The advancement of machine learning and artificial intelligence has created potential for the automation of the time-consuming process of analyzing, evaluating, and interpreting large quantities of imagery. In order for the large-scale deployment of artificial intelligence in image analysis to be made possible, a new image rating system akin to the more traditional National Interpretability Rating Scale (NIIRS) must be created, so as to aid in the defining of system requirements in the development of new sensors. Through prototyping models using the open-source object detection library Detectron2 alongside proxy imagery from the Overhead Imagery Research Data Set (OIRDS), current limitations faced by machine learning algorithms in the context of target identification in aerial imagery are brought to light, including how they may affect a new image-rating scale, and insights into how such limitations could be overcome.