Today’s analysts demand more from machines: a new AI program aims to teach them how

The US Department of Defense (DOD) and the intelligence community require computer systems that can robustly and automatically analyze large amounts of multimedia. These systems must also communicate and cooperate with people to resolve ambiguities and improve performance over time.

However, today’s machine learning approaches result in artificial intelligence (AI) agents that cannot interact with humans through conversation except in limited, specifically designed applications. Current computational paradigms rely on statistical methods and lack sufficiently diverse and representative annotated data for training to achieve the accuracy required for successful implementation. Additionally, these agents lack the ability to understand concepts, such as object properties and abilities, which prevents them from being able to handle previously invisible objects, activities, scenes, or entities.

DARPA’s Environment-Driven Conceptual Learning (ECOLE) program aims to radically improve these technologies by creating AI agents that can continuously learn from linguistic and visual input. The aim is to enable the collaborative human-machine analysis of images, videos and multimedia documents during urgent and critical DOD analysis tasks where reliability and robustness are essential.

“Today’s multimedia analysis systems lack introspection,” said Dr. William Corvey, ECOLE program manager in DARPA’s Office of Information Innovation. “Furthermore, the symbolic representations as they have been constructed in the past are simply not to scale. ECOLE’s central innovation will be to teach AI to learn representations that are faceted and conceptual in nature – such as representations that can be iterated with a human partner; reasonable representations; and easily generalizable representations.

ECOLE’s findings will be broadly applicable to a range of technology sectors – from the Semantic Web community, commercial enterprises that reason about information on the Internet and the robotics industry, to public safety organizations dealing with images or videos for object and activity recognition, and anyone requiring robust and automatic reasoning on image and video data, as required by autonomous vehicles, for example.

Previously, Corvey managed an exploratory effort called Grounded Artificial Intelligence Language Acquisition (GAILA) that investigated aspects of human language acquisition in children. As a result, researchers developed technologies that mirrored a child’s approach to learning. ECOLE seeks to develop this area of ​​research with specific applications to multimedia analysis. The scope of the program will include the development of algorithms capable of identifying, representing and grounding the attributes that form the symbolic and contextual model of a particular object through interactive learning with a human analyst.

“Representing the holistic and extensible representation of media content will require innovation both in unsupervised learning and in elevating that acquired knowledge to the symbolic level,” Corvey said. “Our goal is to bridge the gap between advanced symbolic reasoners that rely on manual feature input and fully unsupervised learning, which currently can only perform tasks such as captioning images at grain of course.”

ECOLE is a four-year program divided into three phases. The first two 18-month phases will involve basic research to create a neuro-symbolic scaffold that will advance the field of computational multimedia analysis for all areas of AI application. During its third 12-month phase, ECOLE will focus on developing concerns related to geospatial intelligence workflows.

Please see the general agency announcement for more information.

Learn more about DARPA

James G. Williams