Meta Teaches AI to See the World Like Us With OpenEQA
Envisioning a World Where AI Understands Our Reality
Meta’s Bold Move: Empowering AI to Decode Our World and Elevate Its Intelligence.
This Thursday, the tech giant unveiled OpenEQA, a groundbreaking initiative aiming to equip AI with a deeper understanding of its surroundings. Open-Vocabulary Embodied Question Answering (OpenEQA) heralds a new era, enabling AI to actively perceive and interact with its environment.
Additionally, by providing sensory inputs and spatial awareness, this open-source framework empowers AI to glean insights, ‘visualize’ its surroundings, and enhance human interactions with abstract AI assistance.
“Imagine an embodied AI agent that acts as the brain of a home robot or a stylish pair of smart glasses,” Meta explained. “Such an agent needs to leverage sensory modalities like vision to understand its surroundings and be capable of communicating in clear, everyday language to effectively assist people.”
That’s why Meta decided to make OpenEQA open source. Creating an AI model that mirrors human perception and offers contextual insights is immensely challenging. Meta envisions a collaborative effort among researchers, technologists, and experts to turn this vision into reality.
OpenEQA: A Breakthrough Benchmark for Embodied AI
Initially, in crafting the OpenEQA dataset, Meta’s researchers embarked on a multi-step process. Initially, they amassed video data and 3D scans of real-world environments. Subsequently, they presented these videos to human participants, soliciting questions they would pose to an AI equipped with access to such visual data.
The resultant 1,636 questions test various perception and reasoning abilities. For example, the question ‘How many chairs are around the dining table?’ requires the AI to recognize and count objects based on spatial concepts. Other inquiries necessitate the AI to possess fundamental knowledge regarding object uses and attributes.
Moreover, each question is accompanied by answers provided by multiple humans, acknowledging the diverse ways questions can be addressed. Consequently, to evaluate AI agent performance, researchers leveraged large language models. The purpose is to automatically gauge the similarity between AI-generated answers and human responses.
Related Article:
Meta Introduces Ego-Exo4D: A Dataset For Video Learning
Newsletter