MMCTAgent: وكيل الذكاء الاصطناعي متعدد الوسائط من Microsoft للاستدلال على بيانات الفيديو والصور واسعة النطاق

Microsoft Research has unveiled MMCTAgent, a novel multimodal AI agent designed to tackle the challenges of reasoning over large-scale, long-form video and image data. Current AI models often struggle with analyzing extended video content or querying across vast multimodal libraries. MMCTAgent, built on Microsoft’s AutoGen, utilizes a Planner-Critic architecture to enable structured reasoning, planning, reflection, and tool-based analysis, addressing the limitations of single-pass inference models.

Key Features and Architecture of MMCTAgent

MMCTAgent stands out with its ability to perform iterative reasoning using modality-specific agents.

• Planner-Critic Architecture: The agent uses a Planner to decompose queries, identify necessary tools, and draft preliminary answers. A Critic then evaluates the Planner’s reasoning, validates evidence, and refines the response for accuracy and consistency.
• Modality-Specific Agents: MMCTAgent incorporates specialized agents like ImageAgent and VideoAgent, each equipped with relevant tools such as get_relevant_query_frames() and object_detection_tool().
• Extensibility: Developers can integrate domain-specific tools, like medical image analyzers or industrial inspection models, making MMCTAgent adaptable to various applications.

VideoAgent: Reasoning Over Long-Form Video

The VideoAgent component is designed explicitly for long-form video analysis and operates in two phases:

• Phase 1: Video Ingestion and Library Creation: This phase involves preprocessing videos to create a searchable knowledge base.

  • Transcription and Translation: Audio is transcribed to text, and multilingual content is translated into a unified language.
  • Key-Frame Identification: Representative frames that signify scene changes are extracted.
  • Semantic Chunking and Chapter Generation: Transcripts and visual summaries are combined into coherent chapters with key frames. Object detection and on-screen text recognition are performed.
  • Multimodal Embedding Creation: Image embeddings are created for keyframes, linking them to corresponding transcript and chapter data.
  • Indexing: Structured metadata is indexed using Azure AI Search for scalable semantic retrieval.

• Phase 2: Video Question Answering and Reasoning: When a user poses a question, the VideoAgent retrieves relevant information and analyzes it using specialized tools.

  • Planner Tools: Includes get_video_analysis, get_context, get_relevant_frames, and query_frame for retrieving video summaries, relevant chapters, key frames, and detailed visual/textual reasoning.
  • Critic Tool: A critic_tool validates the reasoning for temporal alignment, factual accuracy, and coherence.

ImageAgent: Structured Reasoning for Static Visuals

The ImageAgent applies the Planner-Critic architecture to static image analysis, offering modular, tool-based reasoning.

• Planner Tools: Uses tools like vit_tool (Vision Transformer), recog_tool (scene, face, and object recognition), object_detection_tool, and ocr_tool to perform visual understanding and description.
• Critic Tool: The critic_tool validates the Planner’s conclusions, ensuring factual alignment and consistency.

Evaluation and Performance

MMCTAgent has been evaluated using a variety of benchmark datasets, demonstrating improved performance compared to base LLM models. The integration of appropriate tools significantly enhances accuracy. For instance, on the MM-Vet dataset, integrating tools raised GPT-4V’s accuracy from 60.20% to 74.24%. The configurable Critic agent further improves results in critical domains.

Conclusion

MMCTAgent presents a significant advancement in multimodal AI, offering a scalable and adaptable solution for reasoning over large video and image datasets. Its Planner-Critic architecture, modality-specific agents, and extensible toolchain make it a powerful tool for various applications, from medical image analysis to industrial inspection. Microsoft’s commitment to open-source and Azure-native deployment positions MMCTAgent as a valuable asset for developers looking to create innovative multimodal applications. Future efforts will focus on enhancing efficiency, adaptability, and exploring new real-world applications through initiatives like Project Gecko.

Source: Microsoft Research