كي بي لام: إنجاز مايكروسوفت في نماذج اللغة الكبيرة القابلة للتطوير والمدمجة بالمعرفة

Large Language Models (LLMs) are powerful, but struggle with efficiently integrating external knowledge. Microsoft Research has introduced Knowledge Base-Augmented Language Model (KBLaM), a novel approach that directly integrates structured knowledge bases into pre-trained LLMs, offering a scalable, efficient, and interpretable alternative to traditional methods like fine-tuning, Retrieval-Augmented Generation (RAG), and in-context learning. This blog post summarizes the key features and benefits of KBLaM.

Overcoming the Limitations of Existing Knowledge Integration Methods

Existing approaches for integrating external knowledge into LLMs have significant drawbacks:

• Fine-tuning: Requires costly retraining of the entire model for every knowledge update.
• Retrieval-Augmented Generation (RAG): Introduces separate retrieval modules, increasing complexity and preventing end-to-end training. RAG also appends retrieved document chunks to prompts.
• In-Context Learning: Suffers from quadratic computational scaling, making it inefficient for large knowledge bases.

KBLaM is designed to overcome these limitations by directly embedding structured knowledge within the model’s architecture.

KBLaM: A Novel Approach to Knowledge Integration

KBLaM uses a structured knowledge base, consisting of entity, property, and value triples, to consolidate and represent the data. The process comprises a three-step pipeline:

• Knowledge Encoding: Each knowledge triple is transformed into a key-value vector pair using a pre-trained sentence encoder with lightweight linear adapters.
  • The key vector (entity name and property) acts as “index information.”
  • The value vector captures the corresponding property value.
• Integration with LLMs: Key-value pairs are augmented into the model’s attention layers using a specialized rectangular attention structure.
  • This structure allows language tokens (user questions) to attend to all knowledge tokens.
  • Knowledge tokens do not attend to each other or back to the language tokens.
  • This reduces computational cost to linear complexity.
• Efficient Knowledge Retrieval: The model learns to dynamically retrieve relevant knowledge tokens during inference, eliminating the need for a separate retrieval step.

Key Advantages of KBLaM

KBLaM offers several advantages over existing methods:

• Scalability: Achieves linear scaling with the size of the knowledge base, enabling it to handle much larger knowledge repositories than traditional in-context learning. KBLaM can process over 10,000 knowledge triples on a single GPU, equivalent to ~200,000 text tokens.
• Efficiency: Rectangular attention significantly reduces computational cost compared to quadratic scaling approaches. Time to first token is significanlty lower than with RAG-like approaches.
• Dynamic Updates: Allows for dynamic updates to the knowledge base without requiring retraining or re-computation of the entire knowledge base.
• Interpretability: Provides insights into how the model utilizes knowledge tokens through attention weights, making the process more transparent than in-context learning.
• Reliability: Enhances model reliability by learning when not to answer questions if the required information is missing, reducing hallucinations. The model refuses to answer questions that aren’t in its knowledge base.

The Future of Knowledge-Augmented AI

KBLaM represents a significant advancement in knowledge-augmented AI, paving the way for AI systems that can provide more accurate, adaptable, and deeply integrated knowledge-driven responses. This approach has the potential to transform various fields, including medicine, finance, and scientific research. Microsoft Research has released KBLaM’s code and datasets to the research community, with plans for integrations with the Hugging Face transformers library.

While the current model is trained primarily on factual question-answer pairs, future research will focus on expanding its capabilities across more complex reasoning tasks and diverse knowledge domains.

Source: Microsoft Research Blog