CLaRa من Apple: يحقق RAG الأصلي للضغط مع الاستدلال الكامن المستمر ضغطًا دلاليًا بمقدار 16x-128x

Retrieval-Augmented Generation (RAG) systems face challenges in maintaining accuracy and efficiency when dealing with large context windows and disconnected retriever and generator components. A new framework, CLaRa (Continuous Latent Reasoning), developed by Apple researchers, addresses these issues by compressing documents into continuous memory tokens and performing retrieval and generation in a shared latent space. This approach aims to shorten context, avoid double encoding, and optimize the system by allowing the generator to guide the retriever on what information is most relevant.

Key Concepts of CLaRa

• Semantic Compression: CLaRa uses a semantic compressor to attach a small number of learned memory tokens to each document. This compression is achieved during Salient Compressor Pretraining (SCP), where a Mistral 7B-style transformer with LoRA adapters switches between compressor and generator roles. The final layer hidden states of the memory tokens become the compressed representation for the document.
• Joint Retrieval and Generation: After offline compression, each document is represented solely by its memory tokens. CLaRa trains a query reasoner and an answer generator on top of the same backbone. The query reasoner maps an input question into memory tokens. Retrieval becomes a simple cosine similarity search between the query embedding and candidate document embeddings. The best compressed document embeddings are then concatenated with the query tokens and fed into the generator adapter.
• Differentiable Top-k Selection: A key innovation is the use of a differentiable top-k selector, implemented with a Straight-Through estimator. During the forward pass, the model employs hard top-k selection. During the backward pass, a softmax distribution over document scores allows gradients from the generator to flow into the query reasoner parameters. This encourages the retriever to prioritize documents that improve answer likelihood.

Training and Evaluation

• Salient Compressor Pretraining (SCP): Trained on approximately 2 million passages from Wikipedia 2021, SCP uses a Qwen-32B model to generate supervision signals, including simple QA pairs, complex QA pairs, and paraphrases. A verification loop ensures factual consistency and coverage.
• Training Losses: Training involves two loss terms: a cross-entropy term to train the generator and a mean squared error (MSE) term to align the average hidden state of document tokens with the average hidden state of the memory tokens. The MSE loss provides consistent gains by keeping compressed and original representations in the same semantic region.
• QA Datasets: The compressor is evaluated on datasets like Natural Questions, HotpotQA, MuSiQue, and 2WikiMultihopQA. Results show that CLaRa can outperform existing compression and retrieval methods, especially in settings where the correct document is within the candidate set.

Performance Highlights

• CLaRa achieves significant improvements in F1 scores compared to baselines like LLMLingua-2 and PISCO, even outperforming a BGE-based text retriever plus full document Mistral-7B generator.
• The system maintains moderate performance even at high compression ratios (above 32x), with the main bottleneck being weak document relevance rather than compression quality.
• In end-to-end QA tasks, CLaRa demonstrates comparable or slightly better performance than DRO-Mistral-7B, which uses full uncompressed text, while utilizing 16 times shorter document representations.
• CLaRa also performs strongly as a reranker, achieving high Recall at 5, surpassing supervised BGE Reranker baselines.

Model Releases

Apple has released three models on Hugging Face:

• CLaRa-7B-Base
• CLaRa-7B-Instruct (an instruction-tuned unified RAG model with built-in document compression at 16x and 128x)
• CLaRa-7B-E2E

Key Takeaways

• CLaRa leverages QA-guided and paraphrase-guided semantic compression to replace raw documents with continuous memory tokens, preserving reasoning signals even at high compression ratios.
• Retrieval and generation are jointly trained in a shared latent space, optimizing both components with a single language modeling loss.
• The differentiable top-k estimator allows gradients to flow from answer tokens to the retriever, aligning document relevance with answer quality.
• CLaRa outperforms strong text-based baselines on multi-hop QA benchmarks and can even beat full-text BGE/Mistral pipelines.
• Apple has released practical models and the full training pipeline, making the framework accessible for further research and development.

In conclusion, CLaRa represents a significant advancement in RAG systems by integrating semantic document compression and joint optimization in a shared continuous space. This approach offers a compelling alternative to traditional chunk-and-retrieve RAG methods, enabling efficient and accurate question answering with significantly shorter context lengths.

Source: MarkTechPost