نماذج لغة الانحدار: التنبؤ بأداء الأنظمة الكبيرة باستخدام النص إلى النص

Large language models (LLMs) are rapidly evolving, moving beyond purely subjective human feedback to incorporate raw, operational data for enhanced prediction capabilities. A new approach detailed in a Google Research blog post, “Performance Prediction for Large Systems via Text-to-Text Regression,” explores using LLMs for numeric regression tasks by processing input as text strings and outputting numerical predictions, also as structured text strings. This eliminates the need for extensive feature engineering and normalizations typically associated with traditional regression methods that require tabular data. The authors demonstrate the effectiveness of this Regression Language Model (RLM) approach in predicting resource efficiency within Google’s Borg compute infrastructure. This blog post summarizes the key findings and implications of this research.

The Challenge of Traditional Regression

Traditional regression methods often struggle with the complexity and dynamic nature of real-world data. Converting unstructured data, such as configuration files and system logs, into a fixed-length numeric vector (tabular format) is a time-consuming and often lossy process. Furthermore, the emergence of new data types requires restarting the entire process from scratch.

• Traditional methods need inputs to be in a tabular format.
• Converting complex data to tabular formats is laborious.
• New data types require the process to be restarted.

Regression Language Models: A Text-to-Text Approach

The proposed solution leverages the power of LLMs to perform regression directly on string representations of data. The core idea is to treat the input (x) as a structured text string and train the RLM to output the performance metric (y) as another string. This eliminates the need for feature engineering and normalizations. The RLM can be pre-trained or even randomly initialized.

• RLMs take string representations of input data.
• RLMs output numerical predictions as structured text strings.
• RLMs can be pre-trained or randomly initialized.
• RLMs are trained using next token prediction via cross-entropy loss.

Predicting Efficiency in Google’s Borg System

The researchers applied the text-to-text regression method to predict the MIPS per GCU (Millions of Instructions Per Second per Google Compute Unit), a key efficiency metric for Google’s Borg system. They trained an RLM with a relatively small two-layer encoder-decoder architecture (60 million parameters) using large amounts of data from multiple regression tasks. The data included system states represented in YAML or JSON format, containing lists of active jobs, execution traces, and textual metadata. To handle input strings exceeding the model’s token limit, they pre-processed the data by prioritizing the most important features, ensuring that only less important information was truncated.

• The method was applied to predict MIPS per GCU in Google’s Borg system.
• An RLM with a two-layer encoder-decoder architecture was used.
• Data was pre-processed to prioritize important features before truncation.

Key Capabilities of RLMs

The research demonstrated three key capabilities of RLMs:

• Density Capture: The RLM can capture probability distributions of y-values, providing insights into the inherent variability and potential range of outcomes. This allows for modeling aleatoric uncertainty (inherent randomness) and potentially identifying epistemic indicators (uncertainty due to limited observation).
• Uncertainty Quantification: The RLM’s prediction uncertainty is correlated with residual squared error, enabling quantification of the model’s confidence in its predictions. This allows the system to rely more heavily on the regressor when confident and fall back to slower but more accurate simulations when uncertain.
• Near-Perfect, Low-Cost Regression: The RLM achieves very precise pointwise regression with strong alignment between predicted and actual values. This enables few-shot adaptation to diverse prediction tasks, making it an adaptable, universal predictor.

Conclusion

This research presents a novel and effective approach to numeric regression using language models. By processing input as text and outputting numerical predictions as structured text, RLMs eliminate the need for traditional feature engineering and offer several advantages, including density capture, uncertainty quantification, and near-perfect regression. This work demonstrates the potential of RLMs for simulating large systems and paves the way for future breakthroughs in reinforcement learning for language models.

Source: Google Research and Google DeepMind