أبحاث مايكروسوفت تٌحرز تقدماً في التكميم منخفض البت للنماذج اللغوية الكبيرة على الأجهزة الطرفية

Large language models (LLMs) are rapidly transforming various applications, but their substantial computational requirements pose a challenge for deployment on resource-constrained edge devices like smartphones and robots. This blog post from Microsoft Research highlights advancements in low-bit quantization techniques that address this issue, enabling efficient LLM operation on edge devices. The researchers introduce innovative approaches to mixed-precision matrix multiplication (mpGEMM) and hardware architectures, paving the way for a new era of AI accessibility.

The Challenge of LLMs on Edge Devices

The primary hurdle in deploying LLMs on edge devices lies in their massive size and the resulting demand for significant memory and computational power. Low-bit quantization offers a solution by compressing models and reducing memory footprint, but traditional hardware limitations impede the full realization of mixed-precision calculations.

• LLMs require hundreds of millions of parameters, leading to high memory and computational demands.
• Edge devices often lack the necessary resources to efficiently run these models.
• Most hardware supports only symmetric computations, hindering the use of mpGEMM.

Ladder: Bridging the Data Type Gap

Ladder is a data type compiler designed to bridge the gap between custom data formats and hardware limitations. By enabling data storage to be separated from computation, Ladder allows for broader support of custom data types without requiring hardware modifications.

• Ladder converts unsupported data types into hardware-compatible ones without data loss.
• It optimizes performance by translating low-bit data into the most efficient formats for the target hardware.
• Evaluations on NVIDIA and AMD GPUs demonstrate speedups of up to 14.6 times compared to existing DNN compilers.

T-MAC: Multiplication-Free mpGEMM

T-MAC (Table-lookup for mpGEMM) is a novel approach that eliminates the need for dequantization and multiplication, enabling efficient mpGEMM on resource-constrained edge devices. It replaces traditional multiplication operations with bit-wise table lookups, reducing computational overhead.

• T-MAC replaces multiplication with bit-wise table lookups, reducing computational demands.
• It achieves significant performance gains on edge devices like the Surface Laptop 7 and Raspberry Pi 5.
• Testing demonstrated 48 tokens per second for the 3B BitNet-b1.58 model on a Surface Laptop 7.

LUT Tensor Core: Hardware for Efficient mpGEMM

LUT Tensor Core is a software-hardware co-design tailored for low-bit LLM inference. It addresses the overhead associated with traditional LUT-based methods and supports various precision levels in mpGEMM. This allows for greater speed and efficiency in edge device applications.

• It optimizes table precomputation and storage through techniques like software-based DFG transformation and operator fusion.
• The hardware design features an elongated tiling shape to promote table reuse and a bit-serial design for various precision combinations.
• Testing shows that LUT Tensor Core achieves 6.93 times the inference speed using only 38.3% of the area of a traditional Tensor Core.

The Future of Embodied AI

The advancements in low-bit quantization not only improve efficiency but also enable model scaling, which enhances model capabilities and expressiveness. These technologies pave the way for embodied AI systems, such as robots, capable of dynamic perception and real-time environmental interaction.

• Low-bit quantization reduces memory and computational demands, making LLMs more accessible.
• Technologies like T-MAC, Ladder, and LUT Tensor Core support efficient operation on edge devices.
• Embodied AI systems could leverage these advancements for dynamic perception and real-time interaction.

In conclusion, Microsoft Research’s innovations in low-bit quantization represent a significant step toward enabling LLMs on edge devices. By overcoming hardware limitations and developing efficient computation methods, these advancements unlock new possibilities for AI applications across a wide range of scenarios, from smartphones to robots. The open-source availability of T-MAC and Ladder further encourages exploration and innovation in this rapidly evolving field.

Source: Microsoft Research