وحدة المعالجة العصبية MediaTek و LiteRT: إطلاق العنان لقوة الذكاء الاصطناعي على الجهاز

The rise of Neural Processing Units (NPUs) has revolutionized on-device AI, enabling sophisticated generative AI models on edge devices. However, deploying AI on diverse NPUs has been challenging. Google, in collaboration with MediaTek, introduces the LiteRT NeuroPilot Accelerator to simplify NPU deployment and unlock advanced AI capabilities on millions of devices. This blog post summarizes the key features and benefits of this new accelerator.

Simplified and Unified Deployment Workflow

The LiteRT NeuroPilot Accelerator provides a streamlined workflow for deploying AI models on MediaTek NPUs, abstracting away the complexity of individual SDKs. This unified API allows developers to target various MediaTek NPUs with ease.

• Offline (Ahead-of-Time - AOT) Compilation: Ideal for large, complex models where the target SoC is known. AOT compilation reduces initialization costs and lowers memory usage.
• Online (On-Device) Compilation: Suitable for platform-agnostic model distribution of smaller models. The model is compiled on the user’s device during initialization.
• 3-Step Deployment Process: A simplified process that includes AOT compilation (optional), deployment with Google Play for On-device AI (PODAI), and inference using LiteRT Runtime.
• Fallback Mechanism: If the NPU is unavailable, LiteRT automatically falls back to GPU or CPU, ensuring application functionality.

Rich Generative AI Capabilities

The collaboration between Google and MediaTek unlocks the full potential of state-of-the-art open-weight models, including Google’s Gemma family, enabling advanced generative AI features on NPUs.

• Support for Gemma Models: Optimized support for models like Gemma 3 270M, Gemma 3 1B, Gemma 3n E2B, and EmbeddingGemma 300M.
• Performance Boost: Gemma models are accelerated by up to 12x compared to CPU and 10x compared to GPU on MediaTek NPUs.
• LiteRT-LM: A high-level API built on top of LiteRT, designed for easy inference with text generative models.
• EmbeddingGemma Integration: Fits perfectly with LiteRT’s “tensor-in, tensor-out” API for tasks like Retrieval Augmented Generation (RAG) and semantic search.

Efficient, Cross-Platform Development

The LiteRT NeuroPilot Accelerator streamlines development with a new C++ API and Native Hardware Buffer Interoperability.

• Simplified C++ API: An improvement over the previous C API, making it easier to build efficient ML pipelines.
• Native Hardware Buffer Interoperability: Allows for zero-copy data passing from AHardwareBuffer directly to the NPU.
• Zero-Copy Interop: Enables seamless integration between OpenGL/OpenCL buffers and AHardwareBuffer, critical for real-time camera and video applications.
• Reduced Data Conversion: Instead of converting data to and from the CPU, camera frames or video can be passed directly from other ML pipeline components to NPU via LiteRT.

Conclusion

The LiteRT NeuroPilot Accelerator offers a significant advancement in on-device AI deployment, simplifying the process and unlocking the full potential of MediaTek NPUs. With its unified workflow, rich generative AI capabilities, and efficient cross-platform development tools, LiteRT empowers developers to create innovative and performant AI applications that benefit millions of users worldwide. Developers are encouraged to explore the LiteRT example Colab, Sample App, and official Devsite for further documentation and guidance.

Source: Google