E
Google experiments with a tool to enable on-device AI.
Google’s new demo MediaPipe LLM Inference API lets developers run AI models on devices like laptops and phones that don’t have the same computing power as servers.
This new release enables Large Language Models (LLMs) to run fully on-device across platforms. This new capability is particularly transformative considering the memory and compute demands of LLMs, which are over a hundred times larger than traditional on-device models. Optimizations across the on-device stack make this possible, including new ops, quantization, caching, and weight sharing.
Large Language Models On-Device with MediaPipe and TensorFlow Lite - Google for Developers
[developers.googleblog.com]
Follow topics and authors from this story to see more like this in your personalized homepage feed and to receive email updates.
Loading comments
Getting the conversation ready...
