mlx_localllm #

English | 简体中文

English #

A Flutter plugin for high-performance localized LLM inference on macOS using Apple's MLX framework. It provides zero-latency interaction by loading models directly into the application process.

Features #

• 🚀 Native Acceleration: Built on Apple's MLX for optimal performance on Apple Silicon.
• 📦 In-Process Inference: No external sidecars or servers (like Ollama) required.
• 🌐 Robust Downloader: Built-in support for Hugging Face and mirrors with resilient chunk-based downloading.
• 🛠️ Full Lifecycle: From model discovery and download to stateful conversational inference.

Platform Support #

Requirements #

• macOS: 14.0 or higher.
• iOS: 17.0 or higher.
• Hardware: Apple Silicon (M1, M2, M3, etc.) for inference.
• Flutter: 3.0.0 or higher.

Installation #

1. Add the dependency to your pubspec.yaml:

   flutter pub add mlx_localllm
   

2. Platform Setup: The plugin uses Swift Package Manager (SPM) natively (requires Flutter 3.24+).

   • macOS/iOS: Dependencies are automatically resolved by the Flutter toolchain.
   • Architecture:
     • Apple Silicon (ARM64): Fully supported with native MLX acceleration.
     • Intel (x86_64): Supported via Mocking. You can compile and link on Intel Macs, but inference calls will return an "Unsupported architecture" error. This allows for seamless development across different Mac architectures.

Usage #

Check Support

bool supported = await MlxLocalllm().isSupported();

Download Model

// Track progress via modelEvents stream
MlxLocalllm().modelEvents.listen((event) {
  if (event['type'] == 'downloadProgress') {
    print('Download progress: ${event['value'] * 100}%');
  }
});

await MlxLocalllm().downloadModel('mlx-community/Qwen2.5-0.5B-Instruct-4bit');

Run Inference

await MlxLocalllm().loadModel('mlx-community/Qwen2.5-0.5B-Instruct-4bit');
String response = await MlxLocalllm().generate(prompt: 'Hello, who are you?');
print(response);

简体中文 #

基于 Apple MLX 框架的 macOS/iOS 高性能本地大模型推理 Flutter 插件。通过将模型直接加载到应用进程中，提供零延迟的交互体验。

特性 #

• 🚀 原生加速: 基于 Apple MLX 针对 Apple Silicon 芯片深度优化。
• 📦 进程内推理: 无需安装 Ollama 等外部服务，零延迟通信。
• 🌐 鲁棒下载器: 内置分块下载机制，支持 Hugging Face 及其镜像站。
• 🛠️ 多平台支持: 同时支持 macOS 和 iOS 平台。
• 💻 架构兼容: 支持 x86 架构 Mock 编译，确保 Intel Mac 开发者也能正常运行项目。

平台支持 #

系统要求 #

• macOS: 14.0 及以上。
• iOS: 17.0 及以上。
• 硬件: Apple Silicon 系列芯片 (M1, M2, M3 等) 仅在这些芯片上支持推理。
• Flutter: 3.0.0 及以上。

安装指南 #

1. 在 pubspec.yaml 中添加依赖：

   flutter pub add mlx_localllm
   

2. 平台配置:

   • macOS/iOS: 插件使用 Swift Package Manager (SPM) 管理 MLX 依赖。Flutter (3.24+) 会自动处理这些依赖。
   • 架构要求:
     • Apple Silicon (ARM64): 原生支持，具备 MLX 加速。
     • Intel (x86_64): 支持 Mock 编译。您可以在 Intel Mac 上正常编译和链接项目，但推理调用将返回“不支持的架构”错误。这极大地方便了跨架构的协同开发。

快速开始 #

硬件支持检测

bool supported = await MlxLocalllm().isSupported();

模型下载

// 通过 modelEvents 流监听进度
MlxLocalllm().modelEvents.listen((event) {
  if (event['type'] == 'downloadProgress') {
    print('下载进度: ${event['value'] * 100}%');
  }
});

await MlxLocalllm().downloadModel('mlx-community/Qwen2.5-0.5B-Instruct-4bit');

执行推理

await MlxLocalllm().loadModel('mlx-community/Qwen2.5-0.5B-Instruct-4bit');
String response = await MlxLocalllm().generate(prompt: '你好，请做下自我介绍');
print(response);

许可证 #

MIT License.