ExecuTorch Flutter #

A Flutter plugin for on-device ML inference using PyTorch ExecuTorch, supporting Android, iOS, macOS, Windows, Linux, and Web.

pub.dev | Live Demo | Example App

Table of Contents #

• Overview
• Features
• Installation
• Quick Start
• Platform Support
• API Reference
• On-device LLM (Gemma 4)
• Build Configuration
• Advanced Usage
• Web Platform
• Example Application
• Model Export
• Troubleshooting
• Vulkan Backend (Experimental)
• Contributing
• License

Overview #

ExecuTorch Flutter provides a simple Dart API for loading and running ExecuTorch models (.pte files) in your Flutter applications. The package handles all native platform integration, providing you with a straightforward interface for on-device machine learning inference.

Writing a Dart server or command-line tool instead — no Flutter SDK involved? Use executorch_dart directly; this package is a thin wrapper over it that adds Flutter asset-bundle loading and Web support.

Features #

• Cross-Platform: Android (API 23+), iOS (13.0+), macOS (11.0+), Windows, Linux, and Web
• Type-Safe API: dart:ffi bindings with type-safe Dart wrapper classes
• Async Operations: Non-blocking model loading and inference
• Multiple Models: Support for concurrent model instances
• Error Handling: Structured exception handling with clear error messages
• Backend Support: XNNPACK (all platforms), CoreML (Apple), Metal + MLX (macOS), Vulkan (opt-in)
• 13 Tensor Dtypes: float32/64, float16, bfloat16, int8/16/32/64, uint8/16/32/64, bool
• On-device LLM (experimental): streaming text generation with Gemma 4 (XNNPACK CPU + MLX Apple-GPU) — see docs/LLM.md
• Live Camera: Real-time inference with camera stream support

Library Size by Backend #

📊 Download Release Size Comparison (SVG) | Download Debug Size Comparison (SVG) | JSON Report

Installation #

Requirements: Flutter 3.38+ (first version with native assets hooks)

dependencies:
  executorch_flutter: ^0.7.0

Quick Start #

1. Load a Model #

import 'package:executorch_flutter/executorch_flutter.dart';

// Load from Flutter assets (recommended - works on all platforms)
final model = await loadModelFromAsset('assets/models/model.pte');

2. Run Inference #

final inputTensor = TensorData(
  shape: [1, 3, 224, 224],
  dataType: TensorType.float32,
  data: yourImageBytes,
);

final outputs = await model.forward([inputTensor]);

for (var output in outputs) {
  print('Shape: ${output.shape}, Type: ${output.dataType}');
}

3. Clean Up #

await model.dispose();

Model Loading Options #

loadModelFromAsset is a top-level function, not a static method on ExecuTorchModel — this is what changed in 0.6.1 (see CHANGELOG.md).

// From bytes
final byteData = await rootBundle.load('assets/models/model.pte');
final model = await ExecuTorchModel.loadFromBytes(byteData.buffer.asUint8List());

// From file path (native platforms only)
final model = await ExecuTorchModel.load('/path/to/model.pte');

Platform Support #

*Opt-in. Vulkan is experimental — see Vulkan Backend. MLX is the Apple-Silicon GPU runtime used by the LLM path (macOS 14+, arm64) — see docs/LLM.md.

Platform Configuration #

If you encounter deployment target errors, update your project settings:

After updating, run:

flutter clean && flutter pub get && flutter build <platform>

API Reference #

ExecuTorchModel #

// Top-level loader — Flutter asset bundle, all platforms including web
Future<ExecuTorchModel> loadModelFromAsset(String assetPath)

// ExecuTorchModel static factories
static Future<ExecuTorchModel> loadFromBytes(Uint8List modelBytes)
static Future<ExecuTorchModel> load(String filePath)  // Native only

// Inference
Future<List<TensorData>> forward(List<TensorData> inputs)

// Lifecycle
Future<void> dispose()
bool get isDisposed
String get modelId

ExecutorchManager.instance also has loadModelFromAssets(assetPath) — an extension method that does the same thing but caches the model in the manager, like its loadModel/loadModelFromBytes counterparts.

ExecuTorchLLM (experimental) #

On-device generative text — Google Gemma 4 E2B — with token-by-token streaming, separate from the tensor API. Loaded from file paths (weights are 1+ GB) and driven by a stateful decode loop + tokenizer + KV cache. Backends: XNNPACK (CPU, all platforms) and MLX (Apple-Silicon GPU, macOS arm64).

// Load (file paths; mlxMetallibPath is MLX-only)
static Future<ExecuTorchLLM> load({
  required String modelPath,
  required String tokenizerPath,
  String? dataPath,
  String? mlxMetallibPath,
})

// Stream tokens as they decode
Stream<String> generate(String prompt, {GenConfig config})

// Control / lifecycle
void stop();              // cooperative cancel mid-generation
void reset();             // clear KV cache / start a new conversation
Future<void> dispose();   // release native resources

// GenConfig — temperature-only sampling (no top-p/top-k)
const GenConfig({int maxNewTokens, int seqLen, double temperature, bool echo, bool ignoreEos});

final llm = await ExecuTorchLLM.load(
  modelPath: '/path/gemma-4-E2B-it_xnnpack.pte',
  tokenizerPath: '/path/gemma-4-E2B-it_tokenizer.json',
);
// Gemma 4 needs its turn markers around the message:
final prompt = '<bos><|turn>user\nExplain Flutter in one line.<turn|>\n<|turn>model\n';
await for (final piece in llm.generate(prompt,
    config: const GenConfig(maxNewTokens: 512, temperature: 0))) {
  stdout.write(piece);
}
await llm.dispose();

Enable it in pubspec.yaml (hooks.user_defines.executorch_dart):

llm: true
backends: [xnnpack, mlx]   # mlx is auto-dropped off macOS-arm64

📖 Full guide: docs/LLM.md — model export (the Gemma 4 scripts), the chat template, the MLX mlx.metallib shipping step, stopping, platform support, and troubleshooting. A complete streaming chat screen is in example/lib/screens/llm_chat_screen.dart.

TensorData #

final tensor = TensorData(
  shape: [1, 3, 224, 224],       // Dimensions
  dataType: TensorType.float32,  // See dtype table below
  data: Uint8List(...),          // Raw bytes, little-endian
  name: 'input_0',               // Optional
);

Supported dtypes — all 13 map 1:1 to ExecuTorch's native types:

Building data by hand is error-prone, so use ExecutorchManager to encode numeric lists — it handles float16/bfloat16 conversion (round-to-nearest-even) and endianness:

final tensor = ExecutorchManager.instance.createTensorData(
  shape: [1, 4],
  dataType: TensorType.float16,
  data: [1.0, 2.5, -3.25, 0.5],
);

BackendQuery #

Query available backends at runtime:

// Check specific backend
if (BackendQuery.isAvailable(Backend.coreml)) {
  model = await loadModelFromAsset('assets/model_coreml.pte');
} else {
  model = await loadModelFromAsset('assets/model_xnnpack.pte');
}

// List all available backends
final backends = BackendQuery.available;
print('Available: ${backends.map((b) => b.displayName).join(", ")}');

Tokenizer #

Text to token ids and back, independent of any model. Encoder models — embeddings, classification, retrieval — take token ids as an input tensor and produce a vector in a single forward(), so they have no generation loop to borrow a tokenizer from.

final tokenizer = await Tokenizer.load('/path/to/tokenizer.json');

final ids = tokenizer.encode('some text');
final input = TensorData(
  shape: [1, ids.length],
  dataType: TensorType.int64,
  data: Int64List.fromList(ids).buffer.asUint8List(),
);
final outputs = await model.forward([input]);

print(tokenizer.decode(ids));
tokenizer.dispose();

Supported: HuggingFace tokenizer.json built on BPE (GPT-2, Llama, Gemma, Qwen, Mistral), SentencePiece .model, TikToken, llama2.c.

Not supported: WordPiece / BERT-family — that rules out BERT, DistilBERT, MiniLM and most sentence-transformers models. The error names the specific reason rather than reporting a generic parse failure.

Native platforms only. It is dart:ffi throughout with no WebAssembly equivalent, so Web code referencing Tokenizer fails to compile rather than failing at runtime. For generative models use ExecuTorchLLM, which owns its tokenizer internally.

Exception Hierarchy #

ExecuTorchException (base)
├── ExecuTorchModelException      // Model loading/lifecycle
├── ExecuTorchInferenceException  // Inference execution
├── ExecuTorchValidationException // Tensor validation
├── ExecuTorchMemoryException     // Memory/resources
├── ExecuTorchIOException         // File I/O
└── ExecuTorchPlatformException   // Platform communication

Build Configuration #

Configure the native build in your app's pubspec.yaml:

hooks:
  user_defines:
    executorch_dart:
      debug: false              # Enable debug logging
      build_mode: "prebuilt"    # "prebuilt", "local", or "source"
      # prebuilt_version: "1.4.0.6"  # Optional: pin specific native version
      # For source mode: build from local ExecuTorch checkout
      # build_mode: "source"
      # executorch_source: "/path/to/executorch"
      # For local mode: point at pre-compiled libraries
      # local_lib_dir: "/path/to/compiled/libs"
      backends:
        - xnnpack
        - coreml
        - metal

The key under user_defines: is the package that owns the native build — executorch_dart, even though you depend on executorch_flutter. This package used to own the build directly and read executorch_flutter: here; see the 0.6.1 entry in CHANGELOG.md if you're migrating.

Options #

Default Backends by Platform #

Listing backends: replaces the defaults entirely — include every backend you want. vulkan, mlx, and qnn are never on by default. A legacy mps entry is accepted and treated as metal on macOS.

Environment Variables #

Advanced Usage #

Preprocessing Strategies #

The example app demonstrates three preprocessing approaches:

GPU Preprocessing Tutorial - Step-by-step guide with GLSL shader examples.

Web Platform #

Web runs via WebAssembly with XNNPACK backend.

Performance #

When to use Web:

• Demos and prototyping
• Interactive inference (sub-second)
• No app install required

Not recommended for:

• Real-time camera inference
• High-throughput batch processing

Setup #

1. Run setup script:

   dart run executorch_flutter:setup_web
   

2. Add to web/index.html:

   <head>
     <script src="js/executorch_wrapper.js"></script>
   </head>
   

3. Use XNNPACK models (same as native).

Serving models on web #

Host your models on an origin you control, or bundle them as Flutter assets.

On native you can download a .pte from anywhere. On web the browser enforces CORS, and a cross-origin fetch only succeeds if the server sends Access-Control-Allow-Origin. GitHub release assets do not send it, so fetching a model straight from a GitHub release fails in every browser:

Access to fetch at 'https://github.com/.../releases/download/...pte'
blocked by CORS policy: No 'Access-Control-Allow-Origin' header is present

Three approaches that work:

• Bundle the model as an asset and load it with loadModelFromAsset — the simplest option, and the right one when the model ships with the app.
• Serve it from your own origin, same host as the app, so CORS never applies.
• Use a CDN or object store that sends Access-Control-Allow-Origin if the model must be fetched cross-origin.

Remember that label files and any other side-car assets go through the same check — it is easy to fix the model download and still fail on labels.

This repo's own web demo takes the second approach: its deploy workflow copies the models into the published site so they are served from the app's origin.

Example Application #

The example/ directory includes:

• Unified Model Playground - Multiple model types in one interface
• MobileNet V3 - Image classification (1000 ImageNet classes)
• YOLO - Object detection (v5, v8, v11)
• Camera Mode - Real-time inference
• Settings - Thresholds, preprocessing, performance overlay

cd example
flutter run -d macos  # or ios, android, windows, linux, chrome

Converting PyTorch Models to ExecuTorch #

Convert your PyTorch models to .pte format:

Official ExecuTorch Export Guide

Example app models are hosted at executorch_flutter_models and downloaded automatically.

To export manually:

cd models/python
python3 main.py

LLM (Gemma 4) models are exported with dedicated scripts (they need a tokenizer + quantization recipe, not the tensor export path):

python models/python/export_gemma4_xnnpack.py   # CPU model (all platforms)
python models/python/export_gemma4_mlx.py        # Apple-GPU model (macOS)

See docs/LLM.md for the full export recipe, the required tokenizer.json / mlx.metallib, and how to load them with ExecuTorchLLM.

Troubleshooting #

Experimental: Vulkan Backend #

Warning: Vulkan is experimental and opt-in.

Status #

Enable Vulkan #

hooks:
  user_defines:
    executorch_dart:
      backends:
        - xnnpack
        - vulkan

Vulkan Troubleshooting #

Recommendations #

• Production: Use XNNPACK (stable everywhere)
• Apple platforms: Use CoreML (iOS/macOS) or Metal (macOS) instead of Vulkan
• Testing: Report issues with device info and logs

Report Vulkan Issues

Contributing #

Contributions welcome! See CONTRIBUTING.md for guidelines.

Acknowledgments #

• opencv_dart - Referenced for understanding Flutter native assets build patterns and cross-platform FFI packaging

License #

MIT License - see LICENSE.

Support #

• Official ExecuTorch Documentation
• Report Issues
• Roadmap

Built with love for the Flutter and PyTorch communities.