RunAnywhere ONNX Backend #

ONNX Runtime backend for the RunAnywhere Flutter SDK. Provides on-device Speech-to-Text (STT), Text-to-Speech (TTS), and Voice Activity Detection (VAD) capabilities.

Features #

Installation #

Add both the core SDK and this backend to your pubspec.yaml:

dependencies:
  runanywhere: ^0.15.11
  runanywhere_onnx: ^0.15.11

Then run:

flutter pub get

Note: This package requires the core runanywhere package. It won't work standalone.

Platform Support #

Platform Setup #

iOS #

Update ios/Podfile:

platform :ios, '14.0'

target 'Runner' do
  use_frameworks! :linkage => :static  # Required!
  flutter_install_all_ios_pods File.dirname(File.realpath(__FILE__))
end

Add to ios/Runner/Info.plist:

<key>NSMicrophoneUsageDescription</key>
<string>Microphone access is needed for speech recognition</string>

Android #

Add to android/app/src/main/AndroidManifest.xml:

<uses-permission android:name="android.permission.RECORD_AUDIO" />

Quick Start #

1. Initialize & Register #

import 'package:runanywhere/runanywhere.dart';
import 'package:runanywhere_onnx/runanywhere_onnx.dart';

void main() async {
  WidgetsFlutterBinding.ensureInitialized();

  // Initialize SDK
  await RunAnywhere.initialize();

  // Register ONNX backend
  await Onnx.register();

  runApp(MyApp());
}

2. Add Models #

// STT Model (Whisper)
Onnx.addModel(
  id: 'whisper-tiny-en',
  name: 'Whisper Tiny English',
  url: 'https://github.com/RunanywhereAI/sherpa-onnx/releases/download/runanywhere-models-v1/sherpa-onnx-whisper-tiny.en.tar.gz',
  modality: ModelCategory.speechRecognition,
  memoryRequirement: 75000000,  // ~75MB
);

// TTS Model (Piper)
Onnx.addModel(
  id: 'piper-amy-medium',
  name: 'Piper Amy (English)',
  url: 'https://github.com/RunanywhereAI/sherpa-onnx/releases/download/runanywhere-models-v1/vits-piper-en_US-amy-medium.tar.gz',
  modality: ModelCategory.speechSynthesis,
  memoryRequirement: 50000000,  // ~50MB
);

3. Speech-to-Text #

// Download and load STT model
await for (final p in RunAnywhere.downloadModel('whisper-tiny-en')) {
  if (p.state.isCompleted) break;
}
await RunAnywhere.loadSTTModel('whisper-tiny-en');

// Transcribe audio (PCM16 @ 16kHz mono)
final text = await RunAnywhere.transcribe(audioData);
print('Transcription: $text');

// With detailed result
final result = await RunAnywhere.transcribeWithResult(audioData);
print('Text: ${result.text}');
print('Confidence: ${result.confidence}');
print('Language: ${result.language}');

4. Text-to-Speech #

// Download and load TTS model
await for (final p in RunAnywhere.downloadModel('piper-amy-medium')) {
  if (p.state.isCompleted) break;
}
await RunAnywhere.loadTTSVoice('piper-amy-medium');

// Synthesize speech
final result = await RunAnywhere.synthesize(
  'Hello! Welcome to RunAnywhere.',
  rate: 1.0,   // Speech rate
  pitch: 1.0,  // Speech pitch
);

print('Duration: ${result.durationSeconds}s');
print('Sample rate: ${result.sampleRate} Hz');
print('Samples: ${result.samples.length}');

// Play with audioplayers package
// await audioPlayer.play(BytesSource(wavBytes));

API Reference #

Onnx Class #

register()

Register the ONNX backend with the SDK.

static Future<void> register({int priority = 100})

Parameters:

• priority – Backend priority (higher = preferred). Default: 100.

addModel()

Add an ONNX model to the registry.

static void addModel({
  required String id,
  required String name,
  required String url,
  required ModelCategory modality,
  int memoryRequirement = 0,
})

Parameters:

• id – Unique model identifier
• name – Human-readable model name
• url – Download URL (supports .tar.gz, .tar.bz2, .zip)
• modality – Model category (speechRecognition, speechSynthesis)
• memoryRequirement – Estimated memory usage in bytes

Supported Models #

Speech-to-Text (Whisper) #

Recommendation: Use whisper-tiny.en for English-only apps. Use whisper-tiny for multilingual support.

Text-to-Speech (Piper) #

Recommendation: Use amy-medium for good quality English TTS.

Voice Agent Integration #

For full voice assistant functionality, combine STT + LLM + TTS:

import 'package:runanywhere/runanywhere.dart';
import 'package:runanywhere_onnx/runanywhere_onnx.dart';
import 'package:runanywhere_llamacpp/runanywhere_llamacpp.dart';

// Initialize all backends
await RunAnywhere.initialize();
await Onnx.register();
await LlamaCpp.register();

// Load all models
await RunAnywhere.loadSTTModel('whisper-tiny-en');
await RunAnywhere.loadModel('smollm2-360m');
await RunAnywhere.loadTTSVoice('piper-amy-medium');

// Check voice agent readiness
print('Voice agent ready: ${RunAnywhere.isVoiceAgentReady}');

// Start voice session
if (RunAnywhere.isVoiceAgentReady) {
  final session = await RunAnywhere.startVoiceSession();

  session.events.listen((event) {
    if (event is VoiceSessionTranscribed) {
      print('User: ${event.text}');
    } else if (event is VoiceSessionResponded) {
      print('AI: ${event.text}');
    }
  });
}

Audio Format Requirements #

STT Input #

TTS Output #

Troubleshooting #

STT Returns Empty Text #

Possible Causes:

1. Audio too short (< 0.5 seconds)
2. Audio too quiet (no speech detected)
3. Wrong audio format (not PCM16 @ 16kHz)

Solutions:

1. Ensure audio is at least 1 second
2. Check microphone input levels
3. Verify audio format matches requirements

TTS Sounds Robotic #

Solutions:

1. Use *-medium quality models instead of *-low
2. Adjust rate/pitch parameters
3. Try different voice models

Model Loading Fails #

Solutions:

1. Verify model is fully downloaded
2. Check model format compatibility
3. Ensure sufficient memory available

Permission Denied #

iOS:

• Add NSMicrophoneUsageDescription to Info.plist
• Request permission before recording

Android:

• Add RECORD_AUDIO permission to AndroidManifest.xml
• Use permission_handler package to request at runtime

Memory Management #

// Unload STT model to free memory
await RunAnywhere.unloadSTTModel();

// Unload TTS voice
await RunAnywhere.unloadTTSVoice();

// Check current loaded models
print('STT loaded: ${RunAnywhere.isSTTModelLoaded}');
print('TTS loaded: ${RunAnywhere.isTTSVoiceLoaded}');

Related Packages #

• runanywhere — Core SDK (required)
• runanywhere_llamacpp — LLM backend
• runanywhere_onnx — STT/TTS/VAD backend (this package)

Resources #

• Flutter Starter Example
• Documentation
• GitHub Issues

License #

This software is licensed under the RunAnywhere License, which is based on Apache 2.0 with additional terms for commercial use. See LICENSE for details.

For commercial licensing inquiries, contact: san@runanywhere.ai