RunAnywhere LlamaCpp Backend #

LlamaCpp backend for the RunAnywhere Flutter SDK. Provides high-performance on-device LLM text generation capabilities.

Features #

• On-Device LLM Inference: Run large language models directly on device
• Streaming Generation: Real-time token-by-token output
• GGUF Model Support: Compatible with all GGUF quantized models
• Memory Efficient: Optimized for mobile devices
• Privacy-First: All processing happens locally

Installation #

Add both the core SDK and this backend to your pubspec.yaml:

dependencies:
  runanywhere: ^0.15.8
  runanywhere_llamacpp: ^0.15.8

Quick Start #

1. Initialize and Register #

import 'package:runanywhere/runanywhere.dart';
import 'package:runanywhere_llamacpp/runanywhere_llamacpp.dart';

void main() async {
  WidgetsFlutterBinding.ensureInitialized();

  // Initialize the SDK
  await RunAnywhere.initialize();

  // Register the LlamaCpp backend
  await LlamaCpp.register();

  runApp(MyApp());
}

2. Add a Model #

// Add a GGUF model for text generation
LlamaCpp.addModel(
  name: 'SmolLM2 360M',
  url: 'https://your-model-url.com/smollm2-360m-q4.gguf',
  memoryRequirement: 500000000, // ~500MB
);

3. Generate Text #

// Streaming generation
final stream = RunAnywhere.generateStream(
  'Explain quantum computing in simple terms',
  options: RunAnywhereGenerationOptions(
    maxTokens: 256,
    temperature: 0.7,
  ),
);

await for (final token in stream) {
  print(token); // Print each token as it's generated
}

// Non-streaming generation
final response = await RunAnywhere.generate(
  'What is 2 + 2?',
  options: RunAnywhereGenerationOptions(),
);
print(response.text);

Supported Model Formats #

This backend supports GGUF models, which are quantized versions of large language models optimized for efficient inference. Popular model sources include:

• Hugging Face
• TheBloke's GGUF Models

Platform Support #

Requirements #

• runanywhere core SDK package

License #

This software is licensed under the RunAnywhere License, which is based on Apache 2.0 with additional terms for commercial use. See LICENSE for details.

For commercial licensing inquiries, contact: [email protected]

Support #

• GitHub Issues
• Email: [email protected]