Edge-Veda #

A managed on-device AI runtime for Flutter that keeps text and vision models running sustainably on real phones under real constraints — private by default.

~15,900 LOC | 31 C API functions | 21 Dart SDK files | 0 cloud dependencies

Why Edge-Veda Exists #

Modern on-device AI demos break instantly in real usage:

• Thermal throttling collapses throughput
• Memory spikes cause silent crashes
• Sessions longer than ~60 seconds become unstable
• Developers have no visibility into runtime behavior

Edge-Veda exists to make on-device AI predictable, observable, and sustainable — not just runnable.

What Edge-Veda Is #

A supervised on-device AI runtime that:

• Runs text and vision models fully on device
• Keeps models alive across long sessions
• Enforces compute budget contracts (p95 latency, battery, thermal, memory)
• Auto-calibrates to each device's actual performance via adaptive profiles
• Adapts automatically to thermal, memory, and battery pressure
• Applies runtime policies instead of crashing
• Provides structured observability for debugging and analysis
• Is private by default (no network calls during inference)

Installation #

dependencies:
  edge_veda: ^1.2.0

Text Generation #

final edgeVeda = EdgeVeda();

await edgeVeda.init(EdgeVedaConfig(
  modelPath: modelPath,
  contextLength: 2048,
  useGpu: true,
));

// Streaming
await for (final chunk in edgeVeda.generateStream('Explain recursion briefly')) {
  stdout.write(chunk.token);
}

// Blocking
final response = await edgeVeda.generate('Hello from on-device AI');
print(response.text);

Multi-Turn Conversation #

final session = ChatSession(
  edgeVeda: edgeVeda,
  preset: SystemPromptPreset.coder,
);

await for (final chunk in session.sendStream('Write hello world in Python')) {
  stdout.write(chunk.token);
}

// Model remembers the conversation
await for (final chunk in session.sendStream('Now convert it to Rust')) {
  stdout.write(chunk.token);
}

print('Turns: ${session.turnCount}');
print('Context: ${(session.contextUsage * 100).toInt()}%');

// Start fresh (model stays loaded)
session.reset();

Continuous Vision Inference #

final visionWorker = VisionWorker();
await visionWorker.spawn();
await visionWorker.initVision(
  modelPath: vlmModelPath,
  mmprojPath: mmprojPath,
  numThreads: 4,
  contextSize: 2048,
  useGpu: true,
);

// Process camera frames — model stays loaded across all calls
final result = await visionWorker.describeFrame(
  rgbBytes, width, height,
  prompt: 'Describe what you see.',
  maxTokens: 100,
);
print(result.description);

// Clean up when done
await visionWorker.dispose();

Compute Budget Contracts #

Declare runtime guarantees. The Scheduler enforces them.

final scheduler = Scheduler(telemetry: TelemetryService());

// Auto-calibrates to this device's actual performance
scheduler.setBudget(EdgeVedaBudget.adaptive(BudgetProfile.balanced));
scheduler.registerWorkload(WorkloadId.vision, priority: WorkloadPriority.high);
scheduler.start();

// React to violations
scheduler.onBudgetViolation.listen((v) {
  print('${v.constraint}: ${v.currentValue} > ${v.budgetValue}');
});

// After warm-up (~40s), inspect measured baseline
final baseline = scheduler.measuredBaseline;
final resolved = scheduler.resolvedBudget;

Architecture #

Flutter App (Dart)
    |
    +-- ChatSession ---------- Chat templates, context summarization, system presets
    |
    +-- EdgeVeda ------------- generate(), generateStream(), describeImage()
    |
    +-- StreamingWorker ------ Persistent isolate, keeps text model loaded
    +-- VisionWorker --------- Persistent isolate, keeps VLM loaded (~600MB)
    |
    +-- Scheduler ------------ Central budget enforcer, priority-based degradation
    +-- EdgeVedaBudget ------- Declarative constraints (p95, battery, thermal, memory)
    +-- RuntimePolicy -------- Thermal/battery/memory QoS with hysteresis
    +-- TelemetryService ----- iOS thermal, battery, memory polling
    +-- FrameQueue ----------- Drop-newest backpressure for camera frames
    +-- PerfTrace ------------ JSONL flight recorder for offline analysis
    |
    +-- FFI Bindings --------- 31 C functions via DynamicLibrary.process()
         |
    XCFramework (libedge_veda_full.a, ~15MB)
    +-- engine.cpp ----------- Text inference (wraps llama.cpp)
    +-- vision_engine.cpp ---- Vision inference (wraps libmtmd)
    +-- memory_guard.cpp ----- Cross-platform RSS monitoring, pressure callbacks
    +-- llama.cpp b7952 ------ Metal GPU, ARM NEON, GGUF models (unmodified)

Key design constraint: Dart FFI is synchronous — calling llama.cpp directly would freeze the UI. All inference runs in background isolates. Native pointers never cross isolate boundaries. The StreamingWorker and VisionWorker maintain persistent contexts so models load once and stay in memory across the entire session.

Runtime Supervision #

Edge-Veda continuously monitors device thermal state, available memory, and battery level, then dynamically adjusts quality of service:

Escalation is immediate. Restoration requires cooldown (60s per level) to prevent oscillation.

Performance (Vision Soak Test) #

Validated on physical iPhone, continuous vision inference:

Supported Models #

Pre-configured in ModelRegistry with download URLs and SHA-256 checksums:

Any GGUF model compatible with llama.cpp can be loaded by file path.

Platform Status #

Documentation #

• Full README and source
• API Reference
• Example App

License #

Apache 2.0

Built on llama.cpp by Georgi Gerganov and contributors.