hand_detection

Flutter implementation of Google's MediaPipe hand detection and landmark models using TensorFlow Lite. Completely local: no remote API, just pure on-device, offline detection.

Hand Detection with 21-Point Landmarks

Features

• On-device hand detection, runs fully offline
• 21-point hand landmarks with 3D depth information (x, y, z coordinates)
• Handedness detection (left/right hand)
• Gesture recognition: closed fist, open palm, pointing up, thumbs down, thumbs up, victory, I love you
• Truly cross-platform: compatible with Android, iOS, macOS, Windows, and Linux
• The example app illustrates how to detect and render results on images

Quick Start

import 'dart:io';
import 'package:hand_detection/hand_detection.dart';

Future main() async {
  final detector = await HandDetector.create();

  final imageBytes = await File('path/to/image.jpg').readAsBytes();
  List<Hand> hands = await detector.detect(imageBytes);

  for (final hand in hands) {
    final boundingBox = hand.boundingBox;
    final handedness = hand.handedness;

    if (hand.hasLandmarks) {
      final wrist = hand.getLandmark(HandLandmarkType.wrist);
      final indexTip = hand.getLandmark(HandLandmarkType.indexFingerTip);
      print('Wrist: (${wrist?.x}, ${wrist?.y})');
    }
  }

  await detector.dispose();
}

Performance

Hardware Acceleration

HandDetector runs on one of two inference engines, selected at init:

• Interpreter (default). Classic TFLite. CPU via XNNPACK on every platform. GPU only via the platform delegates below, which are deprecated and platform-limited.
• CompiledModel (opt-in: useCompiledModel: true). LiteRT Next. Auto-selects GPU/NPU with automatic CPU fallback on every platform, and it is faster on CPU too (parity-checked: roughly 1.4x to 3.5x vs the plain Interpreter, at or above XNNPACK on most models).

Platform	Interpreter GPU (default engine)	CompiledModel GPU (useCompiledModel: true)
Android	✅ GpuDelegateV2*	✅
iOS / macOS	✅ Metal*	✅
Windows / Linux	❌ CPU only (XNNPACK)	✅
Web	WebGPU via liteRtAccelerator	(n/a)

*Interpreter GPU/Metal delegates are deprecated (removed in flutter_litert 4.0.0). On Windows and Linux, GPU is available only through CompiledModel, because the Interpreter has no desktop GPU delegate.

// Default (Interpreter): CPU everywhere; GPU on Android and Apple only.
final detector = await HandDetector.create();

// CompiledModel: GPU/NPU where available, automatic CPU fallback.
// This is the only GPU path on Windows and Linux.
final detector = await HandDetector.create(useCompiledModel: true);

Accelerator selection (CompiledModel)

When useCompiledModel: true, two optional parameters control the LiteRT Next backend. They have no effect on the default Interpreter engine.

• accelerators (Set<Accelerator>, default {Accelerator.gpu, Accelerator.cpu}). The accelerators the backend may use. The runtime picks the fastest available and falls back through the set. If none initialize it throws, so include Accelerator.cpu to guarantee a fallback. The default requests GPU with CPU fallback.
• precision (Precision, default Precision.fp16). Numeric precision for the compiled graph. Precision.fp32 trades speed for accuracy.

// GPU with automatic CPU fallback (the default).
await HandDetector.create(useCompiledModel: true);

// CPU only, using CompiledModel's fast CPU runtime.
await HandDetector.create(
  useCompiledModel: true,
  accelerators: {Accelerator.cpu},
);

// GPU only. Throws if the GPU backend cannot initialize.
await HandDetector.create(
  useCompiledModel: true,
  accelerators: {Accelerator.gpu},
);

// NPU first, CPU fallback.
await HandDetector.create(
  useCompiledModel: true,
  accelerators: {Accelerator.npu, Accelerator.cpu},
);

// Full fp32 precision.
await HandDetector.create(
  useCompiledModel: true,
  precision: Precision.fp32,
);

Accelerator and Precision are exported from the package.

Advanced Performance Configuration

performanceConfig tunes the Interpreter engine only. It has no effect when useCompiledModel: true.

// Auto mode (default), optimal for each platform
final detector = await HandDetector.create();

// Force XNNPACK (all native platforms)
final detector = await HandDetector.create(
  performanceConfig: PerformanceConfig.xnnpack(numThreads: 4),
);

// Force the Interpreter GPU delegate (Android and Apple only; deprecated, prefer CompiledModel)
final detector = await HandDetector.create(
  performanceConfig: PerformanceConfig.gpu(),
);

// CPU-only (maximum compatibility)
final detector = await HandDetector.create(
  performanceConfig: PerformanceConfig.disabled,
);

Advanced: Direct Mat Input

If you already have a decoded cv.Mat from another OpenCV pipeline, pass it directly:

import 'package:hand_detection/hand_detection.dart';

Future<void> processFrame(Mat frame) async {
  final detector = await HandDetector.create();

  final hands = await detector.detectFromMat(frame);

  frame.dispose(); // always dispose Mats after use
  await detector.dispose();
}

For live camera streams, prefer prepareCameraFrame + detectFromCameraFrame (see below): it keeps cvtColor / rotate / downscale off the UI thread.

Bounding Boxes

The boundingBox property returns a BoundingBox object representing the hand bounding box in absolute pixel coordinates. The BoundingBox provides convenient access to corner points, dimensions (width and height), and the center point.

Accessing Corners

final BoundingBox boundingBox = hand.boundingBox;

// Access individual corners by name (each is a Point with x and y)
final Point topLeft     = boundingBox.topLeft;       // Top-left corner
final Point topRight    = boundingBox.topRight;      // Top-right corner
final Point bottomRight = boundingBox.bottomRight;   // Bottom-right corner
final Point bottomLeft  = boundingBox.bottomLeft;    // Bottom-left corner

// Access coordinates
print('Top-left: (${topLeft.x}, ${topLeft.y})');

Additional Bounding Box Parameters

final BoundingBox boundingBox = hand.boundingBox;

// Access dimensions and center
final double width  = boundingBox.width;     // Width in pixels
final double height = boundingBox.height;    // Height in pixels
final Point center = boundingBox.center;  // Center point

// Access coordinates
print('Size: ${width} x ${height}');
print('Center: (${center.x}, ${center.y})');

// Access all corners as a list (order: top-left, top-right, bottom-right, bottom-left)
final List<Point> allCorners = boundingBox.corners;

Hand Landmarks (21-Point)

The landmarks property returns a list of 21 HandLandmark objects representing key points on the detected hand. Each landmark has 3D coordinates (x, y, z) and a visibility score.

21 Hand Landmarks

Index	Landmark	Description
0	wrist	Wrist
1-4	thumbCMC, thumbMCP, thumbIP, thumbTip	Thumb joints and tip
5-8	indexFingerMCP, indexFingerPIP, indexFingerDIP, indexFingerTip	Index finger
9-12	middleFingerMCP, middleFingerPIP, middleFingerDIP, middleFingerTip	Middle finger
13-16	ringFingerMCP, ringFingerPIP, ringFingerDIP, ringFingerTip	Ring finger
17-20	pinkyMCP, pinkyPIP, pinkyDIP, pinkyTip	Pinky finger

Accessing Landmarks

final Hand hand = hands.first;

// Access specific landmarks by type
final wrist = hand.getLandmark(HandLandmarkType.wrist);
final indexTip = hand.getLandmark(HandLandmarkType.indexFingerTip);
final thumbTip = hand.getLandmark(HandLandmarkType.thumbTip);

if (wrist != null) {
  print('Wrist: (${wrist.x}, ${wrist.y}, ${wrist.z})');
  print('Visibility: ${wrist.visibility}');
}

// Iterate through all landmarks
for (final landmark in hand.landmarks) {
  print('${landmark.type.name}: (${landmark.x}, ${landmark.y})');
}

Drawing Hand Skeleton

Use the handLandmarkConnections constant to draw the hand skeleton:

import 'package:hand_detection/hand_detection.dart';

// Draw skeleton connections
for (final connection in handLandmarkConnections) {
  final start = hand.getLandmark(connection[0]);
  final end = hand.getLandmark(connection[1]);

  if (start != null && end != null) {
    canvas.drawLine(
      Offset(start.x, start.y),
      Offset(end.x, end.y),
      paint,
    );
  }
}

Handedness

The handedness property indicates whether the detected hand is a left or right hand:

final Hand hand = hands.first;

if (hand.handedness == Handedness.left) {
  print('Left hand detected');
} else if (hand.handedness == Handedness.right) {
  print('Right hand detected');
}

Gesture Recognition

Enable gesture recognition to classify hand poses into 7 gestures:

Gesture	Description
closedFist	Closed fist
openPalm	Open palm
pointingUp	Index finger pointing up
thumbDown	Thumbs down
thumbUp	Thumbs up
victory	Victory / peace sign
iLoveYou	"I love you" sign

Enabling Gestures

final detector = HandDetector(
  enableGestures: true,
  gestureMinConfidence: 0.5, // optional, default 0.5
);
await detector.initialize();

final hands = await detector.detect(imageBytes);
for (final hand in hands) {
  if (hand.hasGesture) {
    print('Gesture: ${hand.gesture!.type.name}');
    print('Confidence: ${hand.gesture!.confidence}');
  }
}

Gesture recognition uses a two-stage pipeline (gesture embedder + classifier) and requires HandMode.boxesAndLandmarks (the default mode).

Detection Modes

This package supports two detection modes:

Mode	Features	Speed
boxesAndLandmarks (default)	Bounding boxes + 21 landmarks + handedness	Standard
boxes	Bounding boxes only	Faster

Code Examples

// Full mode (default): bounding boxes + 21 landmarks + handedness
final detector = HandDetector(
  mode: HandMode.boxesAndLandmarks,
);

// Fast mode: bounding boxes only
final detector = HandDetector(
  mode: HandMode.boxes,
);

Configuration Options

The HandDetector constructor accepts several configuration options:

final detector = HandDetector(
  mode: HandMode.boxesAndLandmarks,      // Detection mode
  landmarkModel: HandLandmarkModel.full, // Landmark model variant
  detectorConf: 0.45,                     // Palm detection confidence (0.0-1.0)
  maxDetections: 10,                     // Maximum hands to detect
  minLandmarkScore: 0.5,                 // Minimum landmark confidence (0.0-1.0)
  interpreterPoolSize: 1,                // TFLite interpreter pool size
  performanceConfig: const PerformanceConfig(),    // Performance config (default: auto)
  enableGestures: false,                 // Enable gesture recognition
  gestureMinConfidence: 0.5,             // Minimum gesture confidence (0.0-1.0)
);

Live Camera Detection

For real-time hand detection from a camera feed, use detectFromCameraImage. All processing runs off the UI thread.

Desktop (Windows / macOS / Linux): The default camera package does not include a streaming implementation for desktop platforms. You must also add camera_desktop to your pubspec.yaml, otherwise startImageStream throws UnimplementedError: onStreamedFrameAvailable() is not implemented.

dependencies:
  camera: ^0.12.0
  camera_desktop: ^1.2.0   # required for Windows, macOS, and Linux streaming

import 'package:camera/camera.dart';
import 'package:hand_detection/hand_detection.dart';

final detector = await HandDetector.create();

final cameras = await availableCameras();
final camera = CameraController(
  cameras.first,
  ResolutionPreset.medium,
  enableAudio: false,
  imageFormatGroup: ImageFormatGroup.yuv420, // prevents JPEG fallback on Android; ignored on desktop
);
await camera.initialize();

camera.startImageStream((CameraImage image) async {
  final hands = await detector.detectFromCameraImage(
    image,
    // rotation: rotationForFrame(...), // recommended on Android/iOS
    maxDim: 640,
  );
  // Process hands...
});

Tips:

• Pass rotation: on Android/iOS so the detector sees upright frames. Use rotationForFrame(...) to compute the correct value from sensor orientation and device orientation. On desktop frames are always upright so omit it.
• Pass maxDim: 640 to downscale frames before inference. Recommended: full-res frames waste bandwidth since the model input is much smaller.
• Mirror the overlay on the front camera to match CameraPreview's auto-mirrored texture.
• For advanced use, prepareCameraFrame(...) + detectFromCameraFrame(...) is the lower-level two-step API.

See the full example app for a complete implementation.

Background Processing

All inference runs automatically in a background isolate: the UI thread is never blocked during detection or gesture recognition. No special configuration is needed; HandDetector handles isolate management internally.

Example

The sample code from the pub.dev example tab includes a Flutter app that paints detections onto an image: bounding boxes and 21-point hand landmarks with skeleton connections.