Flutter Automate

A multi-language automation framework for Android. Supports JavaScript, Python, and other scripting languages via WASM runtime.

δΈ­ζ–‡ζ–‡ζ‘£

Features

  • πŸš€ Multi-language Support - JavaScript, Python, Lua (via WASM)
  • πŸ“± Complete Automation API - UI selectors, gestures, app management, device control
  • πŸ“Έ Screen Capture - MediaProjection screenshot support (Android 10+ foreground service)
  • πŸ”§ Pure Kotlin Implementation - No NDK required, no AutoJS dependency
  • 🎯 Chainable API - Fluent API design
  • πŸ”’ Secure - Scripts run in WASM sandbox

Installation

dependencies:
  flutter_automate:
    git:
      url: https://github.com/jiusanzhou/flutter_automate.git

Quick Start

1. Request Permissions

import 'package:flutter_automate/flutter_automate.dart';

final automate = FlutterAutomate.instance;

// Check and request accessibility service
final hasAccessibility = await automate.checkAccessibilityPermission();
if (!hasAccessibility) {
  await automate.requestAccessibilityPermission(wait: true, timeout: 30000);
}

// Check and request screenshot permission
final hasCapture = await automate.permissions.hasMediaProjection();
if (!hasCapture) {
  await automate.permissions.requestMediaProjection();
}

// Other permissions
await automate.permissions.requestStorage();           // Storage
await automate.permissions.requestManageStorage();     // All files access (Android 11+)
await automate.permissions.requestBatteryOptimizationExemption(); // Battery optimization
await automate.permissions.requestNotificationListener(); // Notification listener

2. Screen Capture

// Requires screenshot permission first
// Handle onActivityResult in MainActivity:
// ScreenCapture.onActivityResult(this, resultCode, data)

// Capture screen
final imageData = await automate.capture.capture();
if (imageData != null) {
  // imageData is Uint8List (PNG format)
  Image.memory(imageData);
}

// Capture and save to file
final success = await automate.capture.captureToFile(
  '/sdcard/Download/screenshot.png',
  quality: 90,
);

// Release resources
await automate.capture.release();

3. UI Automation

// Find element
final button = await automate.text("Login").findOne();

// Click
await automate.text("Login").click();

// Set text
await automate.id("username").setText("hello@example.com");

// Wait for element
final element = await automate.textContains("Success").waitFor(timeout: 5000);

// Chainable selector
final result = await automate
    .selector()
    .className("Button")
    .clickable()
    .findAll();

4. Gesture Operations

// Click at coordinates
await automate.click(500, 800);

// Long press
await automate.longClick(500, 800, duration: 1000);

// Swipe
await automate.swipe(100, 500, 100, 1500, duration: 300);

// Quick swipe
await automate.swipeUp();
await automate.swipeDown();

5. Global Actions

await automate.back();
await automate.home();
await automate.recents();
await automate.openNotifications();
await automate.takeScreenshot();

6. App Management

// Launch app
await automate.app.launch("com.example.app");
await automate.app.launchByName("WhatsApp");

// Get current app
final currentPkg = await automate.app.currentPackage();

// Force stop
await automate.app.forceStop("com.example.app");

// Get installed apps
final apps = await automate.app.getInstalled();

7. Device Info

// Device info
final info = await automate.device.info();
print("Model: ${info.model}");
print("Screen: ${info.screenWidth}x${info.screenHeight}");

// Clipboard
final text = await automate.device.getClipboard();
await automate.device.setClipboard("Hello");

// Vibrate
await automate.device.vibrate(duration: 100);

// Battery
final battery = await automate.device.getBattery();

8. Execute Scripts

// JavaScript
final execution = await automate.execute('''
  console.log("Hello from JS!");
  click(text("Login"));
  sleep(1000);
  swipeUp();
''', language: 'js');

// Python (coming soon)
await automate.execute('''
import automate
automate.click(text("Login"))
''', language: 'python');

Android Configuration

MainActivity

Screenshot feature requires handling permission callback in MainActivity:

// MainActivity.kt
package your.package.name

import android.content.Intent
import io.flutter.embedding.android.FlutterActivity
import im.zoe.flutter_automate.core.ScreenCapture

class MainActivity : FlutterActivity() {
    
    override fun onActivityResult(requestCode: Int, resultCode: Int, data: Intent?) {
        if (requestCode == ScreenCapture.REQUEST_CODE) {
            ScreenCapture.onActivityResult(this, resultCode, data)
            return
        }
        super.onActivityResult(requestCode, resultCode, data)
    }
}

AndroidManifest.xml

<!-- Basic permissions -->
<uses-permission android:name="android.permission.SYSTEM_ALERT_WINDOW" />
<uses-permission android:name="android.permission.FOREGROUND_SERVICE" />
<uses-permission android:name="android.permission.FOREGROUND_SERVICE_SPECIAL_USE" />
<uses-permission android:name="android.permission.FOREGROUND_SERVICE_MEDIA_PROJECTION" />

<!-- Storage permissions -->
<uses-permission android:name="android.permission.READ_EXTERNAL_STORAGE" />
<uses-permission android:name="android.permission.WRITE_EXTERNAL_STORAGE" />
<uses-permission android:name="android.permission.MANAGE_EXTERNAL_STORAGE" />

<!-- Other -->
<uses-permission android:name="android.permission.VIBRATE" />
<uses-permission android:name="android.permission.REQUEST_IGNORE_BATTERY_OPTIMIZATIONS" />

Permissions Reference

Permission Purpose API
Accessibility Service UI control, gesture execution permissions.requestAccessibility()
Overlay Display floating control panel FloatwingPlugin
Screen Capture MediaProjection screenshot permissions.requestMediaProjection()
Storage Read/write files permissions.requestStorage()
All Files Access Android 11+ access all files permissions.requestManageStorage()
Battery Optimization Background keep-alive permissions.requestBatteryOptimizationExemption()
Notification Listener Read system notifications permissions.requestNotificationListener()

Architecture

flutter_automate/
β”œβ”€β”€ lib/
β”‚   └── flutter_automate.dart    # Flutter/Dart API
β”œβ”€β”€ android/
β”‚   └── src/main/kotlin/
β”‚       β”œβ”€β”€ core/                 # Core automation modules
β”‚       β”‚   β”œβ”€β”€ AutomateAccessibilityService.kt
β”‚       β”‚   β”œβ”€β”€ UiSelector.kt
β”‚       β”‚   β”œβ”€β”€ UiObject.kt
β”‚       β”‚   β”œβ”€β”€ GestureEngine.kt
β”‚       β”‚   β”œβ”€β”€ ScreenCapture.kt
β”‚       β”‚   β”œβ”€β”€ ScreenCaptureService.kt
β”‚       β”‚   β”œβ”€β”€ AppUtils.kt
β”‚       β”‚   └── DeviceUtils.kt
β”‚       β”œβ”€β”€ wasm/                 # WASM runtime
β”‚       β”‚   β”œβ”€β”€ ScriptEngineManager.kt
β”‚       β”‚   └── QuickJSEngine.kt
β”‚       └── FlutterAutomatePlugin.kt
└── example/                      # Example app

License

MIT License

Contributing

Issues and Pull Requests are welcome!

Libraries

flutter_automate