Flutter UI Copilot 🤖

"Give your App Eyes and Hands"
Empower your users to interact with your Flutter app using natural language. This plugin connects your UI to Multimodal AI (like OpenAI GPT-4o, Gemini, or Qwen2.5-VL), allowing the AI to "see" the screen and "click" widgets automatically.

Pub Version

✨ Advantages

⚡ Zero-Training: No need to train custom models. It works with standard Multimodal LLMs.
👁️ Visual Understanding: The AI sees exactly what the user sees (screenshots).
🖱️ Auto-Pilot: It can find widgets and click them for the user.
🔌 Multi-Provider: Switch between OpenAI, Google Gemini, Hugging Face, or GitHub Copilot easily.
🛠️ Simple Integration: Just wrap widgets with CopilotTag. No complex accessibility trees or XPaths.

🚀 Installation

Add this to your package's pubspec.yaml file:

dependencies:
  flutter_ui_copilot: ^0.0.1

Or install it from the command line:

flutter pub add flutter_ui_copilot

📖 How to Integrate

1. Initialize the Provider

Wrap your root widget (usually MaterialApp) with CopilotProvider.

import 'package:flutter_ui_copilot/flutter_ui_copilot.dart';

void main() {
  runApp(const MyApp());
}

class MyApp extends StatelessWidget {
  const MyApp({super.key});

  @override
  Widget build(BuildContext context) {
    return CopilotProvider(
      apiKey: "YOUR_API_KEY", // e.g., sk-...
      // Select your AI Brain
      aiProvider: OpenAIProvider(
        model: 'gpt-4o', // or 'Qwen/Qwen2.5-VL-7B-Instruct'
      ),
      child: MaterialApp(
        home: HomeScreen(),
      ),
    );
  }
}

2. Tag Interactive Widgets

Wrap any button, text field, or widget you want the AI to control with CopilotTag.

CopilotTag(
  id: "add_item_btn", // Unique ID
  description: "Button to add a new item to the list", // Description for AI
  actionCallback: () {
     // This logic runs when AI decides to click this button
     _addItem(); 
  },
  child: FloatingActionButton(
    onPressed: _addItem,
    child: const Icon(Icons.add),
  ),
)

3. Send Commands

You can trigger the AI from a chat input, a voice command, or a debug button.

// Inside any widget
final copilot = CopilotProvider.of(context);

// Example User Request
await copilot.sendMessage("Add a new item for me");
// The AI will "see" the screen, find the "add_item_btn", and trigger it.

🛠️ Tasks Performed

The plugin handles the complex "Vision-to-Action" loop for you:

Capture: Takes a screenshot of the current UI.
Contextualize: Collects descriptions of all CopilotTag widgets on screen.
Reason: Sends image + widget list to the AI (LLM).
Action:
- Highlight: Draws a visual spotlight on the target widget.
- Execute: Triggers the actionCallback (Click/Tap).
- Navigate: Can push/pop routes if you wire the callback to Navigator.

🔮 Future Updates

🗣️ Voice Mode: Talk to your app directly (Speech-to-Text integration).
📜 Scroll Support: AI will be able to scroll lists to find off-screen items.
📝 Text Input: AI will be able to type text into fields (e.g., "Fill the form with verify data").
🧠 Local Models: Support for on-device small vision models for privacy.

⚠️ Important Notes

API Costs: Using GPT-4o or Gemini may incur costs from the respective providers.
Permissions: Ensure your app has Internet permission.
- Android: <uses-permission android:name="android.permission.INTERNET"/>
- macOS: Enable "Outgoing Connections (Client)".

🆘 Support

Contact the developer for support, contributions, or updates:

Developer: Praveen
Email: [email protected]
Portfolio: praveen-dev.space
Issues: GitHub Issues