AI Automation Factory (ai-gui-auto-core)

End-to-End Automation Agent (auto-core)

The End-to-End Automation Agent is a decoupled, AI-driven automation framework. It utilizes multimodal Large Language Models (LLMs) for visual perception, strategic planning, and autonomous execution. The system is designed to automate complex tasks across various interfaces without relying on brittle DOM selectors by leveraging computer vision and AI reasoning.

Core Features

Autonomous Agent Loop

• Planner: Generates high-level strategic plans based on user-defined goals and environmental context.
• Thinker: Performs multimodal AI reasoning to decide actions based on live visual state captures and screenshots.
• Supervisor: Orchestrates the execution loop between the host system and isolated browser environments.
• Prosecutor: Validates execution results against expected outcomes to ensure mission success.

Mission Control (Generator)

• Mission Dashboard: Real-time tracking of all automation tasks with status updates (Planning, Ready, Running, Completed, Failed).
• Intelligent Planning: Support for Fuzzy (fast) and Precise (KB-aligned) generation modes.
• Procedural Fidelity: Precise mode breaks down complex tasks into logical UI milestones including Navigation, Initialization, Data Entry, and Finalization.
• Environment Configuration: Deep integration with Project Groups, Credentials Vault, and ConfigMaps (Environment Variables) for context-aware automation.
• Strategic Audit: Users can review and modify AI-generated plans before execution begins.
• Live Execution Streaming: WebSocket-based live logs and screenshot streaming during the execution phase.

Craft Space (Lego-based Workshop)

• Component Registry: Access to basic blocks, Meta Lego blocks, and specialized Agentic Skills.
• Visual Assembly Canvas: Horizontal sidebar for step ordering and high-density tab design (60px height) for space efficiency.
• Dynamic Parameter Management: Visual chips indicating filled, injected, or empty parameters with a dedicated popup configuration menu.
• Flow Overview: Real-time vertical visualization of the entire test flow (e.g., LOGIN -> CREATE -> LOGOUT).
• AI Test Architect: Project-aware chat interface for generating flows using project-specific components and knowledge.

Library and Asset Management

• Centralized Management: Unified view for Test Case history and the Lego-based Craft Space.
• Project-Based Filtering: Global project selector that synchronizes component visibility and AI planning context.
• Meta Lego Blocks: Capability to save composed canvas flows as reusable higher-level building blocks.
• ConfigMap Integration: Group-specific environment configurations used to guide AI planning and execution.

Harness Recovery and Resilience

• Protocol Fallback: Automatic detection of infrastructure-level blocks with seamless fallback from HTTPS to HTTP.
• Interstitial Bypass: Automated handling of browser security warnings (e.g., "Your connection is not private") to ensure continuous execution.
• Fail-Fast Navigation: Strict monitoring of browser error pages (404, site not reached) with immediate failure reporting.

Agentic UI/UX Evaluation

• Multi-dimensional Scoring: Automated visual validation across 6 dimensions (Localization, Visibility, Layout, Color, Security, Clarity).
• Native Localization Check: Evaluation of professional terminology and natural phrasing in target languages.
• Comprehensive Reports: Generation of markdown and JSON reports with severity-ranked findings and objective scoring.

High-Performance Knowledge Engine

• Hybrid RAG System: Combines Vector Store Index and BM25 retrieval for high-precision technical documentation search.
• Retrieve-Only Optimization: Fast retrieval mode that bypasses slow LLM reranking and answer generation during the planning phase.
• Project-Scoped RAG: Intelligent filtering of knowledge snippets based on the target project (e.g., FOS, FIC, FTM).

Technical Stack

• Backend: FastAPI (Python 3.11+)
• Task Queue: Celery with Redis (Sentinel Mode supported)
• Database: SQLAlchemy with SQLite
• Vision and Browser: Playwright-CLI with isolated Docker execution
• AI Integration: fortiai180-multimodal or compatible custom endpoints
• Knowledge Retrieval: LlamaIndex with BM25 and Hybrid Search

System Architecture

FastAPI Backend (app/api/)

Provides RESTful endpoints for task management, group administration, and real-time observation via WebSockets.

Celery Worker (app/engine/worker.py)

Orchestrates high-latency tasks including RAG-based planning, autonomous execution, and asset finalization.

Docker Executor (app/engine/docker_executor.py)

Manages ephemeral container environments to provide secure, clean, and reproducible browser sessions.

Installation and Execution

Dependencies

1. Install Python requirements: pip install -r requirements.txt
2. Install agent-browser: npm i -g agent-browser && agent-browser install

Environment Variables

Configure OPENAI_API_KEY, REDIS_URL, and DB_URL in the .env file.

Running the Stack

• Full Stack: docker-compose up --build
• Backend Only: python main.py
• Worker Only: celery -A app.core.celery_app worker --loglevel=info -P solo

Development Conventions

• Line Limit: 80 characters (PEP8 compliant)
• Type Hinting: Required for all signatures
• Context Safety: Always use task_id context for logging and image identification
• Testing: New features must include verification tests using the internal testing framework

Engineering Mandates

For critical technical constraints, hall of shame records, and architectural mandates, refer to GEMINI.md.