Local Screen Agent

A native macOS screen-control agent that uses an OpenAI-compatible tool-calling model plus a LocateAnything grounding server to observe the current screen, locate UI targets, and click them.

Status

This project is a proof of concept and is experimental. It is intended for local development, research, and prototyping, not production use. Expect rough edges around model behavior, permissions, endpoint compatibility, and UI automation reliability.

Features

• SwiftUI macOS chat interface
• Screen Recording and Accessibility permission status
• Full-screen 1x screenshot observation
• LocateAnything-based GUI target localization
• OpenAI-compatible chat completions with tool calls
• Native mouse clicking through Quartz events
• Local settings stored at ~/.local-screen-agent.json

How It Works

1. The app captures the main display and downscales Retina captures to 1x display coordinates.
2. The main model receives the current screen metadata and decides which tool to call.
3. locate_region sends a full-screen or cropped PNG to LocateAnything.
4. LocateAnything returns points/boxes in the same 1x screenshot coordinate space.
5. click_screen maps the selected 1x coordinate back to macOS display coordinates and clicks it.

Requirements

• macOS 13 or newer
• Swift 6.2 toolchain
• A tool-calling OpenAI-compatible chat endpoint
• A LocateAnything-compatible HTTP server exposing /predict-upload
• macOS Screen Recording permission
• macOS Accessibility permission

Build

swift build

Run

For permission-sensitive local testing, use the app bundle script instead of swift run:

./scripts/build_app_bundle.sh debug
open ".build/debug/Local Screen Agent.app"

The script signs the app with LOCAL_SCREEN_AGENT_CODESIGN_IDENTITY when set. If no identity is configured, it falls back to ad-hoc signing for local development.

Configure

Open the app settings and set:

• Main model endpoint, for example http://localhost:11434/v1
• Main model name
• Main model API key, if required
• LocateAnything endpoint, for example http://localhost:8000
• LocateAnything generation mode, output type, temperature, and max tokens
• Request timeout and max tool iterations

The settings file is:

~/.local-screen-agent.json

Permissions

The app needs:

• Screen Recording: capture the current screen
• Accessibility: send mouse click events

Use the Screen and Access buttons in the header to open the relevant System Settings panes. After changing permissions, quit and reopen the app if macOS asks.

App Transport Security

The app includes an Info.plist with NSAllowsArbitraryLoads so local HTTP endpoints such as http://localhost:11434 and http://localhost:8000 can be used during development.

For production distribution, prefer HTTPS endpoints and tighten the ATS policy.

Repository Layout

Sources/
  LocalScreenAgentApp/     SwiftUI app and settings UI
  LocalScreenAgentCore/    Runtime, screenshot capture, grounding, and click automation
scripts/
  build_app_bundle.sh      Builds and signs a local .app bundle

Current Limitations

• Experimental proof-of-concept quality.
• Only mouse clicking is exposed as an automation action.
• Only the main display is captured.
• LocateAnything must be hosted separately.
• The app is designed for local development and has not been notarized.

License

MIT License. See LICENSE.