AutoCua

Open source alternative to OpenAI Operator and Simular Sai

191 stars
MIT
Active this week
Last commit 1 day ago

Last verified Oct 21, 2026

Automate any desktop app, browser, terminal, or iOS device using real OS-level input events and an accessibility-tree view of every screen.

Visit website GitHub191

Read the full breakdownWho it is for, how it works, trade-offs, the quick start and the FAQ

who it's for

Who AutoCua is for#

QA engineers automating cross-application regression tests

AutoCua can run a single test that opens a desktop app, reads data from it, pastes into a browser form, and verifies the result without needing separate automation tools for each surface. The shell-use mode lets the agent write test helper scripts in a real terminal mid-run, keeping the test logic in one place.

Skip if:

Your test suite lives entirely within a single web application. Dedicated browser automation tools like Playwright handle that case with less configuration overhead and a larger ecosystem of plugins and reporters.

AI researchers comparing model performance on computer-use tasks

Switching among 48 models across 9 providers is a single parameter change in run_agent(). With no project telemetry, benchmark runs do not send task details to a vendor. The multi-agent compression keeps long evaluation tasks from failing at token limits, so the same complex task can run to completion across multiple models.

Skip if:

You need a hosted benchmark platform with team access controls, reproducible sandboxed environments, and audit logs. AutoCua has no built-in multi-user management or environment isolation.

Developers building automation pipelines across desktop and browser

A single run_agent() call can move data from a PDF viewer through a spreadsheet and into a browser form in sequence. The accessibility-tree view works across native desktop apps, Chrome, and the shell without requiring separate API integrations for each surface.

Skip if:

Your pipeline involves only web pages with a stable DOM structure. A headless browser library handles that case with less setup than AutoCua requires.

Privacy-sensitive teams running local model inference

Because AutoCua sends only the API call to the model provider you configure, teams running a local endpoint keep all data on-device: no screenshots leave the machine, and no usage data goes to the AutoCua project. This covers Ollama, LM Studio, or any private API that speaks an OpenAI-compatible protocol.

Skip if:

You need enterprise-grade audit trails, role-based access controls, or SLA-backed uptime. AutoCua has no managed service or enterprise support tier.

the problem

The problem it solves#

Automating a real desktop workflow is harder than it looks. Browser automation libraries drive a browser through a debugging port, which browsers flag and anti-bot systems detect. Screen-coordinate automation breaks on high-DPI displays and whenever a window changes size or position. None of these approaches cross application boundaries: a workflow that moves data from a PDF reader into a spreadsheet and then files a form in a browser requires three separate automation tools that were never designed to share state.

Windows UAC elevation prompts stop user-mode automation cold: the secure desktop that UAC shows cannot receive synthetic input from user-level code. Linux desktop automation has historically required knowledge of AT-SPI bindings that most frameworks leave undocumented. Privacy-sensitive teams face a further problem: most managed computer-use services require sending screenshots of your screen to a cloud endpoint with no option to run locally.

how AutoCua solves it

How it solves it#

Cross-Platform Desktop Automation with Real Input Events

Drives any desktop application on macOS, Windows 10/11, and Linux GNOME using the operating system's own input layer: kCGHIDEventTap on macOS, pywinauto's SendInput on Windows, and AT-SPI on Linux. Applications see the same mouse and keyboard events they receive from a physical device, so accessibility APIs, native UI controls, and anti-automation detection all respond normally. Supports clicks, multi-line text paste via clipboard, drag gestures, hotkeys, and AppleScript on macOS.

Browser Agent via Chrome Extension

Drives your own Chrome profile through the AutoCua extension, which Chrome loads at launch without a store install. Chrome shows its own debugging bar and a Cancel button that returns control to you immediately. A headless path opens a separate Chrome instance on a debugging port when needed for unattended runs. The page scanner never calls Runtime.enable and never injects numbered labels into the DOM; interaction points are painted into the screenshot only.

Accessibility Tree and OCR Screen Reading

Each step fuses the OS accessibility tree with OCR to produce a numbered element list. On macOS, Apple Vision runs at the accurate level while the accessibility walk is still in progress; anything under 1 percent visible in the real window stack is discarded. On Windows, WinRT OCR, taskbar walk, Win32, and UIA scans run in parallel and merge at 5 px. The model picks a numbered element, never a pixel coordinate; a number not on the list is rejected before any input fires.

iOS Simulator and iPhone Automation

Drives an iOS Simulator by default or a USB-paired iPhone or iPad from a macOS host. Agent mode is set to 'mobile use, ios' in the run_agent() call. WebDriverAgent is cloned automatically on the first iOS run; full Xcode and git are the only prerequisites, with no manual setup required beyond that. iOS automation requires a macOS host; Android support is not yet available.

Multi-Agent Context Compression

A separate compression agent monitors the transcript during a run and splices a handoff summary into it when the context approaches the limit. This lets long multi-step workflows finish rather than fail at the token boundary. The desktop app's memory ring shows the compression event count in real time. Without this, complex tasks spanning many application interactions would require manual restarts and state recovery between steps.

48 Models Across 9 Providers

Model selection is a single parameter in the run_agent() call. Pass any model name from model_list.txt in the repository along with the provider key in your .env file. Providers include Anthropic, OpenRouter, and others; the full list is in model_list.txt. A model name not on the list is forwarded verbatim to the provider API and returns the provider's own error, making it easy to test new model releases as they ship.

strengths · trade-offs

Strengths and trade-offs#

Strengths

Real OS-Level Input, Not Synthetic AutomationAutoCua posts events at the operating system's own input layer rather than through a debugging port, automation flag, or JavaScript injection. On macOS, events go to kCGHIDEventTap, the same tap a physical mouse feeds. On Windows, real SendInput calls are used; a kernel-mode driver handles UAC-elevated windows. Applications, browsers, and anti-bot systems all see normal user input events, because they are exactly that.
No Telemetry, Local Data StorageAutoCua sends nothing to the project: the only outbound traffic is the API call to the model provider you configure. Screenshots are processed locally. Session data, conversations, provider keys, and browser profiles all write to an AutoCua_data directory on your own machine. Teams running a local model endpoint such as Ollama or LM Studio can operate entirely offline, with no data leaving the device.
MIT License with No Commercial RestrictionsThe MIT license permits commercial use, modification, redistribution, and embedding in proprietary products without restriction. The only dependency with different terms is the optional Interception kernel driver for Windows UAC handling, which is obtained separately from its author under LGPL-3.0 non-commercial terms; it is never required for AutoCua to function, and every other mode works without it.
Prebuilt Binaries on All Major PlatformsThe pip wheel ships prebuilt Rust binaries for the browser agent on Windows x64, Apple Silicon macOS, and Linux x86_64. No Rust toolchain or C compiler is needed on supported platforms. WebDriverAgent for iOS clones itself on first use. The optional Electron desktop app downloads on first launch when you confirm it; without it the same app opens in pywebview with no further setup.

Trade-offs

-Linux Requires One-Time System Package InstallThe Linux wheel requires a GNOME desktop (tested on Ubuntu with Wayland; X11 is not tested) and a one-time apt install of AT-SPI, GTK 3, XDG desktop portal, and related bindings. Ubuntu and Debian also require a virtualenv built on the distro's own Python 3 version to load system PyGObject correctly; a venv on a different minor version fails with a gi import error. This is a documented one-time setup, but it is more involved than the macOS or Windows setup scripts.
-iOS Automation Requires macOS; Android Not Yet AvailableMobile automation only runs on a macOS host. iOS support covers simulators and USB-paired iPhones and iPads; Android automation is listed in the roadmap but not yet available. Teams that need cross-platform mobile automation or that run automation infrastructure on Linux or Windows cannot use the mobile surface at all.
-Windows UAC Requires a Kernel Driver Not Installable via pipAnswering Windows UAC elevation prompts requires the Interception kernel-mode driver, which pip cannot install because it must be loaded at system boot. Without it, every other mode still works, but any task that hits an elevation prompt stops there. Installing the driver requires running setup_windows.bat as administrator and rebooting once; it is never bundled due to its dual-license terms.
-Project Is Under Two Weeks OldAutoCua was created on 2026-10-08 and had 191 stars and 1 open issue at research time. Active development is evident from daily commits, but API stability, long-term maintenance trajectory, and community size are not yet established. Teams evaluating this for production workflows should track the repository's release cadence and issue volume over the coming months before committing to it.

versus alternatives

AutoCua vs alternatives#

AutoCua vs OpenAI Operator

Both automate computer tasks through AI, but they run in different environments. OpenAI Operator is a managed cloud service that operates inside a sandboxed browser; AutoCua is an open source tool you run on your own machine, able to drive any desktop app, browser, terminal, or iOS device.

Feature	AutoCua	OpenAI Operator
License	MIT	Proprietary
Self-hosting	Yes	No
Desktop automation	Yes	No
iOS automation	Yes (macOS host)	No
Model choice	48 models, 9 providers	OpenAI models only
Data handling	Local only	Cloud-processed
Cost	Free (pay for model API)	ChatGPT Plus or API

AutoCua is the better choice when you need native desktop or iOS automation, when screenshots must stay on your machine, or when you want to run the same workflow across different model providers for cost comparison. OpenAI Operator is the right fit when you need only browser automation and prefer a fully managed service with no local setup; its chat interface is also more accessible to non-developers who prefer not to write Python.

AutoCua vs Simular Sai

Simular Sai is a commercial computer-use product for professional desktop automation. Both tools drive desktop apps with OS-level input, but differ in license, hosting model, and platform breadth.

Feature	AutoCua	Simular Sai
License	MIT	Proprietary
Self-hosting	Yes	No
Platforms	Windows, macOS, Linux, iOS	Windows, macOS
Model choice	48 models, 9 providers	Limited
Cost	Free (model API cost only)	Commercial

AutoCua fits teams embedding computer-use into pipelines, running in CI, or keeping data entirely on-premise. Simular Sai may suit teams that want a commercial product with vendor support and no configuration overhead, or non-technical users who need a polished managed interface rather than a pip-based setup.

install · quick start

Quick start#

bash

Install via pip on Python 3.10 or later; Linux users need AT-SPI system packages before running this step.
```bash
pip install AutoCua
```

tech stack · detected from GitHub

What it's built on#

Languages: JavaScriptPythonRust
Frameworks: Flask

frequently asked

FAQ#

Does AutoCua work on Windows and Linux, not just macOS?

AutoCua supports Windows 10 and 11, Linux (GNOME desktop, tested on Ubuntu with Wayland), and macOS on Apple Silicon or Intel. On Linux, a one-time apt install of AT-SPI and GTK bindings is required before use. iOS mobile automation requires a macOS host; there is no iOS path on Windows or Linux.

Which AI models does AutoCua support?

AutoCua supports 48 models across 9 providers. The full list is in model_list.txt in the repository. You pass a provider name and model name as parameters to run_agent(); the names are case-sensitive. A model name not in the list is forwarded verbatim to the provider API and returns the provider's own error response.

Does AutoCua send screenshots or usage data to any external service?

No. AutoCua sends nothing except the API call to your chosen model provider. Screenshots are processed locally and never transmitted by the AutoCua runtime. Session data, conversations, and provider keys write to an AutoCua_data directory on your machine. If you use a local model endpoint, all inference also stays on-premise.

Is AutoCua free for commercial use?

AutoCua is MIT licensed, so commercial use, modification, and redistribution are permitted with no fees to the project. You pay only for the model API calls your runs make. The one dependency with different terms is the optional Interception driver for Windows UAC handling, which is dual-licensed LGPL-3.0 non-commercial for free use; commercial use of that driver requires a paid license from its author.

How is AutoCua different from Playwright or Selenium?

Playwright and Selenium drive a browser through a debugging port and cannot automate native desktop apps. AutoCua drives Chrome through a browser extension and posts input events at the OS level, so it can automate any desktop app, shell terminal, and iOS device in the same run. For web-only testing with a stable DOM, Playwright is simpler and more mature; AutoCua is the right choice when your workflow crosses multiple application surfaces or includes native apps.

also worth a look

Similar open-source tools#

Atomic Agent

Alternative toGrok Bot

Local-first AI agent that runs on your machine, drives real tools.

3,425 GitHub stars
Licence MIT

rakazo

Featured

Alternative toGrok Bot

AI teammates you own: your keys, your model, your machine.

4,904 GitHub stars
Licence Apache-2.0

Voice Assistant

Alternative toSuperwhisper

Voice interface for Claude Code with local Whisper and spoken answers

2 GitHub stars
Licence MIT

OCRmyPDF

Alternative toABBYY FineReader PDF

Make scanned PDFs searchable and copy-pasteable in one command.

35,105 GitHub stars
Licence MPL-2.0

JellyBoxPlayer

Alternative toSymfonium

Native Jellyfin and Emby music player for every platform

750 GitHub stars
Licence AGPL-3.0

FckSignups

InCustomer Support

Open-source tools that work instantly, no signup required

4,511 GitHub stars
Licence GPL-3.0

by category

Open source tools in these categories#

404

4,904 GitHub stars
Licence Apache-2.0
#AI Agents

Omnara

Alternative toClaude Managed Agents

Open-source agent deployment API. Self-host or use Omnara Cloud.

2,957 GitHub stars
Licence Apache-2.0
#AI Agents

Inbox Zero

Alternative toSuperhuman

AI email assistant that drafts replies and clears your inbox

12,712 GitHub stars
Licence AGPL-3.0
#Email

Feature your project

The projects we add and every new guide. Free, unsubscribe any time.

Find open source alternatives to paid software

AutoCua vs OpenAI Operator

Feature	AutoCua	OpenAI Operator
License	MIT	Proprietary
Self-hosting	Yes	No
Desktop automation	Yes	No
iOS automation	Yes (macOS host)	No
Model choice	48 models, 9 providers	OpenAI models only
Data handling	Local only	Cloud-processed
Cost	Free (pay for model API)	ChatGPT Plus or API

AutoCua vs Simular Sai

Simular Sai is a commercial computer-use product for professional desktop automation. Both tools drive desktop apps with OS-level input, but differ in license, hosting model, and platform breadth.

Feature	AutoCua	Simular Sai
License	MIT	Proprietary
Self-hosting	Yes	No
Platforms	Windows, macOS, Linux, iOS	Windows, macOS
Model choice	48 models, 9 providers	Limited
Cost	Free (model API cost only)	Commercial

Who AutoCua is for#

QA engineers automating cross-application regression tests

Skip if:

AI researchers comparing model performance on computer-use tasks

Skip if:

Developers building automation pipelines across desktop and browser

Skip if:

Privacy-sensitive teams running local model inference

Skip if:

The problem it solves#

How it solves it#

Cross-Platform Desktop Automation with Real Input Events

Browser Agent via Chrome Extension

Accessibility Tree and OCR Screen Reading

iOS Simulator and iPhone Automation

Multi-Agent Context Compression

48 Models Across 9 Providers

Strengths and trade-offs#

Strengths

Trade-offs

AutoCua vs alternatives#

AutoCua vs OpenAI Operator

AutoCua vs Simular Sai

Quick start#

What it's built on#

FAQ#

Similar open-source tools#

Open source tools in these categories#

AI & Machine Learning

Developer Tools

Business & Productivity

Find open source alternatives to paid software

Who AutoCua is for#

QA engineers automating cross-application regression tests

Skip if:

AI researchers comparing model performance on computer-use tasks

Skip if:

Developers building automation pipelines across desktop and browser

Skip if:

Privacy-sensitive teams running local model inference

Skip if:

The problem it solves#

How it solves it#

Cross-Platform Desktop Automation with Real Input Events

Browser Agent via Chrome Extension

Accessibility Tree and OCR Screen Reading

iOS Simulator and iPhone Automation

Multi-Agent Context Compression

48 Models Across 9 Providers

Strengths and trade-offs#

Strengths

Trade-offs

AutoCua vs alternatives#

AutoCua vs OpenAI Operator

AutoCua vs Simular Sai

Quick start#

What it's built on#

FAQ#

Similar open-source tools#

Open source tools in these categories#

AI & Machine Learning

Developer Tools

Business & Productivity

Find open source alternatives to paid software