Tongyi-MAI/OpenGCUA
Tongyi-MAI/OpenGCUA has 604 GitHub stars and 52 forks. GitHot measured +2 stars in 24 hours. Last checked 2026-11-15 17:17 UTC. Last push 2026-10-26.
Measured star growth
- in 24h
- +2
- in 7 days
- Update delayed
- in 15 days
- Update delayed
- in 1 month
- Update delayed
Net star changes over overlapping periods, not daily totals. Missing measurements are left blank. Stars measure attention, not software quality.
About
An open-source stack for generalist computer-use agents, combining code execution, GUI interaction, and cross-device control.
Developer ToolsREADME.md
OriginalGPT-6 made computer use the frontier. We make it open.
News · Results · Demos · Quick start · SDK · Open stack · How it works · Contributing · Contact
Computer use has reached a turning point. GPT-6 Astra's demos offered an exciting vision of what computer-use agents could make possible, and the focus is expanding from GUI-only interaction to both visual and programmatic interfaces. The question has changed, from how well a model operates a GUI to how well a system completes work on a real computer by whatever means works.
We call such a system a generalist computer-use agent (GCUA). It uses a computer in more ways than you do. It reads the accessibility tree and calls app APIs, runs shell commands, and operates the GUI when it needs to. With code execution, it writes GUI operations as code and combines them with all of these in one program, with loops and conditions. It can also share the computer with you, working in background windows without interrupting what you are doing.
Yet frontier systems remain largely closed. We therefore introduce Open Generalist CUA (OpenGCUA), an ongoing project to develop and open the full stack for GCUAs with the community, including model, system, trajectories, efficiency optimization, and evaluation.
What we open in this version:
- 🧠 OSWorld integration example with reference screenshot-action and code-action agents for integrating the OpenGCUA design.
- 📜 Evaluation trajectories of the 8 frontier models, including GPT-6 Astra, Claude Opus 5, and Gemini 3.8 Flash.
- 💻 macOS real-device harness (preliminary), a menu-bar app that runs the agent on your own Mac.
- 🔌 Cross-Device SDK (preliminary), one programming interface for computers, browsers, and phones.
🗺️ See the detailed plan for our open stack.
🔥 News
- [2026-10] 🚀 The OpenGCUA research preview is out, with the tech report, the OSWorld integration example, the macOS real-device harness, the Cross-Device SDK, and evaluation trajectories of 8 frontier models.
📊 Results
We test the design on two fronts: OSWorld-v2 with eight frontier models, and real Macs with our real-device harness.
- Every model improves, including GPT-6 Astra, Claude Opus 5, and Gemini 3.8 Flash. Over a GUI-only baseline with batched actions, the mean partial score rises from 45.3 to 64.9 (+19.6 points), and steps per task fall by 53% on average. GPT-6 Astra goes from 102 to 34 steps per task.
- Weaker models gain the most. GLM-5.3-Flash gains 35.2 points and DeepSeek-V4.1-Flash 29.8. The gap between the strongest and the weakest model shrinks from 39.1 to 18.0 points.
- Code execution adds value beyond shell access. With the same GUI and shell access, writing operations as code adds 6.1 points and cuts steps by 14.8% over a GUI+CLI agent (mean of three models).
With our real-device harness held fixed, on four macOS tasks Qwen3.8-Max used 19.4 times the reasoning tokens of GPT-6 Astra and still completed fewer tasks. The case studies trace this gap to model behavior, which is what our next release, the GCUA model, works on. The paper has the full setup, ablations, and case studies.
🎬 Demos
Play the video below, or click a screenshot to watch the other demos.
Do your thing. I'll keep working. Keep watching videos and reading on your Mac while OpenGCUA researches travel essentials and fills your shopping cart. See it report completion, then watch the user open the browser to inspect the cart and request a PowerPoint as a follow-up. This silent edit of a real 26-minute session shows both deliveries, with source timestamps and playback speeds.
https://github.com/user-attachments/assets/b8ac8f0e-5e96-4225-a2df-d45aded005c3
▶ Open background-work video · 1:08
One visa form, four workflows. OpenGCUA, OpenGCUA (prefer GUI), GUI + batch, and Codex work on the same visa application on a real Mac. Playback speeds vary; use the in-video clocks for elapsed task time.
Hotel research. OpenGCUA researches and compares hotel options across desktop apps and the browser.
GUI click sequence. A short excerpt of GUI actions during the visa-form workflow.
▶ Watch the click sequence · 0:25
🚀 Quick Start (macOS source harness)
This is the shortest path from a clean clone to an interactive agent. It requires macOS 14+, uv, Python 3.11–3.14, Rust/Cargo 1.89+, and Xcode Command Line Tools. Python 3.12 is recommended. Prebuilt SDK wheels do not require Rust.
git clone https://github.com/Tongyi-MAI/OpenGCUA.git
cd OpenGCUA
uv venv --python 3.12
source .venv/bin/activate
uv pip install -e ./packages/sdk -e ./packages/harness/runtime
# Install the native driver app. Its daemon is included in the app bundle.
.venv/bin/mu setup macos
.venv/bin/mu doctor macos
.venv/bin/mu setup browser --engine chromium
.venv/bin/mu doctor browser --engine chromium
# Create a private local configuration from the shipped defaults.
cp packages/harness/configs/macos_demo.json packages/harness/configs/local.json
Edit the following fields in packages/harness/configs/local.json:
{
"api_key": "YOUR_API_KEY",
"model_name": "qwen3.8-flash",
"llm_base_url": "https://dashscope.aliyuncs.com/compatible-mode/v1",
"llm_protocol": "openai"
}
Keep the other fields copied from macos_demo.json; the object above shows only the values
normally changed. Chat Completions-compatible gateways use "llm_protocol": "openai";
GPT reasoning endpoints with Responses support can use "openai-thinking" to replay
encrypted reasoning across turns. Native Anthropic Messages endpoints use "anthropic";
set "anthropic_thinking": {"type": "adaptive"} in local.json for Claude models
that support adaptive thinking. The Anthropic path preserves signed thinking blocks
in subsequent requests; reasoning_effort is not an Anthropic thinking switch.
Provider examples and every advanced setting are documented
in Setup, configuration, and troubleshooting.
Grant CuaDriver.app access in System Settings → Privacy & Security → Accessibility and Screen Recording, quit and reopen CuaDriver, then verify the installation:
.venv/bin/mu doctor macos
Start either the terminal agent or the browser demo studio. The terminal command automatically
uses the visual interface when stdin/stdout are an interactive terminal; pipes and TERM=dumb
keep the compatible plain-text interface.
# Persistent visual terminal conversation. Type /exit or press Ctrl-D to stop.
.venv/bin/python packages/harness/runtime/mac_agent.py
# Force the original line-oriented output (also useful for scripts and logs).
.venv/bin/python packages/harness/runtime/mac_agent.py --ui plain
# Or start the demo studio at http://127.0.0.1:8800.
bash packages/harness/runtime/start_demo.sh
If you see PERMISSION_DENIED, DEVICE_OFFLINE, a native-extension import error, or a recording
problem, use the error-indexed troubleshooting guide. For SDK-only
installation and wheel builds, see the SDK guide. The standalone
native macOS app is a separate distribution path with its runtime bundled.
The terminal TUI receives new features more frequently and may change faster. The macOS app is the stability-focused version, with a more conservative update cadence.
Agent Skills
The macOS harness can use local SKILL.md workflows on demand. It advertises each Skill's name
and short description first; the full instructions enter the model context only after you select
one or the agent calls skills.load("name"). The source checkout includes a
PDF-form workflow and a
DingTalk workflow; the macOS app bundles the
same built-ins. The DingTalk Skill can guide an already available MCP tool, authenticated dws
CLI, or GUI session; it does not connect a server or log you in. Skills do not grant computer
permissions or automatically run scripts.
In the visual TUI, /skills lists available workflows, /skill pdf-form selects one for your
next message, /skill new my-workflow creates a personal starter, and /skills reload picks up
edits. You can also put an exact $pdf-form marker in a task message (including plain CLI or demo
hub input). Personal Skills live in ~/Library/Application Support/Generalist/skills/; additional
absolute directories can be listed in skills_dirs in local.json. See the
Skill authoring and troubleshooting guide for the format and limits.
Native macOS app
OpenGCUA for macOS implements the menu bar assistant: enter a task, collapse into a floating control panel, reply to questions, and review saved results. SwiftUI + AppKit, with Python, the SDK and signed CuaDriver bundled inside the app; no browser or separately installed runtime is required. Apple Silicon / macOS 14+, currently a local development build. See the app guide for building, packaging and the remaining public-release validation. Version tags can automatically build the DMG and Python SDK downloads through GitHub Actions.
🔌 Cross-Device SDK
The Cross-Device SDK is the interface the model programs against. It binds every operation to an explicit device, app, window, or tab, keeps accessibility and vision as separate ways to observe and act, and lets one program coordinate several devices in a task.
import asyncio
from qwen_intelligence import Runtime
async def main():
async with Runtime() as runtime:
await runtime.add_macos(name="local-mac")
async with runtime.session() as mu:
mac = await mu.register("local-mac", name="mac")
app = await mac.apps.get("com.apple.TextEdit")
await mu.emit(await app.ui.ax.observe()) # TextEdit's accessibility tree
await mu.emit(await app.windows.list())
asyncio.run(main()) # open TextEdit first
It has adapters for macOS, Android, browsers, and Linux, built on a Rust core with Python bindings.
The source setup above installs it. The package is qwen-intelligence-sdk, imported as qwen_intelligence
(mai_ui is the same API under its internal name). See packages/sdk/ and the examples.
🗺️ The open stack and what comes next
OpenGCUA aims to open the full stack of generalist computer use. This preview releases the first layers, and the rest come next.
- OSWorld integration example with standalone screenshot-action and code-action agents
- macOS real-device harness and Cross-Device SDK, as preliminary open-source versions
- Native OpenGCUA macOS app, a preview menu-bar assistant with a floating task panel, voice, and bundled harness, SDK, and driver
- GCUA model for both computer use and browser use, turning code execution, on-demand observation, and task tracking into learned skills
- Execution efficiency, an essential problem for computer use: optimizing code execution, thinking length, and harness design, and exploring broader system-level improvements
- Real-device environments and benchmark, macOS first, measuring useful work under real conditions
- Reliable work on your devices: tasks across computers, browsers, and phones, with dependable targeting, recovery from failures, and continuity when you step in
We do not commit to dates. The plan follows the evidence and what the community proposes, including directions beyond this list. Details are in ROADMAP.md.
🧩 How it works
Formally, a GCUA is an agent system that completes diverse computer tasks through visual interaction and programmatic access, choosing how to observe, act, and organize its work as each task requires. These are three decisions made at every step. A GUI agent leaves all three to fixed rules in its harness. OpenGCUA lets the model make them.
| Decision | GUI agent | OpenGCUA |
|---|---|---|
| 👁️ What to observe | a screenshot after every action | screenshot(), only when the model needs one |
| ✋ How to act | one click or keystroke at a time | a Python program that combines the accessibility tree and app APIs, shell commands, and GUI actions, with loops, branches, and checks |
| 🗂️ How to organize work | fixed by the harness | the model keeps a todo list and states its <intent> before each program |
Two designs make this work. Code execution lets a model bring the programming skill it already has, and lets intermediate results decide the next operation without another model call. Harness-to-model decision transfer moves choices such as when to look and what to track out of harness rules and into the model. GUI agents, CLI agents, and GUI+CLI hybrids such as our earlier Qwen-UI-Agent are special cases that fix some of these decisions in advance.
Here is one real step by GPT-6 Astra on a CRM task in OSWorld-v2. Lines marked ... are cut, and long lines
are wrapped:
<intent>Apply the 29 evidence-based status changes while preserving all original
customer details.</intent>
<code>
todo.update(id="updates", kind="task", content="Applying 29 reviewed status changes ...",
status="attempted")
r = bash(r"""...
cat > /tmp/apply_lead_statuses.py <<'PY'
...
current=api(p,'/api/crm/workspace')['workspace']
assert current==original,'Workspace changed since original review; stopped before edits.'
for change in changes:
d=api(p,'/api/crm/leads/'+change['leadId'],'PATCH',{'status':change['to']})
...
assert lead['status']==change['to']
...
print('VERIFIED:',len(log),'status changes; all 40 statuses correct; ...')
PY
python /tmp/apply_lead_statuses.py""")
if r.ok:
todo.update(id="status_updates", kind="task",
content="Applied and API-verified all 29 planned status changes. ...",
status="claimed_verified")
</code>
In one step, the model states its intent and records its plan. It then writes and runs a script that stops if the app changed since it last looked, applies 29 edits through the app's own API, and checks each one. Only after the script succeeds does it record the result as verified.
🤝 Contributing
We hope the community will shape not only how GCUAs are built, but also what they should be able to do and how progress is measured. There are three ways to join:
- Improve the harness and SDK. Platform adapters, more reliable observation and control, faster execution, real-device testing, and documentation.
- Build real-device evaluation. Environments, realistic tasks, evaluation criteria, and benchmark tooling, including traces that show where today's agents fail.
- Propose research goals. Workflows agents should handle, capabilities worth building, or questions to study with the released code and trajectories. Open a research question.
Start with CONTRIBUTING.md.
👥 Community
WeChat group![]() |
WeChat assistant![]() |
Discord |
If the WeChat group QR code has expired, scan the WeChat assistant QR code to contact us.
✉️ Contact
| Hanzhang Zhou Project co-lead [email protected] |
Long Li Project co-lead [email protected] |
Yue Wang Project lead [email protected] |
📄 Citation
@misc{opengcua2026,
title = {{OpenGCUA}: Toward a Fully Open Stack for Generalist Computer Use},
author = {Hanzhang Zhou and Long Li and Quyu Kong and Lei Wang and
Panrong Tong and Long Chen and Tianyu Xia and Gongjie Zhang and
Xu Zhang and Chen Liu and Jianan Zhang and Chenglin Cai and
Zhengwen Zeng and Steven Hoi and Yue Wang},
year = {2026},
eprint = {2610.22200},
archivePrefix = {arXiv},
primaryClass = {cs.CV},
url = {https://arxiv.org/abs/2610.22200}
}
🙏 Acknowledgments
OpenGCUA builds on our earlier work, Qwen-UI-Agent. The macOS driver is CuaDriver by trycua. We evaluate on OSWorld-v2, and our definition of a GCUA follows the system-level view of Agents' Last Exam.
⚖️ License
Apache-2.0. See LICENSE.
Built by the MAI-UI Team · Back to top ↑




