A locally-running AI agent that autonomously controls a virtual desktop inside a Docker container using the Qwen3-VL vision-language model. The user issues natural language commands; the agent analyzes live screenshots of the VM, plans multi-step actions, and executes mouse/keyboard inputs to accomplish the task — all running on your own hardware with no cloud APIs required.

Most "computer use" demos rely on cloud-hosted models (GPT-4V, Claude, etc.). This project runs the entire pipeline locally:

1. A Docker container (trycua/cua-xfce) provides a full XFCE Linux desktop accessible via VNC and a REST API.
2. A Planner model (text-only GGUF, e.g. Octopus-Planning Q5_K_M) breaks down complex user objectives into atomic, verifiable steps.
3. A Qwen3-VL 8B vision-language model (GGUF format, accelerated on your NVIDIA GPU via llama-cpp-python) looks at the VM's screen, executes each step, and verifies success.
4. A PyQt6 Mission Control UI lets you issue commands, watch the agent work in real-time, inspect each step, and intervene when needed.

Hierarchical Agent Loop:

User Objective
  → Planner (text-only GGUF) → [Step 1, Step 2, Step 3, ...]
    → For each step:
        Screenshot → Executor (Qwen3-VL) → Action (click/type/scroll)
        Screenshot → Verifier (Qwen3-VL) → Pass/Fail
    → All steps done → Objective complete

• Qwen3-VL 8B vision-language model (GGUF, runs locally on GPU)
• Docker Sandbox — isolated virtual desktop via trycua/cua-xfce container
• Hierarchical Planning — local GGUF planner decomposes complex objectives into atomic steps, verified after execution
• Local Model File Browser — browse and select local .gguf model files directly from the GUI, or download from HuggingFace
• Auto GPU Layer Detection — automatically detects available VRAM and calculates optimal GPU offloading
• Mission Control UI — professional 5-panel PyQt6 interface with planner settings panel
• Live VM Screen — direct mouse/keyboard interaction with the VM
• Agent Trace — step-by-step plan visualization, metrics, structured logs
• Plan Verification — each plan step is verified against success criteria using the vision model
• Safety Guards — repeat detection, coordinate validation, step limit
• Turkish → English Translation — commands are auto-translated (optional)
• JSON Log Export — export structured logs for debugging/analysis

Check if the driver is installed:

nvidia-smi

If not installed:

sudo apt update
sudo apt install -y nvidia-driver-535
sudo reboot

# Install Docker Engine
sudo apt update
sudo apt install -y ca-certificates curl gnupg
sudo install -m 0755 -d /etc/apt/keyrings
curl -fsSL https://download.docker.com/linux/ubuntu/gpg \
  | sudo gpg --dearmor -o /etc/apt/keyrings/docker.gpg
echo "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.gpg] \
  https://download.docker.com/linux/ubuntu $(. /etc/os-release && echo $VERSION_CODENAME) stable" \
  | sudo tee /etc/apt/sources.list.d/docker.list > /dev/null
sudo apt update
sudo apt install -y docker-ce docker-ce-cli containerd.io

# Allow running Docker without sudo (re-login required)
sudo usermod -aG docker $USER
newgrp docker

# Verify
docker run --rm hello-world

docker pull trycua/cua-xfce:latest

If Miniconda is not installed:

wget https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh
bash Miniconda3-latest-Linux-x86_64.sh
# Close and reopen your terminal

Create and activate the environment:

conda create -n cua python=3.10 -y
conda activate cua

pip install torch torchvision --index-url https://download.pytorch.org/whl/cu130

Note: For different CUDA versions, visit https://pytorch.org/get-started/locally/

The standard pip install llama-cpp-python does not include CUDA support. To run on an NVIDIA GPU, use a prebuilt wheel from the JamePeng fork:

1. Go to https://github.com/JamePeng/llama-cpp-python/releases

2. Download the .whl file matching your system:

   • Python version: cp310 (Python 3.10)
   • Platform: linux_x86_64
   • CUDA version: cu130 (CUDA 13.0) or cu124 (CUDA 12.4)
   • Example: llama_cpp_python-0.3.23+cu130-cp310-cp310-linux_x86_64.whl

3. Install the downloaded wheel:

conda activate cua
pip install llama_cpp_python-0.3.23+cu130-cp310-cp310-linux_x86_64.whl

Check your CUDA version: look at the "CUDA Version" line in nvidia-smi output.

conda activate cua
pip install -r requirements.txt

The hierarchical planner uses a separate text-only GGUF model to decompose complex objectives into steps. You can either:

Option A — Use a local .gguf file (recommended): Download any text-only GGUF model (e.g. Octopus-Planning Q5_K_M) and select it in the GUI via the 📂 Browse button.

Option B — Auto-download from HuggingFace: Set PLANNER_GGUF_REPO_ID and PLANNER_GGUF_MODEL_FILENAME in src/config.py.

Note: The planner runs on CPU by default when the executor model (Qwen3-VL) is already using the GPU, to avoid CUDA memory conflicts. Auto GPU detection handles this automatically.

To auto-translate Turkish commands to English:

pip install sentencepiece
# The model (Helsinki-NLP/opus-mt-tc-big-tr-en) downloads automatically on first run.

conda activate cua
python gui_mission_control_local.py

The local-first variant runs the entire pipeline on your hardware with a hierarchical planning system:

• Planner Settings Panel — select local GGUF models via file browser, configure GPU layers (auto/manual), HuggingFace fallback
• Hierarchical Plans — complex objectives are decomposed into verified atomic steps
• Auto GPU Detection — optimal VRAM utilization calculated automatically
• Top Bar — Docker/Model status, step counter, latency
• Left — Command input, preset commands, agent step trace with plan visualization
• Center — Live VM screen (mouse/keyboard active)
• Right — Last action detail, metrics, sandbox info, config
• Bottom — Structured logs with JSON export

conda activate cua
python gui_mission_control.py

Opens a professional 5-panel interface without the hierarchical planner.

conda activate cua
python gui_main.py

conda activate cua
python main.py

CuaOS/
│
├── gui_mission_control_local.py # Local-first Mission Control with hierarchical planner (recommended)
├── gui_mission_control.py       # Standard Mission Control UI
├── gui_mission_control_advance.py # Advanced Mission Control UI
├── gui_main.py                  # Classic UI
├── main.py                      # Terminal-only agent loop
├── setup.py                     # Package setup
├── requirements.txt             # Python dependencies
├── README.md
├── SECURITY.md
│
├── src/                         # Source modules
│   ├── __init__.py
│   ├── config.py                # All configuration parameters
│   ├── sandbox.py               # Docker container REST API wrapper
│   ├── llm_client.py            # Qwen3-VL model loading & inference (with VRAM diagnostics)
│   ├── planner.py               # Plan data models, ABC, system prompt
│   ├── planner_local.py         # Local GGUF planner with auto GPU & text fallback parser
│   ├── verifier.py              # Plan step verification using vision model
│   ├── agent_loop.py            # Hierarchical agent loop (plan → execute → verify)
│   ├── agent_runner_v2.py       # V2 agent runner
│   ├── vision.py                # Screenshot capture, resize, preview
│   ├── actions.py               # Action execution (click, type, scroll)
│   ├── guards.py                # Safety checks (repeat guard, validation)
│   ├── translation.py           # Translation helper
│   ├── design_system.py         # UI design tokens & stylesheet
│   └── panels.py                # UI panel widgets
│
├── tests/                       # Unit tests
│   ├── test_planner.py
│   ├── test_verifier.py
│   └── test_agent_loop.py
│
├── assets/                      # Demo videos & media
│
└── img/                         # Runtime screenshots (auto-generated)
    └── (click previews, screen captures)

All parameters are in src/config.py:

Status Legend: ✅ Done · 🔄 In Progress · ⬜ Not Started

MIT