Computer Use

From CLI to desktop automation — Claude Code builds a complete Computer Use subsystem through 15 dedicated files, bringing screen capture, mouse clicks, and keyboard input into the agent capability surface.

Core Question

How does a CLI tool running in a terminal safely gain the ability to “see the screen and operate the desktop”? Computer Use solves not just “can it click” but “who can click, where, and what happens when something goes wrong.”

Subsystem Overview

Architecture

    flowchart TB
    subgraph Gates ["Gate Layer"]
        GrowthBook["GrowthBook Remote Config<br/>tengu_malort_pedway"]
        Sub["Subscription Check<br/>Max / Pro / Ant"]
    end

    subgraph Adapter ["Host Adaptation Layer"]
        Host["hostAdapter.ts<br/>CLI vs Desktop"]
        Term["getTerminalBundleId()<br/>Terminal Exclusion"]
    end

    subgraph Executor ["Executor Layer"]
        Rust["@ant/computer-use-input<br/>Rust Native Module"]
        Swift["Swift Native Module"]
        Screen["Screenshots"]
        Mouse["Mouse Operations"]
        KB["Keyboard Input"]
    end

    subgraph Cleanup ["Safety Cleanup Layer"]
        Lock["computerUseLock.ts<br/>Session Lock"]
        Clean["cleanup.ts<br/>State Restoration"]
    end

    subgraph Expose ["Exposure Layer"]
        MCP["mcpServer.ts<br/>MCP Server"]
    end

    GrowthBook --> Sub
    Sub --> Host
    Host --> Term
    Term --> Rust
    Term --> Swift
    Rust --> Screen
    Rust --> Mouse
    Rust --> KB
    Screen --> MCP
    Mouse --> Lock
    KB --> Lock
    Lock --> Clean
  

Key Design Decisions

Subscription Gate

hasRequiredSubscription() restricts Computer Use to Max/Pro users. Ant (internal employees) bypass the check. This is typical progressive capability exposure — high-risk capabilities open first to paying users and internal testers.

Terminal Agent Mode

When CLI runs inside a terminal emulator, getTerminalBundleId() detects the hosting terminal app. Screenshots and clicks must exclude the terminal itself — otherwise the agent would capture its own command line or even click its own window.

MCP Exposure

Computer Use is not only for Claude Code’s internal use but also exposed as an MCP Server via mcpServer.ts. External MCP clients can consume Computer Use capabilities, meaning desktop operations can be orchestrated into larger agent workflows.

Session Lock and State Restoration

computerUseLock.ts implements session-level locking. When a Computer Use session is interrupted (user closes terminal, process crashes), cleanup.ts ensures mouse and keyboard state are restored to their original state, preventing desktop control residuals.

Relationship to the Governance System

Computer Use is not a “superpower” that bypasses the permission system. It remains subject to the full four-layer governance:

1. GrowthBook gate: Remote config can shut it down at any time
2. Subscription check: Restricted by user tier
3. Permission mode: Tool execution still passes through the standard permission chain
4. Sandbox constraints: Computer Use capabilities are limited in sandboxed environments

Lessons for Agent Builders

Path Evidence

Further Reading

1. Architecture Map — Where Computer Use sits in the six-layer structure
2. Tool Plane — How Computer Use integrates with tool governance
3. Execution Governance — Gate and permission relationships
4. Signals & Extensions — Computer Use maturity assessment