Claude Computer Use: Prompting for GUI Automation

Computer use is Claude's most ambitious capability — the model can view screenshots, move the mouse, click buttons, type text, and navigate interfaces just like a human operator. This opens automation possibilities that traditional scripting can't handle: legacy apps without APIs, complex multi-step workflows, and interfaces that change between runs.

But computer use prompting is a distinct discipline from standard prompt engineering. You're not describing desired output — you're describing UI targets in visual terms, structuring action sequences with verification checkpoints, and designing error recovery strategies for when things go wrong. The prompts in this section teach Claude to operate interfaces reliably while keeping safety as a first-class concern.

What You'll Find Here

Two complementary resources — one for writing effective computer use prompts, one for designing safety into autonomous workflows:

Computer Use Prompting

How to describe UI targets, structure action sequences, and specify error recovery strategies. The prompt patterns that produce reliable autonomous operation vs. those that lead to confusion and infinite loops. Includes UI target description techniques, checkpoint and branching action patterns, stale screenshot handling, and a full prompt template.

Human-in-the-Loop Patterns

Designing workflows where Claude asks for confirmation at critical junctures. When to delegate fully autonomous action vs. when to require human approval. The risk-based autonomy model (read-only → low-risk → medium-risk → high-risk → destructive), confirmation patterns, timeout behavior, and session management.

Getting Started

Begin with Computer Use Prompting to learn the fundamental prompt structure and UI description techniques. Then implement the safety patterns from Human-in-the-Loop Patterns before deploying any computer use workflow to production. Never skip the safety layer — the cost of an unsupervised bad click can far exceed the cost of implementing proper guardrails.