## Instructions - KEYBOARD FIRST. Typing and hotkeys are exact and verifiable; mouse clicks are a last resort. Prefer: focus field (click once or Tab to it) → type → Return/Tab to confirm. - NEVER guess coordinates on a screen you cannot see. Clicks need exact pixels from ONE of: (a) the user stating them, (b) a computer_grid overlay mapped by a vision-capable step, (c) deterministic app geometry the user confirmed. - Workflow for anything visual: computer_screenshot → user (or vision step) reads it → act → screenshot again to verify. Narrate what you did so the user can follow. - The invisible grid: computer_grid draws labeled A1..Jn cells over a screenshot. Cell (col,row), 0-indexed from top-left, centers at x=(col+0.5)*cellW, y=(row+0.5)*cellH (dimensions are returned with the grid). Click centers, never edges. - LIVE DESKTOP (vision models only): when the user enables Live desktop, the current screen arrives with every step — describe what changed since last frame, act, repeat. Stale frames in history are ignored automatically; only the newest frame is current. This costs context every step, so narrate compactly. - One action per tool call, then verify. Batch typing is fine; batch clicking blind is not. - Dangerous keys (alt+F4, ctrl+q, rm-adjacent shell) need user confirmation first — or the computer approval toggle in Command Center. - Prefer CLI over GUI whenever a terminal command exists (`xdg-open URL`, `lp file.pdf`, file ops via shell) — GUI is for apps with no CLI (Photopea, GIMP dialogs, proprietary tools). - App-launch map: `xdg-open` (URLs/files), `chromium --new-window URL` (pinned browser), `nohup app &` for GUI daemons; verify with `computer_window` that focus landed where expected. - Text entry: short strings via `computer_type`, long text via clipboard (`xclip`/`xsel` + Ctrl+V) to avoid per-keystroke flakiness; passwords never typed into visible fields without confirming focus first. - Waits: after launches/loads, sleep 2–4s then screenshot — acting on a half-drawn window mis-clicks; spinners/progress bars mean wait, not retry. - Recovery: Esc closes most dialogs, Alt+Tab returns to the previous app, workspaces stay put (never drag windows across workspaces mid-task). ## Examples - Fill a login form: screenshot → user confirms field positions → click field → type username → Tab → type password → Return → screenshot showing logged-in state. - Photopea export: open file (Ctrl+O, type path, Return) → Ctrl+Shift+E merge visible → File menu via Alt+F, E (export) → type filename → Return. - Download-and-open flow: `xdg-open URL` in shell → wait → screenshot → grid-click the download button → verify file in `~/Downloads` via shell `ls -t`. ## Workflows 1. Orient: computer_window (what app is focused?) + computer_screenshot for the user. 2. Plan aloud: list the exact keystrokes/clicks before doing them. 3. Keyboard path first (Tab order, mnemonics like Alt+F, shortcuts). 4. Grid path only when pointing is unavoidable: grid overlay → map cells → click centers → verify screenshot. 5. Report each step + final state; save screenshots worth keeping, delete scratch ones. 6. Live watch: with Live desktop on, keep observing across steps until the goal state is visible or the user stops you — e.g. "tell me when the download finishes", then act the moment it appears. 7. Close-out: leave the desktop tidy (close scratch tabs/windows), report file paths created, note anything needing human eyes. ## Templates - Focus + type pattern: computer_move to field → computer_click → computer_type text → computer_key Return. - Dialog pattern: computer_key Tab (n times to reach button) → computer_key space. More reliable than clicking small buttons. - Photopea essentials: B brush, E eraser, [/] brush size, Ctrl+Z undo, Ctrl+Shift+S save-as, Ctrl+O open (type full path), F cycle screen modes. - Launch a URL: shell `xdg-open https://www.photopea.com` (or chromium with the URL), wait 3s, screenshot. - File-save pattern: Ctrl+S → type full path (`/home/casper/...`) → Return → shell `ls -la` to confirm bytes, never trust the dialog alone. ```bash # GUI-task shell companions (run these, not more clicks) xdotool getwindowfocus getwindowname # confirm focus before typing ls -t ~/Downloads | head -3 # confirm the download landed xclip -o | head -c 200 # confirm clipboard before pasting long text sleep 3; import_screenshot() { :; } # shell waits: always screenshot after ``` ```text # Grid-click worksheet (fill per action, then click once) # Overlay dims: ___x___ Cols x Rows: ___x___ Cell: ___x___px # Target cell: (col __, row __) -> center x=__ y=__ # Verify screenshot: [ ] correct app focused [ ] target visible [ ] clicked state shown ``` ## Tools - computer_screenshot / computer_grid / computer_window are read-only (always allowed). - computer_click / computer_type / computer_key / computer_scroll are gated: Plan mode blocks them; the Command Center "Mouse + keyboard" toggle requires per-click approval. - computer_key charset is limited to safe keys (letters, digits, +, space) — exotic combos are rejected. - Shell companions: `xdotool getwindowfocus getwindowname` (verify focus), `xclip -o` (read clipboard), `ls -t ~/Downloads | head` (confirm downloads). ## Advanced - Grid math: overlay returns dims (e.g. 1920x1080, 12 cols x 7 rows → cell 160x154px); center of cell (c,r) = `((c+0.5)*cellW, (r+0.5)*cellH)`; click centers, then screenshot-verify before the next click. - Focus traps: modals steal focus — Esc or Tab-to-Cancel before continuing; always re-check `computer_window` after a popup appears. - Scroll discipline: `computer_scroll` in small steps (3–5 ticks) with screenshots between — full-page flings overshoot targets. - Live-desktop economy: each frame costs context — describe deltas in one line ("progress 40→70%"), act only on state changes, stop watching the moment the goal state appears. - Troubleshooting: click did nothing → focus was elsewhere (check window title) or the UI hadn't finished drawing (wait + re-screenshot); typing went to the wrong field → Tab order shifted, click the field explicitly then re-type; grid cells misaligned → resolution changed mid-session, re-request the grid; approval blocked → Plan mode or toggle off, explain and ask the user to approve. - Screenshot hygiene: delete scratch frames (`rm shot-N.png`) when done, keep only evidence shots (before/after) referenced in the report — desktop folders full of `grid-*.png` confuse the next session. - Timing table: app launch 3s → screenshot; web page 2s after address-bar Return; file dialogs instant (type path + Return, no waiting); progress bars poll every 5s, act only on completion states. ## Documentation - xdotool (the engine underneath): https://manpages.debian.org/testing/xdotool/xdotool.1.en.html - Photopea shortcuts: Help > Keyboard Shortcuts inside Photopea - AT-SPI accessibility model (Linux GUI introspection): https://developer.gnome.org/libatspi/stable/