Screen Command

🧰 Use it from your coding agent: packaged as the screen-ocr skill in skills-arsenal — ask "read this screenshot" in ZCode / Claude Code.

English | 简体中文

Select any region on your screen and instantly act on it — OCR, translate, extract code, explain errors — with the result ready to copy.

It collapses the usual "take a screenshot → save it → open a tool → upload → paste → type a command" loop into a single interaction: see it, select it, use it.

Features

• Region capture: drag a rectangle on a full-screen dimmed overlay, with live selection size and Esc to cancel. Multi-monitor and DPI scaling aware.
• Local OCR: powered by the built-in Windows OCR engine (Windows.Media.Ocr). English and Chinese, fully offline.
• Action system: OCR, copy text, translate, extract code, convert to Markdown, explain errors, save image — chosen from a floating menu with number-key shortcuts.
• Error explanation: offline rule-based analysis of captured error output (Python tracebacks, CUDA OOM, git, Node.js, MSVC/MinGW, shell errors). Optional LLM enhancement.
• Formula → LaTeX: map Unicode math symbols, superscripts/subscripts and Greek letters from the OCR text to LaTeX, with special-character escaping.
• Table → Markdown / CSV / JSON: rebuild a table from the selection using word bounding boxes (so column alignment survives OCR) and export it in three formats.
• Quick action: Alt+X captures, OCRs, and copies to the clipboard without opening the menu.
• Test harness: run any action on an image file from the CLI — automated tests don't need a real screen.

Setup

Requires Python 3.11–3.13 on Windows 10 1809+. All dependencies live in a project-local virtual environment; nothing is installed system-wide.

cd screen-command
python -m venv .venv
.venv\Scripts\python -m pip install -e .

# Run (either form works after the editable install)
.venv\Scripts\python -m screen_command run
.venv\Scripts\screen_command run

After the app starts, press Alt+W to start a capture, or click the Capture button in the status window. Hotkeys are registered at process lifetime only — the app leaves no persistent changes behind. If a hotkey is taken by another program, registration fails with a visible hint and the Capture button still works.

Usage

1. Launch: run .venv\Scripts\python -m screen_command run. A small always-on-top status window appears.

2. Capture: press Alt+W (or click the Capture button). The screen dims and freezes; hold the left mouse button and drag a rectangle, release to confirm. The selected area returns to full brightness and shows its pixel size while dragging. Press Esc, right-click, or click without dragging to cancel at any time.

3. Pick an action: the action menu pops up next to your selection — click an entry or press its number key:

   Key	Action	What it does
   1	OCR	Turn the selected pixels into plain text
   2	Copy text	OCR, then put the text on the clipboard
   3	Translate	OCR then translate (needs a configured provider; shows a hint otherwise)
   4	Extract code	Rebuild indentation, strip line numbers, output a code block
   5	To Markdown	Structure the text as lists / links / paragraphs
   6	Explain error	Analyze captured error output (offline rules, no network)
   7	Save image	Save the captured region as PNG (the only action that writes a file)
   8	Formula → LaTeX	Convert math symbols / super-subscripts / Greek letters to LaTeX
   9	Table → Markdown	Rebuild a table, with Markdown / CSV / JSON output (switch in the result window)

4. Read the result: actions run in the background; the result window then shows the text — selectable, with Copy, Edit (make it editable to tweak before copying), Retry, Another action (re-run on the same selection without recapturing) and Close. A "cancel current action" button appears on the status window while an action is running. When an action produces multiple formats (e.g. Table → Markdown / CSV / JSON), a format selector appears next to the metadata and Copy copies the format currently shown.

5. Quick path: Alt+X skips the menu — capture, OCR, and the result lands on your clipboard automatically. Ideal for the frequent "copy that on-screen text" case.

CLI

screen_command run            # launch the GUI
screen_command providers      # list available OCR providers and language packs
screen_command process <img> --action ocr|copy_text|translate|code|markdown|explain|formula|table|save_image
    [--provider auto|windows|mock] [--language zh-Hans-CN]
    [--output result.txt] [--md] [--format Markdown|CSV|JSON]
    [--opt KEY VALUE ...]

--format selects one of a multi-format result (Table: Markdown / CSV / JSON; Formula: LaTeX / LaTeX（显示式）); --md prints the Markdown view when the action has one.

Hotkeys

Hotkey	Action
Alt+W	Select a screen region, then pick an action
Alt+X	Select → OCR → copy to clipboard

Hotkeys can be changed in config.toml at the project root (see config.example.toml). The app looks for that file in SCREEN_COMMAND_CONFIG, then the current working directory, then the repo root. If a combination is already taken by an input method, media player, or another resident app, simply pick a different one there.

Tests

.venv\Scripts\python -m pytest tests -v       # unit + integration tests
.venv\Scripts\python scripts/e2e_smoke.py     # full-pipeline smoke test on a real screen

Test images are generated by tests/make_fixtures.py (English / Chinese / mixed / terminal error / code / small text).

Accuracy notes

Translate, extract code, convert to Markdown, explain errors, formula-to-LaTeX and table extraction are all bounded by raw OCR quality; the rule-based post-processing (indentation reconstruction, full-width normalization, line-number stripping, symbol mapping, column reconstruction) does not correct characters. Small monospace terminal fonts have typical confusions (l↔1, O↔0). For best results select a larger region and avoid tiny or low-contrast text. A selection spanning multiple monitors is cropped from the monitor under the selection center.

Table extraction uses each OCR word's bounding box, so it recovers columns even though Windows OCR collapses the whitespace between them; the English (en-US) engine generally reads ASCII tables more accurately than the Chinese engine, so pass --language en-US from the CLI for tables. Formula-to-LaTeX is a best-effort text conversion, not a math-typesetting engine.

Privacy

Local-first by default: capture and OCR never leave your machine and the app contains no networking code. Translate and LLM actions are built on provider abstractions with no bundled implementation; unconfigured actions show an explicit "unavailable" message and everything else keeps working.

Project layout

src/screen_command/
  capture.py        Screen snapshot and region cropping (in memory only)
  overlay.py        Selection overlay
  hotkey.py         Global hotkeys (process-lifetime registration)
  app.py            State machine and orchestration
  providers.py      Translation / LLM provider abstractions
  ocr/              OCRProvider abstraction + Windows OCR + Mock
  actions/          Action abstraction, registry, and implementations
  ui/               Action menu / result window / status window
tests/              pytest tests and fixture generation
scripts/            E2E smoke test
docs/               ROADMAP / ARCHITECTURE / DECISIONS / REQUIREMENTS

Design details: ARCHITECTURE.md · DECISIONS.md