deskwork

Supercharge Claude's computer use with retrieval.

A real run, sped up. 3 screens into a form it has never seen, the agent hits 4 regulatory questions, searches a corpus of federal PDFs, and types back what it found with the filename and page number.

See the technical post for more details.

Quickstart

git clone https://github.com/ashnkumar/deskwork && cd deskwork
cp .env.example .env          # then put your key in ANTHROPIC_API_KEY

docker compose up -d --build

docker compose exec agent deskwork ingest
docker compose exec agent deskwork run
docker compose exec agent deskwork verify   # did it file a correct report?

An Anthropic API key is the only credential. Chunks are embedded on CPU with BAAI/bge-small-en-v1.5 into Postgres with pgvector — a local index, so there's no second key and nothing to sign up for, and the model ships inside the image.

While run is going, open http://localhost:6080/vnc.html to watch, or http://localhost:8000 to fill the form in yourself.

The pattern

With computer use, the agent finds out what it has to know by using the software, several steps in, on a screen you wouldn't know existed when you fired it off.

Retrieval that runs before the loop can't target that. Embedding the question and pasting the top passages into the prompt assumes you know the question when the run starts; in our demo case it's behind a Continue button. So retrieval goes in the tools array next to computer, and the model reaches for it when the screen gives it a reason to.

The stakes come from what computer use does. A chat model that invents a regulation number says a wrong thing to someone who can push back. A computer-use agent types it into a system of record where it stays until an audit.

What comes out

A task prompt goes in. A row in submissions comes out: 3 regulatory facts the agent looked up rather than recalled, and the document and page they came from.

	Retrieval on its own	Computer use on its own	Both, as tools
The answer	Correct, and sitting in a chat window	Typed into the right field, and sometimes invented	Retrieved, then typed into the right field
Into the system of record	You do it	It does it	It does it
Provenance	You can cite it if you scroll up	None — the model can't tell you where a remembered fact came from	Filename and page, in the form, because that's what the tool returned
Knowing it worked	You read it	The agent says it worked	deskwork verify reads the row back and grades it

The trace

computer drives the mouse and keyboard; search_regulations returns passages with their filename and page number. Both go out in the same request.

A run is 22 or 23 steps and about 2 minutes. The questions are on page 2 of the form. The agent reaches them at step 9, and at step 10 it says:

Four regulatory questions. Let me look each up in the corpus rather than relying on memory.

then passes that question to search_regulations almost verbatim.

2 things there are the model's: the query text, which it can only get by reading the screen, and the number of searches — 4 questions, 2 calls, unbatched. The policy is ours: prompts.py tells it to search — "You may well believe you already know the answer. Search anyway" — and the model is what turns that standing rule into a query we couldn't have written in advance.

Note: the demo questions were written from the corpus, so every one is answerable.

Architecture

#	Component	Module	What it does
1	Command line	__main__.py	ingest, run, verify
2	The loop	agent.py	Send, run the tool, append the result, repeat. Swaps old screenshots for a text note so a long run doesn't exhaust the context
3	Computer tool	tools/computer.py	computer_20251124 against Xvfb via xdotool and scrot
4	Retrieval tool	tools/search.py	One embedding call, one SQL query, passages with provenance
5	The portal	portal/app.py	3-step form with server-side validation. The target
6	Store	db.py, ingest.py	Postgres + pgvector. HNSW over 384-dim vectors
7	The grader	grading.py, verify in __main__.py	Reads the submissions row and checks each answer, and the page it cites, against the ingested corpus

The request is client.beta.messages.create with betas=["computer-use-2025-11-24"], model claude-opus-5, adaptive thinking, and both tools in one array.

The API can do the screenshot pruning for you: context editing (clear_tool_uses_20250919, behind the context-management-2025-06-27 beta) clears old tool results server-side, as one request parameter. This repo keeps ~30 lines of its own so the pruning is visible in the code, but if you're building rather than reading, use it.

Applying this elsewhere

Four pieces transfer.

Two tools in one request. computer is Anthropic's, declared by type. search_regulations is an ordinary custom tool — a JSON schema and a Python function. Same array, and the model arbitrates; agent.py has no routing logic in it.

A system prompt that separates knowing from looking up. Two rules carry prompts.py: Never state a regulatory fact from memory, and never assume what is on screen. The first stops the model typing a plausible rule identifier out of training data; the second stops it typing into a page that moved 2 steps ago.

Neither rule is enforced. Nothing in the loop inspects a typed string to check it came from a retrieved passage. So the third piece is a check that doesn't ask the agent: deskwork verify reads the filed row and grades it against the corpus. The citation has to name a document that's actually in there, at a page that exists, and that page has to contain the value the answer gives. Fabrication is not prevented here, only caught — and only the kind that cites a page it couldn't have come from. Nothing here understands what a sentence means.

Display geometry that matches what you declare. The X display is sized to exactly the resolution reported to the API, so coordinates map 1:1 and a screenshot whose dimensions disagree is a hard error rather than a drifting misclick. Rescaling is a permanent source of off-by-a-scale-factor bugs.

To point this somewhere else: replace the PDFs in corpus/, re-run ingest, aim the agent at your own software, and rewrite the task prompt. Only the grader has no generic version — it's written against this form's 4 questions. SPEC.md has the data model, the chunking measurements, and what was rejected on the way.

Commands

Command	What it does
deskwork ingest	Chunk and embed the corpus. Re-running replaces a document cleanly
deskwork run	One task, start to finish. --report-id, --quarter, --department
deskwork run --transcript run.json	The same, plus every message and tool call written out
deskwork verify --report-id QI-2025-014	Grade one filed report. Exit 0 or 1

Every tunable is an environment variable, listed with defaults in .env.example — the service addresses are fixed by docker-compose.yml, since they name containers on its own network. Worth changing: DESKWORK_MODEL (claude-opus-5), DESKWORK_EFFORT (high), DESKWORK_MAX_STEPS (40) and DESKWORK_MAX_IMAGES (6 — screenshots dominate the token bill).

Tests

uv run pytest           # 167 tests, no API key, no network
uv run pytest -m live   # 3 more that call the real API and spend money

• The computer tool is asserted against a recording runner — the exact xdotool invocations — then driven against a real Xvfb, typing into an xterm and reading it back.
• The loop runs against a fake client replaying recorded turns, where the subtle bugs live: a dropped tool_result, an unfaithful assistant echo, a pruned thinking block.
• The grader has its own tests, and they're all reports that should fail: a negated answer, an invented filename, a page number that doesn't exist, a real page that doesn't support the answer. A grading bug is the one kind that shows up as PASS.
• The live tier is the only thing that can catch the vendor changing the contract. The fake client will happily replay a request shape the API has stopped accepting, so 3 tests send the real one — including a check that the model config this repo excludes is still excluded for the reason given.
• Retrieval quality is a test. test_chunk_size_is_tuned pins the chunk size, because 1100 characters answered 3 of 5 questions and 500 answers 5.

Tests needing a display skip without one, so that count is what gets collected, not what passes.

Limitations

• 11 out of 11 is a small sample. Every graded run filed a correct report — the ledger is in docs/runs.md, one row per run, regradable from the database with the shipped grader. But 11 trials can't distinguish a reliable agent from a lucky one, and they aren't independent in the way the arithmetic assumes: same corpus, same prompt, same afternoon. The true rate could be as low as 76% and this wouldn't know.
• The portal ships with this repo, and so do its questions — see The trace. A real form would ask things the corpus doesn't cover, and the failure mode there is a confident wrong answer.
• The corpus is 3 documents — enough to make retrieval meaningful, not enough to say anything about retrieval at scale. It's also trusted input, so pointing this at a corpus you don't control is an undefended prompt-injection boundary.

Also: the portal has no authentication, ownership, or CSRF protection, and x11vnc runs with -nopw — which is why compose binds both it and the noVNC desktop to 127.0.0.1. Computer use is a beta API and it misclicks, which is part of why the step budget and the grader exist. Not medical or legal advice.

Corpus

3 US federal publications on HIPAA Administrative Simplification — public domain, checked for patient-identifiable content. Provenance and retrieval dates are in corpus/SOURCES.md.

License

MIT — see LICENSE.