FastAPI backend for MindsHub Cowork. Manages projects, conversations, files, scheduling, memory, and agent orchestration with a SQLite-backed data layer.
This repo is the Python backend. The frontend (Electron shell + React SPA) lives in a separate repo: mindsdb/cowork. They are developed and released independently. At runtime, the frontend spawns cowork-server as a local sidecar and communicates over HTTP (127.0.0.1:26866).
Requires Python 3.12+ and uv.
# Install and run
uv tool install cowork-server
cowork-serverThe server starts on http://127.0.0.1:26866. Confirm with:
curl http://127.0.0.1:26866/api/v1/health/uv.lock is the runtime selection for this repository. CI, Docker and release smoke tests refuse a stale lock. cowork/release-requirements.txt is its generated runtime export, with environment markers and artifact hashes. Every wheel and source distribution carries that exact file; publishing checks both artifacts against the lock.
The release workflow selects Anton from uv.lock. Publishing a newer Anton alone does not change this server release. To move a dependency deliberately:
uv lock --upgrade-package anton-agent
python scripts/release_contract.py
uv sync --locked --group dev
bash scripts/check_wheels_available.sh
make test/unitReview both the lock and export diffs, run the release checks, and publish a new server release. Change a declared compatibility constraint first when the intended version is outside it. Use the same process for any other dependency. Do not run a general upgrade while preparing a source-only release.
Cowork's release installer consumes the contract from the selected wheel or Git commit. Direct pip install and uv tool install cowork-server still resolve the public library compatibility ranges; they do not automatically consume this export. For a standalone server with this repository's reviewed selection, check out a reviewed release tag and run uv sync --locked --no-dev, then uv run --locked --no-dev cowork-server.
The eight platform/Python wheel checks use release constraints. When the lock selects an Anton release candidate, the required wheel gate waits for that exact version to reach PyPI's Simple index before resolving it; the publish runner repeats the wait before building. The weekly --floating check separately probes the public compatibility ranges and never rewrites the release lock. Build backends, the Python interpreter and operating-system packages are separate inputs; this policy fixes the application dependency selection, not every byte of an image.
# Run from source (auto-manages virtualenv + deps)
uv run --locked cowork-serverWhen running alongside the Electron app in dev mode, the app spawns the server automatically — no manual start needed. The Electron app looks for a sibling cowork-server/ directory by convention (override with COWORK_SERVER_DIR).
cowork-server imports anton's artifact folder constants (anton.core.artifacts.internal_files) at startup, with no fallback. An anton without those constants (NON_CONTENT_NAMES and REVISIONS_DIRNAME, which mindsdb/anton#521 added, first in 2.26.10.2.1rc2 and first in a stable release in 2.26.10.4.1) makes the server fail to start with ImportError.
uv run --locked cowork-dev-setupInitializes the database and validates configuration.
make test/unitmake test/unit, make test/unit/coverage and make coverage/html run the unit tests in parallel, one pytest-xdist worker per logical CPU, and CI runs the same target. Each worker is its own process with its own temporary database and COWORK_HOME. Set PYTEST_WORKERS to choose the number of workers, or 0 to run serially, for example make test/unit PYTEST_WORKERS=0. A plain uv run --locked pytest runs serially. A test that passes alone but fails in a parallel run usually depends on state an earlier test left in the process; rerun it serially with the tests that ran before it on the same worker (-v names the worker).
Tests use an isolated in-memory database and temporary directories — no side effects on your local ~/.cowork/ data.
Every unit test runs under a guard for blocking work where it stops other requests (tests/_blocking_guard.py). A test fails when cowork code checks out a pooled database connection on the event loop. It also fails when, in org tenancy, cowork code reads, lists, writes, moves, deletes or closes a file under COWORK_HOME on the event loop, or on a worker thread that run_efs didn't start, such as the file a returned Starlette FileResponse opens on anyio's threads. Move such work into run_db or run_efs (cowork/db/units.py). The sites the code has today are listed in the guard with their reasons, each by the function that blocks and the cowork call nearest the work, past the shared file helpers in cowork/common/paths.py, so new work below a listed function fails too. A listed site that no test reaches fails a full run, so remove its entry when you move the work. A run that can't tell, such as one test file or --setup-only, says in one line that it skipped that check.
The guard sees only what the tests drive. File store work counts only in a test that sets org tenancy in the environment, as the org_deployment fixture does, or on the settings the app already holds. A test that patches one module's own check, such as its _org_mode, switches that module alone, so the guard doesn't see its file work. The guard also sees only the calls it wraps: the os calls in its _PATH_PARAMETERS, open and io.open. A read on a file object opened earlier, fcntl.flock, os.readv, os.writev and os.sendfile go unseen. On a worker thread it watches anyio's threads and the event loop's default executor, not a plain threading.Thread or an executor of your own.
A few tests need a real Redis or PostgreSQL, because hosted Cowork runs on both: two replicas reading one Redis (tests/integration/test_two_replicas.py), the Redis clients through a killed connection, a demoted primary, a connection closed during its handshake and a lost reply (tests/integration/test_redis_failover.py), and the migration and launch-lock contracts on PostgreSQL. They carry the redis_service or postgres_service marker and skip when their service is absent. make test/services runs only them. Point it at a disposable Redis and database. The failover tests empty the database COWORK_TEST_REDIS_URL names and run REPLICAOF, CLIENT KILL and ACL SETUSER on that Redis, putting the server back afterwards, so they run only when the URL is set and skip otherwise, even with a Redis on localhost:6379. The migration test downgrades the database to an empty schema:
COWORK_TEST_REDIS_URL=redis://localhost:6379/1 \
COWORK_TEST_POSTGRES_URI=postgresql://postgres@localhost:5432/cowork_test \
make test/servicesOn every pull request and every push to staging, .github/workflows/tests-services.yml runs them against Redis 8.2.3 and PostgreSQL 16.13 service containers with COWORK_REQUIRE_SERVICES=true, so a missing service fails the job instead of skipping. The desktop wheel release doesn't wait on that job.
A run that requires every prerequisite (COWORK_REQUIRE_INTEGRATION=true or COWORK_REQUIRE_SERVICES=true) fails when a test skips, or is left out with --deselect, unless tests/skip_allowlist.yaml names it with a reason (tests/skip_guard.py). Any run that sets either flag lists what it left out at the end of its report and in the GitHub job summary. The allowlist names two tests: the remote-worker disconnect check, an operator-run proof described under Streaming & scheduling, and the post-deploy cancel check, which skips when the model ends its turn before the turn can be seen running, so there is nothing to cancel.
Post-deploy integration runs use the target cluster's self-hosted runner. Dev and staging obtain the fixed cowork test suite through MindsHub's cluster-only service URL. Production must not mutate that shared @emailsink.dev identity while the fixed password remains committed. The production test path therefore accepts a dedicated COWORK_TEST_API_KEY secret, a reviewed COWORK_TEST_USER_EMAIL variable on the non-staff @mindshub.ai domain, and the immutable dedicated organization id in COWORK_TEST_ORG_ID. The suite resolves and matches that principal and organization through production MindsHub, and refuses an employee-classified or MindsHub-admin identity before testing. A missing or mismatched identity fails the required prod run instead of falling back to provisioning or reporting skipped tests. Standing-identity mode also requires the test target to be exactly https://cowork.mindshub.ai before the first network call, so a changed environment file or workflow cannot send the production key to another origin.
Do not store or use COWORK_TEST_API_KEY yet, or configure its paired email and
organization id for a production run. The live prod GitHub Environment has no
protection rules or deployment-branch policy. Although publish.yml refuses to
enter its production build/deploy job from a non-main ref, a manually selected
branch runs that branch's workflow text and can remove the check. Workflow code
is therefore defense in depth, not the authority that protects an Environment
secret.
Before COWORK_TEST_API_KEY is stored or used, or any production evidence run
starts, the prod Environment must have a nonempty required-reviewer rule with
prevent_self_review: true, can_admins_bypass: false, and a selected-branches
deployment policy whose only entry is the main branch, created as an exact
Branch rule (no tag, wildcard, or second branch rule). Verify the live settings
without reading any secret value:
gh api repos/mindsdb/cowork-server/environments/prod \
--jq '
[.protection_rules[]?
| select(.type == "required_reviewers")
| {
prevent_self_review,
reviewers: [.reviewers[]?
| {type, name: (.reviewer.login // .reviewer.slug)}]
}] as $required_reviewers
| {
can_admins_bypass,
required_reviewers: $required_reviewers,
deployment_branch_policy
}
'
gh api 'repos/mindsdb/cowork-server/environments/prod/deployment-branch-policies?per_page=100' \
--jq '[.branch_policies[] | {name, type}]'The first command must show exactly one required-reviewer rule with at least one
named reviewer and prevent_self_review: true, plus can_admins_bypass: false,
protected_branches: false, and custom_branch_policies: true. The second must
print exactly [{"name":"main","type":"branch"}]. Only after both checks pass
may an operator store COWORK_TEST_API_KEY, COWORK_TEST_USER_EMAIL, and
COWORK_TEST_ORG_ID on that Environment. Evidence must come from a fresh
main-branch run started after the
protection was active, with run_attempt: 1; an eligible reviewer other than the
run's actor and triggering_actor must approve its prod Environment gate. A
rerun of an attempt that began before protection does not count. Keeping the
values at Environment scope prevents non-prod jobs from receiving them, but that
scope is safe only when these Environment controls are active.
The production nightly reads production on mdb-prod at 43 7 * * * once its
schedule is live. It is held today; the Environment prerequisites below say what
turns it on. It uses the same guarded standing identity, but selects only
tests/integration/test_production_read_only.py. That selection is GET-only:
it reads health, conversations, schedules, files, and pins. It never
provisions an identity and does not create conversations, schedules, files,
artifacts, or model turns. The broad integration target excludes the
production_read_only marker, so release and staging callers cannot include
this production-only selection by accident.
Failures and the next recovery use the shared engineering-channel notifier.
The nightly is an alert, not a release gate. Review any new endpoint in
READ_ONLY_ENDPOINTS together with
tests/test_production_read_only_workflow.py, which rejects mutating HTTP calls
and pins the complete request list.
The unattended nightly cannot reference the existing prod Environment.
That Environment must retain its required-reviewer gate for production deploys,
and GitHub pauses every job that references such an Environment until a reviewer
approves it. The monitor instead uses a dedicated prod-read-only Environment
with unique COWORK_PROD_READ_ONLY_API_KEY,
COWORK_PROD_READ_ONLY_USER_EMAIL, and
COWORK_PROD_READ_ONLY_ORG_ID inputs. Unique names prevent a missing
Environment value from falling back to a repository or organization credential.
Before promoting this workflow to main, an operator must create
prod-read-only with no required-reviewer or wait-timer rule and a custom
deployment policy whose only entry is the exact main branch. The repository's
main protection must continue to require a pull-request approval, resolve
review conversations, and apply to administrators. These controls let the
scheduled job start without weakening the separately protected prod
Environment:
gh api --method PUT repos/mindsdb/cowork-server/environments/prod-read-only \
--input - <<'JSON'
{
"wait_timer": 0,
"prevent_self_review": false,
"reviewers": [],
"deployment_branch_policy": {
"protected_branches": false,
"custom_branch_policies": true
}
}
JSON
gh api --method POST \
repos/mindsdb/cowork-server/environments/prod-read-only/deployment-branch-policies \
-f name=main -f type=branchOnly after those controls exist may the operator copy the already reviewed standing identity into the dedicated names. The commands prompt for the secret and variable values and do not print them:
gh secret set COWORK_PROD_READ_ONLY_API_KEY \
--repo mindsdb/cowork-server --env prod-read-only
gh variable set COWORK_PROD_READ_ONLY_USER_EMAIL \
--repo mindsdb/cowork-server --env prod-read-only
gh variable set COWORK_PROD_READ_ONLY_ORG_ID \
--repo mindsdb/cowork-server --env prod-read-onlyVerify policy and names without reading credential values:
gh api repos/mindsdb/cowork-server/environments/prod-read-only \
--jq '{protection_rules, deployment_branch_policy}'
gh api \
'repos/mindsdb/cowork-server/environments/prod-read-only/deployment-branch-policies?per_page=100' \
--jq '[.branch_policies[] | {name, type}]'
gh api repos/mindsdb/cowork-server/branches/main/protection \
--jq '{required_pull_request_reviews, enforce_admins, required_conversation_resolution}'
gh secret list --repo mindsdb/cowork-server --env prod-read-only \
| rg '^COWORK_PROD_READ_ONLY_API_KEY\b'
gh variable list --repo mindsdb/cowork-server --env prod-read-only \
| rg '^COWORK_PROD_READ_ONLY_(USER_EMAIL|ORG_ID)\b'The Environment response must have no required_reviewers or wait_timer
entry and must enable only custom branch policies. The branch-policy response
must be exactly [{"name":"main","type":"branch"}]. The branch-protection
response must show at least one required approval, administrator enforcement,
and required conversation resolution. Do not dispatch until every check passes
and the cowork-server#472 prerequisite has landed.
The nightly schedule is held out of the workflow until those checks pass.
GitHub arms a schedule: trigger as soon as the file reaches the default
branch, so shipping one before the Environment exists does not buy a monitor, it
buys a nightly page that reports a missing credential in the same shape as a
production outage. workflow_dispatch is the only trigger until an operator has
provisioned prod-read-only and watched one dispatched run go green. Restore the
schedule: block with - cron: "43 7 * * *" and delete this paragraph in the
same change; a contract test fails if only one of the two happens.
Set LOG_LEVEL (default WARNING) to control verbosity. Enable file logging with ENABLE_FILE_LOGGING=true (writes to LOG_DIR, defaults to ~/.cowork/logs/).
Database exception records omit SQL, parameter values, driver details and tracebacks, in the owned console and file handlers and in everything Uvicorn logs. setup_logging() attaches the database and provider filters to the uvicorn, uvicorn.error, uvicorn.access and uvicorn.asgi loggers, and to any handler a launcher already installed on them, so the policy holds under every launcher: the cowork-server CLI, python -m uvicorn cowork.server:app, or uvicorn spa_wrapper:app with Uvicorn's default log config. Uvicorn's levels stay as the launcher set them, so lifecycle lines such as Finished server process still print. The shared filter replaces each such record with Database operation failed: error_type=<class> sqlstate=<code or unknown> site=<module>.<function>:<line>, keeping only valid driver SQLSTATE codes. site names the logging call. When that call only relays the error, as Uvicorn's request runner and the server's lifespan guard do, site names the deepest Cowork frame the error passed through outside the session wrappers (cowork.db.scoped, cowork.db.session and cowork.db.units), such as the service line that ran a route's failing statement. Session logs retain error types and client HTTP status, including the 503 for a full pool. A 5xx HTTPException logs its status. Its own detail is not logged, but its cause's class, message and traceback are, unless the cause chain holds a database or provider error, which the filters replace. Any other error, including one that carries a status_code without being an HTTPException, logs only its type in the session line, then reaches Uvicorn, which logs it through the same filters. Exceptions and HTTP errors returned to callers stay unchanged.
A database error while the server starts or stops (a migration, the seed rows, or Postgres not yet reachable after a reboot) logs that safe line through the owned handlers. Uvicorn then prints only RuntimeError: Database operation failed during server startup (or shutdown), because Starlette hands it the failure as plain traceback text, which no filter can read. Other startup errors keep their traceback.
A call site passes the ids that locate a failure as extra=log_context(...) from cowork.common.logger, which sets record attributes: request_id, project_id, schedule_id, conversation_id, and artifact_slug for one artifact or artifact_slugs for several. The filters never change them, and the console and file formatters render them before the message, for example [Req:<id>][Project:<id>][Schedule:<id>][Conversation:<id>][Artifact:'<slug>'] or [Artifacts:'<slug>', '<slug>']. Artifact slugs are folder names, so they render quoted. A site that carries only a request id passes extra={"request_id": ...} directly.
OpenAI and Anthropic SDK errors (any OpenAIError or AnthropicError) and Anton's typed provider errors are found, in the same handlers and Uvicorn loggers, through each exception's cause, then its context (even a suppressed one), then its exception group members. The record becomes Provider operation failed: error_type=<the record's exception class> provider_error=<the first provider error's class> status=<that provider error's HTTP status or unknown>; a wrapper's own status is never used. Cowork's request ID ([Req:<id>]) stays on the line; the provider's own request ID is not logged. Provider messages, bodies, requests and tracebacks are omitted. Routing failure warnings also omit body-derived type, code and parameter fields. Original exceptions, probe verdicts and client HTTP/SSE errors stay unchanged.
Uvicorn's access line keeps each request's method, path and status, and setup_logging() replaces its query with ?[redacted], because a query can carry an OAuth callback's code or a webhook's verify token.
SQLAlchemy query and connection diagnostics stay at WARNING, even when application logging is set to DEBUG. SDK request diagnostics and HTTP client logging stay at ERROR, including Anthropic, httpx2 and the whole httpcore and httpcore2 namespaces, because they can include request bodies, URLs or response headers such as set-cookie.
Releases are automatic on merge; there is no version to bump by hand (the package version comes from the tag).
- Push to
main:publish.ymlruns the unit tests, cuts a CalVer tag and GitHub release (v0.<yy>.<m>.<d>.<seq>), then builds and publishes to PyPI via OIDC trusted publishing. - Push to
staging:publish-staging.ymldoes the same on the rc pre-release stream (v0.<yy>.<m>.<d>.<seq>rc<n>, GitHub and PyPI pre-release). When PyPI's newestanton-agentupload is an rc, it pins that rc into the wheel so the pair installs exactly. When the newest upload is a stable release, the wheel keeps theanton-agentrange frompyproject.toml.
Both take their version, tag, and release from the shared calver-release.yml
reusable in mindsdb/github-actions
(prerelease: true selects the rc stream). The publish jobs stay in these two
workflows: PyPI trusted publishing matches the OIDC claim on the workflow
filename and does not support reusable workflows.
The cowork-server maintainers own the deployed integration signal and its
staging prerequisites. nightly-staging-integration.yml
runs every day at 06:41 UTC and can also be dispatched by hand. It calls the
same tests-integration.yml reusable workflow as a deployment on mdb-dev,
then reports a failure or the first recovery through the shared
engineering-channel notifier. It is a standalone monitor and never gates a
publish, release, or deployment.
The nightly runs staging's copy of the tests, not main's. GitHub starts every scheduled run on the default branch, so the checkout would otherwise take main. main trails staging by every commit waiting for the weekly release. Any behavior change among those commits fails main's tests against staging's pods, even though staging works as intended. So the job passes ref: staging, and tests-integration.yml hands that input to actions/checkout. The deploy callers leave ref empty and keep testing the commit they just deployed. test_nightly_integration_workflow.py fails if a scheduled caller's ref differs from its deploy-env.
codeql.yml runs CodeQL on every pull request and push to staging and main, and weekly on main. Each event's pipeline calls it as a job (dev-build-deploy.yml, publish-staging.yml, publish.yml), so its results sit in that event's one run. It scans Python, JavaScript/TypeScript and the workflows themselves with the default query suite, and needs no build. It replaces CodeQL's default setup, which scans only the default branch and branches under classic branch protection. staging is protected by rulesets, so default setup never scanned a pull request onto it. Default setup must stay off in the repository's code scanning settings, since GitHub refuses a workflow's results while it is on.
The suite may create and delete test conversations, schedules, files, and agent
turns in staging. The fixed test tenant is reserved for the cowork suite, and
the workflow sets COWORK_REQUIRE_INTEGRATION=true for staging so a missing
target, identity source, replica, or port-forward fails instead of becoming a
green skip.
In the packaged Electron app, a background updater checks PyPI on every launch and upgrades automatically (with rollback on failure). See server-updater.ts in the frontend repo.
cowork/
api/v1/endpoints/ # FastAPI route handlers
services/ # Business logic
models/ # SQLModel / DB models
schemas/ # Pydantic request/response schemas
db/ # Database session and migrations
common/ # Shared utilities, settings
harnesses/ # Agent adapters (Anton)
The server is designed to be agent-agnostic — core features (projects, conversations, files) are shared across agents, while agent-specific behavior lives in harness adapters. See docs/DESIGN.md for the full architectural rationale.
The server runs in one of three modes, set by two settings (Configuration):
| Mode | Settings | What it is |
|---|---|---|
| MindsHub mode | COWORK_TENANCY_MODE=org |
The cloud, multi-tenant server. Every request is an organization and a person, from the identity headers the gateway sets, and every resource is kept per organization. Sharing a skill or an artifact with people, teams or everyone at the organization works only here. |
| Single-tenant mode | COWORK_TENANCY_MODE=local and COWORK_TRUST_IDENTITY_HEADERS=true |
One organization with many people, each with their own space (their partition, cowork.principal.personal_partition_id), as the self-hosted VM runs. A person shares one of their own skills with everyone, read-only (Sharing one skill with everyone). |
| Single-user mode | COWORK_TENANCY_MODE=local alone |
The desktop app: one person, no identity headers. |
The code names MindsHub mode TenantScope.org_mode, and the app learns it from /health's org_mode.
- MindsHub: the MindsDB platform service this server asks about organizations, people, teams, permissions and artifact rules. Its public reads go out with the caller's own credential (
cowork/services/mindshub_client.py); its internal artifact-access routes take a shared secret (cowork/services/artifact_access.py). - MindsHub UI: the web app's build flag
VITE_PLATFORM_UI, which shows MindsHub's model, provider and workspace controls. It is set in the app's build, not here, and it is separate from the server's mode: the cloud build turns both on, and neither implies the other. - Audience: who a skill or an artifact is shared with, under MindsHub's artifact rule names:
org_allowed(Everyone at ‹org›),allowed_team_ids(Specific teams),allowed_user_ids(Specific people in the organization) and, for an artifact only,allowed_emails(anyone else, by address). Nothing set is "Only me". - Default team: the organization's team MindsHub marks
is_default, whose members are everyone in it. Naming it is stored asorg_allowed, never as a team. - Always on: a skill shared with everyone or with a team that the agent applies on every task in the turns of the organization or the team, instead of recalling it when a task needs it. It stays on demand for people a skill is shared with personally (Always-on skills).
- Off for me: a person's own switch for one skill (
PUT /api/v1/skills/{skill_id}/me), which leaves it out of their turns only. A skill'senabledturns it off for everyone who reads it; a turn uses a skill only when both are on.
A harness adapts an external agent library (Anton today) to the cowork-server interface. All harnesses implement the HarnessProvider protocol (harnesses/base.py), which exposes streaming responses, skill sync, and memory operations. The active harness is selected via the harness user setting. To add a new agent, implement the protocol and register it with the @register decorator.
Each turn's scratchpads exit when the turn ends. The Anton harness builds a fresh ChatSession for every turn, and each scratchpad that session starts is a child process of the server. When the turn completes, fails or is stopped, close_session_scratchpads (cowork/common/chat_session.py) kills those processes. A Stop or an opted-in client disconnect also interrupts a running cell and ends its turn; the harness does not start another model step. The next turn starts a new process, which reloads the variables the old one saved to the conversation's namespace snapshot after every cell. Only the scratchpads close: full-stack backends the agent launched keep running. Anything a cell starts in the scratchpad's process group, such as a default subprocess.Popen child or a thread, dies with it, so a long-lived service belongs in launch_backend. A child started with start_new_session=True or setsid escapes: it keeps running and holds two of the server's pipe descriptors until it exits. The connector probe closes its scratchpads the same way, and shutdown waits up to 5 seconds for closes still running. A scratchpad runs as python <tmp>/anton_scratchpad_<random>.py, so pgrep -f anton_scratchpad_ counts the live ones; it should drop to 0 once no turn is running.
Agent responses stream to clients via Server-Sent Events (SSE) on POST /responses/. The server tracks in-flight streams and supports cancellation (/responses/cancel) and late-join tailing (/responses/tail).
A streaming API question can set "cancel_on_disconnect": true to stop its own run when the
originating POST connection closes before the answer finishes. The option is a strict boolean,
defaults to false, and requires "stream": true. Browser questions leave it off, so a reload
still reconnects to the detached answer. Closing a /responses/tail observer never cancels a run.
If a client leaves during routing or setup, that request's cleanup prevents it from leaving a
live producer behind. Completion and server shutdown retain their existing outcomes.
A question can set "memory": "off" to read no learned memory and learn nothing, while other questions keep memory, for example a scored evaluation run on a deployment whose testers have it on. The turn runs as every turn does with MEMORY_ENABLED=false: two empty memory tiers in its scratch folder, no background memory passes, and the operator's managed rules still read. A later turn sent without the key in the same conversation learns from the whole conversation, its earlier memory-off turns included, so send the key on every turn of the conversation. The field takes "on" (the default) or "off". A deployment that doesn't trust identity headers, such as the desktop app, still reads stored memory with memory off, so it refuses the question with 501.
Each answer also gets one response.memory_read event, before its response.figure_sources, if any, and its response.completed. read says whether the turn read learned memory, the person's own or the shared store, and entries counts the rules, lessons, profile lines and scratchpad tips anton put in front of the model (ChatSession.memory_read). The operator's managed rules aren't counted. A turn on two empty tiers, as every question with "memory": "off" runs, reads nobody's learned memory, so it gets "read": false and "entries": 0 whether or not the deployment sets COWORK_DEPLOYMENT_KIND. So does a direct answer from the routing gate, which runs no agent. The harness sends the event with the first event anton streams after building the turn's prompt, which anton does before its first model call, or as the turn fails, is stopped or ends, if anton streams nothing after it. So a turn that fails after building its prompt carries it, and so does a turn the user stops after that. The event is saved with the answer's events. An answer that ended before anton built its prompt gets none, and so do a turn on the remote worker and a turn on an anton without ChatSession.memory_read.
The root of response.created carries a run_id UUID for that execution. An API client or proxy
can send {"conversation_id": "...", "expected_run_id": "...", "reason": "client_gone"} to
POST /api/v1/responses/cancel. The guard must match the authorized run; a stale or foreign ID
returns 404 and cannot stop a successor. Turn numbers can repeat, so they are not cancellation
guards. Existing Stop calls without expected_run_id still cancel the conversation's current
authorized run. A 200 response reports whether cancellation was requested; the stream's terminal
event and persisted answer determine the final outcome. Database or worker failures can prevent
successful finalization.
A proxy that keeps reading upstream after its client leaves must relay the disconnect explicitly. The self-hosted recording proxy consumes the option, keeps its upstream connection detached for audit records, and sends an exact-run cancel. Its final record comes from the answer's events. See Disconnect ownership and exact-run cancellation.
tests/test_disconnect_transport.py covers real sockets and a local executing scratchpad.
For the deployed remote-worker check, use the standard post-deploy identity and COWORK_BASE_URL,
then set COWORK_RUN_DISCONNECT_WORKER_TEST=true and COWORK_DISCONNECT_NAMESPACE before running
uv run pytest -q tests/integration/test_disconnect_worker.py.
COWORK_DISCONNECT_CONTEXT optionally selects the Kubernetes context. The runner needs read
access to pods and permission to exec the observer in the worker container. The check starts
its own bounded cell and observes process/container identity and heartbeat cessation; a
cancelled SSE event alone cannot pass it. It skips unless explicitly enabled.
/responses/cancel takes an optional reason. The UI sends "stalled" when it heard nothing for its idle window, and the turn then saves as a response.failed with code stalled, so a reload shows the stall card. Without a reason, or with any other value, the cancel is a Stop and keeps the partial answer with no error row. While anton waits on a model call that sends no output, the server (in cloud, the anton pod) writes a model_wait progress frame every 20 to 25 seconds so the turn's idle bounds do not end it. A model call that sends no output for 600 seconds ends the turn with code model_timeout; anton's ANTON_MODEL_CALL_IDLE_TIMEOUT_S sets that limit, and 0 or less turns it off. Turn liveness and idle bounds lists every bound a quiet turn meets and what happens when each one fires.
POST /responses/ reads what it needs (the turn's settings, its attachments, the conversation, whether an always-on skill reaches the turn and the routing gate's history) as one short database unit in a worker thread (run_db in cowork/db/units.py). A turn that carries attachments, or names no project by id and so may start its conversation in the general project, whose folder is checked, runs that unit on the file store's threads instead (run_efs_unit). Deciding whether an always-on skill reaches the turn can take its skill file, which that unit doesn't read: when one of the organization's skills is always on, the file is read in a unit of its own on the file store's threads (run_efs_unit), and when none reaches the turn, the gate's history is read in one more database unit after it. The request holds no pooled connection while the gate or the stream runs, and a wait for one never stalls the event loop. When no connection frees within POOL_TIMEOUT (5 s by default), the request is refused with 503, a Retry-After header and {"detail": "Cowork is busy. Try again in about 5 seconds.", "code": "server_busy"}. An answer that meets the same wait after its stream started ends with one response.failed frame with code server_busy and the retry_after and retry_at fields rate_limited carries. The question sent again after that frame gets a stream of its own.
That unit also saves the turn's model and reasoning_effort onto the conversation (ConversationService.record_turn_picks), so GET /api/v1/conversations/ and GET /api/v1/conversations/{id} return the latest picks as model and reasoningEffort, and a reopened chat comes back on them. A turn that sends no model or no effort keeps the saved one: Cowork sends no model for a pinned model its picker no longer offers, such as one the wallet locks, so that pin comes back after a top-up. A turn that goes back to the default names the pick in reset_picks ("model" for Model Router or the operator's default model, "reasoning_effort" for the default effort), and that clears the saved one, so a reopened chat comes back on the default. A model or effort the turn sends wins over its own reset. A comparison side keeps the model and effort it was set up with, since its turns run on those whatever the request says.
In a turn this process runs (COWORK_TURN_BACKEND=inprocess), scratchpad call-end events retain their complete input in the stream and saved conversation, including code longer than 65,536 characters. The next turn's write check can therefore read bindings from a long earlier cell. Results and every other tool's input retain their 65,536-character cap. cowork-server adds no cap of its own to a remote pod's scratchpad call input: the pod's own 65,536-character cap (MAX_STEP_CHARS in anton's anton/cloud_turn/__main__.py) is the only one, so a remote turn's saved call-end event can hold a cut call. Remote turns replay no cells and refuse ANTON_WRITE_APPROVAL, so no write check reads it. Replay preserves an explicit cell error; when a truncated result loses that field, only its saved cell_status="ok" establishes success, and the cell otherwise says it failed, timed out or has an unknown outcome. The agent reads that outcome too: its notebook view and compaction summary show such a cell as failed, and anton's end-of-turn memory pass counts it as one. A failed or unconfirmed cell keeps its code for binding analysis but cannot establish ownership of a file while it stays in the pad. A cell that compaction folds into a summary keeps that flag, so an earlier failed write there can't make its file the agent's own either. Plain-text refusals are never replayed as executed cells. Old events whose call and result both lost their code cannot be recovered.
An answer produced in this process (COWORK_TURN_BACKEND=inprocess, the default) runs its own database work as units too. One at the start saves the question (pending) and loads the conversation. The harness reads the history, the attachments and each recall_history lookup in units of its own. One at the end saves the answer and clears the question's pending flag in one commit. Both hold a per-conversation lock that keeps a conversation's message writes in order. So such an answer holds no pooled connection while the model answers or the stream runs. A save that fails at the end of a finished or stopped answer ends the stream with response.failed (server_busy when no connection freed in time), never response.completed. An interrupted or failed answer whose save fails keeps its own response.failed frame, and the next boot's recovery seals an interrupted one into history. A Stop, a second Stop or a shutdown that arrives while a turn saves its question or its answer waits for that save, so the stream's last frame always matches what the database holds. The compacted history summary and new artifact index are database units too. The harness waits for its cleanup through run_to_completion, so a Stop cannot abandon these writes, and a full pool never blocks the event loop. A shutdown can still lose one that has not started when the event loop closes ("Work on the shared file store" below). When the pool stays full, a Stop waits up to POOL_TIMEOUT for each pending write (the answer, a compacted summary and a new artifact's row), so /responses/cancel can take up to three times POOL_TIMEOUT. A Stop during a running cell returns within 5 s only while the pool frees connections. An answer produced by the remote backend (COWORK_TURN_BACKEND=remote) works the same way. Saving the question, saving the compacted summary the pod reports, indexing the artifacts it created and saving the answer are each a unit, and the artifacts' owners are looked up in a unit before they are published. Staging the pod's workspace, reading what the pod is seeded with (its history and the project's memory), building the skills it gets and saving the memory it reports work on the file store, so each runs as one unit on the file store's threads (run_efs_unit, "Work on the shared file store" below). So it holds no pooled connection while the pod answers, and its settings come from the snapshot the request loaded. The turn indexes the artifacts the pod created once, however it ends. A shutdown waits up to 8 s at the end of its teardown for that write, and loses it if it is still running then ("Work on the shared file store" below). A Stop waits for that write and then for the answer's save, so with the pool full /responses/cancel can take up to twice POOL_TIMEOUT. A memory or summary save that is still waiting for a connection when a Stop lands is dropped.
The connector form's stream (POST /api/v1/connectors/submissions/) reads its conversation and the settings that build the probe's model client, and saves the probe's narrative, as units too. So it holds no connection while the credential probe waits on the model, and it never waits for one on the event loop. When no connection frees in time for the conversation read or the settings read, the form shows Cowork is busy. Try again in about 5 seconds. and the probe does not start. The stream ends at that refusal and saves nothing to the conversation. A save of the probe's narrative that fails still ends the stream with response.completed, because the form carries the probe's outcome.
Every other coroutine route reads and writes through database units as well, and takes the caller's tenant scope (get_tenant_scope) instead of the request's session. Search, settings, the MindsHub workspace selector, the channel configuration routes, a model comparison's usage, delete and continue routes, and the read of a skill's stored rule before its sharing save therefore wait for a connection in a worker thread, and a full pool refuses them with the same 503. tests/test_coroutine_routes_hold_no_session.py fails when a coroutine route takes the request's session, and when a coroutine dependency of any route does, since that dependency runs on the event loop even under a sync route. The routes still allowed to are listed there with their reasons, and all four are desktop-only. A channel turn, an inbound Slack, Discord or Telegram message, runs the same way. Its binding, its conversation, its harness, its question, its answer and the artifacts it created are each read or written in a unit, so it holds no pooled connection while the agent answers. The turn's first unit reads its harness and loads its settings. The rest of the turn runs with that snapshot bound, as a browser turn does (use_turn_settings). The pod's seed is read in one unit by the same reader a browser turn uses (ResponsesHandler._read_remote_turn_inputs), less the project memory. That read and each memory save the pod reports take their settings from the snapshot, so no unit waits for a second connection while it holds the first. The workspace staging before the pod starts, and the memory and compaction the pod reports, go through the same helpers a browser turn uses (stage_remote_workspace, save_turn_memory and save_turn_compaction in cowork/handlers/responses.py). The staging, the skills block the pod gets and each memory save work on the file store, so each runs as one unit on the file store's threads (run_efs_unit). A full pool or a busy file store refuses the turn at its staging or its skills block, but a memory or compaction write it refuses is dropped with a WARNING, and the turn goes on to its reply. When the pool or the file store refuses a channel message before the agent answers it, the sender gets one short reply, "I'm overloaded right now and couldn't start on that. Please send it again in a minute.", and the event is still recorded failed. A refusal while the binding is read or made answers only a direct message or a message that mentions the bot, since the channel's trigger rule is not read yet. Once the agent has answered, a full pool never costs the sender the reply: the saves of the question and the answer each try 3 times, 1 and then 2 seconds apart, and a save still refused is dropped with a WARNING naming the conversation, and the reply is delivered anyway. The turn then sends the files of the artifacts it created or touched, as its own artifact cards name them, never another conversation's from the project's shared artifacts folder: at most 3, each a regular file inside the project's .anton/artifacts, read in one run_efs call through descriptors that do not follow a link. A file larger than the adapter's own upload cap (max_attachment_bytes: 10 MiB for Discord, 50 MiB for Slack and Telegram, 100 MiB for WhatsApp, and 100 MiB for an adapter that states none) is skipped on its size before any of it is read. A file store that frees no slot in time sends no files, with a WARNING, and the turn still counts as delivered. Each adapter's send_attachment takes the bytes and the file name, and refuses a file past its own platform's cap. A channel attachment keeps only the last component of its platform file name, as an upload does (upload_basename in cowork/services/files.py); a name that is empty, . or .. is stored as file. In org tenancy a channel turn saves no memory at all: it runs for its organization with no member, and neither memory tier takes an entry without one, so the turn logs one INFO line instead. The webhook records each inbound event in a unit before it schedules the event. A cancel of the intake, such as a credentials change, a lost lease or a shutdown, waits for the event in hand to be both recorded and scheduled, so a recorded event is never left for its redelivery to be dropped as a duplicate. Telegram's poll loop polls a batch again when its intake fails, so a full pool delays those messages and the loop keeps running. A stream (the Discord gateway, Slack Socket Mode) has no offset to resend, and a new connection does not bring a message back, so a batch the pool refuses is taken in again on the same connection, 3 seconds after each refusal, until it is recorded; dedupe drops the events of it already recorded. An event is marked failed only when the runtime itself fails on it. Its status is a record for operators that dedupe does not read, so a status write the pool refuses is dropped with a WARNING. The ingress reconcile, which runs every 30 seconds on every replica, reads every organization's installations in one unit.
A question sent into a conversation whose streamed turn is still answering is refused with 409 and {"detail": "Another question is still being answered in this conversation. Wait for it to finish, then send yours again.", "code": "turn_in_progress"}. It is refused before anything reads or saves it, and the running answer goes on. A turn whose stream has already ended counts as finished, so a question queued behind it is accepted. The check runs before any database read, and RunRegistry.start repeats it under a lock for that conversation before it opens the new turn's buffer. Two questions racing through the gate read the same history, so both compute the same turn number, which names the turn's buffer and which boot recovery reads as the question's position in the history. Once a turn saves its question, the registry refuses a later start at that turn's number or an earlier one, even after that turn has ended. So the second of two racing questions is refused with the same 409 whether it reaches the registry before or after the first turn ends, and two questions never share a buffer. A question that follows a turn that saved nothing (refused for a full pool, say) keeps that turn's number and is accepted. A queued follow-up waits for the preceding turn's internal terminal record once its final client frame is being written; slow buffer creation for one conversation does not hold up other conversations.
With several replicas (COWORK_STREAM_BACKEND=redis), a send that reaches another replica than the one running the conversation's turn gets the same 409. Each turn takes its conversation's turn claim in Redis before it opens its buffer, renews it every 10 s, and releases it once its stream has a terminal record. Only the run that holds the claim writes the conversation's turn index entry, and a channel turn writes it only while no run holds the claim, so Stop and answers keep reaching the run that is answering. A turn also notes there that it saved its question as soon as it saves it, so the second of two racing questions is refused with 409 on every replica, also when it arrives after the first turn ended. A claim that is not renewed lapses 90 s after its last renewal, longer than the stall the liveness probe forgives (up to 65 s) plus the time a claim can have run down when that stall starts. A send that arrives while another replica's turn writes its final frame waits up to 10 s for that turn to release its claim, as a follow-up on the same replica does. When Redis cannot say whether another run holds the conversation, the send is refused with 503, Retry-After: 2 and {"detail": "Cowork is busy. Try again in about 2 seconds.", "code": "server_busy"}. A claim that a refused send may have taken, or that a finished turn could not release, is released again every second until Redis answers, so it is gone about a second after Redis is back. A process that starts in a pod releases the claims an earlier process in that pod left, and writes the controller's cancel flag for each of their turns, which relies on one server process per pod. That covers a container restarted in place, after an OOM kill or a crash, that starts again within 90 s of its last renewal. A liveness restart kills a stalled process only after its claims have lapsed, so it counts as a force-kill here. A pod that is force-killed or lost releases nothing, so a send into a conversation whose turn it was relaying is refused with 409 until that turn's claim lapses, up to 90 s, and is admitted after that. Nothing ends that dead turn's stream or saves it yet: a reader that stays attached ends the stream after its idle cut, and its question stays pending. Where turns run in a sandbox, the dead turn's sandbox turn runs on until it ends or reaches the controller's turn limit, and the next send in that conversation waits behind it, unless a Stop on the dead turn, sent before that next send, cancels it. A conversation delete stops the turn it finds running, on whichever replica it lands. The replica running the turn cancels it and releases its claim. Another replica reads the claim and writes the turn's cancel flag with the cause discarded, so the turn stops without saving into the deleted history or writing to its deleted stream, and releases its claim; the next send is admitted once it has stopped. A delete that cannot write that flag logs turn_discard_cancel_failed, and the next send is then refused with 409 until the turn ends. A conversation delete that cannot reach Redis to forget the conversation's turn index entry logs turn_index_forget_failed, and a send at or below the deleted turn's number is then refused with 409 on every replica until 5 minutes after that turn released its claim. A turn whose process died before its release left no release time, so such a send is refused until the entry expires, an hour after the turn started. That is how long a turn's saved question goes on refusing a send at its number that read the history before the save, far longer than a request spends between reading the history and starting its turn. The claim does not order database writes either: a save that outlives its turn at shutdown can still be writing when a turn on another replica starts. cowork/streaming/turn_claim.py describes the keys. The file backend never reads or writes them.
Where turns run remotely (COWORK_TURN_BACKEND=remote), a replica that has begun ending its turns for a shutdown refuses a new one with 503, Retry-After: 1 and {"detail": "Cowork is busy. Try again in about 1 second.", "code": "server_busy"}, the refusal a full database pool gives. The retry reaches another replica. deployment/cowork-server/README.md describes what a shutdown does to the answers in progress.
A turn that fails ends with one response.failed frame carrying a stable code and the user-facing error (cowork/handlers/turn_errors.py), and the conversation's saved events keep the same payload, so a reload shows the same card. For included_allowance_exhausted and free_serving_paused the frame also carries reset_at, the instant the free way forward comes back. An in-process turn passes the gate's X-MindsHub-Reset-At header through as sent. A hosted turn takes it from the anton worker's turn_failed reply, which scratchpad-controller forwards beside error. The producer keeps it only for those two codes and only when it parses as an ISO-8601 instant with a UTC offset. An older worker sends none, and its frame has no reset_at.
In-process turns pick the billing code from anton's exception type, the same way hosted turns do: WalletEmptyError gives token_limit, AllowanceExhaustedError gives included_allowance_exhausted, FreeServingPausedError gives free_serving_paused, and BudgetExceededError gives budget_exceeded. The locked anton (2.26.11.9.1) predates BudgetExceededError, so until the lock moves to an anton with it, an in-process budget stop is recognized only by the X-MindsHub-Reason header or body code below, and a self-hosted stack with MINDS_URL at https://minds.invalid gets none of them. The other three types hold when a self-hosted stack sets MINDS_URL to https://minds.invalid, because anton builds these types only for a response from a MindsHub host. The X-MindsHub-Reason header counts unless the response provably came from another host, and the body code counts only from the configured minds_url host. In-process, a reason attribute or a class with the same name never selects a code; hosted turns map anton's own type name.
The server never picks a billing code from a 429's status alone: the code comes from a recognized reason or from anton's exception type. anton retries a gateway 429 it cannot classify, and the turn ends as provider_overloaded, in-process and hosted alike. A 429 that reaches the server unclassified ends as anton_error. From the configured gateway, a 402 with no recognized reason gives token_limit and a 503 gives policy_unavailable. A 402 whose reason is budget_exceeded gives its own budget_exceeded code: an admin set a hard spending cap, credits do not lift it, and the frame carries no reset_at.
A turn that stops before it finishes, at a limit, a content filter or another hand-back, still ends with response.completed, so every client sees the answer end and keeps its text. Its response.status is then incomplete, and response.incomplete_details.reason says why: spend_ceiling, round_cap, handback_budget, handback_verifier_failure, handback_stuck, retry_exhausted, content_filter or question_unanswered (anton's ended_by, described on IncompleteDetails in cowork/schemas/responses.py). A turn that finished has status: "completed" and incomplete_details: null. stream: false returns the same status and reason in its JSON body.
An answer whose text holds figures gets one response.figure_sources event just before response.completed. The event lists each figure: its place in the answer (UTF-16 offsets, the unit JavaScript strings index by), its text, its origin (step, question, earlier_answer or none) and the steps it came from. Each step it names is listed once, with its kind, a short label, and the project file or attached file, web page, SQL and requests it shows. A cloud pod sees only its conversation's folder and the project's artifacts, so a step there names a file in one of them or an attached file. The event also names the project the turn ran in (project_id), which a step's file path is relative to. The formatter (cowork/harnesses/anton_harness/stream_formatter.py) finds them in a worker thread once the answer is final, and figure_sources.py does the matching. It matches each figure only against this turn's steps: the scratchpad's cells, the results of the web tools anton runs itself (a provider's own web search never reaches the server, so a figure from it has no source), the files read_text_file read, other than the agent's memory files in .anton/memory, and the results of the connector tools that return the customer's records. Those memory reads, like every other tool, hand back the agent's own text or the user's (an earlier answer from recall_history, an artifact, a memory), so their results hold no figure. A cell that reads a memory file never names it as its file, and a cell that makes no request and whose only source would be the memory files it or an earlier cell read is not a step. That rule misses what the cell reader can't see: a cell that also computes a figure from its own numbers or loads data through a call the reader doesn't classify (a library's download, a pickle) is left out, and a memory path built from a name or found by listing the folder reads as any other file. A connector tool is an MCP tool anton registers as {engine}__{tool}, or {engine}__{connection}__{tool} when the turn has two connections of one engine. anton's returns_records (anton.core.mcp.access) says which ones return records, such as HubSpot's search_crm_objects. Such a step is a query labeled by its engine and tool (hubspot: search_crm_objects), with the connection when the name carries one (hubspot (crm2): search_crm_objects). Its result is read as the tool returned it, less the ISO date-times in its records (2026-03-04T09:12:44.512Z): a record's timestamps are not figures, and the minutes of the created and last-changed times every HubSpot record carries would source almost any small number. A connector call that carries a sql argument, as HubSpot's query_crm_data does, lists that argument as the step's SQL as the call sent it, through the credential sweep and the length cap (MAX_QUERY_CHARS) every step's SQL passes. It is not split into statements or filtered to read statements, as a cell's SQL is, so an argument that also holds a write statement is listed whole. A connector tool that returns docs, schema or account details (HubSpot's tool_guidance or get_properties), a write tool and a tool of an engine anton has not classified are not steps. Neither is any connector tool when the installed anton has no returns_records, as with a local editable anton older than mindsdb/anton#585, which added it. A call of a web tool, read_text_file or a connector tool is a step only when its tool_done event said ok=True, since the turn's tool rows hold a failed call's error as an ordinary result and that error can quote numbers. A figure written in a time unit (12h, 22 hours, a 13.08-day lead time, 22 business days, or the first end of 22-25 hours or 22 to 25 hours) is not held by a step whose output names time units in the plural (lead_time_days, hours=21.98) or right after a number (36h, a 30-day window) but never the figure's, so a 22 in a days column doesn't source a 22 hours the model worked out. Only the output's units rule a step out: a window in its code or label (interval '12 months', "over the last 30 days") doesn't, and a unit its code or label names keeps the figure. A step whose output names no time unit can still hold such a figure, and a rate ($42 per hour, 10 units/day) has no time unit. The rule drops a true source whose output prints the figure's value but names only another unit, such as a cell that turns days into hours and prints the bare hours beside a days column, so that figure has no source. It doesn't fire on a days column named only in the code (SELECT lead_time_days ...) whose output prints a bare row, nor on a range joined by "and" (between 22 and 25 hours), so such a figure keeps its step. A cell that prints a figure also names the earlier cells that read the data it was computed from, such as the cell that loaded a file into a frame the printing cell sums. That cell may be an earlier turn's, since each turn's cells pass on the names and data they bound. The formatter reads the earlier turns only when the answer has figures and this turn ran a cell, and then only their last 50 cells. That never changes a figure's origin; past that, a figure can only lose the earlier cell that loaded its data, and its step the SQL or request of a function or string bound that far back. An earlier turn's step never holds a figure itself. A figure that no step of this turn holds and the question doesn't write gets the origin earlier_answer when one of the conversation's last 20 answers wrote it in the same form with an origin other than none. It names no source, and nothing of that answer is copied. The same form is the same value, exactly, with the same sign, currency sign, scale and percent, and the same unit written right after the digits (3x, 40kg, 12h). A unit after a space (12 hours) and an ISO currency code (4,600 EUR) are not part of it, so 12 hours repeats 12 days, while 12h and 12hrs differ. The turn reads those answers' saved events when it starts, in a database unit of its own (read_earlier_answers in cowork/services/conversations.py), and only for a conversation the caller owns. An answer with no event (one with no figures, a stopped or failed turn) still takes one of the 20 places. When the read fails, the server logs it and no figure is marked as repeated. The cowork client shows that origin's mark; a client that doesn't know an origin drops that figure, so it shows unmarked. The scratchpad's results arrive as stream events, and the other tools' arrive in the turn's tool rows. A call whose result anton marked is_error (one it refused to run, or one an error cut off) is not a step. A figure no step holds counts as the question's when the user typed it, in the question or in an answer to the agent's question during the turn. Context the client appends to the question for the agent (a connect form's state, picked Drive files, attached Notion pages) does not count. The event is saved with the answer's events, so a reload shows the same marks. A direct answer from the routing gate gets the event too, with no steps, so each figure in it is the question's, an earlier answer's or has no source. A non-streaming response carries the same steps and figures as output[0].figure_sources. A turn the user stops, a failed turn and a stalled turn get no event, and neither does an answer whose last text is anton's failure reply ("An unexpected error occurred: ..."). A turn that stops early at a limit or another hand-back (status: "incomplete") gets the event as a finished turn does, since its answer is whole. The number in the sentence anton opens a hand-back with to name the limit it reached ("This task reached its limit of 25 steps per request", from ChatSession._limit_reached) is anton's setting, so it is no figure and gets no mark. A turn that compacts or hard-truncates its context mid-way still hands the formatter every tool row it wrote, the ones the compaction folded into its summary included, when its anton has ChatSession.turn_messages(). In process the server calls it only when the installed anton has it; a cloud pod with it sends those rows in its history frame. Its figures are then matched like any turn's. None of those rows are saved: the turn ran near or over the model's context window, so the next turn replays it as text only. A cloud pod or a local editable anton older than mindsdb/anton#585, which added turn_messages, sends no rows after a compaction. When such a turn called read_text_file on a file outside .anton/memory, a web tool anton runs itself or a connector tool that returns records, and that call's tool_done said it succeeded, a figure that no step holds, and that neither the question nor an earlier answer gives, is left out of the event, so it gets no mark rather than "no source". When finding the sources fails, the server logs it and the answer goes out without them.
A step's label, web page, SQL and requests pass through a best-effort credential sweep: _redacted in cowork/harnesses/anton_harness/figure_sources.py, whose docstring lists every key name and token shape it knows and the forms it still misses, and where a connector with its own secret fields adds them. A secret under a key name the sweep doesn't know, or passed by position, is sent as written, as the step's code in the scratchpad events already shows it. A cell's SQL lists only read statements, each on its own, so a statement after one in the same string is left out; a string that two databases' quoting, comment, bracket or dollar-quote rules would split differently lists no statement, and cell_queries.py names the forms the split still misses, such as a ; inside a bracket before any quote, where Postgres ends a statement, a ' inside Oracle's q'[...]' quoting, and a ; between backticks, which Postgres reads as an operator character. The sweep masks the password a listed statement sets with IDENTIFIED BY, with a quoted value after a secret key's name such as PASSWORD '...', SECRET '...' or SESSION_TOKEN '...', or after REPLACE, as it does in a string literal that holds one. A request carries its method and URL template, never headers, bodies or query values.
A background scheduler loop polls the database every 30 seconds for due schedules, supporting once, every_15_minutes, every_30_minutes, hourly, daily, weekdays, and weekly cadences. The poll and each run's reads and writes are database units, so a full pool refuses them after POOL_TIMEOUT rather than stalling the server. A run whose first write is refused does not start, and a cron slot then stays due for the next poll. So does a run refused before its turn starts by a busy pool, a busy file store or an auth outage, and the empty conversation it made for the turn is deleted. Only its rows go: the run made no folder for it, and the delete runs in a database unit, which a slow file store must not hold. A run's last two writes, its outcome and its finish, try again while the pool stays full, for up to 10 minutes, so a busy moment never leaves a run at running or its slot due. Every replica runs this loop against the same database, and one cron slot still starts one run. A run's row names the slot it fires for (schedule_runs.slot_at), and a unique index on the schedule and that slot makes inserting the row the claim. So when two replicas find the same slot due, the second one's run does nothing. A run renews a lease on its row every 30 seconds while it lives. A run with no renewal for 2 minutes belongs to a server that died without recording how the run ended, and the next poll on any replica fails it. Its slot fires again when its turn never started, and is used up when its turn may have run, so a lost replica neither skips a slot nor runs one twice. A run that leaves its slot due, because its turn never started, a shutdown cut it, or its schedule was paused before it failed, gives the slot back for the next poll to claim. Every run records its outcome: whether its agent acted before it failed, was stopped, or left no record of how it ended. Before handing off execution, the run saves execution_claimed_at; deleting its chat never clears that claim. After a crash, a claimed run is treated as potentially executed. The upgrade conservatively marks older runs as claimed because a missing chat link cannot prove they never started. A schedule has one run going at a time. A process runs at most COWORK_SCHEDULED_RUNS_PER_PROCESS scheduled runs at once (unset: 3 locally, 16 with Redis-backed turns); a due slot over that stays due for a later poll, and Run now answers 409 while the task is running or the process is full. A recurring schedule that fell behind, because the app was not running, runs its latest missed slot once if that slot is no more than seven days old; the slots it passed over are added to missed_runs. A one-off runs up to an hour late; from an hour on it is marked missed and runs only on Run now, which uses a one-off up. Pause records paused_at and stops what would start on its own; a run already going finishes and nothing follows it. Resume clears the pause and the failure streak. A slot that a run gave back before its agent acted, before or during the pause, runs once more after resume, unless a later slot came due meanwhile; otherwise the schedule waits for its next slot. Three cron slots in a row that settle failed set auto_paused_at and pause the schedule; a success resets the streak. Confirmed server-shutdown interruptions leave the streak unchanged; timeouts and interruptions without a shutdown marker still count. Each schedule reports its state: active, paused, auto_paused, missed or done. Run now marks a one-off done only after confirmed successful completion; other endings preserve its previous schedule state. Each run creates a conversation and is tracked in schedule_runs. Deleting that conversation does not delete the run: the run keeps its status, timings, and error as audit history, and only its link to the conversation is released. A channel binding pinned to the conversation is released the same way, so the external chat stays bound to its project and the next inbound message starts a fresh conversation.
Schedule responses include lastRun, the latest finished manual run or cron run that retained its slot. Abandoned cron attempts remain in run history. Each run reports its trigger, outcome, and a readable failure reason. Internal exception details stay in server logs. An ending without reliable evidence neither extends nor resets the failure streak. Terminal records are read with a bounded wait, including when Redis makes the record visible before the producer finishes closing its buffer.
COWORK_SCHEDULED_RUNS_PER_ORG optionally caps running tasks per account across replicas (1–256); unset, no account cap is imposed. Hosted limits from auth are tracked in ENG-4064. A full account is skipped before process admission; claims still check the limit atomically. These deployment settings require a restart.
COWORK_SCHEDULED_RUN_MAX_SECONDS bounds a run (default 30 minutes, 5 minutes to 24 hours, with a floor covering the remote controller's turn and setup budget). A reaped cloud run has its abandoned turn cancelled. This process excludes runs it is actively driving from its own reaper, including after laptop sleep. Settlement and finalization share a locked transaction, so a late writer cannot count a reaped failure again. Deleting a task stops its active turn on the next lease check; pausing it lets the current turn finish.
Run now accepts ?trial=true to try a task without consuming its scheduled occurrence or changing its failure streak. Runs report trigger as cron, manual, or trial; the existing isManual and trial fields remain available. Trial migration 6f1b8d3a5c27 follows the last-run index migration a9e2b6c4d810.
Hosted Cowork reaches Redis through one URL, COWORK_TURN_REDIS_URL, whose Service follows the primary. cowork/turnqueue/redis_client.py builds the server's two Redis clients so that a dropped connection, a Redis restart or a primary failover doesn't fail the call in flight:
- A call that fails with a connection error, a timeout, a refused or timed-out connect, or a
READONLYorLOADINGreply is sent again with jittered backoff (0.25 s base, 2 s cap), until 30 s after the call started. The blocking client, which conversation delete uses from a threadpool thread, stops after 10 s, and so does one delete's whole Redis cleanup: a conversation or turn delete runs its calls under one 10 s deadline, and a project delete runs every conversation's under one, so a Redis that does not answer holds the request for about 10 s. No connect, write or read waits past a deadline: each is cut to the time left, though to no less than 50 ms, so a call ends at most a few of those 50 ms waits past its deadline, and a call that starts after its group's deadline has passed fails at once without reaching Redis. Each new attempt opens a new connection, so after a demoted primary answersREADONLY, the next attempt goes back through the URL and reaches the new primary. - A new connection's handshake (
AUTH,SELECT) is retried with its connect, under the same deadline. redis-py 5 retries only the TCP connect, so a Redis that accepts a connection and closes it mid-handshake, as a restarting Redis or a SentinelCLIENT KILLdoes, would otherwise fail the call at once. - A rejected password (
WRONGPASS,NOAUTH) or a refused permission is raised at once and never sent again. - The 30 s is one deadline per call. redis-py reconnects inside a command's own retry loop, and that reconnect spends what is left of the call's 30 s rather than another 30 s.
redis_deadlineshortens it for a group of calls:/answer's wait for the pod's verdict uses it, so its reads stop retrying when the 15 s wait ends. A transaction whose connection closed while it watched keys ends without a reconnect: the watch ended with the connection, and redis-py 5 would reconnect to sendUNWATCHafter the call, outside its deadline. - A connection opens within 2 s and a reply arrives within 5 s, and neither waits past the call's deadline, or that attempt fails. Every blocking read blocks for 2 s, under the 5 s, so a reader waiting on a quiet stream is never cut as a dead socket. A primary that is demoted answers its blocked readers
-UNBLOCKED, and the reader reads again. - A pooled connection idle for 15 s is PINGed before its next command, so one a failover left half-open is replaced before a write goes out on it.
- redis-py 5's async pool holds its lock while a new connection connects. The server's pool connects after it releases the lock, so callers don't queue behind one caller's connect retries.
The turn claim (cowork/streaming/turn_claim.py) does not ride through a failover. Each of its calls stops after 5 s, and a dropped connection or a READONLY reply inside its WATCH transaction ends that transaction, so a send that arrives during a failover is refused with 503 and Retry-After, and a renewal that fails tries again 10 s later.
The job XADD that queues a remote turn is sent at most once (send_once), since the controller would run a job queued twice. When its reply is lost, or does not arrive within the 5 s read timeout, nobody knows whether Redis applied it, so the producer reads the newest 5 entries of the conversation's job stream and looks for the turn's correlation id. When the job is there, the turn goes on as if the XADD had answered. When it is not, or that read fails too, the producer writes the turn's cancel flag, so a copy the controller picks up later stops at its first check, and the turn ends with response.failed. A READONLY or LOADING reply to the XADD is still retried, since Redis refused the write without running it. Every other write is retried. The RPUSH that delivers an ask_user answer can then queue its entry twice, with one answer_id: the pod answers the question once, nothing renders its verdict on the copy, and /answer returns at the first verdict. A failed /answer would be worse: the client would retry, get 409 and resend the typed text as a new message. Inside the answer's MULTI, a refused RPUSH aborts the EXEC, since Redis runs MULTI and EXEC while it loads. A turn buffer append is retried, so Redis can hold one record twice, and a reader of the buffer yields each seq once. The controller numbers each turn's replies with n, counting up by one. The relay drops a reply whose n equals the last one it applied, which the controller wrote twice. A reply whose n skips a number is applied, and the relay logs remote_reply_gap with both numbers, since a reply never reached the stream. A reply whose n is below the last one comes from a second run of the same job, and the relay fails the turn rather than mix the two runs' answers. A controller that sends no n has every reply applied.
Conversation, turn and project delete clear a conversation's Redis buffers and its turn index entry only with COWORK_STREAM_BACKEND=redis, so desktop and the single-container server never call Redis on a delete. The buffer walk (SCAN) holds one connection of the pool for all its pages, and starts again from cursor 0 when a page went out on a new connection, since after a failover that connection can reach a server the cursor means nothing to. Holding the connection keeps another threadpool thread from reconnecting it unseen between two pages.
In org mode (COWORK_TENANCY_MODE=org) the server pushes to the MindsHub Cowork mobile app through APNs (iOS) and FCM HTTP v1 (Android); everywhere else push does nothing. The app registers each phone with POST /api/v1/push/devices ({"token", "platform": "ios"|"android", "app_version", "environment": "production"|"sandbox"|null}, 204) and removes it on sign-out with DELETE /api/v1/push/devices ({"token"}, 204, only the caller's own token). Outside org mode both answer 404. A token names one phone, so registering it again moves it to whoever registered it last. Rows live in push_devices.
Three events push, each carrying kind, and path (an in-app route, /c/<conversation_id>) or url (the artifact's shared view):
task_finished: a task someone started in the app ends, to its owner: "Finished.", or "Didn't finish." on an error, an interruption, a stall or the idle reaper. A Stop, a server restart and a deleted conversation send nothing, and neither does a turn someone was watching live when it ended: a reader of its stream (the POST that started it orGET /responses/tail, on any replica) holds it back, for 15 seconds after it streamed the end (cowork/streaming/watchers.py).scheduled_run_finished: a scheduled run ends, to the schedule's owner, finished or not. A trial run (?trial=true), a cancelled run and a run whose slot fires again send nothing. A run another replica reaped is pushed by the reaper.artifact_shared: someone saves an artifact's sharing with "Notify people", to each person they named by id, the same people MindsHub emails. Members of a team or organization the share opens to get no push.
Every push is sent off the path of the work that caused it, with short timeouts, and a failure is logged (push_* events) and dropped. A token APNs or FCM reports as gone (410, Unregistered, BadDeviceToken in both environments, FCM UNREGISTERED or 404) is deleted. An iOS token production refuses as BadDeviceToken is tried once in sandbox, and the environment that took it is kept.
Settings, all optional, from the cowork-push Secret in the chart: COWORK_PUSH_APNS_KEY_P8 (the .p8 auth key from Apple Developer > Keys, PEM or base64), COWORK_PUSH_APNS_KEY_ID, COWORK_PUSH_APNS_TEAM_ID, COWORK_PUSH_APNS_TOPIC (default ai.mindshub.cowork), and COWORK_PUSH_FCM_SERVICE_ACCOUNT_JSON (a Firebase service account allowed to send messages, JSON or base64). A platform whose settings are missing sends nothing. The optional COWORK_PUSH_ENABLED key in the same Secret is the kill switch: set it to false to turn push off (default true). Restart the pods after changing these environment-backed settings.
Data lives in two places: a SQLite database for structured records and the filesystem for project files and agent workspaces. Understanding both is essential.
The
~/.coworkpaths below are the default (prod) home. The desktop app runs one of several build channels, each with its own isolated home (~/.cowork-dev,~/.cowork-stable, etc.) selected viaCOWORK_HOME. See the Cowork frontend README → Build Channels for the full mapping.
- Location:
~/.cowork/cowork.db(override withDATABASE_URI) - ORM: SQLModel (SQLAlchemy + Pydantic)
- Migrations: Alembic (
cowork/db/alembic/versions/). Startup runsalembic upgrade head(singular), so the graph must have exactly ONE head: if two branches each added a migration on the same parent, every fresh boot aborts with "Multiple head revisions". After merging or rebasing, checkalembic heads; if it prints two revisions, add a no-op merge revision whosedown_revisionis the tuple of both heads (seef4e2c1a9d3b7for the pattern).
Key tables:
| Table | Purpose |
|---|---|
projects |
Project metadata and filesystem path |
conversations |
Conversation threads, linked to a project |
messages |
Individual messages with role, content (JSON), and harness tag |
message_events |
Streaming event payloads for a message |
files |
Metadata for uploaded files (path points to filesystem) |
schedules / schedule_runs |
Recurring prompts and their execution history |
settings |
Key-value user settings; sensitive values Fernet-encrypted |
pins |
User-pinned items (conversations, artifacts, etc.) |
push_devices |
Phones the mobile app registered for push (org mode) |
channel_* |
Channel installations, bindings, sessions, and events |
shared_resource_attributions |
Creator and latest-editor user IDs for org-shared filesystem resources |
shared_resource_mutations |
Append-only actor trail for allowed org-shared mutations, including deletes |
All models use UUID primary keys with auto-tracked created_at/modified_at timestamps.
~/.cowork/
├── cowork.db # SQLite database
├── .master_key # Fernet encryption key for settings
├── skills/ # COWORK_SKILLS_DIR — canonical SKILL.md store
│ └── <slug>/SKILL.md # one folder per skill (see docs/SKILLS.md)
├── projects/ # COWORK_PROJECTS_DIR
│ ├── general/ # Default project (always exists)
│ └── <project-name>/
│ ├── <user & agent files> # Working directory visible to agents
│ ├── skills/ # symlinks to skills enabled for this project
│ │ └── <slug> -> ~/.cowork/skills/<slug>
│ └── .anton/ # Private agent workspace
│ ├── artifacts/ # Agent-produced outputs (HTML apps, docs, etc.)
│ │ └── <slug>/
│ │ ├── metadata.json
│ │ └── <files>
│ ├── memory/ # Persistent agent memory by category
│ └── context/ # Project context for agent runs
├── files/ # COWORK_FILES_DIR — uploaded files
│ └── <file-id>/<filename>
└── data-vault/ # COWORK_VAULT_DIR — encrypted connector creds
└── <engine>/<connection-name>/
The database holds structured metadata and relationships (which messages belong to which conversation, which conversation belongs to which project). The filesystem holds the actual content agents work with — project files, artifacts, memory entries, and uploaded documents. The files and projects DB tables store filesystem paths that point into the directory tree above.
This split comes from an ongoing migration from a purely filesystem-based architecture. Structured data that benefits from querying and relationships, such as conversations, messages, settings, and schedules, lives in SQLite. Components that are inherently file-based, such as project working directories, agent artifacts, harness-managed memory, the connector vault credentials outside MindsHub mode, and skills, remain on the filesystem by design. In MindsHub mode, OAuth credentials live per user in MindsHub's Data Vault instead. Skills are canonical SKILL.md files, so they can be edited, uploaded, and distributed per project; see docs/SKILLS.md. See docs/SERVER_MIGRATION.md for the full migration story.
Agents (via their harness) have read/write access to their project's working directory and the private .anton/ subdirectory. They do not access the SQLite database directly — all DB interaction flows through the service layer.
On a hosted deployment COWORK_HOME is a shared NFS mount (EFS), mounted hard. A stat that the file server doesn't answer blocks its thread with no time limit. On the event loop such a call stops every request on the pod, liveness included. On anyio's default threads it holds threads that sync routes and every request's sync dependencies share.
The artifact routes' lookups and reads, and a turn's artifact bookkeeping, run their file store work through run_efs in cowork/db/units.py, beside run_db. Other file store work still runs on other threads, and a stuck mount can hold those too. On anyio's default threads: the live-app share and copy routes through run_in_threadpool, and the sync handlers and FileResponse bodies of desktop-only and loopback-only routes. In org tenancy no response body reads the file store there, and neither does any sync route but the loopback-only ones, which require_local refuses to any peer but loopback. On asyncio's default executor through asyncio.to_thread, for example: the publish uploads, an artifact delete's take-down of its live pages, the live-app reads and the startup owner backfills. run_efs works like this:
run_efshas its own 100 threads per process by default (EFS_SLOTS). A started call keeps its thread until it returns, so a stuck mount holds that many threads at most.- One person holds at most 10 of them at once by default (
EFS_SHARE). A heavy client is then refused before anyone else waits. One organization's work that runs for no person, such as a channel turn, shares 10 the same way. Nothing caps an organization: ten busy people fill the slots, and anyone else, in any organization, then waits for one. - A caller waits for a slot until
POOL_TIMEOUT, then getsFilesystemBusy, which the app answers with503andRetry-After. Itsfilesystem_busylog line names the limit that ran out inlimiter:efs_share, the caller's share, orrun_efs, the slots. It names the share's key inshare: the person's user id, ororg:<org_id>for an organization's work with no person. - A turn's artifact bookkeeping (the snapshot at turn start, the diff, autopublish and the cards at turn end) passes
deadline=Falseand waits as long as it takes. In MindsHub mode nothing records a turn's artifacts later, so a refusal there would lose them. The scheduled self-heal of a project's artifacts (below) waits the same way, outside any turn: at most one runs at a time per member and project, and one more per project for its channel turns, and a shutdown cancels it. - A caller cancelled after its work started waits for the work to return, as with
run_db, so a lock the work holds is released before the caller unwinds. - A caller cancelled while it waits for a slot leaves at once, and its work never runs. So a remote turn's turn-end diff and the database record of what it found run as one step under
run_turn_end_record, which isrun_to_completiontracked for shutdown: a Stop or a shutdown that lands while either waits or runs waits for both, and the rows are written once. The in-process harness's cleanup at turn end runs underrun_to_completionthe same way. - During a file store stall, a Stop waits for a slot with no bound, and so does a
/responses/cancelthat reached the replica running the turn. A shutdown waits only until the end of the lifespan teardown, whose last step waits up to 8 s (TURN_END_RECORD_DRAIN_SECONDS) for the turn-end records still running (drain_turn_end_records). A stall that clears within that wait keeps the turn's artifact rows. The event loop's close then cancels every task still running. It cuts a diff that has not returned and a record still waiting for its database slot, so that turn's artifact rows are lost, and the last step logsshutdown_turn_record_lostat ERROR with the conversation, the project,touchedandcount.touchedholds the new artifacts' slugs its diff found, or, when the diff had not returned, the artifacts the pod'sturn_completednamed (artifacts_touched, below; empty for a pod that sends no usable list). It holds the first of them in sorted order, none longer than 255 characters (MAX_ARTIFACT_NAME_CHARSincowork/turnqueue/models.py): at most 50 (TURN_END_RECORD_NAMES_LOGGEDincowork/db/units.py), and only as many as fit in 12 KiB as the line encodes them (TURN_END_RECORD_TOUCHED_BYTESthere), where a multibyte character costs up to 4 bytes and a control character 6. So the line stays under the 16 KiB a container runtime splits a line at, whatever characters the names hold.countsays how many the record held. A web turn whose record is cut there still saves its answer, markedturn_lost, before the loop closes. A web turn whose pod had finished, and which was still recording its artifacts when the shutdown ended it, starts its answer's save once that record ends; a save that starts as the loop closes may not finish. A file store call already running on the stuck mount keeps its worker thread, which is not a daemon thread, so the process stays alive until the kubelet's SIGKILL. - Autopublish releases each publish lock through
run_efs_cleanup, underrun_to_completiontoo, including when a cancel lands while it takes the lock, and when a cancel or a failure lands while it mints its publish key. It keeps one lock on purpose: that of an artifact whose upload it handed to its thread, running or still queued, when a cancel or its publish timeout lands. A running upload may still record the publish, and autopublish cannot tell a queued upload from a running one. So the lock stays until it is 180 s old, three times the 60 s publish timeout, counted from when it was taken. Until then later turns skip that slug aslock_busy. At a shutdown during a file store stall, the event loop's close can also cut a release still waiting for a slot, and that lock stays until the same age.
The artifact routes read in turn. One run_db unit reads the project row, the conversations the caller owns and the artifact's location row (below). One run_efs call then opens the folder that row names, or finds the artifact roots and the folder. The caller's own folder answers; for a reviewer, a co-member's folder answers behind the owner's grant, checked in that same call. A lookup that listed the roots records what it found in one more run_db unit when that can change the row, and a refusal there is only logged. The artifacts list reads its projects, the caller's conversations, the held-root rows and the newest sort-time rows (below) in one run_db unit, then opens its roots and reads its cards in one run_efs call. A route a reviewer may use too reads what it needs in the lookup's run_efs call, once the lookup has let the caller in (review_artifact_and_read), so it waits for a file store slot once. Those routes are the review entry, the draft link, the private draft preview (/drafts/{project}/{artifact}/{path}) and the draft host's preview. A preview checks the file's path and opens the file in that call. It builds a UTF-8 HTML preview there too, and streams anything else from the open file after the call. A route only the owner may use checks ownership in another run_db unit after the lookup. Only then does it read the folder, in one more run_efs call. So a read never runs for a caller that check refuses, and that second wait can still answer 503. A save or a restore that changed the source then takes the artifact's publish lock, to republish the artifact if it is live. The access change takes the same lock. Each try at that lock is another run_efs call, and a busy file store counts as a busy lock. The route tries again every 0.1 s for up to 60 s. If it gets no lock, the access change answers 503, and a save or a restore keeps its change without the republish. The request's own session never takes a connection while a file store call waits. A delete runs in three steps. A file store call pins the folder, checks it and lists its live pages. The pages come down on asyncio's default executor, so a slow publish service holds no file store slot. Where the server keeps owner records for published pages (COWORK_TRUST_IDENTITY_HEADERS), they come down in that first call instead, since each page's owner check holds the publish record's lock across its take-down. A second file store call then records them as unpublished and removes the folder. In MindsHub mode the delete first revokes the artifact's draft review rule, and once that is gone the delete's own file store calls wait for a slot as long as it takes, rather than answer 503 and leave the folder's review marker without its rule. An upload whose database row is refused answers at once, and its folder is removed in the background once a file store slot frees.
In org tenancy each artifact id has a row in artifact_locations: its project, its conversation for a legacy per-conversation root, and its folder name. A lookup reads the row with the project, then opens that one folder inside a root the caller may already read, and serves it only when the folder's metadata.json names the requested id. A metadata.json that is not a regular file, such as a FIFO an agent left in its place, holds no id, and no read of it waits for a writer. Without a row, or when the folder no longer holds the id, the lookup lists the project's roots as before. That scan reads every folder of the project that holds the id, a co-member's legacy roots included, and records what it found: the folder, when one holds the id. An id held by two folders of one project is marked conflicted instead: its lookups keep listing the roots and answer 409 when two of its folders are in the roots that decide the lookup (the caller's own, or every member's when the caller's own hold none), as before the table. While the row's folder holds the id, such a lookup reads the caller's own roots first, without a stat of any co-member's workspace, reads every member's only when the caller's own hold no folder, and records nothing. The mark clears when such a lookup finds the row's folder alone and a strict count of every root of the project, each co-member's legacy roots included, agrees. Each replica runs that count at most once per id every 15 minutes, so once the other folder is gone, the mark clears at the first such lookup on a replica that has not counted the id in the last 15 minutes. The backfill below clears a mark too, but only while its pass runs. A folder whose name is not valid UTF-8 counts toward a conflict, but no row names it, since no row can store it. Only a caller who reads the row's root as their own moves or drops a stale row, and a folder that fails to read leaves the row as it is. There is one row per id in an organization, so an id held in two of its projects names one of them, and each lookup from the other project lists that project's roots and leaves the row as it is. Only such a scan, or the backfill below, writes a row, since each reads every root of the project. A turn end sees only its own folders, so it writes none, and a new artifact's first lookup lists the roots once. A turn's folder whose id has a row naming another folder of the project marks the id conflicted. A copy no turn end claims, such as one a shell cp -r makes, is caught where the project's whole artifacts folder is still listed: at a remote turn end whose pod sends no list of the artifacts it touched, and at the project's scheduled self-heal (below). Each reads the id of every folder it lists, and marks an id conflicted when its row names another folder of the project. A row whose folder the listing read and found gone or holding another id, as after a shell mv, is stale instead, and the id's next lookup moves it. Each listing sorts its own errors by errno: an entry that is gone, is not a folder, or is a link loop (ENOENT, ENOTDIR, ELOOP) holds no artifact, and any other error leaves the entry unreadable, whatever Python's own Path.is_dir would answer (3.12 and 3.13 raise on EIO, 3.14 answers False). The id reader never follows a link, so an entry that is a link, or whose metadata.json is one, is never counted unreadable. Neither is the artifacts folder itself, or any entry below it, when that folder is a link, even one to a folder the server cannot list: no listing of it can ever find an id. No listing includes the project's .locks folder. A folder the listing could not read is never taken for one that is gone, so its row still marks the copy. A listing that could not read a folder, or the project's artifacts folder itself, logs one warning (event artifact_location_listing_unreadable, with the project and a reason: folders_unreadable, with the count of folders it could not read, or artifacts_folder_unreadable, with no count, since it then read no folder below it) and asks for the project's location check again, even when it found no id. So a folder the listing could not read is listed again at the next self-heal, and a copy in it is marked at the first listing that reads it. That check publishes nothing. Each replica keeps its own requests and runs its own poll, and a listing asks again on the replica that made it, so a folder that stays unreadable costs one listing of the project every 15 minutes (SELF_HEAL_INTERVAL_S) on each replica that has listed it, for each member and project, and for each project's channel turns. Those listings stop when the folder reads again or the replica restarts. With the pod's list, a copy made in a turn, a channel turn's included, is marked within about 15 minutes of that turn when its reads succeed, and otherwise at the first later self-heal whose listing reads the copy's folder, whether or not autopublish is on, even when that turn's diff fails or a Stop cuts its turn end, and its id's lookups then answer 409. Without the pod's list, the turn end's own listing marks it; a Stop waits for that listing and its record, and a turn end whose diff or record fails asks for the project's self-heal instead. Rows move with a task's folders and are deleted with the artifact. A one-time pass at startup (cowork/services/artifact_location_backfill.py, sentinel setting _artifact_location_backfill_v1) records the artifacts made before the table, holding one file store slot while it walks. A project whose roots or folders it cannot all read gets no rows from it: only a folder that is gone, is a link, holds no metadata that parses, or is one the server may not read at all (EACCES, EPERM) counts as holding no id. The pass walks the projects in id order and records each one it finishes, and each one it could not read, in the _artifact_location_backfill_v1_progress setting row. An id whose row the database refuses, or whose row another writer deletes mid-write, is skipped and logged with event artifact_location_backfill_skipped. An id whose one folder has a name that is not valid UTF-8 is skipped too. A stop at shutdown, a project it could not read, or any other database error leaves the sentinel unwritten, and the next start walks only the projects the pass has not finished and the ones it could not read that still exist. The pass writes its sentinel, and deletes the progress row, once it reaches the last project with none left unread. Its summary lines (event artifact_location_backfill_summary) count the ids recorded, duplicated, conflicts and skipped, and the projects unread. To walk every project again, delete both setting rows and restart cowork-server. This pass, the held-root backfill and the sort-time backfill (below) run one after another in one startup task. Shutdown sets their stop and waits at most 2 s for the task (LOCATION_BACKFILL_SHUTDOWN_SECONDS), then goes on. The desktop writes no rows.
In org tenancy a held_artifact_roots row names each artifact root the server has seen hold an artifact folder: a project's shared root, or a conversation's legacy root (cowork/services/held_artifact_roots.py). The artifacts list opens only those roots, so a project or a conversation workspace that never held an artifact costs it no file store call. A delete by an artifact's old folder name looks in the same roots, so it finds what the list shows. A turn end that created, changed or listed folders in its project's shared root, or could not read a folder's time there, writes the root's row, and so do the self-heal's full listing of that root and a task's move into a project. The project files PUT (cowork/api/v1/endpoints/project_files.py) writes the row of the root it writes an artifact folder into, a project's shared root or a conversation's legacy root. A comparison side continued into a project writes the row of its carried workspace's legacy root when that root holds a folder. A task's move whose row write fails asks for the destination's location check, whose listing writes the row. A writer that finds the root's row already there locks nothing. Before it adds one, it takes FOR KEY SHARE on the root's conversation, for a legacy root, and then on its project (lock_root_parents), the order a project delete takes them, so the insert never deadlocks a project or conversation delete that cascades into the rows. A row is never removed, except with its project or conversation: a root emptied later costs the list one open. A row is a hint, not an authorization record: the list reads a legacy root's row only for a conversation the caller owns, and opens a row's root only for a project the caller's scope returns. At startup, after the location backfill and in the same task, one replica walks every project's shared root and every conversation workspace once, and writes a row for each that holds a folder the list would open. A root that is missing, is a link, or that the server may not read holds none. Any other read error counts the project as unread (event held_artifact_roots_backfill_unread). So does a project whose folder was there before its walk and is gone after it, or whose row no longer holds the path the pass walked, as when a rename or a delete lands during the walk (event held_artifact_roots_backfill_moved), and it gets no rows. A folder already gone before its walk, at a path the row still holds, holds no root. Both checks of the project's folder, here and in the sort-time backfill (below), take one the server may not read (EACCES, EPERM) for one that is gone. The pass walks the projects in id order and records the last one it finished, and each one it could not read, in the _held_artifact_roots_backfill_v2_progress setting row, so a stop at shutdown or a restart costs only the project it was in: the next start goes on from there. Once the pass has walked the last project, it deletes the progress row and writes the sentinel setting _held_artifact_roots_backfill_v2, whose value lists the projects it counted as unread ({"unread": [...]}). From then on the list trusts the rows for every other project, and finds the listed projects' roots on the file store as before. Each later pass, at a start or at the startup task's next run, walks only the listed projects and rewrites the list, to an empty one once they all read, and a pass that finds the list empty walks nothing. Each replica runs that startup task. It runs the backfills again every 5 minutes (ARTIFACT_BACKFILL_RETRY_SECONDS in cowork/server.py) while this sentinel or the sort-time backfill's (below) is missing, as when another replica held a walk's lock and died, or a walk here failed, so a walker that died does not leave either unwritten until the next deploy. Once both exist, it runs them every 30 minutes (ARTIFACT_BACKFILL_UNREAD_RETRY_SECONDS) while either lists an unread project, since every list reads every folder of such a project. It stops once both exist and list none. Each pass logs one WARNING line, since every hosted deployment runs at LOG_LEVEL=WARNING: its start, with the count of projects it will walk (event held_artifact_roots_backfill_started), or why it walks nothing (held_artifact_roots backfill skipped its walk, with event held_artifact_roots_backfill_skipped and a reason: not_org, finished or locked). Its summary lines (event held_artifact_roots_backfill_summary) count the roots held and the projects unread. Until the sentinel exists, the list finds its roots on the file store as before. To walk every project again, delete this pass's two setting rows, the sort-time backfill's two and every artifact_sort_times row (below) in one transaction, and restart cowork-server. Deleting only this pass's rows is not enough: the sort-time backfill has finished, so it times no root this pass holds anew, and the list hides those roots' folders. Do it after any restore of the database or of the file store, and after running an image that writes no held_artifact_roots or no artifact_sort_times rows (a rollback), before rolling forward again: the sentinels outlive the rollback, and a root that held its first folder meanwhile has no row. The desktop writes no rows.
In org tenancy an artifact_sort_times row holds the time the artifacts list sorts each artifact folder by: the newest modification time of its metadata.json the server has read (cowork/services/artifact_sort_times.py). A writer that read a time no newer than the row's leaves the row as it is. A turn end writes the time of each folder it created or changed, and of each folder its tools named as touched (artifacts_touched, below; every folder for a pod that sends no list), read after its revision captures. So a folder whose only change is its metadata.json moves up too. Every writer times only the folders that hold an id, as the self-heal's listing does (id_holder_times), since a folder that holds none builds no card and its row would only take one of the list's 160 newest rows. The self-heal's full listing of a project's shared root, the heal sweep's included (below), reads the root's rows in a database unit of its own before it lists. It writes the time of each folder it read, an older one too while the row is still the one it read before its listing. When it read the root itself, it also drops the row of each folder it found gone or holding no id, again only while the row is still the one it read. It keeps the row of each folder it could not read. A root that is gone counts as not read, so its rows stay. A self-heal whose root is no longer its project's shared root, as after a rename, lists and records nothing. A turn end, a listing, a project files PUT or a comparison's carry (below) that could not read a folder's time, as on EIO, logs one warning (event artifact_sort_times_unreadable, with the count of those folders). For a shared root it asks for the project's location check (above), whose listing reads the time again. Until then the row keeps the time it had. A task's move carries its folders' rows, with their times, into the destination. A row the destination held under a moved folder's new name names a folder that is gone, so the move deletes it first. A move whose rows could not move asks for the location check of both projects. The artifact delete route drops the deleted folder's row. The project files DELETE of an artifact folder's metadata.json, in a shared root or the caller's own legacy root, drops the folder's row too, since the folder then shows no card. It then reads the folder's time again and records it, so a PUT that wrote the file again meanwhile keeps its row. A folder removed another way keeps its row: in a shared root until the project's next self-heal drops it, and in a legacy root, which no self-heal lists, for as long as the root exists, as after an agent's rm there. Such a row costs the list one of its 160 rows while it is among them (below). The project files PUT writes the row of the artifact folder it writes into, in a project's shared root or in a conversation's legacy root. A PUT that could not write a shared root folder's rows asks for the project's location check. A comparison side continued into a project writes the rows of the folders in its carried workspace's legacy root. The PUT and the carry give the held row and the times their own tries, so a failed hold still writes the times. Each logs a failed hold with event held_artifact_root_not_recorded, and a failed write of the times with event artifact_sort_time_not_recorded. A carry that could not read its workspace's folders writes neither row, and logs held_artifact_root_not_recorded. No self-heal lists a legacy root, so a row the PUT or the carry could not write there waits for the next PUT into the folder. Every writer of these rows, a drop or a move included, takes the held-root writer's locks on the root's conversation and project (above) before it locks or adds a row, and locks the rows in name order, so none deadlocks a project or conversation delete, or another writer of the same folders. A write of times first reads the rows without a lock, then locks and writes only the folders whose time is new or rises, so a root whose times did not change locks nothing. After a conflict on the table's unique index, as when another writer added a folder's row after that read, it writes the whole batch again once, then folder by folder. So the list shows the same cards in the same order as a walk of the folders, also after a turn edits an artifact, an edit of metadata.json alone included. A folder that no turn names, such as a shell copy, shows at the project's next self-heal, and a rewrite of metadata.json that no turn names, such as anton's own file bookkeeping, moves its card then. A rewrite the server makes on its own, such as an id widening (ensure_full_id), puts the file's time back, so it leaves the order as it was. A folder replaced under its name by an older one, as by rm -rf report && mv draft report, or by an archive extract or a cp -a that keeps the files' times, holds an older metadata.json than its row. Its card sorts above where a walk puts it until the project's next self-heal, whose listing writes the older time.
Once the held-root and the sort-time backfills have both written their sentinels, the list reads the 160 newest rows of the held roots it may open (twice the 80 cards it shows) in its plan's database unit, and opens only those folders. The table has an index on (org_id, sort_time) for the list of every project, and one on (project_id, sort_time) for one project's. The list ignores a row whose name a walk skips, such as one that starts with a dot (is_listable_folder_name). A project that either sentinel lists as unread has no rows the list can trust: the list reads every folder of its roots on the file store, as a walk does, and sorts them in with the rows' folders as a walk would, newest first, then by root and name. When more rows exist than the 160 it read, a folder past the cut is no newer than the last of them, so the list keeps only the folders at least as new as that row. The query cuts a run of rows that share one time by name, while a walk sorts such a run by root first. So when the cut falls inside a run, the list reads that run in a second query in the same database unit. When the run holds at most 2,000 rows (_TIED_ROWS in cowork/services/artifact_sort_times.py), the list sorts it by root, then name, as a walk does. Past that cap it keeps only the folders newer than the last row. When the rows were cut and fewer than 80 of the folders it keeps build a card, as when folders were deleted behind the server's back, the list reads every folder of its roots instead. It then asks for the location check of each project whose shared root's rows named a folder it found gone, or a folder that opened but built no card. That check's listing drops the rows of folders gone or holding no id, and keeps those it could not read. So a folder whose read keeps failing, as on EIO, keeps its row, and each list that falls back asks again. Each replica's poll runs that check at most once every 15 minutes per project (SELF_HEAL_INTERVAL_S, below). A legacy root's rows ask for no check, since no check lists a legacy root. The request is kept in memory, as a turn end's is, so the list writes no row.
At startup, after the held-root backfill and in the same task, one replica walks the held roots project by project, in id order, and writes the time of every folder in them. It waits for the held-root sentinel, and leaves out the projects that sentinel lists as unread, whose held rows may not name every root yet. It records the last project it finished, and each one it could not read, in the _artifact_sort_times_backfill_v1_progress setting row, so a stop at shutdown or a restart costs only the project it was in: the next start goes on from there. Once the pass has walked the last project, it deletes the progress row and writes the sentinel setting _artifact_sort_times_backfill_v1, whose value lists the projects it counted as unread and the projects the held-root sentinel lists as unread ({"unread": [...]}). The list reads every folder of those projects' roots, as above. Each later pass, at a start or at the startup task's next run (above), walks only the listed projects that the held-root sentinel no longer lists, and rewrites the list, to an empty one once they all read. A pass that finds the list empty walks nothing. A project whose held roots the pass could not all read logs event artifact_sort_times_backfill_unread and counts as unread. So does a project whose folder was there before its walk and is gone after it, or whose row no longer holds the path the pass walked, as when a rename or a delete lands during the walk (event artifact_sort_times_backfill_moved), and it gets no rows. A first pass counts a folder already gone before its walk the same way, since a rename may have moved the folder and not yet committed its new path. A pass over the sentinel's list counts such a project as read, with no rows, while its row still holds that path, so a folder gone for good leaves the list at that pass. Each pass logs one WARNING line, as the held-root pass does: its start, with a reason (first, resumed or unread) and the count of projects it will walk (event artifact_sort_times_backfill_started), or why it walks nothing (event artifact_sort_times_backfill_skipped, with a reason: not_org, held_pending while the held-root sentinel is missing, finished or locked). Its summary lines (event artifact_sort_times_backfill_summary) count the rows recorded, the projects unread, and the projects the held-root sentinel lists (held_pending). A pass that stops at shutdown logs event artifact_sort_times_backfill_stopped instead. The startup task runs the backfills again while either sentinel is missing or lists an unread project (above). To make both passes walk again, delete both sentinels and both progress rows (_held_artifact_roots_backfill_v2, _held_artifact_roots_backfill_v2_progress, _artifact_sort_times_backfill_v1 and _artifact_sort_times_backfill_v1_progress) and every artifact_sort_times row in one transaction, and restart cowork-server. Do it after any restore of the database or of the file store, and after running an image that writes no held_artifact_roots or no artifact_sort_times rows (a rollback), before rolling forward again: the sentinels outlive the rollback, and the list hides a folder written meanwhile, which has no row. The artifact_sort_times rows go too, since each of those can leave a row newer than its folder's metadata.json. A restore of the file store puts back older files. A restore of the database puts back the row of a folder that is gone, and another folder may hold that name now. An image that writes no artifact_sort_times rows deletes a folder but leaves its row for the next folder of that name. The backfill writes a time only when it is newer than the row's, so a pass that walks again keeps such a row's time, and that folder's card sorts above where a walk puts it. Until the sort-time backfill writes its sentinel again, the list reads every folder of its roots, so it hides no folder meanwhile, and deleting the rows costs only the pass's writes. The desktop writes no rows.
The first rollout of an image that writes the held-root or the sort-time rows leaves the same gap as a rollback. The walks can pass a project while an older pod still serves requests, and an older pod writes neither row. A project files PUT it serves after a walk passed that project can write an artifact folder into a conversation's legacy root, or into the shared root of a project with no recent message, and that folder gets no row. The heal sweep (cowork/services/artifact_heal_sweep.py, below) lists neither: it lists only the shared roots of projects with a recent message. So once the last older pod has stopped, walk every project again, as the held-root paragraph above says.
At the end of a remote turn the pod's turn_completed names the artifacts its tools touched (artifacts_touched, at most 1,000 names: MAX_ARTIFACTS_TOUCHED in cowork/turnqueue/models.py, which anton and scratchpad-controller mirror; a longer list counts as none, and so does a list holding a name longer than 255 characters, MAX_ARTIFACT_NAME_CHARS there). The turn claims a listed folder as created only when it appeared during the turn and its provenance in metadata.json names no other conversation. It reads only the listed folders' content for edits. The list alone is not enough: anton also lists a sibling's folder whose files changed during one of this turn's cells. The turn end opens each listed name as one folder directly below the artifacts folder, never a link. A listed folder it cannot read is left out of the turn and logged as a warning. A pod that sends no list gets the older diff: provenance in metadata.json decides what the turn created, and every folder's content is read for edits. The repair route records each folder it queues an agent repair in (queued_agent_repairs, per conversation and keyed by org), and takes the row back when it refuses the repair. A turn reads its conversation's rows when it starts, so with the pod's list the diff also checks those folders for a repair the turn should finish, and lists nothing else. When the rows can't be read, the diff lists the folder only to find a repair queued to the conversation. Once its cards are out, the turn end drops the rows it read whose folders it checked, by id: a repair recorded again during the turn gets a new row, which stays for the next turn, and so does the row of a folder the diff could not read or whose queued repair the turn end could not finish, such as one whose capture found the artifact's lock held. A pod with no list gets the full listing, and every folder is checked for a queued repair.
With the pod's list, autopublish at the turn end also opens only the turn's own folders, publishes those that need it (never the project's .locks folder), and lists nothing. The rest of the project waits for its self-heal. In org tenancy such a turn end asks for one as its record starts, before the diff (request_turn_end_self_heal), a channel turn's too, whether or not the organization's autopublish (artifact_autopublish_enabled) is on. So a turn end whose diff fails, or that a Stop cuts, still asks, though it never reaches its publish. The scheduler's poll loop (cowork/scheduler.py, every 30 seconds) runs it at most once every 15 minutes per member and project, and once per project for its channel turns (SELF_HEAL_INTERVAL_S in cowork/services/artifact_autopublish.py). A self-heal lists the project's artifacts and checks each one's id against its location row (above). One whose listing could not read a folder or a folder's time, or whose check or one of its writes fails, asks for the next location check, which publishes nothing. Then, only while autopublish is on and for a member's self-heal, it publishes those the member owns that need it, such as an artifact whose publish failed, under a turn end's limits: five publishes and a 60-second budget. Each replica keeps its own requests in memory, so a restart drops them. The heal sweep still checks the locations of every project with a message in the last 35 minutes (twice the interval and a five-minute margin), but a dropped publish waits for the project's next turn. Once every 15 minutes one replica claims the sweep (the _artifact_heal_sweep_started_at setting row, under an advisory lock) and runs the location check, one project at a time, on the shared root of every such project, which writes the root's held row and its folders' sort-time rows (above) and publishes nothing (cowork/services/artifact_heal_sweep.py). A project whose check falls short logs event artifact_heal_sweep_failed and stays on the sweeping replica's retries, in memory, never on its poll's self-heals, so a file store incident never starts every project's listing at once. That replica lists it again at its next pass at least 15 minutes later, whatever the project's message age: its next sweep, or, while other replicas claim the sweeps, a pass over its retries alone (event artifact_heal_sweep_retry), one project at a time, until a check reads every folder. Each pass logs one WARNING line, since every hosted deployment runs at LOG_LEVEL=WARNING: a sweep's (event artifact_heal_sweep) gives the count of projects it checked in full, the failed_count of those that fell short, and the retried_count of retries it listed beyond its window, and a pass over retries alone gives its count and failed_count. So a folder an old pod made during a rollout, or whose turn end lost its record to the shutdown drain or a full pool, shows in the list by the next sweep. A pod with no list asks for a self-heal only when its turn end's own check of the project's locations falls short: its listing could not read a folder, the project's artifacts folder or a folder's time (above), which asks for the location check alone, or its diff, or the record of what the diff listed, fails, which asks for the whole self-heal. Otherwise its turn end lists the project itself, checks every id there, and, while autopublish is on, publishes what needs it.
Routes that work on the file store run their work as one unit on the file store's threads (run_efs_unit): the project file, skill, memory and upload routes, a skill's sharing save, its caller's own switch and the read of its stored rule before the save, a model comparison's create, continue and delete, the deletes of a conversation, a turn, a file or an attachment, an attachment's move into a project, a task's move, and the shared-project reads. A remote turn's workspace staging, its skills block and each memory save its pod reports run there too, for a browser turn and a channel turn alike, and so does a browser turn's seed read, which reads the project's memory. An artifact's access routes read the keys of its rule from the file store through run_efs, then ask MindsHub for the rule on one of anyio's default threads. The Drive un-pick (DELETE /api/v1/connectors/connections/{engine}/{name}/picked-files/{file_id}) reads no database and runs its vault work through run_efs, and so does the OAuth callback (GET /api/v1/connectors/oauth/{service}/callback), which any caller reaches and which reads and clears its pending sign-in in COWORK_HOME. A conversation list or create, or a schedule create, that names no project by id, and a conversation update that names its project by name, may resolve the general project, which checks that project's folder, so they run on run_efs_unit too. A request that names its project by id, lists every project, or changes only a title needs only the database and runs as a run_db unit. In org tenancy the project list reads rows only once the organization has its general project: the organization's first list makes that project and its folder on run_efs_unit, and later lists don't check the folder, which a project file route recreates when it is missing. A new channel thread's binding may start in the general project, so it is made on run_efs_unit too; a known thread's binding is a run_db read. In org tenancy the sync routes left on anyio's default threads touch only the database; the others answer only the desktop (desktop-only or loopback-only routes). A unit opens its own session and closes it when the work returns. After its file store slot it takes one of the file units' own slots (EFS_UNIT_SLOTS, 16 by default) and then a database unit slot, as run_db does, by the same POOL_TIMEOUT deadline, so it is refused about POOL_TIMEOUT after it asked, with FilesystemBusy or DatabaseBusy. Such a unit holds its connection while the file store answers, so units stuck on a slow store hold at most EFS_UNIT_SLOTS connections, and database units keep the rest of the pool. Its thread waits inside a full pool only while a checkout outside the units, such as a sync route's request session, holds the pool's connections; the pool's own timeout then refuses it with DatabaseBusy, up to about two POOL_TIMEOUTs after it asked. On SQLite, the desktop's database, a file store unit takes no unit slot: that one slot orders the database units, and a long upload or delete holding it would refuse every database unit behind it. A 5xx that a unit raises as an HTTPException is logged with its cause (Session error: ... status=503 cause=...), as the request session logs one.
The files these routes serve go out through EfsFileResponse (cowork/api/v1/efs_file_response.py), which opens and stats the file in one call on the same threads before it sends a header, so a busy file store answers 503 there, and a file that is gone by then answers 404. A file or attachment download reads only the file's row before that, in a run_db unit, so it waits for a file store slot once. Project files and drafts stream from their pinned descriptor through PinnedStreamingResponse (cowork/api/v1/pinned_response.py), which reads each chunk on the same threads. A project file's open never waits on what sits at the path, so a FIFO an agent made there answers 404 at once, as anything but a regular file does, rather than holding its file store slot until the pod restarts. Once a response's headers are out, a refusal could only cut its body short, so each read waits for a slot as long as it takes. A client that leaves stops the reads: a send after a disconnect raises on an ASGI 2.4 server, and below 2.4, as uvicorn reports, the response listens for the disconnect and cancels its body. A response's close runs on the file store's threads even when a disconnect, a Stop or a shutdown cancels it (run_efs_cleanup). Starlette's FileResponse reads on anyio's default threads, so only desktop-only routes use it. The request's identity dependencies (get_principal, get_tenant_scope, get_scoped_session and get_session) are coroutines, so authenticating a request takes no thread.
Each turn restores the caller's vault secrets, which history is scrubbed against by exact value, through run_efs. A turn that gets no file store slot within POOL_TIMEOUT is refused, since its history would otherwise reach the model and the pod scrubbed by key shape alone: a web turn answers 503 with Retry-After, a channel turn fails, as it does when the database pool is full, and a scheduled run fails, leaving its cron slot due for the next poll. A vault that can't be read falls back to the shape-only scrubbing.
A daemon thread, cowork/common/loop_watchdog.py, watches for what slips through:
event_loop_stalled(ERROR) means the event loop didn't answer for 2 s. The line carries the loop thread's stack and the requests in flight.event_loop_recovered(WARNING) gives the stall's length once the loop answers again.limiter_saturated(WARNING) means anyio's default threads, therun_dbslots, therun_efsslots or the file units'EFS_UNIT_SLOTS(run_efs_unit) stayed full for 5 s.
Each boot logs startup_finished (WARNING) with the boot's length in duration_ms, counted from the server process's start, and each lifespan step's length in steps. A lifespan step still running 10 s after it starts (_SLOW_STEP_S in cowork/server.py) logs startup_step_running (WARNING) once, from a timer thread, with its name in step and its time so far in duration_ms. A step that took 10 s or more logs startup_step_slow (WARNING) as it ends, with the same fields. A step's timer stops when the step ends. A startup that fails stops the running step's timer when the task running the lifespan ends, so a step that raised is not later logged as still running. On a hosted deployment the liveness probe's delay has to cover that boot (deployment/cowork-server/README.md, "Probes").
Settings use a hybrid approach: user preferences and API keys are stored in the settings DB table (with Fernet encryption for secrets), while desktop connector credentials live in the filesystem vault (data-vault/). In MindsHub mode the OAuth routes proxy the caller's private connection in MindsHub's Data Vault instead.
The MindsHub credential is the exception, and it is stored nowhere. On the
desktop it is the user's own session token, which lives ten minutes, so the
Electron app hands it over at runtime through PUT /api/v1/runtime-credential/minds
and this process keeps it in memory. SettingService._raw_data overlays it onto
the stored rows, so every reader of get_user_settings() sees it without
knowing where it came from, and a value handed over beats any stored row.
The desktop app's own browser is handed over the same way, through
PUT /api/v1/runtime-credential/browser (The agent's browser).
Two properties follow. Nothing survives a restart, so the desktop app re-pushes on every start of this process. And a key the user supplied by hand travels the same way, which is what keeps a long-lived mdb_ key out of both .env and the settings table. The route is loopback-only and refuses in MindsHub mode: there the turn producer mints a per-turn credential and the pods are never handed one.
A live turn re-reads it per request, with one exception. A turn that copied the credential into its provider once would keep sending the token it started with across a hand-over, and take a gateway 401 that looks like a dead account. So build_llm_client passes an api_key_provider into MindsHub-backed providers, and each outbound model call reads the current in-memory value. Two limits are deliberate. Static MindsHub-mode and user-supplied keys keep their construction-time value, because nothing rotates them. And the scratchpad subprocess keeps the token it was started with for the rest of its turn: export_connection_info() hands it a string once and it has no supplier, so a pad-side model call still runs on that value. Refreshing it needs a pad IPC contract, which is out of scope here.
build_llm_client capability-gates the kwarg on inspect.signature, so an
older Anton keeps the static key and logs the degradation rather than failing
every turn.
All endpoints live under /api/v1/. Key resource groups:
| Path | Description |
|---|---|
/health |
Server status: versions, capabilities and configuration state, which the desktop app and the web client poll |
/health/live |
Liveness probe: answers whenever the event loop does, and reads no database, file store or settings |
/health/ready |
Readiness probe: the same, for the kubelet's readiness check |
/projects |
Project CRUD and working-folder management |
/conversations |
Conversation threads and message history |
/responses |
Streaming agent responses (SSE) |
/files |
OpenAI-compatible file uploads |
/schedules |
Recurring task scheduling; scripting it with an API key: docs/SCHEDULED_TASKS_API.md |
/skills |
Agent skill definitions |
/memory |
Persistent agent memory |
/artifacts |
Agent-produced file previews |
/publish |
Publish artifacts to MindsHub (view.mindshub.ai in prod) |
/connectors |
Third-party service connections and OAuth |
/settings |
User preferences and API keys |
/runtime-credential |
Desktop hand-overs, held in memory: the MindsHub credential, and the address and launch key of the app's own browser (write-only, loopback, outside MindsHub mode) |
/browse |
The agent's browser: whether this deployment offers it to the caller and, on the web, a warm-up, viewer links, the person's approval of a held step and signing out of every site. See The agent's browser |
/hub/workspaces |
Which MindsHub workspace this person is working in |
/hub/usage |
The caller's free Air allowance (as a proportion), balance, auto top up and credit spend, for the desktop's usage warnings. An organization with no free grant reads freeTokens.limit 0 and freeTokens.resetsAt null, since nothing refills |
Every route declares its permission at its own definition, and create_app() refuses to start if one does not. Adding a route means picking the class that names the credential the route actually takes:
| Declaration | The credential is |
|---|---|
OpenByDesign |
nothing at all. Pinned route by route, with a written reason, in tests/test_open_by_design_pin.py |
LoopbackOnly |
the caller being on this machine (require_local) |
DesktopOnly |
not MindsHub mode (require_local_tenancy 403s the route there) |
LoopbackDesktopOnly |
both of the above |
PlatformSignature |
the calling platform's HMAC over the body (channel webhooks) |
Authenticated |
a verified Principal |
AuthenticatedInOrgMode |
a verified Principal, in MindsHub mode only; a no-op on desktop |
AuthenticatedOrgAdmin |
... plus org-admin standing |
from cowork.api.v1.permissions import AuthenticatedInOrgMode, require
@router.get("/thing", dependencies=[Depends(require(AuthenticatedInOrgMode))])Declare it on the APIRouter when every route on it takes the same
credential, with one exception: never declare OpenByDesign on a router.
FastAPI adds a route-level dependencies=[...] to its router's rather than
replacing it, so a router carrying the open marker hands it to every route
added later, and the walker cannot tell that apart from a deliberate choice.
Every other class refuses, so inheriting one is safe.
COWORK_TENANCY_MODE=org python -m scripts.dump_routes prints the current
surface with each route's declaration.
The agent drives a browser the person can watch beside the chat and take over. Which browser depends on the deployment, and this server holds no browser credential of its own (cowork/services/browser.py).
- On the web (MindsHub mode) the browser is the browser router at
COWORK_BROWSER_ROUTER_URL, which starts one browser per person. Each/browseroute calls it as the caller, with their own MindsHub credential. Before the producer queues any web turn, it scopes the turn's key with the router (POST {router}/_scope): to the task's session and the project's saved tabs, which also starts the person's browser, or, for a turn with no browser, to no session ({"session_id": null}), so code in the turn that holds the key can't scope it to a session of its own. The router refuses a key it hasn't scoped, and takes a scope only withX-Browser-Scope-Key, read on every scope from the fileCOWORK_BROWSER_SCOPE_KEY_FILEnames, where the chart mounts the router's own Secret,browser-proxy, keyscope_key. The job'sparams.browseris{base_url, profile, project?}and carries no credential: the pod reaches the router with the turn key. A turn whose scope fails runs without a browser and logsbrowser_scope_failedwith the router's code and the wait inelapsed_ms; a person the browser isn't for (their organization switched it off, or their role lacks it) logsbrowser_scope_not_entitledat info instead. No alert reads either yet. - On the desktop the browser runs inside the Cowork app. The app hands over its loopback address and per-launch key through
PUT /api/v1/runtime-credential/browser, with the body{"url": "http://127.0.0.1:<port>", "token": "<43 base64url characters>"}, or both blank to clear it. This process keeps them in memory and hands an in-process turn's anton that browser. The route is loopback-only and refuses MindsHub mode, and a deployment that trusts identity headers ignores the hand-over, so one person's browser never answers another person's turn. - No browser for a channel turn, which comes from a chat others can write to while the browser holds one person's sign-ins; for a person who turned
browser_enabledoff; for a task whose Browser switch is off ("browser": "off"on/responses); and on a MindsHub-signed-in desktop whose organization's admin turned the browser off. That desktop reads the switch from its MindsHub entitlements before each turn and keeps the last answer when MindsHub can't answer; a desktop on the person's own model key asks nothing.
GET /browse/status answers {enabled, available, status}. enabled is the person's switch. available says whether the deployment offers them a browser, whatever the switch says, so they can turn it back on. status is one of these:
none,startingorrunning: the router's answer for the caller's own browser on the web, orrunningfor a registered browser on the desktop;unavailable: no router on the web, or no registered browser on the desktop;organization_off: an admin of the caller's organization turned the browser off, on the web or on a signed-in desktop;not_entitled: thebrowserblock of the caller's MindsHub entitlements (GET /v1/entitlements/me/, read through the same cache as their permissions) says their account may not use it. That read fails closed: when MindsHub can't say, status answers 503mindshub_unreachable.
POST /browse/provision warms the caller's browser with their time zone and language, and answers like status. POST /browse/embed takes {session_id} and answers a single-use viewer link, {session_id, view_url, ticket_expires_at, lease_expires_at}, which is never stored. POST /browse/approve takes {session_id, pending_id} from the commit-hold card and records the person's approval in the browser, so the agent's one resend of that step runs; it doesn't answer the card, which the app answers after a 200. POST /browse/decline takes the same body when the person skips that card, and the browser drops the step and any approval already recorded for it, so a late approval can't let the resend run. DELETE /browse/logins signs the person out of every site in their browser. These five are web-only and answer 409 elsewhere. A refusal from the router keeps its 401, 403 or 429 and its code in detail; any other router failure is 424 with the router's code, since the ingress swaps a 5xx body for its own page.
The response.browser_session_opened event carries the session id only. Whoever opens the pane mints their own viewer link, so a stored event opens nothing.
On the web, sites see the browser's traffic come from MindsDB's own addresses; the desktop app's browser goes out on the person's own connection.
tests/fixtures/browser_router/ pins what this server sends the router and reads back, and tests/fixtures/browser_contract/ holds anton's fixtures for the turn block and the session event.
/api/v1/hub/workspaces backs the workspace selector at the bottom of Cowork's sidebar, which Cowork draws only when the listing carries two or more workspaces: one workspace means nowhere to move to. A MindsHub Workspace is an org-internal container that owns MindsHub resources (API keys, artifacts, model entitlements) and lives in MindsHub. It has nothing to do with the filesystem directories this repo calls workspaces, which is why the stored key is hub_workspace_id.
The selector has no feature gate, and its listing reads no entitlements: it shows whenever the listing carries two or more workspaces.
Four things about it are worth knowing before changing it.
The sidecar makes the call, not the renderer. MindsHub's ingress allows three console origins per environment and no Cowork host, and a per-PR Cowork host cannot be added to a static allow-list because its name carries the PR number. The packaged Electron app would get away with a direct call because it runs with webSecurity: false; the web SPA would not. Reading here works for both shells. The outbound host is derived by default_minds_auth_host() from ENV, which the desktop propagates when it spawns this process, and the credential is the caller's own, never the stored provider key and never minds_url, both of which an org admin can set.
The credential arrives in its own header, X-MindsHub-Authorization. It cannot use Authorization: Electron's main process overwrites that on every request to the loopback server with the server's own token, so the caller's Keycloak JWT can never arrive under that name in the desktop shell. Authorization is still the fallback in MindsHub mode only, where the ingress put the caller's JWT there; on a desktop install that header holds this server's own bearer, and forwarding it to MindsHub would leak the one credential the main process scopes to the loopback origin. mindshub_credential reads both, and it is deliberately a different function from caller_bearer so a client cannot steer the credential on the org model-catalog fetch by setting a header.
Skill and artifact sharing never use mindshub_credential, for the same reason: a caller can put a credential from another organization of theirs in that header, and MindsHub would answer with that organization's permissions, teams and members. Sharing reads all three with caller_bearer. Three routes still take mindshub_credential: the workspace selector's GET /hub/workspaces/ and PUT /hub/workspaces/active, and GET /hub/usage/. PUT /active (set_active_mindshub_workspace in cowork/api/v1/endpoints/mindshub_workspaces.py) answers 403 for a workspace missing from the listing that credential returns, so the listing decides what it stores. The pick it stores grants nothing: the menu shows it only while the listing has it (resolve_active in cowork/services/mindshub_workspaces.py), and every key mint sends it for the gateway's user and organization, where MindsHub refuses a workspace they hold no grant on and the mint falls back to the default workspace and clears the pick (_request_turn_key in cowork/turnqueue/auth_keys.py). Every MindsHub read shares one transport and one per-caller cache (cowork/services/mindshub_client.py).
The listing cache is keyed on the credential, not just the caller. MindsHub answers the listing per caller: an owner or admin sees every workspace in the organization, a member only the ones they hold a grant on. So a cache keyed on the organization alone would serve one admin's menu to every member for the whole TTL, and the grant check on PUT /active reads the same entry. The key is (MindsHub host, organization, user, credential digest) (CallerCacheKey in cowork/services/mindshub_client.py). The digest is not belt-and-braces: user_id comes from the gateway-set principal and is None on every desktop request, because scope_from_principal returns LOCAL_SCOPE outside MindsHub mode, so identity alone collapses to one shared entry and a sign-out/sign-in as another account would be served the previous one's workspaces. A new session means a new token means a new entry. Entries are swept on write, since nothing re-reads a departed caller's key and the dict would otherwise grow for the process lifetime. The TTL follows whether MindsHub answered, not what it said: a reachable listing keeps the long TTL, and an unreachable one the short failure budget, so a degraded MindsHub does not add a round trip to every menu open without also going stale for a long stretch.
Two refusals on PUT /active, and neither may read the stored pick. A
workspace missing from the caller's listing is a 403; one in the listing but
stamped archived is a 409, so the client can say retrying will not help instead
of offering a loop with no exit. The archived check reads the target row's own
archived_at rather than asking whether it is in the set the menu offered, and
that distinction is load-bearing. hub_workspace_id is an untagged
UserSettings field, so PUT /api/v1/settings/hub_workspace_id writes it with
no gate and no listing check, and selectable keeps the active row even when
archived. A refusal phrased against the offered set would therefore have been
talked into accepting an archived workspace by one call to the settings route.
Nothing that refuses a request may read a value any caller can write.
Picking a workspace changes what the client shows, not what a turn is billed
to. Neither turn credential carries a workspace: a desktop turn presents the
user's own session credential, whose organization comes from the token's
active-organization claim, and a cloud turn presents a minted key whose request
body has no workspace field. So nothing on the turn path reads
hub_workspace_id, and a test asserts that.
The pick is stored as an untagged UserSettings field, so it lands per (org, user) in MindsHub mode and in the single global row on a desktop install. That is interim: the shared per-user preference the console reads has no route in MindsHub yet, and moving this onto it is follow-up work.
Single-user mode has one person, so none of this applies: every check below is inert on the desktop.
In MindsHub mode a request acts as a pair, an organization and a user, and it never carries less. The gateway authenticates the credential, asks MindsHub whether that user is still a member of that organization, and injects X-User-Id and X-Organization-Id. cowork-server validates the shape of those headers and then trusts them, so the gateway being the only route to the pod is what makes them trustworthy, and that is a NetworkPolicy rather than anything in this codebase. deployment/cowork-server/templates/network-policy.yaml is that policy, on in staging and prod, and it admits only the nginx ingress controller pods in the infrastructure namespace on port 9010. It is off in PR environments, which take base values, so a PR environment does not enforce this boundary and a forged-header caller inside one is served. A request with no valid pair is answered 401 before any route runs, except on /api/v1/health/, /api/v1/health/live and /api/v1/health/ready, which clients and the kubelet probe with no headers, and the channel webhook paths, which third parties call.
The project, skill, artifact and conversation lists run the same number of database statements whatever the organization's size. The rows a statement reads still grow with the organization. Tests count the statements at two sizes: tests/test_org_list_scaling.py for the project, artifact and conversation lists, and test_the_skill_list_runs_the_same_statements_at_3_and_30_skills in tests/test_org_skill_sharing.py for the skill list.
GET /api/v1/projects/andGET /api/v1/skills/read every attribution row of their kind in one query (SharedResourceAccess.snapshots), after they read the projects or skills. A create or a skill upload commits its claim row (SharedResourceAccess.reserve_claim) before the project row or the skill's folder, so every listed item's claim is in that read. A skill with no sharing row takes its owner from the same attribution rows, read once per list (SkillService.access_rule).- The lists answer from committed state. While a create, upload or rename is still in flight, its item lists as pending, or under its new name with no attribution, until the change commits.
- A create that crashed leaves its claim pending. Once the claim's 5-minute lease runs out, the next list recovers it: it finalizes the item for its creator, or releases the name when nothing was written. Only an item whose claim has expired, or has no expiry, takes the locks. A skill takes the organization's project-references lock and then the skill's own lock, the order every skill writer that takes both uses. A list with no such claim takes no lock.
- Every token check under a lock reads the row as the database holds it now (
SharedResourceAccess._find_for_updatere-reads withpopulate_existing), and so does recovery. A request's session keeps the rows it read past a commit, and another replica may have recovered the same claim meanwhile. A decision from the session's earlier copy would finalize the item a second time and reset itsupdatedBy, or a skill's sharing. GET /api/v1/artifacts/reads the caller's own conversation ids once (ConversationService.owned_ids). It then filters each project's legacy per-conversation workspaces by name before it stats any of them, so a co-member's workspace is never opened, and it skips a workspace with no.anton/artifactsfolder. Project-root owners come from one query and legacy workspaces' owners from one more (resolve_artifact_owners_by_source). In org tenancy, once the held-root backfill has finished, it opens only the roots theheld_artifact_rootsrows name (see Work on the shared file store), so it stats no project or workspace that never held an artifact. Each root is opened from the trusted root, one folder at a time without following links. Once the sort-time backfill has finished too, it opens only the newest folders theartifact_sort_timesrows name, so its file store work stays the same whatever the number of artifacts in its roots. It reads those rows in one query, or two when its limit cuts a run of rows that share one time. When the rows were cut and the folders it keeps build fewer than 80 cards, it reads every folder of its roots instead (see Work on the shared file store). A project that either backfill's sentinel lists as unread costs a full read: the list reads every folder of its roots, and finds the roots themselves on the file store when the held-root sentinel lists it. Before then, and in those projects, inside a root it reads every folder'smetadata.jsontime to sort by it, with one no-follow open and one stat per folder (dir_child_stat). The same tests count its file store calls at 10 and 1,000 projects, at 10 and 1,000 conversation workspaces, and at 80 and 1,000 artifacts in one root, those that share one time included.GET /api/v1/conversations/reads the page's projects in one query and each row's last activity, the time of its newest message, in another (ConversationService.last_messages_at). That read returns one row per conversation, though a turn's tool rows and its answer share onecreated_at.
Canonical Cowork web sends X-Cowork-Expected-Organization-Id with every authenticated browser API request. TrustedHeaderMiddleware._organization_boundary_response compares it with the normalized X-Organization-Id the gateway supplies before the route runs. It applies only to Keycloak-shaped bearer JWTs. MindsDB API keys, opaque service credentials, requests without a bearer, CORS preflights, health checks, and channel webhooks keep their existing behavior.
The boundary always refuses. A missing header returns 426 and a malformed or
changed value returns 409. Both responses carry
X-Cowork-Organization-Reload: required, a JSON code and detail, and
Cache-Control: no-store. The browser reloads instead of letting an old
document continue under a new Keycloak organization. A request already inside a
route keeps the Principal created at its start, so a concurrent session change
cannot retarget that in-flight operation.
There is no setting that relaxes this. The boundary once had an audit position
that logged a violation and served the request anyway; it and the staged rollout
it existed for are gone. An environment that still carries the retired
COWORK_ORGANIZATION_BOUNDARY_MODE key boots normally and ignores it. Deleting
the field is what makes it unreadable: AppSettings sets no env_prefix and
gives every field an explicit validation_alias, so the environment source only
looks up names a field still claims. extra="ignore" only decides what happens
to an unknown key arriving in a .env file, which is not how the overlays
deliver this one. Both halves are pinned, the settings half by
tests/test_app_settings.py::test_stale_organization_boundary_mode_env_var_is_inert
and the request half by
tests/test_principal.py::test_no_environment_variable_reopens_the_organization_boundary.
GET /api/v1/capabilities/organization-switch is authenticated and returns protocol version 1. It reports expectedOrganizationEnforced: true in MindsHub mode with identity enforcement active. Identity enforcement is in that answer because the picker should stay hidden while identity is only audited, not because it gates the boundary: TrustedHeaderMiddleware runs the boundary for every browser JWT request it builds a Principal for, in MindsHub mode alone. It reports enabled: true when those hold and COWORK_ORGANIZATION_SWITCH_ENABLED=true.
COWORK_ORGANIZATION_SWITCH_ENABLED is the product enable for the picker, not a
safety switch, and it is the lever to reach for to hide the picker without a
code change. COWORK_IDENTITY_ENFORCE=audit hides it too, by dropping
expectedOrganizationEnforced, but it reopens the no-principal path and leaves
the boundary refusing anyway.
Backing enforcement out needs the code and values from before cowork-server#524. deployment/cowork-server/README.md records the procedure and why a Helm rollback cannot do it.
Inside one organization, two different rules apply, and which one you get depends on the resource:
- Shared with the organization: projects, project files at the project root, skills, and project memory. Every member reads them.
- Private to whoever created it: conversations and their history, scheduled tasks, personal memory, uploaded files, and everything under a conversation's own workspace at
conversations/<conversation_id>/. Live artifacts are in that last group, because the agent writes them into the conversation it is running in. Cloud OAuth connections are also user-private, held in MindsHub's Data Vault rather than here. cowork-server stores none of it and relays the caller's own credential to MindsHub for the whole lifecycle: the connect handshake and its status poll, the connection catalog, one connection's detail, the Google Picker file grant, a short-lived picker access token, and disconnect. MindsHub resolves the caller from that credential, so each route only ever reaches that user's own connections.
Shared visibility does not imply shared destructive access. Project creators and organization admins can rename or delete a project; the General project is immutable. Any member can create a skill, while only its creator or an admin can edit, disable, or delete it. Packaged skills are immutable in MindsHub mode for every role (desktop keeps its existing editable-copy behavior). The first member to write a non-empty project-memory slot becomes its author, and only that author or an admin can subsequently change it. Project instructions follow the project creator/admin rule. Empty memory writes do not claim a slot, and ordinary project files remain member-writeable.
The API derives these decisions from the trusted principal and returns typed
attribution and capabilities objects for the client. Stable user IDs drive
authorization, and they are the only identity these rows keep: no email address
is stored, and the API fills one in only when the actor is the viewer
themselves, so an attribution never discloses another member's address.
File-backed resource ownership lives in shared_resource_attributions, outside
the agent-writeable project tree, and every allowed protected mutation appends a
shared_resource_mutations row. A delete removes the current attribution but
keeps its mutation event. Unattributed legacy resources fail closed to
admin-only mutation.
The private rule is enforced by the service layer rather than by the routes:
ConversationService._owned, FileService._owned_select and
ScheduleService._owned_select each add created_by == <the caller> when a
request is org-scoped. Two places extend the same rule to the filesystem.
_conversation_workspace_ok in the project-file routes refuses a path under
another member's conversation directory, and artifact_roots drops another
member's conversation directories before the artifact list or delete ever sees
them, because those routes are addressed by project and slug and never receive a
conversation id.
One artifact at a time can leave the private group, and only its owner can put it there. POST /artifacts/workspace/{project}/{artifact}/comments-access records a grant in that artifact's .revisions/draft-review.json and mints the matching rule in MindsHub. A co-member then resolves that one artifact by id (artifact_scope.review_artifact_for_request searches other members' workspaces only after the caller's own, and only accepts a folder that carries the grant) and gets the draft preview and comments, never the source, the edit routes or the delete. Without a grant the answer is 404, so a private draft still cannot be told apart from one that does not exist; with a grant the owner-only routes answer 403 instead, because to a client already looking at the draft a 404 would read as deleted. The artifacts list is unaffected either way: a co-member's artifact never appears in it, so review starts from the link the owner shares.
Outside MindsHub mode the route provisions nothing and answers enabled with "scope": "owner". Its comment routes keep each artifact's threads in that artifact's .revisions/comments.json (services/local_artifact_comments.py), and find the artifact only among the caller's own artifacts roots. In single-tenant mode, while projects are personal, those roots are the person's own partition, so another person's calls to the draft answer 404.
An artifact's journal, .revisions/, holds its revisions, a pending source write, agent repairs, the local comments, the draft-review grant and the lock. The agent's code can write it, so every read and write goes through services/artifact_journal.py: the artifact folder and every folder below it open without following a link (the folders above it were opened the same way when the artifact was found), a file the server reads must be a regular file with one name and under a size cap, and a .revisions, blobs/ or repairs/ that isn't a real folder reads as an empty journal until the first write replaces that entry, never its target, with a real folder. An open that fails for any other reason, such as running out of file descriptors, is an error, never an empty journal, so no write replaces a history it couldn't read. The turn-end capture and the local comment routes run off the event loop, so a journal read or a wait on an artifact's lock holds one worker thread, not every request on the server. Each record the agent's code can write (a manifest entry, a pending source write and a repair record) is checked by its model in services/artifact_revision_records.py before the server uses a field of it, and again where a field names a file. A record that holds a value JSON can't encode, such as one nested too deeply or a lone surrogate, fails its model too. A revision's text is read only as blobs/<sha256> with bytes that match the name. A manifest entry that names any other path, or otherwise fails its model, is left out of the history and answers 404 Revision not found, as a missing revision does, and the next write of the manifest drops it. A repair record that fails its model answers as a missing repair and is never written back, nor is one under any name but its own UUID. A pending source write that fails its model is abandoned. A record keeps any key its model doesn't declare and is written back with the keys it had, so a journal reads and writes the same across builds. Every journal file, and each rewrite of an artifact's metadata.json by the identity and export services, is replaced through dir_atomic_write (common/paths.py): the new bytes go to a file beside it, which is synced and renamed over the name, and then the folder is synced.
MindsHub-mode artifact authorization uses a separate server-owned identity. An agent can edit metadata.json, so its local UUID cannot reserve a global comment key. The artifact_identities SQL table binds each (organization, local UUID) to its immutable owner and a MindsHub-issued canonical key. Draft grants, publishing, and policy deletion use that key; the comments REST/SSE proxy translates the unchanged local UI key through the same table. Read requests never allocate. Existing identities are adopted only when MindsHub confirms a matching durable owner and organization binding. New issuance uses a persisted random request ID so concurrent calls and lost replies cannot allocate different identities.
This requires MindsHub's internal artifact-access/claim/ and allocate/ endpoints and the matching services publisher ownership checks. Apply MindsHub's ownership history migration before enabling new policy writes, then deploy these clients with the coordinated MindsHub changes. Authorization failures preserve files and stop sharing. SQL aliases survive artifact deletion and schema downgrades; they are ownership history, not a cache that can be cleared during rollback.
Filesystem authorization decisions start with a resolved path. The route carries
that decision through opening the file: every
component below the project directory is opened O_NOFOLLOW, and a symlink
planted anywhere in the chain is refused. A pod mounts its own workspace
read-write, so without that a swapped directory component between the check and
the open reaches another member's tree, or another organization's.
A refusal on a private resource answers 404, not 403, with the same body a
genuine miss returns. Telling the two apart would confirm that another member's
file exists, which is most of what an attacker wants to know. Policy refusals
that reveal nothing personal answer 403 instead: the desktop-only routes say
not available in org deployments, and an organization-settings write without
the admin role says so plainly.
The HTML preview hands out a bearer token in a URL (preview-mount-file
returns one, preview-asset spends it), because an iframe cannot send an
Authorization header. The token is random, it is bound to the member and
organization that minted it, and it expires after 30 minutes, so a token that
escapes into a log or a screenshot is not a way in for anyone else.
Being the minter is not enough on its own, because a mount grants a
directory. The gate runs on the file the caller named, and an .html at the
project root sits in a directory every member's conversations/<id>/ hangs off,
so a token minted on a shared file would otherwise read every workspace under it.
A mount therefore reaches only the workspace its own file lived in, and a mount
taken on a shared file reaches no workspace at all, not even the minter's own.
That check holds no session, deliberately: preview-asset serves every sub-asset
a page pulls, and a session there is a database connection per image.
Every minds-cloud role defaults to mindshub_air, for all three roles: planning,
coding and router. MindsHub's catalog declares it, in the
mindshub_model_policy_v1 config that already owns the alias registry, and the
declaration arrives as a default_for list on each /v1/models row. So moving a
default is a config edit plus an apply, not a release. MindsHub Air's usage draws the included allowance, so a user who has
picked no model can finish a whole turn without the wallet being charged for any
part of it.
The two roles a user never sees are why this is the default rather than a premium model. Planning is the model in the picker, so a wrong choice there is visible and fixable. Coding (the completion verifier and the scratchpad) and router (respond-versus-delegate gating and history summarization) run unseen, so a paid default there is denied on an empty wallet with nothing on screen to explain why.
An explicitly stored model is never rewritten by this. Paying for a better model is a pick in the Settings picker, and a funded wallet resolves to the same default as an empty one until that pick is made.
/v1/models marks a model the org cannot currently pay for as enabled: false,
and the map is cached as minds_model_enabled. When a stored pin is flagged
that way, _resolved_model resolves the role to the first affordable model in
the map instead, for all three roles. The alternative is every turn failing on a
denial the user may not be able to see.
The stored row is left exactly as the user set it, which is the load-bearing
half: the moment the wallet can pay again, the next settings load flips the
alias back to enabled: true and the role resolves to the original pick with
nothing to re-select.
Two cases share that path and should not be confused. A pin the map flags
false is a real MindsHub model that is merely unaffordable right now. A pin
absent from a non-empty map is foreign or retired, so it would 404 on every
turn rather than 402, and it is healed the same way with no route back.
enabled: false has one more cause: an admin in the organization restricted the model. /v1/models names the lock on each disabled row in disabled_reason (model_restricted, wallet_empty, included_allowance_exhausted or budget_exceeded), and GET /settings/recommended-models relays it as modelDisabledReasons, so the picker shows a restricted model as restricted and offers no credits for it. The map comes from the MindsHub listing only. A custom openai-compatible endpoint's disabled_reason is dropped, because a BYO endpoint must not be able to claim an admin restricted a model. Resolution treats a restricted pin like any other false one and swaps it, for the members the rule restricts and no one else. A turn that still reaches the gateway on a restricted model fails with the model_restricted code: the gateway keeps the 403's permission_denied reason and names the rule on its X-MindsHub-Deny-Detail header and in the body's error.deny_detail, trusted by the same origin rules as the reason.
A restriction belongs to one member, so it is stored per member. MindsHub applies the model rules for each caller's org, workspace and team, so two members of one org get different listings, and fetch_org_model_catalog caches one listing per org and member. minds_model_enabled is an org row that every member's resolution reads. So in MindsHub mode recommended_models leaves the caller's model_restricted models out of it: each keeps the flag the org map already held for it, or true when it held none. The caller's restricted ids go to their own minds_model_restricted row instead, an untagged setting that SettingService files at the caller's user scope, and UserSettings._minds_enabled_map reads them as false over the org map. One member's restriction therefore never swaps another member off a model, and another member's load never lifts it. A member's restrictions reach resolution when that member's own settings load writes them. A listing that publishes no enabled flags leaves the stored list alone, the same evidence rule the org map follows. A desktop install has one member, so it keeps restrictions in minds_model_enabled.
The listing cache (_minds_models_cache in cowork/services/providers.py) drops expired entries on every write and holds at most _MINDS_MODELS_CACHE_MAX listings, evicting the least recently used, so keeping one entry per member does not grow it for the life of the pod. An entry with no organization in its key, which includes every desktop fetch, is keyed by the credential it was fetched with, held as a pydantic SecretStr so it prints as asterisks in any repr or log, so an account switch in the running app is never served the previous account's restricted or locked rows. Nothing is hashed from it.
The desktop closes the loop at the other end: a model the map locks is not offered in either picker, so a swap only ever applies to a pin that was affordable when it was made. Allowing the pick meant the turn silently ran a different model from the one the picker named.
MODEL_ROLE_DEFAULTS in cowork/common/settings/app_settings.py is still a real
answer, not a legacy layer. Resolution is synchronous and does no network call in
the turn path, so the catalog's declaration has to be persisted before it can be
read, exactly as the availability map is.
| State | What resolves | When you are in it |
|---|---|---|
| A model is stored for the role | that model | the user picked one, or saved Settings once |
minds_role_defaults names the role |
the alias the catalog declares | any install that has loaded Settings since the catalog declared it |
| Nothing persisted | MODEL_ROLE_DEFAULTS |
a fresh install sending its first message, and any install that has never reached the catalog |
The availability map still overrides the answer in every case where the wallet cannot pay for it, so a declared default is not a grant: it says where to start, never what may be called.
GET /settings/recommended-models writes the map, from the same fetch that
already refreshes minds_model_enabled, and it builds the recommendedPair it
serves the picker through minds_role_start_models, the same two steps resolution
takes: the declared default replaces the compiled one, then availability adjusts
it. Those two have to agree, because the picker shows the pair as the model each
role starts on and the desktop writes it back as an explicit pin when a save
repoints a role onto MindsHub. A pair built from the compiled table, or from a
declared default the wallet cannot pay for, would pin a model that turns never
run. The pair is rebuilt from whichever map resolution will read: the live one
when the gateway published defaults, the cached one when it did not.
Why a probe sends a model at all: MindsHub bills per model, so a model the wallet
cannot pay for is denied, and that denial is indistinguishable from a bad key.
Probing a paid model tells an account with an empty wallet that its working key is
invalid. MINDS_PROBE_MODEL (mindshub_air) draws the included allowance
instead of the wallet, so the result reports reachability and key validity, which
is what these endpoints are for.
Two endpoints, and they do not behave identically.
POST /settings/validate-provider (onboarding, and the only caller is the
onboarding screen) probes a chat completion on every branch, and takes an optional
model:
provider: "minds"always sendsMINDS_PROBE_MODELand ignoresmodel.provider: "openai-compatible"sendsmodelas asked, so validating one specific model never reports a pass earned by a different one. Omit it against a MindsHub base URL and it falls back toMINDS_PROBE_MODEL; omit it against any other host and the generic openai-compatible default applies.provider: "anthropic"sendsmodelorclaude-sonnet-4-6.
POST /settings/test-providers (the Settings health dot) probes per provider
type, and only the minds-cloud type is a chat completion, on
MINDS_PROBE_MODEL. The openai-compatible type is a GET {baseUrl}/models
listing probe, so a MindsHub host configured through that card is health-checked
against a route MindsHub does not deploy everywhere; those routes answer 404 or 401
even for a valid key, which is the reason the minds-cloud type does not use one.
An apiKey of "" or "***" on that route means "use the stored one", which is
how the Settings UI re-tests a provider it only ever received masked. A stored key
only goes to a host this deployment saved: the provider cards in providers_json,
the scalar openai_base_url/minds_url, and the vendor host an omitted mindsUrl
already reaches. The comparison is on origin (scheme://host[:port]), not the whole
URL, because the question is which party receives the key; that also means a stored
URL carrying a path still matches a body sending the host alone. A URL the guard
cannot parse names no origin, so it is refused rather than raising. Omitting the URL
stays legal: a minds-cloud probe then goes to the vendor host, and an
openai-compatible one answers missing base URL, which is what it did before this
guard. A key the caller supplied may go anywhere, since nothing stored is at risk.
A refused provider comes back fail with its reason and is never pinged, rather
than failing the whole request. Callers send every configured provider in one call,
so refusing the request would blank the other providers' dots over one bad URL.
Each card in the body is a ProviderProbeCard (cowork/schemas/settings.py), which declares the four fields the ping reads, type, apiKey, baseUrl and mindsUrl, as strings. A card whose type or URL is not a string fails the whole request with a 422 before any key is resolved. The Settings UI does not hit this: it sends the cards it saved itself, with type normalized to a string by backfillProviders in cowork's settingsTransform.js. Every other field the UI holds on a card, such as isDefault or name, is ignored, so a renderer that adds one keeps getting its status dots.
When the MindsHub gateway refuses a minds-cloud probe with a named reason, the response carries it in providerStatusReasons, keyed by provider type: {"minds-cloud": {"code": "free_air_daily_spend_fuse_exceeded", "resetAt": "<ISO instant or null>"}}. The code is the gateway's own X-MindsHub-Reason word, one of wallet_empty, included_allowance_exhausted, free_air_daily_spend_fuse_exceeded, budget_exceeded, rate_limited and policy_unavailable, so the Settings notice can tell a velocity limit or the free-Air fuse from an empty wallet instead of matching on HTTP 429 in the detail string. When a proxy strips that header, the probe reads the same word from the body code, which the gateway sets to match. resetAt carries the gate's X-MindsHub-Reset-At, which it sends on the allowance and fuse denials. The body code counts only when the probe's response came from the configured minds_url host. The header counts unless the response provably came from another host. These are the origin rules the turn-failure mapping applies (gateway_denial in cowork/handlers/turn_errors.py), so a probe aimed at any other host names no reason. A refused provider was never pinged, so it has no entry. providerStatus and providerStatusDetails keep their shape.
Every MindsHub-bound chat probe caps the completion at max_tokens: 20, not 1:
some models refuse a 1-token budget and fail the probe for a perfectly good key
(see _chat_probe). The cap is not sent to a non-MindsHub endpoint, because
OpenAI's reasoning models reject max_tokens and want max_completion_tokens.
The desktop app has a second copy of these validators in its Electron main process
(cowork/src/main/provider-validation.ts, called from the settings:validate IPC
handler in cowork/src/main/index.ts); the endpoints here serve the web build.
Both copies have to change together. One asymmetry worth knowing: the desktop
MindsHub onboarding path signs in through Keycloak rather than validating a pasted
key, so main's validateMinds has no live caller today, and it is the
openai-compatible and anthropic validators there that a packaged build actually
runs.
Hosted turn admission checks product.execute through MindsHub's internal POST /internal/permissions/authorize/ endpoint. This applies to free and paid models, stored provider credentials, direct responses, remote queue submission and scheduled runs. Schedules resolve the acting identity from the stored owner and check it before creating a conversation. Queue submission rechecks current access even when reusing a previously minted credential.
The request contains the server-resolved user and organization IDs and one permission. It uses the existing COWORK_TURN_AUTH_INTERNAL_BASE_URL and COWORK_TURN_AUTH_INTERNAL_SECRET; customer headers cannot choose that host or credential. A confirmed denial returns 403 with permission_denied. Missing configuration, malformed replies, transport errors and service authentication failures return 503 with permission_unavailable. Authorization failures never fall through to another model or delegated execution. Single-user mode keeps its one-person behavior.
Artifact source edits, publishing/access changes, deletion, revision restoration
and repair management require artifact.manage as well as existing artifact
ownership. Starting an agent repair also requires product.execute. Capability
responses reflect these current grants. Generic project-file writes and deletes
apply the artifact grant to artifact storage paths too. Writes atomically replace
the selected directory entry, preserving the existing file mode where descriptor
chmod is available. This detaches a pre-existing hardlink so editing an ordinary
file cannot modify protected artifact bytes through the same inode. Failed writes
leave the original file intact and remove the temporary file.
Remote turns use the anton_turn_v2 controller operation and declare their
workspace authority. The controller checks artifact.manage again when it
dequeues the turn. Without that grant, it mounts saved conversation files read
only and runs the agent in a temporary copy. All workspace edits in that turn
are temporary, including ordinary files; nothing is copied back or published.
The controller replaces warm workers when their storage authority is unsuitable.
Cowork requires the controller's verified workspace acknowledgement before
accepting worker output, and indexes artifacts only for persistent workspaces.
Deploy the companion scratchpad-controller change before enabling this producer;
older controllers fail these turns with permission_unavailable.
Publishing uses a separate artifact_publish mint purpose. It requires artifact management without granting model execution. Execution mints retain their own purpose and credential type; the publisher uses a fresh instance ID and never hands its credential to an inference client. Deploy MindsHub's decision/purpose APIs and the publisher's artifact-only authentication path before this server change. Keep these admission checks in place during rollback while restrictive custom roles remain assigned. No new customer or staff permission grants are introduced.
COWORK_TURN_DATASOURCE_ENABLED (default false) lets a hosted turn query the user's verified PostgreSQL and MySQL connections in the organization. When it is on, the producer lists them from MindsHub under the dedicated producer service role (COWORK_TURN_DATASOURCE_PRODUCER_KEY_ID and COWORK_TURN_DATASOURCE_PRODUCER_KEY), bound to the turn key it has just minted, registers one grant per connection before enqueueing, and puts connection ids and credential versions on the queue, plus each connection's name and connector id when COWORK_TURN_DATASOURCE_LABELS_ENABLED is on. No capability or password crosses Redis; the pod reaches the gateway on the inference host the turn's llm block already names. A listing or registration failure fails the dispatch as permission_unavailable instead of silently dropping the datasources.
Turn the flag on only after MindsHub serves the datasource producer and resolver endpoints, the inference deployment serves /v1/datasources/, the turn controller passes the datasource block on, and the scratchpad image includes the typed helper that reads the gateway from the turn's inference host, with live pods from an older image recycled first. With the flag off nothing in this path runs, so the flag is also the rollback.
COWORK_TURN_DATASOURCE_LABELS_ENABLED (default false in the server, "true" in the hosted chart) adds each connection's name and connector id to its reference, so the agent can tell the connections apart and match the name the user uses. A name or connector id the queue would refuse is left out and anton cleans the rest for the prompt, so neither fails the turn. It acts only where COWORK_TURN_DATASOURCE_ENABLED is on. An anton older than this change refuses a block that carries them, so an environment turns grants on only after its scratchpad image has that anton and no live pod started before the image update; turning labels off is the rollback.
On a local deployment with trusted identity headers and personal projects, an operator can grant read-only file access to one project. Keep projects in COWORK_PERSONAL_RESOURCES. The owner alone can execute tasks, schedule work, change files or edit instructions. Recipient visibility is confined to GET /api/v1/projects/ descriptors and the UUID-addressed GET /api/v1/shared-projects/{project_id}/files, /files/{path} and /files-raw/{path} routes. Ordinary project services, execution and mutation lookups remain scoped to the requester's own partition.
Run these as Cowork's user, with the owner's identity pair:
python -m cowork.ops.projects share --organization-id <deployment-id> --user-id <owner-sub> --project <slug> --recipient <recipient-sub>
python -m cowork.ops.projects share --organization-id <deployment-id> --user-id <owner-sub> --project <slug> --everyone
python -m cowork.ops.projects unshare --organization-id <deployment-id> --user-id <owner-sub> --project-id <project-uuid>
python -m cowork.ops.projects listRepeat --recipient for multiple subjects. An explicit audience is required; resharing replaces it. unshare fails, changing nothing, when no grant matches the project and owner; list also shows grants whose project no longer exists, so they can be revoked by ID. Grants live in <COWORK_SHARED_DIR>/shared/projects.json, outside agent-writable project roots, and name immutable project and subject IDs. A rename retains a grant; a deleted and recreated project or account does not inherit a named grant. Everyone means this deployment only. Commands lock and atomically replace the registry, refusing to overwrite malformed or unreadable bytes. Requests fail closed on a broken registry. No database migration is required. Revoke grants before rolling back application versions.
Recipients never read dot paths, conversations, skills, memory, symlinks, hardlinks or nonregular files through these routes. Every path component is opened relative to pinned directories without following links. Listings visit at most 10,000 entries and 32 levels; text is capped at 2 MiB and downloads at 100 MiB. Downloads use attachment disposition and an opaque content type; no shared HTML preview token exists. Each request rereads the grant, so revocation affects subsequent requests. Previously downloaded bytes cannot be recalled. Single-user mode, MindsHub mode, and deployments that leave all projects globally shared keep their existing behavior.
On a deployment that trusts identity headers and keeps skills in COWORK_PERSONAL_RESOURCES, a person shares one of their own skills with everyone, read-only, or hides it again.
PUT /api/v1/skills/{skill_id}/sharingtakes{"shared": true}or{"shared": false}and returns the skill. Both directions are idempotent.- It answers 401 to a caller with no person: the desktop, or the operator view under
COWORK_IDENTITY_ENFORCE=audit. - It answers 409 "Sharing one skill at a time isn't available here." where this kind of sharing is unavailable: single-user mode, and skills left off
COWORK_PERSONAL_RESOURCES. In MindsHub mode the same route sets the skill's audience instead and ignoresshared(Sharing a skill in MindsHub mode). - It answers 403 "This skill is shared with you and can't be changed here." for a managed skill's name or a skill someone else shares with the caller, and 404 when the caller has no skill by that name.
- A share answers 409 with the reason for a packaged built-in, a name the agent already has from the harness's own skills or anton's packaged ones, a managed skill's name, a skill that can't be read in full, a skill holding a link, and a name someone else already shares a skill under.
- Any change answers 503 "Sharing is busy right now. Try again in a moment." while another change holds the list for 5 seconds, and while the filesystem refuses to read the list.
Each skill's capabilities.canShare is true for the caller's own skill where sharing is available, and false for a packaged built-in's copy, a managed skill and a skill shared with the caller. The skills list's top-level canShare says whether the caller may share here at all; the UI asks "who can use it" on create only when it is true. Both are false on the desktop, where skills are left shared and for the operator view. In MindsHub mode the list's canShare is false, and a skill's capabilities.canShare is true for its owner and for an organization admin, who may change its audience.
POST /api/v1/skills/ takes "shared": true, and POST /api/v1/skills/upload the multipart field shared, to share the new skill as it is created. Both default to false. A refused share answers 409 (503 when busy) and writes nothing; when the skill is written but its entry then can't be added, the new skill is removed and the error stands.
Deleting a shared skill stops sharing it. Renaming one answers 403 "Stop sharing this skill before you rename it.", because the list names a skill by its slug. A create, upload or rename into a name the person's list still has for a skill of theirs that is gone removes that entry first, so the new skill is shared only when they choose it. Once the list exists, every create, upload and rename into a new name takes the list's lock for that check: it answers 503 "Sharing is busy right now. Try again in a moment." while another change holds the list or while the filesystem refuses to read it, and writes nothing. A delete still deletes the skill then, and logs a warning naming the list it couldn't update.
Everyone else lists a shared skill after the packaged built-ins, the managed skills and their own, live from its creator's store, with isBuiltin false, shared true and no capability. A same-named skill earlier in that order hides it, and so does one the agent already has, and a listed skill that no longer exists is skipped; for a name two people share, the first entry wins. Their turns link it from the creator's store into a read-only skill root after the managed skills, with the turn's selection rule (enabled, and scoped to no project or to this project's name) applied to the skill's own metadata. Everyone else's edit, disable, rename or delete of it answers 403 "This skill is shared with you and can't be changed here."; disabling is refused too, because enabled lives in the skill's own SKILL.md. Everyone else's create, upload or rename into its name answers 409 "Someone else shares a skill named ''. Choose another name." Its creator sees it shared with every capability and edits, disables and deletes it as before, and everyone else sees each change.
The list is <COWORK_SHARED_DIR>/shared/skills.json, a JSON list of {user_id, partition_id, skill} in the order shared (cowork.shared_skills). Cowork writes it itself, under an flock on skills.json.lock beside it, replacing the file at once with mode 0644. A missing list shares nothing. A malformed one shares nothing too: every read logs a warning naming the file, the server still starts, and the next change that takes the list's lock replaces it, starting from empty: a share or hide, and a create, upload, rename or delete. A list larger than 1 MiB counts as malformed. A list the filesystem refuses to read (EIO, EMFILE, EACCES), or one that isn't a regular file (a FIFO, device, directory or symlink left in its place), is not malformed, so no change replaces it: while it lasts, readers share nothing, the server still starts, and every change logs an error naming the file, answers busy and writes nothing until an operator removes it. python -m cowork.ops.skills share and unshare change it on a person's behalf (Operator commands).
In MindsHub mode every person in an organization reads one skill store, so who each skill is shared with is a row beside it, in the skill_access table (cowork/services/skill_sharing.py holds the rules, cowork/services/skill_access.py the reads and writes). The row holds the skill's audience, its owner and always_on, keyed by the organization and the skill's folder name: a rename moves it and a delete removes it. Its org and owner are UUID columns, and its audience lists hold lowercase UUIDs.
| Shared with | Who sees and uses it | Who changes it (edit, rename, delete, enable) |
|---|---|---|
| Only me | Its owner | Its owner |
| Specific people | Its owner and each person named | Its owner |
| A team | Its owner and every member of the team | Its owner and every member of the team |
| Everyone at ‹org› | Everyone in the organization | Its owner, and anyone with skills.write_org |
- Changing who a skill is shared with is its owner's: teams, people, and taking it back from everyone. Sharing it with everyone takes
skills.share_orgtoo. A team a change adds must be one the caller belongs to. - An organization admin sees, changes and shares every skill on the skills page. Their turns use only the skills they'd see without the role.
- Built-in and managed skills are shared with everyone, read-only.
- A skill from before sharing gets a row shared with everyone, owned by its recorded creator, the first time the skill store's reader (
SkillService) reads its organization's rows, for a skills listing or a turn's skills. The always-on check at a turn's start (has_always_on_rows) reads the rows without that backfill: a row the backfill writes is never always on, so the check's answer is the same. A settings row (_skill_access_backfill_v1) marks the organization done. A folder with no row after that is its recorded creator's alone, or nobody's with none, so only an admin sees it. PUT /api/v1/skills/{skill_id}/sharingtakes any oforg_allowed,allowed_team_ids,allowed_user_idsandalways_on; a field left out keeps what the skill has. Every id must be a UUID (400 naming it), and a person it adds must be in the organization, asked of MindsHub in one lookup (400 naming each one who isn't).GET /api/v1/skills/sharing-capabilitiesanswersenabled(MindsHub mode),canShareSkillsOrg,canWriteSkillsOrg,canShareArtifactsOrg(MindsHub'sskills.share_org,skills.write_organdartifacts.share_org, all true for an admin),permissionsKnownandteamsKnown. WhilepermissionsKnownorteamsKnownis false, MindsHub couldn't be asked, so a false capability means "couldn't ask", not "may not". It names no organization: the app names the organization from its own list of the caller's organizations.PUT /api/v1/skills/{skill_id}/metakes{"enabled": false}to turn a skill off for the caller alone, andtrueto turn it back on. It works for any skill the caller sees and takes no edit rights. It works in every mode; on the desktop the one person is the nil UUID (LOCAL_PERSON_ID).- The caller's teams and permissions come from MindsHub (
GET /v1/me/teams/andGET /v1/entitlements/me/), cached per caller credential for 60 seconds once MindsHub answers, and 15 seconds when it doesn't. So a team or a capability MindsHub takes away still counts for up to 60 seconds. While MindsHub throttles the caller (a 429), its last answer is served for up to 120 seconds after it gave it. An artifact save that adds a team reads the caller's teams from MindsHub at the save instead (Who an artifact is shared with in MindsHub mode). - While MindsHub can't list the caller's teams, they are in no team: they see their own skills, the ones shared with everyone and the ones naming them, never one shared only with a team. A change whose answer rests on what MindsHub couldn't be asked answers 503 with
{"code": "mindshub_unreachable", "message": ...}, never a 403. A 4xx from MindsHub is a refusal, never "unreachable".
Only a skill shared with everyone or with a team keeps always_on; a change that leaves it shared with neither clears it. Marking a skill shared with everyone always on, or taking that back, takes skills.share_org. Marking a team skill takes edit rights: its owner, a member of one of its teams, or an admin.
An always-on skill reaches the organization or the team it is shared with. A person the skill is shared with personally gets it on demand: a skill shared with people is always on demand. A packaged built-in is never always on.
The routing gate answers some plain questions itself, without the agent, and its answer would not follow an always-on skill. So a turn an always-on skill reaches never meets the gate. The request reads the caller's teams from MindsHub, and then its first database unit reads whether an always-on skill reaches the caller in the conversation's project. When one does, the turn goes to the agent with the route reason always_on_skills, with no gate call and no first take. The skill files that read parses are read on the file store's threads (run_efs_unit), apart from the database unit. When that read fails, the turn goes to the agent with the reason router_unavailable, as when the gate's history can't be read. A caller no always-on skill reaches keeps the gate. While none of the organization's skills is always on, the read parses no skill file.
A cloud turn's request carries every skill the caller sees, the pod's whole recallable set, as {"skills": {<slug>: {"files": {...}, "always_on": true}}} (cowork/schemas/turn_skills.py), with always_on left out for a skill recalled on demand. The always-on skills come first, the ones shared with everyone before the team ones, since the pod puts them in the prompt in that order within its own budget. Every turn sends the block, empty when the caller sees nothing. A browser turn and a channel turn each build it in a unit of their own on the file store's threads (run_efs_unit), since it seeds the built-in skills and reads each selected skill's folder. When the request is over its byte cap, the producer drops the on-demand skills first, then project memory, then the always-on skills, team ones first (cowork/turnqueue/producer.py).
Both read MindsHub with the Authorization the gateway validated (caller_bearer), never the client-set X-MindsHub-Authorization. Outside MindsHub mode each answers empty and reachable. reachable: false means MindsHub couldn't be asked, which a client must not read as "nobody" or "in no team".
GET /api/v1/hub/teams/answers the caller's teams (MindsHub'sGET /v1/me/teams/), cached per caller as above, each withisDefault.GET /api/v1/hub/people/with no parameters answers the organization's directory, the caller left out, read fresh every time. MindsHub returns at most 500 members and saystruncatedwhen it holds more.?ids=a,banswers the members among those user ids, and?emails=a,bthe members who hold those addresses, so the app can share with a member past the directory's cap by id. Not both at once (400). Each is asked of MindsHub 50 at a time, and one request may name at most 200 (400 above that).- Every value is checked first, by the rule MindsHub applies, because MindsHub refuses a whole lookup for one value it doesn't take. An id that isn't a UUID answers 400
{"code": "invalid_id", "detail", "ids": [...]}; an address MindsHub wouldn't take (Django'svalidate_email, ported incowork/services/mindshub_people.py) answers 400{"code": "invalid_email", "detail", "emails": [...]}. MindsHub is not asked in either case. - A 4xx from MindsHub answers 403 or 400
mindshub_refused, or 429mindshub_throttled, neverreachable: false.
GET /api/v1/org-branding and GET /api/v1/hub/org-icons read auth with the caller's own credential (hub_credential_for) and cache each answer per caller for 60 s. Besides the branding or the organizations themselves, auth's 403 is the only reply that answers for the caller: no current organization, which answers enabled: false for branding and no organizations for icons. Any other reply says nothing about the organization:
- The branding read serves the caller's last answer for up to 10 minutes.
- Without one, a 4xx from auth (a 401 for an expired desktop session, a 429, or a 404 while auth runs a release without the route) answers 200 with
reachable: falseand the stock values. The app keeps the look it shows, and the Brand guidelines skill doesn't change. - No answer, a 5xx or a body that isn't auth's branding (not a JSON object with an
appearanceobject, such as{}) answers 503, since auth being down is an outage the ingress alerts should see. The app keeps its look on the 503 too. - The icons read answers 200 with
reachable: falsefor all of these, a body without auth'sresultslist included, and the org switcher keeps the icons it has. - Each reply from auth with nothing usable logs one WARNING line,
org_branding_read_unknownororg_icons_read_unknown, with auth's HTTP status instatus_code. No answer at all logs no such line.
cowork/services/sharing_contract.py holds every bound sharing applies, beside the audience names, the MindsHub capabilities, the MindsHub reads, the code of a refused share, what the agent is told after one, and how an artifact's rule was keyed before it had a stable id. A bound MindsHub owns is mirrored here so a share it would refuse is refused before anything is published. MindsHub's sharing contract module and the app's lib/sharingContract.js give the bounds they share the same names, and the upload service gives LEGACY_RULE_USER_DIR_CHARS and REPORT_ID_PATTERN the same names. MindsHub, the upload service, the app and anton's publisher give the refusal code the same name, and anton's publisher gives the sentence its agent is told the same name; tests/test_sharing_contract.py pins every name and value against tests/fixtures/sharing_contract.yaml.
| Limit | Value | Name | Owned by |
|---|---|---|---|
| People one skill or artifact is shared with by id | 100 | MAX_PEOPLE_PER_RULE |
MindsHub (an artifact's rule); this server applies it to skills |
| Teams one skill or artifact is shared with | 100 | MAX_TEAMS_PER_RULE |
MindsHub (an artifact's rule); this server applies it to skills |
| Addresses one artifact save asks MindsHub to email | 100 | MAX_NOTIFY_EMAILS |
MindsHub |
| Artifact title the share email names | 200 characters | MAX_NOTIFY_TITLE_CHARS |
MindsHub |
| Artifact link the share email carries | 2048 characters | MAX_NOTIFY_URL_CHARS |
MindsHub |
| Ids, and addresses, per MindsHub org-member lookup | 50 each | MAX_IDS_PER_LOOKUP, MAX_EMAILS_PER_LOOKUP |
MindsHub; the app mirrors both |
| Members the directory returns | 500 | MAX_DIRECTORY_ROWS |
MindsHub |
| The longest address MindsHub takes | 320 characters | MAX_EMAIL_CHARS |
MindsHub (its address check) |
| The longest id an artifact's rule is stored under | 512 characters | MAX_ARTIFACT_ID_CHARS |
MindsHub (the artifact_id its artifact-access routes take) |
| Hex digits of the owner's folder in a report's old id | 9 | LEGACY_RULE_USER_DIR_CHARS |
The upload service |
Ids, and addresses, per GET /hub/people/ request |
200 each | MAX_IDS_PER_REQUEST, MAX_EMAILS_PER_REQUEST |
This server |
The ids and emails query values of GET /hub/people/ |
8000 and 64800 characters | MAX_IDS_QUERY_CHARS, MAX_EMAILS_QUERY_CHARS |
This server |
| Concept | On the wire | In this server |
|---|---|---|
| Everyone in the organization | org_allowed (orgAllowed in responses); "Everyone at ‹org›" in the UI |
AudienceField.ORG_ALLOWED |
| Teams | allowed_team_ids (allowedTeamIds); with people, "Specific teams and people" in the UI |
AudienceField.ALLOWED_TEAM_IDS |
| People in the organization | allowed_user_ids (allowedUserIds) |
AudienceField.ALLOWED_USER_IDS |
| People outside the organization | allowed_emails in a save, accessEmails in the access state; anton's publisher calls them emails |
AudienceField.ALLOWED_EMAILS |
| Nothing shared | "Only me" in the UI | an empty audience |
| The capabilities | canShareSkillsOrg, canWriteSkillsOrg, canShareArtifactsOrg |
skills.share_org, skills.write_org, artifacts.share_org |
A share with everyone refused without artifacts.share_org |
403 {"detail": "<the reason>", "code": "share_org_required"} |
SHARE_ORG_REQUIRED_CODE, ShareOrgRequired (cowork/services/publish.py); the agent is told SHARE_ORG_REQUIRED_INSTRUCTION |
| MindsHub couldn't be asked | 503 mindshub_unreachable |
MindsHubUnreachable (cowork/services/mindshub_client.py) |
| The cloud turn's skills block | skills, each entry files and always_on |
TurnSkillEntry |
In MindsHub mode MindsHub's rule for an artifact is the one record of who it is shared with: everyone in the organization (org_allowed), teams, people in the organization by user id, and anyone else by address. The artifact's .published.json sits in a folder the pod can write, so it keeps the access mode, the password and the versions, never the audience.
GET /api/v1/artifacts/workspace/{project}/{artifact_id}/accessreads the rule from MindsHub'sPOST /v1/internal/artifact-access/read/(cowork.services.publish.stored_artifact_rule), under the artifact's identity first. A report published restricted before it had a stable id, and not published since, still has its rule under<user_dir>/<report_id>, whereuser_diris the upload service's folder for the owner (LEGACY_RULE_USER_DIR_CHARSincowork/services/sharing_contract.py). While the identity holds no rule, that one is read, so the save check, the dialog and every re-publish see the audience the report already has. When the entry'sreport_idisn't in the upload service's form (REPORT_ID_PATTERN), or makes the old id longer than MindsHub takes (MAX_ARTIFACT_ID_CHARS), no old id is read: the upload service refuses to publish a report of another form, MindsHub holds no rule under an id that long, and the pod can write thereport_id. MindsHub moves that audience to the stable id only for the report first published under it, so a member re-saving their own artifact already shared with everyone keeps it shared, and nothing else takes its audience. An identity MindsHub reserved before it recorded first reports takes no old audience, so a member's save, live sync or autopublish of such an artifact shared with everyone is refused withshare_org_requireduntil an admin shares it or the owner narrows it. A loose file has no identity, and this server sends the upload service no key for it, so its rule is read under the old id alone. Neither read asks a key the entry recorded, a loose file'sartifact_keyincluded: the pod can write those, and a rule under one may be another artifact's, so its audience would be sent as this one's.PUT .../accesschecks a save against that rule: sharing the artifact with everyone (org_allowed) takesartifacts.share_org, and a team it adds must be one of the caller's, by the caller's teams read from MindsHub at the save rather than the 60-second cache, so a team deleted a moment ago is refused before anything is published (an admin's team they aren't in must be one of the organization's, read the same way). While MindsHub can't list the caller's teams the save is a 503, and while it throttles the caller a 429. The org's default team is never sent: naming it means everyone (org_allowed), so a save that adds a team while the caller's teams can't be read is a 503, an admin's included. When a save changes the people, everyone it names is asked about in one org-member lookup. A person it adds who isn't a member is a 400. A person on the rule who has left the organization is dropped, because MindsHub keeps a departed person's id and refuses it on the next write that sends it. The caller's own address among the emails is a 400, and so is anotifynaming more than 100 addresses or 100 people, the most MindsHub emails from one save. All of this is answered before anything is published.- A save that would newly share the artifact with everyone without
artifacts.share_organswers 403{"detail", "code": "share_org_required"}and publishes nothing.detailis this server's own sentence when its check refuses the save, and MindsHub's reason when MindsHub refuses the rule's write and the upload service passes that on. A reason over 500 characters keeps only its start and its end, as any upload-service text does. The desktop'sPOST /api/v1/publish/and/updateanswer MindsHub's refusal the same way, and the agent's publish tool reportsshare_org_requiredand tells the model not to publish wider instead. MindsHub's refusal is logged once, as a warning withstatus_code403 andreasonshare_org_required, and no traceback. A live sync it refuses, such as a desktop entry that still shares with everyone after the rule stopped, adds no record of its own. - The artifact list (
GET /api/v1/artifacts/) and a chat's inline cards don't say who a restricted artifact is shared with: itsorgAllowed,ownerOnly,allowedTeamIds, andallowedUserIdsare null, and it carries noaccessEmails. OnlyGET .../accessreads the rule. A public or password card's audience fields arefalseand[]. On the desktop, the cards andGET /api/v1/artifacts/statusread the audience from.published.json,ownerOnlyincluded. - A save reads the rule once: the check's read is the rule the publish plans against. Every other restricted publish reads the rule too, so a live sync or an autopublish re-sends what the rule holds and writes no team or person.
- The access mode a live sync or an autopublish keeps comes from
.published.json, so it is checked against the rule. Only a restricted publish leaves a rule, and a public or password publish deletes it. The pod can writemetadata.jsontoo, which moves the artifact's identity, so the rule is read under that identity, under each key the entry recorded for its last publish (artifact_key,lambda_artifact_key), and under the report's<user_dir>/<report_id>id. A key MindsHub refuses as an artifact id (blank once stripped, longer thanMAX_ARTIFACT_ID_CHARS, or holding a NUL or a lone surrogate) holds no rule and isn't asked about. A re-publish whose entry isn't restricted while any of them holds a rule has a mode changed after the owner's last save. A re-publish of a page the entry names no key for can't be checked, because the pod can write thereport_idthat names the old id. Either is refused before anything is uploaded and logged asartifact_access_mode_changed, withreasonrule_heldorno_rule_key. The owner's next save of the artifact's sharing ends it. A key MindsHub binds to another owner, or holds for repair, fails the read (409): the rule there may be this artifact's. A page with no identity of its own, one in a subfolder of its artifact such as one the metadata'sprimarynames, or one whosemetadata.jsonan agent deleted, publishes with no key, and the keys left to check are ones the pod can write. The upload service refuses such a publish of a report that adopted a stable key (409), so an agent that points that page's entry at another report gets nothing published, while a page that has always published with no key keeps publishing. - The owner's own save also records the mode it publishes in, and that the artifact is published, in the
artifact_owner_choicestable, where the pod can't write (cowork/services/artifact_owner_choice.py). Every other re-publish is checked against the narrowest record under the keys above, and refused before anything is uploaded when it would open the artifact wider: public over password over restricted (reasonwider_than_owner_choice), or published while the record says unpublished (unpublished_by_owner). An artifact its owner hasn't saved since the table existed has no record, and keeps the rule check alone. - Before its upload, the owner's save narrows the record to the narrower of it and the new choice, and records the new choice once the upload is live, so the record is never wider than the owner's last confirmed choice. When MindsHub's upload service refuses the save before it changes who can open the artifact or what it serves, the save puts the record back to what it held before. Those answers are 401, 402, 403 (
share_org_requiredincluded), 409, 410, 413 and 429, judged by status alone, plus anton's refusal of a state schema change, which comes before the request. Any other ending can follow the upload service's writes, so the record stays narrowed: a 400, since a full-stack bundle can be refused after its files are uploaded, a 5xx, a timeout, a connection error, or a local error inside anton's publish, before or after its request. So does any refusal from a publisher that isn't MindsHub's (is_mindshub_publish_url), since its order of writes is unknown. Every owner save stamps the row (modified_at) before its upload, even when the mode stays, and every write of the row reads it under its lock. The restore runs only while the row still holds its own narrowing's stamp, so it never undoes another save's write, the same mode included, nor a save that started since and may have changed what is live. A restore that can't reach the database leaves the record narrowed and logs a warning (eventartifact_owner_choice_restore_failed), and the save answers with the upload service's refusal. - What these checks still can't catch:
- An artifact its owner hasn't saved since the table existed has no record, so its re-publishes keep the rule check alone, and that check can't see a password or unpublished entry an agent rewrites to public, since neither leaves a rule. Such an artifact is covered once its owner saves its sharing.
- A password changed inside a password entry: the mode stays password, and the record holds the mode alone.
- An agent that points the metadata id and both entry keys at keys that hold no rule and no narrower record, such as another of the owner's public artifacts.
- An unpublish: MindsHub mode has no unpublish of its own besides deleting the artifact, so no record says unpublished, and the
unpublished_by_ownerrefusal applies only once a route records one (record_owner_choicewithpublished=False).
- A read that fails fails the request or the publish (503) rather than guessing. A 409 or 403 from MindsHub gets the same answer on every try, so it answers 409.
MindsHub has to serve that read before this server is deployed.
On a local deployment with COWORK_TRUST_IDENTITY_HEADERS and a custom ANTON_PUBLISH_URL, the server records each published page's first tester and resolved source path in the database. Only that tester can update or unpublish that page from the original artifact, or delete that artifact while its .published.json names the live page; another tester gets 409, “This artifact was published by another owner.” This applies whether projects are shared or personal. Copies and edited .published.json files confer no ownership: publishing a different artifact or an unknown page id creates a new link, and unpublishing it or deleting its artifact clears only that artifact's local entry. The original page stays unchanged. Same-owner updates retain their link and password unlock, also after the server renames the project or moves the task that holds the artifact. A publish, unpublish or delete of a live page with no authenticated tester, such as a shared schedule's turn or a call with the shared token alone, is refused: the API answers 401 and the agent's publish tool reports the failure.
After schema migration and before readiness, a one-time backfill imports only provable owners from server-side publish history: a unique tester identity from trusted database rows, an existing source inside that partition's recorded project, and a matching page id and URL in the current publisher inventory. A page id that another artifact folder's top-level .published.json also names stays unclaimed, since the history row of its first publisher may have rotated out, and so does a page published from a subfolder of an artifact. Shared-project history contains no tester identity; ambiguous, conflicting, missing or moved sources remain unclaimed. The next publish of an unclaimed entry creates a new link and leaves its old link untouched, and the server logs orphaned_publish with reason=unrecorded_owner. History retains only the latest 100 successful publishes, so this cannot recover every old link. An unreadable history or unavailable publisher inventory leaves the backfill unfinished without stopping startup: each protected publish, unpublish or delete retries it and answers 503 until it completes. A project rename or task move before then can leave an old page unclaimed. Ownership imports and the completion marker commit together.
Ownership rows survive unpublish, restart and a schema downgrade. Database failure refuses a protected publish or delete before calling the publisher. This protects requests through Cowork; the operator's publisher credential still controls the publisher directly. A shared-project member's agent can still rewrite an artifact's .published.json: that can leave the owner unable to manage the page through Cowork until its id is written back, and can change the access the owner's next update sends, but it cannot change or remove the page itself. Hosted publishing and single-user desktop publishing retain their existing authorization.
With anton's ANTON_LIVE_APP_SLOTS set, a full-stack artifact runs as a live app in an app slot outside this server, and only the person who launched it opens it, at the slot's own address. anton launches it (anton.core.artifacts.live_app_slots) from a turn in this process and calls the turn's host (cowork.services.live_apps.CoworkLiveAppHost) around each launch: the host stops the slot's running launch record before the launch, and starts a new one naming the launcher once the slot accepts, so a launch that fails leaves nobody able to open the slot; anton then calls the host's launch_failed, and the app that launch stopped is reported as a stop the person hears of (launch_failed). Live apps open, share and copy by the person's sub and verified email, which the fronting proxy sends as X-User-Id and X-User-Email. A caller without an email (or with one under 3 or over 254 characters) gets 403 "Live apps need an email address on your account. Ask the operator to add one." from the routes that change something and an empty list from the others, and a turn without one is neither offered live apps nor handed a host.
Launch admission is shared across server workers. The host holds one PostgreSQL session advisory lock, or a file lock beside the SQLite database, while Anton rechecks the slots, resets the selected slot, launches, and records ownership. Approval cards and app probes run outside the lock. A competing launch that cannot acquire admission within 30 seconds asks the person to retry. Cancellation releases admission only after any in-flight database write or lock operation finishes.
When Anton refuses a launch because every slot belongs to someone else, the server remembers that person's wait. A fresh empty-slot status records one durable notice for every waiting person before another launch consumes the capacity. A stopped report alone is insufficient while the slot is restarting. Notices survive reloads and remain advisory: they reserve no slot and create no queue. A successful launch clears only its launcher's previous wait.
GET /api/v1/live-apps/<n>/accessanswers the edge in front of slotn's address, which sends the visitor'sCookieand nothing else. It answers 200 when the viewer sign-in (COWORK_LIVE_APPS_VIEWER_LOGIN_URL, its/oauth2/userinfo) names, bysuband by email, the launcher of the app the slot runs now, by the slot's running record and by the slot's ownGET /status; 401 for a visitor who isn't signed in; 403 for anyone else; and 503 when the sign-in or the slot can't answer. The owner never comes from the request, and no reply passes on the sign-in'sSet-Cookie.GET /api/v1/live-apps/<n>/stopped-pageis the page the edge shows at that address instead of a bare 403, 502 or 503: for its launcher, the app by the name its card shows, and when and why it stopped (12:04 UTCtoday,19 Oct 12:04 UTCbefore), with whether another person's app runs there now, or the address it runs at now; a plain refusal for anyone else.POST /api/v1/live-apps/<n>/stoppedtakes slotn's report that its app ended:idleormax_timewith the hours,app_exited,terminatedorreset, and when it ended, which the record keeps within the launch's own life. It takes no credential, since a slot holds none to send: the report is believed only from slotn's own address, with no forwarding header, and only for the boot id of the slot's running record, or of one the slot's status already stopped for the same reason, whose hours and time the report fills in. Aresetchanges nothing: after a claim the record is already stopped, and otherwise the slot's next boot marks a record it still runs lost.GET /api/v1/live-apps/minelists the latest launch of each of the caller's apps, newest first, running or stopped, with the reason as a code and a sentence. A running app whose slot is on a later boot, or that the slot list no longer has, is marked stopped,lost: at the slot's new boot time where the slot gives it, else when it is found. A slot that shows the app's boot exited, with the reason it keeps in its status, stops it for that reason; one that still runs it, or exited without a reason yet, keeps it running until its report says why it ended.GET /api/v1/live-apps/availabilityreturns the authenticated person's{waiting, notice, scope}. A notice contains anidandmessage;scopeis an opaque namespace for that person's browser state. Poll this route within one minute even when/mineis empty. A waiting person's poll checks real slot readiness when no launch holds admission, and records availability for all waiting people, while returning only the caller's notice. The notice's message names the time the slot was seen free (14:05 UTCtoday,19 Oct 14:05 UTCbefore).POST /api/v1/live-apps/availability/<notice_id>/ackacknowledges the caller's notice and returns 204. Another person's id, or an older refusal generation, changes nothing. The existing/mineresponse remains an array.POST /api/v1/artifacts/live-app-shareshares one of the caller's full-stack apps with one address ({"projectId", "slug", "email"}) or with everyone ({"projectId", "slug", "everyone": true}).GET /api/v1/artifacts/live-app-shares?projectId=&slug=lists the caller's shares of that app, newest first, andDELETE /api/v1/artifacts/live-app-share/<shareId>withdraws one of them (404 "This share no longer exists." for anyone else's); copies already made stay.GET /api/v1/artifacts/shared-live-appslists what others shared with the caller's address or with everyone, each app once, with the owner's address; an app whose folder is gone or can't be read drops out.POST /api/v1/artifacts/live-app-duplicatecopies a shared app into a project of the caller's own once the caller confirms the disclaimer ("confirmed": true, JSON true only), and answers the copy's card. The copy runs with the caller's connections. It takesbackend.py,requirements.txt,state_manifest.json,prd.md,spec.mdand every regular file understatic/, read without following links (at the owner's project folder too), and gets a newmetadata.jsonfrom anton's store with the app's name, description, type and declared connections, an empty provenance and a null port. It never takes the log, the publish state, the owner's port or any other file, and a link, a hard-linked file, a missingbackend.py, a folder it can't read or a file over anton's launch caps refuses the whole copy with 422 naming the path. A copy whose received mark can't be recorded is removed again. Each artifact card then carriesreceivedFrom, the owner's address for a copy, from the server's own record of it. Moving a conversation to another project takes its app's shares, received mark and launch records along with the folder.
Every one of these routes answers 404 unless live apps are on, before it reads the request, so a malformed one does too. The three slot routes pass the bearer and identity checks, since their callers carry neither, and each checks its caller itself. A turn's project block then tells the agent to build full-stack apps with generate_artifact and give their address in backticks, and names the person's running apps, each stop they haven't heard of (a turn that fails hands it back to the next), and the copies in the project not launched yet.
POST /api/v1/artifacts/workspace/<project>/<artifact id>/export with {"format": "html" | "docx" | "pdf"} writes the artifact's primary file, converted, as a new file beside it, and never overwrites a file. The server reads the source through the artifact's pinned folder, at most 10 MiB of HTML or 512 KiB of Markdown or text (413 "This file is too large to export." past that).
The conversion runs in a process of its own, which the server starts for each export (python -I -B -m cowork.services.export_child, or -E -P -B in place of -I -B when the server's interpreter uses the user's site folder, from cowork/services/export_runner.py). Python-Markdown spends its time in regular-expression calls that hold the interpreter lock and that nothing can interrupt, so a hostile Markdown file converted in the server would hold every other request for seconds or minutes. In its own process it holds only itself:
- Its limits. The server kills it 30 s after it starts, or once it holds more than 2 GiB (640 MiB in org mode, where a replica has 1 GiB for the server and its conversion together). On Linux it also limits its own address space to the same figure, so the kernel stops it at the limit, and it asks to be the first process the kernel kills when memory runs out (a runtime that kills a whole container at once ignores that). On Linux and macOS it limits its own CPU time to 31 s, can't write data to a file (it can still create an empty one), and can't start a process unless it runs as root. macOS refuses an address-space limit, and Windows sets no limit on the process at all, so the server's watch and wall clock are the limits there, and a process whose memory the watch can't read is killed; on Windows a server that crashes leaves a running conversion to end on its own. It runs at a lower CPU priority than the server everywhere. The inliner and the Word writer each stop at 10 s and still write the file with a note, which fits inside the 30 s.
- What it answers. A conversion the server killed, Markdown with a code-fence marker longer than 32 characters or a repeated heading whose number runs past 4,300 digits, a conversion that ran out of memory, or an output past the 64 MiB the process can send back answers 413 "This file can't be exported.", and nothing is written. One conversion runs at a time in a server process; another export answers 429 "An export is already running; try again in a moment." at once, as one person's or one organization's second export does. A conversion that fails any other way answers 500.
- What it can reach. It gets no path and no folder, and it starts in the filesystem root (on Windows, the drive's root), not the server's folder. It gets only the environment variables that find its home folder and its packages:
HOME, on WindowsSYSTEMROOTandUSERPROFILE, andPYTHONUSERBASEset to the server's own user base when the server's interpreter uses the user's site folder. None holds a credential. It reads the source from the server and asks the server for each image or stylesheet the document names; the server reads each one through the same pinned folder and checks as before, and only below the source's own folder. One export's reads scan at most 200,000 directory entries in all, so a document naming many files in one very large folder gets the last of them left out. It refuses network connections and process starts in Python, a Windows multiprocessing start included, and an export that ever needed that refusal fails. These are checks on the server's own code, not a sandbox: the process runs as the server's user, and a native exploit in a parser could bypass them. - What it needs. The server starts it with its own interpreter, so cowork-server must be installed as a package (
uv sync,uv tool install,pip installinto a venv, orpip install --user). A server run from a source folder throughPYTHONPATHcan't export. - What it logs. A stopped conversion logs
export_conversion_stoppedwith its reason (wall clock, memory, memory unreadable, refused, or a signal), and a refused one also the types and locations of what refused it; a failed one logsexport_conversion_failedwith its error type and where it failed, never the document's values; one that ended without a result logsexport_conversion_no_resultwith its exit code and the end of its error output.
A PDF's inlined HTML goes from the conversion process back to the server, which sends it to the render service (COWORK_EXPORT_RENDER_URL). Org mode refuses export (403) until ORG_EXPORT_ENABLED in cowork/services/artifact_export.py turns it on; scripts/measure_export_capacity.py measures what one export, or one published page, costs a 1 CPU, 1 GiB replica.
Publishing a Markdown artifact renders its page in the same kind of process, under the same limits and the same one slot. Publish reads at most 2 MiB of Markdown, which an ordinary report renders in about 3 s on one fast core. A list nested past Python's recursion limit publishes as its text in a pre block, as it exports. The POST /api/v1/publish/ and /publish/update routes, and the agent's publish tool, answer each refusal with its own status and code:
- a file past 2 MiB: 413
markdown_not_publishable, "This Markdown file is too large to publish. Publish a file under 2 MiB."; - a render the process stopped, or Markdown it refused, such as a repeated heading whose number runs past 4,300 digits: 413
markdown_not_publishable, "This Markdown file can't be published. Converting it took too long, used too much memory, or failed on something in the file."; - a busy slot: 429
conversion_busy, "Another document is converting on this server. Try again in a moment."; - a conversion that failed any other way: 500
markdown_conversion_failed, "This Markdown file couldn't be converted for publishing."
These routes, DELETE /api/v1/publish/, the rollback's POST /api/v1/publish/activate and the preview's GET /api/v1/artifacts/status run in the threadpool, so the server keeps answering while a page renders. Publishes, rollbacks and take-downs of one folder take turns, so two at once never start two pages; one that waits more than 10 s answers 503 "The resource is busy; retry the request". The "changed since publish" badge treats a refused render or a busy slot as can't tell. The server keeps up to 8 MiB of the pages it rendered, counted as UTF-8, so a listing renders an edited artifact once per edit, and it remembers the pages the process stopped, so neither the badge nor autopublish runs one again; a person's publish always tries again. An autopublish that finds the slot busy tries again at the project's next self-heal.
Configuration is read from the database (UserSettings table) and can be managed through the Settings UI in the desktop app or via PUT /api/v1/settings/.
The favicon setting stores a browser tab icon as a data URI, separately from the sidebar's nav_logo. It defaults to an empty string, which keeps the page's original icon. Saving an empty string restores that default. Like the sidebar logo, this is a personal preference scoped to each member within an organization; outside MindsHub mode the instance shares one setting. GET /api/v1/settings/ returns the data URI without masking it.
The show_figure_sources setting (on by default) is read only by the UI, which marks each figure in an answer with the steps it came from while it is on. The server finds and saves every answer's response.figure_sources whatever the setting, so turning it back on shows the marks on earlier answers too. Like favicon, it is a personal preference in an organization, and outside MindsHub mode the instance shares one setting.
Environment variables fall into these groups:
Server-level (COWORK_*) — control the cowork-server process itself. Each one also reaches the desktop sidecar through <COWORK_HOME>/.env, which the server reads below the process environment, so a value written there turns a setting on for the desktop too, except in org tenancy, where the server reads no <COWORK_HOME>/.env. The two COWORK_EXPORT_* PDF settings are the exception: the server reads them from the process environment only.
| Variable | Default | Description |
|---|---|---|
COWORK_LISTEN_PORT |
26866 |
Server port |
COWORK_SERVER_HOST |
127.0.0.1 |
Bind address |
COWORK_DRAFT_PORT |
0 (off) |
MindsHub mode. Port of the draft-preview listener: a second server in this process that serves only signed draft links, on the address in COWORK_SERVER_HOST. Links are issued (by the API) and served (by this listener) only with COWORK_TENANCY_MODE=org and all six draft values set; otherwise the listener answers 404 and web previews stay on srcdoc. /health reports which one as draft_links. |
COWORK_DRAFT_HOSTS |
unset | Comma-separated host names of the draft host (e.g. draft.mindshub.co). The listener answers 404 to any other Host. |
COWORK_DRAFT_ORIGIN |
unset | Origin draft links point at, e.g. https://draft.mindshub.co. |
COWORK_DRAFT_LINK_KEYS |
unset | Draft-link signing keys as JSON {kid: secret}, from the mindsdb-secrets Secret. Two kids can be live during a rotation. A secret shorter than 32 characters is ignored. A pod reads it at start. |
COWORK_DRAFT_LINK_KID |
unset | The kid in COWORK_DRAFT_LINK_KEYS that signs new draft links. |
COWORK_WEB_ORIGIN |
unset | Origin of the cowork web app, e.g. https://cowork.mindshub.ai: the only page allowed to frame a draft (CSP frame-ancestors). |
COWORK_BROWSER_ROUTER_URL |
unset | MindsHub mode only. The browser router's base URL, https:// with an optional path, for example https://cowork.mindshub.ai/browser; the trailing slash is dropped. Unset, the web app offers no browser and makes no browser call. Ignored outside MindsHub mode, where the desktop app runs its own browser. See The agent's browser. |
COWORK_TENANCY_MODE |
local |
local is single-user mode, the desktop sidecar (one person, no organization, no identity headers), or single-tenant mode with COWORK_TRUST_IDENTITY_HEADERS (Server modes and terms). org is MindsHub mode, the cloud deployment, and turns on everything in Who can read what in MindsHub mode above. |
COWORK_DEPLOYMENT_KIND |
unset | self-hosted tells the agent this installation runs in the operator's environment. Requires local, in-process turns and a branding pack. Unset keeps desktop and hosted identity and installation guidance. |
COWORK_DEPLOYMENT_BRAND_FILE |
unset | Absolute path to the selected, read-only pack.json. With COWORK_DEPLOYMENT_KIND=self-hosted, its name becomes the agent's product name. Startup accepts the branding pack loader's JSON encodings, including a UTF-8 BOM and UTF-16/32. It refuses missing, malformed or linked files, duplicate JSON keys and invalid names. The name is fixed until restart; member settings cannot change it. |
COWORK_SURFACE |
inferred | Set web for a self-hosted browser installation. Without an override, MindsHub mode reports web, and single-user and single-tenant mode report desktop. It sets trace attribution and, without COWORK_DEPLOYMENT_KIND, the agent's location line, so web makes the agent say it runs in the browser. With COWORK_DEPLOYMENT_KIND=self-hosted, the deployment identity's browser line applies instead. It does not select a deployment identity. |
COWORK_IDENTITY_ENFORCE |
enforce |
MindsHub mode and single-tenant mode. enforce answers 401 to a request carrying no valid identity headers. audit logs it and lets it through, and has to be asked for. |
COWORK_ORGANIZATION_SWITCH_ENABLED |
false |
Shows the canonical-web organization picker. Requires MindsHub mode and identity enforcement. The product enable, not a safety switch: the expected-organization boundary refuses a mismatched tab whatever this says. Long-lived Helm environments set true. |
COWORK_TRUST_IDENTITY_HEADERS |
false |
With COWORK_TENANCY_MODE=local only, where on is single-tenant mode. Reads X-User-Id and X-Organization-Id on a request that passed the bearer token, and gives each pair its own partition. The partition id is uuid5(X-Organization-Id, X-User-Id) (cowork.principal.personal_partition_id); new rows carry it as org_id and the user as created_by. Only for a deployment behind a proxy that verifies each person, strips client copies of those headers (and of X-User-Email, which the server reads as the person's address and live apps trust; the proxy sets it only from the verified token's email), and refuses a person's write (PUT or DELETE /api/v1/settings/<key>) of an operator-pinned setting: that write lands on the global row every person reads, and nothing here refuses it. Where COWORK_PERSONAL_RESOURCES leaves out memory, the proxy must also refuse a person's PUT or DELETE under /api/v1/memory/: that write lands on the store every person reads, and nothing here refuses it. COWORK_IDENTITY_ENFORCE applies: enforce answers 401 without a valid pair, and audit serves a request with no pair as the operator view, which lists every person's conversations, running answers, projects, files, schedules and pins, and reads skills, connections, artifacts and memory from the deployment's own stores; every artifact route given a person's project_id there (the listing, a delete by id, drafts, workspace and review) answers 404. A pair that is present but broken (one header alone, or an id that is not a UUID) answers 401 under either mode. Paths outside /api/, the web UI's page shell, name no person and log nothing. A path with a decoded . or .. segment answers 400. Only a genuine CORS preflight (Origin with Access-Control-Request-Method) passes the bearer check without the token, so an OPTIONS request can't name a person either. With it on, every connection a save writes records its creator, the person's sub, in the bookkeeping field _created_by: a create stamps it, an update keeps the stored value whatever the request says, and the operator view stamps none. It never reaches the agent's environment or the connections API's fields. Every skill created through the server (create or upload) records its creator's sub in its SKILL.md metadata as created_by, never a value the request or the uploaded file carries; an edit, rename or project-rename rewrite keeps it, and the operator view, the packaged built-ins and the managed skills record none. The server refuses to start with it on in MindsHub mode, with COWORK_REQUIRE_AUTH off, with COWORK_TURN_BACKEND=remote, whose turns keep memory in each project's own folder rather than each person's or the deployment's tiers, with COWORK_CODE_SANDBOX off, and without COWORK_AUTH_TOKEN in its environment. With it on, the token acts as any person, so the server never generates it or copies it into <COWORK_HOME>/.env, and it removes a COWORK_AUTH_TOKEN line an earlier start wrote there. The desktop sidecar reads it from <COWORK_HOME>/.env like every setting here, and the desktop has no Landlock, so turning it on there stops the sidecar from starting. |
COWORK_PERSONAL_RESOURCES |
all nine | With COWORK_TRUST_IDENTITY_HEADERS on, the comma-separated resources each person keeps in their own partition: conversations, projects, files, skills, connections, schedules, pins, memory, settings. A resource left off stays shared across the deployment, as with the setting off. memory left off makes the deployment's own memory one store that every person's turns read and learn into, rules, lessons and profile included: its global tier is COWORK_MEMORY_DIR, and each project's tier sits under it at projects/<project id>/, never in the project's .anton/memory. Each turn reads the shared global tier and its current project's tier, never another project's. The server starts, and logs a warning at start that memory is shared (or, with MEMORY_ENABLED false, that no turn learns and the shared store is kept). Everyone's Memories listing (GET /api/v1/memory/) then holds the shared global tier and every project tier in that store, the caller's own projects included, each under its project id alone: a person sees the learned tier of a project they can't open without its name or files, and gets no capability on it. projects requires conversations, and conversations requires files, pins and schedules; the server refuses to start when the list breaks that, and an unknown name fails settings load. A personal schedule runs as its creator. With memory personal, a person's global tier is <shared>/<partition>/memory/users/<sub>/ and each of their project tiers sits under it at projects/<project id>/, never in the project's .anton/memory; a turn with no person reads and writes no learned memory. With schedules shared (memory alone listed, for one), a scheduled run has no person, so it reads and learns no memory: a shared run that used its creator's memory would show it to everyone. Deleting a project removes its tiers: the deleter's where projects are personal, every person's where they are shared, and the project's tier in the deployment's store, whichever way memory is kept now, since a tier from before a switch stays on disk. Moving memory onto or off the list moves and merges nothing: each person's tiers and the shared store are separate folders, and each comes back as it was. With settings personal, the operator-pinned keys (the model, its endpoint and credentials, memory, the per-turn budgets, and how the agent acts: act_first and proactive_dashboards) still resolve from the global row, then the environment, and a person's own row for them is ignored. Whatever the list holds, a person's PUT or DELETE of one of those keys rewrites that global row for everyone, so the fronting proxy must refuse it. With projects personal, every artifact route resolves a path, a preview token, a project name or an artifact id only inside the caller's own projects, and opens each artifacts folder from the outermost of the server's roots without following links, as COWORK_CODE_SANDBOX opens each folder it grants, so a person's partition folder, projects store or project replaced by a link to someone else's answers 404 or an empty list. So does a project recorded while the person's partition folder was such a link: it names a folder in the other person's partition, and keeps naming it once the link is gone; the server logs an artifact_folder_refused warning with the project's id. A preview counts a backend as running only when this server launched it for that project and artifact, the preview proxy forwards a request only to that live backend (503 otherwise), and publish history is per person. A person's open and reveal (/artifacts/open, /artifacts/reveal) are refused whatever the list holds. Publishing and export work. Export (POST /api/v1/artifacts/workspace/<project>/<artifact id>/export) writes a new Word file beside the artifact's primary file, or a new HTML file for a Markdown or text primary, and never overwrites a file. It builds that file only from the artifact's own folder, opens no network connection, and leaves out every image or stylesheet the source names outside that folder. With projects shared, deleting a project answers 409 while another person still has conversations or schedules in it, the creator's delete included. With projects personal, a person's delete answers 409 too while the operator view has left a conversation or schedule in their project. With projects shared and conversations personal, a conversation's workspace (conversations/<id>/ in the project folder) stays its owner's: anyone else's read, write, delete or listing answers 404. Choosing an existing folder as a project (adopting it) is refused on this deployment whatever the list holds. With skills personal, nothing fans skill links out of the shared store; each turn seeds the read-only builtins into the person's store and links that store's skills into its project, and the skills others share with the person (Sharing one skill with everyone) into a read-only root after the managed ones. With skills shared, everyone uses every skill in the one store, and a person changes, disables, renames or deletes only a skill they created: anyone else's change answers 403 "This skill is shared with you and can't be changed here.", and the skills API marks it shared and gives them no capability. The operator view changes any skill there, one with no recorded creator included (made before creators were recorded, by the operator view, or a packaged built-in's copy). Skills name their projects by name, and a project rename rewrites those names in the renamer's skill store, with one exception: with projects personal and skills shared, a person's rename leaves the shared store alone, because a shared skill scoped to a name applies to every person's project of that name. With skills personal and projects shared, a rename rewrites only the renamer's own skills. Local OAuth sign-in is refused (403) on this deployment whatever the list holds, the operator view included: the provider's redirect to the callback carries no identity. Connect the service with its credentials instead. With connections personal, a turn and an artifact's live backend read only the person's own vault and the connections others share with them (COWORK_MANAGED_DIR). With connections personal and projects shared, trusted-identity sandbox startup refuses the configuration: shared venvs and saved cell state can run one person's code with another person's credentials. Enable personal projects before startup. The live-backend preview and proxy also retain their own 409 refusal for this combination. With connections shared, everyone uses every connection, and a person changes or deletes only a connection they created: anyone else's change answers 403 "This connection is shared with you and can't be changed here.", and the connections API marks it managed and shared; one with no recorded creator stays managed only. Where two connections there have names that map to the same DS_ environment prefix (for example sales-db and sales_db), a person's listing and turn keep their own and hide the other, with a warning in the log. The operator view changes any connection there, one with no recorded creator included (saved before creators were recorded or by the operator view). A turn with no person, such as a scheduled run where schedules are shared, changes none: its agent can't relabel, overwrite or delete a saved connection, and one it saves records no creator. With connections personal or shared, a connection the agent saves from chat updates only one the person can change: one whose identity fields (for postgres, the database alone) match a connection shared with them, or someone else's, is saved as a new connection of their own. Ignored while COWORK_TRUST_IDENTITY_HEADERS is off. |
COWORK_CODE_SANDBOX |
off |
How the agent's code runs (cowork.code_sandbox). off: every scratchpad cell, venv check, package install and artifact check starts as a child of the server, with the server's environment and files, as on the desktop. landlock: each one starts through anton's launcher, which locks it with Landlock, a Linux kernel feature, and hands it an allowlisted environment without COWORK_AUTH_TOKEN, DATABASE_URI or MINDS_API_KEY. A cell then reads, writes and runs code only in the conversation's project folder and the asker's own HOME and TMPDIR (home/ and tmp/ in <shared>/<partition>/sandbox/, each person's own whatever COWORK_PERSONAL_RESOURCES leaves shared, or <COWORK_HOME>/sandbox/ for the operator view); reads and writes <COWORK_HOME>/shared-files, which the server creates at start and every person's agent shares, but runs no file there by its path, so nothing one person leaves there starts as a program in another's cell (an interpreter can still run a script it reads from there, such as python script.py); and only reads the turn's skill folders (the asker's own, the ones shared with them, and each skill in the managed folder's skills/), the conversation's attachment folders, and the system's read-only files. With skills and projects both left off COWORK_PERSONAL_RESOURCES, each project's skills/ links into the one shared skill store, so every cell reads that whole store, read-only. Everything else is denied: the rest of the person's partition, every other person's, <COWORK_HOME>'s own files, the rest of the managed folder, /home and the shared /tmp. The launcher grants what a path resolves to, so the server opens each folder it grants inside <COWORK_HOME>, the shared root, or the projects, files or skills root from the outermost of those roots without following links; a root kept on another disk is set by its own path, not linked into <COWORK_HOME>. A link on the way, which agent code from before the sandbox may have left, fails the turn when it leads to the project, HOME, TMPDIR or shared-files (a link at shared-files itself fails the start), and leaves a skill or attachment folder out of the grants. A turn also refuses a project reached through a link before writing into it, makes a missing project folder again without following links, refuses an ANTON_CONTEXT_DIR outside the project, and makes its own folders in the project (skills/ and the folders in .anton/) without following links. With off, those folders are made as on the desktop. The server itself isn't locked, so a link or a hard-linked file in a person's data vault, the memory tiers a turn hands anton (the deployment's shared ones where memory is shared), or the publish store fails that read or write (the turn, the connection's save or the publish), naming the path: agent code from before the sandbox may have left it there to send the server's writes into another person's space. Everyone else reading the connections that person shares goes on without them, and the server logs an error naming the vault and each connection left out. Whatever COWORK_CODE_SANDBOX says, the artifact routes /api/v1/artifacts/serve, preview, preview-mount and preview-asset answer 404 for a file with a second name, a hard link, as the drafts route does, and export answers 422 for such a primary: a cell can still hard-link a file inside its project, such as .anton/.env, into an artifact folder. The project file routes (files, files-raw and the project preview-asset) and the artifact workspace source answer 404 for the project's own .anton/.env under a second name, and keep serving other hard-linked files, such as a package store's or a backup tool's. A publish refuses an artifact holding a file with a second name. These refusals stop a second name for .anton/.env, not a copy: the agent's own code reads the file and gets its keys as environment variables. With COWORK_TRUST_IDENTITY_HEADERS on and MEMORY_ENABLED false, a turn hands anton no stored tier, so no tier is checked. The agent's prompt names shared-files and says every person's agent reads and changes it. A credential probe hands its candidate credentials to the cell as its DS_* variables instead of a file, both the flat names the probe's prompt gives and the namespaced ones anton's connection listing gives, and one with no conversation runs in probes/ beside the asker's HOME. App backends are refused (a backend preview answers 409 "App backends are off on this deployment because agent code runs in a sandbox.", the preview proxy answers 403 and forwards nothing, and the agent is never offered one unless live apps run it outside this server, Live apps), and every Code mode route answers 403. Required with COWORK_TRUST_IDENTITY_HEADERS. The server refuses to start with landlock in MindsHub mode, with COWORK_TURN_BACKEND=remote, on an anton without the launcher (anton.core.backends.sandbox) or whose policy model refuses the policy this server builds, on a kernel below Landlock ABI 3 (Linux 6.2; Debian 12 needs the bookworm-backports kernel, or Debian 13), with a link at <COWORK_HOME>'s .env, .master_key, streams, tmp or shared, with a link at a person's partition folder in the shared root or at one of its stores (projects, files, memory, data-vault and the rest), and when its startup self-test can't confirm that a path outside a grant is denied. With trusted identity headers, personal connections require personal projects: startup refuses the unsafe combination before probing the kernel. Shared project venvs and saved cell state can execute one person's code with another person's credentials. When both projects and connections are explicitly shared, startup still logs the shared-execution warning. |
COWORK_MANAGED_DIR |
unset | The operator's managed folder, which Cowork only reads and every turn gets, whoever asked it. memory/ holds rules.md and lessons.md (and an empty profile.md): anton's Cortex reads them as a read-only layer ahead of the turn's own memory and never writes them, a turn with no person included. skills/<slug>/SKILL.md are skills every person lists as built-in, with no capability; they follow the harness's own skills in anton's read-only skill roots, a person's same-named skill is never linked into a turn, and every change to a managed name (create, upload, rename into, edit, disable, delete, share) answers 403 "This skill is shared with you and can't be changed here." They are the skills the operator adds, which nobody created. A managed skill folder the server's user can't read after the start is left out of the listing and every turn, logged with the chmod fix, and keeps its name reserved. data-vault/ holds the operator's own connections in the vault's own file format: every vault read lists them first, marked managed in the connections API, a same-named connection of the reader's own stays hidden, and so does one of the reader's own whose name maps to the same DS_ environment prefix, and every change to one answers 403 "This connection is shared with you and can't be changed here.", a delete before its OAuth revoke. shared/connections.json lists the connections people share, a JSON list of {user_id, partition_id, engine, name} that python -m cowork.ops.connections share and unshare write and the server reads live. Where connections are personal, everyone else reads each listed connection after their own, live from its creator's vault, marked managed and shared in the connections API. A same-named connection earlier in that order (the operator's, then the reader's own) hides it, and so does one whose name maps to the same DS_ environment prefix (for example sales-db and sales_db), which is logged as a warning; a listed connection that no longer exists is skipped. Everyone else's change to it (delete, access mode, picked files, token, label, or a save of that engine and name) answers 403 with the same sentence, before any OAuth revoke. Its creator sees it shared but not managed and changes, reconnects and deletes it as before, and everyone else sees each change, a refreshed token included. Once its creator deletes it, its entry shares nothing, and a new connection of theirs under that name answers 403 "This name is still shared for a connection of yours that is gone. Ask your operator to stop sharing it before you use the name again." until the operator unshares it. Blank means unset. The server refuses to start with it set in MindsHub mode or with COWORK_TURN_BACKEND=remote, whose workers receive none of it, when the folder or any of memory/, skills/ and data-vault/ is missing (it names each one), when the server's user can't read and search memory/ (every turn would fail reading its rules), when it can't read and search skills/ or read a managed skill folder or its SKILL.md, when it can't read and search data-vault/ or open a managed connection file read-only (it logs managed_connection_unreadable, with the stored filename in managed_connection_file for a file failure; see Server diagnostics), when shared/connections.json is malformed (it names the file), and when the installed anton's Cortex takes no managed_hc. The connection check opens files without reading their contents, changes no ownership or permissions, and adds no warning when all connections are readable. Repair permissions on the host, then restart Cowork to repeat the check. A missing shared/ folder or file shares nothing, and a registry broken after the start shares nothing and logs an error until it is fixed. The desktop sidecar reads it from <COWORK_HOME>/.env like every setting here. |
COWORK_SHARED_DIR |
~/.cowork |
MindsHub mode and single-tenant mode only. Root of the org-keyed tree: <shared>/<org_id>/{skills,memory,projects,files,data-vault} (and publish for a personal partition), where a personal partition's <org_id> is its partition id. Where skills are personal, it also holds shared/skills.json, the skills people share, which Cowork writes (Sharing one skill with everyone); its server user needs write access to shared/. In MindsHub mode, point it at the durable mount: on the default the data is ephemeral (boot warning). The server resolves its links once at startup, to tell which partition a project's folder sits in without a file store call per request, so a link that changes while the server runs takes effect at the next restart. |
COWORK_PROJECTS_DIR |
~/.cowork/projects |
Project storage root (outside MindsHub mode). Set it by its own path: artifact routes open every folder below the outermost of the server's roots without following links, so a folder between COWORK_HOME and this root replaced by a link leaves its projects' artifacts out, and the server logs an artifact_folder_refused warning. |
COWORK_FILES_DIR |
~/.cowork/files |
Uploaded files root (outside MindsHub mode) |
COWORK_SKILLS_DIR |
~/.cowork/skills |
Skills store root (outside MindsHub mode) |
COWORK_MEMORY_DIR |
~/.cowork/memory |
Memory store root (outside MindsHub mode) |
COWORK_VAULT_DIR |
~/.cowork/data-vault |
Connector credential vault |
ANTON_LIVE_APP_SLOTS |
unset | anton's live-apps switch, the app slots' control addresses, comma-separated, read through anton (live_apps_enabled). With it set, the agent builds full-stack apps and anton launches each into a slot outside this server (Live apps). The server refuses to start, whenever it is set or anton reads live apps as on, with COWORK_CODE_SANDBOX off, COWORK_TRUST_IDENTITY_HEADERS off, projects left off COWORK_PERSONAL_RESOURCES, an anton without anton.core.artifacts.live_app_slots, whose live_app_slots lacks a name this server uses (the switch, the slot addresses, SlotStatus, LiveAppLauncher, LiveAppHost, LiveAppLaunchError, the launch list and its caps, MAX_REPLY_BYTES), whose LiveAppHost lacks launch_lock, slots_full or slots_available, or whose ChatSessionConfig takes no live_app_host, an entry that doesn't parse (a blank one included), COWORK_LIVE_APPS_VIEWER_LOGIN_URL that isn't an http:// or https:// address with a host, no query or fragment and no space around it, or COWORK_LIVE_APPS_LAUNCH_TOKEN shorter than 32 printable ASCII characters or with a space around it. |
COWORK_LIVE_APPS_VIEWER_LOGIN_URL |
unset | With live apps on, the viewer sign-in (oauth2-proxy) the access check asks who a visitor is, for example http://viewer-login:4180: its /oauth2/userinfo, with the visitor's cookie, never through a proxy variable, trusting SSL_CERT_FILE or SSL_CERT_DIR for an https:// one on a private CA. |
COWORK_LIVE_APPS_LAUNCH_TOKEN |
unset | With live apps on, the token anton sends each slot with a launch, and this server with each status read; a slot holds only its SHA-256. With COWORK_CODE_SANDBOX=landlock the agent's code never gets it, because anton's launcher passes no COWORK_* name. |
COWORK_PUBLISH_TOKEN |
unset | Outside MindsHub mode, the bearer the server sends the publish service (ANTON_PUBLISH_URL) for every share, unpublish and version change, in place of the provider's key, whatever the provider. MindsHub mode ignores it: there each publish carries a per-user key the server mints. Without it, the server sends openai_compatible_api_key when openai_base_url is a MindsHub API host (api*.mindshub.ai), and minds_api_key otherwise. Set it for a publisher you run yourself: on a MindsHub endpoint the agent's code holds the model key as its OPENAI_API_KEY, so a publisher that takes that key takes calls from the agent's code too. With COWORK_CODE_SANDBOX=landlock the agent's code never gets this variable, because anton's launcher passes no COWORK_* name. With off, a cell inherits the server's whole environment, this variable included. Blank means unset. |
COWORK_OPENAI_COMPATIBLE_API |
chat_completions |
The API an openai_compatible provider's planning, coding and router roles call. responses moves all three to {base}/responses through anton's openai flavor, where OpenAI and Azure accept function tools together with a reasoning effort. It needs an anton that reports RESPONSES_TRANSPORT_READY; with an older anton all three roles stay on chat completions, and Cowork logs one warning that says so. On the Responses path the agent loop runs without web tools, because OpenAI's hosted web_search reads the web from the provider's side, outside the deployment's egress controls. A Python cell's web_search() is separate and unchanged. The router's gate, first take and history summaries share this selection. Gemini is unaffected. Read once at start. |
COWORK_EXPORT_PDF_ENGINE |
unset | render turns on PDF export through an optional HTML-to-PDF render service at COWORK_EXPORT_RENDER_URL; unset, no artifact offers PDF and no other library makes one. Read from the process environment only, never from <COWORK_HOME>/.env, which POST /settings/raw can write on the desktop. Any other value stops the server's start. |
COWORK_EXPORT_RENDER_URL |
unset | The render service's http:// or https:// base address, with no query or fragment; the server posts each PDF export to <URL>/v1/pdf, with no proxy and no redirect, and waits at most 60 s in all. An https:// service needs a certificate from a public certificate authority: the server checks it against the certifi bundle and ignores SSL_CERT_FILE and SSL_CERT_DIR, like the proxy variables. Read from the process environment only, like COWORK_EXPORT_PDF_ENGINE. With the engine render, a missing or non-http(s) URL stops the server's start. |
A failed router check delegates to the agent and logs the event router_fallback at WARNING, with outcome delegated_agentic, the reason (such as router_timeout or router_unavailable), and the provider and conversation_id when available, without a prompt or provider response body. It also covers failures before the check starts, such as a missing model, unavailable history or a failed read of the turn's always-on skills. Intentional delegation logs router_route at INFO. Every check that runs also logs router_gate_decided at WARNING with its outcome, reason and duration_ms. The gate keeps its two-second first-event, five-second idle and ten-second total budgets; selecting Responses does not make a slow model meet them. A provider failure also logs router_call_failed with the exception class as error_type and the HTTP status as status_code when available, without the provider's text. These are fields of the structured log line (Server diagnostics), not text in its message.
With COWORK_DEPLOYMENT_KIND=self-hosted, outside MindsHub mode and with local turns, each server process also starts one background router probe after loading its configuration. It sends a synthetic greeting with the gate's function tool through the effective router model, API and effort, with a 30-second response deadline and separate two-second cleanup waits (up to eight seconds in all). It executes no tool and creates no Cowork conversation. The call can consume provider tokens; the normal transport's retries and provider retention settings still apply.
Look for the events router_startup_probe_passed, at INFO, and router_startup_probe_unverified, at WARNING with its reason, provider, error_type and status_code when available, without provider bodies, credentials or exception text. A refused request, timeout, empty response or incomplete stream never prevents startup. A pass confirms only this synthetic request, not the gate's latency budget or model quality, and the result may arrive after the first user request. Restart after changing the router configuration to recheck it. Single-user mode and MindsHub mode do not run this probe, and health checks do not call a model.
Database pool: these read their bare names, with no DATABASE_ prefix. A SQLite engine is built without them and waits SQLAlchemy's 30 s for a connection outside a database unit; its database units run one at a time, and POOL_TIMEOUT still bounds a unit's waits.
| Variable | Default | Description |
|---|---|---|
POOL_SIZE |
20 |
Connections the pool keeps open. |
MAX_OVERFLOW |
20 |
Connections the pool may open beyond POOL_SIZE under load. Database units take one of POOL_SIZE plus MAX_OVERFLOW slots before they ask the pool. A negative value means no limit, for the pool and for units. |
EFS_UNIT_SLOTS |
16 |
Database unit slots the units that also work on the shared file store (run_efs_unit) may hold at once, out of POOL_SIZE plus MAX_OVERFLOW. Such a unit holds its connection while the file store answers, so units stuck on a slow store never hold every connection and database units keep the rest. Keep it below POOL_SIZE plus MAX_OVERFLOW. SQLite gives file units no unit slot, so it ignores this. It must be at least 1. |
POOL_TIMEOUT |
5 |
Seconds a database unit may wait in all, for a slot and then for a connection, before it is refused: a request with 503 and a Retry-After of the same number of seconds, a streamed answer with server_busy. On Postgres it is also the pool's own wait, so every route that reads the database answers 503 with Retry-After once it passes, /api/v1/health/ included. /api/v1/health/live and /api/v1/health/ready read no database and keep answering. It also bounds the wait for a shared file store slot (run_efs, see "Work on the shared file store"): a request that gets none in time answers 503 with the same Retry-After, and a download whose headers are already out waits instead. It must be at least 1; a lower value stops the server at startup. |
STATEMENT_TIMEOUT_MS |
0 |
PostgreSQL ends any statement on the app's own connections that runs longer than this many milliseconds, and the failure logs sqlstate=57014. 0 leaves it off. Migrations never carry it: the db-migrate init container opens its own connections, and the server's startup migration turns both timeouts off for its own transaction. SQLite ignores it. |
IDLE_IN_TRANSACTION_TIMEOUT_MS |
0 |
PostgreSQL closes an app connection that sits idle inside an open transaction for longer than this many milliseconds, and the next statement on it fails with sqlstate=25P03. 0 leaves it off. SQLite ignores it. |
SLOW_STATEMENT_LOG_MS |
0 |
Logs a db_statement_slow warning for each PostgreSQL statement that takes at least this many milliseconds, with its duration_ms, its verb and first table as operation, and a statement_fingerprint of its SQL text. The line never carries the SQL or its parameters. 0 leaves it off. The hosted chart sets 1000, so the two timeouts above can be chosen from what hosted statements take. |
Shared file store slots: these read their bare names too, and POOL_TIMEOUT bounds the wait for a slot.
| Variable | Default | Description |
|---|---|---|
EFS_SLOTS |
100 |
Threads run_efs may hold at once per process (see "Work on the shared file store"). A call stuck on the mount keeps its thread, so this is also the most threads a stalled mount can take. It must be at least 1. |
EFS_SHARE |
10 |
Of those, the threads one person may hold at once, or one organization's work that runs for no person. Nothing caps an organization: with the defaults, ten busy people fill the slots. It must be at least 1. |
Harness-level (ANTON_*) — configure a specific agent harness. These are read by the harness adapter, not by cowork-server core. They use the harness prefix because the upstream agent library (anton) defines them:
| Variable | Harness | Description |
|---|---|---|
ANTON_PUBLISH_URL |
Anton | Artifact publish endpoint. Pointed anywhere but MindsHub's service, it takes HTML and Markdown only: sharing a full-stack app is refused before anything is uploaded, because that upload carries the app's connection credentials |
ANTON_SKILLS_ROOT_DIR |
Anton | Skill file storage |
ANTON_GLOBAL_MEMORY_ROOT_DIR |
Anton | Global memory files |
ANTON_WRITE_APPROVAL |
Anton | Off by default. On, anton holds an agent cell that changes connected data, or deletes, renames or moves an existing file, until the person who asked approves it. anton reads it through its own settings, so set it in the server's process environment. The server refuses to start with it on and remote turns (COWORK_TURN_BACKEND=remote), questions off (COWORK_ASK_USER_ENABLED=false), an anton without the write_approval setting, or a value anton's own settings read as off, such as one only in a .env file anton doesn't read. It refuses remote turns and questions off also when only anton's own settings read it as on, from a .env file anton reads that this server doesn't, such as ~/.cowork/.env with COWORK_HOME set elsewhere |
By field name (outside MindsHub mode only): every UserSettings field also reads the environment variable named after it, and a stored row wins over the variable. The router's reasoning effort is set this way, beside PLANNING_REASONING_EFFORT and CODING_REASONING_EFFORT:
| Variable | Default | Description |
|---|---|---|
MEMORY_ENABLED |
true |
Whether the agent reads and learns memory. The key is operator-pinned, so on a deployment that trusts identity headers a stored global row wins over the variable for every person. There false turns learned memory off, whether memory is kept per person or shared: every turn gets two empty learned tiers and the operator's managed rules, runs with memory mode off and no background memory passes, and writes nothing; the Memories listing is empty, and a change or delete of a slot under /api/v1/memory/ answers 409; and stored memory stays on disk, unread, until memory goes back on. Elsewhere (single-user mode, MindsHub mode) false only turns anton's memory mode off, and stored memory is still read. |
ROUTER_MODEL |
the provider's default | The model that writes history summaries. Only for your own inference provider (Anthropic, OpenAI, Gemini or an OpenAI-compatible endpoint): on MindsHub, summaries already run on MindsHub's default model, and Settings no longer shows this choice once turns are routed by Jev or a decision model of your own. It applies to the router's provider (ROUTER_PROVIDER, by default the planning provider). The route gate runs on the provider's fast default whatever this says, except on openai_compatible, which has no default, so there it is the gate's model too. A model picked in Settings is stored and wins over the variable. |
ROUTER_REASONING_EFFORT |
unset | The router model's reasoning effort. The router's provider serves the route gate, first take and history summaries, so the effort reaches all three. On openai_compatible the gate runs ROUTER_MODEL. On other providers the gate runs the provider's default router model, so the effort is sent only while ROUTER_MODEL is that default. Unset or empty sends no effort, which a model that doesn't reason needs, because it refuses any effort. Choose an effort the model accepts on the selected API. Some models require Responses for function tools and reject none; Cowork sends the configured effort unchanged and never substitutes one after a refusal. The composer's per-task effort never applies to the router. |
In Docker/Lightsail deployments, the container also receives ANTON_MINDS_API_KEY, ANTON_OPENAI_API_KEY, etc. — these are consumed by the Anton agent library directly (not by cowork-server settings), and are injected by the provisioning lambda via cloud-init user-data.
On a deployment that trusts identity headers, an operator with a shell where cowork-server runs acts on one person's space with python -m cowork.ops.<module> <action>. Every action takes --organization-id, the deployment's id the proxy sends as X-Organization-Id, and --user-id, the person's sub the proxy sends as X-User-Id, except the memory actions where memory is shared, which act on the deployment's one store and refuse both, and connections list, which takes neither to enumerate all stored spaces. cowork-server works out the space from them with cowork.principal.personal_partition_id, as it does for the person's requests. Results print on stdout as plain text, and errors and log lines on stderr; a command that can't do what it was asked exits with status 1. Every connection share and unshare takes --registry-stdout: it then writes nothing, prints the new registry on stdout (exactly the bytes it would have written) and its message on stderr, so a caller can check the registry and write it itself. A skill share and unshare take no such flag and no --managed-dir: they change the shared-skill list as the person's own Skills page does, so they run inside Cowork's container as the user Cowork runs as, and refuse root, whose shared/ folder or lock file would stop Cowork's own writes.
| Command | What it does |
|---|---|
python -m cowork.ops.memory print ... |
Where memory is kept per person (memory in COWORK_PERSONAL_RESOURCES), prints what the person's agent learned, as their Memories page lists it: the global tier, then each project tier, with the folder each is in, and refuses a call without both --organization-id and --user-id. Where memory is shared, takes neither (either one is refused) and prints the deployment's one store: the global tier, then each project tier, titled Project <project id>. It ends with Nothing learned. when every tier is empty. While MEMORY_ENABLED is false it prints only that memory is off and stored memory is kept, unread, until memory goes back on. |
python -m cowork.ops.memory clear ... |
Where memory is kept per person, removes everything the person's agent learned, in every project of theirs (cowork.services.memory.reset_person_memory). Where memory is shared, takes neither person argument (either one is refused, so a clear meant for one person never clears everyone's) and removes the shared store's global slots and every project tier in it (cowork.services.memory.clear_deployment_memory), and no person's tier or managed file. A link where the store's projects/ folder should be stops print and clear with an error before anything is read or removed. It can't be undone. While MEMORY_ENABLED is false it refuses and removes nothing, so stored memory stays as it is. The person arguments are checked first in every case. |
python -m cowork.ops.skills share ... --skill <slug> |
Shares the person's skill <slug> with everyone, read-only, on their behalf: adds it to <COWORK_SHARED_DIR>/shared/skills.json. Nothing is copied: everyone else reads the skill live from the person's store, and the person still edits, disables and deletes it, but can't rename it until it is hidden. They can hide it again from their Skills page. It runs the checks and prints the refusal texts of a share from the Skills page (Sharing one skill with everyone), plus "User <sub> has no skill named <slug>." for a skill the person doesn't have and "Skill <slug> of user <sub> is already shared." for one already shared. |
python -m cowork.ops.skills unshare ... --skill <slug> |
Removes the person's entry for <slug> from <COWORK_SHARED_DIR>/shared/skills.json, whether or not the skill still exists, so it also clears the entry of an account the identity provider no longer holds. The person's skill stays theirs, in their store. Refused when the person shares no skill by that name. |
python -m cowork.ops.skills snapshot ... [--project <name>] |
Lists the skills the person's agent sees in a project (general unless named), the skills others share with them included, with where each comes from (own, host, managed, shared or anton), and prints one sha256: the sha256 of the lines <sha256 of the file> <slug>/<path in the skill>, one per file of each skill, sorted by slug and path, as sha256sum prints them. Every file the agent can read counts, helper scripts and reference files included; a skill's stats.json, hidden files and links don't. A shared skill's lines hash the files in its creator's store, so the digest changes when its creator edits it. |
python -m cowork.ops.connections list [--organization-id <id> --user-id <sub>] |
Read-only JSON inventory of stored connection metadata on a POSIX trusted-identity local deployment. Without the pair, includes every stored partition, even deleted accounts; with it, selects that partition plus managed and base stores, and opens no other partition's records. Reports engine, name, creator when recorded, partition, origin, visibility and override notes. No credentials, hosts or fields are printed. Inactive stores and stale shares remain visible. Linked, non-regular, malformed or oversized records it opens refuse the whole command without printing record contents. |
python -m cowork.ops.connections share ... --name <name> [--engine <engine>] --managed-dir <folder> |
Shares the person's saved connection named <name> with everyone, read-only: adds it to <folder>/shared/connections.json, where <folder> is the managed folder, writable by the caller. Nothing is copied: everyone else reads the connection live from the person's vault, and the person still changes it. Names are unique per engine only: one Google account connected to Gmail and to Calendar saves two connections under one name, and --engine picks one. Refused where COWORK_PERSONAL_RESOURCES leaves connections shared, when the person has no connection by that name, or more than one and no --engine (the message names --engine), when it is already shared, when someone else already shares one of that engine and name, when the managed data-vault/ holds one, and when the registry is malformed, which the command never rewrites. |
python -m cowork.ops.connections unshare ... --name <name> [--engine <engine>] --managed-dir <folder> |
Removes the person's connection named <name> from <folder>/shared/connections.json. The person's connection stays theirs, in their vault. Refused when the person shares none by that name, and when they share it for more than one engine and no --engine is given. |
The inventory describes stored connections, not one person's picker. visibility is private, shared, operator, inactive or stale. operator marks a base-vault record while connections are personal and COWORK_IDENTITY_ENFORCE is audit: no person reads it, but a request with no identity pair is the operator view, which reads and changes it. inactive means nothing reads the store: the base vault under enforce, and personal vaults while connections are shared. shared_as lists the registry's engine/name pairs that reach each stored connection. Names such as sales.db and sales_db reach the same vault file, so an older share remains active after such an update; registry-name-alias marks this case. The notes identify managed-prefix overrides, same-vault and shared-layer prefix collisions, recipient-owned overrides and duplicate registry entries, using the registered names for shared-layer precedence. Managed connections take priority over personal and shared entries. Two names that normalize to the same DS_* prefix can conflict even when their spellings differ.
Adding connections to COWORK_PERSONAL_RESOURCES switches readers to personal vaults. It leaves old base-vault records stored but unread by testers; under audit, the operator view still reads them. Preserve explicitly chosen team connections in the managed vault, or have their owners recreate and share them. Do not copy private connections to every person. Enable projects too and preserve needed shared-project files before changing that namespace. The new startup refusal deliberately prevents private connections alongside legacy shared executable state; removing that guard does not make such a configuration safe.
The inventory reads up to 1 MiB per record or registry and follows no partition, vault or record links. It never creates a missing store. Run it between connector changes for a stable operator review; it is not a transactional snapshot. A missing creator stamp is null, while the stored partition still identifies the space. One person's listing reads only that partition's records. For other people's shares it reads only the vault's file names, and a vault holding a link or a file with two names shares nothing, as at runtime. So a bad record elsewhere doesn't fail it, and the full listing is the check for every store. The command does not infer current identity-provider membership.
- docs/DESIGN.md — Architectural overview and design decisions
- docs/MIGRATION.md — Migration guide from the legacy server
- docs/MIGRATION_PROGRESS.md — Migration status tracker
See LICENSE.
See structured logging for defaults, troubleshooting overrides, retained evidence streams and rollout requirements. Shared-memory startup warnings state whether turns learn into the shared store or leave it unread, without logging its private path.