feat: api tool agents with settled writes, complete tool calls and redacted records

deepgym/docs/agents.md

@@ -1,7 +1,7 @@
 # Agent harnesses
 
 `deepgym.agents` supplies `AnthropicComputerAgent`, `OpenAIComputerAgent`,
-`ClaudeCodeAgent`, and `CodexAgent`. Each implements
+`AnthropicToolAgent`, `OpenAIToolAgent`, `ClaudeCodeAgent`, and `CodexAgent`. Each implements
 `run(session, initial, *, task, budget) -> AgentResult`. Start the Session first;
 the caller owns grading and cleanup. `DeepGym.run_episode(env, agent, budget=...)`
 does that lifecycle and returns `(AgentResult, RunResult)`. Its existing callable
@@ -54,6 +54,90 @@ structured provider codes use `provider_error`; message-only or unknown-code err
 remain `error`, regardless of their prose. Evidence and fixture limits are recorded
 in `tests/agents/fixtures/README.md`.
 
+## API tool agents
+
+`AnthropicToolAgent` and `OpenAIToolAgent` give the model only the environment's
+declared API tools plus a `done` tool. No hosted or server-side tool (web search, code
+execution, browsing) is requested, and provider keys stay on the host. An environment
+declares its tools as trusted configuration:
+
+```python
+from deepgym.models import MultiTurnEnvironment
+from deepgym.tools import HttpRoute, HttpTool
+
+env = MultiTurnEnvironment(
+    task='your task',
+    snapshot='your-app-snapshot',
+    root_service_start='your trusted service start command',
+    tools=(
+        HttpTool(
+            name='bugzilla',
+            description='Bugzilla 5.0 REST API. Put the API docs the task needs here.',
+            base_url='http://127.0.0.1:8080',
+            routes=(
+                HttpRoute(method='GET', path='/rest/bug'),
+                HttpRoute(method='GET', path='/rest/bug/{id}/comment'),
+                HttpRoute(method='POST', path='/rest/bug/{id}/comment'),
+            ),
+            headers={'X-BUGZILLA-API-KEY': app_key},
+        ),
+    ),
+    allowed_actions=frozenset({'tool', 'done'}),
+    max_steps=None,
+)
+```
+
+`base_url` must be on a loopback host. A call names a method and a path that fits a
+declared route; `{name}` matches one path segment and `{name:path}` also takes an
+encoded `/` (GitLab project paths and file paths). The model supplies only the method,
+path, query and JSON body. Each call is a Session `tool` action: the host checks it,
+then a constant runner sends it with http.client from inside the sandbox, as the agent
+user in the offline network namespace. Its request arrives on stdin from a root-only
+file. Redirects are returned, never followed.
+Header values are credentials: masked in dumps, environment fingerprints and validation
+errors, and every tool's are redacted from every call's record and result, including
+forms an app may echo (escaped, percent-encoded, base64, a Basic pair or password, any
+letter case) and the cookies it sets. A body that isn't UTF-8 comes back base64-encoded,
+with its bytes redacted. Header values must be ASCII, a Basic pair inside one too, so the
+bytes an app sees match what redaction looks for. Pattern redaction cannot catch every
+transformation, so declare only routes that never return credentials. An environment with tools allows exactly
+`tool` and `done`. Tools are trusted configuration, so an inline environment sent to the
+sessions API cannot declare any; a registered environment can, in code or in its
+folder's `metadata.json` (`tools`, `allowed_actions`, `root_tool_settle`).
+
+A route allows exactly a method and a path; the body is checked only as JSON, not
+against the app's operations. A declared GraphQL endpoint (`POST /graphql`) or any
+other endpoint that dispatches on its body (a JSON-RPC method, a batch call) therefore
+allows every operation the tool's credentials permit. Give such a tool an app account
+whose role allows only what the task needs.
+
+The observation, which the model gets back whole, records the full request, the full
+response (status, every header, the body) and timing. A response cut short keeps every
+byte that arrived, marked `"incomplete": true`, and counts as a failed call. A write
+whose response was lost, cut short or stopped by the step timeout is marked
+`"uncertain": true` and is never retried.
+
+An app may still change after it answers a write (a queued job, say) or after its runner
+was stopped, so the session holds every write it sends as unsettled; only a call that
+never connected is cleared. Before the final verifier runs, the environment's trusted
+`root_tool_settle` command (root, offline namespace, `setup_timeout`) must settle them:
+exit 0 only once the app has applied or abandoned every request it received, for example
+by stopping its server or waiting for its queue to drain. Without it, or if it fails,
+the episode gets no verdict (`error_type='unsettled_write'`) whatever its score, and
+counts toward the masked rate. Writes leave no settled state between steps, so an
+environment whose tools have any route other than `GET` cannot have a step verifier.
+
+Only the `done` tool ends the episode; app output that reads `DONE` does not. A reply
+without tool calls gets a reminder. Defaults are uncapped: maximum effort, Anthropic's
+model-maximum `max_tokens`, no OpenAI output limit, and only the turn, time and USD
+limits the caller's `Budget` sets. `AgentResult.messages` holds every model message, full
+responses with their usage included. A provider response that never completed is kept
+too, as what arrived marked `"status": "incomplete"`, and its tool calls never run. Nor
+do the calls of an Anthropic response that stopped for any reason but tool use
+(`max_tokens` mid-call, say): the run ends with an error, and the response is kept with
+its `stop_reason`, usage and `input_json`, each call's input exactly as it streamed. Under
+`dg.trace`, each step also carries the messages that produced it.
+
 ## CLI isolation and its exposure
 
 CLIs must already be installed in the sandbox image; the library installs nothing.

deepgym/src/deepgym/_http_request.py

@@ -0,0 +1,104 @@
+"""Send one HTTP request, read as JSON from stdin, and print what happened as one JSON line.
+
+The session runs this inside the sandbox (``python3 -I``) as the agent user, in the offline
+network namespace, for an environment's declared API tool (see :mod:`deepgym.tools`). Root
+pipes the request in from a file only root can read, so the app credentials it carries
+never sit where the agent user could read them. The host built and checked every part of
+the request; this only sends it with http.client, which parses no URL, uses no proxy and
+follows no redirect: the request line carries exactly the target the host built, to the
+host and port it named. Stdlib only, as the image's python3 runs it.
+"""
+
+import base64
+import http.client
+import json
+import ssl
+import sys
+import time
+
+_READ = 1 << 16
+"""Bytes asked for per read; reads repeat until the body ends, so this bounds nothing."""
+
+
+def _answer(response: http.client.HTTPResponse, raw: bytes) -> dict:
+    """The record of *response*: status, every header, and *raw* as its body."""
+    answer: dict[str, object] = {
+        'status': response.status,
+        'reason': response.reason,
+        'headers': [list(header) for header in response.getheaders()],
+    }
+    try:
+        answer['body'] = raw.decode('utf-8')
+    except UnicodeDecodeError:
+        answer['body_base64'] = base64.b64encode(raw).decode('ascii')
+    return answer
+
+
+def _read(response: http.client.HTTPResponse, received: bytearray) -> None:
+    """Read *response*'s body into *received* as it arrives, so a cut keeps every byte before it.
+
+    Raises:
+        http.client.IncompleteRead: If the connection ended before the declared length.
+        OSError: If the connection failed mid-body (a reset, say).
+    """
+    chunk = response.read1(_READ)
+    while chunk:
+        received += chunk
+        chunk = response.read1(_READ)
+    if response.length:  # read1 ends quietly at a close; the length shows what never came
+        raise http.client.IncompleteRead(bytes(received), response.length)
+
+
+def main() -> int:
+    """Send the request on stdin; print its record; return 0 if a whole response came back.
+
+    A response cut short keeps what arrived (status, headers, every body byte received),
+    marked ``incomplete``.
+    """
+    spec = json.load(sys.stdin)
+    if spec['scheme'] == 'https':
+        context = ssl.create_default_context()
+        connection = http.client.HTTPSConnection(spec['host'], spec['port'], context=context)
+    else:
+        connection = http.client.HTTPConnection(spec['host'], spec['port'])
+    record: dict[str, object] = {'request': spec['record']}
+    status = 1
+    started = time.monotonic()
+    try:
+        connection.connect()
+    except OSError as exc:  # nothing was sent
+        record['error'] = f'No connection: {type(exc).__name__}: {exc}'
+        record['uncertain'] = False
+    else:
+        body = spec['body']
+        response = None
+        received = bytearray()
+        try:
+            connection.request(
+                spec['method'],
+                spec['target'],
+                None if body is None else body.encode('utf-8'),
+                spec['headers'],
+            )
+            response = connection.getresponse()
+            _read(response, received)
+        except (OSError, http.client.HTTPException) as exc:
+            # The request may have reached the app: a write is then neither known done nor not.
+            kind = 'No' if response is None else 'Incomplete'
+            record['error'] = f'{kind} response: {type(exc).__name__}: {exc}'
+            record['uncertain'] = spec['mutation']
+            if response is not None:
+                record['response'] = {**_answer(response, bytes(received)), 'incomplete': True}
+        else:
+            record['response'] = _answer(response, bytes(received))
+            status = 0
+    finally:
+        connection.close()
+    record['elapsed_ms'] = (time.monotonic() - started) * 1000
+    # UTF-8 bytes whatever the sandbox locale, on one line: JSON escapes every newline.
+    sys.stdout.buffer.write(json.dumps(record, ensure_ascii=False).encode('utf-8') + b'\n')
+    return status
+
+
+if __name__ == '__main__':
+    sys.exit(main())

deepgym/src/deepgym/agents/__init__.py

@@ -1,5 +1,6 @@
 """Run provider agents over DeepGym sessions; SDKs are optional and imported on run."""
 
+from deepgym.agents.api import AnthropicToolAgent, OpenAIToolAgent
 from deepgym.agents.base import Agent, AgentResult, Budget, Cost, TokenPrices, Usage
 from deepgym.agents.cli import ClaudeCodeAgent, CodexAgent
 from deepgym.agents.computer import AnthropicComputerAgent, OpenAIComputerAgent
@@ -9,11 +10,13 @@
     'Agent',
     'AgentResult',
     'AnthropicComputerAgent',
+    'AnthropicToolAgent',
     'Budget',
     'ClaudeCodeAgent',
     'CodexAgent',
     'Cost',
     'OpenAIComputerAgent',
+    'OpenAIToolAgent',
     'TokenPrices',
     'Usage',
     'SUPPORTED_CLIS',