Repository navigation
Add Anthropic SDK tool loop integration - #323
cosmo-lgtm wants to merge 1 commit into
Conversation
Records a tool loop written directly against the anthropic Python SDK as AgenTrust Telemetry events (policy decisions, actions, usage, data flows) and signs one log-import TRACE record per finished run. The in-process writer is standard library only and fail-open; the recorder runs out of process. Metadata only; unfinished runs are not signed. Includes an offline example, tests (writer, recorder, interop against the released anthropic SDK with a mocked transport), a path-scoped conformance workflow running trace-tests at Level 0, and the regenerated index and Marketplace catalog. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
noah-ing
left a comment
noah-ing
left a comment
There was a problem hiding this comment.
Reviewed 5bdbad6ec05816030cbd6ba08a09b1c9cf3d05f6 in an isolated checkout. Thanks for keeping the offline SDK test and log-import limitations explicit.
The normal paths pass locally: 28 integration tests (including Anthropic 1.14.0 interop), 28 repository validation tests, the pinned Ruff gate, manifest/compatibility/catalog checks, the offline example and TRACE Level 0. The sdist/wheel build, fresh-wheel writer import, diff check and merge simulation against main 5612f397 also pass. Local execution was Python 3.12 on macOS; I did not reproduce the whole hosted OS/Python matrix.
Requesting changes for the two recorder-recovery cases below, both reproduced separately from the existing tests. One malformed timestamp prevents unrelated valid runs from being processed, and a replaced spool can silently lose a new run. Focused regressions for those cases would close the gaps.
Separately, the current link-check failure is an HTTP 503 in an unchanged README; I have not treated that as a defect introduced here.
| for run_id, rows in list(self.pending.items()): | ||
| ended = any(r["kind"] == "run.end" for r in rows) | ||
| last = max(int(r.get("end") or r.get("t") or 0) for r in rows) | ||
| if ended or final or now - last > self.idle_seconds * 1_000_000_000: | ||
| try: | ||
| self.record(rows, complete=ended) |
There was a problem hiding this comment.
Could the timestamp conversion be included in the per-run validation/error boundary? A run.start row with a valid run ID but t="not-a-timestamp", followed by an otherwise valid completed run, raises ValueError here before the try. The bad run remains pending, the valid run is never recorded, and a second poll() raises again. The existing malformed-run test misses this because its timestamps are valid. Please add that case and ensure rejecting the bad run still lets healthy runs progress.
| if self.spool.exists(): | ||
| if self.spool.stat().st_size < self.offset: # rotated or truncated | ||
| self.offset = 0 | ||
| with self.spool.open("rb") as fh: | ||
| fh.seek(self.offset) |
There was a problem hiding this comment.
The size check does not detect a spool replaced by a new file of equal or greater length. I recorded a valid 560-byte run, renamed its spool aside, and put a new valid 560-byte run at the same path. The next poll() returned [] and the second run was never recorded: the old offset was applied to the new file. Please track file identity/a rotation generation alongside the offset, or define an explicit safe rotation handoff, with an equal-or-larger replacement regression.
imran-siddique
left a comment
imran-siddique
left a comment
There was a problem hiding this comment.
Head is still 5bdbad6 and both of Noah's cases reproduce here. A run.start with t "not-a-timestamp" makes poll() raise ValueError, again from a fresh Recorder. A replacement spool of the same 1310 bytes returns [] and the new run is lost. CI steps pass on Python 3.12: 27 passed, 1 skipped, interop on anthropic 1.14.0, Level 0 PASS. The gate waits on maintainer approval; the links 503 is an unrelated README.
PR description: agentrust-telemetry 0.1.0a6 is on PyPI and the workflow installs it from there.
| governance events, and signs one TRACE v0.2 record per finished run. There is no | ||
| framework to hook into here, so the loop calls the recorder explicitly. | ||
|
|
||
| The code this package was generalised from runs in production in front of two |
There was a problem hiding this comment.
The production claim (two Slack agents) can't be checked from this repo. Link something a reader can open, or drop it.
| self.record(rows, complete=ended) | ||
| except Exception: | ||
| _log.exception("recording %s failed; its rows are dropped", run_id) | ||
| done.append(run_id) |
There was a problem hiding this comment.
done.append(run_id) runs after a failed record() too, so poll() lists a dropped run as recorded, against the docstring at line 184.
This branch has not been deployed
3 participants
Adds
integrations/anthropic-tool-loop/: an integration for agents built directly on theanthropicPython SDK (aclient.messages.createtool-use loop), which the index does not cover today.What it does
spool.py: a standard-library writer with a smallRunAPI (six calls in the loop). It appends metadata-only rows: model calls, tool calls, the caller's gate decision with a reason code, a sha256 digest of each tool input, and timings. It runs on Python 3.9+ and never raises into the agent's reply.recorder.py: an out-of-process recorder that turns those rows into AgenTrust Telemetry events (policy.decision,action.executed,usage.recordedwith anagent_runrollup,data_flow.observed), optional OpenTelemetry spans with the real timestamps, a chained evidence log, and one signed TRACE record per finished run (origin.kind: log-import).incompleteand is not signed.Validation
anthropic0.69.0, 1.12.1, 1.13.0 and 1.14.0.trace-tests verify --level 0: PASS.validate.yml, the pinned ruff gate, and actionlint on the new workflow.Notes from integrating (feedback, not part of the change)
contraindicated. A separate state for "the gate held" would make ledgers easier to read.finalize_traceneeds at least one policy decision, so a plain question-and-answer run cannot be sealed. We record the agent's toolset as a run-level decision.policy_decision: not_evaluated.usage.recordedcarries one timestamp, so the model call's start time lives only in the span.agentrust-telemetryis not on PyPI yet. This package pins0.1.0a6by commit, andintegration.yamlhas no field to declare it.Tier is for maintainers to set; it is listed as community.
🤖 Generated with Claude Code