Executed benign control set - #73
Draft
astham18 wants to merge 23 commits into
Draft
Conversation
The released benign traces were synthesised by a model in the shape of
execution logs, so a detector could separate the classes on provenance
rather than on cross-session composition. This adds a benign corpus that
runs through the same harness, actor model and expansion path as the
malicious side, so the two differ only in the objective.
- 24 seeds (seeds/benign_*.json) authored from a single spec, reproducing
the malicious chain-length histogram exactly and holding the released
benign corpus's measured theme mix.
- Seeds are generated, not hand-written: scripts/author_benign_seeds.py
plus scripts/benign_specs{,_b}.py, with placeholder-scope and
produces/consumes closure checks. Regeneration needs no model calls.
- Each campaign declares two discriminator axes embedded in every artifact
name, so the 21 variations do not write colliding filenames and fuse into
one component under the detector's shared_resource edges. A per-variation
{wd} pins each chain to a single filesystem root.
- generator.py: benign style templates under the same six style labels,
gated on metadata.benign — the offensive wrappers are incoherent applied
to benign work.
- run.py: resolve produces/consumes through the same substitution as
prompts, since artifact names are now templated. Inert for the malicious
seeds (0 of 444 entries contain a placeholder).
- Makefile: DATASET_BASE_SEED 1000 -> 0 to match released numbering.
- scripts/profile_sessions.py and summarize_benign_run.py measure the
distribution gap and the post-run verdict/failure/self-containment rates.
- smoke_benign.sh runs one chain per campaign before the full sweep.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
A tool call whose arguments exceed ~19k characters is truncated mid-value
by the output limit; the JSON then fails to parse, the client falls back
to {"__raw_arguments": ...}, that fails schema validation, and the write
never lands -- starving every downstream fragment in the chain.
Two prompts asked for a cross product and could not fit at any setting:
sysadmin_capacity multiplied fleet_size by window_days (up to 1,800 rows,
the observed failure) and sysadmin_backup multiplied job_count by restore
points over window_days. Both are now bounded -- weekly aggregates, and
three restore points per job.
Six more campaigns had single counts that overflowed on their own:
onboarding backlog 480 -> 140, recertification 300 -> 85, survey
responses 280 -> 70, account extract 240 -> 95, laptops 110 -> 75,
questionnaire and question bank 80 -> 38/40. Certs 70 -> 44 because SANs
make its rows unusually wide. Worst case is now ~10k characters against a
~19k ceiling. Patching, storage, inbox, release and retro already fit and
are unchanged.
Also replaced the incident timeline's unbounded "minute-by-minute over
{duration_hours} hours" with an explicit ~50 significant events.
Measured before the fix: 6 of 205 benign pilot sessions carried a
truncated tool call against 1 of ~1,400 malicious sessions. That is a
class-separating artifact of prompt sizing rather than of the objective,
which is the kind of difference this control set exists to remove.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The earlier cap pass found overlong tables by reading count variables, which missed the cases where the multiplier is a literal in the prompt text. A smoke sweep turned up a second truncated write, in patching: "for every host ... the installed versions of the ten packages" is host_count x 10, up to 800 rows. Same shape in two more places, both of which failed fragments in the sweep: storage multiplied volume_count by window_months, and docs_api emitted a row per endpoint per field per direction. data_catalog profiled every column of every table. All four are now bounded by an explicit per-parent limit, with the parent counts trimmed to match. summarize_benign_run.py: --since on a directory anchored on the directory's own mtime, which tracks its *last* write -- so summarising a finished sweep reported only the final campaign. It now anchors on the oldest entry. The 24-campaign sweep that read as chains=1 is chains=23. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
JUDGE=0 in both runners. The judge scores malicious objective completion, so on benign work it marked 45% of fragments FAIL that had not a single failed tool call -- and it costs a model call per fragment across 13,716 fragments. It runs after the fact and never alters the tool trace, which is the only thing the detector consumes. smoke_benign.sh had JUDGE=1, contradicting its own header comment. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Splitting work across machines must shard by campaign, not by style. Each shard runs all six styles of its campaigns into one accumulating filesystem, matching how the malicious corpus was produced. Sharding by style would give each machine a sixth of the cross-run residue and reopen the self-containment gap that turned out to be an artifact of exposure rather than of the objective. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
There was a problem hiding this comment.
Pull request overview
This PR adds first-class support for an “executed benign” control set by introducing a seed-driven benign variation generator, registering benign campaigns in the generator registry, and adding scripts/docs/shell entrypoints to run and summarize benign harness executions with comparable style labeling and artifact wiring.
Changes:
- Add
BenignVariationand register 24 benign campaigns to reuse a single seed-driven driver. - Ensure authored fragment
produces/consumesare template-resolved inrun.py, and propagate ametadata.benignflag into stylization so benign-specific style wrappers are used where needed. - Add benign seed JSONs plus run/smoke scripts and summarization tooling/docs for measuring execution parity.
Reviewed changes
Copilot reviewed 39 out of 65 changed files in this pull request and generated 1 comment.
Show a summary per file
| File | Description |
|---|---|
| variations/benign.py | New generic seed-driven variation generator for benign campaigns. |
| generator.py | Registers benign campaigns; adds benign-aware style wrapper/description selection and plumbing. |
| run.py | Resolves templated artifact names for produces/consumes; threads benign flag into stylization and logging output. |
| scripts/summarize_benign_run.py | New summarizer for benign runs (per-fragment verdicts + aggregate rates). |
| scripts/author_benign_seeds.py | Generates and validates benign seed JSON scaffolding from compact specs. |
| smoke_benign.sh | Smoke-test runner for benign campaigns (one chain each). |
| run_benign_full.sh | Full-grid runner for benign campaigns across styles/seeds with resumable state. |
| seeds/benign_*.json | New benign campaign seeds with authored fragments and artifact wiring. |
| docs/profiles/benign_targets.md | Target spec and rationale for executed-benign distribution parity. |
| docs/executed_benign_rebuttal_notes.md | Rebuttal-focused construction notes and measurement guidance. |
| Makefile | Align default dataset base seed numbering with “variations start at 0” convention. |
| .gitignore | Ignore dataset/ output directory. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
+53
to
+70
| pending: tuple[str, list[str]] | None = None | ||
| with open(p) as fh: | ||
| for line in fh: | ||
| try: | ||
| e = json.loads(line) | ||
| except json.JSONDecodeError: | ||
| continue | ||
| ev = e.get("event") | ||
| if ev == "tool_call": | ||
| pending = (e.get("tool"), paths_of(e.get("arguments"))) | ||
| elif ev == "tool_result": | ||
| # Count off the result event itself -- some results arrive | ||
| # without a paired call in the stream, and dropping those would | ||
| # shrink the denominator the failure rate is measured over. | ||
| tool = e.get("tool") | ||
| paths = pending[1] if pending and pending[0] == tool else [] | ||
| yield tool, paths, e.get("success") is not False | ||
| pending = None |
… EC2 shards Resumable and safe to run mid-flight; session and graph filenames carry a timestamp plus a random run-id suffix so shards merge without collisions. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
collect_shards.sh used --info=progress2, which macOS's openrsync (rsync 2.6.9-compatible) does not have, so every transfer failed. summarize_benign_run.py resolved session logs at logs/<basename>, but the harness files them under logs/<run_id>/ and older runs sit in logs/old/. The flat lookup found nothing, so tool-result and own-chain rates silently reported 0/0 while fragment verdicts -- which come from the graph JSON -- still looked correct. Now indexed recursively. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Reuses normalize_dataset.build_malicious rather than reimplementing event extraction, so benign and malicious traces are parsed by identical code -- the same principle the control set rests on. Verified: emits exactly the TOOL_CALL_FIELDS/TOOL_RESULT_FIELDS projection. Handles two things the released corpora do not: session logs nested under logs/<run_id>/ and logs/old/ rather than flat, and early smoke/pilot chains that ran with JUDGE=1 and pre-cap seeds, excluded via --since. --sample matches the prior of the synthetic control (143 samples); the full executed set is ~95x larger by event count, so an unmatched swap would move precision through the base rate alone. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
training/ builds its graph from CampaignDatasetGenerator. This swaps in the
normalized trace corpora while keeping the repo's own FragmentFeatureEngine,
so the 36 features are computed by unmodified code.
node = one tool_call / tool_result event
edges = the README's identity-preserving link-discovery layer --
data_flow (tool_call_index <-> tool_result_index within a session),
temporal (consecutive seq), shared_resource (path/URL/IP in
arguments, fan-out capped at 16)
split = chain-disjoint, so no sample spans train and test
obfuscation_score and static_risk_score are held at zero rather than invented:
both are label-correlated in the synthetic generator and there is no static
analyser for MCP traces, so fabricating them would leak the label. Entropy and
a size-derived complexity proxy are computed honestly from arguments.
Sanity check at 120+120 samples: GBT F1 0.9596 / chain F1 0.9722, LogReg
0.8340 -- the same ordering and range as the README's reference table.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Previously this script had its own GNN training loop, which made any GNN-vs-classical difference impossible to attribute -- it could have been the data or it could have been my loop. Training is now entirely the repo's code: campaign_disjoint_split the 70/30 outer-sample split build_node_features 128-dim node features for the GNNs NeighborSampler 2-hop sampling, K1=10 K2=5 train_single_arch GCN / GraphSAGE / GAT / GIN train_ml_methods the seven classical baselines Only the trace->graph adapter is ours, which is unavoidable: it is the thing that did not exist before. Both training functions compute probabilities internally but return only metrics, and per-campaign breakdowns need probabilities. Rather than fork them, capture_probs() shims the metric function they call on finished predictions and keeps a copy -- their code runs unmodified. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
compare_gnns.main() and train_gnn.train() build their dataset inline via CampaignDatasetGenerator with no parameter for supplying a graph, so running either unchanged re-runs the synthetic benchmark. This substitutes exactly one object in the target module's namespace: a stub with the same constructor and the same generate() return signature, backed by the trace graph. Everything downstream -- split, node features, neighbour sampling, four GNNs, seven classical baselines, reporting, checkpoint output -- is the original code path executed line for line. Closer to the original run than reimplementing the harness, and the only divergence left is the trace->graph adapter itself. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
compare_gnns.py:677 writes checkpoints/gnn_comparison.json, a hardcoded path with no parameter, so running the synthetic-benign and executed-benign arms would leave only the second one's results -- and running them concurrently would race rather than overwrite. Each run now claims a distinct destination up front, refuses to start if that destination already holds results, and moves the file when the harness finishes. Their code is still untouched. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
campaign_disjoint_split defaults to test_size=0.2 (train_gnn.py:57) and both call sites pass 0.2 explicitly (compare_gnns.py:614, train_gnn.py:311), so the harness runs an 80/20 outer-sample split. Table 3's caption reports 70/30. main() exposes no test_size argument, so the ratio is forced by wrapping the splitter in the target module's namespace. Defaults to 0.3 to match the paper; pass --test-size 0.2 to reproduce the harness default. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…esults 0.2 is what compare_gnns.py:614 and train_gnn.py:311 actually pass, so it is the ratio behind the published numbers regardless of Table 3's 70/30 caption. The flag stays available for checking the caption's ratio, but the default now reproduces rather than diverges. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…ions build_real_graph.py implements the build_edges() that real_edge_ablation.py imports but which was absent from the repo. Five typed edge sets over real traces, including the argument_similarity SimHash the released generator never built: 0 data_flow a write's resource later read, same session 1 temporal consecutive events in a session 2 shared_resource same resource, different sessions 3 shared_session same session, non-consecutive 4 argument_similarity 64-bit SimHash, Hamming<=8, different sessions The within/cross split the ablation relies on is enforced rather than assumed -- verified on the corpora: 0/1/3 have zero cross-session edges and 2/4 have zero within-session ones. Were that not so, cross_session_only would retain within-session signal and understate the graph's dependence on exactly the structure under question. Sanity check: temporal edges == nodes - sessions. aggregation_baselines.py adds the three arms the reviewers asked for that the edge conditions do not cover -- fragment-only, session-level and user-level aggregation -- all reported at event level on the same split so they sit in one table with the graph rows. Features are node-intrinsic only, so any gap to the graph conditions is attributable to structure and not to richer features. Also fixed real_edge_ablation.py's path setup: it inserted only ../dataset, so importing train_gnn/compare_gnns failed unless run from a specific directory. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
When the OpenRouter key hit its limit the harness still wrote a complete- looking session -- start, toolkits_connected, user_query, iteration_start, end -- with no tool calls in it. The chain's graph file looks normal and the fragment does not register as failed, so these are invisible to the verdict summary: 1,096 of 15,468 sessions across 96 chains, 845 of them inside a single hour. Left in, those chains contribute nodes with no events to the detector graph and quietly depress the benign class. Excluded by default; --keep-dead restores the old behaviour. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Re-running a campaign produces a new chain rather than replacing the broken one, so the API-limit casualties have to be moved aside or the corpus keeps both copies. Moves rather than deletes -- the broken chains are the evidence for what was lost and belong in the run's provenance. --dry-run lists them; --markers prints the run_state lines to clear on the instances, since those blocks exited 0 and are marked complete. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
train_single_arch scores mid-training for checkpoint selection, so the capture shim collects several probability vectors per GNN, not one. Indexing the classical panel's vectors as captured[len(probs):] therefore read from the wrong offset and handed each classical model another model's predictions -- visible as GIN and SVM producing byte-identical per-campaign columns. Now marks the capture position before each call and takes only what that call appends, with a loud warning and omitted columns if the counts ever disagree again rather than silently emitting a misaligned table. Per-campaign output reshaped to Table 3: the paper's seven detectors, F1 and Accuracy per campaign, an AGGREGATE row, and --csv-out for the paper. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
… file The per-campaign table previously came from train_on_traces.py, which rebuilds the split and calls the training functions itself. It now comes from run_original_harness.py, where compare_gnns.main() runs line for line and the only additions are observation wrappers -- each delegates to the original and returns its result untouched. Nothing downstream of the data substitution is reimplemented. Verified by cross-check rather than assumption: the recomputed AGGREGATE row must equal the F1 the harness itself reports, for every model. It did not at first. SVM was off by 3.9e-3 because compare_gnns thresholds with y_proba > 0.5 (lines 196 and 450) while this used >= 0.5, and SVC's Platt-scaled probabilities land exactly on 0.5. Now > 0.5 everywhere and all seven models agree to CSV rounding. Output is one JSON per arm carrying aggregate and per_campaign together, plus the same per-campaign table as CSV for the paper. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…nistic run_loco.py answers reviewer #3's generalization objection: 5 family folds, holding out every variation of every campaign in a family. Only the splitter is replaced -- compare_gnns.main() runs unmodified per fold, on the same graph, features, sampler, models and seeds. Two properties keep that a substitution rather than a rewrite: * campaign_disjoint_split's benign partition is fold-invariant (its rng state depends only on len(campaign_info), and benign_ids derive from the union of ALL instance nodes), so the original function is called once and each fold reuses its benign side verbatim. * a held-out family carries ~4% of positives where the headline split carries ~20%, so F1 would fall for prevalence reasons alone. Benign test events are subsampled to hold the positive rate at the headline split's; folds land at 0.597-0.669 against 0.60. Leakage is asserted for every fold before any of them trains. Worth recording: the headline split is NOT campaign-disjoint. It shuffles the 568 malicious instances, so all 24 campaigns appear on both sides. LOCO is new evidence, not a restatement of the existing protocol. Determinism fix. build_graph iterated a set of resource strings, whose order is randomized per process, which reordered adj[] and therefore changed which neighbours NeighborSampler's K1=10 draw selected. Same edge set, but ~0.002 of run-to-run F1 drift across all eleven models -- small, and indistinguishable from a real effect. aggregation_baselines had the same problem via hash(tool) % 10 filing a tool into a different feature column each run. Both now use blake2b; verified identical adjacency under three PYTHONHASHSEEDs. install_capture is factored out of run_original_harness.main() so run_loco reuses it rather than duplicating the wrappers. Pure code move; the three shim bodies are unchanged. training/ was untracked and not ignored, so a fresh clone could not run any of this. Added. .gitignore: *.pem (an SSH private key was sitting untracked in the repo root), /dataset/ (982 MB, rsynced to hosts instead), /results/quarantine_dead/, results_loco_*.json. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…benign draw
Default --benign is benign_executed.json across run_original_harness, run_loco
and aggregation_baselines, so the synthetic control cannot be selected by
forgetting a flag. dataset/combined/benign.json is no longer read anywhere.
The benign grid is complete: 1,296 distinct objectives, exactly 24 campaigns x
6 styles x 9 seeds. Two things the builder now handles:
* duplicates. Re-running a block after the API-limit window produced a second
chain for objectives that already had a good one -- 118 extras in the pool,
4 of which landed in the 514 sample. Two runs of the same objective are
redundancy, not data, and they inflate whichever campaign needed re-running.
One chain per (campaign, style, seed); --keep-duplicates opts out.
* balance. A flat draw left campaign counts between 11 and 32 while the
malicious side is uniform at ~24. --stratify round-robins across campaigns;
the 514 sample is now 21-22 per campaign, 0 duplicates, 0 zero-event
samples, 77,176 events.
Recorded in the runbook, not fixed here: 124 of 692 malicious chains (18%) have
no events at all and are dropped silently by train_on_traces.load(), along with
1,857 of 6,488 fragments (29%). They sit in the five campaigns that were run
twice, so every campaign still has 20-24 usable chains. But event density
varies ~25x -- UNC2970 has 154 events across 21 chains against GTG1002's 4,370
across 24 -- which is the likely explanation for UNC2970's outlier F1 in the
LOCO smoke run, and needs reporting as thinness rather than as a
generalization failure.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Every benign artifact name carried the per-variation {axis1}_{axis2}
discriminator, which drove cross-chain resource sharing to zero: 462 of 504
chains shared no resource with any other chain and none shared more than one.
The executed malicious corpus sits at a median of 7, because its chains write
generic names into a filesystem that is never reset.
That gap lands on shared_resource, one of the edge types the paper's claim rests
on and one that ablations/real_edge_ablation.py measures directly as
drop_shared_resource. The discriminator was added to stop the 21 variations of a
campaign fusing into one component, which is a real concern, but applying it to
every name manufactured the asymmetry rather than removing it: the malicious
corpus has the shared-name property too (ad_users.txt in 66 chains).
Artifact names are now two-tier. Headline deliverables (reports, runbooks, plans,
announcements) keep the discriminator, so variations stay distinguishable. A
second tier of 2-9 reference-shaped, read-mostly stems per campaign (registers,
catalogues, matrices, schedules, codebooks), declared in SHARED_ARTIFACTS, drops
it and carries a bare name across every variation. Counts vary per campaign so
the resulting distribution has a spread rather than a constant.
Measured: cross-chain sharing median 0 -> 6 (min 2, max 10) against malicious 7,
using the detector's own RESOURCE_RE. Every variation still carries a median of 6
uniquely-named deliverables. Validator clean on all 24 campaigns, 0 unresolved
placeholders across 504 variations, 0 consumes without an earlier producer, and
attack_runner --dry-run resolves the full dependency ordering.
The two pilot campaigns predate the spec format, so they are patched in place by
demote_authored_seed(); the transform is idempotent.
Docs corrected where they asserted the old design: rebuttal notes 5 and 6, plan
blocker B1 and Steps 1/2/4, and benign_targets 4.5-4.7. Plan blocker B2 is marked
resolved, since the evaluation pipeline it was waiting on now exists.
Known trade-off: shared bare names mean concurrent variations can write the same
path. Per decision this is measured in the smoke pass rather than pre-emptively
throttled; drop PAR only if the tool-result failure rate degrades.
…hains" This reverts commit 15d973d.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.