Repository navigation
LOU-J: Cross-harness spec & flagship pipeline - #11
Merged
Merged
Conversation
Extend AgentSpec (LOU-H9) with optional, backward-compatible policy/ triggers fields for the full cross-harness spec, and add generateClaudeCodeSkill() producing a real Claude Code SKILL.md (YAML frontmatter with name/description, full prompt and every tool in the Markdown body). Adds a shared fixture spec reused by the other LOU-J generators. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Add generateCodexConfig() producing a Codex-style agent-config JSON
file ({name, instructions, model:{provider,name}, tools, policy?,
triggers?}). Note: Codex's exact documented config schema was not
verifiable from this repo/session, so the shape follows the ticket's
own illustrative baseline (instructions field for the prompt) -
flagged in the file's doc comment as best-effort, not confirmed
against Codex's real spec.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Add generatePiSkill() producing a Pi skill YAML file, reusing the existing 'yaml' dependency (already used by loadSpec.ts) for stringifying rather than adding a new YAML dependency. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Add examples/ops-pipeline/monitor.ts: ErrorSignal type, an in-memory Set<string> dedup by signature, and handleErrorSignal() which calls the real static AgentExecutor.execute() exactly once per new signature with a prompt built by buildMonitorPrompt(). Wires a POST /webhook HTTP listener reusing src/cli/dev.ts's http.createServer pattern (localhost-default binding, 1MB request body cap). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Add src/tools/built-in/slack.ts following the ToolDescriptor pattern from http.ts/github.ts: builds a Slack Block Kit payload with one actions block containing a "Fix it" button (action_id 'fix_it', value = approvalId) and POSTs it to a webhook URL read from SLACK_WEBHOOK_URL (LOU-F8 env-var convention). Add examples/ops-pipeline/slackInteractions.ts: a POST /slack/interactions route that parses Slack's interaction callback, extracts the approvalId from the clicked button's value, and calls the REAL LOU-C6 resumeAfterApproval() with it - verified by a test that spies on resumeAfterApproval and confirms the deferred tool actually executes, proving this genuinely routes through the real approval gate rather than a bypass. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Add examples/ops-pipeline/fixer.ts: FixRequest type, a fixer AgentConfig/prompt instructing diagnosis from logs and a unified diff output, runFixer() calling the real static AgentExecutor.execute() and extracting a diff block from the response (fenced ```diff, a diff-shaped plain fenced block, or a raw unified-diff marker fallback), throwing typed EmptyPatchError on an empty/unparseable patch. Also createFixerDelegateTool(), wrapping the REAL LOU-D1 createDelegateTool() so the monitor agent can delegate a FixRequest to the fixer agent via ordinary tool-calling and get back an extracted patch the same way runFixer() does directly. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Add examples/ops-pipeline/guardedPr.ts: handleFixerPatch() runs the REAL LOU-E12 runGuardrails() (secret-scan + diff-size-cap by default) against the fixer's patch and awaits it to completion BEFORE the GitHub PR-creation call is even considered - a failing guardrail result means the GitHub tool is never called at all; instead the LOU-J5 Slack tool is called with a message naming which guardrail(s) failed, reusing the original approvalId/channel. A passing result calls the REAL LOU-E13/E14-hardened GitHub tool's PR-creation function exactly once, carrying the patch in the PR body (the only field a PR-creation-scoped token can use to hold diff content). Tests confirm: a patch matching secretScanGuardrail's real patterns, and an oversized patch, both never reach a spied GitHub PR-creation call and instead produce exactly one spied Slack call; a clean patch produces exactly one GitHub call with the patch content and zero Slack calls; and an explicit ordering test proves the guardrail check runs strictly before the GitHub call (sequential, not parallel/best-effort). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Add examples/ops-pipeline/index.ts wiring, in one process entry point: the LOU-J4 monitor webhook listener, a POST /slack/interactions route (LOU-J5), the LOU-J6 fixer (delegated to via a REAL createDelegateTool() flagged needsApproval:true), and the LOU-J7 guardrail-gated PR wiring. The delegate tool's approval requirement is the load-bearing safety property here: the fixer agent's AgentExecutor.execute() call only happens from inside that tool's execute(), which only runs after resumeAfterApproval() (LOU-C6) resolves a real PendingApproval - there is no path that runs the fixer without that approval round-trip. Add examples/ops-pipeline/mocks/ (mock Grafana/Datadog sender, mock Slack API, mock GitHub API - same call shapes as the real tools) and a deterministic zero-network demoProvider.ts, so the whole example runs with zero external network access by default. Add pipeline:demo / pipeline:demo:trigger npm scripts and a README with numbered setup steps. Verified genuinely end-to-end from a fresh `npm run pipeline:demo` process plus `npm run pipeline:demo:trigger` in a second terminal, and via a committed integration test (index.test.ts) that drives the full dedup -> approval-gate -> Slack "Fix it" -> guardrail-gated GitHub PR sequence and asserts each step's call counts. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Add examples/ops-pipeline/pipeline.eval.ts with two complementary checks: 1. A real defineEval() (LOU-G1) call scored with the real toolCallOrder() scorer (LOU-G4) against the monitor agent's own single AgentExecutor.execute() step, asserting it calls the real 'delegate_to_fixer' tool name used by index.ts's wiring. 2. A direct, spy-instrumented assertion of the full cross-stage call order (monitor's Slack alert -> both real guardrails -> GitHub PR creation) - chosen because most of this pipeline's calls are plain ToolDescriptor invocations outside AgentExecutor's own tool-calling loop, so they never populate one single ExecutionResult.toolCalls array for toolCallOrder() to score directly; a deep-equals over the recorded order is the more honest representation of what genuinely happened. Also threads an optional `guardrails` override through startOpsPipeline() so this eval (and any other test) can observe guardrail execution order without altering index.ts's own default guardrails. Verified the eval is a real assertion, not a no-op: ran it passing, then temporarily reordered EXPECTED_ORDER and confirmed it correctly failed with a clear diff, then restored the correct order. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
examples/README.test.ts asserts the gallery index has exactly one heading per example directory - LOU-J8's new examples/ops-pipeline/ directory needed a matching entry. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
- fixer.ts: document runFixer()'s safety contract explicitly (it produces a patch candidate only — no approval gate, no guardrail check — so a future caller wiring it into a new entry point doesn't accidentally bypass the approval/guardrail gates the shipped pipeline relies on). - mockSlackTool.ts: log the approvalId and alert message to the console when the demo posts an alert, since the README's "click Fix it" step otherwise has no way to surface the approvalId in a live terminal run (the mock has no real Slack UI to click a button in). QA found no exploitable bypass in either area — these are documentation/ observability hardening, not fixes to an actual gap in the safety gates. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Implements all 9 tickets from LOU-J (Phase 4, final epic) — cross-harness spec generators and a complete, working flagship pipeline proving every load-bearing primitive built across the whole backlog works together:
generateClaudeCodeSkill(),generateCodexConfig(),generatePiSkill()— each consumes the sameAgentSpec(extended withpolicy/triggers) and emits a native artifact for that harness.examples/ops-pipeline/monitor.ts— a webhook listener that deduplicates error signatures and triggers the realAgentExecutor.execute()exactly once per new signature.src/tools/built-in/slack.ts— a Slack alert tool with a "Fix it" interactive button, plusslackInteractions.tsrouting a button click through the real, unmodifiedresumeAfterApproval()from LOU-C.fixer.ts— a fixer agent delegated to via LOU-D's real delegation primitive, producing a patch candidate from error context and logs.guardedPr.ts— routes every fixer patch through the real, unmodifiedrunGuardrails()/secretScanGuardrail/createDiffSizeGuardrailfrom LOU-E, awaited to completion before any GitHub call is even considered.examples/ops-pipeline/index.ts— the full demo wired against zero-network mocks, with a README andpipeline:demoscript.pipeline.eval.ts— a real, non-no-op eval asserting the pipeline's cross-stage tool-call order.This epic's safety properties were adversarially tested, not just unit-tested
This is explicitly the highest-blast-radius epic in the roadmap — "the fixer agent must never run without approval, and must never open a PR without passing guardrails; both gates are non-negotiable." QA didn't just run the implementer's tests; it actively tried to break both gates:
approvalIdagainst the real/slack/interactionsroute — cleanly rejected with the exact LOU-C6 "unknown or already resolved" error, zero GitHub calls. Audited for a second, ungated entry point to the fixer (the exact bug class found in LOU-F's sandbox review) — foundrunFixer()is exported and could be called directly, but confirmed the shipped wiring never does so; hardened it with an explicit safety-contract doc comment as defense against future misuse.await runGuardrails(...)resolves, never parallel.No bypass was found in either gate. One documentation gap was found and fixed: the demo's mock Slack tool didn't log the
approvalIda user needs to complete the README's manual "click Fix it" step in a live terminal run — now logged.Validation results
npx tsc --noEmit: clean.npx vitest run: 723 passing, 3 skipped. Remaining failures are pre-existing and unrelated: 3 provider-suite tests missing optional peer deps, andpackages/create-loushy-agent's tests needing their ownnpm install/build (confirmed via QA as a build-order artifact, not a regression) plus one pre-existing packaging gap (its E2E test depends on the SDK being published to npm, which it isn't pre-release).npm run pipeline:demo+ the synthetic-error trigger script actually works end-to-end against the mocks.🤖 Generated with Claude Code