Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 12 additions & 0 deletions CLAUDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -21,6 +21,9 @@ proof; this tool checks those proofs resolve in the tree.
- `internal/actrace/` — the library. All logic lives here.
- `actrace.go` — plan parser, status lifecycle, forward gate, backward
gate, `Run()` entrypoint (returns exit code int).
- `config.go` — the optional `.actrace.yml` loader (opt-in features).
- `resolver.go` — custom verify-method prefix→command hook (e.g. `edge:`).
- `journey.go` — the ADR-0077 `**Surface:**` tag + journey-proof gate.
- `reverse.go` — orphan gate, staleness report.
- `report.go` — result model, `--json` renderer.
- `matrix.go` — committed traceability matrix renderer.
Expand All @@ -40,6 +43,15 @@ usage/internal error). Never call `os.Exit` from the library.
`ui-e2e-realfd:`) is Atrium-specific. In a repo with no `ui/` dir the
spec index is empty and all `ui:` checks are no-ops. Do not remove this
logic — Atrium depends on it.
- The ADR-0077 journey-integrity gate (`config.go`, `resolver.go`,
`journey.go`) is **opt-in**: it fires only for a repo that ships an
`.actrace.yml`. Absent config → zero `Config` → every new check off, so
a repo like Airlock is unaffected. `journey_integrity: true` enables the
`**Surface:**` tag + journey-proof gate; `resolvers:` maps a
custom `verify:` prefix (e.g. `edge:`) to a command. The journey-proof
location/substance conventions (`ui/tests/e2e/*.realfd.spec.ts`,
`test/e2e/`, `synthetic`, `idpfake`, `waitForResponse`) are Atrium-shaped
like the `ui:` vocabulary and no-op where those paths don't exist.

## Things that will bite you

Expand Down
46 changes: 46 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -98,6 +98,49 @@ Plans declare a lifecycle on the `**Status:**` line:
Only `landed` plans are gated. A plan names its tests before they are
built, so enforcement waits until its implementation merges.

## Journey integrity (opt-in, ADR-0077)

An optional `.actrace.yml` at the repo root turns on extra checks. Absent,
the tool behaves exactly as above — a repo that ships no config (e.g.
Airlock) is unaffected.

```yaml
# .actrace.yml
journey_integrity: true # enable the Surface tag + journey-proof gate
resolvers:
"edge:": # a custom verify-method prefix
command: ["scripts/resolve-edge.sh"]
```

**Custom `verify:` methods (e.g. `edge:`).** A token like
`edge:agentloop->files.GetFile` is resolved by the command its prefix maps
to in `resolvers`. ac-trace runs that command with the raw token as one
final argv element — never through a shell — and reads the exit code: 0
means the proof holds, non-zero means it does not. A custom-prefix token
whose prefix has no configured resolver is a hard failure, never a silent
pass.

**The `**Surface:**` scenario tag.** With `journey_integrity: true`, each
scenario in a landed plan carries a `**Surface:** user-facing |
backend-foundation` field. A `user-facing` scenario must carry at least one
AC whose `verify:` cites a **journey proof**:

- a `ui-e2e-realfd:` Playwright spec (a `.realfd.spec.ts` under
`ui/tests/e2e/`) that asserts on a real server response
(`waitForResponse`), not on rendered DOM alone; or
- a Go test under `test/e2e/` that is a real-cluster run — `//go:build
e2e`, not `synthetic`, and not importing `idpfake`.

A mock spec, a unit test, a `test/e2e/` test that fakes its seam, or a
`demonstration` / `scenario` / `manual` method cannot satisfy a
`user-facing` scenario. A missing, unrecognised, or ambiguous tag is a hard
failure.

**Grandfathering.** A pre-existing `user-facing` scenario with no journey
proof yet opts out with `journey-ok: <reason>` on an AC's verify line — but
only when the reason cites a tracked issue (`#692` or an issues URL), so the
debt is visible and attributed.

## Wiring into a repo

Add to your `Taskfile.yml`:
Expand Down Expand Up @@ -169,6 +212,9 @@ Run `actrace --matrix` and commit the generated
`actrace.Run`, exits with its return code).
- `internal/actrace/actrace.go` — plan parser, status lifecycle, forward
gate, backward gate, `Run()` entrypoint.
- `internal/actrace/config.go` — the optional `.actrace.yml` loader.
- `internal/actrace/resolver.go` — custom verify-method prefix→command hook.
- `internal/actrace/journey.go` — the ADR-0077 Surface tag + journey-proof gate.
- `internal/actrace/reverse.go` — orphan gate, staleness report.
- `internal/actrace/report.go` — result model, `--json` renderer.
- `internal/actrace/matrix.go` — committed traceability matrix renderer.
Expand Down
6 changes: 4 additions & 2 deletions go.mod
Original file line number Diff line number Diff line change
Expand Up @@ -2,10 +2,12 @@ module github.com/stacklok/ac-trace

go 1.26

require github.com/stretchr/testify v1.11.1
require (
github.com/stretchr/testify v1.11.1
gopkg.in/yaml.v3 v3.0.1
)

require (
github.com/davecgh/go-spew v1.1.1 // indirect
github.com/pmezard/go-difflib v1.0.0 // indirect
gopkg.in/yaml.v3 v3.0.1 // indirect
)
35 changes: 25 additions & 10 deletions internal/actrace/actrace.go
Original file line number Diff line number Diff line change
Expand Up @@ -255,13 +255,19 @@ func Run(args []string) int {
return 2
}

cfg, err := loadConfig(".")
if err != nil {
fmt.Fprintln(os.Stderr, "loading .actrace.yml:", err)
return 2
}

plans, err := resolvePlans(*plan)
if err != nil {
fmt.Fprintln(os.Stderr, "globbing plans:", err)
return 2
}

totalFailures := runPlans(plans, index, fe, ground)
totalFailures := runPlans(plans, index, fe, ground, cfg)

// The backward gate runs once over the whole tree, not per plan: every
// landed TestADR_NNNN_* must name an ADR that exists and is not retired.
Expand Down Expand Up @@ -333,14 +339,14 @@ func resolvePlans(plan string) ([]string, error) {
// runPlans checks each plan and returns the total forward-check failures. The
// README and meta plans are skipped; an unreadable or mis-statused plan is a
// hard error that exits the process.
func runPlans(plans []string, index map[string]string, fe feIndex, ground grounding) int {
func runPlans(plans []string, index map[string]string, fe feIndex, ground grounding, cfg Config) int {
totalFailures := 0
for _, p := range plans {
base := filepath.Base(p)
if base == "README.md" || isMetaPlan(base) {
continue
}
failures, skipped, err := runPlan(p, index, fe, ground)
failures, skipped, err := runPlan(p, index, fe, ground, cfg)
if err != nil {
fmt.Fprintln(os.Stderr, "checking", p, ":", err)
os.Exit(2)
Expand Down Expand Up @@ -594,7 +600,9 @@ func loadGrounding(root string) (grounding, error) {
// number of failures. draft and in-progress plans are reported but never gate,
// so they return zero failures. A superseded plan is skipped (skipped=true). An
// unrecognised status is a hard error.
func runPlan(path string, index map[string]string, fe feIndex, ground grounding) (failures int, skipped bool, err error) {
func runPlan(
path string, index map[string]string, fe feIndex, ground grounding, cfg Config,
) (failures int, skipped bool, err error) {
b, err := os.ReadFile(path)
if err != nil {
return 0, false, err
Expand All @@ -609,18 +617,18 @@ func runPlan(path string, index map[string]string, fe feIndex, ground grounding)
}
acs := parseACs(lines)
if status == "landed" {
return checkLanded(path, acs, index, fe, ground), false, nil
return checkLanded(path, lines, acs, index, fe, ground, cfg), false, nil
}
// draft / in-progress: report only, never gate.
reportPlan(path, status, acs, lines, index, fe)
reportPlan(path, status, acs, lines, index, fe, cfg)
return 0, false, nil
}

// reportPlan prints a draft / in-progress plan's coverage without gating.
func reportPlan(path, status string, acs []acEntry, lines []string, index map[string]string, fe feIndex) {
func reportPlan(path, status string, acs []acEntry, lines []string, index map[string]string, fe feIndex, cfg Config) {
fmt.Printf("(%s, report-only) ", status)
if hasAnyVerify(acs) {
checkStructured(path, acs, index, fe)
checkStructured(path, acs, index, fe, cfg)
return
}
checkProse(path, lines, index)
Expand All @@ -629,7 +637,9 @@ func reportPlan(path, status string, acs []acEntry, lines []string, index map[st
// checkLanded runs the forward gate on a landed plan and returns the number of
// failures: zero structured ACs; an AC with no verify: field; a verify: test
// that does not resolve; or a bare-text ADR / Principle that grounds to nothing.
func checkLanded(path string, acs []acEntry, index map[string]string, fe feIndex, ground grounding) int {
func checkLanded(
path string, lines []string, acs []acEntry, index map[string]string, fe feIndex, ground grounding, cfg Config,
) int {
fmt.Printf("== %s == [landed]\n", strings.TrimPrefix(path, "./"))
if len(acs) == 0 {
fmt.Println(" ✗ landed plan has zero ACx.y criteria — must assert at least one")
Expand All @@ -652,9 +662,13 @@ func checkLanded(path string, acs []acEntry, index map[string]string, fe feIndex
}
failures += checkUIRefs(e, fe)
failures += checkRenderFromWireGate(e, fe)
failures += checkResolverRefs(e, cfg)
}
failures += checkGrounding(e, ground)
}
if cfg.JourneyIntegrity {
failures += checkJourneyIntegrity(lines, acs, fe, index)
}
fmt.Printf(" %d ACs · %d failure(s)\n", len(acs), failures)
return failures
}
Expand Down Expand Up @@ -820,7 +834,7 @@ func hasAnyVerify(acs []acEntry) bool {
// checkStructured evaluates a plan's verify: fields. Test-method criteria have
// every named Go test and `ui:<path>` reference checked; non-test methods are
// accepted.
func checkStructured(path string, acs []acEntry, index map[string]string, fe feIndex) int {
func checkStructured(path string, acs []acEntry, index map[string]string, fe feIndex, cfg Config) int {
fmt.Printf("== %s == [structured]\n", strings.TrimPrefix(path, "./"))
missing, unannotated, testACs, methodACs := 0, 0, 0, 0
for _, e := range acs {
Expand All @@ -839,6 +853,7 @@ func checkStructured(path string, acs []acEntry, index map[string]string, fe feI
}
}
missing += checkUIRefs(e, fe)
missing += checkResolverRefs(e, cfg)
}
}
fmt.Printf(" %d ACs · %d test-verified · %d method · %d unannotated · %d missing\n",
Expand Down
12 changes: 6 additions & 6 deletions internal/actrace/actrace_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -117,7 +117,7 @@ func TestCheckStructured_IgnoresProseAndCountsMissing(t *testing.T) {
index := map[string]string{"TestThing_Exists": "x_test.go"}

acs := parseACs(strings.Split(body, "\n"))
missing := checkStructured(plan, acs, index, nil)
missing := checkStructured(plan, acs, index, nil, Config{})
// Only TestThing_Missing counts; the prose `TestThing_Retired` is ignored.
assert.Equal(t, 1, missing)
}
Expand Down Expand Up @@ -289,7 +289,7 @@ func TestADR_0065_LandedPlanVerifyTestMustExist(t *testing.T) {
content := "# A plan\n\n**Status:** " + tc.status + ", 2026-06-17.\n\n" + tc.body
require.NoError(t, os.WriteFile(plan, []byte(content), 0o644))

fail, skip, err := runPlan(plan, index, nil, ground)
fail, skip, err := runPlan(plan, index, nil, ground, Config{})
if tc.wantStatusErr {
require.Error(t, err, "an unrecognised status must be a hard error")
return
Expand Down Expand Up @@ -694,7 +694,7 @@ func TestCheckLanded_ResolvesUITestReference(t *testing.T) {
// the resolution failures. (AC1.2 has no resolvable FE proof so the gate
// does not fire for it; AC1.3 is ambiguous so the gate also does not
// fire — checkRenderFromWireGate skips cites that don't resolve to one.)
assert.Equal(t, 4, checkLanded("fe.md", acs, map[string]string{}, fe, ground))
assert.Equal(t, 4, checkLanded("fe.md", nil, acs, map[string]string{}, fe, ground, Config{}))
}

// TestADR_0065_RenderFromWireGateDefaultDeny pins the frontend half of the
Expand Down Expand Up @@ -771,7 +771,7 @@ func TestADR_0065_RenderFromWireGateDefaultDeny(t *testing.T) {
t.Parallel()
index := map[string]string{"TestThing_Exists": "x_test.go"}
acs := []acEntry{{id: "AC1.1", verify: tc.verify, hasVerify: true, body: "AC1.1: a behaviour"}}
got := checkLanded("p.md", acs, index, fe, ground)
got := checkLanded("p.md", nil, acs, index, fe, ground, Config{})
assert.Equal(t, tc.wantFail, got > 0, "strict-failure decision")
})
}
Expand Down Expand Up @@ -820,7 +820,7 @@ func TestADR_0065_RenderFromWireGateMislabel(t *testing.T) {
t.Run(tc.name, func(t *testing.T) {
t.Parallel()
acs := []acEntry{{id: "AC1.1", verify: tc.verify, hasVerify: true, body: "AC1.1: a behaviour"}}
got := checkLanded("p.md", acs, map[string]string{}, fe, ground)
got := checkLanded("p.md", nil, acs, map[string]string{}, fe, ground, Config{})
assert.Equal(t, tc.wantFail, got > 0, "strict-failure decision")
})
}
Expand Down Expand Up @@ -907,7 +907,7 @@ func TestADR_0065_RealfdProofType(t *testing.T) { //nolint:paralleltest // captu
acs := []acEntry{{id: "AC1.1", verify: tc.verify, hasVerify: true, body: "AC1.1: a behaviour"}}
var failures int
out := captureOutput(func() {
failures = checkLanded("p.md", acs, map[string]string{}, fe, ground)
failures = checkLanded("p.md", nil, acs, map[string]string{}, fe, ground, Config{})
})
assert.Equal(t, tc.wantFail, failures > 0, "strict-failure decision")
if tc.wantMessage != "" {
Expand Down
61 changes: 61 additions & 0 deletions internal/actrace/config.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,61 @@
// SPDX-FileCopyrightText: Copyright 2026 Stacklok, Inc.
// SPDX-License-Identifier: LicenseRef-Stacklok-Proprietary

package actrace

import (
"fmt"
"os"
"path/filepath"

"gopkg.in/yaml.v3"
)

// Config is the optional `.actrace.yml` at a repo root. It is how a consuming
// repo turns on the opt-in features added for ADR-0077 (Atrium) without
// changing behaviour for a repo that ships no config (Airlock). A missing file
// yields the zero Config, which disables every feature — so the tool behaves
// exactly as it did before this file existed.
type Config struct {
// Resolvers maps a verify-method prefix (including its colon, e.g.
// "edge:") to an external command that decides whether a token with that
// prefix holds. ac-trace invokes the command with the raw verify token as
// a single, final argv element — no shell — and reads the exit code: 0
// means satisfied, non-zero means the proof does not hold. This keeps
// ac-trace domain-agnostic: the consuming repo owns what "edge:" means.
Resolvers map[string]ResolverConfig `yaml:"resolvers"`

// JourneyIntegrity enables the ADR-0077 gate on a landed plan: the
// `**Surface:**` scenario tag, the journey-proof requirement on a
// user-facing scenario, and the rejection of weak proof methods for a
// user-facing AC. Off by default so a repo that has not adopted the
// convention is unaffected.
JourneyIntegrity bool `yaml:"journey_integrity"`
}

// ResolverConfig is one verify-method-prefix resolver: the argv of the command
// ac-trace runs. ac-trace appends the raw verify token as one additional argv
// element, so the token reaches the command as inert data, never through a
// shell.
type ResolverConfig struct {
Command []string `yaml:"command"`
}

// loadConfig reads `.actrace.yml` at root. A missing file is not an error — it
// returns the zero Config (every opt-in feature off). A present-but-malformed
// file is an error: a typo must not silently disable a gate the repo meant to
// turn on.
func loadConfig(root string) (Config, error) {
var c Config
b, err := os.ReadFile(filepath.Join(root, ".actrace.yml"))
if err != nil {
if os.IsNotExist(err) {
return c, nil
}
return c, fmt.Errorf("reading .actrace.yml: %w", err)
}
if err := yaml.Unmarshal(b, &c); err != nil {
return c, fmt.Errorf(".actrace.yml: %w", err)
}
return c, nil
}
54 changes: 54 additions & 0 deletions internal/actrace/config_test.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,54 @@
// SPDX-FileCopyrightText: Copyright 2026 Stacklok, Inc.
// SPDX-License-Identifier: LicenseRef-Stacklok-Proprietary

package actrace

import (
"os"
"path/filepath"
"testing"

"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
)

// TestLoadConfig_MissingFileIsZeroValue pins the opt-in contract: a repo that
// ships no .actrace.yml gets the zero Config (every feature off), not an error.
func TestLoadConfig_MissingFileIsZeroValue(t *testing.T) {
t.Parallel()
c, err := loadConfig(t.TempDir())
require.NoError(t, err)
assert.False(t, c.JourneyIntegrity)
assert.Nil(t, c.Resolvers)
}

// TestLoadConfig_ParsesResolversAndFlag pins parsing of a present config: the
// journey_integrity flag and a prefix→command resolver mapping.
func TestLoadConfig_ParsesResolversAndFlag(t *testing.T) {
t.Parallel()
dir := t.TempDir()
body := "" +
"journey_integrity: true\n" +
"resolvers:\n" +
" \"edge:\":\n" +
" command: [\"scripts/resolve-edge.sh\", \"--graph\", \".seamline/architecture.detail.json\"]\n"
require.NoError(t, os.WriteFile(filepath.Join(dir, ".actrace.yml"), []byte(body), 0o644))

c, err := loadConfig(dir)
require.NoError(t, err)
assert.True(t, c.JourneyIntegrity)
require.Contains(t, c.Resolvers, "edge:")
assert.Equal(t,
[]string{"scripts/resolve-edge.sh", "--graph", ".seamline/architecture.detail.json"},
c.Resolvers["edge:"].Command)
}

// TestLoadConfig_MalformedIsError pins that a present-but-broken config is a
// hard error — a typo must not silently disable a gate the repo turned on.
func TestLoadConfig_MalformedIsError(t *testing.T) {
t.Parallel()
dir := t.TempDir()
require.NoError(t, os.WriteFile(filepath.Join(dir, ".actrace.yml"), []byte("journey_integrity: [not a bool\n"), 0o644))
_, err := loadConfig(dir)
assert.Error(t, err)
}
Loading