Upgrade: career-aligned rigour + BYOK AI (Cachex-AI)

docs/README.md

@@ -0,0 +1,20 @@
+# Documentation
+
+| Document | What it covers |
+| --- | --- |
+| [model-card.md](model-card.md) | Agent card for the `_4399` minimax agent: intended use, provenance, evaluation with intervals, known weaknesses |
+| [ai-use-statement.md](ai-use-statement.md) | What the optional bring-your-own-key AI features do and never do, data sent, human review and the audit log |
+| [decisions/](decisions) | Decision records. A past record is never edited; a new record supersedes it. |
+
+Decision records:
+
+- [DR-001: Evaluation function features and weights](decisions/DR-001-evaluation-function.md)
+- [DR-002: Dynamic depth allocation for minimax](decisions/DR-002-dynamic-depth-allocation.md)
+- [DR-003: Port to TypeScript and run in Web Workers, verified by parity tests](decisions/DR-003-typescript-port-web-workers.md)
+- [DR-004: Compare two methods by their paired difference](decisions/DR-004-paired-comparisons.md)
+- [DR-005: Report uncertainty at the unit that was sampled](decisions/DR-005-report-at-the-sampled-unit.md)
+- [DR-006: Ground commentary in what the search found, and keep the check with the decision](decisions/DR-006-grounded-commentary-audit.md)
+
+These files are rendered on the website under `/methods`. The site reads a synced copy in
+`web/content/docs/` (the Vercel build only sees `web/`); run `pnpm sync-docs` in `web/` after
+editing, and the test suite fails if the copies drift.

docs/ai-use-statement.md

@@ -0,0 +1,73 @@
+# AI use statement
+
+This site has two optional features that call a large language model. Everything else,
+including the agent you play against and every statistic on the site, is ordinary code
+that runs without any AI service. This statement is informed by the Australian Government's
+policy for the responsible use of AI in government (Digital Transformation Agency), the
+transparency principles of the EU AI Act, and the NIST AI Risk Management Framework. It is
+not a claim of compliance or certification with any of them.
+
+## What the AI features do
+
+1. **Move commentator** (on `/play` and `/spectate`). When the minimax agent has searched for
+   a move, you can ask a model to explain it in plain English. The model receives only the
+   agent's own numbers for that move: board size, turn, the move, search depth, nodes visited,
+   the top candidate scores and the six evaluation features with their weights and
+   contributions, plus a sentence saying what the score means at that depth: the features
+   describe the position right after the move, so they explain the score only for a one-ply
+   search, and a forced win found by a deeper search is the reason for the move, not the
+   features. It is instructed to use nothing else.
+2. **LLM as a player** (on `/llm-arena`). A model plays a few short games against the original
+   agent. Each turn it receives the rules, the board, the move history and the list of legal
+   moves, and returns one move as JSON. Answers are scored the same way for both providers, under
+   the same output-token budget: an illegal move, or a reply with no usable move (malformed, cut
+   off at the token limit, or refused), is rejected and asked again, up to three times. The
+   headline is the share of turns whose first answer was rejected, reported next to random,
+   greedy and scripted baselines on the same seeds.
+
+## What they never do
+
+- They never choose the agent's moves, change its evaluation, or feed into the tournament,
+  the A* study or any other statistic on the site.
+- They never run unless you add your own API key and press a button.
+- They never send anything to this site's operators. The site is static and has no server;
+  calls go directly from your browser to the provider you chose.
+- They never store your key anywhere except your own browser (session storage by default,
+  local storage only if you tick "remember on this device"), and never write it to the audit
+  log or to any file in the repository.
+
+## Data sent to the provider
+
+Only the prompt shown in the audit log for that call: game state, rules, legal moves and, for
+the commentator, the agent's evaluation breakdown. No personal information is collected or
+sent. Your API key goes in the request header to the provider (Anthropic or OpenAI), which
+processes the request under its own terms and bills your account.
+
+## Human in the loop and transparency
+
+- Every model output on the site is labelled **AI-generated** with the model name.
+- Commentary is checked automatically against the facts it was given: cited feature
+  directions must match the sign of their contribution (a feature shared by both players may be
+  called neutral), every number and move it mentions must appear in the input, if it says which
+  player moved that must be the player who did, and a forced win must be reported when the search
+  found one and never claimed when it did not. The result of that check is shown next to the
+  text, opened when a check failed, and stored with the call in the audit log.
+- You review each commentary and record a decision: accept, edit (your edit is stored next to
+  the original) or reject. Commentary you did not review where it appeared (for example because
+  the game moved on) stays "awaiting review" and can be decided later on `/ai-log`, where the
+  stored check is shown above the decision buttons.
+- Every call, successful or not, is recorded in the AI audit log at `/ai-log` with its
+  timestamp, feature, provider, model, exact input, output (including the raw reply when it
+  failed validation, was cut off or was refused), latency, token usage when the provider reports
+  it, the grounding check result, and your decision, so the record shows when commentary was
+  accepted despite a failed check. Automated evaluation calls (the LLM player) are
+  marked "n/a". The log stays in your browser and can be exported as JSON or CSV, or cleared.
+
+## Known limitations
+
+- A model can still misdescribe the facts in ways the automatic check does not catch (for
+  example, emphasis or causal language). The check is a guard, not a guarantee; that is why
+  the human decision exists.
+- LLM-as-a-player results from a handful of games have wide intervals and depend on the
+  model, the prompt and the day. Treat them as a demonstration of the evaluation method, not
+  as a ranking of models.

docs/decisions/DR-001-evaluation-function.md

@@ -0,0 +1,51 @@
+# DR-001: Evaluation function features and weights
+
+- **Decision:** score positions with six hand-counted board features and fixed, hand-tuned weights (`weights.json`), from Red's point of view.
+- **Status:** accepted in 2022 (team _4399, COMP30024 Part B). Recorded retrospectively on 6 October 2026 from the code; the 2022 report is not in the repository, so anything about our reasoning that the code does not show is marked as such.
+- **Supersedes:** nothing. **Superseded by:** nothing yet.
+
+## Context
+
+The Part B agent needed a static evaluation for minimax leaves. It had to run inside the referee's CPU-time and memory limits (60 seconds and 100 MB per player when enabled) on Python 3.6, on boards from 3 × 3 to 15 × 15, and be explainable in the report. We had no training data and no self-play infrastructure, and about four weeks.
+
+## Decision
+
+`Eval(board)` returns a weighted sum of six features, each counted for Red and subtracted for Blue (except the shared empty-cell count):
+
+| Feature | Weight | Sign |
+| --- | --- | --- |
+| Empty hexes (shared) | 0.5 | + |
+| Tokens in a solid triangle with two adjacent friendly neighbours | 3 | + |
+| Token count | 8 | + |
+| Positional value (score matrix: rim high, centre low) | 2 | + |
+| Tokens in a half-built diamond the opponent could complete | 4 | − |
+| Tokens with an exposed gap (weak formation) | 3 | − |
+
+A random ×(1 + 10⁻⁵) bias breaks ties between equal scores. The weights live in `coursework/Project Part B/code/utility/weights.json`, which the web port copies verbatim (a test checks the bytes).
+
+## Options considered
+
+What the code shows we built or started:
+
+1. **Hand-tuned linear evaluation over pattern features** (chosen).
+2. **A connection-distance feature.** `_4399/eval_func.py` contains `estimate_steps_to_win`, which uses our Part A A* search to estimate how many cells each side still needs. It is written but commented out of the feature list in `utility/evaluation.py`. The code does not record why; cost per leaf is the likely reason, but that was never measured.
+3. **Learned weights** (for example, tuning against the random agent or by self-play). Not attempted; nothing in the repository suggests it was.
+
+## Why
+
+Pattern counts are cheap (one pass over the board), map directly onto the capture rule (diamonds) and onto shapes that are hard to capture (triangles), and are easy to explain in a report. Fixed weights let us change behaviour quickly by hand during development. The rationale for the specific numbers was "team empirical experience" (the code's own words); there is no record of a systematic search.
+
+## What happened
+
+Measured in October 2026 with the parity-tested TypeScript port (reference round robin, 1,200 games on 4 × 4 to 6 × 6, seed 2022; details on `/tournament`):
+
+- Against the random agent the evaluation is clearly useful: the original agent won 106 of 120 games (88.3%, Wilson 95% CI 81.4% to 92.9%), and greedy one-ply on the same evaluation won 100 of 120 (83.3%, 75.7% to 88.9%). The original Python benchmark (144/160, 90.0%, 84.4% to 93.8%) agrees.
+- Against agents that search, lookahead on the same evaluation is what wins: fixed depth 3 beat the original in 84 of 120 games. Used one ply deep, the evaluation does not separate the original from greedy one-ply at all (61 wins to 59).
+- It has blind spots that are visible in the feature list. Nothing measures connection progress (the actual goal of the game), and the positional matrix rewards the rim over the centre, which is the opposite of standard Hex opening advice. Neither was tested in 2022.
+- By construction, token count carries the largest weight (8 per token), so a capture (two tokens, 16 points) outweighs several shape features at once. Whether that balance is right was never tested.
+
+## What I'd change
+
+- Put connection distance back in, measured: add `estimate_steps_to_win` (or a cheaper two-distance or flow measure) and test it in the round-robin harness with colour-swapped pairs and intervals, rather than by feel.
+- Tune weights against data instead of by hand: even a simple coordinate search over the six weights, scored by win rate against fixed opponents on held-out seeds, would make the numbers defensible.
+- Report evaluation quality directly (for example, how often the evaluation ranks the eventual winner ahead after k moves), not only final win rates.

docs/decisions/DR-002-dynamic-depth-allocation.md

@@ -0,0 +1,49 @@
+# DR-002: Dynamic depth allocation for minimax
+
+- **Decision:** search one ply while at least 15% of cells are empty, then deepen to 2, 3 and 4 plies as the empty share falls below 15%, 10% and 5%, after a hard-coded opening and an instant-win check.
+- **Status:** accepted in 2022 (team _4399). Recorded retrospectively on 6 October 2026. The agent on the website still plays exactly this.
+- **Supersedes:** nothing. **Superseded by:** nothing yet.
+
+## Context
+
+The referee could enforce a CPU-time limit per player for the whole game (60 seconds when enabled) on boards up to 15 × 15, and our minimax copied the whole board (`deepcopy`) at every node, so search cost grew quickly with depth and board size. We needed a rule for how deep to search that would not time out on large boards, but would look further ahead when the game was close to decided.
+
+## Decision
+
+`dynamic_depth_allocation` in `_4399/player.py`:
+
+| Empty cells | Depth |
+| --- | --- |
+| ≥ 15% | 1 |
+| 10% to 15% | 2 |
+| 5% to 10% | 3 |
+| < 5% | 4 |
+
+Before any search, the agent plays a fixed opening on turns 1 and 2 (always cell (1, 1), stealing it as Blue if Red took it) and takes any immediately winning move.
+
+## Options considered
+
+1. **A fixed depth** (2 or 3 throughout). Simple, but at n = 15 a fixed depth 3 with full board copies would not have met the time limit.
+2. **Iterative deepening with a time budget.** The robust textbook answer; it needs move ordering and a clock, and we did not build it.
+3. **Depth by empty-cell ratio** (chosen): cheap, predictable and safe on large boards.
+
+## Why
+
+The ratio rule guaranteed we would not time out on big boards, and the intuition was that the endgame, with few empty cells, is where exact lookahead pays off and is also cheapest. It needed no timing code.
+
+## What happened
+
+Measured in October 2026 (reference round robin, 1,200 games on 4 × 4 to 6 × 6, 40 colour-swapped games per pairing per size, seed 2022; alpha-beta study on 120 positions; details on `/tournament`):
+
+- **The deeper searches almost never run.** Over the whole tournament the original agent searched deeper than one ply on 34 of 4,602 searched moves (0.7%). Most games are decided before fewer than 15% of cells are empty.
+- **So the agent is, in practice, greedy one-ply with an opening book.** Head to head with greedy one-ply it won 61 and lost 59 (50.8%, Wilson 95% CI 42.0% to 59.6%). Bradley-Terry strengths (Elo scale, random = 0): original 339 (95% bootstrap CI 290 to 396) against greedy 324 (274 to 386). No detectable difference.
+- **Depth does matter.** Fixed depth 3 beat the original in 84 of 120 games (70.0%, 61.3% to 77.5%) and rates 490 Elo (432 to 559). Fixed depth 2 sits between them at 369 (321 to 427), and its head-to-head with the original (65 of 120 for depth 2) is not decisive.
+- **Depth is expensive partly because of a bug.** The alpha-beta search visits between 92% and 100% of the nodes that plain minimax visits. In the minimising branch `if beta <= min_score: beta = min_score` can only raise beta, so the window never narrows there. Results are still correct (all 480 test searches returned the same root value as plain minimax), but a textbook update (`beta = min(beta, min_score)`) would visit 18% of the tree instead of 95% on 6 × 6 at depth 3, roughly a fivefold saving.
+- The cost of depth 3 in the tournament was about 27 ms per move against 0.1 ms for the original (TypeScript port on an Apple M4; the 2022 Python is much slower).
+
+## What I'd change
+
+- Fix the beta update first. It is one line, it keeps results identical, and it makes deeper search several times cheaper.
+- Replace the ratio thresholds with iterative deepening under a per-move time budget, with move ordering from the previous iteration, so the agent searches as deep as the clock allows on every move rather than only in the endgame.
+- Choose thresholds or budgets by experiment, using this harness: colour-swapped pairs, fixed seeds, Wilson and bootstrap intervals, and a pre-registered comparison against the current agent.
+- Keep this record: the original thresholds stay on the site as the "as submitted" agent, and any change ships as a new variant with its own row in the tournament.

docs/decisions/DR-003-typescript-port-web-workers.md

@@ -0,0 +1,40 @@
+# DR-003: Port to TypeScript and run in Web Workers, verified by parity tests
+
+- **Decision:** re-implement the original Python (referee rules, Part A A*, Part B agent) line by line in TypeScript, run all search in Web Workers in the browser, and prove equivalence with parity fixtures generated by the unchanged Python.
+- **Status:** accepted on revival of the project (2026, Rin). Recorded on 6 October 2026.
+- **Supersedes:** nothing. **Superseded by:** nothing yet.
+
+## Context
+
+The 2022 code ran only from a terminal against the subject's referee. To make it something people can play, inspect and evaluate, it needed to run on a website. The site should be static (no server to pay for or secure), stay responsive while the agent thinks, and keep the original results faithful: a port that quietly plays differently would make every number on the site meaningless.
+
+## Decision
+
+- Port each module with the original's structure and quirks preserved (for example `game_end` ignoring STEAL, the tie-break bias, and the inverted beta update described in DR-002), not a cleaned-up rewrite.
+- Generate fixtures with `scripts/generate_parity_fixtures.py`, which imports the unchanged Python and records its outputs, and assert in Vitest that the port reproduces them exactly.
+- Where the original depends on CPython internals, simulate them: A* iterates neighbours out of a `set`, so `python-set-order.ts` reproduces CPython 3.12's tuple hashing and set probing to get the same tie-breaking and node counts.
+- For randomness, the port uses a seeded generator (mulberry32). Parity tests run the Python with the move shuffle replaced by a canonical sort and the bias fixed at 1, which removes the only difference.
+- Run the agent, the A* study and the tournament in Web Workers, with a main-thread fallback.
+
+## Options considered
+
+1. **Run the Python in the browser with Pyodide.** Byte-for-byte faithful, but a large download, slow start, and still single-threaded and slow (deep copies at every node).
+2. **Host the Python behind an API.** Faithful, but needs a server, adds latency and cost, and makes the static, inspectable site impossible.
+3. **Port to TypeScript with parity tests** (chosen).
+
+## Why
+
+A port gives speed (the tournament plays 1,200 games in about three minutes on a laptop), a static deployment, and code readers can step through. The risk of a port is silent divergence, and parity fixtures turn that risk into a test: if the port differs from the original on any recorded case, CI fails.
+
+## What happened
+
+- The port reproduces the original exactly on everything recorded: paths and node counts for 184 A* runs, every board state and capture across 40 random refereed games, evaluation features on 237 positions, 179 minimax searches, 143 agent moves and 5 full self-play games.
+- The tournament harness uses the port because the original is far too slow for it: in Python one 6 × 6 self-play game at fixed depth 2 takes about 40 seconds. To check the harness itself, `scripts/crosscheck_tournament.py` replays the pairings that are feasible in Python (boards 4 and 5, 480 games) from the unchanged code. All six pairings agree within sampling error (differences in win rate from −8.8 to +3.7 percentage points, every Newcombe 95% interval containing 0). With 80 games per pairing that check can only rule out large discrepancies, about ±15 points.
+- Faithfulness has a cost: the site plays the original's bugs too (the alpha-beta beta update, rim-favouring positional weights). That is deliberate; they are documented instead of fixed.
+- Random play cannot be compared game for game across languages, because the random streams differ. Comparisons are distributional.
+
+## What I'd change
+
+- Record the original Python's random draws (or patch its RNG to mulberry32) so whole random games, not just deterministic ones, can be compared move for move.
+- Add a property-based test layer (random positions, compare rules and evaluation against the Python on the fly in CI) on top of the fixed fixtures.
+- Keep any improved agent strictly separate from the "as submitted" port, as a named variant with its own tests and its own row in the tournament.