diff --git a/README.md b/README.md index 7ef1ce7..d948e81 100644 --- a/README.md +++ b/README.md @@ -2,7 +2,7 @@ # Cachex Arena -**Play our COMP30024 minimax agent, watch AI vs AI, and step through our A\* search, in the browser.** +**Play our COMP30024 minimax agent, watch AI vs AI, step through our A\* search, and see how strong the agent really is, with confidence intervals.** [![CI](https://github.com/rNLKJA/Cachex-AI/actions/workflows/ci.yml/badge.svg)](https://github.com/rNLKJA/Cachex-AI/actions/workflows/ci.yml) [![Next.js](https://img.shields.io/badge/Next.js-16-000?logo=nextdotjs)](https://nextjs.org/) @@ -16,6 +16,92 @@ --- +## Showcase + +

+ Playing the minimax agent on a 5 × 5 board: the opening steal, a capture, a recapture and the agent's explanation of its move, with a numbered caption for each step +

+ +**[Take the guided tour →](https://cachex-ai.vercel.app/tour)** Three captioned walkthrough videos and +every screenshot below, recorded from a production build of the site by a reproducible Playwright script +([`web/e2e/showcase.spec.ts`](web/e2e/showcase.spec.ts), run with `pnpm showcase`) that also checks +each step, so the same seeds give the same moves and numbers. No real API key appears anywhere: +the AI steps use a placeholder key and a mocked reply, labelled on screen. + +### Key features + +| | | +| --- | --- | +| Landing page in light mode
**Landing page.** The game, the coursework and the key results, each with its interval. | Landing page in dark mode
**Dark mode.** The same page in dark mode. | +| Play page after a capture and a recapture, with the Why that move panel
**Play the agent.** After a capture and a recapture on 5 × 5, with the agent's search for its capture. | AI vs AI game with the evaluation trend chart
**AI vs AI.** A seeded agent-vs-agent game with the evaluation trend. | +| A* Lab showing the 8-cell path for the original sample input
**A\* Lab.** The original sample input: an 8-cell path that matches the recorded output. | Paired heuristic study with bootstrap interval, Wilcoxon test and McNemar test
**Paired heuristic study.** 980 paired boards: bootstrap CI, Wilcoxon test, BFS optimality and McNemar. | +| Tournament results with Wilson and bootstrap intervals
**Tournament harness.** A seeded 24-game round robin: Wilson and bootstrap 95% intervals. | AI settings dialog for bringing your own key
**Bring your own key.** AI settings: Anthropic by default, the key stays in this browser. | +| LLM Arena evaluation setup and baselines
**LLM Arena.** The LLM-as-a-player harness and its baselines on the same seeds (no key needed). | Methods page with data provenance
**Methods.** Provenance, evaluation design, limitations, decision records and AI use. | +| Landing page on a phone
**Mobile: landing.** The landing page at 390 px. | Play page on a phone
**Mobile: play.** Playing the agent on a phone, after the recapture. | +| A* Lab on a phone
**Mobile: A\* Lab.** The A\* Lab with the sample input's path. | **[More on /tour →](https://cachex-ai.vercel.app/tour)** Every screenshot in a lightbox, and the three videos with captions and transcripts. | + +### Workflow walkthrough + +The numbered steps are the captions shown on screen. The videos on +[/tour](https://cachex-ai.vercel.app/tour) play in real time with captions; the GIFs here are +shortened (waits on the app and scrolls are cut, and they play at 1.25× speed). + +#### 1. Play the agent (`/play`) + +Setup: 5 × 5, you play Red against the minimax agent, page seed 4399. (The GIF at the top of this section.) + +1. Open Play: a 5 × 5 board against the minimax agent (seed 4399), coordinates on +2. Red opens on the strong cell (1, 1) +3. Blue steals: Red's opening tile is mirrored across the diagonal and becomes Blue's +4. Red plays (0, 1); the agent answers at (4, 1) +5. Red plays (1, 0), leaving two red tiles inside a diamond +6. Capture: Blue closes the diamond at (0, 0) and removes both red tiles +7. Red retakes (0, 1); the agent plays (1, 4) +8. Recapture: Red plays (1, 0) and removes Blue's (0, 0) and (1, 1) +9. Why that move? Pick the capture in the move log: search depth, candidate scores, features + +#### 2. A\* Lab (`/astar`) + +Setup: preset “Sample input 1” (`code/sample_input.json`); reference study seed 2022. + +1. Open the A* Lab: the Part A search, ported line by line from Python +2. Load the original sample input (code/sample_input.json) +3. Manhattan, as in the recorded output: animate the expansions +4. An 8-cell path that matches sample_output.txt from the original repo +5. Switch to Euclidean and animate the same board +6. Same board, both heuristics: path cost, nodes expanded, queue pushes +7. Paired study on 980 boards: paired bootstrap CI and Wilcoxon test on expansions +8. Optimality against breadth-first search: paired difference and exact McNemar test + +
+Watch the A* Lab walkthrough (GIF, 6.7 MB) +

A* Lab walkthrough: loading the sample input, animating Manhattan and Euclidean, and the paired study

+
+ +#### 3. Tournament and LLM evaluation (`/tournament`, `/llm-arena`, `/ai-log`) + +Setup: round robin of 4 agents on 4 × 4, 2 colour-swapped pairs, seed 4399; LLM Arena 2 games on +4 × 4, seed 2026. Steps 6 to 10 are a **mocked AI response for illustration**: the key is a +placeholder, every request to the provider is intercepted in the browser and answered by a mock +that plays the first legal cell, so the LLM row shows the mock, not a real model's results. + +1. Open the tournament harness: a seeded, colour-swapped round robin +2. Keep 4 agents and seed 4399; 4 × 4 only, 2 colour-swapped pairs: 24 games +3. Run all 24 games in parallel Web Workers +4. Win rates with Wilson 95% CIs, Bradley-Terry strengths with bootstrap CIs +5. AI settings: bring your own key, kept in this browser and sent only to the provider +6. For this demo: a placeholder key and the model id “mock-for-illustration”, no real key +7. LLM Arena: 2 games on 4 × 4 against the minimax agent, on the baselines' seeds +8. Every reply is checked against the legal moves and labelled AI-generated +9. Side by side with random, greedy and scripted baselines, Wilson 95% CIs +10. Every call is in the AI audit log, with JSON and CSV export +11. Forget key: the placeholder is removed from this browser + +
+Watch the tournament and LLM evaluation walkthrough (GIF, 7.4 MB) +

Tournament and LLM evaluation walkthrough: a seeded round robin with intervals, the bring-your-own-key dialog and a mocked LLM Arena run

+
+ ## Overview Cachex is a two-player connection game on an _n_ × _n_ rhombic hex board, based on Hex. @@ -40,31 +126,111 @@ ported to TypeScript line by line and is verified against the original code. | Route | What you can do | | --- | --- | -| `/` | What the coursework asked, what we built, key results from the original code, credits | -| `/play` | Play the minimax agent (or the random baseline) on 3×3 to 10×10, as Red or Blue, with STEAL, capture animations, undo, a move log and a "why that move?" panel (search depth, top candidates, evaluation feature breakdown) | -| `/spectate` | AI vs AI with play/pause/step, a speed control, seeded replays and an evaluation trend chart | -| `/astar` | Paint tiles, move start and goal, animate A\* expansions (open and closed sets), toggle Manhattan/Euclidean and the original block-colour option, load or export the original `sample_input.json` format, compare against the recorded outputs, and rerun the heuristic study | - -Agents and the heuristic study run in **Web Workers**, so the board stays responsive. - -### Key results (from the original Python) - -- The `_4399` agent beat the original random agent in **144 of 160 games** (90%) across - board sizes 4–7, playing both colours (`scripts/benchmark_agent.py`). -- A\* reproduces the recorded outputs for both sample inputs (paths of **8** and **13** cells). +| `/` | What the coursework asked, what we built, key results from the original code (now with their confidence intervals), credits | +| `/play` | Play the minimax agent (or the random baseline) on 3×3 to 10×10, as Red or Blue, with STEAL, capture animations, undo, a move log and a "why that move?" panel (search depth, top candidates, evaluation feature breakdown), plus an optional AI commentary on the move | +| `/spectate` | AI vs AI with play/pause/step, a speed control, seeded replays, an evaluation trend chart and the same optional commentary | +| `/astar` | Paint tiles, move start and goal, animate A\* expansions, toggle Manhattan/Euclidean and the original block-colour option, load or export the original `sample_input.json` format, and a **paired heuristic study**: both heuristics on the same random boards, paired bootstrap interval, Wilcoxon signed-rank test, effect sizes, and an optimality check against breadth-first search | +| `/tournament` | **Round-robin harness**: the original agent vs fixed-depth, greedy and random variants, colour-swapped and seeded, in parallel Web Workers. Wilson intervals for win rates, Bradley-Terry strengths on the Elo scale with bootstrap intervals, first-move effect, move times, CSV export. Shows a precomputed 1,200-game reference tournament, a cross-check against the original Python, the original 144/160 benchmark restated with its interval, and an **alpha-beta efficiency study** | +| `/llm-arena` | **LLM as a player** (bring your own key): a language model plays short games against the minimax agent via structured JSON moves validated against the legal moves; win rate with Wilson interval, the share of turns whose first answer was rejected (illegal move vs no usable move, scored the same way for both providers; interval from a bootstrap over games), forfeits, latency and tokens, side by side with random, greedy and a scripted first-legal-cell baseline on the same seeds (only the games the model finished, if a run stops early), each compared with the model game by game (exact McNemar) | +| `/methods` | Data provenance, method, evaluation design, assumptions, limitations, "what I'd change", the decision records (`/methods/decisions/…`), the agent card (`/methods/agent-card`) and the AI use statement | +| `/ai-log` | The AI audit log: every AI call made from this browser, with exact input, output, latency, tokens and your review decision; JSON and CSV export | +| `/tour` | The guided tour: three captioned walkthrough videos (play the agent, the A\* Lab, the tournament and LLM evaluation) with step lists and transcripts, and every screenshot in a lightbox | + +Agents, the heuristic study and the tournament run in **Web Workers**, so the board stays responsive. Everything works without an API key; the AI features are optional. + +### Key results + +From the original Python: + +- The `_4399` agent beat the original random agent in **144 of 160 games**: 90.0%, Wilson 95% CI + 84.4% to 93.8% (boards 4 to 7, both colours, `scripts/benchmark_agent.py`). +- A\* reproduces the recorded outputs for both sample inputs (paths of **8** and **13** cells, both + confirmed shortest by breadth-first search). - The TypeScript port matches the original **exactly**: paths and node-expansion counts for 184 A\* runs, every board state and capture across 40 random refereed games, evaluation features on - 237 positions, 179 minimax searches, 143 agent moves and 5 full self-play games - (844 Vitest tests in total). + 237 positions, 179 minimax searches, 143 agent moves and 5 full self-play games. + +From the 2026 evaluation (details, seeds and downloadable data on the site): + +- **Dynamic depth barely runs.** In a 1,200-game round robin (4×4 to 6×6), the original searched + deeper than one ply on 34 of 4,602 searched moves (0.7%), and there is no detectable difference + from a greedy one-ply agent on the same evaluation (61 wins to 59; 50.8%, 95% CI 42.0% to + 59.6%; Elo difference +15, 95% CI −24 to +56). That interval only rules out differences larger + than about 10 points of win rate. Fixed depth 3 beats it in 84 of 120 games and is 151 Elo + stronger (95% CI 106 to 195; Elo differences come from the same bootstrap refits, not from + comparing two intervals). +- **It does not block one-move wins.** Searching one ply, it never looks at the opponent's next + move: a scripted Blue that just fills row 0 left to right beats it in 50 of 50 seeded games on + 4×4 (43 of 50 on 5×5, 34 of 50 on 6×6). The LLM arena uses this script as a baseline. +- **Alpha-beta saves almost nothing because of a one-line bug.** The original's + `if beta <= min_score: beta = min_score` never narrows the window: beta stays at +infinity, so + a cut-off can only happen once the search finds a line it scores as a forced win for Red. 444 of + 480 test searches visited every node of the full minimax tree, and the 36 that pruned anything + visited between 0.6% and 92% of it; averaged per board size, depth and move order that is 92% to + 100%, against 17% to 45% for a textbook update. Results are unaffected (all 480 searches return + the same value as plain minimax). The site keeps the agent as submitted (DR-002, DR-005). +- **Manhattan is faster but less often optimal.** On 980 paired random boards Manhattan expands + 110 fewer nodes per board (95% CI 96 to 126 fewer; Wilcoxon p < 0.001, rank-biserial r = −0.81) + but returns a shortest path on 74.3% of solvable boards (71.4% to 77.0%) against 85.6% (83.2% to + 87.7%) for Euclidean. On the same 931 boards that is 11.3 points fewer (paired bootstrap 95% CI + 9.1 to 13.3); where only one heuristic was optimal it was Manhattan on 1 board and Euclidean on + 106 (exact McNemar p < 0.001). Neither heuristic is admissible on this hex grid. +- **No detectable disagreement with the original Python** on the pairings feasible in Python + (480 games; every Newcombe 95% interval for the six win-rate differences contains 0). With 80 + games per pairing this check can only detect differences larger than about ±15 points. + +## Bring your own key (optional AI features) + +The two AI features (the move commentator and LLM-as-a-player) only run if you paste **your own** +API key into **AI settings** (the key icon in the header): + +- **Providers:** Anthropic (default; Claude Haiku 4.5, or Claude Sonnet 5.5) or OpenAI (any model id + you type, default `gpt-5-mini`). +- **Where the key lives:** in your browser only. Session storage by default (cleared when the tab + closes); local storage only if you tick "remember on this device". "Forget key" removes every + saved key, for both providers. +- **Where it goes:** calls go **directly from your browser** to the provider (Anthropic with the + `anthropic-dangerous-direct-browser-access` header via the official SDK; OpenAI's Chat + Completions API). This is a static site with no server, so the key is never sent to us, never + logged and never committed. +- **Grounding and review:** the commentator receives only the agent's own numbers for the move, + including what the score means at that search depth (a deeper search or a forced win is not + explained by the one-ply feature breakdown). An automatic check verifies feature directions, + numbers, moves, which player moved and that a forced win is reported as the search found it; + you then accept, edit or reject the commentary. Every model output is labelled "AI-generated". +- **Audit log:** every call (success or failure) is recorded in IndexedDB with timestamp, feature, + provider, model, exact input, output, latency, token usage, the grounding check result and the + human decision, so an export shows when a reviewer accepted commentary that failed its check. + View it at **`/ai-log`** and export it as JSON or CSV. Keys are never written to it. + +The AI use statement on `/methods` (source: [`docs/ai-use-statement.md`](docs/ai-use-statement.md)) +explains what the features do and never do. It is informed by the Australian DTA policy for the +responsible use of AI in government, EU AI Act transparency principles and the NIST AI RMF; it is +not a claim of compliance with any of them. + +## Methods and decision records + +- [`docs/model-card.md`](docs/model-card.md): the agent card (intended use, provenance, evaluation + with intervals, known weaknesses). +- [`docs/decisions/`](docs/decisions): decision records, each stating the decision first, then + context, options, reasons, what happened (weak numbers included) and what I'd change. + - [DR-001](docs/decisions/DR-001-evaluation-function.md) evaluation function features and weights + - [DR-002](docs/decisions/DR-002-dynamic-depth-allocation.md) dynamic depth allocation + - [DR-003](docs/decisions/DR-003-typescript-port-web-workers.md) TypeScript port, Web Workers and parity testing + - [DR-004](docs/decisions/DR-004-paired-comparisons.md) comparing two methods by their paired difference + - [DR-005](docs/decisions/DR-005-report-at-the-sampled-unit.md) reporting uncertainty at the unit that was sampled (supersedes DR-002's pruning-ratio sentence) + - [DR-006](docs/decisions/DR-006-grounded-commentary-audit.md) grounding commentary in what the search found, and storing the check with the human decision +- All of these are rendered on the site under `/methods`. Past records are never edited; a new + record supersedes an old one. ## Tech stack | | 2022 original | 2026 revival | | --- | --- | --- | | Language | Python 3.6 | TypeScript (strict) | -| Libraries | NumPy, SciPy, Jupyter | Next.js 16 (App Router), React 19, Tailwind CSS v4, shadcn/ui (Radix), lucide-react, next-themes | +| Libraries | NumPy, SciPy, Jupyter | Next.js 16 (App Router), React 19, Tailwind CSS v4, shadcn/ui (Radix), lucide-react, next-themes, zod, Anthropic TypeScript SDK (browser, BYOK), react-markdown | | Compute | CLI / referee | Web Workers, fully client-side; no backend | -| Testing | Notebook experiments | Vitest parity tests against fixtures generated by the original code | +| Testing | Notebook experiments | Vitest parity tests against fixtures generated by the original code; statistics verified against scipy, statsmodels and R; AI clients tested with mocked fetch | | Tooling | conda | pnpm, ESLint, Prettier, GitHub Actions, uv for the Python scripts | ## Repository structure @@ -72,26 +238,32 @@ Agents and the heuristic study run in **Web Workers**, so the board stays respon ``` Cachex-AI/ ├── coursework/ # original submission, unchanged (see coursework/README.md) -│ ├── Project Part A/ # A* search: code/, notebook/, report/, specification/ -│ ├── Project Part B/ # minimax agent: code/ (_4399, utility, referee), skeleton-code-B/ -│ ├── _archive/ # README from the semester -│ └── environment.yml, requirements.txt, pyproject.toml -├── scripts/ # uv scripts that run the original Python to produce artefacts +├── docs/ # agent card, AI use statement, decision records (rendered on /methods) +│ └── showcase/ # README screenshots and GIFs (pnpm showcase) +├── scripts/ # uv scripts that run the ORIGINAL Python │ ├── generate_parity_fixtures.py -│ └── benchmark_agent.py +│ ├── benchmark_agent.py # 144/160 vs random +│ └── crosscheck_tournament.py# tournament pairings feasible in Python ├── web/ # the deployable Next.js app (Vercel root) -│ ├── public/ +│ ├── e2e/ # Playwright guided tour: journeys as end-to-end tests, records the showcase +│ ├── content/docs/ # synced copy of docs/ (pnpm sync-docs; a test checks they match) +│ ├── public/showcase/ # /tour videos (MP4), captions (WebVTT), posters, screenshot copies +│ ├── scripts/ # generate-reference-studies.ts (tsx), sync-docs.mjs, showcase*.mjs │ └── src/ -│ ├── app/ # routes: / , /play, /spectate, /astar, not-found, OG image, icon -│ ├── components/ # ui/ (shadcn), layout/, board/, play/, astar/, landing/ -│ ├── hooks/ # match state, Web Worker clients -│ ├── lib/ # framework-free ports + tests +│ ├── app/ # / , /play, /spectate, /astar, /tournament, /llm-arena, /methods, /ai-log, /tour, CSV routes +│ ├── components/ # ui/, layout/, board/, play/, astar/, tournament/, stats/, ai/, methods/, tour/ +│ ├── hooks/ # match state, Web Worker clients and the tournament worker pool +│ ├── lib/ │ │ ├── cachex/ # referee rules: board, captures, STEAL, win/draw │ │ ├── agent/ # Board_4399, evaluation features, minimax, opening book, random agent │ │ ├── astar/ # A*, CPython set-order simulation, input format, presets, study -│ │ ├── data/ # agent-benchmark.json (generated) +│ │ ├── stats/ # Wilson, Newcombe, bootstrap, Wilcoxon, Bradley-Terry (+ tests) +│ │ ├── tournament/ # agents and variants, schedule, game runner, analysis, Python cross-check +│ │ ├── analysis/ # paired A* study, alpha-beta efficiency +│ │ ├── ai/ # BYOK provider adapters, audit log, commentator, LLM player (+ tests) +│ │ ├── data/ # generated study data (JSON) and their configs │ │ └── __fixtures__/ # parity fixtures (generated) -│ └── workers/ # agent.worker.ts, astar-benchmark.worker.ts +│ └── workers/ # agent, A* study and tournament workers ├── .github/workflows/ci.yml └── LICENSE ``` @@ -107,18 +279,38 @@ pnpm dev # http://localhost:3000 pnpm lint # ESLint pnpm typecheck # tsc --noEmit -pnpm test # Vitest (unit + parity tests) +pnpm test # Vitest (parity, statistics, tournament and AI-client tests) pnpm build && pnpm start + +pnpm gen:reference # regenerate the reference studies (TypeScript port, a few minutes) +pnpm sync-docs # copy ../docs into web/content/docs after editing the docs +``` + +### Guided tour and showcase media + +`pnpm showcase` runs the guided tour in [`web/e2e/showcase.spec.ts`](web/e2e/showcase.spec.ts) with +Playwright on the system Google Chrome (no browser is downloaded; the ffmpeg on `PATH` records the +video). Each journey is an end-to-end test that asserts what it shows (the steal, both captures, +the recorded A\* output, the seeded tournament, the mocked LLM run and the audit log), then +[`web/scripts/showcase-media.mjs`](web/scripts/showcase-media.mjs) writes the screenshots and GIFs +in `docs/showcase/` and the MP4s, captions and posters in `web/public/showcase/`. + +```bash +pnpm showcase # against production (BASE_URL defaults to it) +BASE_URL=http://localhost:3000 pnpm showcase # after pnpm build; the server is started for you +pnpm showcase:test # quick end-to-end check: no pauses, no video ``` ## How the data artefacts are generated -Both scripts import the **original, unchanged** Python from `coursework/` and are run with +The Python scripts import the **original, unchanged** Python from `coursework/` and are run with [uv](https://docs.astral.sh/uv/) (dependencies are declared inline, PEP 723): ```bash uv run scripts/generate_parity_fixtures.py # → web/src/lib/__fixtures__/parity-part-{a,b}.json uv run scripts/benchmark_agent.py # → web/src/lib/data/agent-benchmark.json +uv run scripts/crosscheck_tournament.py # → web/src/lib/data/python-crosscheck.json +cd web && pnpm gen:reference # → web/src/lib/data/{reference-tournament,astar-paired-study,alpha-beta-study}.json ``` - **Parity fixtures** record the original outputs for the sample inputs, the notebook test @@ -134,6 +326,22 @@ uv run scripts/benchmark_agent.py # → web/src/lib/data/agent-benchm The original `get_valid_actions` returns a `set` of `("PLACE", r, q)` tuples, and CPython randomises string hashes per process, so the script re-runs itself with `PYTHONHASHSEED=0`; with that pin the output is byte-for-byte reproducible. +- **Reference studies** (`web/scripts/generate-reference-studies.ts`, configs in + `web/src/lib/data/reference-config.ts`) use the parity-tested TypeScript port, because depth-3 + search in the original Python is far too slow at this scale. Seeds are fixed (2022), so results + are reproducible apart from wall-clock move times. +- **Cross-check.** `scripts/crosscheck_tournament.py` rebuilds the tournament variants from the + unchanged Python (patching only `dynamic_depth_allocation` and the opening-book flag) and + replays the feasible pairings, so the TypeScript harness can be compared with the original. + +### Verifying the statistics + +The helpers in `web/src/lib/stats` are tested against reference values computed with +scipy 1.17 and statsmodels (`uv run --with scipy --with statsmodels`) and cross-checked in R +(`prop.test(correct = FALSE)`, `wilcox.test`, `quantile(type = 7)`): Wilson and Newcombe +intervals, exact and normal-approximation Wilcoxon p-values with ties and zeros, type-7 quantiles, +and Bradley-Terry maximum-likelihood strengths. The reference values are listed in +`web/src/lib/stats/stats.test.ts`. ## Credits diff --git a/docs/README.md b/docs/README.md new file mode 100644 index 0000000..36906af --- /dev/null +++ b/docs/README.md @@ -0,0 +1,20 @@ +# Documentation + +| Document | What it covers | +| --- | --- | +| [model-card.md](model-card.md) | Agent card for the `_4399` minimax agent: intended use, provenance, evaluation with intervals, known weaknesses | +| [ai-use-statement.md](ai-use-statement.md) | What the optional bring-your-own-key AI features do and never do, data sent, human review and the audit log | +| [decisions/](decisions) | Decision records. A past record is never edited; a new record supersedes it. | + +Decision records: + +- [DR-001: Evaluation function features and weights](decisions/DR-001-evaluation-function.md) +- [DR-002: Dynamic depth allocation for minimax](decisions/DR-002-dynamic-depth-allocation.md) +- [DR-003: Port to TypeScript and run in Web Workers, verified by parity tests](decisions/DR-003-typescript-port-web-workers.md) +- [DR-004: Compare two methods by their paired difference](decisions/DR-004-paired-comparisons.md) +- [DR-005: Report uncertainty at the unit that was sampled](decisions/DR-005-report-at-the-sampled-unit.md) +- [DR-006: Ground commentary in what the search found, and keep the check with the decision](decisions/DR-006-grounded-commentary-audit.md) + +These files are rendered on the website under `/methods`. The site reads a synced copy in +`web/content/docs/` (the Vercel build only sees `web/`); run `pnpm sync-docs` in `web/` after +editing, and the test suite fails if the copies drift. diff --git a/docs/ai-use-statement.md b/docs/ai-use-statement.md new file mode 100644 index 0000000..7c61c8c --- /dev/null +++ b/docs/ai-use-statement.md @@ -0,0 +1,73 @@ +# AI use statement + +This site has two optional features that call a large language model. Everything else, +including the agent you play against and every statistic on the site, is ordinary code +that runs without any AI service. This statement is informed by the Australian Government's +policy for the responsible use of AI in government (Digital Transformation Agency), the +transparency principles of the EU AI Act, and the NIST AI Risk Management Framework. It is +not a claim of compliance or certification with any of them. + +## What the AI features do + +1. **Move commentator** (on `/play` and `/spectate`). When the minimax agent has searched for + a move, you can ask a model to explain it in plain English. The model receives only the + agent's own numbers for that move: board size, turn, the move, search depth, nodes visited, + the top candidate scores and the six evaluation features with their weights and + contributions, plus a sentence saying what the score means at that depth: the features + describe the position right after the move, so they explain the score only for a one-ply + search, and a forced win found by a deeper search is the reason for the move, not the + features. It is instructed to use nothing else. +2. **LLM as a player** (on `/llm-arena`). A model plays a few short games against the original + agent. Each turn it receives the rules, the board, the move history and the list of legal + moves, and returns one move as JSON. Answers are scored the same way for both providers, under + the same output-token budget: an illegal move, or a reply with no usable move (malformed, cut + off at the token limit, or refused), is rejected and asked again, up to three times. The + headline is the share of turns whose first answer was rejected, reported next to random, + greedy and scripted baselines on the same seeds. + +## What they never do + +- They never choose the agent's moves, change its evaluation, or feed into the tournament, + the A* study or any other statistic on the site. +- They never run unless you add your own API key and press a button. +- They never send anything to this site's operators. The site is static and has no server; + calls go directly from your browser to the provider you chose. +- They never store your key anywhere except your own browser (session storage by default, + local storage only if you tick "remember on this device"), and never write it to the audit + log or to any file in the repository. + +## Data sent to the provider + +Only the prompt shown in the audit log for that call: game state, rules, legal moves and, for +the commentator, the agent's evaluation breakdown. No personal information is collected or +sent. Your API key goes in the request header to the provider (Anthropic or OpenAI), which +processes the request under its own terms and bills your account. + +## Human in the loop and transparency + +- Every model output on the site is labelled **AI-generated** with the model name. +- Commentary is checked automatically against the facts it was given: cited feature + directions must match the sign of their contribution (a feature shared by both players may be + called neutral), every number and move it mentions must appear in the input, if it says which + player moved that must be the player who did, and a forced win must be reported when the search + found one and never claimed when it did not. The result of that check is shown next to the + text, opened when a check failed, and stored with the call in the audit log. +- You review each commentary and record a decision: accept, edit (your edit is stored next to + the original) or reject. Commentary you did not review where it appeared (for example because + the game moved on) stays "awaiting review" and can be decided later on `/ai-log`, where the + stored check is shown above the decision buttons. +- Every call, successful or not, is recorded in the AI audit log at `/ai-log` with its + timestamp, feature, provider, model, exact input, output (including the raw reply when it + failed validation, was cut off or was refused), latency, token usage when the provider reports + it, the grounding check result, and your decision, so the record shows when commentary was + accepted despite a failed check. Automated evaluation calls (the LLM player) are + marked "n/a". The log stays in your browser and can be exported as JSON or CSV, or cleared. + +## Known limitations + +- A model can still misdescribe the facts in ways the automatic check does not catch (for + example, emphasis or causal language). The check is a guard, not a guarantee; that is why + the human decision exists. +- LLM-as-a-player results from a handful of games have wide intervals and depend on the + model, the prompt and the day. Treat them as a demonstration of the evaluation method, not + as a ranking of models. diff --git a/docs/decisions/DR-001-evaluation-function.md b/docs/decisions/DR-001-evaluation-function.md new file mode 100644 index 0000000..1b4d221 --- /dev/null +++ b/docs/decisions/DR-001-evaluation-function.md @@ -0,0 +1,51 @@ +# DR-001: Evaluation function features and weights + +- **Decision:** score positions with six hand-counted board features and fixed, hand-tuned weights (`weights.json`), from Red's point of view. +- **Status:** accepted in 2022 (team _4399, COMP30024 Part B). Recorded retrospectively on 6 October 2026 from the code; the 2022 report is not in the repository, so anything about our reasoning that the code does not show is marked as such. +- **Supersedes:** nothing. **Superseded by:** nothing yet. + +## Context + +The Part B agent needed a static evaluation for minimax leaves. It had to run inside the referee's CPU-time and memory limits (60 seconds and 100 MB per player when enabled) on Python 3.6, on boards from 3 × 3 to 15 × 15, and be explainable in the report. We had no training data and no self-play infrastructure, and about four weeks. + +## Decision + +`Eval(board)` returns a weighted sum of six features, each counted for Red and subtracted for Blue (except the shared empty-cell count): + +| Feature | Weight | Sign | +| --- | --- | --- | +| Empty hexes (shared) | 0.5 | + | +| Tokens in a solid triangle with two adjacent friendly neighbours | 3 | + | +| Token count | 8 | + | +| Positional value (score matrix: rim high, centre low) | 2 | + | +| Tokens in a half-built diamond the opponent could complete | 4 | − | +| Tokens with an exposed gap (weak formation) | 3 | − | + +A random ×(1 + 10⁻⁵) bias breaks ties between equal scores. The weights live in `coursework/Project Part B/code/utility/weights.json`, which the web port copies verbatim (a test checks the bytes). + +## Options considered + +What the code shows we built or started: + +1. **Hand-tuned linear evaluation over pattern features** (chosen). +2. **A connection-distance feature.** `_4399/eval_func.py` contains `estimate_steps_to_win`, which uses our Part A A* search to estimate how many cells each side still needs. It is written but commented out of the feature list in `utility/evaluation.py`. The code does not record why; cost per leaf is the likely reason, but that was never measured. +3. **Learned weights** (for example, tuning against the random agent or by self-play). Not attempted; nothing in the repository suggests it was. + +## Why + +Pattern counts are cheap (one pass over the board), map directly onto the capture rule (diamonds) and onto shapes that are hard to capture (triangles), and are easy to explain in a report. Fixed weights let us change behaviour quickly by hand during development. The rationale for the specific numbers was "team empirical experience" (the code's own words); there is no record of a systematic search. + +## What happened + +Measured in October 2026 with the parity-tested TypeScript port (reference round robin, 1,200 games on 4 × 4 to 6 × 6, seed 2022; details on `/tournament`): + +- Against the random agent the evaluation is clearly useful: the original agent won 106 of 120 games (88.3%, Wilson 95% CI 81.4% to 92.9%), and greedy one-ply on the same evaluation won 100 of 120 (83.3%, 75.7% to 88.9%). The original Python benchmark (144/160, 90.0%, 84.4% to 93.8%) agrees. +- Against agents that search, lookahead on the same evaluation is what wins: fixed depth 3 beat the original in 84 of 120 games. Used one ply deep, the evaluation does not separate the original from greedy one-ply at all (61 wins to 59). +- It has blind spots that are visible in the feature list. Nothing measures connection progress (the actual goal of the game), and the positional matrix rewards the rim over the centre, which is the opposite of standard Hex opening advice. Neither was tested in 2022. +- By construction, token count carries the largest weight (8 per token), so a capture (two tokens, 16 points) outweighs several shape features at once. Whether that balance is right was never tested. + +## What I'd change + +- Put connection distance back in, measured: add `estimate_steps_to_win` (or a cheaper two-distance or flow measure) and test it in the round-robin harness with colour-swapped pairs and intervals, rather than by feel. +- Tune weights against data instead of by hand: even a simple coordinate search over the six weights, scored by win rate against fixed opponents on held-out seeds, would make the numbers defensible. +- Report evaluation quality directly (for example, how often the evaluation ranks the eventual winner ahead after k moves), not only final win rates. diff --git a/docs/decisions/DR-002-dynamic-depth-allocation.md b/docs/decisions/DR-002-dynamic-depth-allocation.md new file mode 100644 index 0000000..4b65e8f --- /dev/null +++ b/docs/decisions/DR-002-dynamic-depth-allocation.md @@ -0,0 +1,49 @@ +# DR-002: Dynamic depth allocation for minimax + +- **Decision:** search one ply while at least 15% of cells are empty, then deepen to 2, 3 and 4 plies as the empty share falls below 15%, 10% and 5%, after a hard-coded opening and an instant-win check. +- **Status:** accepted in 2022 (team _4399). Recorded retrospectively on 6 October 2026. The agent on the website still plays exactly this. +- **Supersedes:** nothing. **Superseded by:** nothing yet. + +## Context + +The referee could enforce a CPU-time limit per player for the whole game (60 seconds when enabled) on boards up to 15 × 15, and our minimax copied the whole board (`deepcopy`) at every node, so search cost grew quickly with depth and board size. We needed a rule for how deep to search that would not time out on large boards, but would look further ahead when the game was close to decided. + +## Decision + +`dynamic_depth_allocation` in `_4399/player.py`: + +| Empty cells | Depth | +| --- | --- | +| ≥ 15% | 1 | +| 10% to 15% | 2 | +| 5% to 10% | 3 | +| < 5% | 4 | + +Before any search, the agent plays a fixed opening on turns 1 and 2 (always cell (1, 1), stealing it as Blue if Red took it) and takes any immediately winning move. + +## Options considered + +1. **A fixed depth** (2 or 3 throughout). Simple, but at n = 15 a fixed depth 3 with full board copies would not have met the time limit. +2. **Iterative deepening with a time budget.** The robust textbook answer; it needs move ordering and a clock, and we did not build it. +3. **Depth by empty-cell ratio** (chosen): cheap, predictable and safe on large boards. + +## Why + +The ratio rule guaranteed we would not time out on big boards, and the intuition was that the endgame, with few empty cells, is where exact lookahead pays off and is also cheapest. It needed no timing code. + +## What happened + +Measured in October 2026 (reference round robin, 1,200 games on 4 × 4 to 6 × 6, 40 colour-swapped games per pairing per size, seed 2022; alpha-beta study on 120 positions; details on `/tournament`): + +- **The deeper searches almost never run.** Over the whole tournament the original agent searched deeper than one ply on 34 of 4,602 searched moves (0.7%). Most games are decided before fewer than 15% of cells are empty. +- **So the agent is, in practice, greedy one-ply with an opening book.** Head to head with greedy one-ply it won 61 and lost 59 (50.8%, Wilson 95% CI 42.0% to 59.6%). Bradley-Terry strengths (Elo scale, random = 0): original 339 (95% bootstrap CI 290 to 396) against greedy 324 (274 to 386). No detectable difference. +- **Depth does matter.** Fixed depth 3 beat the original in 84 of 120 games (70.0%, 61.3% to 77.5%) and rates 490 Elo (432 to 559). Fixed depth 2 sits between them at 369 (321 to 427), and its head-to-head with the original (65 of 120 for depth 2) is not decisive. +- **Depth is expensive partly because of a bug.** The alpha-beta search visits between 92% and 100% of the nodes that plain minimax visits. In the minimising branch `if beta <= min_score: beta = min_score` can only raise beta, so the window never narrows there. Results are still correct (all 480 test searches returned the same root value as plain minimax), but a textbook update (`beta = min(beta, min_score)`) would visit 18% of the tree instead of 95% on 6 × 6 at depth 3, roughly a fivefold saving. +- The cost of depth 3 in the tournament was about 27 ms per move against 0.1 ms for the original (TypeScript port on an Apple M4; the 2022 Python is much slower). + +## What I'd change + +- Fix the beta update first. It is one line, it keeps results identical, and it makes deeper search several times cheaper. +- Replace the ratio thresholds with iterative deepening under a per-move time budget, with move ordering from the previous iteration, so the agent searches as deep as the clock allows on every move rather than only in the endgame. +- Choose thresholds or budgets by experiment, using this harness: colour-swapped pairs, fixed seeds, Wilson and bootstrap intervals, and a pre-registered comparison against the current agent. +- Keep this record: the original thresholds stay on the site as the "as submitted" agent, and any change ships as a new variant with its own row in the tournament. diff --git a/docs/decisions/DR-003-typescript-port-web-workers.md b/docs/decisions/DR-003-typescript-port-web-workers.md new file mode 100644 index 0000000..0d2c5a2 --- /dev/null +++ b/docs/decisions/DR-003-typescript-port-web-workers.md @@ -0,0 +1,40 @@ +# DR-003: Port to TypeScript and run in Web Workers, verified by parity tests + +- **Decision:** re-implement the original Python (referee rules, Part A A*, Part B agent) line by line in TypeScript, run all search in Web Workers in the browser, and prove equivalence with parity fixtures generated by the unchanged Python. +- **Status:** accepted on revival of the project (2026, Rin). Recorded on 6 October 2026. +- **Supersedes:** nothing. **Superseded by:** nothing yet. + +## Context + +The 2022 code ran only from a terminal against the subject's referee. To make it something people can play, inspect and evaluate, it needed to run on a website. The site should be static (no server to pay for or secure), stay responsive while the agent thinks, and keep the original results faithful: a port that quietly plays differently would make every number on the site meaningless. + +## Decision + +- Port each module with the original's structure and quirks preserved (for example `game_end` ignoring STEAL, the tie-break bias, and the inverted beta update described in DR-002), not a cleaned-up rewrite. +- Generate fixtures with `scripts/generate_parity_fixtures.py`, which imports the unchanged Python and records its outputs, and assert in Vitest that the port reproduces them exactly. +- Where the original depends on CPython internals, simulate them: A* iterates neighbours out of a `set`, so `python-set-order.ts` reproduces CPython 3.12's tuple hashing and set probing to get the same tie-breaking and node counts. +- For randomness, the port uses a seeded generator (mulberry32). Parity tests run the Python with the move shuffle replaced by a canonical sort and the bias fixed at 1, which removes the only difference. +- Run the agent, the A* study and the tournament in Web Workers, with a main-thread fallback. + +## Options considered + +1. **Run the Python in the browser with Pyodide.** Byte-for-byte faithful, but a large download, slow start, and still single-threaded and slow (deep copies at every node). +2. **Host the Python behind an API.** Faithful, but needs a server, adds latency and cost, and makes the static, inspectable site impossible. +3. **Port to TypeScript with parity tests** (chosen). + +## Why + +A port gives speed (the tournament plays 1,200 games in about three minutes on a laptop), a static deployment, and code readers can step through. The risk of a port is silent divergence, and parity fixtures turn that risk into a test: if the port differs from the original on any recorded case, CI fails. + +## What happened + +- The port reproduces the original exactly on everything recorded: paths and node counts for 184 A* runs, every board state and capture across 40 random refereed games, evaluation features on 237 positions, 179 minimax searches, 143 agent moves and 5 full self-play games. +- The tournament harness uses the port because the original is far too slow for it: in Python one 6 × 6 self-play game at fixed depth 2 takes about 40 seconds. To check the harness itself, `scripts/crosscheck_tournament.py` replays the pairings that are feasible in Python (boards 4 and 5, 480 games) from the unchanged code. All six pairings agree within sampling error (differences in win rate from −8.8 to +3.7 percentage points, every Newcombe 95% interval containing 0). With 80 games per pairing that check can only rule out large discrepancies, about ±15 points. +- Faithfulness has a cost: the site plays the original's bugs too (the alpha-beta beta update, rim-favouring positional weights). That is deliberate; they are documented instead of fixed. +- Random play cannot be compared game for game across languages, because the random streams differ. Comparisons are distributional. + +## What I'd change + +- Record the original Python's random draws (or patch its RNG to mulberry32) so whole random games, not just deterministic ones, can be compared move for move. +- Add a property-based test layer (random positions, compare rules and evaluation against the Python on the fly in CI) on top of the fixed fixtures. +- Keep any improved agent strictly separate from the "as submitted" port, as a named variant with its own tests and its own row in the tournament. diff --git a/docs/decisions/DR-004-paired-comparisons.md b/docs/decisions/DR-004-paired-comparisons.md new file mode 100644 index 0000000..f13c9cf --- /dev/null +++ b/docs/decisions/DR-004-paired-comparisons.md @@ -0,0 +1,40 @@ +# DR-004: Compare two methods by their paired difference + +- **Decision:** whenever two methods are measured on the same units, report the difference between them with its own interval (and, for pass/fail outcomes, an exact paired test), rather than two separate intervals whose overlap the reader has to judge. +- **Status:** accepted on 6 October 2026, during review of the 2026 evaluation. Applies to the tournament, the A\* study and the LLM arena. +- **Supersedes:** nothing. It changes how the comparisons behind DR-002 are presented on the site, not what DR-002 records. + +## Context + +The first version of the 2026 evaluation showed some comparisons as two separate 95% intervals. Manhattan's and Euclidean's shortest-path rates were each given a Wilson interval, although both were measured on the same 931 boards. Two agents' Bradley-Terry strengths were compared by whether their intervals overlapped, although both come from one fit anchored to the same agent, so their errors are correlated. A review pointed out that neither display answers the question being asked, which is how far apart the two methods are. + +The LLM arena had a related problem. When a run stopped early (the visitor pressed Stop, or the provider rate-limited the key), the model's row covered only the games it finished, while the baseline rows still covered the whole schedule. The caption said the seeds were the same for every row, and they were not. + +## Decision + +- **Agent strengths:** compare two agents through the difference of their Elo strengths, with a percentile interval taken from the same stratified bootstrap refits as the leaderboard (1,000 resamples, seed 2022 for the reference). The differences are shown in their own table on `/tournament` and in the summary CSV. +- **Pass/fail outcomes on shared units:** report the difference in rates with a paired bootstrap interval that resamples the units (boards), plus an exact McNemar test on the discordant units. The separate Wilson intervals stay, labelled as describing each method on its own. +- **LLM arena:** replay the baselines on exactly the games the model finished, and say so in the caption, including when colours end up unbalanced. Count rejected answers per turn (first answer only), with illegal moves and replies that carry no usable move reported separately, under one rule and one token budget for both providers. + +## Options considered + +1. **Keep the separate intervals and add a note.** Cheap, but it leaves the reader to make a comparison the intervals cannot support: overlapping 95% intervals do not imply no difference, and correlated estimates make overlap even less informative. +2. **Newcombe's or Tango's closed-form interval for a paired difference in proportions.** Good coverage properties, but I could not verify an implementation against a reference package offline, and an unverified formula is worse than none. +3. **Paired bootstrap plus exact McNemar** (chosen). The bootstrap reuses the seeded, tested resampling code, and the exact McNemar test is a two-sided binomial test that I verified against scipy, statsmodels and R. + +## Why + +The design decides the analysis. The A\* study and the tournament were built as paired designs (same boards, same seeds, colour-swapped pairs), so presenting them as if the arms were independent threw away the main advantage of the design. For a pass/fail outcome on shared units, only the units where the methods disagree carry information about the difference, and McNemar's test uses exactly those. For strengths fitted together, the interval of the difference must come from the same replicates, so the correlation is accounted for. + +## What happened + +- **A\*:** on the same boards Manhattan found a shortest path 11.3 percentage points less often than Euclidean (paired 95% bootstrap CI 9.1 to 13.3). Of the 107 boards where only one heuristic was optimal, it was Manhattan on 1 and Euclidean on 106 (exact McNemar p ≈ 1.3 × 10⁻³⁰). The separate intervals pointed the same way; the paired analysis is far sharper, because the 824 boards on which the heuristics agree say nothing about the difference. +- **Tournament:** the original minus greedy one-ply is +15 Elo (95% CI −24 to +56), so DR-002's "no detectable difference" stands, now on the right evidence. Fixed depth 3 minus the original is +151 Elo (106 to 195). Fixed depth 2 minus the original is +30 Elo (−11 to +70): not detectable either way. +- **LLM arena:** a run stopped after one game now compares one game with one game, and the caption says the colours are unbalanced. Truncated and refused replies are no longer counted as illegal moves. +- No original result changed. The per-agent intervals are identical to before, because the bootstrap replicates are the same; only the differences are new. + +## What I'd change + +- Write the comparisons down before running the study, so that the paired analysis is part of the plan rather than added after a review. +- Resample colour-swapped pairs, which are the design unit of the round robin, rather than single games. +- Add a closed-form paired interval (Tango's score interval) as a cross-check once it can be verified against a reference implementation. diff --git a/docs/decisions/DR-005-report-at-the-sampled-unit.md b/docs/decisions/DR-005-report-at-the-sampled-unit.md new file mode 100644 index 0000000..ce1fafd --- /dev/null +++ b/docs/decisions/DR-005-report-at-the-sampled-unit.md @@ -0,0 +1,44 @@ +# DR-005: Report uncertainty at the unit that was sampled + +- **Decision:** every interval accounts for the unit that was actually sampled independently (positions, games, boards), every average is shown with the distribution it summarises when that distribution is lopsided, and every seeded interval is computed on data in a fixed order, so the same seed always gives the same numbers. +- **Status:** accepted on 6 October 2026, after a second review of the 2026 evaluation (PR #12). +- **Supersedes:** the pruning-ratio sentence in DR-002 ("visits between 92% and 100% of the nodes"), and the per-turn interval for the LLM arena described in DR-004. Both records stay as written; this record says what changed and why. + +## Context + +A second review found three places where the numbers on the site were correct but the way they were summarised said more than the data did. + +1. **Alpha-beta.** DR-002, the agent card and the README said the original alpha-beta "visits 92% to 100%" of the full tree. That range is of the *averages* per board size, depth and move order. Individual searches range from under 1% to 100%, and the distribution is all-or-nothing: most searches prune nothing, and a few prune a lot. +2. **LLM arena.** The share of turns whose first answer was rejected had a Wilson interval that treated every turn as an independent trial. Turns in one game share the model, the prompt and an evolving position, and errors tend to repeat, so turns are clustered by game. The arena also showed each baseline's win rate next to the model's, with no paired comparison, although DR-004 lists the arena as a paired design. +3. **Tournament reproducibility.** With the same seed, the in-browser round robin played identical games, but the Bradley-Terry intervals and the order of rows in the games CSV changed between runs. Web Workers return games in the order they finish, and the seeded bootstrap indexes into the arrays in that order. Three runs at seed 4399 gave three different intervals for fixed depth 2: [199.2, 443.3], [187.9, 446.2] and [192.1, 437.2]. + +## Decision + +- **Alpha-beta:** report the per-position distribution next to the averages (how many searches visited every node, and the range among those that pruned anything), and state the mechanism: with the bug, beta never drops below +infinity, so a cut-off can only happen once the search finds a line it scores as a forced win for Red. +- **LLM arena, rejected answers:** a Wilson interval on the effective number of turns, n / deff, where the design effect deff is the cluster-robust variance of the pooled rate (games as clusters) over its variance under independent turns, floored at 1. With one game the design effect cannot be estimated, and the site says the interval then treats turns as independent. +- **LLM arena, versus baselines:** for each baseline, the games won by only the model and by only the baseline, on the same seeds and colours, with an exact McNemar test. +- **Tournament:** sort game records into schedule order (board size, pairing, round, colour leg) before analysis and export. A run played one game at a time, like the reference run, is already in that order, so its numbers do not change. + +## Options considered + +1. **Keep the old summaries and add footnotes.** Cheapest, but a footnote does not stop a reader quoting "92% to 100%" as the range for any search, or reading a Wilson interval over 24 turns as if it came from 24 independent trials. +2. **Cluster bootstrap over games.** Resample whole games and recompute the pooled rate. I implemented this first, because it reuses the seeded bootstrap code. It fails in the wrong direction with few games: when every game happens to have the same rate (3 rejections in 6 turns, four times), every resample gives exactly 50%, a zero-width interval, which is far more confident than 24 independent turns would justify. +3. **Wilson interval with a design effect** (chosen for the arena). The design effect comes from the standard cluster-robust (linearised) variance of a ratio, which I checked against statsmodels' cluster-robust variance for an intercept-only regression. Flooring it at 1 means games that look alike never make the interval narrower than Wilson's over independent turns, while games that differ widen it. With 4 to 8 games the design effect is itself noisy, so the interval is a guide, not a guarantee. +4. **For the tournament, place each record in a slot by its schedule index inside the worker pool.** That fixes the browser, but any other caller could still pass records in another order. Sorting inside the analysis fixes every caller, including CSV export. + +## Why + +An interval is only as honest as its unit of analysis. Positions in the alpha-beta study, games in the arena and boards in the A\* study are the units drawn independently; turns within a game are not. A summary that is a range of averages should not be read as a range for single cases, especially when the underlying distribution is all-or-nothing. And a seed is only useful if it reproduces the numbers on the page, not just the games. + +## What happened + +- **Alpha-beta:** 444 of the 480 searches (92.5%) visited every node of the full tree. The 36 that pruned anything visited between 0.6% and 92% of it (canonical order down to 0.6%, shuffled order down to 12.5%). The averages, 92% to 100% per board size, depth and move order, are unchanged; so is the textbook comparison (17% to 45% on average). The weak number stands: the original's pruning saves almost nothing in almost every search. +- **LLM arena:** in a test with 12 rejected first answers in 24 turns, bunched in 2 of 4 games, the old Wilson interval was 31% to 69%. The design effect is 8, so the 24 turns carry about as much information as 3 independent ones, and the interval becomes 13% to 87%: that is what four games can say about a model that fails in some games and not others. With rejections spread evenly, the interval is the old one. The paired column makes the comparison DR-004 promised; with 4 games, McNemar can only flag a very large difference, and the caption says so. There are no real-provider results yet to compare before and after. +- **Tournament:** the same seed now gives identical intervals and an identical games CSV however the workers finish (tested by shuffling records). Two browser runs at seed 4399 both gave fixed depth 2 an interval of [202.7, 437.3]; the point estimates (302.4 for depth 2, 246.3 for the original, 237.1 for greedy) were already stable and did not change. The reference run's intervals, and every number quoted from it in the agent card and DR-004, are unchanged, because it was played in schedule order. + +## What I'd change + +- Write the unit of analysis down in the analysis plan before collecting data, alongside the sample size. +- Report the alpha-beta distribution as a small histogram on `/tournament` rather than in prose. +- For the arena, report rejected answers per game as well (the share of games with any rejection), which needs no clustering argument at all. +- Run enough games that the design effect can be estimated with some confidence (at least 10 to 20), and say so in the arena's default settings. diff --git a/docs/decisions/DR-006-grounded-commentary-audit.md b/docs/decisions/DR-006-grounded-commentary-audit.md new file mode 100644 index 0000000..f1b005b --- /dev/null +++ b/docs/decisions/DR-006-grounded-commentary-audit.md @@ -0,0 +1,43 @@ +# DR-006: Ground commentary in what the search found, and keep the check with the decision + +- **Decision:** the move commentator is told what the score means at the depth searched (the feature breakdown explains the score only for a one-ply search, and a forced win is the reason for a move, not the features), its automatic grounding check tests that, and the check's result is stored in the audit-log entry for the call, so it is shown and exported next to the human decision. +- **Status:** accepted on 6 October 2026, after a second review of the 2026 evaluation (PR #12). +- **Supersedes:** nothing. It extends the human-in-the-loop design in the AI use statement. + +## Context + +The commentator turns the agent's own numbers for one move into plain English, and a grounding check compares the reply with those numbers. A review found three gaps. + +1. **The facts mixed two scopes.** They paired the chosen score, which at depth 2 or more is the backed-up minimax value (or plus or minus infinity for a forced win), with a feature breakdown of the position one ply after the move. The prompt asked the model to explain "which evaluation features pushed the score". On 4 × 4, seed 76, the original agent as Red searched 3 plies and found a forced win, while the one-ply features summed to −12, favouring Blue. A model that followed the prompt would attribute a forced win to features that point the other way, and every check would still pass, because every number and move it cited was in the input. +2. **The shared feature could not be described correctly.** The empty-hex term adds the same amount whoever moved, so its contribution is always positive. A model that called it neutral, which is right, failed the direction check. +3. **The check was not in the audit trail.** It was shown in the panel and then discarded. A reviewer deciding later on `/ai-log` never saw it, and an exported log could not show that accepted commentary had failed its check, which is the evidence the human-in-the-loop claim rests on. + +## Decision + +- Add `score_scope` to the facts: one sentence, chosen by the depth and score, that says how the score relates to the features. Add `search.forced_win` (Red, Blue or none), and a note on the shared feature saying it is neutral and does not separate the candidates. +- Rewrite the prompt so the model reads `score_scope` first: report a forced win plainly, never present the features as the source of a depth-2+ score, and explain the features only for a one-ply search. +- Add a fifth check, "Forced wins reported as the search found them": the summary must mention a forced win when the search found one, must not give it to the wrong player, and must not claim one the search did not find (a denial such as "no forced win" is fine). Accept neutral for shared features. +- Store `{ passed, checks }` in the audit entry, written once with the output by the call itself, not by the panel afterwards. Export it as `grounding_passed` and `grounding_failures` in CSV and in full in JSON. On `/ai-log`, show it above the review buttons (open when it failed), flag failed entries next to the decision, and count them. Entries logged before this change are re-checked from the logged prompt, labelled as recomputed, and the recomputed check is stored when the reviewer decides. + +## Options considered + +1. **Drop the feature breakdown for depth 2+.** Removes the mismatch, but also removes the only facts that describe the position, and the breakdown is still true of the position after the move. +2. **Send the features of the position at the end of the principal variation.** That is what the score actually summarises, but the agent does not record its principal variation, and changing the agent's search to record it would change the code that is being explained. +3. **Label the scope and check it** (chosen). The facts stay the agent's own numbers, the model is told what they mean, and the check catches the specific failure the review found. +4. **Store the check in the panel's state only, and add a field later.** That is how it was, and it is why a later reviewer could accept failed commentary without knowing. + +## Why + +A grounding check that only verifies that cited numbers exist in the input can pass a confidently wrong explanation built from true numbers. The facts have to say what the numbers mean, and the check has to test the claim that matters most, here whether the move was a forced win. And an audit trail is only evidence of human oversight if it records what the human saw when deciding, including an automated warning they chose to override. + +## What happened + +- The seed-76 position is now a regression test. A reply that explains the move through the features alone passes the old four checks and fails the new one; a reply that reports the forced win for Red passes; one that gives it to Blue fails. +- The check is stored with every new commentary call and appears in both exports; a test accepts commentary that failed the wrong-player check and confirms the CSV row says so. +- The fifth check is narrow by design. It looks for "forced win", "forces a win" and similar phrases, so a model that describes a forced win in other words would fail it and need a human decision; I would rather have that false alarm than a silent pass. +- None of this has been run against a real provider yet; it is tested against mocked replies. How often real models trip the new check is unknown. + +## What I'd change + +- Record the agent's principal variation, so the commentator can describe the position the score actually comes from (option 2), as a new agent variant rather than a change to the original. +- Run a small labelled evaluation of the grounding check itself (replies written to pass and to fail), and report its false-alarm and miss rates with intervals. diff --git a/docs/model-card.md b/docs/model-card.md new file mode 100644 index 0000000..cc2d6a1 --- /dev/null +++ b/docs/model-card.md @@ -0,0 +1,77 @@ +# Agent card: `_4399` minimax agent for Cachex + +A model card for a game-playing agent with no learned parameters. Numbers are from the +reference studies on the website (`/tournament`), which list seeds, sample sizes and how +each interval was computed. Last updated 6 October 2026. + +## Overview + +| Item | Detail | +| --- | --- | +| **What it is** | Minimax with alpha-beta pruning over a hand-tuned six-feature evaluation, plus a two-move opening book and an instant-win check | +| **Authors** | Team _4399: Sunchuangyu "Rin" Huang and Wei Zhao (COMP30024, University of Melbourne, Semester 1 2022) | +| **Versions** | Original Python (`coursework/Project Part B/code/_4399`), unchanged. TypeScript port (`web/src/lib/agent`), verified move for move against the original by parity tests. | +| **Learned parameters** | None. Six weights chosen by hand (`weights.json`). See DR-001. | +| **Search depth** | 1 ply while at least 15% of cells are empty, then 2, 3, 4. See DR-002. | + +## Intended use + +- Teaching and demonstration: playing Cachex in the browser, watching agents play, and stepping through how a classical search agent scores moves. +- A reproducible baseline for experiments on Cachex (the tournament harness, the LLM-as-a-player evaluation). + +**Not intended for:** competitive play claims, or as a reference for strong Hex or Cachex play. It is a student project and, as measured below, a modest one. + +## Training data and provenance + +There is no training data. The evaluation weights were set by the two authors by hand during development in 2022 ("team empirical experience" in the code); the records of how are not in the repository. The opening book (always cell (1, 1), stealing it as Blue) is likewise hand-written. + +Evaluation data on this card is synthetic: games between agents generated by seeded random number generators, produced by `web/scripts/generate-reference-studies.ts` (TypeScript port) and `scripts/benchmark_agent.py` and `scripts/crosscheck_tournament.py` (original Python). + +## Evaluation + +All intervals are 95%. Win rates use Wilson score intervals; strengths are Bradley-Terry fits on the Elo scale with the random agent fixed at 0, with percentile intervals from 1,000 bootstrap resamples of games within each pairing and board size (one virtual drawn game per pairing as a regulariser). + +**Original benchmark (Python, unchanged).** Against the subject's random agent on 4 × 4 to 7 × 7, 20 seeded games per size and colour: 144 wins in 160 games, **90.0% (84.4% to 93.8%)**. + +**Reference round robin (TypeScript port).** 1,200 games on 4 × 4, 5 × 5 and 6 × 6; every pairing played 40 games per size as 20 colour-swapped pairs; seed 2022. + +| Agent | Elo vs random (95% CI) | Win rate over all its games (95% CI) | +| --- | --- | --- | +| Fixed depth 3 (variant) | 490 (432 to 559) | 76.0% (72.0% to 79.6%) | +| Fixed depth 2 (variant) | 369 (321 to 427) | 58.3% (53.9% to 62.7%) | +| **Original (dynamic depth)** | **339 (290 to 396)** | **53.8% (49.3% to 58.2%)** | +| Greedy one-ply (variant) | 324 (274 to 386) | 51.5% (47.0% to 55.9%) | +| Random | 0 (reference) | 10.2% (7.8% to 13.2%) | + +Head to head, the original beat random in 106 of 120 games (88.3%, 81.4% to 92.9%), drew level with greedy one-ply (61 to 59; 50.8%, 42.0% to 59.6%) and lost to fixed depth 3 in 84 of 120 (its win rate 30.0%, 22.5% to 38.7%). Compared directly, from the same bootstrap refits (the two strengths come from one fit, so their separate intervals are correlated and overlap is not a test): the original minus greedy is +15 Elo (95% CI −24 to +56), so no difference is detectable, while fixed depth 3 minus the original is +151 Elo (106 to 195). Red, the first mover, won 50.3% of all games (95% CI 47.5% to 53.2%): no first-move advantage was detectable in these games, which all allow STEAL; a no-STEAL condition was not tested. + +**Cross-check.** The pairings feasible in Python (boards 4 and 5, 80 games each) were re-run from the unchanged original; all six win-rate differences against the TypeScript run have Newcombe 95% intervals that contain 0. That is no detectable disagreement, not proof of agreement: with 80 games per pairing the check can only detect differences larger than about ±15 percentage points. + +**Search efficiency.** On 120 mid-game positions, each searched at depths 2 and 3 in two move orders (480 searches), the original alpha-beta visits on average 92% to 100% of the nodes plain minimax visits, per board size, depth and move order (bootstrap intervals on `/tournament`); a textbook beta update would visit 17% to 45% on average. The averages hide the shape: 444 of the 480 searches visited every node, and the 36 that pruned anything visited between 0.6% and 92% of the tree. Because of the beta bug, a cut-off can only happen once the search finds a line it scores as a forced win for Red (DR-005). All 480 searches returned the same root value as plain minimax. + +## Known weaknesses and failure modes + +- **Effectively one ply deep.** The deeper search triggers on only 0.7% of searched moves (34 of 4,602), because most games end before 85% of the board is full. In practice the agent is greedy one-ply plus an opening book, and plays like it. +- **Does not block one-move wins.** While at least 15% of cells are empty it searches one ply, and its instant-win check looks only for its own wins, so it never considers the opponent's next move. A scripted Blue that just plays the first empty cell in (r, q) order (filling row 0 left to right) beats it as Red in 50 of 50 seeded games on 4 × 4, 43 of 50 on 5 × 5 and 34 of 50 on 6 × 6 (seeds 0 to 49). In 124 of those 127 losses the agent had left Blue a one-move win that a single Red move could have blocked. Reproduce with `cd web && pnpm test` (the "first-legal-cell" tests in `src/lib/ai/ai.test.ts`). The LLM arena includes this scripted line as a baseline, so a model's wins can be judged against it. +- **Pruning bug.** The minimising branch's `if beta <= min_score: beta = min_score` never narrows the window (beta stays at +infinity), so alpha-beta prunes only after finding a forced win for Red and otherwise saves nothing. Correct results, wasted work. +- **No notion of connection.** None of the six features measures progress towards linking the agent's edges, which is the goal of the game. A path-distance feature exists in the code but is switched off. +- **Rim-favouring positional weights.** The score matrix rates the rim above the centre, against standard Hex opening advice; untested either way. +- **Rigid opening.** Always (1, 1) (or steal (1, 1)); easy to anticipate. On 3 × 3 the book plays fixed cells. +- **Tie-breaking noise.** A random ×(1 + 10⁻⁵) bias picks among equal scores, so play varies between seeds even in identical positions. +- **Scale of evidence.** Strength estimates come from 4 × 4 to 6 × 6 boards. Nothing here supports claims about larger boards, where depth 1 is likely to be relatively weaker still. + +## Ethical and practical considerations + +- Low-stakes domain: a board game, no personal data, no decisions about people. +- Academic integrity: the original is preserved for reference; current COMP30024 students should not copy it. +- Compute: the agent itself is light (about 0.1 ms per move in the browser). The optional LLM features use the visitor's own API key and are billed to them; the site shows token usage for every call. +- Honest reporting: variants exist to explain the original, not to replace it. The site keeps the agent exactly as submitted and documents its flaws instead of silently fixing them. + +## How to reproduce + +```bash +uv run scripts/benchmark_agent.py # original Python vs random (144/160) +uv run scripts/crosscheck_tournament.py # original Python, feasible pairings +cd web && pnpm gen:reference # TypeScript reference studies +cd web && pnpm test # parity, statistics and harness tests +``` diff --git a/docs/showcase/01-landing-light.png b/docs/showcase/01-landing-light.png new file mode 100644 index 0000000..7d7b2dc Binary files /dev/null and b/docs/showcase/01-landing-light.png differ diff --git a/docs/showcase/02-landing-dark.png b/docs/showcase/02-landing-dark.png new file mode 100644 index 0000000..1e2b8bc Binary files /dev/null and b/docs/showcase/02-landing-dark.png differ diff --git a/docs/showcase/03-play.png b/docs/showcase/03-play.png new file mode 100644 index 0000000..e9eac86 Binary files /dev/null and b/docs/showcase/03-play.png differ diff --git a/docs/showcase/04-spectate.png b/docs/showcase/04-spectate.png new file mode 100644 index 0000000..091c4c9 Binary files /dev/null and b/docs/showcase/04-spectate.png differ diff --git a/docs/showcase/05-astar-lab.png b/docs/showcase/05-astar-lab.png new file mode 100644 index 0000000..01a839f Binary files /dev/null and b/docs/showcase/05-astar-lab.png differ diff --git a/docs/showcase/06-astar-paired-study.png b/docs/showcase/06-astar-paired-study.png new file mode 100644 index 0000000..6c03e0f Binary files /dev/null and b/docs/showcase/06-astar-paired-study.png differ diff --git a/docs/showcase/07-tournament.png b/docs/showcase/07-tournament.png new file mode 100644 index 0000000..5d235c4 Binary files /dev/null and b/docs/showcase/07-tournament.png differ diff --git a/docs/showcase/08-ai-settings.png b/docs/showcase/08-ai-settings.png new file mode 100644 index 0000000..fd51f5f Binary files /dev/null and b/docs/showcase/08-ai-settings.png differ diff --git a/docs/showcase/09-llm-arena.png b/docs/showcase/09-llm-arena.png new file mode 100644 index 0000000..0f36dbb Binary files /dev/null and b/docs/showcase/09-llm-arena.png differ diff --git a/docs/showcase/10-methods.png b/docs/showcase/10-methods.png new file mode 100644 index 0000000..89ae3b5 Binary files /dev/null and b/docs/showcase/10-methods.png differ diff --git a/docs/showcase/11-mobile-landing.png b/docs/showcase/11-mobile-landing.png new file mode 100644 index 0000000..36b448f Binary files /dev/null and b/docs/showcase/11-mobile-landing.png differ diff --git a/docs/showcase/12-mobile-play.png b/docs/showcase/12-mobile-play.png new file mode 100644 index 0000000..9870259 Binary files /dev/null and b/docs/showcase/12-mobile-play.png differ diff --git a/docs/showcase/13-mobile-astar.png b/docs/showcase/13-mobile-astar.png new file mode 100644 index 0000000..b22faba Binary files /dev/null and b/docs/showcase/13-mobile-astar.png differ diff --git a/docs/showcase/astar-lab.gif b/docs/showcase/astar-lab.gif new file mode 100644 index 0000000..95a2d94 Binary files /dev/null and b/docs/showcase/astar-lab.gif differ diff --git a/docs/showcase/play-the-agent.gif b/docs/showcase/play-the-agent.gif new file mode 100644 index 0000000..cd36ca3 Binary files /dev/null and b/docs/showcase/play-the-agent.gif differ diff --git a/docs/showcase/tournament-llm-eval.gif b/docs/showcase/tournament-llm-eval.gif new file mode 100644 index 0000000..d84b9ba Binary files /dev/null and b/docs/showcase/tournament-llm-eval.gif differ diff --git a/scripts/crosscheck_tournament.py b/scripts/crosscheck_tournament.py new file mode 100644 index 0000000..cb38bb0 --- /dev/null +++ b/scripts/crosscheck_tournament.py @@ -0,0 +1,151 @@ +# /// script +# requires-python = ">=3.12,<3.13" +# dependencies = ["numpy>=1.26", "scipy>=1.11"] +# /// +""" +Cross-check the tournament harness against the ORIGINAL Python agents. + +The reference tournament on /tournament is played by the parity-tested +TypeScript port, because depth-3 search in the original Python (deepcopy of +the board at every node) is far too slow for 1,200 games. This script replays +the subset that IS feasible in Python, with the same protocol: + + * agents built from the original code with exactly one knob changed: + - minimax-dynamic: _4399.Player, unchanged; + - minimax-d2: dynamic_depth_allocation patched to return 2; + - greedy: dynamic_depth_allocation patched to return 1 and + action(enforceGamePlayer=False) (no opening book); + - random: random_play_agent, unchanged; + * every pairing plays colour-swapped pairs of games, both games of a pair + seeded identically (Python's global `random`); + * the subject's referee `Game` validates every action. + +The RNG streams differ from the TypeScript run (Python's Mersenne Twister vs +mulberry32), so individual games differ; the comparison is between win +rates, with intervals, on the web page. + +Like benchmark_agent.py it re-executes itself with PYTHONHASHSEED=0 so the +set iteration order inside get_valid_actions is reproducible. + + uv run scripts/crosscheck_tournament.py +""" + +from __future__ import annotations + +import json +import logging +import os +import platform +import random +import sys +import time +from itertools import combinations +from pathlib import Path + +ROOT = Path(__file__).resolve().parents[1] +PART_B = ROOT / "coursework" / "Project Part B" / "code" +SKELETON = ROOT / "coursework" / "Project Part B" / "skeleton-code-B" +OUT = ROOT / "web" / "src" / "lib" / "data" / "python-crosscheck.json" + +AGENTS = ["minimax-dynamic", "minimax-d2", "greedy", "random"] +SIZES = [4, 5] +ROUNDS = 20 # colour-swapped pairs per pairing per size -> 40 games +SEED = 2022 + + +def pin_hash_seed() -> None: + if os.environ.get("PYTHONHASHSEED") != "0": + os.execve( + sys.executable, + [sys.executable, *sys.argv], + {**os.environ, "PYTHONHASHSEED": "0"}, + ) + + +def main() -> None: + pin_hash_seed() + os.chdir(PART_B) # utility/evaluation.py opens ./utility/weights.json + sys.path.insert(0, str(PART_B)) + sys.path.insert(1, str(SKELETON)) + + import _4399.player as player_module # noqa: E402 + from _4399.player import Player as MinimaxPlayer # noqa: E402 + from random_play_agent.player import Player as RandomPlayer # noqa: E402 + from referee.game import Game # noqa: E402 + + logging.getLogger().setLevel(logging.WARNING) + original_depth = player_module.dynamic_depth_allocation + + class Variant(MinimaxPlayer): + """The original player with the depth (and opening book) overridden per call.""" + + depth: int | None = None + book: bool = True + + def action(self): # type: ignore[override] + if self.depth is not None: + player_module.dynamic_depth_allocation = lambda board, d=self.depth: d + try: + return super().action(enforceGamePlayer=self.book) + finally: + player_module.dynamic_depth_allocation = original_depth + + def make(agent: str, colour: str, n: int): + if agent == "random": + return RandomPlayer(colour, n) + p = Variant(colour, n) + if agent == "minimax-d2": + p.depth = 2 + elif agent == "greedy": + p.depth = 1 + p.book = False + return p + + def play(n: int, red: str, blue: str, seed: int) -> dict: + random.seed(seed) + game = Game(n) + players = [make(red, "red", n), make(blue, "blue", n)] + while not game.over(): + colour = "red" if game.nturns % 2 == 0 else "blue" + action = players[game.nturns % 2].action() + sanitised = game.update(colour, action) + for p in players: + p.turn(colour, sanitised) + winner = None + if game.result == "winner: red": + winner = "red" + elif game.result == "winner: blue": + winner = "blue" + return {"n": n, "red": red, "blue": blue, "seed": seed, "winner": winner, "turns": game.nturns} + + games = [] + started = time.perf_counter() + for n in SIZES: + for a, b in combinations(AGENTS, 2): + t0 = time.perf_counter() + for r in range(ROUNDS): + seed = SEED * 1_000_003 + n * 10_007 + AGENTS.index(a) * 101 + AGENTS.index(b) * 11 + r + games.append(play(n, a, b, seed)) + games.append(play(n, b, a, seed)) + print(f"n={n} {a} vs {b}: {time.perf_counter() - t0:.1f}s", flush=True) + + OUT.parent.mkdir(parents=True, exist_ok=True) + payload = { + "meta": { + "generatedBy": "scripts/crosscheck_tournament.py", + "implementation": "original Python (coursework/Project Part B), variants by patching one knob", + "python": platform.python_version(), + "pythonHashSeed": os.environ.get("PYTHONHASHSEED"), + "config": {"agents": AGENTS, "sizes": SIZES, "rounds": ROUNDS, "seed": SEED}, + "seconds": round(time.perf_counter() - started, 1), + }, + "games": games, + } + with open(OUT, "w") as f: + json.dump(payload, f) + f.write("\n") + print(f"wrote {OUT.relative_to(ROOT)}") + + +if __name__ == "__main__": + main() diff --git a/web/.gitignore b/web/.gitignore index c76a96f..d678d51 100644 --- a/web/.gitignore +++ b/web/.gitignore @@ -44,3 +44,9 @@ yarn-error.log* # typescript *.tsbuildinfo next-env.d.ts + +# showcase tour (pnpm showcase): raw recordings, Playwright output, local ffmpeg link +/.showcase +/.playwright +/test-results +/playwright-report diff --git a/web/.prettierignore b/web/.prettierignore index 753f5a6..1e1670d 100644 --- a/web/.prettierignore +++ b/web/.prettierignore @@ -5,3 +5,9 @@ src/lib/__fixtures__ src/components/ui src/lib/agent/weights.json src/lib/data +content/docs +.showcase +.playwright +test-results +playwright-report +public/showcase diff --git a/web/.vercelignore b/web/.vercelignore new file mode 100644 index 0000000..904070d --- /dev/null +++ b/web/.vercelignore @@ -0,0 +1,9 @@ +# Local-only output that `vercel deploy` would otherwise upload (it does not read .gitignore). +# Showcase tour (pnpm showcase): raw recordings, Playwright output, local ffmpeg link +.showcase +.playwright +test-results +playwright-report +# Test and TypeScript caches +.vitest +*.tsbuildinfo diff --git a/web/content/docs/ai-use-statement.md b/web/content/docs/ai-use-statement.md new file mode 100644 index 0000000..7c61c8c --- /dev/null +++ b/web/content/docs/ai-use-statement.md @@ -0,0 +1,73 @@ +# AI use statement + +This site has two optional features that call a large language model. Everything else, +including the agent you play against and every statistic on the site, is ordinary code +that runs without any AI service. This statement is informed by the Australian Government's +policy for the responsible use of AI in government (Digital Transformation Agency), the +transparency principles of the EU AI Act, and the NIST AI Risk Management Framework. It is +not a claim of compliance or certification with any of them. + +## What the AI features do + +1. **Move commentator** (on `/play` and `/spectate`). When the minimax agent has searched for + a move, you can ask a model to explain it in plain English. The model receives only the + agent's own numbers for that move: board size, turn, the move, search depth, nodes visited, + the top candidate scores and the six evaluation features with their weights and + contributions, plus a sentence saying what the score means at that depth: the features + describe the position right after the move, so they explain the score only for a one-ply + search, and a forced win found by a deeper search is the reason for the move, not the + features. It is instructed to use nothing else. +2. **LLM as a player** (on `/llm-arena`). A model plays a few short games against the original + agent. Each turn it receives the rules, the board, the move history and the list of legal + moves, and returns one move as JSON. Answers are scored the same way for both providers, under + the same output-token budget: an illegal move, or a reply with no usable move (malformed, cut + off at the token limit, or refused), is rejected and asked again, up to three times. The + headline is the share of turns whose first answer was rejected, reported next to random, + greedy and scripted baselines on the same seeds. + +## What they never do + +- They never choose the agent's moves, change its evaluation, or feed into the tournament, + the A* study or any other statistic on the site. +- They never run unless you add your own API key and press a button. +- They never send anything to this site's operators. The site is static and has no server; + calls go directly from your browser to the provider you chose. +- They never store your key anywhere except your own browser (session storage by default, + local storage only if you tick "remember on this device"), and never write it to the audit + log or to any file in the repository. + +## Data sent to the provider + +Only the prompt shown in the audit log for that call: game state, rules, legal moves and, for +the commentator, the agent's evaluation breakdown. No personal information is collected or +sent. Your API key goes in the request header to the provider (Anthropic or OpenAI), which +processes the request under its own terms and bills your account. + +## Human in the loop and transparency + +- Every model output on the site is labelled **AI-generated** with the model name. +- Commentary is checked automatically against the facts it was given: cited feature + directions must match the sign of their contribution (a feature shared by both players may be + called neutral), every number and move it mentions must appear in the input, if it says which + player moved that must be the player who did, and a forced win must be reported when the search + found one and never claimed when it did not. The result of that check is shown next to the + text, opened when a check failed, and stored with the call in the audit log. +- You review each commentary and record a decision: accept, edit (your edit is stored next to + the original) or reject. Commentary you did not review where it appeared (for example because + the game moved on) stays "awaiting review" and can be decided later on `/ai-log`, where the + stored check is shown above the decision buttons. +- Every call, successful or not, is recorded in the AI audit log at `/ai-log` with its + timestamp, feature, provider, model, exact input, output (including the raw reply when it + failed validation, was cut off or was refused), latency, token usage when the provider reports + it, the grounding check result, and your decision, so the record shows when commentary was + accepted despite a failed check. Automated evaluation calls (the LLM player) are + marked "n/a". The log stays in your browser and can be exported as JSON or CSV, or cleared. + +## Known limitations + +- A model can still misdescribe the facts in ways the automatic check does not catch (for + example, emphasis or causal language). The check is a guard, not a guarantee; that is why + the human decision exists. +- LLM-as-a-player results from a handful of games have wide intervals and depend on the + model, the prompt and the day. Treat them as a demonstration of the evaluation method, not + as a ranking of models. diff --git a/web/content/docs/decisions/DR-001-evaluation-function.md b/web/content/docs/decisions/DR-001-evaluation-function.md new file mode 100644 index 0000000..1b4d221 --- /dev/null +++ b/web/content/docs/decisions/DR-001-evaluation-function.md @@ -0,0 +1,51 @@ +# DR-001: Evaluation function features and weights + +- **Decision:** score positions with six hand-counted board features and fixed, hand-tuned weights (`weights.json`), from Red's point of view. +- **Status:** accepted in 2022 (team _4399, COMP30024 Part B). Recorded retrospectively on 6 October 2026 from the code; the 2022 report is not in the repository, so anything about our reasoning that the code does not show is marked as such. +- **Supersedes:** nothing. **Superseded by:** nothing yet. + +## Context + +The Part B agent needed a static evaluation for minimax leaves. It had to run inside the referee's CPU-time and memory limits (60 seconds and 100 MB per player when enabled) on Python 3.6, on boards from 3 × 3 to 15 × 15, and be explainable in the report. We had no training data and no self-play infrastructure, and about four weeks. + +## Decision + +`Eval(board)` returns a weighted sum of six features, each counted for Red and subtracted for Blue (except the shared empty-cell count): + +| Feature | Weight | Sign | +| --- | --- | --- | +| Empty hexes (shared) | 0.5 | + | +| Tokens in a solid triangle with two adjacent friendly neighbours | 3 | + | +| Token count | 8 | + | +| Positional value (score matrix: rim high, centre low) | 2 | + | +| Tokens in a half-built diamond the opponent could complete | 4 | − | +| Tokens with an exposed gap (weak formation) | 3 | − | + +A random ×(1 + 10⁻⁵) bias breaks ties between equal scores. The weights live in `coursework/Project Part B/code/utility/weights.json`, which the web port copies verbatim (a test checks the bytes). + +## Options considered + +What the code shows we built or started: + +1. **Hand-tuned linear evaluation over pattern features** (chosen). +2. **A connection-distance feature.** `_4399/eval_func.py` contains `estimate_steps_to_win`, which uses our Part A A* search to estimate how many cells each side still needs. It is written but commented out of the feature list in `utility/evaluation.py`. The code does not record why; cost per leaf is the likely reason, but that was never measured. +3. **Learned weights** (for example, tuning against the random agent or by self-play). Not attempted; nothing in the repository suggests it was. + +## Why + +Pattern counts are cheap (one pass over the board), map directly onto the capture rule (diamonds) and onto shapes that are hard to capture (triangles), and are easy to explain in a report. Fixed weights let us change behaviour quickly by hand during development. The rationale for the specific numbers was "team empirical experience" (the code's own words); there is no record of a systematic search. + +## What happened + +Measured in October 2026 with the parity-tested TypeScript port (reference round robin, 1,200 games on 4 × 4 to 6 × 6, seed 2022; details on `/tournament`): + +- Against the random agent the evaluation is clearly useful: the original agent won 106 of 120 games (88.3%, Wilson 95% CI 81.4% to 92.9%), and greedy one-ply on the same evaluation won 100 of 120 (83.3%, 75.7% to 88.9%). The original Python benchmark (144/160, 90.0%, 84.4% to 93.8%) agrees. +- Against agents that search, lookahead on the same evaluation is what wins: fixed depth 3 beat the original in 84 of 120 games. Used one ply deep, the evaluation does not separate the original from greedy one-ply at all (61 wins to 59). +- It has blind spots that are visible in the feature list. Nothing measures connection progress (the actual goal of the game), and the positional matrix rewards the rim over the centre, which is the opposite of standard Hex opening advice. Neither was tested in 2022. +- By construction, token count carries the largest weight (8 per token), so a capture (two tokens, 16 points) outweighs several shape features at once. Whether that balance is right was never tested. + +## What I'd change + +- Put connection distance back in, measured: add `estimate_steps_to_win` (or a cheaper two-distance or flow measure) and test it in the round-robin harness with colour-swapped pairs and intervals, rather than by feel. +- Tune weights against data instead of by hand: even a simple coordinate search over the six weights, scored by win rate against fixed opponents on held-out seeds, would make the numbers defensible. +- Report evaluation quality directly (for example, how often the evaluation ranks the eventual winner ahead after k moves), not only final win rates. diff --git a/web/content/docs/decisions/DR-002-dynamic-depth-allocation.md b/web/content/docs/decisions/DR-002-dynamic-depth-allocation.md new file mode 100644 index 0000000..4b65e8f --- /dev/null +++ b/web/content/docs/decisions/DR-002-dynamic-depth-allocation.md @@ -0,0 +1,49 @@ +# DR-002: Dynamic depth allocation for minimax + +- **Decision:** search one ply while at least 15% of cells are empty, then deepen to 2, 3 and 4 plies as the empty share falls below 15%, 10% and 5%, after a hard-coded opening and an instant-win check. +- **Status:** accepted in 2022 (team _4399). Recorded retrospectively on 6 October 2026. The agent on the website still plays exactly this. +- **Supersedes:** nothing. **Superseded by:** nothing yet. + +## Context + +The referee could enforce a CPU-time limit per player for the whole game (60 seconds when enabled) on boards up to 15 × 15, and our minimax copied the whole board (`deepcopy`) at every node, so search cost grew quickly with depth and board size. We needed a rule for how deep to search that would not time out on large boards, but would look further ahead when the game was close to decided. + +## Decision + +`dynamic_depth_allocation` in `_4399/player.py`: + +| Empty cells | Depth | +| --- | --- | +| ≥ 15% | 1 | +| 10% to 15% | 2 | +| 5% to 10% | 3 | +| < 5% | 4 | + +Before any search, the agent plays a fixed opening on turns 1 and 2 (always cell (1, 1), stealing it as Blue if Red took it) and takes any immediately winning move. + +## Options considered + +1. **A fixed depth** (2 or 3 throughout). Simple, but at n = 15 a fixed depth 3 with full board copies would not have met the time limit. +2. **Iterative deepening with a time budget.** The robust textbook answer; it needs move ordering and a clock, and we did not build it. +3. **Depth by empty-cell ratio** (chosen): cheap, predictable and safe on large boards. + +## Why + +The ratio rule guaranteed we would not time out on big boards, and the intuition was that the endgame, with few empty cells, is where exact lookahead pays off and is also cheapest. It needed no timing code. + +## What happened + +Measured in October 2026 (reference round robin, 1,200 games on 4 × 4 to 6 × 6, 40 colour-swapped games per pairing per size, seed 2022; alpha-beta study on 120 positions; details on `/tournament`): + +- **The deeper searches almost never run.** Over the whole tournament the original agent searched deeper than one ply on 34 of 4,602 searched moves (0.7%). Most games are decided before fewer than 15% of cells are empty. +- **So the agent is, in practice, greedy one-ply with an opening book.** Head to head with greedy one-ply it won 61 and lost 59 (50.8%, Wilson 95% CI 42.0% to 59.6%). Bradley-Terry strengths (Elo scale, random = 0): original 339 (95% bootstrap CI 290 to 396) against greedy 324 (274 to 386). No detectable difference. +- **Depth does matter.** Fixed depth 3 beat the original in 84 of 120 games (70.0%, 61.3% to 77.5%) and rates 490 Elo (432 to 559). Fixed depth 2 sits between them at 369 (321 to 427), and its head-to-head with the original (65 of 120 for depth 2) is not decisive. +- **Depth is expensive partly because of a bug.** The alpha-beta search visits between 92% and 100% of the nodes that plain minimax visits. In the minimising branch `if beta <= min_score: beta = min_score` can only raise beta, so the window never narrows there. Results are still correct (all 480 test searches returned the same root value as plain minimax), but a textbook update (`beta = min(beta, min_score)`) would visit 18% of the tree instead of 95% on 6 × 6 at depth 3, roughly a fivefold saving. +- The cost of depth 3 in the tournament was about 27 ms per move against 0.1 ms for the original (TypeScript port on an Apple M4; the 2022 Python is much slower). + +## What I'd change + +- Fix the beta update first. It is one line, it keeps results identical, and it makes deeper search several times cheaper. +- Replace the ratio thresholds with iterative deepening under a per-move time budget, with move ordering from the previous iteration, so the agent searches as deep as the clock allows on every move rather than only in the endgame. +- Choose thresholds or budgets by experiment, using this harness: colour-swapped pairs, fixed seeds, Wilson and bootstrap intervals, and a pre-registered comparison against the current agent. +- Keep this record: the original thresholds stay on the site as the "as submitted" agent, and any change ships as a new variant with its own row in the tournament. diff --git a/web/content/docs/decisions/DR-003-typescript-port-web-workers.md b/web/content/docs/decisions/DR-003-typescript-port-web-workers.md new file mode 100644 index 0000000..0d2c5a2 --- /dev/null +++ b/web/content/docs/decisions/DR-003-typescript-port-web-workers.md @@ -0,0 +1,40 @@ +# DR-003: Port to TypeScript and run in Web Workers, verified by parity tests + +- **Decision:** re-implement the original Python (referee rules, Part A A*, Part B agent) line by line in TypeScript, run all search in Web Workers in the browser, and prove equivalence with parity fixtures generated by the unchanged Python. +- **Status:** accepted on revival of the project (2026, Rin). Recorded on 6 October 2026. +- **Supersedes:** nothing. **Superseded by:** nothing yet. + +## Context + +The 2022 code ran only from a terminal against the subject's referee. To make it something people can play, inspect and evaluate, it needed to run on a website. The site should be static (no server to pay for or secure), stay responsive while the agent thinks, and keep the original results faithful: a port that quietly plays differently would make every number on the site meaningless. + +## Decision + +- Port each module with the original's structure and quirks preserved (for example `game_end` ignoring STEAL, the tie-break bias, and the inverted beta update described in DR-002), not a cleaned-up rewrite. +- Generate fixtures with `scripts/generate_parity_fixtures.py`, which imports the unchanged Python and records its outputs, and assert in Vitest that the port reproduces them exactly. +- Where the original depends on CPython internals, simulate them: A* iterates neighbours out of a `set`, so `python-set-order.ts` reproduces CPython 3.12's tuple hashing and set probing to get the same tie-breaking and node counts. +- For randomness, the port uses a seeded generator (mulberry32). Parity tests run the Python with the move shuffle replaced by a canonical sort and the bias fixed at 1, which removes the only difference. +- Run the agent, the A* study and the tournament in Web Workers, with a main-thread fallback. + +## Options considered + +1. **Run the Python in the browser with Pyodide.** Byte-for-byte faithful, but a large download, slow start, and still single-threaded and slow (deep copies at every node). +2. **Host the Python behind an API.** Faithful, but needs a server, adds latency and cost, and makes the static, inspectable site impossible. +3. **Port to TypeScript with parity tests** (chosen). + +## Why + +A port gives speed (the tournament plays 1,200 games in about three minutes on a laptop), a static deployment, and code readers can step through. The risk of a port is silent divergence, and parity fixtures turn that risk into a test: if the port differs from the original on any recorded case, CI fails. + +## What happened + +- The port reproduces the original exactly on everything recorded: paths and node counts for 184 A* runs, every board state and capture across 40 random refereed games, evaluation features on 237 positions, 179 minimax searches, 143 agent moves and 5 full self-play games. +- The tournament harness uses the port because the original is far too slow for it: in Python one 6 × 6 self-play game at fixed depth 2 takes about 40 seconds. To check the harness itself, `scripts/crosscheck_tournament.py` replays the pairings that are feasible in Python (boards 4 and 5, 480 games) from the unchanged code. All six pairings agree within sampling error (differences in win rate from −8.8 to +3.7 percentage points, every Newcombe 95% interval containing 0). With 80 games per pairing that check can only rule out large discrepancies, about ±15 points. +- Faithfulness has a cost: the site plays the original's bugs too (the alpha-beta beta update, rim-favouring positional weights). That is deliberate; they are documented instead of fixed. +- Random play cannot be compared game for game across languages, because the random streams differ. Comparisons are distributional. + +## What I'd change + +- Record the original Python's random draws (or patch its RNG to mulberry32) so whole random games, not just deterministic ones, can be compared move for move. +- Add a property-based test layer (random positions, compare rules and evaluation against the Python on the fly in CI) on top of the fixed fixtures. +- Keep any improved agent strictly separate from the "as submitted" port, as a named variant with its own tests and its own row in the tournament. diff --git a/web/content/docs/decisions/DR-004-paired-comparisons.md b/web/content/docs/decisions/DR-004-paired-comparisons.md new file mode 100644 index 0000000..f13c9cf --- /dev/null +++ b/web/content/docs/decisions/DR-004-paired-comparisons.md @@ -0,0 +1,40 @@ +# DR-004: Compare two methods by their paired difference + +- **Decision:** whenever two methods are measured on the same units, report the difference between them with its own interval (and, for pass/fail outcomes, an exact paired test), rather than two separate intervals whose overlap the reader has to judge. +- **Status:** accepted on 6 October 2026, during review of the 2026 evaluation. Applies to the tournament, the A\* study and the LLM arena. +- **Supersedes:** nothing. It changes how the comparisons behind DR-002 are presented on the site, not what DR-002 records. + +## Context + +The first version of the 2026 evaluation showed some comparisons as two separate 95% intervals. Manhattan's and Euclidean's shortest-path rates were each given a Wilson interval, although both were measured on the same 931 boards. Two agents' Bradley-Terry strengths were compared by whether their intervals overlapped, although both come from one fit anchored to the same agent, so their errors are correlated. A review pointed out that neither display answers the question being asked, which is how far apart the two methods are. + +The LLM arena had a related problem. When a run stopped early (the visitor pressed Stop, or the provider rate-limited the key), the model's row covered only the games it finished, while the baseline rows still covered the whole schedule. The caption said the seeds were the same for every row, and they were not. + +## Decision + +- **Agent strengths:** compare two agents through the difference of their Elo strengths, with a percentile interval taken from the same stratified bootstrap refits as the leaderboard (1,000 resamples, seed 2022 for the reference). The differences are shown in their own table on `/tournament` and in the summary CSV. +- **Pass/fail outcomes on shared units:** report the difference in rates with a paired bootstrap interval that resamples the units (boards), plus an exact McNemar test on the discordant units. The separate Wilson intervals stay, labelled as describing each method on its own. +- **LLM arena:** replay the baselines on exactly the games the model finished, and say so in the caption, including when colours end up unbalanced. Count rejected answers per turn (first answer only), with illegal moves and replies that carry no usable move reported separately, under one rule and one token budget for both providers. + +## Options considered + +1. **Keep the separate intervals and add a note.** Cheap, but it leaves the reader to make a comparison the intervals cannot support: overlapping 95% intervals do not imply no difference, and correlated estimates make overlap even less informative. +2. **Newcombe's or Tango's closed-form interval for a paired difference in proportions.** Good coverage properties, but I could not verify an implementation against a reference package offline, and an unverified formula is worse than none. +3. **Paired bootstrap plus exact McNemar** (chosen). The bootstrap reuses the seeded, tested resampling code, and the exact McNemar test is a two-sided binomial test that I verified against scipy, statsmodels and R. + +## Why + +The design decides the analysis. The A\* study and the tournament were built as paired designs (same boards, same seeds, colour-swapped pairs), so presenting them as if the arms were independent threw away the main advantage of the design. For a pass/fail outcome on shared units, only the units where the methods disagree carry information about the difference, and McNemar's test uses exactly those. For strengths fitted together, the interval of the difference must come from the same replicates, so the correlation is accounted for. + +## What happened + +- **A\*:** on the same boards Manhattan found a shortest path 11.3 percentage points less often than Euclidean (paired 95% bootstrap CI 9.1 to 13.3). Of the 107 boards where only one heuristic was optimal, it was Manhattan on 1 and Euclidean on 106 (exact McNemar p ≈ 1.3 × 10⁻³⁰). The separate intervals pointed the same way; the paired analysis is far sharper, because the 824 boards on which the heuristics agree say nothing about the difference. +- **Tournament:** the original minus greedy one-ply is +15 Elo (95% CI −24 to +56), so DR-002's "no detectable difference" stands, now on the right evidence. Fixed depth 3 minus the original is +151 Elo (106 to 195). Fixed depth 2 minus the original is +30 Elo (−11 to +70): not detectable either way. +- **LLM arena:** a run stopped after one game now compares one game with one game, and the caption says the colours are unbalanced. Truncated and refused replies are no longer counted as illegal moves. +- No original result changed. The per-agent intervals are identical to before, because the bootstrap replicates are the same; only the differences are new. + +## What I'd change + +- Write the comparisons down before running the study, so that the paired analysis is part of the plan rather than added after a review. +- Resample colour-swapped pairs, which are the design unit of the round robin, rather than single games. +- Add a closed-form paired interval (Tango's score interval) as a cross-check once it can be verified against a reference implementation. diff --git a/web/content/docs/decisions/DR-005-report-at-the-sampled-unit.md b/web/content/docs/decisions/DR-005-report-at-the-sampled-unit.md new file mode 100644 index 0000000..ce1fafd --- /dev/null +++ b/web/content/docs/decisions/DR-005-report-at-the-sampled-unit.md @@ -0,0 +1,44 @@ +# DR-005: Report uncertainty at the unit that was sampled + +- **Decision:** every interval accounts for the unit that was actually sampled independently (positions, games, boards), every average is shown with the distribution it summarises when that distribution is lopsided, and every seeded interval is computed on data in a fixed order, so the same seed always gives the same numbers. +- **Status:** accepted on 6 October 2026, after a second review of the 2026 evaluation (PR #12). +- **Supersedes:** the pruning-ratio sentence in DR-002 ("visits between 92% and 100% of the nodes"), and the per-turn interval for the LLM arena described in DR-004. Both records stay as written; this record says what changed and why. + +## Context + +A second review found three places where the numbers on the site were correct but the way they were summarised said more than the data did. + +1. **Alpha-beta.** DR-002, the agent card and the README said the original alpha-beta "visits 92% to 100%" of the full tree. That range is of the *averages* per board size, depth and move order. Individual searches range from under 1% to 100%, and the distribution is all-or-nothing: most searches prune nothing, and a few prune a lot. +2. **LLM arena.** The share of turns whose first answer was rejected had a Wilson interval that treated every turn as an independent trial. Turns in one game share the model, the prompt and an evolving position, and errors tend to repeat, so turns are clustered by game. The arena also showed each baseline's win rate next to the model's, with no paired comparison, although DR-004 lists the arena as a paired design. +3. **Tournament reproducibility.** With the same seed, the in-browser round robin played identical games, but the Bradley-Terry intervals and the order of rows in the games CSV changed between runs. Web Workers return games in the order they finish, and the seeded bootstrap indexes into the arrays in that order. Three runs at seed 4399 gave three different intervals for fixed depth 2: [199.2, 443.3], [187.9, 446.2] and [192.1, 437.2]. + +## Decision + +- **Alpha-beta:** report the per-position distribution next to the averages (how many searches visited every node, and the range among those that pruned anything), and state the mechanism: with the bug, beta never drops below +infinity, so a cut-off can only happen once the search finds a line it scores as a forced win for Red. +- **LLM arena, rejected answers:** a Wilson interval on the effective number of turns, n / deff, where the design effect deff is the cluster-robust variance of the pooled rate (games as clusters) over its variance under independent turns, floored at 1. With one game the design effect cannot be estimated, and the site says the interval then treats turns as independent. +- **LLM arena, versus baselines:** for each baseline, the games won by only the model and by only the baseline, on the same seeds and colours, with an exact McNemar test. +- **Tournament:** sort game records into schedule order (board size, pairing, round, colour leg) before analysis and export. A run played one game at a time, like the reference run, is already in that order, so its numbers do not change. + +## Options considered + +1. **Keep the old summaries and add footnotes.** Cheapest, but a footnote does not stop a reader quoting "92% to 100%" as the range for any search, or reading a Wilson interval over 24 turns as if it came from 24 independent trials. +2. **Cluster bootstrap over games.** Resample whole games and recompute the pooled rate. I implemented this first, because it reuses the seeded bootstrap code. It fails in the wrong direction with few games: when every game happens to have the same rate (3 rejections in 6 turns, four times), every resample gives exactly 50%, a zero-width interval, which is far more confident than 24 independent turns would justify. +3. **Wilson interval with a design effect** (chosen for the arena). The design effect comes from the standard cluster-robust (linearised) variance of a ratio, which I checked against statsmodels' cluster-robust variance for an intercept-only regression. Flooring it at 1 means games that look alike never make the interval narrower than Wilson's over independent turns, while games that differ widen it. With 4 to 8 games the design effect is itself noisy, so the interval is a guide, not a guarantee. +4. **For the tournament, place each record in a slot by its schedule index inside the worker pool.** That fixes the browser, but any other caller could still pass records in another order. Sorting inside the analysis fixes every caller, including CSV export. + +## Why + +An interval is only as honest as its unit of analysis. Positions in the alpha-beta study, games in the arena and boards in the A\* study are the units drawn independently; turns within a game are not. A summary that is a range of averages should not be read as a range for single cases, especially when the underlying distribution is all-or-nothing. And a seed is only useful if it reproduces the numbers on the page, not just the games. + +## What happened + +- **Alpha-beta:** 444 of the 480 searches (92.5%) visited every node of the full tree. The 36 that pruned anything visited between 0.6% and 92% of it (canonical order down to 0.6%, shuffled order down to 12.5%). The averages, 92% to 100% per board size, depth and move order, are unchanged; so is the textbook comparison (17% to 45% on average). The weak number stands: the original's pruning saves almost nothing in almost every search. +- **LLM arena:** in a test with 12 rejected first answers in 24 turns, bunched in 2 of 4 games, the old Wilson interval was 31% to 69%. The design effect is 8, so the 24 turns carry about as much information as 3 independent ones, and the interval becomes 13% to 87%: that is what four games can say about a model that fails in some games and not others. With rejections spread evenly, the interval is the old one. The paired column makes the comparison DR-004 promised; with 4 games, McNemar can only flag a very large difference, and the caption says so. There are no real-provider results yet to compare before and after. +- **Tournament:** the same seed now gives identical intervals and an identical games CSV however the workers finish (tested by shuffling records). Two browser runs at seed 4399 both gave fixed depth 2 an interval of [202.7, 437.3]; the point estimates (302.4 for depth 2, 246.3 for the original, 237.1 for greedy) were already stable and did not change. The reference run's intervals, and every number quoted from it in the agent card and DR-004, are unchanged, because it was played in schedule order. + +## What I'd change + +- Write the unit of analysis down in the analysis plan before collecting data, alongside the sample size. +- Report the alpha-beta distribution as a small histogram on `/tournament` rather than in prose. +- For the arena, report rejected answers per game as well (the share of games with any rejection), which needs no clustering argument at all. +- Run enough games that the design effect can be estimated with some confidence (at least 10 to 20), and say so in the arena's default settings. diff --git a/web/content/docs/decisions/DR-006-grounded-commentary-audit.md b/web/content/docs/decisions/DR-006-grounded-commentary-audit.md new file mode 100644 index 0000000..f1b005b --- /dev/null +++ b/web/content/docs/decisions/DR-006-grounded-commentary-audit.md @@ -0,0 +1,43 @@ +# DR-006: Ground commentary in what the search found, and keep the check with the decision + +- **Decision:** the move commentator is told what the score means at the depth searched (the feature breakdown explains the score only for a one-ply search, and a forced win is the reason for a move, not the features), its automatic grounding check tests that, and the check's result is stored in the audit-log entry for the call, so it is shown and exported next to the human decision. +- **Status:** accepted on 6 October 2026, after a second review of the 2026 evaluation (PR #12). +- **Supersedes:** nothing. It extends the human-in-the-loop design in the AI use statement. + +## Context + +The commentator turns the agent's own numbers for one move into plain English, and a grounding check compares the reply with those numbers. A review found three gaps. + +1. **The facts mixed two scopes.** They paired the chosen score, which at depth 2 or more is the backed-up minimax value (or plus or minus infinity for a forced win), with a feature breakdown of the position one ply after the move. The prompt asked the model to explain "which evaluation features pushed the score". On 4 × 4, seed 76, the original agent as Red searched 3 plies and found a forced win, while the one-ply features summed to −12, favouring Blue. A model that followed the prompt would attribute a forced win to features that point the other way, and every check would still pass, because every number and move it cited was in the input. +2. **The shared feature could not be described correctly.** The empty-hex term adds the same amount whoever moved, so its contribution is always positive. A model that called it neutral, which is right, failed the direction check. +3. **The check was not in the audit trail.** It was shown in the panel and then discarded. A reviewer deciding later on `/ai-log` never saw it, and an exported log could not show that accepted commentary had failed its check, which is the evidence the human-in-the-loop claim rests on. + +## Decision + +- Add `score_scope` to the facts: one sentence, chosen by the depth and score, that says how the score relates to the features. Add `search.forced_win` (Red, Blue or none), and a note on the shared feature saying it is neutral and does not separate the candidates. +- Rewrite the prompt so the model reads `score_scope` first: report a forced win plainly, never present the features as the source of a depth-2+ score, and explain the features only for a one-ply search. +- Add a fifth check, "Forced wins reported as the search found them": the summary must mention a forced win when the search found one, must not give it to the wrong player, and must not claim one the search did not find (a denial such as "no forced win" is fine). Accept neutral for shared features. +- Store `{ passed, checks }` in the audit entry, written once with the output by the call itself, not by the panel afterwards. Export it as `grounding_passed` and `grounding_failures` in CSV and in full in JSON. On `/ai-log`, show it above the review buttons (open when it failed), flag failed entries next to the decision, and count them. Entries logged before this change are re-checked from the logged prompt, labelled as recomputed, and the recomputed check is stored when the reviewer decides. + +## Options considered + +1. **Drop the feature breakdown for depth 2+.** Removes the mismatch, but also removes the only facts that describe the position, and the breakdown is still true of the position after the move. +2. **Send the features of the position at the end of the principal variation.** That is what the score actually summarises, but the agent does not record its principal variation, and changing the agent's search to record it would change the code that is being explained. +3. **Label the scope and check it** (chosen). The facts stay the agent's own numbers, the model is told what they mean, and the check catches the specific failure the review found. +4. **Store the check in the panel's state only, and add a field later.** That is how it was, and it is why a later reviewer could accept failed commentary without knowing. + +## Why + +A grounding check that only verifies that cited numbers exist in the input can pass a confidently wrong explanation built from true numbers. The facts have to say what the numbers mean, and the check has to test the claim that matters most, here whether the move was a forced win. And an audit trail is only evidence of human oversight if it records what the human saw when deciding, including an automated warning they chose to override. + +## What happened + +- The seed-76 position is now a regression test. A reply that explains the move through the features alone passes the old four checks and fails the new one; a reply that reports the forced win for Red passes; one that gives it to Blue fails. +- The check is stored with every new commentary call and appears in both exports; a test accepts commentary that failed the wrong-player check and confirms the CSV row says so. +- The fifth check is narrow by design. It looks for "forced win", "forces a win" and similar phrases, so a model that describes a forced win in other words would fail it and need a human decision; I would rather have that false alarm than a silent pass. +- None of this has been run against a real provider yet; it is tested against mocked replies. How often real models trip the new check is unknown. + +## What I'd change + +- Record the agent's principal variation, so the commentator can describe the position the score actually comes from (option 2), as a new agent variant rather than a change to the original. +- Run a small labelled evaluation of the grounding check itself (replies written to pass and to fail), and report its false-alarm and miss rates with intervals. diff --git a/web/content/docs/model-card.md b/web/content/docs/model-card.md new file mode 100644 index 0000000..cc2d6a1 --- /dev/null +++ b/web/content/docs/model-card.md @@ -0,0 +1,77 @@ +# Agent card: `_4399` minimax agent for Cachex + +A model card for a game-playing agent with no learned parameters. Numbers are from the +reference studies on the website (`/tournament`), which list seeds, sample sizes and how +each interval was computed. Last updated 6 October 2026. + +## Overview + +| Item | Detail | +| --- | --- | +| **What it is** | Minimax with alpha-beta pruning over a hand-tuned six-feature evaluation, plus a two-move opening book and an instant-win check | +| **Authors** | Team _4399: Sunchuangyu "Rin" Huang and Wei Zhao (COMP30024, University of Melbourne, Semester 1 2022) | +| **Versions** | Original Python (`coursework/Project Part B/code/_4399`), unchanged. TypeScript port (`web/src/lib/agent`), verified move for move against the original by parity tests. | +| **Learned parameters** | None. Six weights chosen by hand (`weights.json`). See DR-001. | +| **Search depth** | 1 ply while at least 15% of cells are empty, then 2, 3, 4. See DR-002. | + +## Intended use + +- Teaching and demonstration: playing Cachex in the browser, watching agents play, and stepping through how a classical search agent scores moves. +- A reproducible baseline for experiments on Cachex (the tournament harness, the LLM-as-a-player evaluation). + +**Not intended for:** competitive play claims, or as a reference for strong Hex or Cachex play. It is a student project and, as measured below, a modest one. + +## Training data and provenance + +There is no training data. The evaluation weights were set by the two authors by hand during development in 2022 ("team empirical experience" in the code); the records of how are not in the repository. The opening book (always cell (1, 1), stealing it as Blue) is likewise hand-written. + +Evaluation data on this card is synthetic: games between agents generated by seeded random number generators, produced by `web/scripts/generate-reference-studies.ts` (TypeScript port) and `scripts/benchmark_agent.py` and `scripts/crosscheck_tournament.py` (original Python). + +## Evaluation + +All intervals are 95%. Win rates use Wilson score intervals; strengths are Bradley-Terry fits on the Elo scale with the random agent fixed at 0, with percentile intervals from 1,000 bootstrap resamples of games within each pairing and board size (one virtual drawn game per pairing as a regulariser). + +**Original benchmark (Python, unchanged).** Against the subject's random agent on 4 × 4 to 7 × 7, 20 seeded games per size and colour: 144 wins in 160 games, **90.0% (84.4% to 93.8%)**. + +**Reference round robin (TypeScript port).** 1,200 games on 4 × 4, 5 × 5 and 6 × 6; every pairing played 40 games per size as 20 colour-swapped pairs; seed 2022. + +| Agent | Elo vs random (95% CI) | Win rate over all its games (95% CI) | +| --- | --- | --- | +| Fixed depth 3 (variant) | 490 (432 to 559) | 76.0% (72.0% to 79.6%) | +| Fixed depth 2 (variant) | 369 (321 to 427) | 58.3% (53.9% to 62.7%) | +| **Original (dynamic depth)** | **339 (290 to 396)** | **53.8% (49.3% to 58.2%)** | +| Greedy one-ply (variant) | 324 (274 to 386) | 51.5% (47.0% to 55.9%) | +| Random | 0 (reference) | 10.2% (7.8% to 13.2%) | + +Head to head, the original beat random in 106 of 120 games (88.3%, 81.4% to 92.9%), drew level with greedy one-ply (61 to 59; 50.8%, 42.0% to 59.6%) and lost to fixed depth 3 in 84 of 120 (its win rate 30.0%, 22.5% to 38.7%). Compared directly, from the same bootstrap refits (the two strengths come from one fit, so their separate intervals are correlated and overlap is not a test): the original minus greedy is +15 Elo (95% CI −24 to +56), so no difference is detectable, while fixed depth 3 minus the original is +151 Elo (106 to 195). Red, the first mover, won 50.3% of all games (95% CI 47.5% to 53.2%): no first-move advantage was detectable in these games, which all allow STEAL; a no-STEAL condition was not tested. + +**Cross-check.** The pairings feasible in Python (boards 4 and 5, 80 games each) were re-run from the unchanged original; all six win-rate differences against the TypeScript run have Newcombe 95% intervals that contain 0. That is no detectable disagreement, not proof of agreement: with 80 games per pairing the check can only detect differences larger than about ±15 percentage points. + +**Search efficiency.** On 120 mid-game positions, each searched at depths 2 and 3 in two move orders (480 searches), the original alpha-beta visits on average 92% to 100% of the nodes plain minimax visits, per board size, depth and move order (bootstrap intervals on `/tournament`); a textbook beta update would visit 17% to 45% on average. The averages hide the shape: 444 of the 480 searches visited every node, and the 36 that pruned anything visited between 0.6% and 92% of the tree. Because of the beta bug, a cut-off can only happen once the search finds a line it scores as a forced win for Red (DR-005). All 480 searches returned the same root value as plain minimax. + +## Known weaknesses and failure modes + +- **Effectively one ply deep.** The deeper search triggers on only 0.7% of searched moves (34 of 4,602), because most games end before 85% of the board is full. In practice the agent is greedy one-ply plus an opening book, and plays like it. +- **Does not block one-move wins.** While at least 15% of cells are empty it searches one ply, and its instant-win check looks only for its own wins, so it never considers the opponent's next move. A scripted Blue that just plays the first empty cell in (r, q) order (filling row 0 left to right) beats it as Red in 50 of 50 seeded games on 4 × 4, 43 of 50 on 5 × 5 and 34 of 50 on 6 × 6 (seeds 0 to 49). In 124 of those 127 losses the agent had left Blue a one-move win that a single Red move could have blocked. Reproduce with `cd web && pnpm test` (the "first-legal-cell" tests in `src/lib/ai/ai.test.ts`). The LLM arena includes this scripted line as a baseline, so a model's wins can be judged against it. +- **Pruning bug.** The minimising branch's `if beta <= min_score: beta = min_score` never narrows the window (beta stays at +infinity), so alpha-beta prunes only after finding a forced win for Red and otherwise saves nothing. Correct results, wasted work. +- **No notion of connection.** None of the six features measures progress towards linking the agent's edges, which is the goal of the game. A path-distance feature exists in the code but is switched off. +- **Rim-favouring positional weights.** The score matrix rates the rim above the centre, against standard Hex opening advice; untested either way. +- **Rigid opening.** Always (1, 1) (or steal (1, 1)); easy to anticipate. On 3 × 3 the book plays fixed cells. +- **Tie-breaking noise.** A random ×(1 + 10⁻⁵) bias picks among equal scores, so play varies between seeds even in identical positions. +- **Scale of evidence.** Strength estimates come from 4 × 4 to 6 × 6 boards. Nothing here supports claims about larger boards, where depth 1 is likely to be relatively weaker still. + +## Ethical and practical considerations + +- Low-stakes domain: a board game, no personal data, no decisions about people. +- Academic integrity: the original is preserved for reference; current COMP30024 students should not copy it. +- Compute: the agent itself is light (about 0.1 ms per move in the browser). The optional LLM features use the visitor's own API key and are billed to them; the site shows token usage for every call. +- Honest reporting: variants exist to explain the original, not to replace it. The site keeps the agent exactly as submitted and documents its flaws instead of silently fixing them. + +## How to reproduce + +```bash +uv run scripts/benchmark_agent.py # original Python vs random (144/160) +uv run scripts/crosscheck_tournament.py # original Python, feasible pairings +cd web && pnpm gen:reference # TypeScript reference studies +cd web && pnpm test # parity, statistics and harness tests +``` diff --git a/web/e2e/mock-ai.ts b/web/e2e/mock-ai.ts new file mode 100644 index 0000000..3e674fb --- /dev/null +++ b/web/e2e/mock-ai.ts @@ -0,0 +1,101 @@ +/** + * A mocked AI provider for the tour. No real key is ever used: the visitor's + * "key" is a placeholder, every request to the provider is intercepted in the + * browser context, and the reply is generated here. + * + * The mocked model plays the first legal cell it is offered (the same rule as + * the scripted baseline), and its reason says that it is a mock, so nothing + * in the recording can be mistaken for a real model's behaviour. + */ +import type { BrowserContext, Request } from "@playwright/test"; + +import { MOCK_MODEL_ID } from "../src/lib/showcase"; + +/** Not a credential: an obviously fake placeholder typed into the BYOK dialog. */ +export const PLACEHOLDER_KEY = "placeholder-not-a-real-key"; + +const CORS = { + "access-control-allow-origin": "*", + "access-control-allow-headers": "*", + "access-control-allow-methods": "POST, OPTIONS", +}; + +/** First "(r, q)" in the prompt's list of legal moves. */ +export function firstLegalCell(prompt: string): [number, number] { + const line = prompt.split("\n").find((l) => l.startsWith("Legal moves")); + const m = line?.match(/\((\d+), (\d+)\)/); + if (!m) throw new Error(`No legal moves in prompt: ${prompt.slice(0, 200)}`); + return [Number(m[1]), Number(m[2])]; +} + +export interface MockAi { + /** Requests that reached the mock (each one an AI call the app made). */ + calls: number; + /** Requests to anything else that carried the placeholder key (must stay empty). */ + leaks: string[]; +} + +export async function mockAiProviders( + context: BrowserContext, + { latencyMs = 350 }: { latencyMs?: number } = {}, +): Promise { + const state: MockAi = { calls: 0, leaks: [] }; + + await context.route("https://api.openai.com/**", async (route) => { + const req = route.request(); + if (req.method() === "OPTIONS") { + await route.fulfill({ status: 204, headers: CORS }); + return; + } + state.calls += 1; + const body = JSON.parse(req.postData() ?? "{}") as { + messages?: { role: string; content: string }[]; + }; + const user = body.messages?.find((m) => m.role === "user")?.content ?? ""; + const [r, q] = firstLegalCell(user); + if (latencyMs > 0) await new Promise((resolve) => setTimeout(resolve, latencyMs)); + await route.fulfill({ + status: 200, + headers: { ...CORS, "content-type": "application/json" }, + body: JSON.stringify({ + id: `chatcmpl-mock-${state.calls}`, + object: "chat.completion", + model: MOCK_MODEL_ID, + choices: [ + { + index: 0, + finish_reason: "stop", + message: { + role: "assistant", + content: JSON.stringify({ + action: "PLACE", + r, + q, + reason: "Mocked response for illustration: the first legal cell.", + }), + }, + }, + ], + // No usage block: the mock has no token counts to report. + }), + }); + }); + + // The tour never uses Anthropic; block it so nothing can leave the browser. + await context.route("https://api.anthropic.com/**", (route) => route.abort("blockedbyclient")); + + context.on("request", (req: Request) => { + if (req.url().startsWith("https://api.openai.com/")) return; + const headers = JSON.stringify(req.headers()); + const data = req.postData() ?? ""; + if ( + headers.includes(PLACEHOLDER_KEY) || + data.includes(PLACEHOLDER_KEY) || + req.url().includes(PLACEHOLDER_KEY) + ) { + state.leaks.push(req.url()); + } + }); + + return state; +} diff --git a/web/e2e/showcase-helpers.ts b/web/e2e/showcase-helpers.ts new file mode 100644 index 0000000..74bf739 --- /dev/null +++ b/web/e2e/showcase-helpers.ts @@ -0,0 +1,426 @@ +/** + * Helpers for the guided tour: an on-screen caption banner and cursor + * highlight (injected into the page, so they appear in the recording), + * human-paced pointer movement, and a cue log that becomes the WebVTT + * captions of each video. + */ +import { mkdirSync, writeFileSync } from "node:fs"; +import path from "node:path"; + +import type { Browser, BrowserContext, Locator, Page } from "@playwright/test"; + +import { MOCK_LABEL, type Walkthrough } from "../src/lib/showcase"; + +/** Raw tour output (screenshots, webm, cue files); post-processed by scripts/showcase-media.mjs. */ +export const RAW_DIR = path.resolve(__dirname, "..", ".showcase"); +export const SHOT_DIR = path.join(RAW_DIR, "screens"); +export const VIDEO_DIR = path.join(RAW_DIR, "video"); + +/** SHOWCASE_FAST=1 runs the journeys as quick end-to-end tests: no pauses, no video. */ +export const FAST = process.env.SHOWCASE_FAST === "1"; +export const RECORD = !FAST && process.env.SHOWCASE_RECORD !== "0"; +const PACE = FAST ? 0 : Number(process.env.SHOWCASE_PACE ?? 1); + +export const VIDEO_SIZE = { width: 1280, height: 800 } as const; +/** Between the sticky site header and the caption banner. */ +const SAFE_AREA = { top: 72, bottom: VIDEO_SIZE.height - 112 } as const; + +export function ensureDirs() { + for (const dir of [RAW_DIR, SHOT_DIR, VIDEO_DIR]) mkdirSync(dir, { recursive: true }); +} + +/** A context for one recorded journey (light theme, 1280 × 800). */ +export async function recordingContext(browser: Browser): Promise { + ensureDirs(); + const context = await browser.newContext({ + viewport: VIDEO_SIZE, + deviceScaleFactor: 1, + colorScheme: "light", + ...(RECORD ? { recordVideo: { dir: path.join(VIDEO_DIR, "tmp"), size: VIDEO_SIZE } } : {}), + }); + await context.addInitScript(overlayInit, { mockLabel: MOCK_LABEL }); + return context; +} + +interface CaptionState { + step: number; + total: number; + text: string; + mock: boolean; +} + +/** + * Runs in the page (as an init script, so it survives navigations). Mounts a + * shadow-DOM overlay with a cursor dot that follows real mouse events and a + * fixed caption banner at the bottom of the viewport. + */ +function overlayInit({ mockLabel }: { mockLabel: string }) { + const HOST_ID = "__showcase-overlay"; + const w = window as unknown as { + __showcaseCaption?: (c: CaptionState | null) => void; + }; + const css = ` + :host { all: initial; } + .cursor { + position: fixed; left: 0; top: 0; width: 26px; height: 26px; margin: -13px 0 0 -13px; + border-radius: 50%; background: rgba(250, 190, 40, 0.28); + border: 2px solid rgba(196, 136, 0, 0.95); + box-shadow: 0 0 0 3px rgba(255, 255, 255, 0.7), 0 2px 10px rgba(0, 0, 0, 0.25); + transition: width .12s, height .12s, margin .12s, background .12s; opacity: 0; + } + .cursor.on { opacity: 1; } + .cursor.down { width: 18px; height: 18px; margin: -9px 0 0 -9px; background: rgba(214, 150, 0, 0.6); } + .ripple { + position: fixed; width: 18px; height: 18px; margin: -9px 0 0 -9px; border-radius: 50%; + border: 2px solid rgba(196, 136, 0, 0.9); animation: ripple .6s ease-out forwards; + } + @keyframes ripple { to { transform: scale(3.4); opacity: 0; } } + .caption { + position: fixed; left: 24px; right: 24px; bottom: 16px; margin: 0 auto; + width: fit-content; max-width: 1180px; box-sizing: border-box; + display: flex; align-items: center; gap: 12px; padding: 10px 16px 10px 10px; + border-radius: 16px; background: rgba(22, 20, 34, 0.93); color: #fff; + border: 1px solid rgba(255, 255, 255, 0.14); box-shadow: 0 12px 34px rgba(0, 0, 0, 0.32); + font: 600 17px/1.35 ui-sans-serif, system-ui, -apple-system, "Segoe UI", sans-serif; + letter-spacing: 0.005em; + } + .text { flex: 1 1 auto; min-width: 0; } + .caption[hidden] { display: none; } + .step { + flex: none; min-width: 44px; padding: 4px 9px; border-radius: 10px; text-align: center; + background: #e8b33a; color: #231a05; font: 700 14px/1.2 ui-monospace, SFMono-Regular, Menlo, monospace; + } + .mock { + flex: none; padding: 4px 9px; border-radius: 10px; background: #fff3d6; color: #5a3b00; + border: 1px dashed #b07a00; font: 700 12px/1.2 ui-sans-serif, system-ui, sans-serif; + text-transform: uppercase; letter-spacing: 0.04em; + } + .mock[hidden] { display: none; } + `; + + let root: ShadowRoot | null = null; + let host: HTMLElement | null = null; + + const readJson = (key: string): T | null => { + try { + return JSON.parse(sessionStorage.getItem(key) ?? "null") as T | null; + } catch { + return null; + } + }; + + const placeCursor = (x: number, y: number) => { + const cursor = root?.querySelector(".cursor"); + if (!cursor) return; + cursor.style.transform = `translate(${x}px, ${y}px)`; + cursor.classList.add("on"); + }; + + const render = (c: CaptionState | null) => { + const box = root?.querySelector(".caption"); + if (!box) return; + if (!c) { + box.hidden = true; + return; + } + box.hidden = false; + box.querySelector(".step")!.textContent = `${c.step}/${c.total}`; + box.querySelector(".text")!.textContent = c.text; + box.querySelector(".mock")!.hidden = !c.mock; + }; + + const mount = () => { + if (host?.isConnected) return; + host = document.createElement("div"); + host.id = HOST_ID; + host.setAttribute("aria-hidden", "true"); + host.style.cssText = "position:fixed;inset:0;pointer-events:none;z-index:2147483647;"; + root = host.attachShadow({ mode: "open" }); + root.innerHTML = `
`; + root.querySelector(".mock")!.textContent = mockLabel; + document.documentElement.appendChild(host); + const pos = readJson<{ x: number; y: number }>("__showcase_cursor"); + if (pos) placeCursor(pos.x, pos.y); + render(readJson("__showcase_caption")); + }; + + w.__showcaseCaption = (c) => { + sessionStorage.setItem("__showcase_caption", JSON.stringify(c)); + mount(); + render(c); + }; + + window.addEventListener( + "mousemove", + (e) => { + placeCursor(e.clientX, e.clientY); + sessionStorage.setItem("__showcase_cursor", JSON.stringify({ x: e.clientX, y: e.clientY })); + }, + { capture: true, passive: true }, + ); + window.addEventListener( + "mousedown", + (e) => { + root?.querySelector(".cursor")?.classList.add("down"); + const ripple = document.createElement("div"); + ripple.className = "ripple"; + ripple.style.left = `${e.clientX}px`; + ripple.style.top = `${e.clientY}px`; + root?.appendChild(ripple); + setTimeout(() => ripple.remove(), 700); + }, + true, + ); + window.addEventListener( + "mouseup", + () => root?.querySelector(".cursor")?.classList.remove("down"), + true, + ); + + if (document.readyState === "loading") document.addEventListener("DOMContentLoaded", mount); + else mount(); + // Hydration may replace foreign nodes; put the overlay back if it goes missing. + setInterval(mount, 250); +} + +interface Cue { + step: number; + text: string; + mock: boolean; + start: number; +} + +const ease = (t: number) => (t < 0.5 ? 2 * t * t : 1 - (-2 * t + 2) ** 2 / 2); + +/** + * Drives one journey: captions, paced pointer movement and the cue log. + * Times are seconds since the page (and so the video) was created. + */ +export class Tour { + private readonly t0 = Date.now(); + private readonly cues: Cue[] = []; + private readonly idle: [number, number][] = []; + /** Scripted scrolls; the GIF drops these frames (a cut instead of a slide). */ + private readonly motion: [number, number][] = []; + private mouse = { x: VIDEO_SIZE.width / 2, y: VIDEO_SIZE.height / 2 }; + private trimStart = 0; + + constructor( + readonly page: Page, + readonly walkthrough: Walkthrough, + ) {} + + private now() { + return (Date.now() - this.t0) / 1000; + } + + async pause(ms: number) { + if (PACE > 0) await this.page.waitForTimeout(Math.round(ms * PACE)); + } + + /** Mark the start of the useful footage (after the first page has rendered). */ + markStart() { + this.trimStart = Math.max(0, this.now() - 0.2); + } + + /** Show step `step` (1-based) of the walkthrough in the caption banner. */ + async caption(step: number) { + const { steps, mockedSteps = [] } = this.walkthrough; + const text = steps[step - 1]; + if (!text) throw new Error(`No step ${step} in ${this.walkthrough.id}`); + const mock = mockedSteps.includes(step); + this.cues.push({ step, text, mock, start: this.now() }); + const state: CaptionState = { step, total: steps.length, text, mock }; + await this.page.evaluate((c) => { + (window as unknown as { __showcaseCaption?: (c: unknown) => void }).__showcaseCaption?.(c); + }, state); + } + + /** Time spent waiting on the app; cut from the GIF to keep it short. */ + async idleWhile(fn: () => Promise): Promise { + const start = this.now(); + const result = await fn(); + this.idle.push([start, this.now()]); + return result; + } + + /** Glide the pointer to (x, y) with an ease-in-out path. */ + async glide(x: number, y: number, ms = 650) { + const from = { ...this.mouse }; + if (PACE === 0) { + await this.page.mouse.move(x, y); + } else { + const steps = Math.max(8, Math.round((ms * PACE) / 16)); + for (let i = 1; i <= steps; i++) { + const t = ease(i / steps); + await this.page.mouse.move(from.x + (x - from.x) * t, from.y + (y - from.y) * t); + await this.page.waitForTimeout(Math.max(1, Math.round((ms * PACE) / steps) - 6)); + } + } + this.mouse = { x, y }; + } + + /** Run a scripted scroll and log its interval. */ + private async scrolling(fn: () => Promise) { + const start = this.now(); + await fn(); + const end = this.now(); + if (end - start > 0.05) this.motion.push([start, end]); + } + + /** Smoothly scroll the window by `delta` px (clamped to the page). */ + private async scrollBy(delta: number, ms = 700) { + await this.scrolling(() => + this.page.evaluate( + async ([delta, ms]) => { + const html = document.documentElement; + const from = window.scrollY; + const to = Math.min(html.scrollHeight - window.innerHeight, Math.max(0, from + delta)); + if (ms <= 0 || Math.abs(to - from) < 1) { + window.scrollTo({ top: to, behavior: "instant" }); + return; + } + const startAt = performance.now(); + await new Promise((resolve) => { + const frame = (t: number) => { + const k = Math.min(1, (t - startAt) / ms); + const e = k < 0.5 ? 2 * k * k : 1 - (-2 * k + 2) ** 2 / 2; + window.scrollTo({ top: from + (to - from) * e, behavior: "instant" }); + if (k < 1) requestAnimationFrame(frame); + else resolve(); + }; + requestAnimationFrame(frame); + }); + }, + [delta, Math.round(ms * PACE)] as const, + ), + ); + } + + /** + * Centre of `target`, scrolling it (smoothly) into the safe area between + * the sticky header and the caption banner first, if needed. + */ + private async centre(target: Locator) { + await target.waitFor({ state: "visible" }); + let box = await target.boundingBox(); + if (!box) throw new Error(`No bounding box for ${target}`); + const { top, bottom } = SAFE_AREA; + if (box.y < top || box.y + box.height > bottom) { + const mid = box.y + box.height / 2; + await this.scrollBy(mid - (top + bottom) / 2); + box = await target.boundingBox(); + } + if (!box || box.y < 0 || box.y + box.height > VIDEO_SIZE.height) { + // Clipped by an inner scroll container (or the window could not move). + await target.scrollIntoViewIfNeeded(); + box = await target.boundingBox(); + } + if (!box) throw new Error(`No bounding box for ${target}`); + return { x: box.x + box.width / 2, y: box.y + box.height / 2 }; + } + + async hover(target: Locator, ms = 650) { + const { x, y } = await this.centre(target); + await this.glide(x, y, ms); + } + + /** Hover something that may already be gone (live UI); never fails the tour. */ + async tryHover(target: Locator, ms = 650) { + try { + await target.waitFor({ state: "visible", timeout: 2000 }); + const box = await target.boundingBox(); + if (box) await this.glide(box.x + box.width / 2, box.y + box.height / 2, ms); + } catch { + // The element changed while we moved; the recording simply keeps going. + } + } + + async click(target: Locator, { before = 250, after = 350 } = {}) { + await target.waitFor({ state: "visible" }); + const { x, y } = await this.centre(target); + await this.glide(x, y); + await this.pause(before); + await this.page.mouse.down(); + await this.page.waitForTimeout(PACE > 0 ? 90 : 10); + await this.page.mouse.up(); + await this.pause(after); + } + + async type(target: Locator, text: string) { + await this.click(target, { after: 150 }); + await target.press("ControlOrMeta+a"); + await this.page.keyboard.type(text, { delay: PACE > 0 ? 45 : 0 }); + await this.pause(300); + } + + /** Smoothly scroll so `target` sits `offset` px below the top of the viewport. */ + async scrollTo(target: Locator, { offset = 84, ms = 1100 } = {}) { + await target.waitFor({ state: "attached" }); + const handle = await target.elementHandle(); + await this.scrolling(() => + this.page.evaluate( + async ([el, offset, ms]) => { + const html = document.documentElement; + const from = window.scrollY; + const to = Math.min( + html.scrollHeight - window.innerHeight, + Math.max(0, (el as Element).getBoundingClientRect().top + from - (offset as number)), + ); + const duration = ms as number; + if (duration <= 0) { + window.scrollTo({ top: to, behavior: "instant" }); + return; + } + const startAt = performance.now(); + await new Promise((resolve) => { + const frame = (t: number) => { + const k = Math.min(1, (t - startAt) / duration); + const e = k < 0.5 ? 2 * k * k : 1 - (-2 * k + 2) ** 2 / 2; + window.scrollTo({ top: from + (to - from) * e, behavior: "instant" }); + if (k < 1) requestAnimationFrame(frame); + else resolve(); + }; + requestAnimationFrame(frame); + }); + }, + [handle, offset, Math.round(ms * PACE)] as const, + ), + ); + await handle?.dispose(); + } + + /** Write the cue file next to the raw video (used for WebVTT, trimming and GIF cuts). */ + save() { + if (!RECORD) return; + const end = this.now(); + const cues = this.cues.map((c, i) => ({ + ...c, + end: i + 1 < this.cues.length ? this.cues[i + 1].start : end, + })); + writeFileSync( + path.join(VIDEO_DIR, `${this.walkthrough.id}.json`), + `${JSON.stringify( + { + id: this.walkthrough.id, + title: this.walkthrough.title, + trimStart: this.trimStart, + end, + cues, + idle: this.idle, + motion: this.motion, + }, + null, + 2, + )}\n`, + ); + } +} + +/** Close the context and move its video to .showcase/video/.webm. */ +export async function finishRecording(context: BrowserContext, page: Page, tour: Tour) { + const video = RECORD ? page.video() : null; + tour.save(); + await context.close(); + if (video) await video.saveAs(path.join(VIDEO_DIR, `${tour.walkthrough.id}.webm`)); +} diff --git a/web/e2e/showcase.spec.ts b/web/e2e/showcase.spec.ts new file mode 100644 index 0000000..6ae32fa --- /dev/null +++ b/web/e2e/showcase.spec.ts @@ -0,0 +1,498 @@ +/** + * The guided tour, as an end-to-end test. + * + * pnpm showcase # production, records media + * BASE_URL=http://localhost:3000 pnpm showcase # a local `pnpm build` first + * SHOWCASE_FAST=1 pnpm showcase:test # journeys only: no pauses, no video + * + * Each journey checks what it shows (the steal, both captures, the recorded + * A* output, the seeded tournament, the mocked LLM run and the audit log), so + * a broken feature fails the tour instead of producing a misleading video. + * Everything is deterministic: the Play page starts at seed 4399, the A* Lab + * uses the original sample input, and the tournament and arena use fixed + * seeds. No real API key is used; AI calls are answered by e2e/mock-ai.ts. + */ +import path from "node:path"; + +import { + type Browser, + type BrowserContext, + type Locator, + type Page, + expect, + test, +} from "@playwright/test"; + +import { MOCK_MODEL_ID, SCREENSHOTS, WALKTHROUGHS, type WalkthroughId } from "../src/lib/showcase"; +import { PLACEHOLDER_KEY, mockAiProviders } from "./mock-ai"; +import { + FAST, + SHOT_DIR, + Tour, + ensureDirs, + finishRecording, + recordingContext, +} from "./showcase-helpers"; + +const walkthrough = (id: WalkthroughId) => WALKTHROUGHS.find((w) => w.id === id)!; + +const cell = (page: Page, r: number, q: number) => + page.getByRole("button", { name: new RegExp(`^Row ${r}, column ${q}:`) }); + +const expectCell = (page: Page, r: number, q: number, content: "red" | "blue" | "empty") => + expect(cell(page, r, q)).toHaveAttribute( + "aria-label", + new RegExp(`^Row ${r}, column ${q}: ${content}`), + ); + +const moveLog = (page: Page) => page.getByRole("list", { name: "Move log" }); + +/** + * The scripted game on /play: 5 × 5, the human is Red, the agent plays Blue + * at the page's initial seed (4399). Found by searching the deterministic + * agent's replies for a short line with a steal, a capture and a recapture. + */ +const PLAY_MOVES = { + open: [1, 1], + first: [0, 1], + diamond: [1, 0], + retake: [0, 1], + recapture: [1, 0], +} as const; + +async function playScriptedGame(page: Page, click: (target: Locator) => Promise) { + await click(cell(page, ...PLAY_MOVES.open)); + await expectCell(page, 1, 1, "blue"); // Blue stole the opening (mirrored onto itself) + await expect(moveLog(page)).toContainText("STEAL"); + await click(cell(page, ...PLAY_MOVES.first)); + await expectCell(page, 4, 1, "blue"); + await click(cell(page, ...PLAY_MOVES.diamond)); + await expectCell(page, 0, 0, "blue"); // Blue captures (0, 1) and (1, 0) + await expectCell(page, 0, 1, "empty"); + await expectCell(page, 1, 0, "empty"); + await click(cell(page, ...PLAY_MOVES.retake)); + await expectCell(page, 1, 4, "blue"); + await click(cell(page, ...PLAY_MOVES.recapture)); + await expectCell(page, 0, 0, "empty"); // Red captures (0, 0) and (1, 1) + await expectCell(page, 1, 1, "empty"); + await expectCell(page, 1, 0, "red"); + await expect(moveLog(page).getByRole("listitem")).toHaveCount(10); +} + +const captureEntry = (page: Page) => + moveLog(page).getByRole("button").filter({ hasText: "(0, 0)" }).filter({ hasText: "captured 2" }); + +async function settle(page: Page) { + await page.evaluate(() => document.fonts.ready); + await page.waitForLoadState("networkidle").catch(() => undefined); +} + +/** + * Viewport screenshot to .showcase/screens/.png, optionally with `align` + * scrolled to `offset` px from the top (applied twice, after layout settles). + */ +async function shot(page: Page, id: string, align?: { target: Locator; offset: number }) { + if (!SCREENSHOTS.some((s) => s.id === id)) throw new Error(`Unknown screenshot ${id}`); + await page.mouse.move(0, 0); + await page.evaluate(() => (document.activeElement as HTMLElement | null)?.blur?.()); + for (let i = 0; i < 2; i++) { + if (align) { + await align.target.evaluate((el, offset) => { + const top = el.getBoundingClientRect().top + window.scrollY - offset; + window.scrollTo({ top, behavior: "instant" }); + }, align.offset); + } + await page.waitForTimeout(450); + } + await page.screenshot({ path: path.join(SHOT_DIR, `${id}.png`) }); +} + +const h1 = (page: Page) => page.getByRole("heading", { level: 1 }); + +test.beforeAll(() => ensureDirs()); + +test.describe("journeys (recorded)", () => { + test("1. play the agent: steal, capture, recapture, why that move", async ({ browser }) => { + const context = await recordingContext(browser); + const page = await context.newPage(); + const tour = new Tour(page, walkthrough("play-the-agent")); + + await page.goto("/play"); + await expect(cell(page, 2, 2)).toBeVisible(); + await settle(page); + tour.markStart(); + + await tour.caption(1); + await tour.pause(1200); + // Board and the coordinates switch below it both fit between header and caption. + await tour.scrollTo(page.getByRole("group", { name: /^Cachex board/ }), { offset: 84 }); + await tour.click(page.getByRole("switch", { name: "Show coordinates" })); + await tour.pause(1200); + + await tour.caption(2); + await tour.pause(600); + await tour.click(cell(page, ...PLAY_MOVES.open), { after: 0 }); + await expectCell(page, 1, 1, "blue"); + + await tour.caption(3); + await tour.pause(900); + await tour.hover(page.getByText("Opening book", { exact: true })); + await tour.pause(2600); + + await tour.caption(4); + await tour.click(cell(page, ...PLAY_MOVES.first), { after: 0 }); + await expectCell(page, 4, 1, "blue"); + await tour.pause(1600); + + await tour.caption(5); + await tour.pause(1200); + await tour.click(cell(page, ...PLAY_MOVES.diamond), { after: 0 }); + await expectCell(page, 0, 0, "blue"); + + await tour.caption(6); + await expectCell(page, 0, 1, "empty"); + await expectCell(page, 1, 0, "empty"); + await tour.pause(2800); + + await tour.caption(7); + await tour.click(cell(page, ...PLAY_MOVES.retake), { after: 0 }); + await expectCell(page, 1, 4, "blue"); + await tour.pause(1400); + + await tour.caption(8); + await tour.pause(800); + await tour.click(cell(page, ...PLAY_MOVES.recapture), { after: 0 }); + await expectCell(page, 0, 0, "empty"); + await expectCell(page, 1, 1, "empty"); + await tour.pause(2600); + await expect(moveLog(page).getByRole("listitem")).toHaveCount(10); + + await tour.caption(9); + await tour.scrollTo(moveLog(page), { offset: 300 }); + await tour.click(captureEntry(page)); + await expect(page.getByText("Turn 6: Blue played")).toBeVisible(); + await tour.scrollTo(page.getByRole("heading", { name: "Why that move?" }), { offset: 90 }); + await tour.hover(page.getByText("Top candidates")); + await tour.pause(2200); + await tour.hover(page.getByRole("region", { name: "Evaluation features after this move" })); + await tour.pause(3200); + + await finishRecording(context, page, tour); + }); + + test("2. A* Lab: sample input, both heuristics, paired study", async ({ browser }) => { + const context = await recordingContext(browser); + const page = await context.newPage(); + const tour = new Tour(page, walkthrough("astar-lab")); + const output = page.getByRole("region", { name: "Search output in the original CLI format" }); + + await page.goto("/astar"); + await expect(page.getByRole("heading", { name: "A* Lab", level: 1 })).toBeVisible(); + await settle(page); + tour.markStart(); + + await tour.caption(1); + await tour.pause(1200); + await tour.hover(page.getByRole("group", { name: /^A\* board/ }), 900); + await tour.pause(1200); + + await tour.caption(2); + await tour.click(page.getByRole("combobox", { name: "Preset" })); + await tour.click(page.getByRole("option", { name: "Sample input 1" })); + await expect(page.getByText("code/sample_input.json → sample_output.txt")).toBeVisible(); + await tour.hover(page.getByText("code/sample_input.json → sample_output.txt")); + await tour.pause(1600); + + await tour.caption(3); + await tour.click(page.getByRole("radio", { name: "Manhattan" })); + await tour.click(page.getByRole("button", { name: "Animate search" })); + await expect(output).toContainText("…searching"); + await expect(output).not.toContainText("…searching", { timeout: 60_000 }); + + await tour.caption(4); + await expect(page.getByText("Matches the output recorded in the original repo")).toBeVisible(); + await expect(output).toContainText("8"); + await tour.hover(output); + await tour.pause(1600); + await tour.hover(page.getByText("Matches the output recorded in the original repo")); + await tour.pause(1800); + + await tour.caption(5); + await tour.click(page.getByRole("radio", { name: "Euclidean" })); + await tour.click(page.getByRole("button", { name: "Animate search" })); + await expect(output).toContainText("…searching"); + await expect(output).not.toContainText("…searching", { timeout: 60_000 }); + await tour.pause(800); + + await tour.caption(6); + const compare = page.getByRole("table", { name: "Both heuristics on this board" }); + await expect(compare).toBeVisible(); + await tour.hover(compare); + await tour.pause(3000); + + await tour.caption(7); + const expansions = page.getByRole("heading", { + name: /^Node expansions \(paired, 980 boards\)/, + }); + await tour.scrollTo(expansions, { offset: 150, ms: 1600 }); + await tour.hover(page.getByText("Mean difference, Manhattan − Euclidean").first()); + await tour.pause(1600); + await tour.hover(page.getByText("Wilcoxon signed-rank (two-sided)").first()); + await tour.pause(2200); + + await tour.caption(8); + await tour.hover(page.getByText("Paired difference, Manhattan − Euclidean").first()); + await tour.pause(1600); + await tour.hover(page.getByText("Exact McNemar test (two-sided)").first()); + await tour.pause(2600); + + await finishRecording(context, page, tour); + }); + + test("3. tournament and LLM evaluation (mocked AI response)", async ({ browser }) => { + const context = await recordingContext(browser); + const ai = await mockAiProviders(context, { latencyMs: FAST ? 150 : 350 }); + const page = await context.newPage(); + const tour = new Tour(page, walkthrough("tournament-llm-eval")); + + await page.goto("/tournament"); + await expect(page.getByRole("heading", { level: 1 })).toBeVisible(); + await settle(page); + tour.markStart(); + + await tour.caption(1); + await tour.pause(1500); + await tour.scrollTo(page.locator("#run"), { offset: 70, ms: 1600 }); + await tour.pause(800); + + await tour.caption(2); + const sizes5 = page.getByRole("button", { name: "5×5", exact: true }); + await expect(sizes5).toHaveAttribute("aria-pressed", "true"); + await tour.click(sizes5); + await tour.click(page.getByRole("button", { name: "2", exact: true })); + await expect(page.getByLabel("Seed", { exact: true })).toHaveValue("4399"); + await tour.hover(page.getByLabel("Seed", { exact: true })); + await tour.pause(900); + + await tour.caption(3); + const runButton = page.getByRole("button", { name: "Run 24 games" }); + await tour.click(runButton, { after: 0 }); + await tour.idleWhile(() => + expect(page.getByText(/^24 games in .* · seed 4399$/)).toBeVisible({ timeout: 120_000 }), + ); + await tour.pause(600); + + await tour.caption(4); + const leaderboard = page.getByText("Your run: Bradley-Terry strengths"); + await tour.scrollTo(leaderboard, { offset: 120, ms: 1300 }); + await tour.hover(leaderboard); + await tour.pause(2400); + const headToHead = page.locator("#run").getByText("Head to head."); + await tour.scrollTo(headToHead, { offset: 140, ms: 1200 }); + await tour.hover(headToHead); + await tour.pause(2400); + + await tour.caption(5); + await tour.click(page.getByRole("button", { name: /^AI settings/ })); + const dialog = page.getByRole("dialog", { name: "AI settings" }); + await expect(dialog).toBeVisible(); + await tour.pause(1400); + await tour.hover(dialog.getByText("Your key stays in this browser.")); + await tour.pause(1800); + await tour.click(dialog.getByRole("combobox", { name: "Model" })); + await tour.pause(900); + await tour.click(page.getByRole("option", { name: /Claude Haiku 4\.5/ })); + await tour.pause(600); + + await tour.caption(6); + await tour.click(dialog.getByRole("radio", { name: "OpenAI" })); + await tour.type(dialog.getByLabel("Model id"), MOCK_MODEL_ID); + await tour.type(dialog.getByLabel("OpenAI API key"), PLACEHOLDER_KEY); + await tour.pause(900); + await tour.click(dialog.getByRole("button", { name: "Save" })); + await expect(dialog).toBeHidden(); + + await tour.caption(7); + await tour.scrollTo(page.locator("body"), { offset: 0, ms: 600 }); + await tour.click( + page.getByRole("navigation", { name: "Main" }).getByRole("link", { name: "LLM Arena" }), + ); + await expect(page).toHaveURL(/\/llm-arena$/); + await expect(page.getByText(`Model: ${MOCK_MODEL_ID}`)).toBeVisible(); + await tour.pause(1000); + await tour.click(page.getByRole("radio", { name: "2", exact: true })); + await expect(page.getByRole("textbox", { name: "Seed" })).toHaveValue("2026"); + await tour.click(page.getByRole("button", { name: "Play 2 games" }), { after: 0 }); + + await tour.caption(8); + await expect( + page.getByText("waiting for the model").or(page.getByText("minimax agent thinking")), + ).toBeVisible(); + const reason = page.getByText("Mocked response for illustration: the first legal cell."); + await expect(reason).toBeVisible(); + await tour.tryHover(reason); + await tour.idleWhile(async () => { + // Both games finished: the run button is back and the games table is shown. + await expect(page.getByRole("button", { name: "Play 2 games" })).toBeVisible({ + timeout: 120_000, + }); + await expect(page.getByRole("heading", { name: "Games", exact: true })).toBeVisible(); + }); + + await tour.caption(9); + const table = page.getByRole("heading", { name: "Side by side against the original agent" }); + await tour.scrollTo(table, { offset: 90, ms: 1000 }); + await expect( + page.getByRole("rowheader", { name: new RegExp(`LLM: ${MOCK_MODEL_ID}`) }), + ).toBeVisible(); + await tour.hover(page.getByRole("rowheader", { name: new RegExp(`LLM: ${MOCK_MODEL_ID}`) })); + await tour.pause(2200); + await tour.hover(page.getByRole("rowheader", { name: /First legal cell/ })); + await tour.pause(2200); + + await tour.caption(10); + await tour.click(page.locator("#main").getByRole("link", { name: "AI audit log" })); + await expect(page).toHaveURL(/\/ai-log$/); + await expect(page.getByText(MOCK_MODEL_ID).first()).toBeVisible(); + await tour.pause(1600); + await tour.hover(page.getByRole("button", { name: /CSV/ }).first()); + await tour.pause(1800); + + await tour.caption(11); + await tour.click(page.getByRole("button", { name: /^AI settings/ })); + await tour.click(page.getByRole("dialog").getByRole("button", { name: /^Forget/ })); + await expect(page.getByText(/No OpenAI key saved/)).toBeVisible(); + await tour.pause(2200); + await page.keyboard.press("Escape"); + await tour.pause(800); + + expect(ai.calls).toBeGreaterThan(0); + expect(ai.leaks, "the placeholder key must only go to the (mocked) provider").toEqual([]); + const stored = await page.evaluate(() => + JSON.stringify({ ...localStorage, ...sessionStorage }), + ); + expect(stored).not.toContain(PLACEHOLDER_KEY); + + await finishRecording(context, page, tour); + }); +}); + +async function desktop(browser: Browser, colorScheme: "light" | "dark" = "light") { + return browser.newContext({ + viewport: { width: 1440, height: 900 }, + deviceScaleFactor: 1, + colorScheme, + }); +} + +async function mobile(browser: Browser): Promise { + return browser.newContext({ + viewport: { width: 390, height: 844 }, + deviceScaleFactor: 2, + isMobile: true, + hasTouch: true, + colorScheme: "light", + }); +} + +const quickClick = (target: Locator) => target.click(); + +test.describe("screenshots", () => { + test("landing, light and dark", async ({ browser }) => { + for (const scheme of ["light", "dark"] as const) { + const context = await desktop(browser, scheme); + const page = await context.newPage(); + await page.goto("/"); + await expect(page.getByRole("heading", { level: 1 })).toContainText("Cachex"); + await settle(page); + await shot(page, `0${scheme === "light" ? 1 : 2}-landing-${scheme}`); + await context.close(); + } + }); + + test("key features at 1440 × 900", async ({ browser }) => { + const context = await desktop(browser); + const page = await context.newPage(); + + await page.goto("/play"); + await settle(page); + await page.getByRole("switch", { name: "Show coordinates" }).click(); + await playScriptedGame(page, quickClick); + await captureEntry(page).click(); + await expect(page.getByText("Turn 6: Blue played")).toBeVisible(); + await shot(page, "03-play", { target: h1(page), offset: 100 }); + + await page.goto("/spectate"); + await settle(page); + await page.getByRole("switch", { name: "Coordinates" }).click(); + const delay = page.getByRole("slider", { name: "Move delay" }); + await delay.focus(); + await page.keyboard.press("Home"); + await page.getByRole("button", { name: "Play", exact: true }).click(); + await expect( + page + .getByRole("status") + .filter({ hasText: /wins|Draw/ }) + .first(), + ).toBeVisible({ + timeout: 120_000, + }); + await shot(page, "04-spectate", { target: h1(page), offset: 100 }); + + await page.goto("/astar"); + await settle(page); + await expect(page.getByText("Matches the output recorded in the original repo")).toBeVisible(); + await shot(page, "05-astar-lab", { target: h1(page), offset: 100 }); + await shot(page, "06-astar-paired-study", { + target: page.locator("#paired-study ul").first(), + offset: 76, + }); + + await page.goto("/tournament#run"); + await settle(page); + await page.getByRole("button", { name: "5×5", exact: true }).click(); + await page.getByRole("button", { name: "2", exact: true }).click(); + await page.getByRole("button", { name: "Run 24 games" }).click(); + await expect(page.getByText(/^24 games in .* · seed 4399$/)).toBeVisible({ timeout: 120_000 }); + await shot(page, "07-tournament", { + target: page.getByText(/^24 games in .* · seed 4399$/), + offset: 84, + }); + + await page.goto("/llm-arena"); + await settle(page); + await expect(page.getByRole("rowheader", { name: /First legal cell/ })).toBeVisible(); + await page.getByRole("button", { name: /^AI settings/ }).click(); + await expect(page.getByRole("dialog", { name: "AI settings" })).toBeVisible(); + await page.waitForTimeout(400); + await shot(page, "08-ai-settings"); + await page.keyboard.press("Escape"); + await expect(page.getByRole("dialog")).toBeHidden(); + await shot(page, "09-llm-arena"); + + await page.goto("/methods"); + await settle(page); + await shot(page, "10-methods"); + await context.close(); + }); + + test("mobile at 390 × 844", async ({ browser }) => { + const context = await mobile(browser); + const page = await context.newPage(); + + await page.goto("/"); + await settle(page); + await shot(page, "11-mobile-landing"); + + await page.goto("/play"); + await settle(page); + await playScriptedGame(page, quickClick); + await shot(page, "12-mobile-play", { target: h1(page), offset: 96 }); + + await page.goto("/astar"); + await settle(page); + await expect(page.getByText("Matches the output recorded in the original repo")).toBeAttached(); + await shot(page, "13-mobile-astar", { target: h1(page), offset: 96 }); + await context.close(); + }); +}); diff --git a/web/eslint.config.mjs b/web/eslint.config.mjs index 05e726d..f533f2b 100644 --- a/web/eslint.config.mjs +++ b/web/eslint.config.mjs @@ -12,6 +12,11 @@ const eslintConfig = defineConfig([ "out/**", "build/**", "next-env.d.ts", + // Showcase tour output (pnpm showcase): raw media and Playwright artefacts. + ".showcase/**", + ".playwright/**", + "test-results/**", + "playwright-report/**", ]), ]); diff --git a/web/next.config.ts b/web/next.config.ts index cb651cd..2a81d21 100644 --- a/web/next.config.ts +++ b/web/next.config.ts @@ -1,5 +1,12 @@ import type { NextConfig } from "next"; -const nextConfig: NextConfig = {}; +const nextConfig: NextConfig = { + env: { + // "Source" links on /methods point at the commit being built when it is + // known (Vercel sets VERCEL_GIT_COMMIT_SHA for Git deployments; CLI + // deployments can pass NEXT_PUBLIC_GIT_REF as a build env), else at main. + NEXT_PUBLIC_GIT_REF: process.env.NEXT_PUBLIC_GIT_REF || process.env.VERCEL_GIT_COMMIT_SHA || "", + }, +}; export default nextConfig; diff --git a/web/package.json b/web/package.json index a5e7359..2277637 100644 --- a/web/package.json +++ b/web/package.json @@ -10,9 +10,15 @@ "typecheck": "tsc --noEmit", "test": "vitest run", "format": "prettier --write .", - "format:check": "prettier --check ." + "format:check": "prettier --check .", + "gen:reference": "tsx scripts/generate-reference-studies.ts", + "sync-docs": "node scripts/sync-docs.mjs", + "showcase": "node scripts/showcase.mjs", + "showcase:test": "SHOWCASE_FAST=1 node scripts/showcase.mjs --test-only", + "showcase:media": "node scripts/showcase-media.mjs" }, "dependencies": { + "@anthropic-ai/sdk": "^0.131.0", "class-variance-authority": "^0.7.1", "cn": "^0.4.0", "lucide-react": "^1.52.0", @@ -21,10 +27,14 @@ "radix-ui": "^1.6.7", "react": "19.2.8", "react-dom": "19.2.8", + "react-markdown": "^10.1.0", + "remark-gfm": "^4.0.1", "shadcn": "^4.21.1", - "tw-animate-css": "^1.4.0" + "tw-animate-css": "^1.4.0", + "zod": "^4.6.5" }, "devDependencies": { + "@playwright/test": "^1.63.0", "@tailwindcss/postcss": "^4", "@types/node": "^22.20.5", "@types/react": "^19", @@ -33,7 +43,9 @@ "eslint-config-next": "16.3.8", "prettier": "^3.9.9", "prettier-plugin-tailwindcss": "^0.8.1", + "sharp": "^0.35.5", "tailwindcss": "^4", + "tsx": "^4.23.15", "typescript": "^5", "vitest": "^5.0.3" }, diff --git a/web/playwright.config.ts b/web/playwright.config.ts new file mode 100644 index 0000000..b8b1055 --- /dev/null +++ b/web/playwright.config.ts @@ -0,0 +1,42 @@ +/** + * Playwright drives the guided tour (e2e/showcase.spec.ts): it checks the main + * journeys end to end and, on the way, captures the README screenshots and + * the /tour recordings. Run it with `pnpm showcase` (see scripts/showcase.mjs). + * + * - BASE_URL picks the site (default: production). A localhost URL starts + * `pnpm start` first, so build before using it. + * - The system Google Chrome is used (channel "chrome"); no browser is + * downloaded. + */ +import { defineConfig } from "@playwright/test"; + +const BASE_URL = (process.env.BASE_URL ?? "https://cachex-ai.vercel.app").replace(/\/$/, ""); +const local = new URL(BASE_URL); +const isLocal = ["localhost", "127.0.0.1"].includes(local.hostname); + +export default defineConfig({ + testDir: "./e2e", + outputDir: "./test-results", + timeout: 6 * 60_000, + expect: { timeout: 30_000 }, + workers: 1, + fullyParallel: false, + forbidOnly: Boolean(process.env.CI), + reporter: [["list"]], + use: { + baseURL: BASE_URL, + channel: "chrome", + headless: true, + locale: "en-AU", + timezoneId: "Australia/Adelaide", + reducedMotion: "no-preference", + }, + webServer: isLocal + ? { + command: `pnpm start --port ${local.port || "3000"}`, + url: BASE_URL, + reuseExistingServer: true, + timeout: 120_000, + } + : undefined, +}); diff --git a/web/pnpm-lock.yaml b/web/pnpm-lock.yaml index 638dfda..1f71821 100644 --- a/web/pnpm-lock.yaml +++ b/web/pnpm-lock.yaml @@ -8,6 +8,9 @@ importers: .: dependencies: + '@anthropic-ai/sdk': + specifier: ^0.131.0 + version: 0.131.0(zod@4.6.5) class-variance-authority: specifier: ^0.7.1 version: 0.7.1 @@ -19,7 +22,7 @@ importers: version: 1.52.0(@types/react@19.3.0)(react@19.2.8) next: specifier: 16.3.8 - version: 16.3.8(@babel/core@7.29.7)(@types/node@22.20.5)(react-dom@19.2.8(react@19.2.8))(react@19.2.8) + version: 16.3.8(@babel/core@7.29.7)(@playwright/test@1.63.0)(@types/node@22.20.5)(react-dom@19.2.8(react@19.2.8))(react@19.2.8) next-themes: specifier: ^0.4.6 version: 0.4.6(react-dom@19.2.8(react@19.2.8))(react@19.2.8) @@ -32,13 +35,25 @@ importers: react-dom: specifier: 19.2.8 version: 19.2.8(react@19.2.8) + react-markdown: + specifier: ^10.1.0 + version: 10.1.0(@types/react@19.3.0)(react@19.2.8) + remark-gfm: + specifier: ^4.0.1 + version: 4.0.1 shadcn: specifier: ^4.21.1 version: 4.21.1(typescript@5.9.3) tw-animate-css: specifier: ^1.4.0 version: 1.4.0 + zod: + specifier: ^4.6.5 + version: 4.6.5 devDependencies: + '@playwright/test': + specifier: ^1.63.0 + version: 1.63.0 '@tailwindcss/postcss': specifier: ^4 version: 4.3.3 @@ -63,15 +78,21 @@ importers: prettier-plugin-tailwindcss: specifier: ^0.8.1 version: 0.8.1(prettier@3.9.9) + sharp: + specifier: ^0.35.5 + version: 0.35.5(@types/node@22.20.5) tailwindcss: specifier: ^4 version: 4.3.3 + tsx: + specifier: ^4.23.15 + version: 4.23.15 typescript: specifier: ^5 version: 5.9.3 vitest: specifier: ^5.0.3 - version: 5.0.3(@types/node@22.20.5)(vite@8.3.2(@types/node@22.20.5)(jiti@2.7.0)) + version: 5.0.3(@types/node@22.20.5)(vite@8.3.2(@types/node@22.20.5)(esbuild@0.28.2)(jiti@2.7.0)(tsx@4.23.15)) packages: @@ -79,6 +100,15 @@ packages: resolution: {integrity: sha512-U4+70Pc5ZS9osnCBCE5Jha/ciHM+Yp+CNMNC/7HvYbNRk1Ldd+f7qO65W5qfhu/TCv+/ozljlXXe9Nj8419DMA==} engines: {node: '>=10'} + '@anthropic-ai/sdk@0.131.0': + resolution: {integrity: sha512-9+PepoU7qVMmM70jSahdnfcMHAvAnzhMOLv2W9WjRLO+OG/MAiBETkwKZqv7KIJ1Cj5h8eZ019bHzZaq+Mbepw==} + hasBin: true + peerDependencies: + zod: ^3.25.0 || ^4.0.0 + peerDependenciesMeta: + zod: + optional: true + '@babel/code-frame@7.29.7': resolution: {integrity: sha512-Aup7aUOfpbAUg2ROOJN6Iw5f9DMBlzu0mIkm/malLQFN/YQgO48wCj0Kxa3sEHJvPVFg7siR+qRInwXd2qhQKw==} engines: {node: '>=6.9.0'} @@ -196,6 +226,10 @@ packages: peerDependencies: '@babel/core': ^7.0.0-0 + '@babel/runtime@7.29.7': + resolution: {integrity: sha512-Nq8OhGWiZIZGV6hLHoyAKLLcJihP/xFeBMGJoUrxTX2psI8dCifzLhZISFb+VWS3wFMRDmCGw5R+dOySCqPLhw==} + engines: {node: '>=6.9.0'} + '@babel/template@7.29.7': resolution: {integrity: sha512-puq+Gf35oI24FeN11LkoUQFqv9uwNeWpxXZi/Ji3rRIoKAzKnxRaZ+Gkj0vKS9ZCiTESfng1N9LyOyXvo+m+Gg==} engines: {node: '>=6.9.0'} @@ -227,6 +261,162 @@ packages: '@emnapi/wasi-threads@1.2.1': resolution: {integrity: sha512-uTII7OYF+/Mes/MrcIOYp5yOtSMLBWSIoLPpcgwipoiKbli6k322tcoFsxoIIxPDqW01SQGAgko4EzZi2BNv2w==} + '@esbuild/aix-ppc64@0.28.2': + resolution: {integrity: sha512-XExcO+dvLKvVtNTibSTBej1NCAbaGhWn9Ww1ZPx80qsahhPFe/8jgWP0IchNe0F3HwkU7n8ejhH8bjonqht8mQ==} + engines: {node: '>=18'} + cpu: [ppc64] + os: [aix] + + '@esbuild/android-arm64@0.28.2': + resolution: {integrity: sha512-5YfKeeI8qWfBZIX+u2xZC3Zlb3Os/gLS2sbEKM+I4ZOcsWmHS2WLysCcQZDAFRslDUU5Oiq44gf6PYN1vGwG5A==} + engines: {node: '>=18'} + cpu: [arm64] + os: [android] + + '@esbuild/android-arm@0.28.2': + resolution: {integrity: sha512-kXXoiPVVGQcnIYGOeaovwOURpniDBpSq4A03qkQ+BMQqtGG6HYap3xne9C1O1yo4TR3qxlCX5IqqmX6fFo2Lqg==} + engines: {node: '>=18'} + cpu: [arm] + os: [android] + + '@esbuild/android-x64@0.28.2': + resolution: {integrity: sha512-O387ite7SzUyCcy3JQX4P4bLtEA7bLLkx+esve5JHnyYfNTxcVpXZo9jhdB0lTKN44gztELTdU7nS8Nr16Fs1Q==} + engines: {node: '>=18'} + cpu: [x64] + os: [android] + + '@esbuild/darwin-arm64@0.28.2': + resolution: {integrity: sha512-n4KqkOQrraxHJcgjM1RvwbigfQKIKJVpM7xp+KsxiyUSrRdIXnt73VhrPAx0fV44hgfmIVKjxMN9J1t5jySVkw==} + engines: {node: '>=18'} + cpu: [arm64] + os: [darwin] + + '@esbuild/darwin-x64@0.28.2': + resolution: {integrity: sha512-uq6suIWYP37qzGddBKPw5QEQPi6HiLGsO7UmkpfyaYNQ3D+rN6w6WfwH+nuqcGXWvawGwxOEroO4YGnFh95azw==} + engines: {node: '>=18'} + cpu: [x64] + os: [darwin] + + '@esbuild/freebsd-arm64@0.28.2': + resolution: {integrity: sha512-n+I0BTSRIoy+d6RPKnEVwql5UwBJolytvY4mAOIEJorKlqgPII8ix6slVVrfZ5Tnj7glIZvloylbB/EJPMWEXw==} + engines: {node: '>=18'} + cpu: [arm64] + os: [freebsd] + + '@esbuild/freebsd-x64@0.28.2': + resolution: {integrity: sha512-78XJTJkvPs0kz2w61301PJjXl4g7q3JqiYMZ/M/yVI73EHBrCRTgkhu9oqG7vPqq+a/yadEW8aD+agKlk5xrmg==} + engines: {node: '>=18'} + cpu: [x64] + os: [freebsd] + + '@esbuild/linux-arm64@0.28.2': + resolution: {integrity: sha512-pW4AC0P3it8c7do9MVM4p51FzHzdM/TZrerurgRcHJ2WTa1VQ1CIq18xncfpBJw4ojkiZZrKW2yIBWBP92j6Ug==} + engines: {node: '>=18'} + cpu: [arm64] + os: [linux] + + '@esbuild/linux-arm@0.28.2': + resolution: {integrity: sha512-XlDnu2q5yoqems+xay6wSAcg9DDD7K9RLKZEBOMZm3ckNpJBvOX20tSfby8KfrrhINDyv9V2YVZKY/SpoGJI8w==} + engines: {node: '>=18'} + cpu: [arm] + os: [linux] + + '@esbuild/linux-ia32@0.28.2': + resolution: {integrity: sha512-CYbnj78HsIeA+DhgUKgFCfvNsTHFhMMrinUrMZpDXJXKN8T3XViTZ/+wtHeVxEWY8ewSzTFN+nRmSwO2tZaLUQ==} + engines: {node: '>=18'} + cpu: [ia32] + os: [linux] + + '@esbuild/linux-loong64@0.28.2': + resolution: {integrity: sha512-buwkd8nsph4R+ajRvw0qM5Hja/TXQow3ptzWO2EbG/cqcIkHloRrdlBtQlshyYGTNFvfkfJ5tpPLVkY4DtsPfQ==} + engines: {node: '>=18'} + cpu: [loong64] + os: [linux] + + '@esbuild/linux-mips64el@0.28.2': + resolution: {integrity: sha512-ZVykbDyk7519VwiNb9Lcj9m8XM6v5V9uKPvrEMkkEedVewf+0itkhahp4HDpgERXhwLRpWFypsGbG/J8s0QjJA==} + engines: {node: '>=18'} + cpu: [mips64el] + os: [linux] + + '@esbuild/linux-ppc64@0.28.2': + resolution: {integrity: sha512-CAXl+Dtd9UUuJd8pKKdwh6MLm3MUMiqMPmhZ3tTSXPqfyQ3vDl6R5hZdZ/kYojK4ofXtdfSv1tFq8XzWx3heNQ==} + engines: {node: '>=18'} + cpu: [ppc64] + os: [linux] + + '@esbuild/linux-riscv64@0.28.2': + resolution: {integrity: sha512-GeXCej4IQtU1B+QlDV8W/RRvbzI3O/Stss+/bCXv4lZls5WGRtu2a+3JkA3i4qIUlMXpcHebWpF8AkJhATowuA==} + engines: {node: '>=18'} + cpu: [riscv64] + os: [linux] + + '@esbuild/linux-s390x@0.28.2': + resolution: {integrity: sha512-3H1weTYZPxt/WOhByszQZybS9w5lKzUn1FDMsgEChbHWQwHYQQRfBxgCcZvPhjHfKyJjIievvMmEUawJrdY9Dg==} + engines: {node: '>=18'} + cpu: [s390x] + os: [linux] + + '@esbuild/linux-x64@0.28.2': + resolution: {integrity: sha512-4xTZr1FUmSoQW4XIWmit3tzQrUTZM+N3P0XV8xROKYF50XfI7xeO90+1bZvNwxIufQ9hDQVRJH5YhgPVF8A/HQ==} + engines: {node: '>=18'} + cpu: [x64] + os: [linux] + + '@esbuild/netbsd-arm64@0.28.2': + resolution: {integrity: sha512-sSATRjPeDBg3pdgHoQfoYBob11Kk1FGa9lui5RIHZCoCkJa9QKlvl3/vKz2usCmYYjs7ymJR/2Nnsqe+Hjt5nw==} + engines: {node: '>=18'} + cpu: [arm64] + os: [netbsd] + + '@esbuild/netbsd-x64@0.28.2': + resolution: {integrity: sha512-lqnzCV+mM0gIADaKihiCg6ifgfU2L3h5E33rNQBN1Y4MaVGnzryzmvvf7UHxprpQdE8hpqLolJ9Rl+SkIRDpyw==} + engines: {node: '>=18'} + cpu: [x64] + os: [netbsd] + + '@esbuild/openbsd-arm64@0.28.2': + resolution: {integrity: sha512-AL2qJILH7lNjrDmCQDvdxMfAUIv8KMNZOvrwAQ8i8//ntL9FflhOyMJ8OZSMBb8/AWXe3/5v5S20y3zCoZWKoQ==} + engines: {node: '>=18'} + cpu: [arm64] + os: [openbsd] + + '@esbuild/openbsd-x64@0.28.2': + resolution: {integrity: sha512-QtiuPytchRyC4rwUKhexJdQKvDuZ6hWloi3igqPQNUJCS1/v9EiO3UTOXR6A3FoMo4fnAKbWJdqaIwhOzh8qEw==} + engines: {node: '>=18'} + cpu: [x64] + os: [openbsd] + + '@esbuild/openharmony-arm64@0.28.2': + resolution: {integrity: sha512-WkhYDmpTjLvGlScA1rwjRUmhl4k8oXR3cIbtqWmELgU/dFeHHlEllxDvdWcNJV9rbzCexB5vz8gtNewWLgCT7Q==} + engines: {node: '>=18'} + cpu: [arm64] + os: [openharmony] + + '@esbuild/sunos-x64@0.28.2': + resolution: {integrity: sha512-GPMSkTOtMnv2U2F8gxe4Io6qmVs+YKyp832Etqqxr0hFngmXQ3rzwytelm3GIn7T4VviRUlf3sOgBOiTdvaf7g==} + engines: {node: '>=18'} + cpu: [x64] + os: [sunos] + + '@esbuild/win32-arm64@0.28.2': + resolution: {integrity: sha512-PIhhEkE9uPBleRBrQEJpUn7MBnibZzbGzYWPmY3x+YoVg/95zbjB4CxPPOQ8l5tYYM4mMaCthF8/1DIfBQQyWQ==} + engines: {node: '>=18'} + cpu: [arm64] + os: [win32] + + '@esbuild/win32-ia32@0.28.2': + resolution: {integrity: sha512-YmJbfTlvU7Sdn9BB+4PRES4oB6pxgS37MAONj+hBr/cpXS1aBPKXxNnDbu+QCWPj0o9dgyxeq79g6c5P8KeuYA==} + engines: {node: '>=18'} + cpu: [ia32] + os: [win32] + + '@esbuild/win32-x64@0.28.2': + resolution: {integrity: sha512-5ebpxr3nWMzrL/rnUI755Jkuee0bHL/Gq0WTF9lvcpv73wAp5eu8MfBUgWK9bhWvZjj7yX8etf/8tI8Ney695g==} + engines: {node: '>=18'} + cpu: [x64] + os: [win32] + '@eslint-community/eslint-utils@4.10.1': resolution: {integrity: sha512-cuadcxVFE8sDK6iWJbs8Sn0av2Nrh2QSGQhVlBW9AaAHqHwjWsZHT8LJ4hFGPh7ASBV2deFdM7H/DPjulmh8rg==} engines: {node: ^12.22.0 || ^14.17.0 || >=16.0.0} @@ -584,6 +774,11 @@ packages: '@oxc-project/types@0.152.0': resolution: {integrity: sha512-oM/5rLBm2tPkg0iBgkH/FOeR3PCDpY19GTgAZjMFM8h9WI9VW7cLgzp6nwtarYKmovavIQZ+Fe/RKX/8C8O/Rw==} + '@playwright/test@1.63.0': + resolution: {integrity: sha512-oxMK4vllB9RK5NQ2l1pq1IfOf2AvnEuj/vYGDj0H2nMtmtZpKtCwt/l00GEO6xjGfpBNAvjovvYdCm50dRQkpQ==} + engines: {node: '>=20'} + hasBin: true + '@radix-ui/number@1.1.3': resolution: {integrity: sha512-Road2bidD0uu/1BGDOWNdPI06g0lIRy6IF9GZcIrDK2KGItfor8IQwQa+yM2ERgHM1MmHxaxpTzk0/Jp42lNfA==} @@ -1387,6 +1582,9 @@ packages: resolution: {integrity: sha512-tlqY9xq5ukxTUZBmoOp+m61cqwQD5pHJtFY3Mn8CA8ps6yghLH/Hw8UPdqg4OLmFW3IFlcXnQNmo/dh8HzXYIQ==} engines: {node: '>=18'} + '@stablelib/base64@1.0.1': + resolution: {integrity: sha512-1bnPQqSxSuc3Ii6MhBysoWCg58j97aUjuCSZrGSmDxNqtytIi0k8utUenAwTZN4V5mXXYGsVUI9zeBqy+jBOSQ==} + '@swc/helpers@0.5.23': resolution: {integrity: sha512-5lSsMOTXURePglDfvuAQUqkGek9Hg2kksOYay2m0+XR++b2NWYL/4sWyuvVBIs8oKnJaxkdi9whaL/sqN13afw==} @@ -1491,18 +1689,33 @@ packages: '@types/chai@5.2.3': resolution: {integrity: sha512-Mw558oeA9fFbv65/y4mHtXDs9bPnFMZAL/jxdPFUpOHHIXX91mcgEHbS5Lahr+pwZFR8A7GQleRWeI6cGFC2UA==} + '@types/debug@4.1.13': + resolution: {integrity: sha512-KSVgmQmzMwPlmtljOomayoR89W4FynCAi3E8PPs7vmDVPe84hT+vGPKkJfThkmXs0x0jAaa9U8uW8bbfyS2fWw==} + '@types/deep-eql@4.0.2': resolution: {integrity: sha512-c9h9dVVMigMPc4bwTvC5dxqtqJZwQPePsWjPlpSOnojbor6pGqdk541lfA7AqFQr5pB1BRdq0juY9db81BwyFw==} + '@types/estree-jsx@1.0.5': + resolution: {integrity: sha512-52CcUVNFyfb1A2ALocQw/Dd1BQFNmSdkuC3BkZ6iqhdMfQz7JWOFRuJFloOzjk+6WijU56m9oKXFAXc7o3Towg==} + '@types/estree@1.0.9': resolution: {integrity: sha512-GhdPgy1el4/ImP05X05Uw4cw2/M93BCUmnEvWZNStlCzEKME4Fkk+YpoA5OiHNQmoS7Cafb8Xa3Pya8m1Qrzeg==} + '@types/hast@3.0.5': + resolution: {integrity: sha512-rp/ezSWaD1m44dPKICGhiskI13nVr7qTloFwDa/IYkhhf5nzwP+zIQcIJh3WIFSBOy/H1PzB40jPjMDksN4F+g==} + '@types/json-schema@7.0.15': resolution: {integrity: sha512-5+fP8P8MFNC+AyZCDxrB2pkZFPGzqQWUzpSeuuVLvm8VMcorNYavBqoFcxK8bQz4Qsbn4oUEEem4wDLfcysGHA==} '@types/json5@0.0.29': resolution: {integrity: sha512-dRLjCWHYg4oaA77cxO64oO+7JwCwnIzkZPdrrC71jQmQtlhM556pwKo5bUzqvZndkVbeFLIIi+9TC40JNF5hNQ==} + '@types/mdast@4.0.4': + resolution: {integrity: sha512-kGaNbPh1k7AFzgpud/gMdvIm5xuECykRR+JnWKQno9TAXVa6WIVCGTPvYGekIDL4uwCZQSYbUxNBSb1aUo79oA==} + + '@types/ms@2.1.0': + resolution: {integrity: sha512-GsCCIZDE/p3i96vtEqx+7dBUGXrc7zeSK3wwPHIaRThS+9OhWIXRqzs4d6k1SVU8g91DrNRWxWUGhp5KXQb2VA==} + '@types/node@22.20.5': resolution: {integrity: sha512-U2+DNr+wSjpsTS/wZGYHq7GcwfuSmKiKvoPvK22zwTlRhU91yOniN4qRR5KhIjvif7ysw/dz/hKmfDH0Ris4aA==} @@ -1514,6 +1727,12 @@ packages: '@types/react@19.3.0': resolution: {integrity: sha512-N0rFCuH9YoxG9/m61l9MfpJKfmLOVU0em7ipIz6TRgSSkvReLB9vL85GB+yr8Bs5leqpvg96JSwF4ZS1s4viQg==} + '@types/unist@2.0.11': + resolution: {integrity: sha512-CmBKiL6NNo/OqgmMn95Fk9Whlp2mtvIv+KNpQKN2F4SjvrEesubTRWGYSg+BnWZOnlCaSTU1sMpsBOzgbYhnsA==} + + '@types/unist@3.0.3': + resolution: {integrity: sha512-ko/gIFJRv177XgZsZcBwnqJN5x/Gien8qNOn0D5bQU/zAzVf9Zt3BlcUiLqhV9y4ARk0GbT3tnUiPNgnTXzc/Q==} + '@types/validate-npm-package-name@4.0.2': resolution: {integrity: sha512-lrpDziQipxCEeK5kWxvljWYhUvOiB2A9izZd9B2AFarYAkqZshb4lPbRs7zKEic6eGtH8V/2qJW+dPp9OtF6bw==} @@ -1576,6 +1795,9 @@ packages: resolution: {integrity: sha512-8eQ9R218XORK+KLosnf4bu/QsUXvUyVwTbArg7/0NMB1Pu87OJKvj4nhFblkYE8gQV73mW1dx1ptlPCkwRGa7A==} engines: {node: ^18.18.0 || ^20.9.0 || >=21.1.0} + '@ungap/structured-clone@1.4.0': + resolution: {integrity: sha512-1mEZtMKPM09vDmQt5y7YvmN2+DFTP7Tg0EWXdic8/C6VRnpb33e4ghisCIE3WZjsE2N8mf+QV1Zqh7ZFYLWInQ==} + '@unrs/resolver-binding-android-arm-eabi@1.12.2': resolution: {integrity: sha512-g5T90pqg1bo/7mytQx6F4iBNC0Wsh9cu+z9veDbFjc7HjpesJFWD7QMS0NGStXM075+7dJPPVvBbpZlnrdpi/w==} cpu: [arm] @@ -1836,6 +2058,9 @@ packages: resolution: {integrity: sha512-qIj0G9wZbMGNLjLmg1PT6v2mE9AH2zlnADJD/2tC6E00hgmhUOfEB6greHPAfLRSufHqROIUTkw6E+M3lH0PTQ==} engines: {node: '>= 0.4'} + bail@2.0.2: + resolution: {integrity: sha512-0xO6mYd7JB2YesxDKplafRpsiOzPt9V02ddPCLbY1xYGPOX24NTyN50qnUxgCPcSoYMhKpAuBTjQoRZCAkUDRw==} + balanced-match@1.0.2: resolution: {integrity: sha512-3oSeUO0TMV67hN1AmbXsK4yaqU7tjiHlbxRDZOpH0KW9+CeX4bRAaX0Anxt0tx2MrpRpWwQaPwIlISEJhYU5Pw==} @@ -1895,6 +2120,9 @@ packages: caniuse-lite@1.0.30001814: resolution: {integrity: sha512-/Uaf1lAzr59XcMpW0o96WoEfr+VXK2OX4U9AgFoiSHsVJ4HppnIFUjtYzsyDH2+tgANaQb2/oxYGwCPapN1FpA==} + ccount@2.0.1: + resolution: {integrity: sha512-eyrF0jiFpY+3drT6383f1qhkbGsLSifNAjA61IUjZjmLCWjItY6LB9ft9YhoDgwfmclB2zhu51Lc7+95b8NRAg==} + chai@6.3.0: resolution: {integrity: sha512-XWAtwJ6OHO+tj0EKCs0Y2UamnyOxseZWltU4x2U2wh8g4AigdjwvtUjvLP2tqkA/avxHEtzxNaqGq/YGNwckKg==} engines: {node: '>=18'} @@ -1907,6 +2135,18 @@ packages: resolution: {integrity: sha512-7NzBL0rN6fMUW+f7A6Io4h40qQlG+xGmtMxfbnH/K7TAtt8JQWVQK+6g0UXKMeVJoyV5EkkNsErQ8pVD3bLHbA==} engines: {node: ^12.17.0 || ^14.13 || >=16.0.0} + character-entities-html4@2.1.0: + resolution: {integrity: sha512-1v7fgQRj6hnSwFpq1Eu0ynr/CDEw0rXo2B61qXrLNdHZmPKgb7fqS1a2JwF0rISo9q77jDI8VMEHoApn8qDoZA==} + + character-entities-legacy@3.0.0: + resolution: {integrity: sha512-RpPp0asT/6ufRm//AJVwpViZbGM/MkjQFxJccQRHmISF/22NBtsHqAWmL+/pmkPWoIUJdWyeVleTl1wydHATVQ==} + + character-entities@2.0.2: + resolution: {integrity: sha512-shx7oQ0Awen/BRIdkjkvz54PnEEI/EjwXDSIZp86/KKdbafHh1Df/RYGBhn4hbe2+uKC9FnT5UCEdyPz3ai9hQ==} + + character-reference-invalid@2.0.1: + resolution: {integrity: sha512-iBZ4F4wRbyORVsu0jPV7gXkOsGYjGHPmAyv+HiHG8gi5PtC9KI2j1+v8/tlibRvjoWX027ypmG/n0HtO5t7unw==} + class-variance-authority@0.7.1: resolution: {integrity: sha512-Ka+9Trutv7G8M6WT6SeiRWz792K5qEqIGEGzXKhAE6xOWAY6pPH8U+9IY3oCMv6kqTmLsv7Xh/2w2RigkePMsg==} @@ -1945,6 +2185,9 @@ packages: color-name@1.1.4: resolution: {integrity: sha512-dOy+3AuW3a2wNbZHIuMZpTcgjGuLU/uBL/ubcZF9OXbDo8ff4O8yVp5Bf0efS8uEoYo5q4Fx7dY9OgQGXgAsQA==} + comma-separated-tokens@2.0.3: + resolution: {integrity: sha512-Fu4hJdvzeylCfQPp9SGWidpzrMs7tTrlu6Vb8XGaRGck8QSNZJJp538Wrb60Lax4fPwR64ViY468OIUTbRlGZg==} + commander@11.1.0: resolution: {integrity: sha512-yPVavfyCcRhmorC7rWlkHn15b4wDVgVmBA7kV4QVBsF7kv/9TKJAbAXVTxvTnwP8HHKjRCJDClKbciiYS7p0DQ==} engines: {node: '>=16'} @@ -2044,6 +2287,9 @@ packages: supports-color: optional: true + decode-named-character-reference@1.3.0: + resolution: {integrity: sha512-GtpQYB283KrPp6nRw50q3U9/VfOutZOe103qlN7BPP6Ad27xYnOIWv4lPzo8HCAL+mMZofJ9KEy30fq6MfaK6Q==} + dedent@1.7.2: resolution: {integrity: sha512-WzMx3mW98SN+zn3hgemf4OzdmyNhhhKz5Ay0pUfQiMQ3e1g+xmTJWp/pKdwKVXhdSkAEGIIzqeuWrL3mV/AXbA==} peerDependencies: @@ -2087,6 +2333,10 @@ packages: resolution: {integrity: sha512-g7nH6P6dyDioJogAAGprGpCtVImJhpPk/roCzdb3fIh61/s/nPsfR6onyMwkCAR/OlC3yBC0lESvUoQEAssIrw==} engines: {node: '>= 0.8'} + dequal@2.0.3: + resolution: {integrity: sha512-0je+qPKHEMohvfRTCEo3CrPG6cAzAYgmzKyxRiYSSDkS6eGJdyVJm7WaYA5ECaAD9wLB2T4EEeymA5aFVcYXCA==} + engines: {node: '>=6'} + detect-libc@2.1.2: resolution: {integrity: sha512-Btj2BOOO83o3WyH59e8MgXsxEQVcarkUOpEYrubB0urwnN10yQ364rsiByU11nZlqWYZm05i/of7io4mzihBtQ==} engines: {node: '>=8'} @@ -2094,6 +2344,9 @@ packages: detect-node-es@1.1.0: resolution: {integrity: sha512-ypdmJU/TbBby2Dxibuv7ZLW3Bs1QEmM7nHjEANfohJLvE0XVujisn1qPJcZxg+qDucsr+bP6fLD1rPS3AhJ7EQ==} + devlop@1.1.0: + resolution: {integrity: sha512-RWmIqhcFf1lRYBvNmr7qTNuyCt/7/ns2jbpp1+PalgE/rDQcBT0fioSMUpJ93irlUhC5hrg4cYqe6U+0ImW0rA==} + diff@8.0.4: resolution: {integrity: sha512-DPi0FmjiSU5EvQV0++GFDOJ9ASQUVFh5kD+OzOnYdi7n3Wpm9hWWGfB/O2blfHcMVTL5WkQXSnRiK9makhrcnw==} engines: {node: '>=0.3.1'} @@ -2184,6 +2437,11 @@ packages: resolution: {integrity: sha512-yPDz7wqpg1/mmHLmS3tcfTfbw5f1eryXvyghYBffGdERwe+mV7ZcWzTR8LR17Kvqt3qfPurjlonmnq3MKXIOXw==} engines: {node: '>= 0.4'} + esbuild@0.28.2: + resolution: {integrity: sha512-HKVLS8dvII+xoKW9kmqxbRKrnWEXfJJr/FZhhJmiqIB0e053QNYFqOBouTMO/k5sID4MvCiUCvv8b9M4h32wIA==} + engines: {node: '>=18'} + hasBin: true + escalade@3.2.0: resolution: {integrity: sha512-WUj2qlxaQtO4g6Pq5c29GTcWGDyd8itL8zTlipgECz3JesAiiOKotd8JU6otB3PACgG6xkJUyVhboMS+bje/jA==} engines: {node: '>=6'} @@ -2195,6 +2453,10 @@ packages: resolution: {integrity: sha512-TtpcNJ3XAzx3Gq8sWRzJaVajRs0uVxA2YAkdb1jm2YkPz4G6egUFAyA3n5vtEIZefPk5Wa4UXbKuS5fKkJWdgA==} engines: {node: '>=10'} + escape-string-regexp@5.0.0: + resolution: {integrity: sha512-/veY75JbMK4j1yjvuUxuVsiS/hr/4iHs9FTT6cgTexxdE0Ly/glccBAkloH/DofkjRbZU3bnoj38mOmhkZ0lHw==} + engines: {node: '>=12'} + eslint-config-next@16.3.8: resolution: {integrity: sha512-81vovwMe6NGnoFsl0KUJWzlS+y239i3fdsxs49gSEX3pDQW2cuKy+fxUPXNkQqqx6OCPheYPtBlSufc9aZ0k+w==} peerDependencies: @@ -2317,6 +2579,9 @@ packages: resolution: {integrity: sha512-MMdARuVEQziNTeJD8DgMqmhwR11BRQ/cBP+pLtYdSTnf3MIO8fFeiINEbX36ZdNlfU/7A9f3gUw49B3oQsvwBA==} engines: {node: '>=4.0'} + estree-util-is-identifier-name@3.0.0: + resolution: {integrity: sha512-hFtqIDZTIUZ9BXLb8y4pYGyk6+wekIivNVTcmvk8NoOh+VeRn5y6cEHzbURrWbfp1fIqdVipilzj+lfaadNZmg==} + estree-walker@3.0.3: resolution: {integrity: sha512-7RUKfXgSMMkzt6ZuXmqapOurLGPPfgj6l9uRZ7lRGolvk0y2yocc35LdcxKC5PQZdn2DMqioAQ2NoWcrTKmm6g==} @@ -2358,6 +2623,9 @@ packages: resolution: {integrity: sha512-hIS4idWWai69NezIdRt2xFVofaF4j+6INOpJlVOLDO8zXGpUVEVzIYk12UUi2JzjEzWL3IOAxcTubgz9Po0yXw==} engines: {node: '>= 18'} + extend@3.0.2: + resolution: {integrity: sha512-fjquC59cD7CyW6urNXK0FBufkZcoiGG80wTuPujX590cB5Ttln20E2UB4S/WARVqhXffZl2LNgS+gQdPIIim/g==} + fast-deep-equal@3.1.3: resolution: {integrity: sha512-f3qQ9oQy9j2AhBe/H9VC91wLmKBCCU/gDOnKNAYG5hswO7BLKj09Hc5HYNz9cGI++xlpDCIgDaitVs03ATR84Q==} @@ -2375,6 +2643,9 @@ packages: fast-levenshtein@2.0.6: resolution: {integrity: sha512-DCXu6Ifhqcks7TZKY3Hxp3y6qphY5SJZmrWMDrKcERSOXWQdMhU9Ig/PYrzyw/ul9jOIyh0N4M0tbC5hodg8dw==} + fast-sha256@1.3.0: + resolution: {integrity: sha512-n11RGP/lrWEFI/bWdygLxhI+pVeo1ZYIVwvvPkW7azl/rOy+F3HYRZ2K5zeE9mmkhQppyv9sQFx0JM9UabnpPQ==} + fast-uri@3.1.8: resolution: {integrity: sha512-GZMtZUTNRpOVIECoXwLNZS5xUGE+mVNbTB8h/7Rwh2TFWcBQiPzTgyZi05BF9UMZKkLJv8XBRJTlU7zg8+ZfMg==} @@ -2552,6 +2823,12 @@ packages: resolution: {integrity: sha512-T2UbfbBEF32wiepXIsMlTW9+dDYC6wMh/t/vYA4tuOMKqWz/n3vr1NFSxQiyP+zk2mXsoMA/i/7qV6LKut1t1A==} engines: {node: '>= 0.4'} + hast-util-to-jsx-runtime@2.3.6: + resolution: {integrity: sha512-zl6s8LwNyo1P9uw+XJGvZtdFF1GdAkOg8ujOw+4Pyb76874fLps4ueHXDhXWdk6YHQ6OgUtinliG7RsYvCbbBg==} + + hast-util-whitespace@3.0.0: + resolution: {integrity: sha512-88JUN06ipLwsnv+dVn+OIYOvAuvBMy/Qoi6O7mQHxdPXpjy+Cd6xRkWwux7DKO+4sYILtLBRIKgsdpS2gQc7qw==} + hermes-estree@0.25.1: resolution: {integrity: sha512-0wUoCcLp+5Ev5pDW2OriHC2MJCbwLwuRx+gAqMTOkGKJJiBCLjtrvy4PWUGn6MIVefecRpzoOZ/UV6iGdOr+Cw==} @@ -2562,6 +2839,9 @@ packages: resolution: {integrity: sha512-CQ46U0ZkAGmbT/4UxdzzGJpacP2IeKgY4a5/tOI9AABbpOMfK739wfDXmv1usCk+3RkKj1hQy4/fjhiwa2xlrA==} engines: {node: '>=16.9.0'} + html-url-attributes@3.0.1: + resolution: {integrity: sha512-ol6UPyBWqsrO6EJySPz2O7ZSr856WDrEzM5zMqp+FJJLGMW35cLYmmZnl0vztAZxRUoNZJFTCohfjuIJ8I4QBQ==} + http-errors@2.0.1: resolution: {integrity: sha512-4FbRdAX+bSdmo4AUFuS0WNiPz8NgFt+r8ThgNWmlrjQjt1Q7ZR9+zTlce2859x4KSXrwIsaeTqDoKQmtP8pLmQ==} engines: {node: '>= 0.8'} @@ -2597,6 +2877,9 @@ packages: inherits@2.0.4: resolution: {integrity: sha512-k/vGaX4/Yla3WzyMCvTQOXYeIHvqOKtnqBduzTHpzpQZzAskKMhZ2K+EnBiSM9zGSoIFeMpXKxa4dYeZIQqewQ==} + inline-style-parser@0.2.7: + resolution: {integrity: sha512-Nb2ctOyNR8DqQoR0OwRG95uNWIC0C1lCgf5Naz5H6Ji72KZ8OcFZLz2P5sNgwlyoJ8Yif11oMuYs5pBQa86csA==} + internal-slot@1.1.0: resolution: {integrity: sha512-4gd7VpWNQNB4UKKCFFVcp1AVv+FMOgs9NKzjHKusc8jTMhd5eL1NqQqOpE0KzMds804/yHlglp3uxgluOqAPLw==} engines: {node: '>= 0.4'} @@ -2609,6 +2892,12 @@ packages: resolution: {integrity: sha512-0KI/607xoxSToH7GjN1FfSbLoU0+btTicjsQSWQlh/hZykN8KpmMf7uYwPW3R+akZ6R/w18ZlXSHBYXiYUPO3g==} engines: {node: '>= 0.10'} + is-alphabetical@2.0.1: + resolution: {integrity: sha512-FWyyY60MeTNyeSRpkM2Iry0G9hpr7/9kD40mD/cGQEuilcZYS4okz8SN2Q6rLCJ8gbCt6fN+rC+6tMGS99LaxQ==} + + is-alphanumerical@2.0.1: + resolution: {integrity: sha512-hmbYhX/9MUMF5uh7tOXyK/n0ZvWpad5caBA17GsC6vyuCqaWliRG5K1qS9inmUhEMaOBIW7/whAnSwveW/LtZw==} + is-array-buffer@3.0.5: resolution: {integrity: sha512-DDfANUiiG2wC1qawP66qlTugJeL5HyzMpfr8lLK+jMQirGzNod0B12cFB/9q838Ru27sBwfw78/rdoU7RERz6A==} engines: {node: '>= 0.4'} @@ -2647,6 +2936,9 @@ packages: resolution: {integrity: sha512-PwwhEakHVKTdRNVOw+/Gyh0+MzlCl4R6qKvkhuvLtPMggI1WAHt9sOwZxQLSGpUaDnrdyDsomoRgNnCfKNSXXg==} engines: {node: '>= 0.4'} + is-decimal@2.0.1: + resolution: {integrity: sha512-AAB9hiomQs5DXWcRB1rqsxGUstbRroFOPPVAomNk/3XHR5JyEZChOyTWe2oayKnsSsr/kcGqF+z6yuH6HHpN0A==} + is-docker@2.2.1: resolution: {integrity: sha512-F+i2BKsFrH66iaUFc0woD8sLy8getkwTwtOBjvs56Cx4CgJDeKQeqfz8wAYiSb8JOprWhHH5p77PbmYCvvUuXQ==} engines: {node: '>=8'} @@ -2677,6 +2969,9 @@ packages: resolution: {integrity: sha512-xelSayHH36ZgE7ZWhli7pW34hNbNl8Ojv5KVmkJD4hBdD3th8Tfk9vYasLM+mXWOZhFkgZfxhLSnrwRr4elSSg==} engines: {node: '>=0.10.0'} + is-hexadecimal@2.0.1: + resolution: {integrity: sha512-DgZQp241c8oO6cA1SbTEWiXeoxV42vlcJxgH+B3hi1AiqqKruZR3ZGF8In3fj4+/y/7rHvlOZLZtgJ/4ttYGZg==} + is-in-ssh@1.0.0: resolution: {integrity: sha512-jYa6Q9rH90kR1vKB6NM7qqd1mge3Fx4Dhw5TVlK1MUBqhEOuCagrEHMevNuCcbECmXZ0ThXkRm+Ymr51HwEPAw==} engines: {node: '>=20'} @@ -2824,6 +3119,10 @@ packages: json-parse-even-better-errors@2.3.1: resolution: {integrity: sha512-xyFwyhro/JEof6Ghe2iz2NcXoj2sloNsWr/XsERDK/oiPCfaNhl5ONfp+jQdAZRQQ0IJWNzH9zIZF7li91kh2w==} + json-schema-to-ts@3.1.1: + resolution: {integrity: sha512-+DWg8jCJG2TEnpy7kOm/7/AxaYoaRbjVB4LFZLySZlWn8exGs3A4OLJR966cVvU26N7X9TWxl+Jsw7dzAqKT6g==} + engines: {node: '>=16'} + json-schema-traverse@0.4.1: resolution: {integrity: sha512-xbbCH5dCYU5T8LcEhhuh7HJ88HXuW3qsI3Y0zOZFKfZEHcpWiHU/Jxzk629Brsab/mMiHQti9wMP+845RPe3Vg==} @@ -3043,6 +3342,9 @@ packages: resolution: {integrity: sha512-i24m8rpwhmPIS4zscNzK6MSEhk0DUWa/8iYQWxhffV8jkI4Phvs3F+quL5xvS0gdQR0FyTCMMH33Y78dDTzzIw==} engines: {node: '>=18'} + longest-streak@3.1.0: + resolution: {integrity: sha512-9Ri+o0JYgehTaVBBDoMqIl8GXtbWg711O3srftcHhZ0dqnETqLaoIK0x17fUw9rFSlK/0NlsKe0Ahhyl5pXE2g==} + loose-envify@1.4.0: resolution: {integrity: sha512-lyuxPGr/Wfhrlem2CL/UcnUc1zcqKAImBDzukY7Y5F/yQiNdko6+fRLevlw1HgMySw7f611UIY408EtxRSoK3Q==} hasBin: true @@ -3065,10 +3367,58 @@ packages: magic-string@1.4.3: resolution: {integrity: sha512-z12OxmaPGE0F4xlpGdjuAUckipLpQlTAuAmyGhNoD7ODpYUzvDfvtxUbjM/jwznyP178oju/KDdyi220wNJ0RQ==} + markdown-table@3.0.4: + resolution: {integrity: sha512-wiYz4+JrLyb/DqW2hkFJxP7Vd7JuTDm77fvbM8VfEQdmSMqcImWeeRbHwZjBjIFki/VaMK2BhFi7oUUZeM5bqw==} + math-intrinsics@1.1.0: resolution: {integrity: sha512-/IXtbwEk5HTPyEwyKX6hGkYXxM9nbj64B+ilVJnC/R6B0pH5G4V3b0pVbL7DBj4tkhBAppbQUlf6F6Xl9LHu1g==} engines: {node: '>= 0.4'} + mdast-util-find-and-replace@3.0.3: + resolution: {integrity: sha512-xPgpDNl0/OXHsI7TlaIs22lWnH3KlpvZdXG1ET/m/YT2Hhdkx8lJV4kLphE6l9simyM4KPDFBwas08BgEob6Jw==} + + mdast-util-from-markdown@2.1.0: + resolution: {integrity: sha512-KRLgmPaP2H1ul2bpoJkOS+9pAcDAAj9Nu1mqLnh8NLeJG5tlvj42TH80PIGBpROWlRXLk0lDrWNuZCFy/tsB4A==} + + mdast-util-gfm-autolink-literal@2.0.1: + resolution: {integrity: sha512-5HVP2MKaP6L+G6YaxPNjuL0BPrq9orG3TsrZ9YXbA3vDw/ACI4MEsnoDpn6ZNm7GnZgtAcONJyPhOP8tNJQavQ==} + + mdast-util-gfm-footnote@2.1.0: + resolution: {integrity: sha512-sqpDWlsHn7Ac9GNZQMeUzPQSMzR6Wv0WKRNvQRg0KqHh02fpTz69Qc1QSseNX29bhz1ROIyNyxExfawVKTm1GQ==} + + mdast-util-gfm-strikethrough@2.0.1: + resolution: {integrity: sha512-OuJHqvr455pwu2OaOrir7dbzqs4jlWKOtlu9L6GjcIjjQwNyw02VntQsr/Ek6/A13vgnXfUhBekWRUP1OkNivw==} + + mdast-util-gfm-table@2.0.0: + resolution: {integrity: sha512-78UEvebzz/rJIxLvE7ZtDd/vIQ0RHv+3Mh5DR96p7cS7HsBhYIICDBCu8csTNWNO6tBWfqXPWekRuj2FNOGOZg==} + + mdast-util-gfm-task-list-item@2.0.0: + resolution: {integrity: sha512-IrtvNvjxC1o06taBAVJznEnkiHxLFTzgonUdy8hzFVeDun0uTjxxrRGVaNFqkU1wJR3RBPEfsxmU6jDWPofrTQ==} + + mdast-util-gfm@3.1.0: + resolution: {integrity: sha512-0ulfdQOM3ysHhCJ1p06l0b0VKlhU0wuQs3thxZQagjcjPrlFRqY215uZGHHJan9GEAXd9MbfPjFJz+qMkVR6zQ==} + + mdast-util-mdx-expression@2.0.1: + resolution: {integrity: sha512-J6f+9hUp+ldTZqKRSg7Vw5V6MqjATc+3E4gf3CFNcuZNWD8XdyI6zQ8GqH7f8169MM6P7hMBRDVGnn7oHB9kXQ==} + + mdast-util-mdx-jsx@3.2.0: + resolution: {integrity: sha512-lj/z8v0r6ZtsN/cGNNtemmmfoLAFZnjMbNyLzBafjzikOM+glrjNHPlf6lQDOTccj9n5b0PPihEBbhneMyGs1Q==} + + mdast-util-mdxjs-esm@2.0.1: + resolution: {integrity: sha512-EcmOpxsZ96CvlP03NghtH1EsLtr0n9Tm4lPUJUBccV9RwUOneqSycg19n5HGzCf+10LozMRSObtVr3ee1WoHtg==} + + mdast-util-phrasing@4.1.0: + resolution: {integrity: sha512-TqICwyvJJpBwvGAMZjj4J2n0X8QWp21b9l0o7eXyVJ25YNWYbJDVIyD1bZXE6WtV6RmKJVYmQAKWa0zWOABz2w==} + + mdast-util-to-hast@13.2.1: + resolution: {integrity: sha512-cctsq2wp5vTsLIcaymblUriiTcZd0CwWtCbLvrOzYCDZoWyMNV8sZ7krj09FSnsiJi3WVsHLM4k6Dq/yaPyCXA==} + + mdast-util-to-markdown@2.2.0: + resolution: {integrity: sha512-2Jn/ADrHFFyyniBMi0bncTbJqU78K3/TEYvzvwrWWzD9MYPUa/Gm4r3nOeKFQg/PDM5BsbDYdc8ackW4NTY6NA==} + + mdast-util-to-string@4.0.0: + resolution: {integrity: sha512-0H44vDimn51F0YwvxSJSm0eCDOJTRlmN0R1yBh4HLj9wiV1Dn0QoXGbvFAWj2hSItVTlCmBF1hqKlIyUBVFLPg==} + media-typer@1.1.1: resolution: {integrity: sha512-yz3xRaG20c6/BOzvYoDaGtPmGscs7YivItZEEqe6GbwNfHuxu9YNmvnEkMzKldAGY4/80pRcQRZSEnhquk9XuQ==} engines: {node: '>= 0.8'} @@ -3084,6 +3434,93 @@ packages: resolution: {integrity: sha512-8q7VEgMJW4J8tcfVPy8g09NcQwZdbwFEqhe/WZkoIzjn/3TGDwtOCYtXGxA3O8tPzpczCCDgv+P2P5y00ZJOOg==} engines: {node: '>= 8'} + micromark-core-commonmark@2.0.4: + resolution: {integrity: sha512-wxEeE8v8XVvOrxn1TZj74qYhAtTQsSqufHVS3uNVyT1MupXxhyaHk8/9xDNPh9BmL77QNDgWTkZ3RsYHB2ry7w==} + + micromark-extension-gfm-autolink-literal@2.1.0: + resolution: {integrity: sha512-oOg7knzhicgQ3t4QCjCWgTmfNhvQbDDnJeVu9v81r7NltNCVmhPy1fJRX27pISafdjL+SVc4d3l48Gb6pbRypw==} + + micromark-extension-gfm-footnote@2.1.0: + resolution: {integrity: sha512-/yPhxI1ntnDNsiHtzLKYnE3vf9JZ6cAisqVDauhp4CEHxlb4uoOTxOCJ+9s51bIB8U1N1FJ1RXOKTIlD5B/gqw==} + + micromark-extension-gfm-strikethrough@2.1.0: + resolution: {integrity: sha512-ADVjpOOkjz1hhkZLlBiYA9cR2Anf8F4HqZUO6e5eDcPQd0Txw5fxLzzxnEkSkfnD0wziSGiv7sYhk/ktvbf1uw==} + + micromark-extension-gfm-table@2.1.2: + resolution: {integrity: sha512-pRzm4kDTu0MjlmBkxmS9yYhw60nncfcEwu9NNdPFSQEFXS95ZKyIIyTSHu/o3ReBUrLKYEq+7YaXCRn/bPB4MA==} + + micromark-extension-gfm-tagfilter@2.0.0: + resolution: {integrity: sha512-xHlTOmuCSotIA8TW1mDIM6X2O1SiX5P9IuDtqGonFhEK0qgRI4yeC6vMxEV2dgyr2TiD+2PQ10o+cOhdVAcwfg==} + + micromark-extension-gfm-task-list-item@2.1.0: + resolution: {integrity: sha512-qIBZhqxqI6fjLDYFTBIa4eivDMnP+OZqsNwmQ3xNLE4Cxwc+zfQEfbs6tzAo2Hjq+bh6q5F+Z8/cksrLFYWQQw==} + + micromark-extension-gfm@3.0.0: + resolution: {integrity: sha512-vsKArQsicm7t0z2GugkCKtZehqUm31oeGBV/KVSorWSy8ZlNAv7ytjFhvaryUiCUJYqs+NoE6AFhpQvBTM6Q4w==} + + micromark-factory-destination@2.0.1: + resolution: {integrity: sha512-Xe6rDdJlkmbFRExpTOmRj9N3MaWmbAgdpSrBQvCFqhezUn4AHqJHbaEnfbVYYiexVSs//tqOdY/DxhjdCiJnIA==} + + micromark-factory-label@2.0.1: + resolution: {integrity: sha512-VFMekyQExqIW7xIChcXn4ok29YE3rnuyveW3wZQWWqF4Nv9Wk5rgJ99KzPvHjkmPXF93FXIbBp6YdW3t71/7Vg==} + + micromark-factory-space@2.1.0: + resolution: {integrity: sha512-fS8hnLIjnjvdQIj39Geug8wWsR0HrYZ43KShKxNfwT7t2LOHo/LbWZEzEaSOjwtjdCsjoE6syHhonEzW0zv0+Q==} + + micromark-factory-title@2.0.1: + resolution: {integrity: sha512-5bZ+3CjhAd9eChYTHsjy6TGxpOFSKgKKJPJxr293jTbfry2KDoWkhBb6TcPVB4NmzaPhMs1Frm9AZH7OD4Cjzw==} + + micromark-factory-whitespace@2.0.1: + resolution: {integrity: sha512-Ob0nuZ3PKt/n0hORHyvoD9uZhr+Za8sFoP+OnMcnWK5lngSzALgQYKMr9RJVOWLqQYuyn6ulqGWSXdwf6F80lQ==} + + micromark-util-character@2.1.1: + resolution: {integrity: sha512-wv8tdUTJ3thSFFFJKtpYKOYiGP2+v96Hvk4Tu8KpCAsTMs6yi+nVmGh1syvSCsaxz45J6Jbw+9DD6g97+NV67Q==} + + micromark-util-chunked@2.0.1: + resolution: {integrity: sha512-QUNFEOPELfmvv+4xiNg2sRYeS/P84pTW0TCgP5zc9FpXetHY0ab7SxKyAQCNCc1eK0459uoLI1y5oO5Vc1dbhA==} + + micromark-util-classify-character@2.0.1: + resolution: {integrity: sha512-K0kHzM6afW/MbeWYWLjoHQv1sgg2Q9EccHEDzSkxiP/EaagNzCm7T/WMKZ3rjMbvIpvBiZgwR3dKMygtA4mG1Q==} + + micromark-util-combine-extensions@2.0.1: + resolution: {integrity: sha512-OnAnH8Ujmy59JcyZw8JSbK9cGpdVY44NKgSM7E9Eh7DiLS2E9RNQf0dONaGDzEG9yjEl5hcqeIsj4hfRkLH/Bg==} + + micromark-util-decode-numeric-character-reference@2.0.2: + resolution: {integrity: sha512-ccUbYk6CwVdkmCQMyr64dXz42EfHGkPQlBj5p7YVGzq8I7CtjXZJrubAYezf7Rp+bjPseiROqe7G6foFd+lEuw==} + + micromark-util-decode-string@2.0.1: + resolution: {integrity: sha512-nDV/77Fj6eH1ynwscYTOsbK7rR//Uj0bZXBwJZRfaLEJ1iGBR6kIfNmlNqaqJf649EP0F3NWNdeJi03elllNUQ==} + + micromark-util-edit-map@1.0.0: + resolution: {integrity: sha512-Pa2ljlsEL6sVwFaYeyrOLSYbQt73JvGbPOYFq+9AElXiSnfI5Q4465RRZ1sHv7lDjDOnGRDDaqPXqClqWK/q1Q==} + + micromark-util-encode@2.0.1: + resolution: {integrity: sha512-c3cVx2y4KqUnwopcO9b/SCdo2O67LwJJ/UyqGfbigahfegL9myoEFoDYZgkT7f36T0bLrM9hZTAaAyH+PCAXjw==} + + micromark-util-html-tag-name@2.0.1: + resolution: {integrity: sha512-2cNEiYDhCWKI+Gs9T0Tiysk136SnR13hhO8yW6BGNyhOC4qYFnwF1nKfD3HFAIXA5c45RrIG1ub11GiXeYd1xA==} + + micromark-util-normalize-identifier@2.0.1: + resolution: {integrity: sha512-sxPqmo70LyARJs0w2UclACPUUEqltCkJ6PhKdMIDuJ3gSf/Q+/GIe3WKl0Ijb/GyH9lOpUkRAO2wp0GVkLvS9Q==} + + micromark-util-resolve-all@2.0.1: + resolution: {integrity: sha512-VdQyxFWFT2/FGJgwQnJYbe1jjQoNTS4RjglmSjTUlpUMa95Htx9NHeYW4rGDJzbjvCsl9eLjMQwGeElsqmzcHg==} + + micromark-util-sanitize-uri@2.0.1: + resolution: {integrity: sha512-9N9IomZ/YuGGZZmQec1MbgxtlgougxTodVwDzzEouPKo3qFWvymFHWcnDi2vzV1ff6kas9ucW+o3yzJK9YB1AQ==} + + micromark-util-subtokenize@2.1.0: + resolution: {integrity: sha512-XQLu552iSctvnEcgXw6+Sx75GflAPNED1qx7eBJ+wydBb2KCbRZe+NwvIEEMM83uml1+2WSXpBAcp9IUCgCYWA==} + + micromark-util-symbol@2.0.1: + resolution: {integrity: sha512-vs5t8Apaud9N28kgCrRUdEed4UJ+wWNvicHLPxCa9ENlYuAY31M0ETy5y1vA33YoNPDFTghEbnh6efaE8h4x0Q==} + + micromark-util-types@2.0.3: + resolution: {integrity: sha512-oxB2Ik03hI0gv+VNn9tnh1t1YEe9MDPptViAEgfdf3YQHsn0pzGTgCdlSCJXcwhqm8phaEuM7zeEu3QQzVBrPg==} + + micromark@4.0.3: + resolution: {integrity: sha512-oGYfQzHSG5dOMovQcJ3fyTmZlWAWpi0XA0sJwJs+i6OT88o1+Jtw/8z0CmdowSLxGhhFK89rQ/oXph/wN02PNw==} + micromatch@4.0.8: resolution: {integrity: sha512-PXwfBhYu0hBCPw8Dn0E+WDYb7af3dSLVWKi3HGv84IdF4TyFoC0ysxFd0Goxw7nSv4T/PzEJQxsYsEiFCKo2BA==} engines: {node: '>=8.6'} @@ -3280,6 +3717,9 @@ packages: resolution: {integrity: sha512-GQ2EWRpQV8/o+Aw8YqtfZZPfNRWZYkbidE9k5rpl/hC3vtHHBfGm2Ifi6qWV+coDGkrUKZAxE3Lot5kcsRlh+g==} engines: {node: '>=6'} + parse-entities@4.0.2: + resolution: {integrity: sha512-GG2AQYWoLgL877gQIKeRPGO1xF9+eG1ujIb5soS5gPvLQ1y2o8FL90w2QWNdf9I361Mpp7726c+lj3U0qK1uGw==} + parse-json@5.2.0: resolution: {integrity: sha512-ayCKvm/phCGxOkYRSCM82iDwct8/EonSEgCSxWxD7ve6jHggsFl4fZVQBPRNgQoKiuV/odhFrGzQXZwbifC8Rg==} engines: {node: '>=8'} @@ -3336,6 +3776,16 @@ packages: resolution: {integrity: sha512-nDywThFk1i4BQK4twPQ6TA4RT8bDY96yeuCVBWL3ePARCiEKDRSrNGbFIgUJpLp+XeIR65v8ra7WuJOFUBtkMA==} engines: {node: '>=8'} + playwright-core@1.63.0: + resolution: {integrity: sha512-rYCsBF/M5HjUch52bbtVONEFjv6Xu8sm8h72dNlR5bzIE1fvC/bxgspzkjSfU+MweEMmPM8KJebG6nnyxo5mCg==} + engines: {node: '>=20'} + hasBin: true + + playwright@1.63.0: + resolution: {integrity: sha512-+7ziBLidS4NaNCdt57SUDT+wYmmd5fmiQejUic/kb+YsYSCPyOOE9sebzMjNmQrsnNpDJqd4WHvV/8lfKfUDUg==} + engines: {node: '>=20'} + hasBin: true + possible-typed-array-names@1.1.0: resolution: {integrity: sha512-/+5VFTchJDoVj3bhoqi6UeymcD00DAwb1nJwamzPvHEszJ4FpF6SNNbUbOS8yI56qHzdV8eK0qEfOSiodkTdxg==} engines: {node: '>= 0.4'} @@ -3431,6 +3881,9 @@ packages: prop-types@15.8.1: resolution: {integrity: sha512-oj87CgZICdulUohogVAR7AjlC0327U4el4L6eAvOqCeudMDVU0NThNaV+b9Df4dXgSP1gXMTnPdhfe/2qDH5cg==} + property-information@7.2.0: + resolution: {integrity: sha512-IAtzIB6sUiWaJYrX9smp3V46pBGbBeLFRGdh25kg1334VcBlD8HzhPeNIWQH9zhGmo2itIe25EHt9dQP7G5hmg==} + proxy-addr@2.0.8: resolution: {integrity: sha512-5nnx0yGyVUcY6t9RnWcARWtwT9F1D8O9rt08htPvnd49W1IgZtmLkhu9WfMzQj1cFxjHIO6connUNVW5k7AVyQ==} engines: {node: '>= 0.10'} @@ -3475,6 +3928,12 @@ packages: react-is@16.13.1: resolution: {integrity: sha512-24e6ynE2H+OKt4kqsOvNd8kBpV65zoxbA4BVsEOB3ARVWQki/DHzaUoC5KuON/BiccDaCCTZBuOcfZs70kR8bQ==} + react-markdown@10.1.0: + resolution: {integrity: sha512-qKxVopLT/TyA6BX3Ue5NwabOsAzm0Q7kAPwq6L+wWDwisYs7R8vZ0nRXqq6rkueboxpkjvLGU9fWifiX/ZZFxQ==} + peerDependencies: + '@types/react': '>=18' + react: '>=18' + react-remove-scroll-bar@2.3.8: resolution: {integrity: sha512-9r+yi9+mgU33AKcj6IbT9oRCO78WriSj6t/cF8DWBZJ9aOGPOTEDvdUDz1FwKim7QXWwmHqtdHnRJfhAxEG46Q==} engines: {node: '>=10'} @@ -3521,6 +3980,18 @@ packages: resolution: {integrity: sha512-dYqgNSZbDwkaJ2ceRd9ojCGjBq+mOm9LmtXnAnEGyHhN/5R7iDW2TRw3h+o/jCFxus3P2LfWIIiwowAjANm7IA==} engines: {node: '>= 0.4'} + remark-gfm@4.0.1: + resolution: {integrity: sha512-1quofZ2RQ9EWdeN34S79+KExV1764+wCUGop5CPL1WGdD0ocPpu91lzPGbwWMECpEpd42kJGQwzRfyov9j4yNg==} + + remark-parse@11.0.0: + resolution: {integrity: sha512-FCxlKLNGknS5ba/1lmpYijMUzX2esxW5xQqjWxw2eHFfS2MSdaHVINFmhjo+qN1WhZhNimq0dZATN9pH0IDrpA==} + + remark-rehype@11.1.2: + resolution: {integrity: sha512-Dh7l57ianaEoIpzbp0PC9UKAdCSVklD8E5Rpw7ETfbTl3FqcOOgq5q2LVDhgGCkaBv7p24JXikPdvhhmHvKMsw==} + + remark-stringify@11.0.0: + resolution: {integrity: sha512-1OSmLd3awB/t8qdoEOMazZkNsfVTeY4fTsgzcQFdXNq8ToTN4ZGwrMnlda4K6smTFKD+GRV6O48i6Z4iKgPPpw==} + require-from-string@2.0.2: resolution: {integrity: sha512-Xf0nWe6RseziFMu+Ap9biiUbmplq6S9/p+7w7YXP/JBHhrUDDUhwa+vANyubuqfZWTveU//DYVGsDG7RKL/vEw==} engines: {node: '>=0.10.0'} @@ -3675,9 +4146,15 @@ packages: resolution: {integrity: sha512-UjgapumWlbMhkBgzT7Ykc5YXUT46F0iKu8SGXq0bcwP5dz/h0Plj6enJqjz1Zbq2l5WaqYnrVbwWOWMyF3F47g==} engines: {node: '>=0.10.0'} + space-separated-tokens@2.0.2: + resolution: {integrity: sha512-PEGlAwrG8yXGXRjW32fGbg66JAlOAwbObuqVoJpv/mRgoWDQfgH1wDPvtzWyUSNAXBGSk8h755YDbbcEy3SH2Q==} + stable-hash@0.0.5: resolution: {integrity: sha512-+L3ccpzibovGXFK+Ap/f8LOS0ahMrHTf3xu7mMLSpEGU0EO9ucaysSylKo9eRDFNhWve/y275iPmIZ4z39a9iA==} + standardwebhooks@1.1.1: + resolution: {integrity: sha512-bCbX9ZEyFkWPsRz7Bl3NuQUJohmwGSev/yhr7vhaGPlc4AfIrspIRa6cPTBuI1ItmrTDJ4d/S2hCsfe4+vQGnQ==} + statuses@2.0.2: resolution: {integrity: sha512-DvEy55V3DB7uknRo+4iOGT5fP1slR8wQohVdknigZPMpMstaKJQWhwiYBACJE3Ul2pTnATihhBYnRhZQHGBiRw==} engines: {node: '>= 0.8'} @@ -3720,6 +4197,9 @@ packages: resolution: {integrity: sha512-UXSH262CSZY1tfu3G3Secr6uGLCFVPMhIqHjlgCUtCCcgihYc/xKs9djMTMUOb2j1mVSeU8EU6NWc/iQKU6Gfg==} engines: {node: '>= 0.4'} + stringify-entities@4.0.4: + resolution: {integrity: sha512-IwfBptatlO+QCJUo19AqvrPNqlVMpW9YEL2LIVY+Rpv2qsjCGxaDLNRgeGsQWJhfItebuJhsGSLjaBbNSQ+ieg==} + stringify-object@5.0.0: resolution: {integrity: sha512-zaJYxz2FtcMb4f+g60KsRNFOpVMUyuJgA51Zi5Z1DOTC3S59+OQiVOzE9GZt0x72uBGWKsQIuBKeF9iusmKFsg==} engines: {node: '>=14.16'} @@ -3748,6 +4228,12 @@ packages: resolution: {integrity: sha512-6fPc+R4ihwqP6N/aIv2f1gMH8lOVtWQHoqC4yK6oSDVVocumAsfCqjkXnqiYMhmMwS/mEHLp7Vehlt3ql6lEig==} engines: {node: '>=8'} + style-to-js@1.1.21: + resolution: {integrity: sha512-RjQetxJrrUJLQPHbLku6U/ocGtzyjbJMP9lCNK7Ag0CNh690nSH8woqWH9u16nMjYBAok+i7JO1NP2pOy8IsPQ==} + + style-to-object@1.0.14: + resolution: {integrity: sha512-LIN7rULI0jBscWQYaSswptyderlarFkjQ+t79nzty8tcIAceVomEVlLzH5VP4Cmsv6MtKhs7qaAiwlcp+Mgaxw==} + styled-jsx@5.1.6: resolution: {integrity: sha512-qSVyDTeMotdvQYoHWLNGwRFJHC+i+ZvdBRYosOFgC+Wg1vx4frN2/RG/NA7SYqqvKNLf39P2LSRA2pu6n0XYZA==} engines: {node: '>= 12.0.0'} @@ -3805,6 +4291,15 @@ packages: resolution: {integrity: sha512-o5sSPKEkg/DIQNmH43V0/uerLrpzVedkUh8tGNvaeXpfpuwjKenlSox/2O/BTlZUtEe+JG7s5YhEz608PlAHRA==} engines: {node: '>=0.6'} + trim-lines@3.0.1: + resolution: {integrity: sha512-kRj8B+YHZCc9kQYdWfJB2/oUl9rA99qbowYYBtr4ui4mZyAQ2JpvVBd/6U2YloATfqBhBTSMhTpgBHtU0Mf3Rg==} + + trough@2.2.0: + resolution: {integrity: sha512-tmMpK00BjZiUyVyvrBK7knerNgmgvcV/KLVyuma/SC+TQN167GrMRciANTz09+k3zW8L8t60jWO1GpfkZdjTaw==} + + ts-algebra@2.0.0: + resolution: {integrity: sha512-FPAhNPFMrkwz76P7cdjdmiShwMynZYN6SgOujD1urY4oNm80Ou9oMdmbR45LotcKOXoy7wSmHkRFE6Mxbrhefw==} + ts-api-utils@2.5.0: resolution: {integrity: sha512-OJ/ibxhPlqrMM0UiNHJ/0CKQkoKF243/AEmplt3qpRgkW8VG7IfOS41h7V8TjITqdByHzrjcS/2si+y4lIh8NA==} engines: {node: '>=18.12'} @@ -3824,6 +4319,11 @@ packages: tslib@2.8.1: resolution: {integrity: sha512-oJFu94HQb+KVduSUQL7wnpmqnfmLsOA/nAh6b6EH0wCEoK0/mPeXU6c3wKDV83MkOuHPRHtSXKKU99IBazS/2w==} + tsx@4.23.15: + resolution: {integrity: sha512-Yiex1Ovn8z2xPpOWckIiysV1SSyRMY9BkLF++q0yKiDxCqRhosKfMg3janKkiLBwZ5c/YryloKwGZcrEmtwxKw==} + engines: {node: '>=18.0.0'} + hasBin: true + tw-animate-css@1.4.0: resolution: {integrity: sha512-7bziOlRqH0hJx80h/3mbicLW7o8qLsH5+RaLR2t+OHM3D0JlWGODQKQ4cxbK7WlvmUxpcj6Kgu6EKqjrGFe3QQ==} @@ -3882,6 +4382,24 @@ packages: resolution: {integrity: sha512-+QBBXBCvifc56fsbuxZQ6Sic3wqqc3WWaqxs58gvJrcOuN83HGTCwz3oS5phzU9LthRNE9VrJCFCLUgHeeFnfA==} engines: {node: '>=18'} + unified@11.0.5: + resolution: {integrity: sha512-xKvGhPWw3k84Qjh8bI3ZeJjqnyadK+GEFtazSfZv/rKeTkTjOJho6mFqh2SM96iIcZokxiOpg78GazTSg8+KHA==} + + unist-util-is@6.0.1: + resolution: {integrity: sha512-LsiILbtBETkDz8I9p1dQ0uyRUWuaQzd/cuEeS1hoRSyW5E5XGmTzlwY1OrNzzakGowI9Dr/I8HVaw4hTtnxy8g==} + + unist-util-position@5.0.0: + resolution: {integrity: sha512-fucsC7HjXvkB5R3kTCO7kUjRdrS0BJt3M/FPxmHMBOm8JQi2BsHAHFsy27E0EolP8rp0NzXsJ+jNPyDWvOJZPA==} + + unist-util-stringify-position@4.0.0: + resolution: {integrity: sha512-0ASV06AAoKCDkS2+xw5RXJywruurpbC4JZSm7nr7MOt1ojAzvyyaO+UxZf18j8FCF6kmzCZKcAgN/yu2gm2XgQ==} + + unist-util-visit-parents@6.0.2: + resolution: {integrity: sha512-goh1s1TBrqSqukSc8wrjwWhL0hiJxgA8m4kFxGlQ+8FYQ3C/m11FcTs4YYem7V664AhHVvgoQLk890Ssdsr2IQ==} + + unist-util-visit@5.1.0: + resolution: {integrity: sha512-m+vIdyeCOpdr/QeQCu2EzxX/ohgS8KbnPDgFni4dQsfSCtpz8UqDyY5GjRru8PDKuYn7Fq19j1CQ+nJSsGKOzg==} + universalify@2.0.1: resolution: {integrity: sha512-gptHNQghINnc/vTGIk0SOFGFNXw7JVrlRUtConJRlvaw6DuX0wO5Jeko9sWrMBhh+PsYAZ7oXAiOnf/UKogyiw==} engines: {node: '>= 10.0.0'} @@ -3933,6 +4451,12 @@ packages: resolution: {integrity: sha512-BNGbWLfd0eUPabhkXUVm0j8uuvREyTh5ovRa/dyow/BqAbZJyC+5fU+IzQOzmAKzYqYRAISoRhdQr3eIZ/PXqg==} engines: {node: '>= 0.8'} + vfile-message@4.0.3: + resolution: {integrity: sha512-QTHzsGd1EhbZs4AsQ20JX1rC3cOlt/IWJruk893DfLRr57lcnOeMaWG4K0JrRta4mIJZKth2Au3mM3u03/JWKw==} + + vfile@6.0.3: + resolution: {integrity: sha512-KzIbH/9tXat2u30jf+smMwFCsno4wHVdNmzFyL+T/L3UGqqk6JKfVqOFOZEpZSHADH1k40ab6NUIXZq422ov3Q==} + vite@8.3.2: resolution: {integrity: sha512-SQr1x6W5vVSbROg7vsyXIaxK9b0G7zsT68acdWWRmnBUsgDieLCRG+Rep9WdZgcposvv/GSnr4GUUBqB3vXq6w==} engines: {node: ^20.19.0 || >=22.12.0} @@ -4091,10 +4615,20 @@ packages: zod@4.6.5: resolution: {integrity: sha512-v5l/aFXZQeai4awLbOpSoHecE9UiMrnfx75tEXLjNonXVARxQ5mOeipTjROUchszUNCqnE+hqAMujRsRHsut2Q==} + zwitch@2.0.4: + resolution: {integrity: sha512-bXE4cR/kVZhKZX/RjPEflHaKVhUVl85noU3v6b8apfQEc1x4A+zBxjZ4lN8LqGd6WZ3dl98pY4o717VFmoPp+A==} + snapshots: '@alloc/quick-lru@5.3.0': {} + '@anthropic-ai/sdk@0.131.0(zod@4.6.5)': + dependencies: + json-schema-to-ts: 3.1.1 + standardwebhooks: 1.1.1 + optionalDependencies: + zod: 4.6.5 + '@babel/code-frame@7.29.7': dependencies: '@babel/helper-validator-identifier': 7.29.7 @@ -4258,6 +4792,8 @@ snapshots: transitivePeerDependencies: - supports-color + '@babel/runtime@7.29.7': {} + '@babel/template@7.29.7': dependencies: '@babel/code-frame': 7.29.7 @@ -4323,6 +4859,84 @@ snapshots: tslib: 2.8.1 optional: true + '@esbuild/aix-ppc64@0.28.2': + optional: true + + '@esbuild/android-arm64@0.28.2': + optional: true + + '@esbuild/android-arm@0.28.2': + optional: true + + '@esbuild/android-x64@0.28.2': + optional: true + + '@esbuild/darwin-arm64@0.28.2': + optional: true + + '@esbuild/darwin-x64@0.28.2': + optional: true + + '@esbuild/freebsd-arm64@0.28.2': + optional: true + + '@esbuild/freebsd-x64@0.28.2': + optional: true + + '@esbuild/linux-arm64@0.28.2': + optional: true + + '@esbuild/linux-arm@0.28.2': + optional: true + + '@esbuild/linux-ia32@0.28.2': + optional: true + + '@esbuild/linux-loong64@0.28.2': + optional: true + + '@esbuild/linux-mips64el@0.28.2': + optional: true + + '@esbuild/linux-ppc64@0.28.2': + optional: true + + '@esbuild/linux-riscv64@0.28.2': + optional: true + + '@esbuild/linux-s390x@0.28.2': + optional: true + + '@esbuild/linux-x64@0.28.2': + optional: true + + '@esbuild/netbsd-arm64@0.28.2': + optional: true + + '@esbuild/netbsd-x64@0.28.2': + optional: true + + '@esbuild/openbsd-arm64@0.28.2': + optional: true + + '@esbuild/openbsd-x64@0.28.2': + optional: true + + '@esbuild/openharmony-arm64@0.28.2': + optional: true + + '@esbuild/sunos-x64@0.28.2': + optional: true + + '@esbuild/win32-arm64@0.28.2': + optional: true + + '@esbuild/win32-ia32@0.28.2': + optional: true + + '@esbuild/win32-x64@0.28.2': + optional: true + '@eslint-community/eslint-utils@4.10.1(eslint@9.39.5(jiti@2.7.0))': dependencies: eslint: 9.39.5(jiti@2.7.0) @@ -4411,8 +5025,7 @@ snapshots: '@humanwhocodes/retry@0.4.3': {} - '@img/colour@1.1.0': - optional: true + '@img/colour@1.1.0': {} '@img/sharp-darwin-arm64@0.35.5': optionalDependencies: @@ -4615,6 +5228,10 @@ snapshots: '@oxc-project/types@0.152.0': {} + '@playwright/test@1.63.0': + dependencies: + playwright: 1.63.0 + '@radix-ui/number@1.1.3': {} '@radix-ui/primitive@1.1.7': {} @@ -5445,6 +6062,8 @@ snapshots: '@sindresorhus/merge-streams@4.0.0': {} + '@stablelib/base64@1.0.1': {} + '@swc/helpers@0.5.23': dependencies: tslib: 2.8.1 @@ -5534,14 +6153,32 @@ snapshots: '@types/deep-eql': 4.0.2 assertion-error: 2.0.1 + '@types/debug@4.1.13': + dependencies: + '@types/ms': 2.1.0 + '@types/deep-eql@4.0.2': {} + '@types/estree-jsx@1.0.5': + dependencies: + '@types/estree': 1.0.9 + '@types/estree@1.0.9': {} + '@types/hast@3.0.5': + dependencies: + '@types/unist': 3.0.3 + '@types/json-schema@7.0.15': {} '@types/json5@0.0.29': {} + '@types/mdast@4.0.4': + dependencies: + '@types/unist': 3.0.3 + + '@types/ms@2.1.0': {} + '@types/node@22.20.5': dependencies: undici-types: 6.21.0 @@ -5554,6 +6191,10 @@ snapshots: dependencies: csstype: 3.2.3 + '@types/unist@2.0.11': {} + + '@types/unist@3.0.3': {} + '@types/validate-npm-package-name@4.0.2': {} '@typescript-eslint/eslint-plugin@8.71.0(@typescript-eslint/parser@8.71.0(eslint@9.39.5(jiti@2.7.0))(typescript@5.9.3))(eslint@9.39.5(jiti@2.7.0))(typescript@5.9.3)': @@ -5647,6 +6288,8 @@ snapshots: '@typescript-eslint/types': 8.71.0 eslint-visitor-keys: 5.0.1 + '@ungap/structured-clone@1.4.0': {} + '@unrs/resolver-binding-android-arm-eabi@1.12.2': optional: true @@ -5717,14 +6360,14 @@ snapshots: '@unrs/resolver-binding-win32-x64-msvc@1.12.2': optional: true - '@vitest/mocker@5.0.3(vite@8.3.2(@types/node@22.20.5)(jiti@2.7.0))': + '@vitest/mocker@5.0.3(vite@8.3.2(@types/node@22.20.5)(esbuild@0.28.2)(jiti@2.7.0)(tsx@4.23.15))': dependencies: '@jridgewell/trace-mapping': 0.3.31 '@vitest/spy': 5.0.3 estree-walker: 3.0.3 magic-string: 1.4.3 optionalDependencies: - vite: 8.3.2(@types/node@22.20.5)(jiti@2.7.0) + vite: 8.3.2(@types/node@22.20.5)(esbuild@0.28.2)(jiti@2.7.0)(tsx@4.23.15) '@vitest/spy@5.0.3': {} @@ -5866,6 +6509,8 @@ snapshots: axobject-query@4.1.0: {} + bail@2.0.2: {} + balanced-match@1.0.2: {} balanced-match@4.0.4: {} @@ -5934,6 +6579,8 @@ snapshots: caniuse-lite@1.0.30001814: {} + ccount@2.0.1: {} + chai@6.3.0: {} chalk@4.1.2: @@ -5943,6 +6590,14 @@ snapshots: chalk@5.6.2: {} + character-entities-html4@2.1.0: {} + + character-entities-legacy@3.0.0: {} + + character-entities@2.0.2: {} + + character-reference-invalid@2.0.1: {} + class-variance-authority@0.7.1: dependencies: clsx: 2.1.1 @@ -5969,6 +6624,8 @@ snapshots: color-name@1.1.4: {} + comma-separated-tokens@2.0.3: {} + commander@11.1.0: {} commander@14.0.3: {} @@ -6056,6 +6713,10 @@ snapshots: dependencies: ms: 2.1.3 + decode-named-character-reference@1.3.0: + dependencies: + character-entities: 2.0.2 + dedent@1.7.2: {} deep-is@0.1.4: {} @@ -6087,10 +6748,16 @@ snapshots: depd@2.0.0: {} + dequal@2.0.3: {} + detect-libc@2.1.2: {} detect-node-es@1.1.0: {} + devlop@1.1.0: + dependencies: + dequal: 2.0.3 + diff@8.0.4: {} doctrine@2.1.0: @@ -6248,12 +6915,43 @@ snapshots: is-date-object: 1.1.0 is-symbol: 1.1.1 + esbuild@0.28.2: + optionalDependencies: + '@esbuild/aix-ppc64': 0.28.2 + '@esbuild/android-arm': 0.28.2 + '@esbuild/android-arm64': 0.28.2 + '@esbuild/android-x64': 0.28.2 + '@esbuild/darwin-arm64': 0.28.2 + '@esbuild/darwin-x64': 0.28.2 + '@esbuild/freebsd-arm64': 0.28.2 + '@esbuild/freebsd-x64': 0.28.2 + '@esbuild/linux-arm': 0.28.2 + '@esbuild/linux-arm64': 0.28.2 + '@esbuild/linux-ia32': 0.28.2 + '@esbuild/linux-loong64': 0.28.2 + '@esbuild/linux-mips64el': 0.28.2 + '@esbuild/linux-ppc64': 0.28.2 + '@esbuild/linux-riscv64': 0.28.2 + '@esbuild/linux-s390x': 0.28.2 + '@esbuild/linux-x64': 0.28.2 + '@esbuild/netbsd-arm64': 0.28.2 + '@esbuild/netbsd-x64': 0.28.2 + '@esbuild/openbsd-arm64': 0.28.2 + '@esbuild/openbsd-x64': 0.28.2 + '@esbuild/openharmony-arm64': 0.28.2 + '@esbuild/sunos-x64': 0.28.2 + '@esbuild/win32-arm64': 0.28.2 + '@esbuild/win32-ia32': 0.28.2 + '@esbuild/win32-x64': 0.28.2 + escalade@3.2.0: {} escape-html@1.0.3: {} escape-string-regexp@4.0.0: {} + escape-string-regexp@5.0.0: {} + eslint-config-next@16.3.8(@typescript-eslint/parser@8.71.0(eslint@9.39.5(jiti@2.7.0))(typescript@5.9.3))(eslint@9.39.5(jiti@2.7.0))(typescript@5.9.3): dependencies: '@next/eslint-plugin-next': 16.3.8(eslint@9.39.5(jiti@2.7.0)) @@ -6459,6 +7157,8 @@ snapshots: estraverse@5.3.0: {} + estree-util-is-identifier-name@3.0.0: {} + estree-walker@3.0.3: dependencies: '@types/estree': 1.0.9 @@ -6543,6 +7243,8 @@ snapshots: transitivePeerDependencies: - supports-color + extend@3.0.2: {} + fast-deep-equal@3.1.3: {} fast-glob@3.3.1: @@ -6565,6 +7267,8 @@ snapshots: fast-levenshtein@2.0.6: {} + fast-sha256@1.3.0: {} + fast-uri@3.1.8: {} fastq@1.20.3: @@ -6737,6 +7441,30 @@ snapshots: dependencies: function-bind: 1.1.2 + hast-util-to-jsx-runtime@2.3.6: + dependencies: + '@types/estree': 1.0.9 + '@types/hast': 3.0.5 + '@types/unist': 3.0.3 + comma-separated-tokens: 2.0.3 + devlop: 1.1.0 + estree-util-is-identifier-name: 3.0.0 + hast-util-whitespace: 3.0.0 + mdast-util-mdx-expression: 2.0.1 + mdast-util-mdx-jsx: 3.2.0 + mdast-util-mdxjs-esm: 2.0.1 + property-information: 7.2.0 + space-separated-tokens: 2.0.2 + style-to-js: 1.1.21 + unist-util-position: 5.0.0 + vfile-message: 4.0.3 + transitivePeerDependencies: + - supports-color + + hast-util-whitespace@3.0.0: + dependencies: + '@types/hast': 3.0.5 + hermes-estree@0.25.1: {} hermes-parser@0.25.1: @@ -6745,6 +7473,8 @@ snapshots: hono@4.13.13: {} + html-url-attributes@3.0.1: {} + http-errors@2.0.1: dependencies: depd: 2.0.0 @@ -6774,6 +7504,8 @@ snapshots: inherits@2.0.4: {} + inline-style-parser@0.2.7: {} + internal-slot@1.1.0: dependencies: es-errors: 1.3.0 @@ -6784,6 +7516,13 @@ snapshots: ipaddr.js@1.9.1: {} + is-alphabetical@2.0.1: {} + + is-alphanumerical@2.0.1: + dependencies: + is-alphabetical: 2.0.1 + is-decimal: 2.0.1 + is-array-buffer@3.0.5: dependencies: call-bind: 1.0.9 @@ -6830,6 +7569,8 @@ snapshots: call-bound: 1.0.4 has-tostringtag: 1.0.2 + is-decimal@2.0.1: {} + is-docker@2.2.1: {} is-docker@3.0.0: {} @@ -6856,6 +7597,8 @@ snapshots: dependencies: is-extglob: 2.1.1 + is-hexadecimal@2.0.1: {} + is-in-ssh@1.0.0: {} is-inside-container@1.0.0: @@ -6971,6 +7714,11 @@ snapshots: json-parse-even-better-errors@2.3.1: {} + json-schema-to-ts@3.1.1: + dependencies: + '@babel/runtime': 7.29.7 + ts-algebra: 2.0.0 + json-schema-traverse@0.4.1: {} json-schema-traverse@1.0.0: {} @@ -7135,6 +7883,8 @@ snapshots: chalk: 5.6.2 is-unicode-supported: 1.3.0 + longest-streak@3.1.0: {} + loose-envify@1.4.0: dependencies: js-tokens: 4.0.0 @@ -7157,8 +7907,165 @@ snapshots: dependencies: '@jridgewell/sourcemap-codec': 1.6.0 + markdown-table@3.0.4: {} + math-intrinsics@1.1.0: {} + mdast-util-find-and-replace@3.0.3: + dependencies: + '@types/mdast': 4.0.4 + escape-string-regexp: 5.0.0 + unist-util-is: 6.0.1 + unist-util-visit-parents: 6.0.2 + + mdast-util-from-markdown@2.1.0: + dependencies: + '@types/mdast': 4.0.4 + '@types/unist': 3.0.3 + decode-named-character-reference: 1.3.0 + devlop: 1.1.0 + mdast-util-to-string: 4.0.0 + micromark: 4.0.3 + micromark-util-decode-numeric-character-reference: 2.0.2 + micromark-util-decode-string: 2.0.1 + micromark-util-normalize-identifier: 2.0.1 + micromark-util-symbol: 2.0.1 + micromark-util-types: 2.0.3 + unist-util-stringify-position: 4.0.0 + transitivePeerDependencies: + - supports-color + + mdast-util-gfm-autolink-literal@2.0.1: + dependencies: + '@types/mdast': 4.0.4 + ccount: 2.0.1 + devlop: 1.1.0 + mdast-util-find-and-replace: 3.0.3 + micromark-util-character: 2.1.1 + + mdast-util-gfm-footnote@2.1.0: + dependencies: + '@types/mdast': 4.0.4 + devlop: 1.1.0 + mdast-util-from-markdown: 2.1.0 + mdast-util-to-markdown: 2.2.0 + micromark-util-normalize-identifier: 2.0.1 + transitivePeerDependencies: + - supports-color + + mdast-util-gfm-strikethrough@2.0.1: + dependencies: + '@types/mdast': 4.0.4 + mdast-util-from-markdown: 2.1.0 + mdast-util-to-markdown: 2.2.0 + transitivePeerDependencies: + - supports-color + + mdast-util-gfm-table@2.0.0: + dependencies: + '@types/mdast': 4.0.4 + devlop: 1.1.0 + markdown-table: 3.0.4 + mdast-util-from-markdown: 2.1.0 + mdast-util-to-markdown: 2.2.0 + transitivePeerDependencies: + - supports-color + + mdast-util-gfm-task-list-item@2.0.0: + dependencies: + '@types/mdast': 4.0.4 + devlop: 1.1.0 + mdast-util-from-markdown: 2.1.0 + mdast-util-to-markdown: 2.2.0 + transitivePeerDependencies: + - supports-color + + mdast-util-gfm@3.1.0: + dependencies: + mdast-util-from-markdown: 2.1.0 + mdast-util-gfm-autolink-literal: 2.0.1 + mdast-util-gfm-footnote: 2.1.0 + mdast-util-gfm-strikethrough: 2.0.1 + mdast-util-gfm-table: 2.0.0 + mdast-util-gfm-task-list-item: 2.0.0 + mdast-util-to-markdown: 2.2.0 + transitivePeerDependencies: + - supports-color + + mdast-util-mdx-expression@2.0.1: + dependencies: + '@types/estree-jsx': 1.0.5 + '@types/hast': 3.0.5 + '@types/mdast': 4.0.4 + devlop: 1.1.0 + mdast-util-from-markdown: 2.1.0 + mdast-util-to-markdown: 2.2.0 + transitivePeerDependencies: + - supports-color + + mdast-util-mdx-jsx@3.2.0: + dependencies: + '@types/estree-jsx': 1.0.5 + '@types/hast': 3.0.5 + '@types/mdast': 4.0.4 + '@types/unist': 3.0.3 + ccount: 2.0.1 + devlop: 1.1.0 + mdast-util-from-markdown: 2.1.0 + mdast-util-to-markdown: 2.2.0 + parse-entities: 4.0.2 + stringify-entities: 4.0.4 + unist-util-stringify-position: 4.0.0 + vfile-message: 4.0.3 + transitivePeerDependencies: + - supports-color + + mdast-util-mdxjs-esm@2.0.1: + dependencies: + '@types/estree-jsx': 1.0.5 + '@types/hast': 3.0.5 + '@types/mdast': 4.0.4 + devlop: 1.1.0 + mdast-util-from-markdown: 2.1.0 + mdast-util-to-markdown: 2.2.0 + transitivePeerDependencies: + - supports-color + + mdast-util-phrasing@4.1.0: + dependencies: + '@types/mdast': 4.0.4 + unist-util-is: 6.0.1 + + mdast-util-to-hast@13.2.1: + dependencies: + '@types/hast': 3.0.5 + '@types/mdast': 4.0.4 + '@ungap/structured-clone': 1.4.0 + devlop: 1.1.0 + micromark-util-sanitize-uri: 2.0.1 + trim-lines: 3.0.1 + unist-util-position: 5.0.0 + unist-util-visit: 5.1.0 + vfile: 6.0.3 + + mdast-util-to-markdown@2.2.0: + dependencies: + '@types/mdast': 4.0.4 + '@types/unist': 3.0.3 + longest-streak: 3.1.0 + mdast-util-phrasing: 4.1.0 + mdast-util-to-string: 4.0.0 + micromark-util-character: 2.1.1 + micromark-util-classify-character: 2.0.1 + micromark-util-decode-string: 2.0.1 + micromark-util-html-tag-name: 2.0.1 + unist-util-visit: 5.1.0 + zwitch: 2.0.4 + + mdast-util-to-string@4.0.0: + dependencies: + '@types/mdast': 4.0.4 + media-typer@1.1.1: {} merge-descriptors@2.0.0: {} @@ -7167,6 +8074,203 @@ snapshots: merge2@1.4.1: {} + micromark-core-commonmark@2.0.4: + dependencies: + decode-named-character-reference: 1.3.0 + devlop: 1.1.0 + micromark-factory-destination: 2.0.1 + micromark-factory-label: 2.0.1 + micromark-factory-space: 2.1.0 + micromark-factory-title: 2.0.1 + micromark-factory-whitespace: 2.0.1 + micromark-util-character: 2.1.1 + micromark-util-chunked: 2.0.1 + micromark-util-classify-character: 2.0.1 + micromark-util-edit-map: 1.0.0 + micromark-util-html-tag-name: 2.0.1 + micromark-util-normalize-identifier: 2.0.1 + micromark-util-resolve-all: 2.0.1 + micromark-util-subtokenize: 2.1.0 + micromark-util-symbol: 2.0.1 + micromark-util-types: 2.0.3 + + micromark-extension-gfm-autolink-literal@2.1.0: + dependencies: + micromark-util-character: 2.1.1 + micromark-util-sanitize-uri: 2.0.1 + micromark-util-symbol: 2.0.1 + micromark-util-types: 2.0.3 + + micromark-extension-gfm-footnote@2.1.0: + dependencies: + devlop: 1.1.0 + micromark-core-commonmark: 2.0.4 + micromark-factory-space: 2.1.0 + micromark-util-character: 2.1.1 + micromark-util-normalize-identifier: 2.0.1 + micromark-util-sanitize-uri: 2.0.1 + micromark-util-symbol: 2.0.1 + micromark-util-types: 2.0.3 + + micromark-extension-gfm-strikethrough@2.1.0: + dependencies: + devlop: 1.1.0 + micromark-util-chunked: 2.0.1 + micromark-util-classify-character: 2.0.1 + micromark-util-resolve-all: 2.0.1 + micromark-util-symbol: 2.0.1 + micromark-util-types: 2.0.3 + + micromark-extension-gfm-table@2.1.2: + dependencies: + devlop: 1.1.0 + micromark-factory-space: 2.1.0 + micromark-util-character: 2.1.1 + micromark-util-symbol: 2.0.1 + micromark-util-types: 2.0.3 + + micromark-extension-gfm-tagfilter@2.0.0: + dependencies: + micromark-util-types: 2.0.3 + + micromark-extension-gfm-task-list-item@2.1.0: + dependencies: + devlop: 1.1.0 + micromark-factory-space: 2.1.0 + micromark-util-character: 2.1.1 + micromark-util-symbol: 2.0.1 + micromark-util-types: 2.0.3 + + micromark-extension-gfm@3.0.0: + dependencies: + micromark-extension-gfm-autolink-literal: 2.1.0 + micromark-extension-gfm-footnote: 2.1.0 + micromark-extension-gfm-strikethrough: 2.1.0 + micromark-extension-gfm-table: 2.1.2 + micromark-extension-gfm-tagfilter: 2.0.0 + micromark-extension-gfm-task-list-item: 2.1.0 + micromark-util-combine-extensions: 2.0.1 + micromark-util-types: 2.0.3 + + micromark-factory-destination@2.0.1: + dependencies: + micromark-util-character: 2.1.1 + micromark-util-symbol: 2.0.1 + micromark-util-types: 2.0.3 + + micromark-factory-label@2.0.1: + dependencies: + devlop: 1.1.0 + micromark-util-character: 2.1.1 + micromark-util-symbol: 2.0.1 + micromark-util-types: 2.0.3 + + micromark-factory-space@2.1.0: + dependencies: + micromark-util-character: 2.1.1 + micromark-util-types: 2.0.3 + + micromark-factory-title@2.0.1: + dependencies: + micromark-factory-space: 2.1.0 + micromark-util-character: 2.1.1 + micromark-util-symbol: 2.0.1 + micromark-util-types: 2.0.3 + + micromark-factory-whitespace@2.0.1: + dependencies: + micromark-factory-space: 2.1.0 + micromark-util-character: 2.1.1 + micromark-util-symbol: 2.0.1 + micromark-util-types: 2.0.3 + + micromark-util-character@2.1.1: + dependencies: + micromark-util-symbol: 2.0.1 + micromark-util-types: 2.0.3 + + micromark-util-chunked@2.0.1: + dependencies: + micromark-util-symbol: 2.0.1 + + micromark-util-classify-character@2.0.1: + dependencies: + micromark-util-character: 2.1.1 + micromark-util-symbol: 2.0.1 + micromark-util-types: 2.0.3 + + micromark-util-combine-extensions@2.0.1: + dependencies: + micromark-util-chunked: 2.0.1 + micromark-util-types: 2.0.3 + + micromark-util-decode-numeric-character-reference@2.0.2: + dependencies: + micromark-util-symbol: 2.0.1 + + micromark-util-decode-string@2.0.1: + dependencies: + decode-named-character-reference: 1.3.0 + micromark-util-character: 2.1.1 + micromark-util-decode-numeric-character-reference: 2.0.2 + micromark-util-symbol: 2.0.1 + + micromark-util-edit-map@1.0.0: + dependencies: + micromark-util-types: 2.0.3 + + micromark-util-encode@2.0.1: {} + + micromark-util-html-tag-name@2.0.1: {} + + micromark-util-normalize-identifier@2.0.1: + dependencies: + micromark-util-symbol: 2.0.1 + + micromark-util-resolve-all@2.0.1: + dependencies: + micromark-util-types: 2.0.3 + + micromark-util-sanitize-uri@2.0.1: + dependencies: + micromark-util-character: 2.1.1 + micromark-util-encode: 2.0.1 + micromark-util-symbol: 2.0.1 + + micromark-util-subtokenize@2.1.0: + dependencies: + devlop: 1.1.0 + micromark-util-chunked: 2.0.1 + micromark-util-symbol: 2.0.1 + micromark-util-types: 2.0.3 + + micromark-util-symbol@2.0.1: {} + + micromark-util-types@2.0.3: {} + + micromark@4.0.3: + dependencies: + '@types/debug': 4.1.13 + debug: 4.4.3 + decode-named-character-reference: 1.3.0 + devlop: 1.1.0 + micromark-core-commonmark: 2.0.4 + micromark-factory-space: 2.1.0 + micromark-util-character: 2.1.1 + micromark-util-chunked: 2.0.1 + micromark-util-combine-extensions: 2.0.1 + micromark-util-decode-numeric-character-reference: 2.0.2 + micromark-util-edit-map: 1.0.0 + micromark-util-encode: 2.0.1 + micromark-util-normalize-identifier: 2.0.1 + micromark-util-resolve-all: 2.0.1 + micromark-util-sanitize-uri: 2.0.1 + micromark-util-subtokenize: 2.1.0 + micromark-util-symbol: 2.0.1 + micromark-util-types: 2.0.3 + transitivePeerDependencies: + - supports-color + micromatch@4.0.8: dependencies: braces: 3.0.3 @@ -7211,7 +8315,7 @@ snapshots: react: 19.2.8 react-dom: 19.2.8(react@19.2.8) - next@16.3.8(@babel/core@7.29.7)(@types/node@22.20.5)(react-dom@19.2.8(react@19.2.8))(react@19.2.8): + next@16.3.8(@babel/core@7.29.7)(@playwright/test@1.63.0)(@types/node@22.20.5)(react-dom@19.2.8(react@19.2.8))(react@19.2.8): dependencies: '@next/env': 16.3.8 '@swc/helpers': 0.5.23 @@ -7230,6 +8334,7 @@ snapshots: '@next/swc-linux-x64-musl': 16.3.8 '@next/swc-win32-arm64-msvc': 16.3.8 '@next/swc-win32-x64-msvc': 16.3.8 + '@playwright/test': 1.63.0 sharp: 0.35.5(@types/node@22.20.5) transitivePeerDependencies: - '@babel/core' @@ -7381,6 +8486,16 @@ snapshots: dependencies: callsites: 3.1.0 + parse-entities@4.0.2: + dependencies: + '@types/unist': 2.0.11 + character-entities-legacy: 3.0.0 + character-reference-invalid: 2.0.1 + decode-named-character-reference: 1.3.0 + is-alphanumerical: 2.0.1 + is-decimal: 2.0.1 + is-hexadecimal: 2.0.1 + parse-json@5.2.0: dependencies: '@babel/code-frame': 7.29.7 @@ -7418,6 +8533,12 @@ snapshots: dependencies: find-up: 3.0.0 + playwright-core@1.63.0: {} + + playwright@1.63.0: + dependencies: + playwright-core: 1.63.0 + possible-typed-array-names@1.1.0: {} postcss-selector-parser@7.1.6: @@ -7462,6 +8583,8 @@ snapshots: object-assign: 4.1.1 react-is: 16.13.1 + property-information@7.2.0: {} + proxy-addr@2.0.8: dependencies: forwarded: 0.2.0 @@ -7555,6 +8678,24 @@ snapshots: react-is@16.13.1: {} + react-markdown@10.1.0(@types/react@19.3.0)(react@19.2.8): + dependencies: + '@types/hast': 3.0.5 + '@types/mdast': 4.0.4 + '@types/react': 19.3.0 + devlop: 1.1.0 + hast-util-to-jsx-runtime: 2.3.6 + html-url-attributes: 3.0.1 + mdast-util-to-hast: 13.2.1 + react: 19.2.8 + remark-parse: 11.0.0 + remark-rehype: 11.1.2 + unified: 11.0.5 + unist-util-visit: 5.1.0 + vfile: 6.0.3 + transitivePeerDependencies: + - supports-color + react-remove-scroll-bar@2.3.8(@types/react@19.3.0)(react@19.2.8): dependencies: react: 19.2.8 @@ -7612,6 +8753,40 @@ snapshots: gopd: 1.2.0 set-function-name: 2.0.2 + remark-gfm@4.0.1: + dependencies: + '@types/mdast': 4.0.4 + mdast-util-gfm: 3.1.0 + micromark-extension-gfm: 3.0.0 + remark-parse: 11.0.0 + remark-stringify: 11.0.0 + unified: 11.0.5 + transitivePeerDependencies: + - supports-color + + remark-parse@11.0.0: + dependencies: + '@types/mdast': 4.0.4 + mdast-util-from-markdown: 2.1.0 + micromark-util-types: 2.0.3 + unified: 11.0.5 + transitivePeerDependencies: + - supports-color + + remark-rehype@11.1.2: + dependencies: + '@types/hast': 3.0.5 + '@types/mdast': 4.0.4 + mdast-util-to-hast: 13.2.1 + unified: 11.0.5 + vfile: 6.0.3 + + remark-stringify@11.0.0: + dependencies: + '@types/mdast': 4.0.4 + mdast-util-to-markdown: 2.2.0 + unified: 11.0.5 + require-from-string@2.0.2: {} resolve-from@4.0.0: {} @@ -7821,7 +8996,6 @@ snapshots: '@img/sharp-win32-ia32': 0.35.5 '@img/sharp-win32-x64': 0.35.5 '@types/node': 22.20.5 - optional: true shebang-command@2.0.0: dependencies: @@ -7874,8 +9048,15 @@ snapshots: source-map@0.6.1: {} + space-separated-tokens@2.0.2: {} + stable-hash@0.0.5: {} + standardwebhooks@1.1.1: + dependencies: + '@stablelib/base64': 1.0.1 + fast-sha256: 1.3.0 + statuses@2.0.2: {} std-env@4.3.0: {} @@ -7944,6 +9125,11 @@ snapshots: define-properties: 1.2.1 es-object-atoms: 1.1.2 + stringify-entities@4.0.4: + dependencies: + character-entities-html4: 2.1.0 + character-entities-legacy: 3.0.0 + stringify-object@5.0.0: dependencies: get-own-enumerable-keys: 1.0.0 @@ -7966,6 +9152,14 @@ snapshots: strip-json-comments@3.1.1: {} + style-to-js@1.1.21: + dependencies: + style-to-object: 1.0.14 + + style-to-object@1.0.14: + dependencies: + inline-style-parser: 0.2.7 + styled-jsx@5.1.6(@babel/core@7.29.7)(react@19.2.8): dependencies: client-only: 0.0.1 @@ -8002,6 +9196,12 @@ snapshots: toidentifier@1.0.1: {} + trim-lines@3.0.1: {} + + trough@2.2.0: {} + + ts-algebra@2.0.0: {} + ts-api-utils@2.5.0(typescript@5.9.3): dependencies: typescript: 5.9.3 @@ -8026,6 +9226,12 @@ snapshots: tslib@2.8.1: {} + tsx@4.23.15: + dependencies: + esbuild: 0.28.2 + optionalDependencies: + fsevents: 2.3.3 + tw-animate-css@1.4.0: {} type-check@0.4.0: @@ -8098,6 +9304,39 @@ snapshots: unicorn-magic@0.3.0: {} + unified@11.0.5: + dependencies: + '@types/unist': 3.0.3 + bail: 2.0.2 + devlop: 1.1.0 + extend: 3.0.2 + is-plain-obj: 4.1.0 + trough: 2.2.0 + vfile: 6.0.3 + + unist-util-is@6.0.1: + dependencies: + '@types/unist': 3.0.3 + + unist-util-position@5.0.0: + dependencies: + '@types/unist': 3.0.3 + + unist-util-stringify-position@4.0.0: + dependencies: + '@types/unist': 3.0.3 + + unist-util-visit-parents@6.0.2: + dependencies: + '@types/unist': 3.0.3 + unist-util-is: 6.0.1 + + unist-util-visit@5.1.0: + dependencies: + '@types/unist': 3.0.3 + unist-util-is: 6.0.1 + unist-util-visit-parents: 6.0.2 + universalify@2.0.1: {} unpipe@1.0.0: {} @@ -8160,7 +9399,17 @@ snapshots: vary@1.1.2: {} - vite@8.3.2(@types/node@22.20.5)(jiti@2.7.0): + vfile-message@4.0.3: + dependencies: + '@types/unist': 3.0.3 + unist-util-stringify-position: 4.0.0 + + vfile@6.0.3: + dependencies: + '@types/unist': 3.0.3 + vfile-message: 4.0.3 + + vite@8.3.2(@types/node@22.20.5)(esbuild@0.28.2)(jiti@2.7.0)(tsx@4.23.15): dependencies: lightningcss: 1.33.0 picomatch: 4.0.7 @@ -8169,13 +9418,15 @@ snapshots: tinyglobby: 0.2.17 optionalDependencies: '@types/node': 22.20.5 + esbuild: 0.28.2 fsevents: 2.3.3 jiti: 2.7.0 + tsx: 4.23.15 - vitest@5.0.3(@types/node@22.20.5)(vite@8.3.2(@types/node@22.20.5)(jiti@2.7.0)): + vitest@5.0.3(@types/node@22.20.5)(vite@8.3.2(@types/node@22.20.5)(esbuild@0.28.2)(jiti@2.7.0)(tsx@4.23.15)): dependencies: '@types/chai': 5.2.3 - '@vitest/mocker': 5.0.3(vite@8.3.2(@types/node@22.20.5)(jiti@2.7.0)) + '@vitest/mocker': 5.0.3(vite@8.3.2(@types/node@22.20.5)(esbuild@0.28.2)(jiti@2.7.0)(tsx@4.23.15)) chai: 6.3.0 es-module-lexer: 2.3.2 expect-type: 1.4.0 @@ -8186,7 +9437,7 @@ snapshots: tinybench: 6.2.0 tinyexec: 1.3.1 tinyglobby: 0.2.17 - vite: 8.3.2(@types/node@22.20.5)(jiti@2.7.0) + vite: 8.3.2(@types/node@22.20.5)(esbuild@0.28.2)(jiti@2.7.0)(tsx@4.23.15) why-is-node-running: 3.2.1 optionalDependencies: '@types/node': 22.20.5 @@ -8274,3 +9525,5 @@ snapshots: zod@3.25.76: {} zod@4.6.5: {} + + zwitch@2.0.4: {} diff --git a/web/public/showcase/astar-lab-poster.webp b/web/public/showcase/astar-lab-poster.webp new file mode 100644 index 0000000..77dfc9f Binary files /dev/null and b/web/public/showcase/astar-lab-poster.webp differ diff --git a/web/public/showcase/astar-lab.mp4 b/web/public/showcase/astar-lab.mp4 new file mode 100644 index 0000000..9272257 Binary files /dev/null and b/web/public/showcase/astar-lab.mp4 differ diff --git a/web/public/showcase/astar-lab.vtt b/web/public/showcase/astar-lab.vtt new file mode 100644 index 0000000..cc8ba6e --- /dev/null +++ b/web/public/showcase/astar-lab.vtt @@ -0,0 +1,35 @@ +WEBVTT + +NOTE A* Lab: captions match the on-screen steps. + +1 +00:00:00.200 --> 00:00:04.824 +Step 1 of 8. Open the A* Lab: the Part A search, ported line by line from Python. + +2 +00:00:04.824 --> 00:00:11.873 +Step 2 of 8. Load the original sample input (code/sample_input.json). + +3 +00:00:11.873 --> 00:00:20.294 +Step 3 of 8. Manhattan, as in the recorded output: animate the expansions. + +4 +00:00:20.294 --> 00:00:26.020 +Step 4 of 8. An 8-cell path that matches sample_output.txt from the original repo. + +5 +00:00:26.020 --> 00:00:35.897 +Step 5 of 8. Switch to Euclidean and animate the same board. + +6 +00:00:35.897 --> 00:00:40.335 +Step 6 of 8. Same board, both heuristics: path cost, nodes expanded, queue pushes. + +7 +00:00:40.335 --> 00:00:47.138 +Step 7 of 8. Paired study on 980 boards: paired bootstrap CI and Wilcoxon test on expansions. + +8 +00:00:47.138 --> 00:00:52.680 +Step 8 of 8. Optimality against breadth-first search: paired difference and exact McNemar test. diff --git a/web/public/showcase/play-the-agent-poster.webp b/web/public/showcase/play-the-agent-poster.webp new file mode 100644 index 0000000..32f3a99 Binary files /dev/null and b/web/public/showcase/play-the-agent-poster.webp differ diff --git a/web/public/showcase/play-the-agent.mp4 b/web/public/showcase/play-the-agent.mp4 new file mode 100644 index 0000000..80680f1 Binary files /dev/null and b/web/public/showcase/play-the-agent.mp4 differ diff --git a/web/public/showcase/play-the-agent.vtt b/web/public/showcase/play-the-agent.vtt new file mode 100644 index 0000000..de3a68f --- /dev/null +++ b/web/public/showcase/play-the-agent.vtt @@ -0,0 +1,39 @@ +WEBVTT + +NOTE Play the agent: captions match the on-screen steps. + +1 +00:00:00.200 --> 00:00:05.147 +Step 1 of 9. Open Play: a 5 × 5 board against the minimax agent (seed 4399), coordinates on. + +2 +00:00:05.147 --> 00:00:07.595 +Step 2 of 9. Red opens on the strong cell (1, 1). + +3 +00:00:07.595 --> 00:00:11.784 +Step 3 of 9. Blue steals: Red's opening tile is mirrored across the diagonal and becomes Blue's. + +4 +00:00:11.784 --> 00:00:15.217 +Step 4 of 9. Red plays (0, 1); the agent answers at (4, 1). + +5 +00:00:15.217 --> 00:00:18.251 +Step 5 of 9. Red plays (1, 0), leaving two red tiles inside a diamond. + +6 +00:00:18.251 --> 00:00:21.066 +Step 6 of 9. Capture: Blue closes the diamond at (0, 0) and removes both red tiles. + +7 +00:00:21.066 --> 00:00:24.310 +Step 7 of 9. Red retakes (0, 1); the agent plays (1, 4). + +8 +00:00:24.310 --> 00:00:28.797 +Step 8 of 9. Recapture: Red plays (1, 0) and removes Blue's (0, 0) and (1, 1). + +9 +00:00:28.797 --> 00:00:39.120 +Step 9 of 9. Why that move? Pick the capture in the move log: search depth, candidate scores, features. diff --git a/web/public/showcase/screens/01-landing-light.webp b/web/public/showcase/screens/01-landing-light.webp new file mode 100644 index 0000000..00c9fa8 Binary files /dev/null and b/web/public/showcase/screens/01-landing-light.webp differ diff --git a/web/public/showcase/screens/02-landing-dark.webp b/web/public/showcase/screens/02-landing-dark.webp new file mode 100644 index 0000000..a83cf43 Binary files /dev/null and b/web/public/showcase/screens/02-landing-dark.webp differ diff --git a/web/public/showcase/screens/03-play.webp b/web/public/showcase/screens/03-play.webp new file mode 100644 index 0000000..5f1803b Binary files /dev/null and b/web/public/showcase/screens/03-play.webp differ diff --git a/web/public/showcase/screens/04-spectate.webp b/web/public/showcase/screens/04-spectate.webp new file mode 100644 index 0000000..1dcb2dc Binary files /dev/null and b/web/public/showcase/screens/04-spectate.webp differ diff --git a/web/public/showcase/screens/05-astar-lab.webp b/web/public/showcase/screens/05-astar-lab.webp new file mode 100644 index 0000000..45e2d38 Binary files /dev/null and b/web/public/showcase/screens/05-astar-lab.webp differ diff --git a/web/public/showcase/screens/06-astar-paired-study.webp b/web/public/showcase/screens/06-astar-paired-study.webp new file mode 100644 index 0000000..5415942 Binary files /dev/null and b/web/public/showcase/screens/06-astar-paired-study.webp differ diff --git a/web/public/showcase/screens/07-tournament.webp b/web/public/showcase/screens/07-tournament.webp new file mode 100644 index 0000000..8472a3f Binary files /dev/null and b/web/public/showcase/screens/07-tournament.webp differ diff --git a/web/public/showcase/screens/08-ai-settings.webp b/web/public/showcase/screens/08-ai-settings.webp new file mode 100644 index 0000000..290c897 Binary files /dev/null and b/web/public/showcase/screens/08-ai-settings.webp differ diff --git a/web/public/showcase/screens/09-llm-arena.webp b/web/public/showcase/screens/09-llm-arena.webp new file mode 100644 index 0000000..2b90c49 Binary files /dev/null and b/web/public/showcase/screens/09-llm-arena.webp differ diff --git a/web/public/showcase/screens/10-methods.webp b/web/public/showcase/screens/10-methods.webp new file mode 100644 index 0000000..48b985f Binary files /dev/null and b/web/public/showcase/screens/10-methods.webp differ diff --git a/web/public/showcase/screens/11-mobile-landing.webp b/web/public/showcase/screens/11-mobile-landing.webp new file mode 100644 index 0000000..8e0a267 Binary files /dev/null and b/web/public/showcase/screens/11-mobile-landing.webp differ diff --git a/web/public/showcase/screens/12-mobile-play.webp b/web/public/showcase/screens/12-mobile-play.webp new file mode 100644 index 0000000..1f41264 Binary files /dev/null and b/web/public/showcase/screens/12-mobile-play.webp differ diff --git a/web/public/showcase/screens/13-mobile-astar.webp b/web/public/showcase/screens/13-mobile-astar.webp new file mode 100644 index 0000000..2b19f4a Binary files /dev/null and b/web/public/showcase/screens/13-mobile-astar.webp differ diff --git a/web/public/showcase/tournament-llm-eval-poster.webp b/web/public/showcase/tournament-llm-eval-poster.webp new file mode 100644 index 0000000..b738147 Binary files /dev/null and b/web/public/showcase/tournament-llm-eval-poster.webp differ diff --git a/web/public/showcase/tournament-llm-eval.mp4 b/web/public/showcase/tournament-llm-eval.mp4 new file mode 100644 index 0000000..b9d7cab Binary files /dev/null and b/web/public/showcase/tournament-llm-eval.mp4 differ diff --git a/web/public/showcase/tournament-llm-eval.vtt b/web/public/showcase/tournament-llm-eval.vtt new file mode 100644 index 0000000..a835de1 --- /dev/null +++ b/web/public/showcase/tournament-llm-eval.vtt @@ -0,0 +1,47 @@ +WEBVTT + +NOTE Tournament and LLM evaluation: captions match the on-screen steps. + +1 +00:00:00.200 --> 00:00:04.141 +Step 1 of 11. Open the tournament harness: a seeded, colour-swapped round robin. + +2 +00:00:04.141 --> 00:00:09.337 +Step 2 of 11. Keep 4 agents and seed 4399; 4 × 4 only, 2 colour-swapped pairs: 24 games. + +3 +00:00:09.337 --> 00:00:11.204 +Step 3 of 11. Run all 24 games in parallel Web Workers. + +4 +00:00:11.204 --> 00:00:19.985 +Step 4 of 11. Win rates with Wilson 95% CIs, Bradley-Terry strengths with bootstrap CIs. + +5 +00:00:19.985 --> 00:00:30.571 +Step 5 of 11. AI settings: bring your own key, kept in this browser and sent only to the provider. + +6 +00:00:30.571 --> 00:00:39.589 +Step 6 of 11. For this demo: a placeholder key and the model id “mock-for-illustration”, no real key. (Mocked AI response for illustration.) + +7 +00:00:39.589 --> 00:00:45.820 +Step 7 of 11. LLM Arena: 2 games on 4 × 4 against the minimax agent, on the baselines' seeds. (Mocked AI response for illustration.) + +8 +00:00:45.820 --> 00:00:50.163 +Step 8 of 11. Every reply is checked against the legal moves and labelled AI-generated. (Mocked AI response for illustration.) + +9 +00:00:50.163 --> 00:00:57.700 +Step 9 of 11. Side by side with random, greedy and scripted baselines, Wilson 95% CIs. (Mocked AI response for illustration.) + +10 +00:00:57.700 --> 00:01:03.184 +Step 10 of 11. Every call is in the AI audit log, with JSON and CSV export. (Mocked AI response for illustration.) + +11 +00:01:03.184 --> 00:01:09.734 +Step 11 of 11. Forget key: the placeholder is removed from this browser. diff --git a/web/scripts/generate-reference-studies.ts b/web/scripts/generate-reference-studies.ts new file mode 100644 index 0000000..c347ccd --- /dev/null +++ b/web/scripts/generate-reference-studies.ts @@ -0,0 +1,109 @@ +/** + * Precompute the reference studies shown on /tournament and /astar, using the + * parity-tested TypeScript port (the original Python is too slow for depth-3 + * search at this scale; scripts/crosscheck_tournament.py re-runs the feasible + * subset with the original code). + * + * cd web && pnpm gen:reference # all three studies + * cd web && pnpm gen:reference tournament # just one + * + * Output (deterministic for the seeds below, apart from wall-clock timings): + * src/lib/data/reference-tournament.json + * src/lib/data/astar-paired-study.json + * src/lib/data/alpha-beta-study.json + */ +import { writeFileSync } from "node:fs"; +import os from "node:os"; +import path from "node:path"; + +import { measurePruning, samplePositions, summarisePruning } from "../src/lib/analysis/alpha-beta"; +import { runPairedStudy, summarisePairedStudy } from "../src/lib/analysis/astar-paired"; +import { AGENT_IDS } from "../src/lib/tournament/agents"; +import { summariseTournament } from "../src/lib/tournament/analyse"; +import { playTournamentGame } from "../src/lib/tournament/play"; +import { buildSchedule } from "../src/lib/tournament/schedule"; +import { + ALPHA_BETA_CONFIG, + ASTAR_STUDY_CONFIG, + REFERENCE_TOURNAMENT_CONFIG, +} from "../src/lib/data/reference-config"; + +const DATA = path.join(import.meta.dirname, "..", "src", "lib", "data"); +const only = process.argv[2]; + +const meta = () => ({ + generatedBy: "web/scripts/generate-reference-studies.ts", + generatedAt: new Date().toISOString().slice(0, 10), + node: process.version, + cpu: os.cpus()[0]?.model ?? "unknown", + implementation: "TypeScript port (parity-tested against the original Python)", +}); + +function write(name: string, payload: unknown) { + const file = path.join(DATA, name); + writeFileSync(file, `${JSON.stringify(payload)}\n`); + console.log(`wrote ${path.relative(process.cwd(), file)}`); +} + +if (!only || only === "tournament") { + const config = REFERENCE_TOURNAMENT_CONFIG; + const schedule = buildSchedule({ ...config, agents: [...AGENT_IDS] }); + console.log(`tournament: ${schedule.length} games`); + const started = Date.now(); + const games = schedule.map((spec, i) => { + const g = playTournamentGame(spec); + if ((i + 1) % 100 === 0) + console.log( + ` ${i + 1}/${schedule.length} (${((Date.now() - started) / 1000).toFixed(0)} s)`, + ); + return { + ...g, + red: { + ...g.red, + totalMs: Number(g.red.totalMs.toFixed(3)), + maxMs: Number(g.red.maxMs.toFixed(3)), + }, + blue: { + ...g.blue, + totalMs: Number(g.blue.totalMs.toFixed(3)), + maxMs: Number(g.blue.maxMs.toFixed(3)), + }, + }; + }); + write("reference-tournament.json", { meta: { ...meta(), config }, games }); + const s = summariseTournament(games, { reps: 1000, seed: config.seed }); + for (const a of s.standings) { + console.log(` deep searches ${a.id}: ${(a.deepSearchShare * 100).toFixed(1)}%`); + console.log( + `${a.id.padEnd(16)} elo ${a.elo.estimate.toFixed(0)} [${a.elo.lower.toFixed(0)}, ${a.elo.upper.toFixed(0)}] win ${(a.winRate.p * 100).toFixed(1)}% [${(a.winRate.lower * 100).toFixed(1)}, ${(a.winRate.upper * 100).toFixed(1)}] ${a.moveMs.estimate.toFixed(2)} ms/move`, + ); + } + for (const p of s.pairs) { + console.log( + `${p.a} vs ${p.b}: ${p.aWins}-${p.bWins}-${p.draws} (${(p.aWinRate.p * 100).toFixed(1)}% [${(p.aWinRate.lower * 100).toFixed(1)}, ${(p.aWinRate.upper * 100).toFixed(1)}])`, + ); + } + console.log(`red wins ${s.colour.redWins}/${s.colour.games}`); +} + +if (!only || only === "astar") { + const rows = runPairedStudy(ASTAR_STUDY_CONFIG); + write("astar-paired-study.json", { meta: { ...meta(), config: ASTAR_STUDY_CONFIG }, rows }); + const s = summarisePairedStudy(rows, { seed: ASTAR_STUDY_CONFIG.seed }); + console.log(JSON.stringify(s, null, 2)); +} + +if (!only || only === "alphabeta") { + const positions = samplePositions(ALPHA_BETA_CONFIG); + const samples = positions.flatMap((p) => + ALPHA_BETA_CONFIG.depths.flatMap((depth) => + (["shuffled", "canonical"] as const).map((order) => measurePruning(p, depth, order)), + ), + ); + write("alpha-beta-study.json", { meta: { ...meta(), config: ALPHA_BETA_CONFIG }, samples }); + for (const g of summarisePruning(samples, { seed: ALPHA_BETA_CONFIG.seed })) { + console.log( + `n=${g.n} d=${g.depth} ${g.order}: ratio ${g.meanRatio.estimate.toFixed(3)} [${g.meanRatio.lower.toFixed(3)}, ${g.meanRatio.upper.toFixed(3)}] full ${g.meanFull.toFixed(0)} pruned ${g.meanPruned.toFixed(0)} match ${g.valueMatches}/${g.positions} | textbook ${g.textbookRatio.estimate.toFixed(3)} [${g.textbookRatio.lower.toFixed(3)}, ${g.textbookRatio.upper.toFixed(3)}] nodes ${g.meanTextbook.toFixed(0)} match ${g.textbookValueMatches}`, + ); + } +} diff --git a/web/scripts/showcase-media.mjs b/web/scripts/showcase-media.mjs new file mode 100644 index 0000000..a6e178f --- /dev/null +++ b/web/scripts/showcase-media.mjs @@ -0,0 +1,260 @@ +/** + * Turn the raw tour output in web/.showcase/ into the committed media: + * + * docs/showcase/NN-*.png screenshots for the README (optimised, < 600 KB each) + * docs/showcase/.gif README GIFs (960 px, 10 fps, waits and scrolls cut, < 8 MB) + * web/public/showcase/.mp4 H.264, CRF 28, faststart (< 8 MB) + * web/public/showcase/.vtt captions from the on-screen steps + * web/public/showcase/-poster.webp + * web/public/showcase/screens/NN-*.webp screenshot copies for the /tour lightbox + * + * Images use sharp (no pngquant needed); video uses the ffmpeg on PATH. + * node scripts/showcase-media.mjs (or: pnpm showcase --media-only) + */ +import { spawnSync } from "node:child_process"; +import { + existsSync, + mkdirSync, + readFileSync, + readdirSync, + rmSync, + statSync, + writeFileSync, +} from "node:fs"; +import path from "node:path"; + +import sharp from "sharp"; + +const web = path.resolve(import.meta.dirname, ".."); +const raw = path.join(web, ".showcase"); +const docsOut = path.resolve(web, "..", "docs", "showcase"); +const publicOut = path.join(web, "public", "showcase"); +const screensOut = path.join(publicOut, "screens"); +const tmp = path.join(raw, "tmp"); + +const KB = 1024; +const MB = 1024 * KB; +const PNG_LIMIT = 600 * KB; +const VIDEO_LIMIT = 8 * MB; + +/** Poster frame: this step's caption, plus a delay (seconds) for the UI to settle. */ +const POSTER = { + "play-the-agent": { step: 9, after: 3.2 }, + "astar-lab": { step: 4, after: 1.6 }, + "tournament-llm-eval": { step: 4, after: 1.8 }, +}; + +for (const dir of [docsOut, publicOut, screensOut, tmp]) mkdirSync(dir, { recursive: true }); + +const size = (file) => statSync(file).size; +const fmt = (bytes) => + bytes >= MB ? `${(bytes / MB).toFixed(2)} MB` : `${Math.round(bytes / KB)} KB`; +const report = []; + +function ffmpeg(args) { + const res = spawnSync("ffmpeg", ["-hide_banner", "-loglevel", "error", "-y", ...args], { + stdio: "inherit", + }); + if (res.status !== 0) throw new Error(`ffmpeg failed: ffmpeg ${args.join(" ")}`); +} + +function probeDuration(file) { + const res = spawnSync( + "ffprobe", + ["-v", "error", "-show_entries", "format=duration", "-of", "default=nw=1:nk=1", file], + { encoding: "utf8" }, + ); + return Number.parseFloat(res.stdout.trim()); +} + +// ---------------------------------------------------------------- screenshots + +async function screenshots() { + const dir = path.join(raw, "screens"); + if (!existsSync(dir)) return; + for (const file of readdirSync(dir) + .filter((f) => f.endsWith(".png")) + .sort()) { + const id = path.basename(file, ".png"); + const src = path.join(dir, file); + const meta = await sharp(src).metadata(); + const isMobile = id.includes("mobile"); + // Mobile shots are taken at 2x (780 × 1688) and downscaled to 1.5x. + const width = isMobile ? Math.round((meta.width * 3) / 4) : meta.width; + const base = () => sharp(src).resize({ width, kernel: "lanczos3" }); + + const png = path.join(docsOut, `${id}.png`); + await base().png({ compressionLevel: 9, effort: 10, adaptiveFiltering: true }).toFile(png); + if (size(png) > PNG_LIMIT) { + // Palette PNG (libimagequant, bundled with sharp): UI screenshots survive 256 colours well. + await base() + .png({ palette: true, quality: 92, colours: 256, effort: 10, dither: 0.6 }) + .toFile(png); + } + if (size(png) > PNG_LIMIT) { + await base() + .png({ palette: true, quality: 80, colours: 160, effort: 10, dither: 0.8 }) + .toFile(png); + } + const webp = path.join(screensOut, `${id}.webp`); + await base().webp({ quality: 82, effort: 6, smartSubsample: true }).toFile(webp); + report.push([path.relative(path.resolve(web, ".."), png), size(png)]); + report.push([path.relative(path.resolve(web, ".."), webp), size(webp)]); + } +} + +// --------------------------------------------------------------------- videos + +const vttTime = (s) => { + const ms = Math.max(0, Math.round(s * 1000)); + const h = String(Math.floor(ms / 3_600_000)).padStart(2, "0"); + const m = String(Math.floor((ms % 3_600_000) / 60_000)).padStart(2, "0"); + const sec = String(Math.floor((ms % 60_000) / 1000)).padStart(2, "0"); + return `${h}:${m}:${sec}.${String(ms % 1000).padStart(3, "0")}`; +}; + +function writeVtt(id, info, duration) { + const lines = ["WEBVTT", "", `NOTE ${info.title}: captions match the on-screen steps.`, ""]; + info.cues.forEach((c, i) => { + const start = Math.max(0, c.start - info.trimStart); + const end = Math.min(duration, c.end - info.trimStart); + if (end <= start) return; + lines.push(String(i + 1)); + lines.push(`${vttTime(start)} --> ${vttTime(end)}`); + lines.push( + `Step ${c.step} of ${info.cues.length}. ${c.text}.${c.mock ? " (Mocked AI response for illustration.)" : ""}`, + ); + lines.push(""); + }); + const file = path.join(publicOut, `${id}.vtt`); + writeFileSync(file, lines.join("\n")); + report.push([path.relative(path.resolve(web, ".."), file), size(file)]); +} + +function encodeMp4(src, out, trimStart) { + for (const crf of [28, 30, 32, 34]) { + ffmpeg([ + "-ss", + trimStart.toFixed(2), + "-i", + src, + "-an", + "-c:v", + "libx264", + "-preset", + "slow", + "-crf", + String(crf), + "-pix_fmt", + "yuv420p", + "-movflags", + "+faststart", + "-r", + "25", + out, + ]); + if (size(out) <= VIDEO_LIMIT) return crf; + } + throw new Error(`${out} is larger than 8 MB even at CRF 34`); +} + +/** + * GIF for the README: 960 px wide, 10 fps, own palette. To stay under 8 MB + * without shrinking it, the GIF is edited rather than squeezed: waits on the + * app are cut, scripted scrolls become cuts (the video slides, the GIF jumps), + * frames that change almost the whole screen at once (page loads, dialogs + * fading in) are dropped in favour of the settled frame, and it plays a + * little faster than real time. Each attempt below trades a bit more speed + * or colour for size; the width never changes. + */ +function encodeGif(src, out, info, duration) { + const rel = ([a, b], padStart, padEnd) => [ + a - info.trimStart + padStart, + b - info.trimStart + padEnd, + ]; + const cuts = [ + ...info.idle.map((iv) => rel(iv, 0.7, -0.5)).filter(([a, b]) => b - a > 0.6), + ...(info.motion ?? []).map((iv) => rel(iv, 0.04, 0.08)).filter(([a, b]) => b - a > 0.1), + ].filter(([a]) => a < duration); + const select = cuts.length + ? `select='not(${cuts.map(([a, b]) => `between(t,${a.toFixed(2)},${b.toFixed(2)})`).join("+")})',setpts=N/FRAME_RATE/TB,` + : ""; + const attempts = [ + { fps: 10, speed: 1.25, colours: 160 }, + { fps: 10, speed: 1.25, colours: 128 }, + { fps: 10, speed: 1.4, colours: 128 }, + { fps: 10, speed: 1.4, colours: 96 }, + { fps: 10, speed: 1.5, colours: 96 }, + { fps: 10, speed: 1.6, colours: 96 }, + { fps: 10, speed: 1.6, colours: 64 }, + ]; + for (const a of attempts) { + const filter = + `${select}setpts=PTS/${a.speed},fps=${a.fps},scale=960:-2:flags=lanczos,` + + `select='lt(scene,0.04)',setpts=N/${a.fps}/TB,split[s0][s1];` + + `[s0]palettegen=max_colors=${a.colours}:stats_mode=diff[p];` + + `[s1][p]paletteuse=dither=bayer:bayer_scale=5:diff_mode=rectangle`; + ffmpeg([ + "-ss", + info.trimStart.toFixed(2), + "-i", + src, + "-filter_complex", + filter, + "-loop", + "0", + out, + ]); + if (size(out) <= VIDEO_LIMIT) return { ...a, seconds: probeDuration(out) }; + } + throw new Error(`${out} is larger than 8 MB`); +} + +async function poster(id, mp4, info) { + const p = POSTER[id]; + const cue = info.cues.find((c) => c.step === p.step) ?? info.cues.at(-1); + const at = Math.max(0, cue.start - info.trimStart + p.after); + const frame = path.join(tmp, `${id}-poster.png`); + ffmpeg(["-ss", at.toFixed(2), "-i", mp4, "-frames:v", "1", frame]); + const out = path.join(publicOut, `${id}-poster.webp`); + await sharp(frame).webp({ quality: 80, effort: 6 }).toFile(out); + report.push([path.relative(path.resolve(web, ".."), out), size(out)]); +} + +async function videos() { + const dir = path.join(raw, "video"); + if (!existsSync(dir)) return; + for (const file of readdirSync(dir) + .filter((f) => f.endsWith(".webm")) + .sort()) { + const id = path.basename(file, ".webm"); + const src = path.join(dir, file); + const cueFile = path.join(dir, `${id}.json`); + if (!existsSync(cueFile)) throw new Error(`Missing ${cueFile}`); + const info = JSON.parse(readFileSync(cueFile, "utf8")); + + const mp4 = path.join(publicOut, `${id}.mp4`); + const crf = encodeMp4(src, mp4, info.trimStart); + const duration = probeDuration(mp4); + report.push([ + `${path.relative(path.resolve(web, ".."), mp4)} (${duration.toFixed(1)} s, CRF ${crf})`, + size(mp4), + ]); + writeVtt(id, info, duration); + await poster(id, mp4, info); + + const gif = path.join(docsOut, `${id}.gif`); + const g = encodeGif(src, gif, info, duration); + report.push([ + `${path.relative(path.resolve(web, ".."), gif)} (${g.seconds.toFixed(1)} s, ${g.fps} fps, ×${g.speed}, ${g.colours} colours)`, + size(gif), + ]); + } +} + +await screenshots(); +await videos(); +rmSync(tmp, { recursive: true, force: true }); + +console.log("\nShowcase media:"); +for (const [name, bytes] of report) console.log(` ${fmt(bytes).padStart(9)} ${name}`); diff --git a/web/scripts/showcase.mjs b/web/scripts/showcase.mjs new file mode 100644 index 0000000..a90bca8 --- /dev/null +++ b/web/scripts/showcase.mjs @@ -0,0 +1,87 @@ +/** + * pnpm showcase: run the guided tour and rebuild the showcase media. + * + * pnpm showcase # against production + * BASE_URL=http://localhost:3000 pnpm showcase # after `pnpm build` (the server is started for you) + * pnpm showcase --test-only # Playwright only (screenshots + raw video) + * pnpm showcase --media-only # post-process the last run only + * pnpm showcase -g "A\* Lab" # other arguments go to `playwright test` + * + * 1. Playwright records video by piping frames to ffmpeg. Instead of + * downloading Playwright's own ffmpeg build, this links the ffmpeg already + * on PATH into a project-local PLAYWRIGHT_BROWSERS_PATH (web/.playwright, + * git-ignored). No browser is downloaded either: the tour runs on the + * system Google Chrome (channel "chrome"). + * 2. `playwright test` runs e2e/showcase.spec.ts, writing raw screenshots, + * webm recordings and caption timings to web/.showcase/. + * 3. scripts/showcase-media.mjs turns those into the committed media. + */ +import { spawnSync } from "node:child_process"; +import { existsSync, mkdirSync, readFileSync, rmSync, symlinkSync } from "node:fs"; +import { createRequire } from "node:module"; +import path from "node:path"; + +const web = path.resolve(import.meta.dirname, ".."); +const args = process.argv.slice(2); +const testOnly = args.includes("--test-only"); +const mediaOnly = args.includes("--media-only"); +const passThrough = args.filter((a) => a !== "--test-only" && a !== "--media-only"); + +function which(cmd) { + const res = spawnSync(process.platform === "win32" ? "where" : "which", [cmd], { + encoding: "utf8", + }); + return res.status === 0 ? res.stdout.split(/\r?\n/)[0].trim() : null; +} + +/** Link the system ffmpeg where Playwright looks for its own build. */ +function ffmpegShim() { + const ffmpeg = which("ffmpeg"); + if (!ffmpeg) { + console.error( + "ffmpeg is not on PATH; install it (e.g. `brew install ffmpeg`) to record video.", + ); + process.exit(1); + } + const req = createRequire(path.join(web, "package.json")); + const pwTest = req.resolve("@playwright/test/package.json"); + const pw = createRequire(pwTest).resolve("playwright/package.json"); + const core = path.dirname(createRequire(pw).resolve("playwright-core/package.json")); + const browsers = JSON.parse(readFileSync(path.join(core, "browsers.json"), "utf8")); + const { revision } = browsers.browsers.find((b) => b.name === "ffmpeg"); + const exe = + process.platform === "darwin" + ? "ffmpeg-mac" + : process.platform === "win32" + ? "ffmpeg-win64.exe" + : "ffmpeg-linux"; + const root = path.join(web, ".playwright"); + const dir = path.join(root, `ffmpeg-${revision}`); + const target = path.join(dir, exe); + if (!existsSync(target)) { + rmSync(dir, { recursive: true, force: true }); + mkdirSync(dir, { recursive: true }); + symlinkSync(ffmpeg, target); + } + return root; +} + +function run(cmd, cmdArgs, env = {}) { + const res = spawnSync(cmd, cmdArgs, { + cwd: web, + stdio: "inherit", + env: { ...process.env, ...env }, + shell: process.platform === "win32", + }); + if (res.status !== 0) process.exit(res.status ?? 1); +} + +if (!mediaOnly) { + const browsersPath = ffmpegShim(); + console.log(`Tour against ${process.env.BASE_URL ?? "https://cachex-ai.vercel.app"}`); + run("pnpm", ["exec", "playwright", "test", ...passThrough], { + PLAYWRIGHT_BROWSERS_PATH: browsersPath, + PLAYWRIGHT_SKIP_BROWSER_DOWNLOAD: "1", + }); +} +if (!testOnly) run("node", [path.join("scripts", "showcase-media.mjs")]); diff --git a/web/scripts/sync-docs.mjs b/web/scripts/sync-docs.mjs new file mode 100644 index 0000000..e26f959 --- /dev/null +++ b/web/scripts/sync-docs.mjs @@ -0,0 +1,23 @@ +/** + * Copy the rendered documentation from the repository's docs/ folder into + * web/content/docs/, because the Vercel build only sees web/. The test + * suite (src/lib/content/docs.test.ts) fails if the copies drift. + * + * cd web && pnpm sync-docs + */ +import { copyFileSync, mkdirSync, readdirSync, rmSync } from "node:fs"; +import path from "node:path"; + +const web = path.resolve(import.meta.dirname, ".."); +const src = path.resolve(web, "..", "docs"); +const dest = path.join(web, "content", "docs"); + +const RENDERED = ["model-card.md", "ai-use-statement.md"]; + +rmSync(dest, { recursive: true, force: true }); +mkdirSync(path.join(dest, "decisions"), { recursive: true }); +for (const file of RENDERED) copyFileSync(path.join(src, file), path.join(dest, file)); +for (const file of readdirSync(path.join(src, "decisions")).filter((f) => f.endsWith(".md"))) { + copyFileSync(path.join(src, "decisions", file), path.join(dest, "decisions", file)); +} +console.log(`synced docs into ${path.relative(process.cwd(), dest) || dest}`); diff --git a/web/src/app/ai-log/page.tsx b/web/src/app/ai-log/page.tsx new file mode 100644 index 0000000..7d42ed6 --- /dev/null +++ b/web/src/app/ai-log/page.tsx @@ -0,0 +1,35 @@ +import type { Metadata } from "next"; +import Link from "next/link"; + +import { AiLogClient } from "@/components/ai/ai-log-client"; +import { PageHeader } from "@/components/layout/page-header"; + +export const metadata: Metadata = { + title: "AI audit log", + description: + "Every AI call made from this browser: feature, provider, model, exact input, output, latency, tokens, the grounding check and your review decision. Exportable as JSON or CSV. API keys are never recorded.", +}; + +export default function AiLogPage() { + return ( +
+ +

+ Every call the optional AI features make is recorded here: what was sent, what came back, + which provider and model, how long it took, the tokens the provider reported, the + automated grounding check for commentary, and your decision on the output (accepted, + edited or rejected; automated evaluation calls are marked n/a). +

+

+ The log lives only in this browser (IndexedDB). This static site has no server, so nothing + here is sent to us. Your API key is never written to the log.{" "} + + AI use statement + + . +

+ + +
+ ); +} diff --git a/web/src/app/astar/page.tsx b/web/src/app/astar/page.tsx index e2c8820..36b5ac1 100644 --- a/web/src/app/astar/page.tsx +++ b/web/src/app/astar/page.tsx @@ -1,19 +1,88 @@ +import { Download } from "lucide-react"; import type { Metadata } from "next"; import { AstarLab } from "@/components/astar/astar-lab"; import { HeuristicStudy } from "@/components/astar/heuristic-study"; +import { PairedStudySummaryView } from "@/components/astar/paired-study-summary"; +import { Finding, Section } from "@/components/layout/page-header"; +import { Button } from "@/components/ui/button"; +import { summarisePairedStudy } from "@/lib/analysis/astar-paired"; +import { loadAstarStudy } from "@/lib/data/load"; +import { ASTAR_STUDY_CONFIG } from "@/lib/data/reference-config"; +import { formatNumber, formatPStatement, formatPct } from "@/lib/stats/format"; export const metadata: Metadata = { title: "A* Lab", description: - "Edit a Cachex board, run the original Part A A* search step by step, and compare Manhattan and Euclidean heuristics.", + "Edit a Cachex board, run the original Part A A* search step by step, and compare Manhattan and Euclidean heuristics in a paired study with intervals and an optimality check.", }; export default function AstarPage() { + const { rows, meta } = loadAstarStudy(); + const summary = summarisePairedStudy(rows, { seed: ASTAR_STUDY_CONFIG.seed }); + const e = summary.expansions; + const o = summary.optimality; + const cfg = ASTAR_STUDY_CONFIG; + // Phrase the paired mean difference as "fewer"/"more" rather than a negative count. + const fewer = e.meanDiff.estimate < 0; + const size = (x: number) => formatNumber(Math.abs(x), 1); + const [near, far] = fewer + ? [e.meanDiff.upper, e.meanDiff.lower] + : [e.meanDiff.lower, e.meanDiff.upper]; + const pp = (x: number) => formatNumber(Math.abs(x) * 100, 1); + const d = o.difference; + const [dNear, dFar] = d.estimate < 0 ? [d.upper, d.lower] : [d.lower, d.upper]; return (
+
+ {cfg.boardsPerDimension} random boards for every size from {cfg.dimensions[0]} ×{" "} + {cfg.dimensions[0]} to {cfg.dimensions.at(-1)} × {cfg.dimensions.at(-1)} from the + notebook's generator (seed {cfg.seed}), each searched with both heuristics and + solved by breadth-first search. Generated by{" "} + {meta.generatedBy} with the A* port that + reproduces the original paths and node counts exactly. +

+ } + > +
    + + The report's direction holds, and is now measured. Manhattan + expanded {size(e.meanDiff.estimate)} {fewer ? "fewer" : "more"} nodes per board on + average (paired 95% CI {size(near)} to {size(far)} {fewer ? "fewer" : "more"}); fewer + on {e.manhattanFewer} boards, more on {e.euclideanFewer}. + + + But it finds a shortest path less often. Manhattan returned an + optimal path on {formatPct(o.manhattan.p)} of solvable boards, Euclidean on{" "} + {formatPct(o.euclidean.p)}: {pp(o.difference.estimate)} points{" "} + {o.difference.estimate < 0 ? "fewer" : "more"} for Manhattan on the same boards + (paired 95% CI {pp(dNear)} to {pp(dFar)}). Where only one heuristic was optimal, it + was Manhattan on {o.mcnemar.b} boards and Euclidean on {o.mcnemar.c} (exact McNemar{" "} + {formatPStatement(o.mcnemar.pValue)}). + + + Why: neither heuristic is admissible here. On this hex grid a step + like (1, −1) costs 1, but Manhattan scores it 2 and Euclidean √2. Overestimating makes + the search greedier: fewer expansions, longer paths. The two sample inputs happen to + be solved optimally (8 and 13 cells, confirmed by BFS). + +
+ + +
diff --git a/web/src/app/data/alpha-beta-study.csv/route.ts b/web/src/app/data/alpha-beta-study.csv/route.ts new file mode 100644 index 0000000..f84ddb1 --- /dev/null +++ b/web/src/app/data/alpha-beta-study.csv/route.ts @@ -0,0 +1,21 @@ +import { toCsv } from "@/lib/csv"; +import { loadAlphaBetaStudy } from "@/lib/data/load"; + +export const dynamic = "force-static"; + +export function GET() { + const rows = loadAlphaBetaStudy().samples.map((s) => ({ + board_n: s.n, + depth: s.depth, + move_order: s.order, + empty_cells: s.empty, + nodes_full_minimax: s.nodesFull, + nodes_original_alphabeta: s.nodesPruned, + ratio_original: Number(s.ratio.toFixed(6)), + original_value_matches: s.valueMatches, + nodes_textbook_alphabeta: s.nodesTextbook, + ratio_textbook: Number(s.ratioTextbook.toFixed(6)), + textbook_value_matches: s.textbookValueMatches, + })); + return new Response(toCsv(rows), { headers: { "Content-Type": "text/csv; charset=utf-8" } }); +} diff --git a/web/src/app/data/astar-paired-study.csv/route.ts b/web/src/app/data/astar-paired-study.csv/route.ts new file mode 100644 index 0000000..e8ca68c --- /dev/null +++ b/web/src/app/data/astar-paired-study.csv/route.ts @@ -0,0 +1,11 @@ +import { pairedStudyCsvRows } from "@/lib/analysis/astar-paired"; +import { toCsv } from "@/lib/csv"; +import { loadAstarStudy } from "@/lib/data/load"; + +export const dynamic = "force-static"; + +export function GET() { + return new Response(toCsv(pairedStudyCsvRows(loadAstarStudy().rows)), { + headers: { "Content-Type": "text/csv; charset=utf-8" }, + }); +} diff --git a/web/src/app/data/reference-tournament-games.csv/route.ts b/web/src/app/data/reference-tournament-games.csv/route.ts new file mode 100644 index 0000000..3f1a033 --- /dev/null +++ b/web/src/app/data/reference-tournament-games.csv/route.ts @@ -0,0 +1,10 @@ +import { loadReferenceTournament } from "@/lib/data/load"; +import { gamesCsv } from "@/lib/tournament/analyse"; + +export const dynamic = "force-static"; + +export function GET() { + return new Response(gamesCsv(loadReferenceTournament().games), { + headers: { "Content-Type": "text/csv; charset=utf-8" }, + }); +} diff --git a/web/src/app/data/reference-tournament-summary.csv/route.ts b/web/src/app/data/reference-tournament-summary.csv/route.ts new file mode 100644 index 0000000..0b69b2c --- /dev/null +++ b/web/src/app/data/reference-tournament-summary.csv/route.ts @@ -0,0 +1,15 @@ +import { loadReferenceTournament } from "@/lib/data/load"; +import { REFERENCE_TOURNAMENT_CONFIG } from "@/lib/data/reference-config"; +import { summariseTournament, summaryCsv } from "@/lib/tournament/analyse"; + +export const dynamic = "force-static"; + +export function GET() { + const summary = summariseTournament(loadReferenceTournament().games, { + reps: 1000, + seed: REFERENCE_TOURNAMENT_CONFIG.seed, + }); + return new Response(summaryCsv(summary), { + headers: { "Content-Type": "text/csv; charset=utf-8" }, + }); +} diff --git a/web/src/app/layout.tsx b/web/src/app/layout.tsx index 0648690..803c404 100644 --- a/web/src/app/layout.tsx +++ b/web/src/app/layout.tsx @@ -1,6 +1,7 @@ import type { Metadata, Viewport } from "next"; import { Bricolage_Grotesque, Geist, Geist_Mono } from "next/font/google"; +import { AiProvider } from "@/components/ai/ai-provider"; import { SiteFooter } from "@/components/layout/site-footer"; import { SiteHeader } from "@/components/layout/site-header"; import { ThemeProvider } from "@/components/layout/theme-provider"; @@ -62,11 +63,13 @@ export default function RootLayout({ children }: { children: React.ReactNode }) enableSystem disableTransitionOnChange > - -
- {children} -
- + + +
+ {children} +
+ + diff --git a/web/src/app/llm-arena/page.tsx b/web/src/app/llm-arena/page.tsx new file mode 100644 index 0000000..415d6f7 --- /dev/null +++ b/web/src/app/llm-arena/page.tsx @@ -0,0 +1,43 @@ +import type { Metadata } from "next"; +import Link from "next/link"; + +import { LlmArenaClient } from "@/components/ai/llm-arena-client"; +import { PageHeader } from "@/components/layout/page-header"; + +export const metadata: Metadata = { + title: "LLM Arena", + description: + "An evaluation harness: a language model (your own API key) plays short games of Cachex against the original minimax agent, with legal-move validation, Wilson intervals and a full audit log.", +}; + +export default function LlmArenaPage() { + return ( +
+
+ +

+ The model plays short games against the original minimax agent. Each turn it gets the + rules, the board, the move history and the full list of legal moves, and must answer + with a JSON move. The referee checks every answer: an illegal move, or a reply with no + usable move (malformed, cut off or refused), is counted and the model is asked again (up + to three times) before it forfeits. Both providers are scored by the same rule. +

+

+ The same seeds and colours are replayed with the random and greedy agents and a scripted + first-legal-cell player in the model's seat, so the comparison is like for like (if + a run stops early, on the games the model finished). This page is an evaluation harness, + not a claim: a few games give wide intervals.{" "} + + What is sent to the provider + + . +

+ + +
+
+ ); +} diff --git a/web/src/app/methods/agent-card/page.tsx b/web/src/app/methods/agent-card/page.tsx new file mode 100644 index 0000000..40de66f --- /dev/null +++ b/web/src/app/methods/agent-card/page.tsx @@ -0,0 +1,47 @@ +import { ArrowLeft } from "lucide-react"; +import type { Metadata } from "next"; +import Link from "next/link"; + +import { PageHeader } from "@/components/layout/page-header"; +import { Markdown } from "@/components/methods/markdown"; +import { readDoc, withoutTitle } from "@/lib/content/docs"; +import { SITE } from "@/lib/site"; + +export const metadata: Metadata = { + title: "Agent card", + description: + "Agent card for the _4399 minimax agent: intended use, provenance, evaluation with confidence intervals, known weaknesses and ethical considerations.", +}; + +export default function AgentCardPage() { + return ( +
+ + Methods & decisions + + +
+ +
+

+ Source:{" "} + + docs/model-card.md + + . Live numbers with downloadable data:{" "} + + /tournament + + . +

+
+ ); +} diff --git a/web/src/app/methods/decisions/[slug]/page.tsx b/web/src/app/methods/decisions/[slug]/page.tsx new file mode 100644 index 0000000..848470c --- /dev/null +++ b/web/src/app/methods/decisions/[slug]/page.tsx @@ -0,0 +1,95 @@ +import { ArrowLeft } from "lucide-react"; +import type { Metadata } from "next"; +import Link from "next/link"; +import { notFound } from "next/navigation"; + +import { PageHeader } from "@/components/layout/page-header"; +import { Markdown } from "@/components/methods/markdown"; +import { listDecisions, readDoc, supersededBy, withoutTitle } from "@/lib/content/docs"; +import { SITE } from "@/lib/site"; + +export const dynamicParams = false; + +export function generateStaticParams() { + return listDecisions().map((d) => ({ slug: d.slug })); +} + +export async function generateMetadata({ + params, +}: { + params: Promise<{ slug: string }>; +}): Promise { + const { slug } = await params; + const d = listDecisions().find((x) => x.slug === slug); + return d + ? { title: `${d.id}: ${d.title}`, description: `Decision record ${d.id}: ${d.title}.` } + : {}; +} + +export default async function DecisionPage({ params }: { params: Promise<{ slug: string }> }) { + const { slug } = await params; + const all = listDecisions(); + const index = all.findIndex((x) => x.slug === slug); + if (index < 0) notFound(); + const d = all[index]; + const markdown = readDoc(`decisions/${slug}.md`); + const prev = all[index - 1]; + const next = all[index + 1]; + const later = supersededBy(d.id); + return ( +
+ + Methods & decisions + + + {later.length > 0 && ( +

+ Part of this record has been superseded by{" "} + {later.map((x, i) => ( + + {i > 0 && ", "} + + {x.id}: {x.title} + + + ))} + . The record below is unchanged, as written at the time. +

+ )} +
+ +
+ +

+ Source:{" "} + + docs/decisions/{slug}.md + +

+
+ ); +} diff --git a/web/src/app/methods/page.tsx b/web/src/app/methods/page.tsx new file mode 100644 index 0000000..06cd6b1 --- /dev/null +++ b/web/src/app/methods/page.tsx @@ -0,0 +1,374 @@ +import { ArrowRight, FileText } from "lucide-react"; +import type { Metadata } from "next"; +import Link from "next/link"; + +import { PageHeader, Section } from "@/components/layout/page-header"; +import { Markdown } from "@/components/methods/markdown"; +import { ScrollRegion } from "@/components/stats/scroll-table"; +import benchmark from "@/lib/data/agent-benchmark.json"; +import { listDecisions, readDoc, withoutTitle } from "@/lib/content/docs"; +import { + loadAlphaBetaStudy, + loadAstarStudy, + loadPythonCrosscheck, + loadReferenceTournament, +} from "@/lib/data/load"; +import { SITE } from "@/lib/site"; + +export const metadata: Metadata = { + title: "Methods & decisions", + description: + "Data provenance, methods, evaluation design, assumptions and limitations behind every number on Cachex Arena, plus decision records, the agent card and the AI use statement.", +}; + +export default function MethodsPage() { + const decisions = listDecisions(); + const ref = loadReferenceTournament().meta; + const astar = loadAstarStudy().meta; + const ab = loadAlphaBetaStudy().meta; + const py = loadPythonCrosscheck().meta; + const provenance = [ + { + what: "Original benchmark: agent vs random, 160 games", + how: "Original Python, unchanged, via the subject's referee", + source: benchmark.meta.generatedBy, + detail: `Python ${benchmark.meta.python}, PYTHONHASHSEED=${benchmark.meta.pythonHashSeed}`, + }, + { + what: "Parity fixtures (A*, rules, evaluation, minimax, moves)", + how: "Original Python, unchanged; shuffle replaced by a sort and bias fixed at 1", + source: "scripts/generate_parity_fixtures.py", + detail: "CPython 3.12", + }, + { + what: "Reference round robin, 1,200 games", + how: ref.implementation, + source: ref.generatedBy, + detail: `seed ${String(ref.config.seed)}; ${ref.cpu}; Node ${ref.node}; ${ref.generatedAt}`, + }, + { + what: "Python cross-check, 480 games", + how: py.implementation, + source: py.generatedBy, + detail: `Python ${py.python}, PYTHONHASHSEED=0; seed ${String(py.config.seed)}`, + }, + { + what: "A* paired heuristic study, 980 boards", + how: astar.implementation, + source: astar.generatedBy, + detail: `seed ${String(astar.config.seed)}; the notebook's board generator`, + }, + { + what: "Alpha-beta efficiency, 480 searches", + how: ab.implementation, + source: ab.generatedBy, + detail: `seed ${String(ab.config.seed)}; positions from greedy-vs-random games`, + }, + ]; + + return ( +
+
+ +

+ Provenance, methods and evaluation design for the tournament, the A* study and the + alpha-beta study; the assumptions and limitations behind them; and the decisions, agent + card and AI use statement that go with them. +

+ + + +
+

+ There is no external dataset. Every result is generated by running code in this + repository with a fixed seed, and the generating script is named next to it. The + original 2022 Python in coursework/ is never + modified; the TypeScript port is checked against it by parity tests (DR-003). +

+ + + + + + + + + + + + {provenance.map((p) => ( + + + + + + + ))} + +
+ Artefact + + Produced by + + Script + + Run details +
+ {p.what} + {p.how} + + {p.source} + + {p.detail}
+ +
+ +
+
+ + The original agent plus variants that each change one knob: depth fixed at 2 or 3, and + greedy one-ply (depth 1, no opening book). The same variants are rebuilt from the + unchanged Python for the cross-check by patching{" "} + dynamic_depth_allocation and the + opening-book flag. Random is the subject's baseline. + + + Every pairing plays the same number of games on each board size, as colour-swapped + pairs: one seed, played twice with the agents swapping Red and Blue, so the first-move + effect cancels within a pairing. The referee port validates every move; an illegal + move or crash forfeits the game. Per-move seeds are derived from the game seed and + move number, so any game can be replayed. + + + Win rates: Wilson score intervals. Strengths: Bradley-Terry fitted by the MM algorithm + (draws count half, one virtual draw per pairing to keep estimates finite), reported on + the Elo scale with random fixed at 0, and percentile intervals from a bootstrap that + resamples games within each pairing and board size. Move times: bootstrap intervals + over per-game means. + + + Paired design: both heuristics search the same random boards from the notebook's + generator, and the unit of analysis is the board. Mean per-board difference with a + paired bootstrap interval, a Wilcoxon signed-rank test (zero differences dropped, + tie-corrected normal approximation), the matched-pairs rank-biserial correlation and + Cohen's d_z. Optimality is checked against breadth-first search on the same + board. + + + The same position is searched at the same depth and move order three ways: plain + minimax, the original alpha-beta, and a textbook alpha-beta. The pruning ratio is the + share of the full tree visited, with bootstrap intervals over positions. Every search + is also checked to return the same root value as plain minimax. + + + All helpers live in web/src/lib/stats with + unit tests against reference values from scipy 1.17, statsmodels and R (Wilson + intervals, Newcombe differences, Wilcoxon exact and approximate p-values, type-7 + quantiles, Bradley-Terry maximum likelihood). Every interval shown records its seed + and number of resamples. + +
+
+ +
+
    +
  • + Questions asked: how strong is the original relative to controlled + variants and a baseline; does dynamic depth help; does the A* heuristic choice trade + speed for optimality; how much work does pruning save; and (with your own key) how + does a language model compare to the random and greedy agents on identical seeds. +
    • +
  • + Sample sizes are stated with every result. The reference round robin + has 40 games per pairing per board size (120 per pairing overall), which resolves + differences of roughly 15 to 20 percentage points in a head-to-head win rate; smaller + differences are reported as not detectable rather than as ties. +
    • +
  • + Paired comparisons where two methods are compared: heuristics on the + same boards (paired bootstrap and Wilcoxon for node counts, paired bootstrap and exact + McNemar for shortest-path rates), search variants on the same positions, + colour-swapped game pairs on the same seeds, agents compared through Elo differences + taken from the same bootstrap refits (never by whether two intervals overlap), and LLM + versus each baseline on exactly the games the model finished (games won by only one of + them, with an exact McNemar test). +
    • +
  • + LLM answers are scored per turn, one rule for both providers: the + rate reported is the share of turns whose first answer was rejected, split into + illegal moves and replies with no usable move (malformed, cut off or refused), because + retries on the same turn are not independent trials. Turns within a game are not + independent either (same model, prompt and position history), so the Wilson interval + is on the effective number of turns after a design effect estimated from how much the + rate varies between games, never below 1 (DR-005). With a handful of games the design + effect is itself rough. +
    • +
  • + Effect sizes, not just p-values: differences in win rate and Elo with + intervals, mean node differences, rank-biserial r and d_z. +
    • +
  • + No self-scored metrics. Results are measured against the referee and + against other agents, never against a score the author chose. +
    • +
  • + Multiple comparisons: ten pairings are reported side by side without + correction. Each interval is valid on its own; reading the table as a whole, expect + about one in twenty 95% intervals to miss. +
    • +
+
+ +
+
+ +
    +
  • + Games are independent given their seeds; colour-swapped pairs share a seed by + design, and the bootstrap resamples games, not pairs. +
    • +
  • + Bradley-Terry assumes strength is one number per agent (transitive, no style + matchups) and pools board sizes in the “all sizes” view. +
    • +
  • + The TypeScript port behaves like the original in random play as well as in the + deterministic cases the parity tests cover; the Python cross-check supports this + within its resolution. +
    • +
+ + +
    +
  • + Board sizes 4 to 6 only for the round robin; the original supports up to 15 × 15, + where depth 1 is likely to be weaker still. +
    • +
  • Move times depend on hardware and are only comparable within one run.
    • +
  • + The Python cross-check has 80 games per pairing and can only detect large + discrepancies (about ±15 points). +
    • +
  • + The A* study uses the notebook's random boards (barriers at random, up to + half the board); other board distributions may behave differently. +
    • +
  • + LLM-as-a-player runs are a handful of games against one opponent, with one prompt; + intervals are wide by design. +
    • +
+ +
+
+ +
+
    +
  • + Fix the one-line beta update and replace ratio-based depth with iterative deepening + under a time budget, then test the change as a new variant against the original + (DR-002). +
    • +
  • + Add a connection-distance feature and tune weights against data, not by hand (DR-001). +
    • +
  • + Pre-register comparisons and sample sizes before running the next tournament, and + extend it to larger boards. +
    • +
  • + Record the original Python's random draws so whole random games can be compared + move for move across languages (DR-003). +
    • +
+
+ +
+

+ Each record states the decision first, then context, options, reasons, what happened + (weak numbers included) and what I would change. Records are never edited after the + fact; a new record supersedes an old one. +

+
    + {decisions.map((d) => ( +
  • + + {d.id} + {d.title} + + Read + + +
    • + ))} +
  • + + + Agent card + + + The _4399 agent: intended use, evaluation with intervals, known weaknesses + + + Read + + +
    • +
+
+ +
+ +

+ + Open the AI audit log + {" "} + to see every call made from this browser. +

+
+
+
+ ); +} + +function Block({ title, children }: { title: string; children: React.ReactNode }) { + return ( +
+

{title}

+
{children}
+
+ ); +} diff --git a/web/src/app/page.tsx b/web/src/app/page.tsx index e8cbcc0..3467f6b 100644 --- a/web/src/app/page.tsx +++ b/web/src/app/page.tsx @@ -1,16 +1,32 @@ -import { ArrowRight, Bot, BrainCircuit, GitBranch, Route, Swords } from "lucide-react"; +import { + ArrowRight, + Bot, + BrainCircuit, + CirclePlay, + FlaskConical, + GitBranch, + NotebookPen, + Route, + Sparkles, + Swords, + Trophy, +} from "lucide-react"; import Link from "next/link"; import { HexBoard } from "@/components/board/hex-board"; import { StaticBoard } from "@/components/landing/static-board"; import { GitHubIcon } from "@/components/layout/github-icon"; import { ColourDot } from "@/components/play/primitives"; +import { IntervalBar } from "@/components/stats/interval"; +import { ScrollRegion } from "@/components/stats/scroll-table"; import { Button } from "@/components/ui/button"; import { FEATURES } from "@/lib/agent/evaluation"; import { TARGET_RATES } from "@/lib/agent/player"; import { deterministicSelfPlay } from "@/lib/agent/self-play"; import benchmark from "@/lib/data/agent-benchmark.json"; import { SITE } from "@/lib/site"; +import { formatPct } from "@/lib/stats/format"; +import { wilson } from "@/lib/stats/proportion"; export default function HomePage() { const { game } = deterministicSelfPlay(7); @@ -25,6 +41,8 @@ export default function HomePage() { { wins: 0, games: 0 }, ); const winRate = Math.round((totals.wins / totals.games) * 100); + const rowCI = (r: { wins: number; games: number }) => wilson(r.wins, r.games); + const winCI = wilson(totals.wins, totals.games); return (
@@ -64,6 +82,22 @@ export default function HomePage() {
+ + + + + + New here? Take the guided tour + · three short walkthroughs + + +
@@ -285,7 +319,7 @@ export default function HomePage() {
-
+

Agent _4399 vs the random baseline

{benchmark.results[0].games} seeded games per row, generated by{" "} - scripts/benchmark_agent.py. + scripts/benchmark_agent.py. Wilson 95% intervals; + with {benchmark.results[0].games} games a row each one is wide, so the pooled rate + is the better summary.

- - - - - - - - - - - - {benchmark.results.map((r) => ( - - - - + + + + ))} + +
- Board - - Agent - Agent plays - - Win rate - - W–L - - Turns - Avg turns -
- {r.n} × {r.n} - - - {r.agentColour} - - - - - + + + + + + + + + + + + + {benchmark.results.map((r) => ( + + + + - - - - ))} - -
+ Board + + Agent + Agent plays + + Win rate (95% CI) + + W–L + + Turns + Avg turns +
+ {r.n} × {r.n} + + + {r.agentColour} - - {Math.round((r.wins / r.games) * 100)}% + + + + + {Math.round(rowCI(r).p * 100)}% + + [{Math.round(rowCI(r).lower * 100)},{" "} + {Math.round(rowCI(r).upper * 100)}] + + - - - {r.wins}–{r.losses} - {r.avgTurns}
+ 		+ {r.wins}–{r.losses} + {r.avgTurns}
+
+ {/* Evaluation */} +
+ + The revival adds a seeded tournament harness, a paired A* study and an optional LLM + evaluation, each reported with sample sizes and confidence intervals, plus the decisions + and weaknesses behind them. + +
+ + + + +
+
+ {/* About */}
+ +

{title}

+

{body}

+ + Open + + + ); +} + function BigStat({ value, label }: { value: string; label: string }) { return (
diff --git a/web/src/app/tour/page.tsx b/web/src/app/tour/page.tsx new file mode 100644 index 0000000..d7fb8b8 --- /dev/null +++ b/web/src/app/tour/page.tsx @@ -0,0 +1,188 @@ +import { ArrowRight, Clapperboard, Info } from "lucide-react"; +import type { Metadata } from "next"; +import Link from "next/link"; + +import { PageHeader, Section } from "@/components/layout/page-header"; +import { LazyVideo } from "@/components/tour/lazy-video"; +import { ScreenshotGallery } from "@/components/tour/screenshot-gallery"; +import { Button } from "@/components/ui/button"; +import { + MOCK_LABEL, + MOCK_MODEL_ID, + SCREENSHOTS, + WALKTHROUGHS, + type Walkthrough, + walkthroughMedia, +} from "@/lib/showcase"; +import { SITE } from "@/lib/site"; + +export const metadata: Metadata = { + title: "Guided tour", + description: + "Three short captioned walkthroughs (play the agent, the A* Lab, the tournament and LLM evaluation) and screenshots of every key feature, recorded by a reproducible Playwright script.", +}; + +const SPEC_URL = `${SITE.repo}/blob/${SITE.ref}/web/e2e/showcase.spec.ts`; + +export default function TourPage() { + return ( +
+
+ +

+ Each video follows one workflow from start to finish, with the step shown on screen and + as captions. They were recorded from this site by a Playwright script that also checks + every step (the steal, both captures, the recorded A* output, the seeded tournament), so + the same seeds reproduce the same moves and numbers. +

+ + + + {WALKTHROUGHS.map((w, i) => ( + + ))} + +
+ Captured by the same script, in light mode at 1440 × 900 (the landing page also in + dark mode) and on a 390 px phone. Select one to enlarge it; use the arrow keys to step + through. +

+ } + > + +
+ +
+ +
+

+ How these were made +

+

+ pnpm showcase runs{" "} + + web/e2e/showcase.spec.ts + {" "} + on the system Chrome: it plays each journey at a human pace with an on-screen caption + and cursor, asserts what it shows, and records it at 1280 × 800. ffmpeg then encodes + the H.264 videos here and the GIFs in the README. The captions on this page are the + same text as the on-screen steps. +

+

+ No real API key is used anywhere in these recordings. Where an AI feature appears, the + key is a placeholder, every request to the provider is intercepted in the browser, and + the reply is a labelled mock. +

+
+
+
+
+ ); +} + +/** "Steps 6 to 10" for a run of consecutive steps, otherwise "Steps 2, 5 and 7". */ +function stepRange(steps: number[]) { + const sorted = [...steps].sort((a, b) => a - b); + const consecutive = sorted.every((s, i) => i === 0 || s === sorted[i - 1] + 1); + if (sorted.length > 2 && consecutive) return `Steps ${sorted[0]} to ${sorted.at(-1)}`; + if (sorted.length === 1) return `Step ${sorted[0]}`; + return `Steps ${sorted.slice(0, -1).join(", ")} and ${sorted.at(-1)}`; +} + +function WalkthroughSection({ + walkthrough: w, + index, +}: { + walkthrough: Walkthrough; + index: number; +}) { + const media = walkthroughMedia(w.id); + const mocked = new Set(w.mockedSteps ?? []); + const titleId = `${w.id}-steps`; + return ( +
{w.summary}

} + > +
+
+ +
+ + Setup: {w.setup} + + + Open the MP4 + +
+ {mocked.size > 0 && ( +

+ + + {MOCK_LABEL}. {stepRange([...mocked])} use a placeholder key and + the model id {MOCK_MODEL_ID}. Requests to the + provider are intercepted in the browser and answered by a mock that plays the first + legal cell; no model was called, so the LLM row shows the mock, not a real + model's results. + +

+ )} +
+ +
+

+ Steps (transcript) +

+
    + {w.steps.map((step, k) => ( +
  1. + + {k + 1} + + + {step} + {mocked.has(k + 1) && ( + + Mocked AI response for illustration + + )} + +
    2. + ))} +
+ +
+
+
+ ); +} diff --git a/web/src/app/tournament/page.tsx b/web/src/app/tournament/page.tsx new file mode 100644 index 0000000..b627e43 --- /dev/null +++ b/web/src/app/tournament/page.tsx @@ -0,0 +1,491 @@ +import { ArrowRight, Download } from "lucide-react"; +import type { Metadata } from "next"; +import Link from "next/link"; + +import { Finding, PageHeader, Section } from "@/components/layout/page-header"; +import { EstimateCI, IntervalAxis, IntervalBar } from "@/components/stats/interval"; +import { AlphaBetaTable } from "@/components/tournament/alpha-beta-table"; +import { ReferenceTournament } from "@/components/tournament/reference-tournament"; +import { TournamentRunner } from "@/components/tournament/tournament-runner"; +import { Button } from "@/components/ui/button"; +import { ScrollTable } from "@/components/stats/scroll-table"; +import { summarisePruning } from "@/lib/analysis/alpha-beta"; +import benchmark from "@/lib/data/agent-benchmark.json"; +import { loadAlphaBetaStudy, loadPythonCrosscheck, loadReferenceTournament } from "@/lib/data/load"; +import { ALPHA_BETA_CONFIG, REFERENCE_TOURNAMENT_CONFIG } from "@/lib/data/reference-config"; +import { formatPct, formatSigned } from "@/lib/stats/format"; +import { wilson } from "@/lib/stats/proportion"; +import { AGENTS } from "@/lib/tournament/agents"; +import { eloDifference, summariseTournament } from "@/lib/tournament/analyse"; +import { compareImplementations } from "@/lib/tournament/crosscheck"; + +export const metadata: Metadata = { + title: "Tournament", + description: + "A seeded, colour-swapped round robin between the original minimax agent and controlled variants, with Wilson and bootstrap intervals, Bradley-Terry strengths and an alpha-beta efficiency study.", +}; + +const WORDS = ["no", "one", "two", "three", "four", "five", "six", "seven", "eight"]; + +const pctCI = (lo: number, hi: number) => `[${(lo * 100).toFixed(1)}, ${(hi * 100).toFixed(1)}]`; + +export default function TournamentPage() { + // Original benchmark (Python), restated with intervals. + const totals = benchmark.results.reduce( + (a, r) => ({ wins: a.wins + r.wins, games: a.games + r.games }), + { wins: 0, games: 0 }, + ); + const overall = wilson(totals.wins, totals.games); + const bySize = [...new Set(benchmark.results.map((r) => r.n))].map((n) => { + const rows = benchmark.results.filter((r) => r.n === n); + const wins = rows.reduce((s, r) => s + r.wins, 0); + const games = rows.reduce((s, r) => s + r.games, 0); + return { n, ci: wilson(wins, games) }; + }); + + // Reference tournament (TypeScript port). + const ref = loadReferenceTournament(); + const cfg = REFERENCE_TOURNAMENT_CONFIG; + const summaries = [ + { + key: "all", + label: "All sizes", + summary: summariseTournament(ref.games, { reps: 1000, seed: cfg.seed }), + }, + ...cfg.sizes.map((n) => ({ + key: String(n), + label: `${n} × ${n}`, + summary: summariseTournament(ref.games, { reps: 1000, seed: cfg.seed, sizes: [n] }), + })), + ]; + const all = summaries[0].summary; + const pair = (a: string, b: string) => all.pairs.find((p) => p.a === a && p.b === b)!; + const dynVsGreedy = pair("minimax-dynamic", "greedy"); + const dynVsD3 = pair("minimax-dynamic", "minimax-d3"); + const dynVsRandom = pair("minimax-dynamic", "random"); + const standing = (id: string) => all.standings.find((s) => s.id === id)!; + const dyn = standing("minimax-dynamic"); + const d3 = standing("minimax-d3"); + // Strength comparisons use differences from the same bootstrap refits, not overlap. + const dynMinusGreedy = eloDifference(all, "minimax-dynamic", "greedy")!; + const d3MinusDyn = eloDifference(all, "minimax-d3", "minimax-dynamic")!; + const elo = (x: number) => formatSigned(x, 0); + const deepSearches = ref.games.reduce( + (acc, g) => { + for (const s of [g.red, g.blue]) { + if (s.agent !== "minimax-dynamic") continue; + acc.searches += s.searches; + acc.deep += s.deepSearches; + } + return acc; + }, + { searches: 0, deep: 0 }, + ); + const hardware = `${ref.meta.cpu ?? "unknown CPU"}, Node ${ref.meta.node ?? "?"}`; + + // Cross-check against the original Python. + const py = loadPythonCrosscheck(); + const cross = compareImplementations(ref.games, py.games); + + // Alpha-beta efficiency. + const ab = loadAlphaBetaStudy(); + const groups = summarisePruning(ab.samples, { seed: ALPHA_BETA_CONFIG.seed }); + const shuffled = groups.filter((g) => g.order === "shuffled"); + const canonical = groups.filter((g) => g.order === "canonical"); + const worst = shuffled.reduce((a, b) => (b.meanRatio.estimate < a.meanRatio.estimate ? b : a)); + const deepest = shuffled.find((g) => g.n === 6 && g.depth === 3)!; + const allMatch = ab.samples.every((s) => s.valueMatches && s.textbookValueMatches); + // The averages hide an all-or-nothing distribution: most searches prune nothing. + const ownOrder = ab.samples.filter((s) => s.order === "shuffled"); + const pruned = ownOrder.filter((s) => s.ratio < 1); + const prunedRange = pruned.length + ? [Math.min(...pruned.map((s) => s.ratio)), Math.max(...pruned.map((s) => s.ratio))] + : null; + + return ( +
+
+ +

+ A seeded round robin between the original{" "} + _4399 agent and variants that each change + exactly one knob, with colour-swapped pairs to cancel the first-move advantage. Every + number comes with its sample size and a 95% interval. +

+

+ Jump to:{" "} + + original benchmark + {" "} + ·{" "} + + reference round robin + {" "} + ·{" "} + + Python cross-check + {" "} + ·{" "} + + run your own + {" "} + ·{" "} + + alpha-beta efficiency + +

+ + +
+ The headline result from the original Python ({benchmark.meta.generatedBy}, Python{" "} + {benchmark.meta.python}): the agent beat the subject's random agent in{" "} + {totals.wins} of {totals.games} games on boards 4 to 7, 20 seeded games per size and + colour. The count is unchanged; the interval is new. +

+ } + > +
+
+
+ {formatPct(overall.p)} +
+

+ {totals.wins} of {totals.games} games won against the random agent +

+
+ +

+ Wilson 95% CI{" "} + + {formatPct(overall.lower)} to {formatPct(overall.upper)} + +

+
+
+ + By board size (both colours pooled, 40 games each). Small samples give wide + intervals. + + } + > + + + + Board + + + Interval + `${t * 100}%`} + /> + + + Win rate (95% CI) + + + + + {bySize.map(({ n, ci }) => ( + + + {n} × {n} + + + + + + + + + ))} + + +
+
+ +
+

+ Every pairing played {cfg.rounds * 2} games per board size ({cfg.rounds} seeds, each + played twice with colours swapped) on{" "} + {cfg.sizes.map((n) => `${n}\u00a0×\u00a0${n}`).join(", ")}. Tournament seed{" "} + {cfg.seed}. Played by the TypeScript port, which reproduces the original Python move + for move in deterministic mode; the original is too slow for depth-3 search at this + scale, so a feasible subset was re-run in Python below. +

+ + } + > +
    + + Depth matters; dynamic depth barely runs. Fixed depth 3 beat the + original in {dynVsD3.bWins} of {dynVsD3.games} games ( + {formatPct(1 - dynVsD3.aWinRate.p)}, 95% CI {formatPct(1 - dynVsD3.aWinRate.upper)} to{" "} + {formatPct(1 - dynVsD3.aWinRate.lower)}). The original searched deeper than one ply on + only {deepSearches.deep} of {deepSearches.searches.toLocaleString("en-AU")} searched + moves ({formatPct(deepSearches.deep / deepSearches.searches)}): its thresholds only + deepen once fewer than 15% of cells are empty, and most games end before that. + + + No detectable difference from greedy one-ply. Head to head the + original won {dynVsGreedy.aWins} and lost {dynVsGreedy.bWins} ( + {formatPct(dynVsGreedy.aWinRate.p)}, 95% CI {formatPct(dynVsGreedy.aWinRate.lower)} to{" "} + {formatPct(dynVsGreedy.aWinRate.upper)}), and its strength minus greedy's is{" "} + {elo(dynMinusGreedy.estimate)} Elo (95% CI {elo(dynMinusGreedy.lower)} to{" "} + {elo(dynMinusGreedy.upper)}). The interval still allows the original to win up to + about {Math.round((dynVsGreedy.aWinRate.upper - 0.5) * 100)} points more or{" "} + {Math.round((0.5 - dynVsGreedy.aWinRate.lower) * 100)} points less than half its games + against greedy, so only larger differences are ruled out; the opening book and the + late deepening add no detectable strength. + + + Against random it reproduces the original benchmark.{" "} + {dynVsRandom.aWins}/{dynVsRandom.games} wins ({formatPct(dynVsRandom.aWinRate.p)}, 95% + CI {formatPct(dynVsRandom.aWinRate.lower)} to {formatPct(dynVsRandom.aWinRate.upper)}) + on boards 4 to 6, consistent with the 90% the Python run recorded on boards 4 to 7. + + + Strength costs time. Depth 3 is {Math.round(d3MinusDyn.estimate)} Elo + above the original (95% CI {Math.round(d3MinusDyn.lower)} to{" "} + {Math.round(d3MinusDyn.upper)}) but spends about{" "} + {Math.round(d3.moveMs.estimate / dyn.moveMs.estimate)}× as long per move, partly + because of the pruning bug described{" "} + + below + + . No first-move advantage was detectable: Red won {formatPct(all.colour.redWinRate.p)}{" "} + (95% CI {formatPct(all.colour.redWinRate.lower)} to{" "} + {formatPct(all.colour.redWinRate.upper)}). + +
+ + +
+ +
+ scripts/crosscheck_tournament.py rebuilt + the same variants from the unchanged Python (patching only{" "} + dynamic_depth_allocation and the + opening-book flag) and replayed the pairings that are feasible in Python on boards{" "} + {cross[0]?.sizes.join(" and ")}, with the same colour-swapped protocol. Random streams + differ between the languages, so games differ; the question is whether win rates + agree. With {cross[0]?.python.n} games per pairing this can only rule out large + discrepancies. +

+ } + > + + Win rate of the first-named agent. Difference = Python − TypeScript, with + Newcombe's 95% interval; an interval that contains 0 is consistent with no + difference. + + } + > + + + + Pairing + + + Original Python + + + TypeScript port + + + Difference interval + `${t > 0 ? "+" : ""}${Math.round(t * 100)}`} + /> + + + Difference (pp) + + + + + {cross.map((r) => ( + + + {AGENTS[r.a].short} vs{" "} + {AGENTS[r.b].short} + + + + + + + + + + + + + + + ))} + + +
+ +
+ Same harness, same statistics, run on your machine in Web Workers. Pick a seed to make + the run reproducible, then export the games and the summary as CSV. +

+ } + > + +
+ +
+

+ {ab.samples.length / (ALPHA_BETA_CONFIG.depths.length * 2)} mid-game positions ( + {ALPHA_BETA_CONFIG.perSize} per board size, from seeded greedy-vs-random games, seed{" "} + {ALPHA_BETA_CONFIG.seed}) were searched three ways at the same depth and move order: + plain minimax (no pruning), the original alpha-beta as submitted, and a textbook + alpha-beta. The share of the full tree each one visits is the pruning ratio; lower + is better. +

+

+ In the agent's own move order the original prunes very little: averaged per + board size and depth it still visits at least {formatPct(worst.meanRatio.estimate)}{" "} + of the tree, and {ownOrder.length - pruned.length} of {ownOrder.length} searches + visited every node. The cause is one line in{" "} + minimax.py: the minimising branch updates + beta with{" "} + if beta <= min_score: beta = min_score + , which can only raise beta, so beta stays at +infinity and a cut-off can only + happen once the search finds a line it scores as a forced win for Red. + {prunedRange && + ` The ${pruned.length} searches where that happened visited ${formatPct(prunedRange[0])} to ${formatPct(prunedRange[1])} of the tree.`}{" "} + The answer is still right ({allMatch ? "all" : "not all"} {ab.samples.length}{" "} + searches returned the same root value as plain minimax), just slow. With{" "} + beta = min(beta, min_score) the same + search on 6 × 6 at depth 3 would visit {formatPct(deepest.textbookRatio.estimate)}{" "} + of the tree instead of {formatPct(deepest.meanRatio.estimate)}. The agent on this + site is left as submitted. +

+ + } + > + +
+ + Canonical (r, q) move order + +
+ +
+
+
+ + +
+
+
+
+ ); +} diff --git a/web/src/components/ai/ai-badge.tsx b/web/src/components/ai/ai-badge.tsx new file mode 100644 index 0000000..be07f93 --- /dev/null +++ b/web/src/components/ai/ai-badge.tsx @@ -0,0 +1,19 @@ +import { Sparkles } from "lucide-react"; + +import { cn } from "@/lib/utils"; + +/** Visible label on every piece of model output. */ +export function AiBadge({ model, className }: { model?: string; className?: string }) { + return ( + + + AI-generated + {model && · {model}} + + ); +} diff --git a/web/src/components/ai/ai-log-client.tsx b/web/src/components/ai/ai-log-client.tsx new file mode 100644 index 0000000..8cb6514 --- /dev/null +++ b/web/src/components/ai/ai-log-client.tsx @@ -0,0 +1,387 @@ +"use client"; + +import { Check, Download, Pencil, RefreshCw, Trash2, X } from "lucide-react"; +import { useCallback, useEffect, useMemo, useState } from "react"; + +import { Segmented } from "@/components/play/primitives"; +import { Button } from "@/components/ui/button"; +import { Textarea } from "@/components/ui/textarea"; +import { + type AuditEntry, + type AuditGrounding, + type AuditStore, + DECISION_LABEL, + type HumanDecision, + auditToCsv, + auditToJson, + onAuditChange, +} from "@/lib/ai/audit-log"; +import { CommentarySchema, commentaryToText, recheckLoggedCommentary } from "@/lib/ai/commentator"; +import { FEATURE_LABEL, PROVIDER_LABEL } from "@/lib/ai/types"; +import { downloadText, localDateStamp } from "@/lib/download"; +import { cn } from "@/lib/utils"; +import { AiBadge } from "./ai-badge"; +import { useAi } from "./ai-provider"; +import { GroundingDetails } from "./grounding-details"; + +type Filter = "all" | "commentator" | "llm-player"; + +interface EntryGrounding { + grounding: AuditGrounding; + /** False when recomputed from the logged prompt (entries logged before checks were stored). */ + stored: boolean; +} + +/** The grounding check for an entry: as stored, or recomputed for older commentary entries. */ +function groundingOf(e: AuditEntry): EntryGrounding | null { + if (e.grounding) return { grounding: e.grounding, stored: true }; + if (e.feature !== "commentator" || e.error) return null; + const g = recheckLoggedCommentary(e.input.user, e.output); + return g ? { grounding: g, stored: false } : null; +} + +export function AiLogClient() { + const { audit } = useAi(); + const [entries, setEntries] = useState(null); + const [error, setError] = useState(null); + const [filter, setFilter] = useState("all"); + const [confirmClear, setConfirmClear] = useState(false); + + const refresh = useCallback(() => { + audit + .list() + .then((list) => { + setEntries(list); + setError(null); + }) + .catch((err: unknown) => setError(err instanceof Error ? err.message : String(err))); + }, [audit]); + + useEffect(() => { + // Subscribe first, then load: both callbacks set state asynchronously. + const unsubscribe = onAuditChange(refresh); + const timer = setTimeout(refresh, 0); + return () => { + unsubscribe(); + clearTimeout(timer); + }; + }, [refresh]); + + const checks = useMemo( + () => new Map((entries ?? []).map((e) => [e.id, groundingOf(e)] as const)), + [entries], + ); + const shown = useMemo( + () => (entries ?? []).filter((e) => filter === "all" || e.feature === filter), + [entries, filter], + ); + + const stats = useMemo(() => { + const list = entries ?? []; + const reviewed = list.filter((e) => + ["accepted", "edited", "rejected"].includes(e.humanDecision), + ); + const checked = list.map((e) => checks.get(e.id)).filter((g) => g != null); + return { + calls: list.length, + errors: list.filter((e) => e.error).length, + checked: checked.length, + failedChecks: checked.filter((g) => !g.grounding.passed).length, + reviewed: reviewed.length, + accepted: reviewed.filter((e) => e.humanDecision === "accepted").length, + edited: reviewed.filter((e) => e.humanDecision === "edited").length, + rejected: reviewed.filter((e) => e.humanDecision === "rejected").length, + }; + }, [entries, checks]); + + return ( +
+
+
+ +
+
+ + + + {confirmClear ? ( + <> + + + + ) : ( + + )} +
+
+ +
+ + + + + +
+ + {error && ( +

+ Could not read the log: {error} +

+ )} + {entries === null && !error &&

Loading…

} + {entries !== null && shown.length === 0 && ( +
+ No AI calls recorded in this browser yet. Use the commentator on /play or /spectate, or + run the LLM Arena, with your own key. +
+ )} + +
    + {shown.map((e) => { + const check = checks.get(e.id) ?? null; + return ( +
  1. +
    + {FEATURE_LABEL[e.feature]} + + {new Date(e.timestamp).toLocaleString("en-AU")} + + + {PROVIDER_LABEL[e.provider]} · {e.model} + + + {Math.round(e.latencyMs).toLocaleString("en-AU")} ms + {e.usage + ? ` · ${e.usage.inputTokens} in / ${e.usage.outputTokens} out tokens` + : ""} + + + {check && !check.grounding.passed && ( + + Grounding check failed + + )} + + {DECISION_LABEL[e.humanDecision]} + + +
    + {e.error ? ( +
    +

    + Error ({e.error.kind}): {e.error.message} +

    + {e.outputText && ( + <> + +

    + Raw reply as received (it did not pass validation): +

    + 
    +                        {e.outputText}
+
    + + )} +
    + ) : ( +
    + + 
    +                    {JSON.stringify(e.output, null, 2)}
+
    +
    + )} + {check && ( + + )} + {e.humanDecision === "pending" && !e.error && ( + + )} + {e.editedOutput !== undefined && ( +
    +

    Human-edited version

    +

    {e.editedOutput}

    +
    + )} +
    + + Exact input sent (system prompt, user prompt, schema) + + 
    +                  {`SYSTEM\n${e.input.system}\n\nUSER\n${e.input.user}\n\nSCHEMA ${e.input.schema}`}
+                  {e.context ? `\n\nCONTEXT ${JSON.stringify(e.context)}` : ""}
+
    +
    +
    2. + ); + })} +
+
+ ); +} + +/** + * Review an entry that was never decided where it was generated (for example + * commentary that arrived after the visitor moved on). Same choices as the + * commentator panel; the decision is written back to the audit entry. + */ +function ReviewControls({ + entry, + check, + audit, + onDone, +}: { + entry: AuditEntry; + /** The check the reviewer saw; stored with the decision if the entry lacks one. */ + check: EntryGrounding | null; + audit: AuditStore; + onDone: () => void; +}) { + const [editing, setEditing] = useState(null); + const decide = async (decision: HumanDecision, edited?: string) => { + await audit.update(entry.id, { + humanDecision: decision, + decidedAt: new Date().toISOString(), + ...(edited !== undefined ? { editedOutput: edited } : {}), + ...(check && !check.stored ? { grounding: check.grounding } : {}), + }); + setEditing(null); + onDone(); + }; + const startEdit = () => { + const parsed = CommentarySchema.safeParse(entry.output); + setEditing( + parsed.success ? commentaryToText(parsed.data) : JSON.stringify(entry.output, null, 2), + ); + }; + if (editing !== null) { + return ( +
+