reportgen: non-canonical kv_cache run layout publishes one run's metrics and silently discards the rest

[FileSystemGuy]

Summary

When a kv_cache workload directory contains multiple run directories that do not match the canonical <model>/run/<timestamp>/ layout, reportgen publishes a results row built from one of those runs and silently discards the rest. The only diagnostic emitted is No valid run directories found, which describes neither the row that was produced nor the measurements that were dropped.

This is worse than dropping the workload entirely: a blank row is visible to a reviewer, whereas a populated row assembled from one of several measured configurations is not.

Observed in

mlcommons/submissions_storage_v3.0 @ 316717f2e, ANL crux-eagle, tool version 3.0.46.

The submission organizes its kv_cache runs by cluster topology instead of by run timestamp:

closed/ANL/results/crux-eagle/kv_cache/llama3-8b-10u/
├── 1nodex8ppn/    (8 ranks)    summary.json + kv_cache_20260723_062638_metadata.json
├── 8nodex8ppn/    (64 ranks)   summary.json + kv_cache_20260723_191548_metadata.json
└── 64nodex8ppn/   (512 ranks)  summary.json + kv_cache_20260724_051511_metadata.json

All three directories are complete: each has a summary.json with options 1/2/3, option subdirectories, a *_metadata.json, and 389 per-rank result files between them. Nothing is missing from the submission.

Behavior

All three runs are discovered and verified. From the reportgen log:

INFO: Created benchmark run: kv_cache_run_llama3.1-8b_20260723_062638
INFO: Created benchmark run: kv_cache_run_llama3.1-8b_20260724_051511
INFO: Created benchmark run: kv_cache_run_llama3.1-8b_20260723_191548
STATUS: Benchmark run qualifies for CLOSED category (... run_datetime='20260723_062638')
STATUS: Benchmark run qualifies for CLOSED category (... run_datetime='20260724_051511')
STATUS: Benchmark run qualifies for CLOSED category (... run_datetime='20260723_191548')
WARNING: No valid run directories found in .../crux-eagle/kv_cache/llama3-8b-10u

The resulting tables disagree with each other:

Table	Rows
crux-eagle/kv_cache/llama3-8b-10u/results.csv	0 (and results.json is [])
crux-eagle/kv_cache/results.csv	1
top-level results.csv	1 (v3.0-0005)

The workload-level table is empty, consistent with the warning. But the system rollup and the global table both carry a row — and its values are 1nodex8ppn's alone:

Cell	Published in v3.0-0005	1nodex8ppn	8nodex8ppn	64nodex8ppn
8b Storage Only tok/s	5865.99	5865.99	48.57	0.0
8b Storage + Mem tok/s	49.84	49.84	170.16	163.96
70b Storage Only tok/s	2521.63	2521.63	1110.03	738.48
8b Storage Only Read B/W	0.0	0.0	0.776	0.0
# Client Nodes	1	1	8	64

The 8-node and 64-node measurements — which have materially different bandwidth and latency figures — do not appear anywhere in any published table. The three zero-valued bandwidth cells and three blank P95 cells are likewise the 1-node run's own values, not placeholders.

Empirically the earliest-timestamped run wins, but I did not pin the selection rule.

Apparent root cause

Two different paths disagree about what constitutes a run:

• Row placement uses the canonical layout. _validate_model_dir (mlpstorage_py/reporting/directory_validator.py:411) accepts a child of the model directory only if _is_datetime_dir(entry.name) or entry.name in EXPECTED_COMMANDS. 1nodex8ppn is neither, so nothing is placed into the workload-level table and the warning at line 428 fires.
• Rollup aggregation consumes the extracted BenchmarkRun objects directly. Those were built successfully — each topology directory has a *_metadata.json, so ResultFilesExtractor.extract (mlpstorage_py/rules/models.py:976) identifies all three as kvcache runs without needing the directory name.

So the runs exist in memory, are excluded from the leaf table, and then collapse into a single row upstream because all three share the same (program, command, model) grouping key and differ only in run_datetime.

Why the warning is inadequate

No valid run directories found is emitted against the workload directory, is a WARNING among 183 others, and names no consequence. Nothing in the output states that a row was published, that it represents one of three configurations, or which two were dropped. In a 179-row table this is not something a reviewer can be expected to catch — the row looks complete.

Suggested fixes

Independent of any decision about whether the layout itself should be accepted:

1. Never silently reduce N measured configurations to one. If multiple runs group to one row and their metrics disagree, that is either an error or a row per configuration — not a silent pick.
2. Make the warning name the consequence, in the style of reportgen silently drops kv_cache/vdb runs missing *_metadata.json — metrics publish as blank #835: say that a row was published from a subset, how many runs were dropped, and which.
3. Make the two paths agree. A run good enough to extract, verify and aggregate into a rollup should not be invisible to the leaf table that is supposed to contain it — the disagreement between an empty llama3-8b-10u/results.csv and a populated parent rollup is itself a bug.
4. Submission-checker gate: a workload whose leaf table has zero rows while its rollup has one should fail validation.

Item 1 is the one that affects published results.

Related

• reportgen silently drops kv_cache/vdb runs missing *_metadata.json — metrics publish as blank #835 — silent drop of kv_cache/VDB run leaves missing *_metadata.json. Same family of problem (a run disappears without the output saying so), different trigger. This one is more severe because it produces a wrong number rather than a blank.
• The layout question itself is a separate WG matter, written up as open question 3 in ApparentProblems.md.

[FileSystemGuy]

Correction: the dropped runs are failed runs, and the layout is a repackaging artifact

Re-examined the tree at file level rather than directory level. Two claims in the original report are wrong, and correcting them changes what the fix should be.

1. This is not a novel layout — it is three systems flattened into one

Each run's own *_metadata.json records the path the tool wrote it to on the run machine:

Dir as submitted	metadata.result_dir
crux-eagle/kv_cache/llama3-8b-10u/1nodex8ppn/	.../results/crux-eagle/kv_cache/llama3.1-8b/run/20260723_062638
crux-eagle/kv_cache/llama3-8b-10u/8nodex8ppn/	.../results/crux-eagle-n8/kv_cache/llama3.1-8b/run/20260723_191548
crux-eagle/kv_cache/llama3-8b-10u/64nodex8ppn/	.../results/crux-eagle-n64/kv_cache/llama3.1-8b/run/20260724_051511

All three were written canonically as <system>/kv_cache/llama3.1-8b/run/<timestamp>/. At packaging time three system directories were merged into one, each run/<timestamp>/ pair was collapsed into a single dir renamed after its topology, and the model dir was renamed llama3.1-8b → llama3-8b-10u.

The same submitter packages aurora-flare, aurora-flare-n8, aurora-flare-n64 and aurora-flare-n128 correctly — four system directories with four .yaml and four .pdf. This is the only submission in the tree with the flattened shape; the other 29 kv_cache runs are canonical.

2. Accepting the layout would not fix the collapse

Worth stating explicitly, since it is the obvious first instinct. Detection would be easy and reliable — the metadata JSON is self-describing and never needs the directory name, which is why extraction already succeeds. But systemname is derived from the on-disk path, so all three runs still produce the grouping key (closed, ANL, crux-eagle, llama3.1-8b, throughput) and still collapse to one row. Accepting the layout would turn the empty leaf results.csv into a populated one and go on publishing one row of three.

Three correct rows requires three system directories, and crux-eagle-n8.{yaml,pdf} / crux-eagle-n64.{yaml,pdf} were not submitted — only crux-eagle.{yaml,pdf}. Two of the three runs have no system description behind them and cannot be published as systems regardless of what the tool does.

3. The two dropped runs failed — "nothing is missing from the submission" was wrong

The original report checked directory-level presence. The runs' own summary.json records otherwise:

Run	partial_failure	missing per-rank result files	mpirun trials with non-zero exit
1nodex8ppn	false	0 / 24	0 of 9
8nodex8ppn	true	403 (~70%)	9 of 9
64nodex8ppn	true	4473 (~97%)	9 of 9

Every mpirun trial in both dropped runs returned non-zero. Their headline figures are fmean over the 3–30% of ranks whose result files survived — which is what the 48.57 tok/s and 0.0 bandwidth values in the original comparison table actually are.

So the accidentally-published row is the only one of the three that should be published. The selection was still wrong — runs[0] is filesystem-discovery order, not a rule — but the outcome was right by luck.

4. The generalizable bug: partial_failure is written and never read

partial_failure and trial_failures are written by _write_run_summary (mlpstorage_py/benchmarks/kvcache.py:919,960) and read by nothing — not reportgen, not the submission checker. A run that lost most of its ranks publishes a row indistinguishable from a clean one.

Tree-wide, 30 kv_cache runs, 3 with partial_failure: true. The third is not ANL's:

open/farmgpu/results/potato8-qlc-np6-open-tnf-70b/kv_cache/llama3.1-70b-instruct/run/20260709_215001
  partial_failure: true, 45 missing rank result files, 0 failed trials

That one is in canonical layout, groups cleanly, and is published today with no indication that a third of its rank data is absent. It is not caught by anything in this issue's original framing.

Revised fix

1. Make the N→1 selection deterministic and annotate the published row — the issues column should state that the row came from 1 of N runs and name the ones dropped, rather than the current bare No valid run directories found against the workload directory.
2. Surface partial_failure / trial_failures as a warning on any kv_cache row built from a run that recorded them, with the missing-file and failed-trial counts.

Not proposing to accept the flattened layout: it would not fix the collapse, and the two runs it would recover are failed runs that lack system descriptions. That stays a submitter-side correction and remains open question 3 for the WG.