[Bug] MLPStorage Validation Fails with "Expected 6 run files, but found 12" for RetinaNet on Both B200 and MI355

[xutao00090]

Issue Summary
When running mlpstorage validate on the aggregated RetinaNet results from both B200 and MI355 platforms, the validation tool reports a file count mismatch error. The tool expects 6 files but finds 12.

Scenario Context
We are conducting run validation for RetinaNet on two accelerator platforms:

B200 – 6 runs

MI355 – 6 runs

This results in a total of 12 runtime files (6 from each platform), which is correct and expected behavior. However, the validation script appears to assume a single-platform context and expects only 6 files, failing when it encounters the combined set of 12.

[ERROR] [2.1.17 runTimestamps] ./closed/xxxx/training/retinanet/run: Expected 6 run files, but found 12. Timestamps

[FileSystemGuy]

@xutao00090 Could you please copy in at least the topmost "Options Summary" section of the output., and ideally all the output, during this run? I'd like to make sure I know what options you provided to mlpstorage. I can believe there is a bug there, but it is hard to diagnose without more information.

[xutao00090]

Hi @FileSystemGuy, Our understanding of the SYSTEM_NAME is that it describes the storage system under test (e.g., the storage hardware topology, number of storage nodes, client nodes, network fabric, etc.), not the accelerator type used in the compute nodes. In our case, the SYSTEM_NAME represents the storage infrastructure (e.g., 3 storage nodes + 16 client nodes + network configuration), which remains the same regardless of whether the compute nodes are equipped with B200 or MI355 accelerators.

The accelerator type (B200 vs MI355) is part of the benchmark workload configuration, not the system identity. Therefore, we believe both sets of runs (6 from B200 + 6 from MI355 = 12 total) should legitimately reside under the same SYSTEM_NAME, as they are testing the same underlying storage system with different compute accelerators.

However, the submission checker (rule 2.1.17) expects exactly 6 run timestamps under each training/retinanet/run/ directory and does not account for this scenario.

Is there a different expected structure for submitting results tested with multiple accelerator configurations against the same storage system?

Our current submission tree:

# tree -d -L 2 closed/
closed/
└── XXXX
    ├── code
    ├── results
    └── systems

# tree -d -L 5 closed/XXXX/results/
closed/XXXX/results/
└── <SYSTEM_NAME>
    └── training
        └── retinanet
            ├── datagen
            │   └── 20260704_161351
            ├── datasize
            │   ├── 20260704_185700
            │   └── 20260705_005922
            └── run
                ├── 20260704_185703
                ├── 20260704_195712
                ├── 20260704_205737
                ├── 20260704_215803
                ├── 20260704_225825
                ├── 20260704_235845
                ├── 20260705_005924
                ├── 20260705_020425
                ├── 20260705_030915
                ├── 20260705_041415
                ├── 20260705_051912
                └── 20260705_062422

Operation:

• First, execute the RetinaNet test for 6 loops with parameters --accelerator-type b200 --results-dir /root/storage_results_v3
• Then, Run the RetinaNet test for 6 loops with the following parameters: --accelerator-type mi355 --results-dir /root/storage_results_v3
  So, it will generate 12 result directories under the "run/" directory.

I hope these details are helpful to you. If you need more information, please let me know clearly.

[FileSystemGuy]

Looking at this against the current code and Rules.md, there's a genuine spec ambiguity here — not just an implementation bug — and I think the reporter's tree layout is defensible under one reading of the rules.

Where the checker enforces the "6 exactly" rule

mlpstorage_py/submission_checker/checks/directory_checks.py:250 (rule 2.1.17 run_files_timestamp_check) does a flat len(timestamps) != RUN_TIMESTAMP_COUNT where RUN_TIMESTAMP_COUNT = 6 (see submission_checker/constants.py:147). It never partitions the timestamps by accelerator type recorded in each run's *_metadata.json. So under the current implementation, any run/ directory with more than 6 timestamps is a hard violation regardless of whether the extra runs represent a legitimate second accelerator configuration against the same storage system.

Rules.md is internally inconsistent on this point

• §2.1.17 (prose): "there must also be exactly 6 subdirectories" — no per-accelerator carve-out.
• §2.1.27 (directory diagram): annotates the run/ tree for both unet3d and retinanet with ... (5x Runs per Emulated Accelerator Type) (Rules.md lines 159 and 170). That phrasing directly implies a valid tree may contain N × 6 timestamps where N = number of distinct accelerator types tested under the same SYSTEM_NAME.
• §2.1.9 (identicalSystemConfig): ties SYSTEM_NAME to the system-under-test being identical across the tree, but the rule is silent on whether the compute-side accelerator type is part of "system identity" or is a workload knob.

The reporter's reading — SYSTEM_NAME = storage system (nodes, network, fabric), accelerator type = workload configuration — is consistent with the §2.1.27 diagram annotation. The current checker enforces the strict §2.1.17 prose reading, which contradicts the diagram.

Suggested resolution shape (pending WG guidance on which reading is authoritative)

If the diagram's "per Emulated Accelerator Type" reading is authoritative:

1. Change run_files_timestamp_check to group self.submissions_logs.run_files by each run's recorded accelerator_type from *_metadata.json, then require each group to have exactly 6 timestamps. Violation message should name the offending accelerator-type group.
2. Update rule 2.1.18 (runTimestampGap) to enforce contiguity within each accelerator-type group, not across the entire flat set — otherwise a B200 batch followed hours later by an MI355 batch will trip the gap check even when each individual group is contiguous.
3. Reconcile §2.1.17's prose ("exactly 6") with §2.1.27's diagram ("5x Runs per Emulated Accelerator Type") — pick one and make the other match.

If the strict prose reading is authoritative, the fix is on the submitter side (separate SYSTEM_NAME per accelerator, or single-accelerator submissions), and §2.1.27's diagram annotation should be removed to avoid future confusion.

Either way, the spec question needs to be answered before the code path can be fixed cleanly.

[Raysmond]

@FileSystemGuy I also found this issue. The rootcause is the benchmark generates two run folders for each run. I check the files, the first folder is empty and the second looks valid. So you will find 12 run folders after 6 actual runs. You can perform a quick run to reproduce this.
The issue not only generates an empty folder first for each run, but also generates empty folders for datasize and datagen steps.

[xutao00090]

Looking at this against the current code and Rules.md, there's a genuine spec ambiguity here — not just an implementation bug — and I think the reporter's tree layout is defensible under one reading of the rules.

Where the checker enforces the "6 exactly" rule

mlpstorage_py/submission_checker/checks/directory_checks.py:250 (rule 2.1.17 run_files_timestamp_check) does a flat len(timestamps) != RUN_TIMESTAMP_COUNT where RUN_TIMESTAMP_COUNT = 6 (see submission_checker/constants.py:147). It never partitions the timestamps by accelerator type recorded in each run's *_metadata.json. So under the current implementation, any run/ directory with more than 6 timestamps is a hard violation regardless of whether the extra runs represent a legitimate second accelerator configuration against the same storage system.

Rules.md is internally inconsistent on this point

• §2.1.17 (prose): "there must also be exactly 6 subdirectories" — no per-accelerator carve-out.
• §2.1.27 (directory diagram): annotates the run/ tree for both unet3d and retinanet with ... (5x Runs per Emulated Accelerator Type) (Rules.md lines 159 and 170). That phrasing directly implies a valid tree may contain N × 6 timestamps where N = number of distinct accelerator types tested under the same SYSTEM_NAME.
• §2.1.9 (identicalSystemConfig): ties SYSTEM_NAME to the system-under-test being identical across the tree, but the rule is silent on whether the compute-side accelerator type is part of "system identity" or is a workload knob.

The reporter's reading — SYSTEM_NAME = storage system (nodes, network, fabric), accelerator type = workload configuration — is consistent with the §2.1.27 diagram annotation. The current checker enforces the strict §2.1.17 prose reading, which contradicts the diagram.

Suggested resolution shape (pending WG guidance on which reading is authoritative)

If the diagram's "per Emulated Accelerator Type" reading is authoritative:

1. Change run_files_timestamp_check to group self.submissions_logs.run_files by each run's recorded accelerator_type from *_metadata.json, then require each group to have exactly 6 timestamps. Violation message should name the offending accelerator-type group.
2. Update rule 2.1.18 (runTimestampGap) to enforce contiguity within each accelerator-type group, not across the entire flat set — otherwise a B200 batch followed hours later by an MI355 batch will trip the gap check even when each individual group is contiguous.
3. Reconcile §2.1.17's prose ("exactly 6") with §2.1.27's diagram ("5x Runs per Emulated Accelerator Type") — pick one and make the other match.

If the strict prose reading is authoritative, the fix is on the submitter side (separate SYSTEM_NAME per accelerator, or single-accelerator submissions), and §2.1.27's diagram annotation should be removed to avoid future confusion.

Either way, the spec question needs to be answered before the code path can be fixed cleanly.

@FileSystemGuy
What was the final conclusion of the discussion on this issue? Is it necessary to make changes at the code level? Or is it about modifying the submission rules?