Skip to content

Add StabiliNNator matrix coverage (S01-S03) - #117

Merged
wilke merged 2 commits into
mainfrom
feat/matrix-stabilinnator-coverage
Aug 25, 2026
Merged

wilke merged 2 commits into
mainfrom
feat/matrix-stabilinnator-coverage

Conversation

@wilke

@wilke wilke commented Aug 24, 2026

Copy link
Copy Markdown
Contributor

StabiliNNator ships in the same container as PredictStructure but had zero matrix coverage, so 59/59 passed silently meant one of the two apps. Its first ever real submission failed at dispatch — tasks 23450684 and 23450690 — long after folding_260821.3 was deployed and reported green.

  • submit_job takes a per-case app key, defaulting to PredictStructure
  • S01/S02/S03 cover analysis_type both / proline / disulfide
  • the --include-alphafold filter no longer assumes every case has a tool key (it crashed with KeyError: 'tool' on the first non-PredictStructure case)
  • 1crn_small.pdb staged into the workspace inputs/ folder so the cases follow the existing path convention

S01 currently FAILS against production, and that is the correct result — it reports the partition => 'normal' dispatch bug that the matrix previously could not see. It should stay red until App-StabiliNNator.pl is fixed.

Companion change in runtime_build (56b46ff): the acceptance harness now asserts the content of policy_data.partition rather than just rc=0, and goes red on the current container while PredictStructure passes the same check.

612 passed, 10 skipped.

wilke and others added 2 commits August 24, 2026 13:28
StabiliNNator ships in the same container but had NO matrix coverage,
so '59/59 passed' silently meant one of the two apps. Its first real
submission failed at dispatch (tasks 23450684, 23450690) long after
folding_260821.3 was deployed and declared green.

- submit_job takes a per-case 'app' key, defaulting to PredictStructure
- S01/S02/S03 cover analysis_type both/proline/disulfide
- the --include-alphafold filter no longer assumes every case has a
  'tool' key
- 1crn_small.pdb staged in the workspace inputs/ folder

S01 currently FAILS against production, which is correct: it reports
the partition => 'normal' dispatch bug the matrix could not see before.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DEWo4xvL59PHC1QUvtTV9F
status == completed is not evidence a job produced anything.
StabiliNNator reported success while uploading its results flat into
output_path instead of the job folder, leaving output_files empty
(tasks 23452794-23452798, CEPI-dxkb/stabiliNNatorApp#18) -- and S01-S03
went green on a job whose own folder was empty and whose report the UI
cannot find. That is the same 'rc=0 is not enough' weakness this suite
was added to close.

- judge() now fails a completed job with empty output_files, and honours
  an optional expect_output suffix (S01-S03 require *_report.html)
- fetch_output_files reads the job object via p3-cat, falling back to
  the folding container since p3-cat is not on the host PATH -- without
  that fallback the helper returned None every time and the assertion
  never ran, which it did on the first attempt
- unreadable (None) is distinguished from empty ([]), so a workspace
  hiccup is not reported as a missing-output bug

Verified both directions: S01 now FAILS with 'completed but no
output_files'; E01 and B01 still PASS with real outputs.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DEWo4xvL59PHC1QUvtTV9F
@wilke

wilke commented Aug 25, 2026

Copy link
Copy Markdown
Contributor Author

Updated — the cases now assert delivered outputs, not just status == completed.

Running S01-S03 against folding_260824.2 exposed why that matters. All three jobs completed on coconut in 12-16 s with success: 1, and the original assertion reported PASS. But the job folders were empty and output_files was []: StabiliNNator uploads its results flat into output_path instead of the job folder (filed as CEPI-dxkb/stabiliNNatorApp#18). The report exists — a 611 KB 1crn_small_report.html — just not where the workspace or the UI can find it.

That is the same weakness I flagged in the acceptance harness earlier: asserting the exit code rather than the effect.

  • judge() fails a completed job whose output_files is empty, and honours an optional expect_output suffix — S01-S03 require *_report.html
  • fetch_output_files reads the job object with p3-cat, falling back to the folding container because p3-cat is not on the host PATH. Without that fallback the helper returned None on every call and the assertion silently never ran — which is exactly what happened on my first attempt, and S01 passed again for the wrong reason
  • unreadable (None) is kept distinct from empty ([]), so a workspace hiccup is not reported as a missing-output bug

Verified both directions: S01 now FAILs with completed but no output_files; E01 and B01 still PASS with real outputs. 612 unit tests pass.

Still holding this PR: S01-S03 are red, correctly, until stabiliNNatorApp#18 is fixed.

@wilke
wilke merged commit 8cf41df into main Aug 25, 2026
1 check passed
@wilke
wilke deleted the feat/matrix-stabilinnator-coverage branch August 25, 2026 18:33
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant