[matthewberry]

Stacked on #160. This changes app/postjob.py after #160's upload handling, so merge #160 first. This PR's base then switches to main.

Problem

A CLEAN run that crashes is reported as completed. The user gets a "ready" email, and the results page shows an error because there is no _maxsep.csv.

The CLEAN command (app/services/clean_service.py) runs:

python CLEAN_infer_fasta.py … 2>&1 | tee $JOB_OUTPUT_DIR/log

Without pipefail, a pipeline's exit status is that of its last command, tee, which succeeds. So when the inference fails:

1. the || touch $JOB_OUTPUT_DIR/error fallback never runs;
2. the job template's && touch "${JOB_OUTPUT_DIR}/finished" (job.tpl.yaml) does run;
3. the job container exits 0, post-job uploads finished and log, and the watcher marks the job completed.

This happened to job f289fd27… on 2026-11-04. Its log ends with RuntimeError: torch.cat(): expected a non-empty list of Tensors. It left finished and log in MinIO, with no error and no _maxsep.csv. (Why that run crashed is a separate question. It isn't addressed here.)

CRISPR-Copies and Mutagenesis use the same | tee log pattern and already set pipefail. A word search for tee across the repo finds only these three commands, and CLEAN was the only one without it.

Why the Job is failed in post-job, not in the job container

Adding pipefail alone (this PR's first commit) makes the job container exit non-zero, but that loses the log:

• in job.tpl.yaml, job is an init container, and post-job (postjob.py, which uploads the output directory) is the pod's main container;
• Kubernetes starts main containers only after every init container succeeds, so post-job never runs;
• error and log stay on the shared volume and never reach MinIO. The f289fd27 diagnosis above was only possible because its log was uploaded.

So a failed inference writes error and exits 0 instead, and post-job fails the Job after uploading.

Change

• clean_service.py: prefix the command with set -o pipefail && , and change the fallback from || (touch error && false) to || touch error. A failed inference now writes error and exits 0; the template then touches finished.
• postjob.py: after the upload, exit 1 if $JOB_OUTPUT_DIR/error exists. With restartPolicy: Never and backoffLimit: 0, the Job fails; the watcher marks the job error and sends the "failed" email.
• If the upload itself fails, fix: exit non-zero when postjob fails to upload a job's outputs #160 already exits 1 before this check, so the job is still marked error.
• Other tools: CRISPR-Copies and Mutagenesis still exit non-zero from the job container (&& false), so their behavior is unchanged. Tools that never write error are unaffected.
• A failed CLEAN run now uploads error, finished and log. Nothing in app/ reads finished or error from MinIO (grepped), but I haven't checked the frontend.

Deploy note

post-job always runs moleculemaker/mmli-backend:main (hardcoded in job.tpl.yaml, imagePullPolicy: Always), whatever backend version created the Job. A backend running this branch's CLEAN command against a :main image without the new postjob.py check would mark failed runs completed. Don't deploy this branch to an environment before :main contains the postjob.py change.

Testing

• tests/test_clean_job_exit_code.py:
  • What it runs: it renders the real job template through create_job with the real build_clean_job_command output, as test_job_shm.py does. It then runs the job init container's bash -c script, with a stand-in python on PATH.
  • Inference exits 1, or is killed with SIGKILL as an OOM kill would do: the container exits 0, and the output directory holds error, finished and log.
  • Inference succeeds: the container exits 0, with _maxsep.csv, finished and log (no error).
  • Without pipefail: both failure cases fail (no error).
• tests/test_postjob_exit_code.py (from fix: exit non-zero when postjob fails to upload a job's outputs #160) gains one case: with error present, the upload runs and then the script exits 1. It fails with the check removed.
• Full suite in the Dockerfile.test image: 630 passed.
• Not run against a real CLEAN image or on the cluster.

[matthewberry]

Review of 829ea8d. No blocking findings.

Ran: Dockerfile.test image built from this branch. The 7 tests in test_postjob_exit_code.py and test_clean_job_exit_code.py pass, and the full suite passes (630). Not run against a real CLEAN image or on the cluster.

Checked and cleared

• pipefail reaches the pipeline. The template's folded bash -c script is ls -al ${JOB_INPUT_DIR} && set -o pipefail && … | tee …. Because it's all one && chain in a single shell, the option is set before the | tee pipeline runs.
• A failed setup step still fails the container. If echo | base64 -d > … fails, the outer && skips the whole group, || touch error included. The job container exits non-zero, post-job never runs, and the job is marked error, as before this PR.
• Either merge order is safe. If :main has the new postjob.py check but a backend still sends the old CLEAN command, the old command's (touch error && false) fails the init container, so nothing is wrongly marked completed. The only risky order is the one your deploy note already covers.
• Only these three commands write error. git grep over app/services/*_service.py and chart/ finds error written only by CLEAN, CRISPR-Copies and Mutagenesis. The latter two still end with && false, so post-job never runs for them.
• No frontend reads finished or error. I searched src/ in clean-frontend (4730799), crispr-copies-frontend (eb80673) and mutagenesis-frontend (9ba1bbf) for those names, /out/ and artifacts. The only reads of job output are results fetches gated on completed. This settles the "I haven't checked the frontend" item in the description.

Notes

• Deploy Fix failed jobs polling forever: check for 'error' status, not 'failed' clean-frontend#159 first. On clean-frontend main, the results page stops polling only for 'completed' or 'failed', and the backend sends 'error' (results.component.ts:144,153). Today a crashed CLEAN run is marked completed, and the page ends up showing something. After this PR the same run is error, and the page polls forever with no failure message. fix: link crispr-copies, mutagenesis and ez-specificity notifications to their result pages #159 fixes that, and it now also shows the failure message when loading a completed job's results fails (d111b3e), which covers jobs that were wrongly marked completed before this PR.
• The error check in postjob.py applies to every job type, not just CLEAN. fix: keep failed CRISPR-Copies and Mutagenesis logs by failing the Job in post-job #169 and fix: a failed novostoic run was reported as completed with no results #171 build on that, and fix: a failed novostoic run was reported as completed with no results #171 adds a comment in chart/values.yaml stating the rule. A tool image that writes its own top-level error file from inside its code would now fail its job. That can't be ruled out by searching this repo, so "tools that never write error are unaffected" holds only for the commands this repo defines.
• Stack: fix: keep failed CRISPR-Copies and Mutagenesis logs by failing the Job in post-job #169 and fix: a failed novostoic run was reported as completed with no results #171 are both based on this branch. If fix: exit non-zero when postjob fails to upload a job's outputs #160 is squash-merged, this branch and both of those will need a rebase. fix: keep failed CRISPR-Copies and Mutagenesis logs by failing the Job in post-job #169 and fix: a failed novostoic run was reported as completed with no results #171 also both edit chart/values.yaml, in different sections; I didn't test-merge them.