Repository navigation
CI: cancel a failed job's own Slurm sims, shared rodinia data, tolerate ReqMem '16?' - #572
Merged
Merged
Conversation
…te ReqMem '16?' - A sim job that fails or is cancelled now scancels its own Slurm jobs (tag gh-run:<run>.<attempt>:<job>) before its runner is reused; the next job's checkout wiped the run dirs and the orphans segfaulted. - PTX-Simulation links the shared gpgpu-sim-appdata data_dirs instead of downloading rodinia_2.0-ft, whose script exits 0 on a truncated download. - common._mem_kb returns None for an unparseable ReqMem (a just-submitted job showed '16?' and crashed the monitor on its first poll). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
JRPan
added a commit
that referenced
this pull request
Oct 21, 2026
…te ReqMem '16?' (#572) - A sim job that fails or is cancelled now scancels its own Slurm jobs (tag gh-run:<run>.<attempt>:<job>) before its runner is reused; the next job's checkout wiped the run dirs and the orphans segfaulted. - PTX-Simulation links the shared gpgpu-sim-appdata data_dirs instead of downloading rodinia_2.0-ft, whose script exits 0 on a truncated download. - common._mem_kb returns None for an unparseable ReqMem (a just-submitted job showed '16?' and crashed the monitor on its first poll). Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes for the orphaned CI sims seen in run 39530862942. The PTX monitor crashed, its Slurm jobs kept running, and the next job's checkout on the same runner deleted their run dirs, so they segfaulted with
Stale file handle.gh-run:<run>.<attempt>:<job>. On failure or cancel, each job scancels its own jobs (.github/scripts/cancel-slurm.sh) before the runner is released.cleanup-slurmstill runs at the end of the workflow and matches the run tag as a prefix./scratch/.../for-sharing/gpgpu-sim-appdata/data_dirs, the same copy the functional CI uses, instead of runningget_regression_data.sh. That script downloads from engineering.purdue.edu, which was failing SSL, and exits 0 on a truncated tarball.common._mem_kbreturns None for a ReqMem it can't parse. sacct printed16?for a just-submitted job, and the monitor crashed on its first poll.🤖 Generated with Claude Code