Repository navigation
Read fcvm-coordinator-release logs for the three failed staging deploys of 2026-10-26 #696
Description
Please read the /aws/lambda/fcvm-coordinator-release log group (us-west-1) for these invocations and report what each one printed: the final JSON line ({"status": ...}) and the exception type and traceback of any failure. This box cannot read those logs.
| fcvm run | invocation start (UTC, from the Duration metric) | duration |
|---|---|---|
| 40261310012 attempt 1 | 2026-10-26 03:06 | 309 s |
| 40265725881 attempt 1 | 2026-10-26 04:00 | 272 s |
| 40274465086 attempt 1 | 2026-10-26 05:51 | 320 s |
| 40273777374 (passed) | 2026-10-26 05:34 and 05:38 | 258 s and 2 s |
Also the throttled invocations in the minutes 03:11, 04:04 and 05:55 (2 throttles each), if they left anything.
In all three failed jobs, publish.py exited with {"status": "failed", "errorType": "RuntimeError"} while the invocation was still running, and the re-run returned "unchanged". The metrics suggest the CLI's call ended about 260 s in and its own retries were throttled by the function's reserved concurrency of 1, so I expect each of the three to have printed {"status": "deployed", ...}. If any printed "failed", the exception decides whether #695 covers it.
Related: #695 (failure result names the step and code; lock handling), ejc3/fcvm#1334 (publish.py retries an invoke that ends without a result).
Activity
ejc3 commentedon Oct 26, 2026
From
/aws/lambda/fcvm-coordinator-release(us-west-1), 03:00 to 06:05 UTC. There is no traceback or exception anywhere in that window; every invocation ended with a status line.deployed, task definition:25deployed,:28deployed,:32deployed:31, thenunchanged:31The re-runs at 04:09:41 and 05:57:53 returned
unchangedwith the same task definition (:28,:32). Each of these runs also has a{"status": "busy"}invocation about a minute before (03:05:46, 03:59:08), and 03:31 to 03:37 is a run of sixbusyreturns beforedeployed:26at 03:42:16.The throttled invocations (03:11, 04:04, 05:55) left nothing in the log group: a throttled invoke never reaches the function.
So the release itself succeeded every time. The
RuntimeErrorcame from publish.py's side of the call, as you expected: the CLI's invoke ended about 260 s in, and its retries were throttled by the reserved concurrency of 1. ejc3/fcvm#1334 is the matching fix; #695 does not need to cover a failed release here.