Repository navigation
worker: can claim beyond its capacity? #992
Description
We've found a case where worker seems to have claimed ~30 runs, despite on a capacity of 2. This is naughty. This resulted in most of those runs being lost, although it's not clear to me whether the over-claiming is directly related to the losses.
link to the rush of claims
link to the rush of fails
There is no obvious cause as to why all these requests go out. I can only wonder if the worker has joined the queue multiple times, and as soon as the capacity was hit it triggered 30 claims at once. But I can't see any evidence of that?
Metadata
Metadata
Assignees
Labels
No labels
Type
Projects
- Status
Activity
josephjclark commentedon Sep 10, 2025
Struggling to reproduce
The claim queue runs in series and there are mitigations to prevent multiple loops starting. Spamming the server locally everything seems fine
But something must be triggering break this loop and send it into overdrive
josephjclark commentedon Sep 10, 2025
Looking into the logs and this is happening a fair bit: 17 occurences of 4/2 capacity in 48 hours GCP link
Feels like there are two claim requests, or two workloops, active for some reason.
josephjclark commentedon Sep 10, 2025
Oh no, it's the work available message!
This will force a claim so long as the worker isn't at capacity - but it doesn't take into account how many active claims there are
So if you send work available, and the server has capacity, it'll claim. If more work comes available while the claim is outstanding, it'll claim again. Thus you can have a whole suite of claims on the wing with nothing to constrain them.