[josephjclark]

We have discovered an issue in the worker related to the work-available (wake up) alert.

The wake up event will instantly trigger a claim, skipping the workloop entirely.

Now, if there are runs in progress, the claim function will abort and say "sorry, I'm already at capacity".

The oversight was: there is no check as to whether or not a claim request has been sent out and is being served.

This means that, particularly if Lightning is being slow to respond, a worker could receive 50 wake up events and send out 50 claim requests before lightning replies. Suddenly the poor worker needs to process 50 runs at a time.

Now there is some mitigation around this in the engine - the engine will buffer its own little queue and ensure that it never has more than MAX_WORKERS running.

But it does mean that the worker can take way more than its share, and significantly increases the risk that a greedy worker could a) blow up or b) timeout a bunch of runs that are sitting in its internal queue.

So this PR adds logic to include outstanding claim requests in the server's capacity, essentially. When a worker makes a claim, it assumes that claim will be honoured and reserves space to satisfy the number of runs returned.

AI Usage

Please disclose how you've used AI in this work (it's cool, we just want to know!):

• Code generation (copilot but not intellisense)
• Learning or fact checking
• Strategy / design
• Optimisation / refactoring
• Translation / spellchecking / doc gen
• Other
• I have not used AI

You can read more details in our Responsible AI Policy

Release branch checklist

Delete this section if this is not a release PR.

If this IS a release branch:

• Run pnpm changeset version from root to bump versions
• Run pnpm install
• Commit the new version numbers
• Run pnpm changeset tag to generate tags
• Push tags git push --tags

Tags may need updating if commits come in after the tags are first generated.