Worker: appears to be over subscribing claims #1041

The worker is over-subscribing claims and using its own internal queuing to defer runs.

smoking gun in GCP

This should not be happening. There are a few problem areas:

1. If the engine is given 3 tasks on capacity 2, the third task won't complete until one of the others does. If the first 2 tasks take an hour, then the run is guaranteed lost.
2. It looks like in this Lost run, the Run was deferred but something when wrong when trying to restore it. The deffered run exists immediately, having done nothing. This cant' be happening all the time or we'd have way more lost runs, right?
3. I suspect that deferred runs like this won't prevent shutdown on sigterm (but I haven't checked)