Skip to content

grpc: close the attach window where an executor negotiates no lease at all #947

Description

@pratyush618

Follow-up to #932, which was fixed by its option 1. Option 2 — "install the lease book
before the door accepts a stream"
— was left open because it is not available as written.
Surfaced by the Go executor e2e suite in #928.

What #932 fixed, and what it left

#932 made the check and the advertisement come off one decision: a capability is in force
only where the executor advertised it and that attach's hello_ack carried it. Frames
are no longer dropped, and the contract sentence is true as written.

What it does not do is make the window stop existing. An executor that attaches before this
process wins the scheduler role is acknowledged without lease, is dispatched without one,
and is fenced on (owner, attempt) alone for the life of that connection. That is exactly
the give-up CAP_LEASE
documents for a peer that does not negotiate the capability, and it is self-healing —
FLEXIQ_GRPC_EXECUTOR_STREAM_MAX_AGE rotates the stream (1800s by default) and the next
attach negotiates leases. So it is a real gap, not a bug.

The gap is requeue_stuck-shaped: while it is open, a dispatch of a job made under one
claim cannot be told from a later dispatch made under a new claim, so an operator
unsticking a job the executor is still running gets two attempts that can both report.

Why the ordering does not allow the obvious fix

The book belongs to the Scheduler. Scheduler::new creates it
(crates/flexiq-core/src/scheduler/mod.rs), worker/runner.rs hands it to the pool via
WorkerDispatcher::set_lease_book, and in flexiq-server the Scheduler only exists once
SchedulerSupervisor wins the role — long after listener::spawn and the gRPC service start
accepting (crates/flexiq-server/src/runtime/mod.rs). There is no point in the current
ordering at which the door could wait for a book.

Proposal: invert who owns the book

Have RemoteDispatcher::new mint a LeaseBook and always advertise CAP_LEASE, and have
the Scheduler adopt the dispatcher's book instead of creating its own. Nothing can be
dispatched before a scheduler exists, so a dispatcher that holds a book from construction is
not promising anything it cannot do.

Two things this buys:

  • Every attach negotiates leases, whatever the election timing. The window closes rather
    than being tolerated.
  • One book spans scheduler-role restarts. Today each generation builds a fresh book, so a
    straggler from the previous generation meets an empty one and is accepted on the storage
    fence alone. A book that outlives the generation keeps it fenced.

Places it touches:

  • crates/flexiq-core/src/scheduler/mod.rsScheduler takes a book rather than
    constructing one.
  • crates/flexiq-core/src/worker/mod.rs — the WorkerDispatcher trait, where
    set_lease_book becomes an adopt-or-offer shape. The default no-op behaviour for pools
    that do not fence (see crates/flexiq/src/pool.rs) has to survive.
  • crates/flexiq-core/src/worker/runner.rs — the wiring at the set_claim_owner /
    set_lease_book pair.
  • crates/flexiq-server/src/runtime/mod.rs and the supervisor — the server's construction
    order.
  • crates/flexiq-core/src/worker/http_target/mod.rs — unaffected in behaviour (a push target
    negotiates nothing), but it implements the same trait method.

Worth confirming while doing it: LeaseBook::retire/forget are already guarded against a
straggler evicting a newer dispatch's entry, which is what makes a longer-lived book safe.

Also left undone

#932's fix documented the rule in worker/protocol.rs and
REMOTE_SDK_CONTRACT.md,
but not in contracts/proto/flexiq/executor/v1/executor_service.proto. The descriptor
embeds proto comments, so a comment-only edit there makes contracts/descriptor.binpb stale
and needs scripts/proto-check.sh --fix with the pinned buf. Cheap to fold into any change
that already regenerates the descriptor — the two spots are HelloAckFrame.capabilities and
JobFrame.lease.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    area/wireNetwork contract: proto, gRPC, JSON facadeenhancementNew feature or request

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions