feat(vmcp): stop session init stalling on backends that never service the notification stream

[aron-muon]

Summary

Background: vMCP opens a persistent connection to every backend during session initialize, and part of that is subscribing to the backend's list_changed notifications. A backend that accepts the subscribe and then never services it stalls the handshake, and clients with their own connect timeout give up before it resolves. Two separate things make that unrecoverable rather than merely slow:

• the subscribe isn't avoidable in practice. createMCPClient only sets WithContinuousListening() when the sink is non-nil, and the comment there is explicit - "some backends hang when this stream is opened against them (Consume backend notifications in vMCP and propagate list_changed #5748 R3), so it must stay opt-in" - but server.go builds a sink for every session, so that gate can't be reached from a running deployment
• the wait isn't boundable. WithBackendInitTimeout exists but nothing outside tests ever called it, so init was pinned to the 30s default, and operational.timeouts only ever raises it (if requestTimeout > initTimeout)

This PR does both halves, because they're the same bug from two ends and they touch the same files:

• operational.listChanged with enabled and disabledWorkloads, plus session.WithListChangedFilter, a per-workload predicate in the same shape as the resolvers already on the factory. Returning false drops the sink for that backend alone, which is what stops the connector opening the stream. This removes the dependency
• operational.timeouts.backendInit, wired to the existing option, and an explicitly configured value is now authoritative so a longer workload request timeout no longer extends it. This bounds what's left
  • the two are genuinely different budgets. A heavy query legitimately wanting 60s shouldn't also grant 60s to a handshake, and that coupling is what made the knob useless here
• defaults are unchanged throughout. No config means every backend is subscribed and init keeps the 30s allowance, exactly as today

An excluded backend loses live propagation and nothing else. Its tools are still aggregated and callable, and they refresh on the next session.

I originally sent these as #6633 and #6635. Folded them together after they conflicted with each other - they share 7 files including the generated CRDs, and whichever landed first would have forced a rebase and full regeneration of the other. #6635 is closed in favour of this.

Type of change

• New feature

Test plan

• Unit tests (task test)
• Linting (task lint-fix)
• Manual testing (describe below)

go test ./pkg/vmcp/... and the operator unit tests pass. golangci-lint run ./pkg/vmcp/... is clean at 0 issues. Regenerated with task operator-generate, task operator-manifests and task crdref-gen; task gen and task license-check leave the tree clean. I also ran the CRD schema check locally with the same SHA-pinned crd-schema-checker and flags CI uses, against the v0.48.0 baseline: exit 0, Compatible.

New tests. TestSessionFactory_ListChangedFilter asserts the connector receives a sink for subscribed backends and nil for excluded ones, since that nil is the whole mechanism, plus a case proving a nil-sink caller is unaffected. TestListChangedFilter and TestBackendInitTimeout cover the config to behaviour mapping, including enabled: false winning over an exclusion list. TestSessionFactory_ExplicitBackendInitTimeoutIsNotExtended covers the new precedence and TestSessionFactory_DefaultBackendInitTimeoutIsExtended keeps the old one under test.

Manual: hit this on 0.47.1 against Grafana Cloud's hosted MCP, which is stateless and mandates per-user auth. Its own audit log shows the handshake succeeding and then the stream sitting there until the client gives up at exactly 30s:

12:45:01  detected initialize method call
12:45:01  upstream response received  status: 200  mcp_session_id: mcp-session-6ae468e9-...
12:45:01  AUDIT mcp_initialize  outcome: success
12:45:01  outbound request to upstream  GET /mcp  accept: text/event-stream
12:45:31  http: proxy error: context canceled
12:45:31  AUDIT sse_connection  outcome: error

vMCP's own log stops after creating session-scoped backends and never reaches the Failed to initialise backend warning, because the client has already gone.

Worth noting why that backend can't dodge this on its own: it's Modern, so initOneBackend would skip the connect entirely, but the revision probe runs unauthenticated from the health monitor, gets a 401, and probeRevision quite reasonably leaves a 401 uncached as a transient auth blip. So it never classifies, and because no session ever completes, no authenticated call reaches dispatch to warm the cache either.

API Compatibility

• This PR does not break the v1beta1 API, OR the api-break-allowed label is applied and the migration guidance is described above.

listChanged and timeouts.backendInit are new optional fields. Nothing is removed or retyped. disabledWorkloads carries listType=set for the SSA tag the checker requires, matching MCPGroup.Servers.

Does this introduce a user-facing change?

Yes. Two new optional settings under operational: listChanged, to exclude backends from live list_changed propagation, and timeouts.backendInit, to cap how long session initialization waits for a single backend. Unset keeps today's behaviour.

Special notes for reviewers

Two calls worth your attention.

I made list_changed exclusion config rather than automatic on purpose - I couldn't find a signal that reliably separates "will never service this stream" from "is just slow right now", and guessing wrong silently drops propagation. If you'd rather infer it, or put the switch somewhere other than operational, happy to move it.

The precedence flip is deliberate: an explicit WithBackendInitTimeout is now a cap rather than a floor. TestNewSessionFactory_WorkloadTimeoutExtendsBackendInit asserted the old contract, so I split it in two rather than delete it. If you'd rather keep the old precedence and have the config field only bite alongside a lowered timeouts.default, say so and I'll rework it, though that combination can't express "generous request budget, tight handshake budget" which is the case I need.

The deeper fix is presumably for the stream to not gate initialize at all, but that sits in mcpcompat/go-sdk rather than here, so this is the smallest thing that makes session init independent of it.

[codecov]

Codecov Report

❌ Patch coverage is 92.85714% with 3 lines in your changes missing coverage. Please review.
✅ Project coverage is 78.92%. Comparing base (e532cf0) to head (d3c27ee).

Files with missing lines	Patch %	Lines
pkg/vmcp/cli/serve.go	89.28%	3 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #6633      +/-   ##
==========================================
- Coverage   78.98%   78.92%   -0.06%     
==========================================
  Files         782      782              
  Lines       78065    78099      +34     
==========================================
- Hits        61658    61639      -19     
- Misses      16402    16455      +53     
  Partials        5        5              

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[aron-muon]

Heads up on the two red checks here, they aren't from this branch.

Lint and Tests fail on pkg/authserver/runner, which this PR doesn't touch:

pkg/authserver/runner/embeddedauthserver_test.go:1689:8: undefined: miniredis (typecheck)

That reproduces on a clean checkout of main at bb4c185 - TestCreateStorage_NoAuthRedisConnects from #6551 calls miniredis.RunT without importing it, so the package fails typecheck and takes its whole test binary with it. This branch inherited it by rebasing onto current main.

Put the one-line fix up as #6636. Once that lands I'll rebase and these should go green.

[jhrozek]

Automated review pass over the list_changed exclusion + authoritative backendInit timeout changes. Design and overall test coverage look solid; two items below, one of which (the data race) will fail task test under -race as currently written.

[aron-muon]

Thanks, both fixed, and rebased onto current main now that #6636 has landed.

Data race - you're right, and it fails exactly as you describe. My mistake: I ran go test ./pkg/vmcp/... without -race, so I never saw it. Reproduced first:

--- FAIL: TestSessionFactory_ListChangedFilter/filter_drops_only_the_excluded_backend
    testing.go:1712: race detected during execution of test

Went with a mutex-guarded recorder rather than sync.Map, since the assertion wants the whole map compared at once and a snapshot under the lock reads better than ranging a sync.Map. Followed your pointer to factory_dialcontrol_test.go for the precedent of synchronising these concurrent-connector assertions. Clean now:

go test -race -run TestSessionFactory_ListChangedFilter ./pkg/vmcp/session/...
ok      github.com/stacklok/toolhive/pkg/vmcp/session   2.086s

I've since run the whole of ./pkg/vmcp/... and the operator packages under -race too, both clean.

Misplaced GoDoc - also correct, and a self-inflicted one: I anchored the insertion on the func line, which dropped listChangedEnabled in between isKnownModern's comment and isKnownModern itself. Each has its own comment back.

One thing worth flagging that isn't from this branch: golangci-lint reports 2 gci issues in pkg/authserver (server/provider.go:422, server_impl.go:195). Those reproduce on a pristine checkout of main at 2bb9996, so they're not from here, but they may bite the lint job.

[aron-muon]

The one remaining red check here (Tests) is also not from this branch.

It's a pre-existing -race failure in pkg/vmcp/server: TestPeriodicStatusReporting_ReactsToVersionChange assigns the package-level versionPollInterval while marked t.Parallel(), and Server.Start reads that variable for reconcileSessionsOnRegistryChange (#6549). The detector then fails the whole binary, so TestListChangedSink_EndToEnd_*, TestAuthzCallGate and a couple of others go red alongside it.

I checked before blaming main - it reproduces on a pristine checkout at 2bb9996, 5 failures, deterministic:

go test -race -count=1 -run 'TestListChangedSink_EndToEnd|TestAuthzCallGate|TestReadinessEndpoint_DynamicMode_CacheNotSynced|TestServeHandlerMetricsOnTransportPort|TestPeriodicStatusReporting_ReactsToVersionChange' ./pkg/vmcp/server/

Fix is up as #6644. With it the same filter is 0 failures and go test -race ./pkg/vmcp/server/... passes. I'll rebase here once it lands.

[aron-muon]

Correction on the gci thing, and you were right about the toolchain bumps - just not in the direction I reported. Please disregard it, there's nothing to fix.

It was my linter, not the tree. I was running a golangci-lint built with Go 1.26 while the project now targets 1.27 (#6639). Rebuilt with the right toolchain and pointed at current main:

$ GOTOOLCHAIN=go1.27.0 go install github.com/golangci/golangci-lint/v2/cmd/golangci-lint@v2.13.2
$ golangci-lint run ./pkg/authserver/...
0 issues.

The stale binary doesn't even load the config on current main any more - the Go language version (go1.26) used to build golangci-lint is lower than the targeted Go version (1.27) - which is what tipped me off. Sorry for the noise, I should have checked my own tooling against the pin before reporting it. mockgen had the same problem locally, same fix.

Meanwhile I've rebased this onto current main (7e52ab9, so Go 1.27) and re-verified with correctly built tooling: golangci-lint ./pkg/vmcp/... 0 issues, go test -race ./pkg/vmcp/... and the operator packages clean, all codegen (operator-generate, operator-manifests, crdref-gen, gen) leaves the tree untouched, and crd-schema-checker is Compatible against the v0.48.0 baseline.

The -race failure in pkg/vmcp/server is still real and still on main, though - I re-checked at 7e52ab9 and it reproduces (versionPollInterval mutated by a t.Parallel() test while Server.Start reads it). Same filter on this branch fails identically, so it isn't from here. #6644 is rebased onto current main and takes it to 0.

[aron-muon]

Go SDK / Verify generated Go SDK is also failing on main itself, not just here - the newest main run at 7e52ab9 has it red alongside Operator CI / Operator Tests Integration.

Two leftovers from the Go 1.27 bump: the job's setup-go reads sdk/go/go.mod (1.26.0) while sdk-verify runs root-module tooling that now needs 1.27, and the committed sdk/go/openapi.{json,yaml} plus the generated client are stale because the bump edited a BuilderImage doc comment that flows into them. Fix is up as #6645.

So the three red checks here are all main's, with fixes queued: #6644 (-race), #6645 (SDK), and Operator Tests Integration which I haven't looked at. Nothing outstanding on this branch itself as far as I can tell.