fix(server): release the resident reranker or embedder before loading its replacement, then drain MLX's cache

[magicnight]

Partially addresses #130 (the cheap half; the pool-budget half stays open).

ensureRerankerLoaded and ensureEmbedderLoaded loaded the replacement engine and only then let the old one go, so both models were resident for the duration of the load. Both now release the current engine first, drain MLX's buffer cache through a new EngineMemory.releaseCachedBuffers() (so HummingbirdServer.swift keeps no MLX import), then load. A failed load therefore leaves no reranker or embedder resident; the next request for the previous model reloads it.

Measured

Same probe as in #130 (real HummingbirdServer, Memory.snapshot() after each /v1/rerank call), two model ids pointing at the same Qwen3-Reranker-0.6B-4bit weights, then a swap to ms-marco:

step	before: active / cache / peak	after: active / cache / peak
load 0.6B	320 / 291 / 591 MB	320 / 291 / 591 MB
swap to the same weights under a second id	320 / 611 / 639 MB	320 / 291 / 591 MB
swap back	320 / 611 / 639 MB	320 / 291 / 591 MB
swap to ms-marco	87 / 932 / 639 MB	87 / 5 / 591 MB

The 639 MB peak is the two 320 MB copies side by side; after the change the same serial probe never rises above its first request's 591 MB high-water mark. The 4B bf16 numbers from #130 (7.66 GB for one copy, 14.98 GB during a swap, 15.5 GB left in the cache) show the old order at scale; the 4B run was not repeated after the change.

Review rounds

• Bugbot found that the already-loaded check suspends the actor, so a concurrent swap could empty the slot under a same-model request that then re-read it and got a 500. Second commit: ensureEmbedderLoaded / ensureRerankerLoaded return the engine for the request and the handlers never re-read the slot. This also closes the mirror race main had since v0.5.3 (embedder) and v0.9.0 (cross-encoder): a re-read after another request's swap finished could pick up the other model's engine and answer under the requested model's name. Recorded as a Fixed entry.
• Critic: the swap ran outside the generation lock, so requests arriving during a swap would each load their own copy (including requests for the model being replaced), and two swaps could load at once. Third commit: the three handlers acquire the generation lock before the swap, as the chat model's handler always has (SRV-2); every early return releases it. The CHANGELOG now states exactly what was measured (serial requests; the 4B run was not repeated; the 591 MB figure is the first request's high-water mark) and says a chat generation waits behind a swap for the load's duration.
• New gated smoke (ms-marco under two ids, then Qwen3-0.6B-4bit, then back; real server): one copy 90.9 MB active, peak during the same-weights swap 100.2 MB (asserted below 1.5 copies); after the swap back 9.4 MB left in MLX's cache (asserted below 64 MB). Two mutations each turn it red: load-then-assign order → swap peak 181.7 MB (two copies); release without the drain → 646.5 MB left in the cache. Full pure suite 971/101 green; no drain on a failed load, because a load that fails before touching MLX must not be the first thing to initialize Metal (the SPM test job has no metallib).

Third round

• Lock contract tests (pure, swift test): a swap on /v1/rerank and on /v1/embeddings queues while a generation holds the lock and releases it after a failed load; the embeddings 404 and 400 do not wait for the lock. Four mutations each turn the expected test red: swap moved before the acquire, either failure path leaking the lock, the kind gate moved under the lock. A regression here is a server-wide deadlock, which is why the critic asked for these before merging.
• /v1/embeddings resolves the model and checks its kind before the lock, as the reranker path already did, so an unknown model or a chat model is answered at once even during a generation; ensureEmbedderLoaded takes the resolved target, the cosine branch reuses the one it had, and one loadFailureResponse maps a failed swap for all three handlers.
• A failed load drains the cache only when this call had just released a resident engine (so Metal is known to be up); a load that fails before touching MLX never initializes it, which the SPM test job depends on.
• Merged the new main (fix(detect): classify sequence-classification heads by what MLXRerankers can score, never as embedders #132): both ### Fixed entries sit under one header; the two test files keep both sides' additions. On the merged head the full pure suite is 980/102 green and all five gated smokes pass (ms-marco 1.9e-6 vs PyTorch, NLI 7.4e-6 relative, Qwen3 ordering, endpoint, swap order 100.2 MB peak against 90.9 MB one copy and 9.4 MB left in the cache).

[cursor]

Cursor Bugbot has reviewed your changes using high effort and found 1 potential issue.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit 1f2c9c5. Configure here.}

[cursor]

Swap 500s concurrent same-model requests

Medium Severity

Releasing embeddingEngine or rerankEngine before the replacement is assigned opens a nil window. A concurrent same-model request can pass the already-loaded check on a local current handle, then the handler re-reads the now-nil property and returns 500 load_failed for a model that was already resident.

Additional Locations (1)

• MacMLXCore/Sources/MacMLXCore/Server/HummingbirdServer.swift#L3813-L3820

 

^{Reviewed by Cursor Bugbot for commit 1f2c9c5. Configure here.}