Skip to content

fix(router): free tier follows the gateway's free cascade - #44

Merged
VickyXAI merged 1 commit into
mainfrom
fix/free-tier-follow-gateway-cascade
Oct 20, 2026
Merged

VickyXAI merged 1 commit into
mainfrom
fix/free-tier-follow-gateway-cascade

Conversation

@VickyXAI

Copy link
Copy Markdown
Contributor

Why

#43 pinned all four Free tiers to nvidia/llama-3.2-11b-vision, because every other free id showed available: false. That flag is not a delisting. /v1/models sets it to isFreeModelUsable(model.id) (blockrun src/app/api/v1/models/route.ts:59), which is the gateway's per-instance free-health circuit breaker (src/lib/free-health.ts). It flips between instances and recovers by itself.

The gateway already handles fallback

The Go router has no per-request fallback, and adding one would only stack waits on top of the gateway's own.

  • Before dispatch: a requested free model whose breaker is open is rerouted to the first usable, capability-compatible rung of FREE_RESCUE_LADDER (chat/completions/route.ts ~1811, pickFreeRescue).
  • On failure: the call cascades nemotron-3-super-120b → nemotron-3-nano-omni-30b-a3b-reasoning → llama-3.2-11b-vision within a 90s budget (route.ts:394/422/452/465, 3463+).
  • Verified live today: a request for nemotron-3.5-lightning came back x-health-reroute: true and x-served-model: nvidia/nemotron-3-super-120b. A request for nano-omni came back served by nvidia/llama-3.2-11b-vision.

Pinning a primary is therefore safe, and the gateway's ladder supplies the fallbacks, ending at llama-3.2-11b-vision.

Free table

router-core (ae6990f) has no free profile; its only free rungs are the eco SIMPLE head. This table mirrors the sibling SDK's free profile, blockrun-llm router_adapter.py FREE_TIERS (a5af583):

Tier Primary Python SDK fallbacks (reference) Gateway reroute ladder
Simple nvidia/nemotron-3-nano-omni-30b-a3b-reasoning nemotron-3.5-lightning, llama-3.2-11b-vision, laguna-xs-2.1 super-120b → nano-omni → llama-3.2-11b-vision
Medium nvidia/nemotron-3.5-lightning nano-omni, laguna-xs-2.1, north-mini-code same
Complex nvidia/nemotron-3.5-lightning north-mini-code, nano-omni, llama-3.2-11b-vision same
Reasoning nvidia/nemotron-3.5-lightning nano-omni, north-mini-code, laguna-xs-2.1 same

The fix from #43 for hidden ids stays: nvidia/nemotron-3-super-120b (hidden:true in models.ts; it is a gateway-internal cascade rung) and moonshot/kimi-k2.x remain in the retired-id test, and Eco/Auto Simple are unchanged.

Other changes

  • TestRoutingTableAvoidsRetiredIDs no longer lists ids whose only problem is available: false.
  • The opt-in live test still fails on unlisted ids, and on free tiers that are not billed as free. A breaker-open routed id is now logged instead of failing the test.
  • The README free-model list and notes and the CHANGELOG Unreleased entry are updated.

Checks

  • go build ./...: ok
  • go vet ./...: ok
  • go test ./...: ok
  • BLOCKRUN_LIVE_CATALOG=1 go test -run 'TestRoutingTable|TestReadme|TestFreeRouting' .: ok. It logged that all four free primaries are breaker-open right now, which the gateway reroutes.

Known limitation

When the gateway reroutes, SmartChatResponse.Model still names the requested primary rather than the model that served the call. The served model is reported in the x-served-model header and the response model field.

available:false in /v1/models is the gateway's per-instance free-health
breaker (isFreeModelUsable), not a delisting, and the gateway already
reroutes a breaker-open free model along its ladder (super-120b ->
nano-omni -> llama-3.2-11b-vision). Free tiers now name the Python SDK's
FREE_TIERS primaries instead of hard-pinning llama-3.2-11b-vision. Hidden
ids (nemotron-3-super-120b, kimi-k2.x) stay out. The live catalog test
logs, rather than fails, on a breaker-open routed id.
@VickyXAI
VickyXAI merged commit 420b7d1 into main Oct 20, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant