Skip to content

fix(router): SmartChatResponse.Model names the model that answered - #45

Merged
VickyXAI merged 1 commit into
mainfrom
fix/report-served-model
Oct 20, 2026
Merged

VickyXAI merged 1 commit into
mainfrom
fix/report-served-model

Conversation

@VickyXAI

Copy link
Copy Markdown
Contributor

Problem

The gateway sometimes serves a different model than the one requested:

  • Health reroute: a free model whose health breaker is open is rerouted before dispatch.
  • Cascade fallback: a failed free call falls through the free cascade.

In both cases SmartChatResponse.Model still reported the id the router requested. Since #44 routes the free tiers to primaries whose breakers can be open, this is the common case on the free profile. Live example: requesting nvidia/nemotron-3.5-lightning was answered by nvidia/nemotron-3-super-120b.

What the gateway sends

Checked read-only in blockrun origin/main, src/app/api/v1/chat/completions/route.ts:

  • Health reroute: X-Health-Reroute: true, X-Original-Model, and X-Served-Model (route.ts:3421-3424).
  • Cascade or rescue fallback: X-Original-Model and X-Fallback-Model (route.ts:3441-3442, 3552, 3621, 3685, 4811).
  • The response body's model is the served model in both cases (confirmed live).

Change

  • SmartChatResponse.Model is now the model that answered. It is resolved in this order:
    1. the response body's model
    2. the X-Served-Model header
    3. the X-Fallback-Model header
    4. the requested id
  • New field SmartChatResponse.RequestedModel (json:"requested_model") holds the router's pick. Routing.Model already held it and is unchanged.
  • The change is additive: no existing field changed name, type or JSON key.
  • Internals: ChatCompletion now wraps a new unexported chatCompletionHeaders, which uses the existing doRequestHeaders, so SmartChat can read the response headers. Its public behaviour is unchanged. When a response comes from the local cache, headers are nil and the body's model still applies.
  • README and the CHANGELOG Unreleased entry are updated.

Tests

router_served_model_test.go uses an httptest server to cover five cases:

  • no reroute
  • body model differs from the request
  • header only: health reroute (X-Served-Model)
  • header only: cascade fallback (X-Fallback-Model)
  • no body model and no headers (falls back to the requested id)

Each case also asserts RequestedModel and Routing.Model. A separate test checks the JSON keys stay additive.

Checks

  • go build ./...: ok
  • go vet ./...: ok
  • go test ./...: ok
  • BLOCKRUN_LIVE_CATALOG=1 go test -run TestRoutingTableMatchesLiveCatalog .: ok

When the gateway reroutes (free health breaker open, or a free-cascade
fallback) Model reported the requested id. It is now the response body's
model, else the X-Served-Model / X-Fallback-Model header, else the
requested id. New SmartChatResponse.RequestedModel keeps the router's
pick; existing fields are unchanged.
@VickyXAI
VickyXAI merged commit 173ae4c into main Oct 20, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant