Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
15 changes: 15 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,21 @@

All notable changes to blockrun-llm-go will be documented in this file.

## Unreleased

- **fix(router): SmartChat no longer routes to ids the gateway does not serve.**
The free profile named `nvidia/nemotron-3-super-120b` (gone from
`/v1/models`) and `nvidia/nemotron-3-nano-omni-30b-a3b-reasoning`
(`available: false`); the gateway redirected both to
`nvidia/llama-3.2-11b-vision`, so `SmartChatResponse.Model` named a model
that never served the call. All four free tiers now route to
`nvidia/llama-3.2-11b-vision`, the only free model the catalog marks
available. Eco and Auto Simple routed to hidden `moonshot/kimi-k2.7`; they
now route to `google/gemini-2.5-flash-lite` and `google/gemini-2.5-flash`
(router-core's eco/auto SIMPLE choices). A retired-id test covers every
profile, and `BLOCKRUN_LIVE_CATALOG=1 go test` checks the table against the
live catalog.

## 0.22.1

- **fix(solana): `MusicResponse.TxHash` is filled for every Solana track.**
Expand Down
42 changes: 22 additions & 20 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -75,26 +75,26 @@ credit account is enough:

```go
// Option 1: call a free model directly
reply, _ := client.Chat(ctx, "nvidia/nemotron-3-nano-omni-30b-a3b-reasoning", "Explain x402 in 1 sentence")
reply, _ := client.Chat(ctx, "nvidia/llama-3.2-11b-vision", "Explain x402 in 1 sentence")

// Option 2: let the smart router pick the best free model per request
result, _ := client.SmartChat(ctx, "What is 2+2?", &blockrun.SmartChatOptions{
RoutingProfile: blockrun.RoutingFree,
})
fmt.Println(result.Model) // e.g. "nvidia/nemotron-3-super-120b"
fmt.Println(result.Model) // "nvidia/llama-3.2-11b-vision"
fmt.Println(result.Response) // "4"
```

**Available free models** (input + output both $0, from the live `/v1/models` catalog, 2026-10-02):
**Available free models** (input + output both $0, `available: true` in the live `/v1/models` catalog, 2026-10-20):

| Model ID | Context | Best For |
|----------|---------|----------|
| `nvidia/nemotron-3-ultra-550b` | 1M | Largest free model — 550B / 55B active MoE reasoning |
| `nvidia/nemotron-3.5-lightning` | 1M | Thinking-mode reasoning, long context |
| `nvidia/nemotron-3-nano-omni-30b-a3b-reasoning` | 256K | Multimodal reasoning — text, images, video, audio |
| `nvidia/llama-3.2-11b-vision` | 128K | Meta Llama 3.2 vision |
| `cohere/north-mini-code` | 256K | Compact coding model |
| `poolside/laguna-xs-2.1` | 131K | Fast compact coding model |
| `nvidia/llama-3.2-11b-vision` | 128K | Meta Llama 3.2 vision — the only free model currently marked available |

The catalog also lists `nvidia/nemotron-3-ultra-550b`, `nvidia/nemotron-3.5-lightning`,
`nvidia/nemotron-3-nano-omni-30b-a3b-reasoning`, `nvidia/muse-glimmer-30b`,
`cohere/north-mini-code` and `poolside/laguna-xs-2.1` as free, but with
`available: false`; calls to them are redirected to another free model.

> The free pool rotates as NVIDIA retires deployments. Retired free IDs keep
> answering through gateway redirects, but `GET /v1/models` is the list to
Expand Down Expand Up @@ -499,9 +499,9 @@ resp, err := client.SmartChat(ctx, "Prove P != NP", &blockrun.SmartChatOptions{

| Profile | Simple | Medium | Complex | Reasoning |
|---------|--------|--------|---------|-----------|
| **free** | nvidia/nemotron-3-super-120b | nvidia/nemotron-3-super-120b | nvidia/nemotron-3-super-120b | nvidia/nemotron-3-nano-omni-30b-a3b-reasoning |
| **eco** | moonshot/kimi-k2.7 | deepseek/deepseek-chat | google/gemini-2.5-pro | deepseek/deepseek-reasoner |
| **auto** | moonshot/kimi-k2.7 | google/gemini-3.5-flash | google/gemini-3.1-pro | deepseek/deepseek-reasoner |
| **free** | nvidia/llama-3.2-11b-vision | nvidia/llama-3.2-11b-vision | nvidia/llama-3.2-11b-vision | nvidia/llama-3.2-11b-vision |
| **eco** | google/gemini-2.5-flash-lite | deepseek/deepseek-chat | google/gemini-2.5-pro | deepseek/deepseek-reasoner |
| **auto** | google/gemini-2.5-flash | google/gemini-3.5-flash | google/gemini-3.1-pro | deepseek/deepseek-reasoner |
| **premium** | google/gemini-3.5-flash | openai/gpt-5.5 | anthropic/claude-opus-4.8 | openai/o3 |

> DeepSeek V4 family launched 2026-04-24. The legacy `deepseek/deepseek-chat`
Expand All @@ -513,14 +513,16 @@ resp, err := client.SmartChat(ctx, "Prove P != NP", &blockrun.SmartChatOptions{
> `deepseek-reasoner` as the eco/auto reasoning primary because V4 Flash
> thinking is cheaper.
>
> Free routing, 2026-10-02: `nvidia/deepseek-v4-flash`,
> `nvidia/llama-4-maverick` and `nvidia/qwen3-coder-480b` left the gateway
> catalog, and the gateway already answered all three with
> `nvidia/nemotron-3-super-120b`. The free Simple / Medium / Complex tiers now
> name that model directly, so `result.Model` matches what serves the call.
> Reasoning stays on `nvidia/nemotron-3-nano-omni-30b-a3b-reasoning`. The free
> pool is load-shed by a gateway circuit breaker, so a free call can still be
> rerouted to another $0 model or refused with `FREE_MODEL_FAILED` under load.
> Live-catalog sweep, 2026-10-20: `nvidia/nemotron-3-super-120b` left
> `/v1/models` and `nvidia/nemotron-3-nano-omni-30b-a3b-reasoning` is listed
> `available: false`; the gateway answers both with
> `nvidia/llama-3.2-11b-vision`, the only free model marked available, so every
> free tier now names it and `result.Model` matches what serves the call.
> `moonshot/kimi-k2.7` is hidden from the catalog: **eco** Simple moved to
> `google/gemini-2.5-flash-lite` and **auto** Simple to `google/gemini-2.5-flash`.
> The free pool is load-shed by a gateway circuit breaker, so a free call can
> still be rerouted to another $0 model or refused with `FREE_MODEL_FAILED`
> under load.

## Streaming

Expand Down
21 changes: 17 additions & 4 deletions router.go
Original file line number Diff line number Diff line change
Expand Up @@ -99,17 +99,30 @@ type SmartChatOptions struct {
// premium/COMPLEX flagship, replacing anthropic/claude-opus-4.5. Opus 4.8 is
// Anthropic's most capable model with a 1M-token context window. Older Opus
// IDs remain available for clients pinned to their pricing.
// Live-catalog sweep, 2026-10-20 (GET /v1/models + a $0 probe of each free id):
// - nvidia/nemotron-3-super-120b is no longer listed; the gateway answers it
// with nvidia/llama-3.2-11b-vision, so the free table was naming a model
// that never served the call.
// - nvidia/nemotron-3-nano-omni-30b-a3b-reasoning (and every other free id
// except llama-3.2-11b-vision) is listed with available:false and is
// redirected too. nvidia/llama-3.2-11b-vision is the only free model the
// catalog marks available on every fetch (the flag is a live health
// signal; laguna-xs-2.1 flapped), so all four free tiers route to it.
// - moonshot/kimi-k2.7 is hidden from /v1/models. Eco/SIMPLE takes
// google/gemini-2.5-flash-lite ($0.10/$0.40) and Auto/SIMPLE takes
// google/gemini-2.5-flash ($0.30/$2.50), matching router-core's eco
// SIMPLE first live rung and auto SIMPLE primary.
var routingTable = map[RoutingProfile]map[RoutingTier]string{
RoutingFree: {
TierSimple: "nvidia/nemotron-3-super-120b", TierMedium: "nvidia/nemotron-3-super-120b",
TierComplex: "nvidia/nemotron-3-super-120b", TierReasoning: "nvidia/nemotron-3-nano-omni-30b-a3b-reasoning",
TierSimple: "nvidia/llama-3.2-11b-vision", TierMedium: "nvidia/llama-3.2-11b-vision",
TierComplex: "nvidia/llama-3.2-11b-vision", TierReasoning: "nvidia/llama-3.2-11b-vision",
},
RoutingEco: {
TierSimple: "moonshot/kimi-k2.7", TierMedium: "deepseek/deepseek-chat",
TierSimple: "google/gemini-2.5-flash-lite", TierMedium: "deepseek/deepseek-chat",
TierComplex: "google/gemini-2.5-pro", TierReasoning: "deepseek/deepseek-reasoner",
},
RoutingAuto: {
TierSimple: "moonshot/kimi-k2.7", TierMedium: "google/gemini-3.5-flash",
TierSimple: "google/gemini-2.5-flash", TierMedium: "google/gemini-3.5-flash",
TierComplex: "google/gemini-3.1-pro", TierReasoning: "deepseek/deepseek-reasoner",
},
RoutingPremium: {
Expand Down
66 changes: 66 additions & 0 deletions router_live_catalog_test.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,66 @@
package blockrun

import (
"encoding/json"
"net/http"
"os"
"testing"
"time"
)

// TestRoutingTableMatchesLiveCatalog holds every routed id to the live
// gateway catalog: listed, not available:false, and for the free profile
// billed as free. "available" is the gateway's live health flag, so an
// available:false failure can be a transient circuit-breaker trip; refetch
// before retargeting. Opt-in because it needs network:
//
// BLOCKRUN_LIVE_CATALOG=1 go test -run TestRoutingTableMatchesLiveCatalog
func TestRoutingTableMatchesLiveCatalog(t *testing.T) {
if os.Getenv("BLOCKRUN_LIVE_CATALOG") == "" {
t.Skip("BLOCKRUN_LIVE_CATALOG not set, skipping live catalog check")
}
url := os.Getenv("BLOCKRUN_CATALOG_URL")
if url == "" {
url = "https://blockrun.ai/api/v1/models"
}
httpClient := &http.Client{Timeout: 30 * time.Second}
resp, err := httpClient.Get(url)
if err != nil {
t.Fatalf("GET %s: %v", url, err)
}
defer resp.Body.Close()
if resp.StatusCode != http.StatusOK {
t.Fatalf("GET %s: HTTP %d", url, resp.StatusCode)
}
var catalog struct {
Data []struct {
ID string `json:"id"`
BillingMode string `json:"billing_mode"`
Available *bool `json:"available"`
} `json:"data"`
}
if err := json.NewDecoder(resp.Body).Decode(&catalog); err != nil {
t.Fatalf("decode catalog: %v", err)
}
type entry struct {
billing string
available bool
}
live := make(map[string]entry, len(catalog.Data))
for _, m := range catalog.Data {
live[m.ID] = entry{billing: m.BillingMode, available: m.Available == nil || *m.Available}
}
for profile, tiers := range routingTable {
for tier, model := range tiers {
e, ok := live[model]
switch {
case !ok:
t.Errorf("%s/%s routes to %s, which /v1/models does not list", profile, tier, model)
case !e.available:
t.Errorf("%s/%s routes to %s, which /v1/models marks available:false", profile, tier, model)
case profile == RoutingFree && e.billing != "free":
t.Errorf("free/%s routes to %s, billed %q", tier, model, e.billing)
}
}
}
}
44 changes: 31 additions & 13 deletions router_readme_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -6,21 +6,39 @@ import (
"testing"
)

// The free profile must not name ids the gateway has dropped from /v1/models.
// Each of these is answered by a MODEL_REDIRECTS target on the gateway; the
// table names that target instead.
func TestFreeRoutingAvoidsRetiredIDs(t *testing.T) {
retired := []string{
"nvidia/deepseek-v4-flash",
"nvidia/llama-4-maverick",
"nvidia/qwen3-coder-480b",
}
for tier, model := range routingTable[RoutingFree] {
for _, id := range retired {
if model == id {
t.Errorf("free/%s routes to retired %s", tier, id)
// retiredRoutingIDs are ids the gateway has dropped from /v1/models, hidden,
// or lists with available:false. Each is answered by a redirect to another
// model, so routing to one makes SmartChatResponse.Model name a model that
// never served the call. TestRoutingTableMatchesLiveCatalog
// (BLOCKRUN_LIVE_CATALOG=1) checks the table against the live catalog itself.
var retiredRoutingIDs = []string{
"nvidia/deepseek-v4-flash",
"nvidia/llama-4-maverick",
"nvidia/qwen3-coder-480b",
"nvidia/nemotron-3-super-120b", // left /v1/models; redirected 2026-10
"nvidia/nemotron-3-nano-omni-30b-a3b-reasoning", // available:false 2026-10
"nvidia/nemotron-3.5-lightning", // available:false 2026-10
"nvidia/nemotron-3-ultra-550b", // available:false 2026-10
"moonshot/kimi-k2.5", // hidden
"moonshot/kimi-k2.6", // hidden
"moonshot/kimi-k2.7", // hidden
}

func TestRoutingTableAvoidsRetiredIDs(t *testing.T) {
for profile, tiers := range routingTable {
for tier, model := range tiers {
for _, id := range retiredRoutingIDs {
if model == id {
t.Errorf("%s/%s routes to retired %s", profile, tier, id)
}
}
}
}
}

// The free profile must stay inside the $0 NVIDIA-hosted tier.
func TestFreeRoutingStaysOnFreeTier(t *testing.T) {
for tier, model := range routingTable[RoutingFree] {
if !strings.HasPrefix(model, "nvidia/") {
t.Errorf("free/%s routes to %s, outside the free NVIDIA tier", tier, model)
}
Expand Down
Loading