Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
64 changes: 62 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -263,12 +263,69 @@ Two rules govern this file, and both are load-bearing:
1. **These are weak priors, never task-quality labels.** They inform the `speed` and `reliability` terms only. A model is not "better" because it is fast.
2. **Historical numbers are never presented as a current provider SLA.** Hosts are expected to inject fresher observations (see below); the committed snapshot exists so the engine is safe and useful when a catalog is temporarily unavailable.

The repository ships 66 live performance profiles (2026-08-29, three samples per model), 9 auditable historical seeds, 74 function-calling probe results, and a built-in snapshot of all 90 text models in the public catalog (2026-10-20 sync). Eleven newly listed models passed a real structured tool-call probe and are admitted as availability fallbacks for tool-required turns. DeepSeek V4 Flash Vision Experimental and Qwen 3.8 Flash did not return a valid tool call and remain fail-closed. GPT-6.1 Sol, Mistral Large 4, DeepSeek V4.1 Flash and Gemini 4 Argon, listed since the last probe, have no probe evidence yet. A model with no probe evidence can serve plain chat but never a tool-required turn.
The repository ships 66 live performance profiles (2026-08-29, three samples per model), 9 auditable historical seeds, 74 function-calling probe results, and a built-in snapshot of all 90 text models in the public catalog (2026-10-20 sync). Eleven newly listed models passed a real structured tool-call probe and are admitted as availability fallbacks for tool-required turns. DeepSeek V4 Flash Vision Experimental and Qwen 3.8 Flash did not return a valid tool call and remain fail-closed. GPT-6.1 Sol and DeepSeek V4.1 Flash also passed a structured tool-call probe in the subsequent capability refresh. Mistral Large 4 and Gemini 4 Argon have no probe evidence yet. A model with no probe evidence can serve plain chat but never a tool-required turn.

Passing the function-call probe does **not** promote a model to primary. New tool-capable models are appended after the configured and trajectory-benchmarked candidate chains with neutral task affinity; they can recover an unavailable route immediately (cheapest first outside Premium), while primary promotion still requires an end-to-end agent evaluation.

---

## One catalog for chat, image, video and music

`parseModelCatalog()` normalizes the same public gateway catalog for every
consumer. A new model ID is discovered from its categories and valid prices;
there is no model-ID allowlist or product-specific picker order in Core.
GPT-6.1 Sol, image generators, video generators and music generators all appear
in one result, grouped by `chat`, `image`, `video` and `music`.

```ts
import {
parseModelCatalog, toRoutingCatalog, route, DEFAULT_ROUTING_CONFIG,
} from "@blockrun/router-core";

// The host selects the gateway (Base or Solana), performs I/O and keeps its
// last successful catalog on refresh errors. Core never fetches or caches it.
const response = await fetch("https://blockrun.ai/api/v1/models");
if (!response.ok) throw new Error(`Catalog unavailable: ${response.status}`);
const catalog = parseModelCatalog(await response.json());

catalog.chat; // GPT-6.1 Sol and the other currently available chat models
catalog.image; // GPT Image, Nano Banana, Grok Imagine, CogView, Seedream, ...
catalog.video; // Seedance, Grok Imagine Video, Sora (where available), ...
catalog.music; // MiniMax Music and subsequent music listings

// Optional adapter: only chat entries enter the LLM router. Pass the same
// config to both functions if you use custom chains or promotions.
const config = DEFAULT_ROUTING_CONFIG;
const routing = toRoutingCatalog(catalog, { config });
const decision = route(prompt, undefined, 4096, { config, ...routing });
// route() retains its existing total-outage recovery behavior. Do not submit
// a configured fallback that the current gateway no longer advertises.
if (!catalog.chat.some(model => model.id === decision.model)) {
throw new Error("No routed model is available in the current catalog");
}
```

- `models` is the deduplicated full list; the four category arrays preserve
gateway order. A model can belong to more than one category. Products own
display names, grouping, preferred order and default selections.
- Hidden/unavailable entries and rows with invalid per-kind prices are omitted.
Last duplicate wins, including a final retirement. Malformed envelopes throw;
an explicit `data: []` produces an empty catalog. Hosts decide how to handle
an empty or stale response and keep Base/Solana caches separate.
- Prices retain gateway units (chat per 1M tokens; media per image/second/track).
The parser carries base prices and known cache, surcharge and duration fields;
it is not a billing quote engine or a media parameter schema. It does not add
margins, infer image sizes, or assume reference-image support.
- The routing adapter uses live context/output limits and vision categories.
Tool support comes from the bundled verified table or explicit host evidence
via `verifiedCapabilities`; a newly listed model is not automatically declared
tool-capable. Missing capacities fall back to verified limits, then zero.
- Discovering a model makes it available to a picker. It does not promote it to
an Auto primary or add media generation to the LLM router. Existing routing
chains, scoring and decision snapshots are unchanged.

---

## Feeding the router live data

The engine never makes a network call. Everything current is **injected** by the host, which keeps routing on the hot path while still tracking a catalog that moves faster than this package releases.
Expand Down Expand Up @@ -333,6 +390,9 @@ Omitted entries fall back to a weak historical prior rather than disqualifying a
| Export | Purpose |
|---|---|
| `route(prompt, systemPrompt, maxOutputTokens, options)` | The entry point. Returns a `RoutingDecision`. |
| `parseModelCatalog(payload)` | Pure normalization of the public chat/image/video/music catalog. |
| `toRoutingCatalog(catalog, options?)` | Chat-only pricing/capability/availability adapter for `route()`. |
| `MODEL_KINDS` | The four supported catalog kinds. |
| `DEFAULT_ROUTING_CONFIG` | Router Core V3.5 config: tiers, profiles, weights, promotions. |
| `DEFAULT_MODEL_CAPABILITIES` | Built-in capability snapshot (context, max output, tools, vision). |
| `LIVE_MODEL_PROFILES` / `HISTORICAL_MODEL_PROFILES` | Performance priors. |
Expand Down Expand Up @@ -371,7 +431,7 @@ npm run sync:capabilities # → model-capabilities.ts fro
UPDATE_DECISION_SNAPSHOT=1 npm test # then review the snapshot diff
```

The two scripts are the only place the catalog enters this repository. `probe:profiles` pays for real, uncached completions (three samples plus one function-calling request per model, ≈$1 for the whole catalog) and writes the speed/reliability priors; `sync:capabilities` rebuilds the built-in capability snapshot from the public model list, taking `supportsTools` **only** from that probe. Hidden gateway ids are never written — the router should only name a model a user can see on [blockrun.ai/models](https://blockrun.ai/models).
These two scripts update the committed capability and performance snapshots. Runtime catalogs enter through the pure `parseModelCatalog` API described above. `probe:profiles` pays for real, uncached completions (three samples plus one function-calling request per model, ≈$1 for the whole catalog) and writes the speed/reliability priors; `sync:capabilities` rebuilds the built-in capability snapshot from the public model list, taking `supportsTools` **only** from that probe. Hidden gateway ids are never written — the router should only name a model a user can see on [blockrun.ai/models](https://blockrun.ai/models).

Three rules keep consumers safe:

Expand Down
149 changes: 149 additions & 0 deletions catalog.test.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,149 @@
import { describe, expect, it, vi } from "vitest";
import {
DEFAULT_ROUTING_CONFIG, MODEL_KINDS, parseModelCatalog, route, toRoutingCatalog,
} from "./index.js";
import type { ModelCapabilities } from "./index.js";

const chat = (id = "openai/gpt-6.1-sol", extra: Record<string, unknown> = {}) => ({
id, name: "GPT-6.1 Sol", categories: ["chat", "vision", "reasoning"],
context_window: 1_050_000, max_output: 128_000, billing_mode: "paid",
pricing: { input: 2, output: 10 }, ...extra,
});
const media = (kind: "image" | "video" | "music", id = `future/${kind}-next`) => ({
id, name: `New ${kind}`, categories: [kind],
pricing: { [kind === "image" ? "per_image" : kind === "video" ? "per_second" : "per_track"]: 0.12 },
});

describe("unified model catalog", () => {
it("discovers current chat and media models and keeps gateway prices in their original units", () => {
const catalog = parseModelCatalog({ data: [chat(), ...[media("image"), media("video"), media("music")]] });
expect(MODEL_KINDS.map(kind => catalog[kind].length)).toEqual([1, 1, 1, 1]);
expect(catalog.models).toHaveLength(4);
expect(catalog.chat[0]).toMatchObject({
id: "openai/gpt-6.1-sol", name: "GPT-6.1 Sol", free: false,
contextWindow: 1_050_000, maxOutputTokens: 128_000, pricing: { input: 2, output: 10 },
});
expect(catalog.image[0].pricing).toEqual({ per_image: 0.12 });
expect(catalog.video[0].pricing).toEqual({ per_second: 0.12 });
expect(catalog.music[0].pricing).toEqual({ per_track: 0.12 });
});

it("accepts newly listed and multi-category IDs, including version and colon suffixes", () => {
const catalog = parseModelCatalog({ data: [
chat("new-provider/model-7.2:free", { name: "Next Model", categories: ["chat", "image", "vision"], pricing: { input: 1, output: 3, per_image: 0.4 } }),
media("image", "xai/grok-imagine-image-pro"), media("image", "zai/cogview-4"),
media("video", "bytedance/seedance-1.5-pro"),
] });
expect(catalog.chat[0].name).toBe("Next Model");
expect(catalog.chat[0].kinds).toEqual(["chat", "image"]);
expect(catalog.image).toHaveLength(3);
expect(catalog.video[0].id).toBe("bytedance/seedance-1.5-pro");
});

it("requires an explicit free billing mode as well as zero token prices", () => {
const catalog = parseModelCatalog({ data: [
chat("acme/zero", { pricing: { input: 0, output: 0 } }),
chat("acme/free", { billing_mode: "free", pricing: { input: 0, output: 0 } }),
chat("acme/not-free", { billing_mode: "free", pricing: { input: 0, output: 1 } }),
] });
expect(catalog.chat.map(m => m.free)).toEqual([false, true, false]);
});

it.each([null, {}, { data: null }, { data: {} }, [], { error: "offline" }])("rejects malformed envelopes rather than clearing a host cache: %j", value => {
expect(() => parseModelCatalog(value)).toThrow(TypeError);
});

it("allows an explicitly empty catalog and does not borrow models from another network", () => {
const base = parseModelCatalog({ data: [chat(), media("video", "azure/sora-2")] });
const solana = parseModelCatalog({ data: [chat()] });
expect(base.video).toHaveLength(1);
expect(solana.video).toEqual([]);
expect(parseModelCatalog({ data: [] }).models).toEqual([]);
});

it("drops hidden, unavailable, malformed and incorrectly priced rows", () => {
const catalog = parseModelCatalog({ data: [
null, 42, chat("../escape"), chat("acme/missing", { categories: null }),
chat("acme/hidden", { hidden: true }), chat("acme/off", { available: false }),
chat("acme/ambiguous", { available: "true" }), chat("acme/bad", { pricing: { input: -1, output: 1 } }),
chat("acme/infinite", { pricing: { input: Infinity, output: 1 } }),
{ ...media("image"), pricing: { per_image: NaN } },
{ ...media("video"), pricing: { per_image: 0.1 } },
{ ...media("music"), pricing: { per_track: "0.1" } },
chat(),
] });
expect(catalog.models.map(m => m.id)).toEqual(["openai/gpt-6.1-sol"]);
});

it("treats the last duplicate as authoritative, including a retirement or invalid replacement", () => {
const catalog = parseModelCatalog({ data: [
chat("acme/update"), chat("acme/remove"), chat("acme/invalid"),
chat("acme/update", { name: "Updated" }), chat("acme/remove", { available: false }),
chat("acme/invalid", { pricing: null }),
] });
expect(catalog.models.map(m => [m.id, m.name])).toEqual([["acme/update", "Updated"]]);
});

it("does not truncate a growing catalog or modify the caller's payload", () => {
const input = { data: Array.from({ length: 600 }, (_, i) => chat(`acme/model-${i}`)) };
const original = structuredClone(input);
const catalog = parseModelCatalog(input);
expect(catalog.chat).toHaveLength(600);
input.data[0].categories.push("music");
expect(catalog.chat[0].categories).not.toContain("music");
expect(input.data.slice(1)).toEqual(original.data.slice(1));
});
});

describe("chat routing catalog adapter", () => {
it("feeds GPT-6.1 Sol's verified tools and live limits into the existing router", () => {
const catalog = parseModelCatalog({ data: [chat(), media("image")] });
const routing = toRoutingCatalog(catalog);
expect(routing.modelPricing.size).toBe(1);
expect(routing.modelCapabilities["openai/gpt-6.1-sol"]).toEqual({
contextWindow: 1_050_000, maxOutputTokens: 128_000, supportsTools: true, supportsVision: true,
});
const decision = route("Search the latest news using web_search", undefined, 256, {
config: DEFAULT_ROUTING_CONFIG, ...routing, hasTools: true, requiresTools: true, toolNames: ["web_search"],
});
expect(decision.model).toBe("openai/gpt-6.1-sol");
expect(decision.candidates?.every(id => id === "openai/gpt-6.1-sol")).toBe(true);
expect(routing.unavailableModels).toContain("openai/gpt-6-astra");
});

it("does not infer tool support from a new model's name or categories", () => {
const catalog = parseModelCatalog({ data: [chat("future/tool-agent", { categories: ["chat", "tools"] })] });
expect(toRoutingCatalog(catalog).modelCapabilities["future/tool-agent"].supportsTools).toBe(false);
const capabilities: ModelCapabilities = { contextWindow: 500_000, maxOutputTokens: 32_000, supportsTools: true, supportsVision: false };
const routing = toRoutingCatalog(catalog, { verifiedCapabilities: { "future/tool-agent": capabilities } });
expect(routing.modelCapabilities["future/tool-agent"].supportsTools).toBe(true);
expect(routing.modelCapabilities["future/tool-agent"].contextWindow).toBe(1_050_000);
});

it("uses verified fallback limits only when the live catalog lacks valid limits", () => {
const routing = toRoutingCatalog(parseModelCatalog({ data: [
chat("openai/gpt-6.1-sol", { context_window: -1, max_output: 0 }),
chat("future/unknown", { context_window: Infinity, max_output: "large" }),
] }));
expect(routing.modelCapabilities["openai/gpt-6.1-sol"].contextWindow).toBe(1_050_000);
expect(routing.modelCapabilities["future/unknown"]).toMatchObject({ contextWindow: 0, maxOutputTokens: 0, supportsTools: false });
});

it("removes absent custom chain and promotion IDs without injecting media into chat", () => {
const config = structuredClone(DEFAULT_ROUTING_CONFIG);
config.tiers.SIMPLE = { primary: "custom/offline", fallback: ["openai/gpt-6.1-sol"] };
config.promotions = [{ name: "test", startDate: "2026-01-01", endDate: "2027-01-01", tierOverrides: { SIMPLE: { primary: "custom/promo" } } }];
const routing = toRoutingCatalog(parseModelCatalog({ data: [chat(), media("image"), media("video"), media("music")] }), { config });
expect(routing.unavailableModels).toEqual(expect.arrayContaining(["custom/offline", "custom/promo"]));
expect([...routing.modelPricing.keys()]).toEqual(["openai/gpt-6.1-sol"]);
});

it("performs no network requests during parsing, adaptation or routing", () => {
const fetch = vi.spyOn(globalThis, "fetch").mockImplementation(() => { throw new Error("unexpected network"); });
try {
const routing = toRoutingCatalog(parseModelCatalog({ data: [chat()] }));
route("hello", undefined, 256, { config: DEFAULT_ROUTING_CONFIG, ...routing });
expect(fetch).not.toHaveBeenCalled();
} finally { fetch.mockRestore(); }
});
});
Loading
Loading