Repository navigation
fix: stop system prompts, substring keywords and chars/4 from steering routing - #6
Merged
Merged
Conversation
added 4 commits
October 18, 2026 14:22
…g routing
Four classifier inputs were carrying signal they should not have, and one
scoring term was decaying in the wrong direction. Each is covered by a test
that failed before the change.
System prompt leakage. rules.ts has read only the user turn since issue #50,
but two other readers remained. RulesStrategy inferred a structured-output
requirement from /json|structured|schema/ in the system prompt, so an agent
host that lists its tool schemas there had every turn lifted off SIMPLE
("what is the capital of france?" went MEDIUM, ~$0 → $0.003). Every consumer
already passes `requiresStructuredOutput` from response_format, so the
inference is dropped and the protocol flag is the only source. In
portfolio.ts, highStakes, latencySensitive and the zh language band read the
full text, so "be quick" or a Mandarin persona in a host's standing
instructions moved candidate scores (0.842 → 0.976 on a web-search turn) and
sent English extraction to the Mandarin band. They now read the user turn;
only the deep-web-research clue detectors still read the full text, and those
are gated on a web-research tool surface.
Token estimate. Both strategies estimated chars/4 regardless of script. CJK
text tokenizes at roughly one token per character, so a 200k-character
Chinese contract read as 50k tokens: under the 80k long_context boundary,
under maxTokensForceComplex, and through the capacity filter for models that
cannot hold it. `estimateTokens` (tokens.ts, also exported for hosts) counts
CJK characters at one each and the rest at a quarter. The scan-window helper
both strategies had copied moves there too.
Keyword matching. Substring tests counted `class` in "classic", `story` in
"history", `edit` in "credit", `pip` in "pipeline", `def`/`define` in
"undefined". Alphabetic keywords now start at a word boundary; ASCII keywords
also end at one after an optional English inflection (classes, defining,
deploying still count). Non-ASCII alphabetic keywords stay trailing-open
because several are deliberate stems (оптимизаци), and CJK keywords keep the
substring test. A cheap substring pre-check keeps the regex off the common
path; routing stays well under a millisecond.
Stale speed. profileScore blended reliability toward its neutral prior as an
observation aged but multiplied speed by freshness, so a model with a
49-day-old three-sample probe scored 0.096 on speed while an unmeasured
model scored 0.5 — the eleven newly admitted tool fallbacks, none of which
has a profile yet, were winning that term by default. Speed and tail speed
now fade toward 0.5 like reliability.
Unpriced models. selectModel reported costEstimate 0 and savings 100% for a
model the host had no price for. Savings are now 0 when the price is unknown;
the charge-accurate calculateModelCost path is unchanged.
Code-task floor. With the accidental `def`/`define` hits gone, the corpus's
stack-trace prompt scored SIMPLE. The portfolio classifier still knows it is
a debug task and the SIMPLE chain has no model with code affinity, so debug
and code_edit now floor at MEDIUM, alongside the existing reasoning floor.
Decision snapshot: 88 rows, one prompt moves tier (the tool-less "rm -rf …
rerun the release pipeline" prose, MEDIUM → SIMPLE, which had only reached
MEDIUM through `pip` in "pipeline"); six rows drop a fabricated 100% savings
on an unpriced model; ten change only the signal text in `reasoning`.
Also: PortfolioStrategy reuses one RulesStrategy and checks eligibility once
per candidate. dist/ rebuilt.
…e floor on code evidence Three findings from an outside review of the previous commit. The end boundary on ASCII keywords accepted only English suffixes, but the lists spell German and Romance keywords in ASCII too: "Beweise den mathematischen Satz." lost both reasoning markers and fell from REASONING to SIMPLE. The accepted inflections are now graded by the keyword's last word: three letters or fewer take only endings that cannot spell another word (fix → fixes, never pip → pipe or def → defer); four to six letters also take the common German and Romance endings (beweis → beweise, klasse → klassen) and a doubled final consonant (debug → debugging, which the previous form missed and which dropped "Keep debugging and editing until it works." from agentic 0.6 to 0.2); longer words stay trailing-open like the substring test they replaced. Arabic joins the substring-matched scripts, since it attaches articles and conjunctions to the word. The MEDIUM floor for debug/code_edit tasks fired on bare verbs — "Rewrite this sentence" and "margin of error" were leaving the SIMPLE chain. It now requires code evidence, and a CamelCase error class (TypeError, NullPointerException) counts as such evidence and as a debug cue, so the stack-trace prompt still gets the floor while prose does not. Decision snapshot footprint versus main is unchanged. dist/ rebuilt.
… artifact
Two further findings from the outside review.
The keyword boundaries excluded any letter, so in a mixed-script prompt
without spaces ("请step by step证明这个结论。") the Chinese characters on
either side of "step by step" read as part of the word and the reasoning
marker was lost. A boundary is now a transition out of the segmented
scripts (Latin, Cyrillic, Greek, digits); a CJK neighbour is a boundary.
The MEDIUM floor for debug/code_edit was gated on hasCode, which a bare
language name satisfies, so "What is Python?" and a translation that
mentions Python left the SIMPLE chain. The floor now requires a code
artifact or coding action (fence, file extension, trace, exception class,
construct-plus-verb, native-language phrase); a language name alone no
longer qualifies.
Decision snapshot footprint versus main is unchanged. dist/ rebuilt.
"Translate into French: Please make an exception." was leaving the SIMPLE chain through the code floor. Only a stack trace, traceback, fence, file extension, CamelCase error class, construct-plus-verb or native-language phrase counts; a thrown exception already surfaces as its class name. Snapshot unchanged. dist/ rebuilt.
This was referenced Oct 18, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Five classifier/scoring defects, each reproduced by a test that failed before the fix, plus the decision-snapshot footprint for review.
RulesStrategyinferred structured output from/json|structured|schema/in the system prompt (every agent-host turn left SIMPLE);portfolio.tsreadhighStakes,latencySensitiveand the zh language band from the full text. All now read the user turn;requiresStructuredOutputis the only structured-output source (every consumer already passes it).long_context,maxTokensForceComplexand the capacity filter. NewestimateTokens(exported) counts CJK characters at one token each.class∈"classic",story∈"history",edit∈"credit",pip∈"pipeline",def∈"undefined". Word-boundary matching with English inflections; Cyrillic stems and CJK keep their current behaviour.selectModelnow reports 0 savings when the price is unknown.Plus a MEDIUM floor for
debug/code_edittasks (the SIMPLE chain has no code-affinity model) and two behaviour-preserving cleanups (shared scan helper, singleRulesStrategy, one eligibility check per candidate).Decision snapshot footprint (88 rows)
pipin "pipeline").reasoning.Test plan
npm run check— typecheck, 139 tests, build🤖 Generated with Claude Code