Repository navigation
feat(constraint): any string as a key or enum value, any type at the schema root - #137
Conversation
…schema root Keys, enum values and const values in a response_format schema were limited to ASCII without quotes, backslashes or control characters, because the automaton matched them as literal bytes and a tokenizer may have no token that spells a multi-byte scalar whole. Apple's TripPlanner sample, whose destination enum holds "Lençóis Maranhenses", was a 400 for that reason. The automaton now matches literals scalar by scalar (LiteralMatch): each scalar may arrive as its raw UTF-8 bytes, split across tokens or not, or as a JSON escape — \uXXXX in either hex case, a surrogate pair above the BMP, or a short escape. An escape is ASCII, which every tokenizer can produce, so a literal the model cannot spell raw is still reachable and a required key can never strand a generation; the compile-time ban is gone. The root of a schema may now be an array, a string, a number, an integer, a boolean, an enum, a const or a $ref as well as an object, which is what @generable emits for [T].self, String.self or an enum. ResponseFormat carries a SchemaValueType root; a root number is complete without a terminator, as under json_object; a root without a type is still an object. Diagnostics at the root say "at the schema root"; nested ones are unchanged. Tests: raw, escaped and mixed spellings, byte-split scalars, surrogate pairs and their dead prefixes, escaped ASCII and control characters; roots of every type; the TripPlanner schema in full, with its destination raw and escaped; the trap search over the escape alphabet; the differential test against the reference validator, which now decodes escapes and generates Unicode literals and roots of every type; the processor reaching an unspellable literal through its escape; a root array and the full itinerary accepted through the server.
On Qwen3.6-27B the first version let a model that wanted "hiking" where the enum says "hotelAndLodging" take the backslash, since it outranked the raw "o", and from there it spelled the rest of the value as ot… — valid JSON, decoded correctly, six tokens a letter and unreadable. An ASCII scalar in a key or enum value is now matched raw only; the escape path stays for scalars outside ASCII, which a tokenizer may not spell whole, and for the quote, the backslash and control characters, which JSON cannot carry raw. The reference validator and the generator apply the same rule. One run of the real-model itinerary test produced a value the automaton rejects when the same text is walked byte by byte (an escaped emoji inside an enum value, then an unpaired high surrogate before the closing quote), and three traced reruns did not reproduce it: every sampled token walked. The mechanism is open. Whatever it is, a sampled token the automaton cannot walk used to leave the state stale, after which every classification ran against a position the text had left and arbitrary bytes passed; the processor now logs that token and its state once and forces EOS at the next step.
…ames by scalar, bound an unbounded array's states The object allow-list no longer admits `$defs`, `definitions`, `$schema` and `$id` on an empty path: the root's were read and stripped before compilation, so an object that still carries one sits below the root — a `$ref` target, or a property named "" — and used to be accepted silently. `isRoot` is now passed explicitly from the root compiler and only words the diagnostics. `required` names, and the program's member index, are compared scalar by scalar like the keys themselves. Swift's `String` equality is canonical, so a decomposed "é" in `required` passed for a precomposed key it could never match; it is a 400 now. An unbounded array's count holds at `minItems`, past which nothing reads it, so such an array has finitely many states and the trap search finishes; the search now asserts that it finished for every hand-written schema instead of silently stopping at the cap. Also: the key diagnostics show the match's progress; `.hex` drops the value it accumulated and never read; the size-limit messages use the root wording at the root; `\b`, `\f` and `\r` join the generator's short escapes, with `/` and control characters in its pools and `/` in its mutation alphabet, and a table test covers every short escape and `\/`; the wedge test primes the vocabulary table the way the sampler does, since `didSample` before any `process` has nothing to classify against; the docs no longer claim every tokenizer can produce an escape.
…unt, name the `$ref` behind a diagnostic State equality compared the schema values the programs were compiled from, whose `String` equality is canonical; it now compares the flattened tables scalar by scalar, so two schemas that differ only in a key's normalisation are different automata, as the matcher treats them. An unbounded array's held count reads `≥N` in diagnostics, so a log line never claims a long array holds `minItems` items. A property named "" is written `""` in paths, so only the root has the empty path and a problem on it is worded as a property's. A problem inside a `$ref` target says which reference led there, since the path names the position, not the definition. A sampled unusable token wedges the processor like an illegal one: it is never legal, so sampling it means the mask was bypassed. The forced-EOS guard gets its CHANGELOG entry. The escape rule is stated the same way in every doc comment (a scalar outside ASCII raw or escaped, the quote, the backslash and control characters escaped, the rest raw), and the error doc no longer lists non-object roots as unsupported. Tests: array positions below and at `minItems`; NFC/NFD pairs in the generator's pools and in the equality, decoder and program tests; `progress:` in the key diagnostics; the size-limit wording at the root and on a property; the count of seeded roots the trap search cannot finish; the gated unusable-token wedge; the model tests walk their output through a fresh automaton.
With a property named "" written `""` in paths, only the root position has the empty path, so the explicit `isRoot` threaded through `compileValue` and `compileObject` duplicated what the path already says: a mutation that read the root off the path survived every test, because the two can no longer disagree. One source of truth now, documented where the path is built.
|
Results on the final head. Pure suite ( Mutation check: 20 guards broken one at a time (list in the description), the constraint and decoder suites (99 tests) run against each: 20/20 caught, 0 survived, tree restored clean between runs. One earlier mutation, reading the root position off the path instead of an explicit flag, survived once the "" path fix landed — the two could no longer disagree — so the explicit flag went (5120634) and that mutation with it. The unusable-token wedge is gated, so I broke it by hand ( Gated processor tests ( Real checkpoint (Qwen3.6-27B-4bit,
Standalone reruns of the throughput test on 4ebfddb read 31.7% and 19.3% while Spotlight was indexing the mutation loop's build output; the figure is informational and not asserted, and the uncontended readings are 7–11%. The run-1 failure is the one the wedge guard answers. The output contained |
1 participant
Two gaps Apple's
@Generablehits, from the roadmap's interoperability axis.Any string as a key, enum or
constvalue. The decoder used to refuse non-ASCII characters, quotes, backslashes and control characters in declared keys and literal values with a 400, because the automaton matched them as literal bytes and a tokenizer may have no token that spells a multi-byte scalar whole (a token cut inside a scalar decodes with U+FFFD and is unusable under the constraint), so a required key could have deadlocked a generation. Apple's TripPlanner sample, whose destination enum holds "Lençóis Maranhenses", was a 400 for exactly that reason.The automaton now matches a key or enum value scalar by scalar (
LiteralMatch). A scalar outside ASCII may arrive as its raw UTF-8 bytes, split across tokens or not, or as a JSON escape:\uXXXXin either hex case, or a surrogate pair above the BMP. The quote, the backslash and control characters take their short escape or\uXXXX. Every other ASCII scalar is matched raw only. I first allowed escapes everywhere, and Qwen3.6-27B, nudged off its preferred word by the mask, drifted into spelling the rest of an enum value as\u006f\u0074\u0065…, since\outranked the raw letter at every step; with ASCII raw-only that cannot start. An escape is ASCII, so a literal the model cannot spell raw is still reachable, and the compile-time ban is gone.Any type at the root.
generating: [T].self,String.selfor an enum type produce a non-object root; the decoder refused everything butobject.ResponseFormat.jsonSchemanow carries aSchemaValueType; the root may be an array, a string, a number, an integer, a boolean, an enum, aconstor a$refas well as an object. Root-only keywords ($defs,definitions,$schema,$id) are read and stripped before the value compiler runs, so they work on every root and are a 400 anywhere below it, including in a$reftarget and under a property named "" (a property named "" is written""in paths, so only the root position has the empty path). A root withouttype(and without$ref,constorenum) is still an object; a root number is complete in a terminal number state, as underjson_object, since nothing terminates it. Diagnostics at the root say "at the schema root"; nested ones are unchanged. No persisted encodings ofGenerateRequestexist (the nested-schemas design note checked), so the largerCodableshape is safe; a round-trip test covers it.A few smaller things the work turned up:
requirednames, the program's member index and state equality are compared scalar by scalar, like the keys themselves. Swift'sStringequality is canonical, so a decomposed "é" inrequiredpassed for a precomposed key it could never match (it is a 400 now), and two states compiled from schemas that differ only in a key's normalisation compared equal (they compare their tables now).minItems, past which nothing reads it, so such an array has finitely many automaton states and the trap search can finish; the search now asserts that it did for every hand-written schema instead of silently stopping at its cap, and counts the seeded schemas too wide to finish. In diagnostics the held count reads≥N.""in diagnostic paths, so only the root has the empty path, and a problem inside a$reftarget names the reference that led there.Wedge guard. One real-model run of the nested itinerary produced an unpaired surrogate inside a string value (
"h\ud83c\udf05…", thenfood…\ud83c"). The pure automaton rejects that text and so does a tokenizer replay of it; three traced reruns walked every sampled token and passed. I could not reproduce or explain it, soJSONConstraintProcessornow notices when a sampled token does not walk, or is unusable: it keeps the last valid state, logs once at error level, and forces EOS at the next step instead of classifying the vocabulary against a position the text has left. One token still reaches the client before the stream ends, since the sampler runs a token ahead. Two gated tests cover the guard, and the model tests now walk their output through a fresh automaton, so a recurrence will say whether the automaton rejects the final text.Tests, all under bare
swift test:\ud83d\ud8,\ud83d\u0, a lone low surrogate), every short escape against its\u00XXspelling and the wrong letter,\/and\u002frefused for/, escaped plain ASCII refused, raw control characters refused;./e/sign or a trailing byte;/, tab, backspace, form feed and carriage return in its pools and/in its mutation alphabet, and produces roots of every type (600 rounds × 12 documents);\is the highest-ranked legal token);const; roots of every kind including$refand$defson an array root; root-only keywords refused in a$reftarget and under a "" property, which takes neither the root's wording nor its empty path; a problem in a$reftarget names the reference;requirednames compared by scalar and compiled names kept scalar-exact; root problems, the size limits included, reported at the root;Codableround trip;minItemsof an unbounded array, the≥Nmarker, NFC/NFD look-alikes unequal, the key match's progress in the log line.Mutation check: 20 guards broken one at a time (hex digit unchecked, raw byte unchecked, low surrogate not required, raw control characters allowed,
\nand\bdropped from the short escapes, plain ASCII allowed escaped, the root object default dropped, root-only keys not stripped, root-only keys allowed below the root,requiredcompared asStringin the decoder and in the program, the unbounded count unclamped, its≥Nmarker dropped, a "" property given the empty path, the$refleft out of its diagnostic, state equality ignoring the program, a root number never complete, a dangling.complete, the reference comparing raw bytes), the constraint and decoder suites run against each; the gated unusable-token wedge was broken by hand and its test run underxcodebuild. Results in the first comment.Real checkpoint:
StructuredOutputModelTests(gated) runs the full itinerary schema with a prompt asking for Lençóis Maranhenses and asserts the decodeddestinationName, and a root[String]schema with exactly three items. Results, including the one failing run above, in the first comment.Note
Medium Risk
Changes core constrained-decoding semantics (literal matching, root types, and failure handling); bugs could accept invalid JSON or truncate streams early, but behavior is heavily tested against a reference validator and real-model gates.
Overview
Expands C2 structured output so Apple's
@Generableshapes compile and decode correctly:response_formatschemas can use any supported type at the root (array, scalar, enum,const,$ref, not only objects), and property keys plus enum/constvalues may be any Unicode string instead of being rejected at compile time for non-ASCII or characters that need JSON escaping.The schema automaton gains scalar-by-scalar literal matching (
LiteralMatch/SchemaLiteral): non-ASCII and escapable characters can appear raw or as JSON escapes (including surrogate pairs), while printable ASCII in literals must be spelled raw so models do not drift into all-\u00XXspellings. The decoder drops the oldrequireLiteralMatchablechecks, comparesrequirednames by Unicode scalar (not canonicalStringequality), and improves root vs nested diagnostics.SchemaConstraintStateandSchemaProgramare rooted onSchemaValueType; unbounded arrays cap their counted length atminItemsfor a finite state space, and root numbers can complete without a trailing delimiter.JSONConstraintProcessoradds a wedge guard: if an illegal or unusable token is sampled anyway, it logs once, keeps the last good automaton state, and forces EOS on the next mask step instead of continuing from a stale position. Changelog/README note TripPlanner ("Lençóis Maranhenses") and root-array@Generablecases; tests and server stubs cover literals, roots, wedge behavior, and full itinerary schemas.Reviewed by Cursor Bugbot for commit 5120634. Configure here.