Skip to content

Commit db70539

Browse files
sanchitrkclaude
andauthored
fix(aikit): convert strict tool schemas to the strict subset (#61)
Port strict JSON-schema conversion with prefer/require fallback, strip optional nulls before validation, pass constrainedSampling through plugin ToolDef, and decode harness args via the JSON codec. Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
1 parent 06862c3 commit db70539

10 files changed

Lines changed: 312 additions & 31 deletions

File tree

‎CHANGELOG.md‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -13,14 +13,17 @@ This file is the canonical source for unreleased changes and published release n
1313

1414
### Added
1515

16+
- Added the `normalizeOptionalNulls` export, which drops the `null` a strict tool call sends for an omitted optional argument whose schema does not accept null. `validateToolArguments` now applies it.
1617
- Added automatic retry of transient provider failures in the harness loop. A turn whose request fails with a rate limit, overload, 5xx, timeout or dropped connection, including one that fails mid-stream, is retried with exponential backoff. Configure it with a top-level `retry` block in `settings.jsonc`: `{ "maxRetries": 3, "baseDelayMs": 2000, "maxDelayMs": 60000 }`, where `maxRetries: 0` turns it off. Quota errors are never retried, and a server that asks to wait longer than `maxDelayMs` fails the turn. Each failed attempt stays in the session as an aborted entry that the model never sees, and clients receive `session.retry.scheduled` and `session.retry.finished` events to show progress.
1718

1819
### Changed
1920

2021
- `model.options.maxRetries`, the retry inside the provider SDK, now defaults to `0` (was `3`). The SDK retried silently, only before the stream started, and also retried quota errors. Combined with the loop retry, one outage would have cost up to 16 requests. Set it per model if an endpoint needs it.
22+
- `resolveOpenAICodexToolConstraint` now resolves grammar constraints only, and `Message.resolveJsonSchemaConstraint` is removed; `convertTools` handles JSON-schema strictness for every provider.
2123

2224
### Fixed
2325

26+
- Fixed tools with `constrainedSampling: { type: "json_schema" }` being rejected by Anthropic and other strict providers. aikit set `strict: true` but sent the original schema; it now converts it to the strict subset (closed objects, every property required, optional ones nullable). A `"prefer"` tool whose schema the provider cannot express in strict mode, such as Anthropic with `minimum`, is sent non-strict; a `"require"` tool throws.
2427
- Fixed an interrupted turn never reporting `session.turn.aborted`. The cleanup that settles the turn's open draft and publishes the event was skipped on interruption, so the draft was only cleaned up on the session's next run.
2528

2629
## [@codeworksh/aikit@0.10.0]

‎packages/aikit/src/index.ts‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -8,7 +8,7 @@ export * as Message from "./message/message.ts";
88
export * as Model from "./model/model.ts";
99
export { stream } from "./stream.ts";
1010
export { createAssistantMessageEventStream, EventStream } from "./utils/eventstream.ts";
11-
export { validateSchema, validateToolArguments, validateToolCall } from "./utils/validation.ts";
11+
export { normalizeOptionalNulls, validateSchema, validateToolArguments, validateToolCall } from "./utils/validation.ts";
1212

1313
export { createOpenAICodex, openaiCodex, openAICodexTools } from "./providers/openai-codex/index.ts";
1414
export type {

‎packages/aikit/src/llm/strict.ts‎

Lines changed: 227 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,227 @@
1+
import type * as Message from "../message/message.ts";
2+
import * as Model from "../model/model.ts";
3+
4+
type JsonSchemaObject = Record<string, unknown> & {
5+
properties?: Record<string, unknown>;
6+
required?: unknown;
7+
};
8+
9+
class UnsupportedStrictSchemaError extends Error {}
10+
11+
/** True when a provider's strict mode rejects this schema keyword with this value. */
12+
type UnsupportedKeyword = (key: string, value: unknown) => boolean;
13+
14+
const UNSUPPORTED_KEYWORDS = [
15+
"$ref",
16+
"$defs",
17+
"definitions",
18+
"allOf",
19+
"oneOf",
20+
"patternProperties",
21+
"dependentSchemas",
22+
"dependencies",
23+
"unevaluatedProperties",
24+
"propertyNames",
25+
"contains",
26+
"prefixItems",
27+
"not",
28+
"if",
29+
"then",
30+
"else",
31+
] as const;
32+
33+
// Anthropic strict tool use rejects these with a 400 for the whole request.
34+
// https://platform.claude.com/docs/en/build-with-claude/structured-outputs#json-schema-limitations
35+
const ANTHROPIC_UNSUPPORTED_KEYWORDS = new Set([
36+
"minimum",
37+
"maximum",
38+
"exclusiveMinimum",
39+
"exclusiveMaximum",
40+
"multipleOf",
41+
"maxItems",
42+
"uniqueItems",
43+
"minContains",
44+
"maxContains",
45+
"minProperties",
46+
"maxProperties",
47+
]);
48+
const ANTHROPIC_STRING_FORMATS = new Set([
49+
"date-time",
50+
"time",
51+
"date",
52+
"duration",
53+
"email",
54+
"hostname",
55+
"uri",
56+
"ipv4",
57+
"ipv6",
58+
"uuid",
59+
]);
60+
61+
const isAnthropicUnsupportedKeyword: UnsupportedKeyword = (key, value) => {
62+
if (ANTHROPIC_UNSUPPORTED_KEYWORDS.has(key)) return true;
63+
if (key === "minItems") return value !== 0 && value !== 1;
64+
if (key === "format") return typeof value !== "string" || !ANTHROPIC_STRING_FORMATS.has(value);
65+
return false;
66+
};
67+
68+
function isJsonSchemaObject(value: unknown): value is JsonSchemaObject {
69+
return typeof value === "object" && value !== null && !Array.isArray(value);
70+
}
71+
72+
function isNullSchema(schema: unknown): boolean {
73+
return isJsonSchemaObject(schema) && schema.type === "null";
74+
}
75+
76+
/** `X` of an `anyOf: [X, null]` schema. */
77+
export function nullableVariant(schema: unknown): unknown {
78+
if (!isJsonSchemaObject(schema) || !Array.isArray(schema.anyOf)) return;
79+
const variants = schema.anyOf.filter((variant) => !isNullSchema(variant));
80+
return variants.length === 1 ? variants[0] : undefined;
81+
}
82+
83+
function isStructuredSchema(schema: unknown): boolean {
84+
if (!isJsonSchemaObject(schema)) return false;
85+
const types = typeof schema.type === "string" ? [schema.type] : Array.isArray(schema.type) ? schema.type : [];
86+
return (
87+
types.includes("object") ||
88+
types.includes("array") ||
89+
schema.properties !== undefined ||
90+
schema.items !== undefined
91+
);
92+
}
93+
94+
const CONSTRAINING_KEYWORDS = [
95+
"type",
96+
"const",
97+
"enum",
98+
"anyOf",
99+
"allOf",
100+
"oneOf",
101+
"not",
102+
"$ref",
103+
"properties",
104+
"items",
105+
];
106+
107+
export function schemaAllowsNull(schema: unknown): boolean {
108+
if (schema === true) return true;
109+
if (!isJsonSchemaObject(schema)) return false;
110+
if (CONSTRAINING_KEYWORDS.every((key) => schema[key] === undefined)) return true;
111+
if (schema.type === "null" || (Array.isArray(schema.type) && schema.type.includes("null"))) return true;
112+
if (schema.const === null || (Array.isArray(schema.enum) && schema.enum.includes(null))) return true;
113+
return Array.isArray(schema.anyOf) && schema.anyOf.some((variant) => schemaAllowsNull(variant));
114+
}
115+
116+
function makeNodeStrict(schema: unknown, isUnsupportedKeyword?: UnsupportedKeyword): void {
117+
if (!isJsonSchemaObject(schema)) throw new UnsupportedStrictSchemaError("boolean schemas are unsupported");
118+
for (const key of UNSUPPORTED_KEYWORDS) {
119+
if (schema[key] !== undefined) throw new UnsupportedStrictSchemaError(`${key} schemas are unsupported`);
120+
}
121+
if (isUnsupportedKeyword) {
122+
for (const [key, value] of Object.entries(schema)) {
123+
if (isUnsupportedKeyword(key, value)) {
124+
throw new UnsupportedStrictSchemaError(`${key}: ${JSON.stringify(value)} is unsupported`);
125+
}
126+
}
127+
}
128+
129+
if (schema.anyOf !== undefined) {
130+
if (!Array.isArray(schema.anyOf) || schema.anyOf.length === 0) {
131+
throw new UnsupportedStrictSchemaError("anyOf must contain at least one schema");
132+
}
133+
// `[X, null]` is the nullable shape this transform itself emits; only real unions of structures are rejected.
134+
const nullable = nullableVariant(schema) !== undefined;
135+
for (const variant of schema.anyOf) {
136+
if (!nullable && isStructuredSchema(variant)) {
137+
throw new UnsupportedStrictSchemaError("object and array unions are unsupported");
138+
}
139+
makeNodeStrict(variant, isUnsupportedKeyword);
140+
}
141+
}
142+
143+
if (schema.items !== undefined) {
144+
if (Array.isArray(schema.items)) throw new UnsupportedStrictSchemaError("tuple schemas are unsupported");
145+
makeNodeStrict(schema.items, isUnsupportedKeyword);
146+
}
147+
148+
const isObjectSchema = schema.type === "object";
149+
if (schema.properties !== undefined && !isObjectSchema) {
150+
throw new UnsupportedStrictSchemaError("properties require type object");
151+
}
152+
if (!isObjectSchema) return;
153+
// Effect Schema emits `additionalProperties: true` for every struct; excess keys are dropped on decode, so
154+
// closing the object loses nothing. Only schema-valued additionalProperties carries meaning.
155+
if (schema.additionalProperties !== undefined && typeof schema.additionalProperties !== "boolean") {
156+
throw new UnsupportedStrictSchemaError("schema-valued additionalProperties is unsupported");
157+
}
158+
if (schema.properties !== undefined && !isJsonSchemaObject(schema.properties)) {
159+
throw new UnsupportedStrictSchemaError("object properties must be a schema map");
160+
}
161+
if (
162+
schema.required !== undefined &&
163+
(!Array.isArray(schema.required) || schema.required.some((key) => typeof key !== "string"))
164+
) {
165+
throw new UnsupportedStrictSchemaError("object required must be a string array");
166+
}
167+
168+
const properties = schema.properties ?? {};
169+
const propertyNames = Object.keys(properties);
170+
const required = new Set(Array.isArray(schema.required) ? schema.required : []);
171+
if ([...required].some((key) => !propertyNames.includes(key))) {
172+
throw new UnsupportedStrictSchemaError("required contains an unknown property");
173+
}
174+
for (const [key, property] of Object.entries(properties)) {
175+
makeNodeStrict(property, isUnsupportedKeyword);
176+
if (!required.has(key) && !schemaAllowsNull(property)) {
177+
properties[key] = { anyOf: [property, { type: "null" }] };
178+
}
179+
}
180+
schema.required = propertyNames;
181+
schema.additionalProperties = false;
182+
}
183+
184+
/**
185+
* Convert a tool's parameter schema to the subset strict constrained sampling accepts: every object closed and
186+
* every property required, with formerly optional properties made nullable. Callers strip those nulls again
187+
* before validation (see `normalizeOptionalNulls`).
188+
*/
189+
function makeStrictJsonSchema(schema: unknown, isUnsupportedKeyword?: UnsupportedKeyword): Record<string, unknown> {
190+
const cloned: unknown = structuredClone(schema);
191+
if (!isJsonSchemaObject(cloned) || cloned.type !== "object") {
192+
throw new UnsupportedStrictSchemaError("root schema must have type object");
193+
}
194+
makeNodeStrict(cloned, isUnsupportedKeyword);
195+
return cloned;
196+
}
197+
198+
/**
199+
* The strict parameter schema to send for a JSON-schema constrained tool, or `undefined` to send it non-strict.
200+
* A `"prefer"` tool whose schema the model's strict mode cannot express falls back to non-strict; a `"require"`
201+
* tool throws instead.
202+
*/
203+
export function resolveStrictParameters(
204+
tool: Message.Tool,
205+
model?: Pick<Model.Info, "protocol" | "compat">,
206+
): Record<string, unknown> | undefined {
207+
const config = tool.constrainedSampling;
208+
if (!config || config.type !== "json_schema") return;
209+
210+
if (model?.compat?.supportsStrictMode ?? true) {
211+
const isAnthropic =
212+
model?.protocol === Model.KnownProviderEnum.anthropic ||
213+
model?.protocol === Model.KnownProviderEnum.googleVertexAnthropic;
214+
try {
215+
return makeStrictJsonSchema(tool.parameters, isAnthropic ? isAnthropicUnsupportedKeyword : undefined);
216+
} catch (error) {
217+
if (!(error instanceof UnsupportedStrictSchemaError)) throw error;
218+
if (config.strict !== "require") return;
219+
throw new Error(`tool "${tool.name}" requires JSON-schema constrained sampling, but ${error.message}.`);
220+
}
221+
}
222+
if (config.strict === "require") {
223+
throw new Error(
224+
`tool "${tool.name}" requires JSON-schema constrained sampling, but strict tools are unsupported.`,
225+
);
226+
}
227+
}

‎packages/aikit/src/llm/transform.ts‎

Lines changed: 5 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -16,6 +16,7 @@ import { resolveOpenAICodexToolConstraint } from "../providers/openai-codex/code
1616
import { shortHash } from "../utils/hash.ts";
1717
import { parseStreamingJson } from "../utils/jsonparse.ts";
1818
import { sanitizeSurrogates } from "../utils/sanitize.ts";
19+
import { resolveStrictParameters } from "./strict.ts";
1920

2021
type ToolCallPart = Extract<Message.AssistantMessage["parts"][number], { type: "toolCall" }>;
2122
type TerminalToolCall = Exclude<ToolCallPart, Message.ToolCallPendingPart | Message.ToolCallRunningPart>;
@@ -367,18 +368,16 @@ export function convertTools(tools?: Message.Tool[], model?: Model.Info): ToolSe
367368

368369
const result: ToolSet = {};
369370
for (const tool of tools) {
370-
const jsonSchemaConstraint = Message.resolveJsonSchemaConstraint(tool, model?.compat);
371+
const strictParameters = resolveStrictParameters(tool, model);
371372
const grammarConstraint =
372373
model?.protocol === Model.KnownProviderEnum.openaiCodex
373374
? resolveOpenAICodexToolConstraint(tool, model.compat)
374375
: undefined;
375376
result[tool.name] = {
376377
description: tool.description,
377-
inputSchema: jsonSchema(tool.parameters as any),
378-
...(jsonSchemaConstraint ? { strict: true } : {}),
379-
...(grammarConstraint?.type === "grammar"
380-
? { providerOptions: { "openai-codex": { grammar: grammarConstraint } } }
381-
: {}),
378+
inputSchema: jsonSchema(strictParameters ?? (tool.parameters as any)),
379+
...(strictParameters ? { strict: true } : {}),
380+
...(grammarConstraint ? { providerOptions: { "openai-codex": { grammar: grammarConstraint } } } : {}),
382381
};
383382
}
384383
return result;

‎packages/aikit/src/message/message.ts‎

Lines changed: 0 additions & 14 deletions
Original file line numberDiff line numberDiff line change
@@ -285,20 +285,6 @@ export function defineTool<TParameters extends TSchema>(tool: Tool<TParameters>)
285285
return tool;
286286
}
287287

288-
export function resolveJsonSchemaConstraint(
289-
tool: Tool,
290-
compat?: Pick<Model.Compatibility, "supportsStrictMode">,
291-
): { type: "json_schema"; strict: true } | undefined {
292-
const config = tool.constrainedSampling;
293-
if (!config || config.type !== "json_schema") return;
294-
if (compat?.supportsStrictMode ?? true) return { type: "json_schema", strict: true };
295-
if (config.strict === "require") {
296-
throw new Error(
297-
`tool "${tool.name}" requires JSON-schema constrained sampling, but strict tools are unsupported.`,
298-
);
299-
}
300-
}
301-
302288
export const ContextSchema = Type.Object({
303289
systemPrompt: Type.Optional(Type.String()),
304290
messages: Type.Array(MessageSchema),

‎packages/aikit/src/providers/openai-codex/codex-tools.ts‎

Lines changed: 2 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -94,10 +94,9 @@ function inferGrammarInputProperty(parameters: TSchema): string {
9494
export function resolveOpenAICodexToolConstraint(
9595
tool: Message.Tool,
9696
compat?: OpenAICodexCompatibility,
97-
): OpenAICodexGrammarConstraint | { type: "json_schema"; strict: true } | undefined {
97+
): OpenAICodexGrammarConstraint | undefined {
9898
const config = tool.constrainedSampling;
99-
if (!config) return undefined;
100-
if (config.type === "json_schema") return Message.resolveJsonSchemaConstraint(tool, compat);
99+
if (!config || config.type !== "grammar") return undefined;
101100
if (!(compat?.supportsOpenAIGrammarTools ?? false)) return undefined;
102101

103102
const lark = config.variants.openai_lark;

‎packages/aikit/src/utils/validation.ts‎

Lines changed: 48 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -6,6 +6,7 @@ import type { Static, TSchema } from "typebox";
66
import Schema from "typebox/schema";
77
import Value from "typebox/value";
88

9+
import { nullableVariant, schemaAllowsNull } from "../llm/strict.ts";
910
import type * as Message from "../message/message.ts";
1011

1112
const validators = new WeakMap<TSchema, ReturnType<typeof Schema.Compile>>();
@@ -28,6 +29,50 @@ function errorPath(error: { instancePath?: string; params?: Record<string, unkno
2829
return "root";
2930
}
3031

32+
type JsonSchemaObject = {
33+
properties?: Record<string, JsonSchemaObject>;
34+
required?: string[];
35+
items?: JsonSchemaObject | JsonSchemaObject[];
36+
$ref?: unknown;
37+
};
38+
39+
/**
40+
* Drop `null` from optional properties whose schema does not accept null, in place. Strict constrained sampling
41+
* makes every property required and nullable, so the model answers an omitted optional argument with `null`.
42+
*/
43+
export function normalizeOptionalNulls(value: unknown, schema: unknown): void {
44+
const variant = nullableVariant(schema);
45+
if (variant !== undefined) return normalizeOptionalNulls(value, variant);
46+
const node = schema as JsonSchemaObject;
47+
if (Array.isArray(value)) {
48+
if (Array.isArray(node.items)) {
49+
for (const [index, item] of value.entries()) {
50+
if (node.items[index]) normalizeOptionalNulls(item, node.items[index]);
51+
}
52+
} else if (node.items) {
53+
for (const item of value) normalizeOptionalNulls(item, node.items);
54+
}
55+
return;
56+
}
57+
if (typeof value !== "object" || value === null || !node.properties) return;
58+
59+
const object = value as Record<string, unknown>;
60+
const required = new Set(node.required ?? []);
61+
for (const [key, property] of Object.entries(node.properties)) {
62+
if (!(key in object)) continue;
63+
if (
64+
object[key] === null &&
65+
!required.has(key) &&
66+
typeof property.$ref !== "string" &&
67+
!schemaAllowsNull(property)
68+
) {
69+
delete object[key];
70+
} else {
71+
normalizeOptionalNulls(object[key], property);
72+
}
73+
}
74+
}
75+
3176
/**
3277
* Validates an arbitrary value against a TypeBox schema and returns the coerced value.
3378
*/
@@ -78,9 +123,7 @@ export function validateToolArguments<T extends Message.Tool>(
78123
tool: T,
79124
toolExecution: Message.ToolCallInFlight,
80125
): Message.ToolArguments<T> {
81-
return validateSchema(
82-
tool.parameters,
83-
toolExecution.rawArgs,
84-
`Tool "${toolExecution.name}"`,
85-
) as Message.ToolArguments<T>;
126+
const args = structuredClone(toolExecution.rawArgs);
127+
normalizeOptionalNulls(args, tool.parameters);
128+
return validateSchema(tool.parameters, args, `Tool "${toolExecution.name}"`) as Message.ToolArguments<T>;
86129
}

‎packages/harness/src/plugin/builtin/tool/bash.ts‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -86,6 +86,7 @@ export const bashDef = Tool.define({
8686
"note says which lines are shown and gives the path of a file holding the full output when one was saved. " +
8787
"A non-zero exit code is reported as an error carrying the captured output.",
8888
parameters: BashParams,
89+
constrainedSampling: { type: "json_schema", strict: "prefer" },
8990
success: BashSuccess,
9091
failure: BashFailure,
9192
// The model reads the command output, not the JSON envelope — and, on failure, why it failed.

0 commit comments

Comments
 (0)