diff --git a/.gitignore b/.gitignore index f246a24..eb6b5a3 100644 --- a/.gitignore +++ b/.gitignore @@ -39,3 +39,9 @@ dist/ __pycache__/ .release-*.md .DS_Store + +# Benchmark harness: dataset cache (TRAIL is evaluation-only and must never be +# committed or reshared), score cache, results and the virtualenv. +bench/.cache/ +bench/results/ +bench/.venv/ diff --git a/CHANGELOG.md b/CHANGELOG.md index d51c51e..d9adddb 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -4,6 +4,88 @@ All notable changes to Tailwise are recorded here. The format follows [Keep a Changelog](https://keepachangelog.com/en/1.1.0/), and versions follow [Semantic Versioning](https://semver.org/spec/v2.0.0.html). +## [Unreleased] + +## [0.2.0] - 2026-11-05 + +Ranks traces for review, checks heuristic rules for free, and reads +code-writing agents. Benchmarked on τ-bench, SWE-agent and TRAIL: per 10,000 +traces with 10% failing, a team reviewing 500 finds 110 to 258 failures where +random review finds 50, for $1.31 to $3.58 of Jev calls (figures in +`bench/published/benchmark.json`). + +### Changed + +- `scorer.evidence` defaults to `grounded`: the model sees each tool call's + result on its line, preferring the results later calls rely on. With + `first`, it saw only the first three results, so on τ-bench it flagged what + it could not check. With `grounded`, τ-bench ROC-AUC rises from 0.59 to 0.64 + (0.63 to 0.71 on the runs whose label the trace can show), false alarms on + successful runs fall from 57% to 33%, and SWE-agent ROC-AUC rises from 0.79 + to 0.80 on a held-out sample. It costs about twice the input tokens. Set + `evidence: first` for the old behaviour. + +- `severity` is the deciding questions' average answer, on every scored trace, + instead of the priority-weighted sum over the flags only. It is the order to + review traces in: a trace several checks doubt ranks above one a single + check is sure of. On τ-bench with the action checks (trace-detectable + runs), when 10% of runs fail and a team reviews the top 5%, the share of + reviewed traces that are failures rises from 23% (ranked by `p_keep`) to + 47% (95% interval 26–66%). The benchmark's headline metric is now that + share. The response's `severity` is `null` on fallback rather than omitted. Rank-order priority weights are gone. A check + that failed is left out of the average rather than counted as 0, so a + scorer outage does not bury failed traces; `action_wrong` on a trace without + actions is 0. + +### Added + +- `bench publish` writes `bench/published/benchmark.json`, the figures the + website shows: failures found per 500 reviews against random review, the + Jev cost per 10,000 traces, and their 95% intervals. The benchmark reports + gain a cost section comparing random review with Tailwise, and each run's + review figures resample whole tasks, so the intervals hold for tasks the + benchmark did not see. Rules run over the exact requests through + `bench/rulecheck`, at no cost. + +- Heuristic rules: free, certain checks that need no model, in the config + (every agent) and in a manifest (one agent; same-name rules override, + `enabled: false` removes). They check the final answer, tool outputs and + arguments (length, regular expressions, JSON, JSON Schema, `ok_status`), + and the run (`max_tool_calls`, `max_repeated_calls`, `requires_tools`, + `forbids_tools`). `on_fail: keep` keeps the trace without asking the + model (rule `keep_rule`, severity 1); `signal` counts 1 or 0 in severity. + Fired rules are in the response's `rules`, the log and `eval.rules`. + +- Action checks, turned on by a manifest `actions` block naming the tools that + act for the user. `action_wrong` checks each action call's arguments against + the user's request and the lookups they came from; `unhandled_request` + checks that every request of those kinds was carried out or declined. Both + can keep a trace, with a reason naming the action or the request. On + τ-bench they raise the failures caught from 54% to 70% (wrong-variant + exchanges from 33% to 90%), keeping 40% of successful runs instead of 26%. +- Traces carry the conversation, from the latest LLM span's messages or the + JSON API's `conversation`, for the completeness check. +- Code-writing agents, such as a smolagents `CodeAgent`, are read: each step + under the `AGENT` root whose model wrote code becomes an `execute_code` + tool call, with the code as its arguments, what it printed as its output, + and the step's error as its status. Such agents record no TOOL spans for + their actions, so the model saw only the question and the answer. On + TRAIL's SWE-bench runs, ROC-AUC rose from 0.33 to 0.67. + +### Fixed + +- An `AGENT` root span with no `output.value` (the run crashed or ran out of + steps) is scored with an empty final answer instead of being discarded as + not scorable. Those runs are failures, and they were dropped unreviewed. +- A trace too long for the model is no longer dropped at random. The middle of + its longest fields (a giant patch, a long log) is cut until the state fits + in 40,000 characters, and the decision says so: `trimmed` in the response + and decision log, `eval.trimmed` on the exported trace. +- A root span whose input is `{"task": ...}`, as smolagents records it, gives + the task as the question instead of the raw JSON. +- A tool call whose output starts with `Error:` is read as failed, even when + its span status is OK, as many tools report failures that way. + ## [0.1.0] - 2026-10-27 The first release of the Tailwise inference engine. @@ -25,15 +107,33 @@ The first release of the Tailwise inference engine. Manifests set the questions a model is asked and are optional. - Five built-in checks asked alongside `answer_wrong` in the same call: `unsupported_answer`, `missing_tool_calls`, `answered_on_failure`, - `ignored_instructions` and `contradicts_knowledge`. A trace is kept when any - of them reaches its threshold. -- A manifest `decision` block: `any_of` lists the questions that can keep a - trace, in priority order after the decision question, and `thresholds` - gives any of them its own threshold. + `ignored_instructions` and `contradicts_knowledge`. Their answers are + recorded with every decision; `answer_wrong` alone decides keep or drop. +- A manifest `decision` block (none in the built-in questions): `any_of` + lists more questions that can keep a trace, in priority order after the + decision question, and `thresholds` gives any of them its own threshold. - Flags and severity on every decision: the questions that reached their threshold, and a priority-weighted score that ranks kept traces for review. Both appear in the decision log, with every answer, and on exported traces as `eval.flags`, `eval.severity` and `eval.answers.`. +- `scorer.system_prompt` (off by default) shows the model the agent's system + prompt, above the question. It comes from the JSON API's `system_prompt`, + the root span's `tailwise.system_prompt`, or the first LLM span's system + message, and is cut to 8,000 characters. Off, or without a prompt, the + model sees exactly what it did before. +- `scorer.evidence: grounded` (experimental; the default, `first`, is + unchanged) shows each tool call's result on its line, within + `scorer.evidence_budget` characters, preferring the results later calls + rely on. The default shows only the first three tool results, so on long + runs the model never sees most of what the agent looked up. Tool calls carry + their `output` (JSON API and OTLP). +- `POST /v1/otlp/traces:score`: scores one trace sent as OTLP spans, + synchronously, built exactly as the OTLP receivers build it. +- Score responses include every answer, the flags, the severity and the + input tokens Jev reported billing. +- A benchmark harness in `bench/` that measures how well Tailwise picks + failed agent runs compared with random sampling, on τ-bench and SWE-agent + runs (and TRAIL). - One model per agent, with its own questions, endpoint or key. - The engine's own metrics and errors over OpenTelemetry. - Docker Compose for a local stack, a Helm chart for Kubernetes, and diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 2e0b1a4..347ac65 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -61,7 +61,9 @@ itself, not a fake. `trace.BuildState` renders a trace into the text the model reads. Tool arguments, and any status that is not a string, are written as compact JSON (`search({"query":"capital of France","limit":3}) -> ok`), and evidence is -cut to 1500 Unicode code points rather than bytes. +cut to 1500 Unicode code points rather than bytes. A system prompt opens the +slice, cut to 8000 code points; without one its section is left out, so the +slice is what it was before system prompts existed. - `testdata/traces/*.json`: input fixtures - `testdata/golden/*.txt`: the slice for each, as `BuildState` writes it diff --git a/README.md b/README.md index fc52e72..14f97a1 100644 --- a/README.md +++ b/README.md @@ -229,6 +229,9 @@ together. [architecture.md](architecture/architecture.md) explains how a file is | `scorer.manifest` | none | Optional manifest: the questions the model is asked. See [Manifests](#manifests) | | `scorer.decision_question` | `answer_wrong` | Question that drives the decision; must be `noul` | | `scorer.reason_question` | `why_wrong` | Choice question whose chosen option is the plain-English `reason`; see [Reasons](#reasons). Empty turns it off | +| `scorer.system_prompt` | `false` | Show the model the agent's system prompt, above the question (see [How spans are read](#how-spans-are-read)). Off by default: in the [benchmark](bench/README.md) it kept more traces without ranking failures better, and it adds the prompt's tokens to every call. Each `agents.models` entry sets its own | +| `scorer.evidence` | `grounded` | How tool results reach the model. `grounded`: each call's result on its line, within `scorer.evidence_budget`, choosing first the results whose values later calls use as arguments, then the most recent; a long result keeps the parts later calls used. `first`: the first three evidence items, each cut to 1,500 characters; tool-call lines show only the status. In the [benchmark](bench/README.md), `first` hid most tool results and the model flagged what it could not check, so it ranked failures worse at about half the cost | +| `scorer.evidence_budget` | `12000` | With `evidence: grounded`, the most characters of tool results shown across all calls | | `scorer.timeout` | `30s` | Timeout per call | | `scorer.max_retries` | `3` | Retries after the first attempt, on the synchronous endpoint only; a queued trace is redelivered by the queue instead | | `scorer.backoff` | `[1s, 5s, 15s]` | Base delay before each synchronous retry; the last value repeats | @@ -312,12 +315,24 @@ engine logs a warning at start-up. They are: | `contradicts_knowledge` | Does it contradict general knowledge? | | `why_wrong` | Which [reason](#reasons) explains what is wrong? | -Each yes/no answer is the probability that the agent failed in that way, and -any of them can keep the trace. To word the questions +Each yes/no answer is the probability that the agent failed in that way. +`answer_wrong` alone decides keep or drop; the other answers are recorded +with every decision (in the log and as `eval.answers.` on the +export) so a reviewer sees which problems the model found. In the +[benchmark](bench/README.md), letting any check keep a trace kept most traces +without ranking failures any better, so the built-in questions have no +`decision` block; a manifest can add one. To word the questions for your agent, start from [`examples/manifests/my-agent.json`](examples/manifests/my-agent.json) and set `scorer.manifest` (or `TAILWISE_MANIFEST`). +How well each check tells failed runs from successful ones depends on the +agent. In the benchmark, `unsupported_answer` was informative on coding +agents (ROC-AUC 0.78 on SWE-agent) but no better than chance on a +customer-service agent whose answers cite its own lookups; `missing_tool_calls` +was informative on both. Treat a check as a hint until you have seen it on +your own agent's traces. + ### Any question can keep a trace A manifest's `decision` block lists the yes/no questions that can keep a @@ -345,14 +360,24 @@ them its own threshold: decision log line (`flags`) and on the exported trace (`eval.flags`), next to every answer (`eval.answers.`). A reviewer can filter on them, for example every kept trace flagged `answered_on_failure`. -- **Priority and severity.** The decision question always ranks first, then - `any_of` in the order written. Ranks become weights that sum to 1, falling - steeply from the top ([rank-order centroid](https://doi.org/10.1287/mnsc.42.11.1515) - weights: with all six built-in questions, 0.41, 0.24, 0.16, 0.10, 0.06 and - 0.03). A trace's **severity** is the sum of weight × answer over its flags: - higher for more problems, and for higher-priority ones. It does not change - whether a trace is kept; it ranks kept traces for review (`severity` in the - log, `eval.severity` on the export). +- **Priority.** The decision question always comes first, then `any_of` in + the order written, then the action checks. Flags are listed in that order. +- **Severity: the review order.** A trace's **severity** is the average + answer of its deciding questions, flagged or not. `action_wrong` on a + trace without actions is 0; a check that failed is left out of the average + rather than counted as 0, so an outage does not bury failed traces. Review + traces highest severity first. It does not change whether a trace is kept + (`severity` in the response and the log, `eval.severity` on the export). + - Why an average and not the highest answer: a trace several checks doubt + is more often a real failure than one a single check is sure of. On + τ-bench with the action checks, when 10% of runs fail and a team reviews + the top 5%, ranking by the highest answer finds 23% failures among the + traces reviewed; ranking by the average finds 47% + ([benchmark](bench/README.md#results)), on the runs whose label the trace + can show. + - Every deciding question counts the same, so list in `any_of` only the + checks you trust on your agent: a check no better than chance pulls good + traces up. - `p_keep` is the highest answer among the deciding questions. - Every listed question must be a `noul` question of the same manifest, listed once, and the model must answer all of them: a missing answer is a @@ -365,6 +390,115 @@ them its own threshold: Without a `decision` block, the decision question decides alone, against `policy.keep_threshold`. +### Checking an agent's actions + +An agent that acts for its users (cancels orders, books flights, sends +money) fails in ways its final answer hides: it exchanges the wrong item, or +never does one of the things the user asked, and then reports success. Name +the tools that act in the manifest's `actions` block, each with what it does, +worded as the user would ask: + +```json +"actions": { + "cancel_order": "cancel an order", + "issue_refund": "get a refund for an order", + "update_shipping_address": "change the shipping address of an order" +} +``` + +Each trace then gets two more checks, each its own call to the model: + +- **`action_wrong`**, once per action call (the last 10): does the call use + an argument that is not what the user asked for? The model sees the user's + request, the earlier tool results that hold the call's argument values + (where the right item or order is identified, trimmed to the parts holding + them), the call, and the actions after it. The trace's answer is the + highest over its calls. +- **`unhandled_request`**, once per trace: for each action, did the user ask + for it, and was every such request carried out or explicitly declined? The + model sees the conversation (or, without one, the user's request), the + actions taken and the final answer. The answer is the highest + min(requested, 1 − handled) over the actions. + +Both are deciding checks after the decision question and any `any_of` +questions, with `policy.keep_threshold` unless `decision.thresholds` sets +their own. They are recorded in `answers` and flags like any question. When +one of them, and not the decision question, keeps a trace, the reason says +what it found: `cause` `wrong_action` ("The agent's action +cancel_order({"order_id":"#W2"}) does not match what the user asked for.") +or `unhandled_request` ("The user asked the agent to get a refund for an +order, and it neither did it nor declined."). A check that fails is left out +of the decision, and reported; on the queued path a system error redelivers +the trace instead. + +**The conversation.** The completeness check reads the agent's turns: they +show a request it declined, or one the user withdrew. Tailwise takes them +from the agent's latest LLM span (the one with the longest +`llm.input_messages`, plus its `llm.output_messages`), or from the JSON API's +optional `conversation`, a list of `{"role": "user" | "assistant", +"content": ...}`. Without them, the check sees only the user's request and +the final answer, and flags far more successful runs (26% against 11% on +τ-bench). + +**What it costs and buys.** On τ-bench's trace-detectable runs, with +`grounded` evidence, the checks raise the failures kept from 54% to 70%, and +the successful runs kept from 26% to 40%; the share of kept runs that failed +barely moves (60% to 57%). They catch the failures the final answer hides: +wrong-variant exchanges the user had described go from 33% to 90%, runs that +left out an action from 50% to 68%. Each trace costs one call per action +call plus one, about 2,000 input tokens each. + +### Heuristic rules + +Some failures need no model to spot: a final answer that should be JSON and +is not, a tool call that errored, an agent calling the same tool with the +same arguments again and again. Write them as rules. They cost nothing, +their verdict is certain, and each fired rule is listed with a plain reason. + +Rules in the config file apply to every agent. An agent's manifest adds its +own under `"rules"`, in the same form: a rule with the same name replaces +the default, and `"enabled": false` turns it off for that agent. + +```yaml +rules: + - name: answer_json + on_fail: keep # certain: keep without asking the model + final_answer: {json_schema: {type: object, required: [order_id, status]}} + - name: tool_failed + on_fail: signal # raises severity, decides nothing + tool_output: {ok_status: true} + - name: looping + run: {max_repeated_calls: 3} +``` + +Each rule sets one of: + +| Key | Checks | Expectations | +| --- | --- | --- | +| `final_answer` | the final answer | `not_empty`, `min_length`, `max_length` (characters), `matches`, `not_matches` (Go regular expressions), `json`, `json_schema` | +| `tool_output` | the output of every call, or every call to `tool` | the same, plus `ok_status`: the call's status is `ok` | +| `tool_args` | the arguments of every call, or every call to `tool` | `json_schema` | +| `run` | the run as a whole | `max_tool_calls`, `max_repeated_calls` (one call, same tool and arguments, more than this many times), `requires_tools`, `forbids_tools` | + +A rule fires when the trace breaks any of its expectations. `on_fail` says +what happens then: + +- **`keep`**: the trace is kept without asking the model, so it costs no + tokens. Rule `keep_rule`, cause `rule`, the rule's reason ("The final + answer does not match the schema: at '': missing property 'status'."), + severity 1, so it is reviewed first, and `p_keep` null. Use it for + failures that are certain. +- **`signal`** (the default): the rule counts in severity, 1 when it fires + and 0 when it does not, next to the model's answers, and decides nothing. + Every signal rule weighs as much as a check, so a rule that fires on many + good traces pulls them up the review order: keep signal rules few and + specific. + +Fired rules are in the response's `rules` (`name`, `on_fail`, `reason`), the +log line's `rules`, and the export's `eval.rules` and `eval.rules.`. A +rule that does not compile (a bad regular expression or schema, two rules +with one name) stops Tailwise at start-up, naming the model and the rule. + ### One model per agent Each agent can have its own model: its own questions, and @@ -513,16 +647,17 @@ attributes that Phoenix's instrumentors emit. | --- | --- | | Root span | The span whose parent is not in the trace, preferring `openinference.span.kind` = `AGENT`, then `CHAIN` | | `agent` | The first of `agents.attributes` set on the root span, else on its resource | +| `system_prompt` | `tailwise.system_prompt` on the root span if set; else the system message of the first `LLM` span that has one (`llm.input_messages..message.role` = `system`, its `content` or its `contents` parts joined) | | `question` | Root span `input.value` | | `final_answer` | Root span `output.value` | -| `tool_calls` | Spans with kind `TOOL`, ordered by start time | +| `tool_calls` | Spans with kind `TOOL`, and the code steps of an `AGENT` root, ordered by start time. A code step is a `CHAIN` child of the root with no `TOOL` span under it, whose `LLM` span wrote a fenced or `` block, as in a smolagents `CodeAgent`: it becomes an `execute_code` call with the code as `{"code": ...}`, its `output.value` as the output, and an error status (with the span's status message) when the step failed | | tool `name` | `tool.name`, else the span name | | tool `args` | `input.value` if it is JSON, else `tool.parameters`, else `input.value` as a string | -| tool `status` | `tailwise.tool.status` if set; otherwise `error` when the span status is ERROR, else `ok` | +| tool `status` | `tailwise.tool.status` if set; otherwise `error` when the span status is ERROR or the output starts with `Error:` (any case), else `ok` | | `evidence` | First of: `tailwise.evidence.` on the root span; documents on `RETRIEVER` spans; the paragraphs (split on blank lines) of successful tool outputs | When `input.value` or `output.value` holds a JSON object, the text is taken -from a `question`/`input`/`query`/`prompt` key (or `answer`/`output`/`result` +from a `question`/`input`/`query`/`prompt`/`task` key (or `answer`/`output`/`result` for outputs) when there is one. A tool output shaped as a function response, `{"response": {"result": "..."}}`, is unwrapped. @@ -532,12 +667,15 @@ A message recorded by the Google ADK instrumentor, its parts, so an ADK agent's own root span works without extra attributes. A trace is **not scorable**, and is discarded with a `spans discarded` log -line, when its root span has no `input.value` or no `output.value`. That -covers ordinary service spans sent to the engine by mistake. +line, when its root span has no `input.value`, or has no `output.value` and +is not an `AGENT` span. That covers ordinary service spans sent to the engine +by mistake. -An agent run that ended without an answer is scored like any other, as long -as its root span sets `output.value`, even to an empty string. The model then -sees an empty final answer. +An agent run that ended without an answer is scored like any other, with an +empty final answer: its root span either sets `output.value` to an empty +string, or is an `AGENT` span with no `output.value`, as when the run crashed +or ran out of steps. Those runs nearly always failed, so they reach review +instead of being dropped. ### Matching what the model was trained on @@ -555,6 +693,7 @@ An agent can remove the difference by setting two attributes itself: | --- | --- | --- | | `tailwise.evidence.0`, `tailwise.evidence.1`, ... | the root span | the evidence items, in order | | `tailwise.tool.status` | each `TOOL` span | the tool call's status | +| `tailwise.system_prompt` | the root span | the agent's system prompt, when its LLM spans do not record it or record a sub-agent's first | With both set, a trace sent as spans yields byte for byte the slice the same trace yields through the JSON API. A test checks this for every fixture, and @@ -572,6 +711,7 @@ unless the embedded server runs without `queue.store_dir`. { "trace_id": "7f3a91c2e8", "agent": "support-bot", + "system_prompt": "You answer geography questions. Cite your sources.", "question": "Which river flows through the capital of France?", "tool_calls": [{"name": "search", "args": {"query": "capital of France"}, "status": "ok"}], "evidence": ["Paris is the capital of France.", "The Seine flows through Paris."], @@ -583,7 +723,16 @@ unless the embedded server runs without `queue.store_dir`. string for a run that ended without an answer. - `trace_id` is optional; the engine generates 32 hex characters if it is missing. - `agent` is optional and picks the model (see [One model per agent](#one-model-per-agent)); at most 128 bytes. -- `args` may be any JSON value. `status` defaults to `ok`. +- `conversation` is optional: the user's and agent's turns, each + `{"role": "user" | "assistant", "content": "..."}`, for the + [action checks](#checking-an-agents-actions). +- `system_prompt` is optional: the instructions the agent ran under. A model + with `scorer.system_prompt: true` sees it, cut to 8,000 characters, above + the question; otherwise it is ignored and the model sees exactly what it + would without one. +- `args` may be any JSON value. `status` defaults to `ok`. A tool call's + optional `output` is what it returned; models with + `scorer.evidence: grounded`, the default, see it. - This is the engine's own JSON API on port 8080. The OTLP receiver also serves a `/v1/traces` path, on port 4318, with a different body. @@ -612,19 +761,41 @@ slow back end never holds up the response. "cause": "contradicts_evidence", "model": "jev-1.14.0", "latency_ms": 357.8, - "exported": true + "exported": true, + "answers": {"answer_wrong": 0.98, "unsupported_answer": 0.12}, + "flags": ["answer_wrong"], + "severity": 0.98, + "input_tokens": 412 } ``` | Field | Values | | --- | --- | | `decision` | `keep` or `drop` | -| `p_keep` | The model's probability that the final answer is wrong (its `noul` for the decision question); `null` on fallback | -| `rule` | Which part of the policy decided: `above_threshold`, `random_floor`, `below_threshold` or `fallback` | +| `p_keep` | The highest answer among the questions that decide (the decision question alone without a `decision` block); `null` on fallback and when a keep rule decided | +| `rule` | Which part of the policy decided: `above_threshold`, `random_floor`, `below_threshold`, `fallback` or `keep_rule` (see [Heuristic rules](#heuristic-rules)) | | `reason` | Why the model thinks the final answer is wrong, or that it is right, in plain English (see [Reasons](#reasons)) | | `cause` | The model's short name for that reason, e.g. `contradicts_evidence` or `correct`; omitted when it gave none | | `latency_ms` | The model's reported latency; the measured time if it reports none, or on fallback | | `exported` | The trace was kept and queued to at least one exporter, which sends it right after | +| `answers` | Every yes/no answer the model gave, by question; omitted on fallback | +| `severity` | The deciding questions' average answer, with each signal rule counting 1 or 0: the order to review traces in, highest first (see [Any question can keep a trace](#any-question-can-keep-a-trace)); 1 when a keep rule decided; `null` on fallback | +| `flags` | The questions that reached their threshold, in priority order; omitted when there are none | +| `rules` | The [heuristic rules](#heuristic-rules) that fired, each `{name, on_fail, reason}`; omitted when none did | +| `input_tokens` | The input tokens the model reported billing for the call; omitted if it reported none | +| `trimmed` | `true` when the trace was too long for the model (over 40,000 characters as rendered), so the middle of its longest fields, such as a giant patch, was cut before scoring; omitted otherwise | + +### `POST /v1/otlp/traces:score` + +Scores one trace sent as OTLP spans, synchronously, and returns the same +response as `POST /v1/traces:score`. The body is an OTLP/HTTP export request +(`application/x-protobuf` or `application/json`, optionally gzipped) holding +the spans of exactly one trace. The trace is built from the spans exactly as +the OTLP receivers build it (see [From spans to a decision](#from-spans-to-a-decision)), +but without the queue or the wait for late spans, so send every span in one +request. It answers 400 for spans of more than one trace, and 422 when no +root span has a question and an answer. Meant for testing and benchmarks: +[bench/](bench/README.md) scores datasets through it. ### API docs: `GET /docs` and `GET /openapi.yaml` @@ -650,25 +821,28 @@ so it does not take the engine out of a load balancer; watch 1. **Route.** The trace's agent picks its model (see [One model per agent](#one-model-per-agent)). -2. **Slice.** `trace.BuildState` renders the trace into the text the model +2. **Rules.** The [heuristic rules](#heuristic-rules) run. When a keep rule + fires, the trace is kept (rule `keep_rule`) and the model is not asked. +3. **Slice.** `trace.BuildState` renders the trace into the text the model reads (see [The slice](CONTRIBUTING.md#the-slice-and-its-golden-files)). -3. **Score.** The slice and the model's questions (its manifest's, or the +4. **Score.** The slice and the model's questions (its manifest's, or the built-in ones) go to the model, in one call. A queued trace that hits a system error goes back to the queue for redelivery. On the synchronous endpoint, which has no queue, the engine retries it with `scorer.max_retries` and jittered `scorer.backoff`. The two kinds of retry never multiply. -4. **Decide.** The trace is kept when any question that decides (the +5. **Decide.** The trace is kept when any question that decides (the decision question, and those in the manifest's `decision.any_of`) reaches its threshold; those questions are its flags (see [Any question can keep a trace](#any-question-can-keep-a-trace)). - A question reached its threshold: keep, rule `above_threshold`. - Otherwise keep with probability `random_floor`, rule `random_floor`. - Otherwise drop, rule `below_threshold`. -5. **Export.** Kept traces are queued to every exporter. Each decision produces one JSON log - line with `trace_id`, `agent`, `decision`, `rule`, `reason`, `cause`, `p_keep`, `model`, `latency_ms`, +6. **Export.** Kept traces are queued to every exporter. Each decision produces one JSON log + line with `trace_id`, `agent`, `decision`, `rule`, `reason`, `cause`, `p_keep`, `severity`, `model`, `latency_ms`, `answers`, every yes/no answer by question, `flags`, the deciding - questions that reached their threshold, and `severity`, as in - `"answers": {"answer_wrong": 0.91, "unsupported_answer": 0.84, ...}, "flags": ["answer_wrong", "unsupported_answer"], "severity": 0.57`. + questions that reached their threshold, `rules`, the heuristic rules + that fired, as in + `"answers": {"answer_wrong": 0.91, "unsupported_answer": 0.84, ...}, "flags": ["answer_wrong", "unsupported_answer"]`. ### Reasons @@ -936,15 +1110,18 @@ Root span attributes: | `tailwise.evidence.` | string | rebuilt | Each evidence item, 0-based, untruncated | | `eval.agent` | string | both | The trace's agent; empty if it named none | | `eval.decision` | string | both | `keep` (dropped traces are not exported) | -| `eval.rule` | string | both | Which part of the policy kept it: `above_threshold`, `random_floor`, or `fallback` (the model could not score it) | +| `eval.rule` | string | both | Which part of the policy kept it: `above_threshold`, `random_floor`, `fallback` (the model could not score it), or `keep_rule` (a heuristic rule) | | `eval.reason` | string | both | Why the model thinks the final answer is wrong, in plain English | | `eval.cause` | string | both | The model's short name for the reason, e.g. `contradicts_evidence`; omitted when it gave none | | `eval.model` | string | both | The scorer model that decided | | `eval.latency_ms` | double | both | Scoring time in milliseconds | | `eval.p_keep` | double | both | The highest answer among the questions that decide; omitted on a fallback keep | | `eval.flags` | string | both | The deciding questions that reached their threshold, comma-separated in priority order, e.g. `answer_wrong,unsupported_answer`; omitted when none did | -| `eval.severity` | double | both | Sum of priority weight × answer over the flags; omitted when there are none | +| `eval.severity` | double | both | The deciding questions' average answer with the signal rules, the review order; 1 for a keep rule; omitted on fallback | +| `eval.rules` | string | both | The heuristic rules that fired, comma-separated; omitted when none did | +| `eval.rules.` | string | both | Each fired rule's reason | | `eval.answers.` | double | both | Each yes/no answer, e.g. `eval.answers.missing_tool_calls`; omitted on a fallback keep | +| `eval.trimmed` | bool | both | `true` when the model saw the trace with the middle of its longest fields cut to fit; omitted otherwise | Tool span attributes (rebuilt traces only; forwarded tool spans are not changed): @@ -1055,7 +1232,9 @@ These are open problems in the current version, not design choices to rely on. which flags keep them, and adjust `decision.thresholds` or the `any_of` list. - **Severity ranks traces but nothing acts on it yet.** Exporters send kept - traces in the order they are decided, not by severity. + traces in the order they are decided, not by severity, and nothing keeps a + fixed share such as the top 5%: the keep decision is still each check's + threshold. Sort by `eval.severity` in your back end to review in order. - **Every scored trace is a billed call to Jev, and leaves your network.** Nothing caps the spend or redacts fields before sending; only `server.workers` limits the rate. diff --git a/bench/README.md b/bench/README.md new file mode 100644 index 0000000..e1aeeec --- /dev/null +++ b/bench/README.md @@ -0,0 +1,436 @@ +# Tailwise benchmark + +Tailwise's job is to pick, from many agent runs, the ones worth a human's +review: the runs that failed. This harness measures how well it does that. +Each labelled run is scored once, the way production would score it, and +the result is compared with keeping runs at random. + +Every score goes through the real service. The harness starts +`bin/tailwise`, sends each run as OTLP spans to +`POST /v1/otlp/traces:score`, and reads back the decision, every answer and +the input tokens Jev billed. + +| Dataset | Runs | A run failed when | +| --- | --- | --- | +| [τ-bench](https://github.com/sierra-research/tau-bench) (MIT) | 1,980 customer-service runs (retail and airline; gpt-4o and claude-3.5-sonnet), about 40% failed | Its reward is below 1: the database did not end in the right state, or the agent did not tell the user what it had to. | +| [SWE-agent trajectories](https://huggingface.co/datasets/nebius/SWE-agent-trajectories) (CC-BY-4.0) | A seeded sample of 1,000 coding runs on real GitHub issues, about 83% failed | Its patch did not make the issue's tests pass. | +| [TRAIL](https://huggingface.co/datasets/PatronusAI/TRAIL) (gated, evaluation only), optional | 148 GAIA and SWE-bench traces | It has a HIGH-impact error. TRAIL labels errors, not outcomes, and nearly every trace has some error, so "any error" would leave almost nothing to tell apart. | + +## Setup + +```sh +make build # in the repository root: bin/tailwise +cd bench +uv sync # Python 3.11+, the dependencies, the bench command +``` + +| Variable | Needed for | +| --- | --- | +| `JEV_API_KEY` | Scoring. By default the harness runs Tailwise under `envchain observer-ai`, so the key never appears in the environment or a file. Set `tailwise.envchain: ""` to use `JEV_API_KEY` from the environment instead. | +| `HF_TOKEN` | Only for TRAIL, which is gated. Accept its terms on Hugging Face, then set `HF_TOKEN` or run `hf auth login`. τ-bench and SWE-agent need no token. | + +## Running + +```sh +uv run bench run --dry-run # projected tokens and cost, no API calls +uv run bench run --limit 5 # a smoke test: 5 runs per dataset +uv run bench run # τ-bench and SWE-agent, in full +uv run bench run --datasets tau,swe,trail # with TRAIL, once you have access +uv run bench run --config path/to/config.yaml # another config +``` + +Per-run overrides: `--manifest manifests/answer-wrong-only.json` (other +questions; `manifests/tau-actions.json` adds τ-bench's action tools, for the +action checks), `--evidence first` (Tailwise's older evidence strategy; `grounded` +is the default), +`--budget 1.00` (a hard cap for this run), and `--baseline ` with +`--notes ` to add a before/after comparison to the report. + +`uv run bench investigate` runs the offline τ-bench diagnosis. It never calls +Jev. It captures the exact requests Tailwise would send under several +settings, checks every write call's arguments against the user's messages, +the policy and earlier tool results, and compares writes with the task's gold +actions. The output lands in `results/_investigation/`. + +`heldout.yaml` samples SWE-agent runs from a parquet file no other run has +read. Use it to confirm a change designed on the main datasets: +`uv run bench run --config heldout.yaml --datasets swe`. + +The exit status is 0 on success, 3 when the run stopped at the budget cap, +and 2 for bad arguments. + +`uv run bench publish ` writes `published/benchmark.json`, +the figures the website shows: one row per dataset, slice (all runs or the +trace-detectable ones) and setup (built-in questions or a manifest, with or +without rules), at the cost section's deployment of 10,000 traces, 10% +failing and 500 reviews. It reads only `summary.json`, so it is free. The +website reads the file at the latest release's tag, so publish it from the +runs a release's figures come from before tagging: + +```sh +uv run bench publish results/ results/ \ + results/ results/ +``` + +`uv run pytest` runs the unit tests, and `uv run mypy src` type-checks the +code. Neither needs the network. + +## Results + +The headline is the share of reviewed runs that failed: a team reviews the +5% of runs Tailwise ranks highest, and asks how many of them are real +failures. That is what a review queue gets; ROC-AUC and the threshold +metrics below are diagnostics. The datasets fail far more often than +production agents, so the columns after the first drop failed runs at random +until 20%, 10% or 5% of runs fail (100 draws, mean). Each draw also +resamples the tasks, since the datasets run each task several times, so the +intervals cover which tasks a dataset happened to contain. Random review +finds failures at the failure rate; a perfect ranker finds 100% in each +column. + +Jev `jev-1.14.0`. Runs are ranked by `severity`, the deciding questions' +average answer. The bold rows and the other `grounded` rows were scored +2026-11-05 on the current code (`results/20261105T121615Z_…` and +`20261105T121625Z_…`); the `first` and held-out rows on 2026-11-04. The +folders under `results/` hold every table. + +| Dataset | Runs (failed) | Questions, evidence | Own rate | 20% failing | 10% failing | 5% failing | ROC-AUC | Cost per 1,000 runs | +| --- | --- | --- | --- | --- | --- | --- | --- | --- | +| τ-bench | 1,980 (40%) | built-in, `first` | 51% | 27% | 13% | 7% | 0.586 | $0.08 | +| | | built-in, `grounded` | 66% | 40% | 22% | 12% | 0.645 | $0.14 | +| | | + action checks³ | **67%** | **42%** | **23%** | **12%** | **0.668** | $0.36 | +| τ-bench, trace-detectable runs | 1,304 (42%) | built-in, `first` | 61% | 33% | 19% | 10% | 0.629 | $0.08 | +| | | built-in, `grounded` | 74% | 49% | 30% | 18% | 0.708 | $0.13 | +| | | + action checks³ | **92%** | **74%** | **47%** | **27%** | **0.732** | $0.33 | +| | | + action checks, ranked by `p_keep`⁴ | 71% | 41% | 23% | 14% | 0.706 | $0.33 | +| SWE-agent | 1,000 (77%) | built-in, `first`¹ | 100% | 59%† | 36%† | 22%† | 0.832 | $0.12 | +| | | built-in, `grounded` | 100% | 78%† | 52%† | 32%† | 0.856 | $0.25 | +| SWE-agent, held out | 1,000 (84%) | built-in, `first` | 100% | 60%† | 36%† | 18%† | 0.786 | $0.14 | +| | | built-in, `grounded` | 98% | 67%† | 41%† | 21%† | 0.801 | $0.27 | +| TRAIL (GAIA and SWE-bench) | 148 (74%) | built-in, `first` | 100% | 55%† | 28%† | 12%† | 0.631 | $0.12 | +| | | built-in, `grounded` | 100% | 53%† | 31%† | 17%† | 0.673 | $0.15 | + +† Rough: these datasets have few successful runs (227, 160 and 38), so at +these rates a 5% review is 3 to 15 runs per draw. On τ-bench it is 33 to 141. + +The intervals are wide everywhere. At 10% failing (95% intervals): +τ-bench trace-detectable with action checks 47% (26–66%), the same runs with +`answer_wrong` alone 30% (17–49%), all τ-bench runs with action checks 23% +(14–34%), SWE-agent 52% (24–83%), and TRAIL 31% (0–100%). + +³ With `manifests/tau-actions.json`: the built-in questions plus τ-bench's +13 action tools, which turn on the [action checks](../README.md#checking-an-agents-actions). +Of failed runs on gold-consistent tasks, they keep wrong-variant exchanges +the user had described 90% of the time (33% without), other wrong arguments +73% (58%), extra actions 68% (57%), and runs that left out an action 68% +(50%). + +⁴ The same answers, ranked by the highest deciding answer, as Tailwise did +before `severity` became the average. Ranked that way, the action checks +made the review queue *worse* than `answer_wrong` alone (23% against 30% at +10% failing): one confident check, usually `action_wrong`, put a third +successful runs at the top. Averaging puts first the runs several checks +doubt. + +**Against `answer_wrong` alone**, on the same resampled runs (points of +reviewed runs that failed, 95% interval): + +| τ-bench, with action checks | 20% failing | 10% failing | 5% failing | +| --- | --- | --- | --- | +| Trace-detectable runs | +24 (+5 to +40) | +17 (−1 to +35) | +9 (−2 to +23) | +| All runs | 0 (−15 to +13) | 0 (−11 to +9) | +1 (−9 to +10) | + +The gain is shown at 20% failing on trace-detectable runs and likely at +10%; at lower rates and on all runs it is not shown. The action checks and +the averaging were designed on τ-bench, so these are in-sample until +confirmed on data they were not built on. + +With the built-in questions, `answer_wrong` is the only deciding question, +so `severity` is its answer. The `first` and held-out rows predate +`severity` and were rebuilt from each run's `runs.csv` (folders named +`rebuilt-from-…`), not re-scored. + +### Cost: random review against Tailwise + +Per 10,000 traces with 10% failing, a team reviewing 500 of them (5%). +Random review costs nothing to choose the traces and finds 50 failures; +Tailwise pays Jev to score every trace ($0.042 per million input tokens, +output free). Built-in questions with `grounded` evidence, and the action +checks for τ-bench: + +| Dataset | Jev cost | Failures found: random → Tailwise | Reviews per failure found | Random reviews for the same failures | Jev cost per extra failure found | +| --- | --- | --- | --- | --- | --- | +| τ-bench, trace-detectable, action checks | $3.32 | 50 → 237 | 10 → 2.1 | 2,366 | $0.018 | +| τ-bench, all runs, action checks | $3.58 | 50 → 115 | 10 → 4.3 | 1,150 | $0.055 | +| τ-bench, trace-detectable, built-in | $1.31 | 50 → 152 | 10 → 3.3 | 1,522 | $0.013 | +| SWE-agent† | $2.50 | 50 → 258 | 10 → 1.9 | 2,579 | $0.012 | +| SWE-agent† with the `no_submit` keep rule | $1.43 | 50 → 280 | 10 → 1.8 | 2,796 | $0.006 | +| TRAIL† | $1.54 | 50 → 156 | 10 → 3.2 | 1,558 | $0.015 | + +Scoring costs cents per 10,000 traces; reviewer time is the real cost, and +Tailwise cuts the reviews needed per failure found by 2.3× to 5×. Every +report has this table (`## Cost`), from that run's own token counts. These +traces are 1,300 to 5,300 input tokens each; longer traces cost +proportionally more. + +### Heuristic rules + +`bench run --rules rules` adds the [heuristic rules](../README.md#heuristic-rules) +in `rules/.yaml`, written from what each agent does before looking +at any result, and never tuned on the labels. Rules do not change what Jev is +asked, so `rulecheck` (Tailwise's own rule and span code) checks each run's +request and the harness combines the verdicts with the cached answers as the +engine does: a rule set costs nothing to measure, and is compared with Jev +alone on the very same answers. Scored 2026-11-05 +(`results/20261105T121633Z_…`, `20261105T121649Z_…`), 10% failing: + +| Dataset | Jev alone | Jev + rules | Gain (95% interval) | Rules alone | +| --- | --- | --- | --- | --- | +| τ-bench, with action checks | 23% | 19% | −4 (−11 to +4) | 12% | +| τ-bench trace-detectable, with action checks | 47% | 42% | −5 (−21 to +15) | 14% | +| SWE-agent | 52%† | 56%† | +4 (−35 to +43) | 55%† | +| TRAIL | 31%† | 24%† | −7 (−67 to +50) | 18%† | + +| Dataset | Rule | Fired on failed | Fired on successful | Fired runs that failed | +| --- | --- | --- | --- | --- | +| SWE-agent | `no_submit` (keep) | 40% | 4% | 97% | +| τ-bench | `tool_failed` (signal) | 34% | 30% | 43% | +| τ-bench | `looping` (signal) | 2% | 0% | 75% | +| TRAIL | `code_failed` (signal) | 18% | 32% | 62% | +| TRAIL | `looping` (signal) | 11% | 5% | 86% | +| TRAIL | `no_answer` (keep) | 5% | 0% | 100% | + +- **A tool error is not a failure signal.** Agents recover from them: + `tool_failed` fires about as often on successful τ-bench runs as on failed + ones, and TRAIL's `code_failed` more often on successful ones. As signal + rules weighing as much as a check, they pulled good runs up the queue. +- **"Certain" needs checking.** SWE-agent's `no_submit` is right 97% of the + time, not always: 4% of successful runs never call `submit` (the harness + can submit for the agent when it runs out of budget). It decides 32% of + SWE-agent runs without Jev, saving a fifth of the three-dataset run's tokens. +- None of the gains is shown. Rules earn their place by saving cost on + certain failures, or as signals only once measured on the agent's own + labelled runs. + +### At the threshold + +The keep decision itself, at 0.5: the share of successful runs kept, and of +failures caught. A team that reviews a fixed share by `severity` does not +depend on these. + +| Dataset | Questions, evidence | Successes kept | Failures caught | +| --- | --- | --- | --- | +| τ-bench | built-in, `first` | 57% | 69% | +| | built-in, `grounded` | 33% | 52% | +| | + action checks³ | 50% | 72% | +| τ-bench, trace-detectable runs | built-in, `first` | 50% | 68% | +| | built-in, `grounded` | 25% | 54% | +| | + action checks³ | 40% | 70% | +| SWE-agent | built-in, `first`¹ | 30% | 82% | +| | built-in, `grounded` | 33% | 90% | +| SWE-agent, held out | built-in, `first` | 42% | 82% | +| | built-in, `grounded` | 46% | 86% | +| TRAIL | built-in, `first` | 68% | 86% | +| | built-in, `grounded` | 68% | 84% | + +¹ Scored before the trimming and tool-status fixes, with +`manifests/answer-wrong-only.json` (the same deciding question): 3 runs were +unscorable. + +TRAIL's runs are smolagents agents. Before Tailwise read their code steps +(each "Step" span whose model wrote code, as an `execute_code` call) and +scored runs that ended without an answer, TRAIL's SWE-bench runs ranked +below random (ROC-AUC 0.33): Tailwise saw only the task and the final +answer, and dropped the 5 runs that crashed, all of them failures. With +both, it is 0.67 (95% interval for the gain: +0.05 to +0.62), and no run is +unscorable. + +**Reading them:** + +- **Tailwise beats random review clearly.** With the action checks, a team + reviewing 5% of τ-bench runs finds a real failure in about half its reviews + when 10% of runs fail (47%, 95% interval 26–66%, against 10% at random), + and in a quarter when 5% fail. +- **How checks are combined matters as much as which checks run.** Ranked by + the highest answer, the action checks lowered the review queue's yield + (30% to 23%); averaged, they raise it to 47%. That gain over `answer_wrong` + alone is likely but not yet shown at 10% failing (interval −1 to +35 + points), and it is in-sample. +- **On all τ-bench runs there is no gain** (23% either way): a third of its + labels rest on things no trace shows, which caps any trace-based ranking. +- `grounded` evidence improves the queue on every dataset. It costs about + twice as much. +- TRAIL's label is "has a HIGH-impact error", not "failed the task", so it + is noisier than the other two. Its figures are a final check, never a + target: its terms forbid tuning on it. + +## What Tailwise sees + +Each run becomes the spans an OpenInference-instrumented agent emits for the +parts Tailwise reads: + +- an AGENT root span whose input is the user's request and whose output is + the agent's final answer; +- an LLM span whose system message (`llm.input_messages.0`) is the agent's + system prompt, where instrumentors record it. Tailwise shows it to the + model only with `tailwise.system_prompt: true` in the bench config, its + `scorer.system_prompt`; it is off by default, as in Tailwise; +- one TOOL span per tool call, with its arguments, its output, and an error + status when the call failed. + +| Dataset | System prompt | Question | Final answer | Tool calls | +| --- | --- | --- | --- | --- | +| τ-bench | The domain's policy | The user's messages, in order | The agent's last message | Each function call with its result; a result starting with "Error" is a failed call | +| SWE-agent | SWE-agent's system message | The issue text | The submitted patch | Each shell command with its output | +| TRAIL | Its own spans, unchanged except that the outermost AGENT span becomes a root, because TRAIL's top span has no input or output for Tailwise to read | | | | + +## Metrics + +Each dataset and each subset (domain and model for τ-bench, model for +SWE-agent) gets the metrics below. Two decision rules come from the same +call: + +- **Tailwise (default):** the built-in questions and decision rule as + shipped. A run is kept when any check reaches the threshold (0.5), and it + is ranked by `severity`, the deciding questions' average answer. +- **answer_wrong only:** the decision question alone, read from the recorded + answers. + +A third row ranks Tailwise's answers by `p_keep`, the highest deciding +answer, to show what averaging adds; it is left out where it ranks the same. + +**The headline: reviewed runs that failed.** A team reviews the +`review_budget` share of runs (5%) that Tailwise ranks highest. The report +gives the share of those that failed at the dataset's own failure rate, and +at each production rate in `reference_failure_rates` (20, 10, 5 and 2%). For +a production rate, failed runs (or, above the dataset's rate, successful +ones) are dropped at random until that share fails, 100 times; the report +gives the mean, its 95% interval in `summary.json`, and marks a cell † when +fewer than 20 runs are reviewed per draw. Ties in score are broken at random. +Random review finds failures at the failure rate; a perfect ranker finds +them in every reviewed run, or failure rate ÷ budget when failures are fewer +than the budget. + +`severity` is computed from the recorded answers, as Tailwise computes it, +so changing how answers combine needs no re-scoring. For each run scored +fresh, the harness checks it against the `severity` Tailwise returned. + +**At the threshold** (no random floor): + +| Metric | What it says | +| --- | --- | +| Kept | Share of runs kept | +| Kept that failed (precision) | Of the runs kept, the share that failed. Random keeping gets the failure rate. | +| Lift | Precision ÷ failure rate | +| Failures caught (recall) | Of the runs that failed, the share kept | +| Successes kept (false positive rate) | Of the runs that succeeded, the share kept | +| At production failure rates | The datasets fail far more often than most production agents. The catch rate and false positive rate do not depend on the failure rate, so the harness also reports the precision they would give if 2, 5, 10 or 20% of runs failed. | + +**Ranking** (for review queues sorted by score): + +- **Budget curve.** At budgets of 1, 5, 10, 20, 30 and 50% of runs, it gives + the share of failures found, and the share of reviewed runs that failed, + when reviewing the highest-scored runs first. Each is compared with the + same number of random runs, averaged over 100 seeded runs and shown with + its 95% interval. The curve also shows the best a perfect ranker could do. +- **ROC-AUC and PR-AUC.** +- **Calibration:** whether `p_keep` (not `severity`, an average) behaves + like a probability of failure, + shown as a reliability diagram with its ECE. + +**Trace-detectable runs (τ-bench).** Some τ-bench labels can't be judged +from the trace, whatever reads it. The report adds a "trace-detectable runs" +row that leaves those runs out: + +- runs on tasks whose logged gold actions disagree with at least one + successful run: there, the gold says little about how runs were scored; +- failed runs whose only mistake is choosing the wrong variant of an item + when the user never stated the option that tells the right one apart. The + right item came from the simulated user's hidden instructions. + +That leaves 1,304 of the 1,980 runs, 554 of them failed. Judge changes to +Tailwise on this row: label noise in the full dataset caps what any +trace-based detector can reach. Each run's labels are in `runs.csv` +(`meta_trace_detectable`, `meta_gold_consistent_task`, +`meta_mixup_not_in_trace`). + +**Diagnostics:** + +- for each check: how often it fires on failed runs and on successful runs; +- the reasons Jev gave, for failed and for successful runs; +- cost per run, and per failure caught. + +**Runs Tailwise cannot score:** Jev can reject a trace. Tailwise cuts one +that is too long (a giant SWE-agent patch, say) to fit before sending it, so +`max_tokens_exceeded` should no longer happen, but on any rejection it falls +back to a random decision. These runs count the way production treats them +here: dropped, and last in any review order. The tables report how many +there were. + +All randomness comes from `seed`: the SWE-agent sample, baselines, tie +breaks and retry jitter. + +## Cost + +- **Pricing.** Jev charges $0.042 per million input tokens; output is free. +- **Real runs.** Cost is billed from the `input_tokens` Jev reports for each + call (`tokens_source: api`). +- **Dry runs.** `--dry-run` estimates with tiktoken `cl100k_base`, because + TypeSafe does not publish Jev's tokenizer. Jev has billed about 25% more + than these estimates. +- **Budget cap.** `cost.budget_usd` (default $3) is a hard cap. + - Before each call, its estimated cost is reserved, multiplied by the + worst estimate-to-billed ratio seen so far. + - A call that could go past the cap is never sent. The run stops cleanly + and exits with status 3. + - Calls that time out inside Tailwise may still be billed without being + recorded, so leave some headroom. +- **Cache.** Every score is cached in `.cache/scores.sqlite`, keyed by + dataset, trace, model, a hash of the settings that change a score (model, + threshold, questions, evidence, and the Tailwise sources that build Jev's + state: `internal/trace/*.go` and `internal/otlp/convert.go`) and a hash of + the exact request sent. + - Rerunning, or resuming after a stop, costs nothing for runs already + scored. + - A change to other Tailwise code does not invalidate the cache; delete + it after such a change if it affects what Jev is sent. +- **Rate limits.** + - `client.concurrency` and `client.requests_per_second` limit the load. + - A 429 or 5xx is retried with exponential backoff and jitter. + - A fallback decision is retried once. + - A 4xx is final, and is written to `errors.json`. + +## Outputs + +Each run writes `results/_/`: + +| File | Contents | +| --- | --- | +| `runs.csv` | One row per run: the label, `p_keep`, the decision, `answer_wrong` and the decision it alone would make, flags, severity, cause, every answer, tokens, cost, dataset metadata | +| `summary.json` | Every metric, per dataset and subset, plus tokens, cost, duration, Tailwise version, Jev model and the config used | +| `plots/*.png` | Budget curve, precision of kept runs, ROC, PR, reliability, checks on failed and successful runs, cost | +| `report.md` | A summary of the results, the main tables and the plots | +| `tailwise.yaml`, `tailwise.log` | The config Tailwise ran with and its log | +| `errors.json` | Runs that could not be scored, if any | + +`results/` is not committed. `published/benchmark.json` is: the figures of the +latest published runs, with the `run` folder each row came from and the +`source` commit. + +Every results folder also holds `source.patch`, the uncommitted changes the +run used. `summary.json` records the base commit and the patch's SHA-256 +under `meta.source`; `git checkout && git apply source.patch` rebuilds +the tree. To reproduce a run, use the commit in the folder's name, the `config` block +in `summary.json` (it pins τ-bench's commit and the SWE-agent sample), and +the same `seed`. Delete `.cache/scores.sqlite` to score from scratch. + +## Data handling + +- **Where the data lives.** Datasets download into `.cache/`, which is + git-ignored along with `results/`. Nothing from them is committed. +- **TRAIL's terms.** + - Use it for evaluation only. + - Never use it to train or tune anything, including questions or + thresholds. + - Never share it outside a gated repository. +- **Test fixtures.** The fixtures in `tests/` are invented, not dataset rows. diff --git a/bench/config.yaml b/bench/config.yaml new file mode 100644 index 0000000..65d996d --- /dev/null +++ b/bench/config.yaml @@ -0,0 +1,55 @@ +# Benchmark settings. Everything that changes a score (model, threshold, +# questions, evidence, and Tailwise's state-building code) is part of the +# config hash, so changing it re-scores instead of reusing the cache. + +seed: 0 # every random choice: samples, baselines, tie breaks, jitter +random_seeds: 100 # random-sampling baseline runs, for the mean and 95% interval +budgets: [0.01, 0.05, 0.10, 0.20, 0.30, 0.50] # review budgets: share of runs a team reviews +review_budget: 0.05 # the headline: share of reviewed runs that failed when a team reviews this share +# Production failure rates to translate precision to: the datasets fail far +# more often (τ-bench ~40%, SWE-agent ~83%) than most production agents. +reference_failure_rates: [0.02, 0.05, 0.10, 0.20] + +tailwise: + # start: the harness runs `binary` with a generated config; false uses a + # Tailwise already listening at `url`. + start: true + binary: ../bin/tailwise # relative to this file + url: http://127.0.0.1:18080 + envchain: observer-ai # namespace holding JEV_API_KEY; "" to use the environment + model: jev-latest + keep_threshold: 0.5 # Tailwise's default + manifest: "" # "" is the built-in questions and decision rule, as shipped + system_prompt: false # Tailwise's scorer.system_prompt; the runs always carry their prompt + evidence: grounded # Tailwise's scorer.evidence: grounded (default) or first + evidence_budget: 12000 # Tailwise's scorer.evidence_budget, for grounded + rules: "" # a folder of .yaml heuristic rules (e.g. rules); checked on the cached answers + +cost: + usd_per_million_input_tokens: 0.042 # Jev; output tokens are free + budget_usd: 3.00 # hard cap: the run stops before going over it + tokenizer: cl100k_base # for --dry-run and pre-call estimates only + +client: + concurrency: 8 + requests_per_second: 8 + max_attempts: 6 + backoff_seconds: 1.0 # doubles each attempt, with jitter + timeout_seconds: 120 + +datasets: + tau: + commit: 59a200c6d575d595120f1cb70fea53cef0632f6b # sierra-research/tau-bench, pinned + files: [gpt-4o-retail, gpt-4o-airline, sonnet-35-new-retail, sonnet-35-new-airline] + swe: + repo: nebius/SWE-agent-trajectories + first_shard: 0 # the first parquet file to sample from + shards: 1 # parquet files to sample from (about 6,700 runs each) + shards_total: 12 + sample: 1000 # runs, drawn with `seed` at the natural failure rate + trail: + repo: PatronusAI/TRAIL # gated: accept its terms on Hugging Face first + splits: [gaia, swe_bench] + +cache_dir: .cache # relative to this file; git-ignored +results_dir: results # relative to this file; git-ignored diff --git a/bench/heldout.yaml b/bench/heldout.yaml new file mode 100644 index 0000000..7eaa66f --- /dev/null +++ b/bench/heldout.yaml @@ -0,0 +1,16 @@ +# A held-out check: SWE-agent runs from a parquet file no benchmark run or +# investigation has read (config.yaml samples file 0). Use it to confirm a +# change designed on the main datasets before treating its numbers as final: +# +# uv run bench run --config heldout.yaml --datasets swe --evidence first +# uv run bench run --config heldout.yaml --datasets swe --evidence grounded +# +# Everything not set here takes the defaults config.yaml also uses. + +seed: 0 + +datasets: + swe: + first_shard: 1 + shards: 1 + sample: 1000 diff --git a/bench/manifests/answer-wrong-only.json b/bench/manifests/answer-wrong-only.json new file mode 100644 index 0000000..a8cb84c --- /dev/null +++ b/bench/manifests/answer-wrong-only.json @@ -0,0 +1,23 @@ +{ + "model": "jev-latest", + "note": "Tailwise 0.1.0 before the five checks (internal/config/builtin_manifest.json at commit f573a15): answer_wrong alone decides keep or drop; why_wrong only gives the plain-English reason.", + "questions": { + "answer_wrong": { + "type": "noul", + "instructions": "Is the final answer wrong or only partially correct for the user's question?" + }, + "why_wrong": { + "type": "choice", + "instructions": "Which best explains what is wrong with the final answer? Choose \"correct\" if nothing is wrong.", + "criteria": { + "correct": "The final answer is correct, complete and supported by the evidence.", + "contradicts_evidence": "The final answer contradicts the evidence the agent gathered.", + "unsupported": "The final answer makes claims the evidence does not support.", + "incomplete": "The final answer leaves out part of what the question asked.", + "wrong_lookup": "The agent looked up the wrong thing, so its evidence does not answer the question.", + "ignored_tool_failure": "A tool call failed and the agent answered anyway without the information it needed.", + "no_answer": "The agent did not give an answer." + } + } + } +} diff --git a/bench/manifests/tau-actions.json b/bench/manifests/tau-actions.json new file mode 100644 index 0000000..658588a --- /dev/null +++ b/bench/manifests/tau-actions.json @@ -0,0 +1,58 @@ +{ + "model": "tau-bench", + "note": "The built-in questions plus τ-bench's actions (retail and airline), for the action checks: each call to one of these tools is checked against the user's request, and each run is checked for requests of these kinds left unhandled. Descriptions are worded as the user would ask.", + "questions": { + "answer_wrong": { + "type": "noul", + "instructions": "Is the final answer wrong or only partially correct for the user's question?" + }, + "unsupported_answer": { + "type": "noul", + "instructions": "Does the final answer make claims that the evidence the agent gathered does not support?" + }, + "missing_tool_calls": { + "type": "noul", + "instructions": "Did the agent leave out tool calls the task needed, so it answered without information it should have looked up?" + }, + "answered_on_failure": { + "type": "noul", + "instructions": "Did the agent answer even though a tool call failed or returned only partial results?" + }, + "ignored_instructions": { + "type": "noul", + "instructions": "Does the final answer ignore or go against instructions given in the user's question?" + }, + "contradicts_knowledge": { + "type": "noul", + "instructions": "Does the final answer contradict well-established general knowledge?" + }, + "why_wrong": { + "type": "choice", + "instructions": "Which best explains what is wrong with the final answer? Choose \"correct\" if nothing is wrong.", + "criteria": { + "correct": "The final answer is correct, complete and supported by the evidence.", + "contradicts_evidence": "The final answer contradicts the evidence the agent gathered.", + "unsupported": "The final answer makes claims the evidence does not support.", + "incomplete": "The final answer leaves out part of what the question asked.", + "wrong_lookup": "The agent looked up the wrong thing, so its evidence does not answer the question.", + "ignored_tool_failure": "A tool call failed and the agent answered anyway without the information it needed.", + "no_answer": "The agent did not give an answer." + } + } + }, + "actions": { + "cancel_pending_order": "cancel a pending order", + "modify_pending_order_items": "change items in a pending order (for example another size or color)", + "modify_pending_order_address": "change the shipping address of a pending order", + "modify_pending_order_payment": "change the payment method of a pending order", + "return_delivered_order_items": "return delivered items for a refund", + "exchange_delivered_order_items": "exchange delivered items for other items", + "modify_user_address": "change their default address on their account", + "book_reservation": "book a new flight reservation", + "update_reservation_flights": "change the flights, dates or cabin of a reservation", + "update_reservation_baggages": "add or change checked bags on a reservation", + "update_reservation_passengers": "change passenger details on a reservation", + "cancel_reservation": "cancel a reservation", + "send_certificate": "get compensation, a certificate or a refund for a problem" + } +} diff --git a/bench/published/benchmark.json b/bench/published/benchmark.json new file mode 100644 index 0000000..ed460bc --- /dev/null +++ b/bench/published/benchmark.json @@ -0,0 +1,423 @@ +{ + "schema": 1, + "date": "2026-11-05", + "jev_model": "jev-1.14.0", + "source": "cb86569a320ee940ece6fd0341cff9202cf3f2c8", + "usd_per_million_input_tokens": 0.042, + "scenario": { + "traces": 10000, + "failure_rate": 0.1, + "review_budget": 0.05, + "reviews": 500, + "random_found": 50.0, + "random_reviews_per_failure": 10.0 + }, + "datasets": { + "tau": { + "name": "τ-bench", + "agent": "Customer-service agents (GPT-4o, Claude 3.5 Sonnet) in airline and retail simulations, acting through tools for a simulated user" + }, + "swe": { + "name": "SWE-agent", + "agent": "SWE-agent on Llama models (8B, 70B, 405B) fixing GitHub issues; a run fails when its patch does not pass the tests" + }, + "trail": { + "name": "TRAIL", + "agent": "smolagents runs on GAIA and SWE-bench, with failures annotated by hand" + } + }, + "results": [ + { + "dataset": "swe", + "slice": "all", + "setup": "builtin", + "rules": false, + "runs": 1000, + "failed": 773, + "rough": true, + "precision": 0.5158885975135975, + "precision_ci95": [ + 0.23916666666666667, + 0.8328671328671327 + ], + "found": 257.94429875679873, + "found_ci95": [ + 119.58333333333333, + 416.43356643356634 + ], + "multiple": 5.158885975135974, + "reviews_per_failure": 1.9384029901409918, + "random_reviews_for_same": 2579.442987567987, + "jev_usd": 2.4975804, + "jev_usd_per_extra_failure": 0.012010814506249317, + "input_tokens_per_run": 5946.62, + "roc_auc": 0.8556570601409917, + "vs_baseline": null, + "run": "20261105T121615Z_cb86569-dirty" + }, + { + "dataset": "tau", + "slice": "all", + "setup": "builtin", + "rules": false, + "runs": 1980, + "failed": 797, + "rough": false, + "precision": 0.21974975969102511, + "precision_ci95": [ + 0.12301136363636364, + 0.31550868486352346 + ], + "found": 109.87487984551255, + "found_ci95": [ + 61.50568181818182, + 157.75434243176173 + ], + "multiple": 2.1974975969102513, + "reviews_per_failure": 4.550630687405668, + "random_reviews_for_same": 1098.7487984551256, + "jev_usd": 1.3813413939393941, + "jev_usd_per_extra_failure": 0.023070466237318416, + "input_tokens_per_run": 3288.9080808080807, + "roc_auc": 0.6452626130746004, + "vs_baseline": null, + "run": "20261105T121615Z_cb86569-dirty" + }, + { + "dataset": "tau", + "slice": "detectable", + "setup": "builtin", + "rules": false, + "runs": 1304, + "failed": 554, + "rough": false, + "precision": 0.3044181139060137, + "precision_ci95": [ + 0.162460716530484, + 0.48247377622377613 + ], + "found": 152.20905695300684, + "found_ci95": [ + 81.230358265242, + 241.23688811188808 + ], + "multiple": 3.0441811390601368, + "reviews_per_failure": 3.284955639363632, + "random_reviews_for_same": 1522.0905695300683, + "jev_usd": 1.312554754601227, + "jev_usd_per_extra_failure": 0.01284186346817295, + "input_tokens_per_run": 3125.1303680981596, + "roc_auc": 0.708395908543923, + "vs_baseline": null, + "run": "20261105T121615Z_cb86569-dirty" + }, + { + "dataset": "trail", + "slice": "all", + "setup": "builtin", + "rules": false, + "runs": 148, + "failed": 110, + "rough": true, + "precision": 0.3116666666666667, + "precision_ci95": [ + 0.0, + 1.0 + ], + "found": 155.83333333333334, + "found_ci95": [ + 0.0, + 500.0 + ], + "multiple": 3.1166666666666667, + "reviews_per_failure": 3.2085561497326203, + "random_reviews_for_same": 1558.3333333333333, + "jev_usd": 1.5403613513513514, + "jev_usd_per_extra_failure": 0.01455459544583954, + "input_tokens_per_run": 3667.527027027027, + "roc_auc": 0.6728468899521531, + "vs_baseline": null, + "run": "20261105T121615Z_cb86569-dirty" + }, + { + "dataset": "tau", + "slice": "all", + "setup": "tau-actions", + "rules": false, + "runs": 1980, + "failed": 797, + "rough": false, + "precision": 0.23004268491897123, + "precision_ci95": [ + 0.1447236473880597, + 0.3435284514925373 + ], + "found": 115.02134245948561, + "found_ci95": [ + 72.36182369402985, + 171.76422574626866 + ], + "multiple": 2.300426849189712, + "reviews_per_failure": 4.347019338399018, + "random_reviews_for_same": 1150.213424594856, + "jev_usd": 3.57612203030303, + "jev_usd_per_extra_failure": 0.0549992032620872, + "input_tokens_per_run": 8514.576262626262, + "roc_auc": 0.6684020062554952, + "vs_baseline": { + "against": "answer_wrong_alone", + "gain": -0.004077857424389931, + "ci95": [ + -0.10691724941724942, + 0.0917560634328358 + ] + }, + "run": "20261105T121625Z_cb86569-dirty" + }, + { + "dataset": "tau", + "slice": "detectable", + "setup": "tau-actions", + "rules": false, + "runs": 1304, + "failed": 554, + "rough": false, + "precision": 0.47313600914341186, + "precision_ci95": [ + 0.2564102564102564, + 0.6633720930232556 + ], + "found": 236.56800457170593, + "found_ci95": [ + 128.2051282051282, + 331.6860465116278 + ], + "multiple": 4.7313600914341185, + "reviews_per_failure": 2.113557160467342, + "random_reviews_for_same": 2365.6800457170593, + "jev_usd": 3.3248749233128834, + "jev_usd_per_extra_failure": 0.017821249313062112, + "input_tokens_per_run": 7916.368865030675, + "roc_auc": 0.7320264741275572, + "vs_baseline": { + "against": "answer_wrong_alone", + "gain": 0.1687674847576362, + "ci95": [ + -0.011931818181818168, + 0.35426829268292664 + ] + }, + "run": "20261105T121625Z_cb86569-dirty" + }, + { + "dataset": "swe", + "slice": "all", + "setup": "builtin", + "rules": true, + "runs": 1000, + "failed": 773, + "rough": true, + "precision": 0.5592766726737315, + "precision_ci95": [ + 0.28568181818181815, + 0.8123529411764704 + ], + "found": 279.63833633686573, + "found_ci95": [ + 142.84090909090907, + 406.1764705882352 + ], + "multiple": 5.592766726737315, + "reviews_per_failure": 1.7880237972724744, + "random_reviews_for_same": 2796.3833633686572, + "jev_usd": 1.4303162999999999, + "jev_usd_per_extra_failure": 0.006228560626313767, + "input_tokens_per_run": 3405.515, + "roc_auc": 0.8656102717828018, + "vs_baseline": { + "against": "jev_alone", + "gain": 0.043388075160133974, + "ci95": [ + -0.34924242424242424, + 0.4312499999999998 + ] + }, + "run": "20261105T121633Z_cb86569-dirty" + }, + { + "dataset": "tau", + "slice": "all", + "setup": "builtin", + "rules": true, + "runs": 1980, + "failed": 797, + "rough": false, + "precision": 0.17873003969313442, + "precision_ci95": [ + 0.08134615384615385, + 0.26741258741258733 + ], + "found": 89.36501984656721, + "found_ci95": [ + 40.67307692307692, + 133.70629370629368 + ], + "multiple": 1.7873003969313441, + "reviews_per_failure": 5.595030369359971, + "random_reviews_for_same": 893.650198465672, + "jev_usd": 1.3813413939393941, + "jev_usd_per_extra_failure": 0.035090580401672344, + "input_tokens_per_run": 3288.9080808080807, + "roc_auc": 0.6059589479143577, + "vs_baseline": { + "against": "jev_alone", + "gain": -0.04101971999789069, + "ci95": [ + -0.1451550839552239, + 0.08739596469104657 + ] + }, + "run": "20261105T121633Z_cb86569-dirty" + }, + { + "dataset": "tau", + "slice": "detectable", + "setup": "builtin", + "rules": true, + "runs": 1304, + "failed": 554, + "rough": false, + "precision": 0.30193760979323914, + "precision_ci95": [ + 0.1794871794871795, + 0.4461337209302325 + ], + "found": 150.96880489661956, + "found_ci95": [ + 89.74358974358974, + 223.06686046511626 + ], + "multiple": 3.0193760979323914, + "reviews_per_failure": 3.311942492638728, + "random_reviews_for_same": 1509.6880489661955, + "jev_usd": 1.312554754601227, + "jev_usd_per_extra_failure": 0.012999606719571774, + "input_tokens_per_run": 3125.1303680981596, + "roc_auc": 0.6765691937424789, + "vs_baseline": { + "against": "jev_alone", + "gain": -0.0024805041127745385, + "ci95": [ + -0.19575845665961947, + 0.18837209302325578 + ] + }, + "run": "20261105T121633Z_cb86569-dirty" + }, + { + "dataset": "trail", + "slice": "all", + "setup": "builtin", + "rules": true, + "runs": 148, + "failed": 110, + "rough": true, + "precision": 0.24166666666666664, + "precision_ci95": [ + 0.0, + 0.8416666666666638 + ], + "found": 120.83333333333331, + "found_ci95": [ + 0.0, + 420.8333333333319 + ], + "multiple": 2.416666666666666, + "reviews_per_failure": 4.137931034482759, + "random_reviews_for_same": 1208.333333333333, + "jev_usd": 1.432690945945946, + "jev_usd_per_extra_failure": 0.02022622511923689, + "input_tokens_per_run": 3411.1689189189187, + "roc_auc": 0.5696172248803828, + "vs_baseline": { + "against": "jev_alone", + "gain": -0.07, + "ci95": [ + -0.6666666666666666, + 0.5 + ] + }, + "run": "20261105T121633Z_cb86569-dirty" + }, + { + "dataset": "tau", + "slice": "all", + "setup": "tau-actions", + "rules": true, + "runs": 1980, + "failed": 797, + "rough": false, + "precision": 0.1948571163520232, + "precision_ci95": [ + 0.11428571428571428, + 0.2791947115384615 + ], + "found": 97.4285581760116, + "found_ci95": [ + 57.14285714285714, + 139.59735576923075 + ], + "multiple": 1.948571163520232, + "reviews_per_failure": 5.13196550745126, + "random_reviews_for_same": 974.285581760116, + "jev_usd": 3.57612203030303, + "jev_usd_per_extra_failure": 0.07540018435794996, + "input_tokens_per_run": 8514.576262626262, + "roc_auc": 0.62842750339131, + "vs_baseline": { + "against": "jev_alone", + "gain": -0.035185568566948024, + "ci95": [ + -0.11195312499999999, + 0.03833333333333324 + ] + }, + "run": "20261105T121649Z_cb86569-dirty" + }, + { + "dataset": "tau", + "slice": "detectable", + "setup": "tau-actions", + "rules": true, + "runs": 1304, + "failed": 554, + "rough": false, + "precision": 0.42016135953947065, + "precision_ci95": [ + 0.27147865853658537, + 0.5857142857142856 + ], + "found": 210.08067976973533, + "found_ci95": [ + 135.7393292682927, + 292.85714285714283 + ], + "multiple": 4.201613595394707, + "reviews_per_failure": 2.380037995631196, + "random_reviews_for_same": 2100.806797697353, + "jev_usd": 3.3248749233128834, + "jev_usd_per_extra_failure": 0.020769995030602565, + "input_tokens_per_run": 7916.368865030675, + "roc_auc": 0.6973044524669073, + "vs_baseline": { + "against": "jev_alone", + "gain": -0.05297464960394119, + "ci95": [ + -0.21191860465116283, + 0.14687499999999964 + ] + }, + "run": "20261105T121649Z_cb86569-dirty" + } + ] +} diff --git a/bench/pyproject.toml b/bench/pyproject.toml new file mode 100644 index 0000000..e401abe --- /dev/null +++ b/bench/pyproject.toml @@ -0,0 +1,36 @@ +[project] +name = "tailwise-bench" +version = "0.1.0" +description = "Measures how well Tailwise finds agent errors against random sampling, on TRAIL and Who&When." +requires-python = ">=3.11" +dependencies = [ + "httpx>=0.27", + "huggingface-hub>=0.24", + "matplotlib>=3.8", + "numpy>=1.26", + "pyarrow>=15", + "pyyaml>=6", + "scikit-learn>=1.4", + "tiktoken>=0.7", +] + +[project.scripts] +bench = "tailwise_bench.cli:main" + +[dependency-groups] +dev = ["pytest>=8", "mypy>=1.10", "types-PyYAML"] + +[build-system] +requires = ["hatchling"] +build-backend = "hatchling.build" + +[tool.hatch.build.targets.wheel] +packages = ["src/tailwise_bench"] + +[tool.pytest.ini_options] +testpaths = ["tests"] + +[tool.mypy] +python_version = "3.11" +strict = true +ignore_missing_imports = true diff --git a/bench/rulecheck/main.go b/bench/rulecheck/main.go new file mode 100644 index 0000000..d087b32 --- /dev/null +++ b/bench/rulecheck/main.go @@ -0,0 +1,106 @@ +// Command rulecheck runs Tailwise's heuristic rules over the benchmark's +// requests without scoring them, so a rule set is measured on cached Jev +// answers at no cost. It reads JSON lines {"id": ..., "request": } on stdin and writes {"id": ..., +// "rules": [{"name", "keep", "fired", "reason"}]} for each, in order. +// +// go run ./bench/rulecheck -rules bench/rules/tau.yaml < requests.jsonl +// +// The trace is assembled exactly as POST /v1/otlp/traces:score does. +package main + +import ( + "bufio" + "encoding/json" + "errors" + "flag" + "fmt" + "os" + + coltracepb "go.opentelemetry.io/proto/otlp/collector/trace/v1" + "gopkg.in/yaml.v3" + + "github.com/tecnosam/tailwise/internal/otlp" + "github.com/tecnosam/tailwise/internal/rules" +) + +type input struct { + ID string `json:"id"` + Request json.RawMessage `json:"request"` +} + +type verdict struct { + Name string `json:"name"` + Keep bool `json:"keep"` + Fired bool `json:"fired"` + Reason string `json:"reason,omitempty"` +} + +func main() { + path := flag.String("rules", "", "a YAML file with a rules: list, as in the Tailwise config") + flag.Parse() + if err := run(*path); err != nil { + fmt.Fprintln(os.Stderr, "rulecheck:", err) + os.Exit(1) + } +} + +func run(path string) error { + raw, err := os.ReadFile(path) + if err != nil { + return err + } + var file struct { + Rules []rules.Config `yaml:"rules"` + } + if err := yaml.Unmarshal(raw, &file); err != nil { + return fmt.Errorf("%s: %w", path, err) + } + set, err := rules.New(file.Rules) + if err != nil { + return fmt.Errorf("%s: %w", path, err) + } + if set == nil { + return fmt.Errorf("%s: no rules", path) + } + in := bufio.NewScanner(os.Stdin) + in.Buffer(make([]byte, 0, 1<<20), 1<<30) + out := bufio.NewWriter(os.Stdout) + defer out.Flush() + enc := json.NewEncoder(out) + assembler := otlp.Assembler{} + for in.Scan() { + var line input + if err := json.Unmarshal(in.Bytes(), &line); err != nil { + return err + } + var req coltracepb.ExportTraceServiceRequest + if err := otlp.UnmarshalJSON(line.Request, &req); err != nil { + return fmt.Errorf("%s: %w", line.ID, err) + } + byTrace, _, err := otlp.Split(req.GetResourceSpans()) + if err != nil { + return fmt.Errorf("%s: %w", line.ID, err) + } + if len(byTrace) != 1 { + return fmt.Errorf("%s: %d traces in one request", line.ID, len(byTrace)) + } + var verdicts []verdict + for id, spans := range byTrace { + t, err := assembler.Assemble(id, [][]byte{spans}) + if err != nil { + return fmt.Errorf("%s: %w", line.ID, err) + } + for _, r := range set.Check(t) { + verdicts = append(verdicts, verdict{Name: r.Name, Keep: r.Keep, Fired: r.Fired, Reason: r.Reason}) + } + } + if err := enc.Encode(map[string]any{"id": line.ID, "rules": verdicts}); err != nil { + return err + } + } + if err := in.Err(); err != nil { + return errors.Join(errors.New("reading stdin"), err) + } + return nil +} diff --git a/bench/rules/swe.yaml b/bench/rules/swe.yaml new file mode 100644 index 0000000..bcd2756 --- /dev/null +++ b/bench/rules/swe.yaml @@ -0,0 +1,8 @@ +# Heuristic rules for SWE-agent, written from how the agent works, before +# looking at any result. Never tuned on the labels. +rules: + # SWE-agent hands in its patch with the submit command; a run that never + # submits produced no patch, so it has certainly not fixed the issue. + - name: no_submit + on_fail: keep + run: {requires_tools: [submit]} diff --git a/bench/rules/tau.yaml b/bench/rules/tau.yaml new file mode 100644 index 0000000..0b7cb00 --- /dev/null +++ b/bench/rules/tau.yaml @@ -0,0 +1,18 @@ +# Heuristic rules for τ-bench's customer-service agents (retail and airline), +# written from what such an agent is for, before looking at any result: a +# team running this agent could write the same on day one. Never tuned on the +# labels; change them only for a reason that holds without the results. +rules: + # A conversation that ends without a reply has certainly failed the user. + - name: no_answer + on_fail: keep + final_answer: {not_empty: true} + # A tool returned an error (an output starting "Error:"): the agent may + # recover, so it only raises the review order. + - name: tool_failed + on_fail: signal + tool_output: {ok_status: true} + # The same lookup or action three times with the same arguments: stuck. + - name: looping + on_fail: signal + run: {max_repeated_calls: 2} diff --git a/bench/rules/trail.yaml b/bench/rules/trail.yaml new file mode 100644 index 0000000..0b1c82d --- /dev/null +++ b/bench/rules/trail.yaml @@ -0,0 +1,17 @@ +# Heuristic rules for TRAIL's smolagents runs (GAIA and SWE-bench), written +# from how the agents work, before looking at any result. TRAIL is for +# evaluation only: these are never tuned on it. +rules: + # A run that ends without an answer has failed its task. + - name: no_answer + on_fail: keep + final_answer: {not_empty: true} + # A code step raised an error: the agent may recover, so it only raises + # the review order. + - name: code_failed + on_fail: signal + tool_output: {tool: execute_code, ok_status: true} + # The same call three times with the same arguments: stuck. + - name: looping + on_fail: signal + run: {max_repeated_calls: 2} diff --git a/bench/src/tailwise_bench/__init__.py b/bench/src/tailwise_bench/__init__.py new file mode 100644 index 0000000..677175c --- /dev/null +++ b/bench/src/tailwise_bench/__init__.py @@ -0,0 +1 @@ +"""Benchmark harness for Tailwise.""" diff --git a/bench/src/tailwise_bench/analyze.py b/bench/src/tailwise_bench/analyze.py new file mode 100644 index 0000000..9e62fd7 --- /dev/null +++ b/bench/src/tailwise_bench/analyze.py @@ -0,0 +1,202 @@ +"""Scores and labels joined into one row per run, and the summary metrics. + +Two decision rules are compared, both from the same call: + +- default: Tailwise as configured (built-in questions: keep when any check + reaches the threshold); runs are ranked for review by severity, the + deciding questions' average answer. +- answer_wrong: the decision question alone, read from the recorded answers. + +Severity is computed here from the recorded answers, as Tailwise computes it, +so a change to how Tailwise combines answers does not need the runs scored +again; for runs scored in this run (not from the cache), it is checked against +the severity Tailwise returned. + +A run Tailwise could not score (Jev rejected it, e.g. as too long) counts as +production treats it: dropped (there is no random floor here) and last in +any review order. +""" + +from __future__ import annotations + +from collections import Counter +from collections.abc import Sequence +from typing import Any + +from . import metrics +from .config import Config +from .types import Run + +DECISION_QUESTION = "answer_wrong" + + +def severity(answers: dict[str, float | None], deciding: list[str], signals: Sequence[float] = ()) -> float: + """Tailwise's severity: the average answer of the deciding questions that + answered, a failed check left out, with each signal rule's 1 or 0 + (internal/pipeline/policy.go).""" + got = [*signals, *(a for a in (answers.get(q) for q in deciding) if a is not None)] + return sum(got) / len(got) if got else 0.0 + + +def apply_rules(row: dict[str, Any], verdicts: list[dict[str, Any]], answers: dict[str, float | None], + deciding: list[str]) -> None: + """The row as Tailwise with these rule verdicts would decide it + (internal/pipeline/engine.go): a keep rule keeps the trace at severity 1 + without asking Jev, so its tokens are not spent; signal rules join the + severity average. The answers are kept, for the comparison rows.""" + keep = any(v["fired"] and v["keep"] for v in verdicts) + signals = [1.0 if v["fired"] else 0.0 for v in verdicts if not v["keep"]] + row["rules_fired"] = "; ".join(v["name"] for v in verdicts if v["fired"]) + row["rule_keep"] = int(keep) + row["severity_rules"] = 1.0 if keep else (sum(signals) / len(signals) if signals else 0.0) + if keep: + row["severity"], row["kept"], row["tokens"], row["cost"] = 1.0, 1, 0, 0.0 + elif row["severity"] is not None: + row["severity"] = severity(answers, deciding, signals) + + +def task_group(run: Run) -> str: + """The task a run attempted: runs of one task are resampled together. + τ-bench repeats each task in several trials and SWE-agent each issue + across models; a TRAIL run is its own task.""" + m = run.meta + if "task_id" in m: + return f"{run.dataset}:{m.get('domain', '')}:{m['task_id']}" + if "instance_id" in m: + return f"{run.dataset}:{m['instance_id']}" + return f"{run.dataset}:{run.run_id}" + + +def rows(runs: list[Run], results: dict[str, dict[str, Any]], errors: dict[str, str], cfg: Config, + questions: list[str], verdicts: dict[str, list[dict[str, Any]]] | None = None) -> list[dict[str, Any]]: + """One row per run. With rule verdicts (by trace_id), severity, kept and + tokens are as Tailwise with the rules would have them; severity_jev and + kept_jev are without them.""" + deciding = cfg.deciding_questions(DECISION_QUESTION) + out = [] + for run in runs: + r = results.get(run.trace_id) + scored = r is not None and r.get("p_keep") is not None + answers = (r or {}).get("answers") or {} + aw = answers.get(DECISION_QUESTION) + tokens = int((r or {}).get("tokens", 0)) + row: dict[str, Any] = { + "dataset": run.dataset, "subset": run.subset, "run_id": run.run_id, "trace_id": run.trace_id, + "task": task_group(run), + "failed": int(run.failed), "tool_calls": len(run.tool_calls), + "p_keep": float(r["p_keep"]) if scored and r else None, + "kept": int(r["decision"] == "keep") if scored and r else None, + "answer_wrong": aw, + "kept_answer_wrong_only": int(aw >= cfg.tailwise.keep_threshold) if aw is not None else None, + "flags": "; ".join((r or {}).get("flags") or []), + "severity": severity(answers, deciding) if scored else None, + "cause": (r or {}).get("cause", ""), + "tokens": tokens, "tokens_source": (r or {}).get("tokens_source", ""), "cost": cfg.cost.usd(tokens), + "error": errors.get(run.trace_id, ""), + } + for q in questions: + row[f"answer_{q}"] = answers.get(q) + for k, v in run.meta.items(): + if isinstance(v, str | int | float): + row[f"meta_{k}"] = v + if scored and r and not r.get("cached") and r.get("severity") is not None \ + and abs(float(r["severity"]) - row["severity"]) > 1e-6: + raise ValueError(f"{run.run_id}: severity {row['severity']} differs from Tailwise's {r['severity']}: " + "analyze.severity no longer matches internal/pipeline/policy.go") + row["severity_jev"], row["kept_jev"] = row["severity"], row["kept"] + if verdicts is not None: + apply_rules(row, verdicts.get(run.trace_id, []), answers, deciding) + out.append(row) + return out + + +def _rule(rows_: list[dict[str, Any]], score: str, kept: str, probability: str, cfg: Config, + baseline: str | None = None) -> dict[str, Any]: + """score ranks runs for review; probability is the one meant as a + probability of failure, for calibration. With baseline, the headline also + compares score with that ranking on the same resampled runs.""" + y = [r["failed"] for r in rows_] + + def scores(key: str) -> list[float]: + return [r[key] if r[key] is not None else 0.0 for r in rows_] + + s = scores(score) + p = [r[probability] if r[probability] is not None else 0.0 for r in rows_] + pred = [r[kept] or 0 for r in rows_] + return { + "review": metrics.review_precision(y, s, cfg.review_budget, cfg.reference_failure_rates, + cfg.random_seeds, cfg.seed, groups=[r["task"] for r in rows_], + baseline=scores(baseline) if baseline else None), + "at_threshold": {"threshold": cfg.tailwise.keep_threshold, + **metrics.threshold_metrics(y, pred, cfg.reference_failure_rates)}, + **metrics.ranking_metrics(y, s), + "budget_curve": metrics.budget_curve(y, s, cfg.budgets, cfg.random_seeds, cfg.seed), + "calibration": metrics.calibration(y, p), + } + + +def group_summary(rows_: list[dict[str, Any]], cfg: Config) -> dict[str, Any]: + scored = [r for r in rows_ if r["p_keep"] is not None] + y = [r["failed"] for r in scored] + ruled = any("rule_keep" in r for r in rows_) + # With rules, compared with Jev alone; without, with answer_wrong alone + # when the two rank differently. + if ruled: + baseline: str | None = "severity_jev" + else: + baseline = "answer_wrong" if any(r["severity"] != r["answer_wrong"] for r in scored) else None + default = _rule(rows_, "severity", "kept", "p_keep", cfg, baseline=baseline) + tokens = sum(r["tokens"] for r in rows_) + caught = default["at_threshold"]["tp"] + return { + "runs": len(rows_), "runs_scored": len(scored), "runs_failed_to_score": len(rows_) - len(scored), + "failed": sum(r["failed"] for r in rows_), + "failure_rate": sum(r["failed"] for r in rows_) / len(rows_) if rows_ else None, + "default": default, + "answer_wrong_only": _rule(rows_, "answer_wrong", "kept_answer_wrong_only", "answer_wrong", cfg), + # The same decisions ranked by the highest deciding answer, as before + # severity was the average: what the ranking adds. + "by_p_keep": _rule(rows_, "p_keep", "kept_jev", "p_keep", cfg), + **({"jev_only": _rule(rows_, "severity_jev", "kept_jev", "p_keep", cfg), + "rules_only": _rule(rows_, "severity_rules", "rule_keep", "severity_rules", cfg), + "rules": metrics.flag_stats([r["failed"] for r in rows_], + [[f for f in r["rules_fired"].split("; ") if f] for r in rows_])} + if ruled else {}), + "flags": metrics.flag_stats(y, [[f for f in r["flags"].split("; ") if f] for r in scored]), + "causes": { + "failed_runs": dict(Counter(r["cause"] or "(none)" for r in scored if r["failed"]).most_common()), + "successful_runs": dict(Counter(r["cause"] or "(none)" for r in scored if not r["failed"]).most_common()), + }, + "tokens": tokens, "cost_usd": cfg.cost.usd(tokens), + "cost_per_run_usd": cfg.cost.usd(tokens) / len(rows_) if rows_ else None, + "cost_per_failure_caught_usd": cfg.cost.usd(tokens) / caught if caught else None, + } + + +def summary(rows_: list[dict[str, Any]], cfg: Config) -> dict[str, Any]: + """Per dataset, per dataset subset, and every run pooled. A dataset whose + runs are labelled with meta_trace_detectable (τ-bench) also gets its + trace-detectable runs alone, under "detectable": the runs whose label a + trace-only reviewer could judge.""" + out: dict[str, Any] = {"datasets": {}, "subsets": {}, "detectable": {}} + for ds in sorted({r["dataset"] for r in rows_}): + out["datasets"][ds] = group_summary([r for r in rows_ if r["dataset"] == ds], cfg) + detectable = [r for r in rows_ if r["dataset"] == ds and r.get("meta_trace_detectable") == 1] + if detectable: + out["detectable"][ds] = group_summary(detectable, cfg) + for sub in sorted({r["subset"] for r in rows_ if r["dataset"] == ds}): + out["subsets"][f"{ds}/{sub}"] = group_summary( + [r for r in rows_ if r["dataset"] == ds and r["subset"] == sub], cfg) + if len(out["datasets"]) > 1: + out["all"] = group_summary(rows_, cfg) + return out + + +def roc_pr_curves(rows_: list[dict[str, Any]]) -> dict[str, Any]: + out = {} + for ds in sorted({r["dataset"] for r in rows_}): + sc = [r for r in rows_ if r["dataset"] == ds and r["severity"] is not None] + c = metrics.curves([r["failed"] for r in sc], [r["severity"] for r in sc]) if sc else None + if c: + out[ds] = c + return out diff --git a/bench/src/tailwise_bench/budget.py b/bench/src/tailwise_bench/budget.py new file mode 100644 index 0000000..b60ee45 --- /dev/null +++ b/bench/src/tailwise_bench/budget.py @@ -0,0 +1,62 @@ +"""The hard spending cap. + +Before a call, its estimated cost is reserved; a call whose reservation would +take spent + reserved past the cap is never made. After the call, the +reservation is replaced by what Jev billed. Estimates can be low, so the +guard also keeps a safety margin of the largest estimate-to-actual ratio +seen so far. +""" + +from __future__ import annotations + +import threading +from dataclasses import dataclass + + +class BudgetExceeded(Exception): + pass + + +@dataclass +class Spend: + tokens: int = 0 + calls: int = 0 + estimated_tokens: int = 0 # of the calls billed, what was estimated + + +class BudgetGuard: + def __init__(self, cap_usd: float, usd_per_token: float) -> None: + self.cap_usd = cap_usd + self.usd_per_token = usd_per_token + self.spend = Spend() + self._reserved = 0 + self._ratio = 1.0 + self._lock = threading.Lock() + + @property + def spent_usd(self) -> float: + return self.spend.tokens * self.usd_per_token + + def reserve(self, estimate: int) -> int: + """Reserve a call; raises BudgetExceeded instead of going over.""" + with self._lock: + held = int(estimate * self._ratio) + 1 + if (self.spend.tokens + self._reserved + held) * self.usd_per_token > self.cap_usd: + raise BudgetExceeded( + f"the next call would take spending past the ${self.cap_usd:.2f} cap " + f"(spent ${self.spent_usd:.4f}, {self._reserved} tokens in flight)") + self._reserved += held + return held + + def settle(self, held: int, estimate: int, actual: int | None) -> None: + """Replace a reservation by the billed tokens, None if the call failed + before Jev billed anything.""" + with self._lock: + self._reserved -= held + if actual is None: + return + self.spend.tokens += actual + self.spend.calls += 1 + self.spend.estimated_tokens += estimate + if estimate > 0: + self._ratio = min(4.0, max(self._ratio, actual / estimate)) diff --git a/bench/src/tailwise_bench/cache.py b/bench/src/tailwise_bench/cache.py new file mode 100644 index 0000000..4922295 --- /dev/null +++ b/bench/src/tailwise_bench/cache.py @@ -0,0 +1,44 @@ +"""Every score, on disk, so a rerun or a resumed run costs nothing. + +Keyed by (dataset, trace_id, span_id, model, config hash); span_id is +"__trace__" for a whole-trace score. Only real scores are stored: a call +that failed or fell back is retried on the next run. +""" + +from __future__ import annotations + +import json +import sqlite3 +import threading +from pathlib import Path +from typing import Any + +TRACE = "__trace__" + + +class ScoreCache: + def __init__(self, path: Path) -> None: + path.parent.mkdir(parents=True, exist_ok=True) + self._db = sqlite3.connect(path, check_same_thread=False) + self._lock = threading.Lock() + self._db.execute( + "CREATE TABLE IF NOT EXISTS scores (dataset TEXT, trace_id TEXT, span_id TEXT, model TEXT," + " config_hash TEXT, result TEXT, PRIMARY KEY (dataset, trace_id, span_id, model, config_hash))" + ) + self._db.commit() + + def get(self, key: tuple[str, str, str, str, str]) -> dict[str, Any] | None: + with self._lock: + row = self._db.execute( + "SELECT result FROM scores WHERE dataset=? AND trace_id=? AND span_id=? AND model=? AND config_hash=?", + key, + ).fetchone() + return json.loads(row[0]) if row else None + + def put(self, key: tuple[str, str, str, str, str], result: dict[str, Any]) -> None: + with self._lock: + self._db.execute("INSERT OR REPLACE INTO scores VALUES (?, ?, ?, ?, ?, ?)", (*key, json.dumps(result))) + self._db.commit() + + def close(self) -> None: + self._db.close() diff --git a/bench/src/tailwise_bench/cli.py b/bench/src/tailwise_bench/cli.py new file mode 100644 index 0000000..6c4d0d0 --- /dev/null +++ b/bench/src/tailwise_bench/cli.py @@ -0,0 +1,251 @@ +"""uv run bench run --datasets tau,swe[,trail] [--dry-run] [--limit N] [--config bench/config.yaml]""" + +from __future__ import annotations + +import argparse +import asyncio +import json +import subprocess +import sys +import time +from dataclasses import replace +from datetime import UTC, datetime +from pathlib import Path +from typing import Any + +from . import analyze, publish, report, rules, source +from .budget import BudgetGuard +from .cache import ScoreCache +from .client import ScoringClient +from .config import Config, load +from .loaders import swe, tau, trail +from .runner import RunResult, estimate, run +from .service import Tailwise, version +from .tokens import Estimator +from .types import Run + +DEFAULT_CONFIG = Path(__file__).resolve().parents[2] / "config.yaml" +DATASETS = ("tau", "swe", "trail") + + +def git_sha(repo: Path) -> str: + def git(*args: str) -> str: + return subprocess.run(["git", *args], cwd=repo, capture_output=True, text=True).stdout.strip() + sha = git("rev-parse", "--short", "HEAD") or "nogit" + return sha + ("-dirty" if git("status", "--porcelain") else "") + + +def load_runs(cfg: Config, names: list[str], limit: int | None) -> tuple[list[Run], dict[str, Any]]: + cache = cfg.resolve(cfg.cache_dir) + runs: list[Run] = [] + stats: dict[str, Any] = {} + for n in names: + d = getattr(cfg.datasets, n) + if n == "tau": + r, stats[n] = tau.load(d["commit"], list(d["files"]), cache, limit) + elif n == "swe": + r, stats[n] = swe.load(d["repo"], int(d["shards"]), int(d["shards_total"]), int(d["sample"]), cfg.seed, + cache, limit, int(d.get("first_shard", 0))) + else: + r, stats[n] = trail.load(d["repo"], list(d["splits"]), cache, limit) + runs += r + return runs, stats + + +def dry_run(cfg: Config, runs: list[Run], est: Estimator, cache: ScoreCache) -> int: + rows = estimate(runs, est, cache, cfg) + print(f"Dry run: no API calls. Tokens are tiktoken {cfg.cost.tokenizer} estimates of what Jev is sent " + f"(state + {est.questions_tokens} tokens of questions per call).\n") + print(f"{'dataset/subset':36} {'runs':>6} {'cached':>7} {'tokens':>12} {'avg/run':>8} {'cost':>9}") + totals: dict[str, list[int]] = {} + for key, r in sorted(rows.items()): + avg = r["tokens"] // r["calls"] if r["calls"] else 0 + print(f"{key:36} {r['calls']:6} {r['cached']:7} {r['tokens']:12,} {avg:8,} ${cfg.cost.usd(r['tokens']):8.4f}") + t = totals.setdefault(key.split("/")[0], [0, 0]) + t[0] += r["calls"] + t[1] += r["tokens"] + print() + for ds, (calls, tokens) in totals.items(): + print(f"{ds:36} {calls:6} {'':7} {tokens:12,} {'':8} ${cfg.cost.usd(tokens):8.4f}") + total = sum(t[1] for t in totals.values()) + print(f"{'total':36} {sum(t[0] for t in totals.values()):6} {'':7} {total:12,} {'':8} ${cfg.cost.usd(total):8.4f}") + verdict = "within" if cfg.cost.usd(total) <= cfg.cost.budget_usd else "OVER" + print(f"\nProjected ${cfg.cost.usd(total):.4f} is {verdict} the ${cfg.cost.budget_usd:.2f} budget cap " + f"(Jev has billed about 25% more than these estimates).") + return 0 + + +def cmd_run(args: argparse.Namespace) -> int: + cfg = load(Path(args.config)) + if args.manifest: + cfg = replace(cfg, tailwise=replace(cfg.tailwise, manifest=str(Path(args.manifest).resolve()))) + if args.evidence: + cfg = replace(cfg, tailwise=replace(cfg.tailwise, evidence=args.evidence)) + if args.rules: + cfg = replace(cfg, tailwise=replace(cfg.tailwise, rules=str(Path(args.rules).resolve()))) + if args.budget is not None: + cfg = replace(cfg, cost=replace(cfg.cost, budget_usd=args.budget)) + names = [d.strip() for d in args.datasets.split(",") if d.strip()] + if unknown := set(names) - set(DATASETS): + print(f"unknown datasets {sorted(unknown)}; choose from {', '.join(DATASETS)}", file=sys.stderr) + return 2 + started = time.monotonic() + runs, stats = load_runs(cfg, names, args.limit) + est = Estimator(cfg.cost.tokenizer, cfg.manifest_text) + cache = ScoreCache(cfg.resolve(cfg.cache_dir) / "scores.sqlite") + try: + if args.dry_run: + return dry_run(cfg, runs, est, cache) + + sha = git_sha(cfg.base.parent) + run_id = f"{datetime.now(UTC).strftime('%Y%m%dT%H%M%SZ')}_{sha}" + out = cfg.resolve(cfg.results_dir) / run_id + out.mkdir(parents=True) + budget = BudgetGuard(cfg.cost.budget_usd, cfg.cost.usd(1)) + print(f"Scoring {len(runs)} runs ({cfg.variant} questions), cap ${cfg.cost.budget_usd:.2f}; results in {out}", + file=sys.stderr) + with Tailwise(cfg, out) as tw: + async def go() -> RunResult: + client = ScoringClient(cfg.tailwise.url, cfg.client, budget, cfg.seed) + try: + return await run(runs, client, cache, est, cfg) + finally: + await client.close() + res = asyncio.run(go()) + tw_version = version(tw.binary) if cfg.tailwise.start else "external" + + sources = sorted({r.get("tokens_source", "") for r in res.results.values()} - {""}) + tokens = sum(int(r.get("tokens", 0)) for r in res.results.values()) + meta = { + "source": source.record(cfg.base.parent, out), + "baseline": Path(args.baseline).name if args.baseline else None, + "run_id": run_id, "git_sha": sha, "tailwise_version": tw_version, + "jev_models": sorted({str(r.get("model", "")) for r in res.results.values()} - {""}), + "variant": cfg.variant, "datasets": names, "limit": args.limit, + "scored": res.scored, "cached": res.cached, "failed": len(res.errors), "aborted": res.aborted, + "total_tokens": tokens, "total_cost_usd": cfg.cost.usd(tokens), + "spent_this_run_usd": budget.spent_usd, "tokens_this_run": budget.spend.tokens, + "estimated_tokens_this_run": budget.spend.estimated_tokens, + "token_source": ("as reported by Jev (usage.input_tokens)" if sources == ["api"] else + f"sources: {', '.join(sources) or 'none'} (estimate = tiktoken {cfg.cost.tokenizer})"), + "duration_seconds": time.monotonic() - started, + "dataset_stats": stats, + "config": cfg.to_dict(), + } + if res.errors: + report.write_json(out / "errors.json", [{"trace_id": k, "error": v} for k, v in res.errors.items()]) + questions = sorted({q for r in res.results.values() for q in (r.get("answers") or {})}) + verdicts = None + if cfg.tailwise.rules: + rule_files = rules.files(cfg.resolve(cfg.tailwise.rules), names) + meta["rules"] = {ds: {"file": str(rule_files[ds]), "sha256": h} for ds, h in rules.digest(rule_files).items()} + verdicts = rules.check(runs, rule_files, cfg.base.parent) + rows = analyze.rows(runs, res.results, res.errors, cfg, questions, verdicts) + if verdicts is not None: + # A keep rule decides without Jev, so with the rules those runs cost nothing. + meta["tokens_with_rules"] = sum(r["tokens"] for r in rows) + summary = analyze.summary(rows, cfg) + report.write_csv(out / "runs.csv", rows) + report.write_json(out / "summary.json", {"meta": meta, **summary}) + plot_paths = report.make_plots(summary, analyze.roc_pr_curves(rows), out / "plots") + baseline = json.loads((Path(args.baseline) / "summary.json").read_text()) if args.baseline else None + notes = Path(args.notes).read_text() if args.notes else "" + report.write_report(out, summary, meta, plot_paths, baseline, notes) + print(f"\n{report.summary_paragraph(summary, meta)}\n\nWrote {out}") + if res.aborted: + print(f"Stopped before the budget cap: {res.aborted}. Rerun to resume from the cache.", file=sys.stderr) + return 3 + return 0 + finally: + cache.close() + + +def cmd_investigate(args: argparse.Namespace) -> int: + from .investigate.run import investigate + + cfg = load(Path(args.config)) + if args.narrow_check or args.completeness_check: + if not args.into: + print("--narrow-check and --completeness-check need --into ", file=sys.stderr) + return 2 + grounded = Path(args.grounded_run) if args.grounded_run else None + if args.narrow_check: + from .investigate.narrow import GROUPS, narrow_check + + groups = [g.strip() for g in args.groups.split(",") if g.strip()] if args.groups else None + if groups and (bad := set(groups) - set(GROUPS)): + print(f"unknown groups {sorted(bad)}; choose from {', '.join(GROUPS)}", file=sys.stderr) + return 2 + result = narrow_check(cfg, Path(args.into), Path(args.baseline), grounded, args.budget, groups, + args.per_group, args.dry_run, "narrow_check_large" if groups else "narrow_check") + else: + from .investigate.completeness import completeness_check + + result = completeness_check(cfg, Path(args.into), Path(args.baseline), grounded, args.budget, + args.control, args.dry_run) + print(json.dumps(result, indent=2)) + return 3 if result.get("aborted") else 0 + run_id = f"{datetime.now(UTC).strftime('%Y%m%dT%H%M%SZ')}_investigation" + out = cfg.resolve(cfg.results_dir) / run_id + out.mkdir(parents=True) + src = source.record(cfg.base.parent, out) + started = time.monotonic() + summary = investigate(cfg, Path(args.baseline), Path(args.prompt_run) if args.prompt_run else None, out, + cfg.resolve(cfg.tailwise.binary), Path(args.answer_wrong_manifest)) + meta = {"run_id": run_id, "source": src, "baseline": Path(args.baseline).name, + "prompt_run": Path(args.prompt_run).name if args.prompt_run else None, + "flag": "answer_wrong >= 0.5 (answer_wrong alone decides)", "seed": cfg.seed, + "duration_seconds": time.monotonic() - started, "jev_calls": 0} + report.write_json(out / "summary.json", {"meta": meta, **summary}) + print(f"Wrote {out}") + return 0 + + +def cmd_publish(args: argparse.Namespace) -> int: + data = publish.write([Path(r) for r in args.runs], Path(args.out)) + print(f"Wrote {args.out}: {len(data['results'])} results") + return 0 + + +def main(argv: list[str] | None = None) -> int: + p = argparse.ArgumentParser(prog="bench", description="Benchmark how well Tailwise picks failed agent runs.") + sub = p.add_subparsers(dest="cmd", required=True) + r = sub.add_parser("run", help="score the datasets and write a results folder") + r.add_argument("--datasets", default="tau,swe", help=f"comma-separated: {', '.join(DATASETS)}") + r.add_argument("--dry-run", action="store_true", help="estimate tokens and cost; no API calls") + r.add_argument("--limit", type=int, default=None, help="at most N runs per dataset (a smoke test)") + r.add_argument("--config", default=str(DEFAULT_CONFIG)) + r.add_argument("--manifest", help="score with this manifest instead of tailwise.manifest (a labelled variant)") + r.add_argument("--budget", type=float, help="hard spending cap in dollars for this run (cost.budget_usd)") + r.add_argument("--evidence", choices=["first", "grounded"], help="Tailwise's scorer.evidence for this run") + r.add_argument("--rules", help="a folder of .yaml heuristic rules, checked on the cached answers (no cost)") + r.add_argument("--baseline", help="an earlier results folder: the report compares this run with it") + r.add_argument("--notes", help="a Markdown file: what changed since the baseline, added to the report") + inv = sub.add_parser("investigate", help="the offline τ-bench diagnosis: what Jev sees, grounding, gold actions") + inv.add_argument("--baseline", default="results/20261031T171853Z_e4c33e4-dirty", + help="a results folder with Jev's answers for every run (system prompt off)") + inv.add_argument("--prompt-run", default="results/20261031T190958Z_e4c33e4-dirty", + help="a results folder scored with the system prompt on, for its token counts") + inv.add_argument("--answer-wrong-manifest", default="manifests/answer-wrong-only.json") + inv.add_argument("--narrow-check", action="store_true", + help="step 7 (calls Jev): narrow checks of each write in the failed-but-rated-fine sample") + inv.add_argument("--into", help="with --narrow-check: the investigation folder to read samples from and write to") + inv.add_argument("--grounded-run", help="with --narrow-check: a results folder scored with evidence grounded") + inv.add_argument("--budget", type=float, default=0.05, help="with an experiment: hard cap in dollars") + inv.add_argument("--groups", help="with --narrow-check: detectable_mixup,grounded_wrong,extra,control " + "(none: the failed-but-rated-fine sample)") + inv.add_argument("--per-group", type=int, default=75, help="with --groups: writes sampled per group") + inv.add_argument("--completeness-check", action="store_true", + help="experiment (calls Jev): did the agent handle every request the customer made?") + inv.add_argument("--control", type=int, default=200, help="with --completeness-check: successful runs sampled") + inv.add_argument("--dry-run", action="store_true", help="with an experiment: count and project cost, no calls") + inv.add_argument("--config", default=str(DEFAULT_CONFIG)) + pub = sub.add_parser("publish", help="write the website's figures from results folders") + pub.add_argument("runs", nargs="+", help="results folders: built-in, manifest and +rules runs") + pub.add_argument("--out", default=str(DEFAULT_CONFIG.parent / "published" / "benchmark.json")) + args = p.parse_args(argv) + return {"investigate": cmd_investigate, "publish": cmd_publish}.get(args.cmd, cmd_run)(args) + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/bench/src/tailwise_bench/client.py b/bench/src/tailwise_bench/client.py new file mode 100644 index 0000000..8de2fd0 --- /dev/null +++ b/bench/src/tailwise_bench/client.py @@ -0,0 +1,102 @@ +"""Scores traces through Tailwise's POST /v1/otlp/traces:score. + +Rate limited, with exponential backoff on 429 and 5xx. A fallback decision +(Tailwise could not get a score from Jev) is retried once. A 422 (nothing to +score) or other 4xx is final. +""" + +from __future__ import annotations + +import asyncio +import random +import time +from dataclasses import dataclass +from typing import Any + +import httpx + +from .budget import BudgetGuard +from .config import Client as ClientConfig + +PATH = "/v1/otlp/traces:score" +RETRY_STATUS = {429, 500, 502, 503, 504} + + +@dataclass +class Outcome: + result: dict[str, Any] | None # Tailwise's response; None on failure + error: str = "" + attempts: int = 0 + + +class RateLimiter: + """At most rate starts per second, spaced evenly.""" + + def __init__(self, rate: float) -> None: + self._interval = 1.0 / rate if rate > 0 else 0.0 + self._next = 0.0 + self._lock = asyncio.Lock() + + async def wait(self) -> None: + async with self._lock: + now = time.monotonic() + if self._next > now: + await asyncio.sleep(self._next - now) + self._next = max(now, self._next) + self._interval + + +class ScoringClient: + def __init__(self, base_url: str, cfg: ClientConfig, budget: BudgetGuard, seed: int) -> None: + self._http = httpx.AsyncClient(base_url=base_url, timeout=cfg.timeout_seconds) + self._cfg = cfg + self._budget = budget + self._limiter = RateLimiter(cfg.requests_per_second) + self._rng = random.Random(seed) + + @property + def spent_usd(self) -> float: + return self._budget.spent_usd + + async def close(self) -> None: + await self._http.aclose() + + async def score(self, payload: dict[str, Any], estimate: int) -> Outcome: + """One trace scored; raises BudgetExceeded before a call that would go + over the cap.""" + last = "" + fallbacks = 0 + for attempt in range(1, self._cfg.max_attempts + 1): + held = self._budget.reserve(estimate) + billed: int | None = None + try: + await self._limiter.wait() + resp = await self._http.post(PATH, json=payload) + if resp.status_code == 200: + body = resp.json() + if body.get("rule") != "fallback": + # Jev reports what it billed; if it ever does not, the + # estimate stands in, and the row says so. + reported = int(body.get("input_tokens") or 0) + body["tokens"] = reported or estimate + body["tokens_source"] = "api" if reported else "estimate" + billed = body["tokens"] + return Outcome(result=body, attempts=attempt) + # Tailwise already retried Jev; a second fallback is most + # likely a rejection (e.g. a trace too long for Jev), which + # retrying does not change. + last = f"fallback: {body.get('reason', '')}" + fallbacks += 1 + if fallbacks >= 2: + return Outcome(result=None, error=last, attempts=attempt) + elif resp.status_code in RETRY_STATUS: + last = f"HTTP {resp.status_code}: {resp.text[:200]}" + else: + return Outcome(result=None, error=f"HTTP {resp.status_code}: {resp.text[:300]}", attempts=attempt) + except httpx.TransportError as e: + last = f"{type(e).__name__}: {e}" + finally: + self._budget.settle(held, estimate, billed) + if attempt < self._cfg.max_attempts: + delay = self._cfg.backoff_seconds * 2 ** (attempt - 1) + await asyncio.sleep(delay * (0.5 + self._rng.random())) + return Outcome(result=None, error=last, attempts=self._cfg.max_attempts) diff --git a/bench/src/tailwise_bench/config.py b/bench/src/tailwise_bench/config.py new file mode 100644 index 0000000..6364f3c --- /dev/null +++ b/bench/src/tailwise_bench/config.py @@ -0,0 +1,178 @@ +"""The benchmark config: bench/config.yaml, typed.""" + +from __future__ import annotations + +import hashlib +import json +from dataclasses import asdict, dataclass, field +from functools import cache +from pathlib import Path +from typing import Any + +import yaml + + +# Tailwise's sources that decide what Jev is sent, relative to the repository. +STATE_CODE = [Path("internal/trace/*.go"), Path("internal/otlp/convert.go")] + + +@dataclass(frozen=True) +class Tailwise: + start: bool = True + binary: str = "../bin/tailwise" + url: str = "http://127.0.0.1:18080" + envchain: str = "observer-ai" + model: str = "jev-latest" + keep_threshold: float = 0.5 + manifest: str = "" + system_prompt: bool = False + evidence: str = "grounded" + evidence_budget: int = 12000 + # A folder of .yaml heuristic rule files; "" runs none. Not part + # of the score cache: rules do not change what Jev is asked. + rules: str = "" + + +@dataclass(frozen=True) +class Cost: + usd_per_million_input_tokens: float = 0.042 + budget_usd: float = 3.0 + tokenizer: str = "cl100k_base" + + def usd(self, tokens: int) -> float: + return tokens * self.usd_per_million_input_tokens / 1_000_000 + + +@dataclass(frozen=True) +class Client: + concurrency: int = 8 + requests_per_second: float = 8.0 + max_attempts: int = 6 + backoff_seconds: float = 1.0 + timeout_seconds: float = 120.0 + + +@dataclass(frozen=True) +class Datasets: + tau: dict[str, Any] = field(default_factory=lambda: { + "commit": "59a200c6d575d595120f1cb70fea53cef0632f6b", + "files": ["gpt-4o-retail", "gpt-4o-airline", "sonnet-35-new-retail", "sonnet-35-new-airline"], + }) + swe: dict[str, Any] = field(default_factory=lambda: { + "repo": "nebius/SWE-agent-trajectories", "first_shard": 0, "shards": 1, "shards_total": 12, "sample": 1000, + }) + trail: dict[str, Any] = field(default_factory=lambda: { + "repo": "PatronusAI/TRAIL", "splits": ["gaia", "swe_bench"], + }) + + +@dataclass(frozen=True) +class Config: + path: Path + seed: int = 0 + random_seeds: int = 100 + budgets: tuple[float, ...] = (0.01, 0.05, 0.10, 0.20, 0.30, 0.50) + review_budget: float = 0.05 + reference_failure_rates: tuple[float, ...] = (0.02, 0.05, 0.10, 0.20) + tailwise: Tailwise = Tailwise() + cost: Cost = Cost() + client: Client = Client() + datasets: Datasets = Datasets() + cache_dir: str = ".cache" + results_dir: str = "results" + + @property + def base(self) -> Path: + return self.path.parent + + def resolve(self, p: str) -> Path: + path = Path(p) + return path if path.is_absolute() else (self.base / path).resolve() + + @property + def manifest_text(self) -> str: + """The manifest's content, "" for the built-in questions.""" + if not self.tailwise.manifest: + return "" + return self.resolve(self.tailwise.manifest).read_text() + + @property + def actions(self) -> dict[str, str]: + """The manifest's actions block: the tools the action checks cover.""" + text = self.manifest_text + return dict(json.loads(text).get("actions") or {}) if text else {} + + def deciding_questions(self, decision_question: str) -> list[str]: + """The questions Tailwise's severity averages: the decision question, + the manifest's decision.any_of, then the action checks.""" + text = self.manifest_text + manifest = json.loads(text) if text else {} + out = [decision_question, *((manifest.get("decision") or {}).get("any_of") or [])] + if manifest.get("actions"): + out += ["action_wrong", "unhandled_request"] + return out + + @property + def variant(self) -> str: + name = Path(self.tailwise.manifest).stem if self.tailwise.manifest else "builtin" + return name + ("+rules" if self.tailwise.rules else "") + + def score_hash(self) -> str: + """Hash of everything that changes what Jev is sent or how it is read, + including the Tailwise code that turns spans into Jev's state.""" + relevant = {"model": self.tailwise.model, "keep_threshold": self.tailwise.keep_threshold, + "manifest": self.manifest_text, "system_prompt": self.tailwise.system_prompt, + "evidence": self.tailwise.evidence, + "evidence_budget": self.tailwise.evidence_budget if self.tailwise.evidence == "grounded" else None, + "state_code": self.state_code_hash()} + return hashlib.sha256(json.dumps(relevant, sort_keys=True).encode()).hexdigest()[:16] + + def state_code_hash(self) -> str: + """Hash of the Tailwise sources that build the state from spans, so a + change to them re-scores instead of reusing the cache.""" + return _state_code_hash(self.base.parent) + + + def to_dict(self) -> dict[str, Any]: + d = asdict(self) + d["path"] = str(self.path) + d["score_hash"] = self.score_hash() + return d + + +@cache +def _state_code_hash(repo: Path) -> str: + """Read once per run, so the hash cannot change while a run is scoring.""" + h = hashlib.sha256() + for path in sorted(STATE_CODE, key=str): + for f in sorted((repo / path.parent).glob(path.name)): + if not f.name.endswith("_test.go"): + h.update(f.name.encode() + f.read_bytes()) + return h.hexdigest()[:16] + + +def load(path: Path) -> Config: + raw = yaml.safe_load(path.read_text()) or {} + known = {"seed", "random_seeds", "budgets", "review_budget", "reference_failure_rates", "tailwise", "cost", "client", + "datasets", "cache_dir", "results_dir"} + if unknown := set(raw) - known: + raise ValueError(f"{path}: unknown keys {sorted(unknown)}") + defaults = Datasets() + ds = raw.get("datasets", {}) + cfg = Config( + path=path.resolve(), + seed=int(raw.get("seed", 0)), + random_seeds=int(raw.get("random_seeds", 100)), + budgets=tuple(float(b) for b in raw.get("budgets", Config.budgets)), + review_budget=float(raw.get("review_budget", Config.review_budget)), + reference_failure_rates=tuple(float(r) for r in raw.get("reference_failure_rates", Config.reference_failure_rates)), + tailwise=Tailwise(**raw.get("tailwise", {})), + cost=Cost(**raw.get("cost", {})), + client=Client(**raw.get("client", {})), + datasets=Datasets(**{k: {**getattr(defaults, k), **v} for k, v in ds.items()}), + cache_dir=str(raw.get("cache_dir", ".cache")), + results_dir=str(raw.get("results_dir", "results")), + ) + if not all(0 < b <= 1 for b in (*cfg.budgets, cfg.review_budget, *cfg.reference_failure_rates)): + raise ValueError("budgets, review_budget and reference_failure_rates must be shares in (0, 1]") + return cfg diff --git a/bench/src/tailwise_bench/investigate/__init__.py b/bench/src/tailwise_bench/investigate/__init__.py new file mode 100644 index 0000000..8f70bcf --- /dev/null +++ b/bench/src/tailwise_bench/investigate/__init__.py @@ -0,0 +1,5 @@ +"""Diagnosis of Tailwise on τ-bench: what Jev is shown, whether the agent's +actions were grounded or wrong, and how that lines up with Tailwise's flags. + +Everything here runs offline except the opt-in experiments. +""" diff --git a/bench/src/tailwise_bench/investigate/audit.py b/bench/src/tailwise_bench/investigate/audit.py new file mode 100644 index 0000000..eb11c47 --- /dev/null +++ b/bench/src/tailwise_bench/investigate/audit.py @@ -0,0 +1,418 @@ +"""Plain-code checks on τ-bench runs, no model involved. + +- grounding: does every argument of a write call come from something the + agent had seen before making it: the user's messages, the policy (system + prompt), or an earlier tool result? +- gold: do the write calls match the task's ground-truth actions? +- visibility: which tool results reached the model in a captured request? +""" + +from __future__ import annotations + +import json +import re +from dataclasses import dataclass, field +from typing import Any + +from ..loaders import tau + +READ_PREFIXES = ("get_", "find_", "list_", "search_", "calculate", "think") +NOT_ACTIONS = {"transfer_to_human_agents"} # hands over; its only argument is a free-text summary +SKIP_KEYS = {"reason", "summary"} # enums and free text the agent writes itself +ENUM_KEYS = {"cabin", "flight_type", "insurance"} +COUNT_KEYS = {"total_baggages", "nonfree_baggages"} +ITEM_WRITES = {"exchange_delivered_order_items", "modify_pending_order_items"} +TARGET_KEYS = ("order_id", "reservation_id", "user_id") +# An id or text argument found nowhere the agent could have read it makes a +# write ungrounded. A date or amount not found word for word is counted as +# derived instead: agents compute them ("one day later", a fare total). +CHECKED_KINDS = ("id", "text") +MONTHS = ["january", "february", "march", "april", "may", "june", "july", "august", "september", "october", + "november", "december"] + + +def is_write(name: str) -> bool: + return not name.startswith(READ_PREFIXES) and name not in NOT_ACTIONS + + +# --- grounding ----------------------------------------------------------------- + +@dataclass +class Leaf: + path: str + value: Any + kind: str # id, date, text, amount, enum, count + source: str = "" # user, system, tool:, or "" when not found + tool_index: int | None = None + seen_in_tool: bool = False # in any earlier tool result, whatever the source + + +@dataclass +class WriteCall: + call_index: int + name: str + args: dict[str, Any] + message_index: int + leaves: list[Leaf] = field(default_factory=list) + output: str = "" + # How far back the furthest tool result it relied on was: tool calls and + # characters of conversation in between; None when no leaf came from a result. + calls_between: int | None = None + chars_between: int | None = None + + @property + def ungrounded(self) -> list[Leaf]: + return [lf for lf in self.leaves if lf.kind in CHECKED_KINDS and not lf.source] + + @property + def derived(self) -> list[Leaf]: + return [lf for lf in self.leaves if lf.kind in ("amount", "date") and not lf.source] + + @property + def acted_without_lookup(self) -> bool: + """Its target (order, reservation, user) was never in a tool result + before the call: the agent acted on it without looking it up.""" + return any(lf.path in TARGET_KEYS and not lf.seen_in_tool for lf in self.leaves) + + +def leaves(obj: Any, path: str = "") -> list[tuple[str, Any]]: + if isinstance(obj, dict): + return [x for k, v in obj.items() for x in leaves(v, f"{path}.{k}" if path else k)] + if isinstance(obj, list): + return [x for i, v in enumerate(obj) for x in leaves(v, f"{path}[{i}]")] + return [(path, obj)] + + +def kind_of(path: str, value: Any) -> str: + key = re.sub(r"\[\d+\]", "", path).split(".")[-1] + if isinstance(value, bool) or value is None or key in SKIP_KEYS: + return "skip" + if isinstance(value, int | float): + return "count" if key in COUNT_KEYS or (float(value).is_integer() and abs(value) < 10) else "amount" + s = str(value).strip() + if not s: + return "skip" + if key in ENUM_KEYS or (re.fullmatch(r"[a-z_]+", s) and len(s) <= 20): + return "enum" + if re.fullmatch(r"\d{4}-\d{2}-\d{2}", s): + return "date" + if re.search(r"\d", s): + return "id" + return "text" + + +def _token_re(value: str) -> re.Pattern[str]: + return re.compile(r"(? list[str]: + forms = {f"{v:.2f}".rstrip("0").rstrip("."), f"{v:.2f}", f"{v:,.2f}"} + if float(v).is_integer(): + forms |= {str(int(v)), f"{int(v):,}"} + return sorted(forms) + + +def found_in(kind: str, value: Any, text: str, natural: bool) -> bool: + """Whether value appears in text (lower-cased). natural allows the forms a + person would type (dates as "May 20").""" + if kind == "id": + return bool(_token_re(str(value).strip().lower().lstrip("#")).search(text)) + if kind == "text": + return " ".join(str(value).lower().split()) in " ".join(text.split()) + if kind == "amount": + return any(_token_re(f).search(text) for f in number_forms(float(value))) + if kind == "date": + s = str(value) + if s in text: + return True + if natural: + y, m, d = (int(x) for x in s.split("-")) + month = MONTHS[m - 1] + for form in (f"{month} {d}", f"{d} {month}", f"{month[:3]} {d}", f"{d} {month[:3]}", f"{m}/{d}", + f"{m:02d}/{d:02d}", f"{month} {d}th", f"{month} {d}st", f"{month} {d}nd", f"{month} {d}rd"): + if _token_re(form).search(text): + return True + return False + return True + + +def ground(traj: list[dict[str, Any]], system_prompt: str) -> list[WriteCall]: + """Every write call in the run, each argument traced to where the agent + could have taken it from before the call.""" + user_text = "" + outputs: list[tuple[int, str]] = [] # (call index, lower-cased result) in the order results arrived + result_msg: dict[int, int] = {} # call index -> message index of its result + # characters of conversation before each message, for distances + before = [0] + for m in traj: + before.append(before[-1] + len(m.get("content") or "") + len(json.dumps(m.get("tool_calls") or ""))) + writes: list[WriteCall] = [] + paired = tau.pair_results(traj) + call_of_msg = {mi: ci for ci, (_, mi) in paired.items()} + system = system_prompt.lower() + n = 0 + for mi, m in enumerate(traj): + role = m.get("role") + if role == "user": + user_text += "\n" + (m.get("content") or "").lower() + elif role == "tool": + ci = call_of_msg.get(mi, -1) + outputs.append((ci, (m.get("content") or "").lower())) + result_msg[ci] = mi + elif role == "assistant": + for tc in m.get("tool_calls") or []: + fn = tc.get("function") or {} + args: Any = fn.get("arguments") or {} + if isinstance(args, str): + try: + args = json.loads(args or "{}") + except json.JSONDecodeError: + args = {} + name = fn.get("name") or "" + if is_write(name) and isinstance(args, dict): + w = WriteCall(n, name, args, mi, output=str(paired[n][0].get("content") or "") if n in paired else "") + for path, value in leaves(args): + lf = Leaf(path, value, kind_of(path, value)) + if lf.kind in ("id", "date", "text", "amount"): + lf.seen_in_tool = any(found_in(lf.kind, value, out, natural=False) for _, out in outputs) + if found_in(lf.kind, value, user_text, natural=True): + lf.source = "user" + elif found_in(lf.kind, value, system, natural=False): + lf.source = "system" + else: + for ci, out in reversed(outputs): + if found_in(lf.kind, value, out, natural=False): + lf.source, lf.tool_index = f"tool:{ci}", ci + break + w.leaves.append(lf) + sources = [lf.tool_index for lf in w.leaves if lf.tool_index is not None] + if sources: + furthest = min(sources) + w.calls_between = n - furthest - 1 + w.chars_between = before[mi] - before[result_msg[furthest] + 1] + writes.append(w) + n += 1 + return writes + + +# --- gold --------------------------------------------------------------------- + +def norm(v: Any) -> Any: + if isinstance(v, dict): + return {k: norm(x) for k, x in v.items()} + if isinstance(v, list): + items = [norm(x) for x in v] + return sorted(items, key=json.dumps) if all(not isinstance(x, dict | list) for x in items) else items + if isinstance(v, bool) or v is None: + return v + if isinstance(v, int | float): + return round(float(v), 2) + return str(v).strip() + + +def target(args: dict[str, Any]) -> tuple[str, Any] | None: + for k in TARGET_KEYS: + if k in args: + return k, norm(args[k]) + return None + + +@dataclass +class GoldResult: + rows: list[dict[str, Any]] # one per run write call and per missed gold action + + def has(self, category: str) -> bool: + return any(r["category"] == category for r in self.rows) + + +def compare_gold(writes: list[WriteCall], gold: list[dict[str, Any]], traj: list[dict[str, Any]] | None = None) -> GoldResult: + gold_writes = [g for g in gold if is_write(g.get("name", ""))] + left = list(range(len(gold_writes))) + rows: list[dict[str, Any]] = [] + pending: list[WriteCall] = [] + for w in writes: + hit = next((i for i in left if gold_writes[i]["name"] == w.name + and norm(gold_writes[i].get("kwargs", {})) == norm(w.args)), None) + if hit is not None: + left.remove(hit) + rows.append({"call_index": w.call_index, "tool": w.name, "category": "correct", "diff": "", + "grounded_wrong": False, "variant_mixup": False, "mixup_detectable_from_trace": None}) + else: + pending.append(w) + for w in pending: + t = target(w.args) + hit = next((i for i in left if gold_writes[i]["name"] == w.name + and target(gold_writes[i].get("kwargs", {})) == t), None) + if hit is None: + rows.append({"call_index": w.call_index, "tool": w.name, "category": "extra", "diff": "", + "grounded_wrong": False, "variant_mixup": False, "mixup_detectable_from_trace": None}) + continue + left.remove(hit) + g = gold_writes[hit].get("kwargs", {}) + traj = traj or [] + keys = sorted(k for k in set(g) | set(w.args) if norm(g.get(k)) != norm(w.args.get(k))) + bad = [lf for lf in w.leaves if lf.path.split(".")[0].split("[")[0] in keys] + grounded = all(lf.source or lf.kind not in CHECKED_KINDS for lf in bad) + detectable = None + if w.name in ITEM_WRITES and "new_item_ids" in keys: + detectable = mixup_detectable(traj, [str(x) for x in w.args.get("new_item_ids") or []], + [str(x) for x in g.get("new_item_ids") or []]) + rows.append({ + "call_index": w.call_index, "tool": w.name, "category": "wrong_args", + "mixup_detectable_from_trace": detectable, + "diff": "; ".join(f"{k}: {json.dumps(w.args.get(k))} vs gold {json.dumps(g.get(k))}" for k in keys), + "grounded_wrong": grounded, + "variant_mixup": w.name in ITEM_WRITES and "new_item_ids" in keys and grounded, + }) + for i in left: + rows.append({"call_index": None, "tool": gold_writes[i]["name"], "category": "missed", + "diff": json.dumps(gold_writes[i].get("kwargs", {}))[:300], + "grounded_wrong": False, "variant_mixup": False, "mixup_detectable_from_trace": None}) + return GoldResult(rows) + + +# --- what reached the model ------------------------------------------------------- + +OUT_KEYS = ["result", "output", "content", "text"] + + +def message_text(v: str) -> str: + """Tailwise's messageText for a tool output (internal/otlp/convert.go).""" + t = v.strip() + if not t: + return "" + if t[0] == '"': + try: + s = json.loads(t) + return s if isinstance(s, str) else v + except json.JSONDecodeError: + return v + if t[0] == "{": + try: + o = json.loads(t) + except json.JSONDecodeError: + return v + if isinstance(o.get("response"), dict): + o = o["response"] + for k in OUT_KEYS: + if isinstance(o.get(k), str): + return str(o[k]) + return v + + +def section(state: str, name: str) -> str: + heads = ["SYSTEM PROMPT", "USER QUESTION", "TOOL CALLS", "EVIDENCE", "FINAL ANSWER"] + start = state.find(name + ":\n") + if start < 0: + return "" + start += len(name) + 2 + ends = [state.find("\n\n" + h + ":\n", start) for h in heads[heads.index(name) + 1:]] + ends = [e for e in ends if e >= 0] + return state[start:min(ends)] if ends else state[start:] + + +def visibility(state: str, outputs: list[str]) -> list[str]: + """For each tool call's output: "full", "partial" or "none", by what the + captured state shows of it, on its tool-call line or in the evidence.""" + shown = ["none"] * len(outputs) + texts = [message_text(o).strip() for o in outputs] + calls = section(state, "TOOL CALLS") + blocks = re.split(r"(?m)^(?=\d+\. )", calls) + for block in blocks: + m = re.match(r"(\d+)\. ", block) + if not m: + continue + i = int(m.group(1)) - 1 + if not 0 <= i < len(outputs) or not texts[i]: + continue + first, _, more = block.partition("\n") + result = "" + for status in ("ok", "error"): + mark = f") -> {status}: " + if mark in first: + result = first.split(mark, 1)[1] + if not result and more.startswith(" "): + result = "\n".join(line[3:] for line in more.rstrip("\n").split("\n")) + if result: + shown[i] = "full" if result.strip() == texts[i] else "partial" + items = re.split(r"(?m)^\[\d+\] ", section(state, "EVIDENCE"))[1:] + for item in items: + item = item.rstrip("\n").strip() + if not item or item == "(none)": + continue + for i, t in enumerate(texts): + if shown[i] == "none" and item[:300] in t: + shown[i] = "full" if item == t else "partial" + break + return shown + + +# --- variant mix-ups: could the trace show them? -------------------------------- + +def variant_options(traj: list[dict[str, Any]]) -> dict[str, dict[str, Any]]: + """item_id -> options, from every product lookup in the run.""" + out: dict[str, dict[str, Any]] = {} + for m in traj: + if m.get("role") != "tool": + continue + try: + o = json.loads(m.get("content") or "") + except json.JSONDecodeError: + continue + if isinstance(o, dict) and isinstance(o.get("variants"), dict): + for k, v in o["variants"].items(): + if isinstance(v, dict): + out[k] = v.get("options") or {} + return out + + +def mixup_detectable(traj: list[dict[str, Any]], chosen: list[str], gold: list[str]) -> bool | None: + """Whether the user ever stated an option that tells a gold item apart + from the closest item the agent chose instead. None when the run never + looked up the variants involved.""" + options = variant_options(traj) + user = " ".join((m.get("content") or "") for m in traj if m.get("role") == "user").lower() + verdict: bool | None = None + for g in gold: + if g in chosen or g not in options: + continue + closest: list[str] | None = None + for c in chosen: + if c not in options: + continue + diff = [str(v).lower() for k, v in options[g].items() if str(v).lower() != str(options[c].get(k)).lower()] + if closest is None or len(diff) < len(closest): + closest = diff + if closest is not None: + found = any(v in user for v in closest) + verdict = found if verdict is None else verdict or found + return verdict + + +# --- which failures a trace can show ----------------------------------------------- + +def trace_detectable(runs: list[tuple[str, dict[str, Any]]]) -> list[dict[str, int]]: + """For each (domain, τ-bench row): whether its label can be judged from + the trace alone. + + - gold_consistent_task: every successful run of the task carried out the + logged gold actions exactly. On the other tasks the gold disagrees with + how runs were scored, so a run's writes say little about its label. + - mixup_not_in_trace: the run failed only by choosing the wrong variant of + an item, and the user never stated the option that tells the gold item + apart; the gold rests on the simulated user's hidden instructions. + - trace_detectable: the first and not the second. + """ + out: list[dict[str, Any]] = [] + for domain, row in runs: + system = next((m.get("content") or "" for m in row["traj"] if m.get("role") == "system"), "") + actions = ((row.get("info") or {}).get("task") or {}).get("actions") or [] + gold = compare_gold(ground(row["traj"], system), actions, row["traj"]) + wrong = [g for g in gold.rows if g["category"] != "correct"] + out.append({"task": f"{domain}:{row['task_id']}", "succeeded": float(row["reward"]) >= 1, "wrong": bool(wrong), + "mixup_not_in_trace": bool(wrong) and all(g["mixup_detectable_from_trace"] is False for g in wrong)}) + bad_tasks = {r["task"] for r in out if r["succeeded"] and r["wrong"]} + return [{"gold_consistent_task": int(r["task"] not in bad_tasks), + "mixup_not_in_trace": int(not r["succeeded"] and r["mixup_not_in_trace"]), + "trace_detectable": int(r["task"] not in bad_tasks and not (not r["succeeded"] and r["mixup_not_in_trace"]))} + for r in out] diff --git a/bench/src/tailwise_bench/investigate/capture.py b/bench/src/tailwise_bench/investigate/capture.py new file mode 100644 index 0000000..6cf5b2d --- /dev/null +++ b/bench/src/tailwise_bench/investigate/capture.py @@ -0,0 +1,121 @@ +"""The exact requests Tailwise sends a System One model, captured without +calling one. + +A local HTTP server stands in for the model: it records every request body +and answers every question with a fixed value. Tailwise runs against it with +the settings under test, and each run is sent through the same OTLP endpoint +the benchmark uses, one at a time, so each captured request belongs to the +run just sent. +""" + +from __future__ import annotations + +import json +import subprocess +import threading +import time +from collections.abc import Iterator +from contextlib import contextmanager +from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer +from pathlib import Path +from typing import Any + +import httpx +import yaml + +from ..otel import run_trace +from ..types import Run + + +class _Recorder(ThreadingHTTPServer): + last: dict[str, Any] | None = None + lock = threading.Lock() + + +class _Handler(BaseHTTPRequestHandler): + server: _Recorder + + def log_message(self, *args: Any) -> None: + pass + + def _reply(self, body: dict[str, Any]) -> None: + data = json.dumps(body).encode() + self.send_response(200) + self.send_header("Content-Type", "application/json") + self.send_header("Content-Length", str(len(data))) + self.end_headers() + self.wfile.write(data) + + def do_GET(self) -> None: # /v1/models, Tailwise's readiness probe + self._reply({"models": [{"name": "capture"}]}) + + def do_POST(self) -> None: + req = json.loads(self.rfile.read(int(self.headers["Content-Length"]))) + with self.server.lock: + self.server.last = req + answers = {q: ({"type": "noul", "noul": 0.1} if spec.get("type") == "noul" + else {"type": "choice", "choice": "correct", "confidence": 1.0, "probabilities": {}}) + for q, spec in req.get("questions", {}).items()} + self._reply({"model": "capture", "answers": answers, "usage": {"input_tokens": 0, "output_tokens": 0}}) + + +@contextmanager +def fake_model(port: int) -> Iterator[_Recorder]: + srv = _Recorder(("127.0.0.1", port), _Handler) + t = threading.Thread(target=srv.serve_forever, daemon=True) + t.start() + try: + yield srv + finally: + srv.shutdown() + + +@contextmanager +def tailwise(binary: Path, workdir: Path, port: int, model_port: int, scorer: dict[str, Any]) -> Iterator[str]: + """Tailwise pointed at the fake model, with the given scorer settings.""" + conf = workdir / f"capture-{port}.yaml" + conf.write_text(yaml.safe_dump({ + "server": {"addr": f"127.0.0.1:{port}"}, + "otlp": {"grpc_addr": "", "http_addr": ""}, + "scorer": {"kind": "kev", "endpoint": f"http://127.0.0.1:{model_port}/v1/systemone", "model": "capture", + "max_retries": 0, **scorer}, + "policy": {"random_floor": 0}, + "exporters": [], + })) + log = (workdir / f"capture-{port}.log").open("wb") + proc = subprocess.Popen([str(binary), "-config", str(conf)], stdout=log, stderr=subprocess.STDOUT) + url = f"http://127.0.0.1:{port}" + try: + for _ in range(120): + try: + if httpx.get(url + "/readyz", timeout=2).status_code == 200: + break + except httpx.TransportError: + pass + if proc.poll() is not None: + raise RuntimeError(f"Tailwise exited; see {log.name}") + time.sleep(0.25) + yield url + finally: + proc.terminate() + proc.wait(timeout=30) + log.close() + + +def capture(runs: list[Run], binary: Path, workdir: Path, scorer: dict[str, Any], + port: int = 18190, model_port: int = 18191) -> dict[str, dict[str, Any]]: + """trace_id -> the request Tailwise sent the model for that run, or + {"error": ...} when Tailwise answered without calling it.""" + out: dict[str, dict[str, Any]] = {} + with fake_model(model_port) as rec, tailwise(binary, workdir, port, model_port, scorer) as url, \ + httpx.Client(base_url=url, timeout=60) as client: + for r in runs: + payload, _ = run_trace(r) + with rec.lock: + rec.last = None + resp = client.post("/v1/otlp/traces:score", json=payload) + with rec.lock: + got = rec.last + out[r.trace_id] = got if got is not None and resp.status_code == 200 else \ + {"error": f"HTTP {resp.status_code}: {resp.text[:200]}"} + return out diff --git a/bench/src/tailwise_bench/investigate/completeness.py b/bench/src/tailwise_bench/investigate/completeness.py new file mode 100644 index 0000000..93027c2 --- /dev/null +++ b/bench/src/tailwise_bench/investigate/completeness.py @@ -0,0 +1,197 @@ +"""Opt-in experiment: did the agent handle every request the customer made? + +Jev answers yes/no and choice questions; it does not write lists. So the +customer's requests are found by fan-out: for each kind of request the domain +allows, one question asks whether the customer made it, and a paired question +asks whether it was fulfilled by an action or explicitly declined by the +agent. A request made (≥ 0.5) and not handled (< 0.5) makes the run +incomplete. One call per run, directly over /v1/systemone, under a hard +budget. +""" + +from __future__ import annotations + +import csv +import json +import os +import random +from pathlib import Path +from typing import Any + +import httpx + +from ..budget import BudgetExceeded, BudgetGuard +from ..config import Config +from ..loaders import tau +from ..report import write_csv, write_json +from ..tokens import Estimator +from . import audit +from .run import load_tau + +ENDPOINT = "https://api.typesafe.ai/v1/systemone" +MAX_AGENT_CHARS = 600 + +# The kinds of request each domain's tools can carry out, and the tool for each. +REQUESTS = { + "retail": { + "cancel_order": ("cancel a pending order", "cancel_pending_order"), + "modify_items": ("change items in a pending order (for example another size or color)", "modify_pending_order_items"), + "modify_order_address": ("change the shipping address of a pending order", "modify_pending_order_address"), + "modify_payment": ("change the payment method of a pending order", "modify_pending_order_payment"), + "return_items": ("return delivered items for a refund", "return_delivered_order_items"), + "exchange_items": ("exchange delivered items for other items", "exchange_delivered_order_items"), + "modify_user_address": ("change their default address on their account", "modify_user_address"), + }, + "airline": { + "book": ("book a new flight reservation", "book_reservation"), + "change_flights": ("change the flights, dates or cabin of a reservation", "update_reservation_flights"), + "change_baggage": ("add or change checked bags on a reservation", "update_reservation_baggages"), + "change_passengers": ("change passenger details on a reservation", "update_reservation_passengers"), + "cancel_reservation": ("cancel a reservation", "cancel_reservation"), + "compensation": ("get compensation, a certificate or a refund for a problem", "send_certificate"), + }, +} +TYPE_OF_TOOL = {tool: kind for reqs in REQUESTS.values() for kind, (_, tool) in reqs.items()} + + +def questions(domain: str) -> dict[str, dict[str, str]]: + out = {} + for kind, (what, _) in REQUESTS[domain].items(): + out[f"requested_{kind}"] = { + "type": "noul", + "instructions": f"At any point in the CONVERSATION, did the customer ask the agent to {what}?", + } + out[f"handled_{kind}"] = { + "type": "noul", + "instructions": f"Was every customer request to {what} HANDLED, as defined in the CONVERSATION's notes?", + } + return out + + +def transcript(traj: list[dict[str, Any]]) -> str: + """The conversation as the customer and agent saw it, with the agent's + actions (not its lookups) in place.""" + paired = tau.pair_results(traj) + lines, n = [], 0 + for m in traj: + role = m.get("role") + if role == "user": + text = (m.get("content") or "").replace(tau.STOP, "").strip() + if text: + lines.append(f"CUSTOMER: {text}") + elif role == "assistant": + text = (m.get("content") or "").strip() + if text: + # Agents' long listings (products, flights) matter less than what they say they did. + lines.append(f"AGENT: {text if len(text) <= MAX_AGENT_CHARS else text[:MAX_AGENT_CHARS] + ' […]'}") + for tc in m.get("tool_calls") or []: + fn = tc.get("function") or {} + name = fn.get("name") or "" + if audit.is_write(name) or name == "transfer_to_human_agents": + args = fn.get("arguments") + args = args if isinstance(args, str) else json.dumps(args, ensure_ascii=False) + result = str(paired[n][0].get("content") or "") if n in paired else "" + lines.append(f"AGENT ACTION: {name}({args}) -> {'error: ' + result[:200] if result.startswith('Error') else 'ok'}") + n += 1 + return (HANDLED_NOTE + "\n\nCONVERSATION:\n" + "\n\n".join(lines)) + + +HANDLED_NOTE = ("NOTES: A customer request is HANDLED when the agent either carried it out with an AGENT ACTION, or " + "explicitly declined it and told the customer why. It is not handled if it was left unaddressed, done " + "only in part, or the agent said it would act but no AGENT ACTION did it. Lookups are not shown.") + + +def completeness_check(cfg: Config, investigation: Path, baseline: Path, grounded_run: Path | None, + budget_usd: float, control: int = 200, dry_run: bool = False) -> dict[str, Any]: + key = os.environ.get("JEV_API_KEY") + if not key and not dry_run: + raise SystemExit("JEV_API_KEY is not set: run under `envchain observer-ai ...`") + with (investigation / "tau_runs.csv").open() as f: + runs = {r["trace_id"]: r for r in csv.DictReader(f)} + with (investigation / "wrong_args.csv").open() as f: + missed: dict[str, set[str]] = {} + for g in csv.DictReader(f): + if g["category"] == "missed" and g["tool"] in TYPE_OF_TOOL: + missed.setdefault(g["trace_id"], set()).add(TYPE_OF_TOOL[g["tool"]]) + failing = sorted(t for t, r in runs.items() if r["failed"] == "1" and r["missed_write"] == "1") + ok_pool = sorted(t for t, r in runs.items() if r["failed"] == "0" and r["gold_consistent_task"] == "1") + ok = sorted(random.Random(cfg.seed).sample(ok_pool, min(control, len(ok_pool)))) + groups = {**{t: "missing_action" for t in failing}, **{t: "control" for t in ok}} + + def answers(folder: Path | None) -> dict[str, dict[str, str]]: + if folder is None: + return {} + with (folder / "runs.csv").open() as f: + return {r["trace_id"]: r for r in csv.DictReader(f)} + + first, grounded = answers(baseline), answers(grounded_run) + est = Estimator(cfg.cost.tokenizer) + work = [] + for run, row in load_tau(cfg): + if run.trace_id in groups: + domain = run.meta["domain"] + work.append((run, domain, transcript(row["traj"]), questions(domain))) + # A seeded order, so a run stopped by the budget cuts evenly across groups. + random.Random(cfg.seed + 1).shuffle(work) + estimate_all = sum(est.count(st) + est.count(json.dumps(q)) for _, _, st, q in work) + if dry_run: + return {"dry_run": True, "runs": len(work), "missing_action_runs": len(failing), "control_runs": len(ok), + "estimated_tokens": estimate_all, "projected_usd_at_jev_rate": cfg.cost.usd(int(estimate_all * 1.33))} + + guard = BudgetGuard(budget_usd, cfg.cost.usd(1)) + rows, requests = [], [] + aborted = "" + with httpx.Client(timeout=120, headers={"Authorization": f"Bearer {key}"}) as client: + for run, domain, state, qs in work: + estimate = est.count(state) + est.count(json.dumps(qs)) + try: + held = guard.reserve(estimate) + except BudgetExceeded as e: + aborted = str(e) + break + billed = None + out: dict[str, Any] = {} + try: + resp = client.post(ENDPOINT, json={"model": cfg.tailwise.model, "state": state, "questions": qs}) + if resp.status_code == 200: + out = resp.json() + billed = int((out.get("usage") or {}).get("input_tokens") or estimate) + else: + out = {"error": f"HTTP {resp.status_code}: {resp.text[:200]}"} + except httpx.TransportError as e: + out = {"error": f"{type(e).__name__}: {e}"} + finally: + guard.settle(held, estimate, billed) + a = out.get("answers") or {} + def noul(q: str) -> float | None: + v = (a.get(q) or {}).get("noul") + return float(v) if v is not None else None + + req = {k: noul(f"requested_{k}") for k in REQUESTS[domain]} + hand = {k: noul(f"handled_{k}") for k in REQUESTS[domain]} + both = {k: (r_, h_) for k in REQUESTS[domain] if (r_ := req[k]) is not None and (h_ := hand[k]) is not None} + unhandled = sorted(k for k, (r_, h_) in both.items() if r_ >= 0.5 and h_ < 0.5) + score = max((min(r_, 1 - h_) for r_, h_ in both.values()), default=None) + gold_missed = sorted(missed.get(run.trace_id, set())) + r = runs[run.trace_id] + rows.append({ + "group": groups[run.trace_id], "run_id": run.run_id, "trace_id": run.trace_id, "domain": domain, + "failed": int(r["failed"]), "gold_consistent_task": r["gold_consistent_task"], + "requested": "; ".join(k for k in REQUESTS[domain] if (req[k] or 0) >= 0.5), + "unhandled": "; ".join(unhandled), "gold_missed": "; ".join(gold_missed), + "flag": int(bool(unhandled)), "score": score, + "right_reason": int(bool(set(unhandled) & set(gold_missed))), + "answer_wrong_first": first.get(run.trace_id, {}).get("answer_answer_wrong", ""), + "answer_wrong_grounded": grounded.get(run.trace_id, {}).get("answer_answer_wrong", ""), + **{f"p_requested_{k}": req[k] for k in REQUESTS[domain]}, + **{f"p_handled_{k}": hand[k] for k in REQUESTS[domain]}, + "tokens": billed, "model": out.get("model", ""), "error": out.get("error", ""), + }) + requests.append({"run_id": run.run_id, "request": {"state": state, "questions": qs}, "response": out}) + write_csv(investigation / "completeness_check.csv", rows) + write_json(investigation / "completeness_check_requests.json", requests) + result = {"runs": len(rows), "planned_runs": len(work), "spent_usd": guard.spent_usd, "tokens": guard.spend.tokens, + "aborted": aborted, "errors": sum(1 for r in rows if r["error"]), + "jev_models": sorted({r["model"] for r in rows if r["model"]})} + write_json(investigation / "completeness_check_summary.json", result) + return result diff --git a/bench/src/tailwise_bench/investigate/experiments.py b/bench/src/tailwise_bench/investigate/experiments.py new file mode 100644 index 0000000..8fb6bd2 --- /dev/null +++ b/bench/src/tailwise_bench/investigate/experiments.py @@ -0,0 +1,90 @@ +"""Tables for the opt-in experiments (narrow and completeness checks), from +their CSVs, against Tailwise's own answer_wrong on the same runs.""" + +from __future__ import annotations + +import csv +from pathlib import Path +from typing import Any + +from .. import metrics + +FLAG_AT = 0.5 + + +def _f(v: str) -> float | None: + return float(v) if v not in ("", None) else None + + +def _rate(rows: list[dict[str, Any]], pred: Any) -> dict[str, Any]: + n = len(rows) + hit = sum(1 for r in rows if pred(r)) + return {"n": n, "flagged": hit, "rate": hit / n if n else None} + + +def narrow_tables(path: Path) -> dict[str, Any]: + with path.open() as f: + rows = [r for r in csv.DictReader(f) if not r["error"]] + out: dict[str, Any] = {} + for g in ("detectable_mixup", "grounded_wrong", "extra", "control"): + rs = [r for r in rows if r["group"] == g] + if not rs: + continue + aw_first = [r for r in rs if _f(r["answer_wrong_first"]) is not None] + aw_grounded = [r for r in rs if _f(r["answer_wrong_grounded"]) is not None] + out[g] = { + "writes": len(rs), "runs": len({r["run_id"] for r in rs}), + "argument_check": _rate(rs, lambda r: r["args_flag"] == "1"), + "policy_check": _rate(rs, lambda r: r["policy_flag"] == "1"), + "either_check": _rate(rs, lambda r: r["args_flag"] == "1" or r["policy_flag"] == "1"), + # Tailwise's own decision on the run the write is in, for comparison. + "answer_wrong_first": _rate(aw_first, lambda r: (_f(r["answer_wrong_first"]) or 0) >= FLAG_AT), + "answer_wrong_grounded": _rate(aw_grounded, lambda r: (_f(r["answer_wrong_grounded"]) or 0) >= FLAG_AT), + } + # Each check's power to tell a wrong write from a correct one (control). + for check, col in (("argument_check", "p_action_wrong_for_request"), ("policy_check", "p_action_violates_policy")): + for g in ("detectable_mixup", "grounded_wrong", "extra"): + rs = [r for r in rows if r["group"] in (g, "control") and _f(r[col]) is not None] + if g in out and rs: + out[g][f"{check}_roc_auc_vs_control"] = metrics.ranking_metrics( + [int(r["group"] == g) for r in rs], [_f(r[col]) or 0.0 for r in rs])["roc_auc"] + return out + + +def completeness_tables(path: Path) -> dict[str, Any]: + with path.open() as f: + rows = [r for r in csv.DictReader(f) if not r["error"] and r["score"] != ""] + out: dict[str, Any] = {} + subsets = { + "missing_action (all tasks)": [r for r in rows if r["group"] == "missing_action"], + "missing_action (gold-consistent tasks)": [r for r in rows if r["group"] == "missing_action" + and r["gold_consistent_task"] == "1"], + "control (successful runs)": [r for r in rows if r["group"] == "control"], + } + for name, rs in subsets.items(): + if not rs: + continue + out[name] = { + "runs": len(rs), + "completeness_check": _rate(rs, lambda r: r["flag"] == "1"), + "flagged_for_the_missing_request": (_rate(rs, lambda r: r["right_reason"] == "1") + if name.startswith("missing") else None), + "answer_wrong_first": _rate([r for r in rs if _f(r["answer_wrong_first"]) is not None], + lambda r: (_f(r["answer_wrong_first"]) or 0) >= FLAG_AT), + "answer_wrong_grounded": _rate([r for r in rs if _f(r["answer_wrong_grounded"]) is not None], + lambda r: (_f(r["answer_wrong_grounded"]) or 0) >= FLAG_AT), + "either_completeness_or_answer_wrong_grounded": _rate( + rs, lambda r: r["flag"] == "1" or (_f(r["answer_wrong_grounded"]) or 0) >= FLAG_AT), + } + pos = subsets["missing_action (all tasks)"] + neg = subsets["control (successful runs)"] + if pos and neg: + y = [1] * len(pos) + [0] * len(neg) + out["roc_auc_missing_vs_control"] = { + "completeness_score": metrics.ranking_metrics(y, [_f(r["score"]) or 0.0 for r in pos + neg])["roc_auc"], + "answer_wrong_first": metrics.ranking_metrics( + y, [_f(r["answer_wrong_first"]) or 0.0 for r in pos + neg])["roc_auc"], + "answer_wrong_grounded": metrics.ranking_metrics( + y, [_f(r["answer_wrong_grounded"]) or 0.0 for r in pos + neg])["roc_auc"], + } + return out diff --git a/bench/src/tailwise_bench/investigate/narrow.py b/bench/src/tailwise_bench/investigate/narrow.py new file mode 100644 index 0000000..0224088 --- /dev/null +++ b/bench/src/tailwise_bench/investigate/narrow.py @@ -0,0 +1,321 @@ +"""Step 7, opt-in: can Jev catch a wrong action when shown only what matters? + +For each write call checked, a short state holds: +- the user's messages before the call; +- the earlier lookup results that contain the call's IDs, each trimmed to + the objects holding them; +- the call itself, and the agent's later actions in the run (so a request + handled in a later step is not mistaken for one left out); +- the one policy section that governs that kind of action. + +Jev is asked whether the call's arguments match what the user asked for, and +whether the policy allows it, directly over /v1/systemone (not through +Tailwise), under a hard budget. +""" + +from __future__ import annotations + +import csv +import json +import os +import random +import re +from pathlib import Path +from typing import Any + +import httpx + +from ..budget import BudgetExceeded, BudgetGuard +from ..config import Config +from ..loaders import tau +from ..report import write_csv, write_json +from ..tokens import Estimator +from . import audit +from .run import load_tau + +ENDPOINT = "https://api.typesafe.ai/v1/systemone" +MAX_LOOKUP_CHARS = 2500 +MAX_LOOKUPS = 3 + +ONE_OF_SEVERAL = (" Judge this ACTION only: the user's other requests may be handled by the LATER ACTIONS, so a " + "request this ACTION does not cover is not a problem with it.") +QUESTIONS = { + "action_matches_request": { + "type": "noul", + "instructions": "Does every argument of the ACTION (each item, product option, order, reservation, flight, " + "amount, address and payment method) match what the user asked for, as identified in the " + "LOOKUP RESULTS?" + ONE_OF_SEVERAL, + }, + "action_wrong_for_request": { + "type": "noul", + "instructions": "Does the ACTION use any argument that is not what the user asked for: a different item or " + "product option, order, reservation, flight, amount, address or payment method than the user " + "requested, according to the USER MESSAGES and the LOOKUP RESULTS?" + ONE_OF_SEVERAL, + }, + "action_allowed_by_policy": { + "type": "noul", + "instructions": "Does the POLICY allow the ACTION, given the user's situation shown in the USER MESSAGES and " + "the LOOKUP RESULTS (for example the order or reservation status, cabin, insurance, dates and " + "the reason the user gave)?", + }, + "action_violates_policy": { + "type": "noul", + "instructions": "Does the ACTION break a rule in the POLICY, given the user's situation shown in the USER " + "MESSAGES and the LOOKUP RESULTS (for example acting on an order or reservation whose status, " + "cabin, insurance, dates or the user's reason the policy does not allow)?", + }, +} + +# The policy section that governs each kind of action: (section, subsection or None). +POLICY_SECTIONS = { + "cancel_pending_order": ("Cancel pending order", None), + "modify_pending_order_address": ("Modify pending order", ""), + "modify_pending_order_payment": ("Modify pending order", "Modify payment"), + "modify_pending_order_items": ("Modify pending order", "Modify items"), + "return_delivered_order_items": ("Return delivered order", None), + "exchange_delivered_order_items": ("Exchange delivered order", None), + "modify_user_address": ("Domain basic", None), + "book_reservation": ("Book flight", None), + "update_reservation_flights": ("Modify flight", None), + "update_reservation_baggages": ("Modify flight", None), + "update_reservation_passengers": ("Modify flight", None), + "cancel_reservation": ("Cancel flight", None), + "send_certificate": ("Refund", None), +} + + +def policy_section(policy: str, action: str) -> str: + """The section of the policy that governs action: a `## ` section, or its + introduction plus one `### ` subsection ("" for the introduction alone).""" + want = POLICY_SECTIONS.get(action) + if not want: + return "" + sections: dict[str, str] = {} + current = None + for line in policy.splitlines(): + if line.startswith("## "): + current = line[3:].strip().lower() + sections[current] = "" + if current is not None: + sections[current] += line + "\n" + text = sections.get(want[0].lower(), "") + if want[1] is None: + return text.strip() + intro, *subs = re.split(r"(?m)^(?=### )", text) + chosen = [s for s in subs if s[4:].strip().lower().startswith(want[1].lower())] if want[1] else [] + return (intro + "".join(chosen)).strip() + + +def _ids(args: dict[str, Any]) -> list[str]: + """The id-like values of a call's arguments.""" + return [str(v).strip().lower().lstrip("#") for p, v in audit.leaves(args) if audit.kind_of(p, v) == "id"] + + +def _trim(output: str, ids: list[str]) -> str: + """The output whole if short, else the JSON objects holding one of ids + (and the top-level scalars), else its start.""" + if len(output) <= MAX_LOOKUP_CHARS: + return output + try: + obj = json.loads(output) + except json.JSONDecodeError: + return output[:MAX_LOOKUP_CHARS] + " [trimmed]" + + def holds(v: Any) -> bool: + return isinstance(v, str | int | float) and str(v).strip().lower().lstrip("#") in ids + + def keep(v: Any, top: bool) -> tuple[Any, bool]: + if isinstance(v, dict): + if any(holds(c) for c in v.values()): + return v, True + kept, found = {}, False + for k, c in v.items(): + sub, ok = keep(c, False) + if ok or holds(k): + kept[k], found = c if holds(k) else sub, True + elif top and not isinstance(c, dict | list): + kept[k] = c + return kept, found + if isinstance(v, list): + items = [s for s, ok in (keep(c, False) for c in v) if ok] + return items, bool(items) + return v, holds(v) + + kept, found = keep(obj, True) + text = json.dumps(kept, ensure_ascii=False) if found else output + return text[:MAX_LOOKUP_CHARS] + " [trimmed]" + + +def slices(traj: list[dict[str, Any]], system_prompt: str, only: set[int] | None = None) -> list[dict[str, Any]]: + """One narrow state per write call (or per call index in only).""" + paired = tau.pair_results(traj) + calls: list[tuple[int, int, str, dict[str, Any]]] = [] # (call index, message index, name, args) + n = 0 + for mi, m in enumerate(traj): + for tc in (m.get("tool_calls") or []) if m.get("role") == "assistant" else []: + fn = tc.get("function") or {} + args: Any = fn.get("arguments") or {} + if isinstance(args, str): + try: + args = json.loads(args or "{}") + except json.JSONDecodeError: + args = {} + calls.append((n, mi, fn.get("name") or "", args if isinstance(args, dict) else {})) + n += 1 + def status_of(i: int) -> str: + return "error" if i in paired and str(paired[i][0].get("content") or "").startswith("Error") else "ok" + + out = [] + for ci, mi, name, args in calls: + if not audit.is_write(name) or (only is not None and ci not in only): + continue + ids = _ids(args) + user = [m.get("content") or "" for m in traj[:mi] if m.get("role") == "user"] + lookups = [] + for cj, mj, nj, aj in calls: + if cj >= ci or cj not in paired or audit.is_write(nj): + continue + result = str(paired[cj][0].get("content") or "") + low = result.lower() + if any(re.search(r"(? {status_of(cj)}" + for cj, _, nj, aj in calls if cj > ci and (audit.is_write(nj) or nj == "transfer_to_human_agents")] + state = ("USER MESSAGES (before the action):\n" + "\n\n".join(u.replace(tau.STOP, "").strip() for u in user if u.strip()) + + "\n\nLOOKUP RESULTS THE ACTION'S IDS CAME FROM:\n" + + ("\n".join(f"[{k + 1}] {nj}({json.dumps(aj, ensure_ascii=False)}) -> {_trim(r, ids)}" + for k, (_, nj, aj, r) in enumerate(lookups)) or "(none)") + + f"\n\nACTION:\n{name}({json.dumps(args, ensure_ascii=False)}) -> {status_of(ci)}" + + "\n\nLATER ACTIONS IN THIS RUN:\n" + ("\n".join(later) or "(none)") + + "\n\nPOLICY (the section for this kind of action):\n" + (policy_section(system_prompt, name) or "(none)")) + out.append({"call_index": ci, "tool": name, "state": state, "lookups": [c for c, *_ in lookups]}) + return out + + +GROUPS = ("detectable_mixup", "grounded_wrong", "extra", "control") + + +def targets(investigation: Path, groups: list[str] | None, per_group: int, seed: int) -> list[tuple[str, int, str]]: + """(trace_id, call index, group) to check. No groups: every write in the + failed-but-rated-fine sample (the first, small run).""" + with (investigation / "wrong_args.csv").open() as f: + gold = [r for r in csv.DictReader(f) if r["call_index"]] + with (investigation / "grounding.csv").open() as f: + writes = list(csv.DictReader(f)) + with (investigation / "tau_runs.csv").open() as f: + runs = {r["trace_id"]: r for r in csv.DictReader(f)} + if not groups: + sample = set(json.loads((investigation / "summary.json").read_text())["step6_samples"]["failed_rated_fine"]) + return [(w["trace_id"], int(w["call_index"]), "sample") for w in writes if w["run_id"] in sample] + consistent = {t for t, r in runs.items() if r["gold_consistent_task"] == "1"} + pools: dict[str, list[tuple[str, int]]] = { + # every failed run whose wrong item the user's own words could have shown + "detectable_mixup": [(g["trace_id"], int(g["call_index"])) for g in gold if g["failed"] == "1" + and g["variant_mixup"] == "True" and g["mixup_detectable_from_trace"] == "True"], + "grounded_wrong": [(g["trace_id"], int(g["call_index"])) for g in gold if g["failed"] == "1" + and g["category"] == "wrong_args" and g["grounded_wrong"] == "True" + and g["variant_mixup"] != "True" and g["trace_id"] in consistent], + "extra": [(g["trace_id"], int(g["call_index"])) for g in gold if g["failed"] == "1" + and g["category"] == "extra" and g["trace_id"] in consistent], + # writes of successful runs on tasks whose gold every success carried out: all correct + "control": [(w["trace_id"], int(w["call_index"])) for w in writes if w["failed"] == "0" + and w["trace_id"] in consistent], + } + rng = random.Random(seed) + out = [] + for g in groups: + pool = sorted(set(pools[g])) + chosen = pool if g == "detectable_mixup" or len(pool) <= per_group else rng.sample(pool, per_group) + out += [(t, c, g) for t, c in sorted(chosen)] + return out + + +def narrow_check(cfg: Config, investigation: Path, baseline: Path, grounded_run: Path | None, budget_usd: float, + groups: list[str] | None = None, per_group: int = 75, dry_run: bool = False, + name: str = "narrow_check") -> dict[str, Any]: + key = os.environ.get("JEV_API_KEY") + if not key and not dry_run: + raise SystemExit("JEV_API_KEY is not set: run under `envchain observer-ai ...`") + todo = targets(investigation, groups, per_group, cfg.seed) + by_trace: dict[str, dict[int, str]] = {} + for t, c, g in todo: + by_trace.setdefault(t, {})[c] = g + with (investigation / "wrong_args.csv").open() as f: + gold = {(r["trace_id"], r["call_index"]): r for r in csv.DictReader(f) if r["call_index"]} + + def answers(folder: Path | None) -> dict[str, dict[str, str]]: + if folder is None: + return {} + with (folder / "runs.csv").open() as f: + return {r["run_id"]: r for r in csv.DictReader(f)} + + first, grounded = answers(baseline), answers(grounded_run) + est = Estimator(cfg.cost.tokenizer) + work = [(run, s, by_trace[run.trace_id][s["call_index"]]) + for run, row in load_tau(cfg) if run.trace_id in by_trace + for s in slices(row["traj"], run.system_prompt, set(by_trace[run.trace_id]))] + # A seeded order, so a run stopped by the budget cuts evenly across groups. + random.Random(cfg.seed + 1).shuffle(work) + estimate_all = sum(est.count(s["state"]) + est.count(json.dumps(QUESTIONS)) for _, s, _ in work) + if dry_run: + groups_n = {g: sum(1 for *_, gg in work if gg == g) for g in {gg for *_, gg in work}} + return {"dry_run": True, "writes": len(work), "by_group": groups_n, "estimated_tokens": estimate_all, + "projected_usd_at_jev_rate": cfg.cost.usd(int(estimate_all * 1.33))} + guard = BudgetGuard(budget_usd, cfg.cost.usd(1)) + rows, requests = [], [] + aborted = "" + with httpx.Client(timeout=120, headers={"Authorization": f"Bearer {key}"}) as client: + for run, s, group in work: + body = {"model": cfg.tailwise.model, "state": s["state"], "questions": QUESTIONS} + estimate = est.count(s["state"]) + est.count(json.dumps(QUESTIONS)) + try: + held = guard.reserve(estimate) + except BudgetExceeded as e: + aborted = str(e) + break + billed = None + out: dict[str, Any] = {} + try: + resp = client.post(ENDPOINT, json=body) + if resp.status_code == 200: + out = resp.json() + billed = int((out.get("usage") or {}).get("input_tokens") or estimate) + else: + out = {"error": f"HTTP {resp.status_code}: {resp.text[:200]}"} + except httpx.TransportError as e: + out = {"error": f"{type(e).__name__}: {e}"} + finally: + guard.settle(held, estimate, billed) + a = out.get("answers") or {} + + def noul(q: str) -> float | None: + v = (a.get(q) or {}).get("noul") + return float(v) if v is not None else None + + p_wrong, p_violates = noul("action_wrong_for_request"), noul("action_violates_policy") + gold_row = gold.get((run.trace_id, str(s["call_index"])), {}) + rows.append({ + "group": group, "run_id": run.run_id, "trace_id": run.trace_id, "call_index": s["call_index"], + "tool": s["tool"], "gold": gold_row.get("category", ""), "gold_diff": gold_row.get("diff", ""), + "variant_mixup": gold_row.get("variant_mixup", ""), + "mixup_detectable_from_trace": gold_row.get("mixup_detectable_from_trace", ""), + "lookups_shown": len(s["lookups"]), + "p_action_matches_request": noul("action_matches_request"), "p_action_wrong_for_request": p_wrong, + "p_action_allowed_by_policy": noul("action_allowed_by_policy"), "p_action_violates_policy": p_violates, + "args_flag": int(p_wrong is not None and p_wrong >= 0.5), + "policy_flag": int(p_violates is not None and p_violates >= 0.5), + "answer_wrong_first": first.get(run.run_id, {}).get("answer_answer_wrong", ""), + "answer_wrong_grounded": grounded.get(run.run_id, {}).get("answer_answer_wrong", ""), + "tokens": billed, "model": out.get("model", ""), "error": out.get("error", ""), + }) + requests.append({"run_id": run.run_id, "call_index": s["call_index"], "group": group, + "request": body, "response": out}) + write_csv(investigation / f"{name}.csv", rows) + write_json(investigation / f"{name}_requests.json", requests) + result = {"writes": len(rows), "runs": len({r["run_id"] for r in rows}), "planned_writes": len(work), + "spent_usd": guard.spent_usd, "tokens": guard.spend.tokens, "aborted": aborted, + "jev_models": sorted({r["model"] for r in rows if r["model"]}), + "errors": sum(1 for r in rows if r["error"])} + write_json(investigation / f"{name}_summary.json", result) + return result diff --git a/bench/src/tailwise_bench/investigate/run.py b/bench/src/tailwise_bench/investigate/run.py new file mode 100644 index 0000000..f21a0b0 --- /dev/null +++ b/bench/src/tailwise_bench/investigate/run.py @@ -0,0 +1,442 @@ +"""Runs the offline investigation (steps 1 to 6) and writes its folder. + +Inputs: the τ-bench and SWE-agent runs the benchmark scores, the scores of an +earlier benchmark run (Jev's answers for every run, system prompt off), and +optionally a run with the system prompt on (for its token counts). +""" + +from __future__ import annotations + +import csv +import json +import math +import random +import statistics as st +from collections import Counter +from collections.abc import Callable, Iterable +from pathlib import Path +from typing import Any + +from ..config import Config +from ..loaders import swe, tau +from ..otel import hex_id +from ..report import write_csv, write_json +from ..tokens import Estimator +from ..types import Run +from . import audit +from .capture import capture + +FLAG_AT = 0.5 +QUESTIONS = ["answer_wrong", "unsupported_answer", "missing_tool_calls", "answered_on_failure", + "ignored_instructions", "contradicts_knowledge"] + + +def read_runs_csv(path: Path) -> dict[str, dict[str, str]]: + with path.open() as f: + return {r["run_id"]: r for r in csv.DictReader(f)} + + +def fnum(v: str | None) -> float | None: + return float(v) if v not in (None, "") else None + + +def load_tau(cfg: Config) -> list[tuple[Run, dict[str, Any]]]: + d = cfg.datasets.tau + out = [] + for file in d["files"]: + rows = json.loads(tau.download(d["commit"], file, cfg.resolve(cfg.cache_dir)).read_text()) + out += [(tau.parse(file, row), row) for row in rows] + return out + + +def p_value(a_hit: int, a_n: int, b_hit: int, b_n: int) -> float | None: + """Two-sided two-proportion z-test.""" + if not a_n or not b_n: + return None + p = (a_hit + b_hit) / (a_n + b_n) + se = math.sqrt(p * (1 - p) * (1 / a_n + 1 / b_n)) + if se == 0: + return None + z = (a_hit / a_n - b_hit / b_n) / se + return math.erfc(abs(z) / math.sqrt(2)) + + +def share(rows: Iterable[dict[str, Any]], pred: Callable[[dict[str, Any]], bool]) -> tuple[int, int]: + rows = list(rows) + return sum(1 for r in rows if pred(r)), len(rows) + + +def pct(hit: int, n: int) -> str: + return f"{hit / n:.0%} ({hit}/{n})" if n else "n/a" + + +def investigate(cfg: Config, baseline: Path, prompt_run: Path | None, out: Path, + binary: Path, answer_wrong_manifest: Path) -> dict[str, Any]: + est = Estimator(cfg.cost.tokenizer) + base = read_runs_csv(baseline / "runs.csv") + on_run = read_runs_csv(prompt_run / "runs.csv") if prompt_run else {} + tau_runs = load_tau(cfg) + runs = [r for r, _ in tau_runs] + d = cfg.datasets.swe + swe_runs, _ = swe.load(d["repo"], int(d["shards"]), int(d["shards_total"]), int(d["sample"]), cfg.seed, + cfg.resolve(cfg.cache_dir)) + work = out / "work" + work.mkdir(parents=True, exist_ok=True) + aw_manifest = str(answer_wrong_manifest.resolve()) + + # Step 1: the exact requests, per setting. first/off is what the baseline sent. + variants: dict[str, tuple[list[Run], dict[str, Any]]] = { + "tau_first_off": (runs, {}), + "tau_first_on": (runs, {"system_prompt": True}), + "tau_grounded_off": (runs, {"evidence": "grounded", "manifest": aw_manifest}), + "swe_first_off": (swe_runs, {}), + "swe_grounded_off": (swe_runs, {"evidence": "grounded", "manifest": aw_manifest}), + } + captured: dict[str, dict[str, dict[str, Any]]] = {} + for i, (name, (rs, scorer)) in enumerate(variants.items()): + captured[name] = capture(rs, binary, work, scorer, port=18190 + 2 * i, model_port=18191 + 2 * i) + + def req_tokens(req: dict[str, Any]) -> int: + return est.count(req["state"]) + est.count(json.dumps(req["questions"])) + + def per_run(name: str, rs: list[Run]) -> list[dict[str, Any]]: + rows = [] + for r in rs: + req = captured[name][r.trace_id] + if "error" in req: + rows.append({"trace_id": r.trace_id, "run_id": r.run_id, "error": req["error"]}) + continue + vis = audit.visibility(req["state"], [c.output for c in r.tool_calls]) + with_out = [v for v, c in zip(vis, r.tool_calls, strict=True) if c.output.strip()] + rows.append({ + "trace_id": r.trace_id, "run_id": r.run_id, "dataset": r.dataset, "failed": int(r.failed), + "tool_calls": len(r.tool_calls), "outputs": len(with_out), + "full": with_out.count("full"), "partial": with_out.count("partial"), "none": with_out.count("none"), + "visible_share": (len(with_out) - with_out.count("none")) / len(with_out) if with_out else 1.0, + "state_chars": len(req["state"]), "est_tokens": req_tokens(req), + "trimmed": req["state"].count(" [trimmed]"), + "system_prompt_cut": int(len(r.system_prompt) > 8000 and "SYSTEM PROMPT:" in str(req["state"])), + "vis": vis, + }) + return rows + + vis_rows = {name: per_run(name, rs) for name, (rs, _) in variants.items()} + + # Steps 2 to 5 on τ-bench. + grounding_rows: list[dict[str, Any]] = [] + gold_rows: list[dict[str, Any]] = [] + run_rows: list[dict[str, Any]] = [] + vis_first = {r["trace_id"]: r["vis"] for r in vis_rows["tau_first_off"] if "vis" in r} + vis_grounded = {r["trace_id"]: r["vis"] for r in vis_rows["tau_grounded_off"] if "vis" in r} + for r, row in tau_runs: + b = base.get(r.run_id, {}) + aw = fnum(b.get("answer_answer_wrong")) + writes = audit.ground(row["traj"], r.system_prompt) + gold = audit.compare_gold(writes, ((row.get("info") or {}).get("task") or {}).get("actions") or [], row["traj"]) + answers = {q: fnum(b.get(f"answer_{q}")) for q in QUESTIONS} + flagged = aw is not None and aw >= FLAG_AT + vf, vg = vis_first.get(r.trace_id, []), vis_grounded.get(r.trace_id, []) + n_ungrounded = 0 + for w in writes: + srcs = sorted({lf.tool_index for lf in w.leaves if lf.tool_index is not None}) + ung = w.ungrounded + n_ungrounded += bool(ung) + grounding_rows.append({ + "run_id": r.run_id, "trace_id": r.trace_id, "subset": r.subset, "failed": int(r.failed), + "answer_wrong": aw, "flagged": int(flagged), + "call_index": w.call_index, "tool": w.name, + "tool_span_id": hex_id("tool", r.trace_id, str(w.call_index), size=8), + "leaves_checked": sum(1 for lf in w.leaves if lf.kind in audit.CHECKED_KINDS), + "ungrounded": int(bool(ung)), + "ungrounded_leaves": "; ".join(f"{lf.path}={lf.value}" for lf in ung), + "derived_values": "; ".join(f"{lf.path}={lf.value}" for lf in w.derived), + "sources": "; ".join(f"{lf.path}<-{lf.source}" for lf in w.leaves if lf.source), + "grounding_results": " ".join(str(s) for s in srcs), + "calls_between": w.calls_between, "chars_between": w.chars_between, + "grounding_visible_first": (int(all(vf[s] != "none" for s in srcs)) if srcs and vf else None), + "grounding_visible_grounded": (int(all(vg[s] != "none" for s in srcs)) if srcs and vg else None), + "acted_without_lookup": int(w.acted_without_lookup), + }) + for g in gold.rows: + gold_rows.append({"run_id": r.run_id, "trace_id": r.trace_id, "subset": r.subset, "failed": int(r.failed), + "answer_wrong": aw, "flagged": int(flagged), **g, + "tool_span_id": (hex_id("tool", r.trace_id, str(g["call_index"]), size=8) + if g["call_index"] is not None else "")}) + vis_share = (sum(1 for v, c in zip(vf, r.tool_calls, strict=False) if c.output.strip() and v != "none") + / max(1, sum(1 for c in r.tool_calls if c.output.strip()))) if vf else None + srcs_all = {lf.tool_index for w in writes for lf in w.leaves if lf.tool_index is not None} + run_rows.append({ + "task": f"{r.meta['domain']}:{r.meta['task_id']}", + "visible_share_first": vis_share, + "grounding_visible_first": (int(all(vf[s] != "none" for s in srcs_all)) if srcs_all and vf else None), + "gold_rows_wrong": sum(1 for g in gold.rows if g["category"] != "correct"), + "run_id": r.run_id, "trace_id": r.trace_id, "subset": r.subset, "failed": int(r.failed), + "answer_wrong": aw, "flagged": int(flagged), "flagged_six_any_of": int(b.get("kept") == "1"), + "cause": b.get("cause", ""), "writes": len(writes), "ungrounded_writes": n_ungrounded, + "acted_without_lookup": int(any(g["acted_without_lookup"] for g in grounding_rows + if g["trace_id"] == r.trace_id)), + "wrong_args": int(gold.has("wrong_args")), + "grounded_wrong": int(any(g["grounded_wrong"] for g in gold.rows if g["category"] == "wrong_args")), + "variant_mixup": int(any(g["variant_mixup"] for g in gold.rows)), + "mixup_detectable": next((int(g["mixup_detectable_from_trace"]) for g in gold.rows + if g["variant_mixup"] and g.get("mixup_detectable_from_trace") is not None), None), + "extra_write": int(gold.has("extra")), "missed_write": int(gold.has("missed")), + "tool_calls": len(r.tool_calls), "tokens_off": fnum(b.get("tokens")), + "tokens_on": fnum(on_run.get(r.run_id, {}).get("tokens")), + **{f"fires_{q}": (int(v >= FLAG_AT) if v is not None else None) for q, v in answers.items()}, + }) + + # Tasks whose gold actions every successful run carried out exactly: on the + # others the logged gold disagrees with how runs were scored. + bad_tasks = {r["task"] for r in run_rows if not r["failed"] and r["gold_rows_wrong"]} + consistent = {row_["trace_id"]: int(row_["task"] not in bad_tasks) for row_ in run_rows} + for row_ in run_rows: + row_["gold_consistent_task"] = consistent[row_["trace_id"]] + for g in gold_rows: + g["gold_consistent_task"] = consistent[g["trace_id"]] + + write_csv(out / "grounding.csv", grounding_rows) + write_csv(out / "wrong_args.csv", gold_rows) + write_csv(out / "tau_runs.csv", run_rows) + for name, rows in vis_rows.items(): + write_csv(work / f"visibility_{name}.csv", [{k: v for k, v in r.items() if k != "vis"} for r in rows]) + + summary = { + "step1_visibility": {name: visibility_summary(rows) for name, rows in vis_rows.items()}, + "step2_grounding": grounding_tables(run_rows, grounding_rows), + "step3_wrong_args": { + "all_tasks": gold_tables(run_rows), + "gold_consistent_tasks": gold_tables([r for r in run_rows if r["gold_consistent_task"]]), + "tasks": len({r["task"] for r in run_rows}), + "gold_consistent_task_count": len({r["task"] for r in run_rows if r["gold_consistent_task"]}), + }, + "step4_audit": audit_tables(run_rows), + "step5_distance": distance_tables(grounding_rows, run_rows), + "step2_counts": {"runs": len(run_rows), "write_calls": len(grounding_rows), + "write_calls_ungrounded": sum(g["ungrounded"] for g in grounding_rows)}, + "h2_tokens": token_summary(run_rows, vis_rows), + } + summary["step6_samples"] = samples(run_rows, captured["tau_first_off"], base, tau_runs, grounding_rows, gold_rows, + out / "samples", cfg.seed) + write_json(out / "summary.json", summary) + return summary + + +def visibility_summary(rows: list[dict[str, Any]]) -> dict[str, Any]: + ok = [r for r in rows if "vis" in r] + outs = sum(r["outputs"] for r in ok) + return { + "runs": len(rows), "errors": len(rows) - len(ok), + "tool_outputs": outs, + "outputs_full": sum(r["full"] for r in ok), "outputs_partial": sum(r["partial"] for r in ok), + "outputs_none": sum(r["none"] for r in ok), + "share_reaching_model": (outs - sum(r["none"] for r in ok)) / outs if outs else None, + "per_run_visible_share_median": st.median(r["visible_share"] for r in ok) if ok else None, + "runs_seeing_every_output": sum(1 for r in ok if r["none"] == 0), + "est_tokens_median": st.median(r["est_tokens"] for r in ok) if ok else None, + "est_tokens_max": max((r["est_tokens"] for r in ok), default=None), + "state_chars_median": st.median(r["state_chars"] for r in ok) if ok else None, + "results_trimmed": sum(r["trimmed"] for r in ok), + "system_prompt_cut": sum(r["system_prompt_cut"] for r in ok), + } + + +def token_summary(run_rows: list[dict[str, Any]], vis_rows: dict[str, list[dict[str, Any]]]) -> dict[str, Any]: + off = [r["tokens_off"] for r in run_rows if r["tokens_off"]] + on = [r["tokens_on"] for r in run_rows if r["tokens_on"]] + est_off = {r["trace_id"]: r["est_tokens"] for r in vis_rows["tau_first_off"] if "est_tokens" in r} + ratio = [r["tokens_off"] / est_off[r["trace_id"]] for r in run_rows + if r["tokens_off"] and est_off.get(r["trace_id"])] + calib = st.median(ratio) if ratio else None + g = [r["est_tokens"] for r in vis_rows["tau_grounded_off"] if "est_tokens" in r] + sg = [r["est_tokens"] for r in vis_rows["swe_grounded_off"] if "est_tokens" in r] + sf = [r["est_tokens"] for r in vis_rows["swe_first_off"] if "est_tokens" in r] + return { + "jev_tokens_system_prompt_off": {"median": st.median(off), "max": max(off), "runs": len(off)} if off else None, + "jev_tokens_system_prompt_on": {"median": st.median(on), "max": max(on), "runs": len(on)} if on else None, + "jev_context_limit_tokens": 64000, + "runs_over_32k_off": sum(1 for t in off if t > 32000), "runs_over_32k_on": sum(1 for t in on if t > 32000), + "estimate_to_jev_ratio_median": calib, + "tau_grounded_est_tokens_median": st.median(g) if g else None, + "tau_grounded_jev_tokens_projected_median": st.median(g) * calib if g and calib else None, + "tau_grounded_projected_cost_usd": sum(g) * (calib or 1) * 0.042 / 1e6 if g else None, + "swe_first_est_tokens_median": st.median(sf) if sf else None, + "swe_grounded_est_tokens_median": st.median(sg) if sg else None, + "swe_grounded_projected_cost_usd": sum(sg) * (calib or 1) * 0.042 / 1e6 if sg else None, + } + + +def grounding_tables(run_rows: list[dict[str, Any]], grounding_rows: list[dict[str, Any]]) -> dict[str, Any]: + out: dict[str, Any] = {"by_run": {}, "by_write": {}} + for outcome in ("failed", "succeeded"): + for ung in (True, False): + rs = [r for r in run_rows if r["writes"] and bool(r["failed"]) == (outcome == "failed") + and (r["ungrounded_writes"] > 0) == ung] + hit, n = share(rs, lambda r: r["flagged"] == 1) + out["by_run"][f"{outcome}, {'≥1 ungrounded write' if ung else 'all writes grounded'}"] = \ + {"runs": n, "flagged": hit, "flag_rate": hit / n if n else None} + rs = [r for r in run_rows if not r["writes"]] + hit, n = share(rs, lambda r: r["flagged"] == 1) + out["by_run"]["no write calls"] = {"runs": n, "flagged": hit, "flag_rate": hit / n if n else None, + "failed": sum(r["failed"] for r in rs)} + kinds = Counter(lf.split("=")[0].split(".")[-1].split("[")[0] for g in grounding_rows + for lf in g["ungrounded_leaves"].split("; ") if lf) + out["ungrounded_argument_keys"] = dict(kinds.most_common(15)) + for tool in sorted({g["tool"] for g in grounding_rows}): + gs = [g for g in grounding_rows if g["tool"] == tool] + out["by_write"][tool] = {"calls": len(gs), "ungrounded": sum(g["ungrounded"] for g in gs)} + ung_runs = [r for r in run_rows if r["ungrounded_writes"]] + out["ungrounded_runs_outcome"] = {"runs": len(ung_runs), "failed": sum(r["failed"] for r in ung_runs)} + return out + + +def gold_tables(run_rows: list[dict[str, Any]]) -> dict[str, Any]: + failed = [r for r in run_rows if r["failed"]] + cats: list[tuple[str, Callable[[dict[str, Any]], bool]]] = [ + ("variant mix-up (grounded wrong item)", lambda r: r["variant_mixup"] == 1), + ("other grounded-but-wrong arguments", lambda r: r["grounded_wrong"] == 1 and r["variant_mixup"] == 0), + ("wrong arguments, not grounded", lambda r: r["wrong_args"] == 1 and r["grounded_wrong"] == 0), + ("extra write (not in gold)", lambda r: r["wrong_args"] == 0 and r["extra_write"] == 1), + ("missed write only", lambda r: r["wrong_args"] == 0 and r["extra_write"] == 0 and r["missed_write"] == 1), + ("writes match gold (failed on something else)", lambda r: not (r["wrong_args"] or r["extra_write"] + or r["missed_write"])), + ] + out: dict[str, Any] = {} + for label, pred in cats: + rs = [r for r in failed if pred(r)] + hit, n = share(rs, lambda r: r["flagged"] == 1) + out[label] = {"failed_runs": n, "flagged": hit, "flag_rate": hit / n if n else None} + mix = [r for r in failed if r["variant_mixup"]] + for label, val in (("variant mix-up, user stated the distinguishing option", 1), + ("variant mix-up, user never stated it", 0)): + rs = [r for r in mix if r["mixup_detectable"] == val] + hit, n = share(rs, lambda r: r["flagged"] == 1) + out[label] = {"failed_runs": n, "flagged": hit, "flag_rate": hit / n if n else None} + out["variant mix-up, could not tell"] = {"failed_runs": sum(1 for r in mix if r["mixup_detectable"] is None)} + succ = [r for r in run_rows if not r["failed"]] + out["_succeeded_with_write_discrepancy"] = sum(1 for r in succ if r["wrong_args"] or r["extra_write"] + or r["missed_write"]) + return out + + +def audit_tables(run_rows: list[dict[str, Any]]) -> dict[str, Any]: + succ = [r for r in run_rows if not r["failed"]] + problem: Callable[[dict[str, Any]], bool] = lambda r: r["ungrounded_writes"] > 0 or r["acted_without_lookup"] == 1 + out: dict[str, Any] = {} + for label, rs in (("flagged successes", [r for r in succ if r["flagged"]]), + ("unflagged successes", [r for r in succ if not r["flagged"]])): + p, n = share(rs, problem) + u, _ = share(rs, lambda r: r["ungrounded_writes"] > 0) + a, _ = share(rs, lambda r: r["acted_without_lookup"] == 1) + out[label] = {"runs": n, "process_problem": p, "share": p / n if n else None, + "ungrounded_write": u, "acted_without_lookup": a} + checks = {} + for q in QUESTIONS: + row: dict[str, Any] = {} + for outcome in ("succeeded", "failed"): + rs = [r for r in run_rows if r["writes"] and bool(r["failed"]) == (outcome == "failed") + and r[f"fires_{q}"] is not None] + ug = [r for r in rs if r["ungrounded_writes"] > 0] + gr = [r for r in rs if r["ungrounded_writes"] == 0] + a, an = share(ug, lambda r: r[f"fires_{q}"] == 1) + b, bn = share(gr, lambda r: r[f"fires_{q}"] == 1) + row[outcome] = {"fires_on_ungrounded": a / an if an else None, "n_ungrounded": an, + "fires_on_grounded": b / bn if bn else None, "n_grounded": bn, + "difference_pts": (a / an - b / bn) * 100 if an and bn else None, + "p_value": p_value(a, an, b, bn)} + s = row["succeeded"] + diff, p = s["difference_pts"], s["p_value"] + if s["n_ungrounded"] < 20: + row["verdict"] = f"too few ungrounded successful runs to tell (n={s['n_ungrounded']})" + else: + row["verdict"] = ("process signal" if diff is not None and diff >= 10 and p is not None and p < 0.01 + else "noise (no link to ungrounded actions)") + checks[q] = row + out["checks_vs_grounding"] = checks + # Does a check fire because the model cannot see what the agent's actions + # rest on? Successful runs only, so the outcome is held fixed. + vis: dict[str, Any] = {} + preds: list[tuple[str, Callable[[dict[str, Any]], bool]]] + for q in QUESTIONS: + row = {} + preds = [("grounding visible", lambda r: r["grounding_visible_first"] == 1), + ("grounding not visible", lambda r: r["grounding_visible_first"] == 0), + ("≥50% of results visible", lambda r: (r["visible_share_first"] or 0) >= 0.5), + ("<25% of results visible", lambda r: (r["visible_share_first"] or 0) < 0.25)] + for label, pred in preds: + rs = [r for r in succ if r[f"fires_{q}"] is not None and pred(r)] + hit, n = share(rs, lambda r: r[f"fires_{q}"] == 1) + row[label] = {"runs": n, "fires": hit / n if n else None} + a, b = row["grounding visible"], row["grounding not visible"] + row["p_value_visible_vs_not"] = p_value(round((a["fires"] or 0) * a["runs"]), a["runs"], + round((b["fires"] or 0) * b["runs"]), b["runs"]) + vis[q] = row + out["checks_vs_visibility_successes"] = vis + return out + + +def distance_tables(grounding_rows: list[dict[str, Any]], run_rows: list[dict[str, Any]]) -> dict[str, Any]: + by_run = {r["trace_id"]: r for r in run_rows} + failed_writes = [g for g in grounding_rows if g["failed"] and g["calls_between"] is not None] + + def miss(gs: list[dict[str, Any]]) -> dict[str, Any]: + runs = {g["trace_id"] for g in gs} + m = sum(1 for t in runs if not by_run[t]["flagged"]) + return {"failed_runs": len(runs), "missed": m, "miss_rate": m / len(runs) if runs else None} + + out: dict[str, Any] = {"by_grounding_visible_first": {}, "by_calls_between": {}, "by_chars_between": {}, + "by_trace_length": {}} + for v in (1, 0): + out["by_grounding_visible_first"]["visible" if v else "not visible"] = \ + miss([g for g in failed_writes if g["grounding_visible_first"] == v]) + for lo, hi in ((0, 1), (2, 3), (4, 99)): + out["by_calls_between"][f"{lo}-{hi if hi < 99 else '+'}"] = \ + miss([g for g in failed_writes if lo <= g["calls_between"] <= hi]) + for lo, hi in ((0, 2000), (2000, 8000), (8000, 10**9)): + out["by_chars_between"][f"{lo}-{hi if hi < 10**9 else '+'} chars"] = \ + miss([g for g in failed_writes if lo <= g["chars_between"] < hi]) + failed = [r for r in run_rows if r["failed"]] + for lo, hi in ((0, 4), (5, 8), (9, 99)): + rs = [r for r in failed if lo <= r["tool_calls"] <= hi] + m = sum(1 for r in rs if not r["flagged"]) + out["by_trace_length"][f"{lo}-{hi if hi < 99 else '+'} tool calls"] = \ + {"failed_runs": len(rs), "missed": m, "miss_rate": m / len(rs) if rs else None} + return out + + +def samples(run_rows: list[dict[str, Any]], requests: dict[str, dict[str, Any]], base: dict[str, dict[str, str]], + tau_runs: list[tuple[Run, dict[str, Any]]], grounding_rows: list[dict[str, Any]], + gold_rows: list[dict[str, Any]], out: Path, seed: int) -> dict[str, list[str]]: + out.mkdir(parents=True, exist_ok=True) + by_id = {r.trace_id: (r, row) for r, row in tau_runs} + classes: dict[str, Callable[[dict[str, Any]], bool]] = { + "failed_rated_fine": lambda r: r["failed"] and not r["flagged"], + "succeeded_flagged": lambda r: not r["failed"] and r["flagged"], + "failed_flagged": lambda r: r["failed"] and r["flagged"], + } + rng = random.Random(seed) + picked: dict[str, list[str]] = {} + for name, pred in classes.items(): + pool = sorted((r for r in run_rows if r["answer_wrong"] is not None and pred(r)), key=lambda r: r["run_id"]) + chosen = rng.sample(pool, min(10, len(pool))) + picked[name] = [r["run_id"] for r in chosen] + for k, r in enumerate(chosen): + run, _ = by_id[r["trace_id"]] + b = base[r["run_id"]] + spans = {"root": hex_id("root", run.trace_id, size=8), + "tools": [{"index": i, "name": c.name, "span_id": hex_id("tool", run.trace_id, str(i), size=8)} + for i, c in enumerate(run.tool_calls)]} + write_json(out / f"{name}_{k:02d}.json", { + "run_id": r["run_id"], "trace_id": run.trace_id, "class": name, "failed": bool(r["failed"]), + "spans": spans, + "jev_request": requests[run.trace_id], + "jev_response": { + "p_keep": fnum(b.get("p_keep")), "decision_six_any_of": "keep" if b.get("kept") == "1" else "drop", + "answers": {q: fnum(b.get(f"answer_{q}")) for q in QUESTIONS}, + "flags": b.get("flags", ""), "cause": b.get("cause", ""), "tokens": fnum(b.get("tokens")), + }, + "grounding": [g for g in grounding_rows if g["trace_id"] == run.trace_id], + "gold": [g for g in gold_rows if g["trace_id"] == run.trace_id], + "tau_reward": run.meta.get("reward"), + }) + return picked diff --git a/bench/src/tailwise_bench/loaders/__init__.py b/bench/src/tailwise_bench/loaders/__init__.py new file mode 100644 index 0000000..819064c --- /dev/null +++ b/bench/src/tailwise_bench/loaders/__init__.py @@ -0,0 +1 @@ +"""Dataset loaders.""" diff --git a/bench/src/tailwise_bench/loaders/swe.py b/bench/src/tailwise_bench/loaders/swe.py new file mode 100644 index 0000000..1b0d570 --- /dev/null +++ b/bench/src/tailwise_bench/loaders/swe.py @@ -0,0 +1,92 @@ +"""SWE-agent trajectories (nebius/SWE-agent-trajectories, CC-BY-4.0): +coding agents fixing real GitHub issues, 80k runs. + +`target` is true when the agent's patch made the issue's tests pass; a run +with target false failed. A seeded sample is drawn from `shards` parquet +files starting at `first_shard`, at the natural failure rate. + +A trajectory alternates `ai` messages (reasoning ending in one fenced +command) and `user` messages (the command's output). Tailwise sees +SWE-agent's system message as the system prompt, the issue as the question, +every command with its output as a tool call, and the submitted patch as the +final answer. +""" + +from __future__ import annotations + +import re +from pathlib import Path +from typing import Any + +import numpy as np +import pyarrow.parquet as pq +from huggingface_hub import hf_hub_download + +from ..otel import hex_id +from ..types import Run, ToolCall + +COLUMNS = ["instance_id", "model_name", "target", "trajectory", "exit_status", "generated_patch"] +_FENCE = re.compile(r"```[^\n]*\n(.*?)```", re.DOTALL) + + +def download(repo: str, shard: int, shards_total: int, cache: Path) -> Path: + name = f"data/train-{shard:05d}-of-{shards_total:05d}.parquet" + return Path(hf_hub_download(repo, name, repo_type="dataset", local_dir=cache / "hf" / repo)) + + +def load(repo: str, shards: int, shards_total: int, sample: int, seed: int, cache: Path, + limit: int | None = None, first_shard: int = 0) -> tuple[list[Run], dict[str, Any]]: + rows: list[dict[str, Any]] = [] + for shard in range(first_shard, first_shard + shards): + rows += pq.read_table(download(repo, shard, shards_total, cache), columns=COLUMNS).to_pylist() + n = min(sample, len(rows)) if limit is None else min(limit, sample, len(rows)) + picked = sorted(np.random.default_rng(seed).choice(len(rows), size=n, replace=False).tolist()) + runs = [parse(rows[i], i) for i in picked] + return runs, {"pool": len(rows), "runs": len(runs), "failed": sum(r.failed for r in runs), "shards": shards, + "first_shard": first_shard} + + +def issue_text(first_user: str) -> str: + """The issue itself, without SWE-agent's instructions around it.""" + m = re.search(r"ISSUE:\n(.*?)\n+INSTRUCTIONS:", first_user, re.DOTALL) + return m.group(1).strip() if m else first_user.strip() + + +def command_of(ai_text: str) -> str: + blocks = _FENCE.findall(ai_text) + return blocks[-1].strip() if blocks else "" + + +def parse(row: dict[str, Any], index: int) -> Run: + for key in COLUMNS: + if key not in row: + raise ValueError(f"SWE-agent row {index}: no {key!r} (keys {sorted(row)})") + traj = row["trajectory"] or [] + question, calls = "", [] + system = next((m.get("system_prompt") or m.get("text") or "" for m in traj if m.get("role") == "system"), "") + pending: str | None = None + for m in traj: + role, text = m.get("role"), m.get("text") or "" + if role == "user" and not question: + question = issue_text(text) + elif role == "ai": + if pending is not None: + calls.append(_call(pending, "")) + pending = command_of(text) + elif role == "user" and pending is not None: + calls.append(_call(pending, text)) + pending = None + if pending: + calls.append(_call(pending, "")) + run_id = f"{row['instance_id']}:{row['model_name']}:{index}" + return Run( + dataset="swe", subset=str(row["model_name"]), run_id=run_id, trace_id=hex_id("swe", run_id), + failed=not bool(row["target"]), system_prompt=system, question=question, tool_calls=calls, + final_answer=str(row["generated_patch"] or ""), + meta={"instance_id": row["instance_id"], "model": row["model_name"], "exit_status": row["exit_status"]}, + ) + + +def _call(command: str, output: str) -> ToolCall: + name = command.split(maxsplit=1)[0] if command.strip() else "(no command)" + return ToolCall(name=name, args={"command": command}, output=output) diff --git a/bench/src/tailwise_bench/loaders/tau.py b/bench/src/tailwise_bench/loaders/tau.py new file mode 100644 index 0000000..436d6a2 --- /dev/null +++ b/bench/src/tailwise_bench/loaders/tau.py @@ -0,0 +1,128 @@ +"""τ-bench (sierra-research/tau-bench): customer-service agents (retail, +airline) talking to a simulated user and calling tools. + +The repository's historical_trajectories/ hold every run of gpt-4o and +claude-3.5-sonnet with its reward: 1 when the database ended in the right +state and the agent told the user what it had to, else 0. A run with +reward < 1 failed. Files are downloaded at a pinned commit (MIT licensed). + +Each trajectory is OpenAI-style messages: system (the policy), user, +assistant (text or tool_calls), tool (a call's result, "Error: ..." when it +failed). Tailwise sees the policy as the system prompt, the user's messages +as the question, every tool call with its result, and the agent's last +message as the final answer. The user's and agent's turns go on the agent's +last LLM call, as its history and reply. +""" + +from __future__ import annotations + +import json +from pathlib import Path +from typing import Any + +import httpx + +from ..otel import hex_id +from ..types import Run, ToolCall + +RAW = "https://raw.githubusercontent.com/sierra-research/tau-bench/{commit}/historical_trajectories/{file}.json" +STOP = "###STOP###" + + +def download(commit: str, file: str, cache: Path) -> Path: + path = cache / "tau-bench" / commit / f"{file}.json" + if not path.exists(): + path.parent.mkdir(parents=True, exist_ok=True) + r = httpx.get(RAW.format(commit=commit, file=file), timeout=120, follow_redirects=True) + r.raise_for_status() + tmp = path.with_suffix(".part") + tmp.write_bytes(r.content) + tmp.rename(path) + return path + + +def load(commit: str, files: list[str], cache: Path, limit: int | None = None) -> tuple[list[Run], dict[str, Any]]: + from ..investigate.audit import trace_detectable # audit imports this module + + runs: list[Run] = [] + raw: list[tuple[str, dict[str, Any]]] = [] + for i, file in enumerate(files): + rows = json.loads(download(commit, file, cache).read_text()) + n = None if limit is None else limit // len(files) + (1 if i < limit % len(files) else 0) + runs += [parse(file, row) for row in rows[:n]] + raw += [(file.rpartition("-")[2], row) for row in rows[:n]] + for run, labels in zip(runs, trace_detectable(raw), strict=True): + run.meta.update(labels) + return runs, {"runs": len(runs), "failed": sum(r.failed for r in runs), "commit": commit, + "trace_detectable": sum(r.meta["trace_detectable"] for r in runs), + "trace_detectable_failed": sum(r.failed and r.meta["trace_detectable"] for r in runs)} + + +def pair_results(msgs: list[dict[str, Any]]) -> dict[int, tuple[dict[str, Any], int]]: + """Each tool result, by the position of the call it answers (in the order + calls were made): {call index: (tool message, its message index)}. + + Results follow their calls in order. A result's tool_call_id picks its + call only when exactly one waiting call has that id: the claude-3.5-sonnet + logs give every call the id "0", and the gpt-4o logs reuse ids on retries, + so pairing by id alone gives calls the wrong results. + """ + waiting: list[tuple[int, str]] = [] # (call index, id) of calls not yet answered + out: dict[int, tuple[dict[str, Any], int]] = {} + n = 0 + for mi, m in enumerate(msgs): + if m.get("role") == "assistant": + for tc in m.get("tool_calls") or []: + waiting.append((n, str(tc.get("id")))) + n += 1 + elif m.get("role") == "tool" and waiting: + same = [w for w in waiting if w[1] == str(m.get("tool_call_id"))] + pick = same[0] if len(same) == 1 else waiting[0] + waiting.remove(pick) + out[pick[0]] = (m, mi) + return out + + +def parse(file: str, row: dict[str, Any]) -> Run: + for key in ("task_id", "reward", "traj"): + if key not in row: + raise ValueError(f"τ-bench {file}: run without {key!r} (keys {sorted(row)})") + model, _, domain = file.rpartition("-") + run_id = f"{file}:{row['task_id']}:{row.get('trial', 0)}" + msgs = row["traj"] + results = pair_results(msgs) + + user_turns: list[str] = [] + turns: list[tuple[str, str]] = [] + calls: list[ToolCall] = [] + final = "" + system = next((m.get("content") or "" for m in msgs if m.get("role") == "system"), "") + for m in msgs: + role, content = m.get("role"), m.get("content") or "" + if role == "user": + text = content.replace(STOP, "").strip() + if text: + user_turns.append(text) + turns.append(("user", text)) + elif role == "assistant": + if content.strip(): + final = content + turns.append(("assistant", content)) + for tc in m.get("tool_calls") or []: + fn = tc.get("function") or {} + args: Any = fn.get("arguments") or {} + if isinstance(args, str): # OpenAI sends a JSON string; some files hold the object + try: + args = json.loads(args or "{}") + except json.JSONDecodeError: + pass + out = str(results[len(calls)][0].get("content") or "") if len(calls) in results else "" + calls.append(ToolCall(name=fn.get("name") or "tool", args=args, output=out, + error=out.startswith("Error"))) + return Run( + dataset="tau", subset=f"{domain}/{model}", run_id=run_id, trace_id=hex_id("tau", run_id), + failed=float(row["reward"]) < 1, system_prompt=system, question="\n\n".join(user_turns), + tool_calls=calls, final_answer=final, messages=turns, + meta={"domain": domain, "model": model, "task_id": row["task_id"], "trial": row.get("trial", 0), + "reward": row["reward"]}, + ) diff --git a/bench/src/tailwise_bench/loaders/trail.py b/bench/src/tailwise_bench/loaders/trail.py new file mode 100644 index 0000000..a15c119 --- /dev/null +++ b/bench/src/tailwise_bench/loaders/trail.py @@ -0,0 +1,106 @@ +"""TRAIL (PatronusAI/TRAIL): real agent traces (GAIA, SWE-bench) with every +error annotated on the span where it happened, with a category and impact. + +TRAIL labels errors, not outcomes, so a run counts as failed when it has at +least one HIGH-impact error: the closest it comes to "this run went wrong". +Nearly every trace has some error, so "any error" would leave almost nothing +to tell apart. + +Gated on Hugging Face and licensed for evaluation only: the data is +downloaded into the git-ignored cache and never written anywhere else, never +used to train or tune anything. + +Each row holds `trace` (a JSON string: {"trace_id", "spans": [...]}, the +spans nested under "child_spans") and `labels` (a JSON string: {"errors": +[{"location": span_id, "category", "impact", ...}], "scores": [...]}). +""" + +from __future__ import annotations + +import json +import re +from pathlib import Path +from typing import Any + +import pyarrow.parquet as pq +from huggingface_hub import HfApi, hf_hub_download +from huggingface_hub.errors import GatedRepoError, HfHubHTTPError + +from ..otel import iso_nanos, otlp_span +from ..types import Run + + +class SchemaError(ValueError): + pass + + +def download(repo: str, splits: list[str], cache: Path) -> dict[str, Path]: + """The parquet file of each split, downloaded once.""" + try: + files = HfApi().list_repo_files(repo, repo_type="dataset") + out = {} + for split in splits: + matches = [f for f in files if f.startswith(f"data/{split}-") and f.endswith(".parquet")] + if len(matches) != 1: + raise SchemaError(f"{repo}: expected one parquet file for split {split!r}, found {matches}") + out[split] = Path(hf_hub_download(repo, matches[0], repo_type="dataset", local_dir=cache / "hf" / repo)) + return out + except GatedRepoError as e: + raise SystemExit( + f"{repo} is gated and this Hugging Face account has no access yet. Accept its terms at " + f"https://huggingface.co/datasets/{repo}, then set HF_TOKEN or run `hf auth login`." + ) from e + except HfHubHTTPError as e: + if e.response is not None and e.response.status_code in (401, 403): + raise SystemExit(f"{repo}: access denied (HTTP {e.response.status_code}). Accept the terms at " + f"https://huggingface.co/datasets/{repo} and check HF_TOKEN.") from e + raise + + +def load(repo: str, splits: list[str], cache: Path, limit: int | None = None) -> tuple[list[Run], dict[str, Any]]: + files = download(repo, splits, cache) + runs: list[Run] = [] + for i, split in enumerate(splits): + rows = pq.read_table(files[split]).to_pylist() + if rows and not {"trace", "labels"} <= rows[0].keys(): + raise SchemaError(f"{repo}/{split}: expected columns trace and labels, got {sorted(rows[0])}") + n = None if limit is None else limit // len(splits) + (1 if i < limit % len(splits) else 0) + runs += [parse(split, loads(r["trace"]), loads(r["labels"])) for r in rows[:n]] + return runs, {"runs": len(runs), "failed": sum(r.failed for r in runs), + "with_any_error": sum(r.meta["errors"] > 0 for r in runs)} + + +_TRAILING_COMMA = re.compile(r",(\s*[\]}])") + + +def loads(text: str) -> Any: + """JSON, tolerating the trailing commas a few hand-edited TRAIL labels + have (one GAIA row has "},\n ]").""" + try: + return json.loads(text) + except json.JSONDecodeError: + return json.loads(_TRAILING_COMMA.sub(r"\1", text)) + + +def parse(split: str, trace: dict[str, Any], labels: dict[str, Any]) -> Run: + if "spans" not in trace or "trace_id" not in trace: + raise SchemaError(f"TRAIL trace without spans or trace_id: keys {sorted(trace)}") + flat: list[tuple[dict[str, Any], str | None]] = [] + + def walk(span: dict[str, Any], parent: str | None) -> None: + flat.append((span, parent)) + for child in span.get("child_spans") or []: + walk(child, span["span_id"]) + + for top in trace["spans"]: + walk(top, top.get("parent_span_id")) + flat.sort(key=lambda sp: iso_nanos(sp[0]["timestamp"])) + + errors = labels.get("errors") or [] + impacts = [str(e.get("impact", "")).strip().upper() for e in errors] + return Run( + dataset="trail", subset=split, run_id=trace["trace_id"], trace_id=trace["trace_id"], + failed="HIGH" in impacts, otlp_spans=[otlp_span(s, p) for s, p in flat], + meta={"errors": len(errors), "high_impact_errors": impacts.count("HIGH"), + "categories": sorted({str(e.get("category", "")).strip() for e in errors})}, + ) diff --git a/bench/src/tailwise_bench/metrics.py b/bench/src/tailwise_bench/metrics.py new file mode 100644 index 0000000..6aa6e6c --- /dev/null +++ b/bench/src/tailwise_bench/metrics.py @@ -0,0 +1,234 @@ +"""The metrics, as pure functions over labels (1 = the run failed) and scores. + +Every random choice takes an explicit seed. Ties in score are broken by a +seeded shuffle, so reviewing "by score" among equal scores is random, not by +position. +""" + +from __future__ import annotations + +import math +from collections.abc import Sequence +from typing import Any + +import numpy as np +from sklearn.metrics import average_precision_score, precision_recall_curve, roc_auc_score, roc_curve + +Array = np.ndarray + + +def _arrays(y: Sequence[int], s: Sequence[float]) -> tuple[Array, Array]: + ya, sa = np.asarray(y, dtype=int), np.asarray(s, dtype=float) + if ya.shape != sa.shape: + raise ValueError("labels and scores differ in length") + return ya, sa + + +def precision_at_rate(tpr: float | None, fpr: float | None, rate: float) -> float | None: + """The precision the same decisions would have if `rate` of runs failed: + TPR and FPR do not depend on the failure rate, precision does.""" + if tpr is None or fpr is None: + return None + kept = tpr * rate + fpr * (1 - rate) + return tpr * rate / kept if kept else None + + +def threshold_metrics(y: Sequence[int], pred: Sequence[int], rates: Sequence[float] = ()) -> dict[str, Any]: + """The keep decisions against the labels. `rates` adds the precision the + decisions would have at those production failure rates.""" + ya, pa = np.asarray(y, dtype=int), np.asarray(pred, dtype=int) + tp = int(((ya == 1) & (pa == 1)).sum()) + fp = int(((ya == 0) & (pa == 1)).sum()) + fn = int(((ya == 1) & (pa == 0)).sum()) + tn = int(((ya == 0) & (pa == 0)).sum()) + n = len(ya) + precision = tp / (tp + fp) if tp + fp else None + recall = tp / (tp + fn) if tp + fn else None + fpr = fp / (fp + tn) if fp + tn else None + f1 = (2 * precision * recall / (precision + recall) + if precision is not None and recall is not None and precision + recall else None) + base = (tp + fn) / n if n else None + return { + "kept_share": (tp + fp) / n if n else None, + "precision": precision, "base_rate": base, + "lift": precision / base if precision is not None and base else None, + "recall": recall, "false_positive_rate": fpr, "f1": f1, + "tp": tp, "fp": fp, "fn": fn, "tn": tn, + "precision_at_failure_rate": {str(r): precision_at_rate(recall, fpr, r) for r in rates}, + } + + +def ranking_metrics(y: Sequence[int], s: Sequence[float]) -> dict[str, Any]: + """ROC-AUC and PR-AUC (average precision); None when one class is missing.""" + ya, sa = _arrays(y, s) + base = float(ya.mean()) if len(ya) else None + if len(set(ya.tolist())) < 2: + return {"roc_auc": None, "pr_auc": None, "pr_auc_random": base} + return {"roc_auc": float(roc_auc_score(ya, sa)), "pr_auc": float(average_precision_score(ya, sa)), + "pr_auc_random": base} + + +def curves(y: Sequence[int], s: Sequence[float]) -> dict[str, Any] | None: + ya, sa = _arrays(y, s) + if len(set(ya.tolist())) < 2: + return None + fpr, tpr, _ = roc_curve(ya, sa) + precision, recall, _ = precision_recall_curve(ya, sa) + return {"fpr": fpr.tolist(), "tpr": tpr.tolist(), "precision": precision.tolist(), "recall": recall.tolist()} + + +def rank_order(s: Array, seed: int) -> Array: + """Indices from highest score to lowest, ties in a seeded random order.""" + tiebreak = np.random.default_rng(seed).permutation(len(s)) + return np.lexsort((tiebreak, -s)) + + +def budget_k(budget: float, n: int) -> int: + return min(n, max(1, math.ceil(budget * n))) if n else 0 + + +def budget_curve(y: Sequence[int], s: Sequence[float], budgets: Sequence[float], seeds: int, seed: int) -> list[dict[str, Any]]: + """Reviewing the top budget share of runs by score, against reviewing the + same number picked at random (seeds runs: mean and 95% percentile + interval): the share of failures found (recall) and the share of reviewed + runs that failed (precision).""" + ya, sa = _arrays(y, s) + total = int(ya.sum()) + order = rank_order(sa, seed) + rng = np.random.default_rng(seed + 1) + perms = [rng.permutation(len(ya)) for _ in range(seeds)] + out = [] + for b in budgets: + k = budget_k(b, len(ya)) + found = int(ya[order[:k]].sum()) + rand = np.array([ya[p[:k]].sum() for p in perms], dtype=float) + + def ci(v: Array) -> list[float]: + return [float(np.percentile(v, 2.5)), float(np.percentile(v, 97.5))] + + out.append({ + "budget": b, "k": k, "n": len(ya), "failures": total, + "tailwise_recall": found / total if total else None, + "random_recall_mean": float(rand.mean()) / total if total else None, + "random_recall_ci95": ci(rand / total) if total else None, + "tailwise_precision": found / k if k else None, + "random_precision_mean": float(rand.mean()) / k if k else None, + "random_precision_ci95": ci(rand / k) if k else None, + "lift": found / float(rand.mean()) if rand.mean() > 0 else None, + # The most a perfect ranker could find at this budget. + "best_possible_recall": min(k, total) / total if total else None, + }) + return out + + +def review_precision(y: Sequence[int], s: Sequence[float], budget: float, rates: Sequence[float], + seeds: int, seed: int, groups: Sequence[str] | None = None, + baseline: Sequence[float] | None = None) -> dict[str, Any]: + """The headline: a team reviews the top `budget` share of runs by score; + what share of those failed? At the dataset's own failure rate, and at each + production failure rate in `rates`. The datasets fail far more often than + production agents, so for each rate, failed runs (or, above the dataset's + rate, successful ones) are dropped at random until `rate` of runs fail. + + Each of the `seeds` draws first resamples the tasks in `groups` with + replacement (each run its own task without them): runs of one task are + not independent, so the 95% interval covers which tasks the dataset + happened to contain, not only which runs were dropped. With `baseline` + scores, each draw also ranks the same runs by them, and "vs_baseline" is + the gain in precision: its mean, 95% interval, and the share of draws in + which the baseline did at least as well. + + Random review finds failures at the failure rate; a perfect ranker finds + min(1, rate / budget).""" + ya, sa = _arrays(y, s) + ba = np.asarray(baseline, dtype=float) if baseline is not None else None + if ba is not None and ba.shape != sa.shape: + raise ValueError("baseline and scores differ in length") + g = np.asarray(groups if groups is not None else range(len(ya))) + if g.shape != ya.shape: + raise ValueError("groups and labels differ in length") + + def top(idx: Array, scores: Array, tie_seed: int) -> float: + k = budget_k(budget, len(idx)) + return float(ya[idx[rank_order(scores[idx], tie_seed)[:k]]].mean()) if k else float("nan") + + def ci(v: Array) -> list[float]: + return [float(np.percentile(v, 2.5)), float(np.percentile(v, 97.5))] + + real = float(ya.mean()) if len(ya) else None + out: dict[str, Any] = { + "budget": budget, + "real": {"failure_rate": real, "precision": top(np.arange(len(ya)), sa, seed) if len(ya) else None, + "best_possible": min(1.0, real / budget) if real is not None else None}, + "at_rate": {}, + } + if not (ya == 1).any() or not (ya == 0).any(): + return out + tasks, members = np.unique(g, return_inverse=True) + by_task = [np.flatnonzero(members == t) for t in range(len(tasks))] + rng = np.random.default_rng(seed + 2) + for rate in rates: + vals, gains = [], [] + for i in range(seeds): + pool = np.concatenate([by_task[t] for t in rng.integers(len(tasks), size=len(tasks))]) + fail, ok = pool[ya[pool] == 1], pool[ya[pool] == 0] + if not len(fail) or not len(ok): + continue + n_fail = round(rate / (1 - rate) * len(ok)) + if n_fail <= len(fail): + idx = np.concatenate([rng.choice(fail, n_fail, replace=False), ok]) + else: + n_ok = min(len(ok), round(len(fail) * (1 - rate) / rate)) + idx = np.concatenate([fail, rng.choice(ok, n_ok, replace=False)]) + vals.append(top(idx, sa, seed + 3 + i)) + if ba is not None: + gains.append(vals[-1] - top(idx, ba, seed + 3 + i)) + v = np.asarray(vals) + n_fail, n_ok = int((ya == 1).sum()), int((ya == 0).sum()) + row: dict[str, Any] = { + "precision": float(v.mean()), "ci95": ci(v), + "lift": float(v.mean()) / rate, "best_possible": min(1.0, rate / budget), + # Reviewed runs per draw: few means a noisy estimate. + "reviewed": budget_k(budget, min(len(ya), round(n_ok / (1 - rate)), round(n_fail / rate))), + } + if gains: + d = np.asarray(gains) + row["vs_baseline"] = {"gain": float(d.mean()), "ci95": ci(d), "not_better": float((d <= 0).mean())} + out["at_rate"][str(rate)] = row + return out + + +def calibration(y: Sequence[int], s: Sequence[float], bins: int = 10) -> dict[str, Any]: + """Reliability-diagram data and expected calibration error, equal-width bins.""" + ya, sa = _arrays(y, s) + edges = np.linspace(0, 1, bins + 1) + index = np.clip(np.digitize(sa, edges[1:-1], right=False), 0, bins - 1) + rows, ece = [], 0.0 + for b in range(bins): + mask = index == b + n = int(mask.sum()) + mean_pred = float(sa[mask].mean()) if n else None + frac_pos = float(ya[mask].mean()) if n else None + if mean_pred is not None and frac_pos is not None: + ece += n / len(ya) * abs(mean_pred - frac_pos) + rows.append({"lo": float(edges[b]), "hi": float(edges[b + 1]), "count": n, + "mean_predicted": mean_pred, "fraction_failed": frac_pos}) + return {"bins": rows, "ece": ece if len(ya) else None} + + +def flag_stats(y: Sequence[int], flags: Sequence[Sequence[str]]) -> dict[str, dict[str, Any]]: + """For each question that flagged at least one run: how often it fires on + failed and on successful runs, and how often a run it flags failed.""" + ya = list(y) + failed, ok = sum(ya), len(ya) - sum(ya) + out: dict[str, dict[str, Any]] = {} + for label, fs in zip(ya, flags, strict=True): + for f in set(fs): + row = out.setdefault(f, {"flagged": 0, "flagged_failed": 0}) + row["flagged"] += 1 + row["flagged_failed"] += label + for row in out.values(): + row["precision"] = row["flagged_failed"] / row["flagged"] + row["recall"] = row["flagged_failed"] / failed if failed else None + row["false_positive_rate"] = (row["flagged"] - row["flagged_failed"]) / ok if ok else None + return dict(sorted(out.items(), key=lambda kv: -kv[1]["flagged"])) diff --git a/bench/src/tailwise_bench/otel.py b/bench/src/tailwise_bench/otel.py new file mode 100644 index 0000000..5476ff7 --- /dev/null +++ b/bench/src/tailwise_bench/otel.py @@ -0,0 +1,205 @@ +"""OTLP/JSON for Tailwise's POST /v1/otlp/traces:score, one request per run. + +A run becomes the spans an OpenInference-instrumented agent emits for the +parts Tailwise reads: an AGENT root span with the question as input.value and +the final answer as output.value, an LLM span whose system message is the +agent's system prompt, a TOOL span per tool call with its arguments, output +and status, and, when the run has its turns, a last LLM span holding the +conversation. TRAIL runs are sent as their own spans. +""" + +from __future__ import annotations + +import hashlib +import json +import re +from datetime import datetime +from typing import Any + +from .types import Run + +# Tailwise's own limits when it renders the state (internal/trace/slice.go). +TAILWISE_MAX_EVIDENCE = 3 +TAILWISE_MAX_EVIDENCE_CHARS = 1500 +TAILWISE_MAX_SYSTEM_PROMPT_CHARS = 8000 + + +def hex_id(*parts: str, size: int = 16) -> str: + """A stable, nonzero hex ID of size bytes from parts.""" + digest = hashlib.sha256("\x1f".join(parts).encode()).hexdigest()[: size * 2] + return digest if set(digest) != {"0"} else "1" + digest[1:] + + +def attr(key: str, value: Any) -> dict[str, Any]: + if isinstance(value, bool): + return {"key": key, "value": {"boolValue": value}} + if isinstance(value, int): + return {"key": key, "value": {"intValue": str(value)}} + if isinstance(value, float): + return {"key": key, "value": {"doubleValue": value}} + if not isinstance(value, str): + value = json.dumps(value, ensure_ascii=False, default=str) + return {"key": key, "value": {"stringValue": value}} + + +def request(spans: list[dict[str, Any]], resource: dict[str, Any] | None = None) -> dict[str, Any]: + """An ExportTraceServiceRequest holding spans.""" + res = [attr(k, v) for k, v in (resource or {}).items()] + return {"resourceSpans": [{ + "resource": {"attributes": res}, + "scopeSpans": [{"scope": {"name": "tailwise-bench"}, "spans": spans}], + }]} + + +def run_trace(run: Run) -> tuple[dict[str, Any], str]: + """The run as OTLP spans, and the state Tailwise will build from them + (for token estimates).""" + if run.otlp_spans is not None: + return native_trace(run) + root_id = hex_id("root", run.trace_id, size=8) + root_attrs = [ + attr("openinference.span.kind", "AGENT"), + attr("input.value", run.question), + attr("output.value", run.final_answer), + ] + if run.agent: + root_attrs.append(attr("agent.name", run.agent)) + end = 10_000 * (len(run.tool_calls) + 2) + spans = [{"traceId": run.trace_id, "spanId": root_id, "name": run.agent or "agent_run", + "startTimeUnixNano": "1000", "endTimeUnixNano": str(end), "attributes": root_attrs}] + if run.system_prompt: + # Where an OpenInference instrumentor records the agent's instructions: + # the system message of its first LLM call. + spans.append({ + "traceId": run.trace_id, "spanId": hex_id("llm", run.trace_id, size=8), "parentSpanId": root_id, + "name": "llm", "startTimeUnixNano": "1500", "endTimeUnixNano": "1900", + "attributes": [attr("openinference.span.kind", "LLM"), + attr("llm.input_messages.0.message.role", "system"), + attr("llm.input_messages.0.message.content", run.system_prompt), + attr("llm.input_messages.1.message.role", "user"), + attr("llm.input_messages.1.message.content", run.question)], + }) + for i, c in enumerate(run.tool_calls): + start = 2000 + 10_000 * i + a = [attr("openinference.span.kind", "TOOL"), attr("tool.name", c.name), + attr("input.value", c.args if isinstance(c.args, str) else json.dumps(c.args, ensure_ascii=False)), + attr("output.value", c.output)] + span: dict[str, Any] = { + "traceId": run.trace_id, "spanId": hex_id("tool", run.trace_id, str(i), size=8), "parentSpanId": root_id, + "name": c.name, "startTimeUnixNano": str(start), "endTimeUnixNano": str(start + 5000), "attributes": a, + } + if c.error: + span["status"] = {"code": 2, "message": "tool error"} + spans.append(span) + if run.messages: + # The agent's last LLM call, where Tailwise reads the conversation: the + # turns before its reply as input, and the reply (the agent's last turn, + # when the conversation ends with one) as output. + history, reply = run.messages, "" + if history[-1][0] == "assistant": + history, reply = history[:-1], history[-1][1] + start = 2000 + 10_000 * len(run.tool_calls) + a = [attr("openinference.span.kind", "LLM")] + for i, (role, text) in enumerate(history): + a += [attr(f"llm.input_messages.{i}.message.role", role), + attr(f"llm.input_messages.{i}.message.content", text)] + if reply: + a += [attr("llm.output_messages.0.message.role", "assistant"), + attr("llm.output_messages.0.message.content", reply)] + spans.append({"traceId": run.trace_id, "spanId": hex_id("llm-last", run.trace_id, size=8), + "parentSpanId": root_id, "name": "llm", "startTimeUnixNano": str(start), + "endTimeUnixNano": str(start + 5000), "attributes": a}) + return request(spans), estimate_state(spans) + + +# --- TRAIL's own spans -------------------------------------------------------- + +_DURATION = re.compile(r"^P(?:(?P\d+)D)?(?:T(?:(?P\d+)H)?(?:(?P\d+)M)?(?:(?P[\d.]+)S)?)?$") +_STATUS = {"ok": 1, "error": 2} + + +def iso_nanos(ts: str) -> int: + dt = datetime.fromisoformat(ts.replace("Z", "+00:00")) + return int(dt.timestamp()) * 1_000_000_000 + dt.microsecond * 1000 + + +def duration_nanos(d: str | None) -> int: + m = _DURATION.match(d or "") + if not m: + return 0 + days, hours, mins = (int(m.group(g) or 0) for g in ("d", "h", "m")) + secs = float(m.group("s") or 0) + return int(((days * 24 + hours) * 60 + mins) * 60 * 1e9 + secs * 1e9) + + +def otlp_span(span: dict[str, Any], parent: str | None) -> dict[str, Any]: + """One TRAIL span, without its children, as an OTLP/JSON span.""" + start = iso_nanos(span["timestamp"]) + out: dict[str, Any] = { + "traceId": span["trace_id"], + "spanId": span["span_id"], + "name": span.get("span_name") or "span", + "startTimeUnixNano": str(start), + "endTimeUnixNano": str(start + duration_nanos(span.get("duration"))), + "attributes": [attr(k, v) for k, v in (span.get("span_attributes") or {}).items() if v is not None], + } + if parent: + out["parentSpanId"] = parent + code = _STATUS.get(str(span.get("status_code", "")).lower()) + if code: + out["status"] = {"code": code, "message": span.get("status_message") or ""} + return out + + +def native_trace(run: Run) -> tuple[dict[str, Any], str]: + """A TRAIL run's own spans. + + TRAIL's top span (`main`) carries no input or output, so Tailwise, which + reads the question and answer from the root, would find nothing to score. + The outermost AGENT span is made a root (its parent cleared) so Tailwise + picks it: the AGENT kind wins among parentless spans. Nothing else changes. + """ + assert run.otlp_spans is not None + spans = run.otlp_spans + agents = [s for s in spans if _value(s, "openinference.span.kind") == "AGENT"] + if agents: + by_id = {s["spanId"]: s for s in spans} + outer = min(agents, key=lambda s: _depth(s, by_id)) + spans = [({k: v for k, v in s.items() if k != "parentSpanId"} if s is outer else s) for s in spans] + return request(spans), estimate_state(spans) + + +def _depth(span: dict[str, Any], by_id: dict[str, dict[str, Any]]) -> int: + depth = 0 + while span.get("parentSpanId") in by_id and depth < 10_000: + span = by_id[span["parentSpanId"]] + depth += 1 + return depth + + +def _value(span: dict[str, Any], key: str) -> str: + for a in span["attributes"]: + if a["key"] == key: + return str(next(iter(a["value"].values()))) + return "" + + +def estimate_state(spans: list[dict[str, Any]]) -> str: + """Roughly what Tailwise's trace.BuildState renders from the spans: the + root's question and answer, one line per tool call, at most three evidence + paragraphs from successful tool outputs. Used only to count tokens.""" + roots = [s for s in spans if "parentSpanId" not in s] + root = next((s for s in roots if _value(s, "openinference.span.kind") == "AGENT"), roots[0] if roots else spans[0]) + tools = sorted((s for s in spans if _value(s, "openinference.span.kind") == "TOOL"), + key=lambda s: int(s["startTimeUnixNano"])) + lines = [f"{i + 1}. {_value(s, 'tool.name') or s['name']}({_value(s, 'input.value')}) -> " + f"{'error' if s.get('status', {}).get('code') == 2 else 'ok'}" for i, s in enumerate(tools)] + evidence = [p[:TAILWISE_MAX_EVIDENCE_CHARS] for s in tools if s.get("status", {}).get("code") != 2 + for p in _value(s, "output.value").split("\n\n") if p.strip()][:TAILWISE_MAX_EVIDENCE] + ev = [f"[{i + 1}] {e}" for i, e in enumerate(evidence)] + llm = next((s for s in spans if _value(s, "openinference.span.kind") == "LLM" + and _value(s, "llm.input_messages.0.message.role") == "system"), None) + system = _value(llm, "llm.input_messages.0.message.content")[:TAILWISE_MAX_SYSTEM_PROMPT_CHARS] if llm else "" + prefix = f"SYSTEM PROMPT:\n{system}\n\n" if system else "" + return (f"{prefix}USER QUESTION:\n{_value(root, 'input.value')}\n\nTOOL CALLS:\n{chr(10).join(lines) or '(none)'}" + f"\n\nEVIDENCE:\n{chr(10).join(ev) or '(none)'}\n\nFINAL ANSWER:\n{_value(root, 'output.value')}") diff --git a/bench/src/tailwise_bench/plots.py b/bench/src/tailwise_bench/plots.py new file mode 100644 index 0000000..6a51f5d --- /dev/null +++ b/bench/src/tailwise_bench/plots.py @@ -0,0 +1,165 @@ +"""The PNG plots. One look throughout: Tailwise in the first series color, +answer_wrong alone in the second, random sampling in neutral gray with its +95% band, recessive axes.""" + +from __future__ import annotations + +from collections.abc import Callable +from pathlib import Path +from typing import Any + +import matplotlib + +matplotlib.use("Agg") +import matplotlib.pyplot as plt # noqa: E402 + +SURFACE = "#fcfcfb" +INK = "#0b0b0b" +INK_2 = "#52514e" +GRID = "#e4e3df" +SERIES = ["#2a78d6", "#eb6834", "#1baf7a"] # fixed order: Tailwise default, answer_wrong only, third +BASELINE = "#8a8984" +NAMES = {"tau": "τ-bench", "swe": "SWE-agent", "trail": "TRAIL"} + +plt.rcParams.update({ + "figure.facecolor": SURFACE, "axes.facecolor": SURFACE, "savefig.facecolor": SURFACE, + "axes.edgecolor": GRID, "axes.labelcolor": INK_2, "axes.titlecolor": INK, "axes.titlesize": 11, + "axes.titleweight": "bold", "axes.labelsize": 9, "xtick.color": INK_2, "ytick.color": INK_2, + "xtick.labelsize": 8, "ytick.labelsize": 8, "axes.grid": True, "grid.color": GRID, "grid.linewidth": 0.6, + "axes.axisbelow": True, "axes.spines.top": False, "axes.spines.right": False, "legend.frameon": False, + "legend.fontsize": 8, "font.family": "sans-serif", "lines.linewidth": 2, "lines.markersize": 5, +}) + + +def name(ds: str) -> str: + return NAMES.get(ds, ds) + + +def _save(fig: Any, path: Path) -> Path: + fig.tight_layout() + fig.savefig(path, dpi=160) + plt.close(fig) + return path + + +def _panels(n: int, height: float = 3.7) -> tuple[Any, list[Any]]: + fig, axes = plt.subplots(1, n, figsize=(4.4 * n, height), squeeze=False) + return fig, list(axes[0]) + + +def budget(datasets: dict[str, Any], out: Path) -> Path | None: + if not datasets: + return None + fig, axes = _panels(len(datasets)) + for ax, (ds, s) in zip(axes, datasets.items(), strict=True): + tw, aw = s["default"]["budget_curve"], s["answer_wrong_only"]["budget_curve"] + xs = [p["budget"] * 100 for p in tw] + ax.fill_between(xs, [p["random_recall_ci95"][0] for p in tw], [p["random_recall_ci95"][1] for p in tw], + color=BASELINE, alpha=0.18, linewidth=0, label="Random, 95% interval") + ax.plot(xs, [p["random_recall_mean"] for p in tw], color=BASELINE, linestyle="--", label="Random sampling") + ax.plot(xs, [p["best_possible_recall"] for p in tw], color=INK_2, linestyle=":", linewidth=1.2, label="Best possible") + ax.plot(xs, [p["tailwise_recall"] for p in aw], color=SERIES[1], marker="o", label="answer_wrong only") + ax.plot(xs, [p["tailwise_recall"] for p in tw], color=SERIES[0], marker="o", label="Tailwise (default)") + ax.set(title=f"{name(ds)}: failed runs found", xlabel="Runs reviewed (% of all, highest score first)", + ylabel="Share of failed runs found", ylim=(0, 1.02), xlim=(0, max(xs) + 2)) + axes[0].legend(loc="upper left") + return _save(fig, out / "budget_curve.png") + + +def kept_precision(datasets: dict[str, Any], out: Path) -> Path | None: + """Of the runs kept at the threshold, the share that failed, against the + share that fails overall (what random keeping gets).""" + if not datasets: + return None + labels = [name(ds) for ds in datasets] + xs = range(len(labels)) + fig, ax = plt.subplots(figsize=(1.9 * len(labels) + 3, 3.8)) + groups: list[tuple[str, str, Callable[[dict[str, Any]], float | None]]] = [("Random keeping (failure rate)", BASELINE, lambda s: s["default"]["at_threshold"]["base_rate"]), + ("answer_wrong only", SERIES[1], lambda s: s["answer_wrong_only"]["at_threshold"]["precision"]), + ("Tailwise (default)", SERIES[0], lambda s: s["default"]["at_threshold"]["precision"])] + w = 0.26 + for i, (label, color, get) in enumerate(groups): + vals = [get(s) or 0 for s in datasets.values()] + bars = ax.bar([x + (i - 1) * w for x in xs], vals, width=w - 0.02, color=color, label=label) + for b, v in zip(bars, vals, strict=True): + ax.text(b.get_x() + b.get_width() / 2, v + 0.015, f"{v:.0%}", ha="center", fontsize=7.5, color=INK_2) + ax.set_xticks(list(xs), labels) + ax.set(title="Of the runs kept, the share that failed", ylabel="Precision", ylim=(0, 1.12)) + ax.grid(axis="x", visible=False) + ax.legend(loc="upper left", ncol=3) + return _save(fig, out / "kept_precision.png") + + +def roc_pr(curves: dict[str, Any], datasets: dict[str, Any], out: Path) -> list[Path]: + paths = [] + for kind, x, y, xl, yl, auc in (("roc", "fpr", "tpr", "False positive rate", "True positive rate", "roc_auc"), + ("pr", "recall", "precision", "Recall", "Precision", "pr_auc")): + fig, ax = plt.subplots(figsize=(4.8, 4.1)) + for i, ds in enumerate(curves): + v = datasets[ds]["default"][auc] + ax.plot(curves[ds][x], curves[ds][y], color=SERIES[i % 3], + label=f"{name(ds)} ({auc.upper().replace('_', '-')} {v:.3f})") + if kind == "pr": + base = datasets[ds]["default"]["pr_auc_random"] + ax.axhline(base, color=SERIES[i % 3], linestyle="--", linewidth=0.8, alpha=0.6) + if kind == "roc": + ax.plot([0, 1], [0, 1], color=BASELINE, linestyle="--", linewidth=1, label="Random") + ax.set(title=f"Tailwise score: {'ROC' if kind == 'roc' else 'precision-recall (dashed: random)'}", + xlabel=xl, ylabel=yl, xlim=(0, 1), ylim=(0, 1.02)) + ax.legend(loc="lower right" if kind == "roc" else "lower left") + paths.append(_save(fig, out / f"{kind}_curve.png")) + return paths + + +def reliability(datasets: dict[str, Any], out: Path) -> Path | None: + if not datasets: + return None + fig, ax = plt.subplots(figsize=(4.8, 4.1)) + ax.plot([0, 1], [0, 1], color=BASELINE, linestyle="--", linewidth=1, label="Perfectly calibrated") + for i, (ds, s) in enumerate(datasets.items()): + cal = s["default"]["calibration"] + pts = [(b["mean_predicted"], b["fraction_failed"]) for b in cal["bins"] if b["count"]] + ax.plot([p[0] for p in pts], [p[1] for p in pts], color=SERIES[i % 3], marker="o", + label=f"{name(ds)} (ECE {cal['ece']:.3f})") + ax.set(title="Is p_keep a probability of failure?", xlabel="Tailwise score (p_keep)", + ylabel="Share of runs that failed", xlim=(0, 1), ylim=(0, 1.02)) + ax.legend(loc="upper left") + return _save(fig, out / "reliability.png") + + +def flags(datasets: dict[str, Any], out: Path) -> Path | None: + """Per question: how often it fires on failed runs and on successful ones.""" + if not datasets: + return None + qs = sorted({q for s in datasets.values() for q in s["flags"]}) + if not qs: + return None + fig, axes = _panels(len(datasets), height=0.42 * len(qs) + 1.6) + for ax, (ds, s) in zip(axes, datasets.items(), strict=True): + ys = range(len(qs)) + fail = [(s["flags"].get(q) or {}).get("recall") or 0 for q in qs] + ok = [(s["flags"].get(q) or {}).get("false_positive_rate") or 0 for q in qs] + ax.barh([y + 0.2 for y in ys], fail, height=0.38, color=SERIES[0], label="Fires on failed runs") + ax.barh([y - 0.2 for y in ys], ok, height=0.38, color=BASELINE, label="Fires on successful runs") + ax.set_yticks(list(ys), qs) + ax.set(title=name(ds), xlabel="Share of runs", xlim=(0, 1.02)) + ax.grid(axis="y", visible=False) + axes[0].legend(loc="lower right") + fig.suptitle("Which checks tell failed runs apart", fontsize=11, fontweight="bold", color=INK) + return _save(fig, out / "flags.png") + + +def cost(datasets: dict[str, Any], out: Path) -> Path | None: + if not datasets: + return None + names = [name(ds) for ds in datasets] + vals = [s["cost_usd"] for s in datasets.values()] + fig, ax = plt.subplots(figsize=(6.6, 0.45 * len(names) + 1.3)) + ax.barh(names, vals, color=SERIES[0], height=0.6) + top = max(vals) or 1 + for i, s in enumerate(datasets.values()): + ax.text(vals[i] + top * 0.01, i, f"${vals[i]:.4f} ({s['tokens']:,} tokens, ${s['cost_per_run_usd'] * 1000:.3f} per 1,000 runs)", + va="center", fontsize=7.5, color=INK_2) + ax.set(title="Cost by dataset", xlabel="US dollars (Jev input tokens)", xlim=(0, top * 2.2)) + ax.grid(axis="y", visible=False) + return _save(fig, out / "cost_by_dataset.png") diff --git a/bench/src/tailwise_bench/publish.py b/bench/src/tailwise_bench/publish.py new file mode 100644 index 0000000..1af6122 --- /dev/null +++ b/bench/src/tailwise_bench/publish.py @@ -0,0 +1,91 @@ +"""The published figures: bench/published/benchmark.json, read by the website +at the latest release's tag. One row per dataset slice and setup, at the +cost section's deployment (report.COST_TRACES, COST_FAILURE_RATE).""" + +from __future__ import annotations + +import json +from pathlib import Path +from typing import Any + +from .report import COST_FAILURE_RATE, COST_TRACES, FEW_REVIEWED, _review_at + +SCHEMA = 1 + +DATASETS = { + "tau": {"name": "τ-bench", "agent": "Customer-service agents (GPT-4o, Claude 3.5 Sonnet) in airline and retail " + "simulations, acting through tools for a simulated user"}, + "swe": {"name": "SWE-agent", "agent": "SWE-agent on Llama models (8B, 70B, 405B) fixing GitHub issues; a run fails " + "when its patch does not pass the tests"}, + "trail": {"name": "TRAIL", "agent": "smolagents runs on GAIA and SWE-bench, with failures annotated by hand"}, +} + + +def _row(ds: str, part: str, s: dict[str, Any], meta: dict[str, Any]) -> dict[str, Any] | None: + budget = meta["config"].get("review_budget", 0.05) + at = _review_at(s["default"], COST_FAILURE_RATE) + if not at or s.get("cost_per_run_usd") is None: + return None + reviews = round(COST_TRACES * budget) + random_found = reviews * COST_FAILURE_RATE + found = reviews * at["precision"] + jev = s["cost_per_run_usd"] * COST_TRACES + variant = meta["variant"] + gain = at.get("vs_baseline") + return { + "dataset": ds, + "slice": part, + "setup": variant.removesuffix("+rules"), + "rules": variant.endswith("+rules"), + "runs": s["runs_scored"], + "failed": s["failed"], + "rough": at["reviewed"] < FEW_REVIEWED, + "precision": at["precision"], + "precision_ci95": at["ci95"], + "found": found, + "found_ci95": [reviews * p for p in at["ci95"]], + "multiple": found / random_found, + "reviews_per_failure": reviews / found, + "random_reviews_for_same": found / COST_FAILURE_RATE, + "jev_usd": jev, + "jev_usd_per_extra_failure": jev / (found - random_found) if found > random_found else None, + "input_tokens_per_run": s["tokens"] / s["runs_scored"], + "roc_auc": s["default"].get("roc_auc"), + "vs_baseline": gain and {"against": "jev_alone" if variant.endswith("+rules") else "answer_wrong_alone", + "gain": gain["gain"], "ci95": gain["ci95"]}, + "run": meta["run_id"], + } + + +def build(runs: list[Path]) -> dict[str, Any]: + rows: list[dict[str, Any]] = [] + first: dict[str, Any] = {} + for folder in runs: + summary = json.loads((folder / "summary.json").read_text()) + meta = summary["meta"] + first = first or meta + for ds, s in summary["datasets"].items(): + for part, group in (("all", s), ("detectable", summary.get("detectable", {}).get(ds))): + if group and (r := _row(ds, part, group, meta)): + rows.append(r) + budget = first["config"].get("review_budget", 0.05) + reviews = round(COST_TRACES * budget) + return { + "schema": SCHEMA, + "date": f"{first['run_id'][:4]}-{first['run_id'][4:6]}-{first['run_id'][6:8]}", + "jev_model": first["jev_models"][0], + "source": first["source"]["base_sha"], + "usd_per_million_input_tokens": first["config"]["cost"]["usd_per_million_input_tokens"], + "scenario": {"traces": COST_TRACES, "failure_rate": COST_FAILURE_RATE, "review_budget": budget, + "reviews": reviews, "random_found": reviews * COST_FAILURE_RATE, + "random_reviews_per_failure": 1 / COST_FAILURE_RATE}, + "datasets": DATASETS, + "results": rows, + } + + +def write(runs: list[Path], out: Path) -> dict[str, Any]: + data = build(runs) + out.parent.mkdir(parents=True, exist_ok=True) + out.write_text(json.dumps(data, indent=2, ensure_ascii=False) + "\n") + return data diff --git a/bench/src/tailwise_bench/report.py b/bench/src/tailwise_bench/report.py new file mode 100644 index 0000000..9562a7a --- /dev/null +++ b/bench/src/tailwise_bench/report.py @@ -0,0 +1,386 @@ +"""Writes the run folder: runs.csv, summary.json, plots and report.md.""" + +from __future__ import annotations + +import csv +import json +import math +from collections.abc import Callable +from pathlib import Path +from typing import Any + +from . import plots +from .plots import name + + +def write_csv(path: Path, rows: list[dict[str, Any]]) -> None: + if not rows: + return + fields = list(rows[0]) + for r in rows[1:]: + fields += [k for k in r if k not in fields] + with path.open("w", newline="") as f: + w = csv.DictWriter(f, fieldnames=fields) + w.writeheader() + w.writerows(rows) + + +def _clean(v: Any) -> Any: + """JSON without NaN.""" + if isinstance(v, float) and math.isnan(v): + return None + if isinstance(v, dict): + return {k: _clean(x) for k, x in v.items()} + if isinstance(v, list | tuple): + return [_clean(x) for x in v] + return v + + +def write_json(path: Path, data: Any) -> None: + path.write_text(json.dumps(_clean(data), indent=2) + "\n") + + +def make_plots(summary: dict[str, Any], curves: dict[str, Any], out: Path) -> list[Path]: + out.mkdir(parents=True, exist_ok=True) + ds = summary["datasets"] + made = [plots.budget(ds, out), plots.kept_precision(ds, out), *plots.roc_pr(curves, ds, out), + plots.reliability(ds, out), plots.flags(ds, out), plots.cost(ds, out)] + return [p for p in made if p is not None] + + +def _pct(v: float | None) -> str: + return "n/a" if v is None else f"{v:.0%}" + + +def _num(v: float | None, digits: int = 2) -> str: + return "n/a" if v is None else f"{v:.{digits}f}" + + +def _at(curve: list[dict[str, Any]], budget: float) -> dict[str, Any]: + return next((p for p in curve if abs(p["budget"] - budget) < 1e-9), {}) + + +# Fewer reviewed runs per draw than this, and a review cell is marked as noisy. +FEW_REVIEWED = 20 + + +def _review_at(r: dict[str, Any], rate: float) -> dict[str, Any]: + at: dict[str, Any] = (r.get("review") or {}).get("at_rate", {}).get(str(rate), {}) + return at + + +def _review_cell(r: dict[str, Any], rate: float | None) -> str: + if rate is None: + return _pct((r.get("review") or {}).get("real", {}).get("precision")) + a = _review_at(r, rate) + if not a: + return "n/a" + return _pct(a["precision"]) + ("†" if a["reviewed"] < FEW_REVIEWED else "") + + +def review_section(summary: dict[str, Any], meta: dict[str, Any]) -> list[str]: + """The headline table: of the runs a team reviews, ranked by Tailwise, + the share that failed.""" + cfg = meta["config"] + budget, rates = cfg.get("review_budget", 0.05), sorted(cfg["reference_failure_rates"], reverse=True) + lines = [f"## Review budget: the share of reviewed runs that failed", "", + f"A team reviews the {budget:.0%} of runs Tailwise ranks highest (by severity). The table shows what share " + "of those runs failed. Random review finds failures at the failure rate; the best any ranking can do is " + f"every reviewed run a failure, or failure rate ÷ {budget:.0%} when failures are fewer than that.", "", + "The datasets fail far more often than production agents, so each column after the first drops failed " + f"runs at random until that share fail ({meta['config']['random_seeds']} draws, mean). Each draw also " + "resamples the tasks, since a dataset runs each task several times, so the intervals below cover which " + "tasks the dataset happened to contain.", "", + "| Ranking | Own rate | " + " | ".join(f"{r:.0%} failing" for r in rates) + " |", + "| --- | --- | " + " | ".join("---" for _ in rates) + " |", + "| Random review | failure rate | " + " | ".join(_pct(r) for r in rates) + " |", + "| Best possible | 100% | " + " | ".join(_pct(min(1.0, r / budget)) for r in rates) + " |"] + groups = [] + for ds, s in summary["datasets"].items(): + whole = [(f"{name(ds)}", s)] + if ds in summary.get("detectable", {}): + whole.append((f"{name(ds)}, trace-detectable", summary["detectable"][ds])) + for label, g in whole: + groups.append((f"{label}, Tailwise" + (" (Jev + rules)" if "jev_only" in g else ""), g, "default")) + if "jev_only" in g: + groups.append((f"{label}, Jev alone", g, "jev_only")) + groups.append((f"{label}, rules alone", g, "rules_only")) + groups.append((f"{label}, ranked by p_keep (highest answer)", g, "by_p_keep")) + groups.append((f"{label}, answer_wrong only", g, "answer_wrong_only")) + for label, s, rule in groups: + r = s[rule] + # A comparison row that ranks exactly as Tailwise does (one deciding + # question) adds nothing. + if rule != "default" and r.get("review") == s["default"].get("review"): + continue + lines.append(f"| {label} ({_pct(s['failure_rate'])} failed) | {_review_cell(r, None)} | " + + " | ".join(_review_cell(r, rate) for rate in rates) + " |") + lines += ["", f"† Fewer than {FEW_REVIEWED} runs reviewed per draw: a rough estimate.", ""] + lines += gain_table([(lbl, s_) for lbl, s_, rule in groups if rule == "default"], rates) + lines += rules_table(summary) + return lines + + +# The deployment the cost table describes. +COST_TRACES = 10_000 +COST_FAILURE_RATE = 0.1 + + +def cost_section(summary: dict[str, Any], meta: dict[str, Any]) -> list[str]: + """What upgrading from random review to Tailwise costs and buys: per + 10,000 traces at a 10% failure rate, with the headline review budget.""" + budget = meta["config"].get("review_budget", 0.05) + reviews = round(COST_TRACES * budget) + random_found = reviews * COST_FAILURE_RATE + usd_per_m = meta["config"]["cost"]["usd_per_million_input_tokens"] + rows = [] + groups = [] + for ds, s in summary["datasets"].items(): + groups.append((name(ds), s)) + if ds in summary.get("detectable", {}): + groups.append((f"{name(ds)}, trace-detectable", summary["detectable"][ds])) + for label, s in groups: + at = _review_at(s["default"], COST_FAILURE_RATE) + per_run = s.get("cost_per_run_usd") + if not at or per_run is None: + continue + found = reviews * at["precision"] + jev = per_run * COST_TRACES + extra = found - random_found + rough = "†" if at["reviewed"] < FEW_REVIEWED else "" + rows.append(f"| {label} | ${jev:.2f} | {random_found:.0f} → {found:.0f}{rough} | " + f"{reviews / random_found:.0f} → {reviews / found:.1f}{rough} | " + f"{found / COST_FAILURE_RATE:,.0f} | " + + (f"${jev / extra:.3f}" if extra > 0 else "no gain") + " |") + if not rows: + return [] + return ["## Cost: random review against Tailwise", "", + f"Per {COST_TRACES:,} traces with {COST_FAILURE_RATE:.0%} failing, a team reviewing {reviews} of them " + f"({budget:.0%}). Random review costs nothing to choose the traces and finds failures at the failure rate; " + f"Tailwise pays Jev to score every trace (${usd_per_m} per million input tokens; output is free). " + "\"Random reviews for the same failures\" is how many traces random review would need to read to find as many " + "failures as Tailwise's queue does.", "", + "| Dataset | Jev cost | Failures found: random → Tailwise | Reviews per failure found | " + "Random reviews for the same failures | Jev cost per extra failure found |", + "| --- | --- | --- | --- | --- | --- |", *rows, "", + "Reviewer time is the larger cost: multiply \"reviews per failure found\" by what a review costs your " + "team. † Rough: few successful runs in the dataset.", ""] + + +def rules_table(summary: dict[str, Any]) -> list[str]: + """What each heuristic rule fired on.""" + rows = [] + for ds, s in summary["datasets"].items(): + for rule, f in (s.get("rules") or {}).items(): + rows.append(f"| {name(ds)} | `{rule}` | {_pct(f['recall'])} | {_pct(f['false_positive_rate'])} | " + f"{_pct(f['precision'])} |") + if not rows: + return [] + return ["### The rules", "", + "How often each rule fired on failed and on successful runs, and the share of runs it fired on that " + "failed. A keep rule's runs are kept without asking Jev.", "", + "| Dataset | Rule | Fired on failed | Fired on successful | Fired runs that failed |", + "| --- | --- | --- | --- | --- |", *rows, ""] + + +def _points(v: float) -> str: + return f"{round(v * 100) + 0:+d}" # + 0 turns -0 into 0 + + +def gain_table(groups: list[tuple[str, dict[str, Any]]], rates: list[float]) -> list[str]: + """Tailwise's ranking against answer_wrong alone, on the same resampled + runs: the points it adds, with the 95% interval.""" + rows = [] + for label, s in groups: + cells = [(_review_at(s["default"], r).get("vs_baseline")) for r in rates] + if not any(cells): + continue + rows.append(f"| {label} | " + " | ".join( + "n/a" if not c else f"{_points(c['gain'])} ({_points(c['ci95'][0])} to {_points(c['ci95'][1])})" + for c in cells) + " |") + if not rows: + return [] + against = "Jev alone" if any("jev_only" in s for _, s in groups) else "`answer_wrong` alone" + return [f"### Against {against.replace('`', '')}", "", + f"Points of reviewed runs that failed, Tailwise's ranking minus {against} on the same draws, " + "with the 95% interval. An interval that includes 0 means the gain is not shown.", "", + "| Ranking | " + " | ".join(f"{r:.0%} failing" for r in rates) + " |", + "| --- | " + " | ".join("---" for _ in rates) + " |", *rows, ""] + + +def dataset_sentence(ds: str, s: dict[str, Any]) -> str: + t = s["default"]["at_threshold"] + rev = s["default"].get("review") or {} + budget = rev.get("budget", 0.05) + at10 = _review_at(s["default"], 0.1) + headline = (f"Reviewing the {budget:.0%} of {name(ds)} runs Tailwise ranks highest, " + f"{_pct(rev.get('real', {}).get('precision'))} failed (failure rate {_pct(s['failure_rate'])})" + + (f"; were 10% of runs failing, {_pct(at10.get('precision'))} would." if at10 else ".") + " ") + p = _at(s["default"]["budget_curve"], 0.10) + lo, hi = p.get("random_precision_ci95") or [None, None] + return headline + ( + f"On {name(ds)} ({s['runs']} runs, {_pct(s['failure_rate'])} failed), Tailwise kept {_pct(t['kept_share'])} " + f"of runs; {_pct(t['precision'])} of those had failed, against {_pct(t['base_rate'])} for runs kept at random " + f"({_num(t['lift'])}× lift), and it caught {_pct(t['recall'])} of all failures while keeping " + f"{_pct(t['false_positive_rate'])} of successful runs. Reviewing only the 10% of runs it scores highest, " + f"{_pct(p.get('tailwise_precision'))} are failures (random: {_pct(p.get('random_precision_mean'))}, " + f"95% interval {_pct(lo)}–{_pct(hi)}). ROC-AUC is {_num(s['default']['roc_auc'], 3)}." + ) + + +def detectable_sentence(ds: str, s: dict[str, Any], whole: dict[str, Any]) -> str: + at10 = _review_at(s["default"], 0.1) + lo, hi = at10.get("ci95") or [None, None] + return (f"On the {s['runs']} {name(ds)} runs whose label the trace can show ({_pct(s['failure_rate'])} failed), " + f"{_pct(at10.get('precision'))} of reviewed runs failed at a 10% failure rate " + f"(95% interval {_pct(lo)}–{_pct(hi)}), and " + f"ROC-AUC is {_num(s['default']['roc_auc'], 3)} (all runs: {_num(whole['default']['roc_auc'], 3)}).") + + +def summary_paragraph(summary: dict[str, Any], meta: dict[str, Any]) -> str: + parts = [] + for ds, s in summary["datasets"].items(): + if s["runs"]: + parts.append(dataset_sentence(ds, s)) + if ds in summary.get("detectable", {}): + parts.append(detectable_sentence(ds, summary["detectable"][ds], s)) + parts.append(f"The scores took {meta['total_tokens']:,} input tokens (${meta['total_cost_usd']:.4f}), " + f"{meta['token_source']}; this run spent ${meta['spent_this_run_usd']:.4f}, the rest from the cache.") + if "tokens_with_rules" in meta: + parts.append(f"With the rules, keep rules decide some runs without Jev: " + f"{meta['tokens_with_rules']:,} tokens ({meta['tokens_with_rules'] / max(1, meta['total_tokens']):.0%}).") + if meta.get("limit"): + parts.append(f"This is a smoke run on at most {meta['limit']} runs per dataset: the numbers are not meaningful.") + return " ".join(parts) + + +def _rule_row(label: str, s: dict[str, Any], rule: str) -> str: + r = s[rule] + t = r["at_threshold"] + p = _at(r["budget_curve"], 0.10) + return (f"| {label} | {s['runs']} | {s['runs_failed_to_score']} | {_pct(s['failure_rate'])} | {_pct(t['kept_share'])} | {_pct(t['precision'])} | " + f"{_num(t['lift'])}× | {_pct(t['recall'])} | {_pct(t['false_positive_rate'])} | " + f"{_pct(p.get('tailwise_precision'))} vs {_pct(p.get('random_precision_mean'))} | {_num(r['roc_auc'], 3)} | " + f"{_num(r['calibration']['ece'], 3)} |") + + +HEADER = ["| Runs | Runs | Not scorable | Failed | Kept | Kept that failed | Lift | Failures caught | Successes kept " + "| Top 10%: failed vs random | ROC-AUC | ECE |", + "| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |"] + + +COMPARED: list[tuple[str, Callable[[dict[str, Any]], float | None], str]] = [ + *[(f"Reviewed runs that failed, {r:.0%} failing", (lambda r: lambda x: _review_at(x, r).get("precision"))(r), "pct") + for r in (0.2, 0.1, 0.05)], + ("Kept", lambda r: r["at_threshold"]["kept_share"], "pct"), + ("Kept that failed (precision)", lambda r: r["at_threshold"]["precision"], "pct"), + ("Lift", lambda r: r["at_threshold"]["lift"], "x"), + ("Failures caught (recall)", lambda r: r["at_threshold"]["recall"], "pct"), + ("Successes kept", lambda r: r["at_threshold"]["false_positive_rate"], "pct"), + ("Top 10%: share that failed", lambda r: _at(r["budget_curve"], 0.10).get("tailwise_precision"), "pct"), + ("Top 10%: failures found", lambda r: _at(r["budget_curve"], 0.10).get("tailwise_recall"), "pct"), + ("ROC-AUC", lambda r: r["roc_auc"], "num"), + ("PR-AUC", lambda r: r["pr_auc"], "num"), + ("ECE (lower is better)", lambda r: r["calibration"]["ece"], "num"), +] + + +def _fmt(v: float | None, kind: str) -> str: + if v is None: + return "n/a" + return {"pct": f"{v:.0%}", "x": f"{v:.2f}×", "num": f"{v:.3f}"}[kind] + + +def _delta(before: float | None, after: float | None, kind: str) -> str: + if before is None or after is None: + return "" + d = after - before + return f"{d * 100:+.0f} pts" if kind == "pct" else f"{d:+.3f}" if kind == "num" else f"{d:+.2f}×" + + +def comparison(summary: dict[str, Any], baseline: dict[str, Any]) -> list[str]: + """Each dataset's default rule against the same dataset in a baseline run.""" + b_meta = baseline.get("meta", {}) + lines = [f"## Compared with {b_meta.get('run_id', 'the baseline')}", "", + f"The baseline run is {b_meta.get('run_id', '?')} (Tailwise {b_meta.get('tailwise_version', '?')}). " + "Both score the same runs; what changed between them is in the run notes below.", ""] + groups = [(ds, name(ds), s, baseline.get("datasets", {}).get(ds)) for ds, s in summary["datasets"].items()] + groups += [(ds, f"{name(ds)}, trace-detectable runs", s, baseline.get("detectable", {}).get(ds)) + for ds, s in summary.get("detectable", {}).items()] + for ds, title, s, b in groups: + if not b: + continue + lines += [f"### {title}", "", "| Metric | Before | After | Change |", "| --- | --- | --- | --- |"] + for label, get, kind in COMPARED: + before, after = get(b["default"]), get(s["default"]) + lines.append(f"| {label} | {_fmt(before, kind)} | {_fmt(after, kind)} | {_delta(before, after, kind)} |") + lines += ["", "| Check | Fires on failed runs, before → after | Fires on successful runs, before → after |", + "| --- | --- | --- |"] + for q in sorted(set(s["flags"]) | set(b["flags"])): + fb, fa = b["flags"].get(q, {}), s["flags"].get(q, {}) + lines.append(f"| {q} | {_pct(fb.get('recall'))} → {_pct(fa.get('recall'))} | " + f"{_pct(fb.get('false_positive_rate'))} → {_pct(fa.get('false_positive_rate'))} |") + lines.append("") + subs = [k for k in summary["subsets"] if k in baseline.get("subsets", {})] + if subs: + lines += ["### ROC-AUC by subset", "", "| Subset | Before | After | Change |", "| --- | --- | --- | --- |"] + for k in subs: + before, after = baseline["subsets"][k]["default"]["roc_auc"], summary["subsets"][k]["default"]["roc_auc"] + lines.append(f"| {k} | {_fmt(before, 'num')} | {_fmt(after, 'num')} | {_delta(before, after, 'num')} |") + lines.append("") + return lines + + +def write_report(out: Path, summary: dict[str, Any], meta: dict[str, Any], plot_paths: list[Path], + baseline: dict[str, Any] | None = None, notes: str = "") -> None: + lines = [f"# Tailwise benchmark: {meta['run_id']}", "", + "Tailwise's job is to pick, from many agent runs, the ones worth a human's review: the failures. " + "Each run here is scored once, as production would score it, and labelled by whether it actually " + "failed. Random keeping is the baseline: it finds failures at the dataset's failure rate.", "", + summary_paragraph(summary, meta), ""] + lines += review_section(summary, meta) + lines += cost_section(summary, meta) + lines += ["## Run", "", + f"- Tailwise {meta['tailwise_version']} (commit {meta['git_sha']}), Jev model {', '.join(meta['jev_models']) or 'n/a'}", + f"- Questions: {meta['variant']}; threshold {meta['config']['tailwise']['keep_threshold']}, no random floor; " + "runs ranked by severity, the deciding questions' average answer", + f"- {meta['scored']} runs scored, {meta['cached']} from the cache, {meta['failed']} failed to score", ""] + if meta.get("aborted"): + lines += [f"**Stopped early:** {meta['aborted']}", ""] + lines += ["## At Tailwise's threshold", "", HEADER[0].replace("| Runs |", "| Dataset |", 1), HEADER[1]] + for ds, s in summary["datasets"].items(): + lines.append(_rule_row(f"{name(ds)}, Tailwise", s, "default")) + lines.append(_rule_row(f"{name(ds)}, answer_wrong only", s, "answer_wrong_only")) + if ds in summary.get("detectable", {}): + lines.append(_rule_row(f"{name(ds)}, trace-detectable runs", summary["detectable"][ds], "default")) + lines += ["", "\"Not scorable\" runs are ones Jev rejected (for example as too long); they count as dropped, as in " + "production. \"Kept that failed\" is precision; random keeping gets the failure rate, so lift is precision ÷ failure " + "rate. \"Failures caught\" is recall.", ""] + if summary.get("detectable"): + lines += ["\"Trace-detectable runs\" leave out runs whose label no trace-only reviewer could judge: runs on " + "tasks whose logged gold actions disagree with a successful run, and failed runs whose only mistake is " + "choosing a variant the user never described. Judge changes to Tailwise on this row; the full " + "dataset caps what any trace-based detector can reach.", ""] + rates = meta["config"]["reference_failure_rates"] + lines += ["### At production failure rates", "", + "The datasets fail far more often than most production agents. With the same catch rate and " + "false-positive rate, this is the share of kept runs that would be failures:", "", + "| Dataset | " + " | ".join(f"{r:.0%} fail" for r in rates) + " |", + "| --- | " + " | ".join("---" for _ in rates) + " |"] + for ds, s in summary["datasets"].items(): + pa = s["default"]["at_threshold"]["precision_at_failure_rate"] + lines.append(f"| {name(ds)} | " + " | ".join(_pct(pa.get(str(r))) for r in rates) + " |") + lines += ["", "## By subset", "", HEADER[0].replace("| Runs |", "| Subset |", 1), HEADER[1]] + for sub, s in summary["subsets"].items(): + lines.append(_rule_row(sub, s, "default")) + lines.append("") + if baseline: + lines += comparison(summary, baseline) + if notes: + lines += ["## Run notes", "", notes, ""] + if plot_paths: + lines += ["## Plots", ""] + for path in plot_paths: + title = path.stem.replace("_", " ").capitalize() + lines += [f"### {title}", "", f"![{title}](plots/{path.name})", ""] + (out / "report.md").write_text("\n".join(lines)) diff --git a/bench/src/tailwise_bench/rules.py b/bench/src/tailwise_bench/rules.py new file mode 100644 index 0000000..0fa9f0a --- /dev/null +++ b/bench/src/tailwise_bench/rules.py @@ -0,0 +1,45 @@ +"""Tailwise's heuristic rules, run over the requests the harness sends. + +Rules do not change what Jev is asked, so they are not part of the score +cache: bench/rulecheck (Go, Tailwise's own rules and span assembly) checks +each run's request, and analyze combines the verdicts with the cached +answers as internal/pipeline does. A rule set is measured at no cost, on the +same answers as the run without it. +""" + +from __future__ import annotations + +import hashlib +import json +import subprocess +from pathlib import Path +from typing import Any + +from .otel import run_trace +from .types import Run + +Verdicts = dict[str, list[dict[str, Any]]] # by trace_id: every rule's verdict + + +def files(rules_dir: Path, datasets: list[str]) -> dict[str, Path]: + """Each dataset's rule file, .yaml in rules_dir, when it has one.""" + return {ds: rules_dir / f"{ds}.yaml" for ds in datasets if (rules_dir / f"{ds}.yaml").is_file()} + + +def digest(paths: dict[str, Path]) -> dict[str, str]: + return {ds: hashlib.sha256(p.read_bytes()).hexdigest()[:16] for ds, p in sorted(paths.items())} + + +def check(runs: list[Run], paths: dict[str, Path], repo: Path) -> Verdicts: + out: Verdicts = {} + for ds, path in sorted(paths.items()): + lines = "".join(json.dumps({"id": r.trace_id, "request": run_trace(r)[0]}) + "\n" + for r in runs if r.dataset == ds) + proc = subprocess.run(["go", "run", "./bench/rulecheck", "-rules", str(path.resolve())], cwd=repo, + input=lines, capture_output=True, text=True) + if proc.returncode: + raise RuntimeError(f"rulecheck {path}: {proc.stderr.strip()}") + for line in proc.stdout.splitlines(): + v = json.loads(line) + out[v["id"]] = v["rules"] or [] + return out diff --git a/bench/src/tailwise_bench/runner.py b/bench/src/tailwise_bench/runner.py new file mode 100644 index 0000000..fe49b6a --- /dev/null +++ b/bench/src/tailwise_bench/runner.py @@ -0,0 +1,126 @@ +"""Scores each run once, through Tailwise, using the cache first.""" + +from __future__ import annotations + +import asyncio +import hashlib +import json +import sys +from dataclasses import dataclass, field +from typing import Any + +from .budget import BudgetExceeded +from .cache import TRACE, ScoreCache +from .client import ScoringClient +from .config import Config +from .otel import _value, run_trace +from .tokens import Estimator +from .types import Run + +Key = tuple[str, str, str, str, str] + +TAILWISE_MAX_STATE_CHARS = 40000 # trace.MaxStateChars: Tailwise cuts longer states to fit +# Rough sizes of the action checks beyond the user's request, for estimates: +# an action check's lookups, action and question; completeness questions. +ACTION_CHECK_TOKENS = 900 +COMPLETENESS_TOKENS_PER_ACTION = 70 + + +def key(run: Run, payload: dict[str, Any], cfg: Config) -> Key: + """The settings that change a score, and the exact request: a change to + what the harness sends re-scores the run.""" + sent = hashlib.sha256(json.dumps(payload, sort_keys=True).encode()).hexdigest()[:16] + return (run.dataset, run.trace_id, TRACE, cfg.tailwise.model, f"{cfg.score_hash()}:{sent}") + + +def tokens_for(run: Run, state: str, est: Estimator, cfg: Config) -> int: + """Estimated input tokens of the call: the state as shown, plus, with the + grounded evidence strategy, the tool results it adds (about 4 characters + per token).""" + tokens = est.call(shown(state, cfg)[:TAILWISE_MAX_STATE_CHARS]) + actions = cfg.actions + if actions: + # The action checks: one call per action call (its request, lookups and + # neighbours), and one for completeness (request, actions, answer). + question = est.count(run.question) + tokens += sum(question + ACTION_CHECK_TOKENS for c in run.tool_calls if c.name in actions) + turns = sum(est.count(text[:600] if role == "assistant" else text) for role, text in run.messages) + tokens += (turns or question) + est.count(run.final_answer) + COMPLETENESS_TOKENS_PER_ACTION * len(actions) + if cfg.tailwise.evidence == "grounded": + outputs = (sum(len(_value(s, "output.value")) for s in run.otlp_spans + if _value(s, "openinference.span.kind") == "TOOL") + if run.otlp_spans else sum(len(c.output) for c in run.tool_calls)) + tokens += min(cfg.tailwise.evidence_budget, outputs) // 4 + return tokens + + +def shown(state: str, cfg: Config) -> str: + """The state as Tailwise shows it: without the system prompt unless the + scorer is set to show it.""" + if cfg.tailwise.system_prompt or not state.startswith("SYSTEM PROMPT:\n"): + return state + return state[state.index("USER QUESTION:\n"):] + + +@dataclass +class RunResult: + results: dict[str, dict[str, Any]] = field(default_factory=dict) # by trace_id + errors: dict[str, str] = field(default_factory=dict) + cached: int = 0 + scored: int = 0 + aborted: str = "" + + +def estimate(runs: list[Run], est: Estimator, cache: ScoreCache, cfg: Config) -> dict[str, dict[str, int]]: + """Projected calls and input tokens per dataset and subset; cached runs cost nothing.""" + out: dict[str, dict[str, int]] = {} + for run in runs: + row = out.setdefault(f"{run.dataset}/{run.subset}", {"calls": 0, "tokens": 0, "cached": 0}) + payload, state = run_trace(run) + if cache.get(key(run, payload, cfg)) is not None: + row["cached"] += 1 + continue + row["calls"] += 1 + row["tokens"] += tokens_for(run, state, est, cfg) + return out + + +async def run(runs: list[Run], client: ScoringClient, cache: ScoreCache, est: Estimator, cfg: Config) -> RunResult: + out = RunResult() + queue: asyncio.Queue[Run] = asyncio.Queue() + for r in runs: + hit = cache.get(key(r, run_trace(r)[0], cfg)) + if hit is not None: + out.results[r.trace_id] = {**hit, "cached": True} + out.cached += 1 + else: + queue.put_nowait(r) + todo, done = queue.qsize(), 0 + + async def worker() -> None: + nonlocal done + while not out.aborted: + try: + r = queue.get_nowait() + except asyncio.QueueEmpty: + return + payload, state = run_trace(r) + tokens = tokens_for(r, state, est, cfg) + try: + o = await client.score(payload, tokens) + except BudgetExceeded as e: + out.aborted = str(e) + return + if o.result is None: + out.errors[r.trace_id] = o.error + else: + result = {**o.result, "estimated_tokens": tokens} + cache.put(key(r, payload, cfg), result) + out.results[r.trace_id] = {**result, "cached": False} + out.scored += 1 + done += 1 + if done % 100 == 0 or done == todo: + print(f" scored {done}/{todo} (spent ${client.spent_usd:.4f})", file=sys.stderr) + + await asyncio.gather(*(worker() for _ in range(max(1, cfg.client.concurrency)))) + return out diff --git a/bench/src/tailwise_bench/service.py b/bench/src/tailwise_bench/service.py new file mode 100644 index 0000000..7a8da7a --- /dev/null +++ b/bench/src/tailwise_bench/service.py @@ -0,0 +1,103 @@ +"""Runs the Tailwise binary for a benchmark run. + +The generated config keeps every decision deterministic and local: no random +floor, no exporters, the OTLP receivers off (the bench scores through the +synchronous OTLP endpoint), and the API on loopback. The Jev key comes from +envchain or the environment and is never written or logged here. +""" + +from __future__ import annotations + +import os +import shutil +import subprocess +import time +from pathlib import Path +from types import TracebackType +from typing import IO +from urllib.parse import urlparse + +import httpx +import yaml + +from .config import Config + + +def version(binary: Path) -> str: + out = subprocess.run([str(binary), "-version"], capture_output=True, text=True, timeout=10) + return out.stdout.strip().removeprefix("tailwise ").strip() or out.stderr.strip() + + +def write_config(cfg: Config, path: Path) -> None: + url = urlparse(cfg.tailwise.url) + scorer: dict[str, object] = {"model": cfg.tailwise.model, "system_prompt": cfg.tailwise.system_prompt, + "evidence": cfg.tailwise.evidence, "evidence_budget": cfg.tailwise.evidence_budget} + if cfg.tailwise.manifest: + scorer["manifest"] = str(cfg.resolve(cfg.tailwise.manifest)) + doc = { + "server": {"addr": f"{url.hostname}:{url.port or 80}"}, + "otlp": {"grpc_addr": "", "http_addr": ""}, + "scorer": scorer, + "policy": {"keep_threshold": cfg.tailwise.keep_threshold, "random_floor": 0}, + "exporters": [], + } + path.write_text("# Generated by the benchmark harness for one run.\n" + yaml.safe_dump(doc, sort_keys=False)) + + +class Tailwise: + """A Tailwise process, started and stopped with the run (unless + tailwise.start is false, when an already running one is used).""" + + def __init__(self, cfg: Config, run_dir: Path) -> None: + self.cfg = cfg + self.run_dir = run_dir + self.binary = cfg.resolve(cfg.tailwise.binary) + self._proc: subprocess.Popen[bytes] | None = None + self._log: IO[bytes] | None = None + + def __enter__(self) -> Tailwise: + if self.cfg.tailwise.start: + self._start() + self._wait_ready() + return self + + def __exit__(self, t: type[BaseException] | None, e: BaseException | None, tb: TracebackType | None) -> None: + if self._proc is not None: + self._proc.terminate() + try: + self._proc.wait(timeout=30) + except subprocess.TimeoutExpired: + self._proc.kill() + if self._log is not None: + self._log.close() + + def _start(self) -> None: + if not self.binary.exists(): + raise SystemExit(f"{self.binary} not found: run `make build` in the repository root") + conf = self.run_dir / "tailwise.yaml" + write_config(self.cfg, conf) + cmd = [str(self.binary), "-config", str(conf)] + if self.cfg.tailwise.envchain: + if not shutil.which("envchain"): + raise SystemExit("tailwise.envchain is set but envchain is not installed") + cmd = ["envchain", self.cfg.tailwise.envchain, *cmd] + elif not os.environ.get("JEV_API_KEY"): + raise SystemExit("JEV_API_KEY is not set (or set tailwise.envchain)") + self._log = (self.run_dir / "tailwise.log").open("wb") + self._proc = subprocess.Popen(cmd, stdout=self._log, stderr=subprocess.STDOUT) + + def _wait_ready(self, timeout: float = 60) -> None: + deadline = time.monotonic() + timeout + last = "" + while time.monotonic() < deadline: + if self._proc is not None and self._proc.poll() is not None: + raise SystemExit(f"Tailwise exited with {self._proc.returncode}; see {self.run_dir / 'tailwise.log'}") + try: + r = httpx.get(self.cfg.tailwise.url + "/readyz", timeout=5) + if r.status_code == 200: + return + last = r.text + except httpx.TransportError as e: + last = str(e) + time.sleep(0.5) + raise SystemExit(f"Tailwise at {self.cfg.tailwise.url} was not ready after {timeout:.0f}s: {last}") diff --git a/bench/src/tailwise_bench/source.py b/bench/src/tailwise_bench/source.py new file mode 100644 index 0000000..02c3c39 --- /dev/null +++ b/bench/src/tailwise_bench/source.py @@ -0,0 +1,47 @@ +"""The exact source a run used, for a tree with uncommitted changes. + +A run records the commit it started from and a patch of everything not +committed: changes to tracked files and new files git does not ignore. +Applying the patch to that commit (`git apply source.patch`) rebuilds the +tree, and the patch's SHA-256 identifies it. +""" + +from __future__ import annotations + +import hashlib +import subprocess +from pathlib import Path +from typing import Any + + +def _git(repo: Path, *args: str, check: bool = True) -> str: + out = subprocess.run(["git", *args], cwd=repo, capture_output=True, text=True) + if check and out.returncode != 0: + raise RuntimeError(f"git {' '.join(args)}: {out.stderr.strip()}") + return out.stdout + + +def patch(repo: Path) -> str: + """Uncommitted changes to tracked files, then every untracked, unignored + file as a new-file diff. Nothing is staged or written to the repository.""" + parts = [_git(repo, "diff", "--binary", "HEAD")] + for name in sorted(_git(repo, "ls-files", "--others", "--exclude-standard", "-z").split("\0")): + if name: + # --no-index exits 1 when the files differ, which they always do here. + parts.append(_git(repo, "diff", "--binary", "--no-index", "--", "/dev/null", name, check=False)) + return "".join(parts) + + +def record(repo: Path, out_dir: Path) -> dict[str, Any]: + """Writes out_dir/source.patch and returns what identifies the source.""" + text = patch(repo) + path = out_dir / "source.patch" + path.write_text(text) + return { + "base_sha": _git(repo, "rev-parse", "HEAD").strip(), + "dirty": bool(text), + "patch_file": path.name, + "patch_sha256": hashlib.sha256(text.encode()).hexdigest(), + "patch_bytes": len(text.encode()), + "rebuild": "git checkout && git apply source.patch", + } diff --git a/bench/src/tailwise_bench/tokens.py b/bench/src/tailwise_bench/tokens.py new file mode 100644 index 0000000..cc20970 --- /dev/null +++ b/bench/src/tailwise_bench/tokens.py @@ -0,0 +1,31 @@ +"""Input-token estimates, for --dry-run and for reserving budget before a call. + +TypeSafe does not publish Jev's tokenizer, so these use tiktoken and are +labelled estimates. Real runs bill from the `input_tokens` Jev reports. +""" + +from __future__ import annotations + +import json +from functools import lru_cache +from pathlib import Path + +import tiktoken + +# The questions Tailwise sends with every call, unless the bench names a manifest. +BUILTIN_MANIFEST = Path(__file__).resolve().parents[3] / "internal" / "config" / "builtin_manifest.json" + + +class Estimator: + def __init__(self, encoding: str, manifest_text: str = "") -> None: + self._enc = tiktoken.get_encoding(encoding) + manifest = json.loads(manifest_text or BUILTIN_MANIFEST.read_text()) + # Jev reads the questions as well as the state. + self.questions_tokens = self.count(json.dumps(manifest.get("questions", manifest))) + + @lru_cache(maxsize=65536) + def count(self, text: str) -> int: + return len(self._enc.encode(text, disallowed_special=())) + + def call(self, state: str) -> int: + return self.count(state) + self.questions_tokens diff --git a/bench/src/tailwise_bench/types.py b/bench/src/tailwise_bench/types.py new file mode 100644 index 0000000..9c2131e --- /dev/null +++ b/bench/src/tailwise_bench/types.py @@ -0,0 +1,36 @@ +"""The run every loader produces and every later stage reads.""" + +from __future__ import annotations + +from dataclasses import dataclass, field +from typing import Any + + +@dataclass +class ToolCall: + name: str + args: Any # a dict when the agent passed structured arguments, else a string + output: str + error: bool = False + + +@dataclass +class Run: + """One agent run: what Tailwise sees, and whether it actually failed.""" + + dataset: str # tau, swe, trail + subset: str # e.g. retail/gpt-4o, swe-agent-llama-70b, gaia + run_id: str # the dataset's own id for the run + trace_id: str # 32 hex characters + failed: bool # the label: the run did not do its job + system_prompt: str = "" + question: str = "" + tool_calls: list[ToolCall] = field(default_factory=list) + final_answer: str = "" + # The user's and agent's turns, (role, text), when the dataset has them: + # sent as the history of the agent's last LLM call. + messages: list[tuple[str, str]] = field(default_factory=list) + agent: str = "" + # TRAIL only: the run's own spans as OTLP/JSON, sent instead of building them. + otlp_spans: list[dict[str, Any]] | None = None + meta: dict[str, Any] = field(default_factory=dict) diff --git a/bench/tests/test_audit.py b/bench/tests/test_audit.py new file mode 100644 index 0000000..8f77a12 --- /dev/null +++ b/bench/tests/test_audit.py @@ -0,0 +1,140 @@ +"""The investigation's plain-code checks, on invented τ-bench-shaped runs.""" + +import json + +from tailwise_bench.investigate import audit + + +def tc(i, name, args): + return {"id": f"c{i}", "type": "function", "function": {"name": name, "arguments": json.dumps(args)}} + + +TRAJ = [ + {"role": "system", "content": "# Policy"}, + {"role": "user", "content": "Exchange the keyboard in order #W100 for the size L one."}, + {"role": "assistant", "content": None, "tool_calls": [tc(0, "get_order_details", {"order_id": "#W100"})]}, + {"role": "tool", "tool_call_id": "c0", "name": "get_order_details", + "content": json.dumps({"order_id": "#W100", "items": [{"item_id": "1111111"}], "payment": "credit_card_42"})}, + {"role": "assistant", "content": None, "tool_calls": [tc(1, "get_product_details", {"product_id": "77"})]}, + {"role": "tool", "tool_call_id": "c1", "name": "get_product_details", + "content": json.dumps({"variants": {"2222222": {"size": "M"}, "3333333": {"size": "L"}}})}, + {"role": "assistant", "content": None, "tool_calls": [tc(2, "exchange_delivered_order_items", { + "order_id": "#W100", "item_ids": ["1111111"], "new_item_ids": ["2222222"], "payment_method_id": "credit_card_42"})]}, + {"role": "tool", "tool_call_id": "c2", "name": "exchange_delivered_order_items", "content": "{\"status\": \"ok\"}"}, + {"role": "assistant", "content": None, "tool_calls": [tc(3, "cancel_pending_order", {"order_id": "#W999", "reason": "no longer needed"})]}, + {"role": "tool", "tool_call_id": "c3", "name": "cancel_pending_order", "content": "Error: not found"}, +] + + +def test_only_writes_are_checked(): + assert audit.is_write("exchange_delivered_order_items") and audit.is_write("book_reservation") + assert not audit.is_write("get_user_details") and not audit.is_write("transfer_to_human_agents") + + +def test_kinds(): + assert audit.kind_of("order_id", "#W100") == "id" + assert audit.kind_of("reason", "no longer needed") == "skip" + assert audit.kind_of("cabin", "economy") == "enum" + assert audit.kind_of("flights[0].date", "2024-05-20") == "date" + assert audit.kind_of("passengers[0].first_name", "Yusuf") == "text" + assert audit.kind_of("total_baggages", 2) == "count" + assert audit.kind_of("payment_methods[0].amount", 348.5) == "amount" + + +def test_grounding_traces_each_argument_and_flags_the_made_up_one(): + writes = audit.ground(TRAJ, "# Policy") + exchange, cancel = writes + src = {lf.path: lf.source for lf in exchange.leaves} + assert src["order_id"] == "user" + assert src["item_ids[0]"] == "tool:0" and src["payment_method_id"] == "tool:0" + assert src["new_item_ids[0]"] == "tool:1" + assert exchange.ungrounded == [] + assert exchange.calls_between == 1 # the order lookup (call 0) was two calls back + assert [lf.path for lf in cancel.ungrounded] == ["order_id"] # #W999 was never seen + + +def test_ids_match_whole_tokens_only(): + assert audit.found_in("id", "#W100", "order #w100 is pending", natural=False) + assert not audit.found_in("id", "#W10", "order #w100 is pending", natural=False) + assert audit.found_in("date", "2024-05-20", "i want to fly on may 20", natural=True) + assert not audit.found_in("date", "2024-05-20", "i want to fly on may 20", natural=False) + assert audit.found_in("amount", 1200.0, "the total is 1,200.00", natural=False) + + +def test_gold_labels_a_grounded_variant_mixup(): + writes = audit.ground(TRAJ, "# Policy") + gold = [{"name": "get_order_details", "kwargs": {"order_id": "#W100"}}, + {"name": "exchange_delivered_order_items", "kwargs": { + "order_id": "#W100", "item_ids": ["1111111"], "new_item_ids": ["3333333"], + "payment_method_id": "credit_card_42"}}] + res = audit.compare_gold(writes, gold) + by_tool = {r["tool"]: r for r in res.rows} + ex = by_tool["exchange_delivered_order_items"] + assert ex["category"] == "wrong_args" and ex["grounded_wrong"] and ex["variant_mixup"] + assert "new_item_ids" in ex["diff"] + assert by_tool["cancel_pending_order"]["category"] == "extra" + + +def test_gold_exact_match_and_missed(): + writes = audit.ground(TRAJ[:8], "# Policy") + gold = [{"name": "exchange_delivered_order_items", "kwargs": { + "order_id": "#W100", "item_ids": ["1111111"], "new_item_ids": ["2222222"], "payment_method_id": "credit_card_42"}}, + {"name": "cancel_pending_order", "kwargs": {"order_id": "#W100", "reason": "ordered by mistake"}}] + cats = sorted(r["category"] for r in audit.compare_gold(writes, gold).rows) + assert cats == ["correct", "missed"] + + +def test_visibility_reads_evidence_and_inline_results(): + outputs = ["yusuf_9620", json.dumps({"a": 1}), json.dumps({"b": 2})] + first = ("USER QUESTION:\nq\n\nTOOL CALLS:\n1. find() -> ok\n2. get() -> ok\n3. get() -> ok\n\n" + "EVIDENCE:\n[1] yusuf_9620\n[2] {\"a\": 1}\n\nFINAL ANSWER:\na") + assert audit.visibility(first, outputs) == ["full", "full", "none"] + grounded = ("USER QUESTION:\nq\n\nTOOL CALLS:\n1. find() -> ok: yusuf_9620\n2. get() -> ok\n" + "3. get() -> ok:\n {\"b\": 2}\n\nEVIDENCE:\n(none)\n\nFINAL ANSWER:\na") + assert audit.visibility(grounded, outputs) == ["full", "none", "full"] + trimmed = "USER QUESTION:\nq\n\nTOOL CALLS:\n1. find() -> ok: yusuf [trimmed]\n\nEVIDENCE:\n(none)\n\nFINAL ANSWER:\na" + assert audit.visibility(trimmed, outputs[:1]) == ["partial"] + + +def test_mixup_detectable_only_when_the_user_named_the_difference(): + # Exact phrases only: a paraphrase ("not water resistant") is not matched. + traj = [ + {"role": "user", "content": "I want the blue one, the not resistant kind."}, + {"role": "tool", "content": json.dumps({"variants": { + "1": {"options": {"color": "blue", "water resistance": "IPX4"}}, + "2": {"options": {"color": "blue", "water resistance": "not resistant"}}}})}, + ] + assert audit.mixup_detectable(traj, chosen=["1"], gold=["2"]) is True + traj[0]["content"] = "I want the blue one." + assert audit.mixup_detectable(traj, chosen=["1"], gold=["2"]) is False + assert audit.mixup_detectable(traj, chosen=["1"], gold=["9"]) is None + + +def _mixup_run(task_id, reward, user, new_item, gold_item): + """A run exchanging item 1 for new_item, where the gold wants gold_item.""" + traj = [ + {"role": "system", "content": "# Policy"}, + {"role": "user", "content": f"Exchange item 1 in order #W1. {user}"}, + {"role": "assistant", "content": None, "tool_calls": [tc(0, "get_product_details", {"product_id": "7"})]}, + {"role": "tool", "tool_call_id": "c0", "content": json.dumps({"variants": { + "2": {"options": {"size": "M"}}, "3": {"options": {"size": "L"}}}})}, + {"role": "assistant", "content": None, "tool_calls": [tc(1, "exchange_delivered_order_items", { + "order_id": "#W1", "item_ids": ["1"], "new_item_ids": [new_item]})]}, + {"role": "tool", "tool_call_id": "c1", "content": "{}"}, + ] + gold = [{"name": "exchange_delivered_order_items", + "kwargs": {"order_id": "#W1", "item_ids": ["1"], "new_item_ids": [gold_item]}}] + return {"task_id": task_id, "reward": reward, "traj": traj, "info": {"task": {"actions": gold}}} + + +def test_trace_detectable_leaves_out_hidden_mixups_and_inconsistent_tasks(): + runs = [ + ("retail", _mixup_run(1, 1.0, "", "3", "3")), # task 1: success, matches gold + ("retail", _mixup_run(1, 0.0, "", "2", "3")), # task 1: mix-up, the user never said L + ("retail", _mixup_run(1, 0.0, "Size L.", "2", "3")), # task 1: mix-up the user ruled out + ("retail", _mixup_run(2, 1.0, "", "2", "3")), # task 2: a success that disagrees with gold + ("retail", _mixup_run(2, 0.0, "Size L.", "2", "3")), + ] + got = [(r["gold_consistent_task"], r["mixup_not_in_trace"], r["trace_detectable"]) + for r in audit.trace_detectable(runs)] + assert got == [(1, 0, 1), (1, 1, 0), (1, 0, 1), (0, 0, 0), (0, 0, 0)] diff --git a/bench/tests/test_budget_cache.py b/bench/tests/test_budget_cache.py new file mode 100644 index 0000000..8538401 --- /dev/null +++ b/bench/tests/test_budget_cache.py @@ -0,0 +1,42 @@ +import pytest + +from tailwise_bench.budget import BudgetExceeded, BudgetGuard +from tailwise_bench.cache import ScoreCache + + +def test_budget_refuses_a_call_that_would_go_over(): + g = BudgetGuard(cap_usd=1.0, usd_per_token=0.01) # 100 tokens + held = g.reserve(60) + with pytest.raises(BudgetExceeded): + g.reserve(60) # 60 in flight + 61 more is over + g.settle(held, 60, 50) + assert g.spent_usd == pytest.approx(0.5) + g.settle(g.reserve(40), 40, 40) # 90 spent + assert g.reserve(5) > 0 # 90 + 6 fits + with pytest.raises(BudgetExceeded): + g.reserve(5) # 90 + 6 in flight + 6 does not + + +def test_budget_learns_when_estimates_run_low(): + g = BudgetGuard(cap_usd=1.0, usd_per_token=0.01) + g.settle(g.reserve(10), 10, 20) # billed twice the estimate + with pytest.raises(BudgetExceeded): + g.reserve(41) # 80 + 41 * 2 > 100 + assert g.reserve(39) > 0 + + +def test_failed_call_releases_its_reservation(): + g = BudgetGuard(cap_usd=1.0, usd_per_token=0.01) + g.settle(g.reserve(90), 90, None) + assert g.spent_usd == 0 and g.reserve(90) > 0 + + +def test_cache_round_trip(tmp_path): + c = ScoreCache(tmp_path / "s.sqlite") + key = ("whowhen", "t", "s", "jev-latest", "abc") + assert c.get(key) is None + c.put(key, {"p_keep": 0.7}) + assert c.get(key) == {"p_keep": 0.7} + assert c.get(("whowhen", "t", "s", "jev-latest", "other")) is None + c.close() + assert ScoreCache(tmp_path / "s.sqlite").get(key) == {"p_keep": 0.7} diff --git a/bench/tests/test_metrics.py b/bench/tests/test_metrics.py new file mode 100644 index 0000000..9622f7a --- /dev/null +++ b/bench/tests/test_metrics.py @@ -0,0 +1,128 @@ +import pytest + +from tailwise_bench import metrics + + +def test_threshold_metrics(): + m = metrics.threshold_metrics([1, 1, 0, 0, 1], [1, 0, 1, 0, 1], rates=(0.1,)) + assert (m["tp"], m["fp"], m["fn"], m["tn"]) == (2, 1, 1, 1) + assert m["precision"] == pytest.approx(2 / 3) + assert m["recall"] == pytest.approx(2 / 3) + assert m["f1"] == pytest.approx(2 / 3) + assert m["kept_share"] == pytest.approx(3 / 5) + assert m["base_rate"] == pytest.approx(3 / 5) + assert m["lift"] == pytest.approx((2 / 3) / (3 / 5)) + assert m["false_positive_rate"] == pytest.approx(1 / 2) + # TPR 2/3, FPR 1/2 at a 10% failure rate: (0.0667) / (0.0667 + 0.45) + assert m["precision_at_failure_rate"]["0.1"] == pytest.approx((2 / 30) / (2 / 30 + 0.45)) + + +def test_threshold_metrics_nothing_kept(): + m = metrics.threshold_metrics([1, 0], [0, 0]) + assert m["precision"] is None and m["recall"] == 0 and m["f1"] is None and m["lift"] is None + + +def test_precision_at_rate(): + assert metrics.precision_at_rate(1.0, 0.0, 0.05) == 1.0 + assert metrics.precision_at_rate(0.5, 0.5, 0.2) == pytest.approx(0.2) # no better than random + assert metrics.precision_at_rate(None, 0.1, 0.1) is None + + +def test_ranking_metrics_perfect_and_single_class(): + m = metrics.ranking_metrics([0, 0, 1, 1], [0.1, 0.2, 0.8, 0.9]) + assert m["roc_auc"] == 1.0 and m["pr_auc"] == 1.0 and m["pr_auc_random"] == 0.5 + assert metrics.ranking_metrics([1, 1], [0.2, 0.3])["roc_auc"] is None + + +def test_budget_k_rounds_up_and_reviews_at_least_one(): + assert metrics.budget_k(0.01, 50) == 1 + assert metrics.budget_k(0.10, 95) == 10 + assert metrics.budget_k(1.0, 7) == 7 + assert metrics.budget_k(0.5, 0) == 0 + + +def test_budget_curve_perfect_ranker_beats_random(): + y = [1] * 10 + [0] * 90 + s = [0.9] * 10 + [0.1] * 90 + curve = metrics.budget_curve(y, s, [0.10, 0.50], seeds=200, seed=0) + p = curve[0] + assert p["k"] == 10 and p["tailwise_recall"] == 1.0 and p["tailwise_precision"] == 1.0 + # Random picks k of n: expected recall k/n, expected precision the failure rate. + assert p["random_recall_mean"] == pytest.approx(0.10, abs=0.03) + assert p["random_precision_mean"] == pytest.approx(0.10, abs=0.03) + lo, hi = p["random_recall_ci95"] + assert lo <= p["random_recall_mean"] <= hi + assert p["lift"] == pytest.approx(1 / p["random_recall_mean"]) + assert p["best_possible_recall"] == 1.0 + assert curve[1]["random_recall_mean"] == pytest.approx(0.50, abs=0.05) + + +def test_budget_curve_best_possible_is_capped_by_the_budget(): + y = [1] * 80 + [0] * 20 + p = metrics.budget_curve(y, list(range(100)), [0.10], seeds=5, seed=0)[0] + assert p["best_possible_recall"] == pytest.approx(10 / 80) + + +def test_budget_curve_is_deterministic(): + y = [1, 0, 1, 0, 0, 1, 0, 0] + s = [0.5] * 8 # all tied: order comes from the seeded tiebreak + assert metrics.budget_curve(y, s, [0.3], 10, seed=3) == metrics.budget_curve(y, s, [0.3], 10, seed=3) + + +def test_rank_order_breaks_ties_randomly_but_reproducibly(): + import numpy as np + s = np.array([0.5, 0.9, 0.5, 0.5]) + a = metrics.rank_order(s, seed=1) + assert a[0] == 1 and sorted(a[1:].tolist()) == [0, 2, 3] + assert (a == metrics.rank_order(s, seed=1)).all() + + +def test_calibration_perfect_and_off(): + assert metrics.calibration([0, 1, 0, 1], [0.0, 1.0, 0.0, 1.0])["ece"] == pytest.approx(0.0) + off = metrics.calibration([0, 0], [0.95, 0.95]) + assert off["ece"] == pytest.approx(0.95) + assert off["bins"][-1]["count"] == 2 and off["bins"][-1]["fraction_failed"] == 0.0 + assert len(off["bins"]) == 10 + + +def test_calibration_score_of_one_lands_in_the_top_bin(): + assert metrics.calibration([1], [1.0])["bins"][-1]["count"] == 1 + + +def test_flag_stats(): + out = metrics.flag_stats([1, 1, 0, 0], [["a", "b"], ["a"], ["a"], []]) + assert out["a"]["flagged"] == 3 and out["a"]["precision"] == pytest.approx(2 / 3) + assert out["a"]["recall"] == 1.0 and out["a"]["false_positive_rate"] == 0.5 + assert out["b"] == {"flagged": 1, "flagged_failed": 1, "precision": 1.0, "recall": 0.5, "false_positive_rate": 0.0} + assert list(out) == ["a", "b"] # most flagged first + + +def test_review_precision_perfect_ranker_and_random(): + y = [1] * 50 + [0] * 950 + perfect = metrics.review_precision(y, [float(v) for v in y], 0.05, (0.10, 0.02), seeds=20, seed=0) + assert perfect["real"]["precision"] == 1.0 # 50 reviewed, all 50 failures + assert perfect["at_rate"]["0.1"]["precision"] == pytest.approx(1.0) # more failures than reviewed + # At 2% failing, the top 5% holds every failure and successes besides. + assert perfect["at_rate"]["0.02"]["precision"] == pytest.approx(0.4, abs=0.02) + assert perfect["at_rate"]["0.02"]["best_possible"] == pytest.approx(0.4) + + flat = metrics.review_precision(y, [0.5] * len(y), 0.05, (0.10,), seeds=200, seed=0) + assert flat["at_rate"]["0.1"]["precision"] == pytest.approx(0.10, abs=0.02) # ties: random review + + +def test_review_precision_single_class_has_no_rates(): + r = metrics.review_precision([0, 0, 0], [0.1, 0.2, 0.3], 0.05, (0.1,), seeds=5, seed=0) + assert r["real"]["precision"] == 0 and r["at_rate"] == {} + + +def test_review_precision_resamples_tasks_and_compares_a_baseline(): + # Two tasks, each run 10 times: the better ranker sees failures first. + y = [1] * 5 + [0] * 5 + [1] * 5 + [0] * 5 + good = [1.0 if v else 0.0 for v in y] + groups = ["a"] * 10 + ["b"] * 10 + out = metrics.review_precision(y, good, 0.1, [0.5], seeds=20, seed=0, groups=groups, + baseline=[0.5] * len(y)) + at = out["at_rate"]["0.5"] + assert at["precision"] == 1.0 + assert at["vs_baseline"]["gain"] > 0 and at["vs_baseline"]["ci95"][0] >= 0 + assert 0 <= at["vs_baseline"]["not_better"] < 1 diff --git a/bench/tests/test_otel.py b/bench/tests/test_otel.py new file mode 100644 index 0000000..6c892f4 --- /dev/null +++ b/bench/tests/test_otel.py @@ -0,0 +1,177 @@ +"""The converters, on tiny invented runs (never real dataset rows).""" + +import json + +from tailwise_bench.loaders import swe, tau, trail +from tailwise_bench.otel import run_trace + +TAU_RUN = { + "task_id": 7, "reward": 0.0, "trial": 1, "info": {}, + "traj": [ + {"role": "system", "content": "# Policy\nOnly refund delivered orders."}, + {"role": "user", "content": "Hi, I want a refund for order #1."}, + {"role": "assistant", "content": None, "tool_calls": [ + {"id": "c1", "type": "function", "function": {"name": "get_order", "arguments": "{\"order_id\": \"#1\"}"}}]}, + {"role": "tool", "tool_call_id": "c1", "name": "get_order", "content": "{\"status\": \"pending\"}"}, + {"role": "assistant", "content": "Let me refund it.", "tool_calls": [ + {"id": "c2", "type": "function", "function": {"name": "refund", "arguments": "{\"order_id\": \"#1\"}"}}]}, + {"role": "tool", "tool_call_id": "c2", "name": "refund", "content": "Error: order is not delivered"}, + {"role": "assistant", "content": "Your refund is done."}, + {"role": "user", "content": "Thanks! ###STOP###"}, + ], +} + +SWE_ROW = { + "instance_id": "acme__widgets-1", "model_name": "swe-agent-llama-70b", "target": True, + "exit_status": "submitted", "generated_patch": "diff --git a/w.py b/w.py\n+fixed\n", + "trajectory": [ + {"role": "system", "text": "You are an autonomous programmer."}, + {"role": "user", "text": "We're solving this issue:\nISSUE:\nWidgets crash on empty input\n\nINSTRUCTIONS:\nFix it."}, + {"role": "ai", "text": "Let's look.\n```\nopen w.py\n```"}, + {"role": "user", "text": "1: def w(x):\n2: return x[0]"}, + {"role": "ai", "text": "Done.\n```\nsubmit\n```"}, + ], +} + + +def spans(req): + return req["resourceSpans"][0]["scopeSpans"][0]["spans"] + + +def attrs(span): + return {a["key"]: next(iter(a["value"].values())) for a in span["attributes"]} + + +def test_tau_run(): + run = tau.parse("gpt-4o-retail", TAU_RUN) + assert run.failed and run.subset == "retail/gpt-4o" and run.run_id == "gpt-4o-retail:7:1" + assert run.question == "Hi, I want a refund for order #1.\n\nThanks!" + assert run.final_answer == "Your refund is done." + assert [c.name for c in run.tool_calls] == ["get_order", "refund"] + assert run.tool_calls[0].args == {"order_id": "#1"} and not run.tool_calls[0].error + assert run.tool_calls[1].error and run.tool_calls[1].output.startswith("Error") + assert "Policy" not in run.question + + +def test_tau_policy_is_sent_as_the_first_llm_calls_system_message(): + run = tau.parse("gpt-4o-retail", TAU_RUN) + assert run.system_prompt.startswith("# Policy") + req, state = run_trace(run) + llm = [s for s in spans(req) if attrs(s)["openinference.span.kind"] == "LLM"] + assert len(llm) == 2 # the first call, with the policy; the last, with the conversation + a = attrs(llm[0]) + assert a["llm.input_messages.0.message.role"] == "system" + assert a["llm.input_messages.0.message.content"] == run.system_prompt + last = attrs(llm[1]) + assert last["llm.input_messages.0.message.role"] == "user" + # The run ends with the user's thanks: it is history, and there is no reply. + assert last[f"llm.input_messages.{len(run.messages) - 1}.message.content"] == "Thanks!" + assert "llm.output_messages.0.message.content" not in last + assert state.startswith("SYSTEM PROMPT:\n# Policy\nOnly refund delivered orders.\n\nUSER QUESTION:") + + +def test_tau_success_is_reward_one(): + assert not tau.parse("gpt-4o-airline", {**TAU_RUN, "reward": 1.0}).failed + + +def test_swe_run(): + run = swe.parse(SWE_ROW, 3) + assert not run.failed and run.subset == "swe-agent-llama-70b" + assert run.question == "Widgets crash on empty input" + assert [(c.name, c.args["command"]) for c in run.tool_calls] == [("open", "open w.py"), ("submit", "submit")] + assert run.tool_calls[0].output.startswith("1: def w(x)") + assert run.final_answer.startswith("diff --git") + + +def test_swe_failed_is_target_false(): + assert swe.parse({**SWE_ROW, "target": False}, 0).failed + + +def test_run_trace_is_one_agent_root_and_a_tool_span_per_call(): + run = tau.parse("gpt-4o-retail", TAU_RUN) + req, state = run_trace(run) + root, *tools = spans(req) + r = attrs(root) + assert r["openinference.span.kind"] == "AGENT" and "parentSpanId" not in root + assert r["input.value"] == run.question and r["output.value"] == "Your refund is done." + tools = [t for t in tools if attrs(t)["openinference.span.kind"] == "TOOL"] + assert len(tools) == 2 and all(t["parentSpanId"] == root["spanId"] for t in tools) + t0, t1 = attrs(tools[0]), attrs(tools[1]) + assert t0["tool.name"] == "get_order" and json.loads(t0["input.value"]) == {"order_id": "#1"} + assert "status" not in tools[0] and tools[1]["status"]["code"] == 2 + assert len(root["traceId"]) == 32 and len({s["spanId"] for s in spans(req)}) == 5 # root, two LLM, two tools + assert "2. refund({\"order_id\": \"#1\"}) -> error" in state + assert "FINAL ANSWER:\nYour refund is done." in state + + +def test_run_trace_ids_are_stable_and_distinct(): + a, b = tau.parse("gpt-4o-retail", TAU_RUN), tau.parse("gpt-4o-retail", {**TAU_RUN, "trial": 2}) + assert a.trace_id == tau.parse("gpt-4o-retail", TAU_RUN).trace_id + assert a.trace_id != b.trace_id + + +TRAIL_TRACE = { + "trace_id": "ab" * 16, + "spans": [{ + "timestamp": "2025-01-01T00:00:00.000000Z", "trace_id": "ab" * 16, "span_id": "0000000000000001", + "parent_span_id": None, "span_name": "main", "duration": "PT2S", "status_code": "Unset", + "span_attributes": {}, + "child_spans": [{ + "timestamp": "2025-01-01T00:00:00.100000Z", "trace_id": "ab" * 16, "span_id": "0000000000000002", + "parent_span_id": "0000000000000001", "span_name": "Agent.run", "duration": "PT1.5S", "status_code": "Ok", + "span_attributes": {"openinference.span.kind": "AGENT", "input.value": "What is 2 + 2?", "output.value": "5"}, + "child_spans": [{ + "timestamp": "2025-01-01T00:00:00.200000Z", "trace_id": "ab" * 16, "span_id": "0000000000000003", + "parent_span_id": "0000000000000002", "span_name": "calc", "duration": "PT0.1S", "status_code": "Error", + "span_attributes": {"openinference.span.kind": "TOOL", "tool.name": "calc", "input.value": "{\"a\": 2}"}, + "child_spans": [], + }], + }], + }], +} + + +def test_trail_failed_means_a_high_impact_error(): + low = {"errors": [{"category": "Formatting Errors", "location": "0000000000000003", "impact": "LOW"}]} + high = {"errors": [*low["errors"], {"category": "Tool-related", "location": "0000000000000003", "impact": "high"}]} + assert not trail.parse("gaia", TRAIL_TRACE, low).failed + run = trail.parse("gaia", TRAIL_TRACE, high) + assert run.failed and run.meta["errors"] == 2 and run.meta["high_impact_errors"] == 1 + + +def test_trail_run_trace_makes_the_agent_span_a_root(): + run = trail.parse("gaia", TRAIL_TRACE, {"errors": []}) + req, state = run_trace(run) + by_id = {s["spanId"]: s for s in spans(req)} + assert "parentSpanId" not in by_id["0000000000000002"] # the AGENT span, re-rooted + assert by_id["0000000000000003"]["parentSpanId"] == "0000000000000002" + start, end = int(by_id["0000000000000002"]["startTimeUnixNano"]), int(by_id["0000000000000002"]["endTimeUnixNano"]) + assert end - start == 1_500_000_000 + assert "USER QUESTION:\nWhat is 2 + 2?" in state and "FINAL ANSWER:\n5" in state + + +def test_tau_results_pair_by_position_when_ids_repeat(): + # claude-3.5-sonnet logs give every call the id "0". + row = {"task_id": 1, "reward": 0.0, "traj": [ + {"role": "user", "content": "hi"}, + {"role": "assistant", "content": None, "tool_calls": [ + {"id": "0", "function": {"name": "find_user", "arguments": "{}"}}]}, + {"role": "tool", "tool_call_id": "0", "name": "find_user", "content": "Error: user not found"}, + {"role": "assistant", "content": None, "tool_calls": [ + {"id": "0", "function": {"name": "find_user_by_zip", "arguments": "{}"}}]}, + {"role": "tool", "tool_call_id": "0", "name": "find_user_by_zip", "content": "mei_8020"}, + {"role": "assistant", "content": "Found you."}, + ]} + run = tau.parse("sonnet-35-new-retail", row) + assert [(c.output, c.error) for c in run.tool_calls] == [("Error: user not found", True), ("mei_8020", False)] + + +def test_tau_results_use_an_unambiguous_id_out_of_order(): + row = {"task_id": 1, "reward": 1.0, "traj": [ + {"role": "assistant", "content": None, "tool_calls": [ + {"id": "a", "function": {"name": "one", "arguments": "{}"}}, + {"id": "b", "function": {"name": "two", "arguments": "{}"}}]}, + {"role": "tool", "tool_call_id": "b", "name": "two", "content": "second"}, + {"role": "tool", "tool_call_id": "a", "name": "one", "content": "first"}, + ]} + assert [c.output for c in tau.parse("gpt-4o-retail", row).tool_calls] == ["first", "second"] diff --git a/bench/tests/test_publish.py b/bench/tests/test_publish.py new file mode 100644 index 0000000..af3e8c3 --- /dev/null +++ b/bench/tests/test_publish.py @@ -0,0 +1,31 @@ +import json +from pathlib import Path + +from tailwise_bench import publish + + +def summary(variant: str) -> dict: + review = {"review": {"at_rate": {"0.1": {"precision": 0.4, "ci95": [0.2, 0.6], "reviewed": 40, + "vs_baseline": {"gain": 0.1, "ci95": [-0.1, 0.3], "not_better": 0.2}}}}, + "roc_auc": 0.7} + group = {"runs_scored": 100, "failed": 40, "tokens": 300_000, "cost_per_run_usd": 0.0002, "default": review} + meta = {"run_id": "20261105T121615Z_abc", "variant": variant, "jev_models": ["jev-1"], + "source": {"base_sha": "abc"}, "config": {"review_budget": 0.05, + "cost": {"usd_per_million_input_tokens": 0.042}}} + return {"meta": meta, "datasets": {"tau": group}, "detectable": {"tau": group}} + + +def test_publish(tmp_path: Path) -> None: + run = tmp_path / "run" + run.mkdir() + (run / "summary.json").write_text(json.dumps(summary("tau-actions+rules"))) + data = publish.write([run], tmp_path / "out" / "benchmark.json") + assert data["date"] == "2026-11-05" + assert data["scenario"]["reviews"] == 500 and data["scenario"]["random_found"] == 50 + assert [(r["slice"], r["setup"], r["rules"]) for r in data["results"]] == [ + ("all", "tau-actions", True), ("detectable", "tau-actions", True)] + r = data["results"][0] + assert r["found"] == 200 and r["found_ci95"] == [100, 300] and r["multiple"] == 4 + assert r["reviews_per_failure"] == 2.5 and r["random_reviews_for_same"] == 2000 + assert abs(r["jev_usd"] - 2) < 1e-9 and abs(r["jev_usd_per_extra_failure"] - 2 / 150) < 1e-9 + assert r["vs_baseline"]["against"] == "jev_alone" and not r["rough"] diff --git a/bench/tests/test_severity.py b/bench/tests/test_severity.py new file mode 100644 index 0000000..f6c30d0 --- /dev/null +++ b/bench/tests/test_severity.py @@ -0,0 +1,41 @@ +import json +from pathlib import Path + +import pytest + +from tailwise_bench import analyze +from tailwise_bench.config import Config + + +def test_severity_averages_the_answered_deciding_questions(): + deciding = ["answer_wrong", "action_wrong", "unhandled_request"] + # action_wrong failed: left out, not counted as 0. + assert analyze.severity({"answer_wrong": 0.6, "unhandled_request": 0.3}, deciding) == pytest.approx(0.45) + assert analyze.severity({"answer_wrong": 0.6, "action_wrong": 0.0, "unhandled_request": 0.3}, deciding) == pytest.approx(0.3) + + +def test_deciding_questions_follow_the_manifest(tmp_path: Path): + assert Config(path=tmp_path / "c.yaml").deciding_questions("answer_wrong") == ["answer_wrong"] + manifest = tmp_path / "m.json" + manifest.write_text(json.dumps({"questions": {}, "decision": {"any_of": ["missing_tool_calls"]}, + "actions": {"cancel": "cancel an order"}})) + cfg = Config(path=tmp_path / "c.yaml") + object.__setattr__(cfg, "tailwise", type(cfg.tailwise)(manifest=str(manifest))) + assert cfg.deciding_questions("answer_wrong") == [ + "answer_wrong", "missing_tool_calls", "action_wrong", "unhandled_request"] + + +def test_apply_rules_mirrors_the_engine(): + deciding = ["answer_wrong"] + answers = {"answer_wrong": 0.2} + row = {"severity": 0.2, "kept": 0, "tokens": 500, "cost": 0.1} + signal = [{"name": "loop", "keep": False, "fired": True}, {"name": "long", "keep": False, "fired": False}] + analyze.apply_rules(row, signal, answers, deciding) + assert row["severity"] == pytest.approx(0.4) and row["kept"] == 0 and row["tokens"] == 500 + assert row["rules_fired"] == "loop" and row["rule_keep"] == 0 and row["severity_rules"] == 0.5 + + row = {"severity": 0.2, "kept": 0, "tokens": 500, "cost": 0.1} + analyze.apply_rules(row, [*signal, {"name": "json", "keep": True, "fired": True}], answers, deciding) + # A keep rule decides without Jev: kept, first in the review order, no tokens. + assert row["severity"] == 1 and row["kept"] == 1 and row["tokens"] == 0 and row["cost"] == 0 + assert row["rules_fired"] == "loop; json" and row["severity_rules"] == 1 diff --git a/bench/uv.lock b/bench/uv.lock new file mode 100644 index 0000000..7b0f7a8 --- /dev/null +++ b/bench/uv.lock @@ -0,0 +1,2039 @@ +version = 1 +revision = 5 +requires-python = ">=3.11" +resolution-markers = [ + "python_full_version >= '3.15'", + "python_full_version >= '3.12' and python_full_version < '3.15'", + "python_full_version < '3.12'", +] + +[[package]] +name = "anyio" +version = "4.15.1" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "idna" }, + { name = "typing-extensions", marker = "python_full_version < '3.15'" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/a9/d2/f4d173e22df740bc37b1db102b386ba719b66e95b0f0d751f556b387e6d2/anyio-4.15.1.tar.gz", hash = "sha256:9f28306018cbd6d329e64a36d58256edff76dd996fe423bc957326e578b82a94", size = 276966, upload-time = "2026-09-05T10:42:39.44Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/12/b8/4bd346e22b28902df4d651910f5242c28d84e4a5c2435ca5c3f797ed7e2e/anyio-4.15.1-py3-none-any.whl", hash = "sha256:6152fdbbf9a77fdec97731721bebf7c4c44f7c29b424b0065826173efc7ed101", size = 132079, upload-time = "2026-09-05T10:42:37.923Z" }, +] + +[[package]] +name = "ast-serialize" +version = "0.13.1" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/21/d5/5dc1ee8a80c3b20a0b63c61b43ba6bc53517faecd1def47bcd752d22268e/ast_serialize-0.13.1.tar.gz", hash = "sha256:401cc84e8888d4d76f2c8d296fa006ed1a5ff993b6aff90dd777c03973efd025", size = 955137, upload-time = "2026-10-17T00:04:17.183Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/f3/a8/fbd136b9f70c59e75f59a8e974c84102976f1201828b4f886480ba956936/ast_serialize-0.13.1-cp314-cp314-pyemscripten_2026_0_wasm32.whl", hash = "sha256:3e129d44b0cffbea24a7f40117ce49e30993c3d119e584d4817ebd6f87246456", size = 897306, upload-time = "2026-10-17T00:02:52.479Z" }, + { url = "https://files.pythonhosted.org/packages/50/3f/2ad51e9823a790e7a70fd781e9595ab3abeace27f3aefcaa1939457efd72/ast_serialize-0.13.1-cp314-cp314t-macosx_10_12_x86_64.whl", hash = "sha256:523d45bab16ddaf1399c04c4fe81bbf180f81e33c2f7bcbe76be56b0f7311ab4", size = 1236082, upload-time = "2026-10-17T00:02:54.82Z" }, + { url = "https://files.pythonhosted.org/packages/c0/5c/8e959db5cc2237f9d4c560a8e2253c98f9746883fb4774651e4cda458901/ast_serialize-0.13.1-cp314-cp314t-macosx_11_0_arm64.whl", hash = "sha256:638333fb8ae72c62753f9e794fda20290883ddc65b0d85e26e08f8bb227567ad", size = 1221242, upload-time = "2026-10-17T00:02:56.395Z" }, + { url = "https://files.pythonhosted.org/packages/d6/05/ab1c56c3b7adcac88764bba609502d6b4989e449b2dd60eb478e1d20e369/ast_serialize-0.13.1-cp314-cp314t-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:cc34c670747d85d4232834665e247009413eaebe47c69eaf60a5c439aaecaed9", size = 1282105, upload-time = "2026-10-17T00:02:58.275Z" }, + { url = "https://files.pythonhosted.org/packages/0d/f4/41981fcd2ddffe0760e5326be9dcbd2eda9478a8430a637e5123bd8fcb26/ast_serialize-0.13.1-cp314-cp314t-manylinux_2_17_armv7l.manylinux2014_armv7l.whl", hash = "sha256:8c8ebd419811c435812d242fb2a173e57dfede198c9ac9a0a584666ebc9b9331", size = 1287905, upload-time = "2026-10-17T00:03:00.068Z" }, + { url = "https://files.pythonhosted.org/packages/6a/43/b783addc5681153707121991b7b47892f7e9c8ba58a0d03b71a3b023ba47/ast_serialize-0.13.1-cp314-cp314t-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl", hash = "sha256:8899c038b1c15ae8b98e379057fb3ef398ceb0b21d1737a8156dd5a0923b7e51", size = 1541806, upload-time = "2026-10-17T00:03:01.503Z" }, + { url = "https://files.pythonhosted.org/packages/47/10/969db095038ee15270f65ad20d1129c5e4da605bf7e93b70ae640f1c8a9f/ast_serialize-0.13.1-cp314-cp314t-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:e14c4de98f965647a74e6eaf37e20207124e45451ccd36e84a6f6ebab45e0372", size = 1309946, upload-time = "2026-10-17T00:03:02.971Z" }, + { url = "https://files.pythonhosted.org/packages/06/dd/7c623d2fb0b868948952d1787e629a43b866f7d4caefd5445e20aefacec4/ast_serialize-0.13.1-cp314-cp314t-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:7f3e8a3eff69af640db985360fef35073fb46956dfc3e5c614cf00920d2746dd", size = 1302773, upload-time = "2026-10-17T00:03:04.378Z" }, + { url = "https://files.pythonhosted.org/packages/d4/c2/4d49469110a19a44fc6269f4fd59679bcf84df068f185f89b73bee4da19b/ast_serialize-0.13.1-cp314-cp314t-manylinux_2_31_riscv64.whl", hash = "sha256:75d7750fe98f6f35a96c43aba8d98a2e623e22c778a0c17ab9ede92eb10f5d82", size = 1299579, upload-time = "2026-10-17T00:03:05.88Z" }, + { url = "https://files.pythonhosted.org/packages/9d/9a/2d381f1f262dda15affc8ad643fba1619eb07e7d9f8bd904deffef183006/ast_serialize-0.13.1-cp314-cp314t-manylinux_2_5_i686.manylinux1_i686.whl", hash = "sha256:43e8ddd63e80fc280c2f790157386f27f9b9f6ccb9c4f8e803b863cbfb085769", size = 1355969, upload-time = "2026-10-17T00:03:07.321Z" }, + { url = "https://files.pythonhosted.org/packages/9d/9c/2266794e6c25fa9ce51ce04a9bf4b7e330e3ac9351ba3fa14e487678acdb/ast_serialize-0.13.1-cp314-cp314t-musllinux_1_2_aarch64.whl", hash = "sha256:9221e09496fb942901993c1812ee87950cc0ecbb514c7c0f1f5fa6d78f701ed8", size = 1460227, upload-time = "2026-10-17T00:03:08.733Z" }, + { url = "https://files.pythonhosted.org/packages/2b/d6/14deb28e5d8f328f626a0778ea2cf98fe908f2bbcccd804cb6f449ee1782/ast_serialize-0.13.1-cp314-cp314t-musllinux_1_2_armv7l.whl", hash = "sha256:7a5c38bdb93669c596f94fb7204dd6ebe4fadb19058ac9a2faf2948c2a175f96", size = 1562691, upload-time = "2026-10-17T00:03:10.25Z" }, + { url = "https://files.pythonhosted.org/packages/f5/19/7a28c0e5ba54f0d3e054f847a2a549b748cb179ab433321a8dc1b39e398d/ast_serialize-0.13.1-cp314-cp314t-musllinux_1_2_i686.whl", hash = "sha256:acf4e9f055d75058ab01536da330648746f2f82b7697a85230547f2d5ebe895c", size = 1558669, upload-time = "2026-10-17T00:03:11.799Z" }, + { url = "https://files.pythonhosted.org/packages/18/9a/f9e2690eab58217f1d395a964a78efa7eaf813a1e38be8dfd9a8d45d0840/ast_serialize-0.13.1-cp314-cp314t-musllinux_1_2_ppc64le.whl", hash = "sha256:87427913c325adc811e81e7c9a757b061f47ef1e29114ac479aa0953a2687a98", size = 1665071, upload-time = "2026-10-17T00:03:13.284Z" }, + { url = "https://files.pythonhosted.org/packages/8e/f3/ef5b159a0d7161b404f05e7ec21cf7735ee7edc10c000cd2cbbc7bb6b69a/ast_serialize-0.13.1-cp314-cp314t-musllinux_1_2_riscv64.whl", hash = "sha256:ad88921f95b92bc144d9ad059a0e16f8c1ae69f5239f2d3c6536fd2774f73dfe", size = 1473364, upload-time = "2026-10-17T00:03:14.563Z" }, + { url = "https://files.pythonhosted.org/packages/1d/87/6e8ac4a865124a12cc4f915bf1bcfac3bba426cd5f58d3712a873ee61012/ast_serialize-0.13.1-cp314-cp314t-musllinux_1_2_x86_64.whl", hash = "sha256:f8b17e2921e56621bcb46792bd3e0b82ca3b85de78d5936e7b8174ba1758b0aa", size = 1500566, upload-time = "2026-10-17T00:03:15.917Z" }, + { url = "https://files.pythonhosted.org/packages/f5/1e/29155f1621433008013ad5c940a03d41a676be0680c41527040472d37583/ast_serialize-0.13.1-cp314-cp314t-win32.whl", hash = "sha256:2f9a9ce8b0994f757b941f7f7fe384a6666b4162b941ebc5c248081baea18e47", size = 1128674, upload-time = "2026-10-17T00:03:17.481Z" }, + { url = "https://files.pythonhosted.org/packages/bb/64/2506da65143ea138de5822e9173e0b72320a2f70e23bd723f044033175f1/ast_serialize-0.13.1-cp314-cp314t-win_amd64.whl", hash = "sha256:5d99910e6b9a9b9360bcca2ae99aee36cc912ce058df2a57d1c33099b6cb2593", size = 1163306, upload-time = "2026-10-17T00:03:19.046Z" }, + { url = "https://files.pythonhosted.org/packages/6d/8f/c7cb22638899b0cf15908809c270560e46395e65604bc59e4b623ab9f405/ast_serialize-0.13.1-cp314-cp314t-win_arm64.whl", hash = "sha256:a789cbe14d0a0e9ec9b32a456e5abc81ad8799479c3cfd88a29e3442262aebe6", size = 1133775, upload-time = "2026-10-17T00:03:20.377Z" }, + { url = "https://files.pythonhosted.org/packages/2b/e2/b0c140fe50fa39dd249e382f7733af6f816c90021fe7959a0b026f86237f/ast_serialize-0.13.1-cp315-abi3.abi3t-macosx_10_12_x86_64.whl", hash = "sha256:23d768369b89d551077dc173da6892fad00443338417523b6732f4e8ba4a7ed5", size = 1236160, upload-time = "2026-10-17T00:03:21.945Z" }, + { url = "https://files.pythonhosted.org/packages/23/c4/a8a2ca21e3943555b48221f8fba546877a7ad39efaf6505506524967d7c4/ast_serialize-0.13.1-cp315-abi3.abi3t-macosx_11_0_arm64.whl", hash = "sha256:68f8829d114dab6c9db4a8bbb8d0e342958faee1b5e15a29022678df639960f8", size = 1222214, upload-time = "2026-10-17T00:03:23.462Z" }, + { url = "https://files.pythonhosted.org/packages/15/19/2f58f2ba64e4799b39de85c140aef09a03aa6ea53be75dd292a9d9b9d8ae/ast_serialize-0.13.1-cp315-abi3.abi3t-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:4e385b17d5bde892a6184003a9aff3518387908b73f8f5ea539313c07c8c0ede", size = 1282979, upload-time = "2026-10-17T00:03:24.695Z" }, + { url = "https://files.pythonhosted.org/packages/b1/af/ca6087fe84210c1ccd43e3182917bb70649fa1e69745a90c1d59ba08fea8/ast_serialize-0.13.1-cp315-abi3.abi3t-manylinux_2_17_armv7l.manylinux2014_armv7l.whl", hash = "sha256:02686b281ec9b8e9614f6f0db27e12e4cac515f336c5d05b678bc9eb73ff2af5", size = 1288596, upload-time = "2026-10-17T00:03:26.04Z" }, + { url = "https://files.pythonhosted.org/packages/b0/2c/4ef964c763329a0347125e745ec0451ba4fbaceec9efe2ac729b7cd55f17/ast_serialize-0.13.1-cp315-abi3.abi3t-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl", hash = "sha256:a8ab2c2585b2e22e48cd4aff2136fce4eab4ced569b85f150ef9a5a5d6f34d49", size = 1545540, upload-time = "2026-10-17T00:03:27.52Z" }, + { url = "https://files.pythonhosted.org/packages/bb/85/aa961dbb75813cfa94de0ce9a16a9510c226b95f604c405421e727f44bb0/ast_serialize-0.13.1-cp315-abi3.abi3t-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:c652d9f5d52d8b81eff3ff2c95c9dd0f281de45f6ad5d3fa95527550c396454a", size = 1309374, upload-time = "2026-10-17T00:03:29.091Z" }, + { url = "https://files.pythonhosted.org/packages/b4/e6/f2b67c327d9f9533e41791804403ccc2dc457821195a2e51f0b587a90b90/ast_serialize-0.13.1-cp315-abi3.abi3t-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:121e1d08d58b75c15491cc3003609cbad71ac2b0016043c36d30a6b2032b6605", size = 1303466, upload-time = "2026-10-17T00:03:30.451Z" }, + { url = "https://files.pythonhosted.org/packages/73/1d/33f18b81d1bb1f5d682943ea92fda8130a6593627f12f6201668d40824fd/ast_serialize-0.13.1-cp315-abi3.abi3t-manylinux_2_31_riscv64.whl", hash = "sha256:30cf89b6a6844542d116c758725d13de5b97b9410a51ee577715f761270d0c1c", size = 1300576, upload-time = "2026-10-17T00:03:32.102Z" }, + { url = "https://files.pythonhosted.org/packages/25/cb/64914144925f1a4abdd9217e717b122255c38fc5ebe4342f92052f3bacbc/ast_serialize-0.13.1-cp315-abi3.abi3t-manylinux_2_5_i686.manylinux1_i686.whl", hash = "sha256:732aaae9e498c973fc63fd7f308009b57c4afa6ff59d0b674d894e57d0bca01a", size = 1355960, upload-time = "2026-10-17T00:03:33.676Z" }, + { url = "https://files.pythonhosted.org/packages/ee/80/bd29379fab2bce19adc0bf9177002b11b23dfe8d8fecf409998b458b4624/ast_serialize-0.13.1-cp315-abi3.abi3t-musllinux_1_2_aarch64.whl", hash = "sha256:d9c1a920b2aa0c1e016bed7721689717034ca0dba64ac3a3ff243b003d4e8dd8", size = 1459779, upload-time = "2026-10-17T00:03:35.09Z" }, + { url = "https://files.pythonhosted.org/packages/83/89/012dbe733de05ffdf8e40ca0f9fa5d417cb16ec93a74463c57c001914198/ast_serialize-0.13.1-cp315-abi3.abi3t-musllinux_1_2_armv7l.whl", hash = "sha256:847cad4c22953823f5e76d0b126c6f9da670cfa45b1df41b9324e22189d77611", size = 1563399, upload-time = "2026-10-17T00:03:37.285Z" }, + { url = "https://files.pythonhosted.org/packages/69/37/ee4bcb0358bee38744eb0f18560446572d095fe3bf6b63d6f739d08a839c/ast_serialize-0.13.1-cp315-abi3.abi3t-musllinux_1_2_i686.whl", hash = "sha256:aca07de3ca8e3d9f2b716c3c31f4f384fe4bd2d03e09912417cff7a3e08036f7", size = 1558808, upload-time = "2026-10-17T00:03:38.711Z" }, + { url = "https://files.pythonhosted.org/packages/54/1c/ff44c5355251d61ef22934c4430bb31a61de4781d25a992154aac9b8c82a/ast_serialize-0.13.1-cp315-abi3.abi3t-musllinux_1_2_ppc64le.whl", hash = "sha256:5db12d843d931b9d1f1519ce66b9cbfe72d8563b75c1204d255322254d1964dd", size = 1666324, upload-time = "2026-10-17T00:03:40.376Z" }, + { url = "https://files.pythonhosted.org/packages/c8/bb/6da4bfc8aa3403e6447607131bf3463acaf954d990c6c8b6787c1902ddfe/ast_serialize-0.13.1-cp315-abi3.abi3t-musllinux_1_2_riscv64.whl", hash = "sha256:fb8021f4dc3296de18737e3b8b34994411da725374d74a0b9eb32a738f2a23d0", size = 1474089, upload-time = "2026-10-17T00:03:41.715Z" }, + { url = "https://files.pythonhosted.org/packages/21/97/897e33bef2eefd880e6350e375cc66bc5bcd00a139be480fd655f5b3f095/ast_serialize-0.13.1-cp315-abi3.abi3t-musllinux_1_2_x86_64.whl", hash = "sha256:0d764913374ba21df8cd618045b371ed104ce55c6276de7c59e554a4f5ba26fd", size = 1501542, upload-time = "2026-10-17T00:03:43.08Z" }, + { url = "https://files.pythonhosted.org/packages/8d/4c/821211ac83870c27e4412381b7c7838336456b40575bf840e4680b2018d9/ast_serialize-0.13.1-cp315-abi3.abi3t-win32.whl", hash = "sha256:2bad302ae2427454ef9dab71bd5c3ad67188860c39d36daee38f72031edb39c4", size = 1126118, upload-time = "2026-10-17T00:03:44.616Z" }, + { url = "https://files.pythonhosted.org/packages/1f/86/2b2d5e6c4fa352874a80618e742ffd32d249dee6c119f4233b51afc6b8f4/ast_serialize-0.13.1-cp315-abi3.abi3t-win_amd64.whl", hash = "sha256:032b09d86c785c3984e975378b7861f2a68ac44e22ba7b07b93a536b2524eeb8", size = 1163766, upload-time = "2026-10-17T00:03:45.986Z" }, + { url = "https://files.pythonhosted.org/packages/77/a3/d610cfcdabaa44fd62a54ca1fc57a7dba120643798b089a14914973562de/ast_serialize-0.13.1-cp315-abi3.abi3t-win_arm64.whl", hash = "sha256:929eda0c419e622ab49df3a4c45f81c57dcc10874bbad55d0470a4a03324dac3", size = 1134043, upload-time = "2026-10-17T00:03:48.453Z" }, + { url = "https://files.pythonhosted.org/packages/f0/89/0e30f9eddf7d44ccf9d25979030754fd0457af61fa4279acb8c293a04f97/ast_serialize-0.13.1-cp315-cp315-pyemscripten_2026_5_wasm32.whl", hash = "sha256:bf53a3ba1cbe45399fc1e4ccd684b8f2a11cca450f89706b79122f478534c6f5", size = 897546, upload-time = "2026-10-17T00:03:50.093Z" }, + { url = "https://files.pythonhosted.org/packages/81/92/d705be88ff452b9ba0088b554a0cbd308fd01879da092b33fd2ad7d1aa60/ast_serialize-0.13.1-cp39-abi3-macosx_10_12_x86_64.whl", hash = "sha256:4bcf1d5f4eefecaa325a34ecf079b2741189e7d0821523f92cb343a389a67489", size = 1242416, upload-time = "2026-10-17T00:03:51.521Z" }, + { url = "https://files.pythonhosted.org/packages/60/c2/98370845a7f35cb8a25782e78284b48e01bf3128339881d48d613f304f8f/ast_serialize-0.13.1-cp39-abi3-macosx_11_0_arm64.whl", hash = "sha256:1b158f1c9e3a0d386913924c57995d6df78224868a4d0f215bdd41587c97b34a", size = 1229729, upload-time = "2026-10-17T00:03:52.825Z" }, + { url = "https://files.pythonhosted.org/packages/d8/8b/2d7ffed76addd9d3fcf37fb171400942735c8912264c763561a85407f6a8/ast_serialize-0.13.1-cp39-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:77a6201b69ce42e919f2fd9a559138d5410b5d8afff7c3a6b1f500b0481ff7d8", size = 1292255, upload-time = "2026-10-17T00:03:54.148Z" }, + { url = "https://files.pythonhosted.org/packages/6b/2a/bb9e0e719d89cc950fadfcfcc9cb4bc7640419c4228c852f031ab8abf440/ast_serialize-0.13.1-cp39-abi3-manylinux_2_17_armv7l.manylinux2014_armv7l.whl", hash = "sha256:6eb0be64c9b8f1ed26637eef345abe75ea90351c92c85a8aec7838b3af971011", size = 1298501, upload-time = "2026-10-17T00:03:55.559Z" }, + { url = "https://files.pythonhosted.org/packages/7a/32/13ce07dce48226f5bd9cfd8f708ec101a8be05aa430cda4918ce31ce8c82/ast_serialize-0.13.1-cp39-abi3-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl", hash = "sha256:7f2427bddfca208e84de700044fadf9aef2769a70bae740a1bdabb89ee9e1117", size = 1553758, upload-time = "2026-10-17T00:03:56.952Z" }, + { url = "https://files.pythonhosted.org/packages/e2/fb/9b51d186db0c20ca28cbaaefd50f03b573c89327b9a2627820cbce22196b/ast_serialize-0.13.1-cp39-abi3-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:88df960e67184a71874e5b2ee014f87d1254a0dbc1f903c7bd7b1ab7733ecaef", size = 1318258, upload-time = "2026-10-17T00:03:58.418Z" }, + { url = "https://files.pythonhosted.org/packages/c5/fa/da311559dd3dae9af5825cf21660775a929b970f72e3786af3f2cd667944/ast_serialize-0.13.1-cp39-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:9f99786e0d62082a5b121b3437f3167562133711df22128de1e9886f57f0b226", size = 1313260, upload-time = "2026-10-17T00:04:00.194Z" }, + { url = "https://files.pythonhosted.org/packages/2b/11/97a4b9e5b4776d72f47fcd9a09a497bbf68ec8c1ae074307a2c51e410d85/ast_serialize-0.13.1-cp39-abi3-manylinux_2_31_riscv64.whl", hash = "sha256:a01b515b25a03184eff4acef25b981ae007916379682e908fc8785b342a0bf3e", size = 1309992, upload-time = "2026-10-17T00:04:01.575Z" }, + { url = "https://files.pythonhosted.org/packages/2e/c4/62f85121a80cd451ef04f550cf0f7ea0ad9ff46cc4e9a5b9ae439b6527b1/ast_serialize-0.13.1-cp39-abi3-manylinux_2_5_i686.manylinux1_i686.whl", hash = "sha256:29422d7bcd9e7f39411571c36cec45ce8a7a9b3ad4b2e7aa64497c15a9e397f6", size = 1362715, upload-time = "2026-10-17T00:04:02.881Z" }, + { url = "https://files.pythonhosted.org/packages/c1/fa/c5f4a545f72a7681e3295bab6e3600ee0862ec4391bed961fe9f735b67e0/ast_serialize-0.13.1-cp39-abi3-musllinux_1_2_aarch64.whl", hash = "sha256:6ffb0f548a7dc126067fdfb7b2325ab5dfcdeecc29faf5e0f0461c96d6ebfa3d", size = 1469931, upload-time = "2026-10-17T00:04:04.181Z" }, + { url = "https://files.pythonhosted.org/packages/38/a9/65aed4f8fa7a001a7cfdc03e68e844517ab041ba6fb614b7d78a1d6ac056/ast_serialize-0.13.1-cp39-abi3-musllinux_1_2_armv7l.whl", hash = "sha256:2876444a5f87a7c76f371af3c07c2fda563a08bc8af52568df812e8aba4bf568", size = 1570945, upload-time = "2026-10-17T00:04:05.778Z" }, + { url = "https://files.pythonhosted.org/packages/fb/8c/44b2da991cede79bfbf52089f33a5f0168945466b3d9cd38b5dcf08e7130/ast_serialize-0.13.1-cp39-abi3-musllinux_1_2_i686.whl", hash = "sha256:885a62ec6481ce58ab24c6357360d7a82023b62dd7e5e7ceccf59e9662a3272b", size = 1569978, upload-time = "2026-10-17T00:04:07.347Z" }, + { url = "https://files.pythonhosted.org/packages/46/e4/109b2a5beca700237767d5add920c2f8e1ba9bd5799b0faf143d0e9cf4ef/ast_serialize-0.13.1-cp39-abi3-musllinux_1_2_ppc64le.whl", hash = "sha256:2015fd0165e463aac3b1aadc6a1fa1e5820c14ca950821f4fe02f8a3f6a388d6", size = 1676082, upload-time = "2026-10-17T00:04:08.7Z" }, + { url = "https://files.pythonhosted.org/packages/73/e0/7df70f6dbb11b5762da5416318f08316a1c51348e0d69ec79ca5f98f836a/ast_serialize-0.13.1-cp39-abi3-musllinux_1_2_riscv64.whl", hash = "sha256:eb41e1906502ac80d24ff343f45008552a3f5b5158607959fe6a07dbfc6eb037", size = 1483207, upload-time = "2026-10-17T00:04:10.162Z" }, + { url = "https://files.pythonhosted.org/packages/1f/bc/523aa1c08bd69b5654621beb3dc1a9035715a4a63762fb8d219d6fb3eb81/ast_serialize-0.13.1-cp39-abi3-musllinux_1_2_x86_64.whl", hash = "sha256:ca9acc67f796aa2554fa3037dccc022a4b08b38b2ad153edfac9c9bb7cb1f3c6", size = 1512140, upload-time = "2026-10-17T00:04:11.589Z" }, + { url = "https://files.pythonhosted.org/packages/66/83/6607060659998f00afdea0d8375c1e6e2b7d515ab905090d9a5a02f48851/ast_serialize-0.13.1-cp39-abi3-win32.whl", hash = "sha256:4f98da562e5901d69c7dd30885594c5438f859edf354e9ac7517d188fb026c3a", size = 1131889, upload-time = "2026-10-17T00:04:12.953Z" }, + { url = "https://files.pythonhosted.org/packages/9f/6f/bafd6490fe7b355627fabe95f4417e3d02da93102c8552550eb9a0774e1e/ast_serialize-0.13.1-cp39-abi3-win_amd64.whl", hash = "sha256:ddf7cbc9c8abf70102c4a088efe154bb86bfb825b0a10e556d2b15972cf99bd7", size = 1171153, upload-time = "2026-10-17T00:04:14.362Z" }, + { url = "https://files.pythonhosted.org/packages/11/25/4235f4b375a67a37ac595bf478a5407a744eaf7b64113c1c3a5432b8d954/ast_serialize-0.13.1-cp39-abi3-win_arm64.whl", hash = "sha256:48f11d96c9ec71119612f8e79a8c55c12f2b136716d68347433c561d27a25f43", size = 1139035, upload-time = "2026-10-17T00:04:15.767Z" }, +] + +[[package]] +name = "certifi" +version = "2026.10.23" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/fd/11/b524cbd9bbdaca58608e4cf4053fd463417549e1959fa8d3aa18319f9555/certifi-2026.10.23.tar.gz", hash = "sha256:a9895e6e2758b371e0545e321a3a8fd1fcfd9829ada8824a936dfb16aaaf36cd", size = 136225, upload-time = "2026-10-23T11:36:16.478Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/32/d1/a3652fe2c19134d0d15922177821d863cbd7e842d53a21bbe7e8e3a1b5ac/certifi-2026.10.23-py3-none-any.whl", hash = "sha256:e9b54de76e530bb6cf0ccd036787103acb58fd42ba930ea8bf948ddbaa2f61d1", size = 135082, upload-time = "2026-10-23T11:36:14.763Z" }, +] + +[[package]] +name = "charset-normalizer" +version = "3.5.2" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/33/1c/f41d4e74c28ab327ff3acd36053f7ea506c55872d7a90b0fa71aa3ab0c89/charset_normalizer-3.5.2.tar.gz", hash = "sha256:39de2a259fc954455c57274dc94c79d5842774e1247a016aff30bc0efed0f4ef", size = 172659, upload-time = "2026-09-30T04:39:23.398Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/22/67/6a0b94a7960d5e1b5eacd2fb529f3fccc47db4644f7f0a7cfdcfc3be578a/charset_normalizer-3.5.2-cp311-cp311-macosx_10_9_universal2.whl", hash = "sha256:3d21b8b13c7592db2ac5e544a6d83187b995257472b0c9e8351b6d507ae37ed6", size = 370642, upload-time = "2026-09-30T04:35:06.91Z" }, + { url = "https://files.pythonhosted.org/packages/fb/94/01009e13b94041599004edf32e56e382c24e570f60f79bab8efe45cfe1eb/charset_normalizer-3.5.2-cp311-cp311-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:d760fe2a4d7c3b226cb9026d6a842868d52a7901bd98420e1baf14e80da85cf5", size = 259280, upload-time = "2026-09-30T04:35:08.448Z" }, + { url = "https://files.pythonhosted.org/packages/66/85/3b5358f60a13210f0b67d3755c168ef758701b021e655d88d4da28554467/charset_normalizer-3.5.2-cp311-cp311-manylinux2014_armv7l.manylinux_2_17_armv7l.manylinux_2_31_armv7l.whl", hash = "sha256:c9790464842f85f437dbbb54417eda1e0e6bfc52dd8d22d6fd1c994b73b2dc74", size = 245335, upload-time = "2026-09-30T04:35:10.104Z" }, + { url = "https://files.pythonhosted.org/packages/74/75/77c1c479b09ecd751d1e767b251ea5c14d4d50ff757bf404afab2692f600/charset_normalizer-3.5.2-cp311-cp311-manylinux2014_ppc64le.manylinux_2_17_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:4685902cf26edf013ed7a3da0f426ebba7a00ebb9541386d835afbf002c11cab", size = 291707, upload-time = "2026-09-30T04:35:11.575Z" }, + { url = "https://files.pythonhosted.org/packages/0b/0d/363f78cacb70f58f15f4b083961bbd9d292f335d3f5c66fc4f1cfe69cb90/charset_normalizer-3.5.2-cp311-cp311-manylinux2014_s390x.manylinux_2_17_s390x.manylinux_2_28_s390x.whl", hash = "sha256:4495c5002a7b28557e7e222e77e0b661183e432b7d6d2e788101e3f240e05b8c", size = 286747, upload-time = "2026-09-30T04:35:13.022Z" }, + { url = "https://files.pythonhosted.org/packages/e4/ed/cf505d3011ffceb12c2067a7a5d3cfe92b875d4d44bb0ff0d69375e2c184/charset_normalizer-3.5.2-cp311-cp311-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:211d5a3eb6af8f513b8d4ca19a8c1b7accab1b5f0d3175f9826b03c1a920dc1f", size = 269972, upload-time = "2026-09-30T04:35:14.606Z" }, + { url = "https://files.pythonhosted.org/packages/15/d8/f0a93a431d170e7ca681d4f6650fee3de934d18560e474e7267eb4b0f987/charset_normalizer-3.5.2-cp311-cp311-manylinux_2_31_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:ef4fcbf3327382cd4c9f540babd61248208af7b93eec4de397b4d5f58a09e288", size = 269339, upload-time = "2026-09-30T04:35:16.087Z" }, + { url = "https://files.pythonhosted.org/packages/86/bd/9b2bd1c5b7af02462c9752d33994834ff972a96b4c483eefde9e594488e2/charset_normalizer-3.5.2-cp311-cp311-musllinux_1_2_aarch64.whl", hash = "sha256:bd16aabe4a02a297c23417aa17ac6299dbd8c49f673bcd645b4929b11f5a4400", size = 261683, upload-time = "2026-09-30T04:35:17.488Z" }, + { url = "https://files.pythonhosted.org/packages/76/a5/cac540ab0fd61f3fec88ad3dbb64509e71424593d73cfdfff5ab3e4db279/charset_normalizer-3.5.2-cp311-cp311-musllinux_1_2_armv7l.whl", hash = "sha256:fb9e68df06293761f9fe66ade60a9bc6d0f5e42b8acf2939a9158af86ab0e5bd", size = 248143, upload-time = "2026-09-30T04:35:18.849Z" }, + { url = "https://files.pythonhosted.org/packages/71/7a/ff467301deef2089fad87f72df9e000a26a78fec7acbb18e1999371b8369/charset_normalizer-3.5.2-cp311-cp311-musllinux_1_2_ppc64le.whl", hash = "sha256:59f63901b0031c3136cf64704dcb21de0bbae62ce2c9529bc39d27665463de37", size = 292358, upload-time = "2026-09-30T04:35:20.326Z" }, + { url = "https://files.pythonhosted.org/packages/ad/77/22d7e785d1e210afc2e2f58600dd1799d17a35665faf84383f002826c5f8/charset_normalizer-3.5.2-cp311-cp311-musllinux_1_2_riscv64.whl", hash = "sha256:304d5463e65a35d7bb0850550e0780395395f6fcf452f04db7d5ca7cecc425ac", size = 268766, upload-time = "2026-09-30T04:35:21.72Z" }, + { url = "https://files.pythonhosted.org/packages/ae/91/e8e946267f1c2d9e2bd651726e2fbd2addf02c4d36cea5069e32ca9d7bb5/charset_normalizer-3.5.2-cp311-cp311-musllinux_1_2_s390x.whl", hash = "sha256:9cf9b1a857e25c4baceeb3624e92a56df3668f398c4acba74e174d81fb4d1d3a", size = 287432, upload-time = "2026-09-30T04:35:23.273Z" }, + { url = "https://files.pythonhosted.org/packages/4e/88/7561d8a88d555e7df6623abe7c0070b4baf47549b9408783a2ae0a1a6cf7/charset_normalizer-3.5.2-cp311-cp311-musllinux_1_2_x86_64.whl", hash = "sha256:114e4d0c92d618409ed82a99e22b5c5e768fe995f2973f78265f4524f49d4640", size = 272470, upload-time = "2026-09-30T04:35:24.655Z" }, + { url = "https://files.pythonhosted.org/packages/35/7e/578c702301ec036f01455f30744a08d2b42f6ab35b9b2d4bf8cae0ef2a80/charset_normalizer-3.5.2-cp311-cp311-win32.whl", hash = "sha256:2625388c6c754520c37abaf3b41eb34d1cc4a373f457898f08606c8e362b891d", size = 186688, upload-time = "2026-09-30T04:35:26.225Z" }, + { url = "https://files.pythonhosted.org/packages/e8/fc/fdf8cf52ff21cd5bf158f20978991cf985325842f74283eb6df26c8a39d8/charset_normalizer-3.5.2-cp311-cp311-win_amd64.whl", hash = "sha256:87e50a3e7cb90af586b6c5faf23e302a970415ac73bd7bd90a515a04b427ef96", size = 214932, upload-time = "2026-09-30T04:35:27.796Z" }, + { url = "https://files.pythonhosted.org/packages/97/66/3e45a506d8110b632541faf9a9470185aa9878f1ed44020f31346c1c5e5b/charset_normalizer-3.5.2-cp311-cp311-win_arm64.whl", hash = "sha256:254eb48b9fa5ee9898a3c445825a1f340fe53712a098904b39b0bddba8ea3cb1", size = 202828, upload-time = "2026-09-30T04:35:29.259Z" }, + { url = "https://files.pythonhosted.org/packages/e7/c8/693809898870237d82785a03f3b2b58fe4c9f14669f84a7d4e623c92a59e/charset_normalizer-3.5.2-cp312-cp312-macosx_10_13_universal2.whl", hash = "sha256:ed2a239c0ea213acc1908150a3037257083c7c083128f1a4cec2ec4b97dca491", size = 367780, upload-time = "2026-09-30T04:35:30.888Z" }, + { url = "https://files.pythonhosted.org/packages/c9/87/2fea8c13dc24b3ca9c6f803a5b2dfdeae73eb4f9e12c7885ed908ff0433c/charset_normalizer-3.5.2-cp312-cp312-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:b91363207bd9dc966a691e959bb47f64b30f7ac4b072be9968b366982f7db77c", size = 246730, upload-time = "2026-09-30T04:35:32.286Z" }, + { url = "https://files.pythonhosted.org/packages/a8/9e/09efac30b937722f46d3110ba30b875b24b2e3a266ed746cc4e376a94d80/charset_normalizer-3.5.2-cp312-cp312-manylinux2014_armv7l.manylinux_2_17_armv7l.manylinux_2_31_armv7l.whl", hash = "sha256:38a873987f3be698494da8b2e3085e29da02da7b633dce73e79c699a113d7bf0", size = 237707, upload-time = "2026-09-30T04:35:33.709Z" }, + { url = "https://files.pythonhosted.org/packages/9e/18/70d76670b13686237863a379928d60bd10e021f17d243ab3d7014c4a5f4e/charset_normalizer-3.5.2-cp312-cp312-manylinux2014_ppc64le.manylinux_2_17_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:355ad8011081dec5412240c087a9a0c9d4d5039f3ed11a3f13e18c2b29b56c51", size = 273050, upload-time = "2026-09-30T04:35:35.138Z" }, + { url = "https://files.pythonhosted.org/packages/54/e2/77a8b09d5adc013ed07b95b01b8b8fa5441c4e810e83ee7e4aae2fa4d91a/charset_normalizer-3.5.2-cp312-cp312-manylinux2014_s390x.manylinux_2_17_s390x.manylinux_2_28_s390x.whl", hash = "sha256:ee21e28f0430bd6dc9086c6e525d5e818a44a5ad19720c8a0ef766792f3eb5e5", size = 270345, upload-time = "2026-09-30T04:35:36.502Z" }, + { url = "https://files.pythonhosted.org/packages/7f/c5/38806a25ab5e65fc178f39affeda20858efafede2fce1ffc2556cfc9fe73/charset_normalizer-3.5.2-cp312-cp312-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:3d31298449090ab8d47b7b1b2a555ff73cac7ed438a08b7ac160980c7ebed649", size = 257601, upload-time = "2026-09-30T04:35:37.919Z" }, + { url = "https://files.pythonhosted.org/packages/ae/8d/213565184708fdb263ae55e2c04ee1ff748129dd65d48ed0e3502da9c85a/charset_normalizer-3.5.2-cp312-cp312-manylinux_2_31_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:5cde776b7cc66e4f6c99612cea4aa7269aa65863f7a15841b2c264f103822f4e", size = 252222, upload-time = "2026-09-30T04:35:39.544Z" }, + { url = "https://files.pythonhosted.org/packages/7e/24/76d2cefc25472531e4c5c7dfff68865eb1c39b78482f0fdc15b46f047830/charset_normalizer-3.5.2-cp312-cp312-musllinux_1_2_aarch64.whl", hash = "sha256:ae4f5fea5b8b8ccff88238cc8569303e5ee95efae67fa62922a311397a71f346", size = 248482, upload-time = "2026-09-30T04:35:41.088Z" }, + { url = "https://files.pythonhosted.org/packages/7d/dc/65a801b66ab4c197e22c433ab25e7ac24324ac6f45a2269aca42cce309bf/charset_normalizer-3.5.2-cp312-cp312-musllinux_1_2_armv7l.whl", hash = "sha256:f7d486c83842422badd511868fd8a9a20e9407ace71564b6af47ce7e60a336c1", size = 241206, upload-time = "2026-09-30T04:35:42.59Z" }, + { url = "https://files.pythonhosted.org/packages/a7/95/ca9b5eabde673002c6f1e7ada1b223916fe18f6d661da7aabd4d643718f1/charset_normalizer-3.5.2-cp312-cp312-musllinux_1_2_ppc64le.whl", hash = "sha256:11a4d68a6ecda3292cb1e50239e111543ba5d709bb62a6b4ea1afcfa729d8875", size = 273190, upload-time = "2026-09-30T04:35:44.347Z" }, + { url = "https://files.pythonhosted.org/packages/2d/8b/803b4d2a3f6e1740f63f1e87b04d14b42f3d4fdfe6ed7d4db2d34102b14f/charset_normalizer-3.5.2-cp312-cp312-musllinux_1_2_riscv64.whl", hash = "sha256:d6734d2ef8a50fbf8445c139477da401f50d62a0606bf00e20ec6d87773fefb1", size = 253527, upload-time = "2026-09-30T04:35:45.915Z" }, + { url = "https://files.pythonhosted.org/packages/a9/55/93c0e5dbd085ae0471346026abbe7e0db9ea2d6fea74e51f0b5a46f233a7/charset_normalizer-3.5.2-cp312-cp312-musllinux_1_2_s390x.whl", hash = "sha256:a815775b6c38d4e0ff7bcffbeba67feded90202bb6a226b8dd35f1c855217413", size = 271285, upload-time = "2026-09-30T04:35:47.49Z" }, + { url = "https://files.pythonhosted.org/packages/95/69/0dbd0e0b9b16cfa816cdfcb3e2e3854a1f680dc07fb1245ea125e7448060/charset_normalizer-3.5.2-cp312-cp312-musllinux_1_2_x86_64.whl", hash = "sha256:23851fb4e1b85ed3f6c2a27b777cdfe2e19fb5b38429a8faf38c7542b7665869", size = 260010, upload-time = "2026-09-30T04:35:48.996Z" }, + { url = "https://files.pythonhosted.org/packages/58/9d/e7b88e7b1bf403590c3b573277b5e1e488c68c7a6fbacca310a2c324e90c/charset_normalizer-3.5.2-cp312-cp312-win32.whl", hash = "sha256:db19d07e2e0129e974a0e65d0064fc222a446cd5122c2fd4184d2af9fc734a9e", size = 184125, upload-time = "2026-09-30T04:35:50.777Z" }, + { url = "https://files.pythonhosted.org/packages/eb/e6/e6e083884cbcfd49c64865af05027fe7011be7b2d9179524f099a1b611f3/charset_normalizer-3.5.2-cp312-cp312-win_amd64.whl", hash = "sha256:780fbe7cab297b81dad9fb8dc5eb003c0468ffb0d9e5f65068c53a34661a96bc", size = 207486, upload-time = "2026-09-30T04:35:52.194Z" }, + { url = "https://files.pythonhosted.org/packages/c4/e3/017aea0911ada7405a825c7d937eb3a13009664e2f5b38e8c4bbf2abf894/charset_normalizer-3.5.2-cp312-cp312-win_arm64.whl", hash = "sha256:e2af3aad578aa6bd1384bcf4750fc285e5a9de53f40b7d41e5a0bf748edeb2b3", size = 196734, upload-time = "2026-09-30T04:35:53.636Z" }, + { url = "https://files.pythonhosted.org/packages/c5/34/68292d68512768591aaff07c59bb53ee31341c87759433a859c4641a50c2/charset_normalizer-3.5.2-cp313-cp313-android_24_arm64_v8a.whl", hash = "sha256:ed905975ab14056a2e5eb1c376cb2e1ebc5396baf84163939c518556fccde9f5", size = 225946, upload-time = "2026-09-30T04:35:55.313Z" }, + { url = "https://files.pythonhosted.org/packages/e3/80/bee0b01b90ccd5322ae1d0abb33fab1bd95b7c2eadaf02aeccf22e04ee83/charset_normalizer-3.5.2-cp313-cp313-android_24_x86_64.whl", hash = "sha256:a66c3bc5ab1f0ff2164fc9965ddd611ff0802173f4b9d24554c563f6ab7e1d6e", size = 238619, upload-time = "2026-09-30T04:35:56.863Z" }, + { url = "https://files.pythonhosted.org/packages/78/6e/60ce52a85a7fd631ae8482ae6d74521014ca2f255892679484dc04d7ef56/charset_normalizer-3.5.2-cp313-cp313-ios_13_0_arm64_iphoneos.whl", hash = "sha256:d2374b62878abb00cd8309b32af6c0b715cd02dec0ca74ef12e5069bdc64144a", size = 206701, upload-time = "2026-09-30T04:35:58.639Z" }, + { url = "https://files.pythonhosted.org/packages/36/8c/71aafad23f971afc84c2b295bc0c560739ce1dac558aad9fec22e39f3639/charset_normalizer-3.5.2-cp313-cp313-ios_13_0_arm64_iphonesimulator.whl", hash = "sha256:d376bbd28b3a8999db1a103b3b388aee6f1ddeb3e51bc2172993efdcd86e064d", size = 210506, upload-time = "2026-09-30T04:36:00.147Z" }, + { url = "https://files.pythonhosted.org/packages/91/da/3c5a7798c046df7d2d68ad653cf5b6c5a8bfee225055a843c6f2f42aac1a/charset_normalizer-3.5.2-cp313-cp313-macosx_10_13_universal2.whl", hash = "sha256:6045373d5a89a5ec71afde535db987ca28e76dfa276c2d4c818265b375d4b055", size = 366626, upload-time = "2026-09-30T04:36:01.77Z" }, + { url = "https://files.pythonhosted.org/packages/e1/16/710ac3de2ee354e2bd1a9c94efe45a2d27b5c6ad39b2d6a905be2c094b6c/charset_normalizer-3.5.2-cp313-cp313-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:849df64e889b2e17230d58410a03dba311a65b163508fd33679b2b737d4b7858", size = 244624, upload-time = "2026-09-30T04:36:03.389Z" }, + { url = "https://files.pythonhosted.org/packages/d6/39/45c7439f5b63d24f7d5b2a1d760f34af7628782d7144b4cc8ded45c2d4bc/charset_normalizer-3.5.2-cp313-cp313-manylinux2014_armv7l.manylinux_2_17_armv7l.manylinux_2_31_armv7l.whl", hash = "sha256:15c44f7edfd477b06f517a5cc317fc1707edb9de2c865f43d4b6513907473234", size = 235731, upload-time = "2026-09-30T04:36:04.987Z" }, + { url = "https://files.pythonhosted.org/packages/4d/34/38f3154785ce92e9f56eb226f4d35bdfae6b008480dd055f58837a89c810/charset_normalizer-3.5.2-cp313-cp313-manylinux2014_ppc64le.manylinux_2_17_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:a89012d6d5476ee112d20d998570ed58df2260a852afb1758809cd6900411d21", size = 270352, upload-time = "2026-09-30T04:36:06.412Z" }, + { url = "https://files.pythonhosted.org/packages/04/f3/859f74e7babc977705026b30593b3be04049632a522fb7000f83c033d747/charset_normalizer-3.5.2-cp313-cp313-manylinux2014_s390x.manylinux_2_17_s390x.manylinux_2_28_s390x.whl", hash = "sha256:0c951d5e6dd9c2ff60609476752bee49da4206adde960ebc247766937f72e718", size = 267393, upload-time = "2026-09-30T04:36:07.865Z" }, + { url = "https://files.pythonhosted.org/packages/4b/85/41d27f234b82e47c167a5f6c0f62501dc0c640585ff4aba79e08a390336a/charset_normalizer-3.5.2-cp313-cp313-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:7218e8f32b0956cfcd048fd42d9d5779809745ca1d86113ca56f66e7ae1549c4", size = 254660, upload-time = "2026-09-30T04:36:09.248Z" }, + { url = "https://files.pythonhosted.org/packages/58/ca/5d1a997587febe5b26d8daffe363b5c1a091cece19828eec6502fd09c5ef/charset_normalizer-3.5.2-cp313-cp313-manylinux_2_31_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:a19a731138fc27d5682277d3b9df22855cea1239bce7fcec5f78f42ef2d1f3c3", size = 249996, upload-time = "2026-09-30T04:36:10.73Z" }, + { url = "https://files.pythonhosted.org/packages/b3/1f/d1e78246f7ed60c8c8d606b4ac27f66ce49cc3e95f24893ccbeba9f77302/charset_normalizer-3.5.2-cp313-cp313-musllinux_1_2_aarch64.whl", hash = "sha256:62603db9a7caa0802eaa28c1c46fecd7b3a263a774069c24c3c28c302448721c", size = 247028, upload-time = "2026-09-30T04:36:12.294Z" }, + { url = "https://files.pythonhosted.org/packages/8e/37/eba316edd4f0c4d3a5d945924c4eeeae59abac4056aa815d8a4268f863a2/charset_normalizer-3.5.2-cp313-cp313-musllinux_1_2_armv7l.whl", hash = "sha256:b6856554c4f44d79fc2307d5768854310a8f0096e501c75637542c82292b0429", size = 240562, upload-time = "2026-09-30T04:36:13.887Z" }, + { url = "https://files.pythonhosted.org/packages/c8/8e/aaa037d40ca9ef045977f1a661048b1aa33f223adfce3452fe9be9f79d14/charset_normalizer-3.5.2-cp313-cp313-musllinux_1_2_ppc64le.whl", hash = "sha256:1bc0baf5ef96b6ede57d47f4b8fe4d9d84019c3bfcbeb20a41edc6a6ee341f1f", size = 271699, upload-time = "2026-09-30T04:36:15.41Z" }, + { url = "https://files.pythonhosted.org/packages/26/19/1c1c9f75974adf523b87f34b8a2adc5a435cd65916812bcbd0dfa45f9a29/charset_normalizer-3.5.2-cp313-cp313-musllinux_1_2_riscv64.whl", hash = "sha256:56bc200a365efb37383b7852e4cc5898d3b2da5987289b543956cf8cad71018a", size = 252030, upload-time = "2026-09-30T04:36:16.839Z" }, + { url = "https://files.pythonhosted.org/packages/bc/90/0660ef18e18df0a4d2a1a0edff7dfbba42d4e50ef2425557a5bb7051f77b/charset_normalizer-3.5.2-cp313-cp313-musllinux_1_2_s390x.whl", hash = "sha256:2c9ad19a6cfcd5ea5c0d41161d22f9df1dcc277e9bef2751391334546a314c00", size = 268565, upload-time = "2026-09-30T04:36:18.468Z" }, + { url = "https://files.pythonhosted.org/packages/79/ba/57adc269824e8658f1a0f97a9e514c247445a9632b3419b97e0ba37f16dc/charset_normalizer-3.5.2-cp313-cp313-musllinux_1_2_x86_64.whl", hash = "sha256:e243bd13217235fc7290c621941c3f5cc8b66e4872495be821d7436ba2fb838d", size = 257627, upload-time = "2026-09-30T04:36:19.938Z" }, + { url = "https://files.pythonhosted.org/packages/9a/85/33abd4315c052d3d4f54c92b1ee49bfbc0dc7115a981e462a793b6d2ab87/charset_normalizer-3.5.2-cp313-cp313-pyemscripten_2025_0_wasm32.whl", hash = "sha256:a090bb2c68df85450502e3e20d665e3a5af9c65a84d6508ed477badd49166fd3", size = 143407, upload-time = "2026-09-30T04:36:21.376Z" }, + { url = "https://files.pythonhosted.org/packages/4f/de/6435e18d1aaa5d910b896d551411c96af1f42a0c56c29afc2016c61ccc2e/charset_normalizer-3.5.2-cp313-cp313-win32.whl", hash = "sha256:2b7b3bbfb4fe8ef40600792d762fbaa9057559f9d3fad209525b7a22b99e91fd", size = 183207, upload-time = "2026-09-30T04:36:22.776Z" }, + { url = "https://files.pythonhosted.org/packages/9c/76/b8ec57f4e9ee3253541abf95e4a462c0175fe8032dcd070f1f2421240942/charset_normalizer-3.5.2-cp313-cp313-win_amd64.whl", hash = "sha256:78456a747de8dc58360ffa581f30a002baf5aa28cb262536545e91f113ed7639", size = 206489, upload-time = "2026-09-30T04:36:24.306Z" }, + { url = "https://files.pythonhosted.org/packages/3e/60/c647c6ae47480221e875ea5d743ff94946f7416e3c69415ab772928e8d32/charset_normalizer-3.5.2-cp313-cp313-win_arm64.whl", hash = "sha256:11912e4bb14baae7c5d8791aa55ba0a3a03ec6729073307b0f57270abaa713d3", size = 195501, upload-time = "2026-09-30T04:36:25.846Z" }, + { url = "https://files.pythonhosted.org/packages/58/ca/7aa91362a2f77ac8e9e28a9b902a74f7d0e11a851ef0d27a74308da8cd90/charset_normalizer-3.5.2-cp314-cp314-android_24_arm64_v8a.whl", hash = "sha256:1afb975bd5d68d5ce9f6b6d44fdf2f7e34b895a35e95708a7a91b20a3b51d187", size = 226010, upload-time = "2026-09-30T04:36:27.669Z" }, + { url = "https://files.pythonhosted.org/packages/a8/cf/ac8878d0322cf88a1aad4c7b147db32ca0bd806eb0060957b2e31486dbe6/charset_normalizer-3.5.2-cp314-cp314-android_24_x86_64.whl", hash = "sha256:bbbfc8e28816f19d7c0f1816664980c0a9875d01b27cdf8eedddb639d9e108ad", size = 239266, upload-time = "2026-09-30T04:36:29.434Z" }, + { url = "https://files.pythonhosted.org/packages/c9/6d/9a08d7e0b29b7208e2c6c01dc56c8e0520e7c7beadbbfb024b58fd69c8a5/charset_normalizer-3.5.2-cp314-cp314-ios_13_0_arm64_iphoneos.whl", hash = "sha256:7967d08cf06dee78443b874f98c98036f624f3a4e73e11f9f64f5be4d25393cf", size = 206783, upload-time = "2026-09-30T04:36:30.872Z" }, + { url = "https://files.pythonhosted.org/packages/82/44/b0aa350280e6ff5a5492d17cf10460dd39d5ee848f872f7ba2df10607f60/charset_normalizer-3.5.2-cp314-cp314-ios_13_0_arm64_iphonesimulator.whl", hash = "sha256:4c2b5031f63e331e3839b40aed2dd6f191e9c07edbde303e7876846ea1946995", size = 210449, upload-time = "2026-09-30T04:36:32.625Z" }, + { url = "https://files.pythonhosted.org/packages/7c/8a/40db9aa9f5907bb0e6f8b6d64064bf8852fb33d4b813ff9414911df7647c/charset_normalizer-3.5.2-cp314-cp314-macosx_10_15_universal2.whl", hash = "sha256:fcff63213e8e6e47770541a4607175404f47cbb3ebea7b6058cc82d524a0e424", size = 367815, upload-time = "2026-09-30T04:36:34.197Z" }, + { url = "https://files.pythonhosted.org/packages/7f/72/9c5e7707b57c8ddfa9ddf7b0b1d009d7fbab9e9e887d5b721060f37e307d/charset_normalizer-3.5.2-cp314-cp314-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:8d86d6fc60743dc916eb79e2eb1ec4818e21e427731543af40a3021851174a13", size = 247095, upload-time = "2026-09-30T04:36:35.803Z" }, + { url = "https://files.pythonhosted.org/packages/83/09/71e453691e927de4ddf792770cfaab3f49d494e222f66ea5e404bbd5e39c/charset_normalizer-3.5.2-cp314-cp314-manylinux2014_armv7l.manylinux_2_17_armv7l.manylinux_2_31_armv7l.whl", hash = "sha256:7a881931aa470808df94a8c380eed2bbbc76cd9dc622310f99665658c821eb6d", size = 233942, upload-time = "2026-09-30T04:36:37.407Z" }, + { url = "https://files.pythonhosted.org/packages/9f/86/85c84e4da8b27dd409577d9437926ff581c5f9d3c66038dc68c1a526de51/charset_normalizer-3.5.2-cp314-cp314-manylinux2014_ppc64le.manylinux_2_17_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:8024d00c3faf3fc0c16e07a69f4405e8eac7cc0ab15f65fe6cf43827c4cf72b4", size = 273762, upload-time = "2026-09-30T04:36:38.904Z" }, + { url = "https://files.pythonhosted.org/packages/92/08/564955a4b5f2ccb410ab480bbe8c6a18063ff27f2d35458731c4a5335df9/charset_normalizer-3.5.2-cp314-cp314-manylinux2014_s390x.manylinux_2_17_s390x.manylinux_2_28_s390x.whl", hash = "sha256:4d48f2d08b9de5864e2c8744d4461b862fb149a18274abc8b698c45975573438", size = 269514, upload-time = "2026-09-30T04:36:40.469Z" }, + { url = "https://files.pythonhosted.org/packages/18/24/bad3ac4271589df29cf5ce2f5ae490518a5739358052bd0d61209e6fea54/charset_normalizer-3.5.2-cp314-cp314-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:34276fd796040bf0993ab33a369aa572e6979c7aab225a88893667ad8eac8f7a", size = 255056, upload-time = "2026-09-30T04:36:42.02Z" }, + { url = "https://files.pythonhosted.org/packages/d6/3e/350d89ad49916b86554d6f5f2d03ec1152148f87e5ff735106c6a03b1a36/charset_normalizer-3.5.2-cp314-cp314-manylinux_2_31_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:0521c5665880b33d603717defa76c094048900010897909952397feb3039da56", size = 251311, upload-time = "2026-09-30T04:36:43.577Z" }, + { url = "https://files.pythonhosted.org/packages/56/5b/4970a2d154df502e133402906dd04e3ae7cada7b3011283c88d0479a2585/charset_normalizer-3.5.2-cp314-cp314-musllinux_1_2_aarch64.whl", hash = "sha256:eff0ac9dbe711a4aee69bf04a83896aa9b85f19641264053a9f6d48573abb7dd", size = 250190, upload-time = "2026-09-30T04:36:45.185Z" }, + { url = "https://files.pythonhosted.org/packages/88/8c/f1a91bddc8fb47c2889e29ea7ea49a194eb0d9868675d786806519c00d76/charset_normalizer-3.5.2-cp314-cp314-musllinux_1_2_armv7l.whl", hash = "sha256:1503bccbeb36d5527790c3930327704c39af22de3112f1b1666a9f3ce15ee204", size = 237098, upload-time = "2026-09-30T04:36:46.689Z" }, + { url = "https://files.pythonhosted.org/packages/24/0e/bb5dace3cc7e79068425386a6589c19b5a2ab5fefc2a46abea6919683332/charset_normalizer-3.5.2-cp314-cp314-musllinux_1_2_ppc64le.whl", hash = "sha256:52aa6992700996af31f375de0c6bacd402b0097fe40b53c426b9f51a90ebabc7", size = 274228, upload-time = "2026-09-30T04:36:48.31Z" }, + { url = "https://files.pythonhosted.org/packages/9d/79/b849ad523017ea9f5a45581bbebed91439e0cf42fd2860a6f64e358eb5a6/charset_normalizer-3.5.2-cp314-cp314-musllinux_1_2_riscv64.whl", hash = "sha256:e09a3942ecbdee5cce73ea9d42da82b81b72ac1bf031ce069b93b5adf4eac8cd", size = 252401, upload-time = "2026-09-30T04:36:50.091Z" }, + { url = "https://files.pythonhosted.org/packages/89/8c/75469d690cf47200bce8f6cad7655724fc23148e147abfc5ce78b5f65863/charset_normalizer-3.5.2-cp314-cp314-musllinux_1_2_s390x.whl", hash = "sha256:c7c9ab723cde841fefb34efbad91e87f00a674b1fe1cd0784fde742bf2c154dc", size = 271846, upload-time = "2026-09-30T04:36:51.719Z" }, + { url = "https://files.pythonhosted.org/packages/26/cd/6d52d3c7437cdcf2e310ce9f28f282e733d4ef60ed19105d1819c356255f/charset_normalizer-3.5.2-cp314-cp314-musllinux_1_2_x86_64.whl", hash = "sha256:ddc7dacc8ece3a182e7f15cb862d1fd616b46d076cb1ae9dd232b2c38b655874", size = 258151, upload-time = "2026-09-30T04:36:53.234Z" }, + { url = "https://files.pythonhosted.org/packages/f7/4c/070b38bdb5f49a70199fce923ec0726a49536a63ab262abbfcaaf351110b/charset_normalizer-3.5.2-cp314-cp314-pyemscripten_2026_0_wasm32.whl", hash = "sha256:ee43c17b173d46a3212baa6ead3ae258eeabdae48c263a01ccf0218c366dd655", size = 143570, upload-time = "2026-09-30T04:36:54.816Z" }, + { url = "https://files.pythonhosted.org/packages/81/84/9ebfc8ed6c8c4fcd8e726ff6bf220cc8deb3966e31dce9be8dd8aa017e64/charset_normalizer-3.5.2-cp314-cp314-win32.whl", hash = "sha256:4f87960d57feabfb618e4e0af6e7371645fa26a277860739d6e5d6e0012c92f0", size = 186412, upload-time = "2026-09-30T04:36:56.643Z" }, + { url = "https://files.pythonhosted.org/packages/d1/78/5ed86f743d4bc350db307e7636419a0a5ee1d91806d30c7f667bd5c80dae/charset_normalizer-3.5.2-cp314-cp314-win_amd64.whl", hash = "sha256:e4e81e09c1578b8df602e3db08b0b3ea0a6947ad612f52bf8dc5ea8d47691f0c", size = 210965, upload-time = "2026-09-30T04:36:58.205Z" }, + { url = "https://files.pythonhosted.org/packages/53/94/a3a7698e9b1a395e1eb99ccd9a324be9347973bff4e72db2a06496d7cd27/charset_normalizer-3.5.2-cp314-cp314-win_arm64.whl", hash = "sha256:80d02b6f04e92601a081dd97b23d3128033098bff5d35d392ddcc0476ea11253", size = 200406, upload-time = "2026-09-30T04:36:59.764Z" }, + { url = "https://files.pythonhosted.org/packages/c1/48/c5dd00d5ef7791f02666de250a5bb6071e29b7e133cf4b835800b6d3bc27/charset_normalizer-3.5.2-cp314-cp314t-macosx_10_15_universal2.whl", hash = "sha256:dca9ab98072a5a54ebacebdc45f53e645336b320c667410b061be1ca588ae709", size = 396871, upload-time = "2026-09-30T04:37:01.543Z" }, + { url = "https://files.pythonhosted.org/packages/12/c8/8379554b42e8368161d898476686947a0fdbd3e8865170d7909dcabfdee8/charset_normalizer-3.5.2-cp314-cp314t-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:f0aa869112ef88429ae17820d99c3dd9504c9e9c671d3c246f3d7442cb051084", size = 249711, upload-time = "2026-09-30T04:37:03.111Z" }, + { url = "https://files.pythonhosted.org/packages/4a/eb/2ddb1035d17320caa9f41682935123a9a250277b261c3efc86b2d2a21343/charset_normalizer-3.5.2-cp314-cp314t-manylinux2014_armv7l.manylinux_2_17_armv7l.manylinux_2_31_armv7l.whl", hash = "sha256:c0afc6800ba57ccc350374c5bd6150419915d95ce93cdbab2d783d75eaf30ecb", size = 231146, upload-time = "2026-09-30T04:37:04.721Z" }, + { url = "https://files.pythonhosted.org/packages/4a/24/2ecb4bde104322cd7859d6594fcfa74649f8d90b3221c9feecbef149875b/charset_normalizer-3.5.2-cp314-cp314t-manylinux2014_ppc64le.manylinux_2_17_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:7dcd882da75ef9adf94903b1e3b9419e8aa8fb4c7396822b834b9ef7fb96954f", size = 271552, upload-time = "2026-09-30T04:37:06.295Z" }, + { url = "https://files.pythonhosted.org/packages/3f/98/9d5f6ebc3aee9fef5d30b4aff11fb2ab7a1222b4064f8ef2c7c87cde217a/charset_normalizer-3.5.2-cp314-cp314t-manylinux2014_s390x.manylinux_2_17_s390x.manylinux_2_28_s390x.whl", hash = "sha256:2e06a3a98f916dd41d27f3105e02e7a40181c98c94b9158733d03a6f80506c09", size = 269810, upload-time = "2026-09-30T04:37:07.905Z" }, + { url = "https://files.pythonhosted.org/packages/09/e1/a3b06a10461b1b7628853c934c644e03bc28e42767116afb52f19a56519b/charset_normalizer-3.5.2-cp314-cp314t-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:6bd128f206a7752ae1f2ab6c61bf8a24ba28913a10df8b14c2637b973ff97a80", size = 259845, upload-time = "2026-09-30T04:37:09.554Z" }, + { url = "https://files.pythonhosted.org/packages/fd/d3/6f561f74a296cf27d61775a1dc665ad13f3bff6a798810ca05907f37a7c4/charset_normalizer-3.5.2-cp314-cp314t-manylinux_2_31_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:c8f3d67aeaf55f017982b73683f0e7342ba2f6635a78f69ce89ebb26aa411e5c", size = 252825, upload-time = "2026-09-30T04:37:11.274Z" }, + { url = "https://files.pythonhosted.org/packages/26/9f/69e13ca3b18f43e0eafcd34c04a45b732ae22a43b54a5fc9e119103356eb/charset_normalizer-3.5.2-cp314-cp314t-musllinux_1_2_aarch64.whl", hash = "sha256:fe9753dfee015c570d73df76f899f18444d41388bffcde097deba51c4fadbb9f", size = 252659, upload-time = "2026-09-30T04:37:12.941Z" }, + { url = "https://files.pythonhosted.org/packages/73/a9/ace29806a0dae18939919c76ba526472d83214afa101105fabff2cf30625/charset_normalizer-3.5.2-cp314-cp314t-musllinux_1_2_armv7l.whl", hash = "sha256:92888bb3187c5ba50500b00b3b310c9f2c651709d28036077680cb5255450a03", size = 238819, upload-time = "2026-09-30T04:37:14.659Z" }, + { url = "https://files.pythonhosted.org/packages/f8/c1/6116d52a2e3311ec80f21f5fb5e17b27405f10b9608af8f6e69516841a1b/charset_normalizer-3.5.2-cp314-cp314t-musllinux_1_2_ppc64le.whl", hash = "sha256:d008d90a7f2471519aef0c90dfbe73b3e6e4d5e66ac48e19154c17e89e98b604", size = 272973, upload-time = "2026-09-30T04:37:16.346Z" }, + { url = "https://files.pythonhosted.org/packages/19/aa/9955c7e93bba10a9c7e8f7a5031b7ced66f3a1883a55c00712b8d5850ff3/charset_normalizer-3.5.2-cp314-cp314t-musllinux_1_2_riscv64.whl", hash = "sha256:31f3930700408d211f13378ccbe1c40845d8da54bd0681fac3a9b5aae81c7aa8", size = 253692, upload-time = "2026-09-30T04:37:18.212Z" }, + { url = "https://files.pythonhosted.org/packages/bb/33/2a6ae7fdc1b10cb581cef91addd8cdfc5f40d50abb5702309369d5834579/charset_normalizer-3.5.2-cp314-cp314t-musllinux_1_2_s390x.whl", hash = "sha256:2a925889534b3748302dae5dead07cc13480de1dac3aea80a941b729b471ef93", size = 268757, upload-time = "2026-09-30T04:37:19.877Z" }, + { url = "https://files.pythonhosted.org/packages/a2/22/80992720a0282cd39bba1db35868e6b9c22f41281160143a836544bc1d8a/charset_normalizer-3.5.2-cp314-cp314t-musllinux_1_2_x86_64.whl", hash = "sha256:f5ec61164adcec446f8969a3358ec3f9b26bbda3b9213e5586d219afa8df2915", size = 262220, upload-time = "2026-09-30T04:37:21.583Z" }, + { url = "https://files.pythonhosted.org/packages/92/9f/181fd07e1bffea1d95cd80c84ac537354f50699c22cfc4d3c02b6fc16208/charset_normalizer-3.5.2-cp314-cp314t-win32.whl", hash = "sha256:598a11a2c7ebaa5334bf698bf29568c9c390abac6a154d8170fedecd1cea38c5", size = 197797, upload-time = "2026-09-30T04:37:23.235Z" }, + { url = "https://files.pythonhosted.org/packages/49/1c/25d8415ec1c4f2f41f1680435e4c87cfb378ff2f677d950946f2a45d0632/charset_normalizer-3.5.2-cp314-cp314t-win_amd64.whl", hash = "sha256:7fdde2c9fd9e3eca40631e024664cf2584272cc8f96308cbe5fdfc930f51d8bc", size = 222765, upload-time = "2026-09-30T04:37:24.891Z" }, + { url = "https://files.pythonhosted.org/packages/3e/b4/46b48f013dadfc0d0d33b375438e31bdf5a989dc68389c6bf627054d4df9/charset_normalizer-3.5.2-cp314-cp314t-win_arm64.whl", hash = "sha256:d1befeed746d247c81127bb14de9dc3d30edb6e5976d34f83f86ed262b1d9105", size = 210985, upload-time = "2026-09-30T04:37:26.634Z" }, + { url = "https://files.pythonhosted.org/packages/ca/e9/34e597dee616d0b8ee4b34d29399e85c2204ade174157a48505d42baa4ff/charset_normalizer-3.5.2-cp315-cp315-macosx_10_15_universal2.whl", hash = "sha256:87475fabc8d9996fd9c27debb395e642e8c838d78a00b6e932227a0e06b81e26", size = 367676, upload-time = "2026-09-30T04:37:28.329Z" }, + { url = "https://files.pythonhosted.org/packages/60/9f/a5d1c91c0263745e2cd344c5a4415d787c575501ab1d449f1148ac6b495d/charset_normalizer-3.5.2-cp315-cp315-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:9409a8bf35cf78353942504b24a57de3d75b708997a1e4bd8db71ac8633ce364", size = 248414, upload-time = "2026-09-30T04:37:30.167Z" }, + { url = "https://files.pythonhosted.org/packages/26/79/e697f77464748a3ee3cf490c83d592459400d4898380d66c38366b03080c/charset_normalizer-3.5.2-cp315-cp315-manylinux2014_armv7l.manylinux_2_17_armv7l.manylinux_2_31_armv7l.whl", hash = "sha256:498dc3188ca05a68231ac3fdbfc7f57eb67e1343c30e0fea17f8218c1599b253", size = 244775, upload-time = "2026-09-30T04:37:31.964Z" }, + { url = "https://files.pythonhosted.org/packages/ca/87/3d42a42e18ea066e2513936fd678a00696e77878b5ae04528976abdbcb83/charset_normalizer-3.5.2-cp315-cp315-manylinux2014_ppc64le.manylinux_2_17_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:e242bb1c5e76e97dfa9e7f209a71e93a01d7f19ffdd5cfbb2e2d55b4f08f8ab0", size = 273877, upload-time = "2026-09-30T04:37:33.661Z" }, + { url = "https://files.pythonhosted.org/packages/c3/76/8a28136f3938ba9836f84280ce0c4d61ed1cf15a036b2034900c62634162/charset_normalizer-3.5.2-cp315-cp315-manylinux2014_s390x.manylinux_2_17_s390x.manylinux_2_28_s390x.whl", hash = "sha256:def79fa35ef0cef8d2accec024f4fdc7ead3012ff02f5215c783f39f03ef8cfc", size = 269455, upload-time = "2026-09-30T04:37:35.573Z" }, + { url = "https://files.pythonhosted.org/packages/a0/a1/4fbf5d0f0f1b2a080474c1cf9a2f12c4c6531bb0e8ba591055e846d2b4e9/charset_normalizer-3.5.2-cp315-cp315-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:3df041de8887954562c9b261cba85ca0e9ded74048daf125f45edcfaa4832229", size = 257336, upload-time = "2026-09-30T04:37:37.397Z" }, + { url = "https://files.pythonhosted.org/packages/ba/a2/8b50aa320adb880ad579518e6f718f24944804b42a88b83d267d5d444125/charset_normalizer-3.5.2-cp315-cp315-manylinux_2_31_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:04851f73ae72b8413dddadb16a49dfee95263553741fd42d546f7d66907e6be5", size = 257584, upload-time = "2026-09-30T04:37:39.522Z" }, + { url = "https://files.pythonhosted.org/packages/a5/57/50e3fed84e175f40349bd0da7a4fce94c87f0378f52d74f511d89e0bdc20/charset_normalizer-3.5.2-cp315-cp315-musllinux_1_2_aarch64.whl", hash = "sha256:183b88127acdb4fabe59d951ab424faf1af7b63cdbb5f776186c1ea2ffcaed98", size = 251854, upload-time = "2026-09-30T04:37:41.23Z" }, + { url = "https://files.pythonhosted.org/packages/d6/54/f7fbb3493c9f49091213b9c2d6dd65800696f1ce1a3f196a4205f50417b1/charset_normalizer-3.5.2-cp315-cp315-musllinux_1_2_armv7l.whl", hash = "sha256:16fa0eccf81304b79c5cd87f9271c3b85dd9dd99245e4422ae9c0dd45e0f99d3", size = 248767, upload-time = "2026-09-30T04:37:42.883Z" }, + { url = "https://files.pythonhosted.org/packages/d9/37/b3a6385acc5a1e45b39ae9c90bfb9cf838a09b9dd37ef2740ab4c6b4a2eb/charset_normalizer-3.5.2-cp315-cp315-musllinux_1_2_ppc64le.whl", hash = "sha256:7441d755b7ab94f8d4eb3e43ec05482d760842fd263d003a99102d742cd835e2", size = 274479, upload-time = "2026-09-30T04:37:44.658Z" }, + { url = "https://files.pythonhosted.org/packages/89/44/809913e2cfd279e635a9294fdbbfb1b1dc62a8189d473d561f649fce98d8/charset_normalizer-3.5.2-cp315-cp315-musllinux_1_2_riscv64.whl", hash = "sha256:ca403d7e4798f525fdfc78e258820419cbbd0f0ecbab9de7840e3c017cf6b8cf", size = 258310, upload-time = "2026-09-30T04:37:46.529Z" }, + { url = "https://files.pythonhosted.org/packages/af/a2/f28400ab13359d91bd39179df8e149376b9bf36588e739a3a4f9de2b84b2/charset_normalizer-3.5.2-cp315-cp315-musllinux_1_2_s390x.whl", hash = "sha256:df29a0a7107f7011e77f4eebdddec4c7331e24d787a0b21a46d63bdf7445da95", size = 271100, upload-time = "2026-09-30T04:37:48.399Z" }, + { url = "https://files.pythonhosted.org/packages/e9/89/9bab37955edf0adb3b66f8a3a6617d9f2f487e0d56f295a6a286cb640aa6/charset_normalizer-3.5.2-cp315-cp315-musllinux_1_2_x86_64.whl", hash = "sha256:f3c96f633825733f735c5a9cf21d21a257d8e1edf0b1cee0a064b9c424ca0f7d", size = 259936, upload-time = "2026-09-30T04:37:50.023Z" }, + { url = "https://files.pythonhosted.org/packages/23/b5/4459e08d45a679f903d50fea08bc52cfa728cca4d7bd02c757b5e5abda2e/charset_normalizer-3.5.2-cp315-cp315-win32.whl", hash = "sha256:281cb91036248400f4cc957495cccd44c275c2e0c5854f7e45ac5cf7dc193847", size = 186301, upload-time = "2026-09-30T04:37:51.722Z" }, + { url = "https://files.pythonhosted.org/packages/98/e8/55d5fd3935b4bce6da4fe0df61898e8c82653e317e677bd58aceb9c60f13/charset_normalizer-3.5.2-cp315-cp315-win_amd64.whl", hash = "sha256:89b53f3cda69831909888e0494f4fa0bcd3537e3e138dabeb620bd6ad946bae8", size = 210858, upload-time = "2026-09-30T04:37:53.427Z" }, + { url = "https://files.pythonhosted.org/packages/a9/5b/974423c2fd8e524c7a7f64318c1e02240ef954912fa2b4d70344107b9c68/charset_normalizer-3.5.2-cp315-cp315-win_arm64.whl", hash = "sha256:6be488a102b8cf28d0391d8c4ba7748938ae28b78ad901f8585520fca33ead1a", size = 200309, upload-time = "2026-09-30T04:37:55.015Z" }, + { url = "https://files.pythonhosted.org/packages/ee/f9/00ee0195db1013d8f7c416fd770fbeb560bb46eb2e36b054d05cb56f6cfa/charset_normalizer-3.5.2-cp315-cp315t-macosx_10_15_universal2.whl", hash = "sha256:915563965d418f986e7e145accc592eae9e1a1be3566ff98a05d7a9ec42a76e1", size = 395343, upload-time = "2026-09-30T04:37:56.743Z" }, + { url = "https://files.pythonhosted.org/packages/04/3a/c00b50e94c964cf934c7899cd47c97952fc11dad71cc5884b3c61795b09b/charset_normalizer-3.5.2-cp315-cp315t-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:65cd72beeeca9d3aaea1201e5923859f308f952f9c71de93f06063c79f0f7a3b", size = 250114, upload-time = "2026-09-30T04:37:58.607Z" }, + { url = "https://files.pythonhosted.org/packages/50/27/d102dc880bbcffd0479ab64dfc1fb96777a854355a55e2bda72a71efadcb/charset_normalizer-3.5.2-cp315-cp315t-manylinux2014_armv7l.manylinux_2_17_armv7l.manylinux_2_31_armv7l.whl", hash = "sha256:b7fd005a73d9e657273b7a10dc71a9e03c8fb9ee6999798d6918ce095b81ac7f", size = 239997, upload-time = "2026-09-30T04:38:00.511Z" }, + { url = "https://files.pythonhosted.org/packages/a5/4a/bf7ef45794dd293fab5f98a9309817977fbb845b9998f171b8cc5d8437a3/charset_normalizer-3.5.2-cp315-cp315t-manylinux2014_ppc64le.manylinux_2_17_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:e54da4baf05720032d527874d40b65fa4d7e5c6c6a43d0c3adbeffcaf275a2b3", size = 272313, upload-time = "2026-09-30T04:38:02.509Z" }, + { url = "https://files.pythonhosted.org/packages/e8/ee/008a2837737991474c5754bb3191010007663860979701990982a502cbaf/charset_normalizer-3.5.2-cp315-cp315t-manylinux2014_s390x.manylinux_2_17_s390x.manylinux_2_28_s390x.whl", hash = "sha256:124fbf1a8ff966d87ae05bb8bd45a71f966055ed8bba320d0c7cf450bc5f4d0e", size = 269986, upload-time = "2026-09-30T04:38:04.435Z" }, + { url = "https://files.pythonhosted.org/packages/93/ad/bd74a283940dc910c5b14f8e4f80a248082bc9c0fcbe1f54530cb6d9cc5e/charset_normalizer-3.5.2-cp315-cp315t-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:28b4f0d66fb834ff90f28209ac7bce77868c45d8c93e26f906709d9b7c2e1af9", size = 260202, upload-time = "2026-09-30T04:38:06.549Z" }, + { url = "https://files.pythonhosted.org/packages/8a/7b/ed341c66f69f688723501fac752be3d63c7159ca0d0d4174fc611e5710bb/charset_normalizer-3.5.2-cp315-cp315t-manylinux_2_31_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:58ca3755ee7ff7f59b57789ec9833c9de9ea275405cdd240eda1f193112e398a", size = 259083, upload-time = "2026-09-30T04:38:08.311Z" }, + { url = "https://files.pythonhosted.org/packages/cc/9d/e41588b777965e5031a43128a1e96173ebb35ac75fc53ec3b517e7c21cd4/charset_normalizer-3.5.2-cp315-cp315t-musllinux_1_2_aarch64.whl", hash = "sha256:443eae2bf318abeaf6f15d785138f71fd6de770e99a92158b8b814265e079115", size = 253081, upload-time = "2026-09-30T04:38:10.402Z" }, + { url = "https://files.pythonhosted.org/packages/81/35/b761eb6d8c1eb218b9b42b9b4d5ac902afdc399fb6dac6f9a9aac7bda589/charset_normalizer-3.5.2-cp315-cp315t-musllinux_1_2_armv7l.whl", hash = "sha256:58f361dcbab699cf8f42db3f47c8e7fd1036f138c23a5d08de9fde5f425a730c", size = 248712, upload-time = "2026-09-30T04:38:12.317Z" }, + { url = "https://files.pythonhosted.org/packages/4d/2c/147169a041b747759f37405c0a97157e8e92de967968373101ff14915cba/charset_normalizer-3.5.2-cp315-cp315t-musllinux_1_2_ppc64le.whl", hash = "sha256:1b4cbc7c3491ccb4aa17fcd8165649d01cf39f76de1696da8631b5f71b85401d", size = 274176, upload-time = "2026-09-30T04:38:14.138Z" }, + { url = "https://files.pythonhosted.org/packages/f0/2d/0ff8db0d373ba8538db686db11cd7e8912031490b9e4f383b41912e8d594/charset_normalizer-3.5.2-cp315-cp315t-musllinux_1_2_riscv64.whl", hash = "sha256:ba0b1d2620edf869789c3879223f52bf2afc5d31b3cb47cc57b3a12c05e2aa9d", size = 259413, upload-time = "2026-09-30T04:38:15.841Z" }, + { url = "https://files.pythonhosted.org/packages/8a/8e/b4a085fb47c9d3a7e43576a4784fdd8fe23f907514a972de8086edaf7a48/charset_normalizer-3.5.2-cp315-cp315t-musllinux_1_2_s390x.whl", hash = "sha256:5e2b6b57e9733d39f0c9fd3185efa6b8e29652c4cd8fe94180272cf6ed9a78c4", size = 269434, upload-time = "2026-09-30T04:38:17.626Z" }, + { url = "https://files.pythonhosted.org/packages/83/1c/d8d8d7322a7c3eecdf3237a4a419cf41d2eaad8e006ce7dfdd9d4c8fa2eb/charset_normalizer-3.5.2-cp315-cp315t-musllinux_1_2_x86_64.whl", hash = "sha256:51cf45226a9b588d0d2b4880c62d686934b63ab0bd79ca23ab0e9762eb27441b", size = 262716, upload-time = "2026-09-30T04:38:19.214Z" }, + { url = "https://files.pythonhosted.org/packages/a0/16/0e4c6ba9b44e97a2da150e52d331e8f9c968b21b358fbffa6c856cebcd89/charset_normalizer-3.5.2-cp315-cp315t-win32.whl", hash = "sha256:5fb29fb8cd1a46c27a1bf9613ad5ec2599310d46b4025d9556404a6b6a292800", size = 197159, upload-time = "2026-09-30T04:38:21.037Z" }, + { url = "https://files.pythonhosted.org/packages/be/33/e90bc2b1374f7f36ef106f56620de5a783907e19ca857efe2277e31cac3e/charset_normalizer-3.5.2-cp315-cp315t-win_amd64.whl", hash = "sha256:a192e2c40070d92c3ccf777e3a5c4ff515573cd2bb7ed0c537fdadbbec5bbf21", size = 221322, upload-time = "2026-09-30T04:38:22.886Z" }, + { url = "https://files.pythonhosted.org/packages/66/89/dfa6dcb08c200b7830ab56439e8c1890f2971d51aafbb3937894a2e7fcfc/charset_normalizer-3.5.2-cp315-cp315t-win_arm64.whl", hash = "sha256:749e97e1b32313717a565abbe321bc2190bc8b35f1a67e4cdbc7c56c8d8ffe58", size = 209192, upload-time = "2026-09-30T04:38:24.648Z" }, + { url = "https://files.pythonhosted.org/packages/8c/ab/176fbfd5b64939c55d652366aa5b9ef1d767af207a3aa6ebeb0d226c484d/charset_normalizer-3.5.2-cp37-abi3-macosx_10_9_universal2.whl", hash = "sha256:4275811936e2f06feff5e598fb42a1b7ae852da8e39605211892b56b81a34efd", size = 331815, upload-time = "2026-09-30T04:38:26.216Z" }, + { url = "https://files.pythonhosted.org/packages/7e/84/371eac6b30bdbcbf2d632a1a01809103459216fcaae61b8b8d922c1bfb8a/charset_normalizer-3.5.2-cp37-abi3-manylinux1_x86_64.manylinux_2_28_x86_64.manylinux_2_5_x86_64.whl", hash = "sha256:1c50fe28bbc2ced33386f298650d91218076c05420e6cbd790b913adc41659e7", size = 253276, upload-time = "2026-09-30T04:38:28.032Z" }, + { url = "https://files.pythonhosted.org/packages/43/6f/c4fbae58febff71709c51bc7e18fdfa55341dc382704740f9f0cbf03817b/charset_normalizer-3.5.2-cp37-abi3-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:d19fbd981a488e22cd04883659ca6b08f50b5974f9fd7c95655ef6a043e5893f", size = 241239, upload-time = "2026-09-30T04:38:29.732Z" }, + { url = "https://files.pythonhosted.org/packages/61/71/458c3f42164a07d0c5210798e9e704b39e540a6793b05aba67f3a35243a9/charset_normalizer-3.5.2-cp37-abi3-manylinux2014_armv7l.manylinux_2_17_armv7l.manylinux_2_31_armv7l.whl", hash = "sha256:0fed1d06615f022ee3b13caf5e8b180cfea32bb2c5aded8a9d44277afc040f93", size = 231121, upload-time = "2026-09-30T04:38:31.462Z" }, + { url = "https://files.pythonhosted.org/packages/09/54/ab9e89367076f6331bb6c65c4bf14a5361fa5191cb6561bf534f18504e1b/charset_normalizer-3.5.2-cp37-abi3-manylinux2014_ppc64le.manylinux_2_17_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:838dcc90063569a0448120554591a1d6c4a4ffe11babf048908793154ab86ade", size = 260350, upload-time = "2026-09-30T04:38:33.239Z" }, + { url = "https://files.pythonhosted.org/packages/7c/c1/061431ecc688d9d76602502cb57cc01e691e682c18f1beb45f9673b5bbd2/charset_normalizer-3.5.2-cp37-abi3-manylinux2014_s390x.manylinux_2_17_s390x.manylinux_2_28_s390x.whl", hash = "sha256:2ce45c6627b22c47e390bc91a41c3d13032192e699fa0bea96e9671b373d69b0", size = 255430, upload-time = "2026-09-30T04:38:34.865Z" }, + { url = "https://files.pythonhosted.org/packages/8d/1f/20c8949f0676f7ab811abdeb7f4d7f1cbc6e61ff20bef08b44edeb092bc8/charset_normalizer-3.5.2-cp37-abi3-manylinux_2_31_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:0774bf9bf620249fee3e0b8b9fd3065de213be30f3aa94ce2494b3b638949e26", size = 250612, upload-time = "2026-09-30T04:38:36.649Z" }, + { url = "https://files.pythonhosted.org/packages/2b/9e/46f2fa4c431fc98c4ae76a8cb5bdca54e0341e3cfc3fcfd8e82740250818/charset_normalizer-3.5.2-cp37-abi3-musllinux_1_2_aarch64.whl", hash = "sha256:1db38f4c5496827c1a501846d64d14c3b80c7e6714e406cd7dc36a9899fa1011", size = 242083, upload-time = "2026-09-30T04:38:38.26Z" }, + { url = "https://files.pythonhosted.org/packages/bd/39/559be29a0c0f086e0bba6922babd38916cc5e0b58ced4de13ee01ea05508/charset_normalizer-3.5.2-cp37-abi3-musllinux_1_2_armv7l.whl", hash = "sha256:304d8e4d493af723536393eee0c689eb7813f4a474c8b479dee63f1fdd98f621", size = 232738, upload-time = "2026-09-30T04:38:39.81Z" }, + { url = "https://files.pythonhosted.org/packages/ff/6c/387b0e4f756a282831c1d9fc6aeb6c51ca4507ca202767c8de15ce9b12e2/charset_normalizer-3.5.2-cp37-abi3-musllinux_1_2_ppc64le.whl", hash = "sha256:9b7f416ff0978e2f2249330527f0ad6fa02f4932e6199692d3b52da2048c19e4", size = 260703, upload-time = "2026-09-30T04:38:41.346Z" }, + { url = "https://files.pythonhosted.org/packages/96/92/1fdf015f09ef449f50d3ac4b67c90887c9c318b727daa95cc4f866e6521d/charset_normalizer-3.5.2-cp37-abi3-musllinux_1_2_riscv64.whl", hash = "sha256:01077390b03f7988f11d700a2194e69b119741a86b1a638b1db88891e3eced8e", size = 247622, upload-time = "2026-09-30T04:38:42.937Z" }, + { url = "https://files.pythonhosted.org/packages/dc/3c/8e7b8a5671ad5d433669fb2a76f1a0164df2d9b1718b0206bc2a16d840cc/charset_normalizer-3.5.2-cp37-abi3-musllinux_1_2_s390x.whl", hash = "sha256:7e841fb9010836c992c9f12fcbd43a831de93a5f726fc1ccd8ca1d0268c5014c", size = 257500, upload-time = "2026-09-30T04:38:44.604Z" }, + { url = "https://files.pythonhosted.org/packages/b4/f0/45b579df5cabc1d5d53ea1cc35e8437d3ca768c0acccc7041517cb6fbb32/charset_normalizer-3.5.2-cp37-abi3-musllinux_1_2_x86_64.whl", hash = "sha256:9cae88599c7219005d879f98e5ed53341e9a122af585e1091200358a3003d2a0", size = 255100, upload-time = "2026-09-30T04:38:46.289Z" }, + { url = "https://files.pythonhosted.org/packages/31/68/fdec18a343f5fb3f310588dd478b09ac4799e0b187dbade3a8cd776f03ef/charset_normalizer-3.5.2-cp37-abi3-win32.whl", hash = "sha256:01b0c0d2262a9e28e8484a278c7e1b5d650e3ac8cf2683d2967e25899f208bdf", size = 174499, upload-time = "2026-09-30T04:38:47.999Z" }, + { url = "https://files.pythonhosted.org/packages/9d/8a/b618149cc5207943a0242068d7a27897f56a62947b5a039085f2a22029f8/charset_normalizer-3.5.2-cp37-abi3-win_amd64.whl", hash = "sha256:9f56f72050826f63dcee7a7f55b0a77168cb3bfc553fd405e7f8f9ece75a4036", size = 200092, upload-time = "2026-09-30T04:38:49.707Z" }, + { url = "https://files.pythonhosted.org/packages/03/cf/4c66866fa9e2b1c78e3c911516d1de497a677b7ac60f1eceda74ce777ca3/charset_normalizer-3.5.2-cp37-abi3-win_arm64.whl", hash = "sha256:40ab6bffa02ae10a0581e6c198be7d2d8ca5c2a0c64e4ed3465d766df457573e", size = 294363, upload-time = "2026-09-30T04:38:51.312Z" }, + { url = "https://files.pythonhosted.org/packages/fc/ad/d07d7862a62ffa6d79d68074d14823243dd235a77c45262acbf6adeb28bf/charset_normalizer-3.5.2-py3-none-any.whl", hash = "sha256:b6b751274acb69d77b3323d6b7dbaa3c7fdfc1eb829b7eb61d262f32e1af9685", size = 68872, upload-time = "2026-09-30T04:39:21.828Z" }, +] + +[[package]] +name = "click" +version = "8.5.0" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/c7/0e/7fa0ef50764b67090eca4114772a2abf8b6148198475e54c660b97caeee6/click-8.5.0.tar.gz", hash = "sha256:ba0d2089de75ea0310e2dde03160e6ca10009947fb95a182f9b54021bb272e34", size = 382235, upload-time = "2026-08-26T13:33:14.56Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/58/50/6c0d534c5f134586a8e1ba4e330569e32f057e33372ae556463212fb4cd3/click-8.5.0-py3-none-any.whl", hash = "sha256:255bc9599cf7748b4b1a446ccc735421bd08a2ae529a8b88597d3de5664ee360", size = 125251, upload-time = "2026-08-26T13:33:12.928Z" }, +] + +[[package]] +name = "cloudpickle" +version = "3.1.2" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/27/fb/576f067976d320f5f0114a8d9fa1215425441bb35627b1993e5afd8111e5/cloudpickle-3.1.2.tar.gz", hash = "sha256:7fda9eb655c9c230dab534f1983763de5835249750e85fbcef43aaa30a9a2414", size = 22330, upload-time = "2025-11-03T09:25:26.604Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/88/39/799be3f2f0f38cc727ee3b4f1445fe6d5e4133064ec2e4115069418a5bb6/cloudpickle-3.1.2-py3-none-any.whl", hash = "sha256:9acb47f6afd73f60dc1df93bb801b472f05ff42fa6c84167d25cb206be1fbf4a", size = 22228, upload-time = "2025-11-03T09:25:25.534Z" }, +] + +[[package]] +name = "colorama" +version = "0.4.6" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/d8/53/6f443c9a4a8358a93a6792e2acffb9d9d5cb0a5cfd8802644b7b1c9a02e4/colorama-0.4.6.tar.gz", hash = "sha256:08695f5cb7ed6e0531a20572697297273c47b8cae5a63ffc6d6ed5c201be6e44", size = 27697, upload-time = "2022-10-25T02:36:22.414Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/d1/d6/3965ed04c63042e047cb6a3e6ed1a63a35087b6a609aa3a15ed8ac56c221/colorama-0.4.6-py2.py3-none-any.whl", hash = "sha256:4f1d9991f5acc0ca119f9d443620b77f9d6b33703e51011c16baf57afb285fc6", size = 25335, upload-time = "2022-10-25T02:36:20.889Z" }, +] + +[[package]] +name = "contourpy" +version = "1.3.3" +source = { registry = "https://pypi.org/simple" } +resolution-markers = [ + "python_full_version < '3.12'", +] +dependencies = [ + { name = "numpy", version = "2.4.6", source = { registry = "https://pypi.org/simple" } }, +] +sdist = { url = "https://files.pythonhosted.org/packages/58/01/1253e6698a07380cd31a736d248a3f2a50a7c88779a1813da27503cadc2a/contourpy-1.3.3.tar.gz", hash = "sha256:083e12155b210502d0bca491432bb04d56dc3432f95a979b429f2848c3dbe880", size = 13466174, upload-time = "2025-07-26T12:03:12.549Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/91/2e/c4390a31919d8a78b90e8ecf87cd4b4c4f05a5b48d05ec17db8e5404c6f4/contourpy-1.3.3-cp311-cp311-macosx_10_9_x86_64.whl", hash = "sha256:709a48ef9a690e1343202916450bc48b9e51c049b089c7f79a267b46cffcdaa1", size = 288773, upload-time = "2025-07-26T12:01:02.277Z" }, + { url = "https://files.pythonhosted.org/packages/0d/44/c4b0b6095fef4dc9c420e041799591e3b63e9619e3044f7f4f6c21c0ab24/contourpy-1.3.3-cp311-cp311-macosx_11_0_arm64.whl", hash = "sha256:23416f38bfd74d5d28ab8429cc4d63fa67d5068bd711a85edb1c3fb0c3e2f381", size = 270149, upload-time = "2025-07-26T12:01:04.072Z" }, + { url = "https://files.pythonhosted.org/packages/30/2e/dd4ced42fefac8470661d7cb7e264808425e6c5d56d175291e93890cce09/contourpy-1.3.3-cp311-cp311-manylinux_2_26_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:929ddf8c4c7f348e4c0a5a3a714b5c8542ffaa8c22954862a46ca1813b667ee7", size = 329222, upload-time = "2025-07-26T12:01:05.688Z" }, + { url = "https://files.pythonhosted.org/packages/f2/74/cc6ec2548e3d276c71389ea4802a774b7aa3558223b7bade3f25787fafc2/contourpy-1.3.3-cp311-cp311-manylinux_2_26_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:9e999574eddae35f1312c2b4b717b7885d4edd6cb46700e04f7f02db454e67c1", size = 377234, upload-time = "2025-07-26T12:01:07.054Z" }, + { url = "https://files.pythonhosted.org/packages/03/b3/64ef723029f917410f75c09da54254c5f9ea90ef89b143ccadb09df14c15/contourpy-1.3.3-cp311-cp311-manylinux_2_26_s390x.manylinux_2_28_s390x.whl", hash = "sha256:0bf67e0e3f482cb69779dd3061b534eb35ac9b17f163d851e2a547d56dba0a3a", size = 380555, upload-time = "2025-07-26T12:01:08.801Z" }, + { url = "https://files.pythonhosted.org/packages/5f/4b/6157f24ca425b89fe2eb7e7be642375711ab671135be21e6faa100f7448c/contourpy-1.3.3-cp311-cp311-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:51e79c1f7470158e838808d4a996fa9bac72c498e93d8ebe5119bc1e6becb0db", size = 355238, upload-time = "2025-07-26T12:01:10.319Z" }, + { url = "https://files.pythonhosted.org/packages/98/56/f914f0dd678480708a04cfd2206e7c382533249bc5001eb9f58aa693e200/contourpy-1.3.3-cp311-cp311-musllinux_1_2_aarch64.whl", hash = "sha256:598c3aaece21c503615fd59c92a3598b428b2f01bfb4b8ca9c4edeecc2438620", size = 1326218, upload-time = "2025-07-26T12:01:12.659Z" }, + { url = "https://files.pythonhosted.org/packages/fb/d7/4a972334a0c971acd5172389671113ae82aa7527073980c38d5868ff1161/contourpy-1.3.3-cp311-cp311-musllinux_1_2_x86_64.whl", hash = "sha256:322ab1c99b008dad206d406bb61d014cf0174df491ae9d9d0fac6a6fda4f977f", size = 1392867, upload-time = "2025-07-26T12:01:15.533Z" }, + { url = "https://files.pythonhosted.org/packages/75/3e/f2cc6cd56dc8cff46b1a56232eabc6feea52720083ea71ab15523daab796/contourpy-1.3.3-cp311-cp311-win32.whl", hash = "sha256:fd907ae12cd483cd83e414b12941c632a969171bf90fc937d0c9f268a31cafff", size = 183677, upload-time = "2025-07-26T12:01:17.088Z" }, + { url = "https://files.pythonhosted.org/packages/98/4b/9bd370b004b5c9d8045c6c33cf65bae018b27aca550a3f657cdc99acdbd8/contourpy-1.3.3-cp311-cp311-win_amd64.whl", hash = "sha256:3519428f6be58431c56581f1694ba8e50626f2dd550af225f82fb5f5814d2a42", size = 225234, upload-time = "2025-07-26T12:01:18.256Z" }, + { url = "https://files.pythonhosted.org/packages/d9/b6/71771e02c2e004450c12b1120a5f488cad2e4d5b590b1af8bad060360fe4/contourpy-1.3.3-cp311-cp311-win_arm64.whl", hash = "sha256:15ff10bfada4bf92ec8b31c62bf7c1834c244019b4a33095a68000d7075df470", size = 193123, upload-time = "2025-07-26T12:01:19.848Z" }, + { url = "https://files.pythonhosted.org/packages/be/45/adfee365d9ea3d853550b2e735f9d66366701c65db7855cd07621732ccfc/contourpy-1.3.3-cp312-cp312-macosx_10_13_x86_64.whl", hash = "sha256:b08a32ea2f8e42cf1d4be3169a98dd4be32bafe4f22b6c4cb4ba810fa9e5d2cb", size = 293419, upload-time = "2025-07-26T12:01:21.16Z" }, + { url = "https://files.pythonhosted.org/packages/53/3e/405b59cfa13021a56bba395a6b3aca8cec012b45bf177b0eaf7a202cde2c/contourpy-1.3.3-cp312-cp312-macosx_11_0_arm64.whl", hash = "sha256:556dba8fb6f5d8742f2923fe9457dbdd51e1049c4a43fd3986a0b14a1d815fc6", size = 273979, upload-time = "2025-07-26T12:01:22.448Z" }, + { url = "https://files.pythonhosted.org/packages/d4/1c/a12359b9b2ca3a845e8f7f9ac08bdf776114eb931392fcad91743e2ea17b/contourpy-1.3.3-cp312-cp312-manylinux_2_26_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:92d9abc807cf7d0e047b95ca5d957cf4792fcd04e920ca70d48add15c1a90ea7", size = 332653, upload-time = "2025-07-26T12:01:24.155Z" }, + { url = "https://files.pythonhosted.org/packages/63/12/897aeebfb475b7748ea67b61e045accdfcf0d971f8a588b67108ed7f5512/contourpy-1.3.3-cp312-cp312-manylinux_2_26_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:b2e8faa0ed68cb29af51edd8e24798bb661eac3bd9f65420c1887b6ca89987c8", size = 379536, upload-time = "2025-07-26T12:01:25.91Z" }, + { url = "https://files.pythonhosted.org/packages/43/8a/a8c584b82deb248930ce069e71576fc09bd7174bbd35183b7943fb1064fd/contourpy-1.3.3-cp312-cp312-manylinux_2_26_s390x.manylinux_2_28_s390x.whl", hash = "sha256:626d60935cf668e70a5ce6ff184fd713e9683fb458898e4249b63be9e28286ea", size = 384397, upload-time = "2025-07-26T12:01:27.152Z" }, + { url = "https://files.pythonhosted.org/packages/cc/8f/ec6289987824b29529d0dfda0d74a07cec60e54b9c92f3c9da4c0ac732de/contourpy-1.3.3-cp312-cp312-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:4d00e655fcef08aba35ec9610536bfe90267d7ab5ba944f7032549c55a146da1", size = 362601, upload-time = "2025-07-26T12:01:28.808Z" }, + { url = "https://files.pythonhosted.org/packages/05/0a/a3fe3be3ee2dceb3e615ebb4df97ae6f3828aa915d3e10549ce016302bd1/contourpy-1.3.3-cp312-cp312-musllinux_1_2_aarch64.whl", hash = "sha256:451e71b5a7d597379ef572de31eeb909a87246974d960049a9848c3bc6c41bf7", size = 1331288, upload-time = "2025-07-26T12:01:31.198Z" }, + { url = "https://files.pythonhosted.org/packages/33/1d/acad9bd4e97f13f3e2b18a3977fe1b4a37ecf3d38d815333980c6c72e963/contourpy-1.3.3-cp312-cp312-musllinux_1_2_x86_64.whl", hash = "sha256:459c1f020cd59fcfe6650180678a9993932d80d44ccde1fa1868977438f0b411", size = 1403386, upload-time = "2025-07-26T12:01:33.947Z" }, + { url = "https://files.pythonhosted.org/packages/cf/8f/5847f44a7fddf859704217a99a23a4f6417b10e5ab1256a179264561540e/contourpy-1.3.3-cp312-cp312-win32.whl", hash = "sha256:023b44101dfe49d7d53932be418477dba359649246075c996866106da069af69", size = 185018, upload-time = "2025-07-26T12:01:35.64Z" }, + { url = "https://files.pythonhosted.org/packages/19/e8/6026ed58a64563186a9ee3f29f41261fd1828f527dd93d33b60feca63352/contourpy-1.3.3-cp312-cp312-win_amd64.whl", hash = "sha256:8153b8bfc11e1e4d75bcb0bff1db232f9e10b274e0929de9d608027e0d34ff8b", size = 226567, upload-time = "2025-07-26T12:01:36.804Z" }, + { url = "https://files.pythonhosted.org/packages/d1/e2/f05240d2c39a1ed228d8328a78b6f44cd695f7ef47beb3e684cf93604f86/contourpy-1.3.3-cp312-cp312-win_arm64.whl", hash = "sha256:07ce5ed73ecdc4a03ffe3e1b3e3c1166db35ae7584be76f65dbbe28a7791b0cc", size = 193655, upload-time = "2025-07-26T12:01:37.999Z" }, + { url = "https://files.pythonhosted.org/packages/68/35/0167aad910bbdb9599272bd96d01a9ec6852f36b9455cf2ca67bd4cc2d23/contourpy-1.3.3-cp313-cp313-macosx_10_13_x86_64.whl", hash = "sha256:177fb367556747a686509d6fef71d221a4b198a3905fe824430e5ea0fda54eb5", size = 293257, upload-time = "2025-07-26T12:01:39.367Z" }, + { url = "https://files.pythonhosted.org/packages/96/e4/7adcd9c8362745b2210728f209bfbcf7d91ba868a2c5f40d8b58f54c509b/contourpy-1.3.3-cp313-cp313-macosx_11_0_arm64.whl", hash = "sha256:d002b6f00d73d69333dac9d0b8d5e84d9724ff9ef044fd63c5986e62b7c9e1b1", size = 274034, upload-time = "2025-07-26T12:01:40.645Z" }, + { url = "https://files.pythonhosted.org/packages/73/23/90e31ceeed1de63058a02cb04b12f2de4b40e3bef5e082a7c18d9c8ae281/contourpy-1.3.3-cp313-cp313-manylinux_2_26_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:348ac1f5d4f1d66d3322420f01d42e43122f43616e0f194fc1c9f5d830c5b286", size = 334672, upload-time = "2025-07-26T12:01:41.942Z" }, + { url = "https://files.pythonhosted.org/packages/ed/93/b43d8acbe67392e659e1d984700e79eb67e2acb2bd7f62012b583a7f1b55/contourpy-1.3.3-cp313-cp313-manylinux_2_26_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:655456777ff65c2c548b7c454af9c6f33f16c8884f11083244b5819cc214f1b5", size = 381234, upload-time = "2025-07-26T12:01:43.499Z" }, + { url = "https://files.pythonhosted.org/packages/46/3b/bec82a3ea06f66711520f75a40c8fc0b113b2a75edb36aa633eb11c4f50f/contourpy-1.3.3-cp313-cp313-manylinux_2_26_s390x.manylinux_2_28_s390x.whl", hash = "sha256:644a6853d15b2512d67881586bd03f462c7ab755db95f16f14d7e238f2852c67", size = 385169, upload-time = "2025-07-26T12:01:45.219Z" }, + { url = "https://files.pythonhosted.org/packages/4b/32/e0f13a1c5b0f8572d0ec6ae2f6c677b7991fafd95da523159c19eff0696a/contourpy-1.3.3-cp313-cp313-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:4debd64f124ca62069f313a9cb86656ff087786016d76927ae2cf37846b006c9", size = 362859, upload-time = "2025-07-26T12:01:46.519Z" }, + { url = "https://files.pythonhosted.org/packages/33/71/e2a7945b7de4e58af42d708a219f3b2f4cff7386e6b6ab0a0fa0033c49a9/contourpy-1.3.3-cp313-cp313-musllinux_1_2_aarch64.whl", hash = "sha256:a15459b0f4615b00bbd1e91f1b9e19b7e63aea7483d03d804186f278c0af2659", size = 1332062, upload-time = "2025-07-26T12:01:48.964Z" }, + { url = "https://files.pythonhosted.org/packages/12/fc/4e87ac754220ccc0e807284f88e943d6d43b43843614f0a8afa469801db0/contourpy-1.3.3-cp313-cp313-musllinux_1_2_x86_64.whl", hash = "sha256:ca0fdcd73925568ca027e0b17ab07aad764be4706d0a925b89227e447d9737b7", size = 1403932, upload-time = "2025-07-26T12:01:51.979Z" }, + { url = "https://files.pythonhosted.org/packages/a6/2e/adc197a37443f934594112222ac1aa7dc9a98faf9c3842884df9a9d8751d/contourpy-1.3.3-cp313-cp313-win32.whl", hash = "sha256:b20c7c9a3bf701366556e1b1984ed2d0cedf999903c51311417cf5f591d8c78d", size = 185024, upload-time = "2025-07-26T12:01:53.245Z" }, + { url = "https://files.pythonhosted.org/packages/18/0b/0098c214843213759692cc638fce7de5c289200a830e5035d1791d7a2338/contourpy-1.3.3-cp313-cp313-win_amd64.whl", hash = "sha256:1cadd8b8969f060ba45ed7c1b714fe69185812ab43bd6b86a9123fe8f99c3263", size = 226578, upload-time = "2025-07-26T12:01:54.422Z" }, + { url = "https://files.pythonhosted.org/packages/8a/9a/2f6024a0c5995243cd63afdeb3651c984f0d2bc727fd98066d40e141ad73/contourpy-1.3.3-cp313-cp313-win_arm64.whl", hash = "sha256:fd914713266421b7536de2bfa8181aa8c699432b6763a0ea64195ebe28bff6a9", size = 193524, upload-time = "2025-07-26T12:01:55.73Z" }, + { url = "https://files.pythonhosted.org/packages/c0/b3/f8a1a86bd3298513f500e5b1f5fd92b69896449f6cab6a146a5d52715479/contourpy-1.3.3-cp313-cp313t-macosx_10_13_x86_64.whl", hash = "sha256:88df9880d507169449d434c293467418b9f6cbe82edd19284aa0409e7fdb933d", size = 306730, upload-time = "2025-07-26T12:01:57.051Z" }, + { url = "https://files.pythonhosted.org/packages/3f/11/4780db94ae62fc0c2053909b65dc3246bd7cecfc4f8a20d957ad43aa4ad8/contourpy-1.3.3-cp313-cp313t-macosx_11_0_arm64.whl", hash = "sha256:d06bb1f751ba5d417047db62bca3c8fde202b8c11fb50742ab3ab962c81e8216", size = 287897, upload-time = "2025-07-26T12:01:58.663Z" }, + { url = "https://files.pythonhosted.org/packages/ae/15/e59f5f3ffdd6f3d4daa3e47114c53daabcb18574a26c21f03dc9e4e42ff0/contourpy-1.3.3-cp313-cp313t-manylinux_2_26_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:e4e6b05a45525357e382909a4c1600444e2a45b4795163d3b22669285591c1ae", size = 326751, upload-time = "2025-07-26T12:02:00.343Z" }, + { url = "https://files.pythonhosted.org/packages/0f/81/03b45cfad088e4770b1dcf72ea78d3802d04200009fb364d18a493857210/contourpy-1.3.3-cp313-cp313t-manylinux_2_26_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:ab3074b48c4e2cf1a960e6bbeb7f04566bf36b1861d5c9d4d8ac04b82e38ba20", size = 375486, upload-time = "2025-07-26T12:02:02.128Z" }, + { url = "https://files.pythonhosted.org/packages/0c/ba/49923366492ffbdd4486e970d421b289a670ae8cf539c1ea9a09822b371a/contourpy-1.3.3-cp313-cp313t-manylinux_2_26_s390x.manylinux_2_28_s390x.whl", hash = "sha256:6c3d53c796f8647d6deb1abe867daeb66dcc8a97e8455efa729516b997b8ed99", size = 388106, upload-time = "2025-07-26T12:02:03.615Z" }, + { url = "https://files.pythonhosted.org/packages/9f/52/5b00ea89525f8f143651f9f03a0df371d3cbd2fccd21ca9b768c7a6500c2/contourpy-1.3.3-cp313-cp313t-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:50ed930df7289ff2a8d7afeb9603f8289e5704755c7e5c3bbd929c90c817164b", size = 352548, upload-time = "2025-07-26T12:02:05.165Z" }, + { url = "https://files.pythonhosted.org/packages/32/1d/a209ec1a3a3452d490f6b14dd92e72280c99ae3d1e73da74f8277d4ee08f/contourpy-1.3.3-cp313-cp313t-musllinux_1_2_aarch64.whl", hash = "sha256:4feffb6537d64b84877da813a5c30f1422ea5739566abf0bd18065ac040e120a", size = 1322297, upload-time = "2025-07-26T12:02:07.379Z" }, + { url = "https://files.pythonhosted.org/packages/bc/9e/46f0e8ebdd884ca0e8877e46a3f4e633f6c9c8c4f3f6e72be3fe075994aa/contourpy-1.3.3-cp313-cp313t-musllinux_1_2_x86_64.whl", hash = "sha256:2b7e9480ffe2b0cd2e787e4df64270e3a0440d9db8dc823312e2c940c167df7e", size = 1391023, upload-time = "2025-07-26T12:02:10.171Z" }, + { url = "https://files.pythonhosted.org/packages/b9/70/f308384a3ae9cd2209e0849f33c913f658d3326900d0ff5d378d6a1422d2/contourpy-1.3.3-cp313-cp313t-win32.whl", hash = "sha256:283edd842a01e3dcd435b1c5116798d661378d83d36d337b8dde1d16a5fc9ba3", size = 196157, upload-time = "2025-07-26T12:02:11.488Z" }, + { url = "https://files.pythonhosted.org/packages/b2/dd/880f890a6663b84d9e34a6f88cded89d78f0091e0045a284427cb6b18521/contourpy-1.3.3-cp313-cp313t-win_amd64.whl", hash = "sha256:87acf5963fc2b34825e5b6b048f40e3635dd547f590b04d2ab317c2619ef7ae8", size = 240570, upload-time = "2025-07-26T12:02:12.754Z" }, + { url = "https://files.pythonhosted.org/packages/80/99/2adc7d8ffead633234817ef8e9a87115c8a11927a94478f6bb3d3f4d4f7d/contourpy-1.3.3-cp313-cp313t-win_arm64.whl", hash = "sha256:3c30273eb2a55024ff31ba7d052dde990d7d8e5450f4bbb6e913558b3d6c2301", size = 199713, upload-time = "2025-07-26T12:02:14.4Z" }, + { url = "https://files.pythonhosted.org/packages/72/8b/4546f3ab60f78c514ffb7d01a0bd743f90de36f0019d1be84d0a708a580a/contourpy-1.3.3-cp314-cp314-macosx_10_13_x86_64.whl", hash = "sha256:fde6c716d51c04b1c25d0b90364d0be954624a0ee9d60e23e850e8d48353d07a", size = 292189, upload-time = "2025-07-26T12:02:16.095Z" }, + { url = "https://files.pythonhosted.org/packages/fd/e1/3542a9cb596cadd76fcef413f19c79216e002623158befe6daa03dbfa88c/contourpy-1.3.3-cp314-cp314-macosx_11_0_arm64.whl", hash = "sha256:cbedb772ed74ff5be440fa8eee9bd49f64f6e3fc09436d9c7d8f1c287b121d77", size = 273251, upload-time = "2025-07-26T12:02:17.524Z" }, + { url = "https://files.pythonhosted.org/packages/b1/71/f93e1e9471d189f79d0ce2497007731c1e6bf9ef6d1d61b911430c3db4e5/contourpy-1.3.3-cp314-cp314-manylinux_2_26_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:22e9b1bd7a9b1d652cd77388465dc358dafcd2e217d35552424aa4f996f524f5", size = 335810, upload-time = "2025-07-26T12:02:18.9Z" }, + { url = "https://files.pythonhosted.org/packages/91/f9/e35f4c1c93f9275d4e38681a80506b5510e9327350c51f8d4a5a724d178c/contourpy-1.3.3-cp314-cp314-manylinux_2_26_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:a22738912262aa3e254e4f3cb079a95a67132fc5a063890e224393596902f5a4", size = 382871, upload-time = "2025-07-26T12:02:20.418Z" }, + { url = "https://files.pythonhosted.org/packages/b5/71/47b512f936f66a0a900d81c396a7e60d73419868fba959c61efed7a8ab46/contourpy-1.3.3-cp314-cp314-manylinux_2_26_s390x.manylinux_2_28_s390x.whl", hash = "sha256:afe5a512f31ee6bd7d0dda52ec9864c984ca3d66664444f2d72e0dc4eb832e36", size = 386264, upload-time = "2025-07-26T12:02:21.916Z" }, + { url = "https://files.pythonhosted.org/packages/04/5f/9ff93450ba96b09c7c2b3f81c94de31c89f92292f1380261bd7195bea4ea/contourpy-1.3.3-cp314-cp314-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:f64836de09927cba6f79dcd00fdd7d5329f3fccc633468507079c829ca4db4e3", size = 363819, upload-time = "2025-07-26T12:02:23.759Z" }, + { url = "https://files.pythonhosted.org/packages/3e/a6/0b185d4cc480ee494945cde102cb0149ae830b5fa17bf855b95f2e70ad13/contourpy-1.3.3-cp314-cp314-musllinux_1_2_aarch64.whl", hash = "sha256:1fd43c3be4c8e5fd6e4f2baeae35ae18176cf2e5cced681cca908addf1cdd53b", size = 1333650, upload-time = "2025-07-26T12:02:26.181Z" }, + { url = "https://files.pythonhosted.org/packages/43/d7/afdc95580ca56f30fbcd3060250f66cedbde69b4547028863abd8aa3b47e/contourpy-1.3.3-cp314-cp314-musllinux_1_2_x86_64.whl", hash = "sha256:6afc576f7b33cf00996e5c1102dc2a8f7cc89e39c0b55df93a0b78c1bd992b36", size = 1404833, upload-time = "2025-07-26T12:02:28.782Z" }, + { url = "https://files.pythonhosted.org/packages/e2/e2/366af18a6d386f41132a48f033cbd2102e9b0cf6345d35ff0826cd984566/contourpy-1.3.3-cp314-cp314-win32.whl", hash = "sha256:66c8a43a4f7b8df8b71ee1840e4211a3c8d93b214b213f590e18a1beca458f7d", size = 189692, upload-time = "2025-07-26T12:02:30.128Z" }, + { url = "https://files.pythonhosted.org/packages/7d/c2/57f54b03d0f22d4044b8afb9ca0e184f8b1afd57b4f735c2fa70883dc601/contourpy-1.3.3-cp314-cp314-win_amd64.whl", hash = "sha256:cf9022ef053f2694e31d630feaacb21ea24224be1c3ad0520b13d844274614fd", size = 232424, upload-time = "2025-07-26T12:02:31.395Z" }, + { url = "https://files.pythonhosted.org/packages/18/79/a9416650df9b525737ab521aa181ccc42d56016d2123ddcb7b58e926a42c/contourpy-1.3.3-cp314-cp314-win_arm64.whl", hash = "sha256:95b181891b4c71de4bb404c6621e7e2390745f887f2a026b2d99e92c17892339", size = 198300, upload-time = "2025-07-26T12:02:32.956Z" }, + { url = "https://files.pythonhosted.org/packages/1f/42/38c159a7d0f2b7b9c04c64ab317042bb6952b713ba875c1681529a2932fe/contourpy-1.3.3-cp314-cp314t-macosx_10_13_x86_64.whl", hash = "sha256:33c82d0138c0a062380332c861387650c82e4cf1747aaa6938b9b6516762e772", size = 306769, upload-time = "2025-07-26T12:02:34.2Z" }, + { url = "https://files.pythonhosted.org/packages/c3/6c/26a8205f24bca10974e77460de68d3d7c63e282e23782f1239f226fcae6f/contourpy-1.3.3-cp314-cp314t-macosx_11_0_arm64.whl", hash = "sha256:ea37e7b45949df430fe649e5de8351c423430046a2af20b1c1961cae3afcda77", size = 287892, upload-time = "2025-07-26T12:02:35.807Z" }, + { url = "https://files.pythonhosted.org/packages/66/06/8a475c8ab718ebfd7925661747dbb3c3ee9c82ac834ccb3570be49d129f4/contourpy-1.3.3-cp314-cp314t-manylinux_2_26_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:d304906ecc71672e9c89e87c4675dc5c2645e1f4269a5063b99b0bb29f232d13", size = 326748, upload-time = "2025-07-26T12:02:37.193Z" }, + { url = "https://files.pythonhosted.org/packages/b4/a3/c5ca9f010a44c223f098fccd8b158bb1cb287378a31ac141f04730dc49be/contourpy-1.3.3-cp314-cp314t-manylinux_2_26_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:ca658cd1a680a5c9ea96dc61cdbae1e85c8f25849843aa799dfd3cb370ad4fbe", size = 375554, upload-time = "2025-07-26T12:02:38.894Z" }, + { url = "https://files.pythonhosted.org/packages/80/5b/68bd33ae63fac658a4145088c1e894405e07584a316738710b636c6d0333/contourpy-1.3.3-cp314-cp314t-manylinux_2_26_s390x.manylinux_2_28_s390x.whl", hash = "sha256:ab2fd90904c503739a75b7c8c5c01160130ba67944a7b77bbf36ef8054576e7f", size = 388118, upload-time = "2025-07-26T12:02:40.642Z" }, + { url = "https://files.pythonhosted.org/packages/40/52/4c285a6435940ae25d7410a6c36bda5145839bc3f0beb20c707cda18b9d2/contourpy-1.3.3-cp314-cp314t-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:b7301b89040075c30e5768810bc96a8e8d78085b47d8be6e4c3f5a0b4ed478a0", size = 352555, upload-time = "2025-07-26T12:02:42.25Z" }, + { url = "https://files.pythonhosted.org/packages/24/ee/3e81e1dd174f5c7fefe50e85d0892de05ca4e26ef1c9a59c2a57e43b865a/contourpy-1.3.3-cp314-cp314t-musllinux_1_2_aarch64.whl", hash = "sha256:2a2a8b627d5cc6b7c41a4beff6c5ad5eb848c88255fda4a8745f7e901b32d8e4", size = 1322295, upload-time = "2025-07-26T12:02:44.668Z" }, + { url = "https://files.pythonhosted.org/packages/3c/b2/6d913d4d04e14379de429057cd169e5e00f6c2af3bb13e1710bcbdb5da12/contourpy-1.3.3-cp314-cp314t-musllinux_1_2_x86_64.whl", hash = "sha256:fd6ec6be509c787f1caf6b247f0b1ca598bef13f4ddeaa126b7658215529ba0f", size = 1391027, upload-time = "2025-07-26T12:02:47.09Z" }, + { url = "https://files.pythonhosted.org/packages/93/8a/68a4ec5c55a2971213d29a9374913f7e9f18581945a7a31d1a39b5d2dfe5/contourpy-1.3.3-cp314-cp314t-win32.whl", hash = "sha256:e74a9a0f5e3fff48fb5a7f2fd2b9b70a3fe014a67522f79b7cca4c0c7e43c9ae", size = 202428, upload-time = "2025-07-26T12:02:48.691Z" }, + { url = "https://files.pythonhosted.org/packages/fa/96/fd9f641ffedc4fa3ace923af73b9d07e869496c9cc7a459103e6e978992f/contourpy-1.3.3-cp314-cp314t-win_amd64.whl", hash = "sha256:13b68d6a62db8eafaebb8039218921399baf6e47bf85006fd8529f2a08ef33fc", size = 250331, upload-time = "2025-07-26T12:02:50.137Z" }, + { url = "https://files.pythonhosted.org/packages/ae/8c/469afb6465b853afff216f9528ffda78a915ff880ed58813ba4faf4ba0b6/contourpy-1.3.3-cp314-cp314t-win_arm64.whl", hash = "sha256:b7448cb5a725bb1e35ce88771b86fba35ef418952474492cf7c764059933ff8b", size = 203831, upload-time = "2025-07-26T12:02:51.449Z" }, + { url = "https://files.pythonhosted.org/packages/a5/29/8dcfe16f0107943fa92388c23f6e05cff0ba58058c4c95b00280d4c75a14/contourpy-1.3.3-pp311-pypy311_pp73-macosx_10_15_x86_64.whl", hash = "sha256:cd5dfcaeb10f7b7f9dc8941717c6c2ade08f587be2226222c12b25f0483ed497", size = 278809, upload-time = "2025-07-26T12:02:52.74Z" }, + { url = "https://files.pythonhosted.org/packages/85/a9/8b37ef4f7dafeb335daee3c8254645ef5725be4d9c6aa70b50ec46ef2f7e/contourpy-1.3.3-pp311-pypy311_pp73-macosx_11_0_arm64.whl", hash = "sha256:0c1fc238306b35f246d61a1d416a627348b5cf0648648a031e14bb8705fcdfe8", size = 261593, upload-time = "2025-07-26T12:02:54.037Z" }, + { url = "https://files.pythonhosted.org/packages/0a/59/ebfb8c677c75605cc27f7122c90313fd2f375ff3c8d19a1694bda74aaa63/contourpy-1.3.3-pp311-pypy311_pp73-manylinux_2_26_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:70f9aad7de812d6541d29d2bbf8feb22ff7e1c299523db288004e3157ff4674e", size = 302202, upload-time = "2025-07-26T12:02:55.947Z" }, + { url = "https://files.pythonhosted.org/packages/3c/37/21972a15834d90bfbfb009b9d004779bd5a07a0ec0234e5ba8f64d5736f4/contourpy-1.3.3-pp311-pypy311_pp73-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:5ed3657edf08512fc3fe81b510e35c2012fbd3081d2e26160f27ca28affec989", size = 329207, upload-time = "2025-07-26T12:02:57.468Z" }, + { url = "https://files.pythonhosted.org/packages/0c/58/bd257695f39d05594ca4ad60df5bcb7e32247f9951fd09a9b8edb82d1daa/contourpy-1.3.3-pp311-pypy311_pp73-win_amd64.whl", hash = "sha256:3d1a3799d62d45c18bafd41c5fa05120b96a28079f2393af559b843d1a966a77", size = 225315, upload-time = "2025-07-26T12:02:58.801Z" }, +] + +[[package]] +name = "contourpy" +version = "1.4.0" +source = { registry = "https://pypi.org/simple" } +resolution-markers = [ + "python_full_version >= '3.15'", + "python_full_version >= '3.12' and python_full_version < '3.15'", +] +dependencies = [ + { name = "numpy", version = "2.5.4", source = { registry = "https://pypi.org/simple" } }, +] +sdist = { url = "https://files.pythonhosted.org/packages/83/5a/a55177dd22553a277388e8a1b3220e92de91bacb28356cdc73caa240121d/contourpy-1.4.0.tar.gz", hash = "sha256:20156f5a1ac4f8ce02656e39a61e82164a3d359796dc8026f75b062783d500e1", size = 13323726, upload-time = "2026-09-11T19:05:05.808Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/97/ba/01bde9753bdbba04a6da9d2bff3881f021bc708f654655e95fae26d3b3a3/contourpy-1.4.0-cp312-cp312-macosx_10_13_x86_64.whl", hash = "sha256:186ba929df36d61b6127da2e89cd1357e4cb79aca647381ab1a6feb0b152877b", size = 297302, upload-time = "2026-09-11T19:02:45.103Z" }, + { url = "https://files.pythonhosted.org/packages/dd/f5/c5bf67522d49a2222f3154fe46879451d26f0e35ece41770758a64ea78cb/contourpy-1.4.0-cp312-cp312-macosx_11_0_arm64.whl", hash = "sha256:c76f3a5318164db7d9401132fc1b5364b784c613c93fa506e3b5e6d1bf353ec8", size = 284921, upload-time = "2026-09-11T19:02:48.049Z" }, + { url = "https://files.pythonhosted.org/packages/0f/cc/cb989599eec12fda312e127eb8e04a8b21a7a6c94bf7ff1bc68273334a42/contourpy-1.4.0-cp312-cp312-manylinux_2_26_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:43c3ccbb32c6294b183dcc8e8c46dacf5ecef809497e3d48a5be298eef185ad0", size = 356332, upload-time = "2026-09-11T19:02:51.408Z" }, + { url = "https://files.pythonhosted.org/packages/47/fb/6f620d7602507817b0b4c21dd790ed68688f1f4ae9c29aacf21f08db5cf1/contourpy-1.4.0-cp312-cp312-manylinux_2_26_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:86d05cec773c9507a3950122e0e40ce77c23c75ceeb2fc189514e71893cbb34b", size = 405878, upload-time = "2026-09-11T19:02:54.093Z" }, + { url = "https://files.pythonhosted.org/packages/32/4e/693c6d6bdece679f0953775eef3a1b66d56be55478d936e7deb1f3004422/contourpy-1.4.0-cp312-cp312-manylinux_2_26_s390x.manylinux_2_28_s390x.whl", hash = "sha256:2deb580178ca19437a84bd77e4bc2cd91a8ccad212413a83c273900868b5978c", size = 408296, upload-time = "2026-09-11T19:02:56.528Z" }, + { url = "https://files.pythonhosted.org/packages/78/f9/b6831508960d559581c448532ba4df312217072975486dbf2b24fcc5b76f/contourpy-1.4.0-cp312-cp312-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:875f42444c9cf48d56f724f2637e60d0f73b3b12c9041e1484580a233edf9591", size = 383243, upload-time = "2026-09-11T19:02:58.638Z" }, + { url = "https://files.pythonhosted.org/packages/f2/21/52903825a0ae7bb625e8bd30a09816d4c3c5d6174a469c10a616166cf780/contourpy-1.4.0-cp312-cp312-musllinux_1_2_aarch64.whl", hash = "sha256:f863c6100bf926cf47d13f3cd75f9bb8ebd98aaab230eeb4468b91f98f39a6b3", size = 1357306, upload-time = "2026-09-11T19:03:01.036Z" }, + { url = "https://files.pythonhosted.org/packages/d8/79/d68b1ce8539e4071518fed9523f558398f34dcd078b8927b109c72dad2ef/contourpy-1.4.0-cp312-cp312-musllinux_1_2_x86_64.whl", hash = "sha256:3863ef2e2b13fe93f8c0ebb08ee400cb07153b8b0e91c5acb26e4537f283634f", size = 1427243, upload-time = "2026-09-11T19:03:03.866Z" }, + { url = "https://files.pythonhosted.org/packages/ac/f0/b75a10e9d0616b97a30de277b6dfe83f1879880854c7330337309530eb48/contourpy-1.4.0-cp312-cp312-win32.whl", hash = "sha256:5450f091ac1be0be3ad3a2a3b3f23b5e443e78c670ced4fd347d626f92a28fd2", size = 347345, upload-time = "2026-09-11T19:03:06.192Z" }, + { url = "https://files.pythonhosted.org/packages/50/a9/dab08786bb4d77ef9a046b3c1be923be4e73c5d07d91f5a54e8ea9b09e41/contourpy-1.4.0-cp312-cp312-win_amd64.whl", hash = "sha256:6e697d94e69f499ff6bebb899cae97a58d5d14f0e1fe9568b43a0248d2f9af8c", size = 234083, upload-time = "2026-09-11T19:03:08.122Z" }, + { url = "https://files.pythonhosted.org/packages/47/b9/3ba509755a970dbde5948e7142ac21f266be72f38017cc4087a05fdceee1/contourpy-1.4.0-cp312-cp312-win_arm64.whl", hash = "sha256:0c7a4c2716a4e98342221954416a836cca77c996c14ddbf22b5f01d5d93ca09c", size = 559521, upload-time = "2026-09-11T19:03:10.047Z" }, + { url = "https://files.pythonhosted.org/packages/b7/8b/62a6eacea08ea0b6ab0159a7783df7ed421ff9c1ff58fef7c31f58fa5e52/contourpy-1.4.0-cp313-cp313-macosx_10_13_x86_64.whl", hash = "sha256:e21d1d4a9db5b3a793649c7ab0ddb4697f2818929eec871a7b9c45fce7b2a1ea", size = 297330, upload-time = "2026-09-11T19:03:11.809Z" }, + { url = "https://files.pythonhosted.org/packages/3f/2f/103cdc5b7077cd569989d3ddb99ee3d4e96ee31bde8df2a6e426b2dc6195/contourpy-1.4.0-cp313-cp313-macosx_11_0_arm64.whl", hash = "sha256:196759a3e4db60e1546167e361090ebb6d199aa3aafc9e34ea17a5ac3b23e814", size = 285037, upload-time = "2026-09-11T19:03:13.781Z" }, + { url = "https://files.pythonhosted.org/packages/bf/25/97001d1fdf940e6f5adfc592d4f5899fff95daf7dceead3988fe09987d2d/contourpy-1.4.0-cp313-cp313-manylinux_2_26_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:9c0e07c691f3b3321913ed9b8161c50ea5f77fadd006f52f5e755359b0dcbfc5", size = 356675, upload-time = "2026-09-11T19:03:15.374Z" }, + { url = "https://files.pythonhosted.org/packages/a6/5e/9a7241a89e2f74402aefa2569e1349ad2750ff77823c04c788eaaedb7fdd/contourpy-1.4.0-cp313-cp313-manylinux_2_26_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:7f861af508fca2384eef392bc1d8377cfa349b2b854cb11b45d89c75780e2561", size = 406990, upload-time = "2026-09-11T19:03:17.44Z" }, + { url = "https://files.pythonhosted.org/packages/36/81/3a4fbbf6a806eada5453a6d16f6c7d96b9f4b3e57a3127368fb545986b8a/contourpy-1.4.0-cp313-cp313-manylinux_2_26_s390x.manylinux_2_28_s390x.whl", hash = "sha256:a03b32c2e7eda8b17757c022162c13c86f5c3d6f937dddf9ba3c3ff7b513953b", size = 407271, upload-time = "2026-09-11T19:03:19.523Z" }, + { url = "https://files.pythonhosted.org/packages/68/77/8f93ecbde1a4d11dfd460cd19a0c159f5487352e751fa87f5d78cc9db08e/contourpy-1.4.0-cp313-cp313-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:f1219a8898523cba821085f2da8a1b962cc696325763f09d7f93538ba43b2d70", size = 384707, upload-time = "2026-09-11T19:03:21.208Z" }, + { url = "https://files.pythonhosted.org/packages/05/07/e5cdf94d68961ee464de6b4fedd3d2f666584f59ccd124bb1290597122ab/contourpy-1.4.0-cp313-cp313-musllinux_1_2_aarch64.whl", hash = "sha256:20da2f86bfaed60dbba729fe738c8241b1dd22b6cf7e8cbbf30df290bd04ba28", size = 1356741, upload-time = "2026-09-11T19:03:23.621Z" }, + { url = "https://files.pythonhosted.org/packages/fd/8a/2301d07accb257ac08daeb631875fada06fedd019c20ff7b00ee07e9d211/contourpy-1.4.0-cp313-cp313-musllinux_1_2_x86_64.whl", hash = "sha256:6ced1670fafee703b8277ab1645072a226e74f167166eae146fb743916119b67", size = 1426793, upload-time = "2026-09-11T19:03:25.955Z" }, + { url = "https://files.pythonhosted.org/packages/7d/5d/a62648f06259d15c139b14eb8cf3c58fe53cb970a55f2bb77d1070e80a41/contourpy-1.4.0-cp313-cp313-pyemscripten_2025_0_wasm32.whl", hash = "sha256:abd0f7e51ecc52bf8c95713bd80b1c17c516a71fa391c54475308a4377903c2b", size = 117961, upload-time = "2026-09-11T19:03:27.713Z" }, + { url = "https://files.pythonhosted.org/packages/71/31/c5b7771ba6982f319ec90c0cf5c425b36891adfeee2d9f667f90b2d0c166/contourpy-1.4.0-cp313-cp313-win32.whl", hash = "sha256:510f7d93d94cf6ebf4e2cd0640bea16ab8affac7547230175b6745b596ce0599", size = 347438, upload-time = "2026-09-11T19:03:29.829Z" }, + { url = "https://files.pythonhosted.org/packages/68/c1/a59df0754bf83a6f3e153cc19d550a5b01f607e389b1c4d57f436f9139e0/contourpy-1.4.0-cp313-cp313-win_amd64.whl", hash = "sha256:6507a016976a75a15aee47809a34605de6244ccc056e6164bf9fb14b27eecad4", size = 234084, upload-time = "2026-09-11T19:03:31.657Z" }, + { url = "https://files.pythonhosted.org/packages/41/48/97224f5decb1efa5a610f912195deb45321f73ec9b9dac485cd6fe2d5a6a/contourpy-1.4.0-cp313-cp313-win_arm64.whl", hash = "sha256:423bd5f4b3f11d54a8c597234e513382e3454bb106b8ec7ae32ba4ed63f8d99b", size = 559559, upload-time = "2026-09-11T19:03:33.528Z" }, + { url = "https://files.pythonhosted.org/packages/75/b0/b548628c8dddf6e5e0a981b7fa8e4c008024df09c7d17c6a9e271edfa0b3/contourpy-1.4.0-cp314-cp314-macosx_10_15_x86_64.whl", hash = "sha256:a3e67bc1a6a4618a1dac7c3053a9ffece5ddfa2046b2670b342cf430bb0b87d1", size = 297354, upload-time = "2026-09-11T19:03:35.017Z" }, + { url = "https://files.pythonhosted.org/packages/04/5a/513484208742f65af2648c519f0b7ab034616e183c4403538024c136a1ab/contourpy-1.4.0-cp314-cp314-macosx_11_0_arm64.whl", hash = "sha256:072a702e2e178f4fcf96f04775d0c059e4c917925abf0d3208ebb6865df3c23d", size = 283729, upload-time = "2026-09-11T19:03:36.818Z" }, + { url = "https://files.pythonhosted.org/packages/35/da/5a6562febc994b2c4bf9d01e57a50458ef8a2051bb3e7b35a00333e0fc26/contourpy-1.4.0-cp314-cp314-manylinux_2_26_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:404dbcd9233513dfd1323b66ea593fef90e899f658f6b8a9ed8e93bd0ca669db", size = 357819, upload-time = "2026-09-11T19:03:39.058Z" }, + { url = "https://files.pythonhosted.org/packages/6d/99/7b358ce888aaa426f755daa096c99c59d17c352037344e836f20e5d3b957/contourpy-1.4.0-cp314-cp314-manylinux_2_26_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:61624f6722480aa7e168fd746164e7bfdf48f8ccb6542f1613741200dc37e2b1", size = 407795, upload-time = "2026-09-11T19:03:41.293Z" }, + { url = "https://files.pythonhosted.org/packages/be/02/34d7548adf08c60967435d79d7a90882eacc84f5093435287974c7e99943/contourpy-1.4.0-cp314-cp314-manylinux_2_26_s390x.manylinux_2_28_s390x.whl", hash = "sha256:2e741a39dfc96babe1722561e81351aeec104526c57dbe91c167c1db606c2d55", size = 408552, upload-time = "2026-09-11T19:03:42.728Z" }, + { url = "https://files.pythonhosted.org/packages/cf/8c/9677dad226e5aaac5b823d5b09d881ebc32ff1648af417597649841e516c/contourpy-1.4.0-cp314-cp314-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:e439ab450c93455feb0218532ded3e946ec7a9b5f459069f122cfa44b89229f6", size = 384850, upload-time = "2026-09-11T19:03:44.402Z" }, + { url = "https://files.pythonhosted.org/packages/08/f5/e3fbeaa489c223629fbb434737cb3de4ecaca9cae4825e77b92490f640f8/contourpy-1.4.0-cp314-cp314-musllinux_1_2_aarch64.whl", hash = "sha256:417be048f7e122cefbe2a34c51a1f2b0410f8eb35796f794d45475ddb7872d9d", size = 1357601, upload-time = "2026-09-11T19:03:46.746Z" }, + { url = "https://files.pythonhosted.org/packages/88/be/55fec293d342c222ed58a5d4056df4362e192af2e59527a3d5d822de8828/contourpy-1.4.0-cp314-cp314-musllinux_1_2_x86_64.whl", hash = "sha256:43d80072a32299bf945de0a6dd4e034ea0162e832261de2e07a274a5adbba9c9", size = 1427340, upload-time = "2026-09-11T19:03:48.745Z" }, + { url = "https://files.pythonhosted.org/packages/86/40/0faabf453edf59b0e3633381d7ca72c3414fb9b72b5cbb08b06ed792d3e5/contourpy-1.4.0-cp314-cp314-pyemscripten_2026_0_wasm32.whl", hash = "sha256:fa1b787362a3856e63dd2b89449f6c384d55ee1beb8f94f4bcc871b40f02462c", size = 118355, upload-time = "2026-09-11T19:03:50.047Z" }, + { url = "https://files.pythonhosted.org/packages/38/0c/7ea75aa3559ae8a6805015e15a19c92b6480752ecc92526361e5776e88c5/contourpy-1.4.0-cp314-cp314-win32.whl", hash = "sha256:738c44fa71735a617f36da58e32810512407d283d5d1bb5b1cecb00d7eeba7cf", size = 356977, upload-time = "2026-09-11T19:04:11.401Z" }, + { url = "https://files.pythonhosted.org/packages/15/41/3df8cc14bb572c8b447f9dfbc876703f0f2f2896058285b5c87a1d4a638d/contourpy-1.4.0-cp314-cp314-win_amd64.whl", hash = "sha256:181bea01bc742734ae672fa00c717d855dd35e1f029536406538c52e7b0cd73d", size = 240101, upload-time = "2026-09-11T19:04:12.772Z" }, + { url = "https://files.pythonhosted.org/packages/66/01/ee6830a5aa4565662a345172f6f35eeeed625d20f1c9d4ecd35318fe4ade/contourpy-1.4.0-cp314-cp314-win_arm64.whl", hash = "sha256:64039341e2d8804f1a13bda8c69083eab935167bbd7e856cf096241f17d5bd45", size = 576359, upload-time = "2026-09-11T19:04:14.983Z" }, + { url = "https://files.pythonhosted.org/packages/51/af/49ee9c9cf012699e505c1b63c531cc99f19df50f205173092c4d541e3ec7/contourpy-1.4.0-cp314-cp314t-macosx_10_15_x86_64.whl", hash = "sha256:ef47fef9a912c77e3d84b014f701e5a9340f25703e9ed3bcebe37f91fb69dc49", size = 313601, upload-time = "2026-09-11T19:03:51.452Z" }, + { url = "https://files.pythonhosted.org/packages/ad/59/ce411ab2be0f805038626d8f4ca9ce563487ff51ff6c363e7512271564ed/contourpy-1.4.0-cp314-cp314t-macosx_11_0_arm64.whl", hash = "sha256:96019f059bcb3774acc0104be3360a57b36d33eaaa2e1d7f77c800ca8e07618b", size = 302812, upload-time = "2026-09-11T19:03:53.022Z" }, + { url = "https://files.pythonhosted.org/packages/95/b0/390915f9af14c1e2d3b2a9ad99d85b98df4352346a51e368cbb825f813c3/contourpy-1.4.0-cp314-cp314t-manylinux_2_26_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:ddf5a2596d716fd3793434844caf31abc7a40b1e8718431420c89858268fb909", size = 348996, upload-time = "2026-09-11T19:03:54.594Z" }, + { url = "https://files.pythonhosted.org/packages/c7/85/e0952576d54322f3e9983b1e049a7a31f42ce82c5c29d01821e6dd3780d7/contourpy-1.4.0-cp314-cp314t-manylinux_2_26_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:dd59df9e2fb0aff8bd7fd1adb9f349b7c835245a9d0f18c2f2deeb537190f2b1", size = 400272, upload-time = "2026-09-11T19:03:56.822Z" }, + { url = "https://files.pythonhosted.org/packages/d6/7d/f13543a5e1598e4a4ad01d9d5e2d65f9c09c37660616429913d606c5977d/contourpy-1.4.0-cp314-cp314t-manylinux_2_26_s390x.manylinux_2_28_s390x.whl", hash = "sha256:b7794ea07cab575633daad8d6e963b662053391022294d3aacc1d9ba9ef54114", size = 412895, upload-time = "2026-09-11T19:03:58.487Z" }, + { url = "https://files.pythonhosted.org/packages/c5/a8/6e5eaff53507dfd2dfc428240eecda32453067277d4b3e7ca71210b2d611/contourpy-1.4.0-cp314-cp314t-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:a1c8a74744fa746eeaa12f0f9ed7f8b4add9d8f1b14d95e3b5e133675eac888f", size = 377060, upload-time = "2026-09-11T19:04:00.122Z" }, + { url = "https://files.pythonhosted.org/packages/16/a3/d00e44d35511b3cead5bd794ee672d4eecf30d9de4ceac57a49d6204064f/contourpy-1.4.0-cp314-cp314t-musllinux_1_2_aarch64.whl", hash = "sha256:107ec46f7aa1664266d69b1181aadc953d58d39f7d8d648fa5b795d4a060b0de", size = 1345688, upload-time = "2026-09-11T19:04:02.705Z" }, + { url = "https://files.pythonhosted.org/packages/1e/01/1e533b0cae32c8edd63fe40435ec3b29345dab87321afcd28722d7f957d1/contourpy-1.4.0-cp314-cp314t-musllinux_1_2_x86_64.whl", hash = "sha256:5a4c89c7f38a0d7a94356d74e3391073c4325c4d47ad2fd065c3fe7dbae16c0c", size = 1416140, upload-time = "2026-09-11T19:04:04.77Z" }, + { url = "https://files.pythonhosted.org/packages/6d/0c/4ee66759f00d6a5d6909782f14f441d639fa18a4a43142e334351efc47f8/contourpy-1.4.0-cp314-cp314t-win32.whl", hash = "sha256:912c6afaa106e2f74ba22b30416b77ef7eb94e3bdc5a4df51ab147845dba9b6d", size = 371982, upload-time = "2026-09-11T19:04:06.233Z" }, + { url = "https://files.pythonhosted.org/packages/e6/ac/b3e5324138c3e3741f526663f4f265f4f69dfdf26415a39428940864a5f6/contourpy-1.4.0-cp314-cp314t-win_amd64.whl", hash = "sha256:ef9440f6f8506246269a82734f5ff9e2e4c5e775b3996fc883cc491c5257eca6", size = 262907, upload-time = "2026-09-11T19:04:07.882Z" }, + { url = "https://files.pythonhosted.org/packages/19/37/c9aa45e47819dc15a38fc5c81a2fb987fde55e9d3b991fbde514e3b6b5f5/contourpy-1.4.0-cp314-cp314t-win_arm64.whl", hash = "sha256:fc9feef8f1f001c5b87decadc67c4a5d1eebb62ca39c4763d1237ff62cf2b707", size = 587071, upload-time = "2026-09-11T19:04:09.898Z" }, + { url = "https://files.pythonhosted.org/packages/cc/74/66d4e6fbea3bcb3fd4e5ae21c039ba31b8fc408b35884b4bc09b88016dce/contourpy-1.4.0-cp315-cp315-macosx_10_15_x86_64.whl", hash = "sha256:79b06c60d5e569ce12c5da8f0c51217cdc484386e2349fa717c30fafb56d2871", size = 297358, upload-time = "2026-09-11T19:04:16.718Z" }, + { url = "https://files.pythonhosted.org/packages/2e/8d/8f7f25e71d73f15d7859943189900d2d348e740388c9ef52a0180d284762/contourpy-1.4.0-cp315-cp315-macosx_11_0_arm64.whl", hash = "sha256:6f963ea9f0d68d2b6a02d861e3c0aae09ae25ba21f243fcf281e9fed468b078d", size = 283732, upload-time = "2026-09-11T19:04:18.087Z" }, + { url = "https://files.pythonhosted.org/packages/f0/90/957549e3b03c407c670e3e191db63f4aef60dab6605bc967af3a806b543e/contourpy-1.4.0-cp315-cp315-manylinux_2_26_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:5545ff38229c15d11d822fc6f7415b932da9f4975f53e305b7b96ebfb01f889c", size = 357820, upload-time = "2026-09-11T19:04:20.077Z" }, + { url = "https://files.pythonhosted.org/packages/56/4c/fa8de87d907b41b05d03f4945a51675cd67319650d01b4c927d209ffa6ee/contourpy-1.4.0-cp315-cp315-manylinux_2_26_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:630d32f06cedf37b7f1dd2b6a12e4492826bd477a4c9c7b2e2fee6ce3a2cea76", size = 407804, upload-time = "2026-09-11T19:04:21.781Z" }, + { url = "https://files.pythonhosted.org/packages/04/e7/96bd4f4a4ae285727befbdb4b332d53a2654cc5da4585d2525b47a063e1f/contourpy-1.4.0-cp315-cp315-manylinux_2_26_s390x.manylinux_2_28_s390x.whl", hash = "sha256:be85d160e7c795113c2a93254dde2fc7e86b375322570eee78b420969426973c", size = 408559, upload-time = "2026-09-11T19:04:23.432Z" }, + { url = "https://files.pythonhosted.org/packages/ae/94/39660e8dc309f96bf7971eb4ec5659f272f84e42b34c9990a96ecce8442c/contourpy-1.4.0-cp315-cp315-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:018227e134b73090b06f911e773a526c77c9af426be29065500ce096a5accd4c", size = 384861, upload-time = "2026-09-11T19:04:25.198Z" }, + { url = "https://files.pythonhosted.org/packages/ab/7b/411dccbf73d5d19da49bf7735786b3baf4ddb1456412d48ae4d4ab1fc761/contourpy-1.4.0-cp315-cp315-musllinux_1_2_aarch64.whl", hash = "sha256:4bffcb2e8cc5c0e837631acab59dd04265fdac376550e7c3e310523877c68cb7", size = 1357612, upload-time = "2026-09-11T19:04:27.614Z" }, + { url = "https://files.pythonhosted.org/packages/a0/39/bf2442130e7d5aae75a4de4504f0c8aca3ef3da3feac3940ee27ca985d83/contourpy-1.4.0-cp315-cp315-musllinux_1_2_x86_64.whl", hash = "sha256:cc87f2ffa84a49a77d76cb792f28a48ff3b05a222aa0b2bddbd839ba78e1d5ca", size = 1427345, upload-time = "2026-09-11T19:04:29.646Z" }, + { url = "https://files.pythonhosted.org/packages/96/60/e5d9196dd83ccd7dd103762982b10ef19a4f72e8ef67ae7157699b9be0bb/contourpy-1.4.0-cp315-cp315-pyemscripten_2026_5_wasm32.whl", hash = "sha256:c8be3392b84cf43373a488874dc1ebf46f60784a70977f92586e18019d71c7ed", size = 118157, upload-time = "2026-09-11T19:04:31.091Z" }, + { url = "https://files.pythonhosted.org/packages/1a/b2/ae3d90f08ba2ed82189d93b55ffbd41e559760e89af4f9ef54d51869f41d/contourpy-1.4.0-cp315-cp315-win32.whl", hash = "sha256:618137ec5778fb76d494c9ec854b104a377fddba0d128bef7136cce2c3bf0df8", size = 356982, upload-time = "2026-09-11T19:04:52.913Z" }, + { url = "https://files.pythonhosted.org/packages/02/a0/36648fff407dcf8b8482985d33367ffc0fe90be72383869312cd0bd00a1b/contourpy-1.4.0-cp315-cp315-win_amd64.whl", hash = "sha256:ea09dc704029930cbd097f75cb0cc1db9852adbcf1a151fac7e9559f7bcbfd19", size = 240101, upload-time = "2026-09-11T19:04:54.587Z" }, + { url = "https://files.pythonhosted.org/packages/07/09/49e987a96706b8f877596cee00d558d7880793a833addc6b33f80805e5dc/contourpy-1.4.0-cp315-cp315-win_arm64.whl", hash = "sha256:a5e2bb871b90cd7a52bdee63bf80ef7acb46398d27efc66d7679240016efc5aa", size = 576375, upload-time = "2026-09-11T19:04:56.515Z" }, + { url = "https://files.pythonhosted.org/packages/e3/18/820ba24a070fe2a834af73e52245c501960746f5c1777ebc64ec0c0dec12/contourpy-1.4.0-cp315-cp315t-macosx_10_15_x86_64.whl", hash = "sha256:432e89f835cf01f8d2123a130a3126b17ee6e0348e4d62c0973872b30aa873bf", size = 313543, upload-time = "2026-09-11T19:04:32.582Z" }, + { url = "https://files.pythonhosted.org/packages/d3/4c/f61a63d8107e0c1c3fa47d30ce45abad90b2ad48a1a9a4510332db7e0176/contourpy-1.4.0-cp315-cp315t-macosx_11_0_arm64.whl", hash = "sha256:3e0a2392533e5e2abdf277c951047516acc8d2f3c42b2dd9dc34c907c9ebabf2", size = 302702, upload-time = "2026-09-11T19:04:34.257Z" }, + { url = "https://files.pythonhosted.org/packages/9d/93/5af95cb5113ecc6e780ca0c6c79756a2cea8833f971bbae112dd4ad73ded/contourpy-1.4.0-cp315-cp315t-manylinux_2_26_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:758e7496cec3195fc28a28945eab3ceef348c47fb95c7afc2eca2f3901209c8c", size = 354370, upload-time = "2026-09-11T19:04:36.158Z" }, + { url = "https://files.pythonhosted.org/packages/47/75/5a9942446647202beb7b4d150891bfcb78fb112c73862484bbe4cc031092/contourpy-1.4.0-cp315-cp315t-manylinux_2_26_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:bee6ff96653e0807cd0f8eaef00174a031e3a2effa5c20f49a5a8d574b05af23", size = 402340, upload-time = "2026-09-11T19:04:37.914Z" }, + { url = "https://files.pythonhosted.org/packages/c3/4f/bfdb0c67775a584f957211936661289789c4d0ec3e988bca2c8c02f8090e/contourpy-1.4.0-cp315-cp315t-manylinux_2_26_s390x.manylinux_2_28_s390x.whl", hash = "sha256:a58b5f130afec61a093d743cdda435cc4047748b6791627e34a68f41ec3a9b07", size = 410721, upload-time = "2026-09-11T19:04:39.925Z" }, + { url = "https://files.pythonhosted.org/packages/65/d3/dddf13d6bb26e145c99debd1f2695018b5f3e24452dd9bfc9904c9d2ec89/contourpy-1.4.0-cp315-cp315t-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:b9dc493e7924e5aa32d89d0b8e50986280f3d4b284968a792787bd7bcffd3cad", size = 382604, upload-time = "2026-09-11T19:04:41.793Z" }, + { url = "https://files.pythonhosted.org/packages/37/b4/89f0a5f6c6ccd322eae2704c7e933c82a4e594a61773933b4b0a18646866/contourpy-1.4.0-cp315-cp315t-musllinux_1_2_aarch64.whl", hash = "sha256:c927ec747633e68c9920bcaad48b0bcefe648812b822f18c34e2d0e2270a3d2a", size = 1354996, upload-time = "2026-09-11T19:04:43.856Z" }, + { url = "https://files.pythonhosted.org/packages/5d/29/fc91cc1ac6f3b591a269f4357b656d6ed0c39eb17c148f41f6da2f9729b9/contourpy-1.4.0-cp315-cp315t-musllinux_1_2_x86_64.whl", hash = "sha256:80c7fc8e7ac217777ce7ba2db3cf8478652b5a4b335283111012ff9eaac9d842", size = 1425030, upload-time = "2026-09-11T19:04:45.981Z" }, + { url = "https://files.pythonhosted.org/packages/d8/b8/bb129f524210c5d3eb8616876e02af280fba8b5554e0ac1afad80b51e06b/contourpy-1.4.0-cp315-cp315t-win32.whl", hash = "sha256:90feb8da006803a95ba573fddc9536357d0c3faddc3a4c9e22816ca49af52878", size = 371854, upload-time = "2026-09-11T19:04:48.106Z" }, + { url = "https://files.pythonhosted.org/packages/d8/b1/bc774d9d200cfc4f073bba78b1bb1cb3b55bf1c8c345eae41fc2db510459/contourpy-1.4.0-cp315-cp315t-win_amd64.whl", hash = "sha256:de503609fdb71f597634ca64fb4bd2f13d27c97f140e912b1e8ed93544840df0", size = 262450, upload-time = "2026-09-11T19:04:49.502Z" }, + { url = "https://files.pythonhosted.org/packages/54/bf/eb4ff492e1c3d027617b5a20e4e6d63a035b41b8cf77a95950159556a3a5/contourpy-1.4.0-cp315-cp315t-win_arm64.whl", hash = "sha256:78ed5c5f962b3e156109f0531b174a65ba80d13cc681a70227b12524c162e184", size = 587064, upload-time = "2026-09-11T19:04:51.148Z" }, +] + +[[package]] +name = "cycler" +version = "0.12.1" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/a9/95/a3dbbb5028f35eafb79008e7522a75244477d2838f38cbb722248dabc2a8/cycler-0.12.1.tar.gz", hash = "sha256:88bb128f02ba341da8ef447245a9e138fae777f6a23943da4540077d3601eb1c", size = 7615, upload-time = "2023-10-07T05:32:18.335Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/e7/05/c19819d5e3d95294a6f5947fb9b9629efb316b96de511b418c53d245aae6/cycler-0.12.1-py3-none-any.whl", hash = "sha256:85cef7cff222d8644161529808465972e51340599459b8ac3ccbac5a854e0d30", size = 8321, upload-time = "2023-10-07T05:32:16.783Z" }, +] + +[[package]] +name = "filelock" +version = "4.2.5" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/aa/4b/2320d9fa03b65b4beb43b521fe1b12b7f87ab91ee42de3a286075016d205/filelock-4.2.5.tar.gz", hash = "sha256:37d8fa4f8b5e9233612e68ab23bf64cae41771322c8797361088ae2895b12894", size = 659943, upload-time = "2026-10-30T19:27:22.237Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/14/2d/2cf36eca330b7a97c0e9076e8fc4272127c29398fa2bb7b24d4321b8b476/filelock-4.2.5-py3-none-any.whl", hash = "sha256:46ce42d27e5cdde09ee7197349663e97cb106679bd9e75db3a09679fbc7aa1a8", size = 163877, upload-time = "2026-10-30T19:27:20.409Z" }, +] + +[[package]] +name = "fonttools" +version = "4.68.0" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/0c/b5/672312ae80ce8b8eb9779c56052a24c8f36652aa7293a1a420553bcf03e5/fonttools-4.68.0.tar.gz", hash = "sha256:73294e233070f71e2b3653b6bd8564af676d57f26d075761fb06d1ab2e4a7a5d", size = 3848583, upload-time = "2026-10-27T16:30:26.451Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/21/51/7c725115425be01aa0fac87eb38f8e8c8c04de80d49a99b08a74898b25ee/fonttools-4.68.0-cp311-cp311-macosx_10_9_universal2.whl", hash = "sha256:362443ab355701d6fd26b10d022cd6c8456e1e248f80103c44cbe5ef0228f87f", size = 3135538, upload-time = "2026-10-27T16:28:02.169Z" }, + { url = "https://files.pythonhosted.org/packages/2e/d5/0a04cbde3e3e997d7bda216f77566bea2eb79018e4ca2adb13ce3c9a7206/fonttools-4.68.0-cp311-cp311-macosx_10_9_x86_64.whl", hash = "sha256:664dfa7a4869b3c827dfc2c38dbbb7a58d53e97933d1bd51ee6729792ee73d50", size = 2632853, upload-time = "2026-10-27T16:28:04.962Z" }, + { url = "https://files.pythonhosted.org/packages/d8/8c/aa4c7b38f4814d6bcb3378276bff802c9610a1227f1d24965a9f23f0a48c/fonttools-4.68.0-cp311-cp311-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:53f2387d7e3816763f30e08d14263be18e28ead53d4dc20a82f9280d0c99d88a", size = 5530571, upload-time = "2026-10-27T16:28:06.952Z" }, + { url = "https://files.pythonhosted.org/packages/00/01/e0f5b3d648ddd2eae35c4bcddcc2cffc059279e9d2cdb325bed10458f606/fonttools-4.68.0-cp311-cp311-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:4b9c89b5876f1a74b1341632dfaf4c4c2be565d76c060445df401ebe7a4ceefa", size = 5493829, upload-time = "2026-10-27T16:28:09.127Z" }, + { url = "https://files.pythonhosted.org/packages/ea/bb/9c5568659275c86e6eebb060d98e8ea77c56fb78ee77ea5654a057f72bde/fonttools-4.68.0-cp311-cp311-musllinux_1_2_aarch64.whl", hash = "sha256:054a4d5de7e1f973ab9789e9ae3f20238774783ca9bf7ee0359aebae4de9400c", size = 5496373, upload-time = "2026-10-27T16:28:11.378Z" }, + { url = "https://files.pythonhosted.org/packages/b0/32/0a35ffd35bfc1b0a754563a45500e87d651e0f336cf006a25e0bc460ba2f/fonttools-4.68.0-cp311-cp311-musllinux_1_2_x86_64.whl", hash = "sha256:b5a015b322275d3d745b06d5e69cf74ef4b1ce2dae02afbe4826cb186d8527a3", size = 5634196, upload-time = "2026-10-27T16:28:13.783Z" }, + { url = "https://files.pythonhosted.org/packages/93/76/3454e20f07dbe4b308b7bc99117292109bb7775a1b595ead768110144553/fonttools-4.68.0-cp311-cp311-win32.whl", hash = "sha256:bd5b047ed8f16b83d0344fef053668712ab9c88e2e46d8612ec41df83c5222e1", size = 1624232, upload-time = "2026-10-27T16:28:16.001Z" }, + { url = "https://files.pythonhosted.org/packages/46/87/4ba592099172764aa56e66919b0c3ddaf944c775433e806e36e165062095/fonttools-4.68.0-cp311-cp311-win_amd64.whl", hash = "sha256:35425bba603cf9d72c9f3c173ea6c98632164f3bdfb1ce1a3dcc928979c6ec6d", size = 1681821, upload-time = "2026-10-27T16:28:17.757Z" }, + { url = "https://files.pythonhosted.org/packages/e1/54/004f12709b21a56bb944fdca6d165b0d90cf9366143675dbc7bb46784ddd/fonttools-4.68.0-cp311-cp311-win_arm64.whl", hash = "sha256:04365b897c9ee29a048b6d4db74c75c38b70aa6d1074726fbdd7ea1a23aa8ed2", size = 1649151, upload-time = "2026-10-27T16:28:19.502Z" }, + { url = "https://files.pythonhosted.org/packages/15/63/f332b2099599b4911786a6f6fd69552fdd5ae6a45cfaffb0c881092f36c8/fonttools-4.68.0-cp312-cp312-macosx_10_13_universal2.whl", hash = "sha256:f265c85432c398d86f26add143423924a4747879b2dce001a3ffe0a5fed30638", size = 3143556, upload-time = "2026-10-27T16:28:21.812Z" }, + { url = "https://files.pythonhosted.org/packages/c8/97/68816bb56843e06631ff11517e4fb35fc62d76aec6e29f11df095e7f33c5/fonttools-4.68.0-cp312-cp312-macosx_10_13_x86_64.whl", hash = "sha256:d225096687fc3bac0ebe3357e95b2dd8b0a49d191f016db0771d537ccdd6f47e", size = 2633689, upload-time = "2026-10-27T16:28:23.54Z" }, + { url = "https://files.pythonhosted.org/packages/21/94/b125c7a6d154d17f4f699150ef3abebd9d8ad0f7eddce9dad649a9e5c1d8/fonttools-4.68.0-cp312-cp312-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:b41e71b2657c9843b2327b4e3c116db1757dc8cd05d978cd47cb5061272d4997", size = 5462605, upload-time = "2026-10-27T16:28:25.774Z" }, + { url = "https://files.pythonhosted.org/packages/7f/58/fe13db71e3f5a020c6875f349806e84f7550f3f24cb3cdbabdd693968b1d/fonttools-4.68.0-cp312-cp312-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:3edce69ecdefa73ab1041901ca22eefed6ebb93a77bc77e889d940479f25bf7d", size = 5443448, upload-time = "2026-10-27T16:28:27.898Z" }, + { url = "https://files.pythonhosted.org/packages/f1/7f/48236f8c626007670586dd6b0d61ed80323a6b17ead763c881d717525cea/fonttools-4.68.0-cp312-cp312-musllinux_1_2_aarch64.whl", hash = "sha256:d8d5674de2b614575f0751671727056af042ac1d61beacb59a298d74282fa492", size = 5399992, upload-time = "2026-10-27T16:28:29.99Z" }, + { url = "https://files.pythonhosted.org/packages/cb/0d/09e9a433430e277e454eace1c4d02c3691a33d5257bd234702a427a9d090/fonttools-4.68.0-cp312-cp312-musllinux_1_2_x86_64.whl", hash = "sha256:7ab8097e7e191784974c5608eb95fb31e1243fe3054cb66778ac82558e9b36f8", size = 5563739, upload-time = "2026-10-27T16:28:32.691Z" }, + { url = "https://files.pythonhosted.org/packages/ac/54/6708f4c6742de2d5bf4280d8bde22fb8aef5120328bdc19ca34a9d2e8e8b/fonttools-4.68.0-cp312-cp312-win32.whl", hash = "sha256:1695cd2e1f9768fe3f14d369ce6ef1b3e8561069cc9e47059571985d1ea42614", size = 2481614, upload-time = "2026-10-27T16:28:35.122Z" }, + { url = "https://files.pythonhosted.org/packages/1c/3e/ef786310a3774ae7aacaa288ca6d1e1dfd03c24214137478e4b630b76af0/fonttools-4.68.0-cp312-cp312-win_amd64.whl", hash = "sha256:fedc183bdbc3fdbfbdca26e0e04566d089cbb404712509a4e824608e9304cc77", size = 2533508, upload-time = "2026-10-27T16:28:37.085Z" }, + { url = "https://files.pythonhosted.org/packages/06/11/2cc4272ddd485c448c0363be9beb0a24d3f429f1cc8c5aafa716c508e899/fonttools-4.68.0-cp312-cp312-win_arm64.whl", hash = "sha256:95628bbc5ea81d5761268f4c4f1ba709af510b970f363498ac801c0c1fd7a4a8", size = 2502049, upload-time = "2026-10-27T16:28:39.355Z" }, + { url = "https://files.pythonhosted.org/packages/05/ae/e2d75b7f582c1cc018ab94b8bbfbad7ab501b340debfa256f48125cae3eb/fonttools-4.68.0-cp313-cp313-macosx_10_13_universal2.whl", hash = "sha256:f15de48b82a816c72f358e772792801bd3788a156caa76e1806cb0aa8148cd61", size = 3141370, upload-time = "2026-10-27T16:28:41.721Z" }, + { url = "https://files.pythonhosted.org/packages/fe/80/73d5297441b42d42b64d91cd4fffdff1541fac6171b0aca0a167c26d1c33/fonttools-4.68.0-cp313-cp313-macosx_10_13_x86_64.whl", hash = "sha256:86d319a1feedfc75c8ff582d947b21fa1854e2829cb8cf9428784d750a05f1d9", size = 2633412, upload-time = "2026-10-27T16:28:43.612Z" }, + { url = "https://files.pythonhosted.org/packages/36/29/16501aa69a586b8e8f810d7be19584aa026a0bca46c598ac516cfdd2a31a/fonttools-4.68.0-cp313-cp313-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:04a4868dcfcc23f00bb033147d5afc1a118a992842644c3eb9da55467ab4fd74", size = 5435968, upload-time = "2026-10-27T16:28:45.651Z" }, + { url = "https://files.pythonhosted.org/packages/79/c6/4b44abbffa01f85d6d0f0b9325a712019b4e2e6208f999ac6846d8da1346/fonttools-4.68.0-cp313-cp313-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:08851f7d2b983412f490b73a7c56f17cbda1d4d85d2ad3f857564bf15b5d9475", size = 5418193, upload-time = "2026-10-27T16:28:49.238Z" }, + { url = "https://files.pythonhosted.org/packages/98/0d/15cffdf13e156bb75ee421eee319fa8c5fb91a48a9a3e84685bf786df5be/fonttools-4.68.0-cp313-cp313-musllinux_1_2_aarch64.whl", hash = "sha256:b6f7c5fabf5b1d9ca79f204664f2c3e2f8218e50aebc43e85e02df0b2d9cc313", size = 5378957, upload-time = "2026-10-27T16:28:51.889Z" }, + { url = "https://files.pythonhosted.org/packages/9f/18/174052f12611c1088b61eeaa4019a8f1aedbcb1e985c33ef8fbe0c1ba674/fonttools-4.68.0-cp313-cp313-musllinux_1_2_x86_64.whl", hash = "sha256:e3ecdb0b248cd0bf43100a8b3cb40f7b45d52ea4be8616edea5dea8fefe90229", size = 5536796, upload-time = "2026-10-27T16:28:54.445Z" }, + { url = "https://files.pythonhosted.org/packages/35/d6/892213f45f63565f3f7eca3bce484a6f52ab7f7eda61851f93b1be829d5e/fonttools-4.68.0-cp313-cp313-win32.whl", hash = "sha256:4da99b83ae17f531838bc750abf56164ff37e1543e8e9b2b5403548f50497ce0", size = 2479972, upload-time = "2026-10-27T16:28:56.749Z" }, + { url = "https://files.pythonhosted.org/packages/61/2b/6c6f578176edeec83c5e02b7e745c1c8029f8cec4e32b4898d2c573bb881/fonttools-4.68.0-cp313-cp313-win_amd64.whl", hash = "sha256:6ed796b220654b4d37ad20af73419ec7969b669dd678cab88afb602c5683a331", size = 2531658, upload-time = "2026-10-27T16:28:59.013Z" }, + { url = "https://files.pythonhosted.org/packages/3c/b9/7cc71534bc954346451003bf5c9c117af835aea681a13a2847222ccf19b2/fonttools-4.68.0-cp313-cp313-win_arm64.whl", hash = "sha256:cd4b0d00523cae1e5e1eaed1d2b85ef342cec000f58d0738e12b064d72a7bbc2", size = 2499695, upload-time = "2026-10-27T16:29:01.511Z" }, + { url = "https://files.pythonhosted.org/packages/1a/70/ee6655887166a111d0c4d7ca61f03c90b141cdc6f70c85138fd0b8bfa2e3/fonttools-4.68.0-cp314-cp314-macosx_10_15_universal2.whl", hash = "sha256:b2ee7b9c1a5e1d637073c39b0978956febad7e862a1269719f527934c65535a4", size = 3145377, upload-time = "2026-10-27T16:29:03.757Z" }, + { url = "https://files.pythonhosted.org/packages/d6/37/66e7ab417660cb2c65889eb5ad8d80cc5a7cb2c09dd1becdbe6d56bac4fe/fonttools-4.68.0-cp314-cp314-macosx_10_15_x86_64.whl", hash = "sha256:9d80813ae74b969ef2166a3040b591826e80ecd879bdeab7f13f4e8eaf08b345", size = 2634889, upload-time = "2026-10-27T16:29:05.651Z" }, + { url = "https://files.pythonhosted.org/packages/0c/ad/da985167af1988b82112cf133e826cf075cecb28d120778589265cb3125d/fonttools-4.68.0-cp314-cp314-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:a6837def5de49772b82683dadd836df214def04b307cb98d0e7723d49e7e15bb", size = 5420416, upload-time = "2026-10-27T16:29:07.502Z" }, + { url = "https://files.pythonhosted.org/packages/b7/67/e9f2eb2aed41d4ed251a3c6f66e104685f0a03d8ded7ed87f1f0fc63898c/fonttools-4.68.0-cp314-cp314-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:cbb63b26d33acc3eb61f544c22a0cadbe01682fc064b266bb0e33fa66cea853f", size = 5363969, upload-time = "2026-10-27T16:29:10.172Z" }, + { url = "https://files.pythonhosted.org/packages/e6/06/c4510ac6b43b374b1b707b99fdd0ba1d4059d2fb516584a412c0396fedef/fonttools-4.68.0-cp314-cp314-musllinux_1_2_aarch64.whl", hash = "sha256:81ea963d4390bb9e79f91a905310d91ac7ff7675d8e1c0cfc0f1712521c41a43", size = 5362373, upload-time = "2026-10-27T16:29:12.298Z" }, + { url = "https://files.pythonhosted.org/packages/e0/ef/ea0bf283a30a88a84a47078ff39f7bdefa0346a8bd7737439fd33084031f/fonttools-4.68.0-cp314-cp314-musllinux_1_2_x86_64.whl", hash = "sha256:3b9b87645bcdbfc3f30240e2758a37bbef4fdd0f708683b2bfceb60975d63e59", size = 5495553, upload-time = "2026-10-27T16:29:14.614Z" }, + { url = "https://files.pythonhosted.org/packages/95/c0/d493f8c8974859e57920ce193ffbe277423da7eebccbb9241bc31e1b5f1d/fonttools-4.68.0-cp314-cp314-win32.whl", hash = "sha256:39913c1f576af86c69519dc351302332561ee62b39fed04dcfa8019d9c2d129f", size = 2484057, upload-time = "2026-10-27T16:29:16.45Z" }, + { url = "https://files.pythonhosted.org/packages/96/41/c9ca629ffe1b883c1422540a8a11a4ea09616366ab585ab4ee9833f2cd75/fonttools-4.68.0-cp314-cp314-win_amd64.whl", hash = "sha256:1782f2cf952eb262c4126b1f57b5f0e027e71bf2b632350a033dec526f448428", size = 2536130, upload-time = "2026-10-27T16:29:18.399Z" }, + { url = "https://files.pythonhosted.org/packages/17/40/f6264c988076393db3e0b00d6d0ea2a53ce40d445e8a196be241d16d362c/fonttools-4.68.0-cp314-cp314-win_arm64.whl", hash = "sha256:f0b39924698a85bee107184a3c2ee9ae27248144977160534f7d558831db50cf", size = 2508386, upload-time = "2026-10-27T16:29:20.318Z" }, + { url = "https://files.pythonhosted.org/packages/c5/d8/a4d710362f3f23aecc16015eeed1d602187189af7f63a63546233ad2fa73/fonttools-4.68.0-cp314-cp314t-macosx_10_15_universal2.whl", hash = "sha256:956230858b7a7bb9f5e0971ae07dc64b6155307304acd8c62aedca30e6aae874", size = 3219047, upload-time = "2026-10-27T16:29:22.337Z" }, + { url = "https://files.pythonhosted.org/packages/4d/4d/0d516a1d3d34e84c56237ba1d8dfed8575c350b0e8d39aa30b518d089daf/fonttools-4.68.0-cp314-cp314t-macosx_10_15_x86_64.whl", hash = "sha256:b637d82a488116da2869afbddccfd93ca14e3c47b404c8ec5a9bad9d4381b08c", size = 2668060, upload-time = "2026-10-27T16:29:24.536Z" }, + { url = "https://files.pythonhosted.org/packages/4d/40/56893c685cb1ee7d7ee30d0ac94adf92020787cf5a8e15c2e4e6944f0bec/fonttools-4.68.0-cp314-cp314t-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:419bf78474adaa6fb320a06822f4f941b19345d598a9de379d6e2f0e4d819df6", size = 5593080, upload-time = "2026-10-27T16:29:26.51Z" }, + { url = "https://files.pythonhosted.org/packages/41/b6/bc0bbad6f1aef22519c4e731c6dc8655e92c6f7204277ec36502ea7e76cd/fonttools-4.68.0-cp314-cp314t-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:fe80d05dd85e6ac58d0cd4fa10ddb96252a6f1046ee0fd00cd1d48f0c25921ac", size = 5399669, upload-time = "2026-10-27T16:29:28.969Z" }, + { url = "https://files.pythonhosted.org/packages/c1/b4/4404a91dc699dd430732d366996cef6d8c8f61c758b5e544a76903db580a/fonttools-4.68.0-cp314-cp314t-musllinux_1_2_aarch64.whl", hash = "sha256:cea1969b9bfe230cb3fd66f1c66da5a63a9a917d9cd25a0839524d3d1a88b40e", size = 5460446, upload-time = "2026-10-27T16:29:31.15Z" }, + { url = "https://files.pythonhosted.org/packages/56/09/66876a0d1a01f7c16eb9f7c8ebcf064ebbd6b249f837691cb5a35780316a/fonttools-4.68.0-cp314-cp314t-musllinux_1_2_x86_64.whl", hash = "sha256:fde43b9785c7f105aa563b6b5481edabea4ab6be7d9ef14ff4d76225950923e8", size = 5495792, upload-time = "2026-10-27T16:29:33.455Z" }, + { url = "https://files.pythonhosted.org/packages/15/7d/1d702dc628e6f5b0d63be906025afc5e5b487cee48128943bb0a71d2bcb3/fonttools-4.68.0-cp314-cp314t-win32.whl", hash = "sha256:a325e6d7f96f20b3004658c5ed17ccfce5b66277b43ccd883035c61be6945275", size = 2517805, upload-time = "2026-10-27T16:29:36.929Z" }, + { url = "https://files.pythonhosted.org/packages/f3/42/8fa0741e08ae16be66a0d2e6d67f918fa0f2961bad70ccef84f9af0767cc/fonttools-4.68.0-cp314-cp314t-win_amd64.whl", hash = "sha256:e80ac66627a01698675ce6e98d90ea9817d1e06644129ec8b0093f66ff1cbeac", size = 2567666, upload-time = "2026-10-27T16:29:40.081Z" }, + { url = "https://files.pythonhosted.org/packages/13/2f/b73cf496570e8d2c66a6099f825505283ef8e944bd588e0b63036d6b2412/fonttools-4.68.0-cp314-cp314t-win_arm64.whl", hash = "sha256:83bd3d558c15f4e1d87b8f3fcfb337c2bce63c7e5c05212abc8fdcf15e216c54", size = 2532338, upload-time = "2026-10-27T16:29:42.073Z" }, + { url = "https://files.pythonhosted.org/packages/24/af/dda3fbb1da1d3cf5bf8cd6a924ab799fd2a483382df7de6f9770b543ef4f/fonttools-4.68.0-cp315-cp315-macosx_10_15_universal2.whl", hash = "sha256:ab6505dfbe3199453d8542b25f55c1ea3260d38029d9db58c0c7304d438abdc0", size = 3139761, upload-time = "2026-10-27T16:29:44.288Z" }, + { url = "https://files.pythonhosted.org/packages/4f/47/4294eececd4e7c38bcb98c78be96003b4bbfc459c30adec13104b83112c1/fonttools-4.68.0-cp315-cp315-macosx_10_15_x86_64.whl", hash = "sha256:03c9ef1272e464747f67c623f76449b888da6d5de7fb46275bb36488b9ceedba", size = 2633458, upload-time = "2026-10-27T16:29:46.202Z" }, + { url = "https://files.pythonhosted.org/packages/df/3d/3ee0940101d12a68cfd5e3d91380933878d99bd082eb9df348eb768b0375/fonttools-4.68.0-cp315-cp315-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:5083a456d68d2995ab78600ccb18ec08c42f9bc14f0ef80c12e2e5b11cc371ce", size = 5417615, upload-time = "2026-10-27T16:29:48.404Z" }, + { url = "https://files.pythonhosted.org/packages/a1/ff/a97925464230d9c39c94e4cd7c5690db950373b34317e7f0ffe7b64b8d49/fonttools-4.68.0-cp315-cp315-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:b65c2f3c4449383429507a7e1b372bd759f60a2df8ddc8d6c1cf9010585576b1", size = 5380964, upload-time = "2026-10-27T16:29:50.809Z" }, + { url = "https://files.pythonhosted.org/packages/ee/51/f467501a75bb383f1acbcc7a7d8ce196c02bea770e2bdc62d269c18a31a1/fonttools-4.68.0-cp315-cp315-musllinux_1_2_aarch64.whl", hash = "sha256:039142b75a068c93f8596ef68787f82c74940b9153d24c224ab31e750eb083a2", size = 5357813, upload-time = "2026-10-27T16:29:52.892Z" }, + { url = "https://files.pythonhosted.org/packages/ba/1b/ebb9c32a19a89e8c5cfc63833d7edb70019d3184281452cbc1a24cadce57/fonttools-4.68.0-cp315-cp315-musllinux_1_2_x86_64.whl", hash = "sha256:1dde2f0ec3fd0fb43cf818593b38a8202eb7f3b59cae86289a8b19fbf6f6313b", size = 5505577, upload-time = "2026-10-27T16:29:54.95Z" }, + { url = "https://files.pythonhosted.org/packages/18/c3/080bf6342ed37c4304cf222cf80afd2a145cd8f37a758d812cda81ba3e12/fonttools-4.68.0-cp315-cp315-win32.whl", hash = "sha256:4bc352572abc800177fba513995c01b1cfbe8c810ac2506cbf29f0795ae061c5", size = 2482863, upload-time = "2026-10-27T16:29:57.029Z" }, + { url = "https://files.pythonhosted.org/packages/f2/b7/8ce34a43072ac7b1a937c5a0ee8e024db5d774bdf07708bb7593f3e5540d/fonttools-4.68.0-cp315-cp315-win_amd64.whl", hash = "sha256:9b6e4ae9763c67df2b0f4c4f1a0b18e47ee6572eb602b82e798acc88b64a6137", size = 2535110, upload-time = "2026-10-27T16:29:58.999Z" }, + { url = "https://files.pythonhosted.org/packages/6d/35/d58d4ac6dd3de099c11c42486fc4a51b1ccb2c1c3259b19956e9244d5623/fonttools-4.68.0-cp315-cp315-win_arm64.whl", hash = "sha256:e00bfc6fc0158f156d34f923829e6f62ca00df2e0ea42ba3b0db4089b6d921c2", size = 2507240, upload-time = "2026-10-27T16:30:01.439Z" }, + { url = "https://files.pythonhosted.org/packages/29/16/034e1eee6264cd14ef4404f70c299a9b0570d83aca24a65ddf3a16ab3b42/fonttools-4.68.0-cp315-cp315t-macosx_10_15_universal2.whl", hash = "sha256:0841a4dac69efda309fca2ed0834186ee66d39787f1d26430ed7f779a699a096", size = 3210579, upload-time = "2026-10-27T16:30:03.601Z" }, + { url = "https://files.pythonhosted.org/packages/17/37/626504f427aa92d1d872336eab454ad71b6bc079063cc1f4af314995a48e/fonttools-4.68.0-cp315-cp315t-macosx_10_15_x86_64.whl", hash = "sha256:93afd51f30ae38f7d826f4632eb799a06bd94e1fb4bc59fdce6d291e93491027", size = 2664302, upload-time = "2026-10-27T16:30:05.639Z" }, + { url = "https://files.pythonhosted.org/packages/62/13/bc455cb7f8e467ed4ea1146c1ecad76d6c78102031d937cae71efc176c42/fonttools-4.68.0-cp315-cp315t-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:82b8024fbdcddc399b1e21575b9b82f46b8391e711a44d6bd4a1616b0aca3681", size = 5570974, upload-time = "2026-10-27T16:30:07.457Z" }, + { url = "https://files.pythonhosted.org/packages/23/06/33ce5c7d3bb59dda92fbf669053d5fdaac2e0d566390fdf20746ee83f39f/fonttools-4.68.0-cp315-cp315t-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:9fee5880486c3fd6f45f5a1679a9f4291ae74e503d725de7f1e30efa2d627440", size = 5392631, upload-time = "2026-10-27T16:30:09.587Z" }, + { url = "https://files.pythonhosted.org/packages/3c/c0/414f221a7da04ddb6d30a8171ecdba36e2ab4351c5c62ec319621ee76494/fonttools-4.68.0-cp315-cp315t-musllinux_1_2_aarch64.whl", hash = "sha256:4be32afa53ba3dea243b95b62e9b2e6ed6cf7096acd741d379c12f4113694ddc", size = 5438942, upload-time = "2026-10-27T16:30:11.966Z" }, + { url = "https://files.pythonhosted.org/packages/98/71/797da0cb0fdb3839b4aa65d30854b850832c74dc14ce0ba5bfe812c6a008/fonttools-4.68.0-cp315-cp315t-musllinux_1_2_x86_64.whl", hash = "sha256:6e73d4b1526343b45bbafa9badc0396a49d902f9a1774666ed4244f1df717629", size = 5485323, upload-time = "2026-10-27T16:30:15.074Z" }, + { url = "https://files.pythonhosted.org/packages/c1/47/a6989b4de0f427817f2e436f69664acc3aa9600d3df5f0eb3444cd073f14/fonttools-4.68.0-cp315-cp315t-win32.whl", hash = "sha256:573ccce8b5fd9b6952dcbc5d896255041ab330cc33520d13b1266fca859d8aba", size = 2514943, upload-time = "2026-10-27T16:30:17.326Z" }, + { url = "https://files.pythonhosted.org/packages/5e/38/7311ade45b67dc2c2a4f9d1af4c18285f92591235e8c6cab4f9cecd36516/fonttools-4.68.0-cp315-cp315t-win_amd64.whl", hash = "sha256:6aea5cdbc32f5fd202cf8dec053a784b08692f8b1992af80eba32bef82bf84c7", size = 2563334, upload-time = "2026-10-27T16:30:19.883Z" }, + { url = "https://files.pythonhosted.org/packages/8a/ee/fd08713d15bf0f6487a72e415e91ef4a6b52ca96fa8f879533107a41e18d/fonttools-4.68.0-cp315-cp315t-win_arm64.whl", hash = "sha256:b612613f0165f560571c4e372af0ebbb1c4b36067de409922dcea349beb8bb20", size = 2528487, upload-time = "2026-10-27T16:30:22.222Z" }, + { url = "https://files.pythonhosted.org/packages/35/74/4ea331ecc65ba05a58b1e4e8ccd604c8a15565f876dfbf93fc2a2ea2e22b/fonttools-4.68.0-py3-none-any.whl", hash = "sha256:d76f8fdcb6936f5e1045a0c9dd721a4bf11112e60d1e243bfa1d39f06a8b3104", size = 1248357, upload-time = "2026-10-27T16:30:24.389Z" }, +] + +[[package]] +name = "fsspec" +version = "2026.9.0" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/77/cd/9be253869fc42e764de7f3dedd6969af7d44ff9c3375214a3442a6f3fc08/fsspec-2026.9.0.tar.gz", hash = "sha256:0f08147951c8cb31d844c3547d631053b127863b60be04cf06e121333ee0e2fe", size = 333545, upload-time = "2026-09-18T17:50:42.825Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/6c/c0/a98505f18594f1bce828bb159cec0fcf9860562f1a2c85913409fc8f3d9e/fsspec-2026.9.0-py3-none-any.whl", hash = "sha256:8dd6e646e99ea382bd85f97a45e6b526a442d79423a7dc673f1e2756d05fcb5f", size = 221738, upload-time = "2026-09-18T17:50:41.341Z" }, +] + +[[package]] +name = "h11" +version = "0.16.0" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/01/ee/02a2c011bdab74c6fb3c75474d40b3052059d95df7e73351460c8588d963/h11-0.16.0.tar.gz", hash = "sha256:4e35b956cf45792e4caa5885e69fba00bdbc6ffafbfa020300e549b208ee5ff1", size = 101250, upload-time = "2025-04-24T03:35:25.427Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/04/4b/29cac41a4d98d144bf5f6d33995617b185d14b22401f75ca86f384e87ff1/h11-0.16.0-py3-none-any.whl", hash = "sha256:63cf8bbe7522de3bf65932fda1d9c2772064ffb3dae62d55932da54b31cb6c86", size = 37515, upload-time = "2025-04-24T03:35:24.344Z" }, +] + +[[package]] +name = "hf-xet" +version = "1.8.0" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/2f/18/d91c20f815df6cc496a5d0ad8db6d274dad5c0cfcd59babfa2440a9aba44/hf_xet-1.8.0.tar.gz", hash = "sha256:10ed23b51dacf7917d64f335c25048deb470efa755e9989b218fc0531386138e", size = 1013014, upload-time = "2026-10-30T10:04:33.741Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/2c/18/db7ebc58820d82fd5506aa405a75611e7a4398f573dc436f7387603bf63d/hf_xet-1.8.0-cp314-cp314t-macosx_10_12_x86_64.whl", hash = "sha256:c74e3751cbcdae9222c5170e8d788e83201a03dc5e5f4308a20f17a5b6023b56", size = 3883873, upload-time = "2026-10-30T10:04:03.24Z" }, + { url = "https://files.pythonhosted.org/packages/37/bd/6c9ed89aafbeb4a7ee0d76d6b7b9365221e3e7ee47958aeecdc2c60a0511/hf_xet-1.8.0-cp314-cp314t-macosx_11_0_arm64.whl", hash = "sha256:56735c2d4f416d7473fc986586b26a8756e067832ca4b7fdaad32a0d6f2ce740", size = 3594510, upload-time = "2026-10-30T10:04:05.364Z" }, + { url = "https://files.pythonhosted.org/packages/0e/1f/c26154b3a053b637c0984db0db9fdacbbfc257bdc2e95e9bc0782b745b0c/hf_xet-1.8.0-cp314-cp314t-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:3ecfe2934448cca55a4c42732b60f985266aecb241905945219b3d388aceb7ba", size = 4281780, upload-time = "2026-10-30T10:04:07.35Z" }, + { url = "https://files.pythonhosted.org/packages/e3/9e/d84c31f8e3acc286c9c6a4fd85f7f332e3db40636595f0de5ad776b69beb/hf_xet-1.8.0-cp314-cp314t-manylinux_2_28_aarch64.whl", hash = "sha256:7edeb21c5d381903ee5f4857fc746fa7b52a4b374035e8811ecfb67ee27add0c", size = 4060223, upload-time = "2026-10-30T10:04:09.029Z" }, + { url = "https://files.pythonhosted.org/packages/a4/70/c142e4ad8a679ff6ffe40964a9d017fa0211aabca9ca4d3d547c3a2fc165/hf_xet-1.8.0-cp314-cp314t-musllinux_1_2_aarch64.whl", hash = "sha256:07a06d7481702616726c2d8578cfb00052a5b3f21beed6a674592bc837a10330", size = 4263619, upload-time = "2026-10-30T10:04:11.085Z" }, + { url = "https://files.pythonhosted.org/packages/9d/08/92e03e3ac4eaeab046359093c447c03c8c30b577c2b1e1687dc747654f1c/hf_xet-1.8.0-cp314-cp314t-musllinux_1_2_x86_64.whl", hash = "sha256:f42940869572dc2b05dfe06f4cc398c6291503e086ddc4eddfb7a6ab9409ad01", size = 4493381, upload-time = "2026-10-30T10:04:13.022Z" }, + { url = "https://files.pythonhosted.org/packages/61/63/a512b2906d0f92b5d325f58169d1fd8c3202a9ee61bd8405ec78c67e5315/hf_xet-1.8.0-cp314-cp314t-win_amd64.whl", hash = "sha256:23c262e9f5e137b18dadbce60becfeea84156b0d43119c320bffe260d32cd2fa", size = 3901344, upload-time = "2026-10-30T10:04:15.051Z" }, + { url = "https://files.pythonhosted.org/packages/2d/5e/b453f82690cb36f176c4b01e64d6d32757fabbf34dbb9bcbdf747e429a79/hf_xet-1.8.0-cp314-cp314t-win_arm64.whl", hash = "sha256:62b056aedb89ac809f78940baf465fe553b70c8b25ee56890fbab3039fffec1b", size = 3731681, upload-time = "2026-10-30T10:04:16.855Z" }, + { url = "https://files.pythonhosted.org/packages/97/74/d89d5235524bc14d8519e12ca8bc5204f45009ae83d63a410f52c29eb4a8/hf_xet-1.8.0-cp38-abi3-macosx_10_12_x86_64.whl", hash = "sha256:e00d8f60170cd3dd55d98306e41dbaf28b27ff1bcfefbfa84279d197221be83e", size = 3875002, upload-time = "2026-10-30T10:04:18.958Z" }, + { url = "https://files.pythonhosted.org/packages/d4/f2/ef25e2085e92dcaeaa675331e443923aed2006f6fe2d984e15d8b8d40766/hf_xet-1.8.0-cp38-abi3-macosx_11_0_arm64.whl", hash = "sha256:3e8934784425781f0f779b91044df441f3a7af4dab47375d7e3aeaecbb8eecb4", size = 3600947, upload-time = "2026-10-30T10:04:20.906Z" }, + { url = "https://files.pythonhosted.org/packages/f8/1e/1e8ecc05ef034e9a95c18ed32d2ec07905a7cd2ca611a9d88e25ee1f4850/hf_xet-1.8.0-cp38-abi3-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:29dab7c904cd1914a0439c6e1306c12c7497c8bd32290fbe563ebf51ca4b6d6d", size = 4292483, upload-time = "2026-10-30T10:04:22.875Z" }, + { url = "https://files.pythonhosted.org/packages/35/c4/127a1548df019ea4c80f06e685f263e146075143335a02e78a3c41dbc4eb/hf_xet-1.8.0-cp38-abi3-manylinux_2_28_aarch64.whl", hash = "sha256:8800e3444770f2c630b361bc6d5c6ebb770b7830646cf632fbb79d0e5bbc9e7f", size = 4069340, upload-time = "2026-10-30T10:04:24.664Z" }, + { url = "https://files.pythonhosted.org/packages/a5/d5/281cd82736c3e181e597588589d2204d3a5c388a464d4a2a8c546995bae9/hf_xet-1.8.0-cp38-abi3-musllinux_1_2_aarch64.whl", hash = "sha256:a584a7b3268da62f23c3187cc7201ca01f7b610985a36b1b6119c66bccb0383c", size = 4272344, upload-time = "2026-10-30T10:04:26.519Z" }, + { url = "https://files.pythonhosted.org/packages/48/77/886f09e91a5eadb054ce03b8855323cd2a72f20db76ab4119e0ecda65eef/hf_xet-1.8.0-cp38-abi3-musllinux_1_2_x86_64.whl", hash = "sha256:1ebaf248c9bef8bb0c9d1d8d71db1fdf0cac14f922cc7a936a81adbf12eb5f01", size = 4500689, upload-time = "2026-10-30T10:04:28.394Z" }, + { url = "https://files.pythonhosted.org/packages/97/4d/2c76a992b370ff77d0b5219275779f8c9a7dabb8ee649deb265dfed84d0f/hf_xet-1.8.0-cp38-abi3-win_amd64.whl", hash = "sha256:92188598dedc1e1f5609bd718d4db8687a6c89915553912571e2f587892eecec", size = 3911559, upload-time = "2026-10-30T10:04:30.165Z" }, + { url = "https://files.pythonhosted.org/packages/87/91/fdb1454a43eff8011047d176ada2f0fb4926cf3d74b87357ff35793f6d98/hf_xet-1.8.0-cp38-abi3-win_arm64.whl", hash = "sha256:42b66fdc688eaa4ff3eb77dc9275e5a0f1258ecc9fdd083ddaf43bb49bec925b", size = 3738458, upload-time = "2026-10-30T10:04:31.882Z" }, +] + +[[package]] +name = "httpcore" +version = "1.0.9" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "certifi" }, + { name = "h11" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/06/94/82699a10bca87a5556c9c59b5963f2d039dbd239f25bc2a63907a05a14cb/httpcore-1.0.9.tar.gz", hash = "sha256:6e34463af53fd2ab5d807f399a9b45ea31c3dfa2276f15a2c3f00afff6e176e8", size = 85484, upload-time = "2025-04-24T22:06:22.219Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/7e/f5/f66802a942d491edb555dd61e3a9961140fd64c90bce1eafd741609d334d/httpcore-1.0.9-py3-none-any.whl", hash = "sha256:2d400746a40668fc9dec9810239072b40b4484b640a8c38fd654a024c7a1bf55", size = 78784, upload-time = "2025-04-24T22:06:20.566Z" }, +] + +[[package]] +name = "httpcore2" +version = "2.16.0" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "h11" }, + { name = "truststore" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/c0/61/afa9d6b7452f9eca73f73d45324eb0bb52f098fd5e6049bf6b8263be9b4e/httpcore2-2.16.0.tar.gz", hash = "sha256:c72cd0413563b17e1cd8354677dff887466f8ec2551679759e93126534428643", size = 69886, upload-time = "2026-10-21T07:09:45.31Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/af/c2/ce417c397afa5181f0e3c6ff01a4d01639414e1adef08730d97571297151/httpcore2-2.16.0-py3-none-any.whl", hash = "sha256:5c4358319f2602d984ad93182c9fee5160f22e9da7babdda870a5e7c4e2dc34f", size = 85070, upload-time = "2026-10-21T07:09:42.802Z" }, +] + +[[package]] +name = "httpx" +version = "0.28.1" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "anyio" }, + { name = "certifi" }, + { name = "httpcore" }, + { name = "idna" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/b1/df/48c586a5fe32a0f01324ee087459e112ebb7224f646c0b5023f5e79e9956/httpx-0.28.1.tar.gz", hash = "sha256:75e98c5f16b0f35b567856f597f06ff2270a374470a5c2392242528e3e3e42fc", size = 141406, upload-time = "2024-12-06T15:37:23.222Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/2a/39/e50c7c3a983047577ee07d2a9e53faf5a69493943ec3f6a384bdc792deb2/httpx-0.28.1-py3-none-any.whl", hash = "sha256:d909fcccc110f8c7faf814ca82a9a4d816bc5a6dbfea25d6591d6985b8ba59ad", size = 73517, upload-time = "2024-12-06T15:37:21.509Z" }, +] + +[[package]] +name = "httpx2" +version = "2.16.0" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "anyio", marker = "sys_platform != 'emscripten'" }, + { name = "httpcore2", marker = "sys_platform != 'emscripten'" }, + { name = "httpx2-jsfetch", marker = "python_full_version >= '3.12' and sys_platform == 'emscripten'" }, + { name = "idna" }, + { name = "truststore", marker = "sys_platform != 'emscripten'" }, + { name = "typing-extensions", marker = "python_full_version < '3.13'" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/43/73/5ca2cedf5e4e50423d1c258877865a6b62793633b561c58499f8aeb33b7e/httpx2-2.16.0.tar.gz", hash = "sha256:0d54c98128e8f2ff7fbbde0fca8c359e5f032edeaaaac443cdb7f31f685bcd7f", size = 103116, upload-time = "2026-10-21T07:09:46.348Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/7b/3d/2e4bc4fd1ef9770871669adb348d612a6e4786b303a23f9935b81c48dd60/httpx2-2.16.0-py3-none-any.whl", hash = "sha256:fd2c5d7909e8d86fe92d53020c310ca0e4e431b838c207c5754b9bbf0308d49a", size = 97880, upload-time = "2026-10-21T07:09:44.01Z" }, +] + +[[package]] +name = "httpx2-jsfetch" +version = "1.0" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/cd/c4/0e5636363151a2a1795e0a77617168b9ca438e1748ec05fc9b5687f93d64/httpx2_jsfetch-1.0.tar.gz", hash = "sha256:70a0e3eabfef7cce5ad9c629f7d01ca05e418f586646f4ddf14782e4c1454c60", size = 6872, upload-time = "2026-08-07T00:13:07.492Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/9b/43/832f631d32e4f1211caa2ba368317739fe71f0b8530e4c9d15dc454bac2a/httpx2_jsfetch-1.0-py3-none-any.whl", hash = "sha256:cb916b707601e69a07721aabc8f3f6659be3a6893bc1ff5c6f9e02241df2da32", size = 6382, upload-time = "2026-08-07T00:13:06.567Z" }, +] + +[[package]] +name = "huggingface-hub" +version = "2.4.0" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "click" }, + { name = "filelock" }, + { name = "fsspec" }, + { name = "hf-xet", marker = "platform_machine == 'AMD64' or platform_machine == 'ARM64' or platform_machine == 'aarch64' or platform_machine == 'amd64' or platform_machine == 'arm64' or platform_machine == 'x86_64'" }, + { name = "httpx2" }, + { name = "packaging" }, + { name = "pyyaml" }, + { name = "tqdm" }, + { name = "typing-extensions" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/b5/3b/2fcef1eabe52aec90fc8cd253cc154da1d02a446d7bbcdec59b4a51d4be7/huggingface_hub-2.4.0.tar.gz", hash = "sha256:13a6158b9e3393d9d3f248ce99074d50c2a0a1449fb9d8e2f710751e2084c188", size = 1066194, upload-time = "2026-10-29T15:30:12.321Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/d4/55/ff47f1b082ecf5f43f4bc1e3b932d68eead171dabbbd1a92666c686301d7/huggingface_hub-2.4.0-py3-none-any.whl", hash = "sha256:42ecfd93ede47bdd333daebbf66a32d7bfe2e18752754fe4d2d4f98a0179d801", size = 857021, upload-time = "2026-10-29T15:30:10.421Z" }, +] + +[[package]] +name = "idna" +version = "3.20" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/f5/08/8eea9d4b8302028f3abb2c0813953f7aec26d33b7a8960ed760e65ff29fa/idna-3.20.tar.gz", hash = "sha256:a7db850025b95ded1eae8a46181a1a6c56c92c96f0e2b005d9ff8dc0210cab44", size = 216463, upload-time = "2026-09-17T14:11:04.752Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/58/a2/bb081bab032533a855d44de1d56f8e8426114ff1ba5d1f07a438a0a654f8/idna-3.20-py3-none-any.whl", hash = "sha256:ab7ae7122974553370f0bdb919e1a960b2cd1bc1ef0276416d896db81c14582c", size = 69583, upload-time = "2026-09-17T14:11:03.168Z" }, +] + +[[package]] +name = "iniconfig" +version = "2.3.1" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/01/e1/2069291243c926a2ff1cd706c7f3eeb9b62144bf60f77c9fb9ff2fb26bd3/iniconfig-2.3.1.tar.gz", hash = "sha256:67f4b9c50da0dedf52af349e7749a80a9057a5031199791b906c3bb3ae878960", size = 21209, upload-time = "2026-10-06T22:48:38.076Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/56/43/4ca9e49d27a1fcf6bece6f6aec0ea46bb9112489b93d4b688fb415457bdb/iniconfig-2.3.1-py3-none-any.whl", hash = "sha256:9121e2c1fdb355232495be3194c8dfe87ccc2d5dee45947b78e68f499790d7a7", size = 7552, upload-time = "2026-10-06T22:48:36.959Z" }, +] + +[[package]] +name = "joblib" +version = "1.6.0" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "cloudpickle" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/d5/1d/537ab090f302b838943a1b56497dd53059b9a9b46a074936470173a2e207/joblib-1.6.0.tar.gz", hash = "sha256:2ccc96785b12046c08fd6d55839c12857831b54a3c1673ffadd2f04bfc4eda03", size = 327903, upload-time = "2026-08-31T09:39:04.122Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/18/53/84099323c2ec4be98d935f63c033ac4151ee83836ca1050ede3b3aadf155/joblib-1.6.0-py3-none-any.whl", hash = "sha256:3dbbf9f6e4b592a2357b854608e980fe6390d131d7a82f011a377ef2ebef7aba", size = 306115, upload-time = "2026-08-31T09:39:02.298Z" }, +] + +[[package]] +name = "kiwisolver" +version = "1.5.1" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/ba/07/bd78e6a8fae171ea041ef5bba3ed21a003522fa088834b069b1909981f30/kiwisolver-1.5.1.tar.gz", hash = "sha256:f1303ef2eec81262a4b708c3e858afe58d7c75ad91c1c05266eda7673369859a", size = 104395, upload-time = "2026-08-28T10:28:27.153Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/94/7b/2de6908edc668427c149af5f93112e931f87e1fa4cab80bac32c5844dccc/kiwisolver-1.5.1-cp311-cp311-macosx_10_9_universal2.whl", hash = "sha256:b3d78f7bb2b9d9a30345be1474b9aaa8685430b54afb51ba3639b5c6c11e9ed6", size = 123364, upload-time = "2026-08-28T10:25:04.359Z" }, + { url = "https://files.pythonhosted.org/packages/8a/24/e70914415c77c97be7e22c80a0740869cb7428768cc380fdcdf6703e7084/kiwisolver-1.5.1-cp311-cp311-macosx_10_9_x86_64.whl", hash = "sha256:5025e36fb4fb275cef0a4e30dbb11cb4ae61d1c83deb90189cb5d7e4cafd6b55", size = 66558, upload-time = "2026-08-28T10:25:05.506Z" }, + { url = "https://files.pythonhosted.org/packages/e8/2b/8b08b11833db4d475b8ef1f36174f8d8a7abd31bedd7e794be78e8814b48/kiwisolver-1.5.1-cp311-cp311-macosx_11_0_arm64.whl", hash = "sha256:dc1a26b8e53395a01c2c611e58602fa47461f136fba7cd5542e6db6d64be1839", size = 64071, upload-time = "2026-08-28T10:25:06.7Z" }, + { url = "https://files.pythonhosted.org/packages/89/00/05c2d0369ac322d22d5c05f84b5c4a6856fa6207fbae42869108a28f0383/kiwisolver-1.5.1-cp311-cp311-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:95a02752aa032eef4aed01cda6d9b687c669bd0396bf4519eef8bba22a286720", size = 1438206, upload-time = "2026-08-28T10:25:08.254Z" }, + { url = "https://files.pythonhosted.org/packages/c0/05/c941a139f27438c1910d630fdc3ccfdab7c8407c72052299ead12ece086e/kiwisolver-1.5.1-cp311-cp311-manylinux_2_24_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:719a35fa1156db3640555f95ebb94f60a444e64d1c69626b0edef5df78eba225", size = 1248975, upload-time = "2026-08-28T10:25:10.053Z" }, + { url = "https://files.pythonhosted.org/packages/58/a1/2669ee5512e39b9d4de25faacaedf788c957f93730c5f7c63993ec4f5933/kiwisolver-1.5.1-cp311-cp311-manylinux_2_24_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:febcce10f2bcdbb80b4ea919238a6a4ac13dbc4c7cadbe8d5d75c3682f8b5404", size = 1266301, upload-time = "2026-08-28T10:25:11.754Z" }, + { url = "https://files.pythonhosted.org/packages/28/b8/353f52f2c7f861a9e90cd2e8f90f85b3ad03060835f823e08298d094c463/kiwisolver-1.5.1-cp311-cp311-manylinux_2_24_s390x.manylinux_2_28_s390x.whl", hash = "sha256:1d852545c4d0e35a72728d072cbaa59e2fa7dd84bdf01e068d670dd0ceb58eb6", size = 1319708, upload-time = "2026-08-28T10:25:13.559Z" }, + { url = "https://files.pythonhosted.org/packages/21/0e/14b83200eadc2c1d63b76bac01c1813bf072aecf567429f303e00b70258e/kiwisolver-1.5.1-cp311-cp311-manylinux_2_39_riscv64.whl", hash = "sha256:2e10ae1bba1899188b33557c10d73affcc12033edd18adddb57d209039976a4c", size = 971720, upload-time = "2026-08-28T10:25:14.934Z" }, + { url = "https://files.pythonhosted.org/packages/86/91/9d43d84d23b1cbff72a142d387ead1ea03db0cba8ff86ed5335addad3cc9/kiwisolver-1.5.1-cp311-cp311-musllinux_1_2_aarch64.whl", hash = "sha256:b69602970994a2ed8bbfa78c2f0394a7435226c6040489702d9f0a0ad0c07052", size = 2200119, upload-time = "2026-08-28T10:25:16.636Z" }, + { url = "https://files.pythonhosted.org/packages/ff/7c/f2bd9616f27ffb5e17cecc0baa5d0bbcee7e55aeddc0ccc871d69e2fc3ee/kiwisolver-1.5.1-cp311-cp311-musllinux_1_2_ppc64le.whl", hash = "sha256:d50de98e8d807dc31822fff96f50293163a62418eb65487a21b42713d72ed0b7", size = 2295005, upload-time = "2026-08-28T10:25:18.374Z" }, + { url = "https://files.pythonhosted.org/packages/ba/17/ee671b72bf8f46a08379d4392c65582541759a542428197562f2898294ad/kiwisolver-1.5.1-cp311-cp311-musllinux_1_2_riscv64.whl", hash = "sha256:3221f78211074f561c44ca42eac0619828171bec15a2c4cf6f7747d07df76e8e", size = 1960982, upload-time = "2026-08-28T10:25:19.893Z" }, + { url = "https://files.pythonhosted.org/packages/2e/f7/0e26b4c05bee3bdb0f048dfa305e4fe701999ea17b51e9c616ef91035bbe/kiwisolver-1.5.1-cp311-cp311-musllinux_1_2_s390x.whl", hash = "sha256:0ba9527afc80ae3d7814ed98b6572d02bf85eaf48065678342c5f0c6dab7a8c7", size = 2464918, upload-time = "2026-08-28T10:25:21.65Z" }, + { url = "https://files.pythonhosted.org/packages/ae/62/6eb431133d30ce656ac1e5ff72fac70dd34d54c3984f4011b9ac8bf77d54/kiwisolver-1.5.1-cp311-cp311-musllinux_1_2_x86_64.whl", hash = "sha256:e12dfea7f5fc2a34a9080efbf79c4c44eb380ec5b9c6fea09407e08f0d1e941d", size = 2270967, upload-time = "2026-08-28T10:25:23.643Z" }, + { url = "https://files.pythonhosted.org/packages/bf/9f/6f9e489c188200e6fb3193935501894811e8c97577c8ffe9033589bf3521/kiwisolver-1.5.1-cp311-cp311-win_amd64.whl", hash = "sha256:1a7587dc335f2c0f5bd577fd0540bd16c66006bdb60f759a1059f025e6c4f071", size = 70744, upload-time = "2026-08-28T10:25:25.061Z" }, + { url = "https://files.pythonhosted.org/packages/c6/6d/dfc430d1d43957061599adea3f08ea982bb6f4ab601a8c974bedcf2ba850/kiwisolver-1.5.1-cp311-cp311-win_arm64.whl", hash = "sha256:e4e4523d6f336708d732516e6cfca7796cf3d96c9474eb5aecf6165f2f1fefc3", size = 68404, upload-time = "2026-08-28T10:25:26.186Z" }, + { url = "https://files.pythonhosted.org/packages/6b/9b/65b302742389c6f96f2956bef5decf26011309feb2fc5d79613af18adea4/kiwisolver-1.5.1-cp312-cp312-macosx_10_13_universal2.whl", hash = "sha256:63fb7294b768f444eb4b068965f2662f28c2fd4161e23bd60fcf3ff27b74c046", size = 123876, upload-time = "2026-08-28T10:25:27.44Z" }, + { url = "https://files.pythonhosted.org/packages/71/74/c21f339956f6f691b2ed7e31d5f3ae767304df6c460192739fc830853051/kiwisolver-1.5.1-cp312-cp312-macosx_10_13_x86_64.whl", hash = "sha256:0ebdef3eae5336568147c39a55be6a2036ffde53faa9ca2d978989ae7c2da12c", size = 66487, upload-time = "2026-08-28T10:25:28.728Z" }, + { url = "https://files.pythonhosted.org/packages/84/e5/bdb34e21523e01dceda064d63713f3bdec91388af24fba1eca7ea5e85864/kiwisolver-1.5.1-cp312-cp312-macosx_11_0_arm64.whl", hash = "sha256:1798e83840c3f627246104c4d8a9639c60fa068adf9ce92b61791781fa8a68c1", size = 64660, upload-time = "2026-08-28T10:25:30.071Z" }, + { url = "https://files.pythonhosted.org/packages/fc/f4/dadfec469313c7f428efa7e84b4aba9732f813c13ea7131a24b7b008ef57/kiwisolver-1.5.1-cp312-cp312-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:34633ecf50d16187ab8e5528b7a2530f2feb4e23f300db4672538b51cfc5cd38", size = 1477929, upload-time = "2026-08-28T10:25:31.495Z" }, + { url = "https://files.pythonhosted.org/packages/6f/35/09c58daac34e6f6ea5c6dee0094b422118e5a7c265586008a95fd135ac5f/kiwisolver-1.5.1-cp312-cp312-manylinux_2_24_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:d27c2123977cb9269c30a49ba45f03a4323017ef693e19db4ec9dbe1299a3002", size = 1278499, upload-time = "2026-08-28T10:25:33.375Z" }, + { url = "https://files.pythonhosted.org/packages/19/32/739765e24fbad29d13f83e546ea4abc215a78cea9d677ca09025b027724d/kiwisolver-1.5.1-cp312-cp312-manylinux_2_24_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:6a797a1cefc8b9c93170db580337e1fe3d011ad18b1299943231279406342048", size = 1296677, upload-time = "2026-08-28T10:25:35.059Z" }, + { url = "https://files.pythonhosted.org/packages/df/32/03304d1010e2cc45e5b3b52cef7e43fed3a2a5cd6c87a89b4a88e1d85b5d/kiwisolver-1.5.1-cp312-cp312-manylinux_2_24_s390x.manylinux_2_28_s390x.whl", hash = "sha256:2551cf9917af48ee7c4b29cc82320489508cf96fd26a51f6fc124de661cd44c7", size = 1346037, upload-time = "2026-08-28T10:25:36.705Z" }, + { url = "https://files.pythonhosted.org/packages/3e/57/4c49377bfd274450dd72ecaa13eaac32ea804a03363e4d1db0c5aa999ceb/kiwisolver-1.5.1-cp312-cp312-manylinux_2_39_riscv64.whl", hash = "sha256:38f6e0deb4d0a4615efe0c4efc5990b06ae450ab50a0b321c0b078b6d238c083", size = 988248, upload-time = "2026-08-28T10:25:38.299Z" }, + { url = "https://files.pythonhosted.org/packages/cb/c3/38df144a08b6c5d75ca4504e5cc3141bb3bfef64c04f4ef48204f42711b6/kiwisolver-1.5.1-cp312-cp312-musllinux_1_2_aarch64.whl", hash = "sha256:bfd1de989b3330420e29de39352f5c049905c9e3ee67233a50d550e3d652c148", size = 2228722, upload-time = "2026-08-28T10:25:40.038Z" }, + { url = "https://files.pythonhosted.org/packages/e7/11/3221838a89cd64d9b386353e000cd8a296069a20fbe3584507fdfd5bebae/kiwisolver-1.5.1-cp312-cp312-musllinux_1_2_ppc64le.whl", hash = "sha256:1209042a623ddfda5497e4066c7b77651dde8e1d3a9dd97599dc7e97f3b9b78c", size = 2325216, upload-time = "2026-08-28T10:25:41.699Z" }, + { url = "https://files.pythonhosted.org/packages/83/d4/075c219230697bb5db910d37262b9bacf880f92b4811a02ab81ed073a253/kiwisolver-1.5.1-cp312-cp312-musllinux_1_2_riscv64.whl", hash = "sha256:26e8268480be5061d509e29669d59103c067a26377a56491630ece11762e3858", size = 1977689, upload-time = "2026-08-28T10:25:43.559Z" }, + { url = "https://files.pythonhosted.org/packages/bb/08/1d219c3c2dd960983d0d4da623d916e9de6385df2b0bab3d1af0e9b8fccc/kiwisolver-1.5.1-cp312-cp312-musllinux_1_2_s390x.whl", hash = "sha256:d79308fa689fac89cbcfbd4dbfc80b5f95c54c5a7fd4d194be221f9d33d026e6", size = 2491443, upload-time = "2026-08-28T10:25:45.242Z" }, + { url = "https://files.pythonhosted.org/packages/ba/d3/024208ec1079d273f1047468d1bdffbf38bb75b7b268090fd3a0301b9d9a/kiwisolver-1.5.1-cp312-cp312-musllinux_1_2_x86_64.whl", hash = "sha256:b03af77d77e50edba2030fd5f7c352ff209314b09030a3cba7c14edf9a09a444", size = 2295200, upload-time = "2026-08-28T10:25:46.984Z" }, + { url = "https://files.pythonhosted.org/packages/6e/7c/7b210498f9f92e1cd7855f260fa69ef056881087b199ee20c208f0e4189a/kiwisolver-1.5.1-cp312-cp312-win_amd64.whl", hash = "sha256:06a6917674de9e0fe3f66f5430787f59a9f2ddb64af9b714eaec547e29ef5c19", size = 70748, upload-time = "2026-08-28T10:25:48.444Z" }, + { url = "https://files.pythonhosted.org/packages/94/61/ef0daa157c8bb23672f7423e0d14c39db1dc6ef8ed47e6bc54c9c1bef3bf/kiwisolver-1.5.1-cp312-cp312-win_arm64.whl", hash = "sha256:ad8b9671348d7c8716715652ae11f85ed0eb99e265a2df2ca490577d69860b2c", size = 68324, upload-time = "2026-08-28T10:25:49.81Z" }, + { url = "https://files.pythonhosted.org/packages/08/c1/88018321d976f53c421e379c43bc6993e70ce0c8a3ec5edc4bfe102257f6/kiwisolver-1.5.1-cp313-cp313-ios_13_0_arm64_iphoneos.whl", hash = "sha256:b6ae6a0328f0bc035741820fdeecdcd67bf4694eee03972e843663107122f450", size = 62272, upload-time = "2026-08-28T10:25:51.02Z" }, + { url = "https://files.pythonhosted.org/packages/85/d2/712bc17ea4f1d216034928069d612defbc6c95a471c55a7203a39faecb1a/kiwisolver-1.5.1-cp313-cp313-ios_13_0_arm64_iphonesimulator.whl", hash = "sha256:886fc26012f0e8b5f69d1cfe6d711f6b11f194621539bf8e6bb1c25c5dc82724", size = 64481, upload-time = "2026-08-28T10:25:52.22Z" }, + { url = "https://files.pythonhosted.org/packages/1b/e6/6c5380d676f43b6d918033962ea5e72360ca69e5a404154bc496b598ffdb/kiwisolver-1.5.1-cp313-cp313-ios_13_0_x86_64_iphonesimulator.whl", hash = "sha256:aefe930d113798330e9462f7874542977869c0613cba3262e2de3a8d5dee8f3a", size = 66260, upload-time = "2026-08-28T10:25:53.387Z" }, + { url = "https://files.pythonhosted.org/packages/92/6a/7087f5822cc8bb272679641404b1966a42504dac9ee74e2b33840475a0aa/kiwisolver-1.5.1-cp313-cp313-macosx_10_13_universal2.whl", hash = "sha256:a5ca5aebae78a0bc13c1943af4af615d4966c5b650b05d5aa83b50e427196fee", size = 123876, upload-time = "2026-08-28T10:25:54.644Z" }, + { url = "https://files.pythonhosted.org/packages/5c/4b/9f385087ca09ee5ab9c09c6832561a7d2f7c78d3e5661d511e669f70e439/kiwisolver-1.5.1-cp313-cp313-macosx_10_13_x86_64.whl", hash = "sha256:1ed0f5e49d0ceff8b72190824d9e59c062fbbc02c231b853112c78474b3f5ec2", size = 66487, upload-time = "2026-08-28T10:25:55.899Z" }, + { url = "https://files.pythonhosted.org/packages/47/8b/40d33ffd2f378094ed462e9a9a0907e59d4de9845e65a59561272da350d4/kiwisolver-1.5.1-cp313-cp313-macosx_11_0_arm64.whl", hash = "sha256:77a4c8187a5948d7f8795adb765a3c7b553d07d86d88e43038fc32fc1fb9a3f3", size = 64673, upload-time = "2026-08-28T10:25:57.048Z" }, + { url = "https://files.pythonhosted.org/packages/ab/43/86aacc027959108b4c66eeae8b73cedb057dfa6eb3a335d05ad65197081c/kiwisolver-1.5.1-cp313-cp313-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:74ad5c3dad54a4641b4c28cd15ded70899d04459c6c7aeacafea716be97cce6d", size = 1477992, upload-time = "2026-08-28T10:25:58.479Z" }, + { url = "https://files.pythonhosted.org/packages/ed/40/b1d0369048c79733a32c8abb0f2718532e6630641368e33a81384246e844/kiwisolver-1.5.1-cp313-cp313-manylinux_2_24_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:21e46b23a2da695c364124817bc01d970effd5483147f8d66a6a7167e3f6b851", size = 1278821, upload-time = "2026-08-28T10:26:00.121Z" }, + { url = "https://files.pythonhosted.org/packages/61/a1/fa71c1792272ff9461432715ae60ffb7e11a4d7ac3bf68961b9cab6c60cf/kiwisolver-1.5.1-cp313-cp313-manylinux_2_24_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:75d9b1cf8258462dbdc1eeda718c96ea7f079324c09067f6daabfcf37712b7fe", size = 1296805, upload-time = "2026-08-28T10:26:01.868Z" }, + { url = "https://files.pythonhosted.org/packages/a3/49/3f0bd94af8e06ecc47eb834b195a06f05d48711ceb2352c56d6835160f0e/kiwisolver-1.5.1-cp313-cp313-manylinux_2_24_s390x.manylinux_2_28_s390x.whl", hash = "sha256:8fca690b00c4c48f6c2a547b0160ed511357093a4e4c9b47e0fadf3128066d89", size = 1346109, upload-time = "2026-08-28T10:26:03.59Z" }, + { url = "https://files.pythonhosted.org/packages/a4/44/3afe6ef9cf06d61220953a8963e94eca978491be1d9547cb01d82a1efa08/kiwisolver-1.5.1-cp313-cp313-manylinux_2_39_riscv64.whl", hash = "sha256:876bbfd276473d3daffe30e8c975df4ed9429967b41a6cb362dbb5155b6f13ad", size = 988252, upload-time = "2026-08-28T10:26:05.306Z" }, + { url = "https://files.pythonhosted.org/packages/e4/30/a12bd7a7285a211e1747c3eec77b8c614dbfcc1dad942f7611a1a6921ae5/kiwisolver-1.5.1-cp313-cp313-musllinux_1_2_aarch64.whl", hash = "sha256:f942903fde7363d1d879057ec5de01310efda2597161784d752fa9953a01a71a", size = 2228846, upload-time = "2026-08-28T10:26:07.312Z" }, + { url = "https://files.pythonhosted.org/packages/a1/6b/233e2958abf0dab7b18d07e52f286e02e519d7651bfbbe97af9347564109/kiwisolver-1.5.1-cp313-cp313-musllinux_1_2_ppc64le.whl", hash = "sha256:c90d3022d8a94778939cda8638c6c8da8fa757b8958dad7ec868ce29c87681b8", size = 2325583, upload-time = "2026-08-28T10:26:09.093Z" }, + { url = "https://files.pythonhosted.org/packages/42/8e/7673060a27b01405b580058510adef34687069d229800239f5e44682d4d0/kiwisolver-1.5.1-cp313-cp313-musllinux_1_2_riscv64.whl", hash = "sha256:8a34616dc2521cc8dc1d7d081734da63539f021ac0450ce950908340c6e7aa2f", size = 1978221, upload-time = "2026-08-28T10:26:11.127Z" }, + { url = "https://files.pythonhosted.org/packages/3f/7b/1b882fc1a8b4a0bb8084e7d1d85004116d08c92c0705d31f2928dec607f2/kiwisolver-1.5.1-cp313-cp313-musllinux_1_2_s390x.whl", hash = "sha256:8bf4df63592c2a66b4f8edc5df2544998c288aa02f96ce0acd880cd1de8c8127", size = 2491819, upload-time = "2026-08-28T10:26:13.348Z" }, + { url = "https://files.pythonhosted.org/packages/39/9c/426deb49e62c5f69464b64bbeca064d3b758a7506b8913d986ef34f4619c/kiwisolver-1.5.1-cp313-cp313-musllinux_1_2_x86_64.whl", hash = "sha256:d09037ca068d784ebc4aec290ef952ca27ac15dd9c0b5801a88c6e1096b83e6b", size = 2295520, upload-time = "2026-08-28T10:26:15.042Z" }, + { url = "https://files.pythonhosted.org/packages/f5/22/deabbb3ad6d918d74b7831b2d8ae7151b09d21c87974e5ee8a456f58c94c/kiwisolver-1.5.1-cp313-cp313-win_amd64.whl", hash = "sha256:dc23390afe9f4ef9ac3bcc72a03a56eebbde03f4c571a32cb38f859cff9a6524", size = 70758, upload-time = "2026-08-28T10:26:16.504Z" }, + { url = "https://files.pythonhosted.org/packages/b9/71/02fa5c2fd92068bb8952847e70ee6c5cb280e7febe11653d17812acc53dd/kiwisolver-1.5.1-cp313-cp313-win_arm64.whl", hash = "sha256:186884a58486651e3c217b6acea0a53eaa9498fdd472057c46f2f0fb5c25aad5", size = 68329, upload-time = "2026-08-28T10:26:17.658Z" }, + { url = "https://files.pythonhosted.org/packages/dd/87/2d5dfad0daf17dcc18d98c48ed2332fc3f051cf599e60be6182a30dd4cf1/kiwisolver-1.5.1-cp314-cp314-ios_13_0_arm64_iphoneos.whl", hash = "sha256:0324cd2567259b7a095f6cf18a52b0ffc6f3de9e69528ff1bc0e7a37bd43ff1a", size = 62337, upload-time = "2026-08-28T10:26:18.778Z" }, + { url = "https://files.pythonhosted.org/packages/08/c8/83e1624f15d6262b470dbcc80b09979fd4d5b2ea3ddfc6b6e3327e235726/kiwisolver-1.5.1-cp314-cp314-ios_13_0_arm64_iphonesimulator.whl", hash = "sha256:74ea337e0ec3f6f342a36a4f1b5cd94dd9affddcd28ba9aae2905af932ee8c6b", size = 64513, upload-time = "2026-08-28T10:26:19.909Z" }, + { url = "https://files.pythonhosted.org/packages/c9/2d/827ec30eb07f528c08d8459ffb318ae91a56d793ee8acbea8b491f0ff906/kiwisolver-1.5.1-cp314-cp314-ios_13_0_x86_64_iphonesimulator.whl", hash = "sha256:ee9df1f0d77b9c6e94f4ac0fec533fbddd5ea3a327807f18d7b069ae019ded80", size = 66287, upload-time = "2026-08-28T10:26:21.078Z" }, + { url = "https://files.pythonhosted.org/packages/53/11/5c43a562529dad8def4b81e5e1877c612a7e0298105a5939b3b409d2079c/kiwisolver-1.5.1-cp314-cp314-macosx_10_15_universal2.whl", hash = "sha256:fc271a6f0a2126958f4090e5507b9da5848927dae331f8f763bd4aa642b3d2cd", size = 123940, upload-time = "2026-08-28T10:26:22.475Z" }, + { url = "https://files.pythonhosted.org/packages/64/db/9bd6c505c95128c258a55236bfbb3a7a3fb6023f863316b6d7d9f3c69052/kiwisolver-1.5.1-cp314-cp314-macosx_10_15_x86_64.whl", hash = "sha256:9b3092d8992a1d69b7a59c3e39f35e1b9be327a17f68a7c35fc17329e337d6f2", size = 66493, upload-time = "2026-08-28T10:26:23.743Z" }, + { url = "https://files.pythonhosted.org/packages/e1/f4/b3007a3ed5c9be73f81161140684cf7d9bdb9c4b632f5f484d2a1c713fb9/kiwisolver-1.5.1-cp314-cp314-macosx_11_0_arm64.whl", hash = "sha256:c2306e8bb53601979fcb3fa09cc65e031876d9ae01eff2fcbcd7a84ef94d5bc1", size = 64720, upload-time = "2026-08-28T10:26:24.95Z" }, + { url = "https://files.pythonhosted.org/packages/a1/13/08188f0cafa3a800403e4ff62b9aad4e7a17f9c4c7e080dc8f18c64794cf/kiwisolver-1.5.1-cp314-cp314-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:18a0cfb124546a4c2e6087c5f3029c7f44b37c85b142e0ced71f73a7599ac208", size = 1475867, upload-time = "2026-08-28T10:26:26.393Z" }, + { url = "https://files.pythonhosted.org/packages/8a/3e/053bdc3c9abdb8f2606225eda398adca25c0c91ab90add8222a69db65ee0/kiwisolver-1.5.1-cp314-cp314-manylinux_2_24_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:34ec467940442c9943016fb2d4c81d1ba84351eeca2f1a78f8bc87f1ba0d414c", size = 1282865, upload-time = "2026-08-28T10:26:28.118Z" }, + { url = "https://files.pythonhosted.org/packages/5e/64/a44c341b36b610588cc2f1e89b3cae072a3119aa8be578e90987cd640751/kiwisolver-1.5.1-cp314-cp314-manylinux_2_24_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:a83ee7107df13abe42a54a6654670eef9bb39425cf2e27f65e0007465e1286ab", size = 1300865, upload-time = "2026-08-28T10:26:30.125Z" }, + { url = "https://files.pythonhosted.org/packages/60/5e/7e7d716dca38c714478b741257a5b4a321d9932b8d851551a136dcaf3984/kiwisolver-1.5.1-cp314-cp314-manylinux_2_24_s390x.manylinux_2_28_s390x.whl", hash = "sha256:bebb89489b279b2f5661bbbb2abcc87bcd4a46607bb4a5c966f04f1db6b8df9a", size = 1348071, upload-time = "2026-08-28T10:26:31.829Z" }, + { url = "https://files.pythonhosted.org/packages/10/1a/2b98fdda8bf45b7be317e48ed12393d44334394d315c74b81f4a14c0e31b/kiwisolver-1.5.1-cp314-cp314-manylinux_2_39_riscv64.whl", hash = "sha256:509735237ae0d849e8a843551d423d2500d2e0a9ac1611a145658b29c0fb9f85", size = 992191, upload-time = "2026-08-28T10:26:33.544Z" }, + { url = "https://files.pythonhosted.org/packages/b6/55/d893f5ede0e50f9e3fcf01f6015f42ec7d9cd221e26772701fe4a98745f9/kiwisolver-1.5.1-cp314-cp314-musllinux_1_2_aarch64.whl", hash = "sha256:824c3d763a05ea9e9003610145186b0e9848c7584a5575c79bac5a8e7cd80bad", size = 2233854, upload-time = "2026-08-28T10:26:35.282Z" }, + { url = "https://files.pythonhosted.org/packages/b1/82/f85f6279555a6ee1639fef7bfe83adb037a03e11a6fc9eaa54b8d0380339/kiwisolver-1.5.1-cp314-cp314-musllinux_1_2_ppc64le.whl", hash = "sha256:1fff05e239575b1481b6ed1a782f6fad616efbf1f0b1f44e6e85c4dfe426e483", size = 2330621, upload-time = "2026-08-28T10:26:36.9Z" }, + { url = "https://files.pythonhosted.org/packages/56/31/e11aea078f66fc2fffcc179d38ca90d9da97652a241b64519169742ba46a/kiwisolver-1.5.1-cp314-cp314-musllinux_1_2_riscv64.whl", hash = "sha256:0627b9bceb9c3cdcf12b8a18655eedfed2692b038df27423383c120d0b7dc2d6", size = 1982848, upload-time = "2026-08-28T10:26:39.01Z" }, + { url = "https://files.pythonhosted.org/packages/af/ea/2956b63bf5140ca46aa2c2818e6aa03e2d5754dd2fa41db1c6b28922940c/kiwisolver-1.5.1-cp314-cp314-musllinux_1_2_s390x.whl", hash = "sha256:8a708a47ade1fe19e8371d5da076bac0dd4b0a5a7985ad6c637f7f7e361b6baa", size = 2494850, upload-time = "2026-08-28T10:26:40.837Z" }, + { url = "https://files.pythonhosted.org/packages/11/d1/3829542258d8b3fc0898d221e7ef0e2c83eca0d348709bb8dbe54f3d4005/kiwisolver-1.5.1-cp314-cp314-musllinux_1_2_x86_64.whl", hash = "sha256:007a5553dfc4f4e8d184f588a0200e2cd4b63a59cc8796df3c39909e679dc7a0", size = 2298067, upload-time = "2026-08-28T10:26:42.803Z" }, + { url = "https://files.pythonhosted.org/packages/4e/0e/49522e1ab5788cbaf63a26fbd3b851f9028616828c961b8a31b35cb96df8/kiwisolver-1.5.1-cp314-cp314-win_amd64.whl", hash = "sha256:f4167e87b397f273dc2356fcf1eaf50a6bac51e6105f45103ef7129c8efb0255", size = 72282, upload-time = "2026-08-28T10:26:44.268Z" }, + { url = "https://files.pythonhosted.org/packages/f5/b6/22e7ca5315d363e6f81c9f37c9472e12e7b298731e77c0428e6a911a2c39/kiwisolver-1.5.1-cp314-cp314-win_arm64.whl", hash = "sha256:5c490db2168a508088f59140dd392556a54b8bd1048fc6383c8baff13c359673", size = 69855, upload-time = "2026-08-28T10:26:45.725Z" }, + { url = "https://files.pythonhosted.org/packages/30/8c/03a9cfbe871964c8758a816eb03ac96c806da2795a9a7cd9bf9648bfb594/kiwisolver-1.5.1-cp314-cp314t-macosx_10_15_universal2.whl", hash = "sha256:4d4ca09bf13cff792b1884f64b98ee6c2467930d632233be25c56b442d99f10e", size = 126289, upload-time = "2026-08-28T10:26:47.023Z" }, + { url = "https://files.pythonhosted.org/packages/f2/e3/14ce3041ca79dff9c9d884ca00c7bf32374e76028a865a9ecd99b4f5a517/kiwisolver-1.5.1-cp314-cp314t-macosx_10_15_x86_64.whl", hash = "sha256:44b8faef94f1857e77fa0238f3390ff1ac51d2ea20a487e2e452a59fd2b5f5ca", size = 67709, upload-time = "2026-08-28T10:26:48.268Z" }, + { url = "https://files.pythonhosted.org/packages/af/c4/45030471a66ec8ef042e9f96ffe1d522c9ab12da180186a0898966fc1385/kiwisolver-1.5.1-cp314-cp314t-macosx_11_0_arm64.whl", hash = "sha256:2ae70bc59790d2af72a3f76f24b272403e135070340281108b447cb77ea70819", size = 65909, upload-time = "2026-08-28T10:26:49.523Z" }, + { url = "https://files.pythonhosted.org/packages/42/73/17dce073a6ae259bb32cf9d686c4079d2e538868bc45967462bf33df914a/kiwisolver-1.5.1-cp314-cp314t-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:43844c1a7ad6d723d5b5b4c4fc7f5bd399c40e288120d16257c7c9e8765c6e85", size = 1584907, upload-time = "2026-08-28T10:26:50.933Z" }, + { url = "https://files.pythonhosted.org/packages/8c/84/ae3c75909f507283cbfcc7e916c7e822579ef962020b97e6882b27b4478f/kiwisolver-1.5.1-cp314-cp314t-manylinux_2_24_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:22d5e5aaad6be121f2515765e3b1c444352cb8eb4c86510801db8f2e50757316", size = 1392474, upload-time = "2026-08-28T10:26:52.638Z" }, + { url = "https://files.pythonhosted.org/packages/34/31/8bcc83caad5bce8fa4577152389848bf6bc110e51e573a2b4e7c2aa34c89/kiwisolver-1.5.1-cp314-cp314t-manylinux_2_24_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:3fa5855898f6d3d01b72ccd48a2d65cbdee301251603fefe34e2025bddba219c", size = 1405246, upload-time = "2026-08-28T10:26:54.248Z" }, + { url = "https://files.pythonhosted.org/packages/55/72/220345537d790cf4ae54f8acfff4b5cc2468e0702a384d651cf7a771c63e/kiwisolver-1.5.1-cp314-cp314t-manylinux_2_24_s390x.manylinux_2_28_s390x.whl", hash = "sha256:d66a64dd5dec136040ec2ae94aa026a912ee60fdd45bc28d3db30037fd809e88", size = 1456099, upload-time = "2026-08-28T10:26:56.042Z" }, + { url = "https://files.pythonhosted.org/packages/cd/10/3725fd2398f66d18c34b4e0f81a8d03764cd4f4f089f58a527f0b4428086/kiwisolver-1.5.1-cp314-cp314t-manylinux_2_39_riscv64.whl", hash = "sha256:9e51c119992ea8820706871c30a4642ec76de20ae82f9b50b9a45517d8e9f810", size = 1073695, upload-time = "2026-08-28T10:26:57.658Z" }, + { url = "https://files.pythonhosted.org/packages/86/cb/28d6e09e66b93e4588b2e6b7d84d020ccefea09e2f4de788510a07efeab7/kiwisolver-1.5.1-cp314-cp314t-musllinux_1_2_aarch64.whl", hash = "sha256:70ed9a45c7484d2b30cdacf60d220f494a1763b9fec1ad03285c6553fa0889f2", size = 2335355, upload-time = "2026-08-28T10:26:59.202Z" }, + { url = "https://files.pythonhosted.org/packages/6a/b3/a0f31d5e4e40af7dc97c36b8a74fdd3a36cf3c8bbd098da9a23466ff6a94/kiwisolver-1.5.1-cp314-cp314t-musllinux_1_2_ppc64le.whl", hash = "sha256:98b208a7cc42c803445ef551d6753cc42a5ea13e9cab1ee66cd8b9cb70195330", size = 2426524, upload-time = "2026-08-28T10:27:01.181Z" }, + { url = "https://files.pythonhosted.org/packages/2f/c9/728f63bd58c72cafdc79fc306abeeac7391bec03b757a48dadeb30906521/kiwisolver-1.5.1-cp314-cp314t-musllinux_1_2_riscv64.whl", hash = "sha256:c6834b92dd2428e2dd85ef3d85f723d3c12f20aaf43a2ddd4f944ca25d833408", size = 2063430, upload-time = "2026-08-28T10:27:03.06Z" }, + { url = "https://files.pythonhosted.org/packages/84/df/ce188b96f92f9a2c958231da140768918cba53c9713dc887b82f85462118/kiwisolver-1.5.1-cp314-cp314t-musllinux_1_2_s390x.whl", hash = "sha256:5d142e352eb13facc7dd047489aebdff6ba78576c239f1ea04931979caaf0567", size = 2597513, upload-time = "2026-08-28T10:27:05.072Z" }, + { url = "https://files.pythonhosted.org/packages/69/d6/76947c8203768968382e5bd74d9cc95654746703a61ea53015f2c74a2e06/kiwisolver-1.5.1-cp314-cp314t-musllinux_1_2_x86_64.whl", hash = "sha256:f9b1c4900736e489a812c529100de4b8fb617d4db075e931e213c57424b83d9b", size = 2394488, upload-time = "2026-08-28T10:27:07.423Z" }, + { url = "https://files.pythonhosted.org/packages/e4/d4/14b21e4eb203c4d15425e8b6a2c625a320b4a1f2f7557eead63ffc30ffb7/kiwisolver-1.5.1-cp314-cp314t-win_amd64.whl", hash = "sha256:5978c3340f16a35c30f8ab2fa7bcf559973c55f1a5ef6970e1f621acf3c4db13", size = 75404, upload-time = "2026-08-28T10:27:08.892Z" }, + { url = "https://files.pythonhosted.org/packages/cb/f5/53157899fc7f45f76421b77b99eb1639dd0f83f26ff9d76300c96bb4a3b0/kiwisolver-1.5.1-cp314-cp314t-win_arm64.whl", hash = "sha256:ca307d6c259e5c98d3cb9ade55342b47a6839762caf2536f3d7b46ee660cc82e", size = 72946, upload-time = "2026-08-28T10:27:10.944Z" }, + { url = "https://files.pythonhosted.org/packages/75/62/f786c3a27f181fa339d851a77e266d208e776b9883cabc40a5b041a31b5a/kiwisolver-1.5.1-cp315-cp315-ios_13_0_arm64_iphoneos.whl", hash = "sha256:bb7c99f0673c03017a3ee01e54a5c2617a05468b11eabe513b0080e063ed95b1", size = 62420, upload-time = "2026-08-28T10:27:12.308Z" }, + { url = "https://files.pythonhosted.org/packages/02/cd/58a91ed25fbad0facdf503297b03768efab04bdf3141e5e3b49a34be7443/kiwisolver-1.5.1-cp315-cp315-ios_13_0_arm64_iphonesimulator.whl", hash = "sha256:0d8924877ce22e17326a99a418c3c82037da078df3c6a260b13eca677444e6e7", size = 64577, upload-time = "2026-08-28T10:27:13.502Z" }, + { url = "https://files.pythonhosted.org/packages/f6/5c/d501ef5a0958b226eac28306d24d5e5f114be0ace50e19cabae7b6b3b197/kiwisolver-1.5.1-cp315-cp315-ios_13_0_x86_64_iphonesimulator.whl", hash = "sha256:534f02c1abb31ed6dbd3515545285c330b2f12d00fdb1fdb71658b9ca5a13a6a", size = 66284, upload-time = "2026-08-28T10:27:14.813Z" }, + { url = "https://files.pythonhosted.org/packages/a3/a6/8fbecaf4fc18c02f31f05e47a84c010a80e3ec391ed2f0bdade1d62b5954/kiwisolver-1.5.1-cp315-cp315-macosx_10_15_universal2.whl", hash = "sha256:cea20da04494e662b83c872683bf4ff2345206043d036315ed0e924b652e7294", size = 124031, upload-time = "2026-08-28T10:27:16.193Z" }, + { url = "https://files.pythonhosted.org/packages/7c/c6/bf3090d2983b4204347cbdbe952116e7c3b2abf62b4e33e50167a13e75ee/kiwisolver-1.5.1-cp315-cp315-macosx_10_15_x86_64.whl", hash = "sha256:7fd82debf43c6acd0a94359d232f6bb516ee13f269a7993736a9ac9f988bb5d9", size = 66489, upload-time = "2026-08-28T10:27:17.506Z" }, + { url = "https://files.pythonhosted.org/packages/85/de/562dddef55fdd7c291da8626d6619e72b5fc0870e6ccca0e149a5731e7f3/kiwisolver-1.5.1-cp315-cp315-macosx_11_0_arm64.whl", hash = "sha256:18170a77ddfecf40ec60d0928268dc95880c881864e015a8f34094ed18b9b9ad", size = 64806, upload-time = "2026-08-28T10:27:18.673Z" }, + { url = "https://files.pythonhosted.org/packages/89/b1/ba7b9c0164ce1cf62bf2872db63b8483289cf0f3110d6f9390eb09e409ed/kiwisolver-1.5.1-cp315-cp315-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:ca7f6fe0f37ca978a1e5eb7a3a68e6413f417e78e838324947ffd420202b198b", size = 1482211, upload-time = "2026-08-28T10:27:20.039Z" }, + { url = "https://files.pythonhosted.org/packages/13/dc/34da54efb4976616d45c20aae32d70e89d6e7395ed908029154d1609ef22/kiwisolver-1.5.1-cp315-cp315-manylinux_2_24_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:5b973887ff782cfd6b67c9904ad8ca542e0bc5e4961503408b423b5a688b4d38", size = 1283739, upload-time = "2026-08-28T10:27:21.82Z" }, + { url = "https://files.pythonhosted.org/packages/5a/3a/30ffb62bee646e266e98a1b5cd276d9c75b6116fbfcb87c1190838c1b6df/kiwisolver-1.5.1-cp315-cp315-manylinux_2_24_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:f76fc85bd054c806960f917ec0f329e24e436f1712267d90588e4c39890caa63", size = 1301681, upload-time = "2026-08-28T10:27:23.876Z" }, + { url = "https://files.pythonhosted.org/packages/a9/8d/13c70be22a8506880b35fdc38dca36629613bc493405c79f4037f2cd2bb9/kiwisolver-1.5.1-cp315-cp315-manylinux_2_24_s390x.manylinux_2_28_s390x.whl", hash = "sha256:828f75af2b0080c8a972e75f649ab46af008e92c6104a57a759157200b835b75", size = 1349159, upload-time = "2026-08-28T10:27:25.899Z" }, + { url = "https://files.pythonhosted.org/packages/82/0e/993972b8ec6767f47cd69818fb3a5ff14510557d29f7d1a839be7574fa1b/kiwisolver-1.5.1-cp315-cp315-manylinux_2_39_riscv64.whl", hash = "sha256:431dc224a1a92a5c8f582d96e505196a3b5997a7271076678da2dfde67b77e9a", size = 997613, upload-time = "2026-08-28T10:27:27.507Z" }, + { url = "https://files.pythonhosted.org/packages/36/82/ca26eddd2eda2420dfc56693449c1f821f78b485da9cbde9904c03af3f93/kiwisolver-1.5.1-cp315-cp315-musllinux_1_2_aarch64.whl", hash = "sha256:61e9a64c7635095a6bfe483e2ff055d437c59bd45f3617a228b37277f0185d62", size = 2235109, upload-time = "2026-08-28T10:27:30.113Z" }, + { url = "https://files.pythonhosted.org/packages/c7/84/97d920881e10840b8d7c7185620298e3e4c88820b05514e3a15a258b08a6/kiwisolver-1.5.1-cp315-cp315-musllinux_1_2_ppc64le.whl", hash = "sha256:3c24cd69455e1b00ddf770c13b6e2c33e07d6dc3f2d34add0bf9277c5c6bbd46", size = 2331207, upload-time = "2026-08-28T10:27:32.795Z" }, + { url = "https://files.pythonhosted.org/packages/e2/ce/34d74b8f25acc58800f4c09268371e8d6159cf0f1206f1e4dc7835629b48/kiwisolver-1.5.1-cp315-cp315-musllinux_1_2_riscv64.whl", hash = "sha256:27add358abe374ebaa3b8763ef380bc99051b5a4b18d94878366a9e4f59efef0", size = 1986696, upload-time = "2026-08-28T10:27:34.628Z" }, + { url = "https://files.pythonhosted.org/packages/0e/01/f892644014612527aef7031d3306a2ffc60b3cb044f802c1561f8e5e14f3/kiwisolver-1.5.1-cp315-cp315-musllinux_1_2_s390x.whl", hash = "sha256:255605693a483db7bd5c79f60437f7bf658f7f520d61aa42722e32257c941951", size = 2496400, upload-time = "2026-08-28T10:27:36.818Z" }, + { url = "https://files.pythonhosted.org/packages/32/6d/d8284e66e697026536e5f418b9cfe56567bffd3c775e3ecfbae373605854/kiwisolver-1.5.1-cp315-cp315-musllinux_1_2_x86_64.whl", hash = "sha256:7d38b0c279c3032e8c9cc013b405c6df8e1668dbf15465779aa7f15f61201812", size = 2302967, upload-time = "2026-08-28T10:27:38.803Z" }, + { url = "https://files.pythonhosted.org/packages/46/0a/69a355e27f32ba50d5b6369949b6a1702e122f5277c89bc76d452b81c1c4/kiwisolver-1.5.1-cp315-cp315-win_amd64.whl", hash = "sha256:958254518717542d02d0688d0d20cbf771da5e415e6f49543f92481c850a4540", size = 72283, upload-time = "2026-08-28T10:27:40.491Z" }, + { url = "https://files.pythonhosted.org/packages/ef/d8/7a95be90c33dcdd52204d4aa6384d731443225b887283bbd8b61e7931f6c/kiwisolver-1.5.1-cp315-cp315-win_arm64.whl", hash = "sha256:da3275833be0edbaf4830fae08bae3dc7219f40ce0c37eaa6c25825957e06612", size = 69860, upload-time = "2026-08-28T10:27:41.835Z" }, + { url = "https://files.pythonhosted.org/packages/31/f8/9bc493e7f5707788ba7f621902c68f82dc3a7ba03c78fbd337b026cef1ed/kiwisolver-1.5.1-cp315-cp315t-macosx_10_15_universal2.whl", hash = "sha256:470d420f98d368d6f010633a20659b544c5fdfa5329e6b70219f2ef08fd4a7ef", size = 126336, upload-time = "2026-08-28T10:27:43.379Z" }, + { url = "https://files.pythonhosted.org/packages/d1/82/3aea86b3f99712db825e9ac5631bf99571e818a8b8961ff98cebd798413e/kiwisolver-1.5.1-cp315-cp315t-macosx_10_15_x86_64.whl", hash = "sha256:83f78128fa28705fa85d01c59771c72fe81c11bd0e6155edbb9f818983a7d761", size = 67698, upload-time = "2026-08-28T10:27:44.613Z" }, + { url = "https://files.pythonhosted.org/packages/84/7d/8daafc5d2e7f9c47a4f78f8865d86d2a9cf399c2a85f86c44a993594410c/kiwisolver-1.5.1-cp315-cp315t-macosx_11_0_arm64.whl", hash = "sha256:9506e892bcc3b409831d363c6f53e5985e1c8d1f6f6b0256d00358684ff85378", size = 65945, upload-time = "2026-08-28T10:27:45.932Z" }, + { url = "https://files.pythonhosted.org/packages/3d/c9/51dc974d9130da70a8c47a96160123443d387ffe1b6b833d6f91d9429339/kiwisolver-1.5.1-cp315-cp315t-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:cea90547bfd93807e0013a004dc76552be44fad3bc1cc2b38610a9e889ed098f", size = 1588030, upload-time = "2026-08-28T10:27:47.678Z" }, + { url = "https://files.pythonhosted.org/packages/e6/86/f3e1a730e7a995149d8d3ff9e313b6d8a17b2cf1d98a8eff139dc30463fb/kiwisolver-1.5.1-cp315-cp315t-manylinux_2_24_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:e8e4d953faaded9ec7ede36824e9814082d22d4c7b1eafbfa079ecba8cd0d076", size = 1390760, upload-time = "2026-08-28T10:27:49.676Z" }, + { url = "https://files.pythonhosted.org/packages/4f/3b/8ba25a2b5a0d2375e046f1b72de5179513f0be95aba6e7b094c89303929f/kiwisolver-1.5.1-cp315-cp315t-manylinux_2_24_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:7e9c01d3dd7ceba4d1d436cc021d40d592466e40b9bc7f5d83dc4e98a5c9cd8c", size = 1403279, upload-time = "2026-08-28T10:27:51.242Z" }, + { url = "https://files.pythonhosted.org/packages/18/d0/278d5cb8be812740027d5ca0a7eda0c375488a88d6dce0fa60fcc2591ad2/kiwisolver-1.5.1-cp315-cp315t-manylinux_2_24_s390x.manylinux_2_28_s390x.whl", hash = "sha256:37f801b5d7cc0e5a548921308e059fd2b057bb42972b591cfa3049f95423c4ed", size = 1454429, upload-time = "2026-08-28T10:27:53.213Z" }, + { url = "https://files.pythonhosted.org/packages/94/37/bcbab41063ec284c1d200efe5087cf087798c2f8916960aa8a20dd303290/kiwisolver-1.5.1-cp315-cp315t-manylinux_2_39_riscv64.whl", hash = "sha256:e68e151428b5384f766cd25739bf77c7e4a3dc93b5ded7a12118d9fbfdf78ab6", size = 1073225, upload-time = "2026-08-28T10:27:55.089Z" }, + { url = "https://files.pythonhosted.org/packages/cb/c5/ab79dcdf5ae28909a51210ae0a1c579e97ff997b3466414f0d04c0994583/kiwisolver-1.5.1-cp315-cp315t-musllinux_1_2_aarch64.whl", hash = "sha256:8f8fddb8e323bd6eee4e54e69a39243beab22689070f4c66b472c4cc88bb89d8", size = 2334335, upload-time = "2026-08-28T10:27:56.78Z" }, + { url = "https://files.pythonhosted.org/packages/8d/8e/71d047468a189041d9c93f3b76844b924f9793b188c44bd149fa258912da/kiwisolver-1.5.1-cp315-cp315t-musllinux_1_2_ppc64le.whl", hash = "sha256:3cc210010fd2f438a3ed430b45f1b501fd13a8618bf984dc2c5ce5b69b78752e", size = 2424982, upload-time = "2026-08-28T10:27:58.639Z" }, + { url = "https://files.pythonhosted.org/packages/7d/37/1347461bbea6d0e1f0580b94ef603b18e72c2be5f667fa1653867361a00b/kiwisolver-1.5.1-cp315-cp315t-musllinux_1_2_riscv64.whl", hash = "sha256:b5664603a253efd3a75716d793d1d3a6a82723b61dc6db767b2460bbbeec4c0f", size = 2062857, upload-time = "2026-08-28T10:28:00.407Z" }, + { url = "https://files.pythonhosted.org/packages/1f/87/c7f0976c9cd0d127643351bc0c9929e0b8899d7f49d4ec238cd909e39c42/kiwisolver-1.5.1-cp315-cp315t-musllinux_1_2_s390x.whl", hash = "sha256:a7b85b2cc6ea45e5f7e8c9a30bc9fabd47cda09106cbb4b967335c3e6c43b69d", size = 2596022, upload-time = "2026-08-28T10:28:02.183Z" }, + { url = "https://files.pythonhosted.org/packages/e3/9a/59a6f6ae6f938c15076be2c21b6cedea973d71bb1349ec84fa485fab82cf/kiwisolver-1.5.1-cp315-cp315t-musllinux_1_2_x86_64.whl", hash = "sha256:ab620eb663952455271ac37f9aaad86b73c969c02f11f53cea405b38e96a4300", size = 2395634, upload-time = "2026-08-28T10:28:03.977Z" }, + { url = "https://files.pythonhosted.org/packages/e9/2d/8982c1fb7926da5bb7ed60318c3665b5c3f941447271ac982960a11b8637/kiwisolver-1.5.1-cp315-cp315t-win_amd64.whl", hash = "sha256:cb6fae641357ed2f6e533c0d3c6504a4a5703621a50c89459e46051d56b61140", size = 75379, upload-time = "2026-08-28T10:28:06.307Z" }, + { url = "https://files.pythonhosted.org/packages/07/78/ba7b6dfa1708b82b373ac056928a30c545d5c1a627df9839dcec3c6c1881/kiwisolver-1.5.1-cp315-cp315t-win_arm64.whl", hash = "sha256:b390aec180a7c054919c04898835e1c77bced23ea8383eb2c570213bf25d1a86", size = 73011, upload-time = "2026-08-28T10:28:07.578Z" }, + { url = "https://files.pythonhosted.org/packages/a9/c4/1407df7512a5b36cc79840e01710dc575733c461b13ab866cae77eaf87f3/kiwisolver-1.5.1-graalpy312-graalpy250_312_native-macosx_11_0_arm64.whl", hash = "sha256:482676e5bd48d70ac99d9fc78863469845421e01184fa83f1f9366dc49f7e974", size = 134002, upload-time = "2026-08-28T10:28:08.881Z" }, + { url = "https://files.pythonhosted.org/packages/16/45/c37a21ad5c0ab581a93c55ad544721aaa1f0ae94edb29c6a678a23d013e6/kiwisolver-1.5.1-graalpy312-graalpy250_312_native-manylinux_2_24_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:072bdb15a3c19a5b5dbc8f8fb1f4e1884bf4f3507eeb4cc6334401274d37a5c0", size = 194292, upload-time = "2026-08-28T10:28:11.06Z" }, + { url = "https://files.pythonhosted.org/packages/a6/c1/69f00d627949580e43d57af0aa465df46868d7c29801c137a55374101294/kiwisolver-1.5.1-graalpy312-graalpy250_312_native-win_amd64.whl", hash = "sha256:a5a00665d1a0e26763a7338d7e911d4598fbc1d50dd0d6b7919b7dc6c5d6569f", size = 73362, upload-time = "2026-08-28T10:28:12.449Z" }, + { url = "https://files.pythonhosted.org/packages/b5/1d/59ba570b1774e95e97fde3a0981b2e22118a7a495f73bf74cedc538566a0/kiwisolver-1.5.1-pp311-pypy311_pp73-macosx_10_15_x86_64.whl", hash = "sha256:416ba7ff9f233b7036689bb5a3783537e838ad483f63558d2a800f75afe738b1", size = 59450, upload-time = "2026-08-28T10:28:20.383Z" }, + { url = "https://files.pythonhosted.org/packages/22/98/a6849f04dc18b5400e8b98affa2cd8fd86ed583085f036e57b32e571f4fa/kiwisolver-1.5.1-pp311-pypy311_pp73-macosx_11_0_arm64.whl", hash = "sha256:8af9b142ad719ae3a911ebf616bc4b78b32bbab84d6a40d3ad2f129670509957", size = 57400, upload-time = "2026-08-28T10:28:21.632Z" }, + { url = "https://files.pythonhosted.org/packages/f4/ce/a7dc71353dd06a4cbe02222773f52d4a28c81e5a452a75797f8ed113dc99/kiwisolver-1.5.1-pp311-pypy311_pp73-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:5daa1f19e097050b9c4d9a78fcc9263cb96c9dfae08037ddc1b7c4ad1889f2a2", size = 79891, upload-time = "2026-08-28T10:28:22.936Z" }, + { url = "https://files.pythonhosted.org/packages/10/b1/d61c61a84ff85d1a36a99df2c152b59ffedb1d356c598902aba44abcdb60/kiwisolver-1.5.1-pp311-pypy311_pp73-manylinux_2_24_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:cdaeeb6c350106df6bf9d873395973e5f066a9713200b72cd64f55d0a3eafab6", size = 77605, upload-time = "2026-08-28T10:28:24.322Z" }, + { url = "https://files.pythonhosted.org/packages/d3/52/5aef56f21a460a6e43ab3cdfc7697d59d7b87deb0ec97a0f7b91aa4a521b/kiwisolver-1.5.1-pp311-pypy311_pp73-win_amd64.whl", hash = "sha256:17851e5dad4484be0cbccbde3b15331deae036de9aebd45eed964487802b172f", size = 98465, upload-time = "2026-08-28T10:28:25.696Z" }, +] + +[[package]] +name = "librt" +version = "0.16.0" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/04/f5/9dc696772d241814bacac7880bac32f2930b5a6ebc1f85317b83161a011c/librt-0.16.0.tar.gz", hash = "sha256:ac38d6d8d66bf3d744148dbbc0b8e193e195a51e364ed55e224631f5721891fc", size = 219839, upload-time = "2026-09-29T00:55:32.891Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/ba/0e/b04fd2d76b6a78a02286938251b2c119fb36ddece92c76cb4886ef7d44b8/librt-0.16.0-cp311-cp311-macosx_10_9_x86_64.whl", hash = "sha256:fe4372c52d4849096c6cc1cda2817d293ec51440c890474ed59ef38d46556f18", size = 148058, upload-time = "2026-09-29T00:44:40.903Z" }, + { url = "https://files.pythonhosted.org/packages/a3/fc/e1ed24aadb32d38603c0c07f579d739eb4d90e64e90d525b024e4a999b4d/librt-0.16.0-cp311-cp311-macosx_11_0_arm64.whl", hash = "sha256:c72c5295a84bd249526da9bdca38f2e176d15c31c13bb0063c5053f4ca023421", size = 153073, upload-time = "2026-09-29T00:44:42.165Z" }, + { url = "https://files.pythonhosted.org/packages/be/38/133ab655409420e0c34a9a9836a7f4e5ae75fba758d7fe177099130d58ba/librt-0.16.0-cp311-cp311-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:be56ba9c884143495b517f23fe794ae367d58cd89ea0fdd6d437e3c024a87f9f", size = 493052, upload-time = "2026-09-29T00:44:44.043Z" }, + { url = "https://files.pythonhosted.org/packages/94/42/aaa663bc9421f5bf7bbbe9d3316d7219749698ec7688ac3b519770a89c04/librt-0.16.0-cp311-cp311-manylinux2014_i686.manylinux_2_17_i686.manylinux_2_28_i686.whl", hash = "sha256:ef46c1a29ffb8c72e882e22618ec618778eacd0578fb22c6e7cf9c11d15f357b", size = 485506, upload-time = "2026-09-29T00:44:45.479Z" }, + { url = "https://files.pythonhosted.org/packages/7b/59/83a6eb5087ba8b45b083859e2085256963e06d4c2531944cdd962670bfb5/librt-0.16.0-cp311-cp311-manylinux2014_ppc64le.manylinux_2_17_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:d3c94211ee0c4f8d649ec06b7c115c0ec4eadb873a0e3154ca15cef3f814b071", size = 498708, upload-time = "2026-09-29T00:44:46.916Z" }, + { url = "https://files.pythonhosted.org/packages/a0/e4/15a8d62b2bc12935080ec7277e2e0b9b233047781eb784a51d3e25af669a/librt-0.16.0-cp311-cp311-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:94aed6a8308818b91677957d1bd03188869cd7aeb23c5dba7912a6c0402f7602", size = 515981, upload-time = "2026-09-29T00:44:48.316Z" }, + { url = "https://files.pythonhosted.org/packages/46/4f/287cc282ac5dd815204c0779e79401bb0a568c3a1335fb319bf0838fb566/librt-0.16.0-cp311-cp311-manylinux_2_34_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:349c0bcb87ebd07481b6ff781e25cdc699723dbe2212e57dabb27f7a13b7b87d", size = 508670, upload-time = "2026-09-29T00:44:49.726Z" }, + { url = "https://files.pythonhosted.org/packages/bf/5b/ab7292add898ade0d2b5b75d32a923aae315861671aa5c39f5491309f7ae/librt-0.16.0-cp311-cp311-musllinux_1_2_aarch64.whl", hash = "sha256:001bfd59a7d45b17e3e75f2a8c6405280b35e7b84471792778e718c4f368950e", size = 530393, upload-time = "2026-09-29T00:44:51.356Z" }, + { url = "https://files.pythonhosted.org/packages/7a/9e/8e6bcf027d4bcbc16a4e864b3d158a1d3e5f55ffa53af0238020cad89271/librt-0.16.0-cp311-cp311-musllinux_1_2_i686.whl", hash = "sha256:28e038895b998d7a0c7798922ce8a1dc157675df5cf1c9ef0aca809ed804b7a1", size = 534302, upload-time = "2026-09-29T00:44:52.893Z" }, + { url = "https://files.pythonhosted.org/packages/96/a4/8c5ab4f3b3a8e905ff8305d76976a2e666f52fe049edd398aed2402beb36/librt-0.16.0-cp311-cp311-musllinux_1_2_ppc64le.whl", hash = "sha256:0dbe4096a7ecc00fa835d24510ad8545a4efef738dac96e0e63516783ccde905", size = 536150, upload-time = "2026-09-29T00:44:54.421Z" }, + { url = "https://files.pythonhosted.org/packages/ea/5e/7c4bc267e3cd4b8bd288de9f62d61fff203829c9cc649ea2aa12e9218cf0/librt-0.16.0-cp311-cp311-musllinux_1_2_riscv64.whl", hash = "sha256:5cd5b092441053364af968ea12084692cb9d4a22f3ce9524e377880bf028761e", size = 514316, upload-time = "2026-09-29T00:44:55.998Z" }, + { url = "https://files.pythonhosted.org/packages/e0/75/c8bc6b50f43c73819334233f5a3259c81216742cef56d2140c3dd377c3f0/librt-0.16.0-cp311-cp311-musllinux_1_2_x86_64.whl", hash = "sha256:3ddeb3c9dedb461bb457c6c7d9aa7fbf35329da313d1a7543d00c8d0f3473c96", size = 557731, upload-time = "2026-09-29T00:44:57.581Z" }, + { url = "https://files.pythonhosted.org/packages/93/fb/e241f91bc8de6c7ac1c6a20ea6dd381bc52312b44d711ffcd97140db1e57/librt-0.16.0-cp311-cp311-win32.whl", hash = "sha256:e05108e0849966f53a8d2d3112a7af881d0efaa479bc735bba91108f9f2350a7", size = 104922, upload-time = "2026-09-29T00:44:59.078Z" }, + { url = "https://files.pythonhosted.org/packages/d1/38/c892383d2bb1a6a80b2ec8ede84970a265a792c56614a06282bf3008517c/librt-0.16.0-cp311-cp311-win_amd64.whl", hash = "sha256:5f49cff01bd608ef7d97104cb035c75455e79c2d70bf4a506cf773338ac1860d", size = 125093, upload-time = "2026-09-29T00:49:01.621Z" }, + { url = "https://files.pythonhosted.org/packages/e4/56/8bbde1ed0b21480163617648f638fdaec9374cf5e9c846ebcee34ec02a42/librt-0.16.0-cp311-cp311-win_arm64.whl", hash = "sha256:1d28ae980ae2218f9c5b95d191e947296f918c9bf0b400d467a9430275bbe678", size = 115910, upload-time = "2026-09-29T00:52:40.131Z" }, + { url = "https://files.pythonhosted.org/packages/ad/76/bbdaeb87b7c47b5c7343e90222b9bfa4e4a8a83f647e02933ad0129225b1/librt-0.16.0-cp312-cp312-macosx_10_13_x86_64.whl", hash = "sha256:fe52bf4641069e7978a14253b036cb9002def1926317e710f2e249f8a8c47742", size = 151028, upload-time = "2026-09-29T00:52:41.508Z" }, + { url = "https://files.pythonhosted.org/packages/fd/0c/ab8ed3dab0085931aec4a792c7eaac8dc6c5ff4691fda3a5360d9d8a9cd2/librt-0.16.0-cp312-cp312-macosx_11_0_arm64.whl", hash = "sha256:5bcc2c4726ced915b00de0c9856a4eeabfb3fddb93e10e0b8f735b7709358b6d", size = 155283, upload-time = "2026-09-29T00:52:42.873Z" }, + { url = "https://files.pythonhosted.org/packages/eb/36/494e79d460c80c1f030661e8287c9eca5e1ad652dc2b2180b6cd42abce0a/librt-0.16.0-cp312-cp312-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:ff7baa55f8e7c69851419e50a666015d02a74198716fd45c0125a2112e0a389f", size = 503185, upload-time = "2026-09-29T00:52:44.638Z" }, + { url = "https://files.pythonhosted.org/packages/9b/34/a8464038dd9db6e4381fa2b6eb73dc9a50888d77102c4c139304ac35cddc/librt-0.16.0-cp312-cp312-manylinux2014_i686.manylinux_2_17_i686.manylinux_2_28_i686.whl", hash = "sha256:b95d5d92ab83d39e760a52091bb1baba664f3a2351e39b1e16801e5747c2f0e9", size = 496738, upload-time = "2026-09-29T00:52:46.441Z" }, + { url = "https://files.pythonhosted.org/packages/6a/53/e0e5e334ef0c6ed27039d323819368b9ef6712be87d55ee2bf9799398afd/librt-0.16.0-cp312-cp312-manylinux2014_ppc64le.manylinux_2_17_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:b6d085d70bce51d43c5c7c36d63490770180d8779e71c49305c87b4213918de7", size = 513818, upload-time = "2026-09-29T00:52:48.068Z" }, + { url = "https://files.pythonhosted.org/packages/ad/f7/7ce72cbf19d0addd05090b339152fd0548a02562c2866a603e6e3b3da2df/librt-0.16.0-cp312-cp312-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:36e53948e99bbe3ffea257124cfcae1cfb01831555c9a9c903c9f9a72db7fd07", size = 531952, upload-time = "2026-09-29T00:52:49.816Z" }, + { url = "https://files.pythonhosted.org/packages/83/22/0b1bcb6a8e723c8b4fd60dfc8ae8ec6461c54073fbc8685efeb8d900d407/librt-0.16.0-cp312-cp312-manylinux_2_34_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:54d11f726aae9df5a6ffbbf0a03a52449bbac84a53ef03669cb41cdfd4ae41bf", size = 524465, upload-time = "2026-09-29T00:52:51.354Z" }, + { url = "https://files.pythonhosted.org/packages/64/2e/e9c23b8b9df1813da1be205deca9606beb7ddd033972246cd426d05374a0/librt-0.16.0-cp312-cp312-musllinux_1_2_aarch64.whl", hash = "sha256:4323193ac0cd025f85af531df8ba91bf24d1973b401697347a6282e8fd3fcf5e", size = 543191, upload-time = "2026-09-29T00:52:53.284Z" }, + { url = "https://files.pythonhosted.org/packages/bc/e5/6a8b21b342c03ed7e230fa3afbfd2edc58e6e87ef1f0d11fa2b9a748c252/librt-0.16.0-cp312-cp312-musllinux_1_2_i686.whl", hash = "sha256:e42f8e098b9c5396fefa05fb1cc7e33b0e08fc51da106b5de4a45fd22aac6743", size = 546979, upload-time = "2026-09-29T00:52:54.932Z" }, + { url = "https://files.pythonhosted.org/packages/84/9e/b5129023eced1be01e01c22757f53be551d463b1bb7264f787927404c1d6/librt-0.16.0-cp312-cp312-musllinux_1_2_ppc64le.whl", hash = "sha256:39ec1d5a14e37baf1450a6cabf03fe552340808bf1ad9d71824ab90117716459", size = 555317, upload-time = "2026-09-29T00:52:56.869Z" }, + { url = "https://files.pythonhosted.org/packages/71/89/28bba5938c725fe91f06bf93f7fa6c6b150229df53a87b454b0d5c2a796e/librt-0.16.0-cp312-cp312-musllinux_1_2_riscv64.whl", hash = "sha256:d1aabe3925cbb4a08d15b7b20ba4011b53019da0c4173a25155139b7b1baed65", size = 535846, upload-time = "2026-09-29T00:52:58.475Z" }, + { url = "https://files.pythonhosted.org/packages/22/92/63773026614f888c5d4e370e395ce42ca604b89f70b3acdfedbf94851b80/librt-0.16.0-cp312-cp312-musllinux_1_2_x86_64.whl", hash = "sha256:300c3ffdc459f4a779a8411ecb188e3ac0b1ff3a3a7b099642555dedae06c69b", size = 573700, upload-time = "2026-09-29T00:53:00.127Z" }, + { url = "https://files.pythonhosted.org/packages/66/8f/347d4677eefb57cd9f8e01d95a1dcee9a91b4e44e664173c32ff6f3752e5/librt-0.16.0-cp312-cp312-win32.whl", hash = "sha256:c17194318e4c0c0348b36f36c2ec7534436fe0a4c15582403162a4f08c80797a", size = 106158, upload-time = "2026-09-29T00:53:02.031Z" }, + { url = "https://files.pythonhosted.org/packages/f0/2c/5193dc81127cd5ddfad031391b046bf32dda219b38463ab872407ca30646/librt-0.16.0-cp312-cp312-win_amd64.whl", hash = "sha256:25a58a19ea8d83b68209f04912df765e9260635ef77646542ed4b4abe6bc7940", size = 126176, upload-time = "2026-09-29T00:53:03.445Z" }, + { url = "https://files.pythonhosted.org/packages/ff/3d/9668a400c8dd81d162eba38b33fa49fa6205f1a64493570a13fbb815c3ee/librt-0.16.0-cp312-cp312-win_arm64.whl", hash = "sha256:f7be7cf555bc30ec12622e9447299cc4a9b8ff307548b634794353db0c2065dc", size = 116572, upload-time = "2026-09-29T00:53:04.815Z" }, + { url = "https://files.pythonhosted.org/packages/46/cd/ae5e0e9dba45d1399aa04a5395bcc0bead40d9fa06dc903634a7b4d7473d/librt-0.16.0-cp313-cp313-macosx_10_13_x86_64.whl", hash = "sha256:c5e6144e68b577f157519f2ba88ca20e3ed61c29b00e5cdfa76cd2d45acf059a", size = 151036, upload-time = "2026-09-29T00:53:06.284Z" }, + { url = "https://files.pythonhosted.org/packages/41/5a/48a16e323c5f9447a94cce7b59babf60fa04e62c3365ecf060c77ed8b320/librt-0.16.0-cp313-cp313-macosx_11_0_arm64.whl", hash = "sha256:33f41443a1f4e1f099331b3d8120e409fbff84b9760bc1cc9ea496f37ddaa5cc", size = 155211, upload-time = "2026-09-29T00:53:07.72Z" }, + { url = "https://files.pythonhosted.org/packages/3f/29/0f59299eb4251a409b2e690ad4b7d9f8a676db829d7817ec961f32b44f7e/librt-0.16.0-cp313-cp313-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:7e510b7770bee609617a3374a96548eb114cae048023e3f049ee449e7ff2db32", size = 502587, upload-time = "2026-09-29T00:53:09.191Z" }, + { url = "https://files.pythonhosted.org/packages/de/ba/d6fb4ef8d1537c396079d72289f16be7cd35a366e5065c51253fea2760b6/librt-0.16.0-cp313-cp313-manylinux2014_i686.manylinux_2_17_i686.manylinux_2_28_i686.whl", hash = "sha256:efc49c462d4516b8a58b00b490078fa64689fd1fe66970cc190131d7afb8027e", size = 496146, upload-time = "2026-09-29T00:53:11.016Z" }, + { url = "https://files.pythonhosted.org/packages/52/fc/8c50dd4d7cc97c0ee8f252c8a3104980f234391cf1519b554e8b9de08b60/librt-0.16.0-cp313-cp313-manylinux2014_ppc64le.manylinux_2_17_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:92caf82ebef5e12d21c72242b70d1e92536f1711cf2a727a4c276de4b4469087", size = 513336, upload-time = "2026-09-29T00:53:12.655Z" }, + { url = "https://files.pythonhosted.org/packages/a9/59/16c409c56f708eda2db9a0553662845d45e3871c77d70a240dae3f3bdc56/librt-0.16.0-cp313-cp313-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:17bac7f7a16b328fff77e440287693eb017abde913595b5827ebccbc21ecd8a6", size = 531689, upload-time = "2026-09-29T00:53:14.39Z" }, + { url = "https://files.pythonhosted.org/packages/88/82/d34772a6c29d1446dcca6e64d74062efd508523ab351aa16625a4689d5cc/librt-0.16.0-cp313-cp313-manylinux_2_34_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:5b976054553670829985ed767feb78fb6bcede0175327c4844dd5c281c1be659", size = 524537, upload-time = "2026-09-29T00:53:16.064Z" }, + { url = "https://files.pythonhosted.org/packages/77/8f/24c5631313746131ccee53bc91fdc8374f9cf25e0082a1fee9c93bb98acc/librt-0.16.0-cp313-cp313-musllinux_1_2_aarch64.whl", hash = "sha256:0058f9d68721094105917254c72ac0569117bb7b13b9769cf45d26d89f9d21cd", size = 543231, upload-time = "2026-09-29T00:53:17.939Z" }, + { url = "https://files.pythonhosted.org/packages/f2/cb/5f8e0d41dbd8b499c2265e939c31acc9ba59845565bf99539ad1c06aebcf/librt-0.16.0-cp313-cp313-musllinux_1_2_i686.whl", hash = "sha256:30b7beaf3f4487b7d8adef1f158b49067cb4d5a19fa7a3bf31a4e7a820e435c5", size = 546499, upload-time = "2026-09-29T00:53:19.666Z" }, + { url = "https://files.pythonhosted.org/packages/be/61/063052de441d1385f59cea4223f184bf9e5d125de1ae3239b490aa1e486e/librt-0.16.0-cp313-cp313-musllinux_1_2_ppc64le.whl", hash = "sha256:468df902df016a06eb0e40b0747dc8d14e47d7a38b18b63b1fb167d85cb94d63", size = 555261, upload-time = "2026-09-29T00:53:21.292Z" }, + { url = "https://files.pythonhosted.org/packages/14/11/a2ada0529372268d6401afa9d457a095b68cd7753532b6f7f33049a19b43/librt-0.16.0-cp313-cp313-musllinux_1_2_riscv64.whl", hash = "sha256:aea7b1f2b125dad5de85f049136651bff256c883c65e6b9209b2da0a1ac3cdef", size = 536029, upload-time = "2026-09-29T00:53:23.07Z" }, + { url = "https://files.pythonhosted.org/packages/23/9d/5bb6d38853382986dca702fc7e06c8256d30f5fa0676d882773b744610dc/librt-0.16.0-cp313-cp313-musllinux_1_2_x86_64.whl", hash = "sha256:a8afb6557920860b7a3a596eb804cf37e09e7cf8a803db2478c202acc72d8c2e", size = 573748, upload-time = "2026-09-29T00:53:24.696Z" }, + { url = "https://files.pythonhosted.org/packages/99/f6/0025cde35ff7f607684dc775a2b2d732cce2561cec050a6ee1fb2e1fc6fe/librt-0.16.0-cp313-cp313-pyemscripten_2025_0_wasm32.whl", hash = "sha256:77c7a2b4fe2c1369e0d5aa1cade26740a7b14be32fbc9a5535d617d20065c39d", size = 79861, upload-time = "2026-09-29T00:53:26.089Z" }, + { url = "https://files.pythonhosted.org/packages/93/93/303b8592909bd583f83f02818ffbea3f7647ca1e22b1cd5465d04b8145fe/librt-0.16.0-cp313-cp313-win32.whl", hash = "sha256:02d89c813d5ff74b17df72d3a34819d132cd168e56b81bf755b809bd9e46b8c4", size = 106236, upload-time = "2026-09-29T00:53:27.43Z" }, + { url = "https://files.pythonhosted.org/packages/cf/24/80bbb463c60ed18e29cb26aba386ddb76609580e4e7160710e550587018c/librt-0.16.0-cp313-cp313-win_amd64.whl", hash = "sha256:14ed6ebe3e4f85f326d7920011ad30ff49ed9334e62cf88caef9ba973d9e3a92", size = 126180, upload-time = "2026-09-29T00:53:28.7Z" }, + { url = "https://files.pythonhosted.org/packages/43/80/b1a6fbdd7da825cdd55c71aa81eb6cfa82c513c360774152eacb50b4a771/librt-0.16.0-cp313-cp313-win_arm64.whl", hash = "sha256:83d4041a3d9b2fd053a8a4e1f22878b3e5833e2712956382d5c048d791454e91", size = 116557, upload-time = "2026-09-29T00:53:30.012Z" }, + { url = "https://files.pythonhosted.org/packages/1e/93/9e0cf7da129a93c3dc7f45bc3cd4a660f2aaa995aa8a6f95c2583ef41239/librt-0.16.0-cp314-cp314-macosx_10_15_x86_64.whl", hash = "sha256:931a0bb0fcac88f263e269e46eb30ba8e21402cd3c62ca40cb97034c0693fab1", size = 149844, upload-time = "2026-09-29T00:53:31.391Z" }, + { url = "https://files.pythonhosted.org/packages/8f/26/8a90d2a8f2b2e471bb486b7aec117b8ae622715ed6c39853aec48ea20073/librt-0.16.0-cp314-cp314-macosx_11_0_arm64.whl", hash = "sha256:1bc17e54e5305f8d40b7ca203671ff5a9e59c1d0f8ea0f625dcca53a3984de11", size = 154111, upload-time = "2026-09-29T00:53:32.718Z" }, + { url = "https://files.pythonhosted.org/packages/35/ce/67abb46258da4d3e42ff5b141db6f38c59357bef84c7979f183b22f924f1/librt-0.16.0-cp314-cp314-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:877698bf6bca5721d8be345f2fe09778e40ecadea8b58c73075f2b1a53666bf2", size = 494234, upload-time = "2026-09-29T00:53:34.466Z" }, + { url = "https://files.pythonhosted.org/packages/12/f9/ea7162414a16f8f1bbd3b493ad6d22b926c5471916e078350bdbca8c4e5f/librt-0.16.0-cp314-cp314-manylinux2014_i686.manylinux_2_17_i686.manylinux_2_28_i686.whl", hash = "sha256:5981c011b306781ce561e18e14230a14524a3d8109b97553666c942c18f31a96", size = 491173, upload-time = "2026-09-29T00:53:36.124Z" }, + { url = "https://files.pythonhosted.org/packages/b1/09/9b3e869060dd33f9989b80ba4fea306f6db8cecefcdfe6d7346ef0603f65/librt-0.16.0-cp314-cp314-manylinux2014_ppc64le.manylinux_2_17_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:afced3dfc17cd805ecf7a3d77996a71cf5f2c75aa66eb0c21a9930f4fc992f86", size = 505547, upload-time = "2026-09-29T00:53:37.686Z" }, + { url = "https://files.pythonhosted.org/packages/50/07/79007d2165f649ea93e08c0962d1d62c70af9cde77965255095bf9d96f9a/librt-0.16.0-cp314-cp314-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:ca8052401c55d7511dda6760719fda7618067e83535d7d0010096d216c34b667", size = 523057, upload-time = "2026-09-29T00:53:39.301Z" }, + { url = "https://files.pythonhosted.org/packages/61/0c/8fbaff66d0ba376d8864653f5acce1569bae27e89648671f26f7eca67ab8/librt-0.16.0-cp314-cp314-manylinux_2_34_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:1e511762a074005bb0aa569166779834e75e438370226930d0ce1866d4b6a33b", size = 515158, upload-time = "2026-09-29T00:53:41.012Z" }, + { url = "https://files.pythonhosted.org/packages/df/2e/23ff0dece76f07a4124413a57682efa0bbeb5765ac0122bc0955513f82ca/librt-0.16.0-cp314-cp314-musllinux_1_2_aarch64.whl", hash = "sha256:f1e8591bd8a5a628cd7f07954c6a1592359a878bf032957a8e9057a41d644311", size = 534183, upload-time = "2026-09-29T00:53:42.451Z" }, + { url = "https://files.pythonhosted.org/packages/26/c4/e11dea21d9a29486eba78887380374189d472734fa32cc50cb37ca44d3d0/librt-0.16.0-cp314-cp314-musllinux_1_2_i686.whl", hash = "sha256:a4aaefb4ba6c07e1aeebb2795c8958148f1d6f9af3b555b53d23d766edb6d67a", size = 540663, upload-time = "2026-09-29T00:53:44.272Z" }, + { url = "https://files.pythonhosted.org/packages/44/75/e873ae158a8b7f5359be33e7fd6c1fbe02d9a78a3e89a77de6b0e837f476/librt-0.16.0-cp314-cp314-musllinux_1_2_ppc64le.whl", hash = "sha256:d92db7a0f6aee44f1baee94750457e8d2d1c6ccea41842de6268d34e8dc7eddd", size = 545908, upload-time = "2026-09-29T00:53:45.812Z" }, + { url = "https://files.pythonhosted.org/packages/ba/36/8939d3f6a93e11bd9592e6fe28d2b44f1c2dc4bed6e22e72359a91e18ffb/librt-0.16.0-cp314-cp314-musllinux_1_2_riscv64.whl", hash = "sha256:378dfaffb38e59c24a87cde5713cd865d51ff7383fa12947f3907f306ea1ca55", size = 523302, upload-time = "2026-09-29T00:53:47.596Z" }, + { url = "https://files.pythonhosted.org/packages/71/14/35309f44a077f0f42ade0e2e7cd88c0cea760c661af205c01ea90d3c0e1f/librt-0.16.0-cp314-cp314-musllinux_1_2_x86_64.whl", hash = "sha256:3e0c39bdc85370422e8b637be76eb1fd07d30967551b03e62267dd156f553152", size = 565911, upload-time = "2026-09-29T00:53:49.278Z" }, + { url = "https://files.pythonhosted.org/packages/78/0c/df6255b94967f3159ebc46f08d8e783c12da6ee269ddb74b2efed63c640c/librt-0.16.0-cp314-cp314-pyemscripten_2026_0_wasm32.whl", hash = "sha256:1b384b90ab79a7bc30b566895809a636e0666f21f3cf12b54823d025b7e83839", size = 79119, upload-time = "2026-09-29T00:53:50.938Z" }, + { url = "https://files.pythonhosted.org/packages/6b/44/d30d5a5461378c9d33f36736c6791c3b4c4ba4b1ffe0da7350aedcb2c9a0/librt-0.16.0-cp314-cp314-win32.whl", hash = "sha256:52327da75a94012e7f932f913d20d3876bed3c102be00e6c3e8600ff7bdd58a7", size = 100140, upload-time = "2026-09-29T00:53:52.205Z" }, + { url = "https://files.pythonhosted.org/packages/c2/98/769712f356a1e897df3581bb0c3100375d054a000de26099360ea65b5111/librt-0.16.0-cp314-cp314-win_amd64.whl", hash = "sha256:3f0b8114c44b2ac06ff5dacd08e07e8e807ff4f46083f2a1602685122559be41", size = 120464, upload-time = "2026-09-29T00:53:53.495Z" }, + { url = "https://files.pythonhosted.org/packages/bf/d3/ae2abccc8bdc8b063c1613a77686e17b74e9d3d60cfe6f12c63fa2821b9f/librt-0.16.0-cp314-cp314-win_arm64.whl", hash = "sha256:8caf96a4ef8fb27d0ac0d1ad8337d26a240acd4a02fe4345d0a8f264753e8f99", size = 110889, upload-time = "2026-09-29T00:53:54.817Z" }, + { url = "https://files.pythonhosted.org/packages/e4/26/0737d4be058dd6376eade7dd8b380d4b869b6394cee929393a5a431c45bb/librt-0.16.0-cp314-cp314t-macosx_10_15_x86_64.whl", hash = "sha256:953107e2f68d0f3512c48f898b0dbf0ce5cc52bba0f318d847c985dc555ee4cc", size = 159838, upload-time = "2026-09-29T00:53:56.22Z" }, + { url = "https://files.pythonhosted.org/packages/01/96/9bc96531d7c620e9949af904470f02e3fa8f35129ab8e8f281c51eaa3788/librt-0.16.0-cp314-cp314t-macosx_11_0_arm64.whl", hash = "sha256:ad37d5b9abd49c9a655dcda7ea52a8a752884062ef1ee71ae17c2f2a0f81fe6a", size = 162018, upload-time = "2026-09-29T00:53:57.532Z" }, + { url = "https://files.pythonhosted.org/packages/0e/fa/b0289dcb186eb3f97221ba00da5f4bd3ba7fa5e99752d48aa8d336615334/librt-0.16.0-cp314-cp314t-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:2c4aa329c17bd1aaea4f6e89335d8ccd494b3a5830b6654462273e50e11023f0", size = 703983, upload-time = "2026-09-29T00:53:59.024Z" }, + { url = "https://files.pythonhosted.org/packages/d9/ab/05ebbbde7530fc5eeb58fbd1581522a9f64f132fa703c4c595eed6e14760/librt-0.16.0-cp314-cp314t-manylinux2014_i686.manylinux_2_17_i686.manylinux_2_28_i686.whl", hash = "sha256:0ead24d2562a49473dddd9efef8581f020007eb0054389c3ee3ffad38b1ca4c9", size = 683847, upload-time = "2026-09-29T00:54:00.671Z" }, + { url = "https://files.pythonhosted.org/packages/a3/75/f52aeecd4dbadbddf80725ba7de126d8bd5d0eb66247eae17a81ed90dd4d/librt-0.16.0-cp314-cp314t-manylinux2014_ppc64le.manylinux_2_17_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:02118f56a9c36ddd07dfd9b919d9ecc117ba20a90987d56aa4c429fa34509188", size = 698045, upload-time = "2026-09-29T00:54:02.247Z" }, + { url = "https://files.pythonhosted.org/packages/e6/55/fa277a835cd6eb42380591ceb85f48c5b4d2b2d7e2cb9869e1e17d24d237/librt-0.16.0-cp314-cp314t-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:4e29522c62e28595ff7e324c6834ade51127707f0e255b18d1c1cf03d39c1048", size = 724640, upload-time = "2026-09-29T00:54:04.078Z" }, + { url = "https://files.pythonhosted.org/packages/25/e4/2cf64354f3fde8ebd591b3b48f96510bee24ac5f7d1e567adbbe210abdbd/librt-0.16.0-cp314-cp314t-manylinux_2_34_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:3ff4b2367926b69c6215635902cccb04048e73094e9862900d27cb2c6bbff143", size = 731748, upload-time = "2026-09-29T00:54:05.741Z" }, + { url = "https://files.pythonhosted.org/packages/f8/c9/c180af3e94e01aa529fa93d7733fec2abc47f222345400cf21d5481d5f8a/librt-0.16.0-cp314-cp314t-musllinux_1_2_aarch64.whl", hash = "sha256:c6f1b27bf1632a7e016af9f145f82be95e1edd7721a646505c21059257cb5a04", size = 754960, upload-time = "2026-09-29T00:54:07.38Z" }, + { url = "https://files.pythonhosted.org/packages/95/d6/01073aa78c58f356b10d9c57b3fe9abb143df338142bcd316df417b89db0/librt-0.16.0-cp314-cp314t-musllinux_1_2_i686.whl", hash = "sha256:5696d7f52e7b37217cb3a8f92c744fe835942602fdd4c1a8bc4741d3bfdce15e", size = 747000, upload-time = "2026-09-29T00:54:09.06Z" }, + { url = "https://files.pythonhosted.org/packages/f7/d8/1de3783908658d697a8cfc00582f61299ffba7796d7260c112e4700b1109/librt-0.16.0-cp314-cp314t-musllinux_1_2_ppc64le.whl", hash = "sha256:6072e92dd876ff6ceeb6cf371e35e51f479349837391341f479b08df4564242b", size = 749569, upload-time = "2026-09-29T00:54:10.702Z" }, + { url = "https://files.pythonhosted.org/packages/b7/32/e817f66c96d6caa8bb8435ff93c4c220624d59efc506be59ab98fcd01d0c/librt-0.16.0-cp314-cp314t-musllinux_1_2_riscv64.whl", hash = "sha256:39ca4f2f2fe05de8e63493da592d84311adabe5bef52b193851981da9816b302", size = 729483, upload-time = "2026-09-29T00:54:12.25Z" }, + { url = "https://files.pythonhosted.org/packages/7c/c9/23992ccd2b9d22798fdd0f61353183a47414e651da782ad83eb680f50833/librt-0.16.0-cp314-cp314t-musllinux_1_2_x86_64.whl", hash = "sha256:f9807485a908f00355820f18e91e045ffdcdc5adb68aaec40a1e2b88c5f7bba1", size = 776584, upload-time = "2026-09-29T00:54:13.837Z" }, + { url = "https://files.pythonhosted.org/packages/67/3b/e8af957f08e6e2e8e566b099e43d2748418aa565df6ee1d9d3331209fdfe/librt-0.16.0-cp314-cp314t-win32.whl", hash = "sha256:94be5cb7bca4df6201f4183e9e4fa2086c655283d20b38cd84500a69057575a7", size = 104402, upload-time = "2026-09-29T00:54:15.568Z" }, + { url = "https://files.pythonhosted.org/packages/ca/1c/946e6443d7cd32347a086043395e421e52fd603c9163d2ec970ceab8eed6/librt-0.16.0-cp314-cp314t-win_amd64.whl", hash = "sha256:d46ca272b251d033dd4527b0dec5f261a28a52bd5fa0f99c117b0a1f8588cc2d", size = 126227, upload-time = "2026-09-29T00:54:17.165Z" }, + { url = "https://files.pythonhosted.org/packages/c2/a3/bc4f9959d3c62bcbf9e5fd3470a8bbd8bf33224ed4c25d7fba173201b8ac/librt-0.16.0-cp314-cp314t-win_arm64.whl", hash = "sha256:b9d6d4b14e92d876f8026b54c20c445f36425214c1081dc76f74e40db386b82b", size = 116493, upload-time = "2026-09-29T00:54:18.567Z" }, + { url = "https://files.pythonhosted.org/packages/b6/4b/10fdb42dfab4c1533e1570e686b18e86ff4328b406361b39fb3016667638/librt-0.16.0-cp315-cp315-macosx_10_15_x86_64.whl", hash = "sha256:6fe436af2eaf630474f491af5d032cbe45f93fcff5c3b9fe4ab194a7255b20ff", size = 149829, upload-time = "2026-09-29T00:54:19.933Z" }, + { url = "https://files.pythonhosted.org/packages/8e/30/a90ca13f1d3d91af1680000a4038536907018fbd51764767287a05d28b8b/librt-0.16.0-cp315-cp315-macosx_11_0_arm64.whl", hash = "sha256:8ff5d26c529336be9bd7ae04483235d77778ee7d6444a95353102b542601ce81", size = 154522, upload-time = "2026-09-29T00:54:21.306Z" }, + { url = "https://files.pythonhosted.org/packages/5c/dd/bcf364eacfa070bb1fc88d503111ae7177914197ba1fa717c748926e7930/librt-0.16.0-cp315-cp315-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:909d8e3c1faee44cb762b1c519ff8613dcc5ceae5c99987a00917b5a31fd1d6a", size = 497593, upload-time = "2026-09-29T00:54:22.798Z" }, + { url = "https://files.pythonhosted.org/packages/18/c1/2c4e81e347bdabfe8346bfc6dc37ae5e154d61c88e30848ec0606025a5e0/librt-0.16.0-cp315-cp315-manylinux2014_i686.manylinux_2_17_i686.manylinux_2_28_i686.whl", hash = "sha256:6d4a64283ee61824b5790de882bc68e2d9d7a5143537cb7a966f7354f71646d4", size = 480459, upload-time = "2026-09-29T00:54:24.364Z" }, + { url = "https://files.pythonhosted.org/packages/ab/d7/fef2a3cb8400701be496f6e459876f650b3451f407e30cb243de571ae615/librt-0.16.0-cp315-cp315-manylinux2014_ppc64le.manylinux_2_17_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:5810ba811297fdf37a1531a57667cb8ace0842013ca8606bf9eb7c24cf4be154", size = 507814, upload-time = "2026-09-29T00:54:25.997Z" }, + { url = "https://files.pythonhosted.org/packages/e1/6f/53762927a32e9dc9eb1d1c1f3528da281290c86d96929a8af652f671266e/librt-0.16.0-cp315-cp315-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:e56aaf8c167548dc8e5d6f3bd0f48dcdd299a23c73be3f744aab79d99e9c7f5d", size = 525114, upload-time = "2026-09-29T00:54:27.747Z" }, + { url = "https://files.pythonhosted.org/packages/6c/67/0b9d031f303c4e8c691a9a8ef9d272f13b530c11cc563b819df621b6a348/librt-0.16.0-cp315-cp315-manylinux_2_34_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:8f36c58e33b304b525c6c9c5076399c6ebf1109e17b9051a05a407b091b9215b", size = 520465, upload-time = "2026-09-29T00:54:29.331Z" }, + { url = "https://files.pythonhosted.org/packages/ae/d5/2056a3a85864e882eb17a203a10ddb26fa748bc9718ec67e79059ab46cae/librt-0.16.0-cp315-cp315-musllinux_1_2_aarch64.whl", hash = "sha256:242e00b3d4fa37c3d3c1ca5f5c9adb7d909ddb1eac9c41f2787320d00caa0af2", size = 537257, upload-time = "2026-09-29T00:54:30.915Z" }, + { url = "https://files.pythonhosted.org/packages/54/57/e0d79790c163cbc0909e209a6f62e30bb713b647f905a176cdc64848fd4d/librt-0.16.0-cp315-cp315-musllinux_1_2_i686.whl", hash = "sha256:0253721561787b8df8443eb347b7a6461015354e5bdd37ee38a41fef220d2bb0", size = 527442, upload-time = "2026-09-29T00:54:32.478Z" }, + { url = "https://files.pythonhosted.org/packages/aa/50/1c0c95aba7af51f4752ea34fbf2eb79b36e7cae3a72536248e8c735de735/librt-0.16.0-cp315-cp315-musllinux_1_2_ppc64le.whl", hash = "sha256:3e483a8d69ede8067db70c0e83007423b6925de6fd53afed01d66160f2e9398c", size = 548139, upload-time = "2026-09-29T00:54:34.183Z" }, + { url = "https://files.pythonhosted.org/packages/98/91/a8a43dd5138d4f55f88846b8f0c85454a4fad69952cebfcff9948f831290/librt-0.16.0-cp315-cp315-musllinux_1_2_riscv64.whl", hash = "sha256:69ba927445cfaaffb4081003ef5224c55a5c2ab67ef956f416ef744916e44121", size = 529708, upload-time = "2026-09-29T00:54:35.761Z" }, + { url = "https://files.pythonhosted.org/packages/2d/41/d5226881ab2b7c20d9d587b37bdd4a0ec8775a96d00ca87ac9f385587db4/librt-0.16.0-cp315-cp315-musllinux_1_2_x86_64.whl", hash = "sha256:d6a365f2ab45a984d0e00eee0dd17f599ceab8cadab6ea07b6111c8132fc0e42", size = 567781, upload-time = "2026-09-29T00:54:37.409Z" }, + { url = "https://files.pythonhosted.org/packages/bb/bf/2345ba57a626e8c78c4ddcc724636a8df5a593fe46c1ee76bbf477e32b0d/librt-0.16.0-cp315-cp315-pyemscripten_2026_5_wasm32.whl", hash = "sha256:f01f3805f2dae4781c0c34b440e31740d082950bdaf89a6f601ad589a28af57a", size = 79177, upload-time = "2026-09-29T00:54:38.792Z" }, + { url = "https://files.pythonhosted.org/packages/1e/40/99e77936cc9207f629b077bf7cbc5e2f0827cddd7c29792f04ef881bd2c3/librt-0.16.0-cp315-cp315-win32.whl", hash = "sha256:b0e3e721c75d2e79a76d4422c79d7ba705fe1bbafec907037fe7a657a480a0e3", size = 100127, upload-time = "2026-09-29T00:54:40.251Z" }, + { url = "https://files.pythonhosted.org/packages/56/1e/801fe26bc622061b9dfd010e166d94142cb774b733217e6d98d1c0cf2638/librt-0.16.0-cp315-cp315-win_amd64.whl", hash = "sha256:bc02954b1295de798bbdb0b4e2d8a28c2117de8b5c73dcbeb27dc32572dfb971", size = 120480, upload-time = "2026-09-29T00:54:41.722Z" }, + { url = "https://files.pythonhosted.org/packages/bb/a9/d533983055bd36e112627384c2c038845d1df882540b8bcefb566475640b/librt-0.16.0-cp315-cp315-win_arm64.whl", hash = "sha256:c5db585d43449a5f54303d4b2774e45e1babd975cfe1630a3d708c0b80c3e560", size = 110915, upload-time = "2026-09-29T00:54:43.052Z" }, + { url = "https://files.pythonhosted.org/packages/55/fe/d62238fa9c653b0e0613290349467cb65711b5800e8e474f45e942a0ad95/librt-0.16.0-cp315-cp315t-macosx_10_15_x86_64.whl", hash = "sha256:f06c689cb14afd9b612727553a5ec5a40febf113ca41c4413a2b0b334285884b", size = 159714, upload-time = "2026-09-29T00:54:44.497Z" }, + { url = "https://files.pythonhosted.org/packages/f3/ac/f31efe7818700be72ba4f9af8a80fa67c39808c8d26dacc55dc1f6f35172/librt-0.16.0-cp315-cp315t-macosx_11_0_arm64.whl", hash = "sha256:13b4e8aba90b0b1c82474e9844aa9ffe7ad3faa484350e1da64cb8188d903134", size = 162086, upload-time = "2026-09-29T00:54:45.968Z" }, + { url = "https://files.pythonhosted.org/packages/d9/16/4d7487bf86a9d7e8e18f37ba5538789233ea693637379b75701bd35ec9de/librt-0.16.0-cp315-cp315t-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:5a269c46ae327d8e6f8c1f85f7516cb52c0fa48127565a1105a4f4a05ff2a0b4", size = 712727, upload-time = "2026-09-29T00:54:47.486Z" }, + { url = "https://files.pythonhosted.org/packages/77/74/50cd550ccc1a517b9ed62347625c01ddf8c4afad66490312677c011458f6/librt-0.16.0-cp315-cp315t-manylinux2014_i686.manylinux_2_17_i686.manylinux_2_28_i686.whl", hash = "sha256:a33e0dae1f8592146a4764d54ce842b278732d21a84e17c3bbe6b1bc158a2248", size = 681106, upload-time = "2026-09-29T00:54:49.126Z" }, + { url = "https://files.pythonhosted.org/packages/df/5d/7293f712975ee6fdd2251411fd9ef1c62bc99c7b83ecbe62dbc999b1fab7/librt-0.16.0-cp315-cp315t-manylinux2014_ppc64le.manylinux_2_17_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:47ada6ea32636492c61aa8ad27ae3b9404bfe7a97e3ba946d1984236cc741da0", size = 705921, upload-time = "2026-09-29T00:54:50.905Z" }, + { url = "https://files.pythonhosted.org/packages/67/f7/8aab946f11d59d1bece9ffc65d994b200c789a8ca5a95c17e17e609f912a/librt-0.16.0-cp315-cp315t-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:c43bd6e642d8a248c114327f98dd25ac5a7cb5aa168ef02f0559b91874df16b8", size = 731786, upload-time = "2026-09-29T00:54:52.584Z" }, + { url = "https://files.pythonhosted.org/packages/43/76/1c42ab31e7cb8384ebf6d3af607213c495222474f4940443ae7639ab7685/librt-0.16.0-cp315-cp315t-manylinux_2_34_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:c3d1bb7841a816ace6449bb26d3f9560dbfa20e71c568d23f0f62bf1e68f50b1", size = 744964, upload-time = "2026-09-29T00:54:54.217Z" }, + { url = "https://files.pythonhosted.org/packages/6f/2e/4b19982d933d2dfced671e840b219e4c1cd3f507df6e6dabb51bbd4e3850/librt-0.16.0-cp315-cp315t-musllinux_1_2_aarch64.whl", hash = "sha256:3931f7a3db322e7f44e02a280e3949326ce9579ad388ee8d691dc7c76da9fb70", size = 765305, upload-time = "2026-09-29T00:54:55.815Z" }, + { url = "https://files.pythonhosted.org/packages/b8/1b/e872583de2dcb3ac7746e7a2321aeeb168274f3952a87dc66e05ddb29faf/librt-0.16.0-cp315-cp315t-musllinux_1_2_i686.whl", hash = "sha256:f4462528b6000afe8f16907b5c7c2553abf1df005ba5140e6eb394541c3624c3", size = 744959, upload-time = "2026-09-29T00:54:57.532Z" }, + { url = "https://files.pythonhosted.org/packages/10/de/a18c6bcfb297af2674233e90b3c3661c3a0af0f1434a0c966d2ef708a835/librt-0.16.0-cp315-cp315t-musllinux_1_2_ppc64le.whl", hash = "sha256:80039ba9b6a7d5f1a0175a4cca6bbefead87bd854c80abad1cb30afe47a830db", size = 758138, upload-time = "2026-09-29T00:54:59.484Z" }, + { url = "https://files.pythonhosted.org/packages/22/1c/0df1d732539c297bb1a093e1fe3204d2faf76a9022cf4e1d1c2fce059790/librt-0.16.0-cp315-cp315t-musllinux_1_2_riscv64.whl", hash = "sha256:7a1d272724b581bb6bc769dfdafed6da2ecc9886ba2450311de55a4ac2e1e9cd", size = 744268, upload-time = "2026-09-29T00:55:01.179Z" }, + { url = "https://files.pythonhosted.org/packages/a2/f7/ccaf31331f20c91a5bd9bd48ffc3f743f9c81bb5720cc7b2a720ca204f0a/librt-0.16.0-cp315-cp315t-musllinux_1_2_x86_64.whl", hash = "sha256:fbe4fb8c5445f7496d7f7f6bb0807875d09d47e6771ffa175fb2df2895fb86ba", size = 785783, upload-time = "2026-09-29T00:55:02.938Z" }, + { url = "https://files.pythonhosted.org/packages/c1/ea/7421d9e6db894cd6cd788b604b3394a163ded6b942052b6f96c23ccf08e1/librt-0.16.0-cp315-cp315t-win32.whl", hash = "sha256:375bfe6b572a8f6cfc398709356046173bf27e64c4c5edaf5f7062f051fb4bf9", size = 104355, upload-time = "2026-09-29T00:55:04.533Z" }, + { url = "https://files.pythonhosted.org/packages/85/6d/7c31a506eb847bc58aeb2402d58e17605e821f68ab5df5e66fe274a6df41/librt-0.16.0-cp315-cp315t-win_amd64.whl", hash = "sha256:bd3150023d3dc2bc70f3784e59ffa1140d56ddba3d8125b3d6f9f85221279bfc", size = 126134, upload-time = "2026-09-29T00:55:06.017Z" }, + { url = "https://files.pythonhosted.org/packages/36/69/7a5d10ac409c4da0355e054a14371871da9b5557fcc42772cd00181c6cce/librt-0.16.0-cp315-cp315t-win_arm64.whl", hash = "sha256:8ceafb70f2a4f0826f11031942e59c0728fd98da112dc346d4352bde1e486866", size = 116391, upload-time = "2026-09-29T00:55:07.484Z" }, +] + +[[package]] +name = "matplotlib" +version = "3.11.2" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "contourpy", version = "1.3.3", source = { registry = "https://pypi.org/simple" }, marker = "python_full_version < '3.12'" }, + { name = "contourpy", version = "1.4.0", source = { registry = "https://pypi.org/simple" }, marker = "python_full_version >= '3.12'" }, + { name = "cycler" }, + { name = "fonttools" }, + { name = "kiwisolver" }, + { name = "numpy", version = "2.4.6", source = { registry = "https://pypi.org/simple" }, marker = "python_full_version < '3.12'" }, + { name = "numpy", version = "2.5.4", source = { registry = "https://pypi.org/simple" }, marker = "python_full_version >= '3.12'" }, + { name = "packaging" }, + { name = "pillow" }, + { name = "pyparsing" }, + { name = "python-dateutil" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/e7/c8/9aa712a0afb882649424dd8de8ad9aa6235e796e84c6052e8f6dc1598d0d/matplotlib-3.11.2.tar.gz", hash = "sha256:cec596316640f2b394b8f0daa0ea61a8eae82d017b620b9f202befb972a59ea4", size = 32660610, upload-time = "2026-09-11T19:05:31.214Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/c0/a9/a7fc5a63c427aeb9a4b99d6a0449f891be0952d3aee2b85229099f7b9f54/matplotlib-3.11.2-cp311-cp311-macosx_10_12_x86_64.whl", hash = "sha256:5e1e923a3fc3326b99ec0a6ff1ab1338ac6c6cc62ad9d8a9c944197c7f8c6221", size = 9461046, upload-time = "2026-09-11T19:02:59.277Z" }, + { url = "https://files.pythonhosted.org/packages/0b/d7/59d8ee63e4dee845d953515c50d67e6b8cefa953b453c6b803f197ae556e/matplotlib-3.11.2-cp311-cp311-macosx_11_0_arm64.whl", hash = "sha256:c27e577ece613ea12a0790e00b4eb80d901c59a3e16cad474f31d8b1529690b9", size = 9290872, upload-time = "2026-09-11T19:03:02.295Z" }, + { url = "https://files.pythonhosted.org/packages/2e/a4/1ca765c63a60e3096ae2ad132eb25f6f7eab0377418cd0472808befd9665/matplotlib-3.11.2-cp311-cp311-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:07d9b9fa60cd4c393692f50d0bb03123242ddf61c99bb0e95e75feb354e7c1a8", size = 9854405, upload-time = "2026-09-11T19:03:04.796Z" }, + { url = "https://files.pythonhosted.org/packages/d7/3d/f28ca971a4ad0ed83ebdd60230b09d99de8b91aba47964fac3a81eb2e2ba/matplotlib-3.11.2-cp311-cp311-manylinux_2_26_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:30ec15d7eefee71de16b7c689b42ba49715a4644650b76a6c7c70d79daf24e91", size = 10660507, upload-time = "2026-09-11T19:03:07.402Z" }, + { url = "https://files.pythonhosted.org/packages/f7/0c/a743556bc64a9b57d2246b14d91b713078adfd2df4199f542df1861737a9/matplotlib-3.11.2-cp311-cp311-musllinux_1_2_x86_64.whl", hash = "sha256:116cdb0eb0eb5644fc98eb2975d4b4dd4ad35e5c8e6b851c22e6976f371f7ab5", size = 10801141, upload-time = "2026-09-11T19:03:11.357Z" }, + { url = "https://files.pythonhosted.org/packages/66/54/91fe3fdfa29100b4668cbf9721577a10acde6e009694618230e4e7a97527/matplotlib-3.11.2-cp311-cp311-win_amd64.whl", hash = "sha256:a24d5fd36e4f0e742c3851dcd20810e56a95633a342e4bf6cb591c678e8fe61f", size = 9329982, upload-time = "2026-09-11T19:03:14.25Z" }, + { url = "https://files.pythonhosted.org/packages/49/c1/13b70871c98d7e93f0d2ed71f226b8d6f873f1372124046068a6b4902fb5/matplotlib-3.11.2-cp311-cp311-win_arm64.whl", hash = "sha256:57b9ea60a835937c2012861923cbb91f47db8565775d326d1c42fc926aa10351", size = 9023542, upload-time = "2026-09-11T19:03:16.878Z" }, + { url = "https://files.pythonhosted.org/packages/cb/ce/1bfcc4873b121597791ad74032943b123218c0613af0b97e8dd05e916fb1/matplotlib-3.11.2-cp312-cp312-macosx_10_13_x86_64.whl", hash = "sha256:ef752769cd962f39ea0b6ffc82d1ea43a0012c5a6157c7a075212fa509cfcff2", size = 9476466, upload-time = "2026-09-11T19:03:19.45Z" }, + { url = "https://files.pythonhosted.org/packages/a6/c4/7f5f3601ee69baf072c0c7d3ce60c03e0618621c5a56c34a62a460e29d11/matplotlib-3.11.2-cp312-cp312-macosx_11_0_arm64.whl", hash = "sha256:ef31985c4dedb5f1424e1aec6849a47dd37689cb7fa3c20b1b82187f26806261", size = 9305583, upload-time = "2026-09-11T19:03:22.39Z" }, + { url = "https://files.pythonhosted.org/packages/b8/90/2b3fd67ee273163faeda6d514be70b5596eaa0fd77b60ffc294ad0b34f5f/matplotlib-3.11.2-cp312-cp312-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:df4f7784aca81a94f254c0a2767d592ee25f407e488f5fa7203e51093fb6ca27", size = 9860353, upload-time = "2026-09-11T19:03:25.049Z" }, + { url = "https://files.pythonhosted.org/packages/f4/84/32549e7a462dc311aed2ab62e5d2538028840b5c53a8be0195c789937c3d/matplotlib-3.11.2-cp312-cp312-manylinux_2_26_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:1b9a7ad579856284135e401ecc918c5f8a017ee30539298862a109f51b971710", size = 10670348, upload-time = "2026-09-11T19:03:28.108Z" }, + { url = "https://files.pythonhosted.org/packages/84/39/02e21b74f7439bd643d717ea846006d46e309e6d29b632b57751265311d6/matplotlib-3.11.2-cp312-cp312-musllinux_1_2_x86_64.whl", hash = "sha256:3aa4b8516fd26659e4363abbf317c703d9116496c5db2e9d0609a2866dd39dd2", size = 10810580, upload-time = "2026-09-11T19:03:31.402Z" }, + { url = "https://files.pythonhosted.org/packages/04/93/0d239bde12b308262265c2d98909b2f0ecad2b5deeae96241642e822e9c7/matplotlib-3.11.2-cp312-cp312-win_amd64.whl", hash = "sha256:c5c1c68ee401fc98271263410f0e5ce88285abacf7627132914e8adf3d70ff43", size = 9349409, upload-time = "2026-09-11T19:03:34.195Z" }, + { url = "https://files.pythonhosted.org/packages/49/a8/06baf901c02246c8a222b655cc4540ef9c15e1549a04e07927f9cde716c3/matplotlib-3.11.2-cp312-cp312-win_arm64.whl", hash = "sha256:643ff850d8e0f5b8319337f87ed3cb59506afb3df3cc48de777d85871233be7b", size = 9028084, upload-time = "2026-09-11T19:03:37.032Z" }, + { url = "https://files.pythonhosted.org/packages/bf/ce/fc58855a3d17f6cb8a87b6f9ccd100f65432a3a7ffed57309cbf14c78987/matplotlib-3.11.2-cp313-cp313-macosx_10_13_x86_64.whl", hash = "sha256:7d43ff8cebb50840648cb6429b2228621dbd709010f3117b3740abb20abf21c0", size = 9476987, upload-time = "2026-09-11T19:03:40.17Z" }, + { url = "https://files.pythonhosted.org/packages/a9/29/2a47b700545aca050e0ed73c89ba21e08407e11f5f42a567421b220de6c1/matplotlib-3.11.2-cp313-cp313-macosx_11_0_arm64.whl", hash = "sha256:2a8285cea8ef4d92aa041d1c33788bcae82248503300f93f1ca2136b9049452f", size = 9306050, upload-time = "2026-09-11T19:03:43.103Z" }, + { url = "https://files.pythonhosted.org/packages/a9/df/e7c10292f348cf55178a5b3c0092a38ea68891a3945f199ae8b7b8ac9dc1/matplotlib-3.11.2-cp313-cp313-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:1944895967f87c84c9b4bad29a707b31f5b36df4a3a2339ea1f8ea3ce5105539", size = 9861060, upload-time = "2026-09-11T19:03:46.044Z" }, + { url = "https://files.pythonhosted.org/packages/e4/b9/b978de6d43d47f1d4d8e653fac32c16d8d2e5f46bbbb822e26d7cb1adece/matplotlib-3.11.2-cp313-cp313-manylinux_2_26_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:af2661f6ac6bbd1d081996f54fdd9385715c625ace8cec0869055c0cfbf38981", size = 10670885, upload-time = "2026-09-11T19:03:48.942Z" }, + { url = "https://files.pythonhosted.org/packages/a3/80/d72f61bb2631ac15d17fada1f17e0f99fecba69d8376d34f17cf93b7eddd/matplotlib-3.11.2-cp313-cp313-musllinux_1_2_x86_64.whl", hash = "sha256:7e5a90f8a707ebb6004a713b1cff091a40cc5df4c7c1cb165e5a505ebc11c292", size = 10811222, upload-time = "2026-09-11T19:03:51.665Z" }, + { url = "https://files.pythonhosted.org/packages/94/2e/6d1d33f242c705b5c6c6fb3f0fc141add51cb1abb62912e59de0835d28c4/matplotlib-3.11.2-cp313-cp313-win_amd64.whl", hash = "sha256:bc067c462a86f0e57bf52fc6e058d90171a5420007dac00c46e90153050c69a7", size = 9349539, upload-time = "2026-09-11T19:03:54.534Z" }, + { url = "https://files.pythonhosted.org/packages/bb/bb/c0aa86a7f4c0e62f1c3c0711e361f6f23f87eb9018732a5e11994bd1b7ec/matplotlib-3.11.2-cp313-cp313-win_arm64.whl", hash = "sha256:3e8576f7c47e02fd4f21f44171302d2d1d58d4471d46da3d71fe8899d19539d9", size = 9028295, upload-time = "2026-09-11T19:03:57.261Z" }, + { url = "https://files.pythonhosted.org/packages/54/b4/4facdb700bc236e4c676495ca3798d8134ab8668cbfe0f6b2e9ebf962a38/matplotlib-3.11.2-cp314-cp314-macosx_10_15_x86_64.whl", hash = "sha256:c9721f81275499da1feeb36a2cf8192ea086283b3bd16b7dc4c9d7aedb7396d6", size = 9478976, upload-time = "2026-09-11T19:04:00.12Z" }, + { url = "https://files.pythonhosted.org/packages/4c/a0/bba05f0a25bfeb0475c2f2a87958b31103aa6b7a5386017afb1a96212825/matplotlib-3.11.2-cp314-cp314-macosx_11_0_arm64.whl", hash = "sha256:1a3040b209f3968b4e84161df7b174f07a9fad33b0f2d7e48ea3bbd3075e2863", size = 9307932, upload-time = "2026-09-11T19:04:02.976Z" }, + { url = "https://files.pythonhosted.org/packages/d4/42/81d5cba4bb39b41b0998efc880e129f2d7cdb94256ca49678fdcd9c21be5/matplotlib-3.11.2-cp314-cp314-manylinux_2_26_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:bf3fe71fbfb8ec0e310e0bc8537c3405a01f38f25f9394ed2135e6202fed542b", size = 10674370, upload-time = "2026-09-11T19:04:06.912Z" }, + { url = "https://files.pythonhosted.org/packages/29/5e/52f56f93d5b20815ee0ae352f7a38eff25580edb180b4fa7df87cfeba68f/matplotlib-3.11.2-cp314-cp314-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:8c8255de28f986d935a64c9ca71c0ec2d2f41d355691f5ea684725dc91413f71", size = 10954597, upload-time = "2026-09-11T19:04:09.948Z" }, + { url = "https://files.pythonhosted.org/packages/6e/ae/f8918a1b564db49c53cd603eb3c8748c5767e4ce0fb2cf1d7c38653e4c5d/matplotlib-3.11.2-cp314-cp314-musllinux_1_2_x86_64.whl", hash = "sha256:a8756cc73d9af9a7fe0deb54ea2e75ef73b01d9e575877e72acad5458e660943", size = 10812583, upload-time = "2026-09-11T19:04:12.887Z" }, + { url = "https://files.pythonhosted.org/packages/ea/61/4315ef0d63937fe56f451887b4a0be3b8e8b22cf1d92a8d157caac54570d/matplotlib-3.11.2-cp314-cp314-win_amd64.whl", hash = "sha256:ecea603dd2fbf8242fd31a305a8b12a4ece2de28096870c65fdd0d1e35b8d9a6", size = 9505972, upload-time = "2026-09-11T19:04:15.788Z" }, + { url = "https://files.pythonhosted.org/packages/d5/00/02398b0a1a62ef73d9af0a957fe1773363eec9a6ca7e433c72ae2790fbce/matplotlib-3.11.2-cp314-cp314-win_arm64.whl", hash = "sha256:01dc8eaaab5a9fce9ff615eca82345728f289e4715b186ee10c6d85272fc26bb", size = 9184534, upload-time = "2026-09-11T19:04:18.691Z" }, + { url = "https://files.pythonhosted.org/packages/ec/c5/1e6ca10baa7e04f113c37856aadb4adc3dbb1c55fc7da6dc830bba1292d7/matplotlib-3.11.2-cp314-cp314t-macosx_10_15_x86_64.whl", hash = "sha256:79a258f58253dfa025af80a9e9bb228d75fced007f0e93ae7423fefbde81a74d", size = 9526415, upload-time = "2026-09-11T19:04:21.39Z" }, + { url = "https://files.pythonhosted.org/packages/e0/93/7561727af07ccc84747953c6a33eee3981cc436d04f465032e707ca0e429/matplotlib-3.11.2-cp314-cp314t-macosx_11_0_arm64.whl", hash = "sha256:c0b83f044ce10a98027b105b3931548719a6e8c7ef986b4362651e0b5367c8dc", size = 9361064, upload-time = "2026-09-11T19:04:24.107Z" }, + { url = "https://files.pythonhosted.org/packages/c1/40/ce270ef2a6794d94a409fe28b8dabedffc07c07f9e5dbb89e0f605bd157d/matplotlib-3.11.2-cp314-cp314t-manylinux_2_26_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:e43b188f0a5b75447bcc197728258166aa64365770ed1caa36595a5e1ca4bbba", size = 10684626, upload-time = "2026-09-11T19:04:27.201Z" }, + { url = "https://files.pythonhosted.org/packages/45/46/396e0307dbf1c1f2d0a3654caa6dbc780dd5379f5f11257369cf8d21d9fe/matplotlib-3.11.2-cp314-cp314t-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:d3304eb5a59442a8867f6920d484591c0fa09ffc29e9260be2feec3351e25869", size = 10960602, upload-time = "2026-09-11T19:04:30.254Z" }, + { url = "https://files.pythonhosted.org/packages/1c/97/4defb84695477340aa88b27c9482cc1ca4de11f36f66b6c8c0ddd685def1/matplotlib-3.11.2-cp314-cp314t-musllinux_1_2_x86_64.whl", hash = "sha256:02329432ae5c6af87cf208ee575b701d698bdf0b1a3bb28cc6d53c36e967e575", size = 10822603, upload-time = "2026-09-11T19:04:32.964Z" }, + { url = "https://files.pythonhosted.org/packages/7d/e7/eab54551c1ec6b72dcd678a078b6c6da8d4c85ec1ff0016e81fe52866304/matplotlib-3.11.2-cp314-cp314t-win_amd64.whl", hash = "sha256:f2ac30cf5eb5dff1b584627ae0b0e1186551a4f69ae3c75073911da497a29170", size = 9551206, upload-time = "2026-09-11T19:04:35.811Z" }, + { url = "https://files.pythonhosted.org/packages/4e/0e/04698032cb4d8fb1e30d8ffec5ded20f6f582b03c8c58b2385c95245f158/matplotlib-3.11.2-cp314-cp314t-win_arm64.whl", hash = "sha256:cf41ecd1b0c0b6f7177ed965a54c2afbe888715c7cf6054dc12d53bc1494002c", size = 9234977, upload-time = "2026-09-11T19:04:38.972Z" }, + { url = "https://files.pythonhosted.org/packages/44/a9/ca827c184b87364378964e36e7c0ae23651d146fdbc4000635205490cf82/matplotlib-3.11.2-cp315-cp315-macosx_10_15_x86_64.whl", hash = "sha256:894a9cbbecbe30ae6787d464df2e8fc7a8d475cfc68f87c3029f7c11152899b3", size = 9478997, upload-time = "2026-09-11T19:04:41.914Z" }, + { url = "https://files.pythonhosted.org/packages/48/60/01aaafc9c3159f3fae9a65bbcddaa4a6d4d72b0aeffdd636d7f6d00a5c8d/matplotlib-3.11.2-cp315-cp315-macosx_11_0_arm64.whl", hash = "sha256:bbf1062991d826ed27e2144f3afa4461afbb8ff56e8f703043e191a8163b1ee9", size = 9307947, upload-time = "2026-09-11T19:04:44.834Z" }, + { url = "https://files.pythonhosted.org/packages/1e/a0/6aa72c7cd7cf4dfac652744d7a3310007cec3ce8e5ab9be607bec7096471/matplotlib-3.11.2-cp315-cp315-manylinux_2_26_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:399fef672f7046ef7d6a57572b2a6f9845f3f5afcff04e7b2df7a363a9f42190", size = 10674370, upload-time = "2026-09-11T19:04:47.745Z" }, + { url = "https://files.pythonhosted.org/packages/a1/97/d7e90225ac145046787a2ee0fa4cdc7e4d5cb6772ea232130b10f8a4095a/matplotlib-3.11.2-cp315-cp315-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:cc82dde2a0d3e3ad472edce04897ad7146b8d8bfd1df8a32992eebb81af18fdc", size = 10954581, upload-time = "2026-09-11T19:04:50.712Z" }, + { url = "https://files.pythonhosted.org/packages/58/18/89cf056a9d2cae01f4b8baf01c7e809d613244fcdfa2a2acf28c1e0d79fa/matplotlib-3.11.2-cp315-cp315-musllinux_1_2_x86_64.whl", hash = "sha256:930efb28f59fda124e39265d177bab302625297bb147d2910de725a2fc2aef54", size = 10812605, upload-time = "2026-09-11T19:04:53.552Z" }, + { url = "https://files.pythonhosted.org/packages/9d/4d/c61b9f4c34af802d4d84e58803e009013d626b5f468bafe1416131863b5b/matplotlib-3.11.2-cp315-cp315-win_amd64.whl", hash = "sha256:3da3bc0cbf7245e7db72cc6d29d12c5abef72cb73059c73b945f14e3545f3eb2", size = 9506068, upload-time = "2026-09-11T19:04:56.519Z" }, + { url = "https://files.pythonhosted.org/packages/03/7b/047f2fa8708d47430d8e7fa45e9fffb0aee5d4e300f978e087be2ee11fbc/matplotlib-3.11.2-cp315-cp315-win_arm64.whl", hash = "sha256:f25446b2981717dca9786bac841cb3fd7efb568e3e3c755dd980481c5cb9228d", size = 9184576, upload-time = "2026-09-11T19:04:59.318Z" }, + { url = "https://files.pythonhosted.org/packages/03/6e/ad9c5989950785a98642667df265a47c6637e924cb69b5dcf68c3299ae7c/matplotlib-3.11.2-cp315-cp315t-macosx_10_15_x86_64.whl", hash = "sha256:fea03cf56568cc1cba08b470be6a0559e71c3a5b688d54b7179bb35ba23d0821", size = 9525988, upload-time = "2026-09-11T19:05:02.304Z" }, + { url = "https://files.pythonhosted.org/packages/a6/c3/25c61701fb655dd412428b15eee2591ce554cdd593e39a045a153373f428/matplotlib-3.11.2-cp315-cp315t-macosx_11_0_arm64.whl", hash = "sha256:254d4ddb2fa8df3b4c689c0c306063aee10521df82cfb438185e499c75fe37c1", size = 9360774, upload-time = "2026-09-11T19:05:05.092Z" }, + { url = "https://files.pythonhosted.org/packages/58/5a/4552878fce8f9d249a5e264c16f5c267c27eac08fb039fa3e207a06feff5/matplotlib-3.11.2-cp315-cp315t-manylinux_2_26_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:e60cf3047a51edecdc4535a9196bbd6a732936b8e9f8184aabf4d16165884aaf", size = 10685892, upload-time = "2026-09-11T19:05:07.686Z" }, + { url = "https://files.pythonhosted.org/packages/af/dd/b3b19ec0124fc4aafd8a4b3bae859c562a7ff0fd8ae3b2099947f78bde19/matplotlib-3.11.2-cp315-cp315t-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:eb3712dc9b464793de0a4e42a7313d50293c751f94bddaf7332a1bc71bccdda9", size = 10964946, upload-time = "2026-09-11T19:05:10.943Z" }, + { url = "https://files.pythonhosted.org/packages/7b/30/687ccce22f66a57fed85942450e88062773bcbf1a42914e70c7148ec71dc/matplotlib-3.11.2-cp315-cp315t-musllinux_1_2_x86_64.whl", hash = "sha256:75b6d88402770e181b5d06a67dda4129c31da7d05004d63a21c310ec78d1b83c", size = 10824960, upload-time = "2026-09-11T19:05:13.9Z" }, + { url = "https://files.pythonhosted.org/packages/d3/34/117fe35a8d05054b4b34d2d29287099927c24c3841d36e06d9fa2f20c1cf/matplotlib-3.11.2-cp315-cp315t-win_amd64.whl", hash = "sha256:a6939df7567114b6bac7f4c5e06c84a67f197c1b2f2e4b234d4eecb3bfec9482", size = 9551204, upload-time = "2026-09-11T19:05:16.48Z" }, + { url = "https://files.pythonhosted.org/packages/63/5a/9f440e7bec8b80d0af2a65d1165b2f4e718982dad785531f62255d9ff19b/matplotlib-3.11.2-cp315-cp315t-win_arm64.whl", hash = "sha256:d480038c83691532ed52ff3147db51fa902fc78cb2d8349993a1cdb684435bff", size = 9233946, upload-time = "2026-09-11T19:05:19.063Z" }, + { url = "https://files.pythonhosted.org/packages/0f/d9/a1218a31fa2d8082eb893548990ad09a2005ebda1b2b89dad88a8308e51a/matplotlib-3.11.2-pp311-pypy311_pp73-macosx_10_15_x86_64.whl", hash = "sha256:854df8d7dfe9fdffcbaa6f39e44a6c24b409cb4d7561fbc09213b157d833f6a6", size = 9457702, upload-time = "2026-09-11T19:05:22.086Z" }, + { url = "https://files.pythonhosted.org/packages/68/d7/a9fde05d87c3c853f233ec20c5b5c3a1c86035e1ed4bfa397cd54d46ec90/matplotlib-3.11.2-pp311-pypy311_pp73-macosx_11_0_arm64.whl", hash = "sha256:eac4b07d4e3743b172451e122ea964f72f152879d4f9adcf3f3d33e518f12ead", size = 9287753, upload-time = "2026-09-11T19:05:24.677Z" }, + { url = "https://files.pythonhosted.org/packages/ec/79/ea2ae4eab84f011c1dc8145864f78e5a63adda8efb95557ef36cc38e6da2/matplotlib-3.11.2-pp311-pypy311_pp73-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:7ef7a53b66780e5d942923724f08577fbc5be1f7322da0f0f3f9dcaa45dd803d", size = 9848259, upload-time = "2026-09-11T19:05:27.455Z" }, +] + +[[package]] +name = "mypy" +version = "2.4.1" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "ast-serialize" }, + { name = "librt", marker = "platform_python_implementation != 'PyPy'" }, + { name = "mypy-extensions" }, + { name = "pathspec" }, + { name = "typing-extensions" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/4a/3c/78ecece71ee4b5445389a4ccd11627cfb9c69e4de554d11060171f376305/mypy-2.4.1.tar.gz", hash = "sha256:63b590965691e540a335fa74dadf31a7069142f6b72e234689bbd244edf55623", size = 4066728, upload-time = "2026-10-28T19:23:13.286Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/4b/8b/2bc7f489528db504221ad7eff6b3f83979ee57d066103c72a85d09046a30/mypy-2.4.1-cp311-cp311-macosx_11_0_arm64.whl", hash = "sha256:79a1e79199e980fb41f219cb3d4746d25c64b0b623840d3ed57df1cdc13a6502", size = 14321495, upload-time = "2026-10-28T19:21:16.612Z" }, + { url = "https://files.pythonhosted.org/packages/18/9a/b2f037b4982dce33645296e4394b031ade60f697f468761fa3ea8e2f8ece/mypy-2.4.1-cp311-cp311-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:4e18d0da3898b3bcd16192f15a206e7b8b8dd0f4a0b3e51aa5bd5c2afe749e2c", size = 14467468, upload-time = "2026-10-28T19:22:10.629Z" }, + { url = "https://files.pythonhosted.org/packages/c4/5a/379f9bfa18bf3f8d43668fbd9ae4f2922dd027a559c461ca9d41cde1d32e/mypy-2.4.1-cp311-cp311-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:b69a15fc2e75858d6b15e91ed3a0dcff1a533265183b64734138791fe0f49aa5", size = 15421496, upload-time = "2026-10-28T19:22:48.005Z" }, + { url = "https://files.pythonhosted.org/packages/83/50/3bde18a2824ec296432cebe01752ccd0ab8a2ad784abe4390040d019b3bf/mypy-2.4.1-cp311-cp311-musllinux_1_2_x86_64.whl", hash = "sha256:70971cc0bcb1d3f03d2eb68fa430a6481ccc03558dc001d9534d7e5fa274c392", size = 15670425, upload-time = "2026-10-28T19:21:57.026Z" }, + { url = "https://files.pythonhosted.org/packages/6a/36/8b722b88c325d317354de7d10d8fb9d4dc0f958433e873a91fc4ed21a06e/mypy-2.4.1-cp311-cp311-win_amd64.whl", hash = "sha256:6f7225951a277dd0d6add900731d487174b7d24ac133c09d670ed63b4d8cb90d", size = 11300972, upload-time = "2026-10-28T19:21:48.724Z" }, + { url = "https://files.pythonhosted.org/packages/c2/d6/fb2bab1245658d68af8abfe776362a780c621fe0f364f2c7b53039a945ee/mypy-2.4.1-cp311-cp311-win_arm64.whl", hash = "sha256:be4c03f33c55500191df2ed4a9b5e7edbdd1fff067fe361a5a9b1b5d3262da6e", size = 10873349, upload-time = "2026-10-28T19:21:32.104Z" }, + { url = "https://files.pythonhosted.org/packages/c0/1a/0be9dc6af5e0e04128f18a5e36b4ca44bfb0085857d37aa6a7de1ac2b0cb/mypy-2.4.1-cp312-cp312-macosx_11_0_arm64.whl", hash = "sha256:d63737426a048a27705a4eedb828669f4bc0873612a1b46b0b9fe892b2ebaaaf", size = 14381321, upload-time = "2026-10-28T19:21:54.223Z" }, + { url = "https://files.pythonhosted.org/packages/0d/a6/7907e36711a8a8c703f961f076fab61793383694140bd1fd87589f3f68d4/mypy-2.4.1-cp312-cp312-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:7344869cad05f073b8cae2826774fa0161e3eb2f5c685b76813574008d77d750", size = 14466373, upload-time = "2026-10-28T19:22:25.211Z" }, + { url = "https://files.pythonhosted.org/packages/5d/b8/7313212b29de570c25c9859d7c8e4ef2251a6d34338a1e20031d94297721/mypy-2.4.1-cp312-cp312-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:a20ff6cd2af7a670c64f089bfa5fbcebe8ff3bf896c5e6ac7c9862c7960e4494", size = 15553666, upload-time = "2026-10-28T19:22:36.511Z" }, + { url = "https://files.pythonhosted.org/packages/23/8f/92966b6082ff49da9142e1e527eb9af4359a9a61632a1cfc313da96b37fe/mypy-2.4.1-cp312-cp312-musllinux_1_2_x86_64.whl", hash = "sha256:9ed1195502cca32be5ef756b1693477f4991db2f2e34eaa90cc4ad6bea4f9531", size = 15802641, upload-time = "2026-10-28T19:21:13.421Z" }, + { url = "https://files.pythonhosted.org/packages/b6/b4/874f6fd54b005020ad63e2cf7a59b178d16a771d478849b29b842cb0cf14/mypy-2.4.1-cp312-cp312-win_amd64.whl", hash = "sha256:4c194c5d878130fca1ae662dfb1feb6274afbab0b40032c80147377e0ea803d5", size = 11383067, upload-time = "2026-10-28T19:22:03.516Z" }, + { url = "https://files.pythonhosted.org/packages/b0/41/5ae8e6528c2faa96e4a921a999d2c7ac7c565bee184b4282bf62ffa76b5d/mypy-2.4.1-cp312-cp312-win_arm64.whl", hash = "sha256:71eda06e5bae51976ce869fda429159c9cec678bd02f25bb197f0265c8f991be", size = 10946882, upload-time = "2026-10-28T19:21:39.51Z" }, + { url = "https://files.pythonhosted.org/packages/6a/65/cd19bc06b67d2df4828220245fd98569ba40fac00c60ba48bef1817d8ac3/mypy-2.4.1-cp313-cp313-macosx_11_0_arm64.whl", hash = "sha256:3f1613bfe6726169a5680ec3dad5fe1f6d96285c12aded51ad432eadedd39d45", size = 14214179, upload-time = "2026-10-28T19:22:27.49Z" }, + { url = "https://files.pythonhosted.org/packages/c9/db/4d54ca4f49a0113d0c956980272aefdc6eec5458f7a72b350785b9ef2249/mypy-2.4.1-cp313-cp313-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:680ba5d9f0e3fe0c746b3d14df15f2a718e4bc308a1145616a1ea691ce7c354b", size = 14376217, upload-time = "2026-10-28T19:21:08.019Z" }, + { url = "https://files.pythonhosted.org/packages/a2/45/089b4724b85f0f408941be5298f817ee89181c1ce031a73d60ebb150c2c7/mypy-2.4.1-cp313-cp313-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:1fb70621535c22fd4b1a0450d37ed1d6c5eb39656a1cebfb189084c12de54ba7", size = 15467623, upload-time = "2026-10-28T19:21:21.555Z" }, + { url = "https://files.pythonhosted.org/packages/27/4c/15edbc64b0b1412bfe94c74192b3b56554ab642b85a31bb06cef6313101f/mypy-2.4.1-cp313-cp313-musllinux_1_2_x86_64.whl", hash = "sha256:4af5ad558033bf71a44a1be149ed61633522824f84c1b6bc2445d3fe2d2a5296", size = 15729867, upload-time = "2026-10-28T19:21:37.557Z" }, + { url = "https://files.pythonhosted.org/packages/76/5b/7d160fb685994b65b0ca86d352f3afb5424f96a9e6914719f7b861fb6701/mypy-2.4.1-cp313-cp313-win_amd64.whl", hash = "sha256:acdd82cb3a2eb10720e87706a128d75785d005a8c47e36a4973221524f41fc56", size = 11390802, upload-time = "2026-10-28T19:22:18.081Z" }, + { url = "https://files.pythonhosted.org/packages/9a/ea/453d27f8d531f57c5f14aa73094eaa48e3fb77f680bdede2045cb64fa58b/mypy-2.4.1-cp313-cp313-win_arm64.whl", hash = "sha256:133886ef1a7a8cf673df55d865e41580c6c46aa9b8e31e1325dfe315c038f745", size = 10950427, upload-time = "2026-10-28T19:22:59.003Z" }, + { url = "https://files.pythonhosted.org/packages/7d/75/282d43d6fea7be984ac63e75ffd8eee146ba2f3b077bb1ca5bf3f973e0c4/mypy-2.4.1-cp314-cp314-macosx_11_0_arm64.whl", hash = "sha256:c45485061ba08f700acc12cfc8b177e6997e6bc353605babfc0abb081e079456", size = 14233251, upload-time = "2026-10-28T19:22:44.977Z" }, + { url = "https://files.pythonhosted.org/packages/ac/a8/9021263cc7e761451c7d8131bdfab3c0a5c2d67aa2d694670019e1ab86a7/mypy-2.4.1-cp314-cp314-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:727890a32f1858c09549eb55a8b234fe990d460450b28f03e8213ef925f1247c", size = 14394953, upload-time = "2026-10-28T19:21:29.492Z" }, + { url = "https://files.pythonhosted.org/packages/16/10/4192a5cea09ebbc41c534f57bd0d8f0fb42a8e810632cda6e37f072c59fd/mypy-2.4.1-cp314-cp314-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:beb254fec368c4e7f7f82a5f310c6470b893ec88a90b66426da971c6826b0453", size = 15442824, upload-time = "2026-10-28T19:22:42.068Z" }, + { url = "https://files.pythonhosted.org/packages/99/e6/d3b0a7474aa4f696594a47a218b6796d9c4346fc81ed288904278cddd9bd/mypy-2.4.1-cp314-cp314-musllinux_1_2_x86_64.whl", hash = "sha256:ec416d5d2ae689e681f5727e328723f18d8b5dc8c903a146b7851dc5727aad8a", size = 15717227, upload-time = "2026-10-28T19:22:54.026Z" }, + { url = "https://files.pythonhosted.org/packages/48/6a/2ec530dacd4f77ca50d0fa05b854098f2d75e5d18c587d2612b44f10d68c/mypy-2.4.1-cp314-cp314-pyemscripten_2026_0_wasm32.whl", hash = "sha256:0e95ad0bec03ced9a73b42514208abd24c789e67f0e1eba476e276dd7a2dea3a", size = 7898637, upload-time = "2026-10-28T19:21:41.709Z" }, + { url = "https://files.pythonhosted.org/packages/cc/56/de9d7bad12e3868d5f1f4279aed20b1b568b2c5e166f2d3a05b211dca0b1/mypy-2.4.1-cp314-cp314-win_amd64.whl", hash = "sha256:51f742d4154824837ca6e92aa81f00eec222700a3c337bd258e5562a3880d75a", size = 11576656, upload-time = "2026-10-28T19:22:13.232Z" }, + { url = "https://files.pythonhosted.org/packages/b8/12/4fa9883b9b59e5c96ff62845ad019dc47786031c45d67eaccf452cffffd8/mypy-2.4.1-cp314-cp314-win_arm64.whl", hash = "sha256:f44dd4c5ab22df21206dd926c98381ebdbf5bfd696df37dd38ded2d23143fcb6", size = 11174403, upload-time = "2026-10-28T19:23:01.346Z" }, + { url = "https://files.pythonhosted.org/packages/3f/f2/827e356fc4f74c60b8da42663336e035b49d948a3c11224b0fc7543086b1/mypy-2.4.1-cp314-cp314t-macosx_11_0_arm64.whl", hash = "sha256:2875280687e64740e6ddcef43bd4c5852ad45ff34549740f6aea3f0373870b93", size = 15713427, upload-time = "2026-10-28T19:21:26.757Z" }, + { url = "https://files.pythonhosted.org/packages/a2/fa/e58a4fcae1d238189eac758036c0921992a86e3e6209190efd96d61fc5d4/mypy-2.4.1-cp314-cp314t-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:ea0714d2bccaf6d010c970e1d3ba2c9c2add79d2030aad2ef03167d788948276", size = 15938481, upload-time = "2026-10-28T19:22:34.371Z" }, + { url = "https://files.pythonhosted.org/packages/a1/54/a20f881801b88900923f2be1541a4b438ad0f0883e03836641fdc3d7adf4/mypy-2.4.1-cp314-cp314t-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:63c6ff0e7ea546943001a91639c9f0c64b60d239268fa5d29e4a6c87f82c2373", size = 17243006, upload-time = "2026-10-28T19:22:39.425Z" }, + { url = "https://files.pythonhosted.org/packages/de/1c/ff84a6133a6e89c390bff6b25d9bad6598e7891b205f4b30b496906d636d/mypy-2.4.1-cp314-cp314t-musllinux_1_2_x86_64.whl", hash = "sha256:a5b5da107dc2c93fb0a22f7a45dbaea5dc131afc7056b5468d78e819f9a138e6", size = 17557728, upload-time = "2026-10-28T19:21:43.607Z" }, + { url = "https://files.pythonhosted.org/packages/15/4a/effa47d39b8fadf679c08e7c78559e523f9769a42a75662d2d15b2b9cdd6/mypy-2.4.1-cp314-cp314t-win_amd64.whl", hash = "sha256:c3a4d0c47053cc90035dd3a2c321112b593ad42935d0cbfd747288e1564fa8ca", size = 12370826, upload-time = "2026-10-28T19:22:29.779Z" }, + { url = "https://files.pythonhosted.org/packages/46/9f/b214ececb1a97fbb61ab3dfd48ae3cffd92ae2c148af4c9dc94c6c56772b/mypy-2.4.1-cp314-cp314t-win_arm64.whl", hash = "sha256:0c5bf267645351887b72903fd2f5fbddcf85234e0e6c4ad94e2fcfb4f0c3f971", size = 11851083, upload-time = "2026-10-28T19:23:11.288Z" }, + { url = "https://files.pythonhosted.org/packages/57/5b/e44a2edb0b25a9f1429de770e1d0238af23302671dd40c77ac5efeb0c559/mypy-2.4.1-cp315-cp315-macosx_11_0_arm64.whl", hash = "sha256:db599d03e7d60d15f2e0a11665c18bc52a88699fcab990fdf8cecd5be8ac801c", size = 14243144, upload-time = "2026-10-28T19:21:10.975Z" }, + { url = "https://files.pythonhosted.org/packages/59/d4/5c99dc3a28e11d2f0fcbb02df63d2d3e574dedc0f9f69d50901df305a7a6/mypy-2.4.1-cp315-cp315-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:c6c7210e40dd423af7d65cbe428b5906d11dc60659337bdae4e5fdf02ce32ed7", size = 14409316, upload-time = "2026-10-28T19:22:15.515Z" }, + { url = "https://files.pythonhosted.org/packages/c3/c9/3108b38f4c30eabd80194f5d9771a87e27196c5e0ddb5bf4b99c6ed1e97e/mypy-2.4.1-cp315-cp315-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:a2d4a51db92059eecc2c33b2a5194b28585ebfd917292c015fe87a77dbff6cd2", size = 15445573, upload-time = "2026-10-28T19:22:05.635Z" }, + { url = "https://files.pythonhosted.org/packages/dd/12/84a53eceaf11c46dfd7931fb6b8902d01fb33787d3a991e81c461458150b/mypy-2.4.1-cp315-cp315-musllinux_1_2_x86_64.whl", hash = "sha256:01b7718552d017f88d1151f3cc9d6556d778a90bd58060a2c3d87d070f9099fb", size = 15732989, upload-time = "2026-10-28T19:23:06.512Z" }, + { url = "https://files.pythonhosted.org/packages/65/ba/bf017dc2abdf83b2f74e29c43834b7af67dc1f1b20202f02dae60b4781a0/mypy-2.4.1-cp315-cp315-pyemscripten_2026_5_wasm32.whl", hash = "sha256:3467b1d13000812ab85dc2a3834b9379ec77798ca1bcabe434e23fb21fead692", size = 7898932, upload-time = "2026-10-28T19:21:59.366Z" }, + { url = "https://files.pythonhosted.org/packages/76/05/0c22b1f0390120a8592d31209cb6b9f932b6a0b426fe643e550de15139e3/mypy-2.4.1-cp315-cp315-win_amd64.whl", hash = "sha256:080cf64a60b90a73435542feecf4171904cc5710658165b40c6869115d4aa100", size = 11577577, upload-time = "2026-10-28T19:22:01.302Z" }, + { url = "https://files.pythonhosted.org/packages/73/fb/ac01e173629697ddac2f11e51efd10347d673f9a7311b03f4aba710ebccd/mypy-2.4.1-cp315-cp315-win_arm64.whl", hash = "sha256:f9118a3f50de264b614861bb11814833a742138ac3c8e2e49a860b6977fb2704", size = 11173540, upload-time = "2026-10-28T19:21:46.444Z" }, + { url = "https://files.pythonhosted.org/packages/5b/e4/b9b9c8eb09e70927ee65b1456cfed35710b6e4ff0b07f364ceaf2202307c/mypy-2.4.1-cp315-cp315t-macosx_11_0_arm64.whl", hash = "sha256:9d7dafd56179d296651df9e1844b91c9517c379398f017a2a435d077a28943c1", size = 15706623, upload-time = "2026-10-28T19:22:20.528Z" }, + { url = "https://files.pythonhosted.org/packages/26/74/1ceb1ca884add0565e618d013dd24333039100dc0fcc42661561f2e1cae5/mypy-2.4.1-cp315-cp315t-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:d923cec43316f0e6736b8298fd00abeaaf348a091c20e7296b3aebceecb63eca", size = 15944095, upload-time = "2026-10-28T19:21:04.779Z" }, + { url = "https://files.pythonhosted.org/packages/2b/68/27bcf9b4efa22733b6050485ae37845b94c9b8851abe4fc60632d905c32d/mypy-2.4.1-cp315-cp315t-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:d6e081b466e88d4343adffe11aff0f02df34b5b9070b4b700053a0b7294c0be1", size = 17223065, upload-time = "2026-10-28T19:23:03.724Z" }, + { url = "https://files.pythonhosted.org/packages/1a/a3/ed5724b317bf585c29f3c8be2792e97d45925d65a9c8af40ceeb267072e6/mypy-2.4.1-cp315-cp315t-musllinux_1_2_x86_64.whl", hash = "sha256:70196470ac300f444a302a738541101893c2c57804078b33690286a7899149e9", size = 17560589, upload-time = "2026-10-28T19:21:34.539Z" }, + { url = "https://files.pythonhosted.org/packages/42/2f/005185e6ab4752fd6cc2d1a6d1421fe21979770afe2490c00c555f34a913/mypy-2.4.1-cp315-cp315t-win_amd64.whl", hash = "sha256:de95648e0a6d46fa9929b956ed2c87c5dc9933d4d59945e6390e9d753c74e7c1", size = 12353324, upload-time = "2026-10-28T19:22:51.481Z" }, + { url = "https://files.pythonhosted.org/packages/a1/d6/20420cc9c780087139662534d128c2f3274361f8bb7e9cf0d5495a846ed5/mypy-2.4.1-cp315-cp315t-win_arm64.whl", hash = "sha256:ef8d5be90032261197ac388d945d8ff4f476c95880798a1911760af64ddfe71f", size = 11845599, upload-time = "2026-10-28T19:22:23.136Z" }, + { url = "https://files.pythonhosted.org/packages/21/5c/6e8010f8f2c4ad817364592d4b3883633417269b913fc9d5174fdeb7e565/mypy-2.4.1-py3-none-any.whl", hash = "sha256:1784648d6ec8c2b5465e3f26241b845a0b3cb4e56c993c64abe16989e7698999", size = 2801286, upload-time = "2026-10-28T19:21:19.516Z" }, +] + +[[package]] +name = "mypy-extensions" +version = "1.1.0" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/a2/6e/371856a3fb9d31ca8dac321cda606860fa4548858c0cc45d9d1d4ca2628b/mypy_extensions-1.1.0.tar.gz", hash = "sha256:52e68efc3284861e772bbcd66823fde5ae21fd2fdb51c62a211403730b916558", size = 6343, upload-time = "2025-04-22T14:54:24.164Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/79/7b/2c79738432f5c924bef5071f933bcc9efd0473bac3b4aa584a6f7c1c8df8/mypy_extensions-1.1.0-py3-none-any.whl", hash = "sha256:1be4cccdb0f2482337c4743e60421de3a356cd97508abadd57d47403e94f5505", size = 4963, upload-time = "2025-04-22T14:54:22.983Z" }, +] + +[[package]] +name = "narwhals" +version = "2.28.0" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/67/43/e11d75a7fb635c6f63dd6e176cb3e050ec2fee35f2f2267b332a6ad351ec/narwhals-2.28.0.tar.gz", hash = "sha256:2d9860c80d6610820826e94c6e42ce4b1d92d89cdd479586e652a8d211dab345", size = 754977, upload-time = "2026-10-27T12:19:33.777Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/cf/82/8ef9c7e0f50493307a49c09cfa7f00c5a778b6d1921f7b2dabe1d3a5809e/narwhals-2.28.0-py3-none-any.whl", hash = "sha256:1f69d6385ec30e2fa6e1fdeb37c562ecabd0f713f6f3e3b58a668c5c9df9567f", size = 490591, upload-time = "2026-10-27T12:19:31.649Z" }, +] + +[[package]] +name = "numpy" +version = "2.4.6" +source = { registry = "https://pypi.org/simple" } +resolution-markers = [ + "python_full_version < '3.12'", +] +sdist = { url = "https://files.pythonhosted.org/packages/d0/ad/fed0499ce6a338d2a03ebae59cd15093910c8875328855781952abf6c2fe/numpy-2.4.6.tar.gz", hash = "sha256:f3a3570c4a2a16746ac2c31a7c7c7b0c186b95ce902e33db6f28094ed7387dda", size = 20735807, upload-time = "2026-05-18T23:37:14.07Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/b3/49/ec46835a70be8fa6446c495126ac84fdb28cb2558e1620ffb87a10c8b64c/numpy-2.4.6-cp311-cp311-macosx_10_9_x86_64.whl", hash = "sha256:0280e0356c0829a18d9de1cb7eee50ec22ca639878d7240307ca0943d73cd2c4", size = 16969194, upload-time = "2026-05-18T23:33:13.503Z" }, + { url = "https://files.pythonhosted.org/packages/0e/0d/f5957185c0ee2f3e12f78715aa9e3b353fd83633316c8532b38faa37e3f6/numpy-2.4.6-cp311-cp311-macosx_11_0_arm64.whl", hash = "sha256:110f8b71aacb688ec69062bb7f6938a0f8acb01b7c1c4beb453c65b6d234584d", size = 14964111, upload-time = "2026-05-18T23:33:17.795Z" }, + { url = "https://files.pythonhosted.org/packages/ad/40/40a40ee0ddf7ceb782c49af278894b686e586d65d8c1889c8b5da01a3d7d/numpy-2.4.6-cp311-cp311-macosx_14_0_arm64.whl", hash = "sha256:4cfe66903cc32a9921a6733d96b19bb6abf310397581bbad89c228f5abaf0ee8", size = 5469159, upload-time = "2026-05-18T23:33:20.654Z" }, + { url = "https://files.pythonhosted.org/packages/63/13/f9a8046535cb21deae82f8d03de9617e08882d274fad2539630761888228/numpy-2.4.6-cp311-cp311-macosx_14_0_x86_64.whl", hash = "sha256:8155154c7c691289fe18f510b5d4657c68c67989f293f0535a91360392ff6538", size = 6798936, upload-time = "2026-05-18T23:33:22.987Z" }, + { url = "https://files.pythonhosted.org/packages/33/a8/6fa8c1a345a8c85dbb21932c447bee07c30a2c2a3f31e369c0a84b300147/numpy-2.4.6-cp311-cp311-manylinux_2_27_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:0ab0a9c4ffb1a6d95ef519fe4247dba8eb6b18ad93999f76b7f657039acabd47", size = 15966692, upload-time = "2026-05-18T23:33:26.62Z" }, + { url = "https://files.pythonhosted.org/packages/02/03/74fe2a4cb3817d94d86402f2506554130a2f01414e299b5a843e5a8a957f/numpy-2.4.6-cp311-cp311-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:89cd468399cfd2504718f0ba50e410dca55a170b61a02ad92bb18c8a65186e93", size = 16918164, upload-time = "2026-05-18T23:33:29.955Z" }, + { url = "https://files.pythonhosted.org/packages/c5/80/3615be3313f7e7696609bc194b9f0101da809df79e859bdb84e0cd043f46/numpy-2.4.6-cp311-cp311-musllinux_1_2_aarch64.whl", hash = "sha256:c2d37ab77531417474168eb79d6d80b14f821a966818505d03013d0833edb7a8", size = 17322877, upload-time = "2026-05-18T23:33:34.724Z" }, + { url = "https://files.pythonhosted.org/packages/ca/ac/a691e0fe2675e370d0e08ff905adc49a1c8830e8cae03efe4477e92cd55d/numpy-2.4.6-cp311-cp311-musllinux_1_2_x86_64.whl", hash = "sha256:f407cb6b8e9d6d8c626bc73c945db1706035af8fd632295547bf1c9e46d092d6", size = 18651487, upload-time = "2026-05-18T23:33:38.217Z" }, + { url = "https://files.pythonhosted.org/packages/15/a7/9bc1cd626d7bf6869bfedf27b91b6ab5dd607758bf8e959d6fa80c6a59cb/numpy-2.4.6-cp311-cp311-win32.whl", hash = "sha256:ddea102b48f9e339f3948bf22040944184627a30fdf7f858667673b9c5f033c8", size = 6233945, upload-time = "2026-05-18T23:33:41.331Z" }, + { url = "https://files.pythonhosted.org/packages/c5/31/7fc6239c12bce7e931463251cca4426c465e1876ba3cc785402ef4dd8f4e/numpy-2.4.6-cp311-cp311-win_amd64.whl", hash = "sha256:1e254a00cdf42b1e4d5b3d68d33af63268d41340d8885df2ab6470f2e1500147", size = 12608406, upload-time = "2026-05-18T23:33:44.131Z" }, + { url = "https://files.pythonhosted.org/packages/27/83/140f85a466595a16382996a1bf06b2b54bcd597488921b0c9daaeeda72af/numpy-2.4.6-cp311-cp311-win_arm64.whl", hash = "sha256:ed9749eef4cbd126da3dc1d6bcb3a57f5eb7ac6a6484146bdbf743f552dfc577", size = 10479528, upload-time = "2026-05-18T23:33:50.725Z" }, + { url = "https://files.pythonhosted.org/packages/95/2a/3d7b5ac8aac24feaf9ad7ed58f45b0bbc06d37e4338ae84c9f2298b570f9/numpy-2.4.6-cp312-cp312-macosx_10_13_x86_64.whl", hash = "sha256:001fbb8e08d942dd57599e781f2472269ee7f2755fae407b4f67b2f0b17da3f1", size = 16689119, upload-time = "2026-05-18T23:33:54.065Z" }, + { url = "https://files.pythonhosted.org/packages/ea/12/92c4c131527599e8288d6918e888d88726f84d805d784b771f32408aeaef/numpy-2.4.6-cp312-cp312-macosx_11_0_arm64.whl", hash = "sha256:ebfb099f8dcf083deef3ac1ca4c1503f387cf76296fcb3816b66f5ecb5f54fdb", size = 14699246, upload-time = "2026-05-18T23:33:57.621Z" }, + { url = "https://files.pythonhosted.org/packages/ad/fe/c0a6b7b2ca128a8fb228575147073b660656734b8ebe4d76c8fd748dcc79/numpy-2.4.6-cp312-cp312-macosx_14_0_arm64.whl", hash = "sha256:3213d622a0283a39a93d188f3cf72b26862df52fbb4ca3697f51705016523d41", size = 5204410, upload-time = "2026-05-18T23:34:00.302Z" }, + { url = "https://files.pythonhosted.org/packages/f3/d4/9770d14ba719432bb90a421bfd443872ed0f70f7264b64bec12ea363d5fd/numpy-2.4.6-cp312-cp312-macosx_14_0_x86_64.whl", hash = "sha256:357cc07a6d7b0b182ff02249616a03742827ebb1277546b5c7cd7f7620a45698", size = 6551240, upload-time = "2026-05-18T23:34:02.852Z" }, + { url = "https://files.pythonhosted.org/packages/c9/c6/50a46a6205feba2343f1d6d17438107c5dc491ed1c736e6ea68689fd906b/numpy-2.4.6-cp312-cp312-manylinux_2_27_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:5f9fb9157b4ce2971008323afe46053787b526ef624fea915b261468a8421a0f", size = 15671012, upload-time = "2026-05-18T23:34:05.485Z" }, + { url = "https://files.pythonhosted.org/packages/99/60/14115e6364fa676c5397c2ad3004e527e9aa487abf5d0706ec81bbd08529/numpy-2.4.6-cp312-cp312-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:90f9849678c75fe7afa2d348ac842c168b0a4d3d61919687216dfc547976d853", size = 16645538, upload-time = "2026-05-18T23:34:09.265Z" }, + { url = "https://files.pythonhosted.org/packages/ae/c5/693cbe59e57db94d2231fa519ca3978dc9e19da5a8f088588f5c6e947ff2/numpy-2.4.6-cp312-cp312-musllinux_1_2_aarch64.whl", hash = "sha256:c1a2af6c6ef86344a6b0db6b97834208bf598db514f2b155042439b62605601a", size = 17020706, upload-time = "2026-05-18T23:34:13.053Z" }, + { url = "https://files.pythonhosted.org/packages/ef/fc/85b7c4eff9b4966ade25c2273cf7e7012e92366c032058653934b37de044/numpy-2.4.6-cp312-cp312-musllinux_1_2_x86_64.whl", hash = "sha256:e5805d5a22fd19c8ccff10a9561f9df94436b0545619ea579db2d3c35294bce2", size = 18368541, upload-time = "2026-05-18T23:34:17.024Z" }, + { url = "https://files.pythonhosted.org/packages/f6/81/e1b27545deedce7f4a0b348618c6b62d74e36a4dc9ccd42f3eb2f85eee32/numpy-2.4.6-cp312-cp312-win32.whl", hash = "sha256:e3eeb0aabd6bd5ce64faae67e9935203a6991b4bc2a485a767fbafb2c5125f45", size = 5962825, upload-time = "2026-05-18T23:34:20.3Z" }, + { url = "https://files.pythonhosted.org/packages/ab/ca/feab00bd44aa5fe1ad2c18f08b4d3bb92e26484b0b1d1443897809ed528c/numpy-2.4.6-cp312-cp312-win_amd64.whl", hash = "sha256:d8e8286dd7cea7895157318d1b91cdacac64c479f3cbc8dce548331728484751", size = 12321687, upload-time = "2026-05-18T23:34:23.095Z" }, + { url = "https://files.pythonhosted.org/packages/63/cf/5a6d34850a39d1093558564f77ee8e8e0bee5061151b8f05a55711001ec7/numpy-2.4.6-cp312-cp312-win_arm64.whl", hash = "sha256:4081eb135ac24158bd51cdfbef16f1c64df7063b1143f24731387137c092bec8", size = 10221482, upload-time = "2026-05-18T23:34:25.876Z" }, + { url = "https://files.pythonhosted.org/packages/fb/82/bdab26d7438c6791ca31b7c024ca37c1eab8b726ba236129005cd4a06e45/numpy-2.4.6-cp313-cp313-macosx_10_13_x86_64.whl", hash = "sha256:511dbaf848decaaaf4b4ca48032619fb3138710c4bf7da7617765edad1ef96b0", size = 16684648, upload-time = "2026-05-18T23:34:29.41Z" }, + { url = "https://files.pythonhosted.org/packages/1b/30/a80189bcc7f5e4258b3fbc3968d909d1756f54d023299ecc39ad6fdb9ef8/numpy-2.4.6-cp313-cp313-macosx_11_0_arm64.whl", hash = "sha256:bf162abab1c1a736333192707cef898e735a5ca00f38f27eeedf44b39d9e85eb", size = 14693902, upload-time = "2026-05-18T23:34:33.013Z" }, + { url = "https://files.pythonhosted.org/packages/97/12/70b5d0d7c15e1ebb8a6a84a8caa1d19e181d84fb58bb6d70aca29099dec1/numpy-2.4.6-cp313-cp313-macosx_14_0_arm64.whl", hash = "sha256:043191bfa8eab18c776647b62723ac9dddece59743b13f49b2016094129c2b3f", size = 5198992, upload-time = "2026-05-18T23:34:36.132Z" }, + { url = "https://files.pythonhosted.org/packages/ba/8c/ebd2a8f8a83541f8d38cc5667e8c2b69cecfd30da6e45693e8158857d44b/numpy-2.4.6-cp313-cp313-macosx_14_0_x86_64.whl", hash = "sha256:6180d8b35af935aed8ece3a85e0a43f87393ae0ac87c8d2c8bd2c993f7270ef3", size = 6546944, upload-time = "2026-05-18T23:34:38.484Z" }, + { url = "https://files.pythonhosted.org/packages/bb/c5/7b863a97a91671a0338f4253bd3b5a3d3852f0692dae91711c9f4a10e787/numpy-2.4.6-cp313-cp313-manylinux_2_27_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:72fbe16c6fac95aedf5937fa873445cec2110be35d8a4e9433d7501fd98dae6b", size = 15669392, upload-time = "2026-05-18T23:34:41.257Z" }, + { url = "https://files.pythonhosted.org/packages/a5/9d/3584b9984ca4c047aea75214ce1a4c4c73d849bd71b604264b7f5653f8a8/numpy-2.4.6-cp313-cp313-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:a7830bab239b79cda9c08c2da014761cafb48da6150e1da17ac06283f43b6089", size = 16633220, upload-time = "2026-05-18T23:34:45.075Z" }, + { url = "https://files.pythonhosted.org/packages/05/ae/7c67fba23bd98caec7c99261f3a16072ade14813486b0282cb29846de832/numpy-2.4.6-cp313-cp313-musllinux_1_2_aarch64.whl", hash = "sha256:ef4aea96ce4d3b074422cb4f2f64e216bf9e213004bb58ecfdf50ea02ea8eb9a", size = 17020800, upload-time = "2026-05-18T23:34:49.065Z" }, + { url = "https://files.pythonhosted.org/packages/d9/5d/3b6725cb31d983c5e66916f5d36f6d7e5521129e4c4404d64f918292a5b6/numpy-2.4.6-cp313-cp313-musllinux_1_2_x86_64.whl", hash = "sha256:dfa20cc6ca228e6b155b11da03825975ce66aea520985dbbddf0f2a5a495c605", size = 18357600, upload-time = "2026-05-18T23:34:52.709Z" }, + { url = "https://files.pythonhosted.org/packages/f7/da/2ccc6c2fe8898dee01d90c75c5f5f914a23daf99e3e0f59516a08760c8b5/numpy-2.4.6-cp313-cp313-win32.whl", hash = "sha256:56b39e5e0622a09a25bf5baf62f4bcf0cb8a41ae6e2819cf49bbc5a74c083f91", size = 5961134, upload-time = "2026-05-18T23:34:55.618Z" }, + { url = "https://files.pythonhosted.org/packages/b5/cd/9cc4dc876fb065d5c220aae4d5e14826b2715331bb7618ce1fb07a679d99/numpy-2.4.6-cp313-cp313-win_amd64.whl", hash = "sha256:c4fc99836233ea196540b17ab0983aff60ed07941751930f5f4d05bc3b3b7359", size = 12318598, upload-time = "2026-05-18T23:34:58.928Z" }, + { url = "https://files.pythonhosted.org/packages/39/1e/c0bcba1f8694116485fe28fd1be698c278fcda4141c5b0e53a2aed8b12a8/numpy-2.4.6-cp313-cp313-win_arm64.whl", hash = "sha256:a7c711e21628b52034bb5ab8d1bce291f752fcc5e92accc615778acee1ff4778", size = 10222272, upload-time = "2026-05-18T23:35:02.167Z" }, + { url = "https://files.pythonhosted.org/packages/63/6d/cc5619247c8f4204e507f5883528372e4ac4bb189e579fb859a12e480b1f/numpy-2.4.6-cp313-cp313t-macosx_11_0_arm64.whl", hash = "sha256:112b06a867b235ef466ed3508ddf0238050df9c727cafb5301ac385b899189a1", size = 14821197, upload-time = "2026-05-18T23:35:05.468Z" }, + { url = "https://files.pythonhosted.org/packages/00/58/f1c39161c87d9e9bed660f1ed4bafc0e403d5ec9650b6dd77aead07d489b/numpy-2.4.6-cp313-cp313t-macosx_14_0_arm64.whl", hash = "sha256:eaf7fa2de5c0be8ae6ff8e9bea2ccd725e980541244521d8d4b5f3354a27babe", size = 5326287, upload-time = "2026-05-18T23:35:08.693Z" }, + { url = "https://files.pythonhosted.org/packages/af/57/3917ab0fd97f271a8694513581b8a36c655f111c446852c302f04ccdb6fc/numpy-2.4.6-cp313-cp313t-macosx_14_0_x86_64.whl", hash = "sha256:7265a2f3d436e54ef9f2b52b5c937e6be778781bd97a590319d7348f1c1ca997", size = 6646763, upload-time = "2026-05-18T23:35:11.459Z" }, + { url = "https://files.pythonhosted.org/packages/eb/0f/037e64c494b67581ae18193d770adef354c41f3f2c8ebf865602d949bf8f/numpy-2.4.6-cp313-cp313t-manylinux_2_27_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:f74a575920ab21fe304421a3fc28793d82e299cae9eccb37084e9fc7f3617c20", size = 15728070, upload-time = "2026-05-18T23:35:14.79Z" }, + { url = "https://files.pythonhosted.org/packages/21/a6/5d2bae9c9542eb4df16dc9c46dc79c186e9bad53805dfa5399a6023c6db0/numpy-2.4.6-cp313-cp313t-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:ede83e07a75dd06bc501566c1eca2afc0d61677c1472ac9ad93fdee6e638a48d", size = 16681752, upload-time = "2026-05-18T23:35:18.836Z" }, + { url = "https://files.pythonhosted.org/packages/92/14/23d1dfb410ae362cd59ce53e936b1513d545eb40db3949ced632e19a459e/numpy-2.4.6-cp313-cp313t-musllinux_1_2_aarch64.whl", hash = "sha256:68bb27509ac1b9a3443094260f6326150663b06abe40b73a2f81160623da5b67", size = 17086024, upload-time = "2026-05-18T23:35:22.52Z" }, + { url = "https://files.pythonhosted.org/packages/4b/6e/23595a2c642cdf3bc567877064bdd7f91c8b0038a4453cf2daf7248eafe9/numpy-2.4.6-cp313-cp313t-musllinux_1_2_x86_64.whl", hash = "sha256:a0df0043bdb289bde1f62da130d20df23d58b45429f752bc7a8fc5325a225ecd", size = 18403398, upload-time = "2026-05-18T23:35:26.398Z" }, + { url = "https://files.pythonhosted.org/packages/8a/90/0ac3bc947217e66dec77e7cbc6a1979d1af70b6461b82f620d3bccd5e4c8/numpy-2.4.6-cp313-cp313t-win32.whl", hash = "sha256:29a287e0cf63ff528da061de6b9f64a4618da591ca1046aafc54062e40ca7eab", size = 6084971, upload-time = "2026-05-18T23:35:29.387Z" }, + { url = "https://files.pythonhosted.org/packages/77/71/5673e351671a1d2bd6063b91b44f70c0affea7d1516fa7a6572941ba4aa1/numpy-2.4.6-cp313-cp313t-win_amd64.whl", hash = "sha256:25c692919ac5a01f170a3bfcd62d745b24fd095c353d50812637d6fcab442e75", size = 12458532, upload-time = "2026-05-18T23:35:32.175Z" }, + { url = "https://files.pythonhosted.org/packages/3f/88/19d3503c5046e688f049274b27a3ef3d771152fa80d3ba3d01a3dff61abe/numpy-2.4.6-cp313-cp313t-win_arm64.whl", hash = "sha256:1e978ec1e8bd0e0e4de6bb75de9d30cbb74db6b6a2bb727618613703ca0167dd", size = 10291881, upload-time = "2026-05-18T23:35:35.465Z" }, + { url = "https://files.pythonhosted.org/packages/f8/91/3ab2044d05fd16d343c5ac2e69b127f1b2854040dd20b193257c78028bd3/numpy-2.4.6-cp314-cp314-macosx_10_15_x86_64.whl", hash = "sha256:06ca2f61ec4385a07a6977c55ba998a4466c123642b4a32694d3128fce18c079", size = 16683458, upload-time = "2026-05-18T23:35:38.353Z" }, + { url = "https://files.pythonhosted.org/packages/8e/62/764ce66fa4147ae6d73071a3abf804ffe606f174618697c571acdf26a7c9/numpy-2.4.6-cp314-cp314-macosx_11_0_arm64.whl", hash = "sha256:38efbc8de75c7a0fc1ac190162d892787f3f47b57cc291231aafee36b80982b7", size = 14704559, upload-time = "2026-05-18T23:35:42.14Z" }, + { url = "https://files.pythonhosted.org/packages/60/61/23f27c172f022e04025b7dc2367f4d63c1a398120607ec896228649a6f48/numpy-2.4.6-cp314-cp314-macosx_14_0_arm64.whl", hash = "sha256:d581b735e177fdcdce6fed8e7e8880a3fb6ee4e3653a3ac6af01c6f4c03effc5", size = 5209716, upload-time = "2026-05-18T23:35:45.377Z" }, + { url = "https://files.pythonhosted.org/packages/03/71/21cf70dc6ea3e3acb95fc53a265b2fc248b981f0194ceb5b475271b8809d/numpy-2.4.6-cp314-cp314-macosx_14_0_x86_64.whl", hash = "sha256:0a041d3d761dc3c35cc56ce0351506a02bcbc25f7b169f652435141a17db9096", size = 6543947, upload-time = "2026-05-18T23:35:47.926Z" }, + { url = "https://files.pythonhosted.org/packages/d5/91/64288395ee1799bd2e0b04a305dce9666da90c961e1f3fe982a05ee1c036/numpy-2.4.6-cp314-cp314-manylinux_2_27_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:40fdc1ae7125e518ea98e53e69a4ebc27e1fd50510c47b7ea130cf21e5e1d42b", size = 15685197, upload-time = "2026-05-18T23:35:50.863Z" }, + { url = "https://files.pythonhosted.org/packages/f3/eb/ebffaa97dc55502df69584a8f0dcf07f69a3e0b3e2323670a2722db9aa39/numpy-2.4.6-cp314-cp314-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:a2c306dea656c12c68f51f4cea133cbe78ca7435eb28c735eac1d3ebe73be6e8", size = 16638245, upload-time = "2026-05-18T23:35:54.752Z" }, + { url = "https://files.pythonhosted.org/packages/b8/0b/54f9da33128d7e350fab89c7455902eeae70349ee52bddb448dc4a576f45/numpy-2.4.6-cp314-cp314-musllinux_1_2_aarch64.whl", hash = "sha256:33111801a01c12a8a1e3721f0a9232f8cfc8ae2c6b7098167e6f623c6073f402", size = 17036587, upload-time = "2026-05-18T23:35:58.355Z" }, + { url = "https://files.pythonhosted.org/packages/b6/f0/fdebc1052db1cc37c64beb22072d67cd6d1c71adca1299f53dec2b5e20d3/numpy-2.4.6-cp314-cp314-musllinux_1_2_x86_64.whl", hash = "sha256:ae506e6902902557576a26ff33eda8695e7ecb3cb36c3b573a0765dee114ebdb", size = 18363226, upload-time = "2026-05-18T23:36:02.845Z" }, + { url = "https://files.pythonhosted.org/packages/aa/b4/298628d98c72b57e57f7165ae6a481a1deaf6f3c28262a6e4c739c275930/numpy-2.4.6-cp314-cp314-win32.whl", hash = "sha256:aaf159caa35993cb1f56fb9b8e4610d35758e7ca005412eb1daa856a78c9c4b1", size = 6010196, upload-time = "2026-05-18T23:36:05.92Z" }, + { url = "https://files.pythonhosted.org/packages/df/ac/46de6dda46478f7942f839e094970be2d4a861e005c4b3bf07c92e291a09/numpy-2.4.6-cp314-cp314-win_amd64.whl", hash = "sha256:b507f5c4c1d508876d1819b6bf9a49d365b96320b5d4993426b33a23ca4b8261", size = 12450334, upload-time = "2026-05-18T23:36:09.107Z" }, + { url = "https://files.pythonhosted.org/packages/78/92/b8b798ac784102c0da830d2257d59358e3d3d90d1e2b3f2575dad976c5cf/numpy-2.4.6-cp314-cp314-win_arm64.whl", hash = "sha256:6f41ae150c4e32db4f3310cdaf64b1593a03dbabe29eec77fc9b50fe64061df6", size = 10495678, upload-time = "2026-05-18T23:36:12.766Z" }, + { url = "https://files.pythonhosted.org/packages/30/34/ec28d1aa8115971537c01469ab2011ee96827930f0a124de1000cc2a7ed7/numpy-2.4.6-cp314-cp314t-macosx_11_0_arm64.whl", hash = "sha256:ece3d2cfe132e7d51f44a832b303895e6f2d499c5e74dfbdb06ee246147a304a", size = 14823672, upload-time = "2026-05-18T23:36:16.473Z" }, + { url = "https://files.pythonhosted.org/packages/16/bd/f6d1fede4e54e8042a7ff97bb495510f3c220f94bcd9e8b228e87c92cc0d/numpy-2.4.6-cp314-cp314t-macosx_14_0_arm64.whl", hash = "sha256:e3e5193ef5a3dc73bceee50f7fdc2c90dbb76c42df8d8fae3d1067a583df579e", size = 5328731, upload-time = "2026-05-18T23:36:19.767Z" }, + { url = "https://files.pythonhosted.org/packages/f4/f0/e105b9e2fd728a9910103884decd6951d9dd73896b914a98d9a231de02ee/numpy-2.4.6-cp314-cp314t-macosx_14_0_x86_64.whl", hash = "sha256:17f9ade344e7d9b464a084d69bcf18fc691cb1db67c62ed80820bf4926d78f0e", size = 6649805, upload-time = "2026-05-18T23:36:22.266Z" }, + { url = "https://files.pythonhosted.org/packages/82/dd/1206a7ca6ab15e3f02069707ca96222e202af681bb73756da7527f3cb837/numpy-2.4.6-cp314-cp314t-manylinux_2_27_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:9cd5ffd25db4e7ba6a375693b3fc0fc1791ec636c17db3720da19bde7180ec43", size = 15730496, upload-time = "2026-05-18T23:36:25.713Z" }, + { url = "https://files.pythonhosted.org/packages/51/e7/38d3ea825dcab85a591734decb2f6c67caa7c8367d374df1a1c3842f9b07/numpy-2.4.6-cp314-cp314t-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:7d92c3819208a60205a12a245c91ad70cb0a85336659b19b834205573ac8456e", size = 16679616, upload-time = "2026-05-18T23:36:29.652Z" }, + { url = "https://files.pythonhosted.org/packages/93/b7/caabfdf53edf663e0b4eb74d7d405d83baef09eb5e83bcd32d601d72b93e/numpy-2.4.6-cp314-cp314t-musllinux_1_2_aarch64.whl", hash = "sha256:e85b752a1e912b70eaad4fafbd4d1238007ab221de2009b9a2f5ae7461239895", size = 17085145, upload-time = "2026-05-18T23:36:33.449Z" }, + { url = "https://files.pythonhosted.org/packages/f9/45/68d7c33a6bcf3e5aa3bdbd57a367e6f615286dfd6482f97e8ffeb734306e/numpy-2.4.6-cp314-cp314t-musllinux_1_2_x86_64.whl", hash = "sha256:29cb7f67d10b479ff07c17d33e39f78c07f71c40ef30d63c153d340e96cd3fb4", size = 18403813, upload-time = "2026-05-18T23:36:37.369Z" }, + { url = "https://files.pythonhosted.org/packages/9c/50/0753655aa844c99cd9e018aacf76f130f1bd81d881bb74bc0aef5d73a8ba/numpy-2.4.6-cp314-cp314t-win32.whl", hash = "sha256:260a5d70215b61ab4fadf5c7baacd64821842975eea312125ed3c39a6391b063", size = 6156982, upload-time = "2026-05-18T23:36:40.817Z" }, + { url = "https://files.pythonhosted.org/packages/b2/d4/7c67becf668f973cb490cec3e98dfd799d866f9c989a54d355672cfa0db6/numpy-2.4.6-cp314-cp314t-win_amd64.whl", hash = "sha256:81a1cca95ed5bb92aa8b10dd2cdc9a0d3853a50fad926c28b5d7e8ea54389627", size = 12638908, upload-time = "2026-05-18T23:36:43.996Z" }, + { url = "https://files.pythonhosted.org/packages/43/bb/e1c71a4295b1b1d1393d50dbb4f2a36283c6859d9d3892e84f00ec5a91d5/numpy-2.4.6-cp314-cp314t-win_arm64.whl", hash = "sha256:0c9136e14ed34a9e343a31c533d78a9813a69a3148332bce5e9821cb2f996e66", size = 10565867, upload-time = "2026-05-18T23:36:47.114Z" }, + { url = "https://files.pythonhosted.org/packages/de/12/b422cc84439adc0d00de605bf4a308890ae5c26f2c71fbd73e5d08fbb0dd/numpy-2.4.6-pp311-pypy311_pp73-macosx_10_15_x86_64.whl", hash = "sha256:55cced7c52e981362f708ad635198e97a752dfba412cc03c23bbf3bd8d5cd662", size = 16847511, upload-time = "2026-05-18T23:36:50.673Z" }, + { url = "https://files.pythonhosted.org/packages/44/53/f481bef68011740f8849418d82db07230e825013f31f4eef5ba5b805316a/numpy-2.4.6-pp311-pypy311_pp73-macosx_11_0_arm64.whl", hash = "sha256:d6da64deb6b8ed903e7560180a92f2d804ee1ba5eeb849ac2748b8c1aba1f6d7", size = 14889064, upload-time = "2026-05-18T23:36:53.879Z" }, + { url = "https://files.pythonhosted.org/packages/7f/57/42ed575c10ced8af951d426bc4e1f8aff16fd851db33f067036215a7f860/numpy-2.4.6-pp311-pypy311_pp73-macosx_14_0_arm64.whl", hash = "sha256:68a5124b13fa6cc2086764a20005d30bc0548146f7f5322f02fce212ca14317f", size = 5394157, upload-time = "2026-05-18T23:36:57.194Z" }, + { url = "https://files.pythonhosted.org/packages/6a/ef/f66cc724fcc36c1e364c67f51ae9146090b8b584f27d58b97fdae3edd737/numpy-2.4.6-pp311-pypy311_pp73-macosx_14_0_x86_64.whl", hash = "sha256:948424b06129ce883307e8cff868c31396d8dc7630a59c61d70d98dbe70f222c", size = 6708728, upload-time = "2026-05-18T23:36:59.575Z" }, + { url = "https://files.pythonhosted.org/packages/1a/9c/c531f2293b91265d8b48e9b329f54fdd7ffae73cb4134ea10cca4237e9cc/numpy-2.4.6-pp311-pypy311_pp73-manylinux_2_27_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:5dbbdb29840ca3d91ee0fece42fc29278886d908280bfec0a5846c6f901a3eb0", size = 15798374, upload-time = "2026-05-18T23:37:02.674Z" }, + { url = "https://files.pythonhosted.org/packages/1a/b0/413077f6b1153ed3cba361401c6783bbad6114804a000cc22eb71c13e190/numpy-2.4.6-pp311-pypy311_pp73-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:8ad03c0965fb3c692200e74d458ca28c1dbb4ce96f9a479a8aa041ad5fabca02", size = 16747286, upload-time = "2026-05-18T23:37:06.327Z" }, + { url = "https://files.pythonhosted.org/packages/15/ce/e5ec180bc41812edcd8daeb8639d205622c0e8c02259d8ab25a0201b3c2a/numpy-2.4.6-pp311-pypy311_pp73-win_amd64.whl", hash = "sha256:2803abfebfc990042cd494d8ce2d5f82e9d847af6d35ec486923aa19dbad5e73", size = 12504263, upload-time = "2026-05-18T23:37:09.715Z" }, +] + +[[package]] +name = "numpy" +version = "2.5.4" +source = { registry = "https://pypi.org/simple" } +resolution-markers = [ + "python_full_version >= '3.15'", + "python_full_version >= '3.12' and python_full_version < '3.15'", +] +sdist = { url = "https://files.pythonhosted.org/packages/95/b0/c7453d0b6e2073c3264468b106ee1563750cecc910965e67357e3698c83e/numpy-2.5.4.tar.gz", hash = "sha256:9a94cf751c9ad8ebaa835bcd3d40dacf8534ad086b88c38029b65123c7999d2a", size = 20866315, upload-time = "2026-10-10T20:05:31.422Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/d0/97/ba2074e92b7befea137e77ea8471e768bbd87c339b7e8c9f5a931949f977/numpy-2.5.4-cp312-cp312-macosx_10_13_x86_64.whl", hash = "sha256:c6342f54c67093cae5c0227eb0eb772fdb79f2a2c37a6eb278b9909ee06aa356", size = 17001609, upload-time = "2026-10-10T20:02:40.843Z" }, + { url = "https://files.pythonhosted.org/packages/ff/a9/bac826765e971d8e16e2064e9ac7525fd69b40ac17c905033a7f5442023f/numpy-2.5.4-cp312-cp312-macosx_11_0_arm64.whl", hash = "sha256:b11e8fda06a7d69f15ebf542660b74466c2e51094800c1fb794f47ad4faeef17", size = 12015718, upload-time = "2026-10-10T20:02:43.45Z" }, + { url = "https://files.pythonhosted.org/packages/31/2f/5ea3570fcb8ccd0882bea99436a513b2c85dad8f774a2057849130a8fb99/numpy-2.5.4-cp312-cp312-macosx_14_0_arm64.whl", hash = "sha256:9cb18a327b49c5c337f972b03682f6a49855525faaf3c0d3e9c96cd0fd8880a8", size = 5451717, upload-time = "2026-10-10T20:02:46.169Z" }, + { url = "https://files.pythonhosted.org/packages/34/f2/b4fc1bafca03868220b5eaf729d2f21ebd7d7b151c0f9e144fe212bbca35/numpy-2.5.4-cp312-cp312-macosx_14_0_x86_64.whl", hash = "sha256:aec3fc4b32ff82421274f5d205c559c51c840c8df66a78efd7f3612dd005a26a", size = 6789926, upload-time = "2026-10-10T20:02:48.139Z" }, + { url = "https://files.pythonhosted.org/packages/dc/96/8319e2457ae4333c62c815c7006b869a4f60985c1e01024c2f8c6c040fe5/numpy-2.5.4-cp312-cp312-manylinux_2_27_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:fe4d21ab149f15e4e6043dfb0de87e6e5f34ac176cde83060e9802981fca2ac2", size = 15695312, upload-time = "2026-10-10T20:02:50.115Z" }, + { url = "https://files.pythonhosted.org/packages/43/a3/c799c62e19c337e6d3770b08e475887fb30ce8477d3c09efca6b2f0228a6/numpy-2.5.4-cp312-cp312-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:fbde6962867ee75b48b0ee29b2b9372ec5d617799dbaf38e82dc0596f2f7738a", size = 16727283, upload-time = "2026-10-10T20:02:53.186Z" }, + { url = "https://files.pythonhosted.org/packages/39/6b/3604e53fb00314d0dc1b94ec9125a1484f649c0a17480b1f0f0c7a9d6250/numpy-2.5.4-cp312-cp312-musllinux_1_2_aarch64.whl", hash = "sha256:381a7a3d2e65e64c0ec302795ab9dc12bb1e73f150904699c153716177eebdaf", size = 17047890, upload-time = "2026-10-10T20:02:56.038Z" }, + { url = "https://files.pythonhosted.org/packages/4a/7a/e8b58a5289a0d464c52885de47c35a935cdd70c03a4c3ab94a5126416dd0/numpy-2.5.4-cp312-cp312-musllinux_1_2_x86_64.whl", hash = "sha256:b89d0aaae2fe498c648f4c4795c084db535af5bd98ef942b2a3681fb74ce8645", size = 18485839, upload-time = "2026-10-10T20:02:59.018Z" }, + { url = "https://files.pythonhosted.org/packages/6f/c9/47094f597015009f310b8c900def59065ef1ff5a6fe7b51fc65ec58ec2c6/numpy-2.5.4-cp312-cp312-win32.whl", hash = "sha256:9968ab7e49b93ac6e1c3b2239732183152c9150f16308d30b66a372cffe3483c", size = 6138936, upload-time = "2026-10-10T20:03:01.626Z" }, + { url = "https://files.pythonhosted.org/packages/12/33/fefe62073dc8acfd0f2b9ed7c003af2f50aa61555e113e6db02b8f79f145/numpy-2.5.4-cp312-cp312-win_amd64.whl", hash = "sha256:a7b1b6353e36a7e50de2973a38d705c88ee93adcf120673cee7f45a4a3fa223a", size = 12573091, upload-time = "2026-10-10T20:03:04.349Z" }, + { url = "https://files.pythonhosted.org/packages/1a/07/161270b0c2eec56e4c905f6d6d22e1b836887b2cb189d3f5820aa588e9dd/numpy-2.5.4-cp312-cp312-win_arm64.whl", hash = "sha256:aa1cce2ff3f8d953de38b76bf44602caeb69f101430208f64a10067f7cb4b1d3", size = 10521630, upload-time = "2026-10-10T20:03:06.767Z" }, + { url = "https://files.pythonhosted.org/packages/67/14/1c3ee0118a8fce08565a5d8482631608426a33af10a01077fada5dc7c119/numpy-2.5.4-cp313-cp313-macosx_10_13_x86_64.whl", hash = "sha256:2377da2dd3ba2c1200956acbab2a358c83b8e1f8531191672d1cd6ad83250d53", size = 16997729, upload-time = "2026-10-10T20:03:09.291Z" }, + { url = "https://files.pythonhosted.org/packages/83/8c/b0ea9477fb1f0d4484bbc5cba21678cc9969704d8d7f3f158d1db35f8e14/numpy-2.5.4-cp313-cp313-macosx_11_0_arm64.whl", hash = "sha256:7415db95818b39ec475a5eea54d9e3b6bc83e3912158e46da3438cdce399804d", size = 12009826, upload-time = "2026-10-10T20:03:11.946Z" }, + { url = "https://files.pythonhosted.org/packages/e2/84/6a3d75b3ba3dfe84ac0053450753d1e6d250a8bf80f66474cc46d1fb643f/numpy-2.5.4-cp313-cp313-macosx_14_0_arm64.whl", hash = "sha256:6d6a71b9d9a97c03633aa12565ef2825ffa036cc1d99cfd50dacf0f128af4fe2", size = 5445803, upload-time = "2026-10-10T20:03:14.329Z" }, + { url = "https://files.pythonhosted.org/packages/61/18/bb993f267ca20b376e07092a16793a5b31ed3138751e9ba480011a14d742/numpy-2.5.4-cp313-cp313-macosx_14_0_x86_64.whl", hash = "sha256:d8200f16437b289a5bb927c6e184eccc3e8389bc0070fea4cd5b9e13c1757959", size = 6786220, upload-time = "2026-10-10T20:03:16.602Z" }, + { url = "https://files.pythonhosted.org/packages/db/b6/135bb0953b61dc21c6cafa14b424ae666944e4899cf140e00c2b322a1a45/numpy-2.5.4-cp313-cp313-manylinux_2_27_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:1c2e71b04c6cad90026e544501bbe0ab9290fa8a4d845e7e8c0d124fb429c988", size = 15689178, upload-time = "2026-10-10T20:03:18.721Z" }, + { url = "https://files.pythonhosted.org/packages/da/24/3bd070f3269dc609d8f26b2643f62ef91bb415841c0b294805aaf7fe06da/numpy-2.5.4-cp313-cp313-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:6ffa07666f8da0eef81d149934a626d0d95fbd6838432a33e66245423a9062c0", size = 16718044, upload-time = "2026-10-10T20:03:21.386Z" }, + { url = "https://files.pythonhosted.org/packages/c7/8e/9d15bd356b0a019c965312b1a3c6a727cac4cae5bc40045fbc12ce4cff9c/numpy-2.5.4-cp313-cp313-musllinux_1_2_aarch64.whl", hash = "sha256:2fa3328f784fc8277fc48026f6cad516f5c561c5d8e2e39b3c9e0c8f23223b34", size = 17048364, upload-time = "2026-10-10T20:03:24.468Z" }, + { url = "https://files.pythonhosted.org/packages/dc/fe/9d5b560db964f15871885f2250795d15945f8699e17ef90c0c2ff4c875b2/numpy-2.5.4-cp313-cp313-musllinux_1_2_x86_64.whl", hash = "sha256:b86966fbe4ad7de710422175572bcdc75fdedadfb54bc6fab7deabccddd7780b", size = 18474904, upload-time = "2026-10-10T20:03:27.895Z" }, + { url = "https://files.pythonhosted.org/packages/e9/98/d27552990f1bd611ef3e7466adadc78312ea2df63b83aad47fdc3d3ca8df/numpy-2.5.4-cp313-cp313-win32.whl", hash = "sha256:5258bc06526964be5face2fc6f756857a3f24f21ec3e72ca131337a75b165d6c", size = 6134537, upload-time = "2026-10-10T20:03:30.511Z" }, + { url = "https://files.pythonhosted.org/packages/90/8c/140a40398a66b4471211be1affdb6ed24c486d581bd28d07b7f2fcb69540/numpy-2.5.4-cp313-cp313-win_amd64.whl", hash = "sha256:8b4d2fd2d34e5f8c9235ee787de5631a37a28402b15cb80814df973d2be54129", size = 12566113, upload-time = "2026-10-10T20:03:32.612Z" }, + { url = "https://files.pythonhosted.org/packages/34/52/01d205e5e8ccb27b2b0b141e801f22b830198c979111b0fa44771438d9a9/numpy-2.5.4-cp313-cp313-win_arm64.whl", hash = "sha256:bc39ac66a7a9a3fbd6134fda43136b60ffde99c8f4501e64e0d2b24da137babf", size = 10519523, upload-time = "2026-10-10T20:03:35.163Z" }, + { url = "https://files.pythonhosted.org/packages/99/ba/005cb5edd580d2f84d7ca3206b92dc17d4388e56e6f87ffe8f2762f83139/numpy-2.5.4-cp314-cp314-macosx_10_15_x86_64.whl", hash = "sha256:c668b2f0d651605b58892644b0e302c7157f7159544227758c896982ef384b18", size = 17005499, upload-time = "2026-10-10T20:03:37.961Z" }, + { url = "https://files.pythonhosted.org/packages/f3/49/fee7587c33ee35f7977f9051d7f2023d4e7246d62710c80f20c2361ea232/numpy-2.5.4-cp314-cp314-macosx_11_0_arm64.whl", hash = "sha256:ffa6ce09a1c6a08e9667dd9c97aa0b14184e8d18f2a14b78b2a2328c9147f076", size = 12019666, upload-time = "2026-10-10T20:03:40.606Z" }, + { url = "https://files.pythonhosted.org/packages/d5/b2/c6ce165acffceb15a82c07b9cc77d391f86b3f379ba62911908ae5d34b91/numpy-2.5.4-cp314-cp314-macosx_14_0_arm64.whl", hash = "sha256:956555e0603a4d38019ae6925711cb9dc43195c076a928accf7ea5d50bddfe53", size = 5455617, upload-time = "2026-10-10T20:03:43.138Z" }, + { url = "https://files.pythonhosted.org/packages/77/7f/dd85ce260a669a89be06842cf355d7353a33e6cfbc590fb8ebb947d88dc9/numpy-2.5.4-cp314-cp314-macosx_14_0_x86_64.whl", hash = "sha256:2c2c4afffdeb7920e445028dd71eb932cac3e704792e964bc2a232426d4f1255", size = 6791932, upload-time = "2026-10-10T20:03:44.874Z" }, + { url = "https://files.pythonhosted.org/packages/63/d6/34b0a2b0741386a63025a65a2c09caaaaaad6d0ca95b66cd65c30dd7fcb5/numpy-2.5.4-cp314-cp314-manylinux_2_27_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:4054173604cd8658796053f1f3bc0befb68ec1c0762c57fdad61e199256a8617", size = 15710899, upload-time = "2026-10-10T20:03:46.839Z" }, + { url = "https://files.pythonhosted.org/packages/16/d5/928078d2b28f26829b138b4a6c3980045022fb409f570657a224ae60ef4e/numpy-2.5.4-cp314-cp314-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:d549420b8858885cea8838a727842249218b9c1da24dd517e25c9c7a948310a3", size = 16721710, upload-time = "2026-10-10T20:03:49.489Z" }, + { url = "https://files.pythonhosted.org/packages/f9/cf/673fd1b8f4cd78eb6320e87ec4c90ac19c095644259e3749853a405c70f4/numpy-2.5.4-cp314-cp314-musllinux_1_2_aarch64.whl", hash = "sha256:823874a507a84af050493b622affde94b6f7c3a0dc22cb2801381bc03b871c00", size = 17066182, upload-time = "2026-10-10T20:03:52.25Z" }, + { url = "https://files.pythonhosted.org/packages/f3/92/a77b5061b1b3e2643928c37976d79ee173e1b171ed158b7a3c61056b41bc/numpy-2.5.4-cp314-cp314-musllinux_1_2_x86_64.whl", hash = "sha256:4e263278bfb5ee6409db8aedbc4cc32973b1b82bc1e8d3c668551d04d83a7e37", size = 18480315, upload-time = "2026-10-10T20:03:55.39Z" }, + { url = "https://files.pythonhosted.org/packages/bb/1d/1486ef3d3fb2279fd93c4c43c1bbbf1ca389a19816696684409f71babaab/numpy-2.5.4-cp314-cp314-win32.whl", hash = "sha256:cfd73180400042a7c532d30c5e287bdd03c59ff9ee1b4c0316af0539e29dfe23", size = 6185739, upload-time = "2026-10-10T20:03:58.186Z" }, + { url = "https://files.pythonhosted.org/packages/52/9a/e1e512ebc948d5b9dd33b08736760f0ebbed2848fd4eda1f553088a6dcee/numpy-2.5.4-cp314-cp314-win_amd64.whl", hash = "sha256:2ca144f15135b6212a5c47b1e2aeca6e412f102f95a2d5d88d8aec77eb255de3", size = 12703552, upload-time = "2026-10-10T20:04:00.28Z" }, + { url = "https://files.pythonhosted.org/packages/2c/05/de709a982d7bbcd688a3fad71f002e9ff80c2db39e03ee726609b610f1d1/numpy-2.5.4-cp314-cp314-win_arm64.whl", hash = "sha256:468397ba3c64427474706e5c9123fe266395496714dc684294eac75cd4930d1e", size = 10803901, upload-time = "2026-10-10T20:04:02.659Z" }, + { url = "https://files.pythonhosted.org/packages/13/34/083570ada3bb2a30fbe5d77c8c6fef9141144a15d33e6f793a67e9749ab8/numpy-2.5.4-cp314-cp314t-macosx_11_0_arm64.whl", hash = "sha256:1ef3aa6d7e29bb13677323114280b05acc57607fa2300e66432d665d5418a162", size = 12138695, upload-time = "2026-10-10T20:04:05.012Z" }, + { url = "https://files.pythonhosted.org/packages/94/06/1f9c24db48eef0c2d1207e3b11fffb0478e39dfd8c1e1be7476936885eed/numpy-2.5.4-cp314-cp314t-macosx_14_0_arm64.whl", hash = "sha256:98b053943e5a0474ec0da309d2cb9d3f18ea57f8a2067c2ab7b5f763d1068380", size = 5574615, upload-time = "2026-10-10T20:04:07.316Z" }, + { url = "https://files.pythonhosted.org/packages/da/0f/593fba2e1560e949123bc7d2fc48b5893d56e58cd4bd5a273d2fbf60b220/numpy-2.5.4-cp314-cp314t-macosx_14_0_x86_64.whl", hash = "sha256:b64a85f40e154983960a4167d4c1d57a50c7f109b3d3264a3a984154e90a8454", size = 6889383, upload-time = "2026-10-10T20:04:09.918Z" }, + { url = "https://files.pythonhosted.org/packages/eb/9f/b799dfdce4e05e80ed4bc815c71ff343a11533b2c0ffc221cae8538cda63/numpy-2.5.4-cp314-cp314t-manylinux_2_27_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:a813ed7719bf45463c51779e6a98d0385fe905e48447526938a4b8337333d551", size = 15753763, upload-time = "2026-10-10T20:04:12.278Z" }, + { url = "https://files.pythonhosted.org/packages/34/88/16c5f12f86f5ad2817c4d103205131fc6c8acb3d1878af05a1a4f23ec859/numpy-2.5.4-cp314-cp314t-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:c9b80cdf5cedba0e90d93fa5f9a333c4d65bd545cd669b71bb97ce2b703c9d73", size = 16757212, upload-time = "2026-10-10T20:04:14.799Z" }, + { url = "https://files.pythonhosted.org/packages/ff/4f/a1fe40e18a898e6a5089f4f0d891f0a493eb0574d5b34458f0fbe5aa3e5c/numpy-2.5.4-cp314-cp314t-musllinux_1_2_aarch64.whl", hash = "sha256:2199ed071f460487c8db2c0e5c0b564494190edb4772fe80f9aad88b2604def5", size = 17116471, upload-time = "2026-10-10T20:04:17.58Z" }, + { url = "https://files.pythonhosted.org/packages/aa/46/e923a11c78e65c1722e7aaad817c06bd591324174b9d28ce5d31eee4d432/numpy-2.5.4-cp314-cp314t-musllinux_1_2_x86_64.whl", hash = "sha256:64f9c9878c1938476365e11ccfb6b770f3b9e5f045ccddc514235041e6959365", size = 18524063, upload-time = "2026-10-10T20:04:20.365Z" }, + { url = "https://files.pythonhosted.org/packages/5a/fa/84ab064514440c1f64a1b21088f2c82756defdd05e07c75ab233899565b2/numpy-2.5.4-cp314-cp314t-win32.whl", hash = "sha256:64d1c8ac28a4077cf987e0a71a7a0ef7e2df70722f07f0baa42dbb7eb6938647", size = 6340926, upload-time = "2026-10-10T20:04:22.865Z" }, + { url = "https://files.pythonhosted.org/packages/7e/7e/6cd886876f435b10685db9b9f7eeb70356f99e052116f4e5f11c5792c714/numpy-2.5.4-cp314-cp314t-win_amd64.whl", hash = "sha256:067374eb538c34c745436365cf7b0112595c1d326f21ce4ff340f61230239fbb", size = 12901584, upload-time = "2026-10-10T20:04:24.99Z" }, + { url = "https://files.pythonhosted.org/packages/38/1b/3c1684f6a06f7307f2335fca6e486cb162847fb97e91d65f8eb5cabad213/numpy-2.5.4-cp314-cp314t-win_arm64.whl", hash = "sha256:e94aef2c639da4a960ad0db8e06471208d8589974953d78b61d345b4eb99e394", size = 10891152, upload-time = "2026-10-10T20:04:27.52Z" }, + { url = "https://files.pythonhosted.org/packages/08/f4/3224deff3af2bef6bc0b175369698d8cb348f3d91d9bb0286cd5c9eae9e0/numpy-2.5.4-cp315-cp315-macosx_10_15_x86_64.whl", hash = "sha256:8dddfbee2e68d26d0d7d7d9cb247b1fd4409241cce32d815a11d97ec2cfde179", size = 17003231, upload-time = "2026-10-10T20:04:30.021Z" }, + { url = "https://files.pythonhosted.org/packages/be/75/fee0b8c6d94b44b2fdfae74f6a4ad5a138739589a8aebaec28ce4e713ed5/numpy-2.5.4-cp315-cp315-macosx_11_0_arm64.whl", hash = "sha256:81e3420b27048b65eb14c3acf0c174a8cb0e023277716110347d2dcb26026dad", size = 12018300, upload-time = "2026-10-10T20:04:32.519Z" }, + { url = "https://files.pythonhosted.org/packages/47/c0/d0b335a499a04b65f532c3f034346ef390f81299060f928492dabc1e0272/numpy-2.5.4-cp315-cp315-macosx_14_0_arm64.whl", hash = "sha256:0b4724a19de67bea8cfc4970798efa78bcbbe2ac2613cfac16721a42d44de2a5", size = 5454250, upload-time = "2026-10-10T20:04:34.943Z" }, + { url = "https://files.pythonhosted.org/packages/5a/0e/461b3783c03d668052e6a21b01b673db6ffcb7831fd32d9aa5368c1cd426/numpy-2.5.4-cp315-cp315-macosx_14_0_x86_64.whl", hash = "sha256:2132418bf8dd124a427ca9e6a1daf9ee1a87185344c95119ceae868b99466da1", size = 6789644, upload-time = "2026-10-10T20:04:37.258Z" }, + { url = "https://files.pythonhosted.org/packages/b3/02/5dad269b02166965a7b4ca14adaddd75dbee0de42435bfecf561b84ba5a6/numpy-2.5.4-cp315-cp315-manylinux_2_27_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:325518d4245b9e331387702aa58c2ce1dc4cdcbb41dfb4ccd5dcbc7e08db1266", size = 15704353, upload-time = "2026-10-10T20:04:39.616Z" }, + { url = "https://files.pythonhosted.org/packages/93/3a/01360c8036822ed9f7aa32189a77d1476567ec1e8e1383522389e4faac45/numpy-2.5.4-cp315-cp315-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:56733449d2544178beaa4545cee357370440cf056c197f9c7bfb19dbfdd0e86d", size = 16718648, upload-time = "2026-10-10T20:04:42.383Z" }, + { url = "https://files.pythonhosted.org/packages/7d/5c/b863a2c093c4d6f21a597fcaf24ead0835c09ab16a8312d5a5a8868af683/numpy-2.5.4-cp315-cp315-musllinux_1_2_aarch64.whl", hash = "sha256:5ec3753760c1a6d8bb91200666e545c3a9728e6269dfb5d6ce02340996698aa3", size = 17059053, upload-time = "2026-10-10T20:04:44.976Z" }, + { url = "https://files.pythonhosted.org/packages/0a/60/ced4f57f9a1258a0af74f17cb0b0c2700b5c67cd6678823c803b263e4df3/numpy-2.5.4-cp315-cp315-musllinux_1_2_x86_64.whl", hash = "sha256:b1185012870173de7ae33d370bd45b1cf5baee747ea4b97036b65f4e93016877", size = 18477406, upload-time = "2026-10-10T20:04:47.863Z" }, + { url = "https://files.pythonhosted.org/packages/f9/bd/0ef22dafaafcc7d4bb3ca26b8d2afbd55dedad8eaba99a8c864e1997456f/numpy-2.5.4-cp315-cp315-win32.whl", hash = "sha256:298eca75243f2cbbfdb460560b9fb2a1792a33cf2ab4286efd43d92e8d3df508", size = 6185133, upload-time = "2026-10-10T20:04:50.467Z" }, + { url = "https://files.pythonhosted.org/packages/50/bc/d2651b155ecc608a77e6f4d15495c11f14f19bb98f8bf0c5b0d38f86dda1/numpy-2.5.4-cp315-cp315-win_amd64.whl", hash = "sha256:332f3378fe077dd850e677ec01bdcc4f22368fb5d50ef10b2c79230b1bf5a592", size = 12703085, upload-time = "2026-10-10T20:04:52.63Z" }, + { url = "https://files.pythonhosted.org/packages/dc/d2/45e404f8abb26fb9eda12b94012936873e827b1be76f2ee7890be128312e/numpy-2.5.4-cp315-cp315-win_arm64.whl", hash = "sha256:d4cccbbc78717966f764cd3af4fb70276fa01fc7a2688af11c78901fa5c04f05", size = 10801451, upload-time = "2026-10-10T20:04:55.677Z" }, + { url = "https://files.pythonhosted.org/packages/c6/c3/2ae14e09cfdb67dc187a342e15308a21c15bf4d2071f8079e6aee5fe56dc/numpy-2.5.4-cp315-cp315t-macosx_10_15_x86_64.whl", hash = "sha256:950ea81d57ef070665581b6e1b5f6a029306423cd1739c5b95fe78aa30db6b9d", size = 17097121, upload-time = "2026-10-10T20:04:58.403Z" }, + { url = "https://files.pythonhosted.org/packages/f5/cf/305ae624ef8a039414317224abe9ec9c2fe7ea3c2e1cf204d43ff6b2ffb9/numpy-2.5.4-cp315-cp315t-macosx_11_0_arm64.whl", hash = "sha256:c05ede731b03fb1b7591faca9389ade3267d2bddf1ad8882bb3f2cc5e101694f", size = 12135439, upload-time = "2026-10-10T20:05:01.65Z" }, + { url = "https://files.pythonhosted.org/packages/a9/a8/f75c63813aef95827bb2c0d13b12803016853056e8792c280058cdbfe783/numpy-2.5.4-cp315-cp315t-macosx_14_0_arm64.whl", hash = "sha256:5fbf7141bbfd63aea22f435c9062a032b9ea0082fe9845dad7f021d3f1234e71", size = 5571451, upload-time = "2026-10-10T20:05:04.135Z" }, + { url = "https://files.pythonhosted.org/packages/6f/0f/f17763f983868b5c49b4101ebd7e00760bd1769478a6bb6a8de6e085bbac/numpy-2.5.4-cp315-cp315t-macosx_14_0_x86_64.whl", hash = "sha256:3573cd22564692a5b899ec344e5d5b9cc4576f2985b96f22af3564ed54f2710f", size = 6883356, upload-time = "2026-10-10T20:05:06.249Z" }, + { url = "https://files.pythonhosted.org/packages/67/a7/8af04c5a79e047996cfa38854dcfbececdd0343a7c933a46fdd03ef6f5da/numpy-2.5.4-cp315-cp315t-manylinux_2_27_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:6c109eac9cd439193678f69d70733c1108487546ca8eafc107b510ae10c1aecd", size = 15750991, upload-time = "2026-10-10T20:05:08.376Z" }, + { url = "https://files.pythonhosted.org/packages/57/7a/648254290d0c504faa8f2d07aa206660c728802c781a6f3fc68ab7cb5d71/numpy-2.5.4-cp315-cp315t-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:80d6ef6e8620eb2c2b4c4caad50b5935d6db3cde2d51581b55dcc79e14016d1d", size = 16757675, upload-time = "2026-10-10T20:05:11.393Z" }, + { url = "https://files.pythonhosted.org/packages/b8/fe/4a8c3cdb0c70400cfe4c5bec42d3099a5673802a95064614b33e07b82aa1/numpy-2.5.4-cp315-cp315t-musllinux_1_2_aarch64.whl", hash = "sha256:77045a4b175bbf5316ec08003880804336c78f92281a1b72222b274ea85ec5ac", size = 17113846, upload-time = "2026-10-10T20:05:14.49Z" }, + { url = "https://files.pythonhosted.org/packages/1b/7e/619692bb67778702c0e9eb2d468568a7573f4e269386ea61aed01ee4e557/numpy-2.5.4-cp315-cp315t-musllinux_1_2_x86_64.whl", hash = "sha256:0f02a46e49cfb6c73bdb7aea1c0d3461dbae9aba613542b65f657cd3d17b9fab", size = 18522915, upload-time = "2026-10-10T20:05:17.33Z" }, + { url = "https://files.pythonhosted.org/packages/b7/b5/4da41c328788f575838f97a098fe8ca691ebc6f6fd73ad4a262ee40b184d/numpy-2.5.4-cp315-cp315t-win32.whl", hash = "sha256:ad62a416ddcf863bf44bba76fbf6b53366ab0692e294f51cae4b5fbe0d246788", size = 6335804, upload-time = "2026-10-10T20:05:19.921Z" }, + { url = "https://files.pythonhosted.org/packages/98/94/6482ddfa3d312490cb9358f375bf2ad56427dbea8769187158e94d653753/numpy-2.5.4-cp315-cp315t-win_amd64.whl", hash = "sha256:38f47be9f74ab870d2633b5456ae519c43758a8d1fd05342f0ce4ecc034396ee", size = 12890095, upload-time = "2026-10-10T20:05:21.875Z" }, + { url = "https://files.pythonhosted.org/packages/48/7f/c2d1b436b6e7cfebac140c2579a298344b85f2991a2ce5c3615cefb29400/numpy-2.5.4-cp315-cp315t-win_arm64.whl", hash = "sha256:7a14a461d9340f1b46b8648578aed9cdb8b3b018a8fac6c1dde2c9192a01a87f", size = 10883718, upload-time = "2026-10-10T20:05:28.547Z" }, +] + +[[package]] +name = "packaging" +version = "26.3" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/7d/fa/3944b40b07da9ce895c0e6303a5ab7d53da063554f534556b134a54d6093/packaging-26.3.tar.gz", hash = "sha256:94edc256424af38762eb31306eed28beb9f0efc50a8837492c9d6fd6004aed79", size = 313412, upload-time = "2026-08-04T18:15:28.737Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/63/34/ba1c580383c9eada3711951fef0795c80b829a078d72188184bcab9dd527/packaging-26.3-py3-none-any.whl", hash = "sha256:d7193f7c8e4e93f444fde0262bf90af30e16fa0ad0ad44cb553c87339b23cd1c", size = 129956, upload-time = "2026-08-04T18:15:27.159Z" }, +] + +[[package]] +name = "pathspec" +version = "1.1.1" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/5a/82/42f767fc1c1143d6fd36efb827202a2d997a375e160a71eb2888a925aac1/pathspec-1.1.1.tar.gz", hash = "sha256:17db5ecd524104a120e173814c90367a96a98d07c45b2e10c2f3919fff91bf5a", size = 135180, upload-time = "2026-04-27T01:46:08.907Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/f1/d9/7fb5aa316bc299258e68c73ba3bddbc499654a07f151cba08f6153988714/pathspec-1.1.1-py3-none-any.whl", hash = "sha256:a00ce642f577bf7f473932318056212bc4f8bfdf53128c78bbd5af0b9b20b189", size = 57328, upload-time = "2026-04-27T01:46:07.06Z" }, +] + +[[package]] +name = "pillow" +version = "13.0.0" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/ce/a4/9cc30690a394a8305e8be4c9036aa16d0dfddadd09079860b582f562496d/pillow-13.0.0.tar.gz", hash = "sha256:cdcd1ba258178f7356d74170c9a81ba4eab6a5d55748bc402a5a2a6fe6be3510", size = 47058631, upload-time = "2026-10-16T22:18:09.718Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/6c/d7/af6af1b8cc55e07cb0489d5b0950bf5a9bb1461eeba825700b6d8d0e26ea/pillow-13.0.0-cp311-cp311-macosx_10_10_x86_64.whl", hash = "sha256:6f38f3f885a09cafc93fd667b1173a92f379e43785d635e7983c893f0da319fb", size = 5592324, upload-time = "2026-10-16T22:15:24.672Z" }, + { url = "https://files.pythonhosted.org/packages/0b/d2/2c12cb4fbb5d95f38d1001ff3d95fafcb830077018cba40e56a8f932dfbc/pillow-13.0.0-cp311-cp311-macosx_11_0_arm64.whl", hash = "sha256:37290fbef0c73e2d51d594eb78fded7dc7170d1c3ad0a890269e6fd8c8c4e061", size = 4856298, upload-time = "2026-10-16T22:15:27.669Z" }, + { url = "https://files.pythonhosted.org/packages/5c/ee/229b27472483a4801ea94990815e710391836afdf987b4b722b62bd75db6/pillow-13.0.0-cp311-cp311-manylinux_2_27_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:7c88c10dd432f3b39ccdafc9ed81d0c21b8fd634502574da76d1bfa4fe2c76d0", size = 6517304, upload-time = "2026-10-16T22:15:29.43Z" }, + { url = "https://files.pythonhosted.org/packages/36/46/f509b74787f40ce0627501089d630b2c741b63b244c1f40b640b3b01e181/pillow-13.0.0-cp311-cp311-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:a625df9a2a5597fa5e5266e01aefc82604bb85d291e3547981e863560d75da6c", size = 7140667, upload-time = "2026-10-16T22:15:31.368Z" }, + { url = "https://files.pythonhosted.org/packages/aa/a4/5a398b8ec2c37145397d6461c0b551c23ec03072aa64f357aed25e582b00/pillow-13.0.0-cp311-cp311-musllinux_1_2_aarch64.whl", hash = "sha256:10b5019883c98f1dd819ca48ffc6f33a91cee04948fe637c605f4345c7cafd6f", size = 6610583, upload-time = "2026-10-16T22:15:34.025Z" }, + { url = "https://files.pythonhosted.org/packages/4f/7b/f2f1109b230fd4e2d57ab2305de31adc37ea9b4ff8719436c623cb484415/pillow-13.0.0-cp311-cp311-musllinux_1_2_x86_64.whl", hash = "sha256:f391dc92a015c478745192fe4f2e120c2af2aba18d9fe0616341cbf537599806", size = 7218241, upload-time = "2026-10-16T22:15:36.214Z" }, + { url = "https://files.pythonhosted.org/packages/ab/d3/069edb9083b267746baf59756282cacef904d3475c5288c7216fb5195a98/pillow-13.0.0-cp311-cp311-win32.whl", hash = "sha256:6a0920cb55df5a4d3f6862964908078c0bd59332bdd1bcd20d14e8c70ef27cc5", size = 6515278, upload-time = "2026-10-16T22:15:38.129Z" }, + { url = "https://files.pythonhosted.org/packages/7d/f6/6e2d96413e4c9dbec12ea584d9e646d602877d4cce3ea669099f959deac6/pillow-13.0.0-cp311-cp311-win_amd64.whl", hash = "sha256:d7710eb676e9bb68965cb045be0d333b4c0d7f9dfbaa7fbf121385b1588683d3", size = 7278471, upload-time = "2026-10-16T22:15:40.286Z" }, + { url = "https://files.pythonhosted.org/packages/70/e7/baff21f8cb60496c8c250609501b88919eeedb4088d3da106f6c62f3f3c0/pillow-13.0.0-cp311-cp311-win_arm64.whl", hash = "sha256:12fbb851cf323cc19aac3eea1df5ad17b57e91e9fe3cfd06100dba7bb8208c13", size = 2698981, upload-time = "2026-10-16T22:15:42.637Z" }, + { url = "https://files.pythonhosted.org/packages/7a/b8/58d79332229482eb0d27c82973850808c19d37e5bdc14443d3417bd373f5/pillow-13.0.0-cp312-cp312-macosx_10_13_x86_64.whl", hash = "sha256:67c8d44ad89fa7eb88de5d6ef6199b1e03dd24974f85a67639fc8badb801fe79", size = 5568643, upload-time = "2026-10-16T22:15:44.474Z" }, + { url = "https://files.pythonhosted.org/packages/39/4b/f2ed53d83aa44ac19599ac0a54897f867fafb95f406980d1e24083f8cbe3/pillow-13.0.0-cp312-cp312-macosx_11_0_arm64.whl", hash = "sha256:fc481f742ccda20e9ed760b8f978dff50f2a4e9bc253ac167fb0a05b0783ef52", size = 4852329, upload-time = "2026-10-16T22:15:46.284Z" }, + { url = "https://files.pythonhosted.org/packages/3b/bb/9e30e8ed18c20800ee327a12107d5249ad8dc444867b3adfcc21654e3c71/pillow-13.0.0-cp312-cp312-manylinux_2_27_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:8bf295fe459343bc6093c8e5e0df60bfa189e0fcb775a22e2c2488fdeccc6c0d", size = 6519309, upload-time = "2026-10-16T22:15:48.14Z" }, + { url = "https://files.pythonhosted.org/packages/e6/99/fdf65cc539e266ab93546c261bdf98411fe165c6e4e13f9dbf597097192b/pillow-13.0.0-cp312-cp312-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:42d63be3af4f9b951b199b0a9d23365d37ab14687e6d77b74940432c8515c4ca", size = 7144542, upload-time = "2026-10-16T22:15:50.093Z" }, + { url = "https://files.pythonhosted.org/packages/26/9b/f38b7c4d8f599537e2d1a6d546cec9beef7536ff8f5c49a3f9c78179d8ac/pillow-13.0.0-cp312-cp312-musllinux_1_2_aarch64.whl", hash = "sha256:4bd024d567eca931b668978d672dba914d4b8e2628e2c3073a2ad178de8a2620", size = 6618042, upload-time = "2026-10-16T22:15:52.054Z" }, + { url = "https://files.pythonhosted.org/packages/a8/7c/45ee28531f89ee4582967d8a17f7a26d71ca64cd902b93c269590cd60a00/pillow-13.0.0-cp312-cp312-musllinux_1_2_x86_64.whl", hash = "sha256:23edf4ec4b6f4f240e8df6e4985e9ee2ced86474632433e55fb1e2548744bf40", size = 7228137, upload-time = "2026-10-16T22:15:54.009Z" }, + { url = "https://files.pythonhosted.org/packages/01/bc/9c2a969cb1b70201d04fa17fd5aedf02a732870b8da86bceaaa30b0ca385/pillow-13.0.0-cp312-cp312-win32.whl", hash = "sha256:02464da86f561f410e13b061ec37c3068a9cb557a095494157742d07b3c24a27", size = 6515423, upload-time = "2026-10-16T22:15:56.25Z" }, + { url = "https://files.pythonhosted.org/packages/ff/cd/73a79ecdc822076002eb24b5e1e4a530b0bb5310fcb6d239abbec877110b/pillow-13.0.0-cp312-cp312-win_amd64.whl", hash = "sha256:ff3fe9eb621fbb9db4c4be33ab946f6bac5f98ab50ca8d3260503653656c961b", size = 7272166, upload-time = "2026-10-16T22:15:58.597Z" }, + { url = "https://files.pythonhosted.org/packages/4a/a3/be5a1236efd085b3984e4305da8f179ae9dab80c991d2ad78f2bd5c4c843/pillow-13.0.0-cp312-cp312-win_arm64.whl", hash = "sha256:c3bc8ee0271387e0b1e3ad0ff893645990a656e79bf9c42f4f734a3642b11d51", size = 2698943, upload-time = "2026-10-16T22:16:00.637Z" }, + { url = "https://files.pythonhosted.org/packages/82/e2/9132d38df320e8099e2642d9fa538f18a7dfebf68a875a31725dc2a115aa/pillow-13.0.0-cp313-cp313-ios_13_0_arm64_iphoneos.whl", hash = "sha256:e98b4f91f99b24b1da72e10621ba181f7281922384a7403094debb4199de6c00", size = 4190620, upload-time = "2026-10-16T22:16:02.527Z" }, + { url = "https://files.pythonhosted.org/packages/cb/be/436dac0996876f65ef1916ffb711c5ef4af2ebc57414b6f16642e07f6b47/pillow-13.0.0-cp313-cp313-ios_13_0_arm64_iphonesimulator.whl", hash = "sha256:34fb3db1149c879343861831728c243c432d4a5afc8f1878ca7612ef8de2a033", size = 4290286, upload-time = "2026-10-16T22:16:04.861Z" }, + { url = "https://files.pythonhosted.org/packages/60/16/7c1007162e4169ea0d5bec47e2e0b2174f56c50cf5b8c02a1dc23e41037e/pillow-13.0.0-cp313-cp313-ios_13_0_x86_64_iphonesimulator.whl", hash = "sha256:2d50500a4a3bc6469adcdb319cf05d688b08ca3bf0f22a05339f3b25b433a599", size = 3723169, upload-time = "2026-10-16T22:16:06.956Z" }, + { url = "https://files.pythonhosted.org/packages/08/bc/578ab2eaa04705a8978bd9be0fe15c3d43a8fc95097c4f7e39540e79e4fa/pillow-13.0.0-cp313-cp313-macosx_10_13_x86_64.whl", hash = "sha256:ef5fe6b7db3f501c56000ba8fe7c7e1a35ca1d45824fbeb95a67daa85cff4dbe", size = 5568469, upload-time = "2026-10-16T22:16:08.664Z" }, + { url = "https://files.pythonhosted.org/packages/98/c8/ccede3dcc4f274b80fb49199eed1ef34ba05566c8b71976b9d1f111da2e3/pillow-13.0.0-cp313-cp313-macosx_11_0_arm64.whl", hash = "sha256:f11a3c1358bb3c68012a7b1f41b223c3db9f7bfde90b481df028cfe1c3363313", size = 4852187, upload-time = "2026-10-16T22:16:10.693Z" }, + { url = "https://files.pythonhosted.org/packages/fb/3c/cd0770e96d20bfd990db024bcbd43897ff3e88da1f4782cf90ea231b1d4f/pillow-13.0.0-cp313-cp313-manylinux_2_27_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:19b66ca0a4d25cc0200b381ce824c7ea090db61427315e73cfb05dbf1b35c369", size = 6515908, upload-time = "2026-10-16T22:16:12.896Z" }, + { url = "https://files.pythonhosted.org/packages/ef/fe/f5cdaa2d1c4086e757430f368c271a58ee29cfe7378e7ae7b9d964015c25/pillow-13.0.0-cp313-cp313-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:5f56499c1a74d2b862beb0c33b8016f99d095858455df9d21583bbbc1c15b582", size = 7142544, upload-time = "2026-10-16T22:16:14.844Z" }, + { url = "https://files.pythonhosted.org/packages/fb/b5/a54f6d44adbcd23737c8c12953f42be107e4cca64e247c0ca3a5b563061c/pillow-13.0.0-cp313-cp313-musllinux_1_2_aarch64.whl", hash = "sha256:a2a14cd73d5e6c927f644b0110ee81f59c4ef506259ec9183904747f2caacc81", size = 6613097, upload-time = "2026-10-16T22:16:16.893Z" }, + { url = "https://files.pythonhosted.org/packages/e6/b4/9e6c1207c21b2e91db0d15c16d30f3701ce5add1bd4da0e787761e66cb6e/pillow-13.0.0-cp313-cp313-musllinux_1_2_x86_64.whl", hash = "sha256:08b826a9bf1265c76de04a42061d61ea6b109418b7826f28ff1ecdfaa26c5502", size = 7224846, upload-time = "2026-10-16T22:16:19.195Z" }, + { url = "https://files.pythonhosted.org/packages/78/9f/d3c3cb0518f3dd1eee0ef501535f53d15597a53c5c4118d178086143397a/pillow-13.0.0-cp313-cp313-win32.whl", hash = "sha256:12b0fa15879fe5a9d9eebe66e4e2d74128a773b1a4448010c9c714f45e04b9d8", size = 6516013, upload-time = "2026-10-16T22:16:21.389Z" }, + { url = "https://files.pythonhosted.org/packages/d7/ed/1b16fb66a37e1b9e41370b93b2c614c7da761a5b2a41dc333f7af4639962/pillow-13.0.0-cp313-cp313-win_amd64.whl", hash = "sha256:9dda0788a9932776a75d8bcc020076edc308ce2508b51f6739a45e3158b3fc42", size = 7284777, upload-time = "2026-10-16T22:16:23.393Z" }, + { url = "https://files.pythonhosted.org/packages/87/7d/8d02eba9f7ecc87f8e90f731e032cc4773802f2a843763afd0c66a535ff4/pillow-13.0.0-cp313-cp313-win_arm64.whl", hash = "sha256:40a266371ece581e5de13b6635ef78d7487824af80561b6ca27694a88c19bc0f", size = 2698826, upload-time = "2026-10-16T22:16:25.382Z" }, + { url = "https://files.pythonhosted.org/packages/8e/45/fbad9c7f5e2f1745c373a434d46072bc84eddfefaa28b852425ac957fda6/pillow-13.0.0-cp314-cp314-ios_13_0_arm64_iphoneos.whl", hash = "sha256:44629d4498c9cb6b2051806155f22141df0d8a929c56134b166655232dfb2f8c", size = 4190721, upload-time = "2026-10-16T22:16:27.524Z" }, + { url = "https://files.pythonhosted.org/packages/a2/bf/2074902a259a90692c5dbf132885d21671644cbbd3f690047516243a55db/pillow-13.0.0-cp314-cp314-ios_13_0_arm64_iphonesimulator.whl", hash = "sha256:bf97ba1b5542050e9b0c74e632323b6d8136cc99d64e193abebee22fe6013c76", size = 4290236, upload-time = "2026-10-16T22:16:29.562Z" }, + { url = "https://files.pythonhosted.org/packages/ff/a5/3d1bb93b8ea38d15d21356c1b8e9f50971ac592aaa248f7fb71739094c0e/pillow-13.0.0-cp314-cp314-ios_13_0_x86_64_iphonesimulator.whl", hash = "sha256:aa694c49787040943570d61c9b8fc01271bd2a2cfdd55f0d896f8aecdce1a981", size = 3723093, upload-time = "2026-10-16T22:16:31.249Z" }, + { url = "https://files.pythonhosted.org/packages/84/a1/be060135d85187b25fa8b2573786c0e45c62ae27527fff5257675eb97f4a/pillow-13.0.0-cp314-cp314-macosx_10_15_x86_64.whl", hash = "sha256:49c12af515073e9bf511e4ef736743bf70d531a643668722279314b937742ba4", size = 5572253, upload-time = "2026-10-16T22:16:33.103Z" }, + { url = "https://files.pythonhosted.org/packages/b5/f4/0bf6f1ac9ba7ee2ffe9914aeb410a1035a0c1a3c42f122388d7fc950bc4b/pillow-13.0.0-cp314-cp314-macosx_11_0_arm64.whl", hash = "sha256:d9b0a2bee3f3e03e9f29057aedb807dbf1e1d96ee1b9d0f422d9db6c8583271d", size = 4852269, upload-time = "2026-10-16T22:16:35.102Z" }, + { url = "https://files.pythonhosted.org/packages/83/16/b603274617b3e6728da3a0133d4fba1ea8d7fbe0c4ea5fef9cbffa398b8b/pillow-13.0.0-cp314-cp314-manylinux_2_27_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:5933fcea5a5ba64c686f55c886a115cbfd92f2900460784a1e79909fdbafd7b6", size = 6516058, upload-time = "2026-10-16T22:16:37.228Z" }, + { url = "https://files.pythonhosted.org/packages/1d/f2/1371707ce438ba9a43f9d24089828494ec6f9396e28cd7903aa704dd6306/pillow-13.0.0-cp314-cp314-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:e91c2b1cbeae768c98752e67e717e283d68dae45717b41f2d91eaee648e37b7d", size = 7142574, upload-time = "2026-10-16T22:16:39.172Z" }, + { url = "https://files.pythonhosted.org/packages/b9/a9/62cf68587f910bc87c5951bcb6e9430c7643a65076adf5f18b4bf350e2bd/pillow-13.0.0-cp314-cp314-musllinux_1_2_aarch64.whl", hash = "sha256:7f6299e494e46253d40c34653cac369412f44ca53b5f85d848d3f95b1ed0546c", size = 6614295, upload-time = "2026-10-16T22:16:41.117Z" }, + { url = "https://files.pythonhosted.org/packages/6c/c9/9e9b48178649730299be0bafbf5f810a5b6977538f5a92d47e4ba5f5d608/pillow-13.0.0-cp314-cp314-musllinux_1_2_x86_64.whl", hash = "sha256:f9bd7d260d986f3553518c5d723544346fa48031f8aef3265d786aeb59845bc0", size = 7224841, upload-time = "2026-10-16T22:16:43.118Z" }, + { url = "https://files.pythonhosted.org/packages/8f/48/c44e91384b1b6c072a714a63d00608acc16a6a968735e478e4be4384e5ac/pillow-13.0.0-cp314-cp314-win32.whl", hash = "sha256:2e31c4e68cffd7b1f116127626813b82fcba19edf6b940a7b0c266d6484d0055", size = 6508694, upload-time = "2026-10-16T22:16:45.883Z" }, + { url = "https://files.pythonhosted.org/packages/50/e2/7ad17747517e4c813dca78257d6bb593466f8231baf1384bc98d92745edb/pillow-13.0.0-cp314-cp314-win_amd64.whl", hash = "sha256:f7cbb7a64e7ec664775195e4f81923bc8a93d9451b963f844e59118f63147a7a", size = 7281830, upload-time = "2026-10-16T22:16:47.725Z" }, + { url = "https://files.pythonhosted.org/packages/41/9f/e5b51f2ce8ef4563215334035403ce61a244f93a2e65c24a940131618f87/pillow-13.0.0-cp314-cp314-win_arm64.whl", hash = "sha256:a6aaa7871aebca081eb5c45f83e845bbd9712ae99fa9286dbfdcee835e63f4b4", size = 2695468, upload-time = "2026-10-16T22:16:49.515Z" }, + { url = "https://files.pythonhosted.org/packages/97/73/a53502f731f6c84805a4ce87a8881e86d07a65aee64728b900d112dc703e/pillow-13.0.0-cp314-cp314t-macosx_10_15_x86_64.whl", hash = "sha256:7b999816b5850a35ada08443e6e03945f1f6488144a83afe2d37b240a552aafa", size = 5578196, upload-time = "2026-10-16T22:16:51.291Z" }, + { url = "https://files.pythonhosted.org/packages/21/50/7e949959a56b5559c78ef172b70fc6025706acb289f3f648a15329e4fa6f/pillow-13.0.0-cp314-cp314t-macosx_11_0_arm64.whl", hash = "sha256:67871a13cf47275201553b536c025da777f997fa562a19eb798af4ad4b97c18f", size = 4857276, upload-time = "2026-10-16T22:16:53.399Z" }, + { url = "https://files.pythonhosted.org/packages/c9/52/43ded0ac08e77e39d17bd6d894162572201b5e8fdc4a022b14c55a1960a4/pillow-13.0.0-cp314-cp314t-manylinux_2_27_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:e8ca8f69b7c77fa96dad99f8638c1fc4f1c0bed4acaa1baf7a3c31bdb80a0ab1", size = 6607008, upload-time = "2026-10-16T22:16:55.525Z" }, + { url = "https://files.pythonhosted.org/packages/0d/2e/4a57ed216d5bf877387cba4d44c02aa87abe51ad2e06404aa6b597c4da39/pillow-13.0.0-cp314-cp314t-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:6bce6a29c3898191f4c9998e4d4e8f69790caeabde46e54da4bc235ce5bde683", size = 7224281, upload-time = "2026-10-16T22:16:57.738Z" }, + { url = "https://files.pythonhosted.org/packages/5c/3d/2929b248e135752d63aaeabf53f6de797e905b7b6263f9c1e2adade20aba/pillow-13.0.0-cp314-cp314t-musllinux_1_2_aarch64.whl", hash = "sha256:816752a5ef76a80484c8c441c85593b7ea9304f6231fc5776ad1fbff7e773058", size = 6695600, upload-time = "2026-10-16T22:16:59.842Z" }, + { url = "https://files.pythonhosted.org/packages/bc/33/fb0da992cf084ed1ee6ee05e96b6191e35c351eb707ca2df874feb24ce47/pillow-13.0.0-cp314-cp314t-musllinux_1_2_x86_64.whl", hash = "sha256:5bdb38ba220b61b4ec31dd1046dea6b016d3c6850a23f43ed8b58332b2fc668e", size = 7300589, upload-time = "2026-10-16T22:17:01.686Z" }, + { url = "https://files.pythonhosted.org/packages/bb/af/c30dd9ef6a3dce84ecb1b2393cac629047a32e66279056c6cb46c3ffd65c/pillow-13.0.0-cp314-cp314t-win32.whl", hash = "sha256:06903aac074d00c988e77a1fc435286dc498506d4ca8d6f69df9babddf4ecc15", size = 6517204, upload-time = "2026-10-16T22:17:03.722Z" }, + { url = "https://files.pythonhosted.org/packages/85/d5/ef3a8ae7609c8ab26570abf18194882534b480d245a9148b895a2262c004/pillow-13.0.0-cp314-cp314t-win_amd64.whl", hash = "sha256:9a635068d9793c7f0d929f39f18b58f1bd86c236e8e7b64167b86af9232ebca0", size = 7286695, upload-time = "2026-10-16T22:17:05.543Z" }, + { url = "https://files.pythonhosted.org/packages/84/3c/0861db0c69a03689ebe3e7599d35668a414deb5aaafd8255215e4e62e3d3/pillow-13.0.0-cp314-cp314t-win_arm64.whl", hash = "sha256:0d1a9cf7bc6917538b20f8d06b9ab2147e542f68849e96fc58eb66bdf6acc692", size = 2698098, upload-time = "2026-10-16T22:17:07.674Z" }, + { url = "https://files.pythonhosted.org/packages/6e/eb/e86a41962a3a31ad495a3d6a912d9002b01bba4a6fa71c28703abdca7f69/pillow-13.0.0-cp315-cp315-ios_13_0_arm64_iphoneos.whl", hash = "sha256:7d5e097eabd5d4915c27ac9f6e9325bb1e01a7c94d7348b5c4451d3fc30e8521", size = 4191213, upload-time = "2026-10-16T22:17:09.449Z" }, + { url = "https://files.pythonhosted.org/packages/dd/04/7e7445e706afec84ef06dc25b2c77f9cecff242c23730c62928ad1089d4a/pillow-13.0.0-cp315-cp315-ios_13_0_arm64_iphonesimulator.whl", hash = "sha256:51ebb529fc5e8abdd55302f6c3f8943037348ec2384329a3e55677f69ae2fdfc", size = 4290650, upload-time = "2026-10-16T22:17:11.196Z" }, + { url = "https://files.pythonhosted.org/packages/46/85/d534089068ac40c0f61f650389a296c812eab745a8a733f81feec39ae955/pillow-13.0.0-cp315-cp315-ios_13_0_x86_64_iphonesimulator.whl", hash = "sha256:c87a04e7f152ee4db5dbe15f0bfa3fe628a1aeedbe66b81c43b87f52c813f9ac", size = 3723078, upload-time = "2026-10-16T22:17:12.772Z" }, + { url = "https://files.pythonhosted.org/packages/ce/2b/00b5c3b401df538b5dbe311d3899730fae5ba1e21cb8b80a128887ac6747/pillow-13.0.0-cp315-cp315-macosx_10_15_x86_64.whl", hash = "sha256:20722cd5375533ad754053363a80b8fb30c79d37930a6f7228030e237ce3bd5d", size = 5572292, upload-time = "2026-10-16T22:17:14.539Z" }, + { url = "https://files.pythonhosted.org/packages/72/f5/a6b0610f946e9b54ae888ba7ad29d6e2f8d9b11f69e212d70416b7186c09/pillow-13.0.0-cp315-cp315-macosx_11_0_arm64.whl", hash = "sha256:04a63720f79e22fd54d897c233a68314c50ca0603360f54ce2bd5db8f0f43782", size = 4852222, upload-time = "2026-10-16T22:17:16.521Z" }, + { url = "https://files.pythonhosted.org/packages/1e/d0/c0ff25bb0427ec70f32aac995453321fc3c4494a46c7292ff02d9d260b04/pillow-13.0.0-cp315-cp315-manylinux_2_27_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:4f8db54177035a2b5f717f94475b74907e12a96dc74f6d487f3966b0a4694071", size = 6527349, upload-time = "2026-10-16T22:17:18.534Z" }, + { url = "https://files.pythonhosted.org/packages/85/11/e46fa0bcc0fc117f1eacc206b3d51f04d583fab328b4b740672d0438d83f/pillow-13.0.0-cp315-cp315-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:8f36aa77f74b14a2103622b196402c71330e4e7cf4122ee45c854e9a01e0eecf", size = 7152873, upload-time = "2026-10-16T22:17:20.415Z" }, + { url = "https://files.pythonhosted.org/packages/7c/53/2bdeca705f3a58679fa9201233193c86a30238340ca29b3a152c3fcc719b/pillow-13.0.0-cp315-cp315-musllinux_1_2_aarch64.whl", hash = "sha256:149a1c323ba366f90b48e6f1a6db6582eb50512df374588de56fc909abc0f0f3", size = 6626871, upload-time = "2026-10-16T22:17:22.361Z" }, + { url = "https://files.pythonhosted.org/packages/14/de/3829030f209722ade3753bb456b805a0e7f885cf885b483407365c82e9ca/pillow-13.0.0-cp315-cp315-musllinux_1_2_x86_64.whl", hash = "sha256:e892354a424b1e82a78ba16785822bdd40de06f19446a3d4a28047df1a4f8932", size = 7236439, upload-time = "2026-10-16T22:17:24.352Z" }, + { url = "https://files.pythonhosted.org/packages/44/b1/67fc37388382bc036a74f04da5bb6fcbf90ec3ac1f45893e06176c373fd3/pillow-13.0.0-cp315-cp315-win32.whl", hash = "sha256:8e459b613cf31f8980f584387b4fe47bf08e076e288d61aa8eba1f7fd4ddbb81", size = 6508688, upload-time = "2026-10-16T22:17:26.314Z" }, + { url = "https://files.pythonhosted.org/packages/44/05/fdb6a79aee10861165526b41b7b440e36ee1ff58613b2fbefe3acbf06d4f/pillow-13.0.0-cp315-cp315-win_amd64.whl", hash = "sha256:d4737f50d23ddd11d6685faacbb9252887dde0bc7997127757247983d74399d6", size = 7281107, upload-time = "2026-10-16T22:17:28.198Z" }, + { url = "https://files.pythonhosted.org/packages/e6/9c/6c6f7dc2a566467baf121394e83de25ac287cffe34cc030726beb4be0224/pillow-13.0.0-cp315-cp315-win_arm64.whl", hash = "sha256:2310e2adfc13c6d457d7d0f2ffdd5c56a74a317650cf765887df8d515a647ac0", size = 2695513, upload-time = "2026-10-16T22:17:30.013Z" }, + { url = "https://files.pythonhosted.org/packages/23/16/e9120defdb8bff693b582d035312acc6f530cdc2bc5238a87f410efccd82/pillow-13.0.0-cp315-cp315t-macosx_10_15_x86_64.whl", hash = "sha256:21c32061a5512dc13f87cc7303e120e4d77c2f70df89a35af0e3ce246e5ac57f", size = 5578129, upload-time = "2026-10-16T22:17:31.642Z" }, + { url = "https://files.pythonhosted.org/packages/47/22/76faf2bb2448f48a84cf894c1e2bb1090d7d2b1aa521ece131cb2f05d631/pillow-13.0.0-cp315-cp315t-macosx_11_0_arm64.whl", hash = "sha256:1c774b9c0ba0d0b356f71530677b39028878a2cae5a93e0a13779cfd17120f88", size = 4857428, upload-time = "2026-10-16T22:17:33.43Z" }, + { url = "https://files.pythonhosted.org/packages/b3/64/0861ea4e03c50b14af9ed9dd8b9f388ab42f445c623a9fcbab19aef7aa55/pillow-13.0.0-cp315-cp315t-manylinux_2_27_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:20822140c9a2ba9c5e42df342b0a892ab2e102fc0e3db967a133bb9ce3f898bb", size = 6612582, upload-time = "2026-10-16T22:17:35.415Z" }, + { url = "https://files.pythonhosted.org/packages/5c/20/cd1e9b3f60a9f3d5ebe46da36955c26004092a501f7f2c42f9fcc26611bc/pillow-13.0.0-cp315-cp315t-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:ffea6b2986a797c84b6d1d27d4f43e0b83c73fc16281a930891cd237cf0d08fe", size = 7230176, upload-time = "2026-10-16T22:17:37.362Z" }, + { url = "https://files.pythonhosted.org/packages/b6/ef/ffb0a2fc85690714fb7ac662956949645d57f15bf4426f520da361e7b6e3/pillow-13.0.0-cp315-cp315t-musllinux_1_2_aarch64.whl", hash = "sha256:05cd2cd39055d0b9650060e4513ef4a9a0aea870083950bd046a0d6490f93a22", size = 6703640, upload-time = "2026-10-16T22:17:39.313Z" }, + { url = "https://files.pythonhosted.org/packages/e0/c2/6402ee3ccb3363aeed8fdb3f9fe56703ac5377a539f0b81b53999d7e121f/pillow-13.0.0-cp315-cp315t-musllinux_1_2_x86_64.whl", hash = "sha256:c6fd031333dedcad7ce3e03d244093f43504f209e86925ebdb2b868fa4037630", size = 7309231, upload-time = "2026-10-16T22:17:41.021Z" }, + { url = "https://files.pythonhosted.org/packages/fc/96/d6b82e8b8c06ad143b66b12e0b31051e4bf489b4fabca1f390a14fe1ebde/pillow-13.0.0-cp315-cp315t-win32.whl", hash = "sha256:c50c80e3591e5cf9a8dfa6007d4f10732f02e32263850999286e56ccd650cfb2", size = 6517200, upload-time = "2026-10-16T22:17:42.886Z" }, + { url = "https://files.pythonhosted.org/packages/60/d4/0062e6005e9b270b165015aafd343364ac378ec5a685431ea42baad33357/pillow-13.0.0-cp315-cp315t-win_amd64.whl", hash = "sha256:adb3c2124c68e480f4632d06cb7caa3b48a51678475f483b4edb1c6dee29ea43", size = 7287488, upload-time = "2026-10-16T22:17:45.56Z" }, + { url = "https://files.pythonhosted.org/packages/e4/81/b8eb349746be7bce47dd95337c6634e84910af91b59aaa9159c034c614fa/pillow-13.0.0-cp315-cp315t-win_arm64.whl", hash = "sha256:4a25756244bdcce3b50580baadf320fa48297f0eeb8e114ba64139a40d4d0fc8", size = 2698073, upload-time = "2026-10-16T22:17:47.556Z" }, + { url = "https://files.pythonhosted.org/packages/68/c0/fb7b1caf25ed5ac5d46116f09b8b0706b9559833c65c667cecb1a53153f2/pillow-13.0.0-pp311-pypy311_pp80-macosx_10_15_x86_64.whl", hash = "sha256:405be6058e206a86fedce2f715ecc2343f94072d7925251632dffa2c05f70dca", size = 5521630, upload-time = "2026-10-16T22:17:50.008Z" }, + { url = "https://files.pythonhosted.org/packages/1a/e6/4d11d5b1137f4fc080605cce410cfaae4f303ad400bdec75d27d3083fa29/pillow-13.0.0-pp311-pypy311_pp80-macosx_11_0_arm64.whl", hash = "sha256:b84be41387fc00db28c13c02e30d42b02d8d8f6a4a741a1aee6012ca7f1f6039", size = 4806023, upload-time = "2026-10-16T22:17:51.932Z" }, + { url = "https://files.pythonhosted.org/packages/f1/45/0923202a658e8ff6aafd72c1a4395ebadcdebaa182c437d8595d3dd3db32/pillow-13.0.0-pp311-pypy311_pp80-manylinux_2_27_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:a192360d512305fe92d3053e4f7627bbad1f7708f1f4ad38cccd9336acabe9ad", size = 5464831, upload-time = "2026-10-16T22:17:53.742Z" }, + { url = "https://files.pythonhosted.org/packages/07/52/d3e38f0cad1dadea2a658b9a7d6ba8a04048f5ae14f011b806e5c3bd6bcb/pillow-13.0.0-pp311-pypy311_pp80-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:83b37abd87b590acd8f47ca8a61f70893322b11834d36e07c2298dacad4d14d2", size = 6043133, upload-time = "2026-10-16T22:17:55.625Z" }, + { url = "https://files.pythonhosted.org/packages/c4/f9/2ed7685980c1d64f8131a7d87a19c369424747577152622b73ddeff163b7/pillow-13.0.0-pp311-pypy311_pp80-win_amd64.whl", hash = "sha256:73ee25cf8cf8b75d5a5aeed7cb6588f5e5dbfbb0fde51707b630bbb23897645c", size = 7365208, upload-time = "2026-10-16T22:17:57.509Z" }, + { url = "https://files.pythonhosted.org/packages/60/da/f6d644e631c5b370c5d0b899f3ccadd2780aab070b29a110527cf4e1fd2c/pillow-13.0.0-pp312-pypy312_pp80-macosx_10_15_x86_64.whl", hash = "sha256:8619685cad8efc4c38c66d3b36aad854523297c43d748b3b6f0cc03c90ba3463", size = 5523188, upload-time = "2026-10-16T22:17:59.334Z" }, + { url = "https://files.pythonhosted.org/packages/e4/b0/2594f202d743a35d4fe80682bfd96f277e5ca7aecb27cf64ef5bfc1d000b/pillow-13.0.0-pp312-pypy312_pp80-macosx_11_0_arm64.whl", hash = "sha256:7e91e97a929232cdb828889ee6ad0e48d7f40f4ec38818f4e4a8968b2b1af22e", size = 4806808, upload-time = "2026-10-16T22:18:01.188Z" }, + { url = "https://files.pythonhosted.org/packages/8e/ed/d504165ef79af6fafe25c76726cf30f544cc236002f7ed054f1d51adc4ff/pillow-13.0.0-pp312-pypy312_pp80-manylinux_2_27_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:93ff9c9c3663ce23aaa5d0b6b6abac55359e2a0c06c7e30f3771829af816be63", size = 5464358, upload-time = "2026-10-16T22:18:02.779Z" }, + { url = "https://files.pythonhosted.org/packages/7b/fe/a10aba5503ca13e9e009b8bd0a1a4ce5584bdce246245fd6f3a83701369c/pillow-13.0.0-pp312-pypy312_pp80-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:e65dd6e75de67b2362a4fa3e16aa627595f65054dc4ec73d64680c84d592fa61", size = 6043514, upload-time = "2026-10-16T22:18:04.538Z" }, + { url = "https://files.pythonhosted.org/packages/c0/37/66b12dc476041fe523bca29c9d58fbf1ea92a4eaf73acfb9e993948f1191/pillow-13.0.0-pp312-pypy312_pp80-win_amd64.whl", hash = "sha256:567f20f1c75897d7663830da6ea8af719e03d712b1d9ee7f813319f9ece42e5a", size = 7366030, upload-time = "2026-10-16T22:18:06.638Z" }, +] + +[[package]] +name = "pluggy" +version = "1.7.0" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/bf/db/7fc19e6f2dc92a966727031389fc2e08b558f0f25eb7403c1119ad4713cd/pluggy-1.7.0.tar.gz", hash = "sha256:d1eaa46ebb595891b860ab086b4d09c8588af65ebd4361b8e8f4bb8920b90ba8", size = 123304, upload-time = "2026-10-15T09:50:58.343Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/40/9e/2b38731e0fc536806f16490e1a12d7f0dc2a1235aa8cc07bcc75416a7daa/pluggy-1.7.0-py3-none-any.whl", hash = "sha256:7dd7b0d8832ba3cb632c306926ded123429211b83641b35dc5c41ad2d34f9bec", size = 27082, upload-time = "2026-10-15T09:50:56.808Z" }, +] + +[[package]] +name = "pyarrow" +version = "26.0.0" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/ec/34/17c34cb38e5d940e38f0f0d9fdfa0e8a506676409ea9b85aff7e3079f831/pyarrow-26.0.0.tar.gz", hash = "sha256:0cccd36e00ea3afeb52ded61f2721ce71f604853d70c45365c58324eb773d6ae", size = 1239433, upload-time = "2026-10-09T08:26:25.315Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/07/68/e0707097cee93be7f693e7e89495fabfeb8bf95ee30619063f8b30fffc29/pyarrow-26.0.0-cp311-cp311-macosx_12_0_arm64.whl", hash = "sha256:fcdd1e04982637c6042337d3e24d472f938f01fdc502e2b994844b726d12c3f4", size = 36370896, upload-time = "2026-10-09T08:13:28.874Z" }, + { url = "https://files.pythonhosted.org/packages/5c/f0/591211c00612aef83236daff1620412b24aeb07c646de08c18a8a6c95a39/pyarrow-26.0.0-cp311-cp311-macosx_12_0_x86_64.whl", hash = "sha256:f800e9e722c145ccd18012d82a864cb21bfee4ba4ceffde77100d25eced511a9", size = 38709806, upload-time = "2026-10-09T08:13:33.417Z" }, + { url = "https://files.pythonhosted.org/packages/50/ea/9b035a9d1556e06e64ea86169d9a985d0fc092d427ac5edbb3af7183289c/pyarrow-26.0.0-cp311-cp311-manylinux_2_28_aarch64.whl", hash = "sha256:7aa12ab8e236789b1ecd2d6ecaef036b4e63d675ddf1864a43c6799d18f2d028", size = 50885975, upload-time = "2026-10-09T08:13:37.737Z" }, + { url = "https://files.pythonhosted.org/packages/e1/81/8e685683897a6d3d5887c3e2fd24f3c14bc5d6d6bb3a2387484e665c580e/pyarrow-26.0.0-cp311-cp311-manylinux_2_28_x86_64.whl", hash = "sha256:6e89dee53aaeb50505ed6152ea55bc7ddfd4f4df264f5427ea255288d8f0e580", size = 53904793, upload-time = "2026-10-09T08:13:42.984Z" }, + { url = "https://files.pythonhosted.org/packages/9a/ad/d474a0b1b00110f3a879aa5df654f857c81929a32b2a4222869240de5220/pyarrow-26.0.0-cp311-cp311-musllinux_1_2_aarch64.whl", hash = "sha256:f1c1b4263fd13abbc339a16f2bf19f3a5cbf2a620853d812b1256f03c5342cb8", size = 54458010, upload-time = "2026-10-09T08:13:47.778Z" }, + { url = "https://files.pythonhosted.org/packages/d4/86/2c2861e905810c59fed4d98c85b994c21e8613730c5c3b436781d89110f2/pyarrow-26.0.0-cp311-cp311-musllinux_1_2_x86_64.whl", hash = "sha256:ff1e816af7abff71f289242e109217036723ce36aca74ad6691e52d964a74afa", size = 57368406, upload-time = "2026-10-09T08:13:52.651Z" }, + { url = "https://files.pythonhosted.org/packages/0e/02/823e606633c15155bb965c7a0f3750c4f20dd47c4ab48213c7693df0e0ba/pyarrow-26.0.0-cp311-cp311-win_amd64.whl", hash = "sha256:13b0972a3dc71b642050d1bc72664a3916e14f59c943d8c1368154d6e4b0c2d5", size = 28522657, upload-time = "2026-10-09T08:13:56.513Z" }, + { url = "https://files.pythonhosted.org/packages/b3/60/6793778f2617cce469383dac0ba08c4f2401cf342df0c7b9ca53939d9b46/pyarrow-26.0.0-cp312-cp312-macosx_12_0_arm64.whl", hash = "sha256:90ddaf7c625307ad52f31a9b25c34fe5e4897c7529ee3481135822b2b6842ff1", size = 36333953, upload-time = "2026-10-09T08:14:00.387Z" }, + { url = "https://files.pythonhosted.org/packages/db/81/f944cc63ce8a753e5fbff25de6d1d475ebd7fffdf9cf98c65130294fc896/pyarrow-26.0.0-cp312-cp312-macosx_12_0_x86_64.whl", hash = "sha256:ee341973f78a0b46e073d065e88e75026a9c584051e97f98a0d05d96c6bac7dd", size = 38688456, upload-time = "2026-10-09T08:14:04.344Z" }, + { url = "https://files.pythonhosted.org/packages/f5/2d/7e5c722fa5d5d9f3b75e62fe11694b34217664d4f05ac88031197166b277/pyarrow-26.0.0-cp312-cp312-manylinux_2_28_aarch64.whl", hash = "sha256:01c863a18bd9c8412453dd0d92de6d0ee7b2b3d6fb079d9734a4b2a3c8bd4453", size = 50867603, upload-time = "2026-10-09T08:14:09.115Z" }, + { url = "https://files.pythonhosted.org/packages/88/e4/9cd356d906e71bd79b0c3fc5c9a54e01a0020dcf14c152ccfbcb503c7298/pyarrow-26.0.0-cp312-cp312-manylinux_2_28_x86_64.whl", hash = "sha256:6a628922ba20705fa964ca73e4ef959c2fb2f14b9bbec5589a6a1e68e6257c85", size = 53931932, upload-time = "2026-10-09T08:14:24.051Z" }, + { url = "https://files.pythonhosted.org/packages/bb/e4/5bae3133b7fe04c24907a20f3bc1fba388cbbde659199e7b76445982047a/pyarrow-26.0.0-cp312-cp312-musllinux_1_2_aarch64.whl", hash = "sha256:954d971b363b16ee41f89389a4053315dc71265f2ce5c2468eb0a910b1166268", size = 54444720, upload-time = "2026-10-09T08:14:31.214Z" }, + { url = "https://files.pythonhosted.org/packages/ba/b4/ee422493bb6dafdbef776cfe2c2a73106a1063a79bf4e78d1e5f51176885/pyarrow-26.0.0-cp312-cp312-musllinux_1_2_x86_64.whl", hash = "sha256:5d5768d03426abe6526d5274adefa00abf00a7f81118c46e98b5a46390f5549e", size = 57388949, upload-time = "2026-10-09T08:14:38.964Z" }, + { url = "https://files.pythonhosted.org/packages/54/3c/1783aab1dac28e175dcf26dfc7123725efc474caecaed91e8a34cb89cad0/pyarrow-26.0.0-cp312-cp312-win_amd64.whl", hash = "sha256:cc903e1069e9dd5e9dcf780324c0112e27e051e422ecfaff574fb33ed65d9160", size = 28567581, upload-time = "2026-10-09T08:14:44.279Z" }, + { url = "https://files.pythonhosted.org/packages/4d/35/ca95493712af97c46a312945c8e9d16b21c5fe2f148be5466168d0290505/pyarrow-26.0.0-cp313-cp313-macosx_12_0_arm64.whl", hash = "sha256:a6ca849f90cf73fe361f08a5762c783ead9671e4548c1f558cc637b54c9103f2", size = 36336700, upload-time = "2026-10-09T08:14:51.399Z" }, + { url = "https://files.pythonhosted.org/packages/69/ef/b1a675f79c9babfd4fcd99af62141d3c2d1a78a524e311b0c6b80110445a/pyarrow-26.0.0-cp313-cp313-macosx_12_0_x86_64.whl", hash = "sha256:c2ba350957076b1b3a22f549261dc3e9c67ca20816d8bd5f79d7b9c69be4c4c2", size = 38698502, upload-time = "2026-10-09T08:14:57.114Z" }, + { url = "https://files.pythonhosted.org/packages/3b/7c/cea852a832a327a8de797b3a68e5c25ce0f5aa1d20503807671bd90ec642/pyarrow-26.0.0-cp313-cp313-manylinux_2_28_aarch64.whl", hash = "sha256:e3b190ba1d3d22a5a8758597f797111b77d433473744352a184a5ee0a42d672e", size = 50865064, upload-time = "2026-10-09T08:20:01.614Z" }, + { url = "https://files.pythonhosted.org/packages/4f/d6/e95834b29360092376fe4da9956ba41bb7b021869efe6ee9d4172d05cb15/pyarrow-26.0.0-cp313-cp313-manylinux_2_28_x86_64.whl", hash = "sha256:240bd18a7487f8767616a948a69dd4e740a8bc36a1c9da49e4dc9a32c5c2faed", size = 53926722, upload-time = "2026-10-09T08:23:10.829Z" }, + { url = "https://files.pythonhosted.org/packages/e0/7f/98257444e2aea2e1fddceee3af3bd2077236d550428413f80393bd1f888d/pyarrow-26.0.0-cp313-cp313-musllinux_1_2_aarch64.whl", hash = "sha256:2b5fcd69c0e1107b79e55839877db5a6ed04651b73fd6fec581d09e230bed5e4", size = 54443093, upload-time = "2026-10-09T08:23:16.971Z" }, + { url = "https://files.pythonhosted.org/packages/88/ca/dac99cfb25cfa62bf7194600cc99abc14a6bd2af50d7fdb7f15eeaf6e202/pyarrow-26.0.0-cp313-cp313-musllinux_1_2_x86_64.whl", hash = "sha256:f7444ea6975c49a857c68f9bd8fa11acae96dede63d120ffb3bf0a603ea82516", size = 57381937, upload-time = "2026-10-09T08:23:24.95Z" }, + { url = "https://files.pythonhosted.org/packages/c0/ed/138d29fddaf803b90f4527e124bb6aaddc18aaf4a6c50fd0a5f577c94989/pyarrow-26.0.0-cp313-cp313-win_amd64.whl", hash = "sha256:3de30a7432b48b98b9decbd9e25a53bb9251d202c2e6c5a29a50869592ccb117", size = 28478571, upload-time = "2026-10-09T08:23:30.535Z" }, + { url = "https://files.pythonhosted.org/packages/8c/32/01858422a37f083911c2bb4d15cc32c5eeaa9d9b2bf5ddedee995a7146a6/pyarrow-26.0.0-cp314-cp314-macosx_12_0_arm64.whl", hash = "sha256:5780d487ff6c6ed7b42298609680d87fe0036e529a9dc2e1105364bce9697f50", size = 36378402, upload-time = "2026-10-09T08:23:36.537Z" }, + { url = "https://files.pythonhosted.org/packages/00/85/f6b5976c2878b752d0804d371684e0495a71de296b6dc6559e6fbaa4311a/pyarrow-26.0.0-cp314-cp314-macosx_12_0_x86_64.whl", hash = "sha256:a0e4e92eeb088f1d7c2c04d6c7de8434c75abb4b4ccf0bbcd045aa7164c68d93", size = 38733074, upload-time = "2026-10-09T08:23:42.873Z" }, + { url = "https://files.pythonhosted.org/packages/81/bc/c90fcbbcf893631e23dab1b0fb3fa29a508a8614326571b03c0894eda00b/pyarrow-26.0.0-cp314-cp314-manylinux_2_28_aarch64.whl", hash = "sha256:eaf9e7cc7ab59f6c760232bbde18f64d559bbc50544841303bfb32be53533297", size = 50929201, upload-time = "2026-10-09T08:23:50.507Z" }, + { url = "https://files.pythonhosted.org/packages/ec/c1/0c1ff38ab7df1b2cf54cf0ad9f19a516c4e416c6c9b4c966cc2c9d587f77/pyarrow-26.0.0-cp314-cp314-manylinux_2_28_x86_64.whl", hash = "sha256:ab6914db225d7f399652ae1f08588dfbc9efe617612715701e3d9d5cfa5ca19f", size = 53951865, upload-time = "2026-10-09T08:23:57.692Z" }, + { url = "https://files.pythonhosted.org/packages/9f/70/6a6b170496925472adad45a32528770fc8632db35fc60d4edd1e9ce1be0b/pyarrow-26.0.0-cp314-cp314-musllinux_1_2_aarch64.whl", hash = "sha256:41dd3661ef40790a78870052ad7a58ad827b27c67a4511f06962eb9e9b74d19b", size = 54496388, upload-time = "2026-10-09T08:24:05.23Z" }, + { url = "https://files.pythonhosted.org/packages/a8/32/033ef9dba80976820190e292a10a5a23e9406572b76bbeb4d685d90e5c8d/pyarrow-26.0.0-cp314-cp314-musllinux_1_2_x86_64.whl", hash = "sha256:6e949744dcfc2d379808f7013c5f9cafaf0f817656dff7d46c6931528dd1784b", size = 57411588, upload-time = "2026-10-09T08:24:12.043Z" }, + { url = "https://files.pythonhosted.org/packages/1e/ff/a74892c50aaf1f9f744a84493e08a2f99221e77c39d2d4a926de21a99edf/pyarrow-26.0.0-cp314-cp314-win_amd64.whl", hash = "sha256:4a5fa8dc70dd50808990ff36faf44088e357b353d86c7682dd92d4b78d4c97d5", size = 29237858, upload-time = "2026-10-09T08:24:58.106Z" }, + { url = "https://files.pythonhosted.org/packages/03/10/f0ee0976ef08a851a743c57608917ac9a47623f688b9ee0efe5429975ba1/pyarrow-26.0.0-cp314-cp314t-macosx_12_0_arm64.whl", hash = "sha256:e2a1856e9565fe2679863b372478c681806aebbf7d0a6e72f33e77f804e647d6", size = 36495870, upload-time = "2026-10-09T08:24:16.479Z" }, + { url = "https://files.pythonhosted.org/packages/27/ca/0bc431a509bf10b4472dbb94f4184752ecbbddeb7f467152dac0fdaed469/pyarrow-26.0.0-cp314-cp314t-macosx_12_0_x86_64.whl", hash = "sha256:4bcba83299cb2b8f8e443d36c6ba6269a5034431879015fb0719495df8a14de2", size = 38819754, upload-time = "2026-10-09T08:24:20.875Z" }, + { url = "https://files.pythonhosted.org/packages/61/59/2be41d26af7a07fb71581fb753cae396403ba1a2978355fd553929d44a9a/pyarrow-26.0.0-cp314-cp314t-manylinux_2_28_aarch64.whl", hash = "sha256:3a4d235876f14b4136b4d616ec42eb469ea0d6ead336cae631aa1dd29b21c962", size = 50933671, upload-time = "2026-10-09T08:24:27.199Z" }, + { url = "https://files.pythonhosted.org/packages/4b/cb/b6d5048cf3178be9678f5c9c60040199894b2f69c3439c87ced91fd24da9/pyarrow-26.0.0-cp314-cp314t-manylinux_2_28_x86_64.whl", hash = "sha256:210cc9b83888b87cdc8f793eebb264f22b20d0dedbedefc73b9687a7047b4747", size = 53906419, upload-time = "2026-10-09T08:24:33.536Z" }, + { url = "https://files.pythonhosted.org/packages/09/2b/23e30fbd776c81d18d134d2592eb60daca13e8a57ab087d0fa042f9d9f3d/pyarrow-26.0.0-cp314-cp314t-musllinux_1_2_aarch64.whl", hash = "sha256:ca77c43ca55bfc9a4eeb1f0cd5f093f08731b77c24cdba0829035f084959b0bb", size = 54527960, upload-time = "2026-10-09T08:24:41.292Z" }, + { url = "https://files.pythonhosted.org/packages/e2/23/fce251cd6b0546dfc181b00d5c8ef1c95a8c4cae83266bc3dfd5f719c62c/pyarrow-26.0.0-cp314-cp314t-musllinux_1_2_x86_64.whl", hash = "sha256:290a74c48e9491b436fd5edacfadf357943f82aa45c81110bd83a69aab33d1cf", size = 57388010, upload-time = "2026-10-09T08:24:48.186Z" }, + { url = "https://files.pythonhosted.org/packages/44/a5/0126fb0ef8d59bf257bdd68bb41623b72afc6e81790a0b4ac863a0f58861/pyarrow-26.0.0-cp314-cp314t-win_amd64.whl", hash = "sha256:515a10dae2a1d236bc9c9209d0317acb6746ea63cd4f98704904af7156d90ed1", size = 29406123, upload-time = "2026-10-09T08:24:53.387Z" }, + { url = "https://files.pythonhosted.org/packages/ed/66/8ada1b5165359d84b4b9b5384742304d1081da670f77d458fd9c9b8a2161/pyarrow-26.0.0-cp315-cp315-macosx_12_0_arm64.whl", hash = "sha256:e890816e5ee89c74a0f8b9379fe8b5ba83f46132b2a0bbb9b1c21359ec30dfda", size = 36373215, upload-time = "2026-10-09T08:25:03.067Z" }, + { url = "https://files.pythonhosted.org/packages/c4/83/74f10c3d803a6834b2acab21847724d4bdbc74d246eb17321432844707f3/pyarrow-26.0.0-cp315-cp315-macosx_12_0_x86_64.whl", hash = "sha256:9db18a9dc0af52135c9eac549d80a7a882696efbe5406cf882b044525d4ecc2e", size = 38730866, upload-time = "2026-10-09T08:25:07.924Z" }, + { url = "https://files.pythonhosted.org/packages/e2/5a/ea2fa2163b1bd8ff73efd39c4060be63fd6ddec03e7887a471acd1e042a4/pyarrow-26.0.0-cp315-cp315-manylinux_2_28_aarch64.whl", hash = "sha256:734312d3d99088d9ec28c5b17bad40389bd8373a1afc10acb60b83fd217af087", size = 50924443, upload-time = "2026-10-09T08:25:13.864Z" }, + { url = "https://files.pythonhosted.org/packages/78/80/8c47b6cf8cfd42826df65193eff026c1cc81fa6cb213a3c3f5d203e6f67a/pyarrow-26.0.0-cp315-cp315-manylinux_2_28_x86_64.whl", hash = "sha256:24f892fdf1ae1942d69d3f7742e2f49960ec95277cfb1a70b8a1d91f4a96d935", size = 53948540, upload-time = "2026-10-09T08:25:19.305Z" }, + { url = "https://files.pythonhosted.org/packages/69/1f/3a506a76d944ec5c5e4b7f01d8d0446b392a6fb384de627a12e503f616b4/pyarrow-26.0.0-cp315-cp315-musllinux_1_2_aarch64.whl", hash = "sha256:879331ddea2a26479fa18fade71e6facf684a6cf19f67daec3775c871569e8e5", size = 54494863, upload-time = "2026-10-09T08:25:24.517Z" }, + { url = "https://files.pythonhosted.org/packages/3d/50/08c4bb04d651788d2eaca78065743f4f6ded974d4ef96ae3c473993e9d0c/pyarrow-26.0.0-cp315-cp315-musllinux_1_2_x86_64.whl", hash = "sha256:5b827650e874f1f9f9392524ea3e9e3e8a245de5ba64acca1f81ab188090afb9", size = 57409877, upload-time = "2026-10-09T08:25:31.157Z" }, + { url = "https://files.pythonhosted.org/packages/d4/f3/c64781fbd7b6d3c07993b698c14944d0d195f07e800fa931c486ae6ab36a/pyarrow-26.0.0-cp315-cp315-win_amd64.whl", hash = "sha256:8e8e28c464552b5ca03e30d4504168c4425ce383884f8611b00e972f9fd933fc", size = 29236658, upload-time = "2026-10-09T08:26:22.607Z" }, + { url = "https://files.pythonhosted.org/packages/06/55/2ee3729daea999f19f061f03898d4895a242c4cd94f26e1324e5fdfbfe10/pyarrow-26.0.0-cp315-cp315t-macosx_12_0_arm64.whl", hash = "sha256:ce28748cbeb0f29c3ce9603782979c7117580fc76f16aa3ca448b38a22281adb", size = 36489011, upload-time = "2026-10-09T08:25:37.64Z" }, + { url = "https://files.pythonhosted.org/packages/6a/7d/3eb17f601f2bf13eda5f2ed28956379ca628b4dda97619cbb1cb1721622d/pyarrow-26.0.0-cp315-cp315t-macosx_12_0_x86_64.whl", hash = "sha256:106bb9290fc6fd9a84138a9440038ef184bac86463543c5ff099229cb30d996c", size = 38808480, upload-time = "2026-10-09T08:25:43.579Z" }, + { url = "https://files.pythonhosted.org/packages/0e/e3/f0047360b0f4bfc031b256dc0aec3837a61f245b2fb70f8363438e2db665/pyarrow-26.0.0-cp315-cp315t-manylinux_2_28_aarch64.whl", hash = "sha256:2e4a413046eba9896e632925066c74095182200ba32e19ff0166bf64d2f936ac", size = 50923273, upload-time = "2026-10-09T08:25:51.445Z" }, + { url = "https://files.pythonhosted.org/packages/38/d9/56d9fb91210407df31cbeb9b91138601c88c7c8fb5f6bf773b20d65509bf/pyarrow-26.0.0-cp315-cp315t-manylinux_2_28_x86_64.whl", hash = "sha256:d58798c4d8d629700058e9afc1e16b9801023f3ce4dc1c92d945e79b5ffe4e98", size = 53900905, upload-time = "2026-10-09T08:25:59.554Z" }, + { url = "https://files.pythonhosted.org/packages/cf/40/8e8a7e9e027c731520c7eb179dd00a153b76ebf0bc11d213c6c8f8502851/pyarrow-26.0.0-cp315-cp315t-musllinux_1_2_aarch64.whl", hash = "sha256:645917e976671debabf854abab6e2b75c571ca4f82adc33a2d338697f7c27d93", size = 54518345, upload-time = "2026-10-09T08:26:07.125Z" }, + { url = "https://files.pythonhosted.org/packages/be/89/1e768a3fdb88d34e708ad2dc00dbf8e4e30290784eb84198d59308963bea/pyarrow-26.0.0-cp315-cp315t-musllinux_1_2_x86_64.whl", hash = "sha256:7c3fda041e7078802589cf257750323ee3d0cd1e56e53a9b20ec845697fb3d28", size = 57379403, upload-time = "2026-10-09T08:26:13.624Z" }, + { url = "https://files.pythonhosted.org/packages/96/be/7b81a44d6a8e70581dcc1d6f01541f9000a973b1e5d75394aec91e7b179a/pyarrow-26.0.0-cp315-cp315t-win_amd64.whl", hash = "sha256:68cd662e9e2b00876a131950cf32336ace2d0865e1f9418763e3d3be8481dfa4", size = 29389953, upload-time = "2026-10-09T08:26:18.277Z" }, +] + +[[package]] +name = "pygments" +version = "2.21.0" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/49/2e/ced460408999b33da6b31b0021b0f37d329e202d4169aeb164493778f25b/pygments-2.21.0.tar.gz", hash = "sha256:610ca751c9bc2492b38eb9a38a7fbc93edbbb2d7182edaf34e66ae493dee5c8c", size = 5005329, upload-time = "2026-08-17T08:02:48.824Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/71/46/17f022dd3e953bf20a04a028a21ec746d942f8d2af30fa0f124fa0e6a684/pygments-2.21.0-py3-none-any.whl", hash = "sha256:2363c69b61c4a97c838da3b130dcd6468f4848992b21a82f2a63ec34377137d9", size = 1250147, upload-time = "2026-08-17T08:02:44.912Z" }, +] + +[[package]] +name = "pyparsing" +version = "3.3.3" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/e4/11/b213bebff182584360cb8d17c72c1677fec5c5c228de439e63bcf8ab1c8f/pyparsing-3.3.3.tar.gz", hash = "sha256:928ae7e20211f3b6f3915a72f06a0cfd29ab9d24279dd6346b6b1a7146397d36", size = 1050487, upload-time = "2026-09-20T20:59:05.609Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/38/bb/d215ee7c73b61497b28a5503f9f53523f294fcc936762b7caf90e0c1c2b5/pyparsing-3.3.3-py3-none-any.whl", hash = "sha256:ece8c00a69cf01b45d0b1dedabb469c90d8caf996d4fda40f147627a122849a4", size = 126420, upload-time = "2026-09-20T20:59:04.025Z" }, +] + +[[package]] +name = "pytest" +version = "9.2.0" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "colorama", marker = "sys_platform == 'win32'" }, + { name = "iniconfig" }, + { name = "packaging" }, + { name = "pluggy" }, + { name = "pygments" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/79/42/bb4641652e58dedeac2ec51abc1f8cae4fee4ec63899a0999016229a0f2a/pytest-9.2.0.tar.gz", hash = "sha256:97c4f3b1f3c7414aa36465e09b8220f1d03627993bd51b1282e193753c80557b", size = 1797555, upload-time = "2026-10-27T21:30:20.728Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/51/39/1f5d8c8877a0fa327db959c19341d4be61660de1ab6b7391f9eb805bcd67/pytest-9.2.0-py3-none-any.whl", hash = "sha256:07acdcf88ec174f907dc01af70bf52d49b014bbab03faeeade31c96b18c2e588", size = 411776, upload-time = "2026-10-27T21:30:19.131Z" }, +] + +[[package]] +name = "python-dateutil" +version = "3.0.0" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "six" }, + { name = "tzdata" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/0e/df/f15f80383f152452d8985b36d95cb8751f901a3e0ff4cd27f7e118f168e2/python_dateutil-3.0.0.tar.gz", hash = "sha256:7ebcb3c29a92f3ceafe79a7a48a5c45677e9e004bbeec5742ac0bbb00560b7a2", size = 271066, upload-time = "2026-10-21T23:41:45.075Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/5a/17/cebee01d1c0c1441174cf0456ddc91b280d79acb65340c05e41d77866ee8/python_dateutil-3.0.0-py2.py3-none-any.whl", hash = "sha256:3c79630202e4152c6add678446984236efe4f57e84d8e892aab0637bab82e9fd", size = 94659, upload-time = "2026-10-21T23:41:43.431Z" }, +] + +[[package]] +name = "pyyaml" +version = "6.0.3" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/05/8e/961c0007c59b8dd7729d542c61a4d537767a59645b82a0b521206e1e25c2/pyyaml-6.0.3.tar.gz", hash = "sha256:d76623373421df22fb4cf8817020cbb7ef15c725b9d5e45f17e189bfc384190f", size = 130960, upload-time = "2025-09-25T21:33:16.546Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/6d/16/a95b6757765b7b031c9374925bb718d55e0a9ba8a1b6a12d25962ea44347/pyyaml-6.0.3-cp311-cp311-macosx_10_13_x86_64.whl", hash = "sha256:44edc647873928551a01e7a563d7452ccdebee747728c1080d881d68af7b997e", size = 185826, upload-time = "2025-09-25T21:31:58.655Z" }, + { url = "https://files.pythonhosted.org/packages/16/19/13de8e4377ed53079ee996e1ab0a9c33ec2faf808a4647b7b4c0d46dd239/pyyaml-6.0.3-cp311-cp311-macosx_11_0_arm64.whl", hash = "sha256:652cb6edd41e718550aad172851962662ff2681490a8a711af6a4d288dd96824", size = 175577, upload-time = "2025-09-25T21:32:00.088Z" }, + { url = "https://files.pythonhosted.org/packages/0c/62/d2eb46264d4b157dae1275b573017abec435397aa59cbcdab6fc978a8af4/pyyaml-6.0.3-cp311-cp311-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:10892704fc220243f5305762e276552a0395f7beb4dbf9b14ec8fd43b57f126c", size = 775556, upload-time = "2025-09-25T21:32:01.31Z" }, + { url = "https://files.pythonhosted.org/packages/10/cb/16c3f2cf3266edd25aaa00d6c4350381c8b012ed6f5276675b9eba8d9ff4/pyyaml-6.0.3-cp311-cp311-manylinux2014_s390x.manylinux_2_17_s390x.manylinux_2_28_s390x.whl", hash = "sha256:850774a7879607d3a6f50d36d04f00ee69e7fc816450e5f7e58d7f17f1ae5c00", size = 882114, upload-time = "2025-09-25T21:32:03.376Z" }, + { url = "https://files.pythonhosted.org/packages/71/60/917329f640924b18ff085ab889a11c763e0b573da888e8404ff486657602/pyyaml-6.0.3-cp311-cp311-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:b8bb0864c5a28024fac8a632c443c87c5aa6f215c0b126c449ae1a150412f31d", size = 806638, upload-time = "2025-09-25T21:32:04.553Z" }, + { url = "https://files.pythonhosted.org/packages/dd/6f/529b0f316a9fd167281a6c3826b5583e6192dba792dd55e3203d3f8e655a/pyyaml-6.0.3-cp311-cp311-musllinux_1_2_aarch64.whl", hash = "sha256:1d37d57ad971609cf3c53ba6a7e365e40660e3be0e5175fa9f2365a379d6095a", size = 767463, upload-time = "2025-09-25T21:32:06.152Z" }, + { url = "https://files.pythonhosted.org/packages/f2/6a/b627b4e0c1dd03718543519ffb2f1deea4a1e6d42fbab8021936a4d22589/pyyaml-6.0.3-cp311-cp311-musllinux_1_2_x86_64.whl", hash = "sha256:37503bfbfc9d2c40b344d06b2199cf0e96e97957ab1c1b546fd4f87e53e5d3e4", size = 794986, upload-time = "2025-09-25T21:32:07.367Z" }, + { url = "https://files.pythonhosted.org/packages/45/91/47a6e1c42d9ee337c4839208f30d9f09caa9f720ec7582917b264defc875/pyyaml-6.0.3-cp311-cp311-win32.whl", hash = "sha256:8098f252adfa6c80ab48096053f512f2321f0b998f98150cea9bd23d83e1467b", size = 142543, upload-time = "2025-09-25T21:32:08.95Z" }, + { url = "https://files.pythonhosted.org/packages/da/e3/ea007450a105ae919a72393cb06f122f288ef60bba2dc64b26e2646fa315/pyyaml-6.0.3-cp311-cp311-win_amd64.whl", hash = "sha256:9f3bfb4965eb874431221a3ff3fdcddc7e74e3b07799e0e84ca4a0f867d449bf", size = 158763, upload-time = "2025-09-25T21:32:09.96Z" }, + { url = "https://files.pythonhosted.org/packages/d1/33/422b98d2195232ca1826284a76852ad5a86fe23e31b009c9886b2d0fb8b2/pyyaml-6.0.3-cp312-cp312-macosx_10_13_x86_64.whl", hash = "sha256:7f047e29dcae44602496db43be01ad42fc6f1cc0d8cd6c83d342306c32270196", size = 182063, upload-time = "2025-09-25T21:32:11.445Z" }, + { url = "https://files.pythonhosted.org/packages/89/a0/6cf41a19a1f2f3feab0e9c0b74134aa2ce6849093d5517a0c550fe37a648/pyyaml-6.0.3-cp312-cp312-macosx_11_0_arm64.whl", hash = "sha256:fc09d0aa354569bc501d4e787133afc08552722d3ab34836a80547331bb5d4a0", size = 173973, upload-time = "2025-09-25T21:32:12.492Z" }, + { url = "https://files.pythonhosted.org/packages/ed/23/7a778b6bd0b9a8039df8b1b1d80e2e2ad78aa04171592c8a5c43a56a6af4/pyyaml-6.0.3-cp312-cp312-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:9149cad251584d5fb4981be1ecde53a1ca46c891a79788c0df828d2f166bda28", size = 775116, upload-time = "2025-09-25T21:32:13.652Z" }, + { url = "https://files.pythonhosted.org/packages/65/30/d7353c338e12baef4ecc1b09e877c1970bd3382789c159b4f89d6a70dc09/pyyaml-6.0.3-cp312-cp312-manylinux2014_s390x.manylinux_2_17_s390x.manylinux_2_28_s390x.whl", hash = "sha256:5fdec68f91a0c6739b380c83b951e2c72ac0197ace422360e6d5a959d8d97b2c", size = 844011, upload-time = "2025-09-25T21:32:15.21Z" }, + { url = "https://files.pythonhosted.org/packages/8b/9d/b3589d3877982d4f2329302ef98a8026e7f4443c765c46cfecc8858c6b4b/pyyaml-6.0.3-cp312-cp312-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:ba1cc08a7ccde2d2ec775841541641e4548226580ab850948cbfda66a1befcdc", size = 807870, upload-time = "2025-09-25T21:32:16.431Z" }, + { url = "https://files.pythonhosted.org/packages/05/c0/b3be26a015601b822b97d9149ff8cb5ead58c66f981e04fedf4e762f4bd4/pyyaml-6.0.3-cp312-cp312-musllinux_1_2_aarch64.whl", hash = "sha256:8dc52c23056b9ddd46818a57b78404882310fb473d63f17b07d5c40421e47f8e", size = 761089, upload-time = "2025-09-25T21:32:17.56Z" }, + { url = "https://files.pythonhosted.org/packages/be/8e/98435a21d1d4b46590d5459a22d88128103f8da4c2d4cb8f14f2a96504e1/pyyaml-6.0.3-cp312-cp312-musllinux_1_2_x86_64.whl", hash = "sha256:41715c910c881bc081f1e8872880d3c650acf13dfa8214bad49ed4cede7c34ea", size = 790181, upload-time = "2025-09-25T21:32:18.834Z" }, + { url = "https://files.pythonhosted.org/packages/74/93/7baea19427dcfbe1e5a372d81473250b379f04b1bd3c4c5ff825e2327202/pyyaml-6.0.3-cp312-cp312-win32.whl", hash = "sha256:96b533f0e99f6579b3d4d4995707cf36df9100d67e0c8303a0c55b27b5f99bc5", size = 137658, upload-time = "2025-09-25T21:32:20.209Z" }, + { url = "https://files.pythonhosted.org/packages/86/bf/899e81e4cce32febab4fb42bb97dcdf66bc135272882d1987881a4b519e9/pyyaml-6.0.3-cp312-cp312-win_amd64.whl", hash = "sha256:5fcd34e47f6e0b794d17de1b4ff496c00986e1c83f7ab2fb8fcfe9616ff7477b", size = 154003, upload-time = "2025-09-25T21:32:21.167Z" }, + { url = "https://files.pythonhosted.org/packages/1a/08/67bd04656199bbb51dbed1439b7f27601dfb576fb864099c7ef0c3e55531/pyyaml-6.0.3-cp312-cp312-win_arm64.whl", hash = "sha256:64386e5e707d03a7e172c0701abfb7e10f0fb753ee1d773128192742712a98fd", size = 140344, upload-time = "2025-09-25T21:32:22.617Z" }, + { url = "https://files.pythonhosted.org/packages/d1/11/0fd08f8192109f7169db964b5707a2f1e8b745d4e239b784a5a1dd80d1db/pyyaml-6.0.3-cp313-cp313-macosx_10_13_x86_64.whl", hash = "sha256:8da9669d359f02c0b91ccc01cac4a67f16afec0dac22c2ad09f46bee0697eba8", size = 181669, upload-time = "2025-09-25T21:32:23.673Z" }, + { url = "https://files.pythonhosted.org/packages/b1/16/95309993f1d3748cd644e02e38b75d50cbc0d9561d21f390a76242ce073f/pyyaml-6.0.3-cp313-cp313-macosx_11_0_arm64.whl", hash = "sha256:2283a07e2c21a2aa78d9c4442724ec1eb15f5e42a723b99cb3d822d48f5f7ad1", size = 173252, upload-time = "2025-09-25T21:32:25.149Z" }, + { url = "https://files.pythonhosted.org/packages/50/31/b20f376d3f810b9b2371e72ef5adb33879b25edb7a6d072cb7ca0c486398/pyyaml-6.0.3-cp313-cp313-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:ee2922902c45ae8ccada2c5b501ab86c36525b883eff4255313a253a3160861c", size = 767081, upload-time = "2025-09-25T21:32:26.575Z" }, + { url = "https://files.pythonhosted.org/packages/49/1e/a55ca81e949270d5d4432fbbd19dfea5321eda7c41a849d443dc92fd1ff7/pyyaml-6.0.3-cp313-cp313-manylinux2014_s390x.manylinux_2_17_s390x.manylinux_2_28_s390x.whl", hash = "sha256:a33284e20b78bd4a18c8c2282d549d10bc8408a2a7ff57653c0cf0b9be0afce5", size = 841159, upload-time = "2025-09-25T21:32:27.727Z" }, + { url = "https://files.pythonhosted.org/packages/74/27/e5b8f34d02d9995b80abcef563ea1f8b56d20134d8f4e5e81733b1feceb2/pyyaml-6.0.3-cp313-cp313-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:0f29edc409a6392443abf94b9cf89ce99889a1dd5376d94316ae5145dfedd5d6", size = 801626, upload-time = "2025-09-25T21:32:28.878Z" }, + { url = "https://files.pythonhosted.org/packages/f9/11/ba845c23988798f40e52ba45f34849aa8a1f2d4af4b798588010792ebad6/pyyaml-6.0.3-cp313-cp313-musllinux_1_2_aarch64.whl", hash = "sha256:f7057c9a337546edc7973c0d3ba84ddcdf0daa14533c2065749c9075001090e6", size = 753613, upload-time = "2025-09-25T21:32:30.178Z" }, + { url = "https://files.pythonhosted.org/packages/3d/e0/7966e1a7bfc0a45bf0a7fb6b98ea03fc9b8d84fa7f2229e9659680b69ee3/pyyaml-6.0.3-cp313-cp313-musllinux_1_2_x86_64.whl", hash = "sha256:eda16858a3cab07b80edaf74336ece1f986ba330fdb8ee0d6c0d68fe82bc96be", size = 794115, upload-time = "2025-09-25T21:32:31.353Z" }, + { url = "https://files.pythonhosted.org/packages/de/94/980b50a6531b3019e45ddeada0626d45fa85cbe22300844a7983285bed3b/pyyaml-6.0.3-cp313-cp313-win32.whl", hash = "sha256:d0eae10f8159e8fdad514efdc92d74fd8d682c933a6dd088030f3834bc8e6b26", size = 137427, upload-time = "2025-09-25T21:32:32.58Z" }, + { url = "https://files.pythonhosted.org/packages/97/c9/39d5b874e8b28845e4ec2202b5da735d0199dbe5b8fb85f91398814a9a46/pyyaml-6.0.3-cp313-cp313-win_amd64.whl", hash = "sha256:79005a0d97d5ddabfeeea4cf676af11e647e41d81c9a7722a193022accdb6b7c", size = 154090, upload-time = "2025-09-25T21:32:33.659Z" }, + { url = "https://files.pythonhosted.org/packages/73/e8/2bdf3ca2090f68bb3d75b44da7bbc71843b19c9f2b9cb9b0f4ab7a5a4329/pyyaml-6.0.3-cp313-cp313-win_arm64.whl", hash = "sha256:5498cd1645aa724a7c71c8f378eb29ebe23da2fc0d7a08071d89469bf1d2defb", size = 140246, upload-time = "2025-09-25T21:32:34.663Z" }, + { url = "https://files.pythonhosted.org/packages/9d/8c/f4bd7f6465179953d3ac9bc44ac1a8a3e6122cf8ada906b4f96c60172d43/pyyaml-6.0.3-cp314-cp314-macosx_10_13_x86_64.whl", hash = "sha256:8d1fab6bb153a416f9aeb4b8763bc0f22a5586065f86f7664fc23339fc1c1fac", size = 181814, upload-time = "2025-09-25T21:32:35.712Z" }, + { url = "https://files.pythonhosted.org/packages/bd/9c/4d95bb87eb2063d20db7b60faa3840c1b18025517ae857371c4dd55a6b3a/pyyaml-6.0.3-cp314-cp314-macosx_11_0_arm64.whl", hash = "sha256:34d5fcd24b8445fadc33f9cf348c1047101756fd760b4dacb5c3e99755703310", size = 173809, upload-time = "2025-09-25T21:32:36.789Z" }, + { url = "https://files.pythonhosted.org/packages/92/b5/47e807c2623074914e29dabd16cbbdd4bf5e9b2db9f8090fa64411fc5382/pyyaml-6.0.3-cp314-cp314-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:501a031947e3a9025ed4405a168e6ef5ae3126c59f90ce0cd6f2bfc477be31b7", size = 766454, upload-time = "2025-09-25T21:32:37.966Z" }, + { url = "https://files.pythonhosted.org/packages/02/9e/e5e9b168be58564121efb3de6859c452fccde0ab093d8438905899a3a483/pyyaml-6.0.3-cp314-cp314-manylinux2014_s390x.manylinux_2_17_s390x.manylinux_2_28_s390x.whl", hash = "sha256:b3bc83488de33889877a0f2543ade9f70c67d66d9ebb4ac959502e12de895788", size = 836355, upload-time = "2025-09-25T21:32:39.178Z" }, + { url = "https://files.pythonhosted.org/packages/88/f9/16491d7ed2a919954993e48aa941b200f38040928474c9e85ea9e64222c3/pyyaml-6.0.3-cp314-cp314-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:c458b6d084f9b935061bc36216e8a69a7e293a2f1e68bf956dcd9e6cbcd143f5", size = 794175, upload-time = "2025-09-25T21:32:40.865Z" }, + { url = "https://files.pythonhosted.org/packages/dd/3f/5989debef34dc6397317802b527dbbafb2b4760878a53d4166579111411e/pyyaml-6.0.3-cp314-cp314-musllinux_1_2_aarch64.whl", hash = "sha256:7c6610def4f163542a622a73fb39f534f8c101d690126992300bf3207eab9764", size = 755228, upload-time = "2025-09-25T21:32:42.084Z" }, + { url = "https://files.pythonhosted.org/packages/d7/ce/af88a49043cd2e265be63d083fc75b27b6ed062f5f9fd6cdc223ad62f03e/pyyaml-6.0.3-cp314-cp314-musllinux_1_2_x86_64.whl", hash = "sha256:5190d403f121660ce8d1d2c1bb2ef1bd05b5f68533fc5c2ea899bd15f4399b35", size = 789194, upload-time = "2025-09-25T21:32:43.362Z" }, + { url = "https://files.pythonhosted.org/packages/23/20/bb6982b26a40bb43951265ba29d4c246ef0ff59c9fdcdf0ed04e0687de4d/pyyaml-6.0.3-cp314-cp314-win_amd64.whl", hash = "sha256:4a2e8cebe2ff6ab7d1050ecd59c25d4c8bd7e6f400f5f82b96557ac0abafd0ac", size = 156429, upload-time = "2025-09-25T21:32:57.844Z" }, + { url = "https://files.pythonhosted.org/packages/f4/f4/a4541072bb9422c8a883ab55255f918fa378ecf083f5b85e87fc2b4eda1b/pyyaml-6.0.3-cp314-cp314-win_arm64.whl", hash = "sha256:93dda82c9c22deb0a405ea4dc5f2d0cda384168e466364dec6255b293923b2f3", size = 143912, upload-time = "2025-09-25T21:32:59.247Z" }, + { url = "https://files.pythonhosted.org/packages/7c/f9/07dd09ae774e4616edf6cda684ee78f97777bdd15847253637a6f052a62f/pyyaml-6.0.3-cp314-cp314t-macosx_10_13_x86_64.whl", hash = "sha256:02893d100e99e03eda1c8fd5c441d8c60103fd175728e23e431db1b589cf5ab3", size = 189108, upload-time = "2025-09-25T21:32:44.377Z" }, + { url = "https://files.pythonhosted.org/packages/4e/78/8d08c9fb7ce09ad8c38ad533c1191cf27f7ae1effe5bb9400a46d9437fcf/pyyaml-6.0.3-cp314-cp314t-macosx_11_0_arm64.whl", hash = "sha256:c1ff362665ae507275af2853520967820d9124984e0f7466736aea23d8611fba", size = 183641, upload-time = "2025-09-25T21:32:45.407Z" }, + { url = "https://files.pythonhosted.org/packages/7b/5b/3babb19104a46945cf816d047db2788bcaf8c94527a805610b0289a01c6b/pyyaml-6.0.3-cp314-cp314t-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:6adc77889b628398debc7b65c073bcb99c4a0237b248cacaf3fe8a557563ef6c", size = 831901, upload-time = "2025-09-25T21:32:48.83Z" }, + { url = "https://files.pythonhosted.org/packages/8b/cc/dff0684d8dc44da4d22a13f35f073d558c268780ce3c6ba1b87055bb0b87/pyyaml-6.0.3-cp314-cp314t-manylinux2014_s390x.manylinux_2_17_s390x.manylinux_2_28_s390x.whl", hash = "sha256:a80cb027f6b349846a3bf6d73b5e95e782175e52f22108cfa17876aaeff93702", size = 861132, upload-time = "2025-09-25T21:32:50.149Z" }, + { url = "https://files.pythonhosted.org/packages/b1/5e/f77dc6b9036943e285ba76b49e118d9ea929885becb0a29ba8a7c75e29fe/pyyaml-6.0.3-cp314-cp314t-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:00c4bdeba853cc34e7dd471f16b4114f4162dc03e6b7afcc2128711f0eca823c", size = 839261, upload-time = "2025-09-25T21:32:51.808Z" }, + { url = "https://files.pythonhosted.org/packages/ce/88/a9db1376aa2a228197c58b37302f284b5617f56a5d959fd1763fb1675ce6/pyyaml-6.0.3-cp314-cp314t-musllinux_1_2_aarch64.whl", hash = "sha256:66e1674c3ef6f541c35191caae2d429b967b99e02040f5ba928632d9a7f0f065", size = 805272, upload-time = "2025-09-25T21:32:52.941Z" }, + { url = "https://files.pythonhosted.org/packages/da/92/1446574745d74df0c92e6aa4a7b0b3130706a4142b2d1a5869f2eaa423c6/pyyaml-6.0.3-cp314-cp314t-musllinux_1_2_x86_64.whl", hash = "sha256:16249ee61e95f858e83976573de0f5b2893b3677ba71c9dd36b9cf8be9ac6d65", size = 829923, upload-time = "2025-09-25T21:32:54.537Z" }, + { url = "https://files.pythonhosted.org/packages/f0/7a/1c7270340330e575b92f397352af856a8c06f230aa3e76f86b39d01b416a/pyyaml-6.0.3-cp314-cp314t-win_amd64.whl", hash = "sha256:4ad1906908f2f5ae4e5a8ddfce73c320c2a1429ec52eafd27138b7f1cbe341c9", size = 174062, upload-time = "2025-09-25T21:32:55.767Z" }, + { url = "https://files.pythonhosted.org/packages/f1/12/de94a39c2ef588c7e6455cfbe7343d3b2dc9d6b6b2f40c4c6565744c873d/pyyaml-6.0.3-cp314-cp314t-win_arm64.whl", hash = "sha256:ebc55a14a21cb14062aa4162f906cd962b28e2e9ea38f9b4391244cd8de4ae0b", size = 149341, upload-time = "2025-09-25T21:32:56.828Z" }, +] + +[[package]] +name = "regex" +version = "2026.9.29" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/fc/f2/af1da9d3ceed77bfcdce40427d49ba0be94e4fe84245e3bfef68c10e75b6/regex-2026.9.29.tar.gz", hash = "sha256:8b5fcc4771732191b2b7d1dd68d8f0353f47f8d90b6150f6dce58bf1112442cb", size = 419199, upload-time = "2026-09-29T00:49:58.298Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/e8/6b/6dea87689c3a06a6e79d254bf824e6f3e3d724b5ba027c6112559aa6cd2c/regex-2026.9.29-cp311-cp311-macosx_10_9_universal2.whl", hash = "sha256:6abb75ab16bc3281714a5b99548a2225db70dba1f995f6d7f7419b76eb5a8fbe", size = 495413, upload-time = "2026-09-29T00:46:14.51Z" }, + { url = "https://files.pythonhosted.org/packages/3a/a5/0c791a0e83ad1013d262c13247c4c77e0f4a8d05bdc167df96aba6681c0d/regex-2026.9.29-cp311-cp311-macosx_10_9_x86_64.whl", hash = "sha256:b7b893976e7fe42053da64f2aa27239c24252fd2ec6df471e1be197c0addc3b1", size = 295145, upload-time = "2026-09-29T00:46:16.292Z" }, + { url = "https://files.pythonhosted.org/packages/b1/07/9bf3607d8d13a12e436ab9d63f9791e10706827d535695b23964ad79fd79/regex-2026.9.29-cp311-cp311-macosx_11_0_arm64.whl", hash = "sha256:066d0e3dbfdd739bce2bf8c2a41dd16f73e3d8adc2eb06dd803a36a307f56075", size = 292123, upload-time = "2026-09-29T00:46:17.646Z" }, + { url = "https://files.pythonhosted.org/packages/64/6b/32c2e6fc617e1d3f247e250fea31a9a35b1265bd32f585968aa13b9999b9/regex-2026.9.29-cp311-cp311-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:7020ed44df30b3aa492c00ee3b52d0548c1f30c2c6c5bb13ae897680900d3413", size = 799537, upload-time = "2026-09-29T00:46:18.976Z" }, + { url = "https://files.pythonhosted.org/packages/bf/72/f041177f3c7a4606f7c81a95fe7eea03e2a0c4e8bff9e439a01432cbc9f2/regex-2026.9.29-cp311-cp311-manylinux2014_ppc64le.manylinux_2_17_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:ae4613d7d9dda60fcba95f846cc6f808017f1843f392cf9daad14a6534493d71", size = 873255, upload-time = "2026-09-29T00:46:20.684Z" }, + { url = "https://files.pythonhosted.org/packages/d0/4e/a78948e11dd715e0e46716c2e0f3404b3fe6a44e2a2e9abdc7d965cab2b3/regex-2026.9.29-cp311-cp311-manylinux2014_s390x.manylinux_2_17_s390x.manylinux_2_28_s390x.whl", hash = "sha256:bec37990e3d6121f29ecfb594bd8f1bf009e9f7926daba2e50e3b27d3892a783", size = 914570, upload-time = "2026-09-29T00:46:22.599Z" }, + { url = "https://files.pythonhosted.org/packages/8a/70/aa08d1d2b294894b365e5f8ba5380fe3f8546acdb81f10639dfd74209c37/regex-2026.9.29-cp311-cp311-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:612b709381c0355b70d89cdb51b7f670591ed5cbbc0e3b5337488019dc667b65", size = 804978, upload-time = "2026-09-29T00:46:23.981Z" }, + { url = "https://files.pythonhosted.org/packages/21/32/1b03534c4715aca3b564416d28d518083ed4dab3bc913267600d2256140d/regex-2026.9.29-cp311-cp311-manylinux_2_31_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:a760da040b47767b4b873adfb7c3b691e9ba2fc60f113f9d0b88f1a62f323e85", size = 779297, upload-time = "2026-09-29T00:46:25.318Z" }, + { url = "https://files.pythonhosted.org/packages/76/a7/378f6f558d9e4444af315a307c5953565a511d1e3666f1bb7bdc82012b6b/regex-2026.9.29-cp311-cp311-musllinux_1_2_aarch64.whl", hash = "sha256:49ee178ca31c94621294bf9b8b676a92a2e6bba8af0529591753719e57edb621", size = 786961, upload-time = "2026-09-29T00:46:26.963Z" }, + { url = "https://files.pythonhosted.org/packages/59/13/79f0b1846f5f342f92ddbd4b27b18bcb86da96d902c1a0be26520bde98d7/regex-2026.9.29-cp311-cp311-musllinux_1_2_ppc64le.whl", hash = "sha256:5eeb8edc6110d9194a4d0d54610f64c37a31c605b5dbb7e407fc6ec7fa34a4a1", size = 863234, upload-time = "2026-09-29T00:46:28.58Z" }, + { url = "https://files.pythonhosted.org/packages/97/19/05af70dec9f2eed6ba34e08d2dcc6a48e7ae5e307659d5fe4201a5d7bbee/regex-2026.9.29-cp311-cp311-musllinux_1_2_riscv64.whl", hash = "sha256:ccb64d887a9db1cd76dbc0f92051a1a478a2a67e7f56c62d915cb881d7734704", size = 766487, upload-time = "2026-09-29T00:46:29.941Z" }, + { url = "https://files.pythonhosted.org/packages/01/e1/9c7486d4afe8fdd1fe0ad60139f8aa91427381f409af6a29b609d8fdcb3a/regex-2026.9.29-cp311-cp311-musllinux_1_2_s390x.whl", hash = "sha256:9e4482589065c8ecd761cff522dcd85f2d39e62f551e37e025d1c7d54772def3", size = 854792, upload-time = "2026-09-29T00:46:31.358Z" }, + { url = "https://files.pythonhosted.org/packages/26/c7/49d008ff5f741d9a9799d7315556f3a12b983ff0fcd2cdfb62904bedafbf/regex-2026.9.29-cp311-cp311-musllinux_1_2_x86_64.whl", hash = "sha256:d60030baaa7bfbb02d650c126cdcddcb6e33dbff14d819434c8fa2fdcaeeeba5", size = 793050, upload-time = "2026-09-29T00:46:32.775Z" }, + { url = "https://files.pythonhosted.org/packages/cb/a1/46ba549e65562ca04608b24179b8a7bb6f146ae0e7c6d7f5e70f3339c8ba/regex-2026.9.29-cp311-cp311-win32.whl", hash = "sha256:18ae8eed4526e35bdb754d61562b90bf5c00a67fdcf3cc1380dd59597486631b", size = 268940, upload-time = "2026-09-29T00:46:34.179Z" }, + { url = "https://files.pythonhosted.org/packages/4d/4a/aab232183c70fdcf77bcf0c51819da02ec522e393e6a0bf00bcf2142e21f/regex-2026.9.29-cp311-cp311-win_amd64.whl", hash = "sha256:1043aedf5917caa861bcb25a9c11460049656bdf0017a90a309fa8f255467725", size = 280641, upload-time = "2026-09-29T00:46:35.484Z" }, + { url = "https://files.pythonhosted.org/packages/33/b1/7c05954af0f51de376df2ba97f7f78a8b79334c7e5b3d2d9f2aead1f4d3d/regex-2026.9.29-cp311-cp311-win_arm64.whl", hash = "sha256:352cf115a810b357caa35193ab656ecf5ef41056855e82f292c99e8514f8d954", size = 279411, upload-time = "2026-09-29T00:46:37.193Z" }, + { url = "https://files.pythonhosted.org/packages/84/48/3fdcde9a0baa84d7d25571223265d6e434e114763b438601d54a8028bf3e/regex-2026.9.29-cp312-cp312-macosx_10_13_universal2.whl", hash = "sha256:dc79d36d0618752265f0d575915bdc5c5130ecb9c9f6b3bcefeae32e4bdfafcf", size = 497903, upload-time = "2026-09-29T00:46:38.938Z" }, + { url = "https://files.pythonhosted.org/packages/2e/1c/4ee3e97c76f53940488dfe7a7e18705e78daac8cd7fb161d246b9e328449/regex-2026.9.29-cp312-cp312-macosx_10_13_x86_64.whl", hash = "sha256:3a21a9509d0ee88e7a70e1ad228cd2f0e0fd1e187458db132e8a8d18c97daf9d", size = 296416, upload-time = "2026-09-29T00:46:40.406Z" }, + { url = "https://files.pythonhosted.org/packages/37/14/f3f0ba083d2094392d5eabf56db5ea6ba469fd6e927afd187042054ea68a/regex-2026.9.29-cp312-cp312-macosx_11_0_arm64.whl", hash = "sha256:f57dc6b8fef170f105d2cf5cdce254f47b137d7755086cf7050f47e16582abba", size = 293633, upload-time = "2026-09-29T00:46:41.959Z" }, + { url = "https://files.pythonhosted.org/packages/c9/72/67e7a8ce17f1aea49df215564048efb49cc8c2b31a0e0fc30f36838f8516/regex-2026.9.29-cp312-cp312-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:f93bc1c3486ef3747e07c9d7c1d0a147b8fbaab975f80e348aed6f71309dfaca", size = 805885, upload-time = "2026-09-29T00:46:43.373Z" }, + { url = "https://files.pythonhosted.org/packages/f6/78/25436bcfd4d2260b4b4090094d55d7ab53ec8a1ab4865a0b8bcb33c7d5c0/regex-2026.9.29-cp312-cp312-manylinux2014_ppc64le.manylinux_2_17_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:9e1d3a4cb7993b708f0ada8d0c84590efd853f169e7147d2202c9da503180242", size = 878344, upload-time = "2026-09-29T00:46:45.328Z" }, + { url = "https://files.pythonhosted.org/packages/97/e6/a09ec3a23ae41d6179880e67f0aace9284b2d95f2d7b326eff203f8eec5e/regex-2026.9.29-cp312-cp312-manylinux2014_s390x.manylinux_2_17_s390x.manylinux_2_28_s390x.whl", hash = "sha256:dabee8f4935e731fb46b2a3091bdda0d3d94b3bbfb907d2b4f12eefce4009619", size = 919181, upload-time = "2026-09-29T00:46:47.041Z" }, + { url = "https://files.pythonhosted.org/packages/26/83/d2fbd2e4e3afb1167daa825187d196f313cbaa1a4768f311fb041bb0e3d2/regex-2026.9.29-cp312-cp312-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:39ab5894d971f9ac68baa6eca5c50387db579cfcacf36ae8df3feceb1815e6d0", size = 807783, upload-time = "2026-09-29T00:46:48.894Z" }, + { url = "https://files.pythonhosted.org/packages/46/0b/eb429a7016610d44fc89a597163f8c9127505f0d7dc724dc9effbb6a3ac0/regex-2026.9.29-cp312-cp312-manylinux_2_31_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:c1a9a6651197fbed6f0212591418b9def774fc3f8324f78d1bf0e6a63e5f8aa1", size = 783465, upload-time = "2026-09-29T00:46:50.64Z" }, + { url = "https://files.pythonhosted.org/packages/1b/07/58a3c0153c7476898430f6a7cf3d9062a1d17fbea4f43399ecaf411c7b4c/regex-2026.9.29-cp312-cp312-musllinux_1_2_aarch64.whl", hash = "sha256:87fb80cbe3557e27e7b28b995c2b2eedf689b8886f941ab93e0e288f0976518a", size = 793519, upload-time = "2026-09-29T00:46:52.396Z" }, + { url = "https://files.pythonhosted.org/packages/2a/e8/161b94d39164520e21a7befe0245569bf7fda4c7cf1fc4e2df2b5def49da/regex-2026.9.29-cp312-cp312-musllinux_1_2_ppc64le.whl", hash = "sha256:3c5c2ef13797466aa64170cbb66ad98a32351dd4127694cea7199f80f213750d", size = 869293, upload-time = "2026-09-29T00:46:54.128Z" }, + { url = "https://files.pythonhosted.org/packages/8f/07/3b02ed829aa2decdc1955d222bd1e2f99d1c8bb4873bbb9a66b2f0a36bff/regex-2026.9.29-cp312-cp312-musllinux_1_2_riscv64.whl", hash = "sha256:59b49507f47479e299a9e1bc41b5cb83a7afda0540625f1dbae886615978acbf", size = 770239, upload-time = "2026-09-29T00:46:56.106Z" }, + { url = "https://files.pythonhosted.org/packages/42/5b/ba61f6fe062eb8562e742367d177bb75370434138ef6c9d2a27114f8d613/regex-2026.9.29-cp312-cp312-musllinux_1_2_s390x.whl", hash = "sha256:0dd8af32e9f7b56b7f95cc1fd79b23054c3bdc172392ae560acc24d57b7ffe71", size = 861973, upload-time = "2026-09-29T00:46:57.665Z" }, + { url = "https://files.pythonhosted.org/packages/cc/27/767259b20e8a842948990f5e99138d6c077248fd42f8b5468b1d9ca4b814/regex-2026.9.29-cp312-cp312-musllinux_1_2_x86_64.whl", hash = "sha256:db5e82ba15c142425b8406690032df89e39cca4a2e8afbbb9a3d84edc2373ac3", size = 796470, upload-time = "2026-09-29T00:46:59.236Z" }, + { url = "https://files.pythonhosted.org/packages/a0/05/2566c4ba849b68a8ab81a6bf428fa79d20aae7ddee83979103c0381df254/regex-2026.9.29-cp312-cp312-win32.whl", hash = "sha256:d0c3082bf79bcd6a614d55916590ad4b8f93200e10b97f463ea5d9d07c9b5f23", size = 269327, upload-time = "2026-09-29T00:47:01.135Z" }, + { url = "https://files.pythonhosted.org/packages/93/19/489bc8db91196381c935752df01ba3f607140daece33b78d88573f028e64/regex-2026.9.29-cp312-cp312-win_amd64.whl", hash = "sha256:fdd88ed5e20b1bcdd234421e454962c971aa44b653bdb7f1ea9ef683e90fb649", size = 280334, upload-time = "2026-09-29T00:47:04.436Z" }, + { url = "https://files.pythonhosted.org/packages/0b/47/fb88ba779d0e5e7d4b0ec1aceeb13845948a2cb876bd572a2d1dfdba090b/regex-2026.9.29-cp312-cp312-win_arm64.whl", hash = "sha256:4fe97894d1b306c919b4e50def1e6f6c522f4d03a7283811f4d108f1ce5d3ac2", size = 279614, upload-time = "2026-09-29T00:47:06.541Z" }, + { url = "https://files.pythonhosted.org/packages/79/d5/6080f7d1a6e7e36aa720f806ac93c035ba39c209ae6cc510e8ef4c0279c6/regex-2026.9.29-cp313-cp313-macosx_10_13_universal2.whl", hash = "sha256:f1a0d5117230dd46b399a30a38afa44f79c99f3168988fdc4f425c3f928b39df", size = 497622, upload-time = "2026-09-29T00:47:08.251Z" }, + { url = "https://files.pythonhosted.org/packages/00/71/c87fc7a2e21a42f9d57489db32951c37eef56d153840459a80d464f0321d/regex-2026.9.29-cp313-cp313-macosx_10_13_x86_64.whl", hash = "sha256:f0fe9834e5aeccaf19a0d8feb296d66a24be1a7c9922002f842a682cd5abb787", size = 296281, upload-time = "2026-09-29T00:47:09.764Z" }, + { url = "https://files.pythonhosted.org/packages/11/9e/aa0f4cde3bc4688c1d58b0cd8415edd708339bc0bc401a195b0b1e8c8f0c/regex-2026.9.29-cp313-cp313-macosx_11_0_arm64.whl", hash = "sha256:c90fcf7804ea0a54b896ce0f2b9565350220b8d4890fd0db461a476a4c687963", size = 293472, upload-time = "2026-09-29T00:47:11.723Z" }, + { url = "https://files.pythonhosted.org/packages/90/d4/e835c487850ed922a8d6074f953b888c8ea99775c76b9ed5f8a4d72eab92/regex-2026.9.29-cp313-cp313-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:e11edba5bc344a32b029a7af9d4b3173982dd79eeafa0b9dbd787364414b0509", size = 805966, upload-time = "2026-09-29T00:47:13.235Z" }, + { url = "https://files.pythonhosted.org/packages/2c/57/ba8809847fbae8d2cbc71367c6ded510a7ec88bf52493c65efc1acf4effb/regex-2026.9.29-cp313-cp313-manylinux2014_ppc64le.manylinux_2_17_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:bb90e7177944b6684738c1fc36aabd2dd00d1de3be7dbe09f91e196f1bc0dc81", size = 878346, upload-time = "2026-09-29T00:47:14.877Z" }, + { url = "https://files.pythonhosted.org/packages/1a/52/e3da19fc3cc15ef67ab67e121e87887c3bccfdb683a7a9ec557c460ca5b7/regex-2026.9.29-cp313-cp313-manylinux2014_s390x.manylinux_2_17_s390x.manylinux_2_28_s390x.whl", hash = "sha256:d06fcdecc10fc7954d7c8f27a03c96055fe525274dc84a7b0dbdc3d6b9e03dab", size = 919250, upload-time = "2026-09-29T00:47:16.622Z" }, + { url = "https://files.pythonhosted.org/packages/9a/8e/c1ed81f55f992f6aa0b699a592a50c1ce9e6d44ff1aee2c14c0537dcef9c/regex-2026.9.29-cp313-cp313-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:d49c18f1ea294cf4adde2e5ac256e98c82ea9d708462ce4bf799dffa7cfe8a2c", size = 807902, upload-time = "2026-09-29T00:47:18.268Z" }, + { url = "https://files.pythonhosted.org/packages/ad/bc/5a6886eb470e41040e21e05b75024a18b6ebfe7ea400b72094a60f949101/regex-2026.9.29-cp313-cp313-manylinux_2_31_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:3e778bfccd63075167709136afbc251c1f683758d5bf49c803c60ac3f894ce6b", size = 783514, upload-time = "2026-09-29T00:47:19.921Z" }, + { url = "https://files.pythonhosted.org/packages/cb/52/6d951d453b023c6edb880f1ba474291b53b8ce1cc438b96a9db6d791d991/regex-2026.9.29-cp313-cp313-musllinux_1_2_aarch64.whl", hash = "sha256:686ac5350fceae63830bb98805fcb8039325bf4c06d9f6f048ff65229d5bffa5", size = 793466, upload-time = "2026-09-29T00:47:21.552Z" }, + { url = "https://files.pythonhosted.org/packages/99/b9/d5a41adc08360f5eee0dc4846c578f002366947211fc8af5a69a64ee7b9f/regex-2026.9.29-cp313-cp313-musllinux_1_2_ppc64le.whl", hash = "sha256:26ec4ccce55aa533fbd603d08911b01101a8fcfec987845ac3ae2c7087b2bde3", size = 869464, upload-time = "2026-09-29T00:47:23.276Z" }, + { url = "https://files.pythonhosted.org/packages/4b/32/d76c9d91f5d798e2e9e67f6f85ec4ae35445ac425f7454797311cecb80ca/regex-2026.9.29-cp313-cp313-musllinux_1_2_riscv64.whl", hash = "sha256:a655d34b2a6943af32401f3d94f72e9d731f6ad16285815550bf2b4ee69d420a", size = 770278, upload-time = "2026-09-29T00:47:25.193Z" }, + { url = "https://files.pythonhosted.org/packages/24/00/aeebdb540c620a0f7317f6d6fad80a47729ecf0599a24b5c34ec155351f5/regex-2026.9.29-cp313-cp313-musllinux_1_2_s390x.whl", hash = "sha256:0c992c19cd45058a4b92f68f139c93db168b48fb1f322c9a7cd620806afb6b51", size = 861949, upload-time = "2026-09-29T00:47:27.005Z" }, + { url = "https://files.pythonhosted.org/packages/12/62/d0314bcedfd3586197e4596931fa220260eb2385bf53184e5b9ae67db24b/regex-2026.9.29-cp313-cp313-musllinux_1_2_x86_64.whl", hash = "sha256:ebb8912f565b8cdbbf27debfe00df04202c20e2f651b9e32767930c5eace3621", size = 796602, upload-time = "2026-09-29T00:47:29.233Z" }, + { url = "https://files.pythonhosted.org/packages/ae/c7/d5a8c13a613facb03e0fb55c1ebaaf7bb35d8e2c1abe8bef8dca809fc1d9/regex-2026.9.29-cp313-cp313-win32.whl", hash = "sha256:4d7d93613b01b0199961330e49cfc52d479b3d5776c56c691db31130c0a07d91", size = 269306, upload-time = "2026-09-29T00:47:31.14Z" }, + { url = "https://files.pythonhosted.org/packages/80/a7/bf93a3a6afa5f7bc16b7afb94ae581b01cae620b8ad56bd8f9572a985959/regex-2026.9.29-cp313-cp313-win_amd64.whl", hash = "sha256:61956f074ecd123f55adca68ee3eab46e6a07ad3f8e64e6db95dfacb444f55c4", size = 280307, upload-time = "2026-09-29T00:47:32.709Z" }, + { url = "https://files.pythonhosted.org/packages/b2/7d/388274e53605a86297f433a08102a7bbdcf9379d47683d307ccaefd88e2c/regex-2026.9.29-cp313-cp313-win_arm64.whl", hash = "sha256:bfc71e6d970419c1309b3640305298643e2a734cad3f7cfb6d2ddee4175ab53d", size = 279599, upload-time = "2026-09-29T00:47:34.674Z" }, + { url = "https://files.pythonhosted.org/packages/93/1f/d9dc6f02f569625faf67a4daec926cd5023472dcd69bb44286dccd5a5ab3/regex-2026.9.29-cp314-cp314-macosx_10_15_universal2.whl", hash = "sha256:957bb708e8057ab1649ba566456429d691ec9b90d1c9ad1af1ba7ffbbeaf05f2", size = 497598, upload-time = "2026-09-29T00:47:36.541Z" }, + { url = "https://files.pythonhosted.org/packages/9c/83/9b693a3fd1451381e812031a8961ec5b3b8f0c8cc6871f14c5223642804d/regex-2026.9.29-cp314-cp314-macosx_10_15_x86_64.whl", hash = "sha256:c9b602fae1e00b7c035d661ce85575365719192a7b46784bd71cf64c68053aa0", size = 296250, upload-time = "2026-09-29T00:47:38.233Z" }, + { url = "https://files.pythonhosted.org/packages/dd/5f/52bc2abc3fef040cd9de76ab29c918d6a717a454ae2b9dd7938b0c95656d/regex-2026.9.29-cp314-cp314-macosx_11_0_arm64.whl", hash = "sha256:0166844493626c5015c6088ee15c9ca2fd060ca15b7641d1657da6a58432ae33", size = 293518, upload-time = "2026-09-29T00:47:39.957Z" }, + { url = "https://files.pythonhosted.org/packages/dc/fc/cf50671215ee0057046980b4571ef8646a005819bb67f0957e779ed107a5/regex-2026.9.29-cp314-cp314-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:b97a38fb4c732b6832db6bf108963adbcd82ef1268ba2025dce390f45af75efa", size = 806037, upload-time = "2026-09-29T00:47:41.676Z" }, + { url = "https://files.pythonhosted.org/packages/14/4b/dddef8fc15c63e4347cc9efb138d0cd306f30e6c98acbcc81a8f780083b9/regex-2026.9.29-cp314-cp314-manylinux2014_ppc64le.manylinux_2_17_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:a540abfab208e1b7ef2df231c40ef3b6cbb30a0aad6204e9b6a81c10a6794628", size = 878881, upload-time = "2026-09-29T00:47:43.755Z" }, + { url = "https://files.pythonhosted.org/packages/9f/cb/38daabed32d28f7e58a06e9344ce00dc67952e9996bc578ed6a29fe1240e/regex-2026.9.29-cp314-cp314-manylinux2014_s390x.manylinux_2_17_s390x.manylinux_2_28_s390x.whl", hash = "sha256:ddfa987262763c3c22a8367d2a49c244b018a74c3a8e3ab1a864119ad45c5633", size = 918684, upload-time = "2026-09-29T00:47:45.594Z" }, + { url = "https://files.pythonhosted.org/packages/a9/4d/041d9458a645fee4fce4d642a89d27271a3cfcd91095104f6dde44da70bf/regex-2026.9.29-cp314-cp314-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:2f7f7aa47b229f2b39a2ae2596d2ad5625d77b5eb9856fac2dab3eb506cdd0a0", size = 807176, upload-time = "2026-09-29T00:47:47.372Z" }, + { url = "https://files.pythonhosted.org/packages/bf/c4/4383eed7aa5aef67616cb1b3f3ad06b7c624c4e6cced48630cd5ce133d85/regex-2026.9.29-cp314-cp314-manylinux_2_31_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:d9b77b25b4f395f92de6099ab08e8ae2bc7e51dfe157f22900902243a5cc90c7", size = 784315, upload-time = "2026-09-29T00:47:49.518Z" }, + { url = "https://files.pythonhosted.org/packages/5c/a6/0086ad31cebb183c637d3198547075aa493afde308e1ff61fccccb29ba6e/regex-2026.9.29-cp314-cp314-musllinux_1_2_aarch64.whl", hash = "sha256:34b6925af9853bf461950e6508910f179fd6e9b1a7ec8548e069606b7e51a26b", size = 793748, upload-time = "2026-09-29T00:47:51.279Z" }, + { url = "https://files.pythonhosted.org/packages/d5/a0/f9005cba3f629a859573fc5d1224ea4e1f97919ec8581d018e03a351a604/regex-2026.9.29-cp314-cp314-musllinux_1_2_ppc64le.whl", hash = "sha256:addd736a0547d553283adaf4e05d7104e7f2c7b0b092e9b4d28756825f14531f", size = 870302, upload-time = "2026-09-29T00:47:53.368Z" }, + { url = "https://files.pythonhosted.org/packages/01/4f/e1a3e46bb5315a4e18b01a990e7a28e2a16595609d50c442baf2815a3c65/regex-2026.9.29-cp314-cp314-musllinux_1_2_riscv64.whl", hash = "sha256:fe3fa1dd453ed5c7f5ea23a26218329790ed7197a99b90e94330e313959a7f52", size = 770299, upload-time = "2026-09-29T00:47:55.606Z" }, + { url = "https://files.pythonhosted.org/packages/2c/fe/f303b4acfda44e1ff1379368748c1ef2dad04a6a8e9c0ecbc970b19d97ca/regex-2026.9.29-cp314-cp314-musllinux_1_2_s390x.whl", hash = "sha256:0cc63b5e47c12a48d90c7e9d7de6a035dd14f62868aaedbb4e0ff8ba2b8bfe7b", size = 861570, upload-time = "2026-09-29T00:47:57.617Z" }, + { url = "https://files.pythonhosted.org/packages/60/b6/b4f7e99249f596017c60ccad5faf9310fc8e3e59bb2244940a90a1b0bdff/regex-2026.9.29-cp314-cp314-musllinux_1_2_x86_64.whl", hash = "sha256:724184b4aafed865e4f13ca313fdcb43024300c028ec67319cfa16847d84685e", size = 795967, upload-time = "2026-09-29T00:47:59.922Z" }, + { url = "https://files.pythonhosted.org/packages/fb/d3/fc865a4638d9f6762192b6bab5b7aa1f33a90e9e99578c2e111e2a63c8c3/regex-2026.9.29-cp314-cp314-win32.whl", hash = "sha256:c6c8fabf1dafc1f1ddcbb67896d3f93efb092e8c4b6322d7389b944e76a484e5", size = 274758, upload-time = "2026-09-29T00:48:01.8Z" }, + { url = "https://files.pythonhosted.org/packages/31/e2/c2b466924ccbeb874862968ca638051b15a8fd29d994a0e99004a5cbf78e/regex-2026.9.29-cp314-cp314-win_amd64.whl", hash = "sha256:1c2a0026062abcc321a53db4a185ceba0b59a66b5d37b0808917a88b55a5257f", size = 283817, upload-time = "2026-09-29T00:48:03.614Z" }, + { url = "https://files.pythonhosted.org/packages/c6/42/ea0f8dbaa924fa75c6338935eaee2f44dab369b27f02db1e03d74344b049/regex-2026.9.29-cp314-cp314-win_arm64.whl", hash = "sha256:121a76a0985db80ceae9e171c337f8c927868e37d01b54e3ce87bc87f9c6a208", size = 283663, upload-time = "2026-09-29T00:48:05.624Z" }, + { url = "https://files.pythonhosted.org/packages/44/48/d58e5081119f5c223bbb37d2340acde3d069e1df8e8cd166c37502eee4da/regex-2026.9.29-cp314-cp314t-macosx_10_15_universal2.whl", hash = "sha256:e31f72490b7c12f7790e1e25c3afffd20503ee1bfb43461d7838b871ff244b19", size = 501393, upload-time = "2026-09-29T00:48:07.833Z" }, + { url = "https://files.pythonhosted.org/packages/72/3c/c49945287d4f9efee7d41f98072f8ad880efb8f430595a612fbdea996a4e/regex-2026.9.29-cp314-cp314t-macosx_10_15_x86_64.whl", hash = "sha256:80ea96f5c1a30bf09007d48466521d9c294bebe197c708c3359096e3e3691632", size = 298237, upload-time = "2026-09-29T00:48:09.684Z" }, + { url = "https://files.pythonhosted.org/packages/f9/1f/688cb61c3d4cf7bcc1ed444b5cc49399eba3e51c469ae285cf87fea3022e/regex-2026.9.29-cp314-cp314t-macosx_11_0_arm64.whl", hash = "sha256:554bffadcbcb6d5f4e5fb10a61cc52084b9a63d1dab5f10bcd2c4343972e8e2c", size = 295936, upload-time = "2026-09-29T00:48:11.454Z" }, + { url = "https://files.pythonhosted.org/packages/26/a3/de43ac6b877b7d09c19a3a426b1bd5acdd209eaaf68f406466f80439ccf6/regex-2026.9.29-cp314-cp314t-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:864e9b87ac33c3fb9fb4ad48166d4fdb579c351d5c77deb0d34bccb36a775cd9", size = 816905, upload-time = "2026-09-29T00:48:13.321Z" }, + { url = "https://files.pythonhosted.org/packages/62/14/9940763201c51d537786304984c67d0fc3d2ed18837ffb6f09a869f6b6c9/regex-2026.9.29-cp314-cp314t-manylinux2014_ppc64le.manylinux_2_17_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:044265d77d94f5e3cb2fd72c76723807c429cb8c533e9d4672d0334a6f14f588", size = 881527, upload-time = "2026-09-29T00:48:15.313Z" }, + { url = "https://files.pythonhosted.org/packages/d3/e1/c842d8df0b23245ebf202f8ab9c39fd48e2db39959454ec39a41c8c72082/regex-2026.9.29-cp314-cp314t-manylinux2014_s390x.manylinux_2_17_s390x.manylinux_2_28_s390x.whl", hash = "sha256:2089fe39c406784d90101c726755ffa1497bb74638fd434300d2b88006186de8", size = 923115, upload-time = "2026-09-29T00:48:17.328Z" }, + { url = "https://files.pythonhosted.org/packages/d8/c1/98622479e3c354a446a75232e522d747d2b3df23092dcd8a5309380a2020/regex-2026.9.29-cp314-cp314t-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:0def9fb6abac55492d6d51cddb7225d07d6f279e774e0adc08569a54a5fc8d46", size = 820674, upload-time = "2026-09-29T00:48:19.32Z" }, + { url = "https://files.pythonhosted.org/packages/6c/d0/5808c95f9c79ed27b5eedaafc3df6239ec56a49f2e23ea8f831b18427c82/regex-2026.9.29-cp314-cp314t-manylinux_2_31_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:888d60953908dcf761aa320c3e390ab8556efbdb551ace63921de90f6ae0848d", size = 793306, upload-time = "2026-09-29T00:48:21.615Z" }, + { url = "https://files.pythonhosted.org/packages/bf/d3/021ca2638671ad20603bcd9b4d5bfa35d2610cd216a043ea7f0b44ea39f6/regex-2026.9.29-cp314-cp314t-musllinux_1_2_aarch64.whl", hash = "sha256:ed511a0708e2297e1d6431e7fb217e3402791e491e02da800658ace4973df1bb", size = 803103, upload-time = "2026-09-29T00:48:23.871Z" }, + { url = "https://files.pythonhosted.org/packages/6b/2d/755c6d13ef9c657378013676c391c7a402166b3f419a464a3e058dcbe533/regex-2026.9.29-cp314-cp314t-musllinux_1_2_ppc64le.whl", hash = "sha256:e1172147d28d8fbcf8cb8d26c41506169f5ad8fe9ec969cb116835a19d4d8eca", size = 872175, upload-time = "2026-09-29T00:48:26.255Z" }, + { url = "https://files.pythonhosted.org/packages/6c/fc/e1cab183b9dafe8597f58c1c766da9bf96204d3b2f232bcf3eeb75ff7b6c/regex-2026.9.29-cp314-cp314t-musllinux_1_2_riscv64.whl", hash = "sha256:92f05c9c42bde5785dc48770bc2194d9f7442544156f951e19cd31b096cec562", size = 777362, upload-time = "2026-09-29T00:48:28.389Z" }, + { url = "https://files.pythonhosted.org/packages/06/7c/e10ea17fba31fb4a1f9d13ed53a2d2a9066a2aea58d7557e263f6d99e7b0/regex-2026.9.29-cp314-cp314t-musllinux_1_2_s390x.whl", hash = "sha256:f37964e4a5e993d2fd45147741e9dff7f34a2d8c00ab94c4ea0514a4677f959e", size = 865606, upload-time = "2026-09-29T00:48:30.4Z" }, + { url = "https://files.pythonhosted.org/packages/8e/6e/69824d9aee1fd41c54ea7264654a47c8d9d84d8a228e11c2bcf4c201ed81/regex-2026.9.29-cp314-cp314t-musllinux_1_2_x86_64.whl", hash = "sha256:951733b1bbdb71e377cec567b409f1a7881b47cfcad84121aa74cb575fa425ea", size = 807945, upload-time = "2026-09-29T00:48:32.375Z" }, + { url = "https://files.pythonhosted.org/packages/89/22/857050a86e21ce60193e02a8ef662521f2e263a645c8b1b905fc136b61a7/regex-2026.9.29-cp314-cp314t-win32.whl", hash = "sha256:65b408d8fcb273e3499e7ef2ce796810da1becd208c7fb4373692a242d79d461", size = 276758, upload-time = "2026-09-29T00:48:34.72Z" }, + { url = "https://files.pythonhosted.org/packages/4d/96/56808fe029553d7d4c703414f2a527faad2ea2bfa9ca094a2e7f8762b530/regex-2026.9.29-cp314-cp314t-win_amd64.whl", hash = "sha256:bf48516e35cf848390ea68850aba53e7c333720d2945b4d2c25b69fc5171723f", size = 286527, upload-time = "2026-09-29T00:48:36.864Z" }, + { url = "https://files.pythonhosted.org/packages/01/aa/074e2cfb3d8101a6a764aba5f7c5d1e21de087483e35bdc0c4ce2eb60364/regex-2026.9.29-cp314-cp314t-win_arm64.whl", hash = "sha256:9173db3be74a35cb6731701094b98120f7ee4876a287882a59cdea1fa7da342f", size = 285954, upload-time = "2026-09-29T00:48:38.901Z" }, + { url = "https://files.pythonhosted.org/packages/a7/dc/d84990386c9dfdf8c377f00f371b241fdc9a2c8aea0e3d66941b2e51be0b/regex-2026.9.29-cp315-cp315-macosx_10_15_universal2.whl", hash = "sha256:c3589f40749acce747510bf5d589d54e376cb0930ea58b35effac97e5312b0c1", size = 497836, upload-time = "2026-09-29T00:48:40.858Z" }, + { url = "https://files.pythonhosted.org/packages/c2/ab/a569ebde875fa12ff8c6c9a30e07503620f195e4be4d54c3d3ee8eecc283/regex-2026.9.29-cp315-cp315-macosx_10_15_x86_64.whl", hash = "sha256:32ab11df9677ca80bcbb5fe4eb1da9109a5019239a054836efc6fa1c64e683cf", size = 296244, upload-time = "2026-09-29T00:48:42.952Z" }, + { url = "https://files.pythonhosted.org/packages/f3/3e/7d548e82a108e7c8b2d5246650e397a2f8db599f9b2e975466939c5b4e70/regex-2026.9.29-cp315-cp315-macosx_11_0_arm64.whl", hash = "sha256:7c03031610e3e6ed1768a2b7a8fc84637c1257b50c5eacaf094c6e17a84fc563", size = 293748, upload-time = "2026-09-29T00:48:44.985Z" }, + { url = "https://files.pythonhosted.org/packages/40/34/a8e19a52f452bbb07b32a2bef70dcdf90c2737049749f74cc12d7486fb4f/regex-2026.9.29-cp315-cp315-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:42e82e578c904445d4c8a35b8f28052cf567593215fa5db06266fbc6f77aaa2e", size = 807840, upload-time = "2026-09-29T00:48:46.948Z" }, + { url = "https://files.pythonhosted.org/packages/88/7b/11fbd4640b3bb82b72822a63c20ade4013d562d291703a9debeedc24e682/regex-2026.9.29-cp315-cp315-manylinux2014_ppc64le.manylinux_2_17_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:0b65c72739f981377c9c22e0c5c3cd7f42da7bd8a3c9209330fac772c7d893ed", size = 879330, upload-time = "2026-09-29T00:48:49.168Z" }, + { url = "https://files.pythonhosted.org/packages/f3/55/de58c74f1f4e31586d83eb39c56872d686c4e0d0966d151884c833b94ced/regex-2026.9.29-cp315-cp315-manylinux2014_s390x.manylinux_2_17_s390x.manylinux_2_28_s390x.whl", hash = "sha256:4408b2b27a95ca8cc48b7411945753773353b5c93b307754781086c99d3a576f", size = 919251, upload-time = "2026-09-29T00:48:51.322Z" }, + { url = "https://files.pythonhosted.org/packages/81/42/a8c480f6dd5ac59fa28ddae79afd9d7ac7e596fdb61813adc65bb6e674b8/regex-2026.9.29-cp315-cp315-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:a714befaacbd10092ffe4cea0d3c5f008fb9efe9bc322c715bcdfdee414b9a3d", size = 808808, upload-time = "2026-09-29T00:48:53.529Z" }, + { url = "https://files.pythonhosted.org/packages/68/60/0bc0d1ec8b37ad64be6fa30e035251f11de9667a0fac9e82ee74517d81be/regex-2026.9.29-cp315-cp315-manylinux_2_31_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:33026515aebc0e70d1c89978e53e8d695d35d9e472f8d5b34465ba3c74028650", size = 789907, upload-time = "2026-09-29T00:48:56.036Z" }, + { url = "https://files.pythonhosted.org/packages/da/84/116a3ef19b3acfe81077f0bf2cbc7714a5e94bc8935b7243ab61cb0f1c3c/regex-2026.9.29-cp315-cp315-musllinux_1_2_aarch64.whl", hash = "sha256:31b003f9a070335e2a8233ee9b14a3ca8e6d792012ae011f741bf0aaf11744c5", size = 795770, upload-time = "2026-09-29T00:48:58.284Z" }, + { url = "https://files.pythonhosted.org/packages/96/ba/e38c3f203e7e7e18c957d48e6cb6dbf96c11e95a44efa4a480522afc5d6d/regex-2026.9.29-cp315-cp315-musllinux_1_2_ppc64le.whl", hash = "sha256:c03c6eb6ece86dfdcbb34799efaa339b093132e1aceed491ba5e08fe06cdf699", size = 870671, upload-time = "2026-09-29T00:49:00.506Z" }, + { url = "https://files.pythonhosted.org/packages/2f/0f/9ee0b0cb76c55f63684bd7fff554978e8773b4fc86e2bcb2d50772dc1086/regex-2026.9.29-cp315-cp315-musllinux_1_2_riscv64.whl", hash = "sha256:a5300757f8a68f5b6cc33f57338d72a0e3589c5cc9ad5f8504ea06f028be582a", size = 778631, upload-time = "2026-09-29T00:49:02.984Z" }, + { url = "https://files.pythonhosted.org/packages/b6/19/e6e3eeb226af5872c4958002f6edef4e4f40ea4cc5f5665023f2019eb045/regex-2026.9.29-cp315-cp315-musllinux_1_2_s390x.whl", hash = "sha256:80c7cadd3fd2bfde5df8aa0787e315812cad0c313a753095d02f4c2b6c01677b", size = 862187, upload-time = "2026-09-29T00:49:05.264Z" }, + { url = "https://files.pythonhosted.org/packages/5b/62/823c102e106bb2711d6b7dfe5981552fe4467b2969c46a20c5c383cf498c/regex-2026.9.29-cp315-cp315-musllinux_1_2_x86_64.whl", hash = "sha256:3f1e6cb402a89457582cd696f982559217d13484a193202c394015297968c86d", size = 798423, upload-time = "2026-09-29T00:49:07.644Z" }, + { url = "https://files.pythonhosted.org/packages/37/e0/e927776258fa70b2f6feffc3be584ffc85ba4c1e20a320f0aee9a632fc7d/regex-2026.9.29-cp315-cp315-win32.whl", hash = "sha256:a64b85a4760337cfefdb27d42da6ed8b58e8cde3f2d57b6ef43e76ef6ea9ef47", size = 274757, upload-time = "2026-09-29T00:49:10.513Z" }, + { url = "https://files.pythonhosted.org/packages/77/04/358de85d1860238e1b4fa98fc2c80c990124a25d2e14739e28cc02c25562/regex-2026.9.29-cp315-cp315-win_amd64.whl", hash = "sha256:b3e445b66c80b4eb4234e855ce94d9adc183eedbd632816228d89930b91b2c5b", size = 283824, upload-time = "2026-09-29T00:49:12.849Z" }, + { url = "https://files.pythonhosted.org/packages/92/d3/d5c5b264784a5ab2b0f8cf620c1eeb4dbf3440d306761905e7d99345bef5/regex-2026.9.29-cp315-cp315-win_arm64.whl", hash = "sha256:8f39588af4731c8923c26810eb3b33f76f17633985e40f59c3cd45a33805a895", size = 283664, upload-time = "2026-09-29T00:49:15.331Z" }, + { url = "https://files.pythonhosted.org/packages/02/dc/f63ec2c201445ce1150fe780f5c56f16a10124d9a9da3a93161dbb0d8892/regex-2026.9.29-cp315-cp315t-macosx_10_15_universal2.whl", hash = "sha256:fb99cc9d45f48895d9d67f6a0b8a57f08d39c174d9f25ad97a313e0470267b1c", size = 501549, upload-time = "2026-09-29T00:49:17.705Z" }, + { url = "https://files.pythonhosted.org/packages/9a/5c/d2a698dc6bfc11fbce03f1cb0249c13284e93b79ed11f893edf6fac431c9/regex-2026.9.29-cp315-cp315t-macosx_10_15_x86_64.whl", hash = "sha256:720537c7ea6f80dc61913184edb0ce2497a306b39ef19f28505b322553d52bdb", size = 298187, upload-time = "2026-09-29T00:49:20.171Z" }, + { url = "https://files.pythonhosted.org/packages/85/b7/88dcdb38cd3935d4ee9e9ce9b8e56cb3b3518d1f020acfa7dd62ad289bf8/regex-2026.9.29-cp315-cp315t-macosx_11_0_arm64.whl", hash = "sha256:0fd2c901cc307a745ad4bc87f20060d7a0825a3371d1e93488af22e7a387f78f", size = 296157, upload-time = "2026-09-29T00:49:22.342Z" }, + { url = "https://files.pythonhosted.org/packages/d3/8e/ba6c01dde33a69fc294b38b43f6677baaa5735a6248f39708031a738158a/regex-2026.9.29-cp315-cp315t-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:b11b589e00095ec69cf79841a76360f9b079e95b0368a25b5ebb951ab0c157ff", size = 818468, upload-time = "2026-09-29T00:49:24.612Z" }, + { url = "https://files.pythonhosted.org/packages/2a/f1/2586693e3a2d6b1247852593d37a6c17b42a92ee44f7cdcb9a0c1494e64a/regex-2026.9.29-cp315-cp315t-manylinux2014_ppc64le.manylinux_2_17_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:d7cab119d0df0b9413f106b4d7fc34f2872d3574ed3806fb48959c830b1537da", size = 882825, upload-time = "2026-09-29T00:49:26.996Z" }, + { url = "https://files.pythonhosted.org/packages/30/51/084f3e7bdcd0e9c33665c938cf5d134dc3548cbb4a75f0197ec7bfd754b1/regex-2026.9.29-cp315-cp315t-manylinux2014_s390x.manylinux_2_17_s390x.manylinux_2_28_s390x.whl", hash = "sha256:b89efc38431793d28b7cd91227e2f952ad7c48df19132b17f43a5fec3c14143b", size = 923314, upload-time = "2026-09-29T00:49:29.822Z" }, + { url = "https://files.pythonhosted.org/packages/5a/f1/066c6fc23b7dc229789c21c880b5ba5ad689fb95fed12e078266f55a1f9b/regex-2026.9.29-cp315-cp315t-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:80a5ea3b4fd9d6a5b9a44f7976a9acaaab35aa3c1f6b29e5bd857dfabaded223", size = 822222, upload-time = "2026-09-29T00:49:32.404Z" }, + { url = "https://files.pythonhosted.org/packages/0a/56/592cd46fdb8f2f8682a1d7fd1310e4d0bcb93fbd0e6bbe4141ac28240227/regex-2026.9.29-cp315-cp315t-manylinux_2_31_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:19959129885356df0e97556856f77eb2888380dac18bed075a7c05c5128c618d", size = 794882, upload-time = "2026-09-29T00:49:35.076Z" }, + { url = "https://files.pythonhosted.org/packages/ee/4d/d65384bb071c864b01aa8314e3a6a687845ebd57588390976edc960c218b/regex-2026.9.29-cp315-cp315t-musllinux_1_2_aarch64.whl", hash = "sha256:6a1a824fbed817e0a891103886b68f063b1e83cc51bc97192a90a60195a9291f", size = 805887, upload-time = "2026-09-29T00:49:37.395Z" }, + { url = "https://files.pythonhosted.org/packages/65/b6/358de0d8f40d5178e4f7e7e121cfd5b961c812b77a055d11f5079e3f8fd7/regex-2026.9.29-cp315-cp315t-musllinux_1_2_ppc64le.whl", hash = "sha256:1ba8c6a416569ce0d37e83e28a254a61dc99a419084dfb6476cea02d997f74fa", size = 872901, upload-time = "2026-09-29T00:49:39.927Z" }, + { url = "https://files.pythonhosted.org/packages/00/06/6bfded72d043240c6b52bbb5e16f639d81affbf7484b4fe2ec45f3d4afc9/regex-2026.9.29-cp315-cp315t-musllinux_1_2_riscv64.whl", hash = "sha256:446654b29bfaa30500d80947eda42cef1449dc8a87f4e3cf061cc8485d3a1f0b", size = 782970, upload-time = "2026-09-29T00:49:42.581Z" }, + { url = "https://files.pythonhosted.org/packages/5a/20/9f418a50baa78b3ed8308fcb0cc49e472dd000b7ef935a7295af202ea744/regex-2026.9.29-cp315-cp315t-musllinux_1_2_s390x.whl", hash = "sha256:bf3c49863c23a1ad6da9c30351aed6cff8d5ddbeb63c5c8420ae54e98c7d0138", size = 865441, upload-time = "2026-09-29T00:49:45.238Z" }, + { url = "https://files.pythonhosted.org/packages/2c/29/817c7eacdeaf8463123e949bd394c39ad024eea1ec38ddf5ad141da2f3bd/regex-2026.9.29-cp315-cp315t-musllinux_1_2_x86_64.whl", hash = "sha256:01000ddf0e3ffef97f2413ceb514f6313040106b6d18a03ee00a4fe35c1eb1db", size = 809431, upload-time = "2026-09-29T00:49:47.878Z" }, + { url = "https://files.pythonhosted.org/packages/63/0b/83aab3b5b739947f744135a7a3a446e25433ebc92b05e01aae197ccbfdda/regex-2026.9.29-cp315-cp315t-win32.whl", hash = "sha256:c4e38dd8f39c43a91d2410ad2b85610701b0979342c3df1d69eaf8e838c757d8", size = 276964, upload-time = "2026-09-29T00:49:50.524Z" }, + { url = "https://files.pythonhosted.org/packages/72/f2/6314b5fc68789b5dcc38885bc6e3d6986b34fb3372b7231088ee5cecaa05/regex-2026.9.29-cp315-cp315t-win_amd64.whl", hash = "sha256:e2c89e9b762c57f59d5e99ee8b20202adb892e35f8d3485741340999ca55058e", size = 286487, upload-time = "2026-09-29T00:49:53.224Z" }, + { url = "https://files.pythonhosted.org/packages/56/bc/97b2245c8c7b2dd01f2db74f2bea003cd33c15009b4996a2447f46b5325c/regex-2026.9.29-cp315-cp315t-win_arm64.whl", hash = "sha256:e8c65ef3862a8ad6e86492b6ed9327805dd66904c012bd3649dc67d822ed6c34", size = 285955, upload-time = "2026-09-29T00:49:55.655Z" }, +] + +[[package]] +name = "requests" +version = "2.34.2" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "certifi" }, + { name = "charset-normalizer" }, + { name = "idna" }, + { name = "urllib3" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/ac/c3/e2a2b89f2d3e2179abd6d00ebd70bff6273f37fb3e0cc209f48b39d00cbf/requests-2.34.2.tar.gz", hash = "sha256:f288924cae4e29463698d6d60bc6a4da69c89185ad1e0bcc4104f584e960b9ed", size = 142856, upload-time = "2026-05-14T19:25:27.735Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/a0/f4/c67b0b3f1b9245e8d266f0f112c500d50e5b4e83cb6f3b71b6528104182a/requests-2.34.2-py3-none-any.whl", hash = "sha256:2a0d60c172f83ac6ab31e4554906c0f3b3588d37b5cb939b1c061f4907e278e0", size = 73075, upload-time = "2026-05-14T19:25:26.443Z" }, +] + +[[package]] +name = "scikit-learn" +version = "1.9.1" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "joblib" }, + { name = "narwhals" }, + { name = "numpy", version = "2.4.6", source = { registry = "https://pypi.org/simple" }, marker = "python_full_version < '3.12'" }, + { name = "numpy", version = "2.5.4", source = { registry = "https://pypi.org/simple" }, marker = "python_full_version >= '3.12'" }, + { name = "scipy", version = "1.17.1", source = { registry = "https://pypi.org/simple" }, marker = "python_full_version < '3.12'" }, + { name = "scipy", version = "1.18.1", source = { registry = "https://pypi.org/simple" }, marker = "python_full_version >= '3.12'" }, + { name = "threadpoolctl" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/d2/eb/eaf5e07fcc0da7149b0e084f24e54edd7441b9a89ce7e034032ae97fe3a0/scikit_learn-1.9.1.tar.gz", hash = "sha256:629cada3e33e2b9bf376cdc7614a47a4140b8aedc1d836579e359736fbd82977", size = 7786908, upload-time = "2026-09-10T18:34:04.679Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/79/67/1904bf57ab9ea77b5ba3899178bcb35ff8d814fc38ef6d1283bcddeaf04a/scikit_learn-1.9.1-cp311-cp311-macosx_10_9_x86_64.whl", hash = "sha256:326c188f92084bf58664229f4578eeab6176313b37cd5dfc85abd92b94130c58", size = 8823771, upload-time = "2026-09-10T18:32:29.891Z" }, + { url = "https://files.pythonhosted.org/packages/23/31/dc50d8780ac647bed9417f10a8b15ae3cf990ad7edb4f4b468f35030f8c5/scikit_learn-1.9.1-cp311-cp311-macosx_12_0_arm64.whl", hash = "sha256:b48b2b5b41d9c5fbafef5f37b042f61110df3318ad2a45baf57287ea5b9ba5a2", size = 8290892, upload-time = "2026-09-10T18:32:32.441Z" }, + { url = "https://files.pythonhosted.org/packages/a1/aa/a7a61517eaaa8acbf6061d255a625034e4afa277caff47e829574cc21d1c/scikit_learn-1.9.1-cp311-cp311-manylinux_2_27_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:4298fcc01b3d8fa9768d36894e99cce0747b3b2dd73bfd80779e393769d0afab", size = 9026402, upload-time = "2026-09-10T18:32:34.482Z" }, + { url = "https://files.pythonhosted.org/packages/69/46/f8b95a7a16fd13b458401d5e85a63d88ecb3679f3649812d5fdbea4c8357/scikit_learn-1.9.1-cp311-cp311-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:52a0703bbc07ad27f560fa63fa68e4c54dd735bfbbf65b4dd3c225dc7547b6df", size = 9315422, upload-time = "2026-09-10T18:32:36.581Z" }, + { url = "https://files.pythonhosted.org/packages/29/8a/8d1e888fe3d0dd4e2a1f3356dad703eed569aaf673e222b6f647005ad30f/scikit_learn-1.9.1-cp311-cp311-win_amd64.whl", hash = "sha256:220fa18152852a5ce29c49e1eaba9d44ec44631cd2e5cf65f5a40eafa5ab3412", size = 8329877, upload-time = "2026-09-10T18:32:38.729Z" }, + { url = "https://files.pythonhosted.org/packages/97/44/8f43f8de3e025c505efbd88b9264008f39490fbaf15a5ca8dbd8dae0cb74/scikit_learn-1.9.1-cp311-cp311-win_arm64.whl", hash = "sha256:8218cb8938d3031e0d23842838425490c229ecf3e99f666776e09d12ed902352", size = 7954229, upload-time = "2026-09-10T18:32:40.661Z" }, + { url = "https://files.pythonhosted.org/packages/df/a7/25f0a43d2fde306e8ef45f45121192f687b79beaf4bae8c21607c46c5e63/scikit_learn-1.9.1-cp312-cp312-macosx_10_13_x86_64.whl", hash = "sha256:0c0f8b5d09b44101cea2767f300680bada1ea27f976fe4b48b83950a4f55a49a", size = 8775209, upload-time = "2026-09-10T18:32:42.804Z" }, + { url = "https://files.pythonhosted.org/packages/60/ea/57e57539ce175d774fc291ed091b0a6d756854b92cd92554c6bb4d0ae498/scikit_learn-1.9.1-cp312-cp312-macosx_12_0_arm64.whl", hash = "sha256:8c14ce41d561f7749f990b41d6703fe02c4669fbc485e598e069e0a1967b488e", size = 8295541, upload-time = "2026-09-10T18:32:45.069Z" }, + { url = "https://files.pythonhosted.org/packages/78/2b/5721a174406bfba49bce20ae997b3b64cf355c3f623a2638284ab6a82156/scikit_learn-1.9.1-cp312-cp312-manylinux_2_27_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:e4c20a6c017d820faa7ac8c783e3d0c6a9a2e297bf9f55332ca17cdf7fd4d04d", size = 8871592, upload-time = "2026-09-10T18:32:46.999Z" }, + { url = "https://files.pythonhosted.org/packages/8e/57/a50162f3d29feb979ab6347c6debda506dfb525bcff3c50dd17606651c7e/scikit_learn-1.9.1-cp312-cp312-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:e5d7b18a5b9dca241a74695f3275fa4c895a9dadc72b3d8df5fa9d1083c9b83e", size = 9166043, upload-time = "2026-09-10T18:32:49.343Z" }, + { url = "https://files.pythonhosted.org/packages/72/8d/27c054166bac671770d1ea0ef7716134fe8119c0df224a58c89fd735a61c/scikit_learn-1.9.1-cp312-cp312-win_amd64.whl", hash = "sha256:4b59abb30618121cc46b45972d6bf53a7128b4df4cd346c6ca6f4d5f9031e49c", size = 8262238, upload-time = "2026-09-10T18:32:51.679Z" }, + { url = "https://files.pythonhosted.org/packages/9d/d6/493086006ea0c68ad62c40a8dece1961b61bf503f45400f133d47f56e5be/scikit_learn-1.9.1-cp312-cp312-win_arm64.whl", hash = "sha256:d5945a2908be62350e2978344e62b56c1552c2ca4f844ebf6277c94944d647dd", size = 7902102, upload-time = "2026-09-10T18:32:53.67Z" }, + { url = "https://files.pythonhosted.org/packages/bb/8d/b60d5e7354ff0ff5cc9400e60273696589d87a30b8b2235886a76d80d062/scikit_learn-1.9.1-cp313-cp313-macosx_10_13_x86_64.whl", hash = "sha256:2c2b312fd8c02951a364fa120ea08c1cec10d863466bf1701b013152d7537835", size = 8739300, upload-time = "2026-09-10T18:32:55.483Z" }, + { url = "https://files.pythonhosted.org/packages/2f/81/3c6392c03665d2899457a76e535a9a6f597dddddf3220fd2e1d790da88c5/scikit_learn-1.9.1-cp313-cp313-macosx_12_0_arm64.whl", hash = "sha256:61cd968ab831a76d0ecbaf0347ab2270268716da28f94fd022497e3d6f205f13", size = 8262364, upload-time = "2026-09-10T18:32:57.966Z" }, + { url = "https://files.pythonhosted.org/packages/0f/35/a15b8653499692879821301d48059376d6e68e8b65cd0f22d19b6ee83cd9/scikit_learn-1.9.1-cp313-cp313-manylinux_2_27_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:5990f9c69e431bfaddcde1a6d7c5355243e026bc9b9e560c13893b90dab53fb4", size = 8823698, upload-time = "2026-09-10T18:33:00.632Z" }, + { url = "https://files.pythonhosted.org/packages/23/e5/688703d357e5393f708d98eb189fd415ae69e39f6de03c6bd4005aef6118/scikit_learn-1.9.1-cp313-cp313-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:55e79d6e9b0923f1a978179822bd43d7f5543f45e970a00fe861f43486380aba", size = 9121732, upload-time = "2026-09-10T18:33:02.825Z" }, + { url = "https://files.pythonhosted.org/packages/96/45/a10add34c08184d373be9384660c75758128ca881ed27b503b6f6a742478/scikit_learn-1.9.1-cp313-cp313-win_amd64.whl", hash = "sha256:2070f271e5375dc42c6bb93b461ab1c0aa5841d4009267e0cfd95a39dca94a43", size = 8237244, upload-time = "2026-09-10T18:33:05.26Z" }, + { url = "https://files.pythonhosted.org/packages/9e/08/7a89bcdadd1fff0d464d01056417b646c9abcbc54f7297a0a1203bba5ebb/scikit_learn-1.9.1-cp313-cp313-win_arm64.whl", hash = "sha256:613f0a783ca05aa844a4e1ac42d48425058f2c52be73f40f8cd98b7cd111acd6", size = 7875398, upload-time = "2026-09-10T18:33:07.506Z" }, + { url = "https://files.pythonhosted.org/packages/64/e3/b58e45082dcf3dcf0eb1192ee03545ec43d8c98441dfe20e88afca8442ce/scikit_learn-1.9.1-cp314-cp314-macosx_10_15_x86_64.whl", hash = "sha256:d5d117952769b563067656784e03c75a2d8235a7a05cf7fffa78a311e75aac08", size = 8747414, upload-time = "2026-09-10T18:33:10.698Z" }, + { url = "https://files.pythonhosted.org/packages/9b/ed/d68115577c8b42b0442ebd8180945d4008880a33176094640e00b585128e/scikit_learn-1.9.1-cp314-cp314-macosx_12_0_arm64.whl", hash = "sha256:8893bc6331f60f18d4ac75e12ed356e2dcf6a564bf767918b5b7ca54c8c8be49", size = 8279175, upload-time = "2026-09-10T18:33:12.846Z" }, + { url = "https://files.pythonhosted.org/packages/c4/6c/06c7eb61a438e389cbf3f7210897069bec5a883dbe03c189ae792781e11a/scikit_learn-1.9.1-cp314-cp314-manylinux_2_27_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:b5492cf2df5226691c32611de8734bcf42148c6547ae53c7f4e6b847793addc0", size = 8881706, upload-time = "2026-09-10T18:33:14.741Z" }, + { url = "https://files.pythonhosted.org/packages/86/4e/0bab75490ca4b85fad8388739c7ebc71d9db553f8c69e39943ee8db0aaae/scikit_learn-1.9.1-cp314-cp314-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:993d332ff80e62efae9e39603b7e872297c418d780f01a01855269a3489c950f", size = 9152030, upload-time = "2026-09-10T18:33:17.436Z" }, + { url = "https://files.pythonhosted.org/packages/9a/13/31c6f8ba1b7eecef9dd9558576c752d2ec5785fd0e456bb9fc59305be23f/scikit_learn-1.9.1-cp314-cp314-win_amd64.whl", hash = "sha256:ca9051447455dae341d4d591eece7deb2d8e3d1020298fc87a81fc51e4da8f53", size = 8377024, upload-time = "2026-09-10T18:33:19.941Z" }, + { url = "https://files.pythonhosted.org/packages/62/e6/6d3cb8a45f5228f915acd66b819dd6b8232ccbe24532f51d278e3991df31/scikit_learn-1.9.1-cp314-cp314-win_arm64.whl", hash = "sha256:90de6573f733a9fb79476ff1371af52a397d41c8b35f9146e20923db010d67b6", size = 8014196, upload-time = "2026-09-10T18:33:22.126Z" }, + { url = "https://files.pythonhosted.org/packages/66/6e/6befb2d5961490d18d9dbc16a5df37aa121d08bc9893316a6a363977a903/scikit_learn-1.9.1-cp314-cp314t-macosx_10_15_x86_64.whl", hash = "sha256:7b5cad1624de8b75e5b9ccb7b0ce1ff1d01306340a3efc56d5529c5ba92392eb", size = 9066066, upload-time = "2026-09-10T18:33:24.396Z" }, + { url = "https://files.pythonhosted.org/packages/cb/18/11271f2f7db337db01f598e358721b1e83989407131272e5dd64214c28a8/scikit_learn-1.9.1-cp314-cp314t-macosx_12_0_arm64.whl", hash = "sha256:d137ce8a6142029fb5c35bd82f470c40cd9e760e5e2f7694b362c497c4ab3fa2", size = 8647654, upload-time = "2026-09-10T18:33:26.649Z" }, + { url = "https://files.pythonhosted.org/packages/e7/04/9c15d201e1b6a2e81b8215865df7646c5a360f560831769c8dc92ac1ab9a/scikit_learn-1.9.1-cp314-cp314t-manylinux_2_27_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:66f852f7325b5070bc28329005aca76055a2def78faac039548ae889aeaa45a6", size = 8889070, upload-time = "2026-09-10T18:33:28.95Z" }, + { url = "https://files.pythonhosted.org/packages/1a/5a/4cb6c85160af4a639e87a3b7bf8b1c25cfc3b504c5af710ca416a6dcfc5f/scikit_learn-1.9.1-cp314-cp314t-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:748bcb0a4cc04aec470652c9e5ec68450948e867387e7dfade647107ade68d25", size = 9131227, upload-time = "2026-09-10T18:33:31.008Z" }, + { url = "https://files.pythonhosted.org/packages/00/0e/361440972ae3d19b90ea88a84791138a51de0e8432a770ba741b2c8d9ced/scikit_learn-1.9.1-cp314-cp314t-win_amd64.whl", hash = "sha256:38cd925e893e5539be704d5edc64dbe081aacdab6b89d8c2977c1f6a7a453ce5", size = 8683001, upload-time = "2026-09-10T18:33:33.188Z" }, + { url = "https://files.pythonhosted.org/packages/e2/8f/a9f405c5c0e2df6f343a871b40c97fb32969e3ccc38e3033dd118f3c261e/scikit_learn-1.9.1-cp314-cp314t-win_arm64.whl", hash = "sha256:b01e5b01735d38474127ca3f49319b592506225a87793b27559816b5c75cea39", size = 8257996, upload-time = "2026-09-10T18:33:35.343Z" }, + { url = "https://files.pythonhosted.org/packages/e5/c5/74a83ea39cef7cd07f53e06cc1cf51e79f35df74f81db956835d59ec34b1/scikit_learn-1.9.1-cp315-cp315-macosx_10_15_x86_64.whl", hash = "sha256:8dec64f31a6e0ec826aca6c1b39a51e16d946e400d4c0904316f3ca72ccfb825", size = 8747991, upload-time = "2026-09-10T18:33:37.4Z" }, + { url = "https://files.pythonhosted.org/packages/64/c9/cc93e8a7fe204e43d70e96e5eb89871643be20025eea05eb4fdaf19afe39/scikit_learn-1.9.1-cp315-cp315-macosx_12_0_arm64.whl", hash = "sha256:e1b468241f4a7a9a7a0d6479ad3cc47681cc151a4046c530f2777c3d68f08942", size = 8279340, upload-time = "2026-09-10T18:33:39.705Z" }, + { url = "https://files.pythonhosted.org/packages/a2/61/0c6080f0d356fb966053009e7f25e9bff6cf74b0b16195ccf0c3757d1ae8/scikit_learn-1.9.1-cp315-cp315-manylinux_2_27_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:8ca869d0080a5723cde2d5a8b54a2da1ff7e68735a9e9adb3da1243183a0fa01", size = 8881917, upload-time = "2026-09-10T18:33:41.912Z" }, + { url = "https://files.pythonhosted.org/packages/47/bb/98a31f10fffbd39edcc2f8bf4119b29652248bb110b7d45c84e68aa293ab/scikit_learn-1.9.1-cp315-cp315-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:6754b7cabfc3df0b1f7b38f7a344f559bbae9d82f0ac5e3d48cccbd19fdcefdf", size = 9152805, upload-time = "2026-09-10T18:33:44.23Z" }, + { url = "https://files.pythonhosted.org/packages/f6/48/858ceff52213cfd97c0a069362071756bcb70b9fac9771b38d87c4cf7f17/scikit_learn-1.9.1-cp315-cp315-win_amd64.whl", hash = "sha256:52cfdb1fed3a34362dbc0bd96f2e761a66fd5724d6901629f5a558f1f3bd9849", size = 8376719, upload-time = "2026-09-10T18:33:46.492Z" }, + { url = "https://files.pythonhosted.org/packages/2a/1e/5337a871bdea53effbd154b61429df048f2665653251de74a3bd8a6dea9e/scikit_learn-1.9.1-cp315-cp315-win_arm64.whl", hash = "sha256:ae6571a4828c6f5019bcd2b4125e5b18c0af3dbc9c99726c891f45f41335ec8e", size = 8014341, upload-time = "2026-09-10T18:33:48.762Z" }, + { url = "https://files.pythonhosted.org/packages/0e/35/150383a42d83ec4c7b39f9c50bd68408ecf04c19fc30ea5198fa42e67d9c/scikit_learn-1.9.1-cp315-cp315t-macosx_10_15_x86_64.whl", hash = "sha256:48fefd8eb42bd4eec3e2d348149368ccd6d71987e20c30706a56a24eb86a6e73", size = 9059953, upload-time = "2026-09-10T18:33:50.951Z" }, + { url = "https://files.pythonhosted.org/packages/9f/dd/aa0d738808540f7eaacfab93e01982db8ef1c1c7473ef0ad38193e6aebd1/scikit_learn-1.9.1-cp315-cp315t-macosx_12_0_arm64.whl", hash = "sha256:09f4d73049cd63575157f6b1060e06a8c83a4bd3488dbfaeedf35ccba7aad712", size = 8646602, upload-time = "2026-09-10T18:33:53.23Z" }, + { url = "https://files.pythonhosted.org/packages/7e/cc/687ae4214c2f598906c3b9fa5f86fbaf834b35e20360625528ff1b713f06/scikit_learn-1.9.1-cp315-cp315t-manylinux_2_27_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:b3da53831534214322d9cb240fa6f390b36cf69eba727a6d4bd3238677630d70", size = 8880366, upload-time = "2026-09-10T18:33:55.764Z" }, + { url = "https://files.pythonhosted.org/packages/c2/03/82215cb78ad1c513a4498777571fb28444621ad26ef636287551767b7732/scikit_learn-1.9.1-cp315-cp315t-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:caae15634feceafa2612566b109a3082d3293167fac388eedaf77bff66b51983", size = 9129640, upload-time = "2026-09-10T18:33:58.186Z" }, + { url = "https://files.pythonhosted.org/packages/d4/90/4b7af4efd7909a4a0524a9f18457e4eb2eb60616eff2c7627eda8e3cdceb/scikit_learn-1.9.1-cp315-cp315t-win_amd64.whl", hash = "sha256:ffbcbbbb44202fbe9bc64bced25a145759adb9ef010b3d37a8064958ac13df2a", size = 8678083, upload-time = "2026-09-10T18:34:00.354Z" }, + { url = "https://files.pythonhosted.org/packages/31/27/068e484d4b83004302e0d9cfc1faca69bcc010d76fbb66a642446095af1b/scikit_learn-1.9.1-cp315-cp315t-win_arm64.whl", hash = "sha256:800dd22dd87fe97dcea484c24e85dd93cf1734d86bd74e668ad18f7967f4d1b5", size = 8259434, upload-time = "2026-09-10T18:34:02.678Z" }, +] + +[[package]] +name = "scipy" +version = "1.17.1" +source = { registry = "https://pypi.org/simple" } +resolution-markers = [ + "python_full_version < '3.12'", +] +dependencies = [ + { name = "numpy", version = "2.4.6", source = { registry = "https://pypi.org/simple" } }, +] +sdist = { url = "https://files.pythonhosted.org/packages/7a/97/5a3609c4f8d58b039179648e62dd220f89864f56f7357f5d4f45c29eb2cc/scipy-1.17.1.tar.gz", hash = "sha256:95d8e012d8cb8816c226aef832200b1d45109ed4464303e997c5b13122b297c0", size = 30573822, upload-time = "2026-02-23T00:26:24.851Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/df/75/b4ce781849931fef6fd529afa6b63711d5a733065722d0c3e2724af9e40a/scipy-1.17.1-cp311-cp311-macosx_10_14_x86_64.whl", hash = "sha256:1f95b894f13729334fb990162e911c9e5dc1ab390c58aa6cbecb389c5b5e28ec", size = 31613675, upload-time = "2026-02-23T00:16:00.13Z" }, + { url = "https://files.pythonhosted.org/packages/f7/58/bccc2861b305abdd1b8663d6130c0b3d7cc22e8d86663edbc8401bfd40d4/scipy-1.17.1-cp311-cp311-macosx_12_0_arm64.whl", hash = "sha256:e18f12c6b0bc5a592ed23d3f7b891f68fd7f8241d69b7883769eb5d5dfb52696", size = 28162057, upload-time = "2026-02-23T00:16:09.456Z" }, + { url = "https://files.pythonhosted.org/packages/6d/ee/18146b7757ed4976276b9c9819108adbc73c5aad636e5353e20746b73069/scipy-1.17.1-cp311-cp311-macosx_14_0_arm64.whl", hash = "sha256:a3472cfbca0a54177d0faa68f697d8ba4c80bbdc19908c3465556d9f7efce9ee", size = 20334032, upload-time = "2026-02-23T00:16:17.358Z" }, + { url = "https://files.pythonhosted.org/packages/ec/e6/cef1cf3557f0c54954198554a10016b6a03b2ec9e22a4e1df734936bd99c/scipy-1.17.1-cp311-cp311-macosx_14_0_x86_64.whl", hash = "sha256:766e0dc5a616d026a3a1cffa379af959671729083882f50307e18175797b3dfd", size = 22709533, upload-time = "2026-02-23T00:16:25.791Z" }, + { url = "https://files.pythonhosted.org/packages/4d/60/8804678875fc59362b0fb759ab3ecce1f09c10a735680318ac30da8cd76b/scipy-1.17.1-cp311-cp311-manylinux_2_27_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:744b2bf3640d907b79f3fd7874efe432d1cf171ee721243e350f55234b4cec4c", size = 33062057, upload-time = "2026-02-23T00:16:36.931Z" }, + { url = "https://files.pythonhosted.org/packages/09/7d/af933f0f6e0767995b4e2d705a0665e454d1c19402aa7e895de3951ebb04/scipy-1.17.1-cp311-cp311-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:43af8d1f3bea642559019edfe64e9b11192a8978efbd1539d7bc2aaa23d92de4", size = 35349300, upload-time = "2026-02-23T00:16:49.108Z" }, + { url = "https://files.pythonhosted.org/packages/b4/3d/7ccbbdcbb54c8fdc20d3b6930137c782a163fa626f0aef920349873421ba/scipy-1.17.1-cp311-cp311-musllinux_1_2_aarch64.whl", hash = "sha256:cd96a1898c0a47be4520327e01f874acfd61fb48a9420f8aa9f6483412ffa444", size = 35127333, upload-time = "2026-02-23T00:17:01.293Z" }, + { url = "https://files.pythonhosted.org/packages/e8/19/f926cb11c42b15ba08e3a71e376d816ac08614f769b4f47e06c3580c836a/scipy-1.17.1-cp311-cp311-musllinux_1_2_x86_64.whl", hash = "sha256:4eb6c25dd62ee8d5edf68a8e1c171dd71c292fdae95d8aeb3dd7d7de4c364082", size = 37741314, upload-time = "2026-02-23T00:17:12.576Z" }, + { url = "https://files.pythonhosted.org/packages/95/da/0d1df507cf574b3f224ccc3d45244c9a1d732c81dcb26b1e8a766ae271a8/scipy-1.17.1-cp311-cp311-win_amd64.whl", hash = "sha256:d30e57c72013c2a4fe441c2fcb8e77b14e152ad48b5464858e07e2ad9fbfceff", size = 36607512, upload-time = "2026-02-23T00:17:23.424Z" }, + { url = "https://files.pythonhosted.org/packages/68/7f/bdd79ceaad24b671543ffe0ef61ed8e659440eb683b66f033454dcee90eb/scipy-1.17.1-cp311-cp311-win_arm64.whl", hash = "sha256:9ecb4efb1cd6e8c4afea0daa91a87fbddbce1b99d2895d151596716c0b2e859d", size = 24599248, upload-time = "2026-02-23T00:17:34.561Z" }, + { url = "https://files.pythonhosted.org/packages/35/48/b992b488d6f299dbe3f11a20b24d3dda3d46f1a635ede1c46b5b17a7b163/scipy-1.17.1-cp312-cp312-macosx_10_14_x86_64.whl", hash = "sha256:35c3a56d2ef83efc372eaec584314bd0ef2e2f0d2adb21c55e6ad5b344c0dcb8", size = 31610954, upload-time = "2026-02-23T00:17:49.855Z" }, + { url = "https://files.pythonhosted.org/packages/b2/02/cf107b01494c19dc100f1d0b7ac3cc08666e96ba2d64db7626066cee895e/scipy-1.17.1-cp312-cp312-macosx_12_0_arm64.whl", hash = "sha256:fcb310ddb270a06114bb64bbe53c94926b943f5b7f0842194d585c65eb4edd76", size = 28172662, upload-time = "2026-02-23T00:18:01.64Z" }, + { url = "https://files.pythonhosted.org/packages/cf/a9/599c28631bad314d219cf9ffd40e985b24d603fc8a2f4ccc5ae8419a535b/scipy-1.17.1-cp312-cp312-macosx_14_0_arm64.whl", hash = "sha256:cc90d2e9c7e5c7f1a482c9875007c095c3194b1cfedca3c2f3291cdc2bc7c086", size = 20344366, upload-time = "2026-02-23T00:18:12.015Z" }, + { url = "https://files.pythonhosted.org/packages/35/f5/906eda513271c8deb5af284e5ef0206d17a96239af79f9fa0aebfe0e36b4/scipy-1.17.1-cp312-cp312-macosx_14_0_x86_64.whl", hash = "sha256:c80be5ede8f3f8eded4eff73cc99a25c388ce98e555b17d31da05287015ffa5b", size = 22704017, upload-time = "2026-02-23T00:18:21.502Z" }, + { url = "https://files.pythonhosted.org/packages/da/34/16f10e3042d2f1d6b66e0428308ab52224b6a23049cb2f5c1756f713815f/scipy-1.17.1-cp312-cp312-manylinux_2_27_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:e19ebea31758fac5893a2ac360fedd00116cbb7628e650842a6691ba7ca28a21", size = 32927842, upload-time = "2026-02-23T00:18:35.367Z" }, + { url = "https://files.pythonhosted.org/packages/01/8e/1e35281b8ab6d5d72ebe9911edcdffa3f36b04ed9d51dec6dd140396e220/scipy-1.17.1-cp312-cp312-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:02ae3b274fde71c5e92ac4d54bc06c42d80e399fec704383dcd99b301df37458", size = 35235890, upload-time = "2026-02-23T00:18:49.188Z" }, + { url = "https://files.pythonhosted.org/packages/c5/5c/9d7f4c88bea6e0d5a4f1bc0506a53a00e9fcb198de372bfe4d3652cef482/scipy-1.17.1-cp312-cp312-musllinux_1_2_aarch64.whl", hash = "sha256:8a604bae87c6195d8b1045eddece0514d041604b14f2727bbc2b3020172045eb", size = 35003557, upload-time = "2026-02-23T00:18:54.74Z" }, + { url = "https://files.pythonhosted.org/packages/65/94/7698add8f276dbab7a9de9fb6b0e02fc13ee61d51c7c3f85ac28b65e1239/scipy-1.17.1-cp312-cp312-musllinux_1_2_x86_64.whl", hash = "sha256:f590cd684941912d10becc07325a3eeb77886fe981415660d9265c4c418d0bea", size = 37625856, upload-time = "2026-02-23T00:19:00.307Z" }, + { url = "https://files.pythonhosted.org/packages/a2/84/dc08d77fbf3d87d3ee27f6a0c6dcce1de5829a64f2eae85a0ecc1f0daa73/scipy-1.17.1-cp312-cp312-win_amd64.whl", hash = "sha256:41b71f4a3a4cab9d366cd9065b288efc4d4f3c0b37a91a8e0947fb5bd7f31d87", size = 36549682, upload-time = "2026-02-23T00:19:07.67Z" }, + { url = "https://files.pythonhosted.org/packages/bc/98/fe9ae9ffb3b54b62559f52dedaebe204b408db8109a8c66fdd04869e6424/scipy-1.17.1-cp312-cp312-win_arm64.whl", hash = "sha256:f4115102802df98b2b0db3cce5cb9b92572633a1197c77b7553e5203f284a5b3", size = 24547340, upload-time = "2026-02-23T00:19:12.024Z" }, + { url = "https://files.pythonhosted.org/packages/76/27/07ee1b57b65e92645f219b37148a7e7928b82e2b5dbeccecb4dff7c64f0b/scipy-1.17.1-cp313-cp313-macosx_10_14_x86_64.whl", hash = "sha256:5e3c5c011904115f88a39308379c17f91546f77c1667cea98739fe0fccea804c", size = 31590199, upload-time = "2026-02-23T00:19:17.192Z" }, + { url = "https://files.pythonhosted.org/packages/ec/ae/db19f8ab842e9b724bf5dbb7db29302a91f1e55bc4d04b1025d6d605a2c5/scipy-1.17.1-cp313-cp313-macosx_12_0_arm64.whl", hash = "sha256:6fac755ca3d2c3edcb22f479fceaa241704111414831ddd3bc6056e18516892f", size = 28154001, upload-time = "2026-02-23T00:19:22.241Z" }, + { url = "https://files.pythonhosted.org/packages/5b/58/3ce96251560107b381cbd6e8413c483bbb1228a6b919fa8652b0d4090e7f/scipy-1.17.1-cp313-cp313-macosx_14_0_arm64.whl", hash = "sha256:7ff200bf9d24f2e4d5dc6ee8c3ac64d739d3a89e2326ba68aaf6c4a2b838fd7d", size = 20325719, upload-time = "2026-02-23T00:19:26.329Z" }, + { url = "https://files.pythonhosted.org/packages/b2/83/15087d945e0e4d48ce2377498abf5ad171ae013232ae31d06f336e64c999/scipy-1.17.1-cp313-cp313-macosx_14_0_x86_64.whl", hash = "sha256:4b400bdc6f79fa02a4d86640310dde87a21fba0c979efff5248908c6f15fad1b", size = 22683595, upload-time = "2026-02-23T00:19:30.304Z" }, + { url = "https://files.pythonhosted.org/packages/b4/e0/e58fbde4a1a594c8be8114eb4aac1a55bcd6587047efc18a61eb1f5c0d30/scipy-1.17.1-cp313-cp313-manylinux_2_27_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:2b64ca7d4aee0102a97f3ba22124052b4bd2152522355073580bf4845e2550b6", size = 32896429, upload-time = "2026-02-23T00:19:35.536Z" }, + { url = "https://files.pythonhosted.org/packages/f5/5f/f17563f28ff03c7b6799c50d01d5d856a1d55f2676f537ca8d28c7f627cd/scipy-1.17.1-cp313-cp313-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:581b2264fc0aa555f3f435a5944da7504ea3a065d7029ad60e7c3d1ae09c5464", size = 35203952, upload-time = "2026-02-23T00:19:42.259Z" }, + { url = "https://files.pythonhosted.org/packages/8d/a5/9afd17de24f657fdfe4df9a3f1ea049b39aef7c06000c13db1530d81ccca/scipy-1.17.1-cp313-cp313-musllinux_1_2_aarch64.whl", hash = "sha256:beeda3d4ae615106d7094f7e7cef6218392e4465cc95d25f900bebabfded0950", size = 34979063, upload-time = "2026-02-23T00:19:47.547Z" }, + { url = "https://files.pythonhosted.org/packages/8b/13/88b1d2384b424bf7c924f2038c1c409f8d88bb2a8d49d097861dd64a57b2/scipy-1.17.1-cp313-cp313-musllinux_1_2_x86_64.whl", hash = "sha256:6609bc224e9568f65064cfa72edc0f24ee6655b47575954ec6339534b2798369", size = 37598449, upload-time = "2026-02-23T00:19:53.238Z" }, + { url = "https://files.pythonhosted.org/packages/35/e5/d6d0e51fc888f692a35134336866341c08655d92614f492c6860dc45bb2c/scipy-1.17.1-cp313-cp313-win_amd64.whl", hash = "sha256:37425bc9175607b0268f493d79a292c39f9d001a357bebb6b88fdfaff13f6448", size = 36510943, upload-time = "2026-02-23T00:20:50.89Z" }, + { url = "https://files.pythonhosted.org/packages/2a/fd/3be73c564e2a01e690e19cc618811540ba5354c67c8680dce3281123fb79/scipy-1.17.1-cp313-cp313-win_arm64.whl", hash = "sha256:5cf36e801231b6a2059bf354720274b7558746f3b1a4efb43fcf557ccd484a87", size = 24545621, upload-time = "2026-02-23T00:20:55.871Z" }, + { url = "https://files.pythonhosted.org/packages/6f/6b/17787db8b8114933a66f9dcc479a8272e4b4da75fe03b0c282f7b0ade8cd/scipy-1.17.1-cp313-cp313t-macosx_10_14_x86_64.whl", hash = "sha256:d59c30000a16d8edc7e64152e30220bfbd724c9bbb08368c054e24c651314f0a", size = 31936708, upload-time = "2026-02-23T00:19:58.694Z" }, + { url = "https://files.pythonhosted.org/packages/38/2e/524405c2b6392765ab1e2b722a41d5da33dc5c7b7278184a8ad29b6cb206/scipy-1.17.1-cp313-cp313t-macosx_12_0_arm64.whl", hash = "sha256:010f4333c96c9bb1a4516269e33cb5917b08ef2166d5556ca2fd9f082a9e6ea0", size = 28570135, upload-time = "2026-02-23T00:20:03.934Z" }, + { url = "https://files.pythonhosted.org/packages/fd/c3/5bd7199f4ea8556c0c8e39f04ccb014ac37d1468e6cfa6a95c6b3562b76e/scipy-1.17.1-cp313-cp313t-macosx_14_0_arm64.whl", hash = "sha256:2ceb2d3e01c5f1d83c4189737a42d9cb2fc38a6eeed225e7515eef71ad301dce", size = 20741977, upload-time = "2026-02-23T00:20:07.935Z" }, + { url = "https://files.pythonhosted.org/packages/d9/b8/8ccd9b766ad14c78386599708eb745f6b44f08400a5fd0ade7cf89b6fc93/scipy-1.17.1-cp313-cp313t-macosx_14_0_x86_64.whl", hash = "sha256:844e165636711ef41f80b4103ed234181646b98a53c8f05da12ca5ca289134f6", size = 23029601, upload-time = "2026-02-23T00:20:12.161Z" }, + { url = "https://files.pythonhosted.org/packages/6d/a0/3cb6f4d2fb3e17428ad2880333cac878909ad1a89f678527b5328b93c1d4/scipy-1.17.1-cp313-cp313t-manylinux_2_27_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:158dd96d2207e21c966063e1635b1063cd7787b627b6f07305315dd73d9c679e", size = 33019667, upload-time = "2026-02-23T00:20:17.208Z" }, + { url = "https://files.pythonhosted.org/packages/f3/c3/2d834a5ac7bf3a0c806ad1508efc02dda3c8c61472a56132d7894c312dea/scipy-1.17.1-cp313-cp313t-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:74cbb80d93260fe2ffa334efa24cb8f2f0f622a9b9febf8b483c0b865bfb3475", size = 35264159, upload-time = "2026-02-23T00:20:23.087Z" }, + { url = "https://files.pythonhosted.org/packages/4d/77/d3ed4becfdbd217c52062fafe35a72388d1bd82c2d0ba5ca19d6fcc93e11/scipy-1.17.1-cp313-cp313t-musllinux_1_2_aarch64.whl", hash = "sha256:dbc12c9f3d185f5c737d801da555fb74b3dcfa1a50b66a1a93e09190f41fab50", size = 35102771, upload-time = "2026-02-23T00:20:28.636Z" }, + { url = "https://files.pythonhosted.org/packages/bd/12/d19da97efde68ca1ee5538bb261d5d2c062f0c055575128f11a2730e3ac1/scipy-1.17.1-cp313-cp313t-musllinux_1_2_x86_64.whl", hash = "sha256:94055a11dfebe37c656e70317e1996dc197e1a15bbcc351bcdd4610e128fe1ca", size = 37665910, upload-time = "2026-02-23T00:20:34.743Z" }, + { url = "https://files.pythonhosted.org/packages/06/1c/1172a88d507a4baaf72c5a09bb6c018fe2ae0ab622e5830b703a46cc9e44/scipy-1.17.1-cp313-cp313t-win_amd64.whl", hash = "sha256:e30bdeaa5deed6bc27b4cc490823cd0347d7dae09119b8803ae576ea0ce52e4c", size = 36562980, upload-time = "2026-02-23T00:20:40.575Z" }, + { url = "https://files.pythonhosted.org/packages/70/b0/eb757336e5a76dfa7911f63252e3b7d1de00935d7705cf772db5b45ec238/scipy-1.17.1-cp313-cp313t-win_arm64.whl", hash = "sha256:a720477885a9d2411f94a93d16f9d89bad0f28ca23c3f8daa521e2dcc3f44d49", size = 24856543, upload-time = "2026-02-23T00:20:45.313Z" }, + { url = "https://files.pythonhosted.org/packages/cf/83/333afb452af6f0fd70414dc04f898647ee1423979ce02efa75c3b0f2c28e/scipy-1.17.1-cp314-cp314-macosx_10_14_x86_64.whl", hash = "sha256:a48a72c77a310327f6a3a920092fa2b8fd03d7deaa60f093038f22d98e096717", size = 31584510, upload-time = "2026-02-23T00:21:01.015Z" }, + { url = "https://files.pythonhosted.org/packages/ed/a6/d05a85fd51daeb2e4ea71d102f15b34fedca8e931af02594193ae4fd25f7/scipy-1.17.1-cp314-cp314-macosx_12_0_arm64.whl", hash = "sha256:45abad819184f07240d8a696117a7aacd39787af9e0b719d00285549ed19a1e9", size = 28170131, upload-time = "2026-02-23T00:21:05.888Z" }, + { url = "https://files.pythonhosted.org/packages/db/7b/8624a203326675d7746a254083a187398090a179335b2e4a20e2ddc46e83/scipy-1.17.1-cp314-cp314-macosx_14_0_arm64.whl", hash = "sha256:3fd1fcdab3ea951b610dc4cef356d416d5802991e7e32b5254828d342f7b7e0b", size = 20342032, upload-time = "2026-02-23T00:21:09.904Z" }, + { url = "https://files.pythonhosted.org/packages/c9/35/2c342897c00775d688d8ff3987aced3426858fd89d5a0e26e020b660b301/scipy-1.17.1-cp314-cp314-macosx_14_0_x86_64.whl", hash = "sha256:7bdf2da170b67fdf10bca777614b1c7d96ae3ca5794fd9587dce41eb2966e866", size = 22678766, upload-time = "2026-02-23T00:21:14.313Z" }, + { url = "https://files.pythonhosted.org/packages/ef/f2/7cdb8eb308a1a6ae1e19f945913c82c23c0c442a462a46480ce487fdc0ac/scipy-1.17.1-cp314-cp314-manylinux_2_27_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:adb2642e060a6549c343603a3851ba76ef0b74cc8c079a9a58121c7ec9fe2350", size = 32957007, upload-time = "2026-02-23T00:21:19.663Z" }, + { url = "https://files.pythonhosted.org/packages/0b/2e/7eea398450457ecb54e18e9d10110993fa65561c4f3add5e8eccd2b9cd41/scipy-1.17.1-cp314-cp314-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:eee2cfda04c00a857206a4330f0c5e3e56535494e30ca445eb19ec624ae75118", size = 35221333, upload-time = "2026-02-23T00:21:25.278Z" }, + { url = "https://files.pythonhosted.org/packages/d9/77/5b8509d03b77f093a0d52e606d3c4f79e8b06d1d38c441dacb1e26cacf46/scipy-1.17.1-cp314-cp314-musllinux_1_2_aarch64.whl", hash = "sha256:d2650c1fb97e184d12d8ba010493ee7b322864f7d3d00d3f9bb97d9c21de4068", size = 35042066, upload-time = "2026-02-23T00:21:31.358Z" }, + { url = "https://files.pythonhosted.org/packages/f9/df/18f80fb99df40b4070328d5ae5c596f2f00fffb50167e31439e932f29e7d/scipy-1.17.1-cp314-cp314-musllinux_1_2_x86_64.whl", hash = "sha256:08b900519463543aa604a06bec02461558a6e1cef8fdbb8098f77a48a83c8118", size = 37612763, upload-time = "2026-02-23T00:21:37.247Z" }, + { url = "https://files.pythonhosted.org/packages/4b/39/f0e8ea762a764a9dc52aa7dabcfad51a354819de1f0d4652b6a1122424d6/scipy-1.17.1-cp314-cp314-win_amd64.whl", hash = "sha256:3877ac408e14da24a6196de0ddcace62092bfc12a83823e92e49e40747e52c19", size = 37290984, upload-time = "2026-02-23T00:22:35.023Z" }, + { url = "https://files.pythonhosted.org/packages/7c/56/fe201e3b0f93d1a8bcf75d3379affd228a63d7e2d80ab45467a74b494947/scipy-1.17.1-cp314-cp314-win_arm64.whl", hash = "sha256:f8885db0bc2bffa59d5c1b72fad7a6a92d3e80e7257f967dd81abb553a90d293", size = 25192877, upload-time = "2026-02-23T00:22:39.798Z" }, + { url = "https://files.pythonhosted.org/packages/96/ad/f8c414e121f82e02d76f310f16db9899c4fcde36710329502a6b2a3c0392/scipy-1.17.1-cp314-cp314t-macosx_10_14_x86_64.whl", hash = "sha256:1cc682cea2ae55524432f3cdff9e9a3be743d52a7443d0cba9017c23c87ae2f6", size = 31949750, upload-time = "2026-02-23T00:21:42.289Z" }, + { url = "https://files.pythonhosted.org/packages/7c/b0/c741e8865d61b67c81e255f4f0a832846c064e426636cd7de84e74d209be/scipy-1.17.1-cp314-cp314t-macosx_12_0_arm64.whl", hash = "sha256:2040ad4d1795a0ae89bfc7e8429677f365d45aa9fd5e4587cf1ea737f927b4a1", size = 28585858, upload-time = "2026-02-23T00:21:47.706Z" }, + { url = "https://files.pythonhosted.org/packages/ed/1b/3985219c6177866628fa7c2595bfd23f193ceebbe472c98a08824b9466ff/scipy-1.17.1-cp314-cp314t-macosx_14_0_arm64.whl", hash = "sha256:131f5aaea57602008f9822e2115029b55d4b5f7c070287699fe45c661d051e39", size = 20757723, upload-time = "2026-02-23T00:21:52.039Z" }, + { url = "https://files.pythonhosted.org/packages/c0/19/2a04aa25050d656d6f7b9e7b685cc83d6957fb101665bfd9369ca6534563/scipy-1.17.1-cp314-cp314t-macosx_14_0_x86_64.whl", hash = "sha256:9cdc1a2fcfd5c52cfb3045feb399f7b3ce822abdde3a193a6b9a60b3cb5854ca", size = 23043098, upload-time = "2026-02-23T00:21:56.185Z" }, + { url = "https://files.pythonhosted.org/packages/86/f1/3383beb9b5d0dbddd030335bf8a8b32d4317185efe495374f134d8be6cce/scipy-1.17.1-cp314-cp314t-manylinux_2_27_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:6e3dcd57ab780c741fde8dc68619de988b966db759a3c3152e8e9142c26295ad", size = 33030397, upload-time = "2026-02-23T00:22:01.404Z" }, + { url = "https://files.pythonhosted.org/packages/41/68/8f21e8a65a5a03f25a79165ec9d2b28c00e66dc80546cf5eb803aeeff35b/scipy-1.17.1-cp314-cp314t-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:a9956e4d4f4a301ebf6cde39850333a6b6110799d470dbbb1e25326ac447f52a", size = 35281163, upload-time = "2026-02-23T00:22:07.024Z" }, + { url = "https://files.pythonhosted.org/packages/84/8d/c8a5e19479554007a5632ed7529e665c315ae7492b4f946b0deb39870e39/scipy-1.17.1-cp314-cp314t-musllinux_1_2_aarch64.whl", hash = "sha256:a4328d245944d09fd639771de275701ccadf5f781ba0ff092ad141e017eccda4", size = 35116291, upload-time = "2026-02-23T00:22:12.585Z" }, + { url = "https://files.pythonhosted.org/packages/52/52/e57eceff0e342a1f50e274264ed47497b59e6a4e3118808ee58ddda7b74a/scipy-1.17.1-cp314-cp314t-musllinux_1_2_x86_64.whl", hash = "sha256:a77cbd07b940d326d39a1d1b37817e2ee4d79cb30e7338f3d0cddffae70fcaa2", size = 37682317, upload-time = "2026-02-23T00:22:18.513Z" }, + { url = "https://files.pythonhosted.org/packages/11/2f/b29eafe4a3fbc3d6de9662b36e028d5f039e72d345e05c250e121a230dd4/scipy-1.17.1-cp314-cp314t-win_amd64.whl", hash = "sha256:eb092099205ef62cd1782b006658db09e2fed75bffcae7cc0d44052d8aa0f484", size = 37345327, upload-time = "2026-02-23T00:22:24.442Z" }, + { url = "https://files.pythonhosted.org/packages/07/39/338d9219c4e87f3e708f18857ecd24d22a0c3094752393319553096b98af/scipy-1.17.1-cp314-cp314t-win_arm64.whl", hash = "sha256:200e1050faffacc162be6a486a984a0497866ec54149a01270adc8a59b7c7d21", size = 25489165, upload-time = "2026-02-23T00:22:29.563Z" }, +] + +[[package]] +name = "scipy" +version = "1.18.1" +source = { registry = "https://pypi.org/simple" } +resolution-markers = [ + "python_full_version >= '3.15'", + "python_full_version >= '3.12' and python_full_version < '3.15'", +] +dependencies = [ + { name = "numpy", version = "2.5.4", source = { registry = "https://pypi.org/simple" } }, +] +sdist = { url = "https://files.pythonhosted.org/packages/7e/74/66de6258867beb2ef08f35f9f2ac017a52cacd5081714d239ff1a442d458/scipy-1.18.1.tar.gz", hash = "sha256:52c4b7422442aba924d03ad4019852b08a92e64ea187b933135687bfe2747307", size = 30781235, upload-time = "2026-08-21T23:28:50.599Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/18/f7/240c110c08693826b4513a52f5717d62ec7c7af72f2920821247c03b17b3/scipy-1.18.1-cp312-cp312-macosx_10_15_x86_64.whl", hash = "sha256:457fd7a2a8edeb044ab6ffbc0aa03ff6cd18491356e5e0c834d76ce621b916d1", size = 31111061, upload-time = "2026-08-21T23:23:44.522Z" }, + { url = "https://files.pythonhosted.org/packages/05/4a/78c6285577c375e7cf27277ea8ee6961224327f1e1a0c44af5f17f23635c/scipy-1.18.1-cp312-cp312-macosx_12_0_arm64.whl", hash = "sha256:e708533e8b2ae2497d65346538a7dcc92814410b25b81432eac66de0f2af8265", size = 28733332, upload-time = "2026-08-21T23:23:50.015Z" }, + { url = "https://files.pythonhosted.org/packages/a5/f6/a5b82f8abbe14d134691b8b903696f701d25a081353a29dc655c364d9e62/scipy-1.18.1-cp312-cp312-macosx_14_0_arm64.whl", hash = "sha256:7bbf207c4453ce1ad2e00b17313852b33310b83090c2311bdaf97f93c0380d12", size = 20475078, upload-time = "2026-08-21T23:23:54.138Z" }, + { url = "https://files.pythonhosted.org/packages/23/22/0858a0bbd6b3e825ceb8cd9baf9eaf3b2f2b1d77727eb6be40500bcdc92f/scipy-1.18.1-cp312-cp312-macosx_14_0_x86_64.whl", hash = "sha256:78c0665edead396b1abb4897c41a5c1d9bf090c8a637a4c20a61678e0a264e66", size = 23108904, upload-time = "2026-08-21T23:23:57.824Z" }, + { url = "https://files.pythonhosted.org/packages/75/9a/2e71719f31eaefe0e3a1706c4a1ded94e664bfd95ffca2b219a671faee01/scipy-1.18.1-cp312-cp312-manylinux_2_27_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:3c085faa2cfa879c5141df483f836f4d691045a078224a670fa570fa01612d89", size = 34025113, upload-time = "2026-08-21T23:24:02.209Z" }, + { url = "https://files.pythonhosted.org/packages/df/64/ff35eb9e54894cf471ff4716abd3c81eb0a0626869217ce3e6ba4ccf17d7/scipy-1.18.1-cp312-cp312-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:f55fa87b6c612ecd6b058f167c53231b1d14e412efe361d3d6e38b3631c73218", size = 35344199, upload-time = "2026-08-21T23:24:07.844Z" }, + { url = "https://files.pythonhosted.org/packages/d3/af/c5538be1792f7034c12c7db6ee67cace58253c7b87b122d68253eaf5de89/scipy-1.18.1-cp312-cp312-musllinux_1_2_aarch64.whl", hash = "sha256:c35d74ce0e193ff740c2f2be2ac913ddc232fe6c1ff40b26cfecb9c670c63314", size = 35639587, upload-time = "2026-08-21T23:24:13.05Z" }, + { url = "https://files.pythonhosted.org/packages/91/4c/075e4f66471bac101141ac739e9e135549be1bae584571bd03a530c056e1/scipy-1.18.1-cp312-cp312-musllinux_1_2_x86_64.whl", hash = "sha256:d2924a03db38dc2e848bca2fe9f077dafb891480b91a00a0963a8cf86dfc31c1", size = 37480330, upload-time = "2026-08-21T23:24:19.608Z" }, + { url = "https://files.pythonhosted.org/packages/39/e7/979fd14e75008623df31ba70d6bb144700f68feadcea042021c06a05bf82/scipy-1.18.1-cp312-cp312-win_amd64.whl", hash = "sha256:5e4d44984abc0020154ea81b247adeddcc3ac5527b975ff798bd1ba0adc513c2", size = 36658278, upload-time = "2026-08-21T23:24:25.463Z" }, + { url = "https://files.pythonhosted.org/packages/c7/0b/e1525354ff9d7d5feb6d1b31af6d14072e5c91e9607b421fa1ec889660b3/scipy-1.18.1-cp312-cp312-win_arm64.whl", hash = "sha256:d65d448389b8436493abcf629cc94ad0cf32aecaf06e1acca1de53cc795f2f12", size = 24400588, upload-time = "2026-08-21T23:24:30.579Z" }, + { url = "https://files.pythonhosted.org/packages/b6/55/4540ee0f9c42a9ad7109d0d1a8cc70de54c3572b01c6693a2b1c70e90ceb/scipy-1.18.1-cp313-cp313-macosx_10_15_x86_64.whl", hash = "sha256:3ab3523da44749156e1f68b464dc56af11ae4cbc5c739a49d05f32b982eca9f3", size = 31089958, upload-time = "2026-08-21T23:24:35.8Z" }, + { url = "https://files.pythonhosted.org/packages/2a/f5/769f36d14922b8071a43e95d24d18b6bdafad10d7f5cf647867e1ac052bc/scipy-1.18.1-cp313-cp313-macosx_12_0_arm64.whl", hash = "sha256:e6fb6a55cc0ba97b59a1f288fb86dc6fce8bdfc0fffcbfd015e3a954bf2a2d93", size = 28715106, upload-time = "2026-08-21T23:24:40.775Z" }, + { url = "https://files.pythonhosted.org/packages/9a/d7/21d890274f75ea37a8209d5519e72da3da90302e3b9fb8397a0918386a62/scipy-1.18.1-cp313-cp313-macosx_14_0_arm64.whl", hash = "sha256:ea324d9dd34c38bfb9bec8ca4d1b407db97dbb74029f566b8e322b1b6fe56fe6", size = 20456846, upload-time = "2026-08-21T23:24:45.066Z" }, + { url = "https://files.pythonhosted.org/packages/ec/01/798430ecea2e78ec7c02663d5f71c007bb6abeca931080debd40d7fa55ea/scipy-1.18.1-cp313-cp313-macosx_14_0_x86_64.whl", hash = "sha256:75b00eb8fb802090aa903f4ea1c7f5a584779f967361e68b7e98e531cc2d7174", size = 23087986, upload-time = "2026-08-21T23:24:49.539Z" }, + { url = "https://files.pythonhosted.org/packages/e6/5f/4634e9d35c68496e4e34cb6946eafab044458e6cedab42b40b6588e475b6/scipy-1.18.1-cp313-cp313-manylinux_2_27_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:d416b16cccfd70fbf62400e84d0bb2f4e6af519a45557f1692c749b37f14b315", size = 33998146, upload-time = "2026-08-21T23:24:54.714Z" }, + { url = "https://files.pythonhosted.org/packages/41/48/6450ed9243315322bbc19ac57b9b70d66a20bf1d38d124c96bc4bf6af9ea/scipy-1.18.1-cp313-cp313-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:fdaf5ea890a6183d0565f51a61799d67081bd5b1cf03c5f4b3fd3732108625c9", size = 35312578, upload-time = "2026-08-21T23:25:00.44Z" }, + { url = "https://files.pythonhosted.org/packages/00/bd/bf5a4be6a3525676499f6dff307991739ff6fdcad1481b1aeb6745339f58/scipy-1.18.1-cp313-cp313-musllinux_1_2_aarch64.whl", hash = "sha256:c825cef2f49e46753726a7181a8e199804a912b29519ada542c6ebc654951899", size = 35612621, upload-time = "2026-08-21T23:25:06.144Z" }, + { url = "https://files.pythonhosted.org/packages/bd/4e/3c45c33e00a77996c4b1cb707929f833ba7b1d522ee29f882512c330676d/scipy-1.18.1-cp313-cp313-musllinux_1_2_x86_64.whl", hash = "sha256:e3b417bf8c2c7c16e8f58ad91db17783ec911ac16e7b50eb6eab6e809b4f5b07", size = 37457323, upload-time = "2026-08-21T23:25:12.483Z" }, + { url = "https://files.pythonhosted.org/packages/93/0e/e0348fbc0dbab65c114cf78957e7dfeb49f8e8b556b4d930cc12ff195e18/scipy-1.18.1-cp313-cp313-win_amd64.whl", hash = "sha256:559ed65f60c1af5a03f3912605a1b5114f522c7c32fb23c3376ae8f03219fe28", size = 36622841, upload-time = "2026-08-21T23:25:18.722Z" }, + { url = "https://files.pythonhosted.org/packages/50/a8/6a77f5f267c555108f0a864b6db714363dab567a8266422a79a385f9232b/scipy-1.18.1-cp313-cp313-win_arm64.whl", hash = "sha256:cd479fc04dd9401e3b4f49e76518768ef99c4f517a98c284eb091fd725719adf", size = 24399315, upload-time = "2026-08-21T23:25:23.458Z" }, + { url = "https://files.pythonhosted.org/packages/06/d5/d8eb4e280ddb56a4ab2c6f02ee49b56b23f6e977cf0802fd6d68dbef14f5/scipy-1.18.1-cp314-cp314-macosx_10_15_x86_64.whl", hash = "sha256:83de5453a7799afc9048b4616bd085cef126e36412f0ea2f6370c36a2a3a51e7", size = 31090936, upload-time = "2026-08-21T23:25:28.686Z" }, + { url = "https://files.pythonhosted.org/packages/2a/49/59ea385dc3a62ff498ddf3cfff7c2b41b0f9f9d3c4122b3f1dcb6d6327fe/scipy-1.18.1-cp314-cp314-macosx_12_0_arm64.whl", hash = "sha256:9554bcc6d715ee87a633a3cc8e7703c6628b100dd29cb8a2efc4c0533c7ff729", size = 28725221, upload-time = "2026-08-21T23:25:33.244Z" }, + { url = "https://files.pythonhosted.org/packages/70/e8/6b0c288c50942d78193696c9f15f9a0874f5178aa0ddf40f83d9924b3e8d/scipy-1.18.1-cp314-cp314-macosx_14_0_arm64.whl", hash = "sha256:011413b7426b75012840e35649e00fe0a2c3bae89fed433876e3a99251572efc", size = 20466839, upload-time = "2026-08-21T23:25:37.516Z" }, + { url = "https://files.pythonhosted.org/packages/4b/e0/54fd3793c729e3b936782f181b59cbb1205bf250ab605a16cb1ba61cdd5e/scipy-1.18.1-cp314-cp314-macosx_14_0_x86_64.whl", hash = "sha256:88f0e784020649f88ea48c9f5ddfa403bf9205820667c0914740b392035afb82", size = 23089121, upload-time = "2026-08-21T23:25:42.019Z" }, + { url = "https://files.pythonhosted.org/packages/0b/56/030af62bea3cf878e0028515dff78c123b01633606a879b63f42d2db99cc/scipy-1.18.1-cp314-cp314-manylinux_2_27_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:2d3ab0e8c69a17dd3559eab8cbb88f258e285c94d572c2719033f90f83290c89", size = 34053851, upload-time = "2026-08-21T23:25:47.998Z" }, + { url = "https://files.pythonhosted.org/packages/6b/89/2a844506d49651e9aa1af6ef95b6bd8031cb1d5a4375edec6155037e04cf/scipy-1.18.1-cp314-cp314-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:ac0333bdf38309aa3dcbe7e3fa7ea29e7a2c37c6ea306a757b700ded8e4596ad", size = 35329183, upload-time = "2026-08-21T23:25:53.522Z" }, + { url = "https://files.pythonhosted.org/packages/eb/56/c7370c3640e92ac9613cbf26cb3f729f9b12ddf1727b55b94b53b24d6f48/scipy-1.18.1-cp314-cp314-musllinux_1_2_aarch64.whl", hash = "sha256:911de823097db8b63f034299d12662db93344e6ffa0b881cbb57748974b70168", size = 35672551, upload-time = "2026-08-21T23:25:59.387Z" }, + { url = "https://files.pythonhosted.org/packages/24/16/ec8536f351421f8bf60a1120930638f83790f4710b8230446aca3d6159d4/scipy-1.18.1-cp314-cp314-musllinux_1_2_x86_64.whl", hash = "sha256:95298364e251be3e60249facbeeca03631d3bb7584f85879516ec55ac717b81f", size = 37469416, upload-time = "2026-08-21T23:26:05.432Z" }, + { url = "https://files.pythonhosted.org/packages/52/94/d73da0d28f16c45bb9b0a5691b91610b0275c5ef0eb5e43c87cf2dc1bf31/scipy-1.18.1-cp314-cp314-win_amd64.whl", hash = "sha256:78a0d7c918e74a232394117160e7e3db503377572a45bcef8826e4ab8a35feba", size = 37362755, upload-time = "2026-08-21T23:26:11.366Z" }, + { url = "https://files.pythonhosted.org/packages/89/25/e996e4dc74e10e227b1e14db5eaf6608bb6dd33884a64851c38f18dd4249/scipy-1.18.1-cp314-cp314-win_arm64.whl", hash = "sha256:cbf38d043c1aa4ab306e1ada6ab6eddacc3322a20b7af1b30bc93254b366fe09", size = 25036090, upload-time = "2026-08-21T23:26:15.887Z" }, + { url = "https://files.pythonhosted.org/packages/fa/c9/c00213f92309d753b48903e6a451b87eb52ff5b7a16e789d1568bbf221c4/scipy-1.18.1-cp314-cp314t-macosx_10_15_x86_64.whl", hash = "sha256:0fcb3c93519f27bb4f0c4b0f7802cdcaca7fcf93267b75edda2e9f4e8a55cbd7", size = 31485550, upload-time = "2026-08-21T23:26:20.776Z" }, + { url = "https://files.pythonhosted.org/packages/74/b2/e3067c487982d4eeab2938928529410370c06fea84a4d3f4925e7d96647d/scipy-1.18.1-cp314-cp314t-macosx_12_0_arm64.whl", hash = "sha256:ddef79fb382df40104a19bb7151b3b23e57c1778fcf857c71ceecd9bd264513f", size = 29174642, upload-time = "2026-08-21T23:26:25.395Z" }, + { url = "https://files.pythonhosted.org/packages/d5/ab/374c9fe2d1ec014e576c781a4b5d8e1ba340e8f6b4638c16f711d2b194f0/scipy-1.18.1-cp314-cp314t-macosx_14_0_arm64.whl", hash = "sha256:0e82073ecc7acc6436fac4b31674109c7e1d3e596789767eda01258a8c9e8123", size = 20916357, upload-time = "2026-08-21T23:26:30.112Z" }, + { url = "https://files.pythonhosted.org/packages/90/38/223915c88a17317cafbf8ca2a42b11c265a9fb1e804aa665544132b5fe8a/scipy-1.18.1-cp314-cp314t-macosx_14_0_x86_64.whl", hash = "sha256:8bcf3c1ba5d6456e2effd30fcbd3459b044d683fcdac79a2e6830f0bdf7de487", size = 23482611, upload-time = "2026-08-21T23:26:34.846Z" }, + { url = "https://files.pythonhosted.org/packages/c4/d1/db0948da8ca57a80b36520ef0a768b967d99f3af65f4b6f1bf6362ad4dd4/scipy-1.18.1-cp314-cp314t-manylinux_2_27_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:cfbf154f2ba187f2ed6cce2639efff7d105f1140573642c0161615b6d91d6a87", size = 34143202, upload-time = "2026-08-21T23:26:40.4Z" }, + { url = "https://files.pythonhosted.org/packages/87/53/39d046cc7574ed6acacb6bd5723e220107ece80bff12faaf3efc4ddeede4/scipy-1.18.1-cp314-cp314t-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:a1d33a7836f7ddc1993427966a0823468ec41bcbdb1a9f9942d1d7e57f803ba3", size = 35380876, upload-time = "2026-08-21T23:26:46.1Z" }, + { url = "https://files.pythonhosted.org/packages/f9/da/32e0e799d875a85ca57d9bde6c78148afcc0e38276df683d95854eadc8c3/scipy-1.18.1-cp314-cp314t-musllinux_1_2_aarch64.whl", hash = "sha256:7f4b8bc363b6d65ee2152bec57568e3c52639bb34c46057b09857a307ed5e21d", size = 35770885, upload-time = "2026-08-21T23:26:51.533Z" }, + { url = "https://files.pythonhosted.org/packages/88/2e/f97a666d362fee68b18f41c9c30ed502ca5c98b549749bfcb52a8b74d1eb/scipy-1.18.1-cp314-cp314t-musllinux_1_2_x86_64.whl", hash = "sha256:11c423f1049c5755ad4409af52a9ada1cff96fe9b50795d4af3619f292901239", size = 37525424, upload-time = "2026-08-21T23:26:56.751Z" }, + { url = "https://files.pythonhosted.org/packages/ca/d5/a9e765a84654ebba8479a1fd1b059ced1af72b168a3b2a3a46540ea38d20/scipy-1.18.1-cp314-cp314t-win_amd64.whl", hash = "sha256:c24acac1e18912761c4700239bbc1fd32f615af690f1584d49b35859be51324d", size = 37416961, upload-time = "2026-08-21T23:27:01.546Z" }, + { url = "https://files.pythonhosted.org/packages/ee/16/e79e0d1c63ef698879d85439d37e9fb434e3b804e506a6991038d086ebd9/scipy-1.18.1-cp314-cp314t-win_arm64.whl", hash = "sha256:9f2897bf7737392ad0d5213ea7b6add72a4edf5679b3153106aeb88b6507b3b9", size = 25331848, upload-time = "2026-08-21T23:27:05.884Z" }, + { url = "https://files.pythonhosted.org/packages/be/4f/1bd37c883b67163e2ca1f60977a399500e6879c15defecac62831c8d078d/scipy-1.18.1-cp315-cp315-macosx_10_15_x86_64.whl", hash = "sha256:eb0dfcf4e28a99c12c999744a2ff67c9b06200e20401c7c88186e33552a46331", size = 31091484, upload-time = "2026-08-21T23:27:11.051Z" }, + { url = "https://files.pythonhosted.org/packages/8c/c5/ba929d7feb9b2332f96827c12e0e924b61973b59b4dea383b603372c65ce/scipy-1.18.1-cp315-cp315-macosx_12_0_arm64.whl", hash = "sha256:30f464bee641fa8e282577c7dce027308403213c6ca8270bba73285c91024bc5", size = 28725057, upload-time = "2026-08-21T23:27:15.9Z" }, + { url = "https://files.pythonhosted.org/packages/a4/19/68f1c50f609d955d230e66d25d02bd3e1e167ec540232135354fb9a4b9e3/scipy-1.18.1-cp315-cp315-macosx_14_0_arm64.whl", hash = "sha256:1bca3b943fc2567ea49cd02c99abde49da4d5178ec46f624bd8255cda8755beb", size = 20466734, upload-time = "2026-08-21T23:27:20.044Z" }, + { url = "https://files.pythonhosted.org/packages/ef/6d/319fa29b73d1802fa80b32a6eaf3f5be456ef81526da2716a9493bcb5501/scipy-1.18.1-cp315-cp315-macosx_14_0_x86_64.whl", hash = "sha256:c9d18a33309122074ea483dd92dd444189166b8b2ec429fe9ed5ac73c7a0aa23", size = 23089664, upload-time = "2026-08-21T23:27:24.345Z" }, + { url = "https://files.pythonhosted.org/packages/b7/db/30992f9b51a63de671daf3888ffd18378b6cb9ec9f2c972264238ffa7fd6/scipy-1.18.1-cp315-cp315-manylinux_2_27_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:82f201b4c878551d48558337aab270d3c6cca5507b8737c8d8a608d234cccde0", size = 34054035, upload-time = "2026-08-21T23:27:29.409Z" }, + { url = "https://files.pythonhosted.org/packages/91/d4/bf3e735dc0b9d5a8ff45079d2540e17d3aff7a2f0048dd8f552ffd031d2b/scipy-1.18.1-cp315-cp315-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:0ac49ea97594532dd44b7136094d35f5440fa06e6d9c6384a74c01764df388c5", size = 35333883, upload-time = "2026-08-21T23:27:34.293Z" }, + { url = "https://files.pythonhosted.org/packages/19/93/12d78ce9f871fe945fca588d32644e6e63f553c2a35c564d73f3b22a3313/scipy-1.18.1-cp315-cp315-musllinux_1_2_aarch64.whl", hash = "sha256:ceb30a00ce7c92d459819443d29ca486d882b83fb6738bdcbb2a1cce94ac5daa", size = 35673124, upload-time = "2026-08-21T23:27:39.059Z" }, + { url = "https://files.pythonhosted.org/packages/70/cd/886219313a1012a48e6ae0ec4f302c837151beb92e1ff0d709ef8fdfc488/scipy-1.18.1-cp315-cp315-musllinux_1_2_x86_64.whl", hash = "sha256:f29633129f9fa7e88a3f0fca835de2d030bfc9643f7799e1a0c46cee24d38fc7", size = 37470753, upload-time = "2026-08-21T23:27:44.435Z" }, + { url = "https://files.pythonhosted.org/packages/17/6c/a776888ce618bee54fbde26172f0f46ac1da70d27b63861797fe78e1904b/scipy-1.18.1-cp315-cp315-win_amd64.whl", hash = "sha256:92c14f5bdbfb6216315ce33e78080474082de8b3830122ba97809bfbe65f75c0", size = 37361483, upload-time = "2026-08-21T23:27:49.334Z" }, + { url = "https://files.pythonhosted.org/packages/ab/09/97b651691322ebee97999b017ffc18a15a0b815103844c97e8da9d469731/scipy-1.18.1-cp315-cp315-win_arm64.whl", hash = "sha256:e402cf31eb68f453dbb2d36fc6d722b33f24a55d68b2ae1d92fa6305ca71c298", size = 25035883, upload-time = "2026-08-21T23:27:53.596Z" }, + { url = "https://files.pythonhosted.org/packages/ed/0f/9ec20467bbabd0d44e2a77d0fd3d124f884b4d67df92af82c91d2d6a486f/scipy-1.18.1-cp315-cp315t-macosx_10_15_x86_64.whl", hash = "sha256:2a0b02f9fc46f8520330c23d45e6560db7e3a0d927232139427637f98943e11d", size = 31474926, upload-time = "2026-08-21T23:27:57.993Z" }, + { url = "https://files.pythonhosted.org/packages/8a/58/dcb79161e56efbedc50079fcd2f5fe427a0ebb53022eb476aa73c015ad8f/scipy-1.18.1-cp315-cp315t-macosx_12_0_arm64.whl", hash = "sha256:1d73131e358976663dd969e1fb4ed1404b815cd977eaaedc3b3a133ba2d81c35", size = 29164940, upload-time = "2026-08-21T23:28:03.062Z" }, + { url = "https://files.pythonhosted.org/packages/71/d3/1eeea80c817fcb8ef7bd4a05a58824977a0e57a375cfc3d7ea7c911c01ad/scipy-1.18.1-cp315-cp315t-macosx_14_0_arm64.whl", hash = "sha256:bff0b729edd992766136b34e39cc76bc2fad905aa58897ee72a9cd000a6d8443", size = 20906742, upload-time = "2026-08-21T23:28:07.642Z" }, + { url = "https://files.pythonhosted.org/packages/54/46/e59350428b6099301a20128108c995e2eb175a43f383af9a346e38824f9b/scipy-1.18.1-cp315-cp315t-macosx_14_0_x86_64.whl", hash = "sha256:10ac20c69d880f77f375db44c22e3e6a644f9fefa291d4cd2fb9790a89fc99fd", size = 23472183, upload-time = "2026-08-21T23:28:12.109Z" }, + { url = "https://files.pythonhosted.org/packages/89/31/cc91623fa98f0621766a0f0aaaadb2c66de74a7ea7e3837164f6e4354260/scipy-1.18.1-cp315-cp315t-manylinux_2_27_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:33a834464fdabc0f26a45508df31b3cc5d028e04dbf6c5ed398541418e0a12fe", size = 34130796, upload-time = "2026-08-21T23:28:17.906Z" }, + { url = "https://files.pythonhosted.org/packages/fc/3e/8572ef536957ddb8aa81bb4090d9e25f257e3b4e05d97deb54319deb8a3a/scipy-1.18.1-cp315-cp315t-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:49023963c193dacee096301452f223ee24d86ec5807f8df93c0f7221d119e305", size = 35374253, upload-time = "2026-08-21T23:28:23.732Z" }, + { url = "https://files.pythonhosted.org/packages/b5/c6/59fdeffb4f1435299f93d9dc8140b43ad2916e6cfc944be6c3041fcec86d/scipy-1.18.1-cp315-cp315t-musllinux_1_2_aarch64.whl", hash = "sha256:d84a09d0dad90ba6525d8ac1c2334b33e64bf3ccfe9e841f02feb867a22681e4", size = 35758543, upload-time = "2026-08-21T23:28:29.431Z" }, + { url = "https://files.pythonhosted.org/packages/cf/d9/135be205d9de8783193aff9cc3bf483a03a38e4b29432c954e8cb66ac14e/scipy-1.18.1-cp315-cp315t-musllinux_1_2_x86_64.whl", hash = "sha256:179ce34a8d0fe273d8883ba59e17e052247d08973dfcb743ca52bb1cce2d60b0", size = 37521946, upload-time = "2026-08-21T23:28:35.245Z" }, + { url = "https://files.pythonhosted.org/packages/5c/a2/5b7d5270621ab7cfa3f7766067bf95dc360b5efb6394694e8143b4156e2b/scipy-1.18.1-cp315-cp315t-win_amd64.whl", hash = "sha256:5632e3ae3d09197c446310cd5187de63e28448ce22f0f67b2b93d97503c0c230", size = 37408295, upload-time = "2026-08-21T23:28:40.724Z" }, + { url = "https://files.pythonhosted.org/packages/63/ad/741c19fcb66755ff953daf9243af8480e4bf3d7fbe57583c178c7d2b6b51/scipy-1.18.1-cp315-cp315t-win_arm64.whl", hash = "sha256:eda632a7981f69730d6281f451db9c1c370993a2c0d7ddb43e2a809a2862b83a", size = 25319710, upload-time = "2026-08-21T23:28:45.713Z" }, +] + +[[package]] +name = "six" +version = "1.17.0" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/94/e7/b2c673351809dca68a0e064b6af791aa332cf192da575fd474ed7d6f16a2/six-1.17.0.tar.gz", hash = "sha256:ff70335d468e7eb6ec65b95b99d3a2836546063f63acc5171de367e834932a81", size = 34031, upload-time = "2024-12-04T17:35:28.174Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/b7/ce/149a00dd41f10bc29e5921b496af8b574d8413afcd5e30dfa0ed46c2cc5e/six-1.17.0-py2.py3-none-any.whl", hash = "sha256:4721f391ed90541fddacab5acf947aa0d3dc7d27b2e1e8eda2be8970586c3274", size = 11050, upload-time = "2024-12-04T17:35:26.475Z" }, +] + +[[package]] +name = "tailwise-bench" +version = "0.1.0" +source = { editable = "." } +dependencies = [ + { name = "httpx" }, + { name = "huggingface-hub" }, + { name = "matplotlib" }, + { name = "numpy", version = "2.4.6", source = { registry = "https://pypi.org/simple" }, marker = "python_full_version < '3.12'" }, + { name = "numpy", version = "2.5.4", source = { registry = "https://pypi.org/simple" }, marker = "python_full_version >= '3.12'" }, + { name = "pyarrow" }, + { name = "pyyaml" }, + { name = "scikit-learn" }, + { name = "tiktoken" }, +] + +[package.dev-dependencies] +dev = [ + { name = "mypy" }, + { name = "pytest" }, + { name = "types-pyyaml" }, +] + +[package.metadata] +requires-dist = [ + { name = "httpx", specifier = ">=0.27" }, + { name = "huggingface-hub", specifier = ">=0.24" }, + { name = "matplotlib", specifier = ">=3.8" }, + { name = "numpy", specifier = ">=1.26" }, + { name = "pyarrow", specifier = ">=15" }, + { name = "pyyaml", specifier = ">=6" }, + { name = "scikit-learn", specifier = ">=1.4" }, + { name = "tiktoken", specifier = ">=0.7" }, +] + +[package.metadata.requires-dev] +dev = [ + { name = "mypy", specifier = ">=1.10" }, + { name = "pytest", specifier = ">=8" }, + { name = "types-pyyaml" }, +] + +[[package]] +name = "threadpoolctl" +version = "3.7.0" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/00/dc/6c58154c1c65f758ea979e7139cb76993a9cfc662d14e9be3c4a667cfb77/threadpoolctl-3.7.0.tar.gz", hash = "sha256:61348cfb77d53b9242e0017029244b559b810c142ced65b4e21eeca1843959a7", size = 31961, upload-time = "2026-09-15T15:46:20.263Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/43/3f/f88a53f60a472b46f4023f56d204dd7de33d34c5d2acbfa0d70a674e639e/threadpoolctl-3.7.0-py3-none-any.whl", hash = "sha256:cd8b60b5641b45c67bbf73c64c843235fc2d8a480c87389f52f5dbee893b86be", size = 26362, upload-time = "2026-09-15T15:46:19.168Z" }, +] + +[[package]] +name = "tiktoken" +version = "0.14.0" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "regex" }, + { name = "requests" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/66/62/167a842aa0429d45f5e797354fd4343a96f6043d67d0513c675c7b8d36e6/tiktoken-0.14.0.tar.gz", hash = "sha256:231dec90efcdccf1b565a1416107736f1e09b1a08fe736ef9d6363e626d03874", size = 38898, upload-time = "2026-08-17T19:49:49.514Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/8f/c5/9d848b7f408241171e1f843deb8bfa626086452bc9c78beee500829583e3/tiktoken-0.14.0-cp311-cp311-macosx_10_12_x86_64.whl", hash = "sha256:c2edf09b381fafbc014ae8e018ed25087abb9a3dafa8465a0ea63c6558c47a79", size = 1094971, upload-time = "2026-08-17T19:48:40.347Z" }, + { url = "https://files.pythonhosted.org/packages/2d/a9/d94302340304328961d6f0c35ca4e60617fbb57a5cf667e2ed1692cb9e57/tiktoken-0.14.0-cp311-cp311-macosx_11_0_arm64.whl", hash = "sha256:cd8ca1305c1c902fe42c486165f2e4808d9997625c98ffb05b9e0366d99d3948", size = 1042916, upload-time = "2026-08-17T19:48:41.541Z" }, + { url = "https://files.pythonhosted.org/packages/c8/b6/31da98ee871383509cae2ba96a9ddef1965e3c4f8cb6dc7bcda3379398db/tiktoken-0.14.0-cp311-cp311-manylinux_2_28_aarch64.whl", hash = "sha256:1f83081065ee5833d35b49e9180f3d8d15622a603dd1c435da0da6cc12b3662f", size = 1188650, upload-time = "2026-08-17T19:48:42.729Z" }, + { url = "https://files.pythonhosted.org/packages/24/65/8c5dddd7cb67f6571d154a58d7c6e2f07da54bf84c49b6a1839965b7c35e/tiktoken-0.14.0-cp311-cp311-manylinux_2_28_x86_64.whl", hash = "sha256:f5e7665f6624e052e5e7f6a36919ab69279decdc976d7b16b4fa15e1897d0513", size = 1206378, upload-time = "2026-08-17T19:48:44.013Z" }, + { url = "https://files.pythonhosted.org/packages/d1/04/522ec59d30dd9a2f3ab837011cd4fc5d1178dc4a2fa07c9fa4b90af6ba9d/tiktoken-0.14.0-cp311-cp311-musllinux_1_2_aarch64.whl", hash = "sha256:144a3fc369f92b7d548995217c5d6e84038d3572157a0f6f34080d65291d0f78", size = 1253694, upload-time = "2026-08-17T19:48:45.597Z" }, + { url = "https://files.pythonhosted.org/packages/69/84/9019e272bad188a1c61ecf44f25a9ba2368744644e3ac1f3d6516f3c9e80/tiktoken-0.14.0-cp311-cp311-musllinux_1_2_x86_64.whl", hash = "sha256:151d37a150c8f3dfc5f4345597b10e101876bd1bd13494e0185af6b508758d2e", size = 1317873, upload-time = "2026-08-17T19:48:46.792Z" }, + { url = "https://files.pythonhosted.org/packages/24/7f/fff1217240343c0c11b5938b98aeae0e3a266cacfac25f86f91cdcd748f0/tiktoken-0.14.0-cp311-cp311-win_amd64.whl", hash = "sha256:c77d4a3e1deb2707819df92046b89aad1ac81d27e07616b797cbff3f62c037da", size = 944395, upload-time = "2026-08-17T19:48:48.028Z" }, + { url = "https://files.pythonhosted.org/packages/8c/da/e273746b9d24a63c776bc60fba914351573ad9c575b52601eb5e60632564/tiktoken-0.14.0-cp312-cp312-macosx_10_13_x86_64.whl", hash = "sha256:8e947aefe98ef74cce94923f90e48c98fe34eb1ec0a6bfdfadfc5a96359bfc36", size = 1094408, upload-time = "2026-08-17T19:48:49.269Z" }, + { url = "https://files.pythonhosted.org/packages/69/9f/fe6b1aca23331aa5271df5a4bd07bf68a7059254d47faee1b8272592a777/tiktoken-0.14.0-cp312-cp312-macosx_11_0_arm64.whl", hash = "sha256:d6cebe67765569df3dafac8474e4eccf5c19d24140492567a5e58a11445732a4", size = 1038499, upload-time = "2026-08-17T19:48:50.666Z" }, + { url = "https://files.pythonhosted.org/packages/0b/35/e9f47647c9e163bd1de30fe1a491669b7248cfc67b7404c35c009a701e1a/tiktoken-0.14.0-cp312-cp312-manylinux_2_28_aarch64.whl", hash = "sha256:7db45b98e94adf4173a5cd7422b150999a7ee11ff847783a14f6e1b80cc38cb6", size = 1186355, upload-time = "2026-08-17T19:48:51.93Z" }, + { url = "https://files.pythonhosted.org/packages/51/11/9976ad86980a00cdef05e730a0127a2578a1bc6d11644d8d47246de2eb26/tiktoken-0.14.0-cp312-cp312-manylinux_2_28_x86_64.whl", hash = "sha256:7896eea257fe497a2b7134474d909156c6744ce8da35bce88011a960e008aa0d", size = 1204197, upload-time = "2026-08-17T19:48:53.18Z" }, + { url = "https://files.pythonhosted.org/packages/d4/9c/7035b0bcfaa68d1ee4803fc5be5214ad865669b05bd20e7105ae8a18afc6/tiktoken-0.14.0-cp312-cp312-musllinux_1_2_aarch64.whl", hash = "sha256:b950248272f1b303dc32986396e2dccfa10cf6d1e83ec8f0bba1776660305482", size = 1250635, upload-time = "2026-08-17T19:48:54.392Z" }, + { url = "https://files.pythonhosted.org/packages/bc/1d/69cabf18bed7f4366da076735816abce0d4db3fae491ae338a6612128777/tiktoken-0.14.0-cp312-cp312-musllinux_1_2_x86_64.whl", hash = "sha256:3de75343041a1c57333b1e707ac8a9769738241d7d6a55d39e12cf84548337c6", size = 1316085, upload-time = "2026-08-17T19:48:55.525Z" }, + { url = "https://files.pythonhosted.org/packages/bd/bd/a2e884fb1402cba5be08836590320012b2d8ada0e2eef9911a64df4bcd2d/tiktoken-0.14.0-cp312-cp312-win_amd64.whl", hash = "sha256:087538c080e5ff421abd3a0785ed63c5111d06af98e6cd0d374dbe5969147ca3", size = 941208, upload-time = "2026-08-17T19:48:56.938Z" }, + { url = "https://files.pythonhosted.org/packages/50/53/ee1453623bf65f019328721ccb6587846d2c5b7b82f34e73ca09101f072e/tiktoken-0.14.0-cp313-cp313-macosx_10_13_x86_64.whl", hash = "sha256:e9c5fe393aab56469f04e432ff851216d3def3436cf5f07e442a240164bf500f", size = 1094198, upload-time = "2026-08-17T19:48:57.955Z" }, + { url = "https://files.pythonhosted.org/packages/ad/5f/6448cfe278c3664ba9ec5b5ac08344341f7dc3d42888476e215a14eda2be/tiktoken-0.14.0-cp313-cp313-macosx_11_0_arm64.whl", hash = "sha256:cbe2cc3bba939bcdaf103e03df9d5039d33887080b315624be28ec69059e5f94", size = 1038820, upload-time = "2026-08-17T19:48:59.015Z" }, + { url = "https://files.pythonhosted.org/packages/69/3b/d67eac1bcce9dee3abe23aff5e3ded3116bbebaf67b80a0811c06d3806fc/tiktoken-0.14.0-cp313-cp313-manylinux_2_28_aarch64.whl", hash = "sha256:2157f52e4b4d7ac5ecc7457b3716834706e7ef9a46f5144029bfeb7cf71f4e06", size = 1186175, upload-time = "2026-08-17T19:49:00.068Z" }, + { url = "https://files.pythonhosted.org/packages/37/62/cae690d9783146b0f81f564ada0f8f611de68178c0c9c7e1e969f0516b48/tiktoken-0.14.0-cp313-cp313-manylinux_2_28_x86_64.whl", hash = "sha256:26e60f6a956ee171ab728b37b8439905d7ea1db435c30f9822f291e9861c861d", size = 1203884, upload-time = "2026-08-17T19:49:01.163Z" }, + { url = "https://files.pythonhosted.org/packages/b9/1e/633e30237b94e383cf814145499079f3bb9cdd4aeafc1bc42e01b0f810a6/tiktoken-0.14.0-cp313-cp313-musllinux_1_2_aarch64.whl", hash = "sha256:380873f330b741c4435574f37edb20813d04603ace2d53e0a63560e1fec83010", size = 1250980, upload-time = "2026-08-17T19:49:02.274Z" }, + { url = "https://files.pythonhosted.org/packages/cb/56/4c12f07b812f84206f38d723eb1ebfdd34bad9309b5dbc0bee6bbcff4cbf/tiktoken-0.14.0-cp313-cp313-musllinux_1_2_x86_64.whl", hash = "sha256:3fd7c14b1cb45b486c39fc9b3443bb341f3e2fc7e6f31247f3435a5836651632", size = 1315434, upload-time = "2026-08-17T19:49:03.434Z" }, + { url = "https://files.pythonhosted.org/packages/c9/e0/c65603f0c44811def666d3fbf611bf2af3b5e1ef613e06c19411419830b3/tiktoken-0.14.0-cp313-cp313-win_amd64.whl", hash = "sha256:90a762670c7f968184723769a06ed51f5cf5ce5dcd1e30164f25c72d85c2d1f1", size = 940883, upload-time = "2026-08-17T19:49:04.583Z" }, + { url = "https://files.pythonhosted.org/packages/59/b0/1cf129f4af8fc513931f931023def596b7c4bfc77026513cd9d851da9e88/tiktoken-0.14.0-cp314-cp314-macosx_10_15_x86_64.whl", hash = "sha256:e067f4cbcc5d036e8aff7fe7a6b530a8f4de2e4616ad9005a24a1879e24e6450", size = 1096273, upload-time = "2026-08-17T19:49:05.807Z" }, + { url = "https://files.pythonhosted.org/packages/62/85/2ae74575e321148484147e10b53c3b1717c59ebaa9edb4fe18b1f5c055f8/tiktoken-0.14.0-cp314-cp314-macosx_11_0_arm64.whl", hash = "sha256:f2af4a336ea56d6c14f27741a0e1d8294a35dd0b038bcf990d232ebb54eb994b", size = 1040269, upload-time = "2026-08-17T19:49:06.943Z" }, + { url = "https://files.pythonhosted.org/packages/89/29/92a1120a12e4bcf2d5464350d1a91b68a433d63ce656bb7f806c27aec09c/tiktoken-0.14.0-cp314-cp314-manylinux_2_28_aarch64.whl", hash = "sha256:f702e0aeeb6506e57687e881c59e844ebe8f0a6a097ddafe20e3ab25f387be4e", size = 1186101, upload-time = "2026-08-17T19:49:08.102Z" }, + { url = "https://files.pythonhosted.org/packages/5b/7d/144af98dc5ad68108451a82e2f5a17f80e2663f5115058b8dfd215c1ad02/tiktoken-0.14.0-cp314-cp314-manylinux_2_28_x86_64.whl", hash = "sha256:e3442bbb2f0c588cec876061e37ae67b455b9df9978b003c8fe30e45f2ef5b42", size = 1204457, upload-time = "2026-08-17T19:49:09.28Z" }, + { url = "https://files.pythonhosted.org/packages/e6/1f/be7cb06ab2108f612f3e92e7b76cf391e192db0db37a984616f0cc32aafc/tiktoken-0.14.0-cp314-cp314-musllinux_1_2_aarch64.whl", hash = "sha256:979c1524f753b662b0f3cd261b135afe6659cce33caaa7a5ea00dd1756b3055c", size = 1251716, upload-time = "2026-08-17T19:49:10.509Z" }, + { url = "https://files.pythonhosted.org/packages/ab/6b/81f158d0f90adb826cd704069c2129a046cb784a2a09861009519fc41cf4/tiktoken-0.14.0-cp314-cp314-musllinux_1_2_x86_64.whl", hash = "sha256:2cc19ac87b41c9493c9778ff5847f0c8bbcf5bd0ec6b87ce06c1c802adc8a771", size = 1315432, upload-time = "2026-08-17T19:49:11.844Z" }, + { url = "https://files.pythonhosted.org/packages/fc/ec/f5fa35ec13f07279fdcaf3cc9c04bbb154ea591d23978651f2b672593e8a/tiktoken-0.14.0-cp314-cp314-win_amd64.whl", hash = "sha256:eceeff0c62419bc78d4b6e70a4762a4d25df3ae8f2d5946e3853ce93e7a57098", size = 988046, upload-time = "2026-08-17T19:49:13.282Z" }, + { url = "https://files.pythonhosted.org/packages/68/c9/7756717408d3d0dfea3f046c9466144b28afde39ff69d5808f2475dcd7f5/tiktoken-0.14.0-cp314-cp314t-macosx_10_15_x86_64.whl", hash = "sha256:6eb94895c45f26bb8f5546e5fd8a069efcf6e3f108ea9d5cbe3bf6f7f3983438", size = 1096261, upload-time = "2026-08-17T19:49:14.351Z" }, + { url = "https://files.pythonhosted.org/packages/79/29/46ad8061f57bd9f8b2ea0aa82bf574e0f2aa040b0857a1582adba9957899/tiktoken-0.14.0-cp314-cp314t-macosx_11_0_arm64.whl", hash = "sha256:86951a971c53979ec857bd8c4a32dc227ab0fd33f6c12a3bd62d3fbf5f0bfcaa", size = 1040183, upload-time = "2026-08-17T19:49:15.707Z" }, + { url = "https://files.pythonhosted.org/packages/5a/7c/3184d17b868456f17b60b1a75f5ec0405618a43aa753336df341d8f11781/tiktoken-0.14.0-cp314-cp314t-manylinux_2_28_aarch64.whl", hash = "sha256:e2eca764c53490f8930dbce329e0769f11108d87d908282a80c5c130e26e7037", size = 1186719, upload-time = "2026-08-17T19:49:16.84Z" }, + { url = "https://files.pythonhosted.org/packages/0b/e8/46de4400d5bf859f640feee85bd7e32235f68ddf25db53c63be78e581e3a/tiktoken-0.14.0-cp314-cp314t-manylinux_2_28_x86_64.whl", hash = "sha256:26cc4b4840fa0e9f4b72ed489883e12f57e00d1021ca794720e3c29a12f0edef", size = 1204660, upload-time = "2026-08-17T19:49:17.987Z" }, + { url = "https://files.pythonhosted.org/packages/29/ce/af8964c38bc8226dd8950305b7a255fa33345d5572f78af7275a313d28e0/tiktoken-0.14.0-cp314-cp314t-musllinux_1_2_aarch64.whl", hash = "sha256:2fc834fbe3f6a0736905c36ab709537e6840dbd63b982dc9e0216ae7d305ba1a", size = 1250932, upload-time = "2026-08-17T19:49:19.28Z" }, + { url = "https://files.pythonhosted.org/packages/1d/4b/323631116fc986d9cc5bbeb2b8223c7c85e61a8bb94ea5ab4951023b149b/tiktoken-0.14.0-cp314-cp314t-musllinux_1_2_x86_64.whl", hash = "sha256:ca4db6ff5c5bf600f9b7761a0070ed44dfe5797a76bd432fb978bc480ef40c58", size = 1315190, upload-time = "2026-08-17T19:49:20.467Z" }, + { url = "https://files.pythonhosted.org/packages/18/8b/ba48a73729c9270989b36f37ab2ed5525e52690d715097c9fa791aaa5d05/tiktoken-0.14.0-cp314-cp314t-win_amd64.whl", hash = "sha256:7aab286a020660a039097912a088236b985d18a3090d73f136c4413d29d37ca0", size = 987717, upload-time = "2026-08-17T19:49:21.704Z" }, + { url = "https://files.pythonhosted.org/packages/1d/10/b73b7e319179e0f60b32475f783b044f9cece872c53b6662664e9084b0d0/tiktoken-0.14.0-cp315-cp315-macosx_10_15_x86_64.whl", hash = "sha256:14b47e3674f2624803a8acc8fb367b7e24fc53055f9df3296482fe9a3a34a232", size = 1096280, upload-time = "2026-08-17T19:49:22.779Z" }, + { url = "https://files.pythonhosted.org/packages/c2/6b/09999a9bf1d559670d1680e8f8e419ac0e2c5f6aac82e9bfdf70f260b30a/tiktoken-0.14.0-cp315-cp315-macosx_11_0_arm64.whl", hash = "sha256:19d643d701fdaa70e5b9c7f8f96abcaffe77ca5e482a3a1a7dde46feb4284695", size = 1040433, upload-time = "2026-08-17T19:49:23.998Z" }, + { url = "https://files.pythonhosted.org/packages/cd/7b/8537be0836f3df99b2a636b44399bfa43cd757f2b8b4097dacb794cf24a7/tiktoken-0.14.0-cp315-cp315-manylinux_2_28_aarch64.whl", hash = "sha256:e4ddf863b59347deaa92302dcd90e5eb003cdc9be06ec2b692c38d1bdd9efd49", size = 1186989, upload-time = "2026-08-17T19:49:25.021Z" }, + { url = "https://files.pythonhosted.org/packages/7c/9d/f9c56d7a943a4468abf9ef37661bb9b8e0cd3aa8aa87368c7146cc3f3222/tiktoken-0.14.0-cp315-cp315-manylinux_2_28_x86_64.whl", hash = "sha256:60c47ca69ddda0dea8256fffd12e1b86f4b59734a20e4a70c61f63cc5f021df4", size = 1204615, upload-time = "2026-08-17T19:49:26.37Z" }, + { url = "https://files.pythonhosted.org/packages/4b/d2/98a38579db25c4a8a84e31dd95d9072ec5f21f7e70de591da0412e29b25b/tiktoken-0.14.0-cp315-cp315-musllinux_1_2_aarch64.whl", hash = "sha256:728303a072163130c5b477b1f20d6211895569c1d5302c24ffc93a3009160871", size = 1251828, upload-time = "2026-08-17T19:49:27.423Z" }, + { url = "https://files.pythonhosted.org/packages/0c/83/467be424746c039c5493c0f4102feab16b9b48eb6f5c089b2a2438e3cde2/tiktoken-0.14.0-cp315-cp315-musllinux_1_2_x86_64.whl", hash = "sha256:3c5349c9f916283bba32bec8af69b763e4faa304dc004d0eaaea66a3cf004c1f", size = 1316260, upload-time = "2026-08-17T19:49:29.101Z" }, + { url = "https://files.pythonhosted.org/packages/02/ee/ddf46ca78e371f5890e96b6e7d089a85b3536432be219851eb0481786ca8/tiktoken-0.14.0-cp315-cp315-win_amd64.whl", hash = "sha256:1b6e4adcfd285c44502aed51df98aaaca4f0fea028165dbf8a9e857b9f98d8ea", size = 988230, upload-time = "2026-08-17T19:49:30.246Z" }, + { url = "https://files.pythonhosted.org/packages/2a/00/5162e90c851a28da18ed382d34898b79a8022548e5619a64e14c03ce7c3d/tiktoken-0.14.0-cp315-cp315t-macosx_10_15_x86_64.whl", hash = "sha256:11d8211b290855d2721334ff17dd9b3a17bfb26872be01f25d73612ef7ece890", size = 1096186, upload-time = "2026-08-17T19:49:31.656Z" }, + { url = "https://files.pythonhosted.org/packages/65/97/a5a7bfccf25b1bb65e82bae8edff11ac3c9c041c374b7b4a823d60c38133/tiktoken-0.14.0-cp315-cp315t-macosx_11_0_arm64.whl", hash = "sha256:d0781223705199b289faa59601bb9c2441712d4c600dd13c43d8fd6a33d22cd5", size = 1039947, upload-time = "2026-08-17T19:49:32.848Z" }, + { url = "https://files.pythonhosted.org/packages/fb/ba/ef427fc638f1439181c5e12dd26b70e881861f89c007aa7e5b36300f8342/tiktoken-0.14.0-cp315-cp315t-manylinux_2_28_aarch64.whl", hash = "sha256:2ea70afba6b9eddbf22c165142e5f0a2ad7aa36a452873c48b57bb2aeb8492ae", size = 1186997, upload-time = "2026-08-17T19:49:34.121Z" }, + { url = "https://files.pythonhosted.org/packages/3e/88/2f3f85a968cdc514152129af0a060ebcccb067005a2f29b0d5ef3c838514/tiktoken-0.14.0-cp315-cp315t-manylinux_2_28_x86_64.whl", hash = "sha256:78571efc311c30b73f31eb949a921d6dac39a5d9dc42d1cfa8f8db157b3447b1", size = 1205211, upload-time = "2026-08-17T19:49:35.284Z" }, + { url = "https://files.pythonhosted.org/packages/4e/f6/80760e98a08e6649d2d68afb6035af713121dfb615acce8c4f73810ec438/tiktoken-0.14.0-cp315-cp315t-musllinux_1_2_aarch64.whl", hash = "sha256:86f66c85e796f5d05d5c4a60ec1d40cbfebc47a32464053528c797163fa9ab89", size = 1251479, upload-time = "2026-08-17T19:49:36.419Z" }, + { url = "https://files.pythonhosted.org/packages/c5/84/50966fb6918a0fb9b32721277e5342bf729a2d74350074d662fbedf9772e/tiktoken-0.14.0-cp315-cp315t-musllinux_1_2_x86_64.whl", hash = "sha256:149d97453c4c98c04b081d64a85e635921269b532710d6faf81e9e82b790e7d3", size = 1316673, upload-time = "2026-08-17T19:49:37.756Z" }, + { url = "https://files.pythonhosted.org/packages/35/5e/9b01afd037bfa22a0033963fa091e0f75b6fb15cd85bffb42ff86e697323/tiktoken-0.14.0-cp315-cp315t-win_amd64.whl", hash = "sha256:561e7580f84a79859af1ef6f676968e9030fcc3fe195700b15235bca64f009c9", size = 987929, upload-time = "2026-08-17T19:49:38.947Z" }, +] + +[[package]] +name = "tqdm" +version = "4.70.1" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "colorama", marker = "sys_platform == 'win32'" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/0d/ea/b2a5bd54b28a324dae8211928b2d730b6547500342c7e6c6dea08bd0a485/tqdm-4.70.1.tar.gz", hash = "sha256:cefd0eca11b2a37a3aee776544d4f4ae913f02688135b5556b8788dfa474afc4", size = 171846, upload-time = "2026-09-11T07:25:16.601Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/a7/03/921a3d3c75785aca9ebfbfcabfbc3a1be12e2ab5265deb026d55a5a3f83e/tqdm-4.70.1-py3-none-any.whl", hash = "sha256:c293e525e6fef9c20e8728fd4612df02a0aa31bb5fe91ecd93e123b1b7bffa73", size = 80199, upload-time = "2026-09-11T07:25:14.599Z" }, +] + +[[package]] +name = "truststore" +version = "0.10.5" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/ee/9f/c5201d42a484c061e528825fc8e2d565f5abd50a4ced6fb7d29c4ec99b2b/truststore-0.10.5.tar.gz", hash = "sha256:30d36967ccaded5cbb38d602c433f53600036c79d502f4533a49b60a03bbefcd", size = 28091, upload-time = "2026-10-12T22:27:31.808Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/51/e9/3a7820be2bb0fe53b6bc9c3be26d3d1158004e4c3ab953aa6840b955b1e9/truststore-0.10.5-py3-none-any.whl", hash = "sha256:9aaaedaefaf06d8b206278cf8b5012bc897f485a874503501e12d776df78951c", size = 19017, upload-time = "2026-10-12T22:27:30.377Z" }, +] + +[[package]] +name = "types-pyyaml" +version = "6.0.12.20261030" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/c7/0c/72b85bc2e3a3cf23419a7f8861c1855893f02080e202b631e33cc0d1a94a/types_pyyaml-6.0.12.20261030.tar.gz", hash = "sha256:9123d707aa31a3883eeb2cb290537c75246dfcb1c00abed7a70f5419d3bd31ae", size = 18878, upload-time = "2026-10-30T07:50:19.43Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/34/99/7e2c7c7202108220fb664ce4c5ac313a564ab5babb153516a3ac0ad8a681/types_pyyaml-6.0.12.20261030-py3-none-any.whl", hash = "sha256:508cc80546918201026a955356222111224e1f4f7deab0b91f2a95933f764e67", size = 21283, upload-time = "2026-10-30T07:50:18.389Z" }, +] + +[[package]] +name = "typing-extensions" +version = "4.16.0" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/f6/cc/6253133b5bb138fc3306cebfbda2c520f545d36b5be2c7255cc528bb45d6/typing_extensions-4.16.0.tar.gz", hash = "sha256:dc983d19a509c94dba722ee6abd33940f7c05a89e243c47e907eb4db6f1a43e5", size = 113555, upload-time = "2026-07-02T08:40:05.92Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/49/d3/b8441a820a491ddfc024b0b0cf0393375b75ea13866d9c66727e54c2fc80/typing_extensions-4.16.0-py3-none-any.whl", hash = "sha256:481caa481374e813c1b176ada14e97f1f67a4539ce9cfeb3f350d78d6370c2e8", size = 45571, upload-time = "2026-07-02T08:40:04.659Z" }, +] + +[[package]] +name = "tzdata" +version = "2026.6" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/7a/29/6a1391d31cd00ff9e927215a224a880e1e84e826b64ba80a499e51c074d8/tzdata-2026.6.tar.gz", hash = "sha256:85d2cd9c09f81f0bb29db27759fc83af4f1e620efb814f8c5bf8b4cc4feece6d", size = 201017, upload-time = "2026-10-28T09:57:09.137Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/0a/66/9a06c2227fedfd4919f0eaaa8b94377d9730882a7e4f95823c9a331ab5e1/tzdata-2026.6-py2.py3-none-any.whl", hash = "sha256:0a10eec6e13ef16b627fabed9a5f2ba61a62b1c67f327305f4e7e412a70b4aec", size = 348210, upload-time = "2026-10-28T09:57:07.276Z" }, +] + +[[package]] +name = "urllib3" +version = "2.8.0" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/e3/05/b17359e1cefb4f909b5e40b1b90a496d987258916dbbf88e842c729f510e/urllib3-2.8.0.tar.gz", hash = "sha256:63bf2ead4c879426ebf22ef2a781eeb4aa3b4ae798a0435506f8687fd5bb9b63", size = 458972, upload-time = "2026-09-15T19:29:36.253Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/92/9d/c4e665119135114480843e7ab388fa94d8480650450e6f8e26b70d323a4c/urllib3-2.8.0-py3-none-any.whl", hash = "sha256:0cf3cae568d36aa9576b28dfb35f11328f1cb974ca7647d9475ebb86c75ac6e3", size = 135717, upload-time = "2026-09-15T19:29:34.577Z" }, +] diff --git a/cmd/tailwise/main.go b/cmd/tailwise/main.go index 24bd39c..ddba852 100644 --- a/cmd/tailwise/main.go +++ b/cmd/tailwise/main.go @@ -224,7 +224,8 @@ func explainAddrInUse(err error) error { } func (a *app) startHTTP(context.Context) error { - srv := server.New(server.Options{Engine: a.engine, Queue: a.queue, Reporter: a.reporter, Metrics: a.metrics}) + srv := server.New(server.Options{Engine: a.engine, Queue: a.queue, Reporter: a.reporter, Metrics: a.metrics, + Assembler: otlp.Assembler{AgentAttributes: a.cfg.OTLP.AgentAttributes}}) a.http = &http.Server{ Addr: a.cfg.Server.Addr, Handler: srv.Handler(), diff --git a/deploy/helm/tailwise/Chart.yaml b/deploy/helm/tailwise/Chart.yaml index 7ce5bff..f896448 100644 --- a/deploy/helm/tailwise/Chart.yaml +++ b/deploy/helm/tailwise/Chart.yaml @@ -4,8 +4,8 @@ description: >- Tailwise inference engine: scores LLM agent traces with a System One model (Jev by default) and exports the ones worth reviewing over OTLP. type: application -version: 0.1.0 -appVersion: "0.1.0" +version: 0.2.0 +appVersion: "0.2.0" keywords: [observability, opentelemetry, llm, agents, evaluation] dependencies: - name: nats diff --git a/deploy/helm/tailwise/README.md b/deploy/helm/tailwise/README.md index 8953929..0d60a99 100644 --- a/deploy/helm/tailwise/README.md +++ b/deploy/helm/tailwise/README.md @@ -171,7 +171,7 @@ Kustomize can install the chart and apply your overlays on top: helmCharts: - name: tailwise repo: oci://ghcr.io/tecnosam/charts # or a local copy: helmGlobals.chartHome - version: 0.1.0 + version: 0.2.0 releaseName: tailwise namespace: tailwise valuesFile: tailwise-values.yaml @@ -188,5 +188,5 @@ tailwise-values.yaml > base/tailwise.yaml`. The engine validates a config and its manifests without starting: ```sh -docker run --rm -v $PWD:/cfg ghcr.io/tecnosam/tailwise:0.1.0 -check -config /cfg/tailwise.yaml +docker run --rm -v $PWD:/cfg ghcr.io/tecnosam/tailwise:0.2.0 -check -config /cfg/tailwise.yaml ``` diff --git a/examples/manifests/support-bot.json b/examples/manifests/support-bot.json index ca69efb..137b70c 100644 --- a/examples/manifests/support-bot.json +++ b/examples/manifests/support-bot.json @@ -1,6 +1,6 @@ { "model": "jev-latest", - "note": "A manifest for support-bot: the questions it is asked. answer_wrong decides keep or drop, and why_wrong picks the plain-English reason.", + "note": "A manifest for support-bot: the questions it is asked. answer_wrong decides keep or drop, and why_wrong picks the plain-English reason. Its actions turn on the action checks: each call to one of these tools is checked against what the user asked for, and each run for requests of these kinds left unhandled.", "questions": { "answer_wrong": { "type": "noul", @@ -19,5 +19,10 @@ "no_answer": "The agent did not give an answer." } } + }, + "actions": { + "cancel_order": "cancel an order", + "issue_refund": "get a refund for an order", + "update_shipping_address": "change the shipping address of an order" } } diff --git a/go.mod b/go.mod index 1cab387..a08c65f 100644 --- a/go.mod +++ b/go.mod @@ -35,6 +35,7 @@ require ( github.com/nats-io/jwt/v2 v2.8.2 // indirect github.com/nats-io/nkeys v0.4.16 // indirect github.com/nats-io/nuid v1.0.1 // indirect + github.com/santhosh-tekuri/jsonschema/v6 v6.0.2 // indirect go.opentelemetry.io/auto/sdk v1.2.1 // indirect go.opentelemetry.io/otel/exporters/otlp/otlptrace v1.47.0 // indirect go.opentelemetry.io/otel/log v1.47.0 // indirect diff --git a/go.sum b/go.sum index ae3e535..4c6ec59 100644 --- a/go.sum +++ b/go.sum @@ -39,6 +39,8 @@ github.com/nats-io/nuid v1.0.1 h1:5iA8DT8V7q8WK2EScv2padNa/rTESc1KdnPw4TC2paw= github.com/nats-io/nuid v1.0.1/go.mod h1:19wcPz3Ph3q0Jbyiqsd0kePYG7A95tJPxeL+1OSON2c= github.com/rogpeppe/go-internal v1.14.1 h1:UQB4HGPB6osV0SQTLymcB4TgvyWu6ZyliaW0tI/otEQ= github.com/rogpeppe/go-internal v1.14.1/go.mod h1:MaRKkUm5W0goXpeCfT7UZI6fk/L7L7so1lCWt35ZSgc= +github.com/santhosh-tekuri/jsonschema/v6 v6.0.2 h1:KRzFb2m7YtdldCEkzs6KqmJw4nqEVZGK7IN2kJkjTuQ= +github.com/santhosh-tekuri/jsonschema/v6 v6.0.2/go.mod h1:JXeL+ps8p7/KNMjDQk3TCwPpBy0wYklyWTfbkIzdIFU= github.com/stretchr/testify v1.12.1 h1:EuwCh5fleGS7H32xRwO3wRGT7DxrDhLAT6FF8MpWDWE= github.com/stretchr/testify v1.12.1/go.mod h1:MDEgiDPPsNp5cuIrHPPCyornHKgEVbtFUmoNlxoYthg= go.opentelemetry.io/auto/sdk v1.2.1 h1:jXsnJ4Lmnqd11kwkBV2LgLoFMZKizbCi5fNZ/ipaZ64= diff --git a/internal/config/builtin_manifest.json b/internal/config/builtin_manifest.json index 9b4bed6..ac89cf2 100644 --- a/internal/config/builtin_manifest.json +++ b/internal/config/builtin_manifest.json @@ -1,6 +1,6 @@ { "model": "built-in", - "note": "The questions the engine asks when a model has no manifest of its own, all in one call. A trace is kept when answer_wrong, or any question in decision.any_of, reaches the keep threshold; any_of is in priority order after answer_wrong. why_wrong picks the plain-English reason.", + "note": "The questions the engine asks when a model has no manifest of its own, all in one call. answer_wrong alone decides keep or drop. The five checks are asked and recorded with every decision, so reviewers see which problems the model found, but they do not keep a trace: in the benchmark, letting any of them keep a trace kept most traces without ranking failures better. why_wrong picks the plain-English reason.", "questions": { "answer_wrong": { "type": "noul", @@ -39,14 +39,5 @@ "no_answer": "The agent did not give an answer." } } - }, - "decision": { - "any_of": [ - "unsupported_answer", - "missing_tool_calls", - "answered_on_failure", - "ignored_instructions", - "contradicts_knowledge" - ] } } diff --git a/internal/config/config.go b/internal/config/config.go index 4569ef2..24e8e09 100644 --- a/internal/config/config.go +++ b/internal/config/config.go @@ -6,6 +6,9 @@ import ( "encoding/json" "slices" "time" + + "github.com/tecnosam/tailwise/internal/rules" + "github.com/tecnosam/tailwise/internal/trace" ) const ( @@ -40,6 +43,9 @@ type File struct { Agents Agents `yaml:"agents"` Policy Policy `yaml:"policy"` Exporters []Exporter `yaml:"exporters"` + // Rules are the heuristic rules every agent gets; a manifest adds its own + // and overrides these by name. + Rules []rules.Config `yaml:"rules"` Observability Observability `yaml:"observability"` } @@ -127,6 +133,9 @@ type Scorer struct { Manifest string `yaml:"manifest"` DecisionQuestion string `yaml:"decision_question"` ReasonQuestion string `yaml:"reason_question"` + SystemPrompt bool `yaml:"system_prompt"` + Evidence string `yaml:"evidence"` + EvidenceBudget int `yaml:"evidence_budget"` Timeout time.Duration `yaml:"timeout"` MaxRetries int `yaml:"max_retries"` Backoff []time.Duration `yaml:"backoff"` @@ -184,8 +193,16 @@ type Manifest struct { // Decision, when set, lists more questions that decide; it stays in the // engine and is not sent to the model. Decision *ManifestDecision `json:"decision"` + // Actions maps each tool that acts for the user to what it does, worded + // as a request ("cancel a pending order"). It turns on the action checks. + Actions map[string]string `json:"actions"` + // Rules are this agent's heuristic rules, on top of the config's. + Rules []rules.Config `json:"rules"` Reasons map[string]string `json:"-"` + // ruleSet is the config's rules merged with Rules, compiled; nil when + // there are none. + ruleSet *rules.Set } // ManifestDecision lists the yes/no questions that decide besides the @@ -245,6 +262,8 @@ func Default() File { Model: JevModel, DecisionQuestion: "answer_wrong", ReasonQuestion: "why_wrong", + Evidence: trace.EvidenceGrounded, + EvidenceBudget: 12000, Timeout: 30 * time.Second, MaxRetries: 3, Backoff: []time.Duration{ diff --git a/internal/config/config_test.go b/internal/config/config_test.go index c94765f..00e42a0 100644 --- a/internal/config/config_test.go +++ b/internal/config/config_test.go @@ -1,6 +1,8 @@ package config import ( + "encoding/json" + "fmt" "os" "path/filepath" "slices" @@ -278,9 +280,17 @@ func TestLoadWithoutAFile(t *testing.T) { if m.Client.ReasonQuestion != "why_wrong" || m.Client.Reasons["contradicts_evidence"] != "The final answer contradicts the evidence the agent gathered." { t.Errorf("reason question %q, reasons %v", m.Client.ReasonQuestion, m.Client.Reasons) } - // Any of the five checks can keep a trace, as well as answer_wrong. - if a := m.Client.AnyOf; len(a) != 5 || a[0] != "unsupported_answer" || a[4] != "contradicts_knowledge" { - t.Errorf("any_of = %q, want the five built-in checks", a) + // answer_wrong alone decides; the five checks are asked and recorded only. + if a := m.Client.AnyOf; a != nil { + t.Errorf("any_of = %q, want none: answer_wrong decides alone", a) + } + if len(m.Checks) != 1 || m.Checks[0].Question != "answer_wrong" { + t.Errorf("checks = %+v, want answer_wrong alone", m.Checks) + } + for _, q := range []string{"unsupported_answer", "missing_tool_calls", "answered_on_failure", "ignored_instructions", "contradicts_knowledge"} { + if !strings.Contains(string(m.Client.Questions), `"`+q+`"`) { + t.Errorf("built-in questions no longer ask %s", q) + } } t.Setenv(DefaultJevAPIKeyEnv, "") if _, err := Load(""); err == nil || !strings.Contains(err.Error(), "$JEV_API_KEY is not set") { @@ -413,7 +423,7 @@ func TestManifestDecisionAnyOf(t *testing.T) { `{"any_of": []}`: "must list questions in any_of or set thresholds", `{"any_of": ["checks", "checks"]}`: `"checks" is listed twice`, `{"any_of": ["answer_wrong"]}`: `"answer_wrong" is listed twice, or is the decision question`, - `{"thresholds": {"checks": 0.6}}`: `"checks" is neither the decision question nor in any_of`, + `{"thresholds": {"checks": 0.6}}`: `"checks" is neither the decision question, in any_of, nor an action check`, `{"thresholds": {"answer_wrong": 0}}`: "want a number in (0, 1]", `{"thresholds": {"answer_wrong": 1.5}}`: "want a number in (0, 1]", `{"any_of": ["tone"]}`: `question "tone" not found`, @@ -426,7 +436,7 @@ func TestManifestDecisionAnyOf(t *testing.T) { } // The deciding questions become checks in priority order, the decision -// question first, weighted by rank, each with its own threshold or none. +// question first, each with its own threshold or none. func TestManifestDecisionChecks(t *testing.T) { manifest := `{"questions": {"answer_wrong": {"type": "noul"}, "a": {"type": "noul"}, "b": {"type": "noul"}},` + ` "decision": {"any_of": ["b", "a"], "thresholds": {"answer_wrong": 0.8, "a": 0.6}}}` @@ -441,16 +451,105 @@ func TestManifestDecisionChecks(t *testing.T) { if checks[0].Threshold != 0.8 || checks[1].Threshold != 0 || checks[2].Threshold != 0.6 { t.Errorf("thresholds = %v %v %v, want 0.8, the policy's (0) and 0.6", checks[0].Threshold, checks[1].Threshold, checks[2].Threshold) } - if !(checks[0].Weight > checks[1].Weight && checks[1].Weight > checks[2].Weight) { - t.Errorf("weights = %v %v %v, want them falling with priority", checks[0].Weight, checks[1].Weight, checks[2].Weight) - } // Without a decision block, the decision question is the one check. cfg, err = Load(write(t, "scorer:\n manifest: manifest.json\n", testManifest)) if err != nil { t.Fatal(err) } - if c := cfg.Models.Default.Checks; len(c) != 1 || c[0] != (pipeline.Check{Question: "answer_wrong", Weight: 1}) { - t.Errorf("checks = %+v, want answer_wrong alone with weight 1", c) + if c := cfg.Models.Default.Checks; len(c) != 1 || c[0] != (pipeline.Check{Question: "answer_wrong"}) { + t.Errorf("checks = %+v, want answer_wrong alone", c) + } +} + +// The system prompt is shown only to a model that asks for it. +func TestSystemPromptIsOptIn(t *testing.T) { + cfg, err := Load("") + if err != nil { + t.Fatal(err) + } + if cfg.Models.Default.SystemPrompt { + t.Error("system_prompt is on by default") + } + cfg, err = Load(write(t, "scorer:\n system_prompt: true\nagents:\n models:\n a:\n endpoint: http://kev:8010/v1/systemone\n model: kev-a\n", "")) + if err != nil { + t.Fatal(err) + } + if !cfg.Models.Default.SystemPrompt || cfg.Models.ByAgent["a"].SystemPrompt { + t.Errorf("scorer: %v, agent a: %v; want on for the scorer only (agent models take nothing from it)", + cfg.Models.Default.SystemPrompt, cfg.Models.ByAgent["a"].SystemPrompt) + } +} + +func TestEvidenceStrategy(t *testing.T) { + cfg, err := Load("") + if err != nil { + t.Fatal(err) + } + if m := cfg.Models.Default; m.Evidence != "grounded" || m.EvidenceBudget != 12000 { + t.Errorf("defaults: evidence %q, budget %d; want grounded, 12000", m.Evidence, m.EvidenceBudget) + } + cfg, err = Load(write(t, "scorer:\n evidence: first\n evidence_budget: 4000\n", "")) + if err != nil { + t.Fatal(err) + } + if m := cfg.Models.Default; m.Evidence != "first" || m.EvidenceBudget != 4000 { + t.Errorf("evidence %q, budget %d", m.Evidence, m.EvidenceBudget) + } + for _, bad := range []string{"scorer:\n evidence: everything\n", "scorer:\n evidence_budget: 0\n"} { + if _, err := Load(write(t, bad, "")); err == nil || !strings.Contains(err.Error(), "evidence") { + t.Errorf("%q: error = %v", bad, err) + } + } +} + +// A manifest's actions block turns on the action checks: their questions, +// and two more deciding checks after the decision question. +func TestManifestActions(t *testing.T) { + cfg, err := Load(write(t, "scorer:\n manifest: manifest.json\n", testManifest)) + if err != nil || cfg.Models.Default.Actions != nil { + t.Fatalf("no actions block: actions %+v, error %v", cfg.Models.Default.Actions, err) + } + + manifest := `{"questions": {"answer_wrong": {"type": "noul"}},` + + ` "actions": {"cancel_order": "cancel an order", "book": "book a flight"},` + + ` "decision": {"thresholds": {"unhandled_request": 0.7}}}` + cfg, err = Load(write(t, "scorer:\n manifest: manifest.json\n", manifest)) + if err != nil { + t.Fatal(err) + } + m := cfg.Models.Default + if m.Actions == nil || m.Actions.Tools["cancel_order"] != "cancel an order" { + t.Fatalf("actions = %+v", m.Actions) + } + var got []string + for _, c := range m.Checks { + got = append(got, fmt.Sprintf("%s@%v", c.Question, c.Threshold)) + } + if want := []string{"answer_wrong@0", "action_wrong@0", "unhandled_request@0.7"}; !slices.Equal(got, want) { + t.Errorf("checks = %q, want %q", got, want) + } + // Completeness asks a requested/handled pair per action, in name order. + qs := string(m.Actions.CompletenessQuestions) + for _, want := range []string{`"requested_book"`, `"handled_book"`, `"requested_cancel_order"`, + `did the user ask the agent to cancel an order?`} { + if !strings.Contains(qs, want) { + t.Errorf("completeness questions lack %q: %s", want, qs) + } + } + if strings.Index(qs, "requested_book") > strings.Index(qs, "requested_cancel_order") { + t.Errorf("completeness questions out of order: %s", qs) + } + if !json.Valid(m.Actions.ActionQuestions) || !strings.Contains(string(m.Actions.ActionQuestions), `"action_wrong"`) { + t.Errorf("action questions = %s", m.Actions.ActionQuestions) + } + + for bad, wantErr := range map[string]string{ + `{"questions": {"answer_wrong": {"type": "noul"}}, "actions": {"cancel": ""}}`: "needs a name and a description", + `{"questions": {"answer_wrong": {"type": "noul"}, "action_wrong": {"type": "noul"}}, "actions": {"cancel": "cancel"}}`: "rename the question", + } { + if _, err := Load(write(t, "scorer:\n manifest: manifest.json\n", bad)); err == nil || !strings.Contains(err.Error(), wantErr) { + t.Errorf("%s: error = %v, want %q", bad, err, wantErr) + } } } diff --git a/internal/config/engine.go b/internal/config/engine.go index 848efd0..6d6c80b 100644 --- a/internal/config/engine.go +++ b/internal/config/engine.go @@ -134,7 +134,7 @@ func (f File) engine(sec secrets, manifests map[string]*Manifest) *Engine { } func model(s Scorer, apiKey string, m *Manifest) scorer.ModelConfig { - return scorer.ModelConfig{ + mc := scorer.ModelConfig{ Client: scorer.Config{ Endpoint: s.Endpoint, Model: s.Model, @@ -146,9 +146,18 @@ func model(s Scorer, apiKey string, m *Manifest) scorer.ModelConfig { APIKey: apiKey, Timeout: s.Timeout, }, - Retry: retry.Policy{Attempts: s.MaxRetries + 1, Backoff: s.Backoff, Jitter: true}, - Checks: m.checks(s.DecisionQuestion), + Retry: retry.Policy{Attempts: s.MaxRetries + 1, Backoff: s.Backoff, Jitter: true}, + Checks: m.checks(s.DecisionQuestion), + SystemPrompt: s.SystemPrompt, + Evidence: s.Evidence, + EvidenceBudget: s.EvidenceBudget, + Actions: m.actions(), } + // A nil *rules.Set would be a non-nil pipeline.Rules. + if m.ruleSet != nil { + mc.Rules = m.ruleSet + } + return mc } func reasonQuestion(s Scorer, m *Manifest) string { diff --git a/internal/config/load.go b/internal/config/load.go index 5435d8c..ab0b452 100644 --- a/internal/config/load.go +++ b/internal/config/load.go @@ -11,6 +11,8 @@ import ( "slices" "gopkg.in/yaml.v3" + + "github.com/tecnosam/tailwise/internal/rules" ) // Load reads the YAML file at path and builds the engine's settings with the @@ -87,6 +89,7 @@ func Build(f File, dir string, getenv func(string) string) (*Engine, error) { func (f File) clone() File { f.Exporters = append([]Exporter(nil), f.Exporters...) + f.Rules = append([]rules.Config(nil), f.Rules...) f.Agents.Models = maps.Clone(f.Agents.Models) return f } @@ -225,6 +228,9 @@ func loadManifests(f File) (map[string]*Manifest, []string, error) { "likely the answer is wrong, not why. Add the choice question, or set %s.reason_question to \"\".", key, s.ReasonQuestion, key)) } + if m.ruleSet, err = rules.New(rules.Merge(f.Rules, m.Rules)); err != nil { + return fmt.Errorf("%s: %w", key, err) + } out[agent] = m return nil } diff --git a/internal/config/manifest.go b/internal/config/manifest.go index ef6efb9..7d1c2cc 100644 --- a/internal/config/manifest.go +++ b/internal/config/manifest.go @@ -6,8 +6,10 @@ import ( _ "embed" "encoding/json" "fmt" + "maps" "os" "slices" + "strings" "github.com/tecnosam/tailwise/internal/pipeline" ) @@ -102,9 +104,82 @@ func (m *Manifest) prepare(decisionQuestion string) error { if head.Type != questionTypeNoul { return fmt.Errorf("decision question %q has type %q, want %q", decisionQuestion, head.Type, questionTypeNoul) } + if err := m.checkActions(questions); err != nil { + return err + } return m.checkDecision(questions, decisionQuestion) } +// checkActions accepts named tools with a description each, and keeps the +// action checks' answer names free. +func (m *Manifest) checkActions(questions map[string]json.RawMessage) error { + for tool, what := range m.Actions { + if strings.TrimSpace(tool) == "" || strings.TrimSpace(what) == "" { + return fmt.Errorf("actions: every tool needs a name and a description of what it does, got %q: %q", tool, what) + } + } + if len(m.Actions) > 0 { + for _, q := range []string{pipeline.ActionWrong, pipeline.UnhandledRequest} { + if _, ok := questions[q]; ok { + return fmt.Errorf("questions: %q is the name of an action check's answer; rename the question", q) + } + } + } + return nil +} + +// actions is the action checks' configuration, nil without an actions block. +func (m *Manifest) actions() *pipeline.Actions { + if len(m.Actions) == 0 { + return nil + } + return &pipeline.Actions{ + Tools: maps.Clone(m.Actions), + ActionQuestions: actionQuestions, + CompletenessQuestions: completenessQuestions(m.Actions), + } +} + +// One-of-several keeps a request handled by another action from being blamed +// on this one. +var actionQuestions = mustQuestions([][2]string{{pipeline.ActionWrong, + "Does the ACTION use any argument that is not what the user asked for: a different item, product " + + "option, order, account, booking, recipient, amount, date, address or payment method than the user " + + "requested, according to the USER REQUEST and the LOOKUP RESULTS? Judge this ACTION only: the user's " + + "other requests may be handled by the LATER ACTIONS, so a request this ACTION does not cover is not " + + "a problem with it."}}) + +// completenessQuestions asks, for each action in name order, whether the user +// asked for it and whether every such request was handled. +func completenessQuestions(actions map[string]string) json.RawMessage { + var qs [][2]string + for _, tool := range slices.Sorted(maps.Keys(actions)) { + what := actions[tool] + qs = append(qs, + [2]string{pipeline.RequestedQuestion(tool), "At any point in the USER REQUEST, did the user ask the agent to " + what + "?"}, + [2]string{pipeline.HandledQuestion(tool), "Was every user request to " + what + " HANDLED, as defined in the NOTES?"}) + } + return mustQuestions(qs) +} + +// mustQuestions encodes yes/no questions as a questions object, in order. +func mustQuestions(qs [][2]string) json.RawMessage { + var b bytes.Buffer + b.WriteByte('{') + for i, q := range qs { + if i > 0 { + b.WriteByte(',') + } + name, _ := json.Marshal(q[0]) + spec, _ := json.Marshal(map[string]string{"type": questionTypeNoul, "instructions": q[1]}) + b.Write(name) + b.WriteByte(':') + b.Write(spec) + } + b.WriteByte('}') + return b.Bytes() +} + // checkDecision accepts a decision block that lists only yes/no questions of // the manifest, each once, and sets thresholds only for questions that decide. func (m *Manifest) checkDecision(questions map[string]json.RawMessage, decisionQuestion string) error { @@ -115,6 +190,9 @@ func (m *Manifest) checkDecision(questions map[string]json.RawMessage, decisionQ return fmt.Errorf("\"decision\" must list questions in any_of or set thresholds") } deciding := map[string]bool{decisionQuestion: true} + if len(m.Actions) > 0 { + deciding[pipeline.ActionWrong], deciding[pipeline.UnhandledRequest] = true, true + } for _, q := range m.Decision.AnyOf { if deciding[q] { return fmt.Errorf("decision.any_of: question %q is listed twice, or is the decision question, which always comes first", q) @@ -133,7 +211,7 @@ func (m *Manifest) checkDecision(questions map[string]json.RawMessage, decisionQ } for q, t := range m.Decision.Thresholds { if !deciding[q] { - return fmt.Errorf("decision.thresholds: question %q is neither the decision question nor in any_of", q) + return fmt.Errorf("decision.thresholds: question %q is neither the decision question, in any_of, nor an action check", q) } if t <= 0 || t > 1 { return fmt.Errorf("decision.thresholds: %q is %v, want a number in (0, 1]", q, t) @@ -150,14 +228,16 @@ func (m *Manifest) anyOf() []string { return slices.Clone(m.Decision.AnyOf) } -// checks are the questions that decide, the decision question first and then -// any_of in order, weighted by that priority. +// checks are the questions that decide, the decision question first, then +// any_of in order, then the action checks. func (m *Manifest) checks(decisionQuestion string) []pipeline.Check { questions := append([]string{decisionQuestion}, m.anyOf()...) - weights := pipeline.PriorityWeights(len(questions)) + if len(m.Actions) > 0 { + questions = append(questions, pipeline.ActionWrong, pipeline.UnhandledRequest) + } checks := make([]pipeline.Check, len(questions)) for i, q := range questions { - checks[i] = pipeline.Check{Question: q, Weight: weights[i]} + checks[i] = pipeline.Check{Question: q} if m.Decision != nil { checks[i].Threshold = pipeline.Probability(m.Decision.Thresholds[q]) } diff --git a/internal/config/rules_test.go b/internal/config/rules_test.go new file mode 100644 index 0000000..53aeaad --- /dev/null +++ b/internal/config/rules_test.go @@ -0,0 +1,65 @@ +package config + +import ( + "os" + "path/filepath" + "strings" + "testing" + + "github.com/tecnosam/tailwise/internal/trace" +) + +const rulesManifest = `{ + "model": "support", + "questions": {"answer_wrong": {"type": "noul", "instructions": "is the support answer wrong?"}}, + "rules": [ + {"name": "answer_json", "on_fail": "signal", "final_answer": {"json": true}}, + {"name": "no_loops", "enabled": false}, + {"name": "order_schema", "on_fail": "keep", "tool_output": {"tool": "get_order", "json_schema": {"required": ["id"]}}} + ] +}` + +const defaultRules = ` +rules: + - name: answer_json + on_fail: keep + final_answer: {json: true} + - name: no_loops + run: {max_repeated_calls: 3} +agents: + models: + support-bot: + endpoint: http://kev-support:8010/v1/systemone + model: kev-support + manifest: rules.json +` + +// The config's rules apply to every model; a manifest overrides them by name, +// turns one off, and adds its own. +func TestRulesMergeWithManifest(t *testing.T) { + path := writeAgents(t, defaultRules) + if err := os.WriteFile(filepath.Join(filepath.Dir(path), "rules.json"), []byte(rulesManifest), 0o644); err != nil { + t.Fatal(err) + } + cfg, err := Load(path) + if err != nil { + t.Fatal(err) + } + looping := trace.Trace{Question: "q", FinalAnswer: "not json", ToolCalls: []trace.ToolCall{ + {Name: "get_order"}, {Name: "get_order"}, {Name: "get_order"}, {Name: "get_order"}}} + def := cfg.Models.Default.Rules.Check(looping) + if len(def) != 2 || !def[0].Keep || !def[0].Fired || def[1].Name != "no_loops" || !def[1].Fired { + t.Errorf("default model: %+v", def) + } + own := cfg.Models.ByAgent["support-bot"].Rules.Check(looping) + if len(own) != 2 || own[0].Name != "answer_json" || own[0].Keep || own[1].Name != "order_schema" || !own[1].Keep { + t.Errorf("support-bot: %+v", own) + } +} + +func TestRulesErrorsNameTheModel(t *testing.T) { + _, err := Load(write(t, "scorer:\n manifest: manifest.json\nrules:\n - name: bad\n final_answer: {matches: '('}\n", testManifest)) + if err == nil || !strings.Contains(err.Error(), "scorer: rules[0] (bad): final_answer: invalid regular expression") { + t.Errorf("err = %v", err) + } +} diff --git a/internal/config/validate.go b/internal/config/validate.go index ec5c20b..6a7ea74 100644 --- a/internal/config/validate.go +++ b/internal/config/validate.go @@ -10,6 +10,7 @@ import ( "time" exportnats "github.com/tecnosam/tailwise/internal/export/natsq" + "github.com/tecnosam/tailwise/internal/trace" ) func (c File) validate(p *problems) { @@ -34,6 +35,10 @@ func validateScorer(p *problems, key string, s Scorer) { p.url(key+".endpoint", s.Endpoint) p.nonEmpty(key+".model", s.Model) p.nonEmpty(key+".decision_question", s.DecisionQuestion) + if s.Evidence != trace.EvidenceFirst && s.Evidence != trace.EvidenceGrounded { + p.add("%s.evidence must be %q or %q, got %q", key, trace.EvidenceFirst, trace.EvidenceGrounded, s.Evidence) + } + p.atLeast(key+".evidence_budget", s.EvidenceBudget, 1) p.positive(key+".timeout", s.Timeout) p.atLeast(key+".max_retries", s.MaxRetries, 0) p.backoff(key+".backoff", s.Backoff) diff --git a/internal/export/natsq/queue.go b/internal/export/natsq/queue.go index 271c127..f13b79f 100644 --- a/internal/export/natsq/queue.go +++ b/internal/export/natsq/queue.go @@ -129,6 +129,14 @@ type recordMessage struct { Answers map[string]pipeline.Probability `json:"answers,omitempty"` Flags []string `json:"flags,omitempty"` + Rules []ruleMessage `json:"rules,omitempty"` +} + +// ruleMessage is a rule that fired. +type ruleMessage struct { + Name string `json:"name"` + Keep bool `json:"keep,omitempty"` + Reason string `json:"reason"` } // A trace too big for the server keeps its decision but loses its original @@ -140,6 +148,9 @@ func encodeRecord(rec pipeline.Record, maxPayload int) ([]byte, error) { Rule: string(d.Rule), Reason: d.Reason, Cause: d.Cause, Severity: d.Severity, Model: string(rec.Model), LatencyNS: int64(rec.Latency), Answers: rec.Answers, Flags: rec.Flags, } + for _, r := range rec.Rules { + m.Rules = append(m.Rules, ruleMessage{Name: r.Name, Keep: r.Keep, Reason: r.Reason}) + } data, err := json.Marshal(m) if err != nil || maxPayload <= 0 || len(data) <= maxPayload || len(m.Trace.OTLP) == 0 { return data, err @@ -153,6 +164,10 @@ func decodeRecord(data []byte) (pipeline.Record, error) { if err := json.Unmarshal(data, &m); err != nil { return pipeline.Record{}, err } + var fired []pipeline.RuleResult + for _, r := range m.Rules { + fired = append(fired, pipeline.RuleResult{Name: r.Name, Keep: r.Keep, Fired: true, Reason: r.Reason}) + } return pipeline.Record{ Trace: m.Trace, Agent: pipeline.AgentLabel(m.Agent), @@ -164,5 +179,6 @@ func decodeRecord(data []byte) (pipeline.Record, error) { Latency: time.Duration(m.LatencyNS), Answers: m.Answers, Flags: m.Flags, + Rules: fired, }, nil } diff --git a/internal/export/natsq/queue_test.go b/internal/export/natsq/queue_test.go index bf5112d..c579a1c 100644 --- a/internal/export/natsq/queue_test.go +++ b/internal/export/natsq/queue_test.go @@ -112,6 +112,7 @@ func TestRecordSurvivesTheStream(t *testing.T) { Latency: 93 * time.Millisecond, Answers: map[string]pipeline.Probability{"answer_wrong": 0.95, "unsupported_answer": 0.75}, Flags: []string{"answer_wrong", "unsupported_answer"}, + Rules: []pipeline.RuleResult{{Name: "answer_json", Keep: true, Fired: true, Reason: "not JSON"}}, } if err := q.Enqueue(context.Background(), want); err != nil { t.Fatal(err) @@ -120,7 +121,7 @@ func TestRecordSurvivesTheStream(t *testing.T) { case rec := <-got: if rec.Trace.TraceID != "t1" || rec.Trace.Agent != "support-bot" || rec.Agent != "support-bot" || string(rec.Trace.OTLP) != "\x01\x02\x03" || rec.Decision != want.Decision || rec.Model != want.Model || rec.Latency != want.Latency || - !maps.Equal(rec.Answers, want.Answers) || !slices.Equal(rec.Flags, want.Flags) { + !maps.Equal(rec.Answers, want.Answers) || !slices.Equal(rec.Flags, want.Flags) || !slices.Equal(rec.Rules, want.Rules) { t.Errorf("received %+v, want %+v", rec, want) } case <-time.After(10 * time.Second): diff --git a/internal/export/otlpbackend/backend.go b/internal/export/otlpbackend/backend.go index c5930fb..cbac2e6 100644 --- a/internal/export/otlpbackend/backend.go +++ b/internal/export/otlpbackend/backend.go @@ -110,9 +110,22 @@ func evalAttrs(rec pipeline.Record) []*commonpb.KeyValue { if d.Scored { attrs = append(attrs, otlp.FloatAttr("eval.p_keep", float64(d.PKeep))) } + if d.Ranked() { + attrs = append(attrs, otlp.FloatAttr("eval.severity", d.Severity)) + } if len(rec.Flags) > 0 { - attrs = append(attrs, otlp.StringAttr("eval.flags", strings.Join(rec.Flags, ",")), - otlp.FloatAttr("eval.severity", d.Severity)) + attrs = append(attrs, otlp.StringAttr("eval.flags", strings.Join(rec.Flags, ","))) + } + if len(rec.Rules) > 0 { + names := make([]string, len(rec.Rules)) + for i, r := range rec.Rules { + names[i] = r.Name + attrs = append(attrs, otlp.StringAttr("eval.rules."+r.Name, r.Reason)) + } + attrs = append(attrs, otlp.StringAttr("eval.rules", strings.Join(names, ","))) + } + if rec.Trimmed { + attrs = append(attrs, otlp.BoolAttr("eval.trimmed", true)) } for _, q := range slices.Sorted(maps.Keys(rec.Answers)) { attrs = append(attrs, otlp.FloatAttr("eval.answers."+q, float64(rec.Answers[q]))) diff --git a/internal/otlp/convert.go b/internal/otlp/convert.go index d7f9fb3..bb0ed76 100644 --- a/internal/otlp/convert.go +++ b/internal/otlp/convert.go @@ -30,17 +30,27 @@ const ( kindAgent = "AGENT" kindChain = "CHAIN" + kindLLM = "LLM" kindTool = "TOOL" kindRetriever = "RETRIEVER" retrievalDocFmt = "retrieval.documents.%d.document.content" + + // An LLM span's prompt, one message per index; a message's text is in + // content, or split over contents parts. + inputMessageRoleFmt = "llm.input_messages.%d.message.role" + inputMessageContentFmt = "llm.input_messages.%d.message.content" + inputMessagePartFmt = "llm.input_messages.%d.message.contents.%d.message_content.text" + outputMessageRole = "llm.output_messages.0.message.role" + outputMessageContent = "llm.output_messages.0.message.content" ) // Attributes an agent can set to say exactly what the model should see, // instead of having it inferred from tool output. const ( - AttrEvidenceFmt = "tailwise.evidence.%d" - AttrToolStatus = "tailwise.tool.status" + AttrEvidenceFmt = "tailwise.evidence.%d" + AttrToolStatus = "tailwise.tool.status" + AttrSystemPrompt = "tailwise.system_prompt" ) // ErrNotScorable means there is no root span with a question and an answer. @@ -106,7 +116,10 @@ func Assemble(traceID string, payloads [][]byte) (trace.Trace, error) { // Assemble merges a trace's span batches into the record the model scores, // carrying the spans in its OTLP field. An empty output.value still scores: -// the agent ran and gave up. +// the agent ran and gave up. So does an AGENT root with no output.value at +// all, as an empty answer: the run crashed or ran out of steps, which is a +// failure worth review. Any other root needs one, which keeps ordinary +// service spans out. func (a Assembler) Assemble(traceID string, payloads [][]byte) (trace.Trace, error) { merged, spans, err := merge(payloads) if err != nil { @@ -121,7 +134,7 @@ func (a Assembler) Assemble(traceID string, payloads [][]byte) (trace.Trace, err return trace.Trace{}, fmt.Errorf("%w: root span %q has no %s", ErrNotScorable, root.GetName(), attrInputValue) } output, answered := lookup(root, attrOutputValue) - if !answered { + if !answered && attr(root, attrSpanKind) != kindAgent { return trace.Trace{}, fmt.Errorf("%w: root span %q has no %s", ErrNotScorable, root.GetName(), attrOutputValue) } answer := messageText(output, answerKeys) @@ -129,14 +142,16 @@ func (a Assembler) Assemble(traceID string, payloads [][]byte) (trace.Trace, err answer = "" } - tools := spansOfKind(spans, kindTool) + calls := toolCallsOf(root, spans) t := trace.Trace{ - TraceID: traceID, - Agent: a.agentOf(root, resourceOf(merged, root)), - Question: question, - FinalAnswer: answer, - ToolCalls: toolCalls(tools), - Evidence: evidence(root, spans, tools), + TraceID: traceID, + Agent: a.agentOf(root, resourceOf(merged, root)), + SystemPrompt: systemPrompt(root, spans), + Question: question, + FinalAnswer: answer, + ToolCalls: calls, + Evidence: evidence(root, spans, calls), + Conversation: conversation(spans), } if t.OTLP, err = proto.Marshal(merged); err != nil { return trace.Trace{}, fmt.Errorf("otlp: encode trace %s: %w", traceID, err) @@ -248,18 +263,14 @@ func spansOfKind(spans []*tracepb.Span, kind string) []*tracepb.Span { return out } -func toolCalls(tools []*tracepb.Span) []trace.ToolCall { - var calls []trace.ToolCall - for _, s := range tools { - name := attr(s, attrToolName) - if name == "" { - name = s.GetName() - } - call := trace.ToolCall{Name: name, Args: toolArgs(s)} - call.Status, _ = json.Marshal(toolStatus(s)) - calls = append(calls, call) +func toolCall(s *tracepb.Span) trace.ToolCall { + name := attr(s, attrToolName) + if name == "" { + name = s.GetName() } - return calls + call := trace.ToolCall{Name: name, Args: toolArgs(s), Output: messageText(attr(s, attrOutputValue), outputKeys)} + call.Status, _ = json.Marshal(toolStatus(s)) + return call } // toolArgs is the span's input if it is JSON, else its declared parameters, @@ -285,13 +296,94 @@ func toolStatus(s *tracepb.Span) string { if s.GetStatus().GetCode() == tracepb.Status_STATUS_CODE_ERROR { return "error" } + // Many tools report a failure as their output and leave the span OK. + if reportsFailure(messageText(attr(s, attrOutputValue), outputKeys)) { + return "error" + } return "ok" } +// reportsFailure is an output that starts "Error:", in any case. +func reportsFailure(output string) bool { + head := strings.TrimSpace(output) + return len(head) >= len("error:") && strings.EqualFold(head[:len("error:")], "error:") +} + +// systemPrompt is tailwise.system_prompt on the root if the agent set it, else +// the system message of the first LLM call that has one: the agent's own +// instructions, as its instrumentor recorded them. +func systemPrompt(root *tracepb.Span, spans []*tracepb.Span) string { + if explicit, ok := lookup(root, AttrSystemPrompt); ok { + return explicit + } + for _, s := range spansOfKind(spans, kindLLM) { + for i := 0; ; i++ { + role, ok := lookup(s, fmt.Sprintf(inputMessageRoleFmt, i)) + if !ok { + break + } + if role == "system" { + if text := messageContent(s, i); strings.TrimSpace(text) != "" { + return text + } + } + } + } + return "" +} + +// conversation is the user's and assistant's turns as the latest LLM call saw +// them: the call with the longest history, its input messages and its reply. +// System and tool messages are left out, and so are empty turns. +func conversation(spans []*tracepb.Span) []trace.Message { + var last *tracepb.Span + most := 0 + for _, s := range spansOfKind(spans, kindLLM) { + n := 0 + for ; ; n++ { + if _, ok := lookup(s, fmt.Sprintf(inputMessageRoleFmt, n)); !ok { + break + } + } + if n > 0 && n >= most { + last, most = s, n + } + } + if last == nil { + return nil + } + var out []trace.Message + add := func(role, content string) { + if (role == "user" || role == "assistant") && strings.TrimSpace(content) != "" { + out = append(out, trace.Message{Role: role, Content: content}) + } + } + for i := range most { + add(attr(last, fmt.Sprintf(inputMessageRoleFmt, i)), messageContent(last, i)) + } + add(cmp.Or(attr(last, outputMessageRole), "assistant"), attr(last, outputMessageContent)) + return out +} + +func messageContent(s *tracepb.Span, i int) string { + if content, ok := lookup(s, fmt.Sprintf(inputMessageContentFmt, i)); ok { + return content + } + var parts []string + for j := 0; ; j++ { + text, ok := lookup(s, fmt.Sprintf(inputMessagePartFmt, i, j)) + if !ok { + break + } + parts = append(parts, text) + } + return strings.Join(parts, "\n") +} + // evidence comes from the most explicit source available: evidence attributes // on the root, then RETRIEVER documents, then paragraphs of successful tool // outputs. -func evidence(root *tracepb.Span, spans, tools []*tracepb.Span) []string { +func evidence(root *tracepb.Span, spans []*tracepb.Span, calls []trace.ToolCall) []string { if explicit := indexed(root, AttrEvidenceFmt); len(explicit) > 0 { return explicit } @@ -311,11 +403,11 @@ func evidence(root *tracepb.Span, spans, tools []*tracepb.Span) []string { if len(out) > 0 { return out } - for _, s := range tools { - if toolStatus(s) != "ok" { + for _, c := range calls { + if c.StatusText() != "ok" { continue } - for _, paragraph := range strings.Split(messageText(attr(s, attrOutputValue), outputKeys), "\n\n") { + for _, paragraph := range strings.Split(c.Output, "\n\n") { add(paragraph) } } @@ -334,7 +426,7 @@ func indexed(s *tracepb.Span, format string) []string { } var ( - questionKeys = []string{"question", "input", "query", "prompt", "text", "content"} + questionKeys = []string{"question", "input", "query", "prompt", "task", "text", "content"} answerKeys = []string{"final_answer", "answer", "output", "result", "response", "text", "content"} outputKeys = []string{"result", "output", "content", "text"} ) @@ -488,3 +580,7 @@ func StringAttr(key, value string) *commonpb.KeyValue { func FloatAttr(key string, value float64) *commonpb.KeyValue { return &commonpb.KeyValue{Key: key, Value: &commonpb.AnyValue{Value: &commonpb.AnyValue_DoubleValue{DoubleValue: value}}} } + +func BoolAttr(key string, value bool) *commonpb.KeyValue { + return &commonpb.KeyValue{Key: key, Value: &commonpb.AnyValue{Value: &commonpb.AnyValue_BoolValue{BoolValue: value}}} +} diff --git a/internal/otlp/convert_test.go b/internal/otlp/convert_test.go index a876c7b..56f6155 100644 --- a/internal/otlp/convert_test.go +++ b/internal/otlp/convert_test.go @@ -213,6 +213,47 @@ func TestAssembleEvidenceSources(t *testing.T) { } } +func TestAssembleSystemPrompt(t *testing.T) { + root := func(extra ...string) *tracepb.Span { + return span(1, 0, "run", 100, append([]string{attrSpanKind, kindAgent, attrInputValue, "q", attrOutputValue, "a"}, extra...)...) + } + msg := func(i int, field string) string { return fmt.Sprintf("llm.input_messages.%d.message.%s", i, field) } + // The first LLM call: a user message, then the system message. + first := span(3, 1, "llm", 200, attrSpanKind, kindLLM, + msg(0, "role"), "user", msg(0, "content"), "hello", + msg(1, "role"), "system", msg(1, "content"), "Only refund delivered orders.") + later := span(4, 1, "llm", 300, attrSpanKind, kindLLM, + msg(0, "role"), "system", msg(0, "content"), "A sub-agent's prompt.") + parts := span(3, 1, "llm", 200, attrSpanKind, kindLLM, + msg(0, "role"), "system", + msg(0, "contents.0.message_content.text"), "Be brief.", + msg(0, "contents.1.message_content.text"), "Cite sources.") + noSystem := span(3, 1, "llm", 200, attrSpanKind, kindLLM, msg(0, "role"), "user", msg(0, "content"), "hi") + + tests := []struct { + name string + spans []*tracepb.Span + want string + }{ + {"the explicit attribute wins", []*tracepb.Span{root(AttrSystemPrompt, "explicit"), first}, "explicit"}, + {"the first LLM call's system message", []*tracepb.Span{root(), later, first}, "Only refund delivered orders."}, + {"a message split into parts", []*tracepb.Span{root(), parts}, "Be brief.\nCite sources."}, + {"no system message", []*tracepb.Span{root(), noSystem}, ""}, + {"no LLM spans", []*tracepb.Span{root()}, ""}, + } + for _, tt := range tests { + t.Run(tt.name, func(t *testing.T) { + got, err := assemble(t, tt.spans) + if err != nil { + t.Fatal(err) + } + if got.SystemPrompt != tt.want { + t.Errorf("system prompt = %q, want %q", got.SystemPrompt, tt.want) + } + }) + } +} + func TestAssembleToolCallDetails(t *testing.T) { failed := span(2, 1, "execute_tool search", 200, attrSpanKind, kindTool, attrToolName, "search", attrInputValue, `{"query": "x"}`) failed.Status = &tracepb.Status{Code: tracepb.Status_STATUS_CODE_ERROR, Message: "boom"} @@ -228,6 +269,12 @@ func TestAssembleToolCallDetails(t *testing.T) { span(5, 1, "fetch", 500, attrSpanKind, kindTool, attrToolName, "fetch", attrToolParameters, `{"url": "u"}`), // No input at all. span(6, 1, "now", 600, attrSpanKind, kindTool, attrToolName, "now"), + // A failure reported in the output, with the span left OK, and one + // that only mentions an error. + span(7, 1, "cancel", 700, attrSpanKind, kindTool, attrToolName, "cancel", + attrOutputValue, " error: order not found"), + span(8, 1, "search", 800, attrSpanKind, kindTool, attrToolName, "search", + attrOutputValue, `{"result": "Error handling in Go uses values."}`), } got, err := assemble(t, spans) if err != nil { @@ -238,8 +285,10 @@ func TestAssembleToolCallDetails(t *testing.T) { "2. lookup({\"title\":\"T\"}) -> not_found\n" + "3. calculator(\"2 + 2\") -> ok\n" + "4. fetch({\"url\":\"u\"}) -> ok\n" + - "5. now() -> ok\n\n" + - "EVIDENCE:\n(none)\n\nFINAL ANSWER:\na" + "5. now() -> ok\n" + + "6. cancel() -> error\n" + + "7. search() -> ok\n\n" + + "EVIDENCE:\n[1] Error handling in Go uses values.\n\nFINAL ANSWER:\na" if state := string(trace.BuildState(got)); state != want { t.Errorf("state =\n%s\nwant\n%s", state, want) } @@ -250,6 +299,8 @@ func TestAssembleQuestionAndAnswerFromJSON(t *testing.T) { {"plain question", "plain answer", "plain question", "plain answer"}, {`{"question": "What?"}`, `{"answer": "This."}`, "What?", "This."}, {`{"input": "What?", "session": 3}`, `{"output": "This."}`, "What?", "This."}, + // smolagents records a run's input as {"task": ...}. + {`{"task": "What?", "additional_args": {}}`, `"This."`, "What?", "This."}, {`"quoted question"`, `"quoted answer"`, "quoted question", "quoted answer"}, // JSON without a recognised key is passed through whole. {`{"messages": []}`, `{"choices": []}`, `{"messages": []}`, `{"choices": []}`}, @@ -304,6 +355,18 @@ func TestAssembleADKInvocation(t *testing.T) { } } +// An agent run that ended without an answer (it crashed, or ran out of +// steps) is scored with an empty one. +func TestAssembleAgentWithoutAnswer(t *testing.T) { + got, err := assemble(t, []*tracepb.Span{span(1, 0, "CodeAgent.run", 100, attrSpanKind, kindAgent, attrInputValue, "q")}) + if err != nil { + t.Fatal(err) + } + if got.Question != "q" || got.FinalAnswer != "" { + t.Errorf("question = %q, final answer = %q", got.Question, got.FinalAnswer) + } +} + func TestAssembleNotScorable(t *testing.T) { tests := []struct { name string @@ -311,7 +374,7 @@ func TestAssembleNotScorable(t *testing.T) { wantErr string }{ {"no question", []*tracepb.Span{span(1, 0, "GET /users", 100, "http.method", "GET")}, "has no input.value"}, - {"no answer", []*tracepb.Span{span(1, 0, "run", 100, attrSpanKind, kindAgent, attrInputValue, "q")}, "has no output.value"}, + {"no answer", []*tracepb.Span{span(1, 0, "run", 100, attrSpanKind, kindChain, attrInputValue, "q")}, "has no output.value"}, // Only children arrived: the best available root is a tool span. {"root span missing", []*tracepb.Span{span(2, 1, "search", 100, attrSpanKind, kindTool, attrInputValue, `{"q": 1}`)}, "has no output.value"}, } diff --git a/internal/otlp/fromtrace.go b/internal/otlp/fromtrace.go index 3900978..51c513c 100644 --- a/internal/otlp/fromtrace.go +++ b/internal/otlp/fromtrace.go @@ -9,9 +9,9 @@ import ( "github.com/tecnosam/tailwise/internal/trace" ) -// SpansFromTrace sets the override attributes (evidence on the root, status on -// each tool), so sending these spans to the receiver reproduces the record -// exactly. +// SpansFromTrace sets the override attributes (evidence and the system prompt +// on the root, status on each tool), so sending these spans to the receiver +// reproduces the record exactly. func SpansFromTrace(traceID []byte, t trace.Trace) []*tracepb.Span { const rootID = 1 id := func(n int) []byte { return []byte{0, 0, 0, 0, 0, 0, byte(n >> 8), byte(n)} } @@ -31,6 +31,9 @@ func SpansFromTrace(traceID []byte, t trace.Trace) []*tracepb.Span { for i, e := range t.Evidence { root.Attributes = append(root.Attributes, StringAttr(fmt.Sprintf(AttrEvidenceFmt, i), e)) } + if t.SystemPrompt != "" { + root.Attributes = append(root.Attributes, StringAttr(AttrSystemPrompt, t.SystemPrompt)) + } spans := []*tracepb.Span{root} for i, c := range t.ToolCalls { @@ -51,6 +54,9 @@ func SpansFromTrace(traceID []byte, t trace.Trace) []*tracepb.Span { if len(c.Args) > 0 { s.Attributes = append(s.Attributes, StringAttr(attrInputValue, string(c.Args))) } + if c.Output != "" { + s.Attributes = append(s.Attributes, StringAttr(attrOutputValue, c.Output)) + } spans = append(spans, s) } return spans diff --git a/internal/otlp/json.go b/internal/otlp/json.go index cfc2262..e381bf9 100644 --- a/internal/otlp/json.go +++ b/internal/otlp/json.go @@ -17,7 +17,8 @@ var idFields = map[string]bool{ "trace_id": true, "span_id": true, "parent_span_id": true, } -func unmarshalJSON(data []byte, req *coltracepb.ExportTraceServiceRequest) error { +// UnmarshalJSON decodes an OTLP/JSON export request, IDs in hex. +func UnmarshalJSON(data []byte, req *coltracepb.ExportTraceServiceRequest) error { dec := json.NewDecoder(bytes.NewReader(data)) dec.UseNumber() // keep 64-bit timestamps exact var doc any diff --git a/internal/otlp/receiver.go b/internal/otlp/receiver.go index fa9e3c7..51b499a 100644 --- a/internal/otlp/receiver.go +++ b/internal/otlp/receiver.go @@ -171,43 +171,49 @@ func (r *Receiver) accept(ctx context.Context, req *coltracepb.ExportTraceServic } func (r *Receiver) handleHTTP(w http.ResponseWriter, req *http.Request) { - contentType, _, _ := mime.ParseMediaType(req.Header.Get("Content-Type")) - if contentType != contentTypeProto && contentType != contentTypeJSON { - // The error has to be written in some encoding; JSON is readable. - writeStatus(w, contentTypeJSON, http.StatusUnsupportedMediaType, codes.InvalidArgument, - "Content-Type must be application/x-protobuf or application/json") + export, contentType, code, err := ReadExport(w, req) + if err != nil { + writeStatus(w, contentType, code, codes.InvalidArgument, err.Error()) return } + if err := r.accept(req.Context(), export); err != nil { + // 503 is one of the statuses OTLP exporters retry. + w.Header().Set("Retry-After", retryAfterSeconds) + writeStatus(w, contentType, http.StatusServiceUnavailable, codes.Unavailable, err.Error()) + return + } + writeMessage(w, contentType, http.StatusOK, &coltracepb.ExportTraceServiceResponse{}) +} +// ReadExport decodes an OTLP/HTTP request body, protobuf or JSON, gzipped or +// not. On failure it returns the HTTP status to answer with, and the content +// type to write the error in. +func ReadExport(w http.ResponseWriter, req *http.Request) (export *coltracepb.ExportTraceServiceRequest, contentType string, code int, err error) { + contentType, _, _ = mime.ParseMediaType(req.Header.Get("Content-Type")) + if contentType != contentTypeProto && contentType != contentTypeJSON { + // The error has to be written in some encoding; JSON is readable. + return nil, contentTypeJSON, http.StatusUnsupportedMediaType, + errors.New("Content-Type must be application/x-protobuf or application/json") + } body, err := readBody(w, req) if err != nil { - code := http.StatusBadRequest + code = http.StatusBadRequest var tooLarge *http.MaxBytesError if errors.As(err, &tooLarge) { code = http.StatusRequestEntityTooLarge } - writeStatus(w, contentType, code, codes.InvalidArgument, err.Error()) - return + return nil, contentType, code, err } - - var export coltracepb.ExportTraceServiceRequest + export = &coltracepb.ExportTraceServiceRequest{} if contentType == contentTypeJSON { - err = unmarshalJSON(body, &export) + err = UnmarshalJSON(body, export) } else { - err = proto.Unmarshal(body, &export) + err = proto.Unmarshal(body, export) } if err != nil { - writeStatus(w, contentType, http.StatusBadRequest, codes.InvalidArgument, "invalid OTLP payload: "+err.Error()) - return - } - - if err := r.accept(req.Context(), &export); err != nil { - // 503 is one of the statuses OTLP exporters retry. - w.Header().Set("Retry-After", retryAfterSeconds) - writeStatus(w, contentType, http.StatusServiceUnavailable, codes.Unavailable, err.Error()) - return + return nil, contentType, http.StatusBadRequest, fmt.Errorf("invalid OTLP payload: %w", err) } - writeMessage(w, contentType, http.StatusOK, &coltracepb.ExportTraceServiceResponse{}) + return export, contentType, http.StatusOK, nil } func readBody(w http.ResponseWriter, req *http.Request) ([]byte, error) { diff --git a/internal/otlp/steps.go b/internal/otlp/steps.go new file mode 100644 index 0000000..0146f11 --- /dev/null +++ b/internal/otlp/steps.go @@ -0,0 +1,110 @@ +package otlp + +import ( + "cmp" + "encoding/json" + "regexp" + "slices" + + tracepb "go.opentelemetry.io/proto/otlp/trace/v1" + + "github.com/tecnosam/tailwise/internal/trace" +) + +// stepToolName names a code step as a tool call. +const stepToolName = "execute_code" + +const llmOutputMessageContent = "llm.output_messages.0.message.content" + +// A fenced code block (smolagents' "Code:\n```py ... ```") or a +// block (newer smolagents). +var codeBlock = regexp.MustCompile("(?s)```[^\\n]*\\n(.*?)```|(.*?)") + +// toolCallsOf is every tool call in start order: the TOOL spans, and the code +// steps of an AGENT root (see codeSteps). +func toolCallsOf(root *tracepb.Span, spans []*tracepb.Span) []trace.ToolCall { + children := map[string][]*tracepb.Span{} + for _, s := range spans { + children[string(s.GetParentSpanId())] = append(children[string(s.GetParentSpanId())], s) + } + steps := codeSteps(root, spans, children) + ordered := append(spansOfKind(spans, kindTool), steps...) + slices.SortStableFunc(ordered, func(a, b *tracepb.Span) int { + return cmp.Compare(a.GetStartTimeUnixNano(), b.GetStartTimeUnixNano()) + }) + calls := make([]trace.ToolCall, len(ordered)) + for i, s := range ordered { + if slices.Contains(steps, s) { + calls[i] = stepCall(s, children) + } else { + calls[i] = toolCall(s) + } + } + return calls +} + +// codeSteps are the steps of a code-writing agent, such as a smolagents +// CodeAgent, which runs the code its model writes instead of calling tools: +// each step is a CHAIN span under the AGENT root, its LLM child wrote the +// code, and its output is what running it printed. No TOOL span records the +// action, so without these the model would see only the question and the +// answer. A step with a TOOL span under it is left out: its tools are already +// calls. So is one whose model wrote no code, which keeps other frameworks' +// CHAIN spans out. +func codeSteps(root *tracepb.Span, spans []*tracepb.Span, children map[string][]*tracepb.Span) []*tracepb.Span { + if attr(root, attrSpanKind) != kindAgent { + return nil + } + byID := make(map[string]*tracepb.Span, len(spans)) + for _, s := range spans { + byID[string(s.GetSpanId())] = s + } + withTools := map[string]bool{} // spans with a TOOL span under them + for _, s := range spans { + if attr(s, attrSpanKind) != kindTool { + continue + } + for p := byID[string(s.GetParentSpanId())]; p != nil && !withTools[string(p.GetSpanId())]; p = byID[string(p.GetParentSpanId())] { + withTools[string(p.GetSpanId())] = true + } + } + var steps []*tracepb.Span + for _, s := range children[string(root.GetSpanId())] { + if attr(s, attrSpanKind) == kindChain && !withTools[string(s.GetSpanId())] && stepCode(s, children) != "" { + steps = append(steps, s) + } + } + return steps +} + +// stepCode is the last code block the step's LLM span wrote, "" if none. +func stepCode(step *tracepb.Span, children map[string][]*tracepb.Span) string { + for _, c := range children[string(step.GetSpanId())] { + if attr(c, attrSpanKind) != kindLLM { + continue + } + text, ok := lookup(c, llmOutputMessageContent) + if !ok { + text = messageText(attr(c, attrOutputValue), outputKeys) + } + if blocks := codeBlock.FindAllStringSubmatch(text, -1); len(blocks) > 0 { + last := blocks[len(blocks)-1] + return cmp.Or(last[1], last[2]) + } + } + return "" +} + +// stepCall is a code step as a tool call: the code as its arguments, what it +// printed as its output, or the error that stopped it. +func stepCall(step *tracepb.Span, children map[string][]*tracepb.Span) trace.ToolCall { + args, _ := json.Marshal(map[string]string{"code": stepCode(step, children)}) + output := messageText(attr(step, attrOutputValue), outputKeys) + status := toolStatus(step) + if output == "" && status == "error" { + output = step.GetStatus().GetMessage() + } + call := trace.ToolCall{Name: stepToolName, Args: args, Output: output} + call.Status, _ = json.Marshal(status) + return call +} diff --git a/internal/otlp/steps_test.go b/internal/otlp/steps_test.go new file mode 100644 index 0000000..51ff6d2 --- /dev/null +++ b/internal/otlp/steps_test.go @@ -0,0 +1,96 @@ +package otlp + +import ( + "slices" + "strings" + "testing" + + tracepb "go.opentelemetry.io/proto/otlp/trace/v1" + + "github.com/tecnosam/tailwise/internal/trace" +) + +// codeAgentRun is the shape of a smolagents CodeAgent run: each Step is a +// CHAIN span whose LLM child wrote code, and whose output is what the code +// printed. The last step calls final_answer, a TOOL span. +func codeAgentRun() []*tracepb.Span { + failed := span(5, 1, "Step 2", 300, attrSpanKind, kindChain) + failed.Status = &tracepb.Status{Code: tracepb.Status_STATUS_CODE_ERROR, Message: "AgentExecutionError: SyntaxError"} + return []*tracepb.Span{ + span(1, 0, "CodeAgent.run", 100, attrSpanKind, kindAgent, attrInputValue, `{"task": "Fix the bug."}`, attrOutputValue, "Done."), + span(2, 1, "Step 1", 200, attrSpanKind, kindChain, attrOutputValue, "Execution logs:\nsrc/app.py"), + span(3, 2, "LiteLLMModel.__call__", 210, attrSpanKind, kindLLM, + llmOutputMessageContent, "Thought: list the files.\nCode:\n```py\nprint(ls())\n```"), + failed, + span(6, 5, "LiteLLMModel.__call__", 310, attrSpanKind, kindLLM, llmOutputMessageContent, "Code:\nprint(f'{x\\n}')"), + // A step whose model wrote no code is not an action. + span(7, 1, "Step 3", 400, attrSpanKind, kindChain, attrOutputValue, "thinking"), + span(8, 7, "LiteLLMModel.__call__", 410, attrSpanKind, kindLLM, llmOutputMessageContent, "Let me think."), + // The last step's code called a tool: the tool is the call. + span(9, 1, "Step 4", 500, attrSpanKind, kindChain, attrOutputValue, "Done."), + span(10, 9, "LiteLLMModel.__call__", 510, attrSpanKind, kindLLM, llmOutputMessageContent, "```py\nfinal_answer('Done.')\n```"), + span(11, 9, "FinalAnswerTool", 520, attrSpanKind, kindTool, attrToolName, "final_answer", + attrInputValue, `{"answer": "Done."}`, attrOutputValue, "Done."), + } +} + +func TestAssembleCodeAgentSteps(t *testing.T) { + got, err := assemble(t, codeAgentRun()) + if err != nil { + t.Fatal(err) + } + if got.Question != "Fix the bug." { + t.Errorf("question = %q", got.Question) + } + want := "TOOL CALLS:\n" + + "1. execute_code({\"code\":\"print(ls())\\n\"}) -> ok:\n Execution logs:\n src/app.py\n" + + "2. execute_code({\"code\":\"print(f'{x\\\\n}')\"}) -> error: AgentExecutionError: SyntaxError\n" + + "3. final_answer({\"answer\":\"Done.\"}) -> ok: Done.\n\n" + if state := string(trace.BuildStateGrounded(got, 12000)); !strings.Contains(state, want) { + t.Errorf("state =\n%s\nwant it to contain\n%s", state, want) + } + // The default evidence holds the steps' output too. + if len(got.Evidence) == 0 || got.Evidence[0] != "Execution logs:\nsrc/app.py" { + t.Errorf("evidence = %q", got.Evidence) + } +} + +// Under a root that is not an AGENT span, CHAIN spans are left alone. +func TestAssembleStepsOnlyUnderAnAgent(t *testing.T) { + spans := codeAgentRun() + spans[0] = span(1, 0, "pipeline", 100, attrSpanKind, kindChain, attrInputValue, "q", attrOutputValue, "a") + got, err := assemble(t, spans) + if err != nil { + t.Fatal(err) + } + if len(got.ToolCalls) != 1 || got.ToolCalls[0].Name != "final_answer" { + t.Errorf("tool calls = %+v, want only final_answer", got.ToolCalls) + } +} + +// The conversation comes from the LLM call with the longest history: its +// user and assistant messages, then its reply. +func TestAssembleConversation(t *testing.T) { + spans := []*tracepb.Span{ + span(1, 0, "run", 100, attrSpanKind, kindAgent, attrInputValue, "Cancel #W2.", attrOutputValue, "Cancelled."), + span(2, 1, "llm", 200, attrSpanKind, kindLLM, + "llm.input_messages.0.message.role", "system", "llm.input_messages.0.message.content", "Be kind.", + "llm.input_messages.1.message.role", "user", "llm.input_messages.1.message.content", "Cancel #W2."), + span(3, 1, "llm", 300, attrSpanKind, kindLLM, + "llm.input_messages.0.message.role", "system", "llm.input_messages.0.message.content", "Be kind.", + "llm.input_messages.1.message.role", "user", "llm.input_messages.1.message.content", "Cancel #W2.", + "llm.input_messages.2.message.role", "assistant", "llm.input_messages.2.message.content", "Are you sure?", + "llm.input_messages.3.message.role", "tool", "llm.input_messages.3.message.content", "{}", + "llm.input_messages.4.message.role", "user", "llm.input_messages.4.message.content", "Yes.", + outputMessageContent, "Cancelled."), + } + got, err := assemble(t, spans) + if err != nil { + t.Fatal(err) + } + want := []trace.Message{{Role: "user", Content: "Cancel #W2."}, {Role: "assistant", Content: "Are you sure?"}, + {Role: "user", Content: "Yes."}, {Role: "assistant", Content: "Cancelled."}} + if !slices.Equal(got.Conversation, want) { + t.Errorf("conversation = %+v", got.Conversation) + } +} diff --git a/internal/pipeline/actions.go b/internal/pipeline/actions.go new file mode 100644 index 0000000..51e1a1b --- /dev/null +++ b/internal/pipeline/actions.go @@ -0,0 +1,201 @@ +package pipeline + +import ( + "context" + "encoding/json" + "fmt" + "maps" + "slices" + "sync" + + "github.com/tecnosam/tailwise/internal/trace" +) + +// The action checks' answers, recorded and deciding like the manifest's +// questions. +const ( + // ActionWrong is the highest probability, over the trace's action calls, + // that a call's arguments are not what the user asked for. + ActionWrong = "action_wrong" + // UnhandledRequest is the highest probability, over the kinds of action, + // that the user asked for one and the agent neither did it nor declined: + // min(requested, 1 − handled). + UnhandledRequest = "unhandled_request" + + // The causes the action checks give a trace they keep. + CauseWrongAction = "wrong_action" + CauseUnhandledRequest = "unhandled_request" + + // MaxActionChecks bounds the action calls checked in one trace, the most + // recent kept; actionConcurrency bounds the calls in flight. + MaxActionChecks = 10 + actionConcurrency = 4 +) + +// Actions configures the action checks for a model: which tools act for the +// user, and the questions asked about them. +type Actions struct { + // Tools maps each action tool to what it does, worded as a request + // ("cancel a pending order"). + Tools map[string]string + // ActionQuestions is asked about one action call; ActionWrong is among + // them. + ActionQuestions json.RawMessage + // CompletenessQuestions asks, for each tool, RequestedQuestion(tool) and + // HandledQuestion(tool). + CompletenessQuestions json.RawMessage +} + +func RequestedQuestion(tool string) string { return "requested_" + tool } +func HandledQuestion(tool string) string { return "handled_" + tool } + +func (a *Actions) isAction(name string) bool { + _, ok := a.Tools[name] + return ok +} + +// Asker asks a model questions of the caller's own about a state of its own; +// a Scorer that implements it can run the action checks. +type Asker interface { + Ask(ctx context.Context, state trace.State, questions json.RawMessage) (Answers, error) +} + +type Answers struct { + P map[string]Probability + InputTokens int +} + +// actionFindings is what the action checks found, for the reason. +type actionFindings struct { + actionWrong Probability + action string // the call with the highest ActionWrong, as name(args) + unhandled Probability + request string // the kind of request most likely unhandled + inputTokens int + checked bool // at least one check answered +} + +// checkActions asks about each action call (the last MaxActionChecks) and +// about completeness, concurrently. A check that fails is reported and left +// out, unless on the queued path a system error should redeliver the trace. +func (e *Engine) checkActions(ctx context.Context, t trace.Trace, model Model, queued bool) (actionFindings, error) { + asker, ok := model.Scorer.(Asker) + if !ok || model.Actions == nil { + return actionFindings{}, nil + } + a := model.Actions + var calls []int + for i, c := range t.ToolCalls { + if a.isAction(c.Name) { + calls = append(calls, i) + } + } + if len(calls) > MaxActionChecks { + calls = calls[len(calls)-MaxActionChecks:] + } + + type job struct { + state trace.State + questions json.RawMessage + call int // -1 for completeness + } + jobs := []job{{trace.BuildCompletenessState(t, a.isAction), a.CompletenessQuestions, -1}} + for _, i := range calls { + jobs = append(jobs, job{trace.BuildActionState(t, i, a.isAction), a.ActionQuestions, i}) + } + + var ( + mu sync.Mutex + f actionFindings + fatal error + wg sync.WaitGroup + slots = make(chan struct{}, actionConcurrency) + ) + for _, j := range jobs { + wg.Add(1) + go func() { + defer wg.Done() + slots <- struct{}{} + defer func() { <-slots }() + var ans Answers + err := e.withRetry(ctx, t, model, queued, "ask action check", func() error { + var err error + ans, err = asker.Ask(ctx, j.state, j.questions) + return err + }) + mu.Lock() + defer mu.Unlock() + if err != nil { + if queued && isSystemError(err) { + fatal = err + } + e.reportCheckFailure(ctx, t, model, err) + return + } + f.inputTokens += ans.InputTokens + f.checked = true + if j.call >= 0 { + if p, ok := ans.P[ActionWrong]; ok && (p > f.actionWrong || f.action == "") { + f.actionWrong, f.action = p, callText(t.ToolCalls[j.call]) + } + return + } + for _, tool := range slices.Sorted(maps.Keys(a.Tools)) { + req, okR := ans.P[RequestedQuestion(tool)] + handled, okH := ans.P[HandledQuestion(tool)] + if !okR || !okH { + continue + } + if p := min(req, 1-handled); p > f.unhandled || f.request == "" { + f.unhandled, f.request = p, a.Tools[tool] + } + } + }() + } + wg.Wait() + return f, fatal +} + +// apply adds the findings to the score: as answers, as deciding answers +// (p_keep is the highest), and in the input billed. A trace with no action +// call has no wrong action: its action_wrong is 0, not missing, since a +// missing answer means the check failed. +func (f actionFindings) apply(score *Score, hasActionCalls bool) { + if score.Answers == nil { + score.Answers = map[string]Probability{} + } + if score.Deciding == nil { + score.Deciding = map[string]Probability{} + } + if !hasActionCalls { + score.Answers[ActionWrong], score.Deciding[ActionWrong] = 0, 0 + } + if !f.checked { + return + } + if hasActionCalls && f.action != "" { + score.Answers[ActionWrong], score.Deciding[ActionWrong] = f.actionWrong, f.actionWrong + score.P = max(score.P, f.actionWrong) + } + if f.request != "" { + score.Answers[UnhandledRequest], score.Deciding[UnhandledRequest] = f.unhandled, f.unhandled + score.P = max(score.P, f.unhandled) + } + score.InputTokens += f.inputTokens +} + +// reason is the plain-English reason when an action check, not the decision +// question, kept the trace. +func (f actionFindings) reason(flag string) (cause, reason string) { + switch flag { + case ActionWrong: + return CauseWrongAction, fmt.Sprintf("The agent's action %s does not match what the user asked for.", f.action) + case UnhandledRequest: + return CauseUnhandledRequest, fmt.Sprintf("The user asked the agent to %s, and it neither did it nor declined.", f.request) + } + return "", "" +} + +func callText(c trace.ToolCall) string { + return c.Name + "(" + c.ArgsText() + ")" +} diff --git a/internal/pipeline/actions_test.go b/internal/pipeline/actions_test.go new file mode 100644 index 0000000..9d8c4d8 --- /dev/null +++ b/internal/pipeline/actions_test.go @@ -0,0 +1,176 @@ +package pipeline + +import ( + "context" + "encoding/json" + "errors" + "math" + "strings" + "sync" + "testing" + + "github.com/tecnosam/tailwise/internal/fault" + "github.com/tecnosam/tailwise/internal/trace" +) + +// askScorer answers the decision question with decide, each action check +// from actionWrong (by tool name), and completeness from requested and +// handled (by tool name). err, when set, fails every action check. +type askScorer struct { + decide Probability + actionWrong map[string]Probability + requested map[string]Probability + handled map[string]Probability + err error + + mu sync.Mutex + asked int +} + +func (s *askScorer) Score(context.Context, trace.State) (Score, error) { + p := map[string]Probability{"answer_wrong": s.decide} + return Score{P: s.decide, Answers: p, Deciding: p, Cause: "correct", Explanation: "model reason", InputTokens: 100}, nil +} + +func (s *askScorer) Ready(context.Context) error { return nil } + +func (s *askScorer) Ask(_ context.Context, state trace.State, _ json.RawMessage) (Answers, error) { + s.mu.Lock() + s.asked++ + s.mu.Unlock() + if s.err != nil { + return Answers{}, s.err + } + p := map[string]Probability{} + if strings.HasPrefix(string(state), trace.HandledNote) { + for tool, r := range s.requested { + p[RequestedQuestion(tool)], p[HandledQuestion(tool)] = r, s.handled[tool] + } + return Answers{P: p, InputTokens: 10}, nil + } + action := strings.SplitN(strings.SplitN(string(state), "ACTION:\n", 2)[1], "(", 2)[0] + p[ActionWrong] = s.actionWrong[action] + return Answers{P: p, InputTokens: 10}, nil +} + +var retailActions = &Actions{Tools: map[string]string{"exchange": "exchange an item", "cancel": "cancel an order"}} + +func actionModel(s Scorer) Model { + return Model{Name: "m", Scorer: s, Actions: retailActions, + Checks: []Check{{Question: "answer_wrong"}, {Question: ActionWrong}, {Question: UnhandledRequest}}} +} + +func actionTrace() trace.Trace { + return trace.Trace{TraceID: "t", Question: "Exchange my keyboard for size L, and cancel #W2.", FinalAnswer: "Done.", + ToolCalls: []trace.ToolCall{ + {Name: "get_product", Args: json.RawMessage(`{"id":"77"}`), Output: `{"2222":"M","3333":"L"}`}, + {Name: "exchange", Args: json.RawMessage(`{"new_item_ids":["2222"]}`), Output: "ok"}, + }} +} + +// A wrong action keeps a trace the decision question passed, with a reason +// naming the action. +func TestActionCheckKeepsAWrongAction(t *testing.T) { + s := &askScorer{decide: 0.2, actionWrong: map[string]Probability{"exchange": 0.8}} + e := NewEngine(Deps{Models: SingleModel(actionModel(s)), Policy: keepAtHalf(), Exporter: nowhere{}, Logger: quiet}) + rec := e.Process(context.Background(), actionTrace()).Record + d := rec.Decision + if !d.Keep || d.PKeep != 0.8 || d.Cause != CauseWrongAction || !strings.Contains(d.Reason, `exchange({"new_item_ids":["2222"]})`) { + t.Errorf("decision = %+v", d) + } + if len(rec.Flags) != 1 || rec.Flags[0] != ActionWrong || rec.Answers[ActionWrong] != 0.8 { + t.Errorf("flags %q, answers %v", rec.Flags, rec.Answers) + } + // One action call and the completeness check, billed on top of the score. + if s.asked != 2 || rec.InputTokens != 120 { + t.Errorf("asked %d, input tokens %d", s.asked, rec.InputTokens) + } +} + +// A request neither done nor declined keeps the trace; one the agent handled +// does not. +func TestCompletenessCheckKeepsAnUnhandledRequest(t *testing.T) { + s := &askScorer{decide: 0.2, + requested: map[string]Probability{"exchange": 0.9, "cancel": 0.9}, + handled: map[string]Probability{"exchange": 0.9, "cancel": 0.1}} + e := NewEngine(Deps{Models: SingleModel(actionModel(s)), Policy: keepAtHalf(), Exporter: nowhere{}, Logger: quiet}) + d := e.Process(context.Background(), actionTrace()).Record.Decision + if !d.Keep || d.Cause != CauseUnhandledRequest || !strings.Contains(d.Reason, "cancel an order") { + t.Errorf("decision = %+v", d) + } + + s.handled["cancel"] = 0.9 + if d := e.Process(context.Background(), actionTrace()).Record.Decision; d.Keep { + t.Errorf("all requests handled: decision = %+v", d) + } +} + +// When the decision question keeps the trace, its reason stands. +func TestDecisionQuestionReasonWins(t *testing.T) { + s := &askScorer{decide: 0.9, actionWrong: map[string]Probability{"exchange": 0.8}} + e := NewEngine(Deps{Models: SingleModel(actionModel(s)), Policy: keepAtHalf(), Exporter: nowhere{}, Logger: quiet}) + if d := e.Process(context.Background(), actionTrace()).Record.Decision; d.Reason != "model reason" || d.Cause != "correct" { + t.Errorf("decision = %+v", d) + } +} + +// A failed check is left out: the synchronous path decides without it, the +// queued path redelivers on a system error. +func TestActionCheckFailure(t *testing.T) { + s := &askScorer{decide: 0.2, err: fault.Unavailable(fault.Scorer, "score trace", errors.New("down"))} + e := NewEngine(Deps{Models: SingleModel(actionModel(s)), Policy: keepAtHalf(), Exporter: nowhere{}, Logger: quiet}) + rec := e.Process(context.Background(), actionTrace()).Record + if !rec.Decision.Scored || rec.Decision.Keep || rec.Answers[ActionWrong] != 0 { + t.Errorf("sync: record = %+v", rec) + } + if _, err := e.ProcessQueued(context.Background(), actionTrace()); err == nil { + t.Error("queued: a system error in a check did not redeliver") + } +} + +// Severity ranks a trace two checks doubt above one a single check is sure +// of, though p_keep, the highest answer, says the opposite. +func TestSeverityRanksAgreementFirst(t *testing.T) { + severity := func(s *askScorer) (Probability, float64) { + e := NewEngine(Deps{Models: SingleModel(actionModel(s)), Policy: keepAtHalf(), Exporter: nowhere{}, Logger: quiet}) + d := e.Process(context.Background(), actionTrace()).Record.Decision + return d.PKeep, d.Severity + } + sureP, sure := severity(&askScorer{decide: 0.1, actionWrong: map[string]Probability{"exchange": 0.95}}) + agreeP, agree := severity(&askScorer{decide: 0.7, actionWrong: map[string]Probability{"exchange": 0.7}}) + if !(agree > sure && agreeP < sureP) { + t.Errorf("two checks at 0.7: p_keep %v, severity %v; one at 0.95: p_keep %v, severity %v", agreeP, agree, sureP, sure) + } +} + +// A failed check is left out of severity, not counted as 0; a trace with no +// action call has action_wrong 0, which does count. +func TestSeverityLeavesOutFailedChecks(t *testing.T) { + process := func(s *askScorer, tr trace.Trace) Record { + e := NewEngine(Deps{Models: SingleModel(actionModel(s)), Policy: keepAtHalf(), Exporter: nowhere{}, Logger: quiet}) + return e.Process(context.Background(), tr).Record + } + failed := &askScorer{decide: 0.6, err: fault.Unavailable(fault.Scorer, "score trace", errors.New("down"))} + if got := process(failed, actionTrace()).Decision.Severity; math.Abs(got-0.6) > 1e-9 { + t.Errorf("checks failed: severity %v, want 0.6", got) + } + noAction := actionTrace() + noAction.ToolCalls = noAction.ToolCalls[:1] + s := &askScorer{decide: 0.6, requested: map[string]Probability{"exchange": 0.9}, handled: map[string]Probability{"exchange": 0}} + rec := process(s, noAction) + if a, ok := rec.Answers[ActionWrong]; !ok || a != 0 || math.Abs(rec.Decision.Severity-0.5) > 1e-9 { + t.Errorf("no action call: action_wrong %v (present %v), severity %v, want 0 and 0.5", a, ok, rec.Decision.Severity) + } +} + +// Without an actions block, no check runs. +func TestNoActionsNoChecks(t *testing.T) { + s := &askScorer{decide: 0.2} + m := actionModel(s) + m.Actions = nil + e := NewEngine(Deps{Models: SingleModel(m), Policy: keepAtHalf(), Exporter: nowhere{}, Logger: quiet}) + e.Process(context.Background(), actionTrace()) + if s.asked != 0 { + t.Errorf("asked %d", s.asked) + } +} diff --git a/internal/pipeline/checks.go b/internal/pipeline/checks.go index 7da4388..822ddd7 100644 --- a/internal/pipeline/checks.go +++ b/internal/pipeline/checks.go @@ -6,20 +6,4 @@ type Check struct { Question string // Threshold is 0 to use the policy's keep threshold. Threshold Probability - // Weight is the question's share of severity, from its priority; see - // PriorityWeights. - Weight float64 -} - -// PriorityWeights gives n questions, ranked 1 (most important) to n, their -// rank-order centroid weights: wᵢ = (1/n) Σ_{k=i..n} 1/k. They sum to 1 and -// fall with rank, steeply at the top: for 3 questions 0.61, 0.28 and 0.11. -func PriorityWeights(n int) []float64 { - w := make([]float64, n) - sum := 0.0 - for i := n; i >= 1; i-- { - sum += 1 / float64(i) - w[i-1] = sum / float64(n) - } - return w } diff --git a/internal/pipeline/engine.go b/internal/pipeline/engine.go index 2e7447b..2cb02d6 100644 --- a/internal/pipeline/engine.go +++ b/internal/pipeline/engine.go @@ -5,6 +5,7 @@ import ( "fmt" "log/slog" "math" + "slices" "time" "github.com/tecnosam/tailwise/internal/fault" @@ -73,45 +74,112 @@ func (e *Engine) run(ctx context.Context, t trace.Trace, deferSystemErrors bool) func (e *Engine) decide(ctx context.Context, t trace.Trace, start time.Time, deferSystemErrors bool) (Record, error) { model := e.deps.Models.Route(t.Agent) - score, err := e.score(ctx, t, model, deferSystemErrors) + var verdicts []RuleResult + if model.Rules != nil { + verdicts = model.Rules.Check(t) + } + // A keep rule is certain, so the model is not asked. + if r, ok := firstKeep(verdicts); ok { + return keptByRule(t, model, r, verdicts), nil + } + state, trimmed := buildState(t, model) + score, err := e.score(ctx, t, state, model, deferSystemErrors) + var findings actionFindings + if err == nil && model.Actions != nil { + findings, err = e.checkActions(ctx, t, model, deferSystemErrors) + findings.apply(&score, hasAction(t, model.Actions)) + } e.deps.Metrics.ScoreFinished(ctx, ScoreResult{Agent: model.Agent, Model: model.Name, Duration: time.Since(start), Err: err}) + var rec Record switch { case err == nil: - return e.scored(t, model, score), nil + rec = e.scored(t, model, score, findings, verdicts) case deferSystemErrors && isSystemError(err): return Record{}, err default: - return e.fallback(ctx, t, model, err, time.Since(start)), nil + rec = e.fallback(ctx, t, model, err, time.Since(start)) + } + rec.Trimmed = trimmed + rec.Rules = fired(verdicts) + return rec, nil +} + +// keptByRule is the decision a keep rule made: kept, first in the review +// order, with the rule's reason. +func keptByRule(t trace.Trace, model Model, r RuleResult, verdicts []RuleResult) Record { + d := Decision{Keep: true, Rule: RuleKeepRule, Reason: r.Reason, Cause: CauseRule, Severity: 1} + return Record{Trace: t, Agent: model.Agent, Decision: d, Model: model.Name, Rules: fired(verdicts)} +} + +// buildState renders the trace as the model is set to see it, cut to +// trace.MaxStateChars. +func buildState(t trace.Trace, model Model) (trace.State, bool) { + if !model.SystemPrompt { + t.SystemPrompt = "" + } + build := trace.BuildState + if model.Evidence == trace.EvidenceGrounded { + build = func(t trace.Trace) trace.State { return trace.BuildStateGrounded(t, model.EvidenceBudget) } } + return trace.Fit(t, trace.MaxStateChars, build) } // On the queued path a system error is returned at once and the queue // redelivers; the synchronous path has no queue, so it retries here. -func (e *Engine) score(ctx context.Context, t trace.Trace, model Model, queued bool) (Score, error) { - state := trace.BuildState(t) +func (e *Engine) score(ctx context.Context, t trace.Trace, state trace.State, model Model, queued bool) (Score, error) { + var score Score + err := e.withRetry(ctx, t, model, queued, "score trace", func() error { + var err error + score, err = model.Scorer.Score(ctx, state) + return err + }) + return score, err +} + +// withRetry makes a model call, retrying it on the synchronous path. +func (e *Engine) withRetry(ctx context.Context, t trace.Trace, model Model, queued bool, op string, call func() error) error { for attempt := 1; ; attempt++ { - score, err := model.Scorer.Score(ctx, state) + err := call() if err == nil || queued || !fault.Retryable(err) || model.Retry.Exhausted(attempt) || ctx.Err() != nil { - return score, err + return err } delay := model.Retry.Delay(attempt) - e.deps.Reporter.Report(ctx, fault.New(fault.KindOf(err), fault.Pipeline, "score trace, will retry", err). + e.deps.Reporter.Report(ctx, fault.New(fault.KindOf(err), fault.Pipeline, fault.Op(op+", will retry"), err). WithTrace(t.TraceID).With("model", model.Name).With("attempt", attempt). With("max_attempts", model.Retry.Attempts).With("retry_in", delay.String())) if err := retry.Sleep(ctx, delay); err != nil { - return Score{}, err + return err } } } -func (e *Engine) scored(t trace.Trace, model Model, score Score) Record { - d, flags := e.deps.Policy.DecideChecks(score.P, model.Checks, score.Deciding) +func (e *Engine) reportCheckFailure(ctx context.Context, t trace.Trace, model Model, err error) { + e.deps.Reporter.Report(ctx, fault.New(fault.KindOf(err), fault.Pipeline, "action check failed, deciding without it", err). + WithTrace(t.TraceID).With("agent", model.Agent).With("model", model.Name)) +} + +func hasAction(t trace.Trace, a *Actions) bool { + for _, c := range t.ToolCalls { + if a.isAction(c.Name) { + return true + } + } + return false +} + +func (e *Engine) scored(t trace.Trace, model Model, score Score, findings actionFindings, verdicts []RuleResult) Record { + d, flags := e.deps.Policy.DecideChecks(score.P, model.Checks, score.Deciding, signals(verdicts)...) d.Cause, d.Reason = score.Cause, score.Explanation + // The decision question's reason explains it best; when only an action + // check kept the trace, say what that check found instead. + if len(flags) > 0 && len(model.Checks) > 0 && !slices.Contains(flags, model.Checks[0].Question) { + d.Cause, d.Reason = findings.reason(flags[0]) + } if d.Reason == "" { d.Reason = probabilityReason(score.P) } return Record{Trace: t, Agent: model.Agent, Decision: d, Model: score.Model, Latency: score.Latency, - Answers: score.Answers, Flags: flags} + Answers: score.Answers, Flags: flags, InputTokens: score.InputTokens} } // probabilityReason is the reason for a model that gave none of its own. @@ -163,8 +231,21 @@ func (e *Engine) logDecision(out Outcome) { if d.Scored { attrs = append(attrs, "p_keep", float64(d.PKeep)) } + if d.Ranked() { + attrs = append(attrs, "severity", d.Severity) + } + if len(rec.Rules) > 0 { + names := make([]string, len(rec.Rules)) + for i, r := range rec.Rules { + names[i] = r.Name + } + attrs = append(attrs, "rules", names) + } if rec.Flags != nil { - attrs = append(attrs, "flags", rec.Flags, "severity", d.Severity) + attrs = append(attrs, "flags", rec.Flags) + } + if rec.Trimmed { + attrs = append(attrs, "trimmed", true) } if len(rec.Answers) > 0 { attrs = append(attrs, "answers", rec.Answers) diff --git a/internal/pipeline/engine_test.go b/internal/pipeline/engine_test.go index c5c82bc..a417be2 100644 --- a/internal/pipeline/engine_test.go +++ b/internal/pipeline/engine_test.go @@ -128,6 +128,60 @@ func TestEngineScoresWithTheAgentsModel(t *testing.T) { } } +// A model sees the trace's system prompt only when it asks for it. +func TestEngineShowsTheSystemPromptOnlyWhenTheModelAsks(t *testing.T) { + var states []trace.State + seen := scorerFunc(func() (Score, error) { return Score{P: 0.2}, nil }) + capture := stateScorer{inner: seen, states: &states} + tr := trace.Trace{SystemPrompt: "Only refund delivered orders.", Question: "q", FinalAnswer: "a"} + for _, show := range []bool{false, true} { + e := NewEngine(Deps{Models: SingleModel(Model{Name: "m", Scorer: capture, SystemPrompt: show}), + Policy: keepAtHalf(), Exporter: nowhere{}, Logger: quiet}) + e.Process(context.Background(), tr) + } + if len(states) != 2 { + t.Fatalf("%d calls, want 2", len(states)) + } + if strings.Contains(string(states[0]), "SYSTEM PROMPT") { + t.Errorf("off: the model saw the system prompt:\n%s", states[0]) + } + if !strings.HasPrefix(string(states[1]), "SYSTEM PROMPT:\nOnly refund delivered orders.") { + t.Errorf("on: state = %q", states[1]) + } +} + +// A model set to the grounded evidence strategy sees tool results on their +// lines; the default shows only the status. +func TestEngineEvidenceStrategy(t *testing.T) { + var states []trace.State + capture := stateScorer{inner: scorerFunc(func() (Score, error) { return Score{P: 0.2}, nil }), states: &states} + tr := trace.Trace{Question: "q", FinalAnswer: "a", + ToolCalls: []trace.ToolCall{{Name: "get_order", Output: `{"status":"pending"}`}}} + for _, evidence := range []string{"", trace.EvidenceGrounded} { + e := NewEngine(Deps{Models: SingleModel(Model{Name: "m", Scorer: capture, Evidence: evidence, EvidenceBudget: 1000}), + Policy: keepAtHalf(), Exporter: nowhere{}, Logger: quiet}) + e.Process(context.Background(), tr) + } + if strings.Contains(string(states[0]), "pending") { + t.Errorf("default: the result was shown:\n%s", states[0]) + } + if !strings.Contains(string(states[1]), `1. get_order() -> ok: {"status":"pending"}`) { + t.Errorf("grounded: state = %q", states[1]) + } +} + +type stateScorer struct { + inner Scorer + states *[]trace.State +} + +func (s stateScorer) Score(ctx context.Context, st trace.State) (Score, error) { + *s.states = append(*s.states, st) + return s.inner.Score(ctx, st) +} + +func (s stateScorer) Ready(ctx context.Context) error { return s.inner.Ready(ctx) } + // A fallback is labelled with the model that failed, and the failure is // recorded with its error. func TestEngineFallbackNamesTheAgentsModel(t *testing.T) { @@ -273,3 +327,23 @@ type scorerFunc func() (Score, error) func (f scorerFunc) Score(context.Context, trace.State) (Score, error) { return f() } func (scorerFunc) Ready(context.Context) error { return nil } + +// A trace too long for the model is cut to fit, and the decision says so. +func TestProcessTrimsLongTraces(t *testing.T) { + for _, evidence := range []string{trace.EvidenceFirst, trace.EvidenceGrounded} { + var states []trace.State + capture := stateScorer{inner: scorerFunc(func() (Score, error) { return Score{P: 0.9}, nil }), states: &states} + e := NewEngine(Deps{Models: SingleModel(Model{Name: "m", Scorer: capture, Evidence: evidence, EvidenceBudget: 12000}), + Policy: keepAtHalf(), Exporter: nowhere{}, Logger: quiet}) + long := sampleTrace("") + long.FinalAnswer = strings.Repeat("+ patch line\n", 20000) + out := e.Process(context.Background(), long) + if !out.Record.Trimmed || !out.Record.Decision.Scored || len(states[0]) > trace.MaxStateChars { + t.Errorf("%s: trimmed = %v, scored = %v, state %d characters", evidence, out.Record.Trimmed, + out.Record.Decision.Scored, len(states[0])) + } + if out := e.Process(context.Background(), sampleTrace("")); out.Record.Trimmed { + t.Errorf("%s: a short trace was trimmed", evidence) + } + } +} diff --git a/internal/pipeline/model.go b/internal/pipeline/model.go index c1e11bc..cbd82c4 100644 --- a/internal/pipeline/model.go +++ b/internal/pipeline/model.go @@ -19,6 +19,19 @@ type Model struct { // Checks are the questions that can keep a trace, in priority order. With // none, p_keep alone decides. Checks []Check + // SystemPrompt shows the model the trace's system prompt. Off, the model + // sees the trace as if it had none. + SystemPrompt bool + // Evidence is how tool results reach the model: trace.EvidenceFirst ("" too) + // or trace.EvidenceGrounded, which shows up to EvidenceBudget characters of + // results on the tool-call lines. + Evidence string + EvidenceBudget int + // Actions turns on the action checks; nil leaves them off. The Scorer + // must be an Asker. + Actions *Actions + // Rules are the heuristic rules; nil has none. + Rules Rules // Agent is set by Route. Agent AgentLabel } diff --git a/internal/pipeline/policy.go b/internal/pipeline/policy.go index 21b34c1..6330d4d 100644 --- a/internal/pipeline/policy.go +++ b/internal/pipeline/policy.go @@ -1,6 +1,9 @@ package pipeline -import "math/rand/v2" +import ( + "math/rand/v2" + "slices" +) type PolicyConfig struct { KeepThreshold Probability @@ -23,33 +26,56 @@ func NewPolicy(cfg PolicyConfig, rnd func() float64) *Policy { } func (p *Policy) Decide(pKeep Probability) Decision { - d := Decision{Scored: true, PKeep: pKeep} + d := Decision{Scored: true, PKeep: pKeep, Severity: float64(pKeep)} d.Keep, d.Rule = p.keepIf(pKeep >= p.cfg.KeepThreshold) return d } // DecideChecks keeps a trace that any check flags. Flags are in the checks' -// order. With no checks, p_keep alone decides, as in Decide. -func (p *Policy) DecideChecks(pKeep Probability, checks []Check, answers map[string]Probability) (Decision, []string) { +// order. Severity is the average answer of the checks that answered, flagged +// or not: a check that failed is left out rather than counted as 0, so an +// outage does not sink a failed trace down the review order. Signals, the +// signal rules' 1 or 0, join the average and decide nothing. With no checks, +// p_keep alone decides, as in Decide. +func (p *Policy) DecideChecks(pKeep Probability, checks []Check, answers map[string]Probability, signals ...Probability) (Decision, []string) { if len(checks) == 0 { - return p.Decide(pKeep), nil + d := p.Decide(pKeep) + d.Severity = average(append([]Probability{pKeep}, signals...)) + return d, nil } d := Decision{Scored: true, PKeep: pKeep} flags := []string{} + counted := slices.Clone(signals) for _, c := range checks { threshold := c.Threshold if threshold == 0 { threshold = p.cfg.KeepThreshold } - if a, ok := answers[c.Question]; ok && a >= threshold { + a, ok := answers[c.Question] + if !ok { + continue + } + if a >= threshold { flags = append(flags, c.Question) - d.Severity += c.Weight * float64(a) } + counted = append(counted, a) } + d.Severity = average(counted) d.Keep, d.Rule = p.keepIf(len(flags) > 0) return d, flags } +func average(ps []Probability) float64 { + if len(ps) == 0 { + return 0 + } + var sum float64 + for _, p := range ps { + sum += float64(p) + } + return sum / float64(len(ps)) +} + func (p *Policy) keepIf(above bool) (bool, Rule) { switch { case above: diff --git a/internal/pipeline/policy_test.go b/internal/pipeline/policy_test.go index d739e93..66c813c 100644 --- a/internal/pipeline/policy_test.go +++ b/internal/pipeline/policy_test.go @@ -101,34 +101,13 @@ func TestDefaultRandomSourceRate(t *testing.T) { } } -func TestPriorityWeights(t *testing.T) { - for n, want := range map[int][]float64{ - 1: {1}, - 2: {0.75, 0.25}, - 3: {0.611, 0.278, 0.111}, - } { - got := PriorityWeights(n) - sum := 0.0 - for i := range got { - sum += got[i] - if math.Abs(got[i]-want[i]) > 1e-3 { - t.Errorf("PriorityWeights(%d) = %v, want %v", n, got, want) - break - } - } - if math.Abs(sum-1) > 1e-9 { - t.Errorf("PriorityWeights(%d) sums to %v, want 1", n, sum) - } - } -} - -// Any check reaching its own threshold keeps the trace and flags it; the -// flagged checks' weight × answer add up to its severity. +// Any check reaching its own threshold keeps the trace and flags it; its +// severity is the checks' average answer, flagged or not. func TestDecideChecks(t *testing.T) { checks := []Check{ - {Question: "answer_wrong", Weight: 0.6}, - {Question: "unsupported", Threshold: 0.8, Weight: 0.3}, - {Question: "missing_tools", Weight: 0.1}, + {Question: "answer_wrong"}, + {Question: "unsupported", Threshold: 0.8}, + {Question: "missing_tools"}, } tests := []struct { name string @@ -139,13 +118,14 @@ func TestDecideChecks(t *testing.T) { wantSeverity float64 }{ {"one low-priority check", map[string]Probability{"answer_wrong": 0.2, "unsupported": 0.1, "missing_tools": 0.9}, - nil, RuleAboveThreshold, []string{"missing_tools"}, 0.09}, + nil, RuleAboveThreshold, []string{"missing_tools"}, 0.4}, {"flags in priority order", map[string]Probability{"answer_wrong": 0.5, "unsupported": 0.9, "missing_tools": 1}, - nil, RuleAboveThreshold, []string{"answer_wrong", "unsupported", "missing_tools"}, 0.3 + 0.27 + 0.1}, + nil, RuleAboveThreshold, []string{"answer_wrong", "unsupported", "missing_tools"}, 0.8}, {"below its own threshold", map[string]Probability{"answer_wrong": 0.2, "unsupported": 0.7, "missing_tools": 0.1}, - []float64{0.9}, RuleBelowThreshold, []string{}, 0}, - {"random floor keeps with no flags", map[string]Probability{"answer_wrong": 0.2, "unsupported": 0.1, "missing_tools": 0.1}, - []float64{0.01}, RuleRandomFloor, []string{}, 0}, + []float64{0.9}, RuleBelowThreshold, []string{}, 1.0 / 3}, + // missing_tools failed: the average is over the two that answered. + {"random floor keeps with no flags", map[string]Probability{"answer_wrong": 0.2, "unsupported": 0.1}, + []float64{0.01}, RuleRandomFloor, []string{}, 0.15}, } for _, tt := range tests { rnd, _ := fixed(t, tt.draw...) @@ -159,7 +139,7 @@ func TestDecideChecks(t *testing.T) { } // With no checks, p_keep alone decides. - if d, flags := NewPolicy(testConfig, nil).DecideChecks(0.6, nil, nil); !d.Keep || d.Rule != RuleAboveThreshold || flags != nil { + if d, flags := NewPolicy(testConfig, nil).DecideChecks(0.6, nil, nil); !d.Keep || d.Rule != RuleAboveThreshold || flags != nil || d.Severity != 0.6 { t.Errorf("no checks: %+v, flags %q", d, flags) } } diff --git a/internal/pipeline/rules.go b/internal/pipeline/rules.go new file mode 100644 index 0000000..897078c --- /dev/null +++ b/internal/pipeline/rules.go @@ -0,0 +1,59 @@ +package pipeline + +import "github.com/tecnosam/tailwise/internal/trace" + +// Rules checks a trace against heuristic rules: certain, free checks that +// need no model, such as "the final answer is valid JSON". +type Rules interface { + // Check returns every rule's verdict, in the config's order. + Check(t trace.Trace) []RuleResult +} + +// RuleResult is one rule's verdict on a trace. +type RuleResult struct { + Name string + // Keep is the rule's effect when it fires: keep the trace without asking + // the model. Otherwise the rule is a signal: it counts 1 in severity when + // it fires and 0 when it does not, and does not decide keep or drop. + Keep bool + Fired bool + Reason string // what the trace broke, in plain English; empty if not fired +} + +// CauseRule is the cause of a decision a keep rule made. +const CauseRule = "rule" + +func fired(results []RuleResult) []RuleResult { + var out []RuleResult + for _, r := range results { + if r.Fired { + out = append(out, r) + } + } + return out +} + +func firstKeep(results []RuleResult) (RuleResult, bool) { + for _, r := range results { + if r.Fired && r.Keep { + return r, true + } + } + return RuleResult{}, false +} + +// signals are the signal rules' contributions to severity: 1 fired, 0 not. +func signals(results []RuleResult) []Probability { + var out []Probability + for _, r := range results { + if r.Keep { + continue + } + if r.Fired { + out = append(out, 1) + } else { + out = append(out, 0) + } + } + return out +} diff --git a/internal/pipeline/rules_test.go b/internal/pipeline/rules_test.go new file mode 100644 index 0000000..7f3bc46 --- /dev/null +++ b/internal/pipeline/rules_test.go @@ -0,0 +1,71 @@ +package pipeline + +import ( + "context" + "errors" + "math" + "testing" + + "github.com/tecnosam/tailwise/internal/fault" + "github.com/tecnosam/tailwise/internal/trace" +) + +// fixedRules returns the same verdicts for every trace. +type fixedRules []RuleResult + +func (f fixedRules) Check(trace.Trace) []RuleResult { return f } + +func ruleEngine(s Scorer, r Rules) *Engine { + m := Model{Name: "m", Scorer: s, Rules: r, Checks: []Check{{Question: "answer_wrong"}}} + return NewEngine(Deps{Models: SingleModel(m), Policy: keepAtHalf(), Exporter: nowhere{}, Logger: quiet}) +} + +// A keep rule keeps the trace without asking the model, first in the review +// order. +func TestKeepRuleSkipsTheModel(t *testing.T) { + s := &stubScorer{raw: 0.1} + rules := fixedRules{{Name: "signal", Fired: true, Reason: "s"}, {Name: "json", Keep: true, Fired: true, Reason: "The final answer is not valid JSON."}} + rec := ruleEngine(s, rules).Process(context.Background(), sampleTrace("")).Record + d := rec.Decision + if s.calls != 0 { + t.Errorf("the model was asked %d times", s.calls) + } + if !d.Keep || d.Rule != RuleKeepRule || d.Scored || d.Severity != 1 || !d.Ranked() || + d.Cause != CauseRule || d.Reason != "The final answer is not valid JSON." { + t.Errorf("decision = %+v", d) + } + if len(rec.Rules) != 2 { + t.Errorf("rules = %+v, want both fired", rec.Rules) + } +} + +// A signal rule counts 1 in severity when it fires, 0 when not, and decides +// nothing. +func TestSignalRulesJoinSeverity(t *testing.T) { + rules := fixedRules{{Name: "loop", Fired: true, Reason: "looped"}, {Name: "long"}, {Name: "json", Keep: true}} + rec := ruleEngine(&askScorer{decide: 0.2}, rules).Process(context.Background(), sampleTrace("")).Record + d := rec.Decision + // answer_wrong 0.2, loop 1, long 0; the keep rule did not fire and is not counted. + if d.Keep || d.Rule != RuleBelowThreshold || math.Abs(d.Severity-0.4) > 1e-9 { + t.Errorf("decision = %+v", d) + } + if len(rec.Rules) != 1 || rec.Rules[0].Name != "loop" { + t.Errorf("rules = %+v, want loop only", rec.Rules) + } +} + +// When the model cannot score, the fired rules are still recorded. +func TestRulesRecordedOnFallback(t *testing.T) { + s := &stubScorer{err: fault.Unavailable(fault.Scorer, "score trace", errors.New("down"))} + rec := ruleEngine(s, fixedRules{{Name: "loop", Fired: true, Reason: "looped"}}).Process(context.Background(), sampleTrace("")).Record + if rec.Decision.Rule != RuleFallback || rec.Decision.Ranked() || len(rec.Rules) != 1 { + t.Errorf("record = %+v", rec) + } +} + +func TestDecideWithSignalsAndNoChecks(t *testing.T) { + d, _ := keepAtHalf().DecideChecks(0.6, nil, nil, 1, 0) + if !d.Keep || math.Abs(d.Severity-(0.6+1)/3) > 1e-9 { + t.Errorf("decision = %+v", d) + } +} diff --git a/internal/pipeline/types.go b/internal/pipeline/types.go index d965aa7..c2ae466 100644 --- a/internal/pipeline/types.go +++ b/internal/pipeline/types.go @@ -21,6 +21,8 @@ const ( RuleRandomFloor Rule = "random_floor" RuleBelowThreshold Rule = "below_threshold" RuleFallback Rule = "fallback" + // RuleKeepRule is a heuristic keep rule firing: the model was not asked. + RuleKeepRule Rule = "keep_rule" ) type Score struct { @@ -38,6 +40,9 @@ type Score struct { // Deciding holds the answers of the questions that decide; P is their // highest. Deciding map[string]Probability + // InputTokens is what the model reported billing for the call, 0 if it + // did not say. + InputTokens int } type Decision struct { @@ -52,11 +57,18 @@ type Decision struct { Reason string // Cause is bounded by the manifest, so safe as a metric attribute. Cause string - // Severity is the sum of the flagged checks' weight × answer: higher for - // more, and higher-priority, problems. 0 when nothing was flagged. + // Severity is the deciding questions' average answer: the review order. + // It is higher the more checks are worried, so a trace several checks + // doubt ranks above one a single check is sure of. A check that failed is + // left out of the average; action_wrong on a trace without actions is 0. + // Each signal rule counts too, 1 when it fired. A keep rule makes it 1. Severity float64 } +// Ranked is whether Severity means anything: the model scored the trace, or +// a keep rule decided it. +func (d Decision) Ranked() bool { return d.Scored || d.Rule == RuleKeepRule } + type Record struct { Trace trace.Trace Agent AgentLabel @@ -68,6 +80,13 @@ type Record struct { // Flags names the checks whose answer reached their threshold, in // priority order: the problems the model found. Flags []string + // Rules are the heuristic rules that fired, in the config's order. + Rules []RuleResult + // InputTokens is the scoring call's billed input, 0 on fallback. + InputTokens int + // Trimmed means the trace was too long for the model, so the middle of its + // longest fields was cut before scoring. + Trimmed bool } type Outcome struct { diff --git a/internal/rules/rules.go b/internal/rules/rules.go new file mode 100644 index 0000000..8e9361e --- /dev/null +++ b/internal/rules/rules.go @@ -0,0 +1,451 @@ +// Package rules checks traces against heuristic rules written in the config: +// the shape of the final answer and of tool results, and limits on the run. +// They need no model, so they cost nothing and their verdict is certain. +package rules + +import ( + "bytes" + "encoding/json" + "errors" + "fmt" + "regexp" + "strings" + "unicode/utf8" + + "github.com/santhosh-tekuri/jsonschema/v6" + + "github.com/tecnosam/tailwise/internal/pipeline" + "github.com/tecnosam/tailwise/internal/trace" +) + +// Effect is what a rule does when it fires. +type Effect string + +const ( + // Keep keeps the trace without asking the model, and ranks it first. + Keep Effect = "keep" + // Signal counts 1 in severity, the review order, and decides nothing. + Signal Effect = "signal" +) + +// Config is one rule as written in the config file or a manifest. It sets +// exactly one of FinalAnswer, ToolOutput, ToolArgs and Run. +type Config struct { + Name string `yaml:"name" json:"name"` + // OnFail is keep or signal; empty is signal. + OnFail Effect `yaml:"on_fail" json:"on_fail"` + // Enabled false turns off a default rule of the same name for one agent. + Enabled *bool `yaml:"enabled" json:"enabled"` + // Reason replaces the rule's own description of what the trace broke. + Reason string `yaml:"reason" json:"reason"` + + FinalAnswer *Text `yaml:"final_answer" json:"final_answer"` + ToolOutput *ToolOutput `yaml:"tool_output" json:"tool_output"` + ToolArgs *ToolArgs `yaml:"tool_args" json:"tool_args"` + Run *Run `yaml:"run" json:"run"` +} + +// Text is what a piece of text must be. Each field set is one expectation; +// the rule fires when any is broken. +type Text struct { + NotEmpty bool `yaml:"not_empty" json:"not_empty"` + MinLength int `yaml:"min_length" json:"min_length"` // in characters + MaxLength int `yaml:"max_length" json:"max_length"` + Matches string `yaml:"matches" json:"matches"` // a regular expression it must match + NotMatches string `yaml:"not_matches" json:"not_matches"` // one it must not + JSON bool `yaml:"json" json:"json"` // must parse as JSON + JSONSchema any `yaml:"json_schema" json:"json_schema"` // must be JSON valid against it +} + +// ToolOutput checks the output of every call to Tool, or of every call when +// Tool is empty. +type ToolOutput struct { + Tool string `yaml:"tool" json:"tool"` + // OKStatus fires on a call whose status is not ok. + OKStatus bool `yaml:"ok_status" json:"ok_status"` + Text `yaml:",inline"` +} + +// ToolArgs checks the arguments of every call to Tool, or of every call when +// Tool is empty. +type ToolArgs struct { + Tool string `yaml:"tool" json:"tool"` + JSONSchema any `yaml:"json_schema" json:"json_schema"` +} + +// Run limits the run as a whole. +type Run struct { + MaxToolCalls int `yaml:"max_tool_calls" json:"max_tool_calls"` + // MaxRepeatedCalls fires when one call, same tool and same arguments, is + // made more than this many times: the agent is looping. + MaxRepeatedCalls int `yaml:"max_repeated_calls" json:"max_repeated_calls"` + RequiresTools []string `yaml:"requires_tools" json:"requires_tools"` + ForbidsTools []string `yaml:"forbids_tools" json:"forbids_tools"` +} + +// Merge is an agent's rules: every default, then its own. One of its own +// replaces the default of the same name; with enabled false, it removes it. +func Merge(defaults, own []Config) []Config { + byName := map[string]Config{} + for _, c := range own { + byName[c.Name] = c + } + var out []Config + add := func(c Config) { + if c.Enabled == nil || *c.Enabled { + out = append(out, c) + } + } + seen := map[string]bool{} + for _, c := range defaults { + seen[c.Name] = true + if o, ok := byName[c.Name]; ok { + c = o + } + add(c) + } + for _, c := range own { + if !seen[c.Name] { + add(c) + } + } + return out +} + +// Set is compiled rules, safe for concurrent use. +type Set struct { + rules []rule +} + +type rule struct { + name string + keep bool + reason string + check func(trace.Trace) string // what was broken, "" when nothing +} + +// New compiles rules, reporting every problem at once. No rules is a nil Set. +func New(cfgs []Config) (*Set, error) { + if len(cfgs) == 0 { + return nil, nil + } + var errs []error + seen := map[string]bool{} + s := &Set{} + for i, c := range cfgs { + key := fmt.Sprintf("rules[%d]", i) + if c.Name != "" { + key = fmt.Sprintf("rules[%d] (%s)", i, c.Name) + } + r, err := compile(c, seen) + if err != nil { + errs = append(errs, fmt.Errorf("%s: %w", key, err)) + continue + } + s.rules = append(s.rules, r) + } + if err := errors.Join(errs...); err != nil { + return nil, err + } + return s, nil +} + +func compile(c Config, seen map[string]bool) (rule, error) { + if c.Name == "" { + return rule{}, errors.New("name is required") + } + if seen[c.Name] { + return rule{}, errors.New("name is used by another rule") + } + seen[c.Name] = true + r := rule{name: c.Name, reason: c.Reason} + switch c.OnFail { + case Keep: + r.keep = true + case Signal, "": + default: + return rule{}, fmt.Errorf("on_fail is %q; want keep or signal", c.OnFail) + } + set := 0 + for _, isSet := range []bool{c.FinalAnswer != nil, c.ToolOutput != nil, c.ToolArgs != nil, c.Run != nil} { + if isSet { + set++ + } + } + if set != 1 { + return rule{}, errors.New("set exactly one of final_answer, tool_output, tool_args and run") + } + var err error + switch { + case c.FinalAnswer != nil: + r.check, err = finalAnswer(*c.FinalAnswer) + case c.ToolOutput != nil: + r.check, err = toolOutput(*c.ToolOutput) + case c.ToolArgs != nil: + r.check, err = toolArgs(*c.ToolArgs) + default: + r.check, err = run(*c.Run) + } + return r, err +} + +// Check returns every rule's verdict, in order. +func (s *Set) Check(t trace.Trace) []pipeline.RuleResult { + out := make([]pipeline.RuleResult, len(s.rules)) + for i, r := range s.rules { + out[i] = pipeline.RuleResult{Name: r.name, Keep: r.keep} + if broken := r.check(t); broken != "" { + out[i].Fired = true + out[i].Reason = r.reason + if out[i].Reason == "" { + out[i].Reason = broken + } + } + } + return out +} + +func (t Text) empty() bool { + return !t.NotEmpty && t.MinLength == 0 && t.MaxLength == 0 && t.Matches == "" && t.NotMatches == "" && + !t.JSON && t.JSONSchema == nil +} + +// textCheck returns what a text broke, "" when nothing. +type textCheck func(string) string + +func compileText(t Text) (textCheck, error) { + var checks []textCheck + if t.NotEmpty { + checks = append(checks, func(s string) string { + if strings.TrimSpace(s) == "" { + return "is empty" + } + return "" + }) + } + if t.MinLength < 0 || t.MaxLength < 0 { + return nil, errors.New("min_length and max_length must not be negative") + } + if t.MinLength > 0 { + checks = append(checks, func(s string) string { + if n := utf8.RuneCountInString(s); n < t.MinLength { + return fmt.Sprintf("is %d characters, under the minimum of %d", n, t.MinLength) + } + return "" + }) + } + if t.MaxLength > 0 { + checks = append(checks, func(s string) string { + if n := utf8.RuneCountInString(s); n > t.MaxLength { + return fmt.Sprintf("is %d characters, over the maximum of %d", n, t.MaxLength) + } + return "" + }) + } + for _, re := range []struct { + pattern string + must bool + }{{t.Matches, true}, {t.NotMatches, false}} { + if re.pattern == "" { + continue + } + compiled, err := regexp.Compile(re.pattern) + if err != nil { + return nil, fmt.Errorf("invalid regular expression %q: %w", re.pattern, err) + } + must, pattern := re.must, re.pattern + checks = append(checks, func(s string) string { + switch matched := compiled.MatchString(s); { + case must && !matched: + return fmt.Sprintf("does not match %q", pattern) + case !must && matched: + return fmt.Sprintf("matches %q", pattern) + } + return "" + }) + } + if t.JSON && t.JSONSchema == nil { + checks = append(checks, func(s string) string { + if !json.Valid([]byte(s)) { + return "is not valid JSON" + } + return "" + }) + } + if t.JSONSchema != nil { + schema, err := compileSchema(t.JSONSchema) + if err != nil { + return nil, err + } + checks = append(checks, func(s string) string { + v, err := jsonschema.UnmarshalJSON(strings.NewReader(s)) + if err != nil { + return "is not valid JSON" + } + return schemaBreak(schema, v) + }) + } + if len(checks) == 0 { + return nil, errors.New("set at least one of not_empty, min_length, max_length, matches, not_matches, json and json_schema") + } + return func(s string) string { + for _, c := range checks { + if broken := c(s); broken != "" { + return broken + } + } + return "" + }, nil +} + +func compileSchema(doc any) (*jsonschema.Schema, error) { + raw, err := json.Marshal(doc) + if err != nil { + return nil, fmt.Errorf("json_schema: %w", err) + } + v, err := jsonschema.UnmarshalJSON(bytes.NewReader(raw)) + if err != nil { + return nil, fmt.Errorf("json_schema: %w", err) + } + c := jsonschema.NewCompiler() + if err := c.AddResource("rule.json", v); err != nil { + return nil, fmt.Errorf("json_schema: %w", err) + } + schema, err := c.Compile("rule.json") + if err != nil { + return nil, fmt.Errorf("json_schema: %w", err) + } + return schema, nil +} + +// maxSchemaReason caps the schema errors quoted in a reason. +const maxSchemaReason = 200 + +func schemaBreak(schema *jsonschema.Schema, v any) string { + err := schema.Validate(v) + if err == nil { + return "" + } + // The first line names the schema; the rest are the problems, one a line. + lines := strings.Split(err.Error(), "\n") + var problems []string + for _, l := range lines[1:] { + if l = strings.TrimSpace(strings.TrimPrefix(strings.TrimSpace(l), "-")); l != "" { + problems = append(problems, l) + } + } + detail := strings.Join(problems, "; ") + if utf8.RuneCountInString(detail) > maxSchemaReason { + detail = string([]rune(detail)[:maxSchemaReason]) + "…" + } + if detail == "" { + return "does not match the schema" + } + return "does not match the schema: " + detail +} + +func finalAnswer(t Text) (func(trace.Trace) string, error) { + check, err := compileText(t) + if err != nil { + return nil, fmt.Errorf("final_answer: %w", err) + } + return func(tr trace.Trace) string { + if broken := check(tr.FinalAnswer); broken != "" { + return "The final answer " + broken + "." + } + return "" + }, nil +} + +func toolOutput(o ToolOutput) (func(trace.Trace) string, error) { + var check textCheck + if !o.Text.empty() { + var err error + if check, err = compileText(o.Text); err != nil { + return nil, fmt.Errorf("tool_output: %w", err) + } + } else if !o.OKStatus { + return nil, errors.New("tool_output: set ok_status or a text expectation") + } + return func(tr trace.Trace) string { + for i, c := range tr.ToolCalls { + if o.Tool != "" && c.Name != o.Tool { + continue + } + if o.OKStatus && c.StatusText() != "ok" { + return fmt.Sprintf("Call %d, to %s, failed with status %s.", i+1, c.Name, c.StatusText()) + } + if check == nil { + continue + } + if broken := check(c.Output); broken != "" { + return fmt.Sprintf("The output of call %d, to %s, %s.", i+1, c.Name, broken) + } + } + return "" + }, nil +} + +func toolArgs(a ToolArgs) (func(trace.Trace) string, error) { + if a.JSONSchema == nil { + return nil, errors.New("tool_args: json_schema is required") + } + schema, err := compileSchema(a.JSONSchema) + if err != nil { + return nil, fmt.Errorf("tool_args: %w", err) + } + return func(tr trace.Trace) string { + for i, c := range tr.ToolCalls { + if a.Tool != "" && c.Name != a.Tool { + continue + } + args := c.Args + if len(args) == 0 { + args = json.RawMessage("null") + } + v, err := jsonschema.UnmarshalJSON(bytes.NewReader(args)) + if err != nil { + return fmt.Sprintf("The arguments of call %d, to %s, are not valid JSON.", i+1, c.Name) + } + if broken := schemaBreak(schema, v); broken != "" { + return fmt.Sprintf("The arguments of call %d, to %s, %s.", i+1, c.Name, broken) + } + } + return "" + }, nil +} + +func run(r Run) (func(trace.Trace) string, error) { + if r.MaxToolCalls < 0 || r.MaxRepeatedCalls < 0 { + return nil, errors.New("run: max_tool_calls and max_repeated_calls must not be negative") + } + if r.MaxToolCalls == 0 && r.MaxRepeatedCalls == 0 && len(r.RequiresTools) == 0 && len(r.ForbidsTools) == 0 { + return nil, errors.New("run: set at least one of max_tool_calls, max_repeated_calls, requires_tools and forbids_tools") + } + return func(tr trace.Trace) string { + if r.MaxToolCalls > 0 && len(tr.ToolCalls) > r.MaxToolCalls { + return fmt.Sprintf("The agent made %d tool calls, over the limit of %d.", len(tr.ToolCalls), r.MaxToolCalls) + } + called := map[string]bool{} + repeats := map[string]int{} + for _, c := range tr.ToolCalls { + called[c.Name] = true + key := c.Name + "(" + c.ArgsText() + ")" + repeats[key]++ + if r.MaxRepeatedCalls > 0 && repeats[key] == r.MaxRepeatedCalls+1 { + return fmt.Sprintf("The agent called %s more than %d times with the same arguments.", c.Name, r.MaxRepeatedCalls) + } + } + for _, tool := range r.RequiresTools { + if !called[tool] { + return fmt.Sprintf("The agent never called %s.", tool) + } + } + for _, tool := range r.ForbidsTools { + if called[tool] { + return fmt.Sprintf("The agent called %s, which it must not.", tool) + } + } + return "" + }, nil +} diff --git a/internal/rules/rules_test.go b/internal/rules/rules_test.go new file mode 100644 index 0000000..8769f29 --- /dev/null +++ b/internal/rules/rules_test.go @@ -0,0 +1,121 @@ +package rules + +import ( + "encoding/json" + "strings" + "testing" + + "github.com/tecnosam/tailwise/internal/trace" +) + +func run1(t *testing.T, c Config, tr trace.Trace) (bool, string) { + t.Helper() + s, err := New([]Config{c}) + if err != nil { + t.Fatalf("New: %v", err) + } + r := s.Check(tr)[0] + return r.Fired, r.Reason +} + +var order = trace.Trace{ + Question: "Where is order #7?", + FinalAnswer: `{"order": 7, "status": "shipped"}`, + ToolCalls: []trace.ToolCall{ + {Name: "get_order", Args: json.RawMessage(`{"id": 7}`), Output: `{"id": 7, "status": "shipped"}`}, + {Name: "search", Args: json.RawMessage(`{"q": "x"}`), Status: json.RawMessage(`"error"`), Output: "Error: timeout"}, + {Name: "search", Args: json.RawMessage(`{"q": "x"}`)}, + {Name: "search", Args: json.RawMessage(`{"q": "x"}`)}, + }, +} + +func TestRules(t *testing.T) { + answerSchema := map[string]any{"type": "object", "required": []any{"order", "status"}, + "properties": map[string]any{"status": map[string]any{"enum": []any{"shipped", "pending"}}}} + tests := []struct { + name string + rule Config + fires bool + reason string // a part of it + }{ + {"answer is JSON", Config{FinalAnswer: &Text{JSON: true}}, false, ""}, + {"answer matches its schema", Config{FinalAnswer: &Text{JSONSchema: answerSchema}}, false, ""}, + {"answer too short", Config{FinalAnswer: &Text{MinLength: 100}}, true, "under the minimum of 100"}, + {"answer too long", Config{FinalAnswer: &Text{MaxLength: 10}}, true, "over the maximum of 10"}, + {"answer must match", Config{FinalAnswer: &Text{Matches: `#\d+`}}, true, `does not match "#\\d+"`}, + {"answer must not match", Config{FinalAnswer: &Text{NotMatches: `shipped`}}, true, `matches "shipped"`}, + {"tool failed", Config{ToolOutput: &ToolOutput{OKStatus: true}}, true, "Call 2, to search, failed with status error"}, + {"other tool's status ignored", Config{ToolOutput: &ToolOutput{Tool: "get_order", OKStatus: true}}, false, ""}, + {"tool output schema", Config{ToolOutput: &ToolOutput{Tool: "get_order", + Text: Text{JSONSchema: map[string]any{"required": []any{"id", "eta"}}}}}, true, "The output of call 1, to get_order, does not match the schema"}, + {"tool args schema", Config{ToolArgs: &ToolArgs{Tool: "get_order", + JSONSchema: map[string]any{"properties": map[string]any{"id": map[string]any{"type": "string"}}}}}, true, "The arguments of call 1, to get_order"}, + {"loop", Config{Run: &Run{MaxRepeatedCalls: 2}}, true, "called search more than 2 times"}, + {"no loop at the limit", Config{Run: &Run{MaxRepeatedCalls: 3}}, false, ""}, + {"too many calls", Config{Run: &Run{MaxToolCalls: 3}}, true, "4 tool calls, over the limit of 3"}, + {"required tool", Config{Run: &Run{RequiresTools: []string{"refund"}}}, true, "never called refund"}, + {"forbidden tool", Config{Run: &Run{ForbidsTools: []string{"search"}}}, true, "called search, which it must not"}, + {"own reason", Config{Reason: "No refund.", Run: &Run{RequiresTools: []string{"refund"}}}, true, "No refund."}, + } + for _, tt := range tests { + tt.rule.Name = "r" + fired, reason := run1(t, tt.rule, order) + if fired != tt.fires || !strings.Contains(reason, tt.reason) { + t.Errorf("%s: fired %v, reason %q; want %v, %q", tt.name, fired, reason, tt.fires, tt.reason) + } + } +} + +func TestEmptyAndInvalidAnswer(t *testing.T) { + empty := trace.Trace{Question: "q", FinalAnswer: " "} + if fired, reason := run1(t, Config{Name: "r", FinalAnswer: &Text{NotEmpty: true}}, empty); !fired || reason != "The final answer is empty." { + t.Errorf("empty: %v %q", fired, reason) + } + prose := trace.Trace{Question: "q", FinalAnswer: "it shipped"} + if fired, reason := run1(t, Config{Name: "r", FinalAnswer: &Text{JSONSchema: map[string]any{"type": "object"}}}, prose); !fired || !strings.Contains(reason, "not valid JSON") { + t.Errorf("schema on prose: %v %q", fired, reason) + } +} + +func TestNewReportsEveryProblem(t *testing.T) { + _, err := New([]Config{ + {FinalAnswer: &Text{JSON: true}}, + {Name: "a", OnFail: "maybe", FinalAnswer: &Text{JSON: true}}, + {Name: "b"}, + {Name: "c", FinalAnswer: &Text{}}, + {Name: "d", FinalAnswer: &Text{Matches: "("}}, + {Name: "e", ToolArgs: &ToolArgs{}}, + {Name: "f", FinalAnswer: &Text{JSONSchema: map[string]any{"type": 5}}}, + {Name: "a", Run: &Run{MaxToolCalls: 1}}, + }) + if err == nil { + t.Fatal("no error") + } + for _, want := range []string{"rules[0]: name is required", "on_fail is \"maybe\"", "(b): set exactly one", + "(c): final_answer: set at least one", "(d): final_answer: invalid regular expression", "(e): tool_args: json_schema is required", + "(f): final_answer: json_schema", "rules[7] (a): name is used"} { + if !strings.Contains(err.Error(), want) { + t.Errorf("error lacks %q:\n%v", want, err) + } + } +} + +func TestMerge(t *testing.T) { + off := false + defaults := []Config{{Name: "json", OnFail: Keep}, {Name: "loop"}, {Name: "length"}} + own := []Config{{Name: "loop", OnFail: Keep}, {Name: "length", Enabled: &off}, {Name: "schema"}} + got := Merge(defaults, own) + var names []string + for _, c := range got { + names = append(names, c.Name+":"+string(c.OnFail)) + } + if strings.Join(names, ",") != "json:keep,loop:keep,schema:" { + t.Errorf("merged %v", names) + } +} + +func TestNoRulesIsNil(t *testing.T) { + if s, err := New(nil); s != nil || err != nil { + t.Errorf("New(nil) = %v, %v", s, err) + } +} diff --git a/internal/scorer/factory.go b/internal/scorer/factory.go index d635552..59e13e7 100644 --- a/internal/scorer/factory.go +++ b/internal/scorer/factory.go @@ -14,6 +14,16 @@ type ModelConfig struct { Retry retry.Policy // Checks are the questions that can keep a trace, in priority order. Checks []pipeline.Check + // SystemPrompt shows the model the agent's system prompt. + SystemPrompt bool + // Evidence is how tool results reach the model (trace.EvidenceFirst or + // trace.EvidenceGrounded); EvidenceBudget bounds the grounded results. + Evidence string + EvidenceBudget int + // Actions turns on the action checks; nil leaves them off. + Actions *pipeline.Actions + // Rules are the heuristic rules; nil has none. + Rules pipeline.Rules } type Models struct { @@ -40,5 +50,7 @@ func buildModel(mc ModelConfig) (pipeline.Model, error) { if err != nil { return pipeline.Model{}, err } - return pipeline.Model{Name: pipeline.ModelName(mc.Client.Model), Scorer: s, Retry: mc.Retry, Checks: mc.Checks}, nil + return pipeline.Model{Name: pipeline.ModelName(mc.Client.Model), Scorer: s, Retry: mc.Retry, Checks: mc.Checks, + SystemPrompt: mc.SystemPrompt, Evidence: mc.Evidence, EvidenceBudget: mc.EvidenceBudget, Actions: mc.Actions, + Rules: mc.Rules}, nil } diff --git a/internal/scorer/scorer.go b/internal/scorer/scorer.go index a42971a..d33b159 100644 --- a/internal/scorer/scorer.go +++ b/internal/scorer/scorer.go @@ -86,6 +86,9 @@ type response struct { Answers map[string]json.RawMessage `json:"answers"` LatencyMS *float64 `json:"latency_ms"` EvaluationTimeMS *float64 `json:"evaluation_time_ms"` + Usage struct { + InputTokens int `json:"input_tokens"` + } `json:"usage"` } type noulAnswer struct { @@ -109,6 +112,30 @@ func (c *Client) Score(ctx context.Context, state trace.State) (pipeline.Score, return score, nil } +// Ask makes one call with questions of the caller's own, for the action +// checks: every well-formed yes/no answer, and the input billed. +func (c *Client) Ask(ctx context.Context, state trace.State, questions json.RawMessage) (pipeline.Answers, error) { + body, err := json.Marshal(request{Model: c.cfg.Model, State: string(state), Questions: questions}) + if err != nil { + return pipeline.Answers{}, fault.Internal(fault.Scorer, "encode request", err) + } + ctx, cancel := context.WithTimeout(ctx, c.cfg.Timeout) + defer cancel() + req, err := c.newRequest(ctx, body) + if err != nil { + return pipeline.Answers{}, classify(err) + } + raw, err := c.send(req) + if err != nil { + return pipeline.Answers{}, classify(err) + } + var out response + if err := json.Unmarshal(raw, &out); err != nil { + return pipeline.Answers{}, classify(fmt.Errorf("decode model response: %w", err)) + } + return pipeline.Answers{P: noulAnswers(out), InputTokens: out.Usage.InputTokens}, nil +} + func (c *Client) attempt(ctx context.Context, body []byte) (pipeline.Score, error) { ctx, cancel := context.WithTimeout(ctx, c.cfg.Timeout) defer cancel() @@ -171,7 +198,8 @@ func (c *Client) parse(raw []byte, elapsed time.Duration) (pipeline.Score, error deciding[q] = a p = max(p, a) } - score := pipeline.Score{P: p, Answers: answers, Deciding: deciding, Model: c.modelName(out), Latency: reportedLatency(out, elapsed)} + score := pipeline.Score{P: p, Answers: answers, Deciding: deciding, Model: c.modelName(out), Latency: reportedLatency(out, elapsed), + InputTokens: out.Usage.InputTokens} score.Cause, score.Explanation = c.reasonAnswer(out) return score, nil } diff --git a/internal/scorer/scorer_test.go b/internal/scorer/scorer_test.go index 15086c5..bc0cb4a 100644 --- a/internal/scorer/scorer_test.go +++ b/internal/scorer/scorer_test.go @@ -401,8 +401,53 @@ func TestScoreKeepsTheAnswersAndTakesTheHighest(t *testing.T) { t.Errorf("with any_of: %+v, %v; want p_keep 0.8 from two deciding answers", res, err) } + if res.InputTokens != 0 { + t.Errorf("input tokens = %d with no usage reported, want 0", res.InputTokens) + } + missing := newClient(t, srv.URL, func(cfg *Config) { cfg.AnyOf = []string{"missing_tool_calls"} }) if _, err = missing.Score(context.Background(), "s"); err == nil || !strings.Contains(err.Error(), `no answer for "missing_tool_calls"`) { t.Errorf("a deciding question without an answer: error = %v", err) } } + +func TestScoreReadsTheInputTokens(t *testing.T) { + srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + w.Write([]byte(`{"answers": {"answer_wrong": {"type": "noul", "noul": 0.2}}, "usage": {"input_tokens": 410, "output_tokens": 3}}`)) + })) + defer srv.Close() + + res, err := newClient(t, srv.URL, nil).Score(context.Background(), "s") + if err != nil || res.InputTokens != 410 { + t.Fatalf("input tokens = %d, %v; want 410", res.InputTokens, err) + } +} + +// Ask sends the caller's state and questions and returns the yes/no answers. +func TestAskSendsItsOwnQuestions(t *testing.T) { + var got request + srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + _ = json.NewDecoder(r.Body).Decode(&got) + io.WriteString(w, `{"model":"m","answers":{"action_wrong":{"type":"noul","noul":0.7},`+ + `"odd":{"type":"choice","choice":"x"}},"usage":{"input_tokens":42}}`) + })) + defer srv.Close() + + c := newClient(t, srv.URL, nil) + ask := json.RawMessage(`{"action_wrong":{"type":"noul","instructions":"wrong?"}}`) + ans, err := c.Ask(context.Background(), "ACTION:\ncancel()", ask) + if err != nil { + t.Fatal(err) + } + if got.State != "ACTION:\ncancel()" || string(got.Questions) != string(ask) { + t.Errorf("request = %+v", got) + } + if len(ans.P) != 1 || ans.P["action_wrong"] != 0.7 || ans.InputTokens != 42 { + t.Errorf("answers = %+v", ans) + } + + srv.Config.Handler = http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { w.WriteHeader(http.StatusBadRequest) }) + if _, err := c.Ask(context.Background(), "s", ask); fault.KindOf(err) != fault.KindRejected { + t.Errorf("400: error = %v, want rejected", err) + } +} diff --git a/internal/server/docs_test.go b/internal/server/docs_test.go index f220713..ca9e068 100644 --- a/internal/server/docs_test.go +++ b/internal/server/docs_test.go @@ -7,6 +7,7 @@ import ( "testing" "github.com/tecnosam/tailwise/internal/pipeline" + "github.com/tecnosam/tailwise/internal/rules" "github.com/tecnosam/tailwise/internal/trace" "gopkg.in/yaml.v3" ) @@ -64,6 +65,7 @@ func TestOpenAPICoversRoutes(t *testing.T) { }{ {"post", "/v1/traces", []string{"202", "400", "413", "503"}}, {"post", "/v1/traces:score", []string{"200", "400", "413"}}, + {"post", "/v1/otlp/traces:score", []string{"200", "400", "413", "415", "422"}}, {"get", "/healthz", []string{"200"}}, {"get", "/readyz", []string{"200", "503"}}, } @@ -88,13 +90,24 @@ func TestOpenAPICoversRoutes(t *testing.T) { } // The Decision schema lists exactly the fields the sync path returns, the -// optional cause included. +// optional ones included. func TestOpenAPIDecisionMatchesResult(t *testing.T) { - withCause := &fakeScorer{score: func(context.Context, trace.State) (pipeline.Score, error) { - return pipeline.Score{P: 0.9, Model: "m", Cause: "incomplete", Explanation: "The final answer leaves out part of what the question asked."}, nil + everything := &fakeScorer{score: func(context.Context, trace.State) (pipeline.Score, error) { + answers := map[string]pipeline.Probability{"answer_wrong": 0.9} + return pipeline.Score{P: 0.9, Model: "m", Cause: "incomplete", Explanation: "The final answer leaves out part of what the question asked.", + Answers: answers, Deciding: answers, InputTokens: 410}, nil }} - h := newHarness(t, harnessOpts{scorer: withCause}) - w := h.do("POST", "/v1/traces:score", `{"question": "q", "final_answer": "a"}`) + // A signal rule the long answer breaks, so the response lists it. + tooLong, err := rules.New([]rules.Config{{Name: "short", FinalAnswer: &rules.Text{MaxLength: 100}}}) + if err != nil { + t.Fatal(err) + } + models := pipeline.SingleModel(pipeline.Model{Name: "m", Scorer: everything, Rules: tooLong, + Checks: []pipeline.Check{{Question: "answer_wrong"}}}) + h := newHarness(t, harnessOpts{scorer: everything, models: models}) + // An answer too long for the model, so the response is marked trimmed. + long := strings.Repeat("a", trace.MaxStateChars) + w := h.do("POST", "/v1/traces:score", `{"question": "q", "final_answer": "`+long+`"}`) got := decodeBody[map[string]any](t, w) schema := parseSpec(t, openAPISpec).Components.Schemas["Decision"] diff --git a/internal/server/openapi.yaml b/internal/server/openapi.yaml index 3dd2632..71881db 100644 --- a/internal/server/openapi.yaml +++ b/internal/server/openapi.yaml @@ -106,6 +106,7 @@ paths: trace_id: wrong-river-001 decision: keep p_keep: 0.98 + severity: 0.98 rule: above_threshold reason: The final answer contradicts the evidence the agent gathered. cause: contradicts_evidence @@ -118,6 +119,7 @@ paths: trace_id: 7f3a91c2e8 decision: drop p_keep: 0.04 + severity: 0.04 rule: below_threshold reason: The final answer is correct, complete and supported by the evidence. cause: correct @@ -130,6 +132,7 @@ paths: trace_id: wrong-river-001 decision: drop p_keep: null + severity: null rule: fallback reason: The model could not score this trace, so it was dropped at random. model: jev-latest @@ -139,6 +142,66 @@ paths: $ref: "#/components/responses/BadRequest" "413": $ref: "#/components/responses/TooLarge" + /v1/otlp/traces:score: + post: + tags: [traces] + operationId: scoreOTLPTrace + summary: Score a trace sent as OTLP spans, synchronously + description: | + Takes one trace as an OTLP/HTTP export request (protobuf or JSON, + optionally gzipped), builds the trace from its spans exactly as the + OTLP receivers do, scores it and returns the decision. There is no + queue and no wait for late spans, so send every span of the trace in + one request. Meant for testing and benchmarks. + + Fallback behaves as for `POST /v1/traces:score`. + requestBody: + required: true + content: + application/x-protobuf: + schema: + type: string + format: binary + description: An `ExportTraceServiceRequest` holding the spans of one trace. + application/json: + schema: + type: object + description: An `ExportTraceServiceRequest` in OTLP/JSON, IDs as hex. + responses: + "200": + description: The decision for the trace. + content: + application/json: + schema: + $ref: "#/components/schemas/Decision" + "400": + description: The body is not an OTLP export request, or holds spans of no trace or of more than one. + content: + application/json: + schema: + $ref: "#/components/schemas/Error" + example: + error: the request must hold the spans of exactly one trace + "413": + description: The request body is larger than 16 MiB. + content: + application/json: + schema: + $ref: "#/components/schemas/Error" + "415": + description: Content-Type is not application/x-protobuf or application/json. + content: + application/json: + schema: + $ref: "#/components/schemas/Error" + "422": + description: No root span has a question (`input.value`) and an answer (`output.value`); an `AGENT` root without an answer is scored with an empty one. + content: + application/json: + schema: + $ref: "#/components/schemas/Error" + example: + error: 'trace is not scorable: root span "GET /users" has no input.value' /healthz: get: tags: [health] @@ -259,6 +322,12 @@ components: the trace (agents.models in the config); an agent with no model of its own, or no agent, gets the default model. example: support-bot + system_prompt: + type: string + description: >- + The instructions the agent ran under (its system prompt), so the + model can judge the run against them. Optional; shown to the model + cut to 8000 characters. question: type: string minLength: 1 @@ -268,6 +337,21 @@ components: description: Tool calls the agent made, in order. items: $ref: "#/components/schemas/ToolCall" + conversation: + type: array + description: >- + The user's and agent's turns, in order. Optional; the action + checks read it to see what the agent said, such as a request it + declined. + items: + type: object + required: [role, content] + properties: + role: + type: string + enum: [user, assistant] + content: + type: string evidence: type: array description: | @@ -294,6 +378,11 @@ components: description: Outcome of the call. default: ok example: ok + output: + type: string + description: >- + What the call returned, as text. Optional. Shown to models set to + scorer.evidence grounded, on the call's line; ignored otherwise. Accepted: type: object required: [trace_id] @@ -304,7 +393,7 @@ components: example: c68863b8105c8e95f34eba6666e18d91 Decision: type: object - required: [trace_id, agent, decision, p_keep, rule, reason, model, latency_ms, exported] + required: [trace_id, agent, decision, p_keep, severity, rule, reason, model, latency_ms, exported] properties: trace_id: type: string @@ -320,16 +409,20 @@ components: nullable: true minimum: 0 maximum: 1 - description: The model's probability that the final answer is wrong. Null on fallback. + description: >- + The model's probability that the final answer is wrong. Null on + fallback, and when a keep rule decided, since the model was not + asked. rule: type: string - enum: [above_threshold, random_floor, below_threshold, fallback] + enum: [above_threshold, random_floor, below_threshold, fallback, keep_rule] description: | Which part of the policy decided: - `above_threshold`: kept because `p_keep` reached the keep threshold - `random_floor`: below the threshold, kept as a random sample - `below_threshold`: below the threshold and not sampled; dropped - `fallback`: the model could not score it; kept or dropped at random + - `keep_rule`: a heuristic rule with `on_fail: keep` fired; kept without asking the model reason: type: string description: >- @@ -357,6 +450,63 @@ components: exported: type: boolean description: The trace was kept and queued for export to at least one back end, which receives it moments later. A failed send is retried and dead-lettered after max_deliver tries. + answers: + type: object + additionalProperties: + type: number + format: double + description: Every yes/no answer the model gave, by question. Omitted on fallback. + example: + answer_wrong: 0.93 + unsupported_answer: 0.71 + flags: + type: array + items: + type: string + description: The deciding questions whose answer reached their threshold, in priority order. Omitted when there are none. + example: [answer_wrong, unsupported_answer] + severity: + type: number + format: double + nullable: true + minimum: 0 + maximum: 1 + description: >- + The deciding questions' average answer: the order to review traces in, highest first. + A trace several checks doubt ranks above one a single check is sure of. Each signal + rule counts in the average too, 1 when it fired and 0 when not. 1 when a keep rule + decided. Null on fallback. + rules: + type: array + description: >- + The heuristic rules that fired, in the config's order. Omitted + when none did. + items: + type: object + required: [name, on_fail, reason] + properties: + name: + type: string + on_fail: + type: string + enum: [keep, signal] + description: "`keep`: the rule kept the trace. `signal`: it raised the trace's severity only." + reason: + type: string + description: What the trace broke, in plain English. + example: + - name: answer_json + on_fail: keep + reason: The final answer is not valid JSON. + input_tokens: + type: integer + description: The input tokens the model reported billing for the call. Omitted if it reported none, and on fallback. + example: 410 + trimmed: + type: boolean + description: >- + True when the trace was too long for the model, so the middle of + its longest fields was cut before scoring. Omitted otherwise. Status: type: object required: [status] diff --git a/internal/server/otlp_e2e_test.go b/internal/server/otlp_e2e_test.go index 4b64807..d654ac7 100644 --- a/internal/server/otlp_e2e_test.go +++ b/internal/server/otlp_e2e_test.go @@ -7,6 +7,7 @@ import ( "encoding/json" "net" "net/http" + "net/http/httptest" "strings" "sync" "testing" @@ -346,3 +347,67 @@ func TestJSONAPIIgnoresOTLPField(t *testing.T) { t.Errorf("published trace carries client-supplied spans: %v", got) } } + +// otlpScore posts spans to the synchronous OTLP endpoint as OTLP/protobuf. +func otlpScore(t *testing.T, h *harness, spans ...*tracepb.Span) *httptest.ResponseRecorder { + t.Helper() + body, err := proto.Marshal(&coltracepb.ExportTraceServiceRequest{ResourceSpans: []*tracepb.ResourceSpans{{ + ScopeSpans: []*tracepb.ScopeSpans{{Spans: spans}}, + }}}) + if err != nil { + t.Fatal(err) + } + req := httptest.NewRequest("POST", "/v1/otlp/traces:score", bytes.NewReader(body)) + req.Header.Set("Content-Type", "application/x-protobuf") + w := httptest.NewRecorder() + h.srv.Handler().ServeHTTP(w, req) + return w +} + +func TestScoreOTLPBuildsTheSameSlice(t *testing.T) { + kev := newFakeKev(t, func(w http.ResponseWriter, _ string) { + w.Write(readFile(t, "kev", "success.json")) // noul 0.93, 410 input tokens + }) + h := newHarness(t, harnessOpts{scorer: kev.client(t)}) + + traceID := bytes.Repeat([]byte{0xab}, 16) + w := otlpScore(t, h, fixtureSpans(t, "wrong_river.json", traceID)...) + if w.Code != http.StatusOK { + t.Fatalf("status = %d, want 200; body %s", w.Code, w.Body) + } + got := decodeBody[Result](t, w) + if got.TraceID != hex.EncodeToString(traceID) || got.Decision != "keep" || got.InputTokens != 410 { + t.Errorf("result = %+v", got) + } + if got.Answers["answer_wrong"] != 0.93 { + t.Errorf("answers = %v, want answer_wrong 0.93", got.Answers) + } + wantState := string(readFile(t, "golden", "wrong_river.txt")) + if len(kev.states) != 1 || kev.states[0] != wantState { + t.Errorf("state sent to Kev:\n%q\nwant:\n%q", kev.states, wantState) + } +} + +func TestScoreOTLPRejects(t *testing.T) { + kev := newFakeKev(t, func(w http.ResponseWriter, _ string) { w.Write(readFile(t, "kev", "success.json")) }) + h := newHarness(t, harnessOpts{scorer: kev.client(t)}) + one, two := bytes.Repeat([]byte{1}, 16), bytes.Repeat([]byte{2}, 16) + + twoTraces := append(fixtureSpans(t, "wrong_river.json", one), fixtureSpans(t, "correct_river.json", two)...) + if w := otlpScore(t, h, twoTraces...); w.Code != http.StatusBadRequest { + t.Errorf("two traces: status = %d, want 400; body %s", w.Code, w.Body) + } + + noAnswer := fixtureSpans(t, "wrong_river.json", one)[1:] // tool spans without their root + if w := otlpScore(t, h, noAnswer...); w.Code != http.StatusUnprocessableEntity { + t.Errorf("no root with an answer: status = %d, want 422; body %s", w.Code, w.Body) + } + + w := h.do("POST", "/v1/otlp/traces:score", "{}") // no Content-Type + if w.Code != http.StatusUnsupportedMediaType { + t.Errorf("no Content-Type: status = %d, want 415", w.Code) + } + if kev.calls.Load() != 0 { + t.Errorf("Kev was called %d times for rejected requests", kev.calls.Load()) + } +} diff --git a/internal/server/result.go b/internal/server/result.go index 4dfef18..b464716 100644 --- a/internal/server/result.go +++ b/internal/server/result.go @@ -2,7 +2,9 @@ package server import "github.com/tecnosam/tailwise/internal/pipeline" -// Result is the POST /v1/traces:score response. p_keep is null on fallback. +// Result is the POST /v1/traces:score response. p_keep is null when the model +// gave no answer (fallback, or a keep rule decided); severity is null on +// fallback. type Result struct { TraceID string `json:"trace_id"` Agent string `json:"agent"` @@ -15,24 +17,66 @@ type Result struct { LatencyMS float64 `json:"latency_ms"` // Exported means queued for export to at least one back end. Exported bool `json:"exported"` + // Severity, Answers and Flags are as in the decision log; Answers and + // Flags are absent on fallback. + Severity *float64 `json:"severity"` + Answers map[string]float64 `json:"answers,omitempty"` + Flags []string `json:"flags,omitempty"` + // Rules are the heuristic rules that fired, absent when none did. + Rules []RuleHit `json:"rules,omitempty"` + // InputTokens is the model's billed input for this trace, absent if it + // did not report it. + InputTokens int `json:"input_tokens,omitempty"` + // Trimmed means the model saw the trace with the middle of its longest + // fields cut, to fit its input limit. + Trimmed bool `json:"trimmed,omitempty"` +} + +// RuleHit is a heuristic rule that fired. +type RuleHit struct { + Name string `json:"name"` + // OnFail is keep or signal: whether the rule kept the trace or only + // raised its severity. + OnFail string `json:"on_fail"` + Reason string `json:"reason"` } func resultOf(out pipeline.Outcome) Result { rec, d := out.Record, out.Record.Decision res := Result{ - TraceID: rec.Trace.TraceID, - Agent: string(rec.Trace.Agent), - Decision: d.Label(), - Rule: string(d.Rule), - Reason: d.Reason, - Cause: d.Cause, - Model: string(rec.Model), - LatencyMS: pipeline.Milliseconds(rec.Latency), - Exported: out.Exported, + TraceID: rec.Trace.TraceID, + Agent: string(rec.Trace.Agent), + Decision: d.Label(), + Rule: string(d.Rule), + Reason: d.Reason, + Cause: d.Cause, + Model: string(rec.Model), + LatencyMS: pipeline.Milliseconds(rec.Latency), + Exported: out.Exported, + Flags: rec.Flags, + InputTokens: rec.InputTokens, + Trimmed: rec.Trimmed, + } + if len(rec.Answers) > 0 { + res.Answers = make(map[string]float64, len(rec.Answers)) + for q, p := range rec.Answers { + res.Answers[q] = float64(p) + } + } + for _, r := range rec.Rules { + effect := "signal" + if r.Keep { + effect = "keep" + } + res.Rules = append(res.Rules, RuleHit{Name: r.Name, OnFail: effect, Reason: r.Reason}) } if d.Scored { pKeep := float64(d.PKeep) res.PKeep = &pKeep } + if d.Ranked() { + severity := d.Severity + res.Severity = &severity + } return res } diff --git a/internal/server/rules_test.go b/internal/server/rules_test.go new file mode 100644 index 0000000..a480277 --- /dev/null +++ b/internal/server/rules_test.go @@ -0,0 +1,51 @@ +package server + +import ( + "net/http" + "testing" + "time" + + "github.com/tecnosam/tailwise/internal/pipeline" + "github.com/tecnosam/tailwise/internal/retry" + "github.com/tecnosam/tailwise/internal/rules" +) + +// A keep rule decides without Kev: the response names the rule, has no +// p_keep, ranks the trace first, and the export carries the rule. +func TestScoreKeepRule(t *testing.T) { + kev := newFakeKev(t, func(w http.ResponseWriter, _ string) { + w.Write(readFile(t, "kev", "success_low.json")) + }) + set, err := rules.New([]rules.Config{ + {Name: "answer_json", OnFail: rules.Keep, FinalAnswer: &rules.Text{JSON: true}}, + {Name: "short", FinalAnswer: &rules.Text{MinLength: 50}}, + }) + if err != nil { + t.Fatal(err) + } + models := pipeline.SingleModel(pipeline.Model{Name: "kev-latest", Scorer: kev.client(t), Rules: set, + Retry: retry.Policy{Attempts: 1, Backoff: []time.Duration{time.Millisecond}}}) + h := newHarness(t, harnessOpts{models: models}) + + got := decodeBody[Result](t, h.do("POST", "/v1/traces:score", `{"question": "q", "final_answer": "not json"}`)) + if got.Decision != "keep" || got.Rule != "keep_rule" || got.Cause != "rule" || + got.Reason != "The final answer is not valid JSON." || got.PKeep != nil || got.Severity == nil || *got.Severity != 1 { + t.Errorf("result = %+v", got) + } + if len(got.Rules) != 2 || got.Rules[0] != (RuleHit{Name: "answer_json", OnFail: "keep", Reason: "The final answer is not valid JSON."}) || + got.Rules[1].Name != "short" || got.Rules[1].OnFail != "signal" { + t.Errorf("rules = %+v", got.Rules) + } + if len(kev.states) != 0 { + t.Errorf("Kev was asked %d times", len(kev.states)) + } + waitUntil(t, "the kept trace to be exported", func() bool { return len(h.recorder.Ended()) > 0 }) + a := rootAttrs(t, h.recorder) + if a["eval.rules"].AsString() != "answer_json,short" || a["eval.rules.answer_json"].AsString() != "The final answer is not valid JSON." || + a["eval.severity"].AsFloat64() != 1 { + t.Errorf("eval attributes = %v", a) + } + if _, ok := a["eval.p_keep"]; ok { + t.Error("eval.p_keep exported for a trace the model did not score") + } +} diff --git a/internal/server/server.go b/internal/server/server.go index 2440598..2e960e2 100644 --- a/internal/server/server.go +++ b/internal/server/server.go @@ -13,6 +13,7 @@ import ( "github.com/tecnosam/tailwise/internal/fault" "github.com/tecnosam/tailwise/internal/metrics" "github.com/tecnosam/tailwise/internal/observe" + "github.com/tecnosam/tailwise/internal/otlp" "github.com/tecnosam/tailwise/internal/pipeline" "github.com/tecnosam/tailwise/internal/trace" ) @@ -30,6 +31,9 @@ type Options struct { Metrics common.RequestMetrics // MaxBodyBytes 0 means 8 MiB. MaxBodyBytes int64 + // Assembler turns the spans sent to POST /v1/otlp/traces:score into a + // trace, as the OTLP receivers' assemblers do. + Assembler otlp.Assembler } type Server struct { @@ -50,6 +54,7 @@ func New(opts Options) *Server { s := &Server{opts: opts, mux: http.NewServeMux()} s.mux.HandleFunc("POST /v1/traces", s.handleEnqueue) s.mux.HandleFunc("POST /v1/traces:score", s.handleScore) + s.mux.HandleFunc("POST /v1/otlp/traces:score", s.handleScoreOTLP) s.mux.HandleFunc("GET /healthz", s.handleHealthz) s.mux.HandleFunc("GET /readyz", s.handleReadyz) s.mux.HandleFunc("GET /openapi.yaml", s.handleOpenAPI) @@ -83,6 +88,33 @@ func (s *Server) handleScore(w http.ResponseWriter, r *http.Request) { writeJSON(w, http.StatusOK, resultOf(s.opts.Engine.Process(r.Context(), t))) } +// handleScoreOTLP scores one trace sent as OTLP spans, the way the receivers' +// assemblers would build it, without the queue or the wait for late spans. +func (s *Server) handleScoreOTLP(w http.ResponseWriter, r *http.Request) { + export, _, code, err := otlp.ReadExport(w, r) + if err != nil { + writeError(w, code, err.Error()) + return + } + byTrace, _, err := otlp.Split(export.GetResourceSpans()) + if err != nil { + writeError(w, http.StatusBadRequest, err.Error()) + return + } + if len(byTrace) != 1 { + writeError(w, http.StatusBadRequest, "the request must hold the spans of exactly one trace") + return + } + for id, spans := range byTrace { + t, err := s.opts.Assembler.Assemble(id, [][]byte{spans}) + if err != nil { + writeError(w, http.StatusUnprocessableEntity, err.Error()) + return + } + writeJSON(w, http.StatusOK, resultOf(s.opts.Engine.Process(r.Context(), t))) + } +} + func (s *Server) handleHealthz(w http.ResponseWriter, _ *http.Request) { writeJSON(w, http.StatusOK, map[string]string{"status": "ok"}) } diff --git a/internal/server/server_test.go b/internal/server/server_test.go index a95bc66..33b40ae 100644 --- a/internal/server/server_test.go +++ b/internal/server/server_test.go @@ -485,7 +485,7 @@ func TestScoreEndToEndKeep(t *testing.T) { t.Errorf("response is missing %q: %s", f, w.Body) } } - for _, f := range []string{"p_keep_raw", "severity"} { + for _, f := range []string{"p_keep_raw"} { if _, ok := fields[f]; ok { t.Errorf("response has the removed field %q: %s", f, w.Body) } @@ -500,6 +500,9 @@ func TestScoreEndToEndKeep(t *testing.T) { if got.PKeep == nil || *got.PKeep != wantP { t.Errorf("p_keep = %v, want %v", deref(got.PKeep), wantP) } + if got.Severity == nil { + t.Error("severity = null, want it on every scored trace") + } // Kev was sent exactly the golden slice. wantState := string(readFile(t, "golden", "wrong_river.txt")) @@ -579,7 +582,7 @@ func TestScoreEndToEndFallback(t *testing.T) { if got.PKeep != nil { t.Errorf("p_keep = %v, want null for a fallback", deref(got.PKeep)) } - if !strings.Contains(w.Body.String(), `"p_keep":null`) { + if !strings.Contains(w.Body.String(), `"p_keep":null`) || !strings.Contains(w.Body.String(), `"severity":null`) { t.Errorf("body = %s, want explicit nulls", w.Body) } if got.Model != "kev-latest" { @@ -661,8 +664,7 @@ func TestDecisionLogLine(t *testing.T) { Answers: map[string]pipeline.Probability{"answer_wrong": 0.95, "unsupported_answer": 0.7, "missing_tool_calls": 0.1}, Deciding: map[string]pipeline.Probability{"answer_wrong": 0.95, "unsupported_answer": 0.7, "missing_tool_calls": 0.1}}, nil } - checks := []pipeline.Check{{Question: "answer_wrong", Weight: 0.6}, {Question: "unsupported_answer", Weight: 0.3}, - {Question: "missing_tool_calls", Weight: 0.1}} + checks := []pipeline.Check{{Question: "answer_wrong"}, {Question: "unsupported_answer"}, {Question: "missing_tool_calls"}} h := newHarness(t, harnessOpts{ models: pipeline.SingleModel(pipeline.Model{Name: "kev-latest", Scorer: &fakeScorer{score: answered}, Checks: checks}), logger: jsonLogger(&buf), @@ -690,8 +692,8 @@ func TestDecisionLogLine(t *testing.T) { if flags, _ := json.Marshal(line["flags"]); string(flags) != `["answer_wrong","unsupported_answer"]` { t.Errorf("log field flags = %s, want the deciding questions at or above the threshold", flags) } - if s, _ := line["severity"].(float64); math.Abs(s-(0.6*0.95+0.3*0.7)) > 1e-9 { - t.Errorf("log field severity = %v, want the flagged checks' weight × answer", line["severity"]) + if s, _ := line["severity"].(float64); math.Abs(s-(0.95+0.7+0.1)/3) > 1e-9 { + t.Errorf("log field severity = %v, want the deciding checks' average answer", line["severity"]) } } diff --git a/internal/trace/actions.go b/internal/trace/actions.go new file mode 100644 index 0000000..2131b3a --- /dev/null +++ b/internal/trace/actions.go @@ -0,0 +1,162 @@ +package trace + +import ( + "strconv" + "strings" +) + +// The states of the two action checks, which look closer than the main +// question can at agents that act for a user (cancel an order, book a flight, +// send money): an action call is one to a tool the manifest lists as an +// action. +const ( + // MaxLookups is how many earlier results an action check shows: the most + // recent ones holding a value the action passes. + MaxLookups = 3 + // MaxLookupChars caps one lookup result, cut to the parts holding a value + // the action passes. + MaxLookupChars = 2500 + // maxStatusChars caps an error message shown with an action's status. + maxStatusChars = 200 + // maxAgentTurnChars caps one agent turn in the completeness check: what + // it said it did matters more than its long listings. + maxAgentTurnChars = 600 + // maxConversationChars bounds the conversation shown; the earliest turns + // go first. + maxConversationChars = 24000 +) + +// HandledNote defines HANDLED for the completeness check's questions. +const HandledNote = "NOTES: A user request is HANDLED when the agent either carried it out with an " + + "AGENT ACTION, or explicitly declined it and told the user why. It is not handled if it was left " + + "unaddressed, done only in part, or the agent said it would act but no AGENT ACTION did it. " + + "Lookups are not shown." + +// BuildActionState is the state for checking one action call, the i-th tool +// call: does it do what the user asked? It holds the user's request, the +// earlier results the action's argument values came from (where the right +// item, order or account is identified), the action, and the actions after +// it, so a request handled in a later step is not blamed on this one. +func BuildActionState(t Trace, i int, isAction func(name string) bool) State { + call := t.ToolCalls[i] + values := argValues(call.Args) + + var lookups []int + for j := i - 1; j >= 0 && len(lookups) < MaxLookups; j-- { + c := t.ToolCalls[j] + if !isAction(c.Name) && containsAny(c.Output, values) { + lookups = append(lookups, j) + } + } + + var b strings.Builder + b.WriteString("USER REQUEST:\n") + b.WriteString(t.Question) + b.WriteString("\n\nLOOKUP RESULTS THE ACTION'S VALUES CAME FROM:\n") + if len(lookups) == 0 { + b.WriteString("(none)") + } + for k := len(lookups) - 1; k >= 0; k-- { + c := t.ToolCalls[lookups[k]] + if k < len(lookups)-1 { + b.WriteByte('\n') + } + b.WriteByte('[') + b.WriteString(strconv.Itoa(len(lookups) - k)) + b.WriteString("] ") + writeCall(&b, c) + b.WriteString(": ") + b.WriteString(fit(strings.TrimSpace(c.Output), values, MaxLookupChars)) + } + b.WriteString("\n\nACTION:\n") + writeCall(&b, call) + b.WriteString("\n\nLATER ACTIONS IN THIS RUN:\n") + later := false + for _, c := range t.ToolCalls[i+1:] { + if isAction(c.Name) { + if later { + b.WriteByte('\n') + } + b.WriteString("- ") + writeCall(&b, c) + later = true + } + } + if !later { + b.WriteString("(none)") + } + return State(b.String()) +} + +// BuildCompletenessState is the state for checking that the agent handled +// every request the user made: the conversation when the trace has one (else +// the user's request), the agent's actions in order (lookups left out), and +// its final answer. The agent's turns show what it declined, and requests the +// user withdrew. +func BuildCompletenessState(t Trace, isAction func(name string) bool) State { + var b strings.Builder + b.WriteString(HandledNote) + if turns := conversationText(t.Conversation); turns != "" { + b.WriteString("\n\nCONVERSATION:\n") + b.WriteString(turns) + } else { + b.WriteString("\n\nUSER REQUEST:\n") + b.WriteString(t.Question) + } + b.WriteString("\n\nAGENT ACTIONS:\n") + n := 0 + for _, c := range t.ToolCalls { + if !isAction(c.Name) { + continue + } + if n > 0 { + b.WriteByte('\n') + } + n++ + b.WriteString(strconv.Itoa(n)) + b.WriteString(". ") + writeCall(&b, c) + if c.StatusText() != "ok" && strings.TrimSpace(c.Output) != "" { + b.WriteString(": ") + b.WriteString(truncateRunes(strings.TrimSpace(c.Output), maxStatusChars)) + } + } + if n == 0 { + b.WriteString("(none)") + } + b.WriteString("\n\nFINAL ANSWER:\n") + b.WriteString(t.FinalAnswer) + return State(b.String()) +} + +// conversationText is the turns as USER: and AGENT: lines, agent turns cut to +// maxAgentTurnChars, the earliest turns dropped beyond maxConversationChars. +func conversationText(turns []Message) string { + lines := make([]string, 0, len(turns)) + for _, m := range turns { + text := strings.TrimSpace(m.Content) + if m.Role == "assistant" { + if len(text) > maxAgentTurnChars { + text = truncateRunes(text, maxAgentTurnChars) + " [...]" + } + lines = append(lines, "AGENT: "+text) + } else { + lines = append(lines, "USER: "+text) + } + } + total := 0 + start := len(lines) + for start > 0 && total+len(lines[start-1]) <= maxConversationChars { + start-- + total += len(lines[start]) + 2 + } + return strings.Join(lines[start:], "\n\n") +} + +func writeCall(b *strings.Builder, c ToolCall) { + b.WriteString(c.Name) + b.WriteByte('(') + b.WriteString(c.ArgsText()) + b.WriteString(") -> ") + b.WriteString(c.StatusText()) +} diff --git a/internal/trace/actions_test.go b/internal/trace/actions_test.go new file mode 100644 index 0000000..8f9b895 --- /dev/null +++ b/internal/trace/actions_test.go @@ -0,0 +1,89 @@ +package trace + +import ( + "encoding/json" + "fmt" + "strings" + "testing" +) + +func isWrite(name string) bool { return name == "exchange" || name == "cancel" } + +// A retail-like run: the agent looks up an order and a product, exchanges an +// item for a variant, then cancels another order. +func exchangeRun() Trace { + return Trace{ + Question: "Exchange my keyboard in order #W100 for the size L one. Also cancel order #W200.", + ToolCalls: []ToolCall{ + call("get_order", `{"order_id":"#W100"}`, `{"order_id":"#W100","items":[{"item_id":"1111"}]}`), + call("get_product", `{"product_id":"77"}`, + `{"variants":{"2222":{"size":"M"},"3333":{"size":"L"}},"name":"keyboard"}`), + call("get_user", `{"user_id":"yusuf_1"}`, `{"name":"Yusuf"}`), + call("exchange", `{"order_id":"#W100","item_ids":["1111"],"new_item_ids":["2222"]}`, `{"status":"ok"}`), + {Name: "cancel", Args: json.RawMessage(`{"order_id":"#W200"}`), Status: json.RawMessage(`"error"`), + Output: "Error: order #W200 is already delivered"}, + }, + FinalAnswer: "Done: exchanged your keyboard. I could not cancel #W200: it was delivered.", + } +} + +func TestActionStateShowsTheLookupsTheActionRestsOn(t *testing.T) { + got := string(BuildActionState(exchangeRun(), 3, isWrite)) + want := "USER REQUEST:\nExchange my keyboard in order #W100 for the size L one. Also cancel order #W200.\n\n" + + "LOOKUP RESULTS THE ACTION'S VALUES CAME FROM:\n" + + `[1] get_order({"order_id":"#W100"}) -> ok: {"order_id":"#W100","items":[{"item_id":"1111"}]}` + "\n" + + `[2] get_product({"product_id":"77"}) -> ok: {"variants":{"2222":{"size":"M"},"3333":{"size":"L"}},"name":"keyboard"}` + "\n\n" + + "ACTION:\n" + `exchange({"order_id":"#W100","item_ids":["1111"],"new_item_ids":["2222"]}) -> ok` + "\n\n" + + "LATER ACTIONS IN THIS RUN:\n" + `- cancel({"order_id":"#W200"}) -> error` + if got != want { + t.Errorf("state =\n%s\nwant\n%s", got, want) + } +} + +// A long lookup is cut to the objects holding the action's values. +func TestActionStateTrimsLongLookups(t *testing.T) { + tr := exchangeRun() + var other strings.Builder + for i := range 300 { + fmt.Fprintf(&other, `"9%03d":{"item_id":"9%03d","size":"XL"},`, i, i) + } + tr.ToolCalls[1].Output = `{"variants":{"2222":{"item_id":"2222","size":"M"},` + other.String() + `"3333":{"item_id":"3333","size":"L"}}}` + got := string(BuildActionState(tr, 3, isWrite)) + if !strings.Contains(got, `"item_id":"2222"`) || strings.Contains(got, "9001") || len(got) > 2*MaxLookupChars { + t.Errorf("lookup not trimmed to the used variant (%d characters):\n%.600s", len(got), got) + } +} + +func TestCompletenessStateListsActionsOnly(t *testing.T) { + got := string(BuildCompletenessState(exchangeRun(), isWrite)) + for _, want := range []string{HandledNote, "USER REQUEST:\nExchange my keyboard", + "AGENT ACTIONS:\n1. exchange(" + `{"order_id":"#W100","item_ids":["1111"],"new_item_ids":["2222"]}` + ") -> ok\n" + + `2. cancel({"order_id":"#W200"}) -> error: Error: order #W200 is already delivered`, + "FINAL ANSWER:\nDone: exchanged your keyboard."} { + if !strings.Contains(got, want) { + t.Errorf("state lacks %q:\n%s", want, got) + } + } + if strings.Contains(got, "get_order") { + t.Errorf("lookups shown:\n%s", got) + } + if got := string(BuildCompletenessState(Trace{Question: "q", FinalAnswer: "a"}, isWrite)); !strings.Contains(got, "AGENT ACTIONS:\n(none)") { + t.Errorf("no actions:\n%s", got) + } +} + +// With a conversation, the completeness check sees what the agent said: here, +// that the user withdrew a request. +func TestCompletenessStateShowsTheConversation(t *testing.T) { + tr := exchangeRun() + tr.Conversation = []Message{ + {Role: "user", Content: "Change my order to the 10x zoom camera."}, + {Role: "assistant", Content: "That variant costs $3,066. Shall I proceed?" + strings.Repeat(" More details.", 100)}, + {Role: "user", Content: "Too expensive. Cancel the order instead."}, + } + got := string(BuildCompletenessState(tr, isWrite)) + if !strings.Contains(got, "CONVERSATION:\nUSER: Change my order to the 10x zoom camera.\n\nAGENT: That variant costs") || + !strings.Contains(got, "[...]\n\nUSER: Too expensive.") || strings.Contains(got, "USER REQUEST:") { + t.Errorf("state:\n%s", got) + } +} diff --git a/internal/trace/fit.go b/internal/trace/fit.go new file mode 100644 index 0000000..8b3ae95 --- /dev/null +++ b/internal/trace/fit.go @@ -0,0 +1,115 @@ +package trace + +import ( + "encoding/json" + "slices" + "unicode/utf8" +) + +// MaxStateChars bounds the state sent to the model. Jev rejects longer +// requests (max_tokens_exceeded), and a rejected trace falls back to a random +// decision. In the benchmark it billed up to 26.5k tokens without rejecting, +// and rejected a 64,000-character patch dense enough to be about 35k: at that +// density this bound is about 22k. +const MaxStateChars = 40000 + +const ( + cutMark = "\n[... cut to fit ...]\n" + // minCutChars is the shortest a field is cut to: below it, Fit gives up + // rather than cut every field to nothing. + minCutChars = 200 +) + +// Fit renders t with build and, while the state is longer than limit +// characters, cuts the middle out of the longest field: the question, the +// final answer, the system prompt, an evidence item, or a tool call's +// arguments or result. It keeps the start and end of each, where a patch's +// header and a log's outcome usually are. trimmed reports whether anything was +// cut; the state can still be over limit when every field is already short. +func Fit(t Trace, limit int, build func(Trace) State) (state State, trimmed bool) { + state = build(t) + if len(state) <= limit { + return state, false + } + t.ToolCalls = slices.Clone(t.ToolCalls) + t.Evidence = slices.Clone(t.Evidence) + fields := textFields(&t) + spent := make([]bool, len(fields)) // cut without shortening the state: not shown, or escaping ate the cut + for len(state) > limit { + i := longest(fields, spent) + if i < 0 { + break + } + f := fields[i] + keep := max(f.len()-(len(state)-limit)-len(cutMark), minCutChars) + f.set(cutMiddle(f.get(), keep)) + trimmed = true + before := len(state) + if state = build(t); len(state) >= before { + spent[i] = true + } + } + return state, trimmed +} + +type textField struct { + get func() string + set func(string) +} + +func (f textField) len() int { return len(f.get()) } + +func textFields(t *Trace) []textField { + fields := []textField{ + {func() string { return t.Question }, func(s string) { t.Question = s }}, + {func() string { return t.FinalAnswer }, func(s string) { t.FinalAnswer = s }}, + {func() string { return t.SystemPrompt }, func(s string) { t.SystemPrompt = s }}, + } + for i := range t.Evidence { + fields = append(fields, textField{func() string { return t.Evidence[i] }, func(s string) { t.Evidence[i] = s }}) + } + for i := range t.ToolCalls { + c := &t.ToolCalls[i] + fields = append(fields, + textField{func() string { return c.Output }, func(s string) { c.Output = s }}, + // Cut arguments become a JSON string of the cut text. + textField{func() string { return argsText(c.Args) }, func(s string) { c.Args, _ = json.Marshal(s) }}) + } + return fields +} + +// longest is the index of the longest field still worth cutting, -1 when +// none is longer than minCutChars. +func longest(fields []textField, spent []bool) int { + best := -1 + for i, f := range fields { + if !spent[i] && f.len() > minCutChars && (best < 0 || f.len() > fields[best].len()) { + best = i + } + } + return best +} + +// argsText is the arguments as text: a JSON string's value, so cutting it +// again does not escape it twice, else the raw JSON. +func argsText(args json.RawMessage) string { + var s string + if json.Unmarshal(args, &s) == nil { + return s + } + return string(args) +} + +// cutMiddle keeps about keep bytes of s, half from each end, on rune +// boundaries, with cutMark between them. +func cutMiddle(s string, keep int) string { + head, tail := keep/2, keep-keep/2 + for head > 0 && !utf8.RuneStart(s[head]) { + head-- + } + start := len(s) - tail + for start < len(s) && !utf8.RuneStart(s[start]) { + start++ + } + return s[:head] + cutMark + s[start:] +} diff --git a/internal/trace/fit_test.go b/internal/trace/fit_test.go new file mode 100644 index 0000000..cf02577 --- /dev/null +++ b/internal/trace/fit_test.go @@ -0,0 +1,72 @@ +package trace + +import ( + "encoding/json" + "strings" + "testing" + "unicode/utf8" +) + +func TestFitLeavesShortStatesAlone(t *testing.T) { + tr := Trace{Question: "q", FinalAnswer: "a"} + state, trimmed := Fit(tr, MaxStateChars, BuildState) + if trimmed || state != BuildState(tr) { + t.Errorf("trimmed = %v, state =\n%s", trimmed, state) + } +} + +// A giant patch as the final answer is cut in the middle, keeping its start +// and end, and the trace itself is untouched. +func TestFitCutsTheLongestField(t *testing.T) { + patch := "diff --git a/x.py b/x.py\n" + strings.Repeat("+line\n", 50000) + "END OF PATCH" + tr := Trace{Question: "Fix the bug.", ToolCalls: []ToolCall{call("ls", `{"path":"."}`, "x.py")}, FinalAnswer: patch} + state, trimmed := Fit(tr, 10000, BuildState) + if !trimmed || len(state) > 10000 { + t.Fatalf("trimmed = %v, %d characters, want at most 10000", trimmed, len(state)) + } + for _, want := range []string{"USER QUESTION:\nFix the bug.", "1. ls({\"path\":\".\"}) -> ok", + "FINAL ANSWER:\ndiff --git a/x.py", cutMark, "END OF PATCH"} { + if !strings.Contains(string(state), want) { + t.Errorf("state lost %q", want) + } + } + if tr.FinalAnswer != patch { + t.Error("Fit changed the caller's trace") + } +} + +func TestFitCutsArgumentsToAJSONString(t *testing.T) { + big, _ := json.Marshal(map[string]string{"content": strings.Repeat("é", 20000)}) + tr := Trace{Question: "q", ToolCalls: []ToolCall{{Name: "write", Args: big}}, FinalAnswer: "a"} + state, trimmed := Fit(tr, 5000, func(t Trace) State { return BuildStateGrounded(t, 12000) }) + if !trimmed || len(state) > 5000 || !utf8.ValidString(string(state)) { + t.Fatalf("trimmed = %v, %d characters, valid UTF-8 %v", trimmed, len(state), utf8.ValidString(string(state))) + } + if !strings.Contains(string(state), `1. write("{\"content\":\"éé`) { + t.Errorf("arguments not cut to a JSON string:\n%.200s", state) + } +} + +// BuildState does not show tool results, so cutting the longest one does +// nothing; Fit moves on to the final answer. +func TestFitSkipsFieldsTheStateDoesNotShow(t *testing.T) { + tr := Trace{Question: "q", ToolCalls: []ToolCall{call("cat", `{}`, strings.Repeat("x", 90000))}, + FinalAnswer: strings.Repeat("y", 30000)} + state, trimmed := Fit(tr, 10000, BuildState) + if !trimmed || len(state) > 10000 { + t.Errorf("trimmed = %v, %d characters, want at most 10000", trimmed, len(state)) + } +} + +// When every field is already short, Fit stops rather than cut them away. +func TestFitGivesUpOnManyShortFields(t *testing.T) { + var calls []ToolCall + for range 500 { + calls = append(calls, call("step", `{"n":1}`, "")) + } + tr := Trace{Question: "q", ToolCalls: calls, FinalAnswer: "a"} + state, _ := Fit(tr, 1000, BuildState) + if !strings.Contains(string(state), "500. step") { + t.Error("Fit dropped tool calls") + } +} diff --git a/internal/trace/grounded.go b/internal/trace/grounded.go new file mode 100644 index 0000000..c3d7e56 --- /dev/null +++ b/internal/trace/grounded.go @@ -0,0 +1,336 @@ +package trace + +import ( + "encoding/json" + "regexp" + "slices" + "strconv" + "strings" +) + +// Evidence strategies: how tool results reach the model. +const ( + // EvidenceFirst shows the first MaxEvidence evidence items (BuildState). + EvidenceFirst = "first" + // EvidenceGrounded shows tool results on their tool-call lines, chosen by + // priority within a budget (BuildStateGrounded). + EvidenceGrounded = "grounded" +) + +const ( + // MaxResultChars caps one tool result in the grounded slice. + MaxResultChars = 2000 + // maxInlineChars is the longest result kept on its tool-call line; a + // longer one, or one spanning lines, goes on indented lines below it. + maxInlineChars = 300 + minResultChars = 80 + trimmedMark = " [trimmed]" + resultIndent = " " +) + +// BuildStateGrounded renders the trace like BuildState, but shows each tool +// call's result on its line, as long as budget characters of results allow. +// Results are chosen in this order: +// +// 1. results holding a value that a later call passes as an argument, or +// that the final answer cites: what the agent's actions rest on; +// 2. the remaining results, most recent first. +// +// A result longer than MaxResultChars keeps the parts later calls used (for +// JSON, each object holding such a value) before it is cut. Evidence items +// that came from tool results are not repeated in the evidence section. +func BuildStateGrounded(t Trace, budget int) State { + results := chooseResults(t, budget) + + var b strings.Builder + if t.SystemPrompt != "" { + b.WriteString("SYSTEM PROMPT:\n") + b.WriteString(truncateRunes(t.SystemPrompt, MaxSystemPromptChars)) + b.WriteString("\n\n") + } + b.WriteString("USER QUESTION:\n") + b.WriteString(t.Question) + + b.WriteString("\n\nTOOL CALLS:\n") + if len(t.ToolCalls) == 0 { + b.WriteString("(none)") + } + for i, c := range t.ToolCalls { + if i > 0 { + b.WriteByte('\n') + } + b.WriteString(strconv.Itoa(i + 1)) + b.WriteString(". ") + b.WriteString(c.Name) + b.WriteByte('(') + b.WriteString(c.ArgsText()) + b.WriteString(") -> ") + b.WriteString(c.StatusText()) + if r := results[i]; r != "" { + if len(r) <= maxInlineChars && !strings.Contains(r, "\n") { + b.WriteString(": ") + b.WriteString(r) + } else { + b.WriteString(":\n") + b.WriteString(resultIndent) + b.WriteString(strings.ReplaceAll(r, "\n", "\n"+resultIndent)) + } + } + } + + b.WriteString("\n\nEVIDENCE:\n") + evidence := otherEvidence(t) + if len(evidence) == 0 { + b.WriteString("(none)") + } + for i, e := range evidence { + if i > 0 { + b.WriteByte('\n') + } + b.WriteByte('[') + b.WriteString(strconv.Itoa(i + 1)) + b.WriteString("] ") + b.WriteString(truncateRunes(e, MaxEvidenceChars)) + } + + b.WriteString("\n\nFINAL ANSWER:\n") + b.WriteString(t.FinalAnswer) + return State(b.String()) +} + +// chooseResults is the text shown for each call's result, "" for none. +func chooseResults(t Trace, budget int) []string { + shown := make([]string, len(t.ToolCalls)) + used := laterValues(t) + var grounding, rest []int + for i := len(t.ToolCalls) - 1; i >= 0; i-- { + out := strings.TrimSpace(t.ToolCalls[i].Output) + switch { + case out == "": + case containsAny(out, used[i]): + grounding = append(grounding, i) + default: + rest = append(rest, i) + } + } + left := budget + for _, i := range append(grounding, rest...) { + if left < minResultChars { + break + } + r := fit(strings.TrimSpace(t.ToolCalls[i].Output), used[i], min(MaxResultChars, left)) + shown[i] = r + left -= len(r) + } + return shown +} + +// fit returns out whole if it fits in limit bytes, else the parts of it that +// hold a used value, else its start; cut to limit. +func fit(out string, used []string, limit int) string { + if len(out) <= limit { + return out + } + if trimmed, ok := keepUsed(out, used); ok && len(trimmed)+len(trimmedMark) <= limit { + return trimmed + trimmedMark + } + cut := truncateBytes(out, limit-len(trimmedMark)) + return cut + trimmedMark +} + +// laterValues lists, for each call, the distinctive values that later calls +// pass as arguments or that the final answer cites. +func laterValues(t Trace) [][]string { + out := make([][]string, len(t.ToolCalls)) + seen := map[string]bool{} + var acc []string + add := func(v string) { + if !seen[v] { + seen[v] = true + acc = append(acc, v) + } + } + for _, v := range idLike.FindAllString(t.FinalAnswer, -1) { + if distinctive(v) { + add(v) + } + } + for i := len(t.ToolCalls) - 1; i >= 0; i-- { + out[i] = slices.Clone(acc) + for _, v := range argValues(t.ToolCalls[i].Args) { + add(v) + } + } + return out +} + +// idLike matches the identifiers a final answer cites: a run of letters, +// digits, # or - with a digit in it (distinctive drops the short ones). +var idLike = regexp.MustCompile(`#?[A-Za-z0-9-]*[0-9][A-Za-z0-9-]*`) + +// argValues is every distinctive string or number in a call's arguments. +func argValues(raw json.RawMessage) []string { + var v any + if len(raw) == 0 || json.Unmarshal(raw, &v) != nil { + return nil + } + var out []string + var walk func(any) + walk = func(v any) { + switch x := v.(type) { + case map[string]any: + for _, c := range x { + walk(c) + } + case []any: + for _, c := range x { + walk(c) + } + case string: + if s := strings.TrimSpace(x); distinctive(s) { + out = append(out, s) + } + case float64: + if s := strconv.FormatFloat(x, 'f', -1, 64); distinctive(s) { + out = append(out, s) + } + } + } + walk(v) + return out +} + +// distinctive keeps values specific enough that finding one in a result +// means the result is where it came from: no short words or small numbers. +func distinctive(s string) bool { + if len(s) > 200 { + return false + } + hasDigit := strings.ContainsAny(s, "0123456789") + return (hasDigit && len(s) >= 4) || len(s) >= 8 +} + +func containsAny(s string, values []string) bool { + for _, v := range values { + if v != "" && strings.Contains(s, v) { + return true + } + } + return false +} + +// keepUsed reduces a JSON result to the objects holding a used value (each +// kept whole) and the top-level scalars, or a text result to its lines +// holding one. ok is false when nothing in it was used. +func keepUsed(out string, used []string) (string, bool) { + var v any + if json.Unmarshal([]byte(out), &v) == nil { + kept, ok := keepJSON(v, used, true) + if !ok { + return "", false + } + b, err := json.Marshal(kept) + return string(b), err == nil + } + var lines []string + for _, line := range strings.Split(out, "\n") { + if containsAny(line, used) { + lines = append(lines, line) + } + } + return strings.Join(lines, "\n"), len(lines) > 0 +} + +// keepJSON keeps an object whole if one of its own scalar fields is a used +// value; otherwise it keeps only the children that hold one (and, at the top +// level, the scalar fields, which name what the result is about). +func keepJSON(v any, used []string, top bool) (any, bool) { + switch x := v.(type) { + case map[string]any: + for _, c := range x { + if scalarUsed(c, used) { + return x, true + } + } + kept := map[string]any{} + found := false + for k, c := range x { + if sub, ok := keepJSON(c, used, false); ok { + kept[k] = sub + found = true + } else if top && isScalar(c) { + kept[k] = c + } + } + return kept, found + case []any: + var kept []any + for _, c := range x { + if sub, ok := keepJSON(c, used, false); ok { + kept = append(kept, sub) + } + } + return kept, len(kept) > 0 + default: + return x, scalarUsed(x, used) + } +} + +func scalarUsed(v any, used []string) bool { + var s string + switch x := v.(type) { + case string: + s = x + case float64: + s = strconv.FormatFloat(x, 'f', -1, 64) + default: + return false + } + return slices.Contains(used, strings.TrimSpace(s)) +} + +func isScalar(v any) bool { + switch v.(type) { + case map[string]any, []any: + return false + } + return true +} + +// otherEvidence is the trace's evidence minus items taken from tool results, +// which the grounded slice shows on the tool-call lines instead. +func otherEvidence(t Trace) []string { + var out []string + for _, e := range t.Evidence { + fromTool := false + for _, c := range t.ToolCalls { + if c.Output != "" && strings.Contains(c.Output, strings.TrimSpace(e)) { + fromTool = true + break + } + } + if !fromTool { + out = append(out, e) + } + } + if len(out) > MaxEvidence { + out = out[:MaxEvidence] + } + return out +} + +// truncateBytes cuts s to at most n bytes without splitting a character. +func truncateBytes(s string, n int) string { + if n <= 0 { + return "" + } + if len(s) <= n { + return s + } + for n > 0 && !utf8RuneStart(s[n]) { + n-- + } + return s[:n] +} + +func utf8RuneStart(b byte) bool { return b&0xC0 != 0x80 } diff --git a/internal/trace/grounded_test.go b/internal/trace/grounded_test.go new file mode 100644 index 0000000..57fc2a3 --- /dev/null +++ b/internal/trace/grounded_test.go @@ -0,0 +1,125 @@ +package trace + +import ( + "encoding/json" + "strconv" + "strings" + "testing" +) + +func call(name, args, output string) ToolCall { + return ToolCall{Name: name, Args: json.RawMessage(args), Output: output} +} + +// A trace with no tool outputs renders exactly as BuildState does. +func TestGroundedWithoutOutputsMatchesBuildState(t *testing.T) { + tr := Trace{ + Question: "Which river flows through Paris?", + ToolCalls: []ToolCall{call("search", `{"q":"river Paris"}`, "")}, + Evidence: []string{"The Seine flows through Paris."}, + FinalAnswer: "The Seine", + } + if got, want := BuildStateGrounded(tr, 12000), BuildState(tr); got != want { + t.Errorf("grounded:\n%s\nwant:\n%s", got, want) + } +} + +func TestGroundedShowsResultsOnTheirLines(t *testing.T) { + tr := Trace{ + Question: "Cancel order #W1.", + ToolCalls: []ToolCall{ + call("get_order", `{"order_id":"#W1"}`, `{"status":"pending"}`), + {Name: "cancel", Args: json.RawMessage(`{"order_id":"#W1"}`), Status: json.RawMessage(`"error"`), + Output: "Error: line one\nline two"}, + }, + FinalAnswer: "Cancelled.", + } + got := string(BuildStateGrounded(tr, 12000)) + if !strings.Contains(got, "1. get_order({\"order_id\":\"#W1\"}) -> ok: {\"status\":\"pending\"}\n") { + t.Errorf("short result not inline:\n%s", got) + } + if !strings.Contains(got, "2. cancel({\"order_id\":\"#W1\"}) -> error:\n Error: line one\n line two\n") { + t.Errorf("multi-line result not indented below its call:\n%s", got) + } +} + +// With a small budget, the result a later call's arguments came from is shown +// before more recent ones that nothing used. +func TestGroundedPrefersResultsLaterCallsUse(t *testing.T) { + filler := strings.Repeat("x", 150) + tr := Trace{ + Question: "Exchange my keyboard.", + ToolCalls: []ToolCall{ + call("get_product", `{"id":"1656367028"}`, `{"variants":{"9690244451":{"item_id":"9690244451","size":"L"}}}`), + call("get_user", `{"id":"u1"}`, `{"note":"`+filler+`"}`), + call("get_policy", `{}`, `{"note":"`+filler+`"}`), + call("exchange", `{"new_item_ids":["9690244451"]}`, `{"ok":true}`), + }, + FinalAnswer: "Done.", + } + got := string(BuildStateGrounded(tr, 200)) + if !strings.Contains(got, `1. get_product({"id":"1656367028"}) -> ok: {"variants":{"9690244451"`) { + t.Errorf("the grounding result is not shown:\n%s", got) + } + if strings.Contains(got, filler) { + t.Errorf("an unused result was shown before the budget ran out:\n%s", got) + } +} + +// A long JSON result keeps the object holding the value a later call used, +// and the top-level fields that say what it is. +func TestGroundedTrimsLongResultsToWhatWasUsed(t *testing.T) { + variants := map[string]any{} + for i := range 60 { + id := 9000000000 + i + variants[strconv.Itoa(id)] = map[string]any{"item_id": strconv.Itoa(id), "options": map[string]string{"size": "S"}} + } + variants["9690244451"] = map[string]any{"item_id": "9690244451", "options": map[string]string{"size": "L"}} + out, _ := json.Marshal(map[string]any{"name": "Keyboard", "product_id": "1656367028", "variants": variants}) + if len(out) <= MaxResultChars { + t.Fatalf("fixture result is only %d bytes", len(out)) + } + tr := Trace{ + Question: "Exchange for size L.", + ToolCalls: []ToolCall{ + call("get_product", `{"id":"1656367028"}`, string(out)), + call("exchange", `{"new_item_ids":["9690244451"]}`, `{"ok":true}`), + }, + FinalAnswer: "Done.", + } + got := string(BuildStateGrounded(tr, 12000)) + want := `{"name":"Keyboard","product_id":"1656367028","variants":{"9690244451":{"item_id":"9690244451","options":{"size":"L"}}}} [trimmed]` + if !strings.Contains(got, want) { + t.Errorf("trimmed result missing %s in:\n%s", want, got) + } +} + +func TestGroundedStaysWithinBudget(t *testing.T) { + var calls []ToolCall + for i := range 30 { + calls = append(calls, call("get", `{"i":`+strconv.Itoa(i)+`}`, strings.Repeat("y", 900))) + } + tr := Trace{Question: "q", ToolCalls: calls, FinalAnswer: "a"} + plain := len(BuildState(tr)) + for _, budget := range []int{500, 5000} { + extra := len(BuildStateGrounded(tr, budget)) - plain + // Each shown result adds ": " (or ":\n" and an indent) to its line. + if extra > budget+len(calls)*(2+len(resultIndent)) { + t.Errorf("budget %d: results added %d characters", budget, extra) + } + } +} + +// Evidence taken from tool results is not repeated; other evidence stays. +func TestGroundedDoesNotRepeatToolEvidence(t *testing.T) { + tr := Trace{ + Question: "q", + ToolCalls: []ToolCall{call("search", `{}`, "Paris is in France.")}, + Evidence: []string{"Paris is in France.", "A retrieved document."}, + FinalAnswer: "a", + } + got := string(BuildStateGrounded(tr, 12000)) + if !strings.Contains(got, "EVIDENCE:\n[1] A retrieved document.\n") || strings.Count(got, "Paris is in France.") != 1 { + t.Errorf("evidence section:\n%s", got) + } +} diff --git a/internal/trace/slice.go b/internal/trace/slice.go index 391b1ab..3e1e002 100644 --- a/internal/trace/slice.go +++ b/internal/trace/slice.go @@ -7,17 +7,24 @@ import ( // Changing these changes what the model sees. const ( - MaxEvidence = 3 - MaxEvidenceChars = 1500 + MaxEvidence = 3 + MaxEvidenceChars = 1500 + MaxSystemPromptChars = 8000 ) type State string // BuildState renders the trace into the model's state string. The golden -// tests pin its exact output. +// tests pin its exact output. The system prompt section is left out when +// there is none, so a trace without one renders exactly as before. func BuildState(t Trace) State { var b strings.Builder + if t.SystemPrompt != "" { + b.WriteString("SYSTEM PROMPT:\n") + b.WriteString(truncateRunes(t.SystemPrompt, MaxSystemPromptChars)) + b.WriteString("\n\n") + } b.WriteString("USER QUESTION:\n") b.WriteString(t.Question) diff --git a/internal/trace/slice_test.go b/internal/trace/slice_test.go index 23a16bb..60baaf4 100644 --- a/internal/trace/slice_test.go +++ b/internal/trace/slice_test.go @@ -28,7 +28,7 @@ func TestBuildStateGolden(t *testing.T) { if len(traces) == 0 { t.Fatal("no trace fixtures found") } - if len(traces) != len(goldens) { + if !*update && len(traces) != len(goldens) { t.Fatalf("%d traces but %d golden files; run `make golden`", len(traces), len(goldens)) } diff --git a/internal/trace/trace.go b/internal/trace/trace.go index ccfd6f6..0bcdeb0 100644 --- a/internal/trace/trace.go +++ b/internal/trace/trace.go @@ -18,26 +18,42 @@ type AgentName string const MaxAgentLength = 128 type Trace struct { - TraceID string `json:"trace_id,omitempty"` - Agent AgentName `json:"agent,omitempty"` - Question string `json:"question"` - ToolCalls []ToolCall `json:"tool_calls,omitempty"` - Evidence []string `json:"evidence,omitempty"` + TraceID string `json:"trace_id,omitempty"` + Agent AgentName `json:"agent,omitempty"` + // SystemPrompt is the instructions the agent ran under, so the model can + // judge the run against them; empty when unknown. + SystemPrompt string `json:"system_prompt,omitempty"` + Question string `json:"question"` + ToolCalls []ToolCall `json:"tool_calls,omitempty"` + Evidence []string `json:"evidence,omitempty"` // FinalAnswer must be present but may be empty: a run that ended without an // answer is still scored. FinalAnswer string `json:"final_answer"` + // Conversation is the user's and the agent's turns in order, when known: + // the completeness check reads it to see what the agent said, such as a + // request it declined. + Conversation []Message `json:"conversation,omitempty"` // OTLP holds the original spans of a trace that came through the OTLP // receivers; a kept trace is forwarded as them instead of being rebuilt. OTLP []byte `json:"otlp_spans,omitempty"` } +// Message is one turn of the conversation. +type Message struct { + Role string `json:"role"` // user or assistant + Content string `json:"content"` +} + // Args and Status stay raw: the slice writes them as JSON and needs to know // whether each was present. type ToolCall struct { Name string `json:"name"` Args json.RawMessage `json:"args,omitempty"` Status json.RawMessage `json:"status,omitempty"` + // Output is what the call returned, as text. BuildState does not show it + // (it shows evidence); BuildStateGrounded shows it on the call's line. + Output string `json:"output,omitempty"` } // StatusText is the string itself, "ok" when absent, or compact JSON. @@ -103,6 +119,11 @@ func (t Trace) Validate() error { return fmt.Errorf("tool_calls[%d].name is required", i) } } + for i, m := range t.Conversation { + if m.Role != "user" && m.Role != "assistant" { + return fmt.Errorf("conversation[%d].role must be user or assistant, got %q", i, m.Role) + } + } return nil } diff --git a/internal/trace/trace_test.go b/internal/trace/trace_test.go index c0d415e..2240a9f 100644 --- a/internal/trace/trace_test.go +++ b/internal/trace/trace_test.go @@ -51,6 +51,7 @@ func TestDecodeErrors(t *testing.T) { {"wrong type", `{"question": 5, "final_answer": "a"}`, "invalid JSON"}, {"evidence not strings", `{"question": "q", "final_answer": "a", "evidence": [1]}`, "invalid JSON"}, {"tool call without name", `{"question": "q", "final_answer": "a", "tool_calls": [{"args": "x"}]}`, "tool_calls[0].name is required"}, + {"conversation turn with another role", `{"question": "q", "final_answer": "a", "conversation": [{"role": "tool", "content": "x"}]}`, "conversation[0].role must be user or assistant"}, {"trailing data", `{"question": "q", "final_answer": "a"} {}`, "unexpected data"}, {"array", `[]`, "invalid JSON"}, {"agent too long", `{"question": "q", "final_answer": "a", "agent": "` + strings.Repeat("x", MaxAgentLength+1) + `"}`, "agent is longer than 128 bytes"}, @@ -92,3 +93,31 @@ func TestNewID(t *testing.T) { t.Error("two generated IDs are equal") } } + +func TestDecodeSystemPrompt(t *testing.T) { + tr, err := Decode(strings.NewReader(`{"system_prompt": "Be kind.", "question": "q", "final_answer": "a"}`)) + if err != nil { + t.Fatal(err) + } + if got := string(BuildState(tr)); !strings.HasPrefix(got, "SYSTEM PROMPT:\nBe kind.\n\nUSER QUESTION:\nq") { + t.Errorf("state = %q", got) + } +} + +// Without a system prompt the section is left out: the state is exactly what +// it was before system prompts existed. +func TestBuildStateWithoutSystemPrompt(t *testing.T) { + got := string(BuildState(Trace{Question: "q", FinalAnswer: "a"})) + if !strings.HasPrefix(got, "USER QUESTION:\nq") || strings.Contains(got, "SYSTEM PROMPT") { + t.Errorf("state = %q", got) + } +} + +func TestBuildStateCutsTheSystemPrompt(t *testing.T) { + long := strings.Repeat("é", MaxSystemPromptChars+50) + got := string(BuildState(Trace{SystemPrompt: long, Question: "q", FinalAnswer: "a"})) + want := "SYSTEM PROMPT:\n" + strings.Repeat("é", MaxSystemPromptChars) + "\n\nUSER QUESTION:" + if !strings.HasPrefix(got, want) { + t.Errorf("system prompt not cut to %d code points", MaxSystemPromptChars) + } +} diff --git a/tailwise.schema.yaml b/tailwise.schema.yaml index b078b94..90635d5 100644 --- a/tailwise.schema.yaml +++ b/tailwise.schema.yaml @@ -35,6 +35,19 @@ $defs: type: number minimum: 0 maximum: 1 + ruleText: + type: object + description: >- + What a piece of text must be. Each key set is one expectation; the rule + fires when any is broken. + properties: + not_empty: {type: boolean, description: Fires when it is empty or only whitespace.} + min_length: {type: integer, minimum: 0, description: Fires under this many characters.} + max_length: {type: integer, minimum: 0, description: Fires over this many characters.} + matches: {type: string, description: A regular expression (Go syntax) it must match.} + not_matches: {type: string, description: A regular expression it must not match.} + json: {type: boolean, description: Fires when it does not parse as JSON.} + json_schema: {type: object, description: 'A JSON Schema (draft 2020-12 unless it says) it must be valid against.'} properties: server: @@ -318,6 +331,38 @@ properties: why_wrong. A manifest without it gives reasons that only say how likely the answer is wrong, with a warning at start-up. Empty turns reasons off. + system_prompt: + type: boolean + description: >- + Show the model the agent's system prompt, above the question. The + prompt comes from the trace's system_prompt (JSON API), the root + span's tailwise.system_prompt, or the first LLM span's system + message, and is cut to 8000 characters. Off by default: in the + benchmark it kept more traces without ranking failures better, + and it adds the prompt's tokens to every call. + evidence: + type: string + enum: [first, grounded] + default: grounded + description: >- + How tool results reach the model. grounded: each tool call's result + on its line, within evidence_budget characters, choosing first the + results whose values later calls use as arguments, then the most + recent; a long result keeps the parts later calls used. first: the + first three evidence items (explicit tailwise.evidence.*, else + RETRIEVER documents, else paragraphs of successful tool outputs), + each cut to 1500 characters; tool-call lines show only the status. + In the benchmark, first hid most tool results and the model flagged + what it could not check, so it ranked failures worse at about half + the cost. + evidence_budget: + type: integer + minimum: 1 + default: 12000 + description: >- + With evidence grounded, the most characters of tool results shown, + across all calls (about 4 characters per token). Ignored with + first. timeout: $ref: '#/$defs/duration' default: 30s @@ -558,3 +603,70 @@ properties: items: {$ref: '#/$defs/duration'} default: [1s, 5s, 30s] description: Wait before each retry; the last value repeats. Empty means the default. + + rules: + type: array + default: [] + description: >- + Heuristic rules every agent gets: certain, free checks that need no + model, such as "the final answer is valid JSON" or "no tool call + failed". An agent's manifest adds its own under "rules", in the same + form; one with the same name replaces the default, and one with + enabled false removes it for that agent. A fired rule is listed in the + response's rules, the log and the export (eval.rules). Rule: names are + unique; each rule sets exactly one of final_answer, tool_output, + tool_args and run. + items: + type: object + additionalProperties: false + required: [name] + properties: + name: + type: string + minLength: 1 + description: Identifies the rule in decisions and in a manifest's overrides. + on_fail: + enum: [keep, signal, ''] + default: signal + description: >- + keep: when the rule fires, the trace is kept without asking the + model (decision rule keep_rule, severity 1, p_keep null). Use it + for failures that are certain, such as broken JSON. signal: the + rule counts 1 in severity when it fires and 0 when it does not, + next to the model's answers, and does not decide keep or drop. + Every signal rule weighs as much as a check, so keep them few. + enabled: + type: boolean + description: false turns off the default rule of the same name, in a manifest. + reason: + type: string + description: Replaces the rule's own description of what the trace broke. + final_answer: + $ref: '#/$defs/ruleText' + description: What the final answer must be. + tool_output: + description: What every call's output must be, or every call to tool's. + allOf: [{$ref: '#/$defs/ruleText'}] + properties: + tool: {type: string, description: Only calls to this tool; empty is every call.} + ok_status: {type: boolean, description: Fires on a call whose status is not ok.} + tool_args: + type: object + additionalProperties: false + required: [json_schema] + description: What every call's arguments must be, or every call to tool's. + properties: + tool: {type: string, description: Only calls to this tool; empty is every call.} + json_schema: {type: object, description: A JSON Schema the arguments must be valid against.} + run: + type: object + additionalProperties: false + description: Limits on the run as a whole. + properties: + max_tool_calls: {type: integer, minimum: 0, description: Fires over this many tool calls.} + max_repeated_calls: + type: integer + minimum: 0 + description: Fires when one call, same tool and arguments, is made more than this many times. + requires_tools: {type: array, items: {type: string}, description: Fires when any is never called.} + forbids_tools: {type: array, items: {type: string}, description: Fires when any is called.} diff --git a/tailwise.yaml b/tailwise.yaml index b7df7d5..b280546 100644 --- a/tailwise.yaml +++ b/tailwise.yaml @@ -55,6 +55,9 @@ scorer: # manifest: manifests/my-agent.json decision_question: answer_wrong reason_question: why_wrong # a choice question; the option chosen is the plain-English reason + system_prompt: false # show the model the agent's system prompt (costs its tokens on every call) + evidence: grounded # how tool results reach the model: grounded (each call's result) or first (3 items) + evidence_budget: 12000 # grounded only: characters of tool results shown timeout: 30s # per call # Retries on the synchronous endpoint only. A queued trace makes one call # per delivery and the queue redelivers it (max_deliver, redelivery_backoff). diff --git a/testdata/golden/system_prompt.txt b/testdata/golden/system_prompt.txt new file mode 100644 index 0000000..b88f781 --- /dev/null +++ b/testdata/golden/system_prompt.txt @@ -0,0 +1,16 @@ +SYSTEM PROMPT: +You are a support agent for Acme. Only refund orders that have been delivered. +Never share another customer's details. + +USER QUESTION: +Please refund my order #1; it has not arrived yet. + +TOOL CALLS: +1. get_order({"order_id":"#1"}) -> ok +2. refund({"order_id":"#1"}) -> ok + +EVIDENCE: +[1] Order #1 status: pending. Not yet delivered. + +FINAL ANSWER: +Your refund for order #1 is done. \ No newline at end of file diff --git a/testdata/traces/system_prompt.json b/testdata/traces/system_prompt.json new file mode 100644 index 0000000..0f9a512 --- /dev/null +++ b/testdata/traces/system_prompt.json @@ -0,0 +1,13 @@ +{ + "trace_id": "system-prompt-001", + "system_prompt": "You are a support agent for Acme. Only refund orders that have been delivered.\nNever share another customer's details.", + "question": "Please refund my order #1; it has not arrived yet.", + "tool_calls": [ + {"name": "get_order", "args": {"order_id": "#1"}, "status": "ok"}, + {"name": "refund", "args": {"order_id": "#1"}, "status": "ok"} + ], + "evidence": [ + "Order #1 status: pending. Not yet delivered." + ], + "final_answer": "Your refund for order #1 is done." +}