[ethan-scitix]

Type

• feature — new benchmark, task, or capability

Summary

Implements vllm_native, the seventh dialect in RFC #25's symbol space and the last one still reserved.

vLLM serves no text /generate. Its native surface is the extension fields its /v1/completions accepts beyond OpenAI's, so vllm_native posts there over raw async_http_json. The OpenAI SDK would move these fields into extra_body and type the reply without them. They are sent as first-class wire:

• Native prompt scoring. prompt_logprobs is its own channel: no echo, and no split at a usage boundary. Completeness is checked against both prompt_token_ids and usage.prompt_tokens, as RFC [RFC]: A capability-based Model IR — decouple model access from the OpenAI protocol #25 §8.2 asks. A short or missing channel is refused.
• Token ids. return_token_ids fills TokenLogprob.token_id on input and sampled tokens, and TopKEntry.token_id on the input top-k. It also identifies the scored token among its top-k: a position lists k entries, or k + 1 with the scored token beside them, and any other count is refused.
• Structured decoding. structured_output is lowered to structured_outputs (json schema or json_object) and lifted to Response.structured_output.
• Sampled top-k. vLLM keys it by decoded text, so tokens that decode alike (byte-fallback pieces) collapse. A map short of k, or one holding another value for the sampled token, is refused instead of misread. A collision that still leaves k entries is undetectable (the map carries no rank).
• Seed: SUPPORTED. Measured: equal seeds gave equal samples at temperature 1, and a different seed gave a different one.
• Parser constraints: Unsupported. vLLM's reasoning and tool parsers act on chat requests only, and this endpoint takes a raw prompt.

Every active request leaf is decided before I/O:

Leaf	Decision
prompt, sampling (incl. seed, n), scoring, structured_output.format / .schema	consumed with wire evidence
scheduling.stream	no-op: one non-streaming POST, because streamed choices carry no prompt_logprobs
structured_output.name	no-op: structured_outputs takes no name
structured_output.strict	no-op when true; false refused (the grammar is always strict)
max_tokens: 0, suffix	refused (vLLM 400s on both)
options min_p, repetition_penalty, stop_token_ids	forwarded as top-level fields
any other option, or one naming an IR leaf	refused: vLLM silently ignores an unknown field

Extension-cost contract (#256). The diff stays inside it. Shared code changes only in dialect_registry.py: one import, one DIALECT_BINDERS entry, and the vllm_native DIALECT_SPECS row moved from RESERVED to active with its decisions. There is no IR, registry-schema, reconcile(), connection-family or existing-dialect change. session.py and resolution.py bind the new id with no edits.

Not changed: no default dialect moves. A vLLM gen model keeps openai_completions unless it sets dialect: vllm_native. This is additive: not in any identity, and no RESULT_SCHEMA_VERSION bump.

Related Issues

Closes #256. Refs #25 (§3.1, §3.2, §3.3, §8.2) and #250, whose sglang_native parity run this follows.

Test Plan

Automated

• ruff check, ruff format --check, bare ty check (tests included): clean
• pytest tests/unit tests/integration --cov --cov-fail-under=95 -n 8: 10174 passed, 1 skipped, total coverage 97.17%. vllm_native.py 98%, dialect_registry.py 97% (at df5c641)
• After the review fix (b492239): pytest tests/unit tests/integration -n 8: 10179 passed, 1 skipped. The 5 new top-k tests fail on df5c641
• scripts/check_preflight.py: all pass. sync_package_stubs.py --check: clean
• New tests/unit/core/models/dialects/test_vllm_native.py (86 tests). Its reply fixtures are vLLM 0.26.0 shapes recorded live, e.g. a scored token ranked 522 listed beside the top 2
• Reserved-path tests (test_dialect_registry, test_validation) now hide vllm_native's binder, so "reserved ⇒ DialectNotImplemented" stays covered with no reserved id left. test_reconcile's unavailable-dialect case uses sglang_legacy

Manual

• Live endpoint (§3.3). 1× H100, image vllm:0.26.0 (/version → 0.26.0, system_fingerprint vllm-0.26.0-0c752b4e), Qwen/Qwen2.5-0.5B, --max-logprobs 100 --seed 0 --no-enable-prefix-caching, behind a logging proxy. Requested model id Qwen/Qwen2.5-0.5B; reported Qwen/Qwen2.5-0.5B (Response.response_model).

• Parity against the OpenAI-compatible route on the input-scoring tasks, concurrency_limit: 1, on one server: openai_completions, then vllm_native, then openai_completions again as a control.

  Task	n	openai_completions	vllm_native	control
  arc_easy_kshot_ppl	30	60.0	60.0	60.0
  hellaswag_kshot_ppl (acc_norm / acc)	20	40.0 / 35.0	40.0 / 35.0	40.0 / 35.0
  mmlu_kshot_clp	60	38.33	38.33	38.33
  gsm8k_kshot_base_gen	15	33.33	33.33	33.33

  Zero fails anywhere. Per call, over 395 calls: 160,545 input-token logprobs compared, max |diff| = 0, with input token text equal at every position. That includes the unscored first token, whose text vllm_native recovers as the prompt prefix the decoded tokens leave. Under openai_completions the input's last token and the generated token both come from the echo. The only record difference is texts on the PPL calls: echo returns prompt + generation, while vllm_native returns the generation only. No PPL/CLP task reads it.

• Wire, redacted. 395 vs 395 requests, all /v1/completions, all 200. Native body shapes:

  • PPL: {model, prompt, stream: false, temperature, max_tokens: 1, return_token_ids: true, prompt_logprobs: 0} → choice has prompt_logprobs (809 positions = usage.prompt_tokens 809) and prompt_token_ids.
  • CLP: {…, return_token_ids: true, logprobs: 100}.
  • Generation: {…, max_tokens: 256, stop: [...]}.

  The OpenAI route sent echo: true, logprobs: 0, stream: true for the same calls.

• Supported cases, live: input top-3 with token ids; CJK input (first token 中国的 recovered); n=2 with per-choice token ids; seed repeat (identical text); json_schema → {"name": "jazznix"}; options min_p + repetition_penalty.

• Refused, live:

  • Before I/O: max_tokens: 0, suffix (plan: fim unavailable), unknown option guided_json, strict: false.
  • By the server, carrying vLLM's message: top-k 101 > --max-logprobs; context overflow (read by context_overflow_message).
  • Output contract: structured output is not valid JSON on a reply cut at max_tokens.

• Prefix caching on (vLLM's default), same probes. Input scoring stays complete; a repeated 363-token prompt kept 363/363 positions. That needs no --no-enable-prefix-caching remedy on 0.26.0, unlike the V1 behaviour RFC [RFC]: A capability-based Model IR — decouple model access from the OpenAI protocol #25 §8.2 guarded against. The length check stays as a safeguard.

• Raw wire probe (no sieval) before the implementation:

  • prompt_logprobs with return_token_ids returns prompt_token_ids.
  • prompt_logprobs and max_tokens: 0 → 400 max_tokens must be at least 1.
  • suffix → 400 suffix is not currently supported.
  • An unknown field → 200, silently ignored.
  • stream: true → no prompt_logprobs in the stream.

Checklist

Required (all PRs)

• PR title follows conventional format (type(scope): description)
• No internal paths, credentials, or personal info in committed files
• AI-generated code has AI-Generated Code - <model> (<provider>) in module docstring
• No new upper-layer dependencies added to core/
• Deleted code verified — no remaining call sites depend on it (none deleted)

🤖 Generated with Claude Code