Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
11 changes: 7 additions & 4 deletions docs/guide/configuration.md
Original file line number Diff line number Diff line change
Expand Up @@ -93,7 +93,8 @@ root model (top level or `args`) and on an inline `grader` / `extractor`, on
every dialect. A derived model (`base:`) uses its root's key.

With neither, the `openai_*` dialects read `OPENAI_API_KEY`; the others send
no key, so an external SGLang server started with `--api-key` needs one here.
no key, so an external SGLang or vLLM server started with `--api-key` needs
one here.

```yaml
models:
Expand Down Expand Up @@ -182,7 +183,7 @@ one entry per rollout, in `texts` order, with the same three fields.

| Dialect | `n: 1` | `n > 1` |
| --- | --- | --- |
| `sglang_native`, `openai_completions`, `openai_chat` | yes | yes |
| `sglang_native`, `vllm_native`, `openai_completions`, `openai_chat` | yes | yes |
| `google_genai` | yes | refused |
| `openai_responses` | yes | refused (no `n`) |
| `anthropic_messages` | refused | refused |
Expand All @@ -194,8 +195,10 @@ so changing them on a resume reruns inference.
With `echo` (`alogprobs(text, echo=True, logprobs=N)` or `infer_args: {echo:
true, top_logprobs: N}`), each input token's top-N lands in
`ModelOutput.input_top_logprobs` (`{}` for the first); `top_logprobs` keeps
the generated tokens'. Served by `sglang_native` and `openai_completions`.
The reply grows by N entries per input token.
the generated tokens'. Served by `sglang_native`, `vllm_native` and
`openai_completions`. The reply grows by N entries per input token.

`sglang_native` and `vllm_native` also fill token ids.

## Model Capabilities (`extra`)

Expand Down
16 changes: 16 additions & 0 deletions docs/guide/infer.md
Original file line number Diff line number Diff line change
Expand Up @@ -257,6 +257,22 @@ task scores input: SGLang's `/v1/completions` cannot. `/generate` has no
per-request `seed`, so an explicit one is refused; a managed engine pins
`--random-seed` instead.

A vLLM `gen` model keeps `openai_completions` unless it sets `dialect:
vllm_native`, which posts to the same `/v1/completions` with vLLM's own
fields (measured on 0.26.0):

| Need | `openai_completions` | `vllm_native` |
| --- | --- | --- |
| Input scoring | `echo`, split at `usage.prompt_tokens` | `prompt_logprobs`, checked against `prompt_token_ids` |
| `texts` of a scoring call | prompt + generation | generation only |
| Token ids | none | input and sampled (`return_token_ids`) |
| Structured output | refused | `structured_outputs` |
| Unknown options | `extra_body` | refused, except `min_p`, `repetition_penalty`, `stop_token_ids` |

It sends one non-streaming POST and offers no reasoning or tool parsers
(they act on chat requests). `suffix` and `max_tokens: 0` are refused before
I/O. Input top-k needs `--max-logprobs` at least the breadth.

### Model lifecycles

A model `sieval run` launches may bound its engine by stage barriers
Expand Down
21 changes: 18 additions & 3 deletions sieval/core/models/dialect_registry.py
Original file line number Diff line number Diff line change
Expand Up @@ -54,6 +54,10 @@
CAPABILITY_DECISIONS as SGLANG_NATIVE_CAPABILITY_DECISIONS,
)
from .dialects.sglang_native import SglangNativeDialect
from .dialects.vllm_native import (
CAPABILITY_DECISIONS as VLLM_NATIVE_CAPABILITY_DECISIONS,
)
from .dialects.vllm_native import VllmNativeDialect
from .ir import (
ChatInput,
ChatMessage,
Expand Down Expand Up @@ -378,6 +382,10 @@ def _bind_sglang_native(connection: Any, requested_model_id: str) -> Dialect:
return SglangNativeDialect(connection, requested_model_id)


def _bind_vllm_native(connection: Any, requested_model_id: str) -> Dialect:
return VllmNativeDialect(connection, requested_model_id)


def _binder(
function: Callable[[Any, str], Dialect],
decisions: Mapping[str, DialectCapabilityDecision],
Expand Down Expand Up @@ -433,6 +441,13 @@ def _binder(
SGLANG_NATIVE_CAPABILITY_DECISIONS,
),
),
"vllm_native": _binder(
_bind_vllm_native,
cast(
Mapping[str, DialectCapabilityDecision],
VLLM_NATIVE_CAPABILITY_DECISIONS,
),
),
}
)

Expand Down Expand Up @@ -504,7 +519,9 @@ def _registered_decisions(
"vllm_native": _spec(
"vllm_native",
"async_http_json",
request_seed_support=RequestSeedSupport.RESERVED,
decisions=_registered_decisions("vllm_native"),
# Measured on 0.26.0: equal seeds give equal samples at temperature 1.
request_seed_support=RequestSeedSupport.SUPPORTED,
input_kinds=("completion",),
input_modalities=("text",),
),
Expand Down Expand Up @@ -628,8 +645,6 @@ def bind_dialect(


def _not_implemented_message(dialect_id: str) -> str:
if dialect_id == "vllm_native":
return "dialect 'vllm_native' is reserved and has no executable binder"
return f"dialect {dialect_id!r} is reserved and has no executable binder"


Expand Down
Loading
Loading