Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
19 changes: 19 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,24 @@ All notable changes to this project will be documented in this file.
The format is based on [Keep a Changelog](https://keepachangelog.com/),
and this project adheres to [Semantic Versioning](https://semver.org/).

## [0.13.1] - 2026-10-25

A patch release: a `max_tokens`-derived context need reserves room for the
prompt, without moving a task that fits under the native window into a scaled
group. No on-disk format changes: `meta.json` `identity` stays schema 4 and
0.13.0 result directories resume, except where the served window changed
(below).

### Migrating from 0.13.0

- **A task's served window changes when its need comes from `max_tokens`, so its resume is refused.** The window is part of the inference identity, and a resume never reruns inference. On a model `sieval run` launches with a known native window, a `max_tokens` below `native` that 0.13.0 served at exactly `max_tokens`, or at the recipe window with fewer than 4096 tokens to spare, is now served at `native`. The other tasks on that model's default plan share the window, so theirs changes too. A `max_tokens` at or above `native` is served in scaled group `<model>@<max_tokens + 4096>`, where 0.13.0 served it at `max_tokens` (in `<model>@<max_tokens>`, or unscaled when the recipe window held it). Start such a run fresh, or pin the recorded window with `infer.overrides` (`context_length` / `max_model_len`). A model whose tasks all fit the recipe window with 4096 to spare, or all set `min_context` or declare `TaskRequirements.min_context`, keeps its window (#239, #240).
- **Where 4096 more no longer fits, the task is refused before launch**: a `max_tokens` above `context_limit - 4096` on an endpoint, above the plan's window minus 4096 when the native window is unknown, or above `context.max - 4096` when scaled. Set the task's `min_context` if its prompts are known to be shorter (#239).

### Fixed

- **A raised serving window leaves room for the prompt.** 0.13.0 raised a plan's window to exactly a task's `max_tokens`, so every request with a non-empty prompt was refused as longer than the context. A need derived from `max_tokens` now adds 4096 tokens for the prompt; `min_context` and a task's declared need already count it and are unchanged. A task with longer prompts sets `min_context` (#239, closes #238).
- **A `max_tokens` below the native window is served at that window, never scaled.** With the 4096 added, a task such as `max_tokens: 38000` on a model with `native: 40960` would move into a scaled (YaRN) group, and scaling changes outputs. Such a task now raises the window to `native`; its prompt gets `native - max_tokens` tokens, and a longer prompt is that sample's context overflow, as before. A `max_tokens` at or above `native` still gets `max_tokens + 4096`, in a scaled group when `context.max` and `scaling` allow (#240, refs #238).

## [0.13.0] - 2026-10-25

A minor bump: `meta.json` `identity` moves to schema 4, where `inference`
Expand Down Expand Up @@ -522,6 +540,7 @@ Mainstream benchmarks registered in `sieval/meta/index.json`:
- Project-wide preflight (`scripts/check_preflight.py`): links, deps, tasks, datasets, imports, examples, meta-index sync, version.
- Tooling: `ruff`, `ty`, `mypy strict`, `pytest`.

[0.13.1]: https://github.com/scitix/sieval/compare/v0.13.0...v0.13.1
[0.13.0]: https://github.com/scitix/sieval/compare/v0.12.0...v0.13.0
[0.12.0]: https://github.com/scitix/sieval/compare/v0.11.0...v0.12.0
[0.11.0]: https://github.com/scitix/sieval/compare/v0.10.2...v0.11.0
Expand Down
4 changes: 2 additions & 2 deletions Dockerfile
Original file line number Diff line number Diff line change
Expand Up @@ -15,8 +15,8 @@ RUN python3 -m nltk.downloader \

WORKDIR /app

COPY ./dist/sieval-0.13.0-py3-none-any.whl /tmp/
RUN pip install /tmp/sieval-0.13.0-py3-none-any.whl && rm /tmp/sieval-0.13.0-py3-none-any.whl
COPY ./dist/sieval-0.13.1-py3-none-any.whl /tmp/
RUN pip install /tmp/sieval-0.13.1-py3-none-any.whl && rm /tmp/sieval-0.13.1-py3-none-any.whl

COPY vendor /app/vendor
# The wheel carries no vendor/; this is the code-evaluator SiEval starts.
Expand Down
Loading