Loop-level continuous batching for recurrent language models.
Documentation · Getting Started · Supported Models · Developer Guide · Releases · Community · CDB Paper
- 2026/11 — v0.2.0: core runtime refactoring around Stage and Loop, wavefront chunked prefill (WCPB), and experimental token-level decode pipelining. See the changelog for model additions, migration notes, and validation scope.
vllm-rlt is a standalone inference and serving engine for recurrent language models. It schedules individual recurrent loops, allowing requests at different depths to share a batch instead of waiting for an entire cohort to finish. Its engine organization is inspired by vLLM, but it runs independently and does not require vLLM to be installed.
- Stage/Loop architecture. A stage owns deployment and resources; a loop is a stage-local execution and scheduling unit. Scheduler selects work and applies results; Executor submits execution. Prefill and decode share this runtime structure.
- WCPB prefill. Wavefront chunked prefill exposes loop boundaries so prompt chunks can advance at different recurrent depths. Compatible non-speculative prefill and decode work can share a batch.
- Decode pipeline. Experimental token-level self-speculation advances different tokens at different loop depths while retaining hidden states. Asynchronous execution separates device progress from CPU output delivery.
Original conceptual animation, not a benchmark or a claim that every mode can be combined. Static illustration · Animation source · Architecture · WCPB · Decode pipeline.
Prefill and decode provide complementary pipelined execution paths, connected through separate PD stages and KV transfer in supported configurations. PD is currently same-host CUDA with supported Triton/FA4 paths; it is not a claim of universal multi-node or model support. Speculative steps are not included in mixed prefill/decode batches.
- Depth-aware paged KV cache: LAST_EXITED and SHARED layouts, with optional prefix reuse, incremental allocation, priorities and preemption.
- Adaptive computation: fixed-depth inference or Ouro's trained early-exit gate, with per-request loop bounds. Changing exit policy can change quality.
- Configurable execution: Torch reference execution, GPU attention backends, CUDA Graphs, and optional asynchronous/speculative paths. An optional native CUDA SiLU-and-Mul operator accelerates the Ouro activation path, with a Torch fallback; it is not a general model-speedup claim.
- Offline and online use: Python API, CLI, and a streaming OpenAI-compatible completions endpoint. This is not a chat-completions API.
Important
v0.2.0 is a research/evaluation milestone. Default execution is synchronous; advanced modes are opt-in and have model-specific constraints. Token-pipeline speculation is experimental. Historical serving measurements have unresolved cross-configuration output-equivalence differences, so they are diagnostic, not equal-quality speedup or production-readiness claims.
Requires Python 3.10+ and PyTorch 2.5+. For GPU inference, use Linux and an NVIDIA GPU with a compatible CUDA-enabled PyTorch installation.
# Activate your Python environment first.
git clone --branch v0.2.0 --depth 1 https://github.com/InferMatrixLab/vllm-rlt.git
cd vllm-rlt
uv pip install -e .
vllm-rlt-serve --helpThis installs text, serving and Linux Triton dependencies. Standard isolated builds use the Torch activation fallback; optional native CUDA compilation requires a compatible toolchain. Model weights are downloaded separately. No PyPI publication is implied.
- Installation and first request
- Python API and CLI
- HTTP serving and streaming — use a trusted network or secure gateway; the server has no built-in authentication or TLS.
- Supported models and Ouro-Thinking recipe
- Configuration and compatibility
- Native CUDA build
- Migration from v0.1.0
| Checkpoint | Implementation | Documented validation scope | Guide |
|---|---|---|---|
| Ouro-1.4B | Native Ouro | Existing runtime and benchmark coverage under their recorded configurations | Runtime, accuracy |
| Ouro-2.6B | Shared Ouro | Shares the Ouro implementation; not separately qualified, and 1.4B quality/performance results are not inherited | Scope notes |
| Ouro-1.4B-Thinking | Reuses Ouro | Bounded BF16, fixed-four-loop synchronous basic integration | Shared recipe |
| Ouro-2.6B-Thinking | Reuses Ouro | Bounded BF16, fixed-four-loop synchronous basic integration | Shared recipe |
| Nanbeige4.2-3B | Native Nanbeige | Fixed two-loop inference | Runtime |
| Huginn-0125 | Native Huginn | Synchronous fixed-depth, LAST_EXITED KV, CUDA Graph decode core and coda | Runtime |
Shared implementation does not qualify every checkpoint/feature combination; see the linked guides for validation scope and limitations. See RFC #69 for the broader roadmap.
Huginn uses native checkpoint loading and depth-aware KV for its recurrent core, with separate prelude and coda KV. It supports synchronous fixed-depth inference and CUDA Graphs for the recurrent core and decode coda.
Huginn does not currently support WCPB/mixed staged prefill, asynchronous execution, self-speculation, or PD. Feature availability is not equivalent to validation of every model/backend/workload combination.
Start with the documentation index, then explore:
| Topic | Guide |
|---|---|
| Runtime and process boundaries | Unified architecture |
| Chunk/depth scheduling | WCPB prefill |
| Asynchronous block execution and token pipelines | Self-speculation |
| Cache ownership and lifetime | KV walkthrough |
| Evaluation and measurement | Accuracy, profiling |
Performance evidence: the October 2026 serving report records Ouro-1.4B BF16 on B300/FA4, workloads, one- versus two-GPU configurations, metric definitions and limitations. Compare matched hardware, workload and quality; do not interpret unequal resource budgets or differing output lengths as isolated runtime speedups.
For ongoing design work, see the architecture RFC and model roadmap. Roadmap proposals are not released capabilities.
We welcome systems, model, evaluation and documentation contributions. Read CONTRIBUTING.md and the Developer Must-Read before implementation. Include reproducible configurations, correctness evidence and the scope of any measurements. For larger changes, start with a design issue.
For the CDB method that informs this project, please cite the original paper: Depth-adaptive Inference of Looped Language Models via Continuous Depth Batching (PDF).
@misc{schwethelm2026continuousdepthbatching,
title = {Depth-adaptive Inference of Looped Language Models via Continuous Depth Batching},
author = {Kristian Schwethelm and Daniel R\"{u}ckert and Georgios Kaissis},
year = {2026},
eprint = {2608.09444},
archivePrefix = {arXiv},
primaryClass = {cs.LG},
url = {https://arxiv.org/abs/2608.09444}
}The paper's measurements are separate from the vllm-rlt performance baselines linked above.
Ask questions, report reproducible problems or discuss designs in GitHub issues. You can also join the existing vllm-rlt developer group through WeChat:
vllm-rlt builds on the published Ouro architecture and Continuous Depth Batching. Its engine and Python API organization are inspired by vLLM. See NOTICE for upstream model attribution.
Apache License 2.0. Model checkpoints retain their own licenses and usage conditions.