[ROCm] Encoder prefill attention on gfx1151: split head-dim block and a narrower KV tile

.agents/skills/model-microbenchmark/SKILL.md

@@ -0,0 +1,115 @@
+---
+name: model-microbenchmark
+description: Time a vLLM model's forward pass on exact batch shapes (decode, prefill, chunked prefill, prefix hit, spec-decode verify, or mixes) with `vllm bench forward`, single-GPU or DP/TP/EP across nodes. Use for per-step model latency, layout/parallelism comparisons, and checking a model-level optimization. Not for single kernels (use kernel-microbenchmark) or end-to-end serving (use `vllm bench serve`).
+---
+
+# Model Microbenchmark (`vllm bench forward`)
+
+Builds the batch without the scheduler, runs it through MRV2's real
+`execute_model`, and times the forward as a one-shot CUDA graph (prefill
+included). Sampling, the speculator, and CPU step overhead are excluded, so
+expect results somewhat below a serving step.
+
+## Usage
+
+```bash
+vllm bench forward --model deepseek-ai/DeepSeek-V4-Flash \
+  --case decode=64*1@8192 --case prefill=8192 --case mix=4096,32*1@8192
+
+# DP/TP/EP: one process per GPU (multi-node: add --nnodes/--node-rank/--master-addr)
+torchrun --nproc-per-node=4 -m vllm.entrypoints.cli.main bench forward \
+  --model ... -dp 4 --enable-expert-parallel --case decode=128*1@100000
+```
+
+Requests: `[N*]Q[@K][c][:D]`, comma-separated, per DP rank.
+
+| Form | Meaning |
+| --- | --- |
+| `8192` | fresh prefill, 8192 tokens |
+| `32*1@8192` | 32 decodes, KV length 8192 (KV length **includes** the query) |
+| `4@4096:3` | spec-decode verify, 3 draft tokens |
+| `2048@10240` | prefill after an 8192-token prefix-cache hit |
+| `2048@10240c` | chunked prefill: the prefix came from earlier chunks |
+
+Verify cases require `--speculative-config` with `num_speculative_tokens`
+at least as large as the case's draft count; insufficient capacity is rejected.
+
+Hit vs chunk only differs on models with SWA bounded replay (DSV4.1): a hit
+also recomputes the last window (128 tokens) of the prefix.
+
+Or `--shapes file.json|yaml`:
+`{"cases": [{"name": ..., "requests": [{"query_len", "kv_len", "num_draft_tokens", "chunked", "count"}]}]}`.
+
+Key flags (plus all `EngineArgs`):
+
+- `--kv-len-jitter 0.5` (default): prefixes drawn within ±50%, same on every
+  DP rank, capped at `max_model_len`. Use `0` for exact lengths.
+- `--routing auto|real|balanced`: `auto` = balanced with dummy weights.
+- `--iters 20 --samples 5`, `--output-json out.json` (per-rank latency stats,
+  `samples_us`, graph-vs-eager check).
+- Defaults: `--load-format dummy`, prefix caching off, no torch.compile.
+
+Read `median_us` (each replay is already the max over ranks).
+
+## Kernel breakdown
+
+Add `--profile-dir DIR` to see where the time goes. After timing, each case is
+replayed 3 more times under `torch.profiler`; every rank writes
+`DIR/<case>.rank<r>.kernels.json` (GPU time per kernel per forward, sorted)
+and `DIR/<case>.rank<r>.trace.json.gz` (open in Perfetto). Rank 0's top
+kernels are printed and included in `--output-json`.
+
+```bash
+python -m vllm.benchmarks.forward.profile DIR/decode.rank0.kernels.json
+# Before/after a change, same case: per-kernel deltas, largest first.
+python -m vllm.benchmarks.forward.profile base/decode.rank0.kernels.json new/decode.rank0.kernels.json
+```
+
+- Kernels are grouped by name without template or function arguments; the
+  JSON keeps one full name per group in `example`.
+- Only the timed forward is included: the in-graph KV restore and rank
+  rendezvous are excluded (`pre_forward_us`). `span_us` (first to last
+  forward kernel) should be close to `median_us`, a bit higher from profiler
+  overhead; `busy_us` can exceed it when streams overlap.
+- Compare the same rank across runs: under EP, ranks differ (e.g. collective
+  wait time shows up inside communication kernels).
+
+For Nsight Systems (NVTX, NCCL, CPU side, GPU metrics), skip `--profile-dir`:
+the two can't share CUPTI. Each case's 3 profiled replays run between
+`cudaProfilerStart`/`Stop` in an NVTX range `forward/<case>`, so a capture range
+records only them, one report per case (and rank):
+
+```bash
+nsys profile --capture-range=cudaProfilerApi --capture-range-end=repeat \
+  --cuda-graph-trace=node -o prof vllm bench forward ...       # prof.1, prof.2, ...
+torchrun --nproc-per-node=4 --no-python nsys profile ... -o prof_rank%q{RANK} \
+  python -m vllm.entrypoints.cli.main bench forward ...
+nsys stats --report cuda_gpu_kern_sum --format csv -o kern prof.1.nsys-rep
+```
+
+Keep `--capture-range-end=repeat`: nsys's default stops after the first case
+and leaves the bench hanging. `nsys stats` also counts the KV restore copies
+that the JSON summary excludes.
+
+## Caveats
+
+- **Dummy weights lie for data-dependent kernels.** Dummy gates route every
+  token to the same experts (hence balanced routing). Cross-check suspicious
+  kernels with real weights.
+- **torch.compile models (e.g. Qwen3/3.5) look slow**: no compile here. Use
+  them for correctness only; trust perf for models under `vllm/models/`.
+- `sample spread > 0.5%` warns of unstable timing. DSV4.1 DEP prefill is
+  inherently ~0.5–1% (MegaMoE cross-rank sync, host engram reads).
+- `graph_vs_eager` may be nonzero for small batches where models branch on
+  `is_current_stream_capturing()`; that's expected, not a bug.
+- KV cache is zero-initialized (`--kv-init zeros`); random KV was within 3%.
+- Uniform DP only: every rank gets the same case; no idle ranks.
+- CUDA only: it hangs on ROCm (untested cause: likely the in-graph timing
+  events).
+- Large prefills/prefix hits need room: raise `--max-num-batched-tokens` /
+  `--max-model-len` / `--gpu-memory-utilization` if build fails on KV blocks.
+- Startup cost is JIT/autotune, not the bench: install FlashInfer jit-cache
+  and cubin; DeepGEMM warmup is skipped (each case's eager pass compiles its
+  own shapes).
+- To compare against serving decode: `vllm bench latency` with 65- vs
+  1-token outputs, divided by 64, with a FULL decode graph.

.agents/skills/model-recipes/SKILL.md

@@ -0,0 +1,157 @@
+---
+name: model-recipes
+description: Find vLLM deployment recipes and configs for LLMs and Hugging Face models by model and hardware. Use for recipe lookup, creation, and updates.
+---
+
+# Model Recipes
+
+Model serving recipes are maintained in this repository's
+[recipes/models/ directory](../../../recipes/models). Omni (image, video,
+audio) recipes stay in vllm-project/recipes for now (moving to vllm-omni),
+and Ascend NPU content is inline in these recipes (moving to vllm-ascend
+overlays). The published catalog, rendered by
+[vllm-project/recipes](https://github.com/vllm-project/recipes), is at
+[recipes.vllm.ai](https://recipes.vllm.ai).
+
+## Usage notes
+
+1. Recipe lookup returns commands and deployment guidance without launching
+   `vllm serve`.
+2. Use published configurations or the recipes repository's command generator
+   for supported selections. Return commands as published, including their
+   flags and parallelism settings.
+3. Recipes are starting points for deployment and may not be tuned for your
+   workload. Check each recipe for its verified configurations, and use
+   benchmarks of your workload to guide further tuning.
+4. To create or update a recipe, follow the
+   [authoring workflow below](#create-or-update-a-recipe).
+
+## Find a recipe
+
+Read recipes from the published JSON on recipes.vllm.ai. It is generated from
+the source YAML with variant, hardware, and strategy settings already
+resolved; the YAML contains comments and conditional overrides, so do not use
+it for lookup. When presenting results, link the website page. Read the source
+YAML only when creating or updating a recipe.
+
+Each recipe has a page at `https://recipes.vllm.ai/<org>/<repo>` and JSON at
+the same path plus `.json`. Paths use the full Hugging Face ID, preserving
+case; do not substitute `meta.slug` or a display title. For example,
+`Qwen/Qwen3.8-27B` maps to
+[recipes.vllm.ai/Qwen/Qwen3.8-27B](https://recipes.vllm.ai/Qwen/Qwen3.8-27B).
+
+Recipe JSON is often over 20 KB, mostly the `guide`. Instead of printing it,
+read only what the request needs: each step below returns the paths for the
+next, so stop once the request is answered. Commands need `curl` and `jq`.
+[scripts/read_recipe.sh](scripts/read_recipe.sh) is in this skill's directory;
+the examples use its path from the vLLM repository root.
+
+### Search the catalog
+
+Skip this step if the exact HF ID is known. Otherwise search by model ID,
+family, or provider using case-insensitive keywords. Each
+whitespace-separated word must occur in the ID, title, or provider:
+
+```bash
+curl -fsSL https://recipes.vllm.ai/models.json |
+  jq --arg q 'qwen 27b' '
+    ($q | ascii_downcase | [scan("\\S+")]) as $words | .[] |
+    ([.hf_id, .title, .provider] | join(" ") | ascii_downcase) as $text |
+    select(all($words[]; . as $word | $text | contains($word))) |
+    {hf_id, json, url, derived_from}'
+```
+
+`json` is the recipe data path; `url` is its page path. Prefix both with
+`https://recipes.vllm.ai`. A `derived_from` entry is a variant of a parent
+recipe: fetch its own `json` to get that checkpoint's command. Its page may
+belong to the parent, so link the returned `url` rather than assuming every
+checkpoint has a separate page.
+
+If there is no exact match, search the family and inspect `variants` before
+concluding the recipe is missing.
+
+### Summarize the recipe
+
+Save the recipe JSON to a file and summarize it:
+
+```bash
+read_recipe=.agents/skills/model-recipes/scripts/read_recipe.sh
+recipe=$(mktemp)
+curl -fsSL https://recipes.vllm.ai/Qwen/Qwen3.8-27B.json -o "$recipe"
+"$read_recipe" "$recipe"
+```
+
+`default` is the recommended hardware, strategy, and variant; `hardware` maps
+each hardware ID to its command path; `recipe_verified_hardware` lists hardware
+where the recipe as a whole was verified end to end. For capability
+requirements, such as tool calling or text-to-video, check `features` and
+`omni_tasks`, then confirm in the guide.
+Some recipes, including Omni, lack `hardware`; skip hardware selection and
+use the guide's launch and task-specific sections.
+
+### Select hardware and strategy
+
+Match a hardware ID to the user's GPU and fetch its path from `hardware`,
+prefixed with `https://recipes.vllm.ai`, rather than constructing a URL. That
+response is the command object itself: single-node objects carry `command`,
+and multi-node objects carry `head_command` and `worker_commands` instead, so
+list its keys before selecting fields.
+Check `hardware_profile` to confirm that the selected configuration matches
+the user's GPU model and specifications. Its `gpu_count` describes the profile,
+not necessarily the number of GPUs used by the command. Check the command's
+parallelism settings, `node_count`, and all `worker_commands` against the
+user's available topology, and compare the variant's `vram_minimum_gb`, an
+estimate, with the total memory of the GPUs the command uses; a configuration
+below that minimum is not a working option unless the guide says it was
+tested. Ask when the user's hardware is unclear. If no
+published command or guide command fits, report the mismatch and return the
+closest one unchanged; list any adjustments the user's setup would need
+separately and mark them unverified instead of editing the command.
+
+For another strategy, follow `alternatives` in the same response. These are
+static files; query parameters do not change their configuration. For other
+feature/topology selections, use the recipes repository's command synthesis.
+
+### Read deployment guidance
+
+Read the guide sections that apply to the selection, such as prerequisites,
+launch notes for the selected hardware or variant, troubleshooting, and known
+limitations. Pass entries from `guide_sections` or prefixes of their text.
+The script prints every matching section with its subsections and fails when
+a section matches nothing:
+
+```bash
+"$read_recipe" "$recipe" prerequisites troubleshooting
+```
+
+Query other fields with `jq` on the saved file, such as `features.<name>`,
+`hardware_overrides`, or `omni.tasks` for request examples.
+
+### Return the result
+
+Return the website link, selected checkpoint and hardware, the requested
+command, and relevant prerequisites and caveats from the guide. Pick one
+command and copy it, including its Docker form, exactly as the command object
+or guide section gives it; do not combine commands from different sections or
+add, drop, or reorder flags. List optional changes separately.
+
+For each command, state its source, either a generated hardware or strategy
+configuration or a named guide section, and whether that source says it was
+tested. A `recipe_verified_hardware` entry covers the recipe on that
+hardware, not every variant, strategy, or guide command, so it does not make a
+command verified. Generated configurations are not automatically verified.
+
+## Create or update a recipe
+
+Recipes cover official or widely used checkpoints, with configurations verified
+end to end on real hardware. Do not add recipes for arbitrary community
+uploads, untested hardware claims, or non-standard flags that only work for one
+setup. Before opening a pull request, confirm that the user ran the
+configuration and can share the results.
+
+Edit the YAML under [recipes/models/](../../../recipes/models) in this
+repository, following the recipes repository's
+[add-recipe skill](https://github.com/vllm-project/recipes/blob/main/.agents/skills/add-recipe/SKILL.md)
+for the schema. Validate and preview it from a checkout of
+`vllm-project/recipes` pointed at this directory, as
+[recipes/README.md](../../../recipes/README.md) describes.

.agents/skills/model-recipes/agents/openai.yaml

@@ -0,0 +1,4 @@
+interface:
+  display_name: "Model Recipes"
+  short_description: "Find vLLM deployment recipes and configs for LLMs and Hugging Face models by model and hardware."
+  default_prompt: "Use $model-recipes to find a recipe for my model and hardware, and return its commands and deployment caveats."

.agents/skills/model-recipes/scripts/read_recipe.sh

@@ -0,0 +1,66 @@
+#!/bin/bash
+# Usage: read_recipe.sh <recipe.json> [guide-section...]
+# Without sections, print a compact summary of a recipes.vllm.ai recipe JSON.
+# With sections, print each guide section whose heading starts with one of
+# them (case-insensitive), including its subsections.
+
+set -euo pipefail
+
+if [ $# -lt 1 ]; then
+    echo "Usage: $0 <recipe.json> [guide-section...]" >&2
+    exit 1
+fi
+
+# Print "<level>\t<heading text>\t<line>" for guide lines, marking headings
+# outside code fences with their level and other lines with 0.
+guide_lines() {
+    jq -r '.guide // ""' "$1" | awk '
+        /^[[:space:]]*(```|~~~)/ { fence = !fence }
+        !fence && match($0, /^#+ /) {
+            print RLENGTH - 1 "\t" substr($0, RLENGTH + 1) "\t" $0
+            next
+        }
+        { print "0\t\t" $0 }'
+}
+
+if [ $# -gt 1 ]; then
+    recipe=$1
+    shift
+    status=0
+    for section in "$@"; do
+        guide_lines "$recipe" | awk -F '\t' -v h="$section" '
+            BEGIN { h = tolower(h); sub(/^#+[[:space:]]*/, "", h) }
+            $1 > 0 && lvl && $1 <= lvl { lvl = 0 }
+            $1 > 1 && !lvl && index(tolower($2), h) == 1 { lvl = $1; found = 1 }
+            lvl { sub(/^[^\t]*\t[^\t]*\t/, ""); print }
+            END {
+                if (!found) {
+                    print "No guide section starts with: " h > "/dev/stderr"
+                    exit 1
+                }
+            }' || status=1
+    done
+    exit $status
+fi
+
+sections=$(guide_lines "$1" | awk -F '\t' '$1 == 2 || $1 == 3 { print $3 }' |
+    jq -R . | jq -s .)
+
+jq --argjson sections "$sections" '{
+    hf_id,
+    title: .meta.title,
+    tasks: .meta.tasks,
+    recipe_verified_hardware: .meta.hardware,
+    omni_tasks: (.omni.tasks // null |
+        if . then map(objects |= {id, label, endpoint, model_id, description})
+        else null end),
+    variants: (.variants // {} | map_values(
+        {model_id, precision, vram_minimum_gb, supported_hardware} |
+        with_entries(select(.value != null)))),
+    features: (.features // {} | keys),
+    opt_in_features,
+    default: (.recommended_command // null |
+        if . then {hardware, strategy, variant} else null end),
+    hardware: .recommended_command.by_hardware,
+    guide_sections: $sections
+} | with_entries(select(.value != null))' "$1"

.agents/skills/pr-checklist/SKILL.md

@@ -1,6 +1,6 @@
 ---
 name: pr-checklist
-description: Prepare a vLLM change for human PR review or re-review, or analyze an open pull request. Use to check design fit, behavioral coverage, performance evidence, diff quality, and closure of previous feedback before requesting maintainer attention.
+description: Prepare a vLLM change for human PR review or re-review, or analyze an open pull request. Use before opening a PR, drafts included (e.g. `gh pr create`), or pushing review fixes, to check design fit, behavioral coverage, performance evidence, diff quality, and closure of previous feedback.
 ---
 
 # PR Checklist
@@ -181,6 +181,8 @@ Summarize model evaluations and benchmarks in compact tables with the configurat
 
 When fixing bugs or issues, provide a clear root-cause analysis. Explain what caused the problem, how it was identified, and why the chosen solution effectively addresses it. A patch which resolves the issue without a proper root-cause analysis may lead to recurring problems and increased maintenance overhead.
 
+For fixes that involve disaggregated prefill/decode (PD) logic, such as the NIXL connector (`vllm/distributed/kv_transfer/kv_connector/v1/nixl/`), make a best effort to include a minimal example of the flow being changed and/or the initial flow that caused the failure. A short numbered sequence (or diagram) of a request's path across the P and D instances, e.g. scheduler admission, prefill execution, KV transfer registration and write, decode-side read, token generation, lets reviewers trace the root cause and the fix without reconstructing the distributed flow from the diff. Keep the example minimal: only the steps and components involved in the change. See the [disaggregated prefill](../../../docs/features/disagg_prefill.md), [NIXL KV push](../../../docs/design/nixl_kv_push_connector.md), and [NIXL KV cache lease](../../../docs/design/nixl_kv_cache_lease.md) docs for the flows involved.
+
 #### 4.4: Implementation Details
 
 Provide a detailed explanation of the implementation, including design decisions, trade-offs, and any relevant technical details. Highlight any non-obvious choices, tradeoffs, potential limitations, and areas where maintainer attention is required.