Repository navigation
feat(ci): run the tutorials unattended and pin their dependencies - #141
Merged
Merged
Conversation
Notebooks that asked the reader to start a server in a terminal now do it in a code cell, so they run the same way for readers and in CI: - ensure_server / stop_server helpers: start a server in the background (unless one already answers), wait for its health URL, and raise with the end of its log if it exits or never comes up. - vLLM / SGLang: openclaw_vllm, build_airbnb_agent_mcp, llama4_profiling_vllm, vllm_v1_DSR1 (V1, then V0, then the assistant server; the benchmarks end after BENCHMARK_SECONDS instead of waiting for a kernel interrupt), SpecForge_SGlang, and power-Google-ADK (its rocm/vllm container, without the --group-add render that the image rejects). - Ollama: power-Google-ADK, voice_pipeline_rag_ollama, rag_ollama_llamaindex (replaces `sudo systemctl`). Install is a %%bash cell with set -e that adds zstd when the installer needs it; `capabilities: [ollama]` is dropped. - ComfyUI: t2v_comfyui_api_mode_instinct moves DOWNLOAD_MODELS into a parameters cell (CI sets it) and asserts every mesh loads with faces and every video has frames; t2v_comfyui_radeon starts ComfyUI in the background, runs the LTX-Video workflow through the HTTP API and asserts the saved video, with git/clone/install steps that fail loudly. - ci.timeout_minutes (checked) for the long notebooks; CI_METADATA.md gains Servers, "make setup steps fail loudly" and "check results" sections. Verified on an MI300X under papermill in rocm/pytorch: t2v_comfyui_radeon passes end to end (97-frame 768x512 video); rag_ollama_llamaindex gets Ollama up and the models pulled, then stops at its own venv step (the image has no python3-venv); the ADK vLLM server cell comes up in 96s and stops.
TwelveNights
requested review from
Mahdi-CV,
edamamez and
shailensobhee
as code owners
October 16, 2026 01:47
…t installs
CI now runs every notebook in a virtual environment layered on the image's
Python, after installing what the notebook declares:
- ci.requirements (a requirements file next to the notebook, or a list of
pip requirements) and ci.system_packages (apt packages), validated by the
check and documented in CI_METADATA.md ("Dependencies").
- 25 notebooks declare the system tools their cells use that minimal ROCm
images lack: git, wget, unzip, cmake, ffmpeg, flex, sudo, zstd.
- rag_ollama_llamaindex and the ADK notebook no longer build their own venv
and splice its site-packages into sys.path: they install with %pip
(rag_ollama also declares its LlamaIndex packages as ci.requirements and
uses the environment's PyTorch instead of installing a ROCm 6.2 one).
- apt installs inside notebook cells are replaced by declarations (Ollama's
zstd, ComfyUI's git, Whisper's ffmpeg, Quark's unzip/wget, UCX's flex);
the one in triton_inference_server_benchmark stays because it runs inside
the notebook's own container.
- fp8_quantization_quark_vllm: the %%bash magic was on the cell's second
line, so the cell could not run.
Found running the notebooks on Radeon Global, whose instances reach the internet through an egress proxy and an HF mirror (HF_ENDPOINT), with ollama.com unreachable: - Ollama notebooks install Ollama with ROCm support from the GitHub release archives (what the official installer downloads) instead of ollama.com. - t2v_comfyui_radeon downloads its models with hf_hub_download (resumable, cached, honours HF_ENDPOINT) and links them into ComfyUI's model folders, instead of raw requests to huggingface.co. - voice_pipeline_rag_ollama declares the packages its prerequisites list as ci.requirements (whisper, LlamaIndex, ChatTTS, ...).
- aiter_mla_decode_kernel, helion_gpu_kernel_dev: install AITER with pip --no-build-isolation; build isolation hid the image's ROCm SDK, so the build could not find the ROCm version. helion clones it as aiter_src (an aiter/ checkout shadowed the package) and adds it to the running kernel's path. - helion_gpu_kernel_dev: Helion 1.5.1 on the container's Triton (Triton 3.5.1 cannot read current ROCm device libraries); the softmax kernel tunes once for all shapes; softmax_triton runs on its own input and output instead of the 8192x8192 tensors from the tuning demo. - unsloth_Llama3_1_8B_GRPO: current unsloth and bitsandbytes from PyPI (both support ROCm) in vllm/vllm-openai-rocm instead of a 2025 fork and a source build; MAX_STEPS parameter (CI runs 5 steps). - qwen_image: build DiffSynth-Studio with setuptools<81 (its setup.py imports pkg_resources) and transformers<5; datasets<5 keeps huggingface-hub compatible; the training launch fails loudly; the stale ROCm 6.4 index is gone; 150-minute timebox. - 3_inference_ver3_HF_vllm: start vllm serve in the background and wait for it, instead of a foreground server on GPU 2 and a second notebook. - torchtune_llama3: MAX_STEPS_PER_EPOCH parameter (CI runs 20 steps per recipe); 90-minute timebox. - triton_kernel_dev: bound the Triton source build to 16 jobs (the unbounded build ran the pod out of memory); the cell fails loudly. - fp8_quantization_quark_vllm: calibration batches on the model's device. - deepseek_janus_cpu_gpu: declare git (Janus installs from its repo). - agentic_ai_profiling: put ~/.local/bin on PATH for the Hermes agent and pass an absolute UTILS_DIR to the setup script; find the Python of current Hermes installs (bundled, no venv) to verify the Kokoro tool.
- ci.image may pin a tag (repo:tag) for a notebook that only works in an older image (torchtitan_deepseek's branch needs rocm/primus:v25.9). - ci.sandbox: false runs a notebook in the image's own Python, for tools that start processes with it (slime's Ray workers).
- fp8_quantization_quark_vllm: AMD Quark 0.12 (0.8.1 does not import on current PyTorch): LLMTemplate config, export_safetensors, the 0.12 quantization script's scheme names and options; QuaRot example dropped (the script offers Rotation instead). vLLM and lm_eval leave room for the memory the notebook still holds; lm_eval exits. Calibration and evaluation sample counts are parameters. - ocr_vllm: vLLM 0.31 no longer supports Llama-3.2-Vision; the default is Qwen2.5-VL-7B and images go through llm.chat, so each model uses its own chat template. huggingface-hub is no longer capped below 1.0. - llama4_profiling_vllm: tensor parallelism is a parameter (CI uses two GPUs); vLLM 0.31's --profiler-config replaces VLLM_TORCH_PROFILER_DIR; removed options vLLM no longer has. - openclaw_vllm: one GPU serves the FP8 model; OpenClaw is installed and onboarded non-interactively, and each workshop step has a cell that drives it through the CLI and checks the result. - t2v_comfyui_radeon: uses model files the platform mounts (MODELS_DIR) before downloading them.
…in CI Each declares a NUM_GPUS parameter (readers keep eight) and CI values that shorten the run, so CI uses one or two GPUs: - setup_tutorial, train_llama_mock_data: Megatron-LM pinned to the commit rocm/megatron-lm ships; TP and model size follow the parameters; a BookCorpus subset; training must finish. - training_with_primus: Primus v26.8 layout and CLI. - torchtitan_llama3: torchtitan v0.2.2 (the last TOML-config release) on the container's PyTorch; training steps parameter. - torchtitan_deepseek: expert parallelism follows the GPU count; pinned to the rocm/primus image its torchtitan-amd branch works in. - SpecForge_SGlang: runs in lmsysorg/sglang; fixes for current transformers and SGLang; training-sample and benchmark limits. - slime_qwen3_4B_GRPO: GPU count, rollouts and lengths are parameters; paths follow the clone; converts the last saved checkpoint.
The training cell had never succeeded unattended: LLaMA-Factory at the pinned commit requires numpy<2, pip then picked pyarrow 26, which needs NumPy 2, and the `!` line hid the ImportError. Pin pyarrow<26, run the training cell under `set -e`, and add a MAX_STEPS parameter (readers keep the recipe's three epochs; CI trains 20 steps). One MI300X is enough.
.claude/skills/annotate-notebook-for-ci walks through declaring metadata.ci, starting servers from cells, making setup fail loudly, asserting on results, CI parameters for shorter runs on fewer GPUs, and checking the result, with what running every tutorial in CI taught us. .github/CI_METADATA.md stays the reference for the keys.
# Conflicts: # docs/notebooks/fine_tune/llama_factory_llama3.ipynb # docs/notebooks/inference/openclaw_vllm.ipynb
- Servers started by the notebook are stopped at the end (airbnb, openclaw, rag_ollama, voice_pipeline, vllm_v1_DSR1), and server cells that must not reuse a running server stop with a clear message. - Commands that produce the tutorials' results fail their cell when they fail (assert on _exit_code, or %%bash with set -e): the Quark and lm_eval runs, tune run, vllm bench serve, the DSR1 benchmarks, Ollama pulls, airbnb's Node.js install, qwen_image's DiffSynth install. - rag_ollama and voice_pipeline install from a requirements file next to the notebook, which ci.requirements also names; readers are told to use a virtual environment; rag_ollama's torch check is optional. - Metadata: vllm/vllm-openai-rocm declared where the code only calls the vLLM CLI; image tags pinned where the code needs that release (Primus v26.8, Megatron-LM v26.1, SGLang v0.5.22, the vLLM nightly whose V0 engine vllm_v1_DSR1 compares); gpu_memory_gb for models that fill a GPU; torchtitan_deepseek limited to gfx942. - setup_tutorial's multi-node example is a markdown block, so the notebook trains once. - Markdown matches the code (SpecForge image and GPU count, Megatron's TP wording, voice_pipeline's Ollama steps); ComfyUI's checks import trimesh and imageio only where used; openclaw installs pytest and each app's requirements before testing it.
Readers start the server's image (SGLang v0.5.22 or vLLM v0.31) and run Jupyter inside it; the notebook starts the server in the background with the server helpers instead of `docker run`, waits for its health check and stops it at the end. This lets CI run them in one container on one GPU. - deepseekr1_sglang: sglang.launch_server; a cell checks the model answers through the OpenAI-compatible API. - speculative_decoding_deep_dive: vLLM 0.31 options (--speculative-config; removed flags dropped); `vllm bench serve` replaces SGLang's bench_serving in a second container, same workload; a cell compares the two runs; NUM_PROMPTS parameter. - power-Google-ADK: `vllm serve` with the same tool-calling options. - graph_multiagent_openclaw: OpenClaw installed and configured from cells (128K context, tool search off so specialists can be spawned), gateway started in the background, section 5.2 and 6 steps driven by cells, clean-up stops the gateway and the server.
Brings in #138. Where both changed a notebook, #141's fixes are kept (system_packages declarations, the AITER build without isolation, the Quark 0.12 port, qwen_image's pins), plus two of #138's changes #141 lacked: unsloth's current model id, and llama-index-readers-file and pypdf for rag_ollama_llamaindex's document reader.
… data - openclaw_vllm, graph_multiagent_openclaw: install OpenClaw 2026.10.4, the version these tutorials were tested with, through the official installer's --version option (reinstalling if another version is present). - setup_tutorial: the training cells never set DATA_PATH, so Megatron trained on mock data after the notebook prepared BookCorpus. They now pass the preprocessed data and the GPT-2 tokenizer it was tokenized with, check the log names that data, and take SEQ_LENGTH as a parameter (the script's fixed 99990/8/2 split needs enough data for one test sequence; CI uses 4M records at 512 tokens).
The notebooks render on the docs site with their code and outputs, and five also run one-click; edits made for unattended runs should read naturally there. - Server helper cells are collapsed (hide-input); plumbing cells that only export variables for %%bash cells are hidden (remove-input). - `assert _exit_code == 0` lines are gone: those commands run in %%bash cells with set -e. Server-reuse guards and comments use reader language. - Install steps readers need are back as guarded installs that do nothing when the tool is present (git, wget, unzip, flex, ffmpeg, zstd); system_packages stays for CI. - Ollama: readers use the official installer from a terminal; the cell installs a pinned release only when missing and running as root; pulls fail loudly. - speculative_decoding's model download is required (the servers load the local paths); stopping a server waits for its port to free. - Markdown matches the code: torchtitan_llama3's expected packages, llama4's image and GPU needs, rag_ollama's PyTorch check, airbnb's terminal command, fp8's note on QuaRot, graph_multiagent's Part 6 link, briefings, troubleshooting and container-restart note, setup_tutorial's settings; openclaw_vllm has one server introduction and readers either chat or run each cell, not both. - Parameter cells sit where the text introduces them; rag_ollama and voice_pipeline link their requirements files and list the packages.
torchaudio.save now needs TorchCodec, which was not installed, so the text-to-speech step failed; its exit(1) does not stop a Jupyter kernel, so the notebook carried on as if it had worked. Write the WAV with soundfile (TorchCodec's PyPI wheels do not load with ROCm PyTorch), and raise instead of exit(1) wherever a step fails. The data-directory check no longer stops the notebook before the cell that downloads sample text.
Installing gradio unpinned pulled huggingface_hub 2.x, which the transformers 4.x in the one-click vLLM image refuses to import. The install cell keeps the huggingface_hub the container ships (pip then picks a gradio that works with it) and fails loudly. Checked in the one-click image and in the current vLLM image.
A review compared each changed notebook with its original on main. Where the changes for unattended runs or current images left the text behind, it now says what changed and why, in the original's terms: - setup_tutorial: says it trains on the BookCorpus data it prepares (with the GPT-2 tokenizer it was tokenized with), and how to use mock data instead. - training_with_primus: the FP8 explanation matches how the FP8 config is selected; why the release branch is cloned. - build_airbnb, 3_inference: reader images and notes match the current vLLM; SpecForge: installs all its runtime requirements, explains the renamed CUDA-graph option; torchtitan: expected packages and model link match the code. - vllm_v1_DSR1 keeps each server's log separate; qwen_image restores its wget guard; prerequisites name the container the notebook runs in.
… taught From the review against the originals on main: - graph_multiagent_openclaw: the triage ends with the coordinator's correlated root cause again (stale findings are cleared first); the 128K context OpenClaw 2026.10.4 needs is explained. - t2v_comfyui_radeon: the scripted run is an optional, browser-free equivalent of the GUI steps and uses the tutorial's prompt. - ocr_vllm, unsloth: say why the default model / the AMD fork and source build changed; unsloth's ROCm version matches its image. - openclaw_vllm: why the skill prompt asks for frontmatter; HTTPS clone URL, terminal and readiness hints for readers. - power-Google-ADK: the agent's pinned packages and vLLM share the container, and why (no environment resolves both); restored the original's model and token notes. - rag_ollama: the optional PyTorch install is back (current ROCm index), with its wget guard.
… them From the review against the originals on main: - speculative_decoding_deep_dive: `vllm bench serve` reproduces the original bench_serving workload (greedy, uniform 1-8192 input and 1-256 output lengths, warm-up, 1 request/s), and the original server settings that vLLM 0.31 still has are back. Speculative decoding was slower than the baseline on vLLM 0.31 with one MI300X at every load tried, with good acceptance; the text now attributes the 2.3x figure to the blog's earlier vLLM and says so, rather than claiming it. - fp8_quantization_quark_vllm: the hand-built FP8 QConfig is back (Quark 0.12 classes, AutoSmoothQuant from the Llama template); KV-cache specs stay out of layer_quant_config, which vLLM 0.31 rejects for the fused QKV layer; explicit export options; EVAL_BATCH_SIZE is 1 for readers. - helion_gpu_kernel_dev: readers keep per-shape tuning (STATIC_SHAPES, CI uses dynamic shapes); the Summary matches the measured results. - llama4_profiling_vllm: the W7900 option is back, with the GPU memory the model needs; current ROCm and profiler notes.
…s Triton The original's "outperforming Triton" did not reproduce on current images: its two-pass Helion kernel reads every input element twice and ties the tutorial's Triton kernel at best. The section already promised a simple wrapper kernel; it is now there (Helion's simple softmax, one read per row when a row fits a tile) and is the benchmark's "Helion Softmax", with the two-pass kernel kept as its own line and a note on when each fits. Measured on MI300X (M=4096, fp32): the simple kernel is about 1.4-1.6x the Triton kernel with per-shape tuning (readers' default) and about 1.2x tuned once (CI); the Summary says so.
On vLLM 0.31 the default ROCm attention backend made the 70B baseline queue up on 4K-token prompts, and speculation's extra work tipped it into collapse (0.57x). Both servers now use AMD's AITER kernels (VLLM_ROCM_USE_AITER=1, ROCM_AITER_UNIFIED_ATTN); the draft proposes 3 tokens (5 cost more than they saved); a warm-up burst compiles the kernels before measuring; and the comparison reports end-to-end latency, the metric the blog's 2.3x measured (on vLLM 0.6.7). Measured on one MI300X: 1.50x lower end-to-end latency and 1.53x lower time per output token, about 3 tokens per target step. The comparison checks that speculation is faster, so a regression fails the notebook.
qwen_image: pin DiffSynth-Studio to v1.1.9 (newest release; the raw commit afd101f was an untagged snapshot). Full CI run passes on MI300X. slime: pin THUDM/slime to v0.2.1, which is exactly the previously raw-pinned commit 0934a0e. Newer tags (v0.2.2+) add a hard 'AMD GPU requires --use_cpu_initialization=True' assertion in convert_hf_to_torch_dist.py that the notebook's convert cells do not satisfy, so v0.2.1 is the newest tag that runs unmodified. Full 34-cell CI run (incl. GRPO training) passes.
Replace unpinned clones and branch pins with release tags so each run is reproducible: - SGlang_PD_Disagg: open-mpi/ompi -b v5.0.x -> v5.0.11 - training_with_primus: AMD-AGI/Primus release/v26.8 -> v26.8.0 (matches image) - t2v_comfyui_radeon: comfyanonymous/ComfyUI (unpinned) -> v0.42.1 - vllm_v1_DSR1: vllm-project/production-stack (unpinned) -> vllm-stack-0.1.13 - opea_deployment_and_evaluation: GenAIExamples + GenAIEval (unpinned) -> v1.5 Tags verified to exist upstream. Not GPU-validated here: these notebooks have no CI target yet (RDMA / SSH-runner / 8-GPU / Radeon), so the owning teams should confirm the pinned tag keeps the paths each notebook references.
…ecForge Replace the zhangnju fork (raw commit ff628da) with upstream sgl-project/SpecForge, pinned to main commit 03f3682 (upstream has no release tags; dated comment records when it was tested). Upstream uses a unified 'specforge' CLI, so update the notebook accordingly: - install --no-deps from upstream and apply its SGLang spec-capture patch (--target v0.5.18, matching the notebook's pinned ci.image) - prepare_data.py writes <dataset>_train.jsonl; use --sample-size for CI - specforge train / export / benchmark (benchmark takes --num-prompts) - markdown updated to describe the upstream install, patch and online-training producer/consumer flow Full 15/15-cell CI run passes on a single MI300X with the pinned lmsysorg/sglang:v0.5.18-rocm720-mi30x image.
Pin the from-source install to release v0.9.5 (was a pinned dev commit) and install it with its metrics extras and deepspeed 0.18.4. The release needs Python 3.11+, so declare ci.image rocm/pytorch:rocm10.0_ubuntu24.04_py3.12_ pytorch_release_2.13.0 and update the setup markdown for Python 3.12. Validated PASS on that image.
… -e bash Both notebooks cloned ROCm/Megatron-LM at an unstable feature-branch commit (7ed22c96). Pin them to the newest release branch head, 246c2bf4 (megatron_release_v25.6): the rocm_dev head imports typing.override (Python 3.12), but rocm/megatron-lm:v26.1 ships Python 3.10, so the release branch is the stable lineage that runs on the image. Move the clone/checkout into a %%bash cell with set -e so a failed checkout fails the cell. Validated PASS CI-way on 1 GPU.
Add the learnings from pinning the tutorials: pin every third-party git clone to a release tag (not a moving branch or unpinned clone) so a floating upstream cannot silently break or change a passing notebook; move off abandoned personal forks to the maintained upstream; and check a newer pinned ref against the image Python version before bumping (a ref needing typing.override breaks on a 3.10 image) and run clone/checkout under set -e.
TwelveNights
changed the title
Add section 7: no code cells back to back without a one-line explanation, split a cell that does too much, merge cells too small to run on their own. A reader runs cells one at a time, so how code is split into cells is part of the tutorial.
A reader runs cells one at a time, so how code is split into cells is part of the tutorial (skill section 7). Where this PR left server-start, ollama-pull, or parameters cells adjacent to their helper/setup cell with no prose between them, add a one-line markdown lead-in saying what the cell does and why. In mxfp4_quantization_quark_vllm, split the one setup %%bash cell into install / download / pip-install steps, each with its own lead-in (independent steps, no shared shell state, so the split is behavior-preserving). Markdown-only everywhere except that one split; no code logic, versions, flags, or arguments changed. nbformat and check_ci_metadata pass on all 21 notebooks.
The lead-in now says the setting feeds both the %%bash benchmark cells (via the exported environment variable) and the Python export, so a reader knows why the parameters cell and its os.environ export are kept separate.
The port-drain cell and the assistant-serve cell sat back to back with a single markdown line describing both. Split the lead-in so each code cell has its own one-line explanation (Rule 1: no adjacent code cells without prose between them).
A render scan of the PR's changed cells found three places where a code cell added for CI sat back to back with another with no prose between: - vllm_v1_DSR1: the Step 5 V1->V0 stop and serve cells (same split as the Demo 2 transition already fixed). - slime_qwen3_4B_GRPO: the %%bash script-patch cell. - t2v_comfyui_api_mode_instinct: the DOWNLOAD_MODELS parameters cell was inserted between the runtime-check intro and its code cell; add a lead-in for the check and drop the now-stale 'the next cell' forward reference. Markdown only; code is byte-identical in all three.
Add check_hardware_coverage.py, which reconciles each tutorial's declared ci.hardware against a curated audit of its Hardware section (source of truth baked in, 2026-10-13 snapshot) and fails on a disagreement. Wire it into the CI metadata workflow as a gating step. Extend the hardware vocabulary to the GPUs the audit names that it could not express: mi250x/mi210 (gfx90a), mi100 (gfx908), radeon8060s (gfx1151), with the cdna1/cdna2/rdna35 families. check_ci_metadata.py now derives its HARDWARE set from PRODUCT_ARCH/ARCH_FAMILIES maps (mirroring the runner's routing.py), which the coverage checker reuses. Fix the one disagreement this surfaces: llama4_profiling_vllm declares ["instinct"] but its Hardware section names MI300X or Radeon Pro W7900, so declare ["instinct", "w7900"]. Update the annotate-notebook-for-ci skill: run the coverage check and keep cells readable after the one-click render (which strips setup cells and can leave two code cells adjacent).
The curated GPU audit was only needed to populate ci.hardware during the initial metadata import; those declarations now live in the notebooks. A snapshot-backed gate that reconciles ci.hardware against a hand-maintained copy of the Hardware sections adds maintenance without validating against ground truth, so remove the checker and its CI step. check_ci_metadata.py still validates the hardware vocabulary.
Embed the GPU inventory (Prerequisites/Hardware section of every tutorial) directly into metadata.ci.hardware, by product rather than the generic instinct/radeon family token. The declaration is now the source of truth for which GPUs a tutorial supports, so routing sends each notebook only to the hardware it names: the two Radeon tutorials to W7900 / RX 7900 XTX instead of any Radeon, the multi-Instinct ones to their listed parts (MI100/MI210/MI300X, MI300X/MI325X, MI350X/MI355X). check_ci_metadata.py validates the vocabulary; no separate coverage audit is needed.
Review of the cells this PR changed surfaced three clarity nits; fix them without altering any code (the concatenated code stream is identical before and after): - fp8_quantization_quark_vllm: split the overloaded vLLM-evaluation cell into free-models / run-inference / release-engine, each with a one-line lead-in, so the memory-management boilerplate no longer hides the inference. - slime_qwen3_4B_GRPO: the conversion lead-in now notes the cell first re-exports the base checkpoint to torch_dist before converting the trained checkpoint back to HF (bash cell left intact to preserve shell state). - vllm_v1_DSR1: merge the BENCHMARK_SECONDS set/export into one cell, clearing the adjacency that survived the one-click strip. One-click render now shows zero post-strip adjacencies on all three.
Bring the PR's changes in line with the AI Developer Hub contribution guidelines: - No performance claims: drop the measured speedups added to the Helion and speculative decoding tutorials. The speculative decoding comparison now checks that draft tokens were accepted instead of asserting a speedup. - GPUs required: state the minimum GPU count CI runs with in the Prerequisites of the ten notebooks whose CI run uses fewer GPUs than the default settings, with the default configuration as the optional one. - Single notebook: the RAG and voice pipeline tutorials list their packages in their install cells (and in ci.requirements) instead of in requirements files next to the notebook. - Concise text: describe the current setup instead of what earlier versions of the tutorials did, and say what the slime patch does for the reader rather than for CI. - Pinned versions: the Helion and SGLang setup text no longer tells readers to use the latest release while the cells pin one. CI_METADATA.md and the annotate-notebook-for-ci skill now carry these rules. Co-Authored-By: Claude <noreply@anthropic.com>
None of these configurations have been tested, so the CI metadata should not claim or route to them. Only MI300X and newer stay declared. - ci.hardware: remove mi100/mi210 (deepseek_janus_cpu_gpu, power-Google-ADK), mi250x (se3transform_intro) and w7900 (llama4_profiling_vllm, voice_pipeline_rag_ollama). - check_ci_metadata.py vocabulary: remove mi250x/mi210/mi100, gfx90a/gfx908, cdna1/cdna2, and radeon8060s/gfx1151/rdna35. - CI_METADATA.md and the annotate-notebook-for-ci skill: same vocabulary, and an example that no longer lists MI100/MI210.
…generalize the skill Restore ci.hardware entries backed by each tutorial's own testing claim: mi100 and mi210 (deepseek_janus_cpu_gpu, power-Google-ADK), mi250x (se3transform_intro), and w7900 (voice_pipeline_rag_ollama), and the matching vocabulary (mi250x/mi210/mi100, gfx90a/gfx908, cdna1/cdna2) in the metadata check and CI_METADATA.md. llama4_profiling_vllm stays mi300x-only: its W7900 mention is an example, not a tested setup, and it needs two MI300X. annotate-notebook-for-ci: describe the no-nested-containers rule and the one-click readability check without details of how CI is run.
Drop them from deepseek_janus_cpu_gpu and power-Google-ADK (Hardware section and ci.hardware), and from the metadata check and CI_METADATA.md (mi210, mi100, gfx908, cdna1). MI250X keeps gfx90a/cdna2.
TwelveNights
force-pushed
the
feat/ci-unattended
branch
from
October 23, 2026 22:19
29ab827 to
1b5d9a2
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Stacked on #140 (base
feat/ci-metadata). This makes the tutorials that needed a person mid-run executable end to end, so CI's mechanical verdict (every code cell ran without an error) means something — and, while running them, pins or migrates every third-party dependency that was floating on an unstable ref so the pass/fail signal stays reproducible.Servers start from a code cell
Each server notebook gets three small helpers (
ensure_server,stop_server,server_is_up):ensure_serverstarts the server in the background (or reuses one that already answers) and waits for its health URL.Readers no longer open terminals. The markdown explaining the commands and flags stays.
openclaw_vllm,build_airbnb_agent_mcp,llama4_profiling_vllmvllm_v1_DSR1BENCHMARK_SECONDSinstead of needing a kernel interruptSpecForge_SGlangpower-Google-ADK…rocm/vllmcontainer, plus Ollama; adds a clean-up stepvoice_pipeline_rag_ollama,rag_ollama_llamaindexsudo systemctl, which containers don't have)Setup that fails loudly
A
!commandline never fails its cell, so a failed!git cloneor!curl … | shwent unnoticed and broke a later cell. Setup that later cells depend on is now a%%bashcell withset -e:zstdwhen the installer needs it;ComfyUI assertions
t2v_comfyui_api_mode_instinct:DOWNLOAD_MODELSmoves to aparameterscell, which CI sets toTrue. After generation, the notebook asserts every.glbloads with faces and every.mp4has frames. With downloads on, it also asserts the checkpoints were in place, so it can no longer pass without generating anything.t2v_comfyui_radeon: ComfyUI starts in the background instead of a foregroundmain.py. A new "Run the workflow from Python" section queues the LTX-Video workflow through/prompt, waits on/history, and asserts the saved video has frames. The browser instructions stay.Dependency versions pinned and migrated
An unpinned
git cloneor a floating branch silently changes under the tutorial when upstream moves, so a notebook that passed last week fails — or quietly produces a different result — with no change of its own. Every third-party dep on an unstable ref was pinned to a release tag or migrated to a maintained upstream. Dates are the commit dates of the old vs. new ref.Migrations off abandoned forks / unstable refs — GPU-validated:
pretrain/SpecForge_SGlangzhangnju/SpecForgeforkff628da(2025-09-10)sgl-project/SpecForge03f3682(2026-10-18)fine_tune/llama_factory_llama3a0d44c65, 0.9.4.dev0 (2025-10-02)pretrain/setup_tutorial+pretrain/train_llama_mock_data7ed22c96(2025-09-09)megatron_release_v25.6246c2bf4(2026-03-10)SpecForge moves ~13 months forward onto the upstream project (new unified
specforgeCLI, SGLang v0.5.18). LLaMA-Factory moves ~8 months forward (dev commit → tagged release; deepspeed 0.16.5 → 0.18.4) and onto the Python 3.12 ROCm image. Megatron moves ~6 months forward onto the newest stable release branch — not the even newerrocm_devhead, which importstyping.override(Python 3.12) and fails on the image's Python 3.10.Reproducibility pins — mutable ref → immutable tag at the same commit (GPU-validated):
fine_tune/qwen_imageafd101f(2025-11-18)fine_tune/slime_qwen3_4B_GRPO0934a0e(2025-12-12)pretrain/training_with_primusrelease/v26.8head (2026-10-09)Clone pins — unpinned/floating → last tagged release the tutorial was written against (tag-verified via
git ls-remote, no CI target routes to these yet):inference/SGlang_PD_Disagg_On_AMD_GPUv5.0.xheadinference/t2v_comfyui_radeonmasterinference/vllm_v1_DSR1maininference/opea_deployment_and_evaluationmainFramework version bumps:
fp8_quantization_quark_vllmspeculative_decoding_deep_dive,llama4_profiling_vllm,openclaw_vllm,ocr_vllmVLLM_*env-var era)openclaw_vllm,graph_multiagent_openclawThe vLLM notebooks were rewritten for the 0.31 V1 engine, which replaced the old
VLLM_*env vars and the--swap-space/--disable-log-requestsflags.Metadata and guide
ci.timeout_minutes, validated by the check, for the long notebooks.CI_METADATA.mdsections: Servers, "Make setup steps fail loudly", and "Check results, not just that cells ran".annotate-notebook-for-ciskill now tells authors to pin third-party clones to a release tag (and to check a newer ref runs on the image's Python version before bumping).Contribution guidelines
The changes follow the AI Developer Hub tutorial contribution guidelines:
slime_qwen3_4B_GRPO,torchtune_llama3,llama4_profiling_vllm,SGlang_PD_Disagg_On_AMD_GPU,SpecForge_SGlang,setup_tutorial,torchtitan_deepseek,torchtitan_llama3,train_llama_mock_data,training_with_primus) state that minimum and the settings it needs, with the default configuration as the optional one.rag_ollama_llamaindexandvoice_pipeline_rag_ollamalist their packages in their install cells and inci.requirements, with no requirements files next to the notebooks.CI_METADATA.mdand theannotate-notebook-for-ciskill carry these rules.Verification
Run on an MI300X under papermill in
rocm/pytorch:rocm10.1.0_ubuntu24.04_py3.12_pytorch_release_2.14.0(except where a notebook pins its own image):t2v_comfyui_radeon: passes end to end. The workflow generated a 768×512, 97-frame video and the assertion checked it.SpecForge_SGlang: PASS 15/15 on the migrated upstream.slime_qwen3_4B_GRPO: PASS 34/34.llama_factory_llama3,qwen_image, and both Megatron notebooks (setup_tutorial,train_llama_mock_data): PASS on the pinned refs.rag_ollama_llamaindex: Ollama installs, the server comes up and the models pull. The notebook then stops at its ownpython3 -m venvstep because the image has nopython3-venv. That's an existing issue, not introduced here.--group-add render, which is removed.openclaw_vllm,llama4_profiling_vllm,vllm_v1_DSR1),voice_pipeline_rag_ollama, and the four clone pins with no CI target yet (verified withgit ls-remote).