[BaoYunkai]

Problem

1. Bare-metal setup (install_baremetal.sh --install-framework) can install SGLang or vLLM, but not ATOM. On an environment without ATOM (for example the plain rocm/pytorch:rocm10.0_ubuntu24.04_py3.12_pytorch_release_2.11.0 image), an ATOM workload has no install path and fails preflight; ATOM had to come from a prebuilt image such as rocm/atom-dev.
2. ATOM registers vllm.platform_plugins, vllm.general_plugins and sglang.srt.plugins entry points, which both engines load by default, so a vLLM or SGLang run on the same Python as ATOM would serve through ATOM's platform, model classes and loader patches.
3. The 12h demo already selects SGLang or vLLM through FRAMEWORK, and the 12h forge demo is the same run with the KernelForge backend. ATOM had its own pair of demo skills (hyperloom-qwen3-14b-fp8-12h-atom and hyperloom-qwen3-14b-fp8-12h-atom-forge) that duplicated both, mixing the framework axis into the backend axis and giving four skills for a 2-option backend choice.

The 12h release run of this branch (8 legs on one MI355X node, SGLang / vLLM / ATOM, docker and bare metal, with and without forge) then hit the following:

4. TraceLens runtime patch skipped on bare-metal vLLM, degrading every profile. On ROCm 10, setup installs vLLM from source into the isolated venv with setup.py develop (Bring up vLLM on ROCm 10 and fix scheduler stalls found in the 6-leg run #1605), so site-packages holds only vllm.egg-link and the package lives in the checkout. _probe_isolated_vllm only globbed site-packages/vllm, returned None, and the patch was skipped.
5. kernel_source_resolution.json, the file forge_handoff hands to KernelForge, was dropped. The active finder stamps its symbol-detected non-patchable verdict with source_resolution_method=active_finder but names no source file. validate_document rejects a location method without a source_file (fix(kernel): stop sentinel strings from short-circuiting source resolution #1107), and one invalid entry drops the whole document.
6. A specialist that wrote its heartbeat.json at the worktree top had it harvested as a creation patch, because it is missing from the task-owned files (specialist_done.json and friends) that the harvest pathspec and vet_patches exclude.
7. TraceLens analysis killed while its sub-agents were still running, so analysis.md was never written and the roofline degraded (twice in one run: "10 of 11 subagents complete. Only the gemm analyzer is still running"). The Agent tool call that launches an async sub-agent returns at once, clearing the single tool_in_flight flag; with the main agent waiting and no SDK message for 300s, the stream-idle watchdog killed the run.
8. Specialist worktrees branched off the InferenceX benchmark checkout. The session names the framework by its package dir (<checkout>/python/sglang), and _pick_worktree_base only accepted a preferred path that itself held .git, so it fell through to the first git root in discovery order, which is InferenceX since Make the ENABLEMENT self-repair loop deliver, and fail loudly when it cannot #1604. Specialists edited InferenceX, integrate_patch stashed it, the benchmark's InferenceX runtime patcher dirtied the same files, git stash pop failed, and two SGLang legs stopped with patch_recovery_incomplete right after measuring +21.8% and +27.8% keeps.
9. The stall watchdog never fired on a dead ATOM engine. An ATOM variant with max_num_batched_tokens=65536 hit a GPU memory access fault; its ModelRunner died while the HTTP front end stayed up and kept answering the KV-metrics GET /metrics polls (feat(kv): collect KV cache occupancy and pressure from the engine #1446). Every poll grew server.log, which the silence gate counts as activity (refactor: retire robustness supervision and simplify benchmark watchdogs #1547), so lm_eval waited on 64 requests nothing would serve until the 7800s hard cap.
10. An integrate with no patch (an AITER tuned GEMM CSV artifact, or server flags only) was placed in the InferenceX checkout. With no <FRAMEWORK>_REPO_PATH or <FRAMEWORK>_DIR set, _resolve_framework_root fell through to the first git root in discovery order, and the session stopped with patch_recovery_incomplete exactly as in 8 (the docker SGLang forge leg and the bare-metal SGLang leg, after the fix for 8 was deployed).
11. Server-flag proposals whose extra_args arrived as a flag -> value mapping, e.g. {"--kv-cache-dtype": "fp8_e4m3"}, went through str() in coerce_args and became "{'--kv-cache-dtype': 'fp8_e4m3'}". The grid argv gate then rejected every such variant (extra_server_args must be argv-like flags) before it was measured; one docker SGLang forge round lost all 3 of its variants this way.
12. TraceLens sub-agents killed by the Claude CLI itself. With 7 fixed, the runner no longer kills the analysis, but in print mode the CLI terminates background tasks still running 600s after the main turn goes quiet (Background tasks still running after 600s; terminating). In the rerun, one leg's rmsnorm analyzer, the last of 11 sub-agents, was killed this way, analysis.md was never written, and the session lost its roofline.
13. The fallback of 10 still landed on the InferenceX checkout for a pip-installed framework. The rocm/vllm image serves vLLM from site-packages (its /app/vllm checkout is not where the import resolves), so there is no framework checkout to take and _resolve_framework_root fell through to the first git root, InferenceX. In the rerun a docker vLLM leg stashed it for a flags-only integrate, the runtime patcher rewrote benchmarks/benchmark_lib.sh and lm_eval_sitecustomize.py, git stash pop refused to overwrite them, and the session stopped with patch_recovery_incomplete.
14. Specialist worktrees still branched off the InferenceX checkout when the framework is pip-installed. With no checkout around the preferred tree, _pick_worktree_base fell through to the first git root, InferenceX. A specialist's tuning scratch (scratch/custom_tune.py and its log) was harvested as _worktree_diff.patch with InferenceX recorded as its apply root, integrate_patch trusted that root and stashed InferenceX, and a docker vLLM leg stopped with patch_recovery_incomplete right after a +10.35% AITER tuned GEMM CSV integrate, as in 8 and 13.

Fix

1. Add --install-framework atom. Like SGLang, it installs into the shared interpreter against the existing ROCm torch: AITER first (newest tag compatible with the installed torch/triton, or AITER_REF), then ATOM from source as an editable checkout under torch/triton constraints. It records FRAMEWORK=atom and rejects --framework-env isolated, as for SGLang. ATOM_REF defaults to fe1099b15ddc52e6873e1932935f722864cdeee0, the ATOM_COMMIT that rocm/atom-dev:v0.1.7-rc0 was built from (ATOM publishes no release tags); ATOM_REPO, ATOM_REF and ATOM_ROOT are overridable. The post-install check imports atom and atom.entrypoints.openai.api_server instead of running openai_server --help, whose help rendering raises on ATOM's unescaped %. Preflight lists atom as setup-installable and prints the install command when ATOM is missing. (1873052, f791125)
2. Keep ATOM out of any Python that serves vLLM or SGLang. --install-framework atom stops when SGLang or vLLM imports from the selected Python or from a ROCm 10 vLLM venv built over it with system site-packages. --install-framework sglang, shared vLLM and ROCm 10 source vLLM stop when ATOM imports there. The self-contained isolated vLLM wheel venv is unaffected. docs/compatibility.rst, the setup skill and the 12h demo's ATOM section say to run ATOM in a separate container instead. (4aa156a)
3. Make ATOM a framework option of the existing 12h pair. hyperloom-qwen3-14b-fp8-12h gains an ## ATOM section with the ATOM-specific run mode, execution shell, Docker/baremetal entry, Python selection and setup (including --install-framework atom), cleanup, launch, resume and status checks; hyperloom-qwen3-14b-fp8-12h-forge points at that section and adds only KERNEL_OPT_BACKEND_ORDER=forge. The two ATOM example directories, their packaged skill entries in pyproject.toml and the setup skill's ATOM routing are removed, and the recipe tests execute the 12h demo's ATOM section. (c1c8ae6)
4. When site-packages holds no vllm/ package, resolve the install root from the venv interpreter's vllm.__file__. (e99dac9)
5. A tier that rules on a kernel without locating a file now projects as unresolved; its verdict stays in reason and reason_class. The validator is unchanged: the entry it rejected was self-contradictory, and the producer now writes a valid one, so the document is kept. (7aacb66)
6. List heartbeat.json with the other task-owned work artifacts excluded from the harvest. (4b347d6)
7. Track background task ids from TaskStartedMessage until a TaskNotificationMessage or a terminal TaskUpdatedMessage, as the SDK documents, and apply the in-flight idle bound while any are still running. (0b07ec6)
8. Walk up from the preferred path to the checkout that contains it, and have the worktree read the preferred tree the same way the patch preflight does (session_framework_tree, then the framework's tree). (29e9d06)
9. An increment made only of GET /metrics|/health|/healthz|/ping access lines is not activity for the silence gate, the same rule the liveness report already applies. (3feb51f)
10. Before falling back to the first git root, take the checkout that holds the framework's own tree (resolve_framework_tree). The walk-up from a package dir to its checkout is shared with the worktree base of 8 as enclosing_checkout. (d83516d)
11. Render a mapping as --flag value pairs (True = bare flag, False = dropped), the same way the list form is already space-joined. (b4096c7)
12. Pass CLAUDE_CODE_PRINT_BG_WAIT_CEILING_MS to the CLI at the runner's in-flight idle bound (0 when that bound is disabled), so the runner's watchdog from 7 decides; an explicit value in the environment still wins. (be837fc)
13. Fixed together with 14.
14. Name the tree a session optimises instead of taking a root by position. framework_apply_tree classifies the tree resolve_framework_tree names: the checkout that tracks it, or the tree itself for a pip-installed package, the form the search roots already give such a package in. The specialist worktree and _resolve_framework_root without patches both use it, and neither falls back to the first git root any more, which also retires the fallback 8 and 10 still left behind. A tree with no checkout gets a session-owned bare git snapshot, refreshed at each dispatch unless an integrate is still deciding on a candidate; the worktree is branched off it, so the specialist edits the code the server imports and its harvest applies to that tree through the existing non-git apply and backup path. Nothing is written into the tree itself. The specialist prompt describes that worktree instead of calling the served tree read-only. resolve_framework_tree maps xdit to its xfuser package so xDiT keeps a named tree without the fallback. The InferenceX special cases of 6686aed and 18a68d9 are removed. (7f442da)

[lishuoshuo-amd]

PR #1694 -- feat(setup): install ATOM on bare metal with --install-framework atom

What it does: bare-metal setup could install SGLang and vLLM but not ATOM, so an ATOM workload needed a host that already shipped it (in practice the rocm/atom-dev image) and the preflight gate could only tell the operator that setup does not install ATOM. The PR adds the missing arm at the layer that already owns framework installation: install_baremetal.sh accepts --install-framework atom, Phase 2 dispatches to install_atom_framework, which installs AITER through the existing ensure_aiter_for_python and then ATOM as an editable checkout of ATOM_REPO@ATOM_REF under the write_rocm_torch_constraints torch/triton constraint file, verifies that atom and atom.entrypoints.openai.api_server import, records FRAMEWORK=atom, and rejects --framework-env isolated exactly as it does for SGLang; preflight.py adds atom to _SETUP_INSTALLABLE_FRAMEWORKS, so the missing-ATOM error now prints an install command that works, while the importable-but-CUDA-torch case still prints none.

Blocking issues: none

Checked: install_baremetal.sh (the --install-framework case arm, resolve_installed_framework, base_preflight's framework loop and its atom parser probe, install_atom_framework/install_atom_from_source against install_sglang_framework/install_sglang_from_source, ensure_aiter_for_python, write_rocm_torch_constraints, install_requested_framework, write_runtime_dotenv, restore_persisted_framework_env, main), cli/preflight.py _check_serving_framework + _SETUP_INSTALLABLE_FRAMEWORKS + _setup_install_command and all their readers, framework_paths.py discovery of an editable checkout, bypass_engine.py:112 and reference_script.py:515 for the launched module, skills/hyperloom-setup/SKILL.md, both ATOM examples, .env.template. Upstream at the pin: atom/entrypoints/openai_server.py is a shim importing atom.entrypoints.openai.api_server (USE_ATOMESH_ENTRYPOINTS defaults to 0), ATOM_MESH_BUILD defaults to 0, ATOM declares no ext_modules and no submodules, aiter is a hard runtime import and is absent from ATOM's pyproject, atom/model_engine/arg_utils.py:441 carries the unescaped % and rendering that help raises TypeError (reproduced), and fe1099b15ddc52e6873e1932935f722864cdeee0 matches ATOM_COMMIT=fe1099b15 in the build history of rocm/atom-dev:v0.1.7-rc0 read from its registry config blob. | Ran: pytest test_setup_cli.py -k atom, test_preflight_serving_framework.py, test_dotenv_precedence_recipes.py, test_install_baremetal_vllm_target.py at head (all pass); the same new tests against the merge base with only the test files copied in: 13 failed, 63 passed, so every added test falsifies. | Base: ec95c36 | Head: 1873052

[xiaofei-zheng]

No blocking issues. LGTM.

Reviewed at c9590afc against merge base 245247f3.

What I checked:

• ATOM bare-metal arm: install_baremetal.sh (--install-framework atom, ATOM_REPO/ATOM_REF/ATOM_ROOT, refuse_atom_beside_vllm_or_sglang, the isolated rejection) against preflight._SETUP_INSTALLABLE_FRAMEWORKS; the removal of the two 12h-atom example skills and their pyproject entries leaves no tracked reference behind, and the docs/examples move with it.
• Framework-tree resolution: FrameworkTree/framework_apply_tree/enclosing_checkout/_tracks reach both consumers (specialists/runner.py, executors/integrate_patch.py); _pick_worktree_base is deleted with its last reader and no reference remains; specialist_patch_preflight_error's narrowed signature matches both production callers.
• tree_snapshot.py: session-owned bare repo outside the tree, per-repo lock, finite timeouts, every git failure surfaced as an error rather than swallowed; reached through asyncio.to_thread. Pinned end-to-end through _maybe_setup_worktree with real git.
• _candidate_resolution_method's pathless set matches the validator in common/kernel_source_contract.py; coerce_args' mapping branch output passes the grid_server_args argv gate; kwargs["env"] is an SDK overlay over the inherited environment, not a replacement; _IncrementScan.active defaults true so non-server log scanners keep their prior meaning.
• Title and the 14-item description match the diff at the current head.

Earlier blockers, both now resolved: the branch was CONFLICTING, which had suppressed the pull_request workflow set for 12 commits; and ruff format --check failed on test_proposal_identity_unit.py:137. After the merge of origin/main the full matrix ran green at 7e308fce (py3.10/py3.11, 6 shards each, plus coverage, CodeQL, pylint, mypy, bandit, reuse, docs, pyproject declarations), and c9590afc is a pure reformat of that one parametrize entry with ruff passing. Test shards for c9590afc were still running when I approved.