Skip to content

fix(atom): align user-selected execution and kernel integration - #1615

Merged
ZhengGong-amd merged 40 commits into
mainfrom
feature/leixin/atom-baremetal-example
Sep 24, 2026
Merged

ZhengGong-amd merged 40 commits into
mainfrom
feature/leixin/atom-baremetal-example

Conversation

@zengleixin-amd

@zengleixin-amd zengleixin-amd commented Sep 23, 2026 •

Copy link
Copy Markdown
Contributor

Description

Support the user-selected ATOM execution mode: direct execution in an existing ATOM/ROCm Python environment, or execution in Docker. This follows #1416 and carries the product fixes the direct-run validation required. It does not include generated experiment artifacts.

  • Stop baremetal setup from persisting an implicit geak backend when none was selected. Explicit environment/.env choices are preserved and the existing framework-specific CLI default applies.

  • Use ATOM's native FlexibleArgumentParser when available. Older installations without that parser are judged by the plain argparse contract of their actual server; broken native-parser dependencies and EngineArgs failures stay unavailable rather than being mistaken for an old installation.

  • Use one stdlib-only AITER cache owner for local patches, CSV preparation and pod patches. Source-patch transactions invalidate the complete serving-module set plus build staging; CSV/GEMM preparation passes its exact selected modules and preserves unrelated ones. module_scope is the single scope definition.

  • Ship that owner to Infera and Ray workers, preserve pod failure/backup policies, and return exit code zero for successful finalization.

  • Align the ATOM example with the shared setup user-choice flow: reuse an explicit HYPERLOOM_RUN_MODE, or ask the operator to select Docker or baremetal before setup and launch. Neither mode is an ATOM-specific default.

Observable effect and scope

Operators select HYPERLOOM_RUN_MODE=docker or baremetal; the example reuses a recorded choice and asks when none exists. Docker runs inside the selected container; baremetal reuses the installed ATOM Python without creating another container or venv. Both paths use the same launch command.

--require-frameworks atom now runs a real import atom and openai_server --help through the selected interpreter, because find_spec alone does not prove the engine runs. Runtime preflight applies VLLM_VENV_ROOT only to vLLM, so ATOM is not handed a vLLM venv on PATH.

The AITER transaction fix belongs to the shared patch owner, not an ATOM-only workaround. The supported example remains single-node. This PR does not change other frameworks' serving parsers, add a backend CLI switch, migrate existing explicit GEAK selections, or change optimization budgets and accuracy policy.

A full source-patch invalidation can require substantial first-use recompilation, depending on the installed cache and workload. CSV preparation does not pay that cost for unrelated modules. A runtime cache whose package identity cannot be verified is rejected before any source or cache mutation, rather than accepting a patch that cannot be safely reverted.

The five examples share the installer's sourceable runtime_env.sh data loader instead of each carrying the same preamble; they no longer execute .env, replay exports, or source generated runtime files. Runtime files are loaded in-process, preserving quoted JSON and multi-line custom headers without executing shell content.

Review round 4: defects fixed

Three defects reported against fb1a12fd1, each reproduced before the fix.

1. The Claude CLI was provisioned only when Forge was the selected kernel backend. The gate returned before the npm install and before ~/.claude/config.json was written, so a GEAK run stopped honouring an explicit HYPERLOOM_CLAUDE_CODE_VERSION, stopped writing credentials, and stopped replacing a missing default CLI that GEAK_CLAUDE_BIN still probes. Isolated base/head comparison across 7 configurations: GEAK with a version pin went from 1 npm request and 1 credential step to 0 and 0.

The CLI is now provisioned regardless of the selected backend, and only a runtime Forge resolves to Claude is validated -- so a Codex pin is no longer judged by the Claude --version check, and an unresolvable FORGE_AGENT_BACKEND is left for the backend to report instead of failing the whole installer.

Note on the report's wording: GEAK's own bootstrap can also install the CLI, but it returns early once the version is at least 2.1.177, reads CLAUDE_VERSION rather than HYPERLOOM_CLAUDE_CODE_VERSION, and does not write credentials. The accurate statement is that non-Forge paths lost Hyperloom's version pin, credential write and missing-CLI repair -- not that GEAK is necessarily left without a CLI.

2. Two cache consumers still used the superseded directory resolution. probe_aiter_jit_cache ignored AITER_JIT_DIR and a read-only package's user cache, so an empty runtime cache was reported warm from package modules. The stale-lock sweep missed the read-only package's ~/.aiter/jit/build entirely, and altered override paths that AITER itself reads verbatim. Both now resolve through the shared owner, and the sweep keeps enumerating every authorized build tree -- including a private cache with no importable package, which the shared context cannot express.

3. AITER csrc Python codegen was invalidated in one editable layout but not the other. A discovered checkout triggered runtime-JIT invalidation; /sgl-workspace/aiter matched its reinstall branch first and fell through to the source-only path, so a patched generator kept serving the previously compiled kernel. Both layouts now invalidate. Python outside csrc stays source-only, so an editable checkout does not run a framework reinstall for an ordinary module, and native sources keep their rebuild command and import probes.

Review round 4: scope reduced

Owner-side mechanisms added beyond what this change needs have been withdrawn:

  • The Forge startup hard failure is gone; _check_node_claude_cli is back to base's presence warning at its original position. ClaudeBackend.preflight again returns early with no explicit executable, and an unusable FORGE_AGENT_CLI falls through to discovery as before. validate_runtime remains only because the installer calls it to check a CLI without loading the SDK.

  • resolve_python, python_venv_root, export_virtualenv_for_python and their main() call site are restored byte for byte to base.

  • Test adaptations that existed only to keep passing under the withdrawn CLI check are reverted with it.

Two suggestions from the report were tested and are not applied, because each would introduce a defect:

  • Restoring the noqa on tier3/dispatch.py:659 fails lint: RUF100 is selected in pyproject.toml and upstream narrowed the rule so BLE001 no longer fires there. That one-line change is required by the upstream merge, not incidental cleanup.

  • Deleting the pod backup pre-check breaks three cases. restore_jit_cache raises ValueError, which the adapter converts to OSError, while callers catch FileNotFoundError; the pre-check also enforces the pod's own backup-path policy, which the shared trust-root check does not replace.

Broadening the ATOM parser's targeted ModuleNotFoundError to a bare except ImportError was also tested and rejected: it silently downgrades a broken native-parser dependency to plain argparse.

runtime_env.sh and the example migration are kept. install.sh sources it unconditionally and its inline copy is deleted, so the asset cannot be withdrawn; the packaged skill and references/operations.md migrated with it, and the four non-ATOM examples point at that skill. The base preamble it replaced fails on a readonly exported variable (base's own test asserted that failure as expected behaviour), leaks the host PYTHON into a container, and executes .env contents. The seven-document shared contract predates this PR. This is one mechanism fewer, not a new one.

Validation

CPU, shell and stub coverage only; no new GPU or live multi-node experiment.

  • AITER source/context and transaction suites: 264 passed, seven existing Windows symlink/POSIX cases excluded locally. The ambient-installed-AITER plus explicit-override regression was reproduced before fixing. A combined cache/CSV/pod run passed 212 tests.

  • Defect 1: 7 configurations compared between base and head through the real installer function with all external work stubbed; the regression and its fix are both visible in the npm/credential counts.

  • Defect 2: independent fixtures reproduce both faults and pass after the fix -- an empty runtime cache now reports cold, and a stale lock under the read-only package's home cache is swept. AITER unit suites: 107 passed, three consecutive runs.

  • Defect 3: RED 2 failed to GREEN 79 passed on Linux; the legacy layout cannot be exercised on Windows because the branch matches a POSIX literal path. Both layouts were also compared field by field through the real decision function.

  • Installer: 26 forge/atom cases passed, including pinned-version repair of a broken default CLI and the new every-backend provisioning case.

  • Shared recipe suite: 109 passed, covering wheel/source layouts, mode selection, readonly roots and launch argv. Shared shell loader: 76 passed.

  • Setup owner: ATOM required-check and check-only workspace-preservation cases pass, along with restored readonly-root coverage.

  • Ruff lint/format and diff checks pass on every changed file. No coverage threshold, omission or skip changes.

  • CI at a0cac276e: all 12 Linux test shards pass; coverage is 91.11% on Python 3.10 and 91.10% on Python 3.11, above the unchanged 90% floor. Lint, CodeQL/security, packaging, REUSE and documentation checks pass. One async-timing case in test_mcp_context_tools_unit.py failed once and passed on re-run; this PR changes no MCP or dispatcher code. The separate Pre-release E2E workflow is still queued for a runner and has not executed.

Windows cannot import parts of this tree (fcntl, os.getuid/geteuid, Path.home() semantics, POSIX literal paths, NTFS permission bits). Those runs used isolated imports and are not a substitute for Linux CI, which is the authority for this PR. test_derive_runtime_paths.py could not be collected locally at all.

Direct-run evidence and limits

The recorded MI355X/Qwen3-14B-FP8 run reached framework optimization, Profile/TraceLens, Forge, patch application and first-use compilation, full E2E accuracy, all 8 concurrency pairs / 16 points, and CLOSE with the session's processes and GPU allocations released.

At fixed concurrency 64, measured output throughput was 4,257.05 to 6,175.67 tok/s (+45.07%), below the 50% target. Full GSM8K accuracy was 0.93404094 over 1,319 samples. These compare the run with its own initial baseline, not Docker with baremetal. The sweep baseline retained the accepted kernel patch and is not a pristine-source baseline.

The run was not uninterrupted or unattended: it required one approved same-session resume and repairs to seven generated numerical-validation drivers, which are not part of this PR. The final optional post-optimization Profile failed on the exhausted session budget and some report fields were stale; those reporting and budget issues remain open. The example updates were checked with local shell and stub tests afterwards, not another Docker/GPU run.

Authoring checklist

  • Linked issue(s): follow-up to feat(atom): support Atom end to end — reap its recovery workers, default its kernel backend to forge, and publish it as supported #1416; no new issue.

  • Single concern: the ATOM direct-run setup-to-kernel-integration contract and its operator example. Shared fixes stay with their existing owners.

  • Simplification: one shared environment loader, one package discovery path and one cache transaction; per-example probes, export replay, duplicate runtime sourcing and the redundant intermediate scope flag are gone, as are three identical ray.init call sites.

  • Breaking changes: no public API or CLI removal. The ATOM example no longer silently chooses Docker for an unset run mode -- it asks. Existing explicit selections remain valid. The setup backend persistence change and the scoped cache behaviour are described above.

  • Upstream ownership: these changes are in Hyperloom's setup, argv adapter, patch owner and recipes. Generated experiment-driver repairs are excluded.

🤖 Generated with Claude Code

Preserve explicit backend selection during setup, probe native ATOM arguments correctly, and include AITER serving modules in the shared JIT rollback transaction. Converge the ATOM example into baremetal-first and Docker execution paths with regression coverage.

Co-Authored-By: Claude <noreply@anthropic.com>
@zengleixin-amd
zengleixin-amd requested a review from a team as a code owner September 23, 2026 03:19
Reuse the shared setup mode-selection flow instead of choosing an ATOM-specific default. Keep Docker and baremetal as peer options, forward the chosen mode into Docker, and test that conflicting dotenv settings cannot replace it.

Co-Authored-By: Claude <noreply@anthropic.com>
@zengleixin-amd zengleixin-amd changed the title fix(atom): enable direct-run setup and reliable kernel integration fix(atom): align user-selected execution and kernel integration Sep 23, 2026
@zengleixin-amd

Copy link
Copy Markdown
Contributor Author

PR #1615 -- fix(atom): align user-selected execution and kernel integration

What it does: Aligns ATOM examples with user-selected Docker/baremetal execution, preserves explicit setup configuration, uses ATOM's native argument parser, and extends the shared AITER patch owner to invalidate and restore serving modules with build artifacts.

Blocking issues: 1

  1. [free:container-python-from-dotenv] Resolve container-local Python before honoring host-written .env interpreter paths. [verified]
    Problem: examples/hyperloom-qwen3-14b-fp8-12h-atom/SKILL.md:173-174 retains PYTHON from the mounted workspace .env and executes it before setup. A prior host setup writes PYTHON, VIRTUAL_ENV and INFERENCE_OPTIMIZER_FORCE_PYTHON there (install_baremetal.sh:2232-2234). Entering Docker without forwarding those host variables does not prevent the required dotenv preamble from reintroducing them. Activating the container's existing ATOM venv, which line 168 explicitly allows as an alternative to exporting PYTHON, changes PATH/VIRTUAL_ENV but not PYTHON.
    Impact: With a persisted host-only interpreter path, the documented Docker activation route exits 127 before setup despite a valid container python3. Head fails in the extracted recipe; the exact merge-base setup and runtime resolver functions select the valid container PATH under the same stale configuration. Explicitly selecting a container-local PYTHON succeeds, so this finding is limited to the advertised activation alternative, not every Docker run.
    Action: Reconcile host-persisted interpreter settings at the container boundary so activation selects the container environment without overriding an intentional container-local PYTHON. Add the conflicting .env plus activation case without pre-setting PYTHON in the test fixture.

Checked: all 13 changed files, 49 selected rules, setup/runtime consumers, native ATOM parser, AITER apply/revert/finalize and stack recovery; independent refutation completed. Ran: 16 ATOM argv tests, 12 resolver tests, five focused transaction tests, CPU base/head reproductions, and the real merge-base interpreter resolver probes. Current-head CI: all applicable checks passed, including all 12 test shards and Python 3.10/3.11 coverage; two main-only jobs skipped as designed. No new Docker/GPU E2E was run for this review. Base: 4473e70 | Head: 10563b5

@ZhengGong-amd

Copy link
Copy Markdown
Collaborator

PR #1615 -- fix(atom): align user-selected execution and kernel integration

What it does: bundles three unrelated fixes (backend default persisted as geak, ATOM argv preflight parser, stale AITER jit/*.so after patch) plus an ATOM example rewrite. The first two are correct and minimal; the AITER fix targets the right root cause at the wrong granularity and regresses an existing caller. Please split and substantially shrink.

Blocking issues: 5

  1. [C1/R2] _invalidate_aiter_jit_build now moves every jit/*.so (apply_kernel_patch.py:1115). _aiter_jit.py:628-644 (prepare_serving_so_for_csvs / drop_serving_so_for_envs, from baseline.py:525, phases/kernel.py:3504/3535/3575/3696, request_handlers.py:5833) unlinks only GEMM modules and expects build-only invalidation, with no restore. Repro, GEMM-only env, 4 modules: base drops 1, head drops 4. Every CSV mismatch now forces a full AITER re-JIT (wheels ship ~80 prebuilt modules). [verified]
  2. [C2] Multi-node twin multi_node/scripts/patch_path_safety.py:86 still moves build only; multi-node runtime-JIT keeps serving stale modules. [verified]
  3. [X3] Now false: inference_optimizer/SKILL.md:1413 ("do NOT delete jit/*.so"), _aiter_jit.py:697,748, docstrings of _aiter_jit_build_dir / _invalidate_aiter_jit_build / _restore_aiter_jit_build, comment apply_kernel_patch.py:1669. [verified]
  4. [X1] Description omits the forced full re-JIT cost. [verified]
  5. mergeable: CONFLICTING -- rebase.

Target shape (fewer mechanisms, not more guards):

  • PR A, backend default: delete install_baremetal.sh:2319, plus one test that main persists nothing when unset. Drop the framework x value matrix; framework does not affect this path.
  • PR B, ATOM parser: keep the switch; mirror the vLLM twin (ImportError -> argparse.ArgumentParser) so ATOM <= v0.1.5 is judged by its real parser instead of going UNAVAILABLE.
  • PR C, AITER:
    • One JIT-dir resolver mirroring AITER get_user_jit_dir, used by every runtime-JIT strategy (wheel, /sgl-workspace, checkout), _aiter_jit, and the multi-node pod.
    • One invalidate/restore transaction taking an explicit module set: apply passes all (state the cost), _aiter_jit passes its selection. _unlink_serving_modules and patch_path_safety.invalidate_aiter_jit_build are deleted.
    • Drop the apply-time symlink rejection (base handled it), the always-true modules_invalidated (module_names already marks the schema), and the re-trust of the import-derived path.
  • Docs/examples ride with the PR whose behaviour they describe. ATOM example = FRAMEWORK=atom + setup run-mode flow + setup --check-only --frameworks atom --require-frameworks. Remove the hand-written import/ROCm probe (duplicates install_baremetal.sh:533,1553), the Forge CLI probe (duplicates ensure_forge_claude_cli), the VLLM_VENV_ROOT workaround (fix preflight if wrong), the readonly-shell trick, and run-specific evidence. Drop the Windows-only test edits.

Checked: all 13 files; _aiter_jit.py, patch_path_safety.py, AITER jit/core.py, ATOM arg_parser.py/api_server.py | Ran: changed test files at head (150 passed), base/head invalidation repro | Base: 4473e70 | Head: 10563b5

Zeng and others added 9 commits September 23, 2026 15:46
Preserve upstream vLLM ROCm 10 source and ROCm 7.2 wheel compatibility guidance alongside the ATOM preinstalled-environment support notes.

Co-Authored-By: Claude <noreply@anthropic.com>
Bring in #1622 without changing coverage thresholds or runtime phase behavior.

Co-Authored-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude <noreply@anthropic.com>
Clarify runtime directory selection, serving-module backups, scoped restore, and existing artifact ownership without changing executable behavior.

Co-Authored-By: Claude <noreply@anthropic.com>
@ZhengGong-amd

ZhengGong-amd commented Sep 23, 2026 •

Copy link
Copy Markdown
Collaborator

Blocking issues: 5

  1. [free:configs-dir-split] CSV prep reads configs/ from a different AITER than the JIT dir it invalidates [verified]
    Problem: _aiter_jit.py:708 takes configs/ from the importable package, while _resolve_serving_jit_dir honours INFERENCE_OPTIMIZER_AITER_JIT_DIR. On a host with aiter installed, test_prepare_serving_so_skips_when_cktile_lives_in_blockscale_cktile_so fails on head (passes on base); a pinned bpreshuffle CSV drops the unrelated module_gemm_a8w8_blockscale_cktile.so. CI is green only because it has no aiter.
    Action: derive jit dir and configs dir from one source (override -> configs beside it, as on base; else the package root). Keep package-root discovery in aiter_jit_cache only, not in three modules.

  2. [X1] Undeclared behaviour changes [verified]
    Problem: _aiter_checkout_root (apply_kernel_patch.py:994) adds a new runtime-JIT strategy: <checkout>/csrc/**/*.py goes from compiled=False to full serving-module invalidation. _aiter_jit._invalidate_jit_build now raises OSError, unhandled at phases/kernel.py:3514/3545/3585/3706 and request_handlers.py:5846.
    Action: state both in the description, or drop the checkout strategy if ATOM does not need it.

  3. [X3] Stale docstrings/comments [verified]
    Problem: _target_is_in_aiter_csrc and _detect_strategy docstrings omit checkout detection; apply_kernel_patch.py:1933/2354/2404/2803 and baseline.py:518 still say only jit/build is moved.
    Action: update them.

  4. [C3] modules_invalidated duplicates module_scope (raised before, unaddressed) [verified]
    Problem: never on main; always True on the apply path; the "legacy full-scope" branch in _record_scope only reads this PR's own intermediate manifests (AGENTS.md Finish the replacement).
    Action: delete the field and legacy branch; revert_kernel_patch:1937 checks status in {ok, clean}.

  5. [D10] Environment gaps patched in the ATOM prompt instead of their owners (raised before, unaddressed) [verified]
    Problem: the example hand-writes the VLLM_VENV_ROOT guard (owner preflight._derive_runtime_paths:297 prepends it for every framework), the Forge CLI probe (owner install.sh ensure_forge_claude_cli only warns), the import/ROCm probe (--require-frameworks already checks), and Docker PYTHON/VIRTUAL_ENV isolation that only the ATOM example of the five sharing that preamble gets (writer install_baremetal.sh:2520-2522). hyperloom-setup/SKILL.md now depends on an example skill.
    Action: fix each in its owner so all examples benefit; reduce the example to FRAMEWORK=atom + setup run-mode resolution + setup --check-only --install-framework none --frameworks atom --require-frameworks + launch. "Installer checks are not equivalent" means extend the installer.

deferred: apply-time build-leaf symlink rejection, import-path re-trust, parser exc.name/getattr layers, 52-case backend matrix, Windows-only test edits -- not wrong behaviour, but each is a mechanism the correct shape does not need.

Checked: aiter_jit_cache.py, apply_kernel_patch.py, _aiter_jit.py, patch_path_safety.py, multi_node/cli.py, preflight.py, install_baremetal.sh, install.sh, all 5 example preambles | Ran: changed test files at head (1 failed / 173 passed on an aiter host), base/head _detect_strategy and CSV-prep repros | Base: 146c113 | Head: 808623f

Zeng and others added 11 commits September 23, 2026 18:32
Retain upstream launcher bundling and common-owner changes alongside the shared AITER transaction dependencies.

Co-Authored-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude <noreply@anthropic.com>
Zeng and others added 5 commits September 23, 2026 21:27
…ipes

Preserve caller selections and readonly workspace roots, isolate Docker interpreter settings, and decode quoted configuration values without executing shell content.

Co-Authored-By: Claude <noreply@anthropic.com>
Keep the shared package-discovery owner while incorporating upstream exception and Forge cleanup changes.

Co-Authored-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude <noreply@anthropic.com>
Preserve explicit Magpie selections while testing runtime imports with and without an existing root. Isolate the local CLI in the sandbox-failure test without bypassing provider preflight or starting a session.

Co-Authored-By: Claude <noreply@anthropic.com>
Preserve unset backend semantics while restoring direct environment writes. Match subprocess bytes/text results and isolate Windows auth configuration paths without bypassing CLI validation.

Co-Authored-By: Claude <noreply@anthropic.com>
@ZhengGong-amd

Copy link
Copy Markdown
Collaborator

PR #1615 -- fix(atom): align user-selected execution and kernel integration

What it does: the AITER cache fix is now in the right shape (common/aiter_jit_cache.py with an explicit module scope, shared by CSV prep and pods), and the round-1/2 findings are addressed. But the PR grew from 13 files (+1.8k) to 56 files (+6.3k, ~75% tests) by re-implementing every removed prompt check as a new, stricter owner-side mechanism. "Fix it in the owner" meant delete the duplicate where the owner already checks, not add a new hard fail. Please shrink back to the fixes.

Blocking issues: 3

  1. [free:claude-cli-gating] agents/kernel/scripts/install.sh:1421-1486 now installs the claude CLI, writes ~/.claude/config.json and honours HYPERLOOM_CLAUDE_CODE_VERSION only when Forge is selected at install time (skip returns first). Base did this for every backend. Consumers on the default GEAK path: specialist dispatch (cli/executors.py:120, falls back to in-process), GEAK_CLAUDE_BIN (install.sh:1271), the gateway version pin. It also gates install on a backend that is often chosen only at launch. Not stated in the description. Revert to base behaviour. [verified by reading]
  2. [C2] Two callers still use the old resolver (no AITER_JIT_DIR, no ~/.aiter/jit): probe_aiter_jit_cache (_aiter_jit.py:75, cold/warm classification) and _resolve_lock_sweep_dirs (:197). Route both through aiter_jit_cache, and delete _resolve_aiter_jit_dir_dynamic. [verified]
  3. [twins] csrc/**/*.py gets full invalidation in a discovered checkout but stays source-only under /sgl-workspace/aiter (apply_kernel_patch.py:1463). One rule for both editable layouts. [verified]

Delete (fallbacks, modes, duplicates):

  • adapter_parsers.py:45 getattr(..., argparse.ArgumentParser) is a silent fallback. Keep only except ImportError -> argparse.ArgumentParser; drop the exc.name filter.
  • _runtime_jit_invalidation_error(result=None) (:1769): the pre-mutation mode duplicates the post-check plus revert. Keep one mode.
  • _trusted_aiter_jit_build_dir (:1071): second import-derived branch.
  • patch_path_safety.restore_aiter_jit_build (:113): its pre-check of the backups repeats restore_jit_cache's.
  • multi_node/cli.py:511: if deps == _KERNEL_NODE_OPS_DEPS picks between two bundlers. Use one.
  • kernel_patch_multinode.py: the ray.init(runtime_env=...) block is copied 3x. Extract one helper.
  • Docker host-Python skip rule exists in both preflight.py:195 and runtime_env.sh:87; .env now has three parsers. One owner.
  • Unrelated: kernelforge/.../tier3/dispatch.py noqa removal and the kernel_node_ops finalize exit code. Separate PRs.

Revert / split out:

  • Forge CLI preflight hard fail and claude.py pin semantics, the resolve_python / python_venv_root rewrite, and the migration of 5 examples to runtime_env.sh. If needed, each is its own PR with its own observable effect.
  • Tests: pin the transaction once in test_aiter_jit_cache; one end-to-end case per caller layer (apply, CSV, pod), not four overlapping suites. Drop fixture churn that only exists to satisfy the new hard fails.
  • Prompts: the "trim + lowercase" rule is repeated 5x across setup SKILL.md and the ATOM example. It is forge_explicitly_enabled; say "exact forge opts in" once.

Target: A) delete install_baremetal.sh :-geak + 1 test; B) parser ImportError fallback; C) aiter_jit_cache with every resolver caller migrated. Nothing else.

Checked: aiter_jit_cache.py, _aiter_jit.py, apply_kernel_patch.py, patch_path_safety.py, multi_node/cli.py, preflight.py, runtime_env.sh, three installers, claude.py, executors.py | Ran: all changed test files on an AITER host (1317 passed; 7 codex_model failures also fail on base, host env) | Base: b461fee | Head: fb1a12f

Zeng and others added 9 commits September 24, 2026 13:32
Installing Node, the claude CLI and its credentials was gated on Forge being
the selected kernel backend, so a GEAK run stopped honouring an explicit
HYPERLOOM_CLAUDE_CODE_VERSION, stopped writing ~/.claude/config.json, and
stopped replacing a missing default CLI that GEAK_CLAUDE_BIN still probes.
Provision the CLI regardless of the selected backend and validate only a
runtime Forge resolves to Claude, so a Codex pin is no longer judged by the
Claude version check and an unresolvable provider is left for the backend to
report.

Resolve the JIT cache probe and the stale-lock sweep through the shared cache
owner: both missed AITER_JIT_DIR or a read-only package cache, so a cold
runtime cache was reported warm from package modules and stale locks under the
home cache were never swept. The sweep keeps enumerating every authorized build
tree, including a private cache with no importable package.

Ray fan-out now initializes through one helper instead of three identical
call sites.

Co-Authored-By: Claude <noreply@anthropic.com>
Python codegen under AITER csrc feeds the JIT compile, but only a discovered
checkout invalidated the cache for it: the editable /sgl-workspace/aiter layout
matched its reinstall branch first and fell through to the source-only path, so
a patched generator kept serving the previously compiled kernel.

Promote that codegen in every layout that owns an AITER JIT build. Python
outside csrc still stays source-only, so an editable checkout does not run a
framework reinstall for an ordinary module, and native sources keep their
existing rebuild command and import probes.

Co-Authored-By: Claude <noreply@anthropic.com>
…rough

Moving a check to its owner meant deleting the copy an example carried, not
turning a startup warning into a hard exit. Restore the presence warning and
its position at the end of preflight, and let an unusable FORGE_AGENT_CLI fall
through to discovery again as it did before.

ClaudeBackend.validate_runtime stays: the kernel installer calls it to check a
CLI without loading the SDK. No startup path calls it.

The auth tests redirect USERPROFILE alongside HOME for the whole module --
Path.home() prefers it on Windows, so the credential writers reached the real
home directory.

Co-Authored-By: Claude <noreply@anthropic.com>
The interpreter rewrite went past what this change needs: resolve_python,
python_venv_root and export_virtualenv_for_python are back to their base
implementations byte for byte, along with the main() call site that skips the
block when no interpreter resolves.

What ATOM actually needs stays: --require-frameworks runs a real import and
openai_server --help through the selected interpreter, since find_spec alone
does not prove the engine runs, and baremetal setup no longer persists an
implicit geak backend.

Coverage for the readonly workspace root is restored -- that branch is still in
the tree and nothing else exercises it.

Co-Authored-By: Claude <noreply@anthropic.com>
Both files only grew stubs and a version-check assertion so they would keep
passing while startup validated a default Claude CLI. That check is gone, so
the adaptations assert a call that no longer happens. Back to base.

Co-Authored-By: Claude <noreply@anthropic.com>
The getattr fallback covered an installation whose parser module exists but
does not define the class, a shape no released ATOM is known to have. Import
the class directly so only a genuinely absent module selects plain argparse,
and drop the constructed case that pinned it.

A broken native parser still reports unavailable rather than being mistaken
for a legacy install, so the targeted ModuleNotFoundError filter stays.

Co-Authored-By: Claude <noreply@anthropic.com>
The finalize subcommand returning 1 on success predates this PR and no caller
here reads that code -- Infera routes on the JSON status. Restoring the base
contract keeps this PR to the ATOM path; the backup-scope coverage that does
belong here stays.

Co-Authored-By: Claude <noreply@anthropic.com>
The finalize subcommand returned 1 on success, which this PR reaches for the
first time: accepting a patch now also deletes the JIT module backups the
shared transaction records, and the pod tests assert that path end to end. The
status list gains finalized so a completed teardown is not read as a failure.

Co-Authored-By: Claude <noreply@anthropic.com>
The exact-forge normalization rule was spelled out five times across the setup
skill and the ATOM example; state it where the switch is defined and refer to
it elsewhere.

Three examples still told operators that startup preflight validates the
selected Claude CLI. That check was withdrawn -- the runtime installer owns it.

Co-Authored-By: Claude <noreply@anthropic.com>
@ZhengGong-amd

ZhengGong-amd commented Sep 24, 2026 •

Copy link
Copy Markdown
Collaborator

PR #1615 -- remaining redundancy at 28e34b0

The previous 3 blocking items are fixed. These remaining items are redundancy, not new bugs. Please delete them before approval.

  1. apply_kernel_patch.py:1775 _runtime_jit_invalidation_error(result=None): the pre-mutation mode duplicates the post-check plus revert. Keep one mode.
  2. apply_kernel_patch.py:1071 _trusted_aiter_jit_build_dir: remove the second, import-derived branch.
  3. patch_path_safety.py:113 restore_aiter_jit_build: its backup-existence pre-check repeats restore_jit_cache. The try/except ModuleNotFoundError dual import (:15) can become a single import path.
  4. multi_node/cli.py:511 if deps == _KERNEL_NODE_OPS_DEPS: two bundlers behind one function. Use one.
  5. adapter_parsers.py:41 exc.name filter: except ImportError -> argparse.ArgumentParser is enough.
  6. agents/kernel/scripts/install.sh:1424 ensure_forge_claude_cli: restore base's install-if-missing/pinned behaviour instead of the loop + embedded Python. Also drop the except (ImportError, ValueError): return None swallow (:1447) and the new die on a present-but-failing claude --version for non-Forge runs. ClaudeBackend.validate_runtime (claude.py:300) exists only for this; remove it with it.
  7. Docker host-Python skip rule is defined twice (preflight.py:195, runtime_env.sh:87), and .env now has three parsers (runtime_env.sh, preflight._parse_env_assignments:320, install_baremetal.sh read_dotenv_var:2175). One owner.

Merge also waits on a review approval and the queued pre-release baremetal E2E (run 35981660502).

Zeng and others added 2 commits September 24, 2026 18:06
The bundler branched on which dependency tuple it was given: kernel ops kept
their modules, launcher helpers were inlined and their imports rewritten.
Registering the modules works for both -- the launcher scripts already import
their helpers by module name -- so the inlining branch and its header-stripping
helper are gone.

Co-Authored-By: Claude <noreply@anthropic.com>
Catching ImportError alongside ValueError treated a damaged installation as an
unnamed provider. kernelforge ships with Hyperloom, so that import failing is a
real fault the installer should report; only an unresolvable provider name is
left to the backend.

Co-Authored-By: Claude <noreply@anthropic.com>
@zengleixin-amd

Copy link
Copy Markdown
Contributor Author

Thanks -- four of these are applied at ae6a9103d. For the rest I reproduced the deletion first; each one changes behaviour, so I am keeping them and recording why.

Applied

4. One bundler. Done. The launcher scripts already import their helpers by module name, so registering the modules serves both callers; the inlining branch, its header-stripping helper and the now-unused re import are gone (multi_node/cli.py, -11 lines).

5. getattr fallback. Deleted -- no released ATOM is known to ship the parser module without the class, so that branch had nothing behind it. The exc.name filter is kept; see below.

6, in part. except (ImportError, ValueError) is narrowed to ValueError. kernelforge ships with Hyperloom, so a failed import is a damaged installation the installer should report, not an unnamed provider.

Earlier rounds. ray.init deduplicated; the backend opt-in rule is stated once instead of five times; three examples no longer claim startup preflight validates the Claude CLI.

Kept, with the check that decided it

1. _runtime_jit_invalidation_error(result=None) -- not a duplicate of the post-check. The pre-mutation call at apply_kernel_patch.py:2222 returns before _claim_backup_dir (:2226) and _copy_to_backup (:2228). Delete it and an untrusted JIT target is patched first and found unrevertable afterwards; the post-check cannot undo a write that already happened.

2. _trusted_aiter_jit_build_dir second branch -- covers a layout the first cannot. The first branch resolves _installed_aiter_runtime_root(path) or _EDITABLE_AITER_ROOT, i.e. a site-packages wheel or /sgl-workspace/aiter. A discovered checkout is neither. Running the real function against a checkout outside both shapes returns True with the branch and False without it, which fails the invalidation this PR just fixed for that layout.

3. pod pre-check and dual import -- deleting the pre-check fails three cases in test_kernel_patch_jit_transaction.py: restore_jit_cache raises ValueError, the adapter maps it to OSError, and callers catch FileNotFoundError (test_missing_baseline_backup_preserves_candidate_cache is base coverage). The pre-check also enforces the pod's own backup-path policy, which the shared trust-root check does not express. The dual import matches the two ways the module is loaded: pods receive it registered under its top-level name, the repository imports it as hyperloom.common.aiter_jit_cache.

5. exc.name filter -- except ImportError is not equivalent here. vLLM can use the broad form because it falls between two known locations of the same class, so a failure there is always an old version. import atom.utils.arg_parser is a single path, and a broken dependency inside a modern parser raises the same ImportError; with the filter removed that installation is silently judged by plain argparse and a valid FP8 argv gets repaired into the BF16 default. test_atom_broken_native_parser_is_unavailable_not_legacy pins this.

6. ensure_forge_claude_cli -- base's shape is what produced the gating defect. Comparing base and head across 7 configurations, GEAK with HYPERLOOM_CLAUDE_CODE_VERSION set went from 1 npm request and 1 credential write to 0 and 0. The loop exists so a reinstall is re-validated rather than reported as ready. ClaudeBackend.validate_runtime is the installer's offline check -- it resolves and runs --version without constructing the SDK, which preflight cannot do. validate_runtime is only called when Forge resolves to Claude, so a Codex pin or an unnamed provider cannot reach that failure, and --check-only warns instead of failing either way.

7. dotenv owners -- install_baremetal.sh:read_dotenv_var is base code this PR does not touch, so the count is not something this PR introduced. Of the two remaining, one is the shell loader and one is the in-process reader: different languages, different entry points. This PR removed the fourth (the inline parser inside _load_dotenv_fallback). The Docker host-Python rule is stated in both because both entry points read .env independently; collapsing either side would drop the rule for that entry point.

Happy to split any of these into a separate change if you would rather decide them outside this PR.

@ZhengGong-amd ZhengGong-amd left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approved

…aremetal-example

# Conflicts:
#	docs/reference/kernel-execution-path.md
@ZhengGong-amd
ZhengGong-amd merged commit 0012585 into main Sep 24, 2026
32 of 33 checks passed
@ZhengGong-amd
ZhengGong-amd deleted the feature/leixin/atom-baremetal-example branch September 24, 2026 11:19
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants