Repository navigation
fix(atom): align user-selected execution and kernel integration - #1615
Conversation
Preserve explicit backend selection during setup, probe native ATOM arguments correctly, and include AITER serving modules in the shared JIT rollback transaction. Converge the ATOM example into baremetal-first and Docker execution paths with regression coverage. Co-Authored-By: Claude <noreply@anthropic.com>
Reuse the shared setup mode-selection flow instead of choosing an ATOM-specific default. Keep Docker and baremetal as peer options, forward the chosen mode into Docker, and test that conflicting dotenv settings cannot replace it. Co-Authored-By: Claude <noreply@anthropic.com>
PR #1615 -- fix(atom): align user-selected execution and kernel integrationWhat it does: Aligns ATOM examples with user-selected Docker/baremetal execution, preserves explicit setup configuration, uses ATOM's native argument parser, and extends the shared AITER patch owner to invalidate and restore serving modules with build artifacts. Blocking issues: 1
Checked: all 13 changed files, 49 selected rules, setup/runtime consumers, native ATOM parser, AITER apply/revert/finalize and stack recovery; independent refutation completed. Ran: 16 ATOM argv tests, 12 resolver tests, five focused transaction tests, CPU base/head reproductions, and the real merge-base interpreter resolver probes. Current-head CI: all applicable checks passed, including all 12 test shards and Python 3.10/3.11 coverage; two main-only jobs skipped as designed. No new Docker/GPU E2E was run for this review. Base: 4473e70 | Head: 10563b5 |
PR #1615 -- fix(atom): align user-selected execution and kernel integrationWhat it does: bundles three unrelated fixes (backend default persisted as Blocking issues: 5
Target shape (fewer mechanisms, not more guards):
Checked: all 13 files; |
Preserve upstream vLLM ROCm 10 source and ROCm 7.2 wheel compatibility guidance alongside the ATOM preinstalled-environment support notes. Co-Authored-By: Claude <noreply@anthropic.com>
Bring in #1622 without changing coverage thresholds or runtime phase behavior. Co-Authored-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude <noreply@anthropic.com>
Clarify runtime directory selection, serving-module backups, scoped restore, and existing artifact ownership without changing executable behavior. Co-Authored-By: Claude <noreply@anthropic.com>
|
Blocking issues: 5
deferred: apply-time build-leaf symlink rejection, import-path re-trust, parser Checked: |
Retain upstream launcher bundling and common-owner changes alongside the shared AITER transaction dependencies. Co-Authored-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude <noreply@anthropic.com>
…ipes Preserve caller selections and readonly workspace roots, isolate Docker interpreter settings, and decode quoted configuration values without executing shell content. Co-Authored-By: Claude <noreply@anthropic.com>
Keep the shared package-discovery owner while incorporating upstream exception and Forge cleanup changes. Co-Authored-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude <noreply@anthropic.com>
Preserve explicit Magpie selections while testing runtime imports with and without an existing root. Isolate the local CLI in the sandbox-failure test without bypassing provider preflight or starting a session. Co-Authored-By: Claude <noreply@anthropic.com>
Preserve unset backend semantics while restoring direct environment writes. Match subprocess bytes/text results and isolate Windows auth configuration paths without bypassing CLI validation. Co-Authored-By: Claude <noreply@anthropic.com>
PR #1615 -- fix(atom): align user-selected execution and kernel integrationWhat it does: the AITER cache fix is now in the right shape ( Blocking issues: 3
Delete (fallbacks, modes, duplicates):
Revert / split out:
Target: A) delete Checked: |
Installing Node, the claude CLI and its credentials was gated on Forge being the selected kernel backend, so a GEAK run stopped honouring an explicit HYPERLOOM_CLAUDE_CODE_VERSION, stopped writing ~/.claude/config.json, and stopped replacing a missing default CLI that GEAK_CLAUDE_BIN still probes. Provision the CLI regardless of the selected backend and validate only a runtime Forge resolves to Claude, so a Codex pin is no longer judged by the Claude version check and an unresolvable provider is left for the backend to report. Resolve the JIT cache probe and the stale-lock sweep through the shared cache owner: both missed AITER_JIT_DIR or a read-only package cache, so a cold runtime cache was reported warm from package modules and stale locks under the home cache were never swept. The sweep keeps enumerating every authorized build tree, including a private cache with no importable package. Ray fan-out now initializes through one helper instead of three identical call sites. Co-Authored-By: Claude <noreply@anthropic.com>
Python codegen under AITER csrc feeds the JIT compile, but only a discovered checkout invalidated the cache for it: the editable /sgl-workspace/aiter layout matched its reinstall branch first and fell through to the source-only path, so a patched generator kept serving the previously compiled kernel. Promote that codegen in every layout that owns an AITER JIT build. Python outside csrc still stays source-only, so an editable checkout does not run a framework reinstall for an ordinary module, and native sources keep their existing rebuild command and import probes. Co-Authored-By: Claude <noreply@anthropic.com>
…rough Moving a check to its owner meant deleting the copy an example carried, not turning a startup warning into a hard exit. Restore the presence warning and its position at the end of preflight, and let an unusable FORGE_AGENT_CLI fall through to discovery again as it did before. ClaudeBackend.validate_runtime stays: the kernel installer calls it to check a CLI without loading the SDK. No startup path calls it. The auth tests redirect USERPROFILE alongside HOME for the whole module -- Path.home() prefers it on Windows, so the credential writers reached the real home directory. Co-Authored-By: Claude <noreply@anthropic.com>
The interpreter rewrite went past what this change needs: resolve_python, python_venv_root and export_virtualenv_for_python are back to their base implementations byte for byte, along with the main() call site that skips the block when no interpreter resolves. What ATOM actually needs stays: --require-frameworks runs a real import and openai_server --help through the selected interpreter, since find_spec alone does not prove the engine runs, and baremetal setup no longer persists an implicit geak backend. Coverage for the readonly workspace root is restored -- that branch is still in the tree and nothing else exercises it. Co-Authored-By: Claude <noreply@anthropic.com>
Both files only grew stubs and a version-check assertion so they would keep passing while startup validated a default Claude CLI. That check is gone, so the adaptations assert a call that no longer happens. Back to base. Co-Authored-By: Claude <noreply@anthropic.com>
The getattr fallback covered an installation whose parser module exists but does not define the class, a shape no released ATOM is known to have. Import the class directly so only a genuinely absent module selects plain argparse, and drop the constructed case that pinned it. A broken native parser still reports unavailable rather than being mistaken for a legacy install, so the targeted ModuleNotFoundError filter stays. Co-Authored-By: Claude <noreply@anthropic.com>
The finalize subcommand returning 1 on success predates this PR and no caller here reads that code -- Infera routes on the JSON status. Restoring the base contract keeps this PR to the ATOM path; the backup-scope coverage that does belong here stays. Co-Authored-By: Claude <noreply@anthropic.com>
The finalize subcommand returned 1 on success, which this PR reaches for the first time: accepting a patch now also deletes the JIT module backups the shared transaction records, and the pod tests assert that path end to end. The status list gains finalized so a completed teardown is not read as a failure. Co-Authored-By: Claude <noreply@anthropic.com>
The exact-forge normalization rule was spelled out five times across the setup skill and the ATOM example; state it where the switch is defined and refer to it elsewhere. Three examples still told operators that startup preflight validates the selected Claude CLI. That check was withdrawn -- the runtime installer owns it. Co-Authored-By: Claude <noreply@anthropic.com>
PR #1615 -- remaining redundancy at 28e34b0The previous 3 blocking items are fixed. These remaining items are redundancy, not new bugs. Please delete them before approval.
Merge also waits on a review approval and the queued pre-release baremetal E2E (run 35981660502). |
The bundler branched on which dependency tuple it was given: kernel ops kept their modules, launcher helpers were inlined and their imports rewritten. Registering the modules works for both -- the launcher scripts already import their helpers by module name -- so the inlining branch and its header-stripping helper are gone. Co-Authored-By: Claude <noreply@anthropic.com>
Catching ImportError alongside ValueError treated a damaged installation as an unnamed provider. kernelforge ships with Hyperloom, so that import failing is a real fault the installer should report; only an unresolvable provider name is left to the backend. Co-Authored-By: Claude <noreply@anthropic.com>
|
Thanks -- four of these are applied at Applied4. One bundler. Done. The launcher scripts already import their helpers by module name, so registering the modules serves both callers; the inlining branch, its header-stripping helper and the now-unused 5. 6, in part. Earlier rounds. Kept, with the check that decided it1. 2. 3. pod pre-check and dual import -- deleting the pre-check fails three cases in 5. 6. 7. dotenv owners -- Happy to split any of these into a separate change if you would rather decide them outside this PR. |
…aremetal-example # Conflicts: # docs/reference/kernel-execution-path.md
Description
Support the user-selected ATOM execution mode: direct execution in an existing ATOM/ROCm Python environment, or execution in Docker. This follows #1416 and carries the product fixes the direct-run validation required. It does not include generated experiment artifacts.
Stop baremetal setup from persisting an implicit
geakbackend when none was selected. Explicit environment/.envchoices are preserved and the existing framework-specific CLI default applies.Use ATOM's native
FlexibleArgumentParserwhen available. Older installations without that parser are judged by the plain argparse contract of their actual server; broken native-parser dependencies and EngineArgs failures stay unavailable rather than being mistaken for an old installation.Use one stdlib-only AITER cache owner for local patches, CSV preparation and pod patches. Source-patch transactions invalidate the complete serving-module set plus build staging; CSV/GEMM preparation passes its exact selected modules and preserves unrelated ones.
module_scopeis the single scope definition.Ship that owner to Infera and Ray workers, preserve pod failure/backup policies, and return exit code zero for successful finalization.
Align the ATOM example with the shared setup user-choice flow: reuse an explicit
HYPERLOOM_RUN_MODE, or ask the operator to select Docker or baremetal before setup and launch. Neither mode is an ATOM-specific default.Observable effect and scope
Operators select
HYPERLOOM_RUN_MODE=dockerorbaremetal; the example reuses a recorded choice and asks when none exists. Docker runs inside the selected container; baremetal reuses the installed ATOM Python without creating another container or venv. Both paths use the same launch command.--require-frameworks atomnow runs a realimport atomandopenai_server --helpthrough the selected interpreter, becausefind_specalone does not prove the engine runs. Runtime preflight appliesVLLM_VENV_ROOTonly to vLLM, so ATOM is not handed a vLLM venv onPATH.The AITER transaction fix belongs to the shared patch owner, not an ATOM-only workaround. The supported example remains single-node. This PR does not change other frameworks' serving parsers, add a backend CLI switch, migrate existing explicit GEAK selections, or change optimization budgets and accuracy policy.
A full source-patch invalidation can require substantial first-use recompilation, depending on the installed cache and workload. CSV preparation does not pay that cost for unrelated modules. A runtime cache whose package identity cannot be verified is rejected before any source or cache mutation, rather than accepting a patch that cannot be safely reverted.
The five examples share the installer's sourceable
runtime_env.shdata loader instead of each carrying the same preamble; they no longer execute.env, replay exports, or source generated runtime files. Runtime files are loaded in-process, preserving quoted JSON and multi-line custom headers without executing shell content.Review round 4: defects fixed
Three defects reported against
fb1a12fd1, each reproduced before the fix.1. The Claude CLI was provisioned only when Forge was the selected kernel backend. The gate returned before the npm install and before
~/.claude/config.jsonwas written, so a GEAK run stopped honouring an explicitHYPERLOOM_CLAUDE_CODE_VERSION, stopped writing credentials, and stopped replacing a missing default CLI thatGEAK_CLAUDE_BINstill probes. Isolated base/head comparison across 7 configurations: GEAK with a version pin went from 1 npm request and 1 credential step to 0 and 0.The CLI is now provisioned regardless of the selected backend, and only a runtime Forge resolves to Claude is validated -- so a Codex pin is no longer judged by the Claude
--versioncheck, and an unresolvableFORGE_AGENT_BACKENDis left for the backend to report instead of failing the whole installer.Note on the report's wording: GEAK's own bootstrap can also install the CLI, but it returns early once the version is at least
2.1.177, readsCLAUDE_VERSIONrather thanHYPERLOOM_CLAUDE_CODE_VERSION, and does not write credentials. The accurate statement is that non-Forge paths lost Hyperloom's version pin, credential write and missing-CLI repair -- not that GEAK is necessarily left without a CLI.2. Two cache consumers still used the superseded directory resolution.
probe_aiter_jit_cacheignoredAITER_JIT_DIRand a read-only package's user cache, so an empty runtime cache was reported warm from package modules. The stale-lock sweep missed the read-only package's~/.aiter/jit/buildentirely, and altered override paths that AITER itself reads verbatim. Both now resolve through the shared owner, and the sweep keeps enumerating every authorized build tree -- including a private cache with no importable package, which the shared context cannot express.3. AITER
csrcPython codegen was invalidated in one editable layout but not the other. A discovered checkout triggered runtime-JIT invalidation;/sgl-workspace/aitermatched its reinstall branch first and fell through to the source-only path, so a patched generator kept serving the previously compiled kernel. Both layouts now invalidate. Python outsidecsrcstays source-only, so an editable checkout does not run a framework reinstall for an ordinary module, and native sources keep their rebuild command and import probes.Review round 4: scope reduced
Owner-side mechanisms added beyond what this change needs have been withdrawn:
The Forge startup hard failure is gone;
_check_node_claude_cliis back to base's presence warning at its original position.ClaudeBackend.preflightagain returns early with no explicit executable, and an unusableFORGE_AGENT_CLIfalls through to discovery as before.validate_runtimeremains only because the installer calls it to check a CLI without loading the SDK.resolve_python,python_venv_root,export_virtualenv_for_pythonand theirmain()call site are restored byte for byte to base.Test adaptations that existed only to keep passing under the withdrawn CLI check are reverted with it.
Two suggestions from the report were tested and are not applied, because each would introduce a defect:
Restoring the
noqaontier3/dispatch.py:659fails lint:RUF100is selected inpyproject.tomland upstream narrowed the rule soBLE001no longer fires there. That one-line change is required by the upstream merge, not incidental cleanup.Deleting the pod backup pre-check breaks three cases.
restore_jit_cacheraisesValueError, which the adapter converts toOSError, while callers catchFileNotFoundError; the pre-check also enforces the pod's own backup-path policy, which the shared trust-root check does not replace.Broadening the ATOM parser's targeted
ModuleNotFoundErrorto a bareexcept ImportErrorwas also tested and rejected: it silently downgrades a broken native-parser dependency to plain argparse.runtime_env.shand the example migration are kept.install.shsources it unconditionally and its inline copy is deleted, so the asset cannot be withdrawn; the packaged skill andreferences/operations.mdmigrated with it, and the four non-ATOM examples point at that skill. The base preamble it replaced fails on a readonly exported variable (base's own test asserted that failure as expected behaviour), leaks the hostPYTHONinto a container, and executes.envcontents. The seven-document shared contract predates this PR. This is one mechanism fewer, not a new one.Validation
CPU, shell and stub coverage only; no new GPU or live multi-node experiment.
AITER source/context and transaction suites: 264 passed, seven existing Windows symlink/POSIX cases excluded locally. The ambient-installed-AITER plus explicit-override regression was reproduced before fixing. A combined cache/CSV/pod run passed 212 tests.
Defect 1: 7 configurations compared between base and head through the real installer function with all external work stubbed; the regression and its fix are both visible in the npm/credential counts.
Defect 2: independent fixtures reproduce both faults and pass after the fix -- an empty runtime cache now reports cold, and a stale lock under the read-only package's home cache is swept. AITER unit suites: 107 passed, three consecutive runs.
Defect 3: RED 2 failed to GREEN 79 passed on Linux; the legacy layout cannot be exercised on Windows because the branch matches a POSIX literal path. Both layouts were also compared field by field through the real decision function.
Installer: 26 forge/atom cases passed, including pinned-version repair of a broken default CLI and the new every-backend provisioning case.
Shared recipe suite: 109 passed, covering wheel/source layouts, mode selection, readonly roots and launch argv. Shared shell loader: 76 passed.
Setup owner: ATOM required-check and check-only workspace-preservation cases pass, along with restored readonly-root coverage.
Ruff lint/format and diff checks pass on every changed file. No coverage threshold, omission or skip changes.
CI at
a0cac276e: all 12 Linux test shards pass; coverage is 91.11% on Python 3.10 and 91.10% on Python 3.11, above the unchanged 90% floor. Lint, CodeQL/security, packaging, REUSE and documentation checks pass. One async-timing case intest_mcp_context_tools_unit.pyfailed once and passed on re-run; this PR changes no MCP or dispatcher code. The separate Pre-release E2E workflow is still queued for a runner and has not executed.Windows cannot import parts of this tree (
fcntl,os.getuid/geteuid,Path.home()semantics, POSIX literal paths, NTFS permission bits). Those runs used isolated imports and are not a substitute for Linux CI, which is the authority for this PR.test_derive_runtime_paths.pycould not be collected locally at all.Direct-run evidence and limits
The recorded MI355X/Qwen3-14B-FP8 run reached framework optimization, Profile/TraceLens, Forge, patch application and first-use compilation, full E2E accuracy, all 8 concurrency pairs / 16 points, and CLOSE with the session's processes and GPU allocations released.
At fixed concurrency 64, measured output throughput was 4,257.05 to 6,175.67 tok/s (+45.07%), below the 50% target. Full GSM8K accuracy was 0.93404094 over 1,319 samples. These compare the run with its own initial baseline, not Docker with baremetal. The sweep baseline retained the accepted kernel patch and is not a pristine-source baseline.
The run was not uninterrupted or unattended: it required one approved same-session resume and repairs to seven generated numerical-validation drivers, which are not part of this PR. The final optional post-optimization Profile failed on the exhausted session budget and some report fields were stale; those reporting and budget issues remain open. The example updates were checked with local shell and stub tests afterwards, not another Docker/GPU run.
Authoring checklist
Linked issue(s): follow-up to feat(atom): support Atom end to end — reap its recovery workers, default its kernel backend to forge, and publish it as supported #1416; no new issue.
Single concern: the ATOM direct-run setup-to-kernel-integration contract and its operator example. Shared fixes stay with their existing owners.
Simplification: one shared environment loader, one package discovery path and one cache transaction; per-example probes, export replay, duplicate runtime sourcing and the redundant intermediate scope flag are gone, as are three identical
ray.initcall sites.Breaking changes: no public API or CLI removal. The ATOM example no longer silently chooses Docker for an unset run mode -- it asks. Existing explicit selections remain valid. The setup backend persistence change and the scoped cache behaviour are described above.
Upstream ownership: these changes are in Hyperloom's setup, argv adapter, patch owner and recipes. Generated experiment-driver repairs are excluded.
🤖 Generated with Claude Code