Investigate output and gradient variability between matched same-source Qwen TrainerRank actors

[bradhilton]

The #848/#870 qualification work produced a numerical failure between two independently loaded actors using the same ART main source, checkpoint, inputs and measured execution context. This is separate from the packing-dependent no-gradient case in #901; a common cause is unproved.

Observed September 15, 2026, Qwen/Qwen3.6-35B-A3B, one H200 per actor, rank-1 LoRA. Both baseline actors use ART main 54ebf50f and the same complete tracked ART source namespace. A third actor uses #898 candidate bec279427003452697a73de09ca35e200ed77cda. All ran the unchanged three-stage warmup/warm1/warm2 diagnostic with 6,330 positions, 224 masked positions, three backwards and no optimizer update.

For all three actors, the pre-run full registered tensor inventory completed: 22,264 tensors / 69,556,700,544 logical bytes. Normalized loaded-state SHA256 1bf758dd9af4309998b980762ab4b8b7467ab466979d99af0bfd66b64a2ee6e9 and stable measured context SHA256 b9b1fbb8e408a50fb5b37b37b4c386effbc4fc6f922a7644875d3a5e069e32c5 match. Complete publication/source/descriptor/closure validation succeeded; current strict readout reports no common-condition mismatches. The inventory also checks registration/version/storage/source/RNG/context stability across capture; it is not a full post-training value rehash or proof of identical unmeasured compiler/kernel state.

Strict warm1 comparisons (unchanged atol1e-5, rtol0.016, relative-L2 limit0.03):

• Main versus independent main replica: output max absolute difference 1.14703369140625, relative L2 0.043881414172583; 2,290/6,330 output elements exceed the elementwise tolerance. All 660 gradient tensors fail at least one declared condition; maximum per-tensor relative L2 1.0555278532309724.
• Main versus Account for grouped checkpoint, head, and enclosing forward buffers #898 candidate: all 6,330 forward outputs are bit-for-bit identical. 442/660 gradient tensors pass; 218 fail. Maximum gradient absolute difference 0.000244140625, maximum per-tensor relative L2 0.053877846678882096.
• All three original within-actor repeatability checks also failed and remain authoritative. All three completed their three backward/clear stages; no OOM was observed in this small case. This is not near-cap memory qualification.

Consequences: neither candidate regression nor candidate numerical equivalence is established. Same-source baseline variability confounds attribution; successful offline reader exits preserve negative scientific results. Do not relax tolerances or promote a failed run to acceptance based on these observations.

Owned follow-up (Schulman): compare the retained exact device/native-library/compiler/runtime-context evidence to identify a meaningful controlled next test. Keep near-cap memory-admission testing moving separately. No art.megatron patch or underlying kernel cause is established.

Durable evidence:

• /var/tmp/art898-loaded-{main,candidate,main-replica}-native-v3-20260915-root/ — exact source plans, original raw logs, projections and independent closure receipts.
• /var/tmp/art898-loaded-v3-readout-20260915-root/main-main-replica-result.json and main-candidate-result.json — original strict complete 661-item comparisons.
• /var/tmp/art898-v3-gpu-local-audit-20260915-backward-hf8vvzuy/, manifest 5972ecd97e59410d29fd54b867909744880bf2d5b5a1909adf44294393fd21c8 — source/actor/terminal joins and preserved original failures.
• All twelve exact GPU job/configmap/secret/pod resource IDs and six local process groups were independently confirmed absent after these runs. No new live run is implied by this issue.