fix[next]: collect closure vars from nested code objects - #2861
Open
havogt wants to merge 1 commit into
Open
Conversation
`get_closure_vars_from_function` used `inspect.getclosurevars`, which disassembles
the enclosing function's own code object and collects the names loaded by its
`LOAD_GLOBAL` instructions. A generator expression compiles to a separate code
object, so a name referenced only inside a tuple comprehension body is recorded
there and nowhere else, and was never collected. Type deduction then failed with
`UndefinedSymbolError` on a name that is visibly imported:
@gtx.field_operator
def testee(tracers: tuple[EField, ...]) -> tuple[CField, ...]:
return tuple(neighbor_sum(t(C2E), axis=C2EDim) for t in tracers)
# UndefinedSymbolError: Undeclared symbol 'neighbor_sum'
This affected every module-level name: builtins, `FieldOffset`s, `Dimension`s
and module-level field operators, so `tuple(_inner(t, ...) for t in tracers)`
did not work either.
Free variables were unaffected, because closing over one forces a cell whose
name is recorded on the enclosing code object as `co_freevars`. Globals need no
cell, so nothing links them back. That is also why the existing comprehension
tests pass: they all define their helpers inside the test function.
Nested code objects are scanned the same way `inspect.getclosurevars` scans the
outer one -- `LOAD_GLOBAL` only, so attribute names in `co_names` are not
mistaken for references to the enclosing namespace -- and resolved against
`__globals__` then the builtins namespace, merged at the same precedence so
nonlocals still win. The comprehension target is not collected, being a local of
the nested code object.
Claude-Session: https://claude.ai/code/session_01VR1cyTQ4wysovMBAwPBAWh
havogt
force-pushed
the
c1-closure-vars-repro
branch
from
September 4, 2026 16:08
aad62d1 to
bb56464
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Targets
sf_n_tracer_support(#2833) rather thanmain, because the bug is only reachable through the tuple comprehensions that PR introduces. Follow-up to this review comment.There is a parallel PR, #2864, fixing the same bug a different way — via the
symtablemodule instead of code-object introspection. They are alternatives; pick one and close the other. A comparison is at the bottom.Problem
A name referenced only inside a tuple comprehension body is not collected as a closure variable, so type deduction rejects it — naming a symbol that is visibly imported at module scope:
This covers every module-level name a real stencil uses: gt4py builtins,
FieldOffsets,Dimensions, and module-level field operators — so the natural factoringtuple(_inner(t, ...) for t in tracers)does not work either.Two things make it more than a limitation: the diagnostic names a symbol that is plainly in scope, and the workaround is to also reference the name outside the comprehension — so whether a stencil compiles depends on an unrelated line elsewhere in the function.
Cause
get_closure_vars_from_functionusesinspect.getclosurevars, which disassembles the enclosing function's own code object and collects the names loaded by itsLOAD_GLOBALinstructions. A generator expression compiles to its own code object, so the names it uses are recorded there and nowhere else:Free variables are unaffected, because closing over one forces a cell whose name is recorded on the enclosing code object as
co_freevars. Globals need no cell, so nothing links them back. That asymmetry is also why the existing comprehension tests are green — they all define their helpers inside the test function, putting them on the working side of the bug.Fix
Walk the nested code objects and scan each the same way
inspect.getclosurevarsscans the outer one:LOAD_GLOBALinstructions only, so attribute names — which also appear inco_names— are not mistaken for references to the enclosing namespace. Resolve against__globals__then the builtins namespace, merged at the same precedence so nonlocals still win. The comprehension target is not collected, being a local of the nested code object rather than a global reference.Tests
test_tuples.py: three integration tests placed next totest_tuple_comprehension_other_fo, so the contrast is visible in one screen — that test defines its helper inside the test (free variable, always worked), the new ones use a module-level operator and a module-level builtin. Plus the used-outside-too variant that documents the workaround.test_tuple_comprehension_closure_vars.py: mechanism-level unit tests pinning that the names live in the nested code object, that the comprehension target is not collected, and that an enclosing local shadowing a module-level name is not picked up as a global.All new tests fail without the source change.
Validation
tests/next_tests/unit_tests/ffront_tests/+type_system_tests/: 443 passed, 2 skipped, 3 xfailed.test_tuples.pygreen on roundtrip/numpy and on gtfn CPU, includingskip_value_mesh.uv run mypy src/: clean (361 files).pre-commit: all hooks pass.Versus #2864
symtable)LOAD_GLOBALinspect.getclosurevarsalready uses, applied recursivelysymtable.Function.get_globals()per scopefrom __future__ import annotationsprefixdisopnamessymtable.Functionclassification + PEP 563https://claude.ai/code/session_01VR1cyTQ4wysovMBAwPBAWh