symbolize: name the module when the symbol is unknown - #86
Merged
Conversation
A GPU profile's libcuda/libcupti internals rendered as bare hex
(0x7f2c945b2c2b) and the whole .pb.gz carried one mapping, 0x0/0x0/0x0.
Those frames were missing not merely a symbol but the module - and the
module is most of the diagnostic value, since NVIDIA ships no symbols
for the internals but "seven frames deep inside libcuda" is an answer.
Two independent losses:
blazesym discards the mapping for an address it cannot name. Its
Symbolized::Unknown variant carries a Reason and nothing else, and the
C API zeroes the whole blaze_sym before setting it (capi/src/
symbolize.rs:1105), so Frame.Module arrives empty for exactly the
frames that need it.
The GPU tools build their pprof with no procmap.Resolver, so
addLocation's only route to a real mapping was closed and every frame
fell through to the unconditional default. Wiring a Resolver into the
builder cannot fix it: those tools build the profile after the workload
has exited, and /proc/<pid>/maps is gone by then.
So the lookup happens in the symbolizer, where the process is still
alive, and the frame carries the mapping forward:
attachModules (only for frames with Reason != FailureNone)
-> ToProfFrames carries Map*/BuildID + an explicit Unresolved bit
-> pprof branch 3b builds a mapping from the frame
-> addLocationByAddr renames to "<module base>+0x<file offset>"
The offset is module-relative, so it survives ASLR and can be fed to
addr2line; it is the same number pprof already stores in
Location.Address. The name goes in Function.Name because go tool pprof
reads the same file and knows none of perf-agent's conventions.
An address in no mapping stays a bare address: attachModules writes
nothing without a Lookup hit, frameMapping refuses a range that does
not contain its own address, and framename.Format refuses the [kernel]
and [jit] sentinels. The two cases are counted apart -
LocalStats.ModulesAttached/ModulesBare and
gpuprobe.Stats.StackFramesModuleOnly.
Two honesty features needed defending against the improvement:
flamegraph.Classify checked module rules before isUnsymbolized, which
was safe only while GPU profiles had no modules. With them, an unnamed
libcuda frame would have been painted as ordinary vendor code and
DomainUnsymbolized - labelled "vendor, no symbols" - would have become
unreachable for vendor libraries. isUnsymbolized now sits above the
module rules, below the kernel rule.
foldedstacks.isAddressOnly now recognizes the module form too, so
AddressOnlyFrames and the "N of M frame slots have no symbol" warning
report the same gap before and after. A test asserts that equality.
For the CPU profilers this is additive - the builder's own Resolver
still runs first, so resolved frames and existing mapping attribution
are unchanged - except that an unresolved frame no longer sets
Mapping.HasFunctions, and gains a module name of its own.
Details, including what still needs a live RTX 3090 run to confirm, in
.superpowers/sdd/unresolved-frame-modules-report.md
dpsoft
force-pushed
the
feat/unresolved-frame-modules
branch
from
August 25, 2026 23:55
8a29f3a to
342fd05
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
An unresolved frame now reads
libcuda.so.550.54.14+0x14b71c6instead of0x7f2c945b2c2b.The RTX 3090 profile's 16-frame joined stack has seven consecutive bare-hex frames — libcuda/libcupti internals. NVIDIA ships no debug symbols for those, so the function name is genuinely unrecoverable. The module is not, and knowing a frame is seven deep inside libcuda is most of the diagnostic value.
Where the module was lost — not where I guessed
I assumed it was dropped downstream. It is not.
blazesym does not populate
Frame.Moduleon a miss.symbolizer.rsreturnsSymbolized::Unknown(reason)— a variant carrying a reason and nothing else — whilehandle_entry_addrhad theMapsEntry, path and all, one frame earlier. The C API then zeroes every other field (capi/src/symbolize.rs:1105). The module never arrived; nothing downstream lost it.A second, independent loss:
cmd/gpu-cuda-profilebuilds with noResolver, soaddLocation's only route to a real mapping never runs. The raw dump shows this beyond the single-mapping line — every location reads0x0 M=1, resolved ones included.And wiring a Resolver into that builder cannot fix it: the GPU tools build the profile after
cmd.Wait(), when/proc/<pid>/mapsis gone. The lookup must happen at symbolization time, which settles the layering.Rendering
Function.Namecarries the module+offset,Location.Addressthe same module-relative file offset, plus a realMapping(path/start/limit/build-id).Function.Namebecausego tool pprofreads this file too and knows none of perf-agent's conventions — one mechanism, both audiences.Module-relative, not absolute:
0x7f2c945b2c2bis meaningless across runs under ASLR;libcuda.so.1+0x14b71c6is stable and can be fed toaddr2linelater.Two honesty features that needed defending against this improvement
flamegraph.Classifychecked module rules beforeisUnsymbolized. That ordering was only safe while GPU profiles had no modules. With them, an unnamed libcuda frame matchesisVendorModuleand gets painted as ordinary vendor code — makingDomainUnsymbolized("vendor, no symbols", the hatched one) unreachable for vendor libraries.isUnsymbolizedmoved above the module rules;[kernel]stays above it.foldedstacks.isAddressOnlynow recognizes the new form, soAddressOnlyFramesand the "N of M frame slots have no symbol" banner read identically before and after. A test asserts that equality directly — otherwise the honest 15.5% gap would have silently become 0% while nothing improved.The no-mapping case
Stays bare hex, behind four guards: a
Lookuphit is required;frameMappingrefuses a range not containing its own address;framename.Formatrefuses[kernel]/[jit]; and it is counted apart viaLocalStats.ModulesAttached/ModulesBareandgpuprobe.Stats.StackFramesModuleOnly. One residual ambiguity is documented and pinned byTestKnownAmbiguityrather than hidden.CPU profilers
Additive. The builder's
Resolverbranch still runs first and wins, so resolved frames and existing mapping attribution are byte-identical. Two deliberate changes: an unresolved frame no longer setsMapping.HasFunctions(it is in the profile because symbols are missing), and hex-named frames with a vendor/system module move to the hatched unsymbolized domain. They also gain module names — the intended benefit, for free.Cannot verify end to end
CapEff: 0, no GPU, andgpu-cuda-45.pb.gzcannot be re-rendered because the mapping data was never written into it. The e2e test transcribes the real 16-frame stack out of that file (raw locations 4..18 then 2):TestGPUStack_Beforereproduces today's output,TestGPUStack_Aftershows all seven becominglibcuda.so.550.54.14+0x…. The mapping ranges are a labelled fixture; the stack, names, addresses and arithmetic are real.A fresh 3090 run should show: >1 mapping in
-raw, locations reading0x<offset> M=<n>, the seven frames named,StackFramesModuleOnlyclose toStackFramesUnresolved(0 means the index is not reaching the symbolizer), and the gap warning still ~15%, not 0%.