Skip to content

symbolize: name the module when the symbol is unknown - #86

Merged
dpsoft merged 1 commit into
mainfrom
feat/unresolved-frame-modules
Aug 26, 2026
Merged

symbolize: name the module when the symbol is unknown#86
dpsoft merged 1 commit into
mainfrom
feat/unresolved-frame-modules

Conversation

@dpsoft

@dpsoft dpsoft commented Aug 25, 2026

Copy link
Copy Markdown
Owner

An unresolved frame now reads libcuda.so.550.54.14+0x14b71c6 instead of 0x7f2c945b2c2b.

The RTX 3090 profile's 16-frame joined stack has seven consecutive bare-hex frames — libcuda/libcupti internals. NVIDIA ships no debug symbols for those, so the function name is genuinely unrecoverable. The module is not, and knowing a frame is seven deep inside libcuda is most of the diagnostic value.

Where the module was lost — not where I guessed

I assumed it was dropped downstream. It is not.

blazesym does not populate Frame.Module on a miss. symbolizer.rs returns Symbolized::Unknown(reason) — a variant carrying a reason and nothing else — while handle_entry_addr had the MapsEntry, path and all, one frame earlier. The C API then zeroes every other field (capi/src/symbolize.rs:1105). The module never arrived; nothing downstream lost it.

A second, independent loss: cmd/gpu-cuda-profile builds with no Resolver, so addLocation's only route to a real mapping never runs. The raw dump shows this beyond the single-mapping line — every location reads 0x0 M=1, resolved ones included.

And wiring a Resolver into that builder cannot fix it: the GPU tools build the profile after cmd.Wait(), when /proc/<pid>/maps is gone. The lookup must happen at symbolization time, which settles the layering.

Rendering

Function.Name carries the module+offset, Location.Address the same module-relative file offset, plus a real Mapping (path/start/limit/build-id). Function.Name because go tool pprof reads this file too and knows none of perf-agent's conventions — one mechanism, both audiences.

Module-relative, not absolute: 0x7f2c945b2c2b is meaningless across runs under ASLR; libcuda.so.1+0x14b71c6 is stable and can be fed to addr2line later.

Two honesty features that needed defending against this improvement

  1. flamegraph.Classify checked module rules before isUnsymbolized. That ordering was only safe while GPU profiles had no modules. With them, an unnamed libcuda frame matches isVendorModule and gets painted as ordinary vendor code — making DomainUnsymbolized ("vendor, no symbols", the hatched one) unreachable for vendor libraries. isUnsymbolized moved above the module rules; [kernel] stays above it.
  2. foldedstacks.isAddressOnly now recognizes the new form, so AddressOnlyFrames and the "N of M frame slots have no symbol" banner read identically before and after. A test asserts that equality directly — otherwise the honest 15.5% gap would have silently become 0% while nothing improved.

The no-mapping case

Stays bare hex, behind four guards: a Lookup hit is required; frameMapping refuses a range not containing its own address; framename.Format refuses [kernel]/[jit]; and it is counted apart via LocalStats.ModulesAttached/ModulesBare and gpuprobe.Stats.StackFramesModuleOnly. One residual ambiguity is documented and pinned by TestKnownAmbiguity rather than hidden.

CPU profilers

Additive. The builder's Resolver branch still runs first and wins, so resolved frames and existing mapping attribution are byte-identical. Two deliberate changes: an unresolved frame no longer sets Mapping.HasFunctions (it is in the profile because symbols are missing), and hex-named frames with a vendor/system module move to the hatched unsymbolized domain. They also gain module names — the intended benefit, for free.

Cannot verify end to end

CapEff: 0, no GPU, and gpu-cuda-45.pb.gz cannot be re-rendered because the mapping data was never written into it. The e2e test transcribes the real 16-frame stack out of that file (raw locations 4..18 then 2): TestGPUStack_Before reproduces today's output, TestGPUStack_After shows all seven becoming libcuda.so.550.54.14+0x…. The mapping ranges are a labelled fixture; the stack, names, addresses and arithmetic are real.

A fresh 3090 run should show: >1 mapping in -raw, locations reading 0x<offset> M=<n>, the seven frames named, StackFramesModuleOnly close to StackFramesUnresolved (0 means the index is not reaching the symbolizer), and the gap warning still ~15%, not 0%.

@dpsoft dpsoft closed this Aug 25, 2026
@dpsoft dpsoft reopened this Aug 25, 2026
A GPU profile's libcuda/libcupti internals rendered as bare hex
(0x7f2c945b2c2b) and the whole .pb.gz carried one mapping, 0x0/0x0/0x0.
Those frames were missing not merely a symbol but the module - and the
module is most of the diagnostic value, since NVIDIA ships no symbols
for the internals but "seven frames deep inside libcuda" is an answer.

Two independent losses:

blazesym discards the mapping for an address it cannot name. Its
Symbolized::Unknown variant carries a Reason and nothing else, and the
C API zeroes the whole blaze_sym before setting it (capi/src/
symbolize.rs:1105), so Frame.Module arrives empty for exactly the
frames that need it.

The GPU tools build their pprof with no procmap.Resolver, so
addLocation's only route to a real mapping was closed and every frame
fell through to the unconditional default. Wiring a Resolver into the
builder cannot fix it: those tools build the profile after the workload
has exited, and /proc/<pid>/maps is gone by then.

So the lookup happens in the symbolizer, where the process is still
alive, and the frame carries the mapping forward:

  attachModules (only for frames with Reason != FailureNone)
    -> ToProfFrames carries Map*/BuildID + an explicit Unresolved bit
    -> pprof branch 3b builds a mapping from the frame
    -> addLocationByAddr renames to "<module base>+0x<file offset>"

The offset is module-relative, so it survives ASLR and can be fed to
addr2line; it is the same number pprof already stores in
Location.Address. The name goes in Function.Name because go tool pprof
reads the same file and knows none of perf-agent's conventions.

An address in no mapping stays a bare address: attachModules writes
nothing without a Lookup hit, frameMapping refuses a range that does
not contain its own address, and framename.Format refuses the [kernel]
and [jit] sentinels. The two cases are counted apart -
LocalStats.ModulesAttached/ModulesBare and
gpuprobe.Stats.StackFramesModuleOnly.

Two honesty features needed defending against the improvement:

flamegraph.Classify checked module rules before isUnsymbolized, which
was safe only while GPU profiles had no modules. With them, an unnamed
libcuda frame would have been painted as ordinary vendor code and
DomainUnsymbolized - labelled "vendor, no symbols" - would have become
unreachable for vendor libraries. isUnsymbolized now sits above the
module rules, below the kernel rule.

foldedstacks.isAddressOnly now recognizes the module form too, so
AddressOnlyFrames and the "N of M frame slots have no symbol" warning
report the same gap before and after. A test asserts that equality.

For the CPU profilers this is additive - the builder's own Resolver
still runs first, so resolved frames and existing mapping attribution
are unchanged - except that an unresolved frame no longer sets
Mapping.HasFunctions, and gains a module name of its own.

Details, including what still needs a live RTX 3090 run to confirm, in
.superpowers/sdd/unresolved-frame-modules-report.md
@dpsoft
dpsoft force-pushed the feat/unresolved-frame-modules branch from 8a29f3a to 342fd05 Compare August 25, 2026 23:55
@dpsoft
dpsoft merged commit 2dd640f into main Aug 26, 2026
10 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant