Repository navigation
Measurements: --state-in-memory beats the C backend on every title and platform tested; without it the LLVM backend loses to C #29
Description
Not a bug report — a measurement writeup, since --state-in-memory is opt-in and off by default and the numbers suggest that default is worth revisiting.
Everything below uses code already in main; nothing here needs a patch.
Result
Control is the C backend. Arms are LLVM variants from the same recompiler binary against the same DOL; only backend and state mode differ. Ratios are arm/control, so >1.0 is faster than C.
| title | --backend llvm |
--backend llvm --state-in-memory |
scenes |
|---|---|---|---|
| Luigi's Mansion (GLME01) | 1.185 | 1.472 | 2 |
| Skyward Sword (SOUE01) | 0.896 | 1.194 | 2 |
| Mario Kart: Double Dash (GM4E01) | 0.906 | 1.145 | 3 |
| Pokémon Colosseum (GC6E01) | 0.827 | 1.040 | 8 |
Two things stand out:
- Without
--state-in-memory, the LLVM backend is slower than C on three of the four titles (0.83–0.91). - With it, LLVM beats C on all four. The swing on Colosseum is 0.827 → 1.040 across 8 independent scenes.
Colosseum's figure is the most sampled: 8 scenes spanning 12.8–37.2 fps, and the llvm arm landed in 0.773–0.876 on every one of them.
Method
- runtime is ModernGekko with the static-recomp core; frames advanced in a fixed window, uncapped
- each scene is a savestate; interleaved forward and reversed control/arm pairs, medians compared
- host input pinned (no device bound,
BackgroundInput = False) and one emulator at a time — both mattered, see below - Colosseum's MSR region-leader fix (Make the instruction after an MSR write a region leader #28) is present in all arms, so it is not what is being measured here
Noise floor was measured, not assumed
The same C module was supplied as both control and arm, so the true ratio is exactly 1.0000:
n=13 null comparisons across 4 titles / 6 scenes
12 of 13 within +/-1.72%
1 of 13 at +14.97% (one scene; 3 repeats of it then returned 1.0164 / 0.9972 / 1.0019)
Single runs therefore carry a rare ~15% excursion. Every number above is a mean over repeated runs. Several earlier single-run figures in this work did not survive repetition and were discarded.
Caveats worth stating
- x86-64 Windows only. No aarch64 or macOS numbers yet, and state-in-memory could plausibly behave differently where register pressure differs.
- Scene counts are uneven (8 for Colosseum, 2–3 elsewhere).
- Savestate hygiene matters more than expected: of 38 Colosseum states available here, 17 carried a widescreen patch in their RAM image and reported
SMC: chunk [0x80005300,0x80005500) hash mismatch. They boot and render normally while that chunk silently falls back to the interpreter. Anyone reproducing this should check the runtime log forhash mismatchand confirmframe_countactually advances.
Why post it
--state-in-memory currently reads as an experimental toggle. On this hardware and these titles it is the difference between the LLVM backend being a regression against C and being a win. If that reproduces elsewhere, the default may be worth changing — and if it does not reproduce on other targets, that is worth knowing too.
Happy to run additional titles or scenes if useful.
Activity
dougchansan commentedon Aug 31, 2026
Correction. One claim in the original post is wrong, and the reason turns out to be more useful than the claim.
I re-ran everything on validated scene sets — 6 scenes per title instead of 2–3 — after scanning every available savestate for boot failures, zero frame progress, and SMC hash mismatches. 38 of 105 states (36%) were rejected, including one (
course-peach-beach.sav) that does not boot at all and had been used in an earlier run of mine.What changed
llvmllvm --state-in-memoryThe headline conclusion is unchanged and slightly stronger:
--state-in-memorywins on all four titles, now +11.7% to +57.8% rather than +4.0% to +47.2%.But the original post said plain
llvmloses to C on three of four titles. That is wrong — it is two of four. Skyward Sword flipped sign, 0.896 → 1.259. My earlier figure came from two scenes; with six it reverses.Why it flipped: the ratio depends on scene weight
Pooling all 48 measurements and splitting by how fast the C backend runs each scene (low C speed = heavy scene):
Per title:
LLVM gains on light scenes and gives ground on heavy ones, and the effect is concentrated in the two titles whose scene sets span a wide load range. So neither of my Skyward Sword numbers is "the" answer: the old one sampled two heavy scenes, the new one has four near-identical light scenes (C ≈ 3.4–3.5) dominating the mean.
A per-title mean is only meaningful alongside the scene weight distribution it was drawn from. I have quoted such means twice now and they moved both times.
What is solid
--state-in-memorybeats C on every title and every individual scene measured — 24 of 24 scene-level comparisons.--state-in-memory1.1165, SE 0.0135, n=6.What I would not claim
--state-in-memory. It is load-dependent, and the sign can change with scene selection.dougchansan commentedon Sep 1, 2026
Apple Silicon data, which the original post said it lacked.
Same DOL (
md5 c55df119fad43d7d4ee976c2242211cb, 3,779,808 bytes) and the same six validated Colosseum scenes as the x86-64 numbers, so the two platforms are directly comparable rather than approximately so.Host: MacBook M5, 10 cores, Apple clang 17, LLVM 20 (Homebrew
llvm@20, matching the 20.1.8 used on Windows). Metal backend. 3 reps per cell, arms interleaved per scene so thermal drift hits all three equally.llvmllvm --state-in-memoryCross-platform agreement
Same direction and similar magnitude on both.
--state-in-memoryturns the LLVM backend from a regression against C into a win on AArch64 as well, which is the part I was least sure would carry over: AArch64 has 31 GPRs against x86-64's 16, so if the benefit were purely about register pressure it could plausibly have vanished here. It does not.Two things worth flagging for anyone reproducing on macOS
The LLVM backend cannot build modules on macOS without #26. Every chunk aborts:
27 of these, then failure. With #26 applied: clean build, zero errors, 145.9 MB dylib. I had assumed that PR was cosmetic; it is not — it gates the whole backend on Darwin.
Partial output from a failed run is reused. After applying #26 the errors persisted until I deleted the 665 MB of partial module output the failed attempts had left behind. Worth clearing the output directory rather than assuming a fix did not work.
Caveat
The macOS figures are frames-advanced over a fixed 20s window (3 reps, interleaved). The x86-64 figures came from the interleaved forward/reversed pair harness with medians. Both control for drift, but they are not the identical statistic, so I would treat the cross-platform agreement as directional rather than as a like-for-like ratio comparison.
[-]Measurements: --state-in-memory beats the C backend on all four titles tested; without it the LLVM backend loses to C on three[/-][+]Measurements: --state-in-memory beats the C backend on every title and platform tested; without it the LLVM backend loses to C[/+]dougchansan commentedon Sep 17, 2026
Ran the same comparison on Linux and got a materially different answer for MKDD, which I think sharpens the conclusion rather than undermining it: the "without
--state-in-memory, LLVM loses to C" half appears to be host-dependent.Numbers
MKDD
race.sav, the interleaved harness ported to Linux (6 forward + 6 reversed pairs, medians decide), headless, idle box:--backend llvm--backend llvm+ SIMBoth Linux comparisons were 12/12 pairs favouring the arm, with tight deltas: +17.3%..+22.1% for
llvm, +49.4%..+60.9% for SIM. Absolute: C 32.97 → LLVM 39.57 → SIM 52.03 fps.So on this host plain LLVM does not lose to C — it wins by 20%, and SIM adds another 30 points on top.
Why the two disagree, most likely
The hosts are two generations and 2× L3 apart, and the thing SIM changes is footprint:
perf statacross the three arms on the 5900X, normalised per frame (the arms run at different fps, so per-instruction normalisation flatters the slow one):Two separable effects fall out:
That predicts exactly the split observed: on a part with enough L3 to absorb plain LLVM's footprint, the codegen win is cancelled by the footprint cost and
llvmlands at or below C — which is what the table in this issue shows. On a cache-starved part the footprint cost dominates first, so both arms win and SIM wins hugely.If that reading is right, the practical consequence cuts the way this issue argues, only harder: SIM is worth more the less cache the host has, so benchmarking it exclusively on a large-L3 part systematically understates it.
Caveats, so these are read at the right precision
LLC-*events are unsupported on this Zen 3 part, socache-missesis the last-level proxy.Host: Ubuntu, Ryzen 9 5900X (12C/24T, 64 MB L3 as 2×32 MB), RTX 3090,
performancegovernor, headless, nothing else running.