Skip to content

test-framework: Run nes-test-roms in CI, and fix the PPU/APU bugs they found - #31

Merged
rene merged 16 commits into
mainfrom
test-framework-ci
Oct 25, 2026
Merged

rene merged 16 commits into
mainfrom
test-framework-ci

Conversation

@rene

@rene rene commented Oct 25, 2026

Copy link
Copy Markdown
Owner

Description

This PR runs the public NES accuracy test ROMs in CI, and fixes four
accuracy bugs that those ROMs found.

Test ROMs in CI. The ROM smoke-test workflow now fetches 112 test ROMs
from christopherpow/nes-test-roms,
pinned to a commit, and checks each one's SHA-256 before running it. The ROMs
cover blargg's CPU, PPU, APU and MMC3 suites, Bisqwit's ppu_read_buffer and
kevtris's nestest. The ROMs are not committed: none of them carries a licence
(see test-framework/roms/README.md).

Each ROM reports its own verdict, and the harness reads it from memory rather
than judging the screen:

  • blargg's $6000 report
  • the result byte of blargg's 2005-era runtime, at $F8 or $F0
    (--result-addr)
  • nestest's $02/$03

Results are compared against testroms-baseline.json. A ROM that used to pass
and now fails breaks the build. A ROM that was already failing stays in the
list, so a fix shows up as FIXED. make -C test-framework check runs the
same job locally.

Harness fixes that make these verdicts trustworthy:

  • a blargg reset request is served only once
  • the nestest sweep stops at its documented end ($C66E)
  • a ROM list with two ROMs under the same baseline key is refused

Accuracy fixes:

  • PPU open-bus decay register (ppu.c). The PPU data latch now decays as
    on hardware, in three groups of bits, each refreshed only by the reads that
    drive it. $2007 gets its own read buffer.
    NESdev: PPU registers, Ports

  • APU frame counter, mode 1 (apu.c). Step 4 of the 5-step sequence never
    clocked the length counters, sweeps and envelopes, so in mode 1 they ran at
    half speed.
    NESdev: APU Frame Counter

  • APU soft reset (apu.c). The new apu_soft_reset() models the reset
    button, separately from power-on:

    • $4015 is cleared
    • the last $4017 value is written again
    • the pulse and noise length counters are zeroed and un-halted
    • the triangle is left alone

    The headless harness uses it to serve blargg's reset requests. The SDL
    frontend has no reset button, so it still only does a power-on reset.
    NESdev: CPU power up state

  • Sprite overflow hardware bug (ppu.c). After 8 sprites are found, the
    overflow check now reads the wrong OAM byte for each later sprite, as the
    hardware does. That produces the same false positives and false negatives
    as a real console.
    NESdev: PPU sprite evaluation

None of these changes trades accuracy, memory or portability against another.
The new state is a few bytes: the decay timers, the read buffer and the last
$4017 value. Nothing is allocated, and nothing platform-specific is added to
the core.

How to test and validate this PR

  • Build: cd src && make. There are no new dependencies. The test ROMs are
    downloaded by make -C test-framework testroms, using only Python's standard
    library.

  • Test ROMs: make -C test-framework check. This runs the full CI job:

    • the harness with -Werror
    • the unit tests (75, all passing)
    • the game smoke sweep
    • the 112 test ROMs

    Results:

    PASS FAIL
    main emulator, original 77 ROMs 52 25
    this PR, same 77 ROMs 58 19
    this PR, all 112 ROMs 75 37

    Six ROMs are fixed by this PR, and none regresses:

    Fixed ROM Fixed by
    ppu_open_bus open-bus decay register
    cpu_dummy_writes_ppumem open-bus decay register
    test_cpu_exec_space_ppuio open-bus decay register
    4017_written APU mode 1 fix + soft reset
    len_ctrs_enabled APU mode 1 fix + soft reset
    sprite_overflow_tests/4.Obscure sprite overflow bug

    Of the 37 failures, most are cycle-timing tests (ppu_vbl_nmi,
    vbl_nmi_timing, cpu_interrupts_v2, instr_timing). The instruction-stepped
    CPU core can't pass those yet. Others are MMC3 IRQ details and registers,
    which needs a CPU reset that keeps A/X/Y. That fix belongs in upstream
    MOS650x.

  • Games: a local sweep of 790 games, before and after the sprite overflow
    change, gave 720 PASS and 70 SKIP (unimplemented mappers) both times.

  • Platform: Debian GNU/Linux forky/sid (kernel 7.2.11), GCC 16.2.0,
    SDL2 2.32.74, native build. Also checked: clang-format 21 (make format-src)
    and cppcheck 2.22.

Checklist

  • I've provided a proper description
  • All my commits are signed off (git commit -s)
  • If a model helped write this, every commit it contributed to names it in a Co-Authored-By: line
  • I've added the proper documentation
  • No copyrighted ROMs are attached or committed
  • I've tested my PR on Linux
  • I've tested my PR on Windows (optional)

🤖 Generated with Claude Code

rene and others added 16 commits October 25, 2026 18:46
The suite so far could only judge a ROM by how it looked: crashes, hangs,
blank or frozen screens. That says nothing about accuracy, and it is the
wrong question to ask a purpose-built test ROM, which states its result
outright. Worse, it gets it backwards -- a blargg test that has finished
parks the CPU in a `jmp *` on a screen that stops changing, which the
heuristics read as "CPU stuck in tight loop".

Teach the harness the two reporting conventions, behind a new --test:

  auto     blargg's suites write $DE $B0 $61 to $6001-$6003 and keep a
           status byte at $6000 ($80 running, $81 wants a soft reset,
           otherwise the result code, 0 = passed), with an ASCII summary
           at $6004. The run stops the moment a ROM reports, so the frame
           budget only bounds ROMs that never do.
  nestest  has no signature and is entered at $C000 rather than at its
           reset vector, leaving its result in $02/$03. The vector is
           patched in the PRG image held in memory, never on disk.

Polling is free: sbus_read() serves $0000-$07FF from CPU RAM and
$6000-$7FFF from PRG-RAM, neither of which has read side effects.

A ROM's own verdict wins over every heuristic, with one exception:
cpu_jammed still comes first. A CPU that hit a JAM/KIL stopped in the
middle, so what the ROM had written down is not the answer it meant to
give -- nestest leaves $02/$03 at zero for every test it never reached,
which would otherwise read as "everything passed".

--test-mode defaults to off, so the game sweep and its baseline are
untouched. ROM lists now take per-ROM overrides after " | " (a pipe, not
whitespace: ROM filenames are full of spaces).

61 ROMs, 21 PASS / 40 FAIL, ~50s including the download. All 16
instr_test-v5 singles pass, official and unofficial opcodes both.

The ROMs are fetched, not committed. blargg's readme.txt files and
nestest.txt document the tests and nothing else: no copyright notice, no
licence, no permission to redistribute, and the upstream archive has no
repository-wide LICENSE either. Running them is uncontroversial;
redistributing them from a BSD-3-Clause repository is a different act
that nobody has authorised. fetch-testroms.py downloads them from a
pinned commit and verifies a recorded SHA-256 per file, so a force-push
upstream or a truncated download fails the fetch instead of quietly
changing a verdict. roms/external/ is git-ignored and CI caches it.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Signed-off-by: Renê de Souza Pinto <rene@renesp.com.br>
Two of the checks CI performs existed only inside the workflow file: the
unit tests and the game smoke sweep, the latter with its --rom-list and
--baseline flags spelled out in YAML. Reproducing the job before pushing
meant reading rom-tests.yml and retyping it.

Add the missing targets -- unit-tests and smoke -- and a `check` that
runs the whole job in CI's order: strict, the harness build, unit-tests,
smoke, testroms. About a minute from scratch, most of it the test ROMs.

Point the workflow at the same targets, so the flags live in the
Makefile only and the two cannot drift. smoke takes SMOKE_EXTRA for CI's
reporting flags, the way testroms already takes TESTROM_EXTRA.

`check` calls sub-makes rather than listing prerequisites: these are
ordered steps, and `make -j` may start prerequisites in any order or all
at once. Verified that `make -j4 check` still runs them in sequence.

Also add `help`, matching src/Makefile's convention.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Signed-off-by: Renê de Souza Pinto <rene@renesp.com.br>
The baseline was recorded against the core as it stood before
59d83e8 ("cpu: Update cpu650x emulator code"); rebasing the test
framework onto the current history left eight ROMs recorded as FAIL
that now pass:

  ppu_vbl_nmi  01-vbl_basics, 03-vbl_clear_time, 09-even_odd_frames
  apu_test     1-len_ctr, 4-jitter, 6-irq_flag_timing, 7-dmc_basics
  instr_timing 2-branch_timing

Attributed, not assumed: building the harness with src/cpu/ taken from
59d83e8~1 and everything else at HEAD flips all eight back to FAIL.

Leaving them stale was not harmless. compare() only fails on PASS ->
FAIL, so CI stayed green and reported the same eight "fixes" on every
run -- and with the baseline still saying FAIL, a regression in any of
them would have compared FAIL to FAIL and gone unnoticed. Recording
them makes them gated.

Sixteen more entries keep their FAIL but change reason: the same core
update carries them further into their suite before they stop. The
verdict totals move from 21/40 to 29 PASS / 32 FAIL over 61 ROMs.

Two consecutive runs produced identical verdicts and identical reason
text, so this is the core's behaviour and not run-to-run noise.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Signed-off-by: Renê de Souza Pinto <rene@renesp.com.br>
nestest's unattended sweep ends with the RTS at $C66E, the last
instruction of the reference trace. The harness enters the ROM by pointing
its reset vector at $C000, which is a JMP -- so that RTS has no return
address to pop, returns into whatever the stack holds, and everything
after it is derailed code that shortly executes a JAM.

The harness was running on past that point to the end of the frame and
then refusing to trust $02/$03 because the CPU had jammed. The jam was an
artifact of how the ROM is entered, not a result: the sweep itself matches
the reference trace instruction for instruction.

Watch for $C66E and stop clocking the moment it is reached, mid-frame
rather than at the frame boundary, so the derailed code never runs.
Reaching it is also a better answer to "did the sweep finish?" than
"did the CPU avoid jamming?" -- every test the sweep never reached leaves
its result byte at zero, which would otherwise read as a pass.

nestest now reports $02=00 $03=00 after 8990 instructions, with no jam:
every one of its ~400 CPU tests passes, unofficial opcodes included.
32 PASS / 29 FAIL over the suite.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Signed-off-by: Renê de Souza Pinto <rene@renesp.com.br>
A soft reset leaves PRG-RAM untouched, so $6000 still reads $81 for as
long as it takes the ROM to restart and write $80 back. The harness read
that as a second request and reset the ROM again, at whatever arbitrary
point it had reached -- which is precisely what a reset test measures.

cpu_reset/registers saw its carefully staged registers replaced by
whatever the second reset caught, and ram_after_reset looped until it hit
the 8-reset cap and never reported at all.

Wait for $80 before honouring another $81. The cap stays as the backstop
for a ROM that really does ask over and over.

cpu_reset/registers and ram_after_reset now pass; registers reports
A=34 X=56 Y=78 P=FF S=0F, the state it set up with S down by 3 and I set.
34 PASS / 27 FAIL over the suite.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Signed-off-by: Renê de Souza Pinto <rene@renesp.com.br>
blargg's older test ROMs -- sprite_hit_tests, sprite_overflow_tests and
the rest of that generation -- predate the $6000 report. They keep a
result byte at $F8, print it on screen, beep it, and then park the CPU in
a `jmp` to itself. The framework had no way to read that, so all 16 of
the sprite ROMs came out as "no protocol" and fell through to the screen
heuristics, which score a finished test as a hang.

Add `--test blargg2005`. Two details make it work:

  - $F8 holds the number of the test currently running, replaced by 1 if
    every test passed, so its value alone does not say whether the run is
    over. The park is what says that.
  - "Parked" cannot just be "few distinct PCs this frame": a vblank wait
    is two instructions and can span most of a frame. It cannot span
    three, so the signal is a streak of quiet frames.

Reading $F8 rather than the screen also makes the runs short -- these ROMs
report within 30 to 124 frames, where the screen heuristics needed the
whole 5400-frame budget to conclude nothing.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Signed-off-by: Renê de Souza Pinto <rene@renesp.com.br>
Sixteen more ROMs, fetched from the same pinned commit and checksummed
like the rest: the eleven sprite_hit_tests, which cover sprite 0 hit down
to which pixel it first reports on, and the five sprite_overflow_tests,
which cover the flag in $2002 bit 5. Both suites use blargg's older
runtime, so they carry test=blargg2005 in the ROM list.

Together they close the largest gap left in the suite. Nothing was
checking sprite 0 hit at all, and a game that gets it wrong shows it as a
status bar that shakes or a split that lands on the wrong scanline.

The fetched bytes are identical to the copies in the resources tree.

Recorded as 55 PASS / 22 FAIL over 77 ROMs. All eleven sprite hit tests
pass, timing included; of the overflow tests 1.Basics and 5.Emulator pass
and three fail, the first of which the next commit fixes.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Signed-off-by: Renê de Souza Pinto <rene@renesp.com.br>
The baseline came from the imp-ci branch, which carries emulator changes
main does not have yet (the PPU open-bus decay register, reset handling,
dummy reads of indexed addressing). Record what this tree produces:
2.Details.nes now passes; cpu_dummy_writes_ppumem, ppu_open_bus,
registers and test_cpu_exec_space_ppuio fail until those changes land.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Renê de Souza Pinto <rene@renesp.com.br>
The PPU data bus has no driver of its own. Reading a write-only port
returns the charge the bus is still holding from the last time something
drove it, and that charge leaks away in roughly 600 ms. rNES held a single
`gen_latch` that never decayed, and doubled it up as the $2007 read
buffer, so the two states overwrote each other.

Give the decay register its own three refresh groups -- bits 4-0, bit 5,
bits 7-6 -- each with its own timer, because a read only refreshes the
bits the PPU actually drove:

  $2002        drives 7-5 from the status register and leaves 4-0 to the
               bus, so it refreshes bit 5 and bits 7-6 only
  $2007        drives all eight from VRAM, but only 5-0 from palette
               memory, which is 6 bits wide -- a palette read leaves
               bits 7-6 to decay
  $2004        drives all eight
  write-only   drives nothing, refreshes nothing
  any write    drives the whole bus

$2007 gets the read buffer it needs of its own, so "one read behind" and
"what the bus is holding" stop being the same byte. The dead
`PPUSTATUS.open_bus` bitfield, which was ORed into every write-only read
and was always zero, goes with it.

https://www.nesdev.org/wiki/PPU_registers#Ports

blargg ppu_open_bus, cpu_dummy_writes_ppumem and
test_cpu_exec_space_ppuio now pass. 55 PASS / 22 FAIL over the test ROM
suite.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Signed-off-by: Renê de Souza Pinto <rene@renesp.com.br>
In the 5-step sequence, step 4 clocks both the envelopes and the length
counters and sweeps, as step 3 does in the 4-step sequence.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Renê de Souza Pinto <rene@renesp.com.br>
apu_reset() puts the APU back in its power-on state, which is not what
the reset button does. On a reset the APU clears $4015, rewrites $4017
with the last value written to it (so the frame counter keeps its mode
and restarts, with its IRQ flag cleared), and zeroes and un-halts the
pulse and noise length counters, leaving the triangle's alone.

Add apu_soft_reset() for that, and remember the last $4017 write so it
can be replayed.

https://www.nesdev.org/wiki/CPU_power_up_state

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Renê de Souza Pinto <rene@renesp.com.br>
A blargg reset request stands for the console's reset button, so the
APU should get apu_soft_reset() rather than its power-on reset.

blargg apu_reset/4017_written and len_ctrs_enabled now pass.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Renê de Souza Pinto <rene@renesp.com.br>
Once secondary OAM holds 8 sprites, the PPU keeps scanning for a 9th to
set the overflow flag, but a hardware bug makes every sprite that is not
in range advance the byte index m along with the sprite index n. The
check then reads the tile, attribute or X byte of later sprites as if it
were their Y coordinate, so the flag can be set by sprites that are not
on the line and missed for sprites that are.

rNES compared the real Y of every remaining sprite, which is the
behaviour the hardware was meant to have. Evaluate the first 8 sprites
as before and model the diagonal scan for the overflow check.

https://www.nesdev.org/wiki/PPU_sprite_evaluation

blargg sprite_overflow_tests/4.Obscure now passes. The 720 games of the
local ROM sweep are unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Renê de Souza Pinto <rene@renesp.com.br>
blargg's 2005-era runtime moved its result byte during that year. The
October suites already in CI (sprite_hit_tests, sprite_overflow_tests)
keep it at $F8, but the July and September ones -- blargg_apu_2005.07.30
and blargg_ppu_tests_2005.09.15b -- keep it at $F0, so the harness read
an untouched $F8 and reported every one of them as "failed #0".

Add --result-addr to the harness, defaulting to $F8, and a matching
result-addr= per-ROM option in the ROM list. Document the blargg2005
mode, which the README's table of test conventions was missing.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Renê de Souza Pinto <rene@renesp.com.br>
A ROM's baseline key is its filename. The same filename in two suites --
sprite_overflow_tests and mmc3_irq_tests both have a 2.Details.nes --
gives both ROMs one baseline entry, so each run overwrites the other's
verdict and a regression in either can hide behind the other.

Reject such a list, naming both lines.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Renê de Souza Pinto <rene@renesp.com.br>
Survey the rest of christopherpow/nes-test-roms at the pinned commit for
ROMs that report their own verdict, run on a mapper rNES implements and
expect an NTSC console, and add the 35 that qualify:

  blargg_apu_2005.07.30         11  APU, result at $F0
  blargg_ppu_tests_2005.09.15b   5  PPU, result at $F0
  branch_timing_tests            3  CPU branch cycles
  vbl_nmi_timing                 7  PPU VBL/NMI timing
  mmc3_irq_tests                 6  mapper 4 scanline IRQ
  sprdma_and_dmc_dma             2  overlapping DMA
  ppu_read_buffer                1  $2007 read buffer (Bisqwit)

mmc3_irq_tests/2.Details.nes lands as 2.MMC3_Details.nes, since
sprite_overflow_tests has a 2.Details.nes and the baseline is keyed by
filename.

17 pass and 18 fail, recorded in the baseline. No existing verdict moves.

Left out: suites that report only on screen (cpu_dummy_reads,
dmc_dma_during_read4, read_joy3, cpu_timing_test6, blargg_nes_cpu_test5),
the PAL-only pal_apu_tests, mappers rNES lacks (5, 22, 28, 34), combined
ROMs that repeat singles CI already runs, and mmc3_test/mmc3_test_2,
which write their report to PRG RAM that rNES's MMC3 keeps disabled
until $A001 enables it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Renê de Souza Pinto <rene@renesp.com.br>
@rene
rene merged commit 0c5b3eb into main Oct 25, 2026
4 checks passed
@rene
rene deleted the test-framework-ci branch October 25, 2026 19:02
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant