Repository navigation
test-framework: Run nes-test-roms in CI, and fix the PPU/APU bugs they found - #31
Merged
Merged
Conversation
The suite so far could only judge a ROM by how it looked: crashes, hangs,
blank or frozen screens. That says nothing about accuracy, and it is the
wrong question to ask a purpose-built test ROM, which states its result
outright. Worse, it gets it backwards -- a blargg test that has finished
parks the CPU in a `jmp *` on a screen that stops changing, which the
heuristics read as "CPU stuck in tight loop".
Teach the harness the two reporting conventions, behind a new --test:
auto blargg's suites write $DE $B0 $61 to $6001-$6003 and keep a
status byte at $6000 ($80 running, $81 wants a soft reset,
otherwise the result code, 0 = passed), with an ASCII summary
at $6004. The run stops the moment a ROM reports, so the frame
budget only bounds ROMs that never do.
nestest has no signature and is entered at $C000 rather than at its
reset vector, leaving its result in $02/$03. The vector is
patched in the PRG image held in memory, never on disk.
Polling is free: sbus_read() serves $0000-$07FF from CPU RAM and
$6000-$7FFF from PRG-RAM, neither of which has read side effects.
A ROM's own verdict wins over every heuristic, with one exception:
cpu_jammed still comes first. A CPU that hit a JAM/KIL stopped in the
middle, so what the ROM had written down is not the answer it meant to
give -- nestest leaves $02/$03 at zero for every test it never reached,
which would otherwise read as "everything passed".
--test-mode defaults to off, so the game sweep and its baseline are
untouched. ROM lists now take per-ROM overrides after " | " (a pipe, not
whitespace: ROM filenames are full of spaces).
61 ROMs, 21 PASS / 40 FAIL, ~50s including the download. All 16
instr_test-v5 singles pass, official and unofficial opcodes both.
The ROMs are fetched, not committed. blargg's readme.txt files and
nestest.txt document the tests and nothing else: no copyright notice, no
licence, no permission to redistribute, and the upstream archive has no
repository-wide LICENSE either. Running them is uncontroversial;
redistributing them from a BSD-3-Clause repository is a different act
that nobody has authorised. fetch-testroms.py downloads them from a
pinned commit and verifies a recorded SHA-256 per file, so a force-push
upstream or a truncated download fails the fetch instead of quietly
changing a verdict. roms/external/ is git-ignored and CI caches it.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Signed-off-by: Renê de Souza Pinto <rene@renesp.com.br>
Two of the checks CI performs existed only inside the workflow file: the unit tests and the game smoke sweep, the latter with its --rom-list and --baseline flags spelled out in YAML. Reproducing the job before pushing meant reading rom-tests.yml and retyping it. Add the missing targets -- unit-tests and smoke -- and a `check` that runs the whole job in CI's order: strict, the harness build, unit-tests, smoke, testroms. About a minute from scratch, most of it the test ROMs. Point the workflow at the same targets, so the flags live in the Makefile only and the two cannot drift. smoke takes SMOKE_EXTRA for CI's reporting flags, the way testroms already takes TESTROM_EXTRA. `check` calls sub-makes rather than listing prerequisites: these are ordered steps, and `make -j` may start prerequisites in any order or all at once. Verified that `make -j4 check` still runs them in sequence. Also add `help`, matching src/Makefile's convention. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Signed-off-by: Renê de Souza Pinto <rene@renesp.com.br>
The baseline was recorded against the core as it stood before 59d83e8 ("cpu: Update cpu650x emulator code"); rebasing the test framework onto the current history left eight ROMs recorded as FAIL that now pass: ppu_vbl_nmi 01-vbl_basics, 03-vbl_clear_time, 09-even_odd_frames apu_test 1-len_ctr, 4-jitter, 6-irq_flag_timing, 7-dmc_basics instr_timing 2-branch_timing Attributed, not assumed: building the harness with src/cpu/ taken from 59d83e8~1 and everything else at HEAD flips all eight back to FAIL. Leaving them stale was not harmless. compare() only fails on PASS -> FAIL, so CI stayed green and reported the same eight "fixes" on every run -- and with the baseline still saying FAIL, a regression in any of them would have compared FAIL to FAIL and gone unnoticed. Recording them makes them gated. Sixteen more entries keep their FAIL but change reason: the same core update carries them further into their suite before they stop. The verdict totals move from 21/40 to 29 PASS / 32 FAIL over 61 ROMs. Two consecutive runs produced identical verdicts and identical reason text, so this is the core's behaviour and not run-to-run noise. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Signed-off-by: Renê de Souza Pinto <rene@renesp.com.br>
nestest's unattended sweep ends with the RTS at $C66E, the last instruction of the reference trace. The harness enters the ROM by pointing its reset vector at $C000, which is a JMP -- so that RTS has no return address to pop, returns into whatever the stack holds, and everything after it is derailed code that shortly executes a JAM. The harness was running on past that point to the end of the frame and then refusing to trust $02/$03 because the CPU had jammed. The jam was an artifact of how the ROM is entered, not a result: the sweep itself matches the reference trace instruction for instruction. Watch for $C66E and stop clocking the moment it is reached, mid-frame rather than at the frame boundary, so the derailed code never runs. Reaching it is also a better answer to "did the sweep finish?" than "did the CPU avoid jamming?" -- every test the sweep never reached leaves its result byte at zero, which would otherwise read as a pass. nestest now reports $02=00 $03=00 after 8990 instructions, with no jam: every one of its ~400 CPU tests passes, unofficial opcodes included. 32 PASS / 29 FAIL over the suite. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Signed-off-by: Renê de Souza Pinto <rene@renesp.com.br>
A soft reset leaves PRG-RAM untouched, so $6000 still reads $81 for as long as it takes the ROM to restart and write $80 back. The harness read that as a second request and reset the ROM again, at whatever arbitrary point it had reached -- which is precisely what a reset test measures. cpu_reset/registers saw its carefully staged registers replaced by whatever the second reset caught, and ram_after_reset looped until it hit the 8-reset cap and never reported at all. Wait for $80 before honouring another $81. The cap stays as the backstop for a ROM that really does ask over and over. cpu_reset/registers and ram_after_reset now pass; registers reports A=34 X=56 Y=78 P=FF S=0F, the state it set up with S down by 3 and I set. 34 PASS / 27 FAIL over the suite. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Signed-off-by: Renê de Souza Pinto <rene@renesp.com.br>
blargg's older test ROMs -- sprite_hit_tests, sprite_overflow_tests and
the rest of that generation -- predate the $6000 report. They keep a
result byte at $F8, print it on screen, beep it, and then park the CPU in
a `jmp` to itself. The framework had no way to read that, so all 16 of
the sprite ROMs came out as "no protocol" and fell through to the screen
heuristics, which score a finished test as a hang.
Add `--test blargg2005`. Two details make it work:
- $F8 holds the number of the test currently running, replaced by 1 if
every test passed, so its value alone does not say whether the run is
over. The park is what says that.
- "Parked" cannot just be "few distinct PCs this frame": a vblank wait
is two instructions and can span most of a frame. It cannot span
three, so the signal is a streak of quiet frames.
Reading $F8 rather than the screen also makes the runs short -- these ROMs
report within 30 to 124 frames, where the screen heuristics needed the
whole 5400-frame budget to conclude nothing.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Signed-off-by: Renê de Souza Pinto <rene@renesp.com.br>
Sixteen more ROMs, fetched from the same pinned commit and checksummed like the rest: the eleven sprite_hit_tests, which cover sprite 0 hit down to which pixel it first reports on, and the five sprite_overflow_tests, which cover the flag in $2002 bit 5. Both suites use blargg's older runtime, so they carry test=blargg2005 in the ROM list. Together they close the largest gap left in the suite. Nothing was checking sprite 0 hit at all, and a game that gets it wrong shows it as a status bar that shakes or a split that lands on the wrong scanline. The fetched bytes are identical to the copies in the resources tree. Recorded as 55 PASS / 22 FAIL over 77 ROMs. All eleven sprite hit tests pass, timing included; of the overflow tests 1.Basics and 5.Emulator pass and three fail, the first of which the next commit fixes. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Signed-off-by: Renê de Souza Pinto <rene@renesp.com.br>
The baseline came from the imp-ci branch, which carries emulator changes main does not have yet (the PPU open-bus decay register, reset handling, dummy reads of indexed addressing). Record what this tree produces: 2.Details.nes now passes; cpu_dummy_writes_ppumem, ppu_open_bus, registers and test_cpu_exec_space_ppuio fail until those changes land. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Renê de Souza Pinto <rene@renesp.com.br>
The PPU data bus has no driver of its own. Reading a write-only port
returns the charge the bus is still holding from the last time something
drove it, and that charge leaks away in roughly 600 ms. rNES held a single
`gen_latch` that never decayed, and doubled it up as the $2007 read
buffer, so the two states overwrote each other.
Give the decay register its own three refresh groups -- bits 4-0, bit 5,
bits 7-6 -- each with its own timer, because a read only refreshes the
bits the PPU actually drove:
$2002 drives 7-5 from the status register and leaves 4-0 to the
bus, so it refreshes bit 5 and bits 7-6 only
$2007 drives all eight from VRAM, but only 5-0 from palette
memory, which is 6 bits wide -- a palette read leaves
bits 7-6 to decay
$2004 drives all eight
write-only drives nothing, refreshes nothing
any write drives the whole bus
$2007 gets the read buffer it needs of its own, so "one read behind" and
"what the bus is holding" stop being the same byte. The dead
`PPUSTATUS.open_bus` bitfield, which was ORed into every write-only read
and was always zero, goes with it.
https://www.nesdev.org/wiki/PPU_registers#Ports
blargg ppu_open_bus, cpu_dummy_writes_ppumem and
test_cpu_exec_space_ppuio now pass. 55 PASS / 22 FAIL over the test ROM
suite.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Signed-off-by: Renê de Souza Pinto <rene@renesp.com.br>
In the 5-step sequence, step 4 clocks both the envelopes and the length counters and sweeps, as step 3 does in the 4-step sequence. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Renê de Souza Pinto <rene@renesp.com.br>
apu_reset() puts the APU back in its power-on state, which is not what the reset button does. On a reset the APU clears $4015, rewrites $4017 with the last value written to it (so the frame counter keeps its mode and restarts, with its IRQ flag cleared), and zeroes and un-halts the pulse and noise length counters, leaving the triangle's alone. Add apu_soft_reset() for that, and remember the last $4017 write so it can be replayed. https://www.nesdev.org/wiki/CPU_power_up_state Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Renê de Souza Pinto <rene@renesp.com.br>
A blargg reset request stands for the console's reset button, so the APU should get apu_soft_reset() rather than its power-on reset. blargg apu_reset/4017_written and len_ctrs_enabled now pass. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Renê de Souza Pinto <rene@renesp.com.br>
Once secondary OAM holds 8 sprites, the PPU keeps scanning for a 9th to set the overflow flag, but a hardware bug makes every sprite that is not in range advance the byte index m along with the sprite index n. The check then reads the tile, attribute or X byte of later sprites as if it were their Y coordinate, so the flag can be set by sprites that are not on the line and missed for sprites that are. rNES compared the real Y of every remaining sprite, which is the behaviour the hardware was meant to have. Evaluate the first 8 sprites as before and model the diagonal scan for the overflow check. https://www.nesdev.org/wiki/PPU_sprite_evaluation blargg sprite_overflow_tests/4.Obscure now passes. The 720 games of the local ROM sweep are unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Renê de Souza Pinto <rene@renesp.com.br>
blargg's 2005-era runtime moved its result byte during that year. The October suites already in CI (sprite_hit_tests, sprite_overflow_tests) keep it at $F8, but the July and September ones -- blargg_apu_2005.07.30 and blargg_ppu_tests_2005.09.15b -- keep it at $F0, so the harness read an untouched $F8 and reported every one of them as "failed #0". Add --result-addr to the harness, defaulting to $F8, and a matching result-addr= per-ROM option in the ROM list. Document the blargg2005 mode, which the README's table of test conventions was missing. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Renê de Souza Pinto <rene@renesp.com.br>
A ROM's baseline key is its filename. The same filename in two suites -- sprite_overflow_tests and mmc3_irq_tests both have a 2.Details.nes -- gives both ROMs one baseline entry, so each run overwrites the other's verdict and a regression in either can hide behind the other. Reject such a list, naming both lines. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Renê de Souza Pinto <rene@renesp.com.br>
Survey the rest of christopherpow/nes-test-roms at the pinned commit for ROMs that report their own verdict, run on a mapper rNES implements and expect an NTSC console, and add the 35 that qualify: blargg_apu_2005.07.30 11 APU, result at $F0 blargg_ppu_tests_2005.09.15b 5 PPU, result at $F0 branch_timing_tests 3 CPU branch cycles vbl_nmi_timing 7 PPU VBL/NMI timing mmc3_irq_tests 6 mapper 4 scanline IRQ sprdma_and_dmc_dma 2 overlapping DMA ppu_read_buffer 1 $2007 read buffer (Bisqwit) mmc3_irq_tests/2.Details.nes lands as 2.MMC3_Details.nes, since sprite_overflow_tests has a 2.Details.nes and the baseline is keyed by filename. 17 pass and 18 fail, recorded in the baseline. No existing verdict moves. Left out: suites that report only on screen (cpu_dummy_reads, dmc_dma_during_read4, read_joy3, cpu_timing_test6, blargg_nes_cpu_test5), the PAL-only pal_apu_tests, mappers rNES lacks (5, 22, 28, 34), combined ROMs that repeat singles CI already runs, and mmc3_test/mmc3_test_2, which write their report to PRG RAM that rNES's MMC3 keeps disabled until $A001 enables it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Renê de Souza Pinto <rene@renesp.com.br>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
This PR runs the public NES accuracy test ROMs in CI, and fixes four
accuracy bugs that those ROMs found.
Test ROMs in CI. The ROM smoke-test workflow now fetches 112 test ROMs
from christopherpow/nes-test-roms,
pinned to a commit, and checks each one's SHA-256 before running it. The ROMs
cover blargg's CPU, PPU, APU and MMC3 suites, Bisqwit's
ppu_read_bufferandkevtris's nestest. The ROMs are not committed: none of them carries a licence
(see
test-framework/roms/README.md).Each ROM reports its own verdict, and the harness reads it from memory rather
than judging the screen:
$6000report$F8or$F0(
--result-addr)$02/$03Results are compared against
testroms-baseline.json. A ROM that used to passand now fails breaks the build. A ROM that was already failing stays in the
list, so a fix shows up as
FIXED.make -C test-framework checkruns thesame job locally.
Harness fixes that make these verdicts trustworthy:
$C66E)Accuracy fixes:
PPU open-bus decay register (
ppu.c). The PPU data latch now decays ason hardware, in three groups of bits, each refreshed only by the reads that
drive it.
$2007gets its own read buffer.NESdev: PPU registers, Ports
APU frame counter, mode 1 (
apu.c). Step 4 of the 5-step sequence neverclocked the length counters, sweeps and envelopes, so in mode 1 they ran at
half speed.
NESdev: APU Frame Counter
APU soft reset (
apu.c). The newapu_soft_reset()models the resetbutton, separately from power-on:
$4015is cleared$4017value is written againThe headless harness uses it to serve blargg's reset requests. The SDL
frontend has no reset button, so it still only does a power-on reset.
NESdev: CPU power up state
Sprite overflow hardware bug (
ppu.c). After 8 sprites are found, theoverflow check now reads the wrong OAM byte for each later sprite, as the
hardware does. That produces the same false positives and false negatives
as a real console.
NESdev: PPU sprite evaluation
None of these changes trades accuracy, memory or portability against another.
The new state is a few bytes: the decay timers, the read buffer and the last
$4017value. Nothing is allocated, and nothing platform-specific is added tothe core.
How to test and validate this PR
Build:
cd src && make. There are no new dependencies. The test ROMs aredownloaded by
make -C test-framework testroms, using only Python's standardlibrary.
Test ROMs:
make -C test-framework check. This runs the full CI job:-WerrorResults:
mainemulator, original 77 ROMsSix ROMs are fixed by this PR, and none regresses:
ppu_open_buscpu_dummy_writes_ppumemtest_cpu_exec_space_ppuio4017_writtenlen_ctrs_enabledsprite_overflow_tests/4.ObscureOf the 37 failures, most are cycle-timing tests (
ppu_vbl_nmi,vbl_nmi_timing,cpu_interrupts_v2,instr_timing). The instruction-steppedCPU core can't pass those yet. Others are MMC3 IRQ details and
registers,which needs a CPU reset that keeps A/X/Y. That fix belongs in upstream
MOS650x.
Games: a local sweep of 790 games, before and after the sprite overflow
change, gave 720 PASS and 70 SKIP (unimplemented mappers) both times.
Platform: Debian GNU/Linux forky/sid (kernel 7.2.11), GCC 16.2.0,
SDL2 2.32.74, native build. Also checked: clang-format 21 (
make format-src)and cppcheck 2.22.
Checklist
git commit -s)Co-Authored-By:line🤖 Generated with Claude Code