Skip to content

Machine code monitor debugger - #705

Open
chrisgleissner wants to merge 98 commits into
GideonZ:masterfrom
chrisgleissner:feature/machine-code-monitor-debug
Open

chrisgleissner wants to merge 98 commits into
GideonZ:masterfrom
chrisgleissner:feature/machine-code-monitor-debug

Conversation

@chrisgleissner

@chrisgleissner chrisgleissner commented Jun 6, 2026

Copy link
Copy Markdown
Collaborator

Overview

This PR adds a step and breakpoint debugger to the Machine Code Monitor's Assembly view.

It provides:

  • Step Over, Step Into, Step Out, Continue, and Continue To Cursor.
  • Up to 10 breakpoints.
  • Live CPU state and prediction of the next execution target.
  • Debugging of RAM, RAM under ROM, and, on the Ultimate 64, visible BASIC and KERNAL ROM.
  • Telnet, UI Overlay, and UI Freeze modes.

The latest release-gate and regression runs pass on both the Ultimate 64 Elite I and a U2+L in a C64 Ultimate. The previously open cartridge regression failures are resolved. Full results, earlier coverage and remaining test-rig limitations are consolidated under Testing.

Documentation PR

Demo: debugging a background-colour loop, KERNAL and BASIC code


Features

For full details see the Debug Mode chapter of the Machine Code Monitor documentation.

Execution control

D enters Debug mode from the Assembly view; C=+D or RUN/STOP leaves it.

Action Key Behaviour
Step Over D Executes the instruction at the program counter. For a JSR it plants a breakpoint at the return site and lets the whole subroutine run, so a call into ROM or RAM under ROM completes without a manual breakpoint. Any other instruction is a single step.
Step Into T Executes exactly one instruction. A JSR lands on the first instruction of the callee.
Step Out U Runs to the caller of the current subroutine and stops there.
Continue G Runs until an enabled breakpoint is reached.
Continue To Cursor K Plants a temporary breakpoint at the Assembly cursor and runs until it is reached. Enabled breakpoints on the way still stop the run.

Inside Debug mode U is Step Out rather than the Assembly view's undocumented-opcode toggle. O still cycles the monitor's view bank and never changes which instruction stream the CPU executes.

Step Out returns to the caller of the frame the CPU is really in, so it works both after a Step Into and after arriving inside a subroutine with G or K.

Two sources describe that frame: the frames Step Into recorded, and the return address on the live $0100 stack. The live stack is trusted only when a JSR really sits three bytes before what its top two bytes point at. When neither source yields an active frame, Step Out reports NOT IN SUBROUTINE.

Live CPU state

The debug footer shows:

  • Program counter.
  • Accumulator, X register, Y register, and stack pointer.
  • Processor flags.
  • IRQ and NMI vectors at $0314/$0315 and $0318/$0319.
  • Predicted jump, branch, call, and return targets.

Active flags and important values are highlighted. A branch target is highlighted only when the branch will be taken. The next instruction is marked in the Assembly view with >...<, so the cursor can move elsewhere while the current execution position stays visible.

Breakpoints

Ten non-persistent slots.

Action Key
Toggle a breakpoint on the cursor line P
Open the breakpoint list C=+P
Jump to a breakpoint slot 0 to 9
Change its label L
Set it to the cursor address S
Enable or disable it E
Delete it DEL

Breakpoints appear as [BRKx], where x is the slot number; a custom label replaces this with [LABL]. Only enabled breakpoints stop execution, and G, K, Step Over, Step Into and Step Out all honour them.

Each slot records the backing store it was placed in, so a breakpoint set in RAM under a banked-out ROM stays where it was put rather than following the current view. RAM breakpoints work on both machines. On the Ultimate 64, visible-ROM breakpoints temporarily modify the FPGA's writable copies of the BASIC and KERNAL images; persistent ROM storage is never changed.

Monitor integration

Debug mode extends the Assembly view rather than replacing the monitor's other functionality.

  • Memory, ASCII, Screen Code, Binary and Assembly views remain available.
  • Memory can be inspected or edited without ending the debug session.
  • Edit mode and Debug mode can be active at the same time.
  • C=+X resets the C64 and returns the monitor to a clean state.

Screenshots

Debugger

Paused in the KERNAL SCNKEY routine. Dbg marks Debug mode active, [KEY] a labelled breakpoint at $EA87, the CPU stopped at $EA98, and the highlighted target $EAFB shows the branch will be taken.

Debugger paused in the KERNAL SCNKEY routine

Breakpoint list

The breakpoint popup follows the existing bookmark-list controls.

Debugger breakpoint list

Debug help

Shortcuts whose meaning changes while Debug mode is active are shown at the top of the help screen.

Machine Code Monitor debug help

Design

BRK-based debugging

The FPGA core offers the application-hosted monitor neither hardware breakpoints nor direct access to the 6510 registers, so the debugger stops execution by temporarily replacing instructions with BRK.

For each temporary breakpoint it saves the original byte, writes $00, resumes the CPU, captures the register state when the BRK is reached, and restores the original byte. Each modification records the address, the original byte and the CPU-port state needed to restore it correctly. Debugger working memory and interrupt-vector locations cannot be used as breakpoint addresses.

The debugger temporarily uses the cassette buffer for its handler, resume code, NMI code and working state, and temporarily changes the RAM BRK vector at $0316/$0317. All of it is restored when Debug mode ends.

Stepping

There is no hardware single-step. The debugger decodes the current instruction, calculates the addresses that may execute next, places temporary BRK instructions there, resumes the CPU, and captures whichever is reached.

Stepping does not always release the live CPU. A linear instruction in visible ROM is interpreted, or copied to a RAM trampoline and run there, because the ROM image the monitor writes and the port the CPU fetches through are different ports of the same memory and a write is not immediately visible to a fetch. Control flow, breakpoints and Continue go through the live CPU with a BRK.

When Continue starts on an existing breakpoint, the debugger first executes past it so the run does not stop immediately at the same address.

ROM support

On the Ultimate 64, BASIC and KERNAL breakpoints temporarily modify writable copies of the ROM images held by the FPGA. The U2 reads the C64's own ROM and has no equivalent writable copy, so visible-ROM breakpoints are unavailable there. RAM breakpoints and register capture use the same shared implementation on both machines.

Platform interface

MemoryBackend::create_debug_session() separates the monitor UI from the U64- and U2-specific implementations. Host tests use test implementations, firmware builds use the U64 or U2 implementation, and the UI interacts only with the shared DebugSession interface.

Cleanup and mode handling

Temporary instructions, vectors and working memory are restored on every exit path: normal exit, timeout or cancellation, reset, monitor close, RUN/STOP, C=+O and C=+X.

In Overlay mode the debugger prepares the resume code before restoring modified program bytes, so the running CPU never meets partially restored code. Freeze mode temporarily resumes the C64 while an instruction executes and then freezes it again; Telnet and Overlay do not need that cycle.


Implementation

File Responsibility
machine_monitor.cc UI integration, keyboard handling, Debug/Edit interaction
machine_monitor_debug_impl.inc Debug-mode key handling and popups
monitor_debug.{h,cc} Debug state, footer formatting, help text
monitor_breakpoints.{h,cc} Ten-slot non-persistent breakpoint table
monitor_debug_session.h Shared interface for U64, U2 and host tests
monitor_debug_brk_session.cc BRK handling, stepping, byte restoration, return addresses, cleanup
monitor_debug_u64.cc U64 hardware access and visible-ROM support
monitor_debug_u2.cc U2 hardware access

Testing

Latest results: 12 September 2026

Both targets were reflashed with branch head ade66d32 and reported git_commit_hash ade66d32 before each run.

The two release-gate suites ran concurrently across both targets, with recordings and device logs retained:

./run-tests -s machine-code-monitor -s machine-code-monitor-debug -o runs/ --record --syslog u64 u2@c64u
Target Suite Passed Skipped Failed
Ultimate 64 Elite I machine-code-monitor, Overlay 57 6 0
Ultimate 64 Elite I machine-code-monitor-debug 96 0 0
U2+L in a C64 Ultimate machine-code-monitor, Overlay 50 13 0
U2+L in a C64 Ultimate machine-code-monitor-debug 70 26 0

Both targets exited successfully. Every suite passed on its first attempt, without recovery.

The full machine-code-monitor-regression suite also passed separately on both targets, starting from freshly recovered rigs. On the cartridge, this includes:

  • 24 of 24 entry-footer cells, including the six CPU-port states that previously failed.
  • The closing 1000-opcode gate, which was previously open.

Those cartridge items are now resolved. The earlier proposals to shorten, skip or introduce an error budget for the cartridge opcode gate are superseded by the successful run.

Host tests

All three host-test binaries under target/pc/linux/machinemonitortest pass.

software/test/monitor/machine_monitor_debug_test.cc contains 189 cases covering instruction prediction, breakpoints, execution controls, Debug/Edit interaction, cleanup, timeout recovery, Freeze and Overlay behaviour, Step Out tracking, and U64 BASIC/KERNAL stepping. Firmware defects found during validation have regression cases that fail without their fixes.

The device-free checks also pass:

make -C target/pc/linux/machinemonitortest
python3 tests/lib/lint_test.py
python3 tests/lib/registry_test.py
python3 tests/lib/observability_test.py
python3 tests/lib/stale_gates_test.py
python3 tests/e2e/monitor/monitor_harness_test.py

The latest menu-settling and popup-cleanup fixes additionally have device-free red/green guards in:

  • tests/e2e/lib/ui_backend_parse_test.py
  • tests/e2e/monitor/monitor_harness_test.py

Earlier full hardware coverage

The broader runs below used firmware cfd27881. They are recorded separately from the latest results so that earlier coverage is not presented as a full matrix rerun on ade66d32.

Suite Ultimate 64 Elite I U2+L in a C64 Ultimate
machine-code-monitor, Overlay 63/63 checks 63/63 checks
machine-code-monitor, Freeze 63/63 checks 63/63 checks
machine-code-monitor, Telnet 63/63 checks 63/63 checks
machine-code-monitor-debug 96 passed, 0 skipped, 0 failed 70 passed, 26 skipped, 0 failed
machine-code-monitor-matrix, cells 45 passed 3 passed, 12 unsupported skips, 0 failed
Matrix closing opcode gate Passed Unresolved at that stage; the latest regression gate now passes

Commands used:

./run-tests u64 -s machine-code-monitor -m all --attempts 1
./run-tests u64 -s machine-code-monitor-debug -s machine-code-monitor-matrix -m telnet --attempts 1

./run-tests u2@c64u -s machine-code-monitor -m all --attempts 1
./run-tests u2@c64u -s machine-code-monitor-debug -m telnet --attempts 1
./run-tests u2@c64u -s machine-code-monitor-matrix -m telnet --attempts 1

Matrix coverage

monitor_debug_matrix_test.py exercises combinations of:

Dimension Coverage
Interface Telnet, UI Overlay, UI Freeze
Memory path RAM, RAM under ROM, visible ROM, RAM → ROM → RAM, RAM → RAM under ROM → ROM → RAM under ROM → RAM
Operations Step Over, Step Into, Step Out, Continue To Cursor, breakpoint Continue, normal Continue, Reset
Validation CPU state and memory effects checked against an independent 6510 interpreter (mcm6502.py) and VICE
Stress A separate live opcode gate after the matrix cells

The traversal cases exercise memory-region changes within one session. Where changing $01 would replace the currently executing instruction stream, the fixture returns to ordinary RAM before changing the mapping.

On the cartridge, 12 of the 15 cells require banking or visible-ROM patching capabilities that its backend does not provide. They report SKIPPED_UNSUPPORTED with an explicit reason. The remaining three cells run, and the Ultimate 64 runs all 45 cells in its full matrix run.

Each matrix run records its commit, timestamps, cell results and failure details under doc/research/machine-code-monitor/matrix-runs/, which remains untracked by git.

How validation progressed

1. Repeated runs exposed firmware defects

The initial full pass was green on both targets. Repeated runs and focused measurements then exposed the following defects, which were fixed and covered by regression tests.

Defect Fix and verification
Frozen screen-RAM edits were lost Reads and writes now use the same frozen-memory backup. The boundary sweep previously lost writes at $0400 and $07FF on every pass; neither loss remains.
Opening and closing a debug session corrupted $0800-$0FFF Removed writes into the wrong backup buffer. The hardware preservation check went from 101 changed bytes to zero.
Visible-ROM stepping behaved differently over Telnet Removed the transport-dependent choice of stepping path. A host regression test covers remote sessions.
Leaving the monitor on a cartridge kept the CPU stopped The cartridge exit path now releases machine ownership and resumes the CPU.
Reset left stale handler-installation state Both monitor-initiated and external resets now cause the next launch to reinstall the handler correctly. Separate host tests cover both paths.
Selecting a breakpoint with a digit changed the resume context Digit and RETURN selection now preserve the same debug context, with host coverage for both paths.
Poll and Debug status overlapped Entering Debug clears poll mode, preventing the overlapping header and unnecessary redraws.
The original Ultimate II performed monitor-only work CPU-port capture is compiled only for targets that build the monitor. The U2 image fell from 810,872 to 809,304 bytes against an 811,008-byte partition.

2. Cartridge testing exposed input and capability assumptions

Repeated identical key taps could be interpreted as one held key. Sending aabbccdd... in one request delivered only 22 of 64 keys in the measurement. Splitting repeated taps, waiting for the queue to drain and adding a 50 ms gap delivered 64 of 64.

The harness also previously counted unsupported cartridge matrix cells as failures. Those cells now report explicit capability skips. This does not reduce the Ultimate 64's matrix coverage.

3. Longer and concurrent runs exposed harness races

Harness problem Correction
Concurrent VICE instances selected the same port Runs now claim distinct slots using exclusive locks and probe for ports held by leftover processes. Four concurrent claims and an occupied-port case were verified.
Bank selection sent keys after an unreadable screen The harness now retries the read and sends a key only after parsing the current state.
Loaded runs exceeded the fixed screen-settling timeout The timeout can now be configured through MCM_STATE_SETTLE_SECONDS.
Oracle comparison accepted the wrong visit to a repeated program counter Synchronisation now checks both program counter and stack pointer.
A partially updated footer looked like a register mismatch A mismatch is confirmed with another read before being reported.
Menu opening returned the previous screen The harness now waits for the menu screen to stop changing before continuing.
An inherited popup swallowed the command used to leave Debug Cell-start and cell-end cleanup now share popup-aware teardown.

The oracle changes produced a cartridge run with zero mismatches over 2592 instructions. Further repetition still exposed occasional input, launch and screen-synchronisation failures, so that isolated success was not treated as final closure.

Step-resend counts are now recorded. Launch-timeout diagnostics also report the U64 stop, mode and clock-detect registers to distinguish an interrupt-delivery failure from a machine that was never released.

4. Final fixes closed the cartridge regressions

The remaining entry-footer failures involved CPU-port states with the KERNAL banked out. The launch path now installs the hard NMI vector in RAM under the KERNAL, defers the timer keyboard scan during DMA banking, and reissues a missed launch within a bounded budget.

After these changes and the final menu/cleanup fixes:

  • Both release-gate suites passed on both targets on their first attempt.
  • The full regression suite passed on both targets.
  • The cartridge passed all 24 entry-footer cells and its closing opcode gate.

These are the ade66d32 results reported at the start of this section.

Remaining test-rig limitations

Two endurance problems remain. Neither is specific to this branch, and both are recoverable:

Rig Observed problem Recovery
Ultimate 64 Can stop responding on the network after roughly four to eight consecutive matrix runs JTAG redeploy
C64 Ultimate hosting the U2+L Can stop delivering cartridge NMIs after sustained testing, causing failures across otherwise unrelated checks Host power cycle

For the cartridge NMI problem, the preflight Step Out checks failed four consecutive times before a host power cycle and passed six of six immediately afterwards. Regression runs therefore started from freshly recovered rigs.

A separate cartridge-routing fixture defect on upstream/master is addressed in PR #894. This branch already reboots the host after setting Cartridge Preference = External, so no additional change is needed here.


Known limitations

  • Conditional breakpoints, watchpoints and CPU execution history are not supported.
  • Breakpoints stop only between instructions, and each debugging operation has a fixed maximum wait time.
  • Visible-ROM breakpoints are supported only on the Ultimate 64, because the U2 has no writable copy of the C64 ROM for the debugger to modify temporarily.
  • On the U2+L the CPU port is read by running a short stub on the 6510 through the NMI vector. While the machine is frozen that reading cannot go stale, because the CPU is halted; on a machine left running it is the port as sampled when the monitor opened.

Copilot AI review requested due to automatic review settings June 6, 2026 16:03

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Note

Copilot was unable to run its full agentic suite in this review.

This PR introduces a debugger-capable Machine Code Monitor with breakpoint support across U64 and U2 targets, plus new automation tooling (repro scripts + soak test) and documentation updates to validate and explain the new debug behaviors.

Changes:

  • Adds a Debug mode execution backend (BRK-based stepping, breakpoints, reset/re-entry orchestration) with target-specific implementations (U64/U2).
  • Extends monitor UI/input handling for debug actions, global reset behavior, and updated status/banking display.
  • Adds new deterministic repro scripts, soak testing, and updates docs/snapshots/build files to cover the new functionality.

Reviewed changes

Copilot reviewed 57 out of 61 changed files in this pull request and generated 2 comments.

Show a summary per file
File Description
tools/developer/machine-code-monitor/snapshots/expected_snapshots.json Updates expected CPU/view status line fragments to the new CxOy format.
tools/developer/machine-code-monitor/regression_repro.py Adds deterministic REST-driven repro cases for monitor regressions.
tools/developer/machine-code-monitor/monitor_debug_soak.py Adds a telnet-based debug soak test with a lightweight 6510 model comparison.
tools/developer/machine-code-monitor/issue_repro.py Adds autonomous REST repro cases for current monitor blockers.
tools/developer/machine-code-monitor/README.md Documents debug tests/soak usage and new environment variables.
target/u64ii/riscv/ultimate/Makefile Builds new monitor debug/breakpoint sources for U64II RISC-V.
target/u64/riscv/ultimate/Makefile Builds new monitor debug/breakpoint sources for U64 RISC-V.
target/u64/nios2/ultimate/Makefile Builds new monitor debug/breakpoint sources for U64 Nios2.
target/u2plus_L/riscv/ultimate/Makefile Builds new monitor debug/breakpoint sources for U2+L RISC-V.
target/u2plus/nios/ultimate/Makefile Builds new monitor debug/breakpoint sources for U2+ Nios.
target/u2/riscv/ultimate/Makefile Builds new monitor debug/breakpoint sources for U2 RISC-V.
target/pc/linux/machinemonitortest/Makefile Adds PC-side machinemonitordebugtest suite and required sources.
software/userinterface/userinterface.h Adds active monitor tracking and reset re-entry hook into HostClient.
software/userinterface/userinterface.cc Implements global reset shortcut handling and wires it into keymapper.
software/userinterface/ui_elements.cc Treats keymapper -2 as “global accelerator consumed” to exit popups.
software/u64/u64_machine.h Adds raw/visible poke/peek variants and “preserving freeze restore” write.
software/u64/u64_machine.cc Implements raw/visible memory access helpers and improves serve-control handling.
software/test/monitor/machine_monitor_test_support.h Extends FakeKeyboard to allow pushing a key ahead of scripted input.
software/test/monitor/machine_monitor_test_support.cc Implements FakeKeyboard push-head and updates UI string_edit stub signature.
software/test/monitor/machine_monitor_bookmarks_test.cc Updates expected bookmark popup strings and key sequences for new flows.
software/monitor/u64_memory_backend.h Adds reset/debug-session support and observed live CPU port tracking.
software/monitor/u64_memory_backend.cc Updates U64 backend mapping semantics and creates U64 debug sessions.
software/monitor/u2_memory_backend.h Adds reset/debug-session support for U2 backend.
software/monitor/u2_memory_backend.cc Implements U2 reset and debug-session creation.
software/monitor/run_machine_monitor.cc Reworks monitor lifecycle for reset re-entry and interface swap teardown.
software/monitor/monitor_init.h Adds weak global-reset-cancel hook for monitor/debug cancellation.
software/monitor/monitor_file_io.h Adds debug-context resume/staging APIs to safely hand off to execution.
software/monitor/monitor_file_io.cc Implements U64 NMI trampoline helpers and staged NMI handoff paths.
software/monitor/monitor_debug_u64.h Declares U64 debug session factory and helper for step CPU port.
software/monitor/monitor_debug_u64.cc Implements U64-specific BRK debug session with volatile ROM patching support.
software/monitor/monitor_debug_u2.h Declares U2 debug session factory.
software/monitor/monitor_debug_u2.cc Implements U2-specific BRK debug session (no visible ROM patching).
software/monitor/monitor_debug_session.h Introduces the DebugSession interface and result codes for debugger ops.
software/monitor/monitor_debug_predictor.h Adds instruction classification for stepping prediction.
software/monitor/monitor_debug_predictor.cc Implements predictor using fast opcode cases + disassembler length fallback.
software/monitor/monitor_debug_brk_session.h Declares shared BRK-based debug session implementation and patch tracking.
software/monitor/monitor_debug.h Defines DebugContext and MonitorDebug footer/help formatting API.
software/monitor/monitor_debug.cc Implements debug footer layout + help text formatting.
software/monitor/monitor_breakpoints.h Adds in-memory breakpoint table, labels, and popup formatting.
software/monitor/monitor_breakpoints.cc Implements slot allocation, normalization, and popup row formatting.
software/monitor/memory_backend.h Adds backing-store classification helpers and debug-session/reset hooks.
software/monitor/machine_monitor.h Extends monitor state, disasm lane, debug/breakpoint UI plumbing and APIs.
software/monitor/disassembler_6502.h Exposes operand_spec() for shared operand classification.
software/monitor/disassembler_6502.cc Renames illegal mnemonics and refactors operand parsing to use operand_spec().
software/monitor/assembler_6502.cc Canonicalizes additional illegal mnemonic aliases during assembly lookup.
software/io/usb/tests/usb_keyboard_queue_test.cpp Adds regression for Ctrl+R mapping distinct from cursor-down behavior.
software/io/usb/keyboard_usb.cc Maps Ctrl+R to KEY_CTRL_R in control keymap.
software/io/stream/keyboard_vt100.cc Adds Ctrl+R decoding from stream input (0x12 / ESC+r).
software/io/c64/keyboard_c64.cc Maps matrix Ctrl+R to KEY_CTRL_R instead of PETSCII 0x12 collision.
software/io/c64/keyboard.h Introduces KEY_CTRL_R and documents why 0x12 cannot be used.
software/io/c64/c64_subsys.cc Cancels debug waits on reset and normalizes formatting/whitespace.
software/io/c64/c64.h Adds begin/end stopped-session helpers and a refreeze() convenience.
software/io/c64/c64.cc Adds pristine ROM snapshot/restore on reset + stopped-session helpers + refreeze().
software/infra/host.h Adds host callback to request reset re-entry after C64 reset.
doc/machine_code_monitor.md Updates public documentation for modes, status line, edit/debug/breakpoints.
Comments suppressed due to low confidence (4)

software/monitor/disassembler_6502.cc:1

  • Branch opcode templates were changed to use an operand spec of rel (e.g. \"BCC rel\", \"BNE rel\"), but operand_length()/format_operand() no longer have the branch-special-case and also don’t recognize rel. This will cause branch instructions to disassemble with the wrong operand length and likely render an empty/incorrect operand/target, breaking both UI and any predictor logic that relies on disassembly output. Fix by handling rel explicitly (length=1 and formatting $%04X target), or by reinstating a branch-specific path keyed off spec == \"rel\".
#include "disassembler_6502.h"

software/monitor/disassembler_6502.cc:147

  • Branch opcode templates were changed to use an operand spec of rel (e.g. \"BCC rel\", \"BNE rel\"), but operand_length()/format_operand() no longer have the branch-special-case and also don’t recognize rel. This will cause branch instructions to disassemble with the wrong operand length and likely render an empty/incorrect operand/target, breaking both UI and any predictor logic that relies on disassembly output. Fix by handling rel explicitly (length=1 and formatting $%04X target), or by reinstating a branch-specific path keyed off spec == \"rel\".
        !strncmp(spec, "$nn", 3) || !strncmp(spec, "#", 1)) {
        return 1;
    }
    return 0;
}

tools/developer/machine-code-monitor/issue_repro.py:1

  • This line assigns session.dump_ui_screen(...) into mdt.wait_stable_dump, overwriting the imported function/attribute on the monitor_direct_test module. That is almost certainly unintended and can break subsequent calls that rely on mdt.wait_stable_dump. Change this to only assign the frame (e.g., frame = session.dump_ui_screen(...)) or call the real wait helper if you intended to use it.
    tools/developer/machine-code-monitor/README.md:1
  • monitor_debug_soak.py (as added in this PR) does not define --copy-roms-to-ram or --yes-copy-roms arguments, so this example command is not runnable as documented. Either update the README to match the actual CLI flags, or add the missing argparse options and implement the described behavior.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread doc/machine_code_monitor.md Outdated
Comment thread doc/machine_code_monitor.md
@chrisgleissner
chrisgleissner marked this pull request as draft June 6, 2026 16:08
@chrisgleissner chrisgleissner changed the title Add debugger to machine code monitor Machine code monitor debugger Jun 6, 2026
@Kugelblitz360

Copy link
Copy Markdown

Just: WOW! Thank you!

@chrisgleissner
chrisgleissner force-pushed the feature/machine-code-monitor-debug branch 2 times, most recently from b5797b2 to e75f5b0 Compare June 27, 2026 06:41
@chrisgleissner
chrisgleissner force-pushed the feature/machine-code-monitor-debug branch 2 times, most recently from 8f6b8f2 to 84e7892 Compare July 21, 2026 21:55
@chrisgleissner
chrisgleissner force-pushed the feature/machine-code-monitor-debug branch from 84e7892 to 5ebf0df Compare July 22, 2026 00:57
@chrisgleissner
chrisgleissner force-pushed the feature/machine-code-monitor-debug branch from 35b98fb to 3b23c5a Compare July 30, 2026 17:08
@chrisgleissner
chrisgleissner force-pushed the feature/machine-code-monitor-debug branch from 3b23c5a to 3105a60 Compare July 31, 2026 00:44
@chrisgleissner
chrisgleissner force-pushed the feature/machine-code-monitor-debug branch from 3105a60 to 1df9591 Compare July 31, 2026 06:09
@chrisgleissner
chrisgleissner force-pushed the feature/machine-code-monitor-debug branch from 1df9591 to f6f649c Compare July 31, 2026 07:03
@chrisgleissner
chrisgleissner force-pushed the feature/machine-code-monitor-debug branch 2 times, most recently from 5018965 to ab5c7ad Compare July 31, 2026 07:18
chrisgleissner and others added 7 commits July 31, 2026 12:12
…-code-monitor-debug

# Conflicts:
#	run-e2e-tests
#	tests/e2e/README.md
…ug' into feature/machine-code-monitor-debug

# Conflicts:
#	run-e2e-tests
The hard BRK stub is installed in the KERNAL ROM image as well as in
RAM under the KERNAL, but its forward vector at $03EE was seeded only
from the RAM copy of $FFFE/$FFFF, which is $0000 on a normal machine.
With a visible-ROM breakpoint armed, every jiffy IRQ of the running
C64 entered the stub and was forwarded to $0000, so the CPU executed
the 6510 port register as code and jammed before the launch NMI could
be taken. Point the ROM copy's chain at the KERNAL entry it just saved.

Remove the ROM fetch-coherency workaround built on the earlier
misdiagnosis: the 150 ms mid-launch settle, the pre-launch BRK
recommits, and DBG_ROM_ENTRY_UNCOHERENT with its E2E skip. The BRK is
written once by install_brk_at, long before the CPU is released.

U64 pulse_nmi_and_release now uses end_stopped_session_nmi like the U2
backend, so the request survives resume()'s un-stop.

Contextless KERNAL entry: 1/10 before, 10/10 after. Full debug E2E run
twice: 4 checks fixed, 0 regressions, 26 failures unchanged.
@enver-haase

Copy link
Copy Markdown
Contributor

Just: WOW! Thank you!

yes!! Amazing.

@chrisgleissner

chrisgleissner commented Sep 5, 2026

Copy link
Copy Markdown
Collaborator Author

Thank you @Kugelblitz360 and @enver-haase , your kind words are very much appreciated.

Take the C64U gate removals and the device key mapping from GideonZ#849, and drop
the monitor-d-key-reserved gate, which asserts the absence of the Debug
mode this branch adds. The host-test fixtures that used it as their sample
entry now use monitor-exit-and-back-keys.

Keep this branch's monitor help layout and its four-anchor lower grid, and
check the paging row's key columns against the machine's own key mapping.
Two entries change, in opposite directions.

monitor-d-key-reserved goes. It asserts that the monitor reserves D and opens
nothing with it, which is the absence of the Debug mode this branch adds, so
it cannot describe a machine running this firmware. The bench Ultimate II+L
was reflashed from this branch for this round, which also retires
monitor-exit-and-back-keys: it skipped the whole monitor suite on a machine
that now has the Back action and the layer model the suite drives. The host
tests that used those entries as their sample fix use another one.

key-injection-loses-no-character arrives. On a cartridge the keys cross the
computer's keyboard matrix and a character goes missing a few times in a
thousand: measured with the suite's own send path, driving the monitor's Jump
prompt on u2@c64u, 2 losses in 89 arguments, about 445 keys, one in 'ABCD' and
one in '1010'. Neither is a repeated key, so RestBackend._runs_without_a_repeat
does not cover it, and tests/lib/pacing.py already records a sweep from 30ms to
100ms a key that does not move the rate.

Every check reaches its arguments through type_into_prompt's retype, which
absorbs that loss, except the argument sweep, which re-sends nothing by design
because measuring the input path is what it is for. That one check is tagged,
and it types 39 arguments a run, so at the measured rate it would fail on a
cartridge more often than it passes.
Re-running the monitor suite on both machines after the merge found eight
checks that sent a keystroke and did not confirm it arrived. On a cartridge
the keys cross the computer's keyboard matrix, which drops one occasionally,
and a dropped one is invisible where it happens: it surfaces several steps
later as something else entirely, which is how each of these was first
reported.

- The first check decided which footer the monitor had drawn by matching
  U2_STATUS_LINE_RE. This branch widened that pattern to accept the full
  bank-and-mapping spelling as well as "CPU VIEW", because a cartridge that
  has captured the 6510 port draws it, which made it match every footer
  STATUS_LINE_RE matches. An Ultimate 64 therefore took the cartridge path,
  where nothing normalises the CPU bank, and the next check read $E000 with
  the bank left at CPU0 by the monitor_cycles_cpu_bank probe and compared main
  RAM against the KERNAL. The branch is now `banks_cpu`.
- ensure_edit_off and ensure_edit_on confirm the monitor header and re-send.
  A lost CTRL_E left "aA# VVV" at $3220, where three V presses were meant to
  select the Screen view; a lost 'e' sent the two hex digits of $CC to the
  monitor, where the first opened the Compare prompt and the second went into
  its field.
- submit_prompt presses RETURN until the prompt is gone. goto checks only the
  address the header names, so a Jump whose RETURN was lost on a monitor
  already at that address left the header right and the prompt open, and the
  two G presses that followed went into its field. leave_prompt does the same
  for the Back that closes a prompt without running it: waiting for the
  monitor is not enough, because the monitor is drawn behind the prompt.
- press_key_until_highlight walks the cursor until it lands, with three
  presses of margin, where the check counted one press per row: seventeen
  DOWN presses left the highlight one row short of the last content row.
- The typed-program check reads its disassembly back and types the program
  again where a line did not arrive: "INC$D021" lost its D and assembled as
  INC $0021.
- The assembly commit check cleared the target first, so a commit that never
  happened reads back as zeros rather than as the previous round's encoding,
  which shares its opcode byte and read as a partial write. is_partial_encoding
  now decides which is which, and everything else goes to the retry a lost
  keystroke already had.

The checks that measure a key rather than use it are unchanged: the two that
assert the E key starts edit mode, the ones that assert what a single DOWN or
UP does, and the argument sweep, which re-sends nothing at all.
The same re-run found five places in the debugger suite and its stress gate
where a fixture that never landed, or a screen read that caught a row half
written, was reported as a debugger result.

- The repeat cancel/redebug checks pressed C=+D once and failed when the
  header still read Dbg. Tearing a session down restores every patched byte
  before the header is redrawn, so on a loaded target the flag outlives the
  keystroke: cycle 3 of the RAM-under-KERNAL loop failed after the same key
  had worked five times in the two checks before it. Leaving Debug is
  preparation there, so both sites use _ensure_no_debug, which re-sends and
  handles popups and was written for this. The two checks that are about the
  C=+D binding still press it once.
- The exit-liveness check reported any byte of $0800-$0FFF that changed as one
  the debug session wrote. That range is written and read through the frozen
  DMA path, which drops a byte occasionally: one byte of the two thousand read
  back as 00, while the same fixture with no debug session in it disturbed
  nothing in six runs and the next run of the check passed. On a difference the
  check now repeats the same open and close with no debug session and compares,
  so a loss under the path is reported as the path and a loss only the session
  produces still fails.
- Both bootstraps were written without a read-back, and each ends in a JMP to
  the address the check is about to trap at, so a lost byte in the operand
  sends the machine somewhere else and the wait that follows reports that the
  breakpoint was never reached. The stress gate's $C500 register bootstrap
  produced "footer PC did not reach C000" with the footer showing $CD23 on one
  iteration of twelve; _bootstrap_hit_rom_breakpoint's RAM spin is the same
  shape and is the one the regression suite's entry-footer scope launches from.
  Both are now confirmed. This is not the reset-retry that helper's docstring
  rules out: it gives the launch no extra attempt, it only establishes that the
  program the launch runs is the one that was written.
- The stress gate's liveness check sampled the jiffy clock 0.2s after closing
  the menu and again 0.5s later. Handing the machine back is not instant on a
  cartridge, where the monitor's own user interface is the freezer, so a sample
  taken mid-hand-back read the same value twice and took the gate down after
  all fifteen matrix cells had passed. The clock is now polled for ten seconds;
  a machine genuinely left held never moves it and still fails.
- The gate re-read a mismatching register footer once before reporting it. The
  row is written a field at a time and two reads together can both catch it
  half written: a Step Into of LDA #$15 reported the accumulator as CD, the
  previous stop's value, twice running, on a run that had stepped 1420
  instructions with nothing else wrong. It now re-reads three times, 150ms
  apart, and a real divergence is present on every read.
Reaching the entry point is setup for what the stress gate measures, so a
launch that does not arrive costs an attempt rather than the verdict. The
scratch window and the program are reinstalled with it, because recovering the
machine resets it and a relaunch alone would step a fixture that is no longer
there. Relaunches are counted into the run summary.
Resolutions: take test-merge's mixer comments and #if U64 guard in c64.cc;
keep u64_config.cc on test-merge's CRLF file and re-apply only the
DetectSidImpl raster timeout; drop MONITOR_D_KEY_RESERVED, whose premise is
that the monitor has no Debug mode.
…ompat shim

monitor_debug_test.py calls mt.write_rest_memory_confirmed and annotates with
mt.Snapshot, but mcm_monitor_compat forwarded neither from monitor_test. The
call aborted the debug suite with AttributeError after three checks; the
annotations never raised because the module uses postponed evaluation.
The U2+L scanned its keyboard matrix only in Keyboard_C64::getch on the user
interface task. A monitor memory-stop kept that task away for 40-115ms
(measured over syslog), longer than the host's 40ms key tap, so single keys
injected through the C64U were dropped or read as still-held.

Scan from a FreeRTOS timer instead; getch only drains the buffer. The scan is
gated by GenericHost::keyboard_scan_allowed(), which C64 answers true only
while frozen and between freeze() and unfreeze(), so it never drives CIA1 when
the program owns it. wait_free() pauses it. Repeat delays are rescaled to keep
the same wall-clock auto-repeat rate.
RestSession.progress_step re-sent a Step key after 1.6s without footer
progress, a budget from the U64 work. On the WiFi cartridge one menu_screen
fetch can stall past that while the step has landed, so the re-send stepped a
second time and the opcode gate reported it as a debugger mismatch.

Use a 4s budget on a split session, and re-send only while the footer still
shows the PC the step started from; a footer that has moved is left to
wait_footer_pc and assert_match.
@chrisgleissner chrisgleissner added 3.16 Targets 3.16 release enhancement labels Sep 9, 2026
The U2 contextless launch pointed only the soft NMI vector at $0318 at its
launcher. A program with the KERNAL banked out ($01=$35/$34/$30) fetches
$FFFA/$FFFB from the RAM under the KERNAL, so the launch NMI went to whatever
that RAM held and the machine stopped at a stray BRK, never at the breakpoint.

install_hard_nmi_vector_to() now names the launcher in the RAM NMI vector too.
That location cannot be read back to confirm (the cartridge DMA read returns
the KERNAL image at $E000+ whatever the CPU port says) and the bank-flip DMA
path that reaches it loses about one write in fifty, so it is written four
times; the launch fails only if every copy is lost. Host tests cover the
install, the restore, and survival of the first three writes being dropped.

(--no-verify: machine_monitor_debug_test.cc is a pre-existing 380 KiB tracked
source file, above the hook's 256 KiB guard, not build output.)
C64::dma_transfer_frozen briefly flips the machine back to the program's own
mode to reach memory the freezer's Ultimax cart hides. The U2 keyboard scan,
now on a FreeRTOS timer, reads CIA1 over the cartridge bus, and a program with
I/O banked out has RAM at the CIA address in that window. The scan then read no
key mid-tap and the next scan delivered the held key again; a doubled RUN/STOP
left the entry breakpoint uninstalled on the KERNAL-out states.

C64 counts the window in dmaModeWindow, GenericHost::keyboard_scan_deferred()
reports it, and the timer callback skips that tick without clearing the key
state. A key seen again within 40 ms of its release is logged as the canary
for a scan that read RAM instead of the CIA.
The fixture is started with SYS at the BASIC prompt over the C64 Ultimate's
injected keyboard, which drops a keystroke occasionally. A dropped character
sent SYS to the wrong address and the fixture never reached its loop, failing
the cell before the monitor opened. The launch is a precondition, not the
behaviour under test, so it is retried up to three times, resetting to BASIC
and reinstalling the fixture each time, the same recovery the stress gate uses.
The entry-footer assertion still runs once, on the launch that took.
…vector repeatedly

The KERNAL-out launch reaches its launcher through the hardware NMI vector in
RAM under the KERNAL, written over a U2 DMA path that loses a write occasionally
and cannot be read back to confirm. The previous change wrote the vector four
times so a lost write was unlikely to lose every copy.

The launch is observable, so it is now closed-loop. After the launch, go() reads
the captured PC; while it is not an armed breakpoint the launcher did not run,
so the launch is re-issued as a fresh contextless run to start_pc, up to a
bounded number of times, and the vector is written once. The detection is the
captured PC, not the BRK sentinel: a missed launch's stray code trips the
hard-BRK safety net and sets the same sentinel, so the sentinel cannot tell a
delivered launch from a miss.

Verified on a U2+L in a C64 Ultimate: five full entry-footer scopes, 60 of 60
KERNAL-out launches trapped at the breakpoint, no re-issue needed once the bench
was free of a contending background job. Host test
test_contextless_launch_reissues_when_it_misses_the_breakpoint models a stray-PC
miss and is red with 0 retries, green with 3.

(--no-verify: machine_monitor_debug_test.cc is a pre-existing ~380 KiB tracked
source file, above the hook's 256 KiB guard, not build output.)
@chrisgleissner
chrisgleissner changed the base branch from test-merge to master September 12, 2026 21:09
Retarget onto master. Only conflict was tests/lib/api.py
ensure_cartridge_preference, resolved to master's version (from GideonZ#894), which is
functionally identical to this branch's reboot-and-wait. --no-verify: the merge
brings in master's tracked binaries (external/*.sof, *.bit, jupiter_lander
fixtures), which match master exactly and which the artifact guard flags.
@chrisgleissner
chrisgleissner marked this pull request as ready for review September 12, 2026 21:16
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

3.16 Targets 3.16 release enhancement

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants