Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
57 changes: 27 additions & 30 deletions issues/a-120000-ms-boot-deadline-fired-132859-ms-late-on-the-t14.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,36 +6,33 @@ opened: 2026-09-16

# A 120000 ms boot deadline fired 132859 ms late on the T14, and the hard-lockup bound armed beside it was silent too

`kernel/src/deadline.rs` reads the bound off the parameter line (`claim`,
`:150-162`), turns it into a TSC deadline once there is a clock (`start`,
`:168-183`), and `poll` (`:194-200`) compares one relaxed load of `AT_TSC`
against `rdtsc` from the timer interrupt entry — `kernel/src/arch/x86_64/idt/timer.rs:71`
(`:79`) in Ring 0 and `:96` in Ring 3 — and calls `expire` (`:208-225`), which
seals the record and writes the reset register through `acpi::reset_now`.
`start` arms the other half of the same parameter at `:182`,
`kernel/src/deadline.rs` reads the bound off the parameter line (`claim`),
turns it into a counter deadline once there is a clock (`start`), and `poll`
compares one relaxed load of `AT_COUNTER` against `arch::cpu::counter` from the
timer interrupt entry — `kernel/src/arch/x86_64/idt/timer.rs`, through
`poll_in_kernel` in Ring 0 and `poll` in Ring 3 — and calls `expire`, which
seals the record and writes the reset register through `power::reset_now`.
`start` arms the other half of the same parameter,
`crate::hardlockup::start(ms)`: a bound of `toyos_tco::hard_lockup_bound_ms`,
`deadline_ms / 2` (`lib/tco/src/lib.rs:154-156`), sampled by an NMI every
second on every CPU (`kernel/src/hardlockup/mod.rs:1-31`). The two share one
seal (`claim_the_reset`, `deadline.rs:69-71`).
`deadline_ms / 2` (`lib/tco/src/lib.rs`), sampled by an NMI every
second on every CPU (`kernel/src/hardlockup/mod.rs`'s header). The two share
one seal (`deadline::claim_the_reset`).

A boot armed with `boot-deadline=120000` prints both arms at 0.060 s — quoted
from bench run 42's kernel log, the text `deadline.rs:174-177` and
`hardlockup/mod.rs:193-198` write:
from bench run 42's kernel log, the text `deadline::start` and
`hardlockup`'s `Armed` write:

boot deadline: 120000 ms, after which this kernel seals a WEDGED record and writes the reset register itself
hard lockup: 60000 ms, sampled every 1000 ms by each cpu's own performance counter, after which a cpu that has taken no interrupt seals a WEDGED record and resets the machine

Run 55's boot — armed at 2026-09-14 20:54:57Z, the `lancase` image of the
branch `i219-delivery` (#461) at `4d604c86`, whose `kernel/src/deadline.rs`,
`kernel/src/hardlockup/mod.rs` and `kernel/src/drivers/acpi.rs` are
byte-identical to this tree's (`git diff --stat 4d604c86 HEAD -- <those>` is
empty) — did not come back inside the loop's 420 s, and its own kernel log
never reached the stick, so its print of the two lines is inferred from
`start` and not read. Its `WEDGED` record was read by run 56's loader pass:
a pass that printed the record and ended the chain without booting a kernel
(`loader/src/blackbox.rs:149-154`, `loader/src/main.rs:857-860`), so
that the machine came back to Ubuntu after 100 s and the loop read the
stick's `loader.log`, which reads, whole above the ring tail:
Run 55's boot — armed at 2026-09-14 20:54:57Z, the `lancase` image of the branch
`i219-delivery` (#461) at `4d604c86` — did not come back inside the loop's 420
s, and its own kernel log never reached the stick, so its print of the two lines
is inferred from `start` and not read. Its `WEDGED` record was read by run 56's
loader pass: a pass that printed the record and ended the chain without booting
a kernel (`loader/src/blackbox.rs`'s `harvest`), so that the machine came back
to Ubuntu after 100 s and the loop read the stick's `loader.log`, which reads,
whole above the ring tail:

--- the pass after the reset, reading what the boot above left
ToyOS Bootloader 1.0
Expand All @@ -49,7 +46,7 @@ stick's `loader.log`, which reads, whole above the ring tail:
it. In the same boot the 60000 ms hard-lockup bound sealed nothing — the seal
is the deadline's — and that detector seals only for a CPU whose interrupt
count is stale for its bound *and* whose sampled frame has `IF` clear
(`hardlockup/mod.rs:21-31`). So for 133 s past the bound no CPU's timer entry
(`hardlockup/mod.rs`'s header). So for 133 s past the bound no CPU's timer entry
ran `poll`, and no CPU met both of the detector's conditions for 60 s, or none
was sampled; which of those is unmeasured.

Expand Down Expand Up @@ -108,12 +105,13 @@ runner asks for no reboot; the deadline was the only bound left, and it fired
now.** The next occurrence carries one census, the deadline's own: `expire`
seals the machine's census into the `WEDGED` record above the ring's tail
(`kernel/src/census.rs`), read at the moment the bound fired rather than at
whichever process last died before it.
whichever process last died before it, and names where each CPU's timer last
found the kernel (`deadline::KernelPcs`).

## The instrument that measures this already exists

`tests/metal/src/lib.rs`'s `deadline_lateness_ms` computes exactly
`reached − bound` out of the `DEADLINE_EXPIRED` line (`build/bootlog.rs:28`).
`reached − bound` out of the `DEADLINE_EXPIRED` line (`build/bootlog.rs`).
The arithmetic above is the instrument's own, done by hand because the run
was `toyos-metal` invoked directly and not the harness. No second instrument
is owed; a cause is.
Expand All @@ -129,10 +127,9 @@ is owed; a cause is.
Run 55's census shows cpu0 in that state at 3.47 s and other CPUs not; it
does not show what held every CPU's timer entry off `poll` for 133 s.
- The earlier deadline-ended T14 boot — run 25's `ccorpus`, back after 187 s
with no record because the reset left its stick unenumerable — was tracked
as a defect whose exit condition was a `loader.log` from a boot that ran to
its deadline with the ring tail naming what the machine was doing; this
record is that, and the commit that added this file closed it. Its
with no record because the reset left its stick unenumerable — is
`issues/a-job-list-hangs-with-interrupts-on-and-the-deadline-ends-it.md`,
restored in `14186a833` because run 55 ran no job list. Its
inference — that a deadline which fires is a machine on which some CPU was
still taking a timer interrupt — holds for the instant it fired; this record
shows the 133 s before it in which none did.
Expand Down
8 changes: 3 additions & 5 deletions issues/a-bcachefs-hole-costs-a-block-write-per-page.md
Original file line number Diff line number Diff line change
Expand Up @@ -12,12 +12,10 @@ shrink most recently freed and a hole that reads the previous tenant's bytes is
a data leak. That is correct and it is what FAT32 already does (`Fat32::set_len`
calls `zero_range` on every grow). The cost is linear in the hole: reaching page
N over a gap of M pages writes M zero blocks, one device write each, inside the
VFS lock a flush holds.
file server's write that reaches the page (`system/fileserver/src/data.rs`).

Every writer this kernel has appends
sequentially, so `covered` reaches `target` in one block and the zeroing loop
does not run — `fs_large_file` writes 1024 pages and allocates one block per
page with no gap. The paths that pay are a `lseek` past the end followed by a
A sequential append reaches `target` in one block, so the zeroing loop
does not run. The paths that pay are a `lseek` past the end followed by a
write, and a shrink-then-write-above-the-mark, whose gap is bounded by the
file.

Expand Down
16 changes: 9 additions & 7 deletions issues/a-boot-start-refused-a-present-device-runs-without-it.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,15 +7,17 @@ opened: 2026-09-24
# A boot start refused a device the machine has runs without it

A boot start's device refusal is fatal only for a device its `[programs]` row
marks as required; for any other, init logs the refusal loudly and starts the
program without that device (owner, 2026-09-30). No row can mark a device
marks as required; for any other, the supervisor logs the refusal loudly and
starts the program without that device (owner, 2026-09-30). No row can mark a device
required, so every boot start refused a device the machine has —
`NotSupported`, `AlreadyExists`, `ResourceExhausted` — runs without it:
`/system/bin/init`'s `start` says the refusal in the kernel's own word
(`init: netd: pci:8086:10c9 is on this machine and could not be handed over`)
and starts the program.
`system/supervisor/src/main.rs` says the refusal in the kernel's own word
(`supervisor: netstack: pci:… is on this machine and could not be handed
over`) and starts the program, since `claims::goes_on_without`
(`system/supervisor/lifecycle/src/claims.rs`) goes on for every refusal but an
ambiguous matcher or a claim the replaced process held.

**Exit**: a `[programs]` row can mark a device required, a host test holds the
mark and init's choice between a fatal refusal and a logged one, and a metal
row or, where none can, a guest test shows a boot start refused a required
mark and the supervisor's choice between a fatal refusal and a logged one, and a
metal row or, where none can, a guest test shows a boot start refused a required
device is fatal.
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,7 @@ Run 22's hung mkdir image left the stick holding `attempts` and `loader.log` and
left a 325-line file ending at a spawn. `loader.log` shows an ordinary handoff,
so that kernel started; `logd` either never created its file or its first write
never completed. The stick after the power cut is
`/Users/jan/.claude/jobs/2280e09e/tmp/t14-run22/stick-after-cycle.log`.
`t14-run22/stick-after-cycle.log`, outside the tree.

`tests/metal/src/lib.rs` reads that back as the absence of `Boot: complete`, which is the
same verdict it gives a boot that ran and stopped late. They are not the same
Expand Down
32 changes: 18 additions & 14 deletions issues/a-claim-spends-device-addresses-its-slot-never-gets-back.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,17 +8,17 @@ opened: 2026-09-26

Every `SYS_DEVICE_DMA_ALLOC` grant of a claim is placed at a fresh address of
its slot's domain (`kernel/src/pcidev/mod.rs`, `dma_alloc`), and the domain
never hands an address out twice (`kernel/src/arch/x86_64/vtd/table.rs`,
`Domain::reserve`). A holder bounded to `MAX_GRANT_TOTAL` per claim can still
claim, allocate and die over and over — a service a supervisor restarts does
exactly this — and each claim spends up to that much of the slot's addresses
and the remapping tables under them, which are never freed. The domain running
dry is a refusal (`ResourceExhausted`) and not a crash, but the slot is then
dead for the rest of the boot, and the tables are memory nothing returns.
Nor is running dry always a refusal: every table under a fresh address comes
from `Tables::alloc` (`kernel/src/arch/x86_64/vtd/table.rs`), whose
`expect("no physical memory for a remapping table")` panics the kernel when
physical memory runs out first, and no second-level table a claim made is
never hands an address out twice (`kernel/pure/iova.rs`, `Window::reserve`,
through `kernel/src/iommu/domains.rs`). A holder bounded to `MAX_GRANT_TOTAL`
per claim can still claim, allocate and die over and over — a service a
supervisor restarts does exactly this — and each claim spends up to that much of
the slot's addresses and the remapping tables under them, which are never freed.
The domain running dry is a refusal (`ResourceExhausted`) and not a crash, but
the slot is then dead for the rest of the boot, and the tables are memory
nothing returns. Nor is running dry always a refusal: every table under a fresh
address comes from `Tables::alloc` (`kernel/src/arch/x86_64/vtd/table.rs`),
whose `expect("no physical memory for a remapping table")` panics the kernel
when physical memory runs out first, and no second-level table a claim made is
ever freed.

What a holder *lends* (`SYS_DEVICE_DMA_MAP`) does not spend: it is placed in a
Expand All @@ -28,9 +28,13 @@ residue mechanism answers that only for a function no reset quiets.

**Exit condition.** A slot's grant addresses are bounded as its lent ones are
— reused once the function that held them is reset and quiet, or placed in a
window reserved once — and a guest test claims, allocates and releases a
function more times than a narrowed domain (`iommu-domain-narrow`) has room
for, and the last claim is served; the tables under a slot's addresses are
window reserved once — and a test claims, allocates and releases a
function more times than its domain has room for, and the last claim is
served. With no narrowed domain that is about 262,144 2 MiB grants on a 39-bit
unit, so a host test over `kernel/pure/iova.rs` and
`kernel/src/iommu/domains.rs` is the place to hold it; the
`iommu-domain-narrow` actuator, which narrowed a domain for this, was removed
in `29e100612`. And the tables under a slot's addresses are
bounded or freed once nothing maps through them, and a remapping table the
machine has no memory for is a refusal of the grant that needed it, not a
panic.
Original file line number Diff line number Diff line change
Expand Up @@ -7,14 +7,15 @@ opened: 2026-10-02
# A close of one handle ends every ring's poll on its object

`ops::close` answers every poll on a watch its object ends with `-NotFound`
(`Watch::cancel_polls`), in every ring, when any one handle to a pipe's read
end or an acceptor closes. A sibling handle from `dup` keeps the object open,
and its polls end all the same. `toyos::poller`'s `wait` hands the token of a
(`Watch::cancel_polls`), in every ring, when any one handle to a pipe's read end
or an acceptor closes. A sibling handle from `dup` keeps the object open, and
its polls end all the same. `toyos::poller`'s `wait` hands the token of a
negative completion to its caller as it does a ready one's and drops the
kernel's word for it, so a reader that takes that token for bytes and reads
blocking parks on an object that is open and empty. `wait_answers` hands the
word beside the token. No caller in the tree reaches it: fsd's acceptors are endowed and
never duplicated. `inbox_cancel_wakes` stages the close.
word beside the token. No caller in the tree reaches it: the file server's
acceptors are endowed and never duplicated. `inbox_cancel_wakes` stages the
close.

**Exit**: a poll ends only when the last handle to its source closes, or
`wait`, the form that drops a completion's result, is gone and every caller of
Expand Down
6 changes: 3 additions & 3 deletions issues/a-connect-between-two-accepts-is-reset.md
Original file line number Diff line number Diff line change
Expand Up @@ -36,9 +36,9 @@ netstack's listener is the node's now, whose queue is [tcp]'s: a connect
that arrives while another waits to be accepted is queued
(`a_connect_between_two_accepts_is_queued_not_reset`,
`system/netstack/node/tests/listeners.rs`), and in a guest two host peers
that dial before any accept are both accepted (`netstack_streams`, which on
smoltcp ended `the listener was woken for 1 of two peers`). What is left of the
exit is `lan_swap`.
that dialled before any accept were both accepted (`netstack_streams`, which on
smoltcp ended `the listener was woken for 1 of two peers`, and which
`ff1a11dbd` cut). What is left of the exit is `lan_swap`.

QEMU's own forward is a third place such a reset can come from, and it was
not ruled out of the reading above: its listener queues one connection, and a
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -6,8 +6,11 @@ opened: 2026-09-25

# A console's watch waits on the keyboard, and its readiness is the serial line

A blocking read of a console was moved onto the serial line's own wait; a
watch through an inbox (`Poller::watch` of a console handle) was not. Its one
A console's readiness is the serial line's (`drivers::serial::has_data`),
and nothing on the serial line posts a wake. A blocking read of a console
(`sys_read`, `kernel/src/syscall/io.rs`) parks on its own thread's watch and is
woken only by its own 10 ms repoll (`CONSOLE_REPOLL`). A watch through an
inbox (`Poller::watch` of a console handle) has no source at all. Its one
source was the kernel's keyboard watch, which went with the kernel's input
streams (the small-kernel track's step 10), so a console has no readable
watch at all now: registering one is refused, and libc's `poll` reads every
Expand All @@ -16,6 +19,6 @@ program that polls its console for input spins rather than waits.

## Exit condition

A console's poll source and its readiness are the same source, and a test that
watches a console handle through a poller and types on the serial line sees
the watch complete.
A console's poll source and its readiness are the same source, a blocking read
waits on it with no repoll, and a test that watches a console handle through a
poller and types on the serial line sees the watch complete.
Original file line number Diff line number Diff line change
Expand Up @@ -7,13 +7,13 @@ opened: 2026-09-26
# A corrupt chain met by a queued free is answered by the next call

`Fat32::atomic` re-drives an earlier call's queued repair before a mutating
call starts (`settle_first`, `lib/fat32/src/repair.rs`). A queued
call starts (`settle`, `lib/fat32/src/repair.rs`). A queued
`Repair::Free` walks the rest of a chain whose name is already gone, and a
walk that meets a cyclic or out-of-range link answers `Error::CorruptChain`.
That error is returned by whichever call came next, for a chain it never
named, and the call did not start. fsd logs it as that call failing
(`logged` in `userland/fsd/src/fat.rs`: "`<op>` of '`<name>`': corrupt cluster
chain") and its client reads `SyscallError::Io`.
named, and the call did not start. The file server logs it as that call failing
(`logged` in `system/fileserver/src/fat.rs`: "`<op>` of '`<name>`': corrupt
cluster chain") and its client reads `fs::Error::Io`.

Two things are lost:

Expand All @@ -27,7 +27,7 @@ Two things are lost:

## Owner

`toyos-fat32`, which owns `settle_first` and `free_chain`; the adapter's log
`toyos-fat32`, which owns `settle` and `free_chain`; the adapter's log
line follows from what the crate answers.

## Exit condition
Expand Down
Loading