[Japabu]

Earlier status (before the rebase): cargo run -- --ci host exit 0 at 13fef0ed0; the other checks below were run at b77180845.

Status: draft. Clang installs and compiles in the guest. The link is blocked on the memory track (item 5 of issues/toyos-builds-itself.md), and this PR does not work around it. Rebased onto main after #988 and #991; head 6f398f518.

What the guest row shows (hosted_clang_hello, QEMU x86_64):

• pkg install exits 0 and installs 279 MB at /apps/clang.
• clang -c hello.c exits 0.
• cc hello.c fails: the driver's posix_spawn of /apps/clang/bin/clang -cc1 is refused ENOMEM. That was read with libc's real strerror applied temporarily; today's strerror says "unknown error" for every code, and the libc follow-up fixes that.
• A small std parent spawns /apps/clang/bin/ld.lld --version fine.

Cause. A program on a file server is spawned from a copy in one memory object. SYS_SHM_CREATE allocates that object physically contiguous (kernel/src/object/shm.rs, alloc_contiguous), about 148 MB for clang. The errno is measured; that this allocation is the one refused is inferred, since the kernel logs nothing for it. The memory track's stages 1 to 3 remove contiguous user memory, and process-memory-is-2-mib-pages… records this row as its check. The row stays as written: it asserts the M2 exit, and is red until those stages land. It runs on demand only.

This round:

• ROOM is renumbered 23, after main's DUP, SEEK and EXPORT.
• The hosted clang case gains usbd drives every xHCI controller; the kernel drives no USB #971's block service and NVMe driver rows; without them DATA was absent and pkg saw no room.
• libc's posix_spawn reads a served program into an image object, as std's Command does.
• The track's table is updated for the package, the link's blocker (new item 5) and the renumbered rows.

Checks at the head: --ci host 0 (87 steps), both builds 0, libc_children passes, hosted_clang_hello 1 as above. Evidence: https://github.com/ToyOSOrg/evidence/tree/bc6bfb7a24cc74dfc9f6759fa454ba3dc3f3dd10/r2

Problem

M2 of issues/toyos-builds-itself.md is clang, LLD and their runtime installed by /system/bin/pkg, and clang hello.c && ./a.out working in the guest. hosted_clang_hello staged the binaries on ROOT because pkg could not install a toolchain: it read the whole archive into memory, inflated it under a 256 MiB ceiling (MAX_INFLATED) and refused every link. The toolchain installs 279 MB.

Resulting behaviour

• pkg streams. An archive is read twice and never held: a survey hashes the whole file and plans it header by header; the unpack meets that plan header for header and the digest again at the end. An archive changed between the reads installs nothing, and the second read writes only paths the first checked. Memory is a 1 MiB buffer and the plan's headers (at most 65536 entries).
• Room is asked, not guessed. A new file-server request, ROOM, answers the free bytes of the volume behind a capability and its allocation unit (bcachefs: spare blocks less the delete reserve; FAT32: free clusters; none for an absent or read-only volume). The survey refuses an entry when the plan passes the room, at its header and before its data, so the room replaces MAX_INFLATED as the bound on what a survey inflates. Metadata that still fills a volume part way meets the existing unwind.
• Durable before done. Before the manifest is written, pkg syncs the volume through /apps's capability, which refuses if any file was left unwritten: a file's last close can be refused, and std's File drop discards that answer. (The guest showed exactly this before bcachefs: a split of a tree's first child keeps its siblings after it #969: pkg reported success over an entry the server failed to write.)
• Links, on the merits. ustar carries symlinks, the package model unpacks an archive as-is, the DATA volume stores symlinks, and a toolchain names its driver several ways. A package's symlink must resolve inside it: relative, .. only first and never past the package's own directory, and nothing is written beneath a link or a file, so no chain of links reaches outside. The package's directory is no link and manifest.toml is the installer's alone. Hard links (the filesystem has none), pax and GNU long names stay refused; GNU tar's header is read without its non-prefix field as a prefix.
• A package's program is <name>/<name> (gbae's shape) or <name>/bin/<name> (a toolchain's); both is refused as unsaid. toyos_manifest::package::program_is_inside accepts a nested file of the package's own directory.
• Verification is unchanged in kind: SHA256SUMS beside the file, now over the streamed bytes, every byte of the file included.
• The toolchain package. cargo run -- --hosted-clang also writes target/hosted-clang/clang-v22.1.8-toyos-x86_64.tar.gz (83.8 MB) and its SHA256SUMS: clang/bin/{clang,ld.lld}, links cc and clang++ to clang, bin/x86_64-unknown-toyos.cfg naming --sysroot=<CFGDIR>/../sysroot (clang's documented configuration-file lookup beside its executable, so a compile needs no --sysroot and the hosted clang's key does not move), clang's resource headers at lib/clang, the C sysroot's include and lib at sysroot, and LLVM's LICENSE.TXT. Deterministic: fixed order, no owner, no time.
• The guest row. hosted_clang_hello stages only the package and its sums, boots tests/images/hostedclangcase.toml (now with diskserver, fileserver and pkg) on a 1 GiB DATA (BootOptions::data_bytes), runs pkg install … --yes, then compiles with /apps/clang/bin/clang -c, links with /apps/clang/bin/cc hello.c, runs it and compares both products with the host's bytes.
• pkg is declared exempt.manages: asking ToyOS's file server for room is a ToyOS-only job.

What stopped the compile before #988 (measured, historical)

• Installed clang's -v: InstalledDir: is empty, so the configuration file beside it is never looked for. LLVM's getMainExecutable takes the dladdr route on ToyOS and realpaths the name; libc's realpath resolves by the kernel's mounts (ROOT and /tmp), which hold no /apps. The staged run of libc starts and waits for children, and clang's driver links inside ToyOS #961 passed because ROOT is the kernel's.
• With --sysroot=/apps/clang/sysroot -resource-dir /apps/clang/lib/clang/22 given explicitly, clang reports ignoring nonexistent directory "/apps/clang/sysroot/include" while std, in the same job, sees it and stdio.h (2073 bytes) in it. libc's open/stat go to the kernel only.

Both are issues/c-programs-name-no-file-a-file-server-holds.md, whose exit is libc on one SDK file-server client that std uses too. The track now says so.

Decisions

• Static PIEs make issues/a-package-cannot-ship-its-own-libraries.md moot for M2: neither binary has a PT_INTERP or a DT_NEEDED. It stays open for M3's rustc launcher.
• The configuration file rather than DEFAULT_SYSROOT: relocatable, upstream's mechanism, no LLVM rebuild.
• Two reads rather than unpack-then-verify: verification and the room check precede the consent prompt and every write.

Checks and evidence

All at b77180845 unless said.

• cargo run -- --ci host: exit 0, 87 steps green (also green at the pre-bcachefs: a split of a tree's first child keeps its siblings after it #969 3c0982343).
• cargo run -- --build-only: exit 0 for x86_64 and for --arch aarch64.
• Host tests added: pkg archive/plan/install (truncated archive, oversized entry and a 4 GiB-claiming bomb refused at its header, path escape, link targets outside the package, a path beneath a link or file, a swapped archive writing nothing outside the package, a changed archive installing nothing, the digest covering bytes past the tar's end, a refused sync unwinding, an existing directory never swept); fileserver ROOM; the package's layout and determinism (build/hostedclang.rs).
• Negative controls, each applied as a patch, built, run and reverted in one script: room bound removed, link check removed, beneath-check removed, second read trusting its own headers, header truncation read as the archive's end, digest not draining the file, room counting the reserve, and both halves of the btree fix: every one turns its test red (exit 101), and the unmutated suite passes. Two earlier mutations survived (a short skip, a partial header) because the next header read refuses either; the redundant checks were deleted.
• Guest cargo test --test toyos-build -- hosted_clang_hello (QEMU x86_64, Headless profile, local): exit 1. pkg install exits 0 (installed clang 22.1.8 at /apps/clang (279248896 bytes)), then clang -c exits 1 on stdio.h as above. Before bcachefs: a split of a tree's first child keeps its siblings after it #969 the same run logged the file server's NotFound entry failure; after it, none.
• Hosted clang of this tree: key 1ed2a0875fe8cbf6, clang 147,727,208 bytes and ld.lld 85,761,568 bytes; readelf -l -d shows no INTERP and no NEEDED.

Evidence, at ToyOSOrg/evidence@371b2e4 (index): ci-host2.log, ci-host.log (pre-#969), build-only-x86_64.log, build-only-aarch64.log, guest-final.log, guest1.log (the NotFound entry failure before #969), guest-diag1.log (InstalledDir: empty), guest-diag2.log (libc vs std on /apps), static-pie.log, mutations/run-final.log with run.sh and the patches beside it, hostedclang2.log.

Risks and gaps

• The guest row cannot pass until libc reaches served files.
• A shell finds no installed command by name (PATH is /system/bin); recorded in the track.
• The package carries LLVM's licence, not the licences of what libtoyos_c.a links; owed before a package is published beyond a test's ROOT; recorded in the track.
• clang++ is shipped as a link and not exercised in the guest.
• AArch64 has no hosted clang yet.
• A DATA volume written before bcachefs: a split of a tree's first child keeps its siblings after it #969 may hold an out-of-order node.

🤖 Generated with Claude Code