Issue 183 riscv64 checkpoint restore

[sachin2605]

Summary

Port the checkpoint/restore engine to riscv64, matching x86_64 functionality:

• Restore stub (C): full #ifdef __riscv64 port with signal frame layout,
  syscall numbers, sc6() ecall wrapper, rt_sigreturn, and _start entry
• Blob builder (Rust): rearm_restartable_syscall (a0 return, a7 syscall_nr,
  4-byte ecall rewind) and build_fpstate_image (raw passthrough)
• Supervisor (Rust): riscv64 synthetic-image restore test with 32-register
  file, a7+ecall machine code, Sv39-safe addresses
• Build system: detect riscv64 target and try cross-compilers
• Integration test: relax runtime gate so test_restore_glibc_vdso_program_resumes
  runs on riscv64
• Unit tests: riscv64 equivalents for all x86_64-gated rearm and fpstate tests

Files changed

File	Change
restore-stub.c	+157/-30: riscv64 port (signal frame, mmap, dup3, rt_sigreturn, _start)
resume.rs	+136/-8: generalize cfg gate, add riscv64 synthetic-image test
restore_blob.rs	+91/-17: rearm_restartable_syscall, build_fpstate_image, +3 riscv64 tests
build.rs	+23/-1: riscv64 cross-compiler detection
sandbox.rs	+4/-1: relax restore gate
test_restore.rs	+4/-2: relax runtime gate

Key design decisions

• Blob format is arch-agnostic: same BLOB_MAGIC, VERSION, HEADER_LEN;
  register count determined by regs_len; both arches are LE
• ecall is 4 bytes → PC rewind by 4 (vs x86_64's 2-byte syscall)
• dup3 (syscall 24) replaces dup2 on riscv64
• No xstate framing: kernel stores __riscv_d_ext_state directly in
  sc_fpregs — no separate fp_buf, no magic framing
• Signal frame: uc_mcontext at offset 0xA8=168; sc_regs[32] is 1:1
  with user_regs_struct — no remapping, no CSGSFS, no arch_prctl
• Sv39 user space ends at 256 GiB — synthetic test addresses kept below
  0x40_0000_0000

[congwang-mk]

Reviewed the riscv64 port. The direction is right and the ABI research (1:1 sc_regs/ptrace ordering, no arch_prctl, no xstate framing, dup3) is accurate, but the code has not run yet: the riscv64 stub does not compile, and CI cannot tell you that because build_static downgrades a compiler failure to a cargo:warning and every riscv64 test then skips on !stub.exists(). The Rust cross-build (riscv64) job is green for the same reason.

I checked the findings below on a real riscv64 host (Sv39, 6.18.3) and against the kernel uapi headers rather than by reading alone.

Six blockers, in the order you'd hit them:

1. restore-stub.c — u8 is never typedef'd; the riscv64 branch fails to compile.
2. restore-stub.c — uc_mcontext is at 176, not 168; rt_sigreturn restores a register file shifted by 8 bytes.
3. build.rs / restore_blob.rs:45 — the 3 TiB STUB_BASE is above the Sv39 user ceiling; the stub SIGSEGVs at exec on real boards.
4. resume.rs — the synthetic payload's lui/addi yields 0x2000_0040, not CODE + 64.
5. restore_blob.rs — the restart re-arm writes the syscall number into a0, which is the first argument on riscv64; and orig_a0 isn't reachable through NT_PRSTATUS.
6. build.rs — a native riscv64 build drops cc from the compiler list, so the stub is never built there at all.

Plus smaller things inline: the #ifdef __riscv guard also matches RV32, and the 544-byte sc_fpregs bound overshoots the kernel's 528-byte union (the clamp itself is dead code).

One doc nit outside the diff: sandbox.rs:1067 still says "x86_64 restore engine only." in the restore_interactive doc comment, three lines above the gate this PR widened.

Since none of this can be caught by the current CI shape, it's worth making the stub build failure loud on the arches we claim to support — a cargo:warning is invisible in a green build, which is how all six of these got this far.

[congwang-mk]

Blocker: this branch does not compile — u8 is never declared.

The typedefs at lines 136-138 are u64, u32, i64 only; u8 was never needed before this PR. Building the stub natively on riscv64 (Debian gcc 14, mmu: sv39) with exactly the flags build.rs passes:

/tmp/stub.c:254:5: error: unknown type name 'u8'
  254 |     u8  fpregs[544];
/tmp/stub.c:265:5: error: unknown type name 'u8'
/tmp/stub.c:271:5: error: unknown type name 'u8'

Adding typedef unsigned char u8; next to the others makes it compile and link.

This is why CI is green while none of the riscv64 work has ever run: build_static turns a compiler failure into a cargo:warning, and every riscv64 test then early-returns on !stub.exists().

[congwang-mk]

Blocker: uc_mcontext is at offset 176 (0xB0), not 168 (0xA8).

struct sigcontext has 16-byte alignment on riscv64 — both __riscv_q_ext_state.f[64] and __riscv_extra_ext_header.__padding carry __attribute__((aligned(16))), so the union and therefore struct sigcontext are 16-aligned. 168 is not a multiple of 16, so the kernel pads to 176. Measured on a riscv64 host against <asm/ucontext.h>:

offsetof(kernel struct ucontext, uc_mcontext) = 176 (0xb0)
sizeof(struct sigcontext) = 784

struct sigctx here has alignment 8, so the compiler places mc at exactly 168 and the whole register file lands 8 bytes low. rt_sigreturn then reads pc from gregs[1] (ra), sp from gregs[2] (gp), and so on. In the new synthetic test ra is 0, so the restored process jumps to 0 and dies.

Fix: __attribute__((aligned(16))) on struct sigctx (or widen __unused to 128 bytes and keep the comment honest).

[congwang-mk]

__riscv is also defined on RV32, where __riscv_xlen == 32. The u64-based struct sigctx/struct uctx mirrors, the SYS_* numbers, and the 32x8-byte sc_regs copy are all RV64-only, and the #else #error added below never fires there — so an RV32 build would compile silently wrong rather than being rejected.

Suggest #if defined(__riscv) && __riscv_xlen == 64.

[congwang-mk]

Two things here:

1. This clamp is dead. Line 349 already does if (h->fpstate_len > FP_MAX) die(3);, and FP_MAX is 544 on riscv64 — the same bound. It also mutates the header in place inside ctrl_buf, which the rest of the stub treats as read-only after validation.

2. If it were reachable, 544 is the wrong bound. The kernel's second sigcontext member is union { union __riscv_fp_state sc_fpregs; struct __riscv_extra_ext_header sc_extdesc; }, which is 528 bytes, with sc_extdesc.reserved at offset 516 and sc_extdesc.hdr at 520 (verified: sizeof(struct sigcontext) == 784 = 256 + 528). A 544-byte copy would overwrite reserved, and restore_sigcontext() returns -EINVAL on a non-zero reserved, so rt_sigreturn would fail instead of resuming.

Not reachable today (riscv NT_PRFPREG yields 264 bytes), but FP_MAX should be 516/528 rather than 544, and the comment above it calls 544 the Q-extension size, which it isn't.

[congwang-mk]

Blocker: on riscv64 a0 is the first argument, not the syscall number, so this corrupts the restarted call.

The x86_64 version works because rax carries both the number and the return value. riscv64 splits them: the number stays in a7 (the kernel never clobbers it) and a0 is overwritten by the return value. The kernel's own fixup is arch/riscv/kernel/signal.c:

case -ERESTARTNOHAND:
case -ERESTARTSYS:
case -ERESTARTNOINTR:
case -ERESTART_RESTARTBLOCK:
        regs->a0 = regs->orig_a0;
        regs->epc = restart_addr;
        break;

Concretely: a checkpoint taken while blocked in read(5, buf, n) has a0 = -ERESTARTSYS, a7 = 63. After this re-arm the ecall re-executes as read(63, buf, n) — EBADF, or silently the wrong fd.

The harder problem is that orig_a0 is not recoverable from ps.regs: riscv_gpr_get() writes only sizeof(struct user_regs_struct) (256 bytes, pc..t6), and orig_a0 sits past status/badaddr/cause in pt_regs. So the re-arm as designed cannot be implemented from the captured register file — it needs an extra capture (e.g. /proc/<pid>/syscall, which does expose the original arg0) or should be left unimplemented on riscv64 rather than writing a wrong a0.

Two related notes:

• For -ERESTART_RESTARTBLOCK the kernel sets a7 = __NR_restart_syscall; it does not re-run the original number.
• The new unit test at line 911 asserts regs[10] == 34, i.e. it locks in this behaviour, so it will need to change with the fix.

[congwang-mk]

Blocker: a native riscv64 build loses the compiler entirely.

TARGET is riscv64gc-unknown-linux-gnu for a plain cargo build on a riscv64 host too, so the candidate list becomes just these two cross-prefixed names — neither of which exists on a riscv64 machine, where the compiler is /usr/bin/cc / /usr/bin/gcc (checked on our riscv64 test host). The stub is then never built, stub_links_at_the_reserved_base, the new synthetic test, and test_restore_glibc_vdso_program_resumes all silently return early, and restore_interactive fails at runtime with "restore-stub was not built" — on exactly the platform this PR is for.

Keep cc in the list, gated on HOST so a genuine cross build doesn't silently produce a host binary:

let host = std::env::var("HOST").unwrap_or_default();
// ... riscv64 branch:
if host.starts_with("riscv64") {
    &["cc", "riscv64-linux-gnu-gcc", "riscv64-unknown-linux-gnu-gcc"][..]
} else {
    &["riscv64-linux-gnu-gcc", "riscv64-unknown-linux-gnu-gcc"][..]
}

Minor, same hunk: || target.starts_with("riscv64gc") on line 39 is subsumed by target.starts_with("riscv64").

[congwang-mk]

Blocker: 3 TiB is above the Sv39 user-address ceiling, so the stub cannot be exec'd on real riscv64 hardware.

restore_blob::STUB_BASE is 0x300_0000_0000 and this link address matches it, but Sv39 (what JH7110, SpacemiT K1, and most riscv64 Linux boards run) gives TASK_SIZE = 256 GiB. The PR's own new comment in resume.rs says addresses must stay below 0x40_0000_0000; this is 12x over that.

Verified on a riscv64 host (mmu: sv39, [stack] at 0x3ff1469000):

mmap STUB_BASE 0x300_0000_0000  -> ENOMEM
mmap test CODE 0x200_0000_0000  -> ENOMEM
mmap            0x2000_0000     -> ok

$ ./restore-stub          # ET_EXEC with p_vaddr = 0x300_0000_0000
Segmentation fault        # elf_map() fails past the point of no return

Relinking the same source at -Wl,-Ttext-segment=0x3000000000 (192 GiB, under the ceiling) makes it exec and run correctly — it reaches the blob parser and exits 3 on a short blob, as designed.

So riscv64 needs its own stub base inside the Sv39 window, chosen here and in restore_blob.rs:45 together (the stub_links_at_the_reserved_base test exists precisely to catch them drifting apart, and it will once the stub actually builds).

Aside, since it's the obvious suspicion with a 3 TiB text segment: -mcmodel is not a problem. GCC on riscv64 Linux defaults to medany, and la sp, stub_stack assembles to auipc/ld through the GOT — the link succeeds at either base.

[congwang-mk]

Blocker: this pair does not materialize CODE + 64.

lui a1, 0x20000 puts 0x20000 in bits 31:12, giving a1 = 0x2000_0000; the addi makes it 0x2000_0040. But CODE + 64 is 0x200_0000_0040 — lui+addi reach 32 bits, and CODE is 2^41. The restored payload calls write(10, 0x2000_0040, 1) against an unmapped address, gets -EFAULT, and the test fails at assert_eq!(n, 1, ...).

The encoded instructions are self-consistent with CODE = 0x2000_0000, so it looks like the constant on line 588 picked up three extra zeros.

That also fixes a second problem in the same test: 0x200_0000_0000 is 2 TiB, which is 8x above the Sv39 ceiling the comment on line 585 cites, and mmap there returns ENOMEM on Sv39 hardware (measured). const CODE = 0x2000_0000 / const STACK = 0x2001_0000 satisfies both the encoding and the comment.