Skip to content

A cold-booted Firecracker VM starts with its clock about a second behind the host #1354

Closed

Description

@ejc3

A Firecracker VM that cold-boots comes up with its clock 0.86 to 1.14 s behind the host, and stays there until chronyd slews or steps it back (up to about 12 s at chronyd's maximum slew rate).

Evidence

Four cold boots (fcvm podman run --no-snapshot, rootless, arm64), guest date +%s.%N bracketed between two host reads right after healthy:

run offset at healthy
default profile -857 ms
default profile -1144 ms
default profile -899 ms
nested profile -960 ms

15 s later each read 0 ms. One boot's console logged system clock synchronized from host (stepped -1.572903 s): the boot sync moved the clock back.

Mechanism

  • src/commands/podman/vm_config.rs stamps host_clock: host_clock_now() while building the boot plan, and src/firecracker/config.rs writes it into MMDS before the VM boots.
  • fc-agent reads it after the kernel and systemd are up and calls bootplan::set_system_clock(&metadata.host_time_ns, Duration::ZERO) (fc-agent/src/mmds.rs), so the guest lands on the instant of the stamp, behind the host by the time from the stamp to the read.
  • Cloud Hypervisor is not affected: its boot-plan listener stamps the host clock at each accept (serve_bootplan). Restores are not affected: the host stamps the restore document after the resume.

This is the lag #1353 attributed to the snapshot pause ("0.88 s slow of NTP time" in a cold-booted baseline). On the default profile the clock keeps running through a pause; only the nested (NV2) profile freezes it, which #1355 fixes.

Fix direction

Stamp at the guest's read rather than at plan build (for example a guest-side CLOCK_BOOTTIME offset from a stamp taken at InstanceStart, or a vsock time query as Cloud Hypervisor does), and do not step a correct clock back to a stale stamp.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions