Skip to content

lemans-evk: ufs/pcie: intermittent boot failures in qcom-deb-images CI #1053

Description

@lool

lemans-evk is intermittently failing to boot in the daily qcom-deb-images CI. The failures are seen in both trixie and forky boot jobs. The common LAVA-level failure is that the board never reaches the login prompt:

lava/login-action: fail
lava/auto-login-action: fail
lava/minimal-boot: fail
login-action timed out after 174 seconds

After comparing failed logs with working lemans-evk logs, the most likely issue is that the UFS-backed root device is not becoming available during early userspace. The failures do not look like LAVA prompt-matching problems.

Good reference jobs

Working forky boot

Good boot sequence:

Booting initrd of Debian GNU/Linux forky/sid dracut-112-2 (Initramfs)
systemd[1]: Expecting device dev-disk-by\x2duuid-b03203d4...device
ufshcd-qcom 1d84000.ufshc: ufshcd_populate_vreg: Unable to find vdd-hba-supply regulator, assuming enabled
ufshcd-qcom 1d84000.ufshc: ufshcd_populate_vreg: Unable to find vccq2-supply regulator, assuming enabled
scsi host0: ufshcd
sda: sda1 sda2
OK Found device dev-disk-by\x2duuid-... - MT128GBCAV2U31AI root.
OK Mounted sysroot.mount - /sysroot.
OK Reached target initrd-switch-root.target - Switch Root.
Matched prompt #16: login:

Approximate timing from the successful boot:

Milestone Approx. kernel time
/init starts ~5.7s
dracut starts waiting for root UUID ~7.1s
UFS probe starts ~8.56s
scsi host0: ufshcd appears ~8.64s
sda: sda1 sda2 appears ~8.83s
root UUID found ~10–11s
sysroot mounted ~11s
login prompt reached LAVA login-action completes in ~29s

Working trixie boot

Good boot sequence:

Run /init as init process
Starting systemd-udevd version 257.13-1~deb13u1
ufshcd-qcom 1d84000.ufshc: ufshcd_populate_vreg: Unable to find vdd-hba-supply regulator, assuming enabled
scsi host0: ufshcd
sda: sda1 sda2
Found device dev-disk-by\x2duuid-...
Matched prompt #16: login:

Approximate timing from the successful boot:

Milestone Approx. kernel time
/init starts ~5.6s
UFS probe starts ~6.0s
scsi host0: ufshcd appears ~6.1s
sda: sda1 sda2 appears ~6.2s
root device found ~11–12s
login prompt reached LAVA login-action completes in ~24s

Failed jobs inspected

trixie failures

Date GitHub Actions run LAVA job Device / serial Observed failure
2026-09-06 34003983904 401508 rb8-hyd-08, 0x32488DEE stalls after early UFS/regulator messages; no SCSI/root/login
2026-09-03 33703237836 398145 rb8-02, 0xAD4628C5 stalls around early initramfs probing; no SCSI/root/login
2026-08-28 33139562175 386333 rb8-hyd-06, 0xEA6665DD stalls after early UFS/regulator messages; no SCSI/root/login
2026-08-27 33032862168 383950 rb8-hyd-02, 0xBFAB4B3D stalls after early UFS/regulator messages; no SCSI/root/login
2026-08-26 32917265094 380521 rb8-hyd-07, 0xA7AA388C stalls after early UFS/regulator messages; no SCSI/root/login
2026-08-22 32542391818 375781 rb8-hyd-07, 0xA7AA388C stalls after early UFS/regulator messages; no SCSI/root/login
2026-08-19 32210651999 369992 rb8-hyd-06, 0xEA6665DD stalls around early initramfs probing; no SCSI/root/login
2026-08-16 31923268701 364499 rb8-02, 0xAD4628C5 stalls after early UFS/regulator messages; no SCSI/root/login
2026-08-15 31860652852 364207 rb8-hyd-06, 0xEA6665DD stalls after early UFS/regulator messages; no SCSI/root/login

Typical failed trixie signature:

Run /init as init process
Starting systemd-udevd version 257.13-1~deb13u1
ufshcd-qcom 1d84000.ufshc: ufshcd_populate_vreg: Unable to find vdd-hba-supply regulator, assuming enabled
ufshcd-qcom 1d84000.ufshc: ufshcd_populate_vreg: Unable to find vccq2-supply regulator, assuming enabled
login-action timed out after 174 seconds

In the failed trixie logs, the following good-boot milestones are generally missing:

scsi host0: ufshcd
sda: sda1 sda2
Found device dev-disk-by\x2duuid-...
Matched prompt #16: login:

forky failures

Date GitHub Actions run LAVA job Device / serial Observed failure
2026-09-01 33459098736 392647 rb8-03, 0x1B9B0BD5 dracut waits for root UUID; no SCSI/root/login before timeout
2026-08-29 33225634472 388571 rb8-hyd-03, 0x60C9B837 dracut waits for root UUID; no SCSI/root/login before timeout
2026-08-25 32795929046 378186 rb8-hyd-06, 0xEA6665DD dracut waits for root UUID; no SCSI/root/login before timeout
2026-08-23 32609318526 376004 rb8-hyd-06, 0xEA6665DD dracut waits for root UUID; no SCSI/root/login before timeout
2026-08-20 32319464158 372276 rb8-hyd-07, 0xA7AA388C dracut waits for root UUID; no SCSI/root/login before timeout
2026-08-18 32093821329 367046 rb8-03, 0x1B9B0BD5 dracut waits for root UUID; no SCSI/root/login before timeout

Typical failed forky signature:

Booting initrd of Debian GNU/Linux forky/sid dracut-112-2 (Initramfs)
systemd[1]: Expecting device dev-disk-by\x2duuid-<root-uuid>.device - /dev/disk/by-uuid/<root-uuid>...
Starting dracut-initqueue.service - dracut initqueue hook...
ufshcd-qcom 1d84000.ufshc: ufshcd_populate_vreg: Unable to find vdd-hba-supply regulator, assuming enabled
ufshcd-qcom 1d84000.ufshc: ufshcd_populate_vreg: Unable to find vccq2-supply regulator, assuming enabled
login-action timed out after 174 seconds

Compared with working LAVA 401846, the failed forky jobs do not reach:

scsi host0: ufshcd
sda: sda1 sda2
OK Found device dev-disk-by\x2duuid-... root
OK Mounted sysroot.mount - /sysroot
Matched prompt #16: login:

Likely cause

The most likely cause is an intermittent UFS/root block-device enumeration failure during early userspace.

Reasoning:

  1. The devices are successfully flashed through QDL.

  2. UEFI starts and reports Boot Device: UFS.

  3. The kernel starts and reaches /init.

  4. In successful boots, the logs quickly continue from early ufshcd-qcom regulator messages to:

    scsi host0: ufshcd
    sda: sda1 sda2
    Found device dev-disk-by\x2duuid-...
    Mounted sysroot.mount - /sysroot
    
  5. In failed boots, those milestones are missing. The system stalls before the root filesystem is mounted or before login can appear.

  6. The same physical boards/hosts have both failures and later passing jobs, so this appears intermittent/racy rather than a permanently bad board.

  7. The recurring kernel/sched/idle.c:269 warning appears in both failing and passing boots, so it is not a sufficient explanation by itself.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions