Skip to content

fix(computers): a failed resolve reports apt's error, not the resolver's traceback - #621

Merged
andrewpark-scaleai merged 2 commits into
mainfrom
andrew.park/resolve-failure-says-why
Nov 11, 2026
Merged

andrewpark-scaleai merged 2 commits into
mainfrom
andrew.park/resolve-failure-says-why

Conversation

@andrewpark-scaleai

@andrewpark-scaleai andrewpark-scaleai commented Nov 11, 2026 •

Copy link
Copy Markdown
Contributor

Why

integration-local-slow failed on most runs between about 16:00 and 18:00 UTC on 11-11, on main and on PRs. Computer env builds failed at the resolve stage, so every test that deploys one got DeployRefused: ... has no ready build. Each failure gave the same reason:

resolving the packages failed: subprocess.CalledProcessError: Command '['apt-get', 'install', '-y', '-q', '--download-only', ...]' returned non-zero exit status 100.

That names the command but not why apt failed. composed.failed reports the last line of the resolver's output, and that line is the resolver's own traceback, which buries apt's E: line printed just above it. So CI can't show the cause.

From five failing runs, I can tell this much:

  • The failures cluster in time. Every run from 16:05 to 17:50 lost most of its amd64 resolves. Later runs, main's rerun at about 18:00 and this PR's at about 19:00, resolved every build.
  • Within one run, some builds resolved and others failed in the same minute.
  • The apps that add the fastfetch Launchpad PPA (notes, desk) failed 19 of 20 times. Chromium, which uses only Ubuntu's archive, failed 6 of 15 times.
  • The same resolves pass locally in the base image.

My best guess is an upstream outage on the Ubuntu or Launchpad mirrors that outlasted apt's own retries, but that's an inference. This change makes the next failure name its cause.

The arm64 resolves that fail in every shard are expected: "this machine can't build linux/arm64: its Docker has no emulation for it".

What

When a command the resolver runs fails, the resolver exits with that command's code and no traceback. A command whose output was captured, such as dpkg-deb, has that output printed first. So the last line, which becomes the build's reason, is the failing command's own error.

Validation

I ran the resolver in the base image on a package that doesn't exist:

  • before: subprocess.CalledProcessError: Command '[...]' returned non-zero exit status 100.
  • after: E: Unable to locate package no-such-package-xyz

Two new unit tests cover the exit code, the missing traceback, and a captured command's error. This PR's integration run passed, resolving all 15 of its amd64 builds.

Not fixed here: the outage itself. If the next failure's reason shows a mirror or the PPA, the follow-up is a targeted change, such as a retry of the whole resolve or a different mirror.

RetriggerView in GreptileConfidence Score: 5/5

The PR appears safe to merge.

Summary

Failed resolver commands now exit without a traceback. Captured errors are printed before exit.

  • Failed computer resolves now report the command's error instead of a traceback.

Reviews (2) · Last reviewed commit: "fix(computers): a failed command whose o..." · Reviewed by Greptile

@andrewpark-scaleai
andrewpark-scaleai requested a review from a team as a code owner November 11, 2026 18:46
@andrewpark-scaleai
andrewpark-scaleai enabled auto-merge (squash) November 11, 2026 19:07
@andrewpark-scaleai
andrewpark-scaleai merged commit f63db8d into main Nov 11, 2026
34 of 35 checks passed
@andrewpark-scaleai
andrewpark-scaleai deleted the andrew.park/resolve-failure-says-why branch November 11, 2026 22:14
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants