Skip to content

Recover from damaged build artifacts instead of failing every build - #1329

Merged
mkarlesky merged 2 commits into
next_versionfrom
bug/damaged-artifact-recovery
Oct 9, 2026
Merged

mkarlesky merged 2 commits into
next_versionfrom
bug/damaged-artifact-recovery

Conversation

@mkarlesky

Copy link
Copy Markdown
Member

A single damaged file under build/ could leave every later build failing until a clobber. Fault-injection stress testing found three such cases. This PR makes each one cost a rebuild of the affected step instead.

Failures

  • A truncated or emptied cached test result (build/test/results/<test>.pass) was reused as-is. The summary report then raised undefined method `size' for nil or undefined method `empty?' for nil.
  • A truncated .d file can end in a directory path. The directory was tracked as a dependency, and hashing it raised Is a directory @ io_fread before the recompile that would have rewritten the file.
  • A deleted build-directives cache (build/test/preprocess/build_directives/<test>/) raised Could not find YAML file …_source_files.yml. An emptied one was worse: it loaded as no directives at all, silently dropping TEST_SOURCE_FILE() sources. That surfaced only as undefined reference at the next relink.

Cause

Two of the three share a cause: a freshness record vouched for an artifact it never checked. Test execution was skipped when <test>.fixture_run was fresh, then reused the result file. Directive parsing was skipped when the preprocessed test file was fresh, then loaded the cache. Neither artifact was tracked, so damage to it was trusted indefinitely.

The .d case was the tracker counting any existing path as a dependency, directories included.

Fix

The dependency tracker records outputs. register accepts outputs:: files a step produces and a later run trusts. mark_fresh records their hashes in the target's cache entry. stale? reads them back from that entry, not this run's registrations, so a step registers outputs only after producing them. A missing or altered output makes the target stale, and the step reruns. An entry without outputs is unchanged, so existing caches need no schema bump.

Only a regular file counts as present for a target, a dependency or an output. A directory named by a damaged .d file reads as missing, so the object recompiles and gcc writes a fresh .d.

Call sites:

  • Test execution records the result file it wrote.
  • Test file preprocessing records its build-directives cache. It now marks its target fresh only after storing the cache, so a failure partway leaves the target stale.
  • A review of every mark_fresh site found the same gap elsewhere. Directives-only preprocessing now records its compacted file, and Partial preprocessing its includes list and full expansion, all of which a skipped run reuses.
  • Mock and Partial implementation headers were tracked as dependencies of the file written beside them, with comments noting they weren't really antecedents. They are now outputs. Detection is the same, and the workaround comments are gone.

Changes

  • lib/ceedling/dependencies/dependency_tracker.rb: register( outputs: ), the regular-file rule, and output checks in stale?, mark_fresh and diagnose, sharing one presence-and-hash check with dependencies.
  • lib/ceedling/dependencies/dependinator.rb, dependency_cache_store.rb, README.md: the passthrough, validation of outputs when present, and documentation.
  • lib/ceedling/test_invoker/test_build_executor.rb, test_build_setup.rb, partials_manager.rb: the call sites above.
  • lib/ceedling/generators/generator.rb: its comment on cached results reflects that damage is now caught first by staleness.

Tests

  • Unit: directory dependencies, output recording and checking, compatibility with entries that have no outputs, diagnose, cache validation, the passthrough, and each call site's registration order.
  • Integration: on real files across two tracker runs, a truncated, emptied or deleted output reads stale, and a .d file naming a directory neither raises nor reads fresh.
  • System, new spec/system/build_artifact_recovery_spec.rb: damages one artifact after a passing build and expects the next build to pass. It covers a truncated .pass, an emptied .pass, a .d naming a directory, a deleted directives cache, an emptied directives cache followed by a code change, and a truncated mock header. Against next_version, five of the six fail with the original errors. The mock-header case was already handled and guards the consolidation.

An untouched second build still skips every stage. Units pass 4018/0 and integration 304/0, along with the delta-build, gcov, Partials and include-handling system specs. RuboCop is clean. The stress rig reran both original seeds with 60 random cycles each and found no build that failed to recover, across 504 builds.

Not covered

A truncated .d file that only loses entries, rather than naming a directory, under-tracks headers without failing. That is a pre-existing limitation of dependency files and is not sticky.

mkarlesky and others added 2 commits October 9, 2026 00:28
A fresh target can now vouch for outputs: files its step produces and a
later run trusts. register accepts outputs:, mark_fresh records their hashes
in the cache entry, and stale? reads them back from that entry, so a missing
or altered output makes the target stale. Entries without outputs are
unchanged and need no cache schema bump.

Only a regular file counts as present for a target, a dependency or an
output. A directory named by a damaged .d file then reads as missing rather
than being hashed, which raised on every build.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A damaged artifact that a skipped step reuses now makes that step stale
instead of failing every later build.

- Test execution records the result file it wrote. A truncated or emptied
  .pass is rerun instead of crashing the report.
- Test file preprocessing records its build directives cache, and marks its
  target fresh only once that cache is stored. A deleted or emptied cache is
  rebuilt instead of raising or silently dropping TEST_SOURCE_FILE().
- Directives-only preprocessing records its compacted file, and Partial
  preprocessing its includes list and full expansion.
- Mock and Partial implementation headers move from dependencies to outputs,
  where they belong.

build_artifact_recovery_spec damages one artifact after a passing build and
expects the next build to pass.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@mkarlesky
mkarlesky merged commit 562e731 into next_version Oct 9, 2026
21 checks passed
@mkarlesky
mkarlesky deleted the bug/damaged-artifact-recovery branch October 9, 2026 11:09
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant