Repository navigation
Recover from damaged build artifacts instead of failing every build - #1329
Merged
Merged
Conversation
A fresh target can now vouch for outputs: files its step produces and a later run trusts. register accepts outputs:, mark_fresh records their hashes in the cache entry, and stale? reads them back from that entry, so a missing or altered output makes the target stale. Entries without outputs are unchanged and need no cache schema bump. Only a regular file counts as present for a target, a dependency or an output. A directory named by a damaged .d file then reads as missing rather than being hashed, which raised on every build. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A damaged artifact that a skipped step reuses now makes that step stale instead of failing every later build. - Test execution records the result file it wrote. A truncated or emptied .pass is rerun instead of crashing the report. - Test file preprocessing records its build directives cache, and marks its target fresh only once that cache is stored. A deleted or emptied cache is rebuilt instead of raising or silently dropping TEST_SOURCE_FILE(). - Directives-only preprocessing records its compacted file, and Partial preprocessing its includes list and full expansion. - Mock and Partial implementation headers move from dependencies to outputs, where they belong. build_artifact_recovery_spec damages one artifact after a passing build and expects the next build to pass. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
A single damaged file under
build/could leave every later build failing until a clobber. Fault-injection stress testing found three such cases. This PR makes each one cost a rebuild of the affected step instead.Failures
build/test/results/<test>.pass) was reused as-is. The summary report then raisedundefined method `size' for nilorundefined method `empty?' for nil..dfile can end in a directory path. The directory was tracked as a dependency, and hashing it raisedIs a directory @ io_freadbefore the recompile that would have rewritten the file.build/test/preprocess/build_directives/<test>/) raisedCould not find YAML file …_source_files.yml. An emptied one was worse: it loaded as no directives at all, silently droppingTEST_SOURCE_FILE()sources. That surfaced only asundefined referenceat the next relink.Cause
Two of the three share a cause: a freshness record vouched for an artifact it never checked. Test execution was skipped when
<test>.fixture_runwas fresh, then reused the result file. Directive parsing was skipped when the preprocessed test file was fresh, then loaded the cache. Neither artifact was tracked, so damage to it was trusted indefinitely.The
.dcase was the tracker counting any existing path as a dependency, directories included.Fix
The dependency tracker records outputs.
registeracceptsoutputs:: files a step produces and a later run trusts.mark_freshrecords their hashes in the target's cache entry.stale?reads them back from that entry, not this run's registrations, so a step registers outputs only after producing them. A missing or altered output makes the target stale, and the step reruns. An entry without outputs is unchanged, so existing caches need no schema bump.Only a regular file counts as present for a target, a dependency or an output. A directory named by a damaged
.dfile reads as missing, so the object recompiles and gcc writes a fresh.d.Call sites:
mark_freshsite found the same gap elsewhere. Directives-only preprocessing now records its compacted file, and Partial preprocessing its includes list and full expansion, all of which a skipped run reuses.Changes
lib/ceedling/dependencies/dependency_tracker.rb:register( outputs: ), the regular-file rule, and output checks instale?,mark_freshanddiagnose, sharing one presence-and-hash check with dependencies.lib/ceedling/dependencies/dependinator.rb,dependency_cache_store.rb,README.md: the passthrough, validation ofoutputswhen present, and documentation.lib/ceedling/test_invoker/test_build_executor.rb,test_build_setup.rb,partials_manager.rb: the call sites above.lib/ceedling/generators/generator.rb: its comment on cached results reflects that damage is now caught first by staleness.Tests
diagnose, cache validation, the passthrough, and each call site's registration order..dfile naming a directory neither raises nor reads fresh.spec/system/build_artifact_recovery_spec.rb: damages one artifact after a passing build and expects the next build to pass. It covers a truncated.pass, an emptied.pass, a.dnaming a directory, a deleted directives cache, an emptied directives cache followed by a code change, and a truncated mock header. Againstnext_version, five of the six fail with the original errors. The mock-header case was already handled and guards the consolidation.An untouched second build still skips every stage. Units pass 4018/0 and integration 304/0, along with the delta-build, gcov, Partials and include-handling system specs. RuboCop is clean. The stress rig reran both original seeds with 60 random cycles each and found no build that failed to recover, across 504 builds.
Not covered
A truncated
.dfile that only loses entries, rather than naming a directory, under-tracks headers without failing. That is a pre-existing limitation of dependency files and is not sticky.