[rtfeldman]

Problem and result

Unchanged dev commands were repeating compile-time demand discovery, evaluation, specialization, test execution, and historical native-pack loading. After an edit, even when most of the program was unchanged, the compiler repeatedly rebuilt checking results, specialization constraints, recursive proofs, and native dependency graphs.

This PR consolidates the incremental-compilation work into one branch. It caches completed command outcomes, makes finer-grained checking and specialization outputs reusable, replaces historical pack scanning with indexed lazy acquisition, reduces repeated in-process analysis, and integrates existing link preparation reuse with the indexed store. It also includes two opt-in specialization-cache experiments, described separately below.

In the latest measured configuration, all nine post-edit roc test --opt=dev samples were below one second. Post-edit roc build --opt=dev was approximately 0.37–0.39s for roc-pg, 1.29–1.32s for roc-pdf, and 1.39–1.99s for Observatory. The subsecond build goal remains unmet for roc-pdf and Observatory. Observatory's first edit was close enough to two seconds that this is not a robust sub-two-second guarantee either.

Measurement scope matters: the tables use a ReleaseFast compiler with both experimental caches enabled. They do not establish the performance of ordinary compiler invocations with those experiments disabled. Timings are from source commit b0f992f1c8, before merging main at 88451f17bf in merge commit e6e0d57517. Post-merge validation is listed below; the timing matrix was not repeated after that merge. These are measurements of this branch, not a controlled before/after comparison against current main.

General principles

• Persist completed, immutable outputs produced by the responsible compiler stage. Do not serialize partially solved state or infer missing facts from whatever happens to remain in memory.
• Separate semantic identities from transient store indices, source positions, and current reporting paths. Restore through explicit mappings to the current compilation's owners, types, captures, and reporting sites.
• Source text alone is insufficient for specialization reuse. Types, callable possibilities, captures, dispatch substitutions, compile-time values, target/code domain, and lowering policy participate wherever they affect the result.
• A cache hit must preserve the command's demanded behavior, diagnostics, compile-time observations, and exit status. A build must not accidentally acquire roc check semantics, and cached tests must still report failures at current source locations.
• Validate persisted data before committing it to live compiler state. Publication exposes completed generations; leases protect artifacts still in use. Invalid optional cache data is a miss; allocation failure propagates.
• Keep lookup proportional to requested work. Retaining old versions is useful for reuse across edits, but does not justify reading every historical record on every invocation.
• Developer experiments and instrumentation use base.dev_diagnostics. The new experiment switches are unavailable in ordinary release builds; this PR does not add release-readable environment variables.

The design contracts are documented in design.md. The cache formats are explicit binary formats, not JSON.

What changes

1. Completed dev builds bypass repeated compile-time work

src/compile/completed_compilation.zig, src/compile/coordinator.zig, src/compile/compile_build.zig, and src/cli/main.zig record and validate completed compilation inputs and outcomes.

The records include resolution/input identities, source and asset content dependencies, relevant directory entries, current module-path bindings, typed diagnostic documents, compile-time observations, demanded roots and their dependencies, literal checks, and target configuration. Input validation happens before loading checked IR when the complete result is eligible for reuse.

A valid completed result can bypass demand rediscovery, fresh compile-time evaluation, and the second specialization that normally follows fresh evaluation. It replays the previous build's actual observations instead of switching to a different command's root-selection behavior. Ordinary compilation still enforces the existing demand and finalization contracts when this complete result is unavailable.

The reused output is not an executable accepted merely because a file exists. Runtime objects and link inputs are reused; builds still relink. Complete executable reuse remains separate work.

2. Completed and individual test outcomes are reusable

src/cli/completed_test.zig, the CLI test pipeline, checked-artifact identities, and test report-site bindings cover both whole-command and individual-test reuse.

Completed outcomes retain pass/fail results, required observations and diagnostics, and exit behavior. Individual outcomes are keyed by their semantic checked dependencies rather than treating an old source offset as the identity of the test. Current report sites supply current locations when an unchanged failing test moves. Cached test roots are removed from the work requiring specialization, and eligible native test procedures can be reused for remaining work.

Changes to dependencies still invalidate affected outcomes. A cached failure remains a failure; success is not the only reusable result. The PR includes CLI fixtures and CI coverage for outcome reuse and reporting relocation.

3. Reuse checking results when the consumed checking contract is unchanged

checking_interface.zig, checking_dependencies.zig, checking_outcome.zig, inference_output.zig, reference_binding.zig, and compile/inference_cache.zig separate the interface consumed by an importer from the implementation identity of its dependency.

The checker publishes explicit immutable interface facts and inference outputs while their producers are live. Reuse validates those facts and explicitly rebinds current foreign references and owners. An implementation-only dependency change need not force importer inference when the checking contract it consumed is unchanged. Declaration layout and diagnostic/reporting changes are handled through the corresponding identities and mappings rather than retaining stale indices.

This extends diagnostic-bearing checking reuse while preserving the canonicalization-warning staging fix already on main (#12573). Pre-evaluation checking diagnostics retain their required snapshot data and explicit source dependencies; diagnostics from evaluation are not silently folded into that checking outcome.

4. Persist immutable specialization interfaces and reuse settled analyses

src/postcheck/monotype/interface_summary.zig encodes request-relative settled constraints, including composite type graphs and executable defaults with explicit checked provenance. It does not persist mutable solver IDs. Validation and transactional import precede mutation of the receiving store.

The Monotype, inhabitation, fixed-point, and Lambda Solved analyses retain completed proof results with the dependencies that can invalidate them. Recursive/SCC reasoning is reused only after it is settled; a provisional cyclic assumption cannot become a cached success. Updates notify the proofs whose inputs actually changed instead of clearing all retained work.

Additional reductions include reusing interned immutable Monotypes before cross-store copying, avoiding continuation frames for already-cached checked-type requests, simplifying builtin instantiation, retaining completed erased-callable traversals within their mutation epoch, and sharing immutable expansion reachability across argument scopes. Nominal ownership and structural backings remain explicit.

Dev procedure reuse uses an explicit owned base ABI rather than multiplying reusable artifacts by caller ARC-mode demand. Dev specialization rewrites are limited to the iterator-fusion work needed by that pipeline, with iterator normalization scope represented in identities. Loop-append promotion qualifies candidates before constructing expensive lifetime topology. This trades some opportunities for caller-specific optimization for reusable dev code; it is not a claim about optimized runtime throughput across arbitrary programs.

5. Native artifacts use indexed, lazy, dependency-aware acquisition

src/backend/dev/CodeExcerpt.zig, PackFile.zig, NativeProcCompiler.zig, src/cli/pack_index.zig, pack_store.zig, and src/lir/pack_program.zig define and consume immutable native excerpts and their dependencies. The ProcArtifact → CodeExcerpt terminology distinguishes reusable generated code from its arrangement into an executable image.

Specialization-key offers lead to selected dependency components and payloads. The loader no longer needs to eagerly decode every historical pack before finding requested specializations. Older artifacts remain eligible when their complete identities and current-source requirements still match. Shared cyclic dependency components are represented explicitly rather than repeatedly flattened into unrelated copies.

This builds on the native procedure compatibility domain already on main (#12578): code compatibility stays separate from platform libraries and final linking, including musl/glibc host-link differences. It also preserves the specialized native-callable implementation already on main (#12574). New reuse paths carry the pending erased-call/tail-drive summaries that cached bodies need, rather than losing those producer facts on a hit.

The latest acquisition changes further reduce overhead:

• A validated namespace advertisement avoids per-key offer-bucket queries in origins with no native offers until the snapshot is refreshed. It is an exact availability record, not a probabilistic filter.
• Retained catalog mappings allow record decoding to borrow mapped bytes while ownership is held. Payload files open relative to the retained shard directory.
• The explicit native dependency graph supplies acquisition groups. Operations are grouped by shard while preserving input/result order and each record's generation lease.
• Each ownership interval is bounded by the existing retention-transaction capacity. File reads, payload hashing, and decoding happen after ownership is released. Topological validation, source eligibility, per-record identities, and error cleanup remain required.

Grouping reduces catalog synchronization, but still opens individual payload files. It does not yet implement the indexed multi-record storage containers proposed below.

6. Indexed storage, retained leases, and incremental cleanup

src/compile/artifact_store.zig, cache_index.zig, cache_maintenance.zig, cache_cleanup.zig, and cache_manager.zig replace foreground whole-store sweeps with indexed acquisition and incremental maintenance.

The store has a compact catalog, explicit published/staging/retiring generations, recovery for interrupted catalog updates, and leases protecting readers. Retained mappings are synchronized before reads; header sequence changes invalidate retained decoded nodes. Retention observations are batched with the exact generation they refer to. Native source-eligibility facts remain usable when a graph gains appended artifacts.

Maintenance has explicit progress/cursor state and distinguishes idle-capacity work from bounded compulsory progress, so constrained machines are not permanently exempt from cleanup. An in-progress compiler or link retains the ownership needed to prevent cleanup from removing its inputs.

7. Integrate existing link reuse with indexed storage

The immutable-object and host-inspection improvements from #12579 are already on main. This PR adapts src/cli/link_artifacts.zig and CLI integration to the indexed artifact store: acquire a catalog generation lease before creating a scratch hard link, and publish newly materialized objects through reserved staging generations. The scratch hard link then pins the inode for the complete link. Linker entry also supplies an explicit maintenance safe point. The existing content identities and avoidance of repeated byte copies remain intact.

Relinking remains intentional. Observatory's remaining link time is a significant floor even when preceding compilation work hits cache.

8. Experimental completed-body replay

Opt-in: ROC_COMPLETED_BODY_CACHE=1, in a compiler built with -Ddev-diagnostics=true.

The prototype persists a completed owned worker body shard, including its required types/names, nested lexical bodies, specialization frontier requests, and compile-time observations. Admission includes source/type/evidence/substitution/method inputs, caller and lowering policy, and compile-time value dependencies.

Static references cross the cache boundary through checked producer/value identity plus constructor/capture paths, not a previous process's ConstNodeId. Full dispatch substitutions and their ownership are retained. Decoding validates before committing type-store transactions; owners and IDs are explicitly rebound. Deferred context-free procedure values are not incorrectly treated as lexical dependencies.

The normal coordinator still resolves requests and accepts restored shards. Replay does not change demanded compile-time behavior. This is completed-body reuse, not yet a dependency-closed, per-calling-module lambda-set cache.

9. Experimental completed lambda analysis

Opt-in: ROC_LAMBDA_ANALYSIS_CACHE=1, also requiring -Ddev-diagnostics=true.

This prototype identifies the exact lifted input consumed by lambda analysis, canonicalizes its reachable type graph, and stores only completed reachable annotations. It excludes abandoned scratch rows, redirects, and lazy solver state. Named-occurrence equality and nominal ownership remain part of validation.

A restored analysis uses the current executable IR, literal values, and reporting metadata. Checked capture provenance that the solver merely carries through is obtained from the corresponding current lifted local. Local structure, symbols, types, callable topology, and ABI inputs still have to match. This removes irrelevant provenance invalidation without pretending a changed solver input is unchanged.

A real hit is inexpensive, but whole-program identity is too coarse to cover every first edit. Native object reuse changes the assembled solver input while the cache settles. In the traced Observatory sequence, function counts changed from 4,366 on cold to 2,701 on the first edit and 2,603 thereafter; the first matching analysis appeared on the third edit. Those are different inputs and must not share a result by weakening validation.

Both experiments remain disabled by default because their cold cost and reuse boundaries need further work.

End-to-end measurements

Machine: AMD Ryzen 9 3900X, 12 cores / 24 threads, x86_64 Linux. Compiler: ReleaseFast, -Ddev-diagnostics=true -Dstrip=true. Build and test use --opt=dev with specialization enabled. roc check has no optimization flag. Both experiment switches above are enabled.

Each project/command starts with an empty Roc artifact cache, followed by three unchanged runs and three consecutive edits. Dependencies and platform artifacts are already available; “cold” does not mean a flushed OS page cache. The edits change a small source literal or expect, so these measurements do not guarantee the same latency for arbitrary structural/type edits.

Build entry points are roc-pg/examples/minimal/main.roc, roc-pdf/examples/quarterly-report/main.roc, and roc-gui/examples/observatory/main.roc. Build edits change their database URL/report title/window title respectively. Package tests use package/main.roc; Observatory tests use its application entry. Test edits change one expect in Numeric.roc, Pdf.roc, or Observatory's History.roc.

The table contains 63 serial plain-command samples. Instrumented profiles are separate. Other compiler process trees were paused during measurement. Peak RSS is from GNU time -v. No Boxy measurements are included.

Wall time, seconds

Project	Command	Cold	Warm range, 3 runs	Edit 1	Edit 2	Edit 3
roc-pg	build	20.961	0.041–0.042	0.389	0.370	0.372
roc-pg	test	68.105	0.098–0.100	0.450	0.430	0.460
roc-pg	check	33.945	0.124–0.135	0.185	0.176	0.180
roc-pdf	build	38.702	0.077–0.082	1.321	1.293	1.311
roc-pdf	test	47.047	0.145–0.147	0.767	0.769	0.783
roc-pdf	check	32.142	0.267–0.291	1.956	0.751	0.757
Observatory	build	19.590	0.270–0.275	1.986	1.838	1.387
Observatory	test	11.819	0.034–0.037	0.925	0.910	0.894
Observatory	check	12.245	0.137–0.139	1.247	0.357	0.390

All 27 plain post-edit samples are below two seconds, but the separate instrumented first Observatory edit was 2.031s. All post-edit test samples meet the one-second target. Warm Observatory builds still miss the original 250ms target slightly. The third Observatory edit is distinguished because it is the first whole-analysis hit in this sequence.

Peak RSS, MiB

Warm/edit columns are the maximum of their three plain samples, not a sum across runs.

Project	Command	Cold	Maximum warm	Maximum edit
roc-pg	build	2,192.0	92.0	439.0
roc-pg	test	10,945.0	117.3	400.6
roc-pg	check	4,581.1	353.8	382.8
roc-pdf	build	3,895.5	135.6	1,125.2
roc-pdf	test	3,287.1	222.7	700.0
roc-pdf	check	3,813.6	639.9	1,188.6
Observatory	build	1,537.8	338.2	1,245.2
Observatory	test	658.0	78.4	496.7
Observatory	check	1,550.3	602.1	1,028.6

Cold build peaks are 2.14 GiB, 3.80 GiB, and 1.50 GiB respectively. The most concerning cold-memory case is roc-pg test, at 10.69 GiB.

Controlled comparisons isolating the experiments

These comparisons each use the same compiler on both sides, but were performed at different stages of the branch. Their savings must not be added together or interpreted as a comparison against main.

Experiment	Disabled samples, seconds	Enabled samples, seconds	Median change
Completed-body replay, analysis cache off, settled Observatory edit	2.242, 2.184, 2.218	1.853, 1.878, 1.893	2.218 → 1.878: 340ms / 15.3%
Whole lambda analysis, body replay on, settled Observatory edit	1.829, 1.875, 1.797	1.398, 1.397, 1.405	1.829 → 1.398: 431ms / 23.6%

The analysis comparison alternated disabled/enabled runs and verified actual hits with counters on both sides. Enabled peak RSS was about 95 MiB lower. One steady Observatory analysis record was about 3.38 MB and restored in roughly 17–23ms.

The isolated grouped-acquisition probe reduced 8,000 steady acquisitions from approximately 96–97ms to 72ms and catalog-header reads from 8,000 to 341. This is a microbenchmark, not an end-to-end saving claim.

Cold-build and disk tradeoffs

The experiments spend time and space recording results that subsequent builds can consume. Their cost is not negligible, and the default remains off.

A controlled same-compiler cold roc-pg test comparison isolated completed-body caching:

Completed-body cache	Cold time	Peak RSS
Disabled	58.825s	9,875.0 MiB
Enabled	64.583s	11,127.4 MiB

That is 5.76s more time and 1.22 GiB more peak RSS, with 320,437,765 bytes of extra body records (about 305.6 MiB). This is not all disk-write time: the representation assembled for the second lambda solve also changed, with solved types increasing from 23,250,385 to 24,585,052 and expansions from 9,331,521 to 10,175,911. The exact attribution between encoding, retained graphs, and additional solving has not been isolated.

The final roc-pg cold-test measurement is 68.105s. It uses a later compiler and a different analysis-cache setting, so its difference from the earlier comparison is not an isolated regression attributable to one change. There is no controlled current-main baseline establishing the net cold cost of this entire PR.

Retained artifact payload size after the measurement sequence

The inventory below was captured after cold + three warm + three edits + an additional instrumented edit, before cleanup. It includes retained versions and is not cold-only write volume or incremental overhead versus main.

It counts artifact *.data files. “Logical” is file length; “allocated” is filesystem blocks for those payloads. Catalogs, leases, scratch objects, output executables, package downloads, and platform libraries are excluded. Body/analysis columns are subsets of all payloads.

Project	Command	Body payload MiB	Analysis payload MiB	All payloads, logical MiB	All payloads, allocated MiB
roc-pg	build	65.55	4.06	492.06	568.84
roc-pg	test	306.02	17.87	1,506.04	1,650.05
roc-pg	check	102.54	5.31	674.35	773.77
roc-pdf	build	230.32	15.65	1,388.03	1,599.85
roc-pdf	test	380.39	29.52	1,516.61	1,797.74
roc-pdf	check	183.20	12.73	1,199.99	1,379.39
Observatory	build	77.18	21.31	805.50	881.92
Observatory	test	42.50	13.81	1,518.36	1,579.03
Observatory	check	50.00	12.76	708.94	745.83

Coarse completed bodies are relatively cheap to consume but duplicate reachable information across records. Whole-analysis records are compact enough to restore quickly, but can accumulate when the assembled program changes. Retention policy and cleanup bound stored history; they do not remove the need to reduce duplicate cold work and representation size.

Remaining work, in recommended order

1. Reduce native artifact acquisition costs. The final additional edit profile spends 477ms in roc-pdf native-provider lookup and 287ms in Observatory. roc-pdf includes about 317ms in dependency-graph acquisition/assembly and 119ms in payload acquisition. Put related immutable records into indexed storage containers, retain a container lease, and read only requested ranges while preserving per-record hashes, graph identities, and source eligibility. Grouped acquisition removes synchronization overhead but leaves many individual files. Codec optimization alone is unlikely to solve this: decoding all 5,235 historical metadata records (5.91 MB) in memory took 17.15ms initially and 2.30–3.72ms subsequently, excluding acquisition and graph work.
2. Cache dependency-closed caller regions of lambda analysis. Reuse unaffected regions on the first edit even when native reuse changes surrounding program topology. Persist actual solved boundary types, callable sets, captures, and dependencies, with explicit current-ID rebinding. Do not treat a larger whole-program result as an interchangeable superset. The measured 431ms full-hit saving establishes an opportunity, not a guarantee for a regional design.
3. Reuse unchanged procedures after their complete callable ABI is known. Observatory still spends about 172ms in LIR generation, 82ms in ARC, and 103ms in the native backend; roc-pdf spends about 54ms, 26ms, and 139ms respectively. A later reuse point with complete specialization/capture/ABI and dependency identities could skip unchanged work. These whole-stage totals are upper bounds, not promised additive savings.
4. Reduce cold body-cache cost before enabling the experiments by default. Investigate duplicate type/graph construction, retained body data, and the changed second-solve representation. Measure cold time, peak RSS, and write volume independently. No arbitrary specialization time/record-size budget was added to conceal this cost.
5. Address the remaining link floor. Observatory linking is about 215ms, versus about 39ms for roc-pdf. Existing link preparation reuse helps edited builds, but validated final-executable reuse would be needed to eliminate that warm floor. It must cover the complete host/library/link-option identity; executable presence alone proves nothing.

For context, the final extra Observatory edit profile has approximately 122ms checking, 445ms Monotype lowering (including the 287ms native query), 18ms lambda analysis, 172ms LIR generation, 82ms ARC, 103ms backend, and 215ms linking. roc-pdf has approximately 187ms checking, 574ms Monotype lowering (including 477ms native query), 3ms lambda analysis, 54ms LIR generation, 26ms ARC, 139ms backend, and 39ms linking. Different instrumentation scopes are not additive: provider-lifetime counters can include consumers outside coordinator lookup, and worker wait includes the workers' computations rather than demonstrating idle time.

Validation

Before the main merge, on the measured source:

• ReleaseFast compiler semantic and native builds passed.
• Focused completed-body/interface/analysis tests covered transactional import, corruption, allocation failures, nominal ownership, capture provenance, callback identities, and current literal/reporting metadata.
• 27 focused native-cache tests and 17 catalog/storage tests passed, covering grouped ordering, duplicates, missing records, generation replacement, retained leases, interrupted publication, corruption, and allocation failure.
• Runtime fixtures covered capture values, imported provider edits, static-value paths, iterator scope, and allocation counts. Builds with 1/4/24 workers produced byte-identical executables in the checked cases.
• The 63 plain measurements completed. Actual test totals were 582 for roc-pg, 610 for roc-pdf, and 125 for Observatory.
• Each project was checked for fresh failure, unchanged cached failure, moved failure reporting, and restored success. Benchmark source edits were restored and verified.

After merging main at 88451f17bf:

• The merge completed without conflicts and is pushed as e6e0d57517.
• A ReleaseFast semantic compile passed using freshly generated build inputs for the merged source. A full native compiler build and the timing matrix were not repeated after the merge.
• All 27 focused native-cache tests passed.
• All 31 focused analysis/interface/frozen-type/body-relocation tests passed in ReleaseFast.
• git diff --check origin/main passed; the working tree is clean.

MiniCI and aggregate eval suites were not run locally. No Debug Roc compiler or Boxy measurements contributed to the results. Investigation reports and machine-local reproduction scripts remain outside the repository.

For reproducibility, the measured compiler SHA256 is f7e8c52866befbf079d98c4f27452d9cafa98200f68577e7513cd56b73f72d96. Its version string retains release-fast-60830bd9 because the frozen build snapshot preceded committing the final acquisition change. A SHA256 audit verified all 1,125 tracked compiler/build/design files in that snapshot against b0f992f1c8. Raw timings, GNU-time resources, profiles, commands, and artifact inventories are retained under /home/rtfeldman/code/roc-dev-investigation/2026-10-27/rebuild-followup/; publication notes are in the adjacent pr-publication/ directory.