Remap def indices for deterministic metadata encoding by aerooneqq · Pull Request #162809 · rust-lang/rust

aerooneqq · 2026-09-15T12:17:30Z

View all comments

Now at the very early stage (however fixes corresponding issues), draft primarily for CI.

cc #t-compiler/const-eval > DefId reproducibility with the parallel frontend

r? @petrochenkov

susitsm · 2026-09-15T13:50:42Z

Are the changes in decoder.rs and def_path_hash_map.rs needed? It seems unrelated to me.

aerooneqq · 2026-09-15T14:01:33Z

Are the changes in decoder.rs and def_path_hash_map.rs needed? It seems unrelated to me.

We need to update map from hashes to def indices while encoding DefPathHashMap, now I inserted simple FxHashMap to test the approach, if I don't remove functions from decoder.rs there will be warnings, because they are not used anywhere else.

petrochenkov · 2026-09-16T12:21:55Z

@bors try @rust-timer queue

Remap def indices for deterministic metadata encoding

rust-bors · 2026-09-16T13:29:18Z

☀️ Try build successful (CI)
Build commit: ceb9839 (ceb98398abf30a57ca7f186c152d2adda7b079f9)
Base parent: 28e8a8c (28e8a8c81bf3b37909edac6c2a76e56f30cd492f)

rust-timer · 2026-09-16T14:13:41Z

Finished benchmarking commit (ceb9839): comparison URL.

Overall result: ❌ regressions - please read:

Benchmarking means the PR may be perf-sensitive. It's automatically marked not fit for rolling up. Overriding is possible but disadvised: it risks changing compiler perf.

Next, please: If you can, justify the regressions found in this try perf run in writing along with @rustbot label: +perf-regression-triaged. If not, fix the regressions and do another perf run. Neutral or positive results will clear the label automatically.

@bors rollup=never rustc-perf
@rustbot label: -S-waiting-on-perf +perf-regression

Instruction count

Our most reliable metric. Used to determine the overall result above. However, even this metric can be noisy.

	mean	range	count
Regressions ❌ (primary)	4.4%	[0.1%, 164.6%]	276
Regressions ❌ (secondary)	11.8%	[0.2%, 130.1%]	350
Improvements ✅ (primary)	-	-	0
Improvements ✅ (secondary)	-	-	0
All ❌✅ (primary)	4.4%	[0.1%, 164.6%]	276

Max RSS (memory usage)

Results (primary 3.0%, secondary 4.5%)

A less reliable metric. May be of interest, but not used to determine the overall result above.

	mean	range	count
Regressions ❌ (primary)	3.0%	[0.8%, 9.4%]	203
Regressions ❌ (secondary)	4.6%	[0.9%, 14.8%]	327
Improvements ✅ (primary)	-2.3%	[-2.3%, -2.3%]	1
Improvements ✅ (secondary)	-3.6%	[-4.0%, -2.9%]	3
All ❌✅ (primary)	3.0%	[-2.3%, 9.4%]	204

Cycles

Results (primary 10.2%, secondary 14.6%)

A less reliable metric. May be of interest, but not used to determine the overall result above.

	mean	range	count
Regressions ❌ (primary)	10.2%	[1.7%, 73.0%]	55
Regressions ❌ (secondary)	14.6%	[1.4%, 66.2%]	179
Improvements ✅ (primary)	-	-	0
Improvements ✅ (secondary)	-	-	0
All ❌✅ (primary)	10.2%	[1.7%, 73.0%]	55

Binary size

Results (primary -0.4%, secondary -0.4%)

A less reliable metric. May be of interest, but not used to determine the overall result above.

	mean	range	count
Regressions ❌ (primary)	-	-	0
Regressions ❌ (secondary)	-	-	0
Improvements ✅ (primary)	-0.4%	[-0.4%, -0.3%]	8
Improvements ✅ (secondary)	-0.4%	[-0.4%, -0.4%]	4
All ❌✅ (primary)	-0.4%	[-0.4%, -0.3%]	8

Bootstrap: 496.137s -> 496.27s (0.03%)
Artifact size: 406.79 MiB -> 406.77 MiB (-0.01%)

petrochenkov · 2026-09-16T14:16:05Z

@bors try @rust-timer queue

Remap def indices for deterministic metadata encoding

susitsm · 2026-09-16T14:34:22Z

+
+    #[inline]
+    fn map_index(&self, index: DefIndex) -> DefIndex {
+        self.def_indexes_remapping.get(&index).copied().unwrap_or(index)


This could be donewith a vec for better performance. Since we know that these indices are dense in a range. Either a full IndexVec mapping every LocalDefId, or a shorter Vec, that only contains the remapped ones, indexed by remapped_index - min_remapped_index

View changes since the review

It is very strong assumption that def ids that are needed to be remapped will be in the end, now it should be true, because resolution and AST -> HIR lowering is executed without any parallelism, however AST -> HIR lowering became more incremental recently by introducing lower_to_hir query which is very easy to parallelize in the beginning of the run_required_analysis function (now it is executed sequentially as we prefetch hir_crate_items which is a sequential HIR visit). If it is parallelized then from each lower_to_hir execution we may invoke arbitrary code that can allocate def ids that do not need to be remapped.

But I like the idea, I was thinking about something similar in a bit different direction: create separate IndexVecs in Definitions struct for different stages of compilation, thus we can maybe pre-allocate and remap ids only for selected parts.

I think that it is reasonable to track the last def id that is allocated during resolution stage, because it is not likely to get parallelized in the nearest time (I hope so) and then to remap all def ids that are allocated after it, I will try it tomorrow.

susitsm · 2026-09-16T14:45:54Z

+    let mut to_remap = vec![];
+    let mut def_ids = vec![];
+    for idx in 0..defs.num_definitions() {
+        let def_id = LocalDefId { local_def_index: idx.into() };


I think this could iterate backwards, stopping when it hits an index not needing remapping. Remapped indices should be at the end

View changes since the review

rust-bors · 2026-09-16T15:16:50Z

☀️ Try build successful (CI)
Build commit: 8bcd37a (8bcd37a7bc50acb44dc7181f601e93127ebf19af)
Base parent: 28e8a8c (28e8a8c81bf3b37909edac6c2a76e56f30cd492f)

rust-timer · 2026-09-16T16:35:40Z

Finished benchmarking commit (8bcd37a): comparison URL.

Overall result: ❌ regressions - BENCHMARK(S) FAILED

Benchmarking means the PR may be perf-sensitive. It's automatically marked not fit for rolling up. Overriding is possible but disadvised: it risks changing compiler perf.

Next, please: If you can, justify the regressions found in this try perf run in writing along with @rustbot label: +perf-regression-triaged. If not, fix the regressions and do another perf run. Neutral or positive results will clear the label automatically.

@bors rollup=never rustc-perf
@rustbot label: -S-waiting-on-perf +perf-regression

❗ ❗ ❗ ❗ ❗
Warning ⚠️: The following benchmark(s) failed to build:

Job failure
cranelift-codegen-0.119.0

❗ ❗ ❗ ❗ ❗

Instruction count

Our most reliable metric. Used to determine the overall result above. However, even this metric can be noisy.

	mean	range	count
Regressions ❌ (primary)	0.3%	[0.1%, 0.6%]	60
Regressions ❌ (secondary)	0.2%	[0.1%, 0.3%]	13
Improvements ✅ (primary)	-	-	0
Improvements ✅ (secondary)	-	-	0
All ❌✅ (primary)	0.3%	[0.1%, 0.6%]	60

Max RSS (memory usage)

Results (secondary -4.1%)

A less reliable metric. May be of interest, but not used to determine the overall result above.

	mean	range	count
Regressions ❌ (primary)	-	-	0
Regressions ❌ (secondary)	-	-	0
Improvements ✅ (primary)	-	-	0
Improvements ✅ (secondary)	-4.1%	[-5.1%, -3.3%]	3
All ❌✅ (primary)	-	-	0

Cycles

Results (secondary -0.2%)

A less reliable metric. May be of interest, but not used to determine the overall result above.

	mean	range	count
Regressions ❌ (primary)	-	-	0
Regressions ❌ (secondary)	4.6%	[4.6%, 4.6%]	1
Improvements ✅ (primary)	-	-	0
Improvements ✅ (secondary)	-2.5%	[-2.7%, -2.3%]	2
All ❌✅ (primary)	-	-	0

Binary size

This perf run didn't have relevant results for this metric.

Bootstrap: 496.137s -> 495.635s (-0.10%)
Artifact size: 406.79 MiB -> 406.74 MiB (-0.01%)

rust-timer · 2026-09-17T14:22:41Z

Finished benchmarking commit (501ad34): comparison URL.

Overall result: ✅ improvements - no action needed

Benchmarking means the PR may be perf-sensitive. It's automatically marked not fit for rolling up. Overriding is possible but disadvised: it risks changing compiler perf.

@bors rollup=never rustc-perf
@rustbot label: -S-waiting-on-perf -perf-regression

Instruction count

Our most reliable metric. Used to determine the overall result above. However, even this metric can be noisy.

	mean	range	count
Regressions ❌ (primary)	-	-	0
Regressions ❌ (secondary)	0.2%	[0.2%, 0.2%]	1
Improvements ✅ (primary)	-	-	0
Improvements ✅ (secondary)	-0.3%	[-0.4%, -0.1%]	8
All ❌✅ (primary)	-	-	0

Max RSS (memory usage)

Results (primary -2.3%, secondary 0.2%)

A less reliable metric. May be of interest, but not used to determine the overall result above.

	mean	range	count
Regressions ❌ (primary)	-	-	0
Regressions ❌ (secondary)	3.4%	[3.1%, 3.7%]	2
Improvements ✅ (primary)	-2.3%	[-2.3%, -2.3%]	1
Improvements ✅ (secondary)	-2.0%	[-3.3%, -0.4%]	3
All ❌✅ (primary)	-2.3%	[-2.3%, -2.3%]	1

Cycles

Results (primary 4.4%, secondary 4.3%)

A less reliable metric. May be of interest, but not used to determine the overall result above.

	mean	range	count
Regressions ❌ (primary)	4.4%	[2.4%, 6.4%]	2
Regressions ❌ (secondary)	4.3%	[4.3%, 4.3%]	1
Improvements ✅ (primary)	-	-	0
Improvements ✅ (secondary)	-	-	0
All ❌✅ (primary)	4.4%	[2.4%, 6.4%]	2

Binary size

This perf run didn't have relevant results for this metric.

Bootstrap: 500.132s -> 498.753s (-0.28%)
Artifact size: 406.81 MiB -> 408.93 MiB (0.52%)

@petrochenkov

…coding, r=petrochenkov Deterministic encoding of `DefPathHashMap` This PR splits DefPathHashMap into two parts: deterministic (`det_part`) and non-deterministic (`non_det_part`). `det_part` is used while we sure that allocation and insertion order of def ids is deterministic, at the moment of writing it happens before we start parallel checks after prefetch of `hir_crate_items` in `run_required_analysis`. Up until this point of compilation the allocation of def ids and their insertion order into `det_part` should be the same between different compilations. Next, when non-determinism starts due to parallel compilation we put all mapping between local hash and def ids into a `SortedMap` which gives us ready-to-use sorted by local hash slice of pairs to encode while encoding metadata. Note, that this PR does not solve the problem of allocation of different def indices to same code entities (meaning same local hash), this problem is solved in rust-lang#162809 by remapping needed local def indices. Also note that we serialize `det_part` as a raw bytes sequence, so during remapping if we do not place all def indices that are needed to be remapped into separate container we will end up with copying `det_part`, changing some entries in it and then serialize it as a bytes sequence. With the approach in this PR we do not copy and modify `det_part`, instead we do all modifications in `non_det_part`. r? @petrochenkov

@petrochenkov

Rollup merge of #162910 - aerooneqq:det-def-path-hash-map-encoding, r=petrochenkov Deterministic encoding of `DefPathHashMap` This PR splits DefPathHashMap into two parts: deterministic (`det_part`) and non-deterministic (`non_det_part`). `det_part` is used while we sure that allocation and insertion order of def ids is deterministic, at the moment of writing it happens before we start parallel checks after prefetch of `hir_crate_items` in `run_required_analysis`. Up until this point of compilation the allocation of def ids and their insertion order into `det_part` should be the same between different compilations. Next, when non-determinism starts due to parallel compilation we put all mapping between local hash and def ids into a `SortedMap` which gives us ready-to-use sorted by local hash slice of pairs to encode while encoding metadata. Note, that this PR does not solve the problem of allocation of different def indices to same code entities (meaning same local hash), this problem is solved in #162809 by remapping needed local def indices. Also note that we serialize `det_part` as a raw bytes sequence, so during remapping if we do not place all def indices that are needed to be remapped into separate container we will end up with copying `det_part`, changing some entries in it and then serialize it as a bytes sequence. With the approach in this PR we do not copy and modify `det_part`, instead we do all modifications in `non_det_part`. r? @petrochenkov

@petrochenkov

…=petrochenkov Deterministic encoding of `DefPathHashMap` This PR splits DefPathHashMap into two parts: deterministic (`det_part`) and non-deterministic (`non_det_part`). `det_part` is used while we sure that allocation and insertion order of def ids is deterministic, at the moment of writing it happens before we start parallel checks after prefetch of `hir_crate_items` in `run_required_analysis`. Up until this point of compilation the allocation of def ids and their insertion order into `det_part` should be the same between different compilations. Next, when non-determinism starts due to parallel compilation we put all mapping between local hash and def ids into a `SortedMap` which gives us ready-to-use sorted by local hash slice of pairs to encode while encoding metadata. Note, that this PR does not solve the problem of allocation of different def indices to same code entities (meaning same local hash), this problem is solved in rust-lang/rust#162809 by remapping needed local def indices. Also note that we serialize `det_part` as a raw bytes sequence, so during remapping if we do not place all def indices that are needed to be remapped into separate container we will end up with copying `det_part`, changing some entries in it and then serialize it as a bytes sequence. With the approach in this PR we do not copy and modify `det_part`, instead we do all modifications in `non_det_part`. r? @petrochenkov

@petrochenkov

…cros, r=petrochenkov Some refactorings around metadata encoding Two refactorings that were extracted from rust-lang#162809: - Use macros to record values in all tables, this unifies writing to tables and makes it much easier to modify or refactor writing process as it was needed in rust-lang#162809, - Moves `proc_macro_quoted_spans` table into `ProcMacroData` as it is filled only when `ProcMacroData` is `Some` (meaning crate type is `proc-macro`), so it is reasonable to place it here, moreover all tables left in `LazyTables` have `DefIndex` as key, this simplifies refactorings and modifications that are connected with `DefIndex`es. r? @petrochenkov

@petrochenkov

…cros, r=petrochenkov Some refactorings around metadata encoding Two refactorings that were extracted from rust-lang#162809: - Use macros to record values in all tables, this unifies writing to tables and makes it much easier to modify or refactor writing process as it was needed in rust-lang#162809, - Moves `proc_macro_quoted_spans` table into `ProcMacroData` as it is filled only when `ProcMacroData` is `Some` (meaning crate type is `proc-macro`), so it is reasonable to place it here, moreover all tables left in `LazyTables` have `DefIndex` as key, this simplifies refactorings and modifications that are connected with `DefIndex`es. r? @petrochenkov

@petrochenkov

…cros, r=petrochenkov Some refactorings around metadata encoding Two refactorings that were extracted from rust-lang#162809: - Use macros to record values in all tables, this unifies writing to tables and makes it much easier to modify or refactor writing process as it was needed in rust-lang#162809, - Moves `proc_macro_quoted_spans` table into `ProcMacroData` as it is filled only when `ProcMacroData` is `Some` (meaning crate type is `proc-macro`), so it is reasonable to place it here, moreover all tables left in `LazyTables` have `DefIndex` as key, this simplifies refactorings and modifications that are connected with `DefIndex`es. r? @petrochenkov

oli-obk · 2026-10-01T07:12:19Z

+
+        self.last_deterministic_index
+            .set(
+                if self.def_id_to_key.is_empty() { 0 } else { self.def_id_to_key.len() - 1 }.into(),


will this ever be empty? shouldn't the first entry always be the crate root?

View changes since the review

oli-obk · 2026-10-01T07:14:12Z

 #[derive(MetadataEncodable, LazyDecodable)]
 pub(crate) struct TraitImpls {
-    trait_id: (u32, DefIndex),
+    trait_id: (u32 /* crate */, u32 /* def index */),


why was this changed?

View changes since the review

oli-obk · 2026-10-01T07:15:51Z

    proc_macro_decls_static: DefIndex,
    stability: Option<Stability>,
    macros: LazyArray<(DefIndex, LazyValue<ProcMacroKind>)>,
+    proc_macro_quoted_spans: LazyTableSingleIdx<usize, Option<LazyValue<Span>>>,


this can be its own commit, it's not really necessary, just moving it to the right place, right?

View changes since the review

oli-obk · 2026-10-01T07:17:05Z

            let def_id = local_id.to_def_id();
            let def_kind = tcx.def_kind(local_id);
-            self.tables.def_kind.set_some(def_id.index, def_kind);
+            record_non_lazy!(self.tables.def_kind[def_id] <- def_kind);


lots of unrelated changes, move them to a separate commit

View changes since the review

Yes, it is a draft now because it is waiting on other PRs like #163321, #162900 that contain not related refactorings, after they are all merged I will mark this PR as ready to review and will also write proper description, sorry, should have explicitly stated that it is blocked now.

oli-obk · 2026-10-01T07:21:12Z

+            && matches!(
+                data.data,
+                DefPathData::SyntheticCoroutineBody
+                    | DefPathData::OpaqueLifetime(..)
+                    | DefPathData::NestedStatic
+                    | DefPathData::AnonAssocTy(..)
+            )


This seems fragile. How do we ensure that we won't end up with new nondeterminism in the future?

View changes since the review

which DefPathDatas aren't on this list but are after the last deterministic index?

rustbot · 2026-10-01T07:25:43Z

Reminder, once the PR becomes ready for a review, use @rustbot ready.

aerooneqq · 2026-10-01T07:30:48Z

Blocked by #163321.

@petrochenkov

…cros, r=petrochenkov Some refactorings around metadata encoding Two refactorings that were extracted from rust-lang#162809: - Use macros to record values in all tables, this unifies writing to tables and makes it much easier to modify or refactor writing process as it was needed in rust-lang#162809, - Moves `proc_macro_quoted_spans` table into `ProcMacroData` as it is filled only when `ProcMacroData` is `Some` (meaning crate type is `proc-macro`), so it is reasonable to place it here, moreover all tables left in `LazyTables` have `DefIndex` as key, this simplifies refactorings and modifications that are connected with `DefIndex`es. r? @petrochenkov

@petrochenkov

…cros, r=petrochenkov Some refactorings around metadata encoding Two refactorings that were extracted from rust-lang#162809: - Use macros to record values in all tables, this unifies writing to tables and makes it much easier to modify or refactor writing process as it was needed in rust-lang#162809, - Moves `proc_macro_quoted_spans` table into `ProcMacroData` as it is filled only when `ProcMacroData` is `Some` (meaning crate type is `proc-macro`), so it is reasonable to place it here, moreover all tables left in `LazyTables` have `DefIndex` as key, this simplifies refactorings and modifications that are connected with `DefIndex`es. r? @petrochenkov

@petrochenkov

…cros, r=petrochenkov Some refactorings around metadata encoding Two refactorings that were extracted from rust-lang#162809: - Use macros to record values in all tables, this unifies writing to tables and makes it much easier to modify or refactor writing process as it was needed in rust-lang#162809, - Moves `proc_macro_quoted_spans` table into `ProcMacroData` as it is filled only when `ProcMacroData` is `Some` (meaning crate type is `proc-macro`), so it is reasonable to place it here, moreover all tables left in `LazyTables` have `DefIndex` as key, this simplifies refactorings and modifications that are connected with `DefIndex`es. r? @petrochenkov

@petrochenkov

…cros, r=petrochenkov Some refactorings around metadata encoding Two refactorings that were extracted from rust-lang#162809: - Use macros to record values in all tables, this unifies writing to tables and makes it much easier to modify or refactor writing process as it was needed in rust-lang#162809, - Moves `proc_macro_quoted_spans` table into `ProcMacroData` as it is filled only when `ProcMacroData` is `Some` (meaning crate type is `proc-macro`), so it is reasonable to place it here, moreover all tables left in `LazyTables` have `DefIndex` as key, this simplifies refactorings and modifications that are connected with `DefIndex`es. r? @petrochenkov

@petrochenkov

…cros, r=petrochenkov Some refactorings around metadata encoding Two refactorings that were extracted from rust-lang#162809: - Use macros to record values in all tables, this unifies writing to tables and makes it much easier to modify or refactor writing process as it was needed in rust-lang#162809, - Moves `proc_macro_quoted_spans` table into `ProcMacroData` as it is filled only when `ProcMacroData` is `Some` (meaning crate type is `proc-macro`), so it is reasonable to place it here, moreover all tables left in `LazyTables` have `DefIndex` as key, this simplifies refactorings and modifications that are connected with `DefIndex`es. r? @petrochenkov

@petrochenkov

Rollup merge of #162900 - aerooneqq:record-tables-through-macros, r=petrochenkov Some refactorings around metadata encoding Two refactorings that were extracted from #162809: - Use macros to record values in all tables, this unifies writing to tables and makes it much easier to modify or refactor writing process as it was needed in #162809, - Moves `proc_macro_quoted_spans` table into `ProcMacroData` as it is filled only when `ProcMacroData` is `Some` (meaning crate type is `proc-macro`), so it is reasonable to place it here, moreover all tables left in `LazyTables` have `DefIndex` as key, this simplifies refactorings and modifications that are connected with `DefIndex`es. r? @petrochenkov

rust-bors · 2026-10-02T05:02:00Z

☔ The latest upstream changes (presumably #163609) made this pull request unmergeable. Please resolve the merge conflicts by rebasing.

rustbot assigned petrochenkov Sep 15, 2026

rustbot added S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. T-compiler Relevant to the compiler team, which will review and decide on the PR/issue. labels Sep 15, 2026

aerooneqq changed the title ~~Remap def indices for deterministic metadata encoing~~ Remap def indices for deterministic metadata encoding Sep 15, 2026