Core: Clean up encryption keys as part of RemoveSnapshots by Hugo-WB · Pull Request #16353 · apache/iceberg

Hugo-WB · 2026-05-15T15:40:29Z

Fixes the first part of: #16352 for ExpireSnapshots
By calling removeEncryptionKey when we remove snapshot from metadata.

This assumes that snapshot <> encryption key is 1:1. I think this is a valid assumption given a new encryption key is created per manifest list here? But keen to get thoughts from others on this invariant, I am relatively unfamiliar with all this. Do people know what the intention here was? Are there cases where key re-use could happen?

Hugo-WB · 2026-05-18T12:21:53Z

@ggershinsky curious to hear your thoughts on above. Specifically the questions I left in the PR description. Thanks!

ggershinsky · 2026-05-18T13:52:48Z

sgtm. manifest list file keys are generated per snapshot, and are not re-used. if a snapshot is removed, no reason to keep its key.

Hugo-WB · 2026-05-18T18:44:26Z

sounds good. @huaxingao could I get a review on the above please!

Hugo-WB · 2026-05-21T14:51:54Z

friendly ping on above! Sorry still new 😅 @ggershinsky would appreciate a review and if you could tag in someone else who could give this a review and is familiar with the encryption space!

ggershinsky · 2026-05-21T15:20:26Z

sure, maybe @huaxingao and @singhpk234 can help

Hugo-WB · 2026-05-26T11:08:22Z

Hey @huaxingao / @singhpk234 friendly ping on above 😄

singhpk234

The change makes sense to me ! can we add an E2E where post removing the expired snapshot (MLK) one is still able to read the table ?

- While adding E2E test found a bug in the HiveTableOperations. - HiveTableOperations, as well as the currently open PR for the RestTableOperations, adds all keys from EncryptionManager into TableMetadata

huaxingao · 2026-06-16T17:07:55Z

@Hugo-WB Could you resolve the conflict?

Hugo-WB

Thanks for the review @huaxingao!

Merged upstream/main and updated a comment.

I still have some concerns about the correctness here long term. Left a comment, would be keen to get the answer to it before merging. Thanks!

Hugo-WB · 2026-06-17T16:59:41Z

+    Set<String> referencedEncryptionKeys =
+        Sets.union(
+            metadata.snapshots().stream()
+                .map(Snapshot::keyId)
+                .filter(Objects::nonNull)
+                .collect(Collectors.toUnmodifiableSet()),
+            metadata.encryptionKeys().stream()
+                .map(EncryptedKey::encryptedById)
+                .filter(Objects::nonNull)
+                .collect(Collectors.toUnmodifiableSet()));
+    TableMetadata.Builder metadataBuilder = TableMetadata.buildFrom(metadata);
+    standardEncryptionManager.encryptionKeys().values().stream()
+        .filter(
+            encryptedKey ->
+                referencedEncryptionKeys.contains(encryptedKey.keyId())
+                    // The KEK may not be referenced but must still be saved in the metadata.
+                    || standardEncryptionManager.keyEncryptionKeyID().equals(encryptedKey.keyId()))
+        .forEach(metadataBuilder::addEncryptionKey);
+    return metadataBuilder.build();


This new logic depends on an invariance that the manifestListKey -> KEK -> table encryption key. are all one "hop". E.g. the -> means encryptedBy.

If there was a world where a key in the encryptionKeys() list is encrypted by another key in the encryptionKey list, which is then in itself encrypted by the KEK, and they were all added in a single commit, this PR would fail to propagate the middle key.
E.g.:

EncryptionKeys:[{key: A, encryptedBy: B}, {key: B, encryptedBy: C}, {key: C, encryptedBy: KEK}] snapshot: [snapshotId: x, keyId: A]

And all of the above happens within one commit.
In the above code, we would not add key B to the metadata file. As none of the encryptionKeys will be part of metadata.encryptionKeys() as that will be empty. We will keep A and C, since A is referenced by snapshot and C is a KEK.

TLDR: question @ggershinsky : we expect the depth of encryption key referencing from manifest list -> KEK to remain at a single hop right, we don't expect there to be another hop?

On second thought a behaviour change here would get caught by e2e tests. But worth noting.

yep, the approach is direct - to save KMS interactions, ml keys are encrypted by an intermediate local KEK, that in turn is encrypted by KMS. So it's always ml key -> KEK -> KMS. No reason to introduce another hop.

Hugo-WB · 2026-06-17T17:01:43Z

-    } else {
-      tableMetadata = metadata;
-    }
+    tableMetadata = EncryptionUtil.addEmKeysToMetadata(metadata, encrManager);


cc @smaheshwar-pltr for: #13225. If you could update your REST implementation to use this utility once it merges.

Hugo-WB · 2026-06-25T10:15:23Z

Friendly ping @huaxingao @singhpk234

Hugo-WB · 2026-07-06T16:01:02Z

Apologies for the ping, but another friendly ping here! 😄 @huaxingao @singhpk234

huaxingao · 2026-07-06T22:07:07Z

@singhpk234 Do you have more comments on this PR? If not, I will merge.

singhpk234

LGTM too, thanks @huaxingao for ping !

singhpk234 · 2026-07-07T00:09:38Z

+    SparkActions.get()
+        .expireSnapshots(table)
+        .expireSnapshotId(snapshotToExpire.snapshotId())
+        .execute();


minor/can be done in follow-up: its would be nice to assert that this snapshot is actually expired as well ?

Updated! 082d3f9

…keys

Hugo-WB · 2026-07-28T09:54:13Z

Friendly ping @huaxingao @singhpk234, if this is still good to merge 🙏

gaborkaszab

Thank you for the PR, @Hugo-WB !

I've bumped into this from a different angle: I'm about to add encryption to statistics files and I was wondering how to do the encryption key cleanup. Thx @ggershinsky to pointing me to this PR.
I think there might be an issue with the approach in some use-cases, probably not that complicated to fix. See in my comment.

gaborkaszab · 2026-08-04T17:40:36Z

          }
          removeStatistics(snapshotId);
          removePartitionStatistics(snapshotId);
+          removeEncryptionKey(snapshot.keyId());


I gave the PR to Claude and it might have found an issue with the current design:
suppressHistoricalSnapshots() method on TableMetadata.Builder may collide with the approach here. Let's say we create TableMetadata like below setting up lazy snapshot loading (considering 2 snapshots, a current and a historical one):

TableMetadata lazyMetadata = TableMetadata.buildFrom(metadata) .suppressHistoricalSnapshots() .setSnapshotsSupplier(() -> ImmutableList.of(historical, current)) .discardChanges() .build();

It will have only the referenced snapshots and omit the rest. Due to the current implementation it removes encryption keys for the unloaded snapshots. When we load the rest of the snapshots through snapshotSupplier we won't have encryption keys for all the snapshots, so time travel to such snapshots would fail.

Whether this is a realistic scenario or not, I can't judge, but you can repro this through the public API of the TableMetadata builder.
WDYT?

Hey! Sorry for the slowness here. Thanks for the review!

I think this is a valid point! And definitely does break the current implementation. It also does seem to be used here, in the exact case where SnapshotMode::REFS, so the rest catalog would've ran suppressHistoricalSnapshots.
Downside here is that SnapshotMode::REFS will now return a lot of the encryption keys 😭 and we are not necessarily reducing the size of the loadtable RPC.
I do feel setSnapshotsSupplier's API does seem to be a bit at odds with the way encryption keys are set up within the TableMetadata 😅 . If the goal of it is to supply snapshots that aren't currently within metadata, we should figure out how it can also supply encryption keys for those snapshots. Curious as to why manifest list encryption keys are stored at the metadata layer and not at the snapshot layer.

I've moved the removeEcnryptionKey into !suppress block so that we don't remove encryption keys for now. Lmk what you think!

Updated: Hugo-WB@d78c752

Thank you for getting back with this @Hugo-WB !

I think encryption keys is not any different than schemas or partition specs for instance. When we suppress snapshots to REFS we still need the complete list of schemes and specs in case we want to load later on the missing snapshots (suppress mode). When expiring snapshot, if no remaining snapshot uses them, they can be removed (!suppress mode).

One thing I think still breaks here: I'm not entirely sure we can assume that the mapping between snapshots and encryption keys is 1 to 1. In the reference implementation it seems to be the case, but there might be custom implementations where this isn't true.
To make this more general, I think we should also account the snapshots retained and their key IDs. Maybe adding the change not here but to RemoveSnapshots.internalApply()?

Agreed! In the current spec encryption keys are stored at the metadata layer, similar to schemas/partition specs. I think this makes sense for the KEK, however for manifest list encryption key, those are per snapshot and not re-used, so would've been nice to have them per snapshot.

As you mention above having the keys in the snapshot itself would also make the invariant of 1:1 snapshot <> encryption key more obvious.

Yep! I had this PR open that I was planning on opening up after this one merged: https://github.com/apache/iceberg/pull/16395/changes which creates the list of references keys and removes all unreferenced.

Agreed on the invariance, I mentioned it in the PR description. It seems like things that are 1:1 with snapshot are removed at TableMetadata::rewriteSnapshotsInternal and things that are shared across snapshots (schemas/partitions etc...) RemoveSnapshots::internalApply::cleanExpiredMetadata.

If we rely on this invariance, which I think people are generally agreed with relying on this invariant/it holding in the long run (Mentioned it in PR description and didn't get any disagreements)? I think it makes sense to remove encryption keys as part of TableMetadata::rewriteSnapshotsInternal.

Thanks! Curious on your thoughts.

I think the point of having encryption keys in TableMetadata and not per snapshot is to enable reusing those keys across files. The current reference implementation might not work that way, but a custom encryption manager/KMS might do. So I don't think we can expect 1:1 relation between snapshots and encryption keys. That's why I proposed to move removing keys to RemoveSnapshots.

👍 I see. If we're in favor of not relying on this invariant, I agree this PR should not be implemented:
Curious for your thoughts/review on: #18102 then!

Will defer to others that are more familiar with the original motivations behind making snapshot encryption keys stored in the snapshot metadata on whether we should rely on this invariant of snapshot -> key being 1:1.

I think it would be nice if encryption keys were removed with the snapshot and not require expire_snapshots with cleanExpiredMetadata.

Curious for thoughts from @ggershinsky and @singhpk234

motivations behind making snapshot encryption keys stored in the snapshot metadata on whether we should rely on this invariant of snapshot -> key being 1:1.

Today, there are two kinds of entries in the TableMetadata "encrypted keys" list. One is the manifest list keys, and the other is key encryption keys (the former are encrypted by the latter). Key encryption keys are re-used for a long time, but still should be cleaned up after all manifest list keys, encrypted by them, are removed.

(the condition for removal of a key encryption key - all of its ML keys are removed, and its lifetime has expired)

I'm quoting from @rdblue , let me know if I misunderstand: "the keys table is intended to have one or two keys that are reused, not a key per encrypted file referenced from the table metadata."
For me this means, that theoretically we can have a 1-many relationship between keys and manifest lists (or whatever else's key we store in the encrypted keys list).
The entire conversation.

#17533 (comment)

github-actions · 2026-09-04T00:24:27Z

This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pull request requires a review, please simply write any comment. If closed, you can revive the PR at any time and @mention a reviewer or discuss it on the dev@iceberg.apache.org list. Thank you for your contributions.

github-actions · 2026-09-11T00:26:40Z

This pull request has been closed due to lack of activity. This is not a judgement on the merit of the PR in any way. It is just a way of keeping the PR queue manageable. If you think that is incorrect, or the pull request requires review, you can revive the PR at any time.

…keys

Hugo-WB · 2026-09-14T10:34:32Z

Hey @huaxingao, sorry for letting this close, was out of office for a while. Could we please re-open this PR given the interest in the mailing list. thanks!

Hugo-WB · 2026-09-14T10:44:20Z

@singhpk234 as well if we're still happy to merge this 😄

gaborkaszab

Hi @Hugo-WB ,

Thanks for resuming the work on this! I can't reopen the PR but some committer can do so for you.
I think there are some conceptual questions still, I left some comments.

gaborkaszab · 2026-09-14T11:25:58Z

          }
          removeStatistics(snapshotId);
          removePartitionStatistics(snapshotId);
+          removeEncryptionKey(snapshot.keyId());


Thank you for getting back with this @Hugo-WB !

I think encryption keys is not any different than schemas or partition specs for instance. When we suppress snapshots to REFS we still need the complete list of schemes and specs in case we want to load later on the missing snapshots (suppress mode). When expiring snapshot, if no remaining snapshot uses them, they can be removed (!suppress mode).

One thing I think still breaks here: I'm not entirely sure we can assume that the mapping between snapshots and encryption keys is 1 to 1. In the reference implementation it seems to be the case, but there might be custom implementations where this isn't true.
To make this more general, I think we should also account the snapshots retained and their key IDs. Maybe adding the change not here but to RemoveSnapshots.internalApply()?

gaborkaszab · 2026-09-14T11:31:21Z

-    } else {
-      tableMetadata = metadata;
-    }
+    tableMetadata = EncryptionUtil.addEmKeysToMetadata(metadata, encrManager);


I'm not entirely sure what problem this part of the PR tries to solve. I had the impression that the current changes in TableMetadata are meant to cover the scope of the PR: "Clean up encryption keys as part of RemoveSnapshots".
This seems like another mechanism to clean up unreferenced encryption keys when committing to a table. If I'm not mistaken here with my assumption, then this is orthogonal and should be covered separately.

Sorry should've added a PR comment. Without this change the tests in TestTableEncryption would not pass. The previous HiveTableOperations constructed the EncryptionManager from the old metadata, then added all old encryption keys to the new metadata. Thus it was impossible to ever actually remove encryption keys.

Agree this is very confusing. Happy to discuss separately. But since EncryptionManager stores it's own state separately to TableMetadata, it needs to be kept in sync with the TableMetadata.

github-actions Bot added the core label May 15, 2026

Remove encryption keys as part of RemoveSnapshots

3d534d6

Hugo-WB force-pushed the cleanup-encryption-keys branch from 3166d6c to 3d534d6 Compare May 15, 2026 16:19

Hugo-WB marked this pull request as ready for review May 18, 2026 12:20

ggershinsky approved these changes May 21, 2026

View reviewed changes

singhpk234 reviewed May 29, 2026

View reviewed changes

Add E2E test + EncryptionUtil::addEmKeysToMetadata.

fdb236e

- While adding E2E test found a bug in the HiveTableOperations. - HiveTableOperations, as well as the currently open PR for the RestTableOperations, adds all keys from EncryptionManager into TableMetadata

github-actions Bot added spark hive labels Jun 16, 2026

Hugo-WB mentioned this pull request Jun 16, 2026

Core: Only add required encrypted keys to metadata #16837

Closed

huaxingao approved these changes Jun 16, 2026

View reviewed changes

Hugo-WB added 2 commits June 17, 2026 17:03

merge upstream/main

095e7e3

update comment

ad9e440

Hugo-WB commented Jun 17, 2026

View reviewed changes

singhpk234 approved these changes Jul 7, 2026

View reviewed changes

Hugo-WB added 2 commits July 15, 2026 18:03

comment: assert snapshot gets expired

082d3f9

Merge remote-tracking branch 'upstream/main' into cleanup-encryption-…

2f4d170

…keys

gaborkaszab reviewed Aug 4, 2026

View reviewed changes

gaborkaszab mentioned this pull request Aug 5, 2026

API, Core, Spark 4.1: Add key-id and encrypted write support for StatisticsFile #17417

Draft

github-actions Bot added the stale label Sep 4, 2026

github-actions Bot closed this Sep 11, 2026

Merge remote-tracking branch 'upstream/main' into cleanup-encryption-…

57ff2a9

…keys

Move removeEncryptionKey into !supress block

d78c752

gaborkaszab reviewed Sep 14, 2026

View reviewed changes

kevinjqliu reopened this Sep 14, 2026

kevinjqliu added not-stale and removed stale labels Sep 14, 2026

Conversation

Hugo-WB commented May 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Hugo-WB commented May 18, 2026

Uh oh!

ggershinsky commented May 18, 2026

Uh oh!

Hugo-WB commented May 18, 2026

Uh oh!

Hugo-WB commented May 21, 2026

Uh oh!

ggershinsky commented May 21, 2026

Uh oh!

Hugo-WB commented May 26, 2026

Uh oh!

singhpk234 left a comment

Choose a reason for hiding this comment

Uh oh!

huaxingao commented Jun 16, 2026

Uh oh!

Hugo-WB left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Hugo-WB Jun 18, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Hugo-WB commented Jun 25, 2026

Uh oh!

Hugo-WB commented Jul 6, 2026

Uh oh!

huaxingao commented Jul 6, 2026

Uh oh!

singhpk234 left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

singhpk234 Jul 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Hugo-WB commented Jul 28, 2026

Uh oh!

gaborkaszab left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Hugo-WB Sep 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Hugo-WB Sep 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

github-actions Bot commented Sep 4, 2026

Hugo-WB commented May 15, 2026 •

edited

Loading

Hugo-WB Jun 18, 2026 •

edited

Loading

singhpk234 left a comment •

edited

Loading

singhpk234 Jul 7, 2026 •

edited

Loading

Hugo-WB Sep 14, 2026 •

edited

Loading

Hugo-WB Sep 14, 2026 •

edited

Loading