Skip to content

Spec: Add key-id and deprecate key-metadata in table and partition statistics #17533

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
gaborkaszab wants to merge 3 commits into
base: main
Choose a base branch
from
Open

Spec: Add key-id and deprecate key-metadata in table and partition statistics #17533

Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
7926a1b
Spec: Add key-id and deprecate key-metadata in table and partition st…
gaborkaszab Aug 5, 2026
54a6682
Don't remove key-metadata, revert regrouping version columns
gaborkaszab Aug 25, 2026
3d2a28b
Remove V4 columns
gaborkaszab Aug 31, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 3 additions & 1 deletion format/spec.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1214,7 +1214,8 @@ Statistics files metadata within `statistics` table metadata field is a struct w
| _required_ | _required_ | **`statistics-path`** | `string` | Path of the statistics file. See [Puffin file format](puffin-spec.md). |
| _required_ | _required_ | **`file-size-in-bytes`** | `long` | Size of the statistics file. |
| _required_ | _required_ | **`file-footer-size-in-bytes`** | `long` | Total size of the statistics file's footer (not the footer payload size). See [Puffin file format](puffin-spec.md) for footer definition. |
| _optional_ | _optional_ | **`key-metadata`** | | Base64-encoded implementation-specific key metadata for encryption. |
| _optional_ | _optional_ | **`key-metadata`** | `string` | Base64-encoded implementation-specific key metadata for encryption. (**Deprecated**: use `key-id` instead) |

@rdblue rdblue Sep 16, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think there is still a problem in that we are mixing file-level keys with table-level keys. The keys table is intended to be a small set of keys that are reused and occasionally rotated, not file-specific.

@ggershinsky, how should we handle these? Do we need a way to store encrypted key metadata? What I'm thinking is that we keep key-metadata but add an optional key-id that is used to encrypt the stats file key?

@ggershinsky ggershinsky Sep 17, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

for manifest lists, we did the following: #12162 #12927

An "encrypted key metadata" is a 4-field record (key-id, encrypted-key-metadata, encrypted-by-id, properties map). All of these records are kept in the TableMetadata.encryption-keys list. Some of them are reusable (key encryption keys), but most of them are specific to manifest files. Each snapshot object keeps the key-id of the "encrypted key metadata" of its manifest list file.

The situation with stats files looks similar (if not identical) to the manifest list files. A straightforward technical approach would be to do the same: write a 4-field "encrypted key metadata" record in the TableMetadata.encryption-keys list, and add pointer to the record (key-id) to the stats file object.

@gaborkaszab gaborkaszab Sep 17, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for looking into this @rdblue and @ggershinsky !

I think the main questions here are 1) reusability of keys and 2) whether reusability requires any special care in the spec.

  1. I did an experiment the other day and for me it seems that each manifest list file adds a new EncryptedKey into the encryption-keys list. There is (are?) additional KEK entry(ies) on top of that in the list. I'm not sure about custom implementations but for me it seems that there is no reuse of EncryptedKey entries across different manifest lists, apart from the KEK used to encrypt the EncryptedKeys.
    Keys for stat files fit into this pattern, they could behave the same way as keys for manifest lists. Let me know if I miss something here.
  2. I think the spec changes in this PR can express both the reused key and the non-reused key scenario. key-id for statistics files is just an ID to be used for finding the EncryptedKey in the encryption-keys list that in turn can be used to decrypt the statistics files (after decrypting the found EncryptedFile itself using the KEK). Whether this is a reused EncryptedFile across files or not, I don't think matters in this context.

While I'm still learning this area of the project, let me know where my assumptions go wrong :)

| _optional_ | _optional_ | **`key-id`** | `string` | ID of the encryption key that encrypts the statistics file key metadata |
| _required_ | _required_ | **`blob-metadata`** | `list<blob metadata>` (see below) | A list of the blob metadata for statistics contained in the file with structure described below. |

Blob metadata is a struct with the following fields:
Expand Down Expand Up @@ -1244,6 +1245,7 @@ Partition statistics file must be registered in the table metadata file to be co
| _required_ | _required_ | _required_ | **`snapshot-id`** | `long` | ID of the Iceberg table's snapshot the partition statistics file is associated with. |
| _required_ | _required_ | _required_ | **`statistics-path`** | `string` | Path of the partition statistics file. See [Partition statistics file](#partition-statistics-file). |
| _required_ | _required_ | _required_ | **`file-size-in-bytes`** | `long` | Size of the partition statistics file. |
| _optional_ | _optional_ | _optional_ | **`key-id`** | `string` | ID of the encryption key that encrypts the partition statistics file key metadata |

##### Partition Statistics File

Expand Down