feat(git): per-repo partial-clone snapshot filters#394
Conversation
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 2be39bef90
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 903a4c67fc
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
Some repos' snapshot artifacts are dominated by history no checkout ever touches. A new snapshot-filters config maps host/org/repo to a git partial-clone filter (e.g. blob:none) applied to the workstation snapshot clone via --no-local --filter. Filtered snapshots keep full commit/tree history but only HEAD blobs; clients lazily fetch older blobs through cachew. Mirror and LFS snapshots are unchanged. Co-authored-by: Claude Code <noreply@anthropic.com> Ai-assisted: true
git writes remote.origin.promisor and partialclonefilter even when the server does not support filtering and the clone silently falls back to a full clone, so the promisor config is no proof the filter applied. Detect the fallback from git's warning in the clone output instead, with LC_ALL=C pinned so the message is not localized. Co-authored-by: Claude Code <noreply@anthropic.com> Ai-assisted: true
903a4c6 to
64dbbc0
Compare
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 64dbbc05d9
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| ZstdThreads int `hcl:"zstd-threads,optional" help:"Threads for zstd compression/decompression. 0 = all CPU cores; useful for short-lived CLI invocations but risky on a long-running server where multiple snapshot/restore operations can run concurrently." default:"4"` | ||
| BundleCacheTTL time.Duration `hcl:"bundle-cache-ttl,optional" help:"TTL of cached server-side git bundles." default:"2h"` | ||
|
|
||
| SnapshotFilters map[string]string `hcl:"snapshot-filters,optional" help:"Per-repository git partial-clone filter applied to workstation snapshots, keyed by host/org/repo (e.g. {\"github.com/org/repo\": \"blob:none\"}). Filtered snapshots contain full history metadata but only the blobs needed for the HEAD checkout; clients lazily fetch historical blobs through cachew on demand."` |
There was a problem hiding this comment.
When snapshot-filters is changed for a repo whose HEAD has not moved, the existing https://...snapshot cache entry is still considered valid: cache hits are served without checking this new setting, and generateAndUploadSnapshot skips regeneration based only on the upstream/head/cache key. That means enabling blob:none can keep returning the old full snapshot, while rolling the filter back can keep returning partial-clone snapshots until max-age/cache expiry or manual deletion. Store and compare the filter in snapshot metadata/coordination, or make the cache key vary by filter, so config changes take effect deterministically.
Useful? React with 👍 / 👎.
2 participants
Adds a
snapshot-filtersoption to the git strategy: a map ofhost/org/repoto a git partial-clone filter (e.g.blob:none) applied to the workstation snapshot clone. Filtered snapshots keep full commit/tree history but only the blobs needed for the HEAD checkout; clients lazily fetch historical blobs through cachew on demand (mirrors already setuploadpack.allowFilterandallowAnySHA1InWant).Motivation: for repos whose history dwarfs their checkout, the snapshot artifact is almost entirely dead weight. One large monorepo we cache serves a snapshot in the tens of GB, dominated by superseded binary test fixtures in history, while its HEAD checkout needs about 1 GiB — a
blob:nonesnapshot of the same repo is roughly 95% smaller, which proportionally cuts snapshot storage, transfer, and the download window in which client timeouts and stalls can occur.Details:
--no-local(git silently ignores--filteron hardlink local clones), which also stops objects unreachable from cloned refs — e.g.refs/pull/*history in the mirror — from riding along into the artifact.Default behavior is unchanged; repos opt in via configuration.