Skip to content

Clone mirrors aside and clean up interrupted clones - #3

Merged
sergeyfast merged 1 commit into
masterfrom
fix/mirror-clone-junk
Oct 21, 2026
Merged

sergeyfast merged 1 commit into
masterfrom
fix/mirror-clone-junk

Conversation

@sergeyfast

Copy link
Copy Markdown
Member

Why

A mirror was cloned in place, and "HEAD exists" meant "the mirror exists". On one instance with little memory, clones were OOM-killed and cut off by the 5-minute timeout again and again. Nothing cleaned up after them: the mirror of a 0.85 GB repository grew to 65 GB of packs with no HEAD. After that every cron tick failed with destination path already exists and is not an empty directory, and the mirror stayed broken until someone removed it by hand.

What changed

  • Clone into .clone-<name>-*, then rename into place after the first fetch. A clone that fails or is killed leaves no directory that looks like a mirror.
  • Junk without HEAD at the mirror path is removed before cloning, so mirrors broken by earlier versions heal on their own. Only a missing HEAD counts; a stat error is returned.
  • repairConfig runs on every Ensure. It scrubs a token from the remote URL and restores a missing fetch refspec, and writes the config through a rename.
  • The whole process group is killed on cancel (pkg/client/proc), for git and for the AST engine, whose npm install is a node shim around the binary. Before, helpers kept writing and held the pipes, so Run outlived its deadline by minutes.
  • The first clone gets its own 30-minute CloneTimeout, and at most 2 clones run at a time. once() lets work outlive a caller that gave up, so the cron limit bounded waiting, not cloning.
  • Prune removes .clone-* directories and objects/pack/tmp_* files that have been untouched longer than a clone plus its fetch, regardless of the disk budget. Removals are reported through Options.OnRemove and logged as storage removed.
  • A new git.ErrCloning is returned when the caller stops waiting for a clone that is still running. Cron logs it as mirror still cloning, and tools answer Timeout with a hint to retry instead of Upstream.

Tests

  • 8 new tests. On the old code they fail as expected: the clone into junk fails with the same error seen in production, and Run hangs for 60 s after cancel.
  • go test ./... and -race on the touched packages pass, golangci-lint: 0 issues.
  • Builds for darwin, linux and windows.

Rollout notes

  • Existing mirrors are untouched: a config that already has the refspec is not rewritten.
  • On rollback to v0.0.3, leftover .clone-* directories are not cleaned by the old code; remove them by hand.

- Clone into a .clone-<name>-* directory and rename it into place after the first fetch, so a clone killed by the timeout, a restart or the OOM killer no longer leaves a directory that looks like a mirror or that git refuses to clone into
- Remove junk without HEAD at the mirror path before cloning, so mirrors broken by earlier versions heal on their own
- Repair clone config on every Ensure in one place: scrub a token from the remote URL and restore a missing fetch refspec; write it through a rename
- Kill the whole process group on cancel (new pkg/client/proc), for git and the AST engine: helpers kept writing and held the pipes, so Run outlived its deadline
- Give the first clone its own 30-minute timeout and limit clones to two at a time: a caller that gives up does not stop its clone, so the cron limit bounded waiting, not cloning
- Remove abandoned clone directories and tmp_* packs in Prune, regardless of the disk budget, and report removals through Options.OnRemove for the log
- Return git.ErrCloning when the caller stops waiting for a clone in progress: log it as "mirror still cloning" and answer tools with Timeout instead of Upstream
@sergeyfast
sergeyfast merged commit ccea9a1 into master Oct 21, 2026
2 checks passed
@sergeyfast
sergeyfast deleted the fix/mirror-clone-junk branch October 21, 2026 18:44
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant