Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 6 additions & 4 deletions .github/workflows/ci.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -28,7 +28,9 @@ jobs:
- name: Create test tmp dir
run: sudo mkdir /mnt/tmp && sudo chmod 1777 /mnt/tmp
- name: test
# 30m timeout: the restart-safety tests in resume_test.go run real
# end-to-end resizes (shrink + copy of a multi-GB fixture), which can
# exceed the default 10m. Use `-short` to skip them in a fast loop.
run: TMPDIR=/mnt/tmp go test -timeout 30m ./...
# 45m timeout: the restart-safety tests in resume_test.go and the
# EVE-patterned end-to-end tests run real resizes (shrink + copy of
# multi-GB fixtures), which together exceed the default 10m. Use
# `-short` to skip them in a fast loop. (The chaos soak test is opt-in
# via RESIZER_CHAOS and never runs here.)
run: TMPDIR=/mnt/tmp go test -timeout 45m ./...
53 changes: 53 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -126,3 +126,56 @@ are read-only and an inconsistent filesystem aborts the resize; pass `fixErrors`
to repair instead. squashfs sources are copied raw and have no applicable check,
so a corrupt squashfs source is reproduced faithfully.

## Testing

The suite has three tiers, selected with `-short` and a few environment
variables:

| Invocation | What runs |
|---|---|
| `go test ./...` | All tests, including end-to-end shrinks/copies of multi-GB fixtures (slow). |
| `go test -short ./...` | Skips those slow end-to-end fixtures; everything else still runs. |
| `RESIZER_CHAOS=1 go test -run '^TestChaosKill$' .` | Adds the SIGKILL resume soak (see below). |

The chaos soak is gated separately from `-short` because it is an open-ended
stress run, not a pass/fail CI gate, so it stays off even in a full `go test
./...`. Its environment knobs:

| Variable | Effect |
|---|---|
| `RESIZER_CHAOS=1` | Enable the chaos soak. Without it (and without `CHAOS_GPT_DELAY`) the test skips. |
| `CHAOS_SEED=<n>` | Seed the kill-timing RNG for a reproducible run. Default: random, logged as `CHAOS_SEED=<n>`. |
| `CHAOS_GPT_DELAY=<dur>` | Enable the soak *and* build the resizer subprocess with `-tags chaos`, delaying around GPT-sector writes so kills can land inside the otherwise-instantaneous `updatePartitions`/`createPartitions` table writes (e.g. `5s`). |
| `CHAOS_COPY_STATE=1` | After each kill, also classify the grow target's copied data as empty/partial/complete. |

`-tags chaos` is not something you pass to `go test`: the chaos test builds the
resizer subprocess with it internally when `CHAOS_GPT_DELAY` is set.

Comment thread
eriknordmark marked this conversation as resolved.
Resolved
### Chaos / resume soak test

`TestChaosKill` is a stress test, not a CI gate. It repeatedly runs the full
two-step resize (shrink the data partition, then grow the image/ESP partitions into the freed space) as a subprocess,
SIGKILLs it at random points across the pipeline, then re-runs to completion and
asserts the result always matches an uninterrupted resize. Because it is slow and
meant to run for minutes-to-hours, `go test ./...` skips it by default; enable it
explicitly:

```sh
# basic chaos run (kills land wherever timing puts them)
RESIZER_CHAOS=1 go test -run '^TestChaosKill$' -timeout 30m .

# reproduce a specific run
RESIZER_CHAOS=1 CHAOS_SEED=12345 go test -run '^TestChaosKill$' -timeout 30m .

# widen the window around GPT-table writes so kills can land inside the
# updatePartitions/createPartitions writes (builds the resizer with -tags chaos)
CHAOS_GPT_DELAY=5s go test -run '^TestChaosKill$' -timeout 40m .
```

Longer soaks are driven by an outer loop that re-invokes the test. It needs
`mksquashfs` (squashfs-tools) in addition to `resize2fs`/`e2fsck`. After each
kill the test logs the pipeline step it interrupted and the on-disk GPT integrity
(primary/backup header and entry-array CRCs, and primary↔backup entry equality);
set `CHAOS_COPY_STATE=1` to also classify the grow target's data copy as
empty/partial/complete.

80 changes: 80 additions & 0 deletions copyprogress_test.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,80 @@
package partitionresizer

import (
"bytes"
"os"
"sort"
"strings"
)

// copyDestState classifies each "<label>_resized2" grow target's data region to
// distinguish "nothing written" from a partial or complete copy -- the certain
// empty-vs-something signal the KILL_STEP marker alone cannot give. For a
// raw-copied source (squashfs/unknown) it compares the target's head and tail to
// the source: a head mismatch means nothing was copied, head match + tail
// mismatch means partial, both match means complete. For a filesystem copy
// (FAT32/ext4) a byte compare is meaningless (fresh metadata differs), so it
// looks for that filesystem's on-disk signature, which CreateFilesystem writes
// before any file data -- present means something was written.
//
// It locates partitions with gptPartitions (a raw entry-array read) for the same
// reason as the other helpers in gpt_raw_test.go: it runs on a post-kill disk
// whose table may be torn.
func copyDestState(path string) string {
f, err := os.Open(path)
if err != nil {
return "copydest:open-error"
}
defer func() { _ = f.Close() }()
parts := gptPartitions(f)
if parts == nil {
return "copydest:no-gpt"
}
const chunk = 4096
var states []string
for label, tgt := range parts {
if !strings.HasSuffix(label, alternateLabelSuffix) {
continue
}
src, ok := parts[strings.TrimSuffix(label, alternateLabelSuffix)]
if !ok {
states = append(states, label+"=no-source")
continue
}
srcStart := src[0] * 512
srcLen := (src[1] - src[0] + 1) * 512
tgtStart := tgt[0] * 512
srcHead, tgtHead := readAt(f, srcStart, chunk), readAt(f, tgtStart, chunk)
var state string
switch {
case srcHead == nil || tgtHead == nil:
state = "read-error"
case len(srcHead) >= 4 && string(srcHead[0:4]) == "hsqs": // squashfs => raw copy
switch {
case !bytes.Equal(srcHead, tgtHead):
state = "empty"
case srcLen <= chunk:
state = "complete"
default:
to := srcLen - chunk
if bytes.Equal(readAt(f, srcStart+to, chunk), readAt(f, tgtStart+to, chunk)) {
state = "complete"
} else {
state = "partial"
}
}
default: // FAT32/ext4 => filesystem copy; detect the target's fs signature
fat := len(tgtHead) >= 512 && tgtHead[510] == 0x55 && tgtHead[511] == 0xAA
ext4sb := readAt(f, tgtStart+1024, 512)
ext4 := len(ext4sb) >= 58 && ext4sb[56] == 0x53 && ext4sb[57] == 0xEF
if fat || ext4 {
state = "written"
} else {
state = "empty"
}
}
states = append(states, label+"="+state)
}
sort.Strings(states)
return "copydest: " + strings.Join(states, " ")
}
79 changes: 79 additions & 0 deletions delaybackend_chaos.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,79 @@
//go:build chaos

package partitionresizer

import (
"os"
"time"

"github.com/diskfs/go-diskfs/backend"
)

// maybeWrapBackend, in -tags chaos builds, wraps the backend so that a write
// touching a GPT metadata sector is followed by a delay
// (RESIZER_GPT_WRITE_DELAY, e.g. "10s"). go-diskfs writes the table as several
// WriteAt calls (backup entries, backup header, primary entries, primary
// header, protective MBR); the delay widens the window between them so a
// crash-injection test can land between the backup and primary writes -- in
// particular inside updatePartitions, which is otherwise a single fast table
// write that random-timed kills almost never catch.
func maybeWrapBackend(b backend.Storage) backend.Storage {
s := os.Getenv("RESIZER_GPT_WRITE_DELAY")
if s == "" {
return b
}
d, err := time.ParseDuration(s)
if err != nil || d <= 0 {
return b
}
var sectors int64
if fi, serr := b.Stat(); serr == nil {
sectors = fi.Size() / 512
}
return &delayBackend{Storage: b, delay: d, diskSectors: sectors}
}

type delayBackend struct {
backend.Storage
delay time.Duration
diskSectors int64
}

func (d *delayBackend) Writable() (backend.WritableFile, error) {
wf, err := d.Storage.Writable()
if err != nil {
return nil, err
}
return &delayWritable{WritableFile: wf, delay: d.delay, diskSectors: d.diskSectors}, nil
}

type delayWritable struct {
backend.WritableFile
delay time.Duration
diskSectors int64
}

func (d *delayWritable) WriteAt(p []byte, off int64) (int, error) {
n, err := d.WritableFile.WriteAt(p, off)
if err == nil && isGPTWrite(off, len(p), d.diskSectors) {
time.Sleep(d.delay)
}
return n, err
}

// isGPTWrite reports whether a write at byte offset off of the given length
// touches a GPT metadata sector (512-byte LBAs): the protective MBR + primary
// header + primary entry array (LBA 0..33), or the backup entry array + backup
// header (the last 33 sectors).
func isGPTWrite(off int64, length int, diskSectors int64) bool {
const sec = 512
start := off / sec
end := (off + int64(length) - 1) / sec
if start <= 33 { // overlaps the primary region (LBA 0..33)
return true
}
if diskSectors > 0 && end >= diskSectors-33 { // overlaps the backup region
return true
}
return false
}
10 changes: 10 additions & 0 deletions delaybackend_default.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,10 @@
//go:build !chaos

package partitionresizer

import "github.com/diskfs/go-diskfs/backend"

// maybeWrapBackend returns the backend unchanged in normal builds. The delaying
// wrapper exists only in -tags chaos builds (see delaybackend_chaos.go), so
// production binaries carry none of that code.
func maybeWrapBackend(b backend.Storage) backend.Storage { return b }
2 changes: 2 additions & 0 deletions go.mod
Original file line number Diff line number Diff line change
Expand Up @@ -22,3 +22,5 @@ require (
github.com/ulikunitz/xz v0.5.15 // indirect
golang.org/x/sys v0.46.0 // indirect
)

replace github.com/diskfs/go-diskfs => github.com/eriknordmark/go-diskfs v0.0.0-20260618133142-470c1e96eea8
19 changes: 3 additions & 16 deletions go.sum
Original file line number Diff line number Diff line change
Expand Up @@ -3,33 +3,22 @@ github.com/anchore/go-lzo v0.1.0/go.mod h1:3kLx0bve2oN1iDwgM1U5zGku1Tfbdb0No5qp1
github.com/cpuguy83/go-md2man/v2 v2.0.6/go.mod h1:oOW0eioCTA6cOiMLiUPZOpcVxMig6NIQQ7OS05n1F4g=
github.com/davecgh/go-spew v1.1.1 h1:vj9j/u1bqnvCEfJOwUhtlOARqs3+rkHYY13jYWTU97c=
github.com/davecgh/go-spew v1.1.1/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
github.com/diskfs/go-diskfs v1.7.1-0.20260212103528-6cfdf34dc382 h1:ViMskxlrdO3gqTCTdEMiHqLYkv3taRSa9u+Rw/rPHZ8=
github.com/diskfs/go-diskfs v1.7.1-0.20260212103528-6cfdf34dc382/go.mod h1:rW9+4MPN1tbMpQqRZlcM3YQsh3Ucc+Q1k1iIqzzmZcg=
github.com/diskfs/go-diskfs v1.9.3 h1:cLciNCeZ4QAXVxyPJDr1ZJ9N9CCG3rQlQ/z/Cs/cNDM=
github.com/diskfs/go-diskfs v1.9.3/go.mod h1:TePJORO83Adh5pb2SqsxAwaP0fofFxKLkxctiS/9OQc=
github.com/diskfs/go-diskfs v1.9.4-0.20260610103445-0e4e146f80a7 h1:tzFdxHtF0ref26PWfQ5WwdtF0coJu3+JMB9eOoxgM6Y=
github.com/diskfs/go-diskfs v1.9.4-0.20260610103445-0e4e146f80a7/go.mod h1:TePJORO83Adh5pb2SqsxAwaP0fofFxKLkxctiS/9OQc=
github.com/djherbis/times v1.6.0 h1:w2ctJ92J8fBvWPxugmXIv7Nz7Q3iDMKNx9v5ocVH20c=
github.com/djherbis/times v1.6.0/go.mod h1:gOHeRAz2h+VJNZ5Gmc/o7iD9k4wW7NMVqieYCY99oc0=
github.com/elliotwutingfeng/asciiset v0.0.0-20260129054604-cfde2086bc57 h1:x5yxNrq8XffV/OoNUeFPM6hxHVi5OTspSTBxr/9pemg=
github.com/elliotwutingfeng/asciiset v0.0.0-20260129054604-cfde2086bc57/go.mod h1:GLo/8fDswSAniFG+BFIaiSPcK610jyzgEhWYPQwuQdw=
github.com/go-test/deep v1.0.8 h1:TDsG77qcSprGbC6vTN8OuXp5g+J+b5Pcguhf7Zt61VM=
github.com/go-test/deep v1.0.8/go.mod h1:5C2ZWiW0ErCdrYzpqxLbTX7MG14M9iiw8DgHncVwcsE=
github.com/eriknordmark/go-diskfs v0.0.0-20260618133142-470c1e96eea8 h1:z/avFLFoPxPdW8Mt7+nQLFComDj01sht6m3bbf/j5rk=
github.com/eriknordmark/go-diskfs v0.0.0-20260618133142-470c1e96eea8/go.mod h1:TePJORO83Adh5pb2SqsxAwaP0fofFxKLkxctiS/9OQc=
github.com/go-test/deep v1.1.1 h1:0r/53hagsehfO4bzD2Pgr/+RgHqhmf+k1Bpse2cTu1U=
github.com/go-test/deep v1.1.1/go.mod h1:5C2ZWiW0ErCdrYzpqxLbTX7MG14M9iiw8DgHncVwcsE=
github.com/google/go-cmp v0.6.0 h1:ofyhxvXcZhMsU5ulbFiLKl/XBFqE1GSq7atu8tAmTRI=
github.com/google/go-cmp v0.6.0/go.mod h1:17dUlkBOakJ0+DkrSSNjCkIjxS6bF9zb3elmeNGIjoY=
github.com/google/go-cmp v0.7.0 h1:wk8382ETsv4JYUZwIsn6YpYiWiBsYLSJiTsyBybVuN8=
github.com/google/go-cmp v0.7.0/go.mod h1:pXiqmnSA92OHEEa9HXL2W4E7lf9JzCmGVUdgjX3N/iU=
github.com/google/uuid v1.6.0 h1:NIvaJDMOsjHA8n1jAhLSgzrAzy1Hgr+hNrb57e+94F0=
github.com/google/uuid v1.6.0/go.mod h1:TIyPZe4MgqvfeYDBFedMoGGpEw/LqOeaOT+nhxU+yHo=
github.com/inconshreveable/mousetrap v1.1.0 h1:wN+x4NVGpMsO7ErUn/mUI3vEoE6Jt13X2s0bqwp9tc8=
github.com/inconshreveable/mousetrap v1.1.0/go.mod h1:vpF70FUmC8bwa3OWnCshd2FqLfsEA9PFc4w1p2J65bw=
github.com/klauspost/compress v1.18.4 h1:RPhnKRAQ4Fh8zU2FY/6ZFDwTVTxgJ/EMydqSTzE9a2c=
github.com/klauspost/compress v1.18.4/go.mod h1:R0h/fSBs8DE4ENlcrlib3PsXS61voFxhIs2DeRhCvJ4=
github.com/klauspost/compress v1.18.6 h1:2jupLlAwFm95+YDR+NwD2MEfFO9d4z4Prjl1XXDjuao=
github.com/klauspost/compress v1.18.6/go.mod h1:cwPg85FWrGar70rWktvGQj8/hthj3wpl0PGDogxkrSQ=
github.com/pierrec/lz4/v4 v4.1.25 h1:kocOqRffaIbU5djlIBr7Wh+cx82C0vtFb0fOurZHqD0=
github.com/pierrec/lz4/v4 v4.1.25/go.mod h1:EoQMVJgeeEOMsCqCzqFm2O0cJvljX2nGZjcRIPL34O4=
github.com/pierrec/lz4/v4 v4.1.27 h1:+PhzhWDrjRj89TH2sw43nE3+4+W8lSxIuQadEHZyjUk=
github.com/pierrec/lz4/v4 v4.1.27/go.mod h1:EoQMVJgeeEOMsCqCzqFm2O0cJvljX2nGZjcRIPL34O4=
github.com/pkg/xattr v0.4.12 h1:rRTkSyFNTRElv6pkA3zpjHpQ90p/OdHQC1GmGh1aTjM=
Expand All @@ -50,8 +39,6 @@ github.com/ulikunitz/xz v0.5.15/go.mod h1:nbz6k7qbPmH4IRqmfOplQw/tblSgqTqBwxkY0o
go.yaml.in/yaml/v3 v3.0.4/go.mod h1:DhzuOOF2ATzADvBadXxruRBLzYTpT36CKvDb3+aBEFg=
golang.org/x/sys v0.0.0-20220408201424-a24fb2fb8a0f/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.0.0-20220615213510-4f61da869c0c/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.41.0 h1:Ivj+2Cp/ylzLiEU89QhWblYnOE9zerudt9Ftecq2C6k=
golang.org/x/sys v0.41.0/go.mod h1:OgkHotnGiDImocRcuBABYBEXf8A9a87e/uXjp9XT3ks=
golang.org/x/sys v0.46.0 h1:noSf2Fq6F8DBgS+LysIkx7rIExoNHJsxOAtPp4rthXw=
golang.org/x/sys v0.46.0/go.mod h1:4GL1E5IUh+htKOUEOaiffhrAeqysfVGipDYzABqnCmw=
gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=
Expand Down
Loading
Loading