diff --git a/README.md b/README.md
index bd53bb1..be19795 100644
--- a/README.md
+++ b/README.md
@@ -1,5 +1,9 @@
# gloak
+
+ 
+
+
@@ -13,10 +17,10 @@
>
> Use it for testing, non-critical data, or secondary backups only. If you store anything important with gloak, keep another verified backup somewhere else.
-Encrypt files, split into chunks with SHA-256 checks, upload, verify, restore. Metadata (filenames, sizes, recipients, hashes) lives inside an age-encrypted manifest. Storage never sees plaintext metadata.
+gloak splits a file into fixed-size chunks, encrypts each chunk with age, uploads them, and verifies them before restore. Metadata such as filenames, sizes, recipients, and hashes lives inside an age-encrypted manifest. Storage only sees backup IDs, public completion markers, and encrypted chunk objects.
```text
-keygen → encrypt → chunk → encrypt manifest → upload → verify → restore
+keygen → chunk → encrypt chunks → upload → resume if needed → verify → restore
```
---
@@ -36,6 +40,9 @@ go build -o gloak .
# encrypt + upload
./gloak upload ./secret.tar --remote myremote:backup_folder
+# resume an interrupted upload
+./gloak resume
+
# verify
./gloak verify --remote myremote:backup_folder -i ~/.gloak/identity.txt
@@ -47,14 +54,14 @@ go build -o gloak .
## Why this exists
-Most backup tools either:
+Most backup tools make at least one tradeoff gloak avoids:
-1. Encrypt a single blob — good luck retrying after a network failure at byte 47 GB of 50.
-2. Split into chunks without integrity metadata — you find out a chunk is corrupt during restore, with no manifest to guide repair.
-3. Store metadata in plaintext — filenames, sizes, recipient keys, and chunk hashes visible to anyone who reads the storage.
+1. Encrypt a single blob. If a network failure hits at byte 47 GB of 50, retrying is painful.
+2. Split into chunks without enough integrity metadata. You find corruption during restore, without a manifest to guide repair.
+3. Store metadata in plaintext. Filenames, sizes, recipient keys, and chunk hashes are visible to anyone who can read the storage.
4. Pull in rclone, restic rest-server, or S3 libraries when you just need local filesystem or a mounted remote.
-gloak is the boring correct version: age encryption, fixed-size chunks, deterministic manifests, integrity checks on every chunk *and* the full payload, and an encrypted manifest envelope so storage never sees filenames or hashes.
+gloak keeps the design narrow: age encryption, fixed-size chunks, chunk and payload hashes, resumable uploads, and an encrypted manifest so the storage backend does not see filenames or hashes.
---
@@ -70,7 +77,7 @@ Build the standalone binary:
go build -o gloak .
```
-Produces `gloak` — a compiled binary. Drop it into containers, cron jobs, or VPS rootfs without needing Node, Bun, or Python installed.
+This produces `gloak`, a compiled binary you can put in containers, cron jobs, or VPS root filesystems without installing Node, Bun, or Python.
Run tests:
@@ -92,8 +99,8 @@ Default location is `~/.gloak/`:
```text
~/.gloak/
- identity.txt # private key — keep this secret
- recipient.txt # public key — safe to share
+ identity.txt # private key, keep this secret
+ recipient.txt # public key, safe to share
```
Use a custom directory:
@@ -116,7 +123,7 @@ After `keygen` runs once, `-i` (identity) and `-r` (recipient) flags become opti
## Commands
-### `setup` — install dependencies
+### `setup`: install dependencies
If you don't have `rclone` installed on your system, `gloak setup` will download a standalone, verified binary to `~/.gloak/bin/rclone` so `gloak` can use it without touching your system packages.
@@ -131,9 +138,9 @@ If you don't have `rclone` installed on your system, `gloak setup` will download
---
-### `upload` — full encrypted backup flow
+### `upload`: create an encrypted backup
-Creates a UUID folder, encrypts the input, splits into chunks, writes SHA-256 checksums into an age-encrypted manifest, and uploads in order: chunks first, manifest second, locator last.
+Creates a UUID folder, splits the input into fixed-size plaintext chunks, encrypts each chunk independently with age, records SHA-256 checksums in an age-encrypted manifest, and uploads chunks first. `manifest.age` and `locator.json` are uploaded after the chunks.
Chunks and manifest are encrypted with the same age recipient. Storage sees filenames like `chunk_00000` and `manifest.age` but cannot read filenames, sizes, or hashes without the identity key.
@@ -160,13 +167,25 @@ Save this UUID for restoration: 7f91c6c7-7a0b-44aa-ae23-997b60e4e998
| `--recipient ` | `-r` | `~/.gloak/recipient.txt` | Recipient public key or recipient file. |
| `--remote` | `-R` | _required_ | Target Rclone destination (e.g. `myremote:backup_folder` or `/mnt/backups`). |
-The locator is uploaded last. If the upload is interrupted, the backup directory exists but has no `locator.json`. Upload refuses to overwrite on retry, so partial uploads never look complete.
+The locator is uploaded last. If the upload is interrupted, the backup directory exists but has no `locator.json`. gloak keeps local state under `~/.gloak/uploads/.json`, which lets `resume` skip chunks that already uploaded cleanly.
---
-### `verify` — verify backup integrity
+### `resume`: continue an interrupted upload
-Downloads and decrypts `manifest.age`, checks every chunk exists with matching size and SHA-256, then checks the concatenated encrypted payload hash against the manifest.
+Reads local upload state from `~/.gloak/uploads`, scans the source file from the beginning, skips chunks whose plaintext size and SHA-256 still match the saved state, uploads missing chunks, then writes `manifest.age` and `locator.json`.
+
+```bash
+./gloak resume 7f91c6c7-7a0b-44aa-ae23-997b60e4e998
+```
+
+Resume requires the original source file at the same path with the same size. gloak does not keep plaintext or encrypted payload spool files locally. It stores only the chunk metadata needed to decide which uploaded chunks can be skipped.
+
+---
+
+### `verify`: check backup integrity
+
+Downloads and decrypts `manifest.age`, checks each encrypted chunk size and SHA-256, decrypts each chunk, checks plaintext size and SHA-256, then compares the full plaintext payload hash with the manifest.
```bash
./gloak verify 7f91c6c7-7a0b-44aa-ae23-997b60e4e998 \
@@ -193,9 +212,9 @@ CHUNK CORRUPT: Hash chunk_00042 mismatch!
---
-### `restore` — restore a backup
+### `restore`: restore a backup
-Verifies the full backup first (every chunk, every hash), then concatenates chunks and decrypts the payload.
+Checks each encrypted chunk, decrypts it, checks the plaintext chunk, then writes the restored file chunk by chunk.
```bash
# with auto-resolved identity
@@ -227,9 +246,9 @@ Restore rejects:
---
-### `cleanup` — remove incomplete backups
+### `cleanup`: remove incomplete backups
-Scans the remote storage for backup directories that are missing `manifest.age` (usually caused by interrupted uploads) and offers to delete them.
+Scans remote storage for backup directories that are missing `manifest.age`, usually from interrupted uploads, and offers to delete them.
```bash
# interactive prompt before deletion
@@ -260,7 +279,7 @@ Scans the remote storage for backup directories that are missing `manifest.age`
chunk_00002
```
-**`locator.json`** is a small public file:
+`locator.json` is a small public file:
```json
{
@@ -269,24 +288,22 @@ Scans the remote storage for backup directories that are missing `manifest.age`
}
```
-The locator has no sensitive metadata. No filename, no recipient, no chunk hashes, no payload hash. It exists so tools can detect backup completion without decrypting anything.
+The locator has no sensitive metadata. It contains no filename, recipient, chunk hashes, or payload hash. Tools can use it to detect backup completion without decrypting anything.
-**`manifest.age`** is the full backup metadata encrypted with age. Once decrypted:
+`manifest.age` is the full backup metadata encrypted with age. Once decrypted:
| Field | Contains |
|-------|----------|
-| `source.original_filename` | Basename of backed-up file |
-| `source.original_size` | Original file size |
-| `payload.encryption.recipients` | Age recipient(s) |
-| `payload.encrypted_file_sha256` | SHA-256 of encrypted payload |
-| `chunking.chunk_size` | Fixed chunk size |
-| `chunking.total_chunks` | Number of chunks |
+| `original_name` | Basename of backed-up file |
+| `original_size` | Original file size |
+| `recipient` | Age recipient |
+| `payload_sha256` | SHA-256 of the full plaintext payload |
| `chunks[].index` | Chunk index (0-based) |
-| `chunks[].name` | Chunk filename (e.g. `000042.chunk`) |
-| `chunks[].size` | Chunk byte size |
-| `chunks[].sha256` | SHA-256 of chunk content |
-
-
+| `chunks[].name` | Chunk filename (e.g. `chunk_00042`) |
+| `chunks[].plain_size` | Plaintext chunk byte size |
+| `chunks[].plain_sha256` | SHA-256 of plaintext chunk content |
+| `chunks[].encrypted_size` | Encrypted chunk byte size |
+| `chunks[].encrypted_sha256` | SHA-256 of encrypted chunk content |
## Architecture
@@ -295,6 +312,7 @@ The locator has no sensitive metadata. No filename, no recipient, no chunk hashe
├── setup.go # rclone downloader command
├── keygen.go # keygen command definition
├── upload.go # upload command definition
+├── resume.go # resume command definition
├── verify.go # verify command definition
├── restore.go # restore command definition
├── cleanup.go # cleanup command definition
@@ -307,6 +325,7 @@ The locator has no sensitive metadata. No filename, no recipient, no chunk hashe
│ └── stream.go # streaming age encrypt/decrypt wrapper
├── flow/
│ ├── upload.go # upload workflow logic
+ │ ├── resume.go # resume workflow logic
│ ├── verify.go # verification workflow logic
│ └── restore.go # restore workflow logic
└── storage/
@@ -317,23 +336,27 @@ The locator has no sensitive metadata. No filename, no recipient, no chunk hashe
```text
upload:
- input file → age encrypt → encrypted payload
- → split into fixed-size chunks
- → SHA-256 each chunk
- → manifest.age (with all metadata)
- → upload chunks → upload manifest.age → upload locator.json
+ input file → split into plaintext chunks
+ → SHA-256 plaintext chunk + full payload
+ → age encrypt each chunk
+ → SHA-256 encrypted chunk
+ → upload chunks → upload manifest.age → upload locator.json
+
+resume:
+ upload state → re-read source chunks
+ → skip chunks whose plaintext hash still matches state
+ → upload missing chunks → upload manifest.age → upload locator.json
verify:
locator.json → completion marker
→ manifest.age → age decrypt → validate manifest
- → check every chunk exists
- → check every chunk size
- → check every chunk SHA-256
- → concatenate chunks
- → check encrypted payload SHA-256
+ → check encrypted chunk size/SHA-256
+ → decrypt each chunk
+ → check plaintext chunk size/SHA-256
+ → check full payload SHA-256
restore:
- verify (full) → download chunks → concatenate → decrypt payload → write output
+ manifest.age → download encrypted chunk → verify → decrypt chunk → write output
```
### Key design decisions
@@ -341,17 +364,16 @@ restore:
| Decision | Rationale |
|----------|-----------|
| Recipient-mode age encryption | One identity decrypts. No passphrase to remember or leak. |
+| Per-chunk encryption | Each chunk is a standalone age payload, so interrupted uploads can resume without deterministic encryption. |
| Fixed-size chunks | Deterministic boundaries regardless of content. Simplifies verification, resumability, and storage layout. |
| Encrypted manifest | Storage never sees filenames, sizes, or hashes. Only backup UUIDs and chunk object names are visible. |
-| Locator uploaded last | Atomicity marker. If upload is interrupted, no `locator.json` — upload refuses to overwrite on retry. |
+| Locator uploaded last | If upload is interrupted, no `locator.json` marks the backup complete. |
| Identity required for verify/restore | Manifest is encrypted. You need the identity key to read it. This is a feature, not a bug. |
-| SHA-256 at chunk + payload | Repair individual chunks without re-downloading everything. Corruption detected before decrypt. |
+| SHA-256 at chunk + payload | Corruption is detected before and after decrypting each chunk. |
| Fixed 20MB chunk size | Prevents metadata bloat from tiny chunks and excessive memory usage. |
---
-
-
## Not built yet
Deferred intentionally:
@@ -359,9 +381,8 @@ Deferred intentionally:
| Feature | Reasoning |
|---------|-----------|
| Named storage profiles | `--remote my-backups` instead of typing URIs. |
-| Resumable uploads | Track uploaded chunks, skip on retry. Needs state file in backup directory. |
| Telegram integration | Hermes Agent plugin for backup/restore/verify via Telegram. |
-| Per-chunk encryption keys | Each chunk encrypted with a unique key, stored in manifest. Limits blast radius of key exposure. |
+| Per-chunk data keys | Each chunk would use its own data key stored inside the encrypted manifest. This would limit key exposure if a key leaks. |
---
diff --git a/assets/gloak-banner.svg b/assets/gloak-banner.svg
new file mode 100644
index 0000000..43c8a69
--- /dev/null
+++ b/assets/gloak-banner.svg
@@ -0,0 +1,89 @@
+
diff --git a/internal/core/chunk.go b/internal/core/chunk.go
index 77e58f9..7627769 100644
--- a/internal/core/chunk.go
+++ b/internal/core/chunk.go
@@ -10,18 +10,32 @@ import (
const ChunkSize = 20 * 1024 * 1024
type ChunkMeta struct {
- Index int `json:"index"`
- Name string `json:"name"`
- Size int64 `json:"size"`
- SHA256 string `json:"sha256"`
+ Index int `json:"index"`
+ Name string `json:"name"`
+ PlainSize int64 `json:"plain_size"`
+ PlainSHA256 string `json:"plain_sha256"`
+ EncryptedSize int64 `json:"encrypted_size"`
+ EncryptedSHA256 string `json:"encrypted_sha256"`
}
func StreamChunker(in io.Reader, uploadFn func(chunkReader io.Reader, index int) error) ([]ChunkMeta, error) {
+ return StreamChunkerWithCallback(in, uploadFn, nil)
+}
+
+func StreamChunkerWithCallback(in io.Reader, uploadFn func(chunkReader io.Reader, index int) error, afterChunkFn func(ChunkMeta) error) ([]ChunkMeta, error) {
+ return StreamChunkerWithSize(in, ChunkSize, uploadFn, afterChunkFn)
+}
+
+func StreamChunkerWithSize(in io.Reader, chunkSize int64, uploadFn func(chunkReader io.Reader, index int) error, afterChunkFn func(ChunkMeta) error) ([]ChunkMeta, error) {
+ if chunkSize <= 0 {
+ return nil, fmt.Errorf("chunk size must be positive")
+ }
+
var metadata []ChunkMeta
chunkIndex := 0
for {
- limitedReader := io.LimitReader(in, ChunkSize)
+ limitedReader := io.LimitReader(in, chunkSize)
hasher := sha256.New()
tee := io.TeeReader(limitedReader, hasher)
@@ -38,14 +52,20 @@ func StreamChunker(in io.Reader, uploadFn func(chunkReader io.Reader, index int)
}
hashString := hex.EncodeToString(hasher.Sum(nil))
- metadata = append(metadata, ChunkMeta{
- Index: chunkIndex,
- Name: fmt.Sprintf("chunk_%05d", chunkIndex),
- Size: counter.BytesRead,
- SHA256: hashString,
- })
-
- if counter.BytesRead < ChunkSize {
+ chunk := ChunkMeta{
+ Index: chunkIndex,
+ Name: fmt.Sprintf("chunk_%05d", chunkIndex),
+ PlainSize: counter.BytesRead,
+ PlainSHA256: hashString,
+ }
+ metadata = append(metadata, chunk)
+ if afterChunkFn != nil {
+ if err := afterChunkFn(chunk); err != nil {
+ return nil, fmt.Errorf("chunk %d callback failed: %w", chunkIndex, err)
+ }
+ }
+
+ if counter.BytesRead < chunkSize {
break
}
diff --git a/internal/core/chunk_test.go b/internal/core/chunk_test.go
index ce76a04..1315d08 100644
--- a/internal/core/chunk_test.go
+++ b/internal/core/chunk_test.go
@@ -7,6 +7,7 @@ import (
"errors"
"fmt"
"io"
+ "strings"
"testing"
)
@@ -42,12 +43,12 @@ func TestStreamChunkerBuildsChunkMetadata(t *testing.T) {
if chunk.Name != wantName {
t.Fatalf("chunk %d name = %q, want %q", i, chunk.Name, wantName)
}
- if chunk.Size != wantSizes[i] {
- t.Fatalf("chunk %d size = %d, want %d", i, chunk.Size, wantSizes[i])
+ if chunk.PlainSize != wantSizes[i] {
+ t.Fatalf("chunk %d size = %d, want %d", i, chunk.PlainSize, wantSizes[i])
}
wantHash := sha256.Sum256(uploaded[i])
- if chunk.SHA256 != hex.EncodeToString(wantHash[:]) {
- t.Fatalf("chunk %d sha256 = %q, want %q", i, chunk.SHA256, hex.EncodeToString(wantHash[:]))
+ if chunk.PlainSHA256 != hex.EncodeToString(wantHash[:]) {
+ t.Fatalf("chunk %d sha256 = %q, want %q", i, chunk.PlainSHA256, hex.EncodeToString(wantHash[:]))
}
}
}
@@ -62,6 +63,73 @@ func TestStreamChunkerPropagatesUploadError(t *testing.T) {
}
}
+func TestStreamChunkerWithCallbackReportsChunkMetadata(t *testing.T) {
+ payload := []byte("payload")
+ var seen []ChunkMeta
+
+ chunks, err := StreamChunkerWithCallback(bytes.NewReader(payload), func(chunkReader io.Reader, index int) error {
+ _, err := io.Copy(io.Discard, chunkReader)
+ return err
+ }, func(chunk ChunkMeta) error {
+ seen = append(seen, chunk)
+ return nil
+ })
+ if err != nil {
+ t.Fatalf("StreamChunkerWithCallback() returned error: %v", err)
+ }
+ if len(chunks) != 1 || len(seen) != 1 {
+ t.Fatalf("chunks = %d, callback chunks = %d, want 1 and 1", len(chunks), len(seen))
+ }
+ if seen[0] != chunks[0] {
+ t.Fatalf("callback chunk = %+v, want %+v", seen[0], chunks[0])
+ }
+}
+
+func TestStreamChunkerWithCallbackPropagatesCallbackError(t *testing.T) {
+ wantErr := errors.New("save state failed")
+ _, err := StreamChunkerWithCallback(bytes.NewReader([]byte("payload")), func(chunkReader io.Reader, index int) error {
+ _, err := io.Copy(io.Discard, chunkReader)
+ return err
+ }, func(ChunkMeta) error {
+ return wantErr
+ })
+ if !errors.Is(err, wantErr) {
+ t.Fatalf("StreamChunkerWithCallback() error = %v, want wrapped %v", err, wantErr)
+ }
+}
+
+func TestStreamChunkerWithSizeUsesCustomChunkSize(t *testing.T) {
+ chunks, err := StreamChunkerWithSize(bytes.NewReader([]byte("abcdefghij")), 4, func(chunkReader io.Reader, index int) error {
+ _, err := io.Copy(io.Discard, chunkReader)
+ return err
+ }, nil)
+ if err != nil {
+ t.Fatalf("StreamChunkerWithSize() returned error: %v", err)
+ }
+
+ wantSizes := []int64{4, 4, 2}
+ if len(chunks) != len(wantSizes) {
+ t.Fatalf("chunk count = %d, want %d", len(chunks), len(wantSizes))
+ }
+ for i, chunk := range chunks {
+ if chunk.PlainSize != wantSizes[i] {
+ t.Fatalf("chunk %d size = %d, want %d", i, chunk.PlainSize, wantSizes[i])
+ }
+ }
+}
+
+func TestStreamChunkerWithSizeRejectsInvalidChunkSize(t *testing.T) {
+ _, err := StreamChunkerWithSize(bytes.NewReader([]byte("payload")), 0, func(io.Reader, int) error {
+ return nil
+ }, nil)
+ if err == nil {
+ t.Fatalf("StreamChunkerWithSize() succeeded with zero chunk size")
+ }
+ if !strings.Contains(err.Error(), "chunk size must be positive") {
+ t.Fatalf("StreamChunkerWithSize() error = %q, want chunk size error", err)
+ }
+}
+
func TestStreamChunkerEmptyInput(t *testing.T) {
called := 0
chunks, err := StreamChunker(bytes.NewReader(nil), func(chunkReader io.Reader, index int) error {
diff --git a/internal/core/manifest.go b/internal/core/manifest.go
index 14de34c..5c4c991 100644
--- a/internal/core/manifest.go
+++ b/internal/core/manifest.go
@@ -12,14 +12,14 @@ import (
)
type Manifest struct {
- Version string `json:"version"`
- UUID string `json:"uuid"`
- OriginalName string `json:"original_name"`
- OriginalSize int64 `json:"original_size"`
- CreatedAt string `json:"created_at"`
- Recipient string `json:"recipient"`
- PayloadEncryptedHash string `json:"payload_encrypted_hash"`
- Chunks []ChunkMeta `json:"chunks"`
+ Version string `json:"version"`
+ UUID string `json:"uuid"`
+ OriginalName string `json:"original_name"`
+ OriginalSize int64 `json:"original_size"`
+ CreatedAt string `json:"created_at"`
+ Recipient string `json:"recipient"`
+ PayloadSHA256 string `json:"payload_sha256"`
+ Chunks []ChunkMeta `json:"chunks"`
}
type Locator struct {
@@ -33,14 +33,14 @@ func GenerateUUID() string {
func CreateManifest(id string, filename string, size int64, recipient string, payloadHash string, chunks []ChunkMeta) Manifest {
return Manifest{
- Version: "1.0",
- UUID: id,
- OriginalName: filename,
- OriginalSize: size,
- CreatedAt: time.Now().UTC().Format(time.RFC3339),
- Recipient: recipient,
- PayloadEncryptedHash: payloadHash,
- Chunks: chunks,
+ Version: "1.0",
+ UUID: id,
+ OriginalName: filename,
+ OriginalSize: size,
+ CreatedAt: time.Now().UTC().Format(time.RFC3339),
+ Recipient: recipient,
+ PayloadSHA256: payloadHash,
+ Chunks: chunks,
}
}
diff --git a/internal/core/manifest_test.go b/internal/core/manifest_test.go
index 88fe486..7fc53de 100644
--- a/internal/core/manifest_test.go
+++ b/internal/core/manifest_test.go
@@ -12,7 +12,7 @@ import (
)
func TestCreateManifestAndLocatorVersions(t *testing.T) {
- chunks := []ChunkMeta{{Index: 0, Name: "chunk_00000", Size: 10, SHA256: "abc"}}
+ chunks := []ChunkMeta{{Index: 0, Name: "chunk_00000", PlainSize: 10, PlainSHA256: "plain", EncryptedSize: 12, EncryptedSHA256: "encrypted"}}
manifest := CreateManifest("backup-id", "secret.tar", 123, "age1example", "payload-hash", chunks)
if manifest.Version != "1.0" {
@@ -78,7 +78,7 @@ func TestEncryptManifestCanBeDecrypted(t *testing.T) {
if err := json.Unmarshal(plain, &got); err != nil {
t.Fatalf("decrypted manifest is not valid JSON: %v", err)
}
- if got.UUID != manifest.UUID || got.OriginalName != manifest.OriginalName || got.PayloadEncryptedHash != manifest.PayloadEncryptedHash {
+ if got.UUID != manifest.UUID || got.OriginalName != manifest.OriginalName || got.PayloadSHA256 != manifest.PayloadSHA256 {
t.Fatalf("decrypted manifest = %+v, want %+v", got, manifest)
}
}
diff --git a/internal/flow/flow_test.go b/internal/flow/flow_test.go
index ed5f8e2..8916320 100644
--- a/internal/flow/flow_test.go
+++ b/internal/flow/flow_test.go
@@ -12,6 +12,7 @@ import (
"testing"
appcrypto "gloak/internal/crypto"
+ "gloak/internal/storage"
)
func TestUploadVerifyRestoreRoundTrip(t *testing.T) {
@@ -85,13 +86,240 @@ func TestVerifyRejectsCorruptChunk(t *testing.T) {
}
}
+func TestRunUploadWithStateDeletesStateAfterSuccess(t *testing.T) {
+ pubKey, _, _, err := appcrypto.GenerateKey()
+ if err != nil {
+ t.Fatalf("GenerateKey() returned error: %v", err)
+ }
+
+ sourcePath := filepath.Join(t.TempDir(), "secret.txt")
+ if err := os.WriteFile(sourcePath, []byte("payload"), 0600); err != nil {
+ t.Fatalf("failed to write source file: %v", err)
+ }
+
+ backend := newMemoryBackend()
+ stateDir := t.TempDir()
+ backupID := "7f91c6c7-7a0b-44aa-ae23-997b60e4e998"
+ if err := runUploadWithState(sourcePath, pubKey, backend, backupID, "memory:backups", stateDir); err != nil {
+ t.Fatalf("runUploadWithState() returned error: %v", err)
+ }
+
+ if _, err := os.Stat(uploadStatePath(stateDir, backupID)); !os.IsNotExist(err) {
+ t.Fatalf("upload state should be deleted after success, stat error: %v", err)
+ }
+}
+
+func TestRunUploadWithStateKeepsUploadedChunksAfterFailure(t *testing.T) {
+ pubKey, _, _, err := appcrypto.GenerateKey()
+ if err != nil {
+ t.Fatalf("GenerateKey() returned error: %v", err)
+ }
+
+ sourcePath := filepath.Join(t.TempDir(), "secret.txt")
+ if err := os.WriteFile(sourcePath, []byte("payload"), 0600); err != nil {
+ t.Fatalf("failed to write source file: %v", err)
+ }
+
+ backupID := "7f91c6c7-7a0b-44aa-ae23-997b60e4e998"
+ backend := &failUploadBackend{
+ memoryBackend: newMemoryBackend(),
+ failPath: backupID + "/manifest.age",
+ }
+ stateDir := t.TempDir()
+ err = runUploadWithState(sourcePath, pubKey, backend, backupID, "memory:backups", stateDir)
+ if err == nil {
+ t.Fatalf("runUploadWithState() succeeded despite manifest upload failure")
+ }
+ if !strings.Contains(err.Error(), "failed to upload manifest.age") {
+ t.Fatalf("runUploadWithState() error = %q, want manifest upload failure", err)
+ }
+
+ state, err := loadUploadState(stateDir, backupID)
+ if err != nil {
+ t.Fatalf("loadUploadState() returned error: %v", err)
+ }
+ if state.Remote != "memory:backups" || state.Recipient != pubKey {
+ t.Fatalf("state remote/recipient = %q/%q, want memory:backups/%q", state.Remote, state.Recipient, pubKey)
+ }
+ absSourcePath, err := filepath.Abs(sourcePath)
+ if err != nil {
+ t.Fatalf("failed to resolve source path: %v", err)
+ }
+ if state.SourcePath != absSourcePath {
+ t.Fatalf("state SourcePath = %q, want %q", state.SourcePath, absSourcePath)
+ }
+ if len(state.UploadedChunks) != 1 {
+ t.Fatalf("uploaded chunks in state = %d, want 1", len(state.UploadedChunks))
+ }
+ if state.UploadedChunks[0].Name != "chunk_00000" || state.UploadedChunks[0].PlainSize == 0 || state.UploadedChunks[0].PlainSHA256 == "" || state.UploadedChunks[0].EncryptedSize == 0 || state.UploadedChunks[0].EncryptedSHA256 == "" {
+ t.Fatalf("uploaded chunk metadata not recorded correctly: %+v", state.UploadedChunks[0])
+ }
+}
+
+func TestRunUploadWithStateStopsAtMidUploadFailure(t *testing.T) {
+ pubKey, _, _, err := appcrypto.GenerateKey()
+ if err != nil {
+ t.Fatalf("GenerateKey() returned error: %v", err)
+ }
+
+ sourcePath := filepath.Join(t.TempDir(), "secret.txt")
+ if err := os.WriteFile(sourcePath, []byte("payload large enough for several encrypted chunks"), 0600); err != nil {
+ t.Fatalf("failed to write source file: %v", err)
+ }
+
+ backupID := "7f91c6c7-7a0b-44aa-ae23-997b60e4e998"
+ backend := &failUploadBackend{
+ memoryBackend: newMemoryBackend(),
+ failPath: backupID + "/chunk_00001",
+ }
+ stateDir := t.TempDir()
+ err = runUploadWithStateAndChunkSize(sourcePath, pubKey, backend, backupID, "memory:backups", stateDir, 8)
+ if err == nil {
+ t.Fatalf("runUploadWithStateAndChunkSize() succeeded despite chunk upload failure")
+ }
+ if !strings.Contains(err.Error(), "chunk 1 failed to process") {
+ t.Fatalf("runUploadWithStateAndChunkSize() error = %q, want chunk 1 failure", err)
+ }
+
+ state, err := loadUploadState(stateDir, backupID)
+ if err != nil {
+ t.Fatalf("loadUploadState() returned error: %v", err)
+ }
+ if state.ChunkSize != 8 {
+ t.Fatalf("state ChunkSize = %d, want 8", state.ChunkSize)
+ }
+ if len(state.UploadedChunks) != 1 {
+ t.Fatalf("uploaded chunks in state = %d, want 1", len(state.UploadedChunks))
+ }
+ if state.UploadedChunks[0].Name != "chunk_00000" {
+ t.Fatalf("recorded chunk = %q, want chunk_00000", state.UploadedChunks[0].Name)
+ }
+
+ if !backend.has(backupID + "/chunk_00000") {
+ t.Fatalf("chunk_00000 should exist after first chunk upload")
+ }
+ for _, path := range []string{backupID + "/chunk_00001", backupID + "/manifest.age", backupID + "/locator.json"} {
+ if backend.has(path) {
+ t.Fatalf("%s should not exist after mid-upload failure", path)
+ }
+ }
+}
+
+func TestRunResumeCompletesInterruptedUpload(t *testing.T) {
+ pubKey, _, identityFile, err := appcrypto.GenerateKey()
+ if err != nil {
+ t.Fatalf("GenerateKey() returned error: %v", err)
+ }
+ identityPath := writeFlowIdentityFile(t, identityFile)
+
+ sourcePath := filepath.Join(t.TempDir(), "secret.txt")
+ wantPayload := []byte("payload large enough for several encrypted chunks")
+ if err := os.WriteFile(sourcePath, wantPayload, 0600); err != nil {
+ t.Fatalf("failed to write source file: %v", err)
+ }
+
+ backupID := "7f91c6c7-7a0b-44aa-ae23-997b60e4e998"
+ memory := newMemoryBackend()
+ failing := &failUploadBackend{
+ memoryBackend: memory,
+ failPath: backupID + "/chunk_00001",
+ }
+ stateDir := t.TempDir()
+ if err := runUploadWithStateAndChunkSize(sourcePath, pubKey, failing, backupID, "memory:backups", stateDir, 8); err == nil {
+ t.Fatalf("runUploadWithStateAndChunkSize() succeeded despite chunk upload failure")
+ }
+
+ if err := runResume(backupID, func(string) storage.Backend { return memory }, stateDir); err != nil {
+ t.Fatalf("runResume() returned error: %v", err)
+ }
+ if _, err := os.Stat(uploadStatePath(stateDir, backupID)); !os.IsNotExist(err) {
+ t.Fatalf("upload state should be deleted after resume, stat error: %v", err)
+ }
+ if memory.uploadCount(backupID+"/chunk_00000") != 1 {
+ t.Fatalf("chunk_00000 upload count = %d, want 1", memory.uploadCount(backupID+"/chunk_00000"))
+ }
+ if memory.uploadCount(backupID+"/chunk_00001") != 1 {
+ t.Fatalf("chunk_00001 upload count = %d, want 1", memory.uploadCount(backupID+"/chunk_00001"))
+ }
+
+ if err := runVerify(backupID, identityPath, memory); err != nil {
+ t.Fatalf("runVerify() returned error after resume: %v", err)
+ }
+
+ outputDir := t.TempDir()
+ if err := runRestore(backupID, identityPath, memory, outputDir); err != nil {
+ t.Fatalf("runRestore() returned error after resume: %v", err)
+ }
+ gotPayload, err := os.ReadFile(filepath.Join(outputDir, "secret.txt"))
+ if err != nil {
+ t.Fatalf("failed to read restored file: %v", err)
+ }
+ if !bytes.Equal(gotPayload, wantPayload) {
+ t.Fatalf("restored payload = %q, want %q", gotPayload, wantPayload)
+ }
+}
+
+func TestRunResumeRejectsStateChunkPlainHashMismatch(t *testing.T) {
+ pubKey, _, _, err := appcrypto.GenerateKey()
+ if err != nil {
+ t.Fatalf("GenerateKey() returned error: %v", err)
+ }
+
+ sourcePath := filepath.Join(t.TempDir(), "secret.txt")
+ if err := os.WriteFile(sourcePath, []byte("payload large enough for several encrypted chunks"), 0600); err != nil {
+ t.Fatalf("failed to write source file: %v", err)
+ }
+
+ backupID := "7f91c6c7-7a0b-44aa-ae23-997b60e4e998"
+ memory := newMemoryBackend()
+ failing := &failUploadBackend{
+ memoryBackend: memory,
+ failPath: backupID + "/chunk_00001",
+ }
+ stateDir := t.TempDir()
+ if err := runUploadWithStateAndChunkSize(sourcePath, pubKey, failing, backupID, "memory:backups", stateDir, 8); err == nil {
+ t.Fatalf("runUploadWithStateAndChunkSize() succeeded despite chunk upload failure")
+ }
+
+ state, err := loadUploadState(stateDir, backupID)
+ if err != nil {
+ t.Fatalf("loadUploadState() returned error: %v", err)
+ }
+ state.UploadedChunks[0].PlainSHA256 = "bad-hash"
+ if err := saveUploadState(stateDir, state); err != nil {
+ t.Fatalf("saveUploadState() returned error: %v", err)
+ }
+
+ err = runResume(backupID, func(string) storage.Backend { return memory }, stateDir)
+ if err == nil {
+ t.Fatalf("runResume() succeeded with mismatched state chunk hash")
+ }
+ if !strings.Contains(err.Error(), "does not match source file") {
+ t.Fatalf("runResume() error = %q, want source mismatch error", err)
+ }
+}
+
type memoryBackend struct {
- mu sync.Mutex
- files map[string][]byte
+ mu sync.Mutex
+ files map[string][]byte
+ uploads map[string]int
+}
+
+type failUploadBackend struct {
+ *memoryBackend
+ failPath string
+}
+
+func (f *failUploadBackend) Upload(remotePath string, in io.Reader) error {
+ if remotePath == f.failPath {
+ _, _ = io.Copy(io.Discard, in)
+ return fmt.Errorf("forced upload failure for %s", remotePath)
+ }
+ return f.memoryBackend.Upload(remotePath, in)
}
func newMemoryBackend() *memoryBackend {
- return &memoryBackend{files: make(map[string][]byte)}
+ return &memoryBackend{files: make(map[string][]byte), uploads: make(map[string]int)}
}
func (m *memoryBackend) Upload(remotePath string, in io.Reader) error {
@@ -100,6 +328,9 @@ func (m *memoryBackend) Upload(remotePath string, in io.Reader) error {
return err
}
m.put(remotePath, data)
+ m.mu.Lock()
+ m.uploads[remotePath]++
+ m.mu.Unlock()
return nil
}
@@ -138,6 +369,12 @@ func (m *memoryBackend) put(remotePath string, data []byte) {
m.files[remotePath] = append([]byte(nil), data...)
}
+func (m *memoryBackend) uploadCount(remotePath string) int {
+ m.mu.Lock()
+ defer m.mu.Unlock()
+ return m.uploads[remotePath]
+}
+
func writeFlowIdentityFile(t *testing.T, content string) string {
t.Helper()
path := filepath.Join(t.TempDir(), "identity.txt")
diff --git a/internal/flow/restore.go b/internal/flow/restore.go
index 9c4055d..8c7c962 100644
--- a/internal/flow/restore.go
+++ b/internal/flow/restore.go
@@ -68,47 +68,53 @@ func runRestore(backupID string, identityPath string, backend storage.Backend, o
WithRemoveWhenDone(false).
Start()
- pr, pw := io.Pipe()
-
- go func() {
- defer pw.Close()
- payloadHasher := sha256.New()
-
- for _, chunk := range manifest.Chunks {
-
- chunkHasher := sha256.New()
- multiWriter := io.MultiWriter(pw, chunkHasher, payloadHasher)
-
- err := backend.Download(fmt.Sprintf("%s/%s", backupID, chunk.Name), multiWriter)
- if err != nil {
- pw.CloseWithError(fmt.Errorf("failed to download chunk %s: %w", chunk.Name, err))
- return
- }
-
- if hex.EncodeToString(chunkHasher.Sum(nil)) != chunk.SHA256 {
- pw.CloseWithError(fmt.Errorf("chunk %s corrupted/tampered! Hash mismatch", chunk.Name))
- return
- }
+ payloadHasher := sha256.New()
+ for _, chunk := range manifest.Chunks {
+ var encryptedChunk bytes.Buffer
+ if err := backend.Download(fmt.Sprintf("%s/%s", backupID, chunk.Name), &encryptedChunk); err != nil {
+ p.Stop()
+ return fmt.Errorf("failed to download chunk %s: %w", chunk.Name, err)
}
- if hex.EncodeToString(payloadHasher.Sum(nil)) != manifest.PayloadEncryptedHash {
- pw.CloseWithError(fmt.Errorf("fatal: overall payload hash does not match manifest"))
- return
+ encryptedBytes := encryptedChunk.Bytes()
+ if int64(len(encryptedBytes)) != chunk.EncryptedSize {
+ p.Stop()
+ return fmt.Errorf("chunk %s corrupted/tampered! Size mismatch", chunk.Name)
+ }
+ if hashBytes(encryptedBytes) != chunk.EncryptedSHA256 {
+ p.Stop()
+ return fmt.Errorf("chunk %s corrupted/tampered! Hash mismatch", chunk.Name)
}
- }()
- decReader, err := crypto.DecryptReader(identityPath, pr)
- if err != nil {
- p.Stop()
- return fmt.Errorf("failed to initialize decryption stream: %w", err)
+ decReader, err := crypto.DecryptReader(identityPath, bytes.NewReader(encryptedBytes))
+ if err != nil {
+ p.Stop()
+ return fmt.Errorf("failed to decrypt chunk %s: %w", chunk.Name, err)
+ }
+ plainBytes, err := io.ReadAll(decReader)
+ if err != nil {
+ p.Stop()
+ return fmt.Errorf("failed to read decrypted chunk %s: %w", chunk.Name, err)
+ }
+ if int64(len(plainBytes)) != chunk.PlainSize {
+ p.Stop()
+ return fmt.Errorf("chunk %s corrupted/tampered! Plain size mismatch", chunk.Name)
+ }
+ if hashBytes(plainBytes) != chunk.PlainSHA256 {
+ p.Stop()
+ return fmt.Errorf("chunk %s corrupted/tampered! Plain hash mismatch", chunk.Name)
+ }
+ if _, err := outFile.Write(plainBytes); err != nil {
+ p.Stop()
+ return fmt.Errorf("failed to write restored chunk %s: %w", chunk.Name, err)
+ }
+ payloadHasher.Write(plainBytes)
+ p.Add(len(plainBytes))
}
- progReader := &RestoreProgressReader{Reader: decReader, pb: p}
-
- _, err = io.Copy(outFile, progReader)
- if err != nil {
+ if hex.EncodeToString(payloadHasher.Sum(nil)) != manifest.PayloadSHA256 {
p.Stop()
- return fmt.Errorf("decryption streaming failed: %w", err)
+ return fmt.Errorf("fatal: overall payload hash does not match manifest")
}
p.Stop()
diff --git a/internal/flow/resume.go b/internal/flow/resume.go
new file mode 100644
index 0000000..fdb6cbe
--- /dev/null
+++ b/internal/flow/resume.go
@@ -0,0 +1,110 @@
+package flow
+
+import (
+ "bytes"
+ "fmt"
+ "os"
+ "strings"
+
+ "github.com/pterm/pterm"
+
+ "gloak/internal/core"
+ "gloak/internal/storage"
+)
+
+func RunResumeValidation(backupID string) (UploadState, error) {
+ return runResumeValidation(backupID, defaultUploadStateDir())
+}
+
+func RunResume(backupID string) error {
+ return runResume(backupID, func(remote string) storage.Backend {
+ return storage.NewRcloneBackend(remote)
+ }, defaultUploadStateDir())
+}
+
+func runResume(backupID string, newBackend func(string) storage.Backend, stateDir string) error {
+ state, err := runResumeValidation(backupID, stateDir)
+ if err != nil {
+ return err
+ }
+
+ file, err := os.Open(state.SourcePath)
+ if err != nil {
+ return fmt.Errorf("failed to open source file from upload state: %w", err)
+ }
+ defer file.Close()
+
+ pterm.Success.Printf("Found upload state: %s\n", state.BackupID)
+ pterm.DefaultBasicText.Printf("Source file : %s\n", state.SourcePath)
+ pterm.DefaultBasicText.Printf("Destination : %s\n", state.Remote)
+ pterm.DefaultBasicText.Printf("Uploaded chunks : %d\n", len(state.UploadedChunks))
+
+ backend := newBackend(state.Remote)
+ p, _ := pterm.DefaultProgressbar.
+ WithTotal(int(state.OriginalSize)).
+ WithTitle("Resuming upload...").
+ WithRemoveWhenDone(false).
+ Start()
+
+ chunksMeta, payloadHash, err := uploadPlaintextChunks(file, state.Recipient, backend, backupID, &state, stateDir, state.ChunkSize, func(chunk core.ChunkMeta) {
+ pterm.Info.Printf("Skipping %s\n", chunk.Name)
+ }, func(n int) {
+ p.Add(n)
+ })
+ if err != nil {
+ p.Stop()
+ return fmt.Errorf("resume upload failed: %w", err)
+ }
+ p.Stop()
+
+ pterm.Info.Println("Building and encrypting manifest...")
+ manifest := core.CreateManifest(state.BackupID, state.OriginalName, state.OriginalSize, state.Recipient, payloadHash, chunksMeta)
+ encryptedManifestBytes, err := core.EncryptManifest(state.Recipient, manifest)
+ if err != nil {
+ return fmt.Errorf("failed to encrypt manifest: %w", err)
+ }
+ if err := backend.Upload(fmt.Sprintf("%s/manifest.age", state.BackupID), bytes.NewReader(encryptedManifestBytes)); err != nil {
+ return fmt.Errorf("failed to upload manifest.age: %w", err)
+ }
+
+ locator := core.CreateLocator(state.BackupID)
+ locatorBytes, _ := core.SerializeLocator(locator)
+ if err := backend.Upload(fmt.Sprintf("%s/locator.json", state.BackupID), bytes.NewReader(locatorBytes)); err != nil {
+ return fmt.Errorf("failed to upload locator.json: %w", err)
+ }
+ if err := deleteUploadState(stateDir, state.BackupID); err != nil {
+ return fmt.Errorf("resume completed but failed to delete upload state: %w", err)
+ }
+
+ pterm.Success.Println("Resume completed successfully!")
+ return nil
+}
+
+func runResumeValidation(backupID string, stateDir string) (UploadState, error) {
+ state, err := loadUploadState(stateDir, backupID)
+ if err != nil {
+ return UploadState{}, err
+ }
+
+ info, err := os.Stat(state.SourcePath)
+ if err != nil {
+ return UploadState{}, fmt.Errorf("failed to stat source file from upload state: %w", err)
+ }
+ if info.IsDir() {
+ return UploadState{}, fmt.Errorf("source path from upload state is a directory: %s", state.SourcePath)
+ }
+ if info.Size() != state.OriginalSize {
+ return UploadState{}, fmt.Errorf("source file size changed: got %d, want %d", info.Size(), state.OriginalSize)
+ }
+ if strings.TrimSpace(state.Recipient) == "" {
+ return UploadState{}, fmt.Errorf("upload state recipient is empty")
+ }
+ if state.Remote == "" {
+ return UploadState{}, fmt.Errorf("upload state remote is empty")
+ }
+ if state.ChunkSize <= 0 {
+ return UploadState{}, fmt.Errorf("upload state chunk size must be positive")
+ }
+
+ return state, nil
+}
diff --git a/internal/flow/resume_test.go b/internal/flow/resume_test.go
new file mode 100644
index 0000000..bfb5587
--- /dev/null
+++ b/internal/flow/resume_test.go
@@ -0,0 +1,104 @@
+package flow
+
+import (
+ "os"
+ "path/filepath"
+ "strings"
+ "testing"
+
+ "gloak/internal/core"
+)
+
+func TestRunResumeValidationRejectsMissingState(t *testing.T) {
+ _, err := runResumeValidation("7f91c6c7-7a0b-44aa-ae23-997b60e4e998", t.TempDir())
+ if err == nil {
+ t.Fatalf("runResumeValidation() succeeded with missing state")
+ }
+ if !strings.Contains(err.Error(), "failed to read upload state") {
+ t.Fatalf("runResumeValidation() error = %q, want missing state error", err)
+ }
+}
+
+func TestRunResumeValidationRejectsMissingSourceFile(t *testing.T) {
+ stateDir := t.TempDir()
+ backupID := "7f91c6c7-7a0b-44aa-ae23-997b60e4e998"
+ state := NewUploadState(backupID, filepath.Join(t.TempDir(), "missing.txt"), "missing.txt", 7, "memory:backups", "age1recipient")
+ if err := saveUploadState(stateDir, state); err != nil {
+ t.Fatalf("saveUploadState() returned error: %v", err)
+ }
+
+ _, err := runResumeValidation(backupID, stateDir)
+ if err == nil {
+ t.Fatalf("runResumeValidation() succeeded with missing source file")
+ }
+ if !strings.Contains(err.Error(), "failed to stat source file from upload state") {
+ t.Fatalf("runResumeValidation() error = %q, want source stat error", err)
+ }
+}
+
+func TestRunResumeValidationRejectsChangedSourceSize(t *testing.T) {
+ stateDir := t.TempDir()
+ backupID := "7f91c6c7-7a0b-44aa-ae23-997b60e4e998"
+ sourcePath := writeResumeSource(t, []byte("current payload"))
+ state := NewUploadState(backupID, sourcePath, filepath.Base(sourcePath), 5, "memory:backups", "age1recipient")
+ if err := saveUploadState(stateDir, state); err != nil {
+ t.Fatalf("saveUploadState() returned error: %v", err)
+ }
+
+ _, err := runResumeValidation(backupID, stateDir)
+ if err == nil {
+ t.Fatalf("runResumeValidation() succeeded with changed source size")
+ }
+ if !strings.Contains(err.Error(), "source file size changed") {
+ t.Fatalf("runResumeValidation() error = %q, want source size error", err)
+ }
+}
+
+func TestRunResumeValidationRejectsEmptyRecipient(t *testing.T) {
+ stateDir := t.TempDir()
+ backupID := "7f91c6c7-7a0b-44aa-ae23-997b60e4e998"
+ sourcePath := writeResumeSource(t, []byte("payload"))
+ state := NewUploadState(backupID, sourcePath, filepath.Base(sourcePath), 7, "memory:backups", " ")
+ if err := saveUploadState(stateDir, state); err != nil {
+ t.Fatalf("saveUploadState() returned error: %v", err)
+ }
+
+ _, err := runResumeValidation(backupID, stateDir)
+ if err == nil {
+ t.Fatalf("runResumeValidation() succeeded with empty recipient")
+ }
+ if !strings.Contains(err.Error(), "upload state recipient is empty") {
+ t.Fatalf("runResumeValidation() error = %q, want empty recipient error", err)
+ }
+}
+
+func TestRunResumeValidationAcceptsValidState(t *testing.T) {
+ stateDir := t.TempDir()
+ backupID := "7f91c6c7-7a0b-44aa-ae23-997b60e4e998"
+ sourcePath := writeResumeSource(t, []byte("payload"))
+ state := NewUploadState(backupID, sourcePath, filepath.Base(sourcePath), 7, "memory:backups", "age1recipient")
+ state.UploadedChunks = []core.ChunkMeta{{Index: 0, Name: "chunk_00000", PlainSize: 7, PlainSHA256: hashBytes([]byte("payload")), EncryptedSize: 12, EncryptedSHA256: "encrypted"}}
+ if err := saveUploadState(stateDir, state); err != nil {
+ t.Fatalf("saveUploadState() returned error: %v", err)
+ }
+
+ got, err := runResumeValidation(backupID, stateDir)
+ if err != nil {
+ t.Fatalf("runResumeValidation() returned error: %v", err)
+ }
+ if got.BackupID != backupID || got.SourcePath != sourcePath || got.Remote != "memory:backups" {
+ t.Fatalf("validated state = %+v, want backup/source/remote preserved", got)
+ }
+ if len(got.UploadedChunks) != 1 || got.UploadedChunks[0].Name != "chunk_00000" {
+ t.Fatalf("validated uploaded chunks = %+v, want chunk_00000", got.UploadedChunks)
+ }
+}
+
+func writeResumeSource(t *testing.T, content []byte) string {
+ t.Helper()
+ path := filepath.Join(t.TempDir(), "source.txt")
+ if err := os.WriteFile(path, content, 0600); err != nil {
+ t.Fatalf("failed to write source file: %v", err)
+ }
+ return path
+}
diff --git a/internal/flow/upload.go b/internal/flow/upload.go
index 6ed18d7..065d377 100644
--- a/internal/flow/upload.go
+++ b/internal/flow/upload.go
@@ -17,10 +17,18 @@ import (
)
func RunUpload(filePath string, recipientKey string, remote string) error {
- return runUpload(filePath, recipientKey, storage.NewRcloneBackend(remote), core.GenerateUUID())
+ return runUploadWithStateAndChunkSize(filePath, recipientKey, storage.NewRcloneBackend(remote), core.GenerateUUID(), remote, defaultUploadStateDir(), core.ChunkSize)
}
func runUpload(filePath string, recipientKey string, backend storage.Backend, backupID string) error {
+ return runUploadWithState(filePath, recipientKey, backend, backupID, "", "")
+}
+
+func runUploadWithState(filePath string, recipientKey string, backend storage.Backend, backupID string, remote string, stateDir string) error {
+ return runUploadWithStateAndChunkSize(filePath, recipientKey, backend, backupID, remote, stateDir, core.ChunkSize)
+}
+
+func runUploadWithStateAndChunkSize(filePath string, recipientKey string, backend storage.Backend, backupID string, remote string, stateDir string, chunkSize int64) error {
file, err := os.Open(filePath)
if err != nil {
return fmt.Errorf("failed to open source file: %w", err)
@@ -33,6 +41,17 @@ func runUpload(filePath string, recipientKey string, backend storage.Backend, ba
}
originalSize := stat.Size()
originalName := filepath.Base(filePath)
+ state := UploadState{}
+ if stateDir != "" {
+ absPath, err := filepath.Abs(filePath)
+ if err != nil {
+ return fmt.Errorf("failed to resolve source path: %w", err)
+ }
+ state = newUploadStateWithChunkSize(backupID, absPath, originalName, originalSize, remote, recipientKey, chunkSize)
+ if err := saveUploadState(stateDir, state); err != nil {
+ return err
+ }
+ }
pterm.Success.Printf("Backup ID: %s\n", backupID)
@@ -42,34 +61,8 @@ func runUpload(filePath string, recipientKey string, backend storage.Backend, ba
WithRemoveWhenDone(false).
Start()
- progressReader := &ProgressReader{Reader: file, pb: p}
-
- pr, pw := io.Pipe()
-
- go func() {
- defer pw.Close()
-
- encWriter, err := crypto.EncryptWriter(recipientKey, pw)
- if err != nil {
- pw.CloseWithError(fmt.Errorf("failed to initialize age encryptor: %w", err))
- return
- }
- defer encWriter.Close()
-
- _, err = io.Copy(encWriter, progressReader)
- if err != nil {
- pw.CloseWithError(fmt.Errorf("failed to stream data: %w", err))
- }
- }()
-
- overallHasher := sha256.New()
- tee := io.TeeReader(pr, overallHasher)
-
- chunksMeta, err := core.StreamChunker(tee, func(chunkReader io.Reader, index int) error {
- chunkName := fmt.Sprintf("chunk_%05d", index)
- remotePath := fmt.Sprintf("%s/%s", backupID, chunkName)
-
- return backend.Upload(remotePath, chunkReader)
+ chunksMeta, payloadHash, err := uploadPlaintextChunks(file, recipientKey, backend, backupID, &state, stateDir, chunkSize, nil, func(n int) {
+ p.Add(n)
})
if err != nil {
@@ -78,10 +71,9 @@ func runUpload(filePath string, recipientKey string, backend storage.Backend, ba
}
p.Stop()
- payloadEncryptedHash := hex.EncodeToString(overallHasher.Sum(nil))
pterm.Info.Println("Building and encrypting manifest...")
- manifest := core.CreateManifest(backupID, originalName, originalSize, recipientKey, payloadEncryptedHash, chunksMeta)
+ manifest := core.CreateManifest(backupID, originalName, originalSize, recipientKey, payloadHash, chunksMeta)
encryptedManifestBytes, err := core.EncryptManifest(recipientKey, manifest)
if err != nil {
@@ -99,12 +91,115 @@ func runUpload(filePath string, recipientKey string, backend storage.Backend, ba
if err != nil {
return fmt.Errorf("failed to upload locator.json: %w", err)
}
+ if stateDir != "" {
+ if err := deleteUploadState(stateDir, backupID); err != nil {
+ return fmt.Errorf("backup completed but failed to delete upload state: %w", err)
+ }
+ }
pterm.Success.Println("Backup completed successfully!")
pterm.DefaultBasicText.Printf("Save this UUID for restoration: %s\n", backupID)
return nil
}
+func uploadPlaintextChunks(file *os.File, recipientKey string, backend storage.Backend, backupID string, state *UploadState, stateDir string, chunkSize int64, onSkip func(core.ChunkMeta), onProgress func(int)) ([]core.ChunkMeta, string, error) {
+ if chunkSize <= 0 {
+ return nil, "", fmt.Errorf("chunk size must be positive")
+ }
+
+ existingChunks := make(map[int]core.ChunkMeta, len(state.UploadedChunks))
+ for _, chunk := range state.UploadedChunks {
+ existingChunks[chunk.Index] = chunk
+ }
+
+ payloadHasher := sha256.New()
+ chunksMeta := make([]core.ChunkMeta, 0, len(state.UploadedChunks))
+ buffer := make([]byte, int(chunkSize))
+
+ for index := 0; ; index++ {
+ n, readErr := io.ReadFull(file, buffer)
+ if readErr == io.EOF {
+ break
+ }
+ if readErr != nil && readErr != io.ErrUnexpectedEOF {
+ return nil, "", fmt.Errorf("failed to read source chunk %d: %w", index, readErr)
+ }
+
+ plainChunk := append([]byte(nil), buffer[:n]...)
+ payloadHasher.Write(plainChunk)
+ plainHash := hashBytes(plainChunk)
+ chunkName := fmt.Sprintf("chunk_%05d", index)
+
+ if existing, ok := existingChunks[index]; ok {
+ if existing.Name != chunkName {
+ return nil, "", fmt.Errorf("upload state chunk %d name mismatch: got %s, want %s", index, existing.Name, chunkName)
+ }
+ if existing.PlainSize != int64(n) || existing.PlainSHA256 != plainHash {
+ return nil, "", fmt.Errorf("upload state chunk %d does not match source file", index)
+ }
+ chunksMeta = append(chunksMeta, existing)
+ if onSkip != nil {
+ onSkip(existing)
+ }
+ } else {
+ encryptedChunk, err := encryptChunk(recipientKey, plainChunk)
+ if err != nil {
+ return nil, "", fmt.Errorf("failed to encrypt chunk %d: %w", index, err)
+ }
+
+ chunk := core.ChunkMeta{
+ Index: index,
+ Name: chunkName,
+ PlainSize: int64(n),
+ PlainSHA256: plainHash,
+ EncryptedSize: int64(len(encryptedChunk)),
+ EncryptedSHA256: hashBytes(encryptedChunk),
+ }
+
+ if err := backend.Upload(fmt.Sprintf("%s/%s", backupID, chunkName), bytes.NewReader(encryptedChunk)); err != nil {
+ return nil, "", fmt.Errorf("chunk %d failed to process: %w", index, err)
+ }
+ chunksMeta = append(chunksMeta, chunk)
+ if stateDir != "" {
+ state.UploadedChunks = append(state.UploadedChunks, chunk)
+ if err := saveUploadState(stateDir, *state); err != nil {
+ return nil, "", fmt.Errorf("chunk %d callback failed: %w", index, err)
+ }
+ }
+ }
+
+ if onProgress != nil {
+ onProgress(n)
+ }
+ if readErr == io.ErrUnexpectedEOF {
+ break
+ }
+ }
+
+ return chunksMeta, hex.EncodeToString(payloadHasher.Sum(nil)), nil
+}
+
+func encryptChunk(recipientKey string, plainChunk []byte) ([]byte, error) {
+ var encrypted bytes.Buffer
+ encWriter, err := crypto.EncryptWriter(recipientKey, &encrypted)
+ if err != nil {
+ return nil, err
+ }
+ if _, err := encWriter.Write(plainChunk); err != nil {
+ _ = encWriter.Close()
+ return nil, err
+ }
+ if err := encWriter.Close(); err != nil {
+ return nil, err
+ }
+ return encrypted.Bytes(), nil
+}
+
+func hashBytes(data []byte) string {
+ hash := sha256.Sum256(data)
+ return hex.EncodeToString(hash[:])
+}
+
type ProgressReader struct {
io.Reader
pb *pterm.ProgressbarPrinter
diff --git a/internal/flow/upload_state.go b/internal/flow/upload_state.go
new file mode 100644
index 0000000..8596e71
--- /dev/null
+++ b/internal/flow/upload_state.go
@@ -0,0 +1,109 @@
+package flow
+
+import (
+ "encoding/json"
+ "fmt"
+ "os"
+ "path/filepath"
+ "time"
+
+ "gloak/internal/core"
+)
+
+const uploadStateVersion = "1.0"
+
+type UploadState struct {
+ Version string `json:"version"`
+ BackupID string `json:"backup_id"`
+ SourcePath string `json:"source_path"`
+ OriginalName string `json:"original_name"`
+ OriginalSize int64 `json:"original_size"`
+ Remote string `json:"remote"`
+ Recipient string `json:"recipient"`
+ ChunkSize int64 `json:"chunk_size"`
+ UploadedChunks []core.ChunkMeta `json:"uploaded_chunks"`
+ CreatedAt string `json:"created_at"`
+ UpdatedAt string `json:"updated_at"`
+}
+
+func NewUploadState(backupID string, sourcePath string, originalName string, originalSize int64, remote string, recipient string) UploadState {
+ return newUploadStateWithChunkSize(backupID, sourcePath, originalName, originalSize, remote, recipient, core.ChunkSize)
+}
+
+func newUploadStateWithChunkSize(backupID string, sourcePath string, originalName string, originalSize int64, remote string, recipient string, chunkSize int64) UploadState {
+ now := uploadStateTimestamp()
+ return UploadState{
+ Version: uploadStateVersion,
+ BackupID: backupID,
+ SourcePath: sourcePath,
+ OriginalName: originalName,
+ OriginalSize: originalSize,
+ Remote: remote,
+ Recipient: recipient,
+ ChunkSize: chunkSize,
+ CreatedAt: now,
+ UpdatedAt: now,
+ }
+}
+
+func uploadStatePath(stateDir string, backupID string) string {
+ return filepath.Join(stateDir, backupID+".json")
+}
+
+func defaultUploadStateDir() string {
+ home, err := os.UserHomeDir()
+ if err != nil {
+ home = "."
+ }
+ return filepath.Join(home, ".gloak", "uploads")
+}
+
+func saveUploadState(stateDir string, state UploadState) error {
+ if err := os.MkdirAll(stateDir, 0700); err != nil {
+ return fmt.Errorf("failed to create upload state directory: %w", err)
+ }
+
+ state.UpdatedAt = uploadStateTimestamp()
+ data, err := json.MarshalIndent(state, "", " ")
+ if err != nil {
+ return fmt.Errorf("failed to encode upload state: %w", err)
+ }
+
+ path := uploadStatePath(stateDir, state.BackupID)
+ if err := os.WriteFile(path, data, 0600); err != nil {
+ return fmt.Errorf("failed to write upload state: %w", err)
+ }
+
+ return nil
+}
+
+func loadUploadState(stateDir string, backupID string) (UploadState, error) {
+ path := uploadStatePath(stateDir, backupID)
+ data, err := os.ReadFile(path)
+ if err != nil {
+ return UploadState{}, fmt.Errorf("failed to read upload state: %w", err)
+ }
+
+ var state UploadState
+ if err := json.Unmarshal(data, &state); err != nil {
+ return UploadState{}, fmt.Errorf("failed to parse upload state: %w", err)
+ }
+ if state.BackupID != backupID {
+ return UploadState{}, fmt.Errorf("upload state backup id mismatch: got %s, want %s", state.BackupID, backupID)
+ }
+
+ return state, nil
+}
+
+func deleteUploadState(stateDir string, backupID string) error {
+ path := uploadStatePath(stateDir, backupID)
+ if err := os.Remove(path); err != nil {
+ return fmt.Errorf("failed to delete upload state: %w", err)
+ }
+
+ return nil
+}
+
+func uploadStateTimestamp() string {
+ return time.Now().UTC().Format(time.RFC3339Nano)
+}
diff --git a/internal/flow/upload_state_test.go b/internal/flow/upload_state_test.go
new file mode 100644
index 0000000..535d176
--- /dev/null
+++ b/internal/flow/upload_state_test.go
@@ -0,0 +1,135 @@
+package flow
+
+import (
+ "os"
+ "path/filepath"
+ "strings"
+ "testing"
+ "time"
+
+ "gloak/internal/core"
+)
+
+func TestNewUploadState(t *testing.T) {
+ state := NewUploadState("backup-id", "/tmp/secret.tar", "secret.tar", 123, "remote:path", "age1recipient")
+
+ if state.Version != "1.0" {
+ t.Fatalf("Version = %q, want 1.0", state.Version)
+ }
+ if state.BackupID != "backup-id" || state.SourcePath != "/tmp/secret.tar" || state.OriginalName != "secret.tar" {
+ t.Fatalf("state identity fields not preserved: %+v", state)
+ }
+ if state.OriginalSize != 123 || state.Remote != "remote:path" || state.Recipient != "age1recipient" {
+ t.Fatalf("state backup fields not preserved: %+v", state)
+ }
+ if state.ChunkSize != core.ChunkSize {
+ t.Fatalf("ChunkSize = %d, want %d", state.ChunkSize, core.ChunkSize)
+ }
+ if state.CreatedAt == "" || state.UpdatedAt == "" {
+ t.Fatalf("timestamps should be set: %+v", state)
+ }
+ if _, err := time.Parse(time.RFC3339Nano, state.CreatedAt); err != nil {
+ t.Fatalf("CreatedAt is not RFC3339Nano: %v", err)
+ }
+}
+
+func TestUploadStateSaveLoadRoundTrip(t *testing.T) {
+ stateDir := t.TempDir()
+ state := NewUploadState("backup-id", "/tmp/secret.tar", "secret.tar", 123, "remote:path", "age1recipient")
+ state.UploadedChunks = []core.ChunkMeta{{Index: 0, Name: "chunk_00000", PlainSize: 10, PlainSHA256: "plain", EncryptedSize: 12, EncryptedSHA256: "encrypted"}}
+
+ if err := saveUploadState(stateDir, state); err != nil {
+ t.Fatalf("saveUploadState() returned error: %v", err)
+ }
+
+ path := uploadStatePath(stateDir, "backup-id")
+ info, err := os.Stat(path)
+ if err != nil {
+ t.Fatalf("state file was not written: %v", err)
+ }
+ if info.Mode().Perm() != 0600 {
+ t.Fatalf("state file permissions = %v, want 0600", info.Mode().Perm())
+ }
+
+ got, err := loadUploadState(stateDir, "backup-id")
+ if err != nil {
+ t.Fatalf("loadUploadState() returned error: %v", err)
+ }
+ if got.BackupID != state.BackupID || got.SourcePath != state.SourcePath || got.Remote != state.Remote {
+ t.Fatalf("loaded state identity fields = %+v, want %+v", got, state)
+ }
+ if len(got.UploadedChunks) != 1 || got.UploadedChunks[0] != state.UploadedChunks[0] {
+ t.Fatalf("loaded chunks = %+v, want %+v", got.UploadedChunks, state.UploadedChunks)
+ }
+ createdAt, err := time.Parse(time.RFC3339Nano, got.CreatedAt)
+ if err != nil {
+ t.Fatalf("loaded CreatedAt is not RFC3339Nano: %v", err)
+ }
+ updatedAt, err := time.Parse(time.RFC3339Nano, got.UpdatedAt)
+ if err != nil {
+ t.Fatalf("loaded UpdatedAt is not RFC3339Nano: %v", err)
+ }
+ if updatedAt.Before(createdAt) {
+ t.Fatalf("UpdatedAt %s is before CreatedAt %s", got.UpdatedAt, got.CreatedAt)
+ }
+}
+
+func TestUploadStatePathUsesBackupID(t *testing.T) {
+ stateDir := t.TempDir()
+ got := uploadStatePath(stateDir, "backup-id")
+ want := filepath.Join(stateDir, "backup-id.json")
+ if got != want {
+ t.Fatalf("uploadStatePath() = %q, want %q", got, want)
+ }
+}
+
+func TestLoadUploadStateRejectsInvalidJSON(t *testing.T) {
+ stateDir := t.TempDir()
+ path := uploadStatePath(stateDir, "backup-id")
+ if err := os.WriteFile(path, []byte("{"), 0600); err != nil {
+ t.Fatalf("failed to write invalid state: %v", err)
+ }
+
+ _, err := loadUploadState(stateDir, "backup-id")
+ if err == nil {
+ t.Fatalf("loadUploadState() succeeded with invalid JSON")
+ }
+ if !strings.Contains(err.Error(), "failed to parse upload state") {
+ t.Fatalf("loadUploadState() error = %q, want parse error", err)
+ }
+}
+
+func TestLoadUploadStateRejectsBackupIDMismatch(t *testing.T) {
+ stateDir := t.TempDir()
+ state := NewUploadState("other-id", "/tmp/secret.tar", "secret.tar", 123, "remote:path", "age1recipient")
+ if err := saveUploadState(stateDir, state); err != nil {
+ t.Fatalf("saveUploadState() returned error: %v", err)
+ }
+
+ if err := os.Rename(uploadStatePath(stateDir, "other-id"), uploadStatePath(stateDir, "backup-id")); err != nil {
+ t.Fatalf("failed to rename state file: %v", err)
+ }
+
+ _, err := loadUploadState(stateDir, "backup-id")
+ if err == nil {
+ t.Fatalf("loadUploadState() succeeded with mismatched backup id")
+ }
+ if !strings.Contains(err.Error(), "upload state backup id mismatch") {
+ t.Fatalf("loadUploadState() error = %q, want backup id mismatch", err)
+ }
+}
+
+func TestDeleteUploadState(t *testing.T) {
+ stateDir := t.TempDir()
+ state := NewUploadState("backup-id", "/tmp/secret.tar", "secret.tar", 123, "remote:path", "age1recipient")
+ if err := saveUploadState(stateDir, state); err != nil {
+ t.Fatalf("saveUploadState() returned error: %v", err)
+ }
+
+ if err := deleteUploadState(stateDir, "backup-id"); err != nil {
+ t.Fatalf("deleteUploadState() returned error: %v", err)
+ }
+ if _, err := os.Stat(uploadStatePath(stateDir, "backup-id")); !os.IsNotExist(err) {
+ t.Fatalf("state file still exists or stat failed unexpectedly: %v", err)
+ }
+}
diff --git a/internal/flow/verify.go b/internal/flow/verify.go
index 93857fe..72b917c 100644
--- a/internal/flow/verify.go
+++ b/internal/flow/verify.go
@@ -55,27 +55,49 @@ func runVerify(backupID string, identityPath string, backend storage.Backend) er
payloadHasher := sha256.New()
for _, chunk := range manifest.Chunks {
- chunkHasher := sha256.New()
-
- multiWriter := io.MultiWriter(chunkHasher, payloadHasher)
-
- err := backend.Download(fmt.Sprintf("%s/%s", backupID, chunk.Name), multiWriter)
+ var encryptedChunk bytes.Buffer
+ err := backend.Download(fmt.Sprintf("%s/%s", backupID, chunk.Name), &encryptedChunk)
if err != nil {
p.Stop()
return fmt.Errorf("failed to download chunk %s: %w", chunk.Name, err)
}
- if hex.EncodeToString(chunkHasher.Sum(nil)) != chunk.SHA256 {
+ encryptedBytes := encryptedChunk.Bytes()
+ if int64(len(encryptedBytes)) != chunk.EncryptedSize {
+ p.Stop()
+ return fmt.Errorf("CHUNK CORRUPT: Size %s mismatch!", chunk.Name)
+ }
+ if hashBytes(encryptedBytes) != chunk.EncryptedSHA256 {
p.Stop()
return fmt.Errorf("CHUNK CORRUPT: Hash %s mismatch!", chunk.Name)
}
+ plainReader, err := crypto.DecryptReader(identityPath, bytes.NewReader(encryptedBytes))
+ if err != nil {
+ p.Stop()
+ return fmt.Errorf("failed to decrypt chunk %s: %w", chunk.Name, err)
+ }
+ plainBytes, err := io.ReadAll(plainReader)
+ if err != nil {
+ p.Stop()
+ return fmt.Errorf("failed to read decrypted chunk %s: %w", chunk.Name, err)
+ }
+ if int64(len(plainBytes)) != chunk.PlainSize {
+ p.Stop()
+ return fmt.Errorf("CHUNK CORRUPT: Plain size %s mismatch!", chunk.Name)
+ }
+ if hashBytes(plainBytes) != chunk.PlainSHA256 {
+ p.Stop()
+ return fmt.Errorf("CHUNK CORRUPT: Plain hash %s mismatch!", chunk.Name)
+ }
+ payloadHasher.Write(plainBytes)
+
p.Add(1)
}
p.Stop()
- if hex.EncodeToString(payloadHasher.Sum(nil)) != manifest.PayloadEncryptedHash {
+ if hex.EncodeToString(payloadHasher.Sum(nil)) != manifest.PayloadSHA256 {
return fmt.Errorf("FATAL: Overall payload hash differs from manifest! Data has been modified")
}
diff --git a/main.go b/main.go
index 32905c7..570390d 100644
--- a/main.go
+++ b/main.go
@@ -15,6 +15,7 @@ var cli struct {
Keygen KeygenCmd `cmd:"" help:"Generate an X25519 identity and recipient file."`
Upload UploadCmd `cmd:"" help:"Encrypt, chunk, and upload a file to a storage destination."`
+ Resume ResumeCmd `cmd:"" help:"Validate saved state for an interrupted upload."`
Restore RestoreCmd `cmd:"" help:"Restore and decrypt a backup from storage."`
Verify VerifyCmd `cmd:"" help:"Verify backup chunks and checksums."`
Cleanup CleanupCmd `cmd:"" help:"Remove orphan backup directories missing manifest.age."`
diff --git a/resume.go b/resume.go
new file mode 100644
index 0000000..3dfe6a9
--- /dev/null
+++ b/resume.go
@@ -0,0 +1,16 @@
+package main
+
+import (
+ "github.com/pterm/pterm"
+
+ "gloak/internal/flow"
+)
+
+type ResumeCmd struct {
+ UUID string `arg:"" name:"uuid" help:"Backup UUID to resume."`
+}
+
+func (r *ResumeCmd) Run() error {
+ pterm.Info.Println("Starting RESUME process...")
+ return flow.RunResume(r.UUID)
+}