Skip to content

feat: add resumable upload #2

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
Syrup merged 7 commits into from
Jun 4, 2026
Merged

feat: add resumable upload #2

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
7 commits
Select commit Hold shift + click to select a range
b077ef6
feat: add resumable upload
Syrup May 28, 2026
6615ea5
feat(resume): add upload state model
Syrup May 29, 2026
addfd25
test(resume): fix upload state timestamp assertion
Syrup May 31, 2026
741c7a8
feat(resume): record upload progress
Syrup May 31, 2026
28519cc
test(resume): cover mid-upload failure state
Syrup Jun 4, 2026
84db2fa
feat(resume): complete interrupted upload resume
Syrup Jun 4, 2026
96ebf63
Merge remote-tracking branch 'origin/main' into feat/resumable-upload
Syrup Jun 4, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
123 changes: 72 additions & 51 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,5 +1,9 @@
# gloak

<p align="center">
<img src="assets/gloak-banner.svg" alt="Gloak" width="620">
</p>

![Status](https://img.shields.io/badge/status-alpha-orange)
![Version](https://img.shields.io/badge/version-v0.1.0--alpha.1-orange)
![Go](https://img.shields.io/badge/go-1.26.3-00ADD8)
Expand All @@ -13,10 +17,10 @@
>
> Use it for testing, non-critical data, or secondary backups only. If you store anything important with gloak, keep another verified backup somewhere else.

Encrypt files, split into chunks with SHA-256 checks, upload, verify, restore. Metadata (filenames, sizes, recipients, hashes) lives inside an age-encrypted manifest. Storage never sees plaintext metadata.
gloak splits a file into fixed-size chunks, encrypts each chunk with age, uploads them, and verifies them before restore. Metadata such as filenames, sizes, recipients, and hashes lives inside an age-encrypted manifest. Storage only sees backup IDs, public completion markers, and encrypted chunk objects.

```text
keygen → encrypt → chunk → encrypt manifest → upload → verify → restore
keygen → chunk → encrypt chunks → upload → resume if needed → verify → restore
```

---
Expand All @@ -36,6 +40,9 @@ go build -o gloak .
# encrypt + upload
./gloak upload ./secret.tar --remote myremote:backup_folder

# resume an interrupted upload
./gloak resume <backup-id>

# verify
./gloak verify <backup-id> --remote myremote:backup_folder -i ~/.gloak/identity.txt

Expand All @@ -47,14 +54,14 @@ go build -o gloak .

## Why this exists

Most backup tools either:
Most backup tools make at least one tradeoff gloak avoids:

1. Encrypt a single blob — good luck retrying after a network failure at byte 47 GB of 50.
2. Split into chunks without integrity metadata — you find out a chunk is corrupt during restore, with no manifest to guide repair.
3. Store metadata in plaintext — filenames, sizes, recipient keys, and chunk hashes visible to anyone who reads the storage.
1. Encrypt a single blob. If a network failure hits at byte 47 GB of 50, retrying is painful.
2. Split into chunks without enough integrity metadata. You find corruption during restore, without a manifest to guide repair.
3. Store metadata in plaintext. Filenames, sizes, recipient keys, and chunk hashes are visible to anyone who can read the storage.
4. Pull in rclone, restic rest-server, or S3 libraries when you just need local filesystem or a mounted remote.

gloak is the boring correct version: age encryption, fixed-size chunks, deterministic manifests, integrity checks on every chunk *and* the full payload, and an encrypted manifest envelope so storage never sees filenames or hashes.
gloak keeps the design narrow: age encryption, fixed-size chunks, chunk and payload hashes, resumable uploads, and an encrypted manifest so the storage backend does not see filenames or hashes.

---

Expand All @@ -70,7 +77,7 @@ Build the standalone binary:
go build -o gloak .
```

Produces `gloak` — a compiled binary. Drop it into containers, cron jobs, or VPS rootfs without needing Node, Bun, or Python installed.
This produces `gloak`, a compiled binary you can put in containers, cron jobs, or VPS root filesystems without installing Node, Bun, or Python.

Run tests:

Expand All @@ -92,8 +99,8 @@ Default location is `~/.gloak/`:

```text
~/.gloak/
identity.txt # private key — keep this secret
recipient.txt # public key — safe to share
identity.txt # private key, keep this secret
recipient.txt # public key, safe to share
```

Use a custom directory:
Expand All @@ -116,7 +123,7 @@ After `keygen` runs once, `-i` (identity) and `-r` (recipient) flags become opti

## Commands

### `setup` — install dependencies
### `setup`: install dependencies

If you don't have `rclone` installed on your system, `gloak setup` will download a standalone, verified binary to `~/.gloak/bin/rclone` so `gloak` can use it without touching your system packages.

Expand All @@ -131,9 +138,9 @@ If you don't have `rclone` installed on your system, `gloak setup` will download

---

### `upload` — full encrypted backup flow
### `upload`: create an encrypted backup

Creates a UUID folder, encrypts the input, splits into chunks, writes SHA-256 checksums into an age-encrypted manifest, and uploads in order: chunks first, manifest second, locator last.
Creates a UUID folder, splits the input into fixed-size plaintext chunks, encrypts each chunk independently with age, records SHA-256 checksums in an age-encrypted manifest, and uploads chunks first. `manifest.age` and `locator.json` are uploaded after the chunks.

Chunks and manifest are encrypted with the same age recipient. Storage sees filenames like `chunk_00000` and `manifest.age` but cannot read filenames, sizes, or hashes without the identity key.

Expand All @@ -160,13 +167,25 @@ Save this UUID for restoration: 7f91c6c7-7a0b-44aa-ae23-997b60e4e998
| `--recipient <key-or-file>` | `-r` | `~/.gloak/recipient.txt` | Recipient public key or recipient file. |
| `--remote` | `-R` | _required_ | Target Rclone destination (e.g. `myremote:backup_folder` or `/mnt/backups`). |

The locator is uploaded last. If the upload is interrupted, the backup directory exists but has no `locator.json`. Upload refuses to overwrite on retry, so partial uploads never look complete.
The locator is uploaded last. If the upload is interrupted, the backup directory exists but has no `locator.json`. gloak keeps local state under `~/.gloak/uploads/<backup-id>.json`, which lets `resume` skip chunks that already uploaded cleanly.

---

### `verify` — verify backup integrity
### `resume`: continue an interrupted upload

Downloads and decrypts `manifest.age`, checks every chunk exists with matching size and SHA-256, then checks the concatenated encrypted payload hash against the manifest.
Reads local upload state from `~/.gloak/uploads`, scans the source file from the beginning, skips chunks whose plaintext size and SHA-256 still match the saved state, uploads missing chunks, then writes `manifest.age` and `locator.json`.

```bash
./gloak resume 7f91c6c7-7a0b-44aa-ae23-997b60e4e998
```

Resume requires the original source file at the same path with the same size. gloak does not keep plaintext or encrypted payload spool files locally. It stores only the chunk metadata needed to decide which uploaded chunks can be skipped.

---

### `verify`: check backup integrity

Downloads and decrypts `manifest.age`, checks each encrypted chunk size and SHA-256, decrypts each chunk, checks plaintext size and SHA-256, then compares the full plaintext payload hash with the manifest.

```bash
./gloak verify 7f91c6c7-7a0b-44aa-ae23-997b60e4e998 \
Expand All @@ -193,9 +212,9 @@ CHUNK CORRUPT: Hash chunk_00042 mismatch!

---

### `restore` — restore a backup
### `restore`: restore a backup

Verifies the full backup first (every chunk, every hash), then concatenates chunks and decrypts the payload.
Checks each encrypted chunk, decrypts it, checks the plaintext chunk, then writes the restored file chunk by chunk.

```bash
# with auto-resolved identity
Expand Down Expand Up @@ -227,9 +246,9 @@ Restore rejects:

---

### `cleanup` — remove incomplete backups
### `cleanup`: remove incomplete backups

Scans the remote storage for backup directories that are missing `manifest.age` (usually caused by interrupted uploads) and offers to delete them.
Scans remote storage for backup directories that are missing `manifest.age`, usually from interrupted uploads, and offers to delete them.

```bash
# interactive prompt before deletion
Expand Down Expand Up @@ -260,7 +279,7 @@ Scans the remote storage for backup directories that are missing `manifest.age`
chunk_00002
```

**`locator.json`** is a small public file:
`locator.json` is a small public file:

```json
{
Expand All @@ -269,24 +288,22 @@ Scans the remote storage for backup directories that are missing `manifest.age`
}
```

The locator has no sensitive metadata. No filename, no recipient, no chunk hashes, no payload hash. It exists so tools can detect backup completion without decrypting anything.
The locator has no sensitive metadata. It contains no filename, recipient, chunk hashes, or payload hash. Tools can use it to detect backup completion without decrypting anything.

**`manifest.age`** is the full backup metadata encrypted with age. Once decrypted:
`manifest.age` is the full backup metadata encrypted with age. Once decrypted:

| Field | Contains |
|-------|----------|
| `source.original_filename` | Basename of backed-up file |
| `source.original_size` | Original file size |
| `payload.encryption.recipients` | Age recipient(s) |
| `payload.encrypted_file_sha256` | SHA-256 of encrypted payload |
| `chunking.chunk_size` | Fixed chunk size |
| `chunking.total_chunks` | Number of chunks |
| `original_name` | Basename of backed-up file |
| `original_size` | Original file size |
| `recipient` | Age recipient |
| `payload_sha256` | SHA-256 of the full plaintext payload |
| `chunks[].index` | Chunk index (0-based) |
| `chunks[].name` | Chunk filename (e.g. `000042.chunk`) |
| `chunks[].size` | Chunk byte size |
| `chunks[].sha256` | SHA-256 of chunk content |


| `chunks[].name` | Chunk filename (e.g. `chunk_00042`) |
| `chunks[].plain_size` | Plaintext chunk byte size |
| `chunks[].plain_sha256` | SHA-256 of plaintext chunk content |
| `chunks[].encrypted_size` | Encrypted chunk byte size |
| `chunks[].encrypted_sha256` | SHA-256 of encrypted chunk content |

## Architecture

Expand All @@ -295,6 +312,7 @@ The locator has no sensitive metadata. No filename, no recipient, no chunk hashe
├── setup.go # rclone downloader command
├── keygen.go # keygen command definition
├── upload.go # upload command definition
├── resume.go # resume command definition
├── verify.go # verify command definition
├── restore.go # restore command definition
├── cleanup.go # cleanup command definition
Expand All @@ -307,6 +325,7 @@ The locator has no sensitive metadata. No filename, no recipient, no chunk hashe
│ └── stream.go # streaming age encrypt/decrypt wrapper
├── flow/
│ ├── upload.go # upload workflow logic
│ ├── resume.go # resume workflow logic
│ ├── verify.go # verification workflow logic
│ └── restore.go # restore workflow logic
└── storage/
Expand All @@ -317,51 +336,53 @@ The locator has no sensitive metadata. No filename, no recipient, no chunk hashe

```text
upload:
input file → age encrypt → encrypted payload
→ split into fixed-size chunks
→ SHA-256 each chunk
→ manifest.age (with all metadata)
→ upload chunks → upload manifest.age → upload locator.json
input file → split into plaintext chunks
→ SHA-256 plaintext chunk + full payload
→ age encrypt each chunk
→ SHA-256 encrypted chunk
→ upload chunks → upload manifest.age → upload locator.json

resume:
upload state → re-read source chunks
→ skip chunks whose plaintext hash still matches state
→ upload missing chunks → upload manifest.age → upload locator.json

verify:
locator.json → completion marker
→ manifest.age → age decrypt → validate manifest
→ check every chunk exists
→ check every chunk size
→ check every chunk SHA-256
→ concatenate chunks
→ check encrypted payload SHA-256
→ check encrypted chunk size/SHA-256
→ decrypt each chunk
→ check plaintext chunk size/SHA-256
→ check full payload SHA-256

restore:
verify (full) → download chunks → concatenate → decrypt payload → write output
manifest.age → download encrypted chunk → verify → decrypt chunk → write output
```

### Key design decisions

| Decision | Rationale |
|----------|-----------|
| Recipient-mode age encryption | One identity decrypts. No passphrase to remember or leak. |
| Per-chunk encryption | Each chunk is a standalone age payload, so interrupted uploads can resume without deterministic encryption. |
| Fixed-size chunks | Deterministic boundaries regardless of content. Simplifies verification, resumability, and storage layout. |
| Encrypted manifest | Storage never sees filenames, sizes, or hashes. Only backup UUIDs and chunk object names are visible. |
| Locator uploaded last | Atomicity marker. If upload is interrupted, no `locator.json` — upload refuses to overwrite on retry. |
| Locator uploaded last | If upload is interrupted, no `locator.json` marks the backup complete. |
| Identity required for verify/restore | Manifest is encrypted. You need the identity key to read it. This is a feature, not a bug. |
| SHA-256 at chunk + payload | Repair individual chunks without re-downloading everything. Corruption detected before decrypt. |
| SHA-256 at chunk + payload | Corruption is detected before and after decrypting each chunk. |
| Fixed 20MB chunk size | Prevents metadata bloat from tiny chunks and excessive memory usage. |

---



## Not built yet

Deferred intentionally:

| Feature | Reasoning |
|---------|-----------|
| Named storage profiles | `--remote my-backups` instead of typing URIs. |
| Resumable uploads | Track uploaded chunks, skip on retry. Needs state file in backup directory. |
| Telegram integration | Hermes Agent plugin for backup/restore/verify via Telegram. |
| Per-chunk encryption keys | Each chunk encrypted with a unique key, stored in manifest. Limits blast radius of key exposure. |
| Per-chunk data keys | Each chunk would use its own data key stored inside the encrypted manifest. This would limit key exposure if a key leaks. |

---

Expand Down
89 changes: 89 additions & 0 deletions assets/gloak-banner.svg
View file
Open in desktop
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading