feat(postgresblob): a PostgreSQL driver for gocloud.dev/blob

.github/workflows/ci.yml

@@ -40,7 +40,7 @@ jobs:
     strategy:
       fail-fast: false
       matrix:
-        module: ["dbkit/postgres", "gonsole/auth/postgres"]
+        module: ["dbkit/postgres", "gonsole/auth/postgres", "postgresblob"]
     defaults:
       run:
         working-directory: ${{ matrix.module }}
@@ -99,7 +99,7 @@ jobs:
     strategy:
       fail-fast: false
       matrix:
-        module: ["dbkit", "dbkit/postgres", "dbkit/sqlite", "mailkit", "goncierge", "gonsole", "gonsole/auth", "gonsole/auth/postgres", "gonsole/auth/sqlite", "pluginkit", "pluginkit/graphwire"]
+        module: ["dbkit", "dbkit/postgres", "dbkit/sqlite", "mailkit", "goncierge", "gonsole", "gonsole/auth", "gonsole/auth/postgres", "gonsole/auth/sqlite", "pluginkit", "pluginkit/graphwire", "postgresblob"]
     steps:
       - uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
         with:
@@ -121,7 +121,7 @@ jobs:
     strategy:
       fail-fast: false
       matrix:
-        module: ["dbkit", "dbkit/postgres", "dbkit/sqlite", "mailkit", "goncierge", "gonsole", "gonsole/auth", "gonsole/auth/postgres", "gonsole/auth/sqlite", "pluginkit", "pluginkit/graphwire"]
+        module: ["dbkit", "dbkit/postgres", "dbkit/sqlite", "mailkit", "goncierge", "gonsole", "gonsole/auth", "gonsole/auth/postgres", "gonsole/auth/sqlite", "pluginkit", "pluginkit/graphwire", "postgresblob"]
     defaults:
       run:
         working-directory: ${{ matrix.module }}

.github/workflows/gocloud-master.yml

@@ -0,0 +1,45 @@
+name: gocloud-master
+
+on:
+  schedule:
+    - cron: "17 4 * * 1"
+  workflow_dispatch:
+
+permissions:
+  contents: read
+
+jobs:
+  postgresblob:
+    runs-on: ubuntu-latest
+    timeout-minutes: 15
+    env:
+      DBKIT_TEST_POSTGRES_URL: "postgres://postgres:postgres@localhost:5434/postgres?sslmode=disable"
+    defaults:
+      run:
+        working-directory: postgresblob
+    services:
+      postgres:
+        image: postgres:18
+        env:
+          POSTGRES_PASSWORD: postgres
+        ports:
+          - 5434:5432
+        options: >-
+          --health-cmd "pg_isready -U postgres"
+          --health-interval 2s
+          --health-timeout 2s
+          --health-retries 15
+    steps:
+      - uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
+        with:
+          persist-credentials: false
+      - uses: actions/setup-go@924ae3a1cded613372ab5595356fb5720e22ba16 # v6.5.0
+        with:
+          go-version-file: postgresblob/go.mod
+          cache-dependency-path: |
+            postgresblob/go.mod
+            postgresblob/go.sum
+      - run: go get gocloud.dev@master
+      - run: go mod tidy
+      - run: go test -race ./...
+      - run: go vet ./...

README.md

@@ -40,6 +40,8 @@ need and ignore the rest.
   wiring.
 - [`pluginkit/graphwire`](pluginkit/graphwire/) generates the GraphQL
   resolver root of an application from the schemas of its plugins.
+- [`postgresblob`](postgresblob/) is a gocloud.dev/blob driver that
+  keeps a program's files in two tables of its PostgreSQL database.
 
 ## Design
 

dbkit/postgres/compose.yaml

@@ -6,7 +6,7 @@ services:
     environment:
       POSTGRES_PASSWORD: postgres
     ports:
-      - "5434:5432"
+      - "127.0.0.1:5434:5432"
     volumes:
       - postgres-data:/var/lib/postgresql
     healthcheck:

postgresblob/.golangci.yml

@@ -0,0 +1,52 @@
+version: "2"
+
+linters:
+  default: standard
+  enable:
+    - cyclop
+    - depguard
+    - gocognit
+    - lll
+    - misspell
+    - revive
+    - unconvert
+    - unparam
+  settings:
+    cyclop:
+      max-complexity: 10
+    gocognit:
+      min-complexity: 15
+    lll:
+      line-length: 120
+    revive:
+      rules:
+        - name: blank-imports
+          disabled: true
+        - name: exported
+    depguard:
+      rules:
+        dependency-purity:
+          list-mode: strict
+          allow:
+            - $gostd
+            - gocloud.dev
+            - github.com/google/uuid
+            - github.com/gopherium/framework/dbkit/postgres
+            - github.com/gopherium/framework/postgresblob
+            - github.com/jackc/pgx/v5
+            - github.com/pressly/goose/v3
+  exclusions:
+    rules:
+      - path: _test\.go
+        linters:
+          - cyclop
+          - gocognit
+
+formatters:
+  enable:
+    - gofmt
+    - goimports
+  settings:
+    goimports:
+      local-prefixes:
+        - github.com/gopherium/framework

postgresblob/CHANGELOG.md

@@ -0,0 +1,38 @@
+# Changelog
+
+All notable changes to the `postgresblob` module are documented in this
+file. The format follows
+[Keep a Changelog](https://keepachangelog.com/en/1.1.0/), and the
+module follows [Semantic Versioning](https://semver.org/). While at
+v0.x, minor releases may contain breaking changes.
+
+Releases of this module are tagged `postgresblob/vX.Y.Z`.
+
+## [Unreleased]
+
+### Added
+
+- A driver of `driver.Bucket` from gocloud.dev/blob v0.47.0 that keeps each object in two PostgreSQL tables of one schema.
+- `OpenBucket`, a bucket on a `*pgxpool.Pool` it never closes, refused when the database encoding is not UTF8.
+- `Options` with `Schema`, `ChunkSize`, `MaxSize` and `CleanupTimeout`, each zero field taking its default.
+- `MaxChunkSize`, 256 MiB, the largest `ChunkSize` a bucket takes.
+- `Migrate`, which creates the schema and its `objects` and `chunks` tables through `dbkit/postgres` under goose's lock.
+- Writes stream in chunks of 1 MiB by default and stay hidden until `Close`, which replaces the old object in one transaction.
+- Writes, copies and deletes take a transaction lock of each key they touch, so `IfNotExist` lets exactly one of many racing writers win.
+- Keys are bytes, so any UTF-8 key up to `MaxKeyBytes` (1,024 bytes) works, NUL included, and listings follow byte order.
+- Each `Read` runs one statement for at most one chunk, or 1 MiB when chunks are smaller.
+- A read whose object was replaced or deleted fails with `FailedPrecondition`.
+- `MaxSize` caps each object, and a write or copy past it fails with `ResourceExhausted` and keeps the old object.
+- A writer whose context ends or whose write fails deletes its pending row within `CleanupTimeout`.
+- `Sweep` and `SweepOptions`, which delete at most `Limit` unfinished uploads untouched for longer than `OlderThan`.
+- An upload `Sweep` deleted fails with `FailedPrecondition` at its next chunk or at `Close`.
+- `Copy` in one transaction, onto the same key too.
+- `URLOpener` on the `postgres` scheme, which takes `schema` out of the query and closes the pool it opened.
+- A URL pgx cannot parse fails with one message that holds no part of the URL.
+- `ErrorCode` names 42P01, 23505, 42501, 22021, the 53 class, 54000 and 57014, and `ErrorAs` exposes `*pgconn.PgError`.
+- `SignedURL` answers `Unimplemented`.
+- The tests run go-cloud's `drivertest` unmodified three ways: plain, through `driver.NewPrefixedBucket` and with 7 byte chunks.
+
+### Security
+
+- Requires `golang.org/x/net` v0.60.0, which fixes GO-2026-6603, GO-2026-6610, GO-2026-6611, GO-2026-6612 and GO-2026-6617.

postgresblob/README.md

@@ -0,0 +1,167 @@
+# postgresblob
+
+A [gocloud.dev/blob](https://gocloud.dev/howto/blob/) driver that keeps
+files in PostgreSQL.
+
+gocloud.dev/blob is one Go API for buckets of files, with drivers for
+S3, Google Cloud Storage, Azure, a local folder and memory. postgresblob
+adds PostgreSQL, so a program that already runs PostgreSQL can keep its
+files there without a second service.
+
+If you know Laravel, gocloud.dev/blob plays the part of Flysystem, and
+postgresblob is one more adapter behind it. Flysystem's own adapters
+offer no disk made of database tables, and that is what this one is.
+
+The guide is at <https://docs.gopherium.org/files/postgres/>.
+
+## How it stores files
+
+Each object is one row in the table `objects`, and its bytes are rows
+in the table `chunks`, 1 MiB each by default. Writes send one chunk at
+a time. Each read fetches at most one chunk, or 1 MiB when chunks are
+smaller, so a large file never sits whole in memory unless you read it
+all at once.
+
+A new object stays hidden until its writer closes. Then it replaces
+the old object under that key in one transaction. Writes, copies and
+deletes of the same key wait for each other, so `IfNotExist` lets
+exactly one of many writers win.
+
+Keys are stored as bytes. Any UTF-8 key of up to 1,024 bytes works, a
+NUL byte included, and a listing follows byte order.
+
+## Creating the tables
+
+```go
+err := postgresblob.Migrate(ctx, db, "files")
+```
+
+`Migrate` takes a `*sql.DB` and creates the schema and both tables
+through `dbkit/postgres`, under goose's migration lock. The schema
+keeps its own version table, `files.goose_db_version`.
+
+The schema must be a lowercase plain name of up to 63 bytes, made of
+letters, digits and underscores, that does not start with a digit. It
+must not be a word PostgreSQL reserves, such as `user` or `order`, and
+must not start with `pg_`. An empty one means `blob`.
+
+## Opening a bucket on your pool
+
+```go
+bucket, err := postgresblob.OpenBucket(ctx, pool, &postgresblob.Options{
+	Schema:  "files",
+	MaxSize: 25 << 20,
+})
+if err != nil {
+	return err
+}
+defer bucket.Close()
+```
+
+`OpenBucket` takes the program's `*pgxpool.Pool` and never closes it.
+It refuses a database whose encoding is not UTF8. Each option left at
+zero takes its default:
+
+| Option | Default | Meaning |
+| --- | --- | --- |
+| `Schema` | `blob` | the schema that holds the tables |
+| `ChunkSize` | 1 MiB | the most bytes one chunk row holds, up to `MaxChunkSize` (256 MiB) |
+| `MaxSize` | no cap | the most bytes one object may hold |
+| `CleanupTimeout` | 10 seconds | how long a failed upload may take to delete its rows |
+
+## Opening a bucket from a URL
+
+```go
+import _ "github.com/gopherium/framework/postgresblob"
+
+bucket, err := blob.OpenBucket(ctx, "postgres://app@db.example.com/site?schema=files")
+```
+
+The `postgres` scheme opens a pool of its own, and `bucket.Close`
+closes it. The opener takes `schema` out of the query and hands the
+rest to `pgxpool.ParseConfig`, so pgx settings such as `sslmode` and
+`pool_max_conns` work. pgx sends every other query parameter to the
+server as a setting, and the server turns down a name it does not
+know. go-cloud's own `prefix` parameter works too.
+
+When pgx cannot parse the URL, the error holds no part of it. A URL
+that Go's `net/url` cannot parse never reaches the driver, and
+go-cloud's error for it repeats the whole URL, password included. So
+keep those errors out of shared logs.
+
+## The size cap
+
+With `MaxSize` set, a write or a copy that would take an object past
+the cap fails with `gcerrors.ResourceExhausted`. The old object under
+that key stays as it was.
+
+## Unfinished uploads
+
+A writer whose context ends, or whose write fails, deletes its own
+rows. A program that stops in the middle of an upload leaves a pending
+row behind, and `Sweep` deletes those:
+
+```go
+deleted, err := postgresblob.Sweep(ctx, pool, postgresblob.SweepOptions{
+	Schema:    "files",
+	OlderThan: time.Hour,
+	Limit:     100,
+})
+```
+
+`Sweep` deletes at most `Limit` uploads that got no new chunk for
+longer than `OlderThan`, the oldest first. Run it from a timer. When it
+returns `Limit`, more may be waiting. An upload that `Sweep` deleted
+fails with `gcerrors.FailedPrecondition` when it writes its next chunk
+or closes.
+
+## Errors
+
+| Code | When |
+| --- | --- |
+| `NotFound` | no object has the key, or the schema has no tables (42P01) |
+| `FailedPrecondition` | `IfNotExist` found an object, the object changed during a read, `Sweep` deleted the upload, or 23505 |
+| `InvalidArgument` | an empty key, a key over 1,024 bytes, or 22021 |
+| `ResourceExhausted` | a write or copy past `MaxSize`, the class 53, or 54000 |
+| `PermissionDenied` | 42501 |
+| `Canceled` | a canceled context, or 57014 |
+| `Unimplemented` | `SignedURL` |
+| `Unknown` | every other error |
+
+Since a schema with no tables answers `NotFound`, call
+`bucket.IsAccessible` when your program starts.
+
+## As
+
+`As` answers false on the bucket, its readers, attributes and list
+objects. `ErrorAs` sets a `*pgconn.PgError` when the error came from
+the server.
+
+## What is not supported
+
+- `SignedURL` answers `gcerrors.Unimplemented`.
+- A read takes no snapshot. When another write replaces or deletes the
+  object during a read, the read fails with
+  `gcerrors.FailedPrecondition`, and never returns short data.
+- Files grow the database and its backups, and the chunks are stored
+  without compression.
+- A large `Copy` runs as one long transaction, and writes to its
+  source or its destination wait until it ends.
+
+## Testing
+
+The tests need a PostgreSQL server where they may create databases.
+`compose.yaml` starts one on port 5434:
+
+```sh
+docker compose up -d
+DBKIT_TEST_POSTGRES_URL='postgres://postgres:postgres@localhost:5434/postgres?sslmode=disable' go test ./...
+```
+
+They run go-cloud's `drivertest` three ways: plain, through
+`driver.NewPrefixedBucket`, and with 7 byte chunks so the small test
+files cross chunk edges.
+
+## License
+
+Apache-2.0.