Skip to content

fix: speed up project document delivery - #1302

Open
nicobao wants to merge 3 commits into
zkorum:mainfrom
nicobao:investigate/project-documents
Open

nicobao wants to merge 3 commits into
zkorum:mainfrom
nicobao:investigate/project-documents

Conversation

@nicobao

@nicobao nicobao commented Nov 14, 2026 •

Copy link
Copy Markdown
Member

Summary

Implement faster delivery of JavaScript-powered project documents without changing their rendering or security policy.

  • Store exactly two encoded copies of textual documents: Brotli quality 6 and gzip. New text uploads never store an uncompressed original; logical file sizes remain unchanged. PDFs remain a single unencoded object.
  • Compression uses Node's built-in zlib directly in the API upload path, not a separate recurring process. The standalone command is only for converting existing files.
  • Negotiate available representations from Accept-Encoding; browsers download directly from S3, without API proxying or per-open recompression.
  • Preserve normalized HTML/CSP bytes, iframe sandboxing, network restrictions, participant/facilitator isolation, authorization on every grant, and restricted-download cache policy.
  • Allow simultaneous public grants from attendees sharing a workshop IP while retaining the facilitator quota.
  • Replace the false 15-second viewer failure with a non-blocking loading indicator and keep manual reload available.
  • Add a dry-run-by-default documents:compress command for existing published files.
  • Coordinate representation publication, cancellation, retry, and bounded cleanup through database state, writer-side transactions, tombstones, and upload deadlines.
  • Switch legacy files to encoded-only delivery only after both copies are ready. Retire their original S3 objects after a seven-day grace period covering previously issued signed URLs.
  • Capture signing timestamps under a file lock to make that retirement safe, and recover successful cutovers when a database COMMIT acknowledgement is lost.

The previous investigation/design documents are removed from the PR's final diff.

Database

Generated through pnpm db:generate:

  • Flyway: services/api/database/flyway/V0115__equal_golden_guardian.sql
  • Matching Drizzle SQL, snapshot, and journal entry.
  • Adds project_document_representation, encoding/status enums, nullable project_document_file.compressed_at, and an index for retiring legacy originals.
  • Enforces parent-file references, unique active encodings/object keys, positive sizes, and failed/deleted-object state constraints.
  • Also enforces valid compressed-content/original-cleanup states. The earlier unapplied migration generation was undone and replaced; this PR contains one new migration, not an additional follow-up migration.

The actual generated SQL was applied and tested only in disposable PostgreSQL. Existing file metadata and script permissions were preserved. No development or production database has been migrated.

Validation

  • 135 focused backend tests, including generated-migration upgrade coverage, encoded-only uploads, gzip fallback, PDF handling, concurrent publication/grants, encoding exclusions, permission revocation, cancellation, cleanup retries, legacy original retirement, lost COMMIT acknowledgements, and legacy backfill.
  • 13 viewer tests, including slow transfers and manual reload.
  • API/frontend typechecks, production TypeScript compilation, targeted lint/format checks, and make check-soft-deletes.
  • Controlled local Chromium test of the real report at 4 Mbit/s and 150 ms latency: identity 24.52 s, gzip 4.00 s, Brotli 1.66 s. Report interaction succeeded in all three cases.
  • Brotli downloads decoded to byte-identical standalone HTML and rendered offline in Chromium.

Local measurements are not production guarantees. Safari/iOS and deployed HTTPS/S3/proxy behavior remain rollout checks. Public-grant IP-quota exemption is intentional; infrastructure-level abuse protection remains important.

Rollout

  1. Apply the new database migration before deploying the API.
  2. Deploy api and agora.
  3. With the API's deployment environment configured, inspect existing-file candidates:
    pnpm documents:compress --project sfcg-amplify-kg
  4. Explicitly apply the backfill:
    pnpm documents:compress --project sfcg-amplify-kg --apply
  5. Verify deployed encoding negotiation, report interaction, restricted access, and offline downloads on supported browsers. Existing files retain identity delivery until backfilled. After conversion, new grants use gzip/Brotli only; the normal API cleanup retires their legacy originals after the seven-day link-expiry grace period.

Deploy: agora, api

Document the live Amplify report investigation and the publication/delivery refactor proposal. The observed bottleneck is transferring a 12.1 MB uncompressed HTML artifact directly from S3, while the viewer can report a misleading timeout before the transfer completes.

Record controlled identity, gzip, and Brotli measurements using the unchanged JavaScript report, along with the limits of those observations and a standalone-report browser compatibility risk.

Propose precompressed HTTP representations, explicit storage and publication invariants, and rollout verification while preserving the existing CSP hashes, iframe sandbox, audience isolation, and authorization on every open. Keep CDN adoption, additional security hardening, and content-format changes conditional on further evidence.

Include authoritative references, migration and rollback considerations, and outstanding HTTPS negotiation and offline-download verification. No application or infrastructure implementation is included.

Deploy: none
@netlify

netlify Bot commented Nov 14, 2026 •

Copy link
Copy Markdown

👷 Deploy request for agoracitizen pending review.

Visit the deploys page to approve it

Name Link
🔨 Latest commit 9958769

Precompress normalized project documents once at publication into Brotli quality 6 and gzip representations, retaining the original object and logical file metadata. Negotiate published representations from Accept-Encoding and keep browser transfers directly on S3 without API proxying or per-request compression.

Add durable representation reservations, checksums, encoding metadata, explicit failure states, a nullable marker for incompressible files, and database constraints. Coordinate cleanup on the writer, tombstone objects before deletion, prevent interrupted publishers from reviving deleted representations, bound upload deadlines and cleanup work, and await all deletion attempts before returning.

Include the generated Drizzle and Flyway migration, snapshot, journal entry, synchronized schema copies, and regenerated database test fixtures. Verify the migration in disposable PostgreSQL against existing document data, preserving file metadata and script permissions while checking foreign keys, active uniqueness, positive sizes, and deletion/failure invariants. No development or production database migration has been applied.

Provide a dry-run-by-default documents:compress command for existing published files, with explicit --apply and scoped project/document selection. Preserve existing CSP bytes, iframe sandboxing, network restrictions, facilitator authorization, restricted download naming, and owner no-store behavior.

Exempt public participant grants from the shared-IP quota so workshop attendees can open the same report simultaneously; retain the facilitator quota and authorize every grant. Replace the misleading 15-second viewer failure state with a non-blocking loading indicator and keep manual reload available during slow transfers.

Remove the earlier investigation/design documents so the PR contains the implementation rather than a plan. Cover compression byte preservation, encoding negotiation, concurrent grants/publishers, cancellation, cleanup retry, legacy backfill, permission revocation, S3 metadata, and slow viewer behavior.

Validation: 128 focused backend tests and 13 viewer tests pass; API/frontend typechecks, production TypeScript compilation, targeted lint/format checks, and the soft-delete policy pass. Local Chromium checks retain report interactions and byte-identical offline downloads; Safari/iOS and deployed HTTPS behavior remain rollout verification steps.

Deploy: agora, api
@nicobao nicobao changed the title docs: investigate project document loading fix: speed up project document delivery Nov 14, 2026
Store exactly gzip and Brotli copies for new textual project documents without uploading an identity object. Prepare both encodings even for tiny or incompressible text so gzip is always available as the compatibility fallback. Keep PDFs as one unencoded object and preserve logical sizes, normalized HTML/CSP bytes, script permissions, authorization, and restricted-download behavior.

Track the atomic compression cutover with compressed_at. Legacy documents remain readable through identity until both encoded copies are available; new grants then use encoded copies only. Retire legacy originals through the existing bounded API cleanup after the maximum seven-day signed-URL lifetime, without tombstoning the live document. Preserve original-deletion acknowledgements on failed document publication and use privacy-aware error reporting for deletion retries.

Centralize upload, stale-reservation, and URL-expiry policy constants. Capture signing timestamps under a file share lock and pass them to the S3 presigner so grants issued before conversion cannot outlive the original-retention window. Prefer gzip when Accept-Encoding is absent, honor explicit encoding exclusions, and do not fall back to identity for converted files.

Recover a successful publication when its database COMMIT acknowledgement is lost instead of invalidating the only live copies. Extend database constraints and cleanup indexing, synchronize canonical schema copies, and regenerate test fixtures.

Undo the earlier unapplied migration generation and replace it with one final Drizzle/Flyway migration, snapshot, and journal entry. Do not introduce an additional follow-up migration. Verify the actual generated SQL against existing document data in disposable PostgreSQL, including unchanged file metadata and script permissions and the new storage-state constraints. No development or production database migration has been applied.

Validation: 135 focused backend tests and 13 viewer tests pass, including encoded-only uploads, both upload-failure paths, gzip fallback, PDF handling, legacy original retirement and retry, concurrency, permission revocation, migration upgrade behavior, and lost-COMMIT recovery. API/frontend typechecks, production TypeScript compilation, targeted lint and formatting checks, git diff checks, and the soft-delete policy pass.

Deploy: api

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant