v1.0.8 and v1.0.5

backend/AGENTS.md

@@ -0,0 +1,52 @@
+# Backend Agent Guide
+
+## Scope
+
+These instructions apply to work under `backend/`. Preserve the dependency
+direction: routes and middleware call controllers; controllers call services;
+services own domain rules and call repositories or integrations; repositories
+own persistence queries. Services must not read Express `req` or write `res`.
+
+## Commands
+
+Run from `backend/`:
+
+```sh
+npm run dev
+npm test
+npm test -- tests/api-videos.test.js
+```
+
+The test runner is Node's built-in serial runner. API tests use MongoDB Memory
+Server by default. If `MONGODB_URI_TEST` is configured, tests drop that
+database; only use a disposable test database.
+
+## Guardrails
+
+- Never commit `.env`, credentials, tokens, service-account JSON, or production
+  data. Environment variables are validated in `src/config/env.js`; update
+  `.env.example` and `docs/SETUP.md` when adding configuration.
+- Keep Zod request validation in `src/middlewares/validators/` and business
+  rules in services. Use shared response and `AppError` helpers.
+- Verify file contents from signatures, not supplied MIME types. Route uploads
+  through the storage service and preserve cleanup for partially completed
+  multi-file operations.
+- MongoDB stores domain records and opaque storage references; media bytes
+  belong to configured storage drivers. Preserve visibility checks for
+  metadata and media routes.
+- Redis is optional Upstash. Cache failures should be logged and fall back to
+  the authoritative database/storage path. Keep cache values small and apply
+  explicit TTLs.
+- Update `src/docs/openapi.js`, `docs/API.md`, tests, and relevant architecture
+  docs when changing endpoints or behavior. Do not silently change append-only
+  submission or account-data retention behavior.
+
+## References
+
+- [Backend docs index](docs/README.md)
+- [Setup and environment](docs/SETUP.md)
+- [API contract](docs/API.md)
+- [Architecture](docs/ARCHITECTURE.md)
+- [Module reference](docs/MODULE-REFERENCE.md)
+- [Design decisions](docs/design-docs/ARCHITECTURE-DECISIONS.md)
+- [Test-case inventory](docs/TEST-CASES.md)

backend/README.md

@@ -47,9 +47,10 @@ business rules.
 `src/services/initialization.js` initializes MongoDB, storage, and the configured
 admin idempotently; `server.js` is the standalone and Vercel entry point.
 
-Read the full references: [API](docs/API.md),
-[architecture](docs/ARCHITECTURE.md), [setup](docs/SETUP.md), and
-[test-case inventory](docs/TEST-CASES.md).
+Read the [documentation index](docs/README.md), [API](docs/API.md),
+[architecture](docs/ARCHITECTURE.md), [module reference](docs/MODULE-REFERENCE.md),
+[setup](docs/SETUP.md), [design decisions](docs/design-docs/ARCHITECTURE-DECISIONS.md),
+and [test-case inventory](docs/TEST-CASES.md).
 
 ## Responses
 

backend/docs/API.md

@@ -82,7 +82,7 @@ All paths below are relative to `/api/v1`.
 | GET | `/videos/:id` | Public or admin | Read a visible video. |
 | GET | `/videos/:id/file` | Public or admin | Stream a visible video, including byte-range requests. |
 | GET | `/videos/:id/poster` | Public or admin | Stream a visible video's poster, if present. |
-| POST | `/videos` | Admin | Upload a reference video and optional poster. |
+| POST | `/videos` | Admin | Upload a reference video or register a direct HTTPS video URL; URL sources require a poster. |
 | PATCH | `/videos/:id` | Admin | Update a video, including status or category. |
 | DELETE | `/videos/:id` | Admin | Delete a reference video and its base files; submissions remain. |
 | GET | `/demo-video` | Public | Read the current public demo-video metadata or `null`. |
@@ -117,12 +117,13 @@ Authentication and profile request schemas are defined in
 client-supplied roles are ignored. Profile updates accept supported `connections`
 fields for GitHub and LinkedIn; `null` disconnects a connection.
 
-The video and demo-video upload routes use `multipart/form-data`:
+The video and demo-video creation routes use `multipart/form-data`:
 
 | Field | Video | Demo video |
 | --- | --- | --- |
-| `video` | Required video file | Required video file |
-| `poster` | Optional image file | Not accepted |
+| `video` | Optional video file; provide this or `videoUrl`, but not both | Required video file |
+| `videoUrl` | Optional direct HTTPS MP4/WebM URL; provide this or `video`, but not both | Not accepted |
+| `poster` | Optional for uploaded files; required with `videoUrl` | Not accepted |
 | `title` | Required title | Required title |
 | `categoryId` | Optional category ID | Not accepted |
 | `status` | Optional video status | Not accepted |
@@ -147,7 +148,15 @@ Uploads are bounded by `MAX_VIDEO_UPLOAD_MB` and
 `MAX_RECORDING_UPLOAD_MB` (defaults: 90 MB and 100 MB). Video signatures are
 checked as MP4, WebM, or MOV. Poster signatures are checked as JPEG, PNG, or WebP;
 posters must not exceed 2 MB. The browser-provided MIME type is not treated as
-proof of file content.
+proof of file content. External URLs must point directly to media playable in a
+native HTML `<video>` element; YouTube and other watch-page URLs are not
+supported. The URL is stored in MongoDB. Its poster image is stored through the
+configured `STORAGE_DRIVER` and streamed from the API.
+
+Poster responses use a private five-minute browser cache. When Upstash Redis is
+configured, the backend also caches poster file metadata for five minutes; it
+does not cache poster image bytes. Redis errors are logged and fall back to
+reading metadata from the configured storage driver.
 
 ## Contact, rate limits, and access restrictions
 

backend/docs/MODULE-REFERENCE.md

@@ -0,0 +1,39 @@
+# Backend module reference
+
+For system startup, request flow, and persistence details, see
+[ARCHITECTURE.md](ARCHITECTURE.md). This page maps common changes to their
+implementation layers.
+
+| Concern | Primary modules | Notes |
+| --- | --- | --- |
+| App composition and startup | `server.js`, `src/app.js`, `src/services/initialization.js` | App composition is importable without binding a listener; initialization is shared and idempotent. |
+| Routes and access policy | `src/routes/`, `src/middlewares/authenticate.js`, `src/middlewares/requireRole.js` | Route definitions compose auth and validation middleware. |
+| Request validation | `src/middlewares/validators/`, `src/middlewares/validate.js` | Zod schemas validate bodies, params, and queries. |
+| HTTP response/error boundary | `src/controllers/`, `src/utils/response.js`, `src/middlewares/errorHandler.js` | Controllers adapt HTTP; services should throw `AppError` for expected failures. |
+| Domain behavior | `src/services/` | Enforce permissions, visibility, retention, cooldown, and multi-step workflows here. |
+| MongoDB persistence | `src/models/`, `src/repositories/` | Schemas define serialization/indexes; repositories encapsulate queries. |
+| Video library | `src/models/Video.js`, `src/services/videoService.js`, `src/routes/videoRoutes.js` | Supports stored uploads and direct HTTPS video URLs; posters are stored files. |
+| File storage | `src/services/storage/`, `src/utils/sendFile.js` | Local and Google Drive drivers; `sendFile` handles byte ranges and headers. |
+| Redis cache | `src/services/cache/upstashRedis.js`, feature service | Optional Upstash Redis REST client. Feature services own key naming, TTL, validation, and fallback logging. |
+| Authentication and consent | `src/services/authService.js`, `src/controllers/authController.js`, auth validators | Signup does not activate until the current consent is accepted. |
+| Submissions and archive | `src/services/submissionService.js`, `src/repositories/submissionRepo.js` | Submission records are append-only; archive storage is selected independently. |
+| API specification | `src/docs/openapi.js`, `docs/API.md` | Update both with route, schema, or response changes. |
+| Tests | `tests/*.test.js` | API-level suites plus focused service, filesystem, and utility suites. |
+
+## Typical request path
+
+```text
+route -> middleware -> controller -> service -> repository/model
+                                  -> storage/provider integration
+```
+
+Keep domain policy in services and persistence mechanics in repositories or
+storage drivers. Preserve error propagation and log provider/cache failures
+using the established logger.
+
+## Validation checklist
+
+For behavior changes, add focused tests at the service layer and API boundary
+where practical. Run the relevant test file, then `npm test` for broader
+changes. API and storage test setup is described in
+[SETUP.md](SETUP.md) and the [test-case inventory](TEST-CASES.md).

backend/docs/README.md

@@ -0,0 +1,11 @@
+# Backend documentation
+
+- [Setup](SETUP.md): local configuration, environment variables, and
+  deployment.
+- [API](API.md): endpoint access, request/response behavior, and limits.
+- [Architecture](ARCHITECTURE.md): startup, request flow, persistence, storage,
+  and security controls.
+- [Module reference](MODULE-REFERENCE.md): where to implement backend changes.
+- [Design decisions](design-docs/ARCHITECTURE-DECISIONS.md): current
+  architecture choices and trade-offs.
+- [Test-case inventory](TEST-CASES.md): API and unit test coverage.

backend/docs/design-docs/ARCHITECTURE-DECISIONS.md

@@ -0,0 +1,81 @@
+# Backend architecture decisions
+
+This is a record of current choices and their trade-offs, not a dated changelog.
+
+## Keep HTTP, domain, and persistence responsibilities separate
+
+**Decision:** Express routes compose middleware; controllers adapt HTTP inputs
+and responses; services enforce domain rules; repositories encapsulate
+MongoDB queries.
+
+**Reason and trade-off:** The same business behavior can be tested without an
+HTTP request, and API response formatting stays at the boundary. New features
+must be threaded through several layers rather than placing all logic in a
+route handler.
+
+**Implementation:** `src/routes/`, `src/controllers/`, `src/services/`, and
+`src/repositories/`.
+
+## Store media bytes outside MongoDB
+
+**Decision:** MongoDB stores metadata and opaque `{ driver, key }` file
+references. Local and Google Drive drivers provide file storage and streaming.
+Reference-video URLs may also be direct HTTPS media URLs; those URLs are stored
+as video metadata while their poster remains a stored file.
+
+**Reason and trade-off:** Database documents remain small and storage can be
+switched or separated by intent. External URLs avoid copying video bytes but
+depend on the remote host's availability and browser media/CORS behavior.
+
+**Implementation:** `src/models/Video.js`, `src/models/storedFile.js`,
+`src/services/storage/`, and `src/services/videoService.js`.
+
+## Preserve byte-range media streaming
+
+**Decision:** Stored video endpoints support HTTP Range requests; the storage
+driver exposes streams for requested byte ranges.
+
+**Reason and trade-off:** Browsers use range access for seeking and native
+playback. Storage drivers must preserve inclusive range semantics, MIME type,
+and file size.
+
+**Implementation:** `src/utils/sendFile.js` and
+`src/services/storage/`.
+
+## Treat contributor submissions as append-only records
+
+**Decision:** Accepted submission records and archived recording bytes are
+retained when an account or base video is deleted. A separate contact request
+allows the team to review recording/data deletion.
+
+**Reason and trade-off:** Contribution history and released model provenance
+remain stable. This increases responsibility for retention and requires an
+explicit process for deletion requests.
+
+**Implementation:** `src/services/submissionService.js`,
+`src/services/userService.js`, and `src/services/contactService.js`.
+
+## Make external integrations optional and failure-tolerant where safe
+
+**Decision:** Local file storage works without Google Drive; Redis caching is
+optional; transient cache failures fall back to authoritative reads. Cache
+values have explicit expirations and do not contain poster image bytes.
+
+**Reason and trade-off:** Local development and core reads do not require every
+provider to be configured. Cache outages can increase storage/database work,
+but should not make cached data appear authoritative.
+
+**Implementation:** `src/config/env.js`, `src/services/cache/upstashRedis.js`,
+`src/services/publicStatsService.js`, and `src/services/videoService.js`.
+
+## Put access and validation at the server boundary
+
+**Decision:** Route guards, authentication, Zod validation, role checks, file
+signature checks, rate limits, and origin checks are enforced server-side.
+Client-side restrictions improve UX but are not security controls.
+
+**Reason and trade-off:** API callers are not limited to the first-party UI.
+Routes and services must retain visibility checks even when the client already
+filters data.
+
+**Implementation:** `src/routes/`, `src/middlewares/`, and `src/services/`.

backend/docs/design-docs/README.md

@@ -0,0 +1,8 @@
+# Backend design documents
+
+These documents record current architectural decisions and their trade-offs.
+They complement the [architecture guide](../ARCHITECTURE.md) and do not replace
+the implementation or API contract. Update a decision when the behavior or
+trade-off changes; do not infer undocumented history.
+
+- [Architecture decisions](ARCHITECTURE-DECISIONS.md)

backend/package.json

@@ -1,6 +1,6 @@
 {
   "name": "backend",
-  "version": "1.0.4",
+  "version": "1.0.5",
   "description": "SignPakCommons API: Express, Node.js and MongoDB",
   "main": "server.js",
   "scripts": {

backend/src/controllers/videoController.js

@@ -20,5 +20,5 @@ export const videoController = {
   update: asyncHandler(async (req, res) => ok(res, await videoService.update(req.params.id, req.body))),
   remove: asyncHandler(async (req, res) => { await videoService.remove(req.params.id); noContent(res) }),
   file: asyncHandler(async (req, res) => sendFile(req, res, await videoService.getFile(req.params.id, req.user))),
-  poster: asyncHandler(async (req, res) => sendFile(req, res, await videoService.getPoster(req.params.id, req.user))),
+  poster: asyncHandler(async (req, res) => sendFile(req, res, await videoService.getPoster(req.params.id, req.user), { cache: 'private, max-age=300' })),
 }

backend/src/docs/openapi.js

@@ -116,6 +116,7 @@ export const openapi = {
           width: { type: 'integer', description: 'Displayed width in pixels; 0 when unknown.' },
           height: { type: 'integer', description: 'Displayed height in pixels; 0 when unknown.' },
           videoUrl: { type: 'string' },
+          videoSource: { type: 'string', enum: ['stored', 'external'] },
           poster: { type: 'string', nullable: true },
           size: { type: 'integer' },
         },
@@ -212,10 +213,12 @@ export const openapi = {
       },
       VideoUploadInput: {
         type: 'object',
-        required: ['video', 'title'],
+        required: ['title'],
+        description: 'Provide exactly one of video or videoUrl. External video URLs must point to a direct HTTPS video file and require a poster image.',
         properties: {
           video: { type: 'string', format: 'binary', description: 'MP4, WebM, or MOV video.' },
-          poster: { type: 'string', format: 'binary', description: 'Optional JPEG, PNG, or WebP image (up to 2 MiB).' },
+          videoUrl: { type: 'string', format: 'uri', description: 'Direct HTTPS MP4/WebM URL; watch-page URLs are not supported.' },
+          poster: { type: 'string', format: 'binary', description: 'JPEG, PNG, or WebP image (up to 2 MiB); required when videoUrl is used.' },
           title: { type: 'string', minLength: 1, maxLength: 120 },
           categoryId: { type: 'string', nullable: true, pattern: '^[a-f\\d]{24}$' },
           status: { type: 'string', enum: ['draft', 'published'], default: 'published' },
@@ -484,7 +487,7 @@ export const openapi = {
       },
       post: {
         tags: ['Videos'],
-        summary: 'Upload a base video',
+        summary: 'Add a base video from an upload or direct URL',
         security: adminSecurity,
         requestBody: { required: true, content: { 'multipart/form-data': { schema: ref('VideoUploadInput') } } },
         responses: { 201: jsonResponse('Created', ref('Video')), ...adminErrors, 413: errorResponse },

backend/src/middlewares/validators/videoSchemas.js

@@ -6,13 +6,20 @@ const title = z.string({ error: 'Give the video a title.' }).trim().min(1, 'Give
 const status = z.enum(VIDEO_STATUS_LIST, { error: `Status must be one of: ${VIDEO_STATUS_LIST.join(', ')}.` })
 // An empty string (from a form's "Unassigned" option) means no category.
 const categoryId = z.preprocess((value) => (value === '' ? null : value), objectId('category').nullable())
+const videoUrl = z.preprocess(
+  (value) => value === '' ? undefined : value,
+  z.url({ error: 'Enter a valid direct video URL.' })
+    .refine((value) => value.startsWith('https://'), 'Video URLs must use HTTPS.')
+    .optional(),
+)
 
 const pixels = z.coerce.number({ error: 'Size must be a number.' }).int().min(0).max(16384).default(0)
 
 export const createVideoSchema = z.object({
   title,
   status: status.default(VIDEO_STATUS.PUBLISHED),
   categoryId: categoryId.optional().default(null),
+  videoUrl,
   durationSec: z.coerce.number({ error: 'Duration must be a number.' }).min(0).max(24 * 60 * 60).default(0),
   width: pixels,
   height: pixels,

backend/src/models/Video.js

@@ -12,7 +12,8 @@ const videoSchema = new mongoose.Schema({
   // Displayed size in pixels as the browser reported it when the admin prepared the file; 0 means unknown (older uploads).
   width: { type: Number, default: 0, min: 0 },
   height: { type: Number, default: 0, min: 0 },
-  videoFile: { type: storedFileSchema, required: true },
+  videoFile: { type: storedFileSchema, default: null },
+  externalVideoUrl: { type: String, trim: true, default: null },
   posterFile: { type: storedFileSchema, default: null },
   createdBy: { type: mongoose.Schema.Types.ObjectId, ref: 'User' },
 }, {
@@ -21,14 +22,22 @@ const videoSchema = new mongoose.Schema({
   toJSON: jsonOptions((doc, ret) => {
     const base = `${API_PREFIX}/videos/${ret.id}`
     ret.categoryId = ret.category ? String(ret.category) : null
-    ret.videoUrl = `${base}/file`
+    ret.videoUrl = ret.externalVideoUrl || `${base}/file`
+    ret.videoSource = ret.externalVideoUrl ? 'external' : 'stored'
     ret.poster = ret.posterFile ? `${base}/poster` : null
     ret.size = ret.videoFile?.size ?? 0
-    for (const key of ['category', 'videoFile', 'posterFile', 'createdBy', 'updatedAt']) delete ret[key]
+    for (const key of ['externalVideoUrl', 'category', 'videoFile', 'posterFile', 'createdBy', 'updatedAt']) delete ret[key]
     return ret
   }),
 })
 
+videoSchema.pre('validate', function validateVideoSource(next) {
+  if (Boolean(this.videoFile) === Boolean(this.externalVideoUrl)) {
+    this.invalidate('videoFile', 'Provide either an uploaded video file or an external video URL.')
+  }
+  next()
+})
+
 videoSchema.index({ status: 1, category: 1, order: 1 })
 
 export const Video = mongoose.model('Video', videoSchema)

backend/src/services/storage/index.js

@@ -139,13 +139,13 @@ export const storageService = {
   },
 
   /** Returns readable file stream descriptor by looking up stored driver name. */
-  async describe(ref) {
+  async describe(ref, cachedMetadata) {
     const driver = driverFor(ref.driver)
     try {
-      const { size } = await driver.stat(ref.key)
+      const { size } = cachedMetadata ?? await driver.stat(ref.key)
       return {
         size,
-        mimeType: ref.mimeType,
+        mimeType: cachedMetadata?.mimeType ?? ref.mimeType,
         createReadStream: (range) => driver.createReadStream(ref.key, range),
       }
     } catch (err) {