[suguanYang]

Summary

We complete the next local performance changes after #448 without changing production settings or indexes:

• Reuse an immutable, content-versioned demo ZIP and raw artifact prefix across materializations. We retain owner-scoped jobs, documents, publication transactions and retrieval authorization. Prefix-aware artifact readers preserve ordinary Worker output and existing per-job results.
• Prepare canonical bundles before user requests with a dry-run-by-default CLI and explicit --upload operation.
• Optionally reuse pure demo tokenization and token-frequency preparation in a bounded process-local cache. Namespace state, generated identifiers, scores and retrieval results are not cached.
• Prepare guarded retirement of only idx_document_map_unit_tokens_lookup. The new migration retains it by default; replayable operator commands apply or restore it at the same Alembic head.

Observed effects

Local observation	Before	After / result
SpaceX materialization with duplicate ZIP/raw uploads	40.422 / 39.098 s	Canonical reuse: 5.483 / 5.337 s
Duplicate stored data per repeated SpaceX materialization	637 objects / 439,190,627 bytes	Zero new or rewritten objects
SpaceX with canonical artifacts, preparation cache disabled	4.887 / 5.051 s	Cache cold: 7.044 s; warm: 2.967 s
Token persistence, disposable 12.42M-token cold clone	4.228 s	2.624 s after legacy lookup retirement
Publication transaction in that clone	11.376 s	10.182 s

The cache has a first-fill cost and a per-process warm-up requirement. These are focused local observations, not production latency guarantees. The large-clone index experiment predates canonical bundle reuse; its total HTTP time remained about 54 seconds. We do not combine separate experiments into a claimed production result or claim full SpaceX parsing below ten seconds.

Verification

• Canonical bundle and publication-owner trace contracts: 6 passed.
• Storage/document lifecycle contracts: 8 passed; existing affected retrieval checks: 48 passed.
• Bundle preparation CLI contracts: 3 passed. An isolated upload-mode run reported reused and retained the object count at 6383.
• Preparation-cache HTTP/PostgreSQL contract compares flag-off, cold-on, warm-on and rollback-off persisted search fields, map-unit semantics, token/hash/frequency rows, IDF statistics and namespace-scoped retrieval.
• Index migration/operator contracts: 8 passed against disposable PostgreSQL databases, including default retention, exact prerequisite checks, idempotent apply/restore at unchanged head and both migration transaction modes.
• Real SpaceX HTTP materialization checks cover ZIP/image/table/page downloads, classic retrieval assets, shared-object ETag/size/mtime equality and archive isolation. All runtime documents were archived.
• After restarting the API and Worker on the final source, real HTTP upload and confirmation traversed Redis/Celery, the actual Markdown parser, publication, classic retrieval and archive: 4.300 s client-observed and 3.814 s job duration. The fresh Worker log confirms ownership of the completed job. The configured summarization provider returned HTTP 401, so this run exercised the Markdown parser's fallback; successful external LLM/PDF parsing was not validated. This is a small Worker-path regression, not a full SpaceX parse benchmark.
• Ruff and targeted Pyright passed for all changed Python files; git diff --check passed. No broad repeated benchmark campaign was run.

Commands for the newly added focused contracts:

uv run pytest -q apps/api/tests/contract/test_canonical_demo_bundle_contract.py \
  apps/api/tests/contract/test_demo_publication_trace_owner_contract.py
uv run pytest -q apps/api/tests/contract/test_demo_bundle_preparation_contract.py
uv run pytest -q apps/api/tests/contract/test_demo_preparation_cache_contract.py
uv run pytest -q apps/api/tests/migrations/test_legacy_token_lookup_retirement_contract.py

Deployment Notes / DevOps

Controls and rollout order

Control	Default	Scope
DEMO_CANONICAL_BUNDLE_ENABLED	false	API creates canonical references for new demo materializations
DEMO_PUBLICATION_PREPARATION_CACHE_ENABLED	false	API caches pure preparation when canonical content versions are available; bounded to 64 MiB / 4096 entries per process
Legacy lookup retirement	Retained	Explicit operator action; neither demo flag changes indexes

1. Deploy the compatible writer and artifact readers. Ordinary deployment records migration 3c4d5e6f7a8b without deleting the lookup index. Queue contracts and the existing publication strategy are unchanged.

2. With the deployed source files and the API's protected storage/Redis environment, inspect and then prepare the bundle:

   uv run --package knowhere-api-app python apps/api/scripts/prepare_demo_bundles.py demo-spacex-s1
   uv run --package knowhere-api-app python apps/api/scripts/prepare_demo_bundles.py demo-spacex-s1 --upload

   Preparation creates no jobs, documents, credits or retrieval state. A completed content version is reused. New, unprepared source versions still need an initial upload.

3. Enable DEMO_CANONICAL_BUNDLE_ENABLED=true on API instances and verify materialization plus actual artifact downloads. Optionally enable the separate preparation cache; each API process fills its own cache.

4. Treat index retirement as a separate, explicitly approved database operation. Supply DATABASE_URL and applicable DB_SSL_* settings through the protected environment, then inspect/apply:

   uv run --package knowhere-api-app python scripts/retire_legacy_token_lookup.py
   uv run --package knowhere-api-app python scripts/retire_legacy_token_lookup.py --apply

   The operator validates replacement definitions and validity/readiness, uses concurrent DDL and reads the catalog back. Keep the binary lookup, unit lookup, unit index and primary key. Observe real publication/retrieval and token-stage I/O before accepting the rollout.

Rollback and artifact lifecycle

• Set DEMO_PUBLICATION_PREPARATION_CACHE_ENABLED=false and restart API processes to disable preparation reuse and clear process memory. Persisted retrieval data is unchanged.

• Set DEMO_CANONICAL_BUNDLE_ENABLED=false to restore per-job uploads for new materializations. Keep prefix-aware readers deployed for existing canonical results. A binary rollback to older readers requires per-job artifact restoration first.

• Preserve shared canonical objects for their full reference lifetime. Per-document archive must not delete them; storage lifecycle rules must preserve the canonical prefix. Reuse checks the marker and ZIP size, not every raw object. If a raw object is deleted after completion, invalidate that exact version's ready marker and rerun preparation against the same source version.

• Restore the legacy index without downgrading schema history:

  uv run --package knowhere-api-app python scripts/retire_legacy_token_lookup.py --restore

  Restoration scans the token table and requires time, disk and I/O; application rollback alone does not recreate it. Coordinate one index operation at a time and inspect interrupted concurrent DDL before retrying.

Detailed evidence and procedures:

• docs/design/canonical-demo-bundle-reuse.md
• docs/design/canonical-demo-preparation.md
• docs/design/demo-publication-preparation-cache.md
• docs/design/legacy-token-lookup-retirement.md
• docs/design/publication-scale-cold-cache-findings.md

Checklist

• Tests were added or updated when behavior changed
• Public docs, examples, or OpenAPI contracts were updated when needed
• Database migrations are idempotent and safe to deploy
• Logs, errors, and validation paths avoid leaking secrets or user data
• The pull request description explains any breaking or user-visible change