[SohamRatnaparkhi]

Adds sync-datalake.yml, which keeps the public docs in the team's internal knowledge data lake (internal-knowledge-data-lake / common-collection). The Hydra MCP will search that database next to the internal wiki, Slack and Linear. It is separate from the website FAQ sync (sync-faq.yml): it uses a different database and a different secret, and the two syncs never touch each other's content.

What gets ingested. Every page in the v2 navigation of docs.json (Guides, Cookbooks, API Reference) becomes one knowledge_base source with id mdoc-<page>. v1 and archive/ are skipped. API reference MDX is mostly a stub, so the operation's parameters, body fields and responses are rendered from openapi.json into the body.

Metadata. These fields are shared with the internal wiki's sync, and scripts/datalake_taxonomy.json must stay identical to the copy in cortex-internal-docs:

Field	Values for Mintlify pages
origin / audience	mintlify_docs / public
doc_kind	guide, plugin, cookbook, api_reference
product_area	comma-joined, mapped by page prefix or OpenAPI path
lifecycle / api_version	shipped / v2
endpoint	for example POST /query, from the openapi: frontmatter
section, path, tickets	nav tab and group, repo path, PRO-xxxx ids from the page's commits

Runs. Every triggered push to main and manual dispatch reconciles the full navigation, recovering any changes missed by a failed run. Each run then deletes data-lake sources this sync owns (filtered by origin=mintlify_docs and the mdoc- prefix) that have left the navigation. An unreadable page or a value outside the vocabulary aborts the run before it ingests or deletes anything.

scripts/ is gitignored, so the new files are force-added the same way sync_faq.py is.

Before merging: add the HYDRADB_DATALAKE_API_KEY repo secret.

Verified with a full run: all 81 pages were indexed. Filtered queries for audience=public with doc_kind=api_reference, endpoint=POST /query, and product_area contains_any [metadata, retrieval] returned only matching pages.

---

^{Need help on this PR? Tag @codesmith-bot with what you need.}

[mintlify]

Preview deployment for your docs. Learn more about Mintlify Previews.

Project	Status	Preview	Updated
cortex-ai	🟢 Ready	View Preview	Oct 22, 2026, 2:23 PM

💡 Tip: Enable Automations to automatically generate PRs for you.

[github-actions]

✅ Mintlify Hygiene

No issues found.

[openhack-agent]

✅ OpenHack Summary

Security review of ci(datalake): sync v2 docs into the team data lake. 6 changed files; 0 findings at or above the low reporting threshold.

0   0   0   0

Confidence Score: 5/5

No reportable security findings were detected in this scan.

Security merge-readiness rubric: 1 = critical, 2 = high, 3 = medium, 4 = low, 5 = no reportable findings. This score reflects scan findings, not a guarantee of correctness or complete coverage.

Files Needing Attention: None

Important Files Changed

• .github/workflows/ci.yml (modified)
• .github/workflows/sync-datalake.yml (added)
• docs/learnings/tooling/001-2026-10-22-14-00-00-sync-outcomes.md (added)
• scripts/datalake_taxonomy.json (added)
• scripts/sync_datalake.py (added)
• scripts/test_sync_datalake.py (added)

Last reviewed commit: 827b237 · View review on OpenHack

---

TIP: Mention @openhack-agent in a PR comment to request a review or ask a question. Use @openhack-agent fix all for every finding, or @openhack-agent fix unresolved threads for open review threads only.

[openhack-agent]

OpenHack reviewed this commit. See the OpenHack Summary for the confidence score and fix actions.

[greptile-apps]

Confidence Score: 5/5

[Medium impact] The PR appears safe to merge after the required HYDRADB_DATALAKE_API_KEY secret is configured.

Summary

Adds a separate sync that copies v2 public documentation into the team's data lake, with shared metadata and OpenAPI details.

• Internal search can find the public v2 docs.
• The data lake stays matched to the current v2 navigation.

Diagram

%%{init: {'theme': 'neutral'}}%%
flowchart TD
  A[Push to main or manual run] --> B[Run sync tests]
  B --> C[Build all v2 pages and metadata]
  C --> D[Add OpenAPI details]
  D --> E[Ensure metadata schema]
  E --> F[Ingest pages and wait until searchable]
  F --> G[List sources owned by this sync]
  G --> H[Delete pages no longer in navigation]
  H --> I[Confirm each deletion]

Loading

_{Reviews (2) · Last reviewed commit: "fix(datalake): recover missed updates an..." · Reviewed by Greptile}