diff --git a/.agents/skills/mcp-tools/SKILL.md b/.agents/skills/mcp-tools/SKILL.md index 16b0d313..eb0c7ac3 100644 --- a/.agents/skills/mcp-tools/SKILL.md +++ b/.agents/skills/mcp-tools/SKILL.md @@ -37,17 +37,20 @@ When adding or changing a tool: AI Search resources in this repo use a common shape: ```text -docs in R2 -> Cloudflare AI Search vector index -> shared MCP runtime -> source-specific MCP tools +Items API uploads -> Cloudflare built-in storage and vector index -> shared MCP runtime -> source-specific MCP tools ``` For a new AI Search-backed document source: -1. Create or adopt the R2 content bucket and AI Search namespaces in `createAgentResources` in `apps/api/infra/resources.ts`. -2. Bind `AI_SEARCH`, `WORKFLOW_AI_SEARCH`, and `AI_SEARCH_CONTENT_BUCKET` in `apps/api/infra/index.ts`. +1. Create or adopt AI Search namespaces in `createAgentResources` in `apps/api/infra/resources.ts`. New instances use built-in storage through the namespace binding. +2. Bind `AI_SEARCH` and `WORKFLOW_AI_SEARCH` in `apps/api/infra/index.ts`. Retain `AI_SEARCH_CONTENT_BUCKET` only for explicit legacy-content transition and saved workflow compatibility; it is not a source for new instances. 3. Add public source metadata in `packages/shared/src/session-tools.ts`. 4. Persist runtime source records through `AiSearchRegistryStore` in `packages/api/src/server/background/db/ai-search.ts`, keeping `maxResults` explicit. Tool names come from `getAiSearchMcpToolNames`. 5. Keep MCP tool registration in `packages/api/src/server/mcp/ai-search-server.ts`. Dispatch into the shared runtime in `packages/api/src/server/mcp/ai-search-runtime.ts`. -6. Run the MCP integration tests and update any session UI source picker affected by the new source. +6. Upload and delete documents through the namespace Items API. Workflows use the `ai-search-upload` storage adapter, with administrator authorization for global sources. +7. Run the MCP integration tests and update any session UI source picker affected by the new source. Verify indexing and retrieval against an isolated real namespace. + +Keep legacy R2/website source records readable and exportable, with an explicit migration state. Do not rewrite their storage type or delete original content automatically. Follow `docs/ai-search.md` for the transition. Each AI Search source should expose two MCP tools: diff --git a/.agents/skills/verify-solzero/SKILL.md b/.agents/skills/verify-solzero/SKILL.md index c2c16fa7..e7944c44 100644 --- a/.agents/skills/verify-solzero/SKILL.md +++ b/.agents/skills/verify-solzero/SKILL.md @@ -29,7 +29,7 @@ nub exec e2e list --changed-since origin/master nub run test:e2e -- --changed-since origin/master ``` -The environment loader reads the selected private file, then `config/.env` and `config/.dev.vars`; existing environment values win. Supply `CLOUDFLARE_ACCOUNT_ID` and `CF_AI_GATEWAY_E2E_TOKEN`. The launcher boots local `alchemy dev` and does not read `CLOUDFLARE_API_TOKEN`. Never print credential values. The launcher creates its owned run’s `.dev.vars` with mode 0600 and generates the isolated administrator password in an ignored file. Use the suite's registered secret handles for browser credential input. +The environment loader reads the selected private file, then `config/.env` and `config/.dev.vars`; existing environment values win. Supply `CLOUDFLARE_ACCOUNT_ID` and `CF_AI_GATEWAY_E2E_TOKEN`. The ordinary core launcher boots local `alchemy dev` and does not read `CLOUDFLARE_API_TOKEN`. The explicit isolated AI Search launcher documented in `docs/e2e.md` supplies that credential only when its profile enables native remote AI Search bindings. Never print credential values. The launcher creates its owned run’s `.dev.vars` with mode 0600 and generates the isolated administrator password in an ignored file. Use the suite's registered secret handles for browser credential input. `nub run test` runs core flows, including Isolate, OpenCode, and Codex prompts against the mock `AiGateway` layer, which echoes the user message. `nub run test:e2e:all` adds the deferred external integrations and Claude Code; `nub run test:e2e:external` selects only those deferred cases. Focus one area with `nub run test:e2e -- tests/.e2e.ts` after reading its feature file under `features/`. Read External fixture preflight in `docs/e2e.md` before deferred suites. Missing integration credentials or linked identities are blockers, and collection alone does not establish coverage. @@ -47,6 +47,6 @@ Cloud-agent proof follows [Cloud-agent PR artifacts](../../../AGENTS.md#cloud-ag Use only the isolated e2e profile and disposable fixtures. Do not run `infra:deploy:*` or `db:copy-d1-to-planetscale` during verification. Preserve pinned Alchemy, Effect, Wrangler, and provider versions unless the requested fix requires a change. The e2e launcher owns process cleanup; preserve unrelated development stacks. Delete temporary integration fixtures and revoke minted tokens in `finally` blocks. -When e2e stops the app target, the target stops `alchemy dev`, removes the run's labelled Docker containers and images, and removes `.e2e/runs//`. It never calls the Cloudflare API. Preserve the replay cache, the shared `default` gateway, and unrelated resources. Runs are local only, so there is no recovery command; after a SIGKILL, stop leftover processes by hand. See `docs/e2e.md` for remote preview limitations. +When e2e stops the app target, the target stops `alchemy dev`, removes the run's labelled Docker containers and images, and removes `.e2e/runs//`. The ordinary core launcher never calls the Cloudflare API. The separate AI Search launcher creates unique owned namespaces, deletes their instances and namespaces, verifies absence, and records ownership receipts. Preserve the replay cache, the shared `default` gateway, and unrelated resources. Runs are local only, so there is no recovery command; after a SIGKILL, stop leftover processes by hand. See `docs/e2e.md` for remote preview limitations. The previous `control-solzero` Chrome driver and its recipes were removed. Keep new regression cases in the shared e2e suite rather than rebuilding that driver. diff --git a/.agents/skills/verify-solzero/features/external-integrations.md b/.agents/skills/verify-solzero/features/external-integrations.md index 243eb46b..dd97bfab 100644 --- a/.agents/skills/verify-solzero/features/external-integrations.md +++ b/.agents/skills/verify-solzero/features/external-integrations.md @@ -30,3 +30,8 @@ Keep these product facts honest: The user deferred live positive results for these fixtures. Authored cases and preflight remain available; do not claim green external coverage without a real passing run. Broader linking/clone/webhook/event paths remain unverified in `docs/e2e.md`. + +Built-in AI Search creation, Items upload/index/search/delete, workflow ingestion, and Admin UI +checks live in `tests/ai-search.external.e2e.ts`. Use the isolated launcher in `docs/e2e.md`; +it creates a fresh namespace pair and verifies cleanup. No seeded standing source is reused. +An API proof does not establish the separate browser authoring flow. diff --git a/.e2e/cache/489d9767ef99fb3548294b05cb669e253fbb6ae1e185ea97edbc6e80bfc46a37.json b/.e2e/cache/489d9767ef99fb3548294b05cb669e253fbb6ae1e185ea97edbc6e80bfc46a37.json new file mode 100644 index 00000000..29942009 --- /dev/null +++ b/.e2e/cache/489d9767ef99fb3548294b05cb669e253fbb6ae1e185ea97edbc6e80bfc46a37.json @@ -0,0 +1,108 @@ +{ + "schemaVersion": "trace-1", + "createdAt": "2026-11-09T16:29:47.957Z", + "payload": { + "actions": [ + { + "name": "tap", + "summary": "tap button \"New Source\"", + "target": { + "role": "button", + "name": "New Source" + } + } + ], + "executor": { + "name": "decision", + "version": "2" + }, + "recordedFor": { + "testId": "tests/ai-search.external.e2e.ts::administrators%20create%20built-in%20sources%20and%20open%20their%20documents", + "targetId": "local-cloudflare", + "instructionDigest": "a09dac185a810615ce3ee09265cc9c5ec6490c161918d7e86dbb71c0863cadc2", + "paramsDigest": "44136fa355b3678a1146ad16f7e8649e94fb4fc21fe77e8310c060f61caaff8a", + "callIndex": 0, + "agent": "default" + }, + "recordedBy": { + "provider": "anthropic.messages", + "model": "anthropic/claude-haiku-5.5" + }, + "summary": "The screen shows the step is done (p=0.968, confidence 0.905).", + "startPath": "/admin/ai-search", + "endPath": "/admin/ai-search", + "endAnchors": [ + { + "role": "heading", + "name": "Create Source" + }, + { + "text": "Configure the Cloudflare AI Search instance exposed to agent sessions." + }, + { + "role": "button", + "name": "Close AI Search source dialog" + }, + { + "text": "ID" + }, + { + "text": "Used as the source ID, Cloudflare AI Search instance ID, and basis for MCP tool names." + }, + { + "role": "textbox", + "name": "ID", + "placeholder": "product-docs" + }, + { + "text": "Label" + }, + { + "text": "Human-readable name shown to admins and used in agent-facing tool descriptions." + } + ], + "goneAnchors": [ + { + "role": "link", + "name": "Fork Fixture" + }, + { + "role": "link", + "name": "Agents" + }, + { + "role": "button", + "name": "Expand agents menu" + }, + { + "role": "link", + "name": "Workflows" + }, + { + "role": "link", + "name": "Bots" + }, + { + "text": "Mini Apps (coming soon)" + }, + { + "text": "Admin" + }, + { + "role": "link", + "name": "AI Search" + } + ], + "endWaitMs": 11299, + "keyedBy": { + "cacheSchema": "trace-1", + "policyVersion": "conservative/6", + "project": "356bb4dad9f823ea5064055664450c1dbb1e4d9cbbfb201ca71c7993a2c18dcd", + "platform": "web", + "engineName": "web", + "engineSpiVersion": 1, + "appIdentity": "e18a7246aa64bc6cabcfc1fff593fa36af4d2bccf2dc172e6d2c8d9ad9cb96af", + "agentContextDigest": "44e79ebe1b494e7f7d7911f15d6f0a211137848d9b4d3549ff598d1cc728f3e8" + } + } +} diff --git a/.e2e/cache/951e6f616c948555ec0d2b95e74d33be36c99ac18030d6d14ff8b7d9397b9e93.json b/.e2e/cache/951e6f616c948555ec0d2b95e74d33be36c99ac18030d6d14ff8b7d9397b9e93.json new file mode 100644 index 00000000..f477402c --- /dev/null +++ b/.e2e/cache/951e6f616c948555ec0d2b95e74d33be36c99ac18030d6d14ff8b7d9397b9e93.json @@ -0,0 +1,75 @@ +{ + "schemaVersion": "trace-1", + "createdAt": "2026-11-09T16:29:47.958Z", + "payload": { + "actions": [ + { + "name": "tap", + "summary": "tap button \"Documents\" in \"{{param:/label}}\"", + "target": { + "role": "button", + "name": "Documents", + "within": "{{param:/label}}" + } + } + ], + "executor": { + "name": "decision", + "version": "2" + }, + "recordedFor": { + "testId": "tests/ai-search.external.e2e.ts::administrators%20create%20built-in%20sources%20and%20open%20their%20documents", + "targetId": "local-cloudflare", + "instructionDigest": "416778188b6ffd4dbdb3a735e3dd1f0e3077defec59f5954becffa183587b79e", + "paramsDigest": "44fcbcfb85a7779b6de0f181cfc52ea37d67de119015c2eaf599e1f9843e0d7f", + "callIndex": 0, + "agent": "default" + }, + "recordedBy": { + "provider": "anthropic.messages", + "model": "anthropic/claude-haiku-5.5" + }, + "summary": "The screen shows the step is done (p=0.924, confidence 0.787).", + "startPath": "/admin/ai-search", + "endPath": "/admin/ai-search", + "endAnchors": [ + { + "text": "Close documents" + }, + { + "text": "Uploads are indexed asynchronously. Refresh to check processing status. A matching filename replaces that document. PDF and text files support up to 10 MiB; other formats support up to 4 MiB." + }, + { + "role": "button", + "name": "Upload document" + }, + { + "text": "Refresh documents" + }, + { + "text": "Previous documents" + }, + { + "text": "Page 1" + }, + { + "text": "Next documents" + }, + { + "role": "button", + "name": "Close documents" + } + ], + "endWaitMs": 11434, + "keyedBy": { + "cacheSchema": "trace-1", + "policyVersion": "conservative/6", + "project": "356bb4dad9f823ea5064055664450c1dbb1e4d9cbbfb201ca71c7993a2c18dcd", + "platform": "web", + "engineName": "web", + "engineSpiVersion": 1, + "appIdentity": "e18a7246aa64bc6cabcfc1fff593fa36af4d2bccf2dc172e6d2c8d9ad9cb96af", + "agentContextDigest": "44e79ebe1b494e7f7d7911f15d6f0a211137848d9b4d3549ff598d1cc728f3e8" + } + } +} diff --git a/.e2e/cache/a4ad674d38365e96c0df243294f7ce5cf6fe5be5ea05f53222c16ae9bdb6bd7f.json b/.e2e/cache/a4ad674d38365e96c0df243294f7ce5cf6fe5be5ea05f53222c16ae9bdb6bd7f.json new file mode 100644 index 00000000..0e97402f --- /dev/null +++ b/.e2e/cache/a4ad674d38365e96c0df243294f7ce5cf6fe5be5ea05f53222c16ae9bdb6bd7f.json @@ -0,0 +1,62 @@ +{ + "schemaVersion": "trace-1", + "createdAt": "2026-11-09T16:29:47.958Z", + "payload": { + "actions": [ + { + "name": "tap", + "summary": "tap button \"Create Source\"", + "target": { + "role": "button", + "name": "Create Source" + } + } + ], + "executor": { + "name": "decision", + "version": "2" + }, + "recordedFor": { + "testId": "tests/ai-search.external.e2e.ts::administrators%20create%20built-in%20sources%20and%20open%20their%20documents", + "targetId": "local-cloudflare", + "instructionDigest": "4cd8aaec5b19f7210f68f2226e1c58bb99802b65701db0dbed3b7b08bdfed84a", + "paramsDigest": "44fcbcfb85a7779b6de0f181cfc52ea37d67de119015c2eaf599e1f9843e0d7f", + "callIndex": 0, + "agent": "default" + }, + "recordedBy": { + "provider": "anthropic.messages", + "model": "anthropic/claude-haiku-5.5" + }, + "summary": "The screen shows the step is done (p=0.658, confidence 0.273).", + "startPath": "/admin/ai-search", + "endPath": "/admin/ai-search", + "endAnchors": [ + { + "role": "status", + "name": "Loading" + }, + { + "role": "button", + "name": "Loading Create Source" + } + ], + "goneAnchors": [ + { + "role": "button", + "name": "Create Source" + } + ], + "endWaitMs": 11113, + "keyedBy": { + "cacheSchema": "trace-1", + "policyVersion": "conservative/6", + "project": "356bb4dad9f823ea5064055664450c1dbb1e4d9cbbfb201ca71c7993a2c18dcd", + "platform": "web", + "engineName": "web", + "engineSpiVersion": 1, + "appIdentity": "e18a7246aa64bc6cabcfc1fff593fa36af4d2bccf2dc172e6d2c8d9ad9cb96af", + "agentContextDigest": "44e79ebe1b494e7f7d7911f15d6f0a211137848d9b4d3549ff598d1cc728f3e8" + } + } +} diff --git a/.github/workflows/deploy.yml b/.github/workflows/deploy.yml index cc73e38c..77bc32ca 100644 --- a/.github/workflows/deploy.yml +++ b/.github/workflows/deploy.yml @@ -79,7 +79,6 @@ jobs: GITHUB_APP_PRIVATE_KEY: ${{ secrets.GH_APP_PRIVATE_KEY }} GITHUB_APP_WEBHOOK_SECRET: ${{ secrets.GH_APP_WEBHOOK_SECRET }} SLACK_TOKEN: ${{ secrets.SLACK_TOKEN }} - CF_AI_SEARCH_SERVICE_TOKEN_ID: ${{ secrets.CF_AI_SEARCH_SERVICE_TOKEN_ID }} run: | nub run config:write-stage-vars -- ${{ inputs.environment }} config/.${{ inputs.environment }}.vars @@ -91,5 +90,4 @@ jobs: CLOUDFLARE_API_TOKEN: ${{ secrets.CLOUDFLARE_API_TOKEN }} PLANETSCALE_SERVICE_TOKEN_ID: ${{ secrets.PLANETSCALE_SERVICE_TOKEN_ID }} PLANETSCALE_SERVICE_TOKEN: ${{ secrets.PLANETSCALE_SERVICE_TOKEN }} - CF_AI_SEARCH_SERVICE_TOKEN_ID: ${{ secrets.CF_AI_SEARCH_SERVICE_TOKEN_ID }} WRANGLER_SEND_METRICS: "false" diff --git a/.github/workflows/pr-preview.yml b/.github/workflows/pr-preview.yml index 3627d997..5603b0a4 100644 --- a/.github/workflows/pr-preview.yml +++ b/.github/workflows/pr-preview.yml @@ -145,7 +145,6 @@ jobs: GITHUB_APP_PRIVATE_KEY: ${{ secrets.GH_APP_PRIVATE_KEY }} GITHUB_APP_WEBHOOK_SECRET: ${{ secrets.GH_APP_WEBHOOK_SECRET }} SLACK_TOKEN: ${{ secrets.SLACK_TOKEN }} - CF_AI_SEARCH_SERVICE_TOKEN_ID: ${{ secrets.CF_AI_SEARCH_SERVICE_TOKEN_ID }} run: | nub run config:write-stage-vars -- pre config/.pre.vars @@ -165,7 +164,6 @@ jobs: CLOUDFLARE_API_TOKEN: ${{ secrets.CLOUDFLARE_API_TOKEN }} PLANETSCALE_SERVICE_TOKEN_ID: ${{ secrets.PLANETSCALE_SERVICE_TOKEN_ID }} PLANETSCALE_SERVICE_TOKEN: ${{ secrets.PLANETSCALE_SERVICE_TOKEN }} - CF_AI_SEARCH_SERVICE_TOKEN_ID: ${{ secrets.CF_AI_SEARCH_SERVICE_TOKEN_ID }} WRANGLER_SEND_METRICS: "false" PREVIEW_STAGE: ${{ steps.deploy-lifetime.outputs.stage }} @@ -264,7 +262,6 @@ jobs: GITHUB_APP_PRIVATE_KEY: ${{ secrets.GH_APP_PRIVATE_KEY }} GITHUB_APP_WEBHOOK_SECRET: ${{ secrets.GH_APP_WEBHOOK_SECRET }} SLACK_TOKEN: ${{ secrets.SLACK_TOKEN }} - CF_AI_SEARCH_SERVICE_TOKEN_ID: ${{ secrets.CF_AI_SEARCH_SERVICE_TOKEN_ID }} run: | nub run config:write-stage-vars -- pre config/.pre.vars @@ -281,7 +278,6 @@ jobs: env: CLOUDFLARE_ACCOUNT_ID: ${{ secrets.CLOUDFLARE_ACCOUNT_ID }} CLOUDFLARE_API_TOKEN: ${{ secrets.CLOUDFLARE_API_TOKEN }} - CF_AI_SEARCH_SERVICE_TOKEN_ID: ${{ secrets.CF_AI_SEARCH_SERVICE_TOKEN_ID }} WRANGLER_SEND_METRICS: "false" PREVIEW_STAGE: ${{ steps.cleanup-lifetime.outputs.stage }} diff --git a/.github/workflows/preview.yml b/.github/workflows/preview.yml index 71ffdc8e..4fd92823 100644 --- a/.github/workflows/preview.yml +++ b/.github/workflows/preview.yml @@ -111,7 +111,6 @@ jobs: GITHUB_APP_PRIVATE_KEY: ${{ secrets.GH_APP_PRIVATE_KEY }} GITHUB_APP_WEBHOOK_SECRET: ${{ secrets.GH_APP_WEBHOOK_SECRET }} SLACK_TOKEN: ${{ secrets.SLACK_TOKEN }} - CF_AI_SEARCH_SERVICE_TOKEN_ID: ${{ secrets.CF_AI_SEARCH_SERVICE_TOKEN_ID }} run: | nub run config:write-stage-vars -- pre config/.pre.vars @@ -122,6 +121,5 @@ jobs: CLOUDFLARE_API_TOKEN: ${{ secrets.CLOUDFLARE_API_TOKEN }} PLANETSCALE_SERVICE_TOKEN_ID: ${{ secrets.PLANETSCALE_SERVICE_TOKEN_ID }} PLANETSCALE_SERVICE_TOKEN: ${{ secrets.PLANETSCALE_SERVICE_TOKEN }} - CF_AI_SEARCH_SERVICE_TOKEN_ID: ${{ secrets.CF_AI_SEARCH_SERVICE_TOKEN_ID }} WRANGLER_SEND_METRICS: "false" ALCHEMY_STAGE: pre diff --git a/.github/workflows/validate.yml b/.github/workflows/validate.yml index a6c8d62f..ee6d229b 100644 --- a/.github/workflows/validate.yml +++ b/.github/workflows/validate.yml @@ -299,7 +299,8 @@ jobs: needs: e2e # A dispatched e2e_record run commits its recordings to the branch it ran on. # The App token push starts pull request CI; a GITHUB_TOKEN push would not. - if: github.event_name == 'workflow_dispatch' && inputs.e2e_record && inputs.e2e_commit && github.ref_name != github.event.repository.default_branch + # Manual runs skip release detection; override inherited skip handling, but require green tests. + if: ${{ !cancelled() && needs.e2e.result == 'success' && github.event_name == 'workflow_dispatch' && inputs.e2e_record && inputs.e2e_commit && github.ref_name != github.event.repository.default_branch }} runs-on: ${{ vars.S0_CI_RUNNER || 'ubuntu-latest' }} timeout-minutes: 10 permissions: diff --git a/.tegami/2026-11-09-ai-search-built-in-storage.md b/.tegami/2026-11-09-ai-search-built-in-storage.md new file mode 100644 index 00000000..05b669e9 --- /dev/null +++ b/.tegami/2026-11-09-ai-search-built-in-storage.md @@ -0,0 +1,16 @@ +--- +packages: + "release:solzero": major +--- + +## Upload documents directly to AI Search + +AI Search sources now use built-in storage. Administrators can upload and delete documents and +inspect indexing status without an external-storage service token. Workflows can ingest text and +JSON with the new Upload to AI Search node. + +R2 and website source creation is removed. Existing source records and original content remain +intact and are marked as requiring migration; these sources are unavailable for retrieval until +operators copy their originals into new built-in sources and update their selections. Follow +`docs/ai-search.md` before cutover. Deployment does not copy or delete legacy content. Workflow +artifact and skill storage remain unchanged. diff --git a/.tegami/2026-11-09-commit-manual-test-recordings.md b/.tegami/2026-11-09-commit-manual-test-recordings.md new file mode 100644 index 00000000..893ca39d --- /dev/null +++ b/.tegami/2026-11-09-commit-manual-test-recordings.md @@ -0,0 +1,9 @@ +--- +packages: + "release:solzero": patch +--- + +## Complete manual test-recording refreshes + +Successful manual validation runs now execute the requested recording-commit step when release +detection is skipped. Failed or cancelled tests cannot publish recordings. diff --git a/apps/api/infra/index.ts b/apps/api/infra/index.ts index ddfdf7cb..e9bdba96 100644 --- a/apps/api/infra/index.ts +++ b/apps/api/infra/index.ts @@ -82,7 +82,6 @@ export interface ApiInfraEnv { readonly GITHUB_APP_SLUG: string readonly GITHUB_APP_WEBHOOK_SECRET: ApiSecretInput readonly SLACK_TOKEN: ApiSecretInput - readonly CF_AI_SEARCH_SERVICE_TOKEN_ID: string } function includeWhenDefined( @@ -125,18 +124,6 @@ function secretValue( }) } -function configuredStringSecretValue( - secretBindings: Readonly>, - reference: SecretReference | null | undefined, -): string { - return Result.liftPredicate( - secretValue(secretBindings, reference), - P.isString, - () => invalidApiConfig(`${reference?.env ?? "Secret"} cannot be generated by Alchemy`), - // oxlint-disable-next-line effect/avoid-option-getorthrow -- Synchronous boundary for non-Effect callers: invalid input throws the typed error by contract. - ).pipe(Result.getOrThrow) -} - /** A literal secret must meet its minimum length. Alchemy-generated secrets are checked by Alchemy. */ const hasMinimumLength = (value: ApiSecretInput, minimum: number) => !P.isString(value) || value.length >= minimum @@ -283,10 +270,6 @@ export function getApiInfraEnv( BETTER_AUTH_SECRET: secretValue(secretBindings, config.security.betterAuthSecret), ...githubAppEnv(config.integrations.githubApp, secretBindings), SLACK_TOKEN: slackToken, - CF_AI_SEARCH_SERVICE_TOKEN_ID: configuredStringSecretValue( - secretBindings, - config.aiSearch.serviceTokenId, - ), } } @@ -465,7 +448,6 @@ function createApiBindings(options: CreateApiBindingsOptions) { GITHUB_APP_CLIENT_ID: env.GITHUB_APP_CLIENT_ID, GITHUB_APP_ID: env.GITHUB_APP_ID, GITHUB_APP_SLUG: env.GITHUB_APP_SLUG, - CF_AI_SEARCH_SERVICE_TOKEN_ID: env.CF_AI_SEARCH_SERVICE_TOKEN_ID, } const fixedSecrets = includeWhenDefined( diff --git a/apps/api/infra/resources.ts b/apps/api/infra/resources.ts index 61e79e75..12f9b60e 100644 --- a/apps/api/infra/resources.ts +++ b/apps/api/infra/resources.ts @@ -123,7 +123,6 @@ export interface CreateAgentResourcesOptions { dev: boolean stageMetadata: StageMetadata migrationsDir: string - tokenId: string databaseEngine: S0DatabaseEngine appDbMode: AppDbMode databaseUrl: string @@ -191,7 +190,7 @@ function createControlPlaneDatabase(options: CreateAgentResourcesOptions) { } export function createAgentResources(options: CreateAgentResourcesOptions) { - const { appName, dev, stageMetadata } = options + const { appName, stageMetadata } = options return Effect.gen(function* () { const sessionNamespace = Cloudflare.DurableObject("session", { @@ -232,12 +231,16 @@ export function createAgentResources(options: CreateAgentResourcesOptions) { name: `${appName}-skills-${stageMetadata.name}`, }) + // Retained for explicit legacy content migration and saved workflow artifacts. + // New ingestion uses built-in storage through the namespace Items API. const aiSearchContentBucket = yield* Cloudflare.R2.Bucket("ai-search-content", { name: `${appName}-ai-search-content-${stageMetadata.name}`, }) - const aiSearchNamespaces = yield* Match.value(dev).pipe( - Match.when(true, () => + const aiSearchNamespaces = yield* Match.value( + stageMetadata.infra.aiSearchNamespaceEnabled, + ).pipe( + Match.when(false, () => Effect.succeed({ aiSearchNamespace: undefined, workflowAiSearchNamespace: undefined, diff --git a/apps/web/src/components/admin-ai-search-panel.tsx b/apps/web/src/components/admin-ai-search-panel.tsx index 79f98c27..9ef6d964 100644 --- a/apps/web/src/components/admin-ai-search-panel.tsx +++ b/apps/web/src/components/admin-ai-search-panel.tsx @@ -17,13 +17,8 @@ import { useMemo, useState } from "react" import { S0Loader, TableCellState } from "@/components/s0-loader" import { CodeSurface } from "@/components/code" import { Dialog } from "@/components/ui/dialog" -import { - DocsSectionHeading, - EnvLockIcon, - Field, - SelectField, -} from "@/components/admin-ai-provider-panel-ui" -import { useDialogSelectPortalRoot } from "@/lib/dialog-select-portal" +import { DocsSectionHeading, EnvLockIcon, Field } from "@/components/admin-ai-provider-panel-ui" +import { AiSearchItemsPanel } from "./ai-search-items-panel" import { circleIconButton } from "@/components/ui/kumo-icon-button" type SourceDraft = { @@ -32,16 +27,6 @@ type SourceDraft = { description: string enabled: boolean maxResults: string - dataSourceType: AdminAiSearchDataSource["type"] - r2BucketName: string - r2Prefix: string - r2Jurisdiction: string - websiteDomain: string - websiteIncludePaths: string - websiteExcludePaths: string - websiteSpecificSitemaps: string - websiteUseBrowserRendering: boolean - websiteIncludeImages: boolean } const EMPTY_DRAFT: SourceDraft = { @@ -50,24 +35,8 @@ const EMPTY_DRAFT: SourceDraft = { description: "", enabled: true, maxResults: "5", - dataSourceType: "r2", - r2BucketName: "", - r2Prefix: "", - r2Jurisdiction: "", - websiteDomain: "", - websiteIncludePaths: "", - websiteExcludePaths: "", - websiteSpecificSitemaps: "", - websiteUseBrowserRendering: false, - websiteIncludeImages: false, } -const DATA_SOURCE_OPTIONS = [ - { value: "built-in", label: "Built-in" }, - { value: "r2", label: "R2" }, - { value: "website", label: "Website" }, -] - const SOURCE_FIELD_HELP = { sourceId: "Used as the source ID, Cloudflare AI Search instance ID, and basis for MCP tool names.", @@ -99,14 +68,14 @@ export function AdminAiSearchPanel({ const [draft, setDraft] = useState(EMPTY_DRAFT) const [fieldError, setFieldError] = useState("") - const dialogSelectPortalContainer = useDialogSelectPortalRoot() + const [documentSourceId, setDocumentSourceId] = useState(null) const editingSource = useMemo( () => data?.sources.find((source) => source.id === editingSourceId) ?? null, [data?.sources, editingSourceId], ) - const sourceLocked = editingSource?.locked ?? false + const sourceLocked = Boolean(editingSource?.locked || editingSource?.migrationRequired) const sourceLockEnvVar = sourceLocked ? editingSource?.envVarName : null const editingMode: "create" | "update" = editingSource ? "update" : "create" @@ -260,12 +229,28 @@ export function AdminAiSearchPanel({
{formatDataSource(source.dataSource)}
+ {source.migrationRequired && ( +

+ Migration required. Export the source and copy its documents to a new + built-in source before replacing references. Existing content is + retained. +

+ )}
+ {!source.migrationRequired && ( + + )} }>
@@ -501,171 +471,18 @@ export function AdminAiSearchPanel({ ) } -function DataSourceFields({ - draft, - disabled, - envVarName, - updateDraft, -}: { - draft: SourceDraft - disabled: boolean - envVarName: string | null | undefined - updateDraft: (key: Key, value: SourceDraft[Key]) => void -}) { - if (draft.dataSourceType === "built-in") { - return ( -

- Built-in sources do not require additional data source fields. -

- ) - } - - if (draft.dataSourceType === "r2") { - return ( -
- updateDraft("r2BucketName", value)} - placeholder="s0-ai-search-content-prod" - disabled={disabled} - disabledEnvVarName={envVarName} - required - /> - updateDraft("r2Prefix", value)} - placeholder="docs/" - disabled={disabled} - disabledEnvVarName={envVarName} - /> - updateDraft("r2Jurisdiction", value)} - placeholder="Optional" - disabled={disabled} - disabledEnvVarName={envVarName} - /> -
- ) - } - - return ( -
- updateDraft("websiteDomain", value)} - placeholder="docs.example.com" - disabled={disabled} - disabledEnvVarName={envVarName} - required - /> -
- updateDraft("websiteIncludePaths", event.target.value)} - className="min-h-28 w-full" - placeholder="/docs/*" - disabled={disabled} - readOnly={disabled} - /> - updateDraft("websiteExcludePaths", event.target.value)} - className="min-h-28 w-full" - placeholder="/blog/*" - disabled={disabled} - readOnly={disabled} - /> - updateDraft("websiteSpecificSitemaps", event.target.value)} - className="min-h-28 w-full" - placeholder="https://docs.example.com/sitemap.xml" - disabled={disabled} - readOnly={disabled} - /> -
-
- updateDraft("websiteUseBrowserRendering", value)} - /> - updateDraft("websiteIncludeImages", value)} - /> -
-
- ) -} - -function ToggleField({ - checked, - disabled, - label, - onChange, -}: { - checked: boolean - disabled: boolean - label: string - onChange: (value: boolean) => void -}) { - return ( - - ) -} - function SourceStatus({ enabled }: { enabled: boolean }) { return {enabled ? "Enabled" : "Disabled"} } function draftFromSource(source: AdminAiSearchSource): SourceDraft { - const base = { - ...EMPTY_DRAFT, + return { id: source.id, label: source.label, description: source.description, enabled: source.enabled, maxResults: String(source.maxResults), - dataSourceType: source.dataSource.type, - } - - if (source.dataSource.type === "r2") { - return { - ...base, - r2BucketName: source.dataSource.bucketName, - r2Prefix: source.dataSource.prefix ?? "", - r2Jurisdiction: source.dataSource.r2Jurisdiction ?? "", - } } - - if (source.dataSource.type === "website") { - return { - ...base, - websiteDomain: source.dataSource.domain, - websiteIncludePaths: source.dataSource.includePaths.join("\n"), - websiteExcludePaths: source.dataSource.excludePaths.join("\n"), - websiteSpecificSitemaps: source.dataSource.specificSitemaps.join("\n"), - websiteUseBrowserRendering: source.dataSource.useBrowserRendering, - websiteIncludeImages: source.dataSource.includeImages, - } - } - - return base } function payloadFromDraft(draft: SourceDraft): AdminAiSearchSourcePayload | string { @@ -681,58 +498,13 @@ function payloadFromDraft(draft: SourceDraft): AdminAiSearchSourcePayload | stri return "Max results must be an integer between 1 and 20." } - const dataSource = dataSourceFromDraft(draft) - - if (typeof dataSource === "string") { - return dataSource - } - return { id, label: draft.label.trim(), description: draft.description.trim(), enabled: draft.enabled, maxResults, - dataSource, - } -} - -function dataSourceFromDraft(draft: SourceDraft): AdminAiSearchDataSource | string { - switch (draft.dataSourceType) { - case "built-in": - return { type: "built-in" } - case "r2": { - const bucketName = draft.r2BucketName.trim() - - if (!bucketName) { - return "R2 bucket is required." - } - - return { - type: "r2", - bucketName, - prefix: draft.r2Prefix.trim() || null, - r2Jurisdiction: draft.r2Jurisdiction.trim() || null, - } - } - - case "website": { - const domain = draft.websiteDomain.trim() - - if (!domain) { - return "Website domain is required." - } - - return { - type: "website", - domain, - includePaths: parseList(draft.websiteIncludePaths), - excludePaths: parseList(draft.websiteExcludePaths), - specificSitemaps: parseList(draft.websiteSpecificSitemaps), - useBrowserRendering: draft.websiteUseBrowserRendering, - includeImages: draft.websiteIncludeImages, - } - } + dataSource: { type: "built-in" }, } } @@ -748,18 +520,3 @@ function formatDataSource(dataSource: AdminAiSearchDataSource): string { return `Website ${dataSource.domain}` } } - -function parseList(value: string): string[] { - return Array.from( - new Set( - value - .split(/[\n,]/) - .map((item) => item.trim()) - .filter(Boolean), - ), - ) -} - -function isDataSourceType(value: string): value is AdminAiSearchDataSource["type"] { - return value === "built-in" || value === "r2" || value === "website" -} diff --git a/apps/web/src/components/ai-search-items-panel.tsx b/apps/web/src/components/ai-search-items-panel.tsx new file mode 100644 index 00000000..ca4fda1d --- /dev/null +++ b/apps/web/src/components/ai-search-items-panel.tsx @@ -0,0 +1,124 @@ +"use client" + +import { useState } from "react" +import { useMutation, useQuery, useQueryClient } from "@tanstack/react-query" +import type { AdminAiSearchItemsResponse } from "@solzero/api" +import { Button } from "@cloudflare/kumo/components/button" +import { requestJson } from "@/lib/admin-console-actions" + +function fileContent(file: File): Promise { + return new Promise((resolve, reject) => { + const reader = new FileReader() + + reader.onload = () => resolve(String(reader.result).split(",", 2)[1]) + reader.onerror = () => reject(new Error("Could not read file")) + reader.readAsDataURL(file) + }) +} + +export function AiSearchItemsPanel({ + sourceId, + onClose, +}: { + sourceId: string + onClose: () => void +}) { + const [page, setPage] = useState(1) + const queryClient = useQueryClient() + const queryKey = ["admin", "ai-search-items", sourceId] + const path = `/api/admin/ai-search/sources/${encodeURIComponent(sourceId)}/items` + + const documents = useQuery({ + queryKey: [...queryKey, page], + queryFn: () => requestJson(`${path}?page=${page}`), + }) + + const upload = useMutation({ + mutationFn: async (file: File) => { + if (file.size === 0 || file.size > 10 * 1024 * 1024) { + throw new Error("Choose a file between 1 byte and 10 MiB.") + } + + const contentBase64 = await fileContent(file) + + await requestJson(path, { + method: "POST", + headers: { "Content-Type": "application/json" }, + body: JSON.stringify({ + name: file.name, + contentType: file.type || "application/octet-stream", + contentBase64, + }), + }) + }, + onSuccess: () => queryClient.invalidateQueries({ queryKey }), + }) + + const remove = useMutation({ + mutationFn: (id: string) => + requestJson(`${path}/${encodeURIComponent(id)}`, { method: "DELETE" }), + onSuccess: () => queryClient.invalidateQueries({ queryKey }), + }) + + const busy = upload.isPending || remove.isPending + const error = upload.error || remove.error || documents.error + const data = documents.data + + return ( +
+
+

Documents: {sourceId}

+ +
+

+ Uploads are indexed asynchronously. Refresh to check processing status. A matching filename + replaces that document. PDF and text files support up to 10 MiB; other formats support up to + 4 MiB. +

+ + + {error &&

{error.message}

} +
    + {data?.items.map((item) => ( +
  • + {item.key} + {item.status} + +
  • + ))} +
+
+ + Page {page} + +
+
+ ) +} diff --git a/apps/web/src/components/workflows/ai-search-upload-fields.tsx b/apps/web/src/components/workflows/ai-search-upload-fields.tsx new file mode 100644 index 00000000..3df4d80f --- /dev/null +++ b/apps/web/src/components/workflows/ai-search-upload-fields.tsx @@ -0,0 +1,36 @@ +import type { WorkflowNodeOptions } from "@solzero/shared" +import { TemplateInput } from "./node-fields" +import type { WorkflowTemplateReferenceOption } from "./types" + +export function AiSearchUploadFields({ + options, + templates, + updateOption, +}: { + options: WorkflowNodeOptions + templates: WorkflowTemplateReferenceOption[] + updateOption: (key: string, value: string) => void +}) { + return ( + <> + updateOption("aiSearchSourceId", value)} + templateOptions={templates} + helpText="Built-in source created by an administrator. A connected sourceId input takes precedence." + /> + updateOption("name", value)} + templateOptions={templates} + helpText="Filename, up to 128 characters. A connected name input takes precedence. Reusing a name replaces its document." + /> +

+ Connect text or JSON to content. Upload requires an administrator workflow owner and queues + indexing asynchronously. +

+ + ) +} diff --git a/apps/web/src/components/workflows/node-inspector.tsx b/apps/web/src/components/workflows/node-inspector.tsx index 9a51750c..7223f227 100644 --- a/apps/web/src/components/workflows/node-inspector.tsx +++ b/apps/web/src/components/workflows/node-inspector.tsx @@ -1,3 +1,4 @@ +import { AiSearchUploadFields } from "./ai-search-upload-fields" import { InputGroup } from "@cloudflare/kumo/components/input-group" import { ChevronRight, Copy, Trash2 } from "lucide-react" import { type ReactNode, useCallback, useEffect, useMemo, useRef, useState } from "react" @@ -874,6 +875,14 @@ export function NodeInspector({ ) : null} + {node.type === "ai-search-upload" ? ( + + ) : null} + {node.type === "r2-put-object" || node.type === "r2-get-object" ? ( <> AI Search; choose an ID returned by `/sessions/ai-search/sources` | +| AI Search | AI Search namespace binding with built-in storage | Seed/index a document, enable the source in Admin > AI Search; choose an ID returned by `/sessions/ai-search/sources` | | OIDC | `auth.providers.` kind=oidc, issuer/clientId and a chosen clientSecret env reference; signIn/provisionUsers/link capabilities | Isolated IdP clients with localhost callback URLs; distinct admin/member users | | LiteLLM/BYOK | `aiProviders.litellm.enabled`, baseUrl, `S0_CONFIG_SECRETS_AI_PROVIDERS_LITELLM_API_KEY`, or a personal provider in settings | Compatible reachable model endpoint; per-user provider credential; never reuse the e2e AI Gateway token as an application provider | @@ -447,3 +447,48 @@ These rules moved here from `AGENTS.md`. They apply to every agent that runs or - Wrap fresh fixture data in e2e `unique(value)` action params so recordings substitute the current value. Keep choices that change the action flow as ordinary params. Prove warm action replay with `self-finalized` steps and zero model calls; API response-cache hits do not prove replay. - The app target stops its own processes and removes its run directory and labelled Docker resources when e2e stops it. Do not add crash-recovery journals, ownership ledgers or retry commands: runs are local only. The launcher boots `alchemy dev` in the run directory and does not create a Cloudflare AI Gateway or account API token, so it does not read `CLOUDFLARE_API_TOKEN`. `CLOUDFLARE_ACCOUNT_ID` stays because the decision model calls the account's default AI Gateway. The application does not receive the e2e token or gateway id. Never sweep shared simulator data or unrelated resources. - Keep AI usage accounting content-free. Report only the suite's own decision-model usage from the e2e report. Do not record application inference in the app, the Worker, or the test report, and do not read the application's gateway logs. Preserve unknown token measurements. Upload sanitized usage summaries only; never raw logs or ledgers. Use the official @e2e-dev/github reporter for CI checks/comments and keep its GitHub token out of application children. Disable SDK telemetry with E2E_TELEMETRY_DISABLED=1. + +## AI Search built-in storage verification + +`nub run test:config` checks the new-source schema, legacy-record/export compatibility, and +additive workflow manifest support without Cloudflare resources. These are contract checks; they +do not exercise the Items API or prove indexing. + +The isolated live suite is `tests/ai-search.external.e2e.ts`. Launch it with: + +```sh +E2E_ENV_FILE=/path/to/isolated-test-account.env nub exec tsx scripts/e2e/ai-search.ts +# Refresh and then verify the changed browser actions against fresh owned resources. +E2E_ENV_FILE=/path/to/isolated-test-account.env nub exec tsx scripts/e2e/ai-search.ts --record --grep administrators +E2E_ENV_FILE=/path/to/isolated-test-account.env nub exec tsx scripts/e2e/ai-search.ts --strict-cache --grep administrators +``` + +The private file must supply `CLOUDFLARE_ACCOUNT_ID`, `CLOUDFLARE_API_TOKEN`, and +`CF_AI_GATEWAY_E2E_TOKEN` for the existing default semantic gateway. No token is minted. +The deployment credential needs AI Search namespace/instance access and permission to establish +Alchemy's native remote binding. The launcher generates a unique ignored `e2e-*` profile with +`deployment.aiSearchEnabledInDev: true`. Ordinary core runs keep this option off and do not +receive the deployment token. Application model responses remain on the configured mock layer; +the native AI Search binding reaches Cloudflare's real service. + +The suite creates a source through the Admin UI, uploads known content, waits for indexing, +retrieves its marker through the application MCP route, deletes the item and checks that the +marker disappears. It runs an administrator-owned workflow using `ai-search-upload`, verifies +its indexing and retrieval, and rejects unauthenticated and malformed uploads. An ordinary-member +identity remains necessary for a separate authenticated non-admin role check. + +The launcher verifies that both generated namespace names are absent before startup, removes only +those owned namespaces and any instances left inside them, and verifies absence after cleanup. +It writes a names-only ownership receipt under `.e2e/ai-search-live//ownership.json`. +The app target separately stops its processes and removes local run state. Docker must be running for the native remote-binding proxy. Docker-backed agent +harnesses are not started by this focused suite. A missing binding, unsuccessful cleanup, startup +failure, or unavailable integration fails the run. Do not reuse standing namespaces or count +collection alone as proof. + +On 2026-11-09 the isolated live API test passed source creation, authenticated Items upload, +index completion, MCP retrieval, deletion with retrieval removal, workflow ingestion and retrieval, +and rejection of unauthenticated and malformed uploads. The separate browser test passed source +creation, persisted built-in configuration, and the Documents panel/upload control. Each run +verified deletion of its namespace pair. Earlier interrupted/failed attempts are retained in local +ownership receipts; they are not passing evidence. This does not prove an authenticated +ordinary-member restriction, external callbacks, pre/prod deployment, or historical-source transfer. diff --git a/e2e.config-check.config.ts b/e2e.config-check.config.ts index 7c9b3034..caef1b93 100644 --- a/e2e.config-check.config.ts +++ b/e2e.config-check.config.ts @@ -7,6 +7,7 @@ export default { "tests/migration.e2e.ts", "tests/storage-transfer.e2e.ts", "tests/api-routing.e2e.ts", + "tests/ai-search-contracts.e2e.ts", ], targets: [{ name: "deployment-cli", platform: "tools" }], workers: 1, diff --git a/packages/api/src/http/groups/admin.ts b/packages/api/src/http/groups/admin.ts index 8e63d97e..415b09d9 100644 --- a/packages/api/src/http/groups/admin.ts +++ b/packages/api/src/http/groups/admin.ts @@ -6,6 +6,10 @@ import { AdminAccessResponse, AdminAiSearchExportResponse, AdminAiSearchResponse, + AdminAiSearchItemParams, + AdminAiSearchItemsQuery, + AdminAiSearchItemsResponse, + AdminAiSearchUploadPayload, AdminAiSearchSourcePayload, AdminAiProvidersResponse, AdminCloudflareAiGatewayProviderKeysPayload, @@ -144,6 +148,23 @@ export class AdminGroup extends HttpApiGroup.make("admin") success: AdminAiSearchExportResponse, error: CommonErrors, }).annotateMerge(OpenApi.annotations({ summary: "Export AI Search runtime registry" })), + HttpApiEndpoint.get("aiSearchItems", "/ai-search/sources/:id/items", { + params: AdminIdParams, + query: AdminAiSearchItemsQuery, + success: AdminAiSearchItemsResponse, + error: CommonErrors, + }), + HttpApiEndpoint.post("uploadAiSearchItem", "/ai-search/sources/:id/items", { + params: AdminIdParams, + payload: AdminAiSearchUploadPayload, + success: AdminAiSearchItemsResponse, + error: CommonErrors, + }), + HttpApiEndpoint.delete("deleteAiSearchItem", "/ai-search/sources/:id/items/:itemId", { + params: AdminAiSearchItemParams, + success: AdminAiSearchItemsResponse, + error: CommonErrors, + }), HttpApiEndpoint.post("createAiSearchSource", "/ai-search/sources", { payload: AdminAiSearchSourcePayload, success: AdminAiSearchResponse, diff --git a/packages/api/src/http/schemas/admin.ts b/packages/api/src/http/schemas/admin.ts index 75e9ad78..3f324929 100644 --- a/packages/api/src/http/schemas/admin.ts +++ b/packages/api/src/http/schemas/admin.ts @@ -359,6 +359,7 @@ export class AdminAiSearchSource extends Schema.Class("Admi source: Schema.String, locked: Schema.Boolean, envVarName: Schema.NullOr(Schema.String), + migrationRequired: Schema.Boolean, createdAt: Schema.Number, updatedAt: Schema.Number, }) {} @@ -406,7 +407,7 @@ export const AdminAiSearchSourcePayload = Schema.Struct({ description: Schema.String, enabled: Schema.Boolean, maxResults: Schema.optionalKey(Schema.NullOr(Schema.Number)), - dataSource: AdminAiSearchDataSource, + dataSource: Schema.Struct({ type: Schema.Literal("built-in") }), }) export type AdminAiSearchSourcePayload = typeof AdminAiSearchSourcePayload.Type @@ -580,3 +581,25 @@ export class AdminActionResponse extends Schema.Class("Admi targetId: Schema.String, runId: Schema.optionalKey(Schema.String), }) {} + +export const AdminAiSearchItemsQuery = { page: Schema.optionalKey(Schema.String) } + +export const AdminAiSearchItemParams = { id: Schema.String, itemId: Schema.String } + +export const AdminAiSearchUploadPayload = Schema.Struct({ + name: Schema.String, + contentBase64: Schema.String, + contentType: Schema.String, +}) + +export type AdminAiSearchUploadPayload = typeof AdminAiSearchUploadPayload.Type + +export class AdminAiSearchItemsResponse extends Schema.Class( + "AdminAiSearchItemsResponse", +)({ + items: Schema.Array( + Schema.Struct({ id: Schema.String, key: Schema.String, status: Schema.String }), + ), + page: Schema.Number, + total: Schema.Number, +}) {} diff --git a/packages/api/src/server/background/db/ai-search-records.ts b/packages/api/src/server/background/db/ai-search-records.ts new file mode 100644 index 00000000..8247488c --- /dev/null +++ b/packages/api/src/server/background/db/ai-search-records.ts @@ -0,0 +1,246 @@ +import { normalizeAiSearchSourceId } from "@solzero/shared" +import * as Match from "effect/Match" +import * as Option from "effect/Option" +import * as Schema from "effect/Schema" +import { + booleanField, + isRecord, + nonBlankStringField, + numberField, + readStringArray, + stringField, +} from "./record-fields" + +export const DEFAULT_AI_SEARCH_MAX_RESULTS = 5 + +export type AiSearchSourceRecordSource = "kv" + +export type AiSearchDataSourceType = "built-in" | "r2" | "website" + +export interface AiSearchBuiltInDataSourceRecord { + type: "built-in" +} + +export interface AiSearchR2DataSourceRecord { + type: "r2" + bucketName: string + prefix: string | null + r2Jurisdiction: string | null +} + +export interface AiSearchWebsiteDataSourceRecord { + type: "website" + domain: string + includePaths: readonly string[] + excludePaths: readonly string[] + specificSitemaps: readonly string[] + useBrowserRendering: boolean + includeImages: boolean +} + +export type AiSearchDataSourceRecord = + | AiSearchBuiltInDataSourceRecord + | AiSearchR2DataSourceRecord + | AiSearchWebsiteDataSourceRecord + +export interface AiSearchSourceRecord { + id: string + label: string + description: string + enabled: boolean + maxResults: number + dataSource: AiSearchDataSourceRecord + createdAt: number + updatedAt: number +} + +export interface AiSearchSourceWithPresence { + source: AiSearchSourceRecord + recordSource: AiSearchSourceRecordSource + locked: boolean + envVarName: string | null +} + +export interface AiSearchSourceUpdate { + id: string + label: string + description: string + enabled: boolean + maxResults?: number | null + dataSource: AiSearchDataSourceRecord +} + +export interface AiSearchInstanceListItem { + id: string + type: string | null + source: string | null + status: string | null + enabled: boolean | null + namespace: string | null + createdAt: string | null + modifiedAt: string | null +} + +export class AiSearchConfigurationError extends Schema.TaggedError()( + "AiSearchConfigurationError", + { + message: Schema.String, + }, +) {} + +export class AiSearchSourceUnavailableError extends Schema.TaggedError()( + "AiSearchSourceUnavailableError", + { + sourceIds: Schema.Array(Schema.String), + }, +) {} + +export function trimmedStringField(record: Record, key: string) { + return Option.getOrNull(Option.map(nonBlankStringField(record, key), (value) => value.trim())) +} + +function normalizeStringList(values: readonly string[] | null | undefined): string[] { + return [...new Set((values ?? []).map((value) => value.trim()).filter(Boolean))] +} + +function isValidMaxResults(value: number) { + return Number.isInteger(value) && value >= 1 && value <= 20 +} + +function requireConfiguration(value: A, valid: (value: A) => boolean, message: string): A { + return Option.liftPredicate(value, valid).pipe( + Option.getOrThrowWith(() => new AiSearchConfigurationError({ message })), + ) +} + +function normalizeMaxResults(value: number | null | undefined): number { + return Option.fromNullishOr(value).pipe( + Option.map((resolved) => + requireConfiguration( + resolved, + isValidMaxResults, + "AI Search max results must be an integer between 1 and 20", + ), + ), + Option.getOrElse(() => DEFAULT_AI_SEARCH_MAX_RESULTS), + ) +} + +function normalizeDomain(value: string): string { + const trimmed = value + .trim() + .replace(/^https?:\/\//, "") + .replace(/\/+$/, "") + + return requireConfiguration(trimmed, (domain) => domain.length > 0, "Website domain is required") +} + +function normalizeDataSource(dataSource: AiSearchDataSourceRecord): AiSearchDataSourceRecord { + return Match.value(dataSource).pipe( + Match.discriminatorsExhaustive("type")({ + "built-in": (): AiSearchDataSourceRecord => ({ type: "built-in" }), + r2: (r2): AiSearchDataSourceRecord => ({ + type: "r2", + bucketName: r2.bucketName.trim(), + prefix: r2.prefix?.trim() || null, + r2Jurisdiction: r2.r2Jurisdiction?.trim() || null, + }), + website: (website): AiSearchDataSourceRecord => ({ + type: "website", + domain: normalizeDomain(website.domain), + includePaths: normalizeStringList(website.includePaths), + excludePaths: normalizeStringList(website.excludePaths), + specificSitemaps: normalizeStringList(website.specificSitemaps), + useBrowserRendering: Boolean(website.useBrowserRendering), + includeImages: Boolean(website.includeImages), + }), + }), + ) +} + +function dataSourceFromRecord(value: Record) { + return Match.value(stringField(value, "type")).pipe( + Match.when("built-in", () => Option.some({ type: "built-in" })), + Match.when("r2", () => + Option.some({ + type: "r2", + bucketName: stringField(value, "bucketName"), + prefix: trimmedStringField(value, "prefix"), + r2Jurisdiction: trimmedStringField(value, "r2Jurisdiction"), + }), + ), + Match.when("website", () => + Option.some({ + type: "website", + domain: stringField(value, "domain"), + includePaths: readStringArray(value.includePaths), + excludePaths: readStringArray(value.excludePaths), + specificSitemaps: readStringArray(value.specificSitemaps), + useBrowserRendering: booleanField(value, "useBrowserRendering", false), + includeImages: booleanField(value, "includeImages", false), + }), + ), + Match.orElse(() => Option.none()), + ) +} + +function aiSearchSourceFromRecord( + record: Record, + dataSource: AiSearchDataSourceRecord, +): AiSearchSourceRecord { + const id = normalizeAiSearchSourceId(stringField(record, "id")) + const now = Date.now() + + return { + id, + label: stringField(record, "label", id), + description: stringField(record, "description"), + enabled: booleanField(record, "enabled", true), + maxResults: normalizeMaxResults( + numberField(record, "maxResults", DEFAULT_AI_SEARCH_MAX_RESULTS), + ), + dataSource: normalizeDataSource(dataSource), + createdAt: numberField(record, "createdAt", now), + updatedAt: numberField(record, "updatedAt", now), + } +} + +export function readAiSearchSourceRecord(value: unknown): Option.Option { + return Option.liftPredicate(value, isRecord).pipe( + Option.flatMap((record) => + Option.liftPredicate(record.dataSource, isRecord).pipe( + Option.flatMap(dataSourceFromRecord), + Option.map((dataSource) => aiSearchSourceFromRecord(record, dataSource)), + ), + ), + ) +} + +export function buildAiSearchSourceRecord( + update: AiSearchSourceUpdate, + existing: Option.Option, +): AiSearchSourceRecord { + const id = normalizeAiSearchSourceId(update.id) + + const dataSource = requireConfiguration( + normalizeDataSource(update.dataSource), + (source) => source.type === "built-in", + "Only built-in storage is supported. Export and migrate legacy content explicitly.", + ) + + const now = Date.now() + + return { + id, + label: update.label.trim() || id, + description: update.description.trim(), + enabled: update.enabled, + maxResults: normalizeMaxResults(update.maxResults), + dataSource, + createdAt: Option.getOrElse( + Option.map(existing, (source) => source.createdAt), + () => now, + ), + updatedAt: now, + } +} diff --git a/packages/api/src/server/background/db/ai-search.ts b/packages/api/src/server/background/db/ai-search.ts index b94c5601..976da65f 100644 --- a/packages/api/src/server/background/db/ai-search.ts +++ b/packages/api/src/server/background/db/ai-search.ts @@ -10,257 +10,24 @@ import * as Match from "effect/Match" import * as Option from "effect/Option" import * as P from "effect/Predicate" import * as Result from "effect/Result" -import * as Schema from "effect/Schema" import { describeError, toError } from "../../lib/effect-errors" import type { Env } from "../types" -import { - booleanField, - isRecord, - nonBlankStringField, - numberField, - readArray, - readStringArray, - stringField, -} from "./record-fields" +import { isRecord, readArray, stringField } from "./record-fields" import { S0_CONFIG_KEYS, S0ConfigStore } from "./s0-config" -export const DEFAULT_AI_SEARCH_MAX_RESULTS = 5 - -export type AiSearchSourceRecordSource = "kv" - -export type AiSearchDataSourceType = "built-in" | "r2" | "website" - -export interface AiSearchBuiltInDataSourceRecord { - type: "built-in" -} - -export interface AiSearchR2DataSourceRecord { - type: "r2" - bucketName: string - prefix: string | null - r2Jurisdiction: string | null -} - -export interface AiSearchWebsiteDataSourceRecord { - type: "website" - domain: string - includePaths: readonly string[] - excludePaths: readonly string[] - specificSitemaps: readonly string[] - useBrowserRendering: boolean - includeImages: boolean -} - -export type AiSearchDataSourceRecord = - | AiSearchBuiltInDataSourceRecord - | AiSearchR2DataSourceRecord - | AiSearchWebsiteDataSourceRecord - -export interface AiSearchSourceRecord { - id: string - label: string - description: string - enabled: boolean - maxResults: number - dataSource: AiSearchDataSourceRecord - createdAt: number - updatedAt: number -} - -export interface AiSearchSourceWithPresence { - source: AiSearchSourceRecord - recordSource: AiSearchSourceRecordSource - locked: boolean - envVarName: string | null -} - -export interface AiSearchSourceUpdate { - id: string - label: string - description: string - enabled: boolean - maxResults?: number | null - dataSource: AiSearchDataSourceRecord -} - -export interface AiSearchInstanceListItem { - id: string - type: string | null - source: string | null - status: string | null - enabled: boolean | null - namespace: string | null - createdAt: string | null - modifiedAt: string | null -} - -export class AiSearchConfigurationError extends Schema.TaggedError()( - "AiSearchConfigurationError", - { - message: Schema.String, - }, -) {} - -export class AiSearchSourceUnavailableError extends Schema.TaggedError()( - "AiSearchSourceUnavailableError", - { - sourceIds: Schema.Array(Schema.String), - }, -) {} - -function trimmedStringField(record: Record, key: string) { - return Option.getOrNull(Option.map(nonBlankStringField(record, key), (value) => value.trim())) -} - -function normalizeStringList(values: readonly string[] | null | undefined): string[] { - return [...new Set((values ?? []).map((value) => value.trim()).filter(Boolean))] -} - -function isValidMaxResults(value: number) { - return Number.isInteger(value) && value >= 1 && value <= 20 -} - -function requireConfiguration(value: A, valid: (value: A) => boolean, message: string): A { - return Option.liftPredicate(value, valid).pipe( - Option.getOrThrowWith(() => new AiSearchConfigurationError({ message })), - ) -} - -function normalizeMaxResults(value: number | null | undefined): number { - return Option.fromNullishOr(value).pipe( - Option.map((resolved) => - requireConfiguration( - resolved, - isValidMaxResults, - "AI Search max results must be an integer between 1 and 20", - ), - ), - Option.getOrElse(() => DEFAULT_AI_SEARCH_MAX_RESULTS), - ) -} - -function normalizeDomain(value: string): string { - const trimmed = value - .trim() - .replace(/^https?:\/\//, "") - .replace(/\/+$/, "") - - return requireConfiguration(trimmed, (domain) => domain.length > 0, "Website domain is required") -} - -function normalizeDataSource(dataSource: AiSearchDataSourceRecord): AiSearchDataSourceRecord { - return Match.value(dataSource).pipe( - Match.discriminatorsExhaustive("type")({ - "built-in": (): AiSearchDataSourceRecord => ({ type: "built-in" }), - r2: (r2): AiSearchDataSourceRecord => ({ - type: "r2", - bucketName: r2.bucketName.trim(), - prefix: r2.prefix?.trim() || null, - r2Jurisdiction: r2.r2Jurisdiction?.trim() || null, - }), - website: (website): AiSearchDataSourceRecord => ({ - type: "website", - domain: normalizeDomain(website.domain), - includePaths: normalizeStringList(website.includePaths), - excludePaths: normalizeStringList(website.excludePaths), - specificSitemaps: normalizeStringList(website.specificSitemaps), - useBrowserRendering: Boolean(website.useBrowserRendering), - includeImages: Boolean(website.includeImages), - }), - }), - ) -} - -function hasRequiredDataSourceFields(dataSource: AiSearchDataSourceRecord) { - return dataSource.type !== "r2" || dataSource.bucketName.trim().length > 0 -} - -function dataSourceFromRecord(value: Record) { - return Match.value(stringField(value, "type")).pipe( - Match.when("built-in", () => Option.some({ type: "built-in" })), - Match.when("r2", () => - Option.some({ - type: "r2", - bucketName: stringField(value, "bucketName"), - prefix: trimmedStringField(value, "prefix"), - r2Jurisdiction: trimmedStringField(value, "r2Jurisdiction"), - }), - ), - Match.when("website", () => - Option.some({ - type: "website", - domain: stringField(value, "domain"), - includePaths: readStringArray(value.includePaths), - excludePaths: readStringArray(value.excludePaths), - specificSitemaps: readStringArray(value.specificSitemaps), - useBrowserRendering: booleanField(value, "useBrowserRendering", false), - includeImages: booleanField(value, "includeImages", false), - }), - ), - Match.orElse(() => Option.none()), - ) -} - -function aiSearchSourceFromRecord( - record: Record, - dataSource: AiSearchDataSourceRecord, -): AiSearchSourceRecord { - const id = normalizeAiSearchSourceId(stringField(record, "id")) - const now = Date.now() - - return { - id, - label: stringField(record, "label", id), - description: stringField(record, "description"), - enabled: booleanField(record, "enabled", true), - maxResults: normalizeMaxResults( - numberField(record, "maxResults", DEFAULT_AI_SEARCH_MAX_RESULTS), - ), - dataSource: normalizeDataSource(dataSource), - createdAt: numberField(record, "createdAt", now), - updatedAt: numberField(record, "updatedAt", now), - } -} - -export function readAiSearchSourceRecord(value: unknown): Option.Option { - return Option.liftPredicate(value, isRecord).pipe( - Option.flatMap((record) => - Option.liftPredicate(record.dataSource, isRecord).pipe( - Option.flatMap(dataSourceFromRecord), - Option.map((dataSource) => aiSearchSourceFromRecord(record, dataSource)), - ), - ), - ) -} - -function buildAiSearchSourceRecord( - update: AiSearchSourceUpdate, - existing: Option.Option, -): AiSearchSourceRecord { - const id = normalizeAiSearchSourceId(update.id) - - const dataSource = requireConfiguration( - normalizeDataSource(update.dataSource), - hasRequiredDataSourceFields, - "R2 bucket name is required", - ) - - const now = Date.now() +export * from "./ai-search-records" - return { - id, - label: update.label.trim() || id, - description: update.description.trim(), - enabled: update.enabled, - maxResults: normalizeMaxResults(update.maxResults), - dataSource, - createdAt: Option.getOrElse( - Option.map(existing, (source) => source.createdAt), - () => now, - ), - updatedAt: now, - } -} +import { + AiSearchConfigurationError, + AiSearchSourceUnavailableError, + buildAiSearchSourceRecord, + readAiSearchSourceRecord, + trimmedStringField, + type AiSearchSourceRecord, + type AiSearchSourceUpdate, + type AiSearchSourceWithPresence, + type AiSearchInstanceListItem, +} from "./ai-search-records" function searchToolName(source: AiSearchSourceRecord) { return getAiSearchMcpToolNames(source).search @@ -297,144 +64,20 @@ type AiSearchInstance = ReturnType type AiSearchUpdateInput = Parameters[0] -interface R2SourceParams { - prefix?: string - r2_jurisdiction?: string -} - -interface WebsiteSourceParams { - include_items?: string[] - exclude_items?: string[] - web_crawler: { - parse_type: "sitemap" - parse_options: { - include_images: boolean - use_browser_rendering: boolean - specific_sitemaps?: string[] - } - } -} +function isDisposableInstance( + instance: AiSearchInstance, +): instance is AiSearchInstance & Disposable { + const handle: AiSearchInstance & Partial = instance -function nonEmptyText(value: string | null) { - return Option.fromNullishOr(value).pipe(Option.filter((text) => text.length > 0)) + return P.isFunction(handle[Symbol.dispose]) } -function nonEmptyList(values: readonly string[]) { - return Option.liftPredicate(values, Arr.isReadonlyArrayNonEmpty) -} - -function r2SourceParams(dataSource: AiSearchR2DataSourceRecord): R2SourceParams { - return { - ...Option.match(nonEmptyText(dataSource.prefix), { - onNone: () => ({}), - onSome: (prefix) => ({ prefix }), - }), - ...Option.match(nonEmptyText(dataSource.r2Jurisdiction), { - onNone: () => ({}), - onSome: (jurisdiction) => ({ r2_jurisdiction: jurisdiction }), - }), - } -} - -function websiteSourceParams(dataSource: AiSearchWebsiteDataSourceRecord): WebsiteSourceParams { - return { - web_crawler: { - parse_type: "sitemap", - parse_options: { - include_images: dataSource.includeImages, - use_browser_rendering: dataSource.useBrowserRendering, - ...Option.match(nonEmptyList(dataSource.specificSitemaps), { - onNone: () => ({}), - onSome: (sitemaps) => ({ specific_sitemaps: [...sitemaps] }), - }), - }, - }, - ...Option.match(nonEmptyList(dataSource.includePaths), { - onNone: () => ({}), - onSome: (paths) => ({ include_items: [...paths] }), - }), - ...Option.match(nonEmptyList(dataSource.excludePaths), { - onNone: () => ({}), - onSome: (paths) => ({ exclude_items: [...paths] }), - }), - } -} - -function sourceParamsFor(dataSource: AiSearchDataSourceRecord) { - return Match.value(dataSource).pipe( - Match.discriminatorsExhaustive("type")({ - "built-in": () => undefined, - r2: r2SourceParams, - website: websiteSourceParams, - }), - ) -} - -function createInputFor( - source: AiSearchSourceRecord, - tokenId?: string | null, -): AiSearchCreateInput { - const base = { - id: source.id, - enable: source.enabled, - max_num_results: source.maxResults, - ...Option.match(nonEmptyText(tokenId?.trim() ?? null), { - onNone: () => ({}), - onSome: (token) => ({ token_id: token }), - }), - } - - // SAFETY: each branch builds the AiSearchCreateInput variant for its data source type. - return Match.value(source.dataSource).pipe( - Match.discriminatorsExhaustive("type")({ - "built-in": () => base as AiSearchCreateInput, - r2: (r2) => - ({ - ...base, - type: "r2", - source: r2.bucketName, - source_params: sourceParamsFor(r2), - }) as AiSearchCreateInput, - website: (website) => - ({ - ...base, - type: "web-crawler", - source: website.domain, - source_params: sourceParamsFor(website), - }) as AiSearchCreateInput, - }), - ) +function createInputFor(source: AiSearchSourceRecord): AiSearchCreateInput { + return { id: source.id, enable: source.enabled, max_num_results: source.maxResults } } function updateInputFor(source: AiSearchSourceRecord): AiSearchUpdateInput { - const update = { - enable: source.enabled, - max_num_results: source.maxResults, - } - - const withParams = { - ...sourceFields(update, source), - ...Option.match(Option.fromUndefinedOr(sourceParamsFor(source.dataSource)), { - onNone: () => ({}), - onSome: (sourceParams) => ({ source_params: sourceParams }), - }), - } - - // SAFETY: Cloudflare AI Search update accepts this enable, source, and source_params bag. Absent fields stay omitted. - return withParams as AiSearchUpdateInput -} - -function sourceFields( - update: { enable: boolean; max_num_results: number }, - source: AiSearchSourceRecord, -) { - return Match.value(source.dataSource).pipe( - Match.discriminatorsExhaustive("type")({ - "built-in": () => update, - r2: (r2) => ({ ...update, source: r2.bucketName }), - website: (website) => ({ ...update, source: website.domain }), - }), - ) + return { enable: source.enabled, max_num_results: source.maxResults } } function normalizeInstanceListItem(item: unknown): Option.Option { @@ -566,7 +209,7 @@ export class AiSearchRegistryStore { ) { const sources = yield* this.listSources() - return sources.filter((source) => source.enabled) + return sources.filter((source) => source.enabled && source.dataSource.type === "built-in") }) listAvailableSourcesByIds = Effect.fn("db.aiSearch.listAvailableSourcesByIds")(function* ( @@ -639,7 +282,7 @@ export class AiSearchRegistryStore { yield* ensureAiSearch(update.id === normalizedSourceId, "AI Search source id cannot be changed") yield* ensureAiSearch( update.dataSource.type === existing.source.dataSource.type, - "AI Search data source type cannot be changed; delete and recreate the source", + "Legacy sources require explicit content migration to a new built-in source; existing content is retained", ) const record = buildAiSearchSourceRecord(update, Option.some(existing.source)) @@ -678,18 +321,99 @@ export class AiSearchRegistryStore { return Arr.getSomes(result.map(normalizeInstanceListItem)) }) + builtInInstance = Effect.fn("db.aiSearch.builtInInstance")(function* ( + this: AiSearchRegistryStore, + sourceId: string, + ) { + const presence = yield* this.getSourceWithPresence(sourceId) + const existing = yield* requireExistingSource(presence, sourceId) + + yield* ensureAiSearch( + existing.source.dataSource.type === "built-in", + "This legacy source requires an explicit content migration to built-in storage before use", + ) + + const namespace = yield* aiSearchNamespace(this.env) + + return namespace.get(existing.source.id) + }) + + listItems = Effect.fn("db.aiSearch.listItems")(function* ( + this: AiSearchRegistryStore, + sourceId: string, + page = 1, + ) { + yield* ensureAiSearch(Number.isInteger(page) && page > 0, "Page must be a positive integer") + + const instance = yield* this.builtInInstance(sourceId) + + return yield* Effect.tryPromise({ + try: () => instance.items.list({ page, per_page: 50 }), + catch: toError, + }) + }) + + uploadItem = Effect.fn("db.aiSearch.uploadItem")(function* ( + this: AiSearchRegistryStore, + sourceId: string, + name: string, + content: Blob | string, + ) { + yield* ensureAiSearch( + name.trim().length > 0 && name.length <= 128, + "Item name must contain 1 to 128 characters", + ) + + const size = Match.value(content).pipe( + Match.when(P.isString, (text) => new TextEncoder().encode(text).length), + Match.orElse((blob) => blob.size), + ) + + yield* ensureAiSearch( + size > 0 && size <= 10 * 1024 * 1024, + "Item must contain 1 byte to 10 MiB", + ) + + const instance = yield* this.builtInInstance(sourceId) + + return yield* Effect.tryPromise({ + try: () => instance.items.upload(name, content), + catch: toError, + }) + }) + + deleteItem = Effect.fn("db.aiSearch.deleteItem")(function* ( + this: AiSearchRegistryStore, + sourceId: string, + itemId: string, + ) { + const instance = yield* this.builtInInstance(sourceId) + + yield* Effect.tryPromise({ try: () => instance.items.delete(itemId), catch: toError }) + }) + private createCloudflareInstance = Effect.fn("db.aiSearch.createCloudflareInstance")(function* ( this: AiSearchRegistryStore, source: AiSearchSourceRecord, ) { const aiSearch = yield* aiSearchNamespace(this.env) - yield* Effect.tryPromise({ - try: () => aiSearch.create(createInputFor(source, this.env.CF_AI_SEARCH_SERVICE_TOKEN_ID)), + + const instance = yield* Effect.tryPromise({ + try: () => aiSearch.create(createInputFor(source)), catch: (cause) => new AiSearchConfigurationError({ message: `Failed to create AI Search instance '${source.id}': ${describeError(cause)}`, }), }) + + // Release a remote RPC handle rather than retaining it until garbage collection. + yield* Effect.try({ + try: () => + Option.map(Option.liftPredicate(instance, isDisposableInstance), (handle) => + handle[Symbol.dispose](), + ), + catch: toError, + }) }) private updateCloudflareInstance = Effect.fn("db.aiSearch.updateCloudflareInstance")(function* ( diff --git a/packages/api/src/server/background/workflows/nodes/storage.ts b/packages/api/src/server/background/workflows/nodes/storage.ts index 98b11f1e..08b0420a 100644 --- a/packages/api/src/server/background/workflows/nodes/storage.ts +++ b/packages/api/src/server/background/workflows/nodes/storage.ts @@ -1,3 +1,6 @@ +import { AiSearchRegistryStore } from "../../db/ai-search" +import { isAdminEmailForEnv } from "../../db/admin-config" +import { runControlPlaneSqlFirst } from "../../../effect/db/control-plane-db" import { WORKFLOW_KV_NAMESPACE_OPTIONS, WORKFLOW_R2_BUCKET_OPTIONS, @@ -25,6 +28,7 @@ import * as Effect from "effect/Effect" import * as Match from "effect/Match" import * as Option from "effect/Option" import * as Result from "effect/Result" +import * as P from "effect/Predicate" export type WorkflowStorageNodeExecutionInput = WorkflowNodeExecutionInput @@ -54,6 +58,7 @@ export const executeWorkflowStorageNode = Effect.fn("workflows.executeStorageNod input: WorkflowStorageNodeExecutionInput, ) { return yield* Match.value(input.node.type).pipe( + Match.when("ai-search-upload", () => runAiSearchUploadNode(input)), Match.when("r2-put-object", () => runR2PutObjectNode(input)), Match.when("r2-get-object", () => runR2GetObjectNode(input)), Match.when("kv-put", () => runKvPutNode(input)), @@ -528,3 +533,62 @@ const runGetSecretNode = Effect.fn("workflows.runGetSecretNode")(function* ( }, } }) + +const runAiSearchUploadNode = Effect.fn("workflows.uploadAiSearchItem")(function* ( + input: WorkflowStorageNodeExecutionInput, +) { + const userId = getActionUserId(input) + + const user = yield* Effect.tryPromise({ + try: () => + runControlPlaneSqlFirst<{ email: string }>( + input.env, + `SELECT "email" FROM "user" WHERE "id" = ?1 LIMIT 1`, + [userId], + ), + catch: toError, + }) + + const allowed = yield* Option.match(Option.fromNullishOr(user), { + onNone: () => Effect.succeed(false), + onSome: (owner) => isAdminEmailForEnv(input.env, owner.email), + }) + + yield* workflowNodeFailWhen( + !allowed, + "Only administrators can ingest documents into global AI Search sources", + ) + + const options = input.node.options ?? {} + + const sourceId = yield* requireString( + Option.orElse(getString(input.inputs.sourceId), () => getString(options.aiSearchSourceId)), + "AI Search source ID is required", + ) + + const name = yield* requireString( + Option.orElse(getString(input.inputs.name), () => getString(options.name)), + "AI Search document name is required", + ) + + const content = Match.value(input.inputs.content).pipe( + Match.when(P.isString, (text) => text), + Match.orElse((value) => stringifyJsonPretty(value ?? null)), + ) + + const context = createNodeContext(input) + + const item = yield* new AiSearchRegistryStore(input.env).uploadItem( + renderTemplate(sourceId, context), + renderTemplate(name, context), + content, + ) + + yield* recordWorkflowNodeRunEvent(input, { + eventType: "ai_search_upload", + message: "AI Search upload queued", + data: { sourceId, itemId: item.id }, + }) + + return { outputs: { itemId: item.id, key: item.key, status: item.status } } +}) diff --git a/packages/api/src/server/effect/handlers/admin/ai-search-items.ts b/packages/api/src/server/effect/handlers/admin/ai-search-items.ts new file mode 100644 index 00000000..90c1478f --- /dev/null +++ b/packages/api/src/server/effect/handlers/admin/ai-search-items.ts @@ -0,0 +1,111 @@ +import type { AdminAiSearchUploadPayload, AdminIdParams } from "@solzero/api" +import * as Effect from "effect/Effect" +import { AiSearchRegistryStore } from "../../../background/db/ai-search" +import { toError } from "../../../lib/effect-errors" +import { + describeError, + failMessage, + failWhen, + json, + runControlPlane, +} from "../shared/control-plane" +import { requireAdmin, withAudit } from "./route-helpers" + +const itemsResponse = Effect.fn("admin.aiSearch.itemsResponse")(function* ( + registry: AiSearchRegistryStore, + sourceId: string, + page = 1, +) { + const response = yield* registry + .listItems(sourceId, page) + .pipe(Effect.catch((cause) => failMessage(describeError(cause), 400))) + + return json({ + items: response.result.map((item) => ({ id: item.id, key: item.key, status: item.status })), + page, + total: response.result_info?.total_count ?? response.result.length, + }) +}) + +export function aiSearchItems({ + params, + query, +}: { + params: AdminIdParams + query: { page?: string } +}) { + return runControlPlane( + Effect.fn("admin.aiSearch.items")(function* (context) { + yield* requireAdmin(context) + + return yield* itemsResponse( + new AiSearchRegistryStore(context.env), + params.id, + Number(query.page ?? 1), + ) + }), + ) +} + +export function uploadAiSearchItem({ + params, + payload, +}: { + params: AdminIdParams + payload: AdminAiSearchUploadPayload +}) { + return runControlPlane( + Effect.fn("admin.aiSearch.uploadItem")(function* (context) { + const admin = yield* requireAdmin(context) + + yield* failWhen(payload.contentBase64.length > 14 * 1024 * 1024, "Upload exceeds 10 MiB", 400) + + const bytes = yield* Effect.try({ + try: () => Uint8Array.from(atob(payload.contentBase64), (c) => c.charCodeAt(0)), + catch: toError, + }).pipe(Effect.catch(() => failMessage("Invalid base64 content", 400))) + + const registry = new AiSearchRegistryStore(context.env) + + return yield* withAudit( + { + context, + admin, + targetType: "ai_search_source", + targetId: params.id, + action: "upload_item", + }, + registry + .uploadItem(params.id, payload.name, new Blob([bytes], { type: payload.contentType })) + .pipe( + Effect.catch((cause) => failMessage(describeError(cause), 400)), + Effect.andThen(() => itemsResponse(registry, params.id)), + ), + ) + }), + ) +} + +export function deleteAiSearchItem({ params }: { params: { id: string; itemId: string } }) { + return runControlPlane( + Effect.fn("admin.aiSearch.deleteItem")(function* (context) { + const admin = yield* requireAdmin(context) + + const registry = new AiSearchRegistryStore(context.env) + + return yield* withAudit( + { + context, + admin, + targetType: "ai_search_source", + targetId: params.id, + action: "delete_item", + }, + registry.deleteItem(params.id, params.itemId).pipe( + Effect.catch((cause) => failMessage(describeError(cause), 400)), + Effect.andThen(() => itemsResponse(registry, params.id)), + ), + ) + }), + ) +} diff --git a/packages/api/src/server/effect/handlers/admin/ai-search.ts b/packages/api/src/server/effect/handlers/admin/ai-search.ts index d7dcba9c..0e42c935 100644 --- a/packages/api/src/server/effect/handlers/admin/ai-search.ts +++ b/packages/api/src/server/effect/handlers/admin/ai-search.ts @@ -26,6 +26,7 @@ function formatSource(input: AiSearchSourceWithPresence) { source: input.recordSource, locked: input.locked, envVarName: input.envVarName, + migrationRequired: input.source.dataSource.type !== "built-in", createdAt: input.source.createdAt, updatedAt: input.source.updatedAt, } diff --git a/packages/api/src/server/effect/routes/admin.ts b/packages/api/src/server/effect/routes/admin.ts index 43976639..594c74f0 100644 --- a/packages/api/src/server/effect/routes/admin.ts +++ b/packages/api/src/server/effect/routes/admin.ts @@ -1,3 +1,8 @@ +import { + aiSearchItems, + uploadAiSearchItem, + deleteAiSearchItem, +} from "../handlers/admin/ai-search-items" import { HttpApiBuilder } from "effect/http-api" import { S0Api } from "@solzero/api" import { @@ -92,6 +97,15 @@ export const HttpAdminLive = HttpApiBuilder.group(S0Api, "admin", (handlers) => .handle("exportAiSearchConfig", () => observeRoute("admin", "exportAiSearchConfig", exportAiSearchConfig()), ) + .handle("aiSearchItems", (input) => + observeRoute("admin", "aiSearchItems", aiSearchItems(input)), + ) + .handle("uploadAiSearchItem", (input) => + observeRoute("admin", "uploadAiSearchItem", uploadAiSearchItem(input)), + ) + .handle("deleteAiSearchItem", (input) => + observeRoute("admin", "deleteAiSearchItem", deleteAiSearchItem(input)), + ) .handle("createAiSearchSource", (input) => observeRoute("admin", "createAiSearchSource", createAiSearchSource(input)), ) diff --git a/packages/infra/src/s0.ts b/packages/infra/src/s0.ts index c33e7a4d..8c2c237b 100644 --- a/packages/infra/src/s0.ts +++ b/packages/infra/src/s0.ts @@ -187,7 +187,6 @@ export function createS0Api(options: CreateS0ApiOptions) { dev, stageMetadata, migrationsDir, - tokenId: apiEnv.CF_AI_SEARCH_SERVICE_TOKEN_ID, databaseEngine, appDbMode, databaseUrl: databaseUrl ?? LOCAL_PGLITE_DATABASE_URL, diff --git a/packages/shared/src/s0-config.ts b/packages/shared/src/s0-config.ts index 1e189adb..22ec3b49 100644 --- a/packages/shared/src/s0-config.ts +++ b/packages/shared/src/s0-config.ts @@ -75,6 +75,7 @@ export const S0DeploymentConfigSchema = Schema.Struct({ zone: Schema.String, webFqdn: Schema.optional(Schema.String), useApiShield: Schema.Boolean, + aiSearchEnabledInDev: Schema.optional(Schema.Boolean), planetscale: Schema.optional(S0PlanetscaleConfigSchema), observability: Schema.Struct({ logsDestinations: Schema.Array(Schema.String), @@ -194,9 +195,6 @@ const S0ResolvedConfigFields = { slack: S0SlackConfigSchema, }), security: S0SecurityConfigSchema, - aiSearch: Schema.Struct({ - serviceTokenId: Schema.optional(Schema.NullOr(SecretReferenceSchema)), - }), } as const export const S0ResolvedConfigSchema = Schema.Struct({ @@ -534,12 +532,6 @@ const normalizeResolvedConfig = (decoded: S0ResolvedConfig) => "security.repositorySecretsEncryptionKey", ), }), - aiSearch: { - serviceTokenId: yield* normalizeOptionalSecret( - decoded.aiSearch.serviceTokenId, - "aiSearch.serviceTokenId", - ), - }, } yield* ensureConfig(normalized.deployment.appName !== "", () => @@ -647,7 +639,6 @@ export function s0ActiveSecretReferences(config: S0ResolvedConfig): SecretRefere config.security.mcpcfProxySigningSecret, config.security.tokenEncryptionKey, config.security.repositorySecretsEncryptionKey, - ...enabledSecrets(true, config.aiSearch.serviceTokenId), ] } diff --git a/packages/shared/src/stageMetadata.ts b/packages/shared/src/stageMetadata.ts index acf8e439..be632b77 100644 --- a/packages/shared/src/stageMetadata.ts +++ b/packages/shared/src/stageMetadata.ts @@ -30,6 +30,7 @@ export interface StageInfraConfig { readonly apiObservabilityTracesDestinations: Option.Option readonly apiObservabilityLogsHeadSamplingRate: Option.Option readonly apiObservabilityTracesHeadSamplingRate: Option.Option + readonly aiSearchEnabledInDev: Option.Option readonly useApiShield: Option.Option readonly alchemyStateStore: Option.Option } @@ -41,6 +42,7 @@ const EMPTY_STAGE_INFRA_CONFIG: StageInfraConfig = { apiObservabilityTracesDestinations: Option.none(), apiObservabilityLogsHeadSamplingRate: Option.none(), apiObservabilityTracesHeadSamplingRate: Option.none(), + aiSearchEnabledInDev: Option.none(), useApiShield: Option.none(), alchemyStateStore: Option.none(), } @@ -62,6 +64,7 @@ function stageInfraConfigFromDeployment(config: S0DeploymentConfig): StageInfraC apiObservabilityTracesHeadSamplingRate: Option.some( config.observability.tracesHeadSamplingRate, ), + aiSearchEnabledInDev: Option.fromUndefinedOr(config.aiSearchEnabledInDev), useApiShield: Option.some(config.useApiShield), alchemyStateStore: Option.none(), } @@ -87,6 +90,7 @@ export interface InfraStageProps { readonly authTrustedOrigins: readonly string[] /** Custom domains to bind to the web worker for this stage. */ readonly webDomains: readonly string[] + readonly aiSearchNamespaceEnabled: boolean readonly useApiShield: boolean /** * Origin (scheme + host + optional port) the sandbox container uses to reach @@ -153,6 +157,7 @@ const InfraStagePropsSchema = Schema.Struct({ authBaseUrl: Schema.String, authTrustedOrigins: Schema.Array(Schema.String), webDomains: Schema.Array(Schema.String), + aiSearchNamespaceEnabled: Schema.Boolean, useApiShield: Schema.Boolean, internalMcpOrigin: Schema.String, internalMcpOutboundHost: Schema.NullOr(Schema.String), @@ -236,6 +241,7 @@ function localInfraStageProps( "http://[::1]:3000", ], webDomains: [], + aiSearchNamespaceEnabled: Option.getOrElse(config.aiSearchEnabledInDev, () => false), useApiShield: Option.getOrElse(config.useApiShield, () => false), internalMcpOrigin: INTERNAL_MCP_LOCAL_ORIGIN, internalMcpOutboundHost: null, @@ -306,6 +312,7 @@ function deployedInfraStageProps(input: { authBaseUrl, authTrustedOrigins: [authBaseUrl], webDomains: [webFqdn], + aiSearchNamespaceEnabled: true, useApiShield: Option.getOrElse(config.useApiShield, () => false), internalMcpOrigin: INTERNAL_MCP_INTERNAL_ORIGIN, internalMcpOutboundHost: INTERNAL_MCP_INTERNAL_HOST, diff --git a/packages/shared/src/workflow-nodes/definitions.ts b/packages/shared/src/workflow-nodes/definitions.ts index ec59491e..bd92ca9e 100644 --- a/packages/shared/src/workflow-nodes/definitions.ts +++ b/packages/shared/src/workflow-nodes/definitions.ts @@ -22,6 +22,7 @@ export const WORKFLOW_NODE_TYPES = [ "slack-remove-reaction", "email-notification", "r2-put-object", + "ai-search-upload", "r2-get-object", "kv-put", "kv-get", @@ -119,6 +120,7 @@ export type WorkflowNodeEditorConfiguration = | "notification" | "slack-action" | "session" + | "ai-search-upload" | "r2-object" | "kv-entry" @@ -154,8 +156,9 @@ export const WORKFLOW_R2_BUCKET_OPTIONS = [ }, { binding: "AI_SEARCH_CONTENT_BUCKET", - label: "AI Search content", - description: "Stores documents that can be indexed by configured AI Search sources.", + label: "Legacy AI Search content", + description: + "Retained for existing workflows and explicit content migration. Use Upload to AI Search for new ingestion.", }, ] as const @@ -634,6 +637,27 @@ export const WORKFLOW_NODE_CATALOG = [ runtime: { kind: "adapter", adapterCategory: "notification" }, editor: { icon: "mail", configuration: "notification" }, }, + { + type: "ai-search-upload", + label: "Upload to AI Search", + description: + "Upload a document to a built-in AI Search source. Requires an administrator workflow owner.", + category: "storage", + inputs: [ + { id: "sourceId", type: "string" }, + { id: "name", type: "string" }, + { id: "content", type: "any" }, + ], + outputs: [ + { id: "itemId", type: "string" }, + { id: "key", type: "string" }, + { id: "status", type: "string" }, + ], + defaults: { strategy: "static", options: {} }, + validation: { optionRules: [], templateReferences: "connected-inputs" }, + runtime: { kind: "adapter", adapterCategory: "storage" }, + editor: { icon: "database", configuration: "ai-search-upload" }, + }, { type: "r2-put-object", label: "Save to R2", diff --git a/packages/shared/src/workflows/manifest-schema.ts b/packages/shared/src/workflows/manifest-schema.ts index 2b19db89..234a7ef6 100644 --- a/packages/shared/src/workflows/manifest-schema.ts +++ b/packages/shared/src/workflows/manifest-schema.ts @@ -96,6 +96,7 @@ export const WorkflowNodeOptionsSchema = Schema.Struct({ ), timestamp: Schema.optional(Schema.String), name: Schema.optional(Schema.String), + aiSearchSourceId: Schema.optional(Schema.String), to: Schema.optional(Schema.String), from: Schema.optional(Schema.String), subject: Schema.optional(Schema.String), diff --git a/packages/shared/src/workflows/manifest-types.ts b/packages/shared/src/workflows/manifest-types.ts index 97a66f45..dd4bd359 100644 --- a/packages/shared/src/workflows/manifest-types.ts +++ b/packages/shared/src/workflows/manifest-types.ts @@ -29,6 +29,8 @@ export interface WorkflowNodeOptions WorkflowSlackAddReactionOptions, WorkflowSlackRemoveReactionOptions, WorkflowEmailNotificationOptions { + aiSearchSourceId?: string + title?: string encoding?: string inputValues?: Record diff --git a/scripts/e2e/ai-search.ts b/scripts/e2e/ai-search.ts new file mode 100644 index 00000000..f31db1d3 --- /dev/null +++ b/scripts/e2e/ai-search.ts @@ -0,0 +1,209 @@ +import { spawn, spawnSync } from "node:child_process" +import { randomUUID } from "node:crypto" +import { mkdir, readFile, writeFile, rm, copyFile } from "node:fs/promises" +import { resolve } from "node:path" +import dotenv from "dotenv" +import { parse } from "jsonc-parser" +import { + getAiSearchNamespaceName, + getWorkflowAiSearchNamespaceName, +} from "../../packages/infra/src/aiSearch" + +if (!process.env.E2E_ENV_FILE) { + throw new Error("Set E2E_ENV_FILE to the isolated test account's private credentials file.") +} + +dotenv.config({ path: process.env.E2E_ENV_FILE, quiet: true }) + +const account = process.env.CLOUDFLARE_ACCOUNT_ID +const token = process.env.CLOUDFLARE_API_TOKEN + +if (!account || !token) throw new Error("Live AI Search requires isolated Cloudflare credentials.") + +if (spawnSync("docker", ["info"], { stdio: "ignore" }).status !== 0) { + throw new Error( + "Start Docker: Alchemy's native live AI Search binding requires its proxy container.", + ) +} + +const args = process.argv.slice(2) +const record = args.includes("--record") +const strict = args.includes("--strict-cache") +const selection = args.filter((arg) => arg !== "--record" && arg !== "--strict-cache") + +if (record && strict) throw new Error("Choose either --record or --strict-cache.") + +const cacheArgs: string[] = [] + +if (strict) cacheArgs.push("--strict-cache") +else if (!record) cacheArgs.push("--no-cache") + +if (selection.length !== 0 && (selection.length !== 2 || selection[0] !== "--grep")) { + throw new Error( + "Usage: scripts/e2e/ai-search.ts [--record | --strict-cache] [--grep ]", + ) +} + +const run = randomUUID() +const profile = `e2e-ai-search-${run.slice(0, 8)}` +const appName = `s0-e2e-as-${run.slice(0, 8)}` +const configPath = resolve(`config/${profile}-dev.config.jsonc`) +const evidencePath = resolve(".e2e/ai-search-live", run) + +const namespaces = [ + getAiSearchNamespaceName({ appName, stageName: "dev" }), + getWorkflowAiSearchNamespaceName({ appName, stageName: "dev" }), +] + +await mkdir(evidencePath, { recursive: true, mode: 0o700 }) + +const root = `https://api.cloudflare.com/client/v4/accounts/${encodeURIComponent(account)}/ai-search/namespaces` + +async function request(path: string, method = "GET") { + return fetch(`${root}/${path}`, { + method, + headers: { Authorization: `Bearer ${token}` }, + signal: AbortSignal.timeout(30_000), + }) +} + +// Establish ownership before Alchemy starts. Never adopt or clean up an existing namespace. +for (const name of namespaces) { + const response = await request(name) + + if (response.status !== 404) { + throw new Error(`Expected unused isolated namespace ${name}; received HTTP ${response.status}.`) + } +} + +const source = parse(await readFile("config/e2e-dev.config.jsonc", "utf8")) + +const config = { + ...source, + deployment: { ...source.deployment, appName, aiSearchEnabledInDev: true }, +} + +await writeFile(configPath, JSON.stringify(config, null, 2), { mode: 0o600, flag: "wx" }) + +const cleanup: Array<{ namespace: string; absent: boolean }> = [] +let exitCode = 1 + +try { + console.log(`AI Search isolated run ${run}; evidence: ${evidencePath}`) + + const child = spawn( + "nub", + [ + "exec", + "e2e", + "run", + "tests/auth.setup.e2e.ts", + "tests/ai-search.external.e2e.ts", + ...cacheArgs, + "--tag", + "ai-search-live", + ...selection, + ], + { + stdio: "inherit", + env: { + ...process.env, + E2E_CONFIG_PROFILE: profile, + E2E_CONTAINER_RUNTIME: "0", + E2E_TELEMETRY_DISABLED: "1", + }, + }, + ) + + for (const signal of ["SIGINT", "SIGTERM"] as const) { + process.once(signal, () => child.kill(signal)) + } + + exitCode = await new Promise((accept, reject) => { + child.once("error", reject) + child.once("exit", (code) => accept(code ?? 1)) + }) +} finally { + // Only these fresh, run-owned namespaces can be touched. The instance pass also handles + // a test that failed after Cloudflare created an instance but before its response arrived. + const results = await Promise.allSettled( + namespaces.map(async (name) => { + const instances = await request(`${name}/instances?per_page=100`) + + if (instances.ok) { + // SAFETY: Cloudflare's instance-list envelope; each path remains inside this owned namespace. + const body = (await instances.json()) as { result: Array<{ id: string }> } + + for (const instance of body.result) { + const removed = await request( + `${name}/instances/${encodeURIComponent(instance.id)}`, + "DELETE", + ) + + if (!removed.ok && removed.status !== 404) + throw new Error(`Instance cleanup HTTP ${removed.status}`) + } + } else if (instances.status !== 404) { + throw new Error(`Instance inventory HTTP ${instances.status}`) + } + + const removed = await request(name, "DELETE") + + if (!removed.ok && removed.status !== 404) + throw new Error(`Namespace cleanup HTTP ${removed.status}`) + + let absent = false + + // Deletion can be briefly visible to a subsequent read. Verify eventual absence. + for (let attempt = 0; attempt < 30; attempt++) { + const verified = await request(name) + + if (verified.status === 404) { + absent = true + break + } + + if (verified.status !== 200) + throw new Error(`Namespace cleanup verification HTTP ${verified.status}`) + + await new Promise((done) => setTimeout(done, 1000)) + } + + cleanup.push({ namespace: name, absent }) + + if (!absent) throw new Error("Namespace still present after cleanup verification deadline") + }), + ) + + await writeFile( + resolve(evidencePath, "ownership.json"), + JSON.stringify( + { + run, + accountId: account, + appName, + namespaces, + cleanup, + testsExitCode: exitCode, + cleanupErrors: results.flatMap((result) => + result.status === "rejected" ? [String(result.reason)] : [], + ), + }, + null, + 2, + ), + { mode: 0o600 }, + ) + await copyFile(".e2e/report.json", resolve(evidencePath, "report.json")).catch(() => undefined) + await copyFile(".e2e/summary.md", resolve(evidencePath, "summary.md")).catch(() => undefined) + await rm(configPath, { force: true }) + + if (results.some((result) => result.status === "rejected")) { + console.error(`AI Search cleanup incomplete. Inspect ${evidencePath}/ownership.json.`) + exitCode = 1 + } else { + console.log("AI Search cleanup verified: both run-owned namespaces are absent.") + } +} + +process.exitCode = exitCode diff --git a/scripts/e2e/dev.ts b/scripts/e2e/dev.ts index 675a2028..d25fa352 100644 --- a/scripts/e2e/dev.ts +++ b/scripts/e2e/dev.ts @@ -5,7 +5,7 @@ import { rmSync, writeFileSync } from "node:fs" import { resolve } from "node:path" import { bundleAgentContainer } from "../agent-container-build" import { AGENT_CONTAINER_ENTRYPOINTS } from "../../packages/agent-container/src/images" -import { appEnvironment } from "./environment" +import { appEnvironment, testDeploymentConfig } from "./environment" const repo = resolve(import.meta.dirname, "../..") const runId = randomUUID() @@ -68,7 +68,15 @@ try { // The e2e profile answers model calls inside the Worker, so local alchemy dev needs no deploy token. // Alchemy selects Bun when npm_execpath contains "bun", which the Nub binary path can match. - const { CLOUDFLARE_API_TOKEN: _deployToken, npm_execpath: _execPath, ...env } = appEnvironment + const { CLOUDFLARE_API_TOKEN: deployToken, npm_execpath: _execPath, ...env } = appEnvironment + + // Live AI Search is explicitly enabled by an isolated profile. Its native remote binding + // needs the deployment credential; ordinary core runs continue to receive none. + if (testDeploymentConfig.deployment.aiSearchEnabledInDev) { + if (!deployToken) throw new Error("Live AI Search requires CLOUDFLARE_API_TOKEN.") + + env.CLOUDFLARE_API_TOKEN = deployToken + } // Alchemy dev leads its own process group, so one signal stops its workerd and watch children. const child = spawn( diff --git a/scripts/sync-github-env-secrets.ts b/scripts/sync-github-env-secrets.ts index 91f26005..b4bd33e0 100644 --- a/scripts/sync-github-env-secrets.ts +++ b/scripts/sync-github-env-secrets.ts @@ -5,7 +5,6 @@ import dotenv from "dotenv" const defaultConfig = { preVarsPath: "config/.pre.vars", prodVarsPath: "config/.prod.vars", - infraEnvPath: "config/.env", } const githubSecretNameMap = { @@ -22,18 +21,13 @@ function isMappedSecretName(name: string): name is keyof typeof githubSecretName // and CLOUDFLARE_API_TOKEN must stay off this public repository. const orgSecretNames = ["CLOUDFLARE_ACCOUNT_ID", "CLOUDFLARE_API_TOKEN"] as const -const repoSecretNames = ["CF_AI_SEARCH_SERVICE_TOKEN_ID"] as const - -const skippedRuntimeSecretNameSet = new Set([...orgSecretNames, ...repoSecretNames]) +const skippedRuntimeSecretNameSet = new Set(orgSecretNames) interface Args { repo: string preVarsPath: string prodVarsPath: string - infraEnvPath: string apply: boolean - syncRuntime: boolean - syncInfra: boolean useGitHubTokenEnv: boolean } @@ -56,9 +50,7 @@ Options: --repo GitHub repository. Required. --pre-vars Preview runtime vars file. Default: ${defaultConfig.preVarsPath} --prod-vars Production runtime vars file. Default: ${defaultConfig.prodVarsPath} - --infra-env Repo-level infra env file. Default: ${defaultConfig.infraEnvPath} - --runtime-only Sync only pre/prod environment secrets. - --infra-only Sync only repo-level infra secrets. + --runtime-only Sync pre/prod environment secrets (the default). --use-github-token-env Preserve GITHUB_TOKEN when invoking gh. --help Show this help. @@ -78,10 +70,7 @@ function parseArgs(argv: string[]): Args { repo: "", preVarsPath: defaultConfig.preVarsPath, prodVarsPath: defaultConfig.prodVarsPath, - infraEnvPath: defaultConfig.infraEnvPath, apply: false, - syncRuntime: true, - syncInfra: true, useGitHubTokenEnv: false, } @@ -103,16 +92,7 @@ function parseArgs(argv: string[]): Args { case "--prod-vars": args.prodVarsPath = readOptionValue(argv, ++i, arg) break - case "--infra-env": - args.infraEnvPath = readOptionValue(argv, ++i, arg) - break case "--runtime-only": - args.syncRuntime = true - args.syncInfra = false - break - case "--infra-only": - args.syncRuntime = false - args.syncInfra = true break case "--use-github-token-env": args.useGitHubTokenEnv = true @@ -173,26 +153,6 @@ function runtimeUploads(environment: string, path: string): SecretUpload[] { }) } -function infraUploads(path: string): SecretUpload[] { - const parsed = parseDotenvFile(path) - - return repoSecretNames.flatMap((name) => { - const value = parsed[name] - - if (!value) { - return [] - } - - return [ - { - scope: "repository", - name, - value, - }, - ] - }) -} - function childEnv(useGitHubTokenEnv: boolean): NodeJS.ProcessEnv { const env = { ...process.env } @@ -290,14 +250,8 @@ function main(): void { const uploads: SecretUpload[] = [] - if (args.syncRuntime) { - uploads.push(...runtimeUploads("pre", args.preVarsPath)) - uploads.push(...runtimeUploads("prod", args.prodVarsPath)) - } - - if (args.syncInfra) { - uploads.push(...infraUploads(args.infraEnvPath)) - } + uploads.push(...runtimeUploads("pre", args.preVarsPath)) + uploads.push(...runtimeUploads("prod", args.prodVarsPath)) printPlan(uploads, args.apply) @@ -305,10 +259,8 @@ function main(): void { return } - if (args.syncRuntime) { - ensureEnvironment(args.repo, "pre", args.useGitHubTokenEnv) - ensureEnvironment(args.repo, "prod", args.useGitHubTokenEnv) - } + ensureEnvironment(args.repo, "pre", args.useGitHubTokenEnv) + ensureEnvironment(args.repo, "prod", args.useGitHubTokenEnv) for (const upload of uploads) { setSecret(args.repo, upload, args.useGitHubTokenEnv) diff --git a/tests/ai-search-contracts.e2e.ts b/tests/ai-search-contracts.e2e.ts new file mode 100644 index 00000000..15861467 --- /dev/null +++ b/tests/ai-search-contracts.e2e.ts @@ -0,0 +1,136 @@ +import { describe, expect, test } from "e2e" +import * as Schema from "effect/Schema" +import * as Option from "effect/Option" +import { + AdminAiSearchSourcePayload, + AdminAiSearchDataSource, +} from "../packages/api/src/http/schemas/admin" +import { readAiSearchSourceRecord } from "../packages/api/src/server/background/db/ai-search-records" +import { + validateWorkflowDraft, + getStageMetadataFromConfigSync, + resolveS0Config, +} from "@solzero/shared" +import { exampleDeploymentConfig } from "./support/deployment-config" + +const source = { + id: "migration-original", + label: "Original source", + description: "Retain until verified", + enabled: true, + maxResults: 5, + createdAt: 1700000000000, + updatedAt: 1700000000001, +} + +const legacySources = [ + { type: "r2", bucketName: "original-documents", prefix: "knowledge/", r2Jurisdiction: null }, + { + type: "website", + domain: "docs.example.test", + includePaths: ["/docs/*"], + excludePaths: ["/private/*"], + specificSitemaps: ["https://docs.example.test/sitemap.xml"], + useBrowserRendering: true, + includeImages: true, + }, +] + +describe("AI Search source transition", { platforms: ["tools"] }, () => { + test("local AI Search is opt-in and standing stages retain their namespace bindings", async () => { + const base = exampleDeploymentConfig() + const config = resolveS0Config(base) + + expect( + getStageMetadataFromConfigSync("dev", config.deployment, config.application).infra + .aiSearchNamespaceEnabled, + ).toBe(false) + expect( + getStageMetadataFromConfigSync("pre", config.deployment, config.application).infra + .aiSearchNamespaceEnabled, + ).toBe(true) + + const local = resolveS0Config({ + ...base, + deployment: { ...base.deployment, aiSearchEnabledInDev: true }, + }) + + expect( + getStageMetadataFromConfigSync("dev", local.deployment, local.application).infra + .aiSearchNamespaceEnabled, + ).toBe(true) + }) + + test("new source requests accept built-in storage and reject external storage", async () => { + const decode = Schema.decodeUnknownSync(AdminAiSearchSourcePayload) + + expect(decode({ ...source, dataSource: { type: "built-in" } }).dataSource.type).toBe("built-in") + + for (const dataSource of legacySources) { + expect( + Option.isNone( + Schema.decodeUnknownOption(AdminAiSearchSourcePayload)({ ...source, dataSource }), + ), + ).toBe(true) + } + }) + + test("legacy records retain identity, timestamps and all source details for export", async () => { + for (const dataSource of legacySources) { + const original = { ...source, dataSource } + const snapshot = JSON.stringify(original) + const parsed = readAiSearchSourceRecord(original) + + expect(Option.isSome(parsed)).toBe(true) + expect(Option.getOrThrow(parsed)).toEqual(original) + expect(Schema.decodeUnknownSync(AdminAiSearchDataSource)(dataSource)).toEqual(dataSource) + expect(JSON.stringify(original)).toBe(snapshot) + } + }) + + test("the upload node compiles as an additive action while existing R2 nodes remain valid", async () => { + for (const action of [ + { + type: "ai-search-upload" as const, + options: { aiSearchSourceId: "knowledge", name: "document.txt" }, + }, + { + type: "r2-put-object" as const, + options: { bucket: "AI_SEARCH_CONTENT_BUCKET", key: "original.txt" }, + }, + { + type: "r2-put-object" as const, + options: { bucket: "WORKFLOW_BUCKET", key: "artifact.txt" }, + }, + ]) { + const result = validateWorkflowDraft({ + version: 4, + name: "Ingestion compatibility", + nodes: [ + { + id: "start", + type: "manual-trigger", + label: "Start", + position: { x: 0, y: 0 }, + options: {}, + }, + { id: "upload", ...action, label: "Upload", position: { x: 300, y: 0 } }, + ], + edges: [ + { + id: "content", + source: "start", + target: "upload", + sourceHandle: "payload", + targetHandle: "content", + }, + ], + }) + + expect(result.errors).toEqual([]) + expect(result.valid).toBe(true) + expect(result.manifest?.nodes[1].type).toBe(action.type) + expect(result.manifest?.nodes[1].options).toEqual(action.options) + } + }) +}) diff --git a/tests/ai-search.external.e2e.ts b/tests/ai-search.external.e2e.ts new file mode 100644 index 00000000..cca09f2a --- /dev/null +++ b/tests/ai-search.external.e2e.ts @@ -0,0 +1,274 @@ +import { test } from "@e2e-dev/web" +import { expect, unique } from "e2e" +import { Client } from "@modelcontextprotocol/sdk/client/index.js" +import { StreamableHTTPClientTransport } from "@modelcontextprotocol/sdk/client/streamableHttp.js" +import type { Browser } from "@e2e-dev/web" +import { api as callApi, authRequest, uniqueName } from "./support/api" +import { testDeploymentConfig } from "../scripts/e2e/environment" + +function api( + browser: Browser, + path: string, + method = "GET", + body?: Body, +) { + return callApi(browser, path, method, body, 45_000) +} + +interface Items { + items: Array<{ id: string; key: string; status: string }> + total: number +} + +async function search(sourceId: string, query: string) { + const client = new Client({ name: "solzero-ai-search-proof", version: "1.0.0" }) + + try { + await client.connect( + new StreamableHTTPClientTransport(new URL("http://localhost:3100/mcp"), { + requestInit: { headers: { "x-s0-ai-search-sources": sourceId } }, + }), + ) + + const result = await client.callTool({ + name: `search_${sourceId.replaceAll("-", "_")}`, + arguments: { query }, + }) + + expect(result.isError).not.toBe(true) + + return JSON.stringify(result.content) + } finally { + await client.close() + } +} + +test( + "built-in documents index, retrieve, delete and ingest through a real workflow", + { + session: "admin", + tags: ["external", "ai-search-live"], + timeout: 600_000, + retries: 0, + }, + async ({ browser }) => { + expect(testDeploymentConfig.deployment.appName.startsWith("s0-e2e-as-")).toBe(true) + expect(testDeploymentConfig.deployment.aiSearchEnabledInDev).toBe(true) + + const sourceId = uniqueName("e2e-docs") + const label = uniqueName("Built-in documents") + const marker = uniqueName("violet-otter") + const name = "round-trip.txt" + const path = `/admin/ai-search/sources/${sourceId}` + let workflowId: string | undefined + + try { + await api(browser, "/admin/ai-search/export", "POST") + console.info("AI Search proof: audited configuration export completed") + await api(browser, "/admin/ai-search/sources", "POST", { + id: sourceId, + label, + description: "Disposable integration proof", + enabled: true, + maxResults: 5, + dataSource: { type: "built-in" }, + }) + + console.info("AI Search proof: source creation completed") + + const sources = await api<{ sources: Array<{ id: string; dataSource: { type: string } }> }>( + browser, + "/admin/ai-search", + ) + + expect(sources.sources.find((source) => source.id === sourceId)?.dataSource.type).toBe( + "built-in", + ) + + const payload = { + name, + contentType: "text/plain", + contentBase64: Buffer.from( + `The archival retrieval marker is ${marker}. Its color is violet and its animal is an otter.`, + ).toString("base64"), + } + + const denied = await fetch(`http://localhost:3100${path}/items`, { + method: "POST", + headers: { "Content-Type": "application/json", Origin: "http://localhost:3000" }, + body: JSON.stringify(payload), + }) + + expect(denied.status).toBe(401) + + const malformed = await authRequest(browser, `${path}/items`, "POST", { + ...payload, + contentBase64: "%%%", + }) + + expect(malformed.status).toBe(400) + await api(browser, `${path}/items`, "POST", payload) + console.info("AI Search proof: upload accepted") + + await expect + .poll( + async () => { + const items = await api(browser, `${path}/items`) + + return items.items.find((item) => item.key === name)?.status + }, + { timeout: 240_000, interval: 2000 }, + ) + .toBe("completed") + await expect + .poll(() => search(sourceId, "What is the archival retrieval marker?"), { + timeout: 120_000, + }) + .toContain(marker) + + const listed = await api(browser, `${path}/items`) + const itemId = listed.items.find((item) => item.key === name)!.id + await api(browser, `${path}/items/${itemId}`, "DELETE") + await expect.poll(async () => (await api(browser, `${path}/items`)).total).toBe(0) + await expect + .poll(() => search(sourceId, "What is the archival retrieval marker?"), { + timeout: 120_000, + }) + .not.toContain(marker) + + const workflowName = uniqueName("AI Search ingestion") + const workflowMarker = uniqueName("copper-orchid") + + const created = await api<{ workflow: { id: string } }>(browser, "/workflows", "POST", { + name: workflowName, + manifest: { + version: 4, + name: workflowName, + nodes: [ + { + id: "start", + type: "manual-trigger", + label: "Start", + position: { x: 0, y: 0 }, + options: {}, + }, + { + id: "upload", + type: "ai-search-upload", + label: "Upload", + position: { x: 300, y: 0 }, + options: { aiSearchSourceId: sourceId, name: "workflow.txt" }, + }, + ], + edges: [ + { + id: "content", + source: "start", + target: "upload", + sourceHandle: "payload", + targetHandle: "content", + }, + ], + }, + }) + + workflowId = created.workflow.id + const workflowPath = `/workflows/${workflowId}` + + const { run } = await api<{ run: { id: string } }>(browser, `${workflowPath}/runs`, "POST", { + trigger: { kind: "manual", payload: { marker: workflowMarker } }, + }) + + await expect + .poll( + async () => + (await api<{ run: { status: string } }>(browser, `${workflowPath}/runs/${run.id}`)).run + .status, + { timeout: 120_000, interval: 2000 }, + ) + .toBe("completed") + + const events = await api<{ events: Array<{ eventType: string }> }>( + browser, + `${workflowPath}/runs/${run.id}/events`, + ) + + expect(events.events.some((event) => event.eventType === "ai_search_upload")).toBe(true) + await expect + .poll( + async () => + (await api(browser, `${path}/items`)).items.find( + (item) => item.key === "workflow.txt", + )?.status, + { timeout: 240_000, interval: 2000 }, + ) + .toBe("completed") + await expect + .poll(() => search(sourceId, "What marker was uploaded by the workflow?"), { + timeout: 120_000, + }) + .toContain(workflowMarker) + } finally { + if (workflowId) await api(browser, `/workflows/${workflowId}`, "DELETE") + + const sources = await api<{ sources: Array<{ id: string }> }>(browser, "/admin/ai-search") + + if (sources.sources.some((source) => source.id === sourceId)) + await api(browser, path, "DELETE") + } + }, +) + +test( + "administrators create built-in sources and open their documents", + { + session: "admin", + tags: ["external", "ai-search-live"], + retries: 0, + agentContext: + "This test has approval to create a disposable built-in AI Search source in its unique isolated Cloudflare namespace. Creating this source is allowed and does not link an external account. Use only the supplied test ID and label.", + }, + async ({ app, browser, agent, screen }) => { + const id = uniqueName("e2e-ui-docs") + const label = uniqueName("Documents UI") + + try { + await app.open("/admin/ai-search") + await expect(screen.getByRole("button", "New Source", { exact: true })).toBeVisible() + await agent.act("Click New Source to open the Create Source dialog.") + await expect(screen.getByRole("dialog", "Create Source", { exact: true })).toBeVisible() + await screen.getByLabel("ID", { exact: true }).fill(id) + await screen.getByLabel("Label", { exact: true }).fill(label) + await agent.act( + "Submit the source with ID {id} and label {label} by clicking Create Source once. Stop when the button shows Loading Create Source or the dialog closes.", + { params: { id: unique(id), label: unique(label) } }, + ) + await expect(screen.getByRole("dialog", "Create Source", { exact: true })).not.toBeVisible({ + timeout: 60_000, + }) + await expect(screen.getByText(label, { exact: true })).toBeVisible() + + const sources = await api<{ sources: Array<{ id: string; dataSource: { type: string } }> }>( + browser, + "/admin/ai-search", + ) + + expect(sources.sources.find((source) => source.id === id)?.dataSource.type).toBe("built-in") + await expect( + screen.getByRole("dialog", "AI Search source created.", { exact: true }), + ).not.toBeVisible({ timeout: 30_000 }) + await agent.act( + "Open Documents for source {id} in the row labelled {label}. Finish when Close documents is visible.", + { params: { id: unique(id), label: unique(label) } }, + ) + await expect(screen.getByRole("button", "Close documents")).toBeVisible() + await expect(screen.getByLabel("Upload document")).toBeVisible() + await app.screenshot("built-in-ai-search-documents") + } finally { + const sources = await api<{ sources: Array<{ id: string }> }>(browser, "/admin/ai-search") + + if (sources.sources.some((source) => source.id === id)) + await api(browser, `/admin/ai-search/sources/${id}`, "DELETE") + } + }, +) diff --git a/tests/support/api.ts b/tests/support/api.ts index 7b791841..445ef9f1 100644 --- a/tests/support/api.ts +++ b/tests/support/api.ts @@ -6,8 +6,9 @@ export async function api( path: string, method = "GET", body?: Body, + requestTimeoutMs?: number, ): Promise { - const response = await authRequest(browser, path, method, body) + const response = await authRequest(browser, path, method, body, requestTimeoutMs) expect(response.status, `${method} ${path}`).toBeGreaterThanOrEqual(200) expect(response.status, `${method} ${path}`).toBeLessThan(300) @@ -20,6 +21,7 @@ export async function authRequest( path: string, method = "GET", body?: Body, + requestTimeoutMs?: number, ): Promise<{ status: number; payload: unknown }> { const cookies = await browser.cookies() @@ -34,6 +36,8 @@ export async function authRequest( if (body) init.body = JSON.stringify(body) + if (requestTimeoutMs) init.signal = AbortSignal.timeout(requestTimeoutMs) + const response = await fetch(new URL(path, "http://localhost:3100"), init) return {