feat(adapter-service,registry,registry-seed): enable the catalog crawler for publishing catalogues

charts/adapter-service/CHANGELOG.md

@@ -5,6 +5,20 @@ All notable changes to this chart are documented here.
 The format follows [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
 and this chart adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
+## [0.7.0] - 2026-10-26
+
+### Added
+- The catalog crawler (the scheduled catalog publisher) is on for the
+  network-adapter role in both deployment modes: its `registry` and `crawler`
+  plugin blocks are in `config/single-adapter/network-adapter-config.yaml` and
+  `config/multi-adapter/network-config.yaml`. Catalogs go to this pod's
+  `/publish` (single) or the provider adapter's (multi), sent as the provider
+  to `discovery.oan.dev`. No template change: the deployment fills the run log
+  DSN placeholder `__ADAPTER_CRAWLER_DB_DSN__` through `extraSubstitutions` and
+  mounts `/tmp/catalogs` through `extraVolumes` (see README). Needs the registry
+  and registry-seed from #96 (a `publishPipelineUrl` on the binding), a run log
+  database of its own (not discovery's), and a crawler-capable image.
+
 ## [0.6.0] - 2026-10-09
 
 ### Added

charts/adapter-service/Chart.yaml

@@ -2,7 +2,7 @@ apiVersion: v2
 name: adapter-service
 description: OAN Beckn adapter; role network-adapter runs all three tiers in single-adapter mode (default), or the network tier in multi-adapter.
 type: application
-version: 0.6.0
+version: 0.7.0
 # The adapter binary, not this chart. `/health` reports the build it is
 # actually running, which is the number to trust over this one.
 appVersion: "v1.9.0"

charts/adapter-service/README.md

@@ -84,6 +84,56 @@ kubectl -n <namespace> rollout restart deploy/<release>
 
 Any other values change rolls the pods on its own.
 
+## Catalog crawler
+
+The network adapter's config files carry the catalogcrawler's `registry` and
+`crawler` plugin blocks (`config/single-adapter/network-adapter-config.yaml`,
+`config/multi-adapter/network-config.yaml`). On every tick it runs the pipeline
+of each registry binding with a `publish` action and POSTs the catalogs to
+`/publish` (this pod in single mode, the provider adapter in multi mode).
+
+The deployment wires it with existing values, no template change:
+
+```yaml
+# The run log DSN, from a key in the keys Secret. Single mode:
+adapterKeys:
+  extraSubstitutions:
+    __ADAPTER_CRAWLER_DB_DSN__: network/crawlerDbDsn
+# Multi mode:
+keys:
+  existingSecret:
+    extraSubstitutions:
+      __ADAPTER_CRAWLER_DB_DSN__: crawlerDbDsn
+# Built catalogs (the root filesystem is read-only).
+extraVolumes:
+  - name: crawler-catalogs
+    emptyDir:
+      sizeLimit: 512Mi
+extraVolumeMounts:
+  - name: crawler-catalogs
+    mountPath: /tmp/catalogs
+```
+
+Without the substitution the pod stops at start-up ("placeholders remain").
+The DSN must hold no `&` or `|` (the substitution uses `sed`); a password
+percent-encoded into it is fine.
+
+It needs, outside this chart:
+
+- an image with the `catalogcrawler` plugin and the `render.mappingByField` engine;
+- the `publish` action in the registry: the registry and registry-seed from #96, and
+  `bindings[].publishPipelineUrl` in registry-seed, pinned to a commit SHA;
+- its own run log database (PostgreSQL), not discovery's: e.g. a `catalogcrawler`
+  database and role on the shared `postgres` cluster. Its DSN goes in the keys
+  Secret (the network tier's Secret in single mode), named by the substitution above;
+- the pipeline's inputs as env: `MANDI_API_URI`, `MANDI_PARTICIPANT_ID`,
+  `APP_NETWORK_ID`, and the provider's Agmarknet credentials
+  `AGMARKNET_ACCESS_NAME` and `AGMARKNET_PASSWORD`. Single mode already has those
+  two from the provider's upstream Secret; in multi mode map them with `extraEnv`
+  `secretKeyRef`s into that Secret rather than copying them into a new one.
+
+To run without it, remove both blocks from the config file (or from `config`).
+
 ## Values and examples
 
 - `values.yaml`: every field, with a comment on what it does.

charts/adapter-service/config/multi-adapter/network-config.yaml

@@ -40,6 +40,41 @@ plugins:
       enableTracing: "true"
       enableLogs: "true"
 
+  # Scheduled catalog publishing (catalogcrawler). On every tick it lists the
+  # registry's bindings, runs each `publish` pipeline that is due and POSTs the
+  # catalogs to the provider adapter's /publish, which signs them as the provider and sends them on
+  # to discovery. Needs a crawler-capable image, the run log database (its DSN
+  # comes from the keys Secret through extraSubstitutions, see README) and the pipeline's
+  # inputs as env (MANDI_*, APP_NETWORK_ID, AGMARKNET_*). Remove both blocks to turn it off.
+  # This registry block is the crawler's own lookup; the modules' is separate.
+  registry:
+    id: sunbirdRegistry
+    config:
+      url: http://registry.registry.svc.cluster.local:8081/api/v1
+      entity: Participant
+      providerEntity: ProviderSchema
+
+  crawler:
+    id: catalogcrawler
+    config:
+      dbDsn: "__ADAPTER_CRAWLER_DB_DSN__"
+      discoveryPushUrl: "http://provider-adapter.provider-adapter.svc.cluster.local:9200/publish"
+      # Who the catalogs are published AS (context.senderId) and TO (receiverId).
+      participantId: "provider.oan.dev"
+      receiverId: "discovery.oan.dev"
+      publishPipelines: "true"
+      publishEnabled: "true"
+      # Every 6 hours.
+      catalogIntervalSeconds: "21600"
+      # An emptyDir from extraVolumes (read-only root filesystem), see README.
+      publishCatalogOutputDir: "/tmp/catalogs"
+      # Per-pipeline overrides, by metadata.name. Once a day, OnDemand catalogs
+      # only. Agmarknet data lags 1-3 weeks: a relative window, not the
+      # pipeline's default (yesterday).
+      publish.mandi-price.schedule: "0 13 * * *"
+      publish.mandi-price.mode: "onDemand"
+      publish.mandi-price.fromDate: "21 days ago"
+
 modules:
   - name: network-adapter
     # A subtree: every action lands here and the payload says which one it is.

charts/adapter-service/config/single-adapter/network-adapter-config.yaml

@@ -33,6 +33,41 @@ plugins:
       enableTracing: "true"
       enableLogs: "true"
 
+  # Scheduled catalog publishing (catalogcrawler). On every tick it lists the
+  # registry's bindings, runs each `publish` pipeline that is due and POSTs the
+  # catalogs to this adapter's own /publish, which signs them as the provider and sends them on
+  # to discovery. Needs a crawler-capable image, the run log database (its DSN
+  # comes from the keys Secret through extraSubstitutions, see README) and the pipeline's
+  # inputs as env (MANDI_*, APP_NETWORK_ID, AGMARKNET_*). Remove both blocks to turn it off.
+  # This registry block is the crawler's own lookup; the modules' is separate.
+  registry:
+    id: sunbirdRegistry
+    config:
+      url: http://registry.registry.svc.cluster.local:8081/api/v1
+      entity: Participant
+      providerEntity: ProviderSchema
+
+  crawler:
+    id: catalogcrawler
+    config:
+      dbDsn: "__ADAPTER_CRAWLER_DB_DSN__"
+      discoveryPushUrl: "http://localhost:9201/publish"
+      # Who the catalogs are published AS (context.senderId) and TO (receiverId).
+      participantId: "__PROVIDER_SUBSCRIBER_ID__"
+      receiverId: "discovery.oan.dev"
+      publishPipelines: "true"
+      publishEnabled: "true"
+      # Every 6 hours.
+      catalogIntervalSeconds: "21600"
+      # An emptyDir from extraVolumes (read-only root filesystem), see README.
+      publishCatalogOutputDir: "/tmp/catalogs"
+      # Per-pipeline overrides, by metadata.name. Once a day, OnDemand catalogs
+      # only. Agmarknet data lags 1-3 weeks: a relative window, not the
+      # pipeline's default (yesterday).
+      publish.mandi-price.schedule: "0 13 * * *"
+      publish.mandi-price.mode: "onDemand"
+      publish.mandi-price.fromDate: "21 days ago"
+
 modules:
   # consumer: unsigned app calls at / (the catch-all mount, on purpose: see ADR-0001);
   # discover to /_internal/network, transactions to /_internal/provider.