diff --git a/charts/adapter-service/CHANGELOG.md b/charts/adapter-service/CHANGELOG.md index bccc2755..fc4d3486 100644 --- a/charts/adapter-service/CHANGELOG.md +++ b/charts/adapter-service/CHANGELOG.md @@ -5,6 +5,20 @@ All notable changes to this chart are documented here. The format follows [Keep a Changelog](https://keepachangelog.com/en/1.1.0/), and this chart adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). +## [0.7.0] - 2026-10-26 + +### Added +- The catalog crawler (the scheduled catalog publisher) is on for the + network-adapter role in both deployment modes: its `registry` and `crawler` + plugin blocks are in `config/single-adapter/network-adapter-config.yaml` and + `config/multi-adapter/network-config.yaml`. Catalogs go to this pod's + `/publish` (single) or the provider adapter's (multi), sent as the provider + to `discovery.oan.dev`. No template change: the deployment fills the run log + DSN placeholder `__ADAPTER_CRAWLER_DB_DSN__` through `extraSubstitutions` and + mounts `/tmp/catalogs` through `extraVolumes` (see README). Needs the registry + and registry-seed from #96 (a `publishPipelineUrl` on the binding), a run log + database of its own (not discovery's), and a crawler-capable image. + ## [0.6.0] - 2026-10-09 ### Added diff --git a/charts/adapter-service/Chart.yaml b/charts/adapter-service/Chart.yaml index b7e45e17..5c97e6b8 100644 --- a/charts/adapter-service/Chart.yaml +++ b/charts/adapter-service/Chart.yaml @@ -2,7 +2,7 @@ apiVersion: v2 name: adapter-service description: OAN Beckn adapter; role network-adapter runs all three tiers in single-adapter mode (default), or the network tier in multi-adapter. type: application -version: 0.6.0 +version: 0.7.0 # The adapter binary, not this chart. `/health` reports the build it is # actually running, which is the number to trust over this one. appVersion: "v1.9.0" diff --git a/charts/adapter-service/README.md b/charts/adapter-service/README.md index ad3c2c48..318d64b2 100644 --- a/charts/adapter-service/README.md +++ b/charts/adapter-service/README.md @@ -84,6 +84,56 @@ kubectl -n rollout restart deploy/ Any other values change rolls the pods on its own. +## Catalog crawler + +The network adapter's config files carry the catalogcrawler's `registry` and +`crawler` plugin blocks (`config/single-adapter/network-adapter-config.yaml`, +`config/multi-adapter/network-config.yaml`). On every tick it runs the pipeline +of each registry binding with a `publish` action and POSTs the catalogs to +`/publish` (this pod in single mode, the provider adapter in multi mode). + +The deployment wires it with existing values, no template change: + +```yaml +# The run log DSN, from a key in the keys Secret. Single mode: +adapterKeys: + extraSubstitutions: + __ADAPTER_CRAWLER_DB_DSN__: network/crawlerDbDsn +# Multi mode: +keys: + existingSecret: + extraSubstitutions: + __ADAPTER_CRAWLER_DB_DSN__: crawlerDbDsn +# Built catalogs (the root filesystem is read-only). +extraVolumes: + - name: crawler-catalogs + emptyDir: + sizeLimit: 512Mi +extraVolumeMounts: + - name: crawler-catalogs + mountPath: /tmp/catalogs +``` + +Without the substitution the pod stops at start-up ("placeholders remain"). +The DSN must hold no `&` or `|` (the substitution uses `sed`); a password +percent-encoded into it is fine. + +It needs, outside this chart: + +- an image with the `catalogcrawler` plugin and the `render.mappingByField` engine; +- the `publish` action in the registry: the registry and registry-seed from #96, and + `bindings[].publishPipelineUrl` in registry-seed, pinned to a commit SHA; +- its own run log database (PostgreSQL), not discovery's: e.g. a `catalogcrawler` + database and role on the shared `postgres` cluster. Its DSN goes in the keys + Secret (the network tier's Secret in single mode), named by the substitution above; +- the pipeline's inputs as env: `MANDI_API_URI`, `MANDI_PARTICIPANT_ID`, + `APP_NETWORK_ID`, and the provider's Agmarknet credentials + `AGMARKNET_ACCESS_NAME` and `AGMARKNET_PASSWORD`. Single mode already has those + two from the provider's upstream Secret; in multi mode map them with `extraEnv` + `secretKeyRef`s into that Secret rather than copying them into a new one. + +To run without it, remove both blocks from the config file (or from `config`). + ## Values and examples - `values.yaml`: every field, with a comment on what it does. diff --git a/charts/adapter-service/config/multi-adapter/network-config.yaml b/charts/adapter-service/config/multi-adapter/network-config.yaml index 07163131..7a075382 100644 --- a/charts/adapter-service/config/multi-adapter/network-config.yaml +++ b/charts/adapter-service/config/multi-adapter/network-config.yaml @@ -40,6 +40,41 @@ plugins: enableTracing: "true" enableLogs: "true" + # Scheduled catalog publishing (catalogcrawler). On every tick it lists the + # registry's bindings, runs each `publish` pipeline that is due and POSTs the + # catalogs to the provider adapter's /publish, which signs them as the provider and sends them on + # to discovery. Needs a crawler-capable image, the run log database (its DSN + # comes from the keys Secret through extraSubstitutions, see README) and the pipeline's + # inputs as env (MANDI_*, APP_NETWORK_ID, AGMARKNET_*). Remove both blocks to turn it off. + # This registry block is the crawler's own lookup; the modules' is separate. + registry: + id: sunbirdRegistry + config: + url: http://registry.registry.svc.cluster.local:8081/api/v1 + entity: Participant + providerEntity: ProviderSchema + + crawler: + id: catalogcrawler + config: + dbDsn: "__ADAPTER_CRAWLER_DB_DSN__" + discoveryPushUrl: "http://provider-adapter.provider-adapter.svc.cluster.local:9200/publish" + # Who the catalogs are published AS (context.senderId) and TO (receiverId). + participantId: "provider.oan.dev" + receiverId: "discovery.oan.dev" + publishPipelines: "true" + publishEnabled: "true" + # Every 6 hours. + catalogIntervalSeconds: "21600" + # An emptyDir from extraVolumes (read-only root filesystem), see README. + publishCatalogOutputDir: "/tmp/catalogs" + # Per-pipeline overrides, by metadata.name. Once a day, OnDemand catalogs + # only. Agmarknet data lags 1-3 weeks: a relative window, not the + # pipeline's default (yesterday). + publish.mandi-price.schedule: "0 13 * * *" + publish.mandi-price.mode: "onDemand" + publish.mandi-price.fromDate: "21 days ago" + modules: - name: network-adapter # A subtree: every action lands here and the payload says which one it is. diff --git a/charts/adapter-service/config/single-adapter/network-adapter-config.yaml b/charts/adapter-service/config/single-adapter/network-adapter-config.yaml index 4acbe2ae..80936874 100644 --- a/charts/adapter-service/config/single-adapter/network-adapter-config.yaml +++ b/charts/adapter-service/config/single-adapter/network-adapter-config.yaml @@ -33,6 +33,41 @@ plugins: enableTracing: "true" enableLogs: "true" + # Scheduled catalog publishing (catalogcrawler). On every tick it lists the + # registry's bindings, runs each `publish` pipeline that is due and POSTs the + # catalogs to this adapter's own /publish, which signs them as the provider and sends them on + # to discovery. Needs a crawler-capable image, the run log database (its DSN + # comes from the keys Secret through extraSubstitutions, see README) and the pipeline's + # inputs as env (MANDI_*, APP_NETWORK_ID, AGMARKNET_*). Remove both blocks to turn it off. + # This registry block is the crawler's own lookup; the modules' is separate. + registry: + id: sunbirdRegistry + config: + url: http://registry.registry.svc.cluster.local:8081/api/v1 + entity: Participant + providerEntity: ProviderSchema + + crawler: + id: catalogcrawler + config: + dbDsn: "__ADAPTER_CRAWLER_DB_DSN__" + discoveryPushUrl: "http://localhost:9201/publish" + # Who the catalogs are published AS (context.senderId) and TO (receiverId). + participantId: "__PROVIDER_SUBSCRIBER_ID__" + receiverId: "discovery.oan.dev" + publishPipelines: "true" + publishEnabled: "true" + # Every 6 hours. + catalogIntervalSeconds: "21600" + # An emptyDir from extraVolumes (read-only root filesystem), see README. + publishCatalogOutputDir: "/tmp/catalogs" + # Per-pipeline overrides, by metadata.name. Once a day, OnDemand catalogs + # only. Agmarknet data lags 1-3 weeks: a relative window, not the + # pipeline's default (yesterday). + publish.mandi-price.schedule: "0 13 * * *" + publish.mandi-price.mode: "onDemand" + publish.mandi-price.fromDate: "21 days ago" + modules: # consumer: unsigned app calls at / (the catch-all mount, on purpose: see ADR-0001); # discover to /_internal/network, transactions to /_internal/provider.