Repository navigation
Catalog crawler: enabling it for publishing catalogues (follow-up to OpenAgriNet/network-adapter#54) #77
Description
OpenAgriNet Network Adapter — Mandi Price Catalog (On Demand) Crawler Helm Integration
Context
OpenAgriNet/network-adapter#54, merged into development on 2026-10-07, introduced the catalog crawler as a plugin within the network adapter process.
On each scheduled tick, the crawler:
- Queries the registry for bindings that have a
publishaction. - Runs the pipeline specified by the action.
- Builds the catalogue from the pipeline output.
- POSTs the catalogue to the provider adapter's
/publishendpoint. - The provider adapter signs the catalogue and sends it through the network adapter to discovery.
This issue tracks enabling the crawler through the Helm charts and quick-start configuration so that catalogues can actually be published.
Activity
feat(adapter-service,registry,registry-seed): enable the catalog craw…
feat(quick-start): enable the catalog crawler for publishing catalogues
refactor(adapter-service): simplify crawler wiring, escape subst values
refactor(quick-start): tidy setup.py publish action and crawler comments
refactor(adapter-service): put the catalog crawler in the config files
refactor(quick-start): write the crawler's publish ids in the config …
refactor(registry,registry-seed): move the publish action to its own PR
refactor(registry-seed): drop the render-time pipeline URL check
fix(registry,registry-seed): keep the existing action schema, simplif…
fix(registry): require https for publish pipeline URLs
fix(quick-start): match the registry's publish action schema
refactor(quick-start): give the catalog crawler its own Postgres
docs(adapter-service): the crawler's run log needs its own database
kelvinprabhu commentedon Oct 28, 2026
Dev cluster rollout tracked in OpenAgriNet/infra-automation#19 (PR OpenAgriNet/infra-automation#20). Order: infra PR + OpenBao values, then #92. #96 any time.
refactor(quick-start): use the existing Agmarknet values for the crawler
refactor(quick-start): check the crawler every 6 hours
refactor(adapter-service): check the crawler every 6 hours, publish O…
chore(registry,registry-seed): drop the version bumps and changelog e…
docs(adapter-service): name #96 rather than registry chart versions
refactor(adapter-service): wire the crawler through existing values, …