feat(quick-start): enable the catalog crawler for publishing catalogues by kelvinprabhu · Pull Request #93 · OpenAgriNet/helmcharts

kelvinprabhu · 2026-10-26T15:17:50Z

Refs #77 · Refs OpenAgriNet/network-adapter#54

Objective

Run the network adapter's catalog crawler in docker compose so catalogues are published, in both deployment modes.

Solution

config/adapters/{single,multi}-adapter/network-adapter.yaml.tmpl: registry and crawler plugin blocks. Catalogs go to localhost:9201/publish (single) or provider-adapter:9200/publish (multi), as the provider identity, to the discovery subscriber id.
bin/setup.py: ensure_binding(publish_url) writes the publish action on the Agmarknet binding and adds or repoints it on an existing row. Default pipeline URL pinned to network-adapter 37f5a09; AGMARKNET_PUBLISH_PIPELINE_URL overrides.
config/registry/schemas/ProviderSchema.json: same as the registry chart in feat(registry,registry-seed): admit and seed the catalog crawler's publish action #96: publish joins the Action enum and MappingPath also accepts an https <...>/publish/<source>.yaml pipeline URL; every action still needs method, path, mappings. setup.py writes the publish action as POST /publish.
docker-compose.yml: new crawler-postgres (own user and volume, POSTGRES_DB=catalogcrawler, no published port) for the crawler's run log; network-adapter waits for it to be healthy. No dependency on discovery's database. Mandi pipeline inputs (MANDI_*, APP_NETWORK_ID) mapped from AGMARKNET_*.
README Appendix O and .env.example, including the steps for an existing stack and how to inspect the run log; make destroy lists the new volume.

- network-adapter.yaml.tmpl (single-adapter and multi-adapter): add the registry and crawler plugin blocks. Catalogs go to this process's /publish in single mode and to provider-adapter:9200/publish in multi mode, published as the provider identity to the discovery subscriber id. - setup.py: ensure_binding(publish_url) writes the publish action on the Agmarknet binding, adding or repointing it on an existing row; defaults to a commit-pinned pipeline URL, AGMARKNET_PUBLISH_PIPELINE_URL overrides. - ProviderSchema.json admits the publish action (same as registry chart 0.5.0). - docker-compose.yml: create the catalogcrawler run-log database on discovery-postgres, make network-adapter wait for it, and pass the Mandi pipeline inputs (MANDI_*, APP_NETWORK_ID) from the AGMARKNET_* values. - README Appendix O and .env.example document it, including the steps for an existing stack and the image requirement. Ported from #69 onto the single-adapter quick-start. Refs #77

- setup.py: put() beside post(), sharing one write(); the publish update is its own ensure_publish_action(), and a non-SUCCESSFUL answer now stops setup, as registry-seed does. - The pinned pipeline URL is the module constant MANDI_PUBLISH_PIPELINE_URL. - Crawler blocks in both network-adapter templates keep short comments and point to README Appendix O. Refs #77

…files Mirror the Helm charts: the crawler's participantId and receiverId are in the network-adapter templates rather than filled by setup.py. - single-adapter: participantId __PROVIDER_SUBSCRIBER_ID__ (the provider tier's identity, already rendered by setup.py). - multi-adapter: participantId provider.oan.dev. - receiverId discovery.oan.dev in both. - setup.py drops __PROVIDER_PUBLISH_ID__ and __DISCOVERY_PUBLISH_ID__. Refs #77

ProviderSchema.json is the chart copy from #96: publish joins the Action enum and MappingPath also accepts an https <...>/publish/<source>.yaml pipeline URL, while method, path and mappings stay required for every action. setup.py now writes the publish action as POST /publish, both on a new binding and when it adds or repoints the action on an existing one. Refs #77

The crawler's run log moves off discovery-postgres to a crawler-postgres container with its own user and volume. POSTGRES_DB creates catalogcrawler on every fresh volume, so an existing stack no longer needs a manual createdb, and the network adapter no longer holds discovery's credentials or waits on its database. discovery-postgres and the top-level configs block are back as on development. The crawler stays in the network adapter's config; the adapter now waits for crawler-postgres to be healthy. README Appendix O and the destroy listing cover the new container and how to inspect the run log. Refs #77

kelvinprabhu · 2026-10-28T04:29:16Z

+# The publish pipeline the network adapter's catalog crawler runs (README,
+# Appendix O). Optional: bin/setup.py defaults to a commit-pinned URL. Pin any
+# override to a commit SHA, with its mappings/ folder beside it.


simple comments

kelvinprabhu · 2026-10-28T04:30:31Z

+      POSTGRES_USER: crawler
+      POSTGRES_PASSWORD: crawler
+      POSTGRES_DB: catalogcrawler


is it safe to have it here ...

kelvinprabhu · 2026-10-28T04:31:23Z

+      MANDI_TOKEN_SECRET: ${AGMARKNET_PASSWORD}
+      # The catalog's provider id, and the network it is visible to.
+      MANDI_PARTICIPANT_ID: ${AGMARKNET_PARTICIPANT_ID}
+      APP_NETWORK_ID: ${BECKN_NETWORK_ID}


Why network id

kelvinprabhu · 2026-10-28T04:32:38Z

      AGMARKNET_ACCESS_NAME: ${AGMARKNET_ACCESS_NAME}
      AGMARKNET_PASSWORD: ${AGMARKNET_PASSWORD}
      MOCKAGMARKNET_TOKEN: ${MOCKAGMARKNET_TOKEN:-local-mandi-token}


these are same as MANDI_API_URI: ${AGMARKNET_BASE_URL}
MANDI_TOKEN_USER: ${AGMARKNET_ACCESS_NAME}
MANDI_TOKEN_SECRET ||. use them only no new addiction

kelvinprabhu · 2026-10-28T04:33:11Z

+
+## Appendix O — Scheduled catalog publishing (the crawler)
+
+The network adapter also runs the `catalogcrawler` plugin, in both modes. On
+every tick (`catalogIntervalSeconds`, 60 s here) it lists the registry's
+bindings, keeps the ones with a `publish` action, and runs each pipeline that
+is due: fetch the upstream, build the catalogs, POST each to `/publish`, which
+signs it as the provider and routes it to discovery.
+
+| Piece | Where |
+|---|---|
+| `registry` and `crawler` plugin blocks | `config/adapters/{single,multi}-adapter/network-adapter.yaml.tmpl` |
+| `publish` action on the Agmarknet binding | `bin/setup.py`, pinned to a commit; override with `AGMARKNET_PUBLISH_PIPELINE_URL` |
+| Schema that admits `publish` | `config/registry/schemas/ProviderSchema.json` |
+| Run log database `catalogcrawler` | `crawler-postgres`, its own container and volume; the network adapter waits for it |
+| Pipeline inputs `MANDI_*`, `APP_NETWORK_ID` | `network-adapter` environment, mapped from `AGMARKNET_*` |
+
+Catalogs go to `http://localhost:9201/publish` in single mode and to
+`http://provider-adapter:9200/publish` in multi mode.


posible keep it short and simple

…alog crawler's publish pipeline

Review on #93: no new environment for the crawler's publish pipeline, and no publish URL or changed code in setup.py. - network-adapter.yaml.tmpl (both modes): the pipeline's non-secret inputs (baseUrl, participantId, networkId) as crawler config overrides, rendered from AGMARKNET_BASE_URL, AGMARKNET_PARTICIPANT_ID and BECKN_NETWORK_ID; publish once a day, OnDemand catalogs only (mode: onDemand) - docker-compose.yml: drop MANDI_* and APP_NETWORK_ID from network-adapter; the pipeline reads AGMARKNET_ACCESS_NAME and AGMARKNET_PASSWORD, which it already gets (OpenAgriNet/network-adapter feat/mandi-publish-agmarknet-env-names) - setup.py: existing code back as on development; new put() and ensure_publish_action() add the publish action to the Agmarknet binding only when AGMARKNET_PUBLISH_PIPELINE_URL is set - bin/crawler-check.sh: shows the crawler ticking and its runs; --rerun clears the run log so the next tick runs the pipeline, --watch follows its log - .env.example: AGMARKNET_PUBLISH_PIPELINE_URL, pinned to a commit - README Appendix O: shorter Refs #77

kelvinprabhu · 2026-10-28T05:17:32Z

+      receiverId: "discovery.oan.dev"
+      publishPipelines: "true"
+      publishEnabled: "true"
+      catalogIntervalSeconds: "60"


make it 6hr crawl

kelvinprabhu · 2026-10-28T05:18:24Z

+      # networkId is required: the network the catalogs are visible to.
+      publish.mandi-price.baseUrl: "__AGMARKNET_BASE_URL__"
+      publish.mandi-price.participantId: "__AGMARKNET_PARTICIPANT_ID__"
+      publish.mandi-price.networkId: "__BECKN_NETWORK_ID__"


its optinal it there then only

Review on #93. - network-adapter.yaml.tmpl (both modes): catalogIntervalSeconds 21600 - the pipeline inputs are used only when AGMARKNET_PUBLISH_PIPELINE_URL is set; say so instead of calling networkId required - README Appendix O: the interval Refs #77

…nDemand only Matches the quick-start (#93). - network-adapter-config.yaml, network-config.yaml: catalogIntervalSeconds 21600; publish.mandi-price.mode onDemand on the daily schedule - README: the pipeline reads the provider's Agmarknet credentials (AGMARKNET_ACCESS_NAME, AGMARKNET_PASSWORD) by their own names (OpenAgriNet/network-adapter feat/mandi-publish-agmarknet-env-names) Refs #77

kelvinprabhu added 3 commits October 26, 2026 20:44

This was referenced Oct 27, 2026

feat(adapter-service,registry,registry-seed): enable the catalog crawler for publishing catalogues #92

Open

feat(registry,registry-seed): admit and seed the catalog crawler's publish action #96

Open

kelvinprabhu added 2 commits October 27, 2026 16:19

kelvinprabhu commented Oct 28, 2026

View reviewed changes

kelvinprabhu added 3 commits October 28, 2026 10:26

refactor(quick-start): update documentation and configuration for cat…

18e66a2

…alog crawler's publish pipeline

refactor(quick-start): remove obsolete crawler-check script

0ef03ea

kelvinprabhu commented Oct 28, 2026

View reviewed changes

kelvinprabhu requested a review from manjudr October 28, 2026 06:06

Base automatically changed from development to release-0.2.0 November 2, 2026 06:48

Navigation Menu

Repository navigation

feat(quick-start): enable the catalog crawler for publishing catalogues - #93

feat(quick-start): enable the catalog crawler for publishing catalogues#93
kelvinprabhu wants to merge 9 commits into
release-0.2.0from
feat/77-catalog-crawler-enabling-it-for-publishing-catalogues-follow-up-to-openagrinetnetwork-adapter54(quickstart)

kelvinprabhu commented Oct 26, 2026 •

edited

Loading

Uh oh!

kelvinprabhu Oct 28, 2026

Uh oh!

kelvinprabhu Oct 28, 2026

Uh oh!

kelvinprabhu Oct 28, 2026

Uh oh!

kelvinprabhu Oct 28, 2026

Uh oh!

kelvinprabhu Oct 28, 2026

Uh oh!

kelvinprabhu Oct 28, 2026

Uh oh!

kelvinprabhu Oct 28, 2026

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

kelvinprabhu commented Oct 26, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Objective

Solution

Uh oh!

kelvinprabhu Oct 28, 2026

Choose a reason for hiding this comment

Uh oh!

kelvinprabhu Oct 28, 2026

Choose a reason for hiding this comment

Uh oh!

kelvinprabhu Oct 28, 2026

Choose a reason for hiding this comment

Uh oh!

kelvinprabhu Oct 28, 2026

Choose a reason for hiding this comment

Uh oh!

kelvinprabhu Oct 28, 2026

Choose a reason for hiding this comment

Uh oh!

kelvinprabhu Oct 28, 2026

Choose a reason for hiding this comment

Uh oh!

kelvinprabhu Oct 28, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

kelvinprabhu commented Oct 26, 2026 •

edited

Loading