Repository navigation
feat(adapter-service,registry,registry-seed): enable the catalog crawler for publishing catalogues - #92
Open
kelvinprabhu wants to merge 9 commits into
Conversation
…ler for publishing catalogues - registry 0.5.0: ProviderSchema admits a `publish` action whose mappings is an https <...>/publish/<source>.yaml pipeline URL (no method or path). - registry-seed 0.2.0: bindings[].publishPipelineUrl writes the publish action, adding or repointing it on an existing binding; the render refuses a URL not pinned to a 40-hex commit SHA unless allowUnpinnedPipeline. - adapter-service 0.7.0: crawler.* values for the network-adapter role, in both deployment modes. Inserts the registry and crawler plugin blocks under plugins:, fills the run log DSN from crawler.dbDsnKey (network tier Secret in single-adapter mode), mounts an emptyDir for built catalogs, and defaults discoveryPushUrl to this pod's /publish (single) or the provider adapter's /publish (multi). Off by default; no render change when off. Ported from #69 onto the single-adapter charts. Refs #77
- subst() escapes \, & and | in the replacement, so a DSN or URL holding one is no longer corrupted (& expanded to the placeholder) or breaks the sed expression. Base64 keys are unaffected. - One crawler DSN subst line for both deployment modes. - crawler.publishCatalogOutputDir defaults to /tmp/catalogs in values; drop the crawlerOutputDir helper and its path check. - publishPipelines is always "true"; drop the value. - Quote crawler.pipelines keys instead of regex-validating them. Refs #77
The registry and crawler plugin blocks now live in config/single-adapter/network-adapter-config.yaml and config/multi-adapter/network-config.yaml, beside otelsetup, instead of being generated by template helpers. The crawler is on for the network-adapter role in both modes. - single: participantId is __PROVIDER_SUBSCRIBER_ID__ (provider tier Secret), push to localhost:9201/publish. - multi: participantId provider.oan.dev, push to the provider adapter's /publish. - receiverId discovery.oan.dev in both. The chart detects the crawler from the __ADAPTER_CRAWLER_DB_DSN__ placeholder: it fills the DSN from crawler.dbDsnKey and mounts the /tmp/catalogs emptyDir, and refuses a config whose publishCatalogOutputDir is not /tmp/catalogs. Drops crawlerBlock, withCrawler, crawlerPushUrl and every crawler.* value except dbDsnKey and outputSizeLimit. Refs #77
…d adapt config file retrieval
The registry schema and registry-seed changes now ship separately, on feat/77-catalog-crawler-registry-publish-action, so they can be deployed before the crawler. This PR keeps the adapter-service side. Refs #77
This was referenced Oct 27, 2026
Collaborator
Author
|
Dev rollout: OpenAgriNet/infra-automation#19 (PR OpenAgriNet/infra-automation#20). Merge this after that PR is merged and its secrets are in OpenBao — dev follows |
…nDemand only Matches the quick-start (#93). - network-adapter-config.yaml, network-config.yaml: catalogIntervalSeconds 21600; publish.mandi-price.mode onDemand on the daily schedule - README: the pipeline reads the provider's Agmarknet credentials (AGMARKNET_ACCESS_NAME, AGMARKNET_PASSWORD) by their own names (OpenAgriNet/network-adapter feat/mandi-publish-agmarknet-env-names) Refs #77
…no template change The run log DSN comes in through extraSubstitutions and /tmp/catalogs through extraVolumes, both already in the chart. templates/ and values.yaml are back as on development; README shows the values. Refs #77
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Refs #77 · Refs OpenAgriNet/network-adapter#54 · Supersedes #69 (Helm part) · Pairs with #96
Objective
Turn on the network adapter's catalog crawler through the Helm charts so catalogues can be published, in both deployment modes.
Solution
adapter-service0.7.0: the crawler is on for the network-adapter role, in both deployment modes.registryandcrawlerplugin blocks are in the config files, besideotelsetup:config/single-adapter/network-adapter-config.yamlandconfig/multi-adapter/network-config.yaml.__PROVIDER_SUBSCRIBER_ID__(provider tier Secret) tolocalhost:9201/publish. Multi: asprovider.oan.devto the provider adapter's/publish.receiverIddiscovery.oan.dev.__ADAPTER_CRAWLER_DB_DSN__through the existingextraSubstitutions(adapterKeysin single mode,keys.existingSecretin multi) and mounts/tmp/catalogsthroughextraVolumes/extraVolumeMounts; the README shows the values (set in OpenAgriNet/infra-automation#20 for dev).The registry schema and registry-seed
publishPipelineUrlmoved to #96. Without thatpublishaction seeded the crawler finds no pipeline and publishes nothing, so deploy #96 first.Ported from #69 onto the single-adapter charts (#87). Quick-start counterpart: #93.