Repository navigation
Crawl another network #76
Description
The problem. A farmer on our network only finds what our own providers publish. If a neighbouring network carries mandi prices we do not, that farmer cannot see them. The two networks may already have agreed to share, and it still makes no difference.
Admitting a network records the agreement. Nothing acts on it. Nothing goes and fetches what that network offers.
What needs to be true
- A farmer searching here finds catalogues from a network we admitted, next to our own.
- We take only the capabilities we chose from that network, not everything it offers.
- A copy is visible to us alone. It never travels on to a third network.
- A copy is served only for as long as the owning network allows.
- When that network stops sharing something, it stops appearing here.
- A farmer searching does not wait on another network. Whatever is shown is already here.
Doc: Share, Notify (projection change).
Parent: #48
Activity
ameersohel45 commentedon Oct 13, 2026
Outbound crawl is ready for review. Networks periodically fetch catalogs from admitted peers, allowing farmers to discover their published content. Verified live across two OAN stacks.
One thing is still in progress: when another network requests our catalogs, we don’t yet verify whether we’ve agreed to share them with that network.
PR: #83
ameersohel45 commentedon Oct 13, 2026
Now done too: when another network asks us for catalogs, the registry refuses its key unless we agreed to share with them, so an unagreed caller fails signature validation. Verified live.
feat(registry): read what we agreed with another network [#76]
feat(registry): telemetry on the peer lookups [#76]
feat(crawler): ask an admitted network what it will share [#76]
feat(crawler): keep a crawled catalog on its queue row [#76]
feat(crawler): queue what a peer returned [#76]
feat(crawler): run the peer crawl on a schedule [#76]
feat(crawler): report peer crawls as numbers, not just logs [#76]
fix(crawler): send a discover where the peer said to [#76]
87 remaining items
refactor(crawler): hold a crawled catalog as JSONB [#76]
refactor(crawler): hold a crawled catalog as JSONB [#76]
refactor(crawler): hold a crawled catalog as JSONB [#76]
feat(crawler): crawl an admitted network and mirror what it shares [#76]
feat(crawler): crawl an admitted network and mirror what it shares [#76]
feat(crawler): crawl an admitted network and mirror what it shares [#76]
feat(crawler): crawl an admitted network and mirror what it shares [#76]
feat(crawler): crawl an admitted network and mirror what it shares [#76]
feat(crawler): crawl an admitted network and mirror what it shares [#76]
feat(crawler): crawl an admitted network and mirror what it shares [#76]
[-]Crawl an admitted network and store what it shares[/-][+]Crawl another network[/+]ameersohel45 commentedon Oct 15, 2026
PR #83 covers this.
We ask every network we admitted for its catalogues, and publish them into our own discovery. A farmer here finds them next to ours.
We take only the packs we picked. Each copy is visible to us alone, so it never travels further. Each one expires after the time that network allows. Anything they stop sharing drops out on its own.
The crawler keeps nothing between passes, so it needs no database.
feat(crawler): crawl an admitted network and mirror what it shares [#76]
feat(crawler): crawl an admitted network and mirror what it shares [#76]
ameersohel45 commentedon Oct 19, 2026
Here is the PR , to crawl the admitted network : #83 .
Status - Inreview.