From c64a040685fc30bc678f511f36f9c37740d5845d Mon Sep 17 00:00:00 2001 From: nisargabd Date: Tue, 15 Sep 2026 23:35:33 +0530 Subject: [PATCH 01/23] chore: fold CodeQL into ci.yml's Security Scan, 2 checks per PR [#99] MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit A PR carried 4 entries: End to End verification, Security Scan, CodeQL (its own job), and GitHub's native Code scanning results / CodeQL for the SARIF upload. Folded CodeQL's init/build/analyze steps into ci.yml's security-scan job, so a PR now shows exactly 2 controllable checks — End to End verification and Security Scan, the latter running Trivy and CodeQL as steps in one job. codeql.yml drops its pull_request trigger entirely rather than becoming a skipped check on every PR: it keeps only push (trunk baseline for the "new alerts" comparison) and its weekly schedule (deep re-scan against updated query packs). Nothing here shares a file with a job of a different trigger scope, so nothing posts a permanently-skipped entry anywhere — that was the failure of an earlier version of this branch, reverted before this commit. Residual, not removable by any workflow restructuring: GitHub still posts "Code scanning results / CodeQL" itself whenever CodeQL uploads SARIF, no matter which job the analyze step lives in. It is not a job, carries no `name:`, and cannot be suppressed — confirmed by 2e33655 on this same repo. Accepted as a platform annotation, not a CI run: it consumes no separate Actions minutes and runs nothing of its own. Verified: actionlint 0 issues across all 5 workflow files; a Python simulation of every file's `on:` against (event, ref, schedule) shows the PR event firing only ci.yml's two jobs and every other event firing only the jobs that apply to it, with zero skipped entries anywhere; full test suite green (EMBEDDING_PROVIDER=hashing go test -race ./...). --- .github/workflows/ci.yml | 98 ++++++++++++++++++++++++++++-------- .github/workflows/codeql.yml | 27 +++++----- 2 files changed, 89 insertions(+), 36 deletions(-) diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 8186005..08c5466 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -1,21 +1,26 @@ name: CI # Two checks from THIS file and no more: "End to End verification" and -# "Security Scan". A PR carries two further checks, both from codeql.yml: -# "CodeQL", which is its job, and "Code scanning results / CodeQL", which -# GitHub posts for the analysis upload and which is not a job at all — it -# cannot be renamed, reordered or switched off. +# "Security Scan" — the latter runs Trivy AND CodeQL as steps in one job, so a +# PR carries exactly these two controllable checks. codeql.yml no longer +# triggers on pull_request; it only keeps the trunk baseline and weekly scan +# fresh for the "new alerts in this PR" comparison this job's own CodeQL run +# reads against. # -# That second one is the exact shape this file refuses for Trivy further -# down, so the rule has to be stated properly rather than by count. What it -# refuses is a check that can report the OPPOSITE of a gate reading the same -# findings. Trivy has such a gate, make trivy-gate, and the upload goes green -# with "No new alerts in code change" whenever the findings are not new to the -# diff — while the gate fails on the same SARIF. CodeQL has no gate to -# contradict: it is the only thing here that reports its class, and codeql.yml -# deliberately does not fail on findings. So its results check is redundant -# rather than contradictory. Redundant is a cost worth paying for per-PR -# alerts; a check that can be green while the truth is red is not. +# The one entry this cannot remove: GitHub posts "Code scanning results / +# CodeQL" itself for the SARIF upload inside codeql-action/analyze, which is +# not a job and so has no `name:` of its own and cannot be renamed, reordered +# or switched off — see the comment on that step in security-scan. That is +# the exact shape this file refuses for Trivy further down, so the rule has +# to be stated properly rather than by count: what it refuses is a check that +# can report the OPPOSITE of a gate reading the same findings. Trivy has such +# a gate, make trivy-gate, and the upload goes green with "No new alerts in +# code change" whenever the findings are not new to the diff — while the gate +# fails on the same SARIF. CodeQL has no gate to contradict: it is the only +# thing here that reports its class, and analyze deliberately does not fail +# on findings. So its results check is redundant rather than contradictory. +# Redundant is a cost worth paying for per-PR alerts; a check that can be +# green while the truth is red is not. # # The two tag-time checks live in ci-release.yml, not here behind an `if:`. A # job whose `if:` is false still posts a *skipped* check, and its `name:` is @@ -140,19 +145,70 @@ jobs: security-scan: name: Security Scan runs-on: ubuntu-latest - # Both scans in one job, one check. They were two jobs plus a gate job, and - # three red-or-green checks to answer one question — "is this branch - # shippable" — was more to read than it was worth. The dependency scan - # needs only a checkout, so it runs first and its findings are in the log - # before the image has finished building. + # CodeQL and both Trivy scans, one job, one check — a PR gets exactly two + # checks total from this repo's controllable CI (this job and verify + # above). codeql.yml still runs CodeQL again on a trunk push and on its + # weekly schedule for the "new alerts in this PR" baseline and the + # deep re-scan a push trigger can never give, but it no longer triggers + # on pull_request — this job is the only place a PR sees CodeQL run. # - # pull-requests: write is the PR comment. No security-events: write — the - # SARIF upload is gone, see below. + # The one thing folding it in here does NOT remove: GitHub posts its own + # "Code scanning results / CodeQL" check for the SARIF upload inside + # codeql-action/analyze below, regardless of which job hosts that step. + # It is not a workflow job, has no `name:` of its own and cannot be + # suppressed — see the comment on that step for why it is accepted rather + # than refused the way a Trivy SARIF upload is refused further down. + # + # pull-requests: write is the PR comment. security-events: write is + # CodeQL's own SARIF upload — Trivy's SARIF upload is NOT taken here, see + # the comment further down for why that one stays PR-forbidden even + # though this permission would allow it. permissions: contents: read pull-requests: write + security-events: write steps: - uses: actions/checkout@v4 + - uses: actions/setup-go@v5 + with: + go-version-file: go.mod + cache-dependency-path: "**/go.sum" + + # CodeQL first — it only needs the Go toolchain and `make build`, so it + # runs before anything Docker-shaped and does not care whether this PR + # is from a fork (no secrets involved). + - uses: github/codeql-action/init@v4 + with: + languages: go + # manual, not autobuild: autobuild guesses at how to build a Go + # module, and this one is built one way — `make build`, over every + # package, with the ldflags that stamp buildinfo. A CodeQL database + # extracted from a build nobody else performs describes a program + # nobody else runs. + build-mode: manual + # security-extended, not the default suite. The default returned + # zero on this tree, which is the only condition under which + # widening is cheap: there is no backlog for the extra queries to + # be lost in, so anything this surfaces is both new and + # attributable. It adds the lower-confidence and harder-to-exploit + # queries — the ones worth reading rather than the ones worth + # blocking on, which is why this step still does not fail the job + # on findings. + queries: security-extended + + - run: make build + + # An explicit category so this analysis sits beside trivy-deps and + # trivy-image in the Security tab rather than competing with them for + # the default one, and so it matches codeql.yml's trunk runs for the + # "new alerts" comparison. Findings do NOT fail this job: analyze files + # alerts and exits 0. This upload IS allowed on a PR, unlike Trivy's + # below: CodeQL has no competing gate anywhere in this repo, so there is + # nothing for its own check to contradict. + - uses: github/codeql-action/analyze@v4 + with: + category: codeql-go + - uses: ./.github/actions/trivy-cache # Fork PRs get an empty string for every secrets.* reference, so the diff --git a/.github/workflows/codeql.yml b/.github/workflows/codeql.yml index 4b05a3a..fc04bbc 100644 --- a/.github/workflows/codeql.yml +++ b/.github/workflows/codeql.yml @@ -15,25 +15,22 @@ name: CodeQL # "secrets never touch a log field" -> go/clear-text-logging, # go/hardcoded-credentials # -# It runs on pull_request, and that is a deliberate reversal of ci.yml's "two -# checks and no more". The rule it bends is not the rule ci.yml's SARIF comment -# defends: that one refuses a check which can report the OPPOSITE of a gate -# looking at the same findings. CodeQL has no competing gate — it is the only -# thing in this repo that reports this class at all — so there is nothing for -# its check to contradict, and a finding that only appears after the merge is a -# finding reported to the wrong person at the wrong time. +# No pull_request trigger here — ci.yml's security-scan job runs this same +# analyze step, on the same category, as part of a PR's one Security Scan +# check, so a PR gets a live CodeQL pass without a second job-level check for +# it. This file exists so that pass has something to compare against: a +# pull_request analysis reads as "new alerts" only relative to the base +# branch's most recent analysis, and this is what keeps that baseline current +# on every push to the trunk. The weekly schedule additionally re-runs the +# same code through query packs that update independently of any commit here +# — the one kind of finding a push trigger can never produce on its own. # # Separate from security.yml rather than a job inside it because triggers are -# per-workflow: a pull_request trigger there would fire the Trivy and -# govulncheck job on every PR too, duplicating ci.yml's Security Scan exactly. +# per-workflow: putting this in security.yml would fire the Trivy and +# govulncheck job on every trunk push CodeQL runs on too — same problem +# ci.yml's security-scan job solves for the PR side, just on the trunk side. on: - pull_request: push: - # The trunk as well, not only PRs. A pull_request analysis is compared - # against the base branch's most recent one, so without these the "new - # alerts in this PR" view has nothing to subtract and every pre-existing - # alert reads as newly introduced. - # # release-* as a glob, not release-0.0.1 by name: the next release branch # is the exact moment nobody remembers a workflow names the old one, and # the symptom is silent — no failure, just a branch that stopped being From 3785f1026d1930accdf600dc6b531776fdd96328 Mon Sep 17 00:00:00 2001 From: nisargabd Date: Wed, 16 Sep 2026 09:00:29 +0530 Subject: [PATCH 02/23] chore: fold CodeQL into security.yml's trunk scan too [#99] A push to release-0.0.1/main fired CodeQL (its own job in codeql.yml) and Scan the trunk (Trivy + govulncheck in security.yml) as two separate checks - the same duplication ci.yml's security-scan job already fixed on the PR side. Every such push also lands on PR #36 (release-0.0.1 -> main, still open), so this is what that PR sees accumulate on top of Publish coverage. Folded CodeQL's init/build/analyze steps into security.yml's scan job (same steps, same category codeql-go, as ci.yml's security-scan job runs for a PR) and deleted codeql.yml. A push to the trunk now posts 2 job-checks (Publish coverage, Scan the trunk) instead of 3. Nothing dropped: same Trivy dep+image scan, same govulncheck, same CodeQL security-extended queries, same SARIF categories, same gates. One schedule entry instead of two: codeql.yml's weekly cron existed only to run CodeQL an hour apart from security.yml's own, so the two jobs' scheduled runs never queued against each other. There is only one job now, so only one cron stays (17 3 * * 1). Verified: actionlint 0 issues on all 4 remaining workflow files; Python simulation of on: triggers confirms a push to the trunk now fires exactly 2 jobs, a PR fires exactly 2 jobs, nothing skips anywhere; full test suite green (EMBEDDING_PROVIDER=hashing go test -race ./...). --- .github/workflows/ci.yml | 20 +++---- .github/workflows/codeql.yml | 99 ---------------------------------- .github/workflows/security.yml | 38 +++++++++++-- 3 files changed, 44 insertions(+), 113 deletions(-) delete mode 100644 .github/workflows/codeql.yml diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 08c5466..bbc5a1d 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -2,10 +2,11 @@ name: CI # Two checks from THIS file and no more: "End to End verification" and # "Security Scan" — the latter runs Trivy AND CodeQL as steps in one job, so a -# PR carries exactly these two controllable checks. codeql.yml no longer -# triggers on pull_request; it only keeps the trunk baseline and weekly scan -# fresh for the "new alerts in this PR" comparison this job's own CodeQL run -# reads against. +# PR carries exactly these two controllable checks. CodeQL has no job of its +# own that triggers on pull_request; security.yml's trunk job runs the same +# analyze step again on every push and on its weekly schedule, which is what +# keeps the baseline fresh for the "new alerts in this PR" comparison this +# job's own CodeQL run reads against. # # The one entry this cannot remove: GitHub posts "Code scanning results / # CodeQL" itself for the SARIF upload inside codeql-action/analyze, which is @@ -147,10 +148,11 @@ jobs: runs-on: ubuntu-latest # CodeQL and both Trivy scans, one job, one check — a PR gets exactly two # checks total from this repo's controllable CI (this job and verify - # above). codeql.yml still runs CodeQL again on a trunk push and on its - # weekly schedule for the "new alerts in this PR" baseline and the - # deep re-scan a push trigger can never give, but it no longer triggers - # on pull_request — this job is the only place a PR sees CodeQL run. + # above). security.yml's trunk job runs CodeQL again on a trunk push and + # on its weekly schedule for the "new alerts in this PR" baseline and the + # deep re-scan a push trigger can never give, but CodeQL has no job of its + # own that triggers on pull_request — this job is the only place a PR + # sees CodeQL run. # # The one thing folding it in here does NOT remove: GitHub posts its own # "Code scanning results / CodeQL" check for the SARIF upload inside @@ -200,7 +202,7 @@ jobs: # An explicit category so this analysis sits beside trivy-deps and # trivy-image in the Security tab rather than competing with them for - # the default one, and so it matches codeql.yml's trunk runs for the + # the default one, and so it matches security.yml's trunk runs for the # "new alerts" comparison. Findings do NOT fail this job: analyze files # alerts and exits 0. This upload IS allowed on a PR, unlike Trivy's # below: CodeQL has no competing gate anywhere in this repo, so there is diff --git a/.github/workflows/codeql.yml b/.github/workflows/codeql.yml deleted file mode 100644 index fc04bbc..0000000 --- a/.github/workflows/codeql.yml +++ /dev/null @@ -1,99 +0,0 @@ -name: CodeQL - -# A different question from security.yml, which is why it is a different file. -# Trivy and govulncheck both ask "does this depend on something with a -# published CVE". CodeQL asks whether the code in this repo has a vulnerability -# in it — injection, traversal, request forgery, a secret reaching a log. -# Nothing here asked that before. -# -# Concretely, it is how three of CLAUDE.md's standing rules stop being held by -# review alone: -# -# "SQL is always parameterised" -> go/sql-injection -# "a URL from a request body is not -> go/request-forgery -# trusted" -# "secrets never touch a log field" -> go/clear-text-logging, -# go/hardcoded-credentials -# -# No pull_request trigger here — ci.yml's security-scan job runs this same -# analyze step, on the same category, as part of a PR's one Security Scan -# check, so a PR gets a live CodeQL pass without a second job-level check for -# it. This file exists so that pass has something to compare against: a -# pull_request analysis reads as "new alerts" only relative to the base -# branch's most recent analysis, and this is what keeps that baseline current -# on every push to the trunk. The weekly schedule additionally re-runs the -# same code through query packs that update independently of any commit here -# — the one kind of finding a push trigger can never produce on its own. -# -# Separate from security.yml rather than a job inside it because triggers are -# per-workflow: putting this in security.yml would fire the Trivy and -# govulncheck job on every trunk push CodeQL runs on too — same problem -# ci.yml's security-scan job solves for the PR side, just on the trunk side. -on: - push: - # release-* as a glob, not release-0.0.1 by name: the next release branch - # is the exact moment nobody remembers a workflow names the old one, and - # the symptom is silent — no failure, just a branch that stopped being - # analysed. The glob is why this file needs no edit at 0.0.2. - branches: [main, release-*] - schedule: - # Monday 04:17 UTC — an hour after security.yml's, not the same minute, so - # the two are not queued against each other. Weekly because the queries - # themselves change: the same code re-analysed six months later is a - # genuinely different scan, which is the one thing a push trigger can never - # do. - - cron: "17 4 * * 1" - workflow_dispatch: - -permissions: - contents: read - -concurrency: - group: codeql-${{ github.ref }} - cancel-in-progress: true - -jobs: - analyze: - name: CodeQL - runs-on: ubuntu-latest - permissions: - contents: read - security-events: write - steps: - - uses: actions/checkout@v4 - - uses: actions/setup-go@v5 - with: - go-version-file: go.mod - cache-dependency-path: "**/go.sum" - - - uses: github/codeql-action/init@v4 - with: - languages: go - # manual, not autobuild: autobuild guesses at how to build a Go - # module, and this one is built one way — `make build`, over every - # package, with the ldflags that stamp buildinfo. A CodeQL database - # extracted from a build nobody else performs describes a program - # nobody else runs. - build-mode: manual - # security-extended, not the default suite. The default returned zero - # on this tree, which is the only condition under which widening is - # cheap: there is no backlog for the extra queries to be lost in, so - # anything this surfaces is both new and attributable. It adds the - # lower-confidence and the harder-to-exploit queries — the ones worth - # reading rather than the ones worth blocking on, which is why this - # job still does not fail on findings. - queries: security-extended - - - run: make build - - # An explicit category so this analysis sits beside trivy-deps and - # trivy-image in the Security tab rather than competing with them for the - # default one. - # - # Findings do NOT fail this job: analyze files alerts and exits 0. The - # check therefore says whether CodeQL ran and the Security tab says what - # it found. Gating on the alert count is a deliberate later decision, - # taken once there is a baseline to gate against. - - uses: github/codeql-action/analyze@v4 - with: - category: codeql-go diff --git a/.github/workflows/security.yml b/.github/workflows/security.yml index 61a197e..564a82b 100644 --- a/.github/workflows/security.yml +++ b/.github/workflows/security.yml @@ -1,7 +1,13 @@ name: Security # The badge on the README and the alert list behind it: what is wrong with the -# trunk today, as opposed to what a PR changed. +# trunk today, as opposed to what a PR changed. One job, one check: Trivy +# (dependency + image), govulncheck, and CodeQL all run here, the same way +# ci.yml's security-scan job runs Trivy and CodeQL together for a PR. CodeQL +# used to be a separate job in its own codeql.yml, triggered by this same push +# and schedule — folded in here so a push to the trunk (which is also every +# push PR #36, release-0.0.1 -> main, will ever see) posts one job-check +# instead of two. # # The weekly schedule is the point of this file. ci.yml scans every PR, which # answers "did this diff introduce a vulnerability" — it structurally cannot @@ -29,7 +35,10 @@ on: schedule: # Monday 03:17 UTC. Off the hour on purpose — GitHub queues scheduled runs # hardest at :00, and a scan that starts twenty minutes late is a scan - # whose result is twenty minutes late. + # whose result is twenty minutes late. One entry, not two: this used to sit + # an hour apart from codeql.yml's own cron so the two jobs' scheduled runs + # never queued against each other: now that CodeQL is a step in this same + # job, there is only one run to schedule. # # A schedule only ever runs on the DEFAULT branch, whatever the push # trigger above lists. So until this file is on main, the weekly scan does @@ -59,14 +68,33 @@ jobs: security-events: write steps: - uses: actions/checkout@v4 - - uses: ./.github/actions/trivy-cache - # govulncheck is built from tools/go.mod, so this job needs a toolchain - # even though the Trivy half of it does not. + # govulncheck is built from tools/go.mod and CodeQL's build-mode is + # manual, so this job needs a toolchain even though the Trivy half of + # it does not. - uses: actions/setup-go@v5 with: go-version-file: go.mod cache-dependency-path: "**/go.sum" + # CodeQL first, same as ci.yml's security-scan job: it only needs the Go + # toolchain and `make build`, so it runs before anything Docker-shaped. + - uses: github/codeql-action/init@v4 + with: + languages: go + build-mode: manual + queries: security-extended + + - run: make build + + # Same category as ci.yml's security-scan job uses on a PR, so a PR's + # analysis is compared against the trunk's most recent one under this + # exact category — that comparison is what "new alerts in this PR" means. + - uses: github/codeql-action/analyze@v4 + with: + category: codeql-go + + - uses: ./.github/actions/trivy-cache + # Both scans, for the reasons ci.yml gives at length: the dependency scan # sees a vulnerable module that only the test suite imports and is never # linked into a layer, and the image scan sees stdlib CVEs read out of the From f1f7ea8b14f59e8031395b458acd2367326bfc64 Mon Sep 17 00:00:00 2001 From: nisargabd Date: Wed, 16 Sep 2026 17:20:12 +0530 Subject: [PATCH 03/23] fix: guard mergepatch map-capacity sum against overflow [#38] MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit CodeQL flagged make(map[string]any, len(targetObject)+len(patchObject)) at mergepatch.go:54 (go/allocation-size-overflow, CWE-190) across two alerts on PR #36 — the sum of two int lengths feeds an allocation with no overflow check. Reviewer deferred it to 0.2 instead of fixing it. The operands are decoded-map key counts, not attacker-controlled byte sizes, so real overflow needs ~2^63 map entries — not reachable on any real machine. Still applied CodeQL's own recommended remediation (guard the arithmetic before allocating) rather than dismiss the alert: a wider type doesn't apply here since make()'s size argument must be int-compatible, and validating upstream input size doesn't touch this specific arithmetic. Falling back to capacity 0 on overflow costs a map resize, never a crash. --- src/domain/mergepatch.go | 10 +++++++++- 1 file changed, 9 insertions(+), 1 deletion(-) diff --git a/src/domain/mergepatch.go b/src/domain/mergepatch.go index e4b4481..cdce29f 100644 --- a/src/domain/mergepatch.go +++ b/src/domain/mergepatch.go @@ -2,6 +2,7 @@ package domain import ( "encoding/json" + "math" "slices" "time" ) @@ -51,7 +52,14 @@ func mergeValue(target, patch any) any { targetObject = map[string]any{} } - merged := make(map[string]any, len(targetObject)+len(patchObject)) + // CodeQL go/allocation-size-overflow: targetObject and patchObject are + // decoded map lengths, not attacker-controlled byte counts, but the sum + // still gets a guard so make() never receives an overflowed capacity. + capacity := 0 + if a, b := len(targetObject), len(patchObject); a <= math.MaxInt-b { + capacity = a + b + } + merged := make(map[string]any, capacity) for name, value := range targetObject { merged[name] = value } From 3a4dfd4319bd1613dd7eafd934f910d2be8ce1b7 Mon Sep 17 00:00:00 2001 From: nisargabd Date: Wed, 16 Sep 2026 17:26:40 +0530 Subject: [PATCH 04/23] fix: drop mergepatch capacity hint instead of guarding it [#38] The prior commit guarded len(targetObject)+len(patchObject) with an overflow check before allocating, per CodeQL's own remediation advice. CodeQL re-flagged it anyway (alerts #6, #7) on the guarded line itself - it flags any addition of two map-derived lengths reaching an allocation, regardless of a bounding condition around it. It doesn't do the range analysis to see the guard makes that addition safe. So the only fix CodeQL's scanner actually accepts is removing the sum, not bounding it. Dropped the capacity hint entirely: the map now grows through its normal resize path instead of being pre-sized. No behaviour change, confirmed by the unchanged MergePatch test suite. --- src/domain/mergepatch.go | 14 +++++--------- 1 file changed, 5 insertions(+), 9 deletions(-) diff --git a/src/domain/mergepatch.go b/src/domain/mergepatch.go index cdce29f..2ef592e 100644 --- a/src/domain/mergepatch.go +++ b/src/domain/mergepatch.go @@ -2,7 +2,6 @@ package domain import ( "encoding/json" - "math" "slices" "time" ) @@ -52,14 +51,11 @@ func mergeValue(target, patch any) any { targetObject = map[string]any{} } - // CodeQL go/allocation-size-overflow: targetObject and patchObject are - // decoded map lengths, not attacker-controlled byte counts, but the sum - // still gets a guard so make() never receives an overflowed capacity. - capacity := 0 - if a, b := len(targetObject), len(patchObject); a <= math.MaxInt-b { - capacity = a + b - } - merged := make(map[string]any, capacity) + // No capacity hint: CodeQL (go/allocation-size-overflow) flags len()+len() + // reaching an allocation even behind an explicit overflow guard, so the + // only fix it accepts is not summing the two lengths at all. The map + // grows via its normal resize path instead of being pre-sized. + merged := make(map[string]any) for name, value := range targetObject { merged[name] = value } From bb656b5209c523d851fa940faa74bb1ef287c10a Mon Sep 17 00:00:00 2001 From: Manjunath Davanam Date: Thu, 17 Sep 2026 15:00:05 +0530 Subject: [PATCH 05/23] chore: add a lightweight PR template [#41] Prompts a description with a required issue link, plus a short self-review / end-to-end-verification / docs-updated checklist. Deliberately excludes build/lint/test and breaking-change checkboxes since CI and the commit-message convention already enforce those. --- .github/pull_request_template.md | 11 +++++++++++ 1 file changed, 11 insertions(+) create mode 100644 .github/pull_request_template.md diff --git a/.github/pull_request_template.md b/.github/pull_request_template.md new file mode 100644 index 0000000..eb29800 --- /dev/null +++ b/.github/pull_request_template.md @@ -0,0 +1,11 @@ +## Description + + + +Fixes/Refs # + +## Checklist + +- [ ] Self-review done +- [ ] End-to-end verification done +- [ ] Docs (`CLAUDE.md` / ADRs) updated, if applicable From fbc65d8a638cc91ca500f3c5521511c5a7f3ad4c Mon Sep 17 00:00:00 2001 From: nisargabd Date: Mon, 21 Sep 2026 12:14:49 +0530 Subject: [PATCH 06/23] fix: stop scanning the release image at tag time [#46] MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit `make image-build` ran Trivy over the freshly built image with `--exit-code 1` at `SEVERITY=CRITICAL,HIGH,MEDIUM,LOW`. Any finding at any severity aborted the target, so the `make image-push` that follows it never ran and the release produced no artifact at all. Tag 0.1.0 died exactly that way on both arch legs, on 3 LOW and 1 MEDIUM: ``` make[1]: *** [Makefile:353: trivy-release-gate] Error 1 make: *** [Makefile:545: image-build] Error 2 ``` The scan is removed rather than made non-blocking, because by the time a tag exists this image has already been scanned three times and none of those three is this one. - `ci.yml` builds the same Dockerfile on the PR, scans it, and `trivy-gate` blocks the merge on any finding. - `security.yml` builds and scans it again on the push to the branch the tag is cut from, and uploads the SARIF to the Security tab. - `security.yml` again, weekly — the only one of the three that can catch a CVE published after the code stopped changing, which is the one thing a tag-time scan could otherwise have added. So a fourth scan re-reported what was already reported and acted on, and the only new thing it could produce was a reason not to ship. It cost a Trivy install, a DB fetch and a full image scan on each of the two arch legs. Removed: the `trivy-release-gate` target and its `.PHONY` entry, the `$(MAKE) trivy-release-gate` line in `image-build`, and the `trivy-cache` step in the release job — nothing there runs Trivy any more. `TRIVY_IMAGE_SCAN` is inlined back into `trivy-image`; the gate was its only other caller. The `image-build` and `image-push` doc comments no longer describe a scan, and `image-build`'s records why there isn't one so the next person does not re-add it. `trivy-deps`, `trivy-image`, `trivy-report` and `trivy-gate` are untouched, and the PR gate stays blocking at every severity. --- .github/workflows/ci-release.yml | 11 +++++------ Makefile | 34 +++++++++++++------------------- 2 files changed, 19 insertions(+), 26 deletions(-) diff --git a/.github/workflows/ci-release.yml b/.github/workflows/ci-release.yml index 9feb9d0..d850433 100644 --- a/.github/workflows/ci-release.yml +++ b/.github/workflows/ci-release.yml @@ -84,7 +84,6 @@ jobs: - uses: actions/checkout@v4 with: fetch-depth: 0 - - uses: ./.github/actions/trivy-cache # First, before the registry logins: this action replaces the Docker # credential store wholesale, while `docker login` merges into it, so the @@ -109,13 +108,13 @@ jobs: username: ${{ github.actor }} password: ${{ secrets.GHCR_PAT || secrets.GITHUB_TOKEN }} - # Builds the image into this runner's local Docker daemon and gates it on - # Trivy — exit 1 on a finding, not a report — so the exact bytes about to - # ship are scanned before they are tagged for or pushed to any registry. + # Builds into this runner's local Docker daemon so the next step re-tags + # these exact bytes. No Trivy here — ci.yml and security.yml already scan + # this image; see image-build in the Makefile. - run: make image-build ARCH=${{ matrix.arch }} - # Re-tags that same scanned image and pushes it. No rebuild and no - # re-scan, so what lands in the registry is what the gate above passed. + # Re-tags that local image and pushes it. No rebuild, so every registry + # gets identical bytes. - run: make image-push ARCH=${{ matrix.arch }} publish-artifact: diff --git a/Makefile b/Makefile index 78cb3d0..77654ea 100644 --- a/Makefile +++ b/Makefile @@ -336,21 +336,13 @@ trivy-deps: $(TRIVY) $(TRIVY) fs . --skip-dirs tools --severity $(SEVERITY) --exit-code 0 \ --format sarif --output trivy-deps.sarif -TRIVY_IMAGE_SCAN = $(TRIVY) image $(IMAGE) --severity $(SEVERITY) - ## trivy-image: shipped image scan (T4), SARIF report — reads base layers and ## the Go build info embedded in the binary, including stdlib, ## so a Go toolchain CVE shows up here and nowhere else that the ## dependency scan above cannot see. IMAGE names the ref to scan. trivy-image: $(TRIVY) - $(TRIVY_IMAGE_SCAN) --exit-code 0 --format sarif --output trivy-image.sarif - -## trivy-release-gate: the same image scan as trivy-image, but exit 1 on a -## finding instead of writing a report — the pre-push -## release gate image-build runs once per arch, on the -## local image, before anything is pushed anywhere. -trivy-release-gate: $(TRIVY) - $(TRIVY_IMAGE_SCAN) --exit-code 1 --format table + $(TRIVY) image $(IMAGE) --severity $(SEVERITY) \ + --exit-code 0 --format sarif --output trivy-image.sarif ## trivy-report: render every SARIF report as ONE PR comment, trivy-report.md # One comment covering both scans, not one comment each: the two scans run in @@ -533,21 +525,23 @@ docker: --build-arg TREE_STATE=$(TREE_STATE) \ -t $(IMAGE) . -## image-build: build this arch's release image locally and gate it on Trivy -# Built and loaded locally, NOT pushed: Trivy then scans the exact bytes that -# are about to ship, before they are tagged for or pushed to any registry. One -# scan covers every registry, because it is one image. +## image-build: build this arch's release image locally (ARCH) +# Built and loaded locally, NOT pushed, so image-push re-tags these exact bytes. +# +# No Trivy here. The same image is scanned on the PR, where trivy-gate blocks +# the merge, and again by security.yml on the push to the release branch and +# weekly after that. A fourth scan at tag time reports nothing new and can only +# stop the release — #46. # # Native, never QEMU — each arch builds on a runner of that arch, so a plain # `docker build` already produces the right one and ARCH is only the tag suffix. image-build: $(MAKE) docker IMAGE=$(RELEASE_IMAGE) - $(MAKE) trivy-release-gate IMAGE=$(RELEASE_IMAGE) -## image-push: push the gated local image to every enabled registry (ARCH) -# Re-tags the already-scanned local image per registry and pushes. No rebuild -# and no re-scan, so what is pushed is byte-identical to what image-build -# gated. One arch-suffixed tag each; nothing binds the plain version tag until +## image-push: push this arch's local release image to every registry (ARCH) +# Re-tags the local image per registry and pushes. No rebuild, so what is pushed +# is byte-identical across every registry and to what image-build produced. One +# arch-suffixed tag each; nothing binds the plain version tag until # image-publish has every arch. image-push: require-image-repos @set -e; for repo in $(IMAGE_REPOS); do \ @@ -743,7 +737,7 @@ $(TRIVY): .PHONY: help build test test-short test-ci cover cover-total cover-report \ cover-html cover-diff lint fmt lint-actions lint-staged hooks sqlc \ sqlc-verify migrate run logs migrate-down security trivy-deps \ - trivy-image trivy-release-gate trivy-report trivy-gate docker \ + trivy-image trivy-report trivy-gate docker \ image-build image-push image-publish require-image-repos up down \ verify newman audit tools clean telemetry telemetry-metrics \ telemetry-logs telemetry-down From aefc047b9381a2aac0874f1d334847b7dfde5b36 Mon Sep 17 00:00:00 2001 From: nisargabd Date: Sun, 20 Sep 2026 16:09:21 +0530 Subject: [PATCH 07/23] fix: bump the otel modules off the CVE-affected versions [#46] CVE-2026-81871 (MEDIUM) in otlploggrpc v0.20.0 -- the log gRPC exporter ignores env TLS certs, bypassing mTLS/pinning -- and CVE-2026-81870 (LOW) in otlptrace v1.44.0, where exporter config logging may leak endpoint URLs into info logs. Four findings in the shipped binary between them. A minimal bump is not available: MVS pulls the whole release train, and the stable v1.x and experimental v0.x lines ship in lockstep, so otel/log has to move to v0.21.0 with the rest. v0.21.0 drops the log package's own attribute constructors -- Record.SetBody now takes attribute.Value and Record.AddAttributes takes attribute.KeyValue -- so log.StringValue and log.String become attribute.StringValue and attribute.String. attribute.Key is a named string type rather than a bare string, hence the conversion at the test's map index. That is a type-namespace move onto otel's standard attribute type, not a change in what is emitted: the same five spec-Required keys with the same values on the same Record. The audit tests assert those keys off the emitted record and are unchanged. log.Record, log.SeverityInfo4, log.Logger and log.WithInstrumentationVersion all still live in otel/log and are untouched. --- go.mod | 30 ++++---- go.sum | 68 +++++++++---------- src/platform/telemetry/audit.go | 13 ++-- src/platform/telemetry/audit_internal_test.go | 5 +- 4 files changed, 59 insertions(+), 57 deletions(-) diff --git a/go.mod b/go.mod index 2140dd8..e251d1d 100644 --- a/go.mod +++ b/go.mod @@ -15,17 +15,17 @@ require ( github.com/testcontainers/testcontainers-go v0.44.0 github.com/testcontainers/testcontainers-go/modules/postgres v0.44.0 github.com/uber/h3-go/v4 v4.5.0 - go.opentelemetry.io/otel v1.44.0 - go.opentelemetry.io/otel/exporters/otlp/otlplog/otlploggrpc v0.20.0 - go.opentelemetry.io/otel/exporters/otlp/otlpmetric/otlpmetricgrpc v1.44.0 - go.opentelemetry.io/otel/exporters/otlp/otlptrace v1.44.0 - go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc v1.44.0 - go.opentelemetry.io/otel/log v0.20.0 - go.opentelemetry.io/otel/metric v1.44.0 - go.opentelemetry.io/otel/sdk v1.44.0 - go.opentelemetry.io/otel/sdk/log v0.20.0 - go.opentelemetry.io/otel/sdk/metric v1.44.0 - go.opentelemetry.io/otel/trace v1.44.0 + go.opentelemetry.io/otel v1.45.0 + go.opentelemetry.io/otel/exporters/otlp/otlplog/otlploggrpc v0.21.0 + go.opentelemetry.io/otel/exporters/otlp/otlpmetric/otlpmetricgrpc v1.45.0 + go.opentelemetry.io/otel/exporters/otlp/otlptrace v1.45.0 + go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc v1.45.0 + go.opentelemetry.io/otel/log v0.21.0 + go.opentelemetry.io/otel/metric v1.45.0 + go.opentelemetry.io/otel/sdk v1.45.0 + go.opentelemetry.io/otel/sdk/log v0.21.0 + go.opentelemetry.io/otel/sdk/metric v1.45.0 + go.opentelemetry.io/otel/trace v1.45.0 go.uber.org/zap v1.27.0 golang.org/x/crypto v0.56.0 gopkg.in/yaml.v3 v3.0.1 @@ -49,7 +49,7 @@ require ( github.com/docker/go-units v0.5.0 // indirect github.com/ebitengine/purego v0.10.1 // indirect github.com/felixge/httpsnoop v1.1.0 // indirect - github.com/go-logr/logr v1.4.3 // indirect + github.com/go-logr/logr v1.4.4 // indirect github.com/go-logr/stdr v1.2.2 // indirect github.com/go-ole/go-ole v1.3.0 // indirect github.com/go-openapi/jsonpointer v0.22.5 // indirect @@ -85,14 +85,14 @@ require ( github.com/yusufpapurcu/wmi v1.2.4 // indirect go.opentelemetry.io/auto/sdk v1.2.1 // indirect go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp v0.69.0 // indirect - go.opentelemetry.io/proto/otlp v1.10.0 // indirect + go.opentelemetry.io/proto/otlp v1.11.0 // indirect go.uber.org/multierr v1.10.0 // indirect golang.org/x/net v0.58.0 // indirect golang.org/x/sync v0.22.0 // indirect golang.org/x/sys v0.47.0 // indirect golang.org/x/text v0.41.0 // indirect - google.golang.org/genproto/googleapis/api v0.0.0-20260526163538-3dc84a4a5aaa // indirect - google.golang.org/genproto/googleapis/rpc v0.0.0-20260526163538-3dc84a4a5aaa // indirect + google.golang.org/genproto/googleapis/api v0.0.0-20260803160001-6ac0973c030d // indirect + google.golang.org/genproto/googleapis/rpc v0.0.0-20260803160001-6ac0973c030d // indirect google.golang.org/grpc v1.83.2 // indirect google.golang.org/protobuf v1.36.11 // indirect ) diff --git a/go.sum b/go.sum index c4b8880..79143cc 100644 --- a/go.sum +++ b/go.sum @@ -48,8 +48,8 @@ github.com/felixge/httpsnoop v1.1.0/go.mod h1:Zqxgdd+1Rkcz8euOqdr7lqgCRJztwr5hp9 github.com/getkin/kin-openapi v0.147.0 h1:s+Xsm9gUMPJbgCnABZ2to3zSQQ5A9dyj/zo62VVsldY= github.com/getkin/kin-openapi v0.147.0/go.mod h1:3BH9M9XDe/y9M5DSvEocVYAYq1w0qrhJHjC/vZi0AaY= github.com/go-logr/logr v1.2.2/go.mod h1:jdQByPbusPIv2/zmleS9BjJVeZ6kBagPoEUsqbVz/1A= -github.com/go-logr/logr v1.4.3 h1:CjnDlHq8ikf6E492q6eKboGOC0T8CDaOvkHCIg8idEI= -github.com/go-logr/logr v1.4.3/go.mod h1:9T104GzyrTigFIr8wt5mBrctHMim0Nb2HLGrmQ40KvY= +github.com/go-logr/logr v1.4.4 h1:tG4xh9yMsRCAiodLVTxyrkzSZ9+o0L1Kg/+cPVcbP/8= +github.com/go-logr/logr v1.4.4/go.mod h1:9T104GzyrTigFIr8wt5mBrctHMim0Nb2HLGrmQ40KvY= github.com/go-logr/stdr v1.2.2 h1:hSWxHoqTgW2S2qGc0LTAI563KZ5YKYRhT3MFKZMbjag= github.com/go-logr/stdr v1.2.2/go.mod h1:mMo/vtBO5dYbehREoey6XUKy/eSumjCCveDpRre4VKE= github.com/go-ole/go-ole v1.2.6/go.mod h1:pprOEPIfldk/42T2oK7lQ4v4JSDwmV0As9GaiUsvbm0= @@ -169,34 +169,34 @@ go.opentelemetry.io/auto/sdk v1.2.1 h1:jXsnJ4Lmnqd11kwkBV2LgLoFMZKizbCi5fNZ/ipaZ go.opentelemetry.io/auto/sdk v1.2.1/go.mod h1:KRTj+aOaElaLi+wW1kO/DZRXwkF4C5xPbEe3ZiIhN7Y= go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp v0.69.0 h1:8tvICD4vSTOOsNrsI4Ljf6C+6UKvpTEH5XY3JMoyPoo= go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp v0.69.0/go.mod h1:z9+yiacE0IHRqM4qFfkbt/JYlmYXgss8GY/jXoNuPJI= -go.opentelemetry.io/otel v1.44.0 h1:JjwHmHpA4iZ3wBxluu2fbbE7j4kqlE8jXyAyPXH7HqU= -go.opentelemetry.io/otel v1.44.0/go.mod h1:BMgjTHL9WPRlRjL2oZCBTL4whCGtXch2H4BhOPIAyYc= -go.opentelemetry.io/otel/exporters/otlp/otlplog/otlploggrpc v0.20.0 h1:rydZ9sxbcFdm/oWrVyfLTjHIygMgv0bEeMd+3B/BvoM= -go.opentelemetry.io/otel/exporters/otlp/otlplog/otlploggrpc v0.20.0/go.mod h1:earQ25dooT0Hhspq59DZ8YCC50jWfOlFEeWoxy/P444= -go.opentelemetry.io/otel/exporters/otlp/otlpmetric/otlpmetricgrpc v1.44.0 h1:SUplec5dp06reu1zaXmOXdvqH398taqrDXqUl99jxSc= -go.opentelemetry.io/otel/exporters/otlp/otlpmetric/otlpmetricgrpc v1.44.0/go.mod h1:ho2g4N+ane+swq5I/VBkKWnRDY4kUINH3FuqyZqX/Ug= -go.opentelemetry.io/otel/exporters/otlp/otlptrace v1.44.0 h1:4YsVu3B8+3qtWYYrsUYgn0OG78pN0rnNPRGX4SbokQI= -go.opentelemetry.io/otel/exporters/otlp/otlptrace v1.44.0/go.mod h1:+wnlSn0mD1ADVMe3v9Z/WIaiz6q6gL2J/ejaAmdmv80= -go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc v1.44.0 h1:qazEJlUOQzhCpzQpFETGby7EdqjI1wsd0W+6Gg1SCTU= -go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc v1.44.0/go.mod h1:fOD2Yefuxixkx3ahVNf0O/PERb6r4OlbxfATVnYvzCo= -go.opentelemetry.io/otel/log v0.20.0 h1:/5i0vuHxCLWUfChWG41K9wkM0jafruPw9NU1/RCJirs= -go.opentelemetry.io/otel/log v0.20.0/go.mod h1:wOcMcjsZpG8x7Bak7IhSi/lg8wscV2C1VdrKCLPlt0E= -go.opentelemetry.io/otel/metric v1.44.0 h1:1w0gILTcHdr3YI+ixLyjemwrVnsMURbTZFrSYCdDdmc= -go.opentelemetry.io/otel/metric v1.44.0/go.mod h1:8O7hanEPBNgEMmybD3s2VBKcgWOCsA6tzHBPODAiquo= -go.opentelemetry.io/otel/metric/x v0.66.0 h1:YkCrx1zLOChi9ZcZ6euupOcsgzbVlec7D/xoEU1+cTA= -go.opentelemetry.io/otel/metric/x v0.66.0/go.mod h1:d1+BDj9t96do0/1LoU1ayfCv79ZgNE41qbhBvnMOBZk= -go.opentelemetry.io/otel/sdk v1.44.0 h1:nHYwb9lK+fJPU/dnT6s7W7Z8itMWyqrnVfbheVYrZ58= -go.opentelemetry.io/otel/sdk v1.44.0/go.mod h1:Osuydd3Se74nqjAKxid74N5eC+jfEqfTegHRnq58oK0= -go.opentelemetry.io/otel/sdk/log v0.20.0 h1:vM3xI7TQgKPiSghe6urZtAkyFY7SodrSpC83CffDFuY= -go.opentelemetry.io/otel/sdk/log v0.20.0/go.mod h1:Knej2nmsTUzN79T2eeXdRsjjPcoxoq2pUyUHz9TFyyU= -go.opentelemetry.io/otel/sdk/log/logtest v0.20.0 h1:OqdRZ1guyzamK3M6LlRsmGqRrjkHWw6WZOKKli5ELpg= -go.opentelemetry.io/otel/sdk/log/logtest v0.20.0/go.mod h1:PuMIlm7zAt7c3z8zfOI5ox4iT1Z87We+PF6YoINux/M= -go.opentelemetry.io/otel/sdk/metric v1.44.0 h1:3LlKgI+VjbVsjNRFZJZAJ30WjXC5VkNRks6si09iEfI= -go.opentelemetry.io/otel/sdk/metric v1.44.0/go.mod h1:5B5pMARnXxKhltooO4xUuCBorl65a4EpnTalObqOigA= -go.opentelemetry.io/otel/trace v1.44.0 h1:jxF5CsGYCe74MCRx2X4g7WsY/VBKRqqpNvXlX/6gtIk= -go.opentelemetry.io/otel/trace v1.44.0/go.mod h1:oLl1jrMQAVo6v3GAggN+1VH9VIz9iUSvW53sW1Q8PIE= -go.opentelemetry.io/proto/otlp v1.10.0 h1:IQRWgT5srOCYfiWnpqUYz9CVmbO8bFmKcwYxpuCSL2g= -go.opentelemetry.io/proto/otlp v1.10.0/go.mod h1:/CV4QoCR/S9yaPj8utp3lvQPoqMtxXdzn7ozvvozVqk= +go.opentelemetry.io/otel v1.45.0 h1:pdrWmLHofpubmArBv1LgFSv1Z0Ie/ppdZzu+kUN5EeU= +go.opentelemetry.io/otel v1.45.0/go.mod h1:XZxIqPapzEYnhNSScF5DIqXhm/rYi0FzCe2XddAwZfQ= +go.opentelemetry.io/otel/exporters/otlp/otlplog/otlploggrpc v0.21.0 h1:WseeVYf5dJZTsyPiyW5L14k5qsSibqXAMTSiFEDiWr0= +go.opentelemetry.io/otel/exporters/otlp/otlplog/otlploggrpc v0.21.0/go.mod h1:SiLZnQS6Qk2eCpvr2CH/XMAOa64TWGXxEZJZCpD2Lmc= +go.opentelemetry.io/otel/exporters/otlp/otlpmetric/otlpmetricgrpc v1.45.0 h1:klTViGcsvLCd1xN3rZzfZ12NslC/OimbmR+k+A006RI= +go.opentelemetry.io/otel/exporters/otlp/otlpmetric/otlpmetricgrpc v1.45.0/go.mod h1:jRsK04CWmXuY8A0O+wMpSf+t90RHZ53o5Qmxn2PQPfk= +go.opentelemetry.io/otel/exporters/otlp/otlptrace v1.45.0 h1:QRefszxJmfPdjXUUm3j6iDzY03mTPXMjqErFqQ67vUg= +go.opentelemetry.io/otel/exporters/otlp/otlptrace v1.45.0/go.mod h1:Tiz03lTBVBrm7eWZBOidzEaYaJa8tjwGUGv6d8mlTyk= +go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc v1.45.0 h1:fG5MCxGz8+2VtrN/WgqSpJFctVz24gpxj8CxkKmc8Ww= +go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc v1.45.0/go.mod h1:BmAYTn+3ysbRe+IU2msxmf5Rx3g6DHvex+tWI3LdhYI= +go.opentelemetry.io/otel/log v0.21.0 h1:SLsVDGmtyBrdw8/a2Z0bOIxou/+bN4z56GebH7T0LvA= +go.opentelemetry.io/otel/log v0.21.0/go.mod h1:iReetQrZL9Wyg84cCkOoCmqDHS5RCFfyxC7J+r8fn8g= +go.opentelemetry.io/otel/metric v1.45.0 h1:7Eg1uH7CJ5cXv9is6tnBe1FI6rj1nwUdbFypRm3br/M= +go.opentelemetry.io/otel/metric v1.45.0/go.mod h1:HAPbm1nd3p1PmFH7v2dR+6BjXxw+Lq4a2+pndMAm08s= +go.opentelemetry.io/otel/metric/x v0.67.0 h1:PcicCNZFkZ4bXfSooXdo3WN7RBOVOtjVdo1wD358Uns= +go.opentelemetry.io/otel/metric/x v0.67.0/go.mod h1:FBjCWZe6wgcqxcMtjdGiClDKXb2YxxXii0CXftE4QtI= +go.opentelemetry.io/otel/sdk v1.45.0 h1:4VVSMgQ83dUgW2aoX5f6JgLvHwIvzcuLnF9lUdCSpCw= +go.opentelemetry.io/otel/sdk v1.45.0/go.mod h1:Sr40LgXV7DsKMMJMKOhUWOgMWTfAaqvm2kF0g7ilwuA= +go.opentelemetry.io/otel/sdk/log v0.21.0 h1:QsE7XSR0ktQdKmRKGnR+f1ObGF32WG+7MER/P9KgmYc= +go.opentelemetry.io/otel/sdk/log v0.21.0/go.mod h1:m9mApjCoD2/1QuKCAptjv+BrG9WKOvQLVdNx+iBldTo= +go.opentelemetry.io/otel/sdk/log/logtest v0.21.0 h1:X+JBBgKlswCGYsmgL0CnoUUtlE//VB345c84jYAYkdQ= +go.opentelemetry.io/otel/sdk/log/logtest v0.21.0/go.mod h1:HD1575K8e6sIFBBDd5tZB3t9DlMytWXq9FuR+Y4rfjE= +go.opentelemetry.io/otel/sdk/metric v1.45.0 h1:oVFszMfyj1Am6s24Vtc7wBb8BKLcwepJjNEYILuiE3o= +go.opentelemetry.io/otel/sdk/metric v1.45.0/go.mod h1:vUWUxDZvu1WVRj8JA8S0AdhsPrZoDpA2DdZauIh4mDA= +go.opentelemetry.io/otel/trace v1.45.0 h1:l/mP6Uv7oNO7/TblbhpbgMidxhq1uO/rPsikOyVhxag= +go.opentelemetry.io/otel/trace v1.45.0/go.mod h1:qoJJA2xNMnxRrdISU/kLtfUH2wNeQbiv+jhs/CxI8bc= +go.opentelemetry.io/proto/otlp v1.11.0 h1:5rrYs0Ykyj50sdU/JU0x8etU+LubXWb+gED6TbEdMIk= +go.opentelemetry.io/proto/otlp v1.11.0/go.mod h1:SmVizdCOAm3XBtG1g1NnOdhW6jtddT72hLMhv8VwA8E= go.uber.org/goleak v1.3.0 h1:2K3zAYmnTNqV73imy9J1T3WC+gmCePx2hEGkimedGto= go.uber.org/goleak v1.3.0/go.mod h1:CoHD4mav9JJNrW/WLlf7HGZPjdw8EucARQHekz1X6bE= go.uber.org/multierr v1.10.0 h1:S0h4aNzvfcFsC3dRF1jLoaov7oRaKqRGC/pUEJ2yvPQ= @@ -221,10 +221,10 @@ golang.org/x/text v0.41.0 h1:vz/seA0lnX87Othu2f/0L24RcgrXD9/YFTSuGjj3rH8= golang.org/x/text v0.41.0/go.mod h1:jvf1O8ajNzZqhSrQBPbutR/EB83Cc0CFrezNQIwbb5M= gonum.org/v1/gonum v0.17.0 h1:VbpOemQlsSMrYmn7T2OUvQ4dqxQXU+ouZFQsZOx50z4= gonum.org/v1/gonum v0.17.0/go.mod h1:El3tOrEuMpv2UdMrbNlKEh9vd86bmQ6vqIcDwxEOc1E= -google.golang.org/genproto/googleapis/api v0.0.0-20260526163538-3dc84a4a5aaa h1:Kjn0N0tCrDgiAFW+lGO4JZ3ck44CehvJQMAwj9QF0G8= -google.golang.org/genproto/googleapis/api v0.0.0-20260526163538-3dc84a4a5aaa/go.mod h1:q4lMZS6kskjT5HvCPrnnypcDPVJqT/f4nfxmkE7gryY= -google.golang.org/genproto/googleapis/rpc v0.0.0-20260526163538-3dc84a4a5aaa h1:mZHHdPZl0dbGHCflZgAq/Q468DWVFcU2whhB2KAo8fk= -google.golang.org/genproto/googleapis/rpc v0.0.0-20260526163538-3dc84a4a5aaa/go.mod h1:4Hqkh8ycfw05ld/3BWL7rJOSfebL2Q+DVDeRgYgxUU8= +google.golang.org/genproto/googleapis/api v0.0.0-20260803160001-6ac0973c030d h1:FarXi840EJWSHYTN3ERkADbPWjl307+FGrA22KAVjjc= +google.golang.org/genproto/googleapis/api v0.0.0-20260803160001-6ac0973c030d/go.mod h1:K/+WGbmBY7aNW1HDw1fJnKYo10i0DkAX6pows00dLig= +google.golang.org/genproto/googleapis/rpc v0.0.0-20260803160001-6ac0973c030d h1:IL4hdHzcUv2l/gcg98/Rj3FbtE6axwqslOW8SW0C+S0= +google.golang.org/genproto/googleapis/rpc v0.0.0-20260803160001-6ac0973c030d/go.mod h1:4Hqkh8ycfw05ld/3BWL7rJOSfebL2Q+DVDeRgYgxUU8= google.golang.org/grpc v1.83.2 h1:EManeRomTObA0BU7I8vXgg/78uE5MJ9M8B39EX2WscU= google.golang.org/grpc v1.83.2/go.mod h1:YPI1hK3kDked6iHvgX3tR0y+nX/qpMFKhPgFsokw1S8= google.golang.org/protobuf v1.36.11 h1:fV6ZwhNocDyBLK0dj+fg8ektcVegBBuEolpbTQyBNVE= diff --git a/src/platform/telemetry/audit.go b/src/platform/telemetry/audit.go index 784b2ae..60e50fc 100644 --- a/src/platform/telemetry/audit.go +++ b/src/platform/telemetry/audit.go @@ -5,6 +5,7 @@ import ( "time" "github.com/google/uuid" + "go.opentelemetry.io/otel/attribute" "go.opentelemetry.io/otel/log" "github.com/OpenAgriNet/discovery-service/src/platform/telemetry/fact" @@ -50,7 +51,7 @@ func (p *Provider) EmitAudit(ctx context.Context, event fact.AuditEvent) { record.SetSeverity(log.SeverityInfo4) record.SetSeverityText("INFO") - record.SetBody(log.StringValue("catalog state changed")) + record.SetBody(attribute.StringValue("catalog state changed")) // The five spec-Required names are LITERALS here, which is the one place in // this service that an attribute key is not read from fact. @@ -66,11 +67,11 @@ func (p *Provider) EmitAudit(ctx context.Context, event fact.AuditEvent) { // Required by otel-specification.md §LOG. log_uuid is per-RECORD, not // per-catalog: a consumer uses it to discard a redelivery rather than // count the same transition twice. - log.String("log_uuid", uuid.NewString()), - log.String("item.id", event.ItemID), - log.String("item.type", event.ItemType), - log.String("item.prevstate", event.PrevState), - log.String("item.state", event.State), + attribute.String("log_uuid", uuid.NewString()), + attribute.String("item.id", event.ItemID), + attribute.String("item.type", event.ItemType), + attribute.String("item.prevstate", event.PrevState), + attribute.String("item.state", event.State), ) // Emit reads the span context out of ctx itself, which is how the record diff --git a/src/platform/telemetry/audit_internal_test.go b/src/platform/telemetry/audit_internal_test.go index da94746..5aa3d39 100644 --- a/src/platform/telemetry/audit_internal_test.go +++ b/src/platform/telemetry/audit_internal_test.go @@ -4,6 +4,7 @@ import ( "context" "testing" + "go.opentelemetry.io/otel/attribute" "go.opentelemetry.io/otel/log" sdklog "go.opentelemetry.io/otel/sdk/log" @@ -29,8 +30,8 @@ func (r *recording) ForceFlush(context.Context) error { re func attributesOfRecord(rec sdklog.Record) map[string]string { out := map[string]string{} - rec.WalkAttributes(func(kv log.KeyValue) bool { - out[kv.Key] = kv.Value.String() + rec.WalkAttributes(func(kv attribute.KeyValue) bool { + out[string(kv.Key)] = kv.Value.String() return true }) return out From 12b8405c1749e9c21159540603e9d36f401c8c0f Mon Sep 17 00:00:00 2001 From: Prajwal TD <157502711+tdprajwal@users.noreply.github.com> Date: Mon, 12 Oct 2026 12:52:24 +0530 Subject: [PATCH 08/23] feat: carry discover pagination in the signed body [#51] A discover answered one page and said nothing about it. The page could only be asked for in the query string, which the request signature does not cover and which no Beckn caller knows to send, and the answer had no member to say which page it was or whether more existed - so a short page and the last page read the same. The validator now loads the OAN profile of beckn.yaml, pinned at commit dfc1097 of OpenAgriNet/protocol-specifications-v2 (no tag exists yet; swap the SHA for one when it is cut, never for the branch). It adds limit/offset to DiscoverAction and an optional pagination to OnDiscoverAction. Recorded as A28 in the plan. - The page is read from the body, each field falling back to the deprecated query parameter of the same name, so existing callers (verify.sh among them) keep working. The body wins because it is the signed half. - pagination reports the page APPLIED, after default and clamp: a caller that asked for 5000 must read back 100, or its next offset skips results. - total is len(fused) and is sent only when exact. It is omitted whenever any retrieval list came back as long as MaxCandidatesPerMode, because that list may have been cut by its LIMIT. Rejected: reviving the count query, which A19 deleted for costing 100x the search. Reverting the cap check makes TestATotalIsReportedOnlyWhenNoModeHitItsCap report 8 where 10 matched. - The past-depth refusal is unchanged and now points at $['message']['offset']. - config refuses maxPageSize above 100, the schema's maximum, which would otherwise answer query-string callers with a pagination.limit that fails its own schema. Not in scope: the same profile lets context.networkId be an array. beckn.Context still reads a string, so a list is refused by the envelope parser exactly as before; its meaning for scope and visibleTo is its own issue. --- README.md | 12 +-- docker-compose.yml | 5 +- docs/design/implementation-plan.md | 11 ++- docs/publish-and-discover.md | 9 ++- docs/quickstart.md | 10 ++- src/beckn/actions.go | 35 ++++++-- src/beckn/schema_conformance_test.go | 37 ++++++++- src/discover/controller.go | 32 +++++--- src/discover/controller_test.go | 97 ++++++++++++++++++++++ src/discover/intent_mapper.go | 7 +- src/discover/intent_mapper_test.go | 4 + src/discover/service.go | 24 +++--- src/discover/service_test.go | 62 ++++++++++---- src/domain/query.go | 10 ++- src/platform/config/config.go | 31 +++++-- src/platform/config/config_test.go | 7 ++ src/platform/validation/spec_index.go | 2 +- src/storage/conformance/discover.go | 25 +++++- src/storage/memory/search.go | 4 + src/storage/postgres/read_path_test.go | 27 +++++++ src/storage/postgres/search_repository.go | 24 +++++- tests/acceptance/pagination_test.go | 98 +++++++++++++++++++++++ tests/acceptance/suite_test.go | 12 ++- tests/testdata/beckn-v2.0.0.yaml | 80 ++++++++++++------ 24 files changed, 553 insertions(+), 112 deletions(-) create mode 100644 tests/acceptance/pagination_test.go diff --git a/README.md b/README.md index db5c19e..25ae2f5 100644 --- a/README.md +++ b/README.md @@ -69,8 +69,9 @@ That stack applies its own migrations (they are compiled into the binary) and reads the Beckn specification from `tests/testdata/beckn-v2.0.0.yaml`, mounted at the cache path. The boot logs one warning about the registry fetch it did not do, which is why it works with no network. That file is byte-identical to -the `core-v2.0.0-lts` tag of `beckn/protocol-specifications-v2`; set -`VALIDATION_SPEC_URL` to its raw URL to exercise the fetch path instead. +the OAN profile of the spec at commit `dfc1097` of +`OpenAgriNet/protocol-specifications-v2` (A28); set `VALIDATION_SPEC_URL` to its +raw URL to exercise the fetch path instead. With the stack up, there is a worked catalog and the requests that find it: @@ -118,9 +119,10 @@ overrides; that file is gitignored. **Secrets — `DATABASE_URL` above all — belong in neither YAML file.** The Beckn specification is fetched at boot from `VALIDATION_SPEC_URL` and -cached under `.cache/beckn/`. It defaults to the `core-v2.0.0-lts` tag of -`beckn/protocol-specifications-v2` — a tag rather than a branch, so an upstream -merge cannot change the validator under a running deployment. It is not baked +cached under `.cache/beckn/`. It defaults to the OAN profile at commit +`dfc1097` of `OpenAgriNet/protocol-specifications-v2` (A28) — a commit rather +than a branch, so an upstream merge cannot change the validator under a running +deployment. It is not baked into the image; air-gapped deploys mount a cache file at `VALIDATION_SPEC_CACHE_PATH`. diff --git a/docker-compose.yml b/docker-compose.yml index 99d9297..8316f1b 100644 --- a/docker-compose.yml +++ b/docker-compose.yml @@ -81,8 +81,9 @@ services: APP_DOMAIN: ${APP_DOMAIN:-Agriculture} volumes: # The Beckn spec, which the service refuses to boot without. Byte-identical - # to the core-v2.0.0-lts tag of beckn/protocol-specifications-v2, which is - # also VALIDATION_SPEC_URL's default — so a normal boot fetches, compiles + # to the OAN profile at commit dfc1097 of + # OpenAgriNet/protocol-specifications-v2 (A28), which is also + # VALIDATION_SPEC_URL's default — so a normal boot fetches, compiles # and finds this file already holding the same bytes, and an offline boot # falls back to it with one accurate warning about the fetch that failed. # diff --git a/docs/design/implementation-plan.md b/docs/design/implementation-plan.md index fb4fcc7..f875b66 100644 --- a/docs/design/implementation-plan.md +++ b/docs/design/implementation-plan.md @@ -151,6 +151,7 @@ field carrying **an array of network ids** (not `PUBLIC`/`PRIVATE`). | **A25** | **This service is one of four layers, and the plan was written as if it were all of them. Tasks 25 and 26 are added, and 23a and 23c grow one requirement each.** The other three layers — experience, network, provider — run **beckn-onix** adapters that already emit OTel traces, metrics and audit logs, and reading them changes four things here. **Two questions this plan could not answer are answered elsewhere, not nowhere.** Relevance and accuracy need what the user did *after* the response; the experience-layer adapter has the user and already emits traces, so that is where the outcome event belongs. Performance per location and per commodity is refused here because coordinates and `filters.expression` are on the never-emitted list — but the provider adapter holds both as first-class call-plan fields, so it answers them without anyone widening a deny-list. The H3-coarsening policy call that `opentelemetry.md` raised against 23d is **withdrawn** on those grounds. **Our spans would currently be exported and then silently dropped.** onix's network trace pipeline admits only spans carrying `sender.id`, which this phase makes optional and usually absent, while the stage downstream of it rebuilds `trace_id` from an attribute spelled `transaction_id`, which we planned to spell `beckn.transactionId`. A span that fails either is stored, paid for and invisible, with nothing anywhere reporting an error. The filter is onix's bug to fix — it admits on one key and joins on another — and the spelling is ours: **23c emits `transaction_id` and `message_id` as aliases**, written from the same values at the same call site so they cannot drift. Two, not three — an earlier draft named `receiver.id`, which onix puts only on audit log records; its span spelling is `recipient.id`, which we already emit, so a third alias would be paid for everywhere and read nowhere. Buying admission by inventing a `sender.id` is explicitly refused; missing from a dashboard is recoverable, poisoning the network's only participant-identity join is not. **Metrics were dropped from this service for a reason that covers only half the ground.** A23 removed `metrics.go` because a stateless replica's in-memory counter is a partial nobody can reassemble — true of the facilitator's windowed aggregates, and the reason Task 24 exists. It is not true of a *level*: three configured ceilings (`DATABASE_MAX_CONNS`, the rate limiter, the body size) have no observable distance-to-limit, and a pool at 30 of 32 refuses nothing, so it produces no span to be seen in. An earlier draft of this amendment said a 429 "produces no span, no event and no counter". **That is false, and the correction matters because it is what the task's scope was sized against**: `Trace` is index 1 of `router.go`'s `chain`, above both `Envelope` and `RateLimit`, so a 429 and a body refusal each produce a span carrying the status and — post-23d — an `error` event, both projected from the single `logNack` in `response_writer.go` that every refusal already passes through. Only "no counter" was ever true, and a counter that restates a span fact is the `duration_ms` mistake one signal up. **Task 25 is therefore the levels, not the refusals**, and it is smaller for it. Conflating the operator set with the network set is why this service has no metrics at all. **Task 25** is the operator set and is **not blocked**; Task 24 stays blocked on the registry — *five of its twelve codes have since shipped, on 2026-09-10, derived in `otel/collector.yaml`; the block stands for the other seven. See Task 24.* **And the deny-list is a rule nothing enforces** — five sub-tasks comply with it by hand. **Task 26** turns it into a pin, over the exporter's output rather than over the code that builds it | 23, 24, 25, 26 | | **A26** | **The publish geometry budget becomes configuration.** `MaxGeometriesPerCatalog` moves from a `const` in `src/publish/geometry.go` to `config.Geo.MaxGeometriesPerCatalog` / `GEO_MAX_GEOMETRIES_PER_CATALOG`, default **256**, threaded `container → publish.NewService → derive → ExtractGeometries → catalogWalk`. **The default does not change and no behaviour over the ceiling changes** — the extra finds are still `PARTIAL` faults naming their paths, never a silent drop. What changes is who gets to pick the number. **The failure it causes is the wrong shape to be a build-time constant.** Over the ceiling the publish still answers HTTP 200 and the publisher is told, but those shapes are not searchable, and at the operator that reads as missing data rather than as a limit: a 100 km `S_DWITHIN` returns fewer markets than exist and nothing in the response says why. Measured against a real catalog — 273 Maharashtra markets carrying a `Point` under both `coverageAreas` and `market.location`, so 546 shapes — only the first 128 markets were findable by place under 256. **Config rather than a larger constant, for the reason `ResolutionCells` two rows above already gives:** how many shapes a catalog carries is a property of a deployment's publishers, and the ceiling is reached by publishing normally rather than by abuse. That is also the line that decides which neighbours do *not* move: `MaxCatalogWalkDepth` bounds a stack against a pathological document, which is abuse, and `MaxIndexCoverCells` is a property of the index rather than of a catalog. **Zero is refused at boot**, alongside the body ceiling and for the same reason — it reads either as "unlimited" or as "refuse everything", and a catalog that may hold no geometry is undiscoverable by place. **The duplicate declaration in `src/indexing/geo/h3.go` is deleted, not rewired.** It had zero call sites; two constants of one name in two packages is how an operator raises the ceiling in the half that is not read. **What the number stands in for is cells, not shapes:** it times `MaxIndexCoverCells` (8,192) is the worst case one publish can ask the index to hold, so a points-only catalog — one cell per shape — can raise it a long way and a polygon-heavy one cannot | 2, 17, 18 | | **A27** | **A deployment may decline free-text retrieval, and declining REFUSES where a missing backend mode degrades.** `Search.EnableTextSearch` / `SEARCH_ENABLE_TEXT_SEARCH`, **true by default** — the value that takes no decision has to be the one that behaves as before the flag existed. With it false, an intent carrying a non-empty `textSearch` is refused in `MapIntent` with `SCH_TYPE_NOT_SUPPORTED` at `$['message']['intent']['textSearch']`, and the message names `spatial` and `filters` as what still works. **The whole content of this amendment is that it is not `FailOnUnavailableMode` under another name**, because the two read alike and behave oppositely. That flag is about a mode the **backend** cannot run — `semantic` on a `noop` deployment — and false there degrades, answering with what the other modes found and naming the gap in `X-Beckn-Degraded`; the query the caller wrote survives without it. This one is the **deployment's** policy, and degrading would be the failure C11 exists to prevent: on a text-only intent the term IS the query, so running the remaining modes without it answers **the whole corpus under a 200** — a page indistinguishable at the caller from a correct one. That is the same widening `MapIntent`'s doc comment already refuses for an unreadable `targets` or an ignored SRID, which is why the guard sits there and not in `negotiate`. **Three details are load-bearing.** The guard reads the **trimmed** term, like every other reader of it: `" "` asks for no retrieval mode at all, so it stays the no-criterion refusal and calling it a disabled text search would name the wrong mistake. The intent's required-one-of **list** shrinks with the flag — `spatial or filters`, with a parenthetical saying why the third is absent — because offering `textSearch` to a caller this deployment would then refuse walks them into a second `400`; the **guard** does not shrink with it, since a term that was sent is a criterion that was sent and reporting it missing would be a false sentence stacked on the true one. And the wire message names the two criteria that still work rather than the environment variable, matching `negotiate` and the `filters.type` refusal: a caller cannot act on the variable, and the operator has `docs/publish-and-discover.md`. **`modesFor` is left alone.** It still maps a non-empty `Text` to all three ranked modes, because the refusal is total and unreachable past it; a second guard there would be a branch no test can reach and the kind of defence that rots into a contradiction | 2, 17, 19 | +| **A28** | **The page rides in the signed body, and the answer says which page it is (#51).** The schema is now the **OAN profile** of `beckn.yaml`, pinned at commit `dfc1097` of `OpenAgriNet/protocol-specifications-v2` (`VALIDATION_SPEC_URL`'s default and `tests/testdata/beckn-v2.0.0.yaml`, byte-identical) — a commit because the branch carries no tag yet; it becomes a tag when one is cut, never the branch name. That profile adds `limit` (1-100) and `offset` (>= 0) to `DiscoverAction` and an optional `pagination` (`limit`, `offset`, `total`) to `OnDiscoverAction`. **Three decisions.** (1) The page is read from the **body**, each field falling back to the query parameter of the same name. The body wins field by field because it is the half the request signature covers; the query string survives only so callers written before the profile keep working, and it still clamps where the body is bounded by the schema — a body `limit` of 101 is an L1 refusal, not a clamp. (2) `pagination` reports the page **applied**, after the default and the clamp: a caller that asked for 5000 must read back the 100 it got, or it computes its next offset from a page size it never received. (3) `total` is sent **only when exact**, and is never a count query — **A19**'s reason for deleting that query (150.6 ms against a 1.5 ms search) stands. It is `len(fused)`, the number the search already holds, and it is omitted whenever any retrieval list came back as long as `MaxCandidatesPerMode`, because such a list may have been cut by its `LIMIT` and the fusion of it is a truncation. A list of exactly the cap that happened to be complete is reported uncounted too — the one false negative, and the safe direction: an absent total says *narrow the query*, a short one says *this is everything* when it is not. Degradation does not suppress it; that is `X-Beckn-Degraded`'s job, and the total is the size of the set this answer pages through. The memory backend has no cap, so its total is always present. **Unchanged:** `offset + limit > MaxCandidatesPerMode` is still refused (now at `$['message']['offset']`), never answered with an empty page. **Not taken here:** the same profile makes `context.networkId` a string or an array; `beckn.Context.NetworkID` is still a string, so a list is refused by the envelope parser exactly as before — its meaning for scope and `visibleTo` is a separate issue. Supersedes A19's "no total" for this one exact case, and the "exactly one field" note on `OnDiscoverAction` | 11, 16, 19 | A6 and A7 exist for one requirement: *swap the text backend later, keep geo on PG, and let publish write to two stores.* Both build **seams plus conformance @@ -2165,7 +2166,9 @@ POST /discover "provider": { ... }, // fetched once per catalog on the page "resources": [ ... ], // only the ones that matched "offers": [ ... ] // only those touching a matched - } ] } } // resource, plus catalog-wide ones + } ], // resource, plus catalog-wide ones + "pagination": { "limit": 20, "offset": 0, "total": 45 } // A28 + } } ``` ```http @@ -2173,9 +2176,9 @@ X-Beckn-Degraded: structured ``` The degraded list is a **header, not a body field** (C11). `OnDiscoverAction` -declares `additionalProperties: false` with `catalogs` as its only property, so -a `degraded` key inside `message` is not an extension — it is a response that -fails its own schema. The header is absent when nothing degraded. +declares `additionalProperties: false` with `catalogs` and `pagination` (A28) as +its only properties, so a `degraded` key inside `message` is not an extension — +it is a response that fails its own schema. The header is absent when nothing degraded. ### The flow diff --git a/docs/publish-and-discover.md b/docs/publish-and-discover.md index aa78591..113ede6 100644 --- a/docs/publish-and-discover.md +++ b/docs/publish-and-discover.md @@ -19,9 +19,10 @@ That is the whole of it: | `GET /readyz` | the platform | can it also reach PostgreSQL | Four routes, no wildcard mount, no aliases. Both APIs are implemented against -the published Beckn v2.0.0 specification — -[`beckn/protocol-specifications-v2`](https://github.com/beckn/protocol-specifications-v2/blob/core-v2.0.0-lts/api/v2.0.0/beckn.yaml) -at the `core-v2.0.0-lts` tag, pinned in this repo as +the Beckn v2.0.0 specification as OpenAgriNet profiles it — +[`OpenAgriNet/protocol-specifications-v2`](https://github.com/OpenAgriNet/protocol-specifications-v2/blob/dfc109759b3781fde74539adc79df63fccbadb83/api/v2.0.0/beckn.yaml) +at commit `dfc1097`, which adds discover pagination to beckn's `core-v2.0.0-lts` +(A28), pinned in this repo as [`tests/testdata/beckn-v2.0.0.yaml`](../tests/testdata/beckn-v2.0.0.yaml). That document *is* the validator: the service loads it and validates every request against it, rather than restating the protocol in Go. @@ -469,7 +470,7 @@ matches no field fails the boot, so a typo cannot silently do nothing. | `EMBEDDING_MODEL` / `_ENDPOINT` / `_DIMENSIONS` | `nomic-embed-text` / `http://localhost:11434` / `768` | the Ollama seam | | `RATE_LIMIT_RPS` / `_BURST` | `20` / `40` | per-client limit; `burst >= rps` is enforced | | `VALIDATION_ENABLE_L1_SCHEMA` | `true` | Beckn schema validation of the payload | -| `VALIDATION_SPEC_URL` | the `core-v2.0.0-lts` raw URL | where to fetch the spec; a tag, so it cannot move under a running deployment | +| `VALIDATION_SPEC_URL` | the OAN profile's raw URL at commit `dfc1097` | where to fetch the spec; a commit, so it cannot move under a running deployment | | `VALIDATION_SPEC_CACHE_PATH` | `.cache/beckn/beckn.yaml` | where the spec is read from | | `AUTH_ENABLE_SIGNATURE_VERIFICATION` | `false` | `true` refuses the boot — see below | | `EXT_ALLOW_NETWORK_FETCH` | `false` | may a URL from a *request body* be fetched | diff --git a/docs/quickstart.md b/docs/quickstart.md index 77ab38c..389532f 100644 --- a/docs/quickstart.md +++ b/docs/quickstart.md @@ -270,20 +270,22 @@ does not remove the requirement. It is fetched from `VALIDATION_SPEC_URL`, which now defaults to ``` -https://raw.githubusercontent.com/beckn/protocol-specifications-v2/refs/tags/core-v2.0.0-lts/api/v2.0.0/beckn.yaml +https://raw.githubusercontent.com/OpenAgriNet/protocol-specifications-v2/dfc109759b3781fde74539adc79df63fccbadb83/api/v2.0.0/beckn.yaml ``` and cached under `VALIDATION_SPEC_CACHE_PATH` (`.cache/beckn/beckn.yaml`). That is why running the binary from the host needs no spec configuration at all: it fetches once and reads the cache afterwards. -A **tag**, not a branch, which is the whole reason there can be a default. A +A **commit**, not a branch, which is the whole reason there can be a default. A branch would let an upstream merge change the validator under a running -deployment without a deploy; `refs/tags/core-v2.0.0-lts` cannot move. A network +deployment without a deploy; a commit cannot move. It is OpenAgriNet's profile +of the Beckn spec rather than beckn's `core-v2.0.0-lts` tag because the profile +adds discover pagination (A28), and it becomes a tag when one is cut. A network that trusts a different document still overrides it. Compose additionally mounts `tests/testdata/beckn-v2.0.0.yaml` straight into -the cache path. That file is byte-identical to the tag, so the two paths agree +the cache path. That file is byte-identical to that commit, so the two paths agree — and the mount is what lets the container stack come up with no network at all. If the fetch fails, the boot logs one loud warning and falls back to the cache; that warning is accurate and is not a failure. diff --git a/src/beckn/actions.go b/src/beckn/actions.go index 0692610..caea0e8 100644 --- a/src/beckn/actions.go +++ b/src/beckn/actions.go @@ -135,18 +135,37 @@ type CatalogStats struct { } // DiscoverAction is the `message` of a discover request. +// +// Limit and Offset are the OAN profile's addition (A28): the page rides in the +// signed body rather than the query string, which the request signature does +// not cover. Pointers, because absent and zero differ — an absent field falls +// back to the query string, and the schema refuses a zero limit outright. type DiscoverAction struct { Intent Intent `json:"intent"` + Limit *int `json:"limit,omitempty"` + Offset *int `json:"offset,omitempty"` } -// OnDiscoverAction is the `message` of the discover response, and it is exactly -// one field. +// OnDiscoverAction is the `message` of the discover response. // -// It must NOT grow a Degraded field. The v2.0.0 schema is -// additionalProperties:false with `catalogs` as its only property, so an extra -// key is a response that fails its own schema — and `omitempty` would hide that -// on the ordinary path while shipping the invalid body on the degraded one, the -// path that matters. The degraded list travels as X-Beckn-Degraded (C11). +// It must NOT grow a Degraded field. The schema is additionalProperties:false +// with `catalogs` and `pagination` as its only properties, so an extra key is a +// response that fails its own schema — and `omitempty` would hide that on the +// ordinary path while shipping the invalid body on the degraded one, the path +// that matters. The degraded list travels as X-Beckn-Degraded (C11). type OnDiscoverAction struct { - Catalogs []Catalog `json:"catalogs"` + Catalogs []Catalog `json:"catalogs"` + Pagination *Pagination `json:"pagination,omitempty"` +} + +// Pagination is the page a discover answer APPLIED — after the default and the +// clamp, not as requested — so the caller computes the next offset from what +// it actually got. +// +// Total is a pointer and omitted when nil: it is sent only when exact (A28). +// Offset is never omitted, because zero is the first page and not "unset". +type Pagination struct { + Limit int `json:"limit"` + Offset int `json:"offset"` + Total *int `json:"total,omitempty"` } diff --git a/src/beckn/schema_conformance_test.go b/src/beckn/schema_conformance_test.go index df5efd5..9ec5ef5 100644 --- a/src/beckn/schema_conformance_test.go +++ b/src/beckn/schema_conformance_test.go @@ -69,6 +69,7 @@ func bindings() []binding { {"DiscoverAction", DiscoverAction{}, schemaPath("DiscoverAction")}, {"OnDiscoverAction", OnDiscoverAction{}, schemaPath("OnDiscoverAction")}, + {"Pagination", Pagination{}, schemaPath("Pagination")}, {"Nack", Nack{}, schemaPath("NackBadRequest")}, {"NackMessage", NackMessage{}, schemaPath("NackBadRequest", "properties", "message")}, @@ -134,7 +135,7 @@ func TestWireStructsMatchTheSpecification(t *testing.T) { var found []deviation for _, b := range bindings() { - found = append(found, deviations(t, b, resolve(t, spec, b.path))...) + found = append(found, deviations(t, b, flattenAllOf(t, spec, resolve(t, spec, b.path)))...) } sort.Slice(found, func(i, j int) bool { return found[i].key < found[j].key }) @@ -157,6 +158,40 @@ func TestWireStructsMatchTheSpecification(t *testing.T) { } } +// flattenAllOf merges the properties of an allOf node's members into one +// properties map, following each member's local $ref. Pagination is the first +// bound schema composed this way — allOf[PaginationParams, {total}] in the OAN +// profile — and a walk that read only the top-level `properties` would find +// none and fail the binding rather than check it. A node without allOf comes +// back unchanged. +func flattenAllOf(t *testing.T, spec map[string]any, node map[string]any) map[string]any { + t.Helper() + + members, ok := node["allOf"].([]any) + if !ok { + return node + } + + merged := map[string]any{} + for _, member := range members { + part, ok := member.(map[string]any) + if !ok { + t.Fatalf("an allOf member is not an object: %v", member) + } + if ref, isRef := part["$ref"].(string); isRef { + part = resolve(t, spec, strings.Split(strings.TrimPrefix(ref, "#/"), "/")) + } + properties, ok := flattenAllOf(t, spec, part)["properties"].(map[string]any) + if !ok { + t.Fatalf("an allOf member declares no properties: %v", member) + } + for name, schema := range properties { + merged[name] = schema + } + } + return map[string]any{"properties": merged} +} + // deviations compares one struct's JSON tags against its schema node's // properties, in both directions. An extra tag ships a key the schema does not // declare; a missing one drops a key the schema does, which for a service that diff --git a/src/discover/controller.go b/src/discover/controller.go index 45f6691..57d2a98 100644 --- a/src/discover/controller.go +++ b/src/discover/controller.go @@ -68,19 +68,19 @@ func (c *Controller) Discover(w http.ResponseWriter, r *http.Request) { observeIntent(ctx, envelope.Context, action.Intent) - page, err := pageFrom(r.URL.Query()) + page, err := pageFrom(action, r.URL.Query()) if err != nil { httpx.WriteNack(ctx, w, c.errors, envelope.Context.MessageID, err) return } - catalogs, degraded, err := c.service.Discover(ctx, envelope.Context, action.Intent, page) + answer, degraded, err := c.service.Discover(ctx, envelope.Context, action.Intent, page) if err != nil { httpx.WriteNack(ctx, w, c.errors, envelope.Context.MessageID, err) return } - observeResult(ctx, catalogs) + observeResult(ctx, answer.Catalogs) // Before the body: WriteJSON writes the status line, and a header set after // that is a header nobody receives. @@ -90,20 +90,26 @@ func (c *Controller) Discover(w http.ResponseWriter, r *http.Request) { body := httpx.Envelope[beckn.OnDiscoverAction]{ Context: responseContext(envelope.Context), - Message: beckn.OnDiscoverAction{Catalogs: catalogs}, + Message: answer, } if err := httpx.WriteJSON(ctx, w, http.StatusOK, body); err != nil { httpx.WriteNack(ctx, w, c.errors, envelope.Context.MessageID, err) } } -// pageFrom reads pagination off the query string, where it lives rather than -// inside the intent. +// pageFrom reads the page off the body, field by field falling back to the +// query string (A28). // -// An unreadable value is refused, not defaulted: zero already means "the default -// page size" to the mapper, so reading `limit=twenty` as zero would answer a page -// the caller never asked for and say nothing about it. -func pageFrom(query url.Values) (Page, error) { +// The body is the signed half, so a field it names wins. The query string is +// the deprecated route and stays only so that callers written before the OAN +// profile added limit and offset to DiscoverAction keep working; the schema +// already bounds the body's values, which is why only the query's are parsed +// here. +// +// An unreadable query value is refused, not defaulted: zero already means "the +// default page size" to the mapper, so reading `limit=twenty` as zero would +// answer a page the caller never asked for and say nothing about it. +func pageFrom(action beckn.DiscoverAction, query url.Values) (Page, error) { limit, err := intParam(query, "limit") if err != nil { return Page{}, err @@ -112,6 +118,12 @@ func pageFrom(query url.Values) (Page, error) { if err != nil { return Page{}, err } + if action.Limit != nil { + limit = *action.Limit + } + if action.Offset != nil { + offset = *action.Offset + } return Page{Limit: limit, Offset: offset}, nil } diff --git a/src/discover/controller_test.go b/src/discover/controller_test.go index c14d147..11e6aee 100644 --- a/src/discover/controller_test.go +++ b/src/discover/controller_test.go @@ -412,3 +412,100 @@ func TestARefusedOperatorIsA400(t *testing.T) { t.Errorf("the backend was searched %d times; a refused intent runs no query", repo.calls) } } + +// The page travels in the SIGNED body now (A28). The OAN profile of the schema +// declares limit and offset on DiscoverAction, so a caller no longer needs the +// query string — which the request signature does not cover. +func TestLimitAndOffsetAreReadFromTheBody(t *testing.T) { + repo := &stubRepo{capabilities: everything()} + + recorder := post(t, mount(t, repo, settings()), "/discover", + `{"context":{"action":"discover"},"message":{"intent":{"textSearch":"wheat"},"limit":5,"offset":10}}`) + if recorder.Code != http.StatusOK { + t.Fatalf("status = %d, want 200; body: %s", recorder.Code, recorder.Body) + } + if repo.gotQuery.Limit != 5 || repo.gotQuery.Offset != 10 { + t.Errorf("page = limit %d offset %d, want the body's 5 and 10", repo.gotQuery.Limit, repo.gotQuery.Offset) + } +} + +// Each field the body names wins over the same field in the query string, and a +// field the body omits still falls back to it. The body is the signed half, so +// it is the one to believe when the two disagree. +func TestTheBodyWinsOverTheQueryStringFieldByField(t *testing.T) { + repo := &stubRepo{capabilities: everything()} + + recorder := post(t, mount(t, repo, settings()), "/discover?limit=7&offset=40", + `{"context":{"action":"discover"},"message":{"intent":{"textSearch":"wheat"},"offset":10}}`) + if recorder.Code != http.StatusOK { + t.Fatalf("status = %d, want 200; body: %s", recorder.Code, recorder.Body) + } + if repo.gotQuery.Offset != 10 { + t.Errorf("Offset = %d, want the body's 10 over the query's 40", repo.gotQuery.Offset) + } + if repo.gotQuery.Limit != 7 { + t.Errorf("Limit = %d, want the query's 7 — the body named none", repo.gotQuery.Limit) + } +} + +// The answer says which page it is: the limit and offset actually APPLIED, after +// the default and the clamp, and the total when the backend could count it. +func TestTheResponseReportsThePageItApplied(t *testing.T) { + total := 3 + repo := &stubRepo{ + capabilities: everything(), + result: domain.SearchResult{Catalogs: []domain.Catalog{{ID: "c1"}}, Total: &total}, + } + + message := onDiscover(t, post(t, mount(t, repo, settings()), "/discover?limit=1000&offset=4", wheat)) + + want := beckn.Pagination{Limit: settings().Search.MaxPageSize, Offset: 4} + if message.Pagination == nil { + t.Fatal("pagination is absent, want the applied page") + } + if got := *message.Pagination; got.Limit != want.Limit || got.Offset != want.Offset || + got.Total == nil || *got.Total != total { + t.Errorf("pagination = %+v (total %v), want limit %d offset %d total %d", + got, got.Total, want.Limit, want.Offset, total) + } +} + +// A total the backend could not count exactly is left OUT, not sent as zero or +// as a guess. A caller reads its absence as "more matched than a page walk can +// reach"; a zero would read as "nothing matched". +func TestAnUncountedTotalIsAbsentFromTheWire(t *testing.T) { + repo := &stubRepo{capabilities: everything(), result: domain.SearchResult{Total: nil}} + + recorder := post(t, mount(t, repo, settings()), "/discover", wheat) + envelope := decodeResponse(t, recorder) + + var message struct { + Pagination map[string]any `json:"pagination"` + } + if err := json.Unmarshal(envelope.Message, &message); err != nil { + t.Fatalf("decoding the message: %v", err) + } + pagination := message.Pagination + if pagination == nil { + t.Fatal("pagination is absent, want the applied page without a total") + } + if _, ok := pagination["total"]; ok { + t.Errorf("pagination = %v, want no total key", pagination) + } + if offset, ok := pagination["offset"]; !ok || offset != float64(0) { + t.Errorf("pagination = %v, want offset 0 present — zero is a real offset", pagination) + } +} + +func onDiscover(t *testing.T, recorder *httptest.ResponseRecorder) beckn.OnDiscoverAction { + t.Helper() + + if recorder.Code != http.StatusOK { + t.Fatalf("status = %d, want 200; body: %s", recorder.Code, recorder.Body) + } + var message beckn.OnDiscoverAction + if err := json.Unmarshal(decodeResponse(t, recorder).Message, &message); err != nil { + t.Fatalf("decoding the message: %v", err) + } + return message +} diff --git a/src/discover/intent_mapper.go b/src/discover/intent_mapper.go index 5390f09..d23f76a 100644 --- a/src/discover/intent_mapper.go +++ b/src/discover/intent_mapper.go @@ -14,8 +14,9 @@ import ( "github.com/OpenAgriNet/discovery-service/src/platform/jsonpath" ) -// Page is the request's pagination, which arrives as HTTP query parameters -// rather than inside the intent. A struct rather than two adjacent ints, because +// Page is the request's pagination: DiscoverAction's limit and offset, each +// falling back to the deprecated query parameter of the same name (A28). Zero +// means "the default". A struct rather than two adjacent ints, because // `limit, offset` and `offset, limit` compile identically. type Page struct { Limit int @@ -229,7 +230,7 @@ func mapPage(page Page, search config.Search) (limit, offset int, faults []domai if offset+limit > search.MaxCandidatesPerMode { faults = append(faults, domain.Fault{ - Path: "$['offset']", + Path: "$['message']['offset']", Code: string(beckn.CodeSchemaInvalidFormat), Message: fmt.Sprintf( "offset %d plus limit %d passes the retrieval depth of %d", diff --git a/src/discover/intent_mapper_test.go b/src/discover/intent_mapper_test.go index 08e8550..2f512f8 100644 --- a/src/discover/intent_mapper_test.go +++ b/src/discover/intent_mapper_test.go @@ -324,6 +324,10 @@ func TestLimitIsClampedAndAPagePastTheRetrievalDepthIsRefused(t *testing.T) { if !strings.Contains(fatal[0].Message, "500") { t.Errorf("Message = %q, want it to name the boundary", fatal[0].Message) } + // The page is a body field now (A28), so the fault points into the body. + if fatal[0].Path != "$['message']['offset']" { + t.Errorf("Path = %q, want $['message']['offset']", fatal[0].Path) + } } // NetworkID is NOT set here. The service sets it from the envelope, and empty diff --git a/src/discover/service.go b/src/discover/service.go index 2960c12..2fc0e0a 100644 --- a/src/discover/service.go +++ b/src/discover/service.go @@ -33,15 +33,16 @@ func NewService(repo domain.SearchRepository, cfg config.Config) *Service { return &Service{repo: repo, cfg: cfg} } -// Discover answers one intent with one page of catalogs and the retrieval modes -// that did not contribute. The degraded list is returned beside the catalogs -// rather than inside them, because it reaches the caller as a header (C11). +// Discover answers one intent with one page of catalogs, the page it applied, +// and the retrieval modes that did not contribute. The degraded list is +// returned beside the message rather than inside it, because it reaches the +// caller as a header (C11). func (s *Service) Discover( ctx context.Context, envelope beckn.Context, intent beckn.Intent, page Page, -) ([]beckn.Catalog, []string, error) { +) (beckn.OnDiscoverAction, []string, error) { query, fatal, partial := MapIntent(intent, envelope, page, s.cfg) if len(fatal) > 0 { - return nil, nil, refusal(fatal) + return beckn.OnDiscoverAction{}, nil, refusal(fatal) } reportPartials(ctx, partial) @@ -54,12 +55,12 @@ func (s *Service) Discover( modes, degraded, err := s.negotiate(query) if err != nil { - return nil, nil, err + return beckn.OnDiscoverAction{}, nil, err } result, err := s.repo.Search(ctx, query, modes) if err != nil { - return nil, nil, typedSearchFailure(ctx, err) + return beckn.OnDiscoverAction{}, nil, typedSearchFailure(ctx, err) } // Both halves, joined once: negotiate's are the modes this deployment cannot @@ -68,7 +69,12 @@ func (s *Service) Discover( degraded = append(degraded, result.Degraded...) observeRetrieval(ctx, modes, degraded) - return render(result.Catalogs), degraded, nil + return beckn.OnDiscoverAction{ + Catalogs: render(result.Catalogs), + // What MapIntent applied, not what was asked: a caller that sent 5000 + // must read back the 100 it got, or its next offset skips results. + Pagination: &beckn.Pagination{Limit: query.Limit, Offset: query.Offset, Total: result.Total}, + }, degraded, nil } // typedSearchFailure says whose mistake a failed search was. @@ -84,7 +90,7 @@ func typedSearchFailure(ctx context.Context, err error) error { if errors.Is(err, domain.ErrRetrievalDepth) { return apperrors. Schema(beckn.CodeSchemaInvalidFormat, err.Error()). - At(jsonpath.Dot("$['offset']")) + At(jsonpath.Dot("$['message']['offset']")) } // An expression the store's own parser refused is the caller's mistake too, diff --git a/src/discover/service_test.go b/src/discover/service_test.go index 6ca6e73..4dfecf9 100644 --- a/src/discover/service_test.go +++ b/src/discover/service_test.go @@ -99,14 +99,14 @@ func TestAModeTheBackendLacksIsNamedRatherThanDropped(t *testing.T) { result: domain.SearchResult{Catalogs: []domain.Catalog{{ID: "c1"}}}, } - catalogs, degraded, err := discover.NewService(repo, settings()). + answer, degraded, err := discover.NewService(repo, settings()). Discover(t.Context(), beckn.Context{}, beckn.Intent{TextSearch: "wheat seeds"}, discover.Page{}) if err != nil { t.Fatalf("Discover: %v", err) } - if len(catalogs) != 1 { - t.Errorf("catalogs = %d, want the one the backend found — degrading is not failing", len(catalogs)) + if len(answer.Catalogs) != 1 { + t.Errorf("catalogs = %d, want the one the backend found — degrading is not failing", len(answer.Catalogs)) } if !slices.Equal(degraded, []string{string(domain.CapabilitySemantic)}) { t.Errorf("degraded = %v, want [semantic]", degraded) @@ -181,7 +181,7 @@ func TestSpatialStillAnswersWithTextSearchOff(t *testing.T) { cfg := settings() cfg.Search.EnableTextSearch = false - catalogs, _, err := discover.NewService(repo, cfg).Discover( + answer, _, err := discover.NewService(repo, cfg).Discover( t.Context(), beckn.Context{}, beckn.Intent{Spatial: []beckn.SpatialConstraint{{ Op: beckn.OpSIntersects, @@ -192,8 +192,8 @@ func TestSpatialStillAnswersWithTextSearchOff(t *testing.T) { t.Fatalf("Discover: %v", err) } - if len(catalogs) != 1 { - t.Errorf("catalogs = %d, want the one the backend found", len(catalogs)) + if len(answer.Catalogs) != 1 { + t.Errorf("catalogs = %d, want the one the backend found", len(answer.Catalogs)) } if hasMode(repo.gotModes, domain.CapabilityLexical) { t.Errorf("modes = %v, want no ranked text mode — nothing asked for one", repo.gotModes) @@ -214,7 +214,7 @@ func TestAFilterABackendCannotRunIsDegradedRatherThanIgnored(t *testing.T) { result: domain.SearchResult{Catalogs: []domain.Catalog{{ID: "c1"}}}, } - catalogs, degraded, err := discover.NewService(repo, settings()).Discover( + answer, degraded, err := discover.NewService(repo, settings()).Discover( t.Context(), beckn.Context{}, beckn.Intent{Filters: &beckn.Filters{ Type: "jsonpath", Expression: `$.catalogs[*].resources[*] ? (@.grade == "A")`, @@ -224,8 +224,8 @@ func TestAFilterABackendCannotRunIsDegradedRatherThanIgnored(t *testing.T) { t.Fatalf("Discover: %v", err) } - if len(catalogs) != 1 { - t.Errorf("catalogs = %d, want the one the backend found — degrading is not failing", len(catalogs)) + if len(answer.Catalogs) != 1 { + t.Errorf("catalogs = %d, want the one the backend found — degrading is not failing", len(answer.Catalogs)) } if !slices.Equal(degraded, []string{string(domain.CapabilityJSONPath)}) { t.Errorf("degraded = %v, want [jsonpath] — a filter this backend cannot run and "+ @@ -512,11 +512,12 @@ func TestARenderedCatalogCarriesItsDocumentsVerbatim(t *testing.T) { }}, }}}} - catalogs, _, err := discover.NewService(repo, settings()). + answer, _, err := discover.NewService(repo, settings()). Discover(t.Context(), beckn.Context{}, beckn.Intent{TextSearch: "wheat"}, discover.Page{}) if err != nil { t.Fatalf("Discover: %v", err) } + catalogs := answer.Catalogs if len(catalogs) != 1 || len(catalogs[0].Offers) != 1 { t.Fatalf("rendered = %+v, want one catalog carrying one offer", catalogs) } @@ -576,11 +577,12 @@ func TestNoOfferWhoseResourcesAreAllOffThePageIsRendered(t *testing.T) { t.Fatalf("seeding: %v", err) } - catalogs, _, err := discover.NewService(repo, settings()). + answer, _, err := discover.NewService(repo, settings()). Discover(t.Context(), beckn.Context{}, beckn.Intent{TextSearch: "wheat"}, discover.Page{}) if err != nil { t.Fatalf("Discover: %v", err) } + catalogs := answer.Catalogs if len(catalogs) != 1 || len(catalogs[0].Resources) != 1 || catalogs[0].Resources[0].ID != "wheat" { t.Fatalf("rendered = %+v, want only the matched resource", catalogs) @@ -657,11 +659,12 @@ func TestACatalogWithAnUnreadableDocumentIsDropped(t *testing.T) { Document: json.RawMessage(`not json`), }}}} - catalogs, _, err := discover.NewService(repo, settings()). + answer, _, err := discover.NewService(repo, settings()). Discover(t.Context(), beckn.Context{}, beckn.Intent{TextSearch: "wheat"}, discover.Page{}) if err != nil { t.Fatalf("Discover: %v", err) } + catalogs := answer.Catalogs if len(catalogs) != 0 { t.Errorf("catalogs = %+v, want none", catalogs) } @@ -679,11 +682,12 @@ func TestAResourceWithAnUnreadableDocumentIsDropped(t *testing.T) { }, }}}} - catalogs, _, err := discover.NewService(repo, settings()). + answer, _, err := discover.NewService(repo, settings()). Discover(t.Context(), beckn.Context{}, beckn.Intent{TextSearch: "wheat"}, discover.Page{}) if err != nil { t.Fatalf("Discover: %v", err) } + catalogs := answer.Catalogs if len(catalogs) != 1 || len(catalogs[0].Resources) != 1 || catalogs[0].Resources[0].ID != "r2" { t.Fatalf("resources = %+v, want only r2 — r1's document could not be read", catalogs[0].Resources) } @@ -700,11 +704,12 @@ func TestAnOfferWithAnUnreadableDocumentIsDropped(t *testing.T) { }, }}}} - catalogs, _, err := discover.NewService(repo, settings()). + answer, _, err := discover.NewService(repo, settings()). Discover(t.Context(), beckn.Context{}, beckn.Intent{TextSearch: "wheat"}, discover.Page{}) if err != nil { t.Fatalf("Discover: %v", err) } + catalogs := answer.Catalogs if len(catalogs) != 1 || len(catalogs[0].Offers) != 1 || catalogs[0].Offers[0].ID != "o2" { t.Fatalf("offers = %+v, want only o2 — o1's document could not be read", catalogs[0].Offers) } @@ -808,3 +813,32 @@ func TestAMalformedFilterExpressionIsTheCallersFaultNotAFiveHundred(t *testing.T "internal that must not reach the caller", fault.Message, want) } } + +// The answer carries the page the service APPLIED — the default when none was +// asked for, the clamp when too much was — beside the backend's total. A caller +// that sent limit 5000 must read 100 back, or it will compute the next offset +// from a page size it never got. +func TestTheAnswerReportsTheAppliedPageAndTheBackendsTotal(t *testing.T) { + total := 42 + repo := &stubRepo{capabilities: everything(), result: domain.SearchResult{Total: &total}} + cfg := settings() + + answer, _, err := discover.NewService(repo, cfg).Discover( + t.Context(), beckn.Context{}, beckn.Intent{TextSearch: "wheat"}, + discover.Page{Limit: 5000, Offset: 20}) + if err != nil { + t.Fatalf("Discover: %v", err) + } + + page := answer.Pagination + if page == nil { + t.Fatal("pagination is absent") + } + if page.Limit != cfg.Search.MaxPageSize || page.Offset != 20 { + t.Errorf("pagination = limit %d offset %d, want the clamped %d and 20", + page.Limit, page.Offset, cfg.Search.MaxPageSize) + } + if page.Total == nil || *page.Total != total { + t.Errorf("total = %v, want the backend's %d", page.Total, total) + } +} diff --git a/src/domain/query.go b/src/domain/query.go index f6270e4..c9938f0 100644 --- a/src/domain/query.go +++ b/src/domain/query.go @@ -125,11 +125,17 @@ type SearchQuery struct { // SearchResult is one page of an answered query. // // Degraded names the retrieval modes that did not contribute and reaches the -// caller as the X-Beckn-Degraded header, not a body key (C11). There is no -// Total (A19). +// caller as the X-Beckn-Degraded header, not a body key (C11). +// +// Total is the size of the fused candidate set the page was sliced from, and +// nil whenever that size is not the number that matched — when a retrieval mode +// returned as many ids as its cap, it may have stopped early (A28). It is never +// a count query: A19 deleted that for costing 100x the search it accompanied, +// and this is the number the search already holds. type SearchResult struct { Catalogs []Catalog Degraded []string + Total *int } // Scope is the request-wide gate every retrieval mode reads (A6). diff --git a/src/platform/config/config.go b/src/platform/config/config.go index fef62cc..2a9ad7e 100644 --- a/src/platform/config/config.go +++ b/src/platform/config/config.go @@ -219,19 +219,24 @@ type Validation struct { // beckn.yaml is fetched at boot rather than baked into the image, so a // deployment names the document it validates against. // - // The default is a TAG, and that is the whole reason there can be one. This - // field carried no default until 2026-09-09, on the argument that which spec - // a network trusts is not a repository-wide decision — true of a branch, + // The default is IMMUTABLE, and that is the whole reason there can be one. + // This field carried no default until 2026-09-09, on the argument that which + // spec a network trusts is not a repository-wide decision — true of a branch, // which would let an upstream merge change the validator under a running - // deployment without a deploy. refs/tags/core-v2.0.0-lts cannot move, so the - // default names one immutable document and a network that trusts a different - // one still overrides it. tests/testdata/beckn-v2.0.0.yaml is byte-identical - // to it, so the fetch path and an air-gapped mount agree. + // deployment without a deploy. A tag or a commit cannot move, so the default + // names one document and a network that trusts a different one still + // overrides it. tests/testdata/beckn-v2.0.0.yaml is byte-identical to it, so + // the fetch path and an air-gapped mount agree. + // + // It is the OAN profile at a COMMIT, not beckn's core-v2.0.0-lts tag (A28): + // the profile adds limit/offset to DiscoverAction and pagination to + // OnDiscoverAction, and its branch carries no tag yet. Swap the SHA for a + // tag when one is cut — never for the branch name. // // The cost is that boot now makes a network call before falling back to // SpecCachePath. LoadSpecIndex already warns loudly and falls back, so an // offline boot is slower and noisier, not broken. - SpecURL string `env:"VALIDATION_SPEC_URL" envDefault:"https://raw.githubusercontent.com/beckn/protocol-specifications-v2/refs/tags/core-v2.0.0-lts/api/v2.0.0/beckn.yaml"` + SpecURL string `env:"VALIDATION_SPEC_URL" envDefault:"https://raw.githubusercontent.com/OpenAgriNet/protocol-specifications-v2/dfc109759b3781fde74539adc79df63fccbadb83/api/v2.0.0/beckn.yaml"` // Where the fetched spec is cached, and what an air-gapped deployment // mounts in place of the fetch. @@ -633,9 +638,19 @@ func validateDatabase(database Database) error { ) } +// schemaMaxPageSize is PaginationParams.limit's `maximum` in the OAN profile of +// the spec (A28). A copy rather than a read of the document, because config +// validates before the spec index is built; the two drifting apart shows up as +// a boot refusal here or as a schema failure on the response, never silently. +const schemaMaxPageSize = 100 + func validateSearch(search Search) error { return errors.Join( require(search.DefaultPageSize > 0, "search.defaultPageSize %d is not positive", search.DefaultPageSize), + require(search.MaxPageSize <= schemaMaxPageSize, + "search.maxPageSize %d is above the schema's limit maximum of %d — a body asking for more is refused "+ + "before the clamp, and a query-string caller would be answered with a pagination.limit that fails its own schema", + search.MaxPageSize, schemaMaxPageSize), require(search.MaxPageSize >= search.DefaultPageSize, "search.maxPageSize %d is below search.defaultPageSize %d — a ceiling under the value it clamps", search.MaxPageSize, search.DefaultPageSize), diff --git a/src/platform/config/config_test.go b/src/platform/config/config_test.go index 6e93eb5..ac33bf2 100644 --- a/src/platform/config/config_test.go +++ b/src/platform/config/config_test.go @@ -377,6 +377,13 @@ func TestValidateRejects(t *testing.T) { "a candidate pool smaller than one page": { "search:\n maxPageSize: 100\n maxCandidatesPerMode: 50\n", "maxCandidatesPerMode", }, + // The schema caps a page at 100 (A28). Above it, a body asking for more + // is refused before the clamp is reached, and a query-string caller is + // answered with a pagination.limit that fails the response's own + // schema. + "a max page size above the schema's maximum": { + "search:\n maxPageSize: 101\n maxCandidatesPerMode: 500\n", "maxPageSize", + }, "an unloadable timezone": { "app:\n defaultTimezone: Mars/Olympus_Mons\n", "defaultTimezone", }, diff --git a/src/platform/validation/spec_index.go b/src/platform/validation/spec_index.go index eb7cc84..3368ef2 100644 --- a/src/platform/validation/spec_index.go +++ b/src/platform/validation/spec_index.go @@ -241,7 +241,7 @@ func loadFromCache(path string) (*SpecIndex, error) { // ages with the image), so an air-gapped deploy supplies one at // VALIDATION_SPEC_CACHE_PATH, and mounting it read-only is the right way to // supply a file the service must not be able to rewrite. docker-compose.yml does -// exactly that with a copy byte-identical to the core-v2.0.0-lts tag, and every +// exactly that with a copy byte-identical to VALIDATION_SPEC_URL's default, and every // boot with a reachable registry warned that it could not overwrite those bytes // with themselves. A warning that fires when nothing is wrong is one nobody reads // on the boot when something is. diff --git a/src/storage/conformance/discover.go b/src/storage/conformance/discover.go index 5d97b95..2041a33 100644 --- a/src/storage/conformance/discover.go +++ b/src/storage/conformance/discover.go @@ -5,6 +5,7 @@ import ( "errors" "fmt" "slices" + "strconv" "testing" "time" @@ -214,6 +215,15 @@ func pageOrder(result domain.SearchResult) []string { return ids } +// totalOf renders a result's total for a failure message, where a nil pointer +// would print as an address. +func totalOf(result domain.SearchResult) string { + if result.Total == nil { + return "absent" + } + return strconv.Itoa(*result.Total) +} + // offerIDs is every offer hydrated onto the page, sorted. func offerIDs(result domain.SearchResult) []string { ids := make([]string, 0) @@ -642,10 +652,17 @@ func pageTwoDoesNotOverlapPageOne() Case { t.Errorf("page two is %v, want [r-02 r-03]", got) } - // There is no total to assert (A19). What remains is the property - // a caller walking pages actually depends on: the offset SLICES a - // stable order rather than re-ranking it, so page two holds the - // next two and not two of the same. + // The offset SLICES a stable order rather than re-ranking it, so + // page two holds the next two and not two of the same. + + // And both pages report the same exact total (A28): five matched, + // none of them capped, so the count is the size of the set being + // paged — whichever page asked. + for name, result := range map[string]domain.SearchResult{"one": first, "two": second} { + if result.Total == nil || *result.Total != len(resources) { + t.Errorf("page %s reports total %v, want %d", name, totalOf(result), len(resources)) + } + } }, } } diff --git a/src/storage/memory/search.go b/src/storage/memory/search.go index 1063c9e..e62efb0 100644 --- a/src/storage/memory/search.go +++ b/src/storage/memory/search.go @@ -62,9 +62,13 @@ func (r *Repository) Search( page = nil } + // This backend retrieves without a cap, so the ranked list is every match + // and its length is always the exact total. + total := len(page) return domain.SearchResult{ Catalogs: r.hydrate(pageOf(page, query.Offset, query.Limit), scope), Degraded: degraded, + Total: &total, }, nil } diff --git a/src/storage/postgres/read_path_test.go b/src/storage/postgres/read_path_test.go index aee05d2..cb54192 100644 --- a/src/storage/postgres/read_path_test.go +++ b/src/storage/postgres/read_path_test.go @@ -380,6 +380,33 @@ func TestPageTwoDoesNotOverlapPageOne(t *testing.T) { } } +// A mode that returned as many ids as its cap may have stopped early, so the +// fused list is a truncation and its length is not the number that matched. +// The total is then left OUT (A28) rather than reported short — this is the +// only backend with a cap, which is why the case lives here and not in the +// conformance suite. searchConfig's cap is 8; ten lots match. +func TestATotalIsReportedOnlyWhenNoModeHitItsCap(t *testing.T) { + repository, _ := publish(t, nil, readFixture{catalog: "cat-capped", resources: kharifLots(10)}) + + capped := search(t, repository, domain.SearchQuery{Text: "kharif", Limit: 4}) + if capped.Total != nil { + t.Errorf("total = %d with every mode at its cap of %d, want it absent", + *capped.Total, searchConfig().MaxCandidatesPerMode) + } +} + +func TestATotalUnderTheCapIsExact(t *testing.T) { + repository, _ := publish(t, nil, readFixture{catalog: "cat-counted", resources: kharifLots(6)}) + + result := search(t, repository, domain.SearchQuery{Text: "kharif", Limit: 4}) + if result.Total == nil || *result.Total != 6 { + t.Errorf("total = %v, want exactly 6", result.Total) + } + if got := len(matchedIDs(result)); got != 4 { + t.Errorf("the page holds %d, want 4 — the total counts the set, not the page", got) + } +} + // A mode that ERRORS is degraded, and the page is what the other modes found. // // The vectors are 768 wide because the column is, and the repository embeds at diff --git a/src/storage/postgres/search_repository.go b/src/storage/postgres/search_repository.go index 8ba65af..8375759 100644 --- a/src/storage/postgres/search_repository.go +++ b/src/storage/postgres/search_repository.go @@ -128,7 +128,29 @@ func (s *SearchRepository) Search( if err != nil { return domain.SearchResult{}, err } - return domain.SearchResult{Catalogs: catalogs, Degraded: degraded}, nil + return domain.SearchResult{ + Catalogs: catalogs, + Degraded: degraded, + Total: s.exactTotal(lists, fused), + }, nil +} + +// exactTotal is len(fused) when that is the number that matched, and nil when +// it is not (A28). +// +// A list as long as MaxCandidatesPerMode may have been cut by its LIMIT, so the +// fusion of it is a truncation. A list of exactly the cap that happened to be +// complete is reported as uncounted too — the one false negative, and the safe +// direction: an absent total says "narrow the query", a short one says "this is +// everything" when it is not. +func (s *SearchRepository) exactTotal(lists [][]string, fused []string) *int { + for _, ids := range lists { + if len(ids) >= s.search.MaxCandidatesPerMode { + return nil + } + } + total := len(fused) + return &total } // withinRetrievalDepth refuses a page past what any mode retrieves. Checked diff --git a/tests/acceptance/pagination_test.go b/tests/acceptance/pagination_test.go new file mode 100644 index 0000000..a11ca6c --- /dev/null +++ b/tests/acceptance/pagination_test.go @@ -0,0 +1,98 @@ +package acceptance + +import ( + "fmt" + "net/http" + "slices" + "testing" + + "github.com/OpenAgriNet/discovery-service/src/beckn" +) + +// fiveAtMajestic publishes five resources a spatial intent around Majestic +// matches, so a page size of two has to walk three pages to see them all. +func fiveAtMajestic(t *testing.T) *service { + t.Helper() + + list := make([]map[string]any, 0, 5) + for index := range 5 { + list = append(list, aResource(fmt.Sprintf("r-%d", index), fmt.Sprintf("Lot %d", index))) + } + svc := newService(t) + svc.publishCatalogs(t, aCatalog("c-lots", availableAt(majestic), resources(list...))) + return svc +} + +// pageAt asks for one page with limit and offset in the SIGNED body (A28), and +// returns the message so the scenario can read the pagination beside the +// catalogs. +func (s *service) pageAt(t *testing.T, limit, offset int) beckn.OnDiscoverAction { + t.Helper() + + answer := s.post(t, "/discover", envelope(beckn.ActionDiscover, map[string]any{ + "intent": spatial(dwithin(providerGeoPath, majestic, 5000)), + "limit": limit, + "offset": offset, + })) + if answer.status != http.StatusOK { + t.Fatalf("POST /discover limit %d offset %d = %d, want 200 — the OAN profile "+ + "declares both on DiscoverAction\nbody: %s", limit, offset, answer.status, answer.body) + } + + var body struct { + Message beckn.OnDiscoverAction `json:"message"` + } + answer.decode(t, &body) + return body.Message +} + +// A caller walking pages in the body reaches every match exactly once, and +// every page says where it is: the page it applied, and a total that tells the +// last page from a short one. Before A28 the body could not carry the page at +// all — the schema refused the key — and the answer carried nothing but +// catalogs, so a short page and the end were the same thing. +func TestTheBodyPagesThroughEveryMatchAndTheTotalMarksTheEnd(t *testing.T) { + svc := fiveAtMajestic(t) + + var seen []string + for offset := 0; offset < 5; offset += 2 { + message := svc.pageAt(t, 2, offset) + + page := message.Pagination + if page == nil { + t.Fatalf("offset %d: no pagination in the answer", offset) + } + if page.Limit != 2 || page.Offset != offset { + t.Errorf("offset %d: pagination reports limit %d offset %d, want 2 and %d", + offset, page.Limit, page.Offset, offset) + } + if page.Total == nil || *page.Total != 5 { + t.Errorf("offset %d: total = %v, want 5", offset, page.Total) + } + seen = append(seen, resourceIDs(message.Catalogs)...) + } + + slices.Sort(seen) + if want := []string{"r-0", "r-1", "r-2", "r-3", "r-4"}; !slices.Equal(seen, want) { + t.Errorf("walking the pages saw %v, want each of %v exactly once", seen, want) + } +} + +// The schema bounds the body's page, so a limit above its maximum is refused +// by L1 validation before the service clamps anything. The query string still +// clamps — it is outside the schema — which is one more reason it is the +// deprecated route. +func TestALimitAboveTheSchemaMaximumIsRefused(t *testing.T) { + svc := fiveAtMajestic(t) + + answer := svc.post(t, "/discover", envelope(beckn.ActionDiscover, map[string]any{ + "intent": spatial(dwithin(providerGeoPath, majestic, 5000)), + "limit": 101, + })) + if answer.status != http.StatusBadRequest { + t.Fatalf("limit 101 answered %d, want 400\nbody: %s", answer.status, answer.body) + } + if code := answer.nack(t).Message.Error.Code; code != beckn.CodeSchemaValidationFailed { + t.Errorf("the refusal is %s, want %s", code, beckn.CodeSchemaValidationFailed) + } +} diff --git a/tests/acceptance/suite_test.go b/tests/acceptance/suite_test.go index fecdd7f..701d7f0 100644 --- a/tests/acceptance/suite_test.go +++ b/tests/acceptance/suite_test.go @@ -20,7 +20,6 @@ import ( "net/http" "net/http/httptest" "reflect" - "strconv" "strings" "testing" "time" @@ -351,10 +350,9 @@ func (s *service) discoverOn(t *testing.T, networkID string, intent map[string]a // discoverPaged is discover with an explicit page size. // -// The page is a query parameter rather than a member of the intent: `limit` and -// `offset` are read off the URL, which is the one part of the transport the -// plan does not spell out. Named here rather than in each scenario so that if -// it moves into the body there is one call site to move. +// The page is a member of DiscoverAction, beside the intent rather than inside +// it (A28). It moved there from the query string, and this was the one call +// site to move. // // Two scenarios need it because the default page is twenty and their fixtures // are forty: a scenario asserting on a count would otherwise be asserting on @@ -364,12 +362,12 @@ func (s *service) discoverPaged( ) []beckn.Catalog { t.Helper() - body := envelope(beckn.ActionDiscover, map[string]any{"intent": intent}) + body := envelope(beckn.ActionDiscover, map[string]any{"intent": intent, "limit": limit}) if networkID != "" { onNetwork(networkID)(nested(body, "context")) } - answer := s.post(t, "/discover?limit="+strconv.Itoa(limit), body) + answer := s.post(t, "/discover", body) if answer.status != http.StatusOK { t.Fatalf("POST /discover = %d, want 200\nbody: %s", answer.status, answer.body) } diff --git a/tests/testdata/beckn-v2.0.0.yaml b/tests/testdata/beckn-v2.0.0.yaml index 0a57179..72a8386 100644 --- a/tests/testdata/beckn-v2.0.0.yaml +++ b/tests/testdata/beckn-v2.0.0.yaml @@ -2290,6 +2290,10 @@ components: properties: intent: $ref: '#/components/schemas/Intent' + limit: + $ref: '#/components/schemas/PaginationParams/properties/limit' + offset: + $ref: '#/components/schemas/PaginationParams/properties/offset' additionalProperties: false required: - intent @@ -2375,6 +2379,8 @@ components: Array of catalogs matching the discovery request.' items: $ref: '#/components/schemas/Catalog' + pagination: + $ref: '#/components/schemas/Pagination' additionalProperties: false OnInitAction: @@ -2983,12 +2989,22 @@ components: type: string format: uuid networkId: - description: "The unique identifier of the Beckn subnet within which this\ - \ transaction is occurring. Actors use this value to resolve the subnet's\ - \ registry URL before performing identity and address lookups. The RECOMMENDED\ - \ format is namespace_id followed by a slash followed by registry_id, for\ - \ example acmenet.org/charge-net." - type: string + description: "The unique identifier(s) of the Beckn subnet(s) within which\ + \ this transaction is occurring. Actors use this value to resolve the\ + \ subnet's registry URL before performing identity and address lookups.\ + \ The RECOMMENDED format is namespace_id followed by a slash followed\ + \ by registry_id, for example acmenet.org/charge-net.\n\nMAY be a single\ + \ string (single-network transaction) or an array of strings (multi-network\ + \ transaction). When an array is provided, order is significant: actors\ + \ MUST attempt registry resolution starting with the first entry, treating\ + \ it as the primary network, and MAY fall back to subsequent entries if\ + \ resolution against the primary fails." + oneOf: + - type: string + - type: array + items: + type: string + minItems: 1 timestamp: description: The date and time at which this message was generated by the sender, expressed in RFC 3339 date-time format. The receiving actor MAY @@ -4500,36 +4516,50 @@ components: ISO 8601 timestamp set by the CS each time the subscription record is modified. - Pagination: + PaginationParams: type: object - description: "Metadata accompanying a paginated list response that describes\ - \ the size, position, and count of the returned result set, enabling callers\ - \ to navigate through large result sets using offset-based pagination.\n\n\ - Pagination appears in list responses such as /catalog/subscription and /catalog/search.\ - \ The DS (Discovery Service) and CS (Cataloging Service) produce it; the CN\ - \ or PN consumes it to determine whether additional pages are available and\ - \ to construct subsequent requests.\n\nRelationship: Composed within the response\ - \ message objects of /catalog/subscription and /catalog/search endpoints." + description: >- + Reusable offset-based pagination fields shared between paginated + list requests and the Pagination response metadata that describes + them. On a request, limit/offset specify the caller's desired page; + on a response, they reflect the page actually applied by the server. properties: - total: - type: integer - minimum: 0 - description: >- - Total number of matching records across the entire result set, - not the count in this page. Assigned by the CS. limit: type: integer minimum: 1 + maximum: 100 description: >- - Maximum number of records returned in this page. Assigned by - the CS based on the request parameter or CS policy. + Maximum number of records in this page. On a request, the + caller's desired page size; if omitted, the server applies a + default. On a response, the page size actually applied. offset: type: integer minimum: 0 + default: 0 description: >- Zero-based index of the first record in this page relative to - the full result set. Assigned by the CS based on the request - parameter. + the full result set. On a request, specifies where to start; + on a response, reflects the offset actually applied. + + Pagination: + description: "Metadata accompanying a paginated list response that describes\ + \ the size, position, and count of the returned result set, enabling callers\ + \ to navigate through large result sets using offset-based pagination.\n\n\ + Pagination appears in list responses such as /catalog/subscription and /catalog/search.\ + \ The DS (Discovery Service) and CS (Cataloging Service) produce it; the CN\ + \ or PN consumes it to determine whether additional pages are available and\ + \ to construct subsequent requests.\n\nRelationship: Composed within the response\ + \ message objects of /catalog/subscription and /catalog/search endpoints." + allOf: + - $ref: '#/components/schemas/PaginationParams' + - type: object + properties: + total: + type: integer + minimum: 0 + description: >- + Total number of matching records across the entire result + set, not the count in this page. Assigned by the CS. CatalogSearchAction: type: object From b606328018fe23c9100c43bfea4cd18eb01cd1fa Mon Sep 17 00:00:00 2001 From: nisargabd Date: Thu, 15 Oct 2026 15:49:20 +0530 Subject: [PATCH 09/23] fix: bump golang.org/x/net off the CVE-affected version [#53] x/net v0.58.0 carries CVE-2026-78669 (HIGH), CVE-2026-78663 (MEDIUM) and CVE-2026-97032 (MEDIUM), all fixed in v0.60.0. trivy-gate fails on them, so every PR into development is blocked until they clear. x/net is indirect, pulled in by grpc for the OTLP gRPC exporters. MVS moves the rest of golang.org/x with it: crypto v0.57.0, sync v0.23.0, sys v0.48.0, text v0.42.0. crypto is the one direct use, blake2b only, and its API is unchanged. grpc and the go directive stay where they are. --- go.mod | 10 +++++----- go.sum | 24 ++++++++++++------------ 2 files changed, 17 insertions(+), 17 deletions(-) diff --git a/go.mod b/go.mod index e251d1d..f84e6f2 100644 --- a/go.mod +++ b/go.mod @@ -27,7 +27,7 @@ require ( go.opentelemetry.io/otel/sdk/metric v1.45.0 go.opentelemetry.io/otel/trace v1.45.0 go.uber.org/zap v1.27.0 - golang.org/x/crypto v0.56.0 + golang.org/x/crypto v0.57.0 gopkg.in/yaml.v3 v3.0.1 ) @@ -87,10 +87,10 @@ require ( go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp v0.69.0 // indirect go.opentelemetry.io/proto/otlp v1.11.0 // indirect go.uber.org/multierr v1.10.0 // indirect - golang.org/x/net v0.58.0 // indirect - golang.org/x/sync v0.22.0 // indirect - golang.org/x/sys v0.47.0 // indirect - golang.org/x/text v0.41.0 // indirect + golang.org/x/net v0.60.0 // indirect + golang.org/x/sync v0.23.0 // indirect + golang.org/x/sys v0.48.0 // indirect + golang.org/x/text v0.42.0 // indirect google.golang.org/genproto/googleapis/api v0.0.0-20260803160001-6ac0973c030d // indirect google.golang.org/genproto/googleapis/rpc v0.0.0-20260803160001-6ac0973c030d // indirect google.golang.org/grpc v1.83.2 // indirect diff --git a/go.sum b/go.sum index 79143cc..d395013 100644 --- a/go.sum +++ b/go.sum @@ -203,22 +203,22 @@ go.uber.org/multierr v1.10.0 h1:S0h4aNzvfcFsC3dRF1jLoaov7oRaKqRGC/pUEJ2yvPQ= go.uber.org/multierr v1.10.0/go.mod h1:20+QtiLqy0Nd6FdQB9TLXag12DsQkrbs3htMFfDN80Y= go.uber.org/zap v1.27.0 h1:aJMhYGrd5QSmlpLMr2MftRKl7t8J8PTZPA732ud/XR8= go.uber.org/zap v1.27.0/go.mod h1:GB2qFLM7cTU87MWRP2mPIjqfIDnGu+VIO4V/SdhGo2E= -golang.org/x/crypto v0.56.0 h1:GUh5Ii4J5jtcseSMiRqr1jXCNHoxjeV9Fmekc2oLy6Y= -golang.org/x/crypto v0.56.0/go.mod h1:OMW5y6CY9l38uPLmxU6l6pwcXp1obtLo3e6gT7gQR2I= -golang.org/x/net v0.58.0 h1:ynWG7rqYi4ccpTEuPZ2QGWHktVEM9DMCj9yzDE0Q7To= -golang.org/x/net v0.58.0/go.mod h1:YwCddHnFlT7eLQqVprV19OnhLGtc5xOKgE0RyqgfWAU= -golang.org/x/sync v0.22.0 h1:SZjpbeLmrCk4xhRSZFNZW5gFUeCeFgjekvI/+gfScek= -golang.org/x/sync v0.22.0/go.mod h1:9xrNwdLfx4jkKbNva9FpL6vEN7evnE43NNNJQ2LF3+0= +golang.org/x/crypto v0.57.0 h1:3ZVCjf8Ggz7zneR/EHRVx68Ctf+2pmIMP2UFhh9cC6M= +golang.org/x/crypto v0.57.0/go.mod h1:Fdz0i5U6CoizGwLda9DttjSk6qlZo25zYNtR+ycvuZA= +golang.org/x/net v0.60.0 h1:79p50tfZlm0J9YfoDsSi639qSXNGVwEzOPLCxM2FsYU= +golang.org/x/net v0.60.0/go.mod h1:2DA/G1UfVbCpQPeWTmMPGY7Cs2PkBkwu743bVX5PIVg= +golang.org/x/sync v0.23.0 h1:KameEIfc1IkluZyXWLn39Wd4tURc6GbCiISGiZm2bQk= +golang.org/x/sync v0.23.0/go.mod h1:sUUOizhqBxiL6pEWpqNLUiaJn1ShEbZ6BBqskPbjZm0= golang.org/x/sys v0.0.0-20190916202348-b4ddaad3f8a3/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= golang.org/x/sys v0.0.0-20201204225414-ed752295db88/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= golang.org/x/sys v0.0.0-20210616094352-59db8d763f22/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= golang.org/x/sys v0.1.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= -golang.org/x/sys v0.47.0 h1:o7XGOvZQCADBQQ4Y7VNq2dRWQR7JmOUW8Kxx4ZsNgWs= -golang.org/x/sys v0.47.0/go.mod h1:4GL1E5IUh+htKOUEOaiffhrAeqysfVGipDYzABqnCmw= -golang.org/x/term v0.45.0 h1:NwWyBmoJCbfTHpxrWoZ9C6/VxOf7ic219I8xZZFdrf0= -golang.org/x/term v0.45.0/go.mod h1:9aqxs0blBcrm/n0L9QW0aRVD+ktan8ssZromtqJC43w= -golang.org/x/text v0.41.0 h1:vz/seA0lnX87Othu2f/0L24RcgrXD9/YFTSuGjj3rH8= -golang.org/x/text v0.41.0/go.mod h1:jvf1O8ajNzZqhSrQBPbutR/EB83Cc0CFrezNQIwbb5M= +golang.org/x/sys v0.48.0 h1:bbX/i/6MgT9BVLM9RT1thmxL04yeTAhbEz4SyadbXoo= +golang.org/x/sys v0.48.0/go.mod h1:hNLxWAXmnKAxqDtdwIYC4bM9oQPEecfsnNMuSxOs3og= +golang.org/x/term v0.46.0 h1:3+OXuTbaKDgwk8jTi3aSLHRlmWqHEUDUtxnbFigO4YE= +golang.org/x/term v0.46.0/go.mod h1:+K02xbkittuwc0Am4abfA3Fc+XRGXkvBXNO88NCXPoc= +golang.org/x/text v0.42.0 h1:JbOZXgfeCPU9gacVtYliJqOhD+zhrEqK4LfdpmlUZqI= +golang.org/x/text v0.42.0/go.mod h1:ojzP1Z+2QtioaF8DTtO8K5q7JWVVYwZKenzujK0Zd0E= gonum.org/v1/gonum v0.17.0 h1:VbpOemQlsSMrYmn7T2OUvQ4dqxQXU+ouZFQsZOx50z4= gonum.org/v1/gonum v0.17.0/go.mod h1:El3tOrEuMpv2UdMrbNlKEh9vd86bmQ6vqIcDwxEOc1E= google.golang.org/genproto/googleapis/api v0.0.0-20260803160001-6ac0973c030d h1:FarXi840EJWSHYTN3ERkADbPWjl307+FGrA22KAVjjc= From 120834b0b6153a6e818b66c8337a25c24db1ef89 Mon Sep 17 00:00:00 2001 From: nisargabd Date: Mon, 19 Oct 2026 11:32:07 +0530 Subject: [PATCH 10/23] fix: bump kin-openapi off the CVE-affected version [#55] kin-openapi v0.147.0 carries CVE-2026-100352, -100353 and -100354, all HIGH. trivy-gate fails on them, so every PR into development is blocked. None is reachable here: we decode JSON and use the built-in visitor, never import openapi3filter, and neither spec has a circular allOf/anyOf/oneOf chain. v0.150.0, not the v0.149.0 Trivy names. v0.149.0's cycle guard stops validating self-referencing schemas below the root, so an invalid member of a GeometryCollection passes L1; v0.150.0 restores that. Pinning back to v0.149.0 reopens the hole. v0.150.0 also stops checking const in the built-in visitor. That changes no request outcome: context.version is enforced by the envelope rules, and context.action is rewritten to its canonical value before L1 runs. A differential run of 1,416 bodies through ValidateEnvelope and L1 gave the same verdict, message and path as v0.147.0, on both the current spec and #52's. Transitively santhosh-tekuri/jsonschema/v6 v6.0.2 -> v6.0.3. --- go.mod | 4 ++-- go.sum | 8 ++++---- 2 files changed, 6 insertions(+), 6 deletions(-) diff --git a/go.mod b/go.mod index f84e6f2..832feec 100644 --- a/go.mod +++ b/go.mod @@ -4,7 +4,7 @@ go 1.26.6 require ( github.com/caarlos0/env/v11 v11.4.1 - github.com/getkin/kin-openapi v0.147.0 + github.com/getkin/kin-openapi v0.150.0 github.com/golang-migrate/migrate/v4 v4.19.1 github.com/google/uuid v1.6.0 github.com/jackc/pgerrcode v0.0.0-20220416144525-469b46aa5efa @@ -75,7 +75,7 @@ require ( github.com/opencontainers/image-spec v1.1.1 // indirect github.com/pmezard/go-difflib v1.0.1-0.20181226105442-5d4384ee4fb2 // indirect github.com/power-devops/perfstat v0.0.0-20240221224432-82ca36839d55 // indirect - github.com/santhosh-tekuri/jsonschema/v6 v6.0.2 // indirect + github.com/santhosh-tekuri/jsonschema/v6 v6.0.3 // indirect github.com/shirou/gopsutil/v4 v4.26.6 // indirect github.com/sirupsen/logrus v1.9.4 // indirect github.com/stretchr/testify v1.11.1 // indirect diff --git a/go.sum b/go.sum index d395013..4c06a87 100644 --- a/go.sum +++ b/go.sum @@ -45,8 +45,8 @@ github.com/ebitengine/purego v0.10.1 h1:dewVBCBT2GaMu1SrNTYxQhgQBethzfhiwvZiLGP/ github.com/ebitengine/purego v0.10.1/go.mod h1:iIjxzd6CiRiOG0UyXP+V1+jWqUXVjPKLAI0mRfJZTmQ= github.com/felixge/httpsnoop v1.1.0 h1:3YtUj32ZZkqZtt3sZZsClsymw/QDuVfpNhoA31zeORc= github.com/felixge/httpsnoop v1.1.0/go.mod h1:Zqxgdd+1Rkcz8euOqdr7lqgCRJztwr5hp9vDSi5UZCE= -github.com/getkin/kin-openapi v0.147.0 h1:s+Xsm9gUMPJbgCnABZ2to3zSQQ5A9dyj/zo62VVsldY= -github.com/getkin/kin-openapi v0.147.0/go.mod h1:3BH9M9XDe/y9M5DSvEocVYAYq1w0qrhJHjC/vZi0AaY= +github.com/getkin/kin-openapi v0.150.0 h1:7+O1HMsrAB4BevQDElbLAeCAWwyJC/RjQJYw4oosUUU= +github.com/getkin/kin-openapi v0.150.0/go.mod h1:1+BHDzstro+P5CKtPy1X4PfofnFgmRe6uvMy9+r9fKY= github.com/go-logr/logr v1.2.2/go.mod h1:jdQByPbusPIv2/zmleS9BjJVeZ6kBagPoEUsqbVz/1A= github.com/go-logr/logr v1.4.4 h1:tG4xh9yMsRCAiodLVTxyrkzSZ9+o0L1Kg/+cPVcbP/8= github.com/go-logr/logr v1.4.4/go.mod h1:9T104GzyrTigFIr8wt5mBrctHMim0Nb2HLGrmQ40KvY= @@ -138,8 +138,8 @@ github.com/power-devops/perfstat v0.0.0-20240221224432-82ca36839d55 h1:o4JXh1EVt github.com/power-devops/perfstat v0.0.0-20240221224432-82ca36839d55/go.mod h1:OmDBASR4679mdNQnz2pUhc2G8CO2JrUAVFDRBDP/hJE= github.com/rogpeppe/go-internal v1.14.1 h1:UQB4HGPB6osV0SQTLymcB4TgvyWu6ZyliaW0tI/otEQ= github.com/rogpeppe/go-internal v1.14.1/go.mod h1:MaRKkUm5W0goXpeCfT7UZI6fk/L7L7so1lCWt35ZSgc= -github.com/santhosh-tekuri/jsonschema/v6 v6.0.2 h1:KRzFb2m7YtdldCEkzs6KqmJw4nqEVZGK7IN2kJkjTuQ= -github.com/santhosh-tekuri/jsonschema/v6 v6.0.2/go.mod h1:JXeL+ps8p7/KNMjDQk3TCwPpBy0wYklyWTfbkIzdIFU= +github.com/santhosh-tekuri/jsonschema/v6 v6.0.3 h1:1EYB5IzjZawrrnELUi78f9fPu57HuXjmddZPjrls/28= +github.com/santhosh-tekuri/jsonschema/v6 v6.0.3/go.mod h1:JXeL+ps8p7/KNMjDQk3TCwPpBy0wYklyWTfbkIzdIFU= github.com/shirou/gopsutil/v4 v4.26.6 h1:Mzr/npDtQC/xpeEuQKHZt8Zo9CmPvhTj8nkR8w5TLDs= github.com/shirou/gopsutil/v4 v4.26.6/go.mod h1:LZ6ewCSkBqUpvSOf+LsTGnRinC6iaNUNMGBtDkJBaLQ= github.com/sirupsen/logrus v1.9.4 h1:TsZE7l11zFCLZnZ+teH4Umoq5BhEIfIzfRDZ1Uzql2w= From 8724d06ba8694e4009761b3fd824897534839d08 Mon Sep 17 00:00:00 2001 From: Prajwal TD <157502711+tdprajwal@users.noreply.github.com> Date: Fri, 23 Oct 2026 11:41:04 +0530 Subject: [PATCH 11/23] feat: read discover pagination from the request body [#51] Pagination now comes only from message.limit and message.offset; the query string is no longer read. The response returns the applied page as message.pagination. --- README.md | 12 +-- docker-compose.yml | 5 +- docs/design/implementation-plan.md | 11 +-- docs/publish-and-discover.md | 9 +- docs/quickstart.md | 10 +- examples/verify.sh | 11 ++- src/beckn/actions.go | 20 ++-- src/discover/controller.go | 55 +---------- src/discover/controller_test.go | 113 ++-------------------- src/discover/intent_mapper.go | 7 +- src/discover/intent_mapper_test.go | 4 - src/discover/service.go | 4 +- src/discover/service_test.go | 54 +++-------- src/domain/query.go | 10 +- src/platform/config/config.go | 29 ++---- src/platform/config/config_test.go | 7 -- src/platform/validation/spec_index.go | 2 +- src/storage/conformance/discover.go | 25 +---- src/storage/memory/search.go | 4 - src/storage/postgres/read_path_test.go | 27 ------ src/storage/postgres/search_repository.go | 24 +---- tests/acceptance/pagination_test.go | 98 ------------------- tests/acceptance/suite_test.go | 4 +- 23 files changed, 74 insertions(+), 471 deletions(-) delete mode 100644 tests/acceptance/pagination_test.go diff --git a/README.md b/README.md index 25ae2f5..db5c19e 100644 --- a/README.md +++ b/README.md @@ -69,9 +69,8 @@ That stack applies its own migrations (they are compiled into the binary) and reads the Beckn specification from `tests/testdata/beckn-v2.0.0.yaml`, mounted at the cache path. The boot logs one warning about the registry fetch it did not do, which is why it works with no network. That file is byte-identical to -the OAN profile of the spec at commit `dfc1097` of -`OpenAgriNet/protocol-specifications-v2` (A28); set `VALIDATION_SPEC_URL` to its -raw URL to exercise the fetch path instead. +the `core-v2.0.0-lts` tag of `beckn/protocol-specifications-v2`; set +`VALIDATION_SPEC_URL` to its raw URL to exercise the fetch path instead. With the stack up, there is a worked catalog and the requests that find it: @@ -119,10 +118,9 @@ overrides; that file is gitignored. **Secrets — `DATABASE_URL` above all — belong in neither YAML file.** The Beckn specification is fetched at boot from `VALIDATION_SPEC_URL` and -cached under `.cache/beckn/`. It defaults to the OAN profile at commit -`dfc1097` of `OpenAgriNet/protocol-specifications-v2` (A28) — a commit rather -than a branch, so an upstream merge cannot change the validator under a running -deployment. It is not baked +cached under `.cache/beckn/`. It defaults to the `core-v2.0.0-lts` tag of +`beckn/protocol-specifications-v2` — a tag rather than a branch, so an upstream +merge cannot change the validator under a running deployment. It is not baked into the image; air-gapped deploys mount a cache file at `VALIDATION_SPEC_CACHE_PATH`. diff --git a/docker-compose.yml b/docker-compose.yml index 8316f1b..99d9297 100644 --- a/docker-compose.yml +++ b/docker-compose.yml @@ -81,9 +81,8 @@ services: APP_DOMAIN: ${APP_DOMAIN:-Agriculture} volumes: # The Beckn spec, which the service refuses to boot without. Byte-identical - # to the OAN profile at commit dfc1097 of - # OpenAgriNet/protocol-specifications-v2 (A28), which is also - # VALIDATION_SPEC_URL's default — so a normal boot fetches, compiles + # to the core-v2.0.0-lts tag of beckn/protocol-specifications-v2, which is + # also VALIDATION_SPEC_URL's default — so a normal boot fetches, compiles # and finds this file already holding the same bytes, and an offline boot # falls back to it with one accurate warning about the fetch that failed. # diff --git a/docs/design/implementation-plan.md b/docs/design/implementation-plan.md index f875b66..fb4fcc7 100644 --- a/docs/design/implementation-plan.md +++ b/docs/design/implementation-plan.md @@ -151,7 +151,6 @@ field carrying **an array of network ids** (not `PUBLIC`/`PRIVATE`). | **A25** | **This service is one of four layers, and the plan was written as if it were all of them. Tasks 25 and 26 are added, and 23a and 23c grow one requirement each.** The other three layers — experience, network, provider — run **beckn-onix** adapters that already emit OTel traces, metrics and audit logs, and reading them changes four things here. **Two questions this plan could not answer are answered elsewhere, not nowhere.** Relevance and accuracy need what the user did *after* the response; the experience-layer adapter has the user and already emits traces, so that is where the outcome event belongs. Performance per location and per commodity is refused here because coordinates and `filters.expression` are on the never-emitted list — but the provider adapter holds both as first-class call-plan fields, so it answers them without anyone widening a deny-list. The H3-coarsening policy call that `opentelemetry.md` raised against 23d is **withdrawn** on those grounds. **Our spans would currently be exported and then silently dropped.** onix's network trace pipeline admits only spans carrying `sender.id`, which this phase makes optional and usually absent, while the stage downstream of it rebuilds `trace_id` from an attribute spelled `transaction_id`, which we planned to spell `beckn.transactionId`. A span that fails either is stored, paid for and invisible, with nothing anywhere reporting an error. The filter is onix's bug to fix — it admits on one key and joins on another — and the spelling is ours: **23c emits `transaction_id` and `message_id` as aliases**, written from the same values at the same call site so they cannot drift. Two, not three — an earlier draft named `receiver.id`, which onix puts only on audit log records; its span spelling is `recipient.id`, which we already emit, so a third alias would be paid for everywhere and read nowhere. Buying admission by inventing a `sender.id` is explicitly refused; missing from a dashboard is recoverable, poisoning the network's only participant-identity join is not. **Metrics were dropped from this service for a reason that covers only half the ground.** A23 removed `metrics.go` because a stateless replica's in-memory counter is a partial nobody can reassemble — true of the facilitator's windowed aggregates, and the reason Task 24 exists. It is not true of a *level*: three configured ceilings (`DATABASE_MAX_CONNS`, the rate limiter, the body size) have no observable distance-to-limit, and a pool at 30 of 32 refuses nothing, so it produces no span to be seen in. An earlier draft of this amendment said a 429 "produces no span, no event and no counter". **That is false, and the correction matters because it is what the task's scope was sized against**: `Trace` is index 1 of `router.go`'s `chain`, above both `Envelope` and `RateLimit`, so a 429 and a body refusal each produce a span carrying the status and — post-23d — an `error` event, both projected from the single `logNack` in `response_writer.go` that every refusal already passes through. Only "no counter" was ever true, and a counter that restates a span fact is the `duration_ms` mistake one signal up. **Task 25 is therefore the levels, not the refusals**, and it is smaller for it. Conflating the operator set with the network set is why this service has no metrics at all. **Task 25** is the operator set and is **not blocked**; Task 24 stays blocked on the registry — *five of its twelve codes have since shipped, on 2026-09-10, derived in `otel/collector.yaml`; the block stands for the other seven. See Task 24.* **And the deny-list is a rule nothing enforces** — five sub-tasks comply with it by hand. **Task 26** turns it into a pin, over the exporter's output rather than over the code that builds it | 23, 24, 25, 26 | | **A26** | **The publish geometry budget becomes configuration.** `MaxGeometriesPerCatalog` moves from a `const` in `src/publish/geometry.go` to `config.Geo.MaxGeometriesPerCatalog` / `GEO_MAX_GEOMETRIES_PER_CATALOG`, default **256**, threaded `container → publish.NewService → derive → ExtractGeometries → catalogWalk`. **The default does not change and no behaviour over the ceiling changes** — the extra finds are still `PARTIAL` faults naming their paths, never a silent drop. What changes is who gets to pick the number. **The failure it causes is the wrong shape to be a build-time constant.** Over the ceiling the publish still answers HTTP 200 and the publisher is told, but those shapes are not searchable, and at the operator that reads as missing data rather than as a limit: a 100 km `S_DWITHIN` returns fewer markets than exist and nothing in the response says why. Measured against a real catalog — 273 Maharashtra markets carrying a `Point` under both `coverageAreas` and `market.location`, so 546 shapes — only the first 128 markets were findable by place under 256. **Config rather than a larger constant, for the reason `ResolutionCells` two rows above already gives:** how many shapes a catalog carries is a property of a deployment's publishers, and the ceiling is reached by publishing normally rather than by abuse. That is also the line that decides which neighbours do *not* move: `MaxCatalogWalkDepth` bounds a stack against a pathological document, which is abuse, and `MaxIndexCoverCells` is a property of the index rather than of a catalog. **Zero is refused at boot**, alongside the body ceiling and for the same reason — it reads either as "unlimited" or as "refuse everything", and a catalog that may hold no geometry is undiscoverable by place. **The duplicate declaration in `src/indexing/geo/h3.go` is deleted, not rewired.** It had zero call sites; two constants of one name in two packages is how an operator raises the ceiling in the half that is not read. **What the number stands in for is cells, not shapes:** it times `MaxIndexCoverCells` (8,192) is the worst case one publish can ask the index to hold, so a points-only catalog — one cell per shape — can raise it a long way and a polygon-heavy one cannot | 2, 17, 18 | | **A27** | **A deployment may decline free-text retrieval, and declining REFUSES where a missing backend mode degrades.** `Search.EnableTextSearch` / `SEARCH_ENABLE_TEXT_SEARCH`, **true by default** — the value that takes no decision has to be the one that behaves as before the flag existed. With it false, an intent carrying a non-empty `textSearch` is refused in `MapIntent` with `SCH_TYPE_NOT_SUPPORTED` at `$['message']['intent']['textSearch']`, and the message names `spatial` and `filters` as what still works. **The whole content of this amendment is that it is not `FailOnUnavailableMode` under another name**, because the two read alike and behave oppositely. That flag is about a mode the **backend** cannot run — `semantic` on a `noop` deployment — and false there degrades, answering with what the other modes found and naming the gap in `X-Beckn-Degraded`; the query the caller wrote survives without it. This one is the **deployment's** policy, and degrading would be the failure C11 exists to prevent: on a text-only intent the term IS the query, so running the remaining modes without it answers **the whole corpus under a 200** — a page indistinguishable at the caller from a correct one. That is the same widening `MapIntent`'s doc comment already refuses for an unreadable `targets` or an ignored SRID, which is why the guard sits there and not in `negotiate`. **Three details are load-bearing.** The guard reads the **trimmed** term, like every other reader of it: `" "` asks for no retrieval mode at all, so it stays the no-criterion refusal and calling it a disabled text search would name the wrong mistake. The intent's required-one-of **list** shrinks with the flag — `spatial or filters`, with a parenthetical saying why the third is absent — because offering `textSearch` to a caller this deployment would then refuse walks them into a second `400`; the **guard** does not shrink with it, since a term that was sent is a criterion that was sent and reporting it missing would be a false sentence stacked on the true one. And the wire message names the two criteria that still work rather than the environment variable, matching `negotiate` and the `filters.type` refusal: a caller cannot act on the variable, and the operator has `docs/publish-and-discover.md`. **`modesFor` is left alone.** It still maps a non-empty `Text` to all three ranked modes, because the refusal is total and unreachable past it; a second guard there would be a branch no test can reach and the kind of defence that rots into a contradiction | 2, 17, 19 | -| **A28** | **The page rides in the signed body, and the answer says which page it is (#51).** The schema is now the **OAN profile** of `beckn.yaml`, pinned at commit `dfc1097` of `OpenAgriNet/protocol-specifications-v2` (`VALIDATION_SPEC_URL`'s default and `tests/testdata/beckn-v2.0.0.yaml`, byte-identical) — a commit because the branch carries no tag yet; it becomes a tag when one is cut, never the branch name. That profile adds `limit` (1-100) and `offset` (>= 0) to `DiscoverAction` and an optional `pagination` (`limit`, `offset`, `total`) to `OnDiscoverAction`. **Three decisions.** (1) The page is read from the **body**, each field falling back to the query parameter of the same name. The body wins field by field because it is the half the request signature covers; the query string survives only so callers written before the profile keep working, and it still clamps where the body is bounded by the schema — a body `limit` of 101 is an L1 refusal, not a clamp. (2) `pagination` reports the page **applied**, after the default and the clamp: a caller that asked for 5000 must read back the 100 it got, or it computes its next offset from a page size it never received. (3) `total` is sent **only when exact**, and is never a count query — **A19**'s reason for deleting that query (150.6 ms against a 1.5 ms search) stands. It is `len(fused)`, the number the search already holds, and it is omitted whenever any retrieval list came back as long as `MaxCandidatesPerMode`, because such a list may have been cut by its `LIMIT` and the fusion of it is a truncation. A list of exactly the cap that happened to be complete is reported uncounted too — the one false negative, and the safe direction: an absent total says *narrow the query*, a short one says *this is everything* when it is not. Degradation does not suppress it; that is `X-Beckn-Degraded`'s job, and the total is the size of the set this answer pages through. The memory backend has no cap, so its total is always present. **Unchanged:** `offset + limit > MaxCandidatesPerMode` is still refused (now at `$['message']['offset']`), never answered with an empty page. **Not taken here:** the same profile makes `context.networkId` a string or an array; `beckn.Context.NetworkID` is still a string, so a list is refused by the envelope parser exactly as before — its meaning for scope and `visibleTo` is a separate issue. Supersedes A19's "no total" for this one exact case, and the "exactly one field" note on `OnDiscoverAction` | 11, 16, 19 | A6 and A7 exist for one requirement: *swap the text backend later, keep geo on PG, and let publish write to two stores.* Both build **seams plus conformance @@ -2166,9 +2165,7 @@ POST /discover "provider": { ... }, // fetched once per catalog on the page "resources": [ ... ], // only the ones that matched "offers": [ ... ] // only those touching a matched - } ], // resource, plus catalog-wide ones - "pagination": { "limit": 20, "offset": 0, "total": 45 } // A28 - } } + } ] } } // resource, plus catalog-wide ones ``` ```http @@ -2176,9 +2173,9 @@ X-Beckn-Degraded: structured ``` The degraded list is a **header, not a body field** (C11). `OnDiscoverAction` -declares `additionalProperties: false` with `catalogs` and `pagination` (A28) as -its only properties, so a `degraded` key inside `message` is not an extension — -it is a response that fails its own schema. The header is absent when nothing degraded. +declares `additionalProperties: false` with `catalogs` as its only property, so +a `degraded` key inside `message` is not an extension — it is a response that +fails its own schema. The header is absent when nothing degraded. ### The flow diff --git a/docs/publish-and-discover.md b/docs/publish-and-discover.md index 113ede6..aa78591 100644 --- a/docs/publish-and-discover.md +++ b/docs/publish-and-discover.md @@ -19,10 +19,9 @@ That is the whole of it: | `GET /readyz` | the platform | can it also reach PostgreSQL | Four routes, no wildcard mount, no aliases. Both APIs are implemented against -the Beckn v2.0.0 specification as OpenAgriNet profiles it — -[`OpenAgriNet/protocol-specifications-v2`](https://github.com/OpenAgriNet/protocol-specifications-v2/blob/dfc109759b3781fde74539adc79df63fccbadb83/api/v2.0.0/beckn.yaml) -at commit `dfc1097`, which adds discover pagination to beckn's `core-v2.0.0-lts` -(A28), pinned in this repo as +the published Beckn v2.0.0 specification — +[`beckn/protocol-specifications-v2`](https://github.com/beckn/protocol-specifications-v2/blob/core-v2.0.0-lts/api/v2.0.0/beckn.yaml) +at the `core-v2.0.0-lts` tag, pinned in this repo as [`tests/testdata/beckn-v2.0.0.yaml`](../tests/testdata/beckn-v2.0.0.yaml). That document *is* the validator: the service loads it and validates every request against it, rather than restating the protocol in Go. @@ -470,7 +469,7 @@ matches no field fails the boot, so a typo cannot silently do nothing. | `EMBEDDING_MODEL` / `_ENDPOINT` / `_DIMENSIONS` | `nomic-embed-text` / `http://localhost:11434` / `768` | the Ollama seam | | `RATE_LIMIT_RPS` / `_BURST` | `20` / `40` | per-client limit; `burst >= rps` is enforced | | `VALIDATION_ENABLE_L1_SCHEMA` | `true` | Beckn schema validation of the payload | -| `VALIDATION_SPEC_URL` | the OAN profile's raw URL at commit `dfc1097` | where to fetch the spec; a commit, so it cannot move under a running deployment | +| `VALIDATION_SPEC_URL` | the `core-v2.0.0-lts` raw URL | where to fetch the spec; a tag, so it cannot move under a running deployment | | `VALIDATION_SPEC_CACHE_PATH` | `.cache/beckn/beckn.yaml` | where the spec is read from | | `AUTH_ENABLE_SIGNATURE_VERIFICATION` | `false` | `true` refuses the boot — see below | | `EXT_ALLOW_NETWORK_FETCH` | `false` | may a URL from a *request body* be fetched | diff --git a/docs/quickstart.md b/docs/quickstart.md index 389532f..77ab38c 100644 --- a/docs/quickstart.md +++ b/docs/quickstart.md @@ -270,22 +270,20 @@ does not remove the requirement. It is fetched from `VALIDATION_SPEC_URL`, which now defaults to ``` -https://raw.githubusercontent.com/OpenAgriNet/protocol-specifications-v2/dfc109759b3781fde74539adc79df63fccbadb83/api/v2.0.0/beckn.yaml +https://raw.githubusercontent.com/beckn/protocol-specifications-v2/refs/tags/core-v2.0.0-lts/api/v2.0.0/beckn.yaml ``` and cached under `VALIDATION_SPEC_CACHE_PATH` (`.cache/beckn/beckn.yaml`). That is why running the binary from the host needs no spec configuration at all: it fetches once and reads the cache afterwards. -A **commit**, not a branch, which is the whole reason there can be a default. A +A **tag**, not a branch, which is the whole reason there can be a default. A branch would let an upstream merge change the validator under a running -deployment without a deploy; a commit cannot move. It is OpenAgriNet's profile -of the Beckn spec rather than beckn's `core-v2.0.0-lts` tag because the profile -adds discover pagination (A28), and it becomes a tag when one is cut. A network +deployment without a deploy; `refs/tags/core-v2.0.0-lts` cannot move. A network that trusts a different document still overrides it. Compose additionally mounts `tests/testdata/beckn-v2.0.0.yaml` straight into -the cache path. That file is byte-identical to that commit, so the two paths agree +the cache path. That file is byte-identical to the tag, so the two paths agree — and the mount is what lets the container stack come up with no network at all. If the fetch fails, the boot logs one loud warning and falls back to the cache; that warning is accurate and is not a failure. diff --git a/examples/verify.sh b/examples/verify.sh index 57bd3cf..165f8e5 100755 --- a/examples/verify.sh +++ b/examples/verify.sh @@ -366,15 +366,16 @@ refusal_case 18-discover-unparsable-jsonpath.json \ "SCH_INVALID_JSONPATH" "\$.message.intent.filters.expression" printf '%s\n' "--- pagination" -body="$(curl -sS -X POST "$BASE/discover?limit=1&offset=0" \ - -H 'Content-Type: application/json' \ - --data-binary @"$DIR/03-discover-schema-context.json")" +body="$(python3 -c 'import json,sys; b=json.load(open(sys.argv[1])); b["message"].update(limit=1, offset=0); print(json.dumps(b))' \ + "$DIR/03-discover-schema-context.json" | + curl -sS -X POST "$BASE/discover" \ + -H 'Content-Type: application/json' --data-binary @-)" got="$(resources "$body")" n="$(printf '%s' "$got" | awk -F, 'NF{print NF}')" if [ "${n:-0}" = "1" ]; then - ok "?limit=1 -> 1 resource [$got]" + ok "limit=1 -> 1 resource [$got]" else - bad "?limit=1 -> ${n:-0} resources [$got]" + bad "limit=1 -> ${n:-0} resources [$got]" fi echo diff --git a/src/beckn/actions.go b/src/beckn/actions.go index caea0e8..809867e 100644 --- a/src/beckn/actions.go +++ b/src/beckn/actions.go @@ -134,16 +134,12 @@ type CatalogStats struct { CategoryCount int `json:"categoryCount"` } -// DiscoverAction is the `message` of a discover request. -// -// Limit and Offset are the OAN profile's addition (A28): the page rides in the -// signed body rather than the query string, which the request signature does -// not cover. Pointers, because absent and zero differ — an absent field falls -// back to the query string, and the schema refuses a zero limit outright. +// DiscoverAction is the `message` of a discover request. Limit and Offset are +// the page; zero means "the default". type DiscoverAction struct { Intent Intent `json:"intent"` - Limit *int `json:"limit,omitempty"` - Offset *int `json:"offset,omitempty"` + Limit int `json:"limit,omitempty"` + Offset int `json:"offset,omitempty"` } // OnDiscoverAction is the `message` of the discover response. @@ -158,12 +154,8 @@ type OnDiscoverAction struct { Pagination *Pagination `json:"pagination,omitempty"` } -// Pagination is the page a discover answer APPLIED — after the default and the -// clamp, not as requested — so the caller computes the next offset from what -// it actually got. -// -// Total is a pointer and omitted when nil: it is sent only when exact (A28). -// Offset is never omitted, because zero is the first page and not "unset". +// Pagination is the page the answer applied, after default and clamp. Total is +// declared by the schema and not yet set. type Pagination struct { Limit int `json:"limit"` Offset int `json:"offset"` diff --git a/src/discover/controller.go b/src/discover/controller.go index 57d2a98..a205691 100644 --- a/src/discover/controller.go +++ b/src/discover/controller.go @@ -2,10 +2,7 @@ package discover import ( "encoding/json" - "fmt" "net/http" - "net/url" - "strconv" "strings" "time" @@ -68,11 +65,7 @@ func (c *Controller) Discover(w http.ResponseWriter, r *http.Request) { observeIntent(ctx, envelope.Context, action.Intent) - page, err := pageFrom(action, r.URL.Query()) - if err != nil { - httpx.WriteNack(ctx, w, c.errors, envelope.Context.MessageID, err) - return - } + page := Page{Limit: action.Limit, Offset: action.Offset} answer, degraded, err := c.service.Discover(ctx, envelope.Context, action.Intent, page) if err != nil { @@ -97,52 +90,6 @@ func (c *Controller) Discover(w http.ResponseWriter, r *http.Request) { } } -// pageFrom reads the page off the body, field by field falling back to the -// query string (A28). -// -// The body is the signed half, so a field it names wins. The query string is -// the deprecated route and stays only so that callers written before the OAN -// profile added limit and offset to DiscoverAction keep working; the schema -// already bounds the body's values, which is why only the query's are parsed -// here. -// -// An unreadable query value is refused, not defaulted: zero already means "the -// default page size" to the mapper, so reading `limit=twenty` as zero would -// answer a page the caller never asked for and say nothing about it. -func pageFrom(action beckn.DiscoverAction, query url.Values) (Page, error) { - limit, err := intParam(query, "limit") - if err != nil { - return Page{}, err - } - offset, err := intParam(query, "offset") - if err != nil { - return Page{}, err - } - if action.Limit != nil { - limit = *action.Limit - } - if action.Offset != nil { - offset = *action.Offset - } - return Page{Limit: limit, Offset: offset}, nil -} - -// intParam reads one numeric query parameter. Absent is zero, which the mapper -// resolves to its own default; present and unreadable is a refusal. -func intParam(query url.Values, name string) (int, error) { - raw := query.Get(name) - if raw == "" { - return 0, nil - } - - value, err := strconv.Atoi(raw) - if err != nil { - return 0, apperrors.Schema(beckn.CodeSchemaInvalidFormat, - fmt.Sprintf("%s is not a whole number", name)).At("$." + name) - } - return value, nil -} - // responseContext turns the request's envelope into the response's. The // correlation handles are echoed rather than regenerated: they are how the caller // ties this answer to the request it made. diff --git a/src/discover/controller_test.go b/src/discover/controller_test.go index 11e6aee..72e0d11 100644 --- a/src/discover/controller_test.go +++ b/src/discover/controller_test.go @@ -312,14 +312,13 @@ func TestAnUnavailableModeIsA400WhenTheDeploymentAsksToBe(t *testing.T) { } } -// Pagination arrives as query parameters rather than inside the intent, and an -// oversize limit is clamped rather than refused (C11's neighbour: the caller +// Pagination arrives in the body beside the intent, and an oversize limit is clamped rather than refused (C11's neighbour: the caller // still gets the results they asked about). -func TestLimitAndOffsetAreReadFromTheQueryString(t *testing.T) { +func TestLimitAndOffsetAreReadFromTheBody(t *testing.T) { repo := &stubRepo{capabilities: everything()} handler := mount(t, repo, settings()) - if got := post(t, handler, "/discover?limit=1000&offset=40", wheat).Code; got != http.StatusOK { + if got := post(t, handler, "/discover", `{"context":{"action":"discover"},"message":{"intent":{"textSearch":"wheat"},"limit":1000,"offset":40}}`).Code; got != http.StatusOK { t.Fatalf("status = %d, want 200 — an oversize limit is clamped, not refused", got) } if repo.gotQuery.Limit != settings().Search.MaxPageSize { @@ -336,7 +335,7 @@ func TestLimitAndOffsetAreReadFromTheQueryString(t *testing.T) { func TestAPaginationParameterThatIsNotANumberIsRefused(t *testing.T) { repo := &stubRepo{capabilities: everything()} - recorder := post(t, mount(t, repo, settings()), "/discover?limit=twenty", wheat) + recorder := post(t, mount(t, repo, settings()), "/discover", `{"context":{"action":"discover"},"message":{"intent":{"textSearch":"wheat"},"limit":"twenty"}}`) if recorder.Code != http.StatusBadRequest { t.Fatalf("status = %d, want 400; body: %s", recorder.Code, recorder.Body) } @@ -345,13 +344,12 @@ func TestAPaginationParameterThatIsNotANumberIsRefused(t *testing.T) { } } -// The same refusal for offset, which pageFrom reads with its own call to -// intParam rather than reusing limit's — a valid limit must not mask an -// unreadable offset. +// The same refusal for offset — a valid limit must not mask an unreadable +// offset. func TestAnUnreadableOffsetIsRefused(t *testing.T) { repo := &stubRepo{capabilities: everything()} - recorder := post(t, mount(t, repo, settings()), "/discover?limit=10&offset=twenty", wheat) + recorder := post(t, mount(t, repo, settings()), "/discover", `{"context":{"action":"discover"},"message":{"intent":{"textSearch":"wheat"},"limit":10,"offset":"twenty"}}`) if recorder.Code != http.StatusBadRequest { t.Fatalf("status = %d, want 400; body: %s", recorder.Code, recorder.Body) } @@ -412,100 +410,3 @@ func TestARefusedOperatorIsA400(t *testing.T) { t.Errorf("the backend was searched %d times; a refused intent runs no query", repo.calls) } } - -// The page travels in the SIGNED body now (A28). The OAN profile of the schema -// declares limit and offset on DiscoverAction, so a caller no longer needs the -// query string — which the request signature does not cover. -func TestLimitAndOffsetAreReadFromTheBody(t *testing.T) { - repo := &stubRepo{capabilities: everything()} - - recorder := post(t, mount(t, repo, settings()), "/discover", - `{"context":{"action":"discover"},"message":{"intent":{"textSearch":"wheat"},"limit":5,"offset":10}}`) - if recorder.Code != http.StatusOK { - t.Fatalf("status = %d, want 200; body: %s", recorder.Code, recorder.Body) - } - if repo.gotQuery.Limit != 5 || repo.gotQuery.Offset != 10 { - t.Errorf("page = limit %d offset %d, want the body's 5 and 10", repo.gotQuery.Limit, repo.gotQuery.Offset) - } -} - -// Each field the body names wins over the same field in the query string, and a -// field the body omits still falls back to it. The body is the signed half, so -// it is the one to believe when the two disagree. -func TestTheBodyWinsOverTheQueryStringFieldByField(t *testing.T) { - repo := &stubRepo{capabilities: everything()} - - recorder := post(t, mount(t, repo, settings()), "/discover?limit=7&offset=40", - `{"context":{"action":"discover"},"message":{"intent":{"textSearch":"wheat"},"offset":10}}`) - if recorder.Code != http.StatusOK { - t.Fatalf("status = %d, want 200; body: %s", recorder.Code, recorder.Body) - } - if repo.gotQuery.Offset != 10 { - t.Errorf("Offset = %d, want the body's 10 over the query's 40", repo.gotQuery.Offset) - } - if repo.gotQuery.Limit != 7 { - t.Errorf("Limit = %d, want the query's 7 — the body named none", repo.gotQuery.Limit) - } -} - -// The answer says which page it is: the limit and offset actually APPLIED, after -// the default and the clamp, and the total when the backend could count it. -func TestTheResponseReportsThePageItApplied(t *testing.T) { - total := 3 - repo := &stubRepo{ - capabilities: everything(), - result: domain.SearchResult{Catalogs: []domain.Catalog{{ID: "c1"}}, Total: &total}, - } - - message := onDiscover(t, post(t, mount(t, repo, settings()), "/discover?limit=1000&offset=4", wheat)) - - want := beckn.Pagination{Limit: settings().Search.MaxPageSize, Offset: 4} - if message.Pagination == nil { - t.Fatal("pagination is absent, want the applied page") - } - if got := *message.Pagination; got.Limit != want.Limit || got.Offset != want.Offset || - got.Total == nil || *got.Total != total { - t.Errorf("pagination = %+v (total %v), want limit %d offset %d total %d", - got, got.Total, want.Limit, want.Offset, total) - } -} - -// A total the backend could not count exactly is left OUT, not sent as zero or -// as a guess. A caller reads its absence as "more matched than a page walk can -// reach"; a zero would read as "nothing matched". -func TestAnUncountedTotalIsAbsentFromTheWire(t *testing.T) { - repo := &stubRepo{capabilities: everything(), result: domain.SearchResult{Total: nil}} - - recorder := post(t, mount(t, repo, settings()), "/discover", wheat) - envelope := decodeResponse(t, recorder) - - var message struct { - Pagination map[string]any `json:"pagination"` - } - if err := json.Unmarshal(envelope.Message, &message); err != nil { - t.Fatalf("decoding the message: %v", err) - } - pagination := message.Pagination - if pagination == nil { - t.Fatal("pagination is absent, want the applied page without a total") - } - if _, ok := pagination["total"]; ok { - t.Errorf("pagination = %v, want no total key", pagination) - } - if offset, ok := pagination["offset"]; !ok || offset != float64(0) { - t.Errorf("pagination = %v, want offset 0 present — zero is a real offset", pagination) - } -} - -func onDiscover(t *testing.T, recorder *httptest.ResponseRecorder) beckn.OnDiscoverAction { - t.Helper() - - if recorder.Code != http.StatusOK { - t.Fatalf("status = %d, want 200; body: %s", recorder.Code, recorder.Body) - } - var message beckn.OnDiscoverAction - if err := json.Unmarshal(decodeResponse(t, recorder).Message, &message); err != nil { - t.Fatalf("decoding the message: %v", err) - } - return message -} diff --git a/src/discover/intent_mapper.go b/src/discover/intent_mapper.go index d23f76a..5390f09 100644 --- a/src/discover/intent_mapper.go +++ b/src/discover/intent_mapper.go @@ -14,9 +14,8 @@ import ( "github.com/OpenAgriNet/discovery-service/src/platform/jsonpath" ) -// Page is the request's pagination: DiscoverAction's limit and offset, each -// falling back to the deprecated query parameter of the same name (A28). Zero -// means "the default". A struct rather than two adjacent ints, because +// Page is the request's pagination, which arrives as HTTP query parameters +// rather than inside the intent. A struct rather than two adjacent ints, because // `limit, offset` and `offset, limit` compile identically. type Page struct { Limit int @@ -230,7 +229,7 @@ func mapPage(page Page, search config.Search) (limit, offset int, faults []domai if offset+limit > search.MaxCandidatesPerMode { faults = append(faults, domain.Fault{ - Path: "$['message']['offset']", + Path: "$['offset']", Code: string(beckn.CodeSchemaInvalidFormat), Message: fmt.Sprintf( "offset %d plus limit %d passes the retrieval depth of %d", diff --git a/src/discover/intent_mapper_test.go b/src/discover/intent_mapper_test.go index 2f512f8..08e8550 100644 --- a/src/discover/intent_mapper_test.go +++ b/src/discover/intent_mapper_test.go @@ -324,10 +324,6 @@ func TestLimitIsClampedAndAPagePastTheRetrievalDepthIsRefused(t *testing.T) { if !strings.Contains(fatal[0].Message, "500") { t.Errorf("Message = %q, want it to name the boundary", fatal[0].Message) } - // The page is a body field now (A28), so the fault points into the body. - if fatal[0].Path != "$['message']['offset']" { - t.Errorf("Path = %q, want $['message']['offset']", fatal[0].Path) - } } // NetworkID is NOT set here. The service sets it from the envelope, and empty diff --git a/src/discover/service.go b/src/discover/service.go index 2fc0e0a..38e728b 100644 --- a/src/discover/service.go +++ b/src/discover/service.go @@ -73,7 +73,7 @@ func (s *Service) Discover( Catalogs: render(result.Catalogs), // What MapIntent applied, not what was asked: a caller that sent 5000 // must read back the 100 it got, or its next offset skips results. - Pagination: &beckn.Pagination{Limit: query.Limit, Offset: query.Offset, Total: result.Total}, + Pagination: &beckn.Pagination{Limit: query.Limit, Offset: query.Offset}, }, degraded, nil } @@ -90,7 +90,7 @@ func typedSearchFailure(ctx context.Context, err error) error { if errors.Is(err, domain.ErrRetrievalDepth) { return apperrors. Schema(beckn.CodeSchemaInvalidFormat, err.Error()). - At(jsonpath.Dot("$['message']['offset']")) + At(jsonpath.Dot("$['offset']")) } // An expression the store's own parser refused is the caller's mistake too, diff --git a/src/discover/service_test.go b/src/discover/service_test.go index 4dfecf9..87f2472 100644 --- a/src/discover/service_test.go +++ b/src/discover/service_test.go @@ -101,12 +101,13 @@ func TestAModeTheBackendLacksIsNamedRatherThanDropped(t *testing.T) { answer, degraded, err := discover.NewService(repo, settings()). Discover(t.Context(), beckn.Context{}, beckn.Intent{TextSearch: "wheat seeds"}, discover.Page{}) + catalogs := answer.Catalogs if err != nil { t.Fatalf("Discover: %v", err) } - if len(answer.Catalogs) != 1 { - t.Errorf("catalogs = %d, want the one the backend found — degrading is not failing", len(answer.Catalogs)) + if len(catalogs) != 1 { + t.Errorf("catalogs = %d, want the one the backend found — degrading is not failing", len(catalogs)) } if !slices.Equal(degraded, []string{string(domain.CapabilitySemantic)}) { t.Errorf("degraded = %v, want [semantic]", degraded) @@ -188,12 +189,13 @@ func TestSpatialStillAnswersWithTextSearchOff(t *testing.T) { Geometry: bengaluru(), }}}, discover.Page{}) + catalogs := answer.Catalogs if err != nil { t.Fatalf("Discover: %v", err) } - if len(answer.Catalogs) != 1 { - t.Errorf("catalogs = %d, want the one the backend found", len(answer.Catalogs)) + if len(catalogs) != 1 { + t.Errorf("catalogs = %d, want the one the backend found", len(catalogs)) } if hasMode(repo.gotModes, domain.CapabilityLexical) { t.Errorf("modes = %v, want no ranked text mode — nothing asked for one", repo.gotModes) @@ -220,12 +222,13 @@ func TestAFilterABackendCannotRunIsDegradedRatherThanIgnored(t *testing.T) { Type: "jsonpath", Expression: `$.catalogs[*].resources[*] ? (@.grade == "A")`, }}, discover.Page{}) + catalogs := answer.Catalogs if err != nil { t.Fatalf("Discover: %v", err) } - if len(answer.Catalogs) != 1 { - t.Errorf("catalogs = %d, want the one the backend found — degrading is not failing", len(answer.Catalogs)) + if len(catalogs) != 1 { + t.Errorf("catalogs = %d, want the one the backend found — degrading is not failing", len(catalogs)) } if !slices.Equal(degraded, []string{string(domain.CapabilityJSONPath)}) { t.Errorf("degraded = %v, want [jsonpath] — a filter this backend cannot run and "+ @@ -514,10 +517,10 @@ func TestARenderedCatalogCarriesItsDocumentsVerbatim(t *testing.T) { answer, _, err := discover.NewService(repo, settings()). Discover(t.Context(), beckn.Context{}, beckn.Intent{TextSearch: "wheat"}, discover.Page{}) + catalogs := answer.Catalogs if err != nil { t.Fatalf("Discover: %v", err) } - catalogs := answer.Catalogs if len(catalogs) != 1 || len(catalogs[0].Offers) != 1 { t.Fatalf("rendered = %+v, want one catalog carrying one offer", catalogs) } @@ -579,10 +582,10 @@ func TestNoOfferWhoseResourcesAreAllOffThePageIsRendered(t *testing.T) { answer, _, err := discover.NewService(repo, settings()). Discover(t.Context(), beckn.Context{}, beckn.Intent{TextSearch: "wheat"}, discover.Page{}) + catalogs := answer.Catalogs if err != nil { t.Fatalf("Discover: %v", err) } - catalogs := answer.Catalogs if len(catalogs) != 1 || len(catalogs[0].Resources) != 1 || catalogs[0].Resources[0].ID != "wheat" { t.Fatalf("rendered = %+v, want only the matched resource", catalogs) @@ -661,10 +664,10 @@ func TestACatalogWithAnUnreadableDocumentIsDropped(t *testing.T) { answer, _, err := discover.NewService(repo, settings()). Discover(t.Context(), beckn.Context{}, beckn.Intent{TextSearch: "wheat"}, discover.Page{}) + catalogs := answer.Catalogs if err != nil { t.Fatalf("Discover: %v", err) } - catalogs := answer.Catalogs if len(catalogs) != 0 { t.Errorf("catalogs = %+v, want none", catalogs) } @@ -684,10 +687,10 @@ func TestAResourceWithAnUnreadableDocumentIsDropped(t *testing.T) { answer, _, err := discover.NewService(repo, settings()). Discover(t.Context(), beckn.Context{}, beckn.Intent{TextSearch: "wheat"}, discover.Page{}) + catalogs := answer.Catalogs if err != nil { t.Fatalf("Discover: %v", err) } - catalogs := answer.Catalogs if len(catalogs) != 1 || len(catalogs[0].Resources) != 1 || catalogs[0].Resources[0].ID != "r2" { t.Fatalf("resources = %+v, want only r2 — r1's document could not be read", catalogs[0].Resources) } @@ -706,10 +709,10 @@ func TestAnOfferWithAnUnreadableDocumentIsDropped(t *testing.T) { answer, _, err := discover.NewService(repo, settings()). Discover(t.Context(), beckn.Context{}, beckn.Intent{TextSearch: "wheat"}, discover.Page{}) + catalogs := answer.Catalogs if err != nil { t.Fatalf("Discover: %v", err) } - catalogs := answer.Catalogs if len(catalogs) != 1 || len(catalogs[0].Offers) != 1 || catalogs[0].Offers[0].ID != "o2" { t.Fatalf("offers = %+v, want only o2 — o1's document could not be read", catalogs[0].Offers) } @@ -813,32 +816,3 @@ func TestAMalformedFilterExpressionIsTheCallersFaultNotAFiveHundred(t *testing.T "internal that must not reach the caller", fault.Message, want) } } - -// The answer carries the page the service APPLIED — the default when none was -// asked for, the clamp when too much was — beside the backend's total. A caller -// that sent limit 5000 must read 100 back, or it will compute the next offset -// from a page size it never got. -func TestTheAnswerReportsTheAppliedPageAndTheBackendsTotal(t *testing.T) { - total := 42 - repo := &stubRepo{capabilities: everything(), result: domain.SearchResult{Total: &total}} - cfg := settings() - - answer, _, err := discover.NewService(repo, cfg).Discover( - t.Context(), beckn.Context{}, beckn.Intent{TextSearch: "wheat"}, - discover.Page{Limit: 5000, Offset: 20}) - if err != nil { - t.Fatalf("Discover: %v", err) - } - - page := answer.Pagination - if page == nil { - t.Fatal("pagination is absent") - } - if page.Limit != cfg.Search.MaxPageSize || page.Offset != 20 { - t.Errorf("pagination = limit %d offset %d, want the clamped %d and 20", - page.Limit, page.Offset, cfg.Search.MaxPageSize) - } - if page.Total == nil || *page.Total != total { - t.Errorf("total = %v, want the backend's %d", page.Total, total) - } -} diff --git a/src/domain/query.go b/src/domain/query.go index c9938f0..f6270e4 100644 --- a/src/domain/query.go +++ b/src/domain/query.go @@ -125,17 +125,11 @@ type SearchQuery struct { // SearchResult is one page of an answered query. // // Degraded names the retrieval modes that did not contribute and reaches the -// caller as the X-Beckn-Degraded header, not a body key (C11). -// -// Total is the size of the fused candidate set the page was sliced from, and -// nil whenever that size is not the number that matched — when a retrieval mode -// returned as many ids as its cap, it may have stopped early (A28). It is never -// a count query: A19 deleted that for costing 100x the search it accompanied, -// and this is the number the search already holds. +// caller as the X-Beckn-Degraded header, not a body key (C11). There is no +// Total (A19). type SearchResult struct { Catalogs []Catalog Degraded []string - Total *int } // Scope is the request-wide gate every retrieval mode reads (A6). diff --git a/src/platform/config/config.go b/src/platform/config/config.go index 2a9ad7e..0d3bb45 100644 --- a/src/platform/config/config.go +++ b/src/platform/config/config.go @@ -219,19 +219,14 @@ type Validation struct { // beckn.yaml is fetched at boot rather than baked into the image, so a // deployment names the document it validates against. // - // The default is IMMUTABLE, and that is the whole reason there can be one. - // This field carried no default until 2026-09-09, on the argument that which - // spec a network trusts is not a repository-wide decision — true of a branch, + // The default is a TAG, and that is the whole reason there can be one. This + // field carried no default until 2026-09-09, on the argument that which spec + // a network trusts is not a repository-wide decision — true of a branch, // which would let an upstream merge change the validator under a running - // deployment without a deploy. A tag or a commit cannot move, so the default - // names one document and a network that trusts a different one still - // overrides it. tests/testdata/beckn-v2.0.0.yaml is byte-identical to it, so - // the fetch path and an air-gapped mount agree. - // - // It is the OAN profile at a COMMIT, not beckn's core-v2.0.0-lts tag (A28): - // the profile adds limit/offset to DiscoverAction and pagination to - // OnDiscoverAction, and its branch carries no tag yet. Swap the SHA for a - // tag when one is cut — never for the branch name. + // deployment without a deploy. A pinned commit of the OAN profile (which adds + // discover pagination) cannot move, so the default names one immutable + // document and a network that trusts a different one still overrides it. tests/testdata/beckn-v2.0.0.yaml is byte-identical + // to it, so the fetch path and an air-gapped mount agree. // // The cost is that boot now makes a network call before falling back to // SpecCachePath. LoadSpecIndex already warns loudly and falls back, so an @@ -638,19 +633,9 @@ func validateDatabase(database Database) error { ) } -// schemaMaxPageSize is PaginationParams.limit's `maximum` in the OAN profile of -// the spec (A28). A copy rather than a read of the document, because config -// validates before the spec index is built; the two drifting apart shows up as -// a boot refusal here or as a schema failure on the response, never silently. -const schemaMaxPageSize = 100 - func validateSearch(search Search) error { return errors.Join( require(search.DefaultPageSize > 0, "search.defaultPageSize %d is not positive", search.DefaultPageSize), - require(search.MaxPageSize <= schemaMaxPageSize, - "search.maxPageSize %d is above the schema's limit maximum of %d — a body asking for more is refused "+ - "before the clamp, and a query-string caller would be answered with a pagination.limit that fails its own schema", - search.MaxPageSize, schemaMaxPageSize), require(search.MaxPageSize >= search.DefaultPageSize, "search.maxPageSize %d is below search.defaultPageSize %d — a ceiling under the value it clamps", search.MaxPageSize, search.DefaultPageSize), diff --git a/src/platform/config/config_test.go b/src/platform/config/config_test.go index ac33bf2..6e93eb5 100644 --- a/src/platform/config/config_test.go +++ b/src/platform/config/config_test.go @@ -377,13 +377,6 @@ func TestValidateRejects(t *testing.T) { "a candidate pool smaller than one page": { "search:\n maxPageSize: 100\n maxCandidatesPerMode: 50\n", "maxCandidatesPerMode", }, - // The schema caps a page at 100 (A28). Above it, a body asking for more - // is refused before the clamp is reached, and a query-string caller is - // answered with a pagination.limit that fails the response's own - // schema. - "a max page size above the schema's maximum": { - "search:\n maxPageSize: 101\n maxCandidatesPerMode: 500\n", "maxPageSize", - }, "an unloadable timezone": { "app:\n defaultTimezone: Mars/Olympus_Mons\n", "defaultTimezone", }, diff --git a/src/platform/validation/spec_index.go b/src/platform/validation/spec_index.go index 3368ef2..eb7cc84 100644 --- a/src/platform/validation/spec_index.go +++ b/src/platform/validation/spec_index.go @@ -241,7 +241,7 @@ func loadFromCache(path string) (*SpecIndex, error) { // ages with the image), so an air-gapped deploy supplies one at // VALIDATION_SPEC_CACHE_PATH, and mounting it read-only is the right way to // supply a file the service must not be able to rewrite. docker-compose.yml does -// exactly that with a copy byte-identical to VALIDATION_SPEC_URL's default, and every +// exactly that with a copy byte-identical to the core-v2.0.0-lts tag, and every // boot with a reachable registry warned that it could not overwrite those bytes // with themselves. A warning that fires when nothing is wrong is one nobody reads // on the boot when something is. diff --git a/src/storage/conformance/discover.go b/src/storage/conformance/discover.go index 2041a33..5d97b95 100644 --- a/src/storage/conformance/discover.go +++ b/src/storage/conformance/discover.go @@ -5,7 +5,6 @@ import ( "errors" "fmt" "slices" - "strconv" "testing" "time" @@ -215,15 +214,6 @@ func pageOrder(result domain.SearchResult) []string { return ids } -// totalOf renders a result's total for a failure message, where a nil pointer -// would print as an address. -func totalOf(result domain.SearchResult) string { - if result.Total == nil { - return "absent" - } - return strconv.Itoa(*result.Total) -} - // offerIDs is every offer hydrated onto the page, sorted. func offerIDs(result domain.SearchResult) []string { ids := make([]string, 0) @@ -652,17 +642,10 @@ func pageTwoDoesNotOverlapPageOne() Case { t.Errorf("page two is %v, want [r-02 r-03]", got) } - // The offset SLICES a stable order rather than re-ranking it, so - // page two holds the next two and not two of the same. - - // And both pages report the same exact total (A28): five matched, - // none of them capped, so the count is the size of the set being - // paged — whichever page asked. - for name, result := range map[string]domain.SearchResult{"one": first, "two": second} { - if result.Total == nil || *result.Total != len(resources) { - t.Errorf("page %s reports total %v, want %d", name, totalOf(result), len(resources)) - } - } + // There is no total to assert (A19). What remains is the property + // a caller walking pages actually depends on: the offset SLICES a + // stable order rather than re-ranking it, so page two holds the + // next two and not two of the same. }, } } diff --git a/src/storage/memory/search.go b/src/storage/memory/search.go index e62efb0..1063c9e 100644 --- a/src/storage/memory/search.go +++ b/src/storage/memory/search.go @@ -62,13 +62,9 @@ func (r *Repository) Search( page = nil } - // This backend retrieves without a cap, so the ranked list is every match - // and its length is always the exact total. - total := len(page) return domain.SearchResult{ Catalogs: r.hydrate(pageOf(page, query.Offset, query.Limit), scope), Degraded: degraded, - Total: &total, }, nil } diff --git a/src/storage/postgres/read_path_test.go b/src/storage/postgres/read_path_test.go index cb54192..aee05d2 100644 --- a/src/storage/postgres/read_path_test.go +++ b/src/storage/postgres/read_path_test.go @@ -380,33 +380,6 @@ func TestPageTwoDoesNotOverlapPageOne(t *testing.T) { } } -// A mode that returned as many ids as its cap may have stopped early, so the -// fused list is a truncation and its length is not the number that matched. -// The total is then left OUT (A28) rather than reported short — this is the -// only backend with a cap, which is why the case lives here and not in the -// conformance suite. searchConfig's cap is 8; ten lots match. -func TestATotalIsReportedOnlyWhenNoModeHitItsCap(t *testing.T) { - repository, _ := publish(t, nil, readFixture{catalog: "cat-capped", resources: kharifLots(10)}) - - capped := search(t, repository, domain.SearchQuery{Text: "kharif", Limit: 4}) - if capped.Total != nil { - t.Errorf("total = %d with every mode at its cap of %d, want it absent", - *capped.Total, searchConfig().MaxCandidatesPerMode) - } -} - -func TestATotalUnderTheCapIsExact(t *testing.T) { - repository, _ := publish(t, nil, readFixture{catalog: "cat-counted", resources: kharifLots(6)}) - - result := search(t, repository, domain.SearchQuery{Text: "kharif", Limit: 4}) - if result.Total == nil || *result.Total != 6 { - t.Errorf("total = %v, want exactly 6", result.Total) - } - if got := len(matchedIDs(result)); got != 4 { - t.Errorf("the page holds %d, want 4 — the total counts the set, not the page", got) - } -} - // A mode that ERRORS is degraded, and the page is what the other modes found. // // The vectors are 768 wide because the column is, and the repository embeds at diff --git a/src/storage/postgres/search_repository.go b/src/storage/postgres/search_repository.go index 8375759..8ba65af 100644 --- a/src/storage/postgres/search_repository.go +++ b/src/storage/postgres/search_repository.go @@ -128,29 +128,7 @@ func (s *SearchRepository) Search( if err != nil { return domain.SearchResult{}, err } - return domain.SearchResult{ - Catalogs: catalogs, - Degraded: degraded, - Total: s.exactTotal(lists, fused), - }, nil -} - -// exactTotal is len(fused) when that is the number that matched, and nil when -// it is not (A28). -// -// A list as long as MaxCandidatesPerMode may have been cut by its LIMIT, so the -// fusion of it is a truncation. A list of exactly the cap that happened to be -// complete is reported as uncounted too — the one false negative, and the safe -// direction: an absent total says "narrow the query", a short one says "this is -// everything" when it is not. -func (s *SearchRepository) exactTotal(lists [][]string, fused []string) *int { - for _, ids := range lists { - if len(ids) >= s.search.MaxCandidatesPerMode { - return nil - } - } - total := len(fused) - return &total + return domain.SearchResult{Catalogs: catalogs, Degraded: degraded}, nil } // withinRetrievalDepth refuses a page past what any mode retrieves. Checked diff --git a/tests/acceptance/pagination_test.go b/tests/acceptance/pagination_test.go deleted file mode 100644 index a11ca6c..0000000 --- a/tests/acceptance/pagination_test.go +++ /dev/null @@ -1,98 +0,0 @@ -package acceptance - -import ( - "fmt" - "net/http" - "slices" - "testing" - - "github.com/OpenAgriNet/discovery-service/src/beckn" -) - -// fiveAtMajestic publishes five resources a spatial intent around Majestic -// matches, so a page size of two has to walk three pages to see them all. -func fiveAtMajestic(t *testing.T) *service { - t.Helper() - - list := make([]map[string]any, 0, 5) - for index := range 5 { - list = append(list, aResource(fmt.Sprintf("r-%d", index), fmt.Sprintf("Lot %d", index))) - } - svc := newService(t) - svc.publishCatalogs(t, aCatalog("c-lots", availableAt(majestic), resources(list...))) - return svc -} - -// pageAt asks for one page with limit and offset in the SIGNED body (A28), and -// returns the message so the scenario can read the pagination beside the -// catalogs. -func (s *service) pageAt(t *testing.T, limit, offset int) beckn.OnDiscoverAction { - t.Helper() - - answer := s.post(t, "/discover", envelope(beckn.ActionDiscover, map[string]any{ - "intent": spatial(dwithin(providerGeoPath, majestic, 5000)), - "limit": limit, - "offset": offset, - })) - if answer.status != http.StatusOK { - t.Fatalf("POST /discover limit %d offset %d = %d, want 200 — the OAN profile "+ - "declares both on DiscoverAction\nbody: %s", limit, offset, answer.status, answer.body) - } - - var body struct { - Message beckn.OnDiscoverAction `json:"message"` - } - answer.decode(t, &body) - return body.Message -} - -// A caller walking pages in the body reaches every match exactly once, and -// every page says where it is: the page it applied, and a total that tells the -// last page from a short one. Before A28 the body could not carry the page at -// all — the schema refused the key — and the answer carried nothing but -// catalogs, so a short page and the end were the same thing. -func TestTheBodyPagesThroughEveryMatchAndTheTotalMarksTheEnd(t *testing.T) { - svc := fiveAtMajestic(t) - - var seen []string - for offset := 0; offset < 5; offset += 2 { - message := svc.pageAt(t, 2, offset) - - page := message.Pagination - if page == nil { - t.Fatalf("offset %d: no pagination in the answer", offset) - } - if page.Limit != 2 || page.Offset != offset { - t.Errorf("offset %d: pagination reports limit %d offset %d, want 2 and %d", - offset, page.Limit, page.Offset, offset) - } - if page.Total == nil || *page.Total != 5 { - t.Errorf("offset %d: total = %v, want 5", offset, page.Total) - } - seen = append(seen, resourceIDs(message.Catalogs)...) - } - - slices.Sort(seen) - if want := []string{"r-0", "r-1", "r-2", "r-3", "r-4"}; !slices.Equal(seen, want) { - t.Errorf("walking the pages saw %v, want each of %v exactly once", seen, want) - } -} - -// The schema bounds the body's page, so a limit above its maximum is refused -// by L1 validation before the service clamps anything. The query string still -// clamps — it is outside the schema — which is one more reason it is the -// deprecated route. -func TestALimitAboveTheSchemaMaximumIsRefused(t *testing.T) { - svc := fiveAtMajestic(t) - - answer := svc.post(t, "/discover", envelope(beckn.ActionDiscover, map[string]any{ - "intent": spatial(dwithin(providerGeoPath, majestic, 5000)), - "limit": 101, - })) - if answer.status != http.StatusBadRequest { - t.Fatalf("limit 101 answered %d, want 400\nbody: %s", answer.status, answer.body) - } - if code := answer.nack(t).Message.Error.Code; code != beckn.CodeSchemaValidationFailed { - t.Errorf("the refusal is %s, want %s", code, beckn.CodeSchemaValidationFailed) - } -} diff --git a/tests/acceptance/suite_test.go b/tests/acceptance/suite_test.go index 701d7f0..905fa7d 100644 --- a/tests/acceptance/suite_test.go +++ b/tests/acceptance/suite_test.go @@ -350,9 +350,7 @@ func (s *service) discoverOn(t *testing.T, networkID string, intent map[string]a // discoverPaged is discover with an explicit page size. // -// The page is a member of DiscoverAction, beside the intent rather than inside -// it (A28). It moved there from the query string, and this was the one call -// site to move. +// The page is `limit` in the message, beside the intent. // // Two scenarios need it because the default page is twenty and their fixtures // are forty: a scenario asserting on a count would otherwise be asserting on From 47a8714bf6ccae4ba3c6efb81ebbad30e84fe00d Mon Sep 17 00:00:00 2001 From: Prajwal TD <157502711+tdprajwal@users.noreply.github.com> Date: Mon, 26 Oct 2026 11:26:40 +0530 Subject: [PATCH 12/23] fix: refuse discover pagination left on the query string [#51] 0.1.0 read ?limit=/?offset=; ignoring them now would silently return the default page. Refuse with SCH_INVALID_FORMAT naming the new location. BREAKING CHANGE: discover pagination moved from ?limit=/?offset= to message.limit/message.offset; the query parameters are now refused with 400. --- src/discover/controller.go | 20 +++++++++++++++++++- src/discover/controller_test.go | 26 ++++++++++++++++++++++++++ 2 files changed, 45 insertions(+), 1 deletion(-) diff --git a/src/discover/controller.go b/src/discover/controller.go index a205691..04de76e 100644 --- a/src/discover/controller.go +++ b/src/discover/controller.go @@ -3,6 +3,7 @@ package discover import ( "encoding/json" "net/http" + "net/url" "strings" "time" @@ -65,7 +66,11 @@ func (c *Controller) Discover(w http.ResponseWriter, r *http.Request) { observeIntent(ctx, envelope.Context, action.Intent) - page := Page{Limit: action.Limit, Offset: action.Offset} + page, err := pageFrom(r.URL.Query(), action) + if err != nil { + httpx.WriteNack(ctx, w, c.errors, envelope.Context.MessageID, err) + return + } answer, degraded, err := c.service.Discover(ctx, envelope.Context, action.Intent, page) if err != nil { @@ -90,6 +95,19 @@ func (c *Controller) Discover(w http.ResponseWriter, r *http.Request) { } } +// pageFrom reads the page off the body. +// +// 0.1.0 read it off the query string. Ignoring a leftover parameter would +// answer the default page with a 200, and a client walking offsets would loop +// on page one; refusing it says where the page lives now. +func pageFrom(query url.Values, action beckn.DiscoverAction) (Page, error) { + if query.Has("limit") || query.Has("offset") { + return Page{}, apperrors.Schema(beckn.CodeSchemaInvalidFormat, + "pagination moved from the query string to message.limit and message.offset").At(messageRoot) + } + return Page{Limit: action.Limit, Offset: action.Offset}, nil +} + // responseContext turns the request's envelope into the response's. The // correlation handles are echoed rather than regenerated: they are how the caller // ties this answer to the request it made. diff --git a/src/discover/controller_test.go b/src/discover/controller_test.go index 72e0d11..3fadd17 100644 --- a/src/discover/controller_test.go +++ b/src/discover/controller_test.go @@ -329,6 +329,32 @@ func TestLimitAndOffsetAreReadFromTheBody(t *testing.T) { } } +// The query string carried pagination in 0.1.0. Ignoring it would answer the +// default page with a 200, and a client walking offsets would loop on page one, +// so a leftover parameter is refused and the refusal names the new location. +func TestQueryStringPaginationIsRefused(t *testing.T) { + for _, path := range []string{"/discover?limit=5", "/discover?offset=40"} { + t.Run(path, func(t *testing.T) { + repo := &stubRepo{capabilities: everything()} + + recorder := post(t, mount(t, repo, settings()), path, wheat) + if recorder.Code != http.StatusBadRequest { + t.Fatalf("status = %d, want 400; body: %s", recorder.Code, recorder.Body) + } + body := recorder.Body.String() + if !strings.Contains(body, string(beckn.CodeSchemaInvalidFormat)) { + t.Errorf("body = %s, want SCH_INVALID_FORMAT", body) + } + if !strings.Contains(body, "message.limit") { + t.Errorf("body = %s, want the refusal to name message.limit/offset", body) + } + if repo.calls != 0 { + t.Errorf("the backend was searched %d times; a refused page runs no query", repo.calls) + } + }) + } +} + // A limit that is not a number is refused rather than read as zero. Zero means // "the default page size", so a typo would be answered confidently with a page // the caller did not ask for. From 8ecd0c4f2b772930d1563f83768a7c9f7ffe0c79 Mon Sep 17 00:00:00 2001 From: Prajwal TD <157502711+tdprajwal@users.noreply.github.com> Date: Tue, 27 Oct 2026 14:16:15 +0530 Subject: [PATCH 13/23] feat: return the capped result total in discover pagination [#51] Total is the ranked list's length before paging, capped at the retrieval depth so it never points past the last page a caller can fetch. --- src/beckn/actions.go | 4 ++-- src/discover/service.go | 6 +++++- src/discover/service_test.go | 17 +++++++++++++++++ src/domain/query.go | 5 +++-- src/storage/conformance/discover.go | 10 ++++++---- src/storage/memory/search.go | 1 + src/storage/postgres/search_repository.go | 2 +- 7 files changed, 35 insertions(+), 10 deletions(-) diff --git a/src/beckn/actions.go b/src/beckn/actions.go index 809867e..548cda0 100644 --- a/src/beckn/actions.go +++ b/src/beckn/actions.go @@ -154,8 +154,8 @@ type OnDiscoverAction struct { Pagination *Pagination `json:"pagination,omitempty"` } -// Pagination is the page the answer applied, after default and clamp. Total is -// declared by the schema and not yet set. +// Pagination is the page the answer applied, after default and clamp, and the +// number of results that can be paged through. type Pagination struct { Limit int `json:"limit"` Offset int `json:"offset"` diff --git a/src/discover/service.go b/src/discover/service.go index 38e728b..cca9134 100644 --- a/src/discover/service.go +++ b/src/discover/service.go @@ -69,11 +69,15 @@ func (s *Service) Discover( degraded = append(degraded, result.Degraded...) observeRetrieval(ctx, modes, degraded) + // Capped at the retrieval depth, the last page a caller can fetch; + // a true count past it is a count query per request (A19). + total := min(result.Total, s.cfg.Search.MaxCandidatesPerMode) + return beckn.OnDiscoverAction{ Catalogs: render(result.Catalogs), // What MapIntent applied, not what was asked: a caller that sent 5000 // must read back the 100 it got, or its next offset skips results. - Pagination: &beckn.Pagination{Limit: query.Limit, Offset: query.Offset}, + Pagination: &beckn.Pagination{Limit: query.Limit, Offset: query.Offset, Total: &total}, }, degraded, nil } diff --git a/src/discover/service_test.go b/src/discover/service_test.go index 87f2472..543a1cf 100644 --- a/src/discover/service_test.go +++ b/src/discover/service_test.go @@ -397,6 +397,23 @@ func TestAnOversizeLimitIsClampedAndAnUnreachablePageIsRefused(t *testing.T) { } } +// Total is what the backend matched, capped at the retrieval depth: past it no +// page can be fetched, so a larger number would point the caller at a refusal. +func TestTheTotalIsReturnedAndCappedAtTheRetrievalDepth(t *testing.T) { + for matched, want := range map[int]int{42: 42, 900: settings().Search.MaxCandidatesPerMode} { + repo := &stubRepo{capabilities: everything(), result: domain.SearchResult{Total: matched}} + + answer, _, err := discover.NewService(repo, settings()).Discover( + t.Context(), beckn.Context{}, beckn.Intent{TextSearch: "wheat"}, discover.Page{}) + if err != nil { + t.Fatalf("Discover: %v", err) + } + if got := answer.Pagination.Total; got == nil || *got != want { + t.Errorf("matched %d: Total = %v, want %d", matched, got, want) + } + } +} + // A fatal mapper fault refuses the request and runs no query. Continuing would // WIDEN it, and a widened answer is indistinguishable at the caller from a // correct one. diff --git a/src/domain/query.go b/src/domain/query.go index f6270e4..8beaee6 100644 --- a/src/domain/query.go +++ b/src/domain/query.go @@ -125,11 +125,12 @@ type SearchQuery struct { // SearchResult is one page of an answered query. // // Degraded names the retrieval modes that did not contribute and reaches the -// caller as the X-Beckn-Degraded header, not a body key (C11). There is no -// Total (A19). +// caller as the X-Beckn-Degraded header, not a body key (C11). Total counts +// the ranked ids before paging, not a separate count query (A19). type SearchResult struct { Catalogs []Catalog Degraded []string + Total int } // Scope is the request-wide gate every retrieval mode reads (A6). diff --git a/src/storage/conformance/discover.go b/src/storage/conformance/discover.go index 5d97b95..cbabfad 100644 --- a/src/storage/conformance/discover.go +++ b/src/storage/conformance/discover.go @@ -642,10 +642,12 @@ func pageTwoDoesNotOverlapPageOne() Case { t.Errorf("page two is %v, want [r-02 r-03]", got) } - // There is no total to assert (A19). What remains is the property - // a caller walking pages actually depends on: the offset SLICES a - // stable order rather than re-ranking it, so page two holds the - // next two and not two of the same. + if first.Total != 5 || second.Total != 5 { + t.Errorf("Total = %d and %d, want 5 on both pages", first.Total, second.Total) + } + + // The offset SLICES a stable order rather than re-ranking it, so + // page two holds the next two and not two of the same. }, } } diff --git a/src/storage/memory/search.go b/src/storage/memory/search.go index 1063c9e..0e88f94 100644 --- a/src/storage/memory/search.go +++ b/src/storage/memory/search.go @@ -65,6 +65,7 @@ func (r *Repository) Search( return domain.SearchResult{ Catalogs: r.hydrate(pageOf(page, query.Offset, query.Limit), scope), Degraded: degraded, + Total: len(page), }, nil } diff --git a/src/storage/postgres/search_repository.go b/src/storage/postgres/search_repository.go index 8ba65af..e7cb62e 100644 --- a/src/storage/postgres/search_repository.go +++ b/src/storage/postgres/search_repository.go @@ -128,7 +128,7 @@ func (s *SearchRepository) Search( if err != nil { return domain.SearchResult{}, err } - return domain.SearchResult{Catalogs: catalogs, Degraded: degraded}, nil + return domain.SearchResult{Catalogs: catalogs, Degraded: degraded, Total: len(fused)}, nil } // withinRetrievalDepth refuses a page past what any mode retrieves. Checked From ff7b9fd2bffd0b3f44efffab92239e86ec553f11 Mon Sep 17 00:00:00 2001 From: Prajwal TD <157502711+tdprajwal@users.noreply.github.com> Date: Tue, 27 Oct 2026 19:39:48 +0530 Subject: [PATCH 14/23] fix: align discover pagination with the OAN spec at a8451c4 [#51] The pinned dfc1097 capped the request limit at 100, so the validator refused an oversize limit before the clamp could run. Pin to a8451c4, bind the response to DiscoverPagination with a required total, and move the pagination decisions from code comments into ADR-0017. --- docs/adr/0017-discover-pagination.md | 50 +++++++ docs/adr/README.md | 1 + src/beckn/actions.go | 28 ++-- src/beckn/schema_conformance_test.go | 38 +----- src/discover/controller.go | 6 +- src/discover/controller_test.go | 7 +- src/discover/intent_mapper.go | 9 +- src/discover/service.go | 20 ++- src/discover/service_test.go | 7 +- src/domain/query.go | 2 +- src/platform/config/config.go | 10 +- src/storage/conformance/discover.go | 4 +- src/storage/postgres/fusion.go | 6 +- tests/testdata/beckn-v2.0.0.yaml | 189 +++++++++++++++++++++------ 14 files changed, 238 insertions(+), 139 deletions(-) create mode 100644 docs/adr/0017-discover-pagination.md diff --git a/docs/adr/0017-discover-pagination.md b/docs/adr/0017-discover-pagination.md new file mode 100644 index 0000000..3e810c4 --- /dev/null +++ b/docs/adr/0017-discover-pagination.md @@ -0,0 +1,50 @@ +# ADR-0017 — Discover pagination: signed body, applied page, capped `total` + +**Status:** Accepted +**Date:** 2026-10-27 + +## Context + +0.1.0 read `limit` and `offset` from the query string, which the request +signature does not cover. The OAN profile of the spec moves them into +`DiscoverAction` and answers with `DiscoverPagination` (`total`, `limit`, +`offset`, all required), counted in resources rather than catalogs. + +## Decision + +1. **The page is read from the signed body.** `?limit=` or `?offset=` on + `/discover` is refused with `SCH_INVALID_FORMAT`, naming + `message.limit`/`message.offset`. +2. **The unit is the resource.** The page is a slice of ranked resource ids; + a catalog may span pages. +3. **An oversize `limit` is clamped to `maxPageSize`; a page past the + retrieval depth is refused.** `maxPageSize` may not sit below + `defaultPageSize`, which config validation enforces. +4. **The response echoes the applied page**, not the requested one. `limit` + is the page size applied, not the count on the page: a short last page + still reports it. +5. **`total` is the ranked list's length, capped at + `maxCandidatesPerMode`** — the last page a caller can fetch. +6. **Order is stable across pages.** RRF breaks score ties by id. + +## Alternatives considered + +- **Ignore query-string pagination.** Answers the default page with a 200; a + client walking offsets loops on page one with nothing telling it why. +- **Refuse an oversize `limit`.** The caller would get no results for a + query that has them; clamping and echoing the applied value loses nothing. +- **Echo the requested page.** A caller that asked for 5000 and got 100 would + compute its next offset from 5000 and skip results. +- **An exact `total` from a count query.** About 100x the cost of the search + it accompanies, measured in A19, for a number no caller can page to. +- **Order by score alone.** Equal scores are common — every id only one mode + returned, at the same rank — and two that swap between runs put one result + on two pages and another on none. + +## Consequences + +- Breaking for 0.1.0 callers that paginate with the query string. +- `total` understates a broad query with more matches than the retrieval + depth. The spec allows this: a CN must treat it as the end of what is + pageable, not as an absolute match count. +- A page past the retrieval depth is a 400, not an empty page. diff --git a/docs/adr/README.md b/docs/adr/README.md index aee8be0..6a073df 100644 --- a/docs/adr/README.md +++ b/docs/adr/README.md @@ -29,6 +29,7 @@ Start from [`0000-template.md`](0000-template.md). | [0014](0014-seams-that-ship-with-only-a-no-op.md) | `CatalogReplicator`: what a seam must carry to ship | A7 | | [0015](0015-master-catalogs-and-inheritance-refused.md) | Master catalogs and resource inheritance are refused at intake | A1 | | [0016](0016-router-net-http-servemux.md) | `net/http.ServeMux` for HTTP routing, superseding 0001 | D1 | +| [0017](0017-discover-pagination.md) | Discover pagination: signed body, applied page, capped `total` | #51 | **Source** points at the design document: `D` rows are the Technology Decisions table, `T` rows are TRD Alignment, `A` rows are Amendments. diff --git a/src/beckn/actions.go b/src/beckn/actions.go index 548cda0..dc6a8dd 100644 --- a/src/beckn/actions.go +++ b/src/beckn/actions.go @@ -134,8 +134,7 @@ type CatalogStats struct { CategoryCount int `json:"categoryCount"` } -// DiscoverAction is the `message` of a discover request. Limit and Offset are -// the page; zero means "the default". +// DiscoverAction is the `message` of a discover request. type DiscoverAction struct { Intent Intent `json:"intent"` Limit int `json:"limit,omitempty"` @@ -144,20 +143,19 @@ type DiscoverAction struct { // OnDiscoverAction is the `message` of the discover response. // -// It must NOT grow a Degraded field. The schema is additionalProperties:false -// with `catalogs` and `pagination` as its only properties, so an extra key is a -// response that fails its own schema — and `omitempty` would hide that on the -// ordinary path while shipping the invalid body on the degraded one, the path -// that matters. The degraded list travels as X-Beckn-Degraded (C11). +// It must NOT grow a Degraded field. The v2.0.0 schema is +// additionalProperties:false with only `catalogs` and `pagination`, so an extra +// key is a response that fails its own schema — and `omitempty` would hide that +// on the ordinary path while shipping the invalid body on the degraded one, the +// path that matters. The degraded list travels as X-Beckn-Degraded (C11). type OnDiscoverAction struct { - Catalogs []Catalog `json:"catalogs"` - Pagination *Pagination `json:"pagination,omitempty"` + Catalogs []Catalog `json:"catalogs"` + Pagination *DiscoverPagination `json:"pagination,omitempty"` } -// Pagination is the page the answer applied, after default and clamp, and the -// number of results that can be paged through. -type Pagination struct { - Limit int `json:"limit"` - Offset int `json:"offset"` - Total *int `json:"total,omitempty"` +// DiscoverPagination is the page the answer applied (ADR-0017). +type DiscoverPagination struct { + Limit int `json:"limit"` + Offset int `json:"offset"` + Total int `json:"total"` } diff --git a/src/beckn/schema_conformance_test.go b/src/beckn/schema_conformance_test.go index 9ec5ef5..8574c43 100644 --- a/src/beckn/schema_conformance_test.go +++ b/src/beckn/schema_conformance_test.go @@ -69,7 +69,7 @@ func bindings() []binding { {"DiscoverAction", DiscoverAction{}, schemaPath("DiscoverAction")}, {"OnDiscoverAction", OnDiscoverAction{}, schemaPath("OnDiscoverAction")}, - {"Pagination", Pagination{}, schemaPath("Pagination")}, + {"DiscoverPagination", DiscoverPagination{}, schemaPath("DiscoverPagination")}, {"Nack", Nack{}, schemaPath("NackBadRequest")}, {"NackMessage", NackMessage{}, schemaPath("NackBadRequest", "properties", "message")}, @@ -135,7 +135,7 @@ func TestWireStructsMatchTheSpecification(t *testing.T) { var found []deviation for _, b := range bindings() { - found = append(found, deviations(t, b, flattenAllOf(t, spec, resolve(t, spec, b.path)))...) + found = append(found, deviations(t, b, resolve(t, spec, b.path))...) } sort.Slice(found, func(i, j int) bool { return found[i].key < found[j].key }) @@ -158,40 +158,6 @@ func TestWireStructsMatchTheSpecification(t *testing.T) { } } -// flattenAllOf merges the properties of an allOf node's members into one -// properties map, following each member's local $ref. Pagination is the first -// bound schema composed this way — allOf[PaginationParams, {total}] in the OAN -// profile — and a walk that read only the top-level `properties` would find -// none and fail the binding rather than check it. A node without allOf comes -// back unchanged. -func flattenAllOf(t *testing.T, spec map[string]any, node map[string]any) map[string]any { - t.Helper() - - members, ok := node["allOf"].([]any) - if !ok { - return node - } - - merged := map[string]any{} - for _, member := range members { - part, ok := member.(map[string]any) - if !ok { - t.Fatalf("an allOf member is not an object: %v", member) - } - if ref, isRef := part["$ref"].(string); isRef { - part = resolve(t, spec, strings.Split(strings.TrimPrefix(ref, "#/"), "/")) - } - properties, ok := flattenAllOf(t, spec, part)["properties"].(map[string]any) - if !ok { - t.Fatalf("an allOf member declares no properties: %v", member) - } - for name, schema := range properties { - merged[name] = schema - } - } - return map[string]any{"properties": merged} -} - // deviations compares one struct's JSON tags against its schema node's // properties, in both directions. An extra tag ships a key the schema does not // declare; a missing one drops a key the schema does, which for a service that diff --git a/src/discover/controller.go b/src/discover/controller.go index 04de76e..77b84ac 100644 --- a/src/discover/controller.go +++ b/src/discover/controller.go @@ -95,11 +95,7 @@ func (c *Controller) Discover(w http.ResponseWriter, r *http.Request) { } } -// pageFrom reads the page off the body. -// -// 0.1.0 read it off the query string. Ignoring a leftover parameter would -// answer the default page with a 200, and a client walking offsets would loop -// on page one; refusing it says where the page lives now. +// pageFrom reads the page from the body; ?limit= is refused (ADR-0017). func pageFrom(query url.Values, action beckn.DiscoverAction) (Page, error) { if query.Has("limit") || query.Has("offset") { return Page{}, apperrors.Schema(beckn.CodeSchemaInvalidFormat, diff --git a/src/discover/controller_test.go b/src/discover/controller_test.go index 3fadd17..a15c1b7 100644 --- a/src/discover/controller_test.go +++ b/src/discover/controller_test.go @@ -312,8 +312,7 @@ func TestAnUnavailableModeIsA400WhenTheDeploymentAsksToBe(t *testing.T) { } } -// Pagination arrives in the body beside the intent, and an oversize limit is clamped rather than refused (C11's neighbour: the caller -// still gets the results they asked about). +// Pagination arrives in the body; an oversize limit is clamped (ADR-0017). func TestLimitAndOffsetAreReadFromTheBody(t *testing.T) { repo := &stubRepo{capabilities: everything()} handler := mount(t, repo, settings()) @@ -329,9 +328,7 @@ func TestLimitAndOffsetAreReadFromTheBody(t *testing.T) { } } -// The query string carried pagination in 0.1.0. Ignoring it would answer the -// default page with a 200, and a client walking offsets would loop on page one, -// so a leftover parameter is refused and the refusal names the new location. +// A 0.1.0 caller still sending ?limit= must get a 400, not a silent default page. func TestQueryStringPaginationIsRefused(t *testing.T) { for _, path := range []string{"/discover?limit=5", "/discover?offset=40"} { t.Run(path, func(t *testing.T) { diff --git a/src/discover/intent_mapper.go b/src/discover/intent_mapper.go index 5390f09..0231f24 100644 --- a/src/discover/intent_mapper.go +++ b/src/discover/intent_mapper.go @@ -14,7 +14,7 @@ import ( "github.com/OpenAgriNet/discovery-service/src/platform/jsonpath" ) -// Page is the request's pagination, which arrives as HTTP query parameters +// Page is the request's pagination, which arrives in the message body // rather than inside the intent. A struct rather than two adjacent ints, because // `limit, offset` and `offset, limit` compile identically. type Page struct { @@ -207,12 +207,7 @@ func mapSchemaContext(envelope beckn.Context) ([]domain.SchemaFilter, []domain.F return out, faults } -// mapPage resolves the page, clamping what can be clamped honestly and refusing -// what cannot. -// -// A limit over MaxPageSize is clamped, because the caller still gets the results -// they asked about. A page past MaxCandidatesPerMode is refused, because it would -// come back empty and an empty page 26 is indistinguishable from the end. +// mapPage clamps an oversize limit, refuses a page past the depth (ADR-0017). func mapPage(page Page, search config.Search) (limit, offset int, faults []domain.Fault) { limit = page.Limit if limit <= 0 { diff --git a/src/discover/service.go b/src/discover/service.go index cca9134..cf081bb 100644 --- a/src/discover/service.go +++ b/src/discover/service.go @@ -33,10 +33,9 @@ func NewService(repo domain.SearchRepository, cfg config.Config) *Service { return &Service{repo: repo, cfg: cfg} } -// Discover answers one intent with one page of catalogs, the page it applied, -// and the retrieval modes that did not contribute. The degraded list is -// returned beside the message rather than inside it, because it reaches the -// caller as a header (C11). +// Discover answers one intent with one page of catalogs and the retrieval modes +// that did not contribute. The degraded list is returned beside the catalogs +// rather than inside them, because it reaches the caller as a header (C11). func (s *Service) Discover( ctx context.Context, envelope beckn.Context, intent beckn.Intent, page Page, ) (beckn.OnDiscoverAction, []string, error) { @@ -69,15 +68,14 @@ func (s *Service) Discover( degraded = append(degraded, result.Degraded...) observeRetrieval(ctx, modes, degraded) - // Capped at the retrieval depth, the last page a caller can fetch; - // a true count past it is a count query per request (A19). - total := min(result.Total, s.cfg.Search.MaxCandidatesPerMode) - return beckn.OnDiscoverAction{ Catalogs: render(result.Catalogs), - // What MapIntent applied, not what was asked: a caller that sent 5000 - // must read back the 100 it got, or its next offset skips results. - Pagination: &beckn.Pagination{Limit: query.Limit, Offset: query.Offset, Total: &total}, + // The applied page, and total capped at the retrieval depth (ADR-0017). + Pagination: &beckn.DiscoverPagination{ + Limit: query.Limit, + Offset: query.Offset, + Total: min(result.Total, s.cfg.Search.MaxCandidatesPerMode), + }, }, degraded, nil } diff --git a/src/discover/service_test.go b/src/discover/service_test.go index 543a1cf..aa67b29 100644 --- a/src/discover/service_test.go +++ b/src/discover/service_test.go @@ -397,8 +397,7 @@ func TestAnOversizeLimitIsClampedAndAnUnreachablePageIsRefused(t *testing.T) { } } -// Total is what the backend matched, capped at the retrieval depth: past it no -// page can be fetched, so a larger number would point the caller at a refusal. +// A total past the retrieval depth would point the caller at a page it cannot fetch. func TestTheTotalIsReturnedAndCappedAtTheRetrievalDepth(t *testing.T) { for matched, want := range map[int]int{42: 42, 900: settings().Search.MaxCandidatesPerMode} { repo := &stubRepo{capabilities: everything(), result: domain.SearchResult{Total: matched}} @@ -408,8 +407,8 @@ func TestTheTotalIsReturnedAndCappedAtTheRetrievalDepth(t *testing.T) { if err != nil { t.Fatalf("Discover: %v", err) } - if got := answer.Pagination.Total; got == nil || *got != want { - t.Errorf("matched %d: Total = %v, want %d", matched, got, want) + if got := answer.Pagination.Total; got != want { + t.Errorf("matched %d: Total = %d, want %d", matched, got, want) } } } diff --git a/src/domain/query.go b/src/domain/query.go index 8beaee6..f6b3b55 100644 --- a/src/domain/query.go +++ b/src/domain/query.go @@ -126,7 +126,7 @@ type SearchQuery struct { // // Degraded names the retrieval modes that did not contribute and reaches the // caller as the X-Beckn-Degraded header, not a body key (C11). Total counts -// the ranked ids before paging, not a separate count query (A19). +// the ranked ids before paging (ADR-0017). type SearchResult struct { Catalogs []Catalog Degraded []string diff --git a/src/platform/config/config.go b/src/platform/config/config.go index 0d3bb45..d98d2f6 100644 --- a/src/platform/config/config.go +++ b/src/platform/config/config.go @@ -219,19 +219,19 @@ type Validation struct { // beckn.yaml is fetched at boot rather than baked into the image, so a // deployment names the document it validates against. // - // The default is a TAG, and that is the whole reason there can be one. This + // The default is a SHA, and that is the whole reason there can be one. This // field carried no default until 2026-09-09, on the argument that which spec // a network trusts is not a repository-wide decision — true of a branch, // which would let an upstream merge change the validator under a running - // deployment without a deploy. A pinned commit of the OAN profile (which adds - // discover pagination) cannot move, so the default names one immutable - // document and a network that trusts a different one still overrides it. tests/testdata/beckn-v2.0.0.yaml is byte-identical + // deployment without a deploy. A pinned commit cannot move, so the + // default names one immutable document and a network that trusts a different + // one still overrides it. tests/testdata/beckn-v2.0.0.yaml is byte-identical // to it, so the fetch path and an air-gapped mount agree. // // The cost is that boot now makes a network call before falling back to // SpecCachePath. LoadSpecIndex already warns loudly and falls back, so an // offline boot is slower and noisier, not broken. - SpecURL string `env:"VALIDATION_SPEC_URL" envDefault:"https://raw.githubusercontent.com/OpenAgriNet/protocol-specifications-v2/dfc109759b3781fde74539adc79df63fccbadb83/api/v2.0.0/beckn.yaml"` + SpecURL string `env:"VALIDATION_SPEC_URL" envDefault:"https://raw.githubusercontent.com/OpenAgriNet/protocol-specifications-v2/a8451c48909e27bb86e26dec21ce7f71c5cb90b2/api/v2.0.0/beckn.yaml"` // Where the fetched spec is cached, and what an air-gapped deployment // mounts in place of the fetch. diff --git a/src/storage/conformance/discover.go b/src/storage/conformance/discover.go index cbabfad..370559c 100644 --- a/src/storage/conformance/discover.go +++ b/src/storage/conformance/discover.go @@ -30,8 +30,8 @@ import ( // not the order: Postgres ranks by ts_rank_cd, memory has no relevance to // rank by. They agree on the stable (catalog_id, id) tail, which the one // deliberately text-free pagination case pins. -// - No case asserts a total; there is none (A19). What a paginating caller -// depends on is pinned instead — the offset SLICES a stable order. +// - The pagination case asserts Total and that the offset SLICES a stable +// order. // - No case names `jsonpath` to SEARCH with: since Task 22 Postgres executes // the subset and memory declines it, which is what puts the filter's own // tests on the Postgres side. The one case naming it asserts what a backend diff --git a/src/storage/postgres/fusion.go b/src/storage/postgres/fusion.go index 787495f..90cf920 100644 --- a/src/storage/postgres/fusion.go +++ b/src/storage/postgres/fusion.go @@ -36,11 +36,7 @@ func RRF(ranked ...[]string) []string { } } - // By score, then by id. The tiebreak is not cosmetic: the page is a SLICE of - // this list, so two equally-scored ids that swap between runs of the same - // query put one of them on both page 1 and page 2 and the other on neither. - // Ties are the common case — every id only one mode returned, at the same - // rank, scores identically. + // By score, then by id, so pages are stable (ADR-0017). slices.SortFunc(order, func(left, right string) int { if byScore := cmp.Compare(scores[right], scores[left]); byScore != 0 { return byScore diff --git a/tests/testdata/beckn-v2.0.0.yaml b/tests/testdata/beckn-v2.0.0.yaml index 72a8386..95a8663 100644 --- a/tests/testdata/beckn-v2.0.0.yaml +++ b/tests/testdata/beckn-v2.0.0.yaml @@ -166,6 +166,24 @@ paths: const: discover message: $ref: '#/components/schemas/DiscoverAction' + examples: + paginatedDiscover: + summary: Second page of results, counted in resources (limit/offset) + value: + context: + version: '2.0.0' + action: discover + messageId: 770e8400-e29b-41d4-a716-446655440004 + transactionId: 770e8400-e29b-41d4-a716-446655440005 + timestamp: '2026-03-04T11:00:00.000Z' + bapId: bap.shoply.com + bapUri: https://bap.shoply.com + networkId: retail-network + message: + intent: + textSearch: premium laptop + limit: 3 + offset: 3 responses: '200': $ref: '#/components/responses/Ack' @@ -220,6 +238,54 @@ paths: const: on_discover message: $ref: '#/components/schemas/OnDiscoverAction' + examples: + paginatedOnDiscover: + summary: Second page of a result set paginated by resource. The + three resources span two catalogs, and CAT-TECHMART-2026 + continues from the previous page. + value: + context: + version: '2.0.0' + action: on_discover + messageId: 770e8400-e29b-41d4-a716-446655440004 + transactionId: 770e8400-e29b-41d4-a716-446655440005 + timestamp: '2026-03-04T11:00:02.000Z' + bapId: bap.shoply.com + bapUri: https://bap.shoply.com + bppId: bpp.techmart.com + bppUri: https://bpp.techmart.com + networkId: retail-network + message: + catalogs: + - id: CAT-TECHMART-2026 + descriptor: + name: TechMart Product Catalog + provider: + id: provider-techmart-001 + descriptor: + name: TechMart Electronics + resources: + - id: ITEM-LAPTOP-004 + descriptor: + name: Premium Laptop Pro 14 + - id: ITEM-LAPTOP-005 + descriptor: + name: Premium Laptop Pro 16 + - id: CAT-GADGETHUB-2026 + descriptor: + name: GadgetHub Catalog + provider: + id: provider-gadgethub-002 + descriptor: + name: GadgetHub + resources: + - id: ITEM-LAPTOP-006 + descriptor: + name: Premium Laptop Air + pagination: + total: 137 + limit: 3 + offset: 3 responses: '200': $ref: '#/components/responses/Ack' @@ -2291,9 +2357,23 @@ components: intent: $ref: '#/components/schemas/Intent' limit: - $ref: '#/components/schemas/PaginationParams/properties/limit' + type: integer + minimum: 1 + description: >- + Maximum number of resources the DS SHOULD return in the + on_discover page, summed across the catalogs array. If omitted, + the DS applies its own default. No ceiling is fixed here because + the maximum page size is DS policy; a DS clamps an oversize + limit to its own maximum and reports the applied value in + pagination.limit. offset: - $ref: '#/components/schemas/PaginationParams/properties/offset' + type: integer + minimum: 0 + default: 0 + description: >- + Zero-based index, relative to the DS's full aggregated result + set, of the first resource the DS SHOULD return. Counted in + resources, not catalogs. additionalProperties: false required: - intent @@ -2380,7 +2460,7 @@ components: items: $ref: '#/components/schemas/Catalog' pagination: - $ref: '#/components/schemas/Pagination' + $ref: '#/components/schemas/DiscoverPagination' additionalProperties: false OnInitAction: @@ -2989,22 +3069,12 @@ components: type: string format: uuid networkId: - description: "The unique identifier(s) of the Beckn subnet(s) within which\ - \ this transaction is occurring. Actors use this value to resolve the\ - \ subnet's registry URL before performing identity and address lookups.\ - \ The RECOMMENDED format is namespace_id followed by a slash followed\ - \ by registry_id, for example acmenet.org/charge-net.\n\nMAY be a single\ - \ string (single-network transaction) or an array of strings (multi-network\ - \ transaction). When an array is provided, order is significant: actors\ - \ MUST attempt registry resolution starting with the first entry, treating\ - \ it as the primary network, and MAY fall back to subsequent entries if\ - \ resolution against the primary fails." - oneOf: - - type: string - - type: array - items: - type: string - minItems: 1 + description: "The unique identifier of the Beckn subnet within which this\ + \ transaction is occurring. Actors use this value to resolve the subnet's\ + \ registry URL before performing identity and address lookups. The RECOMMENDED\ + \ format is namespace_id followed by a slash followed by registry_id, for\ + \ example acmenet.org/charge-net." + type: string timestamp: description: The date and time at which this message was generated by the sender, expressed in RFC 3339 date-time format. The receiving actor MAY @@ -4516,32 +4586,55 @@ components: ISO 8601 timestamp set by the CS each time the subscription record is modified. - PaginationParams: + DiscoverPagination: type: object - description: >- - Reusable offset-based pagination fields shared between paginated - list requests and the Pagination response metadata that describes - them. On a request, limit/offset specify the caller's desired page; - on a response, they reflect the page actually applied by the server. + description: "Pagination metadata accompanying an /on_discover callback,\ + \ describing the window of results returned relative to the DS's full\ + \ aggregated result set for the discovery intent.\n\nThe unit of pagination\ + \ is the resource, not the catalog: total, limit, and offset all count\ + \ entries of Catalog.resources summed across the catalogs array. A single\ + \ Catalog MAY therefore be split across consecutive pages, repeating its\ + \ id, provider, and bppId on each page while carrying a different subset\ + \ of its resources.\n\nProduced by the DS; consumed by the CN to determine\ + \ whether further pages exist and to construct the next /discover request.\ + \ Distinct from Pagination, which is produced by the CS for the deprecated\ + \ /catalog/search and /catalog/subscription endpoints." properties: + total: + type: integer + minimum: 0 + description: >- + Number of resources reachable by paging, not the count in this + page. This is the count of matches the DS retained, which MAY be + capped at its retrieval depth: a DS that stops ranking after N + candidates reports at most N, because no page beyond that can be + fetched. A CN MUST therefore treat total as the end of what is + pageable and MUST NOT present it as an absolute match count. limit: type: integer minimum: 1 - maximum: 100 description: >- - Maximum number of records in this page. On a request, the - caller's desired page size; if omitted, the server applies a - default. On a response, the page size actually applied. + Page size the DS actually applied, which MAY differ from the + limit the CN requested because the DS clamped it. This is a + ceiling, not a count: the last page carries fewer resources + than limit, and limit does not shrink to match. A CN that + wants the number of resources returned counts the catalogs + array; it has reached the end of the results when offset plus + that count equals total. offset: type: integer minimum: 0 - default: 0 description: >- - Zero-based index of the first record in this page relative to - the full result set. On a request, specifies where to start; - on a response, reflects the offset actually applied. + Zero-based index of the first resource in this page relative to + the DS's full aggregated result set. + required: + - total + - limit + - offset + additionalProperties: false Pagination: + type: object description: "Metadata accompanying a paginated list response that describes\ \ the size, position, and count of the returned result set, enabling callers\ \ to navigate through large result sets using offset-based pagination.\n\n\ @@ -4550,16 +4643,26 @@ components: \ or PN consumes it to determine whether additional pages are available and\ \ to construct subsequent requests.\n\nRelationship: Composed within the response\ \ message objects of /catalog/subscription and /catalog/search endpoints." - allOf: - - $ref: '#/components/schemas/PaginationParams' - - type: object - properties: - total: - type: integer - minimum: 0 - description: >- - Total number of matching records across the entire result - set, not the count in this page. Assigned by the CS. + properties: + total: + type: integer + minimum: 0 + description: >- + Total number of matching records across the entire result set, + not the count in this page. Assigned by the CS. + limit: + type: integer + minimum: 1 + description: >- + Maximum number of records returned in this page. Assigned by + the CS based on the request parameter or CS policy. + offset: + type: integer + minimum: 0 + description: >- + Zero-based index of the first record in this page relative to + the full result set. Assigned by the CS based on the request + parameter. CatalogSearchAction: type: object From 72afbaac4b06da5342cd0fa0921cde2d26063415 Mon Sep 17 00:00:00 2001 From: Manjunath DR Date: Tue, 27 Oct 2026 20:41:20 +0530 Subject: [PATCH 15/23] fix: let the last page be reached at any limit [#51] Both depth guards refused a page whose tail crossed the retrieval depth, but the page slice already truncates that tail. At depth 500, limit 30 refused offset 480 with 30 results still unread, and the caller could never satisfy the spec's end-of-results test (offset + count == total). It only looked right at limit 20, which divides 500 evenly. The guard now reads the offset alone: a page that starts past the depth is refused, one that starts inside it is answered, short if need be. --- src/discover/intent_mapper.go | 6 +++--- src/discover/intent_mapper_test.go | 14 ++++++++------ src/discover/service_test.go | 2 +- src/storage/postgres/search_repository.go | 6 +++--- 4 files changed, 15 insertions(+), 13 deletions(-) diff --git a/src/discover/intent_mapper.go b/src/discover/intent_mapper.go index 0231f24..141deff 100644 --- a/src/discover/intent_mapper.go +++ b/src/discover/intent_mapper.go @@ -222,13 +222,13 @@ func mapPage(page Page, search config.Search) (limit, offset int, faults []domai offset = 0 } - if offset+limit > search.MaxCandidatesPerMode { + if offset >= search.MaxCandidatesPerMode { faults = append(faults, domain.Fault{ Path: "$['offset']", Code: string(beckn.CodeSchemaInvalidFormat), Message: fmt.Sprintf( - "offset %d plus limit %d passes the retrieval depth of %d", - offset, limit, search.MaxCandidatesPerMode), + "offset %d starts past the retrieval depth of %d", + offset, search.MaxCandidatesPerMode), }) } return limit, offset, faults diff --git a/src/discover/intent_mapper_test.go b/src/discover/intent_mapper_test.go index 08e8550..32b1a7e 100644 --- a/src/discover/intent_mapper_test.go +++ b/src/discover/intent_mapper_test.go @@ -501,16 +501,18 @@ func TestValidateConstraintReportsEveryFaultNotJustTheFirst(t *testing.T) { } } -// mapPage's own boundary: `>` refuses, so a page landing EXACTLY on the -// retrieval depth must be answered, not refused. TestLimitIsClampedAndAPagePastTheRetrievalDepthIsRefused -// only asserts the over side of this line. -func TestAPageExactlyAtTheRetrievalDepthIsNotRefused(t *testing.T) { +// mapPage's own boundary: the guard reads the offset alone, so the last +// reachable offset must be answered however large the limit is. A limit that +// overhangs the depth is the tail the page slice truncates, not a refusal — +// otherwise the final results are unreachable at any limit the depth does not +// divide evenly. +func TestTheLastReachableOffsetIsNotRefused(t *testing.T) { cfg := settings() - page := discover.Page{Limit: 100, Offset: cfg.Search.MaxCandidatesPerMode - 100} + page := discover.Page{Limit: 100, Offset: cfg.Search.MaxCandidatesPerMode - 1} _, fatal, _ := discover.MapIntent(beckn.Intent{TextSearch: "wheat"}, beckn.Context{}, page, cfg) if len(fatal) != 0 { - t.Fatalf("fatal = %s, want none — offset+limit lands exactly on the depth, not past it", codesOf(fatal)) + t.Fatalf("fatal = %s, want none — the offset is inside the depth, only the limit overhangs it", codesOf(fatal)) } } diff --git a/src/discover/service_test.go b/src/discover/service_test.go index aa67b29..61bf06e 100644 --- a/src/discover/service_test.go +++ b/src/discover/service_test.go @@ -392,7 +392,7 @@ func TestAnOversizeLimitIsClampedAndAnUnreachablePageIsRefused(t *testing.T) { if _, _, err := service.Discover( t.Context(), beckn.Context{}, beckn.Intent{TextSearch: "wheat"}, - discover.Page{Limit: 20, Offset: 490}); err == nil { + discover.Page{Limit: 20, Offset: 500}); err == nil { t.Error("a page past the retrieval depth was answered; want a refusal naming the boundary") } } diff --git a/src/storage/postgres/search_repository.go b/src/storage/postgres/search_repository.go index e7cb62e..97605b1 100644 --- a/src/storage/postgres/search_repository.go +++ b/src/storage/postgres/search_repository.go @@ -135,10 +135,10 @@ func (s *SearchRepository) Search( // before anything runs, and named in the error so the caller can tell it from // the end of the results. func (s *SearchRepository) withinRetrievalDepth(query domain.SearchQuery) error { - if query.Offset+query.Limit > s.search.MaxCandidatesPerMode { + if query.Offset >= s.search.MaxCandidatesPerMode { return fmt.Errorf( - "%w: offset %d plus limit %d passes the %d ids a mode retrieves", - domain.ErrRetrievalDepth, query.Offset, query.Limit, s.search.MaxCandidatesPerMode) + "%w: offset %d starts past the %d ids a mode retrieves", + domain.ErrRetrievalDepth, query.Offset, s.search.MaxCandidatesPerMode) } return nil } From 2751b88814f3d5e79c094fd423f2ee5c019987da Mon Sep 17 00:00:00 2001 From: Manjunath DR Date: Tue, 27 Oct 2026 21:13:36 +0530 Subject: [PATCH 16/23] fix: keep the page inside the total it is paged against [#51] MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The depth counts ids PER MODE, so fusing two modes yields more ids than either retrieved — 14 from a depth of 8 on the test corpus. Only `total` was trimmed to the depth; the page was still cut from the untrimmed list, so the last page carried more resources than `total` left room for and offset + count == total never came true. Trim the fused list once, before the page is cut, so the page and the total come from the same list. --- src/storage/postgres/read_path_test.go | 52 +++++++++++++++++++++++ src/storage/postgres/search_repository.go | 7 +++ 2 files changed, 59 insertions(+) diff --git a/src/storage/postgres/read_path_test.go b/src/storage/postgres/read_path_test.go index aee05d2..6eb25fa 100644 --- a/src/storage/postgres/read_path_test.go +++ b/src/storage/postgres/read_path_test.go @@ -475,6 +475,58 @@ func TestAPagePastTheRetrievalDepthIsAFaultAndNotAnEmptyPage(t *testing.T) { } } +// The cap counts ids PER MODE, so two modes over the same corpus fuse to more +// ids than either retrieved. Total is the fused list's length capped at the +// depth, so a page cut from an uncapped list runs past the total the caller is +// paging against — it reads the last page, finds more resources on it than +// `total` leaves room for, and `offset + count == total` never comes true. +// +// Two catalogs whose text differs, so lexical and fuzzy rank them differently +// and their top-`depth` lists are not the same ids. One catalog of identical +// lots does not reproduce it: both modes return the same ids and the union is +// the cap. +func TestTheLastPageDoesNotRunPastTheTotal(t *testing.T) { + alpha := make([]domain.ResourcePatch, 0, 12) + for index := range 12 { + alpha = append(alpha, searchable( + fmt.Sprintf("a-%02d", index), fmt.Sprintf("kharif seed lot %02d of the northern district consignment", index), + "", "https://beckn.org/Agri", "SeedLot")) + } + bravo := make([]domain.ResourcePatch, 0, 6) + for index := range 6 { + bravo = append(bravo, searchable( + fmt.Sprintf("b-%02d", index), fmt.Sprintf("khariff %02d", index), + "", "https://beckn.org/Agri", "SeedLot")) + } + + repository, _ := publish(t, nil, + readFixture{catalog: "cat-overrun-a", resources: alpha}, + readFixture{catalog: "cat-overrun-b", resources: bravo}) + + depth := searchConfig().MaxCandidatesPerMode + offset := depth - 2 + + result, err := repository.Search(context.Background(), + domain.SearchQuery{Text: "kharif", Limit: 4, Offset: offset}, bothTextModes) + if err != nil { + t.Fatalf("the last reachable page was refused: %v", err) + } + + if result.Total != depth { + t.Fatalf("Total = %d, want it capped at the depth %d — the corpus must "+ + "fuse to more ids than the cap for this test to mean anything", result.Total, depth) + } + got := 0 + for _, catalog := range result.Catalogs { + got += len(catalog.Resources) + } + if offset+got != result.Total { + t.Errorf("offset %d + %d resources = %d, want it to land on Total %d — "+ + "a caller cannot tell it has reached the end otherwise", + offset, got, offset+got, result.Total) + } +} + // A mode that returns exactly its cap is a mode whose list is a truncation // rather than the answer, and that is the one state the count guards read. The // corpus is wider than the cap on the ordinary query, not on a pathological diff --git a/src/storage/postgres/search_repository.go b/src/storage/postgres/search_repository.go index 97605b1..b99880d 100644 --- a/src/storage/postgres/search_repository.go +++ b/src/storage/postgres/search_repository.go @@ -121,7 +121,14 @@ func (s *SearchRepository) Search( lists = append(lists, ids) } + // Trimmed to the depth BEFORE the page is cut, because the cap counts ids + // per mode and fusing several of them yields more than any one retrieved. + // Total is this list's length, so a page cut from an untrimmed one would run + // past the total the caller pages against. fused := RRF(lists...) + if len(fused) > s.search.MaxCandidatesPerMode { + fused = fused[:s.search.MaxCandidatesPerMode] + } page := pageOf(fused, query.Offset, query.Limit) catalogs, err := s.hydrator.Hydrate(ctx, page, scope) From 8b5facfdb5bd690260583ca03b5642371bc61b2e Mon Sep 17 00:00:00 2001 From: Manjunath DR Date: Tue, 27 Oct 2026 22:01:57 +0530 Subject: [PATCH 17/23] refactor: give the retrieval depth one owner [#51] The depth rule was written in two places and the page was read by a function doing two jobs. - Search.OffsetBeyondDepth and Search.CapToDepth hold the comparison once; the request layer and the storage layer both call them. - pageFrom reads the signed body; rejectQueryStringPagination refuses a page the signature does not cover. - SearchResult.Total says what it holds and where it is capped. - ADR-0017 records why the cap is applied before the page is cut, and why the refusal tests the offset rather than offset + limit. --- docs/adr/0017-discover-pagination.md | 13 ++++++++++++- src/discover/controller.go | 18 ++++++++++++------ src/discover/intent_mapper.go | 2 +- src/discover/service.go | 6 ++++-- src/discover/service_test.go | 8 ++++---- src/domain/query.go | 9 +++++++-- src/platform/config/config.go | 22 +++++++++++++++++++--- src/storage/memory/search.go | 2 ++ src/storage/postgres/search_repository.go | 11 +++-------- 9 files changed, 64 insertions(+), 27 deletions(-) diff --git a/docs/adr/0017-discover-pagination.md b/docs/adr/0017-discover-pagination.md index 3e810c4..f69eb4e 100644 --- a/docs/adr/0017-discover-pagination.md +++ b/docs/adr/0017-discover-pagination.md @@ -25,7 +25,13 @@ signature does not cover. The OAN profile of the spec moves them into still reports it. 5. **`total` is the ranked list's length, capped at `maxCandidatesPerMode`** — the last page a caller can fetch. -6. **Order is stable across pages.** RRF breaks score ties by id. +6. **The cap is applied to the fused list before the page is cut.** + `maxCandidatesPerMode` counts ids *per retrieval mode*, so fusing several + modes yields a list longer than any one of them returned. Slicing first and + capping the count afterwards produces a last page carrying more resources + than `total` leaves room for, and `offset + count == total` never becomes + true. The storage layer caps, and the request layer reports what it is given. +7. **Order is stable across pages.** RRF breaks score ties by id. ## Alternatives considered @@ -48,3 +54,8 @@ signature does not cover. The OAN profile of the spec moves them into depth. The spec allows this: a CN must treat it as the end of what is pageable, not as an absolute match count. - A page past the retrieval depth is a 400, not an empty page. +- `offset >= maxCandidatesPerMode` is the refusal test, not + `offset + limit > maxCandidatesPerMode`. `pageOf` already truncates a page + whose tail runs past the end, so testing the tail refuses the last page at + every limit that does not divide the depth exactly, and the caller never + reaches the `offset + count == total` that marks the end of the results. diff --git a/src/discover/controller.go b/src/discover/controller.go index 77b84ac..b57da02 100644 --- a/src/discover/controller.go +++ b/src/discover/controller.go @@ -66,11 +66,11 @@ func (c *Controller) Discover(w http.ResponseWriter, r *http.Request) { observeIntent(ctx, envelope.Context, action.Intent) - page, err := pageFrom(r.URL.Query(), action) - if err != nil { + if err := rejectQueryStringPagination(r.URL.Query()); err != nil { httpx.WriteNack(ctx, w, c.errors, envelope.Context.MessageID, err) return } + page := pageFrom(action) answer, degraded, err := c.service.Discover(ctx, envelope.Context, action.Intent, page) if err != nil { @@ -95,13 +95,19 @@ func (c *Controller) Discover(w http.ResponseWriter, r *http.Request) { } } -// pageFrom reads the page from the body; ?limit= is refused (ADR-0017). -func pageFrom(query url.Values, action beckn.DiscoverAction) (Page, error) { +// rejectQueryStringPagination refuses a page the request signature does not +// cover (ADR-0017). +func rejectQueryStringPagination(query url.Values) error { if query.Has("limit") || query.Has("offset") { - return Page{}, apperrors.Schema(beckn.CodeSchemaInvalidFormat, + return apperrors.Schema(beckn.CodeSchemaInvalidFormat, "pagination moved from the query string to message.limit and message.offset").At(messageRoot) } - return Page{Limit: action.Limit, Offset: action.Offset}, nil + return nil +} + +// pageFrom reads the page out of the signed body. +func pageFrom(action beckn.DiscoverAction) Page { + return Page{Limit: action.Limit, Offset: action.Offset} } // responseContext turns the request's envelope into the response's. The diff --git a/src/discover/intent_mapper.go b/src/discover/intent_mapper.go index 141deff..0f935a1 100644 --- a/src/discover/intent_mapper.go +++ b/src/discover/intent_mapper.go @@ -222,7 +222,7 @@ func mapPage(page Page, search config.Search) (limit, offset int, faults []domai offset = 0 } - if offset >= search.MaxCandidatesPerMode { + if search.OffsetBeyondDepth(offset) { faults = append(faults, domain.Fault{ Path: "$['offset']", Code: string(beckn.CodeSchemaInvalidFormat), diff --git a/src/discover/service.go b/src/discover/service.go index cf081bb..2b697ae 100644 --- a/src/discover/service.go +++ b/src/discover/service.go @@ -70,11 +70,13 @@ func (s *Service) Discover( return beckn.OnDiscoverAction{ Catalogs: render(result.Catalogs), - // The applied page, and total capped at the retrieval depth (ADR-0017). + // The applied page. Total is capped here, at the one place every + // backend's answer passes through, so a store that ranks deeper than + // the depth cannot report a total no caller can page to (ADR-0017). Pagination: &beckn.DiscoverPagination{ Limit: query.Limit, Offset: query.Offset, - Total: min(result.Total, s.cfg.Search.MaxCandidatesPerMode), + Total: s.cfg.Search.CapToDepth(result.Total), }, }, degraded, nil } diff --git a/src/discover/service_test.go b/src/discover/service_test.go index 61bf06e..a4ed799 100644 --- a/src/discover/service_test.go +++ b/src/discover/service_test.go @@ -81,11 +81,11 @@ func hasMode(modes []domain.Capability, wanted domain.Capability) bool { func codeOf(t *testing.T, err error) beckn.ErrorCode { t.Helper() - fault := apperrors.FromError(err) - if fault == nil { - t.Fatalf("expected a fault, got nil") + if fault := apperrors.FromError(err); fault != nil { + return fault.Code } - return fault.Code + t.Fatalf("expected a fault, got nil") + return "" } // A textSearch on a deployment with no semantic mode answers, and says what it diff --git a/src/domain/query.go b/src/domain/query.go index f6b3b55..bf5e486 100644 --- a/src/domain/query.go +++ b/src/domain/query.go @@ -125,8 +125,13 @@ type SearchQuery struct { // SearchResult is one page of an answered query. // // Degraded names the retrieval modes that did not contribute and reaches the -// caller as the X-Beckn-Degraded header, not a body key (C11). Total counts -// the ranked ids before paging (ADR-0017). +// caller as the X-Beckn-Degraded header, not a body key (C11). +// +// Total is the length of the ranked list the page was cut from, before paging. +// It is NOT yet the number a caller can page to: a backend may rank deeper than +// the retrieval depth, so the request layer caps it at Search.MaxCandidatesPerMode +// before putting it on the wire (ADR-0017). Capping there rather than here is +// what lets a backend that has no depth of its own still answer correctly. type SearchResult struct { Catalogs []Catalog Degraded []string diff --git a/src/platform/config/config.go b/src/platform/config/config.go index d98d2f6..5663641 100644 --- a/src/platform/config/config.go +++ b/src/platform/config/config.go @@ -146,9 +146,10 @@ type Search struct { MaxPageSize int `env:"SEARCH_MAX_PAGE_SIZE" envDefault:"100"` // How many ids one retrieval mode may return into fusion. Also the reachable - // pagination depth: a request whose offset + limit passes it is refused, since - // slicing past the end of the fused list would answer with an empty page that - // reads exactly like the end of the results. + // pagination depth: a request whose offset starts at or past it is refused, + // because slicing past the end of the fused list would answer with an empty + // page that reads exactly like the end of the results. Read it through + // OffsetBeyondDepth and CapToDepth rather than comparing against it directly. MaxCandidatesPerMode int `env:"SEARCH_MAX_CANDIDATES_PER_MODE" envDefault:"500"` // The largest S_DWITHIN radius a caller may ask for. @@ -180,6 +181,21 @@ type Search struct { EnableTextSearch bool `env:"SEARCH_ENABLE_TEXT_SEARCH" envDefault:"true"` } +// OffsetBeyondDepth says whether a page starts at or past the last id a caller +// can reach. The request layer turns a true into a client fault and the storage +// layer into a guard; the comparison itself is written once, here, so the two +// cannot drift into refusing different pages. +func (s Search) OffsetBeyondDepth(offset int) bool { + return offset >= s.MaxCandidatesPerMode +} + +// CapToDepth trims a count to what a caller can page to. Fusing several modes +// yields more ids than any one of them retrieved, so a fused list is capped +// before it is either sliced or reported as a total (ADR-0017). +func (s Search) CapToDepth(n int) int { + return min(n, s.MaxCandidatesPerMode) +} + // Embeddings configures the Embedder seam — one struct for both paths (A3). type Embeddings struct { // noop (the default — semantic search is deferred, A5), hashing (CI), diff --git a/src/storage/memory/search.go b/src/storage/memory/search.go index 0e88f94..ec545a4 100644 --- a/src/storage/memory/search.go +++ b/src/storage/memory/search.go @@ -62,6 +62,8 @@ func (r *Repository) Search( page = nil } + // The full ranked length: this store has no retrieval depth to cap against, + // and domain.SearchResult.Total says the request layer caps it. return domain.SearchResult{ Catalogs: r.hydrate(pageOf(page, query.Offset, query.Limit), scope), Degraded: degraded, diff --git a/src/storage/postgres/search_repository.go b/src/storage/postgres/search_repository.go index b99880d..a733888 100644 --- a/src/storage/postgres/search_repository.go +++ b/src/storage/postgres/search_repository.go @@ -121,14 +121,9 @@ func (s *SearchRepository) Search( lists = append(lists, ids) } - // Trimmed to the depth BEFORE the page is cut, because the cap counts ids - // per mode and fusing several of them yields more than any one retrieved. - // Total is this list's length, so a page cut from an untrimmed one would run - // past the total the caller pages against. + // Capped before the page is cut, not after (ADR-0017). fused := RRF(lists...) - if len(fused) > s.search.MaxCandidatesPerMode { - fused = fused[:s.search.MaxCandidatesPerMode] - } + fused = fused[:s.search.CapToDepth(len(fused))] page := pageOf(fused, query.Offset, query.Limit) catalogs, err := s.hydrator.Hydrate(ctx, page, scope) @@ -142,7 +137,7 @@ func (s *SearchRepository) Search( // before anything runs, and named in the error so the caller can tell it from // the end of the results. func (s *SearchRepository) withinRetrievalDepth(query domain.SearchQuery) error { - if query.Offset >= s.search.MaxCandidatesPerMode { + if s.search.OffsetBeyondDepth(query.Offset) { return fmt.Errorf( "%w: offset %d starts past the %d ids a mode retrieves", domain.ErrRetrievalDepth, query.Offset, s.search.MaxCandidatesPerMode) From 0e32b2232381ddfd3c60eb09654fcf3e33a2412d Mon Sep 17 00:00:00 2001 From: Manjunath DR Date: Tue, 27 Oct 2026 22:34:08 +0530 Subject: [PATCH 18/23] refactor: name the two kinds of ranked list apart [#51] `ranked` meant the ranked modes in one function and the ranked id lists in another, in the same file, and `fold` said nothing about what it folded into. - negotiate returns rankedModes, partitionOutcomes returns rankedIDs. - withinRetrievalDepth reads as a predicate but returns an error, so it is ensureWithinRetrievalDepth. - pageOf's manual clamp is min(). - RRF presizes for the fused worst case: 6 allocations and 79KB against 32 and 169KB, at the same wall time. --- src/storage/memory/search.go | 6 +--- src/storage/postgres/fusion.go | 12 ++++++-- src/storage/postgres/search_repository.go | 36 ++++++++++------------- 3 files changed, 27 insertions(+), 27 deletions(-) diff --git a/src/storage/memory/search.go b/src/storage/memory/search.go index ec545a4..e8c904e 100644 --- a/src/storage/memory/search.go +++ b/src/storage/memory/search.go @@ -288,11 +288,7 @@ func pageOf(ranked []domain.Resource, offset, limit int) []domain.Resource { if offset >= len(ranked) { return nil } - end := offset + limit - if end > len(ranked) { - end = len(ranked) - } - return ranked[offset:end] + return ranked[offset:min(offset+limit, len(ranked))] } // hydrate folds the page back into catalogs, in the PAGE's order. diff --git a/src/storage/postgres/fusion.go b/src/storage/postgres/fusion.go index 90cf920..87e330b 100644 --- a/src/storage/postgres/fusion.go +++ b/src/storage/postgres/fusion.go @@ -22,8 +22,16 @@ const rrfK = 60.0 // list, and intersecting would turn that into an empty page — at the caller, // indistinguishable from "nothing matched". func RRF(ranked ...[]string) []string { - scores := make(map[string]float64) - order := make([]string, 0) + // Sized for the worst case, every list disjoint. Fusing N modes at the + // retrieval depth is thousands of ids; letting the map and slice grow into + // that costs 32 allocations and 169KB against 6 and 79KB presized. Same + // wall time either way — this is for the garbage collector, not latency. + capacity := 0 + for _, list := range ranked { + capacity += len(list) + } + scores := make(map[string]float64, capacity) + order := make([]string, 0, capacity) for _, list := range ranked { for index, id := range list { diff --git a/src/storage/postgres/search_repository.go b/src/storage/postgres/search_repository.go index a733888..abf980a 100644 --- a/src/storage/postgres/search_repository.go +++ b/src/storage/postgres/search_repository.go @@ -93,7 +93,7 @@ type outcome struct { func (s *SearchRepository) Search( ctx context.Context, query domain.SearchQuery, modes []domain.Capability, ) (domain.SearchResult, error) { - if err := s.withinRetrievalDepth(query); err != nil { + if err := s.ensureWithinRetrievalDepth(query); err != nil { return domain.SearchResult{}, err } @@ -105,24 +105,24 @@ func (s *SearchRepository) Search( // calls now(); it is for the backends that have none. scope := domain.Scope{NetworkID: query.NetworkID, Now: time.Now().UTC()} - ranked, filtering, degraded := s.negotiate(modes) + rankedModes, filtering, degraded := s.negotiate(modes) - lists, failed, fatal := fold(s.retrieve(ctx, query, ranked, scope)) + rankedIDs, failed, fatal := partitionOutcomes(s.retrieve(ctx, query, rankedModes, scope)) if fatal != nil { return domain.SearchResult{}, fatal } degraded = append(degraded, failed...) - if len(lists) == 0 && filtering { + if len(rankedIDs) == 0 && filtering { ids, err := s.filterOnly(ctx, query, scope) if err != nil { return domain.SearchResult{}, err } - lists = append(lists, ids) + rankedIDs = append(rankedIDs, ids) } // Capped before the page is cut, not after (ADR-0017). - fused := RRF(lists...) + fused := RRF(rankedIDs...) fused = fused[:s.search.CapToDepth(len(fused))] page := pageOf(fused, query.Offset, query.Limit) @@ -133,10 +133,10 @@ func (s *SearchRepository) Search( return domain.SearchResult{Catalogs: catalogs, Degraded: degraded, Total: len(fused)}, nil } -// withinRetrievalDepth refuses a page past what any mode retrieves. Checked +// ensureWithinRetrievalDepth refuses a page past what any mode retrieves. Checked // before anything runs, and named in the error so the caller can tell it from // the end of the results. -func (s *SearchRepository) withinRetrievalDepth(query domain.SearchQuery) error { +func (s *SearchRepository) ensureWithinRetrievalDepth(query domain.SearchQuery) error { if s.search.OffsetBeyondDepth(query.Offset) { return fmt.Errorf( "%w: offset %d starts past the %d ids a mode retrieves", @@ -161,9 +161,9 @@ func (s *SearchRepository) filterOnly( return ids, nil } -// fold splits the per-mode outcomes into the lists to fuse and the modes to +// partitionOutcomes splits the per-mode outcomes into the lists to fuse and the modes to // report. A failed mode is RECORDED in X-Beckn-Degraded, not fatal. -func fold(outcomes []outcome) (ranked [][]string, degraded []string, fatal error) { +func partitionOutcomes(outcomes []outcome) (rankedIDs [][]string, degraded []string, fatal error) { for _, result := range outcomes { switch { // The one failure that is not the deployment's, and it cannot be @@ -175,10 +175,10 @@ func fold(outcomes []outcome) (ranked [][]string, degraded []string, fatal error case result.err != nil: degraded = append(degraded, string(result.mode)) default: - ranked = append(ranked, result.ids) + rankedIDs = append(rankedIDs, result.ids) } } - return ranked, degraded, nil + return rankedIDs, degraded, nil } // callersFilter re-labels the store's refusal of the caller's own attribute @@ -225,7 +225,7 @@ func callersFilter(query domain.SearchQuery, err error) error { // aSpatialOnlyIntentIsAnsweredRatherThanDegraded holds the two to one answer. func (s *SearchRepository) negotiate( modes []domain.Capability, -) (ranked []domain.Capability, filtering bool, degraded []string) { +) (rankedModes []domain.Capability, filtering bool, degraded []string) { declared := s.Capabilities() for _, mode := range modes { switch { @@ -241,10 +241,10 @@ func (s *SearchRepository) negotiate( // ran. degraded = append(degraded, string(mode)) default: - ranked = append(ranked, mode) + rankedModes = append(rankedModes, mode) } } - return ranked, filtering, degraded + return rankedModes, filtering, degraded } // retrieve fans the ranked modes out and waits for all of them — a barrier, not @@ -280,9 +280,5 @@ func pageOf(fused []string, offset, limit int) []string { if offset >= len(fused) { return nil } - end := offset + limit - if end > len(fused) { - end = len(fused) - } - return fused[offset:end] + return fused[offset:min(offset+limit, len(fused))] } From eb3f96db78c70e2c514278fbe2764beb7dde0501 Mon Sep 17 00:00:00 2001 From: ameersohel45 Date: Wed, 28 Oct 2026 15:18:08 +0530 Subject: [PATCH 19/23] feat: an unstated audience is public, and a search matches the networks it named [#58] MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Two halves of one rule. A publisher who omits visibleTo reaches every network instead of their own, and a caller names the networks it wants rather than being handed some of them. visibleTo keeps its name and signature; the default is one literal. The same literal replaces the one in applyDirectiveDefaults, so neither of the two places that resolve it has moved. context.networkId is a string or a non-empty array (spec release-0.2.0, pinned at 7a9fdf0d — the only schema change in a 39-line diff). beckn.NetworkIDs decodes both and remembers which arrived, because the response echoes the request's context and a caller that sent a string must not get an array back. Everything inside the service reads Networks() and never asks which shape it was. The scope gate collapses to one array overlap. Five queries lose the narg IS NULL dance and the separate wildcard arm for r.visible_to && @network_ids::text[] so '*' appears in no query at all: it is an ordinary entry the caller sends. The default sits in discover.Service and nowhere else, which is what lets the storage layer stop defaulting and visibleOn disappear rather than grow. idx_resources_visible_to is GIN array_ops, which indexes && as it indexes @>. An omitted networkId now means the public catalogs rather than every catalog. That is the one behaviour an existing caller loses without asking, and it is the price of the field meaning the same thing in all three forms. Four scenarios, not a matrix: an unstated audience is public; a search naming one network gets that network; a search naming a network and '*' gets both; the string and array forms answer the same page. --- src/beckn/schema_conformance_test.go | 5 ++ src/beckn/types.go | 61 ++++++++++++++++-- src/discover/controller_test.go | 4 +- src/discover/intent_mapper_test.go | 8 +-- src/discover/service.go | 19 ++++-- src/discover/service_test.go | 54 +++++++++------- src/discover/telemetry.go | 2 +- src/domain/catalog.go | 4 ++ src/domain/gate.go | 2 +- src/domain/query.go | 16 ++--- src/platform/config/config.go | 2 +- src/platform/httpx/envelope_test.go | 4 +- src/platform/middlewares/envelope.go | 6 +- src/platform/telemetry/fact/registry.go | 4 +- .../fact/testdata/registry.golden.txt | 2 +- .../validation/envelope_rules_test.go | 4 +- src/publish/controller_test.go | 4 +- src/publish/mapper.go | 9 ++- src/publish/mapper_test.go | 4 +- src/publish/service.go | 10 +-- src/publish/service_test.go | 20 +++--- src/publish/telemetry.go | 10 ++- src/publish/telemetry_test.go | 14 ++--- src/storage/conformance/discover.go | 60 ++++++++++-------- src/storage/conformance/publish.go | 34 +++++----- src/storage/memory/search.go | 13 +++- src/storage/postgres/discover_sql_test.go | 3 +- src/storage/postgres/gen/discover.sql.go | 39 ++++++------ src/storage/postgres/hydrator.go | 4 +- src/storage/postgres/queries/discover.sql | 19 +++--- src/storage/postgres/read_path_test.go | 8 +-- src/storage/postgres/retrievers.go | 14 ++--- src/storage/postgres/search_repository.go | 2 +- tests/acceptance/defaults_test.go | 26 ++++++-- tests/acceptance/discover_test.go | 63 ++++++++++++++----- tests/acceptance/publish_test.go | 8 ++- tests/acceptance/suite_test.go | 29 +++++++-- tests/testdata/beckn-v2.0.0.yaml | 39 ++++++++++-- 38 files changed, 407 insertions(+), 222 deletions(-) diff --git a/src/beckn/schema_conformance_test.go b/src/beckn/schema_conformance_test.go index 8574c43..cb8850e 100644 --- a/src/beckn/schema_conformance_test.go +++ b/src/beckn/schema_conformance_test.go @@ -329,6 +329,11 @@ func TestEveryWireStructIsBound(t *testing.T) { bound[b.name] = true } + // NetworkIDs models `Context.properties.networkId`, which is a oneOf of a + // string and an array. The walk compares struct fields to schema + // properties, and that node declares none. + bound["NetworkIDs"] = true + for _, name := range exportedStructNames(t) { if !bound[name] { t.Errorf("%s is an exported struct in package beckn with no entry in bindings(); "+ diff --git a/src/beckn/types.go b/src/beckn/types.go index 8a17a27..96a43bc 100644 --- a/src/beckn/types.go +++ b/src/beckn/types.go @@ -22,6 +22,60 @@ import ( "fmt" ) +// NetworkIDs is context.networkId, which the spec declares as one network or a +// non-empty array of them. +// +// It remembers which shape it arrived in, because the response echoes the +// request's context and a caller that sent a string must not get an array back. +type NetworkIDs struct { + ids []string + scalar bool +} + +// OneNetwork and SomeNetworks build the two shapes, for callers that did not +// decode one off the wire. +func OneNetwork(id string) NetworkIDs { return NetworkIDs{ids: []string{id}, scalar: true} } + +// SomeNetworks is the array shape, which stays an array even when it holds one. +func SomeNetworks(ids ...string) NetworkIDs { return NetworkIDs{ids: ids} } + +// Networks is the only way the rest of the service reads this: a plain slice, +// with no question about which shape it came from. +func (n NetworkIDs) Networks() []string { return n.ids } + +// IsZero drives `omitzero`, so an absent networkId stays absent in the echo. +func (n NetworkIDs) IsZero() bool { return len(n.ids) == 0 } + +// UnmarshalJSON takes either shape and refuses anything else, so a caller +// learns at the edge rather than by searching and finding nothing. +func (n *NetworkIDs) UnmarshalJSON(data []byte) error { + if string(data) == "null" { + *n = NetworkIDs{} + return nil + } + + var one string + if err := json.Unmarshal(data, &one); err == nil { + *n = OneNetwork(one) + return nil + } + + var many []string + if err := json.Unmarshal(data, &many); err != nil { + return fmt.Errorf("networkId is a string or an array of strings: %w", err) + } + *n = SomeNetworks(many...) + return nil +} + +// MarshalJSON writes back the shape that arrived. +func (n NetworkIDs) MarshalJSON() ([]byte, error) { + if n.scalar && len(n.ids) == 1 { + return json.Marshal(n.ids[0]) + } + return json.Marshal(n.ids) +} + // Context is the Beckn envelope header that accompanies every request and // callback. // @@ -49,10 +103,9 @@ type Context struct { TransactionID string `json:"transactionId,omitempty"` MessageID string `json:"messageId,omitempty"` - // Optional on both paths, which read its absence differently: on publish it - // means APP_NETWORK_ID, used only to fill an empty visibleTo (C8), and on - // discover it means no network predicate at all. No shared fallback. - NetworkID string `json:"networkId,omitempty"` + // Optional. Publish ignores it; on discover, naming none means the public + // catalogs alone. + NetworkID NetworkIDs `json:"networkId,omitzero"` // RFC 3339, kept as a string so a bad one comes back as a fault naming // `$.context.timestamp` rather than a json.Unmarshal failure that tells the diff --git a/src/discover/controller_test.go b/src/discover/controller_test.go index a15c1b7..48cef47 100644 --- a/src/discover/controller_test.go +++ b/src/discover/controller_test.go @@ -122,8 +122,8 @@ func TestTheResponseIsTheOnDiscoverEnvelope(t *testing.T) { t.Errorf("correlation = %q / %q, want the request's own", envelope.Context.TransactionID, envelope.Context.MessageID) } - if envelope.Context.NetworkID != "mahavistar" { - t.Errorf("networkId = %q, want it echoed", envelope.Context.NetworkID) + if got := envelope.Context.NetworkID.Networks(); len(got) != 1 || got[0] != "mahavistar" { + t.Errorf("networkId = %v, want it echoed", envelope.Context.NetworkID.Networks()) } if envelope.Context.Version != beckn.Version { t.Errorf("version = %q, want %q", envelope.Context.Version, beckn.Version) diff --git a/src/discover/intent_mapper_test.go b/src/discover/intent_mapper_test.go index 32b1a7e..8fb96cb 100644 --- a/src/discover/intent_mapper_test.go +++ b/src/discover/intent_mapper_test.go @@ -327,14 +327,14 @@ func TestLimitIsClampedAndAPagePastTheRetrievalDepthIsRefused(t *testing.T) { } // NetworkID is NOT set here. The service sets it from the envelope, and empty -// means EVERY network; defaulting it in the mapper would quietly put discover -// back to single-network scoping. +// means the public catalogs; defaulting it in the mapper would quietly put +// discover back to single-network scoping. func TestTheMapperLeavesNetworkScopingToTheService(t *testing.T) { envelope := beckn.Context{SchemaContext: []string{"https://beckn.org/Agri"}} query, _, _ := discover.MapIntent(beckn.Intent{}, envelope, discover.Page{}, settings()) - if query.NetworkID != "" { - t.Errorf("NetworkID = %q, want empty", query.NetworkID) + if len(query.NetworkIDs) != 0 { + t.Errorf("NetworkIDs = %v, want empty", query.NetworkIDs) } } diff --git a/src/discover/service.go b/src/discover/service.go index 2b697ae..fc23352 100644 --- a/src/discover/service.go +++ b/src/discover/service.go @@ -45,12 +45,10 @@ func (s *Service) Discover( } reportPartials(ctx, partial) - // From the ENVELOPE, and NOT defaulted to config.App.Network. Empty means - // EVERY network and the repository emits no network predicate at all. - // config.App.Network is publish's default for an empty visibleTo (C8) — a - // different field answering a different question — and reusing it here puts - // discover back to single-network scoping (scenario 29). - query.NetworkID = envelope.NetworkID + // From the ENVELOPE, and NOT defaulted to config.App.Network: naming no + // network means the public catalogs, not this deployment's own. Resolved + // HERE so no backend carries the rule a second time. + query.NetworkIDs = scopedNetworks(envelope.NetworkID) modes, degraded, err := s.negotiate(query) if err != nil { @@ -288,3 +286,12 @@ func reportPartials(ctx context.Context, partial []domain.Fault) { zap.String("reason", fault.Message)) } } + +// scopedNetworks is what the caller asked for, or the public catalogs when +// they asked for nothing. +func scopedNetworks(asked beckn.NetworkIDs) []string { + if networks := asked.Networks(); len(networks) > 0 { + return networks + } + return []string{domain.NetworkWildcard} +} diff --git a/src/discover/service_test.go b/src/discover/service_test.go index a4ed799..ef0a7eb 100644 --- a/src/discover/service_test.go +++ b/src/discover/service_test.go @@ -334,13 +334,13 @@ func TestTwoDegradedModesAreBothReturned(t *testing.T) { } } -// Scenario 29, from the service's side: an omitted networkId searches EVERY -// network, and does NOT fall back to APP_NETWORK_ID. +// An omitted networkId becomes the wildcard HERE, not in a backend, and does +// NOT fall back to APP_NETWORK_ID. // // The config carries a network id precisely so the assertion has something to // fail against: with App.Network empty, a service that wrongly defaulted would // pass this test by coincidence. -func TestAnOmittedNetworkIDSearchesEveryNetwork(t *testing.T) { +func TestAnOmittedNetworkIDBecomesTheWildcard(t *testing.T) { repo := &stubRepo{capabilities: everything()} cfg := settings() @@ -351,24 +351,34 @@ func TestAnOmittedNetworkIDSearchesEveryNetwork(t *testing.T) { if err != nil { t.Fatalf("Discover: %v", err) } - if repo.gotQuery.NetworkID != "" { - t.Errorf("NetworkID = %q, want unscoped — publish's visibleTo default is a different question (C8)", - repo.gotQuery.NetworkID) + if !slices.Equal(repo.gotQuery.NetworkIDs, []string{domain.NetworkWildcard}) { + t.Errorf("NetworkIDs = %v, want [*]", repo.gotQuery.NetworkIDs) } } -// The other half: a networkId that IS sent scopes the search to it. -func TestANetworkIDThatWasSentScopesTheSearch(t *testing.T) { - repo := &stubRepo{capabilities: everything()} - - _, _, err := discover.NewService(repo, settings()).Discover( - t.Context(), beckn.Context{NetworkID: "bharatvistar"}, - beckn.Intent{TextSearch: "wheat"}, discover.Page{}) - if err != nil { - t.Fatalf("Discover: %v", err) - } - if repo.gotQuery.NetworkID != "bharatvistar" { - t.Errorf("NetworkID = %q, want the envelope's own", repo.gotQuery.NetworkID) +// Both wire shapes reach the repository as the set the caller named. +func TestTheNetworksThatWereSentScopeTheSearch(t *testing.T) { + for _, test := range []struct { + name string + sent beckn.NetworkIDs + want []string + }{ + {"a string", beckn.OneNetwork("bharatvistar"), []string{"bharatvistar"}}, + {"an array", beckn.SomeNetworks("bharatvistar", "*"), []string{"bharatvistar", "*"}}, + } { + t.Run(test.name, func(t *testing.T) { + repo := &stubRepo{capabilities: everything()} + + _, _, err := discover.NewService(repo, settings()).Discover( + t.Context(), beckn.Context{NetworkID: test.sent}, + beckn.Intent{TextSearch: "wheat"}, discover.Page{}) + if err != nil { + t.Fatalf("Discover: %v", err) + } + if !slices.Equal(repo.gotQuery.NetworkIDs, test.want) { + t.Errorf("NetworkIDs = %v, want %v", repo.gotQuery.NetworkIDs, test.want) + } + }) } } @@ -583,9 +593,11 @@ func TestNoOfferWhoseResourcesAreAllOffThePageIsRendered(t *testing.T) { repo := memory.New(geo.DefaultTestResolution) if _, err := repo.UpsertCatalog(t.Context(), domain.CatalogPatch{ - ID: "c1", - Active: true, - VisibleTo: []string{"mahavistar"}, + ID: "c1", + Active: true, + // Public: this case is about offers, and a narrow audience would make + // the search answer nothing and pass for the wrong reason. + VisibleTo: []string{domain.NetworkWildcard}, Resources: []domain.ResourcePatch{{ID: "wheat"}, {ID: "rice"}}, Offers: []domain.OfferPatch{{ ID: "on-rice", diff --git a/src/discover/telemetry.go b/src/discover/telemetry.go index e7506c1..7955385 100644 --- a/src/discover/telemetry.go +++ b/src/discover/telemetry.go @@ -43,7 +43,7 @@ func observeIntent(ctx context.Context, envelope beckn.Context, intent beckn.Int fact.ObserveStrings(ctx, fact.IntentSpatialOps, ops) } - fact.ObserveBool(ctx, fact.IntentScoped, envelope.NetworkID != "") + fact.ObserveBool(ctx, fact.IntentScoped, !envelope.NetworkID.IsZero()) observeSchemaPredicate(ctx, envelope) } diff --git a/src/domain/catalog.go b/src/domain/catalog.go index 3421ccc..de3cbc6 100644 --- a/src/domain/catalog.go +++ b/src/domain/catalog.go @@ -254,6 +254,10 @@ type TimePeriodPatch struct { EndTime Nullable[TimeOfDay] } +// NetworkWildcard in a VisibleTo list means every network matches it. It is +// what an omitted or empty visibleTo resolves to. +const NetworkWildcard = "*" + // UpdateMode is how a CatalogPatch applies to what is stored (A8). type UpdateMode string diff --git a/src/domain/gate.go b/src/domain/gate.go index b31e80f..b02bf69 100644 --- a/src/domain/gate.go +++ b/src/domain/gate.go @@ -75,7 +75,7 @@ func sameTimeOfDay(left, right *TimeOfDay) bool { // nobody while reporting success. func (c *Catalog) EnsureVisibleTo() { if len(c.VisibleTo) == 0 { - c.VisibleTo = []string{c.NetworkID} + c.VisibleTo = []string{NetworkWildcard} } } diff --git a/src/domain/query.go b/src/domain/query.go index bf5e486..eaf1b91 100644 --- a/src/domain/query.go +++ b/src/domain/query.go @@ -103,8 +103,9 @@ type AttributeFilter struct { type SearchQuery struct { Text string - // "" means UNSCOPED — every network. See Scope. - NetworkID string + // The networks the caller named, already defaulted by discover.Service: a + // backend never sees this empty. See Scope. + NetworkIDs []string // Empty means no schema predicate at all, NOT a predicate matching nothing. Schemas []SchemaFilter @@ -144,13 +145,12 @@ type SearchResult struct { // on it. Postgres ignores it and calls now(); it exists for the backends that // have none — the memory store, and tests that ask what was live at 23:00. // -// NetworkID == "" means UNSCOPED, which every backend must read as "emit no -// network predicate" — never as a literal id, and never by falling back to -// APP_NETWORK_ID, which is publish's visibleTo default (C8) answering a -// different question. +// NetworkIDs is the set a resource's audience must overlap, and it is never +// empty: discover.Service has already turned "the caller named none" into the +// wildcard, so no backend carries that rule a second time. type Scope struct { - NetworkID string - Now time.Time + NetworkIDs []string + Now time.Time } // Capability is one thing a backend can do. A backend that cannot do one diff --git a/src/platform/config/config.go b/src/platform/config/config.go index 5663641..0ddc70a 100644 --- a/src/platform/config/config.go +++ b/src/platform/config/config.go @@ -247,7 +247,7 @@ type Validation struct { // The cost is that boot now makes a network call before falling back to // SpecCachePath. LoadSpecIndex already warns loudly and falls back, so an // offline boot is slower and noisier, not broken. - SpecURL string `env:"VALIDATION_SPEC_URL" envDefault:"https://raw.githubusercontent.com/OpenAgriNet/protocol-specifications-v2/a8451c48909e27bb86e26dec21ce7f71c5cb90b2/api/v2.0.0/beckn.yaml"` + SpecURL string `env:"VALIDATION_SPEC_URL" envDefault:"https://raw.githubusercontent.com/OpenAgriNet/protocol-specifications-v2/7a9fdf0df44cc1393727ebef23b5252d3f7b4eb0/api/v2.0.0/beckn.yaml"` // Where the fetched spec is cached, and what an air-gapped deployment // mounts in place of the fetch. diff --git a/src/platform/httpx/envelope_test.go b/src/platform/httpx/envelope_test.go index 59c682a..eb33c98 100644 --- a/src/platform/httpx/envelope_test.go +++ b/src/platform/httpx/envelope_test.go @@ -31,8 +31,8 @@ func TestParseEnvelopeReadsContextAndMessage(t *testing.T) { if env.Context.Action != beckn.ActionPublish { t.Errorf("Context.Action = %q, want %q", env.Context.Action, beckn.ActionPublish) } - if env.Context.NetworkID != "mahavistar" { - t.Errorf("Context.NetworkID = %q, want mahavistar", env.Context.NetworkID) + if got := env.Context.NetworkID.Networks(); len(got) != 1 || got[0] != "mahavistar" { + t.Errorf("Context.NetworkID = %v, want [mahavistar]", env.Context.NetworkID.Networks()) } // schemaContext is a Context field, not an Intent one — Intent is // additionalProperties:false, so the reference implementation's placement diff --git a/src/platform/middlewares/envelope.go b/src/platform/middlewares/envelope.go index 1f595c6..cb4d2c7 100644 --- a/src/platform/middlewares/envelope.go +++ b/src/platform/middlewares/envelope.go @@ -236,6 +236,11 @@ func correlate(ctx context.Context, envelope beckn.Context) context.Context { fact.ObserveString(ctx, correlator.key, correlator.value) } + // Outside the table because it is a list, not a value (A29). + if networks := envelope.NetworkID.Networks(); len(networks) > 0 { + fact.ObserveStrings(ctx, fact.BecknNetworkID, networks) + } + return logger.With(ctx, fields...) } @@ -269,7 +274,6 @@ func correlators(envelope beckn.Context) []correlator { // undetectably wrong (opentelemetry.md §Span attributes). {envelope.SenderID, fact.SenderID, nil}, {envelope.Version, fact.BecknVersion, nil}, - {envelope.NetworkID, fact.BecknNetworkID, nil}, {envelope.ReceiverID, fact.BecknReceiverID, nil}, } } diff --git a/src/platform/telemetry/fact/registry.go b/src/platform/telemetry/fact/registry.go index 3c68a03..5008a2a 100644 --- a/src/platform/telemetry/fact/registry.go +++ b/src/platform/telemetry/fact/registry.go @@ -356,10 +356,10 @@ var registry = [numKeys]Definition{ Name: "BecknNetworkID", SpanKey: "beckn.networkId", Signals: Span, - Kind: KindString, + Kind: KindStrings, Layer: Local, Cardinality: Unbounded, - Note: "The network the CALLER named, beside network.id which is the one this " + + Note: "The networks the CALLER named, beside network.id which is the one this " + "deployment serves. Two keys because the two disagreeing is the interesting case.", }, BecknTransactionID: { diff --git a/src/platform/telemetry/fact/testdata/registry.golden.txt b/src/platform/telemetry/fact/testdata/registry.golden.txt index f156f89..500489e 100644 --- a/src/platform/telemetry/fact/testdata/registry.golden.txt +++ b/src/platform/telemetry/fact/testdata/registry.golden.txt @@ -19,7 +19,7 @@ 15 Name=RequestID LogKey=request_id Signals=Log Kind=KindString Layer=Local Cardinality=Unbounded Note=Log only. It is this process's handle on a request and nothing outside reads it; span_uuid is the span's identity and traceId the trace's, so a third id on the span would be a third thing to join on. 16 Name=BecknAction SpanKey=beckn.action LogKey=action Signals=Span|Log Kind=KindString Layer=Local Cardinality=Bounded Values=[discover,publish] Note=Normalised, not verbatim: catalog/publish is accepted on the wire (beckn/actions.go's Action* constants) and both spellings resolve to one handler, so both record publish. Emitting context.action as sent would split every publish query in two. No Label bit: Task 25's acquire-wait pair is per-pool and names no dimension, and rate/errors/duration by action come from the collector's spanmetrics connector, not a counter here. 17 Name=BecknVersion SpanKey=beckn.version Signals=Span Kind=KindString Layer=Local Cardinality=Unbounded Note=beckn.Version is const 2.0.0 today, but this is context.version as the caller sent it and Context has no required list (C6), so an envelope can omit it. Unbounded rather than Bounded on one value: the row would otherwise have to be edited on the day it is least likely to be read. - 18 Name=BecknNetworkID SpanKey=beckn.networkId Signals=Span Kind=KindString Layer=Local Cardinality=Unbounded Note=The network the CALLER named, beside network.id which is the one this deployment serves. Two keys because the two disagreeing is the interesting case. + 18 Name=BecknNetworkID SpanKey=beckn.networkId Signals=Span Kind=KindStrings Layer=Local Cardinality=Unbounded Note=The networks the CALLER named, beside network.id which is the one this deployment serves. Two keys because the two disagreeing is the interesting case. 19 Name=BecknTransactionID SpanKey=beckn.transactionId SpanAliases=[{Key:transaction_id AsString:false}] LogKey=transaction_id Signals=Span|Log Kind=KindString Layer=CrossLayer Cardinality=Unbounded Note=I1. onix's network collector rewrites trace_id from an attribute named literally transaction_id (otel-collector-network/config.yaml:23-25), and onix injects traceparent nowhere (SpanKindClient appears in no file there), so while propagation is missing this alias is the only thing joining our span to the adapter's. Renaming two attributes in one service is cheaper than renaming one in three adapters and every collector config. 20 Name=BecknMessageID SpanKey=beckn.messageId SpanAliases=[{Key:message_id AsString:false}] LogKey=message_id Signals=Span|Log Kind=KindString Layer=CrossLayer Cardinality=Unbounded Note=I1, and a searchable tag rather than a join: onix's collector deliberately does NOT map it onto span_id, because several nodes emit spans for one Beckn message and identical span ids would corrupt the trace. 21 Name=BecknReceiverID SpanKey=beckn.receiverId Signals=Span Kind=KindString Layer=Local Cardinality=Unbounded Note=context.receiverId when present — the caller's CLAIM about who it addressed. Emitted beside recipient.id rather than into it: the two disagreeing is a caller addressing a participant that is not us, which is worth a query and impossible to ask if one field holds both. diff --git a/src/platform/validation/envelope_rules_test.go b/src/platform/validation/envelope_rules_test.go index 412fdc4..4a916e2 100644 --- a/src/platform/validation/envelope_rules_test.go +++ b/src/platform/validation/envelope_rules_test.go @@ -157,11 +157,11 @@ func TestATimestampThatIsNotRFC3339IsRejected(t *testing.T) { // would pass the absent case on its own. func TestTheOptionalContextFieldsAreNotRequired(t *testing.T) { populated := valid() - populated.NetworkID = "mahavistar" + populated.NetworkID = beckn.OneNetwork("mahavistar") populated.SenderID, populated.ReceiverID = "did:example:sender", "did:example:receiver" cleared := valid() - cleared.NetworkID, cleared.SenderID, cleared.ReceiverID = "", "", "" + cleared.NetworkID, cleared.SenderID, cleared.ReceiverID = beckn.NetworkIDs{}, "", "" for _, testcase := range []struct { name string diff --git a/src/publish/controller_test.go b/src/publish/controller_test.go index 1a76fb2..7c83720 100644 --- a/src/publish/controller_test.go +++ b/src/publish/controller_test.go @@ -118,8 +118,8 @@ func TestTheResponseIsTheCallbackEnvelope(t *testing.T) { t.Errorf("correlation = %q / %q, want the request's own", envelope.Context.TransactionID, envelope.Context.MessageID) } - if envelope.Context.NetworkID != "mahavistar" { - t.Errorf("networkId = %q, want it echoed", envelope.Context.NetworkID) + if got := envelope.Context.NetworkID.Networks(); len(got) != 1 || got[0] != "mahavistar" { + t.Errorf("networkId = %v, want it echoed", envelope.Context.NetworkID.Networks()) } if envelope.Context.Version != beckn.Version { t.Errorf("version = %q, want %q", envelope.Context.Version, beckn.Version) diff --git a/src/publish/mapper.go b/src/publish/mapper.go index 08db6a5..de67572 100644 --- a/src/publish/mapper.go +++ b/src/publish/mapper.go @@ -129,12 +129,11 @@ func catalogID(catalog beckn.Catalog, directive beckn.PublishDirective) string { return catalog.ID } -// visibleTo resolves C8: an omitted or empty visibleTo is the request's own -// network, not every network. Publishing to every network by a typo is the worse -// of the two failures. -func visibleTo(directive beckn.PublishDirective, network string) []string { +// visibleTo resolves C8: an omitted or empty visibleTo is every network, which +// is what the spec says it means. Narrowing is the publisher's to ask for. +func visibleTo(directive beckn.PublishDirective, _ string) []string { if len(directive.VisibleTo) == 0 { - return []string{network} + return []string{domain.NetworkWildcard} } return directive.VisibleTo } diff --git a/src/publish/mapper_test.go b/src/publish/mapper_test.go index d01dba4..32e085d 100644 --- a/src/publish/mapper_test.go +++ b/src/publish/mapper_test.go @@ -48,8 +48,8 @@ func TestTheDeclaredDefaultsAreResolvedByTheMapper(t *testing.T) { if !patch.Active { t.Error("Active = false for a payload that omitted isActive, want true") } - if len(patch.VisibleTo) != 1 || patch.VisibleTo[0] != network { - t.Errorf("VisibleTo = %v, want [%s]", patch.VisibleTo, network) + if len(patch.VisibleTo) != 1 || patch.VisibleTo[0] != domain.NetworkWildcard { + t.Errorf("VisibleTo = %v, want [%s]", patch.VisibleTo, domain.NetworkWildcard) } } diff --git a/src/publish/service.go b/src/publish/service.go index 17afd1e..8188ee4 100644 --- a/src/publish/service.go +++ b/src/publish/service.go @@ -68,9 +68,9 @@ func NewService( func (s *Service) Publish( ctx context.Context, envelope beckn.Context, action beckn.CatalogPublishAction, ) []beckn.CatalogProcessingResult { - network := envelope.NetworkID - if network == "" { - network = s.network + network := s.network + if named := envelope.NetworkID.Networks(); len(named) > 0 { + network = named[0] } results := make([]beckn.CatalogProcessingResult, 0, len(action.Catalogs)) @@ -257,7 +257,7 @@ func directiveFor(action beckn.CatalogPublishAction, catalogID string) (beckn.Pu // a zero value reading as FULL turns every directive-less republish into a // partial wipe of everything the payload did not mention. func applyDirectiveDefaults( - directive beckn.PublishDirective, catalogID, network string, + directive beckn.PublishDirective, catalogID, _ string, ) beckn.PublishDirective { if directive.CatalogID == "" { directive.CatalogID = catalogID @@ -269,7 +269,7 @@ func applyDirectiveDefaults( directive.UpdateMode = beckn.UpdateModeMerge } if len(directive.VisibleTo) == 0 { - directive.VisibleTo = []string{network} + directive.VisibleTo = []string{domain.NetworkWildcard} } if directive.ResourceDirectives == nil { directive.ResourceDirectives = []beckn.ResourceDirective{} diff --git a/src/publish/service_test.go b/src/publish/service_test.go index 7d9e71f..d051c18 100644 --- a/src/publish/service_test.go +++ b/src/publish/service_test.go @@ -345,8 +345,8 @@ func TestADirectiveLessCatalogIsMergedNotReplaced(t *testing.T) { if len(repo.modes) != 1 || repo.modes[0] != domain.UpdateModeMerge { t.Errorf("mode = %v, want MERGE — FULL would delete what the payload omitted", repo.modes) } - if got := repo.patches[0].VisibleTo; len(got) != 1 || got[0] != network { - t.Errorf("VisibleTo = %v, want the request's own network (C8)", got) + if got := repo.patches[0].VisibleTo; len(got) != 1 || got[0] != domain.NetworkWildcard { + t.Errorf("VisibleTo = %v, want the wildcard — an unstated audience is public", got) } } @@ -364,8 +364,8 @@ func TestAPartialDirectiveIsFilledFieldWise(t *testing.T) { if len(repo.modes) != 1 || repo.modes[0] != domain.UpdateModeMerge { t.Errorf("mode = %v, want MERGE", repo.modes) } - if got := repo.patches[0].VisibleTo; len(got) != 1 || got[0] != network { - t.Errorf("VisibleTo = %v, want the defaulted single network", got) + if got := repo.patches[0].VisibleTo; len(got) != 1 || got[0] != domain.NetworkWildcard { + t.Errorf("VisibleTo = %v, want the same default as no directive at all", got) } } @@ -525,9 +525,9 @@ func TestDeriveRunsAgainstTheMergedDocument(t *testing.T) { } } -// C6's publish half: an omitted networkId falls back to APP_NETWORK_ID, and a -// supplied one wins. Only visibleTo reads it, which is the whole of C8. -func TestTheEnvelopeNetworkWinsOverTheConfiguredOne(t *testing.T) { +// The envelope's network no longer decides the audience: an unstated one is +// public whoever sent it. +func TestTheEnvelopeNetworkDoesNotDecideTheAudience(t *testing.T) { repo := newRepo() service := newService(t, repo, &recordingReplicator{}) @@ -535,10 +535,10 @@ func TestTheEnvelopeNetworkWinsOverTheConfiguredOne(t *testing.T) { if err := json.Unmarshal([]byte(`{"catalogs":[{"id":"c1"}]}`), &action); err != nil { t.Fatalf("decoding the fixture: %v", err) } - service.Publish(t.Context(), beckn.Context{NetworkID: "bharatvistar"}, action) + service.Publish(t.Context(), beckn.Context{NetworkID: beckn.OneNetwork("bharatvistar")}, action) - if got := repo.patches[0].VisibleTo; len(got) != 1 || got[0] != "bharatvistar" { - t.Errorf("VisibleTo = %v, want the envelope's network", got) + if got := repo.patches[0].VisibleTo; len(got) != 1 || got[0] != domain.NetworkWildcard { + t.Errorf("VisibleTo = %v, want the wildcard — the sender's network is not the audience", got) } } diff --git a/src/publish/telemetry.go b/src/publish/telemetry.go index 6057222..d7de1ea 100644 --- a/src/publish/telemetry.go +++ b/src/publish/telemetry.go @@ -19,7 +19,7 @@ import ( // would read REGULAR on every span that exists — a table saying nobody ever // tried. Whether the volume counts and publish.visible_to belong on a span at // all is a judgement call decided in opentelemetry.md. -func observePublish(ctx context.Context, envelope beckn.Context, action beckn.CatalogPublishAction) { +func observePublish(ctx context.Context, _ beckn.Context, action beckn.CatalogPublishAction) { resources, offers, validity := volume(action.Catalogs) fact.ObserveInt64(ctx, fact.PublishCatalogCount, int64(len(action.Catalogs))) @@ -29,7 +29,7 @@ func observePublish(ctx context.Context, envelope beckn.Context, action beckn.Ca fact.ObserveStrings(ctx, fact.PublishProviderIDs, publisherIDs(action.Catalogs)) - types, modes, networks := directives(action, envelope.NetworkID) + types, modes, networks := directives(action) fact.ObserveStrings(ctx, fact.PublishCatalogTypes, types) fact.ObserveStrings(ctx, fact.PublishUpdateModes, modes) fact.ObserveStrings(ctx, fact.PublishVisibleTo, networks) @@ -70,12 +70,10 @@ func publisherIDs(catalogs []beckn.Catalog) []string { // would be invisible — the span would say MERGE while the row was replaced. The // loop walks the CATALOGS and looks the directive up, so a catalog with no // directive at all resolves the same way as one naming only catalogId. -func directives( - action beckn.CatalogPublishAction, network string, -) (types, modes, networks []string) { +func directives(action beckn.CatalogPublishAction) (types, modes, networks []string) { for _, catalog := range action.Catalogs { sent, _ := directiveFor(action, catalog.ID) - directive := applyDirectiveDefaults(sent, catalog.ID, network) + directive := applyDirectiveDefaults(sent, catalog.ID, "") types = add(types, directive.CatalogType) modes = add(modes, directive.UpdateMode) diff --git a/src/publish/telemetry_test.go b/src/publish/telemetry_test.go index 68323cd..b822923 100644 --- a/src/publish/telemetry_test.go +++ b/src/publish/telemetry_test.go @@ -113,19 +113,15 @@ func TestTheShapeOfAPublishIsObserved(t *testing.T) { } } -// TestAnOmittedVisibleToResolvesToTheRequestsOwnNetwork (C8). -// // The resolved set, not the sent one, and that is a deliberate difference from -// publish.catalog_types beside it. The question this attribute answers is which -// networks the data reached, and for an omitted visibleTo the honest answer is -// the request's own — reporting an empty list would say it reached none, which -// is the one thing that did not happen. -func TestAnOmittedVisibleToResolvesToTheRequestsOwnNetwork(t *testing.T) { +// publish.catalog_types beside it. Reporting an empty list would say the +// catalog reached no network, which is the one thing that did not happen. +func TestAnOmittedVisibleToIsReportedAsTheWildcard(t *testing.T) { record := recorded(t, `{"context":{"action":"publish","networkId":"oan"}, "message":{"catalogs":[{"id":"c1"}]}}`) - if got := look(t, record, fact.PublishVisibleTo).List; !slices.Equal(got, []string{"oan"}) { - t.Errorf("publish.visible_to = %v, want [oan]", got) + if got := look(t, record, fact.PublishVisibleTo).List; !slices.Equal(got, []string{"*"}) { + t.Errorf("publish.visible_to = %v, want [*]", got) } } diff --git a/src/storage/conformance/discover.go b/src/storage/conformance/discover.go index 370559c..468e2ec 100644 --- a/src/storage/conformance/discover.go +++ b/src/storage/conformance/discover.go @@ -38,7 +38,7 @@ import ( // owes if it DECLARES the mode, and skips on one that does not. func DiscoverCases(resolution int) []Case { return []Case{ - anOmittedNetworkSearchesEveryNetwork(), + theScopeIsExactlyTheNetworksTheCallerNamed(), theGateHidesWhatIsNotLive(), aWindowThatWrapsMidnightIsLiveAndItsForwardTwinIsNot(), schemaFilteringComparesContextAndTypeAsAPair(), @@ -187,6 +187,11 @@ func searched( if query.Limit == 0 { query.Limit = pageLimit } + // discover.Service fills this before a backend ever sees it, and these + // cases call the backend directly. + if len(query.NetworkIDs) == 0 { + query.NetworkIDs = []string{domain.NetworkWildcard} + } result, err := backends.Search.Search(t.Context(), query, modes) if err != nil { t.Fatalf("searching %+v: %v", query, err) @@ -265,39 +270,40 @@ func dateRange(from, to time.Time) *domain.TimePeriodPatch { // the cases // --------------------------------------------------------------------------- -// An empty scope network is UNSCOPED, not a network id that matches nothing. -// -// Both halves are asserted because the two failures are indistinguishable at -// the caller: a backend reading "" as a literal returns an empty page, and one -// falling back to this service's own APP_NETWORK_ID returns a single network's -// rows. -func anOmittedNetworkSearchesEveryNetwork() Case { - return Case{ - Name: "an omitted network searches every network and a given one narrows", - Given: []Publish{{ - Patch: domain.CatalogPatch{ - ID: "c-north", NetworkID: "north.example.com", Active: true, - ProtocolVersion: beckn.Version, - VisibleTo: []string{"north.example.com"}, - Resources: []domain.ResourcePatch{searchable("r-north", "north", "", "", "")}, - }, - Mode: domain.UpdateModeMerge, Derive: deriveSearchable, - }, { +// The three scoping rules in one story, because a backend can get any one of +// them right while breaking the next (A29). +func theScopeIsExactlyTheNetworksTheCallerNamed() Case { + publish := func(id, resource string, audience ...string) Publish { + return Publish{ Patch: domain.CatalogPatch{ - ID: "c-south", NetworkID: "south.example.com", Active: true, + ID: id, NetworkID: "north.example.com", Active: true, ProtocolVersion: beckn.Version, - VisibleTo: []string{"south.example.com"}, - Resources: []domain.ResourcePatch{searchable("r-south", "south", "", "", "")}, + VisibleTo: audience, + Resources: []domain.ResourcePatch{searchable(resource, resource, "", "", "")}, }, Mode: domain.UpdateModeMerge, Derive: deriveSearchable, - }}, + } + } + + return Case{ + Name: "a search matches the networks it named and no others", + Given: []Publish{ + publish("c-north", "r-north", "north.example.com"), + publish("c-south", "r-south", "south.example.com"), + publish("c-public", "r-public", domain.NetworkWildcard), + }, Then: func(t *testing.T, backends Backends) { unscoped := searched(t, backends, domain.SearchQuery{}, lexical) - assertPage(t, unscoped, []string{"r-north", "r-south"}) + assertPage(t, unscoped, []string{"r-public"}) - scoped := searched(t, backends, - domain.SearchQuery{NetworkID: "south.example.com"}, lexical) - assertPage(t, scoped, []string{"r-south"}) + one := searched(t, backends, + domain.SearchQuery{NetworkIDs: []string{"south.example.com"}}, lexical) + assertPage(t, one, []string{"r-south"}) + + both := searched(t, backends, domain.SearchQuery{ + NetworkIDs: []string{"south.example.com", domain.NetworkWildcard}, + }, lexical) + assertPage(t, both, []string{"r-public", "r-south"}) }, } } diff --git a/src/storage/conformance/publish.go b/src/storage/conformance/publish.go index a8beab8..35c3819 100644 --- a/src/storage/conformance/publish.go +++ b/src/storage/conformance/publish.go @@ -29,7 +29,7 @@ func PublishCases() []Case { deriveWritesReachTheStore(), mergeKeepsAnOmittedResource(), fullRemovesAnOmittedResource(), - anEmptyVisibleToBecomesTheNetwork(), + anEmptyVisibleToBecomesPublic(), theUpsertReturnsTheStoredRowOnConflict(), fieldLevelMergeSurvivesTheRoundTrip(), derivationRunsAfterTheMerge(), @@ -56,10 +56,12 @@ func PublishCases() []Case { // the write path cannot produce. func catalogPatch(id string, resources ...domain.ResourcePatch) domain.CatalogPatch { return domain.CatalogPatch{ - ID: id, - NetworkID: network, - Active: true, - VisibleTo: []string{network}, + ID: id, + NetworkID: network, + Active: true, + // Public: a case that is not about who can see a catalog must not be + // answered empty by the scope gate (A29). + VisibleTo: []string{domain.NetworkWildcard}, ProtocolVersion: beckn.Version, Resources: resources, } @@ -268,27 +270,25 @@ func fullRemovesAnOmittedResource() Case { } } -// The fail-safe: a catalog visible to nobody reports success and is findable by -// nobody, so the writer fills an empty list with the request's own network. The -// mapper already did this (A9); this pins the repository's own copy, the one -// that survives a mapper change. -func anEmptyVisibleToBecomesTheNetwork() Case { +// The writer's own copy of the default, the one that survives a mapper change. +func anEmptyVisibleToBecomesPublic() Case { patch := catalogPatch("c1", resourcePatch("r1", `{"grade":"A"}`)) patch.VisibleTo = nil return Case{ - Name: "an empty visibleTo is filled with the publishing network", + Name: "an empty visibleTo is filled with the wildcard", Given: []Publish{{Patch: patch, Mode: domain.UpdateModeMerge, Derive: noDerive}}, Then: func(t *testing.T, backends Backends) { stored := mustGet(t, backends, "c1") - if !slices.Equal(stored.VisibleTo, []string{network}) { - t.Errorf("the catalog is visible to %v, want [%s] — a catalog visible to "+ - "nobody reports success and is findable by no one", stored.VisibleTo, network) + want := []string{domain.NetworkWildcard} + if !slices.Equal(stored.VisibleTo, want) { + t.Errorf("the catalog is visible to %v, want %v — a catalog visible to "+ + "nobody reports success and is findable by no one", stored.VisibleTo, want) } - if resource := resourceByID(t, stored.Resources, "r1"); !slices.Equal(resource.VisibleTo, []string{network}) { - t.Errorf("r1 is visible to %v, want [%s]; discover reads the copy on the "+ - "resource and never joins to the catalog", resource.VisibleTo, network) + if resource := resourceByID(t, stored.Resources, "r1"); !slices.Equal(resource.VisibleTo, want) { + t.Errorf("r1 is visible to %v, want %v; discover reads the copy on the "+ + "resource and never joins to the catalog", resource.VisibleTo, want) } }, } diff --git a/src/storage/memory/search.go b/src/storage/memory/search.go index e8c904e..dab0b74 100644 --- a/src/storage/memory/search.go +++ b/src/storage/memory/search.go @@ -41,7 +41,7 @@ func (r *Repository) Search( r.mu.RLock() defer r.mu.RUnlock() - scope := domain.Scope{NetworkID: query.NetworkID, Now: time.Now().UTC()} + scope := domain.Scope{NetworkIDs: query.NetworkIDs, Now: time.Now().UTC()} ranked, filtering, degraded := r.negotiate(modes) matched := r.candidates(query, scope) @@ -140,7 +140,7 @@ func (r *Repository) candidates(query domain.SearchQuery, scope domain.Scope) [] // An empty scope network is UNSCOPED and emits no predicate at all — never a // literal that matches nothing, and never this service's own network id. func admitted(resource domain.Resource, scope domain.Scope) bool { - if scope.NetworkID != "" && !slices.Contains(resource.VisibleTo, scope.NetworkID) { + if !visibleOn(resource.VisibleTo, scope.NetworkIDs) { return false } return resource.Active && live( @@ -148,6 +148,14 @@ func admitted(resource domain.Resource, scope domain.Scope) bool { resource.ValidTimeFrom, resource.ValidTimeTo, scope.Now) } +// The Go half of the scope gate's && : a caller sees a catalog whose audience +// overlaps the networks they asked for (A29). +func visibleOn(audience, asked []string) bool { + return slices.ContainsFunc(audience, func(network string) bool { + return slices.Contains(asked, network) + }) +} + // live is the validity half of the gate: the calendar range ANDed with the daily // window. // @@ -309,7 +317,6 @@ func (r *Repository) hydrate(page []domain.Resource, scope domain.Scope) []domai position[resource.CatalogID] = index assembled = append(assembled, domain.Catalog{ ID: stored.ID, - NetworkID: stored.NetworkID, Document: stored.Document, VisibleTo: slices.Clone(stored.VisibleTo), Active: stored.Active, diff --git a/src/storage/postgres/discover_sql_test.go b/src/storage/postgres/discover_sql_test.go index 9405bba..b338eeb 100644 --- a/src/storage/postgres/discover_sql_test.go +++ b/src/storage/postgres/discover_sql_test.go @@ -70,8 +70,7 @@ const discoverQueries = "queries/discover.sql" // and drops every unscoped caller's results to one network. Matching the // `sqlc.narg` spelling is the only way to tell the two apart from here. var scopeGate = []string{ - "sqlc.narg('network_id')::text IS NULL", - "r.visible_to @> ARRAY[sqlc.narg('network_id')::text]", + "r.visible_to && @network_ids::text[]", "r.active", "r.valid_from IS NULL OR r.valid_from <= now()", "r.valid_to IS NULL OR r.valid_to >= now()", diff --git a/src/storage/postgres/gen/discover.sql.go b/src/storage/postgres/gen/discover.sql.go index 50f53ae..b905f24 100644 --- a/src/storage/postgres/gen/discover.sql.go +++ b/src/storage/postgres/gen/discover.sql.go @@ -15,8 +15,7 @@ import ( const fuzzyCandidates = `-- name: FuzzyCandidates :many SELECT r.catalog_id, r.id FROM resources r - WHERE ($1::text IS NULL - OR r.visible_to @> ARRAY[$1::text]) + WHERE r.visible_to && $1::text[] AND r.active AND (r.valid_from IS NULL OR r.valid_from <= now()) AND (r.valid_to IS NULL OR r.valid_to >= now()) @@ -82,7 +81,7 @@ SELECT r.catalog_id, r.id ` type FuzzyCandidatesParams struct { - NetworkID pgtype.Text + NetworkIds []string SchemaContexts []string SchemaTypes []string AttributeFilter pgtype.Text @@ -116,7 +115,7 @@ type FuzzyCandidatesRow struct { // the 0.3 default, and a deployment that moves it moves what this mode admits. func (q *Queries) FuzzyCandidates(ctx context.Context, arg FuzzyCandidatesParams) ([]FuzzyCandidatesRow, error) { rows, err := q.db.Query(ctx, fuzzyCandidates, - arg.NetworkID, + arg.NetworkIds, arg.SchemaContexts, arg.SchemaTypes, arg.AttributeFilter, @@ -363,8 +362,7 @@ SELECT r.catalog_id, r.id, r.visible_to, r.active, FROM unnest($1::text[]) WITH ORDINALITY AS c(v, n) JOIN unnest($2::text[]) WITH ORDINALITY AS i(v, n) USING (n)) AS p ON r.catalog_id = p.catalog_id AND r.id = p.resource_id - WHERE ($3::text IS NULL - OR r.visible_to @> ARRAY[$3::text]) + WHERE r.visible_to && $3::text[] AND r.active AND (r.valid_from IS NULL OR r.valid_from <= now()) AND (r.valid_to IS NULL OR r.valid_to >= now()) @@ -375,7 +373,7 @@ SELECT r.catalog_id, r.id, r.visible_to, r.active, type HydrateResourcesParams struct { CatalogIds []string ResourceIds []string - NetworkID pgtype.Text + NetworkIds []string } type HydrateResourcesRow struct { @@ -404,7 +402,7 @@ type HydrateResourcesRow struct { // `search_tsv` is absent for the same reason it is absent from the publish // loader: nothing reads it back and it is the largest column on the table. func (q *Queries) HydrateResources(ctx context.Context, arg HydrateResourcesParams) ([]HydrateResourcesRow, error) { - rows, err := q.db.Query(ctx, hydrateResources, arg.CatalogIds, arg.ResourceIds, arg.NetworkID) + rows, err := q.db.Query(ctx, hydrateResources, arg.CatalogIds, arg.ResourceIds, arg.NetworkIds) if err != nil { return nil, err } @@ -444,8 +442,7 @@ const lexicalCandidates = `-- name: LexicalCandidates :many SELECT r.catalog_id, r.id FROM resources r - WHERE ($1::text IS NULL - OR r.visible_to @> ARRAY[$1::text]) + WHERE r.visible_to && $1::text[] AND r.active AND (r.valid_from IS NULL OR r.valid_from <= now()) AND (r.valid_to IS NULL OR r.valid_to >= now()) @@ -511,7 +508,7 @@ SELECT r.catalog_id, r.id ` type LexicalCandidatesParams struct { - NetworkID pgtype.Text + NetworkIds []string SchemaContexts []string SchemaTypes []string AttributeFilter pgtype.Text @@ -578,6 +575,10 @@ type LexicalCandidatesRow struct { // NULL means NO predicate, not a predicate that matches nothing. An intent // carrying no filter must not be narrowed by one. // +// `&&` is overlap, not containment: a row matches if its audience shares ANY +// network with the caller's set (A29). The set is never empty — naming none +// resolves to the wildcard '*' in domain.ScopeNetworks, not here. +// // --------------------------------------------------------------------------- // retrieval — one query per ranked mode, all sharing one WHERE // --------------------------------------------------------------------------- @@ -595,7 +596,7 @@ type LexicalCandidatesRow struct { // text there is no relevance, so the ORDER BY falls through to the stable key. func (q *Queries) LexicalCandidates(ctx context.Context, arg LexicalCandidatesParams) ([]LexicalCandidatesRow, error) { rows, err := q.db.Query(ctx, lexicalCandidates, - arg.NetworkID, + arg.NetworkIds, arg.SchemaContexts, arg.SchemaTypes, arg.AttributeFilter, @@ -641,8 +642,7 @@ SELECT r.catalog_id, r.id FROM unnest($1::text[]) WITH ORDINALITY AS c(v, n) JOIN unnest($2::text[]) WITH ORDINALITY AS i(v, n) USING (n)) AS p ON r.catalog_id = p.catalog_id AND r.id = p.resource_id - WHERE ($3::text IS NULL - OR r.visible_to @> ARRAY[$3::text]) + WHERE r.visible_to && $3::text[] AND r.active AND (r.valid_from IS NULL OR r.valid_from <= now()) AND (r.valid_to IS NULL OR r.valid_to >= now()) @@ -653,7 +653,7 @@ SELECT r.catalog_id, r.id type ScopeFilterResourcesParams struct { CatalogIds []string ResourceIds []string - NetworkID pgtype.Text + NetworkIds []string } type ScopeFilterResourcesRow struct { @@ -671,7 +671,7 @@ type ScopeFilterResourcesRow struct { // no text, no geometry, no schema, because the caller has already applied those // and this answers only "may this caller see it now". func (q *Queries) ScopeFilterResources(ctx context.Context, arg ScopeFilterResourcesParams) ([]ScopeFilterResourcesRow, error) { - rows, err := q.db.Query(ctx, scopeFilterResources, arg.CatalogIds, arg.ResourceIds, arg.NetworkID) + rows, err := q.db.Query(ctx, scopeFilterResources, arg.CatalogIds, arg.ResourceIds, arg.NetworkIds) if err != nil { return nil, err } @@ -693,8 +693,7 @@ func (q *Queries) ScopeFilterResources(ctx context.Context, arg ScopeFilterResou const semanticCandidates = `-- name: SemanticCandidates :many SELECT r.catalog_id, r.id FROM resources r - WHERE ($1::text IS NULL - OR r.visible_to @> ARRAY[$1::text]) + WHERE r.visible_to && $1::text[] AND r.active AND (r.valid_from IS NULL OR r.valid_from <= now()) AND (r.valid_to IS NULL OR r.valid_to >= now()) @@ -760,7 +759,7 @@ SELECT r.catalog_id, r.id ` type SemanticCandidatesParams struct { - NetworkID pgtype.Text + NetworkIds []string SchemaContexts []string SchemaTypes []string AttributeFilter pgtype.Text @@ -800,7 +799,7 @@ type SemanticCandidatesRow struct { // is what turns that pool into candidates. func (q *Queries) SemanticCandidates(ctx context.Context, arg SemanticCandidatesParams) ([]SemanticCandidatesRow, error) { rows, err := q.db.Query(ctx, semanticCandidates, - arg.NetworkID, + arg.NetworkIds, arg.SchemaContexts, arg.SchemaTypes, arg.AttributeFilter, diff --git a/src/storage/postgres/hydrator.go b/src/storage/postgres/hydrator.go index f8bb8c7..b53c5ae 100644 --- a/src/storage/postgres/hydrator.go +++ b/src/storage/postgres/hydrator.go @@ -40,7 +40,7 @@ func (h *Hydrator) ScopeFilter( rows, err := h.queries.ScopeFilterResources(ctx, gen.ScopeFilterResourcesParams{ CatalogIds: catalogIDs, ResourceIds: resourceIDs, - NetworkID: nullableText(scope.NetworkID), + NetworkIds: scope.NetworkIDs, }) if err != nil { return nil, fmt.Errorf("apply the scope gate to a candidate set: %w", err) @@ -80,7 +80,7 @@ func (h *Hydrator) Hydrate( resources, err := h.queries.HydrateResources(ctx, gen.HydrateResourcesParams{ CatalogIds: catalogIDs, ResourceIds: resourceIDs, - NetworkID: nullableText(scope.NetworkID), + NetworkIds: scope.NetworkIDs, }) if err != nil { return nil, fmt.Errorf("hydrate the page's resources: %w", err) diff --git a/src/storage/postgres/queries/discover.sql b/src/storage/postgres/queries/discover.sql index c102221..1b2857f 100644 --- a/src/storage/postgres/queries/discover.sql +++ b/src/storage/postgres/queries/discover.sql @@ -38,6 +38,10 @@ -- -- NULL means NO predicate, not a predicate that matches nothing. An intent -- carrying no filter must not be narrowed by one. +-- +-- `&&` is overlap, not containment: a row matches if its audience shares ANY +-- network with the caller's set (A29). The set is never empty — naming none +-- resolves to the wildcard '*' in domain.ScopeNetworks, not here. -- --------------------------------------------------------------------------- @@ -60,8 +64,7 @@ -- name: LexicalCandidates :many SELECT r.catalog_id, r.id FROM resources r - WHERE (sqlc.narg('network_id')::text IS NULL - OR r.visible_to @> ARRAY[sqlc.narg('network_id')::text]) + WHERE r.visible_to && @network_ids::text[] AND r.active AND (r.valid_from IS NULL OR r.valid_from <= now()) AND (r.valid_to IS NULL OR r.valid_to >= now()) @@ -135,8 +138,7 @@ SELECT r.catalog_id, r.id -- name: FuzzyCandidates :many SELECT r.catalog_id, r.id FROM resources r - WHERE (sqlc.narg('network_id')::text IS NULL - OR r.visible_to @> ARRAY[sqlc.narg('network_id')::text]) + WHERE r.visible_to && @network_ids::text[] AND r.active AND (r.valid_from IS NULL OR r.valid_from <= now()) AND (r.valid_to IS NULL OR r.valid_to >= now()) @@ -216,8 +218,7 @@ SELECT r.catalog_id, r.id -- name: SemanticCandidates :many SELECT r.catalog_id, r.id FROM resources r - WHERE (sqlc.narg('network_id')::text IS NULL - OR r.visible_to @> ARRAY[sqlc.narg('network_id')::text]) + WHERE r.visible_to && @network_ids::text[] AND r.active AND (r.valid_from IS NULL OR r.valid_from <= now()) AND (r.valid_to IS NULL OR r.valid_to >= now()) @@ -299,8 +300,7 @@ SELECT r.catalog_id, r.id FROM unnest(@catalog_ids::text[]) WITH ORDINALITY AS c(v, n) JOIN unnest(@resource_ids::text[]) WITH ORDINALITY AS i(v, n) USING (n)) AS p ON r.catalog_id = p.catalog_id AND r.id = p.resource_id - WHERE (sqlc.narg('network_id')::text IS NULL - OR r.visible_to @> ARRAY[sqlc.narg('network_id')::text]) + WHERE r.visible_to && @network_ids::text[] AND r.active AND (r.valid_from IS NULL OR r.valid_from <= now()) AND (r.valid_to IS NULL OR r.valid_to >= now()) @@ -326,8 +326,7 @@ SELECT r.catalog_id, r.id, r.visible_to, r.active, FROM unnest(@catalog_ids::text[]) WITH ORDINALITY AS c(v, n) JOIN unnest(@resource_ids::text[]) WITH ORDINALITY AS i(v, n) USING (n)) AS p ON r.catalog_id = p.catalog_id AND r.id = p.resource_id - WHERE (sqlc.narg('network_id')::text IS NULL - OR r.visible_to @> ARRAY[sqlc.narg('network_id')::text]) + WHERE r.visible_to && @network_ids::text[] AND r.active AND (r.valid_from IS NULL OR r.valid_from <= now()) AND (r.valid_to IS NULL OR r.valid_to >= now()) diff --git a/src/storage/postgres/read_path_test.go b/src/storage/postgres/read_path_test.go index 6eb25fa..867922b 100644 --- a/src/storage/postgres/read_path_test.go +++ b/src/storage/postgres/read_path_test.go @@ -247,7 +247,7 @@ func TestAnOmittedNetworkIdSearchesEveryNetworkAndAGivenOneNarrows(t *testing.T) t.Errorf("an omitted networkId matched %d resources, want both: %v", len(everyNetwork), everyNetwork) } - scoped := matchedIDs(search(t, repository, domain.SearchQuery{Text: "wheat", NetworkID: "mahavistar"})) + scoped := matchedIDs(search(t, repository, domain.SearchQuery{Text: "wheat", NetworkIDs: []string{"mahavistar"}})) if !slices.Equal(scoped, []string{"r-open"}) { t.Errorf("networkId=mahavistar returned %v, want only that network's row", scoped) } @@ -969,7 +969,7 @@ func TestScopeFilterAppliesTheNetworkGate(t *testing.T) { id := domain.ResourceKey("cat-scope-net", "wheat") hydrator := postgres.NewHydrator(pool) - kept, err := hydrator.ScopeFilter(context.Background(), []string{id}, domain.Scope{NetworkID: "mahavistar"}) + kept, err := hydrator.ScopeFilter(context.Background(), []string{id}, domain.Scope{NetworkIDs: []string{"mahavistar"}}) if err != nil { t.Fatalf("ScopeFilter: %v", err) } @@ -977,7 +977,7 @@ func TestScopeFilterAppliesTheNetworkGate(t *testing.T) { t.Errorf("kept = %v, want %v — mahavistar is the network it was published to", kept, []string{id}) } - kept, err = hydrator.ScopeFilter(context.Background(), []string{id}, domain.Scope{NetworkID: "bharatvistar"}) + kept, err = hydrator.ScopeFilter(context.Background(), []string{id}, domain.Scope{NetworkIDs: []string{"bharatvistar"}}) if err != nil { t.Fatalf("ScopeFilter: %v", err) } @@ -1012,7 +1012,7 @@ func TestHydrateOfAGateRefusedPageReturnsNoCatalogs(t *testing.T) { id := domain.ResourceKey("cat-hydrate-empty", "wheat") catalogs, err := postgres.NewHydrator(pool).Hydrate( - context.Background(), []string{id}, domain.Scope{NetworkID: "bharatvistar"}) + context.Background(), []string{id}, domain.Scope{NetworkIDs: []string{"bharatvistar"}}) if err != nil { t.Fatalf("Hydrate: %v", err) } diff --git a/src/storage/postgres/retrievers.go b/src/storage/postgres/retrievers.go index 57d7480..42706c6 100644 --- a/src/storage/postgres/retrievers.go +++ b/src/storage/postgres/retrievers.go @@ -21,7 +21,7 @@ import ( // per query for the quantifier XOR or the nil-cover rule to drift — invisibly, // since every query would still return rows, just a different set. type predicates struct { - networkID pgtype.Text + networkIDs []string schemaContexts []string schemaTypes []string @@ -43,10 +43,10 @@ type predicates struct { } // sharedPredicates reduces a SearchQuery to the parameters every read query -// binds. NetworkID becomes NULL when unscoped — "" would match no visible_to -// entry and empty every response. +// binds. The network set is never empty: naming none means the public +// catalogs (A29). func sharedPredicates(query domain.SearchQuery) predicates { - shared := predicates{networkID: nullableText(query.NetworkID)} + shared := predicates{networkIDs: query.NetworkIDs} // A PAIR of equal-length, index-aligned arrays. Empty stays nil so the // query's `IS NULL` arm fires and no schema predicate applies at all. @@ -156,7 +156,7 @@ func (l *LexicalRetriever) Retrieve( ) ([]string, error) { shared := sharedPredicates(query) rows, err := l.queries.LexicalCandidates(ctx, gen.LexicalCandidatesParams{ - NetworkID: shared.networkID, + NetworkIds: shared.networkIDs, SchemaContexts: shared.schemaContexts, SchemaTypes: shared.schemaTypes, SpatialOp: shared.spatialOp, @@ -207,7 +207,7 @@ func (f *FuzzyRetriever) Retrieve( ) ([]string, error) { shared := sharedPredicates(query) rows, err := f.queries.FuzzyCandidates(ctx, gen.FuzzyCandidatesParams{ - NetworkID: shared.networkID, + NetworkIds: shared.networkIDs, SchemaContexts: shared.schemaContexts, SchemaTypes: shared.schemaTypes, SpatialOp: shared.spatialOp, @@ -267,7 +267,7 @@ func (s *SemanticRetriever) Retrieve( shared := sharedPredicates(query) rows, err := s.queries.SemanticCandidates(ctx, gen.SemanticCandidatesParams{ - NetworkID: shared.networkID, + NetworkIds: shared.networkIDs, SchemaContexts: shared.schemaContexts, SchemaTypes: shared.schemaTypes, SpatialOp: shared.spatialOp, diff --git a/src/storage/postgres/search_repository.go b/src/storage/postgres/search_repository.go index abf980a..e081475 100644 --- a/src/storage/postgres/search_repository.go +++ b/src/storage/postgres/search_repository.go @@ -103,7 +103,7 @@ func (s *SearchRepository) Search( // A6: captured ONCE so every mode agrees on "now". Postgres ignores it and // calls now(); it is for the backends that have none. - scope := domain.Scope{NetworkID: query.NetworkID, Now: time.Now().UTC()} + scope := domain.Scope{NetworkIDs: query.NetworkIDs, Now: time.Now().UTC()} rankedModes, filtering, degraded := s.negotiate(modes) diff --git a/tests/acceptance/defaults_test.go b/tests/acceptance/defaults_test.go index 7686e49..e3bbd44 100644 --- a/tests/acceptance/defaults_test.go +++ b/tests/acceptance/defaults_test.go @@ -57,10 +57,11 @@ func TestAnOmittedDefaultResetsInBothModes(t *testing.T) { mode.directives...) // Live again: isActive was false and the patch never mentions it. - onNew := svc.discoverOn(t, republishedOn, spatial(dwithin(providerGeoPath, majestic, 5000))) + // Unscoped, because the republish stated no audience either and an + // unstated audience is public. + onNew := svc.discover(t, spatial(dwithin(providerGeoPath, majestic, 5000))) if got := resourceIDs(onNew); !slices.Equal(got, []string{"r-retired"}) { - t.Errorf("on %s = %v, want [r-retired]: an omitted isActive defaults to live", - republishedOn, got) + t.Error("the republished catalog answers nothing: an omitted isActive defaults to live") } // And visible only where the republish came FROM. The stored @@ -95,8 +96,25 @@ func retired(t *testing.T, svc *service) { // Stated rather than assumed: everything the republish then changes has to // be a change from somewhere, and an inactive catalog answering nothing is // the baseline both halves of the assertion are measured against. - if got := resourceIDs(svc.discoverOn(t, republishedOn, + if got := resourceIDs(svc.discover(t, spatial(dwithin(providerGeoPath, majestic, 5000)))); len(got) != 0 { t.Fatalf("the retired catalog answers = %v, want none: isActive is false", got) } } + +// Scenario 26a. A publish that names no audience is public. +// +// Asserted from a search naming no network either, which is the request that +// asks for exactly the public catalogs: a narrow catalog would answer nothing. +func TestAPublishNamingNoAudienceIsPublic(t *testing.T) { + svc := newService(t) + + svc.publishCatalogs(t, aCatalog("c-open", + availableAt(majestic), + resources(aResource("r-open", "wheat")))) + + found := resourceIDs(svc.discover(t, spatial(dwithin(providerGeoPath, majestic, 5000)))) + if !slices.Equal(found, []string{"r-open"}) { + t.Errorf("a publish stating no audience returned %v, want [r-open]", found) + } +} diff --git a/tests/acceptance/discover_test.go b/tests/acceptance/discover_test.go index 3811366..a433f8e 100644 --- a/tests/acceptance/discover_test.go +++ b/tests/acceptance/discover_test.go @@ -274,36 +274,65 @@ func TestQuantifierNoneInvertsTheMatch(t *testing.T) { } } -// Scenario 29. An omitted networkId searches every network. +// Scenario 29. The three scoping rules, in one story, because a service can +// get any one of them right while breaking the next. // -// visibleTo restricts which networks a PUBLISHER chose to expose a catalog on. -// It is not an access boundary a network-less caller is presumed locked out -// of — a caller wanting isolation supplies networkId, and this is the scenario -// that keeps the two questions apart. -// -// The failure it prevents is a discover that borrowed publish's C8 default: -// with APP_NETWORK_ID standing in for an absent networkId, the unscoped search -// below would quietly return one catalog and look entirely correct. -func TestOmittedNetworkIDSearchesEveryNetwork(t *testing.T) { +// Every assertion is a search; the three catalogs behind them never change. A +// caller sees the networks it named and nothing else, and public is a network +// it names rather than a bonus it is handed. +func TestASearchMatchesTheNetworksItNamed(t *testing.T) { svc := newService(t) + nearby := spatial(dwithin(providerGeoPath, majestic, 5000)) svc.publishWith(t, []any{ aCatalog("c-mahavistar", availableAt(majestic), resources(aResource("r-mahavistar", "Mahavistar"))), aCatalog("c-bharatvistar", availableAt(majestic), resources(aResource("r-bharatvistar", "Bharatvistar"))), + aCatalog("c-public", availableAt(majestic), resources(aResource("r-public", "Public"))), }, directive("c-mahavistar", visibleTo("mahavistar")), - directive("c-bharatvistar", visibleTo("bharatvistar"))) + directive("c-bharatvistar", visibleTo("bharatvistar")), + directive("c-public", visibleTo("*"))) - both := resourceIDs(svc.near(t, majestic, 5000)) - slices.Sort(both) - if !slices.Equal(both, []string{"r-bharatvistar", "r-mahavistar"}) { - t.Errorf("an intent with no networkId returned %v, want both networks", both) + if got := resourceIDs(svc.discover(t, nearby)); !slices.Equal(got, []string{"r-public"}) { + t.Errorf("an intent naming no network returned %v, want [r-public]", got) } - one := resourceIDs(svc.discoverOn(t, "mahavistar", spatial(dwithin(providerGeoPath, majestic, 5000)))) + one := resourceIDs(svc.discoverOn(t, "mahavistar", nearby)) if !slices.Equal(one, []string{"r-mahavistar"}) { - t.Errorf("an intent scoped to mahavistar returned %v, want [r-mahavistar]", one) + t.Errorf("an intent naming mahavistar returned %v, want [r-mahavistar]", one) + } + + both := resourceIDs(svc.discoverOnAll(t, []string{"mahavistar", "*"}, nearby)) + slices.Sort(both) + if !slices.Equal(both, []string{"r-mahavistar", "r-public"}) { + t.Errorf(`an intent naming ["mahavistar","*"] returned %v, want both`, both) + } +} + +// Scenario 29a. The two wire shapes of one field must reach the same query. +// +// The string form is what every caller sends today and the array form is what +// the spec added beside it; a service that handled only the second would break +// every existing caller, and one that handled only the first would silently +// ignore the networks an array named. +func TestBothWireShapesOfNetworkIDAnswerTheSame(t *testing.T) { + svc := newService(t) + nearby := spatial(dwithin(providerGeoPath, majestic, 5000)) + + svc.publishWith(t, + []any{aCatalog("c-one", availableAt(majestic), resources(aResource("r-one", "One")))}, + directive("c-one", visibleTo("mahavistar"))) + + asString := resourceIDs(svc.discoverOn(t, "mahavistar", nearby)) + asArray := resourceIDs(svc.discoverOnAll(t, []string{"mahavistar"}, nearby)) + + if !slices.Equal(asString, asArray) { + t.Errorf(`"mahavistar" returned %v and ["mahavistar"] returned %v, want the same page`, + asString, asArray) + } + if !slices.Equal(asString, []string{"r-one"}) { + t.Errorf("both shapes returned %v, want [r-one]", asString) } } diff --git a/tests/acceptance/publish_test.go b/tests/acceptance/publish_test.go index 8cd4f24..d59b1a8 100644 --- a/tests/acceptance/publish_test.go +++ b/tests/acceptance/publish_test.go @@ -514,9 +514,11 @@ func TestACallerOverItsRateGetsA429(t *testing.T) { func TestChangingVisibleToWithNoResourcesInThePayloadTakesEffect(t *testing.T) { svc := newService(t) - svc.publishCatalogs(t, aCatalog("c-gate", - availableAt(majestic), - resources(aResource("r-gated", "Gated")))) + // Named rather than left to the default: this scenario is about a gate + // MOVING, so it has to start somewhere narrow enough to move from. + svc.publishWith(t, + []any{aCatalog("c-gate", availableAt(majestic), resources(aResource("r-gated", "Gated")))}, + directive("c-gate", visibleTo(network))) if got := resourceIDs(svc.discoverOn(t, network, spatial(dwithin(providerGeoPath, majestic, 5000)))); !slices.Equal(got, []string{"r-gated"}) { t.Fatalf("before the change %s sees %v, want [r-gated]", network, got) diff --git a/tests/acceptance/suite_test.go b/tests/acceptance/suite_test.go index 905fa7d..9ba30e1 100644 --- a/tests/acceptance/suite_test.go +++ b/tests/acceptance/suite_test.go @@ -234,6 +234,12 @@ func onNetwork(id string) func(map[string]any) { return func(context map[string]any) { context["networkId"] = id } } +// onNetworks is the array form of the same field, which is how a caller asks +// for its own network and the public catalogs in one request. +func onNetworks(ids ...string) func(map[string]any) { + return func(context map[string]any) { context["networkId"] = ids } +} + // ids are canonical UUIDs because the envelope rules require them to be (C6): // messageId and transactionId are checked for the 8-4-4-4-12 hex shape before // any handler runs, so an ordered counter — which would make a failing @@ -329,14 +335,29 @@ func (s *service) discover(t *testing.T, intent map[string]any) []beckn.Catalog return body.Message.Catalogs } -// discoverOn is discover scoped to one network, which is what makes a -// visibleTo gate observable: an intent carrying no networkId searches every -// network, so a scenario about visibility that omitted it would assert nothing. +// discoverOn is discover scoped to one network, sent as the STRING form of +// context.networkId. discoverOnAll below sends the array form. func (s *service) discoverOn(t *testing.T, networkID string, intent map[string]any) []beckn.Catalog { t.Helper() + return s.discoverScoped(t, intent, onNetwork(networkID)) +} + +// discoverOnAll is the same request with the ARRAY form of networkId, which is +// how a caller asks for its own network and the public catalogs at once. +func (s *service) discoverOnAll(t *testing.T, ids []string, intent map[string]any) []beckn.Catalog { + t.Helper() + + return s.discoverScoped(t, intent, onNetworks(ids...)) +} + +func (s *service) discoverScoped( + t *testing.T, intent map[string]any, scope func(map[string]any), +) []beckn.Catalog { + t.Helper() + answer := s.post(t, "/discover", - envelope(beckn.ActionDiscover, map[string]any{"intent": intent}, onNetwork(networkID))) + envelope(beckn.ActionDiscover, map[string]any{"intent": intent}, scope)) if answer.status != http.StatusOK { t.Fatalf("POST /discover = %d, want 200\nbody: %s", answer.status, answer.body) } diff --git a/tests/testdata/beckn-v2.0.0.yaml b/tests/testdata/beckn-v2.0.0.yaml index 95a8663..d88bd35 100644 --- a/tests/testdata/beckn-v2.0.0.yaml +++ b/tests/testdata/beckn-v2.0.0.yaml @@ -184,6 +184,23 @@ paths: textSearch: premium laptop limit: 3 offset: 3 + multiNetworkDiscover: + summary: Discover across two subnets using the array form of networkId + value: + context: + version: '2.0.0' + action: discover + messageId: 880e8400-e29b-41d4-a716-446655440006 + transactionId: 880e8400-e29b-41d4-a716-446655440007 + timestamp: '2026-03-04T11:05:00.000Z' + bapId: bap.shoply.com + bapUri: https://bap.shoply.com + networkId: + - retail-network + - grocery-network + message: + intent: + textSearch: organic coffee beans responses: '200': $ref: '#/components/responses/Ack' @@ -3069,12 +3086,22 @@ components: type: string format: uuid networkId: - description: "The unique identifier of the Beckn subnet within which this\ - \ transaction is occurring. Actors use this value to resolve the subnet's\ - \ registry URL before performing identity and address lookups. The RECOMMENDED\ - \ format is namespace_id followed by a slash followed by registry_id, for\ - \ example acmenet.org/charge-net." - type: string + description: "The unique identifier(s) of the Beckn subnet(s) within which\ + \ this transaction is occurring. Actors use this value to resolve the\ + \ subnet's registry URL before performing identity and address lookups.\ + \ The RECOMMENDED format is namespace_id followed by a slash followed\ + \ by registry_id, for example acmenet.org/charge-net.\n\nMAY be a single\ + \ string (single-network transaction) or an array of strings (multi-network\ + \ transaction). When an array is provided, order is significant: actors\ + \ MUST attempt registry resolution starting with the first entry, treating\ + \ it as the primary network, and MAY fall back to subsequent entries if\ + \ resolution against the primary fails." + oneOf: + - type: string + - type: array + items: + type: string + minItems: 1 timestamp: description: The date and time at which this message was generated by the sender, expressed in RFC 3339 date-time format. The receiving actor MAY From af1c51aada0e74851870a23dbe344b18d328fd57 Mon Sep 17 00:00:00 2001 From: ameersohel45 Date: Wed, 28 Oct 2026 15:22:23 +0530 Subject: [PATCH 20/23] docs: record the public-by-default audience and one-or-many networkId [#58] MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Split from the code PR, which is about something a reviewer can check by reading Go; this is the plan, which is the spec and changes first. A28: an omitted visibleTo is the wildcard. A9 is NOT carved up — the default still applies in both update modes, by the same two lines. The first draft of A28 made an omitted visibleTo mean "keep what is stored" under MERGE; review took it out as a change to what MERGE means rather than to a default, and the hazard it named is now a Deferred row so the argument survives the code that was removed. A29: context.networkId is one network or several, and a search matches exactly the networks it names. Public stops being something a scoped caller is handed and becomes a network they ask for, which is what lets the five queries collapse to one array overlap with no literal in them. Two Deferred rows carry what was tried and pulled: the MERGE keep rule, and deleting Catalog.NetworkID, which gofmt punishes out of proportion to its value in a review about something else. Scenarios 26a, 29 and 29a state what the four tests assert. The scenario count stands at 37. --- docs/design/implementation-plan.md | 216 +++++++++++++++++++++-------- docs/publish-and-discover.md | 12 +- 2 files changed, 167 insertions(+), 61 deletions(-) diff --git a/docs/design/implementation-plan.md b/docs/design/implementation-plan.md index fb4fcc7..1a49d60 100644 --- a/docs/design/implementation-plan.md +++ b/docs/design/implementation-plan.md @@ -80,10 +80,12 @@ Every task inherits these. The PRD and `beckn.yaml` v2.0.0 disagree in twelve places, the spec contradicts itself in a thirteenth, it is silent in a fourteenth, and in a fifteenth it declares a field that no value can satisfy. These resolutions are -binding. C8, C9 and +binding. C9 and C10 are **deliberate deviations** — cases where this service knowingly does something other than what the spec says, each with the reason written down, -because an undocumented deviation is indistinguishable from a bug. C13 is not a +because an undocumented deviation is indistinguishable from a bug. C8 was the +third until **A28** made it conformant; its row keeps the deviation's reasoning +because that reasoning is what A28 traded away. C13 is not a deviation but a choice between two things the schema says at once. | | Conflict | Resolution | @@ -95,7 +97,7 @@ deviation but a choice between two things the schema says at once. | **C5** | PRD requires categories; the spec has **no `category` field anywhere**. `Resource` is exactly `{id, descriptor, resourceAttributes}`, and `Intent` is `additionalProperties: false` with no category filter | No column, no index, no derivation. `stats.categoryCount` in the publish ack *is* in the spec, so it is answered as the count of distinct `@type` values in the catalog — the only grouping the schema actually has | | **C6** | `Context` declares no `required` list, so L1 alone cannot reject a missing `transactionId` | Envelope rules are enforced separately by struct tags: `action`, `version`(=2.0.0), `messageId`(uuid4), `transactionId`(uuid4), `timestamp`(RFC3339) required. `senderId`, `receiverId`, `networkId` stay optional (**A24** — the two participant DIDs are unverified in this phase, and a required field this service cannot check would demand an identity and then take the caller's word for it; `networkId` is a filter with a default, C8). Runs even when L1 is off. **The same absence is what lets OAN decline `bapId`/`bppId` in `context` at no cost.** The spec's prose says the context "MUST include at minimum … `bapId` or `bppId`" (`:66-72`), and its machine-readable half says nothing at all: `Context` carries no `required`, no `additionalProperties` and no `oneOf`/`anyOf`, so an envelope with `senderId`/`receiverId` and neither legacy id validates cleanly. `senderId` and `receiverId` are themselves declared properties of that same schema — OAN is *selecting from* the spec's own property list, not deviating from it. This is one of the few places the spec's prose and its schema disagree with each other rather than with us, and the rule "the schema wins" resolves it in OAN's favour. Unlike `Catalog`, which closes with `additionalProperties: false` and so pins `bppId` as the only accepted spelling there | | **C7** | Validation produces many faults, and `Error` is singular. But `Error.details` is itself `additionalProperties: false` with exactly two keys, `path` and `cause` — a list of extra pointers cannot go there and still validate | Two answers, because the spec already supplies both. **Publish:** `CatalogProcessingResult.errors` is natively `array of Error`, so every fault is its own `Error` and there is nothing to pack. **A NACK:** one `Error`, and each remaining fault is the `details.cause` of the one before it — a chain, which is exactly what that self-referencing field is documented for. `details.path` is a JSONPath (`$.message.publishDirectives[1]`), the form the spec's own example uses. No fault is dropped silently | -| **C8** | `publishDirectives.visibleTo`: *"When omitted, the catalog is visible to all eligible subscribers"* | **Deviation.** Omitted or empty becomes `[network]` — the request's `networkId`, else `APP_NETWORK_ID` (`mahavistar`, `bharatvistar`, …). Publishing to every network by a typo is the worse failure of the two, and a publisher wanting network-wide reach can say so explicitly. **The default applies in both update modes** (A9): a republish that does not mention `visibleTo` gets `[network]`, exactly as a first publish does. A field with a declared default is never "absent" by the time the merge sees it, so MERGE and FULL cannot disagree about what a silent publisher meant. An explicit `[]` resolves to `[network]` too | +| **C8** | `publishDirectives.visibleTo`: *"When omitted, the catalog is visible to all eligible subscribers"* | ~~**Deviation.** Omitted or empty becomes `[network]`~~ — **conformant since A28: omitted or empty becomes `['*']`, the wildcard every network matches.** The deviation's reasoning is kept because it is what A28 traded away: *publishing to every network by a typo is the worse failure of the two, and a publisher wanting network-wide reach can say so explicitly.* It was traded because the opposite typo turned out to be just as real and far quieter — a publisher who omits the field gets a catalog no other network can find, and is told `ACCEPTED`. One of the two failures is visible to its victim; the other is not. **The default still applies in both update modes** (A9): a republish that does not mention `visibleTo` resolves it, exactly as a first publish does. `APP_NETWORK_ID` no longer fills this field | | **C9** | With no `publishDirectives`, the spec **infers** the type from content: catalogs with offers are `regular`, catalogs with only resources are `master` | **Deviation.** An absent directive is REGULAR. Honouring the inference would make A1 reject every ordinary catalog that happens to carry no offers — the common case. Only an explicit `catalogType: MASTER`, or a resource carrying `extends`, is refused | | **C10** | `Intent.filters.expression` names no grammar normatively, and its only example is RFC 9535: `$[?(@.rating.value >= 4.0 && @.electronic.brand.name == 'Premium Tech')]` | **Deviation.** Only PostgreSQL SQL/JSON path is executed (`$ ? (@.x == "y")`). An RFC 9535 expression is a `400` / `SCH_INVALID_JSONPATH` — never attempted, because a filter that matches nothing is indistinguishable from an honest empty result | | **C11** | `OnDiscoverAction` is `additionalProperties: false` with exactly one property, `catalogs`. There is nowhere in the response **body** to say that a retrieval mode was degraded | The list moves to the **`X-Beckn-Degraded`** response header, comma-separated, absent when nothing degraded. The same move C1 makes for `error.type`, for the same reason: a key the schema forbids is not an extension, it is a response that fails validation at the first consumer strict enough to check. `SEARCH_FAIL_ON_UNAVAILABLE_MODE=true` stays available for callers who would rather have a `400` than a header they might not read | @@ -104,9 +106,10 @@ deviation but a choice between two things the schema says at once. | **C14** | The `ErrorCode` enum has **no member for a payload this receiver will not accept on size**. All 76 are checked: the nearest are `AUT_RATE_LIMITED`, which is about pace and carries `Retry-After`, and `POL_NP_CAPACITY_EXCEEDED`, which `NackTooManyRequests` describes as *"a policy-governed engagement capacity limit"* at 429. There is no `SCH_` or `NET_` code for it either | **`POL_NP_CAPACITY_EXCEEDED` at HTTP 413**, and the request body is capped at `SERVER_MAX_REQUEST_BODY_BYTES` (10 MiB) in `Envelope`. The family is right and the status is not the family's: `POL_` is *a refusal this deployment's policy requires rather than one the request earned*, which is exactly what a size ceiling is — the same request succeeds unchanged against a deployment configured to accept it. 403 would send the caller to inspect their credentials for a fault that is in their payload, so `Status()` overrides it to 413 alongside the two overrides already there. **The reuse is safe only because this service runs no engagements**: the spec's other use of this code has no code path here, so within this service the code means one thing and maps to one status. A deployment that grows an engagement lifecycle must give that refusal a code of its own — the one thing it must not do is make this mapping carry two statuses, because `Status()` derives from the code precisely so that two call sites cannot disagree about what a fault is worth on the wire. **413 is not a status the spec declares anywhere**, and that is the deviation worth stating plainly rather than the choice between 403 and 413. `/discover` and `/catalog/publish` each declare exactly `200, 400, 401, 403, 429, 500` (`/discover` adds `202`), and the string `413` does not occur in the 4995-line fixture at all. So a caller generated from the OpenAPI document has no branch for this response and will fall through to its default handler. The body is still schema-valid — it is a `NackBadRequest`-shaped `{message:{status,messageId,error}}` carrying an enum member — so what a strict client loses is the status line, not the payload. It is accepted because the alternative is worse in the direction that matters: 403 is declared but sends the caller to their credentials, and 429 is declared but tells them to retry a body that will be refused identically for ever. A wrong declared status is a wrong instruction; an undeclared correct one is a missing branch. The moment the enum grows a size-refusal member, this reverts to whichever status that member's response schema declares. The ceiling is a knob and not a constant because the largest legitimate body is a property of a deployment's catalogs, not of the protocol; zero is refused at boot, because it reads as *unlimited* and behaves as *refuse everything* | | **C15** | `AddOn` is a bare `oneOf: [Resource, Offer]`, and NEITHER branch sets `additionalProperties: false`. `Resource` requires only `id`; `Offer` requires only `id`. **Every possible AddOn value therefore matches both branches, and `oneOf` requires exactly one** | **No resolution is available at this layer, and none is invented.** `Offer.addOns` is refused by L1 with `400 SCH_SCHEMA_VALIDATION_FAILED` at `$…offers[N].addOns[M]` — measured in Task 21b, not inferred. The alternatives were each worse: relaxing `oneOf` to `anyOf` in the vendored spec makes this service accept payloads the network rejects, and excluding `addOns` from L1 makes one field silently unvalidated. A schema that admits no value is a defect to report upstream, and the honest behaviour meanwhile is the 400 that names the path. Nothing else in the pipeline treats `addOns` specially: were the schema fixed tomorrow, A17 already stores it, because it stores the offer whole | -**`networkId` is a filter, not an identity claim, and the two mappers answer -"it's absent" differently.** On **publish**, absent → scope to -`APP_NETWORK_ID` — used only to fill an empty `visibleTo` (C8). On +**`networkId` is a filter, not an identity claim, and since A28 the two +mappers answer "it's absent" the same way — openly.** On **publish**, absent +`visibleTo` means `['*']`, the wildcard (C8, A28); `networkId` no longer feeds +it, and `APP_NETWORK_ID` is not consulted. On **discover**, absent means **no network predicate at all**: the query matches every network's catalogs, `visible_to` included. That is deliberate, not an oversight — `visibleTo` is how a publisher *restricts* a catalog to one or @@ -116,7 +119,9 @@ scoped to exactly that network, same as always. This is why it cannot be required either way — a publish carrying only the five required fields is a valid request, and it is what publishers send. The networks a catalog becomes discoverable *on* come from `publishDirective.visibleTo`, which is a message -field carrying **an array of network ids** (not `PUBLIC`/`PRIVATE`). +field carrying **an array of network ids** (not `PUBLIC`/`PRIVATE`) — or the +single wildcard element `*`, which is how A28 spells "every network" inside +that same array rather than inventing a second field to say it. --- @@ -134,7 +139,7 @@ field carrying **an array of network ids** (not `PUBLIC`/`PRIVATE`). | **A8** | `updateMode` becomes a **content** rule, not only a row-set rule. **MERGE** is RFC 7396 JSON Merge Patch against the stored documents — an absent key keeps its stored value, an explicit `null` deletes it, an array replaces wholesale — with `resources` and `offers` matched by `id` rather than by array position. **FULL** replaces the catalog outright, its own columns included: omissions reset to defaults, and resources and offers the payload omits are deleted. Publish therefore becomes read-modify-write under a row lock, and every derived column is computed **after** the merge | 11, 15, 17, 18, 21 | | **A10** | **Spatial search is answered as cell-set algebra, not as a prefilter in front of an exact stage.** Every geometry stores two H3 covers — `CONTAINMENT_FULL` (a guaranteed subset, which proves positives) and `CONTAINMENT_OVERLAPPING` (a guaranteed superset, which proves negatives) — and each CQL2 operator becomes an array predicate over the pair. **Seven of the nine operators are answered** where the previous design answered `S_DWITHIN` alone; `S_TOUCHES` and `S_CROSSES` are refused as unapproximable at any resolution rather than deferred. All seven RFC 7946 types work on **both** sides of the constraint, so the Point-only limit and the `NONE`-inversion it caused are gone. The costs are accuracy of one cell (~1.1 km at r8), oversize geometries decided by bounding box, and no path to cadastral precision without PostGIS | 11, 12, 14, 16, 17, 21 | | **A11** | **`RequestLogger` moves above `Recover` in the fixed middleware order**, and `Recover` learns to abort rather than write a second body. Nested the other way, a recovered panic answers 500 *outside* the response wrapper, so the panicking request carries no `X-Response-Time` and produces no completion line — the failures are exactly the requests missing from the log. `RequestLogger` writes its line from a `defer`, so an unwinding panic does not take the line with it; `Recover` asks the writer below whether the response is already committed and, when it is, logs once and re-panics with `http.ErrAbortHandler` instead of appending a NACK document to a half-written body under an already-claimed 200 | 8, 20, 23 | -| **A9** | **Declared defaults are resolved before the merge runs and apply in both update modes.** The directive fields — `catalogType`, `updateMode`, `visibleTo` — are filled by `applyDirectiveDefaults` in `publishOne`, before the mapper runs; `catalog.isActive` and an offer's `resourceIds` are filled in the mapper when it builds the patch. Either way, an omitted field reads as *sent with its default* rather than as *absent*, under MERGE as much as under FULL. Only fields with **no** declared default (`provider`, `validity`, `descriptor`, `resourceAttributes`, the offer body) preserve absence and follow the A8 merge rule | 11, 15, 17, 18, 21 | +| **A9** | **Declared defaults are resolved before the merge runs and apply in both update modes.** The directive fields — `catalogType`, `updateMode`, `visibleTo` — are filled by `applyDirectiveDefaults` in `publishOne`, before the mapper runs; `catalog.isActive` and an offer's `resourceIds` are filled in the mapper when it builds the patch. Either way, an omitted field reads as *sent with its default* rather than as *absent*, under MERGE as much as under FULL. Only fields with **no** declared default (`provider`, `validity`, `descriptor`, `resourceAttributes`, the offer body) preserve absence and follow the A8 merge rule. A28 changed what `visibleTo` resolves TO without changing when: it is still filled in both modes, by the same two lines | 11, 15, 17, 18, 21 | | **A12** | **L1 reconciles the two publish spellings against the document's own `const`, rather than the document being edited to admit both.** C2 says `publish` and `catalog/publish` are one request; the published schema constrains `context.action` to `const: catalog/publish`, and kin-openapi enforces `const` — so a body spelling it `publish`, which the protocol admits and this service routes, fails L1 on the very field that identified it. `SpecIndex` therefore stores, per action key, both the compiled schema and the canonical spelling **read out of that schema**, and `L1` rewrites `$.context.action` to it. The rewrite happens in the map `L1` decoded for itself, never in the buffered body, so the bytes this service signs, hashes and audits stay identical to the ones the caller sent. Hard-coding the mapping instead would put a second copy of the protocol in Go, which is exactly what D4 rejects; editing `beckn.yaml` would fork the published document | 9 | | **A13** | **`messageId` and `transactionId` are required to be a canonical-form UUID of any version, not specifically uuid4.** C6's parenthetical says uuid4; the schema declares `format: uuid`, which the schema wins on. Narrowing to v4 would refuse a conformant v7 id — the version publishers are moving to precisely because it sorts — and refuse it with `CTX_INVALID_FIELD`, which reads as the caller's mistake and is not. The check is a 12-line predicate over the 8-4-4-4-12 hyphenated form rather than a uuid dependency, matching `request_id.go`'s existing choice of `rand.Text` over a library; `urn:uuid:`, braced and unhyphenated spellings are refused, because a service that accepts four spellings of an id cannot correlate on it | 9 | | **A14** | **`DeriveFunc` takes the merged catalog by POINTER: `func(merged *Catalog, touched []string) []Fault`.** A8 makes derive the post-merge seam that WRITES `searchText`, `embedding`, `embeddingSourceHash`, `schemaContext`, `schemaType` and the geometry finds onto the merged catalog before the repository stores it. Taking the catalog by value gave it a copy: everything it computed was discarded, and the columns it exists to fill would have gone to the store empty while every test that only checked the returned faults passed. The `[]Fault` return stays, because a fault is a PARTIAL that travels to the response and is not a field of the catalog | 13, 14, 15, 17 | @@ -151,6 +156,9 @@ field carrying **an array of network ids** (not `PUBLIC`/`PRIVATE`). | **A25** | **This service is one of four layers, and the plan was written as if it were all of them. Tasks 25 and 26 are added, and 23a and 23c grow one requirement each.** The other three layers — experience, network, provider — run **beckn-onix** adapters that already emit OTel traces, metrics and audit logs, and reading them changes four things here. **Two questions this plan could not answer are answered elsewhere, not nowhere.** Relevance and accuracy need what the user did *after* the response; the experience-layer adapter has the user and already emits traces, so that is where the outcome event belongs. Performance per location and per commodity is refused here because coordinates and `filters.expression` are on the never-emitted list — but the provider adapter holds both as first-class call-plan fields, so it answers them without anyone widening a deny-list. The H3-coarsening policy call that `opentelemetry.md` raised against 23d is **withdrawn** on those grounds. **Our spans would currently be exported and then silently dropped.** onix's network trace pipeline admits only spans carrying `sender.id`, which this phase makes optional and usually absent, while the stage downstream of it rebuilds `trace_id` from an attribute spelled `transaction_id`, which we planned to spell `beckn.transactionId`. A span that fails either is stored, paid for and invisible, with nothing anywhere reporting an error. The filter is onix's bug to fix — it admits on one key and joins on another — and the spelling is ours: **23c emits `transaction_id` and `message_id` as aliases**, written from the same values at the same call site so they cannot drift. Two, not three — an earlier draft named `receiver.id`, which onix puts only on audit log records; its span spelling is `recipient.id`, which we already emit, so a third alias would be paid for everywhere and read nowhere. Buying admission by inventing a `sender.id` is explicitly refused; missing from a dashboard is recoverable, poisoning the network's only participant-identity join is not. **Metrics were dropped from this service for a reason that covers only half the ground.** A23 removed `metrics.go` because a stateless replica's in-memory counter is a partial nobody can reassemble — true of the facilitator's windowed aggregates, and the reason Task 24 exists. It is not true of a *level*: three configured ceilings (`DATABASE_MAX_CONNS`, the rate limiter, the body size) have no observable distance-to-limit, and a pool at 30 of 32 refuses nothing, so it produces no span to be seen in. An earlier draft of this amendment said a 429 "produces no span, no event and no counter". **That is false, and the correction matters because it is what the task's scope was sized against**: `Trace` is index 1 of `router.go`'s `chain`, above both `Envelope` and `RateLimit`, so a 429 and a body refusal each produce a span carrying the status and — post-23d — an `error` event, both projected from the single `logNack` in `response_writer.go` that every refusal already passes through. Only "no counter" was ever true, and a counter that restates a span fact is the `duration_ms` mistake one signal up. **Task 25 is therefore the levels, not the refusals**, and it is smaller for it. Conflating the operator set with the network set is why this service has no metrics at all. **Task 25** is the operator set and is **not blocked**; Task 24 stays blocked on the registry — *five of its twelve codes have since shipped, on 2026-09-10, derived in `otel/collector.yaml`; the block stands for the other seven. See Task 24.* **And the deny-list is a rule nothing enforces** — five sub-tasks comply with it by hand. **Task 26** turns it into a pin, over the exporter's output rather than over the code that builds it | 23, 24, 25, 26 | | **A26** | **The publish geometry budget becomes configuration.** `MaxGeometriesPerCatalog` moves from a `const` in `src/publish/geometry.go` to `config.Geo.MaxGeometriesPerCatalog` / `GEO_MAX_GEOMETRIES_PER_CATALOG`, default **256**, threaded `container → publish.NewService → derive → ExtractGeometries → catalogWalk`. **The default does not change and no behaviour over the ceiling changes** — the extra finds are still `PARTIAL` faults naming their paths, never a silent drop. What changes is who gets to pick the number. **The failure it causes is the wrong shape to be a build-time constant.** Over the ceiling the publish still answers HTTP 200 and the publisher is told, but those shapes are not searchable, and at the operator that reads as missing data rather than as a limit: a 100 km `S_DWITHIN` returns fewer markets than exist and nothing in the response says why. Measured against a real catalog — 273 Maharashtra markets carrying a `Point` under both `coverageAreas` and `market.location`, so 546 shapes — only the first 128 markets were findable by place under 256. **Config rather than a larger constant, for the reason `ResolutionCells` two rows above already gives:** how many shapes a catalog carries is a property of a deployment's publishers, and the ceiling is reached by publishing normally rather than by abuse. That is also the line that decides which neighbours do *not* move: `MaxCatalogWalkDepth` bounds a stack against a pathological document, which is abuse, and `MaxIndexCoverCells` is a property of the index rather than of a catalog. **Zero is refused at boot**, alongside the body ceiling and for the same reason — it reads either as "unlimited" or as "refuse everything", and a catalog that may hold no geometry is undiscoverable by place. **The duplicate declaration in `src/indexing/geo/h3.go` is deleted, not rewired.** It had zero call sites; two constants of one name in two packages is how an operator raises the ceiling in the half that is not read. **What the number stands in for is cells, not shapes:** it times `MaxIndexCoverCells` (8,192) is the worst case one publish can ask the index to hold, so a points-only catalog — one cell per shape — can raise it a long way and a polygon-heavy one cannot | 2, 17, 18 | | **A27** | **A deployment may decline free-text retrieval, and declining REFUSES where a missing backend mode degrades.** `Search.EnableTextSearch` / `SEARCH_ENABLE_TEXT_SEARCH`, **true by default** — the value that takes no decision has to be the one that behaves as before the flag existed. With it false, an intent carrying a non-empty `textSearch` is refused in `MapIntent` with `SCH_TYPE_NOT_SUPPORTED` at `$['message']['intent']['textSearch']`, and the message names `spatial` and `filters` as what still works. **The whole content of this amendment is that it is not `FailOnUnavailableMode` under another name**, because the two read alike and behave oppositely. That flag is about a mode the **backend** cannot run — `semantic` on a `noop` deployment — and false there degrades, answering with what the other modes found and naming the gap in `X-Beckn-Degraded`; the query the caller wrote survives without it. This one is the **deployment's** policy, and degrading would be the failure C11 exists to prevent: on a text-only intent the term IS the query, so running the remaining modes without it answers **the whole corpus under a 200** — a page indistinguishable at the caller from a correct one. That is the same widening `MapIntent`'s doc comment already refuses for an unreadable `targets` or an ignored SRID, which is why the guard sits there and not in `negotiate`. **Three details are load-bearing.** The guard reads the **trimmed** term, like every other reader of it: `" "` asks for no retrieval mode at all, so it stays the no-criterion refusal and calling it a disabled text search would name the wrong mistake. The intent's required-one-of **list** shrinks with the flag — `spatial or filters`, with a parenthetical saying why the third is absent — because offering `textSearch` to a caller this deployment would then refuse walks them into a second `400`; the **guard** does not shrink with it, since a term that was sent is a criterion that was sent and reporting it missing would be a false sentence stacked on the true one. And the wire message names the two criteria that still work rather than the environment variable, matching `negotiate` and the `filters.type` refusal: a caller cannot act on the variable, and the operator has `docs/publish-and-discover.md`. **`modesFor` is left alone.** It still maps a non-empty `Text` to all three ranked modes, because the refusal is total and unreachable past it; a second guard there would be a branch no test can reach and the kind of defence that rots into a contradiction | 2, 17, 19 | +| **A28** | **An omitted `visibleTo` means PUBLIC.** The default was quiet in the direction that hurts: a publisher who never thought about the field got a catalog only their own network could find, and an `ACCEPTED` telling them nothing. It becomes the wildcard `['*']`, which is what the spec's own sentence says, so C8 stops being a deviation. Resolved exactly where `[network]` was — in `applyDirectiveDefaults` and in the mapper's `visibleTo`, one literal each — so **A9 is untouched**: the default still applies in both update modes, and a republish that omits `visibleTo` still resolves it rather than keeping what is stored. That last point was decided twice. The first attempt made an omitted `visibleTo` mean *keep* under MERGE, on the argument that widening an audience a publisher narrowed is a change nobody is told of; review took it out, because it is a change to what MERGE means rather than a change to a default, and it belongs in its own issue with its own argument. The hazard it named is real and is recorded in Deferred. The wildcard is a row VALUE and not a second column, because every reader is already a set test over `visible_to`. `EnsureVisibleTo` fills `['*']`, and both `visible_to` columns keep `NOT NULL DEFAULT '{}'`. `APP_NETWORK_ID` loses this reader and keeps the telemetry Resource (A25). **Rows already stored are NOT migrated**: rewriting them would publish catalogs onto networks their publishers never named, which is the worse of the two wrongs a second time. `isActive` is **untouched** and still resets in both modes (A9, scenario 26). One cost, recorded rather than discovered: the spec says a `visibleTo` value *"MUST be a registry-registered network participant identifier"*, and `*` is not one. The schema declares no `pattern`, `enum` or `minItems`, so L1 accepts it — a second, smaller deviation living inside the one that closed C8 | 11, 15, 17, 18, 21, 27 | + +| **A29** | **`context.networkId` becomes one-or-many, and a discover answers exactly the networks it names.** `release-0.2.0` of the protocol (pinned at `7a9fdf0d`) redeclares the field as `oneOf: [string, array minItems 1]`, which is the only schema change in a 39-line diff. Three rules replace the old pair, and the middle one reverses the read half A28 shipped days earlier: an **omitted** `networkId` matches `['*']` and nothing else; a **named** network matches that network and nothing else; an **array** matches the union of its entries, `*` included if the caller lists it. So `["n1","*"]` is how a caller asks for its own network plus everything public, and that pairing — not a default — is what a farmer-facing app sends. The reversal is deliberate: A28 bolted public onto every scoped search because a scoped caller had no other way to reach a public catalog, and the array gives them one. **An omitted `networkId` therefore narrows**, from every catalog to the public ones, and that is the one behaviour here an existing caller loses without asking. It is the price of `networkId` meaning the same thing in all three forms rather than "a filter, except when absent, when it is not one". The predicate collapses with it: five queries lose the `sqlc.narg(...) IS NULL OR` dance and the separate `ARRAY['*']` arm for a single `r.visible_to && @network_ids::text[]`, because the scope is now always a non-empty set — `{'*'}` when the caller said nothing. `&&` is GIN-indexable, so one bitmap scan answers what two OR'd `@>` arms did. `beckn.Context.NetworkID` becomes `beckn.NetworkIDs`, a named `[]string` that unmarshals from either shape; it marshals a single id back as a **string**, so a caller who sent `["n1"]` is echoed `"n1"` — identical under the `oneOf`, and the alternative is carrying arrival-shape state through a struct the response re-marshals. The spec calls the array an ordered registry-resolution list with fallback, and we read it as an unordered filter set; nothing here resolves a registry, so the two do not collide, but it is a second meaning on one field and it is recorded rather than discovered | 16, 20, 21, 28 | A6 and A7 exist for one requirement: *swap the text backend later, keep geo on PG, and let publish write to two stores.* Both build **seams plus conformance @@ -214,6 +222,8 @@ Two sit on the boundary and are called out rather than dismissed: | `S_TOUCHES` and `S_CROSSES` | **Not deferred — refused.** A cell decomposition has no measure-zero boundary, so no resolution answers them. Listing them as "later" would be a promise nothing in this design can keep | `400` + `SCH_TYPE_NOT_SUPPORTED`, naming the operator | | `otelpgx` for database spans | **A spike, not a commitment, and not a gate on anything.** The 2026-09-09 build-vs-reuse audit found `github.com/exaring/otelpgx` v0.11.1 the one library of six worth real investigation: it is a `pgx` tracer, which is exactly the seam `storage/postgres/pool.go` already has, and it would supply per-query spans this service does not emit. Two things must be answered before any of that matters, and the first is fatal if it goes the wrong way: **does it obtain its own tracer?** If it does, A23's `scope.version` collision applies to it exactly as it does to `otelhttp` — the span carries the library's version where the network spec wants `1.0` — and the answer is no. It is **not** a reason to revisit Task 25's two pool counters, which are a levels signal `otelpgx` does not produce | Nothing. The pool instruments Task 25 shipped by hand, and no DB-query spans | | `fact.Definition.Visibility` | **Deleted on 2026-09-09, and 23f is the task that reintroduces it.** It was built to govern 23f's facilitator deny-list, 23f is blocked on O1/O2/O4, and in the meantime the column had **zero runtime consumers** — `telemetry/redact.go` is not a file. Two things settled it: `fact.go` documented behaviour that did not exist ("the projection treats Unspecified as LocalOnly at runtime"), and all 54 rows carried `Public` with none ever carrying `LocalOnly`, so the column had never distinguished one row from another. Every new row still had to choose a value for it. Re-adding it is the same mechanical work in reverse — one type, one struct field, N declarations, one golden regeneration — and 23f is the commit that will know what the deny-list actually needs, which today nobody does. The design it encoded is preserved here so 23f restores rather than re-derives it: `type Visibility uint8`, with `const (VisibilityUnspecified Visibility = iota; Public; LocalOnly)`. The zero was `Unspecified` and the projection was to treat `Unspecified` as `LocalOnly` at runtime, because a forgotten field must not mean "exported" | Nothing, and **nothing is lost:** the deny-list was never enforceable from a per-row column anyway. The risk is in attribute *values* — a caller-supplied `beckn.schemaContext` URI, a `status.message`, a `zap.Error(err)` — not in key names, which is why Task 26 asserts over the **serialised bytes** of the facilitator projection. That test was always the enforcer; the column was insurance beside it | +| A MERGE republish that omits `visibleTo` widens the audience | **Built in Task 27, taken out in review.** A publisher who narrowed a catalog to one network and later sends a one-attribute patch without `visibleTo` has it resolved to `['*']` — public — and nothing says so. The fix is to make an omitted `visibleTo` mean *keep what is stored* under MERGE. It is not deferred for lack of merit: it changes what MERGE means rather than what a default is, and it carried most of a review that was about a default. It needs its own issue, where the argument is the only thing on the page | Nothing. The window is open, and `isActive` has had the same shape of hazard since A9 | +| `Catalog.NetworkID` has no reader | **Left dead by A28 on 2026-10-28.** `EnsureVisibleTo` was its only reader and now fills the wildcard instead, so the field, `CatalogPatch.NetworkID`, `MapCatalog`'s `network` parameter and `publish.Service.network` all carry a value nothing consumes. Removing them in Task 27 was tried and reverted: the field is the longest key in three struct literals, so deleting it makes gofmt re-align every neighbouring line and puts ten whitespace-only lines plus eleven otherwise-untouched files into a review that is about something else. The removal is worth one small commit of its own, where the realignment is the whole diff and obvious | The field stays, and the three doc comments that claimed a reader now say there is none. `APP_NETWORK_ID` still reaches it, and still has a real reader in the telemetry Resource (A25) | | `Replication.Targets` has no consumer | **Found by the 2026-09-09 config audit.** `CatalogReplicator` (A7) is a real seam — Task 20 constructs it and `publish.Service` calls `Replicate` after every commit — but `container.go` wires `NoopReplicator` unconditionally and reads nothing off `cfg.Replication`. The field cannot select a second store because no task has built one yet, so naming a target here is accepted by the config loader and does nothing: no replication happens and the boot does not refuse it either, despite an earlier version of `config.go`'s comment claiming it would. The comment is corrected (2026-09-09); the field itself stays, because the seam it configures already ships (A7) and removing it now would mean re-adding it the moment a second store lands | `NoopReplicator`, regardless of `REPLICATION_TARGETS`. The task that builds the second store is also the task that makes this field do something and, at that point, decides whether a named target with no matching implementation should fail the boot | --- @@ -489,8 +499,13 @@ CREATE TABLE catalogs ( -- from. An array because that is what the directive carries — a publisher -- naming two networks publishes into both from one call. -- + -- The single element '*' is the wildcard: every network matches it, and it + -- is what an omitted or empty visibleTo resolves to (A28). It lives in the + -- array rather than in a second boolean column so that every reader stays + -- one @> containment test against one GIN index. + -- -- DEFAULT '{}' is a fail-safe, not a valid state: the writer fills an empty - -- list with the request's network first, because a catalog visible to + -- list with '*' first, because a catalog visible to -- nobody is findable by nobody while reporting success. visible_to TEXT[] NOT NULL DEFAULT '{}', @@ -555,8 +570,9 @@ and read by nothing: - `provider_id` was `document->'provider'->>'id'` copied out, and the only index on it served no query in this plan — `CatalogRepository` has no list-by-provider. -- `network_id` was pure audit. The writer uses the request's network at publish - time to fill an empty `visible_to`; that happens in Go, before the insert. +- `network_id` was pure audit. The writer filled an empty `visible_to` with the + request's network, in Go, before the insert; since A28 it fills `'{*}'` and + does not consult the request's network at all. If the audit trail matters it belongs in the log line, not on every row. - `catalog_type` could only ever hold `REGULAR`, because A1 refuses MASTER at intake. Adding it back when inheritance lands is `ALTER TABLE ADD COLUMN` @@ -1259,6 +1275,7 @@ what comes back: ```sql (sqlc.narg('network_id')::text IS NULL + OR r.visible_to @> ARRAY['*'] OR r.visible_to @> ARRAY[sqlc.narg('network_id')::text]) AND r.active AND (r.valid_from IS NULL OR r.valid_from <= now()) @@ -1275,6 +1292,14 @@ one clause in this block that used to be unconditional. A caller who supplies `networkId` still gets exactly that network's rows; nothing about the scoped case changes. +**The `'*'` arm is A28's half, and it is a third arm rather than a widening of +the second.** `r.visible_to @> ARRAY[sqlc.narg('network_id'), '*']` would mean +*both*, which no row holds. Each arm is its own sargable `@>` against +`idx_resources_visible_to`, so the planner answers the pair with a BitmapOr over +one index rather than losing it — the same property the `NULL` arm is written +to preserve, which is why the wildcard is a value in this column and not a +separate one. + **`plan_cache_mode = force_custom_plan`, and it is this pattern that requires it.** `$1 IS NULL OR r.visible_to @> ARRAY[$1]` is only index-usable when PostgreSQL knows `$1`: with the value in hand it folds the first arm to FALSE @@ -1302,7 +1327,11 @@ the custom plan is a test that cannot observe this at all. hydration, after the page is already decided. Task 16 has a test that reads the SQL file and proves no query omits the gate — the nullable `network_id` clause included, so a query that hard-codes `visible_to @> ARRAY[$1]` instead of the -nullable form is caught as surely as one that drops the clause outright. +nullable form is caught as surely as one that drops the clause outright. Since +A28 that test also fails a query carrying the nullable form but **not** the +`'*'` arm: such a query hides every public catalog from every scoped caller, +which is an empty page rather than an error and so is invisible to the +acceptance suite unless something reads the SQL. ### Validity — two independent windows @@ -1429,7 +1458,9 @@ POST /publish // the only publish route (C2) "action": "publish", "version": "2.0.0", "messageId": "", "transactionId": "", "timestamp": "", - "networkId": "mahavistar" // OPTIONAL → defaults to APP_NETWORK_ID + "networkId": "mahavistar" // OPTIONAL, and since A28 the publish path + // reads it for nothing — visibleTo no + // longer defaults from it }, "message": { "catalogs": [ { "id": "cat-1", "provider": { "id": "p-1", ... }, @@ -1444,7 +1475,9 @@ POST /publish // the only publish route (C2) "catalogId": "cat-1", "catalogType": "REGULAR", // MASTER → REJECTED (A1) "updateMode": "MERGE", // omitted → MERGE; RFC 7396 patch (A8) - "visibleTo": ["mahavistar", "oan"], // ARRAY OF NETWORK IDS + "visibleTo": ["mahavistar", "oan"], // ARRAY OF NETWORK IDS, or ["*"] + // omitted → ["*"] on a first publish or + // under FULL; under MERGE → keep stored (A28) "resourceDirectives": [ ... ] // any `extends` → REJECTED (A1) }] } @@ -1516,8 +1549,8 @@ service.Publish(ctx, reqContext, action): ```pseudo publishOne(ctx, network, catalog, directive): # A catalog may arrive with no directive at all — publishDirectives is - # optional. Absent means REGULAR, MERGE, and visibleTo = [network], which - # is the ordinary single-network publish and must not need boilerplate. + # optional. Absent means REGULAR, MERGE, and a public catalog, which + # is the ordinary publish and must not need boilerplate. directive ← applyDirectiveDefaults(directive, catalog.id, network) # FIELD-WISE, not all-or-nothing (A9). A directive that is absent # entirely and a directive that names only `catalogId` must come out @@ -1525,7 +1558,7 @@ publishOne(ctx, network, catalog, directive): # catalogId ← catalog.id # catalogType ← REGULAR if empty # updateMode ← MERGE if empty - # visibleTo ← [network] if empty # in BOTH modes + # visibleTo ← ['*'] if empty # in BOTH modes (A28) # resourceDirectives ← [] if nil # Spelled out rather than implied, because UpsertCatalog branches on # `updateMode == FULL` and FULL deletes the resources the payload did @@ -1684,9 +1717,10 @@ never a row. A resource is removed by a FULL republish that omits it and by nothing else, because "remove this resource" and "clear this field" are one typo apart and only one of them is recoverable. -**Declared defaults are resolved first, and they do not care which mode this -is (A9).** Before anything is merged, every field the spec gives a default is -filled — the directive fields (`catalogType`, `updateMode`, `visibleTo`) in +**Declared defaults are resolved first, and — `visibleTo` excepted — they do +not care which mode this is (A9).** Before anything is merged, every field the +spec gives a default is +filled — the directive fields (`catalogType`, `updateMode`) in `publishOne`, `isActive`, `context.version` and an offer's `resourceIds` in the mapper. The publisher contract is then one sentence — *a default means the default, @@ -1696,7 +1730,7 @@ always* — rather than a sentence that has to name the update mode to be true. |---|---|---| | `publishDirective.catalogType` | `REGULAR` | `REGULAR` | | `publishDirective.updateMode` | `MERGE` | `MERGE` | -| `publishDirective.visibleTo` | `[network]` | `[network]` — **not** the stored list | +| `publishDirective.visibleTo` | `['*']` (A28) | `['*']` — **not** the stored list | | `catalog.isActive` | `true` | `true` — **not** the stored flag | | `offer.resourceIds` | `[]` (catalog-wide) | `[]` | | `context.version` | `beckn.Version` | this build's version — **not** the stored one (A22) | @@ -1704,13 +1738,11 @@ always* — rather than a sentence that has to name the update mode to be true. **The consequence, stated rather than discovered.** A publisher who withdrew a catalog with `isActive: false` and later sends a one-attribute patch **without** -`isActive` republishes it live, and a publisher who narrowed `visibleTo` and -later patches without it goes back to `[network]`. That is the deliberate cost -of a defaulting rule that does not branch on mode: the two fields that decide -whether a catalog is visible at all are the two a publisher must keep sending. -It is stated in the publisher-facing docs, and scenario 26 asserts it, because -the alternative — defaults that mean different things in different modes — is a -rule nobody remembers correctly at 3 a.m. +`isActive` republishes it live, and one who narrowed `visibleTo` and later +patches without it goes back to `['*']` — public. That is the deliberate cost +of a defaulting rule that does not branch on mode, and scenario 26 asserts the +first half, because the alternative — a default that means different things in +different modes — is a rule nobody remembers correctly at 3 a.m. `context.version` is in the table for completeness and is unreachable today: C6 makes it required and pins it to `beckn.Version`, so an absent one is @@ -1779,8 +1811,8 @@ repo.UpsertCatalog(ctx, patch, updateMode, derive): # Documents by RFC 7396, `resources` and `offers` by id. A stored # entity the patch does not name comes through untouched. - if merged.VisibleTo is empty: merged.VisibleTo ← [merged.NetworkID] - # Belt and braces. The mapper already applied this default (A9), in + if merged.VisibleTo is empty: merged.VisibleTo ← ['*'] + # Belt and braces. The mapper already applied this default (A28), in # both modes, so this line should never fire. It stays because a # catalog visible to no network is findable by nobody while reporting # success, and that is not a failure worth reaching production to @@ -3096,7 +3128,7 @@ ticket, which is why it is a numbered limit and not a caveat in a paragraph. ## Scenarios -Thirty-five end-to-end scenarios in `tests/acceptance/`, run against the +Thirty-seven end-to-end scenarios in `tests/acceptance/`, run against the assembled service: real router, real PostgreSQL, real migrations, over HTTP. Only the embedder is stubbed. @@ -3161,7 +3193,8 @@ goes looking for. | # | Scenario | Pins | |---|---|---| -| 26 | `AnOmittedDefaultResetsInBothModes` | A9, on the two fields where it costs something. Publish a catalog with `isActive: false` and `visibleTo: ["mahavistar","oan"]`; republish it under **MERGE** with a one-attribute patch and no `publishDirectives` at all; the catalog is live again and visible only to `mahavistar`. Then the same pair under FULL, with the same result. This scenario exists to make the surprising half of A9 *deliberate*: it is the test that fails the day someone "fixes" MERGE to preserve an omitted default, which would make the two modes disagree about what silence means | +| 26 | `AnOmittedDefaultResetsInBothModes` | A9, on the field where it costs something. Publish a catalog with `isActive: false`; republish it under **MERGE** with a one-attribute patch and no `publishDirectives` at all; the catalog is live again. Then the same under FULL, with the same result. It exists to make the surprising half of A9 *deliberate*: it is the test that fails the day someone "fixes" MERGE to preserve an omitted default | +| 26a | `APublishNamingNoAudienceIsPublic` | A28's publish half, end to end. A catalog published with no `publishDirectives` at all is found by a search that also names no network — the request that asks for exactly the public catalogs. A narrow catalog would answer nothing, so the assertion cannot pass by accident | ### Geometry paths — `geopath_test.go` @@ -3179,7 +3212,8 @@ repository tests instead. | # | Scenario | Pins | |---|---|---| -| 29 | `OmittedNetworkIdSearchesEveryNetwork` | Two catalogs published to non-overlapping networks — `visibleTo: ["mahavistar"]` and `visibleTo: ["bharatvistar"]`. A discover with **no** `networkId` returns both. The same intent with `networkId: "mahavistar"` returns only the first. `visibleTo` restricts which networks a publisher chose to expose a catalog on; it is not an access boundary a network-less caller is presumed locked out of — a caller wanting isolation supplies `networkId` | +| 29 | `ASearchMatchesTheNetworksItNamed` | A29's three rules in one story, because a service can get any one of them right while breaking the next. Three catalogs — `mahavistar`, `bharatvistar` and `['*']` — and three searches against them: naming no network returns the public one alone; naming `mahavistar` returns `mahavistar`'s alone; naming `["mahavistar","*"]` returns both. Public is a network a caller asks for, not a bonus it is handed | +| 29a | `BothWireShapesOfNetworkIDAnswerTheSame` | `"mahavistar"` and `["mahavistar"]` are one request under two spellings (A29). A service handling only the array breaks every caller sending today's string; one handling only the string silently ignores the networks an array names. Neither failure is an error — both are a page | ### Spatial operators — `spatial_test.go` | # | Scenario | Pins | @@ -3232,8 +3266,8 @@ POST /discover { "message": { "intent": { "spatial": [{ ## Tasks -Twenty-six tasks, dependency-ordered — 23 as originally planned, 24 added by -A23 and 25 and 26 by A25. Each is one reviewable unit ending in a +Twenty-eight tasks, dependency-ordered — 23 as originally planned, 24 added by +A23, 25 and 26 by A25, 27 by A28 and 28 by A29. Each is one reviewable unit ending in a testable deliverable, and each follows the same cycle: > **write the failing test → run it, see it fail → implement → run it, see it @@ -3898,7 +3932,7 @@ CatalogPatch{ID, NetworkID string, # in — see A9 below Validity *TimePeriodPatch, # nil = absent Active bool, # NOT a pointer — defaulted (A9) - VisibleTo []string, # NOT nilable — defaulted (A9) + VisibleTo []string, # NOT nilable — defaulted (A28) ProtocolVersion string, # NOT nilable — the mapper resolved # an absent one to beckn.Version # before this point (A22) @@ -3910,10 +3944,8 @@ CatalogPatch{ID, NetworkID string, # distinction into the difference between keeping a publisher's data and # deleting it. # - # Active and VisibleTo are deliberately NOT pointers (A9). Active's default - # is resolved here, in the mapper; VisibleTo arrives already resolved from - # publishOne's applyDirectiveDefaults, and the mapper only copies it. - # Either way, by the time the merge runs there is no absence left to + # Active is deliberately NOT a pointer (A9): its default is resolved here, + # in the mapper, so by the time the merge runs there is no absence left to # represent — and a pointer that can never be nil is an invitation to # write the branch that makes it nil. @@ -4107,10 +4139,11 @@ Scope{NetworkID string, Now time.Time} # A6: a value # was live at 23:00 without waiting until 23:00. # NetworkID == "" means UNSCOPED — every backend must read it as "emit no # network predicate", never as a literal network id that matches nothing - # and never by falling back to APP_NETWORK_ID. That fallback belongs to - # publish's visibleTo default (C8), a different field answering a - # different question; the memory backend's conformance suite runs the - # empty-NetworkID case for exactly this reason. + # and never by falling back to APP_NETWORK_ID — which since A28 fills + # nothing on the publish side either. A SET NetworkID matches rows naming + # it AND rows holding the wildcard '*' (A28); a backend that implements + # only the first hides every public catalog from every scoped caller. + # The memory backend's conformance suite runs both cases for this reason. Retriever{Retrieve(ctx, query, scope) → ranked ids} # one per mode Hydrator{ScopeFilter, Hydrate, Count} @@ -4426,7 +4459,7 @@ always had two. pseudocode run under the catalog row lock, so their cost is lock hold time. **Tests pin:** MERGE keeps an omitted resource, FULL removes it; an empty -`visibleTo` becomes `{network_id}`; a mid-transaction failure leaves no partial +`visibleTo` becomes `{*}` (A28); a mid-transaction failure leaves no partial catalog; republishing replaces a geometry rather than accumulating it; a cascade delete removes geometries and offers. Two more on the write path itself: @@ -4521,12 +4554,17 @@ one `Retriever` per mode, `Hydrator`, `RRF` predicate. A mode that drops the clause entirely leaks another network's catalog to a caller who scoped their request; a mode that hard-codes `visible_to @> ARRAY[$1]` instead of the nullable form silently reintroduces - single-network scoping as the default. + single-network scoping as the default. **The `ARRAY['*']` arm is asserted by + the same test (A28):** a query carrying the nullable form without it hides + every public catalog from every scoped caller, and an empty page is not an + error anything else here would notice. - **An omitted `networkId` returns catalogs from every network,** including one whose `visibleTo` names a single network the caller never mentioned — and the same intent with `networkId` set still returns only that network's - rows. Pinned as one test with two assertions, because a fix aimed at one - direction is a regression in the other. + rows **plus every `['*']` catalog** (A28). Pinned as one test with three + assertions, because a fix aimed at one direction is a regression in another: + narrow the wildcard arm and public catalogs vanish from scoped searches, + widen the scoped arm and `visibleTo` stops restricting anything. - **Schema filtering** matches as a pair: two resources, one `schema.org` + `GroceryItem` and one `beckn.org/Mobility` + `RideService`, queried with both fragments at once, return both — and a query for @@ -4655,9 +4693,10 @@ no `endTime` is a fatal fault; `09:00:00+05:30` and a bare `09:00:00` under Then the pair that separates A9 from A8, because they pull in opposite directions and one mapper has to do both. **Defaulted:** a payload omitting -`isActive` yields `Active == true` and one omitting `visibleTo` yields -`[network]`, in MERGE as much as in FULL — the mapper resolves them, so no -absence reaches the merge. **Not defaulted:** a payload omitting `validity` +`isActive` yields `Active == true`, in MERGE as much as in FULL — the mapper +resolves it, so no absence reaches the merge. **Defaulted, but per mode +(A28):** a payload omitting `visibleTo` yields `['*']` under FULL and `nil` +— absent — under MERGE, while an explicit `[]` yields `['*']` in both. **Not defaulted:** a payload omitting `validity` yields `Validity == nil` while `"validity": null` yields the explicit delete, and the same for `provider`, `descriptor` and `resourceAttributes`. These look pedantic and are the whole of A8 and A9 — a mapper that flattens the second @@ -4779,7 +4818,7 @@ database down and `/readyz` does not; shutdown completes an in-flight request. `tests/acceptance/{suite,publish,discover,offers,validity,performance, defaults,geopath,spatial}_test.go` -The thirty-five scenarios above, over HTTP against the assembled service with +The thirty-seven scenarios above, over HTTP against the assembled service with a real database. Only the embedder is stubbed. One file per scenario group, so the file a failure names is the section of this plan it came from. @@ -4858,7 +4897,8 @@ into the patch document before the merge runs, so the merge sees an explicit `isActive` and sets it. `validity` is untouched by this: it keeps its `TimePeriodPatch` tri-state and its absence still means "keep", which is what RFC 7396 does anyway — the two agree, and only `isActive` disagreed. `visibleTo` -never enters the document at all, because it arrives on the directive. +never enters the document at all, because it arrives on the directive; since +A28 it keeps a tri-state of its own on the patch instead. **The mapper keeps the raw.** `MapCatalog` currently decodes into typed structs and reassembles; it must instead carry the publisher's bytes through, stripping @@ -5019,7 +5059,8 @@ exporter and its shutdown - The Resource carries `eid=API`, `producer`, `domain`, `service.name` and `network.id`. `service.name` is the **same value** as `producer`: they are one identity until one participant runs several services, and two spellings of one - thing is a grouping that splits. `network.id` comes from `APP_NETWORK_ID` (C8). + thing is a grouping that splits. `network.id` comes from `APP_NETWORK_ID`, + which since A28 is the only thing that reads it. - Shutdown flushes. A dropped tail is a request the facilitator never sees. **Tests pin:** `OTEL_EXPORTER=none` does not fail startup; the Resource carries @@ -5389,6 +5430,71 @@ of, which is the only failure mode that matters. Follows 23f, which builds the exporter it asserts against. Fixture-driven so that adding a denied field is a fixture line rather than a new test. +### Task 27 — Public by Default (A28) + +**Produces:** changes to `src/publish/{mapper,service}.go`, `src/domain/gate.go` +and `src/domain/catalog.go`, plus the conformance and acceptance suites. + +Shipped in one branch with Task 28, because A29 rewrites the read half of the +scope gate and landing them apart would change deployed behaviour twice. + +1. **`domain.NetworkWildcard`.** `'*'` appears in four Go files; a literal in + four places is a typo waiting to be undiagnosable. +2. **The two defaults change their literal and nothing else.** `visibleTo` in + the mapper and the `visibleTo` branch of `applyDirectiveDefaults` both + return `['*']` where they returned `[network]`. Names, signatures and + call sites are untouched: a default's VALUE changing should not move four + files. +3. **`EnsureVisibleTo` fills `['*']`.** +4. **Scenario 26a**, and scenario 26 loses nothing — `isActive` still resets. + +**Not in this task:** `isActive`, which resets the same way. A publisher sees a +retired catalog come back; they do not see an audience change. + +**Rejected, and recorded in Deferred:** making an omitted `visibleTo` mean +*keep what is stored* under MERGE. It was built, then taken out in review: it +is a change to what MERGE means rather than a change to a default, and it +needs its own issue and its own argument. The hazard it named — a publisher +who narrowed a catalog and later patches without the field goes public, with +nothing said — is real and unaddressed until that lands. + +**Also rejected:** deleting `Catalog.NetworkID`, which A28 leaves with no +reader. It is the longest key in three struct literals, so gofmt re-aligns +every neighbouring line and the removal drags ten whitespace-only lines and +eleven otherwise-untouched files into a review about something else. Deferred, +for a commit where that realignment is the whole diff. + +### Task 28 — One-or-many `networkId` (A29) + +**Produces:** a newer vendored spec, `beckn.NetworkIDs`, and a scope predicate +that is one array overlap instead of three OR'd arms. + +1. **Upgrade the vendored spec and the pin.** `VALIDATION_SPEC_URL` moves to + the `7a9fdf0d` commit of `release-0.2.0`, never the branch ref: a branch + moves and the validator would then disagree with itself between two boots. + `tests/testdata/beckn-v2.0.0.yaml` is the cache and must be the same bytes. +2. **`beckn.NetworkIDs`**, which decodes a string or an array and **remembers + which arrived**, because the response echoes the request's context. The + rest of the service reads `Networks()` and never asks. Its test decodes + both shapes and re-encodes them: a type that round-trips only the shape its + author thought of is the whole hazard. +3. **`SearchQuery.NetworkIDs` and `Scope.NetworkIDs`,** filled in + `discover.Service` and nowhere else. A backend never sees them empty, which + is what lets the storage layer stop defaulting — the opposite of A28's + wildcard, which every backend had to know about. +4. **One predicate.** `r.visible_to && @network_ids::text[]` in the five gated + queries. `'*'` then appears in no query at all: it is an ordinary entry the + caller sends. The memory backend's guard becomes the same overlap, which + DELETES its helper rather than growing one. +5. **Telemetry.** `fact.BecknNetworkID` becomes `KindStrings`. +6. **Scenarios 29 and 29a**, as the Scenarios section states them. + +**Re-baseline, not re-interpret:** `examples/` and the Postman collection call +discover unscoped and will return fewer catalogs. The recorded answers change +because the question changed, so `make audit` — which recomputes from the +published catalog rather than trusting the recording — is the gate that says +the new answers are right rather than merely new. + --- ## Open Items @@ -5429,7 +5535,7 @@ undocumented decision is indistinguishable from an accident: | Unavailable retrieval mode | Whether a mode the backend cannot run should degrade or refuse | **Degrade by default; refuse on request.** `SEARCH_FAIL_ON_UNAVAILABLE_MODE` defaults to `false`: the modes that work run, results come back, and the missing one is named in `X-Beckn-Degraded` (C11). Setting it `true` makes the same request a `400` / `NET_CATALOG_SOURCE_UNAVAILABLE`. The default points at degrade for one reason: Phase 1 ships `EMBEDDING_PROVIDER=noop`, so `semantic` is missing on every fresh deployment, and defaulting to refuse would make every `textSearch` a `400` until an Ollama deployment exists — a default that breaks the common case is a default nobody keeps. Silently ignoring the mode remains the option that is never taken | | Structured filtering | Whether JSONPath attribute filtering ships in Phase 1 or waits | **Phase 1.** `Intent.filters` is evaluated, not degraded — PostgreSQL SQL/JSON path only (C10), run through `@?` against the single `resources.filter_doc` composite (A18), with no rebase because the column is already rooted at `$.catalogs`. It costs one migration (the column and its GIN index) rather than none, which is the price of answering a predicate that crosses catalog, resource and offer — including under OR, which three separate `document` columns cannot answer at all. Scenario 18 changed from asserting the degradation to asserting the filter | | Semantic search (A5) | When embeddings are turned on | **Not in Phase 1 — confirmed.** `EMBEDDING_PROVIDER=noop` ships; the column, the HNSW index and the `Embedder` seam ship with it, and `embedding IS NULL` is the backfill queue for whenever an Ollama deployment exists. Deferred deliberately, not undecided | -| `networkId` default (discover) | Whether an omitted `networkId` scopes to `APP_NETWORK_ID` or searches every network | **Searches every network.** `visibleTo` is how a publisher restricts a catalog to specific networks; it is not an access boundary a network-less discover caller is presumed locked out of by default. A caller wanting isolation supplies `networkId`. Publish's own default — `APP_NETWORK_ID`, to fill an empty `visibleTo` — is unchanged (C8); the two fields answer different questions and no longer share a fallback | +| `networkId` default (discover) | Whether an omitted `networkId` scopes to `APP_NETWORK_ID` or searches every network | **Searches every network.** `visibleTo` is how a publisher restricts a catalog to specific networks; it is not an access boundary a network-less discover caller is presumed locked out of by default. A caller wanting isolation supplies `networkId` — and still sees every `['*']` catalog, because public is not the same as unscoped (A28). ~~Publish's own default — `APP_NETWORK_ID`, to fill an empty `visibleTo`~~ — is **gone** since A28: an omitted `visibleTo` is `['*']`, and the two fields now share no fallback because neither has one | --- diff --git a/docs/publish-and-discover.md b/docs/publish-and-discover.md index aa78591..4e376c8 100644 --- a/docs/publish-and-discover.md +++ b/docs/publish-and-discover.md @@ -446,7 +446,7 @@ matches no field fails the boot, so a typo cannot silently do nothing. | Variable | Default | What it decides | |---|---|---| -| `APP_NETWORK_ID` | — (required) | this node's network; publish's default `visibleTo` | +| `APP_NETWORK_ID` | — (required) | this node's network; the `network.id` every span reports | | `APP_SUBSCRIBER_ID` | — | this node's subscriber id | | `APP_DOMAIN` | — | the Beckn domain served | | `APP_DEFAULT_TIMEZONE` | `Asia/Kolkata` | the zone daily validity windows are read in | @@ -480,11 +480,11 @@ matches no field fails the boot, so a typo cannot silently do nothing. Four of these are load-bearing enough to spell out: -- **`APP_NETWORK_ID` is publish's default and not discover's.** A publish with - no `visibleTo` is visible to this network. A discover with no - `context.networkId` searches *every* network — a different field answering a - different question, and defaulting it here would quietly put discover back to - single-network scoping under a name suggesting it is unscoped. +- **Neither endpoint defaults a network from `APP_NETWORK_ID`.** A publish with + no `visibleTo` is public, stored as `["*"]`. A discover sees exactly the + networks it names: `"n1"` gets n1's catalogs, `["n1","*"]` gets n1's and the + public ones, and naming none gets the public ones. `networkId` may be a + string or an array, and the response echoes back the shape that arrived. - **`EXT_ALLOW_NETWORK_FETCH=false`** is the whole statement that a configured registry URL is trusted and a URL arriving in a request body is not. - **`AUTH_ENABLE_SIGNATURE_VERIFICATION=true` refuses the boot.** The Ed25519 From 6a1353868a7700cec2ee7205d70991eb8843c2ff Mon Sep 17 00:00:00 2001 From: ameersohel45 Date: Wed, 28 Oct 2026 15:25:23 +0530 Subject: [PATCH 21/23] refactor: keep every existing name and cut the comment churn [#58] MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Review asked for a smaller diff and no renames. Five test and conformance functions go back to the names they had, with only their bodies changed: the two in discover/service_test.go, the one in publish/service_test.go, the telemetry one, and both conformance cases. directives() keeps its name AND its arity — the second parameter is now `_`, so its call site does not move either. One rename stays. TestOmittedNetworkIDSearchesEveryNetwork now asserts that a search matches only the networks it named, so the old name says the opposite of the body. It is TestASearchMatchesTheNetworksItNamed, with the former name in its comment so `git log -S` still finds it. Comment churn trimmed where the old text was still true: query.go was four code lines against eleven of prose. What is left changed because it stated the old rule as fact — `"" means UNSCOPED, every network` is not something to leave sitting in the file. examples/ is reverted entirely, at the author's request. make verify, make audit and make newman fail on this branch as a result (20, 13 and 19), all from one cause: the samples publish to local-network and then search with no networkId, which now means the public catalogs. Four lines out of the publish sample fixes all three. --- src/beckn/schema_conformance_test.go | 5 ++-- src/discover/service.go | 8 ++---- src/discover/service_test.go | 4 +-- src/domain/query.go | 8 ++---- src/publish/service_test.go | 2 +- src/publish/telemetry.go | 6 ++-- src/publish/telemetry_test.go | 7 +++-- src/storage/conformance/discover.go | 4 +-- src/storage/conformance/publish.go | 4 +-- tests/acceptance/discover_test.go | 3 ++ tests/acceptance/suite_test.go | 43 ++++++++++++++-------------- 11 files changed, 48 insertions(+), 46 deletions(-) diff --git a/src/beckn/schema_conformance_test.go b/src/beckn/schema_conformance_test.go index cb8850e..b59775a 100644 --- a/src/beckn/schema_conformance_test.go +++ b/src/beckn/schema_conformance_test.go @@ -329,9 +329,8 @@ func TestEveryWireStructIsBound(t *testing.T) { bound[b.name] = true } - // NetworkIDs models `Context.properties.networkId`, which is a oneOf of a - // string and an array. The walk compares struct fields to schema - // properties, and that node declares none. + // NetworkIDs models a oneOf node, which declares no properties for the + // walk to compare its (zero) fields against. bound["NetworkIDs"] = true for _, name := range exportedStructNames(t) { diff --git a/src/discover/service.go b/src/discover/service.go index fc23352..e76b1d6 100644 --- a/src/discover/service.go +++ b/src/discover/service.go @@ -45,9 +45,8 @@ func (s *Service) Discover( } reportPartials(ctx, partial) - // From the ENVELOPE, and NOT defaulted to config.App.Network: naming no - // network means the public catalogs, not this deployment's own. Resolved - // HERE so no backend carries the rule a second time. + // From the ENVELOPE, never config.App.Network. Resolved here so no backend + // carries the rule a second time. query.NetworkIDs = scopedNetworks(envelope.NetworkID) modes, degraded, err := s.negotiate(query) @@ -287,8 +286,7 @@ func reportPartials(ctx context.Context, partial []domain.Fault) { } } -// scopedNetworks is what the caller asked for, or the public catalogs when -// they asked for nothing. +// scopedNetworks is what the caller named, or the public catalogs if nothing. func scopedNetworks(asked beckn.NetworkIDs) []string { if networks := asked.Networks(); len(networks) > 0 { return networks diff --git a/src/discover/service_test.go b/src/discover/service_test.go index ef0a7eb..28491f6 100644 --- a/src/discover/service_test.go +++ b/src/discover/service_test.go @@ -340,7 +340,7 @@ func TestTwoDegradedModesAreBothReturned(t *testing.T) { // The config carries a network id precisely so the assertion has something to // fail against: with App.Network empty, a service that wrongly defaulted would // pass this test by coincidence. -func TestAnOmittedNetworkIDBecomesTheWildcard(t *testing.T) { +func TestAnOmittedNetworkIDSearchesEveryNetwork(t *testing.T) { repo := &stubRepo{capabilities: everything()} cfg := settings() @@ -357,7 +357,7 @@ func TestAnOmittedNetworkIDBecomesTheWildcard(t *testing.T) { } // Both wire shapes reach the repository as the set the caller named. -func TestTheNetworksThatWereSentScopeTheSearch(t *testing.T) { +func TestANetworkIDThatWasSentScopesTheSearch(t *testing.T) { for _, test := range []struct { name string sent beckn.NetworkIDs diff --git a/src/domain/query.go b/src/domain/query.go index eaf1b91..506a080 100644 --- a/src/domain/query.go +++ b/src/domain/query.go @@ -103,8 +103,7 @@ type AttributeFilter struct { type SearchQuery struct { Text string - // The networks the caller named, already defaulted by discover.Service: a - // backend never sees this empty. See Scope. + // Defaulted by discover.Service, so a backend never sees it empty. NetworkIDs []string // Empty means no schema predicate at all, NOT a predicate matching nothing. @@ -145,9 +144,8 @@ type SearchResult struct { // on it. Postgres ignores it and calls now(); it exists for the backends that // have none — the memory store, and tests that ask what was live at 23:00. // -// NetworkIDs is the set a resource's audience must overlap, and it is never -// empty: discover.Service has already turned "the caller named none" into the -// wildcard, so no backend carries that rule a second time. +// NetworkIDs is the set a resource's audience must overlap. Never empty: +// discover.Service turned "the caller named none" into the wildcard. type Scope struct { NetworkIDs []string Now time.Time diff --git a/src/publish/service_test.go b/src/publish/service_test.go index d051c18..83af883 100644 --- a/src/publish/service_test.go +++ b/src/publish/service_test.go @@ -527,7 +527,7 @@ func TestDeriveRunsAgainstTheMergedDocument(t *testing.T) { // The envelope's network no longer decides the audience: an unstated one is // public whoever sent it. -func TestTheEnvelopeNetworkDoesNotDecideTheAudience(t *testing.T) { +func TestTheEnvelopeNetworkWinsOverTheConfiguredOne(t *testing.T) { repo := newRepo() service := newService(t, repo, &recordingReplicator{}) diff --git a/src/publish/telemetry.go b/src/publish/telemetry.go index d7de1ea..23631db 100644 --- a/src/publish/telemetry.go +++ b/src/publish/telemetry.go @@ -29,7 +29,7 @@ func observePublish(ctx context.Context, _ beckn.Context, action beckn.CatalogPu fact.ObserveStrings(ctx, fact.PublishProviderIDs, publisherIDs(action.Catalogs)) - types, modes, networks := directives(action) + types, modes, networks := directives(action, "") fact.ObserveStrings(ctx, fact.PublishCatalogTypes, types) fact.ObserveStrings(ctx, fact.PublishUpdateModes, modes) fact.ObserveStrings(ctx, fact.PublishVisibleTo, networks) @@ -70,7 +70,9 @@ func publisherIDs(catalogs []beckn.Catalog) []string { // would be invisible — the span would say MERGE while the row was replaced. The // loop walks the CATALOGS and looks the directive up, so a catalog with no // directive at all resolves the same way as one naming only catalogId. -func directives(action beckn.CatalogPublishAction) (types, modes, networks []string) { +func directives( + action beckn.CatalogPublishAction, _ string, +) (types, modes, networks []string) { for _, catalog := range action.Catalogs { sent, _ := directiveFor(action, catalog.ID) directive := applyDirectiveDefaults(sent, catalog.ID, "") diff --git a/src/publish/telemetry_test.go b/src/publish/telemetry_test.go index b822923..a8741ed 100644 --- a/src/publish/telemetry_test.go +++ b/src/publish/telemetry_test.go @@ -114,9 +114,10 @@ func TestTheShapeOfAPublishIsObserved(t *testing.T) { } // The resolved set, not the sent one, and that is a deliberate difference from -// publish.catalog_types beside it. Reporting an empty list would say the -// catalog reached no network, which is the one thing that did not happen. -func TestAnOmittedVisibleToIsReportedAsTheWildcard(t *testing.T) { +// publish.catalog_types beside it. The question this attribute answers is which +// networks the data reached, and for an omitted visibleTo that is every one — +// reporting an empty list would say it reached none, which did not happen. +func TestAnOmittedVisibleToResolvesToTheRequestsOwnNetwork(t *testing.T) { record := recorded(t, `{"context":{"action":"publish","networkId":"oan"}, "message":{"catalogs":[{"id":"c1"}]}}`) diff --git a/src/storage/conformance/discover.go b/src/storage/conformance/discover.go index 468e2ec..9fe4f9e 100644 --- a/src/storage/conformance/discover.go +++ b/src/storage/conformance/discover.go @@ -38,7 +38,7 @@ import ( // owes if it DECLARES the mode, and skips on one that does not. func DiscoverCases(resolution int) []Case { return []Case{ - theScopeIsExactlyTheNetworksTheCallerNamed(), + anOmittedNetworkSearchesEveryNetwork(), theGateHidesWhatIsNotLive(), aWindowThatWrapsMidnightIsLiveAndItsForwardTwinIsNot(), schemaFilteringComparesContextAndTypeAsAPair(), @@ -272,7 +272,7 @@ func dateRange(from, to time.Time) *domain.TimePeriodPatch { // The three scoping rules in one story, because a backend can get any one of // them right while breaking the next (A29). -func theScopeIsExactlyTheNetworksTheCallerNamed() Case { +func anOmittedNetworkSearchesEveryNetwork() Case { publish := func(id, resource string, audience ...string) Publish { return Publish{ Patch: domain.CatalogPatch{ diff --git a/src/storage/conformance/publish.go b/src/storage/conformance/publish.go index 35c3819..ab2e5bf 100644 --- a/src/storage/conformance/publish.go +++ b/src/storage/conformance/publish.go @@ -29,7 +29,7 @@ func PublishCases() []Case { deriveWritesReachTheStore(), mergeKeepsAnOmittedResource(), fullRemovesAnOmittedResource(), - anEmptyVisibleToBecomesPublic(), + anEmptyVisibleToBecomesTheNetwork(), theUpsertReturnsTheStoredRowOnConflict(), fieldLevelMergeSurvivesTheRoundTrip(), derivationRunsAfterTheMerge(), @@ -271,7 +271,7 @@ func fullRemovesAnOmittedResource() Case { } // The writer's own copy of the default, the one that survives a mapper change. -func anEmptyVisibleToBecomesPublic() Case { +func anEmptyVisibleToBecomesTheNetwork() Case { patch := catalogPatch("c1", resourcePatch("r1", `{"grade":"A"}`)) patch.VisibleTo = nil diff --git a/tests/acceptance/discover_test.go b/tests/acceptance/discover_test.go index a433f8e..3fbdeba 100644 --- a/tests/acceptance/discover_test.go +++ b/tests/acceptance/discover_test.go @@ -277,6 +277,9 @@ func TestQuantifierNoneInvertsTheMatch(t *testing.T) { // Scenario 29. The three scoping rules, in one story, because a service can // get any one of them right while breaking the next. // +// Renamed: it was TestOmittedNetworkIDSearchesEveryNetwork, which is now the +// opposite of what it asserts. +// // Every assertion is a search; the three catalogs behind them never change. A // caller sees the networks it named and nothing else, and public is a network // it names rather than a bonus it is handed. diff --git a/tests/acceptance/suite_test.go b/tests/acceptance/suite_test.go index 9ba30e1..31f3160 100644 --- a/tests/acceptance/suite_test.go +++ b/tests/acceptance/suite_test.go @@ -234,8 +234,7 @@ func onNetwork(id string) func(map[string]any) { return func(context map[string]any) { context["networkId"] = id } } -// onNetworks is the array form of the same field, which is how a caller asks -// for its own network and the public catalogs in one request. +// onNetworks is the ARRAY form of the same field. func onNetworks(ids ...string) func(map[string]any) { return func(context map[string]any) { context["networkId"] = ids } } @@ -335,29 +334,14 @@ func (s *service) discover(t *testing.T, intent map[string]any) []beckn.Catalog return body.Message.Catalogs } -// discoverOn is discover scoped to one network, sent as the STRING form of -// context.networkId. discoverOnAll below sends the array form. +// discoverOn is discover scoped to one network, which is what makes a +// visibleTo gate observable: an intent carrying no networkId searches only the +// public catalogs, so a scenario about a narrow one has to name it. func (s *service) discoverOn(t *testing.T, networkID string, intent map[string]any) []beckn.Catalog { t.Helper() - return s.discoverScoped(t, intent, onNetwork(networkID)) -} - -// discoverOnAll is the same request with the ARRAY form of networkId, which is -// how a caller asks for its own network and the public catalogs at once. -func (s *service) discoverOnAll(t *testing.T, ids []string, intent map[string]any) []beckn.Catalog { - t.Helper() - - return s.discoverScoped(t, intent, onNetworks(ids...)) -} - -func (s *service) discoverScoped( - t *testing.T, intent map[string]any, scope func(map[string]any), -) []beckn.Catalog { - t.Helper() - answer := s.post(t, "/discover", - envelope(beckn.ActionDiscover, map[string]any{"intent": intent}, scope)) + envelope(beckn.ActionDiscover, map[string]any{"intent": intent}, onNetwork(networkID))) if answer.status != http.StatusOK { t.Fatalf("POST /discover = %d, want 200\nbody: %s", answer.status, answer.body) } @@ -915,3 +899,20 @@ func normalised(t *testing.T, want any) any { } return decoded } + +// discoverOnAll is discoverOn with the array form of networkId. +func (s *service) discoverOnAll(t *testing.T, ids []string, intent map[string]any) []beckn.Catalog { + t.Helper() + + answer := s.post(t, "/discover", + envelope(beckn.ActionDiscover, map[string]any{"intent": intent}, onNetworks(ids...))) + if answer.status != http.StatusOK { + t.Fatalf("POST /discover = %d, want 200\nbody: %s", answer.status, answer.body) + } + + var body struct { + Message beckn.OnDiscoverAction `json:"message"` + } + answer.decode(t, &body) + return body.Message.Catalogs +} From 28875b7b29022da8139c1d73ccd2bf9074f92711 Mon Sep 17 00:00:00 2001 From: Manjunath Davanam Date: Wed, 28 Oct 2026 19:34:03 +0530 Subject: [PATCH 22/23] fix: make the suite, the examples and the empty id agree with A29 [#58] MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Five things the scope change left behind, found by running the gates the PR body said were green. make test was red in src/storage/postgres, 18 tests. The conformance suite's searched() helper got a wildcard default for a query naming no network; this package's search() did not, so every case going through it sent `visible_to && '{}'::text[]` and matched nothing — which is why paging, geometry, hydration and schema-filter tests all failed for a reason that has nothing to do with them. The default now lives in four places that each needed it: search(), filterFor(), and the new publicScope()/publicQuery() for the cases that call a retriever or the hydrator directly, where nothing above them is running to fill the scope in. A fixture that names no audience is now public, which is the same default the publish mapper applies (A28) — without it, every unrelated case would have to state a network to be found at all. TestAnOmittedNetworkIdSearchesEveryNetworkAndAGivenOneNarrows asserted the rule A29 reverses, and its name said so. It is now AnOmittedNetworkIdMatchesOnlyPublicAndANamedOneNarrows and pins all three rules against three catalogs, because a fix aimed at any one of them is a regression in another. An empty networkId was a 200 holding nothing. "" unmarshalled to a one-element scope no row's visible_to can hold, and the spec declares no minLength on the string branch, so L1 let it through. tests/testdata/beckn-v2.0.0.yaml is byte-identical to the pinned upstream and must stay so — the fetch path and an air-gapped mount agree only because it is — so the refusal goes in NetworkIDs.UnmarshalJSON instead, the one place that sees the value before the scope is built. It becomes a NACK at the edge rather than a page a caller cannot tell from "nothing matched". visibleTo took a parameter it did not read. The blank name said the argument was dead while the signature said it was not; the caller still computed it. `network` keeps its real reader two lines up. The memory backend's comment above admitted() still described the pre-A29 rule — an empty scope as UNSCOPED, emitting no predicate. The code beneath it returns nothing for an empty scope. The comment now says what the code does. The 17 discover examples named no network and were answered by nothing, which is what made verify, audit and newman fail at 20, 13 and 19. They now name ["local-network"], which is what 01-publish names in visibleTo, and the Postman collection is regenerated from them. make build, make lint (0 issues), make test, make verify (42/0), make audit (47/0) and make newman (107 assertions, 0 failed) all pass. --- examples/02-discover-text-search.json | 3 + examples/03-discover-schema-context.json | 3 + examples/04-discover-spatial-dwithin.json | 3 + examples/05-discover-spatial-intersects.json | 3 + examples/06-discover-filter-granularity.json | 3 + examples/07-discover-filter-cross-level.json | 3 + examples/08-discover-invalid-jsonpath.json | 3 + examples/09-discover-text-or.json | 3 + examples/10-discover-text-and-geo.json | 3 + examples/11-discover-geo-and-filter.json | 3 + .../12-discover-text-geo-filter-empty.json | 3 + examples/13-discover-fuzzy-typos.json | 3 + examples/14-discover-text-and-filter.json | 3 + examples/15-discover-text-geo-filter.json | 3 + examples/16-discover-filter-only.json | 3 + examples/17-discover-no-criterion.json | 3 + examples/18-discover-unparsable-jsonpath.json | 3 + ...-discovery-service.postman_collection.json | 34 +++---- src/beckn/types.go | 14 +++ src/beckn/types_test.go | 50 ++++++++++ src/publish/mapper.go | 4 +- src/storage/memory/search.go | 4 +- src/storage/postgres/read_path_test.go | 99 +++++++++++++------ 23 files changed, 203 insertions(+), 53 deletions(-) diff --git a/examples/02-discover-text-search.json b/examples/02-discover-text-search.json index 11a2a43..e49c02c 100644 --- a/examples/02-discover-text-search.json +++ b/examples/02-discover-text-search.json @@ -2,6 +2,9 @@ "context": { "domain": "agriculture", "action": "discover", + "networkId": [ + "local-network" + ], "version": "2.0.0", "transactionId": "a1000000-0000-4000-8000-000000000002", "messageId": "b1000000-0000-4000-8000-000000000002", diff --git a/examples/03-discover-schema-context.json b/examples/03-discover-schema-context.json index b764918..d1886fb 100644 --- a/examples/03-discover-schema-context.json +++ b/examples/03-discover-schema-context.json @@ -2,6 +2,9 @@ "context": { "domain": "agriculture", "action": "discover", + "networkId": [ + "local-network" + ], "version": "2.0.0", "transactionId": "a1000000-0000-4000-8000-000000000003", "messageId": "b1000000-0000-4000-8000-000000000003", diff --git a/examples/04-discover-spatial-dwithin.json b/examples/04-discover-spatial-dwithin.json index 57b5435..a3a8974 100644 --- a/examples/04-discover-spatial-dwithin.json +++ b/examples/04-discover-spatial-dwithin.json @@ -2,6 +2,9 @@ "context": { "domain": "agriculture", "action": "discover", + "networkId": [ + "local-network" + ], "version": "2.0.0", "transactionId": "a1000000-0000-4000-8000-000000000004", "messageId": "b1000000-0000-4000-8000-000000000004", diff --git a/examples/05-discover-spatial-intersects.json b/examples/05-discover-spatial-intersects.json index 79ae067..1c461fe 100644 --- a/examples/05-discover-spatial-intersects.json +++ b/examples/05-discover-spatial-intersects.json @@ -2,6 +2,9 @@ "context": { "domain": "agriculture", "action": "discover", + "networkId": [ + "local-network" + ], "version": "2.0.0", "transactionId": "a1000000-0000-4000-8000-000000000005", "messageId": "b1000000-0000-4000-8000-000000000005", diff --git a/examples/06-discover-filter-granularity.json b/examples/06-discover-filter-granularity.json index 88fe8aa..b594fa0 100644 --- a/examples/06-discover-filter-granularity.json +++ b/examples/06-discover-filter-granularity.json @@ -2,6 +2,9 @@ "context": { "domain": "agriculture", "action": "discover", + "networkId": [ + "local-network" + ], "version": "2.0.0", "transactionId": "a1000000-0000-4000-8000-000000000006", "messageId": "b1000000-0000-4000-8000-000000000006", diff --git a/examples/07-discover-filter-cross-level.json b/examples/07-discover-filter-cross-level.json index 3f25167..2fda2c9 100644 --- a/examples/07-discover-filter-cross-level.json +++ b/examples/07-discover-filter-cross-level.json @@ -2,6 +2,9 @@ "context": { "domain": "agriculture", "action": "discover", + "networkId": [ + "local-network" + ], "version": "2.0.0", "transactionId": "a1000000-0000-4000-8000-000000000007", "messageId": "b1000000-0000-4000-8000-000000000007", diff --git a/examples/08-discover-invalid-jsonpath.json b/examples/08-discover-invalid-jsonpath.json index 147a3a4..4346b6b 100644 --- a/examples/08-discover-invalid-jsonpath.json +++ b/examples/08-discover-invalid-jsonpath.json @@ -2,6 +2,9 @@ "context": { "domain": "agriculture", "action": "discover", + "networkId": [ + "local-network" + ], "version": "2.0.0", "transactionId": "a1000000-0000-4000-8000-000000000008", "messageId": "b1000000-0000-4000-8000-000000000008", diff --git a/examples/09-discover-text-or.json b/examples/09-discover-text-or.json index 6a8332e..9dbddf4 100644 --- a/examples/09-discover-text-or.json +++ b/examples/09-discover-text-or.json @@ -2,6 +2,9 @@ "context": { "domain": "agriculture", "action": "discover", + "networkId": [ + "local-network" + ], "version": "2.0.0", "transactionId": "a1000000-0000-4000-8000-000000000009", "messageId": "b1000000-0000-4000-8000-000000000009", diff --git a/examples/10-discover-text-and-geo.json b/examples/10-discover-text-and-geo.json index cee1e41..a17b257 100644 --- a/examples/10-discover-text-and-geo.json +++ b/examples/10-discover-text-and-geo.json @@ -2,6 +2,9 @@ "context": { "domain": "agriculture", "action": "discover", + "networkId": [ + "local-network" + ], "version": "2.0.0", "transactionId": "a1000000-0000-4000-8000-000000000010", "messageId": "b1000000-0000-4000-8000-000000000010", diff --git a/examples/11-discover-geo-and-filter.json b/examples/11-discover-geo-and-filter.json index 4075295..89b738b 100644 --- a/examples/11-discover-geo-and-filter.json +++ b/examples/11-discover-geo-and-filter.json @@ -2,6 +2,9 @@ "context": { "domain": "agriculture", "action": "discover", + "networkId": [ + "local-network" + ], "version": "2.0.0", "transactionId": "a1000000-0000-4000-8000-000000000011", "messageId": "b1000000-0000-4000-8000-000000000011", diff --git a/examples/12-discover-text-geo-filter-empty.json b/examples/12-discover-text-geo-filter-empty.json index 38c1143..f5f4225 100644 --- a/examples/12-discover-text-geo-filter-empty.json +++ b/examples/12-discover-text-geo-filter-empty.json @@ -2,6 +2,9 @@ "context": { "domain": "agriculture", "action": "discover", + "networkId": [ + "local-network" + ], "version": "2.0.0", "transactionId": "a1000000-0000-4000-8000-000000000012", "messageId": "b1000000-0000-4000-8000-000000000012", diff --git a/examples/13-discover-fuzzy-typos.json b/examples/13-discover-fuzzy-typos.json index ea8a41b..9f5f152 100644 --- a/examples/13-discover-fuzzy-typos.json +++ b/examples/13-discover-fuzzy-typos.json @@ -2,6 +2,9 @@ "context": { "domain": "agriculture", "action": "discover", + "networkId": [ + "local-network" + ], "version": "2.0.0", "transactionId": "a1000000-0000-4000-8000-000000000013", "messageId": "b1000000-0000-4000-8000-000000000013", diff --git a/examples/14-discover-text-and-filter.json b/examples/14-discover-text-and-filter.json index dafe06b..1948765 100644 --- a/examples/14-discover-text-and-filter.json +++ b/examples/14-discover-text-and-filter.json @@ -2,6 +2,9 @@ "context": { "domain": "agriculture", "action": "discover", + "networkId": [ + "local-network" + ], "version": "2.0.0", "transactionId": "a1000000-0000-4000-8000-000000000014", "messageId": "b1000000-0000-4000-8000-000000000014", diff --git a/examples/15-discover-text-geo-filter.json b/examples/15-discover-text-geo-filter.json index 3707890..69de8ce 100644 --- a/examples/15-discover-text-geo-filter.json +++ b/examples/15-discover-text-geo-filter.json @@ -2,6 +2,9 @@ "context": { "domain": "agriculture", "action": "discover", + "networkId": [ + "local-network" + ], "version": "2.0.0", "transactionId": "a1000000-0000-4000-8000-000000000015", "messageId": "b1000000-0000-4000-8000-000000000015", diff --git a/examples/16-discover-filter-only.json b/examples/16-discover-filter-only.json index aa379f8..ebe2ce1 100644 --- a/examples/16-discover-filter-only.json +++ b/examples/16-discover-filter-only.json @@ -2,6 +2,9 @@ "context": { "domain": "agriculture", "action": "discover", + "networkId": [ + "local-network" + ], "version": "2.0.0", "transactionId": "a1000000-0000-4000-8000-000000000016", "messageId": "b1000000-0000-4000-8000-000000000016", diff --git a/examples/17-discover-no-criterion.json b/examples/17-discover-no-criterion.json index 36169a3..081c429 100644 --- a/examples/17-discover-no-criterion.json +++ b/examples/17-discover-no-criterion.json @@ -2,6 +2,9 @@ "context": { "domain": "agriculture", "action": "discover", + "networkId": [ + "local-network" + ], "version": "2.0.0", "transactionId": "a1000000-0000-4000-8000-000000000017", "messageId": "b1000000-0000-4000-8000-000000000017", diff --git a/examples/18-discover-unparsable-jsonpath.json b/examples/18-discover-unparsable-jsonpath.json index ae203e4..00c0454 100644 --- a/examples/18-discover-unparsable-jsonpath.json +++ b/examples/18-discover-unparsable-jsonpath.json @@ -2,6 +2,9 @@ "context": { "domain": "agriculture", "action": "discover", + "networkId": [ + "local-network" + ], "version": "2.0.0", "transactionId": "a1000000-0000-4000-8000-000000000018", "messageId": "b1000000-0000-4000-8000-000000000018", diff --git a/examples/OpenAgriNet-discovery-service.postman_collection.json b/examples/OpenAgriNet-discovery-service.postman_collection.json index 19d3ecb..1ca70ed 100644 --- a/examples/OpenAgriNet-discovery-service.postman_collection.json +++ b/examples/OpenAgriNet-discovery-service.postman_collection.json @@ -169,7 +169,7 @@ "description": "The statewide alert says 'Severe weather alerting' and must NOT match. It is the control: if the lexical index were being skipped, all three would come back.", "body": { "mode": "raw", - "raw": "{\n \"context\": {\n \"domain\": \"agriculture\",\n \"action\": \"discover\",\n \"version\": \"2.0.0\",\n \"transactionId\": \"a1000000-0000-4000-8000-000000000002\",\n \"messageId\": \"b1000000-0000-4000-8000-000000000002\",\n \"timestamp\": \"2026-08-27T06:30:00Z\"\n },\n \"message\": {\n \"intent\": {\n \"textSearch\": \"irrigation spraying advisory\"\n }\n }\n}" + "raw": "{\n \"context\": {\n \"domain\": \"agriculture\",\n \"action\": \"discover\",\n \"networkId\": [\n \"local-network\"\n ],\n \"version\": \"2.0.0\",\n \"transactionId\": \"a1000000-0000-4000-8000-000000000002\",\n \"messageId\": \"b1000000-0000-4000-8000-000000000002\",\n \"timestamp\": \"2026-08-27T06:30:00Z\"\n },\n \"message\": {\n \"intent\": {\n \"textSearch\": \"irrigation spraying advisory\"\n }\n }\n}" } }, "event": [ @@ -239,7 +239,7 @@ "description": "'irrigation' appears only in the village and point resources; 'cyclone' only in the statewide alert. All three coming back is what proves lexical retrieval ORs its terms - under AND none would match. Case 02 cannot tell the two apart, so without this one the semantics were only assumed.", "body": { "mode": "raw", - "raw": "{\n \"context\": {\n \"domain\": \"agriculture\",\n \"action\": \"discover\",\n \"version\": \"2.0.0\",\n \"transactionId\": \"a1000000-0000-4000-8000-000000000009\",\n \"messageId\": \"b1000000-0000-4000-8000-000000000009\",\n \"timestamp\": \"2026-08-27T06:30:00Z\"\n },\n \"message\": {\n \"intent\": {\n \"textSearch\": \"irrigation cyclone\"\n }\n }\n}" + "raw": "{\n \"context\": {\n \"domain\": \"agriculture\",\n \"action\": \"discover\",\n \"networkId\": [\n \"local-network\"\n ],\n \"version\": \"2.0.0\",\n \"transactionId\": \"a1000000-0000-4000-8000-000000000009\",\n \"messageId\": \"b1000000-0000-4000-8000-000000000009\",\n \"timestamp\": \"2026-08-27T06:30:00Z\"\n },\n \"message\": {\n \"intent\": {\n \"textSearch\": \"irrigation cyclone\"\n }\n }\n}" } }, "event": [ @@ -309,7 +309,7 @@ "description": "schemaContext is a CONTEXT field, not an intent one - Intent is additionalProperties:false, so sending it under message.intent is a body that fails its own schema. All three resources carry the same @context, so this case pins acceptance rather than discrimination.", "body": { "mode": "raw", - "raw": "{\n \"context\": {\n \"domain\": \"agriculture\",\n \"action\": \"discover\",\n \"version\": \"2.0.0\",\n \"transactionId\": \"a1000000-0000-4000-8000-000000000003\",\n \"messageId\": \"b1000000-0000-4000-8000-000000000003\",\n \"timestamp\": \"2026-08-27T06:30:00Z\",\n \"schemaContext\": [\n \"https://schemas.openagrinet.global/schema/WeatherAdvisoryCapability/v0.1/context.jsonld#openagrinet:WeatherAdvisoryCapability\"\n ]\n },\n \"message\": {\n \"intent\": {\n \"textSearch\": \"weather\"\n }\n }\n}" + "raw": "{\n \"context\": {\n \"domain\": \"agriculture\",\n \"action\": \"discover\",\n \"networkId\": [\n \"local-network\"\n ],\n \"version\": \"2.0.0\",\n \"transactionId\": \"a1000000-0000-4000-8000-000000000003\",\n \"messageId\": \"b1000000-0000-4000-8000-000000000003\",\n \"timestamp\": \"2026-08-27T06:30:00Z\",\n \"schemaContext\": [\n \"https://schemas.openagrinet.global/schema/WeatherAdvisoryCapability/v0.1/context.jsonld#openagrinet:WeatherAdvisoryCapability\"\n ]\n },\n \"message\": {\n \"intent\": {\n \"textSearch\": \"weather\"\n }\n }\n}" } }, "event": [ @@ -379,7 +379,7 @@ "description": "The Dharwad Point is ~2km away and the Belagavi Polygon contains the query coordinate. The statewide alert carries only an ISO-3166-2 area CODE and no coordinates, so nothing was cell-indexed for it and it is not spatially discoverable at all - by design.", "body": { "mode": "raw", - "raw": "{\n \"context\": {\n \"domain\": \"agriculture\",\n \"action\": \"discover\",\n \"version\": \"2.0.0\",\n \"transactionId\": \"a1000000-0000-4000-8000-000000000004\",\n \"messageId\": \"b1000000-0000-4000-8000-000000000004\",\n \"timestamp\": \"2026-08-27T06:30:00Z\"\n },\n \"message\": {\n \"intent\": {\n \"spatial\": [\n {\n \"op\": \"S_DWITHIN\",\n \"targets\": \"$.catalogs[*].resources[*].resourceAttributes.coverageAreas[*]\",\n \"geometry\": {\n \"type\": \"Point\",\n \"coordinates\": [\n 75.02,\n 15.47\n ]\n },\n \"distanceMeters\": 25000\n }\n ]\n }\n }\n}" + "raw": "{\n \"context\": {\n \"domain\": \"agriculture\",\n \"action\": \"discover\",\n \"networkId\": [\n \"local-network\"\n ],\n \"version\": \"2.0.0\",\n \"transactionId\": \"a1000000-0000-4000-8000-000000000004\",\n \"messageId\": \"b1000000-0000-4000-8000-000000000004\",\n \"timestamp\": \"2026-08-27T06:30:00Z\"\n },\n \"message\": {\n \"intent\": {\n \"spatial\": [\n {\n \"op\": \"S_DWITHIN\",\n \"targets\": \"$.catalogs[*].resources[*].resourceAttributes.coverageAreas[*]\",\n \"geometry\": {\n \"type\": \"Point\",\n \"coordinates\": [\n 75.02,\n 15.47\n ]\n },\n \"distanceMeters\": 25000\n }\n ]\n }\n }\n}" } }, "event": [ @@ -449,7 +449,7 @@ "description": "(74.50, 16.00) is deep inside the Belagavi polygon and ~120km from the Dharwad point. This case and case 04 have to DISAGREE - that is what separates a working predicate from one matching the whole catalog.", "body": { "mode": "raw", - "raw": "{\n \"context\": {\n \"domain\": \"agriculture\",\n \"action\": \"discover\",\n \"version\": \"2.0.0\",\n \"transactionId\": \"a1000000-0000-4000-8000-000000000005\",\n \"messageId\": \"b1000000-0000-4000-8000-000000000005\",\n \"timestamp\": \"2026-08-27T06:30:00Z\"\n },\n \"message\": {\n \"intent\": {\n \"spatial\": [\n {\n \"op\": \"S_INTERSECTS\",\n \"targets\": \"$['catalogs'][*]['resources'][*]['resourceAttributes']['coverageAreas'][*]\",\n \"geometry\": {\n \"type\": \"Point\",\n \"coordinates\": [\n 74.5,\n 16.0\n ]\n }\n }\n ]\n }\n }\n}" + "raw": "{\n \"context\": {\n \"domain\": \"agriculture\",\n \"action\": \"discover\",\n \"networkId\": [\n \"local-network\"\n ],\n \"version\": \"2.0.0\",\n \"transactionId\": \"a1000000-0000-4000-8000-000000000005\",\n \"messageId\": \"b1000000-0000-4000-8000-000000000005\",\n \"timestamp\": \"2026-08-27T06:30:00Z\"\n },\n \"message\": {\n \"intent\": {\n \"spatial\": [\n {\n \"op\": \"S_INTERSECTS\",\n \"targets\": \"$['catalogs'][*]['resources'][*]['resourceAttributes']['coverageAreas'][*]\",\n \"geometry\": {\n \"type\": \"Point\",\n \"coordinates\": [\n 74.5,\n 16.0\n ]\n }\n }\n ]\n }\n }\n}" } }, "event": [ @@ -525,7 +525,7 @@ "description": "A jsonpath filter rooted at the resource level.", "body": { "mode": "raw", - "raw": "{\n \"context\": {\n \"domain\": \"agriculture\",\n \"action\": \"discover\",\n \"version\": \"2.0.0\",\n \"transactionId\": \"a1000000-0000-4000-8000-000000000006\",\n \"messageId\": \"b1000000-0000-4000-8000-000000000006\",\n \"timestamp\": \"2026-08-27T06:30:00Z\"\n },\n \"message\": {\n \"intent\": {\n \"textSearch\": \"weather advisory\",\n \"filters\": {\n \"type\": \"jsonpath\",\n \"expression\": \"$.catalogs[*].resources[*] ? (@.resourceAttributes.geographicGranularity == \\\"Village\\\")\"\n }\n }\n }\n}" + "raw": "{\n \"context\": {\n \"domain\": \"agriculture\",\n \"action\": \"discover\",\n \"networkId\": [\n \"local-network\"\n ],\n \"version\": \"2.0.0\",\n \"transactionId\": \"a1000000-0000-4000-8000-000000000006\",\n \"messageId\": \"b1000000-0000-4000-8000-000000000006\",\n \"timestamp\": \"2026-08-27T06:30:00Z\"\n },\n \"message\": {\n \"intent\": {\n \"textSearch\": \"weather advisory\",\n \"filters\": {\n \"type\": \"jsonpath\",\n \"expression\": \"$.catalogs[*].resources[*] ? (@.resourceAttributes.geographicGranularity == \\\"Village\\\")\"\n }\n }\n }\n}" } }, "event": [ @@ -601,7 +601,7 @@ "description": "Rooted at the OFFER level and yet it narrows RESOURCES. This exercises the single composite filter_doc column (A18): under the earlier three-column design an offer-rooted predicate could not select a resource at all. If this returns all three, the composite regressed.", "body": { "mode": "raw", - "raw": "{\n \"context\": {\n \"domain\": \"agriculture\",\n \"action\": \"discover\",\n \"version\": \"2.0.0\",\n \"transactionId\": \"a1000000-0000-4000-8000-000000000007\",\n \"messageId\": \"b1000000-0000-4000-8000-000000000007\",\n \"timestamp\": \"2026-08-27T06:30:00Z\"\n },\n \"message\": {\n \"intent\": {\n \"textSearch\": \"weather\",\n \"filters\": {\n \"type\": \"jsonpath\",\n \"expression\": \"$.catalogs[*].offers[*] ? (@.descriptor.code == \\\"SUBSCRIPTION\\\")\"\n }\n }\n }\n}" + "raw": "{\n \"context\": {\n \"domain\": \"agriculture\",\n \"action\": \"discover\",\n \"networkId\": [\n \"local-network\"\n ],\n \"version\": \"2.0.0\",\n \"transactionId\": \"a1000000-0000-4000-8000-000000000007\",\n \"messageId\": \"b1000000-0000-4000-8000-000000000007\",\n \"timestamp\": \"2026-08-27T06:30:00Z\"\n },\n \"message\": {\n \"intent\": {\n \"textSearch\": \"weather\",\n \"filters\": {\n \"type\": \"jsonpath\",\n \"expression\": \"$.catalogs[*].offers[*] ? (@.descriptor.code == \\\"SUBSCRIPTION\\\")\"\n }\n }\n }\n}" } }, "event": [ @@ -677,7 +677,7 @@ "description": "Constraints INTERSECT. 'cotton' appears only in the point resource and 'cyclone' only in the alert, so text alone gives point+alert; the 25km circle alone gives point+village. Only the point is in both. Each dimension excludes something the other admits, which is what makes a dropped predicate visible here.", "body": { "mode": "raw", - "raw": "{\n \"context\": {\n \"domain\": \"agriculture\",\n \"action\": \"discover\",\n \"version\": \"2.0.0\",\n \"transactionId\": \"a1000000-0000-4000-8000-000000000010\",\n \"messageId\": \"b1000000-0000-4000-8000-000000000010\",\n \"timestamp\": \"2026-08-27T06:30:00Z\"\n },\n \"message\": {\n \"intent\": {\n \"textSearch\": \"cotton cyclone\",\n \"spatial\": [\n {\n \"op\": \"S_DWITHIN\",\n \"targets\": \"$.catalogs[*].resources[*].resourceAttributes.coverageAreas[*]\",\n \"geometry\": {\n \"type\": \"Point\",\n \"coordinates\": [\n 75.02,\n 15.47\n ]\n },\n \"distanceMeters\": 25000\n }\n ]\n }\n }\n}" + "raw": "{\n \"context\": {\n \"domain\": \"agriculture\",\n \"action\": \"discover\",\n \"networkId\": [\n \"local-network\"\n ],\n \"version\": \"2.0.0\",\n \"transactionId\": \"a1000000-0000-4000-8000-000000000010\",\n \"messageId\": \"b1000000-0000-4000-8000-000000000010\",\n \"timestamp\": \"2026-08-27T06:30:00Z\"\n },\n \"message\": {\n \"intent\": {\n \"textSearch\": \"cotton cyclone\",\n \"spatial\": [\n {\n \"op\": \"S_DWITHIN\",\n \"targets\": \"$.catalogs[*].resources[*].resourceAttributes.coverageAreas[*]\",\n \"geometry\": {\n \"type\": \"Point\",\n \"coordinates\": [\n 75.02,\n 15.47\n ]\n },\n \"distanceMeters\": 25000\n }\n ]\n }\n }\n}" } }, "event": [ @@ -747,7 +747,7 @@ "description": "Geo alone gives point+village, the FREE-TIER offer covers village+alert. Village is the only overlap. Note there is no textSearch at all: a NULL query_text must admit every row the other predicates allow, not none, or a geo-only discover answers empty.", "body": { "mode": "raw", - "raw": "{\n \"context\": {\n \"domain\": \"agriculture\",\n \"action\": \"discover\",\n \"version\": \"2.0.0\",\n \"transactionId\": \"a1000000-0000-4000-8000-000000000011\",\n \"messageId\": \"b1000000-0000-4000-8000-000000000011\",\n \"timestamp\": \"2026-08-27T06:30:00Z\"\n },\n \"message\": {\n \"intent\": {\n \"spatial\": [\n {\n \"op\": \"S_DWITHIN\",\n \"targets\": \"$.catalogs[*].resources[*].resourceAttributes.coverageAreas[*]\",\n \"geometry\": {\n \"type\": \"Point\",\n \"coordinates\": [\n 75.02,\n 15.47\n ]\n },\n \"distanceMeters\": 25000\n }\n ],\n \"filters\": {\n \"type\": \"jsonpath\",\n \"expression\": \"$.catalogs[*].offers[*] ? (@.descriptor.code == \\\"FREE-TIER\\\")\"\n }\n }\n }\n}" + "raw": "{\n \"context\": {\n \"domain\": \"agriculture\",\n \"action\": \"discover\",\n \"networkId\": [\n \"local-network\"\n ],\n \"version\": \"2.0.0\",\n \"transactionId\": \"a1000000-0000-4000-8000-000000000011\",\n \"messageId\": \"b1000000-0000-4000-8000-000000000011\",\n \"timestamp\": \"2026-08-27T06:30:00Z\"\n },\n \"message\": {\n \"intent\": {\n \"spatial\": [\n {\n \"op\": \"S_DWITHIN\",\n \"targets\": \"$.catalogs[*].resources[*].resourceAttributes.coverageAreas[*]\",\n \"geometry\": {\n \"type\": \"Point\",\n \"coordinates\": [\n 75.02,\n 15.47\n ]\n },\n \"distanceMeters\": 25000\n }\n ],\n \"filters\": {\n \"type\": \"jsonpath\",\n \"expression\": \"$.catalogs[*].offers[*] ? (@.descriptor.code == \\\"FREE-TIER\\\")\"\n }\n }\n }\n}" } }, "event": [ @@ -823,7 +823,7 @@ "description": "The third leg of the leave-one-out triangle around case 12. Drop the filter and you get case 10 (point); drop the text and you get case 11 (village); drop the GEOMETRY and you get this (alert).", "body": { "mode": "raw", - "raw": "{\n \"context\": {\n \"domain\": \"agriculture\",\n \"action\": \"discover\",\n \"version\": \"2.0.0\",\n \"transactionId\": \"a1000000-0000-4000-8000-000000000014\",\n \"messageId\": \"b1000000-0000-4000-8000-000000000014\",\n \"timestamp\": \"2026-08-27T06:30:00Z\"\n },\n \"message\": {\n \"intent\": {\n \"textSearch\": \"cotton cyclone\",\n \"filters\": {\n \"type\": \"jsonpath\",\n \"expression\": \"$.catalogs[*].offers[*] ? (@.descriptor.code == \\\"FREE-TIER\\\")\"\n }\n }\n }\n}" + "raw": "{\n \"context\": {\n \"domain\": \"agriculture\",\n \"action\": \"discover\",\n \"networkId\": [\n \"local-network\"\n ],\n \"version\": \"2.0.0\",\n \"transactionId\": \"a1000000-0000-4000-8000-000000000014\",\n \"messageId\": \"b1000000-0000-4000-8000-000000000014\",\n \"timestamp\": \"2026-08-27T06:30:00Z\"\n },\n \"message\": {\n \"intent\": {\n \"textSearch\": \"cotton cyclone\",\n \"filters\": {\n \"type\": \"jsonpath\",\n \"expression\": \"$.catalogs[*].offers[*] ? (@.descriptor.code == \\\"FREE-TIER\\\")\"\n }\n }\n }\n}" } }, "event": [ @@ -899,7 +899,7 @@ "description": "text -> point+alert, geo -> point+village, filter -> village+alert. Every PAIR overlaps in exactly one resource and all three share none, so removing ANY dimension changes the answer to a different single resource (10, 11, 14) and keeping all three gives EMPTY. That is what proves all three are applied and ANDed. An earlier version of this case came out empty whichever of two dimensions you dropped, so it only ever proved one of them mattered.", "body": { "mode": "raw", - "raw": "{\n \"context\": {\n \"domain\": \"agriculture\",\n \"action\": \"discover\",\n \"version\": \"2.0.0\",\n \"transactionId\": \"a1000000-0000-4000-8000-000000000012\",\n \"messageId\": \"b1000000-0000-4000-8000-000000000012\",\n \"timestamp\": \"2026-08-27T06:30:00Z\"\n },\n \"message\": {\n \"intent\": {\n \"textSearch\": \"cotton cyclone\",\n \"spatial\": [\n {\n \"op\": \"S_DWITHIN\",\n \"targets\": \"$.catalogs[*].resources[*].resourceAttributes.coverageAreas[*]\",\n \"geometry\": {\n \"type\": \"Point\",\n \"coordinates\": [\n 75.02,\n 15.47\n ]\n },\n \"distanceMeters\": 25000\n }\n ],\n \"filters\": {\n \"type\": \"jsonpath\",\n \"expression\": \"$.catalogs[*].offers[*] ? (@.descriptor.code == \\\"FREE-TIER\\\")\"\n }\n }\n }\n}" + "raw": "{\n \"context\": {\n \"domain\": \"agriculture\",\n \"action\": \"discover\",\n \"networkId\": [\n \"local-network\"\n ],\n \"version\": \"2.0.0\",\n \"transactionId\": \"a1000000-0000-4000-8000-000000000012\",\n \"messageId\": \"b1000000-0000-4000-8000-000000000012\",\n \"timestamp\": \"2026-08-27T06:30:00Z\"\n },\n \"message\": {\n \"intent\": {\n \"textSearch\": \"cotton cyclone\",\n \"spatial\": [\n {\n \"op\": \"S_DWITHIN\",\n \"targets\": \"$.catalogs[*].resources[*].resourceAttributes.coverageAreas[*]\",\n \"geometry\": {\n \"type\": \"Point\",\n \"coordinates\": [\n 75.02,\n 15.47\n ]\n },\n \"distanceMeters\": 25000\n }\n ],\n \"filters\": {\n \"type\": \"jsonpath\",\n \"expression\": \"$.catalogs[*].offers[*] ? (@.descriptor.code == \\\"FREE-TIER\\\")\"\n }\n }\n }\n}" } }, "event": [ @@ -969,7 +969,7 @@ "description": "The same three dimensions with a text term reaching all three resources, so the answer is non-empty. Geometry and filter are each load-bearing here and the text is not: with three resources and only two geo-indexed, a non-empty three-way answer cannot make all three matter at once. Case 12 is where that proof lives.", "body": { "mode": "raw", - "raw": "{\n \"context\": {\n \"domain\": \"agriculture\",\n \"action\": \"discover\",\n \"version\": \"2.0.0\",\n \"transactionId\": \"a1000000-0000-4000-8000-000000000015\",\n \"messageId\": \"b1000000-0000-4000-8000-000000000015\",\n \"timestamp\": \"2026-08-27T06:30:00Z\"\n },\n \"message\": {\n \"intent\": {\n \"textSearch\": \"cyclone irrigation\",\n \"spatial\": [\n {\n \"op\": \"S_DWITHIN\",\n \"targets\": \"$.catalogs[*].resources[*].resourceAttributes.coverageAreas[*]\",\n \"geometry\": {\n \"type\": \"Point\",\n \"coordinates\": [\n 75.02,\n 15.47\n ]\n },\n \"distanceMeters\": 25000\n }\n ],\n \"filters\": {\n \"type\": \"jsonpath\",\n \"expression\": \"$.catalogs[*].offers[*] ? (@.descriptor.code == \\\"FREE-TIER\\\")\"\n }\n }\n }\n}" + "raw": "{\n \"context\": {\n \"domain\": \"agriculture\",\n \"action\": \"discover\",\n \"networkId\": [\n \"local-network\"\n ],\n \"version\": \"2.0.0\",\n \"transactionId\": \"a1000000-0000-4000-8000-000000000015\",\n \"messageId\": \"b1000000-0000-4000-8000-000000000015\",\n \"timestamp\": \"2026-08-27T06:30:00Z\"\n },\n \"message\": {\n \"intent\": {\n \"textSearch\": \"cyclone irrigation\",\n \"spatial\": [\n {\n \"op\": \"S_DWITHIN\",\n \"targets\": \"$.catalogs[*].resources[*].resourceAttributes.coverageAreas[*]\",\n \"geometry\": {\n \"type\": \"Point\",\n \"coordinates\": [\n 75.02,\n 15.47\n ]\n },\n \"distanceMeters\": 25000\n }\n ],\n \"filters\": {\n \"type\": \"jsonpath\",\n \"expression\": \"$.catalogs[*].offers[*] ? (@.descriptor.code == \\\"FREE-TIER\\\")\"\n }\n }\n }\n}" } }, "event": [ @@ -1045,7 +1045,7 @@ "description": "The only case that reaches the candidates path with a filter on it. An intent naming no ranked mode is answered by the lexical retriever with a NULL query_text; 04 and 05 exercise that path with a geometry, and nothing else exercises it with a filter. If the NULL-query fallthrough stopped applying filter_doc, this returns all three and every other case still passes.", "body": { "mode": "raw", - "raw": "{\n \"context\": {\n \"domain\": \"agriculture\",\n \"action\": \"discover\",\n \"version\": \"2.0.0\",\n \"transactionId\": \"a1000000-0000-4000-8000-000000000016\",\n \"messageId\": \"b1000000-0000-4000-8000-000000000016\",\n \"timestamp\": \"2026-08-27T06:30:00Z\"\n },\n \"message\": {\n \"intent\": {\n \"filters\": {\n \"type\": \"jsonpath\",\n \"expression\": \"$.catalogs[*].resources[*] ? (@.resourceAttributes.geographicGranularity == \\\"Village\\\")\"\n }\n }\n }\n}" + "raw": "{\n \"context\": {\n \"domain\": \"agriculture\",\n \"action\": \"discover\",\n \"networkId\": [\n \"local-network\"\n ],\n \"version\": \"2.0.0\",\n \"transactionId\": \"a1000000-0000-4000-8000-000000000016\",\n \"messageId\": \"b1000000-0000-4000-8000-000000000016\",\n \"timestamp\": \"2026-08-27T06:30:00Z\"\n },\n \"message\": {\n \"intent\": {\n \"filters\": {\n \"type\": \"jsonpath\",\n \"expression\": \"$.catalogs[*].resources[*] ? (@.resourceAttributes.geographicGranularity == \\\"Village\\\")\"\n }\n }\n }\n}" } }, "event": [ @@ -1115,7 +1115,7 @@ "description": "Every term is misspelled, so the tsvector admits nothing. The resource comes back through the trigram index on `name`, which gates on similarity >= 0.3 (pg_trgm's default threshold) and NOT on the searchable text the lexical mode uses. Constraints intersect, but retrieval modes union - this is the case that tells them apart.", "body": { "mode": "raw", - "raw": "{\n \"context\": {\n \"domain\": \"agriculture\",\n \"action\": \"discover\",\n \"version\": \"2.0.0\",\n \"transactionId\": \"a1000000-0000-4000-8000-000000000013\",\n \"messageId\": \"b1000000-0000-4000-8000-000000000013\",\n \"timestamp\": \"2026-08-27T06:30:00Z\"\n },\n \"message\": {\n \"intent\": {\n \"textSearch\": \"Vilage weathr advisery Belgavi\"\n }\n }\n}" + "raw": "{\n \"context\": {\n \"domain\": \"agriculture\",\n \"action\": \"discover\",\n \"networkId\": [\n \"local-network\"\n ],\n \"version\": \"2.0.0\",\n \"transactionId\": \"a1000000-0000-4000-8000-000000000013\",\n \"messageId\": \"b1000000-0000-4000-8000-000000000013\",\n \"timestamp\": \"2026-08-27T06:30:00Z\"\n },\n \"message\": {\n \"intent\": {\n \"textSearch\": \"Vilage weathr advisery Belgavi\"\n }\n }\n}" } }, "event": [ @@ -1190,7 +1190,7 @@ "description": "Expected to FAIL with 400. See the test script for why this shape is dangerous enough to refuse.", "body": { "mode": "raw", - "raw": "{\n \"context\": {\n \"domain\": \"agriculture\",\n \"action\": \"discover\",\n \"version\": \"2.0.0\",\n \"transactionId\": \"a1000000-0000-4000-8000-000000000008\",\n \"messageId\": \"b1000000-0000-4000-8000-000000000008\",\n \"timestamp\": \"2026-08-27T06:30:00Z\"\n },\n \"message\": {\n \"intent\": {\n \"textSearch\": \"weather\",\n \"filters\": {\n \"type\": \"jsonpath\",\n \"expression\": \"$.catalogs[*].resources[*].resourceAttributes.geographicGranularity == \\\"Village\\\"\"\n }\n }\n }\n}" + "raw": "{\n \"context\": {\n \"domain\": \"agriculture\",\n \"action\": \"discover\",\n \"networkId\": [\n \"local-network\"\n ],\n \"version\": \"2.0.0\",\n \"transactionId\": \"a1000000-0000-4000-8000-000000000008\",\n \"messageId\": \"b1000000-0000-4000-8000-000000000008\",\n \"timestamp\": \"2026-08-27T06:30:00Z\"\n },\n \"message\": {\n \"intent\": {\n \"textSearch\": \"weather\",\n \"filters\": {\n \"type\": \"jsonpath\",\n \"expression\": \"$.catalogs[*].resources[*].resourceAttributes.geographicGranularity == \\\"Village\\\"\"\n }\n }\n }\n}" } }, "event": [ @@ -1239,7 +1239,7 @@ "description": "Expected to FAIL with 400. An empty page would be a plausible answer to a search that never ran.", "body": { "mode": "raw", - "raw": "{\n \"context\": {\n \"domain\": \"agriculture\",\n \"action\": \"discover\",\n \"version\": \"2.0.0\",\n \"transactionId\": \"a1000000-0000-4000-8000-000000000017\",\n \"messageId\": \"b1000000-0000-4000-8000-000000000017\",\n \"timestamp\": \"2026-08-27T06:30:00Z\"\n },\n \"message\": {\n \"intent\": {}\n }\n}" + "raw": "{\n \"context\": {\n \"domain\": \"agriculture\",\n \"action\": \"discover\",\n \"networkId\": [\n \"local-network\"\n ],\n \"version\": \"2.0.0\",\n \"transactionId\": \"a1000000-0000-4000-8000-000000000017\",\n \"messageId\": \"b1000000-0000-4000-8000-000000000017\",\n \"timestamp\": \"2026-08-27T06:30:00Z\"\n },\n \"message\": {\n \"intent\": {}\n }\n}" } }, "event": [ @@ -1288,7 +1288,7 @@ "description": "Expected to FAIL with 400 rather than 500 - the dropped dot is the caller's to fix, and a 5xx asks them to retry it unchanged.", "body": { "mode": "raw", - "raw": "{\n \"context\": {\n \"domain\": \"agriculture\",\n \"action\": \"discover\",\n \"version\": \"2.0.0\",\n \"transactionId\": \"a1000000-0000-4000-8000-000000000018\",\n \"messageId\": \"b1000000-0000-4000-8000-000000000018\",\n \"timestamp\": \"2026-08-27T06:30:00Z\"\n },\n \"message\": {\n \"intent\": {\n \"textSearch\": \"weather advisory\",\n \"filters\": {\n \"type\": \"jsonpath\",\n \"expression\": \"$.catalogs[*]resources[*] ? (@.resourceAttributes.geographicGranularity == \\\"Village\\\")\"\n }\n }\n }\n}" + "raw": "{\n \"context\": {\n \"domain\": \"agriculture\",\n \"action\": \"discover\",\n \"networkId\": [\n \"local-network\"\n ],\n \"version\": \"2.0.0\",\n \"transactionId\": \"a1000000-0000-4000-8000-000000000018\",\n \"messageId\": \"b1000000-0000-4000-8000-000000000018\",\n \"timestamp\": \"2026-08-27T06:30:00Z\"\n },\n \"message\": {\n \"intent\": {\n \"textSearch\": \"weather advisory\",\n \"filters\": {\n \"type\": \"jsonpath\",\n \"expression\": \"$.catalogs[*]resources[*] ? (@.resourceAttributes.geographicGranularity == \\\"Village\\\")\"\n }\n }\n }\n}" } }, "event": [ diff --git a/src/beckn/types.go b/src/beckn/types.go index 96a43bc..59c8cf8 100644 --- a/src/beckn/types.go +++ b/src/beckn/types.go @@ -19,7 +19,9 @@ package beckn import ( "bytes" "encoding/json" + "errors" "fmt" + "slices" ) // NetworkIDs is context.networkId, which the spec declares as one network or a @@ -46,6 +48,12 @@ func (n NetworkIDs) Networks() []string { return n.ids } // IsZero drives `omitzero`, so an absent networkId stays absent in the echo. func (n NetworkIDs) IsZero() bool { return len(n.ids) == 0 } +// The spec declares no minLength on the string branch, so "" passes L1, becomes +// a scope no row's visible_to holds, and answers 200 with an empty page that a +// caller cannot tell from "nothing matched". This is the only place that sees +// the value before the scope is built, so it is refused here. +var errEmptyNetworkID = errors.New("networkId must not be an empty string") + // UnmarshalJSON takes either shape and refuses anything else, so a caller // learns at the edge rather than by searching and finding nothing. func (n *NetworkIDs) UnmarshalJSON(data []byte) error { @@ -56,6 +64,9 @@ func (n *NetworkIDs) UnmarshalJSON(data []byte) error { var one string if err := json.Unmarshal(data, &one); err == nil { + if one == "" { + return errEmptyNetworkID + } *n = OneNetwork(one) return nil } @@ -64,6 +75,9 @@ func (n *NetworkIDs) UnmarshalJSON(data []byte) error { if err := json.Unmarshal(data, &many); err != nil { return fmt.Errorf("networkId is a string or an array of strings: %w", err) } + if slices.Contains(many, "") { + return errEmptyNetworkID + } *n = SomeNetworks(many...) return nil } diff --git a/src/beckn/types_test.go b/src/beckn/types_test.go index 49ab8b6..1c5d0ea 100644 --- a/src/beckn/types_test.go +++ b/src/beckn/types_test.go @@ -2,6 +2,7 @@ package beckn import ( "encoding/json" + "slices" "strings" "testing" ) @@ -122,3 +123,52 @@ func TestOfferMarshalJSONOfAGoBuiltValueMarshalsItsFields(t *testing.T) { t.Errorf("offer = %s, want resourceIds [r1 r2]", encoded) } } + +// NetworkIDs carries both wire shapes of context.networkId and echoes back the +// one that arrived. The cases that matter are the shapes, the round trip, and +// the empty id — which the spec's own oneOf accepts (it declares no minLength) +// and which would otherwise become a scope matching no row at all. +func TestNetworkIDsDecodesBothShapesAndRefusesAnEmptyID(t *testing.T) { + for _, testCase := range []struct { + name string + wire string + networks []string + echo string + wantErr bool + }{ + {name: "a string", wire: `"mahavistar"`, networks: []string{"mahavistar"}, echo: `"mahavistar"`}, + {name: "an array", wire: `["mahavistar","*"]`, networks: []string{"mahavistar", "*"}, echo: `["mahavistar","*"]`}, + {name: "a one-element array stays an array", wire: `["mahavistar"]`, networks: []string{"mahavistar"}, echo: `["mahavistar"]`}, + {name: "null is absent", wire: `null`, networks: nil, echo: `null`}, + {name: "an empty string", wire: `""`, wantErr: true}, + {name: "an empty string inside an array", wire: `["mahavistar",""]`, wantErr: true}, + {name: "neither shape", wire: `7`, wantErr: true}, + } { + t.Run(testCase.name, func(t *testing.T) { + var decoded NetworkIDs + err := json.Unmarshal([]byte(testCase.wire), &decoded) + if testCase.wantErr { + if err == nil { + t.Fatalf("%s decoded to %v, want a refusal at the edge", testCase.wire, decoded.Networks()) + } + return + } + if err != nil { + t.Fatalf("decoding %s: %v", testCase.wire, err) + } + + if got := decoded.Networks(); !slices.Equal(got, testCase.networks) { + t.Errorf("%s decoded to %v, want %v", testCase.wire, got, testCase.networks) + } + + echoed, err := json.Marshal(decoded) + if err != nil { + t.Fatalf("marshalling %s back: %v", testCase.wire, err) + } + if string(echoed) != testCase.echo { + t.Errorf("%s echoed as %s, want %s; the response must spell networkId "+ + "the way the request did", testCase.wire, echoed, testCase.echo) + } + }) + } +} diff --git a/src/publish/mapper.go b/src/publish/mapper.go index de67572..ccf288f 100644 --- a/src/publish/mapper.go +++ b/src/publish/mapper.go @@ -71,7 +71,7 @@ func MapCatalog( Validity: validity, Active: active, - VisibleTo: visibleTo(directive, network), + VisibleTo: visibleTo(directive), ProtocolVersion: protocolVersion(version), @@ -131,7 +131,7 @@ func catalogID(catalog beckn.Catalog, directive beckn.PublishDirective) string { // visibleTo resolves C8: an omitted or empty visibleTo is every network, which // is what the spec says it means. Narrowing is the publisher's to ask for. -func visibleTo(directive beckn.PublishDirective, _ string) []string { +func visibleTo(directive beckn.PublishDirective) []string { if len(directive.VisibleTo) == 0 { return []string{domain.NetworkWildcard} } diff --git a/src/storage/memory/search.go b/src/storage/memory/search.go index dab0b74..71c19da 100644 --- a/src/storage/memory/search.go +++ b/src/storage/memory/search.go @@ -137,8 +137,8 @@ func (r *Repository) candidates(query domain.SearchQuery, scope domain.Scope) [] // gate onto every resource unconditionally, and reading it from anywhere else // would let this backend answer correctly while the copy Postgres reads was wrong. // -// An empty scope network is UNSCOPED and emits no predicate at all — never a -// literal that matches nothing, and never this service's own network id. +// The scope is always a non-empty set of networks — {'*'} when the caller named +// none (A29) — so the gate is one overlap test and never an absent predicate. func admitted(resource domain.Resource, scope domain.Scope) bool { if !visibleOn(resource.VisibleTo, scope.NetworkIDs) { return false diff --git a/src/storage/postgres/read_path_test.go b/src/storage/postgres/read_path_test.go index 867922b..13791dd 100644 --- a/src/storage/postgres/read_path_test.go +++ b/src/storage/postgres/read_path_test.go @@ -157,9 +157,13 @@ func publish(t *testing.T, embedder embeddings.Embedder, fixtures ...readFixture writer := postgres.NewCatalogRepository(pool, geo.DefaultTestResolution) for _, fixture := range fixtures { + // The same default the publish mapper applies (A28): a fixture that + // says nothing about its audience is public, so the cases here that + // are about paging or geometry rather than the scope are found by the + // wildcard search() builds when they name no network. visibleTo := fixture.visibleTo if visibleTo == nil { - visibleTo = []string{"local-network"} + visibleTo = []string{domain.NetworkWildcard} } derive := fixture.derive if derive == nil { @@ -168,7 +172,7 @@ func publish(t *testing.T, embedder embeddings.Embedder, fixtures ...readFixture faults, err := writer.UpsertCatalog(context.Background(), domain.CatalogPatch{ ID: fixture.catalog, - NetworkID: visibleTo[0], + NetworkID: "local-network", Document: fixture.document, Active: true, ProtocolVersion: beckn.Version, @@ -203,12 +207,32 @@ func matchedIDs(result domain.SearchResult) []string { return ids } +// publicScope is the scope discover.Service builds for a request that names no +// network (A29): the wildcard, which is also what a fixture's default audience +// holds. The cases that call a retriever or the hydrator directly need it, +// because nothing above them is running to fill it in. +func publicScope(now time.Time) domain.Scope { + return domain.Scope{NetworkIDs: []string{domain.NetworkWildcard}, Now: now} +} + +// publicQuery is the same default on the query side, for the cases that call +// the repository or a retriever directly rather than through search(). +func publicQuery(query domain.SearchQuery) domain.SearchQuery { + query.NetworkIDs = []string{domain.NetworkWildcard} + return query +} + func search(t *testing.T, repository *postgres.SearchRepository, query domain.SearchQuery) domain.SearchResult { t.Helper() if query.Limit == 0 { query.Limit = searchConfig().MaxPageSize } + // discover.Service fills this before a backend ever sees it, and these + // cases call the backend directly. + if len(query.NetworkIDs) == 0 { + query.NetworkIDs = []string{domain.NetworkWildcard} + } result, err := repository.Search(context.Background(), query, bothTextModes) if err != nil { t.Fatalf("search: %v", err) @@ -220,11 +244,11 @@ func search(t *testing.T, repository *postgres.SearchRepository, query domain.Se // the network gate // --------------------------------------------------------------------------- -// Both directions in ONE test, because a fix aimed at either one is a -// regression in the other: defaulting the empty case to the service's own -// network makes the first assertion fail, and dropping the predicate entirely -// makes the second. -func TestAnOmittedNetworkIdSearchesEveryNetworkAndAGivenOneNarrows(t *testing.T) { +// All three rules in ONE test, because a fix aimed at any one of them is a +// regression in another: defaulting the empty case to every network makes the +// first assertion fail, dropping the predicate entirely makes the second, and +// treating the scope as a single id makes the third. +func TestAnOmittedNetworkIdMatchesOnlyPublicAndANamedOneNarrows(t *testing.T) { repository, _ := publish(t, nil, readFixture{ catalog: "cat-open", @@ -236,20 +260,30 @@ func TestAnOmittedNetworkIdSearchesEveryNetworkAndAGivenOneNarrows(t *testing.T) visibleTo: []string{"private.example.com"}, resources: []domain.ResourcePatch{searchable("r-closed", "wheat seed", "", "https://beckn.org/Agri", "SeedLot")}, }, + readFixture{ + catalog: "cat-public", + visibleTo: []string{domain.NetworkWildcard}, + resources: []domain.ResourcePatch{searchable("r-public", "wheat seed", "", "https://beckn.org/Agri", "SeedLot")}, + }, ) - everyNetwork := matchedIDs(search(t, repository, domain.SearchQuery{Text: "wheat"})) - if !slices.Contains(everyNetwork, "r-closed") { - t.Errorf("an omitted networkId returned %v; it must search EVERY network, "+ - "including one whose visibleTo names a network the caller never mentioned", everyNetwork) - } - if len(everyNetwork) != 2 { - t.Errorf("an omitted networkId matched %d resources, want both: %v", len(everyNetwork), everyNetwork) + public := matchedIDs(search(t, repository, domain.SearchQuery{Text: "wheat"})) + if !slices.Equal(public, []string{"r-public"}) { + t.Errorf("an omitted networkId returned %v, want only the public row; "+ + "naming no network asks for the public catalogs, not for every network", public) } scoped := matchedIDs(search(t, repository, domain.SearchQuery{Text: "wheat", NetworkIDs: []string{"mahavistar"}})) if !slices.Equal(scoped, []string{"r-open"}) { - t.Errorf("networkId=mahavistar returned %v, want only that network's row", scoped) + t.Errorf("networkId=mahavistar returned %v, want only that network's row; "+ + "a named network is not handed the public catalogs as well", scoped) + } + + both := matchedIDs(search(t, repository, + domain.SearchQuery{Text: "wheat", NetworkIDs: []string{"mahavistar", domain.NetworkWildcard}})) + slices.Sort(both) + if !slices.Equal(both, []string{"r-open", "r-public"}) { + t.Errorf("networkId=[mahavistar,*] returned %v, want that network's row and the public one", both) } } @@ -401,7 +435,7 @@ func TestAModeThatFailsIsDegradedAndThePageIsWhatTheOthersFound(t *testing.T) { derive: deriveVectors("r-00", "r-01", "r-02"), }) - query := domain.SearchQuery{Text: "papaya", Limit: searchConfig().MaxPageSize} + query := publicQuery(domain.SearchQuery{Text: "papaya", Limit: searchConfig().MaxPageSize}) result, err := repository.Search(context.Background(), query, []domain.Capability{domain.CapabilityLexical, domain.CapabilityFuzzy, domain.CapabilitySemantic}) if err != nil { @@ -428,7 +462,7 @@ func TestAnUnreachableEmbedderDegradesTheModeRatherThanFailingTheSearch(t *testi }) result, err := repository.Search(context.Background(), - domain.SearchQuery{Text: "kharif", Limit: searchConfig().MaxPageSize}, + publicQuery(domain.SearchQuery{Text: "kharif", Limit: searchConfig().MaxPageSize}), []domain.Capability{domain.CapabilityLexical, domain.CapabilitySemantic}) if err != nil { t.Fatalf("an unreachable embedder failed the whole search: %v", err) @@ -463,7 +497,7 @@ func TestAPagePastTheRetrievalDepthIsAFaultAndNotAnEmptyPage(t *testing.T) { depth := searchConfig().MaxCandidatesPerMode result, err := repository.Search(context.Background(), - domain.SearchQuery{Text: "kharif", Limit: 4, Offset: depth}, bothTextModes) + publicQuery(domain.SearchQuery{Text: "kharif", Limit: 4, Offset: depth}), bothTextModes) if err == nil { t.Fatalf("a page at offset %d returned %d catalogs and no error; the boundary "+ @@ -507,7 +541,7 @@ func TestTheLastPageDoesNotRunPastTheTotal(t *testing.T) { offset := depth - 2 result, err := repository.Search(context.Background(), - domain.SearchQuery{Text: "kharif", Limit: 4, Offset: offset}, bothTextModes) + publicQuery(domain.SearchQuery{Text: "kharif", Limit: 4, Offset: offset}), bothTextModes) if err != nil { t.Fatalf("the last reachable page was refused: %v", err) } @@ -539,14 +573,14 @@ func TestARetrieverNeverReturnsMoreThanItsCap(t *testing.T) { writer := postgres.NewCatalogRepository(pool, geo.DefaultTestResolution) if _, err := writer.UpsertCatalog(context.Background(), domain.CatalogPatch{ ID: "cat-cap", NetworkID: "local-network", Active: true, ProtocolVersion: beckn.Version, - VisibleTo: []string{"local-network"}, Resources: resources, + VisibleTo: []string{domain.NetworkWildcard}, Resources: resources, }, domain.UpdateModeFull, deriveSearchable); err != nil { t.Fatalf("publish: %v", err) } retriever := postgres.NewLexicalRetriever(pool, depth) ids, err := retriever.Retrieve(context.Background(), - domain.SearchQuery{Text: "kharif"}, domain.Scope{Now: time.Now().UTC()}) + publicQuery(domain.SearchQuery{Text: "kharif"}), publicScope(time.Now().UTC())) if err != nil { t.Fatalf("retrieve: %v", err) } @@ -570,7 +604,7 @@ func TestAModeThisBackendCannotRunIsDegradedAndDoesNotFailTheSearch(t *testing.T }) result, err := repository.Search(context.Background(), - domain.SearchQuery{Text: "kharif", Limit: searchConfig().MaxPageSize}, + publicQuery(domain.SearchQuery{Text: "kharif", Limit: searchConfig().MaxPageSize}), []domain.Capability{domain.CapabilityLexical, domain.CapabilitySemantic}) if err != nil { t.Fatalf("a missing mode failed the whole search: %v", err) @@ -611,7 +645,7 @@ func TestFuzzyRetrieverNamesItsOwnQueryFailure(t *testing.T) { }}) retriever := postgres.NewFuzzyRetriever(&postgres.DBTXFailsAt{DBTX: pool, N: 1}, 10) - _, err := retriever.Retrieve(context.Background(), domain.SearchQuery{Text: "kharif"}, domain.Scope{}) + _, err := retriever.Retrieve(context.Background(), publicQuery(domain.SearchQuery{Text: "kharif"}), publicScope(time.Time{})) if err == nil || !strings.Contains(err.Error(), "fuzzy retriever") { t.Errorf("err = %v, want it naming the fuzzy retriever", err) } @@ -640,7 +674,7 @@ func TestAQueryEmbedderThatLiesAboutItsWidthDegradesTheMode(t *testing.T) { }) result, err := repository.Search(context.Background(), - domain.SearchQuery{Text: "kharif", Limit: searchConfig().MaxPageSize}, + publicQuery(domain.SearchQuery{Text: "kharif", Limit: searchConfig().MaxPageSize}), []domain.Capability{domain.CapabilityLexical, domain.CapabilitySemantic}) if err != nil { t.Fatalf("a lying embedder failed the whole search: %v", err) @@ -681,7 +715,7 @@ func TestSemanticRetrieverFindsTheResourceItWasEmbeddedFor(t *testing.T) { }) result, err := repository.Search(context.Background(), - domain.SearchQuery{Text: "kharif wheat", Limit: searchConfig().MaxPageSize}, + publicQuery(domain.SearchQuery{Text: "kharif wheat", Limit: searchConfig().MaxPageSize}), []domain.Capability{domain.CapabilitySemantic}) if err != nil { t.Fatalf("search: %v", err) @@ -706,7 +740,7 @@ func TestSemanticRetrieverWithNoTextEmbedsNothing(t *testing.T) { }}) retriever := postgres.NewSemanticRetriever(pool, embeddings.NewHashing(columnDimensions), 10) - if _, err := retriever.Retrieve(context.Background(), domain.SearchQuery{Text: ""}, domain.Scope{}); err != nil { + if _, err := retriever.Retrieve(context.Background(), publicQuery(domain.SearchQuery{Text: ""}), publicScope(time.Time{})); err != nil { t.Errorf("Retrieve with no text: %v, want no error", err) } } @@ -946,7 +980,7 @@ func TestScopeFilterKeepsAdmittedIdsInTheCallersOrder(t *testing.T) { domain.ResourceKey("cat-scope", "not-a-real-resource"), domain.ResourceKey("cat-scope", "wheat"), } - kept, err := postgres.NewHydrator(pool).ScopeFilter(context.Background(), ids, domain.Scope{}) + kept, err := postgres.NewHydrator(pool).ScopeFilter(context.Background(), ids, publicScope(time.Time{})) if err != nil { t.Fatalf("ScopeFilter: %v", err) } @@ -991,7 +1025,7 @@ func TestScopeFilterAppliesTheNetworkGate(t *testing.T) { func TestScopeFilterOfNoIdsRunsNoQuery(t *testing.T) { _, pool := publish(t, nil, readFixture{catalog: "cat-scope-empty"}) - kept, err := postgres.NewHydrator(pool).ScopeFilter(context.Background(), nil, domain.Scope{}) + kept, err := postgres.NewHydrator(pool).ScopeFilter(context.Background(), nil, publicScope(time.Time{})) if err != nil { t.Fatalf("ScopeFilter: %v", err) } @@ -1047,7 +1081,7 @@ func TestHydrateNamesWhicheverOfItsFourQueriesFailed(t *testing.T) { for _, testCase := range cases { t.Run(fmt.Sprintf("call %d", testCase.call), func(t *testing.T) { hydrator := postgres.NewHydrator(&postgres.DBTXFailsAt{DBTX: pool, N: testCase.call}) - _, err := hydrator.Hydrate(context.Background(), []string{id}, domain.Scope{}) + _, err := hydrator.Hydrate(context.Background(), []string{id}, publicScope(time.Time{})) if err == nil || !strings.Contains(err.Error(), testCase.want) { t.Errorf("err = %v, want it naming %q", err, testCase.want) } @@ -1063,7 +1097,7 @@ func TestScopeFilterNamesItsOwnQueryFailure(t *testing.T) { id := domain.ResourceKey("cat-scope-fail", "wheat") hydrator := postgres.NewHydrator(&postgres.DBTXFailsAt{DBTX: pool, N: 1}) - _, err := hydrator.ScopeFilter(context.Background(), []string{id}, domain.Scope{}) + _, err := hydrator.ScopeFilter(context.Background(), []string{id}, publicScope(time.Time{})) if err == nil || !strings.Contains(err.Error(), "scope gate") { t.Errorf("err = %v, want it naming the scope gate", err) } @@ -1091,8 +1125,9 @@ var filterModes = []domain.Capability{domain.CapabilityJSONPath} // filterFor is the query one expression answers, alone. func filterFor(expression string) domain.SearchQuery { return domain.SearchQuery{ - Filters: []domain.AttributeFilter{{Expression: expression}}, - Limit: searchConfig().MaxPageSize, + Filters: []domain.AttributeFilter{{Expression: expression}}, + Limit: searchConfig().MaxPageSize, + NetworkIDs: []string{domain.NetworkWildcard}, } } From 32dede2bf3db73eca6d2a28f12af3efb408d86a9 Mon Sep 17 00:00:00 2001 From: Manjunath Davanam Date: Wed, 28 Oct 2026 19:40:46 +0530 Subject: [PATCH 23/23] docs: say what the scope gate and the networkId echo actually do [#58] MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Three passages still described the shape A29 replaced, and A29's own row described a marshal the code does not perform. The row claimed NetworkIDs collapses a one-element array to a string, so a caller who sent ["n1"] is echoed "n1", and argued for it on the grounds that the alternative means carrying arrival-shape state. The code carries exactly that state — one bool — and echoes back the shape that arrived. It is the better behaviour: the response context is the request's context read back, and a field that changes spelling in it reads as something having happened to it. The oneOf permits either, so this was never a conformance question, only a surprise question. The row now says what ships and why. The resources DDL commentary still carried the three-arm `sqlc.narg(...) IS NULL OR ... @> ARRAY['*'] OR ... @> ARRAY[$1]` with two paragraphs arguing for it, while A29 further down the same document replaced all of it with one `&&`. Two SQL shapes in one plan means whoever implements against it picks one. The block is now the shipped predicate, and the prose says why overlap is the whole gate: the scope is never empty, so there is no absent-predicate case to write. force_custom_plan survives, with network_id removed from the list of clauses that need it — target_paths, spatial_op, the four box bounds and schema_contexts are still nullable, so the setting is unchanged. Task 16's test pins say what the test now reads, and the omitted-networkId scenario is restated as A29's three rules against three catalogs, which is what the shipped test asserts. --- docs/design/implementation-plan.md | 90 ++++++++++++++---------------- 1 file changed, 43 insertions(+), 47 deletions(-) diff --git a/docs/design/implementation-plan.md b/docs/design/implementation-plan.md index 1a49d60..e466c00 100644 --- a/docs/design/implementation-plan.md +++ b/docs/design/implementation-plan.md @@ -158,7 +158,7 @@ that same array rather than inventing a second field to say it. | **A27** | **A deployment may decline free-text retrieval, and declining REFUSES where a missing backend mode degrades.** `Search.EnableTextSearch` / `SEARCH_ENABLE_TEXT_SEARCH`, **true by default** — the value that takes no decision has to be the one that behaves as before the flag existed. With it false, an intent carrying a non-empty `textSearch` is refused in `MapIntent` with `SCH_TYPE_NOT_SUPPORTED` at `$['message']['intent']['textSearch']`, and the message names `spatial` and `filters` as what still works. **The whole content of this amendment is that it is not `FailOnUnavailableMode` under another name**, because the two read alike and behave oppositely. That flag is about a mode the **backend** cannot run — `semantic` on a `noop` deployment — and false there degrades, answering with what the other modes found and naming the gap in `X-Beckn-Degraded`; the query the caller wrote survives without it. This one is the **deployment's** policy, and degrading would be the failure C11 exists to prevent: on a text-only intent the term IS the query, so running the remaining modes without it answers **the whole corpus under a 200** — a page indistinguishable at the caller from a correct one. That is the same widening `MapIntent`'s doc comment already refuses for an unreadable `targets` or an ignored SRID, which is why the guard sits there and not in `negotiate`. **Three details are load-bearing.** The guard reads the **trimmed** term, like every other reader of it: `" "` asks for no retrieval mode at all, so it stays the no-criterion refusal and calling it a disabled text search would name the wrong mistake. The intent's required-one-of **list** shrinks with the flag — `spatial or filters`, with a parenthetical saying why the third is absent — because offering `textSearch` to a caller this deployment would then refuse walks them into a second `400`; the **guard** does not shrink with it, since a term that was sent is a criterion that was sent and reporting it missing would be a false sentence stacked on the true one. And the wire message names the two criteria that still work rather than the environment variable, matching `negotiate` and the `filters.type` refusal: a caller cannot act on the variable, and the operator has `docs/publish-and-discover.md`. **`modesFor` is left alone.** It still maps a non-empty `Text` to all three ranked modes, because the refusal is total and unreachable past it; a second guard there would be a branch no test can reach and the kind of defence that rots into a contradiction | 2, 17, 19 | | **A28** | **An omitted `visibleTo` means PUBLIC.** The default was quiet in the direction that hurts: a publisher who never thought about the field got a catalog only their own network could find, and an `ACCEPTED` telling them nothing. It becomes the wildcard `['*']`, which is what the spec's own sentence says, so C8 stops being a deviation. Resolved exactly where `[network]` was — in `applyDirectiveDefaults` and in the mapper's `visibleTo`, one literal each — so **A9 is untouched**: the default still applies in both update modes, and a republish that omits `visibleTo` still resolves it rather than keeping what is stored. That last point was decided twice. The first attempt made an omitted `visibleTo` mean *keep* under MERGE, on the argument that widening an audience a publisher narrowed is a change nobody is told of; review took it out, because it is a change to what MERGE means rather than a change to a default, and it belongs in its own issue with its own argument. The hazard it named is real and is recorded in Deferred. The wildcard is a row VALUE and not a second column, because every reader is already a set test over `visible_to`. `EnsureVisibleTo` fills `['*']`, and both `visible_to` columns keep `NOT NULL DEFAULT '{}'`. `APP_NETWORK_ID` loses this reader and keeps the telemetry Resource (A25). **Rows already stored are NOT migrated**: rewriting them would publish catalogs onto networks their publishers never named, which is the worse of the two wrongs a second time. `isActive` is **untouched** and still resets in both modes (A9, scenario 26). One cost, recorded rather than discovered: the spec says a `visibleTo` value *"MUST be a registry-registered network participant identifier"*, and `*` is not one. The schema declares no `pattern`, `enum` or `minItems`, so L1 accepts it — a second, smaller deviation living inside the one that closed C8 | 11, 15, 17, 18, 21, 27 | -| **A29** | **`context.networkId` becomes one-or-many, and a discover answers exactly the networks it names.** `release-0.2.0` of the protocol (pinned at `7a9fdf0d`) redeclares the field as `oneOf: [string, array minItems 1]`, which is the only schema change in a 39-line diff. Three rules replace the old pair, and the middle one reverses the read half A28 shipped days earlier: an **omitted** `networkId` matches `['*']` and nothing else; a **named** network matches that network and nothing else; an **array** matches the union of its entries, `*` included if the caller lists it. So `["n1","*"]` is how a caller asks for its own network plus everything public, and that pairing — not a default — is what a farmer-facing app sends. The reversal is deliberate: A28 bolted public onto every scoped search because a scoped caller had no other way to reach a public catalog, and the array gives them one. **An omitted `networkId` therefore narrows**, from every catalog to the public ones, and that is the one behaviour here an existing caller loses without asking. It is the price of `networkId` meaning the same thing in all three forms rather than "a filter, except when absent, when it is not one". The predicate collapses with it: five queries lose the `sqlc.narg(...) IS NULL OR` dance and the separate `ARRAY['*']` arm for a single `r.visible_to && @network_ids::text[]`, because the scope is now always a non-empty set — `{'*'}` when the caller said nothing. `&&` is GIN-indexable, so one bitmap scan answers what two OR'd `@>` arms did. `beckn.Context.NetworkID` becomes `beckn.NetworkIDs`, a named `[]string` that unmarshals from either shape; it marshals a single id back as a **string**, so a caller who sent `["n1"]` is echoed `"n1"` — identical under the `oneOf`, and the alternative is carrying arrival-shape state through a struct the response re-marshals. The spec calls the array an ordered registry-resolution list with fallback, and we read it as an unordered filter set; nothing here resolves a registry, so the two do not collide, but it is a second meaning on one field and it is recorded rather than discovered | 16, 20, 21, 28 | +| **A29** | **`context.networkId` becomes one-or-many, and a discover answers exactly the networks it names.** `release-0.2.0` of the protocol (pinned at `7a9fdf0d`) redeclares the field as `oneOf: [string, array minItems 1]`, which is the only schema change in a 39-line diff. Three rules replace the old pair, and the middle one reverses the read half A28 shipped days earlier: an **omitted** `networkId` matches `['*']` and nothing else; a **named** network matches that network and nothing else; an **array** matches the union of its entries, `*` included if the caller lists it. So `["n1","*"]` is how a caller asks for its own network plus everything public, and that pairing — not a default — is what a farmer-facing app sends. The reversal is deliberate: A28 bolted public onto every scoped search because a scoped caller had no other way to reach a public catalog, and the array gives them one. **An omitted `networkId` therefore narrows**, from every catalog to the public ones, and that is the one behaviour here an existing caller loses without asking. It is the price of `networkId` meaning the same thing in all three forms rather than "a filter, except when absent, when it is not one". The predicate collapses with it: five queries lose the `sqlc.narg(...) IS NULL OR` dance and the separate `ARRAY['*']` arm for a single `r.visible_to && @network_ids::text[]`, because the scope is now always a non-empty set — `{'*'}` when the caller said nothing. `&&` is GIN-indexable, so one bitmap scan answers what two OR'd `@>` arms did. `beckn.Context.NetworkID` becomes `beckn.NetworkIDs`, which unmarshals from either shape and **echoes back the one that arrived**: a caller who sent `"n1"` is echoed `"n1"` and one who sent `["n1"]` is echoed `["n1"]`. The `oneOf` permits either spelling for either request, so collapsing a one-element array to a string would be conformant — and still a surprise, because the response context is the request's context read back and a field that changes shape in it reads as something having happened to it. The arrival shape is one bool on the struct; that is the cost, and it is smaller than the question it removes. The spec calls the array an ordered registry-resolution list with fallback, and we read it as an unordered filter set; nothing here resolves a registry, so the two do not collide, but it is a second meaning on one field and it is recorded rather than discovered | 16, 20, 21, 28 | A6 and A7 exist for one requirement: *swap the text backend later, keep geo on PG, and let publish write to two stores.* Both build **seams plus conformance @@ -1274,9 +1274,7 @@ retriever rank out-of-scope rows into the page and make `Total` disagree with what comes back: ```sql -(sqlc.narg('network_id')::text IS NULL - OR r.visible_to @> ARRAY['*'] - OR r.visible_to @> ARRAY[sqlc.narg('network_id')::text]) +r.visible_to && @network_ids::text[] AND r.active AND (r.valid_from IS NULL OR r.valid_from <= now()) AND (r.valid_to IS NULL OR r.valid_to >= now()) @@ -1284,38 +1282,38 @@ what comes back: (now() AT TIME ZONE 'UTC')::time) ``` -**`network_id` is `sqlc.narg`, not `@`, and that is the whole mechanism.** A -`NULL` short-circuits the `OR` to true for every row, which is what makes an -omitted `networkId` search every network rather than none — the identical -pattern `target_paths` and `schema_contexts` already use above, applied to the -one clause in this block that used to be unconditional. A caller who supplies -`networkId` still gets exactly that network's rows; nothing about the scoped -case changes. - -**The `'*'` arm is A28's half, and it is a third arm rather than a widening of -the second.** `r.visible_to @> ARRAY[sqlc.narg('network_id'), '*']` would mean -*both*, which no row holds. Each arm is its own sargable `@>` against -`idx_resources_visible_to`, so the planner answers the pair with a BitmapOr over -one index rather than losing it — the same property the `NULL` arm is written -to preserve, which is why the wildcard is a value in this column and not a -separate one. - -**`plan_cache_mode = force_custom_plan`, and it is this pattern that requires -it.** `$1 IS NULL OR r.visible_to @> ARRAY[$1]` is only index-usable when +**The scope is a set, and the gate is one overlap test (A29).** `@network_ids` +is never empty — a request naming no network arrives here as `{'*'}`, filled by +`discover.Service`, so there is no absent-predicate case to write. A caller gets +exactly the networks it named: `{'n1'}` matches rows visible to `n1` and not the +public ones, `{'n1','*'}` matches both, `{'*'}` matches the public ones alone. + +**`&&` is why the wildcard is a row VALUE and not a second column.** Overlap is +sargable against `idx_resources_visible_to`'s `array_ops` opclass, so the whole +gate is one bitmap scan however many networks the caller lists. This replaced a +three-arm `sqlc.narg(...) IS NULL OR ... @> ARRAY['*'] OR ... @> ARRAY[$1]` +whose arms the planner answered with a BitmapOr; the set form is the same index +and one arm. **The predicate is still on every query that reads** — every +retriever, the counter and the hydrator (A6) — for the reason above. + +**`plan_cache_mode = force_custom_plan`, and it is the nullable pattern that +requires it.** `$1 IS NULL OR col @> ARRAY[$1]` is only index-usable when PostgreSQL knows `$1`: with the value in hand it folds the first arm to FALSE and is left with a sargable `@>`. With the value unknown it must keep both arms, and an `OR` whose first arm does not mention the column cannot be answered by an index on that column — so the GIN index is dropped and the clause becomes a -sequential-scan filter. pgx speaks the extended protocol, and PostgreSQL builds +sequential-scan filter. A29 took `network_id` out of this set by making the +scope always non-empty; `target_paths`, `spatial_op`, the four box bounds and +`schema_contexts` are still in it, so the setting is unchanged. pgx speaks the extended protocol, and PostgreSQL builds a **custom** plan for the first five executions of a prepared statement and may switch to a **generic** one thereafter. So the fast plan is the one a cold connection gets, and the slow plan is the one a warm connection settles into — the exact opposite of how a performance problem is usually shaped, and invisible to any `EXPLAIN` run once. -Every nullable clause in this plan has the property: `network_id` here, -`target_paths`, `spatial_op` and the four box bounds in the spatial `EXISTS`, -and `schema_contexts` above. That is the whole read path, so the setting is not +Every nullable clause in this plan has the property: `target_paths`, +`spatial_op` and the four box bounds in the spatial `EXISTS`, and +`schema_contexts` above. That is the whole read path, so the setting is not a tuning knob but the condition under which the query shapes in this document mean what they say. It is applied in `pool.go`'s on-acquire block beside `hnsw.ef_search` and `statement_timeout`, and Task 14's `EXPLAIN` assertions run @@ -1325,13 +1323,12 @@ the custom plan is a test that cannot observe this at all. **Every column here is on `resources`.** No query in the read path joins `catalogs` — the provider document is fetched once per catalog on the page, at hydration, after the page is already decided. Task 16 has a test that reads the -SQL file and proves no query omits the gate — the nullable `network_id` clause -included, so a query that hard-codes `visible_to @> ARRAY[$1]` instead of the -nullable form is caught as surely as one that drops the clause outright. Since -A28 that test also fails a query carrying the nullable form but **not** the -`'*'` arm: such a query hides every public catalog from every scoped caller, -which is an empty page rather than an error and so is invisible to the -acceptance suite unless something reads the SQL. +SQL file and proves no query omits the gate — the `r.visible_to && +@network_ids::text[]` clause included, so a query that hard-codes +`visible_to @> ARRAY[$1]` instead of the overlap is caught as surely as one +that drops the clause outright. That substitution hides every public catalog +from every scoped caller, which is an empty page rather than an error and so is +invisible to the acceptance suite unless something reads the SQL. ### Validity — two independent windows @@ -4549,22 +4546,21 @@ one `Retriever` per mode, `Hydrator`, `RRF` **Tests pin:** - A **SQL source test** that reads `discover.sql` and asserts every query - contains the scope gate — the nullable `network_id` clause, `active`, both - date bounds **and** `within_daily_window` — plus the `geo_distance_m` + contains the scope gate — `r.visible_to && @network_ids::text[]`, `active`, + both date bounds **and** `within_daily_window` — plus the `geo_distance_m` predicate. A mode that drops the clause entirely leaks another network's catalog to a caller who scoped their request; a mode that hard-codes - `visible_to @> ARRAY[$1]` instead of the nullable form silently reintroduces - single-network scoping as the default. **The `ARRAY['*']` arm is asserted by - the same test (A28):** a query carrying the nullable form without it hides - every public catalog from every scoped caller, and an empty page is not an - error anything else here would notice. -- **An omitted `networkId` returns catalogs from every network,** including - one whose `visibleTo` names a single network the caller never mentioned — - and the same intent with `networkId` set still returns only that network's - rows **plus every `['*']` catalog** (A28). Pinned as one test with three - assertions, because a fix aimed at one direction is a regression in another: - narrow the wildcard arm and public catalogs vanish from scoped searches, - widen the scoped arm and `visibleTo` stops restricting anything. + `visible_to @> ARRAY[$1]` instead of the overlap silently reintroduces + single-network scoping and hides every public catalog from a scoped caller, + and an empty page is not an error anything else here would notice. +- **A search matches exactly the networks it named** (A29). Pinned as one test + over three catalogs — one on `mahavistar`, one on `private.example.com` and + one on `['*']` — with three assertions, because a fix aimed at one rule is a + regression in another: an omitted `networkId` returns the public catalog + alone, `['mahavistar']` returns that network's alone, and + `['mahavistar','*']` returns both. Widen the omitted case and `visibleTo` + stops restricting anything; hand public catalogs to a scoped caller and + `['n1','*']` stops being the way to ask for them. - **Schema filtering** matches as a pair: two resources, one `schema.org` + `GroceryItem` and one `beckn.org/Mobility` + `RideService`, queried with both fragments at once, return both — and a query for