Repository navigation
Release-0.2.0 - #102
Merged
Merged
Release-0.2.0#102
Conversation
unzip() joined an attacker-controlled zip entry name straight into the destination path with no bounds check, so a "../" entry could write outside dest (CodeQL go/zipslip).
CVE-2026-81871 (otlploggrpc v0.19.0, MEDIUM) and CVE-2026-81870 (otlptrace, otlptracegrpc, otel/sdk, LOW) were failing trivy-gate on every PR. Bumps the whole otel release train together -- v1.45.0 for the stable line, v0.21.0 for the log line -- rather than only the four flagged modules, because the two lines are released in lockstep. otel/log v0.21.0 drops its own Value/KeyValue types in favour of the shared go.opentelemetry.io/otel/attribute ones (upstream open-telemetry/opentelemetry-go#8490), so the audit-log call sites move to attribute.String/StringValue/KeyValue. Same keys, same values, same OTLP output -- no behaviour change. A narrower bump is not possible: otlploggrpc v0.21.0 requires otel/log v0.21.0, so module resolution pulls the new API in regardless. stdHandler.go also picks up a gofmt pass: it was already unformatted on development, and the pre-commit hook checks any file the commit touches. Whitespace only, no logic in those lines.
cover-diff grades the whole of every file a PR touches, not just the changed lines, so the security fixes in manager.go and stdHandler.go were being blocked by coverage debt that predates them: 76% and 74% against an 80% floor. Covers what was genuinely untested rather than padding the number. manager.go's CatalogPublisher, CatalogBlobStore, Crawler, Mapper, ProviderStep and SchemaVersionMediator were all at 0% despite being the same provider-lookup shape as Publisher and Cache next to them; each now has the success, provider-error and plugin-missing cases the neighbouring accessors already had. stdHandler.go's LoadKeyManager, loadManifestLoader, loadPayloadStore, LoadPolicyChecker and loadPayloadTransformerStep get their argument-validation branches covered, including the registry that satisfies RegistryLookup but not RegistryMetadataLookup. manager.go 76% -> 90%, stdHandler.go 74% -> 81%, changed-file total 75% -> 86%. Test files only; no production code touched.
…erts fix: close the open code-scanning and dependency findings [#41]
The tag pipeline ran `make trivy-release-gate` after `make image-build`, scanning the digest just pushed with `--exit-code 1` at `CRITICAL,HIGH,MEDIUM,LOW`. Any finding at any severity failed the step, which meant the digest was never uploaded as an artifact, `Publish Artifact` was skipped, and `image-publish` never bound a tag. The layers sat in the registry with nothing pointing at them — no `:<version>`, no `:latest`, nothing anyone could pull. `fail-fast: true` took the sibling arch down with it. A scan is the wrong thing to hold a release on. By the time a tag exists the code is merged, so a finding offers "a tag or no tag", never "a safer image". The image is already scanned three times before a tag can reach it, and none of those three is this one: - ci.yml scans it on the PR, where trivy-gate blocks the merge on any finding. - security.yml scans it again on the push to main / release-*. - security.yml again, weekly — the only one of the three that catches a CVE published after the code stopped changing. Removed: the `trivy-release-gate` target and its `.PHONY` entry, the `trivy-release-gate` step in the release job, and the `trivy-cache` step that existed only to feed it. Nothing in the tag pipeline runs Trivy now. `image-build`'s doc comment records why there is no scan so the next person does not re-add one. `trivy-deps`, `trivy-image`, `trivy-report` and `trivy-gate` are untouched, and the PR gate stays blocking at every severity.
…elease fix: stop scanning the release image at tag time
…data - Implemented main functionality to authenticate and collect market master data across India. - Added configuration handling for base URL, states, date range, and output file. - Introduced data collection logic with error handling for state-specific failures. - Created mappings for market-commodity and master data to facilitate data transformation. - Developed comprehensive tests to ensure functionality and error handling. - Embedded mappings into the binary to avoid external dependencies.
…ctionality - Introduced a new YAML mapping file for generating Beckn catalog payloads from Agmarknet market data. - Implemented HTTP client for publishing catalogs to the provider adapter. - Added tests for handling scenarios with no data from upstream and ensuring proper error reporting. - Created a run function to collect data and build catalogs without generating a collection document. - Enhanced publish functionality to handle transport errors and catalog retirement.
- Updated `build_test.go` to improve market resource validation and ensure accurate reporting of geometry-related issues. - Introduced `chunk_test.go` to validate chunking behavior for states exceeding geometry limits and ensure proper handling of markets without coordinates. - Enhanced `main.go` to improve catalog reporting, including detailed summaries of excluded markets and geometry-less markets. - Added `publish.go` changes to correctly associate catalog files with their respective states, accommodating split states. - Created `report_test.go` to validate the accuracy of reports generated for excluded markets and chunked states. - Updated `catalog.yaml` to streamline market object creation and ensure correct catalog IDs. - Removed redundant tests in `main_test.go` to maintain clarity and focus on essential functionality.
Move serveMappings() and newMapper() from mandi_publish/mappings.go to internal/catalogpublish/mapper.go as public functions that accept an embed.FS and directory name. Update all call sites in mandi_publish to use the new public API. This generalizes the mapper setup for reuse by other tools.
Const block was space-indented instead of tab-indented, with a stray blank line before the following const block. No functional change.
…nd improve build process
…coverage area to catalog mapping Bring the catalog/publish payload closer to the OpenAgriNet target schema: resourceAttributes.source, resource longDesc, provider shortDesc, and an ISO-3166-2 state coverageArea alongside the existing district one. Left market.location.geo out deliberately — coverageAreas is already the sole geometry copy, per the existing per-catalog geometry budget.
…ing, and main execution flow
- Deleted mappings for market-commodity, master-markets, and master-states as they are no longer needed. - Removed associated test files that validated the mappings and their functionality. - Cleaned up the publish wiring and report generation code by removing unused functions and tests. - Ensured that the run logic no longer references the removed collection document, streamlining the catalog generation process.
- Introduced master-markets.yaml and master-states.yaml for fetching market and state data. - Implemented mappings_test.go to validate the presence of mapping files. - Created pipeline_files.go to embed pipeline definitions and mappings. - Developed spec.go for parsing the mandi-price-agmarket.yaml pipeline specification. - Added spec_test.go to ensure correct parsing of the pipeline specification. - Implemented steps.go for handling market data collection and catalog-building logic. - Added steps_test.go to validate the functionality of market data processing steps.
- Add inputs.go to handle the resolution of input values from environment variables and defaults. - Implement input validation with enum checks to ensure correct values are used. - Create inputs_test.go to cover various test cases for input resolution, including environment variable precedence and default rendering. - Introduce publish.go to manage the publishing process, including validation of publishing rules and handling of publish URLs. - Add publish_test.go to test the publishing logic, ensuring correct behavior for accepted and rejected catalog statuses, as well as handling of partial collections.
- Implemented unit tests for the catalog publishing functionality in `publish_test.go`, covering various scenarios including successful publishing, state filtering, dry runs, and error handling. - Created a new `spec.go` file to define the structure and parsing logic for the pipeline specification from `mandi-price-agmarket.yaml`. - Added tests in `spec_test.go` to ensure the correct mapping of YAML fields to Go structs, validating the integrity of the pipeline specification. - Introduced a sample greeting text file in the test data directory for use in mappings.
- Introduced `run_execute_test.go` to validate the execution flow after scheduling, ensuring that the collector's results are correctly processed and written to the output directory. - Added `run_test.go` to cover pre-execution decisions, including capability validation and run logging. - Created `schedule.go` and `schedule_test.go` to manage and test the scheduling logic, ensuring capabilities are allowed to publish and that runs are executed at the correct times. - Implemented unmapped key detection in `unmapped.go` to identify any discrepancies between pipeline files and their expected schema. - Added test data files for minimal and other pipeline configurations to support testing of pipeline loading and execution.
- Add schedule.go to handle tick decisions for publishing capabilities. - Implement functions to determine if a capability is allowed to publish and if it is due for execution based on a cron schedule. - Introduce TickDecision struct to encapsulate the outcome of a tick decision. test: add unit tests for scheduling logic - Create schedule_test.go to validate the behavior of scheduling functions. - Test cases include scenarios for valid and invalid provider records, cron schedule evaluations, and tick decision outcomes. feat: define pipeline specification structure - Add spec.go to define the structure of the pipeline specification based on the YAML schema. - Implement LoadSpec function to read and parse pipeline definitions from embedded files. test: add tests for pipeline specification loading - Create spec_test.go to ensure LoadSpec correctly reads pipeline files and identifies unmapped keys. - Include tests for handling known and unknown keys in the pipeline specification. feat: implement unmapped keys detection - Add unmapped.go to identify keys in the pipeline file that are not represented in the Spec structure. - Implement UnmappedKeys function to return paths of unmapped keys for validation. test: add tests for unmapped keys detection - Create tests for unmapped keys functionality to ensure proper detection of schema mismatches. - Include test cases for complete and incomplete pipeline files. chore: add test data for pipeline specifications - Add testdata directory with minimal and other pipeline YAML files for testing purposes. - Include examples of valid and invalid pipeline configurations to support unit tests.
…e functionalities - Added `steps.go` to define the step runner for executing pipeline steps. - Implemented core primitives: HTTP GET, join, derive, and dedupe. - Introduced error handling for empty outputs and transport errors. - Created tests in `steps_test.go` to validate step execution order, conditional execution, and error handling. - Added fixture mappings for testing in `testdata/mappings/catalog.yaml` and `testdata/mappings/things.yaml`.
Discovery pages by resource and answers with the page it applied, so a page holding fewer catalogs than the limit is not the end. A network that reports no page is still walked on a short page, and one that never advances is stopped rather than asked forever.
release-0.2.0 redeclares context.networkId as one or many, and a discover answers exactly the networks the array names. Ours alone would miss every public catalog.
Indirect, through hashicorp/vault/api. v4.1.4 is a HIGH the trivy gate refuses; 4.1.6 is the fixed version.
feat: crawl another network [#76]
A select naming another network's provider arrives at our provider module like any other. The capability step now asks who it is for before it asks whether it has a binding for it -- a binding key names the upstream, so two networks fronting the same upstream hold identical keys and the binding check cannot tell them apart. Outbound: context.receiverId is compared against the module's own configured id, never the one reqpreprocessor resolved from the body -- on a request addressed to another network that one is THEM. A receiver that is not us must be a network we admitted as a source, or the request is refused. It goes to context.bppUri plus the action, signed as us. Inbound: the registry refuses to hand back a signing key for a network we did not admit as a requester, so the request dies at validateSign rather than reaching any capability. The check lives there and not in a step because a step can be left out of a config, and a signature check cannot proceed without the key. Off unless crossNetworkCheck is set, so a deployment that federates with nobody behaves exactly as before. A participant with no federationRole is an ordinary caller -- a farmer's app -- and is served as it always was.
federationRole becomes a direction-keyed object, so an agreement says which way it runs and which packs it covers. A discover is refused if it names a pack we did not agree to give. Every other action is governed by the direction alone, because only a discover carries context.schemaContext.
] A pack URL names the attributes document and the context document beside it is what a discover asks for, so the suffix is swapped. An agreement recorded with the context document already was having a second one appended, and one with no filename at all lost its separator. Both produced a URL matching nothing, on the asking side and on the comparison side, with no error anywhere.
The agreement check lived inside the registry's key lookup, so it only ran when validateSign did. A mount without signing enforced nothing, while crossNetworkCheck stayed true in the config and said so nowhere. checkCrossNetworkRequester is now a step. It reads the caller from the body and asks our own admission records, so it holds with or without a signature. Listed after validateSign it judges a sender already proved, which is also the order the old placement got backwards: the key lookup ran before the signature was verified. A refusal is a 400 naming its reason, not the 500 an unclassified error became. The registry being unreachable stays a 500, which is ours. crossNetworkCheck is gone. Listing the step is the switch.
…ll them [#78] Moving the check into a step routed it through AdmittedNetwork, which answers a different question: can we CALL this network. That one also wants a network role and a discover address, and the registry search filters on the role before a record is even returned. So a record carrying federationRole but missing either came back as no network at all, and the step then allowed it as an ordinary participant. A network admitted as a source only -- one we read from and never agreed to answer -- was refused before and let in after. A half-filled admission opened the door instead of shutting it. agreementWith reads federationRole from the record and nothing else, which is where whether we agreed to answer someone is actually written. Usability is the crawler's concern and has no bearing on it. In practice admit always writes both fields, so this needed a hand-edited record to reach. It was still a loosening, and not one anybody asked for.
Multi-paragraph headers where one line does. The reasoning belongs in the behaviour doc, not above a function.
Subscription.Agreement was filled by the key lookup for a check that has since moved into a step. Write-only since 31abc1b, with a comment describing a reader that no longer exists. AdmittedNetwork, the singular lookup, has had no caller outside its own tests since the crawler commit added it. operationAdmittedNetwork goes with it. The blank-id guard it was tested for still holds, so that test now covers agreementWith instead of being deleted.
A provider request now carries senderId and receiverId: a module that cannot see who a request is addressed to cannot tell whether it is ours to answer, and guessing "ours" is how one network answers another's request with its own data. The Grievance fixtures predate that and named neither, so every one of them was refused -- 108 tests in one package. They now name both ends, and their StepContext carries the subscriberId to compare against. Nothing about the plugin changes; this is its fixtures meeting the contract. The network names in every federation test were two real deployments. This is a digital public good: net-a.oan.local and net-b.oan.local say the same thing and belong to nobody.
…78] capabilities already names the registry's capabilityCode, which holds openagrinet:MandiPrice, while these are context document URLs.
feat: cross-network calls [#78]
release: bring development into release-0.2.0
🛡️ Trivy security scan (CRITICAL,HIGH,MEDIUM,LOW)Go dependenciesNo findings at CRITICAL,HIGH,MEDIUM,LOW. Container imageNo findings at CRITICAL,HIGH,MEDIUM,LOW. |
|
📊 Test Coverage: ✅ Passed — 87% of changed lines covered, min 80% |
manjudr
approved these changes
Nov 2, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Release 0.2.0 of the network adapter: cross-network calls, crawling another network, and the new plugins.