Skip to content

Commit 9bc091c

Browse files
committed
PM-4777: Report declared and transitive dependencies from embedded poms
Adds three opt-in flags to `scan binary` and `report sbom binary`: - `--include-declared` reads every embedded pom.xml the way Maven does (properties, ${project.*}, the parent chain, dependencyManagement with imported BOMs, dependencies inherited from parents) and reports the compile- and runtime-scope dependencies the artifact does not ship as components marked `declared`, with no hashes and the declaring pom as their path. Test, provided, system and optional are left out because Maven never hands them to a consumer. - `--include-test-scope` lifts that filter so totals line up with SCA tools that count every declared dependency. Scope stays on each component. - `--include-transitive` follows each declared dependency through its own pom on Maven Central: nearest declaration wins, exclusions apply down their branch, optional stops, scope combines per Maven's table, and the root pom's management overrides transitive versions. Each transitive records what pulled it in (`cycode:via`) and the graph links them. Bounded to 10 hops and 250 poms per embedded pom. Parent and dependency poms come from repo1.maven.org behind a PomSource seam, so `--maven-central` is required for parent-managed versions and for transitives. Coordinates read from an untrusted pom are validated before they can shape a URL, and bodies are capped at 1 MB. A version that cannot be established is listed under "Declared, version unresolved" with the reason, never guessed. Motivation: a customer's Checkmarx report on 16 vendor jars listed 29 vulnerabilities, every one on a declared-but-unshipped dependency. With all three flags Cycode reports all 61 of those declared packages and 30 findings; the difference is vulnerability-database content only. Claude-Session: https://claude.ai/code/session_016SxZLL1buHZ8jZ9T8onEZ8
1 parent 04ca4ae commit 9bc091c

22 files changed

Lines changed: 2651 additions & 70 deletions

‎README.md‎

Lines changed: 59 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -50,7 +50,8 @@ This guide walks you through both installation and usage.
5050
2. [Binary Scan Options](#binary-scan-options)
5151
3. [How Components Are Identified](#how-components-are-identified)
5252
4. [Unidentified Components](#unidentified-components)
53-
5. [Limitations](#binary-scan-limitations)
53+
5. [Declared Dependencies](#declared-dependencies)
54+
6. [Limitations](#binary-scan-limitations)
5455
5. [Commit History Scan](#commit-history-scan)
5556
1. [Commit Range Option (Diff Scanning)](#commit-range-option-diff-scanning)
5657
6. [Pre-Commit Scan](#pre-commit-scan)
@@ -1051,7 +1052,10 @@ Nested archives are opened recursively, so a JAR inside a WAR inside an EAR is s
10511052
|---|---|---|
10521053
| `--max-depth` | `3` | Nested-archive recursion limit. An EAR containing WARs containing JARs is depth 3. |
10531054
| `--offline` | off | Identify components from embedded metadata only. Acknowledges and silences the partial-results warning. |
1054-
| `--maven-central` | off | Look up archives that embedded metadata cannot identify on Maven Central by SHA-1. Sends only the digest, never the archive. Cannot be combined with `--offline`. |
1055+
| `--maven-central` | off | Consult Maven Central: look up archives that embedded metadata cannot identify by SHA-1, and with `--include-declared` fetch parent poms to pin declared versions. Sends digests and public coordinates, never the archive. Cannot be combined with `--offline`. |
1056+
| `--include-declared` | off | Also report the compile- and runtime-scope dependencies that embedded `pom.xml` files declare but the artifact does not ship. See [Declared Dependencies](#declared-dependencies). |
1057+
| `--include-test-scope` | off | With `--include-declared`: also report test-, provided- and system-scope declarations, which never reach a consuming build. Matches SCA tools that count every declared dependency. |
1058+
| `--include-transitive` | off | With `--include-declared` and `--maven-central`: follow each declared dependency through its own pom to the dependencies it needs in turn, the way a Maven build would. |
10551059
| `--project-name` | inferred | Override the platform identity when the artifact is detached from its source repository. |
10561060
| `--keep-bom` | off | Write the generated component inventory beside each artifact, for inspection or audit. |
10571061
| `--include-binaries` | off | On `scan path` only. Extract any Java archives encountered during the walk. |
@@ -1095,6 +1099,52 @@ We do not guess a component's identity from its filename.
10951099
`internal-shim.jar` is not evidence of anything, and an admitted gap is more useful than an invented coordinate.
10961100
Unidentified components do not set the exit code on their own.
10971101
1102+
#### Declared Dependencies
1103+
1104+
By default a binary scan reports what is physically inside the artifact.
1105+
That is the right answer for a deployable, but it is not the only question worth asking of a library JAR.
1106+
A build that consumes that JAR inherits its compile- and runtime-scope dependencies, whether or not the vendor shipped them, and a vulnerability in one of those is real in the consuming build.
1107+
1108+
`--include-declared` answers that second question from the same embedded metadata:
1109+
1110+
`cycode scan -t sca --include-declared --maven-central binary ./vendor/woden-core.jar`
1111+
1112+
Every embedded `pom.xml` is read the way Maven reads it.
1113+
Each dependency it declares at `compile` or `runtime` scope, and which is not already present in the artifact, is added to the component inventory as a **declared** component and scanned like any other.
1114+
Dependencies Maven would never hand to a consumer are left out: `test`, `provided` and `system` scope, and anything marked `optional`.
1115+
1116+
Some SCA tools count every declared dependency regardless of scope, so their totals include a vendor's own test libraries.
1117+
To compare against such a report, add `--include-test-scope` and those declarations are reported too.
1118+
Each declared component still records its scope, so a `test` finding can be told apart from one that reaches a build.
1119+
1120+
Versions in a real pom are rarely literal.
1121+
`${property}` references and `${project.version}` are resolved from the pom itself.
1122+
A version managed by a parent pom, or by an imported BOM, lives on Maven Central rather than inside the JAR, so resolving it needs `--maven-central`.
1123+
Without it, or when the parent cannot be fetched, the dependency is listed in its own section with the reason instead of being guessed at:
1124+
1125+
```
1126+
╭─ 📜 Declared, version unresolved (2) ──────────────────────────────────────────────╮
1127+
│ Dependency Version Declared by Why │
1128+
│ commons-logging:commons-log… - woden-core.jar > META-INF/… no version … │
1129+
╰────────────────────────────────────────────────────────────────────────────────────╯
1130+
1131+
1 identified | 0 unidentified | 2 declared (2 unresolved) | 0 vulnerabilities
1132+
```
1133+
1134+
A finding on a declared component says `Declared by` and the pom that made the claim, rather than `Found in`, and its presence is marked `declared` in the generated BOM.
1135+
Declared components carry no hashes, because there are no bytes to digest.
1136+
`--output json` reports the counts under `binary.declared`.
1137+
1138+
A declared dependency has dependencies of its own.
1139+
`--include-transitive` follows each one through its pom on Maven Central and reports what it needs in turn, so it requires `--maven-central`.
1140+
The walk follows Maven's rules: the nearest declaration of a coordinate wins, exclusions apply down their branch, optional dependencies are not followed, and a transitive's scope is combined with the scope it was reached through, so a test dependency's tree stays test scope.
1141+
Each transitive component records the dependency that pulled it in as `cycode:via`, and the dependency graph in the BOM links them.
1142+
On a real vendor jar this typically fetches a few dozen poms and adds under a minute to the scan.
1143+
1144+
> [!NOTE]
1145+
> Version ranges and versions that only a repository's metadata could pin are listed as unresolved, never guessed.
1146+
> Where the consuming build exists, scanning its output or its `pom.xml` remains the most exact answer: a build tool resolves profiles and repository settings this walk cannot see.
1147+
10981148
#### Binary Scan Limitations
10991149

11001150
Read this before relying on a binary scan as your only check.
@@ -1702,9 +1752,13 @@ It uses the same extraction as [Binary Scan](#binary-scan), so the [limitations]
17021752
17031753
The `binary` subcommand supports the following additional option:
17041754
1705-
| Option | Description |
1706-
|---------------|----------------------------------------------------------------------------------------------|
1707-
| `--max-depth` | Nested-archive recursion limit. Defaults to 3. An EAR containing WARs containing JARs is depth 3. |
1755+
| Option | Description |
1756+
|----------------------|----------------------------------------------------------------------------------------------|
1757+
| `--max-depth` | Nested-archive recursion limit. Defaults to 3. An EAR containing WARs containing JARs is depth 3. |
1758+
| `--maven-central` | Consult Maven Central to identify archives by SHA-1 and, with `--include-declared`, to fetch parent poms. |
1759+
| `--include-declared` | Also include the compile- and runtime-scope dependencies that embedded `pom.xml` files declare but the artifact does not ship, marked `declared` rather than `shipped`. See [Declared Dependencies](#declared-dependencies). |
1760+
| `--include-test-scope` | With `--include-declared`: also include test-, provided- and system-scope declarations. |
1761+
| `--include-transitive` | With `--include-declared` and `--maven-central`: follow each declared dependency to what its own pom needs. |
17081762
17091763
# Import Command
17101764

‎cycode/cli/apps/report/sbom/binary/binary_command.py‎

Lines changed: 43 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -34,8 +34,34 @@ def binary_command(
3434
bool,
3535
typer.Option(
3636
'--maven-central',
37-
help='Look archives that embedded metadata cannot identify up on Maven Central by SHA-1. '
38-
'Sends the digest of each such archive, never the archive itself, to search.maven.org.',
37+
help='Consult Maven Central: look archives that embedded metadata cannot identify up by SHA-1, and '
38+
'with --include-declared fetch parent poms to pin declared versions. Sends digests and public '
39+
'coordinates, never the archive itself.',
40+
),
41+
] = False,
42+
include_declared: Annotated[
43+
bool,
44+
typer.Option(
45+
'--include-declared',
46+
help='Also include the compile- and runtime-scope dependencies that embedded pom.xml files declare '
47+
'but the artifact does not ship, marked as declared rather than shipped. '
48+
'Versions managed by a parent pom need --maven-central to resolve.',
49+
),
50+
] = False,
51+
include_test_scope: Annotated[
52+
bool,
53+
typer.Option(
54+
'--include-test-scope',
55+
help='With --include-declared: also include test-, provided- and system-scope declarations, which '
56+
'Maven never passes to a consuming build. Each component still records its scope.',
57+
),
58+
] = False,
59+
include_transitive: Annotated[
60+
bool,
61+
typer.Option(
62+
'--include-transitive',
63+
help='With --include-declared and --maven-central: follow each declared dependency to the '
64+
'dependencies its own pom declares, the way a Maven build would, and include those too.',
3965
),
4066
] = False,
4167
) -> None:
@@ -52,8 +78,23 @@ def binary_command(
5278
Format conversion happens server-side, so every format the path command supports is supported here too.
5379
5480
"""
81+
if include_test_scope and not include_declared:
82+
raise typer.BadParameter(
83+
'--include-test-scope widens what --include-declared reports; it does nothing on its own.',
84+
param_hint='--include-test-scope',
85+
)
86+
if include_transitive and not (include_declared and maven_central):
87+
raise typer.BadParameter(
88+
'--include-transitive follows declared dependencies through their poms on Maven Central; '
89+
'it needs both --include-declared and --maven-central.',
90+
param_hint='--include-transitive',
91+
)
92+
5593
ctx.obj['binary_max_depth'] = max_depth
5694
ctx.obj['maven_central'] = maven_central
95+
ctx.obj['include_declared'] = include_declared
96+
ctx.obj['include_test_scope'] = include_test_scope
97+
ctx.obj['include_transitive'] = include_transitive
5798

5899
client = get_report_cycode_client(ctx)
59100
report_parameters = ctx.obj['report_parameters']

‎cycode/cli/apps/scan/scan_command.py‎

Lines changed: 46 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -129,8 +129,38 @@ def scan_command(
129129
bool,
130130
typer.Option(
131131
'--maven-central',
132-
help='Look archives that embedded metadata cannot identify up on Maven Central by SHA-1. '
133-
'Sends the digest of each such archive, never the archive itself, to search.maven.org.',
132+
help='Consult Maven Central: look archives that embedded metadata cannot identify up by SHA-1, and '
133+
'with --include-declared fetch parent poms to pin declared versions. Sends digests and public '
134+
'coordinates, never the archive itself.',
135+
rich_help_panel=_BINARY_RICH_HELP_PANEL,
136+
),
137+
] = False,
138+
include_declared: Annotated[
139+
bool,
140+
typer.Option(
141+
'--include-declared',
142+
help='Also report the compile- and runtime-scope dependencies that embedded pom.xml files declare '
143+
'but the artifact does not ship: what a build consuming the artifact would pull in. '
144+
'Versions managed by a parent pom need --maven-central to resolve.',
145+
rich_help_panel=_BINARY_RICH_HELP_PANEL,
146+
),
147+
] = False,
148+
include_test_scope: Annotated[
149+
bool,
150+
typer.Option(
151+
'--include-test-scope',
152+
help='With --include-declared: also report test-, provided- and system-scope declarations, which '
153+
'Maven never passes to a consuming build. Matches SCA tools that count every declared dependency. '
154+
'Each component still records its scope.',
155+
rich_help_panel=_BINARY_RICH_HELP_PANEL,
156+
),
157+
] = False,
158+
include_transitive: Annotated[
159+
bool,
160+
typer.Option(
161+
'--include-transitive',
162+
help='With --include-declared and --maven-central: follow each declared dependency to the '
163+
'dependencies its own pom declares, the way a Maven build would, and report those too.',
134164
rich_help_panel=_BINARY_RICH_HELP_PANEL,
135165
),
136166
] = False,
@@ -211,6 +241,17 @@ def scan_command(
211241
'--offline identifies from embedded metadata only; it cannot be combined with --maven-central.',
212242
param_hint='--maven-central',
213243
)
244+
if include_test_scope and not include_declared:
245+
raise typer.BadParameter(
246+
'--include-test-scope widens what --include-declared reports; it does nothing on its own.',
247+
param_hint='--include-test-scope',
248+
)
249+
if include_transitive and not (include_declared and maven_central):
250+
raise typer.BadParameter(
251+
'--include-transitive follows declared dependencies through their poms on Maven Central; '
252+
'it needs both --include-declared and --maven-central.',
253+
param_hint='--include-transitive',
254+
)
214255

215256
# _single_value_callback validated exactly one value was provided; unwrap from list
216257
scan_type = scan_type[0]
@@ -226,6 +267,9 @@ def scan_command(
226267
ctx.obj['binary_max_depth'] = max_depth
227268
ctx.obj['offline'] = offline
228269
ctx.obj['maven_central'] = maven_central
270+
ctx.obj['include_declared'] = include_declared
271+
ctx.obj['include_test_scope'] = include_test_scope
272+
ctx.obj['include_transitive'] = include_transitive
229273
ctx.obj['project_name'] = project_name
230274
ctx.obj['keep_bom'] = keep_bom
231275
ctx.obj['include_binaries'] = include_binaries

‎cycode/cli/consts.py‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -231,6 +231,12 @@
231231
BINARY_MAX_DEPTH = 3
232232
BINARY_MAVEN_CENTRAL_SEARCH_URL = 'https://search.maven.org/solrsearch/select'
233233
BINARY_DIGEST_LOOKUP_TIMEOUT_IN_SECONDS = 10
234+
BINARY_MAVEN_CENTRAL_REPOSITORY_URL = 'https://repo1.maven.org/maven2'
235+
BINARY_POM_MAX_SIZE_IN_BYTES = 1024 * 1024 # 1 MB; the largest poms on Central are a few hundred KB
236+
BINARY_POM_FETCH_BUDGET = 32 # parent and imported poms consulted for one embedded pom
237+
BINARY_POM_PARENT_MAX_DEPTH = 10
238+
BINARY_TRANSITIVE_MAX_DEPTH = 10 # hops from a declared dependency; Maven trees are rarely deeper than 6
239+
BINARY_TRANSITIVE_FETCH_BUDGET = 250 # poms consulted for one embedded pom's whole tree, parents included
234240
BINARY_MAX_ENTRY_COUNT = 100_000
235241
BINARY_MAX_ENTRY_SIZE_IN_BYTES = 512 * 1024 * 1024 # 512 MB
236242
BINARY_MAX_TOTAL_SIZE_IN_BYTES = 2 * 1024 * 1024 * 1024 # 2 GB

‎cycode/cli/files_collector/binary/base_extractor.py‎

Lines changed: 50 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -15,6 +15,11 @@
1515
EVIDENCE_POM_PROPERTIES = 'pom.properties'
1616
EVIDENCE_DIGEST = 'digest'
1717
EVIDENCE_MANIFEST = 'manifest.mf'
18+
EVIDENCE_POM_XML = 'pom.xml' # declared by an embedded pom, not found as bytes; only with --include-declared
19+
20+
# whether the component's bytes are in the artifact, or an embedded pom merely says a consumer will need it
21+
PRESENCE_SHIPPED = 'shipped'
22+
PRESENCE_DECLARED = 'declared'
1823

1924
CONFIDENCE_EXACT = 'exact'
2025
CONFIDENCE_AMBIGUOUS = 'ambiguous'
@@ -49,12 +54,26 @@ class IdentifiedComponent:
4954
group: str
5055
artifact: str
5156
version: str
52-
sha1: str
53-
logical_path: str
57+
sha1: Optional[str] # None only for a declared component: there are no bytes to digest
58+
logical_path: str # for a declared component, the pom that declares it
5459
parent: Optional[str]
5560
evidence: str # one of the EVIDENCE_* constants
5661
confidence: str # one of the CONFIDENCE_* constants
5762
sha256: Optional[str] = None
63+
declared_scope: Optional[str] = None # the Maven scope a declared component was declared with; None if shipped
64+
declared_via: Optional[str] = None # 'group:artifact' that pulled a transitive in; None for a direct declaration
65+
66+
@property
67+
def is_declared(self) -> bool:
68+
return self.declared_scope is not None
69+
70+
@property
71+
def is_transitive(self) -> bool:
72+
return self.declared_via is not None
73+
74+
@property
75+
def presence(self) -> str:
76+
return PRESENCE_DECLARED if self.is_declared else PRESENCE_SHIPPED
5877

5978
@property
6079
def purl(self) -> str:
@@ -73,6 +92,21 @@ class UnidentifiedArtifact:
7392
size: int
7493

7594

95+
@dataclass(frozen=True)
96+
class UnresolvedDeclaration:
97+
"""A declared dependency whose version could not be established. Listed, never guessed."""
98+
99+
group: str
100+
artifact: str
101+
version_expression: Optional[str]
102+
declared_by: str # logical path of the pom
103+
reason: str
104+
105+
@property
106+
def coordinate_key(self) -> str:
107+
return f'{self.group}:{self.artifact}'
108+
109+
76110
@dataclass
77111
class ExtractionResult:
78112
components: list[IdentifiedComponent] = field(default_factory=list)
@@ -84,6 +118,20 @@ class ExtractionResult:
84118
# bom-ref -> the refs it depends on, containment overlaid with real edges recovered from embedded poms
85119
dependency_edges: dict[str, list[str]] = field(default_factory=dict)
86120
has_real_edges: bool = False
121+
# --include-declared only: declared dependencies whose version could not be established
122+
declared_unresolved: list[UnresolvedDeclaration] = field(default_factory=list)
123+
124+
@property
125+
def declared_components(self) -> list[IdentifiedComponent]:
126+
return [component for component in self.components if component.is_declared]
127+
128+
@property
129+
def transitive_components(self) -> list[IdentifiedComponent]:
130+
return [component for component in self.components if component.is_transitive]
131+
132+
@property
133+
def shipped_components(self) -> list[IdentifiedComponent]:
134+
return [component for component in self.components if not component.is_declared]
87135

88136

89137
class BinaryExtractor(ABC):

0 commit comments

Comments
 (0)