Add optional root CA signature verification for client, server and management certificates

[Damans227]

Description

The root-ca plugin never checked that certificates were actually signed by the CloudStack root CA. checkClientTrusted() checked revocation, validity and SAN, but not the signature. checkServerTrusted() did nothing at all, so any server certificate was trusted. isManagementCertificate() trusted any cert with the right SAN, signed or not.

This adds the missing signature check to all three, checked against every CA in the configured chain (so CA rotation still works). It's controlled by a new config key, ca.plugin.root.ca.signature.verification, off by default so upgrades are a no-op.

For checkClientTrusted/checkServerTrusted, this follows the same pattern as the plugin's other checks: it only actually rejects a bad cert when ca.plugin.root.auth.strictness is also true. Otherwise it just logs. isManagementCertificate always enforces it once enabled, since it's not part of that strict/non-strict handshake logic.

Types of changes

• Enhancement (improves an existing feature and functionality)

Feature/Enhancement Scale or Bug Severity

Feature/Enhancement Scale

• Minor

How Has This Been Tested?

Added unit tests for valid certs, certs signed by a rogue CA, certs signed by a non-primary CA in a rotated chain, and the no-CA-available case, in both strict and non-strict mode, across all three checks.

How did you try to break this feature and the system with this change?

Checked the flag defaults to off so nothing changes for existing deployments. Checked a cert signed by a different CA gets rejected once both flags are on. Checked CA rotation still works. Checked the missing-CA case doesn't throw an unhandled exception mid-handshake.

[codecov]

Codecov Report

❌ Patch coverage is 89.06250% with 7 lines in your changes missing coverage. Please review.
✅ Project coverage is 19.65%. Comparing base (4f11707) to head (6a0d895).
⚠️ Report is 166 commits behind head on main.

Files with missing lines	Patch %	Lines
...oudstack/ca/provider/RootCACustomTrustManager.java	84.78%	2 Missing and 5 partials ⚠️

Additional details and impacted files

@@            Coverage Diff            @@
##               main   #13787   +/-   ##
=========================================
  Coverage     19.65%   19.65%           
- Complexity    19792    19808   +16     
=========================================
  Files          6368     6368           
  Lines        574881   574942   +61     
  Branches      70351    70368   +17     
=========================================
+ Hits         112970   113029   +59     
+ Misses       449639   449635    -4     
- Partials      12272    12278    +6     

Flag	Coverage Δ
uitests	3.41% <ø> (ø)
unittests	20.93% <89.06%> (+<0.01%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[kiranchavala]

@blueorangutan package

[blueorangutan]

@kiranchavala a [SL] Jenkins job has been kicked to build packages. It will be bundled with no SystemVM templates. I'll keep you posted as I make progress.

[blueorangutan]

Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ el10 ✔️ debian ✔️ suse15. SL-JID 19278

[kiranchavala]

LGTM , tested manually

Set the following global settings

cmk update configuration name=ca.plugin.root.ca.signature.verification value=true
cmk update configuration name=ca.plugin.root.auth.strictness value=true

On the kvm host, generate a certificate

openssl req -x509 -newkey rsa:2048 -nodes -days 1 \
  -keyout /tmp/rogue.key -out /tmp/rogue.crt -subj "/CN=rogue-agent"

openssl s_client -connect 10.0.35.135:8250 -cert /tmp/rogue.crt -key /tmp/rogue.key </dev/null

Check the management server log

The connection is rejected — management server log:

2026-09-21 03:21:43,735 DEBUG [o.a.c.c.p.RootCACustomTrustManager] (pool-1576-thread-1:[]) (logid:) A client/agent attempting connection from address=10.0.33.195 has presented these certificate(s):
Certificate [1] :
 Serial: 1eed66df38bed40671dcef4ea068043be35377c
  Not Before:Mon Sep 21 03:20:35 UTC 2026
  Not After:Tue Sep 22 03:20:35 UTC 2026
  Signature Algorithm:SHA256withRSA
  Version:3
  Subject DN:CN=rogue-agent
  Issuer DN:CN=rogue-agent
  Alternative Names:null
2026-09-21 03:21:43,735 ERROR [o.a.c.c.p.RootCACustomTrustManager] (pool-1576-thread-1:[]) (logid:) Client certificate is not signed by the root CA, serial=1eed66df38bed40671dcef4ea068043be35377c, subject=CN=rogue-agent from address=10.0.33.195
2026-09-21 03:21:43,739 ERROR [o.a.c.c.p.RootCACustomTrustManager] (pool-1576-thread-1:[]) (logid:) Certificate ownership verification failed for client: 10.0.33.195
2026-09-21 03:21:43,742 ERROR [c.c.u.n.Link] (AgentManager-SSLHandshakeHandler-3:[]) (logid:) SSL error caught during wrap data: (certificate_unknown) Client certificate is not signed by the root CA, serial=1eed66df38bed40671dcef4ea068043be35377c, subject=CN=rogue-agent from address=10.0.33.195. Certificate ownership verification failed for client: 10.0.33.195, for local address=/10.0.34.120:8250, remote address=/10.0.33.195:37184.

set the global setting back to false

cmk update configuration name=ca.plugin.root.ca.signature.verification value=false

run the command

openssl s_client -connect 10.0.35.135:8250 -cert /tmp/rogue.crt -key /tmp/rogue.key </dev/null

Check the management server log

2026-09-21 03:24:58,162 DEBUG [o.a.c.c.p.RootCACustomTrustManager] (pool-1579-thread-1:[]) (logid:) A client/agent attempting connection from address=10.0.33.195 has presented these certificate(s):
Certificate [1] :
 Serial: 1eed66df38bed40671dcef4ea068043be35377c
  Not Before:Mon Sep 21 03:20:35 UTC 2026
  Not After:Tue Sep 22 03:20:35 UTC 2026
  Signature Algorithm:SHA256withRSA
  Version:3
  Subject DN:CN=rogue-agent
  Issuer DN:CN=rogue-agent
  Alternative Names:null
2026-09-21 03:24:58,164 ERROR [o.a.c.c.p.RootCACustomTrustManager] (pool-1579-thread-1:[]) (logid:) Certificate ownership verification failed for client: 10.0.33.195
2026-09-21 03:24:58,164 ERROR [c.c.u.n.Link] (AgentManager-SSLHandshakeHandler-4:[]) (logid:) SSL error caught during wrap data: (certificate_unknown) Certificate ownership verification failed for client: 10.0.33.195, for local address=/10.0.34.120:8250, remote address=/10.0.33.195:52968.

[nvazquez]

@blueorangutan test

[blueorangutan]

@nvazquez a [SL] Trillian-Jenkins test job (ol8 mgmt + kvm-ol8) has been kicked to run smoke tests

[blueorangutan]

[SF] Trillian Build Failed (tid-17041)

[Damans227]

@blueorangutan test

[Damans227]

@nvazquez can you review this one? last smoke run died on lab setup, rerunning