Skip to content

Package diagnostic transport backport for Alpine - #62

Closed
baijupn wants to merge 2 commits into
sonic-net:masterfrom
baijupn:fix/lemctl-transport-backport
Closed

baijupn wants to merge 2 commits into
sonic-net:masterfrom
baijupn:fix/lemctl-transport-backport

Conversation

@baijupn

@baijupn baijupn commented Oct 23, 2026 •

Copy link
Copy Markdown

Why I did it

Make the Lucius diagnostic CLI available through Alpine's normal package, container, and CI artifact paths with consistent connection behavior. This is a temporary backport of openconfig/lemming#754 while upstream review and merge are pending, rather than waiting for a future Lemming pin update.

How I did it

  • Add a lemctl Debian package built from Alpine's existing Lemming pin (1975dcc4cb7f1766445c791ba12e7445b2d74548). Apply the carried patch to a private source archive, leaving the shared submodule unchanged.
  • Verify remote TLS, support private CA trust, reject the old certificate-verification bypass, and restrict plaintext to literal loopback addresses. Cover info, SAI queries, and completion. The independent C++ SAI runtime transport is unchanged; this does not enable TLS on the Lucius server.
  • Build the release binary and its notices with CGO_ENABLED=0, avoiding a target libc dependency. Document the Go resolver/NSS trade-off and scope the intentional static-linking Lintian exception to /usr/bin/lemctl. Lintian remains a local check; CI adds only extracted/installed static-binary checks.
  • Exclude the four Bazel BUILD sections from the carried patch; Go implementation, tests and documentation remain unchanged.
  • Register package/image dependencies and cache inputs, build/install tests, and existing CI artifact/release aggregation.
  • Generate notices for the linked Go dependencies and toolchain. Store them outside the documentation directory removed during container cleanup; missing license texts fail the build.

Remove the patch when the pinned Lemming revision contains the upstream change. Forward-only, zero-fuzz application intentionally rejects incompatible or already-patched sources.

Verification

All results below are local validation, not claims of hosted CI approval.

Check Result
amd64 Debian package, clean Debian CI-style build, installed CLI and lint Passed; no lint errors
Actual SONiC package target and cache save/reload Passed
Actual SONiC cache hashing: patch and Lemming source changes Both invalidate their hashes
Focused Python packaging/cache/notices regressions 9 passed
Patched Go transport tests and amd64 race checks Passed
Real CLI loopback/nonloopback transport matrix 48/49 passed, plus 4/4 certificate-rejection regressions
Complete amd64 docker-sonic-alpinevs.gz build Passed
Binary and retained notices in the loaded final container Passed
ARM64 cross-build and executable tests under QEMU Passed
Full-system ARM64 guest: three affected race-test binaries Passed
ARM64 Debian package build/install, ordinary tests, installed CLI regressions Passed in the ARM64 guest

Key commands:

cd src/lemctl
dpkg-buildpackage -us -uc -b -tc
dpkg-deb -x ../lemctl_1.0-1_amd64.deb package-test
bash ./tests/test-static-binary.sh ./package-test/usr/bin/lemctl
./tests/test-binary.sh ./package-test/usr/bin/lemctl
grep -F 'Go toolchain' package-test/usr/share/lemctl/THIRD_PARTY_NOTICES

In initialized/configured sonic-buildimage with the candidate Alpine tree:

NOJESSIE=1 NOSTRETCH=1 NOBUSTER=1 NOBULLSEYE=1 NOBOOKWORM=1 NOTRIXIE=0 \
  UNATTENDED=y SONIC_BUILD_JOBS=4 \
  make target/debs/trixie/lemctl_1.0-1_amd64.deb
NOJESSIE=1 NOSTRETCH=1 NOBUSTER=1 NOBULLSEYE=1 NOBOOKWORM=1 NOTRIXIE=0 \
  UNATTENDED=y SONIC_BUILD_JOBS=4 make target/docker-sonic-alpinevs.gz

Retained evidence excerpts

All 48 linked dependency modules have accompanying notice sections.
[ CACHE::SAVED ] ...04fc166ebd715385cc62560.tgz
[ CACHE::LOADED ] ...04fc166ebd715385cc62560.tgz
Loaded image: docker-sonic-alpinevs:latest
lemctl 1.0-1
amd64
Go toolchain go1.26.7
Go standard-library source golang.org/fips140@v1.0.0-c2097c7c

Retained artifact SHA256:

  • amd64 Alpine container archive: ff6656e626622ce0e0e6909ab2292a2cf46c3006b71b799965a3851572d71fbe
  • ARM64 Debian package: abf60e328a2981a7581dfa6ada848924ae5835801d02fecf7a2d385549d379fd

Limits and integration follow-up

  • ARM64 results use software emulation, not physical ARM64 hardware. A full ARM64 SONiC image was not built.
  • Lint retains initial-upload-closes-no-bugs and no-manual-page warnings.
  • The amd64 full-image validation used a separate local sonic-buildimage wheel-recipe correction: an isolated native bookworm/trixie Python virtual environment preserving version hooks. It resolves the managed-system-Python installation failure; it is not part of this Alpine PR and needs separate buildimage delivery.
  • Existing CI in this PR builds amd64; ARM64 guest validation was performed locally.
  • After this PR merges, update sonic-buildimage's Alpine submodule pin in a dependent PR. No unmerged fork gitlink is proposed here.

October 26 review follow-up validation

Validated revision: 2a871c40f5060766e102f2a3f9611ac664ea93d9 (same file contents as the tested candidate). Rebuilt the complete amd64 Alpine container and verified its installed binary/notices against the rebuilt package, including actual hostname/TLS and transport RPC checks. In a full-system ARM64 guest, all three affected race binaries, native package build/install, nine Python regressions, Go tests, module verification, static ELF checks, 48 linked-module notices, and installed loopback/nonloopback CLI regressions passed.

Failed/partial attempts were retained: an unchanged ARM64 RPC race test timed out during a concurrent image build, then passed on an isolated rerun without changing assertions or deadlines. Image checks account only for the standard removal of documentation/Lintian metadata; fixture-permission and guest package-lock interruptions were corrected before final passing checks. Hostname checks use localhost; arbitrary DNS/custom NSS environments are not covered. Physical ARM64 and a full ARM64 SONiC image remain outside this validation.

Build the diagnostic CLI from a private copy of the pinned Lemming source, carry the temporary connection-policy patch, and register the package in the Alpine image and existing CI artifacts. Include linked-source notices and focused package/cache tests.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: db2e0f10-3729-4d5d-b4da-3654eb856d74
Signed-off-by: Baiju Parameswaran <baijup@microsoft.com>
@mssonicbld

Copy link
Copy Markdown

/azp run

@azure-pipelines

Copy link
Copy Markdown
No pipelines are associated with this pull request.

Comment thread src/lemctl/Makefile Resolved
@sonikajindal

Copy link
Copy Markdown
Contributor

Hi @baijupn , can you pls tell me where are you using lemctl and why do we need this change (as well as the openconfig/lemming change)?

cc: @hardikharti

@baijupn

baijupn commented Oct 26, 2026

Copy link
Copy Markdown
Author

Hi @baijupn , can you pls tell me where are you using lemctl and why do we need this change (as well as the openconfig/lemming change)?

cc: @hardikharti

Hi @sonikajindal,

lemctl lucius is a diagnostic client used to query Lucius forwarding information and SAI attributes during development and troubleshooting.

The change ensures remote diagnostic connections verify the server’s identity and protect traffic in transit.

The upstream change consistently requires verified TLS for remote endpoints, supports private CA certificates, and limits plaintext connections to literal loopback addresses.

This Alpine PR explicitly packages the diagnostic tool with those connection defaults.

The temporary backport allows Alpine delivery while upstream review and approval are pending. Once upstream merges and Alpine’s Lemming pin includes the change, the carried patch can be removed.

Address Alpine review feedback with a Go-only release build, matching notices, static-binary checks and a documented packaging exception. Keep only Go and documentation changes in the carried patch.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: db2e0f10-3729-4d5d-b4da-3654eb856d74
Signed-off-by: Baiju Parameswaran <baijup@microsoft.com>
@mssonicbld

Copy link
Copy Markdown

/azp run

@azure-pipelines

Copy link
Copy Markdown
No pipelines are associated with this pull request.

@sonikajindal

Copy link
Copy Markdown
Contributor

"The change ensures remote diagnostic connections verify the server’s identity and protect traffic in transit." - I am trying to understand who is the user for lemctl?

Just to add, we own the Lucius part of Lemming and we can work with you to merge the lemming PR, rather than creating this temporary backport.

@baijupn

baijupn commented Oct 26, 2026

Copy link
Copy Markdown
Author

"The change ensures remote diagnostic connections verify the server’s identity and protect traffic in transit." - I am trying to understand who is the user for lemctl?

Just to add, we own the Lucius part of Lemming and we can work with you to merge the lemming PR, rather than creating this temporary backport.

This tool is used for debugging by developers and troubleshooting in deployments.

Validation Results

Scenario Observed result
Normal verified TLS Trusted server accepted; invalid certificates rejected.
--tls_skip_verify Self-signed, expired, and wrong-identity certificates accepted; real diagnostic RPCs completed.
--insecure Real RPCs completed over plaintext, including to an isolated non-loopback address.
Synthetic endpoint responses CLI displayed information supplied by the test endpoint.

After the fix

  • Verified TLS/private-CA connections worked.
  • Certificate bypass and remote plaintext were rejected.
  • Literal-loopback plaintext remained functional.
  • Package, full amd64 image, and ARM64 VM checks passed.

It would be great if you could help with merging the lemming PR, and this backport may not be needed anymore ..
Thanks

@sonikajindal

Copy link
Copy Markdown
Contributor

I mean, are you using this tool? Because we don't use this (so far), hence curious why you ended up using it and found the gap.

@baijupn

baijupn commented Oct 26, 2026

Copy link
Copy Markdown
Author

I mean, are you using this tool? Because we don't use this (so far), hence curious why you ended up using it and found the gap.

Checked again within the team and it looks like this tool is not used even though the source code in present. Closing these PRs!!

@baijupn baijupn closed this Oct 26, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants