Restrict Flex Node bootstrap token RBAC

[Alex Huang (wenhug)]

Summary

• remove the bootstrap-group to system:node privilege escalation path while retaining least-privilege CSR creation and approval
• add an explicit, fail-closed migration for the canonical legacy aks-flex-node-role binding, including optimistic concurrency checks and private local API transport
• preserve already-joined nodes by requiring operators to verify certificate-backed agents before opting into removal; token generation refuses to mint a new token while unsafe legacy bindings remain
• add a real historical migration E2E that installs official v0.1.0, upgrades the same host to HEAD, migrates RBAC idempotently, revokes the token, restarts services, and verifies Node identity/readiness/Lease continuity
• harden E2E state, concurrent-run isolation, and cleanup for both current resources and historical AKS node resource groups/untagged implicit OS disks

Compatibility and migration

Existing joined nodes that use their issued kubelet/daemon certificates continue to work after migration. New and in-progress joins keep the narrowly scoped CSR permissions.

For a cluster configured by an older helper:

1. upgrade every bootstrap-token Flex Node agent to v0.1.1 or later and verify its daemon certificate and service health;
2. run setup-node-rbac --remove-legacy-node-role-binding explicitly;
3. remove old non-expiring bootstrap-token Secrets after confirming they are no longer needed.

Plain setup-node-rbac installs/verifies the safe bindings but deliberately does not remove the legacy binding. generate-node-config --bootstrap-token fails closed while any bootstrap group still has system:node; it does not silently migrate an old node.

Automatic removal is limited to the exact canonical legacy object and uses UID/resourceVersion preconditions. Ambiguous, customized, or concurrently changed bindings are preserved for manual review.

Validation

Local checks on the current branch:

• make check
• go test -race -count=1 ./...
• go test -race -count=10 ./hack/e2e
• ShellCheck and bash -n for changed shell scripts
• Python bytecode compilation for scripts/aks-flex-config
• Bicep compilation for hack/e2e/infra/main.bicep

Real-cluster historical migration assertions passed on fresh Azure infrastructure at:

• AKS/Kubernetes 1.34.9: https://github.com/Azure/AKSFlexNode/actions/runs/32547937112
• AKS/Kubernetes 1.35.0: https://github.com/Azure/AKSFlexNode/actions/runs/32548662068

In both runs, a real Ubuntu VM joined as a Ready node using the official v0.1.0 release, the same host upgraded to the PR build, the migration remained idempotent, bootstrap-token authorization changed from 200 to 403 and then 401 after revocation, and the same Node UID remained Ready while its Lease renewed after kubelet/daemon restarts. Those workflow runs are red only because an older cleanup implementation could not remove the default AKS node resource group; the functional test step passed. Final-SHA reruns including the cleanup fixes are in progress.

This reproduces historical v0.1.0 node/config/RBAC state on newly created clusters. It is not evidence from a long-lived customer cluster with accumulated upgrade history or configuration drift. The repository E2E CSR approver is also not independent validation of the production AKS managed-ownership check; the test documents that boundary.

Related work

This implements the backlog item Fix flex-node-bootstrap node group.

PR #284 must retain the same least-privilege manifest and explicit migration semantics when rebased or merged. A repository-wide invariant test prevents bootstrap groups from being rebound to system:node.

[Copilot]

Pull request overview

This PR tightens AKS Flex Node bootstrap-token RBAC by removing the legacy broad system:node binding for bootstrapper groups, adding migration/cleanup steps to reconcile existing clusters, and documenting the new behavior to ensure least-privilege node joins.

Changes:

• Removes the legacy aks-flex-node-role ClusterRoleBinding from the Python helper RBAC manifest, and adds idempotent cleanup logic that runs after applying least-privilege RBAC (and before minting new bootstrap tokens).
• Updates kubeadm E2E RBAC reconciliation to fail fast and explicitly propagate errors while running under a cluster lock that may disable errexit.
• Adds Go tests that assert the exact RBAC manifest content, ordering, idempotency, and a repository-wide invariant preventing bootstrap groups from being bound to system:node, plus documentation updates describing migration and compatibility.

Reviewed changes

Copilot reviewed 5 out of 5 changed files in this pull request and generated no comments.

Show a summary per file

File	Description
scripts/aks-flex-config	Drops legacy system:node binding, adds legacy-binding cleanup after least-privilege RBAC and before token minting (fail-closed).
scripts/aks_flex_config_test.go	Adds regression tests for RBAC manifest contents, migration ordering/idempotency, fail-closed cleanup, kubeadm reconciliation behavior, and repo-wide invariant scanning.
hack/e2e/lib/node-join-kubeadm.sh	Removes legacy broad binding, adds explicit error propagation for RBAC and ConfigMap apply plus legacy-binding deletion.
docs/usages/joining-nodes.md	Documents least-privilege RBAC setup and legacy binding removal behavior when re-running setup-node-rbac.
docs/usages/aks-flex-config.md	Documents new required permissions and migration steps, including manual legacy binding removal command and fail-closed token behavior.

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[Copilot]

Pull request overview

Copilot reviewed 10 out of 10 changed files in this pull request and generated 1 comment.

[Copilot]

Pull request overview

Copilot reviewed 10 out of 10 changed files in this pull request and generated no new comments.

Suppressed comments (2)

Previously missed (1) — in code that hasn't changed since the last review.

docs/usages/aks-flex-config.md:80

• The docs state the expected result of kubectl get clusterrolebinding aks-flex-node-role is NotFound, but the helper intentionally preserves a same-named binding when it is not the legacy unsafe system:node binding. This verification step should instead instruct users to confirm the binding is absent or that it does not bind the bootstrap group to system:node, to avoid false alarms in clusters with a repurposed/safe same-named binding.

The expected result is `NotFound`. Once certificate issuance has been verified, both the kubelet and long-running Flex daemon use issued client certificates, so removing this binding does not interrupt joined nodes. New and in-progress joins retain the CSR permissions installed above.

Do not roll back a migrated host to an older or development-mode agent that still uses the bootstrap token for ordinary Kubernetes API requests. After this binding is removed, those requests correctly receive `403 Forbidden`. Restore a supported certificate-using agent instead of restoring the broad binding.

scripts/aks-flex-config:320

• The port-selection strategy has a TOCTOU race: the reservation socket binds to an ephemeral port and is then closed before starting kubectl proxy, so another process can claim the port and cause intermittent proxy-start failures. Consider retrying proxy startup on bind failures (e.g., loop and pick a new port if the proxy exits immediately with an “address already in use” error), rather than assuming the reserved port will still be available.

    with socket.socket(socket.AF_INET, socket.SOCK_STREAM) as reservation:
        reservation.bind(("127.0.0.1", 0))
        port = reservation.getsockname()[1]

    proxy = subprocess.Popen(
        [

[Copilot]

Pull request overview

Copilot reviewed 10 out of 10 changed files in this pull request and generated no new comments.

[Copilot]

Pull request overview

Copilot reviewed 20 out of 20 changed files in this pull request and generated 2 comments.

[Copilot]

Pull request overview

Copilot reviewed 20 out of 20 changed files in this pull request and generated no new comments.

Suppressed comments (1)

Previously missed (1) — in code that hasn't changed since the last review.

scripts/aks-flex-config:128

• binding_has_bootstrap_group raises SystemExit("...subjects is not a list") without identifying which ClusterRoleBinding was malformed. If a cluster has drifted RBAC objects, this makes the failure hard to diagnose. Consider including the binding name (from metadata.name) in the error message, or passing name into this helper so the message can point at the exact object that needs manual review.

    subjects = binding.get("subjects", [])
    if subjects is None:
        subjects = []
    if not isinstance(subjects, list):
        raise SystemExit("ERROR: managed ClusterRoleBinding subjects is not a list")

[Copilot]

Pull request overview

Copilot reviewed 23 out of 23 changed files in this pull request and generated no new comments.

[Copilot]

Pull request overview

Copilot reviewed 23 out of 23 changed files in this pull request and generated no new comments.

[Copilot]

Pull request overview

Copilot reviewed 23 out of 23 changed files in this pull request and generated no new comments.

[Alex Huang (wenhug)]

Superseded by a smaller, reviewable three-PR stack:

1. [1/3] Restrict Flex Node bootstrap token RBAC #287 — core bootstrap RBAC restriction
2. [2/3] Harden E2E state, cleanup, and Arc diagnostics #288 — E2E state and cleanup hardening
3. [3/3] Test historical bootstrap RBAC migration #289 — historical RBAC migration coverage

The tip of #289 is byte-for-byte identical to commit d677542 from this PR. Closing this monolithic PR so review can proceed in dependency order, starting with #287.