Skip to content
This repository was archived by the owner on Jul 6, 2026. It is now read-only.

feat(operator): rewrite operator from kopf (Python) to kubebuilder (Go) - #11

Draft
kitsunoff wants to merge 7 commits into
mainfrom
rewrite
Draft

kitsunoff wants to merge 7 commits into
mainfrom
rewrite

Conversation

@kitsunoff

Copy link
Copy Markdown
Contributor

Summary

  • Migrate NixOS Operator from kopf (Python) to kubebuilder (Go) for better performance, type safety, and ecosystem integration
  • Implement Machine and NixosConfiguration controllers with kstatus-compliant conditions (Ready, Reconciling, Stalled)
  • Add Job-based apply mechanism for isolated nixos-rebuild/nixos-anywhere execution

Changes

API Types (api/v1alpha1/):

  • Machine CRD with SSH connectivity tracking and hardware facts storage
  • NixosConfiguration CRD with Job-based apply, additionalFiles injection, and jobTemplate customization
  • kstatus-compliant conditions with observedGeneration

Controllers (internal/controller/):

  • MachineReconciler: SSH connectivity checks, Secret watches, finalizer handling
  • NixosConfigurationReconciler: Job creation/monitoring, concurrency limits, onRemoveFlake support

Infrastructure:

  • SSH client with mock support for testing
  • Prometheus metrics (counters, gauges, histograms)
  • Apply Job runner for git clone + nixos-rebuild/nixos-anywhere

Known Limitations (to address before stable release)

  • SSH host key verification disabled (InsecureIgnoreHostKey) - security risk for MITM
  • DefaultApplyImage hardcoded to :latest - should be configurable via flag/env
  • Machine deletion doesn't check for referencing NixosConfigurations
  • AppliedCommit stores ref name instead of actual git commit SHA
  • NixosConfigurationReconciler lacks unit tests
  • No TTLSecondsAfterFinished on Jobs - old jobs accumulate
  • Status update race condition - needs conflict retry

Test plan

  • go build ./... passes
  • go test ./... passes (MachineReconciler covered)
  • golangci-lint run ./... - 0 issues
  • E2E testing with actual NixOS machine
  • Integration testing in homelab cluster

kitsunoff and others added 6 commits February 19, 2026 19:50
Comprehensive analysis document for migrating nixos-operator from
KOPF (Python) to kubebuilder (Go), covering:

- CRD definitions with kstatus compliance
- Reconciler design patterns
- Long-running operations via Kubernetes Jobs
- Secret watches with field indexes
- Testing strategy with unit test examples
- Owner references and garbage collection
- State machines and lifecycle diagrams

Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: ZverGuy <maximbel2003@gmail.com>
- Replace separate hostname/ipAddress fields with single host field
- Add SecretKeyReference with explicit key field for additionalFiles
- Add jobTemplate for pod customization (image, nodeSelector, tolerations, resources, serviceAccountName)
- Remove Age from additionalPrinterColumns (built-in kubectl column)
- Add complete Go type definitions for MachineSpec

Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: ZverGuy <maximbel2003@gmail.com>
Initialize kubebuilder project with:
- Domain: homystack.com
- Repo: github.com/homystack/nixos-operator
- Go 1.25.5, kubebuilder v4.10.1

Created API scaffolds:
- Machine (nio.homystack.com/v1alpha1)
- NixosConfiguration (nio.homystack.com/v1alpha1)

Ref: Issue #2

Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: ZverGuy <maximbel2003@gmail.com>
…s compliance

Machine type includes:
- Spec: host, sshUser, sshKeySecretRef, sshPasswordSecretRef
- Status: observedGeneration, discoverable, hasConfiguration,
  appliedConfiguration, appliedCommit, hardwareFacts, nixFacterResult
- Conditions: Ready, Reconciling, Stalled, Discoverable, HardwareScanned
- Printer columns for kubectl output

NixosConfiguration type includes:
- Spec: machineRef, gitRepo, ref, credentialsRef, flake, onRemoveFlake,
  configurationSubdir, fullInstall, additionalFiles, jobTemplate
- Status: observedGeneration, fullDiskInstallCompleted, appliedCommit,
  configurationHash, additionalFilesHash, operationState
- Conditions: Ready, Reconciling, Stalled, Applied, GitSynced
- Support for JobTemplate customization

Added condition constants and reasons for kstatus compliance.

Ref: Issue #3

Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: ZverGuy <maximbel2003@gmail.com>
…hecks

MachineReconciler implementation includes:
- SSH Client interface with mock for testing
- SSH connectivity checks using golang.org/x/crypto/ssh
- Key-based and password-based authentication support
- Condition management for kstatus compliance:
  - Ready, Reconciling, Stalled, Discoverable
- Secret watching for SSH credentials updates
- Field indexes for efficient secret-to-machine mapping
- Finalizer handling for clean deletion
- Event recording for status changes
- Periodic requeue for connectivity monitoring

Tests cover:
- Successful SSH connection scenario
- Failed SSH connection scenario
- Missing SSH credentials scenario
- Non-existent resource handling

Ref: Issue #4

Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: ZverGuy <maximbel2003@gmail.com>
…ed apply

NixosConfigurationReconciler implementation includes:
- Machine reference resolution and discovery validation
- Kubernetes Job creation for nixos-rebuild and nixos-anywhere
- Job lifecycle monitoring (pending, running, succeeded, failed)
- Per-machine concurrency protection via labels
- Global concurrency limiting (max 5 concurrent jobs)
- Configuration hash calculation for change detection
- Machine status updates on successful apply
- Finalizer handling for cleanup on deletion
- onRemoveFlake placeholder for deletion cleanup
- Secret mounting for SSH keys and git credentials
- Pod security context with minimal privileges
- JobTemplate customization (nodeSelector, tolerations, resources)

RBAC updates for Job management and pods/logs access.

Ref: Issue #5

Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: ZverGuy <maximbel2003@gmail.com>
Comment on lines +14 to +23
- apiGroups:
- ""
resources:
- pods
- pods/log
- secrets
verbs:
- get
- list
- watch

Check failure

Code scanning / Trivy

Manage secrets Critical

Artifact: go-operator/config/rbac/role.yaml
Type: kubernetes
Vulnerability KSV-0041
Severity: CRITICAL
Message: ClusterRole 'manager-role' shouldn't have access to manage resource 'secrets'
Link: KSV-0041
Comment on lines +60 to +95
containers:
- command:
- /manager
args:
- --leader-elect
- --health-probe-bind-address=:8081
image: controller:latest
name: manager
ports: []
securityContext:
readOnlyRootFilesystem: true
allowPrivilegeEscalation: false
capabilities:
drop:
- "ALL"
livenessProbe:
httpGet:
path: /healthz
port: 8081
initialDelaySeconds: 15
periodSeconds: 20
readinessProbe:
httpGet:
path: /readyz
port: 8081
initialDelaySeconds: 5
periodSeconds: 10
# TODO(user): Configure the resources accordingly based on the project requirements.
# More info: https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/
resources:
limits:
cpu: 500m
memory: 128Mi
requests:
cpu: 10m
memory: 64Mi

Check warning

Code scanning / Trivy

Image tag ":latest" used Medium

Artifact: go-operator/config/manager/manager.yaml
Type: kubernetes
Vulnerability KSV-0013
Severity: MEDIUM
Message: Container 'manager' of Deployment 'controller-manager' should specify an image tag
Link: KSV-0013
Comment on lines +10 to +21
- apiGroups:
- ""
resources:
- configmaps
verbs:
- get
- list
- watch
- create
- update
- patch
- delete

Check warning

Code scanning / Trivy

Manage configmaps Medium

Artifact: go-operator/config/rbac/leader_election_role.yaml
Type: kubernetes
Vulnerability KSV-0049
Severity: MEDIUM
Message: Role 'leader-election-role' should not have access to resource 'configmaps' for verbs ["create", "update", "patch", "delete", "deletecollection", "impersonate", "*"]
Link: KSV-0049
Comment on lines +24 to +35
- apiGroups:
- batch
resources:
- jobs
verbs:
- create
- delete
- get
- list
- patch
- update
- watch

Check warning

Code scanning / Trivy

Manage Kubernetes workloads and pods Medium

Artifact: go-operator/config/rbac/role.yaml
Type: kubernetes
Vulnerability KSV-0048
Severity: MEDIUM
Message: ClusterRole 'manager-role' should not have access to resources ["pods", "deployments", "jobs", "cronjobs", "statefulsets", "daemonsets", "replicasets", "replicationcontrollers"] for verbs ["create", "update", "patch", "delete", "deletecollection", "impersonate", "*"]
Link: KSV-0048
Comment thread go-operator/Dockerfile
@@ -0,0 +1,31 @@
# Build the manager binary

Check notice

Code scanning / Trivy

No HEALTHCHECK defined Low

Artifact: go-operator/Dockerfile
Type: dockerfile
Vulnerability DS-0026
Severity: LOW
Message: Add HEALTHCHECK instruction in your Dockerfile
Link: DS-0026
Comment on lines +60 to +95
containers:
- command:
- /manager
args:
- --leader-elect
- --health-probe-bind-address=:8081
image: controller:latest
name: manager
ports: []
securityContext:
readOnlyRootFilesystem: true
allowPrivilegeEscalation: false
capabilities:
drop:
- "ALL"
livenessProbe:
httpGet:
path: /healthz
port: 8081
initialDelaySeconds: 15
periodSeconds: 20
readinessProbe:
httpGet:
path: /readyz
port: 8081
initialDelaySeconds: 5
periodSeconds: 10
# TODO(user): Configure the resources accordingly based on the project requirements.
# More info: https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/
resources:
limits:
cpu: 500m
memory: 128Mi
requests:
cpu: 10m
memory: 64Mi

Check notice

Code scanning / Trivy

Runs with UID <= 10000 Low

Artifact: go-operator/config/manager/manager.yaml
Type: kubernetes
Vulnerability KSV-0020
Severity: LOW
Message: Container 'manager' of Deployment 'controller-manager' should set 'securityContext.runAsUser' > 10000
Link: KSV-0020
Comment on lines +60 to +95
containers:
- command:
- /manager
args:
- --leader-elect
- --health-probe-bind-address=:8081
image: controller:latest
name: manager
ports: []
securityContext:
readOnlyRootFilesystem: true
allowPrivilegeEscalation: false
capabilities:
drop:
- "ALL"
livenessProbe:
httpGet:
path: /healthz
port: 8081
initialDelaySeconds: 15
periodSeconds: 20
readinessProbe:
httpGet:
path: /readyz
port: 8081
initialDelaySeconds: 5
periodSeconds: 10
# TODO(user): Configure the resources accordingly based on the project requirements.
# More info: https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/
resources:
limits:
cpu: 500m
memory: 128Mi
requests:
cpu: 10m
memory: 64Mi

Check notice

Code scanning / Trivy

Runs with GID <= 10000 Low

Artifact: go-operator/config/manager/manager.yaml
Type: kubernetes
Vulnerability KSV-0021
Severity: LOW
Message: Container 'manager' of Deployment 'controller-manager' should set 'securityContext.runAsGroup' > 10000
Link: KSV-0021
- Add cmd/apply package for Job-based apply execution
- Add internal/applyjob package with git clone and nixos-rebuild/anywhere runner
- Add internal/metrics package with Prometheus metrics (gauges, counters, histograms)
- Add SSH client unit tests
- Add RBAC role for apply jobs
- Update controllers with metrics integration
- Fix Containerfile.ipxe PATH configuration

Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: ZverGuy <maximbel2003@gmail.com>
Comment on lines +9 to +14
- apiGroups:
- ""
resources:
- secrets
verbs:
- get

Check warning

Code scanning / Trivy

Manage namespace secrets Medium

Artifact: go-operator/config/rbac/apply_job_role.yaml
Type: kubernetes
Vulnerability KSV-0113
Severity: MEDIUM
Message: Role 'apply-job-role' shouldn't have access to manage secrets in namespace 'system'
Link: KSV-0113
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants