This document describes the Prometheus metrics exposed by the HyperFleet operator, including their meaning, labels, and example queries for common investigations. It follows the HyperFleet metrics standard, matching the naming and label conventions used by the other components (API, Sentinel, Adapters).
Metrics are exposed at:
- Endpoint:
/metrics - Port: 9090 (default, configurable via
--metrics-bind-address; set0to disable) - Protocol: plain HTTP (
--metrics-secure=falseby default) - Format: OpenMetrics/Prometheus text format
The operator can create a ServiceMonitor at runtime when the Prometheus Operator
API is available. The default ServiceMonitor uses HTTP, matching the default
metrics endpoint. A secure ServiceMonitor is rendered when --metrics-secure=true
and its required trust and authentication references are supplied; see
Secure metrics scraping.
The operator's custom collectors register into controller-runtime's registry, so
they are served on the same /metrics endpoint as the built-in
controller_runtime_* metrics — there is no second metrics server.
All operator-defined series share the hyperfleet_operator_ prefix (Prometheus
Namespace hyperfleet + Subsystem operator) and carry the standard
component and version const labels.
Type: Gauge (always 1)
Description: Build information for the HyperFleet operator component. The
value is always 1; the identity is carried in the labels.
Labels:
| Label | Description | Example Values |
|---|---|---|
component |
Component name (const) | operator |
version |
Application version (const) | v1.2.3, dev |
commit |
Git commit SHA (short) | abc1234 |
go_version |
Go runtime version | go1.26.0 |
Example output:
hyperfleet_operator_build_info{component="operator",version="v1.2.3",commit="abc1234",go_version="go1.26.0"} 1
Version and commit are injected at build time via
-ldflags -X(seeinternal/version). Under a plaingo build/make runthey fall back to the module version and VCS revision from the binary's build info, so the metric is still populated (version="dev"when unavailable).
Type: Gauge
Description: 1 while the operator process is running. Set once at startup.
Distinct from the Prometheus scrape-generated up series.
Labels: component, version (const)
Example output:
hyperfleet_operator_up{component="operator",version="v1.2.3"} 1
These metrics track the HyperFleetConfig reconcile loop.
Type: Histogram
Description: Wall-clock duration of a full reconcile, recorded regardless of outcome.
Labels: component, version (const)
Buckets: 0.005s, 0.01s, 0.025s, 0.05s, 0.1s, 0.25s, 0.5s, 1s, 2.5s, 5s, 10s
Derived metrics:
hyperfleet_operator_reconcile_duration_seconds_sumhyperfleet_operator_reconcile_duration_seconds_counthyperfleet_operator_reconcile_duration_seconds_bucket
Type: Counter
Description: Total number of reconcile errors, labeled by the stage that failed, so error rate can be broken down by cause.
Labels:
| Label | Description | Example Values |
|---|---|---|
component |
Component name (const) | operator |
version |
Application version (const) | v1.2.3 |
reason |
Reconcile stage that failed | get, discovery, secrets, bundle, render, apply |
Example output:
hyperfleet_operator_reconcile_errors_total{component="operator",version="v1.2.3",reason="apply"} 3
These metrics track the operands the operator manages (one per component in the
resolved bundle, e.g. api).
Type: Gauge
Description: Readiness of each operand workload: 1 when the Deployment
reports Available=True, 0 otherwise. Published from the freshly applied state
each reconcile.
Labels:
| Label | Description | Example Values |
|---|---|---|
component |
Component name (const) | operator |
version |
Application version (const) | v1.2.3 |
operand |
Operand component name | api |
Example output:
hyperfleet_operator_operand_ready{component="operator",version="v1.2.3",operand="api"} 1
Type: Counter
Description: Total operand workload rollouts, labeled by operand and the trigger that caused the rollout. Detected before applying, by comparing the live pod template against the desired one via a template-hash annotation.
Labels:
| Label | Description | Example Values |
|---|---|---|
component |
Component name (const) | operator |
version |
Application version (const) | v1.2.3 |
operand |
Operand component name | api |
trigger |
What caused the rollout | create, image, config |
Trigger values:
create— the operand's workload did not exist and was created.image— a rollout caused by a container image change.config— a rollout caused by any other pod-template change (env, resources, ...).
Example output:
hyperfleet_operator_operand_rollouts_total{component="operator",version="v1.2.3",operand="api",trigger="create"} 1
hyperfleet_operator_operand_rollouts_total{component="operator",version="v1.2.3",operand="api",trigger="image"} 4
Type: Gauge (info-style, always 1)
Description: Info metric whose hash label is the digest of the currently
applied configuration: the HyperFleetConfig spec plus every component's
config-rollout hash (rendered config content and referenced-Secret
resourceVersions). Covering only the spec would miss a Secret rotation or a
resolved value that never lands in the CR (e.g. OIDC JWKS discovery), either of
which changes what is actually applied to an operand without changing the spec
itself. Exactly one series exists at a time: the collector is reset before
each set, so a change replaces the previous series rather than accumulating
cardinality.
Labels:
| Label | Description | Example Values |
|---|---|---|
component |
Component name (const) | operator |
version |
Application version (const) | v1.2.3 |
hash |
12-char SHA-256 digest of the applied spec and component config hashes | 9f2a1c4b7d3e |
Example output:
hyperfleet_operator_applied_config_info{component="operator",version="v1.2.3",hash="9f2a1c4b7d3e"} 1
Because the operator's collectors share controller-runtime's registry, the standard controller-runtime metrics are exposed on the same endpoint, including:
| Metric | Type | Description |
|---|---|---|
controller_runtime_reconcile_total |
Counter | Total reconciles per controller, by result (success, error, requeue). |
controller_runtime_reconcile_errors_total |
Counter | Total reconcile errors per controller. |
controller_runtime_reconcile_time_seconds |
Histogram | Reconcile latency per controller. |
controller_runtime_active_workers |
Gauge | Number of active reconcile workers per controller. |
workqueue_depth |
Gauge | Current depth of the reconcile work queue. |
The Prometheus Go client library automatically exposes process and runtime
metrics on the same endpoint, e.g. process_cpu_seconds_total,
process_resident_memory_bytes, process_start_time_seconds, go_goroutines,
and the go_memstats_* family. Use process_start_time_seconds (rather than a
custom build-time label) to answer "when did this binary start".
# Reconcile rate (per second)
rate(hyperfleet_operator_reconcile_duration_seconds_count[5m])
# Reconcile error rate by failing stage
sum by (reason) (rate(hyperfleet_operator_reconcile_errors_total[5m]))
# P99 reconcile latency
histogram_quantile(0.99,
sum(rate(hyperfleet_operator_reconcile_duration_seconds_bucket[5m])) by (le))
# Average reconcile duration
rate(hyperfleet_operator_reconcile_duration_seconds_sum[10m]) /
rate(hyperfleet_operator_reconcile_duration_seconds_count[10m])
# Operands that are not ready
hyperfleet_operator_operand_ready == 0
# Rollout rate by operand and trigger
sum by (operand, trigger) (rate(hyperfleet_operator_operand_rollouts_total[15m]))
# Image-triggered rollouts in the last hour
increase(hyperfleet_operator_operand_rollouts_total{trigger="image"}[1h])
# Currently applied config digest (the single active series)
hyperfleet_operator_applied_config_info
# Operator build identity
hyperfleet_operator_build_info
# Is the operator up?
hyperfleet_operator_up
The operator creates and owns its ServiceMonitor
(controller-manager-metrics-monitor) at runtime when
--metrics-service-monitor-enabled=true (the default). The bootstrap is
conditional: it runs only when the cluster serves the Prometheus Operator API
(monitoring.coreos.com/v1), and is skipped — with a log line — when that CRD is
absent. Metrics remain available on :9090 either way. See the
internal/servicemonitor package.
The ServiceMonitor is intentionally not shipped in the OLM bundle. OLM
applies a bundle's arbitrary manifests but does not install the CRDs they depend
on, so bundling it would fail the InstallPlan — and block the entire operator
install — on any cluster without the Prometheus Operator CRD. Because HyperFleet
targets generic Kubernetes (not only OpenShift, where that CRD is guaranteed), the
runtime bootstrap above degrades gracefully instead.
A cluster that installs the Prometheus Operator after the operator started picks
the ServiceMonitor up on the operator's next restart. For GitOps, the equivalent
static manifest remains available at config/prometheus/monitor.yaml and can be
applied directly (requires the ServiceMonitor CRD). GitOps and the runtime
bootstrap must not manage the same object: set
--metrics-service-monitor-enabled=false when applying the static manifest.
--metrics-secure=true enables controller-runtime TLS plus Kubernetes
authentication and authorization on /metrics. A ServiceMonitor in this mode
must use HTTPS and provide all of the following:
- A mounted, stable server certificate for the manager
(
--metrics-cert-path, normally frommetrics-server-cert). Controller-runtime self-signed development certificates are not a supported production scrape trust source. --metrics-service-monitor-ca-secret-nameand--metrics-service-monitor-ca-secret-key, naming the CA that verifies that server certificate.--metrics-service-monitor-server-name, matching the serving certificate DNS name, for examplehyperfleet-operator-controller-manager-metrics-service.hyperfleet-system.svc.--metrics-service-monitor-authorization-secret-nameand--metrics-service-monitor-authorization-secret-key, naming a Secret in the ServiceMonitor namespace containing a bearer credential.
The credential subject must be authorized to get the non-resource URL
/metrics, normally by binding it to the shipped metrics-reader ClusterRole.
The Prometheus Operator must also be allowed to read the referenced Secrets.
The operator neither creates nor copies scrape credentials, and does not use
bearerTokenFile; cluster owners are responsible for secure credential delivery
and rotation. When runtime ServiceMonitor creation is enabled, the operator
validates these secure scrape flags at startup on every replica, even if the
optional ServiceMonitor API is absent. To serve secure metrics without a runtime
ServiceMonitor, set --metrics-service-monitor-enabled=false and manage scraping
separately.
For GitOps, use the checked-in config/secure-metrics Kustomize overlay, rather
than applying the plain-HTTP config/prometheus base. It configures the manager
with secure serving, mounts metrics-server-cert, disables the runtime
bootstrapper, and emits the corresponding static ServiceMonitor. The overlay
derives both the ServiceMonitor namespace and its TLS server name from the metrics
Service. Changing namespace or namePrefix in config/default/kustomization.yaml,
or setting them in config/secure-metrics/kustomization.yaml, keeps the monitor
in the Service's namespace with the matching <service>.<namespace>.svc name.
If a parent overlay changes these fields after this overlay has rendered, it must
also update the TLS server name; nested replacements do not run again in the parent.
Before applying it, create
metrics-server-cert (ca.crt, tls.crt, and tls.key) and
hyperfleet-metrics-scrape-credentials (token) in the operator namespace.
The secure overlay sets scheme: https, CA verification, and Secret-backed
bearer authorization. It deliberately does not set insecureSkipVerify or
client TLS credentials: the manager uses bearer-token authentication, not mTLS.
- README — observability endpoints quick reference
- HyperFleet metrics standard
- HyperFleet health-endpoints standard