Add read-only HA replication checks - #43
Draft
viliakov wants to merge 3 commits into
Draft
Conversation
Inspect HDFS, Elasticsearch, Kafka and ClickHouse through namespace-scoped Kubernetes discovery and fixed read-only exec queries. Require complete evidence, reject observations whose Kubernetes membership changes, and support bounded waits with a stable healthy period and JSON output. Keep replication checks independent of backup configuration. Include dedicated Job RBAC and a SUSE BCI packaging example. Document that this is sampled replication evidence, not a node-removal safety certificate; live cluster validation remains outstanding. Tracking: StackVista/stackstate#501
Use chart name and component labels instead of requiring a Helm release. Distinguish the HDFS NameNode from its SecondaryNameNode even though both use a namenode container. Clear the inherited broker JMX settings for Kafka CLI invocations while preserving authentication options, and alias the ClickHouse queue subquery. ClickHouse retains last_queue_update_exception after successful queue updates; report it as historical context instead of blocking recovered replicas. Current coordination errors and backlog still fail. Verified against StorageReplicatedMergeTree::queueUpdatingTask in ClickHouse v26.5.1.882-stable. Full Go tests and golangci-lint pass. Read-only validation in stackstate-nightly reports HDFS, Elasticsearch and ClickHouse healthy; Kafka correctly flags three perf-test topics with one assigned replica. Added namespace/component discovery, SecondaryNameNode, JMX isolation and recovered-exception regression coverage.
Discover the namespace's ZooKeeper ensemble using chart labels and query every member with read-only mntr. Require at least three voters, matching configured voting counts, one leader and all expected synchronized followers. Recheck the leader after sampling to detect elections without depending on Kubernetes readiness. Enable the check by default and document the supported chart layout and sampled-maintenance boundary. Cover incomplete membership, lag, elections, unavailable metrics and malformed responses. Full Go tests and golangci-lint pass; nightly ZooKeeper passed a live stable-period wait with three voters.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds
sts-backup replication checkfor HDFS, Elasticsearch, Kafka, ClickHouse and ZooKeeper, with table/JSON output, bounded waiting and nonzero exit for degraded or unverified replication. Discovers one installation per namespace using chart name/component labels; no release name or backup configuration is required. Includes a Kubernetes Job/RBAC example and a SUSE BCI container recipe.This reports replication status and does not certify node-removal safety. Controlled recovery validation remains outstanding.
Tracking: https://github.com/StackVista/stackstate/issues/501
Validation: full Go tests and golangci-lint pass. Read-only nightly validation includes a successful ZooKeeper stable-period wait. The combined run checks all five components and flags HDFS block problems and three Kafka test topics with one replica. Initial static build and local nonroot container smoke test also passed.