Judging how balanced a Kafka Cluster is can be difficult. Looking at charts in Graphana doesn't always give a good feel for the distribution of load across a cluster. It also needs human operators to look at several charts to synthesis a view of the cluster.
Cruise Control has its own built in measure of "balancedness". This is a single value based on subtracting weighted values from 100 for each Cruise Control Goal (a Java class responsible for a specific aspect of the Kafka cluster - such as disk usage) that is not currently met. So the balancedness score is a weighted penalty model, not a statistical measure of load distribution. It tells you "which goals are violated and how important are they?" rather than "how evenly is load spread?". Because it is highly dependant on the configuration the balancedness score is not comparable across clusters. It also obviously needs Cruise Control to be running in order for it to be available.
It would be better to have a more objective view of a cluster's balance in the Console UI. We have access to cluster metrics so we could use these to visualise the distribution of load across the cluster.
We should:
- Investigate what metrics would be best used to judge "balance".
- Investigate how we could produce glanceable summary views for cluster balance.
- Investigate how we could show a more detailed cluster balance visualisation.
- Use that detailed cluster visualization to allow comparison of before and after load distributions from manual rebalance optimization proposals.
Judging how balanced a Kafka Cluster is can be difficult. Looking at charts in Graphana doesn't always give a good feel for the distribution of load across a cluster. It also needs human operators to look at several charts to synthesis a view of the cluster.
Cruise Control has its own built in measure of "balancedness". This is a single value based on subtracting weighted values from 100 for each Cruise Control Goal (a Java class responsible for a specific aspect of the Kafka cluster - such as disk usage) that is not currently met. So the balancedness score is a weighted penalty model, not a statistical measure of load distribution. It tells you "which goals are violated and how important are they?" rather than "how evenly is load spread?". Because it is highly dependant on the configuration the balancedness score is not comparable across clusters. It also obviously needs Cruise Control to be running in order for it to be available.
It would be better to have a more objective view of a cluster's balance in the Console UI. We have access to cluster metrics so we could use these to visualise the distribution of load across the cluster.
We should: