feat(sonarqube): collect historical project metrics and Grafana trends#8983
Open
jbsmith7741 wants to merge 4 commits into
Open
feat(sonarqube): collect historical project metrics and Grafana trends#8983jbsmith7741 wants to merge 4 commits into
jbsmith7741 wants to merge 4 commits into
Conversation
Add collector, extractor, and convertor subtasks that pull historical metric snapshots from the SonarQube measures/search_history and project_analyses/search APIs into a new cq_project_metrics_history domain table, enabling trend analysis for coverage, bugs, code smells, complexity, vulnerabilities, and other quality indicators over time. Signed-off-by: Joshua Smith <jbsmith7741@gmail.com>
* Reject Global and Project analysis tokens on SonarQube Server before authentication validate runs. * Skip prefix validation for SonarCloud endpoints where token prefixes differ.
* Add collapsed "Historical Trends" row with coverage, bugs, vulnerabilities, code smells, and duplication time-series panels to both Server and Cloud dashboards * Panels query cq_project_metrics_history and honor the Grafana time picker via $__timeFilter Signed-off-by: Joshua Smith <jbsmith7741@gmail.com>
* Align struct fields and ApiCollectorArgs literals to satisfy golangci-lint gofmt checks. * Updates cq_project_metrics_history and SonarQube collector tasks flagged in CI.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Closes #8941.
DevLake's SonarQube plugin previously stored only point-in-time metrics. Each sync overwrote prior values, so there was no way to answer questions like "how has coverage trended over the past year?" This PR adds project-level historical collection from SonarQube's
measures/search_historyandproject_analyses/searchAPIs, persists them through the standard Raw → Tool → Domain pipeline, and adds Grafana trend panels that honor the time picker.Data pipeline
flowchart LR SQ[SonarQube API] --> C1[CollectProjectMetricsHistory] SQ --> C2[CollectProjectAnalyses] C1 --> R1["_raw_sonarqube_api_project_metrics_history"] C2 --> R2["_raw_sonarqube_api_project_analyses"] R1 --> T1["_tool_sonarqube_project_metrics_history"] R2 --> T2["_tool_sonarqube_project_analyses"] T1 --> D["cq_project_metrics_history"] T2 --> DSonarQube plugin
_tool_sonarqube_project_metrics_history,_tool_sonarqube_project_analysescq_project_metrics_history(coverage, ncloc, bugs, vulnerabilities, code_smells, ratings, complexity, duplication, etc.)NewStatefulApiCollectorwith sync policyTimeAfterfor initial backfill and incremental runsConnection validation
sqa_) and Project (sqp_) analysis tokens on SonarQube Server at connection test — they authenticate but cannot read metrics via the Web APIconnection_test.goGrafana dashboards
cq_project_metrics_historyand use$__timeFilter(pmh.analysis_date)Does this close any open issues?
Closes #8941
Screenshots
Other Information
How to verify
squ_prefix on Server)._tool_sonarqube_project_metrics_history,_tool_sonarqube_project_analyses, andcq_project_metrics_history.sqa_/sqp_tokens on Server should fail with a clear message;squ_should succeed.cd backend && go test ./plugins/sonarqube/models/... -run TestValidateUserTokenPrefix