Skip to content

feat(sonarqube): collect historical project metrics and Grafana trends#8983

Open
jbsmith7741 wants to merge 4 commits into
apache:mainfrom
jbsmith7741:feat-sonar-8941
Open

feat(sonarqube): collect historical project metrics and Grafana trends#8983
jbsmith7741 wants to merge 4 commits into
apache:mainfrom
jbsmith7741:feat-sonar-8941

Conversation

@jbsmith7741

Copy link
Copy Markdown
Contributor

Summary

Closes #8941.

DevLake's SonarQube plugin previously stored only point-in-time metrics. Each sync overwrote prior values, so there was no way to answer questions like "how has coverage trended over the past year?" This PR adds project-level historical collection from SonarQube's measures/search_history and project_analyses/search APIs, persists them through the standard Raw → Tool → Domain pipeline, and adds Grafana trend panels that honor the time picker.

Data pipeline

flowchart LR
    SQ[SonarQube API] --> C1[CollectProjectMetricsHistory]
    SQ --> C2[CollectProjectAnalyses]
    C1 --> R1["_raw_sonarqube_api_project_metrics_history"]
    C2 --> R2["_raw_sonarqube_api_project_analyses"]
    R1 --> T1["_tool_sonarqube_project_metrics_history"]
    R2 --> T2["_tool_sonarqube_project_analyses"]
    T1 --> D["cq_project_metrics_history"]
    T2 --> D
Loading

SonarQube plugin

  • Five new subtasks: collect/extract project metrics history, collect/extract project analyses, convert to domain
  • New tool tables: _tool_sonarqube_project_metrics_history, _tool_sonarqube_project_analyses
  • New domain table: cq_project_metrics_history (coverage, ncloc, bugs, vulnerabilities, code_smells, ratings, complexity, duplication, etc.)
  • Uses NewStatefulApiCollector with sync policy TimeAfter for initial backfill and incremental runs
  • Metrics collected: coverage, ncloc, bugs, reliability_rating, code_smells, sqale_rating, complexity, cognitive_complexity, vulnerabilities, security_rating, security_hotspots, duplicated_lines_density

Connection validation

  • Reject Global (sqa_) and Project (sqp_) analysis tokens on SonarQube Server at connection test — they authenticate but cannot read metrics via the Web API
  • SonarCloud skips prefix validation (token formats differ)
  • Unit tests in connection_test.go

Grafana dashboards

  • Collapsed Historical Trends row added to SonarQube Server and SonarQube Cloud dashboards (MySQL and PostgreSQL variants)
  • Four time-series panels: coverage, bugs & vulnerabilities, code smells, duplication
  • Panels query cq_project_metrics_history and use $__timeFilter(pmh.analysis_date)
  • Existing snapshot stat panels are unchanged

Does this close any open issues?

Closes #8941

Screenshots

SonarQube_Historic_Metrics

Other Information

How to verify

  1. Configure a SonarQube connection with a User token (squ_ prefix on Server).
  2. Run a blueprint sync for a project with analysis history.
  3. Confirm data in _tool_sonarqube_project_metrics_history, _tool_sonarqube_project_analyses, and cq_project_metrics_history.
  4. Open the SonarQube Server or SonarQube Cloud dashboard in Grafana, select a project, set time range to Last 6 months, and expand Historical Trends.
  5. Connection test: sqa_ / sqp_ tokens on Server should fail with a clear message; squ_ should succeed.
  6. Run unit tests: cd backend && go test ./plugins/sonarqube/models/... -run TestValidateUserTokenPrefix

Add collector, extractor, and convertor subtasks that pull historical
metric snapshots from the SonarQube measures/search_history and
project_analyses/search APIs into a new cq_project_metrics_history
domain table, enabling trend analysis for coverage, bugs, code smells,
complexity, vulnerabilities, and other quality indicators over time.

Signed-off-by: Joshua Smith <jbsmith7741@gmail.com>
  * Reject Global and Project analysis tokens on SonarQube Server before authentication validate runs.
  * Skip prefix validation for SonarCloud endpoints where token prefixes differ.
  * Add collapsed "Historical Trends" row with coverage, bugs, vulnerabilities, code smells, and duplication time-series panels to both Server and Cloud dashboards
  * Panels query cq_project_metrics_history and honor the Grafana time picker via $__timeFilter

Signed-off-by: Joshua Smith <jbsmith7741@gmail.com>
  * Align struct fields and ApiCollectorArgs literals to satisfy golangci-lint gofmt checks.
  * Updates cq_project_metrics_history and SonarQube collector tasks flagged in CI.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Feature][SonarQube] Collect historical project-level metrics from SonarQube

1 participant