Skip to content

perf(analytics): cache the project profile per project per day - #292

Merged
mountainowl merged 2 commits into
mainfrom
perf/cache-project-profile
Oct 22, 2026
Merged

mountainowl merged 2 commits into
mainfrom
perf/cache-project-profile

Conversation

@mountainowl

Copy link
Copy Markdown
Owner

Requested by Ajay Singh · project thread

What changed

  • Before: with analytics on, every review rebuilt the project profile from scratch. That meant a git ls-tree of the checkout, reading up to 100 manifests, and one SCM metadata API call, even when the same project had been reviewed minutes earlier.
  • After: the stack scan and the SCM metadata are cached per project per UTC day. Later reviews that day reuse them. The events sent to PostHog are unchanged.
  • How:
    • Each review runs in its own worker process, so the cache lives in a profiles table in the existing analytics_projects.sqlite, keyed by the random project ID.
    • A failed metadata read (metadata_status: unavailable) isn't cached, so the next review retries it.
    • The declared domain and project_type still come from config on every review, so a config change applies right away.

Verification

  • tests/test_profile_cache.py runs the real collect_profile against a real git checkout and counts stack scans and metadata reads.
  • On main, 3 reviews on one day did 3 scans and 3 reads. With this change they do 1 and 1, and a new day refreshes the cache.
  • The tests also cover separate projects, a failed metadata read being retried, and a declared-profile change applying the same day.
  • The full suite on Python 3.14 gives 1932 passed. The only failures are the 2 container-only test_ui_export failures, which also fail on main.
  • ruff check, ruff format --check and cz check pass.

Generated by Claude Code

Every review ran a git ls-tree, read up to 100 manifests and made an SCM
metadata call to build the analytics project profile. The scan and the
metadata are now cached per project per UTC day in the existing
analytics_projects.sqlite, since each review runs in its own worker
process. A failed metadata read is not cached, and the declared domain and
project type are still read from config on every review.
@mountainowl mountainowl self-assigned this Oct 20, 2026
@mountainowl
mountainowl merged commit 2220593 into main Oct 22, 2026
5 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant