Skip to content

fix: tenant isolation in event queries and billing correctness - #100

Open
MahediBuilds wants to merge 8 commits into
ScrawnDotDev:devfrom
MahediBuilds:fix/tenant-isolation-and-billing-correctness
Open

MahediBuilds wants to merge 8 commits into
ScrawnDotDev:devfrom
MahediBuilds:fix/tenant-isolation-and-billing-correctness

Conversation

@MahediBuilds

@MahediBuilds MahediBuilds commented Oct 13, 2026 •

Copy link
Copy Markdown

Summary

Fixes a cross-tenant data leak in QueryEvents, ClickHouse mode being unable to authenticate any request, and several billing bugs that caused usage to be skipped or charged twice.

Tenant isolation: QueryEvents leaked every project's events

handleQueryEvents received the auth context but never used it, so any API key (including a scrn_test_ key from another project) could list and aggregate usage events from all projects. Both the Postgres and ClickHouse dialects now always AND the caller's filters with project_id, plus mode for production/test keys. Dashboard keys still see both modes of their own project.

ClickHouse mode could not authenticate anything

server.ts only set up the Postgres connection when STORAGE_ADAPTER=postgres, but API keys, users, sessions, tags and expressions always live in Postgres. In ClickHouse mode every gRPC call failed with DATABASE_URL is not defined. Postgres is now always initialized.

Data queries on users / sessions were always empty

Data queries are dashboard-only, and dashboard keys have mode = null, so the filter became mode = NULL, which never matches. A null mode now means "all modes".

Deleting a project failed when it had tags or expressions

tags and expressions reference projects but were never deleted first, so the request hit a foreign key violation and returned 500.

Billing correctness

  • Test payments hid production usage. There was a single users.last_billed_timestamp shared by both modes. The last billed point is now derived per (project, user, mode) as MAX(billed_upto) over succeeded sessions.

  • Paying an older session moved the window backwards, which re-billed usage that had already been paid. Taking the MAX makes the window monotonic. users.last_billed_timestamp is also only ever advanced now (GREATEST).

  • Late or backdated events were never billed. The window was keyed on the client-reported timestamp. It now uses the server-assigned ingested_timestamp over [lastBilled, before), so each event lands in exactly one checkout.

  • ClickHouse re-billed all history on a Postgres hiccup. The last-billed lookup swallowed errors and fell back to an unbounded query. Errors now propagate.

  • ClickHouse double-counted retried events. ReplacingMergeTree only dedupes on background merges and keeps the latest duplicate. ClickHouse ingestion now claims each idempotency key atomically in a new Postgres table, event_idempotency_keys, inside the same transaction as the insert. Only one of several concurrent retries is accepted, and a failed insert rolls the claim back. Reads also use FINAL as a backstop.

  • Events committed during checkout pricing were lost. An event stamped before the cutoff but committed after pricing read the events was excluded from that checkout, and then from every later window. This existed on main too. Ingestion and pricing now follow one ordering rule:

    • Ingestion takes a SHARE lock on the user row, then stamps the event with the database clock (clock_timestamp()).
    • Pricing takes an UPDATE lock on the same row, then reads its cutoff from the database clock.
    • The cutoff pricing returns is what gets stored as the session's billed_upto.

    As a result, anything that commits after pricing has read is stamped after the cutoff and lands in the next window. Database time is used on both sides, so clock skew between server instances can't reorder events.

    • Pricing does all its Postgres reads on that one transaction, including the ClickHouse path's last-billed lookup. Checkout also loads the payment client before opening its transaction. Otherwise concurrent checkouts could use up the connection pool and wait on each other forever.

Tests

  • queryEvents.test.ts seeded data in one project and queried it with a key from another, so it only passed because of the leak (and on main it fails during seeding due to a project-name collision with the shared fixture). It now uses a key scoped to the seeded project and adds isolation cases: another project's events are never returned, and test-mode keys only see test-mode events in lists and aggregations.
  • dataQuery.test.ts used a test-role key, which the endpoint rejects, and had the same project-name collision. It now uses a dashboard key, which exercises the null-mode fix.
  • New billing.test.ts runs against both adapters and covers:
    • only usage since the last paid checkout is billed
    • late or backdated events are billed
    • test payments don't affect production
    • paying an older checkout doesn't reopen a billed window
    • an event ingested while a checkout holds the lock waits, and is still billed after that checkout is paid
    • of two concurrent events with the same idempotency key, exactly one is accepted and it is billed once
    • 25 concurrent checkouts (more than the 15-connection pool) all complete; pricing never needs a second pooled connection
  • The ClickHouse duplicate-idempotency-key test was previously skipped; it now runs on both adapters.

Results (vitest --run):

Adapter Passed Failed
Postgres 108 1 test + 1 suite setup
ClickHouse 108 1 test + 1 suite setup

The two remaining failures also fail on main and are unrelated to this change:

  • auth.test.ts › 1.6 rejects expired key: gets the webhook-endpoint error instead of "expired".
  • createAPIKey.test.ts: setup inserts a key for TEST_PROJECT_ID before that project exists (foreign key violation).

tsgo type-check passes.

Notes for reviewers

  • Billing semantics change: the window is now keyed on ingestion time instead of reported time. Usage that was previously skipped (late or backdated events) will show up in the next checkout. As a one-off edge case during rollout, an event that was reported in the future but ingested before an already-paid billed_upto won't be picked up.
  • Not addressed here: if a user has two pending checkout sessions at once (possible after the 24h reuse window), both are priced from the same starting point, so paying both still charges the overlap twice. Fixing that needs a decision on how to expire or supersede old sessions.
  • Schema change: adds the Postgres table event_idempotency_keys (run bunx drizzle-kit push). It's only written in ClickHouse mode. Existing ClickHouse events aren't backfilled, so a retry of an event ingested before this deploy would still be accepted. Backfilling from ClickHouse closes that if it matters.
  • Locking cost: ingestion takes a SHARE lock on the user row. These locks don't block each other, so ingestion stays concurrent. They only wait while a checkout for that same user is being priced, which takes a few milliseconds.
  • Remaining ClickHouse edge: if the ClickHouse insert succeeds but the Postgres commit then fails, the claim is lost while the row remains. FINAL still prevents double-counting inside one window.

SteakFisher and others added 6 commits July 7, 2026 18:52
handleQueryEvents received the auth context but never applied it, so any
API key could list and aggregate usage events from every project. Both
query dialects now always AND the caller's filters with project_id and,
for production/test keys, mode. ClickHouse queries also read with FINAL so
unmerged duplicates are not returned twice.

The QueryEvents tests seeded data in one project and queried it with a key
from another, which only passed because of this leak. They now use a key
scoped to the seeded project and cover cross-project and cross-mode
isolation.
API keys, projects, users, sessions, tags and expressions live in Postgres
even when events are stored in ClickHouse, but the connection was only
created for the postgres adapter. In ClickHouse mode every authenticated
request failed with "DATABASE_URL is not defined".
Data queries are dashboard-only, and dashboard keys have a null mode, so
the users and sessions tables were filtered with `mode = NULL` and always
came back empty. A null mode now means "all modes".

The DataQuery tests used a test-role key (rejected by the endpoint) and a
project name that collides with the shared fixture; they now use a
dashboard key scoped to the seeded project.
Both tables reference projects, so deleting any project that had tags or
expressions failed with a foreign key violation and returned 500.
- Derive the last billed point per (project, user, mode) from the latest
  succeeded checkout session instead of the shared
  users.last_billed_timestamp. A test-mode payment no longer hides
  production usage, and paying an older session late can no longer move
  the billing window backwards.
- Key the billing window on the server-assigned ingested timestamp,
  [lastBilled, before), instead of the client-reported timestamp, so late
  or backdated events are billed in the next checkout instead of never.
- Keep users.last_billed_timestamp monotonic (GREATEST) for display.
- ClickHouse: propagate errors when looking up the last billed point
  instead of falling back to an unbounded query that re-billed all
  history, read with FINAL, and reject duplicate idempotency keys at
  insert time like the Postgres unique constraint does.
@greptile-apps

greptile-apps Bot commented Oct 13, 2026 •

Copy link
Copy Markdown

RetriggerConfidence Score: 5/5

[Critical impact] No unresolved blocking findings remain in this review; the previously reported ClickHouse connection-pool stall is addressed.

Summary

The PR scopes event queries to the caller’s project and mode, fixes database setup and project cleanup, and corrects billing windows and retry handling.

Since the last review:

  • Event queries stay within the caller’s project and mode.
  • Checkout windows follow each mode’s database-stamped event times.
  • ClickHouse event retries claim each idempotency key once.
  • Project deletion removes its dependent records first.
  • Dashboard data queries include both modes of their project.
  • The server starts Postgres in ClickHouse mode too.

MahediBuilds explicitly deferred overlapping pending checkouts, which can still charge the same usage twice, because fixing the overlap requires a decision on expiring or superseding old sessions. MahediBuilds also acknowledged that existing ClickHouse events are not backfilled into the new idempotency-key table, so retries of pre-deployment events can still be accepted, and that a failed Postgres commit after a successful ClickHouse insert can lose the key claim. Those acknowledged cases remain outside this change’s guarantees.

Reviews (3) · Last reviewed commit: "fix(billing): reuse the pricing transact..." · Reviewed by Greptile

Comment thread src/storage/adapter/clickhouse/utils.ts Resolved
Two races remained in the billing window:

- An event stamped before a checkout's cutoff but committed after pricing
  read the events was excluded from that checkout and, once it was paid,
  from every later window.
- In ClickHouse, the duplicate idempotency check was check-then-insert, so
  concurrent retries could both be accepted; ReplacingMergeTree keeps the
  later copy, whose newer ingested timestamp made the event billable again
  in the next window.

Ingestion now takes a SHARE lock on the user row and stamps events with
the database clock after acquiring it; pricing takes an UPDATE lock on the
same row and reads its cutoff from the database clock. Any event that
commits after pricing has read is therefore stamped after the cutoff and
lands in the next window. The cutoff returned by pricing is stored as the
session's billed_upto, and database time is used on both sides so clock
skew between server instances cannot reorder them.

ClickHouse ingestion runs inside the same Postgres transaction and claims
its idempotency keys atomically in a new event_idempotency_keys table
before inserting, so only one of several concurrent retries is accepted.
A failed insert rolls the claim back.
Comment thread src/services/pricingService.ts Resolved
Checkout pricing holds a pooled Postgres connection for its transaction,
but in ClickHouse mode the last-billed lookup ignored that transaction and
requested a second connection. With as many concurrent checkouts as the
pool has connections, every request waited on another and none finished.

The transaction is now passed through the ClickHouse pricing handlers and
used for the lookup. The checkout handler also loads the payment provider
client before opening its own transaction, since a cache miss there reads
project metadata from the pool.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants