Repository navigation
fix: tenant isolation in event queries and billing correctness - #100
MahediBuilds wants to merge 8 commits into
Conversation
handleQueryEvents received the auth context but never applied it, so any API key could list and aggregate usage events from every project. Both query dialects now always AND the caller's filters with project_id and, for production/test keys, mode. ClickHouse queries also read with FINAL so unmerged duplicates are not returned twice. The QueryEvents tests seeded data in one project and queried it with a key from another, which only passed because of this leak. They now use a key scoped to the seeded project and cover cross-project and cross-mode isolation.
API keys, projects, users, sessions, tags and expressions live in Postgres even when events are stored in ClickHouse, but the connection was only created for the postgres adapter. In ClickHouse mode every authenticated request failed with "DATABASE_URL is not defined".
Data queries are dashboard-only, and dashboard keys have a null mode, so the users and sessions tables were filtered with `mode = NULL` and always came back empty. A null mode now means "all modes". The DataQuery tests used a test-role key (rejected by the endpoint) and a project name that collides with the shared fixture; they now use a dashboard key scoped to the seeded project.
Both tables reference projects, so deleting any project that had tags or expressions failed with a foreign key violation and returned 500.
- Derive the last billed point per (project, user, mode) from the latest succeeded checkout session instead of the shared users.last_billed_timestamp. A test-mode payment no longer hides production usage, and paying an older session late can no longer move the billing window backwards. - Key the billing window on the server-assigned ingested timestamp, [lastBilled, before), instead of the client-reported timestamp, so late or backdated events are billed in the next checkout instead of never. - Keep users.last_billed_timestamp monotonic (GREATEST) for display. - ClickHouse: propagate errors when looking up the last billed point instead of falling back to an unbounded query that re-billed all history, read with FINAL, and reject duplicate idempotency keys at insert time like the Postgres unique constraint does.
|
Two races remained in the billing window: - An event stamped before a checkout's cutoff but committed after pricing read the events was excluded from that checkout and, once it was paid, from every later window. - In ClickHouse, the duplicate idempotency check was check-then-insert, so concurrent retries could both be accepted; ReplacingMergeTree keeps the later copy, whose newer ingested timestamp made the event billable again in the next window. Ingestion now takes a SHARE lock on the user row and stamps events with the database clock after acquiring it; pricing takes an UPDATE lock on the same row and reads its cutoff from the database clock. Any event that commits after pricing has read is therefore stamped after the cutoff and lands in the next window. The cutoff returned by pricing is stored as the session's billed_upto, and database time is used on both sides so clock skew between server instances cannot reorder them. ClickHouse ingestion runs inside the same Postgres transaction and claims its idempotency keys atomically in a new event_idempotency_keys table before inserting, so only one of several concurrent retries is accepted. A failed insert rolls the claim back.
Checkout pricing holds a pooled Postgres connection for its transaction, but in ClickHouse mode the last-billed lookup ignored that transaction and requested a second connection. With as many concurrent checkouts as the pool has connections, every request waited on another and none finished. The transaction is now passed through the ClickHouse pricing handlers and used for the lookup. The checkout handler also loads the payment provider client before opening its own transaction, since a cache miss there reads project metadata from the pool.
Summary
Fixes a cross-tenant data leak in
QueryEvents, ClickHouse mode being unable to authenticate any request, and several billing bugs that caused usage to be skipped or charged twice.Tenant isolation:
QueryEventsleaked every project's eventshandleQueryEventsreceived the auth context but never used it, so any API key (including ascrn_test_key from another project) could list and aggregate usage events from all projects. Both the Postgres and ClickHouse dialects now alwaysANDthe caller's filters withproject_id, plusmodefor production/test keys. Dashboard keys still see both modes of their own project.ClickHouse mode could not authenticate anything
server.tsonly set up the Postgres connection whenSTORAGE_ADAPTER=postgres, but API keys, users, sessions, tags and expressions always live in Postgres. In ClickHouse mode every gRPC call failed withDATABASE_URL is not defined. Postgres is now always initialized.Data queries on
users/sessionswere always emptyData queries are dashboard-only, and dashboard keys have
mode = null, so the filter becamemode = NULL, which never matches. A null mode now means "all modes".Deleting a project failed when it had tags or expressions
tagsandexpressionsreferenceprojectsbut were never deleted first, so the request hit a foreign key violation and returned 500.Billing correctness
Test payments hid production usage. There was a single
users.last_billed_timestampshared by both modes. The last billed point is now derived per(project, user, mode)asMAX(billed_upto)over succeeded sessions.Paying an older session moved the window backwards, which re-billed usage that had already been paid. Taking the MAX makes the window monotonic.
users.last_billed_timestampis also only ever advanced now (GREATEST).Late or backdated events were never billed. The window was keyed on the client-reported timestamp. It now uses the server-assigned
ingested_timestampover[lastBilled, before), so each event lands in exactly one checkout.ClickHouse re-billed all history on a Postgres hiccup. The last-billed lookup swallowed errors and fell back to an unbounded query. Errors now propagate.
ClickHouse double-counted retried events.
ReplacingMergeTreeonly dedupes on background merges and keeps the latest duplicate. ClickHouse ingestion now claims each idempotency key atomically in a new Postgres table,event_idempotency_keys, inside the same transaction as the insert. Only one of several concurrent retries is accepted, and a failed insert rolls the claim back. Reads also useFINALas a backstop.Events committed during checkout pricing were lost. An event stamped before the cutoff but committed after pricing read the events was excluded from that checkout, and then from every later window. This existed on
maintoo. Ingestion and pricing now follow one ordering rule:SHARElock on the user row, then stamps the event with the database clock (clock_timestamp()).UPDATElock on the same row, then reads its cutoff from the database clock.billed_upto.As a result, anything that commits after pricing has read is stamped after the cutoff and lands in the next window. Database time is used on both sides, so clock skew between server instances can't reorder events.
Tests
queryEvents.test.tsseeded data in one project and queried it with a key from another, so it only passed because of the leak (and onmainit fails during seeding due to a project-name collision with the shared fixture). It now uses a key scoped to the seeded project and adds isolation cases: another project's events are never returned, and test-mode keys only see test-mode events in lists and aggregations.dataQuery.test.tsused a test-role key, which the endpoint rejects, and had the same project-name collision. It now uses a dashboard key, which exercises the null-mode fix.billing.test.tsruns against both adapters and covers:Results (
vitest --run):The two remaining failures also fail on
mainand are unrelated to this change:auth.test.ts› 1.6 rejects expired key: gets the webhook-endpoint error instead of "expired".createAPIKey.test.ts: setup inserts a key forTEST_PROJECT_IDbefore that project exists (foreign key violation).tsgotype-check passes.Notes for reviewers
billed_uptowon't be picked up.event_idempotency_keys(runbunx drizzle-kit push). It's only written in ClickHouse mode. Existing ClickHouse events aren't backfilled, so a retry of an event ingested before this deploy would still be accepted. Backfilling from ClickHouse closes that if it matters.SHARElock on the user row. These locks don't block each other, so ingestion stays concurrent. They only wait while a checkout for that same user is being priced, which takes a few milliseconds.FINALstill prevents double-counting inside one window.