Value Metric Updates - #181
Conversation
- Updated docs/experiments.mdx Mintlify-Source: dashboard-editor
Confidence Score: 4/5The denominator contradiction should be resolved before merging because it can mislead customers interpreting Value Metric experiment results. The page now prescribes both exposed and observed-value populations for overlapping Value Metric configurations, while an escaped table heading adds a smaller rendering defect. Files Needing Attention: docs/experiments.mdx
|
| Filename | Overview |
|---|---|
| docs/experiments.mdx | The new Value Metric explanation conflicts with surrounding denominator guidance, and one pricing-table heading will render literal emphasis markers. |
Reviews (1): Last reviewed commit: "Updated mintlify pages" | Re-trigger Greptile
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
| Lift is the percentage difference between the control and variant(s) metrics. $$Lift= { (variant \,group\,rate - control \,group\,rate) \over (control \,group\,rate)}$$ | ||
|
|
||
| The "group rate" in this formula is calculated differently for each metric type. In every case, the denominator is **the number of users exposed to that variant** (i.e., users who received the `$experiment_started` event for that variant): | ||
| The "group rate" in this formula is calculated differently for each metric type. For Total Events, Total Sessions, and Rate Metrics, the denominator is the **number of users exposed** to that variant. Value Metrics are the exception; they use a different denominator, detailed below. |
There was a problem hiding this comment.
When readers evaluate a Value Metric, this exception prescribes observed-value users while the surrounding guidance says exposures normalize lift and all group-identifier calculations, leaving customers unable to determine the correct denominator for interpreting lift and significance.
Knowledge Base Used: Feature Flags & Experiments
Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!
| | ------------ | ---------------- | | ||
| | Small (< 100k) | 50-100% | | ||
|
|
||
| | **MTU bucket** | \*\*Estimated MEU (% MTU) \*\* | |
There was a problem hiding this comment.
The escaped asterisks render as literal ** characters rather than bolding the second heading, leaving the MEU table header visibly inconsistent with the first column.
| | **MTU bucket** | \*\*Estimated MEU (% MTU) \*\* | | |
| | **MTU bucket** | **Estimated MEU (% MTU)** | |
Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!
| - **Variance:** Calculated using Bernoulli distribution: p × (1-p) | ||
| - **Example:** If 25 out of 100 exposed users convert, group rate = 0.25 (25%). Note that this denominator is exposures — not funnel entrants. This is different from the funnel conversion rate you see in the Totals or Chart tabs, which uses funnel entrants as the denominator. | ||
|
|
||
| **Value Metrics (Averages, Sums):** |
There was a problem hiding this comment.
The Value Metrics section is clearer, but could we add one short sentence defining what counts as an “observed value”? Readers may not know whether that means only users who triggered the underlying event, and it would help to explicitly say how missing or zero values are handled.
| - **Variance:** Calculated from the distribution of individual property values, over that same observed-value population | ||
| - **Example:** If the treatment group spent \$5,000 total from 20 users who had a value (out of 100 exposed), the group rate = \$250 average per user with an observed value | ||
|
|
||
| **Why This Matters:** Value metrics report the average based only on users who had an observed value for the metric, not all exposed users, since there's no natural zero for someone who never triggered the underlying event. The exposed count still matters; it's used downstream in the standard error and confidence interval calculation, so overall reach still factors into how confident you can be in the result, just not into the group rate or variance themselves. |
There was a problem hiding this comment.
This is a helpful clarification, but it may still read as if exposures are no longer part of the Value Metrics calculation at all. Could we make the distinction explicit that exposures are excluded from the group rate/variance, but still used in downstream standard error and confidence interval calculations?
|
|
||
| Lift is the percentage difference between the control and variant(s) metrics. | ||
| $Lift= { (variant \,group\,rate - control \,group\,rate) \over (control \,group\,rate)}$ | ||
| Lift is the percentage difference between the control and variant(s) metrics. $$Lift= { (variant \,group\,rate - control \,group\,rate) \over (control \,group\,rate)}$$ |
There was a problem hiding this comment.
The lift explanation is more explicit now, but the formula and the metric-type-specific denominator rules are spread across several paragraphs. Could we consolidate this a bit so readers don’t miss that lift behaves differently depending on the metric type?
| @@ -384,9 +385,10 @@ Benjamini-Hochberg controls the **false discovery rate**, the share of your flag | |||
| This correction operates on the values in your breakdown, which is a different axis from the [Multiple Testing Correction](#bonferroni-correction) you apply across metrics and variants. That means adding more breakdown values doesn't inflate your false positive rate. | |||
|
|
|||
| ### When do we say the Experiment is ready to review? | |||
There was a problem hiding this comment.
“Experiment is ready to review” could be interpreted as statistically ready, not just setup-complete. Could we clarify that this banner reflects configuration completion rather than a recommendation to end the experiment?
2 participants
See Slack thread for details.
https://mixpanel.slack.com/archives/C0547DQTVCN/p1786744670689369
Mintlify
0 threads from 0 users in Mintlify