Skip to content

:hedgehog: PostHog is the leading platform for building self-driving products. Our developer tools – AI observability, analytics, session replay, flags, experiments, error tracking, logs, and more – capture all the context agents need to diagnose problems, uncover opportunities, and ship fixes. Steer it all from Slack, web, desktop, or the MCP.

feat(experiments): show what the experiment can detect in the running time settings (#123698)

Marcel Poelker6566eb7a77c8
diff --git a/frontend/snapshots.yml b/frontend/snapshots.yml
index e5bdaafd7701..ea087fe4d97f 100644
--- a/frontend/snapshots.yml
+++ b/frontend/snapshots.yml
@@ -11252,6 +11252,14 @@ snapshots:
hash: v1.k794b7964.4b4f9908a2552fc97dd30fe179ca38caeb074f454f46ade87b3ca90ef0cd7f70.0GRyu-AtfFknvRJUEg03KJaNlwPBv9-SzOo4hXWpgoY
scenes-app-experiments--experiment-results-row-recording-links-unselectable-metric--light:
hash: v1.k794b7964.08f0ff958ed2d355bc6a0c9f5048f35e7f75b2cd34bacb867dd56f4ae5e4bf93.BNe46IwANTcBPvWEWBrpf78YOLf5cgWoIF1K-cni1J0
+ scenes-app-experiments--experiment-running-time-on-track--dark:
+ hash: v1.k794b7964.afa5d2bc13065d7380ed28208c4e2eabdb1f4c45121ed0208c403ea955b674e6.MO0qXYuaERQK6SXloMRlNHcmGocr0DyiesdR3ZayFtk
+ scenes-app-experiments--experiment-running-time-on-track--light:
+ hash: v1.k794b7964.36487dd075f2235217d9b4fad812b91e9d51f2b22c9500120c81bb0bcc3548c7.0IhzfZMJNdt1pAKXr5f3bvcK9hFC2IvE_EfSlNUcwgQ
+ scenes-app-experiments--experiment-running-time-out-of-reach--dark:
+ hash: v1.k794b7964.76ee3239ae3823c6749b82d2d0f39d9e345f86aec33c8553847b59867ca1993d.RHLG1HcCDNxMpGjZ18XoRArDCNhQC2xDXjeGKXC5aNI
+ scenes-app-experiments--experiment-running-time-out-of-reach--light:
+ hash: v1.k794b7964.9dc2d830c6fc93df43da8d7ca01861af94d85aacaa7ab6ba202ce9d89ff46236.XVTEPecb7FA5Q0n3544d00ftbnOvv5tGWMxl0oR9VeE
scenes-app-experiments--experiment-stopped-with-conclusion--dark:
hash: v1.k794b7964.578ffd24470c381648d034ab3f9454d9f448430ae15d09e34b4e61b6441c3258.aOoEZotWjnQ2NmVYfdoTIz3AF8y1Y8DTCrF9YeavSQk
scenes-app-experiments--experiment-stopped-with-conclusion--light:
diff --git a/frontend/src/scenes/experiments/RunningTimeCalculator/calculations.test.ts b/frontend/src/scenes/experiments/RunningTimeCalculator/calculations.test.ts
index d3a87620eeec..2af8eef2b0cf 100644
--- a/frontend/src/scenes/experiments/RunningTimeCalculator/calculations.test.ts
+++ b/frontend/src/scenes/experiments/RunningTimeCalculator/calculations.test.ts
@@ -1,6 +1,17 @@
import { CachedNewExperimentQueryResponse } from '~/queries/schema/schema-general'
-import { calculateCurrentExposures, calculateDaysElapsed, calculateExposureRate } from './calculations'
+import type { RunningTimeOutlookApi } from 'products/experiments/frontend/generated/api.schemas'
+
+import {
+ buildRunningTimeOutlook,
+ calculateCurrentExposures,
+ calculateDaysElapsed,
+ DetectableEffectCheckpoint,
+ describeCheckpointDay,
+ describeUseButton,
+ formatDetectableEffect,
+ RunningTimeOutlook,
+} from './calculations'
// The sample-size / variance math is verified in the backend
// (products/experiments/backend/test/test_running_time_calculator.py). These tests
@@ -20,20 +31,6 @@ describe('running time calculations', () => {
})
})
- describe('calculateExposureRate', () => {
- it('returns exposures per day', () => {
- expect(calculateExposureRate(1000, 10)).toBe(100)
- })
-
- it('returns null below the minimum elapsed window', () => {
- expect(calculateExposureRate(1000, 0.05)).toBeNull()
- })
-
- it('returns null without exposures', () => {
- expect(calculateExposureRate(null, 10)).toBeNull()
- })
- })
-
describe('calculateDaysElapsed', () => {
it('returns null without a start date', () => {
expect(calculateDaysElapsed(null)).toBeNull()
@@ -45,4 +42,174 @@ describe('running time calculations', () => {
expect(elapsed!).toBeGreaterThan(0)
})
})
+
+ describe('buildRunningTimeOutlook', () => {
+ // Two weeks in at about 143 exposures a day, as the backend projects it.
+ const PROJECTION: RunningTimeOutlookApi = {
+ days_since_launch: 14,
+ exposures_per_day: 2000 / 14,
+ detectable_effect_now: 126.5,
+ upcoming_checkpoints: [
+ { weeks_since_launch: 4, detectable_effect: 89.4 },
+ { weeks_since_launch: 6, detectable_effect: 73 },
+ { weeks_since_launch: 8, detectable_effect: 63.2 },
+ ],
+ reachable_effect: 73,
+ }
+ const outlook = (
+ overrides: Partial<Parameters<typeof buildRunningTimeOutlook>[0]>
+ ): RunningTimeOutlook | null =>
+ buildRunningTimeOutlook({
+ projection: PROJECTION,
+ currentSampleSize: 2000,
+ savedEffect: 30,
+ savedSampleSize: 35556,
+ defaultEffect: 30,
+ enteredEffect: null,
+ selectedKey: null,
+ minimumDetectableEffect: 30,
+ ...overrides,
+ })
+ const rowsOf = (result: RunningTimeOutlook): string[] =>
+ result.checkpoints.map(({ kind, isTarget }) => (isTarget ? `${kind} (target)` : kind))
+
+ it('lays out today, the week marks and the custom row at the saved effect', () => {
+ const result = outlook({})!
+
+ expect(
+ result.checkpoints.map(({ kind, days, exposures, detectableEffect, isTarget }) => [
+ kind,
+ Math.round(days),
+ exposures,
+ detectableEffect,
+ isTarget,
+ ])
+ ).toEqual([
+ ['now', 14, 2000, 126.5, false],
+ ['week', 28, 4000, 89.4, false],
+ ['week', 42, 6000, 73, false],
+ ['week', 56, 8000, 63.2, false],
+ ['custom', 249, 35556, 30, true],
+ ])
+ expect(result.daysToTarget).toBeCloseTo(248.9, 1)
+ })
+
+ it.each([
+ [
+ '"Use" chooses a week mark',
+ { selectedKey: 'week-6' },
+ ['now', 'week', 'week (target)', 'week', 'custom'],
+ 30,
+ 42,
+ ],
+ ['"Use" chooses today', { selectedKey: 'now' }, ['now (target)', 'week', 'week', 'week', 'custom'], 30, 14],
+ [
+ 'the slider moves the custom row',
+ { enteredEffect: 50 },
+ ['now', 'week', 'week', 'week', 'custom (target)'],
+ 50,
+ 89.6,
+ ],
+ ])('keeps the same rows when %s', (_name, overrides, rows, customEffect, daysToTarget) => {
+ const result = outlook(overrides)!
+
+ expect(rowsOf(result)).toEqual(rows)
+ expect(result.checkpoints.find(({ kind }) => kind === 'custom')?.detectableEffect).toEqual(customEffect)
+ expect(result.daysToTarget).toBeCloseTo(daysToTarget, 1)
+ })
+
+ it('reopens on the row whose effect was saved, with the custom row at the default effect', () => {
+ const result = outlook({ savedEffect: 73, savedSampleSize: 5990, minimumDetectableEffect: 73 })!
+
+ expect(rowsOf(result)).toEqual(['now', 'week', 'week (target)', 'week', 'custom'])
+ expect(result.checkpoints.find(({ kind }) => kind === 'custom')?.detectableEffect).toEqual(30)
+ })
+
+ it.each([
+ [80, false],
+ [73, false],
+ [30, true],
+ ])('puts a %s%% target out of reach: %s', (minimumDetectableEffect, isOutOfReach) => {
+ expect(outlook({ minimumDetectableEffect })!.isOutOfReach).toBe(isOutOfReach)
+ })
+
+ it.each([
+ ['no projection', { projection: null }],
+ ['no saved target', { savedSampleSize: null }],
+ ['no exposures so far', { currentSampleSize: null }],
+ ])('gives no outlook without %s', (_name, overrides) => {
+ expect(outlook(overrides)).toBeNull()
+ })
+ })
+
+ describe('describeCheckpointDay', () => {
+ const checkpoint = (overrides: Partial<DetectableEffectCheckpoint>): DetectableEffectCheckpoint => ({
+ key: 'custom',
+ kind: 'custom',
+ days: 18,
+ exposures: 3341,
+ detectableEffect: 30,
+ isTarget: false,
+ ...overrides,
+ })
+
+ beforeEach(() => {
+ jest.useFakeTimers().setSystemTime(new Date('2026-10-27T12:00:00Z'))
+ })
+
+ afterEach(() => {
+ jest.useRealTimers()
+ })
+
+ it.each([
+ ['a week mark', { kind: 'week' as const, days: 14 }, 'week 2'],
+ ['today', { kind: 'now' as const, days: 11.17, exposures: 2000 }, 'day 12'],
+ ['the custom row before it is reached', {}, 'in 7 days'],
+ ['the custom row reached earlier today', { days: 11, exposures: 1970 }, 'day 12'],
+ ])('describes %s', (_name, overrides, description) => {
+ expect(
+ describeCheckpointDay(checkpoint(overrides), {
+ startDate: '2026-10-16T08:00:00Z',
+ currentSampleSize: 2000,
+ })
+ ).toEqual(description)
+ })
+ })
+
+ describe('describeUseButton', () => {
+ it.each([
+ [
+ 'the effect of a week mark as the table shows it',
+ { kind: 'week' as const, detectableEffect: 32.74 },
+ { effect: 32.8, tooltip: 'Set 32.8% as the minimum detectable effect' },
+ ],
+ [
+ 'the custom effect as it is',
+ { kind: 'custom' as const, detectableEffect: 12.345 },
+ { effect: 12.345, tooltip: 'Set 12.345% as the minimum detectable effect' },
+ ],
+ [
+ 'only why an effect over 100% cannot be set',
+ { kind: 'now' as const, detectableEffect: 126.5 },
+ { effect: 127, disabledReason: 'The minimum detectable effect can be at most 100%' },
+ ],
+ ])('offers %s', (_name, overrides, button) => {
+ expect(describeUseButton({ key: 'row', days: 14, exposures: 2000, isTarget: false, ...overrides })).toEqual(
+ button
+ )
+ })
+ })
+
+ describe('formatDetectableEffect', () => {
+ it.each([
+ [63.21, '63.3%'],
+ [30.000000001, '30%'],
+ [8.714, '8.72%'],
+ [100, '100%'],
+ [100.01, '>100%'],
+ [126.49, '>100%'],
+ ])('formats the detectable change %s as %s', (detectableEffect, formatted) => {
+ expect(formatDetectableEffect(detectableEffect)).toEqual(formatted)
+ })
+ })
})
diff --git a/frontend/src/scenes/experiments/RunningTimeCalculator/calculations.ts b/frontend/src/scenes/experiments/RunningTimeCalculator/calculations.ts
index 829a5d07bed4..7d4804fff640 100644
--- a/frontend/src/scenes/experiments/RunningTimeCalculator/calculations.ts
+++ b/frontend/src/scenes/experiments/RunningTimeCalculator/calculations.ts
@@ -2,9 +2,12 @@ import { dayjs } from 'lib/dayjs'
import type { CachedNewExperimentQueryResponse } from '~/queries/schema/schema-general'
-// The target sample lives in the backend (products/experiments/backend/running_time_target.py), reached via
-// GET /experiments/:id/running_time_target, and the manual preview's math via POST
-// /experiments/calculate_running_time. This module keeps only the client-side helpers that read live experiment state.
+import type { RunningTimeOutlookApi } from 'products/experiments/frontend/generated/api.schemas'
+
+// The target sample and its outlook live in the backend (products/experiments/backend/running_time_target.py), reached
+// via GET /experiments/:id/running_time_target, and the manual preview's math via POST
+// /experiments/calculate_running_time. This module keeps the helpers that read live experiment state, rescale the target
+// for an effect being edited, and place that target on the backend's outlook.
// Manual calculator only supports these types (ratio/retention require full baseline data).
export type ManualCalculatorMetricType = 'funnel' | 'mean_count' | 'mean_sum_or_avg'
@@ -28,13 +31,9 @@ export function calculateDaysElapsed(startDate: string | null): number | null {
return dayjs().diff(dayjs(startDate), 'days', true) // fractional days
}
-export function calculateExposureRate(currentExposures: number | null, daysElapsed: number | null): number | null {
- if (!currentExposures || !daysElapsed || daysElapsed < 0.1) {
- return null
- }
-
- return currentExposures / daysElapsed
-}
+// The minimum detectable effects, as a percentage, that the running time settings accept.
+export const MINIMUM_DETECTABLE_EFFECT_MIN = 1
+export const MINIMUM_DETECTABLE_EFFECT_MAX = 100
/**
* The target sample for another minimum detectable effect, by the rule the target endpoint documents: the variances come
@@ -48,3 +47,182 @@ export function rescaleSampleSize(sampleSize: number, fromEffect: number, toEffe
// Round before the ceiling, as the backend does, so that float noise doesn't turn an exact 3,200 into 3,201.
return Math.ceil(Number((sampleSize * (fromEffect / toEffect) ** 2).toFixed(6)))
}
+
+export interface DetectableEffectCheckpoint {
+ key: string
+ kind: 'now' | 'week' | 'custom'
+ // Since launch.
+ days: number
+ // So far for today, projected for the week marks, and the target sample for the custom row.
+ exposures: number
+ // The smallest relative change, as a percentage, that the exposures by then can detect.
+ detectableEffect: number
+ isTarget: boolean
+}
+
+export interface RunningTimeOutlook {
+ // Since launch, to the row the user chose as the target.
+ daysToTarget: number
+ isOutOfReach: boolean
+ // Today, the backend's next week marks and the custom row, by date.
+ checkpoints: DetectableEffectCheckpoint[]
+ // The smallest change detectable within four weeks from today, which the header shows for a target out of reach.
+ reachableEffect: number
+}
+
+/**
+ * Rounded up to three significant figures, because the exposures can't detect a smaller change than the exact value.
+ * Set as the minimum detectable effect, the rounded value needs at most 2% fewer exposures than the row it came from.
+ */
+export function roundUpDetectableEffect(detectableEffect: number): number {
+ if (!(detectableEffect > 0)) {
+ return detectableEffect
+ }
+ const precision = 10 ** (2 - Math.floor(Math.log10(detectableEffect)))
+ // Round before the ceiling, so that float noise doesn't turn an exact 30 into 30.1.
+ return Math.ceil(Number((detectableEffect * precision).toFixed(6))) / precision
+}
+
+/**
+ * The backend's projection of what the experiment can detect over time, as a fixed set of rows: today, the next week
+ * marks, and a custom row for the effect that the slider sets. The slider moves the custom row, and "Use" chooses a row
+ * as the target, so no interaction adds or removes a row. Every row sits where the projection reaches its exposures.
+ */
+export function buildRunningTimeOutlook({
+ projection,
+ currentSampleSize,
+ savedEffect,
+ savedSampleSize,
+ defaultEffect,
+ enteredEffect,
+ selectedKey,
+ minimumDetectableEffect,
+}: {
+ projection: RunningTimeOutlookApi | null
+ currentSampleSize: number | null
+ // The effect the backend sized its target for, and that target.
+ savedEffect: number | null
+ savedSampleSize: number | null
+ defaultEffect: number
+ // The last effect from the slider or the input, null until the user changes it.
+ enteredEffect: number | null
+ // The row the user chose with "Use", null when the custom row is the target.
+ selectedKey: string | null
+ minimumDetectableEffect: number
+}): RunningTimeOutlook | null {
+ if (!projection || currentSampleSize === null || !savedEffect || !savedSampleSize) {
+ return null
+ }
+ const rate = projection.exposures_per_day
+ const now: DetectableEffectCheckpoint = {
+ key: 'now',
+ kind: 'now',
+ days: projection.days_since_launch,
+ exposures: currentSampleSize,
+ detectableEffect: projection.detectable_effect_now,
+ isTarget: false,
+ }
+ const weeks = projection.upcoming_checkpoints.map(
+ (checkpoint): DetectableEffectCheckpoint => ({
+ key: `week-${checkpoint.weeks_since_launch}`,
+ kind: 'week',
+ days: checkpoint.weeks_since_launch * 7,
+ exposures: Math.round(rate * checkpoint.weeks_since_launch * 7),
+ detectableEffect: checkpoint.detectable_effect,
+ isTarget: false,
+ })
+ )
+ // A saved effect that today or a week mark shows came from that row's "Use", so that row is the target. The custom
+ // row then starts at the default effect rather than repeat the row.
+ const savedRow = [now, ...weeks].find(
+ (checkpoint) => roundUpDetectableEffect(checkpoint.detectableEffect) === roundUpDetectableEffect(savedEffect)
+ )
+ const customEffect = enteredEffect ?? (savedRow ? defaultEffect : savedEffect)
+ const customSampleSize = rescaleSampleSize(savedSampleSize, savedEffect, customEffect)
+ if (!customSampleSize) {
+ return null
+ }
+ const custom: DetectableEffectCheckpoint = {
+ key: 'custom',
+ kind: 'custom',
+ days: customSampleSize / rate,
+ exposures: customSampleSize,
+ detectableEffect: customEffect,
+ isTarget: false,
+ }
+ const rows = [now, ...weeks, custom]
+ const targetKey = selectedKey ?? (enteredEffect === null && savedRow ? savedRow.key : custom.key)
+ const target = rows.find((checkpoint) => checkpoint.key === targetKey) ?? custom
+
+ return {
+ daysToTarget: target.days,
+ isOutOfReach: minimumDetectableEffect < projection.reachable_effect,
+ checkpoints: rows
+ .map((checkpoint) => (checkpoint === target ? { ...checkpoint, isTarget: true } : checkpoint))
+ .sort((a, b) => a.days - b.days),
+ reachableEffect: projection.reachable_effect,
+ }
+}
+
+/**
+ * Calendar days from today to a day since launch, at least one, so that the header and the table count alike.
+ */
+export function daysFromToday(startDate: string, daysSinceLaunch: number): number {
+ const date = dayjs(startDate).add(daysSinceLaunch * 24, 'hour')
+ return Math.max(1, date.startOf('day').diff(dayjs().startOf('day'), 'day'))
+}
+
+export function describeCheckpointDay(
+ checkpoint: DetectableEffectCheckpoint,
+ { startDate, currentSampleSize }: { startDate: string; currentSampleSize: number }
+): string {
+ if (checkpoint.kind === 'week') {
+ return `week ${checkpoint.days / 7}`
+ }
+ if (checkpoint.kind === 'custom' && checkpoint.exposures > currentSampleSize) {
+ const days = daysFromToday(startDate, checkpoint.days)
+ return `in ${days.toLocaleString()} ${days === 1 ? 'day' : 'days'}`
+ }
+ // Calendar days, like the dates beside them, so that rows on one date show one day and the launch date is day 1.
+ const launch = dayjs(startDate)
+ const day =
+ launch
+ .add(checkpoint.days * 24, 'hour')
+ .startOf('day')
+ .diff(launch.startOf('day'), 'day') + 1
+ return `day ${day.toLocaleString()}`
+}
+
+export function describeUseButton(checkpoint: DetectableEffectCheckpoint): {
+ effect: number
+ disabledReason?: string
+ tooltip?: string
+} {
+ const effect =
+ checkpoint.kind === 'custom'
+ ? checkpoint.detectableEffect
+ : roundUpDetectableEffect(checkpoint.detectableEffect)
+ if (effect > MINIMUM_DETECTABLE_EFFECT_MAX) {
+ return {
+ effect,
+ disabledReason: `The minimum detectable effect can be at most ${MINIMUM_DETECTABLE_EFFECT_MAX}%`,
+ }
+ }
+ if (effect < MINIMUM_DETECTABLE_EFFECT_MIN) {
+ return {
+ effect,
+ disabledReason: `The minimum detectable effect must be at least ${MINIMUM_DETECTABLE_EFFECT_MIN}%`,
+ }
+ }
+ return { effect, tooltip: `Set ${effect.toLocaleString()}% as the minimum detectable effect` }
+}
+
+/**
+ * Past the largest effect the settings accept, the exact value is too large to act on, so it shows as ">100%".
+ */
+export function formatDetectableEffect(detectableEffect: number): string {
+ const effect = roundUpDetectableEffect(detectableEffect)
+ return effect > MINIMUM_DETECTABLE_EFFECT_MAX
+ ? `>${MINIMUM_DETECTABLE_EFFECT_MAX}%`
+ : `${effect.toLocaleString(undefined, { maximumSignificantDigits: 3 })}%`
+}
diff --git a/frontend/src/scenes/experiments/RunningTimeCalculator/runningTimeLogic.test.ts b/frontend/src/scenes/experiments/RunningTimeCalculator/runningTimeLogic.test.ts
index bddda9c4b730..d6a099830ca5 100644
--- a/frontend/src/scenes/experiments/RunningTimeCalculator/runningTimeLogic.test.ts
+++ b/frontend/src/scenes/experiments/RunningTimeCalculator/runningTimeLogic.test.ts
@@ -1,6 +1,7 @@
import { api } from 'lib/api.mock'
import { expectLogic } from 'kea-test-utils'
+import posthog from 'posthog-js'
import { lemonToast } from 'lib/lemon-ui/LemonToast/LemonToast'
@@ -11,6 +12,7 @@ import { Experiment } from '~/types'
import { experimentLogic } from '../experimentLogic'
import { experimentMetricsLogic } from '../experimentMetricsLogic'
import { modalsLogic } from '../modalsLogic'
+import { roundUpDetectableEffect } from './calculations'
import { runningTimeLogic } from './runningTimeLogic'
jest.mock('lib/lemon-ui/LemonToast/LemonToast', () => ({
@@ -61,6 +63,15 @@ const metricResultWithBaseline = {
variant_results: [{ key: 'test', number_of_samples: 600, sum: 130, sum_squares: 52 }],
}
+// The backend's projection for the result above: 1,200 exposures over ten days.
+const PROJECTION = {
+ days_since_launch: 10,
+ exposures_per_day: 120,
+ detectable_effect_now: 38.7,
+ upcoming_checkpoints: [{ weeks_since_launch: 2, detectable_effect: 32.7 }],
+ reachable_effect: 25.4,
+}
+
// The latest completed run covers the one primary metric, so the metrics logic applies the baseline on mount.
const latestRecalculation = {
id: 'recalc-1',
@@ -123,6 +134,7 @@ describe('runningTimeLogic', () => {
runningTimeTargetMock.mockResolvedValue({
recommended_sample_size: 2000,
minimum_detectable_effect: 30,
+ outlook: PROJECTION,
})
api.update.mockResolvedValue({
...experiment,
@@ -172,6 +184,7 @@ describe('runningTimeLogic', () => {
runningTimeTargetMock.mockResolvedValue({
recommended_sample_size: 2000,
minimum_detectable_effect: 30,
+ outlook: PROJECTION,
})
api.update.mockRejectedValue({
status: 409,
@@ -202,7 +215,7 @@ describe('runningTimeLogic', () => {
logic = runningTimeLogic({ experiment })
logic.mount()
- await expectLogic(logic).toDispatchActions(['loadAutomaticCalculationSuccess']).toFinishAllListeners()
+ await expectLogic(logic).toDispatchActions(['loadAutomaticTargetSuccess']).toFinishAllListeners()
runningTimeTargetMock.mockClear()
logic.actions.setConfig({ mde: 15 })
@@ -223,18 +236,75 @@ describe('runningTimeLogic', () => {
logic = runningTimeLogic({ experiment })
logic.mount()
- await expectLogic(logic).toDispatchActions(['loadAutomaticCalculationSuccess']).toFinishAllListeners()
+ await expectLogic(logic).toDispatchActions(['loadAutomaticTargetSuccess']).toFinishAllListeners()
expect(logic.values.currentExposures).toEqual(1100)
})
+ it('counts the days left on the projection, also before 100 exposures and with results from before today', async () => {
+ // 90 exposures in results that end on day 9 make 10 a day, so the 180 of the target arrive on day 18.
+ runningTimeTargetMock.mockResolvedValue({
+ recommended_sample_size: 180,
+ minimum_detectable_effect: 30,
+ current_sample_size: 90,
+ outlook: { ...PROJECTION, days_since_launch: 10, exposures_per_day: 10 },
+ })
+ api.update.mockResolvedValue({ ...experiment, version: 4 })
+
+ logic = runningTimeLogic({ experiment })
+ logic.mount()
+ await expectLogic(logic).toDispatchActions(['loadAutomaticTargetSuccess']).toFinishAllListeners()
+
+ expect(logic.values.remainingDays).toEqual(8)
+ })
+
+ it('keeps the same rows while the user takes a row with "Use" and moves the slider', async () => {
+ // At 120 exposures a day, the 1,996 that 30% needs arrive after week 2, which can detect 32.7%.
+ runningTimeTargetMock.mockResolvedValue({
+ recommended_sample_size: 1996,
+ minimum_detectable_effect: 30,
+ current_sample_size: 1200,
+ outlook: PROJECTION,
+ })
+ api.update.mockResolvedValue({ ...experiment, version: 4 })
+ logic = runningTimeLogic({ experiment })
+ logic.mount()
+ await expectLogic(logic).toDispatchActions(['loadAutomaticTargetSuccess']).toFinishAllListeners()
+ const rows = (): [number, boolean][] | undefined =>
+ logic.values.outlook?.checkpoints.map(({ detectableEffect, isTarget }) => [detectableEffect, isTarget])
+
+ expect(rows()).toEqual([
+ [38.7, false],
+ [32.7, false],
+ [30, true],
+ ])
+
+ logic.actions.selectCheckpoint('week-2', roundUpDetectableEffect(32.7))
+
+ expect(rows()).toEqual([
+ [38.7, false],
+ [32.7, true],
+ [30, false],
+ ])
+ expect(logic.values.remainingDays).toEqual(4)
+ expect(logic.values.config.mde).toEqual(32.7)
+
+ logic.actions.setConfig({ mde: 36 })
+
+ expect(rows()).toEqual([
+ [38.7, false],
+ [36, true],
+ [32.7, false],
+ ])
+ })
+
it('asks again when the metric sums change without new exposures', async () => {
runningTimeTargetMock.mockResolvedValue({ recommended_sample_size: 2000, minimum_detectable_effect: 30 })
api.update.mockResolvedValue({ ...experiment, version: 4 })
logic = runningTimeLogic({ experiment })
logic.mount()
- await expectLogic(logic).toDispatchActions(['loadAutomaticCalculationSuccess']).toFinishAllListeners()
+ await expectLogic(logic).toDispatchActions(['loadAutomaticTargetSuccess']).toFinishAllListeners()
metricsLogicInstance.actions.setPrimaryMetricsResults([
{
@@ -242,7 +312,7 @@ describe('runningTimeLogic', () => {
baseline: { ...metricResultWithBaseline.baseline, sum: 150, sum_squares: 90 },
} as any,
])
- await expectLogic(logic).toDispatchActions(['loadAutomaticCalculation']).toFinishAllListeners()
+ await expectLogic(logic).toDispatchActions(['loadAutomaticTarget']).toFinishAllListeners()
expect(runningTimeTargetMock).toHaveBeenCalledTimes(2)
})
@@ -253,6 +323,7 @@ describe('runningTimeLogic', () => {
runningTimeTargetMock.mockResolvedValueOnce({
recommended_sample_size: 2000,
minimum_detectable_effect: 30,
+ outlook: PROJECTION,
})
api.update.mockImplementation(async (_url: string, payload: unknown) => ({
...experiment,
@@ -260,19 +331,22 @@ describe('runningTimeLogic', () => {
version: 4,
}))
+ const captureSpy = jest.spyOn(posthog, 'capture')
logic = runningTimeLogic({ experiment })
logic.mount()
- await expectLogic(logic).toDispatchActions(['loadAutomaticCalculationSuccess']).toFinishAllListeners()
+ await expectLogic(logic).toDispatchActions(['loadAutomaticTargetSuccess']).toFinishAllListeners()
+ const savedMde = logic.values.config.mde
modalsLogic.actions.openRunningTimeConfigModal()
logic.actions.setConfig({ mde: 15 })
runningTimeTargetMock.mockResolvedValueOnce({
recommended_sample_size: 8000,
minimum_detectable_effect: 15,
+ outlook: PROJECTION,
})
api.update.mockClear()
await expectLogic(logic, () => logic.actions.save())
- .toDispatchActions(['loadAutomaticCalculationSuccess'])
+ .toDispatchActions(['loadAutomaticTargetSuccess'])
.toFinishAllListeners()
expect(api.update).toHaveBeenLastCalledWith(
@@ -284,6 +358,11 @@ describe('runningTimeLogic', () => {
}),
})
)
+ expect(captureSpy).toHaveBeenCalledWith(
+ 'experiment running time settings saved',
+ expect.objectContaining({ minimum_detectable_effect: 15, previous_minimum_detectable_effect: savedMde })
+ )
+ captureSpy.mockRestore()
})
it('keeps the modal open with the edited config when the experiment save fails', async () => {
@@ -308,7 +387,14 @@ describe('runningTimeLogic', () => {
})
describe('isCalculating', () => {
- it('stays true while the automatic estimate is in flight, then settles on success', async () => {
+ it.each([
+ ['an automatic experiment', {}, null],
+ [
+ 'a manual experiment switched to automatic',
+ { exposure_estimate_config: { conversionRateInputType: 'manual' } },
+ { mode: 'automatic' as const },
+ ],
+ ])('stays true while the estimate of %s is in flight, then settles', async (_name, saved, edit) => {
let resolveCalc: (value: unknown) => void = () => {}
runningTimeTargetMock.mockReturnValue(
new Promise((resolve) => {
@@ -316,15 +402,22 @@ describe('runningTimeLogic', () => {
})
)
api.update.mockResolvedValue({ ...experiment, version: 4 })
+ experimentLogicInstance.actions.setExperiment({
+ ...experiment,
+ running_time_calculation: saved as Experiment['running_time_calculation'],
+ })
logic = runningTimeLogic({ experiment })
logic.mount()
+ if (edit) {
+ logic.actions.setConfig(edit)
+ }
- await expectLogic(logic).toDispatchActions(['loadAutomaticCalculation'])
+ await expectLogic(logic).toDispatchActions(['loadAutomaticTarget'])
expect(logic.values.isCalculating).toBe(true)
resolveCalc({ recommended_sample_size: 2000, minimum_detectable_effect: 30 })
- await expectLogic(logic).toDispatchActions(['loadAutomaticCalculationSuccess']).toFinishAllListeners()
+ await expectLogic(logic).toDispatchActions(['loadAutomaticTargetSuccess']).toFinishAllListeners()
expect(logic.values.isCalculating).toBe(false)
})
@@ -336,7 +429,7 @@ describe('runningTimeLogic', () => {
logic = runningTimeLogic({ experiment })
logic.mount()
- await expectLogic(logic).toDispatchActions(['loadAutomaticCalculationFailure']).toFinishAllListeners()
+ await expectLogic(logic).toDispatchActions(['loadAutomaticTargetFailure']).toFinishAllListeners()
expect(logic.values.isCalculating).toBe(false)
})
@@ -344,12 +437,13 @@ describe('runningTimeLogic', () => {
runningTimeTargetMock.mockResolvedValue({
recommended_sample_size: 2000,
minimum_detectable_effect: 30,
+ outlook: PROJECTION,
})
logic = runningTimeLogic({ experiment })
logic.mount()
// The baseline fires the automatic estimate on mount; let it settle so only the recalculation state moves.
- await expectLogic(logic).toDispatchActions(['loadAutomaticCalculationSuccess']).toFinishAllListeners()
+ await expectLogic(logic).toDispatchActions(['loadAutomaticTargetSuccess']).toFinishAllListeners()
expect(logic.values.isCalculating).toBe(false)
metricsLogicInstance.actions.setRecalculationLoading(true)
diff --git a/frontend/src/scenes/experiments/RunningTimeCalculator/runningTimeLogic.ts b/frontend/src/scenes/experiments/RunningTimeCalculator/runningTimeLogic.ts
index 1c032901e282..7389cbb6bc7f 100644
--- a/frontend/src/scenes/experiments/RunningTimeCalculator/runningTimeLogic.ts
+++ b/frontend/src/scenes/experiments/RunningTimeCalculator/runningTimeLogic.ts
@@ -1,6 +1,7 @@
import { MakeLogicType, actions, connect, kea, key, listeners, path, props, reducers, selectors } from 'kea'
import { loaders } from 'kea-loaders'
import { subscriptions } from 'kea-subscriptions'
+import posthog from 'posthog-js'
import api from 'lib/api'
import { lemonToast } from 'lib/lemon-ui/LemonToast/LemonToast'
@@ -9,7 +10,7 @@ import { objectsEqual } from 'lib/utils/objects'
import { experimentsConfigLogic } from '~/scenes/settings/environment/experimentsConfigLogic'
import { ConversionRateInputType, Experiment } from '~/types'
-import { isLaunched } from 'products/experiments/frontend/experimentStatus'
+import { isExperimentExposureFrozen, isLaunched } from 'products/experiments/frontend/experimentStatus'
import {
experimentsCalculateRunningTimeCreate,
experimentsRunningTimeTargetRetrieve,
@@ -22,15 +23,18 @@ import type {
import type { CachedNewExperimentQueryResponse } from '../../../queries/schema/schema-general'
import { experimentLogic, saveExperimentUpdate } from '../experimentLogic'
import { experimentMetricsLogic } from '../experimentMetricsLogic'
+import { getDefaultMetricTitle } from '../MetricsView/shared/utils'
import { modalsLogic } from '../modalsLogic'
import { getFlagVariants, metricResults, isExperimentConflictError, toConcurrencyPayload } from '../utils'
import type { MetricWithResult } from '../utils'
import {
ManualCalculatorMetricType,
calculateCurrentExposures,
+ buildRunningTimeOutlook,
calculateDaysElapsed,
- calculateExposureRate,
+ daysFromToday,
rescaleSampleSize,
+ RunningTimeOutlook,
} from './calculations'
export interface RunningTimeLogicProps {
@@ -50,6 +54,14 @@ export interface ManualPreview {
runningTime: number | null
}
+// The metric the automatic estimate is sized from.
+export interface EstimateMetric {
+ name: string
+ // The experiment has more than one primary metric, and the estimate reads the first.
+ isFirstOfSeveral: boolean
+ hasOtherMetrics: boolean
+}
+
// The backend sizes the target for the saved minimum detectable effect. The settings rescale it for an edited one, so an
// edit sends no request.
export interface AutomaticTargetRequest {
@@ -68,17 +80,22 @@ export interface runningTimeLogicValues {
primaryMetricsResultsErrors: (unknown | null)[] // experimentMetricsLogic
defaultMinimumDetectableEffect: number // experimentsConfigLogic
isRunningTimeConfigModalOpen: boolean // modalsLogic
- automaticCalculation: RunningTimeTargetApi | null
- automaticCalculationInput: AutomaticTargetRequest | null
- automaticCalculationLoading: boolean
+ automaticTarget: RunningTimeTargetApi | null
+ automaticTargetFailed: boolean
+ automaticTargetLoading: boolean
+ automaticTargetRequest: AutomaticTargetRequest | null
+ canIncludeMoreUsers: boolean
config: RunningTimeConfig
configOverrides: Partial<RunningTimeConfig> | null
currentExposures: number | null
dailyExposureRate: number | null
editedMde: number | null
+ enteredMde: number | null
+ estimateMetric: EstimateMetric | null
initialConfig: RunningTimeConfig
isCalculating: boolean
isComplete: boolean
+ isManualEstimate: boolean
isManualMode: boolean
isSaving: boolean
manualFormPreview: ManualPreview
@@ -89,7 +106,9 @@ export interface runningTimeLogicValues {
metricsWithResults: MetricWithResult[]
mode: RunningTimeConfig['mode']
numberOfVariants: number
+ outlook: RunningTimeOutlook | null
remainingDays: number | null
+ selectedCheckpointKey: string | null
targetEffect: number
targetSampleSize: number | null
}
@@ -105,22 +124,25 @@ export interface runningTimeLogicActions {
closeRunningTimeConfigModal: () => {
value: true
} // modalsLogic
+ openRunningTimeConfigModal: () => {
+ value: true
+ } // modalsLogic
cancel: () => {
value: true
}
- loadAutomaticCalculation: (_: AutomaticTargetRequest) => AutomaticTargetRequest
- loadAutomaticCalculationFailure: (
+ loadAutomaticTarget: (_: AutomaticTargetRequest) => AutomaticTargetRequest
+ loadAutomaticTargetFailure: (
error: string,
errorObject?: any
) => {
error: string
errorObject?: any
}
- loadAutomaticCalculationSuccess: (
- automaticCalculation: RunningTimeTargetApi | null,
+ loadAutomaticTargetSuccess: (
+ automaticTarget: RunningTimeTargetApi | null,
payload?: AutomaticTargetRequest
) => {
- automaticCalculation: RunningTimeTargetApi | null
+ automaticTarget: RunningTimeTargetApi | null
payload?: AutomaticTargetRequest
}
loadManualPreview: (input: RunningTimeCalculationInputApi) => RunningTimeCalculationInputApi
@@ -153,6 +175,13 @@ export interface runningTimeLogicActions {
save: () => {
value: true
}
+ selectCheckpoint: (
+ key: string,
+ effect: number
+ ) => {
+ effect: number
+ key: string
+ }
setConfig: (config: Partial<RunningTimeConfig>) => {
config: Partial<RunningTimeConfig>
}
@@ -178,23 +207,20 @@ export interface runningTimeLogicMeta {
) => RunningTimeConfig
mde: (config: RunningTimeConfig) => number
editedMde: (configOverrides: Partial<RunningTimeConfig> | null) => number | null
- targetEffect: (
- editedMde: number | null,
- automaticCalculation: RunningTimeTargetApi | null,
- mde: number
- ) => number
+ targetEffect: (editedMde: number | null, automaticTarget: RunningTimeTargetApi | null, mde: number) => number
mode: (config: RunningTimeConfig) => RunningTimeConfig['mode']
metricsWithResults: (
experiment: Experiment,
primaryMetricsResults: CachedNewExperimentQueryResponse[],
primaryMetricsResultsErrors: unknown[]
) => MetricWithResult[]
+ isManualEstimate: (isManualMode: boolean, mode: 'automatic' | 'manual') => boolean
currentExposures: (
- isManualMode: boolean,
+ isManualEstimate: boolean,
metricsWithResults: MetricWithResult[],
- automaticCalculation: RunningTimeTargetApi | null
+ automaticTarget: RunningTimeTargetApi | null
) => number | null
- automaticCalculationInput: (
+ automaticTargetRequest: (
mode: 'automatic' | 'manual',
metricsWithResults: MetricWithResult[]
) => AutomaticTargetRequest | null
@@ -203,29 +229,37 @@ export interface runningTimeLogicMeta {
numberOfVariants: number
) => RunningTimeCalculationInputApi | null
targetSampleSize: (
- isManualMode: boolean,
+ isManualEstimate: boolean,
targetEffect: number,
experiment: Experiment,
- automaticCalculationInput: AutomaticTargetRequest | null,
- automaticCalculation: RunningTimeTargetApi | null
+ automaticTargetRequest: AutomaticTargetRequest | null,
+ automaticTarget: RunningTimeTargetApi | null
) => number | null
dailyExposureRate: (
- isManualMode: boolean,
+ isManualEstimate: boolean,
experiment: Experiment,
- currentExposures: number | null
+ automaticTarget: RunningTimeTargetApi | null
) => number | null
remainingDays: (
targetSampleSize: number | null,
currentExposures: number | null,
dailyExposureRate: number | null,
- experiment: Experiment
+ experiment: Experiment,
+ isManualEstimate: boolean,
+ outlook: RunningTimeOutlook | null
) => number | null
+ outlook: (
+ isManualEstimate: boolean,
+ automaticTarget: RunningTimeTargetApi | null,
+ targetEffect: number,
+ defaultMinimumDetectableEffect: number,
+ enteredMde: number | null,
+ selectedCheckpointKey: string | null
+ ) => RunningTimeOutlook | null
+ canIncludeMoreUsers: (experiment: Experiment) => boolean
+ estimateMetric: (experiment: Experiment, metricsWithResults: MetricWithResult[]) => EstimateMetric | null
isComplete: (currentExposures: number | null, targetSampleSize: number | null) => boolean
- isCalculating: (
- isManualMode: boolean,
- isRecalculating: boolean,
- automaticCalculationLoading: boolean
- ) => boolean
+ isCalculating: (isManualEstimate: boolean, isRecalculating: boolean, automaticTargetLoading: boolean) => boolean
manualFormPreview: (
manualPreviewInput: RunningTimeCalculationInputApi | null,
manualPreview: ManualPreview
@@ -262,13 +296,14 @@ export const runningTimeLogic = kea<runningTimeLogicType>([
experimentLogic({ experimentId }),
['setExperiment', 'setUnmodifiedExperiment'],
modalsLogic,
- ['closeRunningTimeConfigModal'],
+ ['openRunningTimeConfigModal', 'closeRunningTimeConfigModal'],
],
}
}),
actions({
setConfig: (config: Partial<RunningTimeConfig>) => ({ config }),
+ selectCheckpoint: (key: string, effect: number) => ({ key, effect }),
resetConfig: true,
save: true,
cancel: true,
@@ -281,6 +316,23 @@ export const runningTimeLogic = kea<runningTimeLogicType>([
null as Partial<RunningTimeConfig> | null,
{
setConfig: (state, { config }) => ({ ...state, ...config }),
+ selectCheckpoint: (state, { effect }) => ({ ...state, mde: effect }),
+ resetConfig: () => null,
+ },
+ ],
+ enteredMde: [
+ null as number | null,
+ {
+ setConfig: (state, { config }) => (config.mde === undefined ? state : config.mde),
+ resetConfig: () => null,
+ },
+ ],
+ // The row chosen with "Use". The slider and the input make the custom row the target again.
+ selectedCheckpointKey: [
+ null as string | null,
+ {
+ selectCheckpoint: (_, { key }) => key,
+ setConfig: (state, { config }) => (config.mde === undefined ? state : null),
resetConfig: () => null,
},
],
@@ -290,14 +342,22 @@ export const runningTimeLogic = kea<runningTimeLogicType>([
setSaving: (_, { saving }) => saving,
},
],
+ automaticTargetFailed: [
+ false,
+ {
+ loadAutomaticTarget: () => false,
+ loadAutomaticTargetSuccess: () => false,
+ loadAutomaticTargetFailure: () => true,
+ },
+ ],
}),
loaders(({ values, props }) => ({
// Automatic mode: the target the backend sizes from the primary metric's stored result and the flag's split.
- automaticCalculation: [
+ automaticTarget: [
null as RunningTimeTargetApi | null,
{
- loadAutomaticCalculation: async (_: AutomaticTargetRequest, breakpoint) => {
+ loadAutomaticTarget: async (_: AutomaticTargetRequest, breakpoint) => {
const experimentId = props.experiment.id
if (typeof experimentId !== 'number') {
return null
@@ -380,12 +440,12 @@ export const runningTimeLogic = kea<runningTimeLogicType>([
// The effect the automatic target is sized for: an edited one, else the one the backend sized for. The backend's
// can differ from `mde` before the team default loads, or when the saved effect isn't a positive number.
targetEffect: [
- (s) => [s.editedMde, s.automaticCalculation, s.mde],
- (editedMde: number | null, automaticCalculation: RunningTimeTargetApi | null, mde: number): number =>
- editedMde ?? automaticCalculation?.minimum_detectable_effect ?? mde,
+ (s) => [s.editedMde, s.automaticTarget, s.mde],
+ (editedMde: number | null, automaticTarget: RunningTimeTargetApi | null, mde: number): number =>
+ editedMde ?? automaticTarget?.minimum_detectable_effect ?? mde,
],
- // A primitive, so that automaticCalculationInput stays stable when the experiment object is replaced (e.g.
- // after persisting an estimate) but the mode is unchanged.
+ // A primitive, so that automaticTargetRequest stays stable when the experiment object is replaced (e.g. after
+ // persisting an estimate) but the mode is unchanged.
mode: [(s) => [s.config], (config: RunningTimeConfig): RunningTimeConfig['mode'] => config.mode],
// Primary metrics zipped with their results, so automatic mode always has a baseline to read.
@@ -398,10 +458,17 @@ export const runningTimeLogic = kea<runningTimeLogicType>([
) => (experiment ? metricResults(experiment)(results, errors, 'primary') : []),
],
+ // Switching a manual experiment to automatic in the settings previews the automatic estimate, so the manual one
+ // applies only while both the saved and the edited mode are manual.
+ isManualEstimate: [
+ (s) => [s.isManualMode, s.mode],
+ (isManualMode: boolean, mode: RunningTimeConfig['mode']): boolean => isManualMode && mode === 'manual',
+ ],
+
currentExposures: [
- (s) => [s.isManualMode, s.metricsWithResults, s.automaticCalculation],
+ (s) => [s.isManualEstimate, s.metricsWithResults, s.automaticTarget],
(
- isManualMode: boolean,
+ isManualEstimate: boolean,
results: {
displayIndex: number
error: any
@@ -409,18 +476,18 @@ export const runningTimeLogic = kea<runningTimeLogicType>([
metricIndex: number
result: any
}[],
- automaticCalculation: RunningTimeTargetApi | null
+ automaticTarget: RunningTimeTargetApi | null
): number | null => {
// The backend counts the exposures the automatic target sizes, so a variant at 0% does not count.
- if (!isManualMode && automaticCalculation?.current_sample_size != null) {
- return automaticCalculation.current_sample_size
+ if (!isManualEstimate && automaticTarget?.current_sample_size != null) {
+ return automaticTarget.current_sample_size
}
return calculateCurrentExposures(results?.[0]?.result ?? null)
},
],
// Request for the automatic-mode target, or null when not applicable (manual mode / no results yet).
- automaticCalculationInput: [
+ automaticTargetRequest: [
(s) => [s.mode, s.metricsWithResults],
(
mode: RunningTimeConfig['mode'],
@@ -472,59 +539,82 @@ export const runningTimeLogic = kea<runningTimeLogicType>([
],
targetSampleSize: [
- (s) => [s.isManualMode, s.targetEffect, s.experiment, s.automaticCalculationInput, s.automaticCalculation],
+ (s) => [s.isManualEstimate, s.targetEffect, s.experiment, s.automaticTargetRequest, s.automaticTarget],
(
- isManualMode: boolean,
+ isManualEstimate: boolean,
targetEffect: number,
experiment: Experiment,
automaticInput: AutomaticTargetRequest | null,
- automaticCalculation: RunningTimeTargetApi | null
+ automaticTarget: RunningTimeTargetApi | null
): number | null => {
- if (isManualMode) {
+ if (isManualEstimate) {
// Persisted by the save listener from the same backend calculation.
return experiment?.running_time_calculation?.recommended_sample_size ?? null
}
- if (!automaticInput || !automaticCalculation?.recommended_sample_size) {
+ if (!automaticInput || !automaticTarget?.recommended_sample_size) {
return null
}
return rescaleSampleSize(
- automaticCalculation.recommended_sample_size,
- automaticCalculation.minimum_detectable_effect,
+ automaticTarget.recommended_sample_size,
+ automaticTarget.minimum_detectable_effect,
targetEffect
)
},
],
dailyExposureRate: [
- (s) => [s.isManualMode, s.experiment, s.currentExposures],
- (isManualMode: boolean, experiment: Experiment, currentExposures: number | null): number | null => {
- if (isManualMode) {
+ (s) => [s.isManualEstimate, s.experiment, s.automaticTarget],
+ (
+ isManualEstimate: boolean,
+ experiment: Experiment,
+ automaticTarget: RunningTimeTargetApi | null
+ ): number | null => {
+ if (isManualEstimate) {
return experiment?.running_time_calculation?.exposure_estimate_config?.manualExposureRate ?? null
}
-
- // Automatic mode: calculate from actual rate
- const daysElapsed = calculateDaysElapsed(experiment?.start_date ?? null)
- return calculateExposureRate(currentExposures, daysElapsed)
+ return automaticTarget?.outlook?.exposures_per_day ?? null
},
],
// Days until we reach target sample size
remainingDays: [
- (s) => [s.targetSampleSize, s.currentExposures, s.dailyExposureRate, s.experiment],
+ (s) => [
+ s.targetSampleSize,
+ s.currentExposures,
+ s.dailyExposureRate,
+ s.experiment,
+ s.isManualEstimate,
+ s.outlook,
+ ],
(
target: number | null,
current: number | null,
rate: number | null,
- experiment: Experiment
+ experiment: Experiment,
+ isManualEstimate: boolean,
+ outlook: RunningTimeOutlook | null
): number | null => {
if (!target || target <= 0 || !rate || rate <= 0) {
return null
}
+ // The automatic rate comes from the backend's outlook, which projects from the first 0.1 days on.
+ if (!isManualEstimate) {
+ if (current !== null && current >= target) {
+ return 0
+ }
+ if (!experiment?.start_date) {
+ return Math.ceil(target / rate)
+ }
+ // The projection reaches the target `target / rate` days after launch, or at the checkpoint whose
+ // effect the target took. Calendar days to that date keep the header, the table's date and its
+ // "in N days" in step, also when the results end before today.
+ return daysFromToday(experiment.start_date, outlook?.daysToTarget ?? target / rate)
+ }
- // Need minimum data for automatic mode
+ // The manual estimate counts down from the exposures so far once there are a day and 100 of them.
const daysElapsed = calculateDaysElapsed(experiment?.start_date ?? null)
if (!daysElapsed || daysElapsed < 1 || !current || current < 100) {
- // Not enough data yet - show total estimated time if we have rate
+ // Before that, the whole planned run.
if (rate > 0) {
return Math.ceil(target / rate)
}
@@ -539,6 +629,65 @@ export const runningTimeLogic = kea<runningTimeLogicType>([
return Math.ceil(remaining / rate)
},
],
+ // What the experiment can detect over time, for the automatic estimate.
+ outlook: [
+ (s) => [
+ s.isManualEstimate,
+ s.automaticTarget,
+ s.targetEffect,
+ s.defaultMinimumDetectableEffect,
+ s.enteredMde,
+ s.selectedCheckpointKey,
+ ],
+ (
+ isManualEstimate: boolean,
+ automaticTarget: RunningTimeTargetApi | null,
+ targetEffect: number,
+ defaultMinimumDetectableEffect: number,
+ enteredMde: number | null,
+ selectedCheckpointKey: string | null
+ ): RunningTimeOutlook | null =>
+ isManualEstimate
+ ? null
+ : buildRunningTimeOutlook({
+ projection: automaticTarget?.outlook ?? null,
+ currentSampleSize: automaticTarget?.current_sample_size ?? null,
+ savedEffect: automaticTarget?.minimum_detectable_effect ?? null,
+ savedSampleSize: automaticTarget?.recommended_sample_size ?? null,
+ defaultEffect: defaultMinimumDetectableEffect,
+ enteredEffect: enteredMde,
+ selectedKey: selectedCheckpointKey,
+ minimumDetectableEffect: targetEffect,
+ }),
+ ],
+ canIncludeMoreUsers: [
+ (s) => [s.experiment],
+ (experiment: Experiment): boolean => {
+ // A group without conditions at 100% already includes every user.
+ const includesEveryone = (experiment?.feature_flag?.filters?.groups ?? []).some(
+ (group) => !group.properties?.length && (group.rollout_percentage ?? 100) >= 100
+ )
+ return !includesEveryone && !isExperimentExposureFrozen(experiment)
+ },
+ ],
+ // The first primary metric in display order, which is the one the backend sizes the target from.
+ estimateMetric: [
+ (s) => [s.experiment, s.metricsWithResults],
+ (experiment: Experiment, metricsWithResults: MetricWithResult[]): EstimateMetric | null => {
+ const metric = metricsWithResults?.[0]?.metric
+ if (!metric) {
+ return null
+ }
+ const secondaryCount =
+ (experiment?.metrics_secondary?.length ?? 0) +
+ (experiment?.saved_metrics ?? []).filter((saved) => saved.metadata?.type === 'secondary').length
+ return {
+ name: metric.name || getDefaultMetricTitle(metric),
+ isFirstOfSeveral: metricsWithResults.length > 1,
+ hasOtherMetrics: metricsWithResults.length > 1 || secondaryCount > 0,
+ }
+ },
+ ],
isComplete: [
(s) => [s.currentExposures, s.targetSampleSize],
(current: number | null, target: number | null): boolean =>
@@ -549,12 +698,12 @@ export const runningTimeLogic = kea<runningTimeLogicType>([
// instead of flashing a pending state during the gap between the two. Manual mode has no async
// estimate to wait on, so it never counts as calculating.
isCalculating: [
- (s) => [s.isManualMode, s.isRecalculating, s.automaticCalculationLoading],
- (isManualMode: boolean, isRecalculating: boolean, automaticCalculationLoading: boolean): boolean => {
- if (isManualMode) {
+ (s) => [s.isManualEstimate, s.isRecalculating, s.automaticTargetLoading],
+ (isManualEstimate: boolean, isRecalculating: boolean, automaticTargetLoading: boolean): boolean => {
+ if (isManualEstimate) {
return false
}
- return isRecalculating || automaticCalculationLoading
+ return isRecalculating || automaticTargetLoading
},
],
manualFormPreview: [
@@ -567,10 +716,10 @@ export const runningTimeLogic = kea<runningTimeLogicType>([
subscriptions(({ actions }) => ({
// kea-subscriptions fires on reference change. The recalc poll replaces the metric-results array every
// tick, so these inputs get a fresh object reference even when their values are identical, which would
- // re-fire the calculation POST on every poll. Skip the dispatch unless the value actually changed.
- automaticCalculationInput: (input: AutomaticTargetRequest | null, oldInput: AutomaticTargetRequest | null) => {
+ // re-fire the request on every poll. Skip the dispatch unless the value actually changed.
+ automaticTargetRequest: (input: AutomaticTargetRequest | null, oldInput: AutomaticTargetRequest | null) => {
if (input && !objectsEqual(input, oldInput)) {
- actions.loadAutomaticCalculation(input)
+ actions.loadAutomaticTarget(input)
}
},
manualPreviewInput: (
@@ -585,7 +734,7 @@ export const runningTimeLogic = kea<runningTimeLogicType>([
listeners(({ actions, values, props }) => ({
// Persist the automatic estimate once the backend calculation lands and the selectors are fresh.
- loadAutomaticCalculationSuccess: () => {
+ loadAutomaticTargetSuccess: () => {
actions.persistRunningTimeEstimate()
},
@@ -659,6 +808,8 @@ export const runningTimeLogic = kea<runningTimeLogicType>([
save: async () => {
const { config, numberOfVariants, experiment, currentProjectId } = values
+ // Read before the save, which replaces the experiment and with it the initial config.
+ const previousMde = values.initialConfig.mde
actions.setSaving(true)
let saved = false
@@ -729,15 +880,33 @@ export const runningTimeLogic = kea<runningTimeLogicType>([
if (!saved) {
return
}
+ // pinned: analytics event name and properties.
+ posthog.capture('experiment running time settings saved', {
+ experiment_id: props.experiment.id,
+ mode: config.mode,
+ minimum_detectable_effect: config.mde,
+ previous_minimum_detectable_effect: previousMde,
+ })
actions.closeRunningTimeConfigModal()
actions.resetConfig()
// The target on hand is sized for the previous effect. The backend sizes it for the saved one, and its
// success persists the estimate.
- if (config.mode === 'automatic' && values.automaticCalculationInput) {
- actions.loadAutomaticCalculation(values.automaticCalculationInput)
+ if (config.mode === 'automatic' && values.automaticTargetRequest) {
+ actions.loadAutomaticTarget(values.automaticTargetRequest)
}
},
+ openRunningTimeConfigModal: () => {
+ const { config, outlook } = values
+ // pinned: analytics event name and properties.
+ posthog.capture('experiment running time settings opened', {
+ experiment_id: props.experiment.id,
+ mode: config.mode,
+ out_of_reach: outlook?.isOutOfReach ?? null,
+ days_to_target: outlook ? Math.round(outlook.daysToTarget) : null,
+ })
+ },
+
cancel: () => {
actions.closeRunningTimeConfigModal()
actions.resetConfig()
diff --git a/frontend/src/scenes/experiments/stories/ExperimentRunningTimeOutlook.stories.tsx b/frontend/src/scenes/experiments/stories/ExperimentRunningTimeOutlook.stories.tsx
new file mode 100644
index 000000000000..43d93bb430e5
--- /dev/null
+++ b/frontend/src/scenes/experiments/stories/ExperimentRunningTimeOutlook.stories.tsx
@@ -0,0 +1,126 @@
+import { Meta, StoryObj } from '@storybook/react'
+import { waitFor } from '@testing-library/dom'
+import userEvent from '@testing-library/user-event'
+
+import { App } from 'scenes/App'
+import { urls } from 'scenes/urls'
+
+import { mswDecorator } from '~/mocks/browser'
+import EXPERIMENT_WITH_MEAN_METRIC from '~/mocks/fixtures/api/experiments/experiment_with_mean_metric.json'
+import EXPOSURE_QUERY_RESULT from '~/mocks/fixtures/api/experiments/exposure_query_result.json'
+import MEAN_METRIC_RESULT from '~/mocks/fixtures/api/experiments/mean_metric_result.json'
+import { NodeKind } from '~/queries/schema/schema-general'
+
+import type { RunningTimeTargetApi } from 'products/experiments/frontend/generated/api.schemas'
+
+import { recalculationMocks, resultsByMetricType } from './recalculationMocks'
+
+const EXPERIMENT_PATH = `/api/projects/:team_id/experiments/${EXPERIMENT_WITH_MEAN_METRIC.id}/`
+const RECALCULATION = recalculationMocks(
+ EXPERIMENT_WITH_MEAN_METRIC,
+ resultsByMetricType(EXPERIMENT_WITH_MEAN_METRIC, { mean: MEAN_METRIC_RESULT })
+)
+
+// The fixture launched on 2025-05-20, so `mockDate` puts the run two weeks in, as each target response's outlook says.
+const targetMocks = (target: RunningTimeTargetApi): Parameters<typeof mswDecorator>[0] => ({
+ get: {
+ ...RECALCULATION.get,
+ [EXPERIMENT_PATH]: EXPERIMENT_WITH_MEAN_METRIC,
+ [`${EXPERIMENT_PATH}running_time_target/`]: target,
+ '/api/projects/:team_id/experiment_holdouts': [],
+ '/api/projects/:team_id/experiment_saved_metrics/': [],
+ [`/api/projects/:team_id/feature_flags/${EXPERIMENT_WITH_MEAN_METRIC.feature_flag.id}/`]:
+ EXPERIMENT_WITH_MEAN_METRIC.feature_flag,
+ [`/api/projects/:team_id/feature_flags/${EXPERIMENT_WITH_MEAN_METRIC.feature_flag.id}/status/`]: {},
+ '/api/environments/:team_id/default_release_conditions/': [],
+ '/api/projects/:team_id/experiments_config/': {},
+ [`${EXPERIMENT_PATH}health/`]: { findings: [], checks: [] },
+ '/api/projects/:team_id/property_definitions/seen_together': {},
+ },
+ post: {
+ ...RECALCULATION.post,
+ '/api/environments/:team_id/query/:kind': async ({ request }) => {
+ const body = (await request.json()) as Record<string, any>
+ return body.query.kind === NodeKind.ExperimentExposureQuery
+ ? [200, EXPOSURE_QUERY_RESULT]
+ : [200, { results: [] }]
+ },
+ },
+ patch: { [EXPERIMENT_PATH]: EXPERIMENT_WITH_MEAN_METRIC },
+})
+
+const openRunningTimeSettings = async (): Promise<void> => {
+ const settings = await waitFor(() => {
+ const button = document.querySelector<HTMLElement>('[data-attr="experiment-running-time-config"]')
+ if (!button) {
+ throw new Error('The running time settings button has not rendered yet')
+ }
+ return button
+ })
+ await userEvent.click(settings)
+}
+
+const meta: Meta = {
+ component: App,
+ title: 'Scenes-App/Experiments',
+ parameters: {
+ layout: 'fullscreen',
+ viewMode: 'story',
+ mockDate: '2025-06-03',
+ pageUrl: urls.experiment(EXPERIMENT_WITH_MEAN_METRIC.id),
+ },
+}
+export default meta
+
+type Story = StoryObj<{}>
+
+// A target days away: the settings place it between the upcoming weeks.
+export const ExperimentRunningTimeOnTrack: Story = {
+ decorators: [
+ mswDecorator(
+ targetMocks({
+ minimum_detectable_effect: 30,
+ recommended_sample_size: 3000,
+ current_sample_size: 2000,
+ outlook: {
+ days_since_launch: 14,
+ exposures_per_day: 2000 / 14,
+ detectable_effect_now: 36.7,
+ upcoming_checkpoints: [
+ { weeks_since_launch: 4, detectable_effect: 26 },
+ { weeks_since_launch: 6, detectable_effect: 21.2 },
+ { weeks_since_launch: 8, detectable_effect: 18.4 },
+ ],
+ reachable_effect: 21.2,
+ },
+ })
+ ),
+ ],
+ play: openRunningTimeSettings,
+}
+
+// A target months away: the header says what the experiment can detect by week 4, and the settings list the ways to
+// finish sooner.
+export const ExperimentRunningTimeOutOfReach: Story = {
+ decorators: [
+ mswDecorator(
+ targetMocks({
+ minimum_detectable_effect: 30,
+ recommended_sample_size: 35556,
+ current_sample_size: 2000,
+ outlook: {
+ days_since_launch: 14,
+ exposures_per_day: 2000 / 14,
+ detectable_effect_now: 126.5,
+ upcoming_checkpoints: [
+ { weeks_since_launch: 4, detectable_effect: 89.4 },
+ { weeks_since_launch: 6, detectable_effect: 73 },
+ { weeks_since_launch: 8, detectable_effect: 63.2 },
+ ],
+ reachable_effect: 73,
+ },
+ })
+ ),
+ ],
+ play: openRunningTimeSettings,
+}
diff --git a/products/experiments/frontend/components/ExperimentMetaBar/ExperimentRemainingTime.tsx b/products/experiments/frontend/components/ExperimentMetaBar/ExperimentRemainingTime.tsx
index 2a9296bde038..6688e902d8dd 100644
--- a/products/experiments/frontend/components/ExperimentMetaBar/ExperimentRemainingTime.tsx
+++ b/products/experiments/frontend/components/ExperimentMetaBar/ExperimentRemainingTime.tsx
@@ -1,9 +1,13 @@
import { useValues } from 'kea'
-import { IconCheck, IconGear, IconHourglass } from '@posthog/icons'
+import { IconCheck, IconGear, IconHourglass, IconWarning } from '@posthog/icons'
import { LemonButton, Tooltip } from '@posthog/lemon-ui'
import { LemonProgressCircle } from 'lib/lemon-ui/LemonProgressCircle/LemonProgressCircle'
+import {
+ MINIMUM_DETECTABLE_EFFECT_MAX,
+ roundUpDetectableEffect,
+} from 'scenes/experiments/RunningTimeCalculator/calculations'
import { runningTimeLogic } from 'scenes/experiments/RunningTimeCalculator/runningTimeLogic'
import { Experiment } from '~/types'
@@ -17,9 +21,8 @@ export function ExperimentRemainingTime({
experiment: Experiment
onConfigure: () => void
}): JSX.Element {
- const { remainingDays, currentExposures, targetSampleSize, isComplete, isCalculating } = useValues(
- runningTimeLogic({ experiment })
- )
+ const { remainingDays, currentExposures, targetSampleSize, isComplete, isCalculating, outlook, targetEffect } =
+ useValues(runningTimeLogic({ experiment }))
const launched = isLaunched(experiment)
const showProgress = currentExposures !== null && targetSampleSize !== null
@@ -37,6 +40,28 @@ export function ExperimentRemainingTime({
<span>Target reached</span>
</span>
)
+ } else if (outlook?.isOutOfReach) {
+ // A countdown of months helps nobody, so the header says what the experiment can detect instead.
+ const reachable = roundUpDetectableEffect(outlook.reachableEffect)
+ const isBeyondMaximum = reachable > MINIMUM_DETECTABLE_EFFECT_MAX
+ content = (
+ <Tooltip
+ title={`Detecting a change of ${targetEffect.toLocaleString()}% needs about ${days.toLocaleString()} more days at the current traffic. Within 4 weeks, the experiment can detect ${
+ isBeyondMaximum
+ ? `only changes over ${MINIMUM_DETECTABLE_EFFECT_MAX}%`
+ : `changes of ${reachable.toLocaleString()}% or more`
+ }. The running time settings show what it can detect at each date.`}
+ >
+ <span className="flex items-center gap-1.5">
+ <IconWarning className="text-warning text-base shrink-0" />
+ <span>
+ {isBeyondMaximum
+ ? `Can detect only changes over ${MINIMUM_DETECTABLE_EFFECT_MAX}% within 4 weeks`
+ : `Can detect ${reachable.toLocaleString()}%+ changes within 4 weeks`}
+ </span>
+ </span>
+ </Tooltip>
+ )
} else {
content = (
<span className="flex items-center gap-1.5">
diff --git a/products/experiments/frontend/modals/RunningTimeConfigModal/DetectableEffectTable.tsx b/products/experiments/frontend/modals/RunningTimeConfigModal/DetectableEffectTable.tsx
new file mode 100644
index 000000000000..dadb13b739d3
--- /dev/null
+++ b/products/experiments/frontend/modals/RunningTimeConfigModal/DetectableEffectTable.tsx
@@ -0,0 +1,137 @@
+import { useActions, useValues } from 'kea'
+
+import { IconCheck, IconWarning } from '@posthog/icons'
+import { LemonButton, LemonTable, LemonTag, Tooltip } from '@posthog/lemon-ui'
+
+import { dayjs } from 'lib/dayjs'
+import { Label } from 'lib/ui/Label/Label'
+import {
+ describeCheckpointDay,
+ describeUseButton,
+ formatDetectableEffect,
+} from 'scenes/experiments/RunningTimeCalculator/calculations'
+import { runningTimeLogic } from 'scenes/experiments/RunningTimeCalculator/runningTimeLogic'
+
+import { Experiment } from '~/types'
+
+export interface DetectableEffectTableProps {
+ experiment: Experiment
+}
+
+function formatDate(date: dayjs.Dayjs): string {
+ return date.format(date.isSame(dayjs(), 'year') ? 'MMM D' : 'MMM D, YYYY')
+}
+
+export function DetectableEffectTable({ experiment: experimentProp }: DetectableEffectTableProps): JSX.Element | null {
+ const logic = runningTimeLogic({ experiment: experimentProp })
+ const {
+ outlook,
+ currentExposures,
+ isComplete,
+ dailyExposureRate,
+ estimateMetric,
+ canIncludeMoreUsers,
+ experiment,
+ } = useValues(logic)
+ const { selectCheckpoint } = useActions(logic)
+
+ const startDate = experiment.start_date
+ if (!outlook || currentExposures === null || dailyExposureRate === null || !startDate) {
+ return null
+ }
+
+ const dateAt = (days: number): dayjs.Dayjs => dayjs(startDate).add(days * 24, 'hour')
+
+ return (
+ <div className="space-y-2" data-attr="experiment-running-time-outlook">
+ <Label intent="menu">What the experiment can detect</Label>
+ <LemonTable
+ size="small"
+ embedded
+ dataSource={outlook.checkpoints}
+ rowKey={(checkpoint) => checkpoint.key}
+ rowStatus={(checkpoint) => (checkpoint.isTarget ? 'highlighted' : null)}
+ rowClassName={(checkpoint) => (checkpoint.isTarget ? 'font-semibold' : null)}
+ columns={[
+ {
+ title: 'When',
+ key: 'when',
+ // Takes the spare width, so that the number columns stay in place while the target row moves.
+ width: '100%',
+ render: (_, checkpoint) => (
+ <span className="flex items-center gap-1.5 whitespace-nowrap">
+ {checkpoint.isTarget && isComplete && (
+ <Tooltip title="Target reached">
+ <IconCheck className="text-success shrink-0" />
+ </Tooltip>
+ )}
+ {checkpoint.isTarget && outlook.isOutOfReach && (
+ <Tooltip title="More than 4 weeks from today. Most experiments end within 4 weeks.">
+ <IconWarning className="text-warning shrink-0" />
+ </Tooltip>
+ )}
+ <span>{checkpoint.kind === 'now' ? 'Today' : formatDate(dateAt(checkpoint.days))}</span>
+ <span className="text-muted font-normal">{`· ${describeCheckpointDay(checkpoint, {
+ startDate,
+ currentSampleSize: currentExposures,
+ })}`}</span>
+ </span>
+ ),
+ },
+ {
+ title: 'Exposures',
+ key: 'exposures',
+ align: 'right',
+ render: (_, checkpoint) => <span>{checkpoint.exposures.toLocaleString()}</span>,
+ },
+ {
+ title: 'Detectable effect',
+ key: 'detectable',
+ align: 'right',
+ render: (_, checkpoint) => <span>{formatDetectableEffect(checkpoint.detectableEffect)}</span>,
+ },
+ {
+ key: 'set-target',
+ align: 'right',
+ width: 0,
+ render: (_, checkpoint) => {
+ if (checkpoint.isTarget) {
+ return (
+ <LemonTag type="highlight" size="small">
+ Target
+ </LemonTag>
+ )
+ }
+ const { effect, disabledReason, tooltip } = describeUseButton(checkpoint)
+ return (
+ <LemonButton
+ size="xsmall"
+ type="tertiary"
+ disabledReason={disabledReason}
+ tooltip={tooltip}
+ onClick={() => selectCheckpoint(checkpoint.key, effect)}
+ data-attr="experiment-running-time-set-target"
+ >
+ Use
+ </LemonButton>
+ )
+ },
+ },
+ ]}
+ />
+ <div className="text-xs text-secondary">
+ <span>Based on </span>
+ <span className="font-semibold">{estimateMetric?.name ?? 'the primary metric'}</span>
+ <span>
+ {`${estimateMetric ? `, the ${estimateMetric.isFirstOfSeveral ? 'first ' : ''}primary metric` : ''}.${
+ estimateMetric?.hasOtherMetrics ? ' Other metrics can need more time.' : ''
+ } Projections assume about ${Math.round(dailyExposureRate).toLocaleString()} exposures a day, the average since launch.${
+ outlook.isOutOfReach && canIncludeMoreUsers
+ ? ' To finish sooner, include more users in the experiment.'
+ : ''
+ }`}
+ </span>
+ </div>
+ </div>
+ )
+}
diff --git a/products/experiments/frontend/modals/RunningTimeConfigModal/MinimumDetectableEffectField.tsx b/products/experiments/frontend/modals/RunningTimeConfigModal/MinimumDetectableEffectField.tsx
new file mode 100644
index 000000000000..f2a030229726
--- /dev/null
+++ b/products/experiments/frontend/modals/RunningTimeConfigModal/MinimumDetectableEffectField.tsx
@@ -0,0 +1,80 @@
+import { useEffect, useState } from 'react'
+
+import { LemonInput } from '@posthog/lemon-ui'
+
+import { LemonSlider } from 'lib/lemon-ui/LemonSlider'
+import { Label } from 'lib/ui/Label/Label'
+import {
+ MINIMUM_DETECTABLE_EFFECT_MAX,
+ MINIMUM_DETECTABLE_EFFECT_MIN,
+} from 'scenes/experiments/RunningTimeCalculator/calculations'
+
+export interface MinimumDetectableEffectFieldProps {
+ value: number
+ onChange: (value: number) => void
+}
+
+const isInRange = (value: number | undefined): value is number =>
+ value !== undefined && value >= MINIMUM_DETECTABLE_EFFECT_MIN && value <= MINIMUM_DETECTABLE_EFFECT_MAX
+
+export function MinimumDetectableEffectField({ value, onChange }: MinimumDetectableEffectFieldProps): JSX.Element {
+ // The input keeps what the user types, so that an empty field or a partial number stays editable. Only a value in
+ // range reaches the estimate.
+ const [draft, setDraft] = useState<number | undefined>(value)
+ useEffect(() => setDraft(value), [value])
+
+ const isDraftEmpty = draft === undefined || Number.isNaN(draft)
+
+ const commitDraft = (): void => {
+ if (draft === undefined || Number.isNaN(draft)) {
+ setDraft(value)
+ } else if (!isInRange(draft)) {
+ const clamped = Math.min(MINIMUM_DETECTABLE_EFFECT_MAX, Math.max(MINIMUM_DETECTABLE_EFFECT_MIN, draft))
+ setDraft(clamped)
+ onChange(clamped)
+ }
+ }
+
+ return (
+ <div>
+ <Label intent="menu">Minimum detectable effect</Label>
+ <div className="flex items-center gap-3">
+ <div className="flex-[3]">
+ <LemonSlider
+ value={value}
+ onChange={onChange}
+ min={MINIMUM_DETECTABLE_EFFECT_MIN}
+ max={MINIMUM_DETECTABLE_EFFECT_MAX}
+ step={1}
+ />
+ </div>
+ <div className="flex-1 flex items-center gap-1">
+ <LemonInput
+ type="number"
+ value={draft}
+ onChange={(next) => {
+ setDraft(next)
+ if (isInRange(next)) {
+ onChange(next)
+ }
+ }}
+ onBlur={commitDraft}
+ onPressEnter={commitDraft}
+ status={isDraftEmpty || isInRange(draft) ? 'default' : 'danger'}
+ min={MINIMUM_DETECTABLE_EFFECT_MIN}
+ max={MINIMUM_DETECTABLE_EFFECT_MAX}
+ step={1}
+ className="flex-1"
+ data-attr="experiment-running-time-mde-input"
+ />
+ <span className="text-muted">%</span>
+ </div>
+ </div>
+ {!isDraftEmpty && !isInRange(draft) && (
+ <div className="text-xs text-danger mt-1">
+ {`Enter a value from ${MINIMUM_DETECTABLE_EFFECT_MIN} to ${MINIMUM_DETECTABLE_EFFECT_MAX}.`}
+ </div>
+ )}
+ </div>
+ )
+}
diff --git a/products/experiments/frontend/modals/RunningTimeConfigModal/MinimumDetectableEffectHelp.tsx b/products/experiments/frontend/modals/RunningTimeConfigModal/MinimumDetectableEffectHelp.tsx
new file mode 100644
index 000000000000..08a473155f71
--- /dev/null
+++ b/products/experiments/frontend/modals/RunningTimeConfigModal/MinimumDetectableEffectHelp.tsx
@@ -0,0 +1,14 @@
+export interface MinimumDetectableEffectHelpProps {
+ // The color too, so that the text matches the other hints where it sits.
+ className: string
+}
+
+export function MinimumDetectableEffectHelp({ className }: MinimumDetectableEffectHelpProps): JSX.Element {
+ return (
+ <div className={`text-xs ${className}`}>
+ The minimum detectable effect is the smallest change you want to detect, relative to the control. Smaller
+ changes are harder to tell apart from random noise, so they need more exposures. It affects only the running
+ time, not how the metrics are calculated.
+ </div>
+ )
+}
diff --git a/products/experiments/frontend/modals/RunningTimeConfigModal/RunningTimeConfigModal.tsx b/products/experiments/frontend/modals/RunningTimeConfigModal/RunningTimeConfigModal.tsx
index 817ab2010a54..f6f3d2078765 100644
--- a/products/experiments/frontend/modals/RunningTimeConfigModal/RunningTimeConfigModal.tsx
+++ b/products/experiments/frontend/modals/RunningTimeConfigModal/RunningTimeConfigModal.tsx
@@ -1,9 +1,7 @@
import { useActions, useValues } from 'kea'
-import { IconCheck } from '@posthog/icons'
-import { LemonBanner, LemonButton, LemonInput, LemonModal, LemonSegmentedButton } from '@posthog/lemon-ui'
+import { LemonBanner, LemonButton, LemonInput, LemonModal, LemonSegmentedButton, Spinner } from '@posthog/lemon-ui'
-import { LemonSlider } from 'lib/lemon-ui/LemonSlider'
import { Label } from 'lib/ui/Label/Label'
import { ManualCalculatorMetricType } from 'scenes/experiments/RunningTimeCalculator/calculations'
import { runningTimeLogic } from 'scenes/experiments/RunningTimeCalculator/runningTimeLogic'
@@ -12,6 +10,10 @@ import { Experiment } from '~/types'
import { isLaunched } from 'products/experiments/frontend/experimentStatus'
+import { DetectableEffectTable } from './DetectableEffectTable'
+import { MinimumDetectableEffectField } from './MinimumDetectableEffectField'
+import { MinimumDetectableEffectHelp } from './MinimumDetectableEffectHelp'
+
export interface RunningTimeConfigModalProps {
experiment: Experiment
}
@@ -48,23 +50,25 @@ export function RunningTimeConfigModal({ experiment: experimentProp }: RunningTi
const {
config,
manualFormPreview,
- currentExposures,
- targetSampleSize,
- dailyExposureRate,
- remainingDays,
isRunningTimeConfigModalOpen,
experiment,
isSaving,
+ outlook,
+ isCalculating,
+ automaticTargetFailed,
+ automaticTargetRequest,
} = useValues(runningTimeLogic({ experiment: experimentProp }))
- const { setConfig, save, cancel } = useActions(runningTimeLogic({ experiment: experimentProp }))
+ const { setConfig, save, cancel, loadAutomaticTarget } = useActions(
+ runningTimeLogic({ experiment: experimentProp })
+ )
- const hasAutomaticData = remainingDays !== null
const isPreLaunch = !isLaunched(experiment)
+ const effectField = <MinimumDetectableEffectField value={config.mde} onChange={(mde) => setConfig({ mde })} />
return (
- <LemonModal isOpen={isRunningTimeConfigModalOpen} onClose={cancel} width={480} simple>
+ <LemonModal isOpen={isRunningTimeConfigModalOpen} onClose={cancel} width={560} simple>
<LemonModal.Header>
- <h3>{isPreLaunch ? 'Estimate running time' : 'Running time configuration'}</h3>
+ <h3>{isPreLaunch ? 'Estimate running time' : 'Running time'}</h3>
</LemonModal.Header>
<LemonModal.Content className="flex-1 min-h-0">
<div className="space-y-6">
@@ -85,11 +89,11 @@ export function RunningTimeConfigModal({ experiment: experimentProp }: RunningTi
value={config.mode}
onChange={(value) => setConfig({ mode: value as 'automatic' | 'manual' })}
/>
- <div className="text-xs text-muted mt-1">
- {config.mode === 'manual'
- ? 'Enter your own values to calculate sample size and running time.'
- : 'Uses actual experiment data to calculate estimates.'}
- </div>
+ {config.mode === 'manual' && (
+ <div className="text-xs text-muted mt-1">
+ Enter your own values to calculate sample size and running time.
+ </div>
+ )}
</div>
{config.mode === 'manual' ? (
@@ -144,35 +148,8 @@ export function RunningTimeConfigModal({ experiment: experimentProp }: RunningTi
</div>
<div>
- <Label intent="menu">Minimum detectable effect</Label>
- <div className="flex items-center gap-3">
- <div className="flex-[3]">
- <LemonSlider
- value={config.mde}
- onChange={(value) => setConfig({ mde: Math.round(value * 10) / 10 })}
- min={1}
- max={100}
- step={0.1}
- />
- </div>
- <div className="flex-1 flex items-center gap-1">
- <LemonInput
- type="number"
- value={config.mde}
- onChange={(value) => setConfig({ mde: Number(value) || 1 })}
- min={1}
- max={100}
- step={0.1}
- className="flex-1"
- />
- <span className="text-muted">%</span>
- </div>
- </div>
- <div className="text-xs text-muted mt-1">
- The smallest delta (change) you want to be able to measure with statistical
- significance. Changing this setting does not impact the statistical analysis, only
- the estimated runtime. Lower values require more data and longer run times.
- </div>
+ {effectField}
+ <MinimumDetectableEffectHelp className="text-muted mt-1" />
</div>
<div>
@@ -192,74 +169,36 @@ export function RunningTimeConfigModal({ experiment: experimentProp }: RunningTi
</>
) : (
<>
- {!hasAutomaticData ? (
- <LemonBanner type="info">
- Waiting for sufficient data to calculate time estimates. Need at least 1 day and 100
- exposures.
- </LemonBanner>
- ) : (
- <>
- <div>
- <Label intent="menu">Remaining time</Label>
- <div className="metric-cell">
- {remainingDays === 0 ? (
- <span className="inline-flex items-center gap-1">
- <IconCheck className="text-success" />
- Complete
- </span>
- ) : (
- `~${Math.ceil(remainingDays)} day${Math.ceil(remainingDays) !== 1 ? 's' : ''}`
- )}
- </div>
+ {effectField}
+ <div className="space-y-2">
+ {!outlook && isCalculating ? (
+ <div className="flex items-center gap-2 text-secondary">
+ <Spinner />
+ <span>Calculating the running time…</span>
</div>
- <div>
- <Label intent="menu">Progress</Label>
- <div className="metric-cell">
- {currentExposures !== null && targetSampleSize !== null
- ? `${currentExposures.toLocaleString()} / ${targetSampleSize.toLocaleString()} exposures`
- : '—'}
- </div>
- </div>
- <div>
- <Label intent="menu">Rate</Label>
- <div className="metric-cell">
- {dailyExposureRate !== null
- ? `~${Math.round(dailyExposureRate).toLocaleString()} exposures/day`
- : '—'}
- </div>
- </div>
- </>
- )}
- <div>
- <Label intent="menu">Minimum detectable effect</Label>
- <div className="flex items-center gap-3">
- <div className="flex-[3]">
- <LemonSlider
- value={config.mde}
- onChange={(value) => setConfig({ mde: Math.round(value * 10) / 10 })}
- min={1}
- max={100}
- step={0.1}
- />
- </div>
- <div className="flex-1 flex items-center gap-1">
- <LemonInput
- type="number"
- value={config.mde}
- onChange={(value) => setConfig({ mde: Number(value) || 1 })}
- min={1}
- max={100}
- step={0.1}
- className="flex-1"
- />
- <span className="text-muted">%</span>
- </div>
- </div>
- <div className="text-xs text-muted mt-1">
- The smallest delta (change) you want to be able to measure with statistical
- significance. Changing this setting does not impact the statistical analysis, only
- the estimated runtime. Lower values require more data and longer run times.
- </div>
+ ) : automaticTargetFailed ? (
+ <LemonBanner
+ type="error"
+ action={
+ automaticTargetRequest
+ ? {
+ children: 'Try again',
+ onClick: () => loadAutomaticTarget(automaticTargetRequest),
+ }
+ : undefined
+ }
+ >
+ Couldn't calculate the running time.
+ </LemonBanner>
+ ) : !outlook ? (
+ <LemonBanner type="info">
+ Not enough data to estimate the running time yet. It needs exposures, and some
+ activity on the primary metric, in each variant.
+ </LemonBanner>
+ ) : (
+ <DetectableEffectTable experiment={experimentProp} />
+ )}
+ <MinimumDetectableEffectHelp className="text-secondary" />
</div>
</>
)}
diff --git a/products/experiments/skills/diagnosing-experiment-health/references/interpretation.md b/products/experiments/skills/diagnosing-experiment-health/references/interpretation.md
index a8674f3f5212..c20f03253a89 100644
--- a/products/experiments/skills/diagnosing-experiment-health/references/interpretation.md
+++ b/products/experiments/skills/diagnosing-experiment-health/references/interpretation.md
@@ -339,6 +339,9 @@ real anomaly worth investigating with the experiment URL.
**Behavior** `[HIGH]`. The duration display next to the experiment's status is an estimate of sample size.
It reads "~N days left" while the exposures are below the target, and "Target reached" once they are at or above it.
+When the target is more than four weeks away, it reads "Can detect X%+ changes within 4 weeks" with a warning icon instead.
+X is the smallest change that the experiment can detect within four weeks at its current traffic, not the minimum detectable effect.
+Above 100% it reads "Can detect only changes over 100% within 4 weeks".
Older versions read "Complete".
On an ended experiment "Complete" is the status tag: every ended experiment carries it, and the duration display is hidden.
Read `status` first when the user says "complete".
@@ -352,6 +355,8 @@ Read `status` first when the user says "complete".
- In manual mode the target comes from the baseline the user typed, not from the results.
- "Target reached" says that an effect of that size would probably have shown.
It says nothing about significance, and it does not end the experiment.
+- The running time settings list the exposures and the change that the experiment can detect today, at the next week marks, and on the date it reaches the target.
+ A change over 100% shows as ">100%".
<!-- Source for maintainers (may rot): ExperimentRemainingTime.tsx in
products/experiments/frontend/components/ExperimentMetaBar; runningTimeLogic.ts in
@@ -365,7 +370,6 @@ frontend/src/scenes/experiments/RunningTimeCalculator; running_time_target.py in
- `exposure_estimate_config.conversionRateInputType` — `manual` means that the user typed the baseline and the daily exposures.
Both numbers above then come from that input: `recommended_running_time` is the whole estimated length, not the days left, and neither number is updated during the run.
The same holds for numbers that were written when the experiment was created, through the API or by an agent, until someone opens the launched experiment.
- A page load in the first day of the run, or before 100 exposures, also saves the whole estimated length, until a later page load replaces it.
A `recommended_running_time` above 0 on an experiment whose exposures have passed the target is one of these cases, or a value saved by a page load before the target was reached.
- `minimum_detectable_effect` — the experiment's own value, or the project default that was copied in when the experiment was created.
When it is absent the project default applies.