:hedgehog: PostHog is the leading platform for building self-driving products. Our developer tools – AI observability, analytics, session replay, flags, experiments, error tracking, logs, and more – capture all the context agents need to diagnose problems, uncover opportunities, and ship fixes. Steer it all from Slack, web, desktop, or the MCP.
Python ab-testingai-analyticsanalyticscdpdata-warehouseexperimentsfeature-flagsjavascriptproduct-analyticspythonreactsession-replaysurveystypescriptweb-analytics
fix(experiments): stop false bias and ended-flag health warnings (#127363)
Marcel Poelker2d1013ae3f29
diff --git a/products/experiments/backend/health/checks/bias_risk.py b/products/experiments/backend/health/checks/bias_risk.pyindex ab60060954de..8d88a98a20d8 100644--- a/products/experiments/backend/health/checks/bias_risk.py+++ b/products/experiments/backend/health/checks/bias_risk.py@@ -1,6 +1,6 @@"""Multi-variant exclusion bias on an uneven split. Pure functions, no I/O."""-from collections.abc import Mapping, Sequence+from collections.abc import Collection, Mapping, Sequencefrom posthog.schema import BiasRisk, MultipleVariantHandling@@ -18,40 +18,48 @@ from products.experiments.backend.health.skips import (CheckOutcome,not_applicable,)-from products.experiments.backend.variant_distribution import is_evenly_distributed+from products.feature_flags.backend.facade.edits import FlagVariantMULTIPLE_VARIANT_KEY = "$multiple"# `$multiple` share above this triggers the warning. Below this, the asymmetric-# exclusion effect on arm means is too small to matter in practice.MULTIPLE_VARIANT_BIAS_THRESHOLD = 0.1 # on the 0-100 scale (0.1 = 0.1%)+# One or two testers who switch between variants must not trigger the warning.+MULTIPLE_VARIANT_BIAS_MINIMUM_COUNT = 5def evaluate_bias_risk(- rollout_percentages: Sequence[float | None],+ variants: Sequence[FlagVariant],+ excluded_variant_keys: Collection[str],multiple_variant_handling: MultipleVariantHandling,total_exposures: Mapping[str, int],) -> BiasRisk | None:"""- Empirically observed multi-variant exclusion bias risk: uneven split + EXCLUDE- handling + observed `$multiple` share above the threshold.+ Empirically observed multi-variant exclusion bias risk: uneven shares of the variants that the flag serves and the+ analysis compares + EXCLUDE handling + observed `$multiple` share above the threshold, from enough users.Returns a `BiasRisk` only when all three conditions hold; `None` otherwise."""if multiple_variant_handling != MultipleVariantHandling.EXCLUDE:return None- if not rollout_percentages:+ # Excluded variants don't count, because the exposure query drops their events before it counts users in several+ # variants. `is_evenly_distributed` needs shares that add up to 100. Shares one point apart are even, like 34/33/33.+ compared = [+ variant.rollout_percentage+ for variant in variants+ if variant.rollout_percentage is not None+ and variant.rollout_percentage > 0+ and variant.key not in excluded_variant_keys+ ]+ if len(compared) < 2 or max(compared) - min(compared) <= 1:return None- if is_evenly_distributed(rollout_percentages):- return None-- total_observed = sum(total_exposures.values())- if total_observed <= 0:+ multiple_observed = total_exposures.get(MULTIPLE_VARIANT_KEY, 0)+ if multiple_observed < MULTIPLE_VARIANT_BIAS_MINIMUM_COUNT:return None- multiple_observed = total_exposures.get(MULTIPLE_VARIANT_KEY, 0)- multiple_variant_percentage = (multiple_observed / total_observed) * 100+ multiple_variant_percentage = (multiple_observed / sum(total_exposures.values())) * 100if multiple_variant_percentage <= MULTIPLE_VARIANT_BIAS_THRESHOLD:return None@@ -66,7 +74,8 @@ def bias_risk_multiple_excluded(ctx: HealthContext) -> CheckOutcome:if ctx.has_ended:return not_applicable(NOT_AFTER_END)risk = evaluate_bias_risk(- rollout_percentages=[variant.rollout_percentage for variant in ctx.flag.variants],+ variants=ctx.flag.variants,+ excluded_variant_keys=ctx.excluded_variant_keys,multiple_variant_handling=ctx.exposures.multiple_variant_handling,total_exposures=ctx.exposures.total_exposures,)diff --git a/products/experiments/backend/health/checks/flag_state.py b/products/experiments/backend/health/checks/flag_state.pyindex 4b22e22e7c1f..ebb3bbfd3bcb 100644--- a/products/experiments/backend/health/checks/flag_state.py+++ b/products/experiments/backend/health/checks/flag_state.py@@ -12,8 +12,9 @@ show. Only the server reports that case.A paused experiment is another exception. Only the server reads the experiment's activity log, so only the servertells a pause from the experiment from a flag disabled elsewhere, and reports the pause with its own subcode.-An ended experiment whose flag has no recorded call since the end is the last exception. Only the server reads the-flag's latest call, so only the server reports that case, with its own subcode.+An ended experiment is the last exception. Only the server reads the flag's latest call and the other experiments on+the flag. So only the server stays silent when the flag now serves a later experiment, and reports as info a flag that+a draft also uses, a flag with no recent call, and a flag whose calls it can't read yet, each with its own subcode."""from datetime import UTC, datetime, timedelta@@ -26,11 +27,13 @@ from products.experiments.backend.facade.contracts import ()from products.experiments.backend.health.context import (ConditionSetTraffic,+ ExperimentOnFlag,FlagHistoryNotChecked,FlagState,HealthContext,PauseHistory,)+from products.experiments.backend.health.delays import ENDED_FLAG_QUIET_PERIODfrom products.experiments.backend.health.skips import NO_FLAG, CheckOutcome, not_applicablefrom products.experiments.backend.health.split_changes import (SplitChangeShape,@@ -52,6 +55,8 @@ RUNNING_BUT_FLAG_DELETED = "running_but_flag_deleted"ASSIGNED_OUTSIDE_FLAG_SERVICE = "assigned_outside_flag_service"PAUSED_FROM_EXPERIMENT = "paused_from_experiment"ENDED_FLAG_NOT_CALLED = "ended_flag_not_called"+ENDED_FLAG_CALLS_UNKNOWN = "ended_flag_calls_unknown"+ENDED_FLAG_SHARED_WITH_DRAFT = "ended_flag_shared_with_draft"# The experiment's Pause and Resume write activity log entries since August 25, 2026, and a pause before then has# none. So a missing entry shows that the flag was disabled outside the experiment only for an experiment launched@@ -121,10 +126,14 @@ def flag_state(ctx: HealthContext) -> CheckOutcome:evidence={"variant_key": variant_key},)elif ctx.has_ended:- if flag.active and _has_multiple_variants_active(flag) and not zero_rollout and not single_variant_shipped:- if ctx.flag_called_after_end is False and ctx.ended_at is not None:- return _ended_flag_not_called(ctx.ended_at, ctx.team_timezone)- return _live_after_end(called_after_end=ctx.flag_called_after_end is True)+ if (+ flag.active+ and _has_multiple_variants_active(flag)+ and not zero_rollout+ and not single_variant_shipped+ and ctx.ended_at is not None+ ):+ return _live_after_end(ctx, ctx.ended_at)elif not ctx.archived:# The draft state is also reached again when resetting experiment measurements.if flag.active and _has_multiple_variants_active(flag) and not zero_rollout:@@ -211,8 +220,14 @@ def _no_rollout(*, observed: bool) -> ExperimentHealthFinding:)-def _live_after_end(*, called_after_end: bool) -> ExperimentHealthFinding:- if called_after_end:+def _live_after_end(ctx: HealthContext, ended_at: datetime) -> ExperimentHealthFinding | None:+ others = ctx.other_experiments_on_flag or ()+ # The flag's state belongs to the later experiment, and that experiment's own check reports it.+ if any(_serves_later_experiment(other, ended_at) for other in others):+ return None+ if any(other.launched_at is None and not other.archived for other in others):+ return _shared_with_draft(ended_at, ctx.team_timezone)+ if ctx.flag_called_after_end is True:return _finding(code=ExperimentHealthFindingCode.FLAG_LIVE_AFTER_END,subcode=ENDED_BUT_MULTIPLE_VARIANTS_ROLLED_OUT,@@ -222,10 +237,14 @@ def _live_after_end(*, called_after_end: bool) -> ExperimentHealthFinding:"traffic across multiple variants. Disable the flag, or resume the experiment."),)- # Without a call recorded after the end, the finding says only what the flag's settings show.+ if ctx.flag_called_after_end is False:+ return _ended_flag_not_called(ended_at, ctx.flag_last_called_at, ctx.team_timezone)+ # Info, because only a recorded call shows that users still get a variant. PostHog can't tell yet in the first+ # days after the end, and never for an organization whose flag calls it can't read.return _finding(code=ExperimentHealthFindingCode.FLAG_LIVE_AFTER_END,- subcode=ENDED_BUT_MULTIPLE_VARIANTS_ROLLED_OUT,+ subcode=ENDED_FLAG_CALLS_UNKNOWN,+ severity=ExperimentHealthFindingSeverity.INFO,title="The experiment has ended, but its feature flag is still active with multiple variants",detail=("This experiment has ended, but the linked feature flag is still active with multiple variants, so users "@@ -234,9 +253,38 @@ def _live_after_end(*, called_after_end: bool) -> ExperimentHealthFinding:)-def _ended_flag_not_called(ended_at: datetime, team_timezone: str) -> ExperimentHealthFinding:+def _serves_later_experiment(other: ExperimentOnFlag, ended_at: datetime) -> bool:+ return other.launched_at is not None and (other.ended_at is None or other.launched_at > ended_at)+++def _shared_with_draft(ended_at: datetime, team_timezone: str) -> ExperimentHealthFinding:+ # Info, because the draft's own check reports a flag that is active before launch.+ return _finding(+ code=ExperimentHealthFindingCode.FLAG_LIVE_AFTER_END,+ subcode=ENDED_FLAG_SHARED_WITH_DRAFT,+ severity=ExperimentHealthFindingSeverity.INFO,+ title="The feature flag is still active, and a draft experiment uses it",+ detail=(+ f"The experiment ended on {format_change_time(ended_at, team_timezone)}, and the linked feature flag is "+ "still active with multiple variants. A draft experiment uses the same flag, so users get its variants "+ "whenever your code reads the flag, before that experiment launches. This is expected while you test its "+ "variants. Otherwise, disable the flag until you launch it."+ ),+ evidence={"ended_at": ended_at.astimezone(UTC).isoformat()},+ )+++def _ended_flag_not_called(+ ended_at: datetime, last_called_at: datetime | None, team_timezone: str+) -> ExperimentHealthFinding:# Info, because a flag that nothing reads exposes nobody. The finding says only what `last_called_at` shows,# because reads that record no call can still get a variant.+ last_call = (+ f"PostHog last recorded a call of the flag on {format_change_time(last_called_at, team_timezone)}, and none "+ f"in the last {ENDED_FLAG_QUIET_PERIOD.days} days."+ if last_called_at is not None and last_called_at > ended_at+ else "PostHog hasn't recorded a call of the flag since then."+ )return _finding(code=ExperimentHealthFindingCode.FLAG_LIVE_AFTER_END,subcode=ENDED_FLAG_NOT_CALLED,@@ -244,12 +292,14 @@ def _ended_flag_not_called(ended_at: datetime, team_timezone: str) -> Experimenttitle="The feature flag is still active after the experiment ended",detail=(f"The experiment ended on {format_change_time(ended_at, team_timezone)}, and the linked feature flag is "- "still active with multiple variants. PostHog hasn't recorded a call of the flag since then. Reading only "- "a payload or a list of all flags records no call, and neither does a call with send_event set to false, "- "so your code may still read the flag that way. If nothing reads the flag anymore, disable or archive it "- "to clean up."+ f"still active with multiple variants. {last_call} Reading only a payload or a list of all flags records "+ "no call, and neither does a call with send_event set to false, so your code may still read the flag that "+ "way. If nothing reads the flag anymore, disable or archive it to clean up."),- evidence={"ended_at": ended_at.astimezone(UTC).isoformat()},+ evidence={+ "ended_at": ended_at.astimezone(UTC).isoformat(),+ "last_called_at": last_called_at.astimezone(UTC).isoformat() if last_called_at is not None else None,+ },)diff --git a/products/experiments/backend/health/context.py b/products/experiments/backend/health/context.pyindex 75da26b2e8ec..840fb975bf68 100644--- a/products/experiments/backend/health/context.py+++ b/products/experiments/backend/health/context.py@@ -48,7 +48,11 @@ from products.experiments.backend.logic.result_stats import MetricResultStatsfrom products.experiments.backend.metric_resolution import MetricRole, saved_metric_link_role, saved_metric_linksfrom products.experiments.backend.metric_utils import get_default_metric_titlefrom products.experiments.backend.metric_validation import defined_event_names, event_last_seen, extract_entity_nodes-from products.experiments.backend.models.experiment import Experiment, ExperimentToSavedMetric+from products.experiments.backend.models.experiment import (+ Experiment,+ ExperimentToSavedMetric,+ live_experiments_for_flag,+)from products.experiments.backend.running_time_target import RunningTimeTargetfrom products.feature_flags.backend.facade.edits import (BucketingChange,@@ -219,6 +223,17 @@ class PauseHistory:paused_at: datetime | None+@frozen+class ExperimentOnFlag:+ """Another experiment of the project that uses the same feature flag. The API lets several experiments share one+ flag, so the flag's state can belong to one of them. It has no name or id, because the reader of this experiment+ may have no access to the other one."""++ launched_at: datetime | None+ ended_at: datetime | None+ archived: bool++@frozenclass PrimaryMetric:"""The experiment's first primary metric, which its running-time estimate sizes, with its stored result."""@@ -553,11 +568,15 @@ class HealthContext:# answer this. None before launch, when the experiment has no flag, when the flag's latest call came after the# end, or when PostHog can't record the calls of this organization's flags.flag_called_during_run: bool | None- # Whether the flag's latest recorded call came after the end. False only when no call is recorded in the- # ENDED_FLAG_QUIET_PERIOD after the end, also for a flag without a recorded call. None before the end, when the- # experiment has no flag, before that period is over, or when PostHog can't record the calls of this- # organization's flags.+ # Whether PostHog has recorded a call of the flag after the end and in the last ENDED_FLAG_QUIET_PERIOD, so that+ # one stray call doesn't count as a flag that is still read. False only once that period after the end is over.+ # None before the end, when the experiment has no flag, before that period is over without a call, or when+ # PostHog can't record the calls of this organization's flags.flag_called_after_end: bool | None+ # The flag's latest recorded call. None when the experiment has no flag, or the flag has no recorded call.+ flag_last_called_at: datetime | None+ # The other non-deleted experiments on the flag. None when not loaded: it is loaded only for an ended experiment.+ other_experiments_on_flag: tuple[ExperimentOnFlag, ...] | None# Whether the team has an active flag, which posthog-js needs to request flags on page load. None when not read:# it is read only for a draft whose flag has no recorded call.team_has_active_flag: bool | None@@ -745,13 +764,29 @@ def _flag_called_after_end(experiment: Experiment) -> bool | None:if not experiment.is_stopped or end_date is None or feature_flag_id is None:return Nonelast_called_at = experiment.feature_flag.last_called_at- if last_called_at is not None and last_called_at > end_date:+ now = timezone.now()+ if last_called_at is not None and last_called_at > max(end_date, now - ENDED_FLAG_QUIET_PERIOD):return True- if timezone.now() - end_date < ENDED_FLAG_QUIET_PERIOD + LAST_CALLED_AT_LAG_DURATION:+ if now - end_date < ENDED_FLAG_QUIET_PERIOD + LAST_CALLED_AT_LAG_DURATION:return Nonereturn None if _flag_calls_unrecorded(experiment) else False+def _other_experiments_on_flag(experiment: Experiment) -> tuple[ExperimentOnFlag, ...] | None:+ if not experiment.is_stopped:+ return None+ others = (+ live_experiments_for_flag(experiment.feature_flag_id)+ .filter(team_id=experiment.team_id)+ .exclude(id=experiment.id)+ .values_list("start_date", "end_date", "archived", named=True)+ )+ return tuple(+ ExperimentOnFlag(launched_at=other.start_date, ended_at=other.end_date, archived=other.archived)+ for other in others+ )++def _flag_calls_unrecorded(experiment: Experiment) -> bool:# The task that sets `last_called_at` reads `$feature_flag_called` from the events table, unless it is set to# read flag_evaluations. On FLAG_EVALUATIONS_ONLY, ingestion writes no `$feature_flag_called` to the events table@@ -1046,6 +1081,8 @@ def load_health_context(flag_called=flag_called,flag_called_during_run=_flag_called_during_run(experiment, flag_called),flag_called_after_end=_flag_called_after_end(experiment),+ flag_last_called_at=experiment.feature_flag.last_called_at if experiment.feature_flag_id else None,+ other_experiments_on_flag=_other_experiments_on_flag(experiment),team_has_active_flag=(has_active_flags(experiment.team) if flag_called is False and not experiment.is_launched else None),diff --git a/products/experiments/backend/health/inputs_payload.py b/products/experiments/backend/health/inputs_payload.pyindex 1045cd444823..93dcfd981247 100644--- a/products/experiments/backend/health/inputs_payload.py+++ b/products/experiments/backend/health/inputs_payload.py@@ -263,7 +263,7 @@ def health_inputs_payload(inputs: HealthInputs) -> dict[str, Any]:"primary_metric_count": ctx.primary_metric_count,"secondary_metric_count": ctx.secondary_metric_count,"flag": _health_inputs_flag(ctx.flag) if ctx.flag else None,- "flag_last_called_at": inputs.flag_last_called_at,+ "flag_last_called_at": ctx.flag_last_called_at,"flag_called": ctx.flag_called,"flag_called_during_run": ctx.flag_called_during_run,"flag_called_after_end": ctx.flag_called_after_end,diff --git a/products/experiments/backend/health/loader.py b/products/experiments/backend/health/loader.pyindex 674aea76be02..7c5d4d8c0d7e 100644--- a/products/experiments/backend/health/loader.py+++ b/products/experiments/backend/health/loader.py@@ -1,5 +1,4 @@from dataclasses import replace-from datetime import datetimefrom posthog.dataclasses import frozenfrom posthog.models.user import User@@ -30,8 +29,6 @@ class HealthInputs:reader_access: ExperimentHealthReaderAccess# None when the experiment has no flag, or the flag has no version.flag_version: int | None- # The flag's latest recorded call. None when the experiment has no flag, or the flag has no recorded call.- flag_last_called_at: datetime | Noneproject: ProjectSetupFacts@@ -53,7 +50,6 @@ def load_experiment_health_inputs(exposures=load_exposure_inputs(experiment, user=user, can_query_events=reader_access.can_query_events),reader_access=reader_access,flag_version=experiment.feature_flag.version if experiment.feature_flag_id else None,- flag_last_called_at=experiment.feature_flag.last_called_at if experiment.feature_flag_id else None,project=build_project_setup_facts(experiment.team,can_query_events=reader_access.can_query_events,diff --git a/products/experiments/backend/health/test/factories.py b/products/experiments/backend/health/test/factories.pyindex 0cdff87ff0fc..fffe0ce44b2d 100644--- a/products/experiments/backend/health/test/factories.py+++ b/products/experiments/backend/health/test/factories.py@@ -27,6 +27,8 @@ RUNNING_CONTEXT = HealthContext(flag_called=None,flag_called_during_run=None,flag_called_after_end=None,+ flag_last_called_at=None,+ other_experiments_on_flag=None,team_has_active_flag=None,baseline_variant_key=None,primary_metric_count=1,diff --git a/products/experiments/backend/health/test/test_bias_risk.py b/products/experiments/backend/health/test/test_bias_risk.pyindex 4b889c1aa904..7842d0755713 100644--- a/products/experiments/backend/health/test/test_bias_risk.py+++ b/products/experiments/backend/health/test/test_bias_risk.py@@ -5,18 +5,25 @@ from parameterized import parameterizedfrom posthog.schema import BiasRisk, MultipleVariantHandlingfrom products.experiments.backend.health.checks.bias_risk import MULTIPLE_VARIANT_BIAS_THRESHOLD, evaluate_bias_risk+from products.feature_flags.backend.facade.edits import FlagVariant-UNEVEN_2WAY = [80, 20]-EVEN_2WAY = [50, 50]++def _split(*percentages: float) -> tuple[FlagVariant, ...]:+ keys = ("control", "test", "third")+ return tuple(FlagVariant(key=key, rollout_percentage=pct) for key, pct in zip(keys, percentages))+++UNEVEN_2WAY = _split(80, 20)+EVEN_2WAY = _split(50, 50)# Auto-distribution for 3 variants — should be treated as even, not uneven.-AUTO_EVEN_3WAY = [34, 33, 33]+AUTO_EVEN_3WAY = _split(34, 33, 33)class TestEvaluateBiasRisk(TestCase):def test_observed_bias_returns_populated_risk(self):# 20 / (800 + 200 + 20) ≈ 1.96%, well above the 0.1% threshold.result = evaluate_bias_risk(- UNEVEN_2WAY, MultipleVariantHandling.EXCLUDE, {"control": 800, "test": 200, "$multiple": 20}+ UNEVEN_2WAY, frozenset(), MultipleVariantHandling.EXCLUDE, {"control": 800, "test": 200, "$multiple": 20})self.assertIsInstance(result, BiasRisk)assert result is not None@@ -25,19 +32,13 @@ class TestEvaluateBiasRisk(TestCase):def test_auto_even_3way_is_treated_as_even(self):# 34/33/33 is what the auto-distribution produces — must NOT be flagged as uneven.result = evaluate_bias_risk(- AUTO_EVEN_3WAY, MultipleVariantHandling.EXCLUDE, {"a": 340, "b": 330, "c": 330, "$multiple": 50}+ AUTO_EVEN_3WAY,+ frozenset(),+ MultipleVariantHandling.EXCLUDE,+ {"a": 340, "b": 330, "c": 330, "$multiple": 50},)self.assertIsNone(result)- def test_reordered_auto_even_is_uneven(self):- # 33/34/33 doesn't match the auto-distribution result (34/33/33) — counts as uneven,- # mirroring the frontend's positional `isEvenlyDistributed` check.- reordered = [33, 34, 33]- result = evaluate_bias_risk(- reordered, MultipleVariantHandling.EXCLUDE, {"a": 330, "b": 340, "c": 330, "$multiple": 50}- )- self.assertIsNotNone(result)-@parameterized.expand([(@@ -72,27 +73,54 @@ class TestEvaluateBiasRisk(TestCase):),("empty_variants",- [],+ (),MultipleVariantHandling.EXCLUDE,{"control": 800, "test": 200, "$multiple": 50},),+ (+ "variant_at_zero_beside_an_even_split",+ _split(50, 50, 0),+ MultipleVariantHandling.EXCLUDE,+ {"control": 500, "test": 500, "$multiple": 50},+ ),+ (+ "one_variant_at_100",+ _split(0, 100),+ MultipleVariantHandling.EXCLUDE,+ {"control": 500, "test": 500, "$multiple": 50},+ ),+ (+ "fewer_users_in_several_variants_than_the_floor",+ UNEVEN_2WAY,+ MultipleVariantHandling.EXCLUDE,+ {"control": 80, "test": 20, "$multiple": 4},+ ),])- def test_returns_none_when_not_at_risk(self, _name, rollout_percentages, handling, exposures):- result = evaluate_bias_risk(rollout_percentages, handling, exposures)+ def test_returns_none_when_not_at_risk(self, _name, variants, handling, exposures):+ result = evaluate_bias_risk(variants, frozenset(), handling, exposures)+ self.assertIsNone(result)++ def test_excluded_variant_leaves_an_even_split_of_the_compared_variants(self):+ result = evaluate_bias_risk(+ _split(50, 25, 25),+ frozenset({"control"}),+ MultipleVariantHandling.EXCLUDE,+ {"test": 500, "third": 500, "$multiple": 50},+ )self.assertIsNone(result)def test_threshold_boundary_strictly_greater_than(self):- # Exactly at threshold (1 / 1000 = 0.1%) should NOT trigger — uses strict `>`.+ # Exactly at threshold (10 / 10000 = 0.1%) should NOT trigger — uses strict `>`.result = evaluate_bias_risk(- UNEVEN_2WAY, MultipleVariantHandling.EXCLUDE, {"control": 799, "test": 200, "$multiple": 1}+ UNEVEN_2WAY, frozenset(), MultipleVariantHandling.EXCLUDE, {"control": 7990, "test": 2000, "$multiple": 10})self.assertIsNone(result)def test_threshold_boundary_just_above(self):- # 2 / 1000 = 0.2% — above the 0.1% threshold.+ # 11 / 10000 = 0.11% — above the 0.1% threshold.result = evaluate_bias_risk(- UNEVEN_2WAY, MultipleVariantHandling.EXCLUDE, {"control": 798, "test": 200, "$multiple": 2}+ UNEVEN_2WAY, frozenset(), MultipleVariantHandling.EXCLUDE, {"control": 7989, "test": 2000, "$multiple": 11})assert result is not Noneself.assertGreater(result.multiple_variant_percentage, MULTIPLE_VARIANT_BIAS_THRESHOLD)diff --git a/products/experiments/backend/health/test/test_flag_state.py b/products/experiments/backend/health/test/test_flag_state.pyindex 5e631160675d..3de6609a00bb 100644--- a/products/experiments/backend/health/test/test_flag_state.py+++ b/products/experiments/backend/health/test/test_flag_state.py@@ -2,8 +2,12 @@ from dataclasses import replacefrom datetime import UTC, datetime, timedeltafrom typing import Any+import time_machine+from posthog.test.base import BaseTestfrom unittest import TestCase+from django.test import override_settings+from parameterized import parameterizedfrom products.experiments.backend.facade.contracts import (@@ -14,15 +18,20 @@ from products.experiments.backend.facade.contracts import (from products.experiments.backend.health.checks.flag_state import flag_statefrom products.experiments.backend.health.context import (ConditionSetTraffic,+ ExperimentOnFlag,HealthContext,PauseHistory,+ load_health_context,parse_flag_state,)from products.experiments.backend.health.test.factories import DRAFT, ENDED, RUNNING, RUNNING_CONTEXT, Status+from products.experiments.backend.models.experiment import Experiment+from products.feature_flags.backend.models.feature_flag import FeatureFlag# The flag documents and cases of the `experimentWarning` test in# frontend/src/scenes/experiments/experimentLogic.test.ts, so that both sides answer alike. A running-# experiment with a deleted flag is the exception: only the server reports it.+# experiment with a deleted flag is one exception: only the server reports it. An ended experiment is another:+# only the server reads the flag's calls and the other experiments on the flag.MULTIVARIANT = {"groups": [{"properties": [], "rollout_percentage": 100}],"multivariate": {@@ -76,6 +85,13 @@ PAUSED_AT = datetime(2026, 10, 12, 13, 4, tzinfo=UTC)ENDED_AT = datetime(2026, 10, 14, 8, 30, tzinfo=UTC)+def _other(*, launched_at: datetime | None, ended_at: datetime | None = None) -> ExperimentOnFlag:+ return ExperimentOnFlag(launched_at=launched_at, ended_at=ended_at, archived=False)+++DRAFT_ON_FLAG = _other(launched_at=None)++def _context(status: Status,*,@@ -86,6 +102,8 @@ def _context(traffic: ConditionSetTraffic | None = None,pause_history: PauseHistory | None = None,flag_called_after_end: bool | None = None,+ flag_last_called_at: datetime | None = None,+ other_experiments: tuple[ExperimentOnFlag, ...] = (),) -> HealthContext:return replace(RUNNING_CONTEXT,@@ -94,6 +112,8 @@ def _context(is_paused=status["is_running"] and not active,archived=archived,flag_called_after_end=flag_called_after_end,+ flag_last_called_at=flag_last_called_at,+ other_experiments_on_flag=other_experiments if status["has_ended"] else None,flag=parse_flag_state(active=active,deleted=deleted,@@ -147,7 +167,7 @@ class TestFlagState(TestCase):("ended_flag_distributing_variants",_context(ENDED, active=True, filters=MULTIVARIANT),- (ExperimentHealthFindingCode.FLAG_LIVE_AFTER_END, "ended_but_multiple_variants_rolled_out", None),+ (ExperimentHealthFindingCode.FLAG_LIVE_AFTER_END, "ended_flag_calls_unknown", None),),("ended_flag_not_called_since_the_end",@@ -159,12 +179,60 @@ class TestFlagState(TestCase):_context(ENDED, active=True, filters=MULTIVARIANT, flag_called_after_end=True),(ExperimentHealthFindingCode.FLAG_LIVE_AFTER_END, "ended_but_multiple_variants_rolled_out", None),),+ (+ "ended_flag_serves_a_running_experiment",+ _context(+ ENDED,+ active=True,+ filters=MULTIVARIANT,+ flag_called_after_end=True,+ other_experiments=(_other(launched_at=ENDED_AT - timedelta(days=3)),),+ ),+ None,+ ),+ (+ "ended_flag_served_a_later_experiment_that_ended",+ _context(+ ENDED,+ active=True,+ filters=MULTIVARIANT,+ flag_called_after_end=True,+ other_experiments=(+ _other(launched_at=ENDED_AT + timedelta(days=1), ended_at=ENDED_AT + timedelta(days=5)),+ ),+ ),+ None,+ ),+ (+ "ended_flag_served_an_earlier_experiment",+ _context(+ ENDED,+ active=True,+ filters=MULTIVARIANT,+ flag_called_after_end=True,+ other_experiments=(+ _other(launched_at=ENDED_AT - timedelta(days=30), ended_at=ENDED_AT - timedelta(days=20)),+ ),+ ),+ (ExperimentHealthFindingCode.FLAG_LIVE_AFTER_END, "ended_but_multiple_variants_rolled_out", None),+ ),+ (+ "ended_flag_shared_with_a_draft",+ _context(+ ENDED,+ active=True,+ filters=MULTIVARIANT,+ flag_called_after_end=True,+ other_experiments=(DRAFT_ON_FLAG,),+ ),+ (ExperimentHealthFindingCode.FLAG_LIVE_AFTER_END, "ended_flag_shared_with_draft", None),+ ),("ended_zero_rollout", _context(ENDED, active=True, filters=ZERO_ROLLOUT), None),("ended_flag_disabled", _context(ENDED, active=False, filters=MULTIVARIANT), None),("ended_single_variant_shipped", _context(ENDED, active=True, filters=SHIPPED_VARIANT), None),("archived_ended_flag_distributing_variants",- _context(ENDED, active=True, filters=MULTIVARIANT, archived=True),+ _context(ENDED, active=True, filters=MULTIVARIANT, archived=True, flag_called_after_end=True),(ExperimentHealthFindingCode.FLAG_LIVE_AFTER_END, "ended_but_multiple_variants_rolled_out", None),),(@@ -297,11 +365,26 @@ class TestFlagState(TestCase):"ended_flag_calls_after_the_end_unknown",_context(ENDED, active=True, filters=MULTIVARIANT),(- "ended_but_multiple_variants_rolled_out",- ExperimentHealthFindingSeverity.WARNING,+ "ended_flag_calls_unknown",+ ExperimentHealthFindingSeverity.INFO,"The experiment has ended, but its feature flag is still active with multiple variants",),),+ (+ "ended_flag_shared_with_a_draft",+ _context(+ ENDED,+ active=True,+ filters=MULTIVARIANT,+ flag_called_after_end=True,+ other_experiments=(DRAFT_ON_FLAG,),+ ),+ (+ "ended_flag_shared_with_draft",+ ExperimentHealthFindingSeverity.INFO,+ "The feature flag is still active, and a draft experiment uses it",+ ),+ ),("running_zero_rollout_traffic_unknown",_context(RUNNING, active=True, filters=ZERO_ROLLOUT),@@ -363,6 +446,37 @@ class TestFlagState(TestCase):"payload or a list of all flags records no call, and neither does a call with send_event set to ""false, so your code may still read the flag that way. If nothing reads the flag anymore, disable or ""archive it to clean up.",+ {"ended_at": "2026-10-14T08:30:00+00:00", "last_called_at": None},+ ),+ (+ "flag_last_called_after_the_end",+ _context(+ ENDED,+ active=True,+ filters=MULTIVARIANT,+ flag_called_after_end=False,+ flag_last_called_at=datetime(2026, 10, 17, 18, 0, tzinfo=UTC),+ ),+ "The experiment ended on October 14, 2026 at 10:30 AM CEST, and the linked feature flag is still "+ "active with multiple variants. PostHog last recorded a call of the flag on October 17, 2026 at 8:00 "+ "PM CEST, and none in the last 3 days. Reading only a payload or a list of all flags records no call, "+ "and neither does a call with send_event set to false, so your code may still read the flag that way. "+ "If nothing reads the flag anymore, disable or archive it to clean up.",+ {"ended_at": "2026-10-14T08:30:00+00:00", "last_called_at": "2026-10-17T18:00:00+00:00"},+ ),+ (+ "flag_shared_with_a_draft",+ _context(+ ENDED,+ active=True,+ filters=MULTIVARIANT,+ flag_called_after_end=True,+ other_experiments=(DRAFT_ON_FLAG,),+ ),+ "The experiment ended on October 14, 2026 at 10:30 AM CEST, and the linked feature flag is still "+ "active with multiple variants. A draft experiment uses the same flag, so users get its variants "+ "whenever your code reads the flag, before that experiment launches. This is expected while you test "+ "its variants. Otherwise, disable the flag until you launch it.",{"ended_at": "2026-10-14T08:30:00+00:00"},),]@@ -392,3 +506,44 @@ class TestFlagState(TestCase):{"people_assigned_outside": 1234, "assigned_outside_since": "2026-10-12T14:04:00+00:00"},),)+++class TestFlagStateOfAnEndedExperiment(BaseTest):+ @parameterized.expand(+ [+ ("running_experiment_on_the_flag", False, None),+ ("deleted_experiment_on_the_flag", True, "ended_but_multiple_variants_rolled_out"),+ ]+ )+ def test_reads_the_other_experiments_on_the_flag(+ self, _name: str, other_deleted: bool, expected_subcode: str | None+ ) -> None:+ flag = FeatureFlag.objects.create(+ team=self.team,+ key="checkout-test",+ created_by=self.user,+ last_called_at=datetime(2026, 10, 19, tzinfo=UTC),+ filters=MULTIVARIANT,+ )+ experiment = Experiment.objects.create(+ team=self.team,+ name="exp",+ feature_flag=flag,+ start_date=datetime(2026, 10, 1, tzinfo=UTC),+ end_date=datetime(2026, 10, 10, tzinfo=UTC),+ )+ Experiment.objects.create(+ team=self.team,+ name="next",+ feature_flag=flag,+ start_date=datetime(2026, 10, 12, tzinfo=UTC),+ deleted=other_deleted,+ )++ with (+ override_settings(FEATURE_FLAG_LAST_CALLED_AT_SYNC_SOURCE="events"),+ time_machine.travel(datetime(2026, 10, 20, 12, 0, tzinfo=UTC), tick=False),+ ):+ finding = flag_state(load_health_context(experiment))++ self.assertEqual(finding.subcode if isinstance(finding, ExperimentHealthFinding) else finding, expected_subcode)diff --git a/products/experiments/backend/health/test/test_zero_exposures.py b/products/experiments/backend/health/test/test_zero_exposures.pyindex 647733dbb431..33a766c8f645 100644--- a/products/experiments/backend/health/test/test_zero_exposures.py+++ b/products/experiments/backend/health/test/test_zero_exposures.py@@ -353,12 +353,21 @@ class TestFlagCalledDuringRun(BaseTest):False,),(- "ended_called_after_the_end",+ "ended_called_after_the_end_but_not_lately",LAUNCHED_AT,ENDED_AT,datetime(2026, 10, 15, tzinfo=UTC),FlagEvaluationsMode.EVENTS,None,+ False,+ ),+ (+ "ended_called_lately",+ LAUNCHED_AT,+ ENDED_AT,+ datetime(2026, 10, 19, tzinfo=UTC),+ FlagEvaluationsMode.EVENTS,+ None,True,),("ended_never_called", LAUNCHED_AT, ENDED_AT, None, FlagEvaluationsMode.EVENTS, False, False),diff --git a/products/experiments/backend/hogql_queries/experiment_exposures_query_runner.py b/products/experiments/backend/hogql_queries/experiment_exposures_query_runner.pyindex 5ffd74509c89..1a9044e223f7 100644--- a/products/experiments/backend/hogql_queries/experiment_exposures_query_runner.py+++ b/products/experiments/backend/hogql_queries/experiment_exposures_query_runner.py@@ -88,6 +88,7 @@ from products.experiments.backend.hogql_queries.exposure_query_logic import (from products.experiments.backend.hogql_queries.exposures_precompute import ExposuresPrecomputefrom products.experiments.backend.models.experiment import Experimentfrom products.experiments.backend.models.team_experiments_config import TeamExperimentsConfig+from products.feature_flags.backend.facade.edits import parse_variantslogger = structlog.get_logger(__name__)@@ -297,17 +298,17 @@ class ExperimentExposuresQueryRunner(ExperimentResultsCacheMixin, QueryRunner):def _evaluate_bias_risk(self, total_exposures: dict[str, int], multiple_variant_handling: MultipleVariantHandling) -> BiasRisk | None:- # Shipping a variant rewrites the flag to 100/0, which would falsely trip the- # uneven-split check on data collected under the original split. The warning also has- # no action after the experiment stops, because both of its CTAs only apply while it runs.+ # After the end the flag often holds the shipped decision, not the split that the data+ # was collected under. The warning also has no action after the experiment stops,+ # because both of its CTAs only apply while it runs.# Read the end from the query (the cache key), so that the running/stopped decision# matches the cached window.if self.window_end_date is not None:return Nonemultivariate_data = (self.query.feature_flag.get("filters") or {}).get("multivariate") or {}- flag_variants = multivariate_data.get("variants") or []return evaluate_bias_risk(- rollout_percentages=[variant.get("rollout_percentage", 0) for variant in flag_variants],+ variants=parse_variants(multivariate_data.get("variants")),+ excluded_variant_keys=self.excluded_variants,multiple_variant_handling=multiple_variant_handling,total_exposures=total_exposures,)@@ -609,7 +610,7 @@ class ExperimentExposuresQueryRunner(ExperimentResultsCacheMixin, QueryRunner):def get_cache_payload(self) -> dict:payload = super().get_cache_payload()- payload["experiment_exposures_response_version"] = 4+ payload["experiment_exposures_response_version"] = 5# The sample ratio finding can name a split change from the flag's history, so a reader who can't read the# history, a userless run included, must not get a cached answer of one who can.if not self._can_read_flag_history():diff --git a/products/experiments/backend/hogql_queries/test/experiment_query_runner/test_experiment_exposures_query_runner.py b/products/experiments/backend/hogql_queries/test/experiment_query_runner/test_experiment_exposures_query_runner.pyindex 126e42048432..425746867fdf 100644--- a/products/experiments/backend/hogql_queries/test/experiment_query_runner/test_experiment_exposures_query_runner.py+++ b/products/experiments/backend/hogql_queries/test/experiment_query_runner/test_experiment_exposures_query_runner.py@@ -2637,18 +2637,16 @@ class TestExperimentExposuresQueryRunner(ExperimentQueryRunnerBaseTest):self.assertLess(result.p_value, 0.01)def test_bias_risk_skipped_when_experiment_has_ended(self):- # Shipping a variant rewrites the flag to 100/0 — that uneven post-ship split- # plus historical $multiple exposures would otherwise produce a false positive.feature_flag = FeatureFlag.objects.create(- name="Test flag (post-ship)",+ name="Test flag (uneven)",key="test-bias-stopped",team=self.team,filters={"groups": [{"properties": [], "rollout_percentage": None}],"multivariate": {"variants": [- {"key": "control", "rollout_percentage": 100},- {"key": "test", "rollout_percentage": 0},+ {"key": "control", "rollout_percentage": 80},+ {"key": "test", "rollout_percentage": 20},]},},diff --git a/products/experiments/backend/models/experiment.py b/products/experiments/backend/models/experiment.pyindex ca1269074a45..5242c4631811 100644--- a/products/experiments/backend/models/experiment.py+++ b/products/experiments/backend/models/experiment.py@@ -331,7 +331,7 @@ class Experiment(Taggable, FileSystemSyncMixin, ModelActivityMixin, RootTeamMixi)-def _live_experiments_for_flag(feature_flag_id: Any) -> "QuerySet[Experiment]":+def live_experiments_for_flag(feature_flag_id: Any) -> "QuerySet[Experiment]":"""Single source of truth for the `has_experiment` predicate: a flag has a linkedexperiment iff a non-deleted Experiment row references it. The Rust flags service(rust/feature-flags/src/flags/feature_flag_list.rs) mirrors this in hand-written SQL —@@ -344,12 +344,12 @@ def _live_experiments_for_flag(feature_flag_id: Any) -> "QuerySet[Experiment]":def live_experiment_exists() -> Exists:"""`Exists` subquery for `.annotate()` over many flags. For a single in-hand flag, use `flag_has_live_experiment`."""- return Exists(_live_experiments_for_flag(OuterRef("pk")))+ return Exists(live_experiments_for_flag(OuterRef("pk")))def flag_has_live_experiment(feature_flag_id: int) -> bool:"""Instance-level companion to `live_experiment_exists` — same predicate, one flag."""- return _live_experiments_for_flag(feature_flag_id).exists()+ return live_experiments_for_flag(feature_flag_id).exists()def metric_display_rank(ordered_uuids: Sequence[str] | None) -> Callable[[str], int]:diff --git a/products/experiments/backend/presentation/serializers.py b/products/experiments/backend/presentation/serializers.pyindex 410777a43cdf..c79c1360c9ff 100644--- a/products/experiments/backend/presentation/serializers.py+++ b/products/experiments/backend/presentation/serializers.py@@ -3633,10 +3633,10 @@ class ExperimentHealthInputsSerializer(serializers.Serializer):flag_called_after_end = serializers.BooleanField(allow_null=True,help_text=(- "Whether PostHog has recorded a call of the flag after the end. The flag state check reads it on an ended "- "experiment whose flag is still active with multiple variants. False only after 3 days without a "- "recorded call. Null before the end, before those 3 days are over, or when the organization's flag "- "calls can't be read."+ "Whether PostHog has recorded a call of the flag after the end and within the last 3 days. The flag "+ "state check reads it on an ended experiment whose flag is still active with multiple variants. False "+ "only once 3 days have passed since the end. Null before the end, before those 3 days are over without a "+ "call, or when the organization's flag calls can't be read."),)exposure_events = ExperimentHealthInputsExposureEventsSerializer(diff --git a/products/experiments/backend/presentation/views.py b/products/experiments/backend/presentation/views.pyindex d918071cf58c..1f62bba90005 100644--- a/products/experiments/backend/presentation/views.py+++ b/products/experiments/backend/presentation/views.py@@ -1080,10 +1080,14 @@ class EnterpriseExperimentsViewSet(`running_but_flag_disabled`, as a warning. A caller who can't view the linked flag'shistory gets the warning, because the pause is not read. A draft whose flag is alreadyactive with multiple variants gets `not_started_but_multiple_variants_rolled_out` as info.- An ended experiment whose flag is still active with multiple variants gets- `ended_but_multiple_variants_rolled_out` as a warning, or `ended_flag_not_called` as info- when the flag's `last_called_at` shows no call in the 3 days after the end. The info waits- an extra hour, because `last_called_at` trails a call.+ An ended experiment whose flag is still active with multiple variants gets no finding when+ another experiment on the flag is running or launched after the end. Otherwise it gets+ `ended_flag_shared_with_draft` as info when a draft experiment uses the flag,+ `ended_but_multiple_variants_rolled_out` as a warning when the flag's `last_called_at` is+ after the end and within the last 3 days, and `ended_flag_not_called` as info when it is+ not, from 3 days and an hour after the end, because `last_called_at` trails a call. Until+ then without such a call, or when the organization's flag calls can't be read, it gets+ `ended_flag_calls_unknown` as info.A running experiment whose first primary metric is a funnel gets the info`early_significance` (subcode `below_detectable_effect`) when a significant variant'sdiff --git a/products/experiments/frontend/generated/api.schemas.ts b/products/experiments/frontend/generated/api.schemas.tsindex 5c200dd3a89e..61e69481a6e4 100644--- a/products/experiments/frontend/generated/api.schemas.ts+++ b/products/experiments/frontend/generated/api.schemas.ts@@ -3786,7 +3786,7 @@ export interface ExperimentHealthInputsApi {*/flag_called_during_run: boolean | null/**- * Whether PostHog has recorded a call of the flag after the end. The flag state check reads it on an ended experiment whose flag is still active with multiple variants. False only after 3 days without a recorded call. Null before the end, before those 3 days are over, or when the organization's flag calls can't be read.+ * Whether PostHog has recorded a call of the flag after the end and within the last 3 days. The flag state check reads it on an ended experiment whose flag is still active with multiple variants. False only once 3 days have passed since the end. Null before the end, before those 3 days are over without a call, or when the organization's flag calls can't be read.* @nullable*/flag_called_after_end: boolean | nulldiff --git a/products/experiments/frontend/generated/api.ts b/products/experiments/frontend/generated/api.tsindex 7c1525836fdb..c51e981ffff3 100644--- a/products/experiments/frontend/generated/api.ts+++ b/products/experiments/frontend/generated/api.ts@@ -825,10 +825,14 @@ export const getExperimentsHealthRetrieveUrl = (projectId: string, id: number) =* `running_but_flag_disabled`, as a warning. A caller who can't view the linked flag's* history gets the warning, because the pause is not read. A draft whose flag is already* active with multiple variants gets `not_started_but_multiple_variants_rolled_out` as info.- * An ended experiment whose flag is still active with multiple variants gets- * `ended_but_multiple_variants_rolled_out` as a warning, or `ended_flag_not_called` as info- * when the flag's `last_called_at` shows no call in the 3 days after the end. The info waits- * an extra hour, because `last_called_at` trails a call.+ * An ended experiment whose flag is still active with multiple variants gets no finding when+ * another experiment on the flag is running or launched after the end. Otherwise it gets+ * `ended_flag_shared_with_draft` as info when a draft experiment uses the flag,+ * `ended_but_multiple_variants_rolled_out` as a warning when the flag's `last_called_at` is+ * after the end and within the last 3 days, and `ended_flag_not_called` as info when it is+ * not, from 3 days and an hour after the end, because `last_called_at` trails a call. Until+ * then without such a call, or when the organization's flag calls can't be read, it gets+ * `ended_flag_calls_unknown` as info.** A running experiment whose first primary metric is a funnel gets the info* `early_significance` (subcode `below_detectable_effect`) when a significant variant'sdiff --git a/products/experiments/frontend/health/debug/healthDebugInputs.ts b/products/experiments/frontend/health/debug/healthDebugInputs.tsindex 20d5deb311b7..62652c0a33d6 100644--- a/products/experiments/frontend/health/debug/healthDebugInputs.ts+++ b/products/experiments/frontend/health/debug/healthDebugInputs.ts@@ -155,7 +155,7 @@ export function buildFlagFacts(experiment: Experiment, state: HealthInputsState){label: 'Called after the end',value: serverValue(state, ({ flag_called_after_end }) => yesNoUnknown(flag_called_after_end)),- help: "When this is No, the flag-state finding for an ended experiment is info. No needs 3 days without a recorded call after the end. Unknown before the end, before those 3 days are over, or when the organization's flag calls skip the events table.",+ help: "Yes needs a recorded call after the end and in the last 3 days. Only Yes makes the flag-state finding for an ended experiment a warning. No only once 3 days have passed since the end. Unknown before the end, before those 3 days are over without a call, or when the organization's flag calls skip the events table.",},]}diff --git a/products/experiments/skills/debugging-experiments/references/pulling-the-data.md b/products/experiments/skills/debugging-experiments/references/pulling-the-data.mdindex 9f1ff46a4de1..4b04ea01559b 100644--- a/products/experiments/skills/debugging-experiments/references/pulling-the-data.md+++ b/products/experiments/skills/debugging-experiments/references/pulling-the-data.md@@ -129,7 +129,7 @@ loaded get `false`/`undefined` and are dropped from their arm instead of showing| Cause | The number that confirms it || ----------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------- |-| Uneven split + Exclude bias | uneven `rollout_percentage` split **and** `$multiple` share > 0.1% (banner threshold), handling = `exclude` |+| Uneven split + Exclude bias | uneven split of the served, analyzed variants **and** `$multiple` share > 0.1% from 5+ people (banner threshold), handling = `exclude` || Sample ratio mismatch | chi-squared p < 0.001 on §2's per-person `total_exposures` (see below); only meaningful once totals are healthy || Assignment override (assignment-side SRM) | the reassignment component carries the gap in the decisive test below; then localize with the SDK split / bootstrap mix below || Capture-by-surface (capture-side SRM) | a `$pathname`/`$screen_name` where one variant's share jumps to ~100% while other paths sit near the split |diff --git a/products/experiments/skills/diagnosing-experiment-health/references/bias-and-skew.md b/products/experiments/skills/diagnosing-experiment-health/references/bias-and-skew.mdindex cddfa93c7a52..73cbf04dd39a 100644--- a/products/experiments/skills/diagnosing-experiment-health/references/bias-and-skew.md+++ b/products/experiments/skills/diagnosing-experiment-health/references/bias-and-skew.md@@ -66,9 +66,9 @@ This is the signal behind the in-app warning "The uneven split may bias the resu**Evidence.** The banner's check reports a finding when **all four** hold:- `multiple_variant_handling == "exclude"` (the default)-- the variant rollouts are uneven.- A variant at 0% counts, so a flag at 0/50/50 is uneven.-- the share of people in more than one variant is above **0.1%**.+- the variants that the flag serves and the analysis keeps have shares more than one point apart.+ A variant at 0% and an excluded variant don't count, so a flag at 0/50/50 is even, and a flag that serves one variant gets no finding.+- the share of people in more than one variant is above **0.1%**, and at least 5 people are in more than one variant.The value is `exposures.bias_risk.multiple_variant_percentage`: the `$multiple` count over all exposures.- the experiment has no end date.After the end the check reports nothing, and the banner is gone.@@ -379,6 +379,11 @@ Five states exist, checked from `experiment-get` alone:| "The experiment is not running, but users are exposed to multiple variants" | Ended, the flag is active, more than one variant is above 0%, and a release condition is above 0% | Disable the flag, or resume the experiment || "The experiment is not running, but users are exposed to multiple variants" | Not launched (also after a reset), not archived, the flag is active, more than one variant is above 0%, and a release condition is above 0% | Disable the flag, or start the experiment |+The table shows the page's own rules. When the page shows the health findings, it takes the banner from the server's flag-state finding instead. For an ended experiment, the `flag_live_after_end` finding also reads the flag's calls and the other experiments on the flag:++- When another experiment on the flag is running, or launched after this one ended, the flag's state belongs to that experiment, and the finding is absent. Do not advise disabling the flag from this experiment, even when the banner says so. Disabling it stops a running experiment.+- A draft experiment on the flag, no recorded call of the flag in the last 3 days, or calls that PostHog can't read yet make the finding info, not a warning. A page that shows the health findings shows no banner then.+"Adjust the variant distribution" on a running experiment is a change of the split, with what that does to the people already exposed (E3 in `mid-run-changes.md`).Say so before the user restores a split.diff --git a/services/mcp/src/api/generated.ts b/services/mcp/src/api/generated.tsindex 9a45d0ff8182..dda705360bff 100644--- a/services/mcp/src/api/generated.ts+++ b/services/mcp/src/api/generated.ts@@ -49476,7 +49476,7 @@ export namespace Schemas {*/flag_called_during_run: boolean | null;/**- * Whether PostHog has recorded a call of the flag after the end. The flag state check reads it on an ended experiment whose flag is still active with multiple variants. False only after 3 days without a recorded call. Null before the end, before those 3 days are over, or when the organization's flag calls can't be read.+ * Whether PostHog has recorded a call of the flag after the end and within the last 3 days. The flag state check reads it on an ended experiment whose flag is still active with multiple variants. False only once 3 days have passed since the end. Null before the end, before those 3 days are over without a call, or when the organization's flag calls can't be read.* @nullable*/flag_called_after_end: boolean | null;