Skip to content

:hedgehog: PostHog is the leading platform for building self-driving products. Our developer tools – AI observability, analytics, session replay, flags, experiments, error tracking, logs, and more – capture all the context agents need to diagnose problems, uncover opportunities, and ship fixes. Steer it all from Slack, web, desktop, or the MCP.

feat(flags): accept experiment rules in v2 writes (#113154)

Andersc0e59692792a
diff --git a/docs/internal/feature-flags/api-writes.md b/docs/internal/feature-flags/api-writes.md
index 84921dde3722..c8ff3ab895d9 100644
--- a/docs/internal/feature-flags/api-writes.md
+++ b/docs/internal/feature-flags/api-writes.md
@@ -83,10 +83,12 @@ A stored version 2 row that is not remote-configured and carries no encrypted pa
- A new row is created disabled with server-owned rule IDs and seeds; the request document is resolved against an empty stored document and validated strictly before it is persisted exactly. No v1 row is converted.
- A supplied `filters` object replaces the whole configuration. There is no top-level merge, no rule merge by index, no filling of omitted fields from the stored document, and no v1 normalization. `{}`, `null`, a list, and an incomplete object are invalid; a complete document with `rules: []` is valid.
-- `return_type` is `boolean`, `string`, `number` or `object` and cannot change after creation. `default_value` is null or the declared type; every rule `value` is the declared type. Strings are non-empty and not `$false`, `$true` or `"null"`, the values v1 also reserves as variant keys; numbers lie within ±(2^53 − 1); objects nest at most 20 object or array levels with numbers in the same range.
-- Writer-only rules (`check_writer_rules` in `facade/config_writes.py`, currently the `$false`, `$true` and `"null"` reservation and the regex check below) apply to a created or replacing document and to the stored document being enabled. The shared validator, and with it the flags caches and the stored side of an update, does not apply them: tightening a writer rule never hides a stored row from evaluation or blocks replacing it. A `"null"` value the stored document already serves passes the reservation, as in v1.
+- `return_type` is `boolean`, `string`, `number` or `object` and cannot change after creation. `default_value` is null or the declared type; every rule `value` and variant `value` is the declared type. Strings are non-empty and not `$false`, `$true` or `"null"`, the values v1 also reserves as variant keys; numbers lie within ±(2^53 − 1); objects nest at most 20 object or array levels with numbers in the same range.
+- Writer-only rules (`check_writer_rules` in `facade/config_writes.py`, currently the `$false`, `$true` and `"null"` reservation and the regex check below) apply to a created or replacing document and to the stored document being enabled. The shared validator, and with it the flags caches and the stored side of an update, does not apply them: tightening a writer rule never hides a stored row from evaluation or blocks replacing it. A `"null"` value the stored document already serves, as the default, a rule value or a variant value, passes the reservation, as in v1.
+- Experiment-rule variant keys follow the v1 variant-key rule for every return type: a new `"null"` key is rejected, writer-only, unless the stored rule with the same ID already uses that key. The key belongs to that rule's experiment, so another rule cannot take it over.
- `regex` and `not_regex` patterns must compile with Python's `re`, as on the v1 write path. This is a writer-only rule: readers store an uncompilable pattern and report an evaluation error when evaluation reaches it. The flags service compiles with `fancy_regex`, whose dialect differs at the edges, so, as in v1, a pattern the stored document already holds is not checked again, and enabling checks no pattern.
-- Rule IDs and percentage-rollout seeds are server-owned and identify rules rather than list positions. An echoed ID keeps its seed through reordering and unrelated edits; a rule that omits its ID is new and receives fresh identity. An unknown ID, an ID used twice, a client-chosen seed, and a changed seed are all rejected.
+- Rule types are `targeted_release`, `percentage_rollout` and `experiment`. An experiment rule names no Experiment: an Experiment links to its rule through the rule ID (`Experiment.feature_flag_rule_id`), and writing the rule creates or links no Experiment row. The rule splits the subjects its `rollout_percentage` enrolls across two to twenty `variants` (unique keys of letters, digits, `-` and `_`; weights with at most two decimal places that total exactly 100; values of the flag's return type, equal values allowed), returns the flag default while `paused` is true, and may carry a `holdout` that belongs to the rule: `{seed, exclusion_percentage}`, where the client chooses `exclusion_percentage` and the server owns the seed. At `/flags?v=3` a variant assignment reports `experiment_split` with the rule ID and the variant key; no response carries an experiment or holdout ID. An `experiment_id` on the rule and an `id` in its holdout are unknown fields, whatever their value, `null` included.
+- Rule IDs, the seeds of percentage rollouts and experiment rules, and holdout seeds are server-owned and identify rules rather than list positions. An echoed ID keeps its seeds through reordering and unrelated edits, and a rule keeps its seed while it stays a percentage rollout or an experiment rule; a rule that omits its ID is new and receives fresh identity. An unknown ID, an ID used twice, a client-chosen seed, and a changed seed are all rejected.
- `FeatureFlag.version` is required on every update and must equal the locked row's value. Unlike v1 there is no `original_flag` merge: a stale token is a conflict even when no individual field clashes. The token applies to disabling and deleting too.
- Identity resolution, validation and the enable check run against the locked row inside the existing update transaction. A stored document with an unsupported shape or semantics is rejected rather than replaced or enabled. Configuration and metadata byte limits apply to a replacement, so an oversized stored document can be edited back under a lowered limit; enabling requires the stored document to fit the current limits.
- Request bytes that repeat a JSON key are rejected before normalization can hide the duplicate. Non-finite numbers and percentages with more than two decimal places are rejected by the validator itself.
@@ -98,4 +100,6 @@ A stored version 2 row that is not remote-configured and carries no encrypted pa
Rollback turns both flags off. Existing version 2 rows stay readable, disableable and deletable; there is no data conversion, and v1-to-v2 dependency protection is not removed.
+A flags service that predates experiment rules fails a flag that has one, in the same way as below. Deploy the flags service that evaluates them before an admitted project writes one, and disable every such enabled flag before reverting that service.
+
A flags service that predates string, number and object values fails such a flag: the flag gets an error entry and the whole `/flags` response sets `errorsWhileComputingFlags`, so posthog-js merges it into its cached flags instead of replacing them. Deploy the flags service that evaluates these types before an admitted project creates one. Turning the writer flags off stops new ones but leaves enabled ones served; disable every enabled string, number and object flag before reverting the flags service.
diff --git a/frontend/src/types.ts b/frontend/src/types.ts
index 81bdf4fb558f..8c44d396892a 100644
--- a/frontend/src/types.ts
+++ b/frontend/src/types.ts
@@ -4528,7 +4528,7 @@ export interface FeatureFlagRulesV2ExperimentRule extends FeatureFlagRulesV2Rule
rule_type: 'experiment'
paused: boolean
variants: { key: string; weight: number; value: JsonType }[]
- holdout?: { id: number; seed: string; exclusion_percentage: number }
+ holdout?: { seed: string; exclusion_percentage: number }
}
export type FeatureFlagRulesV2Rule =
diff --git a/posthog/api/test/test_flag_value.py b/posthog/api/test/test_flag_value.py
index 086522fce2ea..363f7c468c7d 100644
--- a/posthog/api/test/test_flag_value.py
+++ b/posthog/api/test/test_flag_value.py
@@ -36,6 +36,20 @@ def rollout_rule(rule_id: str, value: Any) -> dict[str, Any]:
}
+def split_rule(rule_id: str, *values: Any) -> dict[str, Any]:
+ return {
+ "id": rule_id,
+ "rule_type": "experiment",
+ "targeting": {"properties": []},
+ "paused": False,
+ "rollout_percentage": 100,
+ "on_rollout_miss": "continue",
+ "assignment_algorithm": "sha1_60_v1",
+ "seed": "values-endpoint-split",
+ "variants": [{"key": f"arm_{index}", "weight": 50, "value": value} for index, value in enumerate(values)],
+ }
+
+
def v2_document(return_type: str, default_value: Any, *rules: dict[str, Any]) -> dict[str, Any]:
return {"version": 2, "return_type": return_type, "default_value": default_value, "rules": list(rules)}
@@ -76,6 +90,11 @@ class TestFlagValueViewSet(APIBaseTest):
["compact"],
),
("no_rules", v2_document("string", "standard"), ["standard"]),
+ (
+ "variant_values_in_stored_order",
+ v2_document("string", "standard", targeted_rule(RULE_A, "wide"), split_rule(RULE_B, "compact", "wide")),
+ ["wide", "compact", "standard"],
+ ),
]
)
def test_flag_values_v2_string_flag_lists_its_strings(
@@ -98,6 +117,12 @@ class TestFlagValueViewSet(APIBaseTest):
[
("v2_without_rules_key", {"version": 2, "return_type": "string", "default_value": "standard"}),
("v2_rules_not_a_list", {"version": 2, "return_type": "string", "default_value": None, "rules": "x"}),
+ (
+ "v2_property_not_an_object",
+ v2_document(
+ "string", "standard", {**targeted_rule(RULE_A, "wide"), "targeting": {"properties": ["x"]}}
+ ),
+ ),
(
"unsupported_version",
{"version": 3, "multivariate": {"variants": [{"key": "variant1", "rollout_percentage": 100}]}},
diff --git a/products/feature_flags/backend/api/flag_value.py b/products/feature_flags/backend/api/flag_value.py
index 7f8ac1714b4c..f99102c617f2 100644
--- a/products/feature_flags/backend/api/flag_value.py
+++ b/products/feature_flags/backend/api/flag_value.py
@@ -100,9 +100,12 @@ def _v2_string_values(filters: Mapping[str, Any]) -> list[str]:
"""
try:
config = parse_v2_config(filters)
- except (KeyError, TypeError, ValueError):
+ except (KeyError, TypeError, ValueError, AttributeError):
return []
if config.return_type != "string":
return []
- candidates = [*(rule.value for rule in config.rules), config.default_value]
+ candidates = [
+ *(value for rule in config.rules for value in (rule.value, *rule.variant_values)),
+ config.default_value,
+ ]
return list(dict.fromkeys(value for value in candidates if isinstance(value, str)))
diff --git a/products/feature_flags/backend/api/test/test_feature_flag_config_v2_experiment_rules.py b/products/feature_flags/backend/api/test/test_feature_flag_config_v2_experiment_rules.py
new file mode 100644
index 000000000000..7f26ffd9a0cc
--- /dev/null
+++ b/products/feature_flags/backend/api/test/test_feature_flag_config_v2_experiment_rules.py
@@ -0,0 +1,214 @@
+import copy
+from collections.abc import Callable
+from typing import Any
+
+from unittest.mock import patch
+
+from parameterized import parameterized
+from rest_framework import status
+
+from products.experiments.backend.models.experiment import Experiment
+from products.feature_flags.backend.api.test.test_feature_flag_config_v2_updates import (
+ HOLDOUT_SEED_C,
+ RULE_B,
+ RULE_C,
+ RULE_VALUE_DETAIL,
+ SEED_B,
+ SEED_C,
+ AdmittedV2TestCase,
+ admit_v2,
+ config,
+ experiment,
+ local_holdout,
+ rollout,
+ targeted,
+)
+from products.feature_flags.backend.flags_cache import _get_feature_flags_for_service
+from products.feature_flags.backend.models import FeatureFlag
+
+
+def string_split(first: str, **extra: Any) -> dict:
+ variants = [{"key": "a", "weight": 50, "value": first}, {"key": "b", "weight": 50, "value": "on"}]
+ return config(experiment(variants=variants, **extra), return_type="string", default_value=None)
+
+
+NULL_KEY_DETAIL = 'The variant key "null" is reserved. Choose another key.'
+NULL_KEY_ATTR = "filters__rules__0__variants__0__key"
+
+
+def keyed_split(first_key: str, return_type: str = "boolean", **extra: Any) -> dict:
+ off, on = {"boolean": (False, True), "string": ("off", "on")}[return_type]
+ variants = [{"key": first_key, "weight": 50, "value": off}, {"key": "test", "weight": 50, "value": on}]
+ return config(experiment(variants=variants, **extra), return_type=return_type, default_value=None)
+
+
+class TestExperimentRules(AdmittedV2TestCase):
+ def test_create_assigns_the_rule_and_holdout_seeds(self) -> None:
+ submitted = config(
+ targeted(rule_id=None), experiment(rule_id=None, seed=None, holdout=local_holdout(seed=None))
+ )
+ with admit_v2(self.team.id, creation=True):
+ response = self.post_flag({"key": "split-v2", "filters": submitted})
+ assert response.status_code == status.HTTP_201_CREATED, response.json()
+ flag = FeatureFlag.objects.get(team=self.team, key="split-v2")
+ rule = flag.filters["rules"][1]
+ assert rule["id"] and rule["seed"] and rule["holdout"]["seed"]
+ assert len({rule["seed"], rule["holdout"]["seed"]}) == 2
+ assert "experiment_id" not in rule and set(rule["holdout"]) == {"seed", "exclusion_percentage"}
+ assert not Experiment.objects.filter(feature_flag=flag).exists()
+ assert response.json()["filters"] == flag.filters
+
+ def test_seeds_survive_reorder_and_unrelated_edits(self) -> None:
+ flag = self.flag(config(targeted(), experiment(holdout=local_holdout())))
+ replacement = config(
+ experiment(seed=None, holdout=local_holdout(seed=None, exclusion_percentage=25), paused=True),
+ targeted(value=False),
+ )
+ response = self.patch_flag(flag, {"version": 3, "filters": replacement})
+ assert response.status_code == status.HTTP_200_OK, response.json()
+ flag.refresh_from_db()
+ rule = flag.filters["rules"][0]
+ assert (rule["id"], rule["seed"], rule["holdout"]["seed"]) == (RULE_C, SEED_C, HOLDOUT_SEED_C)
+ assert (rule["paused"], rule["holdout"]["exclusion_percentage"]) == (True, 25)
+
+ def test_a_percentage_rollout_that_becomes_an_experiment_rule_keeps_its_seed(self) -> None:
+ flag = self.flag(config(rollout()))
+ response = self.patch_flag(flag, {"version": 3, "filters": config(experiment(rule_id=RULE_B, seed=None))})
+ assert response.status_code == status.HTTP_200_OK, response.json()
+ flag.refresh_from_db()
+ assert flag.filters["rules"][0]["seed"] == SEED_B
+
+ def test_a_holdout_added_later_gets_a_fresh_seed(self) -> None:
+ flag = self.flag(config(experiment()))
+ response = self.patch_flag(
+ flag, {"version": 3, "filters": config(experiment(holdout=local_holdout(seed=None)))}
+ )
+ assert response.status_code == status.HTTP_200_OK, response.json()
+ flag.refresh_from_db()
+ assert flag.filters["rules"][0]["holdout"]["seed"] not in (None, "", SEED_C, HOLDOUT_SEED_C)
+
+ @parameterized.expand(
+ [
+ (
+ "changed_holdout_seed",
+ config(experiment(holdout=local_holdout(seed="chosen-by-client"))),
+ "filters__rules__0__holdout__seed",
+ ),
+ ("chosen_seed", config(experiment(rule_id=None, seed="chosen-by-client")), "filters__rules__0__seed"),
+ (
+ "chosen_holdout_seed",
+ config(experiment(rule_id=None, seed=None, holdout=local_holdout(seed="chosen"))),
+ "filters__rules__0__holdout__seed",
+ ),
+ ]
+ )
+ def test_invalid_rules_are_rejected(self, _name: str, filters: dict, attr: str) -> None:
+ flag = self.flag(config(experiment(holdout=local_holdout())))
+ stored = copy.deepcopy(flag.filters)
+ response = self.patch_flag(flag, {"version": 3, "filters": filters})
+ assert response.status_code == status.HTTP_400_BAD_REQUEST
+ assert response.json()["attr"] == attr, response.json()
+ flag.refresh_from_db()
+ assert (flag.filters, flag.version) == (stored, 3)
+
+ @parameterized.expand(
+ [
+ ("experiment_id", experiment(experiment_id=42), "filters.rules[0].experiment_id"),
+ ("null_experiment_id", experiment(experiment_id=None), "filters.rules[0].experiment_id"),
+ ("holdout_id", experiment(holdout={**local_holdout(), "id": 7}), "filters.rules[0].holdout.id"),
+ ("null_holdout_id", experiment(holdout={**local_holdout(), "id": None}), "filters.rules[0].holdout.id"),
+ # Reported as the unknown field, not as a client-chosen holdout seed.
+ ("rollout_holdout", rollout(rule_id=None, seed=None, holdout=local_holdout()), "filters.rules[0].holdout"),
+ ]
+ )
+ def test_ids_and_misplaced_holdouts_are_unknown_fields(self, _name: str, rule: dict, path: str) -> None:
+ flag = self.flag(config(experiment(holdout=local_holdout())))
+ stored = copy.deepcopy(flag.filters)
+ response = self.patch_flag(flag, {"version": 3, "filters": config(rule)})
+ assert response.status_code == status.HTTP_400_BAD_REQUEST
+ assert response.json() == {
+ "type": "validation_error",
+ "code": "unknown_field",
+ "detail": f"{path}: Unknown field.",
+ "attr": "filters",
+ }
+ flag.refresh_from_db()
+ assert (flag.filters, flag.version) == (stored, 3)
+
+ def test_create_rejects_a_null_experiment_id(self) -> None:
+ submitted = config(experiment(rule_id=None, seed=None, experiment_id=None))
+ with admit_v2(self.team.id, creation=True):
+ response = self.post_flag({"key": "split-v2", "filters": submitted})
+ assert response.status_code == status.HTTP_400_BAD_REQUEST
+ assert response.json()["detail"] == "filters.rules[0].experiment_id: Unknown field.", response.json()
+ assert not FeatureFlag.objects.filter(team=self.team, key="split-v2").exists()
+
+ @parameterized.expand([("false", "$false"), ("true", "$true"), ("null", "null")])
+ def test_reserved_strings_are_rejected_as_variant_values(self, _name: str, value: str) -> None:
+ flag = self.flag(string_split("off"))
+ response = self.patch_flag(flag, {"version": 3, "filters": string_split(value)})
+ assert response.status_code == status.HTTP_400_BAD_REQUEST
+ body = response.json()
+ assert (body["attr"], body["detail"]) == ("filters__rules__0__variants__0__value", RULE_VALUE_DETAIL), body
+ flag.refresh_from_db()
+ assert (flag.filters, flag.version) == (string_split("off"), 3)
+
+ @parameterized.expand([("variant_value", string_split), ("variant_key", keyed_split)])
+ def test_a_null_variant_the_rule_already_holds_is_kept_and_can_be_enabled(
+ self, _name: str, split: Callable[..., dict]
+ ) -> None:
+ flag = self.flag(split("null"), active=False)
+ replacement = split("null", paused=True)
+ response = self.patch_flag(flag, {"version": 3, "filters": replacement})
+ assert response.status_code == status.HTTP_200_OK, response.json()
+ response = self.patch_flag(flag, {"version": 4, "active": True})
+ assert response.status_code == status.HTTP_200_OK, response.json()
+ flag.refresh_from_db()
+ assert (flag.filters, flag.active, flag.version) == (replacement, True, 5)
+
+ @parameterized.expand([("boolean",), ("string",)])
+ def test_create_rejects_a_null_variant_key(self, return_type: str) -> None:
+ with admit_v2(self.team.id, creation=True):
+ submitted = keyed_split("null", return_type, rule_id=None, seed=None)
+ response = self.post_flag({"key": "split-v2", "filters": submitted})
+ assert response.status_code == status.HTTP_400_BAD_REQUEST
+ body = response.json()
+ assert (body["code"], body["attr"], body["detail"]) == ("invalid_input", NULL_KEY_ATTR, NULL_KEY_DETAIL)
+ assert not FeatureFlag.objects.filter(team=self.team, key="split-v2").exists()
+
+ @parameterized.expand(
+ [
+ ("boolean", keyed_split("control"), keyed_split("null"), NULL_KEY_ATTR),
+ ("string", keyed_split("control", "string"), keyed_split("null", "string"), NULL_KEY_ATTR),
+ (
+ # A held key stays with its rule's experiment, so a new rule cannot take it over.
+ "held_by_another_rule",
+ keyed_split("null"),
+ config(
+ *keyed_split("control")["rules"],
+ *keyed_split("null", rule_id=None, seed=None)["rules"],
+ default_value=None,
+ ),
+ "filters__rules__1__variants__0__key",
+ ),
+ ]
+ )
+ def test_update_rejects_a_new_null_variant_key(self, _name: str, stored: dict, submitted: dict, attr: str) -> None:
+ flag = self.flag(stored)
+ response = self.patch_flag(flag, {"version": 3, "filters": submitted})
+ assert response.status_code == status.HTTP_400_BAD_REQUEST
+ body = response.json()
+ assert (body["code"], body["attr"], body["detail"]) == ("invalid_input", attr, NULL_KEY_DETAIL)
+ flag.refresh_from_db()
+ assert (flag.filters, flag.version) == (stored, 3)
+
+ def test_a_split_above_a_lower_rule_reports_warnings_and_is_cached(self) -> None:
+ flag = self.flag(config(experiment()), active=True)
+ # A 100% split with no holdout returns a value for everyone, so the targeted rule below is never reached.
+ document = config(experiment(), targeted(rule_id=None))
+ with patch("products.feature_flags.backend.api.feature_flag.logger.info") as info:
+ response = self.patch_flag(flag, {"version": 3, "filters": document})
+ assert response.status_code == status.HTTP_200_OK, response.json()
+ assert info.call_args.kwargs["extra"]["codes"] == ["UNREACHABLE_LOWER_RULE"]
+ cached = {f["key"]: f for f in _get_feature_flags_for_service(self.team)["flags"]}
+ assert cached[flag.key]["filters"]["rules"][0]["seed"] == SEED_C
diff --git a/products/feature_flags/backend/api/test/test_feature_flag_config_v2_updates.py b/products/feature_flags/backend/api/test/test_feature_flag_config_v2_updates.py
index 3882ce3686e8..50330cfbe581 100644
--- a/products/feature_flags/backend/api/test/test_feature_flag_config_v2_updates.py
+++ b/products/feature_flags/backend/api/test/test_feature_flag_config_v2_updates.py
@@ -37,6 +37,9 @@ from products.feature_flags.backend.models import FeatureFlag
RULE_A = "3f3b7a9e-8f2e-4f4b-9c7d-2a1e5b6c8d90"
RULE_B = "b1c2d3e4-f5a6-4b7c-8d9e-0f1a2b3c4d5e"
SEED_B = "7c9e6f82-1a2b-4c3d-9e8f-5a6b7c8d9e0f"
+RULE_C = "d3e4f5a6-b7c8-4d9e-a0f1-2b3c4d5e6f70"
+SEED_C = "c0ffee00-1a2b-4c3d-9e8f-5a6b7c8d9e0f"
+HOLDOUT_SEED_C = "f00dcafe-1a2b-4c3d-9e8f-5a6b7c8d9e0f"
UNKNOWN_RULE = "00000000-0000-4000-8000-000000000000"
@@ -75,6 +78,34 @@ def rollout(rule_id: str | None = RULE_B, seed: str | None = SEED_B, **extra: An
return rule
+def experiment(rule_id: str | None = RULE_C, seed: str | None = SEED_C, **extra: Any) -> dict:
+ rule: dict[str, Any] = {
+ "rule_type": "experiment",
+ "targeting": {"properties": []},
+ "paused": False,
+ "rollout_percentage": 100,
+ "on_rollout_miss": "continue",
+ "assignment_algorithm": "sha1_60_v1",
+ "variants": [
+ {"key": "control", "weight": 50, "value": False},
+ {"key": "test", "weight": 50, "value": True},
+ ],
+ **extra,
+ }
+ if rule_id is not None:
+ rule = {"id": rule_id, **rule}
+ if seed is not None:
+ rule["seed"] = seed
+ return rule
+
+
+def local_holdout(seed: str | None = HOLDOUT_SEED_C, exclusion_percentage: float = 10) -> dict:
+ holdout: dict[str, Any] = {"exclusion_percentage": exclusion_percentage}
+ if seed is not None:
+ holdout["seed"] = seed
+ return holdout
+
+
def config(*rules: dict, **extra: Any) -> dict:
return {"version": 2, "return_type": "boolean", "default_value": False, "rules": list(rules), **extra}
@@ -591,7 +622,7 @@ class TestV2AdmissionBoundary(AdmittedV2TestCase):
@parameterized.expand(
[
("malformed", {"version": 2, "rules": "broken"}),
- ("deferred_experiment", config({"id": RULE_A, "rule_type": "experiment", "targeting": {}})),
+ ("malformed_experiment", config({"id": RULE_A, "rule_type": "experiment", "targeting": {}})),
("string_with_boolean_default", config(return_type="string")),
]
)
@@ -603,16 +634,6 @@ class TestV2AdmissionBoundary(AdmittedV2TestCase):
assert flag.filters == stored
assert flag.version == 3
- def test_an_experiment_rule_cannot_be_written(self) -> None:
- flag = self.flag()
- response = self.patch_flag(
- flag,
- {"version": 3, "filters": config({"id": RULE_A, "rule_type": "experiment", "targeting": {}})},
- )
- assert response.status_code == status.HTTP_400_BAD_REQUEST
- flag.refresh_from_db()
- assert flag.version == 3
-
@parameterized.expand(
[
("archived", {"archived": True, "active": False}),
diff --git a/products/feature_flags/backend/facade/config.py b/products/feature_flags/backend/facade/config.py
index 925d2ea86121..dfa0d6fda8a5 100644
--- a/products/feature_flags/backend/facade/config.py
+++ b/products/feature_flags/backend/facade/config.py
@@ -93,6 +93,8 @@ class RuleV2:
rule_type: RuleType
# None for an experiment rule, whose values sit on its variants.
value: FlagValue | None
+ # An experiment rule's variant values in stored order, equal values included; empty for other rules.
+ variant_values: tuple[FlagValue, ...] = ()
def __post_init__(self) -> None:
if self.rule_type not in get_args(RuleType):
@@ -170,4 +172,5 @@ def _rule_v2(rule: Mapping[str, Any]) -> RuleV2:
),
rule_type=rule_type,
value=None if rule_type == "experiment" else rule["value"],
+ variant_values=tuple(variant["value"] for variant in rule["variants"]) if rule_type == "experiment" else (),
)
diff --git a/products/feature_flags/backend/facade/config_validation.py b/products/feature_flags/backend/facade/config_validation.py
index 56755b0fa1ca..c6331f9a11db 100644
--- a/products/feature_flags/backend/facade/config_validation.py
+++ b/products/feature_flags/backend/facade/config_validation.py
@@ -14,14 +14,15 @@ request input into the final document, calls ``rule_warnings.review_config`` (wh
validates through this module and reports warnings) and persists under the existing row
lock. No production caller exists yet.
-Admitted family: person-assigned flags of every return type with targeted_release and
-percentage_rollout rules whose targeting uses person properties. Everything else the
-published contract describes (experiment rules, group assignment, cohort/group/flag
-properties) is rejected with the ``unsupported`` code, so a
-canonical fixture is never relabelled as malformed and no partially checked document is
-accepted. Shape errors use the other codes. The canonical shape is the harness config
-schema 1.0.0 in contract package 2.0.0; the semantic constraints (unique rule ids, two
-decimal places, byte limits) come from its literal registry.
+Admitted family: person-assigned flags of every return type with targeted_release,
+percentage_rollout and experiment rules whose targeting uses person properties. An experiment
+rule names no Experiment (an Experiment points at its rule by rule id) and may carry a holdout
+that belongs to the rule. Everything else the published contract describes (group assignment,
+cohort/group/flag properties) is rejected with the ``unsupported`` code, so a canonical
+fixture is never relabelled as malformed and no partially checked document is accepted. Shape
+errors use the other codes. The canonical shape is the harness config schema in contract
+package 6.0.0; the semantic constraints (unique rule ids and variant keys, two decimal places,
+weights totalling 100, byte limits) come from its literal registry.
"""
import re
@@ -47,13 +48,16 @@ MAX_SEED_LENGTH = 400
MAX_PERCENTAGE_DECIMALS = 2
MAX_SAFE_INTEGER = 2**53 - 1
MAX_OBJECT_DEPTH = 20
+MIN_VARIANTS = 2
+MAX_VARIANTS = 20
ConfigErrorCode = Literal["required", "invalid", "unknown_field", "not_unique", "unsupported", "limit_exceeded"]
RolloutMissPolicy = Literal["continue", "return_default"]
-AdmittedRuleType = Literal["targeted_release", "percentage_rollout"]
RETURN_TYPES: tuple[str, ...] = get_args(FlagReturnType)
RULE_TYPES: tuple[str, ...] = get_args(RuleType)
+# Rule types that assign by hash: they carry a rollout and a server-owned seed.
+RANDOMIZED_RULE_TYPES: tuple[str, ...] = ("percentage_rollout", "experiment")
ROLLOUT_MISS_POLICIES: tuple[RolloutMissPolicy, ...] = get_args(RolloutMissPolicy)
ASSIGNMENT_ALGORITHM = "sha1_60_v1"
PERSON_ASSIGNMENT = "person"
@@ -123,10 +127,15 @@ _V1_ONLY_ROOT_FIELDS = frozenset(
)
_COMMON_RULE_FIELDS = frozenset({"id", "rule_type", "targeting", "description", "metadata", "value"})
_ROLLOUT_RULE_FIELDS = frozenset({"rollout_percentage", "on_rollout_miss", "assignment_algorithm", "seed", "assign_by"})
+_EXPERIMENT_ONLY_FIELDS = frozenset({"paused", "variants", "holdout"})
_RULE_FIELDS: dict[str, frozenset[str]] = {
"targeted_release": _COMMON_RULE_FIELDS,
"percentage_rollout": _COMMON_RULE_FIELDS | _ROLLOUT_RULE_FIELDS,
+ "experiment": (_COMMON_RULE_FIELDS - {"value"}) | _ROLLOUT_RULE_FIELDS | _EXPERIMENT_ONLY_FIELDS,
}
+_VARIANT_FIELDS = frozenset({"key", "weight", "value"})
+_HOLDOUT_FIELDS = frozenset({"seed", "exclusion_percentage"})
+_VARIANT_KEY = re.compile(r"^[a-zA-Z0-9_-]+$")
_PROPERTY_FIELDS = frozenset(
{"key", "value", "type", "operator", "group_type_index", "negation", "cohort_name", "group_key_names", "label"}
)
@@ -193,16 +202,34 @@ class Predicate:
negation: bool
+@frozen
+class ValidatedVariant:
+ key: str
+ weight: Decimal
+ value: str # canonical JSON text, see ``canonical_value``
+
+
+@frozen
+class ValidatedHoldout:
+ """A holdout that belongs to its rule; it has no id of its own."""
+
+ exclusion_percentage: Decimal
+ seed: str = field(repr=False)
+
+
@frozen
class ValidatedRule:
id: str
- rule_type: AdmittedRuleType
+ rule_type: RuleType
predicates: frozenset[Predicate]
- value: str # canonical JSON text, see ``canonical_value``
+ value: str | None # canonical JSON text, see ``canonical_value``; None for an experiment rule
rollout_percentage: Decimal | None = None
on_rollout_miss: RolloutMissPolicy | None = None
# Assignment seeds must not reach logs or tracebacks; the detectors only compare them.
seed: str | None = field(default=None, repr=False)
+ paused: bool = False
+ variants: tuple[ValidatedVariant, ...] = ()
+ holdout: ValidatedHoldout | None = None
@frozen
@@ -263,7 +290,7 @@ def validate_config(document: object, *, limits: ValidationLimits) -> ValidatedC
)
rules: list[ValidatedRule] = []
- if _field(document, "rules", "filters", errors, (lambda v: isinstance(v, list), "Must be an array.")):
+ if _field(document, "rules", "filters", errors, _ARRAY_CHECK):
if len(document["rules"]) > MAX_RULES:
errors.append(
ConfigError(
@@ -320,15 +347,9 @@ def _validate_rule(
if not _field(rule, "rule_type", path, errors, _one_of(RULE_TYPES)):
return None
rule_type = rule["rule_type"]
- if rule_type not in _RULE_FIELDS:
- errors.append(
- ConfigError(code="unsupported", detail="Experiment rules are not available yet.", attr=f"{path}.rule_type")
- )
- return None
for name in rule:
if name not in _RULE_FIELDS[rule_type]:
- rollout_only = "Only percentage_rollout rules have this field."
- errors.append(_unknown_field(f"{path}.{name}", rollout_only if name in _ROLLOUT_RULE_FIELDS else None))
+ errors.append(_unknown_field(f"{path}.{name}", _UNKNOWN_RULE_FIELD_DETAILS.get((rule_type, name))))
before = len(errors)
_field(rule, "id", path, errors, (lambda v: isinstance(v, str) and bool(_UUID.fullmatch(v)), "Must be a UUID."))
@@ -336,20 +357,23 @@ def _validate_rule(
_field(rule, "description", path, errors, (lambda v: isinstance(v, str), "Must be a string."), False)
if "metadata" in rule:
_validate_metadata(rule["metadata"], f"{path}.metadata", limits, errors)
- if value_check is not None:
+ if value_check is not None and rule_type != "experiment":
_field(rule, "value", path, errors, value_check)
rollout: Decimal | None = None
- if rule_type == "percentage_rollout":
- rollout = _validate_percentage(rule, path, errors)
+ if rule_type in RANDOMIZED_RULE_TYPES:
+ rollout = _validate_percentage(rule, "rollout_percentage", path, errors)
_field(rule, "on_rollout_miss", path, errors, _one_of(ROLLOUT_MISS_POLICIES))
_field(rule, "assignment_algorithm", path, errors, _one_of((ASSIGNMENT_ALGORITHM,)))
- seed_ok = (
- lambda v: isinstance(v, str) and 1 <= len(v) <= MAX_SEED_LENGTH,
- f"Must be between 1 and {MAX_SEED_LENGTH} characters.",
- )
- _field(rule, "seed", path, errors, seed_ok)
+ _field(rule, "seed", path, errors, _SEED_CHECK)
_field(rule, "assign_by", path, errors, _one_of((PERSON_ASSIGNMENT,)), False)
+ variants: tuple[ValidatedVariant, ...] = ()
+ holdout: ValidatedHoldout | None = None
+ if rule_type == "experiment":
+ _field(rule, "paused", path, errors, _VALUE_CHECKS["boolean"])
+ variants = _validate_variants(rule, path, value_check, errors)
+ if "holdout" in rule:
+ holdout = _validate_holdout(rule["holdout"], f"{path}.holdout", errors)
if len(errors) > before or predicates is None or value_check is None:
return None
@@ -357,13 +381,95 @@ def _validate_rule(
id=rule["id"],
rule_type=rule_type,
predicates=predicates,
- value=canonical_value(rule["value"]),
+ value=None if rule_type == "experiment" else canonical_value(rule["value"]),
rollout_percentage=rollout,
on_rollout_miss=rule["on_rollout_miss"] if rollout is not None else None,
seed=rule["seed"] if rollout is not None else None,
+ paused=rule.get("paused", False),
+ variants=variants,
+ holdout=holdout,
)
+_ROLLOUT_ONLY = "Only percentage_rollout and experiment rules have this field."
+_UNKNOWN_RULE_FIELD_DETAILS: dict[tuple[str, str], str] = {
+ **{("targeted_release", name): _ROLLOUT_ONLY for name in _ROLLOUT_RULE_FIELDS},
+ ("experiment", "value"): "Experiment rules return the values of their variants.",
+}
+_SEED_CHECK: "_Check" = (
+ lambda v: isinstance(v, str) and 1 <= len(v) <= MAX_SEED_LENGTH,
+ f"Must be between 1 and {MAX_SEED_LENGTH} characters.",
+)
+_ARRAY_CHECK: "_Check" = (lambda v: isinstance(v, list), "Must be an array.")
+_VARIANT_KEY_CHECK: "_Check" = (
+ lambda v: isinstance(v, str) and bool(_VARIANT_KEY.fullmatch(v)),
+ "Must be letters, digits, hyphens or underscores.",
+)
+
+
+def _validate_variants(
+ rule: Mapping[str, Any], path: str, value_check: "_Check | None", errors: list[ConfigError]
+) -> tuple[ValidatedVariant, ...]:
+ if not _field(rule, "variants", path, errors, _ARRAY_CHECK):
+ return ()
+ items, path = rule["variants"], f"{path}.variants"
+ if len(items) > MAX_VARIANTS:
+ errors.append(
+ ConfigError(code="limit_exceeded", detail=f"At most {MAX_VARIANTS} variants are allowed.", attr=path)
+ )
+ return ()
+ if len(items) < MIN_VARIANTS:
+ errors.append(ConfigError(code="invalid", detail=f"At least {MIN_VARIANTS} variants are required.", attr=path))
+ return ()
+ variants: list[ValidatedVariant] = []
+ seen_keys: set[str] = set()
+ for index, variant in enumerate(items):
+ validated = _validate_variant(variant, f"{path}[{index}]", value_check, seen_keys, errors)
+ if validated is not None:
+ variants.append(validated)
+ # Sum only a fully validated list: without a valid return type no variant validates.
+ if len(variants) == len(items) and sum(variant.weight for variant in variants) != 100:
+ errors.append(ConfigError(code="invalid", detail="Variant weights must total exactly 100.", attr=path))
+ return tuple(variants)
+
+
+def _validate_variant(
+ variant: object, path: str, value_check: "_Check | None", seen_keys: set[str], errors: list[ConfigError]
+) -> ValidatedVariant | None:
+ if not isinstance(variant, Mapping):
+ errors.append(ConfigError(code="invalid", detail="Must be an object.", attr=path))
+ return None
+ before = len(errors)
+ for name in variant:
+ if name not in _VARIANT_FIELDS:
+ errors.append(_unknown_field(f"{path}.{name}"))
+ if _field(variant, "key", path, errors, _VARIANT_KEY_CHECK):
+ if variant["key"] in seen_keys:
+ errors.append(ConfigError(code="not_unique", detail="Variant keys must be unique.", attr=f"{path}.key"))
+ seen_keys.add(variant["key"])
+ weight = _validate_percentage(variant, "weight", path, errors)
+ if value_check is not None:
+ _field(variant, "value", path, errors, value_check)
+ if len(errors) > before or weight is None or value_check is None:
+ return None
+ return ValidatedVariant(key=variant["key"], weight=weight, value=canonical_value(variant["value"]))
+
+
+def _validate_holdout(holdout: object, path: str, errors: list[ConfigError]) -> ValidatedHoldout | None:
+ if not isinstance(holdout, Mapping):
+ errors.append(ConfigError(code="invalid", detail="Must be an object.", attr=path))
+ return None
+ before = len(errors)
+ for name in holdout:
+ if name not in _HOLDOUT_FIELDS:
+ errors.append(_unknown_field(f"{path}.{name}"))
+ _field(holdout, "seed", path, errors, _SEED_CHECK)
+ exclusion = _validate_percentage(holdout, "exclusion_percentage", path, errors)
+ if len(errors) > before or exclusion is None:
+ return None
+ return ValidatedHoldout(exclusion_percentage=exclusion, seed=holdout["seed"])
+
+
def _validate_targeting(rule: Mapping[str, Any], path: str, errors: list[ConfigError]) -> frozenset[Predicate] | None:
if not _field(rule, "targeting", path, errors, (lambda v: isinstance(v, Mapping), "Must be an object.")):
return None
@@ -372,7 +478,7 @@ def _validate_targeting(rule: Mapping[str, Any], path: str, errors: list[ConfigE
for name in targeting:
if name != "properties":
errors.append(_unknown_field(f"{path}.{name}"))
- if not _field(targeting, "properties", path, errors, (lambda v: isinstance(v, list), "Must be an array.")):
+ if not _field(targeting, "properties", path, errors, _ARRAY_CHECK):
return None
properties = targeting["properties"]
if len(properties) > MAX_PREDICATES_PER_RULE:
@@ -502,11 +608,11 @@ def _validate_metadata(metadata: object, path: str, limits: ValidationLimits, er
errors.append(ConfigError(code="limit_exceeded", detail="The rule metadata is too large.", attr=path))
-def _validate_percentage(rule: Mapping[str, Any], path: str, errors: list[ConfigError]) -> Decimal | None:
- if not _field(rule, "rollout_percentage", path, errors, (_is_number, "Must be a finite number.")):
+def _validate_percentage(obj: Mapping[str, Any], name: str, path: str, errors: list[ConfigError]) -> Decimal | None:
+ if not _field(obj, name, path, errors, (_is_number, "Must be a finite number.")):
return None
- value = rule["rollout_percentage"]
- path = f"{path}.rollout_percentage"
+ value = obj[name]
+ path = f"{path}.{name}"
if not 0 <= value <= 100:
errors.append(ConfigError(code="invalid", detail="Must be between 0 and 100.", attr=path))
return None
diff --git a/products/feature_flags/backend/facade/config_writes.py b/products/feature_flags/backend/facade/config_writes.py
index bab43e656df3..a8bb18db9c3f 100644
--- a/products/feature_flags/backend/facade/config_writes.py
+++ b/products/feature_flags/backend/facade/config_writes.py
@@ -6,9 +6,9 @@ Four things a v2 write needs that the pure validator deliberately does not do:
internal feature flags evaluated for the project, both off by default. Nothing else grants
admission. Disabling and soft-deleting an existing v2 row need neither, which the serializer
decides; the operation matrix is in ``docs/internal/feature-flags/api-writes.md``.
-- **Identity.** Rule ids and assignment seeds are server-owned and identify rules, not
- list positions. ``resolve_identity`` echoes back existing identity, allocates it for
- genuinely new rules, and rejects a client that tries to choose it.
+- **Identity.** Rule ids, assignment seeds and holdout seeds are server-owned and identify
+ rules, not list positions. ``resolve_identity`` echoes back existing identity, allocates it
+ for genuinely new rules, and rejects a client that tries to choose it.
- **Comparison.** ``review_update`` validates both documents' shape and semantics, so
warnings describe the real before/proposed pair and unsupported stored families are
rejected. Byte limits apply only to the candidate so oversized rows can be reduced.
@@ -36,6 +36,7 @@ from posthog.dataclasses import frozen
from posthog.ph_client import feature_enabled_or_false
from products.feature_flags.backend.facade.config_validation import (
+ RANDOMIZED_RULE_TYPES,
ConfigError,
ConfigValidationError,
ValidatedConfig,
@@ -48,8 +49,6 @@ from products.feature_flags.backend.reserved_variant_keys import is_reserved_var
logger = structlog.get_logger(__name__)
-_SEEDED_RULE_TYPE = "percentage_rollout"
-
# Internal feature flags, targeted at the ``project`` group by id. Both off means closed.
V2_WRITES_FLAG = "feature-flag-rules-v2-writes"
V2_CREATION_FLAG = "feature-flag-rules-v2-creation"
@@ -187,28 +186,40 @@ def _resolve_rule(
claimed.add(rule["id"])
current = stored_rules[rule["id"]]
- if rule.get("rule_type") != _SEEDED_RULE_TYPE:
- # A rule that is not randomized carries no seed; changing type away from a rollout
- # drops it, and changing type towards one allocates a fresh one below.
+ if rule.get("rule_type") not in RANDOMIZED_RULE_TYPES:
+ # A rule that is not randomized carries no seed; changing type away from a randomized
+ # type drops it, and changing type towards one allocates a fresh one below.
return resolved
- stored_seed = current.get("seed") if current is not None and current.get("rule_type") == _SEEDED_RULE_TYPE else None
+ # A rule keeps its seed while it stays randomized, so a rollout that becomes an experiment rule keeps who is enrolled.
+ stored = current if current is not None and current.get("rule_type") in RANDOMIZED_RULE_TYPES else {}
+ resolved["seed"] = _resolve_seed(rule, stored.get("seed"), path, errors)
+ holdout = rule.get("holdout")
+ # Only experiment rules have a holdout; the validator reports one on any other rule as an unknown field.
+ if rule["rule_type"] == "experiment" and isinstance(holdout, Mapping):
+ stored_holdout = stored.get("holdout")
+ stored_seed = stored_holdout.get("seed") if isinstance(stored_holdout, Mapping) else None
+ resolved["holdout"] = {**holdout, "seed": _resolve_seed(holdout, stored_seed, f"{path}.holdout", errors)}
+ return resolved
+
+
+def _resolve_seed(submitted: Mapping[str, Any], stored_seed: object, path: str, errors: list[ConfigError]) -> object:
+ """The stored seed when the client omits or echoes it, a fresh one when nothing is stored."""
if not isinstance(stored_seed, str):
- if "seed" in rule:
+ if "seed" in submitted:
errors.append(
ConfigError(code="invalid", detail="Assignment seeds are server-assigned.", attr=f"{path}.seed")
)
- resolved["seed"] = str(uuid4())
- elif "seed" not in rule or rule["seed"] == stored_seed:
- resolved["seed"] = stored_seed
- else:
- errors.append(
- ConfigError(
- code="invalid",
- detail="Cannot be changed; resetting assignment is a separate operation.",
- attr=f"{path}.seed",
- )
+ return str(uuid4())
+ if "seed" not in submitted or submitted["seed"] == stored_seed:
+ return stored_seed
+ errors.append(
+ ConfigError(
+ code="invalid",
+ detail="Cannot be changed; resetting assignment is a separate operation.",
+ attr=f"{path}.seed",
)
- return resolved
+ )
+ return submitted["seed"]
def review_update(
@@ -282,13 +293,40 @@ def _reserved_string_values(document: Mapping[str, Any], stored: Mapping[str, An
# The flag serves a string value as its variant, so this rule reserves the keys v1 reserves as variant keys.
if document["return_type"] != "string":
return
- held = {stored["default_value"], *(rule.get("value") for rule in stored["rules"])} if stored else set()
+ held = {stored["default_value"], *(value for _, value in _rule_values(stored))} if stored else set()
detail = f"a non-empty string other than {_RESERVED_STRING_VALUES_TEXT}"
if is_reserved_variant_key(document["default_value"], held=held):
yield ConfigError(code="invalid", detail=f"Must be null or {detail}.", attr="filters.default_value")
+ for attr, value in _rule_values(document):
+ if is_reserved_variant_key(value, held=held):
+ yield ConfigError(code="invalid", detail=f"Must be {detail}.", attr=attr)
+
+
+def _rule_values(document: Mapping[str, Any]) -> Iterator[tuple[str, Any]]:
+ """Each rule value and experiment-rule variant value in ``document``, with its attr."""
for index, rule in enumerate(document["rules"]):
- if is_reserved_variant_key(rule.get("value"), held=held):
- yield ConfigError(code="invalid", detail=f"Must be {detail}.", attr=f"filters.rules[{index}].value")
+ yield f"filters.rules[{index}].value", rule.get("value")
+ for variant_index, variant in enumerate(rule.get("variants") or ()):
+ yield f"filters.rules[{index}].variants[{variant_index}].value", variant["value"]
+
+
+def _reserved_variant_keys(document: Mapping[str, Any], stored: Mapping[str, Any]) -> Iterator[ConfigError]:
+ # A variant key does not depend on the return type, so v1's variant-key reservation applies to every flag.
+ # A held key keeps one experiment's continuity, so only the rule that holds it keeps it.
+ held_by_rule = {rule["id"]: _variant_keys(rule) for rule in stored["rules"]} if stored else {}
+ for index, rule in enumerate(document["rules"]):
+ held = held_by_rule.get(rule["id"], set())
+ for variant_index, variant in enumerate(rule.get("variants") or ()):
+ if is_reserved_variant_key(variant["key"], held=held):
+ yield ConfigError(
+ code="invalid",
+ detail=f'The variant key "{variant["key"]}" is reserved. Choose another key.',
+ attr=f"filters.rules[{index}].variants[{variant_index}].key",
+ )
+
+
+def _variant_keys(rule: Mapping[str, Any]) -> set[str]:
+ return {variant["key"] for variant in rule.get("variants") or ()}
def _compilable_patterns(document: Mapping[str, Any], stored: Mapping[str, Any]) -> Iterator[ConfigError]:
@@ -328,5 +366,6 @@ def _compiles(pattern: str) -> bool:
_WRITER_RULES: tuple[Callable[[Mapping[str, Any], Mapping[str, Any]], Iterator[ConfigError]], ...] = (
_reserved_string_values,
+ _reserved_variant_keys,
_compilable_patterns,
)
diff --git a/products/feature_flags/backend/facade/rule_warnings.py b/products/feature_flags/backend/facade/rule_warnings.py
index 7a7e8e23ab79..7c283a01b8bd 100644
--- a/products/feature_flags/backend/facade/rule_warnings.py
+++ b/products/feature_flags/backend/facade/rule_warnings.py
@@ -13,17 +13,20 @@ Every detector reasons over one *population*: the people who satisfy a rule's wh
predicate set. A rule provably applies to that population when its own predicate set is
a subset (rules AND their properties, so fewer predicates means a wider audience); any
other relation is inconclusive and stops the walk, so a warning is never derived from a
-guess about how two different predicates overlap. Within a population, percentage rules
-partition people by their assignment hash: the same seed reuses the same hash, and
-different seeds are modelled as independent dimensions. A rollout of ``p`` percent
-includes the hash interval ``(0, p/100]``; 0 percent includes nobody, 100 percent
-everybody. This is decision-level reasoning about who can reach which rule, not a second
-evaluator, and it never claims an exact affected percentage.
+guess about how two different predicates overlap. Within a population, randomized rules
+partition people by their hashes: the same seed reuses the same hash, and each distinct
+hash (a rollout, variant or holdout hash of a seed) is an independent dimension. A rollout
+of ``p`` percent includes the hash interval ``(0, p/100]``; 0 percent includes nobody, 100
+percent everybody. An experiment rule returns the default while paused, then to its
+holdout share, then splits its enrolled share across its variants by cumulative weight.
+This is decision-level reasoning about who can reach which rule, not a second evaluator,
+and it never claims an exact affected percentage.
"""
from collections.abc import Iterator
+from decimal import Decimal
from fractions import Fraction
-from itertools import product
+from typing import Literal
from posthog.dataclasses import frozen
@@ -36,8 +39,9 @@ from products.feature_flags.backend.facade.config_validation import (
)
from products.feature_flags.backend.facade.warnings import ManagementWarning
-_Interval = tuple[Fraction, Fraction] # (lo, hi] of the assignment hash in (0, 1]
-_Box = dict[str, _Interval] # per seed; an absent seed is unconstrained
+_Interval = tuple[Fraction, Fraction] # (lo, hi] of a hash in (0, 1]
+_Dimension = tuple[Literal["rollout", "variant", "holdout"], str] # (which of the rule's hashes, seed)
+_Box = dict[_Dimension, _Interval] # an absent dimension is unconstrained
_FULL: _Interval = (Fraction(0), Fraction(1))
@@ -76,36 +80,37 @@ def reorder_warnings(current: ValidatedConfig, proposed: ValidatedConfig) -> tup
or whose evaluated content changed, is an edit rather than a reorder, so a value change
it causes is never attributed to the order.
"""
- unchanged = set(current.rules) & set(proposed.rules)
+ current_index = {rule.id: index for index, rule in enumerate(current.rules)}
+ proposed_index = {rule.id: index for index, rule in enumerate(proposed.rules)}
+ unchanged = [rule.id for rule in set(current.rules) & set(proposed.rules)]
+ later_by_earlier: dict[str, list[str]] = {}
+ for earlier in unchanged:
+ for later in unchanged:
+ if current_index[earlier] < current_index[later] and proposed_index[later] < proposed_index[earlier]:
+ later_by_earlier.setdefault(earlier, []).append(later)
+ if not later_by_earlier:
+ return ()
# A terminal miss serves the config default, so when the default is edited too those
# outcomes differ because of the edit; only served rule values stay comparable.
default_edited = current.default_value != proposed.default_value
- current_index = {rule.id: index for index, rule in enumerate(current.rules)}
- proposed_index = {rule.id: index for index, rule in enumerate(proposed.rules)}
+ # A population's walk settles a prefix of the walk every rule applies to, so populations
+ # that reach the same rules in both configs compare the same regions.
+ current_walk, proposed_walk = _walk(current, None, split_variants=True), _walk(proposed, None, split_variants=True)
+ compared: set[tuple[int, int]] = set()
inversions: dict[tuple[str, str], int] = {}
for population in {rule.predicates for rule in (*current.rules, *proposed.rules)}:
- before = _walk(current, population)
- after = _walk(proposed, population)
- for (box_before, outcome_before), (box_after, outcome_after) in product(before.settled, after.settled):
- earlier = current.rules[outcome_before.rule_index]
- later = proposed.rules[outcome_after.rule_index]
- values_differ = outcome_before.value != outcome_after.value
- values_comparable = not default_edited or (outcome_before.served and outcome_after.served)
- # Only a pair present in both configs has both indexes, so this covers "unchanged" too.
- order_flipped = (
- earlier in unchanged
- and later in unchanged
- and current_index[earlier.id] < current_index[later.id]
- and proposed_index[later.id] < proposed_index[earlier.id]
- )
- if (
- values_differ
- and values_comparable
- and order_flipped
- and (earlier.id, later.id) not in inversions
- and _intersects(box_before, box_after)
- ):
- inversions[(earlier.id, later.id)] = proposed_index[later.id]
+ reach = (_reach(current, population), _reach(proposed, population))
+ if reach in compared:
+ continue
+ compared.add(reach)
+ before = _regions_by_rule(current, current_walk, reach[0])
+ after = _regions_by_rule(proposed, proposed_walk, reach[1])
+ for earlier, laters in later_by_earlier.items():
+ for later in laters:
+ if (earlier, later) not in inversions and _value_changes(
+ before.get(earlier, {}), after.get(later, {}), default_edited
+ ):
+ inversions[(earlier, later)] = proposed_index[later]
return tuple(
ManagementWarning(
code="RULE_ORDER_CHANGES_TRAFFIC",
@@ -135,9 +140,10 @@ def _rollout_miss_extensions(config: ValidatedConfig) -> Iterator[ManagementWarn
if not _continues_after_partial_miss(upper):
continue
assert upper.rollout_percentage is not None
+ upper_values = _served_values(upper)
for lower_index in range(upper_index + 1, len(config.rules)):
lower = config.rules[lower_index]
- if lower.value != upper.value:
+ if not upper_values & _served_values(lower):
continue
population = _overlap(upper, lower)
if population is None:
@@ -164,9 +170,14 @@ def _rollout_miss_extensions(config: ValidatedConfig) -> Iterator[ManagementWarn
)
+def _served_values(rule: ValidatedRule) -> set[str | None]:
+ # A variant with weight 0 never serves its value.
+ return {variant.value for variant in rule.variants if variant.weight > 0} if rule.variants else {rule.value}
+
+
def _continues_after_partial_miss(rule: ValidatedRule) -> bool:
return (
- rule.rule_type == "percentage_rollout"
+ not rule.paused
and rule.on_rollout_miss == "continue"
and rule.rollout_percentage is not None
and 0 < rule.rollout_percentage < 100
@@ -184,8 +195,8 @@ def _overlap(a: ValidatedRule, b: ValidatedRule) -> frozenset[Predicate] | None:
@frozen
class _Outcome:
- value: str | None # canonical JSON of the value served: the rule's, or the config default on a terminal miss
- served: bool # True when the rule's own value was served, False for a terminal miss
+ value: str | None # canonical JSON of the value served: the rule's or a variant's, or the config default
+ served: bool # True when the rule served its own value, False when it returned the default
rule_index: int
@@ -195,6 +206,37 @@ class _Walk:
closed_by: int | None # the rule after which nothing was left to evaluate
+_Regions = dict[tuple[str | None, bool], list[_Box]] # one rule's settled boxes by (value, served)
+
+
+def _regions_by_rule(config: ValidatedConfig, walk: _Walk, reach: int) -> dict[str, _Regions]:
+ # Grouping by outcome compares two rules' equal outcomes once instead of box by box.
+ regions: dict[str, _Regions] = {}
+ for box, outcome in walk.settled:
+ if outcome.rule_index >= reach:
+ break # regions settle in rule order
+ rule_id = config.rules[outcome.rule_index].id
+ regions.setdefault(rule_id, {}).setdefault((outcome.value, outcome.served), []).append(box)
+ return regions
+
+
+def _value_changes(before: _Regions, after: _Regions, default_edited: bool) -> bool:
+ return any(
+ value_before != value_after
+ and (not default_edited or (served_before and served_after))
+ and any(_intersects(box_before, box_after) for box_before in boxes_before for box_after in boxes_after)
+ for (value_before, served_before), boxes_before in before.items()
+ for (value_after, served_after), boxes_after in after.items()
+ )
+
+
+def _reach(config: ValidatedConfig, population: frozenset[Predicate]) -> int:
+ """How many leading rules ``_walk`` evaluates for ``population`` before it stops."""
+ if _has_contradictory_presence_checks(population):
+ return 0
+ return next((i for i, rule in enumerate(config.rules) if not rule.predicates <= population), len(config.rules))
+
+
def _has_contradictory_presence_checks(population: frozenset[Predicate]) -> bool:
presence: dict[str, set[bool]] = {}
for predicate in population:
@@ -203,34 +245,53 @@ def _has_contradictory_presence_checks(population: frozenset[Predicate]) -> bool
return any(len(states) > 1 for states in presence.values())
-def _walk(config: ValidatedConfig, population: frozenset[Predicate], *, until: int | None = None) -> _Walk:
+def _walk(
+ config: ValidatedConfig,
+ population: frozenset[Predicate] | None,
+ *,
+ until: int | None = None,
+ split_variants: bool = False,
+) -> _Walk:
"""Evaluate ``population`` through the rules before ``until`` (all rules when None).
+ A ``None`` population is one every rule applies to.
+
Stops at the first rule that does not provably apply to the population; the regions
- settled before it stay valid, the rest is unknown.
+ settled before it stay valid, the rest is unknown. Only value comparisons need an
+ experiment rule's served region split by variant; without ``split_variants`` it has value None.
"""
- if _has_contradictory_presence_checks(population):
+ if population is not None and _has_contradictory_presence_checks(population):
return _Walk(settled=(), closed_by=None)
open_boxes: list[_Box] = [{}]
settled: list[tuple[_Box, _Outcome]] = []
for index, rule in enumerate(config.rules[:until]):
- if not rule.predicates <= population:
+ if population is not None and not rule.predicates <= population:
# Inconclusive: who passes this rule is unknown, so nothing below it can be settled
# and the regions that were still open must not be reported as reaching anything.
return _Walk(settled=tuple(settled), closed_by=None)
next_open: list[_Box] = []
for box in open_boxes:
+ if rule.paused:
+ settled.append((box, _Outcome(value=config.default_value, served=False, rule_index=index)))
+ continue
if rule.rule_type == "targeted_release":
settled.append((box, _Outcome(value=rule.value, served=True, rule_index=index)))
continue
assert rule.seed is not None and rule.rollout_percentage is not None
- lo, hi = box.get(rule.seed, _FULL)
- threshold = Fraction(rule.rollout_percentage) / 100
- if lo < min(hi, threshold):
- included = {**box, rule.seed: (lo, min(hi, threshold))}
- settled.append((included, _Outcome(value=rule.value, served=True, rule_index=index)))
- if max(lo, threshold) < hi:
- missed = {**box, rule.seed: (max(lo, threshold), hi)}
+ enrolling: _Box | None = box
+ if rule.holdout is not None:
+ held = _split(box, ("holdout", rule.holdout.seed), _threshold(rule.holdout.exclusion_percentage))
+ enrolling = held.above
+ if held.below is not None:
+ settled.append((held.below, _Outcome(value=config.default_value, served=False, rule_index=index)))
+ if enrolling is None:
+ continue
+ rollout = _split(enrolling, ("rollout", rule.seed), _threshold(rule.rollout_percentage))
+ included, missed = rollout.below, rollout.above
+ if included is not None:
+ for region, value in _served_regions(included, rule, split_variants):
+ settled.append((region, _Outcome(value=value, served=True, rule_index=index)))
+ if missed is not None:
if rule.on_rollout_miss == "return_default":
settled.append((missed, _Outcome(value=config.default_value, served=False, rule_index=index)))
else:
@@ -241,10 +302,47 @@ def _walk(config: ValidatedConfig, population: frozenset[Predicate], *, until: i
return _Walk(settled=tuple(settled), closed_by=None)
+def _threshold(percentage: Decimal) -> Fraction:
+ return Fraction(percentage) / 100
+
+
+@frozen
+class _Halves:
+ """The parts of a box at or below and above a threshold on one hash, None when empty."""
+
+ below: _Box | None
+ above: _Box | None
+
+
+def _split(box: _Box, dimension: _Dimension, threshold: Fraction) -> _Halves:
+ lo, hi = box.get(dimension, _FULL)
+ return _Halves(
+ below={**box, dimension: (lo, min(hi, threshold))} if lo < min(hi, threshold) else None,
+ above={**box, dimension: (max(lo, threshold), hi)} if max(lo, threshold) < hi else None,
+ )
+
+
+def _served_regions(box: _Box, rule: ValidatedRule, split_variants: bool) -> Iterator[tuple[_Box, str | None]]:
+ if not (split_variants and rule.variants):
+ yield box, rule.value
+ return
+ assert rule.seed is not None
+ rest: _Box | None = box
+ boundary = Fraction(0)
+ for variant in rule.variants:
+ boundary += _threshold(variant.weight)
+ if rest is None:
+ return
+ halves = _split(rest, ("variant", rule.seed), boundary)
+ rest = halves.above
+ if halves.below is not None:
+ yield halves.below, variant.value
+
+
def _intersects(a: _Box, b: _Box) -> bool:
- # Every stored interval is non-empty, so only seeds constrained on both sides can be disjoint.
- for seed in a.keys() & b.keys():
- (a_lo, a_hi), (b_lo, b_hi) = a[seed], b[seed]
+ # Every stored interval is non-empty, so only dimensions constrained on both sides can be disjoint.
+ for dimension in a.keys() & b.keys():
+ (a_lo, a_hi), (b_lo, b_hi) = a[dimension], b[dimension]
if max(a_lo, b_lo) >= min(a_hi, b_hi):
return False
return True
diff --git a/products/feature_flags/backend/test/fixtures/rules_v2_contract/SOURCE.json b/products/feature_flags/backend/test/fixtures/rules_v2_contract/SOURCE.json
index cf3c8f831ab4..082dde6e4fd3 100644
--- a/products/feature_flags/backend/test/fixtures/rules_v2_contract/SOURCE.json
+++ b/products/feature_flags/backend/test/fixtures/rules_v2_contract/SOURCE.json
@@ -1,30 +1,31 @@
{
"repository": "https://github.com/PostHog/posthog-sdk-test-harness",
- "release": "1.15.0",
- "commit": "64ffa6ab19166541880c077d349abe18a7e42395",
+ "release": "1.20.0",
+ "commit": "aa725c89eec72cb8ac75f4e418662bf2e161feb8",
"path": "contracts/feature_flag_rules_v2",
- "contract_version": "2.3.1",
+ "contract_version": "6.0.0",
"files": {
"fixtures/config/invalid/assign_by_not_person.json": "52d356d2bcdcbf705b0fe2673eb147ac352bda13c9b1bc3d53c4bbb1cb01a5aa",
"fixtures/config/invalid/default_value_type_mismatch.json": "516e72b6bd9f761de62166457528242723694f86e417b1aee6eb37b924acc4b5",
"fixtures/config/invalid/empty_seed.json": "f5aa796ba2abf17ab6b2f4706c872f2b15bfa77310cdf0400a3a460db22eda41",
"fixtures/config/invalid/empty_string_value.json": "6fac810e014d81b938f66909808e2a2fc1b7cf4db5462fe6522fde1e045cbdda",
- "fixtures/config/invalid/group_experiment_with_assign_by.json": "cbe5d7e00f3abadd490a72055e6197d511fa59f6b3f7062dbacffea426cc5360",
+ "fixtures/config/invalid/experiment_rule_with_experiment_id.json": "6943387d2644b2d93d3798fafe1c6168d9287340b8464c93efa530469ffa18f7",
+ "fixtures/config/invalid/group_experiment_with_assign_by.json": "6aebbfb2896ec6febb20c661a1d2e9d023210d37e60daf6caa9481a559252cba",
"fixtures/config/invalid/group_percentage_rollout_with_assign_by.json": "82189b24744270ac2e4ad93b97efddbede530e6990d597fe80890cbd2820f70e",
- "fixtures/config/invalid/missing_experiment_paused.json": "f3902f40a82f5f80c88a463768b24922bb787b774cb7cd3c4e77fe0e8748e487",
+ "fixtures/config/invalid/holdout_with_id.json": "134ac9d3509fcf1e469ba761ee9d6061e9ad9424a3b66bdfd89efacb5f9c8721",
+ "fixtures/config/invalid/missing_experiment_paused.json": "1fe269eeb50a25923a95e2ca6bc9531b0b36649d53c7cee77074b78e12013399",
"fixtures/config/invalid/missing_on_rollout_miss.json": "88541b7eb14532becb477a9d62797960d8cfdc38bc101b79c5294c1e611e26ec",
"fixtures/config/invalid/missing_rollout_seed.json": "bfc87e05621e30ec19647c8b4dc129b02433c7f2f2e17a9c52f8bf71ad3711ea",
"fixtures/config/invalid/non_uuid_rule_id.json": "3e35c5e211d8ba159824009be3ed26c4ac6f12c1426109fa1b53b49ff63fc770",
- "fixtures/config/invalid/null_experiment_id.json": "de6c7a55d67adce2042d16db287dfcbe49f502d1e5e2f8e71532d0817e28e169",
"fixtures/config/invalid/null_rule_value.json": "ac6f0c04ffbe348a75be7249a7cce80a2ec0411954c46c142eaa3609d2842bd7",
"fixtures/config/invalid/object_value_too_deep.json": "8809275942a5fb5ed548d70d819baffa84ee4ccfec7414cc93903cc79f39c550",
"fixtures/config/invalid/return_value_type_mismatch.json": "b48ead6599cbb40a2a9eea606544b0ee083bb13ce768d57fc225f2da4924633a",
"fixtures/config/invalid/rollout_percentage_above_maximum.json": "9aeefb4ab1eb4140afb213554a91a2c1c085e54cdbe47734442e65ba9a17f99c",
"fixtures/config/invalid/rollout_percentage_below_minimum.json": "dc4102374a3e075c9501f915ead07e8061d46f07293d7b71c0eaa53d4e06fbcf",
- "fixtures/config/invalid/seed_too_long.json": "2ca76285c4f9724603eecb980080bf7ddf0ff8dea4458eb60a2699dfad020941",
+ "fixtures/config/invalid/seed_too_long.json": "fa04a8b212c909684ab93ce189822a16031f432bcc4aa367f0e6ff83960f2805",
"fixtures/config/invalid/targeted_release_with_rollout_fields.json": "7080ad218d8a04b26884aeca4828ada72a21ea8b85453a4f2ae6fafc567223be",
- "fixtures/config/invalid/too_few_variants.json": "22b5201619d80be17bfb21c9a0bf93472e2231ad4a5f1c6bb8136275e64bb8a4",
- "fixtures/config/invalid/too_many_variants.json": "aacabb57255580136b8e6a984c57cea7367e59ddeffed3d4a40137b804cf4c31",
+ "fixtures/config/invalid/too_few_variants.json": "57504f3d25f3f82ee265ef92f9f0764a5f0e0ebe2b3fd90d6330e6af327bc61e",
+ "fixtures/config/invalid/too_many_variants.json": "0f6e4eaae67c1e9da008783ac7fd0dcce5880336ee5695ceaa1ceae1903ee7d4",
"fixtures/config/invalid/unknown_assignment_algorithm.json": "a2cc3d33443fb50ca284ec0a053c1e29b5b752729808aca7f1a5d5009c3cc5cb",
"fixtures/config/invalid/unknown_config_field.json": "68c83e9c074a0a70e7753a6e52daa414aeb18977abdd9b702ec4c5c61d82c3a4",
"fixtures/config/invalid/unknown_config_version.json": "f5bba7fa0ea88e43834ac3b5e03ed4f7297ad0bf4ab2f09d962241521dd06787",
@@ -33,21 +34,22 @@
"fixtures/config/invalid/unknown_property_type.json": "06f5c1dafff26761dfe321765072161a1230e591e1e80a353e5093a8eca041bd",
"fixtures/config/invalid/unknown_rollout_miss_policy.json": "e1ed66e2d00c56c217af240b2cb5af3cc10d2f7f2d26299ecf998bcb73246c22",
"fixtures/config/invalid/unknown_rule_field.json": "8bcc957c38bb2b6a33e244ad1cfe76612cf6f4918c8f0b0cc22f86bd01cd1932",
- "fixtures/config/invalid/variant_key_invalid_characters.json": "f5d71ea6a2ad839fb5f540c8a7f894438fe7c99f152d7469b86af83e938288db",
- "fixtures/config/invalid/variant_value_type_mismatch.json": "aecfc8a8f49f1fa5586082fc4b70284a5212e14998984fc3a7bde1520b933095",
+ "fixtures/config/invalid/variant_key_invalid_characters.json": "c29268de3f3599a3fff87c1f2266d21fe5a005ae161831adf934440a42d3f7ab",
+ "fixtures/config/invalid/variant_value_type_mismatch.json": "35480f8e0c93315378059e1d3c31a506049dee78039ec0e09327437c3f8c02ee",
"fixtures/config/invalid/version_boolean.json": "49587173259e8b379c8f473e60af7bce7a52ad8f17088068dec021c2a3f9242d",
"fixtures/config/invalid/version_string.json": "4835228c5a390aa2497d4df5bd62a41d775c5ff75363d2f7dca0a5eee9e77649",
- "fixtures/config/valid/boolean_all_rule_types.json": "bd56e3422fee0adf18ee9a4ad13ea96fe9bb4e4d11b3e9e0499e6be985b628e8",
+ "fixtures/config/valid/boolean_all_rule_types.json": "fc9a4e604326deb0dea43370c8a22e300de0610f2cfabaf384a9f465e2fcf3d6",
"fixtures/config/valid/boolean_targeted_and_percentage_rollout.json": "ff39ff54ecdc16a52fe963b3d0e7f08b8e417567a147c7730fc932fc366944e8",
- "fixtures/config/valid/group_all_rule_types.json": "bd163291f26d8a851d91ed8c76e4979e9af40608fe0c8a80eefa27c3cdd0f2ad",
+ "fixtures/config/valid/experiment_rule_local_holdout.json": "717160ad80189b99312392b49aa9891601503de57db4f310af75ac5b83d071fd",
+ "fixtures/config/valid/group_all_rule_types.json": "0681c035f38c5c2e58e93bf13b8ebca0057936ddf920c8ff8dd7a0bd854ad582",
"fixtures/config/valid/object_value_max_depth.json": "fb09e42911934248ff767dbcfe61be60fd39483972f3bb8b1f4b1a4e9e46d944",
"fixtures/config/valid/reserved_number_value.json": "6a3ce5d6c5522c71969f9141c5da06e0222f797a321de2c5a90c391bcb937947",
"fixtures/config/valid/reserved_object_value.json": "316ca827654b1a5585d25c534a889e963e5c603d293e605b3902357117802d27",
"fixtures/config/valid/string_group_assignment.json": "414987b3d374fc5fc2d912616cadce8241be9b346d66d6cf4945ca69c169354d",
"fixtures/config/valid/version_float_literal.json": "a848ae2e3e0dd2119600bb90ebbcba59040b3bb788586ba127a57eab1848a403",
- "manifest.json": "e1dc17a0ef7a8388c05262c1a6c388a8237f028b7942bbce9ac1a8b45fa3fb5d",
- "registries/literals.json": "afd25df141b5e05dfa5c8848e1685a4e8b81f3c80fbafc4769c944b815356069",
- "schemas/config.schema.json": "74e43ed13dbfd578bcb8eedc5a247917788b320829b888eab84ce1126841d63b",
+ "manifest.json": "905674e9eff024a4ea58b8ff12c3c604495c84cb31860457d59e6d56e6748aa0",
+ "registries/literals.json": "3e87e6c7c760f0ebac80289faa5412a6ecd17749cb5c79dabe5ed1efaea22578",
+ "schemas/config.schema.json": "163507e0db1219329caef034a50516d27eb5759b5e4c250f313ceb53ca3f1e08",
"schemas/management_warning.schema.json": "9aafe3c28785298bbc143e170881df43866d1b6f1070f685cbdf3d6e4418f361"
}
}
diff --git a/products/feature_flags/backend/test/fixtures/rules_v2_contract/fixtures/config/invalid/null_experiment_id.json b/products/feature_flags/backend/test/fixtures/rules_v2_contract/fixtures/config/invalid/experiment_rule_with_experiment_id.json
similarity index 85%
rename from products/feature_flags/backend/test/fixtures/rules_v2_contract/fixtures/config/invalid/null_experiment_id.json
rename to products/feature_flags/backend/test/fixtures/rules_v2_contract/fixtures/config/invalid/experiment_rule_with_experiment_id.json
index 7e6c8231080d..9f0159d26586 100644
--- a/products/feature_flags/backend/test/fixtures/rules_v2_contract/fixtures/config/invalid/null_experiment_id.json
+++ b/products/feature_flags/backend/test/fixtures/rules_v2_contract/fixtures/config/invalid/experiment_rule_with_experiment_id.json
@@ -9,11 +9,12 @@
"targeting": {
"properties": []
},
- "experiment_id": null,
+ "experiment_id": 42,
+ "paused": false,
"rollout_percentage": 100,
"on_rollout_miss": "continue",
"assignment_algorithm": "sha1_60_v1",
- "seed": "null-experiment-id",
+ "seed": "experiment-id-field",
"variants": [
{
"key": "control",
@@ -25,8 +26,7 @@
"weight": 50,
"value": true
}
- ],
- "paused": false
+ ]
}
]
}
diff --git a/products/feature_flags/backend/test/fixtures/rules_v2_contract/fixtures/config/invalid/group_experiment_with_assign_by.json b/products/feature_flags/backend/test/fixtures/rules_v2_contract/fixtures/config/invalid/group_experiment_with_assign_by.json
index 6da55c0ad12a..ab995886b210 100644
--- a/products/feature_flags/backend/test/fixtures/rules_v2_contract/fixtures/config/invalid/group_experiment_with_assign_by.json
+++ b/products/feature_flags/backend/test/fixtures/rules_v2_contract/fixtures/config/invalid/group_experiment_with_assign_by.json
@@ -9,7 +9,6 @@
"targeting": {
"properties": []
},
- "experiment_id": 42,
"paused": false,
"rollout_percentage": 100,
"on_rollout_miss": "continue",
@@ -29,7 +28,6 @@
}
],
"holdout": {
- "id": 7,
"seed": "holdout-",
"exclusion_percentage": 5
}
diff --git a/products/feature_flags/backend/test/fixtures/rules_v2_contract/fixtures/config/invalid/holdout_with_id.json b/products/feature_flags/backend/test/fixtures/rules_v2_contract/fixtures/config/invalid/holdout_with_id.json
new file mode 100644
index 000000000000..8452e2126833
--- /dev/null
+++ b/products/feature_flags/backend/test/fixtures/rules_v2_contract/fixtures/config/invalid/holdout_with_id.json
@@ -0,0 +1,36 @@
+{
+ "version": 2,
+ "return_type": "boolean",
+ "default_value": false,
+ "rules": [
+ {
+ "id": "aaaaaaaa-aaaa-4aaa-8aaa-aaaaaaaaaaaa",
+ "rule_type": "experiment",
+ "targeting": {
+ "properties": []
+ },
+ "paused": false,
+ "rollout_percentage": 100,
+ "on_rollout_miss": "continue",
+ "assignment_algorithm": "sha1_60_v1",
+ "seed": "holdout-id-field",
+ "variants": [
+ {
+ "key": "control",
+ "weight": 50,
+ "value": false
+ },
+ {
+ "key": "test",
+ "weight": 50,
+ "value": true
+ }
+ ],
+ "holdout": {
+ "id": 7,
+ "seed": "holdout-",
+ "exclusion_percentage": 5
+ }
+ }
+ ]
+}
diff --git a/products/feature_flags/backend/test/fixtures/rules_v2_contract/fixtures/config/invalid/missing_experiment_paused.json b/products/feature_flags/backend/test/fixtures/rules_v2_contract/fixtures/config/invalid/missing_experiment_paused.json
index 7dabb0282af8..4d9c0b1d8117 100644
--- a/products/feature_flags/backend/test/fixtures/rules_v2_contract/fixtures/config/invalid/missing_experiment_paused.json
+++ b/products/feature_flags/backend/test/fixtures/rules_v2_contract/fixtures/config/invalid/missing_experiment_paused.json
@@ -9,7 +9,6 @@
"targeting": {
"properties": []
},
- "experiment_id": 21,
"rollout_percentage": 100,
"on_rollout_miss": "continue",
"assignment_algorithm": "sha1_60_v1",
diff --git a/products/feature_flags/backend/test/fixtures/rules_v2_contract/fixtures/config/invalid/seed_too_long.json b/products/feature_flags/backend/test/fixtures/rules_v2_contract/fixtures/config/invalid/seed_too_long.json
index 6e706d2fdc66..5cb3ff1392cc 100644
--- a/products/feature_flags/backend/test/fixtures/rules_v2_contract/fixtures/config/invalid/seed_too_long.json
+++ b/products/feature_flags/backend/test/fixtures/rules_v2_contract/fixtures/config/invalid/seed_too_long.json
@@ -9,7 +9,6 @@
"targeting": {
"properties": []
},
- "experiment_id": 42,
"paused": false,
"rollout_percentage": 100,
"on_rollout_miss": "continue",
@@ -29,7 +28,6 @@
}
],
"holdout": {
- "id": 7,
"seed": "holdout-",
"exclusion_percentage": 5
}
diff --git a/products/feature_flags/backend/test/fixtures/rules_v2_contract/fixtures/config/invalid/too_few_variants.json b/products/feature_flags/backend/test/fixtures/rules_v2_contract/fixtures/config/invalid/too_few_variants.json
index eac34c695e3a..467d0fd705ab 100644
--- a/products/feature_flags/backend/test/fixtures/rules_v2_contract/fixtures/config/invalid/too_few_variants.json
+++ b/products/feature_flags/backend/test/fixtures/rules_v2_contract/fixtures/config/invalid/too_few_variants.json
@@ -9,7 +9,6 @@
"targeting": {
"properties": []
},
- "experiment_id": 42,
"paused": false,
"rollout_percentage": 100,
"on_rollout_miss": "continue",
@@ -24,7 +23,6 @@
}
],
"holdout": {
- "id": 7,
"seed": "holdout-",
"exclusion_percentage": 5
}
diff --git a/products/feature_flags/backend/test/fixtures/rules_v2_contract/fixtures/config/invalid/too_many_variants.json b/products/feature_flags/backend/test/fixtures/rules_v2_contract/fixtures/config/invalid/too_many_variants.json
index 60de1041556c..61d74e51d4c4 100644
--- a/products/feature_flags/backend/test/fixtures/rules_v2_contract/fixtures/config/invalid/too_many_variants.json
+++ b/products/feature_flags/backend/test/fixtures/rules_v2_contract/fixtures/config/invalid/too_many_variants.json
@@ -9,7 +9,6 @@
"targeting": {
"properties": []
},
- "experiment_id": 42,
"paused": false,
"rollout_percentage": 100,
"on_rollout_miss": "continue",
@@ -124,7 +123,6 @@
}
],
"holdout": {
- "id": 7,
"seed": "holdout-",
"exclusion_percentage": 5
}
diff --git a/products/feature_flags/backend/test/fixtures/rules_v2_contract/fixtures/config/invalid/variant_key_invalid_characters.json b/products/feature_flags/backend/test/fixtures/rules_v2_contract/fixtures/config/invalid/variant_key_invalid_characters.json
index 292ba97aa4f2..79a8fbb51a0e 100644
--- a/products/feature_flags/backend/test/fixtures/rules_v2_contract/fixtures/config/invalid/variant_key_invalid_characters.json
+++ b/products/feature_flags/backend/test/fixtures/rules_v2_contract/fixtures/config/invalid/variant_key_invalid_characters.json
@@ -9,7 +9,6 @@
"targeting": {
"properties": []
},
- "experiment_id": 42,
"paused": false,
"rollout_percentage": 100,
"on_rollout_miss": "continue",
@@ -29,7 +28,6 @@
}
],
"holdout": {
- "id": 7,
"seed": "holdout-",
"exclusion_percentage": 5
}
diff --git a/products/feature_flags/backend/test/fixtures/rules_v2_contract/fixtures/config/invalid/variant_value_type_mismatch.json b/products/feature_flags/backend/test/fixtures/rules_v2_contract/fixtures/config/invalid/variant_value_type_mismatch.json
index 6c777d1eef6e..d5005ad21a63 100644
--- a/products/feature_flags/backend/test/fixtures/rules_v2_contract/fixtures/config/invalid/variant_value_type_mismatch.json
+++ b/products/feature_flags/backend/test/fixtures/rules_v2_contract/fixtures/config/invalid/variant_value_type_mismatch.json
@@ -9,7 +9,6 @@
"targeting": {
"properties": []
},
- "experiment_id": 21,
"rollout_percentage": 100,
"on_rollout_miss": "continue",
"assignment_algorithm": "sha1_60_v1",
diff --git a/products/feature_flags/backend/test/fixtures/rules_v2_contract/fixtures/config/valid/boolean_all_rule_types.json b/products/feature_flags/backend/test/fixtures/rules_v2_contract/fixtures/config/valid/boolean_all_rule_types.json
index ef901e2a6986..212c736daec8 100644
--- a/products/feature_flags/backend/test/fixtures/rules_v2_contract/fixtures/config/valid/boolean_all_rule_types.json
+++ b/products/feature_flags/backend/test/fixtures/rules_v2_contract/fixtures/config/valid/boolean_all_rule_types.json
@@ -42,7 +42,6 @@
"targeting": {
"properties": []
},
- "experiment_id": 42,
"paused": false,
"rollout_percentage": 100,
"on_rollout_miss": "continue",
@@ -62,7 +61,6 @@
}
],
"holdout": {
- "id": 7,
"seed": "holdout-",
"exclusion_percentage": 5
}
diff --git a/products/feature_flags/backend/test/fixtures/rules_v2_contract/fixtures/config/valid/experiment_rule_local_holdout.json b/products/feature_flags/backend/test/fixtures/rules_v2_contract/fixtures/config/valid/experiment_rule_local_holdout.json
new file mode 100644
index 000000000000..ba765ccdb2ef
--- /dev/null
+++ b/products/feature_flags/backend/test/fixtures/rules_v2_contract/fixtures/config/valid/experiment_rule_local_holdout.json
@@ -0,0 +1,51 @@
+{
+ "version": 2,
+ "return_type": "string",
+ "default_value": "standard",
+ "rules": [
+ {
+ "id": "55555555-5555-4555-8555-555555555555",
+ "rule_type": "experiment",
+ "targeting": {
+ "properties": [
+ {
+ "key": "plan",
+ "type": "person",
+ "operator": "exact",
+ "value": [
+ "pro"
+ ]
+ }
+ ]
+ },
+ "description": "Split checkout layouts with a holdout that belongs to this rule.",
+ "paused": false,
+ "rollout_percentage": 80.5,
+ "on_rollout_miss": "return_default",
+ "assignment_algorithm": "sha1_60_v1",
+ "seed": "checkout-layout-split",
+ "assign_by": "person",
+ "variants": [
+ {
+ "key": "standard",
+ "weight": 33.33,
+ "value": "standard"
+ },
+ {
+ "key": "compact",
+ "weight": 33.33,
+ "value": "compact"
+ },
+ {
+ "key": "compact-copy",
+ "weight": 33.34,
+ "value": "compact"
+ }
+ ],
+ "holdout": {
+ "seed": "checkout-layout-holdout",
+ "exclusion_percentage": 10
+ }
+ }
+ ]
+}
diff --git a/products/feature_flags/backend/test/fixtures/rules_v2_contract/fixtures/config/valid/group_all_rule_types.json b/products/feature_flags/backend/test/fixtures/rules_v2_contract/fixtures/config/valid/group_all_rule_types.json
index 4d6a091e86ee..b42c20fddf01 100644
--- a/products/feature_flags/backend/test/fixtures/rules_v2_contract/fixtures/config/valid/group_all_rule_types.json
+++ b/products/feature_flags/backend/test/fixtures/rules_v2_contract/fixtures/config/valid/group_all_rule_types.json
@@ -41,7 +41,6 @@
"targeting": {
"properties": []
},
- "experiment_id": 42,
"paused": false,
"rollout_percentage": 100,
"on_rollout_miss": "continue",
@@ -60,7 +59,6 @@
}
],
"holdout": {
- "id": 7,
"seed": "holdout-",
"exclusion_percentage": 5
}
diff --git a/products/feature_flags/backend/test/fixtures/rules_v2_contract/manifest.json b/products/feature_flags/backend/test/fixtures/rules_v2_contract/manifest.json
index 1c96529d544a..356bbad43d81 100644
--- a/products/feature_flags/backend/test/fixtures/rules_v2_contract/manifest.json
+++ b/products/feature_flags/backend/test/fixtures/rules_v2_contract/manifest.json
@@ -1,34 +1,9 @@
{
- "manifest_version": 1,
"contract": {
"name": "feature_flag_rules_v2",
- "version": "2.3.1",
+ "version": "6.0.0",
"config_version": 2
},
- "corpus": {
- "version": "1.1.0",
- "config_version_locked": 1,
- "case_id_pattern": "^[a-z][a-z0-9_]*(\\.[a-z0-9_]+)+$",
- "published_versions_are_immutable": true,
- "expectation_change": "Any changed expected value requires a new corpus version and a review explanation."
- },
- "schema_dialect": "https://json-schema.org/draft/2020-12/schema",
- "compatibility": {
- "published_versions_are_immutable": true,
- "patch": "Documentation or integrity-tool changes that do not change accepted configs, wire payloads, frozen literals, or fixture expectations.",
- "minor": "Additive fixtures or artifacts that do not change accepted configs, accepted or rejected wire payloads, frozen literals, or published fixture expectations.",
- "major": "Any change to accepted or rejected configs, accepted or rejected wire payloads (responses, definitions, events, diagnostics), frozen literals, or published fixture expectations."
- },
- "integrity": {
- "algorithm": "sha256",
- "index": "SHA256SUMS",
- "content": "raw_file_bytes",
- "entry_format": "<lowercase_hex_digest><two_spaces><relative_posix_path><line_feed>",
- "entry_order": "bytewise_path",
- "excluded_from_index": [
- "SHA256SUMS"
- ]
- },
"artifacts": [
{
"path": "README.md",
@@ -36,13 +11,11 @@
},
{
"path": "registries/literals.json",
- "kind": "registry",
- "version": "2.0.0"
+ "kind": "registry"
},
{
"path": "schemas/config.schema.json",
- "kind": "schema",
- "version": "1.0.0"
+ "kind": "schema"
},
{
"path": "fixtures/config/valid/boolean_all_rule_types.json",
@@ -210,17 +183,6 @@
"instance_path": "/rules/0/variants/0/value"
}
},
- {
- "path": "fixtures/config/invalid/null_experiment_id.json",
- "kind": "fixture",
- "fixture_id": "config.invalid.null_experiment_id",
- "expected": "invalid",
- "expected_failure": {
- "keyword": "type",
- "instance_path": "/rules/0/experiment_id",
- "message_contains": "integer"
- }
- },
{
"path": "fixtures/config/valid/object_value_max_depth.json",
"kind": "fixture",
@@ -422,842 +384,129 @@
"instance_path": "/rules/0/targeting/properties/0/type"
}
},
+ {
+ "path": "fixtures/config/valid/experiment_rule_local_holdout.json",
+ "kind": "fixture",
+ "fixture_id": "config.valid.experiment_rule_local_holdout",
+ "expected": "valid"
+ },
+ {
+ "path": "fixtures/config/invalid/experiment_rule_with_experiment_id.json",
+ "kind": "fixture",
+ "fixture_id": "config.invalid.experiment_rule_with_experiment_id",
+ "expected": "invalid",
+ "expected_failure": {
+ "keyword": "additionalProperties",
+ "instance_path": "/rules/0",
+ "message_contains": "experiment_id"
+ }
+ },
+ {
+ "path": "fixtures/config/invalid/holdout_with_id.json",
+ "kind": "fixture",
+ "fixture_id": "config.invalid.holdout_with_id",
+ "expected": "invalid",
+ "expected_failure": {
+ "keyword": "additionalProperties",
+ "instance_path": "/rules/0/holdout",
+ "message_contains": "id"
+ }
+ },
{
"path": "schemas/hash_sha1_60_v1.schema.json",
- "kind": "schema",
- "version": "1.1.0"
+ "kind": "schema"
},
{
"path": "schemas/v1_evaluation.schema.json",
- "kind": "schema",
- "version": "1.1.0"
+ "kind": "schema"
},
{
"path": "schemas/legacy_projection.schema.json",
- "kind": "schema",
- "version": "1.1.0"
+ "kind": "schema"
},
{
"path": "corpus/hash_sha1_60_v1.json",
- "kind": "corpus",
- "version": "1.1.0",
- "schema": "schemas/hash_sha1_60_v1.schema.json",
- "case_ids": [
- "hash.rollout.ascii_division_edge",
- "hash.rollout.ascii_excluded",
- "hash.variant.ascii",
- "hash.holdout.literal_prefix",
- "hash.concatenation.no_separator_escaping",
- "hash.identifier.empty",
- "hash.identifier.unicode_utf8",
- "hash.identifier.truncated_ascii",
- "hash.identifier.truncated_scalar_values",
- "hash.identifier.exactly_200_multibyte",
- "hash.identifier.integer_group_key",
- "threshold.equal_is_included",
- "threshold.next_binary64_above_is_excluded",
- "threshold.zero_percent_zero_hash",
- "threshold.zero_percent_smallest_hash",
- "threshold.hundred_percent_short_circuit",
- "threshold.two_decimal_division_equal",
- "threshold.two_decimal_division_excluded",
- "threshold.two_decimal_division_included_below_quotient",
- "variant.zero_hash_first_variant",
- "variant.boundary_is_exclusive",
- "variant.just_below_boundary",
- "variant.accumulated_boundary_left_to_right",
- "variant.accumulated_boundary_third",
- "variant.two_decimal_weights_boundary",
- "variant.hash_one_off_the_end",
- "variant.sum_below_one_off_the_end",
- "parity.rollout_identical_bytes",
- "parity.variant_identical_bytes",
- "parity.holdout_identical_bytes",
- "parity.empty_identifier_divergent",
- "parity.final_variant_fallback_divergent"
- ]
+ "kind": "corpus"
},
{
"path": "corpus/v1_evaluation.json",
- "kind": "corpus",
- "version": "1.1.0",
- "schema": "schemas/v1_evaluation.schema.json",
- "case_ids": [
- "v1.boolean.targeted_match_with_payload",
- "v1.boolean.targeting_miss",
- "v1.boolean.rollout_included",
- "v1.boolean.rollout_excluded",
- "v1.boolean.rollout_miss_continues_to_lower_condition",
- "v1.boolean.rollout_miss_early_exit",
- "v1.boolean.reason_priority_keeps_earlier_rollout_miss",
- "v1.boolean.reason_priority_prefers_later_rollout_miss",
- "v1.boolean.null_rollout_is_100",
- "v1.boolean.empty_distinct_id_included_at_zero_percent",
- "v1.boolean.empty_distinct_id_included_below_prefix_hash",
- "v1.multivariate.variant_from_hash_with_payload",
- "v1.multivariate.variant_from_hash_control",
- "v1.multivariate.variant_override",
- "v1.multivariate.invalid_variant_override_uses_hash",
- "v1.multivariate.partial_rollout_included",
- "v1.multivariate.partial_rollout_excluded",
- "v1.multivariate.empty_distinct_id_first_variant",
- "v1.group.included_by_group_key",
- "v1.group.excluded_by_group_key",
- "v1.group.integer_group_key_normalized",
- "v1.group.missing_group_key",
- "v1.dependency.satisfied",
- "v1.dependency.unsatisfied",
- "v1.dependency.variant_match",
- "v1.dependency.variant_mismatch",
- "v1.dependency.missing_target",
- "v1.holdout.member",
- "v1.holdout.not_member",
- "v1.holdout.empty_distinct_id_hashes_prefix",
- "v1.omission.inactive_flag"
- ]
+ "kind": "corpus"
},
{
"path": "corpus/legacy_projection.json",
- "kind": "corpus",
- "version": "1.1.0",
- "schema": "schemas/legacy_projection.schema.json",
- "case_ids": [
- "projection.boolean_true",
- "projection.boolean_true_with_payload",
- "projection.boolean_false_targeting_miss",
- "projection.boolean_false_out_of_rollout",
- "projection.variant",
- "projection.variant_with_payload",
- "projection.holdout_variant",
- "projection.no_group_type",
- "projection.missing_dependency",
- "projection.failed_evaluation",
- "projection.inactive_omitted"
- ]
+ "kind": "corpus"
},
{
"path": "schemas/definitions_entry.schema.json",
- "kind": "schema",
- "version": "1.0.0"
+ "kind": "schema"
},
{
"path": "schemas/definitions_v2.schema.json",
- "kind": "schema",
- "version": "1.0.0"
+ "kind": "schema"
},
{
"path": "schemas/flags_response_v3.schema.json",
- "kind": "schema",
- "version": "1.0.1"
+ "kind": "schema"
},
{
"path": "schemas/flags_response_v3_presence.schema.json",
- "kind": "schema",
- "version": "1.0.0"
+ "kind": "schema"
},
{
"path": "schemas/management_warning.schema.json",
- "kind": "schema",
- "version": "1.0.0"
+ "kind": "schema"
},
{
"path": "schemas/management_error.schema.json",
- "kind": "schema",
- "version": "1.0.0"
+ "kind": "schema"
},
{
"path": "schemas/feature_flag_called_context.schema.json",
- "kind": "schema",
- "version": "1.0.1"
+ "kind": "schema"
},
{
"path": "schemas/experiment_exposure_properties.schema.json",
- "kind": "schema",
- "version": "1.0.1"
+ "kind": "schema"
},
{
"path": "rules/response_presence.json",
- "kind": "presence_matrix",
- "version": "1.0.0"
+ "kind": "presence_matrix"
},
{
"path": "fixtures/wire/definitions.json",
- "kind": "fixture_set",
- "version": "1.0.0",
- "schema": "schemas/definitions_entry.schema.json",
- "case_ids": [
- "definitions.v2_canonical",
- "definitions.v1_implicit",
- "definitions.v1_explicit",
- "definitions.v2_numeric_version",
- "definitions.inactive",
- "definitions.unsupported_version_future",
- "definitions.unsupported_version_string",
- "definitions.unsupported_version_boolean",
- "definitions.unsupported_version_null",
- "definitions.unknown_semantic_field",
- "definitions.unknown_rule_field",
- "definitions.unsupported_rule_type",
- "definitions.missing_assignment_seed",
- "definitions.v1_null_version",
- "definitions.v1_wrong_type_version",
- "definitions.v1_null_ensure_experience_continuity",
- "definitions.v1_wrong_type_ensure_experience_continuity",
- "definitions.v1_explicit_null_row_fields",
- "definitions.missing_id",
- "definitions.missing_key",
- "definitions.missing_active",
- "definitions.missing_filters",
- "definitions.unknown_field"
- ]
- },
- {
- "path": "fixtures/wire/definitions_feed.json",
- "kind": "fixture_set",
- "version": "1.0.0",
- "schema": "schemas/definitions_v2.schema.json",
- "case_ids": [
- "definitions_feed.mixed_versions",
- "definitions_feed.missing_cohorts",
- "definitions_feed.minimal_events_false",
- "definitions_feed.minimal_events_true",
- "definitions_feed.minimal_events_wrong_type",
- "definitions_feed.null_legacy_row_fields",
- "definitions_feed.missing_flags",
- "definitions_feed.missing_group_type_mapping",
- "definitions_feed.unknown_field"
- ]
+ "kind": "fixture_set"
},
{
"path": "fixtures/wire/responses.json",
- "kind": "fixture_set",
- "version": "1.0.0",
- "schema": "schemas/flags_response_v3.schema.json",
- "case_ids": [
- "responses.targeting_match_targeted_release",
- "responses.missing_id",
- "responses.missing_version",
- "responses.missing_config_version",
- "responses.targeting_match_targeted_release_missing_has_experiment",
- "responses.targeting_match_targeted_release_missing_rule_type",
- "responses.targeting_match_targeted_release_missing_rule_id",
- "responses.targeting_match_targeted_release_forbidden_experiment_id",
- "responses.targeting_match_targeted_release_forbidden_variant_key",
- "responses.targeting_match_targeted_release_forbidden_holdout_id",
- "responses.targeting_match_targeted_release_wrong_condition_index",
- "responses.targeting_match_targeted_release_forbidden_failed",
- "responses.targeting_match_targeted_release_flag_with_experiment",
- "responses.targeting_match_percentage_rollout",
- "responses.targeting_match_percentage_rollout_missing_has_experiment",
- "responses.targeting_match_percentage_rollout_missing_rule_type",
- "responses.targeting_match_percentage_rollout_missing_rule_id",
- "responses.targeting_match_percentage_rollout_forbidden_experiment_id",
- "responses.targeting_match_percentage_rollout_forbidden_variant_key",
- "responses.targeting_match_percentage_rollout_forbidden_holdout_id",
- "responses.targeting_match_percentage_rollout_wrong_condition_index",
- "responses.targeting_match_percentage_rollout_forbidden_failed",
- "responses.experiment_split",
- "responses.experiment_split_missing_has_experiment",
- "responses.experiment_split_missing_rule_type",
- "responses.experiment_split_missing_rule_id",
- "responses.experiment_split_missing_experiment_id",
- "responses.experiment_split_missing_variant_key",
- "responses.experiment_split_forbidden_holdout_id",
- "responses.experiment_split_wrong_condition_index",
- "responses.experiment_split_forbidden_failed",
- "responses.experiment_split_contradicted_has_experiment",
- "responses.rollout_miss_percentage_rollout",
- "responses.rollout_miss_percentage_rollout_missing_has_experiment",
- "responses.rollout_miss_percentage_rollout_missing_rule_type",
- "responses.rollout_miss_percentage_rollout_missing_rule_id",
- "responses.rollout_miss_percentage_rollout_forbidden_experiment_id",
- "responses.rollout_miss_percentage_rollout_forbidden_variant_key",
- "responses.rollout_miss_percentage_rollout_forbidden_holdout_id",
- "responses.rollout_miss_percentage_rollout_wrong_condition_index",
- "responses.rollout_miss_percentage_rollout_forbidden_failed",
- "responses.rollout_miss_experiment",
- "responses.rollout_miss_experiment_missing_has_experiment",
- "responses.rollout_miss_experiment_missing_rule_type",
- "responses.rollout_miss_experiment_missing_rule_id",
- "responses.rollout_miss_experiment_missing_experiment_id",
- "responses.rollout_miss_experiment_forbidden_variant_key",
- "responses.rollout_miss_experiment_forbidden_holdout_id",
- "responses.rollout_miss_experiment_wrong_condition_index",
- "responses.rollout_miss_experiment_forbidden_failed",
- "responses.experiment_paused",
- "responses.experiment_paused_missing_has_experiment",
- "responses.experiment_paused_missing_rule_type",
- "responses.experiment_paused_missing_rule_id",
- "responses.experiment_paused_missing_experiment_id",
- "responses.experiment_paused_forbidden_variant_key",
- "responses.experiment_paused_forbidden_holdout_id",
- "responses.experiment_paused_wrong_condition_index",
- "responses.experiment_paused_forbidden_failed",
- "responses.holdout",
- "responses.holdout_missing_has_experiment",
- "responses.holdout_missing_rule_type",
- "responses.holdout_missing_rule_id",
- "responses.holdout_missing_experiment_id",
- "responses.holdout_missing_holdout_id",
- "responses.holdout_forbidden_variant_key",
- "responses.holdout_wrong_condition_index",
- "responses.holdout_forbidden_failed",
- "responses.no_rule_match",
- "responses.no_rule_match_missing_has_experiment",
- "responses.no_rule_match_forbidden_rule_type",
- "responses.no_rule_match_forbidden_rule_id",
- "responses.no_rule_match_forbidden_experiment_id",
- "responses.no_rule_match_forbidden_variant_key",
- "responses.no_rule_match_forbidden_holdout_id",
- "responses.no_rule_match_wrong_condition_index",
- "responses.no_rule_match_forbidden_failed",
- "responses.missing_group_key",
- "responses.missing_group_key_missing_has_experiment",
- "responses.missing_group_key_forbidden_rule_type",
- "responses.missing_group_key_forbidden_rule_id",
- "responses.missing_group_key_forbidden_experiment_id",
- "responses.missing_group_key_forbidden_variant_key",
- "responses.missing_group_key_forbidden_holdout_id",
- "responses.missing_group_key_wrong_condition_index",
- "responses.missing_group_key_forbidden_failed",
- "responses.dependency_error",
- "responses.dependency_error_missing_has_experiment",
- "responses.dependency_error_forbidden_rule_type",
- "responses.dependency_error_forbidden_rule_id",
- "responses.dependency_error_forbidden_experiment_id",
- "responses.dependency_error_forbidden_variant_key",
- "responses.dependency_error_forbidden_holdout_id",
- "responses.dependency_error_wrong_condition_index",
- "responses.dependency_error_missing_failed",
- "responses.dependency_error_nonnull_value",
- "responses.error",
- "responses.error_missing_has_experiment",
- "responses.error_forbidden_rule_type",
- "responses.error_forbidden_rule_id",
- "responses.error_forbidden_experiment_id",
- "responses.error_forbidden_variant_key",
- "responses.error_forbidden_holdout_id",
- "responses.error_wrong_condition_index",
- "responses.error_missing_failed",
- "responses.error_nonnull_value",
- "responses.v1",
- "responses.omitted",
- "responses.split_wrong_type_config_version",
- "responses.split_wrong_type_rule_type",
- "responses.split_wrong_type_rule_id",
- "responses.split_wrong_type_experiment_id",
- "responses.split_wrong_type_variant_key",
- "responses.forbidden_legacy_enabled",
- "responses.forbidden_legacy_variant",
- "responses.split_arm_b_equal_value",
- "responses.split_string_value",
- "responses.null_default",
- "responses.request_failure_omission",
- "responses.missing_group_nonnull_value",
- "responses.v1_failed",
- "responses.v1_quick_win",
- "responses.v2_forced_variant",
- "responses.key_mismatch",
- "responses.failure_envelope",
- "responses.local_only_reason",
- "responses.nested_seed",
- "responses.passthrough_seed",
- "responses.reserved_number_wire_value",
- "responses.reserved_object_wire_value",
- "responses.targeting_match_targeted_release_wrong_rule_type",
- "responses.experiment_split_wrong_rule_type",
- "responses.rollout_miss_percentage_rollout_wrong_rule_type",
- "responses.rollout_miss_experiment_wrong_rule_type",
- "responses.experiment_paused_wrong_rule_type",
- "responses.holdout_wrong_rule_type",
- "responses.targeting_match_targeted_release_null_value",
- "responses.targeting_match_percentage_rollout_null_value",
- "responses.experiment_split_null_value",
- "responses.empty_v2_string",
- "responses.v1_experiment_without_linkage",
- "responses.v1_ambiguous_experiment",
- "responses.flag_key_seed",
- "responses.seed_named_flag_nested_seed",
- "responses.flag_key_assignment_seed",
- "responses.v1_forbidden_rule_id",
- "responses.v1_forbidden_rule_type",
- "responses.v1_forbidden_holdout_id",
- "responses.v1_object_payload",
- "responses.v1_null_value_not_failed"
- ]
- },
- {
- "path": "fixtures/wire/warnings.json",
- "kind": "fixture_set",
- "version": "1.0.0",
- "schema": "schemas/management_warning.schema.json",
- "case_ids": [
- "warnings.rule_order_changes_traffic",
- "warnings.unreachable_lower_rule",
- "warnings.rollout_miss_can_enter_lower_rule",
- "warnings.assignment_reset_changes_traffic",
- "warnings.conclusion_expands_population",
- "warnings.detail_and_attr",
- "warnings.unknown_code",
- "warnings.missing_code",
- "warnings.wrong_detail_type",
- "warnings.unknown_field"
- ]
- },
- {
- "path": "fixtures/wire/errors.json",
- "kind": "fixture_set",
- "version": "1.0.0",
- "schema": "schemas/management_error.schema.json",
- "case_ids": [
- "errors.validation",
- "errors.nonfield",
- "errors.missing_type",
- "errors.missing_code",
- "errors.missing_detail",
- "errors.missing_attr",
- "errors.wrong_detail_type",
- "errors.unknown_field"
- ]
- },
- {
- "path": "fixtures/wire/calls.json",
- "kind": "fixture_set",
- "version": "1.0.0",
- "schema": "schemas/feature_flag_called_context.schema.json",
- "case_ids": [
- "calls.targeting_match_targeted_release",
- "calls.targeting_match_targeted_release_missing_rule_type",
- "calls.targeting_match_targeted_release_missing_rule_id",
- "calls.targeting_match_targeted_release_forbidden_experiment_id",
- "calls.targeting_match_targeted_release_forbidden_variant_key",
- "calls.targeting_match_targeted_release_forbidden_holdout_id",
- "calls.targeting_match_percentage_rollout",
- "calls.targeting_match_percentage_rollout_missing_rule_type",
- "calls.targeting_match_percentage_rollout_missing_rule_id",
- "calls.targeting_match_percentage_rollout_forbidden_experiment_id",
- "calls.targeting_match_percentage_rollout_forbidden_variant_key",
- "calls.targeting_match_percentage_rollout_forbidden_holdout_id",
- "calls.experiment_split",
- "calls.experiment_split_missing_rule_type",
- "calls.experiment_split_missing_rule_id",
- "calls.experiment_split_missing_experiment_id",
- "calls.experiment_split_missing_variant_key",
- "calls.experiment_split_forbidden_holdout_id",
- "calls.rollout_miss_percentage_rollout",
- "calls.rollout_miss_percentage_rollout_missing_rule_type",
- "calls.rollout_miss_percentage_rollout_missing_rule_id",
- "calls.rollout_miss_percentage_rollout_forbidden_experiment_id",
- "calls.rollout_miss_percentage_rollout_forbidden_variant_key",
- "calls.rollout_miss_percentage_rollout_forbidden_holdout_id",
- "calls.rollout_miss_experiment",
- "calls.rollout_miss_experiment_missing_rule_type",
- "calls.rollout_miss_experiment_missing_rule_id",
- "calls.rollout_miss_experiment_missing_experiment_id",
- "calls.rollout_miss_experiment_forbidden_variant_key",
- "calls.rollout_miss_experiment_forbidden_holdout_id",
- "calls.experiment_paused",
- "calls.experiment_paused_missing_rule_type",
- "calls.experiment_paused_missing_rule_id",
- "calls.experiment_paused_missing_experiment_id",
- "calls.experiment_paused_forbidden_variant_key",
- "calls.experiment_paused_forbidden_holdout_id",
- "calls.holdout",
- "calls.holdout_missing_rule_type",
- "calls.holdout_missing_rule_id",
- "calls.holdout_missing_experiment_id",
- "calls.holdout_missing_holdout_id",
- "calls.holdout_forbidden_variant_key",
- "calls.no_rule_match",
- "calls.no_rule_match_forbidden_rule_type",
- "calls.no_rule_match_forbidden_rule_id",
- "calls.no_rule_match_forbidden_experiment_id",
- "calls.no_rule_match_forbidden_variant_key",
- "calls.no_rule_match_forbidden_holdout_id",
- "calls.missing_group_key",
- "calls.missing_group_key_forbidden_rule_type",
- "calls.missing_group_key_forbidden_rule_id",
- "calls.missing_group_key_forbidden_experiment_id",
- "calls.missing_group_key_forbidden_variant_key",
- "calls.missing_group_key_forbidden_holdout_id",
- "calls.dependency_error",
- "calls.dependency_error_forbidden_rule_type",
- "calls.dependency_error_forbidden_rule_id",
- "calls.dependency_error_forbidden_experiment_id",
- "calls.dependency_error_forbidden_variant_key",
- "calls.dependency_error_forbidden_holdout_id",
- "calls.error",
- "calls.error_forbidden_rule_type",
- "calls.error_forbidden_rule_id",
- "calls.error_forbidden_experiment_id",
- "calls.error_forbidden_variant_key",
- "calls.error_forbidden_holdout_id",
- "calls.v1",
- "calls.missing",
- "calls.inactive",
- "calls.v1_quick_win",
- "calls.arm_b_equal_value",
- "calls.string",
- "calls.v2_forced_variant",
- "calls.forbidden_value_number",
- "calls.forbidden_value_object",
- "calls.forbidden_value_array",
- "calls.forbidden_value_null",
- "calls.forbidden_value_empty_string",
- "calls.partial_split_diagnostic",
- "calls.inactive_forbidden_experiment_id",
- "calls.missing_feature_flag",
- "calls.missing_feature_flag_response",
- "calls.unknown_field"
- ]
- },
- {
- "path": "fixtures/wire/exposures.json",
- "kind": "fixture_set",
- "version": "1.0.0",
- "schema": "schemas/experiment_exposure_properties.schema.json",
- "case_ids": [
- "exposures.remote_split",
- "exposures.local_split",
- "exposures.arm_b_equal_value",
- "exposures.successor_experiment",
- "exposures.string",
- "exposures.missing_feature_flag",
- "exposures.wrong_type_feature_flag",
- "exposures.missing_feature_flag_response",
- "exposures.wrong_type_feature_flag_response",
- "exposures.missing_feature_flag_config_version",
- "exposures.wrong_type_feature_flag_config_version",
- "exposures.missing_feature_flag_reason_code",
- "exposures.wrong_type_feature_flag_reason_code",
- "exposures.missing_feature_flag_variant",
- "exposures.wrong_type_feature_flag_variant",
- "exposures.missing_feature_flag_rule_id",
- "exposures.wrong_type_feature_flag_rule_id",
- "exposures.missing_feature_flag_rule_type",
- "exposures.wrong_type_feature_flag_rule_type",
- "exposures.missing_feature_flag_experiment_id",
- "exposures.wrong_type_feature_flag_experiment_id",
- "exposures.missing_experiment_exposure_source",
- "exposures.wrong_type_experiment_exposure_source",
- "exposures.missing_locally_evaluated",
- "exposures.wrong_type_locally_evaluated",
- "exposures.forbidden_outcome_holdout",
- "exposures.forbidden_outcome_pause",
- "exposures.forbidden_outcome_rollout_miss",
- "exposures.forbidden_outcome_no_rule",
- "exposures.forbidden_outcome_failed",
- "exposures.forbidden_value_object",
- "exposures.forbidden_value_array",
- "exposures.forbidden_value_null",
- "exposures.forbidden_value_empty_string",
- "exposures.holdout_context",
- "exposures.ingestion_source",
- "exposures.v1_config",
- "exposures.forced_variant_context",
- "exposures.unknown_field"
- ]
+ "kind": "fixture_set"
},
{
"path": "fixtures/wire/readers.json",
- "kind": "fixture_set",
- "version": "1.0.0",
- "schema": "schemas/flags_response_v3.schema.json",
- "case_ids": [
- "readers.complete_split",
- "readers.partial_split_missing_config_version",
- "readers.partial_split_missing_rule_type",
- "readers.partial_split_missing_rule_id",
- "readers.partial_split_missing_experiment_id",
- "readers.partial_split_missing_variant_key",
- "readers.split_wrong_type_config_version",
- "readers.split_wrong_type_rule_type",
- "readers.split_wrong_type_rule_id",
- "readers.split_wrong_type_experiment_id",
- "readers.split_wrong_type_variant_key",
- "readers.wrong_rule_type_literal",
- "readers.unknown_fields",
- "readers.old_server_variant",
- "readers.old_server_boolean",
- "readers.old_server_null_variant",
- "readers.old_server_false",
- "readers.v3_value_takes_precedence",
- "readers.missing_flag",
- "readers.targeting_match_targeted_release",
- "readers.targeting_match_percentage_rollout",
- "readers.rollout_miss_percentage_rollout",
- "readers.rollout_miss_experiment",
- "readers.experiment_paused",
- "readers.holdout",
- "readers.holdout_variant_key",
- "readers.no_rule_match",
- "readers.missing_group_key",
- "readers.dependency_error",
- "readers.error",
- "readers.error_with_value",
- "readers.malformed_sibling_isolation",
- "readers.partial_split_string_value",
- "readers.v1_at_v3",
- "readers.v1_failed",
- "readers.integral_json_number_context",
- "readers.flag_key_seed"
- ]
- },
- {
- "path": "fixtures/wire/exposure_transport.json",
- "kind": "fixture_set",
- "version": "1.0.0",
- "schema": "schemas/experiment_exposure_properties.schema.json",
- "case_ids": [
- "exposure_transport.exposure_with_transport",
- "exposure_transport.nested_seed",
- "exposure_transport.nested_holdout_seed",
- "exposure_transport.holdout_context",
- "exposure_transport.forced_variant_context",
- "exposure_transport.group_type_named_seed"
- ]
- },
- {
- "path": "fixtures/wire/call_transport.json",
- "kind": "fixture_set",
- "version": "1.0.0",
- "schema": "schemas/feature_flag_called_context.schema.json",
- "case_ids": [
- "call_transport.call_with_transport",
- "call_transport.nested_assignment_seed",
- "call_transport.prefixed_seed",
- "call_transport.person_property_named_seed"
- ]
+ "kind": "fixture_set"
},
{
"path": "schemas/v2_boolean_evaluation.schema.json",
- "kind": "schema",
- "version": "1.0.0"
+ "kind": "schema"
},
{
"path": "corpus/v2_boolean_evaluation.json",
- "kind": "boolean_evaluation_corpus",
- "version": "1.0.0",
- "schema": "schemas/v2_boolean_evaluation.schema.json",
- "component": "v2_boolean_evaluation",
- "case_ids": [
- "v2_boolean.empty.null",
- "v2_boolean.empty.false",
- "v2_boolean.empty.true",
- "v2_boolean.target.false",
- "v2_boolean.target.true",
- "v2_boolean.order.false_claims",
- "v2_boolean.order.reversed",
- "v2_boolean.order.original_index",
- "v2_boolean.miss.continue",
- "v2_boolean.miss.continue_false_default",
- "v2_boolean.miss.return_default",
- "v2_boolean.miss.return_default_false_default",
- "v2_boolean.miss.consecutive",
- "v2_boolean.miss.exhausted",
- "v2_boolean.order.percentage_false",
- "v2_boolean.order.targeting_miss",
- "v2_boolean.miss.targeting_miss_continues",
- "v2_boolean.property.exact_case",
- "v2_boolean.property.exact_scalar",
- "v2_boolean.property.exact_false",
- "v2_boolean.property.is_not",
- "v2_boolean.property.contains",
- "v2_boolean.property.not_contains",
- "v2_boolean.property.starts",
- "v2_boolean.property.not_starts",
- "v2_boolean.property.ends",
- "v2_boolean.property.not_ends",
- "v2_boolean.property.contains_multi",
- "v2_boolean.property.not_contains_multi",
- "v2_boolean.property.greater",
- "v2_boolean.property.greater_equal",
- "v2_boolean.property.less",
- "v2_boolean.property.less_equal",
- "v2_boolean.property.regex",
- "v2_boolean.property.not_regex",
- "v2_boolean.property.not_regex_negated",
- "v2_boolean.property.regex_search",
- "v2_boolean.property.semver_eq",
- "v2_boolean.property.semver_neq",
- "v2_boolean.property.semver_gt",
- "v2_boolean.property.semver_gte",
- "v2_boolean.property.semver_lt",
- "v2_boolean.property.semver_lte",
- "v2_boolean.property.semver_tilde",
- "v2_boolean.property.semver_caret",
- "v2_boolean.property.semver_wildcard",
- "v2_boolean.property.semver_bad_subject",
- "v2_boolean.property.date_exact",
- "v2_boolean.property.date_after",
- "v2_boolean.property.date_before",
- "v2_boolean.property.date_before_negated",
- "v2_boolean.property.date_relative",
- "v2_boolean.property.relative_unsigned",
- "v2_boolean.property.relative_hours",
- "v2_boolean.property.relative_weeks",
- "v2_boolean.property.relative_year_leap_clamp",
- "v2_boolean.property.relative_month_clamp",
- "v2_boolean.property.date_epoch",
- "v2_boolean.property.date_epoch_string",
- "v2_boolean.property.date_bad_subject",
- "v2_boolean.property.legacy_exact_setting",
- "v2_boolean.property.timezone",
- "v2_boolean.property.wall_clock_no_offset",
- "v2_boolean.property.relative_dst",
- "v2_boolean.property.date_exact_same_day_miss",
- "v2_boolean.property.date_before_equal",
- "v2_boolean.property.date_after_equal",
- "v2_boolean.property.relative_dst_elapsed_miss",
- "v2_boolean.property.relative_dst_fall",
- "v2_boolean.property.relative_dst_overlap_earliest",
- "v2_boolean.property.relative_dst_overlap_later_miss",
- "v2_boolean.property.relative_dst_gap",
- "v2_boolean.property.null_is_set",
- "v2_boolean.property.missing_is_not_set",
- "v2_boolean.property.null_is_not_set",
- "v2_boolean.property.missing_exact",
- "v2_boolean.property.missing_is_not",
- "v2_boolean.property.negated_missing",
- "v2_boolean.property.negation",
- "v2_boolean.property.double_negative",
- "v2_boolean.property.negated_match",
- "v2_boolean.property.and",
- "v2_boolean.property.and_miss",
- "v2_boolean.context.partial_is_not",
- "v2_boolean.context.partial_is_not_set",
- "v2_boolean.context.partial_exact",
- "v2_boolean.context.unavailable",
- "v2_boolean.context.unneeded",
- "v2_boolean.context.partial_present",
- "v2_boolean.context.false_before_missing",
- "v2_boolean.context.missing_before_false",
- "v2_boolean.error.invalid_regex_false",
- "v2_boolean.error.invalid_regex_true",
- "v2_boolean.error.numeric",
- "v2_boolean.error.false_short_circuits",
- "v2_boolean.error.terminal_short_circuits",
- "v2_boolean.hash.ascii",
- "v2_boolean.hash.seed_parity",
- "v2_boolean.hash.empty",
- "v2_boolean.hash.empty_return_default",
- "v2_boolean.hash.ascii_200",
- "v2_boolean.hash.ascii_201",
- "v2_boolean.hash.multibyte_200",
- "v2_boolean.hash.multibyte_201",
- "v2_boolean.hash.supplementary_200",
- "v2_boolean.hash.supplementary_201",
- "v2_boolean.hash.composed",
- "v2_boolean.hash.decomposed",
- "v2_boolean.hash.zero",
- "v2_boolean.hash.full",
- "v2_boolean.hash.empty_full",
- "v2_boolean.hash.partial",
- "v2_boolean.hash.fraction",
- "v2_boolean.hash.seed_changed",
- "v2_boolean.hash.subject_changed",
- "v2_boolean.hash.renamed",
- "v2_boolean.hash.moved",
- "v2_boolean.property.long_value",
- "v2_boolean.threshold.p0_equal",
- "v2_boolean.threshold.p0_above",
- "v2_boolean.threshold.p33_33_equal",
- "v2_boolean.threshold.p33_33_below",
- "v2_boolean.threshold.p33_33_above",
- "v2_boolean.threshold.p50_equal",
- "v2_boolean.threshold.p50_below",
- "v2_boolean.threshold.p50_above",
- "v2_boolean.threshold.p100_equal",
- "v2_boolean.threshold.p100_below",
- "v2_boolean.eligibility.inactive",
- "v2_boolean.eligibility.deleted",
- "v2_boolean.eligibility.request_filtered",
- "v2_boolean.parser.malformed",
- "v2_boolean.parser.unsupported",
- "v2_boolean.error.invalid_not_regex_negated",
- "v2_boolean.target.empty_identifier"
- ]
+ "kind": "boolean_evaluation_corpus"
},
{
"path": "schemas/v2_value_evaluation.schema.json",
- "kind": "schema",
- "version": "1.0.0"
+ "kind": "schema"
},
{
"path": "corpus/v2_value_evaluation.json",
- "kind": "value_evaluation_corpus",
- "version": "1.0.0",
- "schema": "schemas/v2_value_evaluation.schema.json",
- "component": "v2_value_evaluation",
- "case_ids": [
- "v2_value.string.target",
- "v2_value.string.rollout_hit",
- "v2_value.string.miss_return_default",
- "v2_value.string.miss_return_default_null",
- "v2_value.string.miss_continue",
- "v2_value.string.no_match_default",
- "v2_value.string.no_match_null",
- "v2_value.string.order_first_match",
- "v2_value.string.order_targeting_miss",
- "v2_value.number.target",
- "v2_value.number.rollout_hit",
- "v2_value.number.miss_return_default",
- "v2_value.number.miss_return_default_null",
- "v2_value.number.miss_continue",
- "v2_value.number.no_match_default",
- "v2_value.number.no_match_null",
- "v2_value.number.order_first_match",
- "v2_value.number.order_targeting_miss",
- "v2_value.object.target",
- "v2_value.object.rollout_hit",
- "v2_value.object.miss_return_default",
- "v2_value.object.miss_return_default_null",
- "v2_value.object.miss_continue",
- "v2_value.object.no_match_default",
- "v2_value.object.no_match_null",
- "v2_value.object.order_first_match",
- "v2_value.object.order_targeting_miss",
- "v2_value.string.unicode",
- "v2_value.string.looks_boolean",
- "v2_value.number.zero",
- "v2_value.number.negative_decimal",
- "v2_value.number.short_decimal",
- "v2_value.number.exponent",
- "v2_value.number.max_safe_integer",
- "v2_value.number.min_safe_integer",
- "v2_value.object.empty",
- "v2_value.object.nested",
- "v2_value.object.max_depth",
- "v2_value.parser.string_empty_value",
- "v2_value.parser.string_empty_default",
- "v2_value.parser.string_value_mismatch",
- "v2_value.parser.number_value_mismatch",
- "v2_value.parser.number_default_mismatch",
- "v2_value.parser.number_above_safe_integer",
- "v2_value.parser.number_below_safe_integer",
- "v2_value.parser.object_array_value",
- "v2_value.parser.object_default_mismatch",
- "v2_value.parser.object_too_deep",
- "v2_value.parser.object_nested_number_above_safe_integer"
- ]
+ "kind": "value_evaluation_corpus"
+ },
+ {
+ "path": "schemas/v2_experiment_evaluation.schema.json",
+ "kind": "schema"
+ },
+ {
+ "path": "corpus/v2_experiment_evaluation.json",
+ "kind": "experiment_evaluation_corpus"
}
- ],
- "wire_contract": {
- "version": "1.0.1",
- "fixture_version": "1.0.0",
- "published_versions_are_immutable": true,
- "producer_schema_sha256": "98c1397bb7b5a1ceb3f317b34c6c57cff0ab30e94bf5945ee7b56f293536b70f",
- "fixture_encoding": "Each case deep-copies its named template, removes the listed existing JSON Pointer members, then sets members by JSON Pointer. Set requires an existing parent. Templates and expectations are JSON, not executable SDK suites.",
- "compatibility": "New artifacts are additive to contract 1.2.0. Published schema bytes and fixture expectations are immutable. Accepted/rejected wire-contract changes require a new major wire component version; additive cases require a new minor fixture version. Reader expectations never relax producer validation."
- },
- "v2_boolean_evaluation": {
- "version": "1.0.0",
- "config_version_locked": 2,
- "published_versions_are_immutable": true
- },
- "v2_value_evaluation": {
- "version": "1.0.0",
- "config_version_locked": 2,
- "published_versions_are_immutable": true
- }
+ ]
}
diff --git a/products/feature_flags/backend/test/fixtures/rules_v2_contract/registries/literals.json b/products/feature_flags/backend/test/fixtures/rules_v2_contract/registries/literals.json
index ed8065689791..097bc7fd8a57 100644
--- a/products/feature_flags/backend/test/fixtures/rules_v2_contract/registries/literals.json
+++ b/products/feature_flags/backend/test/fixtures/rules_v2_contract/registries/literals.json
@@ -1,5 +1,4 @@
{
- "registry_version": "2.0.0",
"config_versions": {
"v1": 1,
"v2": 2,
@@ -15,7 +14,6 @@
"default_value",
"description",
"exclusion_percentage",
- "experiment_id",
"group_key_names",
"group_type_index",
"holdout",
@@ -146,7 +144,7 @@
"experiment_split": {
"reason": "SPLIT",
"error_code": null,
- "note": "Experiment identity is in flag metadata."
+ "note": "The rule ID and variant key are in flag metadata."
},
"rollout_miss": {
"reason": "DEFAULT",
@@ -231,9 +229,7 @@
"config_version",
"rule_type",
"rule_id",
- "experiment_id",
"variant_key",
- "holdout_id",
"id",
"version",
"reason_code",
diff --git a/products/feature_flags/backend/test/fixtures/rules_v2_contract/schemas/config.schema.json b/products/feature_flags/backend/test/fixtures/rules_v2_contract/schemas/config.schema.json
index f44d12f0c44a..e1b7335ecfa4 100644
--- a/products/feature_flags/backend/test/fixtures/rules_v2_contract/schemas/config.schema.json
+++ b/products/feature_flags/backend/test/fixtures/rules_v2_contract/schemas/config.schema.json
@@ -365,14 +365,10 @@
"holdout": {
"type": "object",
"required": [
- "id",
"seed",
"exclusion_percentage"
],
"properties": {
- "id": {
- "type": "integer"
- },
"seed": {
"$ref": "#/$defs/seed"
},
@@ -466,7 +462,6 @@
"id",
"rule_type",
"targeting",
- "experiment_id",
"paused",
"rollout_percentage",
"on_rollout_miss",
@@ -490,9 +485,6 @@
"metadata": {
"$ref": "#/$defs/metadata"
},
- "experiment_id": {
- "type": "integer"
- },
"paused": {
"type": "boolean"
},
diff --git a/products/feature_flags/backend/test/test_facade_config.py b/products/feature_flags/backend/test/test_facade_config.py
index b47531728059..bbfe1df1346a 100644
--- a/products/feature_flags/backend/test/test_facade_config.py
+++ b/products/feature_flags/backend/test/test_facade_config.py
@@ -66,7 +66,7 @@ V2_BOOLEAN_ALL_RULE_TYPES: dict[str, Any] = {
{"key": "control", "weight": 50, "value": True},
{"key": "test", "weight": 50, "value": True},
],
- "holdout": {"id": 7, "seed": "holdout-", "exclusion_percentage": 5},
+ "holdout": {"seed": "holdout-", "exclusion_percentage": 5},
},
],
}
@@ -145,6 +145,7 @@ class TestParseV2Config:
id="44444444-4444-4444-8444-444444444444",
rule_type="experiment",
value=None,
+ variant_values=(True, True),
),
),
aggregation_group_type_index=None,
diff --git a/products/feature_flags/backend/test/test_facade_config_validation.py b/products/feature_flags/backend/test/test_facade_config_validation.py
index aa46ca7da592..a1b5c9d27a63 100644
--- a/products/feature_flags/backend/test/test_facade_config_validation.py
+++ b/products/feature_flags/backend/test/test_facade_config_validation.py
@@ -17,16 +17,21 @@ from products.feature_flags.backend.api.feature_flag import calculate_filter_siz
from products.feature_flags.backend.facade import config_validation
from products.feature_flags.backend.facade.config_validation import (
_COMMON_RULE_FIELDS,
+ _EXPERIMENT_ONLY_FIELDS,
+ _HOLDOUT_FIELDS,
_PROPERTY_FIELDS,
_ROLLOUT_RULE_FIELDS,
_ROOT_FIELDS,
_UUID as UUID_PATTERN,
+ _VARIANT_FIELDS,
ASSIGNMENT_ALGORITHM,
MAX_OBJECT_DEPTH,
MAX_PREDICATES_PER_RULE,
MAX_RULES,
MAX_SAFE_INTEGER,
MAX_SEED_LENGTH,
+ MAX_VARIANTS,
+ MIN_VARIANTS,
PERSON_ASSIGNMENT,
PROPERTY_OPERATORS,
PROPERTY_TYPES,
@@ -36,20 +41,24 @@ from products.feature_flags.backend.facade.config_validation import (
ConfigValidationError,
Predicate,
ValidatedConfig,
+ ValidatedHoldout,
ValidatedRule,
+ ValidatedVariant,
ValidationLimits,
_encoded_size as config_size_bytes,
canonical_value,
validate_config,
)
-# Vendored from the released harness contract; SOURCE.json records the revision and digests.
+# Vendored from the harness contract; SOURCE.json records the revision and digests.
CONTRACT_DIR = Path(__file__).parent / "fixtures" / "rules_v2_contract"
LIMITS = ValidationLimits(max_config_bytes=64 * 1024, max_metadata_bytes=1024)
TARGETED_ID = "11111111-1111-4111-8111-111111111111"
ROLLOUT_ID = "22222222-2222-4222-8222-222222222222"
+EXPERIMENT_ID = "33333333-3333-4333-8333-333333333333"
SEED = "release-preview"
+HOLDOUT_SEED = "release-preview-holdout"
def targeted(**overrides: Any) -> dict[str, Any]:
@@ -76,6 +85,30 @@ def rollout(**overrides: Any) -> dict[str, Any]:
}
+def variants(*values: Any, weights: tuple[float, ...] = (50, 50)) -> list[dict[str, Any]]:
+ keys = ("control", "test", "third")
+ return [{"key": key, "weight": weight, "value": value} for key, weight, value in zip(keys, weights, values)]
+
+
+def experiment(**overrides: Any) -> dict[str, Any]:
+ return {
+ "id": EXPERIMENT_ID,
+ "rule_type": "experiment",
+ "targeting": {"properties": []},
+ "paused": False,
+ "rollout_percentage": 100,
+ "on_rollout_miss": "continue",
+ "assignment_algorithm": ASSIGNMENT_ALGORITHM,
+ "seed": SEED,
+ "variants": variants(False, True),
+ **overrides,
+ }
+
+
+def holdout(**overrides: Any) -> dict[str, Any]:
+ return {"seed": HOLDOUT_SEED, "exclusion_percentage": 10, **overrides}
+
+
def person(**overrides: Any) -> dict[str, Any]:
return {"key": "account_tier", "type": "person", "operator": "exact", "value": "preview", **overrides}
@@ -200,6 +233,41 @@ VALID_DOCUMENTS: list[tuple[str, dict[str, Any]]] = [
),
("object_max_depth", config(targeted(value=nested(MAX_OBJECT_DEPTH)), return_type="object", default_value=None)),
("max_rules", config(*[targeted(id=f"00000000-0000-4000-8000-{index:012d}") for index in range(MAX_RULES)])),
+ ("experiment_split", config(experiment())),
+ ("experiment_paused_with_holdout", config(experiment(paused=True, holdout=holdout()))),
+ (
+ "experiment_partial_rollout_return_default",
+ config(experiment(rollout_percentage=33.33, on_rollout_miss="return_default", assign_by=PERSON_ASSIGNMENT)),
+ ),
+ ("experiment_equal_values", config(experiment(variants=variants(True, True)))),
+ (
+ "experiment_two_decimal_weights",
+ config(experiment(variants=variants(False, True, True, weights=(33.33, 33.33, 33.34)))),
+ ),
+ ("experiment_zero_weight", config(experiment(variants=variants(False, True, weights=(0, 100))))),
+ (
+ "experiment_max_variants",
+ config(
+ experiment(variants=[{"key": f"arm_{i}", "weight": 5, "value": i % 2 == 0} for i in range(MAX_VARIANTS)])
+ ),
+ ),
+ ("experiment_holdout_bounds", config(experiment(holdout=holdout(exclusion_percentage=0.01)))),
+ (
+ "experiment_string_values",
+ config(experiment(variants=variants("compact", "wide")), return_type="string", default_value=None),
+ ),
+ (
+ "experiment_number_values",
+ config(experiment(variants=variants(0, 12.5)), return_type="number", default_value=-1),
+ ),
+ (
+ "experiment_object_values",
+ config(experiment(variants=variants({}, {"columns": 2})), return_type="object", default_value=None),
+ ),
+ (
+ "experiment_beside_other_rules",
+ config(targeted(targeting={"properties": [person()]}), rollout(), experiment(holdout=holdout())),
+ ),
(
"max_predicates",
config(targeted(targeting={"properties": [person(key=f"k{i}") for i in range(MAX_PREDICATES_PER_RULE)]})),
@@ -228,6 +296,11 @@ INVALID_DOCUMENTS: list[tuple[str, object, list[tuple[str, str]]]] = [
("missing_return_type", without(config(), "return_type"), [("required", "filters.return_type")]),
("return_type_unknown", config(return_type="json"), [("invalid", "filters.return_type")]),
("return_type_null", config(return_type=None), [("invalid", "filters.return_type")]),
+ (
+ "experiment_with_unknown_return_type",
+ config(experiment(), return_type="json"),
+ [("invalid", "filters.return_type")],
+ ),
("string_empty_default", config(return_type="string", default_value=""), [("invalid", "filters.default_value")]),
("string_bool_default", config(return_type="string", default_value=True), [("invalid", "filters.default_value")]),
(
@@ -331,7 +404,128 @@ INVALID_DOCUMENTS: list[tuple[str, object, list[tuple[str, str]]]] = [
("removed_rule_type", config(targeted(rule_type="variant_rollout")), [("invalid", "filters.rules[0].rule_type")]),
("rule_type_null", config(targeted(rule_type=None)), [("invalid", "filters.rules[0].rule_type")]),
("rule_type_list", config(targeted(rule_type=["targeted_release"])), [("invalid", "filters.rules[0].rule_type")]),
- ("experiment_rule", config(targeted(rule_type="experiment")), [("unsupported", "filters.rules[0].rule_type")]),
+ (
+ "experiment_rule_with_targeted_fields",
+ config(targeted(rule_type="experiment")),
+ [
+ ("unknown_field", "filters.rules[0].value"),
+ ("required", "filters.rules[0].rollout_percentage"),
+ ("required", "filters.rules[0].on_rollout_miss"),
+ ("required", "filters.rules[0].assignment_algorithm"),
+ ("required", "filters.rules[0].seed"),
+ ("required", "filters.rules[0].paused"),
+ ("required", "filters.rules[0].variants"),
+ ],
+ ),
+ ("experiment_id", config(experiment(experiment_id=42)), [("unknown_field", "filters.rules[0].experiment_id")]),
+ (
+ "null_experiment_id",
+ config(experiment(experiment_id=None)),
+ [("unknown_field", "filters.rules[0].experiment_id")],
+ ),
+ ("missing_paused", config(without(experiment(), "paused")), [("required", "filters.rules[0].paused")]),
+ ("paused_null", config(experiment(paused=None)), [("invalid", "filters.rules[0].paused")]),
+ (
+ "experiment_missing_seed",
+ config(without(experiment(), "seed")),
+ [("required", "filters.rules[0].seed")],
+ ),
+ ("experiment_group_assign_by", config(experiment(assign_by="group")), [("invalid", "filters.rules[0].assign_by")]),
+ ("missing_variants", config(without(experiment(), "variants")), [("required", "filters.rules[0].variants")]),
+ ("variants_object", config(experiment(variants={})), [("invalid", "filters.rules[0].variants")]),
+ (
+ "one_variant",
+ config(experiment(variants=variants(True, weights=(100,)))),
+ [("invalid", "filters.rules[0].variants")],
+ ),
+ (
+ "too_many_variants",
+ config(experiment(variants=[{"key": f"arm_{i}", "weight": 5, "value": True} for i in range(MAX_VARIANTS + 1)])),
+ [("limit_exceeded", "filters.rules[0].variants")],
+ ),
+ (
+ "variant_not_object",
+ config(experiment(variants=[1, 2])),
+ [("invalid", "filters.rules[0].variants[0]"), ("invalid", "filters.rules[0].variants[1]")],
+ ),
+ (
+ "duplicate_variant_keys",
+ config(experiment(variants=[{**variant, "key": "control"} for variant in variants(False, True)])),
+ [("not_unique", "filters.rules[0].variants[1].key")],
+ ),
+ (
+ "variant_key_characters",
+ config(experiment(variants=[variants(False, True)[0], {**variants(False, True)[1], "key": "test.v2"}])),
+ [("invalid", "filters.rules[0].variants[1].key")],
+ ),
+ (
+ "variant_unknown_field",
+ config(experiment(variants=[variants(False, True)[0], {**variants(False, True)[1], "name": "Test"}])),
+ [("unknown_field", "filters.rules[0].variants[1].name")],
+ ),
+ (
+ "variant_value_type",
+ config(experiment(variants=variants(False, "true"))),
+ [("invalid", "filters.rules[0].variants[1].value")],
+ ),
+ (
+ "variant_null_value",
+ config(experiment(variants=variants(False, None))),
+ [("invalid", "filters.rules[0].variants[1].value")],
+ ),
+ (
+ "variant_missing_weight",
+ config(experiment(variants=[variants(False, True)[0], without(variants(False, True)[1], "weight")])),
+ [("required", "filters.rules[0].variants[1].weight")],
+ ),
+ (
+ "variant_weight_three_decimals",
+ config(experiment(variants=variants(False, True, weights=(33.333, 66.667)))),
+ [("invalid", "filters.rules[0].variants[0].weight"), ("invalid", "filters.rules[0].variants[1].weight")],
+ ),
+ (
+ "variant_weights_below_100",
+ config(experiment(variants=variants(False, True, True, weights=(33.33, 33.33, 33.33)))),
+ [("invalid", "filters.rules[0].variants")],
+ ),
+ (
+ "variant_weights_above_100",
+ config(experiment(variants=variants(False, True, weights=(50.01, 50)))),
+ [("invalid", "filters.rules[0].variants")],
+ ),
+ ("experiment_with_value", config(experiment(value=True)), [("unknown_field", "filters.rules[0].value")]),
+ ("holdout_id", config(experiment(holdout=holdout(id=7))), [("unknown_field", "filters.rules[0].holdout.id")]),
+ (
+ "null_holdout_id",
+ config(experiment(holdout=holdout(id=None))),
+ [("unknown_field", "filters.rules[0].holdout.id")],
+ ),
+ ("holdout_list", config(experiment(holdout=[])), [("invalid", "filters.rules[0].holdout")]),
+ (
+ "holdout_missing_seed",
+ config(experiment(holdout=without(holdout(), "seed"))),
+ [("required", "filters.rules[0].holdout.seed")],
+ ),
+ (
+ "holdout_unknown_field",
+ config(experiment(holdout=holdout(name="x"))),
+ [("unknown_field", "filters.rules[0].holdout.name")],
+ ),
+ (
+ "holdout_three_decimals",
+ config(experiment(holdout=holdout(exclusion_percentage=5.555))),
+ [("invalid", "filters.rules[0].holdout.exclusion_percentage")],
+ ),
+ (
+ "holdout_above_maximum",
+ config(experiment(holdout=holdout(exclusion_percentage=100.5))),
+ [("invalid", "filters.rules[0].holdout.exclusion_percentage")],
+ ),
+ (
+ "rollout_with_experiment_fields",
+ config(rollout(paused=False, variants=[])),
+ [("unknown_field", "filters.rules[0].paused"), ("unknown_field", "filters.rules[0].variants")],
+ ),
(
"unknown_rule_field",
config(targeted(implicit_default=False)),
@@ -872,6 +1066,35 @@ class TestValidateConfig:
assert validated.rules[1].rollout_percentage == Decimal("33.33")
assert SEED not in repr(validated)
+ def test_validated_experiment_rule_carries_its_split_without_seeds(self) -> None:
+ document = config(
+ experiment(
+ paused=True,
+ rollout_percentage=80.5,
+ variants=variants("compact", "compact", weights=(33.33, 66.67)),
+ holdout=holdout(exclusion_percentage=12.5),
+ ),
+ return_type="string",
+ default_value=None,
+ )
+ (rule,) = validate_config(document, limits=LIMITS).rules
+ assert rule == ValidatedRule(
+ id=EXPERIMENT_ID,
+ rule_type="experiment",
+ predicates=frozenset(),
+ value=None,
+ rollout_percentage=Decimal("80.5"),
+ on_rollout_miss="continue",
+ seed=SEED,
+ paused=True,
+ variants=(
+ ValidatedVariant(key="control", weight=Decimal("33.33"), value='"compact"'),
+ ValidatedVariant(key="test", weight=Decimal("66.67"), value='"compact"'),
+ ),
+ holdout=ValidatedHoldout(exclusion_percentage=Decimal("12.5"), seed=HOLDOUT_SEED),
+ )
+ assert SEED not in repr(rule) and HOLDOUT_SEED not in repr(rule)
+
@parameterized.expand([("explicit", {"negation": True}), ("omitted", {})])
def test_predicates_are_canonical_across_equivalent_spellings(self, _name: str, extra: dict[str, Any]) -> None:
plain = validate_config(config(targeted(targeting={"properties": [person(**extra)]})), limits=LIMITS)
@@ -884,8 +1107,8 @@ class TestValidateConfig:
def _contract_path(instance_path: str, expected: dict[str, Any]) -> str:
attr = "filters" + re.sub(r"/(\d+)", r"[\1]", instance_path).replace("/", ".")
- # Upstream writes the bare field name in `message_contains` for these two keywords, so it is
- # the missing or unknown field's path segment. Other keywords carry prose (`2 was expected`).
+ # Upstream writes the field name in `message_contains` for these two keywords,
+ # so it is the missing or unknown field's path segment. Other keywords carry prose (`2 was expected`).
if expected["keyword"] in ("required", "additionalProperties"):
attr += "." + expected["message_contains"]
return attr
@@ -903,11 +1126,7 @@ def _admitted_family(document: dict[str, Any]) -> bool:
for prop in rule["targeting"].get("properties", [])
if isinstance(prop, dict)
]
- return (
- "aggregation_group_type_index" not in document
- and all(rule.get("rule_type") != "experiment" for rule in rules)
- and all(prop.get("type") == "person" for prop in properties)
- )
+ return "aggregation_group_type_index" not in document and all(prop.get("type") == "person" for prop in properties)
class TestReleasedContract:
@@ -945,6 +1164,13 @@ class TestReleasedContract:
assert _ROOT_FIELDS == set(CONFIG_SCHEMA["properties"])
assert _COMMON_RULE_FIELDS == set(defs["targetedReleaseRule"]["properties"])
assert _COMMON_RULE_FIELDS | _ROLLOUT_RULE_FIELDS == set(defs["percentageRolloutRule"]["properties"])
+ assert (_COMMON_RULE_FIELDS - {"value"}) | _ROLLOUT_RULE_FIELDS | _EXPERIMENT_ONLY_FIELDS == set(
+ defs["experimentRule"]["properties"]
+ )
+ assert _VARIANT_FIELDS == set(defs["variant"]["properties"])
+ assert _HOLDOUT_FIELDS == set(defs["holdout"]["properties"])
+ variant_list = defs["experimentRule"]["properties"]["variants"]
+ assert (MIN_VARIANTS, MAX_VARIANTS) == (variant_list["minItems"], variant_list["maxItems"])
assert _PROPERTY_FIELDS == set(defs["propertyFilter"]["properties"])
def test_operator_tables_match_the_v1_cross_field_tier(self) -> None:
diff --git a/products/feature_flags/backend/test/test_facade_rule_warnings.py b/products/feature_flags/backend/test/test_facade_rule_warnings.py
index e6844832cbbb..cedc8804e4c4 100644
--- a/products/feature_flags/backend/test/test_facade_rule_warnings.py
+++ b/products/feature_flags/backend/test/test_facade_rule_warnings.py
@@ -13,7 +13,9 @@ from products.feature_flags.backend.facade.config_validation import (
Predicate,
RolloutMissPolicy,
ValidatedConfig,
+ ValidatedHoldout,
ValidatedRule,
+ ValidatedVariant,
canonical_value,
)
from products.feature_flags.backend.facade.rule_warnings import (
@@ -70,6 +72,37 @@ def rollout(
)
+def split(
+ rule_id: str,
+ percentage: int | float | str = 100,
+ *predicates: Predicate,
+ values: tuple[Any, ...] = ("blue", "green"),
+ seed: str = SEED,
+ miss: RolloutMissPolicy = "continue",
+ paused: bool = False,
+ holdout: int | None = None,
+ weights: tuple[int, ...] | None = None,
+) -> ValidatedRule:
+ shares: tuple[Decimal | int, ...] = weights or (Decimal(100) / len(values),) * len(values)
+ return ValidatedRule(
+ id=rule_id,
+ rule_type="experiment",
+ predicates=frozenset(predicates),
+ value=None,
+ rollout_percentage=Decimal(str(percentage)),
+ on_rollout_miss=miss,
+ seed=seed,
+ paused=paused,
+ variants=tuple(
+ ValidatedVariant(key=f"arm_{index}", weight=Decimal(weight), value=canonical_value(value))
+ for index, (value, weight) in enumerate(zip(values, shares))
+ ),
+ holdout=None
+ if holdout is None
+ else ValidatedHoldout(exclusion_percentage=Decimal(holdout), seed=f"{seed}-holdout"),
+ )
+
+
def cfg(*rules: ValidatedRule, default: Any = False) -> ValidatedConfig:
return ValidatedConfig(default_value=None if default is None else canonical_value(default), rules=rules)
@@ -283,6 +316,77 @@ class TestConfigWarnings:
assert config_warnings(config) == ()
+class TestExperimentRuleWarnings:
+ @parameterized.expand(
+ [
+ ("full_split_blocks_lower", cfg(split(A), targeted(B, "red")), [(UNREACHABLE, "filters.rules[1]")]),
+ (
+ "paused_split_is_terminal",
+ cfg(split(A, 30, paused=True), targeted(B, "red")),
+ [(UNREACHABLE, "filters.rules[1]")],
+ ),
+ (
+ "holdout_is_terminal",
+ cfg(split(A, holdout=10), targeted(B, "red")),
+ [(UNREACHABLE, "filters.rules[1]")],
+ ),
+ (
+ "return_default_miss_is_terminal",
+ cfg(split(A, 30, miss="return_default"), targeted(B, "red")),
+ [(UNREACHABLE, "filters.rules[1]")],
+ ),
+ ("partial_split_continues_to_a_different_value", cfg(split(A, 30), targeted(B, "red")), []),
+ (
+ "partial_split_continues_to_a_variant_value",
+ cfg(split(A, 30), targeted(B, "green")),
+ [(EXTENDS, "filters.rules[1]")],
+ ),
+ ("paused_split_has_no_rollout_to_extend", cfg(split(A, 30, PRO, paused=True), targeted(B, "green")), []),
+ (
+ "zero_weight_variant_serves_nothing",
+ cfg(split(A, 30, weights=(100, 0)), targeted(B, "green")),
+ [],
+ ),
+ (
+ "partial_rollout_continues_to_a_split_serving_its_value",
+ cfg(rollout(A, 30, "green", seed=OTHER_SEED), split(B)),
+ [(EXTENDS, "filters.rules[1]")],
+ ),
+ ("narrower_split_is_inconclusive", cfg(split(A, 100, PRO), targeted(B, "red")), []),
+ (
+ # The rollout above takes the same hash share, so the split enrolls nobody new.
+ "same_seed_rollout_above_leaves_the_split_nobody",
+ cfg(rollout(A, 40, "red"), split(B, 40, values=("red", "blue")), targeted(C, "red")),
+ [(EXTENDS, "filters.rules[2]")],
+ ),
+ ]
+ )
+ def test_experiment_rules_take_part_in_the_walk(
+ self, _name: str, config: ValidatedConfig, expected: list[tuple[str, str]]
+ ) -> None:
+ assert codes(config_warnings(config)) == expected
+
+ def test_variant_values_are_compared_like_rule_values(self) -> None:
+ current = cfg(split(A, values=("red", "red")), targeted(B, "green"))
+ proposed = cfg(targeted(B, "green"), split(A, values=("red", "red")))
+ assert codes(reorder_warnings(current, proposed)) == [(REORDER, "filters.rules[0]")]
+ # Equal-valued arms serve one value, so swapping them with an equal targeted rule changes nothing.
+ current = cfg(split(A, values=("red", "red")), targeted(B, "red"))
+ proposed = cfg(targeted(B, "red"), split(A, values=("red", "red")))
+ assert reorder_warnings(current, proposed) == ()
+
+ @parameterized.expand(
+ [
+ # Only the held-out share gets the default, so only it changes value.
+ ("holdout_serves_the_default", split(A, values=("red", "red"), holdout=10), targeted(B, "red")),
+ # Only the second arm differs from the targeted rule, and the split enrolls half.
+ ("later_arm_of_a_partial_split", split(A, 50, values=("green", "red")), targeted(B, "green")),
+ ]
+ )
+ def test_swapping_reaches_part_of_a_split(self, _name: str, upper: ValidatedRule, lower: ValidatedRule) -> None:
+ assert codes(reorder_warnings(cfg(upper, lower), cfg(lower, upper))) == [(REORDER, "filters.rules[0]")]
+
+
class TestReorderWarnings:
@parameterized.expand(
[
diff --git a/products/feature_flags/frontend/FeatureFlagRulesV2Readonly.test.tsx b/products/feature_flags/frontend/FeatureFlagRulesV2Readonly.test.tsx
index dcb6d36adf82..dbab4e4a6281 100644
--- a/products/feature_flags/frontend/FeatureFlagRulesV2Readonly.test.tsx
+++ b/products/feature_flags/frontend/FeatureFlagRulesV2Readonly.test.tsx
@@ -36,6 +36,38 @@ describe('FeatureFlagRulesV2Readonly', () => {
cleanup()
})
+ it('shows a holdout by its exclusion percentage but never its seed', () => {
+ const config: FeatureFlagRulesV2Config = {
+ ...CONFIG,
+ rules: [
+ {
+ id: 'rule-3',
+ rule_type: 'experiment',
+ targeting: { properties: [] },
+ value: true,
+ rollout_percentage: 100,
+ on_rollout_miss: 'continue',
+ assignment_algorithm: 'sha1_60_v1',
+ seed: 'seed-must-not-render',
+ paused: false,
+ variants: [
+ { key: 'control', weight: 50, value: false },
+ { key: 'test', weight: 50, value: true },
+ ],
+ holdout: { seed: 'holdout-seed-must-not-render', exclusion_percentage: 10 },
+ },
+ ],
+ }
+ const { container } = render(
+ <Provider>
+ <FeatureFlagRulesV2Readonly config={config} />
+ </Provider>
+ )
+
+ expect(container).toHaveTextContent('Holdout excludes 10%')
+ expect(container).not.toHaveTextContent('seed-must-not-render')
+ })
+
it('shows the rule but never its seed', () => {
const { container } = render(
<Provider>
diff --git a/products/feature_flags/frontend/FeatureFlagRulesV2Readonly.tsx b/products/feature_flags/frontend/FeatureFlagRulesV2Readonly.tsx
index c6329ea6dc5a..2c451ab84a4c 100644
--- a/products/feature_flags/frontend/FeatureFlagRulesV2Readonly.tsx
+++ b/products/feature_flags/frontend/FeatureFlagRulesV2Readonly.tsx
@@ -56,7 +56,7 @@ function ValueCell({ rule }: { rule: FeatureFlagRulesV2Rule }): JSX.Element {
))}
{rule.holdout && (
<span className="text-xs text-muted">
- Holdout #{rule.holdout.id} excludes {formatPercentage(rule.holdout.exclusion_percentage)}
+ Holdout excludes {formatPercentage(rule.holdout.exclusion_percentage)}
</span>
)}
</div>
diff --git a/rust/feature-flags/tests/fixtures/flags_cache_config_formats.json b/rust/feature-flags/tests/fixtures/flags_cache_config_formats.json
index f303dd67dc94..1aa268b489e9 100644
--- a/rust/feature-flags/tests/fixtures/flags_cache_config_formats.json
+++ b/rust/feature-flags/tests/fixtures/flags_cache_config_formats.json
@@ -221,21 +221,44 @@
"version": 2,
"return_type": "boolean",
"default_value": false,
+ "aggregation_group_type_index": 0,
"rules": [
{
"id": "e5e5e5e5-e5e5-4e5e-8e5e-e5e5e5e5e5e5",
+ "rule_type": "targeted_release",
+ "targeting": { "properties": [] },
+ "value": true
+ }
+ ]
+ },
+ "expect": "unsupported"
+ },
+ {
+ "key": "v2-experiment-active",
+ "active": true,
+ "filters": {
+ "version": 2,
+ "return_type": "string",
+ "default_value": "standard",
+ "rules": [
+ {
+ "id": "f6f6f6f6-f6f6-4f6f-8f6f-f6f6f6f6f6f6",
"rule_type": "experiment",
"targeting": { "properties": [] },
- "experiment_id": 1,
- "rollout_percentage": 100,
+ "paused": false,
+ "rollout_percentage": 50,
"on_rollout_miss": "continue",
"assignment_algorithm": "sha1_60_v1",
- "seed": "release-experiment",
- "variants": [{ "key": "control", "value": false, "rollout_percentage": 100 }]
+ "seed": "layout-split",
+ "variants": [
+ { "key": "control", "weight": 50, "value": "standard" },
+ { "key": "test", "weight": 50, "value": "compact" }
+ ],
+ "holdout": { "seed": "layout-split-holdout", "exclusion_percentage": 10 }
}
]
},
- "expect": "unsupported"
+ "expect": "kept"
},
{
"key": "v2-valid-inactive",
diff --git a/rust/feature-flags/tests/fixtures/rules_v2_contract/SOURCE.json b/rust/feature-flags/tests/fixtures/rules_v2_contract/SOURCE.json
index 816520f7d019..617b5a55a591 100644
--- a/rust/feature-flags/tests/fixtures/rules_v2_contract/SOURCE.json
+++ b/rust/feature-flags/tests/fixtures/rules_v2_contract/SOURCE.json
@@ -1,9 +1,9 @@
{
"repository": "https://github.com/PostHog/posthog-sdk-test-harness",
- "release": null,
- "commit": "6c4182b71d8021fcb10919a6c20825413cbef3dc",
+ "release": "1.20.0",
+ "commit": "aa725c89eec72cb8ac75f4e418662bf2e161feb8",
"path": "contracts/feature_flag_rules_v2",
- "contract_version": "3.0.0",
+ "contract_version": "6.0.0",
"files": {
"corpus/hash_sha1_60_v1.json": "e9bea9cff58bac0c6c1021e9c8842c53594055b6874a2c58bc4350d0c28a93ca",
"corpus/legacy_projection.json": "536c8386eefa75721d113bd8718ddeb7fb1aad3b4f2cce1fbcf516a346a16e26",
@@ -55,7 +55,7 @@
"fixtures/config/valid/version_float_literal.json": "a848ae2e3e0dd2119600bb90ebbcba59040b3bb788586ba127a57eab1848a403",
"fixtures/wire/definitions.json": "894f37aaf7e6b360ca36e0b5beb5645abd8782dc9a09eb1f76db64285d02fe6d",
"fixtures/wire/responses.json": "5995db3a23667a58aaefb050779c7b0fe3ff154e821729408d4e6c138a7535b1",
- "manifest.json": "6a9c0beffc378afafc1fe67f51297ff51252d2f293d59a1c835cbbaa91d80354",
+ "manifest.json": "905674e9eff024a4ea58b8ff12c3c604495c84cb31860457d59e6d56e6748aa0",
"registries/literals.json": "3e87e6c7c760f0ebac80289faa5412a6ecd17749cb5c79dabe5ed1efaea22578",
"schemas/config.schema.json": "163507e0db1219329caef034a50516d27eb5759b5e4c250f313ceb53ca3f1e08",
"schemas/definitions_entry.schema.json": "6d920c77fb7f20c2031dc188d495ed28c3789b3f362334517a2e86021a226e85",
diff --git a/rust/feature-flags/tests/fixtures/rules_v2_contract/manifest.json b/rust/feature-flags/tests/fixtures/rules_v2_contract/manifest.json
index 7799846a7321..356bbad43d81 100644
--- a/rust/feature-flags/tests/fixtures/rules_v2_contract/manifest.json
+++ b/rust/feature-flags/tests/fixtures/rules_v2_contract/manifest.json
@@ -1,7 +1,7 @@
{
"contract": {
"name": "feature_flag_rules_v2",
- "version": "3.0.0",
+ "version": "6.0.0",
"config_version": 2
},
"artifacts": [
@@ -409,7 +409,7 @@
"expected_failure": {
"keyword": "additionalProperties",
"instance_path": "/rules/0/holdout",
- "message_contains": "'id'"
+ "message_contains": "id"
}
},
{