:hedgehog: PostHog is the leading platform for building self-driving products. Our developer tools – AI observability, analytics, session replay, flags, experiments, error tracking, logs, and more – capture all the context agents need to diagnose problems, uncover opportunities, and ship fixes. Steer it all from Slack, web, desktop, or the MCP.
Python ab-testingai-analyticsanalyticscdpdata-warehouseexperimentsfeature-flagsjavascriptproduct-analyticspythonreactsession-replaysurveystypescriptweb-analytics
chore(devex): give new repo-invariant ratchets a grace period (#118588)
Georges-Antoine Assi7188f8e3ff26
diff --git a/.agents/skills/authoring-ci-workflows/SKILL.md b/.agents/skills/authoring-ci-workflows/SKILL.mdindex 38a5f119c21a..5846eba7bdf5 100644--- a/.agents/skills/authoring-ci-workflows/SKILL.md+++ b/.agents/skills/authoring-ci-workflows/SKILL.md@@ -519,6 +519,8 @@ A workflow edit hits every open PR the instant it merges (it runs against PR-merIf the workflow starts _requiring_ something unrebased branches lack, every in-flight PR fails before its tests run.Make new behavior degrade gracefully when the prerequisite is absent, or gate it.Roll out a new blocking lint the same way: ship `continue-on-error`, clear the inbox, promote to blocking.+A new or tightened baseline ratchet in `posthog/test/repo_invariants/` meets queued PRs that passed CI before it existed, so report its mismatch through `enforce_baseline` in `ratchet.py` with a `blocking_from` date a few days out: new violations only warn until then.+Before that date, fix or add to the baseline every violation that landed during the grace period, or master goes red on it.## New-workflow checklistdiff --git a/.agents/skills/implementing-warehouse-sources/SKILL.md b/.agents/skills/implementing-warehouse-sources/SKILL.mdindex 0849c311f2e7..1aa2083908c6 100644--- a/.agents/skills/implementing-warehouse-sources/SKILL.md+++ b/.agents/skills/implementing-warehouse-sources/SKILL.md@@ -468,8 +468,8 @@ When a fix makes an entry wrong, regenerate the baseline and commit it:```bashSOURCE_CONTRACT_WRITE_BASELINE=1 pytest products/warehouse_sources/backend/temporal/data_imports/sources/tests/test_source_contract.py -p no:xdist-python posthog/test/repo_invariants/test_warehouse_source_static_contract.py-python posthog/test/repo_invariants/test_resume_state_staged_before_yield.py+PYTHONPATH=. python posthog/test/repo_invariants/test_warehouse_source_static_contract.py+PYTHONPATH=. python posthog/test/repo_invariants/test_resume_state_staged_before_yield.py```#### Row order and `sort_mode`diff --git a/.agents/skills/implementing-warehouse-sources/references/testing.md b/.agents/skills/implementing-warehouse-sources/references/testing.mdindex f049c0fddd2d..7c019eec2e9b 100644--- a/.agents/skills/implementing-warehouse-sources/references/testing.md+++ b/.agents/skills/implementing-warehouse-sources/references/testing.md@@ -114,7 +114,7 @@ For a source built on a vendor SDK, mock the SDK client instead.Do not patch `make_tracked_session`, mock `ResumableSourceManager`, or patch a private name in `sources/common`.Each one lets a test pass against wrong behavior, and `posthog/test/repo_invariants/test_warehouse_source_test_shapes.py` fails a new test file that does it.Its baselines list the files that still do, and may only get shorter.-When you move a listed file onto the driver, run `python posthog/test/repo_invariants/test_warehouse_source_test_shapes.py` to remove it from the baseline.+When you move a listed file onto the driver, run `PYTHONPATH=. python posthog/test/repo_invariants/test_warehouse_source_test_shapes.py` to remove it from the baseline.- **Pagination:** one test that walks at least two pages to the terminal page, and asserts the request parameters of each page. Add the cases where the vendor's termination signal is unusual (an empty page with a cursor, a `has_more` flag that lies).- **Request shaping:** an incremental request sends the watermark in the vendor's filter and sort; a full refresh sends no watermark. One parameterized test.diff --git a/.agents/skills/writing-tests/references/database-free-test-classes.md b/.agents/skills/writing-tests/references/database-free-test-classes.mdindex 6f6cd379b335..407bd0eed83e 100644--- a/.agents/skills/writing-tests/references/database-free-test-classes.md+++ b/.agents/skills/writing-tests/references/database-free-test-classes.md@@ -53,7 +53,7 @@ That takes the pure cases off the database without weakening the rest, and it isOnly to ratchet down, or after a rename or a move:```sh-python posthog/test/repo_invariants/test_database_free_test_classes.py+PYTHONPATH=. python posthog/test/repo_invariants/test_database_free_test_classes.py```The file records removals as well as additions, so a class you fixed has to leave the list in the same change.diff --git a/posthog/test/repo_invariants/ratchet.py b/posthog/test/repo_invariants/ratchet.pynew file mode 100644index 000000000000..aad48cd02a91--- /dev/null+++ b/posthog/test/repo_invariants/ratchet.py@@ -0,0 +1,27 @@+import os+import warnings+from collections.abc import Sequence+from datetime import date, timedelta+from pathlib import Path++MAX_GRACE = timedelta(days=14)+++class RatchetGraceWarning(UserWarning):+ pass+++def enforce_baseline(*, added: Sequence[str], removed: Sequence[str], blocking_from: date, message: str) -> None:+ if not added and not removed:+ return+ today = date.today()+ if blocking_from - today > MAX_GRACE:+ raise ValueError(f"blocking_from {blocking_from.isoformat()} is more than {MAX_GRACE.days} days away")+ if removed or today >= blocking_from:+ raise AssertionError(message)+ notice = f"Not blocking until {blocking_from.isoformat()}.\n{message}"+ summary_path = os.environ.get("GITHUB_STEP_SUMMARY")+ if summary_path:+ with Path(summary_path).open("a") as summary:+ summary.write(f"### Ratchet grace period\n\n```text\n{notice}\n```\n\n")+ warnings.warn(notice, RatchetGraceWarning, stacklevel=2)diff --git a/posthog/test/repo_invariants/test_database_free_test_classes.py b/posthog/test/repo_invariants/test_database_free_test_classes.pyindex b3d1ed60fcc3..b12803e88a22 100644--- a/posthog/test/repo_invariants/test_database_free_test_classes.py+++ b/posthog/test/repo_invariants/test_database_free_test_classes.py@@ -8,15 +8,19 @@ carries the long form.import reimport astimport warnings+from datetime import datefrom pathlib import Pathfrom parameterized import parameterized+from posthog.test.repo_invariants.ratchet import enforce_baseline+REPO_ROOT = Path(__file__).parents[3]BASELINE_PATH = Path(__file__).parent / "database_free_test_classes_baseline.txt"+BLOCKING_FROM = date.fromisoformat("2026-09-09")SCANNED_ROOTS = ("posthog", "ee", "products", "common")SKIPPED_DIRS = {"node_modules", ".venv", "venv", "__pycache__", ".git", ".mypy_cache"}-REGENERATE = "python posthog/test/repo_invariants/test_database_free_test_classes.py"+REGENERATE = "PYTHONPATH=. python posthog/test/repo_invariants/test_database_free_test_classes.py"# Must reach a signature split over several lines and a nested class: a base it does# not see is one that gets reported as a candidate.@@ -291,16 +295,21 @@ def test_database_free_test_classes_match_the_baseline() -> None:added = [line for line in scanned if line not in recorded]removed = [line for line in recorded if line not in scanned]report = "\n".join([*(f" + {line}" for line in added), *(f" - {line}" for line in removed)])- raise AssertionError(- f"{BASELINE_PATH.name} no longer matches the repo.\n"- "A '+' line is a test class that inherits a database base class but never names "- "anything that reaches the database, so it needs a database it never reads. "- "Confirm it by swapping the base for django.test.SimpleTestCase and running "- "the class: SimpleTestCase refuses database access, so a passing run means the class "- "never needed one, and that base is the fix. If the class does need a database through "- "a helper this scan cannot see, add the baseline line back in the same change.\n"- f"A '-' line means a candidate went away — good, but the file must record that too. Run: {REGENERATE}\n"- f"{report}"+ enforce_baseline(+ added=added,+ removed=removed,+ blocking_from=BLOCKING_FROM,+ message=(+ f"{BASELINE_PATH.name} no longer matches the repo.\n"+ "A '+' line is a test class that inherits a database base class but never names "+ "anything that reaches the database, so it needs a database it never reads. "+ "Confirm it by swapping the base for django.test.SimpleTestCase and running "+ "the class: SimpleTestCase refuses database access, so a passing run means the class "+ "never needed one, and that base is the fix. If the class does need a database through "+ "a helper this scan cannot see, add the baseline line back in the same change.\n"+ f"A '-' line means a candidate went away — good, but the file must record that too. Run: {REGENERATE}\n"+ f"{report}"+ ),)diff --git a/posthog/test/repo_invariants/test_ratchet.py b/posthog/test/repo_invariants/test_ratchet.pynew file mode 100644index 000000000000..77c7764a18d3--- /dev/null+++ b/posthog/test/repo_invariants/test_ratchet.py@@ -0,0 +1,46 @@+import warnings+from datetime import date+from pathlib import Path++import pytest+import time_machine++from posthog.test.repo_invariants.ratchet import RatchetGraceWarning, enforce_baseline++BLOCKING_FROM = date(2026, 10, 20)+++ ("today", "added", "removed", "expected"),+ [+ ("2026-10-19", ["new"], [], "warns"),+ ("2026-10-20", ["new"], [], "fails"),+ ("2026-10-19", ["new"], ["fixed"], "fails"),+ ],+)+def test_grace_period_boundary(+ today: str,+ added: list[str],+ removed: list[str],+ expected: str,+ tmp_path: Path,+ monkeypatch: pytest.MonkeyPatch,+) -> None:+ step_summary = tmp_path / "step_summary.md"+ monkeypatch.setenv("GITHUB_STEP_SUMMARY", str(step_summary))+ with time_machine.travel(date.fromisoformat(today), tick=False), warnings.catch_warnings(record=True) as caught:+ warnings.simplefilter("always")+ if expected == "fails":+ with pytest.raises(AssertionError, match="baseline mismatch"):+ enforce_baseline(added=added, removed=removed, blocking_from=BLOCKING_FROM, message="baseline mismatch")+ else:+ enforce_baseline(added=added, removed=removed, blocking_from=BLOCKING_FROM, message="baseline mismatch")+ assert [w.category for w in caught] == ([RatchetGraceWarning] if expected == "warns" else [])+ summary_text = step_summary.read_text() if step_summary.exists() else ""+ assert ("baseline mismatch" in summary_text) is (expected == "warns")+++def test_a_grace_period_longer_than_the_cap_is_refused() -> None:+ with time_machine.travel(date(2026, 10, 1), tick=False):+ with pytest.raises(ValueError):+ enforce_baseline(added=["new"], removed=[], blocking_from=BLOCKING_FROM, message="baseline mismatch")diff --git a/posthog/test/repo_invariants/test_resume_state_staged_before_yield.py b/posthog/test/repo_invariants/test_resume_state_staged_before_yield.pyindex dcda15ab1a26..3199bb926c83 100644--- a/posthog/test/repo_invariants/test_resume_state_staged_before_yield.py+++ b/posthog/test/repo_invariants/test_resume_state_staged_before_yield.py@@ -8,15 +8,19 @@ fewer keeps no progress at all. The failure message says how to fix a reported fimport astimport warnings+from datetime import datefrom pathlib import Pathfrom parameterized import parameterized+from posthog.test.repo_invariants.ratchet import enforce_baseline+REPO_ROOT = Path(__file__).parents[3]BASELINE_PATH = Path(__file__).parent / "resume_state_after_yield_baseline.txt"+BLOCKING_FROM = date.fromisoformat("2026-10-06")SOURCES_ROOT = REPO_ROOT / "products/warehouse_sources/backend/temporal/data_imports/sources"SKIPPED_DIRS = {"tests", "test", "__pycache__"}-REGENERATE = "python posthog/test/repo_invariants/test_resume_state_staged_before_yield.py"+REGENERATE = "PYTHONPATH=. python posthog/test/repo_invariants/test_resume_state_staged_before_yield.py"FUNCTIONS = (ast.FunctionDef, ast.AsyncFunctionDef, ast.Lambda)@@ -323,16 +327,21 @@ def test_resume_state_saved_after_a_yield_matches_the_baseline() -> None:added = [line for line in scanned if line not in recorded]removed = [line for line in recorded if line not in scanned]report = "\n".join([*(f" + {line}" for line in added), *(f" - {line}" for line in removed)])- raise AssertionError(- f"{BASELINE_PATH.name} no longer matches the repo.\n"- "A '+' line is a source generator that calls `save_state` after the `yield` the state covers. "- "The pipeline commits staged state right after it writes a batch, and on a worker shutdown it "- "ends the attempt before control returns to the source. That state is lost for the last batch, "- "so the next attempt reads the batch again, and a full refresh appends it twice. "- "Call `save_state` before the `yield` of the batch it covers. "- "Do not add the line to the baseline: the list may only get shorter.\n"- f"A '-' line means a function was fixed. The file must record that too. Run: {REGENERATE}\n"- f"{report}"+ enforce_baseline(+ added=added,+ removed=removed,+ blocking_from=BLOCKING_FROM,+ message=(+ f"{BASELINE_PATH.name} no longer matches the repo.\n"+ "A '+' line is a source generator that calls `save_state` after the `yield` the state covers. "+ "The pipeline commits staged state right after it writes a batch, and on a worker shutdown it "+ "ends the attempt before control returns to the source. That state is lost for the last batch, "+ "so the next attempt reads the batch again, and a full refresh appends it twice. "+ "Call `save_state` before the `yield` of the batch it covers. "+ "Do not add the line to the baseline: the list may only get shorter.\n"+ f"A '-' line means a function was fixed. The file must record that too. Run: {REGENERATE}\n"+ f"{report}"+ ),)diff --git a/posthog/test/repo_invariants/test_signals_scout_tool_citations.py b/posthog/test/repo_invariants/test_signals_scout_tool_citations.pyindex 9d2926b037b8..7887cd822ec1 100644--- a/posthog/test/repo_invariants/test_signals_scout_tool_citations.py+++ b/posthog/test/repo_invariants/test_signals_scout_tool_citations.py@@ -29,14 +29,17 @@ Regenerate the baseline (after confirming a rename, or when a body cites a new t"""import re+from datetime import datefrom pathlib import Pathimport pytestfrom posthog.mcp_tool_definitions import get_mcp_tool_definitions+from posthog.test.repo_invariants.ratchet import enforce_baselineREPO_ROOT = Path(__file__).parents[3]BASELINE_PATH = Path(__file__).parent / "signals_scout_tool_citations_baseline.txt"+BLOCKING_FROM = date.fromisoformat("2026-09-19")TOOL_NAME_VERBS = frozenset({@@ -161,10 +164,15 @@ def test_baseline_tool_citations_still_resolve() -> None:"Repoint each citation at the live tool, then regenerate:\n DEBUG=1 DJANGO_SETTINGS_MODULE=posthog.settings PYTHONPATH=. python posthog/test/repo_invariants/test_signals_scout_tool_citations.py\n"+ "\n".join(f" {name}" for name in missing))- assert not added, (- "Canonical scouts cite tools the baseline does not pin. Confirm each name is a real "- "tool, then regenerate:\n DEBUG=1 DJANGO_SETTINGS_MODULE=posthog.settings PYTHONPATH=. python posthog/test/repo_invariants/test_signals_scout_tool_citations.py\n"- + "\n".join(f" {name}" for name in added)+ enforce_baseline(+ added=added,+ removed=[],+ blocking_from=BLOCKING_FROM,+ message=(+ "Canonical scouts cite tools the baseline does not pin. Confirm each name is a real "+ "tool, then regenerate:\n DEBUG=1 DJANGO_SETTINGS_MODULE=posthog.settings PYTHONPATH=. python posthog/test/repo_invariants/test_signals_scout_tool_citations.py\n"+ + "\n".join(f" {name}" for name in added)+ ),)diff --git a/posthog/test/repo_invariants/test_warehouse_source_static_contract.py b/posthog/test/repo_invariants/test_warehouse_source_static_contract.pyindex 7950e2046542..ba2a1caf87af 100644--- a/posthog/test/repo_invariants/test_warehouse_source_static_contract.py+++ b/posthog/test/repo_invariants/test_warehouse_source_static_contract.py@@ -18,15 +18,18 @@ import astimport sysimport warningsfrom collections.abc import Callable+from datetime import datefrom pathlib import Pathimport pytest+from posthog.test.repo_invariants.ratchet import enforce_baseline+REPO_ROOT = Path(__file__).parents[3]BASELINE_DIR = Path(__file__).parentSOURCES_ROOT = REPO_ROOT / "products/warehouse_sources/backend/temporal/data_imports/sources"SKIPPED_DIRS = {"common", "tests", "test", "__pycache__", "generated_configs"}-REGENERATE = "python posthog/test/repo_invariants/test_warehouse_source_static_contract.py"+REGENERATE = "PYTHONPATH=. python posthog/test/repo_invariants/test_warehouse_source_static_contract.py"FUNCTIONS = (ast.FunctionDef, ast.AsyncFunctionDef, ast.Lambda)HTTP_VERBS = {"get", "post", "put", "patch", "delete", "head", "request", "send"}@@ -209,6 +212,12 @@ CHECKS: dict[str, tuple[Check, tuple[str, ...], str, str]] = {),}+BLOCKING_FROM: dict[str, date] = {+ "raw_sleep": date.fromisoformat("2026-10-07"),+ "request_without_timeout": date.fromisoformat("2026-10-07"),+ "wrapped_resource": date.fromisoformat("2026-10-07"),+}+def collect(check: Check, words: tuple[str, ...]) -> list[str]:found: list[str] = []@@ -308,11 +317,16 @@ def test_source_functions_match_the_baseline(check: str) -> None:added = [line for line in scanned if line not in recorded]removed = [line for line in recorded if line not in scanned]report = "\n".join([*(f" + {line}" for line in added), *(f" - {line}" for line in removed)])- raise AssertionError(- f"{file_name} no longer matches the repo.\n"- f"{explanation} Do not add the line to the baseline: the list may only get shorter.\n"- f"A '-' line means a function was fixed. The file must record that too. Run: {REGENERATE}\n"- f"{report}"+ enforce_baseline(+ added=added,+ removed=removed,+ blocking_from=BLOCKING_FROM[check],+ message=(+ f"{file_name} no longer matches the repo.\n"+ f"{explanation} Do not add the line to the baseline: the list may only get shorter.\n"+ f"A '-' line means a function was fixed. The file must record that too. Run: {REGENERATE}\n"+ f"{report}"+ ),)diff --git a/posthog/test/repo_invariants/test_warehouse_source_test_shapes.py b/posthog/test/repo_invariants/test_warehouse_source_test_shapes.pyindex b795204b8a65..1a479474a4e3 100644--- a/posthog/test/repo_invariants/test_warehouse_source_test_shapes.py+++ b/posthog/test/repo_invariants/test_warehouse_source_test_shapes.py@@ -14,15 +14,18 @@ import astimport sysimport warningsfrom collections.abc import Callable+from datetime import datefrom pathlib import Pathimport pytest+from posthog.test.repo_invariants.ratchet import enforce_baseline+REPO_ROOT = Path(__file__).parents[3]BASELINE_DIR = Path(__file__).parentSOURCES_ROOT = REPO_ROOT / "products/warehouse_sources/backend/temporal/data_imports/sources"SKIPPED_DIRS = {"common", "tests", "test", "__pycache__", "generated_configs"}-REGENERATE = "python posthog/test/repo_invariants/test_warehouse_source_test_shapes.py"+REGENERATE = "PYTHONPATH=. python posthog/test/repo_invariants/test_warehouse_source_test_shapes.py"PRIVATE_SHARED_NAME = re.compile(r"sources\.common\.[A-Za-z0-9_.]+\._[a-z][A-Za-z0-9_]*$")MOCK_BUILDERS = {"MagicMock", "Mock", "NonCallableMagicMock", "create_autospec"}@@ -101,6 +104,12 @@ CHECKS: dict[str, tuple[Check, tuple[str, ...], str, str]] = {),}+BLOCKING_FROM: dict[str, date] = {+ "mocked_resume_manager": date.fromisoformat("2026-10-09"),+ "patched_session": date.fromisoformat("2026-10-09"),+ "private_shared_patch": date.fromisoformat("2026-10-09"),+}+def collect(check: Check, words: tuple[str, ...]) -> list[str]:found: list[str] = []@@ -159,12 +168,17 @@ def test_source_test_files_match_the_baseline(check: str) -> None:added = [line for line in scanned if line not in recorded]removed = [line for line in recorded if line not in scanned]report = "\n".join([*(f" + {line}" for line in added), *(f" - {line}" for line in removed)])- raise AssertionError(- f"{file_name} no longer matches the repo.\n"- f"{explanation}Use `SourceDriver` for an extraction. Use `scripted_network` for a call outside one. "- f"Both are in `{TESTING_PATH}`. Do not add the line to the baseline: the list may only get shorter.\n"- f"A '-' line means a test file was fixed. The file must record that too. Run: {REGENERATE}\n"- f"{report}"+ enforce_baseline(+ added=added,+ removed=removed,+ blocking_from=BLOCKING_FROM[check],+ message=(+ f"{file_name} no longer matches the repo.\n"+ f"{explanation}Use `SourceDriver` for an extraction. Use `scripted_network` for a call outside one. "+ f"Both are in `{TESTING_PATH}`. Do not add the line to the baseline: the list may only get shorter.\n"+ f"A '-' line means a test file was fixed. The file must record that too. Run: {REGENERATE}\n"+ f"{report}"+ ),)