:hedgehog: PostHog is the leading platform for building self-driving products. Our developer tools – AI observability, analytics, session replay, flags, experiments, error tracking, logs, and more – capture all the context agents need to diagnose problems, uncover opportunities, and ship fixes. Steer it all from Slack, web, desktop, or the MCP.
Python ab-testingai-analyticsanalyticscdpdata-warehouseexperimentsfeature-flagsjavascriptproduct-analyticspythonreactsession-replaysurveystypescriptweb-analytics
feat(hogql): staff switch for the native events table (#122489)
Sandy Spicerd282569d0ada
diff --git a/.agents/skills/writing-clickhouse-queries/references/choosing-a-workload.md b/.agents/skills/writing-clickhouse-queries/references/choosing-a-workload.mdindex 2f459b368969..9c4eb22accb3 100644--- a/.agents/skills/writing-clickhouse-queries/references/choosing-a-workload.md+++ b/.agents/skills/writing-clickhouse-queries/references/choosing-a-workload.md@@ -19,13 +19,12 @@ Both feed capacity planning, so pick the workload on purpose.Answer in order and stop at the first yes.-| Question | Workload |-| ------------------------------------------------------------------------------- | -------------------- |-| Does the query read a table that exists only on the logs cluster? | `LOGS` |-| Does the query execute an Endpoints (the product) request? | `ENDPOINTS` |-| Does the query target the native events schema cluster? | `NEW_EVENTS_CLUSTER` |-| Does a person wait on screen for this result (an app request, a Max chat turn)? | `ONLINE` |-| Anything else: scheduled, batch, export, backfill, recompute, eval, digest | `OFFLINE` |+| Question | Workload |+| ------------------------------------------------------------------------------- | ----------- |+| Does the query read a table that exists only on the logs cluster? | `LOGS` |+| Does the query execute an Endpoints (the product) request? | `ENDPOINTS` |+| Does a person wait on screen for this result (an app request, a Max chat turn)? | `ONLINE` |+| Anything else: scheduled, batch, export, backfill, recompute, eval, digest | `OFFLINE` |`ONLINE` is for latency a person feels.A background job does not get faster with `ONLINE`.@@ -54,7 +53,7 @@ The host comes from the environment of the pod that runs the code, in [`get_kwar- `ONLINE` connects to `CLICKHOUSE_HOST`.- `OFFLINE` connects to `CLICKHOUSE_OFFLINE_CLUSTER_HOST` when the deployment sets it, and to `CLICKHOUSE_HOST` when it does not.-- `LOGS`, `ENDPOINTS`, and `NEW_EVENTS_CLUSTER` connect to their own hosts.+- `LOGS` and `ENDPOINTS` connect to their own hosts.A background worker deployment can point `CLICKHOUSE_HOST` at the offline replicas and set `CLICKHOUSE_DEFAULT_WORKLOAD=OFFLINE`.Then `ONLINE` and `OFFLINE` from those pods both run on the offline replicas.diff --git a/docs/internal/web-analytics-query-serving.md b/docs/internal/web-analytics-query-serving.mdindex e0c1bfeec7e6..d6f6ce1601ab 100644--- a/docs/internal/web-analytics-query-serving.md+++ b/docs/internal/web-analytics-query-serving.md@@ -66,6 +66,10 @@ requestA miss costs one live-path serve; the background warm makes the next identical request a bucket hit.The dashboard "enqueues precompute" as a side effect; it never waits on it.+Requests that select native JSON events keep that table choice in precompute job identities and queued revalidation payloads.+Native and legacy revalidations use separate debounce keys, so a trial cannot suppress the project's normal rebuild.+Builders inherit the running query's table choice; an explicit query modifier takes precedence over the staff cookie.+## Marketing search performanceSearch metrics and PostHog landing-page conversions load in separate requests.diff --git a/frontend/src/queries/schema.json b/frontend/src/queries/schema.jsonindex e853c5bb152c..9c8a638e6fb3 100644--- a/frontend/src/queries/schema.json+++ b/frontend/src/queries/schema.json@@ -29960,12 +29960,8 @@"useMaterializedViews": {"type": "boolean"},- "useNewEventsCluster": {- "description": "Read this query from the native events cluster and native JSON events table (`true`), or keep its existing connection (`false` or unset). This is an internal rollout switch for eligible projects. PostHog staff manage eligibility through instance settings. Each query must opt in; project defaults cannot enable it. Only synchronous HogQLQuery SQL comparisons are supported.",- "type": "boolean"- },"useNewEventsSchema": {- "description": "Read events from the native JSON events table (`true`) or the legacy events table (`false`). When unset, the project's stored value applies, then the `CLICKHOUSE_HOGQL_USE_NEW_EVENTS_SCHEMA` instance settings. This is an internal rollout switch. PostHog staff set the project value in Django admin and the project settings API ignores it.",+ "description": "Read events from the native JSON events table (`true`) or the legacy events table (`false`). When unset, a staff person's `ph_new_events_schema` cookie applies, then the project's listing in the `CLICKHOUSE_HOGQL_USE_NEW_EVENTS_SCHEMA_TEAMS` instance setting, then the `CLICKHOUSE_HOGQL_USE_NEW_EVENTS_SCHEMA` instance setting. This is an internal rollout switch, and the project settings API ignores it.","type": "boolean"},"usePreaggregatedIntermediateResults": {diff --git a/frontend/src/queries/schema/schema-general.ts b/frontend/src/queries/schema/schema-general.tsindex a8226ad2b319..60eeec773fc9 100644--- a/frontend/src/queries/schema/schema-general.ts+++ b/frontend/src/queries/schema/schema-general.ts@@ -545,10 +545,8 @@ export interface HogQLQueryModifiers {materializedColumnsOptimizationMode?: 'disabled' | 'optimized'propertyGroupsMode?: 'enabled' | 'disabled' | 'optimized'useMaterializedViews?: boolean- /** Read events from the native JSON events table (`true`) or the legacy events table (`false`). When unset, the project's stored value applies, then the `CLICKHOUSE_HOGQL_USE_NEW_EVENTS_SCHEMA` instance settings. This is an internal rollout switch. PostHog staff set the project value in Django admin and the project settings API ignores it. */+ /** Read events from the native JSON events table (`true`) or the legacy events table (`false`). When unset, a staff person's `ph_new_events_schema` cookie applies, then the project's listing in the `CLICKHOUSE_HOGQL_USE_NEW_EVENTS_SCHEMA_TEAMS` instance setting, then the `CLICKHOUSE_HOGQL_USE_NEW_EVENTS_SCHEMA` instance setting. This is an internal rollout switch, and the project settings API ignores it. */useNewEventsSchema?: boolean- /** Read this query from the native events cluster and native JSON events table (`true`), or keep its existing connection (`false` or unset). This is an internal rollout switch for eligible projects. PostHog staff manage eligibility through instance settings. Each query must opt in; project defaults cannot enable it. Only synchronous HogQLQuery SQL comparisons are supported. */- useNewEventsCluster?: booleancustomChannelTypeRules?: CustomChannelRule[]customBotDefinitions?: CustomBotRule[]/** Do not treat a missing user agent as automation on cookieless events. Positive bot signals and custom project rules still apply. Resolved server-side; not intended to be set by clients. */diff --git a/posthog/api/instance_settings.py b/posthog/api/instance_settings.pyindex 726cc1e94f7e..d01f6fd263e0 100644--- a/posthog/api/instance_settings.py+++ b/posthog/api/instance_settings.py@@ -125,10 +125,11 @@ class InstanceSettingsSerializer(serializers.Serializer):is_secret = serializers.BooleanField(read_only=True)def validate_value(self, value: Any) -> Any:- if self.instance is not None and self.instance.key == "CLICKHOUSE_NATIVE_EVENTS_HOST":- if not isinstance(value, str) or (value and not re.fullmatch(r"[A-Za-z0-9][A-Za-z0-9.-]{0,252}", value)):+ if self.instance is not None and self.instance.key == "CLICKHOUSE_HOGQL_USE_NEW_EVENTS_SCHEMA_TEAMS":+ # Every query parses this list, so a value that fails to parse would fail every query on the instance.+ if not isinstance(value, str) or not re.fullmatch(r"\s*(\d+\s*(,\s*\d+\s*)*)?", value):raise serializers.ValidationError(- "Enter a hostname without a scheme, port or path, or leave it empty to disable the route."+ "Enter project IDs separated by commas, or leave it empty to switch no project.")return valuediff --git a/posthog/api/team/query_modifiers.py b/posthog/api/team/query_modifiers.pyindex ccab259e8357..e6cd46a7adba 100644--- a/posthog/api/team/query_modifiers.py+++ b/posthog/api/team/query_modifiers.py@@ -19,9 +19,6 @@ def merge_modifiers(requested: dict | None, existing: object) -> dict | None:stored = existing if isinstance(existing, dict) else {}if requested is not None:return {**stored, **requested}- # PostHog staff set useNewEventsSchema per project, so a client that clears its modifiers keeps it.- if "useNewEventsSchema" in stored:- return {"useNewEventsSchema": stored["useNewEventsSchema"]}return None@@ -32,10 +29,10 @@ def validate_modifiers(value: dict | None) -> dict | None:if not isinstance(value, dict):raise exceptions.ValidationError("Must provide a dictionary or None.")- # The native JSON events table is an internal rollout switch that PostHog staff set per project in Django- # admin. Dropping the key keeps a client from setting or clearing it, including a settings page that echoes- # the whole modifiers dict back.- value = {key: item for key, item in value.items() if key not in {"useNewEventsSchema", "useNewEventsCluster"}}+ # The native JSON events table is switched per project by an instance setting, never by a stored modifier.+ # Dropping the key keeps a client from storing one, including a settings page that echoes the whole+ # modifiers dict back.+ value = {key: item for key, item in value.items() if key != "useNewEventsSchema"}if "bounceRateDurationSeconds" in value:bounce_rate = value["bounceRateDurationSeconds"]diff --git a/posthog/api/test/test_instance_settings.py b/posthog/api/test/test_instance_settings.pyindex 8c486879eb3d..24514f7dbdfd 100644--- a/posthog/api/test/test_instance_settings.py+++ b/posthog/api/test/test_instance_settings.py@@ -111,21 +111,22 @@ class TestSecretSettingsCoverage(unittest.TestCase):self.assertEqual(_redact_if_secret("EMAIL_HOST", value), value)-class TestHostSettingValidation(SimpleTestCase):+class TestNewEventsSchemaTeamsValidation(SimpleTestCase):@parameterized.expand([- ("native.example.com", True),- ("localhost", True),("", True),- ("https://native.example.com", False),- ("native.example.com:8443", False),- ("native.example.com/path", False),- (True, False),+ ("2", True),+ (" 2, 44 ,123 ", True),+ ("2;44", False),+ ("team 2", False),+ ([2], False),])- def test_host_setting_accepts_hostnames_only(self, value: object, valid: bool) -> None:+ def test_project_list_accepts_comma_separated_ids_only(self, value: object, valid: bool) -> None:serializer = InstanceSettingsSerializer(- instance=InstanceSettingHelper(key="CLICKHOUSE_NATIVE_EVENTS_HOST"), data={"value": value}, partial=True+ instance=InstanceSettingHelper(key="CLICKHOUSE_HOGQL_USE_NEW_EVENTS_SCHEMA_TEAMS"),+ data={"value": value},+ partial=True,)self.assertEqual(serializer.is_valid(), valid, serializer.errors)@@ -215,8 +216,7 @@ class TestInstanceSettings(APIBaseTest):@parameterized.expand([("AUTO_START_ASYNC_MIGRATIONS", False, True),- ("CLICKHOUSE_NATIVE_EVENTS_TEAMS", [], [44]),- ("CLICKHOUSE_NATIVE_EVENTS_HOST", "", "native.example.com"),+ ("CLICKHOUSE_HOGQL_USE_NEW_EVENTS_SCHEMA_TEAMS", "", "2"),])def test_update_setting(self, key: str, before: object, after: object) -> None:@@ -306,8 +306,7 @@ class TestInstanceSettings(APIBaseTest):@parameterized.expand([("AUTO_START_ASYNC_MIGRATIONS", False, True),- ("CLICKHOUSE_NATIVE_EVENTS_TEAMS", [], [44]),- ("CLICKHOUSE_NATIVE_EVENTS_HOST", "", "native.example.com"),+ ("CLICKHOUSE_HOGQL_USE_NEW_EVENTS_SCHEMA_TEAMS", "", "2"),])def test_non_staff_user_cant_update(self, key: str, before: object, after: object) -> None:diff --git a/posthog/api/test/test_team.py b/posthog/api/test/test_team.pyindex bb894d13aa55..fc93166638d3 100644--- a/posthog/api/test/test_team.py+++ b/posthog/api/test/test_team.py@@ -1645,38 +1645,26 @@ def team_api_test_factory():{"id": "test", "channel_type": "Direct", "combiner": "AND", "items": []}]- @parameterized.expand([("useNewEventsSchema",), ("useNewEventsCluster",)])- def test_events_routing_modifiers_are_not_client_writable(self, modifier: str) -> None:+ def test_the_events_schema_modifier_is_not_client_writable(self) -> None:response = self.client.patch(f"/api/environments/{self.team.id}",- {"modifiers": {modifier: True}},+ {"modifiers": {"useNewEventsSchema": True}},)assert response.status_code == status.HTTP_200_OK, response.json()- assert modifier not in response.json()["modifiers"]+ assert "useNewEventsSchema" not in response.json()["modifiers"]- self.team.modifiers = {modifier: True}+ self.team.modifiers = {"useNewEventsSchema": True}self.team.save()response = self.client.patch(f"/api/environments/{self.team.id}",- {"modifiers": {modifier: False, "bounceRateDurationSeconds": 30}},+ {"modifiers": {"useNewEventsSchema": False, "bounceRateDurationSeconds": 30}},)assert response.status_code == status.HTTP_200_OK, response.json()- assert response.json()["modifiers"] == {modifier: True, "bounceRateDurationSeconds": 30}+ assert response.json()["modifiers"] == {"useNewEventsSchema": True, "bounceRateDurationSeconds": 30}@parameterized.expand([- (- "environments",- {"useNewEventsSchema": True, "bounceRateDurationSeconds": 30},- None,- {"useNewEventsSchema": True},- ),- (- "projects",- {"useNewEventsSchema": True, "bounceRateDurationSeconds": 30},- None,- {"useNewEventsSchema": True},- ),+ ("environments", {"useNewEventsSchema": True, "bounceRateDurationSeconds": 30}, None, None),("environments", True, None, None),("projects", ["useNewEventsSchema"], None, None),("environments", True, {"bounceRateDurationSeconds": 30}, {"bounceRateDurationSeconds": 30}),diff --git a/posthog/clickhouse/client/connection.py b/posthog/clickhouse/client/connection.pyindex 586900ac49ea..1b40bbac2c3d 100644--- a/posthog/clickhouse/client/connection.py+++ b/posthog/clickhouse/client/connection.py@@ -455,18 +455,6 @@ def get_kwargs_for_client(creds = get_clickhouse_creds(ch_user)base_kwargs = {"user": creds.user, "password": creds.password}- if workload == Workload.NEW_EVENTS_CLUSTER:- from posthog.models.event.new_events_schema import get_new_events_cluster_host # noqa: PLC0415-- host = get_new_events_cluster_host()- if not host:- raise ValueError("Native events ClickHouse host is not configured")- return {- **get_kwargs_for_client(Workload.DEFAULT, team_id, readonly, ch_user),- "host": host,- "verify": settings.CLICKHOUSE_VERIFY,- }-if team_id is not None and str(team_id) in settings.CLICKHOUSE_PER_TEAM_SETTINGS:user_settings = settings.CLICKHOUSE_PER_TEAM_SETTINGS[str(team_id)]return {**base_kwargs, **user_settings}diff --git a/posthog/clickhouse/client/execute.py b/posthog/clickhouse/client/execute.pyindex 9523cd09b282..875d27a978ae 100644--- a/posthog/clickhouse/client/execute.py+++ b/posthog/clickhouse/client/execute.py@@ -455,7 +455,7 @@ def sync_execute(workload = Workload.OFFLINE if is_api_key_auth else Workload.ONLINEch_user = ClickHouseUser.API if is_api_key_auth else ClickHouseUser.APP- if tags.workload == Workload.ENDPOINTS and workload not in (Workload.LOGS, Workload.NEW_EVENTS_CLUSTER):+ if tags.workload == Workload.ENDPOINTS and workload != Workload.LOGS:workload = Workload.ENDPOINTSif workload == Workload.DEFAULT:diff --git a/posthog/clickhouse/client/test/test_connection.py b/posthog/clickhouse/client/test/test_connection.pyindex bb5eb2175ed9..97020e9f90e3 100644--- a/posthog/clickhouse/client/test/test_connection.py+++ b/posthog/clickhouse/client/test/test_connection.py@@ -143,51 +143,6 @@ def test_signals_readonly_queries_keep_dedicated_credentials_and_offline_host(assert kwargs == {"user": "shared-readonly", "password": "shared-password"}-def test_native_events_pool_is_separate_from_legacy_pool(settings, monkeypatch):- from posthog.hogql_queries.utils.parallel import run_in_parallel_threads- from posthog.models.event.new_events_schema import new_events_cluster_host_scope-- settings.CLICKHOUSE_NATIVE_EVENTS_HOST = "env.example.com"- host = "native.example.com"- monkeypatch.setattr("posthog.models.event.new_events_schema.get_instance_setting", lambda key: host)- settings.CLICKHOUSE_VERIFY = True- settings.CLICKHOUSE_PER_TEAM_SETTINGS = {}- legacy = get_pool(Workload.ONLINE, team_id=44)- native = get_pool(Workload.NEW_EVENTS_CLUSTER, team_id=44)- assert native is not legacy- assert native.connection_args["host"] == "native.example.com"- with new_events_cluster_host_scope():- assert get_pool(Workload.NEW_EVENTS_CLUSTER, team_id=44) is native- host = "replacement.example.com"- pools = []- run_in_parallel_threads([lambda: pools.append(get_pool(Workload.NEW_EVENTS_CLUSTER, team_id=44))])- assert pools == [native]- replacement = get_pool(Workload.NEW_EVENTS_CLUSTER, team_id=44)- assert replacement is not native- assert replacement.connection_args["host"] == host--[email protected]("readonly", [False, True])[email protected]("team_override", [False, True])-def test_native_events_route_preserves_resolved_credentials(settings, readonly, team_override):- settings.CLICKHOUSE_NATIVE_EVENTS_HOST = "native.example.com"- settings.CLICKHOUSE_VERIFY = True- settings.CLICKHOUSE_PER_TEAM_SETTINGS = (- {"44": {"host": "legacy.example.com", "user": "team-user", "password": "example"}} if team_override else {}- )- settings.READONLY_CLICKHOUSE_USER = "reader"- settings.READONLY_CLICKHOUSE_PASSWORD = "example-readonly"- baseline = connection.get_kwargs_for_client(Workload.DEFAULT, team_id=44, readonly=readonly)- native = connection.get_kwargs_for_client(Workload.NEW_EVENTS_CLUSTER, team_id=44, readonly=readonly)- assert native == {**baseline, "host": "native.example.com", "verify": True}---def test_native_events_workload_without_host_fails_closed(settings):- settings.CLICKHOUSE_NATIVE_EVENTS_HOST = None- with pytest.raises(ValueError, match="Native events ClickHouse host is not configured"):- get_pool(Workload.NEW_EVENTS_CLUSTER)--def test_read_password_reads_file_fresh_and_stripped(tmp_path):token = tmp_path / "token"token.write_text("tok-0\n")@@ -310,33 +265,29 @@ def test_deletion_executor_credentials_do_not_fall_back_to_default(monkeypatch):get_clickhouse_creds(ClickHouseUser.DELETION_EXECUTOR)[email protected]("workload", [Workload.ONLINE, Workload.NEW_EVENTS_CLUSTER])-def test_file_backed_pool_is_stable_across_credential_rotation(settings, monkeypatch, tmp_path, workload):- settings.CLICKHOUSE_NATIVE_EVENTS_HOST = "native.example.com"+def test_file_backed_pool_is_stable_across_credential_rotation(settings, monkeypatch, tmp_path):settings.CLICKHOUSE_OFFLINE_CLUSTER_HOST = Nonetoken = tmp_path / "token"token.write_text("tok-0")_file_backed_default(monkeypatch, token)- pool = get_pool(workload)+ pool = get_pool(Workload.ONLINE)assert isinstance(pool, RefreshingChPool)token.write_text("tok-1") # rotate the projected token in place- assert get_pool(workload) is pool+ assert get_pool(Workload.ONLINE) is poolassert make_ch_pool.cache_info().currsize == 1[email protected]("workload", [Workload.ONLINE, Workload.NEW_EVENTS_CLUSTER])-def test_get_http_kwargs_reads_file_backed_credential_fresh(settings, monkeypatch, tmp_path, workload):- settings.CLICKHOUSE_NATIVE_EVENTS_HOST = "native.example.com"+def test_get_http_kwargs_reads_file_backed_credential_fresh(monkeypatch, tmp_path):token = tmp_path / "token"token.write_text("tok-0")_file_backed_default(monkeypatch, token)- assert get_http_kwargs(workload)["password"] == "tok-0"+ assert get_http_kwargs(Workload.ONLINE)["password"] == "tok-0"token.write_text("tok-1") # a rotation must reach the short-lived HTTP client, not a value cached at build- assert get_http_kwargs(workload)["password"] == "tok-1"+ assert get_http_kwargs(Workload.ONLINE)["password"] == "tok-1"def test_get_client_from_pool_http_branch_sends_fresh_token(settings, monkeypatch, tmp_path):@@ -410,12 +361,8 @@ def test_refreshing_pool_stamps_current_credential(tmp_path: Path, failure: str@pytest.fixture(autouse=True)-def reset_state(settings, monkeypatch):+def reset_state():make_ch_pool.cache_clear()- monkeypatch.setattr(- "posthog.models.event.new_events_schema.get_instance_setting",- lambda key: settings.CLICKHOUSE_NATIVE_EVENTS_HOST,- )yield@@ -458,34 +405,3 @@ def test_logs_workload_uses_a_named_user_only_when_registered(expected_user,expected_password,)--- "workload,clickhouse_verify,queryservice_verify,expected_verify",- [- (Workload.NEW_EVENTS_CLUSTER, False, True, True),- (Workload.NEW_EVENTS_CLUSTER, True, False, False),- (Workload.OFFLINE, True, True, True),- ],-)-def test_http_pool_preserves_existing_verification_policy(- settings, monkeypatch, workload, clickhouse_verify, queryservice_verify, expected_verify-):- settings.CLICKHOUSE_NATIVE_EVENTS_HOST = "native.example.com"- settings.CLICKHOUSE_OFFLINE_CLUSTER_HOST = "offline.example.com"- settings.CLICKHOUSE_VERIFY = clickhouse_verify- settings.QUERYSERVICE_VERIFY = queryservice_verify- settings.CLICKHOUSE_SECURE = True- settings.CLICKHOUSE_LOGS_CLUSTER_SECURE = True- settings.CLICKHOUSE_USE_HTTP = True- monkeypatch.setattr(connection, "_default_workload", Workload.OFFLINE)- connection._clickhouse_http_pool_mgr.cache_clear()- try:- with patch("clickhouse_connect.get_client") as client:- with connection.get_client_from_pool(workload):- pass- manager = client.call_args.kwargs["pool_mgr"]- assert client.call_args.kwargs["secure"]- assert manager.connection_pool_kw["cert_reqs"] == ("CERT_REQUIRED" if expected_verify else "CERT_NONE")- finally:- connection._clickhouse_http_pool_mgr.cache_clear()diff --git a/posthog/clickhouse/client/test/test_execute.py b/posthog/clickhouse/client/test/test_execute.pyindex 375be0a5af89..1003a571562a 100644--- a/posthog/clickhouse/client/test/test_execute.py+++ b/posthog/clickhouse/client/test/test_execute.py@@ -83,7 +83,6 @@ def test_celery_cache_warming_workload_routing(client_from_pool, access_method,[(Workload.DEFAULT, Workload.ENDPOINTS),(Workload.LOGS, Workload.LOGS),- (Workload.NEW_EVENTS_CLUSTER, Workload.NEW_EVENTS_CLUSTER),],)def test_endpoints_tag_workload_routing(client_from_pool, workload, expected_workload):diff --git a/posthog/clickhouse/workload.py b/posthog/clickhouse/workload.pyindex dade6763b568..723ac6843046 100644--- a/posthog/clickhouse/workload.py+++ b/posthog/clickhouse/workload.py@@ -12,6 +12,5 @@ class Workload(StrEnum):OFFLINE = "OFFLINE"# Tables that exist only on the logs clusterLOGS = "LOGS"- NEW_EVENTS_CLUSTER = "NEW_EVENTS_CLUSTER"# Endpoints (the product) queriesENDPOINTS = "ENDPOINTS"diff --git a/posthog/hogql/modifiers.py b/posthog/hogql/modifiers.pyindex 23af691149a1..8465ca863914 100644--- a/posthog/hogql/modifiers.py+++ b/posthog/hogql/modifiers.py@@ -58,22 +58,17 @@ def create_default_modifiers_for_team(else:modifiers = modifiers.model_copy()- from posthog.models.event.new_events_schema import use_new_events_cluster # noqa: PLC0415-- if use_new_events_cluster(team.pk, modifiers):- if modifiers.useNewEventsSchema is False:- from posthog.hogql.errors import ExposedHogQLError # noqa: PLC0415-- raise ExposedHogQLError("Native events cluster queries require the native events schema.")- modifiers.useNewEventsSchema = True-if modifiers.useMaterializedViews is None:modifiers.useMaterializedViews = Trueif isinstance(team.modifiers, dict):for key, value in team.modifiers.items():- # Cluster routing requires a query opt-in, even if a project stores the modifier.- if key == "useNewEventsCluster":+ # A modifier removed from the schema can stay stored on a project, and the model rejects unknown fields.+ if key not in HogQLQueryModifiers.model_fields:+ continue+ # The events table is switched per project by CLICKHOUSE_HOGQL_USE_NEW_EVENTS_SCHEMA_TEAMS, which every+ # reader of the table checks. A stored modifier would switch the HogQL readers alone.+ if key == "useNewEventsSchema":continueif getattr(modifiers, key, None) is None:if key == "customChannelTypeRules":diff --git a/posthog/hogql/query.py b/posthog/hogql/query.pyindex 51d7abdea69d..22bfd1eac9cf 100644--- a/posthog/hogql/query.py+++ b/posthog/hogql/query.py@@ -73,11 +73,11 @@ from posthog.hogql.warehouse_warnings import record_warningsfrom posthog.clickhouse.client import sync_executefrom posthog.clickhouse.client.connection import ClickHouseUser, Workload-from posthog.clickhouse.query_tagging import Product, get_query_tag_value, get_query_tags, tag_queries+from posthog.clickhouse.query_tagging import get_query_tag_value, get_query_tags, tag_queriesfrom posthog.dataclasses import frozenfrom posthog.direct_query_cancellation import build_direct_query_cancellation_tokenfrom posthog.errors import CHQueryErrorS3Error, CHQueryErrorS3FileChangedDuringRead, ExposedCHQueryError-from posthog.models.event.new_events_schema import new_events_cluster_host_scope, use_new_events_cluster+from posthog.models.event.new_events_schema import use_new_events_schemafrom posthog.models.team import Teamfrom posthog.models.user import Userfrom posthog.settings import HOGQL_INCREASED_MAX_EXECUTION_TIME@@ -166,8 +166,6 @@ class HogQLQueryExecutor:self.context.user_access_control = self.user_access_controlself.query_modifiers = create_default_modifiers_for_team(self.team, self.modifiers)- if self.connection_id and self.query_modifiers.useNewEventsCluster:- raise ExposedHogQLError("Native events cluster queries cannot use a warehouse connection.")self.debug = self.modifiers is not None and self.modifiers.debugself.error: Optional[str] = Noneself.explain: Optional[list[str]] = None@@ -253,9 +251,10 @@ class HogQLQueryExecutor:@tracer.start_as_current_span("HogQLQueryExecutor._apply_optimizers")def _apply_optimizers(self):- if self.query_modifiers.useNewEventsCluster:- return- if self.query_modifiers.usePreaggregatedTableTransforms:+ # The preaggregated tables are built from the legacy events table, so a read of events_json skips them.+ if self.query_modifiers.usePreaggregatedTableTransforms and not use_new_events_schema(+ self.team.pk, self.query_modifiers+ ):with self.timings.measure("preaggregated_table_transforms"):assert self.hogql_context is not Noneassert self.hogql_context.team is not None@@ -972,19 +971,6 @@ class HogQLQueryExecutor:table_workload = clickhouse_context.workloadif table_workload is not None and table_workload != Workload.DEFAULT and workload in MAIN_CLUSTER_WORKLOADS:workload = table_workload- if use_new_events_cluster(self.team.pk, self.query_modifiers):- if get_query_tags().product == Product.ENDPOINTS or workload not in (- Workload.DEFAULT,- Workload.ONLINE,- Workload.OFFLINE,- Workload.NEW_EVENTS_CLUSTER,- ):- raise ExposedHogQLError(- "New events cluster opt-ins support analytics workloads only. Remove useNewEventsCluster for logs or endpoints queries."- )- if not clickhouse_context.uses_new_events_schema():- raise ExposedHogQLError("New events cluster queries require the native events schema.")- workload = Workload.NEW_EVENTS_CLUSTERdef run_clickhouse_query() -> Any:return sync_execute(@@ -1076,7 +1062,6 @@ class HogQLQueryExecutor:)@tracer.start_as_current_span("HogQLQueryExecutor.execute")- @new_events_cluster_host_scope()def execute(self) -> HogQLQueryResponse:self._executing = Truetrace.get_current_span().set_attribute("team_id", self.team.pk)diff --git a/posthog/hogql/test/test_modifiers.py b/posthog/hogql/test/test_modifiers.pyindex 85305261405b..907f25e716fc 100644--- a/posthog/hogql/test/test_modifiers.py+++ b/posthog/hogql/test/test_modifiers.py@@ -80,6 +80,7 @@ class TestModifiers(BaseTest):== PersonsOnEventsMode.PERSON_ID_OVERRIDE_PROPERTIES_JOINED # no change here)+ # A modifier removed from the schema can stay stored on a project.self.team.modifiers = {"personsOnEventsMode": PersonsOnEventsMode.PERSON_ID_NO_OVERRIDE_PROPERTIES_ON_EVENTS,"useNewEventsCluster": True,@@ -87,7 +88,6 @@ class TestModifiers(BaseTest):self.team.save()modifiers = create_default_modifiers_for_team(self.team)assert modifiers.personsOnEventsMode == PersonsOnEventsMode.PERSON_ID_NO_OVERRIDE_PROPERTIES_ON_EVENTS- assert modifiers.useNewEventsCluster is Nonedef test_unparseable_custom_bot_definitions_are_dropped(self):# modifiers is hand-editable JSON (e.g. Django admin), so an entry that is not a rule objectdiff --git a/posthog/hogql/test/test_new_events_schema_routing.py b/posthog/hogql/test/test_new_events_schema_routing.pyindex ab27e2c45ac9..2d1a02672e09 100644--- a/posthog/hogql/test/test_new_events_schema_routing.py+++ b/posthog/hogql/test/test_new_events_schema_routing.py@@ -1,34 +1,40 @@+import json+from posthog.test.base import APIBaseTest-from unittest.mock import patch+from unittest.mock import MagicMock, patchfrom django.test import SimpleTestCase, override_settings+from django.urls import reversefrom parameterized import parameterized-from posthog.schema import HogQLQuery, HogQLQueryModifiers, PersonsOnEventsMode+from posthog.schema import HogQLQueryModifiers, PersonsOnEventsMode-from posthog.hogql import ast-from posthog.hogql.context import HogQLContext-from posthog.hogql.errors import ExposedHogQLError-from posthog.hogql.query import HogQLQueryExecutor+from posthog.hogql.modifiers import create_default_modifiers_for_team+from posthog.hogql.parser import parse_select+from posthog.hogql.query import execute_hogql_query-from posthog.clickhouse.query_tagging import Product, tags_context-from posthog.clickhouse.workload import Workload-from posthog.hogql_queries.hogql_query_runner import HogQLQueryRunnerfrom posthog.models import Organization, Team, User+from posthog.models.event.new_events_schema import (+ NEW_EVENTS_SCHEMA_COOKIE,+ events_route_scope,+ new_events_schema_for_request,+ use_new_events_schema,+)from posthog.models.instance_setting import override_instance_config+from products.dashboards.backend.models.dashboard import Dashboard+from products.dashboards.backend.models.dashboard_tile import DashboardTile+from products.product_analytics.backend.facade.models import Insight++SQL_ON_EVENTS: dict[str, object] = {"kind": "HogQLQuery", "query": "SELECT count() AS n FROM events"}+-@override_settings(CLICKHOUSE_HOGQL_USE_NEW_EVENTS_SCHEMA=False, CLICKHOUSE_NATIVE_EVENTS_HOST="native.example.com")-class TestNewEventsClusterRouting(SimpleTestCase):+@override_settings(CLICKHOUSE_HOGQL_USE_NEW_EVENTS_SCHEMA=False)+class TestEventsTableSwitch(SimpleTestCase):def setUp(self) -> None:self.team = Team(id=44, modifiers={}, organization=Organization(id=1))- self.user = User(id=123, is_staff=True, is_active=True)- self.modifiers = HogQLQueryModifiers(personsOnEventsMode=PersonsOnEventsMode.DISABLED)- self.instance_settings: dict[str, object] = {- "CLICKHOUSE_NATIVE_EVENTS_TEAMS": [44],- "CLICKHOUSE_NATIVE_EVENTS_HOST": "native.example.com",- }+ self.instance_settings: dict[str, object] = {}self.enterContext(patch("posthog.models.event.new_events_schema.get_instance_setting", self.instance_settings.get))@@ -39,182 +45,104 @@ class TestNewEventsClusterRouting(SimpleTestCase):))- @parameterized.expand(- [- ([], "native.example.com", "not enabled"),- ([44, True], "native.example.com", "not enabled"),- ([44], None, "host is not configured"),- ]- )- def test_unavailable_native_route_fails_closed(self, allowed: object, host: str | None, error: str) -> None:- self.instance_settings.update(CLICKHOUSE_NATIVE_EVENTS_HOST=host, CLICKHOUSE_NATIVE_EVENTS_TEAMS=allowed)- with self.assertRaisesRegex(ExposedHogQLError, error):- HogQLQueryRunner(- query=HogQLQuery(- query="SELECT 1", modifiers=self.modifiers.model_copy(update={"useNewEventsCluster": True})- ),- team=self.team,- )+ def test_a_stored_project_modifier_does_not_switch_the_events_table(self) -> None:+ self.team.modifiers = {"useNewEventsSchema": True, "bounceRateDurationSeconds": 30}+ modifiers = create_default_modifiers_for_team(+ self.team, HogQLQueryModifiers(personsOnEventsMode=PersonsOnEventsMode.DISABLED)+ )+ self.assertIsNone(modifiers.useNewEventsSchema)+ self.assertEqual(modifiers.bounceRateDurationSeconds, 30)- def test_direct_warehouse_connection_cannot_claim_native_cluster_opt_in(self) -> None:- with self.assertRaisesRegex(ExposedHogQLError, "warehouse connection"):- HogQLQueryExecutor(- query="SELECT 1",- team=self.team,- modifiers=self.modifiers.model_copy(update={"useNewEventsCluster": True}),- connection_id="example-connection",+ @parameterized.expand([(False, None, False), (True, None, True), (False, True, True), (True, False, False)])+ def test_nested_reads_inherit_the_querys_events_table(+ self, parent_schema: bool, child_schema: bool | None, expected: bool+ ) -> None:+ with new_events_schema_for_request(), events_route_scope(HogQLQueryModifiers(useNewEventsSchema=parent_schema)):+ self.assertEqual(+ use_new_events_schema(self.team.pk, HogQLQueryModifiers(useNewEventsSchema=child_schema)), expected)- @parameterized.expand([(False,), (True,)])- def test_native_cache_identity_tracks_host_configuration_without_changing_legacy_namespace(- self, native: bool- ) -> None:- runner = HogQLQueryRunner(- query=HogQLQuery(- query="SELECT 1",- modifiers=self.modifiers.model_copy(- update={"useNewEventsSchema": native, "useNewEventsCluster": native}- ),- ),- team=self.team,- )- with (- patch.object(runner, "_get_property_access_restrictions", return_value=None),- patch.object(runner, "_products_modifiers_for_cache", return_value={}),- patch("posthog.hogql_queries.query_runner.events_retention_months_for_team", return_value=None),- ):- keys = []- for host in ("native.example.com", "replacement.example.com"):- self.instance_settings["CLICKHOUSE_NATIVE_EVENTS_HOST"] = host- keys.append(runner.get_cache_key())- self.assertEqual(len(set(keys)), 2 if native else 1)- @parameterized.expand(- [- (- workload,- None,- workload in (Workload.DEFAULT, Workload.ONLINE, Workload.OFFLINE, Workload.NEW_EVENTS_CLUSTER),- )- for workload in Workload- ]- + [(Workload.DEFAULT, Product.ENDPOINTS, False)]- )- def test_native_opt_in_accepts_only_analytics_workloads(- self, workload: Workload, product: Product | None, supported: bool- ) -> None:- executor = HogQLQueryExecutor(- query="SELECT 1",- team=self.team,- workload=workload,- modifiers=self.modifiers.model_copy(update={"useNewEventsCluster": True}),- )- executor.select_query = ast.SelectQuery(select=[ast.Constant(value=1)])- executor.clickhouse_sql = "SELECT 1"- executor.clickhouse_context = HogQLContext(team_id=self.team.pk, modifiers=executor.query_modifiers)- with (- tags_context(product=product),- patch.object(executor, "_detect_warehouse_sources"),- patch("posthog.hogql.query.sync_execute", return_value=([(1,)], [("1", "UInt8")])) as execute,- ):- if supported:- executor._execute_clickhouse_query()- self.assertEqual(execute.call_args.kwargs["workload"], Workload.NEW_EVENTS_CLUSTER)- else:- with self.assertRaisesRegex(ExposedHogQLError, "analytics workloads only"):- executor._execute_clickhouse_query()- execute.assert_not_called()---@override_settings(CLICKHOUSE_NATIVE_EVENTS_HOST="native.example.com", CLICKHOUSE_HOGQL_USE_NEW_EVENTS_SCHEMA=False)-class TestNewEventsClusterPilot(APIBaseTest):+@override_settings(CLICKHOUSE_HOGQL_USE_NEW_EVENTS_SCHEMA=False)+class TestEventsTableSwitchThroughTheApi(APIBaseTest):def setUp(self) -> None:super().setUp()- self.enterContext(override_instance_config("CLICKHOUSE_NATIVE_EVENTS_HOST", "native.example.com"))- self.enterContext(override_instance_config("CLICKHOUSE_NATIVE_EVENTS_TEAMS", [self.team.pk]))self.enterContext(override_instance_config("CLICKHOUSE_HOGQL_USE_NEW_EVENTS_SCHEMA", False))+ def _post_query(self, query: dict[str, object], modifiers: dict[str, bool]) -> MagicMock:+ with patch("posthog.hogql.query.sync_execute", return_value=([], [])) as execute:+ response = self.client.post(+ f"/api/projects/{self.team.id}/query/",+ {"query": {**query, "modifiers": modifiers}, "refresh": "force_blocking"},+ format="json",+ )+ self.assertEqual(response.status_code, 200, response.json())+ return execute+@parameterized.expand([- (modifiers, eligible, native_table, native_cluster)- for modifiers, eligible, native_table, native_cluster in (- ({"useNewEventsCluster": True}, True, True, True),- ({"useNewEventsCluster": True}, False, False, True),- ({}, True, False, False),- ({"useNewEventsSchema": True}, True, True, False),- ({"useNewEventsCluster": False, "useNewEventsSchema": True}, True, True, False),- )- ]- + [- (- {"useNewEventsCluster": True, modifier: True},- True,- True,- True,- sql,- )- for modifier, sql in (- ("usePreaggregatedTableTransforms", "SELECT count() FROM events WHERE event = '$pageview'"),- (- "usePreaggregatedIntermediateResults",- "SELECT toStartOfDay(timestamp), uniqExact(person_id) FROM events WHERE event = '$pageview' AND timestamp >= '2024-01-01' AND timestamp < '2024-01-08' GROUP BY toStartOfDay(timestamp)",- ),- )- ]- + [- ({"useNewEventsCluster": True}, True, True, True, "SELECT 1", "force_async"),- ({"useNewEventsCluster": True}, True, True, True, "SELECT 1", "force_blocking", True),- (- {"useNewEventsCluster": True},- True,- True,- True,- {"kind": "PropertyValuesQuery", "property_key": "$browser", "property_type": "event"},- ),+ ("staff", True, False, {}, True),+ ("staff_query_pinned_to_legacy", True, False, {"useNewEventsSchema": False}, False),+ ("staff_impersonating_a_customer", True, True, {}, True),+ ("not_staff", False, False, {}, False),])- def test_query_route_requires_an_eligible_explicit_blocking_opt_in(- self,- modifiers: dict[str, bool],- eligible: bool,- native_table: bool,- native_cluster: bool,- sql: str | dict[str, str] = "SELECT count() AS n FROM events",- refresh: str = "force_blocking",- async_: bool = False,+ def test_the_cookie_switches_a_staff_persons_requests_to_events_json(+ self, _name: str, staff: bool, impersonating: bool, modifiers: dict[str, bool], events_json: bool) -> None:- query = {"kind": "HogQLQuery", "query": sql} if isinstance(sql, str) else sql- unsupported = native_cluster and (query["kind"] != "HogQLQuery" or refresh != "force_blocking" or async_)- with (- override_instance_config("CLICKHOUSE_NATIVE_EVENTS_TEAMS", [self.team.pk] if eligible else []),- patch(- "products.analytics_platform.backend.lazy_computation.lazy_computation_executor.sync_execute",- return_value=0,- ) as insert,- patch("posthog.hogql.query.sync_execute", return_value=([(7,)], [("n", "UInt64")])) as execute,- patch("posthog.hogql_queries.query_runner.enqueue_process_query_task") as enqueue,- ):- response = self.client.post(- f"/api/projects/{self.team.id}/query/",- {- "query": {**query, "modifiers": modifiers},- "refresh": refresh,- "async": async_,- },- format="json",+ self.user.is_staff = staff+ self.user.save()+ if impersonating:+ customer = User.objects.create_and_join(self.organization, "[email protected]", None)+ # The login-as admin view reads a form post, and this client sends JSON unless told otherwise.+ self.client.post(+ reverse("loginas-user-login", kwargs={"user_id": customer.id}),+ data={"read_only": "false", "reason": "Checking the native table"},+ format="multipart",+ follow=True,+ )+ self.assertEqual(self.client.get("/api/users/@me").json()["email"], customer.email)+ self.client.cookies[NEW_EVENTS_SCHEMA_COOKIE] = "1"++ execute = self._post_query(SQL_ON_EVENTS, modifiers)+ self.assertEqual("events_json" in execute.call_args.args[0], events_json)++ @parameterized.expand([("legacy", False), ("events_json", True)])+ def test_only_a_legacy_read_uses_the_preaggregated_web_tables(self, _name: str, events_json: bool) -> None:+ with patch("posthog.hogql.query.sync_execute", return_value=([], [])) as execute:+ execute_hogql_query(+ parse_select("select count(), uniq(person_id) from events where equals(event, '$pageview')"),+ team=self.team,+ modifiers=HogQLQueryModifiers(usePreaggregatedTableTransforms=True, useNewEventsSchema=events_json),)- rejected = native_cluster and (not eligible or unsupported)- self.assertEqual(response.status_code, 400 if rejected else 200, response.json())- enqueue.assert_not_called()- if rejected:- execute.assert_not_called()- if unsupported:- self.assertIn("Native cluster comparisons", response.json()["detail"])- return- self.assertEqual("events_json" in execute.call_args.args[0], native_table)- self.assertEqual(execute.call_args.kwargs["workload"] == Workload.NEW_EVENTS_CLUSTER, native_cluster)- if native_cluster:- insert.assert_not_called()- self.assertNotIn("pre_aggregated", execute.call_args.args[0])- self.assertNotIn("preaggregation_results", execute.call_args.args[0])- self.assertIs(response.json()["modifiers"]["useNewEventsCluster"], True)+ self.assertEqual("web_pre_aggregated_stats" in execute.call_args.args[0], not events_json)++ def test_the_cookie_reaches_a_streamed_dashboards_cache_lookup(self) -> None:+ self.user.is_staff = True+ self.user.save()+ self.client.cookies[NEW_EVENTS_SCHEMA_COOKIE] = "1"+ insight = Insight.objects.create(+ team=self.team, filters={}, query={"kind": "DataVisualizationNode", "source": SQL_ON_EVENTS}+ )+ dashboard = Dashboard.objects.create(team=self.team, created_by=self.user)+ DashboardTile.objects.create(dashboard=dashboard, insight=insight)+ with patch("posthog.hogql.query.sync_execute", return_value=([(1,)], [("n", "UInt64")])) as execute:+ response = self.client.get(+ f"/api/projects/{self.team.id}/dashboards/{dashboard.id}/?refresh=force_blocking"+ )+ self.assertEqual(response.status_code, 200, response.json())+ self.assertIn("events_json", execute.call_args.args[0])++ # The stream only reads the cache, so a hit here means the switch reached the lookup's cache key.+ self.assertIsNotNone(self._streamed_tile_result(dashboard.id))+ self.client.cookies[NEW_EVENTS_SCHEMA_COOKIE] = "0"+ self.assertIsNone(self._streamed_tile_result(dashboard.id))++ def _streamed_tile_result(self, dashboard_id: int) -> object:+ response = self.client.get(f"/api/projects/{self.team.id}/dashboards/{dashboard_id}/stream_tiles/")+ self.assertEqual(response.status_code, 200)+ content = b"".join(response.streaming_content).decode() # type: ignore[attr-defined]+ metadata = next(+ line[6:] for line in content.split("\n") if line.startswith("data: ") and '"type":"metadata"' in line+ )+ return json.loads(metadata)["dashboard"]["tiles"][0]["insight"]["result"]diff --git a/posthog/hogql/transforms/test/test_preaggregated_table_transformation.py b/posthog/hogql/transforms/test/test_preaggregated_table_transformation.pyindex 14d3cf1f77fd..dcf6b0b42f25 100644--- a/posthog/hogql/transforms/test/test_preaggregated_table_transformation.py+++ b/posthog/hogql/transforms/test/test_preaggregated_table_transformation.py@@ -3,6 +3,8 @@ from datetime import UTC, datetimeimport unittestfrom posthog.test.base import APIBaseTest, BaseTest, ClickhouseTestMixin, QueryMatchingTest, snapshot_clickhouse_queries+from django.test import override_settings+from parameterized import parameterizedfrom posthog.schema import BaseMathType, DateRange, EventsNode, HogQLQueryModifiers, TrendsQuery@@ -827,6 +829,7 @@ GROUP BY date@snapshot_clickhouse_queries+@override_settings(CLICKHOUSE_HOGQL_USE_NEW_EVENTS_SCHEMA=False)class TestPreaggregatedTableTransformationIntegration(APIBaseTest, ClickhouseTestMixin):CLASS_DATA_LEVEL_SETUP = FalseTEST_DATA_DATE = datetime(2024, 11, 24, tzinfo=UTC)diff --git a/posthog/hogql_queries/hogql_query_runner.py b/posthog/hogql_queries/hogql_query_runner.pyindex bc279b13d85f..73dacce6bf4e 100644--- a/posthog/hogql_queries/hogql_query_runner.py+++ b/posthog/hogql_queries/hogql_query_runner.py@@ -68,8 +68,6 @@ class HogQLQueryRunner(AnalyticsQueryRunner[HogQLQueryResponse]):self._direct_engine: str | None = Noneself._managed_warehouse_sql_mode: ManagedWarehouseSQLMode | None = Nonesuper().__init__(*args, **kwargs)- if self.modifiers.useNewEventsCluster and not self._reads_events_route():- raise ExposedHogQLError("Native events cluster queries cannot use a warehouse connection.")def _reads_events_route(self) -> bool:# A direct connection runs on another database, so the events route never applies to it.diff --git a/posthog/hogql_queries/query_runner.py b/posthog/hogql_queries/query_runner.pyindex ebe18cf3149f..e4a7a2cc11fe 100644--- a/posthog/hogql_queries/query_runner.py+++ b/posthog/hogql_queries/query_runner.py@@ -125,7 +125,6 @@ from posthog.hogql import astfrom posthog.hogql.constants import LimitContextfrom posthog.hogql.context import HogQLContextfrom posthog.hogql.database.database import Database-from posthog.hogql.errors import ExposedHogQLErrorfrom posthog.hogql.modifiers import create_default_modifiers_for_userfrom posthog.hogql.printer import prepare_and_print_ast, to_printed_hogqlfrom posthog.hogql.query import create_default_modifiers_for_team@@ -195,8 +194,6 @@ from posthog.models import Team, Userfrom posthog.models.event.new_events_schema import (events_route_cache_modifiers,events_route_scope,- get_new_events_cluster_host,- new_events_cluster_host_scope,strip_events_route_modifiers,use_new_events_schema,with_pinned_events_route,@@ -2472,7 +2469,6 @@ class QueryRunner(ABC, Generic[Q, R, CR]):query_duration_ms = round((perf_counter() - query_start_time) * 1000, 2)return query_result, query_duration_ms- @new_events_cluster_host_scope()@_pin_events_routedef run(self,@@ -2484,12 +2480,6 @@ class QueryRunner(ABC, Generic[Q, R, CR]):cache_age_seconds: Optional[int] = None,analytics_props: Optional[AnalyticsProps] = None,) -> CR | CacheMissResponse | QueryStatusResponse:- if self.modifiers.useNewEventsCluster:- if not isinstance(self.query, HogQLQuery):- raise ExposedHogQLError("Native cluster comparisons currently support HogQLQuery only.")- if execution_mode != ExecutionMode.CALCULATE_BLOCKING_ALWAYS:- raise ExposedHogQLError("Native cluster comparisons require refresh=force_blocking and async=false.")-# Set user for access control during query execution. Some subclasses# (e.g. QueryRunnerWithHogQLContext) construct user-dependent state in# __init__; let them refresh it now that we know the real user. Only on an@@ -3228,8 +3218,6 @@ class QueryRunner(ABC, Generic[Q, R, CR]):"week_start_day": self.team.week_start_day or WeekStartDay.SUNDAY,"version": 2,}- if self.modifiers.useNewEventsCluster:- payload["native_events_host"] = get_new_events_cluster_host()# Include property-level access control restrictions in the cache key so that# users with different property restrictions get separate cache entries.diff --git a/posthog/hogql_queries/test/test_query_runner.py b/posthog/hogql_queries/test/test_query_runner.pyindex 25919d026c95..4aa6d28df76e 100644--- a/posthog/hogql_queries/test/test_query_runner.py+++ b/posthog/hogql_queries/test/test_query_runner.py@@ -943,9 +943,8 @@ class TestQueryRunner(BaseTest):def test_cache_hit_and_fresh_events_carry_one_identity(self) -> None:runner_class = setup_test_query_runner_class()- self.team.modifiers = {"useNewEventsSchema": True}- self.team.save()with (+ override_instance_config("CLICKHOUSE_HOGQL_USE_NEW_EVENTS_SCHEMA_TEAMS", str(self.team.pk)),mock.patch("posthog.hogql_queries.query_runner.create_default_modifiers_for_user", side_effect=_add_user_modifier),diff --git a/posthog/hogql_queries/utils/test/test_timestamp_utils.py b/posthog/hogql_queries/utils/test/test_timestamp_utils.pyindex f07b172fdf58..55ca7c46b45a 100644--- a/posthog/hogql_queries/utils/test/test_timestamp_utils.py+++ b/posthog/hogql_queries/utils/test/test_timestamp_utils.py@@ -22,7 +22,6 @@ from posthog.schema import ActionsNode, DataWarehouseNode, DateRange, EventsNodefrom posthog.hogql.query import execute_hogql_queryfrom posthog.clickhouse.query_tagging import Feature, Product, get_query_tags, tags_context-from posthog.clickhouse.workload import Workloadfrom posthog.hogql_queries.utils.query_date_range import QueryDateRangefrom posthog.hogql_queries.utils.timestamp_utils import (EARLIEST_EVENT_TIMESTAMP,@@ -593,8 +592,8 @@ class TestEarliestTimestampRouting(BaseTest):cache.clear()@parameterized.expand([("series",), ("unfiltered",)])- @override_settings(CLICKHOUSE_HOGQL_USE_NEW_EVENTS_SCHEMA=False, CLICKHOUSE_NATIVE_EVENTS_HOST="native.example.com")- def test_native_route_reads_the_native_cluster_and_skips_the_legacy_cache(self, lookup: str) -> None:+ @override_settings(CLICKHOUSE_HOGQL_USE_NEW_EVENTS_SCHEMA=False)+ def test_native_route_reads_events_json_and_skips_the_legacy_cache(self, lookup: str) -> None:node = EventsNode(event="$pageview")legacy = datetime.datetime(2022, 1, 1, tzinfo=ZoneInfo("UTC"))native = datetime.datetime(2020, 1, 1, tzinfo=ZoneInfo("UTC"))@@ -608,19 +607,14 @@ class TestEarliestTimestampRouting(BaseTest):return get_earliest_timestamp_from_series(self.team, [node])return get_earliest_timestamp_unfiltered(self.team)- with (- override_instance_config("CLICKHOUSE_NATIVE_EVENTS_HOST", "native.example.com"),- override_instance_config("CLICKHOUSE_NATIVE_EVENTS_TEAMS", [self.team.pk]),- patch(- "posthog.hogql.query.sync_execute", return_value=([(native,)], [("timestamp", "DateTime")])- ) as execute,- ):- with events_route_scope(HogQLQueryModifiers(useNewEventsCluster=True, useNewEventsSchema=True)):+ with patch(+ "posthog.hogql.query.sync_execute", return_value=([(native,)], [("timestamp", "DateTime")])+ ) as execute:+ with events_route_scope(HogQLQueryModifiers(useNewEventsSchema=True)):self.assertEqual(earliest(), native)self.assertEqual(earliest(), native)self.assertEqual(earliest(), legacy)execute.assert_called_once()- self.assertEqual(execute.call_args.kwargs["workload"], Workload.NEW_EVENTS_CLUSTER)self.assertIn("events_json", execute.call_args.args[0])def test_route_keys_never_collide_with_legacy_event_keys(self) -> None:@@ -635,17 +629,21 @@ class TestEarliestTimestampRouting(BaseTest):):self.assertEqual(get_earliest_timestamp_from_series(self.team, [EventsNode(event=None)]), native)- @parameterized.expand([("allowlisted_project", True, {}), ("project_setting", False, {"useNewEventsSchema": True})])+ @parameterized.expand(+ [+ ("allowlisted_project", True, {}, True),+ ("stored_project_modifier_is_ignored", False, {"useNewEventsSchema": True}, False),+ ]+ )@override_settings(CLICKHOUSE_HOGQL_USE_NEW_EVENTS_SCHEMA=False)def test_lookup_outside_a_run_keys_on_the_events_table_it_reads(- self, _name: str, allowlisted: bool, team_modifiers: dict[str, bool]+ self, _name: str, allowlisted: bool, team_modifiers: dict[str, bool], reads_events_json: bool) -> None:self.team.modifiers = team_modifiersnode = EventsNode(event="$pageview")- cache.set(- _get_earliest_timestamp_cache_key(self.team, node), datetime.datetime(2022, 1, 1, tzinfo=ZoneInfo("UTC"))- )+ legacy = datetime.datetime(2022, 1, 1, tzinfo=ZoneInfo("UTC"))native = datetime.datetime(2020, 1, 1, tzinfo=ZoneInfo("UTC"))+ cache.set(_get_earliest_timestamp_cache_key(self.team, node), legacy)with (override_instance_config("CLICKHOUSE_HOGQL_USE_NEW_EVENTS_SCHEMA_TEAMS", str(self.team.pk) if allowlisted else ""@@ -654,5 +652,10 @@ class TestEarliestTimestampRouting(BaseTest):"posthog.hogql.query.sync_execute", return_value=([(native,)], [("timestamp", "DateTime")])) as execute,):- self.assertEqual(get_earliest_timestamp_from_series(self.team, [node]), native)- self.assertIn("events_json", execute.call_args.args[0])+ earliest = get_earliest_timestamp_from_series(self.team, [node])+ if reads_events_json:+ self.assertEqual(earliest, native)+ self.assertIn("events_json", execute.call_args.args[0])+ else:+ self.assertEqual(earliest, legacy)+ execute.assert_not_called()diff --git a/posthog/hogql_queries/utils/timestamp_utils.py b/posthog/hogql_queries/utils/timestamp_utils.pyindex 6419bf78f4e1..d2ec736b46db 100644--- a/posthog/hogql_queries/utils/timestamp_utils.py+++ b/posthog/hogql_queries/utils/timestamp_utils.py@@ -33,7 +33,7 @@ from posthog.clickhouse.query_tagging import Feature, Product, get_query_tags, tfrom posthog.hogql_queries.utils.query_date_range import QueryDateRangefrom posthog.models import Team, Userfrom posthog.models.event import DEFAULT_EARLIEST_TIME_DELTA-from posthog.models.event.new_events_schema import get_events_route, get_new_events_cluster_host, use_new_events_schema+from posthog.models.event.new_events_schema import get_events_route, use_new_events_schemafrom posthog.models.team import WeekStartDayfrom posthog.utils import get_safe_cache@@ -143,14 +143,11 @@ def _events_route_modifiers(team: Team) -> HogQLQueryModifiers:def _events_route_cache_key(team: Team, modifiers: HogQLQueryModifiers, cache_key: str) -> str:- # Each events table and cluster holds its own copy of the events, so one route's result must not answer another.- # The route goes in front, because legacy keys end in event names and table names that a suffix could collide with.- native_table = use_new_events_schema(team.pk, modifiers)- native_cluster = bool(modifiers.useNewEventsCluster)- if not (native_table or native_cluster):+ # Each events table holds its own copy of the events, so one table's result must not answer for the other.+ # The table goes in front, because legacy keys end in event names and table names that a suffix could collide with.+ if not use_new_events_schema(team.pk, modifiers):return cache_key- native_host = get_new_events_cluster_host() if native_cluster else None- return f"events_route:{native_table}:{native_host}:{cache_key}"+ return f"events_json:{cache_key}"def _coerce_to_datetime(value: Any, tz: tzinfo) -> datetime:diff --git a/posthog/middleware.py b/posthog/middleware.pyindex 1050eca2a44d..ec7fc297e7dd 100644--- a/posthog/middleware.py+++ b/posthog/middleware.py@@ -57,6 +57,7 @@ from posthog.models.activity_logging.utils import (client_from_header,record_agent_intent,)+from posthog.models.event.new_events_schema import NEW_EVENTS_SCHEMA_COOKIE, new_events_schema_for_requestfrom posthog.session.activity import session_activity_credentialfrom posthog.settings import PROJECT_SWITCHING_TOKEN_ALLOWLIST, SITE_URLfrom posthog.user_permissions import UserPermissions@@ -1227,6 +1228,27 @@ class AutoLogoutImpersonateMiddleware:return self.get_response(request)+class NewEventsSchemaCookieMiddleware:+ """A staff person's requests read the native JSON events table while their browser carries the cookie.++ The switch is per person, so one person can check a project on events_json, an impersonated one included,+ without switching the project for everyone. An impersonated session counts as staff because only staff can+ start one.+ """++ def __init__(self, get_response: Callable[[HttpRequest], HttpResponse]) -> None:+ self.get_response = get_response++ def __call__(self, request: HttpRequest) -> HttpResponse:+ if request.COOKIES.get(NEW_EVENTS_SCHEMA_COOKIE) != "1":+ return self.get_response(request)+ user = cast(User, request.user)+ if not (user.is_staff or is_impersonated_session(request)):+ return self.get_response(request)+ with new_events_schema_for_request():+ return self.get_response(request)++class UnversionedStaticPathMiddleware:"""Serves /static/<version>/<file> as /static/<file>.diff --git a/posthog/models/event/new_events_schema.py b/posthog/models/event/new_events_schema.pyindex aae7ebbe24b4..d8206ab206a2 100644--- a/posthog/models/event/new_events_schema.py+++ b/posthog/models/event/new_events_schema.py@@ -11,10 +11,8 @@ if TYPE_CHECKING:from posthog.schema import HogQLQueryModifiers-# TODO: Remove the per-query rollout option, and the route helpers in this block, after the events cluster-# migration is complete.-EVENTS_ROUTE_MODIFIER_KEYS = ("useNewEventsSchema", "useNewEventsCluster")-_query_host: ContextVar[list[str | None] | None] = ContextVar("new_events_cluster_host", default=None)+# TODO: Remove the route helpers in this block after the events cluster migration is complete.+EVENTS_ROUTE_MODIFIER_KEYS = ("useNewEventsSchema",)_query_route: ContextVar["HogQLQueryModifiers | None"] = ContextVar("new_events_query_route", default=None)@@ -54,10 +52,7 @@ def with_pinned_events_route(query: dict[str, Any], modifiers: "HogQLQueryModifiif "modifiers" not in _copy_source_chain(query)[-1]:return querynodes = _copy_source_chain(strip_events_route_modifiers(query))- route: dict[str, Any] = {"useNewEventsSchema": modifiers.useNewEventsSchema}- if modifiers.useNewEventsCluster is not None:- route["useNewEventsCluster"] = modifiers.useNewEventsCluster- nodes[-1]["modifiers"] = {**(nodes[-1]["modifiers"] or {}), **route}+ nodes[-1]["modifiers"] = {**(nodes[-1]["modifiers"] or {}), "useNewEventsSchema": modifiers.useNewEventsSchema}return nodes[0]@@ -70,80 +65,59 @@ def events_route_cache_modifiers(modifiers: dict[str, Any], *, reads_events: boomodifiers = dict(modifiers)if modifiers.get("useNewEventsSchema") is False or not reads_events:modifiers.pop("useNewEventsSchema", None)- if not modifiers.get("useNewEventsCluster"):- modifiers.pop("useNewEventsCluster", None)return modifiers-def get_new_events_cluster_host() -> str | None:- snapshot = _query_host.get()- if snapshot:- return snapshot[0]- value = get_instance_setting("CLICKHOUSE_NATIVE_EVENTS_HOST")- host = value if isinstance(value, str) and value else None- if snapshot is not None:- snapshot.append(host)- return host---@contextmanager-def new_events_cluster_host_scope() -> Iterator[None]:- # Pin cache lookup and execution to one host even if settings change mid-request.- if _query_host.get() is not None:- yield- return- token = _query_host.set([])- try:- yield- finally:- _query_host.reset(token)--def get_events_route() -> "HogQLQueryModifiers | None":- """The events cluster and table of the query run in progress, or None outside a query run."""+ """The events table of the query run in progress, or None outside a query run."""return _query_route.get()@contextmanagerdef events_route_scope(modifiers: "HogQLQueryModifiers") -> Iterator[None]:- # Lookups inside a run, such as an all-time range's earliest event, must read the query's cluster and table.+ # Lookups inside a run, such as an all-time range's earliest event, must read the query's events table.from posthog.schema import HogQLQueryModifiers # noqa: PLC0415- token = _query_route.set(- HogQLQueryModifiers(- useNewEventsCluster=modifiers.useNewEventsCluster, useNewEventsSchema=modifiers.useNewEventsSchema- )- )+ token = _query_route.set(HogQLQueryModifiers(useNewEventsSchema=modifiers.useNewEventsSchema))try:yieldfinally:_query_route.reset(token)-def use_new_events_cluster(team_id: int | None, modifiers: "HogQLQueryModifiers | None") -> bool:- from posthog.hogql.errors import ExposedHogQLError # noqa: PLC0415+# A staff person's browser sends this cookie to read events_json on every request, impersonated sessions included.+NEW_EVENTS_SCHEMA_COOKIE = "ph_new_events_schema"+_person_switch: ContextVar[bool] = ContextVar("new_events_schema_person_switch", default=False)+- if modifiers is None or not modifiers.useNewEventsCluster:- return False- if not get_new_events_cluster_host():- raise ExposedHogQLError("The native events cluster host is not configured.")- allowed = get_instance_setting("CLICKHOUSE_NATIVE_EVENTS_TEAMS")- if not isinstance(allowed, list) or any(type(value) is not int for value in allowed) or team_id not in allowed:- raise ExposedHogQLError("This project is not enabled for native events cluster queries.")- return True+@contextmanager+def new_events_schema_for_request() -> Iterator[None]:+ """Every events read inside the block targets events_json, unless its query's modifier says otherwise."""+ token = _person_switch.set(True)+ try:+ yield+ finally:+ _person_switch.reset(token)def use_new_events_schema(team_id: int | None = None, modifiers: "HogQLQueryModifiers | None" = None) -> bool:- """Whether HogQL reads should target the native-JSON events tables.-- A query's `useNewEventsSchema` modifier decides when it is set, so one query can be run against either table.- Otherwise the instance setting flips it globally at runtime, and the *_TEAMS instance setting enables it- for individual teams first (both cached for up to 60s per worker). Code running inside a query- should prefer HogQLContext.uses_new_events_schema(), which resolves this once so a mid-query- flip can't mix schemas.+ """Whether events reads target the native JSON events table (events_json) instead of the legacy events table.++ Four switches, each wider than the one before it: the query's useNewEventsSchema modifier, for one query;+ a staff person's cookie, for every request from that browser (see new_events_schema_for_request);+ CLICKHOUSE_HOGQL_USE_NEW_EVENTS_SCHEMA_TEAMS, for a project everywhere, background jobs included; and+ CLICKHOUSE_HOGQL_USE_NEW_EVENTS_SCHEMA, for every project. The instance settings are cached for up to 60s+ per worker. Nested reads inherit the running query's choice unless their own modifier overrides it.+ Code running inside a query should prefer HogQLContext.uses_new_events_schema(), which+ resolves this once so a mid-query flip can't mix schemas."""if modifiers is not None and modifiers.useNewEventsSchema is not None:return modifiers.useNewEventsSchema+ route = get_events_route()+ if route is not None and route.useNewEventsSchema is not None:+ return route.useNewEventsSchema+ if _person_switch.get():+ return Trueif settings.TEST and settings.CLICKHOUSE_HOGQL_USE_NEW_EVENTS_SCHEMA:return Trueif get_instance_setting("CLICKHOUSE_HOGQL_USE_NEW_EVENTS_SCHEMA"):diff --git a/posthog/schema.py b/posthog/schema.pyindex 3c9bd336c5e9..bd1e9d04998e 100644--- a/posthog/schema.py+++ b/posthog/schema.py@@ -6369,25 +6369,15 @@ class HogQLQueryModifiers(BaseModel):),)useMaterializedViews: bool | None = None- useNewEventsCluster: bool | None = Field(- default=None,- description=(- "Read this query from the native events cluster and native JSON events"- " table (`true`), or keep its existing connection (`false` or unset). This"- " is an internal rollout switch for eligible projects. PostHog staff manage"- " eligibility through instance settings. Each query must opt in; project"- " defaults cannot enable it. Only synchronous HogQLQuery SQL comparisons"- " are supported."- ),- )useNewEventsSchema: bool | None = Field(default=None,description=("Read events from the native JSON events table (`true`) or the legacy"- " events table (`false`). When unset, the project's stored value applies,"- " then the `CLICKHOUSE_HOGQL_USE_NEW_EVENTS_SCHEMA` instance settings. This"- " is an internal rollout switch. PostHog staff set the project value in"- " Django admin and the project settings API ignores it."+ " events table (`false`). When unset, a staff person's"+ " `ph_new_events_schema` cookie applies, then the project's listing in the"+ " `CLICKHOUSE_HOGQL_USE_NEW_EVENTS_SCHEMA_TEAMS` instance setting, then the"+ " `CLICKHOUSE_HOGQL_USE_NEW_EVENTS_SCHEMA` instance setting. This is an"+ " internal rollout switch, and the project settings API ignores it."),)usePreaggregatedIntermediateResults: bool | None = Nonediff --git a/posthog/settings/dynamic_settings.py b/posthog/settings/dynamic_settings.pyindex 23a4fe782da3..8f1307e28d8e 100644--- a/posthog/settings/dynamic_settings.py+++ b/posthog/settings/dynamic_settings.py@@ -64,24 +64,14 @@ CONSTANCE_CONFIG = {),"CLICKHOUSE_HOGQL_USE_NEW_EVENTS_SCHEMA": (False,- "Whether HogQL queries read from the native-JSON events tables (events_json) instead of the legacy events table.",+ "Every project reads the native JSON events table (events_json) instead of the legacy events table. The widest of the switches: a query's useNewEventsSchema modifier, a staff person's ph_new_events_schema cookie and CLICKHOUSE_HOGQL_USE_NEW_EVENTS_SCHEMA_TEAMS each switch less.",bool,),"CLICKHOUSE_HOGQL_USE_NEW_EVENTS_SCHEMA_TEAMS": ("",- "Comma-separated team IDs whose HogQL queries read from the native-JSON events tables even while the global setting is off.",+ "Comma-separated project IDs that read the native JSON events table (events_json) in insights, SQL, cohorts and the background jobs that run per project. Changes reach workers within about a minute.",str,),- "CLICKHOUSE_NATIVE_EVENTS_HOST": (- get_from_env("CLICKHOUSE_NATIVE_EVENTS_HOST", ""),- "Hostname for explicit new events cluster queries, without a scheme, port or path. Empty disables the route. Changes reach workers within about a minute.",- str,- ),- "CLICKHOUSE_NATIVE_EVENTS_TEAMS": (- [],- "Project IDs eligible for explicit native events cluster query opt-ins. This does not change query defaults.",- list[int],- ),"PERSON_ON_EVENTS_V2_ENABLED": (# Person-on-events v2 reads are the production default for every project created# since 2024-06-14, and both CI lanes set this variable so their runs match it. A@@ -457,8 +447,6 @@ SETTINGS_ALLOWING_API_OVERRIDE = ("PERSON_ON_EVENTS_V2_ENABLED","CLICKHOUSE_HOGQL_USE_NEW_EVENTS_SCHEMA","CLICKHOUSE_HOGQL_USE_NEW_EVENTS_SCHEMA_TEAMS",- "CLICKHOUSE_NATIVE_EVENTS_HOST",- "CLICKHOUSE_NATIVE_EVENTS_TEAMS","STRICT_CACHING_TEAMS","GITHUB_APP_SLUG","SLACK_APP_CLIENT_ID",diff --git a/posthog/settings/web.py b/posthog/settings/web.pyindex 2cd8ae503637..5cfaa0e7632e 100644--- a/posthog/settings/web.py+++ b/posthog/settings/web.py@@ -231,6 +231,7 @@ MIDDLEWARE = ["posthog.middleware.AutoLogoutImpersonateMiddleware","posthog.middleware.ImpersonationReadOnlyMiddleware","posthog.middleware.ImpersonationBlockedPathsMiddleware",+ "posthog.middleware.NewEventsSchemaCookieMiddleware","django.middleware.clickjacking.XFrameOptionsMiddleware","posthog.middleware.CsvNeverCacheMiddleware","axes.middleware.AxesMiddleware",diff --git a/products/analytics_platform/backend/lazy_computation/lazy_computation_executor.py b/products/analytics_platform/backend/lazy_computation/lazy_computation_executor.pyindex 41a5e3243e68..a9a542609ae5 100644--- a/products/analytics_platform/backend/lazy_computation/lazy_computation_executor.py+++ b/products/analytics_platform/backend/lazy_computation/lazy_computation_executor.py@@ -37,6 +37,7 @@ from posthog.clickhouse.client import sync_executefrom posthog.clickhouse.preaggregation.sql import DISTRIBUTED_PREAGGREGATION_RESULTS_TABLEfrom posthog.clickhouse.query_tagging import tags_contextfrom posthog.dataclasses import frozen+from posthog.models.event.new_events_schema import use_new_events_schemafrom posthog.models.team import Teamfrom posthog.settings import DEBUG, HOGQL_INCREASED_MAX_EXECUTION_TIME, TESTfrom posthog.utils import relative_date_parse_with_delta_mapping@@ -619,6 +620,9 @@ class LazyComputationQuery:timezone: str = "UTC"breakdown_fields: list[str] = field(default_factory=list)cache_key_context: dict[str, str] | None = None+ # The modifiers of the query that made the job, so an inline build prints against the events table the+ # identity names rather than the one the building request's own switches would pick.+ modifiers: HogQLQueryModifiers | None = None@dataclass(frozen=True)@@ -657,6 +661,19 @@ def _oldest_computed_at(jobs: list[PreaggregationJob]) -> datetime | None:return min(stamps) if stamps else None+def events_table_cache_key_context(+ team: Team, modifiers: HogQLQueryModifiers | None, cache_key_context: dict[str, str] | None+) -> dict[str, str] | None:+ """The job identity context with the events table the build reads, when it is not the legacy one.++ The hashed AST names the `events` table whichever ClickHouse table it prints to, so without this a build from+ events_json would be served to readers of the legacy table and the other way round.+ """+ if not use_new_events_schema(team.pk, modifiers):+ return cache_key_context+ return {**(cache_key_context or {}), "events_table": "events_json"}++def compute_query_hash(query_info: LazyComputationQuery) -> str:"""Compute a stable hash for a LazyComputationQuery object.@@ -938,6 +955,7 @@ def build_lazy_computation_insert_sql(time_range_start: datetime,time_range_end: datetime,expires_at: datetime,+ modifiers: HogQLQueryModifiers | None = None,) -> tuple[str, dict]:"""Build the INSERT ... SELECT SQL for populating lazy-computed results.@@ -994,7 +1012,13 @@ def build_lazy_computation_insert_sql(query.where = date_range_filter# Print the SELECT query to ClickHouse SQL- context = HogQLContext(team_id=team.id, team=team, enable_select_queries=True, limit_top_select=False)+ context = HogQLContext(+ team_id=team.id,+ team=team,+ enable_select_queries=True,+ limit_top_select=False,+ modifiers=modifiers if modifiers is not None else HogQLQueryModifiers(),+ )select_sql, _ = prepare_and_print_ast(query,context=context,@@ -1043,6 +1067,7 @@ def run_lazy_computation_insert(time_range_start=job.time_range_start,time_range_end=job.time_range_end,expires_at=ch_expires_at,+ modifiers=query_info.modifiers,)set_ch_query_started(job.id)@@ -1810,6 +1835,11 @@ def ensure_precomputed(}parsed_for_hash = _resolve_insert_query(insert_query, hash_placeholders)+ # The INSERT's printer asks the same switches for its table, so naming the table here keeps the job identity+ # and the rows under it in step, whichever switch (a modifier, the person's cookie, the project or the+ # instance) picked it.+ cache_key_context = events_table_cache_key_context(team, modifiers, cache_key_context)+query_info = LazyComputationQuery(query=parsed_for_hash,table=table,diff --git a/products/analytics_platform/backend/lazy_computation/lazy_computation_transformer.py b/products/analytics_platform/backend/lazy_computation/lazy_computation_transformer.pyindex 5317ce025d4e..ff838b5c2e03 100644--- a/products/analytics_platform/backend/lazy_computation/lazy_computation_transformer.py+++ b/products/analytics_platform/backend/lazy_computation/lazy_computation_transformer.py@@ -1,6 +1,8 @@from datetime import datetime, timedeltafrom typing import Optional, Union+from posthog.schema import HogQLQueryModifiers+from posthog.hogql import astfrom posthog.hogql.ast import CompareOperationOpfrom posthog.hogql.context import HogQLContext@@ -15,6 +17,7 @@ from products.analytics_platform.backend.lazy_computation.lazy_computation_execuLazyComputationQuery,LazyComputationResult,LazyComputationTable,+ events_table_cache_key_context,)PREAGGREGATED_DAILY_UNIQUE_PERSONS_PAGEVIEWS_TABLE_NAME = DISTRIBUTED_PREAGGREGATION_RESULTS_TABLE()@@ -458,6 +461,7 @@ def _run_daily_unique_persons_pageviews(query_to_insert: ast.SelectQuery,start: datetime,end: datetime,+ modifiers: HogQLQueryModifiers,) -> LazyComputationResult:"""Orchestrate lazy computation jobs for daily unique persons pageviews.@@ -468,7 +472,11 @@ def _run_daily_unique_persons_pageviews(3. Returns the result with job IDs for the combiner query"""query_info = LazyComputationQuery(- query=query_to_insert, table=LazyComputationTable.PREAGGREGATION_RESULTS, timezone=team.timezone+ query=query_to_insert,+ table=LazyComputationTable.PREAGGREGATION_RESULTS,+ timezone=team.timezone,+ cache_key_context=events_table_cache_key_context(team, modifiers, None),+ modifiers=modifiers,)executor = LazyComputationExecutor()@@ -514,7 +522,7 @@ class Transformer(CloningVisitor):team = self.context.teamif not team:return transformed_node- result = _run_daily_unique_persons_pageviews(team, query_to_insert, start_dt, end_dt)+ result = _run_daily_unique_persons_pageviews(team, query_to_insert, start_dt, end_dt, self.context.modifiers)if not result.ready:# Lazy computation not ready, fall back to original querydiff --git a/products/analytics_platform/backend/lazy_computation/tests/test_lazy_computation_executor.py b/products/analytics_platform/backend/lazy_computation/tests/test_lazy_computation_executor.pyindex be702d58c235..c990f26c2b4a 100644--- a/products/analytics_platform/backend/lazy_computation/tests/test_lazy_computation_executor.py+++ b/products/analytics_platform/backend/lazy_computation/tests/test_lazy_computation_executor.py@@ -7,7 +7,7 @@ from posthog.test.base import BaseTest, ClickhouseTestMixin, _create_eventfrom unittest.mock import patchfrom django.db import IntegrityError-from django.test import SimpleTestCase+from django.test import SimpleTestCase, override_settingsfrom django.utils import timezone as django_timezonefrom clickhouse_driver.errors import ServerException@@ -21,6 +21,8 @@ from posthog.hogql.query import execute_hogql_queryfrom posthog.clickhouse.client import sync_executefrom posthog.clickhouse.preaggregation.sql import DISTRIBUTED_PREAGGREGATION_RESULTS_TABLE+from posthog.models.event.new_events_schema import new_events_schema_for_request+from posthog.models.instance_setting import override_instance_configfrom posthog.settings import HOGQL_INCREASED_MAX_EXECUTION_TIMEfrom products.analytics_platform.backend.lazy_computation.computation_notifications import (@@ -3929,3 +3931,72 @@ class TestCheckOnlyMode(BaseTest):result = self._execute(datetime(2024, 1, 1, tzinfo=UTC), datetime(2024, 1, 3, tzinfo=UTC))assert result.ready is Falseassert time_mod.monotonic() - started < 5, "check-only must not block on pending jobs"+++class TestBuildsKeepTheirEventsTable(BaseTest):+ INSERT_QUERY = TestInsertSettingsAppliedToInserts.INSERT_QUERY++ @override_settings(CLICKHOUSE_HOGQL_USE_NEW_EVENTS_SCHEMA=False)+ def test_a_switched_persons_build_gets_a_job_of_its_own(self) -> None:+ with (+ override_instance_config("CLICKHOUSE_HOGQL_USE_NEW_EVENTS_SCHEMA", False),+ patch(+ "products.analytics_platform.backend.lazy_computation.lazy_computation_executor.sync_execute",+ return_value=0,+ ) as execute,+ ):+ legacy = ensure_precomputed(+ team=self.team,+ insert_query=self.INSERT_QUERY,+ time_range_start=datetime(2024, 1, 1, tzinfo=UTC),+ time_range_end=datetime(2024, 1, 2, tzinfo=UTC),+ ttl_seconds=7 * 24 * 60 * 60,+ )+ assert "events_json" not in execute.call_args.args[0]+ with new_events_schema_for_request():+ native = ensure_precomputed(+ team=self.team,+ insert_query=self.INSERT_QUERY,+ time_range_start=datetime(2024, 1, 1, tzinfo=UTC),+ time_range_end=datetime(2024, 1, 2, tzinfo=UTC),+ ttl_seconds=7 * 24 * 60 * 60,+ )+ assert "events_json" in execute.call_args.args[0]+ # The two builds read different tables, so neither may be served the other's rows.+ assert set(legacy.job_ids).isdisjoint(native.job_ids)++ @override_settings(CLICKHOUSE_HOGQL_USE_NEW_EVENTS_SCHEMA=False)+ def test_an_ast_build_prints_the_table_its_job_identity_names(self) -> None:+ job = PreaggregationJob.objects.create(+ team=self.team,+ query_hash="test_hash",+ time_range_start=datetime(2024, 1, 1, tzinfo=UTC),+ time_range_end=datetime(2024, 1, 2, tzinfo=UTC),+ status=PreaggregationJob.Status.PENDING,+ expires_at=django_timezone.now() + timedelta(days=7),+ )+ query = parse_select(+ self.INSERT_QUERY,+ placeholders={+ "time_window_min": ast.Constant(value=datetime(2024, 1, 1, tzinfo=UTC)),+ "time_window_max": ast.Constant(value=datetime(2024, 1, 2, tzinfo=UTC)),+ },+ )+ assert isinstance(query, ast.SelectQuery)+ # The query that made the job pinned the legacy table, so the building person's cookie must not change+ # what the build reads.+ query_info = LazyComputationQuery(+ query=query,+ table=LazyComputationTable.PREAGGREGATION_RESULTS,+ modifiers=HogQLQueryModifiers(useNewEventsSchema=False),+ )+ with (+ override_instance_config("CLICKHOUSE_HOGQL_USE_NEW_EVENTS_SCHEMA", False),+ new_events_schema_for_request(),+ patch(+ "products.analytics_platform.backend.lazy_computation.lazy_computation_executor.sync_execute",+ return_value=0,+ ) as execute,+ ):+ run_lazy_computation_insert(self.team, job, query_info)+ assert "events_json" not in execute.call_args.args[0]diff --git a/products/batch_exports/backend/presentation/views/utils.py b/products/batch_exports/backend/presentation/views/utils.pyindex 74a3bc080fa6..7393e8a57378 100644--- a/products/batch_exports/backend/presentation/views/utils.py+++ b/products/batch_exports/backend/presentation/views/utils.py@@ -90,10 +90,6 @@ class HogQLModifiersField(serializers.JSONField):messages.append(f"{location}: {error['msg']}" if location else error["msg"])raise serializers.ValidationError(messages) from eelse:- if modifiers.useNewEventsCluster:- raise serializers.ValidationError(- "Native events cluster queries are only supported by the synchronous query API"- )self.loaded = modifiersreturn modifiers.model_dump(mode="json", exclude_none=True)diff --git a/products/batch_exports/backend/tests/api/test_create.py b/products/batch_exports/backend/tests/api/test_create.pyindex 52caacce1e07..c84f38794021 100644--- a/products/batch_exports/backend/tests/api/test_create.py+++ b/products/batch_exports/backend/tests/api/test_create.py@@ -748,11 +748,6 @@ def test_create_batch_export_with_hogql_model_allows_query_without_placeholders("hogql_modifiers","notAModifier: Extra inputs are not permitted",),- (- {"hogql_modifiers": {"useNewEventsCluster": True}},- "hogql_modifiers",- "Native events cluster queries are only supported by the synchronous query API",- ),({"model": "events", "hogql_modifiers": {"convertToProjectTimezone": False}},"hogql_modifiers",@@ -766,7 +761,6 @@ def test_create_batch_export_with_hogql_model_allows_query_without_placeholders("unaliased-column","filters","unknown-modifier",- "native-cluster-modifier","modifiers-with-events-model",],)diff --git a/products/batch_exports/backend/tests/api/test_serializers.py b/products/batch_exports/backend/tests/api/test_serializers.pyindex df2f4769e20b..b78684b014c1 100644--- a/products/batch_exports/backend/tests/api/test_serializers.py+++ b/products/batch_exports/backend/tests/api/test_serializers.py@@ -23,7 +23,6 @@ from products.batch_exports.backend.presentation.views.batch_export.exports impoBatchExportSerializer,parse_events_hogql_query,)-from products.batch_exports.backend.presentation.views.utils import HogQLModifiersFieldclass TestBatchExportModelValidation(SimpleTestCase):@@ -50,23 +49,6 @@ class TestBatchExportModelValidation(SimpleTestCase):assert serializer.is_valid(), serializer.errorsassert serializer.validated_data == {"name": "Renamed"}- @parameterized.expand(- [- ("unset", {}, {}),- ("false", {"useNewEventsCluster": False}, {"useNewEventsCluster": False}),- ("true", {"useNewEventsCluster": True}, None),- ]- )- def test_cluster_opt_ins_are_not_supported_by_exports(- self, _name: str, modifiers: dict, expected: dict | None- ) -> None:- field = HogQLModifiersField()- if expected is None:- with self.assertRaisesMessage(serializers.ValidationError, "synchronous query API"):- field.run_validation(modifiers)- else:- assert field.run_validation(modifiers) == expected-def prepare_query(query: str, team_id: int) -> ast.SelectQuery:"""Parse and resolve a HogQL query string into a prepared AST."""diff --git a/products/batch_exports/backend/tests/temporal/destinations/file_download/test_file_download_api.py b/products/batch_exports/backend/tests/temporal/destinations/file_download/test_file_download_api.pyindex d854c7df364f..6361d200aeb4 100644--- a/products/batch_exports/backend/tests/temporal/destinations/file_download/test_file_download_api.py+++ b/products/batch_exports/backend/tests/temporal/destinations/file_download/test_file_download_api.py@@ -826,11 +826,6 @@ class TestFileDownloadHogQL:"'data_interval_start' and 'data_interval_end' are required",id="events-model-missing-intervals",),- pytest.param(- {"hogql_modifiers": {"useNewEventsCluster": True}},- "synchronous query API",- id="native-cluster-modifier",- ),pytest.param({"model": "events", "hogql_query": None, "hogql_modifiers": {"convertToProjectTimezone": False}},"'hogql_modifiers' are only supported when 'model' is 'hogql'",@@ -1256,15 +1251,6 @@ class TestFileDownloadHogQL:"Extra inputs are not permitted",id="unknown-modifier",),- pytest.param(- {- "model": "hogql",- "hogql_query": "SELECT event AS event FROM events",- "hogql_modifiers": {"useNewEventsCluster": True},- },- "synchronous query API",- id="native-cluster-modifier",- ),],)@pytest.mark.usefixtures("enable_hogql_flag")diff --git a/products/batch_exports/frontend/generated/api.schemas.ts b/products/batch_exports/frontend/generated/api.schemas.tsindex 05836f264e9c..ce0051bdd3d7 100644--- a/products/batch_exports/frontend/generated/api.schemas.ts+++ b/products/batch_exports/frontend/generated/api.schemas.ts@@ -861,9 +861,7 @@ export interface HogQLQueryModifiersApi {/** Remove provably redundant casts and nullability wrappers (e.g. `toString(String)`, `assumeNotNull(non_nullable)`, dead `ifNull` fallbacks) using inferred expression types */typeAwareCastSimplification?: boolean | nulluseMaterializedViews?: boolean | null- /** Read this query from the native events cluster and native JSON events table (`true`), or keep its existing connection (`false` or unset). This is an internal rollout switch for eligible projects. PostHog staff manage eligibility through instance settings. Each query must opt in; project defaults cannot enable it. Only synchronous HogQLQuery SQL comparisons are supported. */- useNewEventsCluster?: boolean | null- /** Read events from the native JSON events table (`true`) or the legacy events table (`false`). When unset, the project's stored value applies, then the `CLICKHOUSE_HOGQL_USE_NEW_EVENTS_SCHEMA` instance settings. This is an internal rollout switch. PostHog staff set the project value in Django admin and the project settings API ignores it. */+ /** Read events from the native JSON events table (`true`) or the legacy events table (`false`). When unset, a staff person's `ph_new_events_schema` cookie applies, then the project's listing in the `CLICKHOUSE_HOGQL_USE_NEW_EVENTS_SCHEMA_TEAMS` instance setting, then the `CLICKHOUSE_HOGQL_USE_NEW_EVENTS_SCHEMA` instance setting. This is an internal rollout switch, and the project settings API ignores it. */useNewEventsSchema?: boolean | nullusePreaggregatedIntermediateResults?: boolean | null/** Try to automatically convert HogQL queries to use preaggregated tables at the AST level * */diff --git a/products/batch_exports/frontend/generated/api.zod.ts b/products/batch_exports/frontend/generated/api.zod.tsindex 24bfb150c0db..42ebb894ac41 100644--- a/products/batch_exports/frontend/generated/api.zod.ts+++ b/products/batch_exports/frontend/generated/api.zod.ts@@ -669,17 +669,11 @@ export const BatchExportsCreateBody = /* @__PURE__ */ zod'Remove provably redundant casts and nullability wrappers (e.g. `toString(String)`, `assumeNotNull(non_nullable)`, dead `ifNull` fallbacks) using inferred expression types'),useMaterializedViews: zod.union([zod.boolean(), zod.null()]).optional(),- useNewEventsCluster: zod- .union([zod.boolean(), zod.null()])- .optional()- .describe(- 'Read this query from the native events cluster and native JSON events table (`true`), or keep its existing connection (`false` or unset). This is an internal rollout switch for eligible projects. PostHog staff manage eligibility through instance settings. Each query must opt in; project defaults cannot enable it. Only synchronous HogQLQuery SQL comparisons are supported.'- ),useNewEventsSchema: zod.union([zod.boolean(), zod.null()]).optional().describe(- "Read events from the native JSON events table (`true`) or the legacy events table (`false`). When unset, the project's stored value applies, then the `CLICKHOUSE_HOGQL_USE_NEW_EVENTS_SCHEMA` instance settings. This is an internal rollout switch. PostHog staff set the project value in Django admin and the project settings API ignores it."+ "Read events from the native JSON events table (`true`) or the legacy events table (`false`). When unset, a staff person's `ph_new_events_schema` cookie applies, then the project's listing in the `CLICKHOUSE_HOGQL_USE_NEW_EVENTS_SCHEMA_TEAMS` instance setting, then the `CLICKHOUSE_HOGQL_USE_NEW_EVENTS_SCHEMA` instance setting. This is an internal rollout switch, and the project settings API ignores it."),usePreaggregatedIntermediateResults: zod.union([zod.boolean(), zod.null()]).optional(),usePreaggregatedTableTransforms: zod@@ -1655,17 +1649,11 @@ export const BatchExportsUpdateBody = /* @__PURE__ */ zod'Remove provably redundant casts and nullability wrappers (e.g. `toString(String)`, `assumeNotNull(non_nullable)`, dead `ifNull` fallbacks) using inferred expression types'),useMaterializedViews: zod.union([zod.boolean(), zod.null()]).optional(),- useNewEventsCluster: zod- .union([zod.boolean(), zod.null()])- .optional()- .describe(- 'Read this query from the native events cluster and native JSON events table (`true`), or keep its existing connection (`false` or unset). This is an internal rollout switch for eligible projects. PostHog staff manage eligibility through instance settings. Each query must opt in; project defaults cannot enable it. Only synchronous HogQLQuery SQL comparisons are supported.'- ),useNewEventsSchema: zod.union([zod.boolean(), zod.null()]).optional().describe(- "Read events from the native JSON events table (`true`) or the legacy events table (`false`). When unset, the project's stored value applies, then the `CLICKHOUSE_HOGQL_USE_NEW_EVENTS_SCHEMA` instance settings. This is an internal rollout switch. PostHog staff set the project value in Django admin and the project settings API ignores it."+ "Read events from the native JSON events table (`true`) or the legacy events table (`false`). When unset, a staff person's `ph_new_events_schema` cookie applies, then the project's listing in the `CLICKHOUSE_HOGQL_USE_NEW_EVENTS_SCHEMA_TEAMS` instance setting, then the `CLICKHOUSE_HOGQL_USE_NEW_EVENTS_SCHEMA` instance setting. This is an internal rollout switch, and the project settings API ignores it."),usePreaggregatedIntermediateResults: zod.union([zod.boolean(), zod.null()]).optional(),usePreaggregatedTableTransforms: zod@@ -2400,17 +2388,11 @@ export const BatchExportsPartialUpdateBody = /* @__PURE__ */ zod'Remove provably redundant casts and nullability wrappers (e.g. `toString(String)`, `assumeNotNull(non_nullable)`, dead `ifNull` fallbacks) using inferred expression types'),useMaterializedViews: zod.union([zod.boolean(), zod.null()]).optional(),- useNewEventsCluster: zod- .union([zod.boolean(), zod.null()])- .optional()- .describe(- 'Read this query from the native events cluster and native JSON events table (`true`), or keep its existing connection (`false` or unset). This is an internal rollout switch for eligible projects. PostHog staff manage eligibility through instance settings. Each query must opt in; project defaults cannot enable it. Only synchronous HogQLQuery SQL comparisons are supported.'- ),useNewEventsSchema: zod.union([zod.boolean(), zod.null()]).optional().describe(- "Read events from the native JSON events table (`true`) or the legacy events table (`false`). When unset, the project's stored value applies, then the `CLICKHOUSE_HOGQL_USE_NEW_EVENTS_SCHEMA` instance settings. This is an internal rollout switch. PostHog staff set the project value in Django admin and the project settings API ignores it."+ "Read events from the native JSON events table (`true`) or the legacy events table (`false`). When unset, a staff person's `ph_new_events_schema` cookie applies, then the project's listing in the `CLICKHOUSE_HOGQL_USE_NEW_EVENTS_SCHEMA_TEAMS` instance setting, then the `CLICKHOUSE_HOGQL_USE_NEW_EVENTS_SCHEMA` instance setting. This is an internal rollout switch, and the project settings API ignores it."),usePreaggregatedIntermediateResults: zod.union([zod.boolean(), zod.null()]).optional(),usePreaggregatedTableTransforms: zod@@ -3130,17 +3112,11 @@ export const BatchExportsRunTestStepCreateBody = /* @__PURE__ */ zod'Remove provably redundant casts and nullability wrappers (e.g. `toString(String)`, `assumeNotNull(non_nullable)`, dead `ifNull` fallbacks) using inferred expression types'),useMaterializedViews: zod.union([zod.boolean(), zod.null()]).optional(),- useNewEventsCluster: zod- .union([zod.boolean(), zod.null()])- .optional()- .describe(- 'Read this query from the native events cluster and native JSON events table (`true`), or keep its existing connection (`false` or unset). This is an internal rollout switch for eligible projects. PostHog staff manage eligibility through instance settings. Each query must opt in; project defaults cannot enable it. Only synchronous HogQLQuery SQL comparisons are supported.'- ),useNewEventsSchema: zod.union([zod.boolean(), zod.null()]).optional().describe(- "Read events from the native JSON events table (`true`) or the legacy events table (`false`). When unset, the project's stored value applies, then the `CLICKHOUSE_HOGQL_USE_NEW_EVENTS_SCHEMA` instance settings. This is an internal rollout switch. PostHog staff set the project value in Django admin and the project settings API ignores it."+ "Read events from the native JSON events table (`true`) or the legacy events table (`false`). When unset, a staff person's `ph_new_events_schema` cookie applies, then the project's listing in the `CLICKHOUSE_HOGQL_USE_NEW_EVENTS_SCHEMA_TEAMS` instance setting, then the `CLICKHOUSE_HOGQL_USE_NEW_EVENTS_SCHEMA` instance setting. This is an internal rollout switch, and the project settings API ignores it."),usePreaggregatedIntermediateResults: zod.union([zod.boolean(), zod.null()]).optional(),usePreaggregatedTableTransforms: zod@@ -3879,17 +3855,11 @@ export const BatchExportsRunTestStepNewCreateBody = /* @__PURE__ */ zod'Remove provably redundant casts and nullability wrappers (e.g. `toString(String)`, `assumeNotNull(non_nullable)`, dead `ifNull` fallbacks) using inferred expression types'),useMaterializedViews: zod.union([zod.boolean(), zod.null()]).optional(),- useNewEventsCluster: zod- .union([zod.boolean(), zod.null()])- .optional()- .describe(- 'Read this query from the native events cluster and native JSON events table (`true`), or keep its existing connection (`false` or unset). This is an internal rollout switch for eligible projects. PostHog staff manage eligibility through instance settings. Each query must opt in; project defaults cannot enable it. Only synchronous HogQLQuery SQL comparisons are supported.'- ),useNewEventsSchema: zod.union([zod.boolean(), zod.null()]).optional().describe(- "Read events from the native JSON events table (`true`) or the legacy events table (`false`). When unset, the project's stored value applies, then the `CLICKHOUSE_HOGQL_USE_NEW_EVENTS_SCHEMA` instance settings. This is an internal rollout switch. PostHog staff set the project value in Django admin and the project settings API ignores it."+ "Read events from the native JSON events table (`true`) or the legacy events table (`false`). When unset, a staff person's `ph_new_events_schema` cookie applies, then the project's listing in the `CLICKHOUSE_HOGQL_USE_NEW_EVENTS_SCHEMA_TEAMS` instance setting, then the `CLICKHOUSE_HOGQL_USE_NEW_EVENTS_SCHEMA` instance setting. This is an internal rollout switch, and the project settings API ignores it."),usePreaggregatedIntermediateResults: zod.union([zod.boolean(), zod.null()]).optional(),usePreaggregatedTableTransforms: zod@@ -4345,17 +4315,11 @@ export const FileDownloadBatchExportsCreateBody = /* @__PURE__ */ zod.union(['Remove provably redundant casts and nullability wrappers (e.g. `toString(String)`, `assumeNotNull(non_nullable)`, dead `ifNull` fallbacks) using inferred expression types'),useMaterializedViews: zod.union([zod.boolean(), zod.null()]).optional(),- useNewEventsCluster: zod- .union([zod.boolean(), zod.null()])- .optional()- .describe(- 'Read this query from the native events cluster and native JSON events table (`true`), or keep its existing connection (`false` or unset). This is an internal rollout switch for eligible projects. PostHog staff manage eligibility through instance settings. Each query must opt in; project defaults cannot enable it. Only synchronous HogQLQuery SQL comparisons are supported.'- ),useNewEventsSchema: zod.union([zod.boolean(), zod.null()]).optional().describe(- "Read events from the native JSON events table (`true`) or the legacy events table (`false`). When unset, the project's stored value applies, then the `CLICKHOUSE_HOGQL_USE_NEW_EVENTS_SCHEMA` instance settings. This is an internal rollout switch. PostHog staff set the project value in Django admin and the project settings API ignores it."+ "Read events from the native JSON events table (`true`) or the legacy events table (`false`). When unset, a staff person's `ph_new_events_schema` cookie applies, then the project's listing in the `CLICKHOUSE_HOGQL_USE_NEW_EVENTS_SCHEMA_TEAMS` instance setting, then the `CLICKHOUSE_HOGQL_USE_NEW_EVENTS_SCHEMA` instance setting. This is an internal rollout switch, and the project settings API ignores it."),usePreaggregatedIntermediateResults: zod.union([zod.boolean(), zod.null()]).optional(),usePreaggregatedTableTransforms: zod@@ -4615,17 +4579,11 @@ export const FileDownloadBatchExportsCountRowsCreateBody = /* @__PURE__ */ zod'Remove provably redundant casts and nullability wrappers (e.g. `toString(String)`, `assumeNotNull(non_nullable)`, dead `ifNull` fallbacks) using inferred expression types'),useMaterializedViews: zod.union([zod.boolean(), zod.null()]).optional(),- useNewEventsCluster: zod- .union([zod.boolean(), zod.null()])- .optional()- .describe(- 'Read this query from the native events cluster and native JSON events table (`true`), or keep its existing connection (`false` or unset). This is an internal rollout switch for eligible projects. PostHog staff manage eligibility through instance settings. Each query must opt in; project defaults cannot enable it. Only synchronous HogQLQuery SQL comparisons are supported.'- ),useNewEventsSchema: zod.union([zod.boolean(), zod.null()]).optional().describe(- "Read events from the native JSON events table (`true`) or the legacy events table (`false`). When unset, the project's stored value applies, then the `CLICKHOUSE_HOGQL_USE_NEW_EVENTS_SCHEMA` instance settings. This is an internal rollout switch. PostHog staff set the project value in Django admin and the project settings API ignores it."+ "Read events from the native JSON events table (`true`) or the legacy events table (`false`). When unset, a staff person's `ph_new_events_schema` cookie applies, then the project's listing in the `CLICKHOUSE_HOGQL_USE_NEW_EVENTS_SCHEMA_TEAMS` instance setting, then the `CLICKHOUSE_HOGQL_USE_NEW_EVENTS_SCHEMA` instance setting. This is an internal rollout switch, and the project settings API ignores it."),usePreaggregatedIntermediateResults: zod.union([zod.boolean(), zod.null()]).optional(),usePreaggregatedTableTransforms: zoddiff --git a/products/customer_analytics/frontend/generated/api.schemas.ts b/products/customer_analytics/frontend/generated/api.schemas.tsindex 024624af5a9f..606f9b96432d 100644--- a/products/customer_analytics/frontend/generated/api.schemas.ts+++ b/products/customer_analytics/frontend/generated/api.schemas.ts@@ -1791,9 +1791,7 @@ export interface HogQLQueryModifiersApi {/** Remove provably redundant casts and nullability wrappers (e.g. `toString(String)`, `assumeNotNull(non_nullable)`, dead `ifNull` fallbacks) using inferred expression types */typeAwareCastSimplification?: boolean | nulluseMaterializedViews?: boolean | null- /** Read this query from the native events cluster and native JSON events table (`true`), or keep its existing connection (`false` or unset). This is an internal rollout switch for eligible projects. PostHog staff manage eligibility through instance settings. Each query must opt in; project defaults cannot enable it. Only synchronous HogQLQuery SQL comparisons are supported. */- useNewEventsCluster?: boolean | null- /** Read events from the native JSON events table (`true`) or the legacy events table (`false`). When unset, the project's stored value applies, then the `CLICKHOUSE_HOGQL_USE_NEW_EVENTS_SCHEMA` instance settings. This is an internal rollout switch. PostHog staff set the project value in Django admin and the project settings API ignores it. */+ /** Read events from the native JSON events table (`true`) or the legacy events table (`false`). When unset, a staff person's `ph_new_events_schema` cookie applies, then the project's listing in the `CLICKHOUSE_HOGQL_USE_NEW_EVENTS_SCHEMA_TEAMS` instance setting, then the `CLICKHOUSE_HOGQL_USE_NEW_EVENTS_SCHEMA` instance setting. This is an internal rollout switch, and the project settings API ignores it. */useNewEventsSchema?: boolean | nullusePreaggregatedIntermediateResults?: boolean | null/** Try to automatically convert HogQL queries to use preaggregated tables at the AST level * */diff --git a/products/dashboards/backend/api/dashboard.py b/products/dashboards/backend/api/dashboard.pyindex 33b9bca05350..a100bb45ebc5 100644--- a/products/dashboards/backend/api/dashboard.py+++ b/products/dashboards/backend/api/dashboard.py@@ -530,6 +530,14 @@ def serialize_tile_with_context(tile, order: int, context: dict) -> tuple[int, dreturn order, tile_data+def serialize_tile_in_request_context(+ request_context: contextvars.Context, tile: DashboardTile, order: int, context: dict+) -> tuple[int, dict]:+ # A streamed response body runs after the middleware chain has returned, so the request's context variables,+ # the events table switch among them, reach a tile's queries only through a copy taken inside the view.+ return request_context.run(serialize_tile_with_context, tile, order, context)++class ReorderLayout(StrEnum):PRESERVE = "preserve"TWO_COLUMN = "two_column"@@ -3327,6 +3335,8 @@ class DashboardsViewSet(# below runs no queries for them.team_configs_for_cache_key(self.team)+ request_context = contextvars.copy_context()+# Async generator that handles progressive tile serialization and streamingasync def async_tile_stream_generator() -> AsyncGenerator[bytes]:renderer = SafeJSONRenderer()@@ -3340,8 +3350,8 @@ class DashboardsViewSet(tile = sorted_tiles[order]try:order_result, tile_data = await database_sync_to_async(- serialize_tile_with_context, thread_sensitive=True- )(tile, order, context)+ serialize_tile_in_request_context, thread_sensitive=True+ )(request_context, tile, order, context)initial_tiles.append(tile_data)except Exception:logger.exception("Error serializing initial dashboard tile", tile_id=tile.id)@@ -3363,8 +3373,8 @@ class DashboardsViewSet(tile = sorted_tiles[order]try:order_result, tile_data = await database_sync_to_async(- serialize_tile_with_context, thread_sensitive=True- )(tile, order, context)+ serialize_tile_in_request_context, thread_sensitive=True+ )(request_context, tile, order, context)tile_json = renderer.render({"type": "tile", "order": order, "tile": tile_data}).decode()yield f"data: {tile_json}\n\n".encode()except Exception:diff --git a/products/dashboards/frontend/generated/api.schemas.ts b/products/dashboards/frontend/generated/api.schemas.tsindex 80a565b21322..5a36d0c5e52a 100644--- a/products/dashboards/frontend/generated/api.schemas.ts+++ b/products/dashboards/frontend/generated/api.schemas.ts@@ -2454,9 +2454,7 @@ export interface HogQLQueryModifiersApi {/** Remove provably redundant casts and nullability wrappers (e.g. `toString(String)`, `assumeNotNull(non_nullable)`, dead `ifNull` fallbacks) using inferred expression types */typeAwareCastSimplification?: boolean | nulluseMaterializedViews?: boolean | null- /** Read this query from the native events cluster and native JSON events table (`true`), or keep its existing connection (`false` or unset). This is an internal rollout switch for eligible projects. PostHog staff manage eligibility through instance settings. Each query must opt in; project defaults cannot enable it. Only synchronous HogQLQuery SQL comparisons are supported. */- useNewEventsCluster?: boolean | null- /** Read events from the native JSON events table (`true`) or the legacy events table (`false`). When unset, the project's stored value applies, then the `CLICKHOUSE_HOGQL_USE_NEW_EVENTS_SCHEMA` instance settings. This is an internal rollout switch. PostHog staff set the project value in Django admin and the project settings API ignores it. */+ /** Read events from the native JSON events table (`true`) or the legacy events table (`false`). When unset, a staff person's `ph_new_events_schema` cookie applies, then the project's listing in the `CLICKHOUSE_HOGQL_USE_NEW_EVENTS_SCHEMA_TEAMS` instance setting, then the `CLICKHOUSE_HOGQL_USE_NEW_EVENTS_SCHEMA` instance setting. This is an internal rollout switch, and the project settings API ignores it. */useNewEventsSchema?: boolean | nullusePreaggregatedIntermediateResults?: boolean | null/** Try to automatically convert HogQL queries to use preaggregated tables at the AST level * */diff --git a/products/desktop/packages/api-client/src/generated.ts b/products/desktop/packages/api-client/src/generated.tsindex 9fec9351d7e5..e870ed4994b2 100644--- a/products/desktop/packages/api-client/src/generated.ts+++ b/products/desktop/packages/api-client/src/generated.ts@@ -234,7 +234,6 @@ export namespace Schemas {timings: boolean | null;typeAwareCastSimplification: boolean | null;useMaterializedViews: boolean | null;- useNewEventsCluster: boolean | null;useNewEventsSchema: boolean | null;usePreaggregatedIntermediateResults: boolean | null;usePreaggregatedTableTransforms: boolean | null;diff --git a/products/endpoints/backend/logic/execution.py b/products/endpoints/backend/logic/execution.pyindex 152c80896537..a7af4a078cd1 100644--- a/products/endpoints/backend/logic/execution.py+++ b/products/endpoints/backend/logic/execution.py@@ -78,7 +78,6 @@ from products.endpoints.backend.logic.materialized_serving import serving_state_from products.endpoints.backend.logic.pagination import EndpointPaginationfrom products.endpoints.backend.logic.rate_limit_signal import EndpointRateLimitReason, record_endpoint_rate_limitedfrom products.endpoints.backend.logic.strategies import EndpointQueryStrategy, strategy_for-from products.endpoints.backend.logic.validation import NATIVE_CLUSTER_UNSUPPORTEDfrom products.endpoints.backend.logs import build_execution_message, log_endpoint_executionfrom products.endpoints.backend.metrics import (ENDPOINT_CACHE_RESULT_TOTAL,@@ -375,8 +374,6 @@ class EndpointExecutionService(PydanticModelMixin):version: EndpointVersion,offset: int | None = None,) -> None:- if (version.query.get("modifiers") or {}).get("useNewEventsCluster"):- raise ValidationError({"query": NATIVE_CLUSTER_UNSUPPORTED})strategy = strategy_for(endpoint, version, self.team)is_materialized = bool(version.is_materialized and version.saved_query)diff --git a/products/endpoints/backend/logic/validation.py b/products/endpoints/backend/logic/validation.pyindex 01381207f2c1..e88dfb1cc972 100644--- a/products/endpoints/backend/logic/validation.py+++ b/products/endpoints/backend/logic/validation.py@@ -69,9 +69,6 @@ DIRECT_CONNECTION_UNSUPPORTED = ("Endpoints can't query a direct connection. ""Switch the connection to PostHog (ClickHouse) and query a synced table instead.")-NATIVE_CLUSTER_UNSUPPORTED = (- "Endpoints do not support native events cluster opt-ins. Use the query API for comparisons."-)def validate_hogql_query(query: HogQLQuery, team: Team, user: User) -> None:@@ -281,9 +278,6 @@ def validate_optional_breakdown_properties(def validate_endpoint_request(data: EndpointRequest, team: Team, user: User, strict: bool = True) -> None:"""Validate a create/update payload. With strict=True, name and query are required."""query = data.query- modifiers = getattr(query, "modifiers", None)- if modifiers and modifiers.useNewEventsCluster:- raise ValidationError({"query": NATIVE_CLUSTER_UNSUPPORTED})if not query and strict:raise ValidationError({"query": "This field is required."})@@ -316,9 +310,6 @@ def validate_update_request(version_number: int | None = None,) -> None:"""Validate an update payload against the endpoint's resulting state."""- modifiers = getattr(data.query, "modifiers", None)- if modifiers and modifiers.useNewEventsCluster:- raise ValidationError({"query": NATIVE_CLUSTER_UNSUPPORTED})validate_data_freshness(data.data_freshness_seconds)# Determine final states after this request (for validation)diff --git a/products/endpoints/backend/tests/test_endpoint.py b/products/endpoints/backend/tests/test_endpoint.pyindex a27e0963a29a..4e4bfcce639d 100644--- a/products/endpoints/backend/tests/test_endpoint.py+++ b/products/endpoints/backend/tests/test_endpoint.py@@ -69,22 +69,14 @@ class TestEndpoint(ClickhouseTestMixin, APIBaseTest):"series": [{"kind": "EventsNode"}],}- @parameterized.expand([(None,), (True,)])- def test_create_hogql_endpoint(self, native: bool | None):+ def test_create_hogql_endpoint(self):data = {"name": "test_query","description": "Test query description","query": self.sample_hogql_query,}- if native is not None:- data["query"] = {**self.sample_hogql_query, "modifiers": {"useNewEventsCluster": native}}response = self.client.post(f"/api/environments/{self.team.id}/endpoints/", data, format="json")- if native:- self.assertEqual(status.HTTP_400_BAD_REQUEST, response.status_code, response.json())- self.assertIn("not support native events cluster", str(response.json()))- self.assertFalse(Endpoint.objects.filter(team=self.team, name="test_query").exists())- returnself.assertEqual(status.HTTP_201_CREATED, response.status_code, response.json())response_data = response.json()@@ -327,8 +319,7 @@ class TestEndpoint(ClickhouseTestMixin, APIBaseTest):self.assertEqual(status.HTTP_201_CREATED, response.status_code, response.json())- @parameterized.expand([(None,), (True,)])- def test_update_endpoint(self, native: bool | None):+ def test_update_endpoint(self):endpoint = create_endpoint_with_version(name="update_test",team=self.team,@@ -337,26 +328,16 @@ class TestEndpoint(ClickhouseTestMixin, APIBaseTest):created_by=self.user,)- updated_query: dict[str, Any] = {"kind": "HogQLQuery", "query": "SELECT 1"}- if native is not None:- updated_query["modifiers"] = {"useNewEventsCluster": native}-updated_data: dict[str, Any] = {"description": "Updated description","is_active": False,- "query": updated_query,+ "query": {"kind": "HogQLQuery", "query": "SELECT 1"},}response = self.client.put(f"/api/environments/{self.team.id}/endpoints/{endpoint.name}/", updated_data, format="json")- if native:- self.assertEqual(status.HTTP_400_BAD_REQUEST, response.status_code, response.json())- self.assertIn("not support native events cluster", str(response.json()))- self.assertEqual(endpoint.versions.count(), 1)- return-response_data = response.json()self.assertEqual(status.HTTP_200_OK, response.status_code, response_data)diff --git a/products/endpoints/backend/tests/test_endpoint_query_tagging.py b/products/endpoints/backend/tests/test_endpoint_query_tagging.pyindex 0a7218d35017..893d8392f3c8 100644--- a/products/endpoints/backend/tests/test_endpoint_query_tagging.py+++ b/products/endpoints/backend/tests/test_endpoint_query_tagging.py@@ -1,11 +1,9 @@from posthog.test.base import APIBaseTest, ClickhouseTestMixinfrom unittest import mock-from parameterized import parameterizedfrom rest_framework import statusfrom posthog.clickhouse.query_tagging import Feature, Product, tag_queries-from posthog.models.instance_setting import override_instance_configfrom posthog.models.personal_api_key import PersonalAPIKeyfrom posthog.models.utils import generate_random_token_personal, hash_key_value@@ -44,33 +42,9 @@ class TestEndpointQueryTagging(ClickhouseTestMixin, APIBaseTest):scopes=["*"],)- @parameterized.expand([(None, False), (True, False), (True, True)])- def test_run_via_session_auth_tags_endpoint_playground(self, native: bool | None, warm_cache: bool):- version = self.endpoint.get_version()- assert version is not None- if native is not None:- version.query["modifiers"] = {"useNewEventsCluster": native}- version.save()- with (- override_instance_config("CLICKHOUSE_NATIVE_EVENTS_HOST", "native.example.com"),- override_instance_config("CLICKHOUSE_NATIVE_EVENTS_TEAMS", [self.team.pk]),- mock.patch("posthog.hogql.query.sync_execute", return_value=([(1,)], [("1", "UInt8")])) as execute,- mock.patch("products.endpoints.backend.logic.execution.tag_queries", wraps=tag_queries) as spy,- ):- if warm_cache:- response = self.client.post(- f"/api/projects/{self.team.id}/query/",- {"query": version.query, "refresh": "force_blocking", "async": False},- format="json",- )- assert response.status_code == status.HTTP_200_OK, response.json()- execute.reset_mock()+ def test_run_via_session_auth_tags_endpoint_playground(self):+ with mock.patch("products.endpoints.backend.logic.execution.tag_queries", wraps=tag_queries) as spy:response = self.client.post(self.run_url, {}, format="json")- if native:- assert response.status_code == status.HTTP_400_BAD_REQUEST, response.json()- assert "not support native events cluster" in str(response.json())- execute.assert_not_called()- returnassert response.status_code == status.HTTP_200_OK, response.json()call = _find_endpoint_feature_call(spy)assert call is not None, f"no tag_queries(product=ENDPOINTS, feature=...) call found: {spy.call_args_list}"diff --git a/products/marketing_analytics/backend/hogql_queries/marketing_lazy_precompute.py b/products/marketing_analytics/backend/hogql_queries/marketing_lazy_precompute.pyindex c57c41bc4ef2..0049e0fe6104 100644--- a/products/marketing_analytics/backend/hogql_queries/marketing_lazy_precompute.py+++ b/products/marketing_analytics/backend/hogql_queries/marketing_lazy_precompute.py@@ -27,9 +27,12 @@ from typing import Anyimport structlogfrom prometheus_client import Counter+from posthog.schema import HogQLQueryModifiers+from posthog import redisfrom posthog.clickhouse.query_tagging import get_query_tag_value, tag_queriesfrom posthog.models import Team+from posthog.models.event.new_events_schema import events_route_cache_modifiers, use_new_events_schemafrom posthog.token_bucket import BucketDecision, Budget, consumefrom products.analytics_platform.backend.lazy_computation.lazy_computation_executor import (@@ -125,8 +128,13 @@ def marketing_ensure_precomputed(*, team: Team, **kwargs: Any) -> LazyComputatiodef _query_shape_key(query: Any) -> str:+ query_json = query.model_dump(mode="json", exclude_none=True, exclude=set(FINAL_READ_ONLY_FIELDS))+ if "modifiers" in query_json:+ query_json["modifiers"] = events_route_cache_modifiers(query_json["modifiers"], reads_events=True)+ if not query_json["modifiers"]:+ query_json.pop("modifiers")payload = json.dumps(- query.model_dump(mode="json", exclude_none=True, exclude=set(FINAL_READ_ONLY_FIELDS)),+ query_json,sort_keys=True,default=str,)@@ -150,6 +158,17 @@ def _scope_to_revalidation(query: Any) -> Any:return scoped+def _revalidation_query(*, team: Team, query: Any) -> Any:+ """The query the revalidation task rebuilds, which also keys its debounce slot.++ The events table the stale read resolved is pinned, so the task, which runs without the reader's cookie, rebuilds+ from the same table, and each table gets its own slot.+ """+ modifiers = (query.modifiers or HogQLQueryModifiers()).model_copy()+ modifiers.useNewEventsSchema = use_new_events_schema(team.pk, modifiers)+ return _scope_to_revalidation(query.model_copy(update={"modifiers": modifiers}))++def is_on_demand_revalidation() -> bool:"""True inside the Celery revalidation task, which a user read can trigger.@@ -171,7 +190,7 @@ def enqueue_stale_revalidation(*, team: Team, query: Any) -> None:revalidate_marketing_analytics_precompute,)- query = _scope_to_revalidation(query)+ query = _revalidation_query(team=team, query=query)claimed = Falsedebounce_key: str | None = Nonetry:diff --git a/products/marketing_analytics/backend/hogql_queries/test_marketing_cost_trends_query_resolver.py b/products/marketing_analytics/backend/hogql_queries/test_marketing_cost_trends_query_resolver.pyindex 4120c78304bb..d568c2978294 100644--- a/products/marketing_analytics/backend/hogql_queries/test_marketing_cost_trends_query_resolver.py+++ b/products/marketing_analytics/backend/hogql_queries/test_marketing_cost_trends_query_resolver.py@@ -256,14 +256,20 @@ class TestMarketingCostTrendsQueryResolver(SimpleTestCase):self.ensure.side_effect = ensure_precomputedbase_module = "products.marketing_analytics.backend.hogql_queries.marketing_analytics_base_query_runner"with (+ # A pinned events table keeps the warmer's runner from asking the instance settings; this test has no database.patch("posthog.hogql_queries.query_runner.create_default_modifiers_for_team",- return_value=HogQLQueryModifiers(),+ return_value=HogQLQueryModifiers(useNewEventsSchema=False),),patch.object(MarketingSourceFactory, "__init__", initialize_factory),patch.object(MarketingSourceFactory, "create_adapters", create_adapters),patch(f"{base_module}.MarketingSourceFactory", MarketingSourceFactory),patch(f"{base_module}.marketing_ensure_precomputed", side_effect=ensure_precomputed),+ # The job identity asks the instance settings which events table a build reads; this test has no database.+ patch(+ "products.analytics_platform.backend.lazy_computation.lazy_computation_executor.use_new_events_schema",+ return_value=False,+ ),patch.object(LazyComputationExecutor, "execute", return_value=LazyComputationResult(ready=True, job_ids=[])) as execute,diff --git a/products/marketing_analytics/backend/hogql_queries/test_marketing_lazy_precompute.py b/products/marketing_analytics/backend/hogql_queries/test_marketing_lazy_precompute.pyindex 6d6dbbd52d2d..b82748e1f520 100644--- a/products/marketing_analytics/backend/hogql_queries/test_marketing_lazy_precompute.py+++ b/products/marketing_analytics/backend/hogql_queries/test_marketing_lazy_precompute.py@@ -1,12 +1,18 @@from posthog.test.base import BaseTestfrom unittest import mock+from django.test import SimpleTestCase+from parameterized import parameterized-from posthog.schema import CompareFilter, DateRange, MarketingAnalyticsTableQuery+from posthog.schema import CompareFilter, DateRange, HogQLQueryModifiers, MarketingAnalyticsTableQueryfrom posthog import redisfrom posthog.clickhouse.query_tagging import Feature, get_query_tag_value, reset_query_tags, tags_context+from posthog.models import Team+from posthog.models.event.new_events_schema import events_route_scope, new_events_schema_for_request+from posthog.redis import TEST_clear_clients+from posthog.token_bucket import TEST_reset_scriptsfrom products.analytics_platform.backend.lazy_computation.lazy_computation_executor import LazyComputationResultfrom products.marketing_analytics.backend.hogql_queries.marketing_lazy_precompute import (@@ -15,6 +21,7 @@ from products.marketing_analytics.backend.hogql_queries.marketing_lazy_precomputREVALIDATION_TRIGGER,TEAM_WARM_BUDGET,_query_shape_key,+ _revalidation_query,handle_not_ready,handle_stale_served,marketing_ensure_precomputed,@@ -27,11 +34,33 @@ _DELAY = ()+class TestMarketingRevalidationEventsTable(SimpleTestCase):+ def setUp(self) -> None:+ TEST_clear_clients()+ TEST_reset_scripts()+ self.addCleanup(TEST_clear_clients)+ self.addCleanup(TEST_reset_scripts)++ def test_rebuilds_keep_the_events_table_and_debounce_each_table_separately(self) -> None:+ team = Team(id=44, timezone="UTC")+ query = MarketingAnalyticsTableQuery(dateRange=DateRange(date_from="-7d"), properties=[])+ with mock.patch(_DELAY) as delay, new_events_schema_for_request():+ for native in (False, True, False, True):+ with events_route_scope(HogQLQueryModifiers(useNewEventsSchema=native)):+ handle_not_ready(team=team, query=query)++ assert [call.kwargs["query"]["modifiers"]["useNewEventsSchema"] for call in delay.call_args_list] == [+ False,+ True,+ ]+ assert query.modifiers is None++class TestMarketingLazyPrecompute(BaseTest):def setUp(self):super().setUp()self.query = MarketingAnalyticsTableQuery(dateRange=DateRange(date_from="-7d"), properties=[])- redis.get_client().delete(f"ma_swr_reval:{self.team.id}:{_query_shape_key(self.query)}")+ redis.get_client().delete(self._debounce_key(self.query))redis.get_client().delete(f"ma_swr_reval_rate:{self.team.id}")reset_query_tags()@@ -39,6 +68,9 @@ class TestMarketingLazyPrecompute(BaseTest):reset_query_tags()super().tearDown()+ def _debounce_key(self, query: MarketingAnalyticsTableQuery) -> str:+ return f"ma_swr_reval:{self.team.id}:{_query_shape_key(_revalidation_query(team=self.team, query=query))}"+@parameterized.expand([# User-facing read: precompute-only. Never build inline (run_inserts=False), and serve stale@@ -110,7 +142,7 @@ class TestMarketingLazyPrecompute(BaseTest):@mock.patch(_DELAY)def test_debounce_splits_only_on_precompute_identity(self, _name, overrides, expected_enqueues, delay):variant = self.query.model_copy(update=overrides)- redis.get_client().delete(f"ma_swr_reval:{self.team.id}:{_query_shape_key(variant)}")+ redis.get_client().delete(self._debounce_key(variant))handle_stale_served(team=self.team, query=self.query)handle_stale_served(team=self.team, query=variant)@@ -126,11 +158,11 @@ class TestMarketingLazyPrecompute(BaseTest):for days in range(1, TEAM_WARM_BUDGET.burst + 4)]for shape in shapes:- redis.get_client().delete(f"ma_swr_reval:{self.team.id}:{_query_shape_key(shape)}")+ redis.get_client().delete(self._debounce_key(shape))handle_not_ready(team=self.team, query=shape)assert delay.call_count == TEAM_WARM_BUDGET.burst- throttled_key = f"ma_swr_reval:{self.team.id}:{_query_shape_key(shapes[-1])}"+ throttled_key = self._debounce_key(shapes[-1])assert 0 < redis.get_client().ttl(throttled_key) <= ENQUEUE_FAILURE_BACKOFF_SECONDS@mock.patch(_DELAY)@@ -148,7 +180,7 @@ class TestMarketingLazyPrecompute(BaseTest):# The slot is claimed before the enqueue is confirmed. If the enqueue fails, holding it the full# window would suppress every retry for 10 minutes with no rebuild in flight — release it to a# short backoff so revalidation resumes soon after the broker recovers.- key = f"ma_swr_reval:{self.team.id}:{_query_shape_key(self.query)}"+ key = self._debounce_key(self.query)handle_stale_served(team=self.team, query=self.query)diff --git a/products/marketing_analytics/backend/tasks/lazy_precompute_revalidation.py b/products/marketing_analytics/backend/tasks/lazy_precompute_revalidation.pyindex e82591dcd990..df0ff243d72c 100644--- a/products/marketing_analytics/backend/tasks/lazy_precompute_revalidation.py+++ b/products/marketing_analytics/backend/tasks/lazy_precompute_revalidation.py@@ -27,6 +27,7 @@ from posthog.hogql.constants import LimitContextfrom posthog.clickhouse.query_tagging import Feature, Product, tag_queriesfrom posthog.hogql_queries.query_runner import get_query_runnerfrom posthog.models import Team+from posthog.models.event.new_events_schema import events_route_scopefrom posthog.scoping_audit import skip_team_scope_auditfrom posthog.tasks.utils import CeleryQueue@@ -93,7 +94,8 @@ def revalidate_marketing_analytics_precompute(team_id: int, query: dict) -> None# response into the per-team result cache, which a user without access to some of those warehouse# sources could then read. Leaving the result cache alone, the next user read recomputes the# response under its own access — now over hot precomputes — and caches it correctly.- runner.to_query()+ with events_route_scope(runner.modifiers):+ runner.to_query()except Exception:logger.exception("marketing_analytics_swr_revalidation_failed", team_id=team_id, query_kind=query_kind)REVALIDATION_RUN.labels(outcome="failed", query_kind=query_kind).inc()diff --git a/products/marketing_analytics/backend/tasks/test_lazy_precompute_revalidation.py b/products/marketing_analytics/backend/tasks/test_lazy_precompute_revalidation.pyindex 013f5e37e662..ff40c4dc7750 100644--- a/products/marketing_analytics/backend/tasks/test_lazy_precompute_revalidation.py+++ b/products/marketing_analytics/backend/tasks/test_lazy_precompute_revalidation.py@@ -1,9 +1,10 @@from posthog.test.base import BaseTestfrom unittest import mock-from posthog.schema import DateRange, MarketingAnalyticsTableQuery+from posthog.schema import DateRange, HogQLQueryModifiers, MarketingAnalyticsTableQueryfrom posthog.clickhouse.query_tagging import Feature, get_query_tag_value, reset_query_tags+from posthog.models.event.new_events_schema import use_new_events_schemafrom products.analytics_platform.backend.lazy_computation.stale_policy import is_background_warming_requestfrom products.marketing_analytics.backend.hogql_queries.marketing_lazy_precompute import BACKGROUND_WARMING_TRIGGERS@@ -36,6 +37,9 @@ class TestMarketingLazyPrecomputeRevalidation(BaseTest):# a warehouse-restricted user could then read.tags_at_build: dict[str, object] = {}runner = mock.MagicMock()+ runner.modifiers = HogQLQueryModifiers(useNewEventsSchema=True)+ build_schemas: list[bool] = []+ runner.to_query.side_effect = lambda: build_schemas.append(use_new_events_schema(self.team.pk))def capture(**_kwargs):tags_at_build["refresher"] = is_background_warming_request(BACKGROUND_WARMING_TRIGGERS)@@ -50,6 +54,7 @@ class TestMarketingLazyPrecomputeRevalidation(BaseTest):assert tags_at_build == {"refresher": True, "feature": Feature.PREAGGREGATION}runner.to_query.assert_called_once_with()runner.run.assert_not_called()+ assert build_schemas == [True]@mock.patch(f"{_MODULE}.get_query_runner")def test_deleted_team_does_not_raise_into_the_worker(self, get_runner):diff --git a/products/marketing_analytics/frontend/generated/api.schemas.ts b/products/marketing_analytics/frontend/generated/api.schemas.tsindex 7f77fd8f6cf0..66d2dea31fab 100644--- a/products/marketing_analytics/frontend/generated/api.schemas.ts+++ b/products/marketing_analytics/frontend/generated/api.schemas.ts@@ -1748,9 +1748,7 @@ export interface HogQLQueryModifiersApi {/** Remove provably redundant casts and nullability wrappers (e.g. `toString(String)`, `assumeNotNull(non_nullable)`, dead `ifNull` fallbacks) using inferred expression types */typeAwareCastSimplification?: boolean | nulluseMaterializedViews?: boolean | null- /** Read this query from the native events cluster and native JSON events table (`true`), or keep its existing connection (`false` or unset). This is an internal rollout switch for eligible projects. PostHog staff manage eligibility through instance settings. Each query must opt in; project defaults cannot enable it. Only synchronous HogQLQuery SQL comparisons are supported. */- useNewEventsCluster?: boolean | null- /** Read events from the native JSON events table (`true`) or the legacy events table (`false`). When unset, the project's stored value applies, then the `CLICKHOUSE_HOGQL_USE_NEW_EVENTS_SCHEMA` instance settings. This is an internal rollout switch. PostHog staff set the project value in Django admin and the project settings API ignores it. */+ /** Read events from the native JSON events table (`true`) or the legacy events table (`false`). When unset, a staff person's `ph_new_events_schema` cookie applies, then the project's listing in the `CLICKHOUSE_HOGQL_USE_NEW_EVENTS_SCHEMA_TEAMS` instance setting, then the `CLICKHOUSE_HOGQL_USE_NEW_EVENTS_SCHEMA` instance setting. This is an internal rollout switch, and the project settings API ignores it. */useNewEventsSchema?: boolean | nullusePreaggregatedIntermediateResults?: boolean | null/** Try to automatically convert HogQL queries to use preaggregated tables at the AST level * */diff --git a/products/persons/frontend/generated/api.schemas.ts b/products/persons/frontend/generated/api.schemas.tsindex 2b3864ddec4b..1ac403ec39a2 100644--- a/products/persons/frontend/generated/api.schemas.ts+++ b/products/persons/frontend/generated/api.schemas.ts@@ -566,9 +566,7 @@ export interface HogQLQueryModifiersApi {/** Remove provably redundant casts and nullability wrappers (e.g. `toString(String)`, `assumeNotNull(non_nullable)`, dead `ifNull` fallbacks) using inferred expression types */typeAwareCastSimplification?: boolean | nulluseMaterializedViews?: boolean | null- /** Read this query from the native events cluster and native JSON events table (`true`), or keep its existing connection (`false` or unset). This is an internal rollout switch for eligible projects. PostHog staff manage eligibility through instance settings. Each query must opt in; project defaults cannot enable it. Only synchronous HogQLQuery SQL comparisons are supported. */- useNewEventsCluster?: boolean | null- /** Read events from the native JSON events table (`true`) or the legacy events table (`false`). When unset, the project's stored value applies, then the `CLICKHOUSE_HOGQL_USE_NEW_EVENTS_SCHEMA` instance settings. This is an internal rollout switch. PostHog staff set the project value in Django admin and the project settings API ignores it. */+ /** Read events from the native JSON events table (`true`) or the legacy events table (`false`). When unset, a staff person's `ph_new_events_schema` cookie applies, then the project's listing in the `CLICKHOUSE_HOGQL_USE_NEW_EVENTS_SCHEMA_TEAMS` instance setting, then the `CLICKHOUSE_HOGQL_USE_NEW_EVENTS_SCHEMA` instance setting. This is an internal rollout switch, and the project settings API ignores it. */useNewEventsSchema?: boolean | nullusePreaggregatedIntermediateResults?: boolean | null/** Try to automatically convert HogQL queries to use preaggregated tables at the AST level * */diff --git a/products/product_analytics/backend/hogql_queries/funnels/test/test_funnel_insight_actors.py b/products/product_analytics/backend/hogql_queries/funnels/test/test_funnel_insight_actors.pyindex a89004260a6f..c93889f42fa6 100644--- a/products/product_analytics/backend/hogql_queries/funnels/test/test_funnel_insight_actors.py+++ b/products/product_analytics/backend/hogql_queries/funnels/test/test_funnel_insight_actors.py@@ -20,7 +20,6 @@ from posthog.schema import (EventsNode,EventsQuery,FunnelsActorsQuery,- FunnelsFilter,FunnelsQuery,HogQLQueryModifiers,InsightActorsQuery,@@ -31,8 +30,6 @@ from posthog.schema import (from posthog.hogql.context import HogQLContextfrom posthog.hogql.query import execute_hogql_query-from posthog.clickhouse.workload import Workload-from posthog.hogql_queries.actor_strategies import SessionStrategyfrom posthog.hogql_queries.actors_query_runner import ActorsQueryRunnerfrom posthog.hogql_queries.events_query_runner import EventsQueryRunnerfrom posthog.hogql_queries.insight_actors_query_options_runner import InsightActorsQueryOptionsRunner@@ -40,7 +37,6 @@ from posthog.hogql_queries.insight_actors_query_runner import InsightActorsQueryfrom posthog.hogql_queries.query_runner import QueryRunner, get_query_runner, get_query_runner_or_nonefrom posthog.models import Organization, Team, Userfrom posthog.models.group.util import create_group-from posthog.models.instance_setting import override_instance_configfrom posthog.test.test_utils import create_group_type_mapping_without_created_atfrom products.product_analytics.backend.hogql_queries.funnels.funnels_query_runner import FunnelsQueryRunner@@ -165,21 +161,13 @@ class TestFunnelInsightActors(ClickhouseTestMixin, APIBaseTest):)-@override_settings(CLICKHOUSE_HOGQL_USE_NEW_EVENTS_SCHEMA=False, CLICKHOUSE_NATIVE_EVENTS_HOST="native.example.com")+@override_settings(CLICKHOUSE_HOGQL_USE_NEW_EVENTS_SCHEMA=False)class TestNativeEventsActorRouting(SimpleTestCase):def setUp(self) -> None:self.team = Team(id=44, modifiers={}, organization=Organization(id=1))self.user = User(id=123, is_staff=True, is_active=True)self.modifiers = HogQLQueryModifiers(personsOnEventsMode=PersonsOnEventsMode.DISABLED)- self.enterContext(- patch(- "posthog.models.event.new_events_schema.get_instance_setting",- side_effect={- "CLICKHOUSE_NATIVE_EVENTS_TEAMS": [44],- "CLICKHOUSE_NATIVE_EVENTS_HOST": "native.example.com",- }.get,- )- )+ self.enterContext(patch("posthog.models.event.new_events_schema.get_instance_setting", return_value=None))self.enterContext(patch("products.web_analytics.backend.hogql_queries.cookieless_flag.resolve_cookieless_traffic_is_regular_modifier",@@ -197,7 +185,7 @@ class TestNativeEventsActorRouting(SimpleTestCase):def test_actor_queries_inherit_the_source_route_unless_explicitly_overridden(self, kind: str, source_choice: bool | None, wrapper_choice: bool | None) -> None:- modifiers = self.modifiers.model_copy(update={"useNewEventsCluster": source_choice})+ modifiers = self.modifiers.model_copy(update={"useNewEventsSchema": source_choice})source = FunnelsQuery(series=[EventsNode(event="example"), EventsNode(event="finished")], modifiers=modifiers)actors = FunnelsActorsQuery(source=source, funnelStep=1)query: BaseModel@@ -219,15 +207,12 @@ class TestNativeEventsActorRouting(SimpleTestCase):)if wrapper_choice is not None:assert isinstance(query, (ActorsQuery, InsightActorsQuery))- query.modifiers = self.modifiers.model_copy(- update={"useNewEventsSchema": wrapper_choice, "useNewEventsCluster": wrapper_choice}- )+ query.modifiers = self.modifiers.model_copy(update={"useNewEventsSchema": wrapper_choice})original = query.model_dump()runner = get_query_runner(query, self.team, user=self.user)native = source_choice if wrapper_choice is None else wrapper_choiceself.assertEqual(runner.modifiers.personsOnEventsMode, PersonsOnEventsMode.DISABLED)self.assertIs(bool(runner.modifiers.useNewEventsSchema), bool(native))- self.assertIs(runner.modifiers.useNewEventsCluster, native)if kind == "EventsQuery":self.assertIs(runner.modifiers.convertToProjectTimezone, False)self.assertEqual(query.model_dump(), original)@@ -239,23 +224,6 @@ class TestActorExecutionUserContext(APIBaseTest):runner = get_query_runner(InsightActorsQuery(source=source).model_dump(), self.team, user=self.user)self.assertIsInstance(runner.query, InsightActorsQuery)- @override_settings(CLICKHOUSE_HOGQL_USE_NEW_EVENTS_SCHEMA=False, CLICKHOUSE_NATIVE_EVENTS_HOST="native.example.com")- def test_session_actors_hydrate_on_the_selected_cluster(self) -> None:- source = FunnelsQuery(- series=[EventsNode(event="started"), EventsNode(event="finished")],- funnelsFilter=FunnelsFilter(funnelAggregateByHogQL="properties.$session_id"),- modifiers=HogQLQueryModifiers(useNewEventsSchema=True, useNewEventsCluster=True),- )- with (- override_instance_config("CLICKHOUSE_NATIVE_EVENTS_HOST", "native.example.com"),- override_instance_config("CLICKHOUSE_NATIVE_EVENTS_TEAMS", [self.team.pk]),- patch("posthog.hogql.query.sync_execute", return_value=([], [])) as execute,- ):- runner = get_query_runner(ActorsQuery(source=FunnelsActorsQuery(source=source, funnelStep=1)), self.team)- assert isinstance(runner, ActorsQueryRunner) and isinstance(runner.strategy, SessionStrategy)- runner.strategy.get_actors(["a-session"])- self.assertEqual(execute.call_args.kwargs["workload"], Workload.NEW_EVENTS_CLUSTER)-@parameterized.expand([("ActorsQuery",), ("InsightActorsQuery",), ("InsightActorsQueryOptions",), ("EventsQuery",)])def test_nested_query_context_follows_execution_user(self, kind: str) -> None:source = FunnelsQuery(series=[EventsNode(event="started"), EventsNode(event="finished")])diff --git a/products/product_analytics/backend/hogql_queries/trends/test/test_trend_validation_rules.py b/products/product_analytics/backend/hogql_queries/trends/test/test_trend_validation_rules.pyindex b2559f257124..221d7f764763 100644--- a/products/product_analytics/backend/hogql_queries/trends/test/test_trend_validation_rules.py+++ b/products/product_analytics/backend/hogql_queries/trends/test/test_trend_validation_rules.py@@ -22,9 +22,7 @@ from posthog.schema import (TrendsQuery,)-from posthog.clickhouse.workload import Workloadfrom posthog.hogql_queries.validation.validation import QueryValidationContext-from posthog.models.instance_setting import override_instance_configfrom products.product_analytics.backend.hogql_queries.trends.trend_validation_rules import (DisallowDaysOfWeekWithSmoothing,@@ -443,21 +441,16 @@ class TestTrendsQueryRunnerSeriesFanOut(BaseTest):self.assertEqual(len(runner.series), 150)@parameterized.expand([(False,), (True,)])- @override_settings(CLICKHOUSE_HOGQL_USE_NEW_EVENTS_SCHEMA=False, CLICKHOUSE_NATIVE_EVENTS_HOST="native.example.com")- def test_actor_options_execute_on_the_selected_cluster(self, native: bool) -> None:+ @override_settings(CLICKHOUSE_HOGQL_USE_NEW_EVENTS_SCHEMA=False)+ def test_actor_options_read_the_selected_events_table(self, native: bool) -> None:query = TrendsQuery(series=[EventsNode(event="$pageview")],dateRange=DateRange(date_from="2024-01-01", date_to="2024-01-08"),breakdownFilter=BreakdownFilter(breakdown="$browser", breakdown_type=BreakdownType.EVENT),- modifiers=HogQLQueryModifiers(useNewEventsCluster=native, useNewEventsSchema=native),+ modifiers=HogQLQueryModifiers(useNewEventsSchema=native),)- with (- override_instance_config("CLICKHOUSE_NATIVE_EVENTS_HOST", "native.example.com"),- override_instance_config("CLICKHOUSE_NATIVE_EVENTS_TEAMS", [self.team.pk]),- patch("posthog.hogql.query.sync_execute", return_value=([], [])) as execute,- ):+ with patch("posthog.hogql.query.sync_execute", return_value=([], [])) as execute:result = TrendsQueryRunner(query=query, team=self.team).to_actors_query_options()assert result.breakdown == []execute.assert_called_once()- assert execute.call_args.kwargs["workload"] == (Workload.NEW_EVENTS_CLUSTER if native else Workload.DEFAULT)assert ("events_json" in execute.call_args.args[0]) is nativediff --git a/products/product_analytics/frontend/generated/api.schemas.ts b/products/product_analytics/frontend/generated/api.schemas.tsindex 6c2c41fa6559..dbbb33bc4298 100644--- a/products/product_analytics/frontend/generated/api.schemas.ts+++ b/products/product_analytics/frontend/generated/api.schemas.ts@@ -1280,9 +1280,7 @@ export interface HogQLQueryModifiersApi {/** Remove provably redundant casts and nullability wrappers (e.g. `toString(String)`, `assumeNotNull(non_nullable)`, dead `ifNull` fallbacks) using inferred expression types */typeAwareCastSimplification?: boolean | nulluseMaterializedViews?: boolean | null- /** Read this query from the native events cluster and native JSON events table (`true`), or keep its existing connection (`false` or unset). This is an internal rollout switch for eligible projects. PostHog staff manage eligibility through instance settings. Each query must opt in; project defaults cannot enable it. Only synchronous HogQLQuery SQL comparisons are supported. */- useNewEventsCluster?: boolean | null- /** Read events from the native JSON events table (`true`) or the legacy events table (`false`). When unset, the project's stored value applies, then the `CLICKHOUSE_HOGQL_USE_NEW_EVENTS_SCHEMA` instance settings. This is an internal rollout switch. PostHog staff set the project value in Django admin and the project settings API ignores it. */+ /** Read events from the native JSON events table (`true`) or the legacy events table (`false`). When unset, a staff person's `ph_new_events_schema` cookie applies, then the project's listing in the `CLICKHOUSE_HOGQL_USE_NEW_EVENTS_SCHEMA_TEAMS` instance setting, then the `CLICKHOUSE_HOGQL_USE_NEW_EVENTS_SCHEMA` instance setting. This is an internal rollout switch, and the project settings API ignores it. */useNewEventsSchema?: boolean | nullusePreaggregatedIntermediateResults?: boolean | null/** Try to automatically convert HogQL queries to use preaggregated tables at the AST level * */diff --git a/products/web_analytics/backend/hogql_queries/test/test_web_lazy_precompute_common.py b/products/web_analytics/backend/hogql_queries/test/test_web_lazy_precompute_common.pyindex bde11377fc63..b228023f8592 100644--- a/products/web_analytics/backend/hogql_queries/test/test_web_lazy_precompute_common.py+++ b/products/web_analytics/backend/hogql_queries/test/test_web_lazy_precompute_common.py@@ -33,6 +33,9 @@ from posthog.hogql import astfrom posthog import redisfrom posthog.clickhouse.query_tagging import Feature, get_query_tag_value, reset_query_tags, tags_contextfrom posthog.hogql_queries.query_runner import AnalyticsQueryRunner, ExecutionMode+from posthog.models import Team+from posthog.models.event.new_events_schema import events_route_scope, new_events_schema_for_request+from posthog.redis import TEST_clear_clientsfrom products.analytics_platform.backend.lazy_computation.lazy_computation_executor import (LazyComputationResult,@@ -62,6 +65,7 @@ from products.web_analytics.backend.hogql_queries.web_lazy_precompute_common impcheck_common_eligibility,compute_filters_eligibility_hash,compute_shape_cap_key,+ enqueue_stale_revalidation,get_sticky_warm_shapes,handle_stale_served,host_filter_expr,@@ -1134,6 +1138,34 @@ class TestVolumeFloor(BaseTest):assert can_use_lazy_precompute(runner, log_prefix="web_overview") is False+class TestRevalidationEventsTable(SimpleTestCase):+ def setUp(self) -> None:+ TEST_clear_clients()+ self.addCleanup(TEST_clear_clients)++ def test_rebuilds_keep_the_events_table_and_debounce_each_table_separately(self) -> None:+ team = Team(id=44, timezone="UTC")+ query = WebOverviewQuery(dateRange=DateRange(date_from="-7d"), properties=[])+ with (+ mock.patch(+ "products.web_analytics.backend.tasks.lazy_precompute_revalidation"+ ".revalidate_web_analytics_precompute.apply_async"+ ) as enqueue,+ new_events_schema_for_request(),+ ):+ for native in (False, True, False, True):+ with events_route_scope(HogQLQueryModifiers(useNewEventsSchema=native)):+ enqueue_stale_revalidation(team=team, query=query, family="web_overview")++ assert [+ call.kwargs["kwargs"]["query"]["modifiers"]["useNewEventsSchema"] for call in enqueue.call_args_list+ ] == [+ False,+ True,+ ]+ assert query.modifiers is None++class TestStaleRevalidationEnqueue(BaseTest):def setUp(self):super().setUp()diff --git a/products/web_analytics/backend/hogql_queries/test/test_web_vitals_timeseries_lazy_precompute.py b/products/web_analytics/backend/hogql_queries/test/test_web_vitals_timeseries_lazy_precompute.pyindex f90c7f6f94cf..d94ec68a62b5 100644--- a/products/web_analytics/backend/hogql_queries/test/test_web_vitals_timeseries_lazy_precompute.py+++ b/products/web_analytics/backend/hogql_queries/test/test_web_vitals_timeseries_lazy_precompute.py@@ -31,7 +31,6 @@ from posthog.api.services.query import process_query_modelfrom posthog.clickhouse.client import sync_executefrom posthog.clickhouse.query_tagging import Feature, Product, reset_query_tags, tag_queriesfrom posthog.hogql_queries.query_runner import get_query_runner_or_none-from posthog.models.instance_setting import override_instance_configfrom products.analytics_platform.backend.models.preaggregation_job import PreaggregationJobfrom products.product_analytics.backend.facade.queries import TrendsQueryRunner@@ -78,29 +77,25 @@ def _vitals_query(**overrides: Any) -> WebVitalsQuery:class TestVitalsTimeseriesGate(APIBaseTest):@parameterized.expand([(path, outer) for path in ("runner", "source_fallback") for outer in (False, True)])- @override_settings(CLICKHOUSE_HOGQL_USE_NEW_EVENTS_SCHEMA=False, CLICKHOUSE_NATIVE_EVENTS_HOST="native.example.com")- def test_wrapper_cluster_choice_overrides_the_source(self, path: str, outer: bool) -> None:+ @override_settings(CLICKHOUSE_HOGQL_USE_NEW_EVENTS_SCHEMA=False)+ def test_wrapper_table_choice_overrides_the_source(self, path: str, outer: bool) -> None:query = _vitals_query(- modifiers=HogQLQueryModifiers(useNewEventsCluster=outer),- source_overrides={"modifiers": HogQLQueryModifiers(useNewEventsCluster=not outer)},+ modifiers=HogQLQueryModifiers(useNewEventsSchema=outer),+ source_overrides={"modifiers": HogQLQueryModifiers(useNewEventsSchema=not outer)},)- with (- override_instance_config("CLICKHOUSE_NATIVE_EVENTS_HOST", "native.example.com"),- override_instance_config("CLICKHOUSE_NATIVE_EVENTS_TEAMS", [self.team.pk]),- ):- if path == "runner":- runner = WebVitalsQueryRunner(query=query, team=self.team)- else:- with (- patch(- "products.web_analytics.backend.hogql_queries.web_vitals_timeseries_lazy_precompute.is_vitals_precompute_enabled_for_team",- return_value=False,- ),- patch("posthog.api.services.query._run_query_runner", return_value={}) as run_source,- ):- process_query_model(self.team, query)- runner = run_source.call_args.args[0]- self.assertIs(bool(runner.modifiers.useNewEventsCluster), outer)+ if path == "runner":+ runner = WebVitalsQueryRunner(query=query, team=self.team)+ else:+ with (+ patch(+ "products.web_analytics.backend.hogql_queries.web_vitals_timeseries_lazy_precompute.is_vitals_precompute_enabled_for_team",+ return_value=False,+ ),+ patch("posthog.api.services.query._run_query_runner", return_value={}) as run_source,+ ):+ process_query_model(self.team, query)+ runner = run_source.call_args.args[0]+ self.assertIs(bool(runner.modifiers.useNewEventsSchema), outer)def test_canonical_tab_shape_is_servable(self) -> None:assert vitals_timeseries_percentile(_vitals_query()) == "p90"diff --git a/products/web_analytics/backend/hogql_queries/web_lazy_precompute_common.py b/products/web_analytics/backend/hogql_queries/web_lazy_precompute_common.pyindex b5e7d3c8564a..9d733e662272 100644--- a/products/web_analytics/backend/hogql_queries/web_lazy_precompute_common.py+++ b/products/web_analytics/backend/hogql_queries/web_lazy_precompute_common.py@@ -25,7 +25,7 @@ import structlogimport posthoganalyticsfrom prometheus_client import Counter-from posthog.schema import SessionsV2JoinMode, WebAnalyticsPreComputeStrategy+from posthog.schema import HogQLQueryModifiers, SessionsV2JoinMode, WebAnalyticsPreComputeStrategyfrom posthog.hogql import astfrom posthog.hogql.modifiers import create_default_modifiers_for_team@@ -36,6 +36,7 @@ from posthog import redisfrom posthog.clickhouse.query_tagging import clear_tag, get_query_tag_value, tag_queriesfrom posthog.hogql_queries.query_runner import ExecutionModefrom posthog.models import Team+from posthog.models.event.new_events_schema import use_new_events_schemafrom products.access_control.backend.facade.api import team_has_property_access_rulesfrom products.analytics_platform.backend.lazy_computation.lazy_computation_executor import (@@ -528,12 +529,17 @@ def enqueue_stale_revalidation(*, team: Team, query: Any, family: str, debounce_revalidate_web_analytics_precompute,)+ modifiers = (query.modifiers or HogQLQueryModifiers()).model_copy()+ modifiers.useNewEventsSchema = use_new_events_schema(team.pk, modifiers)+ query = query.model_copy(update={"modifiers": modifiers})try:client = redis.get_client()shape_part = compute_filters_eligibility_hash(query, team.timezone)[:16]if debounce_extra:shape_part = hashlib.sha256(f"{shape_part}:{debounce_extra}".encode()).hexdigest()[:16]debounce_key = f"web_swr_reval:{team.id}:{family}:{shape_part}"+ if modifiers.useNewEventsSchema:+ debounce_key += ":events_json"if not client.set(debounce_key, "1", ex=REVALIDATION_DEBOUNCE_SECONDS, nx=True):returnbudget_key = f"web_swr_reval_budget:{team.id}"diff --git a/services/mcp/src/api/generated.ts b/services/mcp/src/api/generated.tsindex 011d2f67ca46..170ec6ffe9de 100644--- a/services/mcp/src/api/generated.ts+++ b/services/mcp/src/api/generated.ts@@ -2808,9 +2808,7 @@ export namespace Schemas {/** Remove provably redundant casts and nullability wrappers (e.g. `toString(String)`, `assumeNotNull(non_nullable)`, dead `ifNull` fallbacks) using inferred expression types */typeAwareCastSimplification?: boolean | null;useMaterializedViews?: boolean | null;- /** Read this query from the native events cluster and native JSON events table (`true`), or keep its existing connection (`false` or unset). This is an internal rollout switch for eligible projects. PostHog staff manage eligibility through instance settings. Each query must opt in; project defaults cannot enable it. Only synchronous HogQLQuery SQL comparisons are supported. */- useNewEventsCluster?: boolean | null;- /** Read events from the native JSON events table (`true`) or the legacy events table (`false`). When unset, the project's stored value applies, then the `CLICKHOUSE_HOGQL_USE_NEW_EVENTS_SCHEMA` instance settings. This is an internal rollout switch. PostHog staff set the project value in Django admin and the project settings API ignores it. */+ /** Read events from the native JSON events table (`true`) or the legacy events table (`false`). When unset, a staff person's `ph_new_events_schema` cookie applies, then the project's listing in the `CLICKHOUSE_HOGQL_USE_NEW_EVENTS_SCHEMA_TEAMS` instance setting, then the `CLICKHOUSE_HOGQL_USE_NEW_EVENTS_SCHEMA` instance setting. This is an internal rollout switch, and the project settings API ignores it. */useNewEventsSchema?: boolean | null;usePreaggregatedIntermediateResults?: boolean | null;/** Try to automatically convert HogQL queries to use preaggregated tables at the AST level * */diff --git a/services/mcp/src/generated/batch_exports/api.ts b/services/mcp/src/generated/batch_exports/api.tsindex 6afad9096d0c..6a9e0094ce86 100644--- a/services/mcp/src/generated/batch_exports/api.ts+++ b/services/mcp/src/generated/batch_exports/api.ts@@ -681,17 +681,11 @@ export const BatchExportsCreateBody = () => zod'Remove provably redundant casts and nullability wrappers (e.g. `toString(String)`, `assumeNotNull(non_nullable)`, dead `ifNull` fallbacks) using inferred expression types'),useMaterializedViews: zod.union([zod.boolean(), zod.null()]).optional(),- useNewEventsCluster: zod- .union([zod.boolean(), zod.null()])- .optional()- .describe(- 'Read this query from the native events cluster and native JSON events table (`true`), or keep its existing connection (`false` or unset). This is an internal rollout switch for eligible projects. PostHog staff manage eligibility through instance settings. Each query must opt in; project defaults cannot enable it. Only synchronous HogQLQuery SQL comparisons are supported.'- ),useNewEventsSchema: zod.union([zod.boolean(), zod.null()]).optional().describe(- "Read events from the native JSON events table (`true`) or the legacy events table (`false`). When unset, the project's stored value applies, then the `CLICKHOUSE_HOGQL_USE_NEW_EVENTS_SCHEMA` instance settings. This is an internal rollout switch. PostHog staff set the project value in Django admin and the project settings API ignores it."+ "Read events from the native JSON events table (`true`) or the legacy events table (`false`). When unset, a staff person's `ph_new_events_schema` cookie applies, then the project's listing in the `CLICKHOUSE_HOGQL_USE_NEW_EVENTS_SCHEMA_TEAMS` instance setting, then the `CLICKHOUSE_HOGQL_USE_NEW_EVENTS_SCHEMA` instance setting. This is an internal rollout switch, and the project settings API ignores it."),usePreaggregatedIntermediateResults: zod.union([zod.boolean(), zod.null()]).optional(),usePreaggregatedTableTransforms: zod@@ -1430,17 +1424,11 @@ export const BatchExportsPartialUpdateBody = () => zod'Remove provably redundant casts and nullability wrappers (e.g. `toString(String)`, `assumeNotNull(non_nullable)`, dead `ifNull` fallbacks) using inferred expression types'),useMaterializedViews: zod.union([zod.boolean(), zod.null()]).optional(),- useNewEventsCluster: zod- .union([zod.boolean(), zod.null()])- .optional()- .describe(- 'Read this query from the native events cluster and native JSON events table (`true`), or keep its existing connection (`false` or unset). This is an internal rollout switch for eligible projects. PostHog staff manage eligibility through instance settings. Each query must opt in; project defaults cannot enable it. Only synchronous HogQLQuery SQL comparisons are supported.'- ),useNewEventsSchema: zod.union([zod.boolean(), zod.null()]).optional().describe(- "Read events from the native JSON events table (`true`) or the legacy events table (`false`). When unset, the project's stored value applies, then the `CLICKHOUSE_HOGQL_USE_NEW_EVENTS_SCHEMA` instance settings. This is an internal rollout switch. PostHog staff set the project value in Django admin and the project settings API ignores it."+ "Read events from the native JSON events table (`true`) or the legacy events table (`false`). When unset, a staff person's `ph_new_events_schema` cookie applies, then the project's listing in the `CLICKHOUSE_HOGQL_USE_NEW_EVENTS_SCHEMA_TEAMS` instance setting, then the `CLICKHOUSE_HOGQL_USE_NEW_EVENTS_SCHEMA` instance setting. This is an internal rollout switch, and the project settings API ignores it."),usePreaggregatedIntermediateResults: zod.union([zod.boolean(), zod.null()]).optional(),usePreaggregatedTableTransforms: zod@@ -1910,17 +1898,11 @@ export const FileDownloadBatchExportsCreateBody = () => zod.union(['Remove provably redundant casts and nullability wrappers (e.g. `toString(String)`, `assumeNotNull(non_nullable)`, dead `ifNull` fallbacks) using inferred expression types'),useMaterializedViews: zod.union([zod.boolean(), zod.null()]).optional(),- useNewEventsCluster: zod- .union([zod.boolean(), zod.null()])- .optional()- .describe(- 'Read this query from the native events cluster and native JSON events table (`true`), or keep its existing connection (`false` or unset). This is an internal rollout switch for eligible projects. PostHog staff manage eligibility through instance settings. Each query must opt in; project defaults cannot enable it. Only synchronous HogQLQuery SQL comparisons are supported.'- ),useNewEventsSchema: zod.union([zod.boolean(), zod.null()]).optional().describe(- "Read events from the native JSON events table (`true`) or the legacy events table (`false`). When unset, the project's stored value applies, then the `CLICKHOUSE_HOGQL_USE_NEW_EVENTS_SCHEMA` instance settings. This is an internal rollout switch. PostHog staff set the project value in Django admin and the project settings API ignores it."+ "Read events from the native JSON events table (`true`) or the legacy events table (`false`). When unset, a staff person's `ph_new_events_schema` cookie applies, then the project's listing in the `CLICKHOUSE_HOGQL_USE_NEW_EVENTS_SCHEMA_TEAMS` instance setting, then the `CLICKHOUSE_HOGQL_USE_NEW_EVENTS_SCHEMA` instance setting. This is an internal rollout switch, and the project settings API ignores it."),usePreaggregatedIntermediateResults: zod.union([zod.boolean(), zod.null()]).optional(),usePreaggregatedTableTransforms: zod@@ -2216,17 +2198,11 @@ export const FileDownloadBatchExportsCountRowsCreateBody = () => zod'Remove provably redundant casts and nullability wrappers (e.g. `toString(String)`, `assumeNotNull(non_nullable)`, dead `ifNull` fallbacks) using inferred expression types'),useMaterializedViews: zod.union([zod.boolean(), zod.null()]).optional(),- useNewEventsCluster: zod- .union([zod.boolean(), zod.null()])- .optional()- .describe(- 'Read this query from the native events cluster and native JSON events table (`true`), or keep its existing connection (`false` or unset). This is an internal rollout switch for eligible projects. PostHog staff manage eligibility through instance settings. Each query must opt in; project defaults cannot enable it. Only synchronous HogQLQuery SQL comparisons are supported.'- ),useNewEventsSchema: zod.union([zod.boolean(), zod.null()]).optional().describe(- "Read events from the native JSON events table (`true`) or the legacy events table (`false`). When unset, the project's stored value applies, then the `CLICKHOUSE_HOGQL_USE_NEW_EVENTS_SCHEMA` instance settings. This is an internal rollout switch. PostHog staff set the project value in Django admin and the project settings API ignores it."+ "Read events from the native JSON events table (`true`) or the legacy events table (`false`). When unset, a staff person's `ph_new_events_schema` cookie applies, then the project's listing in the `CLICKHOUSE_HOGQL_USE_NEW_EVENTS_SCHEMA_TEAMS` instance setting, then the `CLICKHOUSE_HOGQL_USE_NEW_EVENTS_SCHEMA` instance setting. This is an internal rollout switch, and the project settings API ignores it."),usePreaggregatedIntermediateResults: zod.union([zod.boolean(), zod.null()]).optional(),usePreaggregatedTableTransforms: zoddiff --git a/services/mcp/tests/unit/__snapshots__/tool-schemas/batch-export-create.json b/services/mcp/tests/unit/__snapshots__/tool-schemas/batch-export-create.jsonindex 5d27745681e1..b30e4d3790c1 100644--- a/services/mcp/tests/unit/__snapshots__/tool-schemas/batch-export-create.json+++ b/services/mcp/tests/unit/__snapshots__/tool-schemas/batch-export-create.json@@ -1076,17 +1076,6 @@}]},- "useNewEventsCluster": {- "anyOf": [- {- "type": "boolean"- },- {- "type": "null"- }- ],- "description": "Read this query from the native events cluster and native JSON events table (`true`), or keep its existing connection (`false` or unset). This is an internal rollout switch for eligible projects. PostHog staff manage eligibility through instance settings. Each query must opt in; project defaults cannot enable it. Only synchronous HogQLQuery SQL comparisons are supported."- },"useNewEventsSchema": {"anyOf": [{@@ -1096,7 +1085,7 @@"type": "null"}],- "description": "Read events from the native JSON events table (`true`) or the legacy events table (`false`). When unset, the project's stored value applies, then the `CLICKHOUSE_HOGQL_USE_NEW_EVENTS_SCHEMA` instance settings. This is an internal rollout switch. PostHog staff set the project value in Django admin and the project settings API ignores it."+ "description": "Read events from the native JSON events table (`true`) or the legacy events table (`false`). When unset, a staff person's `ph_new_events_schema` cookie applies, then the project's listing in the `CLICKHOUSE_HOGQL_USE_NEW_EVENTS_SCHEMA_TEAMS` instance setting, then the `CLICKHOUSE_HOGQL_USE_NEW_EVENTS_SCHEMA` instance setting. This is an internal rollout switch, and the project settings API ignores it."},"usePreaggregatedIntermediateResults": {"anyOf": [diff --git a/services/mcp/tests/unit/__snapshots__/tool-schemas/batch-export-update.json b/services/mcp/tests/unit/__snapshots__/tool-schemas/batch-export-update.jsonindex e8014b818963..92e3c09c1c56 100644--- a/services/mcp/tests/unit/__snapshots__/tool-schemas/batch-export-update.json+++ b/services/mcp/tests/unit/__snapshots__/tool-schemas/batch-export-update.json@@ -1075,17 +1075,6 @@}]},- "useNewEventsCluster": {- "anyOf": [- {- "type": "boolean"- },- {- "type": "null"- }- ],- "description": "Read this query from the native events cluster and native JSON events table (`true`), or keep its existing connection (`false` or unset). This is an internal rollout switch for eligible projects. PostHog staff manage eligibility through instance settings. Each query must opt in; project defaults cannot enable it. Only synchronous HogQLQuery SQL comparisons are supported."- },"useNewEventsSchema": {"anyOf": [{@@ -1095,7 +1084,7 @@"type": "null"}],- "description": "Read events from the native JSON events table (`true`) or the legacy events table (`false`). When unset, the project's stored value applies, then the `CLICKHOUSE_HOGQL_USE_NEW_EVENTS_SCHEMA` instance settings. This is an internal rollout switch. PostHog staff set the project value in Django admin and the project settings API ignores it."+ "description": "Read events from the native JSON events table (`true`) or the legacy events table (`false`). When unset, a staff person's `ph_new_events_schema` cookie applies, then the project's listing in the `CLICKHOUSE_HOGQL_USE_NEW_EVENTS_SCHEMA_TEAMS` instance setting, then the `CLICKHOUSE_HOGQL_USE_NEW_EVENTS_SCHEMA` instance setting. This is an internal rollout switch, and the project settings API ignores it."},"usePreaggregatedIntermediateResults": {"anyOf": [diff --git a/services/mcp/tests/unit/__snapshots__/tool-schemas/file-download-batch-exports-count-rows-create.json b/services/mcp/tests/unit/__snapshots__/tool-schemas/file-download-batch-exports-count-rows-create.jsonindex 778b5c021315..442048d51b07 100644--- a/services/mcp/tests/unit/__snapshots__/tool-schemas/file-download-batch-exports-count-rows-create.json+++ b/services/mcp/tests/unit/__snapshots__/tool-schemas/file-download-batch-exports-count-rows-create.json@@ -551,17 +551,6 @@}]},- "useNewEventsCluster": {- "anyOf": [- {- "type": "boolean"- },- {- "type": "null"- }- ],- "description": "Read this query from the native events cluster and native JSON events table (`true`), or keep its existing connection (`false` or unset). This is an internal rollout switch for eligible projects. PostHog staff manage eligibility through instance settings. Each query must opt in; project defaults cannot enable it. Only synchronous HogQLQuery SQL comparisons are supported."- },"useNewEventsSchema": {"anyOf": [{@@ -571,7 +560,7 @@"type": "null"}],- "description": "Read events from the native JSON events table (`true`) or the legacy events table (`false`). When unset, the project's stored value applies, then the `CLICKHOUSE_HOGQL_USE_NEW_EVENTS_SCHEMA` instance settings. This is an internal rollout switch. PostHog staff set the project value in Django admin and the project settings API ignores it."+ "description": "Read events from the native JSON events table (`true`) or the legacy events table (`false`). When unset, a staff person's `ph_new_events_schema` cookie applies, then the project's listing in the `CLICKHOUSE_HOGQL_USE_NEW_EVENTS_SCHEMA_TEAMS` instance setting, then the `CLICKHOUSE_HOGQL_USE_NEW_EVENTS_SCHEMA` instance setting. This is an internal rollout switch, and the project settings API ignores it."},"usePreaggregatedIntermediateResults": {"anyOf": [diff --git a/services/mcp/tests/unit/__snapshots__/tool-schemas/file-download-batch-exports-create.json b/services/mcp/tests/unit/__snapshots__/tool-schemas/file-download-batch-exports-create.jsonindex eb97d37b5660..b07225dde207 100644--- a/services/mcp/tests/unit/__snapshots__/tool-schemas/file-download-batch-exports-create.json+++ b/services/mcp/tests/unit/__snapshots__/tool-schemas/file-download-batch-exports-create.json@@ -780,17 +780,6 @@}]},- "useNewEventsCluster": {- "anyOf": [- {- "type": "boolean"- },- {- "type": "null"- }- ],- "description": "Read this query from the native events cluster and native JSON events table (`true`), or keep its existing connection (`false` or unset). This is an internal rollout switch for eligible projects. PostHog staff manage eligibility through instance settings. Each query must opt in; project defaults cannot enable it. Only synchronous HogQLQuery SQL comparisons are supported."- },"useNewEventsSchema": {"anyOf": [{@@ -800,7 +789,7 @@"type": "null"}],- "description": "Read events from the native JSON events table (`true`) or the legacy events table (`false`). When unset, the project's stored value applies, then the `CLICKHOUSE_HOGQL_USE_NEW_EVENTS_SCHEMA` instance settings. This is an internal rollout switch. PostHog staff set the project value in Django admin and the project settings API ignores it."+ "description": "Read events from the native JSON events table (`true`) or the legacy events table (`false`). When unset, a staff person's `ph_new_events_schema` cookie applies, then the project's listing in the `CLICKHOUSE_HOGQL_USE_NEW_EVENTS_SCHEMA_TEAMS` instance setting, then the `CLICKHOUSE_HOGQL_USE_NEW_EVENTS_SCHEMA` instance setting. This is an internal rollout switch, and the project settings API ignores it."},"usePreaggregatedIntermediateResults": {"anyOf": [