Skip to content

:hedgehog: PostHog is the leading platform for building self-driving products. Our developer tools – AI observability, analytics, session replay, flags, experiments, error tracking, logs, and more – capture all the context agents need to diagnose problems, uncover opportunities, and ship fixes. Steer it all from Slack, web, desktop, or the MCP.

feat(hogql): staff switch for the native events table (#122489)

Sandy Spicerd282569d0ada
diff --git a/.agents/skills/writing-clickhouse-queries/references/choosing-a-workload.md b/.agents/skills/writing-clickhouse-queries/references/choosing-a-workload.md
index 2f459b368969..9c4eb22accb3 100644
--- a/.agents/skills/writing-clickhouse-queries/references/choosing-a-workload.md
+++ b/.agents/skills/writing-clickhouse-queries/references/choosing-a-workload.md
@@ -19,13 +19,12 @@ Both feed capacity planning, so pick the workload on purpose.
Answer in order and stop at the first yes.
-| Question | Workload |
-| ------------------------------------------------------------------------------- | -------------------- |
-| Does the query read a table that exists only on the logs cluster? | `LOGS` |
-| Does the query execute an Endpoints (the product) request? | `ENDPOINTS` |
-| Does the query target the native events schema cluster? | `NEW_EVENTS_CLUSTER` |
-| Does a person wait on screen for this result (an app request, a Max chat turn)? | `ONLINE` |
-| Anything else: scheduled, batch, export, backfill, recompute, eval, digest | `OFFLINE` |
+| Question | Workload |
+| ------------------------------------------------------------------------------- | ----------- |
+| Does the query read a table that exists only on the logs cluster? | `LOGS` |
+| Does the query execute an Endpoints (the product) request? | `ENDPOINTS` |
+| Does a person wait on screen for this result (an app request, a Max chat turn)? | `ONLINE` |
+| Anything else: scheduled, batch, export, backfill, recompute, eval, digest | `OFFLINE` |
`ONLINE` is for latency a person feels.
A background job does not get faster with `ONLINE`.
@@ -54,7 +53,7 @@ The host comes from the environment of the pod that runs the code, in [`get_kwar
- `ONLINE` connects to `CLICKHOUSE_HOST`.
- `OFFLINE` connects to `CLICKHOUSE_OFFLINE_CLUSTER_HOST` when the deployment sets it, and to `CLICKHOUSE_HOST` when it does not.
-- `LOGS`, `ENDPOINTS`, and `NEW_EVENTS_CLUSTER` connect to their own hosts.
+- `LOGS` and `ENDPOINTS` connect to their own hosts.
A background worker deployment can point `CLICKHOUSE_HOST` at the offline replicas and set `CLICKHOUSE_DEFAULT_WORKLOAD=OFFLINE`.
Then `ONLINE` and `OFFLINE` from those pods both run on the offline replicas.
diff --git a/docs/internal/web-analytics-query-serving.md b/docs/internal/web-analytics-query-serving.md
index e0c1bfeec7e6..d6f6ce1601ab 100644
--- a/docs/internal/web-analytics-query-serving.md
+++ b/docs/internal/web-analytics-query-serving.md
@@ -66,6 +66,10 @@ request
A miss costs one live-path serve; the background warm makes the next identical request a bucket hit.
The dashboard "enqueues precompute" as a side effect; it never waits on it.
+Requests that select native JSON events keep that table choice in precompute job identities and queued revalidation payloads.
+Native and legacy revalidations use separate debounce keys, so a trial cannot suppress the project's normal rebuild.
+Builders inherit the running query's table choice; an explicit query modifier takes precedence over the staff cookie.
+
## Marketing search performance
Search metrics and PostHog landing-page conversions load in separate requests.
diff --git a/frontend/src/queries/schema.json b/frontend/src/queries/schema.json
index e853c5bb152c..9c8a638e6fb3 100644
--- a/frontend/src/queries/schema.json
+++ b/frontend/src/queries/schema.json
@@ -29960,12 +29960,8 @@
"useMaterializedViews": {
"type": "boolean"
},
- "useNewEventsCluster": {
- "description": "Read this query from the native events cluster and native JSON events table (`true`), or keep its existing connection (`false` or unset). This is an internal rollout switch for eligible projects. PostHog staff manage eligibility through instance settings. Each query must opt in; project defaults cannot enable it. Only synchronous HogQLQuery SQL comparisons are supported.",
- "type": "boolean"
- },
"useNewEventsSchema": {
- "description": "Read events from the native JSON events table (`true`) or the legacy events table (`false`). When unset, the project's stored value applies, then the `CLICKHOUSE_HOGQL_USE_NEW_EVENTS_SCHEMA` instance settings. This is an internal rollout switch. PostHog staff set the project value in Django admin and the project settings API ignores it.",
+ "description": "Read events from the native JSON events table (`true`) or the legacy events table (`false`). When unset, a staff person's `ph_new_events_schema` cookie applies, then the project's listing in the `CLICKHOUSE_HOGQL_USE_NEW_EVENTS_SCHEMA_TEAMS` instance setting, then the `CLICKHOUSE_HOGQL_USE_NEW_EVENTS_SCHEMA` instance setting. This is an internal rollout switch, and the project settings API ignores it.",
"type": "boolean"
},
"usePreaggregatedIntermediateResults": {
diff --git a/frontend/src/queries/schema/schema-general.ts b/frontend/src/queries/schema/schema-general.ts
index a8226ad2b319..60eeec773fc9 100644
--- a/frontend/src/queries/schema/schema-general.ts
+++ b/frontend/src/queries/schema/schema-general.ts
@@ -545,10 +545,8 @@ export interface HogQLQueryModifiers {
materializedColumnsOptimizationMode?: 'disabled' | 'optimized'
propertyGroupsMode?: 'enabled' | 'disabled' | 'optimized'
useMaterializedViews?: boolean
- /** Read events from the native JSON events table (`true`) or the legacy events table (`false`). When unset, the project's stored value applies, then the `CLICKHOUSE_HOGQL_USE_NEW_EVENTS_SCHEMA` instance settings. This is an internal rollout switch. PostHog staff set the project value in Django admin and the project settings API ignores it. */
+ /** Read events from the native JSON events table (`true`) or the legacy events table (`false`). When unset, a staff person's `ph_new_events_schema` cookie applies, then the project's listing in the `CLICKHOUSE_HOGQL_USE_NEW_EVENTS_SCHEMA_TEAMS` instance setting, then the `CLICKHOUSE_HOGQL_USE_NEW_EVENTS_SCHEMA` instance setting. This is an internal rollout switch, and the project settings API ignores it. */
useNewEventsSchema?: boolean
- /** Read this query from the native events cluster and native JSON events table (`true`), or keep its existing connection (`false` or unset). This is an internal rollout switch for eligible projects. PostHog staff manage eligibility through instance settings. Each query must opt in; project defaults cannot enable it. Only synchronous HogQLQuery SQL comparisons are supported. */
- useNewEventsCluster?: boolean
customChannelTypeRules?: CustomChannelRule[]
customBotDefinitions?: CustomBotRule[]
/** Do not treat a missing user agent as automation on cookieless events. Positive bot signals and custom project rules still apply. Resolved server-side; not intended to be set by clients. */
diff --git a/posthog/api/instance_settings.py b/posthog/api/instance_settings.py
index 726cc1e94f7e..d01f6fd263e0 100644
--- a/posthog/api/instance_settings.py
+++ b/posthog/api/instance_settings.py
@@ -125,10 +125,11 @@ class InstanceSettingsSerializer(serializers.Serializer):
is_secret = serializers.BooleanField(read_only=True)
def validate_value(self, value: Any) -> Any:
- if self.instance is not None and self.instance.key == "CLICKHOUSE_NATIVE_EVENTS_HOST":
- if not isinstance(value, str) or (value and not re.fullmatch(r"[A-Za-z0-9][A-Za-z0-9.-]{0,252}", value)):
+ if self.instance is not None and self.instance.key == "CLICKHOUSE_HOGQL_USE_NEW_EVENTS_SCHEMA_TEAMS":
+ # Every query parses this list, so a value that fails to parse would fail every query on the instance.
+ if not isinstance(value, str) or not re.fullmatch(r"\s*(\d+\s*(,\s*\d+\s*)*)?", value):
raise serializers.ValidationError(
- "Enter a hostname without a scheme, port or path, or leave it empty to disable the route."
+ "Enter project IDs separated by commas, or leave it empty to switch no project."
)
return value
diff --git a/posthog/api/team/query_modifiers.py b/posthog/api/team/query_modifiers.py
index ccab259e8357..e6cd46a7adba 100644
--- a/posthog/api/team/query_modifiers.py
+++ b/posthog/api/team/query_modifiers.py
@@ -19,9 +19,6 @@ def merge_modifiers(requested: dict | None, existing: object) -> dict | None:
stored = existing if isinstance(existing, dict) else {}
if requested is not None:
return {**stored, **requested}
- # PostHog staff set useNewEventsSchema per project, so a client that clears its modifiers keeps it.
- if "useNewEventsSchema" in stored:
- return {"useNewEventsSchema": stored["useNewEventsSchema"]}
return None
@@ -32,10 +29,10 @@ def validate_modifiers(value: dict | None) -> dict | None:
if not isinstance(value, dict):
raise exceptions.ValidationError("Must provide a dictionary or None.")
- # The native JSON events table is an internal rollout switch that PostHog staff set per project in Django
- # admin. Dropping the key keeps a client from setting or clearing it, including a settings page that echoes
- # the whole modifiers dict back.
- value = {key: item for key, item in value.items() if key not in {"useNewEventsSchema", "useNewEventsCluster"}}
+ # The native JSON events table is switched per project by an instance setting, never by a stored modifier.
+ # Dropping the key keeps a client from storing one, including a settings page that echoes the whole
+ # modifiers dict back.
+ value = {key: item for key, item in value.items() if key != "useNewEventsSchema"}
if "bounceRateDurationSeconds" in value:
bounce_rate = value["bounceRateDurationSeconds"]
diff --git a/posthog/api/test/test_instance_settings.py b/posthog/api/test/test_instance_settings.py
index 8c486879eb3d..24514f7dbdfd 100644
--- a/posthog/api/test/test_instance_settings.py
+++ b/posthog/api/test/test_instance_settings.py
@@ -111,21 +111,22 @@ class TestSecretSettingsCoverage(unittest.TestCase):
self.assertEqual(_redact_if_secret("EMAIL_HOST", value), value)
-class TestHostSettingValidation(SimpleTestCase):
+class TestNewEventsSchemaTeamsValidation(SimpleTestCase):
@parameterized.expand(
[
- ("native.example.com", True),
- ("localhost", True),
("", True),
- ("https://native.example.com", False),
- ("native.example.com:8443", False),
- ("native.example.com/path", False),
- (True, False),
+ ("2", True),
+ (" 2, 44 ,123 ", True),
+ ("2;44", False),
+ ("team 2", False),
+ ([2], False),
]
)
- def test_host_setting_accepts_hostnames_only(self, value: object, valid: bool) -> None:
+ def test_project_list_accepts_comma_separated_ids_only(self, value: object, valid: bool) -> None:
serializer = InstanceSettingsSerializer(
- instance=InstanceSettingHelper(key="CLICKHOUSE_NATIVE_EVENTS_HOST"), data={"value": value}, partial=True
+ instance=InstanceSettingHelper(key="CLICKHOUSE_HOGQL_USE_NEW_EVENTS_SCHEMA_TEAMS"),
+ data={"value": value},
+ partial=True,
)
self.assertEqual(serializer.is_valid(), valid, serializer.errors)
@@ -215,8 +216,7 @@ class TestInstanceSettings(APIBaseTest):
@parameterized.expand(
[
("AUTO_START_ASYNC_MIGRATIONS", False, True),
- ("CLICKHOUSE_NATIVE_EVENTS_TEAMS", [], [44]),
- ("CLICKHOUSE_NATIVE_EVENTS_HOST", "", "native.example.com"),
+ ("CLICKHOUSE_HOGQL_USE_NEW_EVENTS_SCHEMA_TEAMS", "", "2"),
]
)
def test_update_setting(self, key: str, before: object, after: object) -> None:
@@ -306,8 +306,7 @@ class TestInstanceSettings(APIBaseTest):
@parameterized.expand(
[
("AUTO_START_ASYNC_MIGRATIONS", False, True),
- ("CLICKHOUSE_NATIVE_EVENTS_TEAMS", [], [44]),
- ("CLICKHOUSE_NATIVE_EVENTS_HOST", "", "native.example.com"),
+ ("CLICKHOUSE_HOGQL_USE_NEW_EVENTS_SCHEMA_TEAMS", "", "2"),
]
)
def test_non_staff_user_cant_update(self, key: str, before: object, after: object) -> None:
diff --git a/posthog/api/test/test_team.py b/posthog/api/test/test_team.py
index bb894d13aa55..fc93166638d3 100644
--- a/posthog/api/test/test_team.py
+++ b/posthog/api/test/test_team.py
@@ -1645,38 +1645,26 @@ def team_api_test_factory():
{"id": "test", "channel_type": "Direct", "combiner": "AND", "items": []}
]
- @parameterized.expand([("useNewEventsSchema",), ("useNewEventsCluster",)])
- def test_events_routing_modifiers_are_not_client_writable(self, modifier: str) -> None:
+ def test_the_events_schema_modifier_is_not_client_writable(self) -> None:
response = self.client.patch(
f"/api/environments/{self.team.id}",
- {"modifiers": {modifier: True}},
+ {"modifiers": {"useNewEventsSchema": True}},
)
assert response.status_code == status.HTTP_200_OK, response.json()
- assert modifier not in response.json()["modifiers"]
+ assert "useNewEventsSchema" not in response.json()["modifiers"]
- self.team.modifiers = {modifier: True}
+ self.team.modifiers = {"useNewEventsSchema": True}
self.team.save()
response = self.client.patch(
f"/api/environments/{self.team.id}",
- {"modifiers": {modifier: False, "bounceRateDurationSeconds": 30}},
+ {"modifiers": {"useNewEventsSchema": False, "bounceRateDurationSeconds": 30}},
)
assert response.status_code == status.HTTP_200_OK, response.json()
- assert response.json()["modifiers"] == {modifier: True, "bounceRateDurationSeconds": 30}
+ assert response.json()["modifiers"] == {"useNewEventsSchema": True, "bounceRateDurationSeconds": 30}
@parameterized.expand(
[
- (
- "environments",
- {"useNewEventsSchema": True, "bounceRateDurationSeconds": 30},
- None,
- {"useNewEventsSchema": True},
- ),
- (
- "projects",
- {"useNewEventsSchema": True, "bounceRateDurationSeconds": 30},
- None,
- {"useNewEventsSchema": True},
- ),
+ ("environments", {"useNewEventsSchema": True, "bounceRateDurationSeconds": 30}, None, None),
("environments", True, None, None),
("projects", ["useNewEventsSchema"], None, None),
("environments", True, {"bounceRateDurationSeconds": 30}, {"bounceRateDurationSeconds": 30}),
diff --git a/posthog/clickhouse/client/connection.py b/posthog/clickhouse/client/connection.py
index 586900ac49ea..1b40bbac2c3d 100644
--- a/posthog/clickhouse/client/connection.py
+++ b/posthog/clickhouse/client/connection.py
@@ -455,18 +455,6 @@ def get_kwargs_for_client(
creds = get_clickhouse_creds(ch_user)
base_kwargs = {"user": creds.user, "password": creds.password}
- if workload == Workload.NEW_EVENTS_CLUSTER:
- from posthog.models.event.new_events_schema import get_new_events_cluster_host # noqa: PLC0415
-
- host = get_new_events_cluster_host()
- if not host:
- raise ValueError("Native events ClickHouse host is not configured")
- return {
- **get_kwargs_for_client(Workload.DEFAULT, team_id, readonly, ch_user),
- "host": host,
- "verify": settings.CLICKHOUSE_VERIFY,
- }
-
if team_id is not None and str(team_id) in settings.CLICKHOUSE_PER_TEAM_SETTINGS:
user_settings = settings.CLICKHOUSE_PER_TEAM_SETTINGS[str(team_id)]
return {**base_kwargs, **user_settings}
diff --git a/posthog/clickhouse/client/execute.py b/posthog/clickhouse/client/execute.py
index 9523cd09b282..875d27a978ae 100644
--- a/posthog/clickhouse/client/execute.py
+++ b/posthog/clickhouse/client/execute.py
@@ -455,7 +455,7 @@ def sync_execute(
workload = Workload.OFFLINE if is_api_key_auth else Workload.ONLINE
ch_user = ClickHouseUser.API if is_api_key_auth else ClickHouseUser.APP
- if tags.workload == Workload.ENDPOINTS and workload not in (Workload.LOGS, Workload.NEW_EVENTS_CLUSTER):
+ if tags.workload == Workload.ENDPOINTS and workload != Workload.LOGS:
workload = Workload.ENDPOINTS
if workload == Workload.DEFAULT:
diff --git a/posthog/clickhouse/client/test/test_connection.py b/posthog/clickhouse/client/test/test_connection.py
index bb5eb2175ed9..97020e9f90e3 100644
--- a/posthog/clickhouse/client/test/test_connection.py
+++ b/posthog/clickhouse/client/test/test_connection.py
@@ -143,51 +143,6 @@ def test_signals_readonly_queries_keep_dedicated_credentials_and_offline_host(
assert kwargs == {"user": "shared-readonly", "password": "shared-password"}
-def test_native_events_pool_is_separate_from_legacy_pool(settings, monkeypatch):
- from posthog.hogql_queries.utils.parallel import run_in_parallel_threads
- from posthog.models.event.new_events_schema import new_events_cluster_host_scope
-
- settings.CLICKHOUSE_NATIVE_EVENTS_HOST = "env.example.com"
- host = "native.example.com"
- monkeypatch.setattr("posthog.models.event.new_events_schema.get_instance_setting", lambda key: host)
- settings.CLICKHOUSE_VERIFY = True
- settings.CLICKHOUSE_PER_TEAM_SETTINGS = {}
- legacy = get_pool(Workload.ONLINE, team_id=44)
- native = get_pool(Workload.NEW_EVENTS_CLUSTER, team_id=44)
- assert native is not legacy
- assert native.connection_args["host"] == "native.example.com"
- with new_events_cluster_host_scope():
- assert get_pool(Workload.NEW_EVENTS_CLUSTER, team_id=44) is native
- host = "replacement.example.com"
- pools = []
- run_in_parallel_threads([lambda: pools.append(get_pool(Workload.NEW_EVENTS_CLUSTER, team_id=44))])
- assert pools == [native]
- replacement = get_pool(Workload.NEW_EVENTS_CLUSTER, team_id=44)
- assert replacement is not native
- assert replacement.connection_args["host"] == host
-
-
[email protected]("readonly", [False, True])
[email protected]("team_override", [False, True])
-def test_native_events_route_preserves_resolved_credentials(settings, readonly, team_override):
- settings.CLICKHOUSE_NATIVE_EVENTS_HOST = "native.example.com"
- settings.CLICKHOUSE_VERIFY = True
- settings.CLICKHOUSE_PER_TEAM_SETTINGS = (
- {"44": {"host": "legacy.example.com", "user": "team-user", "password": "example"}} if team_override else {}
- )
- settings.READONLY_CLICKHOUSE_USER = "reader"
- settings.READONLY_CLICKHOUSE_PASSWORD = "example-readonly"
- baseline = connection.get_kwargs_for_client(Workload.DEFAULT, team_id=44, readonly=readonly)
- native = connection.get_kwargs_for_client(Workload.NEW_EVENTS_CLUSTER, team_id=44, readonly=readonly)
- assert native == {**baseline, "host": "native.example.com", "verify": True}
-
-
-def test_native_events_workload_without_host_fails_closed(settings):
- settings.CLICKHOUSE_NATIVE_EVENTS_HOST = None
- with pytest.raises(ValueError, match="Native events ClickHouse host is not configured"):
- get_pool(Workload.NEW_EVENTS_CLUSTER)
-
-
def test_read_password_reads_file_fresh_and_stripped(tmp_path):
token = tmp_path / "token"
token.write_text("tok-0\n")
@@ -310,33 +265,29 @@ def test_deletion_executor_credentials_do_not_fall_back_to_default(monkeypatch):
get_clickhouse_creds(ClickHouseUser.DELETION_EXECUTOR)
[email protected]("workload", [Workload.ONLINE, Workload.NEW_EVENTS_CLUSTER])
-def test_file_backed_pool_is_stable_across_credential_rotation(settings, monkeypatch, tmp_path, workload):
- settings.CLICKHOUSE_NATIVE_EVENTS_HOST = "native.example.com"
+def test_file_backed_pool_is_stable_across_credential_rotation(settings, monkeypatch, tmp_path):
settings.CLICKHOUSE_OFFLINE_CLUSTER_HOST = None
token = tmp_path / "token"
token.write_text("tok-0")
_file_backed_default(monkeypatch, token)
- pool = get_pool(workload)
+ pool = get_pool(Workload.ONLINE)
assert isinstance(pool, RefreshingChPool)
token.write_text("tok-1") # rotate the projected token in place
- assert get_pool(workload) is pool
+ assert get_pool(Workload.ONLINE) is pool
assert make_ch_pool.cache_info().currsize == 1
[email protected]("workload", [Workload.ONLINE, Workload.NEW_EVENTS_CLUSTER])
-def test_get_http_kwargs_reads_file_backed_credential_fresh(settings, monkeypatch, tmp_path, workload):
- settings.CLICKHOUSE_NATIVE_EVENTS_HOST = "native.example.com"
+def test_get_http_kwargs_reads_file_backed_credential_fresh(monkeypatch, tmp_path):
token = tmp_path / "token"
token.write_text("tok-0")
_file_backed_default(monkeypatch, token)
- assert get_http_kwargs(workload)["password"] == "tok-0"
+ assert get_http_kwargs(Workload.ONLINE)["password"] == "tok-0"
token.write_text("tok-1") # a rotation must reach the short-lived HTTP client, not a value cached at build
- assert get_http_kwargs(workload)["password"] == "tok-1"
+ assert get_http_kwargs(Workload.ONLINE)["password"] == "tok-1"
def test_get_client_from_pool_http_branch_sends_fresh_token(settings, monkeypatch, tmp_path):
@@ -410,12 +361,8 @@ def test_refreshing_pool_stamps_current_credential(tmp_path: Path, failure: str
@pytest.fixture(autouse=True)
-def reset_state(settings, monkeypatch):
+def reset_state():
make_ch_pool.cache_clear()
- monkeypatch.setattr(
- "posthog.models.event.new_events_schema.get_instance_setting",
- lambda key: settings.CLICKHOUSE_NATIVE_EVENTS_HOST,
- )
yield
@@ -458,34 +405,3 @@ def test_logs_workload_uses_a_named_user_only_when_registered(
expected_user,
expected_password,
)
-
-
- "workload,clickhouse_verify,queryservice_verify,expected_verify",
- [
- (Workload.NEW_EVENTS_CLUSTER, False, True, True),
- (Workload.NEW_EVENTS_CLUSTER, True, False, False),
- (Workload.OFFLINE, True, True, True),
- ],
-)
-def test_http_pool_preserves_existing_verification_policy(
- settings, monkeypatch, workload, clickhouse_verify, queryservice_verify, expected_verify
-):
- settings.CLICKHOUSE_NATIVE_EVENTS_HOST = "native.example.com"
- settings.CLICKHOUSE_OFFLINE_CLUSTER_HOST = "offline.example.com"
- settings.CLICKHOUSE_VERIFY = clickhouse_verify
- settings.QUERYSERVICE_VERIFY = queryservice_verify
- settings.CLICKHOUSE_SECURE = True
- settings.CLICKHOUSE_LOGS_CLUSTER_SECURE = True
- settings.CLICKHOUSE_USE_HTTP = True
- monkeypatch.setattr(connection, "_default_workload", Workload.OFFLINE)
- connection._clickhouse_http_pool_mgr.cache_clear()
- try:
- with patch("clickhouse_connect.get_client") as client:
- with connection.get_client_from_pool(workload):
- pass
- manager = client.call_args.kwargs["pool_mgr"]
- assert client.call_args.kwargs["secure"]
- assert manager.connection_pool_kw["cert_reqs"] == ("CERT_REQUIRED" if expected_verify else "CERT_NONE")
- finally:
- connection._clickhouse_http_pool_mgr.cache_clear()
diff --git a/posthog/clickhouse/client/test/test_execute.py b/posthog/clickhouse/client/test/test_execute.py
index 375be0a5af89..1003a571562a 100644
--- a/posthog/clickhouse/client/test/test_execute.py
+++ b/posthog/clickhouse/client/test/test_execute.py
@@ -83,7 +83,6 @@ def test_celery_cache_warming_workload_routing(client_from_pool, access_method,
[
(Workload.DEFAULT, Workload.ENDPOINTS),
(Workload.LOGS, Workload.LOGS),
- (Workload.NEW_EVENTS_CLUSTER, Workload.NEW_EVENTS_CLUSTER),
],
)
def test_endpoints_tag_workload_routing(client_from_pool, workload, expected_workload):
diff --git a/posthog/clickhouse/workload.py b/posthog/clickhouse/workload.py
index dade6763b568..723ac6843046 100644
--- a/posthog/clickhouse/workload.py
+++ b/posthog/clickhouse/workload.py
@@ -12,6 +12,5 @@ class Workload(StrEnum):
OFFLINE = "OFFLINE"
# Tables that exist only on the logs cluster
LOGS = "LOGS"
- NEW_EVENTS_CLUSTER = "NEW_EVENTS_CLUSTER"
# Endpoints (the product) queries
ENDPOINTS = "ENDPOINTS"
diff --git a/posthog/hogql/modifiers.py b/posthog/hogql/modifiers.py
index 23af691149a1..8465ca863914 100644
--- a/posthog/hogql/modifiers.py
+++ b/posthog/hogql/modifiers.py
@@ -58,22 +58,17 @@ def create_default_modifiers_for_team(
else:
modifiers = modifiers.model_copy()
- from posthog.models.event.new_events_schema import use_new_events_cluster # noqa: PLC0415
-
- if use_new_events_cluster(team.pk, modifiers):
- if modifiers.useNewEventsSchema is False:
- from posthog.hogql.errors import ExposedHogQLError # noqa: PLC0415
-
- raise ExposedHogQLError("Native events cluster queries require the native events schema.")
- modifiers.useNewEventsSchema = True
-
if modifiers.useMaterializedViews is None:
modifiers.useMaterializedViews = True
if isinstance(team.modifiers, dict):
for key, value in team.modifiers.items():
- # Cluster routing requires a query opt-in, even if a project stores the modifier.
- if key == "useNewEventsCluster":
+ # A modifier removed from the schema can stay stored on a project, and the model rejects unknown fields.
+ if key not in HogQLQueryModifiers.model_fields:
+ continue
+ # The events table is switched per project by CLICKHOUSE_HOGQL_USE_NEW_EVENTS_SCHEMA_TEAMS, which every
+ # reader of the table checks. A stored modifier would switch the HogQL readers alone.
+ if key == "useNewEventsSchema":
continue
if getattr(modifiers, key, None) is None:
if key == "customChannelTypeRules":
diff --git a/posthog/hogql/query.py b/posthog/hogql/query.py
index 51d7abdea69d..22bfd1eac9cf 100644
--- a/posthog/hogql/query.py
+++ b/posthog/hogql/query.py
@@ -73,11 +73,11 @@ from posthog.hogql.warehouse_warnings import record_warnings
from posthog.clickhouse.client import sync_execute
from posthog.clickhouse.client.connection import ClickHouseUser, Workload
-from posthog.clickhouse.query_tagging import Product, get_query_tag_value, get_query_tags, tag_queries
+from posthog.clickhouse.query_tagging import get_query_tag_value, get_query_tags, tag_queries
from posthog.dataclasses import frozen
from posthog.direct_query_cancellation import build_direct_query_cancellation_token
from posthog.errors import CHQueryErrorS3Error, CHQueryErrorS3FileChangedDuringRead, ExposedCHQueryError
-from posthog.models.event.new_events_schema import new_events_cluster_host_scope, use_new_events_cluster
+from posthog.models.event.new_events_schema import use_new_events_schema
from posthog.models.team import Team
from posthog.models.user import User
from posthog.settings import HOGQL_INCREASED_MAX_EXECUTION_TIME
@@ -166,8 +166,6 @@ class HogQLQueryExecutor:
self.context.user_access_control = self.user_access_control
self.query_modifiers = create_default_modifiers_for_team(self.team, self.modifiers)
- if self.connection_id and self.query_modifiers.useNewEventsCluster:
- raise ExposedHogQLError("Native events cluster queries cannot use a warehouse connection.")
self.debug = self.modifiers is not None and self.modifiers.debug
self.error: Optional[str] = None
self.explain: Optional[list[str]] = None
@@ -253,9 +251,10 @@ class HogQLQueryExecutor:
@tracer.start_as_current_span("HogQLQueryExecutor._apply_optimizers")
def _apply_optimizers(self):
- if self.query_modifiers.useNewEventsCluster:
- return
- if self.query_modifiers.usePreaggregatedTableTransforms:
+ # The preaggregated tables are built from the legacy events table, so a read of events_json skips them.
+ if self.query_modifiers.usePreaggregatedTableTransforms and not use_new_events_schema(
+ self.team.pk, self.query_modifiers
+ ):
with self.timings.measure("preaggregated_table_transforms"):
assert self.hogql_context is not None
assert self.hogql_context.team is not None
@@ -972,19 +971,6 @@ class HogQLQueryExecutor:
table_workload = clickhouse_context.workload
if table_workload is not None and table_workload != Workload.DEFAULT and workload in MAIN_CLUSTER_WORKLOADS:
workload = table_workload
- if use_new_events_cluster(self.team.pk, self.query_modifiers):
- if get_query_tags().product == Product.ENDPOINTS or workload not in (
- Workload.DEFAULT,
- Workload.ONLINE,
- Workload.OFFLINE,
- Workload.NEW_EVENTS_CLUSTER,
- ):
- raise ExposedHogQLError(
- "New events cluster opt-ins support analytics workloads only. Remove useNewEventsCluster for logs or endpoints queries."
- )
- if not clickhouse_context.uses_new_events_schema():
- raise ExposedHogQLError("New events cluster queries require the native events schema.")
- workload = Workload.NEW_EVENTS_CLUSTER
def run_clickhouse_query() -> Any:
return sync_execute(
@@ -1076,7 +1062,6 @@ class HogQLQueryExecutor:
)
@tracer.start_as_current_span("HogQLQueryExecutor.execute")
- @new_events_cluster_host_scope()
def execute(self) -> HogQLQueryResponse:
self._executing = True
trace.get_current_span().set_attribute("team_id", self.team.pk)
diff --git a/posthog/hogql/test/test_modifiers.py b/posthog/hogql/test/test_modifiers.py
index 85305261405b..907f25e716fc 100644
--- a/posthog/hogql/test/test_modifiers.py
+++ b/posthog/hogql/test/test_modifiers.py
@@ -80,6 +80,7 @@ class TestModifiers(BaseTest):
== PersonsOnEventsMode.PERSON_ID_OVERRIDE_PROPERTIES_JOINED # no change here
)
+ # A modifier removed from the schema can stay stored on a project.
self.team.modifiers = {
"personsOnEventsMode": PersonsOnEventsMode.PERSON_ID_NO_OVERRIDE_PROPERTIES_ON_EVENTS,
"useNewEventsCluster": True,
@@ -87,7 +88,6 @@ class TestModifiers(BaseTest):
self.team.save()
modifiers = create_default_modifiers_for_team(self.team)
assert modifiers.personsOnEventsMode == PersonsOnEventsMode.PERSON_ID_NO_OVERRIDE_PROPERTIES_ON_EVENTS
- assert modifiers.useNewEventsCluster is None
def test_unparseable_custom_bot_definitions_are_dropped(self):
# modifiers is hand-editable JSON (e.g. Django admin), so an entry that is not a rule object
diff --git a/posthog/hogql/test/test_new_events_schema_routing.py b/posthog/hogql/test/test_new_events_schema_routing.py
index ab27e2c45ac9..2d1a02672e09 100644
--- a/posthog/hogql/test/test_new_events_schema_routing.py
+++ b/posthog/hogql/test/test_new_events_schema_routing.py
@@ -1,34 +1,40 @@
+import json
+
from posthog.test.base import APIBaseTest
-from unittest.mock import patch
+from unittest.mock import MagicMock, patch
from django.test import SimpleTestCase, override_settings
+from django.urls import reverse
from parameterized import parameterized
-from posthog.schema import HogQLQuery, HogQLQueryModifiers, PersonsOnEventsMode
+from posthog.schema import HogQLQueryModifiers, PersonsOnEventsMode
-from posthog.hogql import ast
-from posthog.hogql.context import HogQLContext
-from posthog.hogql.errors import ExposedHogQLError
-from posthog.hogql.query import HogQLQueryExecutor
+from posthog.hogql.modifiers import create_default_modifiers_for_team
+from posthog.hogql.parser import parse_select
+from posthog.hogql.query import execute_hogql_query
-from posthog.clickhouse.query_tagging import Product, tags_context
-from posthog.clickhouse.workload import Workload
-from posthog.hogql_queries.hogql_query_runner import HogQLQueryRunner
from posthog.models import Organization, Team, User
+from posthog.models.event.new_events_schema import (
+ NEW_EVENTS_SCHEMA_COOKIE,
+ events_route_scope,
+ new_events_schema_for_request,
+ use_new_events_schema,
+)
from posthog.models.instance_setting import override_instance_config
+from products.dashboards.backend.models.dashboard import Dashboard
+from products.dashboards.backend.models.dashboard_tile import DashboardTile
+from products.product_analytics.backend.facade.models import Insight
+
+SQL_ON_EVENTS: dict[str, object] = {"kind": "HogQLQuery", "query": "SELECT count() AS n FROM events"}
+
-@override_settings(CLICKHOUSE_HOGQL_USE_NEW_EVENTS_SCHEMA=False, CLICKHOUSE_NATIVE_EVENTS_HOST="native.example.com")
-class TestNewEventsClusterRouting(SimpleTestCase):
+@override_settings(CLICKHOUSE_HOGQL_USE_NEW_EVENTS_SCHEMA=False)
+class TestEventsTableSwitch(SimpleTestCase):
def setUp(self) -> None:
self.team = Team(id=44, modifiers={}, organization=Organization(id=1))
- self.user = User(id=123, is_staff=True, is_active=True)
- self.modifiers = HogQLQueryModifiers(personsOnEventsMode=PersonsOnEventsMode.DISABLED)
- self.instance_settings: dict[str, object] = {
- "CLICKHOUSE_NATIVE_EVENTS_TEAMS": [44],
- "CLICKHOUSE_NATIVE_EVENTS_HOST": "native.example.com",
- }
+ self.instance_settings: dict[str, object] = {}
self.enterContext(
patch("posthog.models.event.new_events_schema.get_instance_setting", self.instance_settings.get)
)
@@ -39,182 +45,104 @@ class TestNewEventsClusterRouting(SimpleTestCase):
)
)
- @parameterized.expand(
- [
- ([], "native.example.com", "not enabled"),
- ([44, True], "native.example.com", "not enabled"),
- ([44], None, "host is not configured"),
- ]
- )
- def test_unavailable_native_route_fails_closed(self, allowed: object, host: str | None, error: str) -> None:
- self.instance_settings.update(CLICKHOUSE_NATIVE_EVENTS_HOST=host, CLICKHOUSE_NATIVE_EVENTS_TEAMS=allowed)
- with self.assertRaisesRegex(ExposedHogQLError, error):
- HogQLQueryRunner(
- query=HogQLQuery(
- query="SELECT 1", modifiers=self.modifiers.model_copy(update={"useNewEventsCluster": True})
- ),
- team=self.team,
- )
+ def test_a_stored_project_modifier_does_not_switch_the_events_table(self) -> None:
+ self.team.modifiers = {"useNewEventsSchema": True, "bounceRateDurationSeconds": 30}
+ modifiers = create_default_modifiers_for_team(
+ self.team, HogQLQueryModifiers(personsOnEventsMode=PersonsOnEventsMode.DISABLED)
+ )
+ self.assertIsNone(modifiers.useNewEventsSchema)
+ self.assertEqual(modifiers.bounceRateDurationSeconds, 30)
- def test_direct_warehouse_connection_cannot_claim_native_cluster_opt_in(self) -> None:
- with self.assertRaisesRegex(ExposedHogQLError, "warehouse connection"):
- HogQLQueryExecutor(
- query="SELECT 1",
- team=self.team,
- modifiers=self.modifiers.model_copy(update={"useNewEventsCluster": True}),
- connection_id="example-connection",
+ @parameterized.expand([(False, None, False), (True, None, True), (False, True, True), (True, False, False)])
+ def test_nested_reads_inherit_the_querys_events_table(
+ self, parent_schema: bool, child_schema: bool | None, expected: bool
+ ) -> None:
+ with new_events_schema_for_request(), events_route_scope(HogQLQueryModifiers(useNewEventsSchema=parent_schema)):
+ self.assertEqual(
+ use_new_events_schema(self.team.pk, HogQLQueryModifiers(useNewEventsSchema=child_schema)), expected
)
- @parameterized.expand([(False,), (True,)])
- def test_native_cache_identity_tracks_host_configuration_without_changing_legacy_namespace(
- self, native: bool
- ) -> None:
- runner = HogQLQueryRunner(
- query=HogQLQuery(
- query="SELECT 1",
- modifiers=self.modifiers.model_copy(
- update={"useNewEventsSchema": native, "useNewEventsCluster": native}
- ),
- ),
- team=self.team,
- )
- with (
- patch.object(runner, "_get_property_access_restrictions", return_value=None),
- patch.object(runner, "_products_modifiers_for_cache", return_value={}),
- patch("posthog.hogql_queries.query_runner.events_retention_months_for_team", return_value=None),
- ):
- keys = []
- for host in ("native.example.com", "replacement.example.com"):
- self.instance_settings["CLICKHOUSE_NATIVE_EVENTS_HOST"] = host
- keys.append(runner.get_cache_key())
- self.assertEqual(len(set(keys)), 2 if native else 1)
- @parameterized.expand(
- [
- (
- workload,
- None,
- workload in (Workload.DEFAULT, Workload.ONLINE, Workload.OFFLINE, Workload.NEW_EVENTS_CLUSTER),
- )
- for workload in Workload
- ]
- + [(Workload.DEFAULT, Product.ENDPOINTS, False)]
- )
- def test_native_opt_in_accepts_only_analytics_workloads(
- self, workload: Workload, product: Product | None, supported: bool
- ) -> None:
- executor = HogQLQueryExecutor(
- query="SELECT 1",
- team=self.team,
- workload=workload,
- modifiers=self.modifiers.model_copy(update={"useNewEventsCluster": True}),
- )
- executor.select_query = ast.SelectQuery(select=[ast.Constant(value=1)])
- executor.clickhouse_sql = "SELECT 1"
- executor.clickhouse_context = HogQLContext(team_id=self.team.pk, modifiers=executor.query_modifiers)
- with (
- tags_context(product=product),
- patch.object(executor, "_detect_warehouse_sources"),
- patch("posthog.hogql.query.sync_execute", return_value=([(1,)], [("1", "UInt8")])) as execute,
- ):
- if supported:
- executor._execute_clickhouse_query()
- self.assertEqual(execute.call_args.kwargs["workload"], Workload.NEW_EVENTS_CLUSTER)
- else:
- with self.assertRaisesRegex(ExposedHogQLError, "analytics workloads only"):
- executor._execute_clickhouse_query()
- execute.assert_not_called()
-
-
-@override_settings(CLICKHOUSE_NATIVE_EVENTS_HOST="native.example.com", CLICKHOUSE_HOGQL_USE_NEW_EVENTS_SCHEMA=False)
-class TestNewEventsClusterPilot(APIBaseTest):
+@override_settings(CLICKHOUSE_HOGQL_USE_NEW_EVENTS_SCHEMA=False)
+class TestEventsTableSwitchThroughTheApi(APIBaseTest):
def setUp(self) -> None:
super().setUp()
- self.enterContext(override_instance_config("CLICKHOUSE_NATIVE_EVENTS_HOST", "native.example.com"))
- self.enterContext(override_instance_config("CLICKHOUSE_NATIVE_EVENTS_TEAMS", [self.team.pk]))
self.enterContext(override_instance_config("CLICKHOUSE_HOGQL_USE_NEW_EVENTS_SCHEMA", False))
+ def _post_query(self, query: dict[str, object], modifiers: dict[str, bool]) -> MagicMock:
+ with patch("posthog.hogql.query.sync_execute", return_value=([], [])) as execute:
+ response = self.client.post(
+ f"/api/projects/{self.team.id}/query/",
+ {"query": {**query, "modifiers": modifiers}, "refresh": "force_blocking"},
+ format="json",
+ )
+ self.assertEqual(response.status_code, 200, response.json())
+ return execute
+
@parameterized.expand(
[
- (modifiers, eligible, native_table, native_cluster)
- for modifiers, eligible, native_table, native_cluster in (
- ({"useNewEventsCluster": True}, True, True, True),
- ({"useNewEventsCluster": True}, False, False, True),
- ({}, True, False, False),
- ({"useNewEventsSchema": True}, True, True, False),
- ({"useNewEventsCluster": False, "useNewEventsSchema": True}, True, True, False),
- )
- ]
- + [
- (
- {"useNewEventsCluster": True, modifier: True},
- True,
- True,
- True,
- sql,
- )
- for modifier, sql in (
- ("usePreaggregatedTableTransforms", "SELECT count() FROM events WHERE event = '$pageview'"),
- (
- "usePreaggregatedIntermediateResults",
- "SELECT toStartOfDay(timestamp), uniqExact(person_id) FROM events WHERE event = '$pageview' AND timestamp >= '2024-01-01' AND timestamp < '2024-01-08' GROUP BY toStartOfDay(timestamp)",
- ),
- )
- ]
- + [
- ({"useNewEventsCluster": True}, True, True, True, "SELECT 1", "force_async"),
- ({"useNewEventsCluster": True}, True, True, True, "SELECT 1", "force_blocking", True),
- (
- {"useNewEventsCluster": True},
- True,
- True,
- True,
- {"kind": "PropertyValuesQuery", "property_key": "$browser", "property_type": "event"},
- ),
+ ("staff", True, False, {}, True),
+ ("staff_query_pinned_to_legacy", True, False, {"useNewEventsSchema": False}, False),
+ ("staff_impersonating_a_customer", True, True, {}, True),
+ ("not_staff", False, False, {}, False),
]
)
- def test_query_route_requires_an_eligible_explicit_blocking_opt_in(
- self,
- modifiers: dict[str, bool],
- eligible: bool,
- native_table: bool,
- native_cluster: bool,
- sql: str | dict[str, str] = "SELECT count() AS n FROM events",
- refresh: str = "force_blocking",
- async_: bool = False,
+ def test_the_cookie_switches_a_staff_persons_requests_to_events_json(
+ self, _name: str, staff: bool, impersonating: bool, modifiers: dict[str, bool], events_json: bool
) -> None:
- query = {"kind": "HogQLQuery", "query": sql} if isinstance(sql, str) else sql
- unsupported = native_cluster and (query["kind"] != "HogQLQuery" or refresh != "force_blocking" or async_)
- with (
- override_instance_config("CLICKHOUSE_NATIVE_EVENTS_TEAMS", [self.team.pk] if eligible else []),
- patch(
- "products.analytics_platform.backend.lazy_computation.lazy_computation_executor.sync_execute",
- return_value=0,
- ) as insert,
- patch("posthog.hogql.query.sync_execute", return_value=([(7,)], [("n", "UInt64")])) as execute,
- patch("posthog.hogql_queries.query_runner.enqueue_process_query_task") as enqueue,
- ):
- response = self.client.post(
- f"/api/projects/{self.team.id}/query/",
- {
- "query": {**query, "modifiers": modifiers},
- "refresh": refresh,
- "async": async_,
- },
- format="json",
+ self.user.is_staff = staff
+ self.user.save()
+ if impersonating:
+ customer = User.objects.create_and_join(self.organization, "[email protected]", None)
+ # The login-as admin view reads a form post, and this client sends JSON unless told otherwise.
+ self.client.post(
+ reverse("loginas-user-login", kwargs={"user_id": customer.id}),
+ data={"read_only": "false", "reason": "Checking the native table"},
+ format="multipart",
+ follow=True,
+ )
+ self.assertEqual(self.client.get("/api/users/@me").json()["email"], customer.email)
+ self.client.cookies[NEW_EVENTS_SCHEMA_COOKIE] = "1"
+
+ execute = self._post_query(SQL_ON_EVENTS, modifiers)
+ self.assertEqual("events_json" in execute.call_args.args[0], events_json)
+
+ @parameterized.expand([("legacy", False), ("events_json", True)])
+ def test_only_a_legacy_read_uses_the_preaggregated_web_tables(self, _name: str, events_json: bool) -> None:
+ with patch("posthog.hogql.query.sync_execute", return_value=([], [])) as execute:
+ execute_hogql_query(
+ parse_select("select count(), uniq(person_id) from events where equals(event, '$pageview')"),
+ team=self.team,
+ modifiers=HogQLQueryModifiers(usePreaggregatedTableTransforms=True, useNewEventsSchema=events_json),
)
- rejected = native_cluster and (not eligible or unsupported)
- self.assertEqual(response.status_code, 400 if rejected else 200, response.json())
- enqueue.assert_not_called()
- if rejected:
- execute.assert_not_called()
- if unsupported:
- self.assertIn("Native cluster comparisons", response.json()["detail"])
- return
- self.assertEqual("events_json" in execute.call_args.args[0], native_table)
- self.assertEqual(execute.call_args.kwargs["workload"] == Workload.NEW_EVENTS_CLUSTER, native_cluster)
- if native_cluster:
- insert.assert_not_called()
- self.assertNotIn("pre_aggregated", execute.call_args.args[0])
- self.assertNotIn("preaggregation_results", execute.call_args.args[0])
- self.assertIs(response.json()["modifiers"]["useNewEventsCluster"], True)
+ self.assertEqual("web_pre_aggregated_stats" in execute.call_args.args[0], not events_json)
+
+ def test_the_cookie_reaches_a_streamed_dashboards_cache_lookup(self) -> None:
+ self.user.is_staff = True
+ self.user.save()
+ self.client.cookies[NEW_EVENTS_SCHEMA_COOKIE] = "1"
+ insight = Insight.objects.create(
+ team=self.team, filters={}, query={"kind": "DataVisualizationNode", "source": SQL_ON_EVENTS}
+ )
+ dashboard = Dashboard.objects.create(team=self.team, created_by=self.user)
+ DashboardTile.objects.create(dashboard=dashboard, insight=insight)
+ with patch("posthog.hogql.query.sync_execute", return_value=([(1,)], [("n", "UInt64")])) as execute:
+ response = self.client.get(
+ f"/api/projects/{self.team.id}/dashboards/{dashboard.id}/?refresh=force_blocking"
+ )
+ self.assertEqual(response.status_code, 200, response.json())
+ self.assertIn("events_json", execute.call_args.args[0])
+
+ # The stream only reads the cache, so a hit here means the switch reached the lookup's cache key.
+ self.assertIsNotNone(self._streamed_tile_result(dashboard.id))
+ self.client.cookies[NEW_EVENTS_SCHEMA_COOKIE] = "0"
+ self.assertIsNone(self._streamed_tile_result(dashboard.id))
+
+ def _streamed_tile_result(self, dashboard_id: int) -> object:
+ response = self.client.get(f"/api/projects/{self.team.id}/dashboards/{dashboard_id}/stream_tiles/")
+ self.assertEqual(response.status_code, 200)
+ content = b"".join(response.streaming_content).decode() # type: ignore[attr-defined]
+ metadata = next(
+ line[6:] for line in content.split("\n") if line.startswith("data: ") and '"type":"metadata"' in line
+ )
+ return json.loads(metadata)["dashboard"]["tiles"][0]["insight"]["result"]
diff --git a/posthog/hogql/transforms/test/test_preaggregated_table_transformation.py b/posthog/hogql/transforms/test/test_preaggregated_table_transformation.py
index 14d3cf1f77fd..dcf6b0b42f25 100644
--- a/posthog/hogql/transforms/test/test_preaggregated_table_transformation.py
+++ b/posthog/hogql/transforms/test/test_preaggregated_table_transformation.py
@@ -3,6 +3,8 @@ from datetime import UTC, datetime
import unittest
from posthog.test.base import APIBaseTest, BaseTest, ClickhouseTestMixin, QueryMatchingTest, snapshot_clickhouse_queries
+from django.test import override_settings
+
from parameterized import parameterized
from posthog.schema import BaseMathType, DateRange, EventsNode, HogQLQueryModifiers, TrendsQuery
@@ -827,6 +829,7 @@ GROUP BY date
@snapshot_clickhouse_queries
+@override_settings(CLICKHOUSE_HOGQL_USE_NEW_EVENTS_SCHEMA=False)
class TestPreaggregatedTableTransformationIntegration(APIBaseTest, ClickhouseTestMixin):
CLASS_DATA_LEVEL_SETUP = False
TEST_DATA_DATE = datetime(2024, 11, 24, tzinfo=UTC)
diff --git a/posthog/hogql_queries/hogql_query_runner.py b/posthog/hogql_queries/hogql_query_runner.py
index bc279b13d85f..73dacce6bf4e 100644
--- a/posthog/hogql_queries/hogql_query_runner.py
+++ b/posthog/hogql_queries/hogql_query_runner.py
@@ -68,8 +68,6 @@ class HogQLQueryRunner(AnalyticsQueryRunner[HogQLQueryResponse]):
self._direct_engine: str | None = None
self._managed_warehouse_sql_mode: ManagedWarehouseSQLMode | None = None
super().__init__(*args, **kwargs)
- if self.modifiers.useNewEventsCluster and not self._reads_events_route():
- raise ExposedHogQLError("Native events cluster queries cannot use a warehouse connection.")
def _reads_events_route(self) -> bool:
# A direct connection runs on another database, so the events route never applies to it.
diff --git a/posthog/hogql_queries/query_runner.py b/posthog/hogql_queries/query_runner.py
index ebe18cf3149f..e4a7a2cc11fe 100644
--- a/posthog/hogql_queries/query_runner.py
+++ b/posthog/hogql_queries/query_runner.py
@@ -125,7 +125,6 @@ from posthog.hogql import ast
from posthog.hogql.constants import LimitContext
from posthog.hogql.context import HogQLContext
from posthog.hogql.database.database import Database
-from posthog.hogql.errors import ExposedHogQLError
from posthog.hogql.modifiers import create_default_modifiers_for_user
from posthog.hogql.printer import prepare_and_print_ast, to_printed_hogql
from posthog.hogql.query import create_default_modifiers_for_team
@@ -195,8 +194,6 @@ from posthog.models import Team, User
from posthog.models.event.new_events_schema import (
events_route_cache_modifiers,
events_route_scope,
- get_new_events_cluster_host,
- new_events_cluster_host_scope,
strip_events_route_modifiers,
use_new_events_schema,
with_pinned_events_route,
@@ -2472,7 +2469,6 @@ class QueryRunner(ABC, Generic[Q, R, CR]):
query_duration_ms = round((perf_counter() - query_start_time) * 1000, 2)
return query_result, query_duration_ms
- @new_events_cluster_host_scope()
@_pin_events_route
def run(
self,
@@ -2484,12 +2480,6 @@ class QueryRunner(ABC, Generic[Q, R, CR]):
cache_age_seconds: Optional[int] = None,
analytics_props: Optional[AnalyticsProps] = None,
) -> CR | CacheMissResponse | QueryStatusResponse:
- if self.modifiers.useNewEventsCluster:
- if not isinstance(self.query, HogQLQuery):
- raise ExposedHogQLError("Native cluster comparisons currently support HogQLQuery only.")
- if execution_mode != ExecutionMode.CALCULATE_BLOCKING_ALWAYS:
- raise ExposedHogQLError("Native cluster comparisons require refresh=force_blocking and async=false.")
-
# Set user for access control during query execution. Some subclasses
# (e.g. QueryRunnerWithHogQLContext) construct user-dependent state in
# __init__; let them refresh it now that we know the real user. Only on an
@@ -3228,8 +3218,6 @@ class QueryRunner(ABC, Generic[Q, R, CR]):
"week_start_day": self.team.week_start_day or WeekStartDay.SUNDAY,
"version": 2,
}
- if self.modifiers.useNewEventsCluster:
- payload["native_events_host"] = get_new_events_cluster_host()
# Include property-level access control restrictions in the cache key so that
# users with different property restrictions get separate cache entries.
diff --git a/posthog/hogql_queries/test/test_query_runner.py b/posthog/hogql_queries/test/test_query_runner.py
index 25919d026c95..4aa6d28df76e 100644
--- a/posthog/hogql_queries/test/test_query_runner.py
+++ b/posthog/hogql_queries/test/test_query_runner.py
@@ -943,9 +943,8 @@ class TestQueryRunner(BaseTest):
def test_cache_hit_and_fresh_events_carry_one_identity(self) -> None:
runner_class = setup_test_query_runner_class()
- self.team.modifiers = {"useNewEventsSchema": True}
- self.team.save()
with (
+ override_instance_config("CLICKHOUSE_HOGQL_USE_NEW_EVENTS_SCHEMA_TEAMS", str(self.team.pk)),
mock.patch(
"posthog.hogql_queries.query_runner.create_default_modifiers_for_user", side_effect=_add_user_modifier
),
diff --git a/posthog/hogql_queries/utils/test/test_timestamp_utils.py b/posthog/hogql_queries/utils/test/test_timestamp_utils.py
index f07b172fdf58..55ca7c46b45a 100644
--- a/posthog/hogql_queries/utils/test/test_timestamp_utils.py
+++ b/posthog/hogql_queries/utils/test/test_timestamp_utils.py
@@ -22,7 +22,6 @@ from posthog.schema import ActionsNode, DataWarehouseNode, DateRange, EventsNode
from posthog.hogql.query import execute_hogql_query
from posthog.clickhouse.query_tagging import Feature, Product, get_query_tags, tags_context
-from posthog.clickhouse.workload import Workload
from posthog.hogql_queries.utils.query_date_range import QueryDateRange
from posthog.hogql_queries.utils.timestamp_utils import (
EARLIEST_EVENT_TIMESTAMP,
@@ -593,8 +592,8 @@ class TestEarliestTimestampRouting(BaseTest):
cache.clear()
@parameterized.expand([("series",), ("unfiltered",)])
- @override_settings(CLICKHOUSE_HOGQL_USE_NEW_EVENTS_SCHEMA=False, CLICKHOUSE_NATIVE_EVENTS_HOST="native.example.com")
- def test_native_route_reads_the_native_cluster_and_skips_the_legacy_cache(self, lookup: str) -> None:
+ @override_settings(CLICKHOUSE_HOGQL_USE_NEW_EVENTS_SCHEMA=False)
+ def test_native_route_reads_events_json_and_skips_the_legacy_cache(self, lookup: str) -> None:
node = EventsNode(event="$pageview")
legacy = datetime.datetime(2022, 1, 1, tzinfo=ZoneInfo("UTC"))
native = datetime.datetime(2020, 1, 1, tzinfo=ZoneInfo("UTC"))
@@ -608,19 +607,14 @@ class TestEarliestTimestampRouting(BaseTest):
return get_earliest_timestamp_from_series(self.team, [node])
return get_earliest_timestamp_unfiltered(self.team)
- with (
- override_instance_config("CLICKHOUSE_NATIVE_EVENTS_HOST", "native.example.com"),
- override_instance_config("CLICKHOUSE_NATIVE_EVENTS_TEAMS", [self.team.pk]),
- patch(
- "posthog.hogql.query.sync_execute", return_value=([(native,)], [("timestamp", "DateTime")])
- ) as execute,
- ):
- with events_route_scope(HogQLQueryModifiers(useNewEventsCluster=True, useNewEventsSchema=True)):
+ with patch(
+ "posthog.hogql.query.sync_execute", return_value=([(native,)], [("timestamp", "DateTime")])
+ ) as execute:
+ with events_route_scope(HogQLQueryModifiers(useNewEventsSchema=True)):
self.assertEqual(earliest(), native)
self.assertEqual(earliest(), native)
self.assertEqual(earliest(), legacy)
execute.assert_called_once()
- self.assertEqual(execute.call_args.kwargs["workload"], Workload.NEW_EVENTS_CLUSTER)
self.assertIn("events_json", execute.call_args.args[0])
def test_route_keys_never_collide_with_legacy_event_keys(self) -> None:
@@ -635,17 +629,21 @@ class TestEarliestTimestampRouting(BaseTest):
):
self.assertEqual(get_earliest_timestamp_from_series(self.team, [EventsNode(event=None)]), native)
- @parameterized.expand([("allowlisted_project", True, {}), ("project_setting", False, {"useNewEventsSchema": True})])
+ @parameterized.expand(
+ [
+ ("allowlisted_project", True, {}, True),
+ ("stored_project_modifier_is_ignored", False, {"useNewEventsSchema": True}, False),
+ ]
+ )
@override_settings(CLICKHOUSE_HOGQL_USE_NEW_EVENTS_SCHEMA=False)
def test_lookup_outside_a_run_keys_on_the_events_table_it_reads(
- self, _name: str, allowlisted: bool, team_modifiers: dict[str, bool]
+ self, _name: str, allowlisted: bool, team_modifiers: dict[str, bool], reads_events_json: bool
) -> None:
self.team.modifiers = team_modifiers
node = EventsNode(event="$pageview")
- cache.set(
- _get_earliest_timestamp_cache_key(self.team, node), datetime.datetime(2022, 1, 1, tzinfo=ZoneInfo("UTC"))
- )
+ legacy = datetime.datetime(2022, 1, 1, tzinfo=ZoneInfo("UTC"))
native = datetime.datetime(2020, 1, 1, tzinfo=ZoneInfo("UTC"))
+ cache.set(_get_earliest_timestamp_cache_key(self.team, node), legacy)
with (
override_instance_config(
"CLICKHOUSE_HOGQL_USE_NEW_EVENTS_SCHEMA_TEAMS", str(self.team.pk) if allowlisted else ""
@@ -654,5 +652,10 @@ class TestEarliestTimestampRouting(BaseTest):
"posthog.hogql.query.sync_execute", return_value=([(native,)], [("timestamp", "DateTime")])
) as execute,
):
- self.assertEqual(get_earliest_timestamp_from_series(self.team, [node]), native)
- self.assertIn("events_json", execute.call_args.args[0])
+ earliest = get_earliest_timestamp_from_series(self.team, [node])
+ if reads_events_json:
+ self.assertEqual(earliest, native)
+ self.assertIn("events_json", execute.call_args.args[0])
+ else:
+ self.assertEqual(earliest, legacy)
+ execute.assert_not_called()
diff --git a/posthog/hogql_queries/utils/timestamp_utils.py b/posthog/hogql_queries/utils/timestamp_utils.py
index 6419bf78f4e1..d2ec736b46db 100644
--- a/posthog/hogql_queries/utils/timestamp_utils.py
+++ b/posthog/hogql_queries/utils/timestamp_utils.py
@@ -33,7 +33,7 @@ from posthog.clickhouse.query_tagging import Feature, Product, get_query_tags, t
from posthog.hogql_queries.utils.query_date_range import QueryDateRange
from posthog.models import Team, User
from posthog.models.event import DEFAULT_EARLIEST_TIME_DELTA
-from posthog.models.event.new_events_schema import get_events_route, get_new_events_cluster_host, use_new_events_schema
+from posthog.models.event.new_events_schema import get_events_route, use_new_events_schema
from posthog.models.team import WeekStartDay
from posthog.utils import get_safe_cache
@@ -143,14 +143,11 @@ def _events_route_modifiers(team: Team) -> HogQLQueryModifiers:
def _events_route_cache_key(team: Team, modifiers: HogQLQueryModifiers, cache_key: str) -> str:
- # Each events table and cluster holds its own copy of the events, so one route's result must not answer another.
- # The route goes in front, because legacy keys end in event names and table names that a suffix could collide with.
- native_table = use_new_events_schema(team.pk, modifiers)
- native_cluster = bool(modifiers.useNewEventsCluster)
- if not (native_table or native_cluster):
+ # Each events table holds its own copy of the events, so one table's result must not answer for the other.
+ # The table goes in front, because legacy keys end in event names and table names that a suffix could collide with.
+ if not use_new_events_schema(team.pk, modifiers):
return cache_key
- native_host = get_new_events_cluster_host() if native_cluster else None
- return f"events_route:{native_table}:{native_host}:{cache_key}"
+ return f"events_json:{cache_key}"
def _coerce_to_datetime(value: Any, tz: tzinfo) -> datetime:
diff --git a/posthog/middleware.py b/posthog/middleware.py
index 1050eca2a44d..ec7fc297e7dd 100644
--- a/posthog/middleware.py
+++ b/posthog/middleware.py
@@ -57,6 +57,7 @@ from posthog.models.activity_logging.utils import (
client_from_header,
record_agent_intent,
)
+from posthog.models.event.new_events_schema import NEW_EVENTS_SCHEMA_COOKIE, new_events_schema_for_request
from posthog.session.activity import session_activity_credential
from posthog.settings import PROJECT_SWITCHING_TOKEN_ALLOWLIST, SITE_URL
from posthog.user_permissions import UserPermissions
@@ -1227,6 +1228,27 @@ class AutoLogoutImpersonateMiddleware:
return self.get_response(request)
+class NewEventsSchemaCookieMiddleware:
+ """A staff person's requests read the native JSON events table while their browser carries the cookie.
+
+ The switch is per person, so one person can check a project on events_json, an impersonated one included,
+ without switching the project for everyone. An impersonated session counts as staff because only staff can
+ start one.
+ """
+
+ def __init__(self, get_response: Callable[[HttpRequest], HttpResponse]) -> None:
+ self.get_response = get_response
+
+ def __call__(self, request: HttpRequest) -> HttpResponse:
+ if request.COOKIES.get(NEW_EVENTS_SCHEMA_COOKIE) != "1":
+ return self.get_response(request)
+ user = cast(User, request.user)
+ if not (user.is_staff or is_impersonated_session(request)):
+ return self.get_response(request)
+ with new_events_schema_for_request():
+ return self.get_response(request)
+
+
class UnversionedStaticPathMiddleware:
"""Serves /static/<version>/<file> as /static/<file>.
diff --git a/posthog/models/event/new_events_schema.py b/posthog/models/event/new_events_schema.py
index aae7ebbe24b4..d8206ab206a2 100644
--- a/posthog/models/event/new_events_schema.py
+++ b/posthog/models/event/new_events_schema.py
@@ -11,10 +11,8 @@ if TYPE_CHECKING:
from posthog.schema import HogQLQueryModifiers
-# TODO: Remove the per-query rollout option, and the route helpers in this block, after the events cluster
-# migration is complete.
-EVENTS_ROUTE_MODIFIER_KEYS = ("useNewEventsSchema", "useNewEventsCluster")
-_query_host: ContextVar[list[str | None] | None] = ContextVar("new_events_cluster_host", default=None)
+# TODO: Remove the route helpers in this block after the events cluster migration is complete.
+EVENTS_ROUTE_MODIFIER_KEYS = ("useNewEventsSchema",)
_query_route: ContextVar["HogQLQueryModifiers | None"] = ContextVar("new_events_query_route", default=None)
@@ -54,10 +52,7 @@ def with_pinned_events_route(query: dict[str, Any], modifiers: "HogQLQueryModifi
if "modifiers" not in _copy_source_chain(query)[-1]:
return query
nodes = _copy_source_chain(strip_events_route_modifiers(query))
- route: dict[str, Any] = {"useNewEventsSchema": modifiers.useNewEventsSchema}
- if modifiers.useNewEventsCluster is not None:
- route["useNewEventsCluster"] = modifiers.useNewEventsCluster
- nodes[-1]["modifiers"] = {**(nodes[-1]["modifiers"] or {}), **route}
+ nodes[-1]["modifiers"] = {**(nodes[-1]["modifiers"] or {}), "useNewEventsSchema": modifiers.useNewEventsSchema}
return nodes[0]
@@ -70,80 +65,59 @@ def events_route_cache_modifiers(modifiers: dict[str, Any], *, reads_events: boo
modifiers = dict(modifiers)
if modifiers.get("useNewEventsSchema") is False or not reads_events:
modifiers.pop("useNewEventsSchema", None)
- if not modifiers.get("useNewEventsCluster"):
- modifiers.pop("useNewEventsCluster", None)
return modifiers
-def get_new_events_cluster_host() -> str | None:
- snapshot = _query_host.get()
- if snapshot:
- return snapshot[0]
- value = get_instance_setting("CLICKHOUSE_NATIVE_EVENTS_HOST")
- host = value if isinstance(value, str) and value else None
- if snapshot is not None:
- snapshot.append(host)
- return host
-
-
-@contextmanager
-def new_events_cluster_host_scope() -> Iterator[None]:
- # Pin cache lookup and execution to one host even if settings change mid-request.
- if _query_host.get() is not None:
- yield
- return
- token = _query_host.set([])
- try:
- yield
- finally:
- _query_host.reset(token)
-
-
def get_events_route() -> "HogQLQueryModifiers | None":
- """The events cluster and table of the query run in progress, or None outside a query run."""
+ """The events table of the query run in progress, or None outside a query run."""
return _query_route.get()
@contextmanager
def events_route_scope(modifiers: "HogQLQueryModifiers") -> Iterator[None]:
- # Lookups inside a run, such as an all-time range's earliest event, must read the query's cluster and table.
+ # Lookups inside a run, such as an all-time range's earliest event, must read the query's events table.
from posthog.schema import HogQLQueryModifiers # noqa: PLC0415
- token = _query_route.set(
- HogQLQueryModifiers(
- useNewEventsCluster=modifiers.useNewEventsCluster, useNewEventsSchema=modifiers.useNewEventsSchema
- )
- )
+ token = _query_route.set(HogQLQueryModifiers(useNewEventsSchema=modifiers.useNewEventsSchema))
try:
yield
finally:
_query_route.reset(token)
-def use_new_events_cluster(team_id: int | None, modifiers: "HogQLQueryModifiers | None") -> bool:
- from posthog.hogql.errors import ExposedHogQLError # noqa: PLC0415
+# A staff person's browser sends this cookie to read events_json on every request, impersonated sessions included.
+NEW_EVENTS_SCHEMA_COOKIE = "ph_new_events_schema"
+_person_switch: ContextVar[bool] = ContextVar("new_events_schema_person_switch", default=False)
+
- if modifiers is None or not modifiers.useNewEventsCluster:
- return False
- if not get_new_events_cluster_host():
- raise ExposedHogQLError("The native events cluster host is not configured.")
- allowed = get_instance_setting("CLICKHOUSE_NATIVE_EVENTS_TEAMS")
- if not isinstance(allowed, list) or any(type(value) is not int for value in allowed) or team_id not in allowed:
- raise ExposedHogQLError("This project is not enabled for native events cluster queries.")
- return True
+@contextmanager
+def new_events_schema_for_request() -> Iterator[None]:
+ """Every events read inside the block targets events_json, unless its query's modifier says otherwise."""
+ token = _person_switch.set(True)
+ try:
+ yield
+ finally:
+ _person_switch.reset(token)
def use_new_events_schema(team_id: int | None = None, modifiers: "HogQLQueryModifiers | None" = None) -> bool:
- """Whether HogQL reads should target the native-JSON events tables.
-
- A query's `useNewEventsSchema` modifier decides when it is set, so one query can be run against either table.
- Otherwise the instance setting flips it globally at runtime, and the *_TEAMS instance setting enables it
- for individual teams first (both cached for up to 60s per worker). Code running inside a query
- should prefer HogQLContext.uses_new_events_schema(), which resolves this once so a mid-query
- flip can't mix schemas.
+ """Whether events reads target the native JSON events table (events_json) instead of the legacy events table.
+
+ Four switches, each wider than the one before it: the query's useNewEventsSchema modifier, for one query;
+ a staff person's cookie, for every request from that browser (see new_events_schema_for_request);
+ CLICKHOUSE_HOGQL_USE_NEW_EVENTS_SCHEMA_TEAMS, for a project everywhere, background jobs included; and
+ CLICKHOUSE_HOGQL_USE_NEW_EVENTS_SCHEMA, for every project. The instance settings are cached for up to 60s
+ per worker. Nested reads inherit the running query's choice unless their own modifier overrides it.
+ Code running inside a query should prefer HogQLContext.uses_new_events_schema(), which
+ resolves this once so a mid-query flip can't mix schemas.
"""
if modifiers is not None and modifiers.useNewEventsSchema is not None:
return modifiers.useNewEventsSchema
+ route = get_events_route()
+ if route is not None and route.useNewEventsSchema is not None:
+ return route.useNewEventsSchema
+ if _person_switch.get():
+ return True
if settings.TEST and settings.CLICKHOUSE_HOGQL_USE_NEW_EVENTS_SCHEMA:
return True
if get_instance_setting("CLICKHOUSE_HOGQL_USE_NEW_EVENTS_SCHEMA"):
diff --git a/posthog/schema.py b/posthog/schema.py
index 3c9bd336c5e9..bd1e9d04998e 100644
--- a/posthog/schema.py
+++ b/posthog/schema.py
@@ -6369,25 +6369,15 @@ class HogQLQueryModifiers(BaseModel):
),
)
useMaterializedViews: bool | None = None
- useNewEventsCluster: bool | None = Field(
- default=None,
- description=(
- "Read this query from the native events cluster and native JSON events"
- " table (`true`), or keep its existing connection (`false` or unset). This"
- " is an internal rollout switch for eligible projects. PostHog staff manage"
- " eligibility through instance settings. Each query must opt in; project"
- " defaults cannot enable it. Only synchronous HogQLQuery SQL comparisons"
- " are supported."
- ),
- )
useNewEventsSchema: bool | None = Field(
default=None,
description=(
"Read events from the native JSON events table (`true`) or the legacy"
- " events table (`false`). When unset, the project's stored value applies,"
- " then the `CLICKHOUSE_HOGQL_USE_NEW_EVENTS_SCHEMA` instance settings. This"
- " is an internal rollout switch. PostHog staff set the project value in"
- " Django admin and the project settings API ignores it."
+ " events table (`false`). When unset, a staff person's"
+ " `ph_new_events_schema` cookie applies, then the project's listing in the"
+ " `CLICKHOUSE_HOGQL_USE_NEW_EVENTS_SCHEMA_TEAMS` instance setting, then the"
+ " `CLICKHOUSE_HOGQL_USE_NEW_EVENTS_SCHEMA` instance setting. This is an"
+ " internal rollout switch, and the project settings API ignores it."
),
)
usePreaggregatedIntermediateResults: bool | None = None
diff --git a/posthog/settings/dynamic_settings.py b/posthog/settings/dynamic_settings.py
index 23a4fe782da3..8f1307e28d8e 100644
--- a/posthog/settings/dynamic_settings.py
+++ b/posthog/settings/dynamic_settings.py
@@ -64,24 +64,14 @@ CONSTANCE_CONFIG = {
),
"CLICKHOUSE_HOGQL_USE_NEW_EVENTS_SCHEMA": (
False,
- "Whether HogQL queries read from the native-JSON events tables (events_json) instead of the legacy events table.",
+ "Every project reads the native JSON events table (events_json) instead of the legacy events table. The widest of the switches: a query's useNewEventsSchema modifier, a staff person's ph_new_events_schema cookie and CLICKHOUSE_HOGQL_USE_NEW_EVENTS_SCHEMA_TEAMS each switch less.",
bool,
),
"CLICKHOUSE_HOGQL_USE_NEW_EVENTS_SCHEMA_TEAMS": (
"",
- "Comma-separated team IDs whose HogQL queries read from the native-JSON events tables even while the global setting is off.",
+ "Comma-separated project IDs that read the native JSON events table (events_json) in insights, SQL, cohorts and the background jobs that run per project. Changes reach workers within about a minute.",
str,
),
- "CLICKHOUSE_NATIVE_EVENTS_HOST": (
- get_from_env("CLICKHOUSE_NATIVE_EVENTS_HOST", ""),
- "Hostname for explicit new events cluster queries, without a scheme, port or path. Empty disables the route. Changes reach workers within about a minute.",
- str,
- ),
- "CLICKHOUSE_NATIVE_EVENTS_TEAMS": (
- [],
- "Project IDs eligible for explicit native events cluster query opt-ins. This does not change query defaults.",
- list[int],
- ),
"PERSON_ON_EVENTS_V2_ENABLED": (
# Person-on-events v2 reads are the production default for every project created
# since 2024-06-14, and both CI lanes set this variable so their runs match it. A
@@ -457,8 +447,6 @@ SETTINGS_ALLOWING_API_OVERRIDE = (
"PERSON_ON_EVENTS_V2_ENABLED",
"CLICKHOUSE_HOGQL_USE_NEW_EVENTS_SCHEMA",
"CLICKHOUSE_HOGQL_USE_NEW_EVENTS_SCHEMA_TEAMS",
- "CLICKHOUSE_NATIVE_EVENTS_HOST",
- "CLICKHOUSE_NATIVE_EVENTS_TEAMS",
"STRICT_CACHING_TEAMS",
"GITHUB_APP_SLUG",
"SLACK_APP_CLIENT_ID",
diff --git a/posthog/settings/web.py b/posthog/settings/web.py
index 2cd8ae503637..5cfaa0e7632e 100644
--- a/posthog/settings/web.py
+++ b/posthog/settings/web.py
@@ -231,6 +231,7 @@ MIDDLEWARE = [
"posthog.middleware.AutoLogoutImpersonateMiddleware",
"posthog.middleware.ImpersonationReadOnlyMiddleware",
"posthog.middleware.ImpersonationBlockedPathsMiddleware",
+ "posthog.middleware.NewEventsSchemaCookieMiddleware",
"django.middleware.clickjacking.XFrameOptionsMiddleware",
"posthog.middleware.CsvNeverCacheMiddleware",
"axes.middleware.AxesMiddleware",
diff --git a/products/analytics_platform/backend/lazy_computation/lazy_computation_executor.py b/products/analytics_platform/backend/lazy_computation/lazy_computation_executor.py
index 41a5e3243e68..a9a542609ae5 100644
--- a/products/analytics_platform/backend/lazy_computation/lazy_computation_executor.py
+++ b/products/analytics_platform/backend/lazy_computation/lazy_computation_executor.py
@@ -37,6 +37,7 @@ from posthog.clickhouse.client import sync_execute
from posthog.clickhouse.preaggregation.sql import DISTRIBUTED_PREAGGREGATION_RESULTS_TABLE
from posthog.clickhouse.query_tagging import tags_context
from posthog.dataclasses import frozen
+from posthog.models.event.new_events_schema import use_new_events_schema
from posthog.models.team import Team
from posthog.settings import DEBUG, HOGQL_INCREASED_MAX_EXECUTION_TIME, TEST
from posthog.utils import relative_date_parse_with_delta_mapping
@@ -619,6 +620,9 @@ class LazyComputationQuery:
timezone: str = "UTC"
breakdown_fields: list[str] = field(default_factory=list)
cache_key_context: dict[str, str] | None = None
+ # The modifiers of the query that made the job, so an inline build prints against the events table the
+ # identity names rather than the one the building request's own switches would pick.
+ modifiers: HogQLQueryModifiers | None = None
@dataclass(frozen=True)
@@ -657,6 +661,19 @@ def _oldest_computed_at(jobs: list[PreaggregationJob]) -> datetime | None:
return min(stamps) if stamps else None
+def events_table_cache_key_context(
+ team: Team, modifiers: HogQLQueryModifiers | None, cache_key_context: dict[str, str] | None
+) -> dict[str, str] | None:
+ """The job identity context with the events table the build reads, when it is not the legacy one.
+
+ The hashed AST names the `events` table whichever ClickHouse table it prints to, so without this a build from
+ events_json would be served to readers of the legacy table and the other way round.
+ """
+ if not use_new_events_schema(team.pk, modifiers):
+ return cache_key_context
+ return {**(cache_key_context or {}), "events_table": "events_json"}
+
+
def compute_query_hash(query_info: LazyComputationQuery) -> str:
"""
Compute a stable hash for a LazyComputationQuery object.
@@ -938,6 +955,7 @@ def build_lazy_computation_insert_sql(
time_range_start: datetime,
time_range_end: datetime,
expires_at: datetime,
+ modifiers: HogQLQueryModifiers | None = None,
) -> tuple[str, dict]:
"""
Build the INSERT ... SELECT SQL for populating lazy-computed results.
@@ -994,7 +1012,13 @@ def build_lazy_computation_insert_sql(
query.where = date_range_filter
# Print the SELECT query to ClickHouse SQL
- context = HogQLContext(team_id=team.id, team=team, enable_select_queries=True, limit_top_select=False)
+ context = HogQLContext(
+ team_id=team.id,
+ team=team,
+ enable_select_queries=True,
+ limit_top_select=False,
+ modifiers=modifiers if modifiers is not None else HogQLQueryModifiers(),
+ )
select_sql, _ = prepare_and_print_ast(
query,
context=context,
@@ -1043,6 +1067,7 @@ def run_lazy_computation_insert(
time_range_start=job.time_range_start,
time_range_end=job.time_range_end,
expires_at=ch_expires_at,
+ modifiers=query_info.modifiers,
)
set_ch_query_started(job.id)
@@ -1810,6 +1835,11 @@ def ensure_precomputed(
}
parsed_for_hash = _resolve_insert_query(insert_query, hash_placeholders)
+ # The INSERT's printer asks the same switches for its table, so naming the table here keeps the job identity
+ # and the rows under it in step, whichever switch (a modifier, the person's cookie, the project or the
+ # instance) picked it.
+ cache_key_context = events_table_cache_key_context(team, modifiers, cache_key_context)
+
query_info = LazyComputationQuery(
query=parsed_for_hash,
table=table,
diff --git a/products/analytics_platform/backend/lazy_computation/lazy_computation_transformer.py b/products/analytics_platform/backend/lazy_computation/lazy_computation_transformer.py
index 5317ce025d4e..ff838b5c2e03 100644
--- a/products/analytics_platform/backend/lazy_computation/lazy_computation_transformer.py
+++ b/products/analytics_platform/backend/lazy_computation/lazy_computation_transformer.py
@@ -1,6 +1,8 @@
from datetime import datetime, timedelta
from typing import Optional, Union
+from posthog.schema import HogQLQueryModifiers
+
from posthog.hogql import ast
from posthog.hogql.ast import CompareOperationOp
from posthog.hogql.context import HogQLContext
@@ -15,6 +17,7 @@ from products.analytics_platform.backend.lazy_computation.lazy_computation_execu
LazyComputationQuery,
LazyComputationResult,
LazyComputationTable,
+ events_table_cache_key_context,
)
PREAGGREGATED_DAILY_UNIQUE_PERSONS_PAGEVIEWS_TABLE_NAME = DISTRIBUTED_PREAGGREGATION_RESULTS_TABLE()
@@ -458,6 +461,7 @@ def _run_daily_unique_persons_pageviews(
query_to_insert: ast.SelectQuery,
start: datetime,
end: datetime,
+ modifiers: HogQLQueryModifiers,
) -> LazyComputationResult:
"""
Orchestrate lazy computation jobs for daily unique persons pageviews.
@@ -468,7 +472,11 @@ def _run_daily_unique_persons_pageviews(
3. Returns the result with job IDs for the combiner query
"""
query_info = LazyComputationQuery(
- query=query_to_insert, table=LazyComputationTable.PREAGGREGATION_RESULTS, timezone=team.timezone
+ query=query_to_insert,
+ table=LazyComputationTable.PREAGGREGATION_RESULTS,
+ timezone=team.timezone,
+ cache_key_context=events_table_cache_key_context(team, modifiers, None),
+ modifiers=modifiers,
)
executor = LazyComputationExecutor()
@@ -514,7 +522,7 @@ class Transformer(CloningVisitor):
team = self.context.team
if not team:
return transformed_node
- result = _run_daily_unique_persons_pageviews(team, query_to_insert, start_dt, end_dt)
+ result = _run_daily_unique_persons_pageviews(team, query_to_insert, start_dt, end_dt, self.context.modifiers)
if not result.ready:
# Lazy computation not ready, fall back to original query
diff --git a/products/analytics_platform/backend/lazy_computation/tests/test_lazy_computation_executor.py b/products/analytics_platform/backend/lazy_computation/tests/test_lazy_computation_executor.py
index be702d58c235..c990f26c2b4a 100644
--- a/products/analytics_platform/backend/lazy_computation/tests/test_lazy_computation_executor.py
+++ b/products/analytics_platform/backend/lazy_computation/tests/test_lazy_computation_executor.py
@@ -7,7 +7,7 @@ from posthog.test.base import BaseTest, ClickhouseTestMixin, _create_event
from unittest.mock import patch
from django.db import IntegrityError
-from django.test import SimpleTestCase
+from django.test import SimpleTestCase, override_settings
from django.utils import timezone as django_timezone
from clickhouse_driver.errors import ServerException
@@ -21,6 +21,8 @@ from posthog.hogql.query import execute_hogql_query
from posthog.clickhouse.client import sync_execute
from posthog.clickhouse.preaggregation.sql import DISTRIBUTED_PREAGGREGATION_RESULTS_TABLE
+from posthog.models.event.new_events_schema import new_events_schema_for_request
+from posthog.models.instance_setting import override_instance_config
from posthog.settings import HOGQL_INCREASED_MAX_EXECUTION_TIME
from products.analytics_platform.backend.lazy_computation.computation_notifications import (
@@ -3929,3 +3931,72 @@ class TestCheckOnlyMode(BaseTest):
result = self._execute(datetime(2024, 1, 1, tzinfo=UTC), datetime(2024, 1, 3, tzinfo=UTC))
assert result.ready is False
assert time_mod.monotonic() - started < 5, "check-only must not block on pending jobs"
+
+
+class TestBuildsKeepTheirEventsTable(BaseTest):
+ INSERT_QUERY = TestInsertSettingsAppliedToInserts.INSERT_QUERY
+
+ @override_settings(CLICKHOUSE_HOGQL_USE_NEW_EVENTS_SCHEMA=False)
+ def test_a_switched_persons_build_gets_a_job_of_its_own(self) -> None:
+ with (
+ override_instance_config("CLICKHOUSE_HOGQL_USE_NEW_EVENTS_SCHEMA", False),
+ patch(
+ "products.analytics_platform.backend.lazy_computation.lazy_computation_executor.sync_execute",
+ return_value=0,
+ ) as execute,
+ ):
+ legacy = ensure_precomputed(
+ team=self.team,
+ insert_query=self.INSERT_QUERY,
+ time_range_start=datetime(2024, 1, 1, tzinfo=UTC),
+ time_range_end=datetime(2024, 1, 2, tzinfo=UTC),
+ ttl_seconds=7 * 24 * 60 * 60,
+ )
+ assert "events_json" not in execute.call_args.args[0]
+ with new_events_schema_for_request():
+ native = ensure_precomputed(
+ team=self.team,
+ insert_query=self.INSERT_QUERY,
+ time_range_start=datetime(2024, 1, 1, tzinfo=UTC),
+ time_range_end=datetime(2024, 1, 2, tzinfo=UTC),
+ ttl_seconds=7 * 24 * 60 * 60,
+ )
+ assert "events_json" in execute.call_args.args[0]
+ # The two builds read different tables, so neither may be served the other's rows.
+ assert set(legacy.job_ids).isdisjoint(native.job_ids)
+
+ @override_settings(CLICKHOUSE_HOGQL_USE_NEW_EVENTS_SCHEMA=False)
+ def test_an_ast_build_prints_the_table_its_job_identity_names(self) -> None:
+ job = PreaggregationJob.objects.create(
+ team=self.team,
+ query_hash="test_hash",
+ time_range_start=datetime(2024, 1, 1, tzinfo=UTC),
+ time_range_end=datetime(2024, 1, 2, tzinfo=UTC),
+ status=PreaggregationJob.Status.PENDING,
+ expires_at=django_timezone.now() + timedelta(days=7),
+ )
+ query = parse_select(
+ self.INSERT_QUERY,
+ placeholders={
+ "time_window_min": ast.Constant(value=datetime(2024, 1, 1, tzinfo=UTC)),
+ "time_window_max": ast.Constant(value=datetime(2024, 1, 2, tzinfo=UTC)),
+ },
+ )
+ assert isinstance(query, ast.SelectQuery)
+ # The query that made the job pinned the legacy table, so the building person's cookie must not change
+ # what the build reads.
+ query_info = LazyComputationQuery(
+ query=query,
+ table=LazyComputationTable.PREAGGREGATION_RESULTS,
+ modifiers=HogQLQueryModifiers(useNewEventsSchema=False),
+ )
+ with (
+ override_instance_config("CLICKHOUSE_HOGQL_USE_NEW_EVENTS_SCHEMA", False),
+ new_events_schema_for_request(),
+ patch(
+ "products.analytics_platform.backend.lazy_computation.lazy_computation_executor.sync_execute",
+ return_value=0,
+ ) as execute,
+ ):
+ run_lazy_computation_insert(self.team, job, query_info)
+ assert "events_json" not in execute.call_args.args[0]
diff --git a/products/batch_exports/backend/presentation/views/utils.py b/products/batch_exports/backend/presentation/views/utils.py
index 74a3bc080fa6..7393e8a57378 100644
--- a/products/batch_exports/backend/presentation/views/utils.py
+++ b/products/batch_exports/backend/presentation/views/utils.py
@@ -90,10 +90,6 @@ class HogQLModifiersField(serializers.JSONField):
messages.append(f"{location}: {error['msg']}" if location else error["msg"])
raise serializers.ValidationError(messages) from e
else:
- if modifiers.useNewEventsCluster:
- raise serializers.ValidationError(
- "Native events cluster queries are only supported by the synchronous query API"
- )
self.loaded = modifiers
return modifiers.model_dump(mode="json", exclude_none=True)
diff --git a/products/batch_exports/backend/tests/api/test_create.py b/products/batch_exports/backend/tests/api/test_create.py
index 52caacce1e07..c84f38794021 100644
--- a/products/batch_exports/backend/tests/api/test_create.py
+++ b/products/batch_exports/backend/tests/api/test_create.py
@@ -748,11 +748,6 @@ def test_create_batch_export_with_hogql_model_allows_query_without_placeholders(
"hogql_modifiers",
"notAModifier: Extra inputs are not permitted",
),
- (
- {"hogql_modifiers": {"useNewEventsCluster": True}},
- "hogql_modifiers",
- "Native events cluster queries are only supported by the synchronous query API",
- ),
(
{"model": "events", "hogql_modifiers": {"convertToProjectTimezone": False}},
"hogql_modifiers",
@@ -766,7 +761,6 @@ def test_create_batch_export_with_hogql_model_allows_query_without_placeholders(
"unaliased-column",
"filters",
"unknown-modifier",
- "native-cluster-modifier",
"modifiers-with-events-model",
],
)
diff --git a/products/batch_exports/backend/tests/api/test_serializers.py b/products/batch_exports/backend/tests/api/test_serializers.py
index df2f4769e20b..b78684b014c1 100644
--- a/products/batch_exports/backend/tests/api/test_serializers.py
+++ b/products/batch_exports/backend/tests/api/test_serializers.py
@@ -23,7 +23,6 @@ from products.batch_exports.backend.presentation.views.batch_export.exports impo
BatchExportSerializer,
parse_events_hogql_query,
)
-from products.batch_exports.backend.presentation.views.utils import HogQLModifiersField
class TestBatchExportModelValidation(SimpleTestCase):
@@ -50,23 +49,6 @@ class TestBatchExportModelValidation(SimpleTestCase):
assert serializer.is_valid(), serializer.errors
assert serializer.validated_data == {"name": "Renamed"}
- @parameterized.expand(
- [
- ("unset", {}, {}),
- ("false", {"useNewEventsCluster": False}, {"useNewEventsCluster": False}),
- ("true", {"useNewEventsCluster": True}, None),
- ]
- )
- def test_cluster_opt_ins_are_not_supported_by_exports(
- self, _name: str, modifiers: dict, expected: dict | None
- ) -> None:
- field = HogQLModifiersField()
- if expected is None:
- with self.assertRaisesMessage(serializers.ValidationError, "synchronous query API"):
- field.run_validation(modifiers)
- else:
- assert field.run_validation(modifiers) == expected
-
def prepare_query(query: str, team_id: int) -> ast.SelectQuery:
"""Parse and resolve a HogQL query string into a prepared AST."""
diff --git a/products/batch_exports/backend/tests/temporal/destinations/file_download/test_file_download_api.py b/products/batch_exports/backend/tests/temporal/destinations/file_download/test_file_download_api.py
index d854c7df364f..6361d200aeb4 100644
--- a/products/batch_exports/backend/tests/temporal/destinations/file_download/test_file_download_api.py
+++ b/products/batch_exports/backend/tests/temporal/destinations/file_download/test_file_download_api.py
@@ -826,11 +826,6 @@ class TestFileDownloadHogQL:
"'data_interval_start' and 'data_interval_end' are required",
id="events-model-missing-intervals",
),
- pytest.param(
- {"hogql_modifiers": {"useNewEventsCluster": True}},
- "synchronous query API",
- id="native-cluster-modifier",
- ),
pytest.param(
{"model": "events", "hogql_query": None, "hogql_modifiers": {"convertToProjectTimezone": False}},
"'hogql_modifiers' are only supported when 'model' is 'hogql'",
@@ -1256,15 +1251,6 @@ class TestFileDownloadHogQL:
"Extra inputs are not permitted",
id="unknown-modifier",
),
- pytest.param(
- {
- "model": "hogql",
- "hogql_query": "SELECT event AS event FROM events",
- "hogql_modifiers": {"useNewEventsCluster": True},
- },
- "synchronous query API",
- id="native-cluster-modifier",
- ),
],
)
@pytest.mark.usefixtures("enable_hogql_flag")
diff --git a/products/batch_exports/frontend/generated/api.schemas.ts b/products/batch_exports/frontend/generated/api.schemas.ts
index 05836f264e9c..ce0051bdd3d7 100644
--- a/products/batch_exports/frontend/generated/api.schemas.ts
+++ b/products/batch_exports/frontend/generated/api.schemas.ts
@@ -861,9 +861,7 @@ export interface HogQLQueryModifiersApi {
/** Remove provably redundant casts and nullability wrappers (e.g. `toString(String)`, `assumeNotNull(non_nullable)`, dead `ifNull` fallbacks) using inferred expression types */
typeAwareCastSimplification?: boolean | null
useMaterializedViews?: boolean | null
- /** Read this query from the native events cluster and native JSON events table (`true`), or keep its existing connection (`false` or unset). This is an internal rollout switch for eligible projects. PostHog staff manage eligibility through instance settings. Each query must opt in; project defaults cannot enable it. Only synchronous HogQLQuery SQL comparisons are supported. */
- useNewEventsCluster?: boolean | null
- /** Read events from the native JSON events table (`true`) or the legacy events table (`false`). When unset, the project's stored value applies, then the `CLICKHOUSE_HOGQL_USE_NEW_EVENTS_SCHEMA` instance settings. This is an internal rollout switch. PostHog staff set the project value in Django admin and the project settings API ignores it. */
+ /** Read events from the native JSON events table (`true`) or the legacy events table (`false`). When unset, a staff person's `ph_new_events_schema` cookie applies, then the project's listing in the `CLICKHOUSE_HOGQL_USE_NEW_EVENTS_SCHEMA_TEAMS` instance setting, then the `CLICKHOUSE_HOGQL_USE_NEW_EVENTS_SCHEMA` instance setting. This is an internal rollout switch, and the project settings API ignores it. */
useNewEventsSchema?: boolean | null
usePreaggregatedIntermediateResults?: boolean | null
/** Try to automatically convert HogQL queries to use preaggregated tables at the AST level * */
diff --git a/products/batch_exports/frontend/generated/api.zod.ts b/products/batch_exports/frontend/generated/api.zod.ts
index 24bfb150c0db..42ebb894ac41 100644
--- a/products/batch_exports/frontend/generated/api.zod.ts
+++ b/products/batch_exports/frontend/generated/api.zod.ts
@@ -669,17 +669,11 @@ export const BatchExportsCreateBody = /* @__PURE__ */ zod
'Remove provably redundant casts and nullability wrappers (e.g. `toString(String)`, `assumeNotNull(non_nullable)`, dead `ifNull` fallbacks) using inferred expression types'
),
useMaterializedViews: zod.union([zod.boolean(), zod.null()]).optional(),
- useNewEventsCluster: zod
- .union([zod.boolean(), zod.null()])
- .optional()
- .describe(
- 'Read this query from the native events cluster and native JSON events table (`true`), or keep its existing connection (`false` or unset). This is an internal rollout switch for eligible projects. PostHog staff manage eligibility through instance settings. Each query must opt in; project defaults cannot enable it. Only synchronous HogQLQuery SQL comparisons are supported.'
- ),
useNewEventsSchema: zod
.union([zod.boolean(), zod.null()])
.optional()
.describe(
- "Read events from the native JSON events table (`true`) or the legacy events table (`false`). When unset, the project's stored value applies, then the `CLICKHOUSE_HOGQL_USE_NEW_EVENTS_SCHEMA` instance settings. This is an internal rollout switch. PostHog staff set the project value in Django admin and the project settings API ignores it."
+ "Read events from the native JSON events table (`true`) or the legacy events table (`false`). When unset, a staff person's `ph_new_events_schema` cookie applies, then the project's listing in the `CLICKHOUSE_HOGQL_USE_NEW_EVENTS_SCHEMA_TEAMS` instance setting, then the `CLICKHOUSE_HOGQL_USE_NEW_EVENTS_SCHEMA` instance setting. This is an internal rollout switch, and the project settings API ignores it."
),
usePreaggregatedIntermediateResults: zod.union([zod.boolean(), zod.null()]).optional(),
usePreaggregatedTableTransforms: zod
@@ -1655,17 +1649,11 @@ export const BatchExportsUpdateBody = /* @__PURE__ */ zod
'Remove provably redundant casts and nullability wrappers (e.g. `toString(String)`, `assumeNotNull(non_nullable)`, dead `ifNull` fallbacks) using inferred expression types'
),
useMaterializedViews: zod.union([zod.boolean(), zod.null()]).optional(),
- useNewEventsCluster: zod
- .union([zod.boolean(), zod.null()])
- .optional()
- .describe(
- 'Read this query from the native events cluster and native JSON events table (`true`), or keep its existing connection (`false` or unset). This is an internal rollout switch for eligible projects. PostHog staff manage eligibility through instance settings. Each query must opt in; project defaults cannot enable it. Only synchronous HogQLQuery SQL comparisons are supported.'
- ),
useNewEventsSchema: zod
.union([zod.boolean(), zod.null()])
.optional()
.describe(
- "Read events from the native JSON events table (`true`) or the legacy events table (`false`). When unset, the project's stored value applies, then the `CLICKHOUSE_HOGQL_USE_NEW_EVENTS_SCHEMA` instance settings. This is an internal rollout switch. PostHog staff set the project value in Django admin and the project settings API ignores it."
+ "Read events from the native JSON events table (`true`) or the legacy events table (`false`). When unset, a staff person's `ph_new_events_schema` cookie applies, then the project's listing in the `CLICKHOUSE_HOGQL_USE_NEW_EVENTS_SCHEMA_TEAMS` instance setting, then the `CLICKHOUSE_HOGQL_USE_NEW_EVENTS_SCHEMA` instance setting. This is an internal rollout switch, and the project settings API ignores it."
),
usePreaggregatedIntermediateResults: zod.union([zod.boolean(), zod.null()]).optional(),
usePreaggregatedTableTransforms: zod
@@ -2400,17 +2388,11 @@ export const BatchExportsPartialUpdateBody = /* @__PURE__ */ zod
'Remove provably redundant casts and nullability wrappers (e.g. `toString(String)`, `assumeNotNull(non_nullable)`, dead `ifNull` fallbacks) using inferred expression types'
),
useMaterializedViews: zod.union([zod.boolean(), zod.null()]).optional(),
- useNewEventsCluster: zod
- .union([zod.boolean(), zod.null()])
- .optional()
- .describe(
- 'Read this query from the native events cluster and native JSON events table (`true`), or keep its existing connection (`false` or unset). This is an internal rollout switch for eligible projects. PostHog staff manage eligibility through instance settings. Each query must opt in; project defaults cannot enable it. Only synchronous HogQLQuery SQL comparisons are supported.'
- ),
useNewEventsSchema: zod
.union([zod.boolean(), zod.null()])
.optional()
.describe(
- "Read events from the native JSON events table (`true`) or the legacy events table (`false`). When unset, the project's stored value applies, then the `CLICKHOUSE_HOGQL_USE_NEW_EVENTS_SCHEMA` instance settings. This is an internal rollout switch. PostHog staff set the project value in Django admin and the project settings API ignores it."
+ "Read events from the native JSON events table (`true`) or the legacy events table (`false`). When unset, a staff person's `ph_new_events_schema` cookie applies, then the project's listing in the `CLICKHOUSE_HOGQL_USE_NEW_EVENTS_SCHEMA_TEAMS` instance setting, then the `CLICKHOUSE_HOGQL_USE_NEW_EVENTS_SCHEMA` instance setting. This is an internal rollout switch, and the project settings API ignores it."
),
usePreaggregatedIntermediateResults: zod.union([zod.boolean(), zod.null()]).optional(),
usePreaggregatedTableTransforms: zod
@@ -3130,17 +3112,11 @@ export const BatchExportsRunTestStepCreateBody = /* @__PURE__ */ zod
'Remove provably redundant casts and nullability wrappers (e.g. `toString(String)`, `assumeNotNull(non_nullable)`, dead `ifNull` fallbacks) using inferred expression types'
),
useMaterializedViews: zod.union([zod.boolean(), zod.null()]).optional(),
- useNewEventsCluster: zod
- .union([zod.boolean(), zod.null()])
- .optional()
- .describe(
- 'Read this query from the native events cluster and native JSON events table (`true`), or keep its existing connection (`false` or unset). This is an internal rollout switch for eligible projects. PostHog staff manage eligibility through instance settings. Each query must opt in; project defaults cannot enable it. Only synchronous HogQLQuery SQL comparisons are supported.'
- ),
useNewEventsSchema: zod
.union([zod.boolean(), zod.null()])
.optional()
.describe(
- "Read events from the native JSON events table (`true`) or the legacy events table (`false`). When unset, the project's stored value applies, then the `CLICKHOUSE_HOGQL_USE_NEW_EVENTS_SCHEMA` instance settings. This is an internal rollout switch. PostHog staff set the project value in Django admin and the project settings API ignores it."
+ "Read events from the native JSON events table (`true`) or the legacy events table (`false`). When unset, a staff person's `ph_new_events_schema` cookie applies, then the project's listing in the `CLICKHOUSE_HOGQL_USE_NEW_EVENTS_SCHEMA_TEAMS` instance setting, then the `CLICKHOUSE_HOGQL_USE_NEW_EVENTS_SCHEMA` instance setting. This is an internal rollout switch, and the project settings API ignores it."
),
usePreaggregatedIntermediateResults: zod.union([zod.boolean(), zod.null()]).optional(),
usePreaggregatedTableTransforms: zod
@@ -3879,17 +3855,11 @@ export const BatchExportsRunTestStepNewCreateBody = /* @__PURE__ */ zod
'Remove provably redundant casts and nullability wrappers (e.g. `toString(String)`, `assumeNotNull(non_nullable)`, dead `ifNull` fallbacks) using inferred expression types'
),
useMaterializedViews: zod.union([zod.boolean(), zod.null()]).optional(),
- useNewEventsCluster: zod
- .union([zod.boolean(), zod.null()])
- .optional()
- .describe(
- 'Read this query from the native events cluster and native JSON events table (`true`), or keep its existing connection (`false` or unset). This is an internal rollout switch for eligible projects. PostHog staff manage eligibility through instance settings. Each query must opt in; project defaults cannot enable it. Only synchronous HogQLQuery SQL comparisons are supported.'
- ),
useNewEventsSchema: zod
.union([zod.boolean(), zod.null()])
.optional()
.describe(
- "Read events from the native JSON events table (`true`) or the legacy events table (`false`). When unset, the project's stored value applies, then the `CLICKHOUSE_HOGQL_USE_NEW_EVENTS_SCHEMA` instance settings. This is an internal rollout switch. PostHog staff set the project value in Django admin and the project settings API ignores it."
+ "Read events from the native JSON events table (`true`) or the legacy events table (`false`). When unset, a staff person's `ph_new_events_schema` cookie applies, then the project's listing in the `CLICKHOUSE_HOGQL_USE_NEW_EVENTS_SCHEMA_TEAMS` instance setting, then the `CLICKHOUSE_HOGQL_USE_NEW_EVENTS_SCHEMA` instance setting. This is an internal rollout switch, and the project settings API ignores it."
),
usePreaggregatedIntermediateResults: zod.union([zod.boolean(), zod.null()]).optional(),
usePreaggregatedTableTransforms: zod
@@ -4345,17 +4315,11 @@ export const FileDownloadBatchExportsCreateBody = /* @__PURE__ */ zod.union([
'Remove provably redundant casts and nullability wrappers (e.g. `toString(String)`, `assumeNotNull(non_nullable)`, dead `ifNull` fallbacks) using inferred expression types'
),
useMaterializedViews: zod.union([zod.boolean(), zod.null()]).optional(),
- useNewEventsCluster: zod
- .union([zod.boolean(), zod.null()])
- .optional()
- .describe(
- 'Read this query from the native events cluster and native JSON events table (`true`), or keep its existing connection (`false` or unset). This is an internal rollout switch for eligible projects. PostHog staff manage eligibility through instance settings. Each query must opt in; project defaults cannot enable it. Only synchronous HogQLQuery SQL comparisons are supported.'
- ),
useNewEventsSchema: zod
.union([zod.boolean(), zod.null()])
.optional()
.describe(
- "Read events from the native JSON events table (`true`) or the legacy events table (`false`). When unset, the project's stored value applies, then the `CLICKHOUSE_HOGQL_USE_NEW_EVENTS_SCHEMA` instance settings. This is an internal rollout switch. PostHog staff set the project value in Django admin and the project settings API ignores it."
+ "Read events from the native JSON events table (`true`) or the legacy events table (`false`). When unset, a staff person's `ph_new_events_schema` cookie applies, then the project's listing in the `CLICKHOUSE_HOGQL_USE_NEW_EVENTS_SCHEMA_TEAMS` instance setting, then the `CLICKHOUSE_HOGQL_USE_NEW_EVENTS_SCHEMA` instance setting. This is an internal rollout switch, and the project settings API ignores it."
),
usePreaggregatedIntermediateResults: zod.union([zod.boolean(), zod.null()]).optional(),
usePreaggregatedTableTransforms: zod
@@ -4615,17 +4579,11 @@ export const FileDownloadBatchExportsCountRowsCreateBody = /* @__PURE__ */ zod
'Remove provably redundant casts and nullability wrappers (e.g. `toString(String)`, `assumeNotNull(non_nullable)`, dead `ifNull` fallbacks) using inferred expression types'
),
useMaterializedViews: zod.union([zod.boolean(), zod.null()]).optional(),
- useNewEventsCluster: zod
- .union([zod.boolean(), zod.null()])
- .optional()
- .describe(
- 'Read this query from the native events cluster and native JSON events table (`true`), or keep its existing connection (`false` or unset). This is an internal rollout switch for eligible projects. PostHog staff manage eligibility through instance settings. Each query must opt in; project defaults cannot enable it. Only synchronous HogQLQuery SQL comparisons are supported.'
- ),
useNewEventsSchema: zod
.union([zod.boolean(), zod.null()])
.optional()
.describe(
- "Read events from the native JSON events table (`true`) or the legacy events table (`false`). When unset, the project's stored value applies, then the `CLICKHOUSE_HOGQL_USE_NEW_EVENTS_SCHEMA` instance settings. This is an internal rollout switch. PostHog staff set the project value in Django admin and the project settings API ignores it."
+ "Read events from the native JSON events table (`true`) or the legacy events table (`false`). When unset, a staff person's `ph_new_events_schema` cookie applies, then the project's listing in the `CLICKHOUSE_HOGQL_USE_NEW_EVENTS_SCHEMA_TEAMS` instance setting, then the `CLICKHOUSE_HOGQL_USE_NEW_EVENTS_SCHEMA` instance setting. This is an internal rollout switch, and the project settings API ignores it."
),
usePreaggregatedIntermediateResults: zod.union([zod.boolean(), zod.null()]).optional(),
usePreaggregatedTableTransforms: zod
diff --git a/products/customer_analytics/frontend/generated/api.schemas.ts b/products/customer_analytics/frontend/generated/api.schemas.ts
index 024624af5a9f..606f9b96432d 100644
--- a/products/customer_analytics/frontend/generated/api.schemas.ts
+++ b/products/customer_analytics/frontend/generated/api.schemas.ts
@@ -1791,9 +1791,7 @@ export interface HogQLQueryModifiersApi {
/** Remove provably redundant casts and nullability wrappers (e.g. `toString(String)`, `assumeNotNull(non_nullable)`, dead `ifNull` fallbacks) using inferred expression types */
typeAwareCastSimplification?: boolean | null
useMaterializedViews?: boolean | null
- /** Read this query from the native events cluster and native JSON events table (`true`), or keep its existing connection (`false` or unset). This is an internal rollout switch for eligible projects. PostHog staff manage eligibility through instance settings. Each query must opt in; project defaults cannot enable it. Only synchronous HogQLQuery SQL comparisons are supported. */
- useNewEventsCluster?: boolean | null
- /** Read events from the native JSON events table (`true`) or the legacy events table (`false`). When unset, the project's stored value applies, then the `CLICKHOUSE_HOGQL_USE_NEW_EVENTS_SCHEMA` instance settings. This is an internal rollout switch. PostHog staff set the project value in Django admin and the project settings API ignores it. */
+ /** Read events from the native JSON events table (`true`) or the legacy events table (`false`). When unset, a staff person's `ph_new_events_schema` cookie applies, then the project's listing in the `CLICKHOUSE_HOGQL_USE_NEW_EVENTS_SCHEMA_TEAMS` instance setting, then the `CLICKHOUSE_HOGQL_USE_NEW_EVENTS_SCHEMA` instance setting. This is an internal rollout switch, and the project settings API ignores it. */
useNewEventsSchema?: boolean | null
usePreaggregatedIntermediateResults?: boolean | null
/** Try to automatically convert HogQL queries to use preaggregated tables at the AST level * */
diff --git a/products/dashboards/backend/api/dashboard.py b/products/dashboards/backend/api/dashboard.py
index 33b9bca05350..a100bb45ebc5 100644
--- a/products/dashboards/backend/api/dashboard.py
+++ b/products/dashboards/backend/api/dashboard.py
@@ -530,6 +530,14 @@ def serialize_tile_with_context(tile, order: int, context: dict) -> tuple[int, d
return order, tile_data
+def serialize_tile_in_request_context(
+ request_context: contextvars.Context, tile: DashboardTile, order: int, context: dict
+) -> tuple[int, dict]:
+ # A streamed response body runs after the middleware chain has returned, so the request's context variables,
+ # the events table switch among them, reach a tile's queries only through a copy taken inside the view.
+ return request_context.run(serialize_tile_with_context, tile, order, context)
+
+
class ReorderLayout(StrEnum):
PRESERVE = "preserve"
TWO_COLUMN = "two_column"
@@ -3327,6 +3335,8 @@ class DashboardsViewSet(
# below runs no queries for them.
team_configs_for_cache_key(self.team)
+ request_context = contextvars.copy_context()
+
# Async generator that handles progressive tile serialization and streaming
async def async_tile_stream_generator() -> AsyncGenerator[bytes]:
renderer = SafeJSONRenderer()
@@ -3340,8 +3350,8 @@ class DashboardsViewSet(
tile = sorted_tiles[order]
try:
order_result, tile_data = await database_sync_to_async(
- serialize_tile_with_context, thread_sensitive=True
- )(tile, order, context)
+ serialize_tile_in_request_context, thread_sensitive=True
+ )(request_context, tile, order, context)
initial_tiles.append(tile_data)
except Exception:
logger.exception("Error serializing initial dashboard tile", tile_id=tile.id)
@@ -3363,8 +3373,8 @@ class DashboardsViewSet(
tile = sorted_tiles[order]
try:
order_result, tile_data = await database_sync_to_async(
- serialize_tile_with_context, thread_sensitive=True
- )(tile, order, context)
+ serialize_tile_in_request_context, thread_sensitive=True
+ )(request_context, tile, order, context)
tile_json = renderer.render({"type": "tile", "order": order, "tile": tile_data}).decode()
yield f"data: {tile_json}\n\n".encode()
except Exception:
diff --git a/products/dashboards/frontend/generated/api.schemas.ts b/products/dashboards/frontend/generated/api.schemas.ts
index 80a565b21322..5a36d0c5e52a 100644
--- a/products/dashboards/frontend/generated/api.schemas.ts
+++ b/products/dashboards/frontend/generated/api.schemas.ts
@@ -2454,9 +2454,7 @@ export interface HogQLQueryModifiersApi {
/** Remove provably redundant casts and nullability wrappers (e.g. `toString(String)`, `assumeNotNull(non_nullable)`, dead `ifNull` fallbacks) using inferred expression types */
typeAwareCastSimplification?: boolean | null
useMaterializedViews?: boolean | null
- /** Read this query from the native events cluster and native JSON events table (`true`), or keep its existing connection (`false` or unset). This is an internal rollout switch for eligible projects. PostHog staff manage eligibility through instance settings. Each query must opt in; project defaults cannot enable it. Only synchronous HogQLQuery SQL comparisons are supported. */
- useNewEventsCluster?: boolean | null
- /** Read events from the native JSON events table (`true`) or the legacy events table (`false`). When unset, the project's stored value applies, then the `CLICKHOUSE_HOGQL_USE_NEW_EVENTS_SCHEMA` instance settings. This is an internal rollout switch. PostHog staff set the project value in Django admin and the project settings API ignores it. */
+ /** Read events from the native JSON events table (`true`) or the legacy events table (`false`). When unset, a staff person's `ph_new_events_schema` cookie applies, then the project's listing in the `CLICKHOUSE_HOGQL_USE_NEW_EVENTS_SCHEMA_TEAMS` instance setting, then the `CLICKHOUSE_HOGQL_USE_NEW_EVENTS_SCHEMA` instance setting. This is an internal rollout switch, and the project settings API ignores it. */
useNewEventsSchema?: boolean | null
usePreaggregatedIntermediateResults?: boolean | null
/** Try to automatically convert HogQL queries to use preaggregated tables at the AST level * */
diff --git a/products/desktop/packages/api-client/src/generated.ts b/products/desktop/packages/api-client/src/generated.ts
index 9fec9351d7e5..e870ed4994b2 100644
--- a/products/desktop/packages/api-client/src/generated.ts
+++ b/products/desktop/packages/api-client/src/generated.ts
@@ -234,7 +234,6 @@ export namespace Schemas {
timings: boolean | null;
typeAwareCastSimplification: boolean | null;
useMaterializedViews: boolean | null;
- useNewEventsCluster: boolean | null;
useNewEventsSchema: boolean | null;
usePreaggregatedIntermediateResults: boolean | null;
usePreaggregatedTableTransforms: boolean | null;
diff --git a/products/endpoints/backend/logic/execution.py b/products/endpoints/backend/logic/execution.py
index 152c80896537..a7af4a078cd1 100644
--- a/products/endpoints/backend/logic/execution.py
+++ b/products/endpoints/backend/logic/execution.py
@@ -78,7 +78,6 @@ from products.endpoints.backend.logic.materialized_serving import serving_state_
from products.endpoints.backend.logic.pagination import EndpointPagination
from products.endpoints.backend.logic.rate_limit_signal import EndpointRateLimitReason, record_endpoint_rate_limited
from products.endpoints.backend.logic.strategies import EndpointQueryStrategy, strategy_for
-from products.endpoints.backend.logic.validation import NATIVE_CLUSTER_UNSUPPORTED
from products.endpoints.backend.logs import build_execution_message, log_endpoint_execution
from products.endpoints.backend.metrics import (
ENDPOINT_CACHE_RESULT_TOTAL,
@@ -375,8 +374,6 @@ class EndpointExecutionService(PydanticModelMixin):
version: EndpointVersion,
offset: int | None = None,
) -> None:
- if (version.query.get("modifiers") or {}).get("useNewEventsCluster"):
- raise ValidationError({"query": NATIVE_CLUSTER_UNSUPPORTED})
strategy = strategy_for(endpoint, version, self.team)
is_materialized = bool(version.is_materialized and version.saved_query)
diff --git a/products/endpoints/backend/logic/validation.py b/products/endpoints/backend/logic/validation.py
index 01381207f2c1..e88dfb1cc972 100644
--- a/products/endpoints/backend/logic/validation.py
+++ b/products/endpoints/backend/logic/validation.py
@@ -69,9 +69,6 @@ DIRECT_CONNECTION_UNSUPPORTED = (
"Endpoints can't query a direct connection. "
"Switch the connection to PostHog (ClickHouse) and query a synced table instead."
)
-NATIVE_CLUSTER_UNSUPPORTED = (
- "Endpoints do not support native events cluster opt-ins. Use the query API for comparisons."
-)
def validate_hogql_query(query: HogQLQuery, team: Team, user: User) -> None:
@@ -281,9 +278,6 @@ def validate_optional_breakdown_properties(
def validate_endpoint_request(data: EndpointRequest, team: Team, user: User, strict: bool = True) -> None:
"""Validate a create/update payload. With strict=True, name and query are required."""
query = data.query
- modifiers = getattr(query, "modifiers", None)
- if modifiers and modifiers.useNewEventsCluster:
- raise ValidationError({"query": NATIVE_CLUSTER_UNSUPPORTED})
if not query and strict:
raise ValidationError({"query": "This field is required."})
@@ -316,9 +310,6 @@ def validate_update_request(
version_number: int | None = None,
) -> None:
"""Validate an update payload against the endpoint's resulting state."""
- modifiers = getattr(data.query, "modifiers", None)
- if modifiers and modifiers.useNewEventsCluster:
- raise ValidationError({"query": NATIVE_CLUSTER_UNSUPPORTED})
validate_data_freshness(data.data_freshness_seconds)
# Determine final states after this request (for validation)
diff --git a/products/endpoints/backend/tests/test_endpoint.py b/products/endpoints/backend/tests/test_endpoint.py
index a27e0963a29a..4e4bfcce639d 100644
--- a/products/endpoints/backend/tests/test_endpoint.py
+++ b/products/endpoints/backend/tests/test_endpoint.py
@@ -69,22 +69,14 @@ class TestEndpoint(ClickhouseTestMixin, APIBaseTest):
"series": [{"kind": "EventsNode"}],
}
- @parameterized.expand([(None,), (True,)])
- def test_create_hogql_endpoint(self, native: bool | None):
+ def test_create_hogql_endpoint(self):
data = {
"name": "test_query",
"description": "Test query description",
"query": self.sample_hogql_query,
}
- if native is not None:
- data["query"] = {**self.sample_hogql_query, "modifiers": {"useNewEventsCluster": native}}
response = self.client.post(f"/api/environments/{self.team.id}/endpoints/", data, format="json")
- if native:
- self.assertEqual(status.HTTP_400_BAD_REQUEST, response.status_code, response.json())
- self.assertIn("not support native events cluster", str(response.json()))
- self.assertFalse(Endpoint.objects.filter(team=self.team, name="test_query").exists())
- return
self.assertEqual(status.HTTP_201_CREATED, response.status_code, response.json())
response_data = response.json()
@@ -327,8 +319,7 @@ class TestEndpoint(ClickhouseTestMixin, APIBaseTest):
self.assertEqual(status.HTTP_201_CREATED, response.status_code, response.json())
- @parameterized.expand([(None,), (True,)])
- def test_update_endpoint(self, native: bool | None):
+ def test_update_endpoint(self):
endpoint = create_endpoint_with_version(
name="update_test",
team=self.team,
@@ -337,26 +328,16 @@ class TestEndpoint(ClickhouseTestMixin, APIBaseTest):
created_by=self.user,
)
- updated_query: dict[str, Any] = {"kind": "HogQLQuery", "query": "SELECT 1"}
- if native is not None:
- updated_query["modifiers"] = {"useNewEventsCluster": native}
-
updated_data: dict[str, Any] = {
"description": "Updated description",
"is_active": False,
- "query": updated_query,
+ "query": {"kind": "HogQLQuery", "query": "SELECT 1"},
}
response = self.client.put(
f"/api/environments/{self.team.id}/endpoints/{endpoint.name}/", updated_data, format="json"
)
- if native:
- self.assertEqual(status.HTTP_400_BAD_REQUEST, response.status_code, response.json())
- self.assertIn("not support native events cluster", str(response.json()))
- self.assertEqual(endpoint.versions.count(), 1)
- return
-
response_data = response.json()
self.assertEqual(status.HTTP_200_OK, response.status_code, response_data)
diff --git a/products/endpoints/backend/tests/test_endpoint_query_tagging.py b/products/endpoints/backend/tests/test_endpoint_query_tagging.py
index 0a7218d35017..893d8392f3c8 100644
--- a/products/endpoints/backend/tests/test_endpoint_query_tagging.py
+++ b/products/endpoints/backend/tests/test_endpoint_query_tagging.py
@@ -1,11 +1,9 @@
from posthog.test.base import APIBaseTest, ClickhouseTestMixin
from unittest import mock
-from parameterized import parameterized
from rest_framework import status
from posthog.clickhouse.query_tagging import Feature, Product, tag_queries
-from posthog.models.instance_setting import override_instance_config
from posthog.models.personal_api_key import PersonalAPIKey
from posthog.models.utils import generate_random_token_personal, hash_key_value
@@ -44,33 +42,9 @@ class TestEndpointQueryTagging(ClickhouseTestMixin, APIBaseTest):
scopes=["*"],
)
- @parameterized.expand([(None, False), (True, False), (True, True)])
- def test_run_via_session_auth_tags_endpoint_playground(self, native: bool | None, warm_cache: bool):
- version = self.endpoint.get_version()
- assert version is not None
- if native is not None:
- version.query["modifiers"] = {"useNewEventsCluster": native}
- version.save()
- with (
- override_instance_config("CLICKHOUSE_NATIVE_EVENTS_HOST", "native.example.com"),
- override_instance_config("CLICKHOUSE_NATIVE_EVENTS_TEAMS", [self.team.pk]),
- mock.patch("posthog.hogql.query.sync_execute", return_value=([(1,)], [("1", "UInt8")])) as execute,
- mock.patch("products.endpoints.backend.logic.execution.tag_queries", wraps=tag_queries) as spy,
- ):
- if warm_cache:
- response = self.client.post(
- f"/api/projects/{self.team.id}/query/",
- {"query": version.query, "refresh": "force_blocking", "async": False},
- format="json",
- )
- assert response.status_code == status.HTTP_200_OK, response.json()
- execute.reset_mock()
+ def test_run_via_session_auth_tags_endpoint_playground(self):
+ with mock.patch("products.endpoints.backend.logic.execution.tag_queries", wraps=tag_queries) as spy:
response = self.client.post(self.run_url, {}, format="json")
- if native:
- assert response.status_code == status.HTTP_400_BAD_REQUEST, response.json()
- assert "not support native events cluster" in str(response.json())
- execute.assert_not_called()
- return
assert response.status_code == status.HTTP_200_OK, response.json()
call = _find_endpoint_feature_call(spy)
assert call is not None, f"no tag_queries(product=ENDPOINTS, feature=...) call found: {spy.call_args_list}"
diff --git a/products/marketing_analytics/backend/hogql_queries/marketing_lazy_precompute.py b/products/marketing_analytics/backend/hogql_queries/marketing_lazy_precompute.py
index c57c41bc4ef2..0049e0fe6104 100644
--- a/products/marketing_analytics/backend/hogql_queries/marketing_lazy_precompute.py
+++ b/products/marketing_analytics/backend/hogql_queries/marketing_lazy_precompute.py
@@ -27,9 +27,12 @@ from typing import Any
import structlog
from prometheus_client import Counter
+from posthog.schema import HogQLQueryModifiers
+
from posthog import redis
from posthog.clickhouse.query_tagging import get_query_tag_value, tag_queries
from posthog.models import Team
+from posthog.models.event.new_events_schema import events_route_cache_modifiers, use_new_events_schema
from posthog.token_bucket import BucketDecision, Budget, consume
from products.analytics_platform.backend.lazy_computation.lazy_computation_executor import (
@@ -125,8 +128,13 @@ def marketing_ensure_precomputed(*, team: Team, **kwargs: Any) -> LazyComputatio
def _query_shape_key(query: Any) -> str:
+ query_json = query.model_dump(mode="json", exclude_none=True, exclude=set(FINAL_READ_ONLY_FIELDS))
+ if "modifiers" in query_json:
+ query_json["modifiers"] = events_route_cache_modifiers(query_json["modifiers"], reads_events=True)
+ if not query_json["modifiers"]:
+ query_json.pop("modifiers")
payload = json.dumps(
- query.model_dump(mode="json", exclude_none=True, exclude=set(FINAL_READ_ONLY_FIELDS)),
+ query_json,
sort_keys=True,
default=str,
)
@@ -150,6 +158,17 @@ def _scope_to_revalidation(query: Any) -> Any:
return scoped
+def _revalidation_query(*, team: Team, query: Any) -> Any:
+ """The query the revalidation task rebuilds, which also keys its debounce slot.
+
+ The events table the stale read resolved is pinned, so the task, which runs without the reader's cookie, rebuilds
+ from the same table, and each table gets its own slot.
+ """
+ modifiers = (query.modifiers or HogQLQueryModifiers()).model_copy()
+ modifiers.useNewEventsSchema = use_new_events_schema(team.pk, modifiers)
+ return _scope_to_revalidation(query.model_copy(update={"modifiers": modifiers}))
+
+
def is_on_demand_revalidation() -> bool:
"""True inside the Celery revalidation task, which a user read can trigger.
@@ -171,7 +190,7 @@ def enqueue_stale_revalidation(*, team: Team, query: Any) -> None:
revalidate_marketing_analytics_precompute,
)
- query = _scope_to_revalidation(query)
+ query = _revalidation_query(team=team, query=query)
claimed = False
debounce_key: str | None = None
try:
diff --git a/products/marketing_analytics/backend/hogql_queries/test_marketing_cost_trends_query_resolver.py b/products/marketing_analytics/backend/hogql_queries/test_marketing_cost_trends_query_resolver.py
index 4120c78304bb..d568c2978294 100644
--- a/products/marketing_analytics/backend/hogql_queries/test_marketing_cost_trends_query_resolver.py
+++ b/products/marketing_analytics/backend/hogql_queries/test_marketing_cost_trends_query_resolver.py
@@ -256,14 +256,20 @@ class TestMarketingCostTrendsQueryResolver(SimpleTestCase):
self.ensure.side_effect = ensure_precomputed
base_module = "products.marketing_analytics.backend.hogql_queries.marketing_analytics_base_query_runner"
with (
+ # A pinned events table keeps the warmer's runner from asking the instance settings; this test has no database.
patch(
"posthog.hogql_queries.query_runner.create_default_modifiers_for_team",
- return_value=HogQLQueryModifiers(),
+ return_value=HogQLQueryModifiers(useNewEventsSchema=False),
),
patch.object(MarketingSourceFactory, "__init__", initialize_factory),
patch.object(MarketingSourceFactory, "create_adapters", create_adapters),
patch(f"{base_module}.MarketingSourceFactory", MarketingSourceFactory),
patch(f"{base_module}.marketing_ensure_precomputed", side_effect=ensure_precomputed),
+ # The job identity asks the instance settings which events table a build reads; this test has no database.
+ patch(
+ "products.analytics_platform.backend.lazy_computation.lazy_computation_executor.use_new_events_schema",
+ return_value=False,
+ ),
patch.object(
LazyComputationExecutor, "execute", return_value=LazyComputationResult(ready=True, job_ids=[])
) as execute,
diff --git a/products/marketing_analytics/backend/hogql_queries/test_marketing_lazy_precompute.py b/products/marketing_analytics/backend/hogql_queries/test_marketing_lazy_precompute.py
index 6d6dbbd52d2d..b82748e1f520 100644
--- a/products/marketing_analytics/backend/hogql_queries/test_marketing_lazy_precompute.py
+++ b/products/marketing_analytics/backend/hogql_queries/test_marketing_lazy_precompute.py
@@ -1,12 +1,18 @@
from posthog.test.base import BaseTest
from unittest import mock
+from django.test import SimpleTestCase
+
from parameterized import parameterized
-from posthog.schema import CompareFilter, DateRange, MarketingAnalyticsTableQuery
+from posthog.schema import CompareFilter, DateRange, HogQLQueryModifiers, MarketingAnalyticsTableQuery
from posthog import redis
from posthog.clickhouse.query_tagging import Feature, get_query_tag_value, reset_query_tags, tags_context
+from posthog.models import Team
+from posthog.models.event.new_events_schema import events_route_scope, new_events_schema_for_request
+from posthog.redis import TEST_clear_clients
+from posthog.token_bucket import TEST_reset_scripts
from products.analytics_platform.backend.lazy_computation.lazy_computation_executor import LazyComputationResult
from products.marketing_analytics.backend.hogql_queries.marketing_lazy_precompute import (
@@ -15,6 +21,7 @@ from products.marketing_analytics.backend.hogql_queries.marketing_lazy_precomput
REVALIDATION_TRIGGER,
TEAM_WARM_BUDGET,
_query_shape_key,
+ _revalidation_query,
handle_not_ready,
handle_stale_served,
marketing_ensure_precomputed,
@@ -27,11 +34,33 @@ _DELAY = (
)
+class TestMarketingRevalidationEventsTable(SimpleTestCase):
+ def setUp(self) -> None:
+ TEST_clear_clients()
+ TEST_reset_scripts()
+ self.addCleanup(TEST_clear_clients)
+ self.addCleanup(TEST_reset_scripts)
+
+ def test_rebuilds_keep_the_events_table_and_debounce_each_table_separately(self) -> None:
+ team = Team(id=44, timezone="UTC")
+ query = MarketingAnalyticsTableQuery(dateRange=DateRange(date_from="-7d"), properties=[])
+ with mock.patch(_DELAY) as delay, new_events_schema_for_request():
+ for native in (False, True, False, True):
+ with events_route_scope(HogQLQueryModifiers(useNewEventsSchema=native)):
+ handle_not_ready(team=team, query=query)
+
+ assert [call.kwargs["query"]["modifiers"]["useNewEventsSchema"] for call in delay.call_args_list] == [
+ False,
+ True,
+ ]
+ assert query.modifiers is None
+
+
class TestMarketingLazyPrecompute(BaseTest):
def setUp(self):
super().setUp()
self.query = MarketingAnalyticsTableQuery(dateRange=DateRange(date_from="-7d"), properties=[])
- redis.get_client().delete(f"ma_swr_reval:{self.team.id}:{_query_shape_key(self.query)}")
+ redis.get_client().delete(self._debounce_key(self.query))
redis.get_client().delete(f"ma_swr_reval_rate:{self.team.id}")
reset_query_tags()
@@ -39,6 +68,9 @@ class TestMarketingLazyPrecompute(BaseTest):
reset_query_tags()
super().tearDown()
+ def _debounce_key(self, query: MarketingAnalyticsTableQuery) -> str:
+ return f"ma_swr_reval:{self.team.id}:{_query_shape_key(_revalidation_query(team=self.team, query=query))}"
+
@parameterized.expand(
[
# User-facing read: precompute-only. Never build inline (run_inserts=False), and serve stale
@@ -110,7 +142,7 @@ class TestMarketingLazyPrecompute(BaseTest):
@mock.patch(_DELAY)
def test_debounce_splits_only_on_precompute_identity(self, _name, overrides, expected_enqueues, delay):
variant = self.query.model_copy(update=overrides)
- redis.get_client().delete(f"ma_swr_reval:{self.team.id}:{_query_shape_key(variant)}")
+ redis.get_client().delete(self._debounce_key(variant))
handle_stale_served(team=self.team, query=self.query)
handle_stale_served(team=self.team, query=variant)
@@ -126,11 +158,11 @@ class TestMarketingLazyPrecompute(BaseTest):
for days in range(1, TEAM_WARM_BUDGET.burst + 4)
]
for shape in shapes:
- redis.get_client().delete(f"ma_swr_reval:{self.team.id}:{_query_shape_key(shape)}")
+ redis.get_client().delete(self._debounce_key(shape))
handle_not_ready(team=self.team, query=shape)
assert delay.call_count == TEAM_WARM_BUDGET.burst
- throttled_key = f"ma_swr_reval:{self.team.id}:{_query_shape_key(shapes[-1])}"
+ throttled_key = self._debounce_key(shapes[-1])
assert 0 < redis.get_client().ttl(throttled_key) <= ENQUEUE_FAILURE_BACKOFF_SECONDS
@mock.patch(_DELAY)
@@ -148,7 +180,7 @@ class TestMarketingLazyPrecompute(BaseTest):
# The slot is claimed before the enqueue is confirmed. If the enqueue fails, holding it the full
# window would suppress every retry for 10 minutes with no rebuild in flight — release it to a
# short backoff so revalidation resumes soon after the broker recovers.
- key = f"ma_swr_reval:{self.team.id}:{_query_shape_key(self.query)}"
+ key = self._debounce_key(self.query)
handle_stale_served(team=self.team, query=self.query)
diff --git a/products/marketing_analytics/backend/tasks/lazy_precompute_revalidation.py b/products/marketing_analytics/backend/tasks/lazy_precompute_revalidation.py
index e82591dcd990..df0ff243d72c 100644
--- a/products/marketing_analytics/backend/tasks/lazy_precompute_revalidation.py
+++ b/products/marketing_analytics/backend/tasks/lazy_precompute_revalidation.py
@@ -27,6 +27,7 @@ from posthog.hogql.constants import LimitContext
from posthog.clickhouse.query_tagging import Feature, Product, tag_queries
from posthog.hogql_queries.query_runner import get_query_runner
from posthog.models import Team
+from posthog.models.event.new_events_schema import events_route_scope
from posthog.scoping_audit import skip_team_scope_audit
from posthog.tasks.utils import CeleryQueue
@@ -93,7 +94,8 @@ def revalidate_marketing_analytics_precompute(team_id: int, query: dict) -> None
# response into the per-team result cache, which a user without access to some of those warehouse
# sources could then read. Leaving the result cache alone, the next user read recomputes the
# response under its own access — now over hot precomputes — and caches it correctly.
- runner.to_query()
+ with events_route_scope(runner.modifiers):
+ runner.to_query()
except Exception:
logger.exception("marketing_analytics_swr_revalidation_failed", team_id=team_id, query_kind=query_kind)
REVALIDATION_RUN.labels(outcome="failed", query_kind=query_kind).inc()
diff --git a/products/marketing_analytics/backend/tasks/test_lazy_precompute_revalidation.py b/products/marketing_analytics/backend/tasks/test_lazy_precompute_revalidation.py
index 013f5e37e662..ff40c4dc7750 100644
--- a/products/marketing_analytics/backend/tasks/test_lazy_precompute_revalidation.py
+++ b/products/marketing_analytics/backend/tasks/test_lazy_precompute_revalidation.py
@@ -1,9 +1,10 @@
from posthog.test.base import BaseTest
from unittest import mock
-from posthog.schema import DateRange, MarketingAnalyticsTableQuery
+from posthog.schema import DateRange, HogQLQueryModifiers, MarketingAnalyticsTableQuery
from posthog.clickhouse.query_tagging import Feature, get_query_tag_value, reset_query_tags
+from posthog.models.event.new_events_schema import use_new_events_schema
from products.analytics_platform.backend.lazy_computation.stale_policy import is_background_warming_request
from products.marketing_analytics.backend.hogql_queries.marketing_lazy_precompute import BACKGROUND_WARMING_TRIGGERS
@@ -36,6 +37,9 @@ class TestMarketingLazyPrecomputeRevalidation(BaseTest):
# a warehouse-restricted user could then read.
tags_at_build: dict[str, object] = {}
runner = mock.MagicMock()
+ runner.modifiers = HogQLQueryModifiers(useNewEventsSchema=True)
+ build_schemas: list[bool] = []
+ runner.to_query.side_effect = lambda: build_schemas.append(use_new_events_schema(self.team.pk))
def capture(**_kwargs):
tags_at_build["refresher"] = is_background_warming_request(BACKGROUND_WARMING_TRIGGERS)
@@ -50,6 +54,7 @@ class TestMarketingLazyPrecomputeRevalidation(BaseTest):
assert tags_at_build == {"refresher": True, "feature": Feature.PREAGGREGATION}
runner.to_query.assert_called_once_with()
runner.run.assert_not_called()
+ assert build_schemas == [True]
@mock.patch(f"{_MODULE}.get_query_runner")
def test_deleted_team_does_not_raise_into_the_worker(self, get_runner):
diff --git a/products/marketing_analytics/frontend/generated/api.schemas.ts b/products/marketing_analytics/frontend/generated/api.schemas.ts
index 7f77fd8f6cf0..66d2dea31fab 100644
--- a/products/marketing_analytics/frontend/generated/api.schemas.ts
+++ b/products/marketing_analytics/frontend/generated/api.schemas.ts
@@ -1748,9 +1748,7 @@ export interface HogQLQueryModifiersApi {
/** Remove provably redundant casts and nullability wrappers (e.g. `toString(String)`, `assumeNotNull(non_nullable)`, dead `ifNull` fallbacks) using inferred expression types */
typeAwareCastSimplification?: boolean | null
useMaterializedViews?: boolean | null
- /** Read this query from the native events cluster and native JSON events table (`true`), or keep its existing connection (`false` or unset). This is an internal rollout switch for eligible projects. PostHog staff manage eligibility through instance settings. Each query must opt in; project defaults cannot enable it. Only synchronous HogQLQuery SQL comparisons are supported. */
- useNewEventsCluster?: boolean | null
- /** Read events from the native JSON events table (`true`) or the legacy events table (`false`). When unset, the project's stored value applies, then the `CLICKHOUSE_HOGQL_USE_NEW_EVENTS_SCHEMA` instance settings. This is an internal rollout switch. PostHog staff set the project value in Django admin and the project settings API ignores it. */
+ /** Read events from the native JSON events table (`true`) or the legacy events table (`false`). When unset, a staff person's `ph_new_events_schema` cookie applies, then the project's listing in the `CLICKHOUSE_HOGQL_USE_NEW_EVENTS_SCHEMA_TEAMS` instance setting, then the `CLICKHOUSE_HOGQL_USE_NEW_EVENTS_SCHEMA` instance setting. This is an internal rollout switch, and the project settings API ignores it. */
useNewEventsSchema?: boolean | null
usePreaggregatedIntermediateResults?: boolean | null
/** Try to automatically convert HogQL queries to use preaggregated tables at the AST level * */
diff --git a/products/persons/frontend/generated/api.schemas.ts b/products/persons/frontend/generated/api.schemas.ts
index 2b3864ddec4b..1ac403ec39a2 100644
--- a/products/persons/frontend/generated/api.schemas.ts
+++ b/products/persons/frontend/generated/api.schemas.ts
@@ -566,9 +566,7 @@ export interface HogQLQueryModifiersApi {
/** Remove provably redundant casts and nullability wrappers (e.g. `toString(String)`, `assumeNotNull(non_nullable)`, dead `ifNull` fallbacks) using inferred expression types */
typeAwareCastSimplification?: boolean | null
useMaterializedViews?: boolean | null
- /** Read this query from the native events cluster and native JSON events table (`true`), or keep its existing connection (`false` or unset). This is an internal rollout switch for eligible projects. PostHog staff manage eligibility through instance settings. Each query must opt in; project defaults cannot enable it. Only synchronous HogQLQuery SQL comparisons are supported. */
- useNewEventsCluster?: boolean | null
- /** Read events from the native JSON events table (`true`) or the legacy events table (`false`). When unset, the project's stored value applies, then the `CLICKHOUSE_HOGQL_USE_NEW_EVENTS_SCHEMA` instance settings. This is an internal rollout switch. PostHog staff set the project value in Django admin and the project settings API ignores it. */
+ /** Read events from the native JSON events table (`true`) or the legacy events table (`false`). When unset, a staff person's `ph_new_events_schema` cookie applies, then the project's listing in the `CLICKHOUSE_HOGQL_USE_NEW_EVENTS_SCHEMA_TEAMS` instance setting, then the `CLICKHOUSE_HOGQL_USE_NEW_EVENTS_SCHEMA` instance setting. This is an internal rollout switch, and the project settings API ignores it. */
useNewEventsSchema?: boolean | null
usePreaggregatedIntermediateResults?: boolean | null
/** Try to automatically convert HogQL queries to use preaggregated tables at the AST level * */
diff --git a/products/product_analytics/backend/hogql_queries/funnels/test/test_funnel_insight_actors.py b/products/product_analytics/backend/hogql_queries/funnels/test/test_funnel_insight_actors.py
index a89004260a6f..c93889f42fa6 100644
--- a/products/product_analytics/backend/hogql_queries/funnels/test/test_funnel_insight_actors.py
+++ b/products/product_analytics/backend/hogql_queries/funnels/test/test_funnel_insight_actors.py
@@ -20,7 +20,6 @@ from posthog.schema import (
EventsNode,
EventsQuery,
FunnelsActorsQuery,
- FunnelsFilter,
FunnelsQuery,
HogQLQueryModifiers,
InsightActorsQuery,
@@ -31,8 +30,6 @@ from posthog.schema import (
from posthog.hogql.context import HogQLContext
from posthog.hogql.query import execute_hogql_query
-from posthog.clickhouse.workload import Workload
-from posthog.hogql_queries.actor_strategies import SessionStrategy
from posthog.hogql_queries.actors_query_runner import ActorsQueryRunner
from posthog.hogql_queries.events_query_runner import EventsQueryRunner
from posthog.hogql_queries.insight_actors_query_options_runner import InsightActorsQueryOptionsRunner
@@ -40,7 +37,6 @@ from posthog.hogql_queries.insight_actors_query_runner import InsightActorsQuery
from posthog.hogql_queries.query_runner import QueryRunner, get_query_runner, get_query_runner_or_none
from posthog.models import Organization, Team, User
from posthog.models.group.util import create_group
-from posthog.models.instance_setting import override_instance_config
from posthog.test.test_utils import create_group_type_mapping_without_created_at
from products.product_analytics.backend.hogql_queries.funnels.funnels_query_runner import FunnelsQueryRunner
@@ -165,21 +161,13 @@ class TestFunnelInsightActors(ClickhouseTestMixin, APIBaseTest):
)
-@override_settings(CLICKHOUSE_HOGQL_USE_NEW_EVENTS_SCHEMA=False, CLICKHOUSE_NATIVE_EVENTS_HOST="native.example.com")
+@override_settings(CLICKHOUSE_HOGQL_USE_NEW_EVENTS_SCHEMA=False)
class TestNativeEventsActorRouting(SimpleTestCase):
def setUp(self) -> None:
self.team = Team(id=44, modifiers={}, organization=Organization(id=1))
self.user = User(id=123, is_staff=True, is_active=True)
self.modifiers = HogQLQueryModifiers(personsOnEventsMode=PersonsOnEventsMode.DISABLED)
- self.enterContext(
- patch(
- "posthog.models.event.new_events_schema.get_instance_setting",
- side_effect={
- "CLICKHOUSE_NATIVE_EVENTS_TEAMS": [44],
- "CLICKHOUSE_NATIVE_EVENTS_HOST": "native.example.com",
- }.get,
- )
- )
+ self.enterContext(patch("posthog.models.event.new_events_schema.get_instance_setting", return_value=None))
self.enterContext(
patch(
"products.web_analytics.backend.hogql_queries.cookieless_flag.resolve_cookieless_traffic_is_regular_modifier",
@@ -197,7 +185,7 @@ class TestNativeEventsActorRouting(SimpleTestCase):
def test_actor_queries_inherit_the_source_route_unless_explicitly_overridden(
self, kind: str, source_choice: bool | None, wrapper_choice: bool | None
) -> None:
- modifiers = self.modifiers.model_copy(update={"useNewEventsCluster": source_choice})
+ modifiers = self.modifiers.model_copy(update={"useNewEventsSchema": source_choice})
source = FunnelsQuery(series=[EventsNode(event="example"), EventsNode(event="finished")], modifiers=modifiers)
actors = FunnelsActorsQuery(source=source, funnelStep=1)
query: BaseModel
@@ -219,15 +207,12 @@ class TestNativeEventsActorRouting(SimpleTestCase):
)
if wrapper_choice is not None:
assert isinstance(query, (ActorsQuery, InsightActorsQuery))
- query.modifiers = self.modifiers.model_copy(
- update={"useNewEventsSchema": wrapper_choice, "useNewEventsCluster": wrapper_choice}
- )
+ query.modifiers = self.modifiers.model_copy(update={"useNewEventsSchema": wrapper_choice})
original = query.model_dump()
runner = get_query_runner(query, self.team, user=self.user)
native = source_choice if wrapper_choice is None else wrapper_choice
self.assertEqual(runner.modifiers.personsOnEventsMode, PersonsOnEventsMode.DISABLED)
self.assertIs(bool(runner.modifiers.useNewEventsSchema), bool(native))
- self.assertIs(runner.modifiers.useNewEventsCluster, native)
if kind == "EventsQuery":
self.assertIs(runner.modifiers.convertToProjectTimezone, False)
self.assertEqual(query.model_dump(), original)
@@ -239,23 +224,6 @@ class TestActorExecutionUserContext(APIBaseTest):
runner = get_query_runner(InsightActorsQuery(source=source).model_dump(), self.team, user=self.user)
self.assertIsInstance(runner.query, InsightActorsQuery)
- @override_settings(CLICKHOUSE_HOGQL_USE_NEW_EVENTS_SCHEMA=False, CLICKHOUSE_NATIVE_EVENTS_HOST="native.example.com")
- def test_session_actors_hydrate_on_the_selected_cluster(self) -> None:
- source = FunnelsQuery(
- series=[EventsNode(event="started"), EventsNode(event="finished")],
- funnelsFilter=FunnelsFilter(funnelAggregateByHogQL="properties.$session_id"),
- modifiers=HogQLQueryModifiers(useNewEventsSchema=True, useNewEventsCluster=True),
- )
- with (
- override_instance_config("CLICKHOUSE_NATIVE_EVENTS_HOST", "native.example.com"),
- override_instance_config("CLICKHOUSE_NATIVE_EVENTS_TEAMS", [self.team.pk]),
- patch("posthog.hogql.query.sync_execute", return_value=([], [])) as execute,
- ):
- runner = get_query_runner(ActorsQuery(source=FunnelsActorsQuery(source=source, funnelStep=1)), self.team)
- assert isinstance(runner, ActorsQueryRunner) and isinstance(runner.strategy, SessionStrategy)
- runner.strategy.get_actors(["a-session"])
- self.assertEqual(execute.call_args.kwargs["workload"], Workload.NEW_EVENTS_CLUSTER)
-
@parameterized.expand([("ActorsQuery",), ("InsightActorsQuery",), ("InsightActorsQueryOptions",), ("EventsQuery",)])
def test_nested_query_context_follows_execution_user(self, kind: str) -> None:
source = FunnelsQuery(series=[EventsNode(event="started"), EventsNode(event="finished")])
diff --git a/products/product_analytics/backend/hogql_queries/trends/test/test_trend_validation_rules.py b/products/product_analytics/backend/hogql_queries/trends/test/test_trend_validation_rules.py
index b2559f257124..221d7f764763 100644
--- a/products/product_analytics/backend/hogql_queries/trends/test/test_trend_validation_rules.py
+++ b/products/product_analytics/backend/hogql_queries/trends/test/test_trend_validation_rules.py
@@ -22,9 +22,7 @@ from posthog.schema import (
TrendsQuery,
)
-from posthog.clickhouse.workload import Workload
from posthog.hogql_queries.validation.validation import QueryValidationContext
-from posthog.models.instance_setting import override_instance_config
from products.product_analytics.backend.hogql_queries.trends.trend_validation_rules import (
DisallowDaysOfWeekWithSmoothing,
@@ -443,21 +441,16 @@ class TestTrendsQueryRunnerSeriesFanOut(BaseTest):
self.assertEqual(len(runner.series), 150)
@parameterized.expand([(False,), (True,)])
- @override_settings(CLICKHOUSE_HOGQL_USE_NEW_EVENTS_SCHEMA=False, CLICKHOUSE_NATIVE_EVENTS_HOST="native.example.com")
- def test_actor_options_execute_on_the_selected_cluster(self, native: bool) -> None:
+ @override_settings(CLICKHOUSE_HOGQL_USE_NEW_EVENTS_SCHEMA=False)
+ def test_actor_options_read_the_selected_events_table(self, native: bool) -> None:
query = TrendsQuery(
series=[EventsNode(event="$pageview")],
dateRange=DateRange(date_from="2024-01-01", date_to="2024-01-08"),
breakdownFilter=BreakdownFilter(breakdown="$browser", breakdown_type=BreakdownType.EVENT),
- modifiers=HogQLQueryModifiers(useNewEventsCluster=native, useNewEventsSchema=native),
+ modifiers=HogQLQueryModifiers(useNewEventsSchema=native),
)
- with (
- override_instance_config("CLICKHOUSE_NATIVE_EVENTS_HOST", "native.example.com"),
- override_instance_config("CLICKHOUSE_NATIVE_EVENTS_TEAMS", [self.team.pk]),
- patch("posthog.hogql.query.sync_execute", return_value=([], [])) as execute,
- ):
+ with patch("posthog.hogql.query.sync_execute", return_value=([], [])) as execute:
result = TrendsQueryRunner(query=query, team=self.team).to_actors_query_options()
assert result.breakdown == []
execute.assert_called_once()
- assert execute.call_args.kwargs["workload"] == (Workload.NEW_EVENTS_CLUSTER if native else Workload.DEFAULT)
assert ("events_json" in execute.call_args.args[0]) is native
diff --git a/products/product_analytics/frontend/generated/api.schemas.ts b/products/product_analytics/frontend/generated/api.schemas.ts
index 6c2c41fa6559..dbbb33bc4298 100644
--- a/products/product_analytics/frontend/generated/api.schemas.ts
+++ b/products/product_analytics/frontend/generated/api.schemas.ts
@@ -1280,9 +1280,7 @@ export interface HogQLQueryModifiersApi {
/** Remove provably redundant casts and nullability wrappers (e.g. `toString(String)`, `assumeNotNull(non_nullable)`, dead `ifNull` fallbacks) using inferred expression types */
typeAwareCastSimplification?: boolean | null
useMaterializedViews?: boolean | null
- /** Read this query from the native events cluster and native JSON events table (`true`), or keep its existing connection (`false` or unset). This is an internal rollout switch for eligible projects. PostHog staff manage eligibility through instance settings. Each query must opt in; project defaults cannot enable it. Only synchronous HogQLQuery SQL comparisons are supported. */
- useNewEventsCluster?: boolean | null
- /** Read events from the native JSON events table (`true`) or the legacy events table (`false`). When unset, the project's stored value applies, then the `CLICKHOUSE_HOGQL_USE_NEW_EVENTS_SCHEMA` instance settings. This is an internal rollout switch. PostHog staff set the project value in Django admin and the project settings API ignores it. */
+ /** Read events from the native JSON events table (`true`) or the legacy events table (`false`). When unset, a staff person's `ph_new_events_schema` cookie applies, then the project's listing in the `CLICKHOUSE_HOGQL_USE_NEW_EVENTS_SCHEMA_TEAMS` instance setting, then the `CLICKHOUSE_HOGQL_USE_NEW_EVENTS_SCHEMA` instance setting. This is an internal rollout switch, and the project settings API ignores it. */
useNewEventsSchema?: boolean | null
usePreaggregatedIntermediateResults?: boolean | null
/** Try to automatically convert HogQL queries to use preaggregated tables at the AST level * */
diff --git a/products/web_analytics/backend/hogql_queries/test/test_web_lazy_precompute_common.py b/products/web_analytics/backend/hogql_queries/test/test_web_lazy_precompute_common.py
index bde11377fc63..b228023f8592 100644
--- a/products/web_analytics/backend/hogql_queries/test/test_web_lazy_precompute_common.py
+++ b/products/web_analytics/backend/hogql_queries/test/test_web_lazy_precompute_common.py
@@ -33,6 +33,9 @@ from posthog.hogql import ast
from posthog import redis
from posthog.clickhouse.query_tagging import Feature, get_query_tag_value, reset_query_tags, tags_context
from posthog.hogql_queries.query_runner import AnalyticsQueryRunner, ExecutionMode
+from posthog.models import Team
+from posthog.models.event.new_events_schema import events_route_scope, new_events_schema_for_request
+from posthog.redis import TEST_clear_clients
from products.analytics_platform.backend.lazy_computation.lazy_computation_executor import (
LazyComputationResult,
@@ -62,6 +65,7 @@ from products.web_analytics.backend.hogql_queries.web_lazy_precompute_common imp
check_common_eligibility,
compute_filters_eligibility_hash,
compute_shape_cap_key,
+ enqueue_stale_revalidation,
get_sticky_warm_shapes,
handle_stale_served,
host_filter_expr,
@@ -1134,6 +1138,34 @@ class TestVolumeFloor(BaseTest):
assert can_use_lazy_precompute(runner, log_prefix="web_overview") is False
+class TestRevalidationEventsTable(SimpleTestCase):
+ def setUp(self) -> None:
+ TEST_clear_clients()
+ self.addCleanup(TEST_clear_clients)
+
+ def test_rebuilds_keep_the_events_table_and_debounce_each_table_separately(self) -> None:
+ team = Team(id=44, timezone="UTC")
+ query = WebOverviewQuery(dateRange=DateRange(date_from="-7d"), properties=[])
+ with (
+ mock.patch(
+ "products.web_analytics.backend.tasks.lazy_precompute_revalidation"
+ ".revalidate_web_analytics_precompute.apply_async"
+ ) as enqueue,
+ new_events_schema_for_request(),
+ ):
+ for native in (False, True, False, True):
+ with events_route_scope(HogQLQueryModifiers(useNewEventsSchema=native)):
+ enqueue_stale_revalidation(team=team, query=query, family="web_overview")
+
+ assert [
+ call.kwargs["kwargs"]["query"]["modifiers"]["useNewEventsSchema"] for call in enqueue.call_args_list
+ ] == [
+ False,
+ True,
+ ]
+ assert query.modifiers is None
+
+
class TestStaleRevalidationEnqueue(BaseTest):
def setUp(self):
super().setUp()
diff --git a/products/web_analytics/backend/hogql_queries/test/test_web_vitals_timeseries_lazy_precompute.py b/products/web_analytics/backend/hogql_queries/test/test_web_vitals_timeseries_lazy_precompute.py
index f90c7f6f94cf..d94ec68a62b5 100644
--- a/products/web_analytics/backend/hogql_queries/test/test_web_vitals_timeseries_lazy_precompute.py
+++ b/products/web_analytics/backend/hogql_queries/test/test_web_vitals_timeseries_lazy_precompute.py
@@ -31,7 +31,6 @@ from posthog.api.services.query import process_query_model
from posthog.clickhouse.client import sync_execute
from posthog.clickhouse.query_tagging import Feature, Product, reset_query_tags, tag_queries
from posthog.hogql_queries.query_runner import get_query_runner_or_none
-from posthog.models.instance_setting import override_instance_config
from products.analytics_platform.backend.models.preaggregation_job import PreaggregationJob
from products.product_analytics.backend.facade.queries import TrendsQueryRunner
@@ -78,29 +77,25 @@ def _vitals_query(**overrides: Any) -> WebVitalsQuery:
class TestVitalsTimeseriesGate(APIBaseTest):
@parameterized.expand([(path, outer) for path in ("runner", "source_fallback") for outer in (False, True)])
- @override_settings(CLICKHOUSE_HOGQL_USE_NEW_EVENTS_SCHEMA=False, CLICKHOUSE_NATIVE_EVENTS_HOST="native.example.com")
- def test_wrapper_cluster_choice_overrides_the_source(self, path: str, outer: bool) -> None:
+ @override_settings(CLICKHOUSE_HOGQL_USE_NEW_EVENTS_SCHEMA=False)
+ def test_wrapper_table_choice_overrides_the_source(self, path: str, outer: bool) -> None:
query = _vitals_query(
- modifiers=HogQLQueryModifiers(useNewEventsCluster=outer),
- source_overrides={"modifiers": HogQLQueryModifiers(useNewEventsCluster=not outer)},
+ modifiers=HogQLQueryModifiers(useNewEventsSchema=outer),
+ source_overrides={"modifiers": HogQLQueryModifiers(useNewEventsSchema=not outer)},
)
- with (
- override_instance_config("CLICKHOUSE_NATIVE_EVENTS_HOST", "native.example.com"),
- override_instance_config("CLICKHOUSE_NATIVE_EVENTS_TEAMS", [self.team.pk]),
- ):
- if path == "runner":
- runner = WebVitalsQueryRunner(query=query, team=self.team)
- else:
- with (
- patch(
- "products.web_analytics.backend.hogql_queries.web_vitals_timeseries_lazy_precompute.is_vitals_precompute_enabled_for_team",
- return_value=False,
- ),
- patch("posthog.api.services.query._run_query_runner", return_value={}) as run_source,
- ):
- process_query_model(self.team, query)
- runner = run_source.call_args.args[0]
- self.assertIs(bool(runner.modifiers.useNewEventsCluster), outer)
+ if path == "runner":
+ runner = WebVitalsQueryRunner(query=query, team=self.team)
+ else:
+ with (
+ patch(
+ "products.web_analytics.backend.hogql_queries.web_vitals_timeseries_lazy_precompute.is_vitals_precompute_enabled_for_team",
+ return_value=False,
+ ),
+ patch("posthog.api.services.query._run_query_runner", return_value={}) as run_source,
+ ):
+ process_query_model(self.team, query)
+ runner = run_source.call_args.args[0]
+ self.assertIs(bool(runner.modifiers.useNewEventsSchema), outer)
def test_canonical_tab_shape_is_servable(self) -> None:
assert vitals_timeseries_percentile(_vitals_query()) == "p90"
diff --git a/products/web_analytics/backend/hogql_queries/web_lazy_precompute_common.py b/products/web_analytics/backend/hogql_queries/web_lazy_precompute_common.py
index b5e7d3c8564a..9d733e662272 100644
--- a/products/web_analytics/backend/hogql_queries/web_lazy_precompute_common.py
+++ b/products/web_analytics/backend/hogql_queries/web_lazy_precompute_common.py
@@ -25,7 +25,7 @@ import structlog
import posthoganalytics
from prometheus_client import Counter
-from posthog.schema import SessionsV2JoinMode, WebAnalyticsPreComputeStrategy
+from posthog.schema import HogQLQueryModifiers, SessionsV2JoinMode, WebAnalyticsPreComputeStrategy
from posthog.hogql import ast
from posthog.hogql.modifiers import create_default_modifiers_for_team
@@ -36,6 +36,7 @@ from posthog import redis
from posthog.clickhouse.query_tagging import clear_tag, get_query_tag_value, tag_queries
from posthog.hogql_queries.query_runner import ExecutionMode
from posthog.models import Team
+from posthog.models.event.new_events_schema import use_new_events_schema
from products.access_control.backend.facade.api import team_has_property_access_rules
from products.analytics_platform.backend.lazy_computation.lazy_computation_executor import (
@@ -528,12 +529,17 @@ def enqueue_stale_revalidation(*, team: Team, query: Any, family: str, debounce_
revalidate_web_analytics_precompute,
)
+ modifiers = (query.modifiers or HogQLQueryModifiers()).model_copy()
+ modifiers.useNewEventsSchema = use_new_events_schema(team.pk, modifiers)
+ query = query.model_copy(update={"modifiers": modifiers})
try:
client = redis.get_client()
shape_part = compute_filters_eligibility_hash(query, team.timezone)[:16]
if debounce_extra:
shape_part = hashlib.sha256(f"{shape_part}:{debounce_extra}".encode()).hexdigest()[:16]
debounce_key = f"web_swr_reval:{team.id}:{family}:{shape_part}"
+ if modifiers.useNewEventsSchema:
+ debounce_key += ":events_json"
if not client.set(debounce_key, "1", ex=REVALIDATION_DEBOUNCE_SECONDS, nx=True):
return
budget_key = f"web_swr_reval_budget:{team.id}"
diff --git a/services/mcp/src/api/generated.ts b/services/mcp/src/api/generated.ts
index 011d2f67ca46..170ec6ffe9de 100644
--- a/services/mcp/src/api/generated.ts
+++ b/services/mcp/src/api/generated.ts
@@ -2808,9 +2808,7 @@ export namespace Schemas {
/** Remove provably redundant casts and nullability wrappers (e.g. `toString(String)`, `assumeNotNull(non_nullable)`, dead `ifNull` fallbacks) using inferred expression types */
typeAwareCastSimplification?: boolean | null;
useMaterializedViews?: boolean | null;
- /** Read this query from the native events cluster and native JSON events table (`true`), or keep its existing connection (`false` or unset). This is an internal rollout switch for eligible projects. PostHog staff manage eligibility through instance settings. Each query must opt in; project defaults cannot enable it. Only synchronous HogQLQuery SQL comparisons are supported. */
- useNewEventsCluster?: boolean | null;
- /** Read events from the native JSON events table (`true`) or the legacy events table (`false`). When unset, the project's stored value applies, then the `CLICKHOUSE_HOGQL_USE_NEW_EVENTS_SCHEMA` instance settings. This is an internal rollout switch. PostHog staff set the project value in Django admin and the project settings API ignores it. */
+ /** Read events from the native JSON events table (`true`) or the legacy events table (`false`). When unset, a staff person's `ph_new_events_schema` cookie applies, then the project's listing in the `CLICKHOUSE_HOGQL_USE_NEW_EVENTS_SCHEMA_TEAMS` instance setting, then the `CLICKHOUSE_HOGQL_USE_NEW_EVENTS_SCHEMA` instance setting. This is an internal rollout switch, and the project settings API ignores it. */
useNewEventsSchema?: boolean | null;
usePreaggregatedIntermediateResults?: boolean | null;
/** Try to automatically convert HogQL queries to use preaggregated tables at the AST level * */
diff --git a/services/mcp/src/generated/batch_exports/api.ts b/services/mcp/src/generated/batch_exports/api.ts
index 6afad9096d0c..6a9e0094ce86 100644
--- a/services/mcp/src/generated/batch_exports/api.ts
+++ b/services/mcp/src/generated/batch_exports/api.ts
@@ -681,17 +681,11 @@ export const BatchExportsCreateBody = () => zod
'Remove provably redundant casts and nullability wrappers (e.g. `toString(String)`, `assumeNotNull(non_nullable)`, dead `ifNull` fallbacks) using inferred expression types'
),
useMaterializedViews: zod.union([zod.boolean(), zod.null()]).optional(),
- useNewEventsCluster: zod
- .union([zod.boolean(), zod.null()])
- .optional()
- .describe(
- 'Read this query from the native events cluster and native JSON events table (`true`), or keep its existing connection (`false` or unset). This is an internal rollout switch for eligible projects. PostHog staff manage eligibility through instance settings. Each query must opt in; project defaults cannot enable it. Only synchronous HogQLQuery SQL comparisons are supported.'
- ),
useNewEventsSchema: zod
.union([zod.boolean(), zod.null()])
.optional()
.describe(
- "Read events from the native JSON events table (`true`) or the legacy events table (`false`). When unset, the project's stored value applies, then the `CLICKHOUSE_HOGQL_USE_NEW_EVENTS_SCHEMA` instance settings. This is an internal rollout switch. PostHog staff set the project value in Django admin and the project settings API ignores it."
+ "Read events from the native JSON events table (`true`) or the legacy events table (`false`). When unset, a staff person's `ph_new_events_schema` cookie applies, then the project's listing in the `CLICKHOUSE_HOGQL_USE_NEW_EVENTS_SCHEMA_TEAMS` instance setting, then the `CLICKHOUSE_HOGQL_USE_NEW_EVENTS_SCHEMA` instance setting. This is an internal rollout switch, and the project settings API ignores it."
),
usePreaggregatedIntermediateResults: zod.union([zod.boolean(), zod.null()]).optional(),
usePreaggregatedTableTransforms: zod
@@ -1430,17 +1424,11 @@ export const BatchExportsPartialUpdateBody = () => zod
'Remove provably redundant casts and nullability wrappers (e.g. `toString(String)`, `assumeNotNull(non_nullable)`, dead `ifNull` fallbacks) using inferred expression types'
),
useMaterializedViews: zod.union([zod.boolean(), zod.null()]).optional(),
- useNewEventsCluster: zod
- .union([zod.boolean(), zod.null()])
- .optional()
- .describe(
- 'Read this query from the native events cluster and native JSON events table (`true`), or keep its existing connection (`false` or unset). This is an internal rollout switch for eligible projects. PostHog staff manage eligibility through instance settings. Each query must opt in; project defaults cannot enable it. Only synchronous HogQLQuery SQL comparisons are supported.'
- ),
useNewEventsSchema: zod
.union([zod.boolean(), zod.null()])
.optional()
.describe(
- "Read events from the native JSON events table (`true`) or the legacy events table (`false`). When unset, the project's stored value applies, then the `CLICKHOUSE_HOGQL_USE_NEW_EVENTS_SCHEMA` instance settings. This is an internal rollout switch. PostHog staff set the project value in Django admin and the project settings API ignores it."
+ "Read events from the native JSON events table (`true`) or the legacy events table (`false`). When unset, a staff person's `ph_new_events_schema` cookie applies, then the project's listing in the `CLICKHOUSE_HOGQL_USE_NEW_EVENTS_SCHEMA_TEAMS` instance setting, then the `CLICKHOUSE_HOGQL_USE_NEW_EVENTS_SCHEMA` instance setting. This is an internal rollout switch, and the project settings API ignores it."
),
usePreaggregatedIntermediateResults: zod.union([zod.boolean(), zod.null()]).optional(),
usePreaggregatedTableTransforms: zod
@@ -1910,17 +1898,11 @@ export const FileDownloadBatchExportsCreateBody = () => zod.union([
'Remove provably redundant casts and nullability wrappers (e.g. `toString(String)`, `assumeNotNull(non_nullable)`, dead `ifNull` fallbacks) using inferred expression types'
),
useMaterializedViews: zod.union([zod.boolean(), zod.null()]).optional(),
- useNewEventsCluster: zod
- .union([zod.boolean(), zod.null()])
- .optional()
- .describe(
- 'Read this query from the native events cluster and native JSON events table (`true`), or keep its existing connection (`false` or unset). This is an internal rollout switch for eligible projects. PostHog staff manage eligibility through instance settings. Each query must opt in; project defaults cannot enable it. Only synchronous HogQLQuery SQL comparisons are supported.'
- ),
useNewEventsSchema: zod
.union([zod.boolean(), zod.null()])
.optional()
.describe(
- "Read events from the native JSON events table (`true`) or the legacy events table (`false`). When unset, the project's stored value applies, then the `CLICKHOUSE_HOGQL_USE_NEW_EVENTS_SCHEMA` instance settings. This is an internal rollout switch. PostHog staff set the project value in Django admin and the project settings API ignores it."
+ "Read events from the native JSON events table (`true`) or the legacy events table (`false`). When unset, a staff person's `ph_new_events_schema` cookie applies, then the project's listing in the `CLICKHOUSE_HOGQL_USE_NEW_EVENTS_SCHEMA_TEAMS` instance setting, then the `CLICKHOUSE_HOGQL_USE_NEW_EVENTS_SCHEMA` instance setting. This is an internal rollout switch, and the project settings API ignores it."
),
usePreaggregatedIntermediateResults: zod.union([zod.boolean(), zod.null()]).optional(),
usePreaggregatedTableTransforms: zod
@@ -2216,17 +2198,11 @@ export const FileDownloadBatchExportsCountRowsCreateBody = () => zod
'Remove provably redundant casts and nullability wrappers (e.g. `toString(String)`, `assumeNotNull(non_nullable)`, dead `ifNull` fallbacks) using inferred expression types'
),
useMaterializedViews: zod.union([zod.boolean(), zod.null()]).optional(),
- useNewEventsCluster: zod
- .union([zod.boolean(), zod.null()])
- .optional()
- .describe(
- 'Read this query from the native events cluster and native JSON events table (`true`), or keep its existing connection (`false` or unset). This is an internal rollout switch for eligible projects. PostHog staff manage eligibility through instance settings. Each query must opt in; project defaults cannot enable it. Only synchronous HogQLQuery SQL comparisons are supported.'
- ),
useNewEventsSchema: zod
.union([zod.boolean(), zod.null()])
.optional()
.describe(
- "Read events from the native JSON events table (`true`) or the legacy events table (`false`). When unset, the project's stored value applies, then the `CLICKHOUSE_HOGQL_USE_NEW_EVENTS_SCHEMA` instance settings. This is an internal rollout switch. PostHog staff set the project value in Django admin and the project settings API ignores it."
+ "Read events from the native JSON events table (`true`) or the legacy events table (`false`). When unset, a staff person's `ph_new_events_schema` cookie applies, then the project's listing in the `CLICKHOUSE_HOGQL_USE_NEW_EVENTS_SCHEMA_TEAMS` instance setting, then the `CLICKHOUSE_HOGQL_USE_NEW_EVENTS_SCHEMA` instance setting. This is an internal rollout switch, and the project settings API ignores it."
),
usePreaggregatedIntermediateResults: zod.union([zod.boolean(), zod.null()]).optional(),
usePreaggregatedTableTransforms: zod
diff --git a/services/mcp/tests/unit/__snapshots__/tool-schemas/batch-export-create.json b/services/mcp/tests/unit/__snapshots__/tool-schemas/batch-export-create.json
index 5d27745681e1..b30e4d3790c1 100644
--- a/services/mcp/tests/unit/__snapshots__/tool-schemas/batch-export-create.json
+++ b/services/mcp/tests/unit/__snapshots__/tool-schemas/batch-export-create.json
@@ -1076,17 +1076,6 @@
}
]
},
- "useNewEventsCluster": {
- "anyOf": [
- {
- "type": "boolean"
- },
- {
- "type": "null"
- }
- ],
- "description": "Read this query from the native events cluster and native JSON events table (`true`), or keep its existing connection (`false` or unset). This is an internal rollout switch for eligible projects. PostHog staff manage eligibility through instance settings. Each query must opt in; project defaults cannot enable it. Only synchronous HogQLQuery SQL comparisons are supported."
- },
"useNewEventsSchema": {
"anyOf": [
{
@@ -1096,7 +1085,7 @@
"type": "null"
}
],
- "description": "Read events from the native JSON events table (`true`) or the legacy events table (`false`). When unset, the project's stored value applies, then the `CLICKHOUSE_HOGQL_USE_NEW_EVENTS_SCHEMA` instance settings. This is an internal rollout switch. PostHog staff set the project value in Django admin and the project settings API ignores it."
+ "description": "Read events from the native JSON events table (`true`) or the legacy events table (`false`). When unset, a staff person's `ph_new_events_schema` cookie applies, then the project's listing in the `CLICKHOUSE_HOGQL_USE_NEW_EVENTS_SCHEMA_TEAMS` instance setting, then the `CLICKHOUSE_HOGQL_USE_NEW_EVENTS_SCHEMA` instance setting. This is an internal rollout switch, and the project settings API ignores it."
},
"usePreaggregatedIntermediateResults": {
"anyOf": [
diff --git a/services/mcp/tests/unit/__snapshots__/tool-schemas/batch-export-update.json b/services/mcp/tests/unit/__snapshots__/tool-schemas/batch-export-update.json
index e8014b818963..92e3c09c1c56 100644
--- a/services/mcp/tests/unit/__snapshots__/tool-schemas/batch-export-update.json
+++ b/services/mcp/tests/unit/__snapshots__/tool-schemas/batch-export-update.json
@@ -1075,17 +1075,6 @@
}
]
},
- "useNewEventsCluster": {
- "anyOf": [
- {
- "type": "boolean"
- },
- {
- "type": "null"
- }
- ],
- "description": "Read this query from the native events cluster and native JSON events table (`true`), or keep its existing connection (`false` or unset). This is an internal rollout switch for eligible projects. PostHog staff manage eligibility through instance settings. Each query must opt in; project defaults cannot enable it. Only synchronous HogQLQuery SQL comparisons are supported."
- },
"useNewEventsSchema": {
"anyOf": [
{
@@ -1095,7 +1084,7 @@
"type": "null"
}
],
- "description": "Read events from the native JSON events table (`true`) or the legacy events table (`false`). When unset, the project's stored value applies, then the `CLICKHOUSE_HOGQL_USE_NEW_EVENTS_SCHEMA` instance settings. This is an internal rollout switch. PostHog staff set the project value in Django admin and the project settings API ignores it."
+ "description": "Read events from the native JSON events table (`true`) or the legacy events table (`false`). When unset, a staff person's `ph_new_events_schema` cookie applies, then the project's listing in the `CLICKHOUSE_HOGQL_USE_NEW_EVENTS_SCHEMA_TEAMS` instance setting, then the `CLICKHOUSE_HOGQL_USE_NEW_EVENTS_SCHEMA` instance setting. This is an internal rollout switch, and the project settings API ignores it."
},
"usePreaggregatedIntermediateResults": {
"anyOf": [
diff --git a/services/mcp/tests/unit/__snapshots__/tool-schemas/file-download-batch-exports-count-rows-create.json b/services/mcp/tests/unit/__snapshots__/tool-schemas/file-download-batch-exports-count-rows-create.json
index 778b5c021315..442048d51b07 100644
--- a/services/mcp/tests/unit/__snapshots__/tool-schemas/file-download-batch-exports-count-rows-create.json
+++ b/services/mcp/tests/unit/__snapshots__/tool-schemas/file-download-batch-exports-count-rows-create.json
@@ -551,17 +551,6 @@
}
]
},
- "useNewEventsCluster": {
- "anyOf": [
- {
- "type": "boolean"
- },
- {
- "type": "null"
- }
- ],
- "description": "Read this query from the native events cluster and native JSON events table (`true`), or keep its existing connection (`false` or unset). This is an internal rollout switch for eligible projects. PostHog staff manage eligibility through instance settings. Each query must opt in; project defaults cannot enable it. Only synchronous HogQLQuery SQL comparisons are supported."
- },
"useNewEventsSchema": {
"anyOf": [
{
@@ -571,7 +560,7 @@
"type": "null"
}
],
- "description": "Read events from the native JSON events table (`true`) or the legacy events table (`false`). When unset, the project's stored value applies, then the `CLICKHOUSE_HOGQL_USE_NEW_EVENTS_SCHEMA` instance settings. This is an internal rollout switch. PostHog staff set the project value in Django admin and the project settings API ignores it."
+ "description": "Read events from the native JSON events table (`true`) or the legacy events table (`false`). When unset, a staff person's `ph_new_events_schema` cookie applies, then the project's listing in the `CLICKHOUSE_HOGQL_USE_NEW_EVENTS_SCHEMA_TEAMS` instance setting, then the `CLICKHOUSE_HOGQL_USE_NEW_EVENTS_SCHEMA` instance setting. This is an internal rollout switch, and the project settings API ignores it."
},
"usePreaggregatedIntermediateResults": {
"anyOf": [
diff --git a/services/mcp/tests/unit/__snapshots__/tool-schemas/file-download-batch-exports-create.json b/services/mcp/tests/unit/__snapshots__/tool-schemas/file-download-batch-exports-create.json
index eb97d37b5660..b07225dde207 100644
--- a/services/mcp/tests/unit/__snapshots__/tool-schemas/file-download-batch-exports-create.json
+++ b/services/mcp/tests/unit/__snapshots__/tool-schemas/file-download-batch-exports-create.json
@@ -780,17 +780,6 @@
}
]
},
- "useNewEventsCluster": {
- "anyOf": [
- {
- "type": "boolean"
- },
- {
- "type": "null"
- }
- ],
- "description": "Read this query from the native events cluster and native JSON events table (`true`), or keep its existing connection (`false` or unset). This is an internal rollout switch for eligible projects. PostHog staff manage eligibility through instance settings. Each query must opt in; project defaults cannot enable it. Only synchronous HogQLQuery SQL comparisons are supported."
- },
"useNewEventsSchema": {
"anyOf": [
{
@@ -800,7 +789,7 @@
"type": "null"
}
],
- "description": "Read events from the native JSON events table (`true`) or the legacy events table (`false`). When unset, the project's stored value applies, then the `CLICKHOUSE_HOGQL_USE_NEW_EVENTS_SCHEMA` instance settings. This is an internal rollout switch. PostHog staff set the project value in Django admin and the project settings API ignores it."
+ "description": "Read events from the native JSON events table (`true`) or the legacy events table (`false`). When unset, a staff person's `ph_new_events_schema` cookie applies, then the project's listing in the `CLICKHOUSE_HOGQL_USE_NEW_EVENTS_SCHEMA_TEAMS` instance setting, then the `CLICKHOUSE_HOGQL_USE_NEW_EVENTS_SCHEMA` instance setting. This is an internal rollout switch, and the project settings API ignores it."
},
"usePreaggregatedIntermediateResults": {
"anyOf": [