Skip to content

:hedgehog: PostHog is the leading platform for building self-driving products. Our developer tools – AI observability, analytics, session replay, flags, experiments, error tracking, logs, and more – capture all the context agents need to diagnose problems, uncover opportunities, and ship fixes. Steer it all from Slack, web, desktop, or the MCP.

fix(hogql): export JSON text and keep SAMPLE in wrapped event joins (#126098)

Sandy Spicer2345e2a0faf8
diff --git a/docs/published/handbook/engineering/databases/hogql-python.md b/docs/published/handbook/engineering/databases/hogql-python.md
index dae538973a2d..64586dc61f23 100644
--- a/docs/published/handbook/engineering/databases/hogql-python.md
+++ b/docs/published/handbook/engineering/databases/hogql-python.md
@@ -191,11 +191,19 @@ DateTime property definitions keep their conversions when compared with strings,
Boolean conversion reads the native path directly because object and array text cannot match `true` or `false`; numeric conversions already read native paths directly.
Constants starting with `[` or `{` keep the full property read because arrays and objects serialize differently from scalars.
Single-property reads serialize sub-objects with ClickHouse's JSON formatter; whole-document reads still remove declared defaults.
+A JSON function on the native `properties` column needs JSON text, and rebuilding the whole document as text from every path is the slowest way to get it.
+A call with a constant first key, such as `JSONLength(properties, 'details')`, serializes only that key's value and runs the function on it. This includes calls through `toString(properties)` and keys stored among the temporary properties.
+String, floating-point and Boolean extraction skip the serialization as well: they read the native value and apply the function's conversion and missing-value rules, independently of property-definition types. A stored floating-point value keeps the parser for string extraction, because JSON number formatting decides the result. It reads as false for Boolean extraction and as `Double` for `JSONType`, because the serialized form of an integral value is an integer token.
+Integer extraction keeps ClickHouse's JSON parser on the serialized value, because its coercion rules cost more SQL than the parse saves.
+`JSONHas`, a key computed per row and a call with no key still rebuild the whole document.
Grouping keys and expressions evaluated after grouping use the general property reader so conversions reference the same expression as GROUP BY.
This includes window inputs, HAVING, ORDER BY and LIMIT BY; ordinary aggregate inputs and row-level predicates retain direct-path optimizations.
When a join constraint needs a lazy join of the joined events table, HogQL wraps that table in a subquery and the outer query reads the subquery's columns.
The subquery exports the columns, lazy-join fields and property keys the outer query reads, found by their resolved table binding rather than by alias text.
A property key is exported as its stored value and the outer read keeps its own conversion, so a Numeric definition and an explicit string parser read the same export.
+A key that a JSON function parses, as in `JSONExtractRaw(t.properties, 'key')`, is exported as its JSON text instead, so an object or an array is not quoted a second time.
+A user-written `toJSONString(t.properties.key)` exports the JSON text for a key without a property definition and the stored value for a key with one, so the definition's conversion applies before the serialization as in a direct read.
+In the clauses of a grouped query that run after grouping, the select list, `GROUP BY`, `HAVING` and `ORDER BY`, every key is exported as its stored value, because a direct read there serializes the value too, and a `GROUP BY` key then shares its column with the serialized read of the same key.
The whole `properties` document is exported only where the outer query reads it whole, and each wrapped table decides this on its own.
Raw SQL property readers preserve backslashes before forward slashes whether ClickHouse's `output_format_json_escape_forward_slashes` setting is enabled or disabled.
Native queries disable forward-slash escaping so object and array text retains `/`.
diff --git a/posthog/hogql/ast.py b/posthog/hogql/ast.py
index fde4e3546b44..6d198f417797 100644
--- a/posthog/hogql/ast.py
+++ b/posthog/hogql/ast.py
@@ -768,14 +768,19 @@ class UnresolvedFieldType(Type):
return UnknownType()
-@dataclass(kw_only=True, slots=True)
+@dataclass(kw_only=True, slots=True, frozen=False)
class PropertyType(Type):
chain: list[str | int]
field_type: FieldType
+ # The read yields the property's JSON text, as the document holds it, so no property-type conversion applies.
+ as_json: bool = False
+
# The property has been moved into a field we query from a joined subquery
joined_subquery: Optional[SelectQueryAliasType] = field(default=None, init=False)
joined_subquery_field_name: Optional[str] = field(default=None, init=False)
+ # That field holds the property's JSON text, serialized inside the subquery, so the printer must not serialize it again.
+ joined_subquery_json_text: bool = field(default=False, init=False)
def get_child(self, name: str | int, context: HogQLContext) -> Type:
return PropertyType(chain=[*self.chain, name], field_type=self.field_type)
@@ -786,6 +791,8 @@ class PropertyType(Type):
def resolve_constant_type(self, context: HogQLContext) -> ConstantType:
if self.joined_subquery is not None and self.joined_subquery_field_name is not None:
return self.joined_subquery.resolve_column_constant_type(self.joined_subquery_field_name, context)
+ if self.as_json:
+ return StringType(nullable=True)
database_field = self.field_type.resolve_database_field(context)
if isinstance(database_field, StructDatabaseField):
diff --git a/posthog/hogql/functions/clickhouse/json.py b/posthog/hogql/functions/clickhouse/json.py
index 329d500f2dc7..21b42d39dccd 100644
--- a/posthog/hogql/functions/clickhouse/json.py
+++ b/posthog/hogql/functions/clickhouse/json.py
@@ -177,3 +177,10 @@ JSON_FUNCTIONS: dict[str, HogQLFunctionMeta] = {
),
"JSON_VALUE": HogQLFunctionMeta("JSON_VALUE", 2, 2, signatures=[((StringType(), StringType()), StringType())]),
}
+
+# The functions that take the document first and then a key path. On the native events table, a call with a constant
+# first key reads that key's subcolumn instead of the whole rebuilt document (`PropertySwapper._rewrite_json_property_call`).
+# The excluded functions take no key path (`isValidJSON`, `JSONArrayLength`) or a JSONPath string (`JSON_VALUE`).
+JSON_KEY_PATH_FUNCTIONS = frozenset(JSON_FUNCTIONS) - {"isValidJSON", "JSONArrayLength", "JSON_VALUE"}
+# Key-path functions whose last argument is the ClickHouse type to extract, so the key path ends before it.
+JSON_TYPED_PATH_FUNCTIONS = frozenset({"JSONExtract", "JSONExtractKeysAndValues"})
diff --git a/posthog/hogql/printer/test/test_printer.py b/posthog/hogql/printer/test/test_printer.py
index dcfe152b3ef5..2ecfb48da712 100644
--- a/posthog/hogql/printer/test/test_printer.py
+++ b/posthog/hogql/printer/test/test_printer.py
@@ -7113,9 +7113,8 @@ class TestMaterializedColumnOptimization(ClickhouseTestMixin, APIBaseTest):
with materialized("events", "test_prop", is_nullable=False) as mat_col:
printed = self._expr("JSONExtractString(properties, 'test_prop')")
if settings.CLICKHOUSE_HOGQL_USE_NEW_EVENTS_SCHEMA:
- assert printed == (
- f"JSONExtractString(ifNull({self._json_dynamic_property_expr('test_prop', as_json=True)}, ''))"
- )
+ assert "events.properties.test_prop" in printed, printed
+ assert "JSONExtractString(ifNull(" not in printed, printed
assert mat_col.name not in printed
else:
assert printed == f"nullIf(nullIf(events.{mat_col.name}, ''), 'null')"
@@ -7140,9 +7139,8 @@ class TestMaterializedColumnOptimization(ClickhouseTestMixin, APIBaseTest):
assert "mat_numeric_prop" not in printed
if settings.CLICKHOUSE_HOGQL_USE_NEW_EVENTS_SCHEMA:
- assert printed == (
- f"JSONExtractString(ifNull({self._json_dynamic_property_expr('numeric_prop', as_json=True)}, ''))"
- )
+ assert "events.properties.numeric_prop" in printed, printed
+ assert "JSONExtractString(ifNull(" not in printed, printed
else:
assert "JSONExtractString(events.properties" in printed
diff --git a/posthog/hogql/test/__snapshots__/test_resolver.ambr b/posthog/hogql/test/__snapshots__/test_resolver.ambr
index 8826c47033e2..60027a174528 100644
--- a/posthog/hogql/test/__snapshots__/test_resolver.ambr
+++ b/posthog/hogql/test/__snapshots__/test_resolver.ambr
@@ -2046,6 +2046,7 @@
from_asterisk: False
start: 7
type: {
+ as_json: False
chain: [
"$exception_issue_id"
]
@@ -2053,6 +2054,7 @@
name: "properties"
table_type: <recursion ...>
}
+ joined_subquery_json_text: False
}
}
from_asterisk: False
@@ -3117,6 +3119,7 @@
from_asterisk: False
start: 7
type: {
+ as_json: False
chain: [
"$exception_issue_id"
]
@@ -3124,6 +3127,7 @@
name: "properties"
table_type: <recursion ...>
}
+ joined_subquery_json_text: False
}
}
from_asterisk: False
@@ -8390,6 +8394,7 @@
from_asterisk: False
start: 7
type: {
+ as_json: False
chain: [
"$exception_issue_id"
]
@@ -8397,6 +8402,7 @@
name: "properties"
table_type: <recursion ...>
}
+ joined_subquery_json_text: False
}
}
from_asterisk: False
@@ -9040,6 +9046,7 @@
from_asterisk: False
start: 7
type: {
+ as_json: False
chain: [
"$exception_issue_id"
]
@@ -9047,6 +9054,7 @@
name: "properties"
table_type: <recursion ...>
}
+ joined_subquery_json_text: False
}
}
from_asterisk: False
@@ -10342,6 +10350,7 @@
from_asterisk: False
start: 7
type: {
+ as_json: False
chain: [
"$exception_issue_id"
]
@@ -10349,6 +10358,7 @@
name: "properties"
table_type: <recursion ...>
}
+ joined_subquery_json_text: False
}
}
from_asterisk: False
@@ -12039,6 +12049,7 @@
from_asterisk: False
start: 7
type: {
+ as_json: False
chain: [
"$exception_issue_id"
]
@@ -12046,6 +12057,7 @@
name: "properties"
table_type: <recursion ...>
}
+ joined_subquery_json_text: False
}
}
from_asterisk: False
diff --git a/posthog/hogql/transforms/clickhouse_property_resolution.py b/posthog/hogql/transforms/clickhouse_property_resolution.py
index 117661eda1e6..716d9ab463d8 100644
--- a/posthog/hogql/transforms/clickhouse_property_resolution.py
+++ b/posthog/hogql/transforms/clickhouse_property_resolution.py
@@ -39,7 +39,8 @@ from posthog.hogql.context import HogQLContext
from posthog.hogql.database.models import DatabaseField, MapStringDatabaseField, StringJSONDatabaseField
from posthog.hogql.database.schema.util.null_sentinels import MAT_COL_NULL_SENTINELS, null_if_sentinels
from posthog.hogql.errors import QueryError
-from posthog.hogql.functions.mapping import HOGQL_COMPARISON_MAPPING, find_hogql_aggregation
+from posthog.hogql.functions.clickhouse.json import JSON_KEY_PATH_FUNCTIONS
+from posthog.hogql.functions.mapping import HOGQL_COMPARISON_MAPPING
from posthog.hogql.printer.base import resolve_field_type
from posthog.hogql.printer.clickhouse import AI_BLOOM_FILTER_PROPERTIES, COLUMNS_WITH_HACKY_OPTIMIZED_NULL_HANDLING
from posthog.hogql.restricted_properties import (
@@ -49,6 +50,12 @@ from posthog.hogql.restricted_properties import (
restricted_key_path,
restricted_property_keys_for_table_type,
)
+from posthog.hogql.transforms.grouped_clauses import (
+ GroupedClauseTracker,
+ before_grouping,
+ is_grouped,
+ visit_after_grouping,
+)
from posthog.hogql.type_system import (
ComparisonCompatibility,
comparison_compatibility,
@@ -262,6 +269,15 @@ def _blob_field_type_of(node: ast.PropertyAccess) -> ast.FieldType | None:
return expr_type if isinstance(expr_type, ast.FieldType) else None
+def _exported_json_text_read(expr: ast.Expr) -> ast.Field | None:
+ """The field behind `expr` if it reads a column the wrapper around a joined table filled with a property's JSON text."""
+ while isinstance(expr, ast.Alias):
+ expr = expr.expr
+ if isinstance(expr, ast.Field) and isinstance(expr.type, ast.PropertyType) and expr.type.joined_subquery_json_text:
+ return expr
+ return None
+
+
# --- value substitution: rebuild the printer's materialized-column read as AST ----------------------------------------
@@ -649,31 +665,6 @@ def _is_moved_events_property(field_type: ast.FieldType, key: str, context: HogQ
)
-# JSON functions that take a key path and that nothing earlier rewrites to read one property. The printer passes them
-# the serialized `properties` document, which never holds a key the native cleaner moves to `temporary_properties`.
-_TEMPORARY_PROPERTY_JSON_PATH_FUNCTIONS = frozenset(
- {
- "JSONExtractArrayRaw",
- "JSONExtractKeys",
- "JSONExtractKeysAndValues",
- "JSONExtractKeysAndValuesRaw",
- "JSONLength",
- "JSONType",
- }
-)
-
-_JSON_KEY_PATH_FUNCTIONS = _TEMPORARY_PROPERTY_JSON_PATH_FUNCTIONS | {
- "JSONExtract",
- "JSONExtractBool",
- "JSONExtractFloat",
- "JSONExtractInt",
- "JSONExtractRaw",
- "JSONExtractString",
- "JSONExtractUInt",
- "JSONHas",
-}
-
-
def _names_temporary_event_property(key: ast.Expr) -> ast.Expr:
"""The SQL form of `is_temporary_event_property`, for a key that is computed per row."""
name = _call("toString", [key])
@@ -1164,7 +1155,7 @@ class _OptimizableProperty:
return ast.ArrayAccess(array=self.bare_column(), property=_const(self.key), type=ast.StringType(nullable=False))
-class ClickHousePropertyResolver(CloningVisitor):
+class ClickHousePropertyResolver(GroupedClauseTracker, CloningVisitor):
"""Rewrites lowered `PropertyAccess` reads (and comparisons over them) to read backing columns where they exist.
After this pass, a backed property is an ordinary column expression; an unbacked (or restricted) one stays a
@@ -1183,7 +1174,6 @@ class ClickHousePropertyResolver(CloningVisitor):
# Depth of `indexHint(...)` calls above the node being visited. A hint is an optimizer directive only, so a
# comparison inside one must not grow a second, nested hint.
self._index_hint_depth = 0
- self._in_grouped_clause = False
def visit_select_query(self, node: ast.SelectQuery) -> ast.SelectQuery:
scope: set[int] = set()
@@ -1195,40 +1185,16 @@ class ClickHousePropertyResolver(CloningVisitor):
table_type = getattr(table_type, "table_type", None)
join = join.next_join
self._tables_in_scope.append(scope)
- was_in_grouped_clause = self._in_grouped_clause
- self._in_grouped_clause = False
try:
- if not node.group_by and node.group_by_mode != "all":
- return super().visit_select_query(node)
- result = super().visit_select_query(
- replace(
- node,
- select=[],
- group_by=None,
- having=None,
- order_by=None,
- qualify=None,
- window_exprs=None,
- limit_by=None,
- interpolate=None,
- )
- )
- self._in_grouped_clause = True
- result.select = [self.visit(expr) for expr in node.select]
- result.group_by = [self.visit(expr) for expr in node.group_by] if node.group_by else None
- result.having = self.visit(node.having)
- result.order_by = [self.visit(expr) for expr in node.order_by] if node.order_by else None
- result.qualify = self.visit(node.qualify)
- result.window_exprs = (
- {name: self.visit(expr) for name, expr in node.window_exprs.items()} if node.window_exprs else None
- )
- result.limit_by = self.visit(node.limit_by)
- result.interpolate = (
- [self.visit(expr) for expr in node.interpolate] if node.interpolate is not None else None
- )
+ with self._grouped_clause(False):
+ if not is_grouped(node):
+ return super().visit_select_query(node)
+ result = super().visit_select_query(before_grouping(node))
+ with self._grouped_clause(True):
+ for clause, value in visit_after_grouping(node, self.visit).items():
+ setattr(result, clause, value)
return result
finally:
- self._in_grouped_clause = was_in_grouped_clause
self._tables_in_scope.pop()
def _property_table_in_scope(self, field_type: ast.FieldType) -> bool:
@@ -1346,17 +1312,136 @@ class ClickHousePropertyResolver(CloningVisitor):
# --- comparison / call rewrites ---
+ def _rewrite_scalar_json_extract(self, node: ast.Call) -> ast.Expr | None:
+ """String, float or Bool extraction of one native property, or its JSON type, computed from its subcolumn.
+
+ Covers `JSONExtractString`, `JSONExtractFloat`, `JSONExtractBool`, `JSONExtract` to String or Float64, and
+ `JSONType`. The property pass has replaced the document with `ifNull(toJSONString(<subcolumn>), '')`, so the
+ function would serialize one value and parse it back. This reads the `Dynamic` value instead and reproduces
+ the function's result per stored type: a string passes through, a number or Bool becomes its text for a string
+ target, an integer becomes a number for a float target, a Bool does so only for typed `JSONExtract`, and a
+ missing key gives '' or 0 unless the target is Nullable. A stored Float64 keeps the function for a string
+ target, because JSON number formatting (an integer token for 2.0, exponent notation) decides the result. It
+ reads as false for a Bool target and as `Double` for `JSONType`, which the integer token of an integral float
+ would hide.
+
+ Integer extraction keeps ClickHouse's JSON parser: its coercion rules (fractional and exponent strings truncate,
+ Bool reads as 0, overflow reads as 0) take more SQL to emulate than the parse of one serialized value costs.
+ """
+ scalar_types = {
+ "JSONExtractString": "String",
+ "JSONExtractFloat": "Float64",
+ "JSONExtractBool": "Bool",
+ "JSONType": "String",
+ }
+ target = scalar_types.get(node.name)
+ nullable = False
+ path_args = node.args[1:]
+ if node.name == "JSONExtract" and len(node.args) >= 2:
+ type_arg = node.args[-1]
+ if not isinstance(type_arg, ast.Constant) or not isinstance(type_arg.value, str):
+ return None
+ target = type_arg.value
+ nullable = target.startswith("Nullable(") and target.endswith(")")
+ if nullable:
+ target = target[len("Nullable(") : -1]
+ if target not in ("String", "Float64"):
+ return None
+ path_args = node.args[1:-1]
+ if target is None or not node.args:
+ return None
+ document = node.args[0]
+ if not (
+ isinstance(document, ast.Call)
+ and document.name == "ifNull"
+ and len(document.args) == 2
+ and isinstance(document.args[1], ast.Constant)
+ and document.args[1].value == ""
+ and isinstance(document.args[0], ast.Call)
+ and document.args[0].name == "toJSONString"
+ and len(document.args[0].args) == 1
+ ):
+ return None
+ access = self._lowered_property_operand(document.args[0].args[0])
+ if access is None or not all(isinstance(key, str) for key in access.keys):
+ return None
+ if not all(isinstance(arg, ast.Constant) and isinstance(arg.value, str) for arg in path_args):
+ return None
+ field_type = _blob_field_type_of(access)
+ assert field_type is not None
+ keys = cast(list[str], access.keys) + [cast(ast.Constant, arg).value for arg in path_args]
+ if self._is_virtual_feature_flag_property(field_type, keys[0]):
+ return None
+ source = resolve_materialized_property_source(field_type, keys[0], self.context)
+ if source is None or source.kind != "json_subcolumn":
+ return None
+ value = _call(
+ "accurateCast",
+ [_json_subcolumn_access(field_type, keys, source=source, is_nullable=True), _const("Dynamic")],
+ )
+ value_type = _call("dynamicType", [clone_expr(value)])
+ if node.name == "JSONType":
+ json_text = _json_subcolumn_value_expr(field_type, keys, source=source, as_json=True)
+ json_type = _call(
+ "if",
+ [
+ _call("equals", [value_type, _const("Float64")]),
+ _const("Double"),
+ _call("JSONType", [_call("ifNull", [json_text, _const("")])]),
+ ],
+ )
+ json_type.type = node.type
+ return json_type
+ if target == "String":
+ result = _json_subcolumn_value_expr(field_type, keys, source=source)
+ elif node.name == "JSONExtractBool":
+ result = _call(
+ "and",
+ [
+ _call("in", [value_type, ast.Tuple(exprs=[_const(t) for t in ("Bool", "Int64", "UInt64")])]),
+ _call("notIn", [_call("toString", [value]), ast.Tuple(exprs=[_const("0"), _const("false")])]),
+ ],
+ )
+ else:
+ result = _call("toFloat64OrNull", [_call("toString", [clone_expr(value)])])
+ if node.name == "JSONExtract":
+ result = _call(
+ "if",
+ [
+ _call("equals", [value_type, _const("Bool")]),
+ _call("accurateCastOrNull", [value, _const(target)]),
+ result,
+ ],
+ )
+ if target == "String":
+ # Float serialization can become an integer token or overflow JSON's integer parser. Keep its defaults.
+ float_extract = replace(
+ node,
+ args=[
+ _call("toJSONString", [clone_expr(value)]),
+ *([node.args[-1]] if node.name == "JSONExtract" else []),
+ ],
+ )
+ result = _call("if", [_call("equals", [value_type, _const("Float64")]), float_extract, result])
+ if not nullable:
+ default = _const("") if target == "String" else _call("accurateCast", [_const(0), _const(target)])
+ result = _call("ifNull", [result, default])
+ result.type = node.type
+ return result
+
def visit_call(self, node: ast.Call) -> ast.Expr:
+ if node.name == "toJSONString" and len(node.args) == 1:
+ exported_json_text = _exported_json_text_read(node.args[0])
+ if exported_json_text is not None:
+ # The column already holds the JSON text, so serializing it again would quote it. The read is a plain
+ # column either way, so the drop stays safe after grouping and runs before the grouped shortcut below.
+ return self.visit(exported_json_text)
+
if self._in_grouped_clause:
- if find_hogql_aggregation(node.name):
- self._in_grouped_clause = False
- try:
- return super().visit_call(node)
- finally:
- self._in_grouped_clause = True
# Shortcuts can introduce raw columns unavailable after grouping. WindowFunction is a separate node,
# so its inputs keep the grouped context while ordinary aggregates above read original rows.
- return super().visit_call(node)
+ with self._call_arguments(node):
+ return super().visit_call(node)
if node.name == "indexHint":
self._index_hint_depth += 1
@@ -1387,14 +1472,14 @@ class ClickHousePropertyResolver(CloningVisitor):
if boolean_conversion is not None:
return boolean_conversion
+ scalar_json_extract = self._rewrite_scalar_json_extract(node)
+ if scalar_json_extract is not None:
+ return scalar_json_extract
+
json_string_on_events_json = self._rewrite_to_json_string_on_events_json_subcolumn(node)
if json_string_on_events_json is not None:
return json_string_on_events_json
- temporary_property_json_function = self._rewrite_json_function_on_temporary_property(node)
- if temporary_property_json_function is not None:
- return temporary_property_json_function
-
runtime_key_json_function = self._rewrite_json_function_with_runtime_key(node)
if runtime_key_json_function is not None:
return runtime_key_json_function
@@ -1520,50 +1605,6 @@ class ClickHousePropertyResolver(CloningVisitor):
args=[ast.Call(name="ifNull", args=[value, _sentinel("")]), *[self.visit(arg) for arg in node.args[2:]]],
)
- def _rewrite_json_function_on_temporary_property(self, node: ast.Call) -> ast.Expr | None:
- """`f(properties, '$set', ...)` on native events, applied to the JSON of `$set` from `temporary_properties`."""
- if (
- not self.context.uses_new_events_schema()
- or node.name not in _TEMPORARY_PROPERTY_JSON_PATH_FUNCTIONS
- or len(node.args) < 2
- ):
- return None
- field_type = resolve_field_type(node.args[0])
- if not isinstance(field_type, ast.FieldType) or not _is_events_properties(field_type, self.context):
- return None
- if not isinstance(node.args[1], ast.Constant):
- return None
- first_key = node.args[1].value
- if not isinstance(first_key, str) or not is_temporary_event_property(first_key):
- return None
- source = resolve_json_subcolumn_source(
- field_type, DISTRIBUTED_EVENTS_JSON_TABLE, "properties", first_key, self.context
- )
- if source is None:
- # A restriction on the key or on one of its children leaves no source.
- args: list[ast.Expr] = [
- _temporary_properties_document(field_type),
- *[self.visit(arg) for arg in node.args[1:]],
- ]
- else:
- json_value = _json_subcolumn_value_expr(field_type, [first_key], source=source, as_json=True)
- args = [
- ast.Call(name="ifNull", args=[json_value, _sentinel("")], type=ast.StringType(nullable=False)),
- *[self.visit(arg) for arg in node.args[2:]],
- ]
- return ast.Call(
- start=node.start,
- end=node.end,
- type=node.type,
- name=node.name,
- args=args,
- params=node.params,
- distinct=node.distinct,
- within_group=node.within_group,
- order_by=node.order_by,
- filter_expr=node.filter_expr,
- )
-
def _rewrite_json_function_with_runtime_key(self, node: ast.Call) -> ast.Expr | None:
"""`f(properties, <key computed per row>, ...)` on native events, reading the document that holds the key.
@@ -1573,7 +1614,7 @@ class ClickHousePropertyResolver(CloningVisitor):
"""
if (
not self.context.uses_new_events_schema()
- or node.name not in _JSON_KEY_PATH_FUNCTIONS
+ or node.name not in JSON_KEY_PATH_FUNCTIONS
or len(node.args) < 2
or isinstance(node.args[1], ast.Constant)
):
diff --git a/posthog/hogql/transforms/grouped_clauses.py b/posthog/hogql/transforms/grouped_clauses.py
new file mode 100644
index 000000000000..793b21f81ab6
--- /dev/null
+++ b/posthog/hogql/transforms/grouped_clauses.py
@@ -0,0 +1,65 @@
+"""The clause split of a grouped `SELECT`, for the passes that must not rewrite an expression after grouping."""
+
+from collections.abc import Callable, Iterator
+from contextlib import contextmanager
+from dataclasses import replace
+from typing import Any
+
+from posthog.hogql import ast
+from posthog.hogql.functions.mapping import find_hogql_aggregation
+
+# The clauses that see the grouped rows. FROM, WHERE, PREWHERE and ARRAY JOIN see the source rows.
+GROUPED_CLAUSES = ("select", "group_by", "having", "order_by", "qualify", "window_exprs", "limit_by", "interpolate")
+
+
+def is_grouped(node: ast.SelectQuery) -> bool:
+ return bool(node.group_by) or node.group_by_mode == "all"
+
+
+def before_grouping(node: ast.SelectQuery) -> ast.SelectQuery:
+ """A copy of the query with only the clauses that see the source rows."""
+ empty: dict[str, Any] = {clause: [] if clause == "select" else None for clause in GROUPED_CLAUSES}
+ return replace(node, **empty)
+
+
+def visit_after_grouping(node: ast.SelectQuery, visit: Callable[[Any], Any]) -> dict[str, Any]:
+ """Visits each expression of the clauses that see the grouped rows, and returns the clauses rebuilt from the results."""
+ rebuilt: dict[str, Any] = {}
+ for clause in GROUPED_CLAUSES:
+ value = getattr(node, clause)
+ if isinstance(value, list):
+ rebuilt[clause] = [visit(expr) for expr in value]
+ elif isinstance(value, dict):
+ rebuilt[clause] = {name: visit(expr) for name, expr in value.items()}
+ else:
+ rebuilt[clause] = visit(value) if value is not None else None
+ return rebuilt
+
+
+class GroupedClauseTracker:
+ """Tracks whether the expression under visit sees the grouped rows.
+
+ An expression in a grouped clause has to stay a function of the group keys, so a pass with this tracker skips the
+ rewrites that would read a source column there. An aggregate's arguments see the source rows whatever clause the
+ aggregate is in, and a nested query starts its own split.
+ """
+
+ _in_grouped_clause: bool = False
+
+ @contextmanager
+ def _grouped_clause(self, in_grouped_clause: bool) -> Iterator[None]:
+ was_in_grouped_clause = self._in_grouped_clause
+ self._in_grouped_clause = in_grouped_clause
+ try:
+ yield
+ finally:
+ self._in_grouped_clause = was_in_grouped_clause
+
+ @contextmanager
+ def _call_arguments(self, node: ast.Call) -> Iterator[None]:
+ """The context of a call's arguments, which see the source rows when the call is an aggregate."""
+ if self._in_grouped_clause and find_hogql_aggregation(node.name):
+ with self._grouped_clause(False):
+ yield
+ else:
+ yield
diff --git a/posthog/hogql/transforms/lazy_tables.py b/posthog/hogql/transforms/lazy_tables.py
index d3bad3766aee..3c7734af706f 100644
--- a/posthog/hogql/transforms/lazy_tables.py
+++ b/posthog/hogql/transforms/lazy_tables.py
@@ -9,6 +9,12 @@ from posthog.hogql.database.models import LazyJoinToAdd, LazyTableToAdd
from posthog.hogql.errors import ResolutionError
from posthog.hogql.resolver import ResolverFactory, resolve_types
from posthog.hogql.resolver_utils import get_long_table_name
+from posthog.hogql.transforms.grouped_clauses import (
+ GroupedClauseTracker,
+ before_grouping,
+ is_grouped,
+ visit_after_grouping,
+)
from posthog.hogql.transforms.logical_property_lowering import lower_property_access
from posthog.hogql.transforms.property_types import PropertySwapper
from posthog.hogql.visitor import TraversingVisitor, clone_expr
@@ -112,9 +118,11 @@ class WrappedTableExport:
property_path: tuple[str | int, ...]
# Lower the read to an unconverted `PropertyAccess`, so the outer read keeps its own conversion context.
raw_property: bool
+ # Export the value's JSON text instead of its stored text, because the outer read parses it as JSON.
+ as_json: bool
-class WrappedTableReferenceCollector(TraversingVisitor):
+class WrappedTableReferenceCollector(GroupedClauseTracker, TraversingVisitor):
"""Collects the outer reads of one table binding before a subquery replaces that table.
Reads are matched by their resolved binding rather than by the alias they were written with. That includes the
@@ -122,15 +130,27 @@ class WrappedTableReferenceCollector(TraversingVisitor):
and it excludes a nested query's alias of the same name.
"""
- def __init__(self, table: ast.TableOrSelectType) -> None:
+ def __init__(self, table: ast.TableOrSelectType, context: HogQLContext) -> None:
super().__init__()
self.table = table
+ self.context = context
self.exports: dict[str, WrappedTableExport] = {}
self.fields: dict[int, tuple[ast.FieldType, str]] = {}
self.properties: dict[int, tuple[ast.PropertyType, str, bool]] = {}
self.reads_lazy_join = False
self._names: dict[tuple[object, ...], str] = {}
+ def visit_select_query(self, node: ast.SelectQuery) -> None:
+ # The property resolver keeps the calls around a read that sees the grouped rows, so such a read exports the
+ # value (`_collect_property`) and the clause split has to match the resolver's.
+ with self._grouped_clause(False):
+ if not is_grouped(node):
+ super().visit_select_query(node)
+ return
+ super().visit_select_query(before_grouping(node))
+ with self._grouped_clause(True):
+ visit_after_grouping(node, self.visit)
+
def visit_field(self, node: ast.Field) -> None:
field_type = node.type
while isinstance(field_type, ast.FieldAliasType):
@@ -145,21 +165,50 @@ class WrappedTableReferenceCollector(TraversingVisitor):
if hops is None:
return
field_chain = (*(hop.field for hop in hops), field_type.name)
- name = self._export(("field", field_chain), field_chain, (), raw_property=False)
+ name = self._export(("field", field_chain), field_chain, (), raw_property=False, as_json=False)
self.fields[id(field_type)] = (field_type, name)
self._note_hops(hops)
- def _collect_property(self, property_type: ast.PropertyType) -> None:
+ def visit_call(self, node: ast.Call) -> None:
+ with self._call_arguments(node):
+ if node.name == "toJSONString" and len(node.args) == 1 and self._collect_serialized_read(node.args[0]):
+ return
+ super().visit_call(node)
+
+ def _collect_serialized_read(self, expr: ast.Expr) -> bool:
+ """Collects a property of this table under a user-written `toJSONString`, exporting what a direct read serializes.
+
+ A key with a property definition exports its stored value and the outer call stays, so the result is the
+ converted value as in a direct read. A key without one exports its JSON text, because serializing the stored
+ text again would quote an object, an array or a number. The compiler's own serialization, from a JSON function
+ rewrite, arrives with its read already marked `as_json` and takes the second path whatever the definition.
+ """
+ while isinstance(expr, ast.Alias):
+ expr = expr.expr
+ if not isinstance(expr, ast.Field) or not isinstance(expr.type, ast.PropertyType):
+ return False
+ hops = _hops_to_table(expr.type.field_type.table_type, self.table)
+ if hops is None or any(isinstance(hop, ast.LazyJoinType) for hop in hops):
+ return False
+ swapper = self.context.property_swapper
+ as_json = expr.type.as_json or swapper is None or not swapper.converts(expr.type)
+ self._collect_property(expr.type, as_json=as_json)
+ return True
+
+ def _collect_property(self, property_type: ast.PropertyType, as_json: bool = False) -> None:
document = property_type.field_type
hops = _hops_to_table(document.table_type, self.table)
if hops is None:
return
through_lazy_join = any(isinstance(hop, ast.LazyJoinType) for hop in hops)
+ # A lazy join's subquery exports the value itself, so a read through one keeps its outer `toJSONString`. After
+ # grouping the export is the value as well, so a GROUP BY key on the same property shares the column.
+ as_json = (as_json or property_type.as_json) and not through_lazy_join and not self._in_grouped_clause
field_chain = (*(hop.field for hop in hops), document.name)
property_path = tuple(property_type.chain)
# An integer array index and a string key with the same text are different paths.
- key = ("property", field_chain, *((isinstance(link, int), link) for link in property_path))
- name = self._export(key, field_chain, property_path, raw_property=not through_lazy_join)
+ key = ("property", as_json, field_chain, *((isinstance(link, int), link) for link in property_path))
+ name = self._export(key, field_chain, property_path, raw_property=not through_lazy_join, as_json=as_json)
self.properties[id(property_type)] = (property_type, name, through_lazy_join)
self._note_hops(hops)
@@ -173,6 +222,7 @@ class WrappedTableReferenceCollector(TraversingVisitor):
field_chain: tuple[str, ...],
property_path: tuple[str | int, ...],
raw_property: bool,
+ as_json: bool,
) -> str:
name = self._names.get(key)
if name is None:
@@ -186,7 +236,11 @@ class WrappedTableReferenceCollector(TraversingVisitor):
name = f"{base}_{counter}"
self._names[key] = name
self.exports[name] = WrappedTableExport(
- name=name, field_chain=field_chain, property_path=property_path, raw_property=raw_property
+ name=name,
+ field_chain=field_chain,
+ property_path=property_path,
+ raw_property=raw_property,
+ as_json=as_json,
)
return name
@@ -245,7 +299,7 @@ class LazyTableResolver(TraversingVisitor):
for table_alias, table_type in select_type.tables.items():
if table_type is not join_ptr.type:
continue
- collector = WrappedTableReferenceCollector(table_type)
+ collector = WrappedTableReferenceCollector(table_type, self.context)
collector.visit(join_ptr.constraint)
if collector.reads_lazy_join:
tables_to_wrap.append(table_alias)
@@ -264,7 +318,7 @@ class LazyTableResolver(TraversingVisitor):
assert join_ptr is not None and isinstance(join_ptr.table, ast.Field), f"Table {table_alias} not found"
# Collect while the outer reads still identify the table they belong to.
- collector = WrappedTableReferenceCollector(original_table_type)
+ collector = WrappedTableReferenceCollector(original_table_type, self.context)
collector.visit(node)
subquery = ast.SelectQuery(
@@ -272,7 +326,12 @@ class LazyTableResolver(TraversingVisitor):
ast.Alias(alias=export.name, expr=ast.Field(chain=[table_alias, *export.field_chain]))
for export in collector.exports.values()
],
- select_from=ast.JoinExpr(table=ast.Field(chain=list(join_ptr.table.chain)), alias=table_alias),
+ # A SAMPLE clause applies to the table, not to the subquery that replaces it.
+ select_from=ast.JoinExpr(
+ table=ast.Field(chain=list(join_ptr.table.chain)),
+ alias=table_alias,
+ sample=clone_expr(join_ptr.sample, clear_types=True) if join_ptr.sample is not None else None,
+ ),
)
subquery = cast(
ast.SelectQuery,
@@ -286,14 +345,22 @@ class LazyTableResolver(TraversingVisitor):
self._read_property_path(subquery.type, column, export.property_path)
resolve_lazy_tables(subquery, self.dialect, [subquery], self.context, resolver_factory=self.resolver_factory)
for column in columns:
- if collector.exports[column.alias].raw_property:
+ export = collector.exports[column.alias]
+ if export.raw_property:
# The property-type pass runs after this and would convert a typed property inside the subquery.
# The outer read must receive the stored value, because it may sit under an explicit string parser
# (toFloatOrZero) that rejects the converted type, so the inner read is lowered to a raw extract now.
column.expr = lower_property_access(column.expr, self.context)
+ if export.as_json:
+ # The property resolver drops the outer `toJSONString` of a read from this column, so the column has
+ # to hold the JSON text itself.
+ column.expr = ast.Call(name="toJSONString", args=[column.expr], type=ast.StringType(nullable=True))
+ column.type = ast.FieldAliasType(alias=column.alias, type=ast.StringType(nullable=True))
+ subquery.type.columns[column.alias] = column.type
join_ptr.table = subquery
join_ptr.alias = table_alias
+ join_ptr.sample = None
join_ptr.type = ast.SelectQueryAliasType(alias=table_alias, select_query_type=subquery.type)
select_type.tables[table_alias] = join_ptr.type
@@ -306,6 +373,7 @@ class LazyTableResolver(TraversingVisitor):
# definition, while the printer reads the exported column.
property_type.joined_subquery = join_ptr.type
property_type.joined_subquery_field_name = export_name
+ property_type.joined_subquery_json_text = collector.exports[export_name].as_json
if through_lazy_join:
# The lazy join's subquery converted the value already, as it does without this wrapper.
property_type.field_type.table_type = join_ptr.type
diff --git a/posthog/hogql/transforms/property_types.py b/posthog/hogql/transforms/property_types.py
index 9c3d992ebc28..73b255f405c8 100644
--- a/posthog/hogql/transforms/property_types.py
+++ b/posthog/hogql/transforms/property_types.py
@@ -16,6 +16,7 @@ from posthog.hogql.database.schema.flag_evaluations import FlagEvaluationsPerson
from posthog.hogql.database.schema.groups import GroupsTable
from posthog.hogql.database.schema.persons import PersonsTable, RawPersonsTable
from posthog.hogql.escape_sql import escape_hogql_identifier
+from posthog.hogql.functions.clickhouse.json import JSON_KEY_PATH_FUNCTIONS, JSON_TYPED_PATH_FUNCTIONS
from posthog.hogql.helpers.timestamp_visitor import parse_zoned_datetime_string
from posthog.hogql.property_metadata import load_property_metadata
from posthog.hogql.property_planner import PropertySourceKind, plan_property_access
@@ -239,7 +240,7 @@ class PropertySwapper(CloningVisitor):
)
def visit_call(self, node: ast.Call):
- rewritten = self._try_rewrite_json_extract_to_mat_column(node)
+ rewritten = self._rewrite_json_property_call(node)
if rewritten is not None:
return rewritten
@@ -303,19 +304,21 @@ class PropertySwapper(CloningVisitor):
),
)
- def _try_rewrite_json_extract_to_mat_column(self, node: ast.Call) -> ast.Expr | None:
- """Rewrite safe direct JSON property extraction to avoid reading the raw JSON blob.
+ def _rewrite_json_property_call(self, node: ast.Call) -> ast.Expr | None:
+ """A JSON function on `properties` with a constant first key reads that key instead of the whole document.
- When users write raw JSONExtractString(properties, '$foo') in HogQL, ClickHouse otherwise decompresses the full
- properties JSON blob. Under the native JSON events schema, scalar extracts with constant string paths become
- direct JSON subcolumn reads plus an explicit cast. Under the legacy schema, simple extracts still rewrite through
- property access only when that can resolve to an equivalent materialized column.
+ On the native events table `properties` is a JSON column, so a JSON function over it needs the document rebuilt
+ as text from every subcolumn, once per row. With a constant first key, only that key's value is serialized and
+ the function runs on it, for the same answer at a fraction of the cost. The ClickHouse property resolver later
+ removes even that serialization for string, float and Bool extraction (`_rewrite_scalar_json_extract`).
- Complex typed JSONExtract(...) values are left alone because ClickHouse's native JSON subcolumn access exposes
- leaf values reliably, but not every object/array path round-trips as a standalone subcolumn value.
+ The read is marked `as_json`, so the metadata pass leaves it unconverted and the JSON function determines
+ conversion and defaults. `JSONHas` keeps the whole document: a key holding JSON null and a missing key read the
+ same from a subcolumn, but `JSONHas` tells them apart. Legacy calls rewrite only when an equivalent materialized
+ column exists.
"""
property_path = self._simple_json_extract_property_path(node)
- if node.name not in ("JSONExtract", "JSONExtractRaw", *_JSON_EXTRACT_SCALAR_CASTS):
+ if node.name not in JSON_KEY_PATH_FUNCTIONS or node.name == "JSONHas":
return None
if not node.args:
return None
@@ -324,6 +327,15 @@ class PropertySwapper(CloningVisitor):
field_arg = node.args[0]
if isinstance(field_arg, ast.Alias):
field_arg = field_arg.expr
+ if (
+ self.context.uses_new_events_schema()
+ and isinstance(field_arg, ast.Call)
+ and field_arg.name == "toString"
+ and len(field_arg.args) == 1
+ ):
+ field_arg = field_arg.args[0]
+ if isinstance(field_arg, ast.Alias):
+ field_arg = field_arg.expr
if not isinstance(field_arg, ast.Field):
return None
@@ -405,14 +417,14 @@ class PropertySwapper(CloningVisitor):
@staticmethod
def _simple_json_extract_property_path(node: ast.Call) -> list[str | int] | None:
- if node.name == "JSONExtract":
+ if node.name in JSON_TYPED_PATH_FUNCTIONS:
if len(node.args) < 3:
return None
type_arg = node.args[-1]
if not isinstance(type_arg, ast.Constant) or not isinstance(type_arg.value, str):
return None
path_args = node.args[1:-1]
- elif node.name in _JSON_EXTRACT_SCALAR_CASTS or node.name == "JSONExtractRaw":
+ elif node.name in JSON_KEY_PATH_FUNCTIONS:
path_args = node.args[1:]
else:
return None
@@ -429,7 +441,7 @@ class PropertySwapper(CloningVisitor):
@staticmethod
def _json_extract_first_key(node: ast.Call) -> str | int | None:
- path_args = node.args[1:-1] if node.name == "JSONExtract" else node.args[1:]
+ path_args = node.args[1:-1] if node.name in JSON_TYPED_PATH_FUNCTIONS else node.args[1:]
if not path_args or not isinstance(path_args[0], ast.Constant):
return None
first_key = path_args[0].value
@@ -468,15 +480,17 @@ class PropertySwapper(CloningVisitor):
order_by=node.order_by,
filter_expr=node.filter_expr,
)
- property_field = ast.Field(
+ property_value = ast.Field(
start=node.start,
end=node.end,
chain=[*field_arg.chain, first_key],
- type=ast.PropertyType(chain=[first_key], field_type=field_type),
+ # The function parses this value as JSON, so the read is marked `as_json` and the metadata pass leaves a
+ # typed key unconverted.
+ type=ast.PropertyType(chain=[first_key], field_type=field_type, as_json=True),
)
property_document: ast.Expr = ast.Call(
name="toJSONString",
- args=[property_field],
+ args=[property_value],
type=ast.StringType(nullable=True),
)
property_document = ast.Call(
@@ -488,9 +502,7 @@ class PropertySwapper(CloningVisitor):
type=ast.StringType(nullable=False),
)
- # Keep ClickHouse's JSONExtract implementation as the source of truth for coercion and
- # default-value semantics. Only replace the full document with the first requested
- # property's serialized value, then apply any remaining path components unchanged.
+ # Keep remaining indices on the JSON function: integer indices can address both arrays and object members.
return ast.Call(
start=node.start,
end=node.end,
@@ -685,52 +697,72 @@ class PropertySwapper(CloningVisitor):
return self._field_type_to_property_call(node, prop_type)
type = node.type
- if isinstance(type, ast.PropertyType) and type.field_type.name == "properties" and len(type.chain) == 1:
+ if isinstance(type, ast.PropertyType) and type.as_json:
+ # The read is consumed as JSON text, so it keeps the stored value whatever the property's type.
+ return node
+ if isinstance(type, ast.PropertyType):
+ definition = self._property_definition(type)
+ if definition is not None:
+ return self._convert_string_property_to_type(node, *definition)
+ return node
+
+ def _property_definition(self, type: ast.PropertyType) -> tuple[Literal["event", "person", "group"], str] | None:
+ """The metadata entry that applies to this property read, as its kind and key, or None when none does."""
+ if type.field_type.name == "properties" and len(type.chain) == 1:
property_name = str(type.chain[0])
if (
isinstance(type.field_type.table_type, ast.VirtualTableType)
and type.field_type.table_type.field == "poe"
):
if property_name in self.person_properties:
- return self._convert_string_property_to_type(node, "person", property_name)
+ return ("person", property_name)
elif isinstance(type.field_type.table_type, ast.BaseTableType):
table_type = type.field_type.table_type
resolved_table = table_type.resolve_database_table(self.context)
if isinstance(resolved_table, (PersonsTable, RawPersonsTable)):
if property_name in self.person_properties:
- return self._convert_string_property_to_type(node, "person", property_name)
+ return ("person", property_name)
if isinstance(resolved_table, GroupsTable):
if isinstance(table_type, ast.LazyJoinType):
if table_type.field.startswith("group_"):
group_id = int(table_type.field.split("_")[1])
if f"{group_id}_{property_name}" in self.group_properties:
- return self._convert_string_property_to_type(
- node, "group", f"{group_id}_{property_name}"
- )
+ return ("group", f"{group_id}_{property_name}")
elif isinstance(table_type, ast.LazyTableType):
global_group_id: Optional[int] = (
self.context.globals.get("group_id") if self.context.globals else None
)
if isinstance(global_group_id, int):
if f"{global_group_id}_{property_name}" in self.group_properties:
- return self._convert_string_property_to_type(
- node, "group", f"{global_group_id}_{property_name}"
- )
+ return ("group", f"{global_group_id}_{property_name}")
if isinstance(resolved_table, EventsPersonSubTable | FlagEvaluationsPersonSubTable):
if property_name in self.person_properties:
- return self._convert_string_property_to_type(node, "person", property_name)
+ return ("person", property_name)
elif isinstance(resolved_table, EventsTable | FlagEvaluationsTable):
if property_name in self.event_properties:
- return self._convert_string_property_to_type(node, "event", property_name)
- if isinstance(type, ast.PropertyType) and type.field_type.name == "person_properties" and len(type.chain) == 1:
+ return ("event", property_name)
+ if type.field_type.name == "person_properties" and len(type.chain) == 1:
property_name = str(type.chain[0])
if isinstance(type.field_type.table_type, ast.BaseTableType):
resolved_table = type.field_type.table_type.resolve_database_table(self.context)
if isinstance(resolved_table, EVENTS_TABLE_TYPES):
if property_name in self.person_properties:
- return self._convert_string_property_to_type(node, "person", property_name)
+ return ("person", property_name)
+ return None
- return node
+ def converts(self, type: ast.PropertyType) -> bool:
+ """Whether this pass replaces the property read with a typed conversion, so a caller that serializes the read
+ itself knows whether the stored value or the converted value is what a direct read yields."""
+ definition = self._property_definition(type)
+ if definition is None:
+ return False
+ kind, property_name = definition
+ properties_by_type = {
+ "event": self.event_properties,
+ "person": self.person_properties,
+ "group": self.group_properties,
+ }
+ return properties_by_type[kind].get(property_name, {}).get("type") in ("Numeric", "Boolean", "DateTime")
def _convert_string_property_to_type(
self,
diff --git a/posthog/hogql/transforms/test/test_lazy_tables.py b/posthog/hogql/transforms/test/test_lazy_tables.py
index 164e08256d26..a4500da80dc6 100644
--- a/posthog/hogql/transforms/test/test_lazy_tables.py
+++ b/posthog/hogql/transforms/test/test_lazy_tables.py
@@ -1,5 +1,6 @@
import re
import json
+from collections import Counter
from typing import Any
from uuid import UUID
@@ -151,6 +152,66 @@ class TestWrappedEventJoins(SimpleTestCase):
printed, _ = self._compile(f"SELECT {select} {WRAPPED_JOIN} {clause}", native=native)
assert self._exports(printed, "joined", native) == expected, printed
+ @parameterized.expand([("legacy", False), ("native", True)])
+ def test_exports_json_text_for_reads_that_parse_it(self, _name: str, native: bool) -> None:
+ printed, _ = self._compile(
+ "SELECT JSONExtractRaw(joined.properties, 'a'), JSONExtractString(joined.properties, 'a', 'b'), "
+ "JSONExtractRaw(joined.properties, 'amount'), toJSONString(joined.properties.amount), "
+ f"toJSONString(joined.properties.a), joined.properties.a {WRAPPED_JOIN}",
+ native=native,
+ )
+ outer = printed.split(" FROM ", 1)[0]
+ # Alias names echo the HogQL source, so they are stripped before counting the calls that remain outside.
+ assert re.sub(r" AS `[^`]*`", "", outer).count("toJSONString(") == 1, printed
+ assert "toJSONString(accurateCastOrNull(joined.properties___amount" in outer, printed
+ assert re.search(r", joined\.properties___a(_\d+)? AS a$", outer), printed
+ inner = self._wrapper_select_list(printed, "joined", native)
+ assert "accurateCastOrNull" not in inner, printed
+ if native:
+ assert "JSONExtractRaw(ifNull(joined.properties___a, ''))" in outer, printed
+ assert "JSONExtractString(ifNull(joined.properties___a, ''), %(hogql_val_" in outer, printed
+ assert "JSONExtractRaw(ifNull(joined.properties___amount, ''))" in outer, printed
+ assert self._export_counts(printed, "joined", native) == {"properties___a": 2, "properties___amount": 2}
+ assert "toJSONString(joined.properties.^a)" in inner, printed
+ assert "JSONExtractKeysAndValuesRaw" not in inner, printed
+ else:
+ assert "JSONExtractRaw(joined.properties, %(hogql_val_" in outer, printed
+ assert self._export_counts(printed, "joined", native) == {
+ "properties": 1,
+ "properties___a": 2,
+ "properties___amount": 1,
+ }
+ assert "toJSONString(replaceRegexpAll(nullIf(nullIf(JSONExtractRaw(joined.properties, %(hogql_val_" in inner
+
+ @parameterized.expand([("legacy", False), ("native", True)])
+ def test_grouped_clauses_export_the_value_under_a_serialization(self, _name: str, native: bool) -> None:
+ printed, _ = self._compile(
+ "SELECT toJSONString(joined.properties.a) AS a, JSONExtractString(joined.properties, 'label') AS label, "
+ f"any(toJSONString(joined.properties.arr)), count() {WRAPPED_JOIN} GROUP BY joined.properties.a, label "
+ "HAVING toJSONString(joined.properties.a) != '' ORDER BY toJSONString(joined.properties.a)",
+ native=native,
+ )
+ outer = re.sub(r" AS `[^`]*`", "", printed.split(" FROM ", 1)[0])
+ assert "toJSONString(joined.properties___a) AS a" in outer, printed
+ assert "any(joined.properties___arr)" in outer, printed
+ tail = printed.rsplit(") AS joined ON ", 1)[1]
+ assert "GROUP BY joined.properties___a," in tail, printed
+ assert tail.count("toJSONString(joined.properties___a)") == 2, printed
+ inner = self._wrapper_select_list(printed, "joined", native)
+ if native:
+ assert "JSONExtractString(ifNull(toJSONString(joined.properties___label), ''))" in outer, printed
+ assert self._exports(printed, "joined", native) == {
+ "properties___a",
+ "properties___label",
+ "properties___arr",
+ }
+ assert "toString(joined.properties.a)" in inner, printed
+ assert "toString(joined.properties.arr)" not in inner, printed
+ else:
+ assert "JSONExtractString(joined.properties, %(hogql_val_" in outer, printed
+ assert self._exports(printed, "joined", native) == {"properties___a", "properties", "properties___arr"}
+ assert inner.count("toJSONString(") == 1, printed
+
def test_decides_exports_per_source(self) -> None:
printed, _ = self._compile(
"SELECT b.properties, c.properties.amount FROM events AS a "
@@ -188,6 +249,14 @@ class TestWrappedEventJoins(SimpleTestCase):
"AS joined ON equals(initial__pdi.person_id, joined.pdi___person_id) WHERE equals(initial.team_id, 1) LIMIT 50000"
)
+ def test_moves_the_sample_clause_into_the_wrapper(self) -> None:
+ printed, _ = self._compile(
+ "SELECT count() FROM events AS initial JOIN events AS joined SAMPLE 0.1 "
+ "ON initial.person_id = joined.person_id"
+ )
+ assert "FROM events AS joined SAMPLE 0.1 " in printed, printed
+ assert printed.count("SAMPLE") == 1, printed
+
def test_corrects_the_person_id_inside_the_wrapper_only(self) -> None:
printed, _ = self._compile(f"SELECT joined.person_id {WRAPPED_JOIN}")
assert printed.startswith(
@@ -215,10 +284,9 @@ class TestWrappedEventJoins(SimpleTestCase):
printed, context = self._compile(
f"SELECT joined.properties.a.b, joined.properties.a___b, joined.properties.arr[1] {WRAPPED_JOIN}"
)
- assert self._exports(printed, "joined", native=False) == {
- "properties___a___b",
- "properties___a___b_2",
- "properties___arr___1",
+ assert self._export_counts(printed, "joined", native=False) == {
+ "properties___a___b": 2,
+ "properties___arr___1": 1,
}
assert 1 in context.values.values()
assert "1" not in context.values.values()
@@ -322,6 +390,10 @@ class TestWrappedEventJoins(SimpleTestCase):
names = {name.strip("`") for name in re.findall(r" AS (`[^`]+`|\w+)", select_list)}
return names - JOIN_KEY_EXPORTS
+ def _export_counts(self, printed: str, alias: str, native: bool) -> dict[str, int]:
+ """The exports by name, with the counter a taken name gets removed, so a test pins the columns and not a suffix."""
+ return dict(Counter(re.sub(r"(?<=[^_])_\d+$", "", name) for name in self._exports(printed, alias, native)))
+
class TestLazyJoins(BaseTest):
snapshot: Any
@@ -356,10 +428,32 @@ class TestLazyJoins(BaseTest):
"sum(related.properties.amount) OVER (PARTITION BY related.$session_id)",
(7.0,),
),
+ (
+ "json_functions",
+ "JSONExtractRaw(related.properties, 'a'), JSONExtractString(related.properties, 'a', 'b'), "
+ "JSONLength(related.properties, 'a'), JSONExtractRaw(related.properties, 'amount'), "
+ "toJSONString(related.properties.amount), toJSONString(related.properties.rating)",
+ ('{"b":"nested"}', "nested", 1, '"7"', "7", None),
+ ),
+ (
+ "grouped",
+ "toJSONString(related.properties.label), JSONExtractString(related.properties, 'label'), "
+ "toJSONString(related.properties.amount), any(toJSONString(related.properties.label)), "
+ "any(JSONExtractRaw(related.properties, 'a')), count()",
+ ('"ready"', "ready", "7", '"ready"', '{"b":"nested"}', 1),
+ "GROUP BY toJSONString(related.properties.label), JSONExtractString(related.properties, 'label'), "
+ "toJSONString(related.properties.amount)",
+ ),
+ (
+ "grouped_by_key",
+ "toJSONString(related.properties.label), toJSONString(related.properties.a), count()",
+ ('"ready"', '"{\\"b\\":\\"nested\\"}"', 1),
+ "GROUP BY related.properties.label, related.properties.a",
+ ),
]
)
def test_wrapped_events_join_executes_exported_property_reads(
- self, _name: str, select: str, expected: tuple[object, ...]
+ self, _name: str, select: str, expected: tuple[object, ...], clause: str = ""
) -> None:
for name, property_type in [
("amount", "Numeric"),
@@ -388,7 +482,7 @@ class TestLazyJoins(BaseTest):
result = execute_hogql_query(
f"SELECT {select} FROM events AS source "
"JOIN events AS related ON source.person_id = related.person_id "
- "WHERE source.uuid = {event_uuid} AND related.uuid = {event_uuid}",
+ f"WHERE source.uuid = {{event_uuid}} AND related.uuid = {{event_uuid}} {clause}",
team=self.team,
placeholders={"event_uuid": ast.Constant(value=event_uuid)},
modifiers=HogQLQueryModifiers(
diff --git a/posthog/hogql/transforms/test/test_property_types.py b/posthog/hogql/transforms/test/test_property_types.py
index cef29e066c4e..228daabf56d7 100644
--- a/posthog/hogql/transforms/test/test_property_types.py
+++ b/posthog/hogql/transforms/test/test_property_types.py
@@ -140,6 +140,70 @@ class _NewEventsSchemaArraySubcolumnsHelpers:
@override_settings(CLICKHOUSE_HOGQL_USE_NEW_EVENTS_SCHEMA=True)
class TestNewEventsSchemaArraySubcolumns(_NewEventsSchemaArraySubcolumnsHelpers, SimpleTestCase):
+ def test_json_function_rewrites_ignore_property_metadata(self) -> None:
+ printed = self._print_select(
+ "SELECT JSONExtractRaw(properties, 'value') AS value FROM events",
+ event_properties={"value": {"type": "Numeric"}},
+ )
+ assert "events.properties.value" in printed
+ assert "accurateCastOrNull" not in printed
+ assert "JSONExtractKeysAndValuesRaw(events.properties)" not in printed
+
+ @parameterized.expand(
+ [
+ ("float", "JSONExtractFloat(properties, 'value')"),
+ ("typed", "JSONExtract(properties, 'value', 'Nullable(Float64)')"),
+ ]
+ )
+ def test_float_json_functions_do_not_serialize_the_path(self, _name: str, expression: str) -> None:
+ printed = self._print_select(f"SELECT {expression} AS value FROM events")
+ assert "events.properties.value" in printed
+ assert "toJSONString" not in printed
+ assert "JSONExtract" not in printed
+
+ @parameterized.expand(
+ [
+ (name, f"{name}(properties, 'value')")
+ for name in (
+ "JSONLength",
+ "JSONType",
+ "JSONExtractKeys",
+ "JSONExtractArrayRaw",
+ "JSONExtractKeysAndValuesRaw",
+ )
+ ]
+ + [("typed_pairs", "JSONExtractKeysAndValues(properties, 'value', 'String')")]
+ )
+ def test_json_functions_read_only_the_requested_property(self, _name: str, expression: str) -> None:
+ printed = self._print_select(f"SELECT {expression} AS value FROM events")
+ assert "events.properties.^value" in printed
+ assert "JSONExtractKeysAndValuesRaw(events.properties)" not in printed
+ assert "events.temporary_properties" not in printed
+
+ def test_json_string_functions_ignore_property_metadata(self) -> None:
+ context = self._context(use_new_events_schema=True, event_properties={"value": {"type": "Numeric"}})
+ with patch("posthog.hogql.printer.utils.build_property_swapper"):
+ printed, _ = prepare_and_print_ast(
+ parse_select("SELECT JSONExtractString(toString(properties), 'value') AS value FROM events"),
+ context,
+ "clickhouse",
+ )
+ assert "events.properties.value" in printed
+ assert "accurateCastOrNull(events.properties.value, 'Float64')" not in printed
+ assert "JSONExtractKeysAndValuesRaw(events.properties)" not in printed
+
+ def test_json_functions_keep_nested_property_restrictions(self) -> None:
+ printed = self._print_select(
+ "SELECT JSONLength(properties, 'value'), JSONExtractKeys(properties, 'value'), "
+ "JSONExtractString(properties, 'value', 'secret') FROM events",
+ restricted_properties={
+ RestrictedProperty(name="value.secret", property_type=PropertyDefinition.Type.EVENT)
+ },
+ )
+ assert "JSONDropKeys" in printed
+ assert "events.properties.value.secret" not in printed
+ assert "events.properties.^value" not in printed
+
@parameterized.expand(
[
(f"{conversion}_{windowed}", conversion, windowed)
@@ -548,6 +612,59 @@ class TestNewEventsSchemaArraySubcolumns(_NewEventsSchemaArraySubcolumnsHelpers,
@override_settings(CLICKHOUSE_HOGQL_USE_NEW_EVENTS_SCHEMA=True)
@pytest.mark.usefixtures("clickhouse_database")
class TestNewEventsSchemaArraySubcolumnsClickhouse(_NewEventsSchemaArraySubcolumnsHelpers, SimpleTestCase):
+ def test_json_function_conversions_and_defaults_match_the_blob(self) -> None:
+ documents = [
+ "{}",
+ '{"value":null}',
+ '{"value":""}',
+ '{"value":"hello"}',
+ '{"value":"2.5"}',
+ '{"value":"1e3"}',
+ '{"value":true}',
+ '{"value":false}',
+ '{"value":1.5}',
+ '{"value":2.0}',
+ '{"value":1e21}',
+ '{"value":-1.5e-7}',
+ '{"value":-2}',
+ '{"value":0}',
+ '{"value":[1,"two"]}',
+ '{"value":{"name":"example"}}',
+ ]
+ expressions = [
+ "JSONExtractString(properties, 'value')",
+ "JSONExtractInt(properties, 'value')",
+ "JSONExtractUInt(properties, 'value')",
+ "JSONExtractFloat(properties, 'value')",
+ "JSONExtractBool(properties, 'value')",
+ "JSONExtract(properties, 'value', 'Nullable(Int64)')",
+ "JSONExtract(properties, 'value', 'Nullable(Bool)')",
+ "JSONExtract(properties, 'value', 'String')",
+ "JSONExtract(properties, 'value', 'Nullable(String)')",
+ "JSONExtract(properties, 'value', 'Float64')",
+ "JSONExtract(properties, 'value', 'Nullable(Float64)')",
+ "JSONLength(properties, 'value')",
+ "JSONType(properties, 'value')",
+ ]
+ query = "SELECT distinct_id, " + ", ".join(expressions) + " FROM events ORDER BY distinct_id"
+ results = []
+ for native in (False, True):
+ context = self._context(use_new_events_schema=native)
+ context.modifiers.propertyGroupsMode = PropertyGroupsMode.DISABLED
+ context.modifiers.materializationMode = MaterializationMode.DISABLED
+ with patch("posthog.hogql.printer.utils.build_property_swapper"):
+ sql, _ = prepare_and_print_ast(parse_select(query), context, "clickhouse")
+ table = "events_json" if native else "events"
+ properties = "CAST(document, 'JSON')" if native else "document"
+ prefix = (
+ f"WITH {table} AS (SELECT 1 AS team_id, toString(item.1) AS distinct_id, item.2 AS document, "
+ f"{properties} AS properties FROM (SELECT "
+ "arrayJoin(arrayZip(range(length(%(documents)s)), %(documents)s)) AS item)) "
+ )
+ rows = sync_execute(prefix + sql, {**context.values, "documents": documents})
+ results.append(rows)
+ assert results[0] == results[1]
+
@parameterized.expand(
[
(
@@ -1612,9 +1729,8 @@ class TestJSONExtractToMaterializedColumn(ClickhouseTestMixin, BaseTest):
assert "events_json" in printed, printed
assert "properties.`$browser`" in printed, printed
assert "mat_$browser" not in printed, printed
- assert (
- f"JSONExtractString(ifNull({json_dynamic_read_sql(root, ['$browser'], as_json=True)}, " in printed
- ), printed
+ assert f"toString({root}.`$browser`)" in printed, printed
+ assert "JSONExtractString(ifNull(" not in printed, printed
return
assert "mat_$browser" in printed, f"Expected mat_$browser in output, got: {printed}"
@@ -1630,9 +1746,7 @@ class TestJSONExtractToMaterializedColumn(ClickhouseTestMixin, BaseTest):
assert "properties.`$browser`" in printed, printed
assert "properties.`$os`" in printed, printed
assert "mat_" not in printed, printed
- assert printed.count("JSONExtractString(ifNull(if(notEquals(toJSONString(events.properties.^") == 2, (
- printed
- )
+ assert "JSONExtractString(ifNull(" not in printed, printed
return
assert "mat_$browser" in printed, printed
@@ -1663,7 +1777,7 @@ class TestJSONExtractToMaterializedColumn(ClickhouseTestMixin, BaseTest):
if settings.CLICKHOUSE_HOGQL_USE_NEW_EVENTS_SCHEMA:
assert "events_json" in printed, printed
assert "events.properties.typed_json_float" in printed, printed
- assert "JSONExtract(ifNull(" in printed, printed
+ assert "JSONExtract(ifNull(" not in printed, printed
assert "mat_typed_json_float" not in printed, printed
return
@@ -1677,7 +1791,7 @@ class TestJSONExtractToMaterializedColumn(ClickhouseTestMixin, BaseTest):
)
if settings.CLICKHOUSE_HOGQL_USE_NEW_EVENTS_SCHEMA:
assert "events.properties.typed_json_float" in printed, printed
- assert "JSONExtract(ifNull(" in printed, printed
+ assert "JSONExtract(ifNull(" not in printed, printed
assert "mat_typed_json_float" not in printed, printed
return
@@ -1705,7 +1819,7 @@ class TestJSONExtractToMaterializedColumn(ClickhouseTestMixin, BaseTest):
printed = self._print_select("select JSONExtract(properties, '$browser', 'String') from events")
if settings.CLICKHOUSE_HOGQL_USE_NEW_EVENTS_SCHEMA:
assert "events.properties.`$browser`" in printed, printed
- assert "JSONExtract(ifNull(if(notEquals(toJSONString(" in printed, printed
+ assert "JSONExtract(ifNull(if(notEquals(toJSONString(" not in printed, printed
assert "mat_" not in printed, printed
return
@@ -1736,17 +1850,17 @@ class TestJSONExtractToMaterializedColumn(ClickhouseTestMixin, BaseTest):
assert "events.properties.score" in printed, printed
assert "events.properties.email" in printed, printed
assert "JSONExtractInt(ifNull(" in printed, printed
- assert "JSONExtract(ifNull(" in printed, printed
- assert "JSONExtractString(ifNull(" in printed, printed
+ assert "JSONExtract(ifNull(" not in printed, printed
+ assert "JSONExtractString(ifNull(" not in printed, printed
assert "toJSONString(events.properties)" not in printed, printed
- assert "toJSONString(events.properties.email)" in printed, printed
+ assert "toString(events.properties.email)" in printed, printed
@override_settings(CLICKHOUSE_HOGQL_USE_NEW_EVENTS_SCHEMA=True)
def test_new_events_schema_nested_jsonextractstring_uses_string_default(self):
printed = self._print_select("select JSONExtractString(properties, 'metadata', 'score') from events")
- assert "JSONExtractString(ifNull(" in printed, printed
- assert "events.properties.^metadata" in printed, printed
+ assert "JSONExtractString(ifNull(" not in printed, printed
+ assert "events.properties.metadata.score" in printed, printed
assert "%(hogql_val_0)s" in printed, printed
assert "JSONExtractKeysAndValuesRaw" not in printed, printed
@@ -1755,7 +1869,7 @@ class TestJSONExtractToMaterializedColumn(ClickhouseTestMixin, BaseTest):
printed = self._print_select("select JSONExtract(properties, 'score', 'Float64') from events")
assert "events.properties.score" in printed, printed
- assert "JSONExtract(ifNull(" in printed, printed
+ assert "JSONExtract(ifNull(" not in printed, printed
assert "JSONExtractKeysAndValuesRaw" not in printed, printed
@override_settings(CLICKHOUSE_HOGQL_USE_NEW_EVENTS_SCHEMA=True)
@@ -1796,7 +1910,7 @@ class TestJSONExtractToMaterializedColumn(ClickhouseTestMixin, BaseTest):
("bool", "JSONExtractBool(properties, 'flag')", "Bool", "0"),
]
)
- def test_scalar_jsonextract_uses_original_function_on_json_subcolumn(
+ def test_scalar_jsonextract_reads_json_subcolumns_without_parsing(
self, _name: str, expression: str, _cast_type: str, _default_value: str
):
printed = self._print_select(f"select {expression} from events")
@@ -1806,7 +1920,9 @@ class TestJSONExtractToMaterializedColumn(ClickhouseTestMixin, BaseTest):
property_name = "flag" if "flag" in expression else "metric"
assert f"events.properties.{property_name}" in printed, printed
- assert expression.split("(", 1)[0] in printed, printed
+ # Integer extraction keeps ClickHouse's parser on the serialized value; the other functions read the subcolumn.
+ parses = _name in ("int", "uint")
+ assert (f"{expression.split('(', 1)[0]}(ifNull(" in printed) == parses, printed
assert "JSONExtractKeysAndValuesRaw" not in printed, printed
def test_scalar_jsonextract_preserves_clickhouse_coercions(self):
@@ -1919,7 +2035,7 @@ class TestJSONExtractToMaterializedColumn(ClickhouseTestMixin, BaseTest):
if settings.CLICKHOUSE_HOGQL_USE_NEW_EVENTS_SCHEMA:
assert "events_json" in sql, sql
assert "events.properties.`$browser`" in sql, sql
- assert "JSONExtractString(ifNull(if(notEquals(toJSONString(" in sql, sql
+ assert "JSONExtractString(ifNull(if(notEquals(toJSONString(" not in sql, sql
assert "mat_$browser" not in sql, sql
else:
assert "JSONExtractString(events.properties" in sql, sql
diff --git a/posthog/hogql_queries/ai/test/__snapshots__/test_event_taxonomy_query_runner.new_events_schema.ambr b/posthog/hogql_queries/ai/test/__snapshots__/test_event_taxonomy_query_runner.new_events_schema.ambr
index bf97f497f806..136209e366bb 100644
--- a/posthog/hogql_queries/ai/test/__snapshots__/test_event_taxonomy_query_runner.new_events_schema.ambr
+++ b/posthog/hogql_queries/ai/test/__snapshots__/test_event_taxonomy_query_runner.new_events_schema.ambr
@@ -58,10 +58,10 @@
count() AS value_count,
max(timestamp) AS latest_seen
FROM
- (SELECT [tuple('zero_duration_recording_count_in_period', JSONExtractString(ifNull(if(notEquals(toJSONString(events.properties.^zero_duration_recording_count_in_period), '{}'), toJSONString(events.properties.^zero_duration_recording_count_in_period), if(isNull(events.properties.zero_duration_recording_count_in_period), NULL, toJSONString(events.properties.zero_duration_recording_count_in_period))), '')))] AS kv,
+ (SELECT [tuple('zero_duration_recording_count_in_period', ifNull(if(ifNull(equals(dynamicType(accurateCast(events.properties.zero_duration_recording_count_in_period, 'Dynamic')), 'Float64'), 0), JSONExtractString(toJSONString(accurateCast(events.properties.zero_duration_recording_count_in_period, 'Dynamic'))), if(notEquals(toJSONString(events.properties.^zero_duration_recording_count_in_period), '{}'), toJSONString(events.properties.^zero_duration_recording_count_in_period), if(isNull(events.properties.zero_duration_recording_count_in_period), NULL, if(and(in(ascii(toString(events.properties.zero_duration_recording_count_in_period)), tuple(91, 123)), in(ascii(toJSONString(events.properties.zero_duration_recording_count_in_period)), tuple(91, 123))), toJSONString(events.properties.zero_duration_recording_count_in_period), toString(events.properties.zero_duration_recording_count_in_period))))), ''))] AS kv,
toTimeZone(events.timestamp, 'UTC') AS timestamp
FROM events_json AS events
- WHERE and(equals(events.team_id, 99999), greaterOrEquals(events.timestamp, toDateTime64(minus(now64(6, 'UTC'), toIntervalDay(30)), 6, 'UTC')), equals(events.event, 'organization usage report'), notEquals(JSONExtractString(ifNull(if(notEquals(toJSONString(events.properties.^zero_duration_recording_count_in_period), '{}'), toJSONString(events.properties.^zero_duration_recording_count_in_period), if(isNull(events.properties.zero_duration_recording_count_in_period), NULL, toJSONString(events.properties.zero_duration_recording_count_in_period))), '')), ''))) ARRAY
+ WHERE and(equals(events.team_id, 99999), greaterOrEquals(events.timestamp, toDateTime64(minus(now64(6, 'UTC'), toIntervalDay(30)), 6, 'UTC')), equals(events.event, 'organization usage report'), notEquals(ifNull(if(ifNull(equals(dynamicType(accurateCast(events.properties.zero_duration_recording_count_in_period, 'Dynamic')), 'Float64'), 0), JSONExtractString(toJSONString(accurateCast(events.properties.zero_duration_recording_count_in_period, 'Dynamic'))), if(notEquals(toJSONString(events.properties.^zero_duration_recording_count_in_period), '{}'), toJSONString(events.properties.^zero_duration_recording_count_in_period), if(isNull(events.properties.zero_duration_recording_count_in_period), NULL, if(and(in(ascii(toString(events.properties.zero_duration_recording_count_in_period)), tuple(91, 123)), in(ascii(toJSONString(events.properties.zero_duration_recording_count_in_period)), tuple(91, 123))), toJSONString(events.properties.zero_duration_recording_count_in_period), toString(events.properties.zero_duration_recording_count_in_period))))), ''), ''))) ARRAY
JOIN (kv).1 AS key,
(kv).2 AS value
WHERE and(isNotNull(value), ifNull(notEquals(value, ''), 1))
diff --git a/products/surveys/backend/api/test/__snapshots__/test_survey.new_events_schema.ambr b/products/surveys/backend/api/test/__snapshots__/test_survey.new_events_schema.ambr
index 3f273f58cdc9..97498ef0839b 100644
--- a/products/surveys/backend/api/test/__snapshots__/test_survey.new_events_schema.ambr
+++ b/products/surveys/backend/api/test/__snapshots__/test_survey.new_events_schema.ambr
@@ -8,7 +8,7 @@
(SELECT if(notEquals(toJSONString(events.properties.^`$survey_id`), '{}'), toJSONString(events.properties.^`$survey_id`), if(isNull(events.properties.`$survey_id`), NULL, if(and(in(ascii(toString(events.properties.`$survey_id`)), tuple(91, 123)), in(ascii(toJSONString(events.properties.`$survey_id`)), tuple(91, 123))), toJSONString(events.properties.`$survey_id`), toString(events.properties.`$survey_id`)))) AS survey_id,
argMax(events.uuid, tuple(toTimeZone(events.timestamp, 'UTC'), events.uuid)) AS response_uuid
FROM events_json AS events
- WHERE and(equals(events.team_id, 99999), or(equals(events.event, 'survey sent'), and(in(events.event, tuple('survey dismissed', 'survey abandoned')), equals(coalesce(JSONExtractString(ifNull(if(notEquals(toJSONString(events.properties.^`$survey_partially_completed`), '{}'), toJSONString(events.properties.^`$survey_partially_completed`), if(isNull(events.properties.`$survey_partially_completed`), NULL, toJSONString(events.properties.`$survey_partially_completed`))), '')), ''), 'true'))), greaterOrEquals(events.timestamp, toDateTime64('2024-01-21 14:40:09.000000', 6, 'UTC')))
+ WHERE and(equals(events.team_id, 99999), or(equals(events.event, 'survey sent'), and(in(events.event, tuple('survey dismissed', 'survey abandoned')), equals(coalesce(ifNull(if(ifNull(equals(dynamicType(accurateCast(events.properties.`$survey_partially_completed`, 'Dynamic')), 'Float64'), 0), JSONExtractString(toJSONString(accurateCast(events.properties.`$survey_partially_completed`, 'Dynamic'))), if(notEquals(toJSONString(events.properties.^`$survey_partially_completed`), '{}'), toJSONString(events.properties.^`$survey_partially_completed`), if(isNull(events.properties.`$survey_partially_completed`), NULL, if(and(in(ascii(toString(events.properties.`$survey_partially_completed`)), tuple(91, 123)), in(ascii(toJSONString(events.properties.`$survey_partially_completed`)), tuple(91, 123))), toJSONString(events.properties.`$survey_partially_completed`), toString(events.properties.`$survey_partially_completed`))))), ''), ''), 'true'))), greaterOrEquals(events.timestamp, toDateTime64('2024-01-21 14:40:09.000000', 6, 'UTC')))
GROUP BY survey_id,
if(ifNull(equals(coalesce(if(notEquals(toJSONString(events.properties.^`$survey_submission_id`), '{}'), toJSONString(events.properties.^`$survey_submission_id`), if(isNull(events.properties.`$survey_submission_id`), NULL, if(and(in(ascii(toString(events.properties.`$survey_submission_id`)), tuple(91, 123)), in(ascii(toJSONString(events.properties.`$survey_submission_id`)), tuple(91, 123))), toJSONString(events.properties.`$survey_submission_id`), toString(events.properties.`$survey_submission_id`)))), ''), ''), 0), toString(events.uuid), if(notEquals(toJSONString(events.properties.^`$survey_submission_id`), '{}'), toJSONString(events.properties.^`$survey_submission_id`), if(isNull(events.properties.`$survey_submission_id`), NULL, if(and(in(ascii(toString(events.properties.`$survey_submission_id`)), tuple(91, 123)), in(ascii(toJSONString(events.properties.`$survey_submission_id`)), tuple(91, 123))), toJSONString(events.properties.`$survey_submission_id`), toString(events.properties.`$survey_submission_id`))))))
GROUP BY survey_id
@@ -37,7 +37,7 @@
(SELECT if(notEquals(toJSONString(events.properties.^`$survey_id`), '{}'), toJSONString(events.properties.^`$survey_id`), if(isNull(events.properties.`$survey_id`), NULL, if(and(in(ascii(toString(events.properties.`$survey_id`)), tuple(91, 123)), in(ascii(toJSONString(events.properties.`$survey_id`)), tuple(91, 123))), toJSONString(events.properties.`$survey_id`), toString(events.properties.`$survey_id`)))) AS survey_id,
argMax(events.uuid, tuple(toTimeZone(events.timestamp, 'UTC'), events.uuid)) AS response_uuid
FROM events_json AS events
- WHERE and(equals(events.team_id, 99999), or(equals(events.event, 'survey sent'), and(in(events.event, tuple('survey dismissed', 'survey abandoned')), equals(coalesce(JSONExtractString(ifNull(if(notEquals(toJSONString(events.properties.^`$survey_partially_completed`), '{}'), toJSONString(events.properties.^`$survey_partially_completed`), if(isNull(events.properties.`$survey_partially_completed`), NULL, toJSONString(events.properties.`$survey_partially_completed`))), '')), ''), 'true'))), greaterOrEquals(events.timestamp, toDateTime64('2024-04-25 14:40:09.000000', 6, 'UTC')))
+ WHERE and(equals(events.team_id, 99999), or(equals(events.event, 'survey sent'), and(in(events.event, tuple('survey dismissed', 'survey abandoned')), equals(coalesce(ifNull(if(ifNull(equals(dynamicType(accurateCast(events.properties.`$survey_partially_completed`, 'Dynamic')), 'Float64'), 0), JSONExtractString(toJSONString(accurateCast(events.properties.`$survey_partially_completed`, 'Dynamic'))), if(notEquals(toJSONString(events.properties.^`$survey_partially_completed`), '{}'), toJSONString(events.properties.^`$survey_partially_completed`), if(isNull(events.properties.`$survey_partially_completed`), NULL, if(and(in(ascii(toString(events.properties.`$survey_partially_completed`)), tuple(91, 123)), in(ascii(toJSONString(events.properties.`$survey_partially_completed`)), tuple(91, 123))), toJSONString(events.properties.`$survey_partially_completed`), toString(events.properties.`$survey_partially_completed`))))), ''), ''), 'true'))), greaterOrEquals(events.timestamp, toDateTime64('2024-04-25 14:40:09.000000', 6, 'UTC')))
GROUP BY survey_id,
if(ifNull(equals(coalesce(if(notEquals(toJSONString(events.properties.^`$survey_submission_id`), '{}'), toJSONString(events.properties.^`$survey_submission_id`), if(isNull(events.properties.`$survey_submission_id`), NULL, if(and(in(ascii(toString(events.properties.`$survey_submission_id`)), tuple(91, 123)), in(ascii(toJSONString(events.properties.`$survey_submission_id`)), tuple(91, 123))), toJSONString(events.properties.`$survey_submission_id`), toString(events.properties.`$survey_submission_id`)))), ''), ''), 0), toString(events.uuid), if(notEquals(toJSONString(events.properties.^`$survey_submission_id`), '{}'), toJSONString(events.properties.^`$survey_submission_id`), if(isNull(events.properties.`$survey_submission_id`), NULL, if(and(in(ascii(toString(events.properties.`$survey_submission_id`)), tuple(91, 123)), in(ascii(toJSONString(events.properties.`$survey_submission_id`)), tuple(91, 123))), toJSONString(events.properties.`$survey_submission_id`), toString(events.properties.`$survey_submission_id`))))))
GROUP BY survey_id
@@ -66,7 +66,7 @@
(SELECT if(notEquals(toJSONString(events.properties.^`$survey_id`), '{}'), toJSONString(events.properties.^`$survey_id`), if(isNull(events.properties.`$survey_id`), NULL, if(and(in(ascii(toString(events.properties.`$survey_id`)), tuple(91, 123)), in(ascii(toJSONString(events.properties.`$survey_id`)), tuple(91, 123))), toJSONString(events.properties.`$survey_id`), toString(events.properties.`$survey_id`)))) AS survey_id,
argMax(events.uuid, tuple(toTimeZone(events.timestamp, 'UTC'), events.uuid)) AS response_uuid
FROM events_json AS events
- WHERE and(equals(events.team_id, 99999), or(equals(events.event, 'survey sent'), and(in(events.event, tuple('survey dismissed', 'survey abandoned')), equals(coalesce(JSONExtractString(ifNull(if(notEquals(toJSONString(events.properties.^`$survey_partially_completed`), '{}'), toJSONString(events.properties.^`$survey_partially_completed`), if(isNull(events.properties.`$survey_partially_completed`), NULL, toJSONString(events.properties.`$survey_partially_completed`))), '')), ''), 'true'))), greaterOrEquals(events.timestamp, toDateTime64('2024-06-10 11:00:00.000000', 6, 'UTC')))
+ WHERE and(equals(events.team_id, 99999), or(equals(events.event, 'survey sent'), and(in(events.event, tuple('survey dismissed', 'survey abandoned')), equals(coalesce(ifNull(if(ifNull(equals(dynamicType(accurateCast(events.properties.`$survey_partially_completed`, 'Dynamic')), 'Float64'), 0), JSONExtractString(toJSONString(accurateCast(events.properties.`$survey_partially_completed`, 'Dynamic'))), if(notEquals(toJSONString(events.properties.^`$survey_partially_completed`), '{}'), toJSONString(events.properties.^`$survey_partially_completed`), if(isNull(events.properties.`$survey_partially_completed`), NULL, if(and(in(ascii(toString(events.properties.`$survey_partially_completed`)), tuple(91, 123)), in(ascii(toJSONString(events.properties.`$survey_partially_completed`)), tuple(91, 123))), toJSONString(events.properties.`$survey_partially_completed`), toString(events.properties.`$survey_partially_completed`))))), ''), ''), 'true'))), greaterOrEquals(events.timestamp, toDateTime64('2024-06-10 11:00:00.000000', 6, 'UTC')))
GROUP BY survey_id,
if(ifNull(equals(coalesce(if(notEquals(toJSONString(events.properties.^`$survey_submission_id`), '{}'), toJSONString(events.properties.^`$survey_submission_id`), if(isNull(events.properties.`$survey_submission_id`), NULL, if(and(in(ascii(toString(events.properties.`$survey_submission_id`)), tuple(91, 123)), in(ascii(toJSONString(events.properties.`$survey_submission_id`)), tuple(91, 123))), toJSONString(events.properties.`$survey_submission_id`), toString(events.properties.`$survey_submission_id`)))), ''), ''), 0), toString(events.uuid), if(notEquals(toJSONString(events.properties.^`$survey_submission_id`), '{}'), toJSONString(events.properties.^`$survey_submission_id`), if(isNull(events.properties.`$survey_submission_id`), NULL, if(and(in(ascii(toString(events.properties.`$survey_submission_id`)), tuple(91, 123)), in(ascii(toJSONString(events.properties.`$survey_submission_id`)), tuple(91, 123))), toJSONString(events.properties.`$survey_submission_id`), toString(events.properties.`$survey_submission_id`))))))
GROUP BY survey_id