Skip to content

:hedgehog: PostHog is the leading platform for building self-driving products. Our developer tools – AI observability, analytics, session replay, flags, experiments, error tracking, logs, and more – capture all the context agents need to diagnose problems, uncover opportunities, and ship fixes. Steer it all from Slack, web, desktop, or the MCP.

feat(sessions): ship the sessions v3 schema with more ad click ids (#104566)

Lucas Ricoy367aabc54d1a
diff --git a/frontend/src/scenes/web-analytics/contributing.md b/frontend/src/scenes/web-analytics/contributing.md
index c027da7a4ebd..ae212399e34e 100644
--- a/frontend/src/scenes/web-analytics/contributing.md
+++ b/frontend/src/scenes/web-analytics/contributing.md
@@ -58,6 +58,30 @@ The sessions table stores pre-computed session properties essential for web anal
- Feature flag values seen during the session
- Session replay presence
+### How entry and exit values are chosen
+
+Entry and exit values come from page views, not from the first or last event of any kind.
+
+- The entry URL, entry referring domain, UTM parameters, click IDs, ad IDs and channel type inputs come from the session's first `$pageview` or `$screen`.
+- The exit URL comes from the session's last `$pageview` or `$screen`.
+- Other events (autocapture, `$pageleave`, custom events) set these values only when the session has no `$pageview` or `$screen`.
+ The materialized view does this by moving the timestamp of every other event one year later (`pageview_prio_timestamp_min`) before it takes `argMin`.
+- Device and geography values (browser, OS, viewport, country, city) come from the first event that has a value.
+- An empty `gclid` or `fbclid` property does not count as a click ID, so it does not make `$entry_has_gclid` or the channel type say paid.
+
+For the discussion behind the page view rule, see [this thread](https://posthog.slack.com/archives/C05LJK1N3CP/p1748876828748129).
+
+One known inconsistency is accepted:
+
+- `entry_utm_*` and `entry_referring_domain` are separate `argMin` columns over nullable values, and `argMin` skips `NULL`.
+ When the entry page view has no UTM source, `$entry_utm_source` takes the value from a later event in the session.
+- `entry_channel_type_properties`, which `$channel_type` reads, is one tuple that is never `NULL`, so it always holds the entry page view's values.
+- So, for a session that lands without UTMs and gets one later, `$entry_utm_source` shows that later value while `$channel_type` can say Direct.
+
+The materialized views also drop events whose timestamp is more than 31 days from the timestamp in their UUIDv7 session ID.
+A client can send a session ID with a broken timestamp (for example 1970), which would put the session in its own monthly partition and stretch its duration across months.
+The bound is symmetric because the client creates the session ID with its own clock while the server corrects event timestamps, so a client clock that is a few days ahead still gives a real session.
+
### Why session properties are central to web analytics
Session properties enable fast, attribution-focused analysis:
diff --git a/posthog/api/test/test_session.py b/posthog/api/test/test_session.py
index 68ea53776497..46d7078e7e91 100644
--- a/posthog/api/test/test_session.py
+++ b/posthog/api/test/test_session.py
@@ -137,6 +137,11 @@ class TestSessionsAPI(APIBaseTest):
"$entry_utm_source",
"$entry_utm_term",
}
+ if version == "v3":
+ for ad_id in SESSION_V3_LOWER_TIER_AD_IDS:
+ if "utm" in ad_id:
+ expected_properties.add(f"$entry_{ad_id}")
+ expected_properties.add(f"$entry_has_{ad_id}")
assert actual_properties == expected_properties
def test_empty_search_session_properties(self):
diff --git a/posthog/clickhouse/hcl/golden/dev/data.hcl b/posthog/clickhouse/hcl/golden/dev/data.hcl
index 436c085a09a4..b5a186520df2 100644
--- a/posthog/clickhouse/hcl/golden/dev/data.hcl
+++ b/posthog/clickhouse/hcl/golden/dev/data.hcl
@@ -4062,9 +4062,6 @@ database "posthog" {
column "entry_has_fbclid" {
type = "AggregateFunction(argMin, Bool, DateTime64(6, 'UTC'))"
}
- column "entry_ad_ids_map" {
- type = "AggregateFunction(argMin, Map(String, String), DateTime64(6, 'UTC'))"
- }
column "entry_ad_ids_set" {
type = "AggregateFunction(argMin, Array(String), DateTime64(6, 'UTC'))"
}
@@ -4072,13 +4069,13 @@ database "posthog" {
type = "AggregateFunction(argMin, Tuple(Nullable(String), Nullable(String), Nullable(String), Nullable(String), Bool, Bool, Nullable(String)), DateTime64(6, 'UTC'))"
}
column "pageview_uniq" {
- type = "AggregateFunction(uniqExact, Nullable(UUID))"
+ type = "AggregateFunction(uniq, Nullable(UUID))"
}
column "autocapture_uniq" {
- type = "AggregateFunction(uniqExact, Nullable(UUID))"
+ type = "AggregateFunction(uniq, Nullable(UUID))"
}
column "screen_uniq" {
- type = "AggregateFunction(uniqExact, Nullable(UUID))"
+ type = "AggregateFunction(uniq, Nullable(UUID))"
}
column "page_screen_uniq_up_to" {
type = "AggregateFunction(uniqUpTo(1), Nullable(UUID))"
@@ -4089,12 +4086,6 @@ database "posthog" {
column "flag_key_values" {
type = "SimpleAggregateFunction(groupUniqArrayArray(10000), Array(String))"
}
- column "flag_values" {
- type = "AggregateFunction(groupUniqArrayMap, Map(String, String))"
- }
- column "flag_keys" {
- type = "SimpleAggregateFunction(groupUniqArrayArray, Array(String))"
- }
column "event_names" {
type = "SimpleAggregateFunction(groupUniqArrayArray, Array(String))"
}
@@ -6599,9 +6590,6 @@ database "posthog" {
column "entry_has_fbclid" {
type = "AggregateFunction(argMin, Bool, DateTime64(6, 'UTC'))"
}
- column "entry_ad_ids_map" {
- type = "AggregateFunction(argMin, Map(String, String), DateTime64(6, 'UTC'))"
- }
column "entry_ad_ids_set" {
type = "AggregateFunction(argMin, Array(String), DateTime64(6, 'UTC'))"
}
@@ -6609,13 +6597,13 @@ database "posthog" {
type = "AggregateFunction(argMin, Tuple(Nullable(String), Nullable(String), Nullable(String), Nullable(String), Bool, Bool, Nullable(String)), DateTime64(6, 'UTC'))"
}
column "pageview_uniq" {
- type = "AggregateFunction(uniqExact, Nullable(UUID))"
+ type = "AggregateFunction(uniq, Nullable(UUID))"
}
column "autocapture_uniq" {
- type = "AggregateFunction(uniqExact, Nullable(UUID))"
+ type = "AggregateFunction(uniq, Nullable(UUID))"
}
column "screen_uniq" {
- type = "AggregateFunction(uniqExact, Nullable(UUID))"
+ type = "AggregateFunction(uniq, Nullable(UUID))"
}
column "page_screen_uniq_up_to" {
type = "AggregateFunction(uniqUpTo(1), Nullable(UUID))"
@@ -6626,12 +6614,6 @@ database "posthog" {
column "flag_key_values" {
type = "SimpleAggregateFunction(groupUniqArrayArray(10000), Array(String))"
}
- column "flag_values" {
- type = "AggregateFunction(groupUniqArrayMap, Map(String, String))"
- }
- column "flag_keys" {
- type = "SimpleAggregateFunction(groupUniqArrayArray, Array(String))"
- }
column "event_names" {
type = "SimpleAggregateFunction(groupUniqArrayArray, Array(String))"
}
@@ -6649,15 +6631,10 @@ database "posthog" {
type = "bloom_filter()"
granularity = 1
}
- index "flag_keys_bloom_filter" {
- expr = "flag_keys"
- type = "bloom_filter()"
- granularity = 1
- }
- index "flag_key_values_bloom_filter" {
+ index "flag_key_values_text" {
expr = "flag_key_values"
- type = "bloom_filter()"
- granularity = 1
+ type = "text(tokenizer = splitByString(['=']))"
+ granularity = 100000000
}
index "hosts_bloom_filter" {
expr = "hosts"
@@ -9019,9 +8996,6 @@ database "posthog" {
column "entry_has_fbclid" {
type = "AggregateFunction(argMin, Bool, DateTime64(6, 'UTC'))"
}
- column "entry_ad_ids_map" {
- type = "AggregateFunction(argMin, Map(String, String), DateTime64(6, 'UTC'))"
- }
column "entry_ad_ids_set" {
type = "AggregateFunction(argMin, Array(String), DateTime64(6, 'UTC'))"
}
@@ -9029,13 +9003,13 @@ database "posthog" {
type = "AggregateFunction(argMin, Tuple(Nullable(String), Nullable(String), Nullable(String), Nullable(String), Bool, Bool, Nullable(String)), DateTime64(6, 'UTC'))"
}
column "pageview_uniq" {
- type = "AggregateFunction(uniqExact, Nullable(UUID))"
+ type = "AggregateFunction(uniq, Nullable(UUID))"
}
column "autocapture_uniq" {
- type = "AggregateFunction(uniqExact, Nullable(UUID))"
+ type = "AggregateFunction(uniq, Nullable(UUID))"
}
column "screen_uniq" {
- type = "AggregateFunction(uniqExact, Nullable(UUID))"
+ type = "AggregateFunction(uniq, Nullable(UUID))"
}
column "page_screen_uniq_up_to" {
type = "AggregateFunction(uniqUpTo(1), Nullable(UUID))"
@@ -9046,12 +9020,6 @@ database "posthog" {
column "flag_key_values" {
type = "SimpleAggregateFunction(groupUniqArrayArray(10000), Array(String))"
}
- column "flag_values" {
- type = "AggregateFunction(groupUniqArrayMap, Map(String, String))"
- }
- column "flag_keys" {
- type = "SimpleAggregateFunction(groupUniqArrayArray, Array(String))"
- }
column "event_names" {
type = "SimpleAggregateFunction(groupUniqArrayArray, Array(String))"
}
@@ -10590,16 +10558,14 @@ SELECT
argMinMerge(entry_fbclid) AS entry_fbclid,
argMinMerge(entry_has_gclid) AS entry_has_gclid,
argMinMerge(entry_has_fbclid) AS entry_has_fbclid,
- argMinMerge(entry_ad_ids_map) AS entry_ad_ids_map,
argMinMerge(entry_ad_ids_set) AS entry_ad_ids_set,
argMinMerge(entry_channel_type_properties) AS entry_channel_type_properties,
- uniqExactMerge(pageview_uniq) AS pageview_uniq,
- uniqExactMerge(autocapture_uniq) AS autocapture_uniq,
- uniqExactMerge(screen_uniq) AS screen_uniq,
+ uniqMerge(pageview_uniq) AS pageview_uniq,
+ uniqMerge(autocapture_uniq) AS autocapture_uniq,
+ uniqMerge(screen_uniq) AS screen_uniq,
uniqUpToMerge(1)(page_screen_uniq_up_to) AS page_screen_uniq_up_to,
max(has_autocapture) AS has_autocapture,
groupUniqArrayArray(10000)(flag_key_values) AS flag_key_values,
- groupUniqArrayArray(flag_keys) AS flag_keys,
groupUniqArrayArray(2000)(event_names) AS event_names,
groupUniqArrayArray(100)(hosts) AS hosts,
groupUniqArrayArray(10)(emails) AS emails,
diff --git a/posthog/clickhouse/hcl/golden/dev/sessions.hcl b/posthog/clickhouse/hcl/golden/dev/sessions.hcl
index 7e1fe6893464..160b935370be 100644
--- a/posthog/clickhouse/hcl/golden/dev/sessions.hcl
+++ b/posthog/clickhouse/hcl/golden/dev/sessions.hcl
@@ -1010,9 +1010,6 @@ database "posthog" {
column "entry_has_fbclid" {
type = "AggregateFunction(argMin, Bool, DateTime64(6, 'UTC'))"
}
- column "entry_ad_ids_map" {
- type = "AggregateFunction(argMin, Map(String, String), DateTime64(6, 'UTC'))"
- }
column "entry_ad_ids_set" {
type = "AggregateFunction(argMin, Array(String), DateTime64(6, 'UTC'))"
}
@@ -1020,13 +1017,13 @@ database "posthog" {
type = "AggregateFunction(argMin, Tuple(Nullable(String), Nullable(String), Nullable(String), Nullable(String), Bool, Bool, Nullable(String)), DateTime64(6, 'UTC'))"
}
column "pageview_uniq" {
- type = "AggregateFunction(uniqExact, Nullable(UUID))"
+ type = "AggregateFunction(uniq, Nullable(UUID))"
}
column "autocapture_uniq" {
- type = "AggregateFunction(uniqExact, Nullable(UUID))"
+ type = "AggregateFunction(uniq, Nullable(UUID))"
}
column "screen_uniq" {
- type = "AggregateFunction(uniqExact, Nullable(UUID))"
+ type = "AggregateFunction(uniq, Nullable(UUID))"
}
column "page_screen_uniq_up_to" {
type = "AggregateFunction(uniqUpTo(1), Nullable(UUID))"
@@ -1037,12 +1034,6 @@ database "posthog" {
column "flag_key_values" {
type = "SimpleAggregateFunction(groupUniqArrayArray(10000), Array(String))"
}
- column "flag_values" {
- type = "AggregateFunction(groupUniqArrayMap, Map(String, String))"
- }
- column "flag_keys" {
- type = "SimpleAggregateFunction(groupUniqArrayArray, Array(String))"
- }
column "event_names" {
type = "SimpleAggregateFunction(groupUniqArrayArray, Array(String))"
}
@@ -1060,15 +1051,10 @@ database "posthog" {
type = "bloom_filter()"
granularity = 1
}
- index "flag_keys_bloom_filter" {
- expr = "flag_keys"
- type = "bloom_filter()"
- granularity = 1
- }
- index "flag_key_values_bloom_filter" {
+ index "flag_key_values_text" {
expr = "flag_key_values"
- type = "bloom_filter()"
- granularity = 1
+ type = "text(tokenizer = splitByString(['=']))"
+ granularity = 100000000
}
engine "replicated_aggregating_merge_tree" {
zoo_path = "/clickhouse/tables/sessions/noshard/posthog.raw_sessions_v3"
diff --git a/posthog/clickhouse/hcl/golden/local-multi/data.hcl b/posthog/clickhouse/hcl/golden/local-multi/data.hcl
index 8dc5fb3408e6..6b49ca428ab1 100644
--- a/posthog/clickhouse/hcl/golden/local-multi/data.hcl
+++ b/posthog/clickhouse/hcl/golden/local-multi/data.hcl
@@ -4421,9 +4421,6 @@ database "posthog" {
column "entry_has_fbclid" {
type = "AggregateFunction(argMin, Bool, DateTime64(6, 'UTC'))"
}
- column "entry_ad_ids_map" {
- type = "AggregateFunction(argMin, Map(String, String), DateTime64(6, 'UTC'))"
- }
column "entry_ad_ids_set" {
type = "AggregateFunction(argMin, Array(String), DateTime64(6, 'UTC'))"
}
@@ -4431,13 +4428,13 @@ database "posthog" {
type = "AggregateFunction(argMin, Tuple(Nullable(String), Nullable(String), Nullable(String), Nullable(String), Bool, Bool, Nullable(String)), DateTime64(6, 'UTC'))"
}
column "pageview_uniq" {
- type = "AggregateFunction(uniqExact, Nullable(UUID))"
+ type = "AggregateFunction(uniq, Nullable(UUID))"
}
column "autocapture_uniq" {
- type = "AggregateFunction(uniqExact, Nullable(UUID))"
+ type = "AggregateFunction(uniq, Nullable(UUID))"
}
column "screen_uniq" {
- type = "AggregateFunction(uniqExact, Nullable(UUID))"
+ type = "AggregateFunction(uniq, Nullable(UUID))"
}
column "page_screen_uniq_up_to" {
type = "AggregateFunction(uniqUpTo(1), Nullable(UUID))"
@@ -4448,9 +4445,6 @@ database "posthog" {
column "flag_key_values" {
type = "SimpleAggregateFunction(groupUniqArrayArray(10000), Array(String))"
}
- column "flag_keys" {
- type = "SimpleAggregateFunction(groupUniqArrayArray, Array(String))"
- }
column "event_names" {
type = "SimpleAggregateFunction(groupUniqArrayArray(2000), Array(String))"
}
@@ -6869,9 +6863,6 @@ database "posthog" {
column "entry_has_fbclid" {
type = "AggregateFunction(argMin, Bool, DateTime64(6, 'UTC'))"
}
- column "entry_ad_ids_map" {
- type = "AggregateFunction(argMin, Map(String, String), DateTime64(6, 'UTC'))"
- }
column "entry_ad_ids_set" {
type = "AggregateFunction(argMin, Array(String), DateTime64(6, 'UTC'))"
}
@@ -6879,13 +6870,13 @@ database "posthog" {
type = "AggregateFunction(argMin, Tuple(Nullable(String), Nullable(String), Nullable(String), Nullable(String), Bool, Bool, Nullable(String)), DateTime64(6, 'UTC'))"
}
column "pageview_uniq" {
- type = "AggregateFunction(uniqExact, Nullable(UUID))"
+ type = "AggregateFunction(uniq, Nullable(UUID))"
}
column "autocapture_uniq" {
- type = "AggregateFunction(uniqExact, Nullable(UUID))"
+ type = "AggregateFunction(uniq, Nullable(UUID))"
}
column "screen_uniq" {
- type = "AggregateFunction(uniqExact, Nullable(UUID))"
+ type = "AggregateFunction(uniq, Nullable(UUID))"
}
column "page_screen_uniq_up_to" {
type = "AggregateFunction(uniqUpTo(1), Nullable(UUID))"
@@ -6896,9 +6887,6 @@ database "posthog" {
column "flag_key_values" {
type = "SimpleAggregateFunction(groupUniqArrayArray(10000), Array(String))"
}
- column "flag_keys" {
- type = "SimpleAggregateFunction(groupUniqArrayArray, Array(String))"
- }
column "event_names" {
type = "SimpleAggregateFunction(groupUniqArrayArray(2000), Array(String))"
}
@@ -6916,15 +6904,10 @@ database "posthog" {
type = "bloom_filter()"
granularity = 1
}
- index "flag_key_values_bloom_filter" {
+ index "flag_key_values_text" {
expr = "flag_key_values"
- type = "bloom_filter()"
- granularity = 1
- }
- index "flag_keys_bloom_filter" {
- expr = "flag_keys"
- type = "bloom_filter()"
- granularity = 1
+ type = "text(tokenizer = splitByString(['=']))"
+ granularity = 100000000
}
index "hosts_bloom_filter" {
expr = "hosts"
@@ -9488,9 +9471,6 @@ database "posthog" {
column "entry_has_fbclid" {
type = "AggregateFunction(argMin, Bool, DateTime64(6, 'UTC'))"
}
- column "entry_ad_ids_map" {
- type = "AggregateFunction(argMin, Map(String, String), DateTime64(6, 'UTC'))"
- }
column "entry_ad_ids_set" {
type = "AggregateFunction(argMin, Array(String), DateTime64(6, 'UTC'))"
}
@@ -9498,13 +9478,13 @@ database "posthog" {
type = "AggregateFunction(argMin, Tuple(Nullable(String), Nullable(String), Nullable(String), Nullable(String), Bool, Bool, Nullable(String)), DateTime64(6, 'UTC'))"
}
column "pageview_uniq" {
- type = "AggregateFunction(uniqExact, Nullable(UUID))"
+ type = "AggregateFunction(uniq, Nullable(UUID))"
}
column "autocapture_uniq" {
- type = "AggregateFunction(uniqExact, Nullable(UUID))"
+ type = "AggregateFunction(uniq, Nullable(UUID))"
}
column "screen_uniq" {
- type = "AggregateFunction(uniqExact, Nullable(UUID))"
+ type = "AggregateFunction(uniq, Nullable(UUID))"
}
column "page_screen_uniq_up_to" {
type = "AggregateFunction(uniqUpTo(1), Nullable(UUID))"
@@ -9515,9 +9495,6 @@ database "posthog" {
column "flag_key_values" {
type = "SimpleAggregateFunction(groupUniqArrayArray(10000), Array(String))"
}
- column "flag_keys" {
- type = "SimpleAggregateFunction(groupUniqArrayArray, Array(String))"
- }
column "event_names" {
type = "SimpleAggregateFunction(groupUniqArrayArray(2000), Array(String))"
}
@@ -11320,6 +11297,507 @@ SQL
}
}
+ materialized_view "raw_sessions_v3_mv" {
+ to_table = "posthog.writable_raw_sessions_v3"
+ query = <<SQL
+WITH
+ JSONExtract(properties, 'Tuple(\n `$current_url` Nullable(String),\n `$external_click_url` Nullable(String),\n `$browser` Nullable(String),\n `$browser_version` Nullable(String),\n `$os` Nullable(String),\n `$os_version` Nullable(String),\n `$device_type` Nullable(String),\n `$viewport_width` Nullable(Int64),\n `$viewport_height` Nullable(Int64),\n `$geoip_country_code` Nullable(String),\n `$geoip_subdivision_1_code` Nullable(String),\n `$geoip_subdivision_1_name` Nullable(String),\n `$geoip_city_name` Nullable(String),\n `$geoip_time_zone` Nullable(String),\n `$referring_domain` Nullable(String),\n `utm_source` Nullable(String),\n `utm_campaign` Nullable(String),\n `utm_medium` Nullable(String),\n `utm_term` Nullable(String),\n `utm_content` Nullable(String),\n `gclid` Nullable(String),\n `gad_source` Nullable(String),\n `fbclid` Nullable(String),\n `$host` Nullable(String),\n `gclsrc` Nullable(String),\n `dclid` Nullable(String),\n `gbraid` Nullable(String),\n `wbraid` Nullable(String),\n `gad_campaignid` Nullable(String),\n `srsltid` Nullable(String),\n `msclkid` Nullable(String),\n `twclid` Nullable(String),\n `li_fat_id` Nullable(String),\n `mc_cid` Nullable(String),\n `igshid` Nullable(String),\n `ttclid` Nullable(String),\n `rdt_cid` Nullable(String),\n `epik` Nullable(String),\n `qclid` Nullable(String),\n `sccid` Nullable(String),\n `_kx` Nullable(String),\n `irclid` Nullable(String),\n `irclickid` Nullable(String),\n `utm_id` Nullable(String),\n `utm_source_platform` Nullable(String),\n `utm_creative_format` Nullable(String),\n `utm_marketing_tactic` Nullable(String),\n `yclid` Nullable(String),\n `tblci` Nullable(String),\n `dicbo` Nullable(String),\n `ndclid` Nullable(String),\n `cjevent` Nullable(String),\n `awc` Nullable(String),\n `syclid` Nullable(String),\n `_bhlid` Nullable(String),\n `rtid` Nullable(String),\n `oppref` Nullable(String),\n `bfcid` Nullable(String)\n )') AS p,
+ JSONExtractString(person_properties, 'email') AS _person_email,
+ tupleElement(p, '$current_url') AS _current_url,
+ if((event = '$pageview') OR (event = '$screen'), left(coalesce(_current_url, ''), 16384), '') AS _attribution_url,
+ tupleElement(p, '$external_click_url') AS _external_click_url,
+ tupleElement(p, '$browser') AS _browser,
+ tupleElement(p, '$browser_version') AS _browser_version,
+ tupleElement(p, '$os') AS _os,
+ tupleElement(p, '$os_version') AS _os_version,
+ tupleElement(p, '$device_type') AS _device_type,
+ tupleElement(p, '$viewport_width') AS _viewport_width,
+ tupleElement(p, '$viewport_height') AS _viewport_height,
+ tupleElement(p, '$geoip_country_code') AS _geoip_country_code,
+ tupleElement(p, '$geoip_subdivision_1_code') AS _geoip_subdivision_1_code,
+ tupleElement(p, '$geoip_subdivision_1_name') AS _geoip_subdivision_1_name,
+ tupleElement(p, '$geoip_city_name') AS _geoip_subdivision_city_name,
+ tupleElement(p, '$geoip_time_zone') AS _geoip_time_zone,
+ tupleElement(p, '$referring_domain') AS _referring_domain,
+ tupleElement(p, 'utm_source') AS _utm_source,
+ tupleElement(p, 'utm_campaign') AS _utm_campaign,
+ tupleElement(p, 'utm_medium') AS _utm_medium,
+ tupleElement(p, 'utm_term') AS _utm_term,
+ tupleElement(p, 'utm_content') AS _utm_content,
+ tupleElement(p, 'gclid') AS _gclid,
+ tupleElement(p, 'gad_source') AS _gad_source,
+ tupleElement(p, 'fbclid') AS _fbclid,
+ coalesce(nullIf(tupleElement(p, 'gclsrc'), ''), nullIf(toValidUTF8(left(decodeURLFormComponent(extractURLParameter(_attribution_url, 'gclsrc')), 2048)), '')) AS gclsrc,
+ coalesce(nullIf(tupleElement(p, 'dclid'), ''), nullIf(toValidUTF8(left(decodeURLFormComponent(extractURLParameter(_attribution_url, 'dclid')), 2048)), '')) AS dclid,
+ coalesce(nullIf(tupleElement(p, 'gbraid'), ''), nullIf(toValidUTF8(left(decodeURLFormComponent(extractURLParameter(_attribution_url, 'gbraid')), 2048)), '')) AS gbraid,
+ coalesce(nullIf(tupleElement(p, 'wbraid'), ''), nullIf(toValidUTF8(left(decodeURLFormComponent(extractURLParameter(_attribution_url, 'wbraid')), 2048)), '')) AS wbraid,
+ coalesce(nullIf(tupleElement(p, 'gad_campaignid'), ''), nullIf(toValidUTF8(left(decodeURLFormComponent(extractURLParameter(_attribution_url, 'gad_campaignid')), 2048)), '')) AS gad_campaignid,
+ coalesce(nullIf(tupleElement(p, 'srsltid'), ''), nullIf(toValidUTF8(left(decodeURLFormComponent(extractURLParameter(_attribution_url, 'srsltid')), 2048)), '')) AS srsltid,
+ coalesce(nullIf(tupleElement(p, 'msclkid'), ''), nullIf(toValidUTF8(left(decodeURLFormComponent(extractURLParameter(_attribution_url, 'msclkid')), 2048)), '')) AS msclkid,
+ coalesce(nullIf(tupleElement(p, 'twclid'), ''), nullIf(toValidUTF8(left(decodeURLFormComponent(extractURLParameter(_attribution_url, 'twclid')), 2048)), '')) AS twclid,
+ coalesce(nullIf(tupleElement(p, 'li_fat_id'), ''), nullIf(toValidUTF8(left(decodeURLFormComponent(extractURLParameter(_attribution_url, 'li_fat_id')), 2048)), '')) AS li_fat_id,
+ coalesce(nullIf(tupleElement(p, 'mc_cid'), ''), nullIf(toValidUTF8(left(decodeURLFormComponent(extractURLParameter(_attribution_url, 'mc_cid')), 2048)), '')) AS mc_cid,
+ coalesce(nullIf(tupleElement(p, 'igshid'), ''), nullIf(toValidUTF8(left(decodeURLFormComponent(extractURLParameter(_attribution_url, 'igshid')), 2048)), '')) AS igshid,
+ coalesce(nullIf(tupleElement(p, 'ttclid'), ''), nullIf(toValidUTF8(left(decodeURLFormComponent(extractURLParameter(_attribution_url, 'ttclid')), 2048)), '')) AS ttclid,
+ coalesce(nullIf(tupleElement(p, 'rdt_cid'), ''), nullIf(toValidUTF8(left(decodeURLFormComponent(extractURLParameter(_attribution_url, 'rdt_cid')), 2048)), '')) AS rdt_cid,
+ coalesce(nullIf(tupleElement(p, 'epik'), ''), nullIf(toValidUTF8(left(decodeURLFormComponent(extractURLParameter(_attribution_url, 'epik')), 2048)), '')) AS epik,
+ coalesce(nullIf(tupleElement(p, 'qclid'), ''), nullIf(toValidUTF8(left(decodeURLFormComponent(extractURLParameter(_attribution_url, 'qclid')), 2048)), '')) AS qclid,
+ coalesce(nullIf(tupleElement(p, 'sccid'), ''), nullIf(toValidUTF8(left(decodeURLFormComponent(extractURLParameter(_attribution_url, 'sccid')), 2048)), ''), nullIf(toValidUTF8(left(decodeURLFormComponent(extractURLParameter(_attribution_url, 'ScCid')), 2048)), ''), nullIf(toValidUTF8(left(decodeURLFormComponent(extractURLParameter(_attribution_url, 'SsCid')), 2048)), '')) AS sccid,
+ coalesce(nullIf(tupleElement(p, '_kx'), ''), nullIf(toValidUTF8(left(decodeURLFormComponent(extractURLParameter(_attribution_url, '_kx')), 2048)), '')) AS _kx,
+ coalesce(nullIf(tupleElement(p, 'irclid'), ''), nullIf(toValidUTF8(left(decodeURLFormComponent(extractURLParameter(_attribution_url, 'irclid')), 2048)), '')) AS irclid,
+ coalesce(nullIf(tupleElement(p, 'irclickid'), ''), nullIf(toValidUTF8(left(decodeURLFormComponent(extractURLParameter(_attribution_url, 'irclickid')), 2048)), '')) AS irclickid,
+ coalesce(nullIf(tupleElement(p, 'utm_id'), ''), nullIf(toValidUTF8(left(decodeURLFormComponent(extractURLParameter(_attribution_url, 'utm_id')), 2048)), '')) AS utm_id,
+ coalesce(nullIf(tupleElement(p, 'utm_source_platform'), ''), nullIf(toValidUTF8(left(decodeURLFormComponent(extractURLParameter(_attribution_url, 'utm_source_platform')), 2048)), '')) AS utm_source_platform,
+ coalesce(nullIf(tupleElement(p, 'utm_creative_format'), ''), nullIf(toValidUTF8(left(decodeURLFormComponent(extractURLParameter(_attribution_url, 'utm_creative_format')), 2048)), '')) AS utm_creative_format,
+ coalesce(nullIf(tupleElement(p, 'utm_marketing_tactic'), ''), nullIf(toValidUTF8(left(decodeURLFormComponent(extractURLParameter(_attribution_url, 'utm_marketing_tactic')), 2048)), '')) AS utm_marketing_tactic,
+ coalesce(nullIf(tupleElement(p, 'yclid'), ''), nullIf(toValidUTF8(left(decodeURLFormComponent(extractURLParameter(_attribution_url, 'yclid')), 2048)), '')) AS yclid,
+ coalesce(nullIf(tupleElement(p, 'tblci'), ''), nullIf(toValidUTF8(left(decodeURLFormComponent(extractURLParameter(_attribution_url, 'tblci')), 2048)), '')) AS tblci,
+ coalesce(nullIf(tupleElement(p, 'dicbo'), ''), nullIf(toValidUTF8(left(decodeURLFormComponent(extractURLParameter(_attribution_url, 'dicbo')), 2048)), '')) AS dicbo,
+ coalesce(nullIf(tupleElement(p, 'ndclid'), ''), nullIf(toValidUTF8(left(decodeURLFormComponent(extractURLParameter(_attribution_url, 'ndclid')), 2048)), '')) AS ndclid,
+ coalesce(nullIf(tupleElement(p, 'cjevent'), ''), nullIf(toValidUTF8(left(decodeURLFormComponent(extractURLParameter(_attribution_url, 'cjevent')), 2048)), '')) AS cjevent,
+ coalesce(nullIf(tupleElement(p, 'awc'), ''), nullIf(toValidUTF8(left(decodeURLFormComponent(extractURLParameter(_attribution_url, 'awc')), 2048)), '')) AS awc,
+ coalesce(nullIf(tupleElement(p, 'syclid'), ''), nullIf(toValidUTF8(left(decodeURLFormComponent(extractURLParameter(_attribution_url, 'syclid')), 2048)), '')) AS syclid,
+ coalesce(nullIf(tupleElement(p, '_bhlid'), ''), nullIf(toValidUTF8(left(decodeURLFormComponent(extractURLParameter(_attribution_url, '_bhlid')), 2048)), '')) AS _bhlid,
+ coalesce(nullIf(tupleElement(p, 'rtid'), ''), nullIf(toValidUTF8(left(decodeURLFormComponent(extractURLParameter(_attribution_url, 'rtid')), 2048)), '')) AS rtid,
+ coalesce(nullIf(tupleElement(p, 'oppref'), ''), nullIf(toValidUTF8(left(decodeURLFormComponent(extractURLParameter(_attribution_url, 'oppref')), 2048)), '')) AS oppref,
+ coalesce(nullIf(tupleElement(p, 'bfcid'), ''), nullIf(toValidUTF8(left(decodeURLFormComponent(extractURLParameter(_attribution_url, 'bfcid')), 2048)), '')) AS bfcid,
+ CAST(arrayFilter(x -> (x IS NOT NULL), [if(gclsrc IS NOT NULL, concat('gclsrc=', gclsrc), NULL), if(dclid IS NOT NULL, concat('dclid=', dclid), NULL), if(gbraid IS NOT NULL, concat('gbraid=', gbraid), NULL), if(wbraid IS NOT NULL, concat('wbraid=', wbraid), NULL), if(gad_campaignid IS NOT NULL, concat('gad_campaignid=', gad_campaignid), NULL), if(srsltid IS NOT NULL, concat('srsltid=', srsltid), NULL), if(msclkid IS NOT NULL, concat('msclkid=', msclkid), NULL), if(twclid IS NOT NULL, concat('twclid=', twclid), NULL), if(li_fat_id IS NOT NULL, concat('li_fat_id=', li_fat_id), NULL), if(mc_cid IS NOT NULL, concat('mc_cid=', mc_cid), NULL), if(igshid IS NOT NULL, concat('igshid=', igshid), NULL), if(ttclid IS NOT NULL, concat('ttclid=', ttclid), NULL), if(rdt_cid IS NOT NULL, concat('rdt_cid=', rdt_cid), NULL), if(epik IS NOT NULL, concat('epik=', epik), NULL), if(qclid IS NOT NULL, concat('qclid=', qclid), NULL), if(sccid IS NOT NULL, concat('sccid=', sccid), NULL), if(_kx IS NOT NULL, concat('_kx=', _kx), NULL), if(irclid IS NOT NULL, concat('irclid=', irclid), NULL), if(irclickid IS NOT NULL, concat('irclickid=', irclickid), NULL), if(utm_id IS NOT NULL, concat('utm_id=', utm_id), NULL), if(utm_source_platform IS NOT NULL, concat('utm_source_platform=', utm_source_platform), NULL), if(utm_creative_format IS NOT NULL, concat('utm_creative_format=', utm_creative_format), NULL), if(utm_marketing_tactic IS NOT NULL, concat('utm_marketing_tactic=', utm_marketing_tactic), NULL), if(yclid IS NOT NULL, concat('yclid=', yclid), NULL), if(tblci IS NOT NULL, concat('tblci=', tblci), NULL), if(dicbo IS NOT NULL, concat('dicbo=', dicbo), NULL), if(ndclid IS NOT NULL, concat('ndclid=', ndclid), NULL), if(cjevent IS NOT NULL, concat('cjevent=', cjevent), NULL), if(awc IS NOT NULL, concat('awc=', awc), NULL), if(syclid IS NOT NULL, concat('syclid=', syclid), NULL), if(_bhlid IS NOT NULL, concat('_bhlid=', _bhlid), NULL), if(rtid IS NOT NULL, concat('rtid=', rtid), NULL), if(oppref IS NOT NULL, concat('oppref=', oppref), NULL), if(bfcid IS NOT NULL, concat('bfcid=', bfcid), NULL)]), 'Array(String)') AS ad_ids_set,
+ tupleElement(p, '$host') AS _host,
+ if((event = '$pageview') OR (event = '$screen'), timestamp, timestamp + toIntervalYear(1)) AS pageview_prio_timestamp_min,
+ if((event = '$pageview') OR (event = '$screen'), timestamp, timestamp - toIntervalYear(1)) AS pageview_prio_timestamp_max
+SELECT
+ team_id,
+ `$session_id_uuid` AS session_id_v7,
+ initializeAggregation('argMaxState', source_table.distinct_id, timestamp) AS distinct_id,
+ initializeAggregation('groupUniqArrayState', source_table.distinct_id) AS distinct_ids,
+ timestamp AS min_timestamp,
+ timestamp AS max_timestamp,
+ inserted_at AS max_inserted_at,
+ if(
+ (_current_url IS NOT NULL) AND ((event = '$pageview') OR (event = '$screen')),
+ [_current_url],
+ []
+ ) AS urls,
+ initializeAggregation('argMinState', _current_url, pageview_prio_timestamp_min) AS entry_url,
+ initializeAggregation('argMaxState', _current_url, pageview_prio_timestamp_max) AS end_url,
+ initializeAggregation('argMaxState', _external_click_url, timestamp) AS last_external_click_url,
+ initializeAggregation('argMinState', _browser, timestamp) AS browser,
+ initializeAggregation('argMinState', _browser_version, timestamp) AS browser_version,
+ initializeAggregation('argMinState', _os, timestamp) AS os,
+ initializeAggregation('argMinState', _os_version, timestamp) AS os_version,
+ initializeAggregation('argMinState', _device_type, timestamp) AS device_type,
+ initializeAggregation('argMinState', _viewport_width, timestamp) AS viewport_width,
+ initializeAggregation('argMinState', _viewport_height, timestamp) AS viewport_height,
+ initializeAggregation('argMinState', _geoip_country_code, timestamp) AS geoip_country_code,
+ initializeAggregation('argMinState', _geoip_subdivision_1_code, timestamp) AS geoip_subdivision_1_code,
+ initializeAggregation('argMinState', _geoip_subdivision_1_name, timestamp) AS geoip_subdivision_1_name,
+ initializeAggregation('argMinState', _geoip_subdivision_city_name, timestamp) AS geoip_subdivision_city_name,
+ initializeAggregation('argMinState', _geoip_time_zone, timestamp) AS geoip_time_zone,
+ initializeAggregation('argMinState', _referring_domain, pageview_prio_timestamp_min) AS entry_referring_domain,
+ initializeAggregation('argMinState', _utm_source, pageview_prio_timestamp_min) AS entry_utm_source,
+ initializeAggregation('argMinState', _utm_campaign, pageview_prio_timestamp_min) AS entry_utm_campaign,
+ initializeAggregation('argMinState', _utm_medium, pageview_prio_timestamp_min) AS entry_utm_medium,
+ initializeAggregation('argMinState', _utm_term, pageview_prio_timestamp_min) AS entry_utm_term,
+ initializeAggregation('argMinState', _utm_content, pageview_prio_timestamp_min) AS entry_utm_content,
+ initializeAggregation('argMinState', _gclid, pageview_prio_timestamp_min) AS entry_gclid,
+ initializeAggregation('argMinState', _gad_source, pageview_prio_timestamp_min) AS entry_gad_source,
+ initializeAggregation('argMinState', _fbclid, pageview_prio_timestamp_min) AS entry_fbclid,
+ initializeAggregation('argMinState', nullIf(_gclid, '') IS NOT NULL, pageview_prio_timestamp_min) AS entry_has_gclid,
+ initializeAggregation('argMinState', nullIf(_fbclid, '') IS NOT NULL, pageview_prio_timestamp_min) AS entry_has_fbclid,
+ initializeAggregation('argMinState', ad_ids_set, pageview_prio_timestamp_min) AS entry_ad_ids_set,
+ initializeAggregation(
+ 'argMinState',
+ tuple(
+ _utm_source,
+ _utm_medium,
+ _utm_campaign,
+ _referring_domain,
+ nullIf(_gclid, '') IS NOT NULL,
+ nullIf(_fbclid, '') IS NOT NULL,
+ _gad_source
+ ),
+ pageview_prio_timestamp_min
+ ) AS entry_channel_type_properties,
+ initializeAggregation('uniqState', if(event = '$pageview', uuid, NULL)) AS pageview_uniq,
+ initializeAggregation('uniqState', if(event = '$autocapture', uuid, NULL)) AS autocapture_uniq,
+ initializeAggregation('uniqState', if(event = '$screen', uuid, NULL)) AS screen_uniq,
+ initializeAggregation(
+ 'uniqUpToState(1)',
+ if((event = '$pageview') OR (event = '$screen'), uuid, NULL)
+ ) AS page_screen_uniq_up_to,
+ event = '$autocapture' AS has_autocapture,
+ arrayMap(
+ (k, v) -> concat(k, '=', v),
+ mapKeys(properties_group_feature_flags),
+ mapValues(properties_group_feature_flags)
+ ) AS flag_key_values,
+ [event] AS event_names,
+ if((_host IS NOT NULL) AND (_host != ''), [_host], []) AS hosts,
+ if((_person_email IS NOT NULL) AND (_person_email != ''), [_person_email], []) AS emails,
+ false AS has_replay_events
+FROM posthog.sharded_events AS source_table
+WHERE
+ (bitAnd(bitShiftRight(toUInt128(accurateCastOrNull(`$session_id`, 'UUID')), 76), 15) = 7)
+AND
+ ((fromUnixTimestamp64Milli(toUInt64(bitShiftRight(`$session_id_uuid`, 80))) >= (timestamp - toIntervalDay(31))) AND (fromUnixTimestamp64Milli(toUInt64(bitShiftRight(`$session_id_uuid`, 80))) <= (timestamp + toIntervalDay(31))))
+AND
+ true
+SQL
+
+ column "team_id" {
+ type = "Int64"
+ }
+ column "session_id_v7" {
+ type = "Nullable(UInt128)"
+ }
+ column "distinct_id" {
+ type = "AggregateFunction(argMax, String, DateTime64(6, 'UTC'))"
+ }
+ column "distinct_ids" {
+ type = "AggregateFunction(groupUniqArray, String)"
+ }
+ column "min_timestamp" {
+ type = "DateTime64(6, 'UTC')"
+ }
+ column "max_timestamp" {
+ type = "DateTime64(6, 'UTC')"
+ }
+ column "max_inserted_at" {
+ type = "Nullable(DateTime64(6, 'UTC'))"
+ }
+ column "urls" {
+ type = "Array(Nullable(String))"
+ }
+ column "entry_url" {
+ type = "AggregateFunction(argMin, Nullable(String), DateTime64(6, 'UTC'))"
+ }
+ column "end_url" {
+ type = "AggregateFunction(argMax, Nullable(String), DateTime64(6, 'UTC'))"
+ }
+ column "last_external_click_url" {
+ type = "AggregateFunction(argMax, Nullable(String), DateTime64(6, 'UTC'))"
+ }
+ column "browser" {
+ type = "AggregateFunction(argMin, Nullable(String), DateTime64(6, 'UTC'))"
+ }
+ column "browser_version" {
+ type = "AggregateFunction(argMin, Nullable(String), DateTime64(6, 'UTC'))"
+ }
+ column "os" {
+ type = "AggregateFunction(argMin, Nullable(String), DateTime64(6, 'UTC'))"
+ }
+ column "os_version" {
+ type = "AggregateFunction(argMin, Nullable(String), DateTime64(6, 'UTC'))"
+ }
+ column "device_type" {
+ type = "AggregateFunction(argMin, Nullable(String), DateTime64(6, 'UTC'))"
+ }
+ column "viewport_width" {
+ type = "AggregateFunction(argMin, Nullable(Int64), DateTime64(6, 'UTC'))"
+ }
+ column "viewport_height" {
+ type = "AggregateFunction(argMin, Nullable(Int64), DateTime64(6, 'UTC'))"
+ }
+ column "geoip_country_code" {
+ type = "AggregateFunction(argMin, Nullable(String), DateTime64(6, 'UTC'))"
+ }
+ column "geoip_subdivision_1_code" {
+ type = "AggregateFunction(argMin, Nullable(String), DateTime64(6, 'UTC'))"
+ }
+ column "geoip_subdivision_1_name" {
+ type = "AggregateFunction(argMin, Nullable(String), DateTime64(6, 'UTC'))"
+ }
+ column "geoip_subdivision_city_name" {
+ type = "AggregateFunction(argMin, Nullable(String), DateTime64(6, 'UTC'))"
+ }
+ column "geoip_time_zone" {
+ type = "AggregateFunction(argMin, Nullable(String), DateTime64(6, 'UTC'))"
+ }
+ column "entry_referring_domain" {
+ type = "AggregateFunction(argMin, Nullable(String), DateTime64(6, 'UTC'))"
+ }
+ column "entry_utm_source" {
+ type = "AggregateFunction(argMin, Nullable(String), DateTime64(6, 'UTC'))"
+ }
+ column "entry_utm_campaign" {
+ type = "AggregateFunction(argMin, Nullable(String), DateTime64(6, 'UTC'))"
+ }
+ column "entry_utm_medium" {
+ type = "AggregateFunction(argMin, Nullable(String), DateTime64(6, 'UTC'))"
+ }
+ column "entry_utm_term" {
+ type = "AggregateFunction(argMin, Nullable(String), DateTime64(6, 'UTC'))"
+ }
+ column "entry_utm_content" {
+ type = "AggregateFunction(argMin, Nullable(String), DateTime64(6, 'UTC'))"
+ }
+ column "entry_gclid" {
+ type = "AggregateFunction(argMin, Nullable(String), DateTime64(6, 'UTC'))"
+ }
+ column "entry_gad_source" {
+ type = "AggregateFunction(argMin, Nullable(String), DateTime64(6, 'UTC'))"
+ }
+ column "entry_fbclid" {
+ type = "AggregateFunction(argMin, Nullable(String), DateTime64(6, 'UTC'))"
+ }
+ column "entry_has_gclid" {
+ type = "AggregateFunction(argMin, UInt8, DateTime64(6, 'UTC'))"
+ }
+ column "entry_has_fbclid" {
+ type = "AggregateFunction(argMin, UInt8, DateTime64(6, 'UTC'))"
+ }
+ column "entry_ad_ids_set" {
+ type = "AggregateFunction(argMin, Array(String), DateTime64(6, 'UTC'))"
+ }
+ column "entry_channel_type_properties" {
+ type = "AggregateFunction(argMin, Tuple(Nullable(String), Nullable(String), Nullable(String), Nullable(String), UInt8, UInt8, Nullable(String)), DateTime64(6, 'UTC'))"
+ }
+ column "pageview_uniq" {
+ type = "AggregateFunction(uniq, Nullable(UUID))"
+ }
+ column "autocapture_uniq" {
+ type = "AggregateFunction(uniq, Nullable(UUID))"
+ }
+ column "screen_uniq" {
+ type = "AggregateFunction(uniq, Nullable(UUID))"
+ }
+ column "page_screen_uniq_up_to" {
+ type = "AggregateFunction(uniqUpTo(1), Nullable(UUID))"
+ }
+ column "has_autocapture" {
+ type = "UInt8"
+ }
+ column "flag_key_values" {
+ type = "Array(String)"
+ }
+ column "event_names" {
+ type = "Array(String)"
+ }
+ column "hosts" {
+ type = "Array(Nullable(String))"
+ }
+ column "emails" {
+ type = "Array(String)"
+ }
+ column "has_replay_events" {
+ type = "Bool"
+ }
+ }
+
+ materialized_view "raw_sessions_v3_recordings_mv" {
+ to_table = "posthog.writable_raw_sessions_v3"
+ query = <<SQL
+WITH
+ min_first_timestamp AS timestamp,
+ CAST(fromUnixTimestamp64Milli(9223372036854775), 'DateTime64(6)') AS max_ts_64,
+ CAST(fromUnixTimestamp64Milli(-9223372036854775), 'DateTime64(6)') AS min_ts_64,
+ CAST(NULL, 'Nullable(String)') AS null_s,
+ CAST(NULL, 'Nullable(Int64)') AS null_i64,
+ CAST(NULL, 'Nullable(UUID)') AS null_uuid
+SELECT
+ team_id,
+ toUInt128(accurateCast(session_id, 'UUID')) AS session_id_v7,
+ fromUnixTimestamp64Milli(toUInt64(bitShiftRight(session_id_v7, 80))) AS session_timestamp,
+ initializeAggregation('argMaxState', source_table.distinct_id, min_ts_64) AS distinct_id,
+ initializeAggregation('groupUniqArrayState', source_table.distinct_id) AS distinct_ids,
+ timestamp AS min_timestamp,
+ timestamp AS max_timestamp,
+ fromUnixTimestamp(0) AS max_inserted_at,
+ CAST([], 'Array(String)') AS urls,
+ initializeAggregation('argMinState', null_s, max_ts_64) AS entry_url,
+ initializeAggregation('argMaxState', null_s, min_ts_64) AS end_url,
+ initializeAggregation('argMaxState', null_s, min_ts_64) AS last_external_click_url,
+ initializeAggregation('argMinState', null_s, max_ts_64) AS browser,
+ initializeAggregation('argMinState', null_s, max_ts_64) AS browser_version,
+ initializeAggregation('argMinState', null_s, max_ts_64) AS os,
+ initializeAggregation('argMinState', null_s, max_ts_64) AS os_version,
+ initializeAggregation('argMinState', null_s, max_ts_64) AS device_type,
+ initializeAggregation('argMinState', null_i64, max_ts_64) AS viewport_width,
+ initializeAggregation('argMinState', null_i64, max_ts_64) AS viewport_height,
+ initializeAggregation('argMinState', null_s, max_ts_64) AS geoip_country_code,
+ initializeAggregation('argMinState', null_s, max_ts_64) AS geoip_subdivision_1_code,
+ initializeAggregation('argMinState', null_s, max_ts_64) AS geoip_subdivision_1_name,
+ initializeAggregation('argMinState', null_s, max_ts_64) AS geoip_subdivision_city_name,
+ initializeAggregation('argMinState', null_s, max_ts_64) AS geoip_time_zone,
+ initializeAggregation('argMinState', null_s, max_ts_64) AS entry_referring_domain,
+ initializeAggregation('argMinState', null_s, max_ts_64) AS entry_utm_source,
+ initializeAggregation('argMinState', null_s, max_ts_64) AS entry_utm_campaign,
+ initializeAggregation('argMinState', null_s, max_ts_64) AS entry_utm_medium,
+ initializeAggregation('argMinState', null_s, max_ts_64) AS entry_utm_term,
+ initializeAggregation('argMinState', null_s, max_ts_64) AS entry_utm_content,
+ initializeAggregation('argMinState', null_s, max_ts_64) AS entry_gclid,
+ initializeAggregation('argMinState', null_s, max_ts_64) AS entry_gad_source,
+ initializeAggregation('argMinState', null_s, max_ts_64) AS entry_fbclid,
+ initializeAggregation('argMinState', false, max_ts_64) AS entry_has_gclid,
+ initializeAggregation('argMinState', false, max_ts_64) AS entry_has_fbclid,
+ initializeAggregation('argMinState', CAST([], 'Array(String)'), max_ts_64) AS entry_ad_ids_set,
+ initializeAggregation(
+ 'argMinState',
+ tuple(null_s, null_s, null_s, null_s, false, false, null_s),
+ max_ts_64
+ ) AS entry_channel_type_properties,
+ initializeAggregation('uniqState', null_uuid) AS pageview_uniq,
+ initializeAggregation('uniqState', null_uuid) AS autocapture_uniq,
+ initializeAggregation('uniqState', null_uuid) AS screen_uniq,
+ initializeAggregation('uniqUpToState(1)', null_uuid) AS page_screen_uniq_up_to,
+ false AS has_autocapture,
+ CAST([], 'Array(String)') AS flag_key_values,
+ CAST([], 'Array(String)') AS event_names,
+ CAST([], 'Array(String)') AS hosts,
+ CAST([], 'Array(String)') AS emails,
+ true AS has_replay_events
+FROM posthog.sharded_session_replay_events AS source_table
+WHERE
+ (bitAnd(bitShiftRight(toUInt128(accurateCastOrNull(session_id, 'UUID')), 76), 15) = 7)
+AND
+ ((fromUnixTimestamp64Milli(toUInt64(bitShiftRight(toUInt128(accurateCastOrNull(session_id, 'UUID')), 80))) >= (min_first_timestamp - toIntervalDay(31))) AND (fromUnixTimestamp64Milli(toUInt64(bitShiftRight(toUInt128(accurateCastOrNull(session_id, 'UUID')), 80))) <= (min_first_timestamp + toIntervalDay(31))))
+AND
+ true
+SQL
+
+ column "team_id" {
+ type = "Int64"
+ }
+ column "session_id_v7" {
+ type = "UInt128"
+ }
+ column "session_timestamp" {
+ type = "DateTime64(3)"
+ }
+ column "distinct_id" {
+ type = "AggregateFunction(argMax, String, DateTime64(6))"
+ }
+ column "distinct_ids" {
+ type = "AggregateFunction(groupUniqArray, String)"
+ }
+ column "min_timestamp" {
+ type = "SimpleAggregateFunction(min, DateTime64(6, 'UTC'))"
+ }
+ column "max_timestamp" {
+ type = "SimpleAggregateFunction(min, DateTime64(6, 'UTC'))"
+ }
+ column "max_inserted_at" {
+ type = "DateTime"
+ }
+ column "urls" {
+ type = "Array(String)"
+ }
+ column "entry_url" {
+ type = "AggregateFunction(argMin, Nullable(String), DateTime64(6))"
+ }
+ column "end_url" {
+ type = "AggregateFunction(argMax, Nullable(String), DateTime64(6))"
+ }
+ column "last_external_click_url" {
+ type = "AggregateFunction(argMax, Nullable(String), DateTime64(6))"
+ }
+ column "browser" {
+ type = "AggregateFunction(argMin, Nullable(String), DateTime64(6))"
+ }
+ column "browser_version" {
+ type = "AggregateFunction(argMin, Nullable(String), DateTime64(6))"
+ }
+ column "os" {
+ type = "AggregateFunction(argMin, Nullable(String), DateTime64(6))"
+ }
+ column "os_version" {
+ type = "AggregateFunction(argMin, Nullable(String), DateTime64(6))"
+ }
+ column "device_type" {
+ type = "AggregateFunction(argMin, Nullable(String), DateTime64(6))"
+ }
+ column "viewport_width" {
+ type = "AggregateFunction(argMin, Nullable(Int64), DateTime64(6))"
+ }
+ column "viewport_height" {
+ type = "AggregateFunction(argMin, Nullable(Int64), DateTime64(6))"
+ }
+ column "geoip_country_code" {
+ type = "AggregateFunction(argMin, Nullable(String), DateTime64(6))"
+ }
+ column "geoip_subdivision_1_code" {
+ type = "AggregateFunction(argMin, Nullable(String), DateTime64(6))"
+ }
+ column "geoip_subdivision_1_name" {
+ type = "AggregateFunction(argMin, Nullable(String), DateTime64(6))"
+ }
+ column "geoip_subdivision_city_name" {
+ type = "AggregateFunction(argMin, Nullable(String), DateTime64(6))"
+ }
+ column "geoip_time_zone" {
+ type = "AggregateFunction(argMin, Nullable(String), DateTime64(6))"
+ }
+ column "entry_referring_domain" {
+ type = "AggregateFunction(argMin, Nullable(String), DateTime64(6))"
+ }
+ column "entry_utm_source" {
+ type = "AggregateFunction(argMin, Nullable(String), DateTime64(6))"
+ }
+ column "entry_utm_campaign" {
+ type = "AggregateFunction(argMin, Nullable(String), DateTime64(6))"
+ }
+ column "entry_utm_medium" {
+ type = "AggregateFunction(argMin, Nullable(String), DateTime64(6))"
+ }
+ column "entry_utm_term" {
+ type = "AggregateFunction(argMin, Nullable(String), DateTime64(6))"
+ }
+ column "entry_utm_content" {
+ type = "AggregateFunction(argMin, Nullable(String), DateTime64(6))"
+ }
+ column "entry_gclid" {
+ type = "AggregateFunction(argMin, Nullable(String), DateTime64(6))"
+ }
+ column "entry_gad_source" {
+ type = "AggregateFunction(argMin, Nullable(String), DateTime64(6))"
+ }
+ column "entry_fbclid" {
+ type = "AggregateFunction(argMin, Nullable(String), DateTime64(6))"
+ }
+ column "entry_has_gclid" {
+ type = "AggregateFunction(argMin, Bool, DateTime64(6))"
+ }
+ column "entry_has_fbclid" {
+ type = "AggregateFunction(argMin, Bool, DateTime64(6))"
+ }
+ column "entry_ad_ids_set" {
+ type = "AggregateFunction(argMin, Array(String), DateTime64(6))"
+ }
+ column "entry_channel_type_properties" {
+ type = "AggregateFunction(argMin, Tuple(Nullable(String), Nullable(String), Nullable(String), Nullable(String), Bool, Bool, Nullable(String)), DateTime64(6))"
+ }
+ column "pageview_uniq" {
+ type = "AggregateFunction(uniq, Nullable(UUID))"
+ }
+ column "autocapture_uniq" {
+ type = "AggregateFunction(uniq, Nullable(UUID))"
+ }
+ column "screen_uniq" {
+ type = "AggregateFunction(uniq, Nullable(UUID))"
+ }
+ column "page_screen_uniq_up_to" {
+ type = "AggregateFunction(uniqUpTo(1), Nullable(UUID))"
+ }
+ column "has_autocapture" {
+ type = "Bool"
+ }
+ column "flag_key_values" {
+ type = "Array(String)"
+ }
+ column "event_names" {
+ type = "Array(String)"
+ }
+ column "hosts" {
+ type = "Array(String)"
+ }
+ column "emails" {
+ type = "Array(String)"
+ }
+ column "has_replay_events" {
+ type = "Bool"
+ }
+ }
+
materialized_view "sessions_mv" {
to_table = "posthog.writable_sessions"
query = <<SQL
@@ -11960,16 +12438,14 @@ SELECT
argMinMerge(entry_fbclid) AS entry_fbclid,
argMinMerge(entry_has_gclid) AS entry_has_gclid,
argMinMerge(entry_has_fbclid) AS entry_has_fbclid,
- argMinMerge(entry_ad_ids_map) AS entry_ad_ids_map,
argMinMerge(entry_ad_ids_set) AS entry_ad_ids_set,
argMinMerge(entry_channel_type_properties) AS entry_channel_type_properties,
- uniqExactMerge(pageview_uniq) AS pageview_uniq,
- uniqExactMerge(autocapture_uniq) AS autocapture_uniq,
- uniqExactMerge(screen_uniq) AS screen_uniq,
+ uniqMerge(pageview_uniq) AS pageview_uniq,
+ uniqMerge(autocapture_uniq) AS autocapture_uniq,
+ uniqMerge(screen_uniq) AS screen_uniq,
uniqUpToMerge(1)(page_screen_uniq_up_to) AS page_screen_uniq_up_to,
max(has_autocapture) AS has_autocapture,
groupUniqArrayArray(10000)(flag_key_values) AS flag_key_values,
- groupUniqArrayArray(flag_keys) AS flag_keys,
groupUniqArrayArray(2000)(event_names) AS event_names,
groupUniqArrayArray(100)(hosts) AS hosts,
groupUniqArrayArray(10)(emails) AS emails,
diff --git a/posthog/clickhouse/hcl/golden/local-single/all.hcl b/posthog/clickhouse/hcl/golden/local-single/all.hcl
index a144006d9c37..9895dad26e39 100644
--- a/posthog/clickhouse/hcl/golden/local-single/all.hcl
+++ b/posthog/clickhouse/hcl/golden/local-single/all.hcl
@@ -10052,9 +10052,6 @@ SQL
column "entry_has_fbclid" {
type = "AggregateFunction(argMin, Bool, DateTime64(6, 'UTC'))"
}
- column "entry_ad_ids_map" {
- type = "AggregateFunction(argMin, Map(String, String), DateTime64(6, 'UTC'))"
- }
column "entry_ad_ids_set" {
type = "AggregateFunction(argMin, Array(String), DateTime64(6, 'UTC'))"
}
@@ -10062,13 +10059,13 @@ SQL
type = "AggregateFunction(argMin, Tuple(Nullable(String), Nullable(String), Nullable(String), Nullable(String), Bool, Bool, Nullable(String)), DateTime64(6, 'UTC'))"
}
column "pageview_uniq" {
- type = "AggregateFunction(uniqExact, Nullable(UUID))"
+ type = "AggregateFunction(uniq, Nullable(UUID))"
}
column "autocapture_uniq" {
- type = "AggregateFunction(uniqExact, Nullable(UUID))"
+ type = "AggregateFunction(uniq, Nullable(UUID))"
}
column "screen_uniq" {
- type = "AggregateFunction(uniqExact, Nullable(UUID))"
+ type = "AggregateFunction(uniq, Nullable(UUID))"
}
column "page_screen_uniq_up_to" {
type = "AggregateFunction(uniqUpTo(1), Nullable(UUID))"
@@ -10079,9 +10076,6 @@ SQL
column "flag_key_values" {
type = "SimpleAggregateFunction(groupUniqArrayArray(10000), Array(String))"
}
- column "flag_keys" {
- type = "SimpleAggregateFunction(groupUniqArrayArray, Array(String))"
- }
column "event_names" {
type = "SimpleAggregateFunction(groupUniqArrayArray(2000), Array(String))"
}
@@ -13703,9 +13697,6 @@ SQL
column "entry_has_fbclid" {
type = "AggregateFunction(argMin, Bool, DateTime64(6, 'UTC'))"
}
- column "entry_ad_ids_map" {
- type = "AggregateFunction(argMin, Map(String, String), DateTime64(6, 'UTC'))"
- }
column "entry_ad_ids_set" {
type = "AggregateFunction(argMin, Array(String), DateTime64(6, 'UTC'))"
}
@@ -13713,13 +13704,13 @@ SQL
type = "AggregateFunction(argMin, Tuple(Nullable(String), Nullable(String), Nullable(String), Nullable(String), Bool, Bool, Nullable(String)), DateTime64(6, 'UTC'))"
}
column "pageview_uniq" {
- type = "AggregateFunction(uniqExact, Nullable(UUID))"
+ type = "AggregateFunction(uniq, Nullable(UUID))"
}
column "autocapture_uniq" {
- type = "AggregateFunction(uniqExact, Nullable(UUID))"
+ type = "AggregateFunction(uniq, Nullable(UUID))"
}
column "screen_uniq" {
- type = "AggregateFunction(uniqExact, Nullable(UUID))"
+ type = "AggregateFunction(uniq, Nullable(UUID))"
}
column "page_screen_uniq_up_to" {
type = "AggregateFunction(uniqUpTo(1), Nullable(UUID))"
@@ -13730,9 +13721,6 @@ SQL
column "flag_key_values" {
type = "SimpleAggregateFunction(groupUniqArrayArray(10000), Array(String))"
}
- column "flag_keys" {
- type = "SimpleAggregateFunction(groupUniqArrayArray, Array(String))"
- }
column "event_names" {
type = "SimpleAggregateFunction(groupUniqArrayArray(2000), Array(String))"
}
@@ -13750,15 +13738,10 @@ SQL
type = "bloom_filter()"
granularity = 1
}
- index "flag_key_values_bloom_filter" {
+ index "flag_key_values_text" {
expr = "flag_key_values"
- type = "bloom_filter()"
- granularity = 1
- }
- index "flag_keys_bloom_filter" {
- expr = "flag_keys"
- type = "bloom_filter()"
- granularity = 1
+ type = "text(tokenizer = splitByString(['=']))"
+ granularity = 100000000
}
index "hosts_bloom_filter" {
expr = "hosts"
@@ -19222,9 +19205,6 @@ SQL
column "entry_has_fbclid" {
type = "AggregateFunction(argMin, Bool, DateTime64(6, 'UTC'))"
}
- column "entry_ad_ids_map" {
- type = "AggregateFunction(argMin, Map(String, String), DateTime64(6, 'UTC'))"
- }
column "entry_ad_ids_set" {
type = "AggregateFunction(argMin, Array(String), DateTime64(6, 'UTC'))"
}
@@ -19232,13 +19212,13 @@ SQL
type = "AggregateFunction(argMin, Tuple(Nullable(String), Nullable(String), Nullable(String), Nullable(String), Bool, Bool, Nullable(String)), DateTime64(6, 'UTC'))"
}
column "pageview_uniq" {
- type = "AggregateFunction(uniqExact, Nullable(UUID))"
+ type = "AggregateFunction(uniq, Nullable(UUID))"
}
column "autocapture_uniq" {
- type = "AggregateFunction(uniqExact, Nullable(UUID))"
+ type = "AggregateFunction(uniq, Nullable(UUID))"
}
column "screen_uniq" {
- type = "AggregateFunction(uniqExact, Nullable(UUID))"
+ type = "AggregateFunction(uniq, Nullable(UUID))"
}
column "page_screen_uniq_up_to" {
type = "AggregateFunction(uniqUpTo(1), Nullable(UUID))"
@@ -19249,9 +19229,6 @@ SQL
column "flag_key_values" {
type = "SimpleAggregateFunction(groupUniqArrayArray(10000), Array(String))"
}
- column "flag_keys" {
- type = "SimpleAggregateFunction(groupUniqArrayArray, Array(String))"
- }
column "event_names" {
type = "SimpleAggregateFunction(groupUniqArrayArray(2000), Array(String))"
}
@@ -24577,6 +24554,507 @@ SQL
}
}
+ materialized_view "raw_sessions_v3_mv" {
+ to_table = "posthog.writable_raw_sessions_v3"
+ query = <<SQL
+WITH
+ JSONExtract(properties, 'Tuple(\n `$current_url` Nullable(String),\n `$external_click_url` Nullable(String),\n `$browser` Nullable(String),\n `$browser_version` Nullable(String),\n `$os` Nullable(String),\n `$os_version` Nullable(String),\n `$device_type` Nullable(String),\n `$viewport_width` Nullable(Int64),\n `$viewport_height` Nullable(Int64),\n `$geoip_country_code` Nullable(String),\n `$geoip_subdivision_1_code` Nullable(String),\n `$geoip_subdivision_1_name` Nullable(String),\n `$geoip_city_name` Nullable(String),\n `$geoip_time_zone` Nullable(String),\n `$referring_domain` Nullable(String),\n `utm_source` Nullable(String),\n `utm_campaign` Nullable(String),\n `utm_medium` Nullable(String),\n `utm_term` Nullable(String),\n `utm_content` Nullable(String),\n `gclid` Nullable(String),\n `gad_source` Nullable(String),\n `fbclid` Nullable(String),\n `$host` Nullable(String),\n `gclsrc` Nullable(String),\n `dclid` Nullable(String),\n `gbraid` Nullable(String),\n `wbraid` Nullable(String),\n `gad_campaignid` Nullable(String),\n `srsltid` Nullable(String),\n `msclkid` Nullable(String),\n `twclid` Nullable(String),\n `li_fat_id` Nullable(String),\n `mc_cid` Nullable(String),\n `igshid` Nullable(String),\n `ttclid` Nullable(String),\n `rdt_cid` Nullable(String),\n `epik` Nullable(String),\n `qclid` Nullable(String),\n `sccid` Nullable(String),\n `_kx` Nullable(String),\n `irclid` Nullable(String),\n `irclickid` Nullable(String),\n `utm_id` Nullable(String),\n `utm_source_platform` Nullable(String),\n `utm_creative_format` Nullable(String),\n `utm_marketing_tactic` Nullable(String),\n `yclid` Nullable(String),\n `tblci` Nullable(String),\n `dicbo` Nullable(String),\n `ndclid` Nullable(String),\n `cjevent` Nullable(String),\n `awc` Nullable(String),\n `syclid` Nullable(String),\n `_bhlid` Nullable(String),\n `rtid` Nullable(String),\n `oppref` Nullable(String),\n `bfcid` Nullable(String)\n )') AS p,
+ JSONExtractString(person_properties, 'email') AS _person_email,
+ tupleElement(p, '$current_url') AS _current_url,
+ if((event = '$pageview') OR (event = '$screen'), left(coalesce(_current_url, ''), 16384), '') AS _attribution_url,
+ tupleElement(p, '$external_click_url') AS _external_click_url,
+ tupleElement(p, '$browser') AS _browser,
+ tupleElement(p, '$browser_version') AS _browser_version,
+ tupleElement(p, '$os') AS _os,
+ tupleElement(p, '$os_version') AS _os_version,
+ tupleElement(p, '$device_type') AS _device_type,
+ tupleElement(p, '$viewport_width') AS _viewport_width,
+ tupleElement(p, '$viewport_height') AS _viewport_height,
+ tupleElement(p, '$geoip_country_code') AS _geoip_country_code,
+ tupleElement(p, '$geoip_subdivision_1_code') AS _geoip_subdivision_1_code,
+ tupleElement(p, '$geoip_subdivision_1_name') AS _geoip_subdivision_1_name,
+ tupleElement(p, '$geoip_city_name') AS _geoip_subdivision_city_name,
+ tupleElement(p, '$geoip_time_zone') AS _geoip_time_zone,
+ tupleElement(p, '$referring_domain') AS _referring_domain,
+ tupleElement(p, 'utm_source') AS _utm_source,
+ tupleElement(p, 'utm_campaign') AS _utm_campaign,
+ tupleElement(p, 'utm_medium') AS _utm_medium,
+ tupleElement(p, 'utm_term') AS _utm_term,
+ tupleElement(p, 'utm_content') AS _utm_content,
+ tupleElement(p, 'gclid') AS _gclid,
+ tupleElement(p, 'gad_source') AS _gad_source,
+ tupleElement(p, 'fbclid') AS _fbclid,
+ coalesce(nullIf(tupleElement(p, 'gclsrc'), ''), nullIf(toValidUTF8(left(decodeURLFormComponent(extractURLParameter(_attribution_url, 'gclsrc')), 2048)), '')) AS gclsrc,
+ coalesce(nullIf(tupleElement(p, 'dclid'), ''), nullIf(toValidUTF8(left(decodeURLFormComponent(extractURLParameter(_attribution_url, 'dclid')), 2048)), '')) AS dclid,
+ coalesce(nullIf(tupleElement(p, 'gbraid'), ''), nullIf(toValidUTF8(left(decodeURLFormComponent(extractURLParameter(_attribution_url, 'gbraid')), 2048)), '')) AS gbraid,
+ coalesce(nullIf(tupleElement(p, 'wbraid'), ''), nullIf(toValidUTF8(left(decodeURLFormComponent(extractURLParameter(_attribution_url, 'wbraid')), 2048)), '')) AS wbraid,
+ coalesce(nullIf(tupleElement(p, 'gad_campaignid'), ''), nullIf(toValidUTF8(left(decodeURLFormComponent(extractURLParameter(_attribution_url, 'gad_campaignid')), 2048)), '')) AS gad_campaignid,
+ coalesce(nullIf(tupleElement(p, 'srsltid'), ''), nullIf(toValidUTF8(left(decodeURLFormComponent(extractURLParameter(_attribution_url, 'srsltid')), 2048)), '')) AS srsltid,
+ coalesce(nullIf(tupleElement(p, 'msclkid'), ''), nullIf(toValidUTF8(left(decodeURLFormComponent(extractURLParameter(_attribution_url, 'msclkid')), 2048)), '')) AS msclkid,
+ coalesce(nullIf(tupleElement(p, 'twclid'), ''), nullIf(toValidUTF8(left(decodeURLFormComponent(extractURLParameter(_attribution_url, 'twclid')), 2048)), '')) AS twclid,
+ coalesce(nullIf(tupleElement(p, 'li_fat_id'), ''), nullIf(toValidUTF8(left(decodeURLFormComponent(extractURLParameter(_attribution_url, 'li_fat_id')), 2048)), '')) AS li_fat_id,
+ coalesce(nullIf(tupleElement(p, 'mc_cid'), ''), nullIf(toValidUTF8(left(decodeURLFormComponent(extractURLParameter(_attribution_url, 'mc_cid')), 2048)), '')) AS mc_cid,
+ coalesce(nullIf(tupleElement(p, 'igshid'), ''), nullIf(toValidUTF8(left(decodeURLFormComponent(extractURLParameter(_attribution_url, 'igshid')), 2048)), '')) AS igshid,
+ coalesce(nullIf(tupleElement(p, 'ttclid'), ''), nullIf(toValidUTF8(left(decodeURLFormComponent(extractURLParameter(_attribution_url, 'ttclid')), 2048)), '')) AS ttclid,
+ coalesce(nullIf(tupleElement(p, 'rdt_cid'), ''), nullIf(toValidUTF8(left(decodeURLFormComponent(extractURLParameter(_attribution_url, 'rdt_cid')), 2048)), '')) AS rdt_cid,
+ coalesce(nullIf(tupleElement(p, 'epik'), ''), nullIf(toValidUTF8(left(decodeURLFormComponent(extractURLParameter(_attribution_url, 'epik')), 2048)), '')) AS epik,
+ coalesce(nullIf(tupleElement(p, 'qclid'), ''), nullIf(toValidUTF8(left(decodeURLFormComponent(extractURLParameter(_attribution_url, 'qclid')), 2048)), '')) AS qclid,
+ coalesce(nullIf(tupleElement(p, 'sccid'), ''), nullIf(toValidUTF8(left(decodeURLFormComponent(extractURLParameter(_attribution_url, 'sccid')), 2048)), ''), nullIf(toValidUTF8(left(decodeURLFormComponent(extractURLParameter(_attribution_url, 'ScCid')), 2048)), ''), nullIf(toValidUTF8(left(decodeURLFormComponent(extractURLParameter(_attribution_url, 'SsCid')), 2048)), '')) AS sccid,
+ coalesce(nullIf(tupleElement(p, '_kx'), ''), nullIf(toValidUTF8(left(decodeURLFormComponent(extractURLParameter(_attribution_url, '_kx')), 2048)), '')) AS _kx,
+ coalesce(nullIf(tupleElement(p, 'irclid'), ''), nullIf(toValidUTF8(left(decodeURLFormComponent(extractURLParameter(_attribution_url, 'irclid')), 2048)), '')) AS irclid,
+ coalesce(nullIf(tupleElement(p, 'irclickid'), ''), nullIf(toValidUTF8(left(decodeURLFormComponent(extractURLParameter(_attribution_url, 'irclickid')), 2048)), '')) AS irclickid,
+ coalesce(nullIf(tupleElement(p, 'utm_id'), ''), nullIf(toValidUTF8(left(decodeURLFormComponent(extractURLParameter(_attribution_url, 'utm_id')), 2048)), '')) AS utm_id,
+ coalesce(nullIf(tupleElement(p, 'utm_source_platform'), ''), nullIf(toValidUTF8(left(decodeURLFormComponent(extractURLParameter(_attribution_url, 'utm_source_platform')), 2048)), '')) AS utm_source_platform,
+ coalesce(nullIf(tupleElement(p, 'utm_creative_format'), ''), nullIf(toValidUTF8(left(decodeURLFormComponent(extractURLParameter(_attribution_url, 'utm_creative_format')), 2048)), '')) AS utm_creative_format,
+ coalesce(nullIf(tupleElement(p, 'utm_marketing_tactic'), ''), nullIf(toValidUTF8(left(decodeURLFormComponent(extractURLParameter(_attribution_url, 'utm_marketing_tactic')), 2048)), '')) AS utm_marketing_tactic,
+ coalesce(nullIf(tupleElement(p, 'yclid'), ''), nullIf(toValidUTF8(left(decodeURLFormComponent(extractURLParameter(_attribution_url, 'yclid')), 2048)), '')) AS yclid,
+ coalesce(nullIf(tupleElement(p, 'tblci'), ''), nullIf(toValidUTF8(left(decodeURLFormComponent(extractURLParameter(_attribution_url, 'tblci')), 2048)), '')) AS tblci,
+ coalesce(nullIf(tupleElement(p, 'dicbo'), ''), nullIf(toValidUTF8(left(decodeURLFormComponent(extractURLParameter(_attribution_url, 'dicbo')), 2048)), '')) AS dicbo,
+ coalesce(nullIf(tupleElement(p, 'ndclid'), ''), nullIf(toValidUTF8(left(decodeURLFormComponent(extractURLParameter(_attribution_url, 'ndclid')), 2048)), '')) AS ndclid,
+ coalesce(nullIf(tupleElement(p, 'cjevent'), ''), nullIf(toValidUTF8(left(decodeURLFormComponent(extractURLParameter(_attribution_url, 'cjevent')), 2048)), '')) AS cjevent,
+ coalesce(nullIf(tupleElement(p, 'awc'), ''), nullIf(toValidUTF8(left(decodeURLFormComponent(extractURLParameter(_attribution_url, 'awc')), 2048)), '')) AS awc,
+ coalesce(nullIf(tupleElement(p, 'syclid'), ''), nullIf(toValidUTF8(left(decodeURLFormComponent(extractURLParameter(_attribution_url, 'syclid')), 2048)), '')) AS syclid,
+ coalesce(nullIf(tupleElement(p, '_bhlid'), ''), nullIf(toValidUTF8(left(decodeURLFormComponent(extractURLParameter(_attribution_url, '_bhlid')), 2048)), '')) AS _bhlid,
+ coalesce(nullIf(tupleElement(p, 'rtid'), ''), nullIf(toValidUTF8(left(decodeURLFormComponent(extractURLParameter(_attribution_url, 'rtid')), 2048)), '')) AS rtid,
+ coalesce(nullIf(tupleElement(p, 'oppref'), ''), nullIf(toValidUTF8(left(decodeURLFormComponent(extractURLParameter(_attribution_url, 'oppref')), 2048)), '')) AS oppref,
+ coalesce(nullIf(tupleElement(p, 'bfcid'), ''), nullIf(toValidUTF8(left(decodeURLFormComponent(extractURLParameter(_attribution_url, 'bfcid')), 2048)), '')) AS bfcid,
+ CAST(arrayFilter(x -> (x IS NOT NULL), [if(gclsrc IS NOT NULL, concat('gclsrc=', gclsrc), NULL), if(dclid IS NOT NULL, concat('dclid=', dclid), NULL), if(gbraid IS NOT NULL, concat('gbraid=', gbraid), NULL), if(wbraid IS NOT NULL, concat('wbraid=', wbraid), NULL), if(gad_campaignid IS NOT NULL, concat('gad_campaignid=', gad_campaignid), NULL), if(srsltid IS NOT NULL, concat('srsltid=', srsltid), NULL), if(msclkid IS NOT NULL, concat('msclkid=', msclkid), NULL), if(twclid IS NOT NULL, concat('twclid=', twclid), NULL), if(li_fat_id IS NOT NULL, concat('li_fat_id=', li_fat_id), NULL), if(mc_cid IS NOT NULL, concat('mc_cid=', mc_cid), NULL), if(igshid IS NOT NULL, concat('igshid=', igshid), NULL), if(ttclid IS NOT NULL, concat('ttclid=', ttclid), NULL), if(rdt_cid IS NOT NULL, concat('rdt_cid=', rdt_cid), NULL), if(epik IS NOT NULL, concat('epik=', epik), NULL), if(qclid IS NOT NULL, concat('qclid=', qclid), NULL), if(sccid IS NOT NULL, concat('sccid=', sccid), NULL), if(_kx IS NOT NULL, concat('_kx=', _kx), NULL), if(irclid IS NOT NULL, concat('irclid=', irclid), NULL), if(irclickid IS NOT NULL, concat('irclickid=', irclickid), NULL), if(utm_id IS NOT NULL, concat('utm_id=', utm_id), NULL), if(utm_source_platform IS NOT NULL, concat('utm_source_platform=', utm_source_platform), NULL), if(utm_creative_format IS NOT NULL, concat('utm_creative_format=', utm_creative_format), NULL), if(utm_marketing_tactic IS NOT NULL, concat('utm_marketing_tactic=', utm_marketing_tactic), NULL), if(yclid IS NOT NULL, concat('yclid=', yclid), NULL), if(tblci IS NOT NULL, concat('tblci=', tblci), NULL), if(dicbo IS NOT NULL, concat('dicbo=', dicbo), NULL), if(ndclid IS NOT NULL, concat('ndclid=', ndclid), NULL), if(cjevent IS NOT NULL, concat('cjevent=', cjevent), NULL), if(awc IS NOT NULL, concat('awc=', awc), NULL), if(syclid IS NOT NULL, concat('syclid=', syclid), NULL), if(_bhlid IS NOT NULL, concat('_bhlid=', _bhlid), NULL), if(rtid IS NOT NULL, concat('rtid=', rtid), NULL), if(oppref IS NOT NULL, concat('oppref=', oppref), NULL), if(bfcid IS NOT NULL, concat('bfcid=', bfcid), NULL)]), 'Array(String)') AS ad_ids_set,
+ tupleElement(p, '$host') AS _host,
+ if((event = '$pageview') OR (event = '$screen'), timestamp, timestamp + toIntervalYear(1)) AS pageview_prio_timestamp_min,
+ if((event = '$pageview') OR (event = '$screen'), timestamp, timestamp - toIntervalYear(1)) AS pageview_prio_timestamp_max
+SELECT
+ team_id,
+ `$session_id_uuid` AS session_id_v7,
+ initializeAggregation('argMaxState', source_table.distinct_id, timestamp) AS distinct_id,
+ initializeAggregation('groupUniqArrayState', source_table.distinct_id) AS distinct_ids,
+ timestamp AS min_timestamp,
+ timestamp AS max_timestamp,
+ inserted_at AS max_inserted_at,
+ if(
+ (_current_url IS NOT NULL) AND ((event = '$pageview') OR (event = '$screen')),
+ [_current_url],
+ []
+ ) AS urls,
+ initializeAggregation('argMinState', _current_url, pageview_prio_timestamp_min) AS entry_url,
+ initializeAggregation('argMaxState', _current_url, pageview_prio_timestamp_max) AS end_url,
+ initializeAggregation('argMaxState', _external_click_url, timestamp) AS last_external_click_url,
+ initializeAggregation('argMinState', _browser, timestamp) AS browser,
+ initializeAggregation('argMinState', _browser_version, timestamp) AS browser_version,
+ initializeAggregation('argMinState', _os, timestamp) AS os,
+ initializeAggregation('argMinState', _os_version, timestamp) AS os_version,
+ initializeAggregation('argMinState', _device_type, timestamp) AS device_type,
+ initializeAggregation('argMinState', _viewport_width, timestamp) AS viewport_width,
+ initializeAggregation('argMinState', _viewport_height, timestamp) AS viewport_height,
+ initializeAggregation('argMinState', _geoip_country_code, timestamp) AS geoip_country_code,
+ initializeAggregation('argMinState', _geoip_subdivision_1_code, timestamp) AS geoip_subdivision_1_code,
+ initializeAggregation('argMinState', _geoip_subdivision_1_name, timestamp) AS geoip_subdivision_1_name,
+ initializeAggregation('argMinState', _geoip_subdivision_city_name, timestamp) AS geoip_subdivision_city_name,
+ initializeAggregation('argMinState', _geoip_time_zone, timestamp) AS geoip_time_zone,
+ initializeAggregation('argMinState', _referring_domain, pageview_prio_timestamp_min) AS entry_referring_domain,
+ initializeAggregation('argMinState', _utm_source, pageview_prio_timestamp_min) AS entry_utm_source,
+ initializeAggregation('argMinState', _utm_campaign, pageview_prio_timestamp_min) AS entry_utm_campaign,
+ initializeAggregation('argMinState', _utm_medium, pageview_prio_timestamp_min) AS entry_utm_medium,
+ initializeAggregation('argMinState', _utm_term, pageview_prio_timestamp_min) AS entry_utm_term,
+ initializeAggregation('argMinState', _utm_content, pageview_prio_timestamp_min) AS entry_utm_content,
+ initializeAggregation('argMinState', _gclid, pageview_prio_timestamp_min) AS entry_gclid,
+ initializeAggregation('argMinState', _gad_source, pageview_prio_timestamp_min) AS entry_gad_source,
+ initializeAggregation('argMinState', _fbclid, pageview_prio_timestamp_min) AS entry_fbclid,
+ initializeAggregation('argMinState', nullIf(_gclid, '') IS NOT NULL, pageview_prio_timestamp_min) AS entry_has_gclid,
+ initializeAggregation('argMinState', nullIf(_fbclid, '') IS NOT NULL, pageview_prio_timestamp_min) AS entry_has_fbclid,
+ initializeAggregation('argMinState', ad_ids_set, pageview_prio_timestamp_min) AS entry_ad_ids_set,
+ initializeAggregation(
+ 'argMinState',
+ tuple(
+ _utm_source,
+ _utm_medium,
+ _utm_campaign,
+ _referring_domain,
+ nullIf(_gclid, '') IS NOT NULL,
+ nullIf(_fbclid, '') IS NOT NULL,
+ _gad_source
+ ),
+ pageview_prio_timestamp_min
+ ) AS entry_channel_type_properties,
+ initializeAggregation('uniqState', if(event = '$pageview', uuid, NULL)) AS pageview_uniq,
+ initializeAggregation('uniqState', if(event = '$autocapture', uuid, NULL)) AS autocapture_uniq,
+ initializeAggregation('uniqState', if(event = '$screen', uuid, NULL)) AS screen_uniq,
+ initializeAggregation(
+ 'uniqUpToState(1)',
+ if((event = '$pageview') OR (event = '$screen'), uuid, NULL)
+ ) AS page_screen_uniq_up_to,
+ event = '$autocapture' AS has_autocapture,
+ arrayMap(
+ (k, v) -> concat(k, '=', v),
+ mapKeys(properties_group_feature_flags),
+ mapValues(properties_group_feature_flags)
+ ) AS flag_key_values,
+ [event] AS event_names,
+ if((_host IS NOT NULL) AND (_host != ''), [_host], []) AS hosts,
+ if((_person_email IS NOT NULL) AND (_person_email != ''), [_person_email], []) AS emails,
+ false AS has_replay_events
+FROM posthog.sharded_events AS source_table
+WHERE
+ (bitAnd(bitShiftRight(toUInt128(accurateCastOrNull(`$session_id`, 'UUID')), 76), 15) = 7)
+AND
+ ((fromUnixTimestamp64Milli(toUInt64(bitShiftRight(`$session_id_uuid`, 80))) >= (timestamp - toIntervalDay(31))) AND (fromUnixTimestamp64Milli(toUInt64(bitShiftRight(`$session_id_uuid`, 80))) <= (timestamp + toIntervalDay(31))))
+AND
+ true
+SQL
+
+ column "team_id" {
+ type = "Int64"
+ }
+ column "session_id_v7" {
+ type = "Nullable(UInt128)"
+ }
+ column "distinct_id" {
+ type = "AggregateFunction(argMax, String, DateTime64(6, 'UTC'))"
+ }
+ column "distinct_ids" {
+ type = "AggregateFunction(groupUniqArray, String)"
+ }
+ column "min_timestamp" {
+ type = "DateTime64(6, 'UTC')"
+ }
+ column "max_timestamp" {
+ type = "DateTime64(6, 'UTC')"
+ }
+ column "max_inserted_at" {
+ type = "Nullable(DateTime64(6, 'UTC'))"
+ }
+ column "urls" {
+ type = "Array(Nullable(String))"
+ }
+ column "entry_url" {
+ type = "AggregateFunction(argMin, Nullable(String), DateTime64(6, 'UTC'))"
+ }
+ column "end_url" {
+ type = "AggregateFunction(argMax, Nullable(String), DateTime64(6, 'UTC'))"
+ }
+ column "last_external_click_url" {
+ type = "AggregateFunction(argMax, Nullable(String), DateTime64(6, 'UTC'))"
+ }
+ column "browser" {
+ type = "AggregateFunction(argMin, Nullable(String), DateTime64(6, 'UTC'))"
+ }
+ column "browser_version" {
+ type = "AggregateFunction(argMin, Nullable(String), DateTime64(6, 'UTC'))"
+ }
+ column "os" {
+ type = "AggregateFunction(argMin, Nullable(String), DateTime64(6, 'UTC'))"
+ }
+ column "os_version" {
+ type = "AggregateFunction(argMin, Nullable(String), DateTime64(6, 'UTC'))"
+ }
+ column "device_type" {
+ type = "AggregateFunction(argMin, Nullable(String), DateTime64(6, 'UTC'))"
+ }
+ column "viewport_width" {
+ type = "AggregateFunction(argMin, Nullable(Int64), DateTime64(6, 'UTC'))"
+ }
+ column "viewport_height" {
+ type = "AggregateFunction(argMin, Nullable(Int64), DateTime64(6, 'UTC'))"
+ }
+ column "geoip_country_code" {
+ type = "AggregateFunction(argMin, Nullable(String), DateTime64(6, 'UTC'))"
+ }
+ column "geoip_subdivision_1_code" {
+ type = "AggregateFunction(argMin, Nullable(String), DateTime64(6, 'UTC'))"
+ }
+ column "geoip_subdivision_1_name" {
+ type = "AggregateFunction(argMin, Nullable(String), DateTime64(6, 'UTC'))"
+ }
+ column "geoip_subdivision_city_name" {
+ type = "AggregateFunction(argMin, Nullable(String), DateTime64(6, 'UTC'))"
+ }
+ column "geoip_time_zone" {
+ type = "AggregateFunction(argMin, Nullable(String), DateTime64(6, 'UTC'))"
+ }
+ column "entry_referring_domain" {
+ type = "AggregateFunction(argMin, Nullable(String), DateTime64(6, 'UTC'))"
+ }
+ column "entry_utm_source" {
+ type = "AggregateFunction(argMin, Nullable(String), DateTime64(6, 'UTC'))"
+ }
+ column "entry_utm_campaign" {
+ type = "AggregateFunction(argMin, Nullable(String), DateTime64(6, 'UTC'))"
+ }
+ column "entry_utm_medium" {
+ type = "AggregateFunction(argMin, Nullable(String), DateTime64(6, 'UTC'))"
+ }
+ column "entry_utm_term" {
+ type = "AggregateFunction(argMin, Nullable(String), DateTime64(6, 'UTC'))"
+ }
+ column "entry_utm_content" {
+ type = "AggregateFunction(argMin, Nullable(String), DateTime64(6, 'UTC'))"
+ }
+ column "entry_gclid" {
+ type = "AggregateFunction(argMin, Nullable(String), DateTime64(6, 'UTC'))"
+ }
+ column "entry_gad_source" {
+ type = "AggregateFunction(argMin, Nullable(String), DateTime64(6, 'UTC'))"
+ }
+ column "entry_fbclid" {
+ type = "AggregateFunction(argMin, Nullable(String), DateTime64(6, 'UTC'))"
+ }
+ column "entry_has_gclid" {
+ type = "AggregateFunction(argMin, UInt8, DateTime64(6, 'UTC'))"
+ }
+ column "entry_has_fbclid" {
+ type = "AggregateFunction(argMin, UInt8, DateTime64(6, 'UTC'))"
+ }
+ column "entry_ad_ids_set" {
+ type = "AggregateFunction(argMin, Array(String), DateTime64(6, 'UTC'))"
+ }
+ column "entry_channel_type_properties" {
+ type = "AggregateFunction(argMin, Tuple(Nullable(String), Nullable(String), Nullable(String), Nullable(String), UInt8, UInt8, Nullable(String)), DateTime64(6, 'UTC'))"
+ }
+ column "pageview_uniq" {
+ type = "AggregateFunction(uniq, Nullable(UUID))"
+ }
+ column "autocapture_uniq" {
+ type = "AggregateFunction(uniq, Nullable(UUID))"
+ }
+ column "screen_uniq" {
+ type = "AggregateFunction(uniq, Nullable(UUID))"
+ }
+ column "page_screen_uniq_up_to" {
+ type = "AggregateFunction(uniqUpTo(1), Nullable(UUID))"
+ }
+ column "has_autocapture" {
+ type = "UInt8"
+ }
+ column "flag_key_values" {
+ type = "Array(String)"
+ }
+ column "event_names" {
+ type = "Array(String)"
+ }
+ column "hosts" {
+ type = "Array(Nullable(String))"
+ }
+ column "emails" {
+ type = "Array(String)"
+ }
+ column "has_replay_events" {
+ type = "Bool"
+ }
+ }
+
+ materialized_view "raw_sessions_v3_recordings_mv" {
+ to_table = "posthog.writable_raw_sessions_v3"
+ query = <<SQL
+WITH
+ min_first_timestamp AS timestamp,
+ CAST(fromUnixTimestamp64Milli(9223372036854775), 'DateTime64(6)') AS max_ts_64,
+ CAST(fromUnixTimestamp64Milli(-9223372036854775), 'DateTime64(6)') AS min_ts_64,
+ CAST(NULL, 'Nullable(String)') AS null_s,
+ CAST(NULL, 'Nullable(Int64)') AS null_i64,
+ CAST(NULL, 'Nullable(UUID)') AS null_uuid
+SELECT
+ team_id,
+ toUInt128(accurateCast(session_id, 'UUID')) AS session_id_v7,
+ fromUnixTimestamp64Milli(toUInt64(bitShiftRight(session_id_v7, 80))) AS session_timestamp,
+ initializeAggregation('argMaxState', source_table.distinct_id, min_ts_64) AS distinct_id,
+ initializeAggregation('groupUniqArrayState', source_table.distinct_id) AS distinct_ids,
+ timestamp AS min_timestamp,
+ timestamp AS max_timestamp,
+ fromUnixTimestamp(0) AS max_inserted_at,
+ CAST([], 'Array(String)') AS urls,
+ initializeAggregation('argMinState', null_s, max_ts_64) AS entry_url,
+ initializeAggregation('argMaxState', null_s, min_ts_64) AS end_url,
+ initializeAggregation('argMaxState', null_s, min_ts_64) AS last_external_click_url,
+ initializeAggregation('argMinState', null_s, max_ts_64) AS browser,
+ initializeAggregation('argMinState', null_s, max_ts_64) AS browser_version,
+ initializeAggregation('argMinState', null_s, max_ts_64) AS os,
+ initializeAggregation('argMinState', null_s, max_ts_64) AS os_version,
+ initializeAggregation('argMinState', null_s, max_ts_64) AS device_type,
+ initializeAggregation('argMinState', null_i64, max_ts_64) AS viewport_width,
+ initializeAggregation('argMinState', null_i64, max_ts_64) AS viewport_height,
+ initializeAggregation('argMinState', null_s, max_ts_64) AS geoip_country_code,
+ initializeAggregation('argMinState', null_s, max_ts_64) AS geoip_subdivision_1_code,
+ initializeAggregation('argMinState', null_s, max_ts_64) AS geoip_subdivision_1_name,
+ initializeAggregation('argMinState', null_s, max_ts_64) AS geoip_subdivision_city_name,
+ initializeAggregation('argMinState', null_s, max_ts_64) AS geoip_time_zone,
+ initializeAggregation('argMinState', null_s, max_ts_64) AS entry_referring_domain,
+ initializeAggregation('argMinState', null_s, max_ts_64) AS entry_utm_source,
+ initializeAggregation('argMinState', null_s, max_ts_64) AS entry_utm_campaign,
+ initializeAggregation('argMinState', null_s, max_ts_64) AS entry_utm_medium,
+ initializeAggregation('argMinState', null_s, max_ts_64) AS entry_utm_term,
+ initializeAggregation('argMinState', null_s, max_ts_64) AS entry_utm_content,
+ initializeAggregation('argMinState', null_s, max_ts_64) AS entry_gclid,
+ initializeAggregation('argMinState', null_s, max_ts_64) AS entry_gad_source,
+ initializeAggregation('argMinState', null_s, max_ts_64) AS entry_fbclid,
+ initializeAggregation('argMinState', false, max_ts_64) AS entry_has_gclid,
+ initializeAggregation('argMinState', false, max_ts_64) AS entry_has_fbclid,
+ initializeAggregation('argMinState', CAST([], 'Array(String)'), max_ts_64) AS entry_ad_ids_set,
+ initializeAggregation(
+ 'argMinState',
+ tuple(null_s, null_s, null_s, null_s, false, false, null_s),
+ max_ts_64
+ ) AS entry_channel_type_properties,
+ initializeAggregation('uniqState', null_uuid) AS pageview_uniq,
+ initializeAggregation('uniqState', null_uuid) AS autocapture_uniq,
+ initializeAggregation('uniqState', null_uuid) AS screen_uniq,
+ initializeAggregation('uniqUpToState(1)', null_uuid) AS page_screen_uniq_up_to,
+ false AS has_autocapture,
+ CAST([], 'Array(String)') AS flag_key_values,
+ CAST([], 'Array(String)') AS event_names,
+ CAST([], 'Array(String)') AS hosts,
+ CAST([], 'Array(String)') AS emails,
+ true AS has_replay_events
+FROM posthog.sharded_session_replay_events AS source_table
+WHERE
+ (bitAnd(bitShiftRight(toUInt128(accurateCastOrNull(session_id, 'UUID')), 76), 15) = 7)
+AND
+ ((fromUnixTimestamp64Milli(toUInt64(bitShiftRight(toUInt128(accurateCastOrNull(session_id, 'UUID')), 80))) >= (min_first_timestamp - toIntervalDay(31))) AND (fromUnixTimestamp64Milli(toUInt64(bitShiftRight(toUInt128(accurateCastOrNull(session_id, 'UUID')), 80))) <= (min_first_timestamp + toIntervalDay(31))))
+AND
+ true
+SQL
+
+ column "team_id" {
+ type = "Int64"
+ }
+ column "session_id_v7" {
+ type = "UInt128"
+ }
+ column "session_timestamp" {
+ type = "DateTime64(3)"
+ }
+ column "distinct_id" {
+ type = "AggregateFunction(argMax, String, DateTime64(6))"
+ }
+ column "distinct_ids" {
+ type = "AggregateFunction(groupUniqArray, String)"
+ }
+ column "min_timestamp" {
+ type = "SimpleAggregateFunction(min, DateTime64(6, 'UTC'))"
+ }
+ column "max_timestamp" {
+ type = "SimpleAggregateFunction(min, DateTime64(6, 'UTC'))"
+ }
+ column "max_inserted_at" {
+ type = "DateTime"
+ }
+ column "urls" {
+ type = "Array(String)"
+ }
+ column "entry_url" {
+ type = "AggregateFunction(argMin, Nullable(String), DateTime64(6))"
+ }
+ column "end_url" {
+ type = "AggregateFunction(argMax, Nullable(String), DateTime64(6))"
+ }
+ column "last_external_click_url" {
+ type = "AggregateFunction(argMax, Nullable(String), DateTime64(6))"
+ }
+ column "browser" {
+ type = "AggregateFunction(argMin, Nullable(String), DateTime64(6))"
+ }
+ column "browser_version" {
+ type = "AggregateFunction(argMin, Nullable(String), DateTime64(6))"
+ }
+ column "os" {
+ type = "AggregateFunction(argMin, Nullable(String), DateTime64(6))"
+ }
+ column "os_version" {
+ type = "AggregateFunction(argMin, Nullable(String), DateTime64(6))"
+ }
+ column "device_type" {
+ type = "AggregateFunction(argMin, Nullable(String), DateTime64(6))"
+ }
+ column "viewport_width" {
+ type = "AggregateFunction(argMin, Nullable(Int64), DateTime64(6))"
+ }
+ column "viewport_height" {
+ type = "AggregateFunction(argMin, Nullable(Int64), DateTime64(6))"
+ }
+ column "geoip_country_code" {
+ type = "AggregateFunction(argMin, Nullable(String), DateTime64(6))"
+ }
+ column "geoip_subdivision_1_code" {
+ type = "AggregateFunction(argMin, Nullable(String), DateTime64(6))"
+ }
+ column "geoip_subdivision_1_name" {
+ type = "AggregateFunction(argMin, Nullable(String), DateTime64(6))"
+ }
+ column "geoip_subdivision_city_name" {
+ type = "AggregateFunction(argMin, Nullable(String), DateTime64(6))"
+ }
+ column "geoip_time_zone" {
+ type = "AggregateFunction(argMin, Nullable(String), DateTime64(6))"
+ }
+ column "entry_referring_domain" {
+ type = "AggregateFunction(argMin, Nullable(String), DateTime64(6))"
+ }
+ column "entry_utm_source" {
+ type = "AggregateFunction(argMin, Nullable(String), DateTime64(6))"
+ }
+ column "entry_utm_campaign" {
+ type = "AggregateFunction(argMin, Nullable(String), DateTime64(6))"
+ }
+ column "entry_utm_medium" {
+ type = "AggregateFunction(argMin, Nullable(String), DateTime64(6))"
+ }
+ column "entry_utm_term" {
+ type = "AggregateFunction(argMin, Nullable(String), DateTime64(6))"
+ }
+ column "entry_utm_content" {
+ type = "AggregateFunction(argMin, Nullable(String), DateTime64(6))"
+ }
+ column "entry_gclid" {
+ type = "AggregateFunction(argMin, Nullable(String), DateTime64(6))"
+ }
+ column "entry_gad_source" {
+ type = "AggregateFunction(argMin, Nullable(String), DateTime64(6))"
+ }
+ column "entry_fbclid" {
+ type = "AggregateFunction(argMin, Nullable(String), DateTime64(6))"
+ }
+ column "entry_has_gclid" {
+ type = "AggregateFunction(argMin, Bool, DateTime64(6))"
+ }
+ column "entry_has_fbclid" {
+ type = "AggregateFunction(argMin, Bool, DateTime64(6))"
+ }
+ column "entry_ad_ids_set" {
+ type = "AggregateFunction(argMin, Array(String), DateTime64(6))"
+ }
+ column "entry_channel_type_properties" {
+ type = "AggregateFunction(argMin, Tuple(Nullable(String), Nullable(String), Nullable(String), Nullable(String), Bool, Bool, Nullable(String)), DateTime64(6))"
+ }
+ column "pageview_uniq" {
+ type = "AggregateFunction(uniq, Nullable(UUID))"
+ }
+ column "autocapture_uniq" {
+ type = "AggregateFunction(uniq, Nullable(UUID))"
+ }
+ column "screen_uniq" {
+ type = "AggregateFunction(uniq, Nullable(UUID))"
+ }
+ column "page_screen_uniq_up_to" {
+ type = "AggregateFunction(uniqUpTo(1), Nullable(UUID))"
+ }
+ column "has_autocapture" {
+ type = "Bool"
+ }
+ column "flag_key_values" {
+ type = "Array(String)"
+ }
+ column "event_names" {
+ type = "Array(String)"
+ }
+ column "hosts" {
+ type = "Array(String)"
+ }
+ column "emails" {
+ type = "Array(String)"
+ }
+ column "has_replay_events" {
+ type = "Bool"
+ }
+ }
+
materialized_view "session_replay_events_mv" {
to_table = "posthog.writable_session_replay_events"
query = <<SQL
@@ -26298,16 +26776,14 @@ SELECT
argMinMerge(entry_fbclid) AS entry_fbclid,
argMinMerge(entry_has_gclid) AS entry_has_gclid,
argMinMerge(entry_has_fbclid) AS entry_has_fbclid,
- argMinMerge(entry_ad_ids_map) AS entry_ad_ids_map,
argMinMerge(entry_ad_ids_set) AS entry_ad_ids_set,
argMinMerge(entry_channel_type_properties) AS entry_channel_type_properties,
- uniqExactMerge(pageview_uniq) AS pageview_uniq,
- uniqExactMerge(autocapture_uniq) AS autocapture_uniq,
- uniqExactMerge(screen_uniq) AS screen_uniq,
+ uniqMerge(pageview_uniq) AS pageview_uniq,
+ uniqMerge(autocapture_uniq) AS autocapture_uniq,
+ uniqMerge(screen_uniq) AS screen_uniq,
uniqUpToMerge(1)(page_screen_uniq_up_to) AS page_screen_uniq_up_to,
max(has_autocapture) AS has_autocapture,
groupUniqArrayArray(10000)(flag_key_values) AS flag_key_values,
- groupUniqArrayArray(flag_keys) AS flag_keys,
groupUniqArrayArray(2000)(event_names) AS event_names,
groupUniqArrayArray(100)(hosts) AS hosts,
groupUniqArrayArray(10)(emails) AS emails,
diff --git a/posthog/clickhouse/hcl/golden/prod-eu/data.hcl b/posthog/clickhouse/hcl/golden/prod-eu/data.hcl
index b434e375ebe2..6ba9707ca10e 100644
--- a/posthog/clickhouse/hcl/golden/prod-eu/data.hcl
+++ b/posthog/clickhouse/hcl/golden/prod-eu/data.hcl
@@ -4672,9 +4672,6 @@ database "posthog" {
column "entry_has_fbclid" {
type = "AggregateFunction(argMin, Bool, DateTime64(6, 'UTC'))"
}
- column "entry_ad_ids_map" {
- type = "AggregateFunction(argMin, Map(String, String), DateTime64(6, 'UTC'))"
- }
column "entry_ad_ids_set" {
type = "AggregateFunction(argMin, Array(String), DateTime64(6, 'UTC'))"
}
@@ -4682,13 +4679,13 @@ database "posthog" {
type = "AggregateFunction(argMin, Tuple(Nullable(String), Nullable(String), Nullable(String), Nullable(String), Bool, Bool, Nullable(String)), DateTime64(6, 'UTC'))"
}
column "pageview_uniq" {
- type = "AggregateFunction(uniqExact, Nullable(UUID))"
+ type = "AggregateFunction(uniq, Nullable(UUID))"
}
column "autocapture_uniq" {
- type = "AggregateFunction(uniqExact, Nullable(UUID))"
+ type = "AggregateFunction(uniq, Nullable(UUID))"
}
column "screen_uniq" {
- type = "AggregateFunction(uniqExact, Nullable(UUID))"
+ type = "AggregateFunction(uniq, Nullable(UUID))"
}
column "page_screen_uniq_up_to" {
type = "AggregateFunction(uniqUpTo(1), Nullable(UUID))"
@@ -4699,12 +4696,6 @@ database "posthog" {
column "flag_key_values" {
type = "SimpleAggregateFunction(groupUniqArrayArray(10000), Array(String))"
}
- column "flag_values" {
- type = "AggregateFunction(groupUniqArrayMap, Map(String, String))"
- }
- column "flag_keys" {
- type = "SimpleAggregateFunction(groupUniqArrayArray, Array(String))"
- }
column "event_names" {
type = "SimpleAggregateFunction(groupUniqArrayArray, Array(String))"
}
@@ -6562,9 +6553,6 @@ database "posthog" {
column "entry_has_fbclid" {
type = "AggregateFunction(argMin, Bool, DateTime64(6, 'UTC'))"
}
- column "entry_ad_ids_map" {
- type = "AggregateFunction(argMin, Map(String, String), DateTime64(6, 'UTC'))"
- }
column "entry_ad_ids_set" {
type = "AggregateFunction(argMin, Array(String), DateTime64(6, 'UTC'))"
}
@@ -6572,13 +6560,13 @@ database "posthog" {
type = "AggregateFunction(argMin, Tuple(Nullable(String), Nullable(String), Nullable(String), Nullable(String), Bool, Bool, Nullable(String)), DateTime64(6, 'UTC'))"
}
column "pageview_uniq" {
- type = "AggregateFunction(uniqExact, Nullable(UUID))"
+ type = "AggregateFunction(uniq, Nullable(UUID))"
}
column "autocapture_uniq" {
- type = "AggregateFunction(uniqExact, Nullable(UUID))"
+ type = "AggregateFunction(uniq, Nullable(UUID))"
}
column "screen_uniq" {
- type = "AggregateFunction(uniqExact, Nullable(UUID))"
+ type = "AggregateFunction(uniq, Nullable(UUID))"
}
column "page_screen_uniq_up_to" {
type = "AggregateFunction(uniqUpTo(1), Nullable(UUID))"
@@ -6589,12 +6577,6 @@ database "posthog" {
column "flag_key_values" {
type = "SimpleAggregateFunction(groupUniqArrayArray(10000), Array(String))"
}
- column "flag_values" {
- type = "AggregateFunction(groupUniqArrayMap, Map(String, String))"
- }
- column "flag_keys" {
- type = "SimpleAggregateFunction(groupUniqArrayArray, Array(String))"
- }
column "event_names" {
type = "SimpleAggregateFunction(groupUniqArrayArray, Array(String))"
}
@@ -6612,15 +6594,10 @@ database "posthog" {
type = "bloom_filter()"
granularity = 1
}
- index "flag_keys_bloom_filter" {
- expr = "flag_keys"
- type = "bloom_filter()"
- granularity = 1
- }
- index "flag_key_values_bloom_filter" {
+ index "flag_key_values_text" {
expr = "flag_key_values"
- type = "bloom_filter()"
- granularity = 1
+ type = "text(tokenizer = splitByString(['=']))"
+ granularity = 100000000
}
index "hosts_bloom_filter" {
expr = "hosts"
@@ -9231,9 +9208,6 @@ database "posthog" {
column "entry_has_fbclid" {
type = "AggregateFunction(argMin, Bool, DateTime64(6, 'UTC'))"
}
- column "entry_ad_ids_map" {
- type = "AggregateFunction(argMin, Map(String, String), DateTime64(6, 'UTC'))"
- }
column "entry_ad_ids_set" {
type = "AggregateFunction(argMin, Array(String), DateTime64(6, 'UTC'))"
}
@@ -9241,13 +9215,13 @@ database "posthog" {
type = "AggregateFunction(argMin, Tuple(Nullable(String), Nullable(String), Nullable(String), Nullable(String), Bool, Bool, Nullable(String)), DateTime64(6, 'UTC'))"
}
column "pageview_uniq" {
- type = "AggregateFunction(uniqExact, Nullable(UUID))"
+ type = "AggregateFunction(uniq, Nullable(UUID))"
}
column "autocapture_uniq" {
- type = "AggregateFunction(uniqExact, Nullable(UUID))"
+ type = "AggregateFunction(uniq, Nullable(UUID))"
}
column "screen_uniq" {
- type = "AggregateFunction(uniqExact, Nullable(UUID))"
+ type = "AggregateFunction(uniq, Nullable(UUID))"
}
column "page_screen_uniq_up_to" {
type = "AggregateFunction(uniqUpTo(1), Nullable(UUID))"
@@ -9258,12 +9232,6 @@ database "posthog" {
column "flag_key_values" {
type = "SimpleAggregateFunction(groupUniqArrayArray(10000), Array(String))"
}
- column "flag_values" {
- type = "AggregateFunction(groupUniqArrayMap, Map(String, String))"
- }
- column "flag_keys" {
- type = "SimpleAggregateFunction(groupUniqArrayArray, Array(String))"
- }
column "event_names" {
type = "SimpleAggregateFunction(groupUniqArrayArray, Array(String))"
}
@@ -10981,16 +10949,14 @@ SELECT
argMinMerge(entry_fbclid) AS entry_fbclid,
argMinMerge(entry_has_gclid) AS entry_has_gclid,
argMinMerge(entry_has_fbclid) AS entry_has_fbclid,
- argMinMerge(entry_ad_ids_map) AS entry_ad_ids_map,
argMinMerge(entry_ad_ids_set) AS entry_ad_ids_set,
argMinMerge(entry_channel_type_properties) AS entry_channel_type_properties,
- uniqExactMerge(pageview_uniq) AS pageview_uniq,
- uniqExactMerge(autocapture_uniq) AS autocapture_uniq,
- uniqExactMerge(screen_uniq) AS screen_uniq,
+ uniqMerge(pageview_uniq) AS pageview_uniq,
+ uniqMerge(autocapture_uniq) AS autocapture_uniq,
+ uniqMerge(screen_uniq) AS screen_uniq,
uniqUpToMerge(1)(page_screen_uniq_up_to) AS page_screen_uniq_up_to,
max(has_autocapture) AS has_autocapture,
groupUniqArrayArray(10000)(flag_key_values) AS flag_key_values,
- groupUniqArrayArray(flag_keys) AS flag_keys,
groupUniqArrayArray(2000)(event_names) AS event_names,
groupUniqArrayArray(100)(hosts) AS hosts,
groupUniqArrayArray(10)(emails) AS emails,
diff --git a/posthog/clickhouse/hcl/golden/prod-eu/sessions.hcl b/posthog/clickhouse/hcl/golden/prod-eu/sessions.hcl
index 2949718c96d4..8b313a626a8a 100644
--- a/posthog/clickhouse/hcl/golden/prod-eu/sessions.hcl
+++ b/posthog/clickhouse/hcl/golden/prod-eu/sessions.hcl
@@ -1006,9 +1006,6 @@ database "posthog" {
column "entry_has_fbclid" {
type = "AggregateFunction(argMin, Bool, DateTime64(6, 'UTC'))"
}
- column "entry_ad_ids_map" {
- type = "AggregateFunction(argMin, Map(String, String), DateTime64(6, 'UTC'))"
- }
column "entry_ad_ids_set" {
type = "AggregateFunction(argMin, Array(String), DateTime64(6, 'UTC'))"
}
@@ -1016,13 +1013,13 @@ database "posthog" {
type = "AggregateFunction(argMin, Tuple(Nullable(String), Nullable(String), Nullable(String), Nullable(String), Bool, Bool, Nullable(String)), DateTime64(6, 'UTC'))"
}
column "pageview_uniq" {
- type = "AggregateFunction(uniqExact, Nullable(UUID))"
+ type = "AggregateFunction(uniq, Nullable(UUID))"
}
column "autocapture_uniq" {
- type = "AggregateFunction(uniqExact, Nullable(UUID))"
+ type = "AggregateFunction(uniq, Nullable(UUID))"
}
column "screen_uniq" {
- type = "AggregateFunction(uniqExact, Nullable(UUID))"
+ type = "AggregateFunction(uniq, Nullable(UUID))"
}
column "page_screen_uniq_up_to" {
type = "AggregateFunction(uniqUpTo(1), Nullable(UUID))"
@@ -1033,12 +1030,6 @@ database "posthog" {
column "flag_key_values" {
type = "SimpleAggregateFunction(groupUniqArrayArray(10000), Array(String))"
}
- column "flag_values" {
- type = "AggregateFunction(groupUniqArrayMap, Map(String, String))"
- }
- column "flag_keys" {
- type = "SimpleAggregateFunction(groupUniqArrayArray, Array(String))"
- }
column "event_names" {
type = "SimpleAggregateFunction(groupUniqArrayArray, Array(String))"
}
@@ -1056,15 +1047,10 @@ database "posthog" {
type = "bloom_filter()"
granularity = 1
}
- index "flag_keys_bloom_filter" {
- expr = "flag_keys"
- type = "bloom_filter()"
- granularity = 1
- }
- index "flag_key_values_bloom_filter" {
+ index "flag_key_values_text" {
expr = "flag_key_values"
- type = "bloom_filter()"
- granularity = 1
+ type = "text(tokenizer = splitByString(['=']))"
+ granularity = 100000000
}
engine "replicated_aggregating_merge_tree" {
zoo_path = "/clickhouse/tables/sessions/noshard/posthog.raw_sessions_v3"
diff --git a/posthog/clickhouse/hcl/golden/prod-us/data.hcl b/posthog/clickhouse/hcl/golden/prod-us/data.hcl
index 0a03c13f8977..7c977c4bbce9 100644
--- a/posthog/clickhouse/hcl/golden/prod-us/data.hcl
+++ b/posthog/clickhouse/hcl/golden/prod-us/data.hcl
@@ -4467,9 +4467,6 @@ database "posthog" {
column "entry_has_fbclid" {
type = "AggregateFunction(argMin, Bool, DateTime64(6, 'UTC'))"
}
- column "entry_ad_ids_map" {
- type = "AggregateFunction(argMin, Map(String, String), DateTime64(6, 'UTC'))"
- }
column "entry_ad_ids_set" {
type = "AggregateFunction(argMin, Array(String), DateTime64(6, 'UTC'))"
}
@@ -4477,13 +4474,13 @@ database "posthog" {
type = "AggregateFunction(argMin, Tuple(Nullable(String), Nullable(String), Nullable(String), Nullable(String), Bool, Bool, Nullable(String)), DateTime64(6, 'UTC'))"
}
column "pageview_uniq" {
- type = "AggregateFunction(uniqExact, Nullable(UUID))"
+ type = "AggregateFunction(uniq, Nullable(UUID))"
}
column "autocapture_uniq" {
- type = "AggregateFunction(uniqExact, Nullable(UUID))"
+ type = "AggregateFunction(uniq, Nullable(UUID))"
}
column "screen_uniq" {
- type = "AggregateFunction(uniqExact, Nullable(UUID))"
+ type = "AggregateFunction(uniq, Nullable(UUID))"
}
column "page_screen_uniq_up_to" {
type = "AggregateFunction(uniqUpTo(1), Nullable(UUID))"
@@ -4494,12 +4491,6 @@ database "posthog" {
column "flag_key_values" {
type = "SimpleAggregateFunction(groupUniqArrayArray(10000), Array(String))"
}
- column "flag_values" {
- type = "AggregateFunction(groupUniqArrayMap, Map(String, String))"
- }
- column "flag_keys" {
- type = "SimpleAggregateFunction(groupUniqArrayArray, Array(String))"
- }
column "event_names" {
type = "SimpleAggregateFunction(groupUniqArrayArray, Array(String))"
}
@@ -4513,9 +4504,9 @@ database "posthog" {
type = "SimpleAggregateFunction(max, Bool)"
}
engine "distributed" {
- cluster_name = "posthog"
+ cluster_name = "sessions"
remote_database = "posthog"
- remote_table = "sharded_raw_sessions_v3"
+ remote_table = "raw_sessions_v3"
sharding_key = "cityHash64(session_id_v7)"
}
}
@@ -6402,9 +6393,6 @@ database "posthog" {
column "entry_has_fbclid" {
type = "AggregateFunction(argMin, Bool, DateTime64(6, 'UTC'))"
}
- column "entry_ad_ids_map" {
- type = "AggregateFunction(argMin, Map(String, String), DateTime64(6, 'UTC'))"
- }
column "entry_ad_ids_set" {
type = "AggregateFunction(argMin, Array(String), DateTime64(6, 'UTC'))"
}
@@ -6412,13 +6400,13 @@ database "posthog" {
type = "AggregateFunction(argMin, Tuple(Nullable(String), Nullable(String), Nullable(String), Nullable(String), Bool, Bool, Nullable(String)), DateTime64(6, 'UTC'))"
}
column "pageview_uniq" {
- type = "AggregateFunction(uniqExact, Nullable(UUID))"
+ type = "AggregateFunction(uniq, Nullable(UUID))"
}
column "autocapture_uniq" {
- type = "AggregateFunction(uniqExact, Nullable(UUID))"
+ type = "AggregateFunction(uniq, Nullable(UUID))"
}
column "screen_uniq" {
- type = "AggregateFunction(uniqExact, Nullable(UUID))"
+ type = "AggregateFunction(uniq, Nullable(UUID))"
}
column "page_screen_uniq_up_to" {
type = "AggregateFunction(uniqUpTo(1), Nullable(UUID))"
@@ -6429,12 +6417,6 @@ database "posthog" {
column "flag_key_values" {
type = "SimpleAggregateFunction(groupUniqArrayArray(10000), Array(String))"
}
- column "flag_values" {
- type = "AggregateFunction(groupUniqArrayMap, Map(String, String))"
- }
- column "flag_keys" {
- type = "SimpleAggregateFunction(groupUniqArrayArray, Array(String))"
- }
column "event_names" {
type = "SimpleAggregateFunction(groupUniqArrayArray, Array(String))"
}
@@ -6452,15 +6434,10 @@ database "posthog" {
type = "bloom_filter()"
granularity = 1
}
- index "flag_keys_bloom_filter" {
- expr = "flag_keys"
- type = "bloom_filter()"
- granularity = 1
- }
- index "flag_key_values_bloom_filter" {
+ index "flag_key_values_text" {
expr = "flag_key_values"
- type = "bloom_filter()"
- granularity = 1
+ type = "text(tokenizer = splitByString(['=']))"
+ granularity = 100000000
}
index "hosts_bloom_filter" {
expr = "hosts"
@@ -9053,9 +9030,6 @@ database "posthog" {
column "entry_has_fbclid" {
type = "AggregateFunction(argMin, Bool, DateTime64(6, 'UTC'))"
}
- column "entry_ad_ids_map" {
- type = "AggregateFunction(argMin, Map(String, String), DateTime64(6, 'UTC'))"
- }
column "entry_ad_ids_set" {
type = "AggregateFunction(argMin, Array(String), DateTime64(6, 'UTC'))"
}
@@ -9063,13 +9037,13 @@ database "posthog" {
type = "AggregateFunction(argMin, Tuple(Nullable(String), Nullable(String), Nullable(String), Nullable(String), Bool, Bool, Nullable(String)), DateTime64(6, 'UTC'))"
}
column "pageview_uniq" {
- type = "AggregateFunction(uniqExact, Nullable(UUID))"
+ type = "AggregateFunction(uniq, Nullable(UUID))"
}
column "autocapture_uniq" {
- type = "AggregateFunction(uniqExact, Nullable(UUID))"
+ type = "AggregateFunction(uniq, Nullable(UUID))"
}
column "screen_uniq" {
- type = "AggregateFunction(uniqExact, Nullable(UUID))"
+ type = "AggregateFunction(uniq, Nullable(UUID))"
}
column "page_screen_uniq_up_to" {
type = "AggregateFunction(uniqUpTo(1), Nullable(UUID))"
@@ -9080,12 +9054,6 @@ database "posthog" {
column "flag_key_values" {
type = "SimpleAggregateFunction(groupUniqArrayArray(10000), Array(String))"
}
- column "flag_values" {
- type = "AggregateFunction(groupUniqArrayMap, Map(String, String))"
- }
- column "flag_keys" {
- type = "SimpleAggregateFunction(groupUniqArrayArray, Array(String))"
- }
column "event_names" {
type = "SimpleAggregateFunction(groupUniqArrayArray, Array(String))"
}
@@ -10766,16 +10734,14 @@ SELECT
argMinMerge(entry_fbclid) AS entry_fbclid,
argMinMerge(entry_has_gclid) AS entry_has_gclid,
argMinMerge(entry_has_fbclid) AS entry_has_fbclid,
- argMinMerge(entry_ad_ids_map) AS entry_ad_ids_map,
argMinMerge(entry_ad_ids_set) AS entry_ad_ids_set,
argMinMerge(entry_channel_type_properties) AS entry_channel_type_properties,
- uniqExactMerge(pageview_uniq) AS pageview_uniq,
- uniqExactMerge(autocapture_uniq) AS autocapture_uniq,
- uniqExactMerge(screen_uniq) AS screen_uniq,
+ uniqMerge(pageview_uniq) AS pageview_uniq,
+ uniqMerge(autocapture_uniq) AS autocapture_uniq,
+ uniqMerge(screen_uniq) AS screen_uniq,
uniqUpToMerge(1)(page_screen_uniq_up_to) AS page_screen_uniq_up_to,
max(has_autocapture) AS has_autocapture,
groupUniqArrayArray(10000)(flag_key_values) AS flag_key_values,
- groupUniqArrayArray(flag_keys) AS flag_keys,
groupUniqArrayArray(2000)(event_names) AS event_names,
groupUniqArrayArray(100)(hosts) AS hosts,
groupUniqArrayArray(10)(emails) AS emails,
diff --git a/posthog/clickhouse/hcl/golden/prod-us/sessions.hcl b/posthog/clickhouse/hcl/golden/prod-us/sessions.hcl
index a9f5131a9e8b..0ab554ed0917 100644
--- a/posthog/clickhouse/hcl/golden/prod-us/sessions.hcl
+++ b/posthog/clickhouse/hcl/golden/prod-us/sessions.hcl
@@ -1017,9 +1017,6 @@ database "posthog" {
column "entry_has_fbclid" {
type = "AggregateFunction(argMin, Bool, DateTime64(6, 'UTC'))"
}
- column "entry_ad_ids_map" {
- type = "AggregateFunction(argMin, Map(String, String), DateTime64(6, 'UTC'))"
- }
column "entry_ad_ids_set" {
type = "AggregateFunction(argMin, Array(String), DateTime64(6, 'UTC'))"
}
@@ -1027,13 +1024,13 @@ database "posthog" {
type = "AggregateFunction(argMin, Tuple(Nullable(String), Nullable(String), Nullable(String), Nullable(String), Bool, Bool, Nullable(String)), DateTime64(6, 'UTC'))"
}
column "pageview_uniq" {
- type = "AggregateFunction(uniqExact, Nullable(UUID))"
+ type = "AggregateFunction(uniq, Nullable(UUID))"
}
column "autocapture_uniq" {
- type = "AggregateFunction(uniqExact, Nullable(UUID))"
+ type = "AggregateFunction(uniq, Nullable(UUID))"
}
column "screen_uniq" {
- type = "AggregateFunction(uniqExact, Nullable(UUID))"
+ type = "AggregateFunction(uniq, Nullable(UUID))"
}
column "page_screen_uniq_up_to" {
type = "AggregateFunction(uniqUpTo(1), Nullable(UUID))"
@@ -1044,12 +1041,6 @@ database "posthog" {
column "flag_key_values" {
type = "SimpleAggregateFunction(groupUniqArrayArray(10000), Array(String))"
}
- column "flag_values" {
- type = "AggregateFunction(groupUniqArrayMap, Map(String, String))"
- }
- column "flag_keys" {
- type = "SimpleAggregateFunction(groupUniqArrayArray, Array(String))"
- }
column "event_names" {
type = "SimpleAggregateFunction(groupUniqArrayArray, Array(String))"
}
@@ -1067,15 +1058,10 @@ database "posthog" {
type = "bloom_filter()"
granularity = 1
}
- index "flag_keys_bloom_filter" {
- expr = "flag_keys"
- type = "bloom_filter()"
- granularity = 1
- }
- index "flag_key_values_bloom_filter" {
+ index "flag_key_values_text" {
expr = "flag_key_values"
- type = "bloom_filter()"
- granularity = 1
+ type = "text(tokenizer = splitByString(['=']))"
+ granularity = 100000000
}
engine "replicated_aggregating_merge_tree" {
zoo_path = "/clickhouse/tables/sessions/noshard/posthog.raw_sessions_v3"
diff --git a/posthog/clickhouse/hcl/roles/data/cloud/tables.hcl b/posthog/clickhouse/hcl/roles/data/cloud/tables.hcl
index daa3ab21931e..8d0dbc806581 100644
--- a/posthog/clickhouse/hcl/roles/data/cloud/tables.hcl
+++ b/posthog/clickhouse/hcl/roles/data/cloud/tables.hcl
@@ -417,9 +417,6 @@ database "posthog" {
column "entry_has_fbclid" {
type = "AggregateFunction(argMin, Bool, DateTime64(6, 'UTC'))"
}
- column "entry_ad_ids_map" {
- type = "AggregateFunction(argMin, Map(String, String), DateTime64(6, 'UTC'))"
- }
column "entry_ad_ids_set" {
type = "AggregateFunction(argMin, Array(String), DateTime64(6, 'UTC'))"
}
@@ -427,13 +424,13 @@ database "posthog" {
type = "AggregateFunction(argMin, Tuple(Nullable(String), Nullable(String), Nullable(String), Nullable(String), Bool, Bool, Nullable(String)), DateTime64(6, 'UTC'))"
}
column "pageview_uniq" {
- type = "AggregateFunction(uniqExact, Nullable(UUID))"
+ type = "AggregateFunction(uniq, Nullable(UUID))"
}
column "autocapture_uniq" {
- type = "AggregateFunction(uniqExact, Nullable(UUID))"
+ type = "AggregateFunction(uniq, Nullable(UUID))"
}
column "screen_uniq" {
- type = "AggregateFunction(uniqExact, Nullable(UUID))"
+ type = "AggregateFunction(uniq, Nullable(UUID))"
}
column "page_screen_uniq_up_to" {
type = "AggregateFunction(uniqUpTo(1), Nullable(UUID))"
@@ -444,12 +441,6 @@ database "posthog" {
column "flag_key_values" {
type = "SimpleAggregateFunction(groupUniqArrayArray(10000), Array(String))"
}
- column "flag_values" {
- type = "AggregateFunction(groupUniqArrayMap, Map(String, String))"
- }
- column "flag_keys" {
- type = "SimpleAggregateFunction(groupUniqArrayArray, Array(String))"
- }
column "event_names" {
type = "SimpleAggregateFunction(groupUniqArrayArray, Array(String))"
}
@@ -1060,9 +1051,6 @@ database "posthog" {
column "entry_has_fbclid" {
type = "AggregateFunction(argMin, Bool, DateTime64(6, 'UTC'))"
}
- column "entry_ad_ids_map" {
- type = "AggregateFunction(argMin, Map(String, String), DateTime64(6, 'UTC'))"
- }
column "entry_ad_ids_set" {
type = "AggregateFunction(argMin, Array(String), DateTime64(6, 'UTC'))"
}
@@ -1070,13 +1058,13 @@ database "posthog" {
type = "AggregateFunction(argMin, Tuple(Nullable(String), Nullable(String), Nullable(String), Nullable(String), Bool, Bool, Nullable(String)), DateTime64(6, 'UTC'))"
}
column "pageview_uniq" {
- type = "AggregateFunction(uniqExact, Nullable(UUID))"
+ type = "AggregateFunction(uniq, Nullable(UUID))"
}
column "autocapture_uniq" {
- type = "AggregateFunction(uniqExact, Nullable(UUID))"
+ type = "AggregateFunction(uniq, Nullable(UUID))"
}
column "screen_uniq" {
- type = "AggregateFunction(uniqExact, Nullable(UUID))"
+ type = "AggregateFunction(uniq, Nullable(UUID))"
}
column "page_screen_uniq_up_to" {
type = "AggregateFunction(uniqUpTo(1), Nullable(UUID))"
@@ -1087,12 +1075,6 @@ database "posthog" {
column "flag_key_values" {
type = "SimpleAggregateFunction(groupUniqArrayArray(10000), Array(String))"
}
- column "flag_values" {
- type = "AggregateFunction(groupUniqArrayMap, Map(String, String))"
- }
- column "flag_keys" {
- type = "SimpleAggregateFunction(groupUniqArrayArray, Array(String))"
- }
column "event_names" {
type = "SimpleAggregateFunction(groupUniqArrayArray, Array(String))"
}
diff --git a/posthog/clickhouse/hcl/roles/data/common/shared_declarations.hcl b/posthog/clickhouse/hcl/roles/data/common/shared_declarations.hcl
index 29297717ed2c..6104d7810b27 100644
--- a/posthog/clickhouse/hcl/roles/data/common/shared_declarations.hcl
+++ b/posthog/clickhouse/hcl/roles/data/common/shared_declarations.hcl
@@ -970,9 +970,6 @@ SQL
column "entry_has_fbclid" {
type = "AggregateFunction(argMin, Bool, DateTime64(6, 'UTC'))"
}
- column "entry_ad_ids_map" {
- type = "AggregateFunction(argMin, Map(String, String), DateTime64(6, 'UTC'))"
- }
column "entry_ad_ids_set" {
type = "AggregateFunction(argMin, Array(String), DateTime64(6, 'UTC'))"
}
@@ -980,13 +977,13 @@ SQL
type = "AggregateFunction(argMin, Tuple(Nullable(String), Nullable(String), Nullable(String), Nullable(String), Bool, Bool, Nullable(String)), DateTime64(6, 'UTC'))"
}
column "pageview_uniq" {
- type = "AggregateFunction(uniqExact, Nullable(UUID))"
+ type = "AggregateFunction(uniq, Nullable(UUID))"
}
column "autocapture_uniq" {
- type = "AggregateFunction(uniqExact, Nullable(UUID))"
+ type = "AggregateFunction(uniq, Nullable(UUID))"
}
column "screen_uniq" {
- type = "AggregateFunction(uniqExact, Nullable(UUID))"
+ type = "AggregateFunction(uniq, Nullable(UUID))"
}
column "page_screen_uniq_up_to" {
type = "AggregateFunction(uniqUpTo(1), Nullable(UUID))"
@@ -997,9 +994,6 @@ SQL
column "flag_key_values" {
type = "SimpleAggregateFunction(groupUniqArrayArray(10000), Array(String))"
}
- column "flag_keys" {
- type = "SimpleAggregateFunction(groupUniqArrayArray, Array(String))"
- }
column "event_names" {
type = "SimpleAggregateFunction(groupUniqArrayArray(2000), Array(String))"
}
@@ -1058,16 +1052,14 @@ SELECT
argMinMerge(entry_fbclid) AS entry_fbclid,
argMinMerge(entry_has_gclid) AS entry_has_gclid,
argMinMerge(entry_has_fbclid) AS entry_has_fbclid,
- argMinMerge(entry_ad_ids_map) AS entry_ad_ids_map,
argMinMerge(entry_ad_ids_set) AS entry_ad_ids_set,
argMinMerge(entry_channel_type_properties) AS entry_channel_type_properties,
- uniqExactMerge(pageview_uniq) AS pageview_uniq,
- uniqExactMerge(autocapture_uniq) AS autocapture_uniq,
- uniqExactMerge(screen_uniq) AS screen_uniq,
+ uniqMerge(pageview_uniq) AS pageview_uniq,
+ uniqMerge(autocapture_uniq) AS autocapture_uniq,
+ uniqMerge(screen_uniq) AS screen_uniq,
uniqUpToMerge(1)(page_screen_uniq_up_to) AS page_screen_uniq_up_to,
max(has_autocapture) AS has_autocapture,
groupUniqArrayArray(10000)(flag_key_values) AS flag_key_values,
- groupUniqArrayArray(flag_keys) AS flag_keys,
groupUniqArrayArray(2000)(event_names) AS event_names,
groupUniqArrayArray(100)(hosts) AS hosts,
groupUniqArrayArray(10)(emails) AS emails,
@@ -1237,9 +1229,6 @@ SQL
column "entry_has_fbclid" {
type = "AggregateFunction(argMin, Bool, DateTime64(6, 'UTC'))"
}
- column "entry_ad_ids_map" {
- type = "AggregateFunction(argMin, Map(String, String), DateTime64(6, 'UTC'))"
- }
column "entry_ad_ids_set" {
type = "AggregateFunction(argMin, Array(String), DateTime64(6, 'UTC'))"
}
@@ -1247,13 +1236,13 @@ SQL
type = "AggregateFunction(argMin, Tuple(Nullable(String), Nullable(String), Nullable(String), Nullable(String), Bool, Bool, Nullable(String)), DateTime64(6, 'UTC'))"
}
column "pageview_uniq" {
- type = "AggregateFunction(uniqExact, Nullable(UUID))"
+ type = "AggregateFunction(uniq, Nullable(UUID))"
}
column "autocapture_uniq" {
- type = "AggregateFunction(uniqExact, Nullable(UUID))"
+ type = "AggregateFunction(uniq, Nullable(UUID))"
}
column "screen_uniq" {
- type = "AggregateFunction(uniqExact, Nullable(UUID))"
+ type = "AggregateFunction(uniq, Nullable(UUID))"
}
column "page_screen_uniq_up_to" {
type = "AggregateFunction(uniqUpTo(1), Nullable(UUID))"
@@ -1264,9 +1253,6 @@ SQL
column "flag_key_values" {
type = "SimpleAggregateFunction(groupUniqArrayArray(10000), Array(String))"
}
- column "flag_keys" {
- type = "SimpleAggregateFunction(groupUniqArrayArray, Array(String))"
- }
column "event_names" {
type = "SimpleAggregateFunction(groupUniqArrayArray(2000), Array(String))"
}
@@ -1284,15 +1270,10 @@ SQL
type = "bloom_filter()"
granularity = 1
}
- index "flag_key_values_bloom_filter" {
+ index "flag_key_values_text" {
expr = "flag_key_values"
- type = "bloom_filter()"
- granularity = 1
- }
- index "flag_keys_bloom_filter" {
- expr = "flag_keys"
- type = "bloom_filter()"
- granularity = 1
+ type = "text(tokenizer = splitByString(['=']))"
+ granularity = 100000000
}
index "hosts_bloom_filter" {
expr = "hosts"
@@ -1937,9 +1918,6 @@ SQL
column "entry_has_fbclid" {
type = "AggregateFunction(argMin, Bool, DateTime64(6, 'UTC'))"
}
- column "entry_ad_ids_map" {
- type = "AggregateFunction(argMin, Map(String, String), DateTime64(6, 'UTC'))"
- }
column "entry_ad_ids_set" {
type = "AggregateFunction(argMin, Array(String), DateTime64(6, 'UTC'))"
}
@@ -1947,13 +1925,13 @@ SQL
type = "AggregateFunction(argMin, Tuple(Nullable(String), Nullable(String), Nullable(String), Nullable(String), Bool, Bool, Nullable(String)), DateTime64(6, 'UTC'))"
}
column "pageview_uniq" {
- type = "AggregateFunction(uniqExact, Nullable(UUID))"
+ type = "AggregateFunction(uniq, Nullable(UUID))"
}
column "autocapture_uniq" {
- type = "AggregateFunction(uniqExact, Nullable(UUID))"
+ type = "AggregateFunction(uniq, Nullable(UUID))"
}
column "screen_uniq" {
- type = "AggregateFunction(uniqExact, Nullable(UUID))"
+ type = "AggregateFunction(uniq, Nullable(UUID))"
}
column "page_screen_uniq_up_to" {
type = "AggregateFunction(uniqUpTo(1), Nullable(UUID))"
@@ -1964,9 +1942,6 @@ SQL
column "flag_key_values" {
type = "SimpleAggregateFunction(groupUniqArrayArray(10000), Array(String))"
}
- column "flag_keys" {
- type = "SimpleAggregateFunction(groupUniqArrayArray, Array(String))"
- }
column "event_names" {
type = "SimpleAggregateFunction(groupUniqArrayArray(2000), Array(String))"
}
diff --git a/posthog/clickhouse/hcl/roles/data/dev/tables.hcl b/posthog/clickhouse/hcl/roles/data/dev/tables.hcl
index e91470263ce3..7095450030f0 100644
--- a/posthog/clickhouse/hcl/roles/data/dev/tables.hcl
+++ b/posthog/clickhouse/hcl/roles/data/dev/tables.hcl
@@ -1181,9 +1181,6 @@ database "posthog" {
column "entry_has_fbclid" {
type = "AggregateFunction(argMin, Bool, DateTime64(6, 'UTC'))"
}
- column "entry_ad_ids_map" {
- type = "AggregateFunction(argMin, Map(String, String), DateTime64(6, 'UTC'))"
- }
column "entry_ad_ids_set" {
type = "AggregateFunction(argMin, Array(String), DateTime64(6, 'UTC'))"
}
@@ -1191,13 +1188,13 @@ database "posthog" {
type = "AggregateFunction(argMin, Tuple(Nullable(String), Nullable(String), Nullable(String), Nullable(String), Bool, Bool, Nullable(String)), DateTime64(6, 'UTC'))"
}
column "pageview_uniq" {
- type = "AggregateFunction(uniqExact, Nullable(UUID))"
+ type = "AggregateFunction(uniq, Nullable(UUID))"
}
column "autocapture_uniq" {
- type = "AggregateFunction(uniqExact, Nullable(UUID))"
+ type = "AggregateFunction(uniq, Nullable(UUID))"
}
column "screen_uniq" {
- type = "AggregateFunction(uniqExact, Nullable(UUID))"
+ type = "AggregateFunction(uniq, Nullable(UUID))"
}
column "page_screen_uniq_up_to" {
type = "AggregateFunction(uniqUpTo(1), Nullable(UUID))"
@@ -1208,12 +1205,6 @@ database "posthog" {
column "flag_key_values" {
type = "SimpleAggregateFunction(groupUniqArrayArray(10000), Array(String))"
}
- column "flag_values" {
- type = "AggregateFunction(groupUniqArrayMap, Map(String, String))"
- }
- column "flag_keys" {
- type = "SimpleAggregateFunction(groupUniqArrayArray, Array(String))"
- }
column "event_names" {
type = "SimpleAggregateFunction(groupUniqArrayArray, Array(String))"
}
@@ -1231,15 +1222,10 @@ database "posthog" {
type = "bloom_filter()"
granularity = 1
}
- index "flag_keys_bloom_filter" {
- expr = "flag_keys"
- type = "bloom_filter()"
- granularity = 1
- }
- index "flag_key_values_bloom_filter" {
+ index "flag_key_values_text" {
expr = "flag_key_values"
- type = "bloom_filter()"
- granularity = 1
+ type = "text(tokenizer = splitByString(['=']))"
+ granularity = 100000000
}
index "hosts_bloom_filter" {
expr = "hosts"
diff --git a/posthog/clickhouse/hcl/roles/data/local/sql/raw_sessions_v3_mv.sql b/posthog/clickhouse/hcl/roles/data/local/sql/raw_sessions_v3_mv.sql
new file mode 100644
index 000000000000..818c0a9a6df2
--- /dev/null
+++ b/posthog/clickhouse/hcl/roles/data/local/sql/raw_sessions_v3_mv.sql
@@ -0,0 +1,142 @@
+WITH
+ JSONExtract(properties, 'Tuple(\n `$current_url` Nullable(String),\n `$external_click_url` Nullable(String),\n `$browser` Nullable(String),\n `$browser_version` Nullable(String),\n `$os` Nullable(String),\n `$os_version` Nullable(String),\n `$device_type` Nullable(String),\n `$viewport_width` Nullable(Int64),\n `$viewport_height` Nullable(Int64),\n `$geoip_country_code` Nullable(String),\n `$geoip_subdivision_1_code` Nullable(String),\n `$geoip_subdivision_1_name` Nullable(String),\n `$geoip_city_name` Nullable(String),\n `$geoip_time_zone` Nullable(String),\n `$referring_domain` Nullable(String),\n `utm_source` Nullable(String),\n `utm_campaign` Nullable(String),\n `utm_medium` Nullable(String),\n `utm_term` Nullable(String),\n `utm_content` Nullable(String),\n `gclid` Nullable(String),\n `gad_source` Nullable(String),\n `fbclid` Nullable(String),\n `$host` Nullable(String),\n `gclsrc` Nullable(String),\n `dclid` Nullable(String),\n `gbraid` Nullable(String),\n `wbraid` Nullable(String),\n `gad_campaignid` Nullable(String),\n `srsltid` Nullable(String),\n `msclkid` Nullable(String),\n `twclid` Nullable(String),\n `li_fat_id` Nullable(String),\n `mc_cid` Nullable(String),\n `igshid` Nullable(String),\n `ttclid` Nullable(String),\n `rdt_cid` Nullable(String),\n `epik` Nullable(String),\n `qclid` Nullable(String),\n `sccid` Nullable(String),\n `_kx` Nullable(String),\n `irclid` Nullable(String),\n `irclickid` Nullable(String),\n `utm_id` Nullable(String),\n `utm_source_platform` Nullable(String),\n `utm_creative_format` Nullable(String),\n `utm_marketing_tactic` Nullable(String),\n `yclid` Nullable(String),\n `tblci` Nullable(String),\n `dicbo` Nullable(String),\n `ndclid` Nullable(String),\n `cjevent` Nullable(String),\n `awc` Nullable(String),\n `syclid` Nullable(String),\n `_bhlid` Nullable(String),\n `rtid` Nullable(String),\n `oppref` Nullable(String),\n `bfcid` Nullable(String)\n )') AS p,
+ JSONExtractString(person_properties, 'email') AS _person_email,
+ tupleElement(p, '$current_url') AS _current_url,
+ if((event = '$pageview') OR (event = '$screen'), left(coalesce(_current_url, ''), 16384), '') AS _attribution_url,
+ tupleElement(p, '$external_click_url') AS _external_click_url,
+ tupleElement(p, '$browser') AS _browser,
+ tupleElement(p, '$browser_version') AS _browser_version,
+ tupleElement(p, '$os') AS _os,
+ tupleElement(p, '$os_version') AS _os_version,
+ tupleElement(p, '$device_type') AS _device_type,
+ tupleElement(p, '$viewport_width') AS _viewport_width,
+ tupleElement(p, '$viewport_height') AS _viewport_height,
+ tupleElement(p, '$geoip_country_code') AS _geoip_country_code,
+ tupleElement(p, '$geoip_subdivision_1_code') AS _geoip_subdivision_1_code,
+ tupleElement(p, '$geoip_subdivision_1_name') AS _geoip_subdivision_1_name,
+ tupleElement(p, '$geoip_city_name') AS _geoip_subdivision_city_name,
+ tupleElement(p, '$geoip_time_zone') AS _geoip_time_zone,
+ tupleElement(p, '$referring_domain') AS _referring_domain,
+ tupleElement(p, 'utm_source') AS _utm_source,
+ tupleElement(p, 'utm_campaign') AS _utm_campaign,
+ tupleElement(p, 'utm_medium') AS _utm_medium,
+ tupleElement(p, 'utm_term') AS _utm_term,
+ tupleElement(p, 'utm_content') AS _utm_content,
+ tupleElement(p, 'gclid') AS _gclid,
+ tupleElement(p, 'gad_source') AS _gad_source,
+ tupleElement(p, 'fbclid') AS _fbclid,
+ coalesce(nullIf(tupleElement(p, 'gclsrc'), ''), nullIf(toValidUTF8(left(decodeURLFormComponent(extractURLParameter(_attribution_url, 'gclsrc')), 2048)), '')) AS gclsrc,
+ coalesce(nullIf(tupleElement(p, 'dclid'), ''), nullIf(toValidUTF8(left(decodeURLFormComponent(extractURLParameter(_attribution_url, 'dclid')), 2048)), '')) AS dclid,
+ coalesce(nullIf(tupleElement(p, 'gbraid'), ''), nullIf(toValidUTF8(left(decodeURLFormComponent(extractURLParameter(_attribution_url, 'gbraid')), 2048)), '')) AS gbraid,
+ coalesce(nullIf(tupleElement(p, 'wbraid'), ''), nullIf(toValidUTF8(left(decodeURLFormComponent(extractURLParameter(_attribution_url, 'wbraid')), 2048)), '')) AS wbraid,
+ coalesce(nullIf(tupleElement(p, 'gad_campaignid'), ''), nullIf(toValidUTF8(left(decodeURLFormComponent(extractURLParameter(_attribution_url, 'gad_campaignid')), 2048)), '')) AS gad_campaignid,
+ coalesce(nullIf(tupleElement(p, 'srsltid'), ''), nullIf(toValidUTF8(left(decodeURLFormComponent(extractURLParameter(_attribution_url, 'srsltid')), 2048)), '')) AS srsltid,
+ coalesce(nullIf(tupleElement(p, 'msclkid'), ''), nullIf(toValidUTF8(left(decodeURLFormComponent(extractURLParameter(_attribution_url, 'msclkid')), 2048)), '')) AS msclkid,
+ coalesce(nullIf(tupleElement(p, 'twclid'), ''), nullIf(toValidUTF8(left(decodeURLFormComponent(extractURLParameter(_attribution_url, 'twclid')), 2048)), '')) AS twclid,
+ coalesce(nullIf(tupleElement(p, 'li_fat_id'), ''), nullIf(toValidUTF8(left(decodeURLFormComponent(extractURLParameter(_attribution_url, 'li_fat_id')), 2048)), '')) AS li_fat_id,
+ coalesce(nullIf(tupleElement(p, 'mc_cid'), ''), nullIf(toValidUTF8(left(decodeURLFormComponent(extractURLParameter(_attribution_url, 'mc_cid')), 2048)), '')) AS mc_cid,
+ coalesce(nullIf(tupleElement(p, 'igshid'), ''), nullIf(toValidUTF8(left(decodeURLFormComponent(extractURLParameter(_attribution_url, 'igshid')), 2048)), '')) AS igshid,
+ coalesce(nullIf(tupleElement(p, 'ttclid'), ''), nullIf(toValidUTF8(left(decodeURLFormComponent(extractURLParameter(_attribution_url, 'ttclid')), 2048)), '')) AS ttclid,
+ coalesce(nullIf(tupleElement(p, 'rdt_cid'), ''), nullIf(toValidUTF8(left(decodeURLFormComponent(extractURLParameter(_attribution_url, 'rdt_cid')), 2048)), '')) AS rdt_cid,
+ coalesce(nullIf(tupleElement(p, 'epik'), ''), nullIf(toValidUTF8(left(decodeURLFormComponent(extractURLParameter(_attribution_url, 'epik')), 2048)), '')) AS epik,
+ coalesce(nullIf(tupleElement(p, 'qclid'), ''), nullIf(toValidUTF8(left(decodeURLFormComponent(extractURLParameter(_attribution_url, 'qclid')), 2048)), '')) AS qclid,
+ coalesce(nullIf(tupleElement(p, 'sccid'), ''), nullIf(toValidUTF8(left(decodeURLFormComponent(extractURLParameter(_attribution_url, 'sccid')), 2048)), ''), nullIf(toValidUTF8(left(decodeURLFormComponent(extractURLParameter(_attribution_url, 'ScCid')), 2048)), ''), nullIf(toValidUTF8(left(decodeURLFormComponent(extractURLParameter(_attribution_url, 'SsCid')), 2048)), '')) AS sccid,
+ coalesce(nullIf(tupleElement(p, '_kx'), ''), nullIf(toValidUTF8(left(decodeURLFormComponent(extractURLParameter(_attribution_url, '_kx')), 2048)), '')) AS _kx,
+ coalesce(nullIf(tupleElement(p, 'irclid'), ''), nullIf(toValidUTF8(left(decodeURLFormComponent(extractURLParameter(_attribution_url, 'irclid')), 2048)), '')) AS irclid,
+ coalesce(nullIf(tupleElement(p, 'irclickid'), ''), nullIf(toValidUTF8(left(decodeURLFormComponent(extractURLParameter(_attribution_url, 'irclickid')), 2048)), '')) AS irclickid,
+ coalesce(nullIf(tupleElement(p, 'utm_id'), ''), nullIf(toValidUTF8(left(decodeURLFormComponent(extractURLParameter(_attribution_url, 'utm_id')), 2048)), '')) AS utm_id,
+ coalesce(nullIf(tupleElement(p, 'utm_source_platform'), ''), nullIf(toValidUTF8(left(decodeURLFormComponent(extractURLParameter(_attribution_url, 'utm_source_platform')), 2048)), '')) AS utm_source_platform,
+ coalesce(nullIf(tupleElement(p, 'utm_creative_format'), ''), nullIf(toValidUTF8(left(decodeURLFormComponent(extractURLParameter(_attribution_url, 'utm_creative_format')), 2048)), '')) AS utm_creative_format,
+ coalesce(nullIf(tupleElement(p, 'utm_marketing_tactic'), ''), nullIf(toValidUTF8(left(decodeURLFormComponent(extractURLParameter(_attribution_url, 'utm_marketing_tactic')), 2048)), '')) AS utm_marketing_tactic,
+ coalesce(nullIf(tupleElement(p, 'yclid'), ''), nullIf(toValidUTF8(left(decodeURLFormComponent(extractURLParameter(_attribution_url, 'yclid')), 2048)), '')) AS yclid,
+ coalesce(nullIf(tupleElement(p, 'tblci'), ''), nullIf(toValidUTF8(left(decodeURLFormComponent(extractURLParameter(_attribution_url, 'tblci')), 2048)), '')) AS tblci,
+ coalesce(nullIf(tupleElement(p, 'dicbo'), ''), nullIf(toValidUTF8(left(decodeURLFormComponent(extractURLParameter(_attribution_url, 'dicbo')), 2048)), '')) AS dicbo,
+ coalesce(nullIf(tupleElement(p, 'ndclid'), ''), nullIf(toValidUTF8(left(decodeURLFormComponent(extractURLParameter(_attribution_url, 'ndclid')), 2048)), '')) AS ndclid,
+ coalesce(nullIf(tupleElement(p, 'cjevent'), ''), nullIf(toValidUTF8(left(decodeURLFormComponent(extractURLParameter(_attribution_url, 'cjevent')), 2048)), '')) AS cjevent,
+ coalesce(nullIf(tupleElement(p, 'awc'), ''), nullIf(toValidUTF8(left(decodeURLFormComponent(extractURLParameter(_attribution_url, 'awc')), 2048)), '')) AS awc,
+ coalesce(nullIf(tupleElement(p, 'syclid'), ''), nullIf(toValidUTF8(left(decodeURLFormComponent(extractURLParameter(_attribution_url, 'syclid')), 2048)), '')) AS syclid,
+ coalesce(nullIf(tupleElement(p, '_bhlid'), ''), nullIf(toValidUTF8(left(decodeURLFormComponent(extractURLParameter(_attribution_url, '_bhlid')), 2048)), '')) AS _bhlid,
+ coalesce(nullIf(tupleElement(p, 'rtid'), ''), nullIf(toValidUTF8(left(decodeURLFormComponent(extractURLParameter(_attribution_url, 'rtid')), 2048)), '')) AS rtid,
+ coalesce(nullIf(tupleElement(p, 'oppref'), ''), nullIf(toValidUTF8(left(decodeURLFormComponent(extractURLParameter(_attribution_url, 'oppref')), 2048)), '')) AS oppref,
+ coalesce(nullIf(tupleElement(p, 'bfcid'), ''), nullIf(toValidUTF8(left(decodeURLFormComponent(extractURLParameter(_attribution_url, 'bfcid')), 2048)), '')) AS bfcid,
+ CAST(arrayFilter(x -> (x IS NOT NULL), [if(gclsrc IS NOT NULL, concat('gclsrc=', gclsrc), NULL), if(dclid IS NOT NULL, concat('dclid=', dclid), NULL), if(gbraid IS NOT NULL, concat('gbraid=', gbraid), NULL), if(wbraid IS NOT NULL, concat('wbraid=', wbraid), NULL), if(gad_campaignid IS NOT NULL, concat('gad_campaignid=', gad_campaignid), NULL), if(srsltid IS NOT NULL, concat('srsltid=', srsltid), NULL), if(msclkid IS NOT NULL, concat('msclkid=', msclkid), NULL), if(twclid IS NOT NULL, concat('twclid=', twclid), NULL), if(li_fat_id IS NOT NULL, concat('li_fat_id=', li_fat_id), NULL), if(mc_cid IS NOT NULL, concat('mc_cid=', mc_cid), NULL), if(igshid IS NOT NULL, concat('igshid=', igshid), NULL), if(ttclid IS NOT NULL, concat('ttclid=', ttclid), NULL), if(rdt_cid IS NOT NULL, concat('rdt_cid=', rdt_cid), NULL), if(epik IS NOT NULL, concat('epik=', epik), NULL), if(qclid IS NOT NULL, concat('qclid=', qclid), NULL), if(sccid IS NOT NULL, concat('sccid=', sccid), NULL), if(_kx IS NOT NULL, concat('_kx=', _kx), NULL), if(irclid IS NOT NULL, concat('irclid=', irclid), NULL), if(irclickid IS NOT NULL, concat('irclickid=', irclickid), NULL), if(utm_id IS NOT NULL, concat('utm_id=', utm_id), NULL), if(utm_source_platform IS NOT NULL, concat('utm_source_platform=', utm_source_platform), NULL), if(utm_creative_format IS NOT NULL, concat('utm_creative_format=', utm_creative_format), NULL), if(utm_marketing_tactic IS NOT NULL, concat('utm_marketing_tactic=', utm_marketing_tactic), NULL), if(yclid IS NOT NULL, concat('yclid=', yclid), NULL), if(tblci IS NOT NULL, concat('tblci=', tblci), NULL), if(dicbo IS NOT NULL, concat('dicbo=', dicbo), NULL), if(ndclid IS NOT NULL, concat('ndclid=', ndclid), NULL), if(cjevent IS NOT NULL, concat('cjevent=', cjevent), NULL), if(awc IS NOT NULL, concat('awc=', awc), NULL), if(syclid IS NOT NULL, concat('syclid=', syclid), NULL), if(_bhlid IS NOT NULL, concat('_bhlid=', _bhlid), NULL), if(rtid IS NOT NULL, concat('rtid=', rtid), NULL), if(oppref IS NOT NULL, concat('oppref=', oppref), NULL), if(bfcid IS NOT NULL, concat('bfcid=', bfcid), NULL)]), 'Array(String)') AS ad_ids_set,
+ tupleElement(p, '$host') AS _host,
+ if((event = '$pageview') OR (event = '$screen'), timestamp, timestamp + toIntervalYear(1)) AS pageview_prio_timestamp_min,
+ if((event = '$pageview') OR (event = '$screen'), timestamp, timestamp - toIntervalYear(1)) AS pageview_prio_timestamp_max
+SELECT
+ team_id,
+ `$session_id_uuid` AS session_id_v7,
+ initializeAggregation('argMaxState', source_table.distinct_id, timestamp) AS distinct_id,
+ initializeAggregation('groupUniqArrayState', source_table.distinct_id) AS distinct_ids,
+ timestamp AS min_timestamp,
+ timestamp AS max_timestamp,
+ inserted_at AS max_inserted_at,
+ if(
+ (_current_url IS NOT NULL) AND ((event = '$pageview') OR (event = '$screen')),
+ [_current_url],
+ []
+ ) AS urls,
+ initializeAggregation('argMinState', _current_url, pageview_prio_timestamp_min) AS entry_url,
+ initializeAggregation('argMaxState', _current_url, pageview_prio_timestamp_max) AS end_url,
+ initializeAggregation('argMaxState', _external_click_url, timestamp) AS last_external_click_url,
+ initializeAggregation('argMinState', _browser, timestamp) AS browser,
+ initializeAggregation('argMinState', _browser_version, timestamp) AS browser_version,
+ initializeAggregation('argMinState', _os, timestamp) AS os,
+ initializeAggregation('argMinState', _os_version, timestamp) AS os_version,
+ initializeAggregation('argMinState', _device_type, timestamp) AS device_type,
+ initializeAggregation('argMinState', _viewport_width, timestamp) AS viewport_width,
+ initializeAggregation('argMinState', _viewport_height, timestamp) AS viewport_height,
+ initializeAggregation('argMinState', _geoip_country_code, timestamp) AS geoip_country_code,
+ initializeAggregation('argMinState', _geoip_subdivision_1_code, timestamp) AS geoip_subdivision_1_code,
+ initializeAggregation('argMinState', _geoip_subdivision_1_name, timestamp) AS geoip_subdivision_1_name,
+ initializeAggregation('argMinState', _geoip_subdivision_city_name, timestamp) AS geoip_subdivision_city_name,
+ initializeAggregation('argMinState', _geoip_time_zone, timestamp) AS geoip_time_zone,
+ initializeAggregation('argMinState', _referring_domain, pageview_prio_timestamp_min) AS entry_referring_domain,
+ initializeAggregation('argMinState', _utm_source, pageview_prio_timestamp_min) AS entry_utm_source,
+ initializeAggregation('argMinState', _utm_campaign, pageview_prio_timestamp_min) AS entry_utm_campaign,
+ initializeAggregation('argMinState', _utm_medium, pageview_prio_timestamp_min) AS entry_utm_medium,
+ initializeAggregation('argMinState', _utm_term, pageview_prio_timestamp_min) AS entry_utm_term,
+ initializeAggregation('argMinState', _utm_content, pageview_prio_timestamp_min) AS entry_utm_content,
+ initializeAggregation('argMinState', _gclid, pageview_prio_timestamp_min) AS entry_gclid,
+ initializeAggregation('argMinState', _gad_source, pageview_prio_timestamp_min) AS entry_gad_source,
+ initializeAggregation('argMinState', _fbclid, pageview_prio_timestamp_min) AS entry_fbclid,
+ initializeAggregation('argMinState', nullIf(_gclid, '') IS NOT NULL, pageview_prio_timestamp_min) AS entry_has_gclid,
+ initializeAggregation('argMinState', nullIf(_fbclid, '') IS NOT NULL, pageview_prio_timestamp_min) AS entry_has_fbclid,
+ initializeAggregation('argMinState', ad_ids_set, pageview_prio_timestamp_min) AS entry_ad_ids_set,
+ initializeAggregation(
+ 'argMinState',
+ tuple(
+ _utm_source,
+ _utm_medium,
+ _utm_campaign,
+ _referring_domain,
+ nullIf(_gclid, '') IS NOT NULL,
+ nullIf(_fbclid, '') IS NOT NULL,
+ _gad_source
+ ),
+ pageview_prio_timestamp_min
+ ) AS entry_channel_type_properties,
+ initializeAggregation('uniqState', if(event = '$pageview', uuid, NULL)) AS pageview_uniq,
+ initializeAggregation('uniqState', if(event = '$autocapture', uuid, NULL)) AS autocapture_uniq,
+ initializeAggregation('uniqState', if(event = '$screen', uuid, NULL)) AS screen_uniq,
+ initializeAggregation(
+ 'uniqUpToState(1)',
+ if((event = '$pageview') OR (event = '$screen'), uuid, NULL)
+ ) AS page_screen_uniq_up_to,
+ event = '$autocapture' AS has_autocapture,
+ arrayMap(
+ (k, v) -> concat(k, '=', v),
+ mapKeys(properties_group_feature_flags),
+ mapValues(properties_group_feature_flags)
+ ) AS flag_key_values,
+ [event] AS event_names,
+ if((_host IS NOT NULL) AND (_host != ''), [_host], []) AS hosts,
+ if((_person_email IS NOT NULL) AND (_person_email != ''), [_person_email], []) AS emails,
+ false AS has_replay_events
+FROM posthog.sharded_events AS source_table
+WHERE
+ (bitAnd(bitShiftRight(toUInt128(accurateCastOrNull(`$session_id`, 'UUID')), 76), 15) = 7)
+AND
+ ((fromUnixTimestamp64Milli(toUInt64(bitShiftRight(`$session_id_uuid`, 80))) >= (timestamp - toIntervalDay(31))) AND (fromUnixTimestamp64Milli(toUInt64(bitShiftRight(`$session_id_uuid`, 80))) <= (timestamp + toIntervalDay(31))))
+AND
+ true
diff --git a/posthog/clickhouse/hcl/roles/data/local/sql/raw_sessions_v3_recordings_mv.sql b/posthog/clickhouse/hcl/roles/data/local/sql/raw_sessions_v3_recordings_mv.sql
new file mode 100644
index 000000000000..39fa96669a0a
--- /dev/null
+++ b/posthog/clickhouse/hcl/roles/data/local/sql/raw_sessions_v3_recordings_mv.sql
@@ -0,0 +1,66 @@
+WITH
+ min_first_timestamp AS timestamp,
+ CAST(fromUnixTimestamp64Milli(9223372036854775), 'DateTime64(6)') AS max_ts_64,
+ CAST(fromUnixTimestamp64Milli(-9223372036854775), 'DateTime64(6)') AS min_ts_64,
+ CAST(NULL, 'Nullable(String)') AS null_s,
+ CAST(NULL, 'Nullable(Int64)') AS null_i64,
+ CAST(NULL, 'Nullable(UUID)') AS null_uuid
+SELECT
+ team_id,
+ toUInt128(accurateCast(session_id, 'UUID')) AS session_id_v7,
+ fromUnixTimestamp64Milli(toUInt64(bitShiftRight(session_id_v7, 80))) AS session_timestamp,
+ initializeAggregation('argMaxState', source_table.distinct_id, min_ts_64) AS distinct_id,
+ initializeAggregation('groupUniqArrayState', source_table.distinct_id) AS distinct_ids,
+ timestamp AS min_timestamp,
+ timestamp AS max_timestamp,
+ fromUnixTimestamp(0) AS max_inserted_at,
+ CAST([], 'Array(String)') AS urls,
+ initializeAggregation('argMinState', null_s, max_ts_64) AS entry_url,
+ initializeAggregation('argMaxState', null_s, min_ts_64) AS end_url,
+ initializeAggregation('argMaxState', null_s, min_ts_64) AS last_external_click_url,
+ initializeAggregation('argMinState', null_s, max_ts_64) AS browser,
+ initializeAggregation('argMinState', null_s, max_ts_64) AS browser_version,
+ initializeAggregation('argMinState', null_s, max_ts_64) AS os,
+ initializeAggregation('argMinState', null_s, max_ts_64) AS os_version,
+ initializeAggregation('argMinState', null_s, max_ts_64) AS device_type,
+ initializeAggregation('argMinState', null_i64, max_ts_64) AS viewport_width,
+ initializeAggregation('argMinState', null_i64, max_ts_64) AS viewport_height,
+ initializeAggregation('argMinState', null_s, max_ts_64) AS geoip_country_code,
+ initializeAggregation('argMinState', null_s, max_ts_64) AS geoip_subdivision_1_code,
+ initializeAggregation('argMinState', null_s, max_ts_64) AS geoip_subdivision_1_name,
+ initializeAggregation('argMinState', null_s, max_ts_64) AS geoip_subdivision_city_name,
+ initializeAggregation('argMinState', null_s, max_ts_64) AS geoip_time_zone,
+ initializeAggregation('argMinState', null_s, max_ts_64) AS entry_referring_domain,
+ initializeAggregation('argMinState', null_s, max_ts_64) AS entry_utm_source,
+ initializeAggregation('argMinState', null_s, max_ts_64) AS entry_utm_campaign,
+ initializeAggregation('argMinState', null_s, max_ts_64) AS entry_utm_medium,
+ initializeAggregation('argMinState', null_s, max_ts_64) AS entry_utm_term,
+ initializeAggregation('argMinState', null_s, max_ts_64) AS entry_utm_content,
+ initializeAggregation('argMinState', null_s, max_ts_64) AS entry_gclid,
+ initializeAggregation('argMinState', null_s, max_ts_64) AS entry_gad_source,
+ initializeAggregation('argMinState', null_s, max_ts_64) AS entry_fbclid,
+ initializeAggregation('argMinState', false, max_ts_64) AS entry_has_gclid,
+ initializeAggregation('argMinState', false, max_ts_64) AS entry_has_fbclid,
+ initializeAggregation('argMinState', CAST([], 'Array(String)'), max_ts_64) AS entry_ad_ids_set,
+ initializeAggregation(
+ 'argMinState',
+ tuple(null_s, null_s, null_s, null_s, false, false, null_s),
+ max_ts_64
+ ) AS entry_channel_type_properties,
+ initializeAggregation('uniqState', null_uuid) AS pageview_uniq,
+ initializeAggregation('uniqState', null_uuid) AS autocapture_uniq,
+ initializeAggregation('uniqState', null_uuid) AS screen_uniq,
+ initializeAggregation('uniqUpToState(1)', null_uuid) AS page_screen_uniq_up_to,
+ false AS has_autocapture,
+ CAST([], 'Array(String)') AS flag_key_values,
+ CAST([], 'Array(String)') AS event_names,
+ CAST([], 'Array(String)') AS hosts,
+ CAST([], 'Array(String)') AS emails,
+ true AS has_replay_events
+FROM posthog.sharded_session_replay_events AS source_table
+WHERE
+ (bitAnd(bitShiftRight(toUInt128(accurateCastOrNull(session_id, 'UUID')), 76), 15) = 7)
+AND
+ ((fromUnixTimestamp64Milli(toUInt64(bitShiftRight(toUInt128(accurateCastOrNull(session_id, 'UUID')), 80))) >= (min_first_timestamp - toIntervalDay(31))) AND (fromUnixTimestamp64Milli(toUInt64(bitShiftRight(toUInt128(accurateCastOrNull(session_id, 'UUID')), 80))) <= (min_first_timestamp + toIntervalDay(31))))
+AND
+ true
diff --git a/posthog/clickhouse/hcl/roles/data/local/tables.hcl b/posthog/clickhouse/hcl/roles/data/local/tables.hcl
index 3543b2d9d979..58bf02876f9a 100644
--- a/posthog/clickhouse/hcl/roles/data/local/tables.hcl
+++ b/posthog/clickhouse/hcl/roles/data/local/tables.hcl
@@ -1260,6 +1260,297 @@ database "posthog" {
type = "AggregateFunction(argMin, Nullable(Float64), DateTime64(6, 'UTC'))"
}
}
+ materialized_view "raw_sessions_v3_mv" {
+ to_table = "posthog.writable_raw_sessions_v3"
+ query = file("sql/raw_sessions_v3_mv.sql")
+
+ column "team_id" {
+ type = "Int64"
+ }
+ column "session_id_v7" {
+ type = "Nullable(UInt128)"
+ }
+ column "distinct_id" {
+ type = "AggregateFunction(argMax, String, DateTime64(6, 'UTC'))"
+ }
+ column "distinct_ids" {
+ type = "AggregateFunction(groupUniqArray, String)"
+ }
+ column "min_timestamp" {
+ type = "DateTime64(6, 'UTC')"
+ }
+ column "max_timestamp" {
+ type = "DateTime64(6, 'UTC')"
+ }
+ column "max_inserted_at" {
+ type = "Nullable(DateTime64(6, 'UTC'))"
+ }
+ column "urls" {
+ type = "Array(Nullable(String))"
+ }
+ column "entry_url" {
+ type = "AggregateFunction(argMin, Nullable(String), DateTime64(6, 'UTC'))"
+ }
+ column "end_url" {
+ type = "AggregateFunction(argMax, Nullable(String), DateTime64(6, 'UTC'))"
+ }
+ column "last_external_click_url" {
+ type = "AggregateFunction(argMax, Nullable(String), DateTime64(6, 'UTC'))"
+ }
+ column "browser" {
+ type = "AggregateFunction(argMin, Nullable(String), DateTime64(6, 'UTC'))"
+ }
+ column "browser_version" {
+ type = "AggregateFunction(argMin, Nullable(String), DateTime64(6, 'UTC'))"
+ }
+ column "os" {
+ type = "AggregateFunction(argMin, Nullable(String), DateTime64(6, 'UTC'))"
+ }
+ column "os_version" {
+ type = "AggregateFunction(argMin, Nullable(String), DateTime64(6, 'UTC'))"
+ }
+ column "device_type" {
+ type = "AggregateFunction(argMin, Nullable(String), DateTime64(6, 'UTC'))"
+ }
+ column "viewport_width" {
+ type = "AggregateFunction(argMin, Nullable(Int64), DateTime64(6, 'UTC'))"
+ }
+ column "viewport_height" {
+ type = "AggregateFunction(argMin, Nullable(Int64), DateTime64(6, 'UTC'))"
+ }
+ column "geoip_country_code" {
+ type = "AggregateFunction(argMin, Nullable(String), DateTime64(6, 'UTC'))"
+ }
+ column "geoip_subdivision_1_code" {
+ type = "AggregateFunction(argMin, Nullable(String), DateTime64(6, 'UTC'))"
+ }
+ column "geoip_subdivision_1_name" {
+ type = "AggregateFunction(argMin, Nullable(String), DateTime64(6, 'UTC'))"
+ }
+ column "geoip_subdivision_city_name" {
+ type = "AggregateFunction(argMin, Nullable(String), DateTime64(6, 'UTC'))"
+ }
+ column "geoip_time_zone" {
+ type = "AggregateFunction(argMin, Nullable(String), DateTime64(6, 'UTC'))"
+ }
+ column "entry_referring_domain" {
+ type = "AggregateFunction(argMin, Nullable(String), DateTime64(6, 'UTC'))"
+ }
+ column "entry_utm_source" {
+ type = "AggregateFunction(argMin, Nullable(String), DateTime64(6, 'UTC'))"
+ }
+ column "entry_utm_campaign" {
+ type = "AggregateFunction(argMin, Nullable(String), DateTime64(6, 'UTC'))"
+ }
+ column "entry_utm_medium" {
+ type = "AggregateFunction(argMin, Nullable(String), DateTime64(6, 'UTC'))"
+ }
+ column "entry_utm_term" {
+ type = "AggregateFunction(argMin, Nullable(String), DateTime64(6, 'UTC'))"
+ }
+ column "entry_utm_content" {
+ type = "AggregateFunction(argMin, Nullable(String), DateTime64(6, 'UTC'))"
+ }
+ column "entry_gclid" {
+ type = "AggregateFunction(argMin, Nullable(String), DateTime64(6, 'UTC'))"
+ }
+ column "entry_gad_source" {
+ type = "AggregateFunction(argMin, Nullable(String), DateTime64(6, 'UTC'))"
+ }
+ column "entry_fbclid" {
+ type = "AggregateFunction(argMin, Nullable(String), DateTime64(6, 'UTC'))"
+ }
+ column "entry_has_gclid" {
+ type = "AggregateFunction(argMin, UInt8, DateTime64(6, 'UTC'))"
+ }
+ column "entry_has_fbclid" {
+ type = "AggregateFunction(argMin, UInt8, DateTime64(6, 'UTC'))"
+ }
+ column "entry_ad_ids_set" {
+ type = "AggregateFunction(argMin, Array(String), DateTime64(6, 'UTC'))"
+ }
+ column "entry_channel_type_properties" {
+ type = "AggregateFunction(argMin, Tuple(Nullable(String), Nullable(String), Nullable(String), Nullable(String), UInt8, UInt8, Nullable(String)), DateTime64(6, 'UTC'))"
+ }
+ column "pageview_uniq" {
+ type = "AggregateFunction(uniq, Nullable(UUID))"
+ }
+ column "autocapture_uniq" {
+ type = "AggregateFunction(uniq, Nullable(UUID))"
+ }
+ column "screen_uniq" {
+ type = "AggregateFunction(uniq, Nullable(UUID))"
+ }
+ column "page_screen_uniq_up_to" {
+ type = "AggregateFunction(uniqUpTo(1), Nullable(UUID))"
+ }
+ column "has_autocapture" {
+ type = "UInt8"
+ }
+ column "flag_key_values" {
+ type = "Array(String)"
+ }
+ column "event_names" {
+ type = "Array(String)"
+ }
+ column "hosts" {
+ type = "Array(Nullable(String))"
+ }
+ column "emails" {
+ type = "Array(String)"
+ }
+ column "has_replay_events" {
+ type = "Bool"
+ }
+ }
+
+ materialized_view "raw_sessions_v3_recordings_mv" {
+ to_table = "posthog.writable_raw_sessions_v3"
+ query = file("sql/raw_sessions_v3_recordings_mv.sql")
+
+ column "team_id" {
+ type = "Int64"
+ }
+ column "session_id_v7" {
+ type = "UInt128"
+ }
+ column "session_timestamp" {
+ type = "DateTime64(3)"
+ }
+ column "distinct_id" {
+ type = "AggregateFunction(argMax, String, DateTime64(6))"
+ }
+ column "distinct_ids" {
+ type = "AggregateFunction(groupUniqArray, String)"
+ }
+ column "min_timestamp" {
+ type = "SimpleAggregateFunction(min, DateTime64(6, 'UTC'))"
+ }
+ column "max_timestamp" {
+ type = "SimpleAggregateFunction(min, DateTime64(6, 'UTC'))"
+ }
+ column "max_inserted_at" {
+ type = "DateTime"
+ }
+ column "urls" {
+ type = "Array(String)"
+ }
+ column "entry_url" {
+ type = "AggregateFunction(argMin, Nullable(String), DateTime64(6))"
+ }
+ column "end_url" {
+ type = "AggregateFunction(argMax, Nullable(String), DateTime64(6))"
+ }
+ column "last_external_click_url" {
+ type = "AggregateFunction(argMax, Nullable(String), DateTime64(6))"
+ }
+ column "browser" {
+ type = "AggregateFunction(argMin, Nullable(String), DateTime64(6))"
+ }
+ column "browser_version" {
+ type = "AggregateFunction(argMin, Nullable(String), DateTime64(6))"
+ }
+ column "os" {
+ type = "AggregateFunction(argMin, Nullable(String), DateTime64(6))"
+ }
+ column "os_version" {
+ type = "AggregateFunction(argMin, Nullable(String), DateTime64(6))"
+ }
+ column "device_type" {
+ type = "AggregateFunction(argMin, Nullable(String), DateTime64(6))"
+ }
+ column "viewport_width" {
+ type = "AggregateFunction(argMin, Nullable(Int64), DateTime64(6))"
+ }
+ column "viewport_height" {
+ type = "AggregateFunction(argMin, Nullable(Int64), DateTime64(6))"
+ }
+ column "geoip_country_code" {
+ type = "AggregateFunction(argMin, Nullable(String), DateTime64(6))"
+ }
+ column "geoip_subdivision_1_code" {
+ type = "AggregateFunction(argMin, Nullable(String), DateTime64(6))"
+ }
+ column "geoip_subdivision_1_name" {
+ type = "AggregateFunction(argMin, Nullable(String), DateTime64(6))"
+ }
+ column "geoip_subdivision_city_name" {
+ type = "AggregateFunction(argMin, Nullable(String), DateTime64(6))"
+ }
+ column "geoip_time_zone" {
+ type = "AggregateFunction(argMin, Nullable(String), DateTime64(6))"
+ }
+ column "entry_referring_domain" {
+ type = "AggregateFunction(argMin, Nullable(String), DateTime64(6))"
+ }
+ column "entry_utm_source" {
+ type = "AggregateFunction(argMin, Nullable(String), DateTime64(6))"
+ }
+ column "entry_utm_campaign" {
+ type = "AggregateFunction(argMin, Nullable(String), DateTime64(6))"
+ }
+ column "entry_utm_medium" {
+ type = "AggregateFunction(argMin, Nullable(String), DateTime64(6))"
+ }
+ column "entry_utm_term" {
+ type = "AggregateFunction(argMin, Nullable(String), DateTime64(6))"
+ }
+ column "entry_utm_content" {
+ type = "AggregateFunction(argMin, Nullable(String), DateTime64(6))"
+ }
+ column "entry_gclid" {
+ type = "AggregateFunction(argMin, Nullable(String), DateTime64(6))"
+ }
+ column "entry_gad_source" {
+ type = "AggregateFunction(argMin, Nullable(String), DateTime64(6))"
+ }
+ column "entry_fbclid" {
+ type = "AggregateFunction(argMin, Nullable(String), DateTime64(6))"
+ }
+ column "entry_has_gclid" {
+ type = "AggregateFunction(argMin, Bool, DateTime64(6))"
+ }
+ column "entry_has_fbclid" {
+ type = "AggregateFunction(argMin, Bool, DateTime64(6))"
+ }
+ column "entry_ad_ids_set" {
+ type = "AggregateFunction(argMin, Array(String), DateTime64(6))"
+ }
+ column "entry_channel_type_properties" {
+ type = "AggregateFunction(argMin, Tuple(Nullable(String), Nullable(String), Nullable(String), Nullable(String), Bool, Bool, Nullable(String)), DateTime64(6))"
+ }
+ column "pageview_uniq" {
+ type = "AggregateFunction(uniq, Nullable(UUID))"
+ }
+ column "autocapture_uniq" {
+ type = "AggregateFunction(uniq, Nullable(UUID))"
+ }
+ column "screen_uniq" {
+ type = "AggregateFunction(uniq, Nullable(UUID))"
+ }
+ column "page_screen_uniq_up_to" {
+ type = "AggregateFunction(uniqUpTo(1), Nullable(UUID))"
+ }
+ column "has_autocapture" {
+ type = "Bool"
+ }
+ column "flag_key_values" {
+ type = "Array(String)"
+ }
+ column "event_names" {
+ type = "Array(String)"
+ }
+ column "hosts" {
+ type = "Array(String)"
+ }
+ column "emails" {
+ type = "Array(String)"
+ }
+ column "has_replay_events" {
+ type = "Bool"
+ }
+ }
+
materialized_view "sessions_mv" {
to_table = "posthog.writable_sessions"
query = file("sql/sessions_mv.sql")
diff --git a/posthog/clickhouse/hcl/roles/data/prod-eu/tables.hcl b/posthog/clickhouse/hcl/roles/data/prod-eu/tables.hcl
index 306c500007f7..bd4d4c6a356c 100644
--- a/posthog/clickhouse/hcl/roles/data/prod-eu/tables.hcl
+++ b/posthog/clickhouse/hcl/roles/data/prod-eu/tables.hcl
@@ -725,9 +725,6 @@ database "posthog" {
column "entry_has_fbclid" {
type = "AggregateFunction(argMin, Bool, DateTime64(6, 'UTC'))"
}
- column "entry_ad_ids_map" {
- type = "AggregateFunction(argMin, Map(String, String), DateTime64(6, 'UTC'))"
- }
column "entry_ad_ids_set" {
type = "AggregateFunction(argMin, Array(String), DateTime64(6, 'UTC'))"
}
@@ -735,13 +732,13 @@ database "posthog" {
type = "AggregateFunction(argMin, Tuple(Nullable(String), Nullable(String), Nullable(String), Nullable(String), Bool, Bool, Nullable(String)), DateTime64(6, 'UTC'))"
}
column "pageview_uniq" {
- type = "AggregateFunction(uniqExact, Nullable(UUID))"
+ type = "AggregateFunction(uniq, Nullable(UUID))"
}
column "autocapture_uniq" {
- type = "AggregateFunction(uniqExact, Nullable(UUID))"
+ type = "AggregateFunction(uniq, Nullable(UUID))"
}
column "screen_uniq" {
- type = "AggregateFunction(uniqExact, Nullable(UUID))"
+ type = "AggregateFunction(uniq, Nullable(UUID))"
}
column "page_screen_uniq_up_to" {
type = "AggregateFunction(uniqUpTo(1), Nullable(UUID))"
@@ -752,12 +749,6 @@ database "posthog" {
column "flag_key_values" {
type = "SimpleAggregateFunction(groupUniqArrayArray(10000), Array(String))"
}
- column "flag_values" {
- type = "AggregateFunction(groupUniqArrayMap, Map(String, String))"
- }
- column "flag_keys" {
- type = "SimpleAggregateFunction(groupUniqArrayArray, Array(String))"
- }
column "event_names" {
type = "SimpleAggregateFunction(groupUniqArrayArray, Array(String))"
}
@@ -775,15 +766,10 @@ database "posthog" {
type = "bloom_filter()"
granularity = 1
}
- index "flag_keys_bloom_filter" {
- expr = "flag_keys"
- type = "bloom_filter()"
- granularity = 1
- }
- index "flag_key_values_bloom_filter" {
+ index "flag_key_values_text" {
expr = "flag_key_values"
- type = "bloom_filter()"
- granularity = 1
+ type = "text(tokenizer = splitByString(['=']))"
+ granularity = 100000000
}
index "hosts_bloom_filter" {
expr = "hosts"
diff --git a/posthog/clickhouse/hcl/roles/data/prod-us/tables.hcl b/posthog/clickhouse/hcl/roles/data/prod-us/tables.hcl
index 0fbcb00fa84e..62429d6afac1 100644
--- a/posthog/clickhouse/hcl/roles/data/prod-us/tables.hcl
+++ b/posthog/clickhouse/hcl/roles/data/prod-us/tables.hcl
@@ -795,9 +795,6 @@ database "posthog" {
column "entry_has_fbclid" {
type = "AggregateFunction(argMin, Bool, DateTime64(6, 'UTC'))"
}
- column "entry_ad_ids_map" {
- type = "AggregateFunction(argMin, Map(String, String), DateTime64(6, 'UTC'))"
- }
column "entry_ad_ids_set" {
type = "AggregateFunction(argMin, Array(String), DateTime64(6, 'UTC'))"
}
@@ -805,13 +802,13 @@ database "posthog" {
type = "AggregateFunction(argMin, Tuple(Nullable(String), Nullable(String), Nullable(String), Nullable(String), Bool, Bool, Nullable(String)), DateTime64(6, 'UTC'))"
}
column "pageview_uniq" {
- type = "AggregateFunction(uniqExact, Nullable(UUID))"
+ type = "AggregateFunction(uniq, Nullable(UUID))"
}
column "autocapture_uniq" {
- type = "AggregateFunction(uniqExact, Nullable(UUID))"
+ type = "AggregateFunction(uniq, Nullable(UUID))"
}
column "screen_uniq" {
- type = "AggregateFunction(uniqExact, Nullable(UUID))"
+ type = "AggregateFunction(uniq, Nullable(UUID))"
}
column "page_screen_uniq_up_to" {
type = "AggregateFunction(uniqUpTo(1), Nullable(UUID))"
@@ -822,12 +819,6 @@ database "posthog" {
column "flag_key_values" {
type = "SimpleAggregateFunction(groupUniqArrayArray(10000), Array(String))"
}
- column "flag_values" {
- type = "AggregateFunction(groupUniqArrayMap, Map(String, String))"
- }
- column "flag_keys" {
- type = "SimpleAggregateFunction(groupUniqArrayArray, Array(String))"
- }
column "event_names" {
type = "SimpleAggregateFunction(groupUniqArrayArray, Array(String))"
}
@@ -845,15 +836,10 @@ database "posthog" {
type = "bloom_filter()"
granularity = 1
}
- index "flag_keys_bloom_filter" {
- expr = "flag_keys"
- type = "bloom_filter()"
- granularity = 1
- }
- index "flag_key_values_bloom_filter" {
+ index "flag_key_values_text" {
expr = "flag_key_values"
- type = "bloom_filter()"
- granularity = 1
+ type = "text(tokenizer = splitByString(['=']))"
+ granularity = 100000000
}
index "hosts_bloom_filter" {
expr = "hosts"
@@ -1673,6 +1659,15 @@ SQL
}
}
+ patch_table "raw_sessions_v3" {
+ engine "distributed" {
+ cluster_name = "sessions"
+ remote_database = "posthog"
+ remote_table = "raw_sessions_v3"
+ sharding_key = "cityHash64(session_id_v7)"
+ }
+ }
+
patch_table "writable_raw_sessions_v3" {
engine "distributed" {
cluster_name = "sessions"
diff --git a/posthog/clickhouse/hcl/roles/sessions/shared/tables.hcl b/posthog/clickhouse/hcl/roles/sessions/shared/tables.hcl
index 06d7130ad4f3..5f0c97e296a7 100644
--- a/posthog/clickhouse/hcl/roles/sessions/shared/tables.hcl
+++ b/posthog/clickhouse/hcl/roles/sessions/shared/tables.hcl
@@ -388,9 +388,6 @@ database "posthog" {
column "entry_has_fbclid" {
type = "AggregateFunction(argMin, Bool, DateTime64(6, 'UTC'))"
}
- column "entry_ad_ids_map" {
- type = "AggregateFunction(argMin, Map(String, String), DateTime64(6, 'UTC'))"
- }
column "entry_ad_ids_set" {
type = "AggregateFunction(argMin, Array(String), DateTime64(6, 'UTC'))"
}
@@ -398,13 +395,13 @@ database "posthog" {
type = "AggregateFunction(argMin, Tuple(Nullable(String), Nullable(String), Nullable(String), Nullable(String), Bool, Bool, Nullable(String)), DateTime64(6, 'UTC'))"
}
column "pageview_uniq" {
- type = "AggregateFunction(uniqExact, Nullable(UUID))"
+ type = "AggregateFunction(uniq, Nullable(UUID))"
}
column "autocapture_uniq" {
- type = "AggregateFunction(uniqExact, Nullable(UUID))"
+ type = "AggregateFunction(uniq, Nullable(UUID))"
}
column "screen_uniq" {
- type = "AggregateFunction(uniqExact, Nullable(UUID))"
+ type = "AggregateFunction(uniq, Nullable(UUID))"
}
column "page_screen_uniq_up_to" {
type = "AggregateFunction(uniqUpTo(1), Nullable(UUID))"
@@ -415,12 +412,6 @@ database "posthog" {
column "flag_key_values" {
type = "SimpleAggregateFunction(groupUniqArrayArray(10000), Array(String))"
}
- column "flag_values" {
- type = "AggregateFunction(groupUniqArrayMap, Map(String, String))"
- }
- column "flag_keys" {
- type = "SimpleAggregateFunction(groupUniqArrayArray, Array(String))"
- }
column "event_names" {
type = "SimpleAggregateFunction(groupUniqArrayArray, Array(String))"
}
@@ -438,15 +429,10 @@ database "posthog" {
type = "bloom_filter()"
granularity = 1
}
- index "flag_keys_bloom_filter" {
- expr = "flag_keys"
- type = "bloom_filter()"
- granularity = 1
- }
- index "flag_key_values_bloom_filter" {
+ index "flag_key_values_text" {
expr = "flag_key_values"
- type = "bloom_filter()"
- granularity = 1
+ type = "text(tokenizer = splitByString(['=']))"
+ granularity = 100000000
}
engine "replicated_aggregating_merge_tree" {
zoo_path = "/clickhouse/tables/sessions/noshard/posthog.raw_sessions_v3"
diff --git a/posthog/clickhouse/hcl/sql/dev/data.sql b/posthog/clickhouse/hcl/sql/dev/data.sql
index 45d354945d44..ce3aed1d9771 100644
--- a/posthog/clickhouse/hcl/sql/dev/data.sql
+++ b/posthog/clickhouse/hcl/sql/dev/data.sql
@@ -1439,24 +1439,20 @@ CREATE TABLE posthog.sharded_raw_sessions_v3 (
entry_fbclid AggregateFunction(argMin, Nullable(String), DateTime64(6, 'UTC')),
entry_has_gclid AggregateFunction(argMin, Bool, DateTime64(6, 'UTC')),
entry_has_fbclid AggregateFunction(argMin, Bool, DateTime64(6, 'UTC')),
- entry_ad_ids_map AggregateFunction(argMin, Map(String, String), DateTime64(6, 'UTC')),
entry_ad_ids_set AggregateFunction(argMin, Array(String), DateTime64(6, 'UTC')),
entry_channel_type_properties AggregateFunction(argMin, Tuple(Nullable(String), Nullable(String), Nullable(String), Nullable(String), Bool, Bool, Nullable(String)), DateTime64(6, 'UTC')),
- pageview_uniq AggregateFunction(uniqExact, Nullable(UUID)),
- autocapture_uniq AggregateFunction(uniqExact, Nullable(UUID)),
- screen_uniq AggregateFunction(uniqExact, Nullable(UUID)),
+ pageview_uniq AggregateFunction(uniq, Nullable(UUID)),
+ autocapture_uniq AggregateFunction(uniq, Nullable(UUID)),
+ screen_uniq AggregateFunction(uniq, Nullable(UUID)),
page_screen_uniq_up_to AggregateFunction(uniqUpTo(1), Nullable(UUID)),
has_autocapture SimpleAggregateFunction(max, Bool),
flag_key_values SimpleAggregateFunction(groupUniqArrayArray(10000), Array(String)),
- flag_values AggregateFunction(groupUniqArrayMap, Map(String, String)),
- flag_keys SimpleAggregateFunction(groupUniqArrayArray, Array(String)),
event_names SimpleAggregateFunction(groupUniqArrayArray, Array(String)),
hosts SimpleAggregateFunction(groupUniqArrayArray(100), Array(String)),
emails SimpleAggregateFunction(groupUniqArrayArray(10), Array(String)),
has_replay_events SimpleAggregateFunction(max, Bool),
INDEX event_names_bloom_filter event_names TYPE bloom_filter() GRANULARITY 1,
- INDEX flag_keys_bloom_filter flag_keys TYPE bloom_filter() GRANULARITY 1,
- INDEX flag_key_values_bloom_filter flag_key_values TYPE bloom_filter() GRANULARITY 1,
+ INDEX flag_key_values_text flag_key_values TYPE text(tokenizer = splitByString(['='])) GRANULARITY 100000000,
INDEX hosts_bloom_filter hosts TYPE bloom_filter() GRANULARITY 1,
INDEX emails_bloom_filter emails TYPE bloom_filter() GRANULARITY 1
) ENGINE = ReplicatedAggregatingMergeTree('/clickhouse/tables/{shard}/posthog.raw_sessions_v3', '{replica}') ORDER BY (team_id, session_timestamp, session_id_v7) PARTITION BY toYYYYMM(session_timestamp) SETTINGS index_granularity = 8192;
@@ -2190,17 +2186,14 @@ CREATE TABLE posthog.writable_raw_sessions_v3 (
entry_fbclid AggregateFunction(argMin, Nullable(String), DateTime64(6, 'UTC')),
entry_has_gclid AggregateFunction(argMin, Bool, DateTime64(6, 'UTC')),
entry_has_fbclid AggregateFunction(argMin, Bool, DateTime64(6, 'UTC')),
- entry_ad_ids_map AggregateFunction(argMin, Map(String, String), DateTime64(6, 'UTC')),
entry_ad_ids_set AggregateFunction(argMin, Array(String), DateTime64(6, 'UTC')),
entry_channel_type_properties AggregateFunction(argMin, Tuple(Nullable(String), Nullable(String), Nullable(String), Nullable(String), Bool, Bool, Nullable(String)), DateTime64(6, 'UTC')),
- pageview_uniq AggregateFunction(uniqExact, Nullable(UUID)),
- autocapture_uniq AggregateFunction(uniqExact, Nullable(UUID)),
- screen_uniq AggregateFunction(uniqExact, Nullable(UUID)),
+ pageview_uniq AggregateFunction(uniq, Nullable(UUID)),
+ autocapture_uniq AggregateFunction(uniq, Nullable(UUID)),
+ screen_uniq AggregateFunction(uniq, Nullable(UUID)),
page_screen_uniq_up_to AggregateFunction(uniqUpTo(1), Nullable(UUID)),
has_autocapture SimpleAggregateFunction(max, Bool),
flag_key_values SimpleAggregateFunction(groupUniqArrayArray(10000), Array(String)),
- flag_values AggregateFunction(groupUniqArrayMap, Map(String, String)),
- flag_keys SimpleAggregateFunction(groupUniqArrayArray, Array(String)),
event_names SimpleAggregateFunction(groupUniqArrayArray, Array(String)),
hosts SimpleAggregateFunction(groupUniqArrayArray(100), Array(String)),
emails SimpleAggregateFunction(groupUniqArrayArray(10), Array(String)),
@@ -3041,17 +3034,14 @@ CREATE TABLE posthog.raw_sessions_v3 (
entry_fbclid AggregateFunction(argMin, Nullable(String), DateTime64(6, 'UTC')),
entry_has_gclid AggregateFunction(argMin, Bool, DateTime64(6, 'UTC')),
entry_has_fbclid AggregateFunction(argMin, Bool, DateTime64(6, 'UTC')),
- entry_ad_ids_map AggregateFunction(argMin, Map(String, String), DateTime64(6, 'UTC')),
entry_ad_ids_set AggregateFunction(argMin, Array(String), DateTime64(6, 'UTC')),
entry_channel_type_properties AggregateFunction(argMin, Tuple(Nullable(String), Nullable(String), Nullable(String), Nullable(String), Bool, Bool, Nullable(String)), DateTime64(6, 'UTC')),
- pageview_uniq AggregateFunction(uniqExact, Nullable(UUID)),
- autocapture_uniq AggregateFunction(uniqExact, Nullable(UUID)),
- screen_uniq AggregateFunction(uniqExact, Nullable(UUID)),
+ pageview_uniq AggregateFunction(uniq, Nullable(UUID)),
+ autocapture_uniq AggregateFunction(uniq, Nullable(UUID)),
+ screen_uniq AggregateFunction(uniq, Nullable(UUID)),
page_screen_uniq_up_to AggregateFunction(uniqUpTo(1), Nullable(UUID)),
has_autocapture SimpleAggregateFunction(max, Bool),
flag_key_values SimpleAggregateFunction(groupUniqArrayArray(10000), Array(String)),
- flag_values AggregateFunction(groupUniqArrayMap, Map(String, String)),
- flag_keys SimpleAggregateFunction(groupUniqArrayArray, Array(String)),
event_names SimpleAggregateFunction(groupUniqArrayArray, Array(String)),
hosts SimpleAggregateFunction(groupUniqArrayArray(100), Array(String)),
emails SimpleAggregateFunction(groupUniqArrayArray(10), Array(String)),
@@ -3403,16 +3393,14 @@ CREATE VIEW posthog.raw_sessions_v3_v AS SELECT
argMinMerge(entry_fbclid) AS entry_fbclid,
argMinMerge(entry_has_gclid) AS entry_has_gclid,
argMinMerge(entry_has_fbclid) AS entry_has_fbclid,
- argMinMerge(entry_ad_ids_map) AS entry_ad_ids_map,
argMinMerge(entry_ad_ids_set) AS entry_ad_ids_set,
argMinMerge(entry_channel_type_properties) AS entry_channel_type_properties,
- uniqExactMerge(pageview_uniq) AS pageview_uniq,
- uniqExactMerge(autocapture_uniq) AS autocapture_uniq,
- uniqExactMerge(screen_uniq) AS screen_uniq,
+ uniqMerge(pageview_uniq) AS pageview_uniq,
+ uniqMerge(autocapture_uniq) AS autocapture_uniq,
+ uniqMerge(screen_uniq) AS screen_uniq,
uniqUpToMerge(1)(page_screen_uniq_up_to) AS page_screen_uniq_up_to,
max(has_autocapture) AS has_autocapture,
groupUniqArrayArray(10000)(flag_key_values) AS flag_key_values,
- groupUniqArrayArray(flag_keys) AS flag_keys,
groupUniqArrayArray(2000)(event_names) AS event_names,
groupUniqArrayArray(100)(hosts) AS hosts,
groupUniqArrayArray(10)(emails) AS emails,
diff --git a/posthog/clickhouse/hcl/sql/dev/sessions.sql b/posthog/clickhouse/hcl/sql/dev/sessions.sql
index e8f3a288322e..f789c2ed6c8b 100644
--- a/posthog/clickhouse/hcl/sql/dev/sessions.sql
+++ b/posthog/clickhouse/hcl/sql/dev/sessions.sql
@@ -320,24 +320,20 @@ CREATE TABLE posthog.raw_sessions_v3 (
entry_fbclid AggregateFunction(argMin, Nullable(String), DateTime64(6, 'UTC')),
entry_has_gclid AggregateFunction(argMin, Bool, DateTime64(6, 'UTC')),
entry_has_fbclid AggregateFunction(argMin, Bool, DateTime64(6, 'UTC')),
- entry_ad_ids_map AggregateFunction(argMin, Map(String, String), DateTime64(6, 'UTC')),
entry_ad_ids_set AggregateFunction(argMin, Array(String), DateTime64(6, 'UTC')),
entry_channel_type_properties AggregateFunction(argMin, Tuple(Nullable(String), Nullable(String), Nullable(String), Nullable(String), Bool, Bool, Nullable(String)), DateTime64(6, 'UTC')),
- pageview_uniq AggregateFunction(uniqExact, Nullable(UUID)),
- autocapture_uniq AggregateFunction(uniqExact, Nullable(UUID)),
- screen_uniq AggregateFunction(uniqExact, Nullable(UUID)),
+ pageview_uniq AggregateFunction(uniq, Nullable(UUID)),
+ autocapture_uniq AggregateFunction(uniq, Nullable(UUID)),
+ screen_uniq AggregateFunction(uniq, Nullable(UUID)),
page_screen_uniq_up_to AggregateFunction(uniqUpTo(1), Nullable(UUID)),
has_autocapture SimpleAggregateFunction(max, Bool),
flag_key_values SimpleAggregateFunction(groupUniqArrayArray(10000), Array(String)),
- flag_values AggregateFunction(groupUniqArrayMap, Map(String, String)),
- flag_keys SimpleAggregateFunction(groupUniqArrayArray, Array(String)),
event_names SimpleAggregateFunction(groupUniqArrayArray, Array(String)),
hosts SimpleAggregateFunction(groupUniqArrayArray(100), Array(String)),
emails SimpleAggregateFunction(groupUniqArrayArray(10), Array(String)),
has_replay_events SimpleAggregateFunction(max, Bool),
INDEX event_names_bloom_filter event_names TYPE bloom_filter() GRANULARITY 1,
- INDEX flag_keys_bloom_filter flag_keys TYPE bloom_filter() GRANULARITY 1,
- INDEX flag_key_values_bloom_filter flag_key_values TYPE bloom_filter() GRANULARITY 1
+ INDEX flag_key_values_text flag_key_values TYPE text(tokenizer = splitByString(['='])) GRANULARITY 100000000
) ENGINE = ReplicatedAggregatingMergeTree('/clickhouse/tables/sessions/noshard/posthog.raw_sessions_v3', '{shard}-{replica}') ORDER BY (team_id, session_timestamp, session_id_v7) PARTITION BY toYYYYMM(session_timestamp) SETTINGS index_granularity = 8192, storage_policy = 's3_tiered';
CREATE TABLE posthog.sessions (
session_id String,
diff --git a/posthog/clickhouse/hcl/sql/local-multi/data.sql b/posthog/clickhouse/hcl/sql/local-multi/data.sql
index 5bce7827aabb..035e152ffff5 100644
--- a/posthog/clickhouse/hcl/sql/local-multi/data.sql
+++ b/posthog/clickhouse/hcl/sql/local-multi/data.sql
@@ -1436,23 +1436,20 @@ CREATE TABLE posthog.sharded_raw_sessions_v3 (
entry_fbclid AggregateFunction(argMin, Nullable(String), DateTime64(6, 'UTC')),
entry_has_gclid AggregateFunction(argMin, Bool, DateTime64(6, 'UTC')),
entry_has_fbclid AggregateFunction(argMin, Bool, DateTime64(6, 'UTC')),
- entry_ad_ids_map AggregateFunction(argMin, Map(String, String), DateTime64(6, 'UTC')),
entry_ad_ids_set AggregateFunction(argMin, Array(String), DateTime64(6, 'UTC')),
entry_channel_type_properties AggregateFunction(argMin, Tuple(Nullable(String), Nullable(String), Nullable(String), Nullable(String), Bool, Bool, Nullable(String)), DateTime64(6, 'UTC')),
- pageview_uniq AggregateFunction(uniqExact, Nullable(UUID)),
- autocapture_uniq AggregateFunction(uniqExact, Nullable(UUID)),
- screen_uniq AggregateFunction(uniqExact, Nullable(UUID)),
+ pageview_uniq AggregateFunction(uniq, Nullable(UUID)),
+ autocapture_uniq AggregateFunction(uniq, Nullable(UUID)),
+ screen_uniq AggregateFunction(uniq, Nullable(UUID)),
page_screen_uniq_up_to AggregateFunction(uniqUpTo(1), Nullable(UUID)),
has_autocapture SimpleAggregateFunction(max, Bool),
flag_key_values SimpleAggregateFunction(groupUniqArrayArray(10000), Array(String)),
- flag_keys SimpleAggregateFunction(groupUniqArrayArray, Array(String)),
event_names SimpleAggregateFunction(groupUniqArrayArray(2000), Array(String)),
hosts SimpleAggregateFunction(groupUniqArrayArray(100), Array(String)),
emails SimpleAggregateFunction(groupUniqArrayArray(10), Array(String)),
has_replay_events SimpleAggregateFunction(max, Bool),
INDEX event_names_bloom_filter event_names TYPE bloom_filter() GRANULARITY 1,
- INDEX flag_key_values_bloom_filter flag_key_values TYPE bloom_filter() GRANULARITY 1,
- INDEX flag_keys_bloom_filter flag_keys TYPE bloom_filter() GRANULARITY 1,
+ INDEX flag_key_values_text flag_key_values TYPE text(tokenizer = splitByString(['='])) GRANULARITY 100000000,
INDEX hosts_bloom_filter hosts TYPE bloom_filter() GRANULARITY 1,
INDEX emails_bloom_filter emails TYPE bloom_filter() GRANULARITY 1
) ENGINE = ReplicatedAggregatingMergeTree('/clickhouse/tables/{shard}/posthog.raw_sessions_v3', '{replica}') ORDER BY (team_id, session_timestamp, session_id_v7) PARTITION BY toYYYYMM(session_timestamp) SETTINGS index_granularity = 8192, max_delay_to_insert = 10, parts_to_delay_insert = 250, parts_to_throw_insert = 1000;
@@ -2248,16 +2245,14 @@ CREATE TABLE posthog.writable_raw_sessions_v3 (
entry_fbclid AggregateFunction(argMin, Nullable(String), DateTime64(6, 'UTC')),
entry_has_gclid AggregateFunction(argMin, Bool, DateTime64(6, 'UTC')),
entry_has_fbclid AggregateFunction(argMin, Bool, DateTime64(6, 'UTC')),
- entry_ad_ids_map AggregateFunction(argMin, Map(String, String), DateTime64(6, 'UTC')),
entry_ad_ids_set AggregateFunction(argMin, Array(String), DateTime64(6, 'UTC')),
entry_channel_type_properties AggregateFunction(argMin, Tuple(Nullable(String), Nullable(String), Nullable(String), Nullable(String), Bool, Bool, Nullable(String)), DateTime64(6, 'UTC')),
- pageview_uniq AggregateFunction(uniqExact, Nullable(UUID)),
- autocapture_uniq AggregateFunction(uniqExact, Nullable(UUID)),
- screen_uniq AggregateFunction(uniqExact, Nullable(UUID)),
+ pageview_uniq AggregateFunction(uniq, Nullable(UUID)),
+ autocapture_uniq AggregateFunction(uniq, Nullable(UUID)),
+ screen_uniq AggregateFunction(uniq, Nullable(UUID)),
page_screen_uniq_up_to AggregateFunction(uniqUpTo(1), Nullable(UUID)),
has_autocapture SimpleAggregateFunction(max, Bool),
flag_key_values SimpleAggregateFunction(groupUniqArrayArray(10000), Array(String)),
- flag_keys SimpleAggregateFunction(groupUniqArrayArray, Array(String)),
event_names SimpleAggregateFunction(groupUniqArrayArray(2000), Array(String)),
hosts SimpleAggregateFunction(groupUniqArrayArray(100), Array(String)),
emails SimpleAggregateFunction(groupUniqArrayArray(10), Array(String)),
@@ -2756,6 +2751,214 @@ WHERE
bitAnd(bitShiftRight(toUInt128(accurateCastOrNull(`$session_id`, 'UUID')), 76), 15) = 7
GROUP BY
team_id, toStartOfHour(fromUnixTimestamp(intDiv(toUInt64(bitShiftRight(session_id_v7, 80)), 1000))), cityHash64(session_id_v7), session_id_v7;
+CREATE MATERIALIZED VIEW posthog.raw_sessions_v3_mv TO posthog.writable_raw_sessions_v3 (team_id Int64, session_id_v7 Nullable(UInt128), distinct_id AggregateFunction(argMax, String, DateTime64(6, 'UTC')), distinct_ids AggregateFunction(groupUniqArray, String), min_timestamp DateTime64(6, 'UTC'), max_timestamp DateTime64(6, 'UTC'), max_inserted_at Nullable(DateTime64(6, 'UTC')), urls Array(Nullable(String)), entry_url AggregateFunction(argMin, Nullable(String), DateTime64(6, 'UTC')), end_url AggregateFunction(argMax, Nullable(String), DateTime64(6, 'UTC')), last_external_click_url AggregateFunction(argMax, Nullable(String), DateTime64(6, 'UTC')), browser AggregateFunction(argMin, Nullable(String), DateTime64(6, 'UTC')), browser_version AggregateFunction(argMin, Nullable(String), DateTime64(6, 'UTC')), os AggregateFunction(argMin, Nullable(String), DateTime64(6, 'UTC')), os_version AggregateFunction(argMin, Nullable(String), DateTime64(6, 'UTC')), device_type AggregateFunction(argMin, Nullable(String), DateTime64(6, 'UTC')), viewport_width AggregateFunction(argMin, Nullable(Int64), DateTime64(6, 'UTC')), viewport_height AggregateFunction(argMin, Nullable(Int64), DateTime64(6, 'UTC')), geoip_country_code AggregateFunction(argMin, Nullable(String), DateTime64(6, 'UTC')), geoip_subdivision_1_code AggregateFunction(argMin, Nullable(String), DateTime64(6, 'UTC')), geoip_subdivision_1_name AggregateFunction(argMin, Nullable(String), DateTime64(6, 'UTC')), geoip_subdivision_city_name AggregateFunction(argMin, Nullable(String), DateTime64(6, 'UTC')), geoip_time_zone AggregateFunction(argMin, Nullable(String), DateTime64(6, 'UTC')), entry_referring_domain AggregateFunction(argMin, Nullable(String), DateTime64(6, 'UTC')), entry_utm_source AggregateFunction(argMin, Nullable(String), DateTime64(6, 'UTC')), entry_utm_campaign AggregateFunction(argMin, Nullable(String), DateTime64(6, 'UTC')), entry_utm_medium AggregateFunction(argMin, Nullable(String), DateTime64(6, 'UTC')), entry_utm_term AggregateFunction(argMin, Nullable(String), DateTime64(6, 'UTC')), entry_utm_content AggregateFunction(argMin, Nullable(String), DateTime64(6, 'UTC')), entry_gclid AggregateFunction(argMin, Nullable(String), DateTime64(6, 'UTC')), entry_gad_source AggregateFunction(argMin, Nullable(String), DateTime64(6, 'UTC')), entry_fbclid AggregateFunction(argMin, Nullable(String), DateTime64(6, 'UTC')), entry_has_gclid AggregateFunction(argMin, UInt8, DateTime64(6, 'UTC')), entry_has_fbclid AggregateFunction(argMin, UInt8, DateTime64(6, 'UTC')), entry_ad_ids_set AggregateFunction(argMin, Array(String), DateTime64(6, 'UTC')), entry_channel_type_properties AggregateFunction(argMin, Tuple(Nullable(String), Nullable(String), Nullable(String), Nullable(String), UInt8, UInt8, Nullable(String)), DateTime64(6, 'UTC')), pageview_uniq AggregateFunction(uniq, Nullable(UUID)), autocapture_uniq AggregateFunction(uniq, Nullable(UUID)), screen_uniq AggregateFunction(uniq, Nullable(UUID)), page_screen_uniq_up_to AggregateFunction(uniqUpTo(1), Nullable(UUID)), has_autocapture UInt8, flag_key_values Array(String), event_names Array(String), hosts Array(Nullable(String)), emails Array(String), has_replay_events Bool) AS WITH
+ JSONExtract(properties, 'Tuple(\n `$current_url` Nullable(String),\n `$external_click_url` Nullable(String),\n `$browser` Nullable(String),\n `$browser_version` Nullable(String),\n `$os` Nullable(String),\n `$os_version` Nullable(String),\n `$device_type` Nullable(String),\n `$viewport_width` Nullable(Int64),\n `$viewport_height` Nullable(Int64),\n `$geoip_country_code` Nullable(String),\n `$geoip_subdivision_1_code` Nullable(String),\n `$geoip_subdivision_1_name` Nullable(String),\n `$geoip_city_name` Nullable(String),\n `$geoip_time_zone` Nullable(String),\n `$referring_domain` Nullable(String),\n `utm_source` Nullable(String),\n `utm_campaign` Nullable(String),\n `utm_medium` Nullable(String),\n `utm_term` Nullable(String),\n `utm_content` Nullable(String),\n `gclid` Nullable(String),\n `gad_source` Nullable(String),\n `fbclid` Nullable(String),\n `$host` Nullable(String),\n `gclsrc` Nullable(String),\n `dclid` Nullable(String),\n `gbraid` Nullable(String),\n `wbraid` Nullable(String),\n `gad_campaignid` Nullable(String),\n `srsltid` Nullable(String),\n `msclkid` Nullable(String),\n `twclid` Nullable(String),\n `li_fat_id` Nullable(String),\n `mc_cid` Nullable(String),\n `igshid` Nullable(String),\n `ttclid` Nullable(String),\n `rdt_cid` Nullable(String),\n `epik` Nullable(String),\n `qclid` Nullable(String),\n `sccid` Nullable(String),\n `_kx` Nullable(String),\n `irclid` Nullable(String),\n `irclickid` Nullable(String),\n `utm_id` Nullable(String),\n `utm_source_platform` Nullable(String),\n `utm_creative_format` Nullable(String),\n `utm_marketing_tactic` Nullable(String),\n `yclid` Nullable(String),\n `tblci` Nullable(String),\n `dicbo` Nullable(String),\n `ndclid` Nullable(String),\n `cjevent` Nullable(String),\n `awc` Nullable(String),\n `syclid` Nullable(String),\n `_bhlid` Nullable(String),\n `rtid` Nullable(String),\n `oppref` Nullable(String),\n `bfcid` Nullable(String)\n )') AS p,
+ JSONExtractString(person_properties, 'email') AS _person_email,
+ tupleElement(p, '$current_url') AS _current_url,
+ if((event = '$pageview') OR (event = '$screen'), left(coalesce(_current_url, ''), 16384), '') AS _attribution_url,
+ tupleElement(p, '$external_click_url') AS _external_click_url,
+ tupleElement(p, '$browser') AS _browser,
+ tupleElement(p, '$browser_version') AS _browser_version,
+ tupleElement(p, '$os') AS _os,
+ tupleElement(p, '$os_version') AS _os_version,
+ tupleElement(p, '$device_type') AS _device_type,
+ tupleElement(p, '$viewport_width') AS _viewport_width,
+ tupleElement(p, '$viewport_height') AS _viewport_height,
+ tupleElement(p, '$geoip_country_code') AS _geoip_country_code,
+ tupleElement(p, '$geoip_subdivision_1_code') AS _geoip_subdivision_1_code,
+ tupleElement(p, '$geoip_subdivision_1_name') AS _geoip_subdivision_1_name,
+ tupleElement(p, '$geoip_city_name') AS _geoip_subdivision_city_name,
+ tupleElement(p, '$geoip_time_zone') AS _geoip_time_zone,
+ tupleElement(p, '$referring_domain') AS _referring_domain,
+ tupleElement(p, 'utm_source') AS _utm_source,
+ tupleElement(p, 'utm_campaign') AS _utm_campaign,
+ tupleElement(p, 'utm_medium') AS _utm_medium,
+ tupleElement(p, 'utm_term') AS _utm_term,
+ tupleElement(p, 'utm_content') AS _utm_content,
+ tupleElement(p, 'gclid') AS _gclid,
+ tupleElement(p, 'gad_source') AS _gad_source,
+ tupleElement(p, 'fbclid') AS _fbclid,
+ coalesce(nullIf(tupleElement(p, 'gclsrc'), ''), nullIf(toValidUTF8(left(decodeURLFormComponent(extractURLParameter(_attribution_url, 'gclsrc')), 2048)), '')) AS gclsrc,
+ coalesce(nullIf(tupleElement(p, 'dclid'), ''), nullIf(toValidUTF8(left(decodeURLFormComponent(extractURLParameter(_attribution_url, 'dclid')), 2048)), '')) AS dclid,
+ coalesce(nullIf(tupleElement(p, 'gbraid'), ''), nullIf(toValidUTF8(left(decodeURLFormComponent(extractURLParameter(_attribution_url, 'gbraid')), 2048)), '')) AS gbraid,
+ coalesce(nullIf(tupleElement(p, 'wbraid'), ''), nullIf(toValidUTF8(left(decodeURLFormComponent(extractURLParameter(_attribution_url, 'wbraid')), 2048)), '')) AS wbraid,
+ coalesce(nullIf(tupleElement(p, 'gad_campaignid'), ''), nullIf(toValidUTF8(left(decodeURLFormComponent(extractURLParameter(_attribution_url, 'gad_campaignid')), 2048)), '')) AS gad_campaignid,
+ coalesce(nullIf(tupleElement(p, 'srsltid'), ''), nullIf(toValidUTF8(left(decodeURLFormComponent(extractURLParameter(_attribution_url, 'srsltid')), 2048)), '')) AS srsltid,
+ coalesce(nullIf(tupleElement(p, 'msclkid'), ''), nullIf(toValidUTF8(left(decodeURLFormComponent(extractURLParameter(_attribution_url, 'msclkid')), 2048)), '')) AS msclkid,
+ coalesce(nullIf(tupleElement(p, 'twclid'), ''), nullIf(toValidUTF8(left(decodeURLFormComponent(extractURLParameter(_attribution_url, 'twclid')), 2048)), '')) AS twclid,
+ coalesce(nullIf(tupleElement(p, 'li_fat_id'), ''), nullIf(toValidUTF8(left(decodeURLFormComponent(extractURLParameter(_attribution_url, 'li_fat_id')), 2048)), '')) AS li_fat_id,
+ coalesce(nullIf(tupleElement(p, 'mc_cid'), ''), nullIf(toValidUTF8(left(decodeURLFormComponent(extractURLParameter(_attribution_url, 'mc_cid')), 2048)), '')) AS mc_cid,
+ coalesce(nullIf(tupleElement(p, 'igshid'), ''), nullIf(toValidUTF8(left(decodeURLFormComponent(extractURLParameter(_attribution_url, 'igshid')), 2048)), '')) AS igshid,
+ coalesce(nullIf(tupleElement(p, 'ttclid'), ''), nullIf(toValidUTF8(left(decodeURLFormComponent(extractURLParameter(_attribution_url, 'ttclid')), 2048)), '')) AS ttclid,
+ coalesce(nullIf(tupleElement(p, 'rdt_cid'), ''), nullIf(toValidUTF8(left(decodeURLFormComponent(extractURLParameter(_attribution_url, 'rdt_cid')), 2048)), '')) AS rdt_cid,
+ coalesce(nullIf(tupleElement(p, 'epik'), ''), nullIf(toValidUTF8(left(decodeURLFormComponent(extractURLParameter(_attribution_url, 'epik')), 2048)), '')) AS epik,
+ coalesce(nullIf(tupleElement(p, 'qclid'), ''), nullIf(toValidUTF8(left(decodeURLFormComponent(extractURLParameter(_attribution_url, 'qclid')), 2048)), '')) AS qclid,
+ coalesce(nullIf(tupleElement(p, 'sccid'), ''), nullIf(toValidUTF8(left(decodeURLFormComponent(extractURLParameter(_attribution_url, 'sccid')), 2048)), ''), nullIf(toValidUTF8(left(decodeURLFormComponent(extractURLParameter(_attribution_url, 'ScCid')), 2048)), ''), nullIf(toValidUTF8(left(decodeURLFormComponent(extractURLParameter(_attribution_url, 'SsCid')), 2048)), '')) AS sccid,
+ coalesce(nullIf(tupleElement(p, '_kx'), ''), nullIf(toValidUTF8(left(decodeURLFormComponent(extractURLParameter(_attribution_url, '_kx')), 2048)), '')) AS _kx,
+ coalesce(nullIf(tupleElement(p, 'irclid'), ''), nullIf(toValidUTF8(left(decodeURLFormComponent(extractURLParameter(_attribution_url, 'irclid')), 2048)), '')) AS irclid,
+ coalesce(nullIf(tupleElement(p, 'irclickid'), ''), nullIf(toValidUTF8(left(decodeURLFormComponent(extractURLParameter(_attribution_url, 'irclickid')), 2048)), '')) AS irclickid,
+ coalesce(nullIf(tupleElement(p, 'utm_id'), ''), nullIf(toValidUTF8(left(decodeURLFormComponent(extractURLParameter(_attribution_url, 'utm_id')), 2048)), '')) AS utm_id,
+ coalesce(nullIf(tupleElement(p, 'utm_source_platform'), ''), nullIf(toValidUTF8(left(decodeURLFormComponent(extractURLParameter(_attribution_url, 'utm_source_platform')), 2048)), '')) AS utm_source_platform,
+ coalesce(nullIf(tupleElement(p, 'utm_creative_format'), ''), nullIf(toValidUTF8(left(decodeURLFormComponent(extractURLParameter(_attribution_url, 'utm_creative_format')), 2048)), '')) AS utm_creative_format,
+ coalesce(nullIf(tupleElement(p, 'utm_marketing_tactic'), ''), nullIf(toValidUTF8(left(decodeURLFormComponent(extractURLParameter(_attribution_url, 'utm_marketing_tactic')), 2048)), '')) AS utm_marketing_tactic,
+ coalesce(nullIf(tupleElement(p, 'yclid'), ''), nullIf(toValidUTF8(left(decodeURLFormComponent(extractURLParameter(_attribution_url, 'yclid')), 2048)), '')) AS yclid,
+ coalesce(nullIf(tupleElement(p, 'tblci'), ''), nullIf(toValidUTF8(left(decodeURLFormComponent(extractURLParameter(_attribution_url, 'tblci')), 2048)), '')) AS tblci,
+ coalesce(nullIf(tupleElement(p, 'dicbo'), ''), nullIf(toValidUTF8(left(decodeURLFormComponent(extractURLParameter(_attribution_url, 'dicbo')), 2048)), '')) AS dicbo,
+ coalesce(nullIf(tupleElement(p, 'ndclid'), ''), nullIf(toValidUTF8(left(decodeURLFormComponent(extractURLParameter(_attribution_url, 'ndclid')), 2048)), '')) AS ndclid,
+ coalesce(nullIf(tupleElement(p, 'cjevent'), ''), nullIf(toValidUTF8(left(decodeURLFormComponent(extractURLParameter(_attribution_url, 'cjevent')), 2048)), '')) AS cjevent,
+ coalesce(nullIf(tupleElement(p, 'awc'), ''), nullIf(toValidUTF8(left(decodeURLFormComponent(extractURLParameter(_attribution_url, 'awc')), 2048)), '')) AS awc,
+ coalesce(nullIf(tupleElement(p, 'syclid'), ''), nullIf(toValidUTF8(left(decodeURLFormComponent(extractURLParameter(_attribution_url, 'syclid')), 2048)), '')) AS syclid,
+ coalesce(nullIf(tupleElement(p, '_bhlid'), ''), nullIf(toValidUTF8(left(decodeURLFormComponent(extractURLParameter(_attribution_url, '_bhlid')), 2048)), '')) AS _bhlid,
+ coalesce(nullIf(tupleElement(p, 'rtid'), ''), nullIf(toValidUTF8(left(decodeURLFormComponent(extractURLParameter(_attribution_url, 'rtid')), 2048)), '')) AS rtid,
+ coalesce(nullIf(tupleElement(p, 'oppref'), ''), nullIf(toValidUTF8(left(decodeURLFormComponent(extractURLParameter(_attribution_url, 'oppref')), 2048)), '')) AS oppref,
+ coalesce(nullIf(tupleElement(p, 'bfcid'), ''), nullIf(toValidUTF8(left(decodeURLFormComponent(extractURLParameter(_attribution_url, 'bfcid')), 2048)), '')) AS bfcid,
+ CAST(arrayFilter(x -> (x IS NOT NULL), [if(gclsrc IS NOT NULL, concat('gclsrc=', gclsrc), NULL), if(dclid IS NOT NULL, concat('dclid=', dclid), NULL), if(gbraid IS NOT NULL, concat('gbraid=', gbraid), NULL), if(wbraid IS NOT NULL, concat('wbraid=', wbraid), NULL), if(gad_campaignid IS NOT NULL, concat('gad_campaignid=', gad_campaignid), NULL), if(srsltid IS NOT NULL, concat('srsltid=', srsltid), NULL), if(msclkid IS NOT NULL, concat('msclkid=', msclkid), NULL), if(twclid IS NOT NULL, concat('twclid=', twclid), NULL), if(li_fat_id IS NOT NULL, concat('li_fat_id=', li_fat_id), NULL), if(mc_cid IS NOT NULL, concat('mc_cid=', mc_cid), NULL), if(igshid IS NOT NULL, concat('igshid=', igshid), NULL), if(ttclid IS NOT NULL, concat('ttclid=', ttclid), NULL), if(rdt_cid IS NOT NULL, concat('rdt_cid=', rdt_cid), NULL), if(epik IS NOT NULL, concat('epik=', epik), NULL), if(qclid IS NOT NULL, concat('qclid=', qclid), NULL), if(sccid IS NOT NULL, concat('sccid=', sccid), NULL), if(_kx IS NOT NULL, concat('_kx=', _kx), NULL), if(irclid IS NOT NULL, concat('irclid=', irclid), NULL), if(irclickid IS NOT NULL, concat('irclickid=', irclickid), NULL), if(utm_id IS NOT NULL, concat('utm_id=', utm_id), NULL), if(utm_source_platform IS NOT NULL, concat('utm_source_platform=', utm_source_platform), NULL), if(utm_creative_format IS NOT NULL, concat('utm_creative_format=', utm_creative_format), NULL), if(utm_marketing_tactic IS NOT NULL, concat('utm_marketing_tactic=', utm_marketing_tactic), NULL), if(yclid IS NOT NULL, concat('yclid=', yclid), NULL), if(tblci IS NOT NULL, concat('tblci=', tblci), NULL), if(dicbo IS NOT NULL, concat('dicbo=', dicbo), NULL), if(ndclid IS NOT NULL, concat('ndclid=', ndclid), NULL), if(cjevent IS NOT NULL, concat('cjevent=', cjevent), NULL), if(awc IS NOT NULL, concat('awc=', awc), NULL), if(syclid IS NOT NULL, concat('syclid=', syclid), NULL), if(_bhlid IS NOT NULL, concat('_bhlid=', _bhlid), NULL), if(rtid IS NOT NULL, concat('rtid=', rtid), NULL), if(oppref IS NOT NULL, concat('oppref=', oppref), NULL), if(bfcid IS NOT NULL, concat('bfcid=', bfcid), NULL)]), 'Array(String)') AS ad_ids_set,
+ tupleElement(p, '$host') AS _host,
+ if((event = '$pageview') OR (event = '$screen'), timestamp, timestamp + toIntervalYear(1)) AS pageview_prio_timestamp_min,
+ if((event = '$pageview') OR (event = '$screen'), timestamp, timestamp - toIntervalYear(1)) AS pageview_prio_timestamp_max
+SELECT
+ team_id,
+ `$session_id_uuid` AS session_id_v7,
+ initializeAggregation('argMaxState', source_table.distinct_id, timestamp) AS distinct_id,
+ initializeAggregation('groupUniqArrayState', source_table.distinct_id) AS distinct_ids,
+ timestamp AS min_timestamp,
+ timestamp AS max_timestamp,
+ inserted_at AS max_inserted_at,
+ if(
+ (_current_url IS NOT NULL) AND ((event = '$pageview') OR (event = '$screen')),
+ [_current_url],
+ []
+ ) AS urls,
+ initializeAggregation('argMinState', _current_url, pageview_prio_timestamp_min) AS entry_url,
+ initializeAggregation('argMaxState', _current_url, pageview_prio_timestamp_max) AS end_url,
+ initializeAggregation('argMaxState', _external_click_url, timestamp) AS last_external_click_url,
+ initializeAggregation('argMinState', _browser, timestamp) AS browser,
+ initializeAggregation('argMinState', _browser_version, timestamp) AS browser_version,
+ initializeAggregation('argMinState', _os, timestamp) AS os,
+ initializeAggregation('argMinState', _os_version, timestamp) AS os_version,
+ initializeAggregation('argMinState', _device_type, timestamp) AS device_type,
+ initializeAggregation('argMinState', _viewport_width, timestamp) AS viewport_width,
+ initializeAggregation('argMinState', _viewport_height, timestamp) AS viewport_height,
+ initializeAggregation('argMinState', _geoip_country_code, timestamp) AS geoip_country_code,
+ initializeAggregation('argMinState', _geoip_subdivision_1_code, timestamp) AS geoip_subdivision_1_code,
+ initializeAggregation('argMinState', _geoip_subdivision_1_name, timestamp) AS geoip_subdivision_1_name,
+ initializeAggregation('argMinState', _geoip_subdivision_city_name, timestamp) AS geoip_subdivision_city_name,
+ initializeAggregation('argMinState', _geoip_time_zone, timestamp) AS geoip_time_zone,
+ initializeAggregation('argMinState', _referring_domain, pageview_prio_timestamp_min) AS entry_referring_domain,
+ initializeAggregation('argMinState', _utm_source, pageview_prio_timestamp_min) AS entry_utm_source,
+ initializeAggregation('argMinState', _utm_campaign, pageview_prio_timestamp_min) AS entry_utm_campaign,
+ initializeAggregation('argMinState', _utm_medium, pageview_prio_timestamp_min) AS entry_utm_medium,
+ initializeAggregation('argMinState', _utm_term, pageview_prio_timestamp_min) AS entry_utm_term,
+ initializeAggregation('argMinState', _utm_content, pageview_prio_timestamp_min) AS entry_utm_content,
+ initializeAggregation('argMinState', _gclid, pageview_prio_timestamp_min) AS entry_gclid,
+ initializeAggregation('argMinState', _gad_source, pageview_prio_timestamp_min) AS entry_gad_source,
+ initializeAggregation('argMinState', _fbclid, pageview_prio_timestamp_min) AS entry_fbclid,
+ initializeAggregation('argMinState', nullIf(_gclid, '') IS NOT NULL, pageview_prio_timestamp_min) AS entry_has_gclid,
+ initializeAggregation('argMinState', nullIf(_fbclid, '') IS NOT NULL, pageview_prio_timestamp_min) AS entry_has_fbclid,
+ initializeAggregation('argMinState', ad_ids_set, pageview_prio_timestamp_min) AS entry_ad_ids_set,
+ initializeAggregation(
+ 'argMinState',
+ tuple(
+ _utm_source,
+ _utm_medium,
+ _utm_campaign,
+ _referring_domain,
+ nullIf(_gclid, '') IS NOT NULL,
+ nullIf(_fbclid, '') IS NOT NULL,
+ _gad_source
+ ),
+ pageview_prio_timestamp_min
+ ) AS entry_channel_type_properties,
+ initializeAggregation('uniqState', if(event = '$pageview', uuid, NULL)) AS pageview_uniq,
+ initializeAggregation('uniqState', if(event = '$autocapture', uuid, NULL)) AS autocapture_uniq,
+ initializeAggregation('uniqState', if(event = '$screen', uuid, NULL)) AS screen_uniq,
+ initializeAggregation(
+ 'uniqUpToState(1)',
+ if((event = '$pageview') OR (event = '$screen'), uuid, NULL)
+ ) AS page_screen_uniq_up_to,
+ event = '$autocapture' AS has_autocapture,
+ arrayMap(
+ (k, v) -> concat(k, '=', v),
+ mapKeys(properties_group_feature_flags),
+ mapValues(properties_group_feature_flags)
+ ) AS flag_key_values,
+ [event] AS event_names,
+ if((_host IS NOT NULL) AND (_host != ''), [_host], []) AS hosts,
+ if((_person_email IS NOT NULL) AND (_person_email != ''), [_person_email], []) AS emails,
+ false AS has_replay_events
+FROM posthog.sharded_events AS source_table
+WHERE
+ (bitAnd(bitShiftRight(toUInt128(accurateCastOrNull(`$session_id`, 'UUID')), 76), 15) = 7)
+AND
+ ((fromUnixTimestamp64Milli(toUInt64(bitShiftRight(`$session_id_uuid`, 80))) >= (timestamp - toIntervalDay(31))) AND (fromUnixTimestamp64Milli(toUInt64(bitShiftRight(`$session_id_uuid`, 80))) <= (timestamp + toIntervalDay(31))))
+AND
+ true;
+CREATE MATERIALIZED VIEW posthog.raw_sessions_v3_recordings_mv TO posthog.writable_raw_sessions_v3 (team_id Int64, session_id_v7 UInt128, session_timestamp DateTime64(3), distinct_id AggregateFunction(argMax, String, DateTime64(6)), distinct_ids AggregateFunction(groupUniqArray, String), min_timestamp SimpleAggregateFunction(min, DateTime64(6, 'UTC')), max_timestamp SimpleAggregateFunction(min, DateTime64(6, 'UTC')), max_inserted_at DateTime, urls Array(String), entry_url AggregateFunction(argMin, Nullable(String), DateTime64(6)), end_url AggregateFunction(argMax, Nullable(String), DateTime64(6)), last_external_click_url AggregateFunction(argMax, Nullable(String), DateTime64(6)), browser AggregateFunction(argMin, Nullable(String), DateTime64(6)), browser_version AggregateFunction(argMin, Nullable(String), DateTime64(6)), os AggregateFunction(argMin, Nullable(String), DateTime64(6)), os_version AggregateFunction(argMin, Nullable(String), DateTime64(6)), device_type AggregateFunction(argMin, Nullable(String), DateTime64(6)), viewport_width AggregateFunction(argMin, Nullable(Int64), DateTime64(6)), viewport_height AggregateFunction(argMin, Nullable(Int64), DateTime64(6)), geoip_country_code AggregateFunction(argMin, Nullable(String), DateTime64(6)), geoip_subdivision_1_code AggregateFunction(argMin, Nullable(String), DateTime64(6)), geoip_subdivision_1_name AggregateFunction(argMin, Nullable(String), DateTime64(6)), geoip_subdivision_city_name AggregateFunction(argMin, Nullable(String), DateTime64(6)), geoip_time_zone AggregateFunction(argMin, Nullable(String), DateTime64(6)), entry_referring_domain AggregateFunction(argMin, Nullable(String), DateTime64(6)), entry_utm_source AggregateFunction(argMin, Nullable(String), DateTime64(6)), entry_utm_campaign AggregateFunction(argMin, Nullable(String), DateTime64(6)), entry_utm_medium AggregateFunction(argMin, Nullable(String), DateTime64(6)), entry_utm_term AggregateFunction(argMin, Nullable(String), DateTime64(6)), entry_utm_content AggregateFunction(argMin, Nullable(String), DateTime64(6)), entry_gclid AggregateFunction(argMin, Nullable(String), DateTime64(6)), entry_gad_source AggregateFunction(argMin, Nullable(String), DateTime64(6)), entry_fbclid AggregateFunction(argMin, Nullable(String), DateTime64(6)), entry_has_gclid AggregateFunction(argMin, Bool, DateTime64(6)), entry_has_fbclid AggregateFunction(argMin, Bool, DateTime64(6)), entry_ad_ids_set AggregateFunction(argMin, Array(String), DateTime64(6)), entry_channel_type_properties AggregateFunction(argMin, Tuple(Nullable(String), Nullable(String), Nullable(String), Nullable(String), Bool, Bool, Nullable(String)), DateTime64(6)), pageview_uniq AggregateFunction(uniq, Nullable(UUID)), autocapture_uniq AggregateFunction(uniq, Nullable(UUID)), screen_uniq AggregateFunction(uniq, Nullable(UUID)), page_screen_uniq_up_to AggregateFunction(uniqUpTo(1), Nullable(UUID)), has_autocapture Bool, flag_key_values Array(String), event_names Array(String), hosts Array(String), emails Array(String), has_replay_events Bool) AS WITH
+ min_first_timestamp AS timestamp,
+ CAST(fromUnixTimestamp64Milli(9223372036854775), 'DateTime64(6)') AS max_ts_64,
+ CAST(fromUnixTimestamp64Milli(-9223372036854775), 'DateTime64(6)') AS min_ts_64,
+ CAST(NULL, 'Nullable(String)') AS null_s,
+ CAST(NULL, 'Nullable(Int64)') AS null_i64,
+ CAST(NULL, 'Nullable(UUID)') AS null_uuid
+SELECT
+ team_id,
+ toUInt128(accurateCast(session_id, 'UUID')) AS session_id_v7,
+ fromUnixTimestamp64Milli(toUInt64(bitShiftRight(session_id_v7, 80))) AS session_timestamp,
+ initializeAggregation('argMaxState', source_table.distinct_id, min_ts_64) AS distinct_id,
+ initializeAggregation('groupUniqArrayState', source_table.distinct_id) AS distinct_ids,
+ timestamp AS min_timestamp,
+ timestamp AS max_timestamp,
+ fromUnixTimestamp(0) AS max_inserted_at,
+ CAST([], 'Array(String)') AS urls,
+ initializeAggregation('argMinState', null_s, max_ts_64) AS entry_url,
+ initializeAggregation('argMaxState', null_s, min_ts_64) AS end_url,
+ initializeAggregation('argMaxState', null_s, min_ts_64) AS last_external_click_url,
+ initializeAggregation('argMinState', null_s, max_ts_64) AS browser,
+ initializeAggregation('argMinState', null_s, max_ts_64) AS browser_version,
+ initializeAggregation('argMinState', null_s, max_ts_64) AS os,
+ initializeAggregation('argMinState', null_s, max_ts_64) AS os_version,
+ initializeAggregation('argMinState', null_s, max_ts_64) AS device_type,
+ initializeAggregation('argMinState', null_i64, max_ts_64) AS viewport_width,
+ initializeAggregation('argMinState', null_i64, max_ts_64) AS viewport_height,
+ initializeAggregation('argMinState', null_s, max_ts_64) AS geoip_country_code,
+ initializeAggregation('argMinState', null_s, max_ts_64) AS geoip_subdivision_1_code,
+ initializeAggregation('argMinState', null_s, max_ts_64) AS geoip_subdivision_1_name,
+ initializeAggregation('argMinState', null_s, max_ts_64) AS geoip_subdivision_city_name,
+ initializeAggregation('argMinState', null_s, max_ts_64) AS geoip_time_zone,
+ initializeAggregation('argMinState', null_s, max_ts_64) AS entry_referring_domain,
+ initializeAggregation('argMinState', null_s, max_ts_64) AS entry_utm_source,
+ initializeAggregation('argMinState', null_s, max_ts_64) AS entry_utm_campaign,
+ initializeAggregation('argMinState', null_s, max_ts_64) AS entry_utm_medium,
+ initializeAggregation('argMinState', null_s, max_ts_64) AS entry_utm_term,
+ initializeAggregation('argMinState', null_s, max_ts_64) AS entry_utm_content,
+ initializeAggregation('argMinState', null_s, max_ts_64) AS entry_gclid,
+ initializeAggregation('argMinState', null_s, max_ts_64) AS entry_gad_source,
+ initializeAggregation('argMinState', null_s, max_ts_64) AS entry_fbclid,
+ initializeAggregation('argMinState', false, max_ts_64) AS entry_has_gclid,
+ initializeAggregation('argMinState', false, max_ts_64) AS entry_has_fbclid,
+ initializeAggregation('argMinState', CAST([], 'Array(String)'), max_ts_64) AS entry_ad_ids_set,
+ initializeAggregation(
+ 'argMinState',
+ tuple(null_s, null_s, null_s, null_s, false, false, null_s),
+ max_ts_64
+ ) AS entry_channel_type_properties,
+ initializeAggregation('uniqState', null_uuid) AS pageview_uniq,
+ initializeAggregation('uniqState', null_uuid) AS autocapture_uniq,
+ initializeAggregation('uniqState', null_uuid) AS screen_uniq,
+ initializeAggregation('uniqUpToState(1)', null_uuid) AS page_screen_uniq_up_to,
+ false AS has_autocapture,
+ CAST([], 'Array(String)') AS flag_key_values,
+ CAST([], 'Array(String)') AS event_names,
+ CAST([], 'Array(String)') AS hosts,
+ CAST([], 'Array(String)') AS emails,
+ true AS has_replay_events
+FROM posthog.sharded_session_replay_events AS source_table
+WHERE
+ (bitAnd(bitShiftRight(toUInt128(accurateCastOrNull(session_id, 'UUID')), 76), 15) = 7)
+AND
+ ((fromUnixTimestamp64Milli(toUInt64(bitShiftRight(toUInt128(accurateCastOrNull(session_id, 'UUID')), 80))) >= (min_first_timestamp - toIntervalDay(31))) AND (fromUnixTimestamp64Milli(toUInt64(bitShiftRight(toUInt128(accurateCastOrNull(session_id, 'UUID')), 80))) <= (min_first_timestamp + toIntervalDay(31))))
+AND
+ true;
CREATE MATERIALIZED VIEW posthog.sessions_mv TO posthog.writable_sessions (session_id String, team_id Int64, distinct_id String, min_timestamp DateTime64(6, 'UTC'), max_timestamp DateTime64(6, 'UTC'), urls Array(String), entry_url AggregateFunction(argMin, String, DateTime64(6, 'UTC')), exit_url AggregateFunction(argMax, String, DateTime64(6, 'UTC')), initial_referring_domain AggregateFunction(argMin, String, DateTime64(6, 'UTC')), initial_utm_source AggregateFunction(argMin, String, DateTime64(6, 'UTC')), initial_utm_campaign AggregateFunction(argMin, String, DateTime64(6, 'UTC')), initial_utm_medium AggregateFunction(argMin, String, DateTime64(6, 'UTC')), initial_utm_term AggregateFunction(argMin, String, DateTime64(6, 'UTC')), initial_utm_content AggregateFunction(argMin, String, DateTime64(6, 'UTC')), initial_gclid AggregateFunction(argMin, String, DateTime64(6, 'UTC')), initial_gad_source AggregateFunction(argMin, String, DateTime64(6, 'UTC')), initial_gclsrc AggregateFunction(argMin, String, DateTime64(6, 'UTC')), initial_dclid AggregateFunction(argMin, String, DateTime64(6, 'UTC')), initial_gbraid AggregateFunction(argMin, String, DateTime64(6, 'UTC')), initial_wbraid AggregateFunction(argMin, String, DateTime64(6, 'UTC')), initial_fbclid AggregateFunction(argMin, String, DateTime64(6, 'UTC')), initial_msclkid AggregateFunction(argMin, String, DateTime64(6, 'UTC')), initial_twclid AggregateFunction(argMin, String, DateTime64(6, 'UTC')), initial_li_fat_id AggregateFunction(argMin, String, DateTime64(6, 'UTC')), initial_mc_cid AggregateFunction(argMin, String, DateTime64(6, 'UTC')), initial_igshid AggregateFunction(argMin, String, DateTime64(6, 'UTC')), initial_ttclid AggregateFunction(argMin, String, DateTime64(6, 'UTC')), initial_epik AggregateFunction(argMin, String, DateTime64(6, 'UTC')), initial_qclid AggregateFunction(argMin, String, DateTime64(6, 'UTC')), initial_sccid AggregateFunction(argMin, String, DateTime64(6, 'UTC')), event_count_map Map(String, UInt64), pageview_count UInt64, autocapture_count UInt64) AS SELECT
`$session_id` AS session_id,
team_id,
@@ -3456,16 +3659,14 @@ CREATE TABLE posthog.raw_sessions_v3 (
entry_fbclid AggregateFunction(argMin, Nullable(String), DateTime64(6, 'UTC')),
entry_has_gclid AggregateFunction(argMin, Bool, DateTime64(6, 'UTC')),
entry_has_fbclid AggregateFunction(argMin, Bool, DateTime64(6, 'UTC')),
- entry_ad_ids_map AggregateFunction(argMin, Map(String, String), DateTime64(6, 'UTC')),
entry_ad_ids_set AggregateFunction(argMin, Array(String), DateTime64(6, 'UTC')),
entry_channel_type_properties AggregateFunction(argMin, Tuple(Nullable(String), Nullable(String), Nullable(String), Nullable(String), Bool, Bool, Nullable(String)), DateTime64(6, 'UTC')),
- pageview_uniq AggregateFunction(uniqExact, Nullable(UUID)),
- autocapture_uniq AggregateFunction(uniqExact, Nullable(UUID)),
- screen_uniq AggregateFunction(uniqExact, Nullable(UUID)),
+ pageview_uniq AggregateFunction(uniq, Nullable(UUID)),
+ autocapture_uniq AggregateFunction(uniq, Nullable(UUID)),
+ screen_uniq AggregateFunction(uniq, Nullable(UUID)),
page_screen_uniq_up_to AggregateFunction(uniqUpTo(1), Nullable(UUID)),
has_autocapture SimpleAggregateFunction(max, Bool),
flag_key_values SimpleAggregateFunction(groupUniqArrayArray(10000), Array(String)),
- flag_keys SimpleAggregateFunction(groupUniqArrayArray, Array(String)),
event_names SimpleAggregateFunction(groupUniqArrayArray(2000), Array(String)),
hosts SimpleAggregateFunction(groupUniqArrayArray(100), Array(String)),
emails SimpleAggregateFunction(groupUniqArrayArray(10), Array(String)),
@@ -3786,16 +3987,14 @@ CREATE VIEW posthog.raw_sessions_v3_v AS SELECT
argMinMerge(entry_fbclid) AS entry_fbclid,
argMinMerge(entry_has_gclid) AS entry_has_gclid,
argMinMerge(entry_has_fbclid) AS entry_has_fbclid,
- argMinMerge(entry_ad_ids_map) AS entry_ad_ids_map,
argMinMerge(entry_ad_ids_set) AS entry_ad_ids_set,
argMinMerge(entry_channel_type_properties) AS entry_channel_type_properties,
- uniqExactMerge(pageview_uniq) AS pageview_uniq,
- uniqExactMerge(autocapture_uniq) AS autocapture_uniq,
- uniqExactMerge(screen_uniq) AS screen_uniq,
+ uniqMerge(pageview_uniq) AS pageview_uniq,
+ uniqMerge(autocapture_uniq) AS autocapture_uniq,
+ uniqMerge(screen_uniq) AS screen_uniq,
uniqUpToMerge(1)(page_screen_uniq_up_to) AS page_screen_uniq_up_to,
max(has_autocapture) AS has_autocapture,
groupUniqArrayArray(10000)(flag_key_values) AS flag_key_values,
- groupUniqArrayArray(flag_keys) AS flag_keys,
groupUniqArrayArray(2000)(event_names) AS event_names,
groupUniqArrayArray(100)(hosts) AS hosts,
groupUniqArrayArray(10)(emails) AS emails,
diff --git a/posthog/clickhouse/hcl/sql/local-single/all.sql b/posthog/clickhouse/hcl/sql/local-single/all.sql
index 87260bcb07db..e74e7dc93895 100644
--- a/posthog/clickhouse/hcl/sql/local-single/all.sql
+++ b/posthog/clickhouse/hcl/sql/local-single/all.sql
@@ -2682,23 +2682,20 @@ CREATE TABLE posthog.sharded_raw_sessions_v3 (
entry_fbclid AggregateFunction(argMin, Nullable(String), DateTime64(6, 'UTC')),
entry_has_gclid AggregateFunction(argMin, Bool, DateTime64(6, 'UTC')),
entry_has_fbclid AggregateFunction(argMin, Bool, DateTime64(6, 'UTC')),
- entry_ad_ids_map AggregateFunction(argMin, Map(String, String), DateTime64(6, 'UTC')),
entry_ad_ids_set AggregateFunction(argMin, Array(String), DateTime64(6, 'UTC')),
entry_channel_type_properties AggregateFunction(argMin, Tuple(Nullable(String), Nullable(String), Nullable(String), Nullable(String), Bool, Bool, Nullable(String)), DateTime64(6, 'UTC')),
- pageview_uniq AggregateFunction(uniqExact, Nullable(UUID)),
- autocapture_uniq AggregateFunction(uniqExact, Nullable(UUID)),
- screen_uniq AggregateFunction(uniqExact, Nullable(UUID)),
+ pageview_uniq AggregateFunction(uniq, Nullable(UUID)),
+ autocapture_uniq AggregateFunction(uniq, Nullable(UUID)),
+ screen_uniq AggregateFunction(uniq, Nullable(UUID)),
page_screen_uniq_up_to AggregateFunction(uniqUpTo(1), Nullable(UUID)),
has_autocapture SimpleAggregateFunction(max, Bool),
flag_key_values SimpleAggregateFunction(groupUniqArrayArray(10000), Array(String)),
- flag_keys SimpleAggregateFunction(groupUniqArrayArray, Array(String)),
event_names SimpleAggregateFunction(groupUniqArrayArray(2000), Array(String)),
hosts SimpleAggregateFunction(groupUniqArrayArray(100), Array(String)),
emails SimpleAggregateFunction(groupUniqArrayArray(10), Array(String)),
has_replay_events SimpleAggregateFunction(max, Bool),
INDEX event_names_bloom_filter event_names TYPE bloom_filter() GRANULARITY 1,
- INDEX flag_key_values_bloom_filter flag_key_values TYPE bloom_filter() GRANULARITY 1,
- INDEX flag_keys_bloom_filter flag_keys TYPE bloom_filter() GRANULARITY 1,
+ INDEX flag_key_values_text flag_key_values TYPE text(tokenizer = splitByString(['='])) GRANULARITY 100000000,
INDEX hosts_bloom_filter hosts TYPE bloom_filter() GRANULARITY 1,
INDEX emails_bloom_filter emails TYPE bloom_filter() GRANULARITY 1
) ENGINE = ReplicatedAggregatingMergeTree('/clickhouse/tables/{shard}/posthog.raw_sessions_v3', '{replica}') ORDER BY (team_id, session_timestamp, session_id_v7) PARTITION BY toYYYYMM(session_timestamp) SETTINGS index_granularity = 8192, max_delay_to_insert = 10, parts_to_delay_insert = 250, parts_to_throw_insert = 1000;
@@ -4236,16 +4233,14 @@ CREATE TABLE posthog.writable_raw_sessions_v3 (
entry_fbclid AggregateFunction(argMin, Nullable(String), DateTime64(6, 'UTC')),
entry_has_gclid AggregateFunction(argMin, Bool, DateTime64(6, 'UTC')),
entry_has_fbclid AggregateFunction(argMin, Bool, DateTime64(6, 'UTC')),
- entry_ad_ids_map AggregateFunction(argMin, Map(String, String), DateTime64(6, 'UTC')),
entry_ad_ids_set AggregateFunction(argMin, Array(String), DateTime64(6, 'UTC')),
entry_channel_type_properties AggregateFunction(argMin, Tuple(Nullable(String), Nullable(String), Nullable(String), Nullable(String), Bool, Bool, Nullable(String)), DateTime64(6, 'UTC')),
- pageview_uniq AggregateFunction(uniqExact, Nullable(UUID)),
- autocapture_uniq AggregateFunction(uniqExact, Nullable(UUID)),
- screen_uniq AggregateFunction(uniqExact, Nullable(UUID)),
+ pageview_uniq AggregateFunction(uniq, Nullable(UUID)),
+ autocapture_uniq AggregateFunction(uniq, Nullable(UUID)),
+ screen_uniq AggregateFunction(uniq, Nullable(UUID)),
page_screen_uniq_up_to AggregateFunction(uniqUpTo(1), Nullable(UUID)),
has_autocapture SimpleAggregateFunction(max, Bool),
flag_key_values SimpleAggregateFunction(groupUniqArrayArray(10000), Array(String)),
- flag_keys SimpleAggregateFunction(groupUniqArrayArray, Array(String)),
event_names SimpleAggregateFunction(groupUniqArrayArray(2000), Array(String)),
hosts SimpleAggregateFunction(groupUniqArrayArray(100), Array(String)),
emails SimpleAggregateFunction(groupUniqArrayArray(10), Array(String)),
@@ -5644,6 +5639,214 @@ WHERE
bitAnd(bitShiftRight(toUInt128(accurateCastOrNull(`$session_id`, 'UUID')), 76), 15) = 7
GROUP BY
team_id, toStartOfHour(fromUnixTimestamp(intDiv(toUInt64(bitShiftRight(session_id_v7, 80)), 1000))), cityHash64(session_id_v7), session_id_v7;
+CREATE MATERIALIZED VIEW posthog.raw_sessions_v3_mv TO posthog.writable_raw_sessions_v3 (team_id Int64, session_id_v7 Nullable(UInt128), distinct_id AggregateFunction(argMax, String, DateTime64(6, 'UTC')), distinct_ids AggregateFunction(groupUniqArray, String), min_timestamp DateTime64(6, 'UTC'), max_timestamp DateTime64(6, 'UTC'), max_inserted_at Nullable(DateTime64(6, 'UTC')), urls Array(Nullable(String)), entry_url AggregateFunction(argMin, Nullable(String), DateTime64(6, 'UTC')), end_url AggregateFunction(argMax, Nullable(String), DateTime64(6, 'UTC')), last_external_click_url AggregateFunction(argMax, Nullable(String), DateTime64(6, 'UTC')), browser AggregateFunction(argMin, Nullable(String), DateTime64(6, 'UTC')), browser_version AggregateFunction(argMin, Nullable(String), DateTime64(6, 'UTC')), os AggregateFunction(argMin, Nullable(String), DateTime64(6, 'UTC')), os_version AggregateFunction(argMin, Nullable(String), DateTime64(6, 'UTC')), device_type AggregateFunction(argMin, Nullable(String), DateTime64(6, 'UTC')), viewport_width AggregateFunction(argMin, Nullable(Int64), DateTime64(6, 'UTC')), viewport_height AggregateFunction(argMin, Nullable(Int64), DateTime64(6, 'UTC')), geoip_country_code AggregateFunction(argMin, Nullable(String), DateTime64(6, 'UTC')), geoip_subdivision_1_code AggregateFunction(argMin, Nullable(String), DateTime64(6, 'UTC')), geoip_subdivision_1_name AggregateFunction(argMin, Nullable(String), DateTime64(6, 'UTC')), geoip_subdivision_city_name AggregateFunction(argMin, Nullable(String), DateTime64(6, 'UTC')), geoip_time_zone AggregateFunction(argMin, Nullable(String), DateTime64(6, 'UTC')), entry_referring_domain AggregateFunction(argMin, Nullable(String), DateTime64(6, 'UTC')), entry_utm_source AggregateFunction(argMin, Nullable(String), DateTime64(6, 'UTC')), entry_utm_campaign AggregateFunction(argMin, Nullable(String), DateTime64(6, 'UTC')), entry_utm_medium AggregateFunction(argMin, Nullable(String), DateTime64(6, 'UTC')), entry_utm_term AggregateFunction(argMin, Nullable(String), DateTime64(6, 'UTC')), entry_utm_content AggregateFunction(argMin, Nullable(String), DateTime64(6, 'UTC')), entry_gclid AggregateFunction(argMin, Nullable(String), DateTime64(6, 'UTC')), entry_gad_source AggregateFunction(argMin, Nullable(String), DateTime64(6, 'UTC')), entry_fbclid AggregateFunction(argMin, Nullable(String), DateTime64(6, 'UTC')), entry_has_gclid AggregateFunction(argMin, UInt8, DateTime64(6, 'UTC')), entry_has_fbclid AggregateFunction(argMin, UInt8, DateTime64(6, 'UTC')), entry_ad_ids_set AggregateFunction(argMin, Array(String), DateTime64(6, 'UTC')), entry_channel_type_properties AggregateFunction(argMin, Tuple(Nullable(String), Nullable(String), Nullable(String), Nullable(String), UInt8, UInt8, Nullable(String)), DateTime64(6, 'UTC')), pageview_uniq AggregateFunction(uniq, Nullable(UUID)), autocapture_uniq AggregateFunction(uniq, Nullable(UUID)), screen_uniq AggregateFunction(uniq, Nullable(UUID)), page_screen_uniq_up_to AggregateFunction(uniqUpTo(1), Nullable(UUID)), has_autocapture UInt8, flag_key_values Array(String), event_names Array(String), hosts Array(Nullable(String)), emails Array(String), has_replay_events Bool) AS WITH
+ JSONExtract(properties, 'Tuple(\n `$current_url` Nullable(String),\n `$external_click_url` Nullable(String),\n `$browser` Nullable(String),\n `$browser_version` Nullable(String),\n `$os` Nullable(String),\n `$os_version` Nullable(String),\n `$device_type` Nullable(String),\n `$viewport_width` Nullable(Int64),\n `$viewport_height` Nullable(Int64),\n `$geoip_country_code` Nullable(String),\n `$geoip_subdivision_1_code` Nullable(String),\n `$geoip_subdivision_1_name` Nullable(String),\n `$geoip_city_name` Nullable(String),\n `$geoip_time_zone` Nullable(String),\n `$referring_domain` Nullable(String),\n `utm_source` Nullable(String),\n `utm_campaign` Nullable(String),\n `utm_medium` Nullable(String),\n `utm_term` Nullable(String),\n `utm_content` Nullable(String),\n `gclid` Nullable(String),\n `gad_source` Nullable(String),\n `fbclid` Nullable(String),\n `$host` Nullable(String),\n `gclsrc` Nullable(String),\n `dclid` Nullable(String),\n `gbraid` Nullable(String),\n `wbraid` Nullable(String),\n `gad_campaignid` Nullable(String),\n `srsltid` Nullable(String),\n `msclkid` Nullable(String),\n `twclid` Nullable(String),\n `li_fat_id` Nullable(String),\n `mc_cid` Nullable(String),\n `igshid` Nullable(String),\n `ttclid` Nullable(String),\n `rdt_cid` Nullable(String),\n `epik` Nullable(String),\n `qclid` Nullable(String),\n `sccid` Nullable(String),\n `_kx` Nullable(String),\n `irclid` Nullable(String),\n `irclickid` Nullable(String),\n `utm_id` Nullable(String),\n `utm_source_platform` Nullable(String),\n `utm_creative_format` Nullable(String),\n `utm_marketing_tactic` Nullable(String),\n `yclid` Nullable(String),\n `tblci` Nullable(String),\n `dicbo` Nullable(String),\n `ndclid` Nullable(String),\n `cjevent` Nullable(String),\n `awc` Nullable(String),\n `syclid` Nullable(String),\n `_bhlid` Nullable(String),\n `rtid` Nullable(String),\n `oppref` Nullable(String),\n `bfcid` Nullable(String)\n )') AS p,
+ JSONExtractString(person_properties, 'email') AS _person_email,
+ tupleElement(p, '$current_url') AS _current_url,
+ if((event = '$pageview') OR (event = '$screen'), left(coalesce(_current_url, ''), 16384), '') AS _attribution_url,
+ tupleElement(p, '$external_click_url') AS _external_click_url,
+ tupleElement(p, '$browser') AS _browser,
+ tupleElement(p, '$browser_version') AS _browser_version,
+ tupleElement(p, '$os') AS _os,
+ tupleElement(p, '$os_version') AS _os_version,
+ tupleElement(p, '$device_type') AS _device_type,
+ tupleElement(p, '$viewport_width') AS _viewport_width,
+ tupleElement(p, '$viewport_height') AS _viewport_height,
+ tupleElement(p, '$geoip_country_code') AS _geoip_country_code,
+ tupleElement(p, '$geoip_subdivision_1_code') AS _geoip_subdivision_1_code,
+ tupleElement(p, '$geoip_subdivision_1_name') AS _geoip_subdivision_1_name,
+ tupleElement(p, '$geoip_city_name') AS _geoip_subdivision_city_name,
+ tupleElement(p, '$geoip_time_zone') AS _geoip_time_zone,
+ tupleElement(p, '$referring_domain') AS _referring_domain,
+ tupleElement(p, 'utm_source') AS _utm_source,
+ tupleElement(p, 'utm_campaign') AS _utm_campaign,
+ tupleElement(p, 'utm_medium') AS _utm_medium,
+ tupleElement(p, 'utm_term') AS _utm_term,
+ tupleElement(p, 'utm_content') AS _utm_content,
+ tupleElement(p, 'gclid') AS _gclid,
+ tupleElement(p, 'gad_source') AS _gad_source,
+ tupleElement(p, 'fbclid') AS _fbclid,
+ coalesce(nullIf(tupleElement(p, 'gclsrc'), ''), nullIf(toValidUTF8(left(decodeURLFormComponent(extractURLParameter(_attribution_url, 'gclsrc')), 2048)), '')) AS gclsrc,
+ coalesce(nullIf(tupleElement(p, 'dclid'), ''), nullIf(toValidUTF8(left(decodeURLFormComponent(extractURLParameter(_attribution_url, 'dclid')), 2048)), '')) AS dclid,
+ coalesce(nullIf(tupleElement(p, 'gbraid'), ''), nullIf(toValidUTF8(left(decodeURLFormComponent(extractURLParameter(_attribution_url, 'gbraid')), 2048)), '')) AS gbraid,
+ coalesce(nullIf(tupleElement(p, 'wbraid'), ''), nullIf(toValidUTF8(left(decodeURLFormComponent(extractURLParameter(_attribution_url, 'wbraid')), 2048)), '')) AS wbraid,
+ coalesce(nullIf(tupleElement(p, 'gad_campaignid'), ''), nullIf(toValidUTF8(left(decodeURLFormComponent(extractURLParameter(_attribution_url, 'gad_campaignid')), 2048)), '')) AS gad_campaignid,
+ coalesce(nullIf(tupleElement(p, 'srsltid'), ''), nullIf(toValidUTF8(left(decodeURLFormComponent(extractURLParameter(_attribution_url, 'srsltid')), 2048)), '')) AS srsltid,
+ coalesce(nullIf(tupleElement(p, 'msclkid'), ''), nullIf(toValidUTF8(left(decodeURLFormComponent(extractURLParameter(_attribution_url, 'msclkid')), 2048)), '')) AS msclkid,
+ coalesce(nullIf(tupleElement(p, 'twclid'), ''), nullIf(toValidUTF8(left(decodeURLFormComponent(extractURLParameter(_attribution_url, 'twclid')), 2048)), '')) AS twclid,
+ coalesce(nullIf(tupleElement(p, 'li_fat_id'), ''), nullIf(toValidUTF8(left(decodeURLFormComponent(extractURLParameter(_attribution_url, 'li_fat_id')), 2048)), '')) AS li_fat_id,
+ coalesce(nullIf(tupleElement(p, 'mc_cid'), ''), nullIf(toValidUTF8(left(decodeURLFormComponent(extractURLParameter(_attribution_url, 'mc_cid')), 2048)), '')) AS mc_cid,
+ coalesce(nullIf(tupleElement(p, 'igshid'), ''), nullIf(toValidUTF8(left(decodeURLFormComponent(extractURLParameter(_attribution_url, 'igshid')), 2048)), '')) AS igshid,
+ coalesce(nullIf(tupleElement(p, 'ttclid'), ''), nullIf(toValidUTF8(left(decodeURLFormComponent(extractURLParameter(_attribution_url, 'ttclid')), 2048)), '')) AS ttclid,
+ coalesce(nullIf(tupleElement(p, 'rdt_cid'), ''), nullIf(toValidUTF8(left(decodeURLFormComponent(extractURLParameter(_attribution_url, 'rdt_cid')), 2048)), '')) AS rdt_cid,
+ coalesce(nullIf(tupleElement(p, 'epik'), ''), nullIf(toValidUTF8(left(decodeURLFormComponent(extractURLParameter(_attribution_url, 'epik')), 2048)), '')) AS epik,
+ coalesce(nullIf(tupleElement(p, 'qclid'), ''), nullIf(toValidUTF8(left(decodeURLFormComponent(extractURLParameter(_attribution_url, 'qclid')), 2048)), '')) AS qclid,
+ coalesce(nullIf(tupleElement(p, 'sccid'), ''), nullIf(toValidUTF8(left(decodeURLFormComponent(extractURLParameter(_attribution_url, 'sccid')), 2048)), ''), nullIf(toValidUTF8(left(decodeURLFormComponent(extractURLParameter(_attribution_url, 'ScCid')), 2048)), ''), nullIf(toValidUTF8(left(decodeURLFormComponent(extractURLParameter(_attribution_url, 'SsCid')), 2048)), '')) AS sccid,
+ coalesce(nullIf(tupleElement(p, '_kx'), ''), nullIf(toValidUTF8(left(decodeURLFormComponent(extractURLParameter(_attribution_url, '_kx')), 2048)), '')) AS _kx,
+ coalesce(nullIf(tupleElement(p, 'irclid'), ''), nullIf(toValidUTF8(left(decodeURLFormComponent(extractURLParameter(_attribution_url, 'irclid')), 2048)), '')) AS irclid,
+ coalesce(nullIf(tupleElement(p, 'irclickid'), ''), nullIf(toValidUTF8(left(decodeURLFormComponent(extractURLParameter(_attribution_url, 'irclickid')), 2048)), '')) AS irclickid,
+ coalesce(nullIf(tupleElement(p, 'utm_id'), ''), nullIf(toValidUTF8(left(decodeURLFormComponent(extractURLParameter(_attribution_url, 'utm_id')), 2048)), '')) AS utm_id,
+ coalesce(nullIf(tupleElement(p, 'utm_source_platform'), ''), nullIf(toValidUTF8(left(decodeURLFormComponent(extractURLParameter(_attribution_url, 'utm_source_platform')), 2048)), '')) AS utm_source_platform,
+ coalesce(nullIf(tupleElement(p, 'utm_creative_format'), ''), nullIf(toValidUTF8(left(decodeURLFormComponent(extractURLParameter(_attribution_url, 'utm_creative_format')), 2048)), '')) AS utm_creative_format,
+ coalesce(nullIf(tupleElement(p, 'utm_marketing_tactic'), ''), nullIf(toValidUTF8(left(decodeURLFormComponent(extractURLParameter(_attribution_url, 'utm_marketing_tactic')), 2048)), '')) AS utm_marketing_tactic,
+ coalesce(nullIf(tupleElement(p, 'yclid'), ''), nullIf(toValidUTF8(left(decodeURLFormComponent(extractURLParameter(_attribution_url, 'yclid')), 2048)), '')) AS yclid,
+ coalesce(nullIf(tupleElement(p, 'tblci'), ''), nullIf(toValidUTF8(left(decodeURLFormComponent(extractURLParameter(_attribution_url, 'tblci')), 2048)), '')) AS tblci,
+ coalesce(nullIf(tupleElement(p, 'dicbo'), ''), nullIf(toValidUTF8(left(decodeURLFormComponent(extractURLParameter(_attribution_url, 'dicbo')), 2048)), '')) AS dicbo,
+ coalesce(nullIf(tupleElement(p, 'ndclid'), ''), nullIf(toValidUTF8(left(decodeURLFormComponent(extractURLParameter(_attribution_url, 'ndclid')), 2048)), '')) AS ndclid,
+ coalesce(nullIf(tupleElement(p, 'cjevent'), ''), nullIf(toValidUTF8(left(decodeURLFormComponent(extractURLParameter(_attribution_url, 'cjevent')), 2048)), '')) AS cjevent,
+ coalesce(nullIf(tupleElement(p, 'awc'), ''), nullIf(toValidUTF8(left(decodeURLFormComponent(extractURLParameter(_attribution_url, 'awc')), 2048)), '')) AS awc,
+ coalesce(nullIf(tupleElement(p, 'syclid'), ''), nullIf(toValidUTF8(left(decodeURLFormComponent(extractURLParameter(_attribution_url, 'syclid')), 2048)), '')) AS syclid,
+ coalesce(nullIf(tupleElement(p, '_bhlid'), ''), nullIf(toValidUTF8(left(decodeURLFormComponent(extractURLParameter(_attribution_url, '_bhlid')), 2048)), '')) AS _bhlid,
+ coalesce(nullIf(tupleElement(p, 'rtid'), ''), nullIf(toValidUTF8(left(decodeURLFormComponent(extractURLParameter(_attribution_url, 'rtid')), 2048)), '')) AS rtid,
+ coalesce(nullIf(tupleElement(p, 'oppref'), ''), nullIf(toValidUTF8(left(decodeURLFormComponent(extractURLParameter(_attribution_url, 'oppref')), 2048)), '')) AS oppref,
+ coalesce(nullIf(tupleElement(p, 'bfcid'), ''), nullIf(toValidUTF8(left(decodeURLFormComponent(extractURLParameter(_attribution_url, 'bfcid')), 2048)), '')) AS bfcid,
+ CAST(arrayFilter(x -> (x IS NOT NULL), [if(gclsrc IS NOT NULL, concat('gclsrc=', gclsrc), NULL), if(dclid IS NOT NULL, concat('dclid=', dclid), NULL), if(gbraid IS NOT NULL, concat('gbraid=', gbraid), NULL), if(wbraid IS NOT NULL, concat('wbraid=', wbraid), NULL), if(gad_campaignid IS NOT NULL, concat('gad_campaignid=', gad_campaignid), NULL), if(srsltid IS NOT NULL, concat('srsltid=', srsltid), NULL), if(msclkid IS NOT NULL, concat('msclkid=', msclkid), NULL), if(twclid IS NOT NULL, concat('twclid=', twclid), NULL), if(li_fat_id IS NOT NULL, concat('li_fat_id=', li_fat_id), NULL), if(mc_cid IS NOT NULL, concat('mc_cid=', mc_cid), NULL), if(igshid IS NOT NULL, concat('igshid=', igshid), NULL), if(ttclid IS NOT NULL, concat('ttclid=', ttclid), NULL), if(rdt_cid IS NOT NULL, concat('rdt_cid=', rdt_cid), NULL), if(epik IS NOT NULL, concat('epik=', epik), NULL), if(qclid IS NOT NULL, concat('qclid=', qclid), NULL), if(sccid IS NOT NULL, concat('sccid=', sccid), NULL), if(_kx IS NOT NULL, concat('_kx=', _kx), NULL), if(irclid IS NOT NULL, concat('irclid=', irclid), NULL), if(irclickid IS NOT NULL, concat('irclickid=', irclickid), NULL), if(utm_id IS NOT NULL, concat('utm_id=', utm_id), NULL), if(utm_source_platform IS NOT NULL, concat('utm_source_platform=', utm_source_platform), NULL), if(utm_creative_format IS NOT NULL, concat('utm_creative_format=', utm_creative_format), NULL), if(utm_marketing_tactic IS NOT NULL, concat('utm_marketing_tactic=', utm_marketing_tactic), NULL), if(yclid IS NOT NULL, concat('yclid=', yclid), NULL), if(tblci IS NOT NULL, concat('tblci=', tblci), NULL), if(dicbo IS NOT NULL, concat('dicbo=', dicbo), NULL), if(ndclid IS NOT NULL, concat('ndclid=', ndclid), NULL), if(cjevent IS NOT NULL, concat('cjevent=', cjevent), NULL), if(awc IS NOT NULL, concat('awc=', awc), NULL), if(syclid IS NOT NULL, concat('syclid=', syclid), NULL), if(_bhlid IS NOT NULL, concat('_bhlid=', _bhlid), NULL), if(rtid IS NOT NULL, concat('rtid=', rtid), NULL), if(oppref IS NOT NULL, concat('oppref=', oppref), NULL), if(bfcid IS NOT NULL, concat('bfcid=', bfcid), NULL)]), 'Array(String)') AS ad_ids_set,
+ tupleElement(p, '$host') AS _host,
+ if((event = '$pageview') OR (event = '$screen'), timestamp, timestamp + toIntervalYear(1)) AS pageview_prio_timestamp_min,
+ if((event = '$pageview') OR (event = '$screen'), timestamp, timestamp - toIntervalYear(1)) AS pageview_prio_timestamp_max
+SELECT
+ team_id,
+ `$session_id_uuid` AS session_id_v7,
+ initializeAggregation('argMaxState', source_table.distinct_id, timestamp) AS distinct_id,
+ initializeAggregation('groupUniqArrayState', source_table.distinct_id) AS distinct_ids,
+ timestamp AS min_timestamp,
+ timestamp AS max_timestamp,
+ inserted_at AS max_inserted_at,
+ if(
+ (_current_url IS NOT NULL) AND ((event = '$pageview') OR (event = '$screen')),
+ [_current_url],
+ []
+ ) AS urls,
+ initializeAggregation('argMinState', _current_url, pageview_prio_timestamp_min) AS entry_url,
+ initializeAggregation('argMaxState', _current_url, pageview_prio_timestamp_max) AS end_url,
+ initializeAggregation('argMaxState', _external_click_url, timestamp) AS last_external_click_url,
+ initializeAggregation('argMinState', _browser, timestamp) AS browser,
+ initializeAggregation('argMinState', _browser_version, timestamp) AS browser_version,
+ initializeAggregation('argMinState', _os, timestamp) AS os,
+ initializeAggregation('argMinState', _os_version, timestamp) AS os_version,
+ initializeAggregation('argMinState', _device_type, timestamp) AS device_type,
+ initializeAggregation('argMinState', _viewport_width, timestamp) AS viewport_width,
+ initializeAggregation('argMinState', _viewport_height, timestamp) AS viewport_height,
+ initializeAggregation('argMinState', _geoip_country_code, timestamp) AS geoip_country_code,
+ initializeAggregation('argMinState', _geoip_subdivision_1_code, timestamp) AS geoip_subdivision_1_code,
+ initializeAggregation('argMinState', _geoip_subdivision_1_name, timestamp) AS geoip_subdivision_1_name,
+ initializeAggregation('argMinState', _geoip_subdivision_city_name, timestamp) AS geoip_subdivision_city_name,
+ initializeAggregation('argMinState', _geoip_time_zone, timestamp) AS geoip_time_zone,
+ initializeAggregation('argMinState', _referring_domain, pageview_prio_timestamp_min) AS entry_referring_domain,
+ initializeAggregation('argMinState', _utm_source, pageview_prio_timestamp_min) AS entry_utm_source,
+ initializeAggregation('argMinState', _utm_campaign, pageview_prio_timestamp_min) AS entry_utm_campaign,
+ initializeAggregation('argMinState', _utm_medium, pageview_prio_timestamp_min) AS entry_utm_medium,
+ initializeAggregation('argMinState', _utm_term, pageview_prio_timestamp_min) AS entry_utm_term,
+ initializeAggregation('argMinState', _utm_content, pageview_prio_timestamp_min) AS entry_utm_content,
+ initializeAggregation('argMinState', _gclid, pageview_prio_timestamp_min) AS entry_gclid,
+ initializeAggregation('argMinState', _gad_source, pageview_prio_timestamp_min) AS entry_gad_source,
+ initializeAggregation('argMinState', _fbclid, pageview_prio_timestamp_min) AS entry_fbclid,
+ initializeAggregation('argMinState', nullIf(_gclid, '') IS NOT NULL, pageview_prio_timestamp_min) AS entry_has_gclid,
+ initializeAggregation('argMinState', nullIf(_fbclid, '') IS NOT NULL, pageview_prio_timestamp_min) AS entry_has_fbclid,
+ initializeAggregation('argMinState', ad_ids_set, pageview_prio_timestamp_min) AS entry_ad_ids_set,
+ initializeAggregation(
+ 'argMinState',
+ tuple(
+ _utm_source,
+ _utm_medium,
+ _utm_campaign,
+ _referring_domain,
+ nullIf(_gclid, '') IS NOT NULL,
+ nullIf(_fbclid, '') IS NOT NULL,
+ _gad_source
+ ),
+ pageview_prio_timestamp_min
+ ) AS entry_channel_type_properties,
+ initializeAggregation('uniqState', if(event = '$pageview', uuid, NULL)) AS pageview_uniq,
+ initializeAggregation('uniqState', if(event = '$autocapture', uuid, NULL)) AS autocapture_uniq,
+ initializeAggregation('uniqState', if(event = '$screen', uuid, NULL)) AS screen_uniq,
+ initializeAggregation(
+ 'uniqUpToState(1)',
+ if((event = '$pageview') OR (event = '$screen'), uuid, NULL)
+ ) AS page_screen_uniq_up_to,
+ event = '$autocapture' AS has_autocapture,
+ arrayMap(
+ (k, v) -> concat(k, '=', v),
+ mapKeys(properties_group_feature_flags),
+ mapValues(properties_group_feature_flags)
+ ) AS flag_key_values,
+ [event] AS event_names,
+ if((_host IS NOT NULL) AND (_host != ''), [_host], []) AS hosts,
+ if((_person_email IS NOT NULL) AND (_person_email != ''), [_person_email], []) AS emails,
+ false AS has_replay_events
+FROM posthog.sharded_events AS source_table
+WHERE
+ (bitAnd(bitShiftRight(toUInt128(accurateCastOrNull(`$session_id`, 'UUID')), 76), 15) = 7)
+AND
+ ((fromUnixTimestamp64Milli(toUInt64(bitShiftRight(`$session_id_uuid`, 80))) >= (timestamp - toIntervalDay(31))) AND (fromUnixTimestamp64Milli(toUInt64(bitShiftRight(`$session_id_uuid`, 80))) <= (timestamp + toIntervalDay(31))))
+AND
+ true;
+CREATE MATERIALIZED VIEW posthog.raw_sessions_v3_recordings_mv TO posthog.writable_raw_sessions_v3 (team_id Int64, session_id_v7 UInt128, session_timestamp DateTime64(3), distinct_id AggregateFunction(argMax, String, DateTime64(6)), distinct_ids AggregateFunction(groupUniqArray, String), min_timestamp SimpleAggregateFunction(min, DateTime64(6, 'UTC')), max_timestamp SimpleAggregateFunction(min, DateTime64(6, 'UTC')), max_inserted_at DateTime, urls Array(String), entry_url AggregateFunction(argMin, Nullable(String), DateTime64(6)), end_url AggregateFunction(argMax, Nullable(String), DateTime64(6)), last_external_click_url AggregateFunction(argMax, Nullable(String), DateTime64(6)), browser AggregateFunction(argMin, Nullable(String), DateTime64(6)), browser_version AggregateFunction(argMin, Nullable(String), DateTime64(6)), os AggregateFunction(argMin, Nullable(String), DateTime64(6)), os_version AggregateFunction(argMin, Nullable(String), DateTime64(6)), device_type AggregateFunction(argMin, Nullable(String), DateTime64(6)), viewport_width AggregateFunction(argMin, Nullable(Int64), DateTime64(6)), viewport_height AggregateFunction(argMin, Nullable(Int64), DateTime64(6)), geoip_country_code AggregateFunction(argMin, Nullable(String), DateTime64(6)), geoip_subdivision_1_code AggregateFunction(argMin, Nullable(String), DateTime64(6)), geoip_subdivision_1_name AggregateFunction(argMin, Nullable(String), DateTime64(6)), geoip_subdivision_city_name AggregateFunction(argMin, Nullable(String), DateTime64(6)), geoip_time_zone AggregateFunction(argMin, Nullable(String), DateTime64(6)), entry_referring_domain AggregateFunction(argMin, Nullable(String), DateTime64(6)), entry_utm_source AggregateFunction(argMin, Nullable(String), DateTime64(6)), entry_utm_campaign AggregateFunction(argMin, Nullable(String), DateTime64(6)), entry_utm_medium AggregateFunction(argMin, Nullable(String), DateTime64(6)), entry_utm_term AggregateFunction(argMin, Nullable(String), DateTime64(6)), entry_utm_content AggregateFunction(argMin, Nullable(String), DateTime64(6)), entry_gclid AggregateFunction(argMin, Nullable(String), DateTime64(6)), entry_gad_source AggregateFunction(argMin, Nullable(String), DateTime64(6)), entry_fbclid AggregateFunction(argMin, Nullable(String), DateTime64(6)), entry_has_gclid AggregateFunction(argMin, Bool, DateTime64(6)), entry_has_fbclid AggregateFunction(argMin, Bool, DateTime64(6)), entry_ad_ids_set AggregateFunction(argMin, Array(String), DateTime64(6)), entry_channel_type_properties AggregateFunction(argMin, Tuple(Nullable(String), Nullable(String), Nullable(String), Nullable(String), Bool, Bool, Nullable(String)), DateTime64(6)), pageview_uniq AggregateFunction(uniq, Nullable(UUID)), autocapture_uniq AggregateFunction(uniq, Nullable(UUID)), screen_uniq AggregateFunction(uniq, Nullable(UUID)), page_screen_uniq_up_to AggregateFunction(uniqUpTo(1), Nullable(UUID)), has_autocapture Bool, flag_key_values Array(String), event_names Array(String), hosts Array(String), emails Array(String), has_replay_events Bool) AS WITH
+ min_first_timestamp AS timestamp,
+ CAST(fromUnixTimestamp64Milli(9223372036854775), 'DateTime64(6)') AS max_ts_64,
+ CAST(fromUnixTimestamp64Milli(-9223372036854775), 'DateTime64(6)') AS min_ts_64,
+ CAST(NULL, 'Nullable(String)') AS null_s,
+ CAST(NULL, 'Nullable(Int64)') AS null_i64,
+ CAST(NULL, 'Nullable(UUID)') AS null_uuid
+SELECT
+ team_id,
+ toUInt128(accurateCast(session_id, 'UUID')) AS session_id_v7,
+ fromUnixTimestamp64Milli(toUInt64(bitShiftRight(session_id_v7, 80))) AS session_timestamp,
+ initializeAggregation('argMaxState', source_table.distinct_id, min_ts_64) AS distinct_id,
+ initializeAggregation('groupUniqArrayState', source_table.distinct_id) AS distinct_ids,
+ timestamp AS min_timestamp,
+ timestamp AS max_timestamp,
+ fromUnixTimestamp(0) AS max_inserted_at,
+ CAST([], 'Array(String)') AS urls,
+ initializeAggregation('argMinState', null_s, max_ts_64) AS entry_url,
+ initializeAggregation('argMaxState', null_s, min_ts_64) AS end_url,
+ initializeAggregation('argMaxState', null_s, min_ts_64) AS last_external_click_url,
+ initializeAggregation('argMinState', null_s, max_ts_64) AS browser,
+ initializeAggregation('argMinState', null_s, max_ts_64) AS browser_version,
+ initializeAggregation('argMinState', null_s, max_ts_64) AS os,
+ initializeAggregation('argMinState', null_s, max_ts_64) AS os_version,
+ initializeAggregation('argMinState', null_s, max_ts_64) AS device_type,
+ initializeAggregation('argMinState', null_i64, max_ts_64) AS viewport_width,
+ initializeAggregation('argMinState', null_i64, max_ts_64) AS viewport_height,
+ initializeAggregation('argMinState', null_s, max_ts_64) AS geoip_country_code,
+ initializeAggregation('argMinState', null_s, max_ts_64) AS geoip_subdivision_1_code,
+ initializeAggregation('argMinState', null_s, max_ts_64) AS geoip_subdivision_1_name,
+ initializeAggregation('argMinState', null_s, max_ts_64) AS geoip_subdivision_city_name,
+ initializeAggregation('argMinState', null_s, max_ts_64) AS geoip_time_zone,
+ initializeAggregation('argMinState', null_s, max_ts_64) AS entry_referring_domain,
+ initializeAggregation('argMinState', null_s, max_ts_64) AS entry_utm_source,
+ initializeAggregation('argMinState', null_s, max_ts_64) AS entry_utm_campaign,
+ initializeAggregation('argMinState', null_s, max_ts_64) AS entry_utm_medium,
+ initializeAggregation('argMinState', null_s, max_ts_64) AS entry_utm_term,
+ initializeAggregation('argMinState', null_s, max_ts_64) AS entry_utm_content,
+ initializeAggregation('argMinState', null_s, max_ts_64) AS entry_gclid,
+ initializeAggregation('argMinState', null_s, max_ts_64) AS entry_gad_source,
+ initializeAggregation('argMinState', null_s, max_ts_64) AS entry_fbclid,
+ initializeAggregation('argMinState', false, max_ts_64) AS entry_has_gclid,
+ initializeAggregation('argMinState', false, max_ts_64) AS entry_has_fbclid,
+ initializeAggregation('argMinState', CAST([], 'Array(String)'), max_ts_64) AS entry_ad_ids_set,
+ initializeAggregation(
+ 'argMinState',
+ tuple(null_s, null_s, null_s, null_s, false, false, null_s),
+ max_ts_64
+ ) AS entry_channel_type_properties,
+ initializeAggregation('uniqState', null_uuid) AS pageview_uniq,
+ initializeAggregation('uniqState', null_uuid) AS autocapture_uniq,
+ initializeAggregation('uniqState', null_uuid) AS screen_uniq,
+ initializeAggregation('uniqUpToState(1)', null_uuid) AS page_screen_uniq_up_to,
+ false AS has_autocapture,
+ CAST([], 'Array(String)') AS flag_key_values,
+ CAST([], 'Array(String)') AS event_names,
+ CAST([], 'Array(String)') AS hosts,
+ CAST([], 'Array(String)') AS emails,
+ true AS has_replay_events
+FROM posthog.sharded_session_replay_events AS source_table
+WHERE
+ (bitAnd(bitShiftRight(toUInt128(accurateCastOrNull(session_id, 'UUID')), 76), 15) = 7)
+AND
+ ((fromUnixTimestamp64Milli(toUInt64(bitShiftRight(toUInt128(accurateCastOrNull(session_id, 'UUID')), 80))) >= (min_first_timestamp - toIntervalDay(31))) AND (fromUnixTimestamp64Milli(toUInt64(bitShiftRight(toUInt128(accurateCastOrNull(session_id, 'UUID')), 80))) <= (min_first_timestamp + toIntervalDay(31))))
+AND
+ true;
CREATE MATERIALIZED VIEW posthog.session_replay_events_mv TO posthog.writable_session_replay_events (session_id String, team_id Int64, distinct_id String, min_first_timestamp DateTime64(6, 'UTC'), max_last_timestamp DateTime64(6, 'UTC'), block_first_timestamps SimpleAggregateFunction(groupArrayArray, Array(DateTime64(6, 'UTC'))), block_last_timestamps SimpleAggregateFunction(groupArrayArray, Array(DateTime64(6, 'UTC'))), block_urls SimpleAggregateFunction(groupArrayArray, Array(String)), first_url AggregateFunction(argMin, Nullable(String), DateTime64(6, 'UTC')), all_urls SimpleAggregateFunction(groupUniqArrayArray, Array(String)), click_count Int64, keypress_count Int64, mouse_activity_count Int64, active_milliseconds Int64, console_log_count Int64, console_warn_count Int64, console_error_count Int64, size Int64, message_count Int64, event_count Int64, snapshot_source AggregateFunction(argMin, Nullable(String), DateTime64(6, 'UTC')), snapshot_library AggregateFunction(argMin, Nullable(String), DateTime64(6, 'UTC')), snapshot_mode_v2 AggregateFunction(argMin, Nullable(String), DateTime64(6, 'UTC')), _timestamp Nullable(DateTime), retention_period_days SimpleAggregateFunction(max, Nullable(Int64)), is_deleted SimpleAggregateFunction(max, UInt8), ai_tags_fixed SimpleAggregateFunction(groupUniqArrayArray, Array(String)), ai_tags_freeform SimpleAggregateFunction(groupUniqArrayArray, Array(String)), ai_highlighted SimpleAggregateFunction(max, UInt8), surfacing_score SimpleAggregateFunction(max, Nullable(Float32))) AS SELECT
session_id,
team_id,
@@ -7335,16 +7538,14 @@ CREATE TABLE posthog.raw_sessions_v3 (
entry_fbclid AggregateFunction(argMin, Nullable(String), DateTime64(6, 'UTC')),
entry_has_gclid AggregateFunction(argMin, Bool, DateTime64(6, 'UTC')),
entry_has_fbclid AggregateFunction(argMin, Bool, DateTime64(6, 'UTC')),
- entry_ad_ids_map AggregateFunction(argMin, Map(String, String), DateTime64(6, 'UTC')),
entry_ad_ids_set AggregateFunction(argMin, Array(String), DateTime64(6, 'UTC')),
entry_channel_type_properties AggregateFunction(argMin, Tuple(Nullable(String), Nullable(String), Nullable(String), Nullable(String), Bool, Bool, Nullable(String)), DateTime64(6, 'UTC')),
- pageview_uniq AggregateFunction(uniqExact, Nullable(UUID)),
- autocapture_uniq AggregateFunction(uniqExact, Nullable(UUID)),
- screen_uniq AggregateFunction(uniqExact, Nullable(UUID)),
+ pageview_uniq AggregateFunction(uniq, Nullable(UUID)),
+ autocapture_uniq AggregateFunction(uniq, Nullable(UUID)),
+ screen_uniq AggregateFunction(uniq, Nullable(UUID)),
page_screen_uniq_up_to AggregateFunction(uniqUpTo(1), Nullable(UUID)),
has_autocapture SimpleAggregateFunction(max, Bool),
flag_key_values SimpleAggregateFunction(groupUniqArrayArray(10000), Array(String)),
- flag_keys SimpleAggregateFunction(groupUniqArrayArray, Array(String)),
event_names SimpleAggregateFunction(groupUniqArrayArray(2000), Array(String)),
hosts SimpleAggregateFunction(groupUniqArrayArray(100), Array(String)),
emails SimpleAggregateFunction(groupUniqArrayArray(10), Array(String)),
@@ -8125,16 +8326,14 @@ CREATE VIEW posthog.raw_sessions_v3_v AS SELECT
argMinMerge(entry_fbclid) AS entry_fbclid,
argMinMerge(entry_has_gclid) AS entry_has_gclid,
argMinMerge(entry_has_fbclid) AS entry_has_fbclid,
- argMinMerge(entry_ad_ids_map) AS entry_ad_ids_map,
argMinMerge(entry_ad_ids_set) AS entry_ad_ids_set,
argMinMerge(entry_channel_type_properties) AS entry_channel_type_properties,
- uniqExactMerge(pageview_uniq) AS pageview_uniq,
- uniqExactMerge(autocapture_uniq) AS autocapture_uniq,
- uniqExactMerge(screen_uniq) AS screen_uniq,
+ uniqMerge(pageview_uniq) AS pageview_uniq,
+ uniqMerge(autocapture_uniq) AS autocapture_uniq,
+ uniqMerge(screen_uniq) AS screen_uniq,
uniqUpToMerge(1)(page_screen_uniq_up_to) AS page_screen_uniq_up_to,
max(has_autocapture) AS has_autocapture,
groupUniqArrayArray(10000)(flag_key_values) AS flag_key_values,
- groupUniqArrayArray(flag_keys) AS flag_keys,
groupUniqArrayArray(2000)(event_names) AS event_names,
groupUniqArrayArray(100)(hosts) AS hosts,
groupUniqArrayArray(10)(emails) AS emails,
diff --git a/posthog/clickhouse/hcl/sql/prod-eu/data.sql b/posthog/clickhouse/hcl/sql/prod-eu/data.sql
index ce33ba61df44..491415a21438 100644
--- a/posthog/clickhouse/hcl/sql/prod-eu/data.sql
+++ b/posthog/clickhouse/hcl/sql/prod-eu/data.sql
@@ -1181,17 +1181,14 @@ CREATE TABLE posthog.raw_sessions_v3 (
entry_fbclid AggregateFunction(argMin, Nullable(String), DateTime64(6, 'UTC')),
entry_has_gclid AggregateFunction(argMin, Bool, DateTime64(6, 'UTC')),
entry_has_fbclid AggregateFunction(argMin, Bool, DateTime64(6, 'UTC')),
- entry_ad_ids_map AggregateFunction(argMin, Map(String, String), DateTime64(6, 'UTC')),
entry_ad_ids_set AggregateFunction(argMin, Array(String), DateTime64(6, 'UTC')),
entry_channel_type_properties AggregateFunction(argMin, Tuple(Nullable(String), Nullable(String), Nullable(String), Nullable(String), Bool, Bool, Nullable(String)), DateTime64(6, 'UTC')),
- pageview_uniq AggregateFunction(uniqExact, Nullable(UUID)),
- autocapture_uniq AggregateFunction(uniqExact, Nullable(UUID)),
- screen_uniq AggregateFunction(uniqExact, Nullable(UUID)),
+ pageview_uniq AggregateFunction(uniq, Nullable(UUID)),
+ autocapture_uniq AggregateFunction(uniq, Nullable(UUID)),
+ screen_uniq AggregateFunction(uniq, Nullable(UUID)),
page_screen_uniq_up_to AggregateFunction(uniqUpTo(1), Nullable(UUID)),
has_autocapture SimpleAggregateFunction(max, Bool),
flag_key_values SimpleAggregateFunction(groupUniqArrayArray(10000), Array(String)),
- flag_values AggregateFunction(groupUniqArrayMap, Map(String, String)),
- flag_keys SimpleAggregateFunction(groupUniqArrayArray, Array(String)),
event_names SimpleAggregateFunction(groupUniqArrayArray, Array(String)),
hosts SimpleAggregateFunction(groupUniqArrayArray(100), Array(String)),
emails SimpleAggregateFunction(groupUniqArrayArray(10), Array(String)),
@@ -1650,24 +1647,20 @@ CREATE TABLE posthog.sharded_raw_sessions_v3 (
entry_fbclid AggregateFunction(argMin, Nullable(String), DateTime64(6, 'UTC')),
entry_has_gclid AggregateFunction(argMin, Bool, DateTime64(6, 'UTC')),
entry_has_fbclid AggregateFunction(argMin, Bool, DateTime64(6, 'UTC')),
- entry_ad_ids_map AggregateFunction(argMin, Map(String, String), DateTime64(6, 'UTC')),
entry_ad_ids_set AggregateFunction(argMin, Array(String), DateTime64(6, 'UTC')),
entry_channel_type_properties AggregateFunction(argMin, Tuple(Nullable(String), Nullable(String), Nullable(String), Nullable(String), Bool, Bool, Nullable(String)), DateTime64(6, 'UTC')),
- pageview_uniq AggregateFunction(uniqExact, Nullable(UUID)),
- autocapture_uniq AggregateFunction(uniqExact, Nullable(UUID)),
- screen_uniq AggregateFunction(uniqExact, Nullable(UUID)),
+ pageview_uniq AggregateFunction(uniq, Nullable(UUID)),
+ autocapture_uniq AggregateFunction(uniq, Nullable(UUID)),
+ screen_uniq AggregateFunction(uniq, Nullable(UUID)),
page_screen_uniq_up_to AggregateFunction(uniqUpTo(1), Nullable(UUID)),
has_autocapture SimpleAggregateFunction(max, Bool),
flag_key_values SimpleAggregateFunction(groupUniqArrayArray(10000), Array(String)),
- flag_values AggregateFunction(groupUniqArrayMap, Map(String, String)),
- flag_keys SimpleAggregateFunction(groupUniqArrayArray, Array(String)),
event_names SimpleAggregateFunction(groupUniqArrayArray, Array(String)),
hosts SimpleAggregateFunction(groupUniqArrayArray(100), Array(String)),
emails SimpleAggregateFunction(groupUniqArrayArray(10), Array(String)),
has_replay_events SimpleAggregateFunction(max, Bool),
INDEX event_names_bloom_filter event_names TYPE bloom_filter() GRANULARITY 1,
- INDEX flag_keys_bloom_filter flag_keys TYPE bloom_filter() GRANULARITY 1,
- INDEX flag_key_values_bloom_filter flag_key_values TYPE bloom_filter() GRANULARITY 1,
+ INDEX flag_key_values_text flag_key_values TYPE text(tokenizer = splitByString(['='])) GRANULARITY 100000000,
INDEX hosts_bloom_filter hosts TYPE bloom_filter() GRANULARITY 1,
INDEX emails_bloom_filter emails TYPE bloom_filter() GRANULARITY 1
) ENGINE = ReplicatedAggregatingMergeTree('/clickhouse/tables/{shard}/posthog.raw_sessions_v3', '{replica}') ORDER BY (team_id, session_timestamp, session_id_v7) PARTITION BY toYYYYMM(session_timestamp) SETTINGS index_granularity = 8192;
@@ -2475,17 +2468,14 @@ CREATE TABLE posthog.writable_raw_sessions_v3 (
entry_fbclid AggregateFunction(argMin, Nullable(String), DateTime64(6, 'UTC')),
entry_has_gclid AggregateFunction(argMin, Bool, DateTime64(6, 'UTC')),
entry_has_fbclid AggregateFunction(argMin, Bool, DateTime64(6, 'UTC')),
- entry_ad_ids_map AggregateFunction(argMin, Map(String, String), DateTime64(6, 'UTC')),
entry_ad_ids_set AggregateFunction(argMin, Array(String), DateTime64(6, 'UTC')),
entry_channel_type_properties AggregateFunction(argMin, Tuple(Nullable(String), Nullable(String), Nullable(String), Nullable(String), Bool, Bool, Nullable(String)), DateTime64(6, 'UTC')),
- pageview_uniq AggregateFunction(uniqExact, Nullable(UUID)),
- autocapture_uniq AggregateFunction(uniqExact, Nullable(UUID)),
- screen_uniq AggregateFunction(uniqExact, Nullable(UUID)),
+ pageview_uniq AggregateFunction(uniq, Nullable(UUID)),
+ autocapture_uniq AggregateFunction(uniq, Nullable(UUID)),
+ screen_uniq AggregateFunction(uniq, Nullable(UUID)),
page_screen_uniq_up_to AggregateFunction(uniqUpTo(1), Nullable(UUID)),
has_autocapture SimpleAggregateFunction(max, Bool),
flag_key_values SimpleAggregateFunction(groupUniqArrayArray(10000), Array(String)),
- flag_values AggregateFunction(groupUniqArrayMap, Map(String, String)),
- flag_keys SimpleAggregateFunction(groupUniqArrayArray, Array(String)),
event_names SimpleAggregateFunction(groupUniqArrayArray, Array(String)),
hosts SimpleAggregateFunction(groupUniqArrayArray(100), Array(String)),
emails SimpleAggregateFunction(groupUniqArrayArray(10), Array(String)),
@@ -3080,16 +3070,14 @@ CREATE VIEW posthog.raw_sessions_v3_v AS SELECT
argMinMerge(entry_fbclid) AS entry_fbclid,
argMinMerge(entry_has_gclid) AS entry_has_gclid,
argMinMerge(entry_has_fbclid) AS entry_has_fbclid,
- argMinMerge(entry_ad_ids_map) AS entry_ad_ids_map,
argMinMerge(entry_ad_ids_set) AS entry_ad_ids_set,
argMinMerge(entry_channel_type_properties) AS entry_channel_type_properties,
- uniqExactMerge(pageview_uniq) AS pageview_uniq,
- uniqExactMerge(autocapture_uniq) AS autocapture_uniq,
- uniqExactMerge(screen_uniq) AS screen_uniq,
+ uniqMerge(pageview_uniq) AS pageview_uniq,
+ uniqMerge(autocapture_uniq) AS autocapture_uniq,
+ uniqMerge(screen_uniq) AS screen_uniq,
uniqUpToMerge(1)(page_screen_uniq_up_to) AS page_screen_uniq_up_to,
max(has_autocapture) AS has_autocapture,
groupUniqArrayArray(10000)(flag_key_values) AS flag_key_values,
- groupUniqArrayArray(flag_keys) AS flag_keys,
groupUniqArrayArray(2000)(event_names) AS event_names,
groupUniqArrayArray(100)(hosts) AS hosts,
groupUniqArrayArray(10)(emails) AS emails,
diff --git a/posthog/clickhouse/hcl/sql/prod-eu/sessions.sql b/posthog/clickhouse/hcl/sql/prod-eu/sessions.sql
index 085954cd688b..315ecb4af203 100644
--- a/posthog/clickhouse/hcl/sql/prod-eu/sessions.sql
+++ b/posthog/clickhouse/hcl/sql/prod-eu/sessions.sql
@@ -320,24 +320,20 @@ CREATE TABLE posthog.raw_sessions_v3 (
entry_fbclid AggregateFunction(argMin, Nullable(String), DateTime64(6, 'UTC')),
entry_has_gclid AggregateFunction(argMin, Bool, DateTime64(6, 'UTC')),
entry_has_fbclid AggregateFunction(argMin, Bool, DateTime64(6, 'UTC')),
- entry_ad_ids_map AggregateFunction(argMin, Map(String, String), DateTime64(6, 'UTC')),
entry_ad_ids_set AggregateFunction(argMin, Array(String), DateTime64(6, 'UTC')),
entry_channel_type_properties AggregateFunction(argMin, Tuple(Nullable(String), Nullable(String), Nullable(String), Nullable(String), Bool, Bool, Nullable(String)), DateTime64(6, 'UTC')),
- pageview_uniq AggregateFunction(uniqExact, Nullable(UUID)),
- autocapture_uniq AggregateFunction(uniqExact, Nullable(UUID)),
- screen_uniq AggregateFunction(uniqExact, Nullable(UUID)),
+ pageview_uniq AggregateFunction(uniq, Nullable(UUID)),
+ autocapture_uniq AggregateFunction(uniq, Nullable(UUID)),
+ screen_uniq AggregateFunction(uniq, Nullable(UUID)),
page_screen_uniq_up_to AggregateFunction(uniqUpTo(1), Nullable(UUID)),
has_autocapture SimpleAggregateFunction(max, Bool),
flag_key_values SimpleAggregateFunction(groupUniqArrayArray(10000), Array(String)),
- flag_values AggregateFunction(groupUniqArrayMap, Map(String, String)),
- flag_keys SimpleAggregateFunction(groupUniqArrayArray, Array(String)),
event_names SimpleAggregateFunction(groupUniqArrayArray, Array(String)),
hosts SimpleAggregateFunction(groupUniqArrayArray(100), Array(String)),
emails SimpleAggregateFunction(groupUniqArrayArray(10), Array(String)),
has_replay_events SimpleAggregateFunction(max, Bool),
INDEX event_names_bloom_filter event_names TYPE bloom_filter() GRANULARITY 1,
- INDEX flag_keys_bloom_filter flag_keys TYPE bloom_filter() GRANULARITY 1,
- INDEX flag_key_values_bloom_filter flag_key_values TYPE bloom_filter() GRANULARITY 1
+ INDEX flag_key_values_text flag_key_values TYPE text(tokenizer = splitByString(['='])) GRANULARITY 100000000
) ENGINE = ReplicatedAggregatingMergeTree('/clickhouse/tables/sessions/noshard/posthog.raw_sessions_v3', '{shard}-{replica}') ORDER BY (team_id, session_timestamp, session_id_v7) PARTITION BY toYYYYMM(session_timestamp) SETTINGS index_granularity = 8192, storage_policy = 's3_tiered';
CREATE TABLE posthog.sessions (
session_id String,
diff --git a/posthog/clickhouse/hcl/sql/prod-us/data.sql b/posthog/clickhouse/hcl/sql/prod-us/data.sql
index 1ee516670807..b1acd23b5ffb 100644
--- a/posthog/clickhouse/hcl/sql/prod-us/data.sql
+++ b/posthog/clickhouse/hcl/sql/prod-us/data.sql
@@ -1014,6 +1014,55 @@ CREATE TABLE posthog.raw_sessions (
page_screen_autocapture_uniq_up_to AggregateFunction(uniqUpTo(1), Nullable(UUID)),
vitals_lcp AggregateFunction(argMin, Nullable(Float64), DateTime64(6, 'UTC'))
) ENGINE = Distributed('sessions', 'posthog', 'raw_sessions', cityHash64(session_id_v7));
+CREATE TABLE posthog.raw_sessions_v3 (
+ team_id Int64,
+ session_id_v7 UInt128,
+ session_timestamp DateTime64(3) MATERIALIZED fromUnixTimestamp64Milli(toUInt64(bitShiftRight(session_id_v7, 80))),
+ distinct_id AggregateFunction(argMax, String, DateTime64(6, 'UTC')),
+ distinct_ids AggregateFunction(groupUniqArray, String),
+ min_timestamp SimpleAggregateFunction(min, DateTime64(6, 'UTC')),
+ max_timestamp SimpleAggregateFunction(max, DateTime64(6, 'UTC')),
+ max_inserted_at SimpleAggregateFunction(max, DateTime64(6, 'UTC')),
+ urls SimpleAggregateFunction(groupUniqArrayArray(2000), Array(String)),
+ entry_url AggregateFunction(argMin, Nullable(String), DateTime64(6, 'UTC')),
+ end_url AggregateFunction(argMax, Nullable(String), DateTime64(6, 'UTC')),
+ last_external_click_url AggregateFunction(argMax, Nullable(String), DateTime64(6, 'UTC')),
+ browser AggregateFunction(argMin, Nullable(String), DateTime64(6, 'UTC')),
+ browser_version AggregateFunction(argMin, Nullable(String), DateTime64(6, 'UTC')),
+ os AggregateFunction(argMin, Nullable(String), DateTime64(6, 'UTC')),
+ os_version AggregateFunction(argMin, Nullable(String), DateTime64(6, 'UTC')),
+ device_type AggregateFunction(argMin, Nullable(String), DateTime64(6, 'UTC')),
+ viewport_width AggregateFunction(argMin, Nullable(Int64), DateTime64(6, 'UTC')),
+ viewport_height AggregateFunction(argMin, Nullable(Int64), DateTime64(6, 'UTC')),
+ geoip_country_code AggregateFunction(argMin, Nullable(String), DateTime64(6, 'UTC')),
+ geoip_subdivision_1_code AggregateFunction(argMin, Nullable(String), DateTime64(6, 'UTC')),
+ geoip_subdivision_1_name AggregateFunction(argMin, Nullable(String), DateTime64(6, 'UTC')),
+ geoip_subdivision_city_name AggregateFunction(argMin, Nullable(String), DateTime64(6, 'UTC')),
+ geoip_time_zone AggregateFunction(argMin, Nullable(String), DateTime64(6, 'UTC')),
+ entry_referring_domain AggregateFunction(argMin, Nullable(String), DateTime64(6, 'UTC')),
+ entry_utm_source AggregateFunction(argMin, Nullable(String), DateTime64(6, 'UTC')),
+ entry_utm_campaign AggregateFunction(argMin, Nullable(String), DateTime64(6, 'UTC')),
+ entry_utm_medium AggregateFunction(argMin, Nullable(String), DateTime64(6, 'UTC')),
+ entry_utm_term AggregateFunction(argMin, Nullable(String), DateTime64(6, 'UTC')),
+ entry_utm_content AggregateFunction(argMin, Nullable(String), DateTime64(6, 'UTC')),
+ entry_gclid AggregateFunction(argMin, Nullable(String), DateTime64(6, 'UTC')),
+ entry_gad_source AggregateFunction(argMin, Nullable(String), DateTime64(6, 'UTC')),
+ entry_fbclid AggregateFunction(argMin, Nullable(String), DateTime64(6, 'UTC')),
+ entry_has_gclid AggregateFunction(argMin, Bool, DateTime64(6, 'UTC')),
+ entry_has_fbclid AggregateFunction(argMin, Bool, DateTime64(6, 'UTC')),
+ entry_ad_ids_set AggregateFunction(argMin, Array(String), DateTime64(6, 'UTC')),
+ entry_channel_type_properties AggregateFunction(argMin, Tuple(Nullable(String), Nullable(String), Nullable(String), Nullable(String), Bool, Bool, Nullable(String)), DateTime64(6, 'UTC')),
+ pageview_uniq AggregateFunction(uniq, Nullable(UUID)),
+ autocapture_uniq AggregateFunction(uniq, Nullable(UUID)),
+ screen_uniq AggregateFunction(uniq, Nullable(UUID)),
+ page_screen_uniq_up_to AggregateFunction(uniqUpTo(1), Nullable(UUID)),
+ has_autocapture SimpleAggregateFunction(max, Bool),
+ flag_key_values SimpleAggregateFunction(groupUniqArrayArray(10000), Array(String)),
+ event_names SimpleAggregateFunction(groupUniqArrayArray, Array(String)),
+ hosts SimpleAggregateFunction(groupUniqArrayArray(100), Array(String)),
+ emails SimpleAggregateFunction(groupUniqArrayArray(10), Array(String)),
+ has_replay_events SimpleAggregateFunction(max, Bool)
+) ENGINE = Distributed('sessions', 'posthog', 'raw_sessions_v3', cityHash64(session_id_v7));
CREATE TABLE posthog.session_recording_events (
uuid UUID,
timestamp DateTime64(6, 'UTC'),
@@ -1499,24 +1548,20 @@ CREATE TABLE posthog.sharded_raw_sessions_v3 (
entry_fbclid AggregateFunction(argMin, Nullable(String), DateTime64(6, 'UTC')),
entry_has_gclid AggregateFunction(argMin, Bool, DateTime64(6, 'UTC')),
entry_has_fbclid AggregateFunction(argMin, Bool, DateTime64(6, 'UTC')),
- entry_ad_ids_map AggregateFunction(argMin, Map(String, String), DateTime64(6, 'UTC')),
entry_ad_ids_set AggregateFunction(argMin, Array(String), DateTime64(6, 'UTC')),
entry_channel_type_properties AggregateFunction(argMin, Tuple(Nullable(String), Nullable(String), Nullable(String), Nullable(String), Bool, Bool, Nullable(String)), DateTime64(6, 'UTC')),
- pageview_uniq AggregateFunction(uniqExact, Nullable(UUID)),
- autocapture_uniq AggregateFunction(uniqExact, Nullable(UUID)),
- screen_uniq AggregateFunction(uniqExact, Nullable(UUID)),
+ pageview_uniq AggregateFunction(uniq, Nullable(UUID)),
+ autocapture_uniq AggregateFunction(uniq, Nullable(UUID)),
+ screen_uniq AggregateFunction(uniq, Nullable(UUID)),
page_screen_uniq_up_to AggregateFunction(uniqUpTo(1), Nullable(UUID)),
has_autocapture SimpleAggregateFunction(max, Bool),
flag_key_values SimpleAggregateFunction(groupUniqArrayArray(10000), Array(String)),
- flag_values AggregateFunction(groupUniqArrayMap, Map(String, String)),
- flag_keys SimpleAggregateFunction(groupUniqArrayArray, Array(String)),
event_names SimpleAggregateFunction(groupUniqArrayArray, Array(String)),
hosts SimpleAggregateFunction(groupUniqArrayArray(100), Array(String)),
emails SimpleAggregateFunction(groupUniqArrayArray(10), Array(String)),
has_replay_events SimpleAggregateFunction(max, Bool),
INDEX event_names_bloom_filter event_names TYPE bloom_filter() GRANULARITY 1,
- INDEX flag_keys_bloom_filter flag_keys TYPE bloom_filter() GRANULARITY 1,
- INDEX flag_key_values_bloom_filter flag_key_values TYPE bloom_filter() GRANULARITY 1,
+ INDEX flag_key_values_text flag_key_values TYPE text(tokenizer = splitByString(['='])) GRANULARITY 100000000,
INDEX hosts_bloom_filter hosts TYPE bloom_filter() GRANULARITY 1,
INDEX emails_bloom_filter emails TYPE bloom_filter() GRANULARITY 1
) ENGINE = ReplicatedAggregatingMergeTree('/clickhouse/tables/reshard/{shard}/posthog.raw_sessions_v3', '{replica}') ORDER BY (team_id, session_timestamp, session_id_v7) PARTITION BY toYYYYMM(session_timestamp) SETTINGS index_granularity = 8192;
@@ -2290,6 +2335,55 @@ CREATE TABLE posthog.writable_raw_sessions_primary (
vitals_lcp AggregateFunction(argMin, Nullable(Float64), DateTime64(6, 'UTC')),
initial__kx AggregateFunction(argMin, String, DateTime64(6, 'UTC'))
) ENGINE = Distributed('posthog', 'posthog', 'sharded_raw_sessions', cityHash64(session_id_v7));
+CREATE TABLE posthog.writable_raw_sessions_v3 (
+ team_id Int64,
+ session_id_v7 UInt128,
+ session_timestamp DateTime64(3) MATERIALIZED fromUnixTimestamp64Milli(toUInt64(bitShiftRight(session_id_v7, 80))),
+ distinct_id AggregateFunction(argMax, String, DateTime64(6, 'UTC')),
+ distinct_ids AggregateFunction(groupUniqArray, String),
+ min_timestamp SimpleAggregateFunction(min, DateTime64(6, 'UTC')),
+ max_timestamp SimpleAggregateFunction(max, DateTime64(6, 'UTC')),
+ max_inserted_at SimpleAggregateFunction(max, DateTime64(6, 'UTC')),
+ urls SimpleAggregateFunction(groupUniqArrayArray(2000), Array(String)),
+ entry_url AggregateFunction(argMin, Nullable(String), DateTime64(6, 'UTC')),
+ end_url AggregateFunction(argMax, Nullable(String), DateTime64(6, 'UTC')),
+ last_external_click_url AggregateFunction(argMax, Nullable(String), DateTime64(6, 'UTC')),
+ browser AggregateFunction(argMin, Nullable(String), DateTime64(6, 'UTC')),
+ browser_version AggregateFunction(argMin, Nullable(String), DateTime64(6, 'UTC')),
+ os AggregateFunction(argMin, Nullable(String), DateTime64(6, 'UTC')),
+ os_version AggregateFunction(argMin, Nullable(String), DateTime64(6, 'UTC')),
+ device_type AggregateFunction(argMin, Nullable(String), DateTime64(6, 'UTC')),
+ viewport_width AggregateFunction(argMin, Nullable(Int64), DateTime64(6, 'UTC')),
+ viewport_height AggregateFunction(argMin, Nullable(Int64), DateTime64(6, 'UTC')),
+ geoip_country_code AggregateFunction(argMin, Nullable(String), DateTime64(6, 'UTC')),
+ geoip_subdivision_1_code AggregateFunction(argMin, Nullable(String), DateTime64(6, 'UTC')),
+ geoip_subdivision_1_name AggregateFunction(argMin, Nullable(String), DateTime64(6, 'UTC')),
+ geoip_subdivision_city_name AggregateFunction(argMin, Nullable(String), DateTime64(6, 'UTC')),
+ geoip_time_zone AggregateFunction(argMin, Nullable(String), DateTime64(6, 'UTC')),
+ entry_referring_domain AggregateFunction(argMin, Nullable(String), DateTime64(6, 'UTC')),
+ entry_utm_source AggregateFunction(argMin, Nullable(String), DateTime64(6, 'UTC')),
+ entry_utm_campaign AggregateFunction(argMin, Nullable(String), DateTime64(6, 'UTC')),
+ entry_utm_medium AggregateFunction(argMin, Nullable(String), DateTime64(6, 'UTC')),
+ entry_utm_term AggregateFunction(argMin, Nullable(String), DateTime64(6, 'UTC')),
+ entry_utm_content AggregateFunction(argMin, Nullable(String), DateTime64(6, 'UTC')),
+ entry_gclid AggregateFunction(argMin, Nullable(String), DateTime64(6, 'UTC')),
+ entry_gad_source AggregateFunction(argMin, Nullable(String), DateTime64(6, 'UTC')),
+ entry_fbclid AggregateFunction(argMin, Nullable(String), DateTime64(6, 'UTC')),
+ entry_has_gclid AggregateFunction(argMin, Bool, DateTime64(6, 'UTC')),
+ entry_has_fbclid AggregateFunction(argMin, Bool, DateTime64(6, 'UTC')),
+ entry_ad_ids_set AggregateFunction(argMin, Array(String), DateTime64(6, 'UTC')),
+ entry_channel_type_properties AggregateFunction(argMin, Tuple(Nullable(String), Nullable(String), Nullable(String), Nullable(String), Bool, Bool, Nullable(String)), DateTime64(6, 'UTC')),
+ pageview_uniq AggregateFunction(uniq, Nullable(UUID)),
+ autocapture_uniq AggregateFunction(uniq, Nullable(UUID)),
+ screen_uniq AggregateFunction(uniq, Nullable(UUID)),
+ page_screen_uniq_up_to AggregateFunction(uniqUpTo(1), Nullable(UUID)),
+ has_autocapture SimpleAggregateFunction(max, Bool),
+ flag_key_values SimpleAggregateFunction(groupUniqArrayArray(10000), Array(String)),
+ event_names SimpleAggregateFunction(groupUniqArrayArray, Array(String)),
+ hosts SimpleAggregateFunction(groupUniqArrayArray(100), Array(String)),
+ emails SimpleAggregateFunction(groupUniqArrayArray(10), Array(String)),
+ has_replay_events SimpleAggregateFunction(max, Bool)
+) ENGINE = Distributed('sessions', 'posthog', 'raw_sessions_v3', cityHash64(session_id_v7));
CREATE TABLE posthog.writable_session_recording_events (
uuid UUID,
timestamp DateTime64(6, 'UTC'),
@@ -2843,6 +2937,57 @@ CREATE VIEW posthog.raw_sessions_v AS SELECT
FROM posthog.raw_sessions
GROUP BY
session_id_v7, team_id;
+CREATE VIEW posthog.raw_sessions_v3_v AS SELECT
+ session_id_v7,
+ session_timestamp,
+ team_id,
+ argMaxMerge(distinct_id) AS distinct_id,
+ groupUniqArrayMerge(distinct_ids) AS distinct_ids,
+ min(min_timestamp) AS min_timestamp,
+ max(max_timestamp) AS max_timestamp,
+ max(max_inserted_at) AS max_inserted_at,
+ groupUniqArrayArray(2000)(urls) AS urls,
+ argMinMerge(entry_url) AS entry_url,
+ argMaxMerge(end_url) AS end_url,
+ argMaxMerge(last_external_click_url) AS last_external_click_url,
+ argMinMerge(browser) AS browser,
+ argMinMerge(browser_version) AS browser_version,
+ argMinMerge(os) AS os,
+ argMinMerge(os_version) AS os_version,
+ argMinMerge(device_type) AS device_type,
+ argMinMerge(viewport_width) AS viewport_width,
+ argMinMerge(viewport_height) AS viewport_height,
+ argMinMerge(geoip_country_code) AS geoip_country_code,
+ argMinMerge(geoip_subdivision_1_code) AS geoip_subdivision_1_code,
+ argMinMerge(geoip_subdivision_1_name) AS geoip_subdivision_1_name,
+ argMinMerge(geoip_subdivision_city_name) AS geoip_subdivision_city_name,
+ argMinMerge(geoip_time_zone) AS geoip_time_zone,
+ argMinMerge(entry_utm_source) AS entry_utm_source,
+ argMinMerge(entry_utm_campaign) AS entry_utm_campaign,
+ argMinMerge(entry_utm_medium) AS entry_utm_medium,
+ argMinMerge(entry_utm_term) AS entry_utm_term,
+ argMinMerge(entry_utm_content) AS entry_utm_content,
+ argMinMerge(entry_referring_domain) AS entry_referring_domain,
+ argMinMerge(entry_gclid) AS entry_gclid,
+ argMinMerge(entry_gad_source) AS entry_gad_source,
+ argMinMerge(entry_fbclid) AS entry_fbclid,
+ argMinMerge(entry_has_gclid) AS entry_has_gclid,
+ argMinMerge(entry_has_fbclid) AS entry_has_fbclid,
+ argMinMerge(entry_ad_ids_set) AS entry_ad_ids_set,
+ argMinMerge(entry_channel_type_properties) AS entry_channel_type_properties,
+ uniqMerge(pageview_uniq) AS pageview_uniq,
+ uniqMerge(autocapture_uniq) AS autocapture_uniq,
+ uniqMerge(screen_uniq) AS screen_uniq,
+ uniqUpToMerge(1)(page_screen_uniq_up_to) AS page_screen_uniq_up_to,
+ max(has_autocapture) AS has_autocapture,
+ groupUniqArrayArray(10000)(flag_key_values) AS flag_key_values,
+ groupUniqArrayArray(2000)(event_names) AS event_names,
+ groupUniqArrayArray(100)(hosts) AS hosts,
+ groupUniqArrayArray(10)(emails) AS emails,
+ max(has_replay_events) AS has_replay_events
+FROM posthog.raw_sessions_v3
+GROUP BY
+ session_id_v7, session_timestamp, team_id;
CREATE VIEW posthog.sessions_v AS SELECT
session_id,
team_id,
@@ -3175,58 +3320,6 @@ CREATE TABLE posthog.preaggregation_results (
breakdown_value Array(String),
uniq_exact_state AggregateFunction(uniqExact, UUID)
) ENGINE = Distributed('posthog', 'posthog', 'sharded_preaggregation_results', sipHash64(job_id));
-CREATE TABLE posthog.raw_sessions_v3 (
- team_id Int64,
- session_id_v7 UInt128,
- session_timestamp DateTime64(3) MATERIALIZED fromUnixTimestamp64Milli(toUInt64(bitShiftRight(session_id_v7, 80))),
- distinct_id AggregateFunction(argMax, String, DateTime64(6, 'UTC')),
- distinct_ids AggregateFunction(groupUniqArray, String),
- min_timestamp SimpleAggregateFunction(min, DateTime64(6, 'UTC')),
- max_timestamp SimpleAggregateFunction(max, DateTime64(6, 'UTC')),
- max_inserted_at SimpleAggregateFunction(max, DateTime64(6, 'UTC')),
- urls SimpleAggregateFunction(groupUniqArrayArray(2000), Array(String)),
- entry_url AggregateFunction(argMin, Nullable(String), DateTime64(6, 'UTC')),
- end_url AggregateFunction(argMax, Nullable(String), DateTime64(6, 'UTC')),
- last_external_click_url AggregateFunction(argMax, Nullable(String), DateTime64(6, 'UTC')),
- browser AggregateFunction(argMin, Nullable(String), DateTime64(6, 'UTC')),
- browser_version AggregateFunction(argMin, Nullable(String), DateTime64(6, 'UTC')),
- os AggregateFunction(argMin, Nullable(String), DateTime64(6, 'UTC')),
- os_version AggregateFunction(argMin, Nullable(String), DateTime64(6, 'UTC')),
- device_type AggregateFunction(argMin, Nullable(String), DateTime64(6, 'UTC')),
- viewport_width AggregateFunction(argMin, Nullable(Int64), DateTime64(6, 'UTC')),
- viewport_height AggregateFunction(argMin, Nullable(Int64), DateTime64(6, 'UTC')),
- geoip_country_code AggregateFunction(argMin, Nullable(String), DateTime64(6, 'UTC')),
- geoip_subdivision_1_code AggregateFunction(argMin, Nullable(String), DateTime64(6, 'UTC')),
- geoip_subdivision_1_name AggregateFunction(argMin, Nullable(String), DateTime64(6, 'UTC')),
- geoip_subdivision_city_name AggregateFunction(argMin, Nullable(String), DateTime64(6, 'UTC')),
- geoip_time_zone AggregateFunction(argMin, Nullable(String), DateTime64(6, 'UTC')),
- entry_referring_domain AggregateFunction(argMin, Nullable(String), DateTime64(6, 'UTC')),
- entry_utm_source AggregateFunction(argMin, Nullable(String), DateTime64(6, 'UTC')),
- entry_utm_campaign AggregateFunction(argMin, Nullable(String), DateTime64(6, 'UTC')),
- entry_utm_medium AggregateFunction(argMin, Nullable(String), DateTime64(6, 'UTC')),
- entry_utm_term AggregateFunction(argMin, Nullable(String), DateTime64(6, 'UTC')),
- entry_utm_content AggregateFunction(argMin, Nullable(String), DateTime64(6, 'UTC')),
- entry_gclid AggregateFunction(argMin, Nullable(String), DateTime64(6, 'UTC')),
- entry_gad_source AggregateFunction(argMin, Nullable(String), DateTime64(6, 'UTC')),
- entry_fbclid AggregateFunction(argMin, Nullable(String), DateTime64(6, 'UTC')),
- entry_has_gclid AggregateFunction(argMin, Bool, DateTime64(6, 'UTC')),
- entry_has_fbclid AggregateFunction(argMin, Bool, DateTime64(6, 'UTC')),
- entry_ad_ids_map AggregateFunction(argMin, Map(String, String), DateTime64(6, 'UTC')),
- entry_ad_ids_set AggregateFunction(argMin, Array(String), DateTime64(6, 'UTC')),
- entry_channel_type_properties AggregateFunction(argMin, Tuple(Nullable(String), Nullable(String), Nullable(String), Nullable(String), Bool, Bool, Nullable(String)), DateTime64(6, 'UTC')),
- pageview_uniq AggregateFunction(uniqExact, Nullable(UUID)),
- autocapture_uniq AggregateFunction(uniqExact, Nullable(UUID)),
- screen_uniq AggregateFunction(uniqExact, Nullable(UUID)),
- page_screen_uniq_up_to AggregateFunction(uniqUpTo(1), Nullable(UUID)),
- has_autocapture SimpleAggregateFunction(max, Bool),
- flag_key_values SimpleAggregateFunction(groupUniqArrayArray(10000), Array(String)),
- flag_values AggregateFunction(groupUniqArrayMap, Map(String, String)),
- flag_keys SimpleAggregateFunction(groupUniqArrayArray, Array(String)),
- event_names SimpleAggregateFunction(groupUniqArrayArray, Array(String)),
- hosts SimpleAggregateFunction(groupUniqArrayArray(100), Array(String)),
- emails SimpleAggregateFunction(groupUniqArrayArray(10), Array(String)),
- has_replay_events SimpleAggregateFunction(max, Bool)
-) ENGINE = Distributed('posthog', 'posthog', 'sharded_raw_sessions_v3', cityHash64(session_id_v7));
CREATE TABLE posthog.session_replay_embeddings (
session_id String,
team_id Int64,
@@ -3267,58 +3360,6 @@ CREATE TABLE posthog.session_replay_events (
surfacing_score SimpleAggregateFunction(max, Nullable(Float32)),
snapshot_mode_v2 AggregateFunction(argMin, Nullable(String), DateTime64(6, 'UTC'))
) ENGINE = Distributed('posthog', 'posthog', 'sharded_session_replay_events', sipHash64(distinct_id));
-CREATE TABLE posthog.writable_raw_sessions_v3 (
- team_id Int64,
- session_id_v7 UInt128,
- session_timestamp DateTime64(3) MATERIALIZED fromUnixTimestamp64Milli(toUInt64(bitShiftRight(session_id_v7, 80))),
- distinct_id AggregateFunction(argMax, String, DateTime64(6, 'UTC')),
- distinct_ids AggregateFunction(groupUniqArray, String),
- min_timestamp SimpleAggregateFunction(min, DateTime64(6, 'UTC')),
- max_timestamp SimpleAggregateFunction(max, DateTime64(6, 'UTC')),
- max_inserted_at SimpleAggregateFunction(max, DateTime64(6, 'UTC')),
- urls SimpleAggregateFunction(groupUniqArrayArray(2000), Array(String)),
- entry_url AggregateFunction(argMin, Nullable(String), DateTime64(6, 'UTC')),
- end_url AggregateFunction(argMax, Nullable(String), DateTime64(6, 'UTC')),
- last_external_click_url AggregateFunction(argMax, Nullable(String), DateTime64(6, 'UTC')),
- browser AggregateFunction(argMin, Nullable(String), DateTime64(6, 'UTC')),
- browser_version AggregateFunction(argMin, Nullable(String), DateTime64(6, 'UTC')),
- os AggregateFunction(argMin, Nullable(String), DateTime64(6, 'UTC')),
- os_version AggregateFunction(argMin, Nullable(String), DateTime64(6, 'UTC')),
- device_type AggregateFunction(argMin, Nullable(String), DateTime64(6, 'UTC')),
- viewport_width AggregateFunction(argMin, Nullable(Int64), DateTime64(6, 'UTC')),
- viewport_height AggregateFunction(argMin, Nullable(Int64), DateTime64(6, 'UTC')),
- geoip_country_code AggregateFunction(argMin, Nullable(String), DateTime64(6, 'UTC')),
- geoip_subdivision_1_code AggregateFunction(argMin, Nullable(String), DateTime64(6, 'UTC')),
- geoip_subdivision_1_name AggregateFunction(argMin, Nullable(String), DateTime64(6, 'UTC')),
- geoip_subdivision_city_name AggregateFunction(argMin, Nullable(String), DateTime64(6, 'UTC')),
- geoip_time_zone AggregateFunction(argMin, Nullable(String), DateTime64(6, 'UTC')),
- entry_referring_domain AggregateFunction(argMin, Nullable(String), DateTime64(6, 'UTC')),
- entry_utm_source AggregateFunction(argMin, Nullable(String), DateTime64(6, 'UTC')),
- entry_utm_campaign AggregateFunction(argMin, Nullable(String), DateTime64(6, 'UTC')),
- entry_utm_medium AggregateFunction(argMin, Nullable(String), DateTime64(6, 'UTC')),
- entry_utm_term AggregateFunction(argMin, Nullable(String), DateTime64(6, 'UTC')),
- entry_utm_content AggregateFunction(argMin, Nullable(String), DateTime64(6, 'UTC')),
- entry_gclid AggregateFunction(argMin, Nullable(String), DateTime64(6, 'UTC')),
- entry_gad_source AggregateFunction(argMin, Nullable(String), DateTime64(6, 'UTC')),
- entry_fbclid AggregateFunction(argMin, Nullable(String), DateTime64(6, 'UTC')),
- entry_has_gclid AggregateFunction(argMin, Bool, DateTime64(6, 'UTC')),
- entry_has_fbclid AggregateFunction(argMin, Bool, DateTime64(6, 'UTC')),
- entry_ad_ids_map AggregateFunction(argMin, Map(String, String), DateTime64(6, 'UTC')),
- entry_ad_ids_set AggregateFunction(argMin, Array(String), DateTime64(6, 'UTC')),
- entry_channel_type_properties AggregateFunction(argMin, Tuple(Nullable(String), Nullable(String), Nullable(String), Nullable(String), Bool, Bool, Nullable(String)), DateTime64(6, 'UTC')),
- pageview_uniq AggregateFunction(uniqExact, Nullable(UUID)),
- autocapture_uniq AggregateFunction(uniqExact, Nullable(UUID)),
- screen_uniq AggregateFunction(uniqExact, Nullable(UUID)),
- page_screen_uniq_up_to AggregateFunction(uniqUpTo(1), Nullable(UUID)),
- has_autocapture SimpleAggregateFunction(max, Bool),
- flag_key_values SimpleAggregateFunction(groupUniqArrayArray(10000), Array(String)),
- flag_values AggregateFunction(groupUniqArrayMap, Map(String, String)),
- flag_keys SimpleAggregateFunction(groupUniqArrayArray, Array(String)),
- event_names SimpleAggregateFunction(groupUniqArrayArray, Array(String)),
- hosts SimpleAggregateFunction(groupUniqArrayArray(100), Array(String)),
- emails SimpleAggregateFunction(groupUniqArrayArray(10), Array(String)),
- has_replay_events SimpleAggregateFunction(max, Bool)
-) ENGINE = Distributed('sessions', 'posthog', 'raw_sessions_v3', cityHash64(session_id_v7));
CREATE VIEW posthog.custom_metrics AS SELECT * REPLACE(toFloat64(value) AS value)
FROM posthog.custom_metrics_test
UNION ALL
@@ -3494,59 +3535,6 @@ SELECT
_partition,
'text-embedding-3-large-3072' AS model_name
FROM posthog.distributed_posthog_document_embeddings_text_embedding_3_large_3072;
-CREATE VIEW posthog.raw_sessions_v3_v AS SELECT
- session_id_v7,
- session_timestamp,
- team_id,
- argMaxMerge(distinct_id) AS distinct_id,
- groupUniqArrayMerge(distinct_ids) AS distinct_ids,
- min(min_timestamp) AS min_timestamp,
- max(max_timestamp) AS max_timestamp,
- max(max_inserted_at) AS max_inserted_at,
- groupUniqArrayArray(2000)(urls) AS urls,
- argMinMerge(entry_url) AS entry_url,
- argMaxMerge(end_url) AS end_url,
- argMaxMerge(last_external_click_url) AS last_external_click_url,
- argMinMerge(browser) AS browser,
- argMinMerge(browser_version) AS browser_version,
- argMinMerge(os) AS os,
- argMinMerge(os_version) AS os_version,
- argMinMerge(device_type) AS device_type,
- argMinMerge(viewport_width) AS viewport_width,
- argMinMerge(viewport_height) AS viewport_height,
- argMinMerge(geoip_country_code) AS geoip_country_code,
- argMinMerge(geoip_subdivision_1_code) AS geoip_subdivision_1_code,
- argMinMerge(geoip_subdivision_1_name) AS geoip_subdivision_1_name,
- argMinMerge(geoip_subdivision_city_name) AS geoip_subdivision_city_name,
- argMinMerge(geoip_time_zone) AS geoip_time_zone,
- argMinMerge(entry_utm_source) AS entry_utm_source,
- argMinMerge(entry_utm_campaign) AS entry_utm_campaign,
- argMinMerge(entry_utm_medium) AS entry_utm_medium,
- argMinMerge(entry_utm_term) AS entry_utm_term,
- argMinMerge(entry_utm_content) AS entry_utm_content,
- argMinMerge(entry_referring_domain) AS entry_referring_domain,
- argMinMerge(entry_gclid) AS entry_gclid,
- argMinMerge(entry_gad_source) AS entry_gad_source,
- argMinMerge(entry_fbclid) AS entry_fbclid,
- argMinMerge(entry_has_gclid) AS entry_has_gclid,
- argMinMerge(entry_has_fbclid) AS entry_has_fbclid,
- argMinMerge(entry_ad_ids_map) AS entry_ad_ids_map,
- argMinMerge(entry_ad_ids_set) AS entry_ad_ids_set,
- argMinMerge(entry_channel_type_properties) AS entry_channel_type_properties,
- uniqExactMerge(pageview_uniq) AS pageview_uniq,
- uniqExactMerge(autocapture_uniq) AS autocapture_uniq,
- uniqExactMerge(screen_uniq) AS screen_uniq,
- uniqUpToMerge(1)(page_screen_uniq_up_to) AS page_screen_uniq_up_to,
- max(has_autocapture) AS has_autocapture,
- groupUniqArrayArray(10000)(flag_key_values) AS flag_key_values,
- groupUniqArrayArray(flag_keys) AS flag_keys,
- groupUniqArrayArray(2000)(event_names) AS event_names,
- groupUniqArrayArray(100)(hosts) AS hosts,
- groupUniqArrayArray(10)(emails) AS emails,
- max(has_replay_events) AS has_replay_events
-FROM posthog.raw_sessions_v3
-GROUP BY
- session_id_v7, session_timestamp, team_id;
CREATE VIEW posthog.team_events_last_month_view AS SELECT team_id, count() AS event_count
FROM posthog.events
WHERE
diff --git a/posthog/clickhouse/hcl/sql/prod-us/sessions.sql b/posthog/clickhouse/hcl/sql/prod-us/sessions.sql
index 59c8f4370c4a..73452c2a55bf 100644
--- a/posthog/clickhouse/hcl/sql/prod-us/sessions.sql
+++ b/posthog/clickhouse/hcl/sql/prod-us/sessions.sql
@@ -320,24 +320,20 @@ CREATE TABLE posthog.raw_sessions_v3 (
entry_fbclid AggregateFunction(argMin, Nullable(String), DateTime64(6, 'UTC')),
entry_has_gclid AggregateFunction(argMin, Bool, DateTime64(6, 'UTC')),
entry_has_fbclid AggregateFunction(argMin, Bool, DateTime64(6, 'UTC')),
- entry_ad_ids_map AggregateFunction(argMin, Map(String, String), DateTime64(6, 'UTC')),
entry_ad_ids_set AggregateFunction(argMin, Array(String), DateTime64(6, 'UTC')),
entry_channel_type_properties AggregateFunction(argMin, Tuple(Nullable(String), Nullable(String), Nullable(String), Nullable(String), Bool, Bool, Nullable(String)), DateTime64(6, 'UTC')),
- pageview_uniq AggregateFunction(uniqExact, Nullable(UUID)),
- autocapture_uniq AggregateFunction(uniqExact, Nullable(UUID)),
- screen_uniq AggregateFunction(uniqExact, Nullable(UUID)),
+ pageview_uniq AggregateFunction(uniq, Nullable(UUID)),
+ autocapture_uniq AggregateFunction(uniq, Nullable(UUID)),
+ screen_uniq AggregateFunction(uniq, Nullable(UUID)),
page_screen_uniq_up_to AggregateFunction(uniqUpTo(1), Nullable(UUID)),
has_autocapture SimpleAggregateFunction(max, Bool),
flag_key_values SimpleAggregateFunction(groupUniqArrayArray(10000), Array(String)),
- flag_values AggregateFunction(groupUniqArrayMap, Map(String, String)),
- flag_keys SimpleAggregateFunction(groupUniqArrayArray, Array(String)),
event_names SimpleAggregateFunction(groupUniqArrayArray, Array(String)),
hosts SimpleAggregateFunction(groupUniqArrayArray(100), Array(String)),
emails SimpleAggregateFunction(groupUniqArrayArray(10), Array(String)),
has_replay_events SimpleAggregateFunction(max, Bool),
INDEX event_names_bloom_filter event_names TYPE bloom_filter() GRANULARITY 1,
- INDEX flag_keys_bloom_filter flag_keys TYPE bloom_filter() GRANULARITY 1,
- INDEX flag_key_values_bloom_filter flag_key_values TYPE bloom_filter() GRANULARITY 1
+ INDEX flag_key_values_text flag_key_values TYPE text(tokenizer = splitByString(['='])) GRANULARITY 100000000
) ENGINE = ReplicatedAggregatingMergeTree('/clickhouse/tables/sessions/noshard/posthog.raw_sessions_v3', '{shard}-{replica}') ORDER BY (team_id, session_timestamp, session_id_v7) PARTITION BY toYYYYMM(session_timestamp) SETTINGS index_granularity = 8192, max_replicated_merges_in_queue = 5, parts_to_delay_insert = 1500, parts_to_throw_insert = 2000, prefer_fetch_merged_part_size_threshold = 1, prefer_fetch_merged_part_time_threshold = 0, replicated_deduplication_window = 0, replicated_deduplication_window_for_async_inserts = 0, storage_policy = 's3_tiered';
CREATE TABLE posthog.sessions (
session_id String,
diff --git a/posthog/clickhouse/migrations/0325_sessions_v3_flag_key_values.py b/posthog/clickhouse/migrations/0325_sessions_v3_flag_key_values.py
index 3a1244654224..ef6be6907211 100644
--- a/posthog/clickhouse/migrations/0325_sessions_v3_flag_key_values.py
+++ b/posthog/clickhouse/migrations/0325_sessions_v3_flag_key_values.py
@@ -10,7 +10,6 @@ from posthog.models.raw_sessions.migrations_v3 import (
)
from posthog.models.raw_sessions.sessions_v3 import (
DISTRIBUTED_RAW_SESSIONS_TABLE_V3,
- RAW_SESSIONS_CREATE_OR_REPLACE_VIEW_SQL_V3,
SHARDED_RAW_SESSIONS_TABLE_V3,
TABLE_BASE_NAME_V3,
WRITABLE_RAW_SESSIONS_TABLE_V3,
@@ -42,8 +41,7 @@ operations = [
sharded=True,
is_alter_on_replicated_table=True,
),
- # hosts and emails were only ever added by hand, so the main-cluster copies never got
- # them; the recreated view below selects both, so the columns must exist first.
+ # hosts and emails were only ever added by hand, so the main-cluster copies never got them.
run_sql_with_exceptions(
ADD_HOSTS.format(table_name=SHARDED_RAW_SESSIONS_TABLE_V3()),
node_roles=[NodeRole.DATA],
@@ -92,10 +90,6 @@ operations = [
sharded=True,
is_alter_on_replicated_table=True,
),
- run_sql_with_exceptions(
- RAW_SESSIONS_CREATE_OR_REPLACE_VIEW_SQL_V3(),
- node_roles=[NodeRole.DATA],
- ),
]
# In cloud envs the live copy is the single-shard raw_sessions_v3 MergeTree on the sessions
diff --git a/posthog/clickhouse/migrations/0364_sessions_v3_more_ad_ids.py b/posthog/clickhouse/migrations/0364_sessions_v3_more_ad_ids.py
new file mode 100644
index 000000000000..893fe51b2995
--- /dev/null
+++ b/posthog/clickhouse/migrations/0364_sessions_v3_more_ad_ids.py
@@ -0,0 +1,77 @@
+from posthog.clickhouse.client.connection import NodeRole
+from posthog.clickhouse.client.migration_tools import run_sql_with_exceptions
+from posthog.models.raw_sessions.migrations_v3 import (
+ DROP_FLAG_KEY_VALUES_BLOOM_FILTER,
+ DROP_FLAG_KEYS,
+ DROP_FLAG_KEYS_BLOOM_FILTER,
+)
+from posthog.models.raw_sessions.sessions_v3 import (
+ DISTRIBUTED_RAW_SESSIONS_TABLE_V3,
+ RAW_SESSIONS_TABLE_MV_RECORDINGS_SQL_V3,
+ RAW_SESSIONS_TABLE_MV_SQL_V3,
+ SHARDED_RAW_SESSIONS_TABLE_V3,
+ WRITABLE_RAW_SESSIONS_TABLE_V3,
+)
+from posthog.run_mode import run_mode
+
+# raw_sessions_v3 has a new shape: no entry_ad_ids_map or flag_keys, the ad-ids set holds
+# 'key=value' pairs, and the counts are uniq states instead of uniqExact. A shape change cannot be
+# applied with ALTER, so an existing v3 table has to be rebuilt with a backfill.
+#
+# Cloud is applied by hand by the ClickHouse team, because the rebuild needs a backfill and
+# dropping the ingestion MV stops v3 ingestion. As of 2026-10-28, prod-us is done; dev and EU are
+# still to do. The declared cloud schema is the HCL in posthog/clickhouse/hcl/.
+#
+# Everywhere else (fresh installs, CI, the test database) this migration is required: the earlier
+# v3 migrations render the current table and view definitions, but 0178 dropped the MVs, and this
+# is the only migration that creates them again. Both statements are CREATE ... IF NOT EXISTS, so
+# an existing MV is kept. An install that created v3 before the shape change keeps the old shape.
+# To use sessionTableVersion v3 there, drop raw_sessions_v3_mv, raw_sessions_v3_recordings_mv,
+# raw_sessions_v3_v, raw_sessions_v3, writable_raw_sessions_v3 and sharded_raw_sessions_v3, then
+# create them again from the definitions in posthog/models/raw_sessions/sessions_v3.py.
+#
+# The drops come first. 0180 and 0325 replay on a fresh cluster and put flag_keys and the two
+# bloom filter indexes back on top of today's base SQL, which no longer declares them, so without
+# these statements a fresh install ends up with a table the declared HCL does not describe.
+DROP_SUPERSEDED_FLAG_COLUMNS = [
+ run_sql_with_exceptions(
+ DROP_FLAG_KEYS_BLOOM_FILTER.format(table_name=SHARDED_RAW_SESSIONS_TABLE_V3()),
+ node_roles=[NodeRole.DATA],
+ sharded=True,
+ is_alter_on_replicated_table=True,
+ ),
+ run_sql_with_exceptions(
+ DROP_FLAG_KEY_VALUES_BLOOM_FILTER.format(table_name=SHARDED_RAW_SESSIONS_TABLE_V3()),
+ node_roles=[NodeRole.DATA],
+ sharded=True,
+ is_alter_on_replicated_table=True,
+ ),
+ run_sql_with_exceptions(
+ DROP_FLAG_KEYS.format(table_name=SHARDED_RAW_SESSIONS_TABLE_V3()),
+ node_roles=[NodeRole.DATA],
+ sharded=True,
+ is_alter_on_replicated_table=True,
+ ),
+ run_sql_with_exceptions(
+ DROP_FLAG_KEYS.format(table_name=WRITABLE_RAW_SESSIONS_TABLE_V3()),
+ node_roles=[NodeRole.DATA],
+ sharded=False,
+ is_alter_on_replicated_table=False,
+ ),
+ run_sql_with_exceptions(
+ DROP_FLAG_KEYS.format(table_name=DISTRIBUTED_RAW_SESSIONS_TABLE_V3()),
+ node_roles=[NodeRole.DATA],
+ sharded=False,
+ is_alter_on_replicated_table=False,
+ ),
+]
+
+operations = (
+ []
+ if run_mode().is_deployed_cloud
+ else [
+ *DROP_SUPERSEDED_FLAG_COLUMNS,
+ run_sql_with_exceptions(RAW_SESSIONS_TABLE_MV_SQL_V3(), node_roles=[NodeRole.DATA]),
+ run_sql_with_exceptions(RAW_SESSIONS_TABLE_MV_RECORDINGS_SQL_V3(), node_roles=[NodeRole.DATA]),
+ ]
+)
diff --git a/posthog/clickhouse/migrations/max_migration.txt b/posthog/clickhouse/migrations/max_migration.txt
index 452e42cade07..7432923099bf 100644
--- a/posthog/clickhouse/migrations/max_migration.txt
+++ b/posthog/clickhouse/migrations/max_migration.txt
@@ -1 +1 @@
-0363_email_status
+0364_sessions_v3_more_ad_ids
diff --git a/posthog/clickhouse/test/__snapshots__/test_raw_sessions_v3_model.ambr b/posthog/clickhouse/test/__snapshots__/test_raw_sessions_v3_model.ambr
index 18be1c646860..c8c3c2419ed1 100644
--- a/posthog/clickhouse/test/__snapshots__/test_raw_sessions_v3_model.ambr
+++ b/posthog/clickhouse/test/__snapshots__/test_raw_sessions_v3_model.ambr
@@ -1,5 +1,5 @@
# serializer version: 1
-# name: TestRawSessionsModel.test_ad_ids_map_and_set
+# name: TestRawSessionsModel.test_ad_id_property_vs_url_param_precedence_0_property_wins
'''
select *
@@ -8,7 +8,88 @@
AND team_id = 99999
'''
# ---
-# name: TestRawSessionsModel.test_autocapture_does_not_set_attribution_when_pageview_present
+# name: TestRawSessionsModel.test_ad_id_property_vs_url_param_precedence_1_empty_property_falls_back
+ '''
+
+ select *
+ from raw_sessions_v3_v
+ where session_id_v7 = toUInt128(toUUID('00000000-0000-0000-0000-000000000000'))
+ AND team_id = 99999
+ '''
+# ---
+# name: TestRawSessionsModel.test_ad_id_url_param_decodes_plus_as_space
+ '''
+
+ select *
+ from raw_sessions_v3_v
+ where session_id_v7 = toUInt128(toUUID('00000000-0000-0000-0000-000000000000'))
+ AND team_id = 99999
+ '''
+# ---
+# name: TestRawSessionsModel.test_ad_id_url_param_with_undecodable_bytes_is_normalized
+ '''
+
+ select *
+ from raw_sessions_v3_v
+ where session_id_v7 = toUInt128(toUUID('00000000-0000-0000-0000-000000000000'))
+ AND team_id = 99999
+ '''
+# ---
+# name: TestRawSessionsModel.test_ad_ids_fall_back_to_url_params_0_gad_campaignid
+ '''
+
+ select *
+ from raw_sessions_v3_v
+ where session_id_v7 = toUInt128(toUUID('00000000-0000-0000-0000-000000000000'))
+ AND team_id = 99999
+ '''
+# ---
+# name: TestRawSessionsModel.test_ad_ids_fall_back_to_url_params_1_utm_id
+ '''
+
+ select *
+ from raw_sessions_v3_v
+ where session_id_v7 = toUInt128(toUUID('00000000-0000-0000-0000-000000000000'))
+ AND team_id = 99999
+ '''
+# ---
+# name: TestRawSessionsModel.test_ad_ids_fall_back_to_url_params_2_irclickid
+ '''
+
+ select *
+ from raw_sessions_v3_v
+ where session_id_v7 = toUInt128(toUUID('00000000-0000-0000-0000-000000000000'))
+ AND team_id = 99999
+ '''
+# ---
+# name: TestRawSessionsModel.test_ad_ids_fall_back_to_url_params_3_snapchat_canonical
+ '''
+
+ select *
+ from raw_sessions_v3_v
+ where session_id_v7 = toUInt128(toUUID('00000000-0000-0000-0000-000000000000'))
+ AND team_id = 99999
+ '''
+# ---
+# name: TestRawSessionsModel.test_ad_ids_fall_back_to_url_params_4_snapchat_sccid_cased
+ '''
+
+ select *
+ from raw_sessions_v3_v
+ where session_id_v7 = toUInt128(toUUID('00000000-0000-0000-0000-000000000000'))
+ AND team_id = 99999
+ '''
+# ---
+# name: TestRawSessionsModel.test_ad_ids_fall_back_to_url_params_5_snapchat_sscid_cased
+ '''
+
+ select *
+ from raw_sessions_v3_v
+ where session_id_v7 = toUInt128(toUUID('00000000-0000-0000-0000-000000000000'))
+ AND team_id = 99999
+ '''
+# ---
+# name: TestRawSessionsModel.test_ad_ids_set_collects_kv_pairs
'''
select *
@@ -26,7 +107,16 @@
AND team_id = 99999
'''
# ---
-# name: TestRawSessionsModel.test_channel_type_properties
+# name: TestRawSessionsModel.test_channel_type_properties_0_click_ids_present
+ '''
+
+ select *
+ from raw_sessions_v3_v
+ where session_id_v7 = toUInt128(toUUID('00000000-0000-0000-0000-000000000000'))
+ AND team_id = 99999
+ '''
+# ---
+# name: TestRawSessionsModel.test_channel_type_properties_1_empty_click_ids_are_not_present
'''
select *
@@ -44,7 +134,7 @@
AND team_id = 99999
'''
# ---
-# name: TestRawSessionsModel.test_emails_are_collected
+# name: TestRawSessionsModel.test_drops_events_far_from_the_session_id_timestamp_0_garbage_session_timestamp
'''
select *
@@ -53,7 +143,7 @@
AND team_id = 99999
'''
# ---
-# name: TestRawSessionsModel.test_empty_hosts_and_emails
+# name: TestRawSessionsModel.test_drops_events_far_from_the_session_id_timestamp_1_session_id_over_31_days_after_event
'''
select *
@@ -62,7 +152,25 @@
AND team_id = 99999
'''
# ---
-# name: TestRawSessionsModel.test_event_names_are_collected
+# name: TestRawSessionsModel.test_drops_events_far_from_the_session_id_timestamp_2_client_clock_days_ahead
+ '''
+
+ select *
+ from raw_sessions_v3_v
+ where session_id_v7 = toUInt128(toUUID('00000000-0000-0000-0000-000000000000'))
+ AND team_id = 99999
+ '''
+# ---
+# name: TestRawSessionsModel.test_drops_events_far_from_the_session_id_timestamp_3_late_event_within_31_days
+ '''
+
+ select *
+ from raw_sessions_v3_v
+ where session_id_v7 = toUInt128(toUUID('00000000-0000-0000-0000-000000000000'))
+ AND team_id = 99999
+ '''
+# ---
+# name: TestRawSessionsModel.test_emails_are_collected
'''
select *
@@ -71,7 +179,16 @@
AND team_id = 99999
'''
# ---
-# name: TestRawSessionsModel.test_flag_keys_are_collected
+# name: TestRawSessionsModel.test_empty_hosts_and_emails
+ '''
+
+ select *
+ from raw_sessions_v3_v
+ where session_id_v7 = toUInt128(toUUID('00000000-0000-0000-0000-000000000000'))
+ AND team_id = 99999
+ '''
+# ---
+# name: TestRawSessionsModel.test_event_names_are_collected
'''
select *
@@ -184,6 +301,42 @@
AND team_id = 99999
'''
# ---
+# name: TestRawSessionsModel.test_non_pageview_event_does_not_set_attribution_when_pageview_present_0_autocapture_first
+ '''
+
+ select *
+ from raw_sessions_v3_v
+ where session_id_v7 = toUInt128(toUUID('00000000-0000-0000-0000-000000000000'))
+ AND team_id = 99999
+ '''
+# ---
+# name: TestRawSessionsModel.test_non_pageview_event_does_not_set_attribution_when_pageview_present_1_pageleave_first
+ '''
+
+ select *
+ from raw_sessions_v3_v
+ where session_id_v7 = toUInt128(toUUID('00000000-0000-0000-0000-000000000000'))
+ AND team_id = 99999
+ '''
+# ---
+# name: TestRawSessionsModel.test_non_pageview_event_does_not_set_attribution_when_pageview_present_2_custom_event_first
+ '''
+
+ select *
+ from raw_sessions_v3_v
+ where session_id_v7 = toUInt128(toUUID('00000000-0000-0000-0000-000000000000'))
+ AND team_id = 99999
+ '''
+# ---
+# name: TestRawSessionsModel.test_non_pageview_event_does_not_set_attribution_when_pageview_present_3_pageview_inserted_before_earlier_event
+ '''
+
+ select *
+ from raw_sessions_v3_v
+ where session_id_v7 = toUInt128(toUUID('00000000-0000-0000-0000-000000000000'))
+ AND team_id = 99999
+ '''
+# ---
# name: TestRawSessionsModel.test_select_from_sessions
'''
diff --git a/posthog/clickhouse/test/__snapshots__/test_schema.ambr b/posthog/clickhouse/test/__snapshots__/test_schema.ambr
index e27b3b78382e..7c927ce9fc14 100644
--- a/posthog/clickhouse/test/__snapshots__/test_schema.ambr
+++ b/posthog/clickhouse/test/__snapshots__/test_schema.ambr
@@ -6954,8 +6954,8 @@
entry_has_gclid AggregateFunction(argMin, Boolean, DateTime64(6, 'UTC')),
entry_has_fbclid AggregateFunction(argMin, Boolean, DateTime64(6, 'UTC')),
- -- for lower-tier ad ids, just put them in a map, and set of the ones present
- entry_ad_ids_map AggregateFunction(argMin, Map(String, String), DateTime64(6, 'UTC')),
+ -- lower-tier ad ids as a flat 'key=value' array - like flag_key_values, a per-key map
+ -- state made merges much more expensive
entry_ad_ids_set AggregateFunction(argMin, Array(String), DateTime64(6, 'UTC')),
-- channel type properties tuple - to reduce redundant reading of the timestamp when loading all of these columns
@@ -6963,12 +6963,13 @@
entry_channel_type_properties AggregateFunction(argMin, Tuple(Nullable(String), Nullable(String), Nullable(String), Nullable(String), Boolean, Boolean, Nullable(String)), DateTime64(6, 'UTC')),
-- Count pageview, autocapture, and screen events for providing totals.
- -- Use uniqExact instead of count, so that inserting events can be idempotent. This is necessary as sometimes we see
+ -- Use uniq instead of count, so that inserting events can be idempotent. This is necessary as sometimes we see
-- events being inserted multiple times to be deduped later, but that can trigger multiple rows here.
-- Additionally, idempotency is useful for backfilling, as we can just reinsert the same events without worrying.
- pageview_uniq AggregateFunction(uniqExact, Nullable(UUID)),
- autocapture_uniq AggregateFunction(uniqExact, Nullable(UUID)),
- screen_uniq AggregateFunction(uniqExact, Nullable(UUID)),
+ -- uniq (not uniqExact) keeps the state small; per-session counts are tiny so it is exact in practice.
+ pageview_uniq AggregateFunction(uniq, Nullable(UUID)),
+ autocapture_uniq AggregateFunction(uniq, Nullable(UUID)),
+ screen_uniq AggregateFunction(uniq, Nullable(UUID)),
-- As a performance optimisation, also keep track of the uniq events for all of these combined.
-- This is a much more efficient way of calculating the bounce rate, as >2 means not a bounce
@@ -6977,8 +6978,8 @@
-- Flags - store every seen 'key=value' pair per flag. A flat array instead of a
-- groupUniqArrayMap state because per-key sub-aggregators made merges ~8-37x more expensive.
+ -- Key-only searches use the text index below (tokenized on '='), so no separate keys column.
flag_key_values SimpleAggregateFunction(groupUniqArrayArray(10000), Array(String)),
- flag_keys SimpleAggregateFunction(groupUniqArrayArray, Array(String)),
-- Event names - store unique event names seen in this session
event_names SimpleAggregateFunction(groupUniqArrayArray(2000), Array(String)),
@@ -7017,7 +7018,7 @@
`$geoip_country_code` Nullable(String),
`$geoip_subdivision_1_code` Nullable(String),
`$geoip_subdivision_1_name` Nullable(String),
- `$geoip_subdivision_city_name` Nullable(String),
+ `$geoip_city_name` Nullable(String),
`$geoip_time_zone` Nullable(String),
`$referring_domain` Nullable(String),
`utm_source` Nullable(String),
@@ -7033,20 +7034,40 @@
`dclid` Nullable(String),
`gbraid` Nullable(String),
`wbraid` Nullable(String),
+ `gad_campaignid` Nullable(String),
+ `srsltid` Nullable(String),
`msclkid` Nullable(String),
`twclid` Nullable(String),
`li_fat_id` Nullable(String),
`mc_cid` Nullable(String),
`igshid` Nullable(String),
`ttclid` Nullable(String),
+ `rdt_cid` Nullable(String),
`epik` Nullable(String),
`qclid` Nullable(String),
`sccid` Nullable(String),
`_kx` Nullable(String),
- `irclid` Nullable(String)
+ `irclid` Nullable(String),
+ `irclickid` Nullable(String),
+ `utm_id` Nullable(String),
+ `utm_source_platform` Nullable(String),
+ `utm_creative_format` Nullable(String),
+ `utm_marketing_tactic` Nullable(String),
+ `yclid` Nullable(String),
+ `tblci` Nullable(String),
+ `dicbo` Nullable(String),
+ `ndclid` Nullable(String),
+ `cjevent` Nullable(String),
+ `awc` Nullable(String),
+ `syclid` Nullable(String),
+ `_bhlid` Nullable(String),
+ `rtid` Nullable(String),
+ `oppref` Nullable(String),
+ `bfcid` Nullable(String)
)') as p,
JSONExtractString(person_properties, 'email') as _person_email,
tupleElement(p, '$current_url') as _current_url,
+ if(event = '$pageview' OR event = '$screen', left(coalesce(_current_url, ''), 16384), '') as _attribution_url,
tupleElement(p, '$external_click_url') as _external_click_url,
tupleElement(p, '$browser') as _browser,
tupleElement(p, '$browser_version') as _browser_version,
@@ -7058,7 +7079,7 @@
tupleElement(p, '$geoip_country_code') as _geoip_country_code,
tupleElement(p, '$geoip_subdivision_1_code') as _geoip_subdivision_1_code,
tupleElement(p, '$geoip_subdivision_1_name') as _geoip_subdivision_1_name,
- tupleElement(p, '$geoip_subdivision_city_name') as _geoip_subdivision_city_name,
+ tupleElement(p, '$geoip_city_name') as _geoip_subdivision_city_name,
tupleElement(p, '$geoip_time_zone') as _geoip_time_zone,
tupleElement(p, '$referring_domain') as _referring_domain,
tupleElement(p, 'utm_source') as _utm_source,
@@ -7069,54 +7090,75 @@
tupleElement(p, 'gclid') as _gclid,
tupleElement(p, 'gad_source') as _gad_source,
tupleElement(p, 'fbclid') as _fbclid,
- tupleElement(p, 'gclsrc') as gclsrc,
- tupleElement(p, 'dclid') as dclid,
- tupleElement(p, 'gbraid') as gbraid,
- tupleElement(p, 'wbraid') as wbraid,
- tupleElement(p, 'msclkid') as msclkid,
- tupleElement(p, 'twclid') as twclid,
- tupleElement(p, 'li_fat_id') as li_fat_id,
- tupleElement(p, 'mc_cid') as mc_cid,
- tupleElement(p, 'igshid') as igshid,
- tupleElement(p, 'ttclid') as ttclid,
- tupleElement(p, 'epik') as epik,
- tupleElement(p, 'qclid') as qclid,
- tupleElement(p, 'sccid') as sccid,
- tupleElement(p, '_kx') as _kx,
- tupleElement(p, 'irclid') as irclid,
- CAST(mapFilter((k, v) -> v IS NOT NULL, map(
- 'gclsrc', gclsrc,
- 'dclid', dclid,
- 'gbraid', gbraid,
- 'wbraid', wbraid,
- 'msclkid', msclkid,
- 'twclid', twclid,
- 'li_fat_id', li_fat_id,
- 'mc_cid', mc_cid,
- 'igshid', igshid,
- 'ttclid', ttclid,
- 'epik', epik,
- 'qclid', qclid,
- 'sccid', sccid,
- '_kx', _kx,
- 'irclid', irclid
- )) AS Map(String, String)) as ad_ids_map,
+ coalesce(nullIf(tupleElement(p, 'gclsrc'), ''), nullIf(toValidUTF8(left(decodeURLFormComponent(extractURLParameter(_attribution_url, 'gclsrc')), 2048)), '')) as gclsrc,
+ coalesce(nullIf(tupleElement(p, 'dclid'), ''), nullIf(toValidUTF8(left(decodeURLFormComponent(extractURLParameter(_attribution_url, 'dclid')), 2048)), '')) as dclid,
+ coalesce(nullIf(tupleElement(p, 'gbraid'), ''), nullIf(toValidUTF8(left(decodeURLFormComponent(extractURLParameter(_attribution_url, 'gbraid')), 2048)), '')) as gbraid,
+ coalesce(nullIf(tupleElement(p, 'wbraid'), ''), nullIf(toValidUTF8(left(decodeURLFormComponent(extractURLParameter(_attribution_url, 'wbraid')), 2048)), '')) as wbraid,
+ coalesce(nullIf(tupleElement(p, 'gad_campaignid'), ''), nullIf(toValidUTF8(left(decodeURLFormComponent(extractURLParameter(_attribution_url, 'gad_campaignid')), 2048)), '')) as gad_campaignid,
+ coalesce(nullIf(tupleElement(p, 'srsltid'), ''), nullIf(toValidUTF8(left(decodeURLFormComponent(extractURLParameter(_attribution_url, 'srsltid')), 2048)), '')) as srsltid,
+ coalesce(nullIf(tupleElement(p, 'msclkid'), ''), nullIf(toValidUTF8(left(decodeURLFormComponent(extractURLParameter(_attribution_url, 'msclkid')), 2048)), '')) as msclkid,
+ coalesce(nullIf(tupleElement(p, 'twclid'), ''), nullIf(toValidUTF8(left(decodeURLFormComponent(extractURLParameter(_attribution_url, 'twclid')), 2048)), '')) as twclid,
+ coalesce(nullIf(tupleElement(p, 'li_fat_id'), ''), nullIf(toValidUTF8(left(decodeURLFormComponent(extractURLParameter(_attribution_url, 'li_fat_id')), 2048)), '')) as li_fat_id,
+ coalesce(nullIf(tupleElement(p, 'mc_cid'), ''), nullIf(toValidUTF8(left(decodeURLFormComponent(extractURLParameter(_attribution_url, 'mc_cid')), 2048)), '')) as mc_cid,
+ coalesce(nullIf(tupleElement(p, 'igshid'), ''), nullIf(toValidUTF8(left(decodeURLFormComponent(extractURLParameter(_attribution_url, 'igshid')), 2048)), '')) as igshid,
+ coalesce(nullIf(tupleElement(p, 'ttclid'), ''), nullIf(toValidUTF8(left(decodeURLFormComponent(extractURLParameter(_attribution_url, 'ttclid')), 2048)), '')) as ttclid,
+ coalesce(nullIf(tupleElement(p, 'rdt_cid'), ''), nullIf(toValidUTF8(left(decodeURLFormComponent(extractURLParameter(_attribution_url, 'rdt_cid')), 2048)), '')) as rdt_cid,
+ coalesce(nullIf(tupleElement(p, 'epik'), ''), nullIf(toValidUTF8(left(decodeURLFormComponent(extractURLParameter(_attribution_url, 'epik')), 2048)), '')) as epik,
+ coalesce(nullIf(tupleElement(p, 'qclid'), ''), nullIf(toValidUTF8(left(decodeURLFormComponent(extractURLParameter(_attribution_url, 'qclid')), 2048)), '')) as qclid,
+ coalesce(nullIf(tupleElement(p, 'sccid'), ''), nullIf(toValidUTF8(left(decodeURLFormComponent(extractURLParameter(_attribution_url, 'sccid')), 2048)), ''), nullIf(toValidUTF8(left(decodeURLFormComponent(extractURLParameter(_attribution_url, 'ScCid')), 2048)), ''), nullIf(toValidUTF8(left(decodeURLFormComponent(extractURLParameter(_attribution_url, 'SsCid')), 2048)), '')) as sccid,
+ coalesce(nullIf(tupleElement(p, '_kx'), ''), nullIf(toValidUTF8(left(decodeURLFormComponent(extractURLParameter(_attribution_url, '_kx')), 2048)), '')) as _kx,
+ coalesce(nullIf(tupleElement(p, 'irclid'), ''), nullIf(toValidUTF8(left(decodeURLFormComponent(extractURLParameter(_attribution_url, 'irclid')), 2048)), '')) as irclid,
+ coalesce(nullIf(tupleElement(p, 'irclickid'), ''), nullIf(toValidUTF8(left(decodeURLFormComponent(extractURLParameter(_attribution_url, 'irclickid')), 2048)), '')) as irclickid,
+ coalesce(nullIf(tupleElement(p, 'utm_id'), ''), nullIf(toValidUTF8(left(decodeURLFormComponent(extractURLParameter(_attribution_url, 'utm_id')), 2048)), '')) as utm_id,
+ coalesce(nullIf(tupleElement(p, 'utm_source_platform'), ''), nullIf(toValidUTF8(left(decodeURLFormComponent(extractURLParameter(_attribution_url, 'utm_source_platform')), 2048)), '')) as utm_source_platform,
+ coalesce(nullIf(tupleElement(p, 'utm_creative_format'), ''), nullIf(toValidUTF8(left(decodeURLFormComponent(extractURLParameter(_attribution_url, 'utm_creative_format')), 2048)), '')) as utm_creative_format,
+ coalesce(nullIf(tupleElement(p, 'utm_marketing_tactic'), ''), nullIf(toValidUTF8(left(decodeURLFormComponent(extractURLParameter(_attribution_url, 'utm_marketing_tactic')), 2048)), '')) as utm_marketing_tactic,
+ coalesce(nullIf(tupleElement(p, 'yclid'), ''), nullIf(toValidUTF8(left(decodeURLFormComponent(extractURLParameter(_attribution_url, 'yclid')), 2048)), '')) as yclid,
+ coalesce(nullIf(tupleElement(p, 'tblci'), ''), nullIf(toValidUTF8(left(decodeURLFormComponent(extractURLParameter(_attribution_url, 'tblci')), 2048)), '')) as tblci,
+ coalesce(nullIf(tupleElement(p, 'dicbo'), ''), nullIf(toValidUTF8(left(decodeURLFormComponent(extractURLParameter(_attribution_url, 'dicbo')), 2048)), '')) as dicbo,
+ coalesce(nullIf(tupleElement(p, 'ndclid'), ''), nullIf(toValidUTF8(left(decodeURLFormComponent(extractURLParameter(_attribution_url, 'ndclid')), 2048)), '')) as ndclid,
+ coalesce(nullIf(tupleElement(p, 'cjevent'), ''), nullIf(toValidUTF8(left(decodeURLFormComponent(extractURLParameter(_attribution_url, 'cjevent')), 2048)), '')) as cjevent,
+ coalesce(nullIf(tupleElement(p, 'awc'), ''), nullIf(toValidUTF8(left(decodeURLFormComponent(extractURLParameter(_attribution_url, 'awc')), 2048)), '')) as awc,
+ coalesce(nullIf(tupleElement(p, 'syclid'), ''), nullIf(toValidUTF8(left(decodeURLFormComponent(extractURLParameter(_attribution_url, 'syclid')), 2048)), '')) as syclid,
+ coalesce(nullIf(tupleElement(p, '_bhlid'), ''), nullIf(toValidUTF8(left(decodeURLFormComponent(extractURLParameter(_attribution_url, '_bhlid')), 2048)), '')) as _bhlid,
+ coalesce(nullIf(tupleElement(p, 'rtid'), ''), nullIf(toValidUTF8(left(decodeURLFormComponent(extractURLParameter(_attribution_url, 'rtid')), 2048)), '')) as rtid,
+ coalesce(nullIf(tupleElement(p, 'oppref'), ''), nullIf(toValidUTF8(left(decodeURLFormComponent(extractURLParameter(_attribution_url, 'oppref')), 2048)), '')) as oppref,
+ coalesce(nullIf(tupleElement(p, 'bfcid'), ''), nullIf(toValidUTF8(left(decodeURLFormComponent(extractURLParameter(_attribution_url, 'bfcid')), 2048)), '')) as bfcid,
CAST(arrayFilter(x -> x IS NOT NULL, [
- if(gclsrc IS NOT NULL, 'gclsrc', NULL),
- if(dclid IS NOT NULL, 'dclid', NULL),
- if(gbraid IS NOT NULL, 'gbraid', NULL),
- if(wbraid IS NOT NULL, 'wbraid', NULL),
- if(msclkid IS NOT NULL, 'msclkid', NULL),
- if(twclid IS NOT NULL, 'twclid', NULL),
- if(li_fat_id IS NOT NULL, 'li_fat_id', NULL),
- if(mc_cid IS NOT NULL, 'mc_cid', NULL),
- if(igshid IS NOT NULL, 'igshid', NULL),
- if(ttclid IS NOT NULL, 'ttclid', NULL),
- if(epik IS NOT NULL, 'epik', NULL),
- if(qclid IS NOT NULL, 'qclid', NULL),
- if(sccid IS NOT NULL, 'sccid', NULL),
- if(_kx IS NOT NULL, '_kx', NULL),
- if(irclid IS NOT NULL, 'irclid', NULL)
+ if(gclsrc IS NOT NULL, concat('gclsrc=', gclsrc), NULL),
+ if(dclid IS NOT NULL, concat('dclid=', dclid), NULL),
+ if(gbraid IS NOT NULL, concat('gbraid=', gbraid), NULL),
+ if(wbraid IS NOT NULL, concat('wbraid=', wbraid), NULL),
+ if(gad_campaignid IS NOT NULL, concat('gad_campaignid=', gad_campaignid), NULL),
+ if(srsltid IS NOT NULL, concat('srsltid=', srsltid), NULL),
+ if(msclkid IS NOT NULL, concat('msclkid=', msclkid), NULL),
+ if(twclid IS NOT NULL, concat('twclid=', twclid), NULL),
+ if(li_fat_id IS NOT NULL, concat('li_fat_id=', li_fat_id), NULL),
+ if(mc_cid IS NOT NULL, concat('mc_cid=', mc_cid), NULL),
+ if(igshid IS NOT NULL, concat('igshid=', igshid), NULL),
+ if(ttclid IS NOT NULL, concat('ttclid=', ttclid), NULL),
+ if(rdt_cid IS NOT NULL, concat('rdt_cid=', rdt_cid), NULL),
+ if(epik IS NOT NULL, concat('epik=', epik), NULL),
+ if(qclid IS NOT NULL, concat('qclid=', qclid), NULL),
+ if(sccid IS NOT NULL, concat('sccid=', sccid), NULL),
+ if(_kx IS NOT NULL, concat('_kx=', _kx), NULL),
+ if(irclid IS NOT NULL, concat('irclid=', irclid), NULL),
+ if(irclickid IS NOT NULL, concat('irclickid=', irclickid), NULL),
+ if(utm_id IS NOT NULL, concat('utm_id=', utm_id), NULL),
+ if(utm_source_platform IS NOT NULL, concat('utm_source_platform=', utm_source_platform), NULL),
+ if(utm_creative_format IS NOT NULL, concat('utm_creative_format=', utm_creative_format), NULL),
+ if(utm_marketing_tactic IS NOT NULL, concat('utm_marketing_tactic=', utm_marketing_tactic), NULL),
+ if(yclid IS NOT NULL, concat('yclid=', yclid), NULL),
+ if(tblci IS NOT NULL, concat('tblci=', tblci), NULL),
+ if(dicbo IS NOT NULL, concat('dicbo=', dicbo), NULL),
+ if(ndclid IS NOT NULL, concat('ndclid=', ndclid), NULL),
+ if(cjevent IS NOT NULL, concat('cjevent=', cjevent), NULL),
+ if(awc IS NOT NULL, concat('awc=', awc), NULL),
+ if(syclid IS NOT NULL, concat('syclid=', syclid), NULL),
+ if(_bhlid IS NOT NULL, concat('_bhlid=', _bhlid), NULL),
+ if(rtid IS NOT NULL, concat('rtid=', rtid), NULL),
+ if(oppref IS NOT NULL, concat('oppref=', oppref), NULL),
+ if(bfcid IS NOT NULL, concat('bfcid=', bfcid), NULL)
]) AS Array(String)) as ad_ids_set,
tupleElement(p, '$host') as _host,
-- attribution properties from non-pageview/screen events should be deprioritized, so make the timestamp +/- 1 year so they sort last
@@ -7168,21 +7210,21 @@
initializeAggregation('argMinState', _fbclid, pageview_prio_timestamp_min) as entry_fbclid,
-- has gclid/fbclid for reading fewer bytes when calculating channel type
- initializeAggregation('argMinState', _gclid IS NOT NULL, pageview_prio_timestamp_min) as entry_has_gclid,
- initializeAggregation('argMinState', _fbclid IS NOT NULL, pageview_prio_timestamp_min) as entry_has_fbclid,
+ -- clients send "gclid": "" and that is not a click id, so an empty value must not count as present
+ initializeAggregation('argMinState', nullIf(_gclid, '') IS NOT NULL, pageview_prio_timestamp_min) as entry_has_gclid,
+ initializeAggregation('argMinState', nullIf(_fbclid, '') IS NOT NULL, pageview_prio_timestamp_min) as entry_has_fbclid,
-- other ad ids
- initializeAggregation('argMinState', ad_ids_map, pageview_prio_timestamp_min) as entry_ad_ids_map,
initializeAggregation('argMinState', ad_ids_set, pageview_prio_timestamp_min) as entry_ad_ids_set,
-- channel type
- initializeAggregation('argMinState', tuple(_utm_source, _utm_medium, _utm_campaign, _referring_domain, _gclid IS NOT NULL, _fbclid IS NOT NULL, _gad_source), pageview_prio_timestamp_min) as entry_channel_type_properties,
+ initializeAggregation('argMinState', tuple(_utm_source, _utm_medium, _utm_campaign, _referring_domain, nullIf(_gclid, '') IS NOT NULL, nullIf(_fbclid, '') IS NOT NULL, _gad_source), pageview_prio_timestamp_min) as entry_channel_type_properties,
-- counts
- initializeAggregation('uniqExactState', if(event='$pageview', uuid, NULL)) as pageview_uniq,
- initializeAggregation('uniqExactState', if(event='$autocapture', uuid, NULL)) as autocapture_uniq,
- initializeAggregation('uniqExactState', if(event='$screen', uuid, NULL)) as screen_uniq,
+ initializeAggregation('uniqState', if(event='$pageview', uuid, NULL)) as pageview_uniq,
+ initializeAggregation('uniqState', if(event='$autocapture', uuid, NULL)) as autocapture_uniq,
+ initializeAggregation('uniqState', if(event='$screen', uuid, NULL)) as screen_uniq,
-- perf
initializeAggregation('uniqUpToState(1)', if(event='$pageview' OR event='$screen', uuid, NULL)) as page_screen_uniq_up_to,
@@ -7190,7 +7232,6 @@
-- flags
arrayMap((k, v) -> concat(k, '=', v), mapKeys(properties_group_feature_flags), mapValues(properties_group_feature_flags)) as flag_key_values,
- mapKeys(properties_group_feature_flags) as flag_keys,
-- event names
[event] as event_names,
@@ -7204,6 +7245,11 @@
false as has_replay_events
FROM posthog_test.sharded_events AS source_table
WHERE bitAnd(bitShiftRight(toUInt128(accurateCastOrNull(`$session_id`, 'UUID')), 76), 0xF) == 7 -- has a session id and is valid uuidv7
+ -- Clients can send a UUIDv7 session id with a garbage timestamp (1970, year 6339). Each one puts its session in its
+ -- own monthly partition, and its events stretch the session duration across months. Keep only events within 31 days
+ -- of the session id's timestamp. The bound is symmetric because the client mints the session id with its own clock
+ -- while the server corrects event timestamps, so a client clock that runs days ahead is still a real session.
+ AND fromUnixTimestamp64Milli(toUInt64(bitShiftRight(`$session_id_uuid`, 80))) BETWEEN timestamp - INTERVAL 31 DAY AND timestamp + INTERVAL 31 DAY
AND TRUE
@@ -7272,16 +7318,15 @@
initializeAggregation('argMinState', false, max_ts_64) as entry_has_fbclid,
-- other ad ids
- initializeAggregation('argMinState', CAST(map(), 'Map(String, String)'), max_ts_64) as entry_ad_ids_map,
initializeAggregation('argMinState', CAST([], 'Array(String)'), max_ts_64) as entry_ad_ids_set,
-- channel type
initializeAggregation('argMinState', tuple(null_s, null_s, null_s, null_s, false, false, null_s), max_ts_64) as entry_channel_type_properties,
-- counts
- initializeAggregation('uniqExactState', null_uuid) as pageview_uniq,
- initializeAggregation('uniqExactState', null_uuid) as autocapture_uniq,
- initializeAggregation('uniqExactState', null_uuid) as screen_uniq,
+ initializeAggregation('uniqState', null_uuid) as pageview_uniq,
+ initializeAggregation('uniqState', null_uuid) as autocapture_uniq,
+ initializeAggregation('uniqState', null_uuid) as screen_uniq,
-- perf
initializeAggregation('uniqUpToState(1)', null_uuid) as page_screen_uniq_up_to,
@@ -7289,7 +7334,6 @@
-- flags
CAST([], 'Array(String)') as flag_key_values,
- CAST([], 'Array(String)') as flag_keys,
-- event names
CAST([], 'Array(String)') as event_names,
@@ -7304,6 +7348,9 @@
true as has_replay_events
FROM posthog_test.sharded_session_replay_events AS source_table
WHERE bitAnd(bitShiftRight(toUInt128(accurateCastOrNull(session_id, 'UUID')), 76), 0xF) == 7 -- has a session id and is valid uuidv7
+ -- same session timestamp guard as the events SELECT. accurateCastOrNull because accurateCast throws on a malformed
+ -- session_id, and ClickHouse can evaluate this condition before the uuidv7 check above filters those rows out
+ AND fromUnixTimestamp64Milli(toUInt64(bitShiftRight(toUInt128(accurateCastOrNull(session_id, 'UUID')), 80))) BETWEEN min_first_timestamp - INTERVAL 31 DAY AND min_first_timestamp + INTERVAL 31 DAY
AND TRUE
@@ -9048,8 +9095,8 @@
entry_has_gclid AggregateFunction(argMin, Boolean, DateTime64(6, 'UTC')),
entry_has_fbclid AggregateFunction(argMin, Boolean, DateTime64(6, 'UTC')),
- -- for lower-tier ad ids, just put them in a map, and set of the ones present
- entry_ad_ids_map AggregateFunction(argMin, Map(String, String), DateTime64(6, 'UTC')),
+ -- lower-tier ad ids as a flat 'key=value' array - like flag_key_values, a per-key map
+ -- state made merges much more expensive
entry_ad_ids_set AggregateFunction(argMin, Array(String), DateTime64(6, 'UTC')),
-- channel type properties tuple - to reduce redundant reading of the timestamp when loading all of these columns
@@ -9057,12 +9104,13 @@
entry_channel_type_properties AggregateFunction(argMin, Tuple(Nullable(String), Nullable(String), Nullable(String), Nullable(String), Boolean, Boolean, Nullable(String)), DateTime64(6, 'UTC')),
-- Count pageview, autocapture, and screen events for providing totals.
- -- Use uniqExact instead of count, so that inserting events can be idempotent. This is necessary as sometimes we see
+ -- Use uniq instead of count, so that inserting events can be idempotent. This is necessary as sometimes we see
-- events being inserted multiple times to be deduped later, but that can trigger multiple rows here.
-- Additionally, idempotency is useful for backfilling, as we can just reinsert the same events without worrying.
- pageview_uniq AggregateFunction(uniqExact, Nullable(UUID)),
- autocapture_uniq AggregateFunction(uniqExact, Nullable(UUID)),
- screen_uniq AggregateFunction(uniqExact, Nullable(UUID)),
+ -- uniq (not uniqExact) keeps the state small; per-session counts are tiny so it is exact in practice.
+ pageview_uniq AggregateFunction(uniq, Nullable(UUID)),
+ autocapture_uniq AggregateFunction(uniq, Nullable(UUID)),
+ screen_uniq AggregateFunction(uniq, Nullable(UUID)),
-- As a performance optimisation, also keep track of the uniq events for all of these combined.
-- This is a much more efficient way of calculating the bounce rate, as >2 means not a bounce
@@ -9071,8 +9119,8 @@
-- Flags - store every seen 'key=value' pair per flag. A flat array instead of a
-- groupUniqArrayMap state because per-key sub-aggregators made merges ~8-37x more expensive.
+ -- Key-only searches use the text index below (tokenized on '='), so no separate keys column.
flag_key_values SimpleAggregateFunction(groupUniqArrayArray(10000), Array(String)),
- flag_keys SimpleAggregateFunction(groupUniqArrayArray, Array(String)),
-- Event names - store unique event names seen in this session
event_names SimpleAggregateFunction(groupUniqArrayArray(2000), Array(String)),
@@ -9089,8 +9137,7 @@
-- Indexes
INDEX event_names_bloom_filter event_names TYPE bloom_filter() GRANULARITY 1,
- INDEX flag_key_values_bloom_filter flag_key_values TYPE bloom_filter() GRANULARITY 1,
- INDEX flag_keys_bloom_filter flag_keys TYPE bloom_filter() GRANULARITY 1,
+ INDEX flag_key_values_text flag_key_values TYPE text(tokenizer = splitByString(['='])),
INDEX hosts_bloom_filter hosts TYPE bloom_filter() GRANULARITY 1,
INDEX emails_bloom_filter emails TYPE bloom_filter() GRANULARITY 1
) ENGINE = ReplicatedAggregatingMergeTree('/clickhouse/tables/77f1df52-4b43-11e9-910f-b8ca3a9b9f3e_{shard}/posthog.raw_sessions_v3', '{replica}')
@@ -11544,8 +11591,8 @@
entry_has_gclid AggregateFunction(argMin, Boolean, DateTime64(6, 'UTC')),
entry_has_fbclid AggregateFunction(argMin, Boolean, DateTime64(6, 'UTC')),
- -- for lower-tier ad ids, just put them in a map, and set of the ones present
- entry_ad_ids_map AggregateFunction(argMin, Map(String, String), DateTime64(6, 'UTC')),
+ -- lower-tier ad ids as a flat 'key=value' array - like flag_key_values, a per-key map
+ -- state made merges much more expensive
entry_ad_ids_set AggregateFunction(argMin, Array(String), DateTime64(6, 'UTC')),
-- channel type properties tuple - to reduce redundant reading of the timestamp when loading all of these columns
@@ -11553,12 +11600,13 @@
entry_channel_type_properties AggregateFunction(argMin, Tuple(Nullable(String), Nullable(String), Nullable(String), Nullable(String), Boolean, Boolean, Nullable(String)), DateTime64(6, 'UTC')),
-- Count pageview, autocapture, and screen events for providing totals.
- -- Use uniqExact instead of count, so that inserting events can be idempotent. This is necessary as sometimes we see
+ -- Use uniq instead of count, so that inserting events can be idempotent. This is necessary as sometimes we see
-- events being inserted multiple times to be deduped later, but that can trigger multiple rows here.
-- Additionally, idempotency is useful for backfilling, as we can just reinsert the same events without worrying.
- pageview_uniq AggregateFunction(uniqExact, Nullable(UUID)),
- autocapture_uniq AggregateFunction(uniqExact, Nullable(UUID)),
- screen_uniq AggregateFunction(uniqExact, Nullable(UUID)),
+ -- uniq (not uniqExact) keeps the state small; per-session counts are tiny so it is exact in practice.
+ pageview_uniq AggregateFunction(uniq, Nullable(UUID)),
+ autocapture_uniq AggregateFunction(uniq, Nullable(UUID)),
+ screen_uniq AggregateFunction(uniq, Nullable(UUID)),
-- As a performance optimisation, also keep track of the uniq events for all of these combined.
-- This is a much more efficient way of calculating the bounce rate, as >2 means not a bounce
@@ -11567,8 +11615,8 @@
-- Flags - store every seen 'key=value' pair per flag. A flat array instead of a
-- groupUniqArrayMap state because per-key sub-aggregators made merges ~8-37x more expensive.
+ -- Key-only searches use the text index below (tokenized on '='), so no separate keys column.
flag_key_values SimpleAggregateFunction(groupUniqArrayArray(10000), Array(String)),
- flag_keys SimpleAggregateFunction(groupUniqArrayArray, Array(String)),
-- Event names - store unique event names seen in this session
event_names SimpleAggregateFunction(groupUniqArrayArray(2000), Array(String)),
@@ -14939,8 +14987,8 @@
entry_has_gclid AggregateFunction(argMin, Boolean, DateTime64(6, 'UTC')),
entry_has_fbclid AggregateFunction(argMin, Boolean, DateTime64(6, 'UTC')),
- -- for lower-tier ad ids, just put them in a map, and set of the ones present
- entry_ad_ids_map AggregateFunction(argMin, Map(String, String), DateTime64(6, 'UTC')),
+ -- lower-tier ad ids as a flat 'key=value' array - like flag_key_values, a per-key map
+ -- state made merges much more expensive
entry_ad_ids_set AggregateFunction(argMin, Array(String), DateTime64(6, 'UTC')),
-- channel type properties tuple - to reduce redundant reading of the timestamp when loading all of these columns
@@ -14948,12 +14996,13 @@
entry_channel_type_properties AggregateFunction(argMin, Tuple(Nullable(String), Nullable(String), Nullable(String), Nullable(String), Boolean, Boolean, Nullable(String)), DateTime64(6, 'UTC')),
-- Count pageview, autocapture, and screen events for providing totals.
- -- Use uniqExact instead of count, so that inserting events can be idempotent. This is necessary as sometimes we see
+ -- Use uniq instead of count, so that inserting events can be idempotent. This is necessary as sometimes we see
-- events being inserted multiple times to be deduped later, but that can trigger multiple rows here.
-- Additionally, idempotency is useful for backfilling, as we can just reinsert the same events without worrying.
- pageview_uniq AggregateFunction(uniqExact, Nullable(UUID)),
- autocapture_uniq AggregateFunction(uniqExact, Nullable(UUID)),
- screen_uniq AggregateFunction(uniqExact, Nullable(UUID)),
+ -- uniq (not uniqExact) keeps the state small; per-session counts are tiny so it is exact in practice.
+ pageview_uniq AggregateFunction(uniq, Nullable(UUID)),
+ autocapture_uniq AggregateFunction(uniq, Nullable(UUID)),
+ screen_uniq AggregateFunction(uniq, Nullable(UUID)),
-- As a performance optimisation, also keep track of the uniq events for all of these combined.
-- This is a much more efficient way of calculating the bounce rate, as >2 means not a bounce
@@ -14962,8 +15011,8 @@
-- Flags - store every seen 'key=value' pair per flag. A flat array instead of a
-- groupUniqArrayMap state because per-key sub-aggregators made merges ~8-37x more expensive.
+ -- Key-only searches use the text index below (tokenized on '='), so no separate keys column.
flag_key_values SimpleAggregateFunction(groupUniqArrayArray(10000), Array(String)),
- flag_keys SimpleAggregateFunction(groupUniqArrayArray, Array(String)),
-- Event names - store unique event names seen in this session
event_names SimpleAggregateFunction(groupUniqArrayArray(2000), Array(String)),
@@ -14980,8 +15029,7 @@
-- Indexes
INDEX event_names_bloom_filter event_names TYPE bloom_filter() GRANULARITY 1,
- INDEX flag_key_values_bloom_filter flag_key_values TYPE bloom_filter() GRANULARITY 1,
- INDEX flag_keys_bloom_filter flag_keys TYPE bloom_filter() GRANULARITY 1,
+ INDEX flag_key_values_text flag_key_values TYPE text(tokenizer = splitByString(['='])),
INDEX hosts_bloom_filter hosts TYPE bloom_filter() GRANULARITY 1,
INDEX emails_bloom_filter emails TYPE bloom_filter() GRANULARITY 1
) ENGINE = ReplicatedAggregatingMergeTree('/clickhouse/tables/77f1df52-4b43-11e9-910f-b8ca3a9b9f3e_{shard}/posthog.raw_sessions_v3', '{replica}')
diff --git a/posthog/clickhouse/test/test_raw_sessions_v3_model.py b/posthog/clickhouse/test/test_raw_sessions_v3_model.py
index 160d0f6d2898..d5ee351a105b 100644
--- a/posthog/clickhouse/test/test_raw_sessions_v3_model.py
+++ b/posthog/clickhouse/test/test_raw_sessions_v3_model.py
@@ -8,6 +8,8 @@ from posthog.test.base import (
snapshot_clickhouse_queries,
)
+from parameterized import parameterized
+
from posthog.clickhouse.client import query_with_columns, sync_execute
from posthog.models.raw_sessions.sessions_v3 import (
DISTRIBUTED_RAW_SESSIONS_TABLE_V3,
@@ -28,10 +30,12 @@ def create_distinct_id():
return f"d{distinct_id_counter}"
-def create_session_id():
+def create_session_id(start: str = "2024-03-08T00:00:00+00:00") -> str:
+ # The sessions MV drops events that are far from the session id's own timestamp, so the id has to
+ # start at the same time as the events these tests write.
global session_id_counter
session_id_counter += 1
- return str(uuid7(random=session_id_counter))
+ return str(uuid7(start, random=session_id_counter))
@snapshot_clickhouse_queries
@@ -373,7 +377,7 @@ class TestRawSessionsModel(ClickhouseTestMixin, BaseTest):
# assert that it's close to now, allowing for a small margin of error because we're running this on CI in the cloud somewhere with preempting
assert abs((max_inserted_at - now).total_seconds()) < 10
- def test_ad_ids_map_and_set(self):
+ def test_ad_ids_set_collects_kv_pairs(self):
distinct_id = create_distinct_id()
session_id = create_session_id()
@@ -391,13 +395,44 @@ class TestRawSessionsModel(ClickhouseTestMixin, BaseTest):
result = self.select_by_session_id(session_id)
- assert result[0]["entry_ad_ids_map"][present_ad_id] == value
- assert missing_ad_id not in result[0]["entry_ad_ids_map"].keys()
+ ad_ids_set = result[0]["entry_ad_ids_set"]
+ assert f"{present_ad_id}={value}" in ad_ids_set
+ assert not any(entry.startswith(f"{missing_ad_id}=") for entry in ad_ids_set)
+
+ @parameterized.expand(
+ [
+ ("gad_campaignid", "gad_campaignid", "123456789"),
+ ("utm_id", "utm_id", "spring_sale"),
+ ("irclickid", "irclickid", "imp_abc"),
+ ("snapchat_canonical", "sccid", "sc-lower"),
+ ("snapchat_sccid_cased", "ScCid", "sc-cased"),
+ ("snapchat_sscid_cased", "SsCid", "sc-alt"),
+ ]
+ )
+ def test_ad_ids_fall_back_to_url_params(self, _name, url_param, value):
+ distinct_id = create_distinct_id()
+ session_id = create_session_id()
+
+ _create_event(
+ team=self.team,
+ event="$pageview",
+ distinct_id=distinct_id,
+ properties={"$session_id": session_id, "$current_url": f"https://example.com/?{url_param}={value}"},
+ timestamp="2024-03-08",
+ )
+
+ result = self.select_by_session_id(session_id)
- assert present_ad_id in result[0]["entry_ad_ids_set"]
- assert missing_ad_id not in result[0]["entry_ad_ids_set"]
+ canonical_key = "sccid" if url_param in ("ScCid", "SsCid") else url_param
+ assert f"{canonical_key}={value}" in result[0]["entry_ad_ids_set"]
- def test_channel_type_properties(self):
+ @parameterized.expand(
+ [
+ ("property_wins", "from_property", "from_property"),
+ ("empty_property_falls_back", "", "from_url"),
+ ]
+ )
+ def test_ad_id_property_vs_url_param_precedence(self, _name, property_value, expected):
distinct_id = create_distinct_id()
session_id = create_session_id()
@@ -407,54 +442,144 @@ class TestRawSessionsModel(ClickhouseTestMixin, BaseTest):
distinct_id=distinct_id,
properties={
"$session_id": session_id,
- "gad_source": "1",
- "gclid": "some_gclid",
- "$referring_domain": "google.com",
- "utm_campaign": "some_campaign",
- "utm_medium": "some_medium",
- "utm_source": "some_source",
+ "ttclid": property_value,
+ "$current_url": "https://example.com/?ttclid=from_url",
},
timestamp="2024-03-08",
)
result = self.select_by_session_id(session_id)
- assert result[0]["entry_channel_type_properties"] == (
- "some_source",
- "some_medium",
- "some_campaign",
- "google.com",
- True,
- False,
- "1",
+ assert f"ttclid={expected}" in result[0]["entry_ad_ids_set"]
+
+ def test_ad_id_url_param_decodes_plus_as_space(self):
+ distinct_id = create_distinct_id()
+ session_id = create_session_id()
+
+ _create_event(
+ team=self.team,
+ event="$pageview",
+ distinct_id=distinct_id,
+ properties={
+ "$session_id": session_id,
+ "$current_url": "https://example.com/?utm_id=spring+sale",
+ },
+ timestamp="2024-03-08",
)
- def test_autocapture_does_not_set_attribution_when_pageview_present(self):
+ result = self.select_by_session_id(session_id)
+
+ # The SDK promotes this property as 'spring sale', so a fallback that kept the plus
+ # would split one campaign across two values.
+ assert "utm_id=spring sale" in result[0]["entry_ad_ids_set"]
+
+ def test_ad_id_url_param_with_undecodable_bytes_is_normalized(self):
distinct_id = create_distinct_id()
session_id = create_session_id()
_create_event(
team=self.team,
- event="$autocapture",
+ event="$pageview",
distinct_id=distinct_id,
properties={
"$session_id": session_id,
- "$current_url": "/1",
- "utm_source": "source1",
+ "$current_url": "https://example.com/?ttclid=%FF",
},
timestamp="2024-03-08",
)
+
+ result = self.select_by_session_id(session_id)
+
+ # Without toValidUTF8 this value comes back as bytes, which breaks text readers.
+ assert "ttclid=\ufffd" in result[0]["entry_ad_ids_set"]
+
+ @parameterized.expand(
+ [
+ ("click_ids_present", "some_gclid", "some_fbclid", True, True),
+ ("empty_click_ids_are_not_present", "", "", False, False),
+ ]
+ )
+ def test_channel_type_properties(self, _name, gclid, fbclid, expected_has_gclid, expected_has_fbclid):
+ distinct_id = create_distinct_id()
+ session_id = create_session_id()
+
_create_event(
team=self.team,
event="$pageview",
distinct_id=distinct_id,
properties={
"$session_id": session_id,
- "$current_url": "/2",
- "utm_source": "source2",
+ "gad_source": "1",
+ "gclid": gclid,
+ "fbclid": fbclid,
+ "$referring_domain": "google.com",
+ "utm_campaign": "some_campaign",
+ "utm_medium": "some_medium",
+ "utm_source": "some_source",
},
- timestamp="2024-03-09",
+ timestamp="2024-03-08",
+ )
+
+ result = self.select_by_session_id(session_id)
+
+ assert result[0]["entry_channel_type_properties"] == (
+ "some_source",
+ "some_medium",
+ "some_campaign",
+ "google.com",
+ expected_has_gclid,
+ expected_has_fbclid,
+ "1",
)
+ assert result[0]["entry_has_gclid"] is expected_has_gclid
+ assert result[0]["entry_has_fbclid"] is expected_has_fbclid
+
+ @parameterized.expand(
+ [
+ ("autocapture_first", "$autocapture", False),
+ ("pageleave_first", "$pageleave", False),
+ ("custom_event_first", "$feature_flag_called", False),
+ ("pageview_inserted_before_earlier_event", "$autocapture", True),
+ ]
+ )
+ def test_non_pageview_event_does_not_set_attribution_when_pageview_present(
+ self, _name, first_event, insert_pageview_first
+ ):
+ distinct_id = create_distinct_id()
+ session_id = create_session_id()
+
+ def create_first_event() -> None:
+ _create_event(
+ team=self.team,
+ event=first_event,
+ distinct_id=distinct_id,
+ properties={
+ "$session_id": session_id,
+ "$current_url": "/1",
+ "utm_source": "source1",
+ },
+ timestamp="2024-03-08",
+ )
+
+ def create_pageview() -> None:
+ _create_event(
+ team=self.team,
+ event="$pageview",
+ distinct_id=distinct_id,
+ properties={
+ "$session_id": session_id,
+ "$current_url": "/2",
+ "utm_source": "source2",
+ },
+ timestamp="2024-03-09",
+ )
+
+ if insert_pageview_first:
+ create_pageview()
+ create_first_event()
+ else:
+ create_first_event()
+ create_pageview()
result = self.select_by_session_id(session_id)
@@ -645,6 +770,35 @@ class TestRawSessionsModel(ClickhouseTestMixin, BaseTest):
k: v for k, v in result_2[0].items() if k not in {"has_replay_events", "max_inserted_at"}
}
+ @parameterized.expand(
+ [
+ ("garbage_session_timestamp", "1970-01-02T00:00:00+00:00", "2024-03-08", False),
+ ("session_id_over_31_days_after_event", "2024-04-10T00:00:00+00:00", "2024-03-08", False),
+ ("client_clock_days_ahead", "2024-03-11T00:00:00+00:00", "2024-03-08", True),
+ ("late_event_within_31_days", "2024-03-08T00:00:00+00:00", "2024-04-07", True),
+ ]
+ )
+ def test_drops_events_far_from_the_session_id_timestamp(self, _name, session_start, event_timestamp, kept):
+ distinct_id = create_distinct_id()
+ session_id = create_session_id(session_start)
+ _create_event(
+ team=self.team,
+ event="$pageview",
+ distinct_id=distinct_id,
+ properties={"$current_url": "/", "$session_id": session_id},
+ timestamp=event_timestamp,
+ )
+ produce_replay_summary(
+ team_id=self.team.pk,
+ distinct_id=distinct_id,
+ session_id=session_id,
+ first_timestamp=event_timestamp,
+ last_timestamp=event_timestamp,
+ ensure_analytics_event_in_session=False,
+ )
+
+ assert len(self.select_by_session_id(session_id)) == (1 if kept else 0)
+
def test_event_names_are_collected(self):
distinct_id = create_distinct_id()
session_id = create_session_id()
@@ -682,38 +836,6 @@ class TestRawSessionsModel(ClickhouseTestMixin, BaseTest):
assert set(result[0]["event_names"]) == {"$pageview", "$autocapture", "custom_event"}
- def test_flag_keys_are_collected(self):
- distinct_id = create_distinct_id()
- session_id = create_session_id()
-
- _create_event(
- team=self.team,
- event="$pageview",
- distinct_id=distinct_id,
- properties={
- "$session_id": session_id,
- "$feature/flag_a": "value1",
- "$feature/flag_b": "value2",
- },
- timestamp="2024-03-08",
- )
- _create_event(
- team=self.team,
- event="$autocapture",
- distinct_id=distinct_id,
- properties={
- "$session_id": session_id,
- "$feature/flag_a": "different_value",
- "$feature/flag_c": "value3",
- },
- timestamp="2024-03-08",
- )
-
- result = self.select_by_session_id(session_id)
-
- # Should have all unique flag keys (not values)
- assert set(result[0]["flag_keys"]) == {"$feature/flag_a", "$feature/flag_b", "$feature/flag_c"}
-
def test_hosts_are_collected(self):
distinct_id = create_distinct_id()
session_id = create_session_id()
diff --git a/posthog/hogql/database/schema/sessions_v3.py b/posthog/hogql/database/schema/sessions_v3.py
index 0262f3abc834..444b41a29b66 100644
--- a/posthog/hogql/database/schema/sessions_v3.py
+++ b/posthog/hogql/database/schema/sessions_v3.py
@@ -75,7 +75,6 @@ RAW_SESSIONS_FIELDS: dict[str, FieldOrTable] = {
"entry_fbclid": DatabaseField(name="entry_fbclid", nullable=False),
"entry_has_gclid": DatabaseField(name="entry_has_gclid", nullable=False),
"entry_has_fbclid": DatabaseField(name="entry_has_fbclid", nullable=False),
- "entry_ad_ids_map": DatabaseField(name="entry_ad_ids_map", nullable=False),
"entry_ad_ids_set": DatabaseField(name="entry_ad_ids_set", nullable=False),
"entry_channel_type_properties": DatabaseField(name="entry_channel_type_properties", nullable=False),
"pageview_uniq": DatabaseField(name="pageview_uniq", nullable=False),
@@ -156,7 +155,7 @@ LAZY_SESSIONS_FIELDS: dict[str, FieldOrTable] = {
"$entry_gad_source": DatabaseField(name="$entry_gad_source"),
"$entry_has_fbclid": BooleanDatabaseField(name="$entry_has_fbclid"),
"$entry_has_gclid": BooleanDatabaseField(name="$entry_has_gclid"),
- # "counts" - actually uses uniqExact behind the scenes
+ # "counts" - actually uses uniq behind the scenes
"$pageview_count": IntegerDatabaseField(name="$pageview_count"),
"$autocapture_count": IntegerDatabaseField(name="$autocapture_count"),
"$screen_count": IntegerDatabaseField(name="$screen_count"),
@@ -195,6 +194,43 @@ for session_ad_id in SESSION_V3_LOWER_TIER_AD_IDS:
LAZY_SESSIONS_FIELDS["$entry_has_" + session_ad_id] = BooleanDatabaseField(name="$entry_has_" + session_ad_id)
+# entry_ad_ids_set stores flat "key=value" strings; these helpers pull one key's value or presence out of it
+def ad_id_value_from_set_expr(set_expr: ast.Expr, session_ad_id: str) -> ast.Call:
+ prefix = session_ad_id + "="
+ # arrayFirst guarantees the prefix sits at position 0, so replaceOne strips exactly the prefix
+ return ast.Call(
+ name="replaceOne",
+ args=[
+ ast.Call(
+ name="arrayFirst",
+ args=[
+ ast.Lambda(
+ args=["x"],
+ expr=ast.Call(name="startsWith", args=[ast.Field(chain=["x"]), ast.Constant(value=prefix)]),
+ ),
+ set_expr,
+ ],
+ ),
+ ast.Constant(value=prefix),
+ ast.Constant(value=""),
+ ],
+ )
+
+
+def ad_id_present_in_set_expr(set_expr: ast.Expr, session_ad_id: str) -> ast.Call:
+ prefix = session_ad_id + "="
+ return ast.Call(
+ name="arrayExists",
+ args=[
+ ast.Lambda(
+ args=["x"],
+ expr=ast.Call(name="startsWith", args=[ast.Field(chain=["x"]), ast.Constant(value=prefix)]),
+ ),
+ set_expr,
+ ],
+ )
+
+
def get_binary_fields(table: Table) -> set[str]:
return {key for key, val in table.fields.items() if val.__class__ == DatabaseField}
@@ -295,9 +331,9 @@ def select_from_sessions_table_v3(
"$entry_has_gclid": arg_min_merge_field("entry_has_gclid"),
"$entry_has_fbclid": arg_min_merge_field("entry_has_fbclid"),
# the count columns here do not come from the "count" columns in the raw table, instead aggregate the uniq columns
- "$pageview_count": ast.Call(name="uniqExactMerge", args=[ast.Field(chain=[table_name, "pageview_uniq"])]),
- "$screen_count": ast.Call(name="uniqExactMerge", args=[ast.Field(chain=[table_name, "screen_uniq"])]),
- "$autocapture_count": ast.Call(name="uniqExactMerge", args=[ast.Field(chain=[table_name, "autocapture_uniq"])]),
+ "$pageview_count": ast.Call(name="uniqMerge", args=[ast.Field(chain=[table_name, "pageview_uniq"])]),
+ "$screen_count": ast.Call(name="uniqMerge", args=[ast.Field(chain=[table_name, "screen_uniq"])]),
+ "$autocapture_count": ast.Call(name="uniqMerge", args=[ast.Field(chain=[table_name, "autocapture_uniq"])]),
"$page_screen_count_up_to": ast.Call(
name="uniqUpToMerge",
params=[ast.Constant(value=1)],
@@ -307,7 +343,6 @@ def select_from_sessions_table_v3(
name="max",
args=[ast.Field(chain=[table_name, "has_autocapture"])],
),
- "$entry_ad_ids_map": arg_min_merge_field("entry_ad_ids_map"),
"$entry_ad_ids_set": arg_min_merge_field("entry_ad_ids_set"),
"$entry_channel_type_properties": arg_min_merge_field("entry_channel_type_properties"),
"$has_replay_events": ast.Call(name="max", args=[ast.Field(chain=[table_name, "has_replay_events"])]),
@@ -318,16 +353,11 @@ def select_from_sessions_table_v3(
# Add ad ids fields
for session_ad_id in SESSION_V3_LOWER_TIER_AD_IDS:
- aggregate_fields["$entry_" + session_ad_id] = ast.Call(
- name="arrayElement",
- args=[
- aggregate_fields["$entry_ad_ids_map"],
- ast.Constant(value=session_ad_id),
- ],
+ aggregate_fields["$entry_" + session_ad_id] = ad_id_value_from_set_expr(
+ aggregate_fields["$entry_ad_ids_set"], session_ad_id
)
- aggregate_fields["$entry_has_" + session_ad_id] = ast.Call(
- name="has",
- args=[aggregate_fields["$entry_ad_ids_set"], ast.Constant(value=session_ad_id)],
+ aggregate_fields["$entry_has_" + session_ad_id] = ad_id_present_in_set_expr(
+ aggregate_fields["$entry_ad_ids_set"], session_ad_id
)
# Some fields are calculated from others. It'd be good to actually deduplicate common sub expressions in SQL, but
@@ -601,9 +631,8 @@ SESSION_PROPERTY_TO_RAW_SESSIONS_EXPR: dict[str, ast.Expr] = {
}
for session_ad_id in SESSION_V3_LOWER_TIER_AD_IDS:
- SESSION_PROPERTY_TO_RAW_SESSIONS_EXPR["$entry_" + session_ad_id] = ast.Call(
- name="arrayElement",
- args=[finalize_aggregation("entry_ad_ids_map"), ast.Constant(value=session_ad_id)],
+ SESSION_PROPERTY_TO_RAW_SESSIONS_EXPR["$entry_" + session_ad_id] = ad_id_value_from_set_expr(
+ finalize_aggregation("entry_ad_ids_set"), session_ad_id
)
diff --git a/posthog/hogql/database/schema/test/__snapshots__/test_sessions_v3.ambr b/posthog/hogql/database/schema/test/__snapshots__/test_sessions_v3.ambr
index 8560c75ea4fa..de42c3865df3 100644
--- a/posthog/hogql/database/schema/test/__snapshots__/test_sessions_v3.ambr
+++ b/posthog/hogql/database/schema/test/__snapshots__/test_sessions_v3.ambr
@@ -60,7 +60,232 @@
use_hive_partitioning=0
'''
# ---
-# name: TestSessionsV3.test_channel_type
+# name: TestSessionsV3.test_channel_type_0_gad_source
+ '''
+ SELECT sessions.`$channel_type` AS `$channel_type`
+ FROM
+ (SELECT multiIf(match(lower(nullIf(nullIf(tupleElement(argMinMerge(raw_sessions_v3.entry_channel_type_properties), 3), ''), 'null')), 'cross-network'), 'Cross Network', or(in(lower(nullIf(nullIf(tupleElement(argMinMerge(raw_sessions_v3.entry_channel_type_properties), 2), ''), 'null')), tuple('cpc', 'cpm', 'cpv', 'cpa', 'ppc', 'retargeting')), startsWith(lower(nullIf(nullIf(tupleElement(argMinMerge(raw_sessions_v3.entry_channel_type_properties), 2), ''), 'null')), 'paid'), tupleElement(argMinMerge(raw_sessions_v3.entry_channel_type_properties), 5), isNotNull(nullIf(nullIf(tupleElement(argMinMerge(raw_sessions_v3.entry_channel_type_properties), 7), ''), 'null'))), coalesce(coalesce(dictGetOrNull('posthog_test.channel_definition_dict', 'type_if_paid', (coalesce(lower(nullIf(nullIf(tupleElement(argMinMerge(raw_sessions_v3.entry_channel_type_properties), 1), ''), 'null')), ''), 'source')), dictGetOrNull('posthog_test.channel_definition_dict', 'type_if_paid', (cutToFirstSignificantSubdomain(coalesce(lower(nullIf(nullIf(tupleElement(argMinMerge(raw_sessions_v3.entry_channel_type_properties), 1), ''), 'null')), '')), 'source'))), if(match(lower(nullIf(nullIf(tupleElement(argMinMerge(raw_sessions_v3.entry_channel_type_properties), 3), ''), 'null')), '^(.*(([^a-df-z]|^)shop|shopping).*)$'), 'Paid Shopping', NULL), dictGetOrNull('posthog_test.channel_definition_dict', 'type_if_paid', (coalesce(lower(nullIf(nullIf(tupleElement(argMinMerge(raw_sessions_v3.entry_channel_type_properties), 2), ''), 'null')), ''), 'medium')), coalesce(dictGetOrNull('posthog_test.channel_definition_dict', 'type_if_paid', (coalesce(tupleElement(argMinMerge(raw_sessions_v3.entry_channel_type_properties), 4), ''), 'source')), dictGetOrNull('posthog_test.channel_definition_dict', 'type_if_paid', (cutToFirstSignificantSubdomain(coalesce(tupleElement(argMinMerge(raw_sessions_v3.entry_channel_type_properties), 4), '')), 'source'))), multiIf(ifNull(equals(nullIf(nullIf(tupleElement(argMinMerge(raw_sessions_v3.entry_channel_type_properties), 7), ''), 'null'), '1'), 0), 'Paid Search', match(lower(nullIf(nullIf(tupleElement(argMinMerge(raw_sessions_v3.entry_channel_type_properties), 3), ''), 'null')), '^(.*video.*)$'), 'Paid Video', tupleElement(argMinMerge(raw_sessions_v3.entry_channel_type_properties), 6), 'Paid Social', 'Paid Unknown')), and(ifNull(equals(tupleElement(argMinMerge(raw_sessions_v3.entry_channel_type_properties), 4), '$direct'), 0), isNull(lower(nullIf(nullIf(tupleElement(argMinMerge(raw_sessions_v3.entry_channel_type_properties), 2), ''), 'null'))), or(isNull(lower(nullIf(nullIf(tupleElement(argMinMerge(raw_sessions_v3.entry_channel_type_properties), 1), ''), 'null'))), in(lower(nullIf(nullIf(tupleElement(argMinMerge(raw_sessions_v3.entry_channel_type_properties), 1), ''), 'null')), tuple('(direct)', 'direct', '$direct'))), not(tupleElement(argMinMerge(raw_sessions_v3.entry_channel_type_properties), 6))), 'Direct', coalesce(coalesce(dictGetOrNull('posthog_test.channel_definition_dict', 'type_if_organic', (coalesce(lower(nullIf(nullIf(tupleElement(argMinMerge(raw_sessions_v3.entry_channel_type_properties), 1), ''), 'null')), ''), 'source')), dictGetOrNull('posthog_test.channel_definition_dict', 'type_if_organic', (cutToFirstSignificantSubdomain(coalesce(lower(nullIf(nullIf(tupleElement(argMinMerge(raw_sessions_v3.entry_channel_type_properties), 1), ''), 'null')), '')), 'source'))), if(match(lower(nullIf(nullIf(tupleElement(argMinMerge(raw_sessions_v3.entry_channel_type_properties), 3), ''), 'null')), '^(.*(([^a-df-z]|^)shop|shopping).*)$'), 'Organic Shopping', NULL), dictGetOrNull('posthog_test.channel_definition_dict', 'type_if_organic', (coalesce(lower(nullIf(nullIf(tupleElement(argMinMerge(raw_sessions_v3.entry_channel_type_properties), 2), ''), 'null')), ''), 'medium')), coalesce(dictGetOrNull('posthog_test.channel_definition_dict', 'type_if_organic', (coalesce(tupleElement(argMinMerge(raw_sessions_v3.entry_channel_type_properties), 4), ''), 'source')), dictGetOrNull('posthog_test.channel_definition_dict', 'type_if_organic', (cutToFirstSignificantSubdomain(coalesce(tupleElement(argMinMerge(raw_sessions_v3.entry_channel_type_properties), 4), '')), 'source'))), multiIf(match(lower(nullIf(nullIf(tupleElement(argMinMerge(raw_sessions_v3.entry_channel_type_properties), 3), ''), 'null')), '^(.*video.*)$'), 'Organic Video', match(lower(nullIf(nullIf(tupleElement(argMinMerge(raw_sessions_v3.entry_channel_type_properties), 2), ''), 'null')), 'push$'), 'Push', tupleElement(argMinMerge(raw_sessions_v3.entry_channel_type_properties), 6), 'Organic Social', ifNull(equals(tupleElement(argMinMerge(raw_sessions_v3.entry_channel_type_properties), 4), '$direct'), 0), 'Direct', isNotNull(tupleElement(argMinMerge(raw_sessions_v3.entry_channel_type_properties), 4)), 'Referral', 'Unknown'))) AS `$channel_type`,
+ toString(reinterpretAsUUID(bitOr(bitShiftLeft(raw_sessions_v3.session_id_v7, 64), bitShiftRight(raw_sessions_v3.session_id_v7, 64)))) AS session_id,
+ raw_sessions_v3.session_id_v7 AS session_id_v7
+ FROM raw_sessions_v3
+ WHERE and(equals(raw_sessions_v3.team_id, 99999), equals(raw_sessions_v3.session_timestamp, fromUnixTimestamp64Milli(toUInt64(bitShiftRight(toUInt128(accurateCastOrNull('00000000-0000-0000-0000-000000000000', 'UUID')), 80)))))
+ GROUP BY raw_sessions_v3.session_id_v7) AS sessions
+ WHERE equals(sessions.session_id, '00000000-0000-0000-0000-000000000000')
+ LIMIT 100 SETTINGS readonly=2,
+ max_execution_time=60,
+ allow_experimental_object_type=1,
+ max_ast_elements=4000000,
+ max_expanded_ast_elements=4000000,
+ max_bytes_before_external_group_by=0,
+ transform_null_in=1,
+ optimize_min_equality_disjunction_chain_length=4294967295,
+ optimize_rewrite_aggregate_function_with_if=0,
+ optimize_min_inequality_conjunction_chain_length=4294967295,
+ allow_experimental_join_condition=1,
+ use_hive_partitioning=0
+ '''
+# ---
+# name: TestSessionsV3.test_channel_type_1_gclid_from_google
+ '''
+ SELECT sessions.`$channel_type` AS `$channel_type`
+ FROM
+ (SELECT multiIf(match(lower(nullIf(nullIf(tupleElement(argMinMerge(raw_sessions_v3.entry_channel_type_properties), 3), ''), 'null')), 'cross-network'), 'Cross Network', or(in(lower(nullIf(nullIf(tupleElement(argMinMerge(raw_sessions_v3.entry_channel_type_properties), 2), ''), 'null')), tuple('cpc', 'cpm', 'cpv', 'cpa', 'ppc', 'retargeting')), startsWith(lower(nullIf(nullIf(tupleElement(argMinMerge(raw_sessions_v3.entry_channel_type_properties), 2), ''), 'null')), 'paid'), tupleElement(argMinMerge(raw_sessions_v3.entry_channel_type_properties), 5), isNotNull(nullIf(nullIf(tupleElement(argMinMerge(raw_sessions_v3.entry_channel_type_properties), 7), ''), 'null'))), coalesce(coalesce(dictGetOrNull('posthog_test.channel_definition_dict', 'type_if_paid', (coalesce(lower(nullIf(nullIf(tupleElement(argMinMerge(raw_sessions_v3.entry_channel_type_properties), 1), ''), 'null')), ''), 'source')), dictGetOrNull('posthog_test.channel_definition_dict', 'type_if_paid', (cutToFirstSignificantSubdomain(coalesce(lower(nullIf(nullIf(tupleElement(argMinMerge(raw_sessions_v3.entry_channel_type_properties), 1), ''), 'null')), '')), 'source'))), if(match(lower(nullIf(nullIf(tupleElement(argMinMerge(raw_sessions_v3.entry_channel_type_properties), 3), ''), 'null')), '^(.*(([^a-df-z]|^)shop|shopping).*)$'), 'Paid Shopping', NULL), dictGetOrNull('posthog_test.channel_definition_dict', 'type_if_paid', (coalesce(lower(nullIf(nullIf(tupleElement(argMinMerge(raw_sessions_v3.entry_channel_type_properties), 2), ''), 'null')), ''), 'medium')), coalesce(dictGetOrNull('posthog_test.channel_definition_dict', 'type_if_paid', (coalesce(tupleElement(argMinMerge(raw_sessions_v3.entry_channel_type_properties), 4), ''), 'source')), dictGetOrNull('posthog_test.channel_definition_dict', 'type_if_paid', (cutToFirstSignificantSubdomain(coalesce(tupleElement(argMinMerge(raw_sessions_v3.entry_channel_type_properties), 4), '')), 'source'))), multiIf(ifNull(equals(nullIf(nullIf(tupleElement(argMinMerge(raw_sessions_v3.entry_channel_type_properties), 7), ''), 'null'), '1'), 0), 'Paid Search', match(lower(nullIf(nullIf(tupleElement(argMinMerge(raw_sessions_v3.entry_channel_type_properties), 3), ''), 'null')), '^(.*video.*)$'), 'Paid Video', tupleElement(argMinMerge(raw_sessions_v3.entry_channel_type_properties), 6), 'Paid Social', 'Paid Unknown')), and(ifNull(equals(tupleElement(argMinMerge(raw_sessions_v3.entry_channel_type_properties), 4), '$direct'), 0), isNull(lower(nullIf(nullIf(tupleElement(argMinMerge(raw_sessions_v3.entry_channel_type_properties), 2), ''), 'null'))), or(isNull(lower(nullIf(nullIf(tupleElement(argMinMerge(raw_sessions_v3.entry_channel_type_properties), 1), ''), 'null'))), in(lower(nullIf(nullIf(tupleElement(argMinMerge(raw_sessions_v3.entry_channel_type_properties), 1), ''), 'null')), tuple('(direct)', 'direct', '$direct'))), not(tupleElement(argMinMerge(raw_sessions_v3.entry_channel_type_properties), 6))), 'Direct', coalesce(coalesce(dictGetOrNull('posthog_test.channel_definition_dict', 'type_if_organic', (coalesce(lower(nullIf(nullIf(tupleElement(argMinMerge(raw_sessions_v3.entry_channel_type_properties), 1), ''), 'null')), ''), 'source')), dictGetOrNull('posthog_test.channel_definition_dict', 'type_if_organic', (cutToFirstSignificantSubdomain(coalesce(lower(nullIf(nullIf(tupleElement(argMinMerge(raw_sessions_v3.entry_channel_type_properties), 1), ''), 'null')), '')), 'source'))), if(match(lower(nullIf(nullIf(tupleElement(argMinMerge(raw_sessions_v3.entry_channel_type_properties), 3), ''), 'null')), '^(.*(([^a-df-z]|^)shop|shopping).*)$'), 'Organic Shopping', NULL), dictGetOrNull('posthog_test.channel_definition_dict', 'type_if_organic', (coalesce(lower(nullIf(nullIf(tupleElement(argMinMerge(raw_sessions_v3.entry_channel_type_properties), 2), ''), 'null')), ''), 'medium')), coalesce(dictGetOrNull('posthog_test.channel_definition_dict', 'type_if_organic', (coalesce(tupleElement(argMinMerge(raw_sessions_v3.entry_channel_type_properties), 4), ''), 'source')), dictGetOrNull('posthog_test.channel_definition_dict', 'type_if_organic', (cutToFirstSignificantSubdomain(coalesce(tupleElement(argMinMerge(raw_sessions_v3.entry_channel_type_properties), 4), '')), 'source'))), multiIf(match(lower(nullIf(nullIf(tupleElement(argMinMerge(raw_sessions_v3.entry_channel_type_properties), 3), ''), 'null')), '^(.*video.*)$'), 'Organic Video', match(lower(nullIf(nullIf(tupleElement(argMinMerge(raw_sessions_v3.entry_channel_type_properties), 2), ''), 'null')), 'push$'), 'Push', tupleElement(argMinMerge(raw_sessions_v3.entry_channel_type_properties), 6), 'Organic Social', ifNull(equals(tupleElement(argMinMerge(raw_sessions_v3.entry_channel_type_properties), 4), '$direct'), 0), 'Direct', isNotNull(tupleElement(argMinMerge(raw_sessions_v3.entry_channel_type_properties), 4)), 'Referral', 'Unknown'))) AS `$channel_type`,
+ toString(reinterpretAsUUID(bitOr(bitShiftLeft(raw_sessions_v3.session_id_v7, 64), bitShiftRight(raw_sessions_v3.session_id_v7, 64)))) AS session_id,
+ raw_sessions_v3.session_id_v7 AS session_id_v7
+ FROM raw_sessions_v3
+ WHERE and(equals(raw_sessions_v3.team_id, 99999), equals(raw_sessions_v3.session_timestamp, fromUnixTimestamp64Milli(toUInt64(bitShiftRight(toUInt128(accurateCastOrNull('00000000-0000-0000-0000-000000000000', 'UUID')), 80)))))
+ GROUP BY raw_sessions_v3.session_id_v7) AS sessions
+ WHERE equals(sessions.session_id, '00000000-0000-0000-0000-000000000000')
+ LIMIT 100 SETTINGS readonly=2,
+ max_execution_time=60,
+ allow_experimental_object_type=1,
+ max_ast_elements=4000000,
+ max_expanded_ast_elements=4000000,
+ max_bytes_before_external_group_by=0,
+ transform_null_in=1,
+ optimize_min_equality_disjunction_chain_length=4294967295,
+ optimize_rewrite_aggregate_function_with_if=0,
+ optimize_min_inequality_conjunction_chain_length=4294967295,
+ allow_experimental_join_condition=1,
+ use_hive_partitioning=0
+ '''
+# ---
+# name: TestSessionsV3.test_channel_type_2_empty_gclid_is_not_a_click
+ '''
+ SELECT sessions.`$channel_type` AS `$channel_type`
+ FROM
+ (SELECT multiIf(match(lower(nullIf(nullIf(tupleElement(argMinMerge(raw_sessions_v3.entry_channel_type_properties), 3), ''), 'null')), 'cross-network'), 'Cross Network', or(in(lower(nullIf(nullIf(tupleElement(argMinMerge(raw_sessions_v3.entry_channel_type_properties), 2), ''), 'null')), tuple('cpc', 'cpm', 'cpv', 'cpa', 'ppc', 'retargeting')), startsWith(lower(nullIf(nullIf(tupleElement(argMinMerge(raw_sessions_v3.entry_channel_type_properties), 2), ''), 'null')), 'paid'), tupleElement(argMinMerge(raw_sessions_v3.entry_channel_type_properties), 5), isNotNull(nullIf(nullIf(tupleElement(argMinMerge(raw_sessions_v3.entry_channel_type_properties), 7), ''), 'null'))), coalesce(coalesce(dictGetOrNull('posthog_test.channel_definition_dict', 'type_if_paid', (coalesce(lower(nullIf(nullIf(tupleElement(argMinMerge(raw_sessions_v3.entry_channel_type_properties), 1), ''), 'null')), ''), 'source')), dictGetOrNull('posthog_test.channel_definition_dict', 'type_if_paid', (cutToFirstSignificantSubdomain(coalesce(lower(nullIf(nullIf(tupleElement(argMinMerge(raw_sessions_v3.entry_channel_type_properties), 1), ''), 'null')), '')), 'source'))), if(match(lower(nullIf(nullIf(tupleElement(argMinMerge(raw_sessions_v3.entry_channel_type_properties), 3), ''), 'null')), '^(.*(([^a-df-z]|^)shop|shopping).*)$'), 'Paid Shopping', NULL), dictGetOrNull('posthog_test.channel_definition_dict', 'type_if_paid', (coalesce(lower(nullIf(nullIf(tupleElement(argMinMerge(raw_sessions_v3.entry_channel_type_properties), 2), ''), 'null')), ''), 'medium')), coalesce(dictGetOrNull('posthog_test.channel_definition_dict', 'type_if_paid', (coalesce(tupleElement(argMinMerge(raw_sessions_v3.entry_channel_type_properties), 4), ''), 'source')), dictGetOrNull('posthog_test.channel_definition_dict', 'type_if_paid', (cutToFirstSignificantSubdomain(coalesce(tupleElement(argMinMerge(raw_sessions_v3.entry_channel_type_properties), 4), '')), 'source'))), multiIf(ifNull(equals(nullIf(nullIf(tupleElement(argMinMerge(raw_sessions_v3.entry_channel_type_properties), 7), ''), 'null'), '1'), 0), 'Paid Search', match(lower(nullIf(nullIf(tupleElement(argMinMerge(raw_sessions_v3.entry_channel_type_properties), 3), ''), 'null')), '^(.*video.*)$'), 'Paid Video', tupleElement(argMinMerge(raw_sessions_v3.entry_channel_type_properties), 6), 'Paid Social', 'Paid Unknown')), and(ifNull(equals(tupleElement(argMinMerge(raw_sessions_v3.entry_channel_type_properties), 4), '$direct'), 0), isNull(lower(nullIf(nullIf(tupleElement(argMinMerge(raw_sessions_v3.entry_channel_type_properties), 2), ''), 'null'))), or(isNull(lower(nullIf(nullIf(tupleElement(argMinMerge(raw_sessions_v3.entry_channel_type_properties), 1), ''), 'null'))), in(lower(nullIf(nullIf(tupleElement(argMinMerge(raw_sessions_v3.entry_channel_type_properties), 1), ''), 'null')), tuple('(direct)', 'direct', '$direct'))), not(tupleElement(argMinMerge(raw_sessions_v3.entry_channel_type_properties), 6))), 'Direct', coalesce(coalesce(dictGetOrNull('posthog_test.channel_definition_dict', 'type_if_organic', (coalesce(lower(nullIf(nullIf(tupleElement(argMinMerge(raw_sessions_v3.entry_channel_type_properties), 1), ''), 'null')), ''), 'source')), dictGetOrNull('posthog_test.channel_definition_dict', 'type_if_organic', (cutToFirstSignificantSubdomain(coalesce(lower(nullIf(nullIf(tupleElement(argMinMerge(raw_sessions_v3.entry_channel_type_properties), 1), ''), 'null')), '')), 'source'))), if(match(lower(nullIf(nullIf(tupleElement(argMinMerge(raw_sessions_v3.entry_channel_type_properties), 3), ''), 'null')), '^(.*(([^a-df-z]|^)shop|shopping).*)$'), 'Organic Shopping', NULL), dictGetOrNull('posthog_test.channel_definition_dict', 'type_if_organic', (coalesce(lower(nullIf(nullIf(tupleElement(argMinMerge(raw_sessions_v3.entry_channel_type_properties), 2), ''), 'null')), ''), 'medium')), coalesce(dictGetOrNull('posthog_test.channel_definition_dict', 'type_if_organic', (coalesce(tupleElement(argMinMerge(raw_sessions_v3.entry_channel_type_properties), 4), ''), 'source')), dictGetOrNull('posthog_test.channel_definition_dict', 'type_if_organic', (cutToFirstSignificantSubdomain(coalesce(tupleElement(argMinMerge(raw_sessions_v3.entry_channel_type_properties), 4), '')), 'source'))), multiIf(match(lower(nullIf(nullIf(tupleElement(argMinMerge(raw_sessions_v3.entry_channel_type_properties), 3), ''), 'null')), '^(.*video.*)$'), 'Organic Video', match(lower(nullIf(nullIf(tupleElement(argMinMerge(raw_sessions_v3.entry_channel_type_properties), 2), ''), 'null')), 'push$'), 'Push', tupleElement(argMinMerge(raw_sessions_v3.entry_channel_type_properties), 6), 'Organic Social', ifNull(equals(tupleElement(argMinMerge(raw_sessions_v3.entry_channel_type_properties), 4), '$direct'), 0), 'Direct', isNotNull(tupleElement(argMinMerge(raw_sessions_v3.entry_channel_type_properties), 4)), 'Referral', 'Unknown'))) AS `$channel_type`,
+ toString(reinterpretAsUUID(bitOr(bitShiftLeft(raw_sessions_v3.session_id_v7, 64), bitShiftRight(raw_sessions_v3.session_id_v7, 64)))) AS session_id,
+ raw_sessions_v3.session_id_v7 AS session_id_v7
+ FROM raw_sessions_v3
+ WHERE and(equals(raw_sessions_v3.team_id, 99999), equals(raw_sessions_v3.session_timestamp, fromUnixTimestamp64Milli(toUInt64(bitShiftRight(toUInt128(accurateCastOrNull('00000000-0000-0000-0000-000000000000', 'UUID')), 80)))))
+ GROUP BY raw_sessions_v3.session_id_v7) AS sessions
+ WHERE equals(sessions.session_id, '00000000-0000-0000-0000-000000000000')
+ LIMIT 100 SETTINGS readonly=2,
+ max_execution_time=60,
+ allow_experimental_object_type=1,
+ max_ast_elements=4000000,
+ max_expanded_ast_elements=4000000,
+ max_bytes_before_external_group_by=0,
+ transform_null_in=1,
+ optimize_min_equality_disjunction_chain_length=4294967295,
+ optimize_rewrite_aggregate_function_with_if=0,
+ optimize_min_inequality_conjunction_chain_length=4294967295,
+ allow_experimental_join_condition=1,
+ use_hive_partitioning=0
+ '''
+# ---
+# name: TestSessionsV3.test_channel_type_3_gad_source_without_referrer
+ '''
+ SELECT sessions.`$channel_type` AS `$channel_type`
+ FROM
+ (SELECT multiIf(match(lower(nullIf(nullIf(tupleElement(argMinMerge(raw_sessions_v3.entry_channel_type_properties), 3), ''), 'null')), 'cross-network'), 'Cross Network', or(in(lower(nullIf(nullIf(tupleElement(argMinMerge(raw_sessions_v3.entry_channel_type_properties), 2), ''), 'null')), tuple('cpc', 'cpm', 'cpv', 'cpa', 'ppc', 'retargeting')), startsWith(lower(nullIf(nullIf(tupleElement(argMinMerge(raw_sessions_v3.entry_channel_type_properties), 2), ''), 'null')), 'paid'), tupleElement(argMinMerge(raw_sessions_v3.entry_channel_type_properties), 5), isNotNull(nullIf(nullIf(tupleElement(argMinMerge(raw_sessions_v3.entry_channel_type_properties), 7), ''), 'null'))), coalesce(coalesce(dictGetOrNull('posthog_test.channel_definition_dict', 'type_if_paid', (coalesce(lower(nullIf(nullIf(tupleElement(argMinMerge(raw_sessions_v3.entry_channel_type_properties), 1), ''), 'null')), ''), 'source')), dictGetOrNull('posthog_test.channel_definition_dict', 'type_if_paid', (cutToFirstSignificantSubdomain(coalesce(lower(nullIf(nullIf(tupleElement(argMinMerge(raw_sessions_v3.entry_channel_type_properties), 1), ''), 'null')), '')), 'source'))), if(match(lower(nullIf(nullIf(tupleElement(argMinMerge(raw_sessions_v3.entry_channel_type_properties), 3), ''), 'null')), '^(.*(([^a-df-z]|^)shop|shopping).*)$'), 'Paid Shopping', NULL), dictGetOrNull('posthog_test.channel_definition_dict', 'type_if_paid', (coalesce(lower(nullIf(nullIf(tupleElement(argMinMerge(raw_sessions_v3.entry_channel_type_properties), 2), ''), 'null')), ''), 'medium')), coalesce(dictGetOrNull('posthog_test.channel_definition_dict', 'type_if_paid', (coalesce(tupleElement(argMinMerge(raw_sessions_v3.entry_channel_type_properties), 4), ''), 'source')), dictGetOrNull('posthog_test.channel_definition_dict', 'type_if_paid', (cutToFirstSignificantSubdomain(coalesce(tupleElement(argMinMerge(raw_sessions_v3.entry_channel_type_properties), 4), '')), 'source'))), multiIf(ifNull(equals(nullIf(nullIf(tupleElement(argMinMerge(raw_sessions_v3.entry_channel_type_properties), 7), ''), 'null'), '1'), 0), 'Paid Search', match(lower(nullIf(nullIf(tupleElement(argMinMerge(raw_sessions_v3.entry_channel_type_properties), 3), ''), 'null')), '^(.*video.*)$'), 'Paid Video', tupleElement(argMinMerge(raw_sessions_v3.entry_channel_type_properties), 6), 'Paid Social', 'Paid Unknown')), and(ifNull(equals(tupleElement(argMinMerge(raw_sessions_v3.entry_channel_type_properties), 4), '$direct'), 0), isNull(lower(nullIf(nullIf(tupleElement(argMinMerge(raw_sessions_v3.entry_channel_type_properties), 2), ''), 'null'))), or(isNull(lower(nullIf(nullIf(tupleElement(argMinMerge(raw_sessions_v3.entry_channel_type_properties), 1), ''), 'null'))), in(lower(nullIf(nullIf(tupleElement(argMinMerge(raw_sessions_v3.entry_channel_type_properties), 1), ''), 'null')), tuple('(direct)', 'direct', '$direct'))), not(tupleElement(argMinMerge(raw_sessions_v3.entry_channel_type_properties), 6))), 'Direct', coalesce(coalesce(dictGetOrNull('posthog_test.channel_definition_dict', 'type_if_organic', (coalesce(lower(nullIf(nullIf(tupleElement(argMinMerge(raw_sessions_v3.entry_channel_type_properties), 1), ''), 'null')), ''), 'source')), dictGetOrNull('posthog_test.channel_definition_dict', 'type_if_organic', (cutToFirstSignificantSubdomain(coalesce(lower(nullIf(nullIf(tupleElement(argMinMerge(raw_sessions_v3.entry_channel_type_properties), 1), ''), 'null')), '')), 'source'))), if(match(lower(nullIf(nullIf(tupleElement(argMinMerge(raw_sessions_v3.entry_channel_type_properties), 3), ''), 'null')), '^(.*(([^a-df-z]|^)shop|shopping).*)$'), 'Organic Shopping', NULL), dictGetOrNull('posthog_test.channel_definition_dict', 'type_if_organic', (coalesce(lower(nullIf(nullIf(tupleElement(argMinMerge(raw_sessions_v3.entry_channel_type_properties), 2), ''), 'null')), ''), 'medium')), coalesce(dictGetOrNull('posthog_test.channel_definition_dict', 'type_if_organic', (coalesce(tupleElement(argMinMerge(raw_sessions_v3.entry_channel_type_properties), 4), ''), 'source')), dictGetOrNull('posthog_test.channel_definition_dict', 'type_if_organic', (cutToFirstSignificantSubdomain(coalesce(tupleElement(argMinMerge(raw_sessions_v3.entry_channel_type_properties), 4), '')), 'source'))), multiIf(match(lower(nullIf(nullIf(tupleElement(argMinMerge(raw_sessions_v3.entry_channel_type_properties), 3), ''), 'null')), '^(.*video.*)$'), 'Organic Video', match(lower(nullIf(nullIf(tupleElement(argMinMerge(raw_sessions_v3.entry_channel_type_properties), 2), ''), 'null')), 'push$'), 'Push', tupleElement(argMinMerge(raw_sessions_v3.entry_channel_type_properties), 6), 'Organic Social', ifNull(equals(tupleElement(argMinMerge(raw_sessions_v3.entry_channel_type_properties), 4), '$direct'), 0), 'Direct', isNotNull(tupleElement(argMinMerge(raw_sessions_v3.entry_channel_type_properties), 4)), 'Referral', 'Unknown'))) AS `$channel_type`,
+ toString(reinterpretAsUUID(bitOr(bitShiftLeft(raw_sessions_v3.session_id_v7, 64), bitShiftRight(raw_sessions_v3.session_id_v7, 64)))) AS session_id,
+ raw_sessions_v3.session_id_v7 AS session_id_v7
+ FROM raw_sessions_v3
+ WHERE and(equals(raw_sessions_v3.team_id, 99999), equals(raw_sessions_v3.session_timestamp, fromUnixTimestamp64Milli(toUInt64(bitShiftRight(toUInt128(accurateCastOrNull('00000000-0000-0000-0000-000000000000', 'UUID')), 80)))))
+ GROUP BY raw_sessions_v3.session_id_v7) AS sessions
+ WHERE equals(sessions.session_id, '00000000-0000-0000-0000-000000000000')
+ LIMIT 100 SETTINGS readonly=2,
+ max_execution_time=60,
+ allow_experimental_object_type=1,
+ max_ast_elements=4000000,
+ max_expanded_ast_elements=4000000,
+ max_bytes_before_external_group_by=0,
+ transform_null_in=1,
+ optimize_min_equality_disjunction_chain_length=4294967295,
+ optimize_rewrite_aggregate_function_with_if=0,
+ optimize_min_inequality_conjunction_chain_length=4294967295,
+ allow_experimental_join_condition=1,
+ use_hive_partitioning=0
+ '''
+# ---
+# name: TestSessionsV3.test_channel_type_4_fbclid_without_referrer
+ '''
+ SELECT sessions.`$channel_type` AS `$channel_type`
+ FROM
+ (SELECT multiIf(match(lower(nullIf(nullIf(tupleElement(argMinMerge(raw_sessions_v3.entry_channel_type_properties), 3), ''), 'null')), 'cross-network'), 'Cross Network', or(in(lower(nullIf(nullIf(tupleElement(argMinMerge(raw_sessions_v3.entry_channel_type_properties), 2), ''), 'null')), tuple('cpc', 'cpm', 'cpv', 'cpa', 'ppc', 'retargeting')), startsWith(lower(nullIf(nullIf(tupleElement(argMinMerge(raw_sessions_v3.entry_channel_type_properties), 2), ''), 'null')), 'paid'), tupleElement(argMinMerge(raw_sessions_v3.entry_channel_type_properties), 5), isNotNull(nullIf(nullIf(tupleElement(argMinMerge(raw_sessions_v3.entry_channel_type_properties), 7), ''), 'null'))), coalesce(coalesce(dictGetOrNull('posthog_test.channel_definition_dict', 'type_if_paid', (coalesce(lower(nullIf(nullIf(tupleElement(argMinMerge(raw_sessions_v3.entry_channel_type_properties), 1), ''), 'null')), ''), 'source')), dictGetOrNull('posthog_test.channel_definition_dict', 'type_if_paid', (cutToFirstSignificantSubdomain(coalesce(lower(nullIf(nullIf(tupleElement(argMinMerge(raw_sessions_v3.entry_channel_type_properties), 1), ''), 'null')), '')), 'source'))), if(match(lower(nullIf(nullIf(tupleElement(argMinMerge(raw_sessions_v3.entry_channel_type_properties), 3), ''), 'null')), '^(.*(([^a-df-z]|^)shop|shopping).*)$'), 'Paid Shopping', NULL), dictGetOrNull('posthog_test.channel_definition_dict', 'type_if_paid', (coalesce(lower(nullIf(nullIf(tupleElement(argMinMerge(raw_sessions_v3.entry_channel_type_properties), 2), ''), 'null')), ''), 'medium')), coalesce(dictGetOrNull('posthog_test.channel_definition_dict', 'type_if_paid', (coalesce(tupleElement(argMinMerge(raw_sessions_v3.entry_channel_type_properties), 4), ''), 'source')), dictGetOrNull('posthog_test.channel_definition_dict', 'type_if_paid', (cutToFirstSignificantSubdomain(coalesce(tupleElement(argMinMerge(raw_sessions_v3.entry_channel_type_properties), 4), '')), 'source'))), multiIf(ifNull(equals(nullIf(nullIf(tupleElement(argMinMerge(raw_sessions_v3.entry_channel_type_properties), 7), ''), 'null'), '1'), 0), 'Paid Search', match(lower(nullIf(nullIf(tupleElement(argMinMerge(raw_sessions_v3.entry_channel_type_properties), 3), ''), 'null')), '^(.*video.*)$'), 'Paid Video', tupleElement(argMinMerge(raw_sessions_v3.entry_channel_type_properties), 6), 'Paid Social', 'Paid Unknown')), and(ifNull(equals(tupleElement(argMinMerge(raw_sessions_v3.entry_channel_type_properties), 4), '$direct'), 0), isNull(lower(nullIf(nullIf(tupleElement(argMinMerge(raw_sessions_v3.entry_channel_type_properties), 2), ''), 'null'))), or(isNull(lower(nullIf(nullIf(tupleElement(argMinMerge(raw_sessions_v3.entry_channel_type_properties), 1), ''), 'null'))), in(lower(nullIf(nullIf(tupleElement(argMinMerge(raw_sessions_v3.entry_channel_type_properties), 1), ''), 'null')), tuple('(direct)', 'direct', '$direct'))), not(tupleElement(argMinMerge(raw_sessions_v3.entry_channel_type_properties), 6))), 'Direct', coalesce(coalesce(dictGetOrNull('posthog_test.channel_definition_dict', 'type_if_organic', (coalesce(lower(nullIf(nullIf(tupleElement(argMinMerge(raw_sessions_v3.entry_channel_type_properties), 1), ''), 'null')), ''), 'source')), dictGetOrNull('posthog_test.channel_definition_dict', 'type_if_organic', (cutToFirstSignificantSubdomain(coalesce(lower(nullIf(nullIf(tupleElement(argMinMerge(raw_sessions_v3.entry_channel_type_properties), 1), ''), 'null')), '')), 'source'))), if(match(lower(nullIf(nullIf(tupleElement(argMinMerge(raw_sessions_v3.entry_channel_type_properties), 3), ''), 'null')), '^(.*(([^a-df-z]|^)shop|shopping).*)$'), 'Organic Shopping', NULL), dictGetOrNull('posthog_test.channel_definition_dict', 'type_if_organic', (coalesce(lower(nullIf(nullIf(tupleElement(argMinMerge(raw_sessions_v3.entry_channel_type_properties), 2), ''), 'null')), ''), 'medium')), coalesce(dictGetOrNull('posthog_test.channel_definition_dict', 'type_if_organic', (coalesce(tupleElement(argMinMerge(raw_sessions_v3.entry_channel_type_properties), 4), ''), 'source')), dictGetOrNull('posthog_test.channel_definition_dict', 'type_if_organic', (cutToFirstSignificantSubdomain(coalesce(tupleElement(argMinMerge(raw_sessions_v3.entry_channel_type_properties), 4), '')), 'source'))), multiIf(match(lower(nullIf(nullIf(tupleElement(argMinMerge(raw_sessions_v3.entry_channel_type_properties), 3), ''), 'null')), '^(.*video.*)$'), 'Organic Video', match(lower(nullIf(nullIf(tupleElement(argMinMerge(raw_sessions_v3.entry_channel_type_properties), 2), ''), 'null')), 'push$'), 'Push', tupleElement(argMinMerge(raw_sessions_v3.entry_channel_type_properties), 6), 'Organic Social', ifNull(equals(tupleElement(argMinMerge(raw_sessions_v3.entry_channel_type_properties), 4), '$direct'), 0), 'Direct', isNotNull(tupleElement(argMinMerge(raw_sessions_v3.entry_channel_type_properties), 4)), 'Referral', 'Unknown'))) AS `$channel_type`,
+ toString(reinterpretAsUUID(bitOr(bitShiftLeft(raw_sessions_v3.session_id_v7, 64), bitShiftRight(raw_sessions_v3.session_id_v7, 64)))) AS session_id,
+ raw_sessions_v3.session_id_v7 AS session_id_v7
+ FROM raw_sessions_v3
+ WHERE and(equals(raw_sessions_v3.team_id, 99999), equals(raw_sessions_v3.session_timestamp, fromUnixTimestamp64Milli(toUInt64(bitShiftRight(toUInt128(accurateCastOrNull('00000000-0000-0000-0000-000000000000', 'UUID')), 80)))))
+ GROUP BY raw_sessions_v3.session_id_v7) AS sessions
+ WHERE equals(sessions.session_id, '00000000-0000-0000-0000-000000000000')
+ LIMIT 100 SETTINGS readonly=2,
+ max_execution_time=60,
+ allow_experimental_object_type=1,
+ max_ast_elements=4000000,
+ max_expanded_ast_elements=4000000,
+ max_bytes_before_external_group_by=0,
+ transform_null_in=1,
+ optimize_min_equality_disjunction_chain_length=4294967295,
+ optimize_rewrite_aggregate_function_with_if=0,
+ optimize_min_inequality_conjunction_chain_length=4294967295,
+ allow_experimental_join_condition=1,
+ use_hive_partitioning=0
+ '''
+# ---
+# name: TestSessionsV3.test_channel_type_5_unknown_source_stripped_referrer
+ '''
+ SELECT sessions.`$channel_type` AS `$channel_type`
+ FROM
+ (SELECT multiIf(match(lower(nullIf(nullIf(tupleElement(argMinMerge(raw_sessions_v3.entry_channel_type_properties), 3), ''), 'null')), 'cross-network'), 'Cross Network', or(in(lower(nullIf(nullIf(tupleElement(argMinMerge(raw_sessions_v3.entry_channel_type_properties), 2), ''), 'null')), tuple('cpc', 'cpm', 'cpv', 'cpa', 'ppc', 'retargeting')), startsWith(lower(nullIf(nullIf(tupleElement(argMinMerge(raw_sessions_v3.entry_channel_type_properties), 2), ''), 'null')), 'paid'), tupleElement(argMinMerge(raw_sessions_v3.entry_channel_type_properties), 5), isNotNull(nullIf(nullIf(tupleElement(argMinMerge(raw_sessions_v3.entry_channel_type_properties), 7), ''), 'null'))), coalesce(coalesce(dictGetOrNull('posthog_test.channel_definition_dict', 'type_if_paid', (coalesce(lower(nullIf(nullIf(tupleElement(argMinMerge(raw_sessions_v3.entry_channel_type_properties), 1), ''), 'null')), ''), 'source')), dictGetOrNull('posthog_test.channel_definition_dict', 'type_if_paid', (cutToFirstSignificantSubdomain(coalesce(lower(nullIf(nullIf(tupleElement(argMinMerge(raw_sessions_v3.entry_channel_type_properties), 1), ''), 'null')), '')), 'source'))), if(match(lower(nullIf(nullIf(tupleElement(argMinMerge(raw_sessions_v3.entry_channel_type_properties), 3), ''), 'null')), '^(.*(([^a-df-z]|^)shop|shopping).*)$'), 'Paid Shopping', NULL), dictGetOrNull('posthog_test.channel_definition_dict', 'type_if_paid', (coalesce(lower(nullIf(nullIf(tupleElement(argMinMerge(raw_sessions_v3.entry_channel_type_properties), 2), ''), 'null')), ''), 'medium')), coalesce(dictGetOrNull('posthog_test.channel_definition_dict', 'type_if_paid', (coalesce(tupleElement(argMinMerge(raw_sessions_v3.entry_channel_type_properties), 4), ''), 'source')), dictGetOrNull('posthog_test.channel_definition_dict', 'type_if_paid', (cutToFirstSignificantSubdomain(coalesce(tupleElement(argMinMerge(raw_sessions_v3.entry_channel_type_properties), 4), '')), 'source'))), multiIf(ifNull(equals(nullIf(nullIf(tupleElement(argMinMerge(raw_sessions_v3.entry_channel_type_properties), 7), ''), 'null'), '1'), 0), 'Paid Search', match(lower(nullIf(nullIf(tupleElement(argMinMerge(raw_sessions_v3.entry_channel_type_properties), 3), ''), 'null')), '^(.*video.*)$'), 'Paid Video', tupleElement(argMinMerge(raw_sessions_v3.entry_channel_type_properties), 6), 'Paid Social', 'Paid Unknown')), and(ifNull(equals(tupleElement(argMinMerge(raw_sessions_v3.entry_channel_type_properties), 4), '$direct'), 0), isNull(lower(nullIf(nullIf(tupleElement(argMinMerge(raw_sessions_v3.entry_channel_type_properties), 2), ''), 'null'))), or(isNull(lower(nullIf(nullIf(tupleElement(argMinMerge(raw_sessions_v3.entry_channel_type_properties), 1), ''), 'null'))), in(lower(nullIf(nullIf(tupleElement(argMinMerge(raw_sessions_v3.entry_channel_type_properties), 1), ''), 'null')), tuple('(direct)', 'direct', '$direct'))), not(tupleElement(argMinMerge(raw_sessions_v3.entry_channel_type_properties), 6))), 'Direct', coalesce(coalesce(dictGetOrNull('posthog_test.channel_definition_dict', 'type_if_organic', (coalesce(lower(nullIf(nullIf(tupleElement(argMinMerge(raw_sessions_v3.entry_channel_type_properties), 1), ''), 'null')), ''), 'source')), dictGetOrNull('posthog_test.channel_definition_dict', 'type_if_organic', (cutToFirstSignificantSubdomain(coalesce(lower(nullIf(nullIf(tupleElement(argMinMerge(raw_sessions_v3.entry_channel_type_properties), 1), ''), 'null')), '')), 'source'))), if(match(lower(nullIf(nullIf(tupleElement(argMinMerge(raw_sessions_v3.entry_channel_type_properties), 3), ''), 'null')), '^(.*(([^a-df-z]|^)shop|shopping).*)$'), 'Organic Shopping', NULL), dictGetOrNull('posthog_test.channel_definition_dict', 'type_if_organic', (coalesce(lower(nullIf(nullIf(tupleElement(argMinMerge(raw_sessions_v3.entry_channel_type_properties), 2), ''), 'null')), ''), 'medium')), coalesce(dictGetOrNull('posthog_test.channel_definition_dict', 'type_if_organic', (coalesce(tupleElement(argMinMerge(raw_sessions_v3.entry_channel_type_properties), 4), ''), 'source')), dictGetOrNull('posthog_test.channel_definition_dict', 'type_if_organic', (cutToFirstSignificantSubdomain(coalesce(tupleElement(argMinMerge(raw_sessions_v3.entry_channel_type_properties), 4), '')), 'source'))), multiIf(match(lower(nullIf(nullIf(tupleElement(argMinMerge(raw_sessions_v3.entry_channel_type_properties), 3), ''), 'null')), '^(.*video.*)$'), 'Organic Video', match(lower(nullIf(nullIf(tupleElement(argMinMerge(raw_sessions_v3.entry_channel_type_properties), 2), ''), 'null')), 'push$'), 'Push', tupleElement(argMinMerge(raw_sessions_v3.entry_channel_type_properties), 6), 'Organic Social', ifNull(equals(tupleElement(argMinMerge(raw_sessions_v3.entry_channel_type_properties), 4), '$direct'), 0), 'Direct', isNotNull(tupleElement(argMinMerge(raw_sessions_v3.entry_channel_type_properties), 4)), 'Referral', 'Unknown'))) AS `$channel_type`,
+ toString(reinterpretAsUUID(bitOr(bitShiftLeft(raw_sessions_v3.session_id_v7, 64), bitShiftRight(raw_sessions_v3.session_id_v7, 64)))) AS session_id,
+ raw_sessions_v3.session_id_v7 AS session_id_v7
+ FROM raw_sessions_v3
+ WHERE and(equals(raw_sessions_v3.team_id, 99999), equals(raw_sessions_v3.session_timestamp, fromUnixTimestamp64Milli(toUInt64(bitShiftRight(toUInt128(accurateCastOrNull('00000000-0000-0000-0000-000000000000', 'UUID')), 80)))))
+ GROUP BY raw_sessions_v3.session_id_v7) AS sessions
+ WHERE equals(sessions.session_id, '00000000-0000-0000-0000-000000000000')
+ LIMIT 100 SETTINGS readonly=2,
+ max_execution_time=60,
+ allow_experimental_object_type=1,
+ max_ast_elements=4000000,
+ max_expanded_ast_elements=4000000,
+ max_bytes_before_external_group_by=0,
+ transform_null_in=1,
+ optimize_min_equality_disjunction_chain_length=4294967295,
+ optimize_rewrite_aggregate_function_with_if=0,
+ optimize_min_inequality_conjunction_chain_length=4294967295,
+ allow_experimental_join_condition=1,
+ use_hive_partitioning=0
+ '''
+# ---
+# name: TestSessionsV3.test_channel_type_6_unknown_referrer
+ '''
+ SELECT sessions.`$channel_type` AS `$channel_type`
+ FROM
+ (SELECT multiIf(match(lower(nullIf(nullIf(tupleElement(argMinMerge(raw_sessions_v3.entry_channel_type_properties), 3), ''), 'null')), 'cross-network'), 'Cross Network', or(in(lower(nullIf(nullIf(tupleElement(argMinMerge(raw_sessions_v3.entry_channel_type_properties), 2), ''), 'null')), tuple('cpc', 'cpm', 'cpv', 'cpa', 'ppc', 'retargeting')), startsWith(lower(nullIf(nullIf(tupleElement(argMinMerge(raw_sessions_v3.entry_channel_type_properties), 2), ''), 'null')), 'paid'), tupleElement(argMinMerge(raw_sessions_v3.entry_channel_type_properties), 5), isNotNull(nullIf(nullIf(tupleElement(argMinMerge(raw_sessions_v3.entry_channel_type_properties), 7), ''), 'null'))), coalesce(coalesce(dictGetOrNull('posthog_test.channel_definition_dict', 'type_if_paid', (coalesce(lower(nullIf(nullIf(tupleElement(argMinMerge(raw_sessions_v3.entry_channel_type_properties), 1), ''), 'null')), ''), 'source')), dictGetOrNull('posthog_test.channel_definition_dict', 'type_if_paid', (cutToFirstSignificantSubdomain(coalesce(lower(nullIf(nullIf(tupleElement(argMinMerge(raw_sessions_v3.entry_channel_type_properties), 1), ''), 'null')), '')), 'source'))), if(match(lower(nullIf(nullIf(tupleElement(argMinMerge(raw_sessions_v3.entry_channel_type_properties), 3), ''), 'null')), '^(.*(([^a-df-z]|^)shop|shopping).*)$'), 'Paid Shopping', NULL), dictGetOrNull('posthog_test.channel_definition_dict', 'type_if_paid', (coalesce(lower(nullIf(nullIf(tupleElement(argMinMerge(raw_sessions_v3.entry_channel_type_properties), 2), ''), 'null')), ''), 'medium')), coalesce(dictGetOrNull('posthog_test.channel_definition_dict', 'type_if_paid', (coalesce(tupleElement(argMinMerge(raw_sessions_v3.entry_channel_type_properties), 4), ''), 'source')), dictGetOrNull('posthog_test.channel_definition_dict', 'type_if_paid', (cutToFirstSignificantSubdomain(coalesce(tupleElement(argMinMerge(raw_sessions_v3.entry_channel_type_properties), 4), '')), 'source'))), multiIf(ifNull(equals(nullIf(nullIf(tupleElement(argMinMerge(raw_sessions_v3.entry_channel_type_properties), 7), ''), 'null'), '1'), 0), 'Paid Search', match(lower(nullIf(nullIf(tupleElement(argMinMerge(raw_sessions_v3.entry_channel_type_properties), 3), ''), 'null')), '^(.*video.*)$'), 'Paid Video', tupleElement(argMinMerge(raw_sessions_v3.entry_channel_type_properties), 6), 'Paid Social', 'Paid Unknown')), and(ifNull(equals(tupleElement(argMinMerge(raw_sessions_v3.entry_channel_type_properties), 4), '$direct'), 0), isNull(lower(nullIf(nullIf(tupleElement(argMinMerge(raw_sessions_v3.entry_channel_type_properties), 2), ''), 'null'))), or(isNull(lower(nullIf(nullIf(tupleElement(argMinMerge(raw_sessions_v3.entry_channel_type_properties), 1), ''), 'null'))), in(lower(nullIf(nullIf(tupleElement(argMinMerge(raw_sessions_v3.entry_channel_type_properties), 1), ''), 'null')), tuple('(direct)', 'direct', '$direct'))), not(tupleElement(argMinMerge(raw_sessions_v3.entry_channel_type_properties), 6))), 'Direct', coalesce(coalesce(dictGetOrNull('posthog_test.channel_definition_dict', 'type_if_organic', (coalesce(lower(nullIf(nullIf(tupleElement(argMinMerge(raw_sessions_v3.entry_channel_type_properties), 1), ''), 'null')), ''), 'source')), dictGetOrNull('posthog_test.channel_definition_dict', 'type_if_organic', (cutToFirstSignificantSubdomain(coalesce(lower(nullIf(nullIf(tupleElement(argMinMerge(raw_sessions_v3.entry_channel_type_properties), 1), ''), 'null')), '')), 'source'))), if(match(lower(nullIf(nullIf(tupleElement(argMinMerge(raw_sessions_v3.entry_channel_type_properties), 3), ''), 'null')), '^(.*(([^a-df-z]|^)shop|shopping).*)$'), 'Organic Shopping', NULL), dictGetOrNull('posthog_test.channel_definition_dict', 'type_if_organic', (coalesce(lower(nullIf(nullIf(tupleElement(argMinMerge(raw_sessions_v3.entry_channel_type_properties), 2), ''), 'null')), ''), 'medium')), coalesce(dictGetOrNull('posthog_test.channel_definition_dict', 'type_if_organic', (coalesce(tupleElement(argMinMerge(raw_sessions_v3.entry_channel_type_properties), 4), ''), 'source')), dictGetOrNull('posthog_test.channel_definition_dict', 'type_if_organic', (cutToFirstSignificantSubdomain(coalesce(tupleElement(argMinMerge(raw_sessions_v3.entry_channel_type_properties), 4), '')), 'source'))), multiIf(match(lower(nullIf(nullIf(tupleElement(argMinMerge(raw_sessions_v3.entry_channel_type_properties), 3), ''), 'null')), '^(.*video.*)$'), 'Organic Video', match(lower(nullIf(nullIf(tupleElement(argMinMerge(raw_sessions_v3.entry_channel_type_properties), 2), ''), 'null')), 'push$'), 'Push', tupleElement(argMinMerge(raw_sessions_v3.entry_channel_type_properties), 6), 'Organic Social', ifNull(equals(tupleElement(argMinMerge(raw_sessions_v3.entry_channel_type_properties), 4), '$direct'), 0), 'Direct', isNotNull(tupleElement(argMinMerge(raw_sessions_v3.entry_channel_type_properties), 4)), 'Referral', 'Unknown'))) AS `$channel_type`,
+ toString(reinterpretAsUUID(bitOr(bitShiftLeft(raw_sessions_v3.session_id_v7, 64), bitShiftRight(raw_sessions_v3.session_id_v7, 64)))) AS session_id,
+ raw_sessions_v3.session_id_v7 AS session_id_v7
+ FROM raw_sessions_v3
+ WHERE and(equals(raw_sessions_v3.team_id, 99999), equals(raw_sessions_v3.session_timestamp, fromUnixTimestamp64Milli(toUInt64(bitShiftRight(toUInt128(accurateCastOrNull('00000000-0000-0000-0000-000000000000', 'UUID')), 80)))))
+ GROUP BY raw_sessions_v3.session_id_v7) AS sessions
+ WHERE equals(sessions.session_id, '00000000-0000-0000-0000-000000000000')
+ LIMIT 100 SETTINGS readonly=2,
+ max_execution_time=60,
+ allow_experimental_object_type=1,
+ max_ast_elements=4000000,
+ max_expanded_ast_elements=4000000,
+ max_bytes_before_external_group_by=0,
+ transform_null_in=1,
+ optimize_min_equality_disjunction_chain_length=4294967295,
+ optimize_rewrite_aggregate_function_with_if=0,
+ optimize_min_inequality_conjunction_chain_length=4294967295,
+ allow_experimental_join_condition=1,
+ use_hive_partitioning=0
+ '''
+# ---
+# name: TestSessionsV3.test_channel_type_7_self_referral_with_paid_utms
+ '''
+ SELECT sessions.`$channel_type` AS `$channel_type`
+ FROM
+ (SELECT multiIf(match(lower(nullIf(nullIf(tupleElement(argMinMerge(raw_sessions_v3.entry_channel_type_properties), 3), ''), 'null')), 'cross-network'), 'Cross Network', or(in(lower(nullIf(nullIf(tupleElement(argMinMerge(raw_sessions_v3.entry_channel_type_properties), 2), ''), 'null')), tuple('cpc', 'cpm', 'cpv', 'cpa', 'ppc', 'retargeting')), startsWith(lower(nullIf(nullIf(tupleElement(argMinMerge(raw_sessions_v3.entry_channel_type_properties), 2), ''), 'null')), 'paid'), tupleElement(argMinMerge(raw_sessions_v3.entry_channel_type_properties), 5), isNotNull(nullIf(nullIf(tupleElement(argMinMerge(raw_sessions_v3.entry_channel_type_properties), 7), ''), 'null'))), coalesce(coalesce(dictGetOrNull('posthog_test.channel_definition_dict', 'type_if_paid', (coalesce(lower(nullIf(nullIf(tupleElement(argMinMerge(raw_sessions_v3.entry_channel_type_properties), 1), ''), 'null')), ''), 'source')), dictGetOrNull('posthog_test.channel_definition_dict', 'type_if_paid', (cutToFirstSignificantSubdomain(coalesce(lower(nullIf(nullIf(tupleElement(argMinMerge(raw_sessions_v3.entry_channel_type_properties), 1), ''), 'null')), '')), 'source'))), if(match(lower(nullIf(nullIf(tupleElement(argMinMerge(raw_sessions_v3.entry_channel_type_properties), 3), ''), 'null')), '^(.*(([^a-df-z]|^)shop|shopping).*)$'), 'Paid Shopping', NULL), dictGetOrNull('posthog_test.channel_definition_dict', 'type_if_paid', (coalesce(lower(nullIf(nullIf(tupleElement(argMinMerge(raw_sessions_v3.entry_channel_type_properties), 2), ''), 'null')), ''), 'medium')), coalesce(dictGetOrNull('posthog_test.channel_definition_dict', 'type_if_paid', (coalesce(tupleElement(argMinMerge(raw_sessions_v3.entry_channel_type_properties), 4), ''), 'source')), dictGetOrNull('posthog_test.channel_definition_dict', 'type_if_paid', (cutToFirstSignificantSubdomain(coalesce(tupleElement(argMinMerge(raw_sessions_v3.entry_channel_type_properties), 4), '')), 'source'))), multiIf(ifNull(equals(nullIf(nullIf(tupleElement(argMinMerge(raw_sessions_v3.entry_channel_type_properties), 7), ''), 'null'), '1'), 0), 'Paid Search', match(lower(nullIf(nullIf(tupleElement(argMinMerge(raw_sessions_v3.entry_channel_type_properties), 3), ''), 'null')), '^(.*video.*)$'), 'Paid Video', tupleElement(argMinMerge(raw_sessions_v3.entry_channel_type_properties), 6), 'Paid Social', 'Paid Unknown')), and(ifNull(equals(tupleElement(argMinMerge(raw_sessions_v3.entry_channel_type_properties), 4), '$direct'), 0), isNull(lower(nullIf(nullIf(tupleElement(argMinMerge(raw_sessions_v3.entry_channel_type_properties), 2), ''), 'null'))), or(isNull(lower(nullIf(nullIf(tupleElement(argMinMerge(raw_sessions_v3.entry_channel_type_properties), 1), ''), 'null'))), in(lower(nullIf(nullIf(tupleElement(argMinMerge(raw_sessions_v3.entry_channel_type_properties), 1), ''), 'null')), tuple('(direct)', 'direct', '$direct'))), not(tupleElement(argMinMerge(raw_sessions_v3.entry_channel_type_properties), 6))), 'Direct', coalesce(coalesce(dictGetOrNull('posthog_test.channel_definition_dict', 'type_if_organic', (coalesce(lower(nullIf(nullIf(tupleElement(argMinMerge(raw_sessions_v3.entry_channel_type_properties), 1), ''), 'null')), ''), 'source')), dictGetOrNull('posthog_test.channel_definition_dict', 'type_if_organic', (cutToFirstSignificantSubdomain(coalesce(lower(nullIf(nullIf(tupleElement(argMinMerge(raw_sessions_v3.entry_channel_type_properties), 1), ''), 'null')), '')), 'source'))), if(match(lower(nullIf(nullIf(tupleElement(argMinMerge(raw_sessions_v3.entry_channel_type_properties), 3), ''), 'null')), '^(.*(([^a-df-z]|^)shop|shopping).*)$'), 'Organic Shopping', NULL), dictGetOrNull('posthog_test.channel_definition_dict', 'type_if_organic', (coalesce(lower(nullIf(nullIf(tupleElement(argMinMerge(raw_sessions_v3.entry_channel_type_properties), 2), ''), 'null')), ''), 'medium')), coalesce(dictGetOrNull('posthog_test.channel_definition_dict', 'type_if_organic', (coalesce(tupleElement(argMinMerge(raw_sessions_v3.entry_channel_type_properties), 4), ''), 'source')), dictGetOrNull('posthog_test.channel_definition_dict', 'type_if_organic', (cutToFirstSignificantSubdomain(coalesce(tupleElement(argMinMerge(raw_sessions_v3.entry_channel_type_properties), 4), '')), 'source'))), multiIf(match(lower(nullIf(nullIf(tupleElement(argMinMerge(raw_sessions_v3.entry_channel_type_properties), 3), ''), 'null')), '^(.*video.*)$'), 'Organic Video', match(lower(nullIf(nullIf(tupleElement(argMinMerge(raw_sessions_v3.entry_channel_type_properties), 2), ''), 'null')), 'push$'), 'Push', tupleElement(argMinMerge(raw_sessions_v3.entry_channel_type_properties), 6), 'Organic Social', ifNull(equals(tupleElement(argMinMerge(raw_sessions_v3.entry_channel_type_properties), 4), '$direct'), 0), 'Direct', isNotNull(tupleElement(argMinMerge(raw_sessions_v3.entry_channel_type_properties), 4)), 'Referral', 'Unknown'))) AS `$channel_type`,
+ toString(reinterpretAsUUID(bitOr(bitShiftLeft(raw_sessions_v3.session_id_v7, 64), bitShiftRight(raw_sessions_v3.session_id_v7, 64)))) AS session_id,
+ raw_sessions_v3.session_id_v7 AS session_id_v7
+ FROM raw_sessions_v3
+ WHERE and(equals(raw_sessions_v3.team_id, 99999), equals(raw_sessions_v3.session_timestamp, fromUnixTimestamp64Milli(toUInt64(bitShiftRight(toUInt128(accurateCastOrNull('00000000-0000-0000-0000-000000000000', 'UUID')), 80)))))
+ GROUP BY raw_sessions_v3.session_id_v7) AS sessions
+ WHERE equals(sessions.session_id, '00000000-0000-0000-0000-000000000000')
+ LIMIT 100 SETTINGS readonly=2,
+ max_execution_time=60,
+ allow_experimental_object_type=1,
+ max_ast_elements=4000000,
+ max_expanded_ast_elements=4000000,
+ max_bytes_before_external_group_by=0,
+ transform_null_in=1,
+ optimize_min_equality_disjunction_chain_length=4294967295,
+ optimize_rewrite_aggregate_function_with_if=0,
+ optimize_min_inequality_conjunction_chain_length=4294967295,
+ allow_experimental_join_condition=1,
+ use_hive_partitioning=0
+ '''
+# ---
+# name: TestSessionsV3.test_channel_type_8_self_referral_with_unknown_source
+ '''
+ SELECT sessions.`$channel_type` AS `$channel_type`
+ FROM
+ (SELECT multiIf(match(lower(nullIf(nullIf(tupleElement(argMinMerge(raw_sessions_v3.entry_channel_type_properties), 3), ''), 'null')), 'cross-network'), 'Cross Network', or(in(lower(nullIf(nullIf(tupleElement(argMinMerge(raw_sessions_v3.entry_channel_type_properties), 2), ''), 'null')), tuple('cpc', 'cpm', 'cpv', 'cpa', 'ppc', 'retargeting')), startsWith(lower(nullIf(nullIf(tupleElement(argMinMerge(raw_sessions_v3.entry_channel_type_properties), 2), ''), 'null')), 'paid'), tupleElement(argMinMerge(raw_sessions_v3.entry_channel_type_properties), 5), isNotNull(nullIf(nullIf(tupleElement(argMinMerge(raw_sessions_v3.entry_channel_type_properties), 7), ''), 'null'))), coalesce(coalesce(dictGetOrNull('posthog_test.channel_definition_dict', 'type_if_paid', (coalesce(lower(nullIf(nullIf(tupleElement(argMinMerge(raw_sessions_v3.entry_channel_type_properties), 1), ''), 'null')), ''), 'source')), dictGetOrNull('posthog_test.channel_definition_dict', 'type_if_paid', (cutToFirstSignificantSubdomain(coalesce(lower(nullIf(nullIf(tupleElement(argMinMerge(raw_sessions_v3.entry_channel_type_properties), 1), ''), 'null')), '')), 'source'))), if(match(lower(nullIf(nullIf(tupleElement(argMinMerge(raw_sessions_v3.entry_channel_type_properties), 3), ''), 'null')), '^(.*(([^a-df-z]|^)shop|shopping).*)$'), 'Paid Shopping', NULL), dictGetOrNull('posthog_test.channel_definition_dict', 'type_if_paid', (coalesce(lower(nullIf(nullIf(tupleElement(argMinMerge(raw_sessions_v3.entry_channel_type_properties), 2), ''), 'null')), ''), 'medium')), coalesce(dictGetOrNull('posthog_test.channel_definition_dict', 'type_if_paid', (coalesce(tupleElement(argMinMerge(raw_sessions_v3.entry_channel_type_properties), 4), ''), 'source')), dictGetOrNull('posthog_test.channel_definition_dict', 'type_if_paid', (cutToFirstSignificantSubdomain(coalesce(tupleElement(argMinMerge(raw_sessions_v3.entry_channel_type_properties), 4), '')), 'source'))), multiIf(ifNull(equals(nullIf(nullIf(tupleElement(argMinMerge(raw_sessions_v3.entry_channel_type_properties), 7), ''), 'null'), '1'), 0), 'Paid Search', match(lower(nullIf(nullIf(tupleElement(argMinMerge(raw_sessions_v3.entry_channel_type_properties), 3), ''), 'null')), '^(.*video.*)$'), 'Paid Video', tupleElement(argMinMerge(raw_sessions_v3.entry_channel_type_properties), 6), 'Paid Social', 'Paid Unknown')), and(ifNull(equals(tupleElement(argMinMerge(raw_sessions_v3.entry_channel_type_properties), 4), '$direct'), 0), isNull(lower(nullIf(nullIf(tupleElement(argMinMerge(raw_sessions_v3.entry_channel_type_properties), 2), ''), 'null'))), or(isNull(lower(nullIf(nullIf(tupleElement(argMinMerge(raw_sessions_v3.entry_channel_type_properties), 1), ''), 'null'))), in(lower(nullIf(nullIf(tupleElement(argMinMerge(raw_sessions_v3.entry_channel_type_properties), 1), ''), 'null')), tuple('(direct)', 'direct', '$direct'))), not(tupleElement(argMinMerge(raw_sessions_v3.entry_channel_type_properties), 6))), 'Direct', coalesce(coalesce(dictGetOrNull('posthog_test.channel_definition_dict', 'type_if_organic', (coalesce(lower(nullIf(nullIf(tupleElement(argMinMerge(raw_sessions_v3.entry_channel_type_properties), 1), ''), 'null')), ''), 'source')), dictGetOrNull('posthog_test.channel_definition_dict', 'type_if_organic', (cutToFirstSignificantSubdomain(coalesce(lower(nullIf(nullIf(tupleElement(argMinMerge(raw_sessions_v3.entry_channel_type_properties), 1), ''), 'null')), '')), 'source'))), if(match(lower(nullIf(nullIf(tupleElement(argMinMerge(raw_sessions_v3.entry_channel_type_properties), 3), ''), 'null')), '^(.*(([^a-df-z]|^)shop|shopping).*)$'), 'Organic Shopping', NULL), dictGetOrNull('posthog_test.channel_definition_dict', 'type_if_organic', (coalesce(lower(nullIf(nullIf(tupleElement(argMinMerge(raw_sessions_v3.entry_channel_type_properties), 2), ''), 'null')), ''), 'medium')), coalesce(dictGetOrNull('posthog_test.channel_definition_dict', 'type_if_organic', (coalesce(tupleElement(argMinMerge(raw_sessions_v3.entry_channel_type_properties), 4), ''), 'source')), dictGetOrNull('posthog_test.channel_definition_dict', 'type_if_organic', (cutToFirstSignificantSubdomain(coalesce(tupleElement(argMinMerge(raw_sessions_v3.entry_channel_type_properties), 4), '')), 'source'))), multiIf(match(lower(nullIf(nullIf(tupleElement(argMinMerge(raw_sessions_v3.entry_channel_type_properties), 3), ''), 'null')), '^(.*video.*)$'), 'Organic Video', match(lower(nullIf(nullIf(tupleElement(argMinMerge(raw_sessions_v3.entry_channel_type_properties), 2), ''), 'null')), 'push$'), 'Push', tupleElement(argMinMerge(raw_sessions_v3.entry_channel_type_properties), 6), 'Organic Social', ifNull(equals(tupleElement(argMinMerge(raw_sessions_v3.entry_channel_type_properties), 4), '$direct'), 0), 'Direct', isNotNull(tupleElement(argMinMerge(raw_sessions_v3.entry_channel_type_properties), 4)), 'Referral', 'Unknown'))) AS `$channel_type`,
+ toString(reinterpretAsUUID(bitOr(bitShiftLeft(raw_sessions_v3.session_id_v7, 64), bitShiftRight(raw_sessions_v3.session_id_v7, 64)))) AS session_id,
+ raw_sessions_v3.session_id_v7 AS session_id_v7
+ FROM raw_sessions_v3
+ WHERE and(equals(raw_sessions_v3.team_id, 99999), equals(raw_sessions_v3.session_timestamp, fromUnixTimestamp64Milli(toUInt64(bitShiftRight(toUInt128(accurateCastOrNull('00000000-0000-0000-0000-000000000000', 'UUID')), 80)))))
+ GROUP BY raw_sessions_v3.session_id_v7) AS sessions
+ WHERE equals(sessions.session_id, '00000000-0000-0000-0000-000000000000')
+ LIMIT 100 SETTINGS readonly=2,
+ max_execution_time=60,
+ allow_experimental_object_type=1,
+ max_ast_elements=4000000,
+ max_expanded_ast_elements=4000000,
+ max_bytes_before_external_group_by=0,
+ transform_null_in=1,
+ optimize_min_equality_disjunction_chain_length=4294967295,
+ optimize_rewrite_aggregate_function_with_if=0,
+ optimize_min_inequality_conjunction_chain_length=4294967295,
+ allow_experimental_join_condition=1,
+ use_hive_partitioning=0
+ '''
+# ---
+# name: TestSessionsV3.test_channel_type_9_msclkid_only_in_url
'''
SELECT sessions.`$channel_type` AS `$channel_type`
FROM
@@ -91,9 +316,9 @@
sessions.`$autocapture_count` AS `$autocapture_count`,
sessions.`$screen_count` AS `$screen_count`
FROM
- (SELECT uniqExactMerge(raw_sessions_v3.pageview_uniq) AS `$pageview_count`,
- uniqExactMerge(raw_sessions_v3.autocapture_uniq) AS `$autocapture_count`,
- uniqExactMerge(raw_sessions_v3.screen_uniq) AS `$screen_count`,
+ (SELECT uniqMerge(raw_sessions_v3.pageview_uniq) AS `$pageview_count`,
+ uniqMerge(raw_sessions_v3.autocapture_uniq) AS `$autocapture_count`,
+ uniqMerge(raw_sessions_v3.screen_uniq) AS `$screen_count`,
toString(reinterpretAsUUID(bitOr(bitShiftLeft(raw_sessions_v3.session_id_v7, 64), bitShiftRight(raw_sessions_v3.session_id_v7, 64)))) AS id,
raw_sessions_v3.session_id_v7 AS session_id_v7
FROM raw_sessions_v3
@@ -198,9 +423,9 @@
sessions.`$autocapture_count` AS `$autocapture_count`,
sessions.`$screen_count` AS `$screen_count`
FROM
- (SELECT uniqExactMerge(raw_sessions_v3.pageview_uniq) AS `$pageview_count`,
- uniqExactMerge(raw_sessions_v3.autocapture_uniq) AS `$autocapture_count`,
- uniqExactMerge(raw_sessions_v3.screen_uniq) AS `$screen_count`,
+ (SELECT uniqMerge(raw_sessions_v3.pageview_uniq) AS `$pageview_count`,
+ uniqMerge(raw_sessions_v3.autocapture_uniq) AS `$autocapture_count`,
+ uniqMerge(raw_sessions_v3.screen_uniq) AS `$screen_count`,
toString(reinterpretAsUUID(bitOr(bitShiftLeft(raw_sessions_v3.session_id_v7, 64), bitShiftRight(raw_sessions_v3.session_id_v7, 64)))) AS id,
raw_sessions_v3.session_id_v7 AS session_id_v7
FROM raw_sessions_v3
@@ -221,6 +446,151 @@
use_hive_partitioning=0
'''
# ---
+# name: TestSessionsV3.test_entry_utm_and_channel_type_across_events_0_utm_only_in_url
+ '''
+ SELECT sessions.`$entry_utm_source` AS `$entry_utm_source`,
+ sessions.`$entry_utm_campaign` AS `$entry_utm_campaign`,
+ sessions.`$channel_type` AS `$channel_type`
+ FROM
+ (SELECT argMinMerge(raw_sessions_v3.entry_utm_source) AS `$entry_utm_source`,
+ argMinMerge(raw_sessions_v3.entry_utm_campaign) AS `$entry_utm_campaign`,
+ multiIf(match(lower(nullIf(nullIf(tupleElement(argMinMerge(raw_sessions_v3.entry_channel_type_properties), 3), ''), 'null')), 'cross-network'), 'Cross Network', or(in(lower(nullIf(nullIf(tupleElement(argMinMerge(raw_sessions_v3.entry_channel_type_properties), 2), ''), 'null')), tuple('cpc', 'cpm', 'cpv', 'cpa', 'ppc', 'retargeting')), startsWith(lower(nullIf(nullIf(tupleElement(argMinMerge(raw_sessions_v3.entry_channel_type_properties), 2), ''), 'null')), 'paid'), tupleElement(argMinMerge(raw_sessions_v3.entry_channel_type_properties), 5), isNotNull(nullIf(nullIf(tupleElement(argMinMerge(raw_sessions_v3.entry_channel_type_properties), 7), ''), 'null'))), coalesce(coalesce(dictGetOrNull('posthog_test.channel_definition_dict', 'type_if_paid', (coalesce(lower(nullIf(nullIf(tupleElement(argMinMerge(raw_sessions_v3.entry_channel_type_properties), 1), ''), 'null')), ''), 'source')), dictGetOrNull('posthog_test.channel_definition_dict', 'type_if_paid', (cutToFirstSignificantSubdomain(coalesce(lower(nullIf(nullIf(tupleElement(argMinMerge(raw_sessions_v3.entry_channel_type_properties), 1), ''), 'null')), '')), 'source'))), if(match(lower(nullIf(nullIf(tupleElement(argMinMerge(raw_sessions_v3.entry_channel_type_properties), 3), ''), 'null')), '^(.*(([^a-df-z]|^)shop|shopping).*)$'), 'Paid Shopping', NULL), dictGetOrNull('posthog_test.channel_definition_dict', 'type_if_paid', (coalesce(lower(nullIf(nullIf(tupleElement(argMinMerge(raw_sessions_v3.entry_channel_type_properties), 2), ''), 'null')), ''), 'medium')), coalesce(dictGetOrNull('posthog_test.channel_definition_dict', 'type_if_paid', (coalesce(tupleElement(argMinMerge(raw_sessions_v3.entry_channel_type_properties), 4), ''), 'source')), dictGetOrNull('posthog_test.channel_definition_dict', 'type_if_paid', (cutToFirstSignificantSubdomain(coalesce(tupleElement(argMinMerge(raw_sessions_v3.entry_channel_type_properties), 4), '')), 'source'))), multiIf(ifNull(equals(nullIf(nullIf(tupleElement(argMinMerge(raw_sessions_v3.entry_channel_type_properties), 7), ''), 'null'), '1'), 0), 'Paid Search', match(lower(nullIf(nullIf(tupleElement(argMinMerge(raw_sessions_v3.entry_channel_type_properties), 3), ''), 'null')), '^(.*video.*)$'), 'Paid Video', tupleElement(argMinMerge(raw_sessions_v3.entry_channel_type_properties), 6), 'Paid Social', 'Paid Unknown')), and(ifNull(equals(tupleElement(argMinMerge(raw_sessions_v3.entry_channel_type_properties), 4), '$direct'), 0), isNull(lower(nullIf(nullIf(tupleElement(argMinMerge(raw_sessions_v3.entry_channel_type_properties), 2), ''), 'null'))), or(isNull(lower(nullIf(nullIf(tupleElement(argMinMerge(raw_sessions_v3.entry_channel_type_properties), 1), ''), 'null'))), in(lower(nullIf(nullIf(tupleElement(argMinMerge(raw_sessions_v3.entry_channel_type_properties), 1), ''), 'null')), tuple('(direct)', 'direct', '$direct'))), not(tupleElement(argMinMerge(raw_sessions_v3.entry_channel_type_properties), 6))), 'Direct', coalesce(coalesce(dictGetOrNull('posthog_test.channel_definition_dict', 'type_if_organic', (coalesce(lower(nullIf(nullIf(tupleElement(argMinMerge(raw_sessions_v3.entry_channel_type_properties), 1), ''), 'null')), ''), 'source')), dictGetOrNull('posthog_test.channel_definition_dict', 'type_if_organic', (cutToFirstSignificantSubdomain(coalesce(lower(nullIf(nullIf(tupleElement(argMinMerge(raw_sessions_v3.entry_channel_type_properties), 1), ''), 'null')), '')), 'source'))), if(match(lower(nullIf(nullIf(tupleElement(argMinMerge(raw_sessions_v3.entry_channel_type_properties), 3), ''), 'null')), '^(.*(([^a-df-z]|^)shop|shopping).*)$'), 'Organic Shopping', NULL), dictGetOrNull('posthog_test.channel_definition_dict', 'type_if_organic', (coalesce(lower(nullIf(nullIf(tupleElement(argMinMerge(raw_sessions_v3.entry_channel_type_properties), 2), ''), 'null')), ''), 'medium')), coalesce(dictGetOrNull('posthog_test.channel_definition_dict', 'type_if_organic', (coalesce(tupleElement(argMinMerge(raw_sessions_v3.entry_channel_type_properties), 4), ''), 'source')), dictGetOrNull('posthog_test.channel_definition_dict', 'type_if_organic', (cutToFirstSignificantSubdomain(coalesce(tupleElement(argMinMerge(raw_sessions_v3.entry_channel_type_properties), 4), '')), 'source'))), multiIf(match(lower(nullIf(nullIf(tupleElement(argMinMerge(raw_sessions_v3.entry_channel_type_properties), 3), ''), 'null')), '^(.*video.*)$'), 'Organic Video', match(lower(nullIf(nullIf(tupleElement(argMinMerge(raw_sessions_v3.entry_channel_type_properties), 2), ''), 'null')), 'push$'), 'Push', tupleElement(argMinMerge(raw_sessions_v3.entry_channel_type_properties), 6), 'Organic Social', ifNull(equals(tupleElement(argMinMerge(raw_sessions_v3.entry_channel_type_properties), 4), '$direct'), 0), 'Direct', isNotNull(tupleElement(argMinMerge(raw_sessions_v3.entry_channel_type_properties), 4)), 'Referral', 'Unknown'))) AS `$channel_type`,
+ toString(reinterpretAsUUID(bitOr(bitShiftLeft(raw_sessions_v3.session_id_v7, 64), bitShiftRight(raw_sessions_v3.session_id_v7, 64)))) AS session_id,
+ raw_sessions_v3.session_id_v7 AS session_id_v7
+ FROM raw_sessions_v3
+ WHERE and(equals(raw_sessions_v3.team_id, 99999), equals(raw_sessions_v3.session_timestamp, fromUnixTimestamp64Milli(toUInt64(bitShiftRight(toUInt128(accurateCastOrNull('00000000-0000-0000-0000-000000000000', 'UUID')), 80)))))
+ GROUP BY raw_sessions_v3.session_id_v7) AS sessions
+ WHERE equals(sessions.session_id, '00000000-0000-0000-0000-000000000000')
+ LIMIT 100 SETTINGS readonly=2,
+ max_execution_time=60,
+ allow_experimental_object_type=1,
+ max_ast_elements=4000000,
+ max_expanded_ast_elements=4000000,
+ max_bytes_before_external_group_by=0,
+ transform_null_in=1,
+ optimize_min_equality_disjunction_chain_length=4294967295,
+ optimize_rewrite_aggregate_function_with_if=0,
+ optimize_min_inequality_conjunction_chain_length=4294967295,
+ allow_experimental_join_condition=1,
+ use_hive_partitioning=0
+ '''
+# ---
+# name: TestSessionsV3.test_entry_utm_and_channel_type_across_events_1_first_event_is_not_a_pageview
+ '''
+ SELECT sessions.`$entry_utm_source` AS `$entry_utm_source`,
+ sessions.`$entry_utm_campaign` AS `$entry_utm_campaign`,
+ sessions.`$channel_type` AS `$channel_type`
+ FROM
+ (SELECT argMinMerge(raw_sessions_v3.entry_utm_source) AS `$entry_utm_source`,
+ argMinMerge(raw_sessions_v3.entry_utm_campaign) AS `$entry_utm_campaign`,
+ multiIf(match(lower(nullIf(nullIf(tupleElement(argMinMerge(raw_sessions_v3.entry_channel_type_properties), 3), ''), 'null')), 'cross-network'), 'Cross Network', or(in(lower(nullIf(nullIf(tupleElement(argMinMerge(raw_sessions_v3.entry_channel_type_properties), 2), ''), 'null')), tuple('cpc', 'cpm', 'cpv', 'cpa', 'ppc', 'retargeting')), startsWith(lower(nullIf(nullIf(tupleElement(argMinMerge(raw_sessions_v3.entry_channel_type_properties), 2), ''), 'null')), 'paid'), tupleElement(argMinMerge(raw_sessions_v3.entry_channel_type_properties), 5), isNotNull(nullIf(nullIf(tupleElement(argMinMerge(raw_sessions_v3.entry_channel_type_properties), 7), ''), 'null'))), coalesce(coalesce(dictGetOrNull('posthog_test.channel_definition_dict', 'type_if_paid', (coalesce(lower(nullIf(nullIf(tupleElement(argMinMerge(raw_sessions_v3.entry_channel_type_properties), 1), ''), 'null')), ''), 'source')), dictGetOrNull('posthog_test.channel_definition_dict', 'type_if_paid', (cutToFirstSignificantSubdomain(coalesce(lower(nullIf(nullIf(tupleElement(argMinMerge(raw_sessions_v3.entry_channel_type_properties), 1), ''), 'null')), '')), 'source'))), if(match(lower(nullIf(nullIf(tupleElement(argMinMerge(raw_sessions_v3.entry_channel_type_properties), 3), ''), 'null')), '^(.*(([^a-df-z]|^)shop|shopping).*)$'), 'Paid Shopping', NULL), dictGetOrNull('posthog_test.channel_definition_dict', 'type_if_paid', (coalesce(lower(nullIf(nullIf(tupleElement(argMinMerge(raw_sessions_v3.entry_channel_type_properties), 2), ''), 'null')), ''), 'medium')), coalesce(dictGetOrNull('posthog_test.channel_definition_dict', 'type_if_paid', (coalesce(tupleElement(argMinMerge(raw_sessions_v3.entry_channel_type_properties), 4), ''), 'source')), dictGetOrNull('posthog_test.channel_definition_dict', 'type_if_paid', (cutToFirstSignificantSubdomain(coalesce(tupleElement(argMinMerge(raw_sessions_v3.entry_channel_type_properties), 4), '')), 'source'))), multiIf(ifNull(equals(nullIf(nullIf(tupleElement(argMinMerge(raw_sessions_v3.entry_channel_type_properties), 7), ''), 'null'), '1'), 0), 'Paid Search', match(lower(nullIf(nullIf(tupleElement(argMinMerge(raw_sessions_v3.entry_channel_type_properties), 3), ''), 'null')), '^(.*video.*)$'), 'Paid Video', tupleElement(argMinMerge(raw_sessions_v3.entry_channel_type_properties), 6), 'Paid Social', 'Paid Unknown')), and(ifNull(equals(tupleElement(argMinMerge(raw_sessions_v3.entry_channel_type_properties), 4), '$direct'), 0), isNull(lower(nullIf(nullIf(tupleElement(argMinMerge(raw_sessions_v3.entry_channel_type_properties), 2), ''), 'null'))), or(isNull(lower(nullIf(nullIf(tupleElement(argMinMerge(raw_sessions_v3.entry_channel_type_properties), 1), ''), 'null'))), in(lower(nullIf(nullIf(tupleElement(argMinMerge(raw_sessions_v3.entry_channel_type_properties), 1), ''), 'null')), tuple('(direct)', 'direct', '$direct'))), not(tupleElement(argMinMerge(raw_sessions_v3.entry_channel_type_properties), 6))), 'Direct', coalesce(coalesce(dictGetOrNull('posthog_test.channel_definition_dict', 'type_if_organic', (coalesce(lower(nullIf(nullIf(tupleElement(argMinMerge(raw_sessions_v3.entry_channel_type_properties), 1), ''), 'null')), ''), 'source')), dictGetOrNull('posthog_test.channel_definition_dict', 'type_if_organic', (cutToFirstSignificantSubdomain(coalesce(lower(nullIf(nullIf(tupleElement(argMinMerge(raw_sessions_v3.entry_channel_type_properties), 1), ''), 'null')), '')), 'source'))), if(match(lower(nullIf(nullIf(tupleElement(argMinMerge(raw_sessions_v3.entry_channel_type_properties), 3), ''), 'null')), '^(.*(([^a-df-z]|^)shop|shopping).*)$'), 'Organic Shopping', NULL), dictGetOrNull('posthog_test.channel_definition_dict', 'type_if_organic', (coalesce(lower(nullIf(nullIf(tupleElement(argMinMerge(raw_sessions_v3.entry_channel_type_properties), 2), ''), 'null')), ''), 'medium')), coalesce(dictGetOrNull('posthog_test.channel_definition_dict', 'type_if_organic', (coalesce(tupleElement(argMinMerge(raw_sessions_v3.entry_channel_type_properties), 4), ''), 'source')), dictGetOrNull('posthog_test.channel_definition_dict', 'type_if_organic', (cutToFirstSignificantSubdomain(coalesce(tupleElement(argMinMerge(raw_sessions_v3.entry_channel_type_properties), 4), '')), 'source'))), multiIf(match(lower(nullIf(nullIf(tupleElement(argMinMerge(raw_sessions_v3.entry_channel_type_properties), 3), ''), 'null')), '^(.*video.*)$'), 'Organic Video', match(lower(nullIf(nullIf(tupleElement(argMinMerge(raw_sessions_v3.entry_channel_type_properties), 2), ''), 'null')), 'push$'), 'Push', tupleElement(argMinMerge(raw_sessions_v3.entry_channel_type_properties), 6), 'Organic Social', ifNull(equals(tupleElement(argMinMerge(raw_sessions_v3.entry_channel_type_properties), 4), '$direct'), 0), 'Direct', isNotNull(tupleElement(argMinMerge(raw_sessions_v3.entry_channel_type_properties), 4)), 'Referral', 'Unknown'))) AS `$channel_type`,
+ toString(reinterpretAsUUID(bitOr(bitShiftLeft(raw_sessions_v3.session_id_v7, 64), bitShiftRight(raw_sessions_v3.session_id_v7, 64)))) AS session_id,
+ raw_sessions_v3.session_id_v7 AS session_id_v7
+ FROM raw_sessions_v3
+ WHERE and(equals(raw_sessions_v3.team_id, 99999), equals(raw_sessions_v3.session_timestamp, fromUnixTimestamp64Milli(toUInt64(bitShiftRight(toUInt128(accurateCastOrNull('00000000-0000-0000-0000-000000000000', 'UUID')), 80)))))
+ GROUP BY raw_sessions_v3.session_id_v7) AS sessions
+ WHERE equals(sessions.session_id, '00000000-0000-0000-0000-000000000000')
+ LIMIT 100 SETTINGS readonly=2,
+ max_execution_time=60,
+ allow_experimental_object_type=1,
+ max_ast_elements=4000000,
+ max_expanded_ast_elements=4000000,
+ max_bytes_before_external_group_by=0,
+ transform_null_in=1,
+ optimize_min_equality_disjunction_chain_length=4294967295,
+ optimize_rewrite_aggregate_function_with_if=0,
+ optimize_min_inequality_conjunction_chain_length=4294967295,
+ allow_experimental_join_condition=1,
+ use_hive_partitioning=0
+ '''
+# ---
+# name: TestSessionsV3.test_entry_utm_and_channel_type_across_events_2_utm_added_on_second_pageview
+ '''
+ SELECT sessions.`$entry_utm_source` AS `$entry_utm_source`,
+ sessions.`$entry_utm_campaign` AS `$entry_utm_campaign`,
+ sessions.`$channel_type` AS `$channel_type`
+ FROM
+ (SELECT argMinMerge(raw_sessions_v3.entry_utm_source) AS `$entry_utm_source`,
+ argMinMerge(raw_sessions_v3.entry_utm_campaign) AS `$entry_utm_campaign`,
+ multiIf(match(lower(nullIf(nullIf(tupleElement(argMinMerge(raw_sessions_v3.entry_channel_type_properties), 3), ''), 'null')), 'cross-network'), 'Cross Network', or(in(lower(nullIf(nullIf(tupleElement(argMinMerge(raw_sessions_v3.entry_channel_type_properties), 2), ''), 'null')), tuple('cpc', 'cpm', 'cpv', 'cpa', 'ppc', 'retargeting')), startsWith(lower(nullIf(nullIf(tupleElement(argMinMerge(raw_sessions_v3.entry_channel_type_properties), 2), ''), 'null')), 'paid'), tupleElement(argMinMerge(raw_sessions_v3.entry_channel_type_properties), 5), isNotNull(nullIf(nullIf(tupleElement(argMinMerge(raw_sessions_v3.entry_channel_type_properties), 7), ''), 'null'))), coalesce(coalesce(dictGetOrNull('posthog_test.channel_definition_dict', 'type_if_paid', (coalesce(lower(nullIf(nullIf(tupleElement(argMinMerge(raw_sessions_v3.entry_channel_type_properties), 1), ''), 'null')), ''), 'source')), dictGetOrNull('posthog_test.channel_definition_dict', 'type_if_paid', (cutToFirstSignificantSubdomain(coalesce(lower(nullIf(nullIf(tupleElement(argMinMerge(raw_sessions_v3.entry_channel_type_properties), 1), ''), 'null')), '')), 'source'))), if(match(lower(nullIf(nullIf(tupleElement(argMinMerge(raw_sessions_v3.entry_channel_type_properties), 3), ''), 'null')), '^(.*(([^a-df-z]|^)shop|shopping).*)$'), 'Paid Shopping', NULL), dictGetOrNull('posthog_test.channel_definition_dict', 'type_if_paid', (coalesce(lower(nullIf(nullIf(tupleElement(argMinMerge(raw_sessions_v3.entry_channel_type_properties), 2), ''), 'null')), ''), 'medium')), coalesce(dictGetOrNull('posthog_test.channel_definition_dict', 'type_if_paid', (coalesce(tupleElement(argMinMerge(raw_sessions_v3.entry_channel_type_properties), 4), ''), 'source')), dictGetOrNull('posthog_test.channel_definition_dict', 'type_if_paid', (cutToFirstSignificantSubdomain(coalesce(tupleElement(argMinMerge(raw_sessions_v3.entry_channel_type_properties), 4), '')), 'source'))), multiIf(ifNull(equals(nullIf(nullIf(tupleElement(argMinMerge(raw_sessions_v3.entry_channel_type_properties), 7), ''), 'null'), '1'), 0), 'Paid Search', match(lower(nullIf(nullIf(tupleElement(argMinMerge(raw_sessions_v3.entry_channel_type_properties), 3), ''), 'null')), '^(.*video.*)$'), 'Paid Video', tupleElement(argMinMerge(raw_sessions_v3.entry_channel_type_properties), 6), 'Paid Social', 'Paid Unknown')), and(ifNull(equals(tupleElement(argMinMerge(raw_sessions_v3.entry_channel_type_properties), 4), '$direct'), 0), isNull(lower(nullIf(nullIf(tupleElement(argMinMerge(raw_sessions_v3.entry_channel_type_properties), 2), ''), 'null'))), or(isNull(lower(nullIf(nullIf(tupleElement(argMinMerge(raw_sessions_v3.entry_channel_type_properties), 1), ''), 'null'))), in(lower(nullIf(nullIf(tupleElement(argMinMerge(raw_sessions_v3.entry_channel_type_properties), 1), ''), 'null')), tuple('(direct)', 'direct', '$direct'))), not(tupleElement(argMinMerge(raw_sessions_v3.entry_channel_type_properties), 6))), 'Direct', coalesce(coalesce(dictGetOrNull('posthog_test.channel_definition_dict', 'type_if_organic', (coalesce(lower(nullIf(nullIf(tupleElement(argMinMerge(raw_sessions_v3.entry_channel_type_properties), 1), ''), 'null')), ''), 'source')), dictGetOrNull('posthog_test.channel_definition_dict', 'type_if_organic', (cutToFirstSignificantSubdomain(coalesce(lower(nullIf(nullIf(tupleElement(argMinMerge(raw_sessions_v3.entry_channel_type_properties), 1), ''), 'null')), '')), 'source'))), if(match(lower(nullIf(nullIf(tupleElement(argMinMerge(raw_sessions_v3.entry_channel_type_properties), 3), ''), 'null')), '^(.*(([^a-df-z]|^)shop|shopping).*)$'), 'Organic Shopping', NULL), dictGetOrNull('posthog_test.channel_definition_dict', 'type_if_organic', (coalesce(lower(nullIf(nullIf(tupleElement(argMinMerge(raw_sessions_v3.entry_channel_type_properties), 2), ''), 'null')), ''), 'medium')), coalesce(dictGetOrNull('posthog_test.channel_definition_dict', 'type_if_organic', (coalesce(tupleElement(argMinMerge(raw_sessions_v3.entry_channel_type_properties), 4), ''), 'source')), dictGetOrNull('posthog_test.channel_definition_dict', 'type_if_organic', (cutToFirstSignificantSubdomain(coalesce(tupleElement(argMinMerge(raw_sessions_v3.entry_channel_type_properties), 4), '')), 'source'))), multiIf(match(lower(nullIf(nullIf(tupleElement(argMinMerge(raw_sessions_v3.entry_channel_type_properties), 3), ''), 'null')), '^(.*video.*)$'), 'Organic Video', match(lower(nullIf(nullIf(tupleElement(argMinMerge(raw_sessions_v3.entry_channel_type_properties), 2), ''), 'null')), 'push$'), 'Push', tupleElement(argMinMerge(raw_sessions_v3.entry_channel_type_properties), 6), 'Organic Social', ifNull(equals(tupleElement(argMinMerge(raw_sessions_v3.entry_channel_type_properties), 4), '$direct'), 0), 'Direct', isNotNull(tupleElement(argMinMerge(raw_sessions_v3.entry_channel_type_properties), 4)), 'Referral', 'Unknown'))) AS `$channel_type`,
+ toString(reinterpretAsUUID(bitOr(bitShiftLeft(raw_sessions_v3.session_id_v7, 64), bitShiftRight(raw_sessions_v3.session_id_v7, 64)))) AS session_id,
+ raw_sessions_v3.session_id_v7 AS session_id_v7
+ FROM raw_sessions_v3
+ WHERE and(equals(raw_sessions_v3.team_id, 99999), equals(raw_sessions_v3.session_timestamp, fromUnixTimestamp64Milli(toUInt64(bitShiftRight(toUInt128(accurateCastOrNull('00000000-0000-0000-0000-000000000000', 'UUID')), 80)))))
+ GROUP BY raw_sessions_v3.session_id_v7) AS sessions
+ WHERE equals(sessions.session_id, '00000000-0000-0000-0000-000000000000')
+ LIMIT 100 SETTINGS readonly=2,
+ max_execution_time=60,
+ allow_experimental_object_type=1,
+ max_ast_elements=4000000,
+ max_expanded_ast_elements=4000000,
+ max_bytes_before_external_group_by=0,
+ transform_null_in=1,
+ optimize_min_equality_disjunction_chain_length=4294967295,
+ optimize_rewrite_aggregate_function_with_if=0,
+ optimize_min_inequality_conjunction_chain_length=4294967295,
+ allow_experimental_join_condition=1,
+ use_hive_partitioning=0
+ '''
+# ---
+# name: TestSessionsV3.test_entry_utm_and_channel_type_across_events_3_two_campaigns_keep_the_first
+ '''
+ SELECT sessions.`$entry_utm_source` AS `$entry_utm_source`,
+ sessions.`$entry_utm_campaign` AS `$entry_utm_campaign`,
+ sessions.`$channel_type` AS `$channel_type`
+ FROM
+ (SELECT argMinMerge(raw_sessions_v3.entry_utm_source) AS `$entry_utm_source`,
+ argMinMerge(raw_sessions_v3.entry_utm_campaign) AS `$entry_utm_campaign`,
+ multiIf(match(lower(nullIf(nullIf(tupleElement(argMinMerge(raw_sessions_v3.entry_channel_type_properties), 3), ''), 'null')), 'cross-network'), 'Cross Network', or(in(lower(nullIf(nullIf(tupleElement(argMinMerge(raw_sessions_v3.entry_channel_type_properties), 2), ''), 'null')), tuple('cpc', 'cpm', 'cpv', 'cpa', 'ppc', 'retargeting')), startsWith(lower(nullIf(nullIf(tupleElement(argMinMerge(raw_sessions_v3.entry_channel_type_properties), 2), ''), 'null')), 'paid'), tupleElement(argMinMerge(raw_sessions_v3.entry_channel_type_properties), 5), isNotNull(nullIf(nullIf(tupleElement(argMinMerge(raw_sessions_v3.entry_channel_type_properties), 7), ''), 'null'))), coalesce(coalesce(dictGetOrNull('posthog_test.channel_definition_dict', 'type_if_paid', (coalesce(lower(nullIf(nullIf(tupleElement(argMinMerge(raw_sessions_v3.entry_channel_type_properties), 1), ''), 'null')), ''), 'source')), dictGetOrNull('posthog_test.channel_definition_dict', 'type_if_paid', (cutToFirstSignificantSubdomain(coalesce(lower(nullIf(nullIf(tupleElement(argMinMerge(raw_sessions_v3.entry_channel_type_properties), 1), ''), 'null')), '')), 'source'))), if(match(lower(nullIf(nullIf(tupleElement(argMinMerge(raw_sessions_v3.entry_channel_type_properties), 3), ''), 'null')), '^(.*(([^a-df-z]|^)shop|shopping).*)$'), 'Paid Shopping', NULL), dictGetOrNull('posthog_test.channel_definition_dict', 'type_if_paid', (coalesce(lower(nullIf(nullIf(tupleElement(argMinMerge(raw_sessions_v3.entry_channel_type_properties), 2), ''), 'null')), ''), 'medium')), coalesce(dictGetOrNull('posthog_test.channel_definition_dict', 'type_if_paid', (coalesce(tupleElement(argMinMerge(raw_sessions_v3.entry_channel_type_properties), 4), ''), 'source')), dictGetOrNull('posthog_test.channel_definition_dict', 'type_if_paid', (cutToFirstSignificantSubdomain(coalesce(tupleElement(argMinMerge(raw_sessions_v3.entry_channel_type_properties), 4), '')), 'source'))), multiIf(ifNull(equals(nullIf(nullIf(tupleElement(argMinMerge(raw_sessions_v3.entry_channel_type_properties), 7), ''), 'null'), '1'), 0), 'Paid Search', match(lower(nullIf(nullIf(tupleElement(argMinMerge(raw_sessions_v3.entry_channel_type_properties), 3), ''), 'null')), '^(.*video.*)$'), 'Paid Video', tupleElement(argMinMerge(raw_sessions_v3.entry_channel_type_properties), 6), 'Paid Social', 'Paid Unknown')), and(ifNull(equals(tupleElement(argMinMerge(raw_sessions_v3.entry_channel_type_properties), 4), '$direct'), 0), isNull(lower(nullIf(nullIf(tupleElement(argMinMerge(raw_sessions_v3.entry_channel_type_properties), 2), ''), 'null'))), or(isNull(lower(nullIf(nullIf(tupleElement(argMinMerge(raw_sessions_v3.entry_channel_type_properties), 1), ''), 'null'))), in(lower(nullIf(nullIf(tupleElement(argMinMerge(raw_sessions_v3.entry_channel_type_properties), 1), ''), 'null')), tuple('(direct)', 'direct', '$direct'))), not(tupleElement(argMinMerge(raw_sessions_v3.entry_channel_type_properties), 6))), 'Direct', coalesce(coalesce(dictGetOrNull('posthog_test.channel_definition_dict', 'type_if_organic', (coalesce(lower(nullIf(nullIf(tupleElement(argMinMerge(raw_sessions_v3.entry_channel_type_properties), 1), ''), 'null')), ''), 'source')), dictGetOrNull('posthog_test.channel_definition_dict', 'type_if_organic', (cutToFirstSignificantSubdomain(coalesce(lower(nullIf(nullIf(tupleElement(argMinMerge(raw_sessions_v3.entry_channel_type_properties), 1), ''), 'null')), '')), 'source'))), if(match(lower(nullIf(nullIf(tupleElement(argMinMerge(raw_sessions_v3.entry_channel_type_properties), 3), ''), 'null')), '^(.*(([^a-df-z]|^)shop|shopping).*)$'), 'Organic Shopping', NULL), dictGetOrNull('posthog_test.channel_definition_dict', 'type_if_organic', (coalesce(lower(nullIf(nullIf(tupleElement(argMinMerge(raw_sessions_v3.entry_channel_type_properties), 2), ''), 'null')), ''), 'medium')), coalesce(dictGetOrNull('posthog_test.channel_definition_dict', 'type_if_organic', (coalesce(tupleElement(argMinMerge(raw_sessions_v3.entry_channel_type_properties), 4), ''), 'source')), dictGetOrNull('posthog_test.channel_definition_dict', 'type_if_organic', (cutToFirstSignificantSubdomain(coalesce(tupleElement(argMinMerge(raw_sessions_v3.entry_channel_type_properties), 4), '')), 'source'))), multiIf(match(lower(nullIf(nullIf(tupleElement(argMinMerge(raw_sessions_v3.entry_channel_type_properties), 3), ''), 'null')), '^(.*video.*)$'), 'Organic Video', match(lower(nullIf(nullIf(tupleElement(argMinMerge(raw_sessions_v3.entry_channel_type_properties), 2), ''), 'null')), 'push$'), 'Push', tupleElement(argMinMerge(raw_sessions_v3.entry_channel_type_properties), 6), 'Organic Social', ifNull(equals(tupleElement(argMinMerge(raw_sessions_v3.entry_channel_type_properties), 4), '$direct'), 0), 'Direct', isNotNull(tupleElement(argMinMerge(raw_sessions_v3.entry_channel_type_properties), 4)), 'Referral', 'Unknown'))) AS `$channel_type`,
+ toString(reinterpretAsUUID(bitOr(bitShiftLeft(raw_sessions_v3.session_id_v7, 64), bitShiftRight(raw_sessions_v3.session_id_v7, 64)))) AS session_id,
+ raw_sessions_v3.session_id_v7 AS session_id_v7
+ FROM raw_sessions_v3
+ WHERE and(equals(raw_sessions_v3.team_id, 99999), equals(raw_sessions_v3.session_timestamp, fromUnixTimestamp64Milli(toUInt64(bitShiftRight(toUInt128(accurateCastOrNull('00000000-0000-0000-0000-000000000000', 'UUID')), 80)))))
+ GROUP BY raw_sessions_v3.session_id_v7) AS sessions
+ WHERE equals(sessions.session_id, '00000000-0000-0000-0000-000000000000')
+ LIMIT 100 SETTINGS readonly=2,
+ max_execution_time=60,
+ allow_experimental_object_type=1,
+ max_ast_elements=4000000,
+ max_expanded_ast_elements=4000000,
+ max_bytes_before_external_group_by=0,
+ transform_null_in=1,
+ optimize_min_equality_disjunction_chain_length=4294967295,
+ optimize_rewrite_aggregate_function_with_if=0,
+ optimize_min_inequality_conjunction_chain_length=4294967295,
+ allow_experimental_join_condition=1,
+ use_hive_partitioning=0
+ '''
+# ---
+# name: TestSessionsV3.test_entry_utm_and_channel_type_across_events_4_sdk_session_entry_properties_are_not_read
+ '''
+ SELECT sessions.`$entry_utm_source` AS `$entry_utm_source`,
+ sessions.`$entry_utm_campaign` AS `$entry_utm_campaign`,
+ sessions.`$channel_type` AS `$channel_type`
+ FROM
+ (SELECT argMinMerge(raw_sessions_v3.entry_utm_source) AS `$entry_utm_source`,
+ argMinMerge(raw_sessions_v3.entry_utm_campaign) AS `$entry_utm_campaign`,
+ multiIf(match(lower(nullIf(nullIf(tupleElement(argMinMerge(raw_sessions_v3.entry_channel_type_properties), 3), ''), 'null')), 'cross-network'), 'Cross Network', or(in(lower(nullIf(nullIf(tupleElement(argMinMerge(raw_sessions_v3.entry_channel_type_properties), 2), ''), 'null')), tuple('cpc', 'cpm', 'cpv', 'cpa', 'ppc', 'retargeting')), startsWith(lower(nullIf(nullIf(tupleElement(argMinMerge(raw_sessions_v3.entry_channel_type_properties), 2), ''), 'null')), 'paid'), tupleElement(argMinMerge(raw_sessions_v3.entry_channel_type_properties), 5), isNotNull(nullIf(nullIf(tupleElement(argMinMerge(raw_sessions_v3.entry_channel_type_properties), 7), ''), 'null'))), coalesce(coalesce(dictGetOrNull('posthog_test.channel_definition_dict', 'type_if_paid', (coalesce(lower(nullIf(nullIf(tupleElement(argMinMerge(raw_sessions_v3.entry_channel_type_properties), 1), ''), 'null')), ''), 'source')), dictGetOrNull('posthog_test.channel_definition_dict', 'type_if_paid', (cutToFirstSignificantSubdomain(coalesce(lower(nullIf(nullIf(tupleElement(argMinMerge(raw_sessions_v3.entry_channel_type_properties), 1), ''), 'null')), '')), 'source'))), if(match(lower(nullIf(nullIf(tupleElement(argMinMerge(raw_sessions_v3.entry_channel_type_properties), 3), ''), 'null')), '^(.*(([^a-df-z]|^)shop|shopping).*)$'), 'Paid Shopping', NULL), dictGetOrNull('posthog_test.channel_definition_dict', 'type_if_paid', (coalesce(lower(nullIf(nullIf(tupleElement(argMinMerge(raw_sessions_v3.entry_channel_type_properties), 2), ''), 'null')), ''), 'medium')), coalesce(dictGetOrNull('posthog_test.channel_definition_dict', 'type_if_paid', (coalesce(tupleElement(argMinMerge(raw_sessions_v3.entry_channel_type_properties), 4), ''), 'source')), dictGetOrNull('posthog_test.channel_definition_dict', 'type_if_paid', (cutToFirstSignificantSubdomain(coalesce(tupleElement(argMinMerge(raw_sessions_v3.entry_channel_type_properties), 4), '')), 'source'))), multiIf(ifNull(equals(nullIf(nullIf(tupleElement(argMinMerge(raw_sessions_v3.entry_channel_type_properties), 7), ''), 'null'), '1'), 0), 'Paid Search', match(lower(nullIf(nullIf(tupleElement(argMinMerge(raw_sessions_v3.entry_channel_type_properties), 3), ''), 'null')), '^(.*video.*)$'), 'Paid Video', tupleElement(argMinMerge(raw_sessions_v3.entry_channel_type_properties), 6), 'Paid Social', 'Paid Unknown')), and(ifNull(equals(tupleElement(argMinMerge(raw_sessions_v3.entry_channel_type_properties), 4), '$direct'), 0), isNull(lower(nullIf(nullIf(tupleElement(argMinMerge(raw_sessions_v3.entry_channel_type_properties), 2), ''), 'null'))), or(isNull(lower(nullIf(nullIf(tupleElement(argMinMerge(raw_sessions_v3.entry_channel_type_properties), 1), ''), 'null'))), in(lower(nullIf(nullIf(tupleElement(argMinMerge(raw_sessions_v3.entry_channel_type_properties), 1), ''), 'null')), tuple('(direct)', 'direct', '$direct'))), not(tupleElement(argMinMerge(raw_sessions_v3.entry_channel_type_properties), 6))), 'Direct', coalesce(coalesce(dictGetOrNull('posthog_test.channel_definition_dict', 'type_if_organic', (coalesce(lower(nullIf(nullIf(tupleElement(argMinMerge(raw_sessions_v3.entry_channel_type_properties), 1), ''), 'null')), ''), 'source')), dictGetOrNull('posthog_test.channel_definition_dict', 'type_if_organic', (cutToFirstSignificantSubdomain(coalesce(lower(nullIf(nullIf(tupleElement(argMinMerge(raw_sessions_v3.entry_channel_type_properties), 1), ''), 'null')), '')), 'source'))), if(match(lower(nullIf(nullIf(tupleElement(argMinMerge(raw_sessions_v3.entry_channel_type_properties), 3), ''), 'null')), '^(.*(([^a-df-z]|^)shop|shopping).*)$'), 'Organic Shopping', NULL), dictGetOrNull('posthog_test.channel_definition_dict', 'type_if_organic', (coalesce(lower(nullIf(nullIf(tupleElement(argMinMerge(raw_sessions_v3.entry_channel_type_properties), 2), ''), 'null')), ''), 'medium')), coalesce(dictGetOrNull('posthog_test.channel_definition_dict', 'type_if_organic', (coalesce(tupleElement(argMinMerge(raw_sessions_v3.entry_channel_type_properties), 4), ''), 'source')), dictGetOrNull('posthog_test.channel_definition_dict', 'type_if_organic', (cutToFirstSignificantSubdomain(coalesce(tupleElement(argMinMerge(raw_sessions_v3.entry_channel_type_properties), 4), '')), 'source'))), multiIf(match(lower(nullIf(nullIf(tupleElement(argMinMerge(raw_sessions_v3.entry_channel_type_properties), 3), ''), 'null')), '^(.*video.*)$'), 'Organic Video', match(lower(nullIf(nullIf(tupleElement(argMinMerge(raw_sessions_v3.entry_channel_type_properties), 2), ''), 'null')), 'push$'), 'Push', tupleElement(argMinMerge(raw_sessions_v3.entry_channel_type_properties), 6), 'Organic Social', ifNull(equals(tupleElement(argMinMerge(raw_sessions_v3.entry_channel_type_properties), 4), '$direct'), 0), 'Direct', isNotNull(tupleElement(argMinMerge(raw_sessions_v3.entry_channel_type_properties), 4)), 'Referral', 'Unknown'))) AS `$channel_type`,
+ toString(reinterpretAsUUID(bitOr(bitShiftLeft(raw_sessions_v3.session_id_v7, 64), bitShiftRight(raw_sessions_v3.session_id_v7, 64)))) AS session_id,
+ raw_sessions_v3.session_id_v7 AS session_id_v7
+ FROM raw_sessions_v3
+ WHERE and(equals(raw_sessions_v3.team_id, 99999), equals(raw_sessions_v3.session_timestamp, fromUnixTimestamp64Milli(toUInt64(bitShiftRight(toUInt128(accurateCastOrNull('00000000-0000-0000-0000-000000000000', 'UUID')), 80)))))
+ GROUP BY raw_sessions_v3.session_id_v7) AS sessions
+ WHERE equals(sessions.session_id, '00000000-0000-0000-0000-000000000000')
+ LIMIT 100 SETTINGS readonly=2,
+ max_execution_time=60,
+ allow_experimental_object_type=1,
+ max_ast_elements=4000000,
+ max_expanded_ast_elements=4000000,
+ max_bytes_before_external_group_by=0,
+ transform_null_in=1,
+ optimize_min_equality_disjunction_chain_length=4294967295,
+ optimize_rewrite_aggregate_function_with_if=0,
+ optimize_min_inequality_conjunction_chain_length=4294967295,
+ allow_experimental_join_condition=1,
+ use_hive_partitioning=0
+ '''
+# ---
# name: TestSessionsV3.test_event_sessions_where
'''
SELECT count() AS `count()`
@@ -277,9 +647,9 @@
sessions.`$autocapture_count` AS `$autocapture_count`,
sessions.`$screen_count` AS `$screen_count`
FROM
- (SELECT uniqExactMerge(raw_sessions_v3.pageview_uniq) AS `$pageview_count`,
- uniqExactMerge(raw_sessions_v3.autocapture_uniq) AS `$autocapture_count`,
- uniqExactMerge(raw_sessions_v3.screen_uniq) AS `$screen_count`,
+ (SELECT uniqMerge(raw_sessions_v3.pageview_uniq) AS `$pageview_count`,
+ uniqMerge(raw_sessions_v3.autocapture_uniq) AS `$autocapture_count`,
+ uniqMerge(raw_sessions_v3.screen_uniq) AS `$screen_count`,
toString(reinterpretAsUUID(bitOr(bitShiftLeft(raw_sessions_v3.session_id_v7, 64), bitShiftRight(raw_sessions_v3.session_id_v7, 64)))) AS id,
raw_sessions_v3.session_id_v7 AS session_id_v7
FROM raw_sessions_v3
@@ -386,6 +756,72 @@
use_hive_partitioning=0
'''
# ---
+# name: TestSessionsV3.test_replay_rows_shape_the_session_0_replay_starts_before_first_pageview
+ '''
+ SELECT sessions.`$start_timestamp` AS `$start_timestamp`,
+ sessions.`$session_duration` AS `$session_duration`,
+ sessions.`$pageview_count` AS `$pageview_count`,
+ sessions.`$is_bounce` AS `$is_bounce`,
+ sessions.`$has_replay_events` AS `$has_replay_events`
+ FROM
+ (SELECT min(toTimeZone(raw_sessions_v3.min_timestamp, 'UTC')) AS `$start_timestamp`,
+ dateDiff('second', min(toTimeZone(raw_sessions_v3.min_timestamp, 'UTC')), max(toTimeZone(raw_sessions_v3.max_timestamp, 'UTC'))) AS `$session_duration`,
+ uniqMerge(raw_sessions_v3.pageview_uniq) AS `$pageview_count`,
+ if(equals(uniqUpToMerge(1)(raw_sessions_v3.page_screen_uniq_up_to), 0), NULL, not(or(greaterOrEquals(uniqUpToMerge(1)(raw_sessions_v3.page_screen_uniq_up_to), 2), max(raw_sessions_v3.has_autocapture), greaterOrEquals(dateDiff('second', min(toTimeZone(raw_sessions_v3.min_timestamp, 'UTC')), max(toTimeZone(raw_sessions_v3.max_timestamp, 'UTC'))), 10)))) AS `$is_bounce`,
+ max(raw_sessions_v3.has_replay_events) AS `$has_replay_events`,
+ toString(reinterpretAsUUID(bitOr(bitShiftLeft(raw_sessions_v3.session_id_v7, 64), bitShiftRight(raw_sessions_v3.session_id_v7, 64)))) AS session_id,
+ raw_sessions_v3.session_id_v7 AS session_id_v7
+ FROM raw_sessions_v3
+ WHERE and(equals(raw_sessions_v3.team_id, 99999), equals(raw_sessions_v3.session_timestamp, fromUnixTimestamp64Milli(toUInt64(bitShiftRight(toUInt128(accurateCastOrNull('00000000-0000-0000-0000-000000000000', 'UUID')), 80)))))
+ GROUP BY raw_sessions_v3.session_id_v7) AS sessions
+ WHERE equals(sessions.session_id, '00000000-0000-0000-0000-000000000000')
+ LIMIT 100 SETTINGS readonly=2,
+ max_execution_time=60,
+ allow_experimental_object_type=1,
+ max_ast_elements=4000000,
+ max_expanded_ast_elements=4000000,
+ max_bytes_before_external_group_by=0,
+ transform_null_in=1,
+ optimize_min_equality_disjunction_chain_length=4294967295,
+ optimize_rewrite_aggregate_function_with_if=0,
+ optimize_min_inequality_conjunction_chain_length=4294967295,
+ allow_experimental_join_condition=1,
+ use_hive_partitioning=0
+ '''
+# ---
+# name: TestSessionsV3.test_replay_rows_shape_the_session_1_replay_only
+ '''
+ SELECT sessions.`$start_timestamp` AS `$start_timestamp`,
+ sessions.`$session_duration` AS `$session_duration`,
+ sessions.`$pageview_count` AS `$pageview_count`,
+ sessions.`$is_bounce` AS `$is_bounce`,
+ sessions.`$has_replay_events` AS `$has_replay_events`
+ FROM
+ (SELECT min(toTimeZone(raw_sessions_v3.min_timestamp, 'UTC')) AS `$start_timestamp`,
+ dateDiff('second', min(toTimeZone(raw_sessions_v3.min_timestamp, 'UTC')), max(toTimeZone(raw_sessions_v3.max_timestamp, 'UTC'))) AS `$session_duration`,
+ uniqMerge(raw_sessions_v3.pageview_uniq) AS `$pageview_count`,
+ if(equals(uniqUpToMerge(1)(raw_sessions_v3.page_screen_uniq_up_to), 0), NULL, not(or(greaterOrEquals(uniqUpToMerge(1)(raw_sessions_v3.page_screen_uniq_up_to), 2), max(raw_sessions_v3.has_autocapture), greaterOrEquals(dateDiff('second', min(toTimeZone(raw_sessions_v3.min_timestamp, 'UTC')), max(toTimeZone(raw_sessions_v3.max_timestamp, 'UTC'))), 10)))) AS `$is_bounce`,
+ max(raw_sessions_v3.has_replay_events) AS `$has_replay_events`,
+ toString(reinterpretAsUUID(bitOr(bitShiftLeft(raw_sessions_v3.session_id_v7, 64), bitShiftRight(raw_sessions_v3.session_id_v7, 64)))) AS session_id,
+ raw_sessions_v3.session_id_v7 AS session_id_v7
+ FROM raw_sessions_v3
+ WHERE and(equals(raw_sessions_v3.team_id, 99999), equals(raw_sessions_v3.session_timestamp, fromUnixTimestamp64Milli(toUInt64(bitShiftRight(toUInt128(accurateCastOrNull('00000000-0000-0000-0000-000000000000', 'UUID')), 80)))))
+ GROUP BY raw_sessions_v3.session_id_v7) AS sessions
+ WHERE equals(sessions.session_id, '00000000-0000-0000-0000-000000000000')
+ LIMIT 100 SETTINGS readonly=2,
+ max_execution_time=60,
+ allow_experimental_object_type=1,
+ max_ast_elements=4000000,
+ max_expanded_ast_elements=4000000,
+ max_bytes_before_external_group_by=0,
+ transform_null_in=1,
+ optimize_min_equality_disjunction_chain_length=4294967295,
+ optimize_rewrite_aggregate_function_with_if=0,
+ optimize_min_inequality_conjunction_chain_length=4294967295,
+ allow_experimental_join_condition=1,
+ use_hive_partitioning=0
+ '''
+# ---
# name: TestSessionsV3.test_select_session_replay_session_duration
'''
SELECT raw_session_replay_events__session.duration AS duration
@@ -480,6 +916,10 @@
sessions.`$entry_has_gbraid` AS `$entry_has_gbraid`,
sessions.`$entry_wbraid` AS `$entry_wbraid`,
sessions.`$entry_has_wbraid` AS `$entry_has_wbraid`,
+ sessions.`$entry_gad_campaignid` AS `$entry_gad_campaignid`,
+ sessions.`$entry_has_gad_campaignid` AS `$entry_has_gad_campaignid`,
+ sessions.`$entry_srsltid` AS `$entry_srsltid`,
+ sessions.`$entry_has_srsltid` AS `$entry_has_srsltid`,
sessions.`$entry_msclkid` AS `$entry_msclkid`,
sessions.`$entry_has_msclkid` AS `$entry_has_msclkid`,
sessions.`$entry_twclid` AS `$entry_twclid`,
@@ -492,6 +932,8 @@
sessions.`$entry_has_igshid` AS `$entry_has_igshid`,
sessions.`$entry_ttclid` AS `$entry_ttclid`,
sessions.`$entry_has_ttclid` AS `$entry_has_ttclid`,
+ sessions.`$entry_rdt_cid` AS `$entry_rdt_cid`,
+ sessions.`$entry_has_rdt_cid` AS `$entry_has_rdt_cid`,
sessions.`$entry_epik` AS `$entry_epik`,
sessions.`$entry_has_epik` AS `$entry_has_epik`,
sessions.`$entry_qclid` AS `$entry_qclid`,
@@ -501,7 +943,39 @@
sessions.`$entry__kx` AS `$entry__kx`,
sessions.`$entry_has__kx` AS `$entry_has__kx`,
sessions.`$entry_irclid` AS `$entry_irclid`,
- sessions.`$entry_has_irclid` AS `$entry_has_irclid`
+ sessions.`$entry_has_irclid` AS `$entry_has_irclid`,
+ sessions.`$entry_irclickid` AS `$entry_irclickid`,
+ sessions.`$entry_has_irclickid` AS `$entry_has_irclickid`,
+ sessions.`$entry_utm_id` AS `$entry_utm_id`,
+ sessions.`$entry_has_utm_id` AS `$entry_has_utm_id`,
+ sessions.`$entry_utm_source_platform` AS `$entry_utm_source_platform`,
+ sessions.`$entry_has_utm_source_platform` AS `$entry_has_utm_source_platform`,
+ sessions.`$entry_utm_creative_format` AS `$entry_utm_creative_format`,
+ sessions.`$entry_has_utm_creative_format` AS `$entry_has_utm_creative_format`,
+ sessions.`$entry_utm_marketing_tactic` AS `$entry_utm_marketing_tactic`,
+ sessions.`$entry_has_utm_marketing_tactic` AS `$entry_has_utm_marketing_tactic`,
+ sessions.`$entry_yclid` AS `$entry_yclid`,
+ sessions.`$entry_has_yclid` AS `$entry_has_yclid`,
+ sessions.`$entry_tblci` AS `$entry_tblci`,
+ sessions.`$entry_has_tblci` AS `$entry_has_tblci`,
+ sessions.`$entry_dicbo` AS `$entry_dicbo`,
+ sessions.`$entry_has_dicbo` AS `$entry_has_dicbo`,
+ sessions.`$entry_ndclid` AS `$entry_ndclid`,
+ sessions.`$entry_has_ndclid` AS `$entry_has_ndclid`,
+ sessions.`$entry_cjevent` AS `$entry_cjevent`,
+ sessions.`$entry_has_cjevent` AS `$entry_has_cjevent`,
+ sessions.`$entry_awc` AS `$entry_awc`,
+ sessions.`$entry_has_awc` AS `$entry_has_awc`,
+ sessions.`$entry_syclid` AS `$entry_syclid`,
+ sessions.`$entry_has_syclid` AS `$entry_has_syclid`,
+ sessions.`$entry__bhlid` AS `$entry__bhlid`,
+ sessions.`$entry_has__bhlid` AS `$entry_has__bhlid`,
+ sessions.`$entry_rtid` AS `$entry_rtid`,
+ sessions.`$entry_has_rtid` AS `$entry_has_rtid`,
+ sessions.`$entry_oppref` AS `$entry_oppref`,
+ sessions.`$entry_has_oppref` AS `$entry_has_oppref`,
+ sessions.`$entry_bfcid` AS `$entry_bfcid`,
+ sessions.`$entry_has_bfcid` AS `$entry_has_bfcid`
FROM
(SELECT toString(reinterpretAsUUID(bitOr(bitShiftLeft(raw_sessions_v3.session_id_v7, 64), bitShiftRight(raw_sessions_v3.session_id_v7, 64)))) AS session_id,
raw_sessions_v3.session_timestamp AS session_timestamp,
@@ -527,45 +1001,83 @@
argMinMerge(raw_sessions_v3.entry_fbclid) AS `$entry_fbclid`,
argMinMerge(raw_sessions_v3.entry_has_fbclid) AS `$entry_has_fbclid`,
argMinMerge(raw_sessions_v3.entry_has_gclid) AS `$entry_has_gclid`,
- uniqExactMerge(raw_sessions_v3.pageview_uniq) AS `$pageview_count`,
- uniqExactMerge(raw_sessions_v3.autocapture_uniq) AS `$autocapture_count`,
- uniqExactMerge(raw_sessions_v3.screen_uniq) AS `$screen_count`,
+ uniqMerge(raw_sessions_v3.pageview_uniq) AS `$pageview_count`,
+ uniqMerge(raw_sessions_v3.autocapture_uniq) AS `$autocapture_count`,
+ uniqMerge(raw_sessions_v3.screen_uniq) AS `$screen_count`,
multiIf(match(lower(nullIf(nullIf(tupleElement(argMinMerge(raw_sessions_v3.entry_channel_type_properties), 3), ''), 'null')), 'cross-network'), 'Cross Network', or(in(lower(nullIf(nullIf(tupleElement(argMinMerge(raw_sessions_v3.entry_channel_type_properties), 2), ''), 'null')), tuple('cpc', 'cpm', 'cpv', 'cpa', 'ppc', 'retargeting')), startsWith(lower(nullIf(nullIf(tupleElement(argMinMerge(raw_sessions_v3.entry_channel_type_properties), 2), ''), 'null')), 'paid'), tupleElement(argMinMerge(raw_sessions_v3.entry_channel_type_properties), 5), isNotNull(nullIf(nullIf(tupleElement(argMinMerge(raw_sessions_v3.entry_channel_type_properties), 7), ''), 'null'))), coalesce(coalesce(dictGetOrNull('posthog_test.channel_definition_dict', 'type_if_paid', (coalesce(lower(nullIf(nullIf(tupleElement(argMinMerge(raw_sessions_v3.entry_channel_type_properties), 1), ''), 'null')), ''), 'source')), dictGetOrNull('posthog_test.channel_definition_dict', 'type_if_paid', (cutToFirstSignificantSubdomain(coalesce(lower(nullIf(nullIf(tupleElement(argMinMerge(raw_sessions_v3.entry_channel_type_properties), 1), ''), 'null')), '')), 'source'))), if(match(lower(nullIf(nullIf(tupleElement(argMinMerge(raw_sessions_v3.entry_channel_type_properties), 3), ''), 'null')), '^(.*(([^a-df-z]|^)shop|shopping).*)$'), 'Paid Shopping', NULL), dictGetOrNull('posthog_test.channel_definition_dict', 'type_if_paid', (coalesce(lower(nullIf(nullIf(tupleElement(argMinMerge(raw_sessions_v3.entry_channel_type_properties), 2), ''), 'null')), ''), 'medium')), coalesce(dictGetOrNull('posthog_test.channel_definition_dict', 'type_if_paid', (coalesce(tupleElement(argMinMerge(raw_sessions_v3.entry_channel_type_properties), 4), ''), 'source')), dictGetOrNull('posthog_test.channel_definition_dict', 'type_if_paid', (cutToFirstSignificantSubdomain(coalesce(tupleElement(argMinMerge(raw_sessions_v3.entry_channel_type_properties), 4), '')), 'source'))), multiIf(ifNull(equals(nullIf(nullIf(tupleElement(argMinMerge(raw_sessions_v3.entry_channel_type_properties), 7), ''), 'null'), '1'), 0), 'Paid Search', match(lower(nullIf(nullIf(tupleElement(argMinMerge(raw_sessions_v3.entry_channel_type_properties), 3), ''), 'null')), '^(.*video.*)$'), 'Paid Video', tupleElement(argMinMerge(raw_sessions_v3.entry_channel_type_properties), 6), 'Paid Social', 'Paid Unknown')), and(ifNull(equals(tupleElement(argMinMerge(raw_sessions_v3.entry_channel_type_properties), 4), '$direct'), 0), isNull(lower(nullIf(nullIf(tupleElement(argMinMerge(raw_sessions_v3.entry_channel_type_properties), 2), ''), 'null'))), or(isNull(lower(nullIf(nullIf(tupleElement(argMinMerge(raw_sessions_v3.entry_channel_type_properties), 1), ''), 'null'))), in(lower(nullIf(nullIf(tupleElement(argMinMerge(raw_sessions_v3.entry_channel_type_properties), 1), ''), 'null')), tuple('(direct)', 'direct', '$direct'))), not(tupleElement(argMinMerge(raw_sessions_v3.entry_channel_type_properties), 6))), 'Direct', coalesce(coalesce(dictGetOrNull('posthog_test.channel_definition_dict', 'type_if_organic', (coalesce(lower(nullIf(nullIf(tupleElement(argMinMerge(raw_sessions_v3.entry_channel_type_properties), 1), ''), 'null')), ''), 'source')), dictGetOrNull('posthog_test.channel_definition_dict', 'type_if_organic', (cutToFirstSignificantSubdomain(coalesce(lower(nullIf(nullIf(tupleElement(argMinMerge(raw_sessions_v3.entry_channel_type_properties), 1), ''), 'null')), '')), 'source'))), if(match(lower(nullIf(nullIf(tupleElement(argMinMerge(raw_sessions_v3.entry_channel_type_properties), 3), ''), 'null')), '^(.*(([^a-df-z]|^)shop|shopping).*)$'), 'Organic Shopping', NULL), dictGetOrNull('posthog_test.channel_definition_dict', 'type_if_organic', (coalesce(lower(nullIf(nullIf(tupleElement(argMinMerge(raw_sessions_v3.entry_channel_type_properties), 2), ''), 'null')), ''), 'medium')), coalesce(dictGetOrNull('posthog_test.channel_definition_dict', 'type_if_organic', (coalesce(tupleElement(argMinMerge(raw_sessions_v3.entry_channel_type_properties), 4), ''), 'source')), dictGetOrNull('posthog_test.channel_definition_dict', 'type_if_organic', (cutToFirstSignificantSubdomain(coalesce(tupleElement(argMinMerge(raw_sessions_v3.entry_channel_type_properties), 4), '')), 'source'))), multiIf(match(lower(nullIf(nullIf(tupleElement(argMinMerge(raw_sessions_v3.entry_channel_type_properties), 3), ''), 'null')), '^(.*video.*)$'), 'Organic Video', match(lower(nullIf(nullIf(tupleElement(argMinMerge(raw_sessions_v3.entry_channel_type_properties), 2), ''), 'null')), 'push$'), 'Push', tupleElement(argMinMerge(raw_sessions_v3.entry_channel_type_properties), 6), 'Organic Social', ifNull(equals(tupleElement(argMinMerge(raw_sessions_v3.entry_channel_type_properties), 4), '$direct'), 0), 'Direct', isNotNull(tupleElement(argMinMerge(raw_sessions_v3.entry_channel_type_properties), 4)), 'Referral', 'Unknown'))) AS `$channel_type`,
dateDiff('second', min(toTimeZone(raw_sessions_v3.min_timestamp, 'UTC')), max(toTimeZone(raw_sessions_v3.max_timestamp, 'UTC'))) AS `$session_duration`,
if(equals(uniqUpToMerge(1)(raw_sessions_v3.page_screen_uniq_up_to), 0), NULL, not(or(greaterOrEquals(uniqUpToMerge(1)(raw_sessions_v3.page_screen_uniq_up_to), 2), max(raw_sessions_v3.has_autocapture), greaterOrEquals(dateDiff('second', min(toTimeZone(raw_sessions_v3.min_timestamp, 'UTC')), max(toTimeZone(raw_sessions_v3.max_timestamp, 'UTC'))), 10)))) AS `$is_bounce`,
arrayDistinct(arrayFlatten(groupArray(raw_sessions_v3.hosts))) AS `$hosts`,
arrayDistinct(arrayFlatten(groupArray(raw_sessions_v3.emails))) AS `$emails`,
max(raw_sessions_v3.has_replay_events) AS `$has_replay_events`,
- arrayElement(argMinMerge(raw_sessions_v3.entry_ad_ids_map), 'gclsrc') AS `$entry_gclsrc`,
- has(argMinMerge(raw_sessions_v3.entry_ad_ids_set), 'gclsrc') AS `$entry_has_gclsrc`,
- arrayElement(argMinMerge(raw_sessions_v3.entry_ad_ids_map), 'dclid') AS `$entry_dclid`,
- has(argMinMerge(raw_sessions_v3.entry_ad_ids_set), 'dclid') AS `$entry_has_dclid`,
- arrayElement(argMinMerge(raw_sessions_v3.entry_ad_ids_map), 'gbraid') AS `$entry_gbraid`,
- has(argMinMerge(raw_sessions_v3.entry_ad_ids_set), 'gbraid') AS `$entry_has_gbraid`,
- arrayElement(argMinMerge(raw_sessions_v3.entry_ad_ids_map), 'wbraid') AS `$entry_wbraid`,
- has(argMinMerge(raw_sessions_v3.entry_ad_ids_set), 'wbraid') AS `$entry_has_wbraid`,
- arrayElement(argMinMerge(raw_sessions_v3.entry_ad_ids_map), 'msclkid') AS `$entry_msclkid`,
- has(argMinMerge(raw_sessions_v3.entry_ad_ids_set), 'msclkid') AS `$entry_has_msclkid`,
- arrayElement(argMinMerge(raw_sessions_v3.entry_ad_ids_map), 'twclid') AS `$entry_twclid`,
- has(argMinMerge(raw_sessions_v3.entry_ad_ids_set), 'twclid') AS `$entry_has_twclid`,
- arrayElement(argMinMerge(raw_sessions_v3.entry_ad_ids_map), 'li_fat_id') AS `$entry_li_fat_id`,
- has(argMinMerge(raw_sessions_v3.entry_ad_ids_set), 'li_fat_id') AS `$entry_has_li_fat_id`,
- arrayElement(argMinMerge(raw_sessions_v3.entry_ad_ids_map), 'mc_cid') AS `$entry_mc_cid`,
- has(argMinMerge(raw_sessions_v3.entry_ad_ids_set), 'mc_cid') AS `$entry_has_mc_cid`,
- arrayElement(argMinMerge(raw_sessions_v3.entry_ad_ids_map), 'igshid') AS `$entry_igshid`,
- has(argMinMerge(raw_sessions_v3.entry_ad_ids_set), 'igshid') AS `$entry_has_igshid`,
- arrayElement(argMinMerge(raw_sessions_v3.entry_ad_ids_map), 'ttclid') AS `$entry_ttclid`,
- has(argMinMerge(raw_sessions_v3.entry_ad_ids_set), 'ttclid') AS `$entry_has_ttclid`,
- arrayElement(argMinMerge(raw_sessions_v3.entry_ad_ids_map), 'epik') AS `$entry_epik`,
- has(argMinMerge(raw_sessions_v3.entry_ad_ids_set), 'epik') AS `$entry_has_epik`,
- arrayElement(argMinMerge(raw_sessions_v3.entry_ad_ids_map), 'qclid') AS `$entry_qclid`,
- has(argMinMerge(raw_sessions_v3.entry_ad_ids_set), 'qclid') AS `$entry_has_qclid`,
- arrayElement(argMinMerge(raw_sessions_v3.entry_ad_ids_map), 'sccid') AS `$entry_sccid`,
- has(argMinMerge(raw_sessions_v3.entry_ad_ids_set), 'sccid') AS `$entry_has_sccid`,
- arrayElement(argMinMerge(raw_sessions_v3.entry_ad_ids_map), '_kx') AS `$entry__kx`,
- has(argMinMerge(raw_sessions_v3.entry_ad_ids_set), '_kx') AS `$entry_has__kx`,
- arrayElement(argMinMerge(raw_sessions_v3.entry_ad_ids_map), 'irclid') AS `$entry_irclid`,
- has(argMinMerge(raw_sessions_v3.entry_ad_ids_set), 'irclid') AS `$entry_has_irclid`,
+ replaceOne(arrayFirst(x -> startsWith(x, 'gclsrc='), argMinMerge(raw_sessions_v3.entry_ad_ids_set)), 'gclsrc=', '') AS `$entry_gclsrc`,
+ arrayExists(x -> startsWith(x, 'gclsrc='), argMinMerge(raw_sessions_v3.entry_ad_ids_set)) AS `$entry_has_gclsrc`,
+ replaceOne(arrayFirst(x -> startsWith(x, 'dclid='), argMinMerge(raw_sessions_v3.entry_ad_ids_set)), 'dclid=', '') AS `$entry_dclid`,
+ arrayExists(x -> startsWith(x, 'dclid='), argMinMerge(raw_sessions_v3.entry_ad_ids_set)) AS `$entry_has_dclid`,
+ replaceOne(arrayFirst(x -> startsWith(x, 'gbraid='), argMinMerge(raw_sessions_v3.entry_ad_ids_set)), 'gbraid=', '') AS `$entry_gbraid`,
+ arrayExists(x -> startsWith(x, 'gbraid='), argMinMerge(raw_sessions_v3.entry_ad_ids_set)) AS `$entry_has_gbraid`,
+ replaceOne(arrayFirst(x -> startsWith(x, 'wbraid='), argMinMerge(raw_sessions_v3.entry_ad_ids_set)), 'wbraid=', '') AS `$entry_wbraid`,
+ arrayExists(x -> startsWith(x, 'wbraid='), argMinMerge(raw_sessions_v3.entry_ad_ids_set)) AS `$entry_has_wbraid`,
+ replaceOne(arrayFirst(x -> startsWith(x, 'gad_campaignid='), argMinMerge(raw_sessions_v3.entry_ad_ids_set)), 'gad_campaignid=', '') AS `$entry_gad_campaignid`,
+ arrayExists(x -> startsWith(x, 'gad_campaignid='), argMinMerge(raw_sessions_v3.entry_ad_ids_set)) AS `$entry_has_gad_campaignid`,
+ replaceOne(arrayFirst(x -> startsWith(x, 'srsltid='), argMinMerge(raw_sessions_v3.entry_ad_ids_set)), 'srsltid=', '') AS `$entry_srsltid`,
+ arrayExists(x -> startsWith(x, 'srsltid='), argMinMerge(raw_sessions_v3.entry_ad_ids_set)) AS `$entry_has_srsltid`,
+ replaceOne(arrayFirst(x -> startsWith(x, 'msclkid='), argMinMerge(raw_sessions_v3.entry_ad_ids_set)), 'msclkid=', '') AS `$entry_msclkid`,
+ arrayExists(x -> startsWith(x, 'msclkid='), argMinMerge(raw_sessions_v3.entry_ad_ids_set)) AS `$entry_has_msclkid`,
+ replaceOne(arrayFirst(x -> startsWith(x, 'twclid='), argMinMerge(raw_sessions_v3.entry_ad_ids_set)), 'twclid=', '') AS `$entry_twclid`,
+ arrayExists(x -> startsWith(x, 'twclid='), argMinMerge(raw_sessions_v3.entry_ad_ids_set)) AS `$entry_has_twclid`,
+ replaceOne(arrayFirst(x -> startsWith(x, 'li_fat_id='), argMinMerge(raw_sessions_v3.entry_ad_ids_set)), 'li_fat_id=', '') AS `$entry_li_fat_id`,
+ arrayExists(x -> startsWith(x, 'li_fat_id='), argMinMerge(raw_sessions_v3.entry_ad_ids_set)) AS `$entry_has_li_fat_id`,
+ replaceOne(arrayFirst(x -> startsWith(x, 'mc_cid='), argMinMerge(raw_sessions_v3.entry_ad_ids_set)), 'mc_cid=', '') AS `$entry_mc_cid`,
+ arrayExists(x -> startsWith(x, 'mc_cid='), argMinMerge(raw_sessions_v3.entry_ad_ids_set)) AS `$entry_has_mc_cid`,
+ replaceOne(arrayFirst(x -> startsWith(x, 'igshid='), argMinMerge(raw_sessions_v3.entry_ad_ids_set)), 'igshid=', '') AS `$entry_igshid`,
+ arrayExists(x -> startsWith(x, 'igshid='), argMinMerge(raw_sessions_v3.entry_ad_ids_set)) AS `$entry_has_igshid`,
+ replaceOne(arrayFirst(x -> startsWith(x, 'ttclid='), argMinMerge(raw_sessions_v3.entry_ad_ids_set)), 'ttclid=', '') AS `$entry_ttclid`,
+ arrayExists(x -> startsWith(x, 'ttclid='), argMinMerge(raw_sessions_v3.entry_ad_ids_set)) AS `$entry_has_ttclid`,
+ replaceOne(arrayFirst(x -> startsWith(x, 'rdt_cid='), argMinMerge(raw_sessions_v3.entry_ad_ids_set)), 'rdt_cid=', '') AS `$entry_rdt_cid`,
+ arrayExists(x -> startsWith(x, 'rdt_cid='), argMinMerge(raw_sessions_v3.entry_ad_ids_set)) AS `$entry_has_rdt_cid`,
+ replaceOne(arrayFirst(x -> startsWith(x, 'epik='), argMinMerge(raw_sessions_v3.entry_ad_ids_set)), 'epik=', '') AS `$entry_epik`,
+ arrayExists(x -> startsWith(x, 'epik='), argMinMerge(raw_sessions_v3.entry_ad_ids_set)) AS `$entry_has_epik`,
+ replaceOne(arrayFirst(x -> startsWith(x, 'qclid='), argMinMerge(raw_sessions_v3.entry_ad_ids_set)), 'qclid=', '') AS `$entry_qclid`,
+ arrayExists(x -> startsWith(x, 'qclid='), argMinMerge(raw_sessions_v3.entry_ad_ids_set)) AS `$entry_has_qclid`,
+ replaceOne(arrayFirst(x -> startsWith(x, 'sccid='), argMinMerge(raw_sessions_v3.entry_ad_ids_set)), 'sccid=', '') AS `$entry_sccid`,
+ arrayExists(x -> startsWith(x, 'sccid='), argMinMerge(raw_sessions_v3.entry_ad_ids_set)) AS `$entry_has_sccid`,
+ replaceOne(arrayFirst(x -> startsWith(x, '_kx='), argMinMerge(raw_sessions_v3.entry_ad_ids_set)), '_kx=', '') AS `$entry__kx`,
+ arrayExists(x -> startsWith(x, '_kx='), argMinMerge(raw_sessions_v3.entry_ad_ids_set)) AS `$entry_has__kx`,
+ replaceOne(arrayFirst(x -> startsWith(x, 'irclid='), argMinMerge(raw_sessions_v3.entry_ad_ids_set)), 'irclid=', '') AS `$entry_irclid`,
+ arrayExists(x -> startsWith(x, 'irclid='), argMinMerge(raw_sessions_v3.entry_ad_ids_set)) AS `$entry_has_irclid`,
+ replaceOne(arrayFirst(x -> startsWith(x, 'irclickid='), argMinMerge(raw_sessions_v3.entry_ad_ids_set)), 'irclickid=', '') AS `$entry_irclickid`,
+ arrayExists(x -> startsWith(x, 'irclickid='), argMinMerge(raw_sessions_v3.entry_ad_ids_set)) AS `$entry_has_irclickid`,
+ replaceOne(arrayFirst(x -> startsWith(x, 'utm_id='), argMinMerge(raw_sessions_v3.entry_ad_ids_set)), 'utm_id=', '') AS `$entry_utm_id`,
+ arrayExists(x -> startsWith(x, 'utm_id='), argMinMerge(raw_sessions_v3.entry_ad_ids_set)) AS `$entry_has_utm_id`,
+ replaceOne(arrayFirst(x -> startsWith(x, 'utm_source_platform='), argMinMerge(raw_sessions_v3.entry_ad_ids_set)), 'utm_source_platform=', '') AS `$entry_utm_source_platform`,
+ arrayExists(x -> startsWith(x, 'utm_source_platform='), argMinMerge(raw_sessions_v3.entry_ad_ids_set)) AS `$entry_has_utm_source_platform`,
+ replaceOne(arrayFirst(x -> startsWith(x, 'utm_creative_format='), argMinMerge(raw_sessions_v3.entry_ad_ids_set)), 'utm_creative_format=', '') AS `$entry_utm_creative_format`,
+ arrayExists(x -> startsWith(x, 'utm_creative_format='), argMinMerge(raw_sessions_v3.entry_ad_ids_set)) AS `$entry_has_utm_creative_format`,
+ replaceOne(arrayFirst(x -> startsWith(x, 'utm_marketing_tactic='), argMinMerge(raw_sessions_v3.entry_ad_ids_set)), 'utm_marketing_tactic=', '') AS `$entry_utm_marketing_tactic`,
+ arrayExists(x -> startsWith(x, 'utm_marketing_tactic='), argMinMerge(raw_sessions_v3.entry_ad_ids_set)) AS `$entry_has_utm_marketing_tactic`,
+ replaceOne(arrayFirst(x -> startsWith(x, 'yclid='), argMinMerge(raw_sessions_v3.entry_ad_ids_set)), 'yclid=', '') AS `$entry_yclid`,
+ arrayExists(x -> startsWith(x, 'yclid='), argMinMerge(raw_sessions_v3.entry_ad_ids_set)) AS `$entry_has_yclid`,
+ replaceOne(arrayFirst(x -> startsWith(x, 'tblci='), argMinMerge(raw_sessions_v3.entry_ad_ids_set)), 'tblci=', '') AS `$entry_tblci`,
+ arrayExists(x -> startsWith(x, 'tblci='), argMinMerge(raw_sessions_v3.entry_ad_ids_set)) AS `$entry_has_tblci`,
+ replaceOne(arrayFirst(x -> startsWith(x, 'dicbo='), argMinMerge(raw_sessions_v3.entry_ad_ids_set)), 'dicbo=', '') AS `$entry_dicbo`,
+ arrayExists(x -> startsWith(x, 'dicbo='), argMinMerge(raw_sessions_v3.entry_ad_ids_set)) AS `$entry_has_dicbo`,
+ replaceOne(arrayFirst(x -> startsWith(x, 'ndclid='), argMinMerge(raw_sessions_v3.entry_ad_ids_set)), 'ndclid=', '') AS `$entry_ndclid`,
+ arrayExists(x -> startsWith(x, 'ndclid='), argMinMerge(raw_sessions_v3.entry_ad_ids_set)) AS `$entry_has_ndclid`,
+ replaceOne(arrayFirst(x -> startsWith(x, 'cjevent='), argMinMerge(raw_sessions_v3.entry_ad_ids_set)), 'cjevent=', '') AS `$entry_cjevent`,
+ arrayExists(x -> startsWith(x, 'cjevent='), argMinMerge(raw_sessions_v3.entry_ad_ids_set)) AS `$entry_has_cjevent`,
+ replaceOne(arrayFirst(x -> startsWith(x, 'awc='), argMinMerge(raw_sessions_v3.entry_ad_ids_set)), 'awc=', '') AS `$entry_awc`,
+ arrayExists(x -> startsWith(x, 'awc='), argMinMerge(raw_sessions_v3.entry_ad_ids_set)) AS `$entry_has_awc`,
+ replaceOne(arrayFirst(x -> startsWith(x, 'syclid='), argMinMerge(raw_sessions_v3.entry_ad_ids_set)), 'syclid=', '') AS `$entry_syclid`,
+ arrayExists(x -> startsWith(x, 'syclid='), argMinMerge(raw_sessions_v3.entry_ad_ids_set)) AS `$entry_has_syclid`,
+ replaceOne(arrayFirst(x -> startsWith(x, '_bhlid='), argMinMerge(raw_sessions_v3.entry_ad_ids_set)), '_bhlid=', '') AS `$entry__bhlid`,
+ arrayExists(x -> startsWith(x, '_bhlid='), argMinMerge(raw_sessions_v3.entry_ad_ids_set)) AS `$entry_has__bhlid`,
+ replaceOne(arrayFirst(x -> startsWith(x, 'rtid='), argMinMerge(raw_sessions_v3.entry_ad_ids_set)), 'rtid=', '') AS `$entry_rtid`,
+ arrayExists(x -> startsWith(x, 'rtid='), argMinMerge(raw_sessions_v3.entry_ad_ids_set)) AS `$entry_has_rtid`,
+ replaceOne(arrayFirst(x -> startsWith(x, 'oppref='), argMinMerge(raw_sessions_v3.entry_ad_ids_set)), 'oppref=', '') AS `$entry_oppref`,
+ arrayExists(x -> startsWith(x, 'oppref='), argMinMerge(raw_sessions_v3.entry_ad_ids_set)) AS `$entry_has_oppref`,
+ replaceOne(arrayFirst(x -> startsWith(x, 'bfcid='), argMinMerge(raw_sessions_v3.entry_ad_ids_set)), 'bfcid=', '') AS `$entry_bfcid`,
+ arrayExists(x -> startsWith(x, 'bfcid='), argMinMerge(raw_sessions_v3.entry_ad_ids_set)) AS `$entry_has_bfcid`,
raw_sessions_v3.session_id_v7 AS session_id_v7
FROM raw_sessions_v3
WHERE and(equals(raw_sessions_v3.team_id, 99999), equals(raw_sessions_v3.session_timestamp, fromUnixTimestamp64Milli(toUInt64(bitShiftRight(toUInt128(accurateCastOrNull('00000000-0000-0000-0000-000000000000', 'UUID')), 80)))))
diff --git a/posthog/hogql/database/schema/test/test_sessions_v3.py b/posthog/hogql/database/schema/test/test_sessions_v3.py
index 80bbe2538448..b49b8277c149 100644
--- a/posthog/hogql/database/schema/test/test_sessions_v3.py
+++ b/posthog/hogql/database/schema/test/test_sessions_v3.py
@@ -1,5 +1,5 @@
import uuid
-from time import time_ns
+from datetime import datetime, timedelta
import pytest
from posthog.test.base import (
@@ -24,6 +24,7 @@ from posthog.hogql.database.schema.sessions_v3 import (
from posthog.hogql.parser import parse_select
from posthog.hogql.query import execute_hogql_query
+from posthog.session_recordings.queries.test.session_replay_sql import produce_replay_summary
from posthog.uuidt import uuid7
from products.event_definitions.backend.models.property_definition import PropertyType
@@ -130,14 +131,44 @@ class TestSessionsV3(ClickhouseTestMixin, APIBaseTest):
0, # just making sure the query runs
)
- def test_channel_type(self):
+ @parameterized.expand(
+ [
+ ("gad_source", {"gad_source": "1"}, "Paid Search"),
+ ("gclid_from_google", {"gclid": "abc", "$referring_domain": "www.google.com"}, "Paid Search"),
+ ("empty_gclid_is_not_a_click", {"gclid": "", "$referring_domain": "www.google.com"}, "Organic Search"),
+ ("gad_source_without_referrer", {"gad_source": "1", "$referring_domain": "$direct"}, "Paid Search"),
+ ("fbclid_without_referrer", {"fbclid": "xyz", "$referring_domain": "$direct"}, "Organic Social"),
+ (
+ "unknown_source_stripped_referrer",
+ {"utm_source": "newsletter-2026", "$referring_domain": "$direct"},
+ "Direct",
+ ),
+ ("unknown_referrer", {"$referring_domain": "blog.example.org"}, "Referral"),
+ (
+ "self_referral_with_paid_utms",
+ {"$referring_domain": "example.com", "utm_source": "google", "utm_medium": "cpc"},
+ "Paid Search",
+ ),
+ (
+ "self_referral_with_unknown_source",
+ {"$referring_domain": "example.com", "utm_source": "newsletter-2026"},
+ "Referral",
+ ),
+ (
+ "msclkid_only_in_url",
+ {"$current_url": "https://example.com/?msclkid=m1", "$referring_domain": "$direct"},
+ "Direct",
+ ),
+ ]
+ )
+ def test_channel_type(self, _name, properties, expected_channel_type):
session_id = str(uuid7())
_create_event(
event="$pageview",
team=self.team,
distinct_id="d1",
- properties={"gad_source": "1", "$session_id": session_id},
+ properties={**properties, "$session_id": session_id},
)
response = self.__execute(
@@ -150,9 +181,105 @@ class TestSessionsV3(ClickhouseTestMixin, APIBaseTest):
result = (response.results or [])[0]
self.assertEqual(
result[0],
- "Paid Search",
+ expected_channel_type,
+ )
+
+ @parameterized.expand(
+ [
+ (
+ "utm_only_in_url",
+ [("$pageview", 0, {"$current_url": "https://example.com/?utm_source=newsletter&utm_campaign=fall"})],
+ (None, None, "Direct"),
+ ),
+ (
+ "first_event_is_not_a_pageview",
+ [
+ ("$feature_flag_called", 0, {}),
+ ("$pageview", 2, {"utm_source": "google", "utm_campaign": "spring"}),
+ ],
+ ("google", "spring", "Organic Search"),
+ ),
+ (
+ # entry_utm_* skip a NULL on the landing page view and take a later value, while the channel type
+ # tuple keeps the landing page view's empty values, so the two disagree for this session
+ "utm_added_on_second_pageview",
+ [
+ ("$pageview", 0, {}),
+ ("$pageview", 1, {"utm_source": "google", "utm_medium": "cpc", "utm_campaign": "spring"}),
+ ],
+ ("google", "spring", "Direct"),
+ ),
+ (
+ "two_campaigns_keep_the_first",
+ [
+ ("$pageview", 0, {"utm_source": "newsletter", "utm_campaign": "A"}),
+ ("$pageview", 30, {"utm_source": "newsletter", "utm_campaign": "B"}),
+ ],
+ ("newsletter", "A", "Direct"),
+ ),
+ (
+ "sdk_session_entry_properties_are_not_read",
+ [("$pageview", 0, {"$session_entry_utm_source": "email"})],
+ (None, None, "Direct"),
+ ),
+ ]
+ )
+ def test_entry_utm_and_channel_type_across_events(self, _name, events, expected):
+ session_id = str(uuid7("2024-03-08T00:00:00+00:00"))
+ start = datetime.fromisoformat("2024-03-08T00:00:00+00:00")
+ for event, offset_seconds, properties in events:
+ _create_event(
+ event=event,
+ team=self.team,
+ distinct_id="d1",
+ properties={"$referring_domain": "$direct", **properties, "$session_id": session_id},
+ timestamp=start + timedelta(seconds=offset_seconds),
+ )
+
+ response = self.__execute(
+ parse_select(
+ "select $entry_utm_source, $entry_utm_campaign, $channel_type from sessions where session_id = {session_id}",
+ placeholders={"session_id": ast.Constant(value=session_id)},
+ ),
+ )
+
+ self.assertEqual((response.results or [])[0], expected)
+
+ @parameterized.expand([("replay_starts_before_first_pageview", True), ("replay_only", False)])
+ def test_replay_rows_shape_the_session(self, _name, with_pageview):
+ session_id = str(uuid7("2024-03-08T00:00:00+00:00"))
+ produce_replay_summary(
+ team_id=self.team.pk,
+ distinct_id="d1",
+ session_id=session_id,
+ first_timestamp="2024-03-08T00:00:00",
+ last_timestamp="2024-03-08T00:00:00",
+ ensure_analytics_event_in_session=False,
+ )
+ if with_pageview:
+ _create_event(
+ event="$pageview",
+ team=self.team,
+ distinct_id="d1",
+ properties={"$session_id": session_id, "$current_url": "https://example.com/"},
+ timestamp="2024-03-08T00:00:05",
+ )
+
+ response = self.__execute(
+ parse_select(
+ "select $start_timestamp, $session_duration, $pageview_count, $is_bounce, $has_replay_events from sessions where session_id = {session_id}",
+ placeholders={"session_id": ast.Constant(value=session_id)},
+ ),
)
+ start_timestamp, duration, pageview_count, is_bounce, has_replay_events = (response.results or [])[0]
+ self.assertEqual(start_timestamp.isoformat()[:19], "2024-03-08T00:00:00")
+ self.assertTrue(has_replay_events)
+ if with_pageview:
+ self.assertEqual((duration, pageview_count, is_bounce), (5, 1, True))
+ else:
+ self.assertEqual((duration, pageview_count, is_bounce), (0, 0, None))
+
def test_session_dot_channel_type(self):
session_id = str(uuid7())
@@ -227,7 +354,7 @@ class TestSessionsV3(ClickhouseTestMixin, APIBaseTest):
self.assertEqual(response.results or [], [(0, 0, 0)])
def test_counts(self):
- s1 = str(uuid7())
+ s1 = str(uuid7("2023-12-02T00:00:00+00:00"))
d1 = "d1"
_create_event(
event="$pageview",
@@ -261,7 +388,7 @@ class TestSessionsV3(ClickhouseTestMixin, APIBaseTest):
self.assertEqual(response.results or [], [(1, 2, 3)])
def test_idempotent_event_counts(self):
- s1 = str(uuid7())
+ s1 = str(uuid7("2023-12-02T00:00:00+00:00"))
d1 = "d1"
pageview_uuid = str(uuid.uuid4())
autocapture_uuid = str(uuid.uuid4())
@@ -303,7 +430,7 @@ class TestSessionsV3(ClickhouseTestMixin, APIBaseTest):
self.assertEqual(response.results or [], [(1, 1, 1)])
def test_page_screen_autocapture_count_up_to(self):
- time = time_ns() // (10**6)
+ time = int(datetime.fromisoformat("2023-12-02T00:00:00+00:00").timestamp() * 1000)
# two pageviews
s1 = str(uuid7(time))
@@ -407,7 +534,7 @@ class TestSessionsV3(ClickhouseTestMixin, APIBaseTest):
assert results == [(2, False), (1, True), (1, False), (2, False), (1, False), (0, True)]
def test_bounce_rate(self):
- time = time_ns() // (10**6)
+ time = int(datetime.fromisoformat("2023-12-02T00:00:00+00:00").timestamp() * 1000)
# ensure the sessions ids are sortable by giving them different time components
s1a = str(uuid7(time))
s1b = str(uuid7(time + 1))
@@ -517,7 +644,7 @@ class TestSessionsV3(ClickhouseTestMixin, APIBaseTest):
]
def test_custom_bounce_rate_duration(self):
- time = time_ns() // (10**6)
+ time = int(datetime.fromisoformat("2023-12-11T00:00:00+00:00").timestamp() * 1000)
# ensure the sessions ids are sortable by giving them different time components
s = str(uuid7(time))
@@ -694,50 +821,88 @@ class TestGetLazySessionProperties(ClickhouseTestMixin, APIBaseTest):
"$end_hostname",
"$end_pathname",
"$end_timestamp",
+ "$entry__bhlid",
"$entry__kx",
+ "$entry_awc",
+ "$entry_bfcid",
+ "$entry_cjevent",
"$entry_current_url",
"$entry_dclid",
+ "$entry_dicbo",
"$entry_epik",
"$entry_fbclid",
+ "$entry_gad_campaignid",
"$entry_gad_source",
"$entry_gbraid",
"$entry_gclid",
"$entry_gclsrc",
+ "$entry_has__bhlid",
"$entry_has__kx",
+ "$entry_has_awc",
+ "$entry_has_bfcid",
+ "$entry_has_cjevent",
"$entry_has_dclid",
+ "$entry_has_dicbo",
"$entry_has_epik",
"$entry_has_fbclid",
+ "$entry_has_gad_campaignid",
"$entry_has_gbraid",
"$entry_has_gclid",
"$entry_has_gclsrc",
"$entry_has_igshid",
+ "$entry_has_irclickid",
"$entry_has_irclid",
"$entry_has_li_fat_id",
"$entry_has_mc_cid",
"$entry_has_msclkid",
+ "$entry_has_ndclid",
+ "$entry_has_oppref",
"$entry_has_qclid",
+ "$entry_has_rdt_cid",
+ "$entry_has_rtid",
"$entry_has_sccid",
+ "$entry_has_srsltid",
+ "$entry_has_syclid",
+ "$entry_has_tblci",
"$entry_has_ttclid",
"$entry_has_twclid",
+ "$entry_has_utm_creative_format",
+ "$entry_has_utm_id",
+ "$entry_has_utm_marketing_tactic",
+ "$entry_has_utm_source_platform",
"$entry_has_wbraid",
+ "$entry_has_yclid",
"$entry_hostname",
"$entry_igshid",
+ "$entry_irclickid",
"$entry_irclid",
"$entry_li_fat_id",
"$entry_mc_cid",
"$entry_msclkid",
+ "$entry_ndclid",
+ "$entry_oppref",
"$entry_pathname",
"$entry_qclid",
+ "$entry_rdt_cid",
"$entry_referring_domain",
+ "$entry_rtid",
"$entry_sccid",
+ "$entry_srsltid",
+ "$entry_syclid",
+ "$entry_tblci",
"$entry_ttclid",
"$entry_twclid",
"$entry_utm_campaign",
"$entry_utm_content",
+ "$entry_utm_creative_format",
+ "$entry_utm_id",
+ "$entry_utm_marketing_tactic",
"$entry_utm_medium",
"$entry_utm_source",
+ "$entry_utm_source_platform",
"$entry_utm_term",
"$entry_wbraid",
+ "$entry_yclid",
"$is_bounce",
"$last_external_click_url",
"$pageview_count",
@@ -764,32 +929,45 @@ class TestGetLazySessionProperties(ClickhouseTestMixin, APIBaseTest):
def test_source(self):
results = get_lazy_session_table_properties_v3("source")
self.assertEqual(
- results,
+ [result["name"] for result in results],
[
- {
- "id": "$entry_utm_source",
- "is_numerical": False,
- "is_seen_on_filtered_events": None,
- "name": "$entry_utm_source",
- "property_type": PropertyType.String,
- "tags": [],
- },
- {
- "id": "$entry_gad_source",
- "is_numerical": False,
- "is_seen_on_filtered_events": None,
- "name": "$entry_gad_source",
- "property_type": PropertyType.String,
- "tags": [],
- },
+ "$entry_utm_source",
+ "$entry_gad_source",
+ "$entry_utm_source_platform",
+ "$entry_has_utm_source_platform",
],
)
+ self.assertEqual(
+ results[0],
+ {
+ "id": "$entry_utm_source",
+ "is_numerical": False,
+ "is_seen_on_filtered_events": None,
+ "name": "$entry_utm_source",
+ "property_type": PropertyType.String,
+ "tags": [],
+ },
+ )
def test_entry_utm(self):
results = get_lazy_session_table_properties_v3("entry utm")
self.assertEqual(
[result["name"] for result in results],
- ["$entry_utm_source", "$entry_utm_campaign", "$entry_utm_medium", "$entry_utm_term", "$entry_utm_content"],
+ [
+ "$entry_utm_source",
+ "$entry_utm_campaign",
+ "$entry_utm_medium",
+ "$entry_utm_term",
+ "$entry_utm_content",
+ "$entry_utm_id",
+ "$entry_has_utm_id",
+ "$entry_utm_source_platform",
+ "$entry_has_utm_source_platform",
+ "$entry_utm_creative_format",
+ "$entry_has_utm_creative_format",
+ "$entry_utm_marketing_tactic",
+ "$entry_has_utm_marketing_tactic",
+ ],
)
def test_can_get_values_for_all(self):
diff --git a/posthog/models/raw_sessions/migrations_v3.py b/posthog/models/raw_sessions/migrations_v3.py
index b105e0c19be5..5191c1109e5b 100644
--- a/posthog/models/raw_sessions/migrations_v3.py
+++ b/posthog/models/raw_sessions/migrations_v3.py
@@ -6,9 +6,11 @@ DROP COLUMN IF EXISTS person_id
"""
+# These statements replay on fresh clusters against a table created from today's base SQL,
+# so an AFTER clause may only name a column that base SQL still creates.
ADD_HAS_REPLAY_EVENTS = """
ALTER TABLE {table_name}
-ADD COLUMN IF NOT EXISTS has_replay_events SimpleAggregateFunction(max, Boolean) AFTER flag_values;
+ADD COLUMN IF NOT EXISTS has_replay_events SimpleAggregateFunction(max, Boolean);
"""
SPLIT_BOUNCE_RATE = """
@@ -23,7 +25,7 @@ ADD COLUMN IF NOT EXISTS has_autocapture SimpleAggregateFunction(max, Boolean) A
ADD_EVENT_NAMES = """
ALTER TABLE {table_name}
-ADD COLUMN IF NOT EXISTS event_names SimpleAggregateFunction(groupUniqArrayArray, Array(String)) AFTER flag_values
+ADD COLUMN IF NOT EXISTS event_names SimpleAggregateFunction(groupUniqArrayArray, Array(String))
"""
@@ -35,7 +37,7 @@ ADD INDEX IF NOT EXISTS event_names_bloom_filter event_names TYPE bloom_filter()
ADD_FLAG_KEYS = """
ALTER TABLE {table_name}
-ADD COLUMN IF NOT EXISTS flag_keys SimpleAggregateFunction(groupUniqArrayArray, Array(String)) AFTER flag_values
+ADD COLUMN IF NOT EXISTS flag_keys SimpleAggregateFunction(groupUniqArrayArray, Array(String))
"""
@@ -91,3 +93,21 @@ ADD_FLAG_KEY_VALUES_BLOOM_FILTER = """
ALTER TABLE {table_name}
ADD INDEX IF NOT EXISTS flag_key_values_bloom_filter flag_key_values TYPE bloom_filter() GRANULARITY 1
"""
+
+
+DROP_FLAG_KEYS = """
+ALTER TABLE {table_name}
+DROP COLUMN IF EXISTS flag_keys
+"""
+
+
+DROP_FLAG_KEYS_BLOOM_FILTER = """
+ALTER TABLE {table_name}
+DROP INDEX IF EXISTS flag_keys_bloom_filter
+"""
+
+
+DROP_FLAG_KEY_VALUES_BLOOM_FILTER = """
+ALTER TABLE {table_name}
+DROP INDEX IF EXISTS flag_key_values_bloom_filter
+"""
diff --git a/posthog/models/raw_sessions/sessions_v3.py b/posthog/models/raw_sessions/sessions_v3.py
index 7752a6d1c460..50b5c14f5e93 100644
--- a/posthog/models/raw_sessions/sessions_v3.py
+++ b/posthog/models/raw_sessions/sessions_v3.py
@@ -24,7 +24,7 @@ table should always aggregate again on session_id (the HogQL session table will
don't need to consider this).
Upgrades over v2:
-* Has a property map for storing lower-tier ad ids, making it easier to add new ad ids in the future
+* Stores lower-tier ad ids as a flat 'key=value' array, making it easier to add new ad ids in the future
* Stores presence of ad ids separately from the value, so e.g. channel type calculations only need to read 1 bit instead of a gclid string up to 100 chars
* Parses JSON only once per event rather than once per column per event, saving CPU usage
* Removes a lot of deprecated fields that are no longer used
@@ -148,8 +148,8 @@ CREATE TABLE IF NOT EXISTS {table_name}
entry_has_gclid AggregateFunction(argMin, Boolean, DateTime64(6, 'UTC')),
entry_has_fbclid AggregateFunction(argMin, Boolean, DateTime64(6, 'UTC')),
- -- for lower-tier ad ids, just put them in a map, and set of the ones present
- entry_ad_ids_map AggregateFunction(argMin, Map(String, String), DateTime64(6, 'UTC')),
+ -- lower-tier ad ids as a flat 'key=value' array - like flag_key_values, a per-key map
+ -- state made merges much more expensive
entry_ad_ids_set AggregateFunction(argMin, Array(String), DateTime64(6, 'UTC')),
-- channel type properties tuple - to reduce redundant reading of the timestamp when loading all of these columns
@@ -157,12 +157,13 @@ CREATE TABLE IF NOT EXISTS {table_name}
entry_channel_type_properties AggregateFunction(argMin, Tuple(Nullable(String), Nullable(String), Nullable(String), Nullable(String), Boolean, Boolean, Nullable(String)), DateTime64(6, 'UTC')),
-- Count pageview, autocapture, and screen events for providing totals.
- -- Use uniqExact instead of count, so that inserting events can be idempotent. This is necessary as sometimes we see
+ -- Use uniq instead of count, so that inserting events can be idempotent. This is necessary as sometimes we see
-- events being inserted multiple times to be deduped later, but that can trigger multiple rows here.
-- Additionally, idempotency is useful for backfilling, as we can just reinsert the same events without worrying.
- pageview_uniq AggregateFunction(uniqExact, Nullable(UUID)),
- autocapture_uniq AggregateFunction(uniqExact, Nullable(UUID)),
- screen_uniq AggregateFunction(uniqExact, Nullable(UUID)),
+ -- uniq (not uniqExact) keeps the state small; per-session counts are tiny so it is exact in practice.
+ pageview_uniq AggregateFunction(uniq, Nullable(UUID)),
+ autocapture_uniq AggregateFunction(uniq, Nullable(UUID)),
+ screen_uniq AggregateFunction(uniq, Nullable(UUID)),
-- As a performance optimisation, also keep track of the uniq events for all of these combined.
-- This is a much more efficient way of calculating the bounce rate, as >2 means not a bounce
@@ -171,8 +172,8 @@ CREATE TABLE IF NOT EXISTS {table_name}
-- Flags - store every seen 'key=value' pair per flag. A flat array instead of a
-- groupUniqArrayMap state because per-key sub-aggregators made merges ~8-37x more expensive.
+ -- Key-only searches use the text index below (tokenized on '='), so no separate keys column.
flag_key_values SimpleAggregateFunction(groupUniqArrayArray(10000), Array(String)),
- flag_keys SimpleAggregateFunction(groupUniqArrayArray, Array(String)),
-- Event names - store unique event names seen in this session
event_names SimpleAggregateFunction(groupUniqArrayArray(2000), Array(String)),
@@ -207,8 +208,7 @@ def SHARDED_RAW_SESSIONS_TABLE_SQL_V3():
-- Indexes
INDEX event_names_bloom_filter event_names TYPE bloom_filter() GRANULARITY 1,
- INDEX flag_key_values_bloom_filter flag_key_values TYPE bloom_filter() GRANULARITY 1,
- INDEX flag_keys_bloom_filter flag_keys TYPE bloom_filter() GRANULARITY 1,
+ INDEX flag_key_values_text flag_key_values TYPE text(tokenizer = splitByString(['='])),
INDEX hosts_bloom_filter hosts TYPE bloom_filter() GRANULARITY 1,
INDEX emails_bloom_filter emails TYPE bloom_filter() GRANULARITY 1
) ENGINE = {engine}""",
@@ -247,10 +247,29 @@ def ALTER_SHARDED_RAW_SESSIONS_TABLE_SETTINGS_V3():
# Re-exported from the Django-free posthog.raw_sessions_v3_ad_ids module so the HogQL schema can
# use it without booting Django; kept importable here for existing callers.
-from posthog.raw_sessions_v3_ad_ids import SESSION_V3_LOWER_TIER_AD_IDS # noqa: E402
+from posthog.raw_sessions_v3_ad_ids import SESSION_V3_LOWER_TIER_AD_IDS, session_ad_id_url_params # noqa: E402
new_line = "\n"
+
+def _ad_id_value_sql(ad_id: str) -> str:
+ # The SDKs only promote a fixed set of URL params to event properties, so fall back to
+ # parsing the event's URL — this is what lets a new ad id apply to backfilled history.
+ # The lookups read _attribution_url, which is empty outside $pageview/$screen, so the
+ # per-event string scans stay off the bulk of the ingestion hot path.
+ # Form decoding, not plain: a query string spells a space as '+', and the SDK decodes it that
+ # way before promoting the property, so plain decoding would split one campaign into two values.
+ # left() caps what a client can pad into the URL; observed click IDs stay under ~120 chars
+ # toValidUTF8 wraps the truncation because both inner steps can emit invalid UTF-8:
+ # decodeURLFormComponent turns '%FF' into a raw byte, and left() counts bytes so it can cut a
+ # multi-byte character in half. Stored bytes then break every reader that expects text.
+ url_lookups = ", ".join(
+ f"nullIf(toValidUTF8(left(decodeURLFormComponent(extractURLParameter(_attribution_url, '{param}')), 2048)), '')"
+ for param in session_ad_id_url_params(ad_id)
+ )
+ return f"coalesce(nullIf(tupleElement(p, '{ad_id}'), ''), {url_lookups}) as {ad_id}"
+
+
# See https://kb.altinity.com/altinity-kb-queries-and-syntax/jsonextract-to-parse-many-attributes-at-a-time/
# Or https://posthog.slack.com/archives/C02JQ320FV3/p1721406540313379?thread_ts=1721334861.073739&cid=C02JQ320FV3
PROPERTIES = f"""
@@ -267,7 +286,7 @@ PROPERTIES = f"""
`$geoip_country_code` Nullable(String),
`$geoip_subdivision_1_code` Nullable(String),
`$geoip_subdivision_1_name` Nullable(String),
- `$geoip_subdivision_city_name` Nullable(String),
+ `$geoip_city_name` Nullable(String),
`$geoip_time_zone` Nullable(String),
`$referring_domain` Nullable(String),
`utm_source` Nullable(String),
@@ -283,6 +302,7 @@ PROPERTIES = f"""
)') as p,
JSONExtractString(person_properties, 'email') as _person_email,
tupleElement(p, '$current_url') as _current_url,
+ if(event = '$pageview' OR event = '$screen', left(coalesce(_current_url, ''), 16384), '') as _attribution_url,
tupleElement(p, '$external_click_url') as _external_click_url,
tupleElement(p, '$browser') as _browser,
tupleElement(p, '$browser_version') as _browser_version,
@@ -294,7 +314,7 @@ PROPERTIES = f"""
tupleElement(p, '$geoip_country_code') as _geoip_country_code,
tupleElement(p, '$geoip_subdivision_1_code') as _geoip_subdivision_1_code,
tupleElement(p, '$geoip_subdivision_1_name') as _geoip_subdivision_1_name,
- tupleElement(p, '$geoip_subdivision_city_name') as _geoip_subdivision_city_name,
+ tupleElement(p, '$geoip_city_name') as _geoip_subdivision_city_name,
tupleElement(p, '$geoip_time_zone') as _geoip_time_zone,
tupleElement(p, '$referring_domain') as _referring_domain,
tupleElement(p, 'utm_source') as _utm_source,
@@ -305,12 +325,9 @@ PROPERTIES = f"""
tupleElement(p, 'gclid') as _gclid,
tupleElement(p, 'gad_source') as _gad_source,
tupleElement(p, 'fbclid') as _fbclid,
-{f",{new_line}".join([f" tupleElement(p, '{ad_id}') as {ad_id}" for ad_id in SESSION_V3_LOWER_TIER_AD_IDS])},
- CAST(mapFilter((k, v) -> v IS NOT NULL, map(
-{f",{new_line}".join([f" '{ad_id}', {ad_id}" for ad_id in SESSION_V3_LOWER_TIER_AD_IDS])}
- )) AS Map(String, String)) as ad_ids_map,
+{f",{new_line}".join([f" {_ad_id_value_sql(ad_id)}" for ad_id in SESSION_V3_LOWER_TIER_AD_IDS])},
CAST(arrayFilter(x -> x IS NOT NULL, [
-{f",{new_line}".join([f" if({ad_id} IS NOT NULL, '{ad_id}', NULL)" for ad_id in SESSION_V3_LOWER_TIER_AD_IDS])}
+{f",{new_line}".join([f" if({ad_id} IS NOT NULL, concat('{ad_id}=', {ad_id}), NULL)" for ad_id in SESSION_V3_LOWER_TIER_AD_IDS])}
]) AS Array(String)) as ad_ids_set,
tupleElement(p, '$host') as _host"""
@@ -368,21 +385,21 @@ SELECT
initializeAggregation('argMinState', _fbclid, pageview_prio_timestamp_min) as entry_fbclid,
-- has gclid/fbclid for reading fewer bytes when calculating channel type
- initializeAggregation('argMinState', _gclid IS NOT NULL, pageview_prio_timestamp_min) as entry_has_gclid,
- initializeAggregation('argMinState', _fbclid IS NOT NULL, pageview_prio_timestamp_min) as entry_has_fbclid,
+ -- clients send "gclid": "" and that is not a click id, so an empty value must not count as present
+ initializeAggregation('argMinState', nullIf(_gclid, '') IS NOT NULL, pageview_prio_timestamp_min) as entry_has_gclid,
+ initializeAggregation('argMinState', nullIf(_fbclid, '') IS NOT NULL, pageview_prio_timestamp_min) as entry_has_fbclid,
-- other ad ids
- initializeAggregation('argMinState', ad_ids_map, pageview_prio_timestamp_min) as entry_ad_ids_map,
initializeAggregation('argMinState', ad_ids_set, pageview_prio_timestamp_min) as entry_ad_ids_set,
-- channel type
- initializeAggregation('argMinState', tuple(_utm_source, _utm_medium, _utm_campaign, _referring_domain, _gclid IS NOT NULL, _fbclid IS NOT NULL, _gad_source), pageview_prio_timestamp_min) as entry_channel_type_properties,
+ initializeAggregation('argMinState', tuple(_utm_source, _utm_medium, _utm_campaign, _referring_domain, nullIf(_gclid, '') IS NOT NULL, nullIf(_fbclid, '') IS NOT NULL, _gad_source), pageview_prio_timestamp_min) as entry_channel_type_properties,
-- counts
- initializeAggregation('uniqExactState', if(event='$pageview', uuid, NULL)) as pageview_uniq,
- initializeAggregation('uniqExactState', if(event='$autocapture', uuid, NULL)) as autocapture_uniq,
- initializeAggregation('uniqExactState', if(event='$screen', uuid, NULL)) as screen_uniq,
+ initializeAggregation('uniqState', if(event='$pageview', uuid, NULL)) as pageview_uniq,
+ initializeAggregation('uniqState', if(event='$autocapture', uuid, NULL)) as autocapture_uniq,
+ initializeAggregation('uniqState', if(event='$screen', uuid, NULL)) as screen_uniq,
-- perf
initializeAggregation('uniqUpToState(1)', if(event='$pageview' OR event='$screen', uuid, NULL)) as page_screen_uniq_up_to,
@@ -390,7 +407,6 @@ SELECT
-- flags
arrayMap((k, v) -> concat(k, '=', v), mapKeys(properties_group_feature_flags), mapValues(properties_group_feature_flags)) as flag_key_values,
- mapKeys(properties_group_feature_flags) as flag_keys,
-- event names
[event] as event_names,
@@ -404,6 +420,11 @@ SELECT
false as has_replay_events
FROM {source_table} AS source_table
WHERE bitAnd(bitShiftRight(toUInt128(accurateCastOrNull(`$session_id`, 'UUID')), 76), 0xF) == 7 -- has a session id and is valid uuidv7
+-- Clients can send a UUIDv7 session id with a garbage timestamp (1970, year 6339). Each one puts its session in its
+-- own monthly partition, and its events stretch the session duration across months. Keep only events within 31 days
+-- of the session id's timestamp. The bound is symmetric because the client mints the session id with its own clock
+-- while the server corrects event timestamps, so a client clock that runs days ahead is still a real session.
+AND fromUnixTimestamp64Milli(toUInt64(bitShiftRight(`$session_id_uuid`, 80))) BETWEEN timestamp - INTERVAL 31 DAY AND timestamp + INTERVAL 31 DAY
AND {where}
""".format(
source_table=source_table,
@@ -589,16 +610,15 @@ SELECT
initializeAggregation('argMinState', false, max_ts_64) as entry_has_fbclid,
-- other ad ids
- initializeAggregation('argMinState', CAST(map(), 'Map(String, String)'), max_ts_64) as entry_ad_ids_map,
initializeAggregation('argMinState', CAST([], 'Array(String)'), max_ts_64) as entry_ad_ids_set,
-- channel type
initializeAggregation('argMinState', tuple(null_s, null_s, null_s, null_s, false, false, null_s), max_ts_64) as entry_channel_type_properties,
-- counts
- initializeAggregation('uniqExactState', null_uuid) as pageview_uniq,
- initializeAggregation('uniqExactState', null_uuid) as autocapture_uniq,
- initializeAggregation('uniqExactState', null_uuid) as screen_uniq,
+ initializeAggregation('uniqState', null_uuid) as pageview_uniq,
+ initializeAggregation('uniqState', null_uuid) as autocapture_uniq,
+ initializeAggregation('uniqState', null_uuid) as screen_uniq,
-- perf
initializeAggregation('uniqUpToState(1)', null_uuid) as page_screen_uniq_up_to,
@@ -606,7 +626,6 @@ SELECT
-- flags
CAST([], 'Array(String)') as flag_key_values,
- CAST([], 'Array(String)') as flag_keys,
-- event names
CAST([], 'Array(String)') as event_names,
@@ -621,6 +640,9 @@ SELECT
true as has_replay_events
FROM {source_table} AS source_table
WHERE bitAnd(bitShiftRight(toUInt128(accurateCastOrNull(session_id, 'UUID')), 76), 0xF) == 7 -- has a session id and is valid uuidv7
+-- same session timestamp guard as the events SELECT. accurateCastOrNull because accurateCast throws on a malformed
+-- session_id, and ClickHouse can evaluate this condition before the uuidv7 check above filters those rows out
+AND fromUnixTimestamp64Milli(toUInt64(bitShiftRight(toUInt128(accurateCastOrNull(session_id, 'UUID')), 80))) BETWEEN min_first_timestamp - INTERVAL 31 DAY AND min_first_timestamp + INTERVAL 31 DAY
AND {where}
""".format(
source_table=source_table,
@@ -815,15 +837,14 @@ SELECT
argMinMerge(entry_has_gclid) as entry_has_gclid,
argMinMerge(entry_has_fbclid) as entry_has_fbclid,
- argMinMerge(entry_ad_ids_map) as entry_ad_ids_map,
argMinMerge(entry_ad_ids_set) as entry_ad_ids_set,
argMinMerge(entry_channel_type_properties) as entry_channel_type_properties,
-- counts
- uniqExactMerge(pageview_uniq) as pageview_uniq,
- uniqExactMerge(autocapture_uniq) as autocapture_uniq,
- uniqExactMerge(screen_uniq) as screen_uniq,
+ uniqMerge(pageview_uniq) as pageview_uniq,
+ uniqMerge(autocapture_uniq) as autocapture_uniq,
+ uniqMerge(screen_uniq) as screen_uniq,
-- perf
uniqUpToMerge(1)(page_screen_uniq_up_to) as page_screen_uniq_up_to,
@@ -831,7 +852,6 @@ SELECT
-- flags
groupUniqArrayArray(10000)(flag_key_values) as flag_key_values,
- groupUniqArrayArray(flag_keys) as flag_keys,
-- event names
groupUniqArrayArray(2000)(event_names) as event_names,
diff --git a/posthog/raw_sessions_v3_ad_ids.py b/posthog/raw_sessions_v3_ad_ids.py
index bb5e36f4c265..a4a08e4136ec 100644
--- a/posthog/raw_sessions_v3_ad_ids.py
+++ b/posthog/raw_sessions_v3_ad_ids.py
@@ -1,22 +1,60 @@
+import re
+
# Django-free home for the raw-sessions-v3 lower-tier ad-id list. Kept out of
# posthog.models.raw_sessions.sessions_v3 (whose package __init__ boots the Django ORM) so the
# HogQL sessions_v3 schema table can build its field catalog at import without booting Django.
# posthog.models.raw_sessions.sessions_v3 re-exports this for existing callers.
SESSION_V3_LOWER_TIER_AD_IDS = [
- "gclsrc",
- "dclid",
- "gbraid",
- "wbraid",
- "msclkid",
- "twclid",
- "li_fat_id",
- "mc_cid",
- "igshid",
- "ttclid",
- "epik",
- "qclid",
- "sccid",
- "_kx",
- "irclid",
+ "gclsrc", # google ads 360
+ "dclid", # google display ads
+ "gbraid", # google ads, web to app
+ "wbraid", # google ads, app to web
+ "gad_campaignid", # google ads campaign id
+ "srsltid", # google merchant center auto-tagging
+ "msclkid", # microsoft
+ "twclid", # twitter
+ "li_fat_id", # linkedin
+ "mc_cid", # mailchimp campaign id
+ "igshid", # instagram
+ "ttclid", # tiktok
+ "rdt_cid", # reddit
+ "epik", # pinterest
+ "qclid", # quora
+ "sccid", # snapchat
+ "_kx", # klaviyo
+ "irclid", # impact (legacy spelling, kept for property parity)
+ "irclickid", # impact
+ "utm_id", # GA4 campaign id
+ "utm_source_platform", # GA4
+ "utm_creative_format", # GA4
+ "utm_marketing_tactic", # GA4
+ "yclid", # yandex
+ "tblci", # taboola
+ "dicbo", # outbrain
+ "ndclid", # nextdoor
+ "cjevent", # cj affiliate
+ "awc", # awin
+ "syclid", # shopify
+ "_bhlid", # beehiiv
+ "rtid", # rokt
+ "oppref", # openai ads
+ "bfcid", # freebuff
]
+
+# URL query-string spellings that differ from the canonical property key. Snapchat documents
+# all three casings for its click ID (https://businesshelp.snapchat.com/s/article/click-id).
+SESSION_V3_AD_ID_URL_ALIASES: dict[str, list[str]] = {
+ "sccid": ["sccid", "ScCid", "SsCid"],
+}
+
+
+def session_ad_id_url_params(ad_id: str) -> list[str]:
+ return SESSION_V3_AD_ID_URL_ALIASES.get(ad_id, [ad_id])
+
+
+# Every entry is interpolated into ClickHouse DDL as a quoted literal and a bare alias, so a
+# stray quote or metacharacter in a future entry would become injected DDL. Fail at import.
+for _param in SESSION_V3_LOWER_TIER_AD_IDS + [p for aliases in SESSION_V3_AD_ID_URL_ALIASES.values() for p in aliases]:
+ if not re.fullmatch(r"[A-Za-z_][A-Za-z0-9_]*", _param):
+ raise ValueError(f"ad id {_param!r} is not a safe SQL identifier")