:hedgehog: PostHog is the leading platform for building self-driving products. Our developer tools – AI observability, analytics, session replay, flags, experiments, error tracking, logs, and more – capture all the context agents need to diagnose problems, uncover opportunities, and ship fixes. Steer it all from Slack, web, desktop, or the MCP.
Python ab-testingai-analyticsanalyticscdpdata-warehouseexperimentsfeature-flagsjavascriptproduct-analyticspythonreactsession-replaysurveystypescriptweb-analytics
chore(persons): retire one-off person repair jobs and guard dev seed tools on cloud (#126689)
Eli Reisman4cc8200599ab
diff --git a/.semgrep/rules/security/no-direct-kafka-producer.yaml b/.semgrep/rules/security/no-direct-kafka-producer.yamlindex 526df01665dc..c71f4df2f1b8 100644--- a/.semgrep/rules/security/no-direct-kafka-producer.yaml+++ b/.semgrep/rules/security/no-direct-kafka-producer.yaml@@ -24,7 +24,6 @@ rules:- /posthog/kafka_client/test/test_client.py# Existing integration tests that explicitly opt out of the test stub# to exercise a real broker. These are gated behind KAFKA_ROUNDTRIP_TESTS.- - /posthog/dags/tests/test_person_property_reconciliation.py- /posthog/temporal/tests/dlq_replay/test_activities.py- /posthog/temporal/tests/common/test_logger.py# DLQ replay needs per-message header passthrough, which confluent'sdiff --git a/posthog/dags/PERSON_PROPERTY_RECONCILIATION.md b/posthog/dags/PERSON_PROPERTY_RECONCILIATION.mddeleted file mode 100644index 57dc6de1b7ff..000000000000--- a/posthog/dags/PERSON_PROPERTY_RECONCILIATION.md+++ /dev/null@@ -1,401 +0,0 @@-# Person Property Reconciliation Job--This document provides a comprehensive overview of the person property reconciliation job, including architecture diagrams, parameter reference, and example configurations.--For detailed query explanations, see [person_property_reconciliation_query_explanation.md](person_property_reconciliation_query_explanation.md).--## Overview--The reconciliation job fixes person properties that were missed due to a bug where `PERSON_BATCH_WRITING_DB_WRITE_MODE=ASSERT_VERSION` caused `updatePersonAssertVersion()` to not properly merge properties.--The job:--1. Reads events from ClickHouse to find property updates (`$set`, `$set_once`, `$unset`) within a bug window-2. Compares with current person properties in ClickHouse-3. Applies any missed updates to Postgres-4. Publishes updated persons to Kafka for ClickHouse ingestion--## Architecture--### Non-Windowed Mode (Default)--Best for: Smaller teams or short bug windows where a single ClickHouse query can handle the data.--```text-┌─────────────────────────────────────────────────────────────────────────────┐-│ Non-Windowed Flow │-└─────────────────────────────────────────────────────────────────────────────┘-- ┌──────────────────────────────────────────────────────────────────┐- │ ClickHouse: Single Query │- │ ┌────────────────────────────────────────────────────────────┐ │- │ │ get_person_property_updates_from_clickhouse() │ │- │ │ │ │- │ │ • Scans events from bug_window_start to now() │ │- │ │ • Joins with person_distinct_id_overrides (merge handling) │ │- │ │ • Joins with person table for current state │ │- │ │ • Computes diffs IN SQL (efficient) │ │- │ │ • Returns only persons with actual differences │ │- │ └────────────────────────────────────────────────────────────┘ │- └──────────────────────────────────────────────────────────────────┘- │- ▼- ┌───────────────────────────────┐- │ filter_event_person_properties │- │ (resolve set/unset conflicts) │- └───────────────────────────────┘- │- ▼- ┌───────────────────────────────┐- │ process_persons_in_batches │- │ • Update Postgres │- │ • Publish to Kafka │- │ • Commit in batches │- └───────────────────────────────┘-```--### Windowed Batched Mode--Best for: Large teams with millions of persons where a single query would OOM.--```text-┌─────────────────────────────────────────────────────────────────────────────┐-│ Windowed Batched Flow │-└─────────────────────────────────────────────────────────────────────────────┘-- ┌──────────────────────────────────────────────────────────────────┐- │ Step 1: Identify Affected Persons (BOUNDED) │- │ ┌────────────────────────────────────────────────────────────┐ │- │ │ get_affected_person_ids_from_clickhouse() │ │- │ │ │ │- │ │ • Aggregates event properties within the bug window │ │- │ │ • Compares against current person state in CH │ │- │ │ • Returns ONLY person_ids with actual diffs │ │- │ └────────────────────────────────────────────────────────────┘ │- └──────────────────────────────────────────────────────────────────┘- │- ▼- ┌──────────────────────────────────────────────────────────────────┐- │ Step 2: Process Persons in Batches (~10K each) │- │ │- │ FOR EACH person_batch (chunk of ~10K person_ids): │- │ │- │ ┌────────────────────────────────────────────────────────────┐ │- │ │ Step 2a: Windowed Event Aggregation │ │- │ │ │ │- │ │ FOR EACH time_window (1 hour chunks from start to now): │ │- │ │ • Query events filtered to this batch's person_ids │ │- │ │ • Merge into batch accumulator (timestamp-based) │ │- │ └────────────────────────────────────────────────────────────┘ │- │ │ │- │ ▼ │- │ ┌────────────────────────────────────────────────────────────┐ │- │ │ Step 2b: Compare with Person State │ │- │ │ │ │- │ │ • Query current person properties from ClickHouse │ │- │ │ • Filter to actual diffs (set: value differs, etc.) │ │- │ └────────────────────────────────────────────────────────────┘ │- │ │ │- │ ▼ │- │ ┌────────────────────────────────────────────────────────────┐ │- │ │ Step 2c: Process & Commit Batch │ │- │ │ │ │- │ │ • Filter set/unset conflicts │ │- │ │ • Update Postgres │ │- │ │ • Publish to Kafka │ │- │ │ • Commit transaction │ │- │ │ • Clear memory for next batch │ │- │ └────────────────────────────────────────────────────────────┘ │- │ │- │ END FOR EACH person_batch │- └──────────────────────────────────────────────────────────────────┘-```--### Why Windowed Batched Mode?--Step 1 aggregates event properties within the bug window and compares against current person state,-returning only person_ids with actual diffs.-Step 2 then reads events up to `now()` to capture all property updates for those persons.--This prevents the problem where the original windowed implementation returned ALL persons-with property-setting events (regardless of whether they had diffs), causing 100x over-selection and OOM.--## Parameter Reference--### Job Configuration (`PersonPropertyReconciliationConfig`)--| Parameter | Type | Default | Description |-| ------------------------------------ | ------------------- | -------- | ------------------------------------------------------------------------ |-| `bug_window_start` | `str` | Required | Start of bug window (ClickHouse format: "YYYY-MM-DD HH:MM:SS", UTC) |-| `bug_window_end` | `str \| None` | `None` | End of bug window. Required if `team_ids` not supplied |-| `team_ids` | `list[int] \| None` | `None` | Explicit list of team IDs to process |-| `min_team_id` | `int \| None` | `None` | Minimum team_id (inclusive) for range scanning |-| `max_team_id` | `int \| None` | `None` | Maximum team_id (inclusive) for range scanning |-| `exclude_team_ids` | `list[int] \| None` | `None` | Team IDs to exclude from processing |-| `dry_run` | `bool` | `False` | Log changes without applying to Postgres |-| `backup_enabled` | `bool` | `True` | Store before/after state in backup table |-| `batch_size` | `int` | `100` | Commit Postgres transaction every N persons (0 = single commit) |-| `teams_per_chunk` | `int` | `100` | Teams per k8s task (reduces pod overhead) |-| `team_ch_props_fetch_window_seconds` | `int` | `0` | **Mode selector**: 0 = non-windowed, >0 = windowed with N-second windows |-| `person_batch_size` | `int` | `10000` | Persons per batch in windowed mode (keeps IN clauses under limit) |--### Sensor Configuration (`ReconciliationSchedulerConfig`)--The sensor automates launching multiple reconciliation jobs across teams.--| Parameter | Type | Default | Description |-| ------------------------------------ | ------------------- | ----------------------- | ----------------------------------------- |-| **Team Selection** | | | |-| `team_ids` | `list[int] \| None` | `None` | Explicit list of team IDs (Option 1) |-| `range_start` | `int \| None` | `None` | First team_id for range scan (Option 2) |-| `range_end` | `int \| None` | `None` | Last team_id for range scan (Option 2) |-| **Concurrency** | | | |-| `chunk_size` | `int` | `1000` | Teams per job run |-| `max_concurrent_jobs` | `int` | `5` | Max reconciliation jobs at once (cap: 50) |-| `max_concurrent_tasks` | `int` | `10` | Max k8s pods per job (cap: 100) |-| **Bug Window** | | | |-| `bug_window_start` | `str` | Required | Start of bug window |-| `bug_window_end` | `str` | Required | End of bug window |-| **Processing** | | | |-| `dry_run` | `bool` | `False` | Log changes without applying |-| `backup_enabled` | `bool` | `True` | Store before/after state |-| `batch_size` | `int` | `100` | Postgres commit batch size |-| `teams_per_chunk` | `int` | `100` | Teams per k8s task |-| `team_ch_props_fetch_window_seconds` | `int` | `0` | Mode selector (0 = non-windowed) |-| `person_batch_size` | `int` | `10000` | Persons per batch in windowed mode |-| **Resources** | | | |-| `persons_db_env_var` | `str` | `PERSONS_DB_WRITER_URL` | Env var for Postgres connection |--## Example Sensor Cursor Configurations--Set the sensor cursor in Dagster UI to one of these JSON configurations.--### Non-Windowed Mode (Small Teams)--Best for teams with < 100K affected persons or short bug windows.--```json-{- "team_ids": [123, 456, 789],- "bug_window_start": "2026-01-06 20:01:00",- "bug_window_end": "2026-01-07 14:52:00",- "team_ch_props_fetch_window_seconds": 0,- "dry_run": true,- "backup_enabled": true,- "batch_size": 100-}-```--### Windowed Batched Mode (Large Teams)--Best for teams with 100K+ affected persons. Uses 1-hour windows and 10K person batches.--```json-{- "team_ids": [2],- "bug_window_start": "2026-01-06 20:01:00",- "bug_window_end": "2026-01-07 14:52:00",- "team_ch_props_fetch_window_seconds": 3600,- "person_batch_size": 10000,- "dry_run": true,- "backup_enabled": true,- "batch_size": 100-}-```--### Range-Based Scanning (All Teams)--Process all teams in a team_id range, in chunks.--```json-{- "range_start": 1,- "range_end": 100000,- "chunk_size": 1000,- "bug_window_start": "2026-01-06 20:01:00",- "bug_window_end": "2026-01-07 14:52:00",- "team_ch_props_fetch_window_seconds": 0,- "max_concurrent_jobs": 5,- "max_concurrent_tasks": 10,- "dry_run": false-}-```--### Production Run with Windowed Mode--Full production configuration for a large team.--```json-{- "team_ids": [2],- "bug_window_start": "2026-01-06 20:01:00",- "bug_window_end": "2026-01-07 14:52:00",- "team_ch_props_fetch_window_seconds": 3600,- "person_batch_size": 10000,- "dry_run": false,- "backup_enabled": true,- "batch_size": 100,- "teams_per_chunk": 1,- "max_concurrent_jobs": 1,- "max_concurrent_tasks": 1-}-```--## Choosing the Right Mode--| Scenario | Recommended Mode | Config |-| ---------------------------- | ---------------- | ------------------------------------------ |-| Small team (< 100K persons) | Non-windowed | `team_ch_props_fetch_window_seconds: 0` |-| Short bug window (< 1 day) | Non-windowed | `team_ch_props_fetch_window_seconds: 0` |-| Large team (100K+ persons) | Windowed batched | `team_ch_props_fetch_window_seconds: 3600` |-| Long bug window (days/weeks) | Windowed batched | `team_ch_props_fetch_window_seconds: 3600` |-| Unknown size, being cautious | Windowed batched | Start with `dry_run: true` |--## Recommended Workflow--For a full reconciliation across all teams, use this two-pass approach:--### Pass 1: Non-Windowed Mode for Most Teams--Run the sensor with non-windowed mode to process the majority of teams quickly:--```json-{- "range_start": 1,- "range_end": 100000,- "chunk_size": 1000,- "bug_window_start": "2026-01-06 20:01:00",- "bug_window_end": "2026-01-07 14:52:00",- "team_ch_props_fetch_window_seconds": 0,- "max_concurrent_jobs": 5,- "max_concurrent_tasks": 10,- "dry_run": false,- "backup_enabled": true-}-```--This will complete successfully for most teams but may fail (OOM or timeout) for very large teams.--### Extract Failed Teams--Use the extraction script to identify which teams failed:--```bash-export DAGSTER_CLOUD_TOKEN="<your-user-token>"--# Extract failed teams from recent runs-python posthog/dags/scripts/extract_reconciliation_results.py \- --org posthog \- --deployment prod-us \- --since "2026-01-27" \- --until "2026-01-30" \- --verbose \- --output-csv failed_teams.csv--# The script outputs:-# - Summary of succeeded/failed teams-# - List of failed team IDs-# - CSV with error details for each failed team-```--See the script's `--help` for additional options (filter by status, specific run ID, etc.).--### Pass 2: Windowed Mode for Failed Teams--Re-run the sensor targeting only the failed teams using windowed batched mode:--```json-{- "team_ids": [2, 15847, 23456],- "bug_window_start": "2026-01-06 20:01:00",- "bug_window_end": "2026-01-07 14:52:00",- "team_ch_props_fetch_window_seconds": 3600,- "person_batch_size": 10000,- "max_concurrent_jobs": 1,- "max_concurrent_tasks": 1,- "dry_run": false,- "backup_enabled": true,- "teams_per_chunk": 1-}-```--Key differences for Pass 2:--- `team_ids`: Explicit list of failed teams from the extraction script-- `team_ch_props_fetch_window_seconds: 3600`: Enable windowed mode (1-hour windows)-- `person_batch_size: 10000`: Process persons in batches to avoid OOM-- `max_concurrent_jobs: 1` and `max_concurrent_tasks: 1`: Serialize for large teams-- `teams_per_chunk: 1`: One team per task for better isolation--If a team still fails, try reducing `person_batch_size` (e.g., 5000) or check the error logs for specific issues.--## Monitoring and Troubleshooting--### Logs to Watch--In windowed batched mode, look for these log patterns:--```text-# Step 1: Affected persons query-Querying affected persons: team_id={id}, bug_window=[{start}, {end}]-Found {N} affected persons for team_id={id}--# Step 2: Per-batch progress-Starting batched windowed processing: team_id={id}, {N} batches of ~10000 persons, ~{M} windows of 3600s each-Processing batch {n}/{total} for team_id={id}, {N} persons-Batch {n}: processed window {w}/{total_windows}, accumulated {N} persons-Batch {n}: comparing {N} persons against current state-Batch {n}: yielding {N} persons with diffs--# Completion-Completed team_id={id} (windowed batched): processed={N}, updated={M}, skipped={K}, person_batches={B}-```--### Error Logs--When failures occur, errors are logged via `logger.exception()` which includes the full stack trace. Look for these message patterns:--```text-# Failed to query affected persons-Failed to query affected persons for team_id={id}, bug_window=[{start}, {end}]-[full stack trace follows]--# Failed during a specific window query-Batch {n}: failed to query window {w}/{total} [{window_start}, {window_end}) for team_id={id}-[full stack trace follows]-Batch {n}: failed during windowed event collection for team_id={id} after {w} windows, accumulated {N} persons-[full stack trace follows]--# Failed during person state comparison-Batch {n}: failed to compare {N} persons against current state for team_id={id}-[full stack trace follows]-```--**Partial Progress**: If a failure occurs mid-processing, already-committed batches are persisted to Postgres and Kafka. The job will be marked as failed, but partial progress is saved.--### Common Issues--1. **OOM in windowed mode**: Reduce `person_batch_size` (e.g., 5000 instead of 10000)-2. **Query size exceeded**: The code handles this by batching person_ids, but if you see this error, reduce `person_batch_size`-3. **Slow progress**: Increase `team_ch_props_fetch_window_seconds` (e.g., 7200 for 2-hour windows) to reduce query count--### Metrics--The job emits these metrics to ClickHouse:--- `person_property_reconciliation_persons_processed_total`-- `person_property_reconciliation_persons_updated_total`-- `person_property_reconciliation_persons_skipped_total`-- `person_property_reconciliation_teams_succeeded_total`-- `person_property_reconciliation_teams_failed_total`-- `person_property_reconciliation_duration_seconds_total`-- `person_property_reconciliation_error` (on failures)--## Backup and Recovery--When `backup_enabled: true`, the job stores before/after state in `posthog_person_reconciliation_backup`:--- `job_id`: Dagster run ID-- `properties` / `properties_after`: Before/after JSON-- `version` / `version_after`: Before/after version numbers-- `pending_operations`: Array of operations that were applieddiff --git a/posthog/dags/common/resources.py b/posthog/dags/common/resources.pyindex 7148fe30f847..567f8980eae6 100644--- a/posthog/dags/common/resources.py+++ b/posthog/dags/common/resources.py@@ -1,10 +1,8 @@import jsonimport socketimport asyncio-from collections.abc import Generatorfrom dataclasses import dataclassfrom typing import Any-from urllib.parse import urlparseimport dagsterimport psycopg2@@ -23,9 +21,6 @@ from posthog.clickhouse.client.connection import (is_file_backed_user,)from posthog.clickhouse.cluster import ClickhouseCluster, RetryPolicy, get_cluster-from posthog.kafka_client.client import _KafkaProducer-from posthog.kafka_client.profiles import KafkaClusterProfile-from posthog.kafka_client.routing import get_producerfrom posthog.redis import get_client, redisfrom posthog.utils import initialize_self_capture_api_token@@ -284,27 +279,6 @@ class PostHogAnalyticsResource(dagster.ConfigurableResource):return None-class PostgresURLResource(dagster.ConfigurableResource):- """- Postgres connection that parses a connection URL.- Delegates to PostgresResource for actual connection logic.- Expects format: postgres://user:pass@host:port/dbname- """-- connection_url: str-- def create_resource(self, context: dagster.InitResourceContext) -> psycopg2.extensions.connection:- parsed = urlparse(self.connection_url)- pg = PostgresResource(- host=parsed.hostname or "",- port=str(parsed.port or 5432),- database=parsed.path.lstrip("/"),- user=parsed.username or "",- password=parsed.password or "",- )- return pg.create_resource(context)--class PostgresURL(dagster.ConfigurableResource):"""A Postgres connection URL handed to the op unopened.@@ -319,25 +293,6 @@ class PostgresURL(dagster.ConfigurableResource):return self.connection_url-def kafka_producer_resource(context: dagster.InitResourceContext) -> Generator[_KafkaProducer]:- """Yield a singleton Kafka producer bound to the INGESTION (WarpStream) profile; flush on teardown.-- Consumers of this resource produce to `clickhouse_person` / `clickhouse_person_distinct_id`,- which the routing map sends to the INGESTION profile. Binding the resource here keeps- that explicit so a chart misconfiguration (missing `KAFKA_INGESTION_HOSTS`)- fails loud rather than silently dropping writes via the DEFAULT fallback.-- Ops producing to topics on a different profile must call- `posthog.kafka_client.routing.get_producer(topic=...)` directly.- """- producer = get_producer(profile=KafkaClusterProfile.INGESTION)- try:- yield producer- finally:- producer.flush()--def _is_retryable_http_error(exception: BaseException) -> bool:"""Check if an HTTP error should be retried."""if isinstance(exception, requests.exceptions.HTTPError):diff --git a/posthog/dags/fix_person_id_overrides.py b/posthog/dags/fix_person_id_overrides.pydeleted file mode 100644index 687072b9b5a9..000000000000--- a/posthog/dags/fix_person_id_overrides.py+++ /dev/null@@ -1,182 +0,0 @@-"""Dagster job to insert person_id overrides for distinct_ids.--Given a list of "bugged" distinct_ids, for each one:-1. Find its person_id in person_distinct_id2-2. Find ALL other distinct_ids mapped to that same person_id-3. Insert ALL of them into person_distinct_id_overrides-"""--from typing import Optional--import dagster-import pydantic--from posthog.clickhouse.client import sync_execute-from posthog.dags.common import JobOwners---class FixPersonIdOverridesConfig(dagster.Config):- """Configuration for the fix person_id overrides job."""-- team_id: int = pydantic.Field(description="Team ID to process")- distinct_ids: str = pydantic.Field(description="Comma-separated list of bugged distinct_ids")- dry_run: bool = pydantic.Field(- default=True,- description="If true, don't actually insert, just log what would be inserted",- )---def get_person_id_from_pdi2(team_id: int, distinct_id: str) -> Optional[tuple[str, int]]:- """Get the current person_id and version for a distinct_id from person_distinct_id2."""- result = sync_execute(- """- SELECT- argMax(person_id, version) as pid,- max(version) as max_version- FROM person_distinct_id2- WHERE team_id = %(team_id)s- AND distinct_id = %(distinct_id)s- GROUP BY distinct_id- HAVING argMax(is_deleted, version) = 0- """,- {"team_id": team_id, "distinct_id": distinct_id},- )- if result and result[0]:- return str(result[0][0]), int(result[0][1])- return None---def get_all_distinct_ids_for_person(team_id: int, person_id: str) -> list[tuple[str, int]]:- """Get ALL distinct_ids mapped to a person_id in person_distinct_id2."""- result = sync_execute(- """- SELECT- distinct_id,- max(version) as max_version- FROM person_distinct_id2- WHERE team_id = %(team_id)s- GROUP BY distinct_id- HAVING argMax(person_id, version) = %(person_id)s- AND argMax(is_deleted, version) = 0- """,- {"team_id": team_id, "person_id": person_id},- )- return [(row[0], int(row[1])) for row in result] if result else []---def get_existing_override(team_id: int, distinct_id: str) -> Optional[tuple[str, int]]:- """Check if an override already exists."""- result = sync_execute(- """- SELECT- argMax(person_id, version) as pid,- max(version) as max_version- FROM person_distinct_id_overrides- WHERE team_id = %(team_id)s- AND distinct_id = %(distinct_id)s- GROUP BY distinct_id- HAVING argMax(is_deleted, version) = 0- """,- {"team_id": team_id, "distinct_id": distinct_id},- )- if result and result[0]:- return str(result[0][0]), int(result[0][1])- return None---def insert_override_batch(overrides: list[tuple[int, str, str, int]]) -> None:- """Insert a batch of records into person_distinct_id_overrides.-- Args:- overrides: List of (team_id, distinct_id, person_id, version) tuples- """- if not overrides:- return-- from datetime import datetime-- now = datetime.now()- sync_execute(- """- INSERT INTO person_distinct_id_overrides- (team_id, distinct_id, person_id, is_deleted, version, _timestamp, _offset, _partition)- VALUES- """,- [- (team_id, distinct_id, person_id, 0, version, now, 0, 0)- for team_id, distinct_id, person_id, version in overrides- ],- )---def fix_person_id_overrides_op(- context: dagster.OpExecutionContext,- config: FixPersonIdOverridesConfig,-) -> None:- """- For each bugged distinct_id:- 1. Find its person_id in person_distinct_id2- 2. Find ALL distinct_ids mapped to that person_id- 3. Insert ALL mappings into person_distinct_id_overrides- """- # Parse comma-separated distinct_ids- bugged_distinct_ids = [d.strip() for d in config.distinct_ids.split(",") if d.strip()]-- context.log.info(f"Processing {len(bugged_distinct_ids)} bugged distinct_ids")- context.log.info(f"Dry run: {config.dry_run}")-- # Track which person_ids we've already processed to avoid duplicates- processed_person_ids: set[str] = set()-- # Batch of overrides to insert: (team_id, distinct_id, person_id, version)- pending_overrides: list[tuple[int, str, str, int]] = []-- for _i, bugged_distinct_id in enumerate(bugged_distinct_ids):- # Step 1: Get person_id for this bugged distinct_id- pdi2_result = get_person_id_from_pdi2(config.team_id, bugged_distinct_id)- if not pdi2_result:- context.log.warning(f"No person found for bugged distinct_id={bugged_distinct_id}")- continue-- person_id, _ = pdi2_result-- # Skip if we already processed this person- if person_id in processed_person_ids:- context.log.info(f"Already processed person_id={person_id}... for distinct_id={bugged_distinct_id}")- continue-- processed_person_ids.add(person_id)-- # Step 2: Get ALL distinct_ids mapped to this person- all_distinct_ids = get_all_distinct_ids_for_person(config.team_id, person_id)- context.log.info(- f"Person {person_id}... has {len(all_distinct_ids)} distinct_ids: {', '.join(did for did, _ in all_distinct_ids)} "- )-- # Step 3: Collect overrides for each distinct_id- for distinct_id, version in all_distinct_ids:- # Check if override already exists- existing = get_existing_override(config.team_id, distinct_id)- if existing:- continue-- # Use version > 0 so it gets picked up by the overrides MV- new_version = max(version, 1)-- if config.dry_run:- context.log.info(f" [DRY RUN] Would insert -> Distinct ID: {distinct_id}, Person ID: {person_id}...")- else:- pending_overrides.append((config.team_id, distinct_id, person_id, new_version))- context.log.info(f" Queued: {distinct_id} -> {person_id}...")-- # Insert all overrides at once- if not config.dry_run and pending_overrides:- insert_override_batch(pending_overrides)- context.log.info(f"Inserted {len(pending_overrides)} overrides")--[email protected](tags={"owner": JobOwners.TEAM_CLICKHOUSE.value})-def fix_person_id_overrides_job():- """Job to insert person_id overrides for specified distinct_ids."""- fix_person_id_overrides_op()diff --git a/posthog/dags/locations/__init__.py b/posthog/dags/locations/__init__.pyindex 432c49fec540..1ceb2ee2e40a 100644--- a/posthog/dags/locations/__init__.py+++ b/posthog/dags/locations/__init__.py@@ -11,10 +11,8 @@ from posthog.dags.common.resources import (ClickhouseClusterResource,PostgresResource,PostgresURL,- PostgresURLResource,PostHogAnalyticsResource,RedisResource,- kafka_producer_resource,)# Default loggers for every code location's jobs. Overrides Dagster's@@ -46,15 +44,9 @@ resources_by_env = {"posthoganalytics": dagster.ResourceDefinition.none_resource(description="Dummy PostHogAnalytics resource since posthoganalytics is configured properly in production."),- # Persons DB resource (parses connection URL)- "persons_database": PostgresURLResource(- connection_url=dagster.EnvVar("PERSONS_DB_WRITER_URL"),- ),"persons_database_url": PostgresURL(connection_url=dagster.EnvVar("PERSONS_DB_WRITER_URL"),),- # Kafka producer (auto-configured from Django settings)- "kafka_producer": kafka_producer_resource,# Clay webhook for job switchers pipeline"clay_webhook_job_switchers": ClayWebhookResource(webhook_url=dagster.EnvVar("CLAY_JOB_SWITCHERS_WEBHOOK_URL"),@@ -85,15 +77,9 @@ resources_by_env = {password=dagster.EnvVar("POSTGRES_PASSWORD"),),"posthoganalytics": PostHogAnalyticsResource(personal_api_key=dagster.EnvVar("PERSONAL_API_KEY")),- # Persons DB resource (parses connection URL)- "persons_database": PostgresURLResource(- connection_url=dagster.EnvVar("PERSONS_DB_WRITER_URL"),- ),"persons_database_url": PostgresURL(connection_url=dagster.EnvVar("PERSONS_DB_WRITER_URL"),),- # Kafka producer (auto-configured from Django settings)- "kafka_producer": kafka_producer_resource,# Clay webhook for job switchers pipeline"clay_webhook_job_switchers": ClayWebhookResource(webhook_url=dagster.EnvVar("CLAY_JOB_SWITCHERS_WEBHOOK_URL"),diff --git a/posthog/dags/locations/clickhouse.py b/posthog/dags/locations/clickhouse.pyindex b65d5d788cde..ed1c4ff2abe4 100644--- a/posthog/dags/locations/clickhouse.py+++ b/posthog/dags/locations/clickhouse.py@@ -11,7 +11,6 @@ from posthog.dags import (deletes,drop_materialized_column,export_query_log_archive_to_s3,- fix_person_id_overrides,flag_evaluations_backfill,flag_evaluations_person_id_repair,part_breaker,@@ -45,7 +44,6 @@ defs = dagster.Definitions(deletes.monthly_retention_events_cleanup_job,export_query_log_archive_to_s3.export_query_log_archive_to_s3,backfill_materialized_column.backfill_materialized_column,- fix_person_id_overrides.fix_person_id_overrides_job,flag_evaluations_backfill.flag_evaluations_backfill_job,flag_evaluations_person_id_repair.cleanup_orphaned_flag_evaluations_person_id_repair,flag_evaluations_person_id_repair.flag_evaluations_person_id_repair_job,diff --git a/posthog/dags/locations/ingestion.py b/posthog/dags/locations/ingestion.pyindex 13fb3456f155..b77c2b17c2fc 100644--- a/posthog/dags/locations/ingestion.py+++ b/posthog/dags/locations/ingestion.py@@ -1,6 +1,6 @@import dagster-from posthog.dags import distinct_id_usage, ingestion_assets, person_property_reconciliation+from posthog.dags import distinct_id_usage, ingestion_assetsfrom . import loggers, resources@@ -10,14 +10,10 @@ defs = dagster.Definitions(],jobs=[distinct_id_usage.distinct_id_usage_monitoring,- person_property_reconciliation.person_property_reconciliation_job,],schedules=[distinct_id_usage.distinct_id_usage_monitoring_schedule,],- sensors=[- person_property_reconciliation.person_property_reconciliation_scheduler,- ],loggers=loggers,resources=resources,)diff --git a/posthog/dags/owners.yaml b/posthog/dags/owners.yamlindex 65a0f1eec679..813dd2db800f 100644--- a/posthog/dags/owners.yaml+++ b/posthog/dags/owners.yaml@@ -1,11 +1,6 @@version: 1owners: []rules:- - match:- - '/fix_person_id_overrides.py'- - '/person_property_reconciliation.py'- - '/tests/test_person_property_reconciliation.py'- owners: team-ingestion- match: '/sessions.py'owners: team-analytics-platform- match:diff --git a/posthog/dags/person_property_reconciliation.py b/posthog/dags/person_property_reconciliation.pydeleted file mode 100644index 1a948d145257..000000000000--- a/posthog/dags/person_property_reconciliation.py+++ /dev/null@@ -1,2584 +0,0 @@-"""Dagster job for reconciling person properties that were missed due to a bug where-PERSON_BATCH_WRITING_DB_WRITE_MODE=ASSERT_VERSION caused updatePersonAssertVersion()-to not properly merge properties.--This job reads events from ClickHouse to find property updates ($set, $set_once, $unset)-within a bug window, compares timestamps with properties_last_updated_at in Postgres,-and applies any missed updates.--For comprehensive documentation including architecture diagrams, parameter reference,-and example sensor configurations, see:- PERSON_PROPERTY_RECONCILIATION.md--For detailed query explanations, see:- person_property_reconciliation_query_explanation.md-"""--import json-import time-from collections.abc import Iterator-from dataclasses import dataclass-from datetime import UTC, datetime, timedelta-from typing import Any--from django.conf import settings--import dagster-import psycopg2-from dagster_k8s import k8s_job_executor--from posthog.clickhouse.client import sync_execute-from posthog.clickhouse.cluster import ClickhouseCluster-from posthog.clickhouse.custom_metrics import MetricsClient-from posthog.dags.common import JobOwners-from posthog.dataclasses import frozen-from posthog.kafka_client.client import _KafkaProducer-from posthog.kafka_client.topics import KAFKA_PERSON--# Use in_process_executor locally (k8s_job_executor doesn't work outside k8s)-executor_def = dagster.in_process_executor if settings.DEBUG else k8s_job_executor---class PersonPropertyReconciliationConfig(dagster.Config):- """Configuration for the person property reconciliation job."""-- bug_window_start: str # ClickHouse format: "YYYY-MM-DD HH:MM:SS" (assumed UTC)- team_ids: list[int] | None = None # Optional: filter to specific teams- bug_window_end: str | None = None # Optional: required if team_ids not supplied- min_team_id: int | None = None # Optional: only process teams with id >= this value- max_team_id: int | None = None # Optional: only process teams with id <= this value- exclude_team_ids: list[int] | None = None # Optional: exclude specific team_ids- dry_run: bool = False # Log changes without applying- backup_enabled: bool = True # Store before/after state in backup table- batch_size: int = 100 # Commit Postgres transaction every N persons (0 = single commit at end)- teams_per_chunk: int = 100 # Number of teams to process per task (reduces task overhead)- team_ch_props_fetch_window_seconds: int = 0 # 0 = single query; >0 = split into N-second windows- person_batch_size: int = 10000 # Persons per batch in windowed mode (keeps IN clauses small)---@dataclass-class PropertyValue:- """A property value with its timestamp."""-- timestamp: datetime # UTC datetime- value: Any | None # None for unset operations---@dataclass-class PersonPropertyDiffs:- """Property diffs for a single person, organized by operation type."""-- person_id: str- person_version: int # Version from CH person table (baseline for conflict detection)- set_updates: dict[str, PropertyValue] # key -> PropertyValue- set_once_updates: dict[str, PropertyValue] # key -> PropertyValue- unset_updates: dict[str, PropertyValue] # key -> PropertyValue (value is always None)---@dataclass-class RawPersonPropertyUpdates:- """Raw property updates from events, before comparison with person state.-- Used in windowed queries where we aggregate across multiple time windows- before doing a single comparison against person properties.- """-- person_id: str- set_updates: dict[str, PropertyValue] # key -> PropertyValue- set_once_updates: dict[str, PropertyValue] # key -> PropertyValue- unset_updates: dict[str, PropertyValue] # key -> PropertyValue (value is always None)---class SkipReason:- """Reasons why a person update was skipped."""-- SUCCESS = "success" # Not skipped - successfully updated- NO_CHANGES = "no_changes" # No changes needed after reconciliation- NOT_FOUND = "not_found" # Person not found in Postgres- VERSION_CONFLICT = "version_conflict" # Version mismatch after max retries---# Properties that should NOT trigger a person update on their own.-# These change frequently but aren't valuable enough to update the person record for.-# Keep in sync with: nodejs/src/common/persons/person-property-utils.ts-FILTERED_PERSON_UPDATE_PROPERTIES = frozenset(- [- # URL/navigation properties - change on every page view- "$current_url",- "$pathname",- "$referring_domain",- "$referrer",- # Screen/viewport dimensions - can change on window resize- "$screen_height",- "$screen_width",- "$viewport_height",- "$viewport_width",- # Browser/device properties - change less frequently but still filtered- "$browser",- "$browser_version",- "$device_type",- "$raw_user_agent",- "$os",- "$os_name",- "$os_version",- # GeoIP properties - filtered because they change frequently- # Note: $geoip_country_name and $geoip_city_name DO trigger updates (not listed here)- "$geoip_postal_code",- "$geoip_time_zone",- "$geoip_latitude",- "$geoip_longitude",- "$geoip_accuracy_radius",- "$geoip_subdivision_1_code",- "$geoip_subdivision_1_name",- "$geoip_subdivision_2_code",- "$geoip_subdivision_2_name",- "$geoip_subdivision_3_code",- "$geoip_subdivision_3_name",- "$geoip_city_confidence",- "$geoip_country_confidence",- "$geoip_postal_code_confidence",- "$geoip_subdivision_1_confidence",- "$geoip_subdivision_2_confidence",- ]-)---def ensure_utc_datetime(ts: datetime) -> datetime:- """Ensure timestamp is a UTC-aware datetime."""- if ts.tzinfo is None:- return ts.replace(tzinfo=UTC)- return ts---def get_person_property_updates_from_clickhouse(- team_id: int,- bug_window_start: str,-) -> list[PersonPropertyDiffs]:- """- Query ClickHouse to get property updates per person that differ from current state.-- This query:- 1. Joins events with person_distinct_id_overrides to resolve merged persons- 2. Extracts $set (argMax for latest), $set_once (argMin for first), and $unset properties- 3. Compares with current person properties in ClickHouse- 4. Returns only persons where properties differ, are missing, or need removal-- Returns:- List of PersonPropertyDiffs, each containing 3 maps (set, set_once, unset) keyed by property key- """- query = """- -- CTE 1: Get all overrides for the team (no filtering)- WITH overrides AS (- SELECT- argMax(person_distinct_id_overrides.person_id, person_distinct_id_overrides.version) AS person_id,- person_distinct_id_overrides.distinct_id AS distinct_id- FROM person_distinct_id_overrides- WHERE equals(person_distinct_id_overrides.team_id, %(team_id)s)- GROUP BY person_distinct_id_overrides.distinct_id- HAVING ifNull(equals(argMax(person_distinct_id_overrides.is_deleted, person_distinct_id_overrides.version), 0), 0)- ),- -- CTE 2: Extract properties from events, grouped by RESOLVED person_id (after overrides)- -- This ensures argMax/argMin aggregates across ALL distinct_ids for the same person- event_properties_raw AS (- SELECT- person_id,- arrayMap(x -> x.1, arrayFilter(x -> x.4 = 'set', grouped_props)) AS set_keys,- arrayMap(x -> x.2, arrayFilter(x -> x.4 = 'set', grouped_props)) AS set_values,- arrayMap(x -> x.3, arrayFilter(x -> x.4 = 'set', grouped_props)) AS set_timestamps,- arrayMap(x -> x.1, arrayFilter(x -> x.4 = 'set_once', grouped_props)) AS set_once_keys,- arrayMap(x -> x.2, arrayFilter(x -> x.4 = 'set_once', grouped_props)) AS set_once_values,- arrayMap(x -> x.3, arrayFilter(x -> x.4 = 'set_once', grouped_props)) AS set_once_timestamps,- arrayMap(x -> x.1, arrayFilter(x -> x.4 = 'unset', grouped_props)) AS unset_keys,- arrayMap(x -> x.3, arrayFilter(x -> x.4 = 'unset', grouped_props)) AS unset_timestamps- FROM (- SELECT- person_id,- groupArray(tuple(key, value, kv_timestamp, prop_type)) AS grouped_props- FROM (- SELECT- -- Apply overrides to get resolved person_id- if(notEmpty(o.distinct_id), o.person_id, e.person_id) AS person_id,- kv_tuple.2 AS key,- kv_tuple.1 AS prop_type,- -- $set: newest event wins, $set_once: first event wins, $unset: newest event wins- if(kv_tuple.1 = 'set',- argMaxIf(kv_tuple.3, e.timestamp, kv_tuple.3 IS NOT NULL AND kv_tuple.3 != ''),- argMinIf(kv_tuple.3, e.timestamp, kv_tuple.3 IS NOT NULL AND kv_tuple.3 != '')- ) AS value,- if(kv_tuple.1 = 'set_once', min(e.timestamp), max(e.timestamp)) AS kv_timestamp- FROM events e- LEFT JOIN overrides o ON e.distinct_id = o.distinct_id- ARRAY JOIN- arrayConcat(- arrayFilter(x -> x.3 IS NOT NULL AND x.3 != '' AND x.3 != 'null' AND x.2 NOT IN %(filtered_properties)s,- arrayMap(x -> tuple('set', x.1, toString(x.2)),- arrayFilter(x -> x.2 IS NOT NULL, JSONExtractKeysAndValuesRaw(e.properties, '$set'))- )- ),- arrayFilter(x -> x.3 IS NOT NULL AND x.3 != '' AND x.3 != 'null' AND x.2 NOT IN %(filtered_properties)s,- arrayMap(x -> tuple('set_once', x.1, toString(x.2)),- arrayFilter(x -> x.2 IS NOT NULL, JSONExtractKeysAndValuesRaw(e.properties, '$set_once'))- )- ),- arrayFilter(x -> x.2 NOT IN %(filtered_properties)s,- arrayMap(x -> tuple('unset', JSON_VALUE(x, '$'), ''),- JSONExtractArrayRaw(e.properties, '$unset')- )- )- ) AS kv_tuple- WHERE e.team_id = %(team_id)s- AND e.timestamp >= %(bug_window_start)s- AND e.timestamp < now()- AND (JSONExtractString(e.properties, '$set') != '' OR JSONExtractString(e.properties, '$set_once') != '' OR notEmpty(JSONExtractArrayRaw(e.properties, '$unset')))- -- Group by resolved person_id (not distinct_id) so argMax works across all distinct_ids- GROUP BY if(notEmpty(o.distinct_id), o.person_id, e.person_id), kv_tuple.2, kv_tuple.1- )- GROUP BY person_id- )- ),- -- CTE 3: Filter to only persons with non-empty property sets- event_properties_flat AS (- SELECT *- FROM event_properties_raw- WHERE length(set_keys) > 0 OR length(set_once_keys) > 0 OR length(unset_keys) > 0- ),- -- CTE 4: Get person properties and version only for affected persons- person_props AS (- SELECT- id,- argMax(properties, version) as person_properties,- argMax(version, version) as person_version- FROM person- WHERE team_id = %(team_id)s- AND id IN (SELECT person_id FROM event_properties_flat)- GROUP BY id- HAVING argMax(is_deleted, version) = 0- )- SELECT- ep.person_id,- p.person_version,- -- For $set: only include properties where the key exists in person properties AND the value differs- arrayMap(i -> (ep.set_keys[i], ep.set_values[i], ep.set_timestamps[i]), arrayFilter(- i -> (- indexOf(keys2, ep.set_keys[i]) > 0- AND ep.set_values[i] != vals2[indexOf(keys2, ep.set_keys[i])]- ),- arrayEnumerate(ep.set_keys)- )) AS set_diff,- -- For $set_once: only include properties where the key does NOT exist in person properties- arrayFilter(- kv -> indexOf(keys2, kv.1) = 0,- arrayMap(i -> (ep.set_once_keys[i], ep.set_once_values[i], ep.set_once_timestamps[i]), arrayEnumerate(ep.set_once_keys))- ) AS set_once_diff,- -- For $unset: only include keys that EXIST in person properties (need removal)- arrayFilter(- kv -> indexOf(keys2, kv.1) > 0,- arrayMap(i -> (ep.unset_keys[i], ep.unset_timestamps[i]), arrayEnumerate(ep.unset_keys))- ) AS unset_diff- FROM event_properties_flat ep- INNER JOIN (- SELECT- id,- person_version,- person_properties,- arrayMap(x -> x.1, JSONExtractKeysAndValuesRaw(person_properties)) AS keys2,- arrayMap(x -> toString(x.2), JSONExtractKeysAndValuesRaw(person_properties)) AS vals2- FROM person_props- ) AS p ON p.id = ep.person_id- WHERE length(set_diff) > 0 OR length(set_once_diff) > 0 OR length(unset_diff) > 0- ORDER BY ep.person_id- SETTINGS- readonly=2,- max_execution_time=1200,- allow_experimental_object_type=1,- format_csv_allow_double_quotes=0,- max_ast_elements=4000000,- max_expanded_ast_elements=4000000,- transform_null_in=1,- optimize_min_equality_disjunction_chain_length=4294967295,- optimize_min_inequality_conjunction_chain_length=4294967295,- allow_experimental_join_condition=1,- use_hive_partitioning=0- """-- params = {- "team_id": team_id,- "bug_window_start": bug_window_start,- "filtered_properties": tuple(FILTERED_PERSON_UPDATE_PROPERTIES),- }-- rows = sync_execute(query, params)-- results: list[PersonPropertyDiffs] = []- for row in rows:- person_id, person_version, set_diff, set_once_diff, unset_diff = row-- set_updates: dict[str, PropertyValue] = {}- for key, value, timestamp in set_diff:- set_updates[str(key)] = PropertyValue(- timestamp=ensure_utc_datetime(timestamp),- value=json.loads(value),- )-- set_once_updates: dict[str, PropertyValue] = {}- for key, value, timestamp in set_once_diff:- set_once_updates[str(key)] = PropertyValue(- timestamp=ensure_utc_datetime(timestamp),- value=json.loads(value),- )-- unset_updates: dict[str, PropertyValue] = {}- for key, timestamp in unset_diff:- unset_updates[str(key)] = PropertyValue(- timestamp=ensure_utc_datetime(timestamp),- value=None,- )-- if set_updates or set_once_updates or unset_updates:- results.append(- PersonPropertyDiffs(- person_id=str(person_id),- person_version=int(person_version),- set_updates=set_updates,- set_once_updates=set_once_updates,- unset_updates=unset_updates,- )- )-- return results---def get_raw_person_property_updates_from_clickhouse(- team_id: int,- bug_window_start: str,- bug_window_end: str,- person_ids: tuple[str, ...],-) -> list[RawPersonPropertyUpdates]:- """- Query ClickHouse to get raw property updates from events WITHOUT comparing to person state.-- This is used for windowed queries where we need to aggregate events across multiple- time windows before doing a single comparison against person properties.-- Returns raw aggregated event data:- - $set: argMax for latest value per key- - $set_once: argMin for first value per key- - $unset: max timestamp per key-- Args:- team_id: Team ID to query- bug_window_start: Start of time window (ClickHouse format: "YYYY-MM-DD HH:MM:SS")- bug_window_end: End of time window (ClickHouse format: "YYYY-MM-DD HH:MM:SS")- person_ids: Tuple of person_ids to filter to (required for batched processing)-- Returns:- List of RawPersonPropertyUpdates with aggregated event data (no person comparison)- """- query = """- -- CTE 1: Get all overrides for the team- WITH overrides AS (- SELECT- argMax(person_distinct_id_overrides.person_id, person_distinct_id_overrides.version) AS person_id,- person_distinct_id_overrides.distinct_id AS distinct_id- FROM person_distinct_id_overrides- WHERE equals(person_distinct_id_overrides.team_id, %(team_id)s)- GROUP BY person_distinct_id_overrides.distinct_id- HAVING ifNull(equals(argMax(person_distinct_id_overrides.is_deleted, person_distinct_id_overrides.version), 0), 0)- ),- -- CTE 2: Extract and aggregate properties from events- event_properties_raw AS (- SELECT- person_id,- arrayMap(x -> x.1, arrayFilter(x -> x.4 = 'set', grouped_props)) AS set_keys,- arrayMap(x -> x.2, arrayFilter(x -> x.4 = 'set', grouped_props)) AS set_values,- arrayMap(x -> x.3, arrayFilter(x -> x.4 = 'set', grouped_props)) AS set_timestamps,- arrayMap(x -> x.1, arrayFilter(x -> x.4 = 'set_once', grouped_props)) AS set_once_keys,- arrayMap(x -> x.2, arrayFilter(x -> x.4 = 'set_once', grouped_props)) AS set_once_values,- arrayMap(x -> x.3, arrayFilter(x -> x.4 = 'set_once', grouped_props)) AS set_once_timestamps,- arrayMap(x -> x.1, arrayFilter(x -> x.4 = 'unset', grouped_props)) AS unset_keys,- arrayMap(x -> x.3, arrayFilter(x -> x.4 = 'unset', grouped_props)) AS unset_timestamps- FROM (- SELECT- person_id,- groupArray(tuple(key, value, kv_timestamp, prop_type)) AS grouped_props- FROM (- SELECT- if(notEmpty(o.distinct_id), o.person_id, e.person_id) AS person_id,- kv_tuple.2 AS key,- kv_tuple.1 AS prop_type,- if(kv_tuple.1 = 'set',- argMaxIf(kv_tuple.3, e.timestamp, kv_tuple.3 IS NOT NULL AND kv_tuple.3 != ''),- argMinIf(kv_tuple.3, e.timestamp, kv_tuple.3 IS NOT NULL AND kv_tuple.3 != '')- ) AS value,- if(kv_tuple.1 = 'set_once', min(e.timestamp), max(e.timestamp)) AS kv_timestamp- FROM events e- LEFT JOIN overrides o ON e.distinct_id = o.distinct_id- ARRAY JOIN- arrayConcat(- arrayFilter(x -> x.3 IS NOT NULL AND x.3 != '' AND x.3 != 'null' AND x.2 NOT IN %(filtered_properties)s,- arrayMap(x -> tuple('set', x.1, toString(x.2)),- arrayFilter(x -> x.2 IS NOT NULL, JSONExtractKeysAndValuesRaw(e.properties, '$set'))- )- ),- arrayFilter(x -> x.3 IS NOT NULL AND x.3 != '' AND x.3 != 'null' AND x.2 NOT IN %(filtered_properties)s,- arrayMap(x -> tuple('set_once', x.1, toString(x.2)),- arrayFilter(x -> x.2 IS NOT NULL, JSONExtractKeysAndValuesRaw(e.properties, '$set_once'))- )- ),- arrayFilter(x -> x.2 NOT IN %(filtered_properties)s,- arrayMap(x -> tuple('unset', JSON_VALUE(x, '$'), ''),- JSONExtractArrayRaw(e.properties, '$unset')- )- )- ) AS kv_tuple- WHERE e.team_id = %(team_id)s- AND e.timestamp >= %(bug_window_start)s- AND e.timestamp < %(bug_window_end)s- AND (JSONExtractString(e.properties, '$set') != '' OR JSONExtractString(e.properties, '$set_once') != '' OR notEmpty(JSONExtractArrayRaw(e.properties, '$unset')))- AND if(notEmpty(o.distinct_id), o.person_id, e.person_id) IN %(person_ids)s- GROUP BY if(notEmpty(o.distinct_id), o.person_id, e.person_id), kv_tuple.2, kv_tuple.1- )- GROUP BY person_id- )- )- -- Return raw aggregated event data (no person comparison)- SELECT- person_id,- arrayMap(i -> (set_keys[i], set_values[i], set_timestamps[i]), arrayEnumerate(set_keys)) AS set_data,- arrayMap(i -> (set_once_keys[i], set_once_values[i], set_once_timestamps[i]), arrayEnumerate(set_once_keys)) AS set_once_data,- arrayMap(i -> (unset_keys[i], unset_timestamps[i]), arrayEnumerate(unset_keys)) AS unset_data- FROM event_properties_raw- WHERE length(set_keys) > 0 OR length(set_once_keys) > 0 OR length(unset_keys) > 0- ORDER BY person_id- SETTINGS- readonly=2,- max_execution_time=600,- allow_experimental_object_type=1,- format_csv_allow_double_quotes=0,- max_ast_elements=4000000,- max_expanded_ast_elements=4000000,- transform_null_in=1,- optimize_min_equality_disjunction_chain_length=4294967295,- optimize_min_inequality_conjunction_chain_length=4294967295,- allow_experimental_join_condition=1,- use_hive_partitioning=0- """-- params = {- "team_id": team_id,- "bug_window_start": bug_window_start,- "bug_window_end": bug_window_end,- "filtered_properties": tuple(FILTERED_PERSON_UPDATE_PROPERTIES),- "person_ids": person_ids,- }-- rows = sync_execute(query, params)-- results: list[RawPersonPropertyUpdates] = []- for row in rows:- person_id, set_data, set_once_data, unset_data = row-- set_updates: dict[str, PropertyValue] = {}- for key, value, timestamp in set_data:- set_updates[str(key)] = PropertyValue(- timestamp=ensure_utc_datetime(timestamp),- value=json.loads(value),- )-- set_once_updates: dict[str, PropertyValue] = {}- for key, value, timestamp in set_once_data:- set_once_updates[str(key)] = PropertyValue(- timestamp=ensure_utc_datetime(timestamp),- value=json.loads(value),- )-- unset_updates: dict[str, PropertyValue] = {}- for key, timestamp in unset_data:- unset_updates[str(key)] = PropertyValue(- timestamp=ensure_utc_datetime(timestamp),- value=None,- )-- if set_updates or set_once_updates or unset_updates:- results.append(- RawPersonPropertyUpdates(- person_id=str(person_id),- set_updates=set_updates,- set_once_updates=set_once_updates,- unset_updates=unset_updates,- )- )-- return results---def get_affected_person_ids_from_clickhouse(- team_id: int,- bug_window_start: str,- bug_window_end: str,-) -> list[str]:- """- Query ClickHouse for person_ids whose events in the bug window produce property- diffs when compared against current person state.-- Uses the same aggregation + person-state comparison structure as the- non-windowed query (get_person_property_updates_from_clickhouse) but:- - Bounded by bug_window_end (not now()) to keep the scan lightweight- - Returns only person_ids (no full diff data)-- This means only persons whose bug-window event aggregation actually differs- from their current person properties are returned, dramatically reducing the- set compared to a simple event-presence check.-- Possible false positives (acceptable): a person whose bug-window aggregation- shows a diff that disappears when events up to now() are included. These are- filtered out by the downstream windowed comparison step.-- Args:- team_id: Team ID to query- bug_window_start: Start of bug window (ClickHouse format: "YYYY-MM-DD HH:MM:SS")- bug_window_end: End of bug window (ClickHouse format: "YYYY-MM-DD HH:MM:SS")-- Returns:- List of person_id strings (UUIDs) affected in the bug window- """- query = """- -- CTE 1: Get all overrides for the team (no filtering)- WITH overrides AS (- SELECT- argMax(person_distinct_id_overrides.person_id, person_distinct_id_overrides.version) AS person_id,- person_distinct_id_overrides.distinct_id AS distinct_id- FROM person_distinct_id_overrides- WHERE equals(person_distinct_id_overrides.team_id, %(team_id)s)- GROUP BY person_distinct_id_overrides.distinct_id- HAVING ifNull(equals(argMax(person_distinct_id_overrides.is_deleted, person_distinct_id_overrides.version), 0), 0)- ),- -- CTE 2: Extract properties from events, grouped by RESOLVED person_id (after overrides)- event_properties_raw AS (- SELECT- person_id,- arrayMap(x -> x.1, arrayFilter(x -> x.4 = 'set', grouped_props)) AS set_keys,- arrayMap(x -> x.2, arrayFilter(x -> x.4 = 'set', grouped_props)) AS set_values,- arrayMap(x -> x.1, arrayFilter(x -> x.4 = 'set_once', grouped_props)) AS set_once_keys,- arrayMap(x -> x.1, arrayFilter(x -> x.4 = 'unset', grouped_props)) AS unset_keys- FROM (- SELECT- person_id,- groupArray(tuple(key, value, kv_timestamp, prop_type)) AS grouped_props- FROM (- SELECT- if(notEmpty(o.distinct_id), o.person_id, e.person_id) AS person_id,- kv_tuple.2 AS key,- kv_tuple.1 AS prop_type,- if(kv_tuple.1 = 'set',- argMaxIf(kv_tuple.3, e.timestamp, kv_tuple.3 IS NOT NULL AND kv_tuple.3 != ''),- argMinIf(kv_tuple.3, e.timestamp, kv_tuple.3 IS NOT NULL AND kv_tuple.3 != '')- ) AS value,- if(kv_tuple.1 = 'set_once', min(e.timestamp), max(e.timestamp)) AS kv_timestamp- FROM events e- LEFT JOIN overrides o ON e.distinct_id = o.distinct_id- ARRAY JOIN- arrayConcat(- arrayFilter(x -> x.3 IS NOT NULL AND x.3 != '' AND x.3 != 'null' AND x.2 NOT IN %(filtered_properties)s,- arrayMap(x -> tuple('set', x.1, toString(x.2)),- arrayFilter(x -> x.2 IS NOT NULL, JSONExtractKeysAndValuesRaw(e.properties, '$set'))- )- ),- arrayFilter(x -> x.3 IS NOT NULL AND x.3 != '' AND x.3 != 'null' AND x.2 NOT IN %(filtered_properties)s,- arrayMap(x -> tuple('set_once', x.1, toString(x.2)),- arrayFilter(x -> x.2 IS NOT NULL, JSONExtractKeysAndValuesRaw(e.properties, '$set_once'))- )- ),- arrayFilter(x -> x.2 NOT IN %(filtered_properties)s,- arrayMap(x -> tuple('unset', JSON_VALUE(x, '$'), ''),- JSONExtractArrayRaw(e.properties, '$unset')- )- )- ) AS kv_tuple- WHERE e.team_id = %(team_id)s- AND e.timestamp >= %(bug_window_start)s- AND e.timestamp < %(bug_window_end)s- AND (JSONExtractString(e.properties, '$set') != '' OR JSONExtractString(e.properties, '$set_once') != '' OR notEmpty(JSONExtractArrayRaw(e.properties, '$unset')))- GROUP BY if(notEmpty(o.distinct_id), o.person_id, e.person_id), kv_tuple.2, kv_tuple.1- )- GROUP BY person_id- )- ),- -- CTE 3: Filter to only persons with non-empty property sets- event_properties_flat AS (- SELECT *- FROM event_properties_raw- WHERE length(set_keys) > 0 OR length(set_once_keys) > 0 OR length(unset_keys) > 0- ),- -- CTE 4: Get person properties only for affected persons- person_props AS (- SELECT- id,- argMax(properties, version) as person_properties- FROM person- WHERE team_id = %(team_id)s- AND id IN (SELECT person_id FROM event_properties_flat)- GROUP BY id- HAVING argMax(is_deleted, version) = 0- )- -- Return only person_ids that have actual diffs against current state- SELECT ep.person_id- FROM event_properties_flat ep- INNER JOIN (- SELECT- id,- person_properties,- arrayMap(x -> x.1, JSONExtractKeysAndValuesRaw(person_properties)) AS keys2,- arrayMap(x -> toString(x.2), JSONExtractKeysAndValuesRaw(person_properties)) AS vals2- FROM person_props- ) AS p ON p.id = ep.person_id- WHERE- -- $set: key exists in person AND value differs- length(arrayFilter(- i -> (- indexOf(p.keys2, ep.set_keys[i]) > 0- AND ep.set_values[i] != p.vals2[indexOf(p.keys2, ep.set_keys[i])]- ),- arrayEnumerate(ep.set_keys)- )) > 0- -- $set_once: key does NOT exist in person- OR length(arrayFilter(- i -> indexOf(p.keys2, ep.set_once_keys[i]) = 0,- arrayEnumerate(ep.set_once_keys)- )) > 0- -- $unset: key EXISTS in person- OR length(arrayFilter(- i -> indexOf(p.keys2, ep.unset_keys[i]) > 0,- arrayEnumerate(ep.unset_keys)- )) > 0- ORDER BY ep.person_id- SETTINGS- readonly=2,- max_execution_time=1200,- allow_experimental_object_type=1,- format_csv_allow_double_quotes=0,- max_ast_elements=4000000,- max_expanded_ast_elements=4000000,- transform_null_in=1,- optimize_min_equality_disjunction_chain_length=4294967295,- optimize_min_inequality_conjunction_chain_length=4294967295,- allow_experimental_join_condition=1,- use_hive_partitioning=0- """-- params = {- "team_id": team_id,- "bug_window_start": bug_window_start,- "bug_window_end": bug_window_end,- "filtered_properties": tuple(FILTERED_PERSON_UPDATE_PROPERTIES),- }-- rows = sync_execute(query, params)- return [str(row[0]) for row in rows]---# Default batch size for processing persons in windowed mode.-# Keeps IN clauses under ClickHouse's max_query_size limit.-# 10K UUIDs × ~40 chars ≈ 400KB query text.-WINDOWED_PERSON_BATCH_SIZE = 10000---def merge_raw_person_property_updates(- accumulated: dict[str, RawPersonPropertyUpdates],- new_updates: list[RawPersonPropertyUpdates],-) -> None:- """- Merge new raw updates into accumulated dict, keyed by person_id.-- This is used when iterating through time windows to accumulate raw property updates- across multiple query results BEFORE comparing with person state.-- Merge rules:- - For $set: newer timestamp replaces older- - For $set_once: earlier timestamp wins (first-writer-wins semantics)- - For $unset: newer timestamp replaces older-- Cross-map conflicts (set/set_once vs unset) are resolved downstream by- filter_event_person_properties after comparison with person state.-- Args:- accumulated: Dict of person_id -> RawPersonPropertyUpdates to merge into (mutated in place)- new_updates: List of new RawPersonPropertyUpdates from the latest window query- """- for new_update in new_updates:- person_id = new_update.person_id-- if person_id not in accumulated:- accumulated[person_id] = RawPersonPropertyUpdates(- person_id=person_id,- set_updates=dict(new_update.set_updates),- set_once_updates=dict(new_update.set_once_updates),- unset_updates=dict(new_update.unset_updates),- )- continue-- existing = accumulated[person_id]-- # Process $set updates - newer timestamp wins- for key, new_pv in new_update.set_updates.items():- if key in existing.set_updates:- if new_pv.timestamp > existing.set_updates[key].timestamp:- existing.set_updates[key] = new_pv- else:- existing.set_updates[key] = new_pv-- # Process $set_once updates - earlier timestamp wins- for key, new_pv in new_update.set_once_updates.items():- if key in existing.set_once_updates:- if new_pv.timestamp < existing.set_once_updates[key].timestamp:- existing.set_once_updates[key] = new_pv- else:- existing.set_once_updates[key] = new_pv-- # Process $unset updates - newer timestamp wins- for key, new_pv in new_update.unset_updates.items():- if key in existing.unset_updates:- if new_pv.timestamp > existing.unset_updates[key].timestamp:- existing.unset_updates[key] = new_pv- else:- existing.unset_updates[key] = new_pv---# Max number of person_ids per batch when querying person state.-# Prevents "Query size exceeded" errors when IN clause becomes too large.-# 10K UUIDs × ~40 chars ≈ 400KB query text, under typical max_query_size limits.-PERSON_STATE_BATCH_SIZE = 10000---def compare_raw_updates_with_person_state(- team_id: int,- raw_updates: list[RawPersonPropertyUpdates],-) -> list[PersonPropertyDiffs]:- """- Compare merged raw event updates against current person state in ClickHouse.-- This is the second step of the windowed query flow:- 1. get_raw_person_property_updates_from_clickhouse() per window- 2. merge_raw_person_property_updates() to combine windows- 3. compare_raw_updates_with_person_state() to filter to actual diffs-- Comparison rules (matching the original single-query behavior):- - $set: only include if key EXISTS in person AND value DIFFERS- - $set_once: only include if key does NOT exist in person- - $unset: only include if key EXISTS in person-- Note on comparison semantics:- This function uses Python object comparison, so semantically equal values- like 123 (int) and 123.0 (float) are considered equal. This differs from- the non-windowed SQL path (get_person_property_updates_from_clickhouse)- which compares raw JSON string representations. The Python approach is- preferable for reconciliation as it avoids unnecessary updates when- values are semantically equivalent.-- Note on batching:- For high-volume teams with many affected persons, we batch the person- state queries to avoid "Query size exceeded" errors from overly large- IN clauses. Each batch queries up to PERSON_STATE_BATCH_SIZE persons.-- Args:- team_id: Team ID- raw_updates: Merged raw updates from all time windows-- Returns:- List of PersonPropertyDiffs with only actual differences- """- if not raw_updates:- return []-- # Extract person_ids for batching- person_ids = [u.person_id for u in raw_updates]-- # Query person properties and versions in batches to avoid "Query size exceeded"- person_data: dict[str, tuple[dict, int]] = {}-- for batch_start in range(0, len(person_ids), PERSON_STATE_BATCH_SIZE):- batch_person_ids = person_ids[batch_start : batch_start + PERSON_STATE_BATCH_SIZE]-- query = """- SELECT- id,- argMax(properties, version) as properties,- argMax(version, version) as person_version- FROM person- WHERE team_id = %(team_id)s- AND id IN %(person_ids)s- GROUP BY id- HAVING argMax(is_deleted, version) = 0- """-- params = {- "team_id": team_id,- "person_ids": tuple(batch_person_ids),- }-- rows = sync_execute(query, params)-- for row in rows:- person_id, properties_str, version = row- if properties_str:- properties = json.loads(properties_str) if isinstance(properties_str, str) else properties_str- else:- properties = {}- person_data[str(person_id)] = (properties, int(version))-- results: list[PersonPropertyDiffs] = []-- for raw_update in raw_updates:- if raw_update.person_id not in person_data:- continue-- person_properties, person_version = person_data[raw_update.person_id]- person_keys = set(person_properties.keys())-- # Filter $set: key must exist in person AND value must differ- filtered_set: dict[str, PropertyValue] = {}- for key, pv in raw_update.set_updates.items():- if key in person_keys:- person_val = person_properties[key]- if pv.value != person_val:- filtered_set[key] = pv-- # Filter $set_once: key must NOT exist in person- filtered_set_once: dict[str, PropertyValue] = {}- for key, pv in raw_update.set_once_updates.items():- if key not in person_keys:- filtered_set_once[key] = pv-- # Filter $unset: key must exist in person- filtered_unset: dict[str, PropertyValue] = {}- for key, pv in raw_update.unset_updates.items():- if key in person_keys:- filtered_unset[key] = pv-- if filtered_set or filtered_set_once or filtered_unset:- results.append(- PersonPropertyDiffs(- person_id=raw_update.person_id,- person_version=person_version,- set_updates=filtered_set,- set_once_updates=filtered_set_once,- unset_updates=filtered_unset,- )- )-- return results---def parse_ch_timestamp(ts: str) -> datetime:- """Parse a ClickHouse timestamp string to a UTC datetime."""- return datetime.strptime(ts, "%Y-%m-%d %H:%M:%S").replace(tzinfo=UTC)---def format_ch_timestamp(dt: datetime) -> str:- """Format a datetime to ClickHouse timestamp string (UTC)."""- if dt.tzinfo is None:- # Assume naive datetimes are already UTC- return dt.strftime("%Y-%m-%d %H:%M:%S")- # Convert to UTC before formatting- utc_dt = dt.astimezone(UTC)- return utc_dt.strftime("%Y-%m-%d %H:%M:%S")---def get_person_property_updates_windowed(- team_id: int,- bug_window_start: str,- window_seconds: int,- bug_window_end: str | None = None,- logger: Any = None,-) -> list[PersonPropertyDiffs]:- """- Fetch person property updates, optionally in time windows.-- When window_seconds > 0, this function uses a two-step flow:- 1. Query affected person_ids first (returns early if none)- 2. Query raw event aggregates per window (no person comparison)- 3. Merge raw updates across windows with timestamp-based deduplication- 4. Compare final merged updates against current person state-- When window_seconds <= 0, uses the original single-query approach.-- Args:- team_id: Team ID to query- bug_window_start: Start of time window (ClickHouse format: "YYYY-MM-DD HH:MM:SS")- window_seconds: Size of each query window in seconds. If <= 0, single query is used.- bug_window_end: End of time window (defaults to now if not provided)- logger: Optional logger for progress tracking-- Returns:- List of PersonPropertyDiffs with merged results across all windows- """- if window_seconds <= 0:- if logger:- logger.info(f"Using single-query mode for team_id={team_id}")- return get_person_property_updates_from_clickhouse(team_id, bug_window_start)-- now = datetime.now(UTC)- effective_end = bug_window_end or format_ch_timestamp(now)-- # Step 1: Get affected person_ids first (early exit if none)- affected_person_ids = get_affected_person_ids_from_clickhouse(- team_id=team_id,- bug_window_start=bug_window_start,- bug_window_end=effective_end,- )-- if logger:- logger.info(f"Found {len(affected_person_ids)} affected persons for team_id={team_id}")-- if not affected_person_ids:- return []-- person_ids_tuple = tuple(affected_person_ids)-- # Step 2: Query raw updates per window and merge- accumulated: dict[str, RawPersonPropertyUpdates] = {}- current_start = parse_ch_timestamp(bug_window_start)- window_count = 0- total_windows = int((now - current_start).total_seconds() / window_seconds) + 1-- if logger:- logger.info(f"Starting windowed query: team_id={team_id}, ~{total_windows} windows of {window_seconds}s each")-- while current_start < now:- current_end = min(current_start + timedelta(seconds=window_seconds), now)- window_updates = get_raw_person_property_updates_from_clickhouse(- team_id,- format_ch_timestamp(current_start),- format_ch_timestamp(current_end),- person_ids=person_ids_tuple,- )- merge_raw_person_property_updates(accumulated, window_updates)- window_count += 1-- if logger and window_count % 10 == 0:- logger.info(- f"Processed window {window_count}/{total_windows} for team_id={team_id}, "- f"accumulated {len(accumulated)} unique persons"- )-- current_start = current_end-- if logger:- logger.info(- f"Completed all {window_count} windows for team_id={team_id}, total unique persons: {len(accumulated)}"- )-- # Step 3: Compare merged raw updates against person state- if logger:- logger.info(f"Comparing {len(accumulated)} persons against current state for team_id={team_id}")-- return compare_raw_updates_with_person_state(team_id, list(accumulated.values()))---def get_person_property_updates_windowed_batched(- team_id: int,- bug_window_start: str,- bug_window_end: str,- window_seconds: int,- person_batch_size: int = WINDOWED_PERSON_BATCH_SIZE,- logger: Any = None,-) -> Iterator[list[PersonPropertyDiffs]]:- """- Yield batches of PersonPropertyDiffs, processing persons in chunks.-- This is the memory-efficient version of get_person_property_updates_windowed()- that processes persons in batches to prevent OOM on large teams.-- Flow:- 1. Get affected person_ids (bounded by bug_window_end)- 2. For each batch of person_ids:- a. For each time window [start → now()]:- - Query raw events filtered to this batch's person_ids- - Merge into batch accumulator- b. Compare batch with person state- c. Yield batch results (allows caller to process and clear memory)-- Args:- team_id: Team ID to query- bug_window_start: Start of bug window (ClickHouse format: "YYYY-MM-DD HH:MM:SS")- bug_window_end: End of bug window - used to bound affected persons- window_seconds: Size of each query window in seconds- person_batch_size: Number of persons per batch (default 10K)- logger: Optional logger for progress tracking-- Yields:- List of PersonPropertyDiffs for each batch of persons- """- # Step 1: Get affected person_ids (bounded by bug_window_end)- if logger:- logger.info(f"Querying affected persons: team_id={team_id}, bug_window=[{bug_window_start}, {bug_window_end}]")-- try:- affected_person_ids = get_affected_person_ids_from_clickhouse(- team_id=team_id,- bug_window_start=bug_window_start,- bug_window_end=bug_window_end,- )- except Exception:- if logger:- logger.exception(- f"Failed to query affected persons for team_id={team_id}, "- f"bug_window=[{bug_window_start}, {bug_window_end}]"- )- raise-- if logger:- logger.info(f"Found {len(affected_person_ids)} affected persons for team_id={team_id}")-- if not affected_person_ids:- return-- # Calculate total batches and windows for logging- total_batches = (len(affected_person_ids) + person_batch_size - 1) // person_batch_size- now = datetime.now(UTC)- current_start_for_count = parse_ch_timestamp(bug_window_start)- total_windows = int((now - current_start_for_count).total_seconds() / window_seconds) + 1-- if logger:- logger.info(- f"Starting batched windowed processing: team_id={team_id}, "- f"{total_batches} batches of ~{person_batch_size} persons, "- f"~{total_windows} windows of {window_seconds}s each"- )-- # Step 2: Process each batch of persons- for batch_idx in range(0, len(affected_person_ids), person_batch_size):- batch_person_ids = affected_person_ids[batch_idx : batch_idx + person_batch_size]- batch_num = (batch_idx // person_batch_size) + 1-- if logger:- logger.info(- f"Processing batch {batch_num}/{total_batches} for team_id={team_id}, {len(batch_person_ids)} persons"- )-- # Step 2a: Query raw updates per window for this batch and merge- accumulated: dict[str, RawPersonPropertyUpdates] = {}- current_start = parse_ch_timestamp(bug_window_start)- window_count = 0-- try:- while current_start < now:- current_end = min(current_start + timedelta(seconds=window_seconds), now)- try:- window_updates = get_raw_person_property_updates_from_clickhouse(- team_id,- format_ch_timestamp(current_start),- format_ch_timestamp(current_end),- person_ids=tuple(batch_person_ids),- )- except Exception:- if logger:- logger.exception(- f"Batch {batch_num}: failed to query window {window_count + 1}/{total_windows} "- f"[{format_ch_timestamp(current_start)}, {format_ch_timestamp(current_end)}) "- f"for team_id={team_id}"- )- raise-- merge_raw_person_property_updates(accumulated, window_updates)- window_count += 1-- if logger and window_count % 10 == 0:- logger.info(- f"Batch {batch_num}: processed window {window_count}/{total_windows}, "- f"accumulated {len(accumulated)} persons"- )-- current_start = current_end-- except Exception:- if logger:- logger.exception(- f"Batch {batch_num}: failed during windowed event collection for team_id={team_id} "- f"after {window_count} windows, accumulated {len(accumulated)} persons"- )- raise-- if logger:- logger.info(- f"Batch {batch_num}: completed {window_count} windows, total persons with updates: {len(accumulated)}"- )-- # Step 2b: Compare batch with person state- if accumulated:- if logger:- logger.info(f"Batch {batch_num}: comparing {len(accumulated)} persons against current state")-- try:- batch_diffs = compare_raw_updates_with_person_state(team_id, list(accumulated.values()))- except Exception:- if logger:- logger.exception(- f"Batch {batch_num}: failed to compare {len(accumulated)} persons "- f"against current state for team_id={team_id}"- )- raise-- if batch_diffs:- if logger:- logger.info(f"Batch {batch_num}: yielding {len(batch_diffs)} persons with diffs")- # Step 2c: Yield batch results- yield batch_diffs- else:- if logger:- logger.info(f"Batch {batch_num}: no diffs found after comparison")- else:- if logger:- logger.info(f"Batch {batch_num}: no updates found in time windows")-- if logger:- logger.info(f"Completed all {total_batches} batches for team_id={team_id}")---def filter_event_person_properties(- person_diffs: list[PersonPropertyDiffs],-) -> list[PersonPropertyDiffs]:- """- Filter conflicting set/unset operations by timestamp.-- For each key that appears in both set_updates and unset_updates:- - If set timestamp > unset timestamp: keep set, remove unset- - If unset timestamp >= set timestamp: keep unset, remove set-- Returns filtered PersonPropertyDiffs with conflicts resolved.- """- results: list[PersonPropertyDiffs] = []-- for diffs in person_diffs:- # Copy the maps so we don't mutate the input- set_updates = dict(diffs.set_updates)- set_once_updates = dict(diffs.set_once_updates)- unset_updates = dict(diffs.unset_updates)-- # Find keys that exist in both maps- # Use original unset keys for iteration so deletions in first loop don't affect second loop- original_unset_keys = list(diffs.unset_updates.keys())-- for key in original_unset_keys:- if key in set_updates:- set_ts = set_updates[key].timestamp- unset_ts = diffs.unset_updates[key].timestamp-- if set_ts > unset_ts:- # set is more recent - remove unset- del unset_updates[key]- else:- # unset is more recent or equal - remove set- del set_updates[key]-- # Filter set_once vs unset conflicts- for key in original_unset_keys:- if key in set_once_updates:- set_once_ts = set_once_updates[key].timestamp- unset_ts = diffs.unset_updates[key].timestamp-- if set_once_ts > unset_ts:- # Only delete if not already removed by set loop- if key in unset_updates:- del unset_updates[key]- else:- del set_once_updates[key]-- results.append(- PersonPropertyDiffs(- person_id=diffs.person_id,- person_version=diffs.person_version,- set_updates=set_updates,- set_once_updates=set_once_updates,- unset_updates=unset_updates,- )- )-- return results---def reconcile_person_properties(- person: dict,- person_property_diffs: PersonPropertyDiffs,-) -> dict | None:- """- Compute updated properties by comparing ClickHouse updates with current Postgres state.-- The CH query pre-filters to only return:- - $set properties where the CH value differs from current person state- - $set_once properties that are missing from current person state- - $unset properties that exist in current person state-- Args:- person: From Postgres with uuid, properties, properties_last_updated_at, properties_last_operation- person_property_diffs: Property diffs from ClickHouse-- Returns:- Dict with updated properties/metadata if changes needed, None otherwise.- """- properties = dict(person["properties"] or {})- properties_last_updated_at = dict(person["properties_last_updated_at"] or {})- properties_last_operation = dict(person["properties_last_operation"] or {})-- changed = False-- # 1. set_once: only update if key not in properties- for key, pv in person_property_diffs.set_once_updates.items():- if key not in properties:- properties[key] = pv.value- properties_last_updated_at[key] = pv.timestamp.isoformat()- properties_last_operation[key] = "set_once"- changed = True-- # 2. set: always update- for key, pv in person_property_diffs.set_updates.items():- properties[key] = pv.value- properties_last_updated_at[key] = pv.timestamp.isoformat()- properties_last_operation[key] = "set"- changed = True-- # 3. unset: delete from all maps- for key in person_property_diffs.unset_updates.keys():- if key in properties:- del properties[key]- if key in properties_last_updated_at:- del properties_last_updated_at[key]- if key in properties_last_operation:- del properties_last_operation[key]- changed = True-- if changed:- return {- "properties": properties,- "properties_last_updated_at": properties_last_updated_at,- "properties_last_operation": properties_last_operation,- }- return None---def fetch_person_properties_from_clickhouse(team_id: int, person_uuid: str, min_version: int) -> dict | None:- """- Fetch person properties from ClickHouse for conflict resolution.-- Fetches the oldest available version >= min_version. This handles the case where- the exact version we computed diffs against may have been merged away by- ReplacingMergeTree background merges.-- Returns the properties at the oldest available version >= min_version, or None if not found.- """- query = """- SELECT argMin(properties, version) as properties- FROM person- WHERE team_id = %(team_id)s AND id = %(person_id)s AND version >= %(min_version)s- GROUP BY id- """- rows = sync_execute(query, {"team_id": team_id, "person_id": person_uuid, "min_version": min_version})- if not rows:- return None- properties_str = rows[0][0]- if not properties_str:- return {}- return json.loads(properties_str) if isinstance(properties_str, str) else properties_str---def reconcile_with_concurrent_changes(- ch_properties: dict,- postgres_person: dict,- person_property_diffs: PersonPropertyDiffs,-) -> dict | None:- """- 3-way merge: apply event changes while respecting concurrent Postgres changes.-- - Base: ch_properties (the state when event diffs were computed)- - Theirs: postgres_person properties (current state, may have concurrent changes)- - Ours: event_diffs (changes from events)-- Conflict resolution: Postgres wins (concurrent changes take precedence).- This is conservative - we don't overwrite changes made by other processes.- """- postgres_properties = dict(postgres_person["properties"] or {})- properties_last_updated_at = dict(postgres_person["properties_last_updated_at"] or {})- properties_last_operation = dict(postgres_person["properties_last_operation"] or {})-- # Identify what changed in Postgres since CH (concurrent changes)- concurrent_changed_keys: set[str] = set()- all_keys = set(ch_properties.keys()) | set(postgres_properties.keys())- for key in all_keys:- ch_val = ch_properties.get(key)- pg_val = postgres_properties.get(key)- if ch_val != pg_val:- concurrent_changed_keys.add(key)-- changed = False-- # 1. set_once: only update if key not in properties AND not concurrently changed- for key, pv in person_property_diffs.set_once_updates.items():- if key not in postgres_properties and key not in concurrent_changed_keys:- postgres_properties[key] = pv.value- properties_last_updated_at[key] = pv.timestamp.isoformat()- properties_last_operation[key] = "set_once"- changed = True-- # 2. set: only update if not concurrently changed- for key, pv in person_property_diffs.set_updates.items():- if key not in concurrent_changed_keys:- postgres_properties[key] = pv.value- properties_last_updated_at[key] = pv.timestamp.isoformat()- properties_last_operation[key] = "set"- changed = True-- # 3. unset: only delete if not concurrently changed- for key in person_property_diffs.unset_updates.keys():- if key in postgres_properties and key not in concurrent_changed_keys:- del postgres_properties[key]- if key in properties_last_updated_at:- del properties_last_updated_at[key]- if key in properties_last_operation:- del properties_last_operation[key]- changed = True-- if changed:- return {- "properties": postgres_properties,- "properties_last_updated_at": properties_last_updated_at,- "properties_last_operation": properties_last_operation,- }- return None---def publish_person_to_kafka(person_data: dict, producer: _KafkaProducer) -> None:- """- Publish a person update to the Kafka topic for ClickHouse ingestion.-- Args:- person_data: Dict with id, team_id, properties, is_identified, is_deleted, created_at, version- producer: Kafka producer instance (injected resource)- """- from django.utils.timezone import now-- # Format data for Kafka/ClickHouse- created_at = person_data.get("created_at")- if created_at is not None and hasattr(created_at, "strftime"):- created_at_str = created_at.strftime("%Y-%m-%d %H:%M:%S.%f")- else:- created_at_str = str(created_at) if created_at else now().strftime("%Y-%m-%d %H:%M:%S.%f")-- kafka_data = {- "id": str(person_data["id"]),- "team_id": person_data["team_id"],- "properties": json.dumps(person_data["properties"]),- "is_identified": int(person_data.get("is_identified", False)),- "is_deleted": int(person_data.get("is_deleted", 0)),- "created_at": created_at_str,- "version": person_data["version"],- }-- producer.produce(topic=KAFKA_PERSON, data=kafka_data)---def fetch_person_from_postgres(cursor, team_id: int, person_uuid: str) -> dict | None:- """Fetch a single person record from Postgres.-- Works with both RealDictCursor (returns dict rows with auto-parsed JSONB)- and regular cursor (returns tuple rows with JSONB as strings).- """-- columns = [- "id",- "uuid",- "properties",- "properties_last_updated_at",- "properties_last_operation",- "version",- "is_identified",- "created_at",- ]-- cursor.execute(- """- SELECT- id,- uuid::text,- properties,- properties_last_updated_at,- properties_last_operation,- version,- is_identified,- created_at- FROM posthog_person- WHERE team_id = %s AND uuid = %s::uuid AND is_deleted = false- """,- (team_id, person_uuid),- )- row = cursor.fetchone()- if row is None:- return None- # Handle both dict rows (RealDictCursor) and tuple rows (regular cursor)- if isinstance(row, dict):- result = dict(row)- else:- result = dict(zip(columns, row))-- # Parse JSONB columns if they're strings (Django cursor doesn't auto-parse)- for json_col in ["properties", "properties_last_updated_at", "properties_last_operation"]:- if isinstance(result.get(json_col), str):- result[json_col] = json.loads(result[json_col])-- return result---def backup_person_with_computed_state(- cursor,- job_id: str,- team_id: int,- person: dict,- person_property_diffs: PersonPropertyDiffs,- computed_update: dict,- new_version: int,-) -> bool:- """- Store person state in backup table with both before and after states.- Called after computing the update but before applying it.- Returns True if backup was successful, False otherwise.- """- pending_operations = []- for key, pv in person_property_diffs.set_updates.items():- pending_operations.append(- {"key": key, "value": pv.value, "timestamp": pv.timestamp.isoformat(), "operation": "set"}- )- for key, pv in person_property_diffs.set_once_updates.items():- pending_operations.append(- {"key": key, "value": pv.value, "timestamp": pv.timestamp.isoformat(), "operation": "set_once"}- )- for key, pv in person_property_diffs.unset_updates.items():- pending_operations.append(- {"key": key, "value": None, "timestamp": pv.timestamp.isoformat(), "operation": "unset"}- )-- cursor.execute(- """- INSERT INTO posthog_person_reconciliation_backup (- job_id, team_id, person_id, uuid,- properties, properties_last_updated_at, properties_last_operation,- version, is_identified, created_at, is_user_id,- pending_operations,- properties_after, properties_last_updated_at_after,- properties_last_operation_after, version_after- ) VALUES (- %s, %s, %s, %s::uuid,- %s, %s, %s,- %s, %s, %s, %s,- %s,- %s, %s, %s, %s- )- ON CONFLICT (job_id, team_id, person_id) DO NOTHING- """,- (- job_id,- team_id,- person["id"],- str(person["uuid"]),- json.dumps(person.get("properties", {})),- json.dumps(person.get("properties_last_updated_at")),- json.dumps(person.get("properties_last_operation")),- person.get("version"),- person.get("is_identified", False),- person.get("created_at"),- person.get("is_user_id"),- json.dumps(pending_operations),- json.dumps(computed_update["properties"]),- json.dumps(computed_update["properties_last_updated_at"]),- json.dumps(computed_update["properties_last_operation"]),- new_version,- ),- )- return cursor.rowcount > 0---@frozen-class PersonUpdateResult:- success: bool- updated_person_data: dict | None- backup_created: bool- skip_reason: str---def update_person_with_version_check(- cursor,- job_id: str,- team_id: int,- person_uuid: str,- person_property_diffs: PersonPropertyDiffs,- dry_run: bool = False,- backup_enabled: bool = True,- max_retries: int = 3,-) -> PersonUpdateResult:- """- Update a person's properties with optimistic locking and conflict resolution.-- Flow:- 1. Fetch person from Postgres- 2. If Postgres version == CH version: apply diffs normally, UPDATE WHERE version = ch_version- 3. If versions differ (conflict): fetch CH properties, do 3-way merge, UPDATE WHERE version = postgres_version- 4. On UPDATE failure, retry-- Args:- cursor: Database cursor- job_id: Dagster run ID for backup tracking- team_id: Team ID- person_uuid: Person UUID- person_property_diffs: Property diffs from ClickHouse (includes person_version)- dry_run: If True, don't actually write the UPDATE- backup_enabled: If True, store before/after state in backup table- max_retries: Maximum retry attempts on version mismatch-- Returns:- PersonUpdateResult where:- - updated_person_data contains the final state for Kafka publishing- - backup_created indicates if a backup row was inserted- - skip_reason indicates why the person was skipped (see SkipReason class)- """- ch_version = person_property_diffs.person_version-- for _attempt in range(max_retries):- # Fetch current person state from Postgres- person = fetch_person_from_postgres(cursor, team_id, person_uuid)- if not person:- return PersonUpdateResult(- success=False, updated_person_data=None, backup_created=False, skip_reason=SkipReason.NOT_FOUND- )-- postgres_version = person.get("version") or 0-- # Check if Postgres and CH are in sync- if postgres_version == ch_version:- # Simple case: no concurrent changes, apply diffs directly- update = reconcile_person_properties(person, person_property_diffs)- target_version = ch_version- else:- # Conflict: Postgres has different version than CH- # Fetch CH properties at the version we computed diffs against for 3-way merge- ch_properties = fetch_person_properties_from_clickhouse(team_id, person_uuid, ch_version)- if ch_properties is None:- # Person version doesn't exist in CH anymore, skip- return PersonUpdateResult(- success=False, updated_person_data=None, backup_created=False, skip_reason=SkipReason.NOT_FOUND- )-- # 3-way merge: apply our changes while respecting concurrent Postgres changes- update = reconcile_with_concurrent_changes(ch_properties, person, person_property_diffs)- target_version = postgres_version-- if not update:- # No changes needed (either no diffs or all conflicts resolved to Postgres values)- return PersonUpdateResult(- success=True, updated_person_data=None, backup_created=False, skip_reason=SkipReason.NO_CHANGES- )-- # Backup before and after state for audit/rollback- backup_created = False- if backup_enabled:- backup_created = backup_person_with_computed_state(- cursor, job_id, team_id, person, person_property_diffs, update, target_version + 1- )-- if dry_run:- return PersonUpdateResult(- success=True, updated_person_data=None, backup_created=backup_created, skip_reason=SkipReason.SUCCESS- )-- # Write with version check- cursor.execute(- """- UPDATE posthog_person SET- properties = %s,- properties_last_updated_at = %s,- properties_last_operation = %s,- version = %s- WHERE team_id = %s AND uuid = %s::uuid AND version = %s- RETURNING uuid- """,- (- json.dumps(update["properties"]),- json.dumps(update["properties_last_updated_at"]),- json.dumps(update["properties_last_operation"]),- target_version + 1,- team_id,- person_uuid,- target_version,- ),- )-- if cursor.rowcount > 0:- # Success - return data for Kafka publishing- return PersonUpdateResult(- success=True,- updated_person_data={- "id": person_uuid,- "team_id": team_id,- "properties": update["properties"],- "is_identified": person.get("is_identified", False),- "is_deleted": 0,- "created_at": person.get("created_at"),- "version": target_version + 1,- },- backup_created=backup_created,- skip_reason=SkipReason.SUCCESS,- )-- # Version mismatch during UPDATE - retry with fresh data- # (loop will re-fetch person)-- # Exhausted retries- return PersonUpdateResult(- success=False, updated_person_data=None, backup_created=False, skip_reason=SkipReason.VERSION_CONFLICT- )---@dataclass-class BatchProcessingResult:- """Result of processing persons in batches."""-- total_processed: int- total_updated: int- total_skipped: int- total_commits: int---def process_persons_in_batches(- person_property_diffs: list[PersonPropertyDiffs],- cursor: Any,- job_id: str,- team_id: int,- batch_size: int,- dry_run: bool,- backup_enabled: bool,- commit_fn: Any, # Callable to commit transaction- on_person_updated: Any = None, # Optional callback for Kafka publishing etc- on_batch_committed: Any = None, # Optional callback after each batch commit- logger: Any = None, # Optional logger for logging skipped persons-) -> BatchProcessingResult:- """- Process person property updates in batches with configurable commit frequency.-- This function is separated from the Dagster op for testability.-- Args:- person_property_diffs: List of persons with their property diffs- cursor: Database cursor (already open in transaction)- job_id: Job identifier for backups- team_id: Team being processed- batch_size: Number of persons per batch (0 = all in one batch)- dry_run: If True, don't actually apply changes- backup_enabled: If True, create backup rows- commit_fn: Function to call for committing the transaction- on_person_updated: Optional callback(person_data) when a person is updated- on_batch_committed: Optional callback(batch_num, batch_persons) after each commit- logger: Optional logger for logging skipped persons (e.g., context.log from Dagster)-- Returns:- BatchProcessingResult with counts- """- if not person_property_diffs:- return BatchProcessingResult(- total_processed=0,- total_updated=0,- total_skipped=0,- total_commits=0,- )-- # Determine effective batch size- effective_batch_size = batch_size if batch_size > 0 else len(person_property_diffs)-- total_processed = 0- total_updated = 0- total_skipped = 0- total_commits = 0-- batch_persons_to_publish: list[dict] = []- batch_has_backups = False- batch_num = 1-- def commit_batch() -> None:- nonlocal total_commits, batch_persons_to_publish, batch_has_backups, batch_num-- if not batch_persons_to_publish and not batch_has_backups:- return-- # Commit if we have updates OR backups- should_commit = (batch_persons_to_publish and not dry_run) or batch_has_backups- if should_commit:- commit_fn()- total_commits += 1-- # Notify callback- if on_batch_committed:- on_batch_committed(batch_num, batch_persons_to_publish)-- # Reset for next batch- batch_persons_to_publish = []- batch_has_backups = False- batch_num += 1-- for i, person_diffs in enumerate(person_property_diffs):- update_result = update_person_with_version_check(- cursor=cursor,- job_id=job_id,- team_id=team_id,- person_uuid=person_diffs.person_id,- person_property_diffs=person_diffs,- dry_run=dry_run,- backup_enabled=backup_enabled,- )-- if update_result.backup_created:- batch_has_backups = True-- if not update_result.success:- total_skipped += 1- if logger:- logger.warning(- f"Skipped person uuid={person_diffs.person_id} team_id={team_id} reason={update_result.skip_reason}"- )- elif update_result.updated_person_data:- batch_persons_to_publish.append(update_result.updated_person_data)- total_updated += 1- if on_person_updated:- on_person_updated(update_result.updated_person_data)- else:- # No changes needed (success=True but updated_person_data=None)- total_skipped += 1-- total_processed += 1-- # Commit batch if we've reached batch_size- persons_in_batch = (i % effective_batch_size) + 1- if persons_in_batch == effective_batch_size:- commit_batch()-- # Commit final batch (if any remaining)- if batch_persons_to_publish or batch_has_backups:- commit_batch()-- return BatchProcessingResult(- total_processed=total_processed,- total_updated=total_updated,- total_skipped=total_skipped,- total_commits=total_commits,- )---def query_team_ids_from_clickhouse(- bug_window_start: str,- bug_window_end: str,- min_team_id: int | None = None,- max_team_id: int | None = None,- exclude_team_ids: list[int] | None = None,- include_team_ids: list[int] | None = None,-) -> list[int]:- """- Query ClickHouse for distinct team_ids with property-setting events in the bug window.-- Args:- bug_window_start: Start of bug window (CH format: "YYYY-MM-DD HH:MM:SS")- bug_window_end: End of bug window (CH format: "YYYY-MM-DD HH:MM:SS")- min_team_id: Optional minimum team_id (inclusive)- max_team_id: Optional maximum team_id (inclusive)- exclude_team_ids: Optional list of team_ids to exclude- include_team_ids: Optional list of team_ids to include (only these teams will be queried)-- Returns:- List of team_ids sorted ascending-- Raises:- ValueError: If min_team_id > max_team_id- """- if min_team_id is not None and max_team_id is not None and min_team_id > max_team_id:- raise ValueError(- f"Invalid team_id range: min_team_id ({min_team_id}) cannot be greater than max_team_id ({max_team_id})"- )-- team_id_filters = []- params: dict[str, Any] = {- "bug_window_start": bug_window_start,- "bug_window_end": bug_window_end,- }-- if include_team_ids:- team_id_filters.append("team_id IN %(include_team_ids)s")- params["include_team_ids"] = tuple(include_team_ids)-- if min_team_id is not None:- team_id_filters.append("team_id >= %(min_team_id)s")- params["min_team_id"] = min_team_id-- if max_team_id is not None:- team_id_filters.append("team_id <= %(max_team_id)s")- params["max_team_id"] = max_team_id-- if exclude_team_ids:- team_id_filters.append("team_id NOT IN %(exclude_team_ids)s")- params["exclude_team_ids"] = tuple(exclude_team_ids)-- team_id_filter_clause = (" AND " + " AND ".join(team_id_filters)) if team_id_filters else ""-- query = f"""- SELECT DISTINCT team_id- FROM events- WHERE timestamp >= %(bug_window_start)s- AND timestamp < %(bug_window_end)s- AND (- JSONHas(properties, '$set') = 1- OR JSONHas(properties, '$set_once') = 1- OR JSONHas(properties, '$unset') = 1- ){team_id_filter_clause}- ORDER BY team_id- """-- results = sync_execute(query, params)- return [int(row[0]) for row in results]---def get_team_ids_to_reconcile(- context: dagster.OpExecutionContext,- config: PersonPropertyReconciliationConfig,- cluster: dagster.ResourceParam[ClickhouseCluster],-) -> list[int]:- """- Query ClickHouse for distinct team_ids with property-setting events in the bug window.- """- if not config.bug_window_end:- raise dagster.Failure(- description="bug_window_end must be provided",- metadata={- "bug_window_start": dagster.MetadataValue.text(config.bug_window_start),- },- )-- filter_info_parts = []- if config.team_ids:- filter_info_parts.append(f"team_ids: {config.team_ids}")- if config.min_team_id is not None or config.max_team_id is not None:- filter_info_parts.append(f"range: {config.min_team_id or 'any'} to {config.max_team_id or 'any'}")- if config.exclude_team_ids:- filter_info_parts.append(f"excluding: {config.exclude_team_ids}")- filter_info = f" ({', '.join(filter_info_parts)})" if filter_info_parts else ""-- context.log.info(- f"Querying for team_ids with property events between {config.bug_window_start} and {config.bug_window_end}{filter_info}"- )-- team_ids = query_team_ids_from_clickhouse(- bug_window_start=config.bug_window_start,- bug_window_end=config.bug_window_end,- min_team_id=config.min_team_id,- max_team_id=config.max_team_id,- exclude_team_ids=config.exclude_team_ids,- include_team_ids=config.team_ids,- )-- if not team_ids:- context.log.info("No team IDs found with property events in bug window")- return []-- context.log.info(f"Found {len(team_ids)} teams with property events")- context.log.info(f"Sample of teams to process: {team_ids[:10]}" + ("..." if len(team_ids) > 10 else ""))- context.add_output_metadata({"team_count": dagster.MetadataValue.int(len(team_ids))})-- return team_ids--[email protected](out=dagster.DynamicOut(list[int]))-def create_team_chunks(- context: dagster.OpExecutionContext,- config: PersonPropertyReconciliationConfig,- team_ids: list[int],-):- """- Create chunks of team_ids based on teams_per_chunk config.- Yields DynamicOutput for each chunk containing one or more teams.- """- if not team_ids:- context.log.info("No teams to process")- return-- teams_per_chunk = max(1, config.teams_per_chunk)- num_chunks = (len(team_ids) + teams_per_chunk - 1) // teams_per_chunk- context.log.info(f"Creating {num_chunks} chunks for {len(team_ids)} teams (teams_per_chunk={teams_per_chunk})")-- for i in range(0, len(team_ids), teams_per_chunk):- chunk = team_ids[i : i + teams_per_chunk]- chunk_key = f"teams_{chunk[0]}_{chunk[-1]}" if len(chunk) > 1 else f"team_{chunk[0]}"- context.log.info(- f"Yielding chunk with {len(chunk)} teams: {chunk[0]}-{chunk[-1] if len(chunk) > 1 else chunk[0]}"- )- yield dagster.DynamicOutput(- value=chunk,- mapping_key=chunk_key,- )---@dataclass-class TeamReconciliationResult:- """Result of reconciling a single team."""-- team_id: int- persons_processed: int- persons_updated: int- persons_skipped: int---def reconcile_single_team(- team_id: int,- bug_window_start: str,- bug_window_end: str,- run_id: str,- batch_size: int,- dry_run: bool,- backup_enabled: bool,- team_ch_props_fetch_window_seconds: int,- person_batch_size: int,- persons_database: psycopg2.extensions.connection,- kafka_producer: _KafkaProducer,- logger: Any,-) -> TeamReconciliationResult:- """- Reconcile person properties for all affected persons in a single team.-- This function supports two modes:- - Non-windowed (team_ch_props_fetch_window_seconds <= 0): Single query to ClickHouse- - Windowed batched (team_ch_props_fetch_window_seconds > 0): Process persons in batches,- each batch iterating through time windows. This prevents OOM on large teams.-- This function does not catch exceptions - the caller is responsible for error handling.- """-- # Callback for batch commits - handles Kafka publishing after each batch- def on_batch_committed(batch_num: int, batch_persons: list[dict]) -> None:- if batch_persons and not dry_run:- for person_data in batch_persons:- try:- publish_person_to_kafka(person_data, kafka_producer)- except Exception as kafka_error:- logger.warning(f"Failed to publish person to Kafka: {person_data['id']}, error: {kafka_error}")- try:- kafka_producer.flush()- except Exception as flush_error:- logger.warning(f"Failed to flush Kafka producer: {flush_error}")-- logger.info(f"Batch {batch_num}: committed {len(batch_persons)} updates for team_id={team_id}")- elif batch_persons and dry_run:- logger.info(f"[DRY RUN] Batch {batch_num}: would apply {len(batch_persons)} updates for team_id={team_id}")-- # Use windowed batched mode for large teams (window_seconds > 0)- if team_ch_props_fetch_window_seconds > 0:- logger.info(- f"Using windowed batched mode: team_id={team_id}, "- f"bug_window=[{bug_window_start}, {bug_window_end}], "- f"window_seconds={team_ch_props_fetch_window_seconds}, "- f"person_batch_size={person_batch_size}"- )-- total_processed = 0- total_updated = 0- total_skipped = 0- person_batch_num = 0-- with persons_database.cursor() as cursor:- cursor.execute("SET application_name = 'person_property_reconciliation'")- cursor.execute("SET lock_timeout = '5s'")- cursor.execute("SET statement_timeout = '30min'")- cursor.execute("SET synchronous_commit = off")-- # Process each batch of persons from the iterator- for person_batch in get_person_property_updates_windowed_batched(- team_id=team_id,- bug_window_start=bug_window_start,- bug_window_end=bug_window_end,- window_seconds=team_ch_props_fetch_window_seconds,- person_batch_size=person_batch_size,- logger=logger,- ):- person_batch_num += 1-- # Filter conflicting set/unset operations for this batch- filtered_batch = filter_event_person_properties(person_batch)-- if not filtered_batch:- logger.info(f"Person batch {person_batch_num}: no diffs after filtering for team_id={team_id}")- continue-- logger.info(- f"Person batch {person_batch_num}: processing {len(filtered_batch)} persons for team_id={team_id}"- )-- # Process this batch- result = process_persons_in_batches(- person_property_diffs=filtered_batch,- cursor=cursor,- job_id=run_id,- team_id=team_id,- batch_size=batch_size,- dry_run=dry_run,- backup_enabled=backup_enabled,- commit_fn=persons_database.commit,- on_batch_committed=on_batch_committed,- logger=logger,- )-- total_processed += result.total_processed- total_updated += result.total_updated- total_skipped += result.total_skipped-- logger.info(- f"Completed team_id={team_id} (windowed batched): processed={total_processed}, "- f"updated={total_updated}, skipped={total_skipped}, person_batches={person_batch_num}"- )-- return TeamReconciliationResult(- team_id=team_id,- persons_processed=total_processed,- persons_updated=total_updated,- persons_skipped=total_skipped,- )-- # Non-windowed mode (original behavior)- logger.info(f"Using non-windowed mode: team_id={team_id}, bug_window_start={bug_window_start}")- person_property_diffs = get_person_property_updates_from_clickhouse(team_id, bug_window_start)- logger.info(f"Found {len(person_property_diffs)} persons with property diffs for team_id={team_id}")-- # Filter conflicting set/unset operations- person_property_diffs = filter_event_person_properties(person_property_diffs)-- if not person_property_diffs:- logger.info(f"No persons to reconcile for team_id={team_id}")- return TeamReconciliationResult(- team_id=team_id,- persons_processed=0,- persons_updated=0,- persons_skipped=0,- )-- logger.info(f"Processing {len(person_property_diffs)} persons for team_id={team_id}")-- with persons_database.cursor() as cursor:- cursor.execute("SET application_name = 'person_property_reconciliation'")- cursor.execute("SET lock_timeout = '5s'")- cursor.execute("SET statement_timeout = '30min'")- cursor.execute("SET synchronous_commit = off")-- # Process persons in batches using the helper function- result = process_persons_in_batches(- person_property_diffs=person_property_diffs,- cursor=cursor,- job_id=run_id,- team_id=team_id,- batch_size=batch_size,- dry_run=dry_run,- backup_enabled=backup_enabled,- commit_fn=persons_database.commit,- on_batch_committed=on_batch_committed,- logger=logger,- )-- logger.info(- f"Completed team_id={team_id}: processed={result.total_processed}, "- f"updated={result.total_updated}, skipped={result.total_skipped}"- )-- return TeamReconciliationResult(- team_id=team_id,- persons_processed=result.total_processed,- persons_updated=result.total_updated,- persons_skipped=result.total_skipped,- )---def reconcile_team_chunk(- context: dagster.OpExecutionContext,- config: PersonPropertyReconciliationConfig,- chunk: list[int],- persons_database: dagster.ResourceParam[psycopg2.extensions.connection],- cluster: dagster.ResourceParam[ClickhouseCluster],- kafka_producer: dagster.ResourceParam[_KafkaProducer],-) -> dict[str, Any]:- """- Reconcile person properties for all affected persons in a chunk of teams.- """- team_ids = chunk- chunk_id = f"teams_{team_ids[0]}_{team_ids[-1]}" if len(team_ids) > 1 else f"team_{team_ids[0]}"- job_name = context.run.job_name- run_id = context.run.run_id-- metrics_client = MetricsClient(cluster)-- context.log.info(- f"Starting reconciliation for {len(team_ids)} teams: {team_ids}, "- f"bug_window_start: {config.bug_window_start}, "- f"dry_run: {config.dry_run}"- )-- total_persons_processed = 0- total_persons_updated = 0- total_persons_skipped = 0- total_teams_succeeded = 0- total_teams_failed = 0- teams_results: list[dict[str, Any]] = []-- start_time = time.time()-- for team_id in team_ids:- team_result: dict[str, Any] = {- "team_id": team_id,- "status": "success",- "persons_processed": 0,- "persons_updated": 0,- "persons_skipped": 0,- }-- try:- result = reconcile_single_team(- team_id=team_id,- bug_window_start=config.bug_window_start,- bug_window_end=config.bug_window_end or config.bug_window_start, # Fallback for backward compat- run_id=run_id,- batch_size=config.batch_size,- dry_run=config.dry_run,- backup_enabled=config.backup_enabled,- team_ch_props_fetch_window_seconds=config.team_ch_props_fetch_window_seconds,- person_batch_size=config.person_batch_size,- persons_database=persons_database,- kafka_producer=kafka_producer,- logger=context.log,- )- team_result["persons_processed"] = result.persons_processed- team_result["persons_updated"] = result.persons_updated- team_result["persons_skipped"] = result.persons_skipped- total_teams_succeeded += 1-- except Exception as e:- team_result["status"] = "failed"- team_result["error"] = str(e)- total_teams_failed += 1-- context.log.exception(f"Failed team_id={team_id}: {e}")-- try:- metrics_client.increment(- "person_property_reconciliation_error",- labels={"job_name": job_name, "chunk_id": chunk_id, "team_id": str(team_id), "reason": "error"},- value=1.0,- ).result()- except Exception:- pass-- total_persons_processed += team_result["persons_processed"]- total_persons_updated += team_result["persons_updated"]- total_persons_skipped += team_result["persons_skipped"]- teams_results.append(team_result)-- # Track metrics for the entire chunk- duration = time.time() - start_time- try:- metrics_client.increment(- "person_property_reconciliation_persons_processed_total",- labels={"job_name": job_name, "chunk_id": chunk_id},- value=float(total_persons_processed),- ).result()- metrics_client.increment(- "person_property_reconciliation_duration_seconds_total",- labels={"job_name": job_name, "chunk_id": chunk_id},- value=duration,- ).result()- metrics_client.increment(- "person_property_reconciliation_persons_updated_total",- labels={"job_name": job_name, "chunk_id": chunk_id},- value=float(total_persons_updated),- ).result()- metrics_client.increment(- "person_property_reconciliation_persons_skipped_total",- labels={"job_name": job_name, "chunk_id": chunk_id},- value=float(total_persons_skipped),- ).result()- metrics_client.increment(- "person_property_reconciliation_teams_succeeded_total",- labels={"job_name": job_name, "chunk_id": chunk_id},- value=float(total_teams_succeeded),- ).result()- metrics_client.increment(- "person_property_reconciliation_teams_failed_total",- labels={"job_name": job_name, "chunk_id": chunk_id},- value=float(total_teams_failed),- ).result()- except Exception:- pass-- # Log failed teams for easy searching- failed_teams = [r for r in teams_results if r["status"] == "failed"]- if failed_teams:- failed_team_ids = [r["team_id"] for r in failed_teams]- context.log.warning(f"Failed teams in chunk {chunk_id}: {failed_team_ids}")-- context.log.info(- f"Completed chunk {chunk_id}: teams={len(team_ids)} (succeeded={total_teams_succeeded}, failed={total_teams_failed}), "- f"persons: processed={total_persons_processed}, updated={total_persons_updated}, skipped={total_persons_skipped}"- )-- context.add_output_metadata(- {- "team_ids": dagster.MetadataValue.text(str(team_ids)),- "teams_count": dagster.MetadataValue.int(len(team_ids)),- "teams_succeeded": dagster.MetadataValue.int(total_teams_succeeded),- "teams_failed": dagster.MetadataValue.int(total_teams_failed),- "failed_team_ids": dagster.MetadataValue.text(str([r["team_id"] for r in failed_teams])),- "persons_processed": dagster.MetadataValue.int(total_persons_processed),- "persons_updated": dagster.MetadataValue.int(total_persons_updated),- "persons_skipped": dagster.MetadataValue.int(total_persons_skipped),- "teams_results": dagster.MetadataValue.json(teams_results),- }- )-- return {- "team_ids": team_ids,- "teams_count": len(team_ids),- "teams_succeeded": total_teams_succeeded,- "teams_failed": total_teams_failed,- "persons_processed": total_persons_processed,- "persons_updated": total_persons_updated,- "persons_skipped": total_persons_skipped,- "teams_results": teams_results,- }--- tags={"owner": JobOwners.TEAM_INGESTION.value},- executor_def=executor_def,-)-def person_property_reconciliation_job():- """- One-time job to reconcile person properties that were missed due to the- ASSERT_VERSION bug.-- IMPORTANT: This job uses k8s_job_executor which spawns a separate Kubernetes pod- for each team. Without concurrency limits, this could spin up thousands of pods- simultaneously. Before running:-- 1. Set step concurrency limits in the Dagster UI (Deployment > Configuration)- or via the run coordinator's concurrency settings- 2. Consider using the `team_ids` config to process a subset of teams first- 3. Monitor cluster resources during execution- """- team_ids = get_team_ids_to_reconcile()- chunks = create_team_chunks(team_ids)- chunks.map(reconcile_team_chunk)---# --- Sensor-based scheduler for automated batch reconciliation ------class ReconciliationSchedulerConfig(dagster.Config):- """Configuration for the reconciliation scheduler sensor.-- This sensor automates launching multiple reconciliation jobs across a range- of team IDs while respecting concurrency limits.-- You must provide EITHER:- - team_ids: explicit list of team IDs to process- - OR both range_start and range_end: to scan a team ID range- """-- # Option 1: Explicit list of team IDs- team_ids: list[int] | None = None-- # Option 2: Range-based scanning- range_start: int | None = None # First team_id (inclusive)- range_end: int | None = None # Last team_id (inclusive)-- chunk_size: int = 1000 # Teams per job run (for range mode) or list chunk size (for team_ids mode)- max_concurrent_jobs: int = 5 # Max reconciliation jobs the sensor can schedule at once- max_concurrent_tasks: int = 10 # Max k8s pods per job (executor concurrency)-- # Base job configuration (applied to all runs)- bug_window_start: str # ClickHouse format: "YYYY-MM-DD HH:MM:SS"- bug_window_end: str # ClickHouse format: "YYYY-MM-DD HH:MM:SS"- dry_run: bool = False- backup_enabled: bool = True- batch_size: int = 100- teams_per_chunk: int = 100 # Number of teams to process per task- team_ch_props_fetch_window_seconds: int = 0 # 0 = single query; >0 = split into N-second windows- person_batch_size: int = 10000 # Persons per batch in windowed mode (keeps IN clauses small)-- # Resource configuration - the env var name for the PG connection string- persons_db_env_var: str = "PERSONS_DB_WRITER_URL" # Env var for Postgres connection URL---def build_reconciliation_run_config(- config: ReconciliationSchedulerConfig,- min_team_id: int | None = None,- max_team_id: int | None = None,- team_ids: list[int] | None = None,-) -> dict[str, Any]:- """Build run config for a single reconciliation job.-- Either provide team_ids (explicit list) OR min_team_id/max_team_id (range scan).- """- op_config: dict[str, Any] = {- "bug_window_start": config.bug_window_start,- "bug_window_end": config.bug_window_end,- "dry_run": config.dry_run,- "backup_enabled": config.backup_enabled,- "batch_size": config.batch_size,- "teams_per_chunk": config.teams_per_chunk,- "team_ch_props_fetch_window_seconds": config.team_ch_props_fetch_window_seconds,- "person_batch_size": config.person_batch_size,- }-- if team_ids is not None:- op_config["team_ids"] = team_ids- else:- op_config["min_team_id"] = min_team_id- op_config["max_team_id"] = max_team_id-- return {- "execution": {"config": {"max_concurrent": config.max_concurrent_tasks}},- "ops": {- "get_team_ids_to_reconcile": {"config": op_config},- "create_team_chunks": {"config": op_config},- "reconcile_team_chunk": {"config": op_config},- },- "resources": {- "cluster": {- "config": {- "client_settings": {- "lightweight_deletes_sync": "0",- "max_execution_time": "0",- "max_memory_usage": "0",- "mutations_sync": "0",- "receive_timeout": "900",- }- }- },- "persons_database": {"config": {"connection_url": {"env": config.persons_db_env_var}}},- },- }--- job=person_property_reconciliation_job,- minimum_interval_seconds=30,- default_status=dagster.DefaultSensorStatus.STOPPED,-)-def person_property_reconciliation_scheduler(context: dagster.SensorEvaluationContext):- """- Sensor that automatically schedules person property reconciliation jobs.-- Supports two modes:- 1. team_ids mode: Process an explicit list of team IDs- 2. range mode: Scan a team_id range (range_start to range_end)-- Splits work into chunks and launches jobs up to max_concurrent_jobs,- tracking progress via cursor. Enable via Dagster UI after configuring.-- Cursor format (team_ids mode):- {"team_ids": [1, 2, 3, ...], "next_team_index": 0, ...}- Cursor format (range mode):- {"range_start": 1, "range_end": 10000, "next_chunk_start": 1, ...}-- Configuration (set via Dagster UI or launchpad):- - team_ids: Explicit list of team IDs (mutually exclusive with range_start/range_end)- - range_start/range_end: Team ID range to process (inclusive)- - chunk_size: Number of teams per job (default 1000)- - max_concurrent_jobs: Max reconciliation jobs sensor can schedule at once (default 5, cap 50)- - max_concurrent_tasks: Max k8s pods per job / executor concurrency (default 10, cap 100)- - bug_window_start/end: Time window for the reconciliation- - dry_run, backup_enabled, batch_size: Passed to each job- """- sensor_config_raw = context.cursor-- if not sensor_config_raw:- return dagster.SkipReason(- "No cursor set. Initialize by setting cursor to JSON. "- 'For team_ids mode: {"team_ids": [1, 2, 3], "chunk_size": 100, ...}. '- 'For range mode: {"range_start": 1, "range_end": 10000, "chunk_size": 1000, ...}. '- 'Common fields: "max_concurrent_jobs", "max_concurrent_tasks", "bug_window_start", "bug_window_end", '- '"dry_run", "backup_enabled", "batch_size"'- )-- try:- cursor_data = json.loads(sensor_config_raw)- except json.JSONDecodeError:- return dagster.SkipReason(f"Invalid cursor JSON: {sensor_config_raw[:100]}...")-- # Extract common config- chunk_size = cursor_data.get("chunk_size", 1000)- max_concurrent_jobs = cursor_data.get("max_concurrent_jobs", 5)- max_concurrent_tasks = cursor_data.get("max_concurrent_tasks", 10)- bug_window_start = cursor_data.get("bug_window_start", "")- bug_window_end = cursor_data.get("bug_window_end", "")-- # Determine mode: team_ids vs range- team_ids = cursor_data.get("team_ids")- range_start = cursor_data.get("range_start")- range_end = cursor_data.get("range_end")-- has_team_ids = team_ids is not None and len(team_ids) > 0- has_range = range_start is not None and range_end is not None-- # Validate: must have exactly one mode- if has_team_ids and has_range:- return dagster.SkipReason(- "Invalid config: cannot specify both team_ids and range_start/range_end. Choose one mode."- )- if not has_team_ids and not has_range:- return dagster.SkipReason(- "Invalid config: must specify either team_ids (list) OR both range_start and range_end"- )-- # Validate common config- MAX_CONCURRENT_JOBS_CAP = 50- MAX_CONCURRENT_TASKS_CAP = 100- if chunk_size <= 0:- return dagster.SkipReason(f"Invalid config: chunk_size must be > 0, got {chunk_size}")- if max_concurrent_jobs <= 0:- return dagster.SkipReason(f"Invalid config: max_concurrent_jobs must be > 0, got {max_concurrent_jobs}")- if max_concurrent_jobs > MAX_CONCURRENT_JOBS_CAP:- return dagster.SkipReason(- f"Invalid config: max_concurrent_jobs ({max_concurrent_jobs}) exceeds cap of {MAX_CONCURRENT_JOBS_CAP}"- )- if max_concurrent_tasks <= 0:- return dagster.SkipReason(f"Invalid config: max_concurrent_tasks must be > 0, got {max_concurrent_tasks}")- if max_concurrent_tasks > MAX_CONCURRENT_TASKS_CAP:- return dagster.SkipReason(- f"Invalid config: max_concurrent_tasks ({max_concurrent_tasks}) exceeds cap of {MAX_CONCURRENT_TASKS_CAP}"- )- if not bug_window_start:- return dagster.SkipReason("Invalid config: bug_window_start is required")- if not bug_window_end:- return dagster.SkipReason("Invalid config: bug_window_end is required")-- # Mode-specific validation- if has_range and range_start > range_end:- return dagster.SkipReason(- f"Invalid config: range_start ({range_start}) cannot be greater than range_end ({range_end})"- )-- # Build config object for run config generation- config = ReconciliationSchedulerConfig(- team_ids=team_ids if has_team_ids else None,- range_start=range_start if has_range else None,- range_end=range_end if has_range else None,- chunk_size=chunk_size,- max_concurrent_jobs=max_concurrent_jobs,- max_concurrent_tasks=max_concurrent_tasks,- bug_window_start=bug_window_start,- bug_window_end=bug_window_end,- dry_run=cursor_data.get("dry_run", False),- backup_enabled=cursor_data.get("backup_enabled", True),- batch_size=cursor_data.get("batch_size", 100),- teams_per_chunk=cursor_data.get("teams_per_chunk", 100),- team_ch_props_fetch_window_seconds=cursor_data.get("team_ch_props_fetch_window_seconds", 0),- person_batch_size=cursor_data.get("person_batch_size", 10000),- persons_db_env_var=cursor_data.get("persons_db_env_var", "PERSONS_DB_WRITER_URL"),- )-- # Count active runs for this job- active_runs = context.instance.get_run_records(- dagster.RunsFilter(- job_name="person_property_reconciliation_job",- statuses=[- dagster.DagsterRunStatus.QUEUED,- dagster.DagsterRunStatus.NOT_STARTED,- dagster.DagsterRunStatus.STARTING,- dagster.DagsterRunStatus.STARTED,- ],- )- )- active_count = len(active_runs)-- available_slots = max(0, max_concurrent_jobs - active_count)- if available_slots == 0:- return dagster.SkipReason(f"At max concurrency ({active_count}/{max_concurrent_jobs} jobs), waiting for slots")-- context.log.info(f"Active runs: {active_count}/{max_concurrent_jobs}, available slots: {available_slots}")-- # Generate run requests based on mode- run_requests: list[dagster.RunRequest] = []-- if has_team_ids:- # Team IDs mode: chunk the list- next_team_index = cursor_data.get("next_team_index", 0)-- if next_team_index >= len(team_ids):- return dagster.SkipReason(f"Reconciliation complete: processed all {len(team_ids)} teams")-- while len(run_requests) < available_slots and next_team_index < len(team_ids):- chunk_end_index = min(next_team_index + chunk_size, len(team_ids))- chunk_team_ids = team_ids[next_team_index:chunk_end_index]-- run_config = build_reconciliation_run_config(- config=config,- team_ids=chunk_team_ids,- )-- context.log.info(- f"Scheduling run for team_ids[{next_team_index}:{chunk_end_index}] ({len(chunk_team_ids)} teams)"- )-- run_requests.append(- dagster.RunRequest(- run_config=run_config,- tags={- "reconciliation_team_ids_range": f"{next_team_index}-{chunk_end_index - 1}",- "reconciliation_team_count": str(len(chunk_team_ids)),- "owner": JobOwners.TEAM_INGESTION.value,- },- )- )-- next_team_index = chunk_end_index-- # Update cursor- cursor_data["next_team_index"] = next_team_index- new_cursor = json.dumps(cursor_data)- context.log.info(f"Scheduled {len(run_requests)} runs, next_team_index: {next_team_index}")-- else:- # Range mode: existing behavior- next_chunk_start = cursor_data.get("next_chunk_start", range_start)-- if next_chunk_start > range_end:- return dagster.SkipReason(f"Reconciliation complete: processed all teams up to {range_end}")-- current_start = next_chunk_start-- while len(run_requests) < available_slots and current_start <= range_end:- chunk_end = min(current_start + chunk_size - 1, range_end)-- run_config = build_reconciliation_run_config(- config=config,- min_team_id=current_start,- max_team_id=chunk_end,- )-- context.log.info(f"Scheduling run for teams {current_start}-{chunk_end}")-- run_requests.append(- dagster.RunRequest(- run_config=run_config,- tags={- "reconciliation_range": f"{current_start}-{chunk_end}",- "owner": JobOwners.TEAM_INGESTION.value,- },- )- )-- current_start = chunk_end + 1-- # Update cursor- cursor_data["next_chunk_start"] = current_start- new_cursor = json.dumps(cursor_data)- context.log.info(f"Scheduled {len(run_requests)} runs, next_chunk_start: {current_start}")-- return dagster.SensorResult(run_requests=run_requests, cursor=new_cursor)diff --git a/posthog/dags/person_property_reconciliation_query_explanation.md b/posthog/dags/person_property_reconciliation_query_explanation.mddeleted file mode 100644index 035093cfd314..000000000000--- a/posthog/dags/person_property_reconciliation_query_explanation.md+++ /dev/null@@ -1,506 +0,0 @@-# Person Property Reconciliation Query Explanation--This document explains the ClickHouse query in `get_person_property_updates_from_clickhouse` (person_property_reconciliation.py:98-211).--The query finds person properties that need reconciliation by comparing event-derived properties with current person state.--## Query Structure Overview--The query has 5 nested levels:--1. **Core Event Extraction** - Extract properties from events, resolve merged persons, aggregate per property-2. **Group into Arrays** - Collect all properties per person into one array-3. **Split by Operation Type** - Separate `$set` and `$set_once` into parallel arrays-4. **Join with Person State** - Get current person properties from ClickHouse-5. **Compute Diff** - Find properties that differ or are missing-------## Level 1: Core Event Extraction with Person Resolution (Innermost)--```sql-SELECT- if(notEmpty(overrides.distinct_id), overrides.person_id, e.person_id) AS person_id,- kv_tuple.2 AS key,- kv_tuple.1 AS prop_type,- if(kv_tuple.1 = 'set',- argMaxIf(kv_tuple.3, e.timestamp, kv_tuple.3 IS NOT NULL AND kv_tuple.3 != ''),- argMinIf(kv_tuple.3, e.timestamp, kv_tuple.3 IS NOT NULL AND kv_tuple.3 != '')- ) AS value,- if(kv_tuple.1 = 'set', max(e.timestamp), min(e.timestamp)) AS kv_timestamp-FROM events e-LEFT OUTER JOIN (- SELECT- argMax(person_distinct_id_overrides.person_id, person_distinct_id_overrides.version) AS person_id,- person_distinct_id_overrides.distinct_id AS distinct_id- FROM person_distinct_id_overrides- WHERE equals(person_distinct_id_overrides.team_id, %(team_id)s)- GROUP BY person_distinct_id_overrides.distinct_id- HAVING ifNull(equals(argMax(person_distinct_id_overrides.is_deleted, person_distinct_id_overrides.version), 0), 0)-) AS overrides ON e.distinct_id = overrides.distinct_id-ARRAY JOIN- arrayConcat(- arrayFilter(x -> x.3 IS NOT NULL AND x.3 != '' AND x.3 != 'null',- arrayMap(x -> tuple('set', x.1, toString(x.2)),- arrayFilter(x -> x.2 IS NOT NULL, JSONExtractKeysAndValuesRaw(e.properties, '$set'))- )- ),- arrayFilter(x -> x.3 IS NOT NULL AND x.3 != '' AND x.3 != 'null',- arrayMap(x -> tuple('set_once', x.1, toString(x.2)),- arrayFilter(x -> x.2 IS NOT NULL, JSONExtractKeysAndValuesRaw(e.properties, '$set_once'))- )- )- ) AS kv_tuple-WHERE e.team_id = %(team_id)s- AND e.timestamp > %(bug_window_start)s- AND e.timestamp < %(bug_window_end)s-GROUP BY person_id, kv_tuple.2, kv_tuple.1-```--### The ARRAY JOIN--`ARRAY JOIN` "unrolls" an array into multiple rows. One input row becomes N output rows (one per array element).--The right side of the ARRAY JOIN builds an array of property tuples from each event:--```sql-arrayConcat(- -- $set properties- arrayFilter(x -> x.3 IS NOT NULL AND x.3 != '' AND x.3 != 'null',- arrayMap(x -> tuple('set', x.1, toString(x.2)),- arrayFilter(x -> x.2 IS NOT NULL, JSONExtractKeysAndValuesRaw(e.properties, '$set'))- )- ),- -- $set_once properties (same structure)- arrayFilter(x -> x.3 IS NOT NULL AND x.3 != '' AND x.3 != 'null',- arrayMap(x -> tuple('set_once', x.1, toString(x.2)),- arrayFilter(x -> x.2 IS NOT NULL, JSONExtractKeysAndValuesRaw(e.properties, '$set_once'))- )- )-) AS kv_tuple-```--Reading innermost to outermost (for the `$set` branch):--1. **`JSONExtractKeysAndValuesRaw(e.properties, '$set')`** - Extract `$set` object as `[(key, raw_json_value), ...]`-2. **`arrayFilter(x -> x.2 IS NOT NULL, ...)`** - Drop pairs where value is NULL-3. **`arrayMap(x -> tuple('set', x.1, toString(x.2)), ...)`** - Transform each `(key, value)` into `('set', key, value_string)`-4. **`arrayFilter(x -> x.3 IS NOT NULL AND x.3 != '' AND x.3 != 'null', ...)`** - Drop tuples where the string value is empty or literal `'null'`-5. **`arrayConcat(..., ...)`** - Combine `$set` and `$set_once` arrays into one--The resulting `kv_tuple` structure:--- `kv_tuple.1` = operation type (`'set'` or `'set_once'` - only these two values, hardcoded)-- `kv_tuple.2` = property key-- `kv_tuple.3` = property value (as JSON string)--Example: if an event has:--```json-{- "$set": { "email": "[email protected]", "name": null },- "$set_once": { "created": "2024-01-01" }-}-```--The result is:--```text-[('set', 'email', '"[email protected]"'), ('set_once', 'created', '"2024-01-01"')]-```--(The `name` property is filtered out because its value is null)--### The LEFT OUTER JOIN (Person Merge Resolution)--```sql-LEFT OUTER JOIN (- SELECT- argMax(person_distinct_id_overrides.person_id, person_distinct_id_overrides.version) AS person_id,- person_distinct_id_overrides.distinct_id AS distinct_id- FROM person_distinct_id_overrides- WHERE equals(person_distinct_id_overrides.team_id, %(team_id)s)- GROUP BY person_distinct_id_overrides.distinct_id- HAVING ifNull(equals(argMax(person_distinct_id_overrides.is_deleted, person_distinct_id_overrides.version), 0), 0)-) AS overrides ON e.distinct_id = overrides.distinct_id-```--This handles **person merges**. When two persons are merged, the `person_distinct_id_overrides` table records that a `distinct_id` now belongs to a different `person_id`.--The subquery:--1. **`GROUP BY distinct_id`** - One row per distinct_id-2. **`argMax(person_id, version)`** - Get the person_id from the row with the highest version (most recent assignment)-3. **`HAVING ifNull(equals(argMax(is_deleted, version), 0), 0)`** - Only include overrides where the latest version is not deleted--The HAVING clause breakdown:--- `argMax(is_deleted, version)` - Get `is_deleted` value from row with highest version-- `equals(..., 0)` - Returns `1` if not deleted, `0` if deleted, `NULL` if NULL-- `ifNull(..., 0)` - Convert NULL to 0 (falsy), so NULL values get filtered out-- Result: `1` (truthy) keeps the row, `0` (falsy) filters it out--The join:--- `LEFT OUTER JOIN` keeps all events, even without an override-- Then: `if(notEmpty(overrides.distinct_id), overrides.person_id, e.person_id)` uses the override if it exists--### The SELECT Columns--Grouped by `(person_id, property_key, operation_type)`, producing one row per person per property per operation:--1. **`person_id`** - Use override if exists, else event's person_id-2. **`key`** - Property key (`kv_tuple.2`)-3. **`prop_type`** - `'set'` or `'set_once'` (`kv_tuple.1`)-4. **`value`** - The winning value:- - For `$set`: `argMaxIf(value, timestamp, ...)` → value from the **latest** event- - For `$set_once`: `argMinIf(value, timestamp, ...)` → value from the **earliest** event- - The `If` suffix adds a condition: only consider rows where value is not null/empty-5. **`kv_timestamp`** - The timestamp of the winning event:- - For `$set`: `max(timestamp)` → latest- - For `$set_once`: `min(timestamp)` → earliest--This implements the semantics: `$set` overwrites (latest wins), `$set_once` only sets if not already set (first wins).-------## Level 2: Group Properties into Arrays per Person--```sql-SELECT- person_id,- groupArray(tuple(key, value, kv_timestamp, prop_type)) AS grouped_props-FROM (... level 1 query ...)-GROUP BY person_id-```--**Input:** Multiple rows per person (one for each property/operation combo)--| person_id | key | value | kv_timestamp | prop_type |-| --------- | ------- | ------------ | ------------ | --------- |-| abc | email | "[email protected]" | 2024-01-05 | set |-| abc | plan | "pro" | 2024-01-03 | set |-| abc | created | "2024-01-01" | 2024-01-01 | set_once |-| xyz | email | "[email protected]" | 2024-01-04 | set |--**Output:** One row per person, all properties in an array of tuples--| person_id | grouped_props |-| --------- | -------------------------------------------------------------------------------------------------------------------------------------- |-| abc | [('email', '"[email protected]"', 2024-01-05, 'set'), ('plan', '"pro"', 2024-01-03, 'set'), ('created', '"2024-01-01"', 2024-01-01, 'set_once')] |-| xyz | [('email', '"[email protected]"', 2024-01-04, 'set')] |--`groupArray()` is ClickHouse's aggregation function that collects all values in the group into an array.-------## Level 3: Split Arrays by Operation Type--```sql-SELECT- person_id,- arrayMap(x -> x.1, arrayFilter(x -> x.4 = 'set', grouped_props)) AS set_keys,- arrayMap(x -> x.2, arrayFilter(x -> x.4 = 'set', grouped_props)) AS set_values,- arrayMap(x -> x.3, arrayFilter(x -> x.4 = 'set', grouped_props)) AS set_timestamps,- arrayMap(x -> x.1, arrayFilter(x -> x.4 = 'set_once', grouped_props)) AS set_once_keys,- arrayMap(x -> x.2, arrayFilter(x -> x.4 = 'set_once', grouped_props)) AS set_once_values,- arrayMap(x -> x.3, arrayFilter(x -> x.4 = 'set_once', grouped_props)) AS set_once_timestamps-FROM (... level 2 query ...)-```--Separates `$set` and `$set_once` properties into parallel arrays (keys, values, timestamps) for easier comparison in the diff step.-------## Level 4: Join with Current Person Properties--```sql-SELECT ...-FROM (...) AS merged-INNER JOIN (- SELECT- id,- argMax(properties, version) as person_properties- FROM person- WHERE team_id = %(team_id)s- AND _timestamp > %(bug_window_start)s- AND _timestamp < %(bug_window_end)s- GROUP BY id-) AS p ON p.id = merged.person_id-```--Gets the current person properties from ClickHouse's `person` table (using `argMax` to get the latest version).-------## Level 5: Compute the Diff (Outermost)--```sql-SELECT- with_person_props.person_id,- -- For $set: only include properties where the key exists AND value differs- arrayMap(i -> (set_keys[i], set_values[i], set_timestamps[i]), arrayFilter(- i -> (- indexOf(keys2, set_keys[i]) > 0- AND set_values[i] != vals2[indexOf(keys2, set_keys[i])]- ),- arrayEnumerate(set_keys)- )) AS set_diff,- -- For $set_once: only include properties where the key does NOT exist- arrayFilter(- kv -> indexOf(keys2, kv.1) = 0,- arrayMap(i -> (set_once_keys[i], set_once_values[i], set_once_timestamps[i]), arrayEnumerate(set_once_keys))- ) AS set_once_diff-FROM (...) AS with_person_props-WHERE length(set_diff) > 0 OR length(set_once_diff) > 0-```--- **`set_diff`**: Properties from events where the value **differs** from current person properties-- **`set_once_diff`**: Properties from events that are **missing** from current person properties--Only returns persons where at least one property needs reconciliation.-------## Summary--The query flows like this:--1. **Extract** `$set`/`$set_once` from events, resolving merged persons-2. **Aggregate** to find the winning value per property (latest for `$set`, first for `$set_once`)-3. **Group** all properties per person into arrays-4. **Join** with current person state from ClickHouse-5. **Diff** to find properties that need updating (different values for `$set`, missing keys for `$set_once`)-------## Open Concerns (TODO: write tests)--### 1. Deleted persons filtered in person join (FIXED)--The query now filters out deleted persons using `HAVING argMax(is_deleted, version) = 0`:--```sql-INNER JOIN (- SELECT- id,- argMax(properties, version) as person_properties- FROM person- WHERE team_id = %(team_id)s- AND _timestamp > %(bug_window_start)s- AND _timestamp < %(bug_window_end)s- GROUP BY id- -- Filter out deleted persons (latest version has is_deleted=1)- HAVING argMax(is_deleted, version) = 0-) AS p ON p.id = merged.person_id-```--Test: `test_deleted_person_filtered_out`--### 2. Person state only from bug window - potential to overwrite newer updates--The person join only selects persons modified within the bug window (`_timestamp > bug_window_start AND _timestamp < bug_window_end`).--If a person received legitimate property updates **after** the bug window ended, will this reconciliation overwrite those newer updates with older values from the bug window?--The reconciliation logic in `reconcile_person_properties()` does compare timestamps via `properties_last_updated_at`, but need to verify this handles the edge case correctly.-------## $unset Support (Implemented)--The reconciliation job now supports `$unset` operations. Here's how it works:--### Query changes--`$unset` extraction in the ARRAY JOIN (note: `$unset` is an array of keys, not key-value pairs):--```sql--- In ARRAY JOIN arrayConcat:-arrayMap(x -> tuple('unset', x, ''),- JSONExtractArrayRaw(e.properties, '$unset')-)-```--Add `unset_*` arrays in Level 3:--```sql-arrayMap(x -> x.1, arrayFilter(x -> x.4 = 'unset', grouped_props)) AS unset_keys,-arrayMap(x -> x.3, arrayFilter(x -> x.4 = 'unset', grouped_props)) AS unset_timestamps-```--Add `unset_diff` in Level 5 (keys where unset happened and key still exists):--```sql-arrayFilter(- kv -> indexOf(keys2, kv.1) > 0,- arrayMap(i -> (unset_keys[i], unset_timestamps[i]), arrayEnumerate(unset_keys))-) AS unset_diff-```--### Python resolution logic--Operation semantics:--- `$set_once`: "If not set, set it" - operation-level check, not a permanent property attribute-- `$set`: "Set it to this value" - always overwrites-- `$unset`: "Remove it" - always removes--Simulate applying operations in timestamp order:--```python-def resolve_property_updates(set_diff, set_once_diff, unset_diff):- """- Simulate applying operations in timestamp order.-- - set_once: sets value only if property is currently unset- - set: always sets value- - unset: always removes property- """- results_by_key = {}-- # Collect all operations by key- ops_by_key = defaultdict(list)-- for key, value, ts in set_diff:- ops_by_key[key].append(('set', value, ts))-- for key, value, ts in set_once_diff:- ops_by_key[key].append(('set_once', value, ts))-- for key, ts in unset_diff:- ops_by_key[key].append(('unset', None, ts))-- for key, ops in ops_by_key.items():- # Sort by timestamp ascending- ops.sort(key=lambda x: x[2])-- # Simulate applying each operation- current_value = None # None means unset- current_ts = None-- for op_type, value, ts in ops:- if op_type == 'set':- current_value = value- current_ts = ts- elif op_type == 'set_once':- if current_value is None:- current_value = value- current_ts = ts- # else: ignore, property already has a value- elif op_type == 'unset':- current_value = None- current_ts = ts-- results_by_key[key] = (current_value, current_ts)-- # Split into updates and deletions- updates = {k: v for k, v in results_by_key.items() if v[0] is not None}- deletions = {k for k, v in results_by_key.items() if v[0] is None}-- return updates, deletions-```--Example timeline for key `email`:--```text-t1: set_once "[email protected]" → value = "[email protected]"-t2: set_once "[email protected]" → ignored (already set)-t3: set "[email protected]" → value = "[email protected]" (set overwrites)-t4: unset → value = None-t5: set_once "[email protected]" → value = "[email protected]" (was unset, so set_once applies)-```--### Potential inconsistency with $unset--**Problem:** If an `$unset` happened during the bug window, but a `$set` happened AFTER the bug window, this reconciliation would incorrectly delete the property.--Example:--```text-t1 (bug window): $unset "email" → resolved as deletion-t2 (after window): $set "email"="x" → not seen by reconciliation-```--The reconciliation would delete `email`, even though it was legitimately set after the bug window.--**Solution:** Compare resolved operation timestamp against `properties_last_updated_at` from Postgres before applying:--```python-def reconcile_person_properties(person, resolved_updates, resolved_deletions):- properties = dict(person["properties"] or {})- properties_last_updated_at = dict(person["properties_last_updated_at"] or {})-- # Apply updates (only if our timestamp is newer)- for key, (value, event_ts) in resolved_updates.items():- existing_ts = properties_last_updated_at.get(key)- if existing_ts is None or event_ts > parse_datetime(existing_ts):- properties[key] = value- properties_last_updated_at[key] = event_ts.isoformat()-- # Apply deletions (only if our timestamp is newer)- for key, event_ts in resolved_deletions.items():- existing_ts = properties_last_updated_at.get(key)- if existing_ts is None or event_ts > parse_datetime(existing_ts):- properties.pop(key, None)- properties_last_updated_at.pop(key, None)-- return properties, properties_last_updated_at-```--This ensures we only apply operations that are actually newer than what's currently in Postgres, preventing overwrites of legitimate post-bug-window updates.--### Critical issue: properties_last_updated_at is NOT maintained--**Discovery:** `properties_last_updated_at` is only set at person creation time (`person-create-service.ts:34-42`). It is NOT updated when properties change after creation.--This means:--- If a property is added after person creation, it has no timestamp in `properties_last_updated_at`-- The reconciliation logic checks `if existing_ts_str is None` and applies the update-- This causes post-bug-window updates to be incorrectly overwritten--**Test case:** `TestPostBugWindowUpdatePreservation::test_post_bug_window_update_should_not_be_overwritten`--Timeline:--```text-t1: Person created WITHOUT property P-t2: Bug window starts-t2.5: Event sets P=V1 (missed due to bug)-t3: Bug window ends-t3.5: Event sets P=V2 (correctly applied)-t4: Reconciliation runs-```--Current state in Postgres:--- `properties["P"] = "V2"` (from t3.5)-- `properties_last_updated_at["P"] = undefined` (not set because P was added after creation)--Reconciliation logic:--```python-existing_ts_str = properties_last_updated_at.get("P") # None!-if existing_ts_str is None or event_ts > parse_datetime(existing_ts_str):- properties["P"] = "V1" # BUG: overwrites V2 with older V1-```--**This test currently FAILS** - documenting the bug that needs to be fixed.--**Integration tests confirming this behavior:**-`nodejs/src/ingestion/person-properties-metadata.e2e.test.ts`--These tests verify that:--For `properties_last_updated_at`:--1. IS set at person creation (via `$set` or `$identify`)-2. Is NOT updated when an existing property is changed-3. Does NOT get a new entry when a new property is added after creation--For `properties_last_operation`:--1. IS set at person creation with correct operation type (`set` or `set_once`)-2. Is NOT updated when properties change after creation-3. Does NOT get a new entry when a new property is added after creationdiff --git a/posthog/dags/scripts/extract_reconciliation_results.py b/posthog/dags/scripts/extract_reconciliation_results.pydeleted file mode 100644index 8a3f8f942f80..000000000000--- a/posthog/dags/scripts/extract_reconciliation_results.py+++ /dev/null@@ -1,648 +0,0 @@-#!/usr/bin/env python3-# ruff: noqa: T201 - CLI script uses print for output-"""-Extract failed team IDs from person property reconciliation job runs via Dagster Cloud GraphQL API.--Usage:- export DAGSTER_CLOUD_TOKEN="<USER_TOKEN>"- python posthog/dags/scripts/extract_reconciliation_results.py \- --org posthog \- --deployment <prod-us | prod-eu> \- [--all | --limit <INT> | --since "2026-01-01" --until "2026-01-12"]--Prerequisites:- 1. Get a Dagster Cloud user token:- - Go to Dagster Cloud UI (e.g., https://posthog.dagster.cloud/)- - Click your avatar (top-right) > "Organization settings" > "Tokens" tab- - Create a User Token and copy it- - REMEMBER: revoke the token when you're done!- 2. Set the DAGSTER_CLOUD_TOKEN environment variable-"""--import os-import ast-import csv-import sys-import json-import argparse-from dataclasses import dataclass-from datetime import datetime-from typing import Any--import requests--# GraphQL query to get runs for the reconciliation job-RUNS_QUERY = """-query GetRuns($limit: Int!, $cursor: String, $filter: RunsFilter!) {- runsOrError(filter: $filter, limit: $limit, cursor: $cursor) {- __typename- ... on Runs {- results {- runId- status- startTime- endTime- tags {- key- value- }- }- }- ... on InvalidPipelineRunsFilterError {- message- }- ... on PythonError {- message- stack- }- }-}-"""--# GraphQL query to get a single run by ID-RUN_QUERY = """-query GetRun($runId: ID!) {- runOrError(runId: $runId) {- __typename- ... on Run {- runId- status- startTime- endTime- tags {- key- value- }- }- ... on RunNotFoundError {- message- }- ... on PythonError {- message- stack- }- }-}-"""--# GraphQL query to get step output events for a run-STEP_OUTPUTS_QUERY = """-query GetStepOutputs($runId: ID!) {- logsForRun(runId: $runId) {- __typename- ... on EventConnection {- events {- __typename- ... on ExecutionStepOutputEvent {- stepKey- metadataEntries {- __typename- label- ... on TextMetadataEntry {- text- }- ... on JsonMetadataEntry {- jsonString- }- ... on IntMetadataEntry {- intValue- }- }- }- }- }- ... on PythonError {- message- stack- }- }-}-"""---@dataclass-class TeamResult:- """Result for a single team from a reconciliation run."""-- team_id: int- status: str- persons_processed: int- persons_updated: int- persons_skipped: int- error: str | None- run_id: str- step_key: str---@dataclass-class RunSummary:- """Summary of a single reconciliation job run."""-- run_id: str- status: str- start_time: datetime | None- end_time: datetime | None- team_range: str | None- teams_count: int- teams_succeeded: int- teams_failed: int- failed_team_ids: list[int]- team_results: list[TeamResult]- persons_processed: int- persons_updated: int- persons_skipped: int---class DagsterCloudClient:- """Client for Dagster Cloud GraphQL API."""-- def __init__(self, org: str, deployment: str, token: str):- self.base_url = f"https://{org}.dagster.cloud/{deployment}/graphql"- self.headers = {- "Dagster-Cloud-Api-Token": token,- "Content-Type": "application/json",- }-- def _execute_query(self, query: str, variables: dict[str, Any] | None = None) -> dict[str, Any]:- """Execute a GraphQL query and return the result."""- payload: dict[str, Any] = {"query": query}- if variables:- payload["variables"] = variables-- response = requests.post(self.base_url, headers=self.headers, json=payload, timeout=60)- response.raise_for_status()-- result = response.json()- if "errors" in result:- raise RuntimeError(f"GraphQL errors: {result['errors']}")-- return result["data"]-- def get_runs(- self,- job_name: str = "person_property_reconciliation_job",- statuses: list[str] | None = None,- created_after: float | None = None,- created_before: float | None = None,- limit: int = 100,- cursor: str | None = None,- ) -> tuple[list[dict[str, Any]], str | None]:- """- Get runs for the specified job.-- Args:- job_name: Name of the Dagster job- statuses: Filter by run statuses- created_after: Unix timestamp - only return runs created after this time- created_before: Unix timestamp - only return runs created before this time- limit: Max runs to return per page- cursor: Pagination cursor-- Returns:- Tuple of (runs list, next cursor for pagination)- """- run_filter: dict[str, Any] = {"pipelineName": job_name}- if statuses:- run_filter["statuses"] = statuses- if created_after is not None:- # Dagster API uses "updatedAfter" (no "createdAfter" exists)- run_filter["updatedAfter"] = created_after- if created_before is not None:- run_filter["createdBefore"] = created_before-- variables = {- "filter": run_filter,- "limit": limit,- "cursor": cursor,- }-- data = self._execute_query(RUNS_QUERY, variables)- runs_result = data["runsOrError"]-- if runs_result["__typename"] == "Runs":- results = runs_result["results"]- # Get cursor for next page (last run's runId if we got a full page)- next_cursor = results[-1]["runId"] if len(results) == limit else None- return results, next_cursor- elif runs_result["__typename"] == "InvalidPipelineRunsFilterError":- raise RuntimeError(f"Invalid filter: {runs_result['message']}")- else:- raise RuntimeError(f"Query error: {runs_result.get('message', 'Unknown error')}")-- def get_run(self, run_id: str) -> dict[str, Any] | None:- """Get a single run by ID. Returns None if not found."""- data = self._execute_query(RUN_QUERY, {"runId": run_id})- result = data["runOrError"]-- if result["__typename"] == "Run":- return result- elif result["__typename"] == "RunNotFoundError":- return None- else:- raise RuntimeError(f"Query error: {result.get('message', 'Unknown error')}")-- def get_step_outputs(self, run_id: str) -> list[dict[str, Any]]:- """Get step output events for a run, filtered to reconcile_team_chunk steps."""- data = self._execute_query(STEP_OUTPUTS_QUERY, {"runId": run_id})- logs_result = data["logsForRun"]-- if logs_result["__typename"] == "PythonError":- raise RuntimeError(f"Query error: {logs_result.get('message', 'Unknown error')}")-- events = logs_result.get("events", [])-- # Filter to only ExecutionStepOutputEvent for reconcile_team_chunk steps- step_outputs = []- for event in events:- if event["__typename"] == "ExecutionStepOutputEvent":- step_key = event.get("stepKey", "")- # Match reconcile_team_chunk steps (they have dynamic suffixes like [teams_1_100])- if "reconcile_team_chunk" in step_key:- step_outputs.append(event)-- return step_outputs---def parse_metadata_entries(entries: list[dict[str, Any]]) -> dict[str, Any]:- """Parse metadata entries into a dictionary."""- result: dict[str, Any] = {}- for entry in entries:- label = entry.get("label", "")- typename = entry.get("__typename", "")-- if typename == "TextMetadataEntry":- result[label] = entry.get("text", "")- elif typename == "JsonMetadataEntry":- json_str = entry.get("jsonString", "")- try:- result[label] = json.loads(json_str) if json_str else None- except json.JSONDecodeError:- result[label] = json_str- elif typename == "IntMetadataEntry":- result[label] = entry.get("intValue")-- return result---def parse_failed_team_ids(failed_team_ids_str: str) -> list[int]:- """Parse the failed_team_ids string (Python list repr) into a list of ints."""- if not failed_team_ids_str or failed_team_ids_str == "[]":- return []-- try:- # The string is a Python list repr like "[123, 456, 789]"- parsed = ast.literal_eval(failed_team_ids_str)- if isinstance(parsed, list):- return [int(x) for x in parsed]- except (ValueError, SyntaxError):- pass-- return []---def extract_run_results(client: DagsterCloudClient, run: dict[str, Any]) -> RunSummary:- """Extract results from a single run."""- run_id = run["runId"]- status = run["status"]-- # Parse timestamps- start_time = None- end_time = None- if run.get("startTime"):- start_time = datetime.fromtimestamp(run["startTime"])- if run.get("endTime"):- end_time = datetime.fromtimestamp(run["endTime"])-- # Extract team range from tags- team_range = None- for tag in run.get("tags", []):- if tag["key"] == "reconciliation_range":- team_range = tag["value"]- break-- # Get step outputs- step_outputs = client.get_step_outputs(run_id)-- all_failed_team_ids: list[int] = []- all_team_results: list[TeamResult] = []- total_teams_count = 0- total_teams_succeeded = 0- total_teams_failed = 0- total_persons_processed = 0- total_persons_updated = 0- total_persons_skipped = 0-- for step_output in step_outputs:- step_key = step_output.get("stepKey", "")- metadata = parse_metadata_entries(step_output.get("metadataEntries", []))-- # Aggregate counts- total_teams_count += metadata.get("teams_count", 0) or 0- total_teams_succeeded += metadata.get("teams_succeeded", 0) or 0- total_teams_failed += metadata.get("teams_failed", 0) or 0-- # Parse failed team IDs- failed_ids_str = metadata.get("failed_team_ids", "[]")- failed_ids = parse_failed_team_ids(failed_ids_str)- all_failed_team_ids.extend(failed_ids)-- # Parse detailed team results- teams_results = metadata.get("teams_results", [])- if isinstance(teams_results, list):- for team_result in teams_results:- if isinstance(team_result, dict):- persons_processed = team_result.get("persons_processed", 0) or 0- persons_updated = team_result.get("persons_updated", 0) or 0- persons_skipped = team_result.get("persons_skipped", 0) or 0-- total_persons_processed += persons_processed- total_persons_updated += persons_updated- total_persons_skipped += persons_skipped-- all_team_results.append(- TeamResult(- team_id=team_result.get("team_id", 0),- status=team_result.get("status", "unknown"),- persons_processed=persons_processed,- persons_updated=persons_updated,- persons_skipped=persons_skipped,- error=team_result.get("error"),- run_id=run_id,- step_key=step_key,- )- )-- return RunSummary(- run_id=run_id,- status=status,- start_time=start_time,- end_time=end_time,- team_range=team_range,- teams_count=total_teams_count,- teams_succeeded=total_teams_succeeded,- teams_failed=total_teams_failed,- failed_team_ids=sorted(set(all_failed_team_ids)),- team_results=all_team_results,- persons_processed=total_persons_processed,- persons_updated=total_persons_updated,- persons_skipped=total_persons_skipped,- )---def main():- parser = argparse.ArgumentParser(- description="Extract failed team IDs from person property reconciliation job runs",- formatter_class=argparse.RawDescriptionHelpFormatter,- epilog=__doc__,- )- parser.add_argument("--org", required=True, help="Dagster Cloud organization name (e.g., 'posthog')")- parser.add_argument("--deployment", required=True, help="Dagster Cloud deployment name (e.g., 'prod')")- parser.add_argument(- "--token",- default=os.environ.get("DAGSTER_CLOUD_TOKEN"),- help="Dagster Cloud API token (defaults to DAGSTER_CLOUD_TOKEN env var)",- )- parser.add_argument(- "--status",- nargs="*",- choices=["QUEUED", "NOT_STARTED", "STARTING", "STARTED", "SUCCESS", "FAILURE", "CANCELING", "CANCELED"],- help="Filter runs by status (default: all statuses)",- )- parser.add_argument(- "--limit",- type=int,- default=100,- help="Maximum number of runs to fetch (default: 100, use --all for unlimited)",- )- parser.add_argument(- "--all",- action="store_true",- dest="fetch_all",- help="Fetch ALL runs (ignores --limit)",- )- parser.add_argument(- "--since",- help="Only include runs created after this date/time in local timezone (format: YYYY-MM-DD or YYYY-MM-DD HH:MM:SS)",- )- parser.add_argument(- "--until",- help="Only include runs created before this date/time in local timezone (format: YYYY-MM-DD or YYYY-MM-DD HH:MM:SS)",- )- parser.add_argument(- "--run-id",- help="Extract results from a specific run ID only",- )- parser.add_argument(- "--output-csv",- help="Output failed teams to a CSV file",- )- parser.add_argument(- "--verbose",- "-v",- action="store_true",- help="Show detailed output including per-team results",- )-- args = parser.parse_args()-- if not args.token:- print("Error: DAGSTER_CLOUD_TOKEN environment variable not set and --token not provided", file=sys.stderr)- print("\nTo get a token:", file=sys.stderr)- print(" 1. Go to Dagster Cloud UI (e.g., https://posthog.dagster.cloud/)", file=sys.stderr)- print(" 2. Click your avatar (top-right) > 'Organization settings' > 'Tokens' tab", file=sys.stderr)- print(" 3. Create and copy the token", file=sys.stderr)- sys.exit(1)-- client = DagsterCloudClient(args.org, args.deployment, args.token)-- def parse_date_arg(value: str, arg_name: str, end_of_day: bool = False) -> float:- """Parse a date string to unix timestamp. Assumes input is in local timezone."""- for fmt in ["%Y-%m-%d %H:%M:%S", "%Y-%m-%d"]:- try:- dt = datetime.strptime(value, fmt)- if end_of_day and fmt == "%Y-%m-%d":- dt = dt.replace(hour=23, minute=59, second=59)- return dt.timestamp()- except ValueError:- continue- print(f"Error: Invalid {arg_name} date format: {value}", file=sys.stderr)- print("Use ISO format: YYYY-MM-DD or YYYY-MM-DD HH:MM:SS", file=sys.stderr)- sys.exit(1)-- created_after = parse_date_arg(args.since, "--since") if args.since else None- created_before = parse_date_arg(args.until, "--until", end_of_day=True) if args.until else None-- all_summaries: list[RunSummary] = []- all_team_results_by_id: dict[int, TeamResult] = {}-- if args.run_id:- print(f"Fetching results for run {args.run_id}...")- run = client.get_run(args.run_id)- if not run:- print(f"Error: Run {args.run_id} not found", file=sys.stderr)- sys.exit(1)- runs_to_process = [run]- else:- # Fetch runs with pagination- filter_info = []- if args.since:- filter_info.append(f"since {args.since}")- if args.until:- filter_info.append(f"until {args.until}")- if args.status:- filter_info.append(f"status={args.status}")- filter_str = f" ({', '.join(filter_info)})" if filter_info else ""-- limit_str = "all" if args.fetch_all else str(args.limit)- print(f"Fetching {limit_str} runs for person_property_reconciliation_job{filter_str}...")-- runs_to_process = []- cursor = None- max_runs = float("inf") if args.fetch_all else args.limit-- while len(runs_to_process) < max_runs:- batch_limit = 100 if args.fetch_all else min(100, int(max_runs) - len(runs_to_process))- runs, cursor = client.get_runs(- statuses=args.status,- created_after=created_after,- created_before=created_before,- limit=batch_limit,- cursor=cursor,- )- runs_to_process.extend(runs)- print(f" Fetched {len(runs_to_process)} runs so far...")- if not cursor or len(runs) == 0:- break-- print(f"Found {len(runs_to_process)} runs total")-- # Sort runs by start time (newest first) so we process most recent results first- runs_to_process.sort(key=lambda r: r.get("startTime") or 0, reverse=True)-- # Process each run- for i, run in enumerate(runs_to_process, 1):- run_id = run["runId"]- print(f"Processing run {i}/{len(runs_to_process)}: {run_id} ({run['status']})...")-- try:- summary = extract_run_results(client, run)- all_summaries.append(summary)-- # Collect team results, keeping only the first (most recent) result per team- for result in summary.team_results:- if result.team_id not in all_team_results_by_id:- all_team_results_by_id[result.team_id] = result-- except Exception as e:- print(f" Error processing run {run_id}: {e}", file=sys.stderr)-- # Compute deduplicated aggregates (most recent result per team)- deduped_results = list(all_team_results_by_id.values())- deduped_teams_count = len(deduped_results)- deduped_teams_succeeded = sum(1 for r in deduped_results if r.status == "success")- deduped_teams_failed = sum(1 for r in deduped_results if r.status == "failed")- deduped_persons_processed = sum(r.persons_processed for r in deduped_results)- deduped_persons_updated = sum(r.persons_updated for r in deduped_results)- deduped_persons_skipped = sum(r.persons_skipped for r in deduped_results)- deduped_failed_team_ids = sorted(r.team_id for r in deduped_results if r.status == "failed")-- # Compute raw totals (includes retries)- raw_teams_count = sum(s.teams_count for s in all_summaries)-- # Print summary- print("\n" + "=" * 80)- print("SUMMARY (deduplicated by team - most recent run per team)")- print("=" * 80)-- print(f"Unique teams processed: {deduped_teams_count}")- print(f"Teams succeeded: {deduped_teams_succeeded}")- print(f"Teams failed: {deduped_teams_failed}")- print(f"Persons processed: {deduped_persons_processed}")- print(f"Persons updated: {deduped_persons_updated}")- print(f"Persons skipped: {deduped_persons_skipped}")-- if deduped_failed_team_ids:- print(f"\nFailed team IDs: {deduped_failed_team_ids}")-- print("\n" + "-" * 80)- print("RAW TOTALS (all runs, includes retries)")- print("-" * 80)- print(f"Runs analyzed: {len(all_summaries)}")- print(f"Raw team count: {raw_teams_count}")-- # Per-run breakdown- if args.verbose and all_summaries:- print("\n" + "-" * 80)- print("PER-RUN BREAKDOWN")- print("-" * 80)-- for summary in all_summaries:- start_str = summary.start_time.strftime("%Y-%m-%d %H:%M:%S") if summary.start_time else "N/A"- print(f"\nRun: {summary.run_id}")- print(f" Status: {summary.status}")- print(f" Started: {start_str}")- print(f" Team range: {summary.team_range or 'N/A'}")- print(- f" Teams: {summary.teams_count} total, {summary.teams_succeeded} succeeded, {summary.teams_failed} failed"- )- print(- f" Persons: {summary.persons_processed} processed, {summary.persons_updated} updated, {summary.persons_skipped} skipped"- )-- if summary.failed_team_ids:- print(f" Failed team IDs: {summary.failed_team_ids}")-- # Show error details for failed teams (from deduplicated results)- failed_team_results = [r for r in deduped_results if r.status == "failed"]- if args.verbose and failed_team_results:- print("\n" + "-" * 80)- print("FAILED TEAM DETAILS")- print("-" * 80)-- for result in failed_team_results:- print(f"\nTeam {result.team_id}:")- print(f" Run: {result.run_id}")- print(f" Step: {result.step_key}")- print(f" Error: {result.error or 'No error message'}")-- # Export to CSV if requested (uses deduplicated results)- if args.output_csv and failed_team_results:- print(f"\nWriting failed teams to {args.output_csv}...")- with open(args.output_csv, "w", newline="") as csvfile:- writer = csv.DictWriter(- csvfile,- fieldnames=[- "team_id",- "status",- "error",- "run_id",- "step_key",- "persons_processed",- "persons_updated",- "persons_skipped",- ],- )- writer.writeheader()- for result in failed_team_results:- writer.writerow(- {- "team_id": result.team_id,- "status": result.status,- "error": result.error or "",- "run_id": result.run_id,- "step_key": result.step_key,- "persons_processed": result.persons_processed,- "persons_updated": result.persons_updated,- "persons_skipped": result.persons_skipped,- }- )- print(f"Wrote {len(failed_team_results)} failed team records to {args.output_csv}")-- # Exit with error code if there were failures- if deduped_failed_team_ids:- sys.exit(1)- sys.exit(0)---if __name__ == "__main__":- main()diff --git a/posthog/dags/tests/test_fix_person_id_overrides.py b/posthog/dags/tests/test_fix_person_id_overrides.pydeleted file mode 100644index 5101c3fb94c0..000000000000--- a/posthog/dags/tests/test_fix_person_id_overrides.py+++ /dev/null@@ -1,318 +0,0 @@-from datetime import datetime-from uuid import UUID--from posthog.test.base import BaseTest, ClickhouseTestMixin--from posthog.clickhouse.client import sync_execute-from posthog.dags.fix_person_id_overrides import (- fix_person_id_overrides_job,- get_all_distinct_ids_for_person,- get_existing_override,- get_person_id_from_pdi2,- insert_override_batch,-)---def insert_pdi2_records(records: list[tuple[int, str, UUID, int, int]]) -> None:- """Insert records into person_distinct_id2: (team_id, distinct_id, person_id, version, is_deleted)"""- sync_execute(- "INSERT INTO person_distinct_id2 (team_id, distinct_id, person_id, version, is_deleted) VALUES",- records,- )---def insert_override_records(records: list[tuple[int, str, UUID, int, int]]) -> None:- """Insert records into person_distinct_id_overrides: (team_id, distinct_id, person_id, version, is_deleted)"""- sync_execute(- "INSERT INTO person_distinct_id_overrides (team_id, distinct_id, person_id, version, is_deleted, _timestamp, _offset, _partition) VALUES",- [(r[0], r[1], r[2], r[3], r[4], datetime(2025, 1, 1), 0, 0) for r in records],- )---def get_all_overrides(team_id: int) -> list[tuple[str, str]]:- """Get all overrides for a team: (distinct_id, person_id)"""- result = sync_execute(- """- SELECT distinct_id, argMax(person_id, version) as person_id- FROM person_distinct_id_overrides- WHERE team_id = %(team_id)s- GROUP BY distinct_id- HAVING argMax(is_deleted, version) = 0- """,- {"team_id": team_id},- )- return [(row[0], str(row[1])) for row in result]---class TestGetPersonIdFromPdi2(ClickhouseTestMixin, BaseTest):- def test_returns_person_id_and_version(self):- person_id = UUID(int=1)- insert_pdi2_records([(self.team.id, "user_123", person_id, 1, 0)])-- result = get_person_id_from_pdi2(self.team.id, "user_123")-- assert result is not None- assert result[0] == str(person_id)- assert result[1] == 1-- def test_returns_none_for_missing_distinct_id(self):- result = get_person_id_from_pdi2(self.team.id, "nonexistent")- assert result is None-- def test_returns_none_for_deleted_mapping(self):- person_id = UUID(int=2)- insert_pdi2_records(- [- (self.team.id, "deleted_user", person_id, 1, 0),- (self.team.id, "deleted_user", person_id, 2, 1), # deleted- ]- )-- result = get_person_id_from_pdi2(self.team.id, "deleted_user")- assert result is None-- def test_returns_latest_person_id_by_version(self):- old_person_id = UUID(int=10)- new_person_id = UUID(int=20)- insert_pdi2_records(- [- (self.team.id, "merged_user", old_person_id, 1, 0),- (self.team.id, "merged_user", new_person_id, 2, 0), # newer version- ]- )-- result = get_person_id_from_pdi2(self.team.id, "merged_user")-- assert result is not None- assert result[0] == str(new_person_id)- assert result[1] == 2---class TestGetAllDistinctIdsForPerson(ClickhouseTestMixin, BaseTest):- def test_returns_all_distinct_ids_for_person(self):- person_id = UUID(int=100)- insert_pdi2_records(- [- (self.team.id, "anon_id", person_id, 1, 0),- (self.team.id, "[email protected]", person_id, 1, 0),- (self.team.id, "phone_123", person_id, 1, 0),- ]- )-- result = get_all_distinct_ids_for_person(self.team.id, str(person_id))-- distinct_ids = {did for did, _ in result}- assert distinct_ids == {"anon_id", "[email protected]", "phone_123"}-- def test_excludes_deleted_mappings(self):- person_id = UUID(int=101)- insert_pdi2_records(- [- (self.team.id, "active_id", person_id, 1, 0),- (self.team.id, "deleted_id", person_id, 1, 0),- (self.team.id, "deleted_id", person_id, 2, 1), # deleted- ]- )-- result = get_all_distinct_ids_for_person(self.team.id, str(person_id))-- distinct_ids = {did for did, _ in result}- assert distinct_ids == {"active_id"}-- def test_returns_empty_for_nonexistent_person(self):- result = get_all_distinct_ids_for_person(self.team.id, str(UUID(int=999)))- assert result == []---class TestGetExistingOverride(ClickhouseTestMixin, BaseTest):- def test_returns_override_when_exists(self):- person_id = UUID(int=200)- insert_override_records([(self.team.id, "override_user", person_id, 1, 0)])-- result = get_existing_override(self.team.id, "override_user")-- assert result is not None- assert result[0] == str(person_id)- assert result[1] == 1-- def test_returns_none_when_no_override(self):- result = get_existing_override(self.team.id, "no_override_user")- assert result is None-- def test_returns_none_for_deleted_override(self):- person_id = UUID(int=201)- insert_override_records(- [- (self.team.id, "deleted_override", person_id, 1, 0),- (self.team.id, "deleted_override", person_id, 2, 1), # deleted- ]- )-- result = get_existing_override(self.team.id, "deleted_override")- assert result is None---class TestInsertOverrideBatch(ClickhouseTestMixin, BaseTest):- def test_inserts_batch_of_overrides(self):- person_id_1 = UUID(int=300)- person_id_2 = UUID(int=301)-- insert_override_batch(- [- (self.team.id, "batch_user_1", str(person_id_1), 1),- (self.team.id, "batch_user_2", str(person_id_1), 1),- (self.team.id, "batch_user_3", str(person_id_2), 2),- ]- )-- result = get_existing_override(self.team.id, "batch_user_1")- assert result is not None- assert result[0] == str(person_id_1)-- result = get_existing_override(self.team.id, "batch_user_2")- assert result is not None- assert result[0] == str(person_id_1)-- result = get_existing_override(self.team.id, "batch_user_3")- assert result is not None- assert result[0] == str(person_id_2)-- def test_handles_empty_batch(self):- insert_override_batch([])---class TestFixPersonIdOverridesJob(ClickhouseTestMixin, BaseTest):- def test_inserts_overrides_for_all_distinct_ids_of_person(self):- person_id = UUID(int=400)- insert_pdi2_records(- [- (self.team.id, "anon_abc123", person_id, 1, 0),- (self.team.id, "[email protected]", person_id, 1, 0),- ]- )-- fix_person_id_overrides_job.execute_in_process(- run_config={- "ops": {- "fix_person_id_overrides_op": {- "config": {- "team_id": self.team.id,- "distinct_ids": "anon_abc123",- "dry_run": False,- }- }- }- }- )-- overrides = get_all_overrides(self.team.id)- override_dict = dict(overrides)-- assert "anon_abc123" in override_dict- assert "[email protected]" in override_dict- assert override_dict["anon_abc123"] == str(person_id)- assert override_dict["[email protected]"] == str(person_id)-- def test_dry_run_does_not_insert(self):- person_id = UUID(int=401)- insert_pdi2_records([(self.team.id, "dry_run_user", person_id, 1, 0)])-- fix_person_id_overrides_job.execute_in_process(- run_config={- "ops": {- "fix_person_id_overrides_op": {- "config": {- "team_id": self.team.id,- "distinct_ids": "dry_run_user",- "dry_run": True,- }- }- }- }- )-- overrides = get_all_overrides(self.team.id)- assert len(overrides) == 0-- def test_skips_existing_overrides(self):- person_id = UUID(int=402)- existing_person_id = UUID(int=403)- insert_pdi2_records(- [- (self.team.id, "existing_override_user", person_id, 1, 0),- (self.team.id, "new_user", person_id, 1, 0),- ]- )- insert_override_records([(self.team.id, "existing_override_user", existing_person_id, 1, 0)])-- fix_person_id_overrides_job.execute_in_process(- run_config={- "ops": {- "fix_person_id_overrides_op": {- "config": {- "team_id": self.team.id,- "distinct_ids": "existing_override_user",- "dry_run": False,- }- }- }- }- )-- overrides = get_all_overrides(self.team.id)- override_dict = dict(overrides)-- # Existing override should not be changed- assert override_dict["existing_override_user"] == str(existing_person_id)- # New user should have override inserted- assert override_dict["new_user"] == str(person_id)-- def test_handles_multiple_distinct_ids_mapping_to_same_person(self):- person_id = UUID(int=404)- insert_pdi2_records(- [- (self.team.id, "id_a", person_id, 1, 0),- (self.team.id, "id_b", person_id, 1, 0),- (self.team.id, "id_c", person_id, 1, 0),- ]- )-- # Pass multiple distinct_ids that map to the same person- fix_person_id_overrides_job.execute_in_process(- run_config={- "ops": {- "fix_person_id_overrides_op": {- "config": {- "team_id": self.team.id,- "distinct_ids": "id_a,id_b",- "dry_run": False,- }- }- }- }- )-- overrides = get_all_overrides(self.team.id)- override_dict = dict(overrides)-- # All three should have overrides (even id_c which wasn't in the input)- assert len(override_dict) == 3- assert all(v == str(person_id) for v in override_dict.values())-- def test_handles_nonexistent_distinct_id(self):- # Should complete without error- result = fix_person_id_overrides_job.execute_in_process(- run_config={- "ops": {- "fix_person_id_overrides_op": {- "config": {- "team_id": self.team.id,- "distinct_ids": "nonexistent_id",- "dry_run": False,- }- }- }- }- )-- assert result.successdiff --git a/posthog/dags/tests/test_person_property_reconciliation.py b/posthog/dags/tests/test_person_property_reconciliation.pydeleted file mode 100644index 732c6074b5d8..000000000000--- a/posthog/dags/tests/test_person_property_reconciliation.py+++ /dev/null@@ -1,8236 +0,0 @@-"""Tests for the person property reconciliation job.--For comprehensive documentation including architecture diagrams and example configs, see:- posthog/dags/PERSON_PROPERTY_RECONCILIATION.md-"""--import os-import json-from datetime import UTC, datetime, timedelta-from typing import Any-from uuid import UUID--import pytest-from unittest.mock import MagicMock, patch--from clickhouse_driver import Client--from posthog.clickhouse.cluster import ClickhouseCluster-from posthog.dags.person_property_reconciliation import (- FILTERED_PERSON_UPDATE_PROPERTIES,- PersonPropertyDiffs,- PropertyValue,- RawPersonPropertyUpdates,- SkipReason,- compare_raw_updates_with_person_state,- fetch_person_properties_from_clickhouse,- filter_event_person_properties,- format_ch_timestamp,- get_affected_person_ids_from_clickhouse,- get_person_property_updates_from_clickhouse,- get_person_property_updates_windowed,- get_person_property_updates_windowed_batched,- get_raw_person_property_updates_from_clickhouse,- merge_raw_person_property_updates,- parse_ch_timestamp,- query_team_ids_from_clickhouse,- reconcile_person_properties,- reconcile_with_concurrent_changes,- update_person_with_version_check,-)-from posthog.dags.tests.conftest import isolated_clickhouse_cluster, refresh_person_from_persons_db---class TestClickHouseResultParsing:- """Test that ClickHouse query results are correctly parsed into PersonPropertyDiffs objects."""-- def test_parses_set_diff_tuples(self):- """Test that set_diff array of tuples is correctly parsed."""- # Simulate ClickHouse returning: (person_id, person_version, set_diff, set_once_diff, unset_diff)- # set_diff is an array of (key, value, timestamp) tuples- # Values are raw JSON strings that get parsed via json.loads()- mock_rows: list[- tuple[- str, int, list[tuple[str, Any, datetime]], list[tuple[str, Any, datetime]], list[tuple[str, datetime]]- ]- ] = [- (- "018d1234-5678-0000-0000-000000000001", # person_id (UUID as string)- 1, # person_version- [ # set_diff - array of (key, raw_json_value, timestamp) tuples- ("email", '"[email protected]"', datetime(2024, 1, 15, 12, 0, 0)),- ("name", '"John Doe"', datetime(2024, 1, 15, 12, 30, 0)),- ],- [], # set_once_diff - empty- [], # unset_diff - empty- )- ]-- with patch("posthog.dags.person_property_reconciliation.sync_execute", return_value=mock_rows):- results = get_person_property_updates_from_clickhouse(- team_id=1,- bug_window_start="2024-01-01T00:00:00Z",- )-- assert len(results) == 1- person_diffs = results[0]- assert person_diffs.person_id == "018d1234-5678-0000-0000-000000000001"- assert person_diffs.person_version == 1- assert len(person_diffs.set_updates) == 2- assert len(person_diffs.set_once_updates) == 0- assert len(person_diffs.unset_updates) == 0-- # Verify set updates - values are parsed from raw JSON to native types- assert "email" in person_diffs.set_updates- assert person_diffs.set_updates["email"].value == "[email protected]"- assert person_diffs.set_updates["email"].timestamp == datetime(2024, 1, 15, 12, 0, 0, tzinfo=UTC)-- assert "name" in person_diffs.set_updates- assert person_diffs.set_updates["name"].value == "John Doe"- assert person_diffs.set_updates["name"].timestamp == datetime(2024, 1, 15, 12, 30, 0, tzinfo=UTC)-- def test_parses_set_once_diff_tuples(self):- """Test that set_once_diff array of tuples is correctly parsed."""- # Values are raw JSON strings that get parsed via json.loads()- mock_rows: list[- tuple[- str, int, list[tuple[str, Any, datetime]], list[tuple[str, Any, datetime]], list[tuple[str, datetime]]- ]- ] = [- (- "018d1234-5678-0000-0000-000000000002",- 1, # person_version- [], # set_diff - empty- [ # set_once_diff - array of (key, raw_json_value, timestamp) tuples- ("initial_referrer", '"google.com"', datetime(2024, 1, 10, 8, 0, 0)),- ("first_seen", '"2024-01-10"', datetime(2024, 1, 10, 8, 0, 0)),- ],- [], # unset_diff - empty- )- ]-- with patch("posthog.dags.person_property_reconciliation.sync_execute", return_value=mock_rows):- results = get_person_property_updates_from_clickhouse(- team_id=1,- bug_window_start="2024-01-01T00:00:00Z",- )-- assert len(results) == 1- person_diffs = results[0]- assert len(person_diffs.set_updates) == 0- assert len(person_diffs.set_once_updates) == 2- assert len(person_diffs.unset_updates) == 0-- # Verify set_once updates - values are parsed from raw JSON to native types- assert "initial_referrer" in person_diffs.set_once_updates- assert person_diffs.set_once_updates["initial_referrer"].value == "google.com"- assert person_diffs.set_once_updates["initial_referrer"].timestamp == datetime(2024, 1, 10, 8, 0, 0, tzinfo=UTC)-- assert "first_seen" in person_diffs.set_once_updates- assert person_diffs.set_once_updates["first_seen"].value == "2024-01-10"- assert person_diffs.set_once_updates["first_seen"].timestamp == datetime(2024, 1, 10, 8, 0, 0, tzinfo=UTC)-- def test_parses_mixed_set_and_set_once(self):- """Test parsing when both set_diff and set_once_diff have values."""- # Values are raw JSON strings that get parsed via json.loads()- mock_rows: list[- tuple[- str, int, list[tuple[str, Any, datetime]], list[tuple[str, Any, datetime]], list[tuple[str, datetime]]- ]- ] = [- (- "018d1234-5678-0000-0000-000000000003",- 1, # person_version- [("email", '"[email protected]"', datetime(2024, 1, 15, 12, 0, 0))], # set_diff- [("initial_source", '"organic"', datetime(2024, 1, 10, 8, 0, 0))], # set_once_diff- [], # unset_diff - empty- )- ]-- with patch("posthog.dags.person_property_reconciliation.sync_execute", return_value=mock_rows):- results = get_person_property_updates_from_clickhouse(- team_id=1,- bug_window_start="2024-01-01T00:00:00Z",- )-- assert len(results) == 1- person_diffs = results[0]- assert len(person_diffs.set_updates) == 1- assert len(person_diffs.set_once_updates) == 1- assert len(person_diffs.unset_updates) == 0-- # Verify set update- assert "email" in person_diffs.set_updates- assert person_diffs.set_updates["email"].value == "[email protected]"-- # Verify set_once update- assert "initial_source" in person_diffs.set_once_updates- assert person_diffs.set_once_updates["initial_source"].value == "organic"-- def test_handles_multiple_persons(self):- """Test parsing results for multiple persons."""- # Values are raw JSON strings that get parsed via json.loads()- mock_rows: list[- tuple[- str, int, list[tuple[str, Any, datetime]], list[tuple[str, Any, datetime]], list[tuple[str, datetime]]- ]- ] = [- (- "018d1234-0000-0000-0000-000000000001",- 1, # person_version- [("prop1", '"val1"', datetime(2024, 1, 15, 12, 0, 0))],- [],- [], # unset_diff- ),- (- "018d1234-0000-0000-0000-000000000002",- 2, # person_version- [("prop2", '"val2"', datetime(2024, 1, 15, 13, 0, 0))],- [],- [], # unset_diff- ),- (- "018d1234-0000-0000-0000-000000000003",- 3, # person_version- [],- [("prop3", '"val3"', datetime(2024, 1, 10, 8, 0, 0))],- [], # unset_diff- ),- ]-- with patch("posthog.dags.person_property_reconciliation.sync_execute", return_value=mock_rows):- results = get_person_property_updates_from_clickhouse(- team_id=1,- bug_window_start="2024-01-01T00:00:00Z",- )-- assert len(results) == 3-- # First person has set update- assert results[0].person_id == "018d1234-0000-0000-0000-000000000001"- assert results[0].set_updates["prop1"].value == "val1"-- # Second person has set update- assert results[1].person_id == "018d1234-0000-0000-0000-000000000002"- assert results[1].set_updates["prop2"].value == "val2"-- # Third person has set_once update- assert results[2].person_id == "018d1234-0000-0000-0000-000000000003"- assert results[2].set_once_updates["prop3"].value == "val3"-- def test_skips_persons_with_no_updates(self):- """Test that persons with empty set_diff, set_once_diff, and unset_diff are skipped."""- mock_rows: list[- tuple[- str, int, list[tuple[str, Any, datetime]], list[tuple[str, Any, datetime]], list[tuple[str, datetime]]- ]- ] = [- (- "018d1234-0000-0000-0000-000000000001",- 1, # person_version- [], # empty set_diff- [], # empty set_once_diff- [], # empty unset_diff- ),- ]-- with patch("posthog.dags.person_property_reconciliation.sync_execute", return_value=mock_rows):- results = get_person_property_updates_from_clickhouse(- team_id=1,- bug_window_start="2024-01-01T00:00:00Z",- )-- # Person with no updates should be filtered out- assert len(results) == 0-- def test_handles_empty_results(self):- """Test handling when ClickHouse returns no rows."""- with patch("posthog.dags.person_property_reconciliation.sync_execute", return_value=[]):- results = get_person_property_updates_from_clickhouse(- team_id=1,- bug_window_start="2024-01-01T00:00:00Z",- )-- assert results == []-- def test_parses_raw_json_value_types(self):- """Test that raw JSON values from CH are correctly parsed to Python types."""- # Raw JSON representations are parsed via json.loads() to native Python types:- # - strings: "hello" -> "hello"- # - numbers: 123, 3.14 -> int, float- # - booleans: true, false -> True, False- # - null -> None- mock_rows: list[- tuple[- str, int, list[tuple[str, Any, datetime]], list[tuple[str, Any, datetime]], list[tuple[str, datetime]]- ]- ] = [- (- "018d1234-5678-0000-0000-000000000001",- 1, # person_version- [- ("string_prop", '"hello"', datetime(2024, 1, 15, 12, 0, 0)),- ("int_prop", "123", datetime(2024, 1, 15, 12, 0, 0)),- ("float_prop", "3.14", datetime(2024, 1, 15, 12, 0, 0)),- ("bool_true_prop", "true", datetime(2024, 1, 15, 12, 0, 0)),- ("bool_false_prop", "false", datetime(2024, 1, 15, 12, 0, 0)),- ("null_prop", "null", datetime(2024, 1, 15, 12, 0, 0)),- ],- [], # set_once_diff- [], # unset_diff- )- ]-- with patch("posthog.dags.person_property_reconciliation.sync_execute", return_value=mock_rows):- results = get_person_property_updates_from_clickhouse(- team_id=1,- bug_window_start="2024-01-01T00:00:00Z",- )-- assert len(results) == 1- person_diffs = results[0]- assert len(person_diffs.set_updates) == 6-- # String - parsed from JSON string- assert person_diffs.set_updates["string_prop"].value == "hello"- assert isinstance(person_diffs.set_updates["string_prop"].value, str)-- # Integer- assert person_diffs.set_updates["int_prop"].value == 123- assert isinstance(person_diffs.set_updates["int_prop"].value, int)-- # Float- assert person_diffs.set_updates["float_prop"].value == 3.14- assert isinstance(person_diffs.set_updates["float_prop"].value, float)-- # Boolean true- assert person_diffs.set_updates["bool_true_prop"].value is True-- # Boolean false- assert person_diffs.set_updates["bool_false_prop"].value is False-- # Null- assert person_diffs.set_updates["null_prop"].value is None-- def test_parses_unset_diff_tuples(self):- """Test that unset_diff array of tuples is correctly parsed."""- # unset_diff is an array of (key, timestamp) tuples- # Keys are already parsed in the query with JSON_VALUE (consistent with $set/$set_once)- mock_rows: list[- tuple[- str, int, list[tuple[str, Any, datetime]], list[tuple[str, Any, datetime]], list[tuple[str, datetime]]- ]- ] = [- (- "018d1234-5678-0000-0000-000000000001",- 1, # person_version- [], # set_diff - empty- [], # set_once_diff - empty- [ # unset_diff - array of (key, timestamp) tuples- ("email", datetime(2024, 1, 15, 12, 0, 0)),- ("old_property", datetime(2024, 1, 15, 12, 30, 0)),- ],- )- ]-- with patch("posthog.dags.person_property_reconciliation.sync_execute", return_value=mock_rows):- results = get_person_property_updates_from_clickhouse(- team_id=1,- bug_window_start="2024-01-01T00:00:00Z",- )-- assert len(results) == 1- person_diffs = results[0]- assert len(person_diffs.set_updates) == 0- assert len(person_diffs.set_once_updates) == 0- assert len(person_diffs.unset_updates) == 2-- # Verify first unset- assert "email" in person_diffs.unset_updates- assert person_diffs.unset_updates["email"].value is None- assert person_diffs.unset_updates["email"].timestamp == datetime(2024, 1, 15, 12, 0, 0, tzinfo=UTC)-- # Verify second unset- assert "old_property" in person_diffs.unset_updates- assert person_diffs.unset_updates["old_property"].value is None- assert person_diffs.unset_updates["old_property"].timestamp == datetime(2024, 1, 15, 12, 30, 0, tzinfo=UTC)-- def test_parses_mixed_set_set_once_and_unset(self):- """Test parsing when all three operation types have values."""- mock_rows: list[- tuple[- str, int, list[tuple[str, Any, datetime]], list[tuple[str, Any, datetime]], list[tuple[str, datetime]]- ]- ] = [- (- "018d1234-5678-0000-0000-000000000004",- 1, # person_version- [("email", '"[email protected]"', datetime(2024, 1, 15, 12, 0, 0))], # set_diff- [("initial_source", '"organic"', datetime(2024, 1, 10, 8, 0, 0))], # set_once_diff- [("old_field", datetime(2024, 1, 14, 10, 0, 0))], # unset_diff - keys are already parsed- )- ]-- with patch("posthog.dags.person_property_reconciliation.sync_execute", return_value=mock_rows):- results = get_person_property_updates_from_clickhouse(- team_id=1,- bug_window_start="2024-01-01T00:00:00Z",- )-- assert len(results) == 1- person_diffs = results[0]- assert len(person_diffs.set_updates) == 1- assert len(person_diffs.set_once_updates) == 1- assert len(person_diffs.unset_updates) == 1-- # Verify set update- assert "email" in person_diffs.set_updates- assert person_diffs.set_updates["email"].value == "[email protected]"-- # Verify set_once update- assert "initial_source" in person_diffs.set_once_updates- assert person_diffs.set_once_updates["initial_source"].value == "organic"-- # Verify unset update- assert "old_field" in person_diffs.unset_updates- assert person_diffs.unset_updates["old_field"].value is None-- def test_no_bug_window_end_parameter_regression(self):- """Regression test: get_person_property_updates_from_clickhouse should NOT accept bug_window_end.-- The function uses ClickHouse's now() for the upper bound, ensuring all events- from bug_window_start until the current time are included. This is intentional:- - bug_window_start/end is used to FIND teams with affected events- - When processing a team, we read ALL events from bug_window_start to now()-- This prevents missing events that occurred after the initial bug window discovery.- """- import inspect-- sig = inspect.signature(get_person_property_updates_from_clickhouse)- param_names = list(sig.parameters.keys())-- assert "bug_window_end" not in param_names, (- "bug_window_end should not be a parameter. The function should use ClickHouse's now() for the upper bound."- )- assert "team_id" in param_names- assert "bug_window_start" in param_names---class TestReconcilePersonProperties:- """Test the reconcile_person_properties function."""-- def test_set_creates_new_property(self):- """Test that $set creates a property that doesn't exist in PG."""- person = {- "uuid": "018d1234-5678-0000-0000-000000000001",- "properties": {},- "properties_last_updated_at": {},- "properties_last_operation": {},- }- person_diffs = PersonPropertyDiffs(- person_id="018d1234-5678-0000-0000-000000000001",- person_version=1,- set_updates={"email": PropertyValue(timestamp=datetime(2024, 1, 15, 12, 0, 0), value="[email protected]")},- set_once_updates={},- unset_updates={},- )-- result = reconcile_person_properties(person, person_diffs)-- assert result is not None- assert result["properties"]["email"] == "[email protected]"- assert "email" in result["properties_last_updated_at"]- assert result["properties_last_operation"]["email"] == "set"-- def test_set_always_applies(self):- """Test that $set always applies (CH query pre-filters to only return diffs)."""- person = {- "uuid": "018d1234-5678-0000-0000-000000000001",- "properties": {"email": "[email protected]"},- "properties_last_updated_at": {"email": "2024-01-10T00:00:00+00:00"},- "properties_last_operation": {"email": "set"},- }- person_diffs = PersonPropertyDiffs(- person_id="018d1234-5678-0000-0000-000000000001",- person_version=1,- set_updates={- "email": PropertyValue(timestamp=datetime(2024, 1, 15, 12, 0, 0, tzinfo=UTC), value="[email protected]")- },- set_once_updates={},- unset_updates={},- )-- result = reconcile_person_properties(person, person_diffs)-- assert result is not None- assert result["properties"]["email"] == "[email protected]"-- def test_set_once_creates_missing_property(self):- """Test that $set_once creates a property that doesn't exist in PG."""- person = {- "uuid": "018d1234-5678-0000-0000-000000000001",- "properties": {"other_prop": "value"},- "properties_last_updated_at": {},- "properties_last_operation": {},- }- person_diffs = PersonPropertyDiffs(- person_id="018d1234-5678-0000-0000-000000000001",- person_version=1,- set_updates={},- set_once_updates={- "initial_referrer": PropertyValue(timestamp=datetime(2024, 1, 10, 8, 0, 0), value="google.com")- },- unset_updates={},- )-- result = reconcile_person_properties(person, person_diffs)-- assert result is not None- assert result["properties"]["initial_referrer"] == "google.com"- assert result["properties_last_operation"]["initial_referrer"] == "set_once"-- def test_set_once_skips_existing_property(self):- """Test that $set_once doesn't overwrite an existing property."""- person = {- "uuid": "018d1234-5678-0000-0000-000000000001",- "properties": {"initial_referrer": "facebook.com"},- "properties_last_updated_at": {"initial_referrer": "2024-01-05T00:00:00+00:00"},- "properties_last_operation": {"initial_referrer": "set_once"},- }- person_diffs = PersonPropertyDiffs(- person_id="018d1234-5678-0000-0000-000000000001",- person_version=1,- set_updates={},- set_once_updates={- "initial_referrer": PropertyValue(timestamp=datetime(2024, 1, 10, 8, 0, 0), value="google.com")- },- unset_updates={},- )-- result = reconcile_person_properties(person, person_diffs)-- # No changes - property already exists- assert result is None-- def test_handles_multiple_updates(self):- """Test processing multiple updates for different properties."""- person = {- "uuid": "018d1234-5678-0000-0000-000000000001",- "properties": {"existing": "value"},- "properties_last_updated_at": {},- "properties_last_operation": {},- }- person_diffs = PersonPropertyDiffs(- person_id="018d1234-5678-0000-0000-000000000001",- person_version=1,- set_updates={- "prop1": PropertyValue(timestamp=datetime(2024, 1, 15, 12, 0, 0), value="val1"),- "prop2": PropertyValue(timestamp=datetime(2024, 1, 15, 12, 0, 0), value="val2"),- },- set_once_updates={- "prop3": PropertyValue(timestamp=datetime(2024, 1, 10, 8, 0, 0), value="val3"),- },- unset_updates={},- )-- result = reconcile_person_properties(person, person_diffs)-- assert result is not None- assert result["properties"]["prop1"] == "val1"- assert result["properties"]["prop2"] == "val2"- assert result["properties"]["prop3"] == "val3"- # Original property preserved- assert result["properties"]["existing"] == "value"-- def test_handles_null_properties(self):- """Test handling when person has None for properties fields."""- person = {- "uuid": "018d1234-5678-0000-0000-000000000001",- "properties": None,- "properties_last_updated_at": None,- "properties_last_operation": None,- }- person_diffs = PersonPropertyDiffs(- person_id="018d1234-5678-0000-0000-000000000001",- person_version=1,- set_updates={"email": PropertyValue(timestamp=datetime(2024, 1, 15, 12, 0, 0), value="[email protected]")},- set_once_updates={},- unset_updates={},- )-- result = reconcile_person_properties(person, person_diffs)-- assert result is not None- assert result["properties"]["email"] == "[email protected]"-- def test_returns_none_when_empty_diffs(self):- """Test that None is returned when diffs are empty."""- person = {- "uuid": "018d1234-5678-0000-0000-000000000001",- "properties": {"email": "[email protected]"},- "properties_last_updated_at": {"email": "2024-01-20T00:00:00+00:00"},- "properties_last_operation": {"email": "set"},- }- person_diffs = PersonPropertyDiffs(- person_id="018d1234-5678-0000-0000-000000000001",- person_version=1,- set_updates={},- set_once_updates={},- unset_updates={},- )-- result = reconcile_person_properties(person, person_diffs)- assert result is None-- def test_unset_removes_existing_property(self):- """Test that $unset removes a property."""- person = {- "uuid": "018d1234-5678-0000-0000-000000000001",- "properties": {"email": "[email protected]", "name": "Test User"},- "properties_last_updated_at": {"email": "2024-01-10T00:00:00+00:00", "name": "2024-01-10T00:00:00+00:00"},- "properties_last_operation": {"email": "set", "name": "set"},- }- person_diffs = PersonPropertyDiffs(- person_id="018d1234-5678-0000-0000-000000000001",- person_version=1,- set_updates={},- set_once_updates={},- unset_updates={"email": PropertyValue(timestamp=datetime(2024, 1, 15, 12, 0, 0, tzinfo=UTC), value=None)},- )-- result = reconcile_person_properties(person, person_diffs)-- assert result is not None- assert "email" not in result["properties"]- assert "email" not in result["properties_last_updated_at"]- assert "email" not in result["properties_last_operation"]- # Other properties should remain unchanged- assert result["properties"]["name"] == "Test User"-- def test_unset_marks_changed_even_when_property_not_exists(self):- """Test that $unset marks changed even when property doesn't exist in properties map."""- person = {- "uuid": "018d1234-5678-0000-0000-000000000001",- "properties": {"name": "Test User"},- "properties_last_updated_at": {"name": "2024-01-10T00:00:00+00:00"},- "properties_last_operation": {"name": "set"},- }- person_diffs = PersonPropertyDiffs(- person_id="018d1234-5678-0000-0000-000000000001",- person_version=1,- set_updates={},- set_once_updates={},- unset_updates={"email": PropertyValue(timestamp=datetime(2024, 1, 15, 12, 0, 0, tzinfo=UTC), value=None)},- )-- result = reconcile_person_properties(person, person_diffs)-- # Unset always marks changed (even if key wasn't in properties)- assert result is not None-- def test_unset_removes_property_with_no_timestamp(self):- """Test that $unset removes property when there's no existing timestamp."""- person = {- "uuid": "018d1234-5678-0000-0000-000000000001",- "properties": {"email": "[email protected]"},- "properties_last_updated_at": {}, # No timestamp for email- "properties_last_operation": {},- }- person_diffs = PersonPropertyDiffs(- person_id="018d1234-5678-0000-0000-000000000001",- person_version=1,- set_updates={},- set_once_updates={},- unset_updates={"email": PropertyValue(timestamp=datetime(2024, 1, 15, 12, 0, 0, tzinfo=UTC), value=None)},- )-- result = reconcile_person_properties(person, person_diffs)-- assert result is not None- assert "email" not in result["properties"]---class TestUpdatePersonWithVersionCheck:- """Test the update_person_with_version_check function."""-- def create_mock_cursor(self, person_data=None, update_success=True):- """Create a mock cursor with configurable behavior."""- cursor = MagicMock()-- # Mock fetchone to return person data- cursor.fetchone.return_value = person_data-- # Mock rowcount for UPDATE success/failure- cursor.rowcount = 1 if update_success else 0-- return cursor-- def test_successful_update(self):- """Test successful update with version check."""- person_data = {- "id": 123,- "uuid": "018d1234-5678-0000-0000-000000000001",- "properties": {"existing": "value"},- "properties_last_updated_at": {},- "properties_last_operation": {},- "version": 5,- "is_identified": True,- "created_at": datetime(2024, 1, 1, 0, 0, 0),- }- cursor = self.create_mock_cursor(person_data=person_data, update_success=True)-- person_diffs = PersonPropertyDiffs(- person_id="018d1234-5678-0000-0000-000000000001",- person_version=5,- set_updates={"email": PropertyValue(timestamp=datetime(2024, 1, 15, 12, 0, 0), value="[email protected]")},- set_once_updates={},- unset_updates={},- )-- result = update_person_with_version_check(- cursor=cursor,- job_id="test-job-id",- team_id=1,- person_uuid="018d1234-5678-0000-0000-000000000001",- person_property_diffs=person_diffs,- dry_run=False,- )-- assert result.success is True- assert result.updated_person_data is not None- assert result.updated_person_data["version"] == 6 # version incremented- assert result.updated_person_data["properties"]["email"] == "[email protected]"- assert result.updated_person_data["properties"]["existing"] == "value"-- # Verify UPDATE was executed- update_calls = [call for call in cursor.execute.call_args_list if "UPDATE posthog_person" in str(call)]- assert len(update_calls) == 1-- # Verify backup INSERT was executed- backup_calls = [- call- for call in cursor.execute.call_args_list- if "INSERT INTO posthog_person_reconciliation_backup" in str(call)- ]- assert len(backup_calls) == 1-- # Verify backup contains correct data- backup_call_args = backup_calls[0][0] # (query, params)- backup_params = backup_call_args[1]- assert backup_params[0] == "test-job-id" # job_id- assert backup_params[1] == 1 # team_id- assert backup_params[2] == 123 # person_id- assert backup_params[3] == "018d1234-5678-0000-0000-000000000001" # uuid- # Before state- assert '"existing": "value"' in backup_params[4] # properties (before)- assert backup_params[7] == 5 # version (before)- # After state- assert '"email": "[email protected]"' in backup_params[12] # properties_after- assert backup_params[15] == 6 # version_after-- def test_dry_run_does_not_write(self):- """Test that dry_run=True doesn't execute UPDATE."""- person_data = {- "id": 123,- "uuid": "018d1234-5678-0000-0000-000000000001",- "properties": {},- "properties_last_updated_at": {},- "properties_last_operation": {},- "version": 1,- "is_identified": False,- "created_at": datetime(2024, 1, 1, 0, 0, 0),- }- cursor = self.create_mock_cursor(person_data=person_data)-- person_diffs = PersonPropertyDiffs(- person_id="018d1234-5678-0000-0000-000000000001",- person_version=1,- set_updates={"email": PropertyValue(timestamp=datetime(2024, 1, 15, 12, 0, 0), value="[email protected]")},- set_once_updates={},- unset_updates={},- )-- result = update_person_with_version_check(- cursor=cursor,- job_id="test-job-id",- team_id=1,- person_uuid="018d1234-5678-0000-0000-000000000001",- person_property_diffs=person_diffs,- dry_run=True,- )-- assert result.success is True- assert result.updated_person_data is None # No data returned for Kafka in dry run-- # Verify UPDATE was NOT executed- update_calls = [call for call in cursor.execute.call_args_list if "UPDATE posthog_person" in str(call)]- assert len(update_calls) == 0-- # Verify backup INSERT was STILL executed (for audit trail)- backup_calls = [- call- for call in cursor.execute.call_args_list- if "INSERT INTO posthog_person_reconciliation_backup" in str(call)- ]- assert len(backup_calls) == 1-- def test_person_not_found(self):- """Test handling when person doesn't exist in PG."""- cursor = self.create_mock_cursor(person_data=None)-- person_diffs = PersonPropertyDiffs(- person_id="018d1234-5678-0000-0000-nonexistent",- person_version=1,- set_updates={"email": PropertyValue(timestamp=datetime(2024, 1, 15, 12, 0, 0), value="[email protected]")},- set_once_updates={},- unset_updates={},- )-- result = update_person_with_version_check(- cursor=cursor,- job_id="test-job-id",- team_id=1,- person_uuid="018d1234-5678-0000-0000-nonexistent",- person_property_diffs=person_diffs,- )-- assert result.success is False- assert result.updated_person_data is None- assert result.skip_reason == SkipReason.NOT_FOUND-- @patch("posthog.dags.person_property_reconciliation.fetch_person_properties_from_clickhouse")- def test_version_mismatch_retry(self, mock_fetch_ch_properties):- """Test retry on version mismatch (concurrent modification).-- When the first UPDATE fails due to version mismatch:- 1. On retry, if Postgres version differs from CH version (person_version in diffs)- 2. We fetch CH properties for 3-way merge- 3. Apply changes and retry UPDATE with new target version- """- person_data_v1 = {- "id": 123,- "uuid": "018d1234-5678-0000-0000-000000000001",- "properties": {},- "properties_last_updated_at": {},- "properties_last_operation": {},- "version": 1,- "is_identified": False,- "created_at": datetime(2024, 1, 1, 0, 0, 0),- }- person_data_v2 = {- "id": 123,- "uuid": "018d1234-5678-0000-0000-000000000001",- "properties": {"other": "concurrent_change"}, # Concurrent change to Postgres- "properties_last_updated_at": {"other": "2024-01-14T00:00:00"},- "properties_last_operation": {"other": "set"},- "version": 2, # Version changed by concurrent update- "is_identified": False,- "created_at": datetime(2024, 1, 1, 0, 0, 0),- }-- # CH properties at version 1 (baseline for 3-way merge)- mock_fetch_ch_properties.return_value = {}-- cursor = MagicMock()- # First fetch returns v1, second fetch returns v2- cursor.fetchone.side_effect = [person_data_v1, person_data_v2]- # First update fails (version mismatch), second succeeds- cursor.rowcount = 0 # Will be set by side_effect-- update_attempt = [0]-- def execute_side_effect(query, *args):- if "UPDATE posthog_person" in query:- update_attempt[0] += 1- # First attempt fails, second succeeds- cursor.rowcount = 1 if update_attempt[0] > 1 else 0-- cursor.execute.side_effect = execute_side_effect-- person_diffs = PersonPropertyDiffs(- person_id="018d1234-5678-0000-0000-000000000001",- person_version=1,- set_updates={"email": PropertyValue(timestamp=datetime(2024, 1, 15, 12, 0, 0), value="[email protected]")},- set_once_updates={},- unset_updates={},- )-- result = update_person_with_version_check(- cursor=cursor,- job_id="test-job-id",- team_id=1,- person_uuid="018d1234-5678-0000-0000-000000000001",- person_property_diffs=person_diffs,- max_retries=3,- )-- assert result.success is True- assert result.updated_person_data is not None- assert result.updated_person_data["version"] == 3 # v2 + 1- # Concurrent change should be preserved, our change should be applied- assert result.updated_person_data["properties"]["other"] == "concurrent_change"- assert result.updated_person_data["properties"]["email"] == "[email protected]"-- def test_exhausted_retries(self):- """Test failure after exhausting all retries."""- person_data = {- "id": 123,- "uuid": "018d1234-5678-0000-0000-000000000001",- "properties": {},- "properties_last_updated_at": {},- "properties_last_operation": {},- "version": 1,- "is_identified": False,- "created_at": datetime(2024, 1, 1, 0, 0, 0),- }-- cursor = MagicMock()- cursor.fetchone.return_value = person_data- cursor.rowcount = 0 # Always fail version check-- person_diffs = PersonPropertyDiffs(- person_id="018d1234-5678-0000-0000-000000000001",- person_version=1,- set_updates={"email": PropertyValue(timestamp=datetime(2024, 1, 15, 12, 0, 0), value="[email protected]")},- set_once_updates={},- unset_updates={},- )-- result = update_person_with_version_check(- cursor=cursor,- job_id="test-job-id",- team_id=1,- person_uuid="018d1234-5678-0000-0000-000000000001",- person_property_diffs=person_diffs,- max_retries=3,- )-- assert result.success is False- assert result.updated_person_data is None- assert result.skip_reason == SkipReason.VERSION_CONFLICT---class TestBatchCommits:- """Test that batch commits work correctly with different batch sizes.-- These tests use the process_persons_in_batches helper function directly,- which is separated from the Dagster op for testability.- """-- @pytest.mark.parametrize(- "num_persons,batch_size,expected_commits",- [- # 5 persons, batch_size=2: batches of [2, 2, 1] = 3 commits- (5, 2, 3),- # 4 persons, batch_size=2: batches of [2, 2] = 2 commits (no partial)- (4, 2, 2),- # 3 persons, batch_size=5: batches of [3] = 1 commit (all in one partial batch)- (3, 5, 1),- # 1 person, batch_size=100: batches of [1] = 1 commit- (1, 100, 1),- # 6 persons, batch_size=0 (disabled): single commit at end- (6, 0, 1),- ],- )- def test_batch_commits_correct_number_of_times(self, num_persons: int, batch_size: int, expected_commits: int):- """- Test that commits happen the correct number of times based on batch_size.-- With batch_size=2 and 5 persons:- - Batch 1: persons 0,1 → commit- - Batch 2: persons 2,3 → commit- - Batch 3: person 4 → commit (partial final batch)- Total: 3 commits- """- from posthog.dags.person_property_reconciliation import process_persons_in_batches-- # Create mock persons with updates (timestamps must be timezone-aware)- person_diffs_list = []- for i in range(num_persons):- person_diffs_list.append(- PersonPropertyDiffs(- person_id=f"person-uuid-{i}",- person_version=1,- set_updates={- "prop": PropertyValue(timestamp=datetime(2024, 1, 15, 12, 0, 0, tzinfo=UTC), value=f"value_{i}")- },- set_once_updates={},- unset_updates={},- )- )-- # Mock cursor- mock_cursor = MagicMock()- mock_cursor.fetchone.return_value = {- "id": 123,- "uuid": "test-uuid",- "properties": {"prop": "old_value"},- "properties_last_updated_at": {"prop": "2024-01-01T00:00:00+00:00"},- "properties_last_operation": {"prop": "set"},- "version": 1,- "is_identified": False,- "created_at": datetime(2024, 1, 1, 0, 0, 0),- }- mock_cursor.rowcount = 1 # UPDATE succeeds-- # Track commits- commit_count = [0]-- def mock_commit():- commit_count[0] += 1-- # Track batch callbacks- batch_callbacks = []-- def on_batch_committed(batch_num, batch_persons):- batch_callbacks.append((batch_num, len(batch_persons)))-- # Call the helper function directly- result = process_persons_in_batches(- person_property_diffs=person_diffs_list,- cursor=mock_cursor,- job_id="test-job-id",- team_id=1,- batch_size=batch_size,- dry_run=False,- backup_enabled=False,- commit_fn=mock_commit,- on_batch_committed=on_batch_committed,- )-- # Verify commit was called the expected number of times- assert result.total_commits == expected_commits, (- f"Expected {expected_commits} commits for {num_persons} persons with batch_size={batch_size}, "- f"got {result.total_commits}"- )-- # Verify all persons were processed- assert result.total_processed == num_persons- assert result.total_updated == num_persons-- # Verify batch callback was called for each batch- assert len(batch_callbacks) == expected_commits-- def test_batch_commits_partial_final_batch_all_persons_updated(self):- """- Specifically test that the partial final batch commits correctly- and all persons are updated (not just full batches).-- 5 persons with batch_size=2:- - Batch 1: persons 0,1 → commit ✓- - Batch 2: persons 2,3 → commit ✓- - Batch 3: person 4 → commit ✓ (THIS is the partial batch we're testing)- """- from posthog.dags.person_property_reconciliation import process_persons_in_batches-- num_persons = 5- batch_size = 2-- person_uuids = [f"uuid-{i}" for i in range(num_persons)]- person_diffs_list = [- PersonPropertyDiffs(- person_id=uuid,- person_version=1,- set_updates={- "email": PropertyValue(- timestamp=datetime(2024, 1, 15, 12, 0, 0, tzinfo=UTC), value=f"user{i}@example.com"- )- },- set_once_updates={},- unset_updates={},- )- for i, uuid in enumerate(person_uuids)- ]-- mock_cursor = MagicMock()- mock_cursor.fetchone.return_value = {- "id": 123,- "uuid": "test-uuid",- "properties": {"email": "[email protected]"},- "properties_last_updated_at": {"email": "2024-01-01T00:00:00+00:00"},- "properties_last_operation": {"email": "set"},- "version": 1,- "is_identified": False,- "created_at": datetime(2024, 1, 1, 0, 0, 0),- }- mock_cursor.rowcount = 1-- commit_count = [0]-- def mock_commit():- commit_count[0] += 1-- # Track batch sizes to verify partial final batch- batch_sizes = []-- def on_batch_committed(batch_num, batch_persons):- batch_sizes.append(len(batch_persons))-- result = process_persons_in_batches(- person_property_diffs=person_diffs_list,- cursor=mock_cursor,- job_id="test-job-id",- team_id=1,- batch_size=batch_size,- dry_run=False,- backup_enabled=False,- commit_fn=mock_commit,- on_batch_committed=on_batch_committed,- )-- # All 5 persons should be processed and updated- assert result.total_processed == 5- assert result.total_updated == 5- assert result.total_skipped == 0-- # 3 commits: batch 1 (2), batch 2 (2), batch 3 (1)- assert result.total_commits == 3-- # Verify batch sizes: [2, 2, 1]- assert batch_sizes == [2, 2, 1], f"Expected batches [2, 2, 1], got {batch_sizes}"-- def test_empty_person_list_no_commits(self):- """Test that empty person list results in zero commits."""- from posthog.dags.person_property_reconciliation import process_persons_in_batches-- mock_cursor = MagicMock()- commit_count = [0]-- def mock_commit():- commit_count[0] += 1-- result = process_persons_in_batches(- person_property_diffs=[],- cursor=mock_cursor,- job_id="test-job-id",- team_id=1,- batch_size=10,- dry_run=False,- backup_enabled=False,- commit_fn=mock_commit,- )-- assert result.total_processed == 0- assert result.total_commits == 0- assert commit_count[0] == 0-- def test_dry_run_no_commits(self):- """Test that dry_run=True doesn't commit when there are no backups."""- from posthog.dags.person_property_reconciliation import process_persons_in_batches-- person_diffs_list = [- PersonPropertyDiffs(- person_id="person-uuid-0",- person_version=1,- set_updates={- "prop": PropertyValue(timestamp=datetime(2024, 1, 15, 12, 0, 0, tzinfo=UTC), value="value_0")- },- set_once_updates={},- unset_updates={},- )- ]-- mock_cursor = MagicMock()- # Return None from fetchone to simulate person not found (no backup, no update)- mock_cursor.fetchone.return_value = None-- commit_count = [0]-- def mock_commit():- commit_count[0] += 1-- result = process_persons_in_batches(- person_property_diffs=person_diffs_list,- cursor=mock_cursor,- job_id="test-job-id",- team_id=1,- batch_size=10,- dry_run=True,- backup_enabled=False,- commit_fn=mock_commit,- )-- # Person not found, so skipped- assert result.total_processed == 1- assert result.total_skipped == 1- assert result.total_commits == 0---class TestBackupFunctionality:- """Test the backup functionality for person property reconciliation."""-- def test_backup_contains_pending_operations(self):- """Test that backup stores the pending operations correctly."""- import json-- person_data = {- "id": 123,- "uuid": "018d1234-5678-0000-0000-000000000001",- "properties": {"existing": "value"},- "properties_last_updated_at": {"existing": "2024-01-01T00:00:00"},- "properties_last_operation": {"existing": "set"},- "version": 5,- "is_identified": True,- "created_at": datetime(2024, 1, 1, 0, 0, 0),- }-- cursor = MagicMock()- cursor.fetchone.return_value = person_data- cursor.rowcount = 1-- person_diffs = PersonPropertyDiffs(- person_id="018d1234-5678-0000-0000-000000000001",- person_version=5,- set_updates={"email": PropertyValue(timestamp=datetime(2024, 1, 15, 12, 0, 0), value="[email protected]")},- set_once_updates={"name": PropertyValue(timestamp=datetime(2024, 1, 15, 12, 0, 0), value="Test User")},- unset_updates={},- )-- update_person_with_version_check(- cursor=cursor,- job_id="test-job-id",- team_id=1,- person_uuid="018d1234-5678-0000-0000-000000000001",- person_property_diffs=person_diffs,- dry_run=False,- )-- # Find the backup INSERT call- backup_calls = [- call- for call in cursor.execute.call_args_list- if "INSERT INTO posthog_person_reconciliation_backup" in str(call)- ]- assert len(backup_calls) == 1-- backup_params = backup_calls[0][0][1]- pending_operations_json = backup_params[11] # pending_operations is at index 11- pending_operations = json.loads(pending_operations_json)-- assert len(pending_operations) == 2- # Find operations by key since dict iteration order may vary- email_op = next(op for op in pending_operations if op["key"] == "email")- name_op = next(op for op in pending_operations if op["key"] == "name")- assert email_op["value"] == "[email protected]"- assert email_op["operation"] == "set"- assert name_op["value"] == "Test User"- assert name_op["operation"] == "set_once"-- def test_backup_preserves_before_and_after_state(self):- """Test that backup correctly stores both before and after state."""- import json-- person_data = {- "id": 456,- "uuid": "018d1234-5678-0000-0000-000000000002",- "properties": {"name": "Old Name", "count": 10},- "properties_last_updated_at": {},- "properties_last_operation": {},- "version": 3,- "is_identified": False,- "is_user_id": None,- "created_at": datetime(2024, 1, 1, 0, 0, 0),- }-- cursor = MagicMock()- cursor.fetchone.return_value = person_data- cursor.rowcount = 1-- person_diffs = PersonPropertyDiffs(- person_id="018d1234-5678-0000-0000-000000000002",- person_version=3,- set_updates={"name": PropertyValue(timestamp=datetime(2024, 1, 15, 12, 0, 0), value="New Name")},- set_once_updates={},- unset_updates={},- )-- update_person_with_version_check(- cursor=cursor,- job_id="run-abc-123",- team_id=42,- person_uuid="018d1234-5678-0000-0000-000000000002",- person_property_diffs=person_diffs,- dry_run=False,- )-- backup_calls = [- call- for call in cursor.execute.call_args_list- if "INSERT INTO posthog_person_reconciliation_backup" in str(call)- ]- assert len(backup_calls) == 1-- backup_params = backup_calls[0][0][1]-- # Verify identifiers- assert backup_params[0] == "run-abc-123" # job_id- assert backup_params[1] == 42 # team_id- assert backup_params[2] == 456 # person_id-- # Verify before state- properties_before = json.loads(backup_params[4])- assert properties_before == {"name": "Old Name", "count": 10}- assert backup_params[7] == 3 # version before-- # Verify after state- properties_after = json.loads(backup_params[12])- assert properties_after["name"] == "New Name"- assert properties_after["count"] == 10 # unchanged property preserved- assert backup_params[15] == 4 # version after (3 + 1)-- def test_backup_disabled_does_not_create_backup(self):- """Test that backup_enabled=False skips backup creation."""- person_data = {- "id": 123,- "uuid": "018d1234-5678-0000-0000-000000000001",- "properties": {"existing": "value"},- "properties_last_updated_at": {},- "properties_last_operation": {},- "version": 5,- "is_identified": True,- "created_at": datetime(2024, 1, 1, 0, 0, 0),- }-- cursor = MagicMock()- cursor.fetchone.return_value = person_data- cursor.rowcount = 1-- person_diffs = PersonPropertyDiffs(- person_id="018d1234-5678-0000-0000-000000000001",- person_version=5,- set_updates={"email": PropertyValue(timestamp=datetime(2024, 1, 15, 12, 0, 0), value="[email protected]")},- set_once_updates={},- unset_updates={},- )-- result = update_person_with_version_check(- cursor=cursor,- job_id="test-job-id",- team_id=1,- person_uuid="018d1234-5678-0000-0000-000000000001",- person_property_diffs=person_diffs,- dry_run=False,- backup_enabled=False,- )-- assert result.success is True- assert result.updated_person_data is not None- assert result.backup_created is False-- # Verify backup INSERT was NOT executed- backup_calls = [- call- for call in cursor.execute.call_args_list- if "INSERT INTO posthog_person_reconciliation_backup" in str(call)- ]- assert len(backup_calls) == 0-- # But UPDATE should still happen- update_calls = [call for call in cursor.execute.call_args_list if "UPDATE posthog_person" in str(call)]- assert len(update_calls) == 1-- def test_backup_created_return_value_true_when_enabled(self):- """Test that backup_created=True when backup is enabled and changes made."""- person_data = {- "id": 123,- "uuid": "018d1234-5678-0000-0000-000000000001",- "properties": {},- "properties_last_updated_at": {},- "properties_last_operation": {},- "version": 1,- "is_identified": False,- "created_at": datetime(2024, 1, 1, 0, 0, 0),- }-- cursor = MagicMock()- cursor.fetchone.return_value = person_data- cursor.rowcount = 1-- person_diffs = PersonPropertyDiffs(- person_id="018d1234-5678-0000-0000-000000000001",- person_version=1,- set_updates={"email": PropertyValue(timestamp=datetime(2024, 1, 15, 12, 0, 0), value="[email protected]")},- set_once_updates={},- unset_updates={},- )-- result = update_person_with_version_check(- cursor=cursor,- job_id="test-job-id",- team_id=1,- person_uuid="018d1234-5678-0000-0000-000000000001",- person_property_diffs=person_diffs,- dry_run=False,- backup_enabled=True,- )-- assert result.success is True- assert result.backup_created is True-- def test_backup_created_false_when_conflict(self):- """- Regression test: backup_created should be False when ON CONFLICT DO NOTHING- means no row was actually inserted (duplicate backup for same person/job).- """- person_data = {- "id": 123,- "uuid": "018d1234-5678-0000-0000-000000000001",- "properties": {},- "properties_last_updated_at": {},- "properties_last_operation": {},- "version": 1,- "is_identified": False,- "created_at": datetime(2024, 1, 1, 0, 0, 0),- }-- cursor = MagicMock()- cursor.fetchone.return_value = person_data-- # Track which query is being executed to return different rowcounts- call_count = [0]- original_execute = cursor.execute-- def execute_side_effect(*args, **kwargs):- call_count[0] += 1- query = args[0] if args else ""- if "INSERT INTO posthog_person_reconciliation_backup" in query:- # Simulate ON CONFLICT DO NOTHING - no row inserted- cursor.rowcount = 0- else:- # Person SELECT/UPDATE succeeds- cursor.rowcount = 1- return original_execute(*args, **kwargs)-- cursor.execute = MagicMock(side_effect=execute_side_effect)-- person_diffs = PersonPropertyDiffs(- person_id="018d1234-5678-0000-0000-000000000001",- person_version=1,- set_updates={"email": PropertyValue(timestamp=datetime(2024, 1, 15, 12, 0, 0), value="[email protected]")},- set_once_updates={},- unset_updates={},- )-- result = update_person_with_version_check(- cursor=cursor,- job_id="test-job-id",- team_id=1,- person_uuid="018d1234-5678-0000-0000-000000000001",- person_property_diffs=person_diffs,- dry_run=False,- backup_enabled=True,- )-- assert result.success is True- # Key assertion: backup_created should be False because rowcount was 0- assert result.backup_created is False---class TestFilterEventPersonProperties:- """Test the filter_event_person_properties function for conflict resolution."""-- def test_set_wins_when_newer_than_unset(self):- """Test that $set wins over $unset when set timestamp is newer."""- person_diffs = [- PersonPropertyDiffs(- person_id="018d1234-5678-0000-0000-000000000001",- person_version=1,- set_updates={- "email": PropertyValue(- timestamp=datetime(2024, 1, 15, 12, 0, 0, tzinfo=UTC), value="[email protected]"- )- },- set_once_updates={},- unset_updates={- "email": PropertyValue(timestamp=datetime(2024, 1, 10, 12, 0, 0, tzinfo=UTC), value=None)- },- )- ]-- result = filter_event_person_properties(person_diffs)-- assert len(result) == 1- # set should win (newer), unset should be removed- assert "email" in result[0].set_updates- assert "email" not in result[0].unset_updates-- def test_unset_wins_when_newer_than_set(self):- """Test that $unset wins over $set when unset timestamp is newer or equal."""- person_diffs = [- PersonPropertyDiffs(- person_id="018d1234-5678-0000-0000-000000000001",- person_version=1,- set_updates={- "email": PropertyValue(- timestamp=datetime(2024, 1, 10, 12, 0, 0, tzinfo=UTC), value="[email protected]"- )- },- set_once_updates={},- unset_updates={- "email": PropertyValue(timestamp=datetime(2024, 1, 15, 12, 0, 0, tzinfo=UTC), value=None)- },- )- ]-- result = filter_event_person_properties(person_diffs)-- assert len(result) == 1- # unset should win (newer), set should be removed- assert "email" not in result[0].set_updates- assert "email" in result[0].unset_updates-- def test_set_once_wins_when_newer_than_unset(self):- """Test that $set_once wins over $unset when set_once timestamp is newer."""- person_diffs = [- PersonPropertyDiffs(- person_id="018d1234-5678-0000-0000-000000000001",- person_version=1,- set_updates={},- set_once_updates={- "referrer": PropertyValue(timestamp=datetime(2024, 1, 15, 12, 0, 0, tzinfo=UTC), value="google.com")- },- unset_updates={- "referrer": PropertyValue(timestamp=datetime(2024, 1, 10, 12, 0, 0, tzinfo=UTC), value=None)- },- )- ]-- result = filter_event_person_properties(person_diffs)-- assert len(result) == 1- # set_once should win (newer), unset should be removed- assert "referrer" in result[0].set_once_updates- assert "referrer" not in result[0].unset_updates-- def test_unset_wins_when_newer_than_set_once(self):- """Test that $unset wins over $set_once when unset timestamp is newer."""- person_diffs = [- PersonPropertyDiffs(- person_id="018d1234-5678-0000-0000-000000000001",- person_version=1,- set_updates={},- set_once_updates={- "referrer": PropertyValue(timestamp=datetime(2024, 1, 10, 12, 0, 0, tzinfo=UTC), value="google.com")- },- unset_updates={- "referrer": PropertyValue(timestamp=datetime(2024, 1, 15, 12, 0, 0, tzinfo=UTC), value=None)- },- )- ]-- result = filter_event_person_properties(person_diffs)-- assert len(result) == 1- # unset should win (newer), set_once should be removed- assert "referrer" not in result[0].set_once_updates- assert "referrer" in result[0].unset_updates-- def test_no_conflict_passes_through(self):- """Test that operations on different keys pass through unchanged."""- person_diffs = [- PersonPropertyDiffs(- person_id="018d1234-5678-0000-0000-000000000001",- person_version=1,- set_updates={- "email": PropertyValue(- timestamp=datetime(2024, 1, 15, 12, 0, 0, tzinfo=UTC), value="[email protected]"- )- },- set_once_updates={- "referrer": PropertyValue(timestamp=datetime(2024, 1, 10, 8, 0, 0, tzinfo=UTC), value="google.com")- },- unset_updates={- "old_field": PropertyValue(timestamp=datetime(2024, 1, 14, 10, 0, 0, tzinfo=UTC), value=None)- },- )- ]-- result = filter_event_person_properties(person_diffs)-- assert len(result) == 1- assert "email" in result[0].set_updates- assert "referrer" in result[0].set_once_updates- assert "old_field" in result[0].unset_updates-- def test_multiple_persons_filtered_independently(self):- """Test that multiple persons are filtered independently."""- person_diffs = [- PersonPropertyDiffs(- person_id="person1",- person_version=1,- set_updates={- "email": PropertyValue(- timestamp=datetime(2024, 1, 15, 12, 0, 0, tzinfo=UTC), value="[email protected]"- )- },- set_once_updates={},- unset_updates={- "email": PropertyValue(timestamp=datetime(2024, 1, 10, 12, 0, 0, tzinfo=UTC), value=None)- },- ),- PersonPropertyDiffs(- person_id="person2",- person_version=1,- set_updates={- "email": PropertyValue(- timestamp=datetime(2024, 1, 10, 12, 0, 0, tzinfo=UTC), value="[email protected]"- )- },- set_once_updates={},- unset_updates={- "email": PropertyValue(timestamp=datetime(2024, 1, 15, 12, 0, 0, tzinfo=UTC), value=None)- },- ),- ]-- result = filter_event_person_properties(person_diffs)-- assert len(result) == 2- # Person 1: set wins (newer)- assert "email" in result[0].set_updates- assert "email" not in result[0].unset_updates- # Person 2: unset wins (newer)- assert "email" not in result[1].set_updates- assert "email" in result[1].unset_updates---class TestMergeRawPersonPropertyUpdates:- """Test the merge_raw_person_property_updates function for windowed query accumulation."""-- def test_first_person_added_to_empty_accumulator(self):- """Test that first person updates are added directly to empty accumulator."""- accumulated: dict[str, RawPersonPropertyUpdates] = {}- new_updates = [- RawPersonPropertyUpdates(- person_id="person-1",- set_updates={"email": PropertyValue(datetime(2024, 1, 15, 12, 0, 0, tzinfo=UTC), "[email protected]")},- set_once_updates={},- unset_updates={},- )- ]-- merge_raw_person_property_updates(accumulated, new_updates)-- assert "person-1" in accumulated- assert accumulated["person-1"].set_updates["email"].value == "[email protected]"-- def test_set_newer_timestamp_replaces_older(self):- """Test that $set with newer timestamp replaces older value."""- accumulated: dict[str, RawPersonPropertyUpdates] = {- "person-1": RawPersonPropertyUpdates(- person_id="person-1",- set_updates={"email": PropertyValue(datetime(2024, 1, 15, 12, 0, 0, tzinfo=UTC), "[email protected]")},- set_once_updates={},- unset_updates={},- )- }- new_updates = [- RawPersonPropertyUpdates(- person_id="person-1",- set_updates={"email": PropertyValue(datetime(2024, 1, 15, 13, 0, 0, tzinfo=UTC), "[email protected]")},- set_once_updates={},- unset_updates={},- )- ]-- merge_raw_person_property_updates(accumulated, new_updates)-- assert accumulated["person-1"].set_updates["email"].value == "[email protected]"-- def test_set_once_earlier_timestamp_wins(self):- """Test that $set_once with earlier timestamp wins (first-writer-wins)."""- accumulated: dict[str, RawPersonPropertyUpdates] = {- "person-1": RawPersonPropertyUpdates(- person_id="person-1",- set_updates={},- set_once_updates={- "signup_source": PropertyValue(datetime(2024, 1, 15, 13, 0, 0, tzinfo=UTC), "late-source")- },- unset_updates={},- )- }- new_updates = [- RawPersonPropertyUpdates(- person_id="person-1",- set_updates={},- set_once_updates={- "signup_source": PropertyValue(datetime(2024, 1, 15, 12, 0, 0, tzinfo=UTC), "early-source")- },- unset_updates={},- )- ]-- merge_raw_person_property_updates(accumulated, new_updates)-- assert accumulated["person-1"].set_once_updates["signup_source"].value == "early-source"-- def test_set_and_unset_both_retained_for_downstream_filtering(self):- """Test that both $set and $unset are retained for downstream conflict resolution."""- accumulated: dict[str, RawPersonPropertyUpdates] = {- "person-1": RawPersonPropertyUpdates(- person_id="person-1",- set_updates={"email": PropertyValue(datetime(2024, 1, 15, 12, 0, 0, tzinfo=UTC), "[email protected]")},- set_once_updates={},- unset_updates={},- )- }- new_updates = [- RawPersonPropertyUpdates(- person_id="person-1",- set_updates={},- set_once_updates={},- unset_updates={"email": PropertyValue(datetime(2024, 1, 15, 13, 0, 0, tzinfo=UTC), None)},- )- ]-- merge_raw_person_property_updates(accumulated, new_updates)-- assert "email" in accumulated["person-1"].set_updates- assert "email" in accumulated["person-1"].unset_updates-- def test_unset_and_set_both_retained_for_downstream_filtering(self):- """Test that both $unset and $set are retained for downstream conflict resolution."""- accumulated: dict[str, RawPersonPropertyUpdates] = {- "person-1": RawPersonPropertyUpdates(- person_id="person-1",- set_updates={},- set_once_updates={},- unset_updates={"email": PropertyValue(datetime(2024, 1, 15, 12, 0, 0, tzinfo=UTC), None)},- )- }- new_updates = [- RawPersonPropertyUpdates(- person_id="person-1",- set_updates={"email": PropertyValue(datetime(2024, 1, 15, 13, 0, 0, tzinfo=UTC), "[email protected]")},- set_once_updates={},- unset_updates={},- )- ]-- merge_raw_person_property_updates(accumulated, new_updates)-- assert "email" in accumulated["person-1"].set_updates- assert "email" in accumulated["person-1"].unset_updates-- def test_multiple_persons_merged_independently(self):- """Test that multiple persons are merged independently."""- accumulated: dict[str, RawPersonPropertyUpdates] = {- "person-1": RawPersonPropertyUpdates(- person_id="person-1",- set_updates={"key1": PropertyValue(datetime(2024, 1, 15, 12, 0, 0, tzinfo=UTC), "value1")},- set_once_updates={},- unset_updates={},- )- }- new_updates = [- RawPersonPropertyUpdates(- person_id="person-1",- set_updates={"key1": PropertyValue(datetime(2024, 1, 15, 13, 0, 0, tzinfo=UTC), "value1-updated")},- set_once_updates={},- unset_updates={},- ),- RawPersonPropertyUpdates(- person_id="person-2",- set_updates={"key2": PropertyValue(datetime(2024, 1, 15, 12, 0, 0, tzinfo=UTC), "value2")},- set_once_updates={},- unset_updates={},- ),- ]-- merge_raw_person_property_updates(accumulated, new_updates)-- assert len(accumulated) == 2- assert accumulated["person-1"].set_updates["key1"].value == "value1-updated"- assert accumulated["person-2"].set_updates["key2"].value == "value2"-- def test_cross_operation_conflict_resolved_by_downstream_filter(self):- """Verify set/unset conflicts from different windows are correctly resolved downstream.-- This test documents the intentional design: merge_raw_person_property_updates- retains both operations, and filter_event_person_properties resolves conflicts.- This matches the non-windowed CH query path behavior.- """- accumulated: dict[str, RawPersonPropertyUpdates] = {}-- # Simulate window 1: set(email) at t=12:00- window1 = [- RawPersonPropertyUpdates(- person_id="person-1",- set_updates={"email": PropertyValue(datetime(2024, 1, 15, 12, 0, 0, tzinfo=UTC), "[email protected]")},- set_once_updates={},- unset_updates={},- )- ]- merge_raw_person_property_updates(accumulated, window1)-- # Simulate window 2: unset(email) at t=13:00- window2 = [- RawPersonPropertyUpdates(- person_id="person-1",- set_updates={},- set_once_updates={},- unset_updates={"email": PropertyValue(datetime(2024, 1, 15, 13, 0, 0, tzinfo=UTC), None)},- )- ]- merge_raw_person_property_updates(accumulated, window2)-- # After merge: both operations retained (no cross-map resolution in merge)- assert "email" in accumulated["person-1"].set_updates- assert "email" in accumulated["person-1"].unset_updates-- # Convert to PersonPropertyDiffs (simulating what compare_raw_updates_with_person_state returns)- diffs = [- PersonPropertyDiffs(- person_id="person-1",- person_version=1,- set_updates=accumulated["person-1"].set_updates,- set_once_updates=accumulated["person-1"].set_once_updates,- unset_updates=accumulated["person-1"].unset_updates,- )- ]-- # After filter_event_person_properties: unset wins (newer timestamp)- result = filter_event_person_properties(diffs)- assert len(result) == 1- assert "email" not in result[0].set_updates- assert "email" in result[0].unset_updates---class TestCompareRawUpdatesWithPersonState:- """Test the compare_raw_updates_with_person_state function for windowed query flow.-- This function compares merged raw event updates against current person state in ClickHouse.- It filters to only return actual differences that need to be applied.- """-- @patch("posthog.dags.person_property_reconciliation.sync_execute")- def test_empty_raw_updates_returns_empty(self, mock_sync_execute):- """Test that empty raw_updates returns empty result without querying CH."""- result = compare_raw_updates_with_person_state(team_id=1, raw_updates=[])-- assert result == []- mock_sync_execute.assert_not_called()-- @patch("posthog.dags.person_property_reconciliation.sync_execute")- def test_person_not_found_excluded_from_results(self, mock_sync_execute):- """Test that persons not found in CH are excluded from results."""- mock_sync_execute.return_value = [] # No person found-- raw_updates = [- RawPersonPropertyUpdates(- person_id="person-1",- set_updates={"email": PropertyValue(datetime(2024, 1, 15, 12, 0, 0, tzinfo=UTC), "[email protected]")},- set_once_updates={},- unset_updates={},- )- ]-- result = compare_raw_updates_with_person_state(team_id=1, raw_updates=raw_updates)-- assert result == []-- @patch("posthog.dags.person_property_reconciliation.sync_execute")- def test_set_key_exists_value_differs_included(self, mock_sync_execute):- """Test that $set is included when key exists in person AND value differs."""- mock_sync_execute.return_value = [("person-1", '{"email": "[email protected]"}', 5)]-- raw_updates = [- RawPersonPropertyUpdates(- person_id="person-1",- set_updates={"email": PropertyValue(datetime(2024, 1, 15, 12, 0, 0, tzinfo=UTC), "[email protected]")},- set_once_updates={},- unset_updates={},- )- ]-- result = compare_raw_updates_with_person_state(team_id=1, raw_updates=raw_updates)-- assert len(result) == 1- assert "email" in result[0].set_updates- assert result[0].set_updates["email"].value == "[email protected]"-- @patch("posthog.dags.person_property_reconciliation.sync_execute")- def test_set_key_exists_value_same_excluded(self, mock_sync_execute):- """Test that $set is excluded when key exists but value is the same."""- mock_sync_execute.return_value = [("person-1", '{"email": "[email protected]"}', 5)]-- raw_updates = [- RawPersonPropertyUpdates(- person_id="person-1",- set_updates={"email": PropertyValue(datetime(2024, 1, 15, 12, 0, 0, tzinfo=UTC), "[email protected]")},- set_once_updates={},- unset_updates={},- )- ]-- result = compare_raw_updates_with_person_state(team_id=1, raw_updates=raw_updates)-- assert result == []-- @patch("posthog.dags.person_property_reconciliation.sync_execute")- def test_set_key_not_exists_excluded(self, mock_sync_execute):- """Test that $set is excluded when key does NOT exist in person properties."""- mock_sync_execute.return_value = [("person-1", '{"other_key": "value"}', 5)]-- raw_updates = [- RawPersonPropertyUpdates(- person_id="person-1",- set_updates={"email": PropertyValue(datetime(2024, 1, 15, 12, 0, 0, tzinfo=UTC), "[email protected]")},- set_once_updates={},- unset_updates={},- )- ]-- result = compare_raw_updates_with_person_state(team_id=1, raw_updates=raw_updates)-- assert result == []-- @patch("posthog.dags.person_property_reconciliation.sync_execute")- def test_set_once_key_not_exists_included(self, mock_sync_execute):- """Test that $set_once is included when key does NOT exist in person."""- mock_sync_execute.return_value = [("person-1", '{"other_key": "value"}', 5)]-- raw_updates = [- RawPersonPropertyUpdates(- person_id="person-1",- set_updates={},- set_once_updates={"referrer": PropertyValue(datetime(2024, 1, 15, 12, 0, 0, tzinfo=UTC), "google.com")},- unset_updates={},- )- ]-- result = compare_raw_updates_with_person_state(team_id=1, raw_updates=raw_updates)-- assert len(result) == 1- assert "referrer" in result[0].set_once_updates- assert result[0].set_once_updates["referrer"].value == "google.com"-- @patch("posthog.dags.person_property_reconciliation.sync_execute")- def test_set_once_key_exists_excluded(self, mock_sync_execute):- """Test that $set_once is excluded when key already exists in person."""- mock_sync_execute.return_value = [("person-1", '{"referrer": "existing.com"}', 5)]-- raw_updates = [- RawPersonPropertyUpdates(- person_id="person-1",- set_updates={},- set_once_updates={"referrer": PropertyValue(datetime(2024, 1, 15, 12, 0, 0, tzinfo=UTC), "new.com")},- unset_updates={},- )- ]-- result = compare_raw_updates_with_person_state(team_id=1, raw_updates=raw_updates)-- assert result == []-- @patch("posthog.dags.person_property_reconciliation.sync_execute")- def test_unset_key_exists_included(self, mock_sync_execute):- """Test that $unset is included when key exists in person."""- mock_sync_execute.return_value = [("person-1", '{"email": "[email protected]"}', 5)]-- raw_updates = [- RawPersonPropertyUpdates(- person_id="person-1",- set_updates={},- set_once_updates={},- unset_updates={"email": PropertyValue(datetime(2024, 1, 15, 12, 0, 0, tzinfo=UTC), None)},- )- ]-- result = compare_raw_updates_with_person_state(team_id=1, raw_updates=raw_updates)-- assert len(result) == 1- assert "email" in result[0].unset_updates-- @patch("posthog.dags.person_property_reconciliation.sync_execute")- def test_unset_key_not_exists_excluded(self, mock_sync_execute):- """Test that $unset is excluded when key does NOT exist in person."""- mock_sync_execute.return_value = [("person-1", '{"other_key": "value"}', 5)]-- raw_updates = [- RawPersonPropertyUpdates(- person_id="person-1",- set_updates={},- set_once_updates={},- unset_updates={"email": PropertyValue(datetime(2024, 1, 15, 12, 0, 0, tzinfo=UTC), None)},- )- ]-- result = compare_raw_updates_with_person_state(team_id=1, raw_updates=raw_updates)-- assert result == []-- @patch("posthog.dags.person_property_reconciliation.sync_execute")- def test_multiple_persons_processed_independently(self, mock_sync_execute):- """Test that multiple persons are processed independently."""- mock_sync_execute.return_value = [- ("person-1", '{"email": "[email protected]"}', 5),- ("person-2", "{}", 3), # Empty properties- ]-- raw_updates = [- RawPersonPropertyUpdates(- person_id="person-1",- set_updates={"email": PropertyValue(datetime(2024, 1, 15, 12, 0, 0, tzinfo=UTC), "[email protected]")},- set_once_updates={},- unset_updates={},- ),- RawPersonPropertyUpdates(- person_id="person-2",- set_updates={},- set_once_updates={"source": PropertyValue(datetime(2024, 1, 15, 12, 0, 0, tzinfo=UTC), "organic")},- unset_updates={},- ),- ]-- result = compare_raw_updates_with_person_state(team_id=1, raw_updates=raw_updates)-- assert len(result) == 2- # Person 1: $set included (key exists, value differs)- p1 = next(r for r in result if r.person_id == "person-1")- assert "email" in p1.set_updates- # Person 2: $set_once included (key doesn't exist)- p2 = next(r for r in result if r.person_id == "person-2")- assert "source" in p2.set_once_updates-- @patch("posthog.dags.person_property_reconciliation.sync_execute")- def test_person_version_propagated_correctly(self, mock_sync_execute):- """Test that person_version from CH is propagated to PersonPropertyDiffs."""- mock_sync_execute.return_value = [- ("person-1", '{"email": "[email protected]"}', 42) # version=42- ]-- raw_updates = [- RawPersonPropertyUpdates(- person_id="person-1",- set_updates={"email": PropertyValue(datetime(2024, 1, 15, 12, 0, 0, tzinfo=UTC), "[email protected]")},- set_once_updates={},- unset_updates={},- )- ]-- result = compare_raw_updates_with_person_state(team_id=1, raw_updates=raw_updates)-- assert len(result) == 1- assert result[0].person_version == 42-- @patch("posthog.dags.person_property_reconciliation.sync_execute")- def test_type_coercion_equality(self, mock_sync_execute):- """Test that Python type coercion treats 123 and 123.0 as equal.-- This documents the semantic comparison behavior: the windowed path uses- Python object comparison, so numerically equivalent values like 123 (int)- and 123.0 (float) are considered equal. This differs from the non-windowed- SQL path which compares raw JSON strings.-- The Python approach is preferable for reconciliation as it avoids- unnecessary updates when values are semantically equivalent.- """- # Person has integer 123, event sets float 123.0- mock_sync_execute.return_value = [("person-1", '{"count": 123}', 5)]-- raw_updates = [- RawPersonPropertyUpdates(- person_id="person-1",- set_updates={"count": PropertyValue(datetime(2024, 1, 15, 12, 0, 0, tzinfo=UTC), 123.0)},- set_once_updates={},- unset_updates={},- )- ]-- result = compare_raw_updates_with_person_state(team_id=1, raw_updates=raw_updates)-- # 123 == 123.0 in Python, so no diff is generated- assert result == []-- @patch("posthog.dags.person_property_reconciliation.PERSON_STATE_BATCH_SIZE", 2)- @patch("posthog.dags.person_property_reconciliation.sync_execute")- def test_batches_large_person_lists(self, mock_sync_execute):- """Test that large person lists are batched to avoid query size limits.-- When there are more persons than PERSON_STATE_BATCH_SIZE, the function- should split the queries into multiple batches and aggregate results.- """- # With batch size of 2, 5 persons should result in 3 batches- mock_sync_execute.side_effect = [- # Batch 1: persons 1-2- [- ("person-1", '{"email": "[email protected]"}', 5),- ("person-2", '{"email": "[email protected]"}', 6),- ],- # Batch 2: persons 3-4- [- ("person-3", '{"email": "[email protected]"}', 7),- ("person-4", "{}", 8), # Empty properties- ],- # Batch 3: person 5- [- ("person-5", '{"email": "[email protected]"}', 9),- ],- ]-- raw_updates = [- RawPersonPropertyUpdates(- person_id=f"person-{i}",- set_updates={- "email": PropertyValue(datetime(2024, 1, 15, 12, 0, 0, tzinfo=UTC), f"new{i}@example.com")- },- set_once_updates={},- unset_updates={},- )- for i in range(1, 6)- ]-- result = compare_raw_updates_with_person_state(team_id=1, raw_updates=raw_updates)-- # Should have called sync_execute 3 times (one per batch)- assert mock_sync_execute.call_count == 3-- # Should return 4 results (person-4 has empty properties so $set is excluded)- assert len(result) == 4- result_ids = {r.person_id for r in result}- assert result_ids == {"person-1", "person-2", "person-3", "person-5"}---class TestGetPersonPropertyUpdatesWindowed:- """Test the get_person_property_updates_windowed function."""-- @patch("posthog.dags.person_property_reconciliation.get_person_property_updates_from_clickhouse")- def test_window_seconds_zero_calls_original_once(self, mock_get_updates):- """Test that window_seconds=0 calls original function once."""- mock_get_updates.return_value = [- PersonPropertyDiffs(- person_id="person-1",- person_version=1,- set_updates={"email": PropertyValue(datetime(2024, 1, 15, 12, 0, 0, tzinfo=UTC), "[email protected]")},- set_once_updates={},- unset_updates={},- )- ]-- result = get_person_property_updates_windowed(- team_id=1,- bug_window_start="2024-01-01 00:00:00",- window_seconds=0,- )-- mock_get_updates.assert_called_once_with(1, "2024-01-01 00:00:00")- assert len(result) == 1-- @patch("posthog.dags.person_property_reconciliation.get_person_property_updates_from_clickhouse")- def test_window_seconds_negative_calls_original_once(self, mock_get_updates):- """Test that negative window_seconds calls original function once."""- mock_get_updates.return_value = []-- get_person_property_updates_windowed(- team_id=1,- bug_window_start="2024-01-01 00:00:00",- window_seconds=-100,- )-- mock_get_updates.assert_called_once_with(1, "2024-01-01 00:00:00")-- @patch("posthog.dags.person_property_reconciliation.compare_raw_updates_with_person_state")- @patch("posthog.dags.person_property_reconciliation.datetime")- @patch("posthog.dags.person_property_reconciliation.get_raw_person_property_updates_from_clickhouse")- @patch("posthog.dags.person_property_reconciliation.get_affected_person_ids_from_clickhouse")- def test_window_seconds_positive_creates_windows(- self, mock_get_affected, mock_get_raw, mock_datetime, mock_compare- ):- """Test that positive window_seconds creates multiple query windows."""- mock_now = datetime(2024, 1, 1, 2, 0, 0, tzinfo=UTC)- mock_datetime.now.return_value = mock_now- mock_datetime.strptime = datetime.strptime- mock_get_affected.return_value = ["person-1"]- mock_get_raw.return_value = []- mock_compare.return_value = []-- get_person_property_updates_windowed(- team_id=1,- bug_window_start="2024-01-01 00:00:00",- window_seconds=3600, # 1 hour windows- )-- # Should first fetch affected person_ids- mock_get_affected.assert_called_once()- # Should have 2 calls: 00:00-01:00 and 01:00-02:00, with person_ids filter- assert mock_get_raw.call_count == 2- calls = mock_get_raw.call_args_list- assert calls[0][0] == (1, "2024-01-01 00:00:00", "2024-01-01 01:00:00")- assert calls[0][1] == {"person_ids": ("person-1",)}- assert calls[1][0] == (1, "2024-01-01 01:00:00", "2024-01-01 02:00:00")- assert calls[1][1] == {"person_ids": ("person-1",)}- # compare_raw_updates_with_person_state should be called once at the end- mock_compare.assert_called_once()-- @patch("posthog.dags.person_property_reconciliation.compare_raw_updates_with_person_state")- @patch("posthog.dags.person_property_reconciliation.datetime")- @patch("posthog.dags.person_property_reconciliation.get_raw_person_property_updates_from_clickhouse")- @patch("posthog.dags.person_property_reconciliation.get_affected_person_ids_from_clickhouse")- def test_window_merges_results_across_windows(self, mock_get_affected, mock_get_raw, mock_datetime, mock_compare):- """Test that results are properly merged across windows."""- mock_now = datetime(2024, 1, 1, 2, 0, 0, tzinfo=UTC)- mock_datetime.now.return_value = mock_now- mock_datetime.strptime = datetime.strptime- mock_get_affected.return_value = ["person-1"]-- # First window returns one update, second window returns a newer update for same key- mock_get_raw.side_effect = [- [- RawPersonPropertyUpdates(- person_id="person-1",- set_updates={"email": PropertyValue(datetime(2024, 1, 1, 0, 30, 0, tzinfo=UTC), "[email protected]")},- set_once_updates={},- unset_updates={},- )- ],- [- RawPersonPropertyUpdates(- person_id="person-1",- set_updates={"email": PropertyValue(datetime(2024, 1, 1, 1, 30, 0, tzinfo=UTC), "[email protected]")},- set_once_updates={},- unset_updates={},- )- ],- ]-- # Mock the comparison to return the expected result- mock_compare.return_value = [- PersonPropertyDiffs(- person_id="person-1",- person_version=1,- set_updates={"email": PropertyValue(datetime(2024, 1, 1, 1, 30, 0, tzinfo=UTC), "[email protected]")},- set_once_updates={},- unset_updates={},- )- ]-- result = get_person_property_updates_windowed(- team_id=1,- bug_window_start="2024-01-01 00:00:00",- window_seconds=3600,- )-- assert len(result) == 1- assert result[0].set_updates["email"].value == "[email protected]"-- # Verify compare was called with merged raw updates (newer email value)- compare_call_args = mock_compare.call_args[0]- assert compare_call_args[0] == 1 # team_id- raw_updates_list = compare_call_args[1]- assert len(raw_updates_list) == 1- assert raw_updates_list[0].set_updates["email"].value == "[email protected]"---class TestGetAffectedPersonIdsFromClickhouse:- """Test the get_affected_person_ids_from_clickhouse function.-- This query aggregates event properties within the bug window, compares against- current person state, and returns only person_ids with actual diffs.- """-- @patch("posthog.dags.person_property_reconciliation.sync_execute")- def test_returns_person_ids_in_bug_window(self, mock_sync_execute):- """Test that query is bounded by bug_window_end."""- mock_sync_execute.return_value = [- ("person-1",),- ("person-2",),- ("person-3",),- ]-- result = get_affected_person_ids_from_clickhouse(- team_id=1,- bug_window_start="2024-01-01 00:00:00",- bug_window_end="2024-01-02 00:00:00",- )-- assert result == ["person-1", "person-2", "person-3"]-- # Verify the query uses bug_window_end (not now())- query = mock_sync_execute.call_args[0][0]- params = mock_sync_execute.call_args[0][1]- assert "bug_window_end" in params- assert params["bug_window_end"] == "2024-01-02 00:00:00"- assert "e.timestamp < %(bug_window_end)s" in query-- @patch("posthog.dags.person_property_reconciliation.sync_execute")- def test_returns_empty_list_when_no_persons(self, mock_sync_execute):- """Test that empty list is returned when no affected persons."""- mock_sync_execute.return_value = []-- result = get_affected_person_ids_from_clickhouse(- team_id=1,- bug_window_start="2024-01-01 00:00:00",- bug_window_end="2024-01-02 00:00:00",- )-- assert result == []-- @patch("posthog.dags.person_property_reconciliation.sync_execute")- def test_filters_out_filtered_person_update_properties(self, mock_sync_execute):- """Detection query must exclude events that only set FILTERED_PERSON_UPDATE_PROPERTIES."""- mock_sync_execute.return_value = []-- get_affected_person_ids_from_clickhouse(- team_id=1,- bug_window_start="2024-01-01 00:00:00",- bug_window_end="2024-01-02 00:00:00",- )-- query = mock_sync_execute.call_args[0][0]- params = mock_sync_execute.call_args[0][1]-- assert "filtered_properties" in params- assert params["filtered_properties"] == tuple(FILTERED_PERSON_UPDATE_PROPERTIES)- assert "NOT IN %(filtered_properties)s" in query---class TestGetRawPersonPropertyUpdatesWithPersonIds:- """Test the person_ids filter parameter in get_raw_person_property_updates_from_clickhouse."""-- @patch("posthog.dags.person_property_reconciliation.sync_execute")- def test_filters_by_person_ids(self, mock_sync_execute):- """Test that query filters to specific person_ids (required param)."""- mock_sync_execute.return_value = []-- get_raw_person_property_updates_from_clickhouse(- team_id=1,- bug_window_start="2024-01-01 00:00:00",- bug_window_end="2024-01-02 00:00:00",- person_ids=("person-1", "person-2"),- )-- params = mock_sync_execute.call_args[0][1]- # person_ids is required and used directly in IN clause- assert params["person_ids"] == ("person-1", "person-2")---class TestGetPersonPropertyUpdatesWindowedBatched:- """Test the get_person_property_updates_windowed_batched iterator function."""-- @patch("posthog.dags.person_property_reconciliation.compare_raw_updates_with_person_state")- @patch("posthog.dags.person_property_reconciliation.get_raw_person_property_updates_from_clickhouse")- @patch("posthog.dags.person_property_reconciliation.datetime")- @patch("posthog.dags.person_property_reconciliation.get_affected_person_ids_from_clickhouse")- def test_yields_batches_of_persons(self, mock_get_affected, mock_datetime, mock_get_raw, mock_compare):- """Test that function yields batches when person count exceeds batch size."""- # Setup: 3 affected persons, batch size of 2 = 2 batches- mock_get_affected.return_value = ["person-1", "person-2", "person-3"]-- mock_now = datetime(2024, 1, 1, 1, 0, 0, tzinfo=UTC)- mock_datetime.now.return_value = mock_now- mock_datetime.strptime = datetime.strptime-- # Return raw updates so accumulated dict is populated- mock_get_raw.side_effect = [- # First batch, first window- [- RawPersonPropertyUpdates(- person_id="person-1",- set_updates={"key": PropertyValue(datetime(2024, 1, 1, 0, 30, 0, tzinfo=UTC), "val1")},- set_once_updates={},- unset_updates={},- )- ],- # Second batch, first window- [- RawPersonPropertyUpdates(- person_id="person-3",- set_updates={"key": PropertyValue(datetime(2024, 1, 1, 0, 30, 0, tzinfo=UTC), "val3")},- set_once_updates={},- unset_updates={},- )- ],- ]- mock_compare.side_effect = [- [ # First batch result- PersonPropertyDiffs(- person_id="person-1",- person_version=1,- set_updates={"key": PropertyValue(datetime(2024, 1, 1, 0, 30, 0, tzinfo=UTC), "val1")},- set_once_updates={},- unset_updates={},- ),- ],- [ # Second batch result- PersonPropertyDiffs(- person_id="person-3",- person_version=1,- set_updates={"key": PropertyValue(datetime(2024, 1, 1, 0, 30, 0, tzinfo=UTC), "val3")},- set_once_updates={},- unset_updates={},- ),- ],- ]-- results = list(- get_person_property_updates_windowed_batched(- team_id=1,- bug_window_start="2024-01-01 00:00:00",- bug_window_end="2024-01-01 01:00:00",- window_seconds=3600,- person_batch_size=2, # Small batch size for testing- )- )-- assert len(results) == 2 # Two batches yielded- assert len(results[0]) == 1 # First batch has 1 person with diffs- assert results[0][0].person_id == "person-1"- assert len(results[1]) == 1 # Second batch has 1 person with diffs- assert results[1][0].person_id == "person-3"-- @patch("posthog.dags.person_property_reconciliation.get_affected_person_ids_from_clickhouse")- def test_yields_nothing_when_no_affected_persons(self, mock_get_affected):- """Test that function yields nothing when no affected persons."""- mock_get_affected.return_value = []-- results = list(- get_person_property_updates_windowed_batched(- team_id=1,- bug_window_start="2024-01-01 00:00:00",- bug_window_end="2024-01-02 00:00:00",- window_seconds=3600,- )- )-- assert results == []-- @patch("posthog.dags.person_property_reconciliation.compare_raw_updates_with_person_state")- @patch("posthog.dags.person_property_reconciliation.get_raw_person_property_updates_from_clickhouse")- @patch("posthog.dags.person_property_reconciliation.datetime")- @patch("posthog.dags.person_property_reconciliation.get_affected_person_ids_from_clickhouse")- def test_filters_raw_queries_by_person_batch(self, mock_get_affected, mock_datetime, mock_get_raw, mock_compare):- """Test that windowed queries are filtered to the current person batch."""- mock_get_affected.return_value = ["person-1", "person-2", "person-3"]-- mock_now = datetime(2024, 1, 1, 2, 0, 0, tzinfo=UTC)- mock_datetime.now.return_value = mock_now- mock_datetime.strptime = datetime.strptime-- mock_get_raw.return_value = []- mock_compare.return_value = []-- # Consume the iterator with batch_size=2- list(- get_person_property_updates_windowed_batched(- team_id=1,- bug_window_start="2024-01-01 00:00:00",- bug_window_end="2024-01-01 01:00:00",- window_seconds=3600,- person_batch_size=2,- )- )-- # Check that get_raw was called with person_ids filter- # First batch should have person-1, person-2- # Second batch should have person-3- calls = mock_get_raw.call_args_list-- # There should be 4 calls total: 2 windows × 2 batches- assert len(calls) == 4-- # Check first batch calls include person_ids filter- first_batch_call = calls[0]- assert first_batch_call[1]["person_ids"] == ("person-1", "person-2")-- # Check second batch calls include person_ids filter- third_call = calls[2] # First call of second batch- assert third_call[1]["person_ids"] == ("person-3",)-- @patch("posthog.dags.person_property_reconciliation.compare_raw_updates_with_person_state")- @patch("posthog.dags.person_property_reconciliation.get_raw_person_property_updates_from_clickhouse")- @patch("posthog.dags.person_property_reconciliation.datetime")- @patch("posthog.dags.person_property_reconciliation.get_affected_person_ids_from_clickhouse")- def test_uses_bug_window_end_for_affected_persons_query(- self, mock_get_affected, mock_datetime, mock_get_raw, mock_compare- ):- """Test that affected persons query uses bug_window_end (bounded)."""- mock_get_affected.return_value = []-- list(- get_person_property_updates_windowed_batched(- team_id=1,- bug_window_start="2024-01-01 00:00:00",- bug_window_end="2024-01-02 00:00:00", # Specific end time- window_seconds=3600,- )- )-- # Verify get_affected_person_ids_from_clickhouse was called with bug_window_end- mock_get_affected.assert_called_once_with(- team_id=1,- bug_window_start="2024-01-01 00:00:00",- bug_window_end="2024-01-02 00:00:00",- )-- @patch("posthog.dags.person_property_reconciliation.get_affected_person_ids_from_clickhouse")- def test_logs_error_when_affected_persons_query_fails(self, mock_get_affected):- """Test that errors during affected persons query are logged with context."""- mock_get_affected.side_effect = Exception("ClickHouse connection failed")- mock_logger = MagicMock()-- with pytest.raises(Exception, match="ClickHouse connection failed"):- list(- get_person_property_updates_windowed_batched(- team_id=1,- bug_window_start="2024-01-01 00:00:00",- bug_window_end="2024-01-02 00:00:00",- window_seconds=3600,- logger=mock_logger,- )- )-- # Verify error was logged with context- mock_logger.exception.assert_called()- error_call = mock_logger.exception.call_args[0][0]- assert "team_id=1" in error_call- assert "Failed to query affected persons" in error_call-- @patch("posthog.dags.person_property_reconciliation.get_raw_person_property_updates_from_clickhouse")- @patch("posthog.dags.person_property_reconciliation.datetime")- @patch("posthog.dags.person_property_reconciliation.get_affected_person_ids_from_clickhouse")- def test_logs_error_when_window_query_fails(self, mock_get_affected, mock_datetime, mock_get_raw):- """Test that errors during window queries are logged with batch and window context."""- mock_get_affected.return_value = ["person-1"]-- mock_now = datetime(2024, 1, 1, 2, 0, 0, tzinfo=UTC)- mock_datetime.now.return_value = mock_now- mock_datetime.strptime = datetime.strptime-- # First window succeeds, second window fails- mock_get_raw.side_effect = [- [], # First window OK- Exception("Query timeout"), # Second window fails- ]- mock_logger = MagicMock()-- with pytest.raises(Exception, match="Query timeout"):- list(- get_person_property_updates_windowed_batched(- team_id=1,- bug_window_start="2024-01-01 00:00:00",- bug_window_end="2024-01-01 01:00:00",- window_seconds=3600,- logger=mock_logger,- )- )-- # Verify error was logged with window context- error_calls = list(mock_logger.exception.call_args_list)- assert len(error_calls) >= 1- # Should have both the inner window error and the outer batch error- error_messages = [c[0][0] for c in error_calls]- assert any("failed to query window" in msg for msg in error_messages)- assert any("Batch 1" in msg for msg in error_messages)- assert any("team_id=1" in msg for msg in error_messages)-- @patch("posthog.dags.person_property_reconciliation.compare_raw_updates_with_person_state")- @patch("posthog.dags.person_property_reconciliation.get_raw_person_property_updates_from_clickhouse")- @patch("posthog.dags.person_property_reconciliation.datetime")- @patch("posthog.dags.person_property_reconciliation.get_affected_person_ids_from_clickhouse")- def test_logs_error_when_compare_fails(self, mock_get_affected, mock_datetime, mock_get_raw, mock_compare):- """Test that errors during person state comparison are logged with context."""- mock_get_affected.return_value = ["person-1"]-- mock_now = datetime(2024, 1, 1, 1, 0, 0, tzinfo=UTC)- mock_datetime.now.return_value = mock_now- mock_datetime.strptime = datetime.strptime-- mock_get_raw.return_value = [- RawPersonPropertyUpdates(- person_id="person-1",- set_updates={"key": PropertyValue(datetime(2024, 1, 1, 0, 30, 0, tzinfo=UTC), "val")},- set_once_updates={},- unset_updates={},- )- ]- mock_compare.side_effect = Exception("Person table query failed")- mock_logger = MagicMock()-- with pytest.raises(Exception, match="Person table query failed"):- list(- get_person_property_updates_windowed_batched(- team_id=1,- bug_window_start="2024-01-01 00:00:00",- bug_window_end="2024-01-01 01:00:00",- window_seconds=3600,- logger=mock_logger,- )- )-- # Verify error was logged with comparison context- error_calls = [c[0][0] for c in mock_logger.exception.call_args_list]- assert any("failed to compare" in msg for msg in error_calls)- assert any("1 persons" in msg for msg in error_calls)-- @patch("posthog.dags.person_property_reconciliation.compare_raw_updates_with_person_state")- @patch("posthog.dags.person_property_reconciliation.get_raw_person_property_updates_from_clickhouse")- @patch("posthog.dags.person_property_reconciliation.datetime")- @patch("posthog.dags.person_property_reconciliation.get_affected_person_ids_from_clickhouse")- def test_yields_nothing_when_compare_returns_empty(- self, mock_get_affected, mock_datetime, mock_get_raw, mock_compare- ):- """Test that batch yields nothing when compare finds no actual diffs."""- mock_get_affected.return_value = ["person-1"]-- mock_now = datetime(2024, 1, 1, 1, 0, 0, tzinfo=UTC)- mock_datetime.now.return_value = mock_now- mock_datetime.strptime = datetime.strptime-- # Raw updates exist but compare finds no actual diffs (person state matches)- mock_get_raw.return_value = [- RawPersonPropertyUpdates(- person_id="person-1",- set_updates={"key": PropertyValue(datetime(2024, 1, 1, 0, 30, 0, tzinfo=UTC), "val")},- set_once_updates={},- unset_updates={},- )- ]- mock_compare.return_value = [] # No diffs after comparison-- results = list(- get_person_property_updates_windowed_batched(- team_id=1,- bug_window_start="2024-01-01 00:00:00",- bug_window_end="2024-01-01 01:00:00",- window_seconds=3600,- )- )-- assert results == []-- @patch("posthog.dags.person_property_reconciliation.compare_raw_updates_with_person_state")- @patch("posthog.dags.person_property_reconciliation.get_raw_person_property_updates_from_clickhouse")- @patch("posthog.dags.person_property_reconciliation.datetime")- @patch("posthog.dags.person_property_reconciliation.get_affected_person_ids_from_clickhouse")- def test_merges_across_windows_correctly(self, mock_get_affected, mock_datetime, mock_get_raw, mock_compare):- """Test that $set takes newest timestamp and $set_once takes earliest when merging windows."""- mock_get_affected.return_value = ["person-1"]-- mock_now = datetime(2024, 1, 1, 2, 0, 0, tzinfo=UTC)- mock_datetime.now.return_value = mock_now- mock_datetime.strptime = datetime.strptime-- # First window: older set value, older set_once value- # Second window: newer set value, newer set_once value (should be ignored for set_once)- mock_get_raw.side_effect = [- [- RawPersonPropertyUpdates(- person_id="person-1",- set_updates={"prop": PropertyValue(datetime(2024, 1, 1, 0, 30, 0, tzinfo=UTC), "old_val")},- set_once_updates={- "first_prop": PropertyValue(datetime(2024, 1, 1, 0, 15, 0, tzinfo=UTC), "first_val")- },- unset_updates={},- )- ],- [- RawPersonPropertyUpdates(- person_id="person-1",- set_updates={"prop": PropertyValue(datetime(2024, 1, 1, 1, 30, 0, tzinfo=UTC), "new_val")},- set_once_updates={- "first_prop": PropertyValue(datetime(2024, 1, 1, 1, 15, 0, tzinfo=UTC), "ignored_val")- },- unset_updates={},- )- ],- ]-- # Capture what compare receives- captured_raw_updates = []-- def capture_compare(team_id, raw_updates):- captured_raw_updates.extend(raw_updates)- return []-- mock_compare.side_effect = capture_compare-- list(- get_person_property_updates_windowed_batched(- team_id=1,- bug_window_start="2024-01-01 00:00:00",- bug_window_end="2024-01-01 01:00:00",- window_seconds=3600,- )- )-- # Verify merge semantics: $set takes newest, $set_once takes earliest- assert len(captured_raw_updates) == 1- merged = captured_raw_updates[0]- assert merged.set_updates["prop"].value == "new_val" # Newer timestamp wins- assert merged.set_once_updates["first_prop"].value == "first_val" # Earlier timestamp wins---class TestParseFormatChTimestamp:- """Test the parse_ch_timestamp and format_ch_timestamp helper functions."""-- def test_parse_ch_timestamp(self):- """Test parsing ClickHouse timestamp string to datetime."""- result = parse_ch_timestamp("2024-01-15 12:30:45")- assert result == datetime(2024, 1, 15, 12, 30, 45, tzinfo=UTC)- assert result.tzinfo == UTC-- def test_format_ch_timestamp(self):- """Test formatting datetime to ClickHouse timestamp string."""- dt = datetime(2024, 1, 15, 12, 30, 45, tzinfo=UTC)- result = format_ch_timestamp(dt)- assert result == "2024-01-15 12:30:45"-- def test_roundtrip(self):- """Test that parse and format are inverse operations."""- original = "2024-06-20 08:15:30"- parsed = parse_ch_timestamp(original)- formatted = format_ch_timestamp(parsed)- assert formatted == original-- def test_format_ch_timestamp_converts_to_utc(self):- """Test that non-UTC datetime is converted to UTC before formatting."""- from datetime import timedelta, timezone-- est = timezone(timedelta(hours=-5))- dt_est = datetime(2024, 1, 15, 12, 30, 45, tzinfo=est) # 12:30 EST = 17:30 UTC- result = format_ch_timestamp(dt_est)- assert result == "2024-01-15 17:30:45"---class TestPropertyValueDataclass:- """Test the PropertyValue dataclass structure."""-- def test_property_value_fields(self):- """Test that PropertyValue has expected fields."""- pv = PropertyValue(- timestamp=datetime(2024, 1, 15, 12, 0, 0),- value="test_value",- )-- assert pv.timestamp == datetime(2024, 1, 15, 12, 0, 0)- assert pv.value == "test_value"-- def test_property_value_accepts_any_value_type(self):- """Test that value field accepts various types."""- # String- pv1 = PropertyValue(timestamp=datetime.now(), value="string")- assert pv1.value == "string"-- # Number- pv2 = PropertyValue(timestamp=datetime.now(), value=123)- assert pv2.value == 123-- # Boolean- pv3 = PropertyValue(timestamp=datetime.now(), value=True)- assert pv3.value is True-- # None (for unset)- pv4 = PropertyValue(timestamp=datetime.now(), value=None)- assert pv4.value is None---class TestPersonPropertyDiffsDataclass:- """Test the PersonPropertyDiffs dataclass structure."""-- def test_person_property_diffs_fields(self):- """Test that PersonPropertyDiffs has expected fields."""- person_diffs = PersonPropertyDiffs(- person_id="018d1234-5678-0000-0000-000000000001",- person_version=1,- set_updates={"email": PropertyValue(timestamp=datetime.now(), value="[email protected]")},- set_once_updates={"referrer": PropertyValue(timestamp=datetime.now(), value="google.com")},- unset_updates={"old_field": PropertyValue(timestamp=datetime.now(), value=None)},- )-- assert person_diffs.person_id == "018d1234-5678-0000-0000-000000000001"- assert person_diffs.person_version == 1- assert len(person_diffs.set_updates) == 1- assert len(person_diffs.set_once_updates) == 1- assert len(person_diffs.unset_updates) == 1- assert person_diffs.set_updates["email"].value == "[email protected]"- assert person_diffs.set_once_updates["referrer"].value == "google.com"- assert person_diffs.unset_updates["old_field"].value is None---class TestClickHouseQueryIntegration:- """Integration tests that insert data into ClickHouse and run the actual query."""-- @pytest.fixture(scope="class")- def cluster(self, django_db_setup):- with isolated_clickhouse_cluster() as clickhouse_cluster:- yield clickhouse_cluster-- def test_unset_uses_latest_timestamp_regression(self, cluster: ClickhouseCluster):- """- Regression test: $unset should use max(timestamp), not min(timestamp).-- If a person has two $unset operations on the same key at different times,- the returned timestamp should be the latest one (max), not the earliest (min).- This was a bug where $unset incorrectly used min(e.timestamp).- """-- team_id = 99901- person_id = UUID("11111111-1111-1111-1111-000000000001")- # Use naive datetimes since ClickHouse returns naive datetimes- now = datetime.now().replace(microsecond=0)- bug_window_start = now - timedelta(days=10)-- # Two timestamps for the same $unset operation- earlier_ts = now - timedelta(days=5)- later_ts = now - timedelta(days=2)-- # Insert events with $unset at different timestamps- events = [- # First $unset at earlier timestamp- (- team_id,- "distinct_id_1",- person_id,- earlier_ts,- json.dumps({"$unset": ["email"]}),- ),- # Second $unset at later timestamp (should be the winner)- (- team_id,- "distinct_id_1",- person_id,- later_ts,- json.dumps({"$unset": ["email"]}),- ),- ]-- def insert_events(client: Client) -> None:- client.execute(- """INSERT INTO writable_events (team_id, distinct_id, person_id, timestamp, properties)- VALUES""",- events,- )-- cluster.any_host(insert_events).result()-- # Insert person with the "email" property (so $unset has something to unset)- person_data = [- (- team_id,- person_id,- json.dumps({"email": "[email protected]", "name": "Test User"}),- 1, # version- now - timedelta(days=8), # _timestamp within bug window- )- ]-- def insert_person(client: Client) -> None:- client.execute(- """INSERT INTO person (team_id, id, properties, version, _timestamp)- VALUES""",- person_data,- )-- cluster.any_host(insert_person).result()-- # Run the actual query- results = get_person_property_updates_from_clickhouse(- team_id=team_id,- bug_window_start=bug_window_start.strftime("%Y-%m-%d %H:%M:%S"),- )-- assert len(results) == 1- person_diffs = results[0]- assert str(person_diffs.person_id) == str(person_id)-- # Find the unset operation- assert len(person_diffs.unset_updates) == 1- assert "email" in person_diffs.unset_updates- unset_update = person_diffs.unset_updates["email"]-- assert unset_update.value is None-- # Regression check: timestamp should be the LATER one (max), not earlier (min)- # Allow 1 second tolerance for timestamp comparison- # Strip timezone for comparison since CH returns tz-aware but we created naive timestamps- result_ts = unset_update.timestamp.replace(tzinfo=None)- assert abs((result_ts - later_ts).total_seconds()) < 1, (- f"$unset should use max(timestamp). "- f"Expected ~{later_ts}, got {result_ts}. "- f"Earlier timestamp was {earlier_ts}."- )-- def test_set_uses_latest_timestamp(self, cluster: ClickhouseCluster):- """$set should use max(timestamp) - latest value wins."""-- team_id = 99902- person_id = UUID("22222222-2222-2222-2222-000000000002")- # Use naive datetimes since ClickHouse returns naive datetimes- now = datetime.now().replace(microsecond=0)- bug_window_start = now - timedelta(days=10)-- earlier_ts = now - timedelta(days=5)- later_ts = now - timedelta(days=2)-- events = [- (team_id, "distinct_id_2", person_id, earlier_ts, json.dumps({"$set": {"name": "First Name"}})),- (team_id, "distinct_id_2", person_id, later_ts, json.dumps({"$set": {"name": "Latest Name"}})),- ]-- def insert_events(client: Client) -> None:- client.execute(- """INSERT INTO writable_events (team_id, distinct_id, person_id, timestamp, properties)- VALUES""",- events,- )-- cluster.any_host(insert_events).result()-- # Person with different value for "name" so set_diff includes it- person_data = [(team_id, person_id, json.dumps({"name": "Old Name"}), 1, now - timedelta(days=8))]-- def insert_person(client: Client) -> None:- client.execute(- """INSERT INTO person (team_id, id, properties, version, _timestamp)- VALUES""",- person_data,- )-- cluster.any_host(insert_person).result()-- results = get_person_property_updates_from_clickhouse(- team_id=team_id,- bug_window_start=bug_window_start.strftime("%Y-%m-%d %H:%M:%S"),- )-- assert len(results) == 1- person_diffs = results[0]- assert len(person_diffs.set_updates) == 1- assert "name" in person_diffs.set_updates-- # Should have latest value and latest timestamp- assert person_diffs.set_updates["name"].value == "Latest Name"- result_ts = person_diffs.set_updates["name"].timestamp.replace(tzinfo=None)- assert abs((result_ts - later_ts).total_seconds()) < 1-- def test_set_once_uses_earliest_timestamp(self, cluster: ClickhouseCluster):- """$set_once should use min(timestamp) - first value wins."""-- team_id = 99903- person_id = UUID("33333333-3333-3333-3333-000000000003")- # Use naive datetimes since ClickHouse returns naive datetimes- now = datetime.now().replace(microsecond=0)- bug_window_start = now - timedelta(days=10)-- earlier_ts = now - timedelta(days=5)- later_ts = now - timedelta(days=2)-- events = [- (team_id, "distinct_id_3", person_id, earlier_ts, json.dumps({"$set_once": {"referrer": "google.com"}})),- (team_id, "distinct_id_3", person_id, later_ts, json.dumps({"$set_once": {"referrer": "facebook.com"}})),- ]-- def insert_events(client: Client) -> None:- client.execute(- """INSERT INTO writable_events (team_id, distinct_id, person_id, timestamp, properties)- VALUES""",- events,- )-- cluster.any_host(insert_events).result()-- # Person WITHOUT "referrer" so set_once_diff includes it- person_data = [(team_id, person_id, json.dumps({"other_prop": "value"}), 1, now - timedelta(days=8))]-- def insert_person(client: Client) -> None:- client.execute(- """INSERT INTO person (team_id, id, properties, version, _timestamp)- VALUES""",- person_data,- )-- cluster.any_host(insert_person).result()-- results = get_person_property_updates_from_clickhouse(- team_id=team_id,- bug_window_start=bug_window_start.strftime("%Y-%m-%d %H:%M:%S"),- )-- assert len(results) == 1- person_diffs = results[0]- assert len(person_diffs.set_once_updates) == 1- assert "referrer" in person_diffs.set_once_updates-- # Should have first value and earliest timestamp- assert person_diffs.set_once_updates["referrer"].value == "google.com"- result_ts = person_diffs.set_once_updates["referrer"].timestamp.replace(tzinfo=None)- assert abs((result_ts - earlier_ts).total_seconds()) < 1-- def test_multiple_operations_same_key_in_batch(self, cluster: ClickhouseCluster):- """- Test multiple operation types on same key: $set then $unset.-- Both operations are for the same key but different operation types.- Each should be returned separately.- """-- team_id = 99904- person_id = UUID("44444444-4444-4444-4444-000000000004")- # Use naive datetimes since ClickHouse returns naive datetimes- now = datetime.now().replace(microsecond=0)- bug_window_start = now - timedelta(days=10)-- set_ts = now - timedelta(days=5)- unset_ts = now - timedelta(days=2)-- events = [- # First $set the property- (team_id, "distinct_id_4", person_id, set_ts, json.dumps({"$set": {"email": "[email protected]"}})),- # Then $unset the same property (later)- (team_id, "distinct_id_4", person_id, unset_ts, json.dumps({"$unset": ["email"]})),- ]-- def insert_events(client: Client) -> None:- client.execute(- """INSERT INTO writable_events (team_id, distinct_id, person_id, timestamp, properties)- VALUES""",- events,- )-- cluster.any_host(insert_events).result()-- # Person with existing email (different value so $set is in diff, and exists so $unset is in diff)- person_data = [(team_id, person_id, json.dumps({"email": "[email protected]"}), 1, now - timedelta(days=8))]-- def insert_person(client: Client) -> None:- client.execute(- """INSERT INTO person (team_id, id, properties, version, _timestamp)- VALUES""",- person_data,- )-- cluster.any_host(insert_person).result()-- results = get_person_property_updates_from_clickhouse(- team_id=team_id,- bug_window_start=bug_window_start.strftime("%Y-%m-%d %H:%M:%S"),- )-- assert len(results) == 1- person_diffs = results[0]-- # Both should be returned - $set with value, $unset without- assert len(person_diffs.set_updates) == 1- assert "email" in person_diffs.set_updates- assert person_diffs.set_updates["email"].value == "[email protected]"-- assert len(person_diffs.unset_updates) == 1- assert "email" in person_diffs.unset_updates- assert person_diffs.unset_updates["email"].value is None-- def test_mixed_operations_different_keys(self, cluster: ClickhouseCluster):- """Test $set, $set_once, and $unset on different keys in same event."""-- team_id = 99905- person_id = UUID("55555555-5555-5555-5555-000000000005")- # Use naive datetimes since ClickHouse returns naive datetimes- now = datetime.now().replace(microsecond=0)- bug_window_start = now - timedelta(days=10)-- event_ts = now - timedelta(days=3)-- events = [- (- team_id,- "distinct_id_5",- person_id,- event_ts,- json.dumps(- {- "$set": {"name": "New Name"},- "$set_once": {"first_visit": "2024-01-01"},- "$unset": ["deprecated_field"],- }- ),- ),- ]-- def insert_events(client: Client) -> None:- client.execute(- """INSERT INTO writable_events (team_id, distinct_id, person_id, timestamp, properties)- VALUES""",- events,- )-- cluster.any_host(insert_events).result()-- # Person with name (different), no first_visit, has deprecated_field- person_data = [- (- team_id,- person_id,- json.dumps({"name": "Old Name", "deprecated_field": "to_remove"}),- 1,- now - timedelta(days=8),- )- ]-- def insert_person(client: Client) -> None:- client.execute(- """INSERT INTO person (team_id, id, properties, version, _timestamp)- VALUES""",- person_data,- )-- cluster.any_host(insert_person).result()-- results = get_person_property_updates_from_clickhouse(- team_id=team_id,- bug_window_start=bug_window_start.strftime("%Y-%m-%d %H:%M:%S"),- )-- assert len(results) == 1- person_diffs = results[0]-- assert len(person_diffs.set_updates) == 1- assert "name" in person_diffs.set_updates- assert person_diffs.set_updates["name"].value == "New Name"-- assert len(person_diffs.set_once_updates) == 1- assert "first_visit" in person_diffs.set_once_updates- assert person_diffs.set_once_updates["first_visit"].value == "2024-01-01"-- assert len(person_diffs.unset_updates) == 1- assert "deprecated_field" in person_diffs.unset_updates- assert person_diffs.unset_updates["deprecated_field"].value is None-- def test_set_with_various_json_types(self, cluster: ClickhouseCluster):- """Test $set with various JSON value types: string, number, boolean, null, array, object."""- team_id = 99906- person_id = UUID("66666666-6666-6666-6666-000000000006")- now = datetime.now().replace(microsecond=0)- bug_window_start = now - timedelta(days=10)-- event_ts = now - timedelta(days=3)-- # Event with various JSON types in $set- # Note: null values are filtered out in the query (use $unset for removal)- events = [- (- team_id,- "distinct_id_6",- person_id,- event_ts,- json.dumps(- {- "$set": {- "string_prop": "hello world",- "int_prop": 42,- "float_prop": 3.14159,- "bool_true_prop": True,- "bool_false_prop": False,- "array_prop": [1, "two", True],- "object_prop": {"nested": "value", "count": 123},- }- }- ),- ),- ]-- def insert_events(client):- client.execute(- """INSERT INTO writable_events (team_id, distinct_id, person_id, timestamp, properties)- VALUES""",- events,- )-- cluster.any_host(insert_events).result()-- # Person with different values for all properties (so set_diff includes them)- person_data = [- (- team_id,- person_id,- json.dumps(- {- "string_prop": "old",- "int_prop": 0,- "float_prop": 0.0,- "bool_true_prop": False,- "bool_false_prop": True,- "array_prop": [],- "object_prop": {},- }- ),- 1,- now - timedelta(days=8),- )- ]-- def insert_person(client):- client.execute(- """INSERT INTO person (team_id, id, properties, version, _timestamp)- VALUES""",- person_data,- )-- cluster.any_host(insert_person).result()-- results = get_person_property_updates_from_clickhouse(- team_id=team_id,- bug_window_start=bug_window_start.strftime("%Y-%m-%d %H:%M:%S"),- )-- assert len(results) == 1- updates = {key: pv.value for key, pv in results[0].set_updates.items()}-- # String- assert updates["string_prop"] == "hello world"- assert isinstance(updates["string_prop"], str)-- # Integer- assert updates["int_prop"] == 42- assert isinstance(updates["int_prop"], int)-- # Float- assert updates["float_prop"] == 3.14159- assert isinstance(updates["float_prop"], float)-- # Boolean true- assert updates["bool_true_prop"] is True- assert isinstance(updates["bool_true_prop"], bool)-- # Boolean false- assert updates["bool_false_prop"] is False- assert isinstance(updates["bool_false_prop"], bool)-- # Array- assert updates["array_prop"] == [1, "two", True]- assert isinstance(updates["array_prop"], list)-- # Object- assert updates["object_prop"] == {"nested": "value", "count": 123}- assert isinstance(updates["object_prop"], dict)-- def test_set_with_null_value_is_filtered_out(self, cluster: ClickhouseCluster):- """Test that $set with null value is filtered out (use $unset for removal)."""- team_id = 99907- person_id = UUID("77777777-7777-7777-7777-000000000007")- now = datetime.now().replace(microsecond=0)- bug_window_start = now - timedelta(days=10)-- event_ts = now - timedelta(days=3)-- # Event with null value in $set - should be filtered out- events = [- (- team_id,- "distinct_id_7",- person_id,- event_ts,- json.dumps(- {- "$set": {- "keep_prop": "value",- "null_prop": None, # This should be filtered out- }- }- ),- ),- ]-- def insert_events(client):- client.execute(- """INSERT INTO writable_events (team_id, distinct_id, person_id, timestamp, properties)- VALUES""",- events,- )-- cluster.any_host(insert_events).result()-- # Person with different values (so set_diff includes them)- person_data = [- (- team_id,- person_id,- json.dumps(- {- "keep_prop": "old_value",- "null_prop": "existing_value", # This exists, but $set null should NOT update it- }- ),- 1,- now - timedelta(days=8),- )- ]-- def insert_person(client):- client.execute(- """INSERT INTO person (team_id, id, properties, version, _timestamp)- VALUES""",- person_data,- )-- cluster.any_host(insert_person).result()-- results = get_person_property_updates_from_clickhouse(- team_id=team_id,- bug_window_start=bug_window_start.strftime("%Y-%m-%d %H:%M:%S"),- )-- assert len(results) == 1- updates = {key: pv.value for key, pv in results[0].set_updates.items()}-- # keep_prop should be included- assert "keep_prop" in updates- assert updates["keep_prop"] == "value"-- # null_prop should NOT be included (filtered out because value is null)- assert "null_prop" not in updates, (- "$set with null value should be filtered out. Use $unset to remove properties."- )-- def test_set_once_with_various_json_types(self, cluster: ClickhouseCluster):- """Test $set_once with various JSON value types: string, number, boolean, array, object."""- team_id = 99908- person_id = UUID("88888888-8888-8888-8888-000000000008")- now = datetime.now().replace(microsecond=0)- bug_window_start = now - timedelta(days=10)-- event_ts = now - timedelta(days=3)-- # Event with various JSON types in $set_once- # Note: null values are filtered out in the query- events = [- (- team_id,- "distinct_id_8",- person_id,- event_ts,- json.dumps(- {- "$set_once": {- "string_prop": "hello world",- "int_prop": 42,- "float_prop": 3.14159,- "bool_true_prop": True,- "bool_false_prop": False,- "array_prop": [1, "two", True],- "object_prop": {"nested": "value", "count": 123},- }- }- ),- ),- ]-- def insert_events(client):- client.execute(- """INSERT INTO writable_events (team_id, distinct_id, person_id, timestamp, properties)- VALUES""",- events,- )-- cluster.any_host(insert_events).result()-- # Person WITHOUT these properties (so set_once_diff includes them)- person_data = [- (- team_id,- person_id,- json.dumps({"other_prop": "value"}), # None of the $set_once keys exist- 1,- now - timedelta(days=8),- )- ]-- def insert_person(client):- client.execute(- """INSERT INTO person (team_id, id, properties, version, _timestamp)- VALUES""",- person_data,- )-- cluster.any_host(insert_person).result()-- results = get_person_property_updates_from_clickhouse(- team_id=team_id,- bug_window_start=bug_window_start.strftime("%Y-%m-%d %H:%M:%S"),- )-- assert len(results) == 1- updates = {key: pv.value for key, pv in results[0].set_once_updates.items()}-- # String- assert updates["string_prop"] == "hello world"- assert isinstance(updates["string_prop"], str)-- # Integer- assert updates["int_prop"] == 42- assert isinstance(updates["int_prop"], int)-- # Float- assert updates["float_prop"] == 3.14159- assert isinstance(updates["float_prop"], float)-- # Boolean true- assert updates["bool_true_prop"] is True- assert isinstance(updates["bool_true_prop"], bool)-- # Boolean false- assert updates["bool_false_prop"] is False- assert isinstance(updates["bool_false_prop"], bool)-- # Array- assert updates["array_prop"] == [1, "two", True]- assert isinstance(updates["array_prop"], list)-- # Object- assert updates["object_prop"] == {"nested": "value", "count": 123}- assert isinstance(updates["object_prop"], dict)-- def test_set_once_with_null_value_is_filtered_out(self, cluster: ClickhouseCluster):- """Test that $set_once with null value is filtered out."""- team_id = 99909- person_id = UUID("99999999-9999-9999-9999-000000000009")- now = datetime.now().replace(microsecond=0)- bug_window_start = now - timedelta(days=10)-- event_ts = now - timedelta(days=3)-- # Event with null value in $set_once - should be filtered out- events = [- (- team_id,- "distinct_id_9",- person_id,- event_ts,- json.dumps(- {- "$set_once": {- "keep_prop": "value",- "null_prop": None, # This should be filtered out- }- }- ),- ),- ]-- def insert_events(client):- client.execute(- """INSERT INTO writable_events (team_id, distinct_id, person_id, timestamp, properties)- VALUES""",- events,- )-- cluster.any_host(insert_events).result()-- # Person without these properties (so set_once_diff would include them if not filtered)- person_data = [- (- team_id,- person_id,- json.dumps({"other_prop": "value"}),- 1,- now - timedelta(days=8),- )- ]-- def insert_person(client):- client.execute(- """INSERT INTO person (team_id, id, properties, version, _timestamp)- VALUES""",- person_data,- )-- cluster.any_host(insert_person).result()-- results = get_person_property_updates_from_clickhouse(- team_id=team_id,- bug_window_start=bug_window_start.strftime("%Y-%m-%d %H:%M:%S"),- )-- assert len(results) == 1- updates = {key: pv.value for key, pv in results[0].set_once_updates.items()}-- # keep_prop should be included- assert "keep_prop" in updates- assert updates["keep_prop"] == "value"-- # null_prop should NOT be included (filtered out because value is null)- assert "null_prop" not in updates, "$set_once with null value should be filtered out."-- # ==================== Person Merge Tests ====================-- def test_person_merge_uses_override_person_id(self, cluster: ClickhouseCluster):- """- When a distinct_id has an override in person_distinct_id_overrides,- events should be attributed to the override person_id, not the original.-- This test verifies that:- 1. Events with an overridden distinct_id are attributed to the merged person- 2. The $set property update is correctly associated with the merged person- """- team_id = 99910- original_person_id = UUID("aaaa0000-0000-0000-0000-000000000001")- merged_person_id = UUID("aaaa0000-0000-0000-0000-000000000002")- now = datetime.now().replace(microsecond=0)- bug_window_start = now - timedelta(days=10)-- event_ts = now - timedelta(days=3)-- # Event with original person_id, but distinct_id will be overridden- # Using $set on a property that EXISTS in the merged person but has DIFFERENT value- events = [- (- team_id,- "merged_distinct_id",- original_person_id, # Original person_id in event- event_ts,- json.dumps({"$set": {"existing": "new_value"}}), # Update existing property- ),- ]-- def insert_events(client):- client.execute(- """INSERT INTO writable_events (team_id, distinct_id, person_id, timestamp, properties)- VALUES""",- events,- )-- cluster.any_host(insert_events).result()-- # Insert override: merged_distinct_id -> merged_person_id- # Must include _timestamp for the data to be visible- override_data = [- (- team_id,- "merged_distinct_id",- merged_person_id, # Override to this person- now - timedelta(days=4), # _timestamp - must be provided- 1, # version- 0, # is_deleted = false- ),- ]-- def insert_override(client):- client.execute(- """INSERT INTO person_distinct_id_overrides- (team_id, distinct_id, person_id, _timestamp, version, is_deleted)- VALUES""",- override_data,- )-- cluster.any_host(insert_override).result()-- # Insert the MERGED person (not the original)- # Has "existing" property with old value that event will update- person_data = [- (- team_id,- merged_person_id,- json.dumps({"existing": "old_value"}),- 1,- now - timedelta(days=8),- )- ]-- def insert_person(client):- client.execute(- """INSERT INTO person (team_id, id, properties, version, _timestamp)- VALUES""",- person_data,- )-- cluster.any_host(insert_person).result()-- results = get_person_property_updates_from_clickhouse(- team_id=team_id,- bug_window_start=bug_window_start.strftime("%Y-%m-%d %H:%M:%S"),- )-- # Should get results for merged_person_id, not original_person_id- assert len(results) == 1- assert results[0].person_id == str(merged_person_id)-- updates = {key: pv.value for key, pv in results[0].set_updates.items()}- assert "existing" in updates- assert updates["existing"] == "new_value"-- def test_multiple_distinct_ids_same_person(self, cluster: ClickhouseCluster):- """- Events from multiple distinct_ids that map to the same person- should be aggregated together.- """- team_id = 99911- person_id = UUID("bbbb0000-0000-0000-0000-000000000001")- now = datetime.now().replace(microsecond=0)- bug_window_start = now - timedelta(days=10)-- earlier_ts = now - timedelta(days=5)- later_ts = now - timedelta(days=2)-- # Two events from different distinct_ids, same person- events = [- (team_id, "distinct_id_A", person_id, earlier_ts, json.dumps({"$set": {"name": "First"}})),- (team_id, "distinct_id_B", person_id, later_ts, json.dumps({"$set": {"name": "Latest"}})),- ]-- def insert_events(client):- client.execute(- """INSERT INTO writable_events (team_id, distinct_id, person_id, timestamp, properties)- VALUES""",- events,- )-- cluster.any_host(insert_events).result()-- person_data = [(team_id, person_id, json.dumps({"name": "Old"}), 1, now - timedelta(days=8))]-- def insert_person(client):- client.execute(- """INSERT INTO person (team_id, id, properties, version, _timestamp)- VALUES""",- person_data,- )-- cluster.any_host(insert_person).result()-- results = get_person_property_updates_from_clickhouse(- team_id=team_id,- bug_window_start=bug_window_start.strftime("%Y-%m-%d %H:%M:%S"),- )-- # Should have one result with the latest value (aggregated from both distinct_ids)- assert len(results) == 1- updates = {key: pv.value for key, pv in results[0].set_updates.items()}- assert updates["name"] == "Latest"-- def test_deleted_override_not_used(self, cluster: ClickhouseCluster):- """- When an override has is_deleted=1, it should not be used.- Events should use the original person_id from the event.- """- team_id = 99920- original_person_id = UUID("eeee0000-0000-0000-0000-000000000001")- merged_person_id = UUID("eeee0000-0000-0000-0000-000000000002")- now = datetime.now().replace(microsecond=0)- bug_window_start = now - timedelta(days=10)-- event_ts = now - timedelta(days=3)-- events = [- (- team_id,- "deleted_override_distinct",- original_person_id,- event_ts,- json.dumps({"$set": {"prop": "value"}}),- ),- ]-- def insert_events(client):- client.execute(- """INSERT INTO writable_events (team_id, distinct_id, person_id, timestamp, properties)- VALUES""",- events,- )-- cluster.any_host(insert_events).result()-- # Insert a DELETED override (is_deleted=1)- override_data = [- (- team_id,- "deleted_override_distinct",- merged_person_id,- now - timedelta(days=4), # _timestamp- 1, # version- 1, # is_deleted = TRUE- ),- ]-- def insert_override(client):- client.execute(- """INSERT INTO person_distinct_id_overrides- (team_id, distinct_id, person_id, _timestamp, version, is_deleted)- VALUES""",- override_data,- )-- cluster.any_host(insert_override).result()-- # Insert the ORIGINAL person (not the merged one)- person_data = [(team_id, original_person_id, json.dumps({"prop": "old"}), 1, now - timedelta(days=8))]-- def insert_person(client):- client.execute(- """INSERT INTO person (team_id, id, properties, version, _timestamp)- VALUES""",- person_data,- )-- cluster.any_host(insert_person).result()-- results = get_person_property_updates_from_clickhouse(- team_id=team_id,- bug_window_start=bug_window_start.strftime("%Y-%m-%d %H:%M:%S"),- )-- # Since override is deleted, should use original_person_id- assert len(results) == 1- assert results[0].person_id == str(original_person_id)-- def test_override_version_ordering(self, cluster: ClickhouseCluster):- """- When multiple override versions exist, the highest version should win.- """- team_id = 99921- original_person_id = UUID("ffff0000-0000-0000-0000-000000000001")- first_merged_person_id = UUID("ffff0000-0000-0000-0000-000000000002")- final_merged_person_id = UUID("ffff0000-0000-0000-0000-000000000003")- now = datetime.now().replace(microsecond=0)- bug_window_start = now - timedelta(days=10)-- event_ts = now - timedelta(days=3)-- events = [- (- team_id,- "multi_version_distinct",- original_person_id,- event_ts,- json.dumps({"$set": {"prop": "value"}}),- ),- ]-- def insert_events(client):- client.execute(- """INSERT INTO writable_events (team_id, distinct_id, person_id, timestamp, properties)- VALUES""",- events,- )-- cluster.any_host(insert_events).result()-- # Insert multiple override versions - version 2 should win- override_data = [- (team_id, "multi_version_distinct", first_merged_person_id, now - timedelta(days=5), 1, 0),- (team_id, "multi_version_distinct", final_merged_person_id, now - timedelta(days=4), 2, 0),- ]-- def insert_override(client):- client.execute(- """INSERT INTO person_distinct_id_overrides- (team_id, distinct_id, person_id, _timestamp, version, is_deleted)- VALUES""",- override_data,- )-- cluster.any_host(insert_override).result()-- # Insert the FINAL merged person- person_data = [(team_id, final_merged_person_id, json.dumps({"prop": "old"}), 1, now - timedelta(days=8))]-- def insert_person(client):- client.execute(- """INSERT INTO person (team_id, id, properties, version, _timestamp)- VALUES""",- person_data,- )-- cluster.any_host(insert_person).result()-- results = get_person_property_updates_from_clickhouse(- team_id=team_id,- bug_window_start=bug_window_start.strftime("%Y-%m-%d %H:%M:%S"),- )-- # Should use final_merged_person_id (version 2)- assert len(results) == 1- assert results[0].person_id == str(final_merged_person_id)-- def test_deleted_person_filtered_out(self, cluster: ClickhouseCluster):- """- Persons with is_deleted=1 in the person table should be filtered out.- There's no point updating properties on a deleted person.- """- team_id = 99922- person_id = UUID("11110000-0000-0000-0000-000000000001")- now = datetime.now().replace(microsecond=0)- bug_window_start = now - timedelta(days=10)-- event_ts = now - timedelta(days=3)-- events = [- (team_id, "deleted_person_distinct", person_id, event_ts, json.dumps({"$set": {"prop": "new_value"}})),- ]-- def insert_events(client):- client.execute(- """INSERT INTO writable_events (team_id, distinct_id, person_id, timestamp, properties)- VALUES""",- events,- )-- cluster.any_host(insert_events).result()-- # Insert a DELETED person (is_deleted=1)- person_data = [- (- team_id,- person_id,- json.dumps({"prop": "old_value"}),- 1, # version- 1, # is_deleted = TRUE- now - timedelta(days=8), # _timestamp- )- ]-- def insert_person(client):- client.execute(- """INSERT INTO person (team_id, id, properties, version, is_deleted, _timestamp)- VALUES""",- person_data,- )-- cluster.any_host(insert_person).result()-- results = get_person_property_updates_from_clickhouse(- team_id=team_id,- bug_window_start=bug_window_start.strftime("%Y-%m-%d %H:%M:%S"),- )-- # Deleted persons should NOT be processed- assert len(results) == 0-- def test_person_multiple_versions_uses_latest(self, cluster: ClickhouseCluster):- """- When a person has multiple versions in CH, argMax(properties, version)- should select the properties from the highest version.- """- team_id = 99923- person_id = UUID("22220000-0000-0000-0000-000000000002")- now = datetime.now().replace(microsecond=0)- bug_window_start = now - timedelta(days=10)-- event_ts = now - timedelta(days=3)-- events = [- (- team_id,- "multi_version_person_distinct",- person_id,- event_ts,- json.dumps({"$set": {"prop": "event_value"}}),- ),- ]-- def insert_events(client):- client.execute(- """INSERT INTO writable_events (team_id, distinct_id, person_id, timestamp, properties)- VALUES""",- events,- )-- cluster.any_host(insert_events).result()-- # Insert multiple versions of the same person- # Version 2 has different properties than version 1- person_data = [- (team_id, person_id, json.dumps({"prop": "v1_value"}), 1, now - timedelta(days=8)),- (team_id, person_id, json.dumps({"prop": "v2_value"}), 2, now - timedelta(days=7)),- ]-- def insert_person(client):- client.execute(- """INSERT INTO person (team_id, id, properties, version, _timestamp)- VALUES""",- person_data,- )-- cluster.any_host(insert_person).result()-- results = get_person_property_updates_from_clickhouse(- team_id=team_id,- bug_window_start=bug_window_start.strftime("%Y-%m-%d %H:%M:%S"),- )-- # Should compare against v2_value (latest version)- # Event has "event_value", person v2 has "v2_value" - different, so should be in diff- assert len(results) == 1- updates = {key: pv.value for key, pv in results[0].set_updates.items()}- assert updates["prop"] == "event_value"-- # ==================== Same Key Operation Tests ====================-- def test_set_and_set_once_on_same_key(self, cluster: ClickhouseCluster):- """- When $set and $set_once both target the same key, both should be returned- as separate operations (query doesn't merge them, Python logic handles precedence).- """- team_id = 99912- person_id = UUID("cccc0000-0000-0000-0000-000000000001")- now = datetime.now().replace(microsecond=0)- bug_window_start = now - timedelta(days=10)-- set_once_ts = now - timedelta(days=5)- set_ts = now - timedelta(days=2)-- events = [- (team_id, "distinct_1", person_id, set_once_ts, json.dumps({"$set_once": {"email": "[email protected]"}})),- (team_id, "distinct_1", person_id, set_ts, json.dumps({"$set": {"email": "[email protected]"}})),- ]-- def insert_events(client):- client.execute(- """INSERT INTO writable_events (team_id, distinct_id, person_id, timestamp, properties)- VALUES""",- events,- )-- cluster.any_host(insert_events).result()-- # Person WITHOUT email (so set_once_diff includes it)- # Person WITH different email for set_diff- # Actually, to get both in results, we need person to NOT have email (for set_once)- # and have a different email (for set). That's contradictory.- # Let's test with person having a different email - set_diff should include it,- # set_once_diff should NOT (key exists).- person_data = [(team_id, person_id, json.dumps({"email": "[email protected]"}), 1, now - timedelta(days=8))]-- def insert_person(client):- client.execute(- """INSERT INTO person (team_id, id, properties, version, _timestamp)- VALUES""",- person_data,- )-- cluster.any_host(insert_person).result()-- results = get_person_property_updates_from_clickhouse(- team_id=team_id,- bug_window_start=bug_window_start.strftime("%Y-%m-%d %H:%M:%S"),- )-- assert len(results) == 1-- # $set should be included (value differs from person)- assert len(results[0].set_updates) == 1- assert "email" in results[0].set_updates- assert results[0].set_updates["email"].value == "[email protected]"-- # $set_once should NOT be included (key already exists in person)- assert len(results[0].set_once_updates) == 0-- # ==================== Diff Filtering Tests ====================-- def test_set_with_same_value_not_in_diff(self, cluster: ClickhouseCluster):- """- When $set value equals current person property value,- it should NOT be in set_diff (no change needed).- """- team_id = 99913- person_id = UUID("dddd0000-0000-0000-0000-000000000001")- now = datetime.now().replace(microsecond=0)- bug_window_start = now - timedelta(days=10)-- event_ts = now - timedelta(days=3)-- events = [- (team_id, "distinct_1", person_id, event_ts, json.dumps({"$set": {"name": "Same Value"}})),- ]-- def insert_events(client):- client.execute(- """INSERT INTO writable_events (team_id, distinct_id, person_id, timestamp, properties)- VALUES""",- events,- )-- cluster.any_host(insert_events).result()-- # Person with SAME value- person_data = [(team_id, person_id, json.dumps({"name": "Same Value"}), 1, now - timedelta(days=8))]-- def insert_person(client):- client.execute(- """INSERT INTO person (team_id, id, properties, version, _timestamp)- VALUES""",- person_data,- )-- cluster.any_host(insert_person).result()-- results = get_person_property_updates_from_clickhouse(- team_id=team_id,- bug_window_start=bug_window_start.strftime("%Y-%m-%d %H:%M:%S"),- )-- # Should be empty - no diff when values are the same- assert len(results) == 0-- def test_set_on_missing_property_not_in_diff(self, cluster: ClickhouseCluster):- """- When $set targets a property that doesn't exist in person,- it should NOT be in set_diff (set_diff only updates existing different values).- """- team_id = 99914- person_id = UUID("eeee0000-0000-0000-0000-000000000001")- now = datetime.now().replace(microsecond=0)- bug_window_start = now - timedelta(days=10)-- event_ts = now - timedelta(days=3)-- events = [- (team_id, "distinct_1", person_id, event_ts, json.dumps({"$set": {"new_prop": "value"}})),- ]-- def insert_events(client):- client.execute(- """INSERT INTO writable_events (team_id, distinct_id, person_id, timestamp, properties)- VALUES""",- events,- )-- cluster.any_host(insert_events).result()-- # Person WITHOUT new_prop- person_data = [(team_id, person_id, json.dumps({"other_prop": "value"}), 1, now - timedelta(days=8))]-- def insert_person(client):- client.execute(- """INSERT INTO person (team_id, id, properties, version, _timestamp)- VALUES""",- person_data,- )-- cluster.any_host(insert_person).result()-- results = get_person_property_updates_from_clickhouse(- team_id=team_id,- bug_window_start=bug_window_start.strftime("%Y-%m-%d %H:%M:%S"),- )-- # Should be empty - set_diff doesn't include missing properties- assert len(results) == 0-- def test_unset_on_missing_property_not_in_diff(self, cluster: ClickhouseCluster):- """- When $unset targets a property that doesn't exist in person,- it should NOT be in unset_diff.- """- team_id = 99915- person_id = UUID("ffff0000-0000-0000-0000-000000000001")- now = datetime.now().replace(microsecond=0)- bug_window_start = now - timedelta(days=10)-- event_ts = now - timedelta(days=3)-- events = [- (team_id, "distinct_1", person_id, event_ts, json.dumps({"$unset": ["nonexistent_prop"]})),- ]-- def insert_events(client):- client.execute(- """INSERT INTO writable_events (team_id, distinct_id, person_id, timestamp, properties)- VALUES""",- events,- )-- cluster.any_host(insert_events).result()-- # Person WITHOUT nonexistent_prop- person_data = [(team_id, person_id, json.dumps({"other_prop": "value"}), 1, now - timedelta(days=8))]-- def insert_person(client):- client.execute(- """INSERT INTO person (team_id, id, properties, version, _timestamp)- VALUES""",- person_data,- )-- cluster.any_host(insert_person).result()-- results = get_person_property_updates_from_clickhouse(- team_id=team_id,- bug_window_start=bug_window_start.strftime("%Y-%m-%d %H:%M:%S"),- )-- # Should be empty - unset_diff doesn't include missing properties- assert len(results) == 0-- # ==================== Empty Value Tests ====================-- def test_empty_set_object_no_results(self, cluster: ClickhouseCluster):- """Empty $set object should not cause issues or return results."""- team_id = 99916- person_id = UUID("11110000-0000-0000-0000-000000000001")- now = datetime.now().replace(microsecond=0)- bug_window_start = now - timedelta(days=10)-- event_ts = now - timedelta(days=3)-- events = [- (team_id, "distinct_1", person_id, event_ts, json.dumps({"$set": {}})),- ]-- def insert_events(client):- client.execute(- """INSERT INTO writable_events (team_id, distinct_id, person_id, timestamp, properties)- VALUES""",- events,- )-- cluster.any_host(insert_events).result()-- person_data = [(team_id, person_id, json.dumps({"prop": "value"}), 1, now - timedelta(days=8))]-- def insert_person(client):- client.execute(- """INSERT INTO person (team_id, id, properties, version, _timestamp)- VALUES""",- person_data,- )-- cluster.any_host(insert_person).result()-- results = get_person_property_updates_from_clickhouse(- team_id=team_id,- bug_window_start=bug_window_start.strftime("%Y-%m-%d %H:%M:%S"),- )-- # Empty $set should produce no results- assert len(results) == 0-- def test_empty_string_value_is_valid(self, cluster: ClickhouseCluster):- """Empty string is a valid value in $set - different from null."""- team_id = 99917- person_id = UUID("22220000-0000-0000-0000-000000000001")- now = datetime.now().replace(microsecond=0)- bug_window_start = now - timedelta(days=10)-- event_ts = now - timedelta(days=3)-- events = [- (- team_id,- "distinct_1",- person_id,- event_ts,- json.dumps({"$set": {"empty_prop": "", "valid_prop": "value"}}),- ),- ]-- def insert_events(client):- client.execute(- """INSERT INTO writable_events (team_id, distinct_id, person_id, timestamp, properties)- VALUES""",- events,- )-- cluster.any_host(insert_events).result()-- person_data = [- (team_id, person_id, json.dumps({"empty_prop": "old", "valid_prop": "old"}), 1, now - timedelta(days=8))- ]-- def insert_person(client):- client.execute(- """INSERT INTO person (team_id, id, properties, version, _timestamp)- VALUES""",- person_data,- )-- cluster.any_host(insert_person).result()-- results = get_person_property_updates_from_clickhouse(- team_id=team_id,- bug_window_start=bug_window_start.strftime("%Y-%m-%d %H:%M:%S"),- )-- assert len(results) == 1- updates = {key: pv.value for key, pv in results[0].set_updates.items()}-- # valid_prop should be included (changed from "old" to "value")- assert "valid_prop" in updates- assert updates["valid_prop"] == "value"-- # empty_prop should be included - empty string is a valid value, not filtered like null- # It changes from "old" to "" (empty string)- assert "empty_prop" in updates- assert updates["empty_prop"] == ""-- # ==================== Window Filtering Tests ====================-- def test_events_before_bug_window_start_filtered(self, cluster: ClickhouseCluster):- """Events with timestamp before bug_window_start should be filtered.-- Events after bug_window_start are included (up to now()).- """- team_id = 99918- person_id = UUID("33330000-0000-0000-0000-000000000001")- now = datetime.now().replace(microsecond=0)- bug_window_start = now - timedelta(days=5)-- # Event BEFORE bug_window_start - should be filtered- before_ts = now - timedelta(days=7)- # Event AFTER bug_window_start - should be included (reads up to now())- after_start_ts = now - timedelta(days=1)- # Event IN window (also after bug_window_start)- in_window_ts = now - timedelta(days=3)-- events = [- (team_id, "distinct_1", person_id, before_ts, json.dumps({"$set": {"before": "value"}})),- (team_id, "distinct_1", person_id, after_start_ts, json.dumps({"$set": {"after_start": "value"}})),- (team_id, "distinct_1", person_id, in_window_ts, json.dumps({"$set": {"in_window": "value"}})),- ]-- def insert_events(client):- client.execute(- """INSERT INTO writable_events (team_id, distinct_id, person_id, timestamp, properties)- VALUES""",- events,- )-- cluster.any_host(insert_events).result()-- person_data = [- (- team_id,- person_id,- json.dumps({"before": "old", "after_start": "old", "in_window": "old"}),- 1,- now - timedelta(days=4), # Person _timestamp after bug_window_start- )- ]-- def insert_person(client):- client.execute(- """INSERT INTO person (team_id, id, properties, version, _timestamp)- VALUES""",- person_data,- )-- cluster.any_host(insert_person).result()-- results = get_person_property_updates_from_clickhouse(- team_id=team_id,- bug_window_start=bug_window_start.strftime("%Y-%m-%d %H:%M:%S"),- )-- assert len(results) == 1- updates = {key: pv.value for key, pv in results[0].set_updates.items()}-- # Events after bug_window_start should be included- assert "in_window" in updates- assert "after_start" in updates- # Events before bug_window_start should NOT be included- assert "before" not in updates-- def test_person_created_before_bug_window_with_events_after_is_included(self, cluster: ClickhouseCluster):- """Regression test: Person created before bug_window_start with events after should be included.-- Scenario:- - Person was created/last modified BEFORE bug_window_start (their _timestamp is old)- - After bug_window_start, there are $set events for this person- - These events SHOULD be included in the reconciliation-- This is important because we want to reconcile ALL persons that have property-setting- events after bug_window_start, regardless of when the person was originally created.- The bug window is about when the ingestion bug occurred, not when persons were created.- """- team_id = 99919- person_id = UUID("55550000-0000-0000-0000-000000000002")- now = datetime.now().replace(microsecond=0)- bug_window_start = now - timedelta(days=5)-- # Event AFTER bug_window_start - should be included- event_ts = now - timedelta(days=3)-- events = [- (team_id, "distinct_1", person_id, event_ts, json.dumps({"$set": {"email": "[email protected]"}})),- ]-- def insert_events(client):- client.execute(- """INSERT INTO writable_events (team_id, distinct_id, person_id, timestamp, properties)- VALUES""",- events,- )-- cluster.any_host(insert_events).result()-- # Person _timestamp BEFORE bug_window_start (person was created before the bug window)- # but with a property that differs from the event's $set value- person_data = [(team_id, person_id, json.dumps({"email": "[email protected]"}), 1, now - timedelta(days=10))]-- def insert_person(client):- client.execute(- """INSERT INTO person (team_id, id, properties, version, _timestamp)- VALUES""",- person_data,- )-- cluster.any_host(insert_person).result()-- results = get_person_property_updates_from_clickhouse(- team_id=team_id,- bug_window_start=bug_window_start.strftime("%Y-%m-%d %H:%M:%S"),- )-- # Person SHOULD be included because they have events after bug_window_start- # The person's _timestamp being before bug_window_start should NOT exclude them- assert len(results) == 1- assert str(results[0].person_id) == str(person_id)- assert "email" in results[0].set_updates- assert results[0].set_updates["email"].value == "[email protected]"-- def test_timestamp_window_filtering_all_permutations(self, cluster: ClickhouseCluster):- """Comprehensive test of all permutations of person _timestamp and event timestamp.-- Tests the matrix of:- - Person _timestamp: BEFORE / DURING / AFTER bug window- - Event timestamp: BEFORE / DURING / AFTER bug window-- Expected behavior:- - Person _timestamp does NOT affect filtering (we removed that filter)- - Events BEFORE bug_window_start are filtered out- - Events DURING or AFTER bug_window_start are included (query uses now() as upper bound)-- So expected results:- | Person _timestamp | Event timestamp | Included? |- |-------------------|-----------------|-----------|- | BEFORE | BEFORE | NO |- | BEFORE | DURING | YES |- | BEFORE | AFTER | YES |- | DURING | BEFORE | NO |- | DURING | DURING | YES |- | DURING | AFTER | YES |- | AFTER | BEFORE | NO |- | AFTER | DURING | YES |- | AFTER | AFTER | YES |- """- # Use two teams to ensure isolation- team_id_1 = 99950- team_id_2 = 99951-- now = datetime.now().replace(microsecond=0)-- # Define time periods- # Bug window is conceptually from bug_window_start to bug_window_end (for team discovery)- # But the CH query reads from bug_window_start to now()- bug_window_start = now - timedelta(days=10)- _bug_window_end = now - timedelta(days=5) # Used conceptually, not in query-- # Timestamps for each period- ts_before = now - timedelta(days=15) # Before bug_window_start- ts_during = now - timedelta(days=7) # Between bug_window_start and bug_window_end- ts_after = now - timedelta(days=2) # After bug_window_end but before now()-- # Create 9 persons for team 1 - all permutations- # Format: (person_id_suffix, person_ts, event_ts, should_be_included)- test_cases_team1 = [- # Person _timestamp BEFORE bug_window_start- ("001", ts_before, ts_before, False, "person_before_event_before"),- ("002", ts_before, ts_during, True, "person_before_event_during"),- ("003", ts_before, ts_after, True, "person_before_event_after"),- # Person _timestamp DURING bug_window- ("004", ts_during, ts_before, False, "person_during_event_before"),- ("005", ts_during, ts_during, True, "person_during_event_during"),- ("006", ts_during, ts_after, True, "person_during_event_after"),- # Person _timestamp AFTER bug_window_end- ("007", ts_after, ts_before, False, "person_after_event_before"),- ("008", ts_after, ts_during, True, "person_after_event_during"),- ("009", ts_after, ts_after, True, "person_after_event_after"),- ]-- # Create 3 persons for team 2 - subset to verify team isolation- test_cases_team2 = [- ("101", ts_before, ts_during, True, "team2_person_before_event_during"),- ("102", ts_during, ts_after, True, "team2_person_during_event_after"),- ("103", ts_after, ts_before, False, "team2_person_after_event_before"),- ]-- # Helper to create UUID from suffix- def make_uuid(suffix: str) -> UUID:- return UUID(f"99990000-0000-0000-0000-000000000{suffix}")-- # Insert events for team 1- events_team1 = []- for suffix, _person_ts, event_ts, _expected, prop_name in test_cases_team1:- person_id = make_uuid(suffix)- events_team1.append(- (- team_id_1,- f"distinct_{suffix}",- person_id,- event_ts,- json.dumps({"$set": {prop_name: "new_value"}}),- )- )-- def insert_events_team1(client):- client.execute(- """INSERT INTO writable_events (team_id, distinct_id, person_id, timestamp, properties)- VALUES""",- events_team1,- )-- cluster.any_host(insert_events_team1).result()-- # Insert events for team 2- events_team2 = []- for suffix, _person_ts, event_ts, _expected, prop_name in test_cases_team2:- person_id = make_uuid(suffix)- events_team2.append(- (- team_id_2,- f"distinct_{suffix}",- person_id,- event_ts,- json.dumps({"$set": {prop_name: "new_value"}}),- )- )-- def insert_events_team2(client):- client.execute(- """INSERT INTO writable_events (team_id, distinct_id, person_id, timestamp, properties)- VALUES""",- events_team2,- )-- cluster.any_host(insert_events_team2).result()-- # Insert persons for team 1 (with different property values so we get a diff)- persons_team1 = []- for suffix, person_ts, _event_ts, _expected, prop_name in test_cases_team1:- person_id = make_uuid(suffix)- persons_team1.append(- (- team_id_1,- person_id,- json.dumps({prop_name: "old_value"}),- 1, # version- person_ts, # _timestamp- )- )-- def insert_persons_team1(client):- client.execute(- """INSERT INTO person (team_id, id, properties, version, _timestamp)- VALUES""",- persons_team1,- )-- cluster.any_host(insert_persons_team1).result()-- # Insert persons for team 2- persons_team2 = []- for suffix, person_ts, _event_ts, _expected, prop_name in test_cases_team2:- person_id = make_uuid(suffix)- persons_team2.append(- (- team_id_2,- person_id,- json.dumps({prop_name: "old_value"}),- 1,- person_ts,- )- )-- def insert_persons_team2(client):- client.execute(- """INSERT INTO person (team_id, id, properties, version, _timestamp)- VALUES""",- persons_team2,- )-- cluster.any_host(insert_persons_team2).result()-- # Query for team 1- results_team1 = get_person_property_updates_from_clickhouse(- team_id=team_id_1,- bug_window_start=bug_window_start.strftime("%Y-%m-%d %H:%M:%S"),- )-- # Query for team 2- results_team2 = get_person_property_updates_from_clickhouse(- team_id=team_id_2,- bug_window_start=bug_window_start.strftime("%Y-%m-%d %H:%M:%S"),- )-- # Verify team 1 results- result_person_ids_team1 = {str(r.person_id) for r in results_team1}- expected_person_ids_team1 = {- str(make_uuid(suffix))- for suffix, _person_ts, _event_ts, expected, _prop_name in test_cases_team1- if expected- }-- assert result_person_ids_team1 == expected_person_ids_team1, (- f"Team 1 mismatch.\n"- f"Expected: {expected_person_ids_team1}\n"- f"Got: {result_person_ids_team1}\n"- f"Missing: {expected_person_ids_team1 - result_person_ids_team1}\n"- f"Extra: {result_person_ids_team1 - expected_person_ids_team1}"- )-- # Verify team 2 results- result_person_ids_team2 = {str(r.person_id) for r in results_team2}- expected_person_ids_team2 = {- str(make_uuid(suffix))- for suffix, _person_ts, _event_ts, expected, _prop_name in test_cases_team2- if expected- }-- assert result_person_ids_team2 == expected_person_ids_team2, (- f"Team 2 mismatch.\nExpected: {expected_person_ids_team2}\nGot: {result_person_ids_team2}"- )-- # Verify the property values are correct for included persons- for result in results_team1:- # Each person should have exactly one set_update with value "new_value"- assert len(result.set_updates) == 1, f"Person {result.person_id} should have 1 update"- prop_value = next(iter(result.set_updates.values()))- assert prop_value.value == "new_value", f"Person {result.person_id} should have new_value"-- # Verify specific cases to make the test more explicit- # Person 002: _timestamp BEFORE, event DURING → should be included- assert str(make_uuid("002")) in result_person_ids_team1, (- "Person with _timestamp BEFORE but event DURING should be included"- )-- # Person 003: _timestamp BEFORE, event AFTER → should be included- assert str(make_uuid("003")) in result_person_ids_team1, (- "Person with _timestamp BEFORE but event AFTER should be included"- )-- # Person 007: _timestamp AFTER, event BEFORE → should NOT be included- assert str(make_uuid("007")) not in result_person_ids_team1, (- "Person with event BEFORE bug_window_start should NOT be included"- )-- # Person 001: _timestamp BEFORE, event BEFORE → should NOT be included- assert str(make_uuid("001")) not in result_person_ids_team1, (- "Person with event BEFORE bug_window_start should NOT be included"- )-- # ==================== Edge Case Tests ====================-- def test_same_timestamp_deterministic(self, cluster: ClickhouseCluster):- """- When multiple events have exact same timestamp,- argMax/argMin should still produce deterministic results.- """- team_id = 99924- person_id = UUID("55550000-0000-0000-0000-000000000001")- now = datetime.now().replace(microsecond=0)- bug_window_start = now - timedelta(days=10)-- # Same timestamp for both events- same_ts = now - timedelta(days=3)-- events = [- (team_id, "distinct_1", person_id, same_ts, json.dumps({"$set": {"name": "Value A"}})),- (team_id, "distinct_2", person_id, same_ts, json.dumps({"$set": {"name": "Value B"}})),- ]-- def insert_events(client):- client.execute(- """INSERT INTO writable_events (team_id, distinct_id, person_id, timestamp, properties)- VALUES""",- events,- )-- cluster.any_host(insert_events).result()-- person_data = [(team_id, person_id, json.dumps({"name": "Old"}), 1, now - timedelta(days=8))]-- def insert_person(client):- client.execute(- """INSERT INTO person (team_id, id, properties, version, _timestamp)- VALUES""",- person_data,- )-- cluster.any_host(insert_person).result()-- results = get_person_property_updates_from_clickhouse(- team_id=team_id,- bug_window_start=bug_window_start.strftime("%Y-%m-%d %H:%M:%S"),- )-- # Should get exactly one result with one of the values- assert len(results) == 1- updates = {key: pv.value for key, pv in results[0].set_updates.items()}- assert updates["name"] in ["Value A", "Value B"]-- def test_special_characters_in_property_keys(self, cluster: ClickhouseCluster):- """Property keys with special characters should work correctly."""- team_id = 99925- person_id = UUID("66660000-0000-0000-0000-000000000001")- now = datetime.now().replace(microsecond=0)- bug_window_start = now - timedelta(days=10)-- event_ts = now - timedelta(days=3)-- events = [- (- team_id,- "distinct_1",- person_id,- event_ts,- json.dumps(- {- "$set": {- "key with spaces": "value1",- "key.with.dots": "value2",- "key-with-dashes": "value3",- "key_with_underscores": "value4",- "キー": "unicode value", # Japanese "key"- }- }- ),- ),- ]-- def insert_events(client):- client.execute(- """INSERT INTO writable_events (team_id, distinct_id, person_id, timestamp, properties)- VALUES""",- events,- )-- cluster.any_host(insert_events).result()-- # Person with different values for these keys- person_data = [- (- team_id,- person_id,- json.dumps(- {- "key with spaces": "old1",- "key.with.dots": "old2",- "key-with-dashes": "old3",- "key_with_underscores": "old4",- "キー": "old unicode",- }- ),- 1,- now - timedelta(days=8),- )- ]-- def insert_person(client):- client.execute(- """INSERT INTO person (team_id, id, properties, version, _timestamp)- VALUES""",- person_data,- )-- cluster.any_host(insert_person).result()-- results = get_person_property_updates_from_clickhouse(- team_id=team_id,- bug_window_start=bug_window_start.strftime("%Y-%m-%d %H:%M:%S"),- )-- assert len(results) == 1- updates = {key: pv.value for key, pv in results[0].set_updates.items()}-- assert updates["key with spaces"] == "value1"- assert updates["key.with.dots"] == "value2"- assert updates["key-with-dashes"] == "value3"- assert updates["key_with_underscores"] == "value4"- assert updates["キー"] == "unicode value"-- def test_filtered_properties_are_excluded(self, cluster: ClickhouseCluster):- """Properties in FILTERED_PERSON_UPDATE_PROPERTIES should be excluded from results.-- These are high-frequency properties like $current_url that change often- but aren't valuable enough to trigger person updates.- """- team_id = 99926- person_id = UUID("77770000-0000-0000-0000-000000000001")- now = datetime.now().replace(microsecond=0)- bug_window_start = now - timedelta(days=10)-- event_ts = now - timedelta(days=3)-- # Event with both filtered and non-filtered properties- events = [- (- team_id,- "distinct_1",- person_id,- event_ts,- json.dumps(- {- "$set": {- # Filtered properties - should NOT appear in results- "$current_url": "https://example.com/page",- "$pathname": "/page",- "$browser": "Chrome",- "$os": "Mac OS X",- # Non-filtered properties - SHOULD appear in results- "email": "[email protected]",- "name": "Test User",- },- "$set_once": {- # Filtered- "$referring_domain": "google.com",- # Non-filtered- "initial_campaign": "summer_sale",- },- }- ),- ),- ]-- def insert_events(client):- client.execute(- """INSERT INTO writable_events (team_id, distinct_id, person_id, timestamp, properties)- VALUES""",- events,- )-- cluster.any_host(insert_events).result()-- # Person with different values so diffs are detected- person_data = [- (- team_id,- person_id,- json.dumps(- {- "$current_url": "https://example.com/old",- "email": "[email protected]",- "name": "Old Name",- }- ),- 1,- now - timedelta(days=8),- )- ]-- def insert_person(client):- client.execute(- """INSERT INTO person (team_id, id, properties, version, _timestamp)- VALUES""",- person_data,- )-- cluster.any_host(insert_person).result()-- results = get_person_property_updates_from_clickhouse(- team_id=team_id,- bug_window_start=bug_window_start.strftime("%Y-%m-%d %H:%M:%S"),- )-- assert len(results) == 1- person_diffs = results[0]-- # Non-filtered properties should be present- set_keys = set(person_diffs.set_updates.keys())- set_once_keys = set(person_diffs.set_once_updates.keys())-- assert "email" in set_keys, "Non-filtered property 'email' should be in results"- assert "name" in set_keys, "Non-filtered property 'name' should be in results"- assert "initial_campaign" in set_once_keys, "Non-filtered property 'initial_campaign' should be in results"-- # Filtered properties should NOT be present- assert "$current_url" not in set_keys, "Filtered property '$current_url' should NOT be in results"- assert "$pathname" not in set_keys, "Filtered property '$pathname' should NOT be in results"- assert "$browser" not in set_keys, "Filtered property '$browser' should NOT be in results"- assert "$os" not in set_keys, "Filtered property '$os' should NOT be in results"- assert "$referring_domain" not in set_once_keys, (- "Filtered property '$referring_domain' should NOT be in results"- )-- def test_person_ids_filter_only_returns_specified_persons(self, cluster: ClickhouseCluster):- """- Integration test: verify person_ids filter correctly restricts query results.-- This tests the fix for the NULL IS NULL bug - when person_ids is provided,- only those persons should be returned.- """- team_id = 99970- now = datetime.now().replace(microsecond=0)- bug_window_start = now - timedelta(days=10)- event_ts = now - timedelta(days=5)-- # Create 5 persons, but we'll only query for 2 of them- person_ids = [UUID(f"77770000-0000-0000-0000-00000000000{i}") for i in range(1, 6)]- target_person_ids = (str(person_ids[0]), str(person_ids[2])) # Person 1 and 3-- # Insert events for all 5 persons- events = []- for i, pid in enumerate(person_ids):- events.append(- (- team_id,- f"distinct_{i}",- pid,- event_ts,- json.dumps({"$set": {"email": f"person{i}@example.com"}}),- )- )-- def insert_events(client: Client) -> None:- client.execute(- """INSERT INTO writable_events (team_id, distinct_id, person_id, timestamp, properties)- VALUES""",- events,- )-- cluster.any_host(insert_events).result()-- # Insert all 5 persons in ClickHouse with different email (so diff is detected)- person_data = []- for i, pid in enumerate(person_ids):- person_data.append(- (- team_id,- pid,- json.dumps({"email": f"old{i}@example.com"}),- 1,- now - timedelta(days=8),- )- )-- def insert_persons(client: Client) -> None:- client.execute(- """INSERT INTO person (team_id, id, properties, version, _timestamp)- VALUES""",- person_data,- )-- cluster.any_host(insert_persons).result()-- # Query with person_ids filter - should only return the 2 specified persons- results = get_raw_person_property_updates_from_clickhouse(- team_id=team_id,- bug_window_start=bug_window_start.strftime("%Y-%m-%d %H:%M:%S"),- bug_window_end=now.strftime("%Y-%m-%d %H:%M:%S"),- person_ids=target_person_ids,- )-- # Should only have 2 results (the filtered persons)- result_person_ids = {r.person_id for r in results}- expected_person_ids = set(target_person_ids)-- assert result_person_ids == expected_person_ids, (- f"person_ids filter failed.\n"- f"Expected: {expected_person_ids}\n"- f"Got: {result_person_ids}\n"- f"This may indicate the NULL IS NULL bug has regressed!"- )-- # Verify the data is correct for the returned persons- for result in results:- assert len(result.set_updates) == 1- assert "email" in result.set_updates-- def test_interleaved_events_multiple_persons_realistic_scenario(self, cluster: ClickhouseCluster):- """- Integration test: realistic scenario with multiple persons and interleaved events.-- Simulates a real-world scenario where:- - Multiple users are active simultaneously- - Events arrive in arbitrary order (not grouped by person)- - Different operation types ($set, $set_once, $unset) are mixed- - Some properties conflict (same key, different operations)-- This tests that the query correctly aggregates per-person despite- events being interleaved in the events table.- """- team_id = 99971- now = datetime.now().replace(microsecond=0)- bug_window_start = now - timedelta(days=10)-- # Three users with different activity patterns- alice = UUID("aaaa0000-0000-0000-0000-000000000001")- bob = UUID("bbbb0000-0000-0000-0000-000000000002")- charlie = UUID("cccc0000-0000-0000-0000-000000000003")-- # Events arrive interleaved (not grouped by person) - this is realistic- # Time ordering simulates events arriving from different user sessions- events = [- # t+1h: Alice sets email- (team_id, "alice_1", alice, now - timedelta(hours=9), json.dumps({"$set": {"email": "[email protected]"}})),- # t+1h: Bob sets name- (team_id, "bob_1", bob, now - timedelta(hours=9), json.dumps({"$set": {"name": "Bob v1"}})),- # t+2h: Charlie sets email and set_once for referrer- (- team_id,- "charlie_1",- charlie,- now - timedelta(hours=8),- json.dumps({"$set": {"email": "[email protected]"}, "$set_once": {"referrer": "google"}}),- ),- # t+3h: Alice updates email (should win over v1)- (team_id, "alice_2", alice, now - timedelta(hours=7), json.dumps({"$set": {"email": "[email protected]"}})),- # t+3h: Bob also tries set_once referrer (should use first value)- (team_id, "bob_2", bob, now - timedelta(hours=7), json.dumps({"$set_once": {"referrer": "facebook"}})),- # t+4h: Alice sets name- (team_id, "alice_3", alice, now - timedelta(hours=6), json.dumps({"$set": {"name": "Alice"}})),- # t+4h: Bob updates name (should win over v1)- (team_id, "bob_3", bob, now - timedelta(hours=6), json.dumps({"$set": {"name": "Bob v2"}})),- # t+5h: Charlie tries set_once referrer again (should NOT override google)- (- team_id,- "charlie_2",- charlie,- now - timedelta(hours=5),- json.dumps({"$set_once": {"referrer": "twitter"}}),- ),- # t+6h: Alice unsets email (should override the $set)- (team_id, "alice_4", alice, now - timedelta(hours=4), json.dumps({"$unset": ["email"]})),- # t+7h: Bob's set_once for referrer again (still should be facebook - first one)- (team_id, "bob_4", bob, now - timedelta(hours=3), json.dumps({"$set_once": {"referrer": "linkedin"}})),- ]-- def insert_events(client: Client) -> None:- client.execute(- """INSERT INTO writable_events (team_id, distinct_id, person_id, timestamp, properties)- VALUES""",- events,- )-- cluster.any_host(insert_events).result()-- # Create persons with existing properties- # Alice: has old email (different, so $set shows in diff) and no name- # Bob: has old name (different), no referrer- # Charlie: has old email (different), no referrer- person_data = [- (team_id, alice, json.dumps({"email": "[email protected]"}), 1, now - timedelta(days=8)),- (team_id, bob, json.dumps({"name": "Old Bob"}), 1, now - timedelta(days=8)),- (team_id, charlie, json.dumps({"email": "[email protected]"}), 1, now - timedelta(days=8)),- ]-- def insert_persons(client: Client) -> None:- client.execute(- """INSERT INTO person (team_id, id, properties, version, _timestamp)- VALUES""",- person_data,- )-- cluster.any_host(insert_persons).result()-- # Query and verify- results = get_person_property_updates_from_clickhouse(- team_id=team_id,- bug_window_start=bug_window_start.strftime("%Y-%m-%d %H:%M:%S"),- )-- results_by_person = {str(r.person_id): r for r in results}-- # Alice assertions:- # - email: $set [email protected] at t+3h, then $unset at t+6h → both should appear- # - name: $set "Alice" at t+4h → but no name in person, so NOT in set_diff- alice_result = results_by_person[str(alice)]- assert "email" in alice_result.set_updates, "Alice should have email in set_updates"- assert alice_result.set_updates["email"].value == "[email protected]", "Alice email should be v2"- assert "email" in alice_result.unset_updates, "Alice should have email in unset_updates"- # name not in person, so $set won't create a diff- assert "name" not in alice_result.set_updates, "Alice name should NOT be in set_updates (key doesn't exist)"-- # Bob assertions:- # - name: latest is "Bob v2" at t+4h, person has different value- # - referrer: first set_once is "facebook" at t+3h, person doesn't have it- bob_result = results_by_person[str(bob)]- assert "name" in bob_result.set_updates, "Bob should have name in set_updates"- assert bob_result.set_updates["name"].value == "Bob v2", "Bob name should be v2"- assert "referrer" in bob_result.set_once_updates, "Bob should have referrer in set_once"- assert bob_result.set_once_updates["referrer"].value == "facebook", "Bob referrer should be facebook (first)"-- # Charlie assertions:- # - email: "[email protected]", person has different value- # - referrer: "google" (first set_once), person doesn't have it- charlie_result = results_by_person[str(charlie)]- assert "email" in charlie_result.set_updates, "Charlie should have email"- assert charlie_result.set_updates["email"].value == "[email protected]"- assert "referrer" in charlie_result.set_once_updates, "Charlie should have referrer"- assert charlie_result.set_once_updates["referrer"].value == "google", (- "Charlie referrer should be google (first)"- )-- def test_unicode_and_emoji_in_property_keys_and_values(self, cluster: ClickhouseCluster):- """- Integration test: verify Unicode and emoji characters work in property keys and values.-- Real users often have international names, emoji in custom properties, etc.- This ensures the query handles UTF-8 properly throughout the pipeline.- """- team_id = 99972- person_id = UUID("88880000-0000-0000-0000-000000000001")- now = datetime.now().replace(microsecond=0)- bug_window_start = now - timedelta(days=10)- event_ts = now - timedelta(days=5)-- # Unicode and emoji test cases- events = [- (- team_id,- "distinct_unicode",- person_id,- event_ts,- json.dumps(- {- "$set": {- "名前": "田中太郎", # Japanese: name -> Taro Tanaka- "город": "Москва", # Russian: city -> Moscow- "emoji_status": "🎉🚀💯", # Emoji value- "Ñoño": "Señor", # Spanish with ñ- "العربية": "مرحبا", # Arabic- },- "$set_once": {- "🔑_key": "emoji_key_value", # Emoji in key- "first_emoji": "👋",- },- }- ),- )- ]-- def insert_events(client: Client) -> None:- client.execute(- """INSERT INTO writable_events (team_id, distinct_id, person_id, timestamp, properties)- VALUES""",- events,- )-- cluster.any_host(insert_events).result()-- # Person with different values for the Unicode keys- person_data = [- (- team_id,- person_id,- json.dumps(- {- "名前": "山田花子", # Different Japanese name- "город": "Санкт-Петербург", # Different Russian city- "emoji_status": "😢",- "Ñoño": "Señora",- "العربية": "وداعا",- }- ),- 1,- now - timedelta(days=8),- )- ]-- def insert_persons(client: Client) -> None:- client.execute(- """INSERT INTO person (team_id, id, properties, version, _timestamp)- VALUES""",- person_data,- )-- cluster.any_host(insert_persons).result()-- results = get_person_property_updates_from_clickhouse(- team_id=team_id,- bug_window_start=bug_window_start.strftime("%Y-%m-%d %H:%M:%S"),- )-- assert len(results) == 1- person_diffs = results[0]-- # Verify all Unicode keys are present with correct values- assert "名前" in person_diffs.set_updates- assert person_diffs.set_updates["名前"].value == "田中太郎"-- assert "город" in person_diffs.set_updates- assert person_diffs.set_updates["город"].value == "Москва"-- assert "emoji_status" in person_diffs.set_updates- assert person_diffs.set_updates["emoji_status"].value == "🎉🚀💯"-- assert "Ñoño" in person_diffs.set_updates- assert person_diffs.set_updates["Ñoño"].value == "Señor"-- assert "العربية" in person_diffs.set_updates- assert person_diffs.set_updates["العربية"].value == "مرحبا"-- # Verify set_once with emoji key- assert "🔑_key" in person_diffs.set_once_updates- assert person_diffs.set_once_updates["🔑_key"].value == "emoji_key_value"-- assert "first_emoji" in person_diffs.set_once_updates- assert person_diffs.set_once_updates["first_emoji"].value == "👋"-- def test_no_affected_persons_returns_empty_results(self, cluster: ClickhouseCluster):- """- Integration test: verify graceful handling when no persons are affected.-- This ensures the query doesn't error when:- - No events exist in the bug window- - Events exist but properties match (no diff)- - Team has no data at all- """- team_id = 99973- now = datetime.now().replace(microsecond=0)-- # Bug window that has no events (far in the past, unique team_id)- bug_window_start = now - timedelta(days=1000)- bug_window_end = now - timedelta(days=999)-- # Query should return empty, not error- results = get_person_property_updates_from_clickhouse(- team_id=team_id,- bug_window_start=bug_window_start.strftime("%Y-%m-%d %H:%M:%S"),- )-- assert results == [], f"Expected empty results for team with no data, got {len(results)} results"-- # Also test get_affected_person_ids_from_clickhouse- affected_ids = get_affected_person_ids_from_clickhouse(- team_id=team_id,- bug_window_start=bug_window_start.strftime("%Y-%m-%d %H:%M:%S"),- bug_window_end=bug_window_end.strftime("%Y-%m-%d %H:%M:%S"),- )-- assert affected_ids == [], f"Expected no affected persons, got {len(affected_ids)}"-- def test_rapid_fire_events_within_same_second(self, cluster: ClickhouseCluster):- """- Integration test: multiple events for same person within the same second.-- Real-world scenario: rapid client-side events, SDK batching, etc.- Verifies deterministic behavior when timestamps collide.- """- team_id = 99974- person_id = UUID("99990000-0000-0000-0000-000000000001")- now = datetime.now().replace(microsecond=0)- bug_window_start = now - timedelta(days=10)-- # Same timestamp for all events (within same second)- same_ts = now - timedelta(days=5)-- # Multiple rapid-fire $set events for same key with same timestamp- # ClickHouse argMax should pick one deterministically- events = [- (team_id, "rapid_1", person_id, same_ts, json.dumps({"$set": {"counter": 1}})),- (team_id, "rapid_2", person_id, same_ts, json.dumps({"$set": {"counter": 2}})),- (team_id, "rapid_3", person_id, same_ts, json.dumps({"$set": {"counter": 3}})),- (team_id, "rapid_4", person_id, same_ts, json.dumps({"$set": {"counter": 4}})),- (team_id, "rapid_5", person_id, same_ts, json.dumps({"$set": {"counter": 5}})),- ]-- def insert_events(client: Client) -> None:- client.execute(- """INSERT INTO writable_events (team_id, distinct_id, person_id, timestamp, properties)- VALUES""",- events,- )-- cluster.any_host(insert_events).result()-- # Person with different counter value- person_data = [(team_id, person_id, json.dumps({"counter": 0}), 1, now - timedelta(days=8))]-- def insert_persons(client: Client) -> None:- client.execute(- """INSERT INTO person (team_id, id, properties, version, _timestamp)- VALUES""",- person_data,- )-- cluster.any_host(insert_persons).result()-- # Run query multiple times - result should be consistent (deterministic)- results_1 = get_person_property_updates_from_clickhouse(- team_id=team_id,- bug_window_start=bug_window_start.strftime("%Y-%m-%d %H:%M:%S"),- )- results_2 = get_person_property_updates_from_clickhouse(- team_id=team_id,- bug_window_start=bug_window_start.strftime("%Y-%m-%d %H:%M:%S"),- )-- assert len(results_1) == 1- assert len(results_2) == 1-- # Should be deterministic - same value both times- counter_1 = results_1[0].set_updates["counter"].value- counter_2 = results_2[0].set_updates["counter"].value-- assert counter_1 == counter_2, (- f"Rapid-fire events should produce deterministic results. Got {counter_1} then {counter_2}"- )-- # Value should be one of the inserted values (1-5)- assert counter_1 in [1, 2, 3, 4, 5], f"Counter should be one of 1-5, got {counter_1}"-- def test_affected_person_ids_excludes_filtered_property_only_events(self, cluster: ClickhouseCluster):- """- Integration test: get_affected_person_ids_from_clickhouse should exclude persons- whose events only set FILTERED_PERSON_UPDATE_PROPERTIES ($browser, $os, etc.).-- Person A: events with ONLY filtered properties -> excluded- Person B: events with at least one non-filtered property that differs from person state -> included- """- team_id = 99980- person_a = UUID("aaaa0000-0000-0000-0000-000000000001")- person_b = UUID("bbbb0000-0000-0000-0000-000000000001")- now = datetime.now().replace(microsecond=0)- bug_window_start = now - timedelta(hours=3)- bug_window_end = now - timedelta(hours=1)- event_ts = now - timedelta(hours=2)-- events = [- # Person A: only filtered properties- (- team_id,- "filtered_only_user",- person_a,- event_ts,- json.dumps(- {- "$set": {- "$browser": "Chrome",- "$os": "Mac OS X",- "$current_url": "https://example.com/page",- "$pathname": "/page",- },- }- ),- ),- # Person B: has a non-filtered property with a value that differs from person state- (- team_id,- "real_update_user",- person_b,- event_ts,- json.dumps(- {- "$set": {- "$browser": "Firefox",- "email": "[email protected]",- },- }- ),- ),- ]-- def insert_events(client: Client) -> None:- client.execute(- """INSERT INTO writable_events (team_id, distinct_id, person_id, timestamp, properties)- VALUES""",- events,- )-- cluster.any_host(insert_events).result()-- person_data = [- (team_id, person_a, json.dumps({"$browser": "Safari"}), 1, now - timedelta(hours=4)),- (team_id, person_b, json.dumps({"email": "[email protected]"}), 1, now - timedelta(hours=4)),- ]-- def insert_persons(client: Client) -> None:- client.execute(- """INSERT INTO person (team_id, id, properties, version, _timestamp)- VALUES""",- person_data,- )-- cluster.any_host(insert_persons).result()-- affected_ids = get_affected_person_ids_from_clickhouse(- team_id=team_id,- bug_window_start=bug_window_start.strftime("%Y-%m-%d %H:%M:%S"),- bug_window_end=bug_window_end.strftime("%Y-%m-%d %H:%M:%S"),- )-- assert str(person_b) in affected_ids, "Person with non-filtered property diff should be detected"- assert str(person_a) not in affected_ids, "Person with only filtered properties should be excluded"-- def test_affected_person_ids_includes_set_once_and_unset_with_non_filtered_keys(self, cluster: ClickhouseCluster):- """- Integration test: detection covers $set_once and $unset with non-filtered keys- that produce actual diffs against person state.-- person_set_once: $set_once for key not on person -> included- person_unset: $unset for key that exists on person -> included- """- team_id = 99981- person_set_once = UUID("cccc0000-0000-0000-0000-000000000001")- person_unset = UUID("dddd0000-0000-0000-0000-000000000002")- now = datetime.now().replace(microsecond=0)- bug_window_start = now - timedelta(hours=3)- bug_window_end = now - timedelta(hours=1)- event_ts = now - timedelta(hours=2)-- events = [- (- team_id,- "set_once_user",- person_set_once,- event_ts,- json.dumps({"$set_once": {"initial_campaign": "summer_sale"}}),- ),- (- team_id,- "unset_user",- person_unset,- event_ts,- json.dumps({"$unset": ["email"]}),- ),- ]-- def insert_events(client: Client) -> None:- client.execute(- """INSERT INTO writable_events (team_id, distinct_id, person_id, timestamp, properties)- VALUES""",- events,- )-- cluster.any_host(insert_events).result()-- person_data = [- # person_set_once: does NOT have "initial_campaign" -> $set_once will produce a diff- (team_id, person_set_once, json.dumps({"name": "Alice"}), 1, now - timedelta(hours=4)),- # person_unset: HAS "email" -> $unset will produce a diff- (team_id, person_unset, json.dumps({"email": "[email protected]"}), 1, now - timedelta(hours=4)),- ]-- def insert_persons(client: Client) -> None:- client.execute(- """INSERT INTO person (team_id, id, properties, version, _timestamp)- VALUES""",- person_data,- )-- cluster.any_host(insert_persons).result()-- affected_ids = get_affected_person_ids_from_clickhouse(- team_id=team_id,- bug_window_start=bug_window_start.strftime("%Y-%m-%d %H:%M:%S"),- bug_window_end=bug_window_end.strftime("%Y-%m-%d %H:%M:%S"),- )-- assert str(person_set_once) in affected_ids, "$set_once with non-filtered key should be detected"- assert str(person_unset) in affected_ids, "$unset with non-filtered key should be detected"-- def test_affected_person_ids_excludes_persons_whose_properties_already_match(self, cluster: ClickhouseCluster):- """- Integration test: persons whose events set non-filtered properties to values- that already match the current person state should NOT be returned.-- This is the core fix for the 100x over-selection bug: the old query returned- anyone with property-setting events, the new query compares against person- state and only returns persons with actual diffs.-- person_match: $set email to same value already on person -> excluded- person_diff: $set email to different value -> included- person_set_once_exists: $set_once for key already on person -> excluded- person_unset_missing: $unset for key NOT on person -> excluded- """- team_id = 99983- person_match = UUID("ff000000-0000-0000-0000-000000000001")- person_diff = UUID("ff000000-0000-0000-0000-000000000002")- person_set_once_exists = UUID("ff000000-0000-0000-0000-000000000003")- person_unset_missing = UUID("ff000000-0000-0000-0000-000000000004")- now = datetime.now().replace(microsecond=0)- bug_window_start = now - timedelta(hours=3)- bug_window_end = now - timedelta(hours=1)- event_ts = now - timedelta(hours=2)-- events = [- # person_match: $set email to value that already matches person state- (team_id, "match_user", person_match, event_ts, json.dumps({"$set": {"email": "[email protected]"}})),- # person_diff: $set email to different value- (team_id, "diff_user", person_diff, event_ts, json.dumps({"$set": {"email": "[email protected]"}})),- # person_set_once_exists: $set_once for key already on person (no-op)- (- team_id,- "set_once_exists_user",- person_set_once_exists,- event_ts,- json.dumps({"$set_once": {"signup_source": "google"}}),- ),- # person_unset_missing: $unset for key not on person (no-op)- (- team_id,- "unset_missing_user",- person_unset_missing,- event_ts,- json.dumps({"$unset": ["nonexistent_key"]}),- ),- ]-- def insert_events(client: Client) -> None:- client.execute(- """INSERT INTO writable_events (team_id, distinct_id, person_id, timestamp, properties)- VALUES""",- events,- )-- cluster.any_host(insert_events).result()-- person_data = [- # person_match: email already matches the event value- (team_id, person_match, json.dumps({"email": "[email protected]"}), 1, now - timedelta(hours=4)),- # person_diff: email differs from event value- (team_id, person_diff, json.dumps({"email": "[email protected]"}), 1, now - timedelta(hours=4)),- # person_set_once_exists: already has the key -> $set_once is a no-op- (team_id, person_set_once_exists, json.dumps({"signup_source": "facebook"}), 1, now - timedelta(hours=4)),- # person_unset_missing: does NOT have the key -> $unset is a no-op- (team_id, person_unset_missing, json.dumps({"name": "Alice"}), 1, now - timedelta(hours=4)),- ]-- def insert_persons(client: Client) -> None:- client.execute(- """INSERT INTO person (team_id, id, properties, version, _timestamp)- VALUES""",- person_data,- )-- cluster.any_host(insert_persons).result()-- affected_ids = get_affected_person_ids_from_clickhouse(- team_id=team_id,- bug_window_start=bug_window_start.strftime("%Y-%m-%d %H:%M:%S"),- bug_window_end=bug_window_end.strftime("%Y-%m-%d %H:%M:%S"),- )-- assert str(person_diff) in affected_ids, "$set with different value should be detected"- assert str(person_match) not in affected_ids, "$set with matching value should be excluded"- assert str(person_set_once_exists) not in affected_ids, "$set_once for existing key should be excluded"- assert str(person_unset_missing) not in affected_ids, "$unset for missing key should be excluded"-- def test_windowed_query_produces_same_result_as_single_query(self, cluster: ClickhouseCluster):- """- Integration test: windowed queries should produce equivalent results to single query.-- This test inserts events spread across multiple time windows with $set, $set_once,- and $unset operations, then verifies that:- 1. A single query returns the expected merged result- 2. A windowed query (multiple smaller queries merged) returns the same result-- The CH query behavior:- - $set: only returns keys that EXIST in person AND have different values- - $set_once: only returns keys that DON'T exist in person- - $unset: only returns keys that EXIST in person- """- team_id = 99930- person_id = UUID("11111111-1111-1111-1111-000000000030")- now = datetime.now().replace(microsecond=0)- bug_window_start = now - timedelta(hours=6)-- # Events spread across 6 hours (will use 2-hour windows = 3 windows)- # Window 1: hour 0-2- # - $set email to "[email protected]" at hour 1- # - $set name to "Name1" at hour 1- # - $set_once signup_source to "source1" at hour 1- # Window 2: hour 2-4- # - $set email to "[email protected]" at hour 3 (newer, should win)- # - $set_once signup_source to "source2" at hour 3 (older should win - source1)- # Window 3: hour 4-6- # - $unset email at hour 5 (newest, should remove email from set)- # - $set name to "Name3" at hour 5 (newest, should win)-- events = [- # Window 1 events (hour 1)- (- team_id,- "distinct_id_1",- person_id,- bug_window_start + timedelta(hours=1),- json.dumps(- {- "$set": {"email": "[email protected]", "name": "Name1"},- "$set_once": {"signup_source": "source1"},- }- ),- ),- # Window 2 events (hour 3)- (- team_id,- "distinct_id_1",- person_id,- bug_window_start + timedelta(hours=3),- json.dumps({"$set": {"email": "[email protected]"}, "$set_once": {"signup_source": "source2"}}),- ),- # Window 3 events (hour 5)- (- team_id,- "distinct_id_1",- person_id,- bug_window_start + timedelta(hours=5),- json.dumps({"$unset": ["email"], "$set": {"name": "Name3"}}),- ),- ]-- def insert_events(client: Client) -> None:- client.execute(- """INSERT INTO writable_events (team_id, distinct_id, person_id, timestamp, properties)- VALUES""",- events,- )-- cluster.any_host(insert_events).result()-- # Person has email and name (so $set/$unset can modify them)- # Person does NOT have signup_source (so $set_once will add it)- person_data = [- (- team_id,- person_id,- json.dumps({"email": "[email protected]", "name": "OldName"}),- 1,- now - timedelta(hours=7),- )- ]-- def insert_person(client: Client) -> None:- client.execute(- """INSERT INTO person (team_id, id, properties, version, _timestamp)- VALUES""",- person_data,- )-- cluster.any_host(insert_person).result()-- bug_window_start_str = bug_window_start.strftime("%Y-%m-%d %H:%M:%S")-- # Run single query (current behavior)- single_results = get_person_property_updates_from_clickhouse(- team_id=team_id,- bug_window_start=bug_window_start_str,- )-- # Run windowed query with 2-hour windows- windowed_results = get_person_property_updates_windowed(- team_id=team_id,- bug_window_start=bug_window_start_str,- window_seconds=7200, # 2 hours- )-- # Both should return exactly one person- assert len(single_results) == 1- assert len(windowed_results) == 1-- single_diff = single_results[0]- windowed_diff = windowed_results[0]-- # Both should have the same person_id- assert single_diff.person_id == windowed_diff.person_id == str(person_id)-- # Core assertion: windowed query should produce equivalent results to single query- # The key properties we expect (based on event data):- # - name: $set to "Name3" at hour 5 (argMax picks latest)- # - email: $unset at hour 5 (person has email, events want to unset)-- # Verify both queries return the same keys in each category- assert set(single_diff.set_updates.keys()) == set(windowed_diff.set_updates.keys()), (- f"set_updates keys differ: single={set(single_diff.set_updates.keys())}, "- f"windowed={set(windowed_diff.set_updates.keys())}"- )- assert set(single_diff.set_once_updates.keys()) == set(windowed_diff.set_once_updates.keys()), (- f"set_once_updates keys differ: single={set(single_diff.set_once_updates.keys())}, "- f"windowed={set(windowed_diff.set_once_updates.keys())}"- )- assert set(single_diff.unset_updates.keys()) == set(windowed_diff.unset_updates.keys()), (- f"unset_updates keys differ: single={set(single_diff.unset_updates.keys())}, "- f"windowed={set(windowed_diff.unset_updates.keys())}"- )-- # Verify values match for keys that exist- for key in single_diff.set_updates:- assert single_diff.set_updates[key].value == windowed_diff.set_updates[key].value, (- f"set_updates[{key}] value differs"- )-- for key in single_diff.set_once_updates:- assert single_diff.set_once_updates[key].value == windowed_diff.set_once_updates[key].value, (- f"set_once_updates[{key}] value differs"- )-- # Verify expected properties based on test data- assert "name" in single_diff.set_updates, "name should be in set_updates"- assert single_diff.set_updates["name"].value == "Name3", "name should be Name3"- assert "email" in single_diff.unset_updates, "email should be in unset_updates"-- def test_windowed_batched_produces_same_result_as_single_query(self, cluster: ClickhouseCluster):- """- Integration test: the production windowed-batched path- (get_person_property_updates_windowed_batched) must produce identical- diffs to the non-windowed single-query path.-- This test includes:- - Events with both filtered and non-filtered properties- - Events spread across multiple time windows- - A person with ONLY filtered-property events (should be excluded by both)- - A person with non-filtered properties that MATCH current state (no diff, excluded by both)- - $set, $set_once, and $unset operations- """- team_id = 99982- person_real = UUID("eeee0000-0000-0000-0000-000000000001")- person_filtered_only = UUID("eeee0000-0000-0000-0000-000000000002")- person_no_diff = UUID("eeee0000-0000-0000-0000-000000000003")- now = datetime.now().replace(microsecond=0)- bug_window_start = now - timedelta(hours=6)- bug_window_end = now - timedelta(minutes=5)-- events = [- # Person with real (non-filtered) property updates across windows- # Window 1 (hour 1): $set email, $set_once signup_source, plus filtered props- (- team_id,- "real_user",- person_real,- bug_window_start + timedelta(hours=1),- json.dumps(- {- "$set": {- "email": "[email protected]",- "name": "Name1",- "$browser": "Chrome",- "$os": "Mac OS X",- },- "$set_once": {"signup_source": "google"},- }- ),- ),- # Window 2 (hour 3): $set email overwrites, $set_once ignored (first wins)- (- team_id,- "real_user",- person_real,- bug_window_start + timedelta(hours=3),- json.dumps(- {- "$set": {"email": "[email protected]"},- "$set_once": {"signup_source": "bing"},- }- ),- ),- # Window 3 (hour 5): $unset email, $set name overwrites- (- team_id,- "real_user",- person_real,- bug_window_start + timedelta(hours=5),- json.dumps(- {- "$unset": ["email"],- "$set": {"name": "Name3"},- }- ),- ),- # Person with ONLY filtered-property events (should not appear in results)- (- team_id,- "filtered_user",- person_filtered_only,- bug_window_start + timedelta(hours=2),- json.dumps(- {- "$set": {- "$browser": "Firefox",- "$os": "Windows",- "$current_url": "https://example.com",- },- }- ),- ),- # Person with non-filtered properties that already match person state (no diff).- # This is the exact scenario causing 100x over-selection in production:- # the old affected persons query returned these, the new one should not.- (- team_id,- "no_diff_user",- person_no_diff,- bug_window_start + timedelta(hours=2),- json.dumps(- {- "$set": {"plan": "enterprise", "company": "Acme Corp"},- "$set_once": {"signup_source": "organic"},- }- ),- ),- ]-- def insert_events(client: Client) -> None:- client.execute(- """INSERT INTO writable_events (team_id, distinct_id, person_id, timestamp, properties)- VALUES""",- events,- )-- cluster.any_host(insert_events).result()-- person_data = [- (- team_id,- person_real,- json.dumps({"email": "[email protected]", "name": "OldName"}),- 1,- now - timedelta(hours=7),- ),- (- team_id,- person_filtered_only,- json.dumps({"$browser": "Safari", "$os": "Linux"}),- 1,- now - timedelta(hours=7),- ),- # person_no_diff: properties already match what the events would set- (- team_id,- person_no_diff,- json.dumps({"plan": "enterprise", "company": "Acme Corp", "signup_source": "organic"}),- 1,- now - timedelta(hours=7),- ),- ]-- def insert_persons(client: Client) -> None:- client.execute(- """INSERT INTO person (team_id, id, properties, version, _timestamp)- VALUES""",- person_data,- )-- cluster.any_host(insert_persons).result()-- bug_window_start_str = bug_window_start.strftime("%Y-%m-%d %H:%M:%S")- bug_window_end_str = bug_window_end.strftime("%Y-%m-%d %H:%M:%S")-- # Run non-windowed single query- single_results = get_person_property_updates_from_clickhouse(- team_id=team_id,- bug_window_start=bug_window_start_str,- )-- # Run windowed-batched (production path) with 2-hour windows- batched_results: list[PersonPropertyDiffs] = []- for batch in get_person_property_updates_windowed_batched(- team_id=team_id,- bug_window_start=bug_window_start_str,- bug_window_end=bug_window_end_str,- window_seconds=7200,- person_batch_size=100,- ):- batched_results.extend(batch)-- # Both should return exactly one person (the real one, not the others)- assert len(single_results) == 1, f"single query returned {len(single_results)}, expected 1"- assert len(batched_results) == 1, f"batched windowed returned {len(batched_results)}, expected 1"-- single_diff = single_results[0]- batched_diff = batched_results[0]-- # Same person_id- assert single_diff.person_id == batched_diff.person_id == str(person_real)-- # No-diff and filtered-only persons must not appear in either mode- result_person_ids = {r.person_id for r in single_results} | {r.person_id for r in batched_results}- assert str(person_filtered_only) not in result_person_ids- assert str(person_no_diff) not in result_person_ids, (- "Person whose properties already match event values should be excluded by both modes"- )-- # Same keys in each diff category- assert set(single_diff.set_updates.keys()) == set(batched_diff.set_updates.keys()), (- f"set_updates keys differ: single={set(single_diff.set_updates.keys())}, "- f"batched={set(batched_diff.set_updates.keys())}"- )- assert set(single_diff.set_once_updates.keys()) == set(batched_diff.set_once_updates.keys()), (- f"set_once_updates keys differ: single={set(single_diff.set_once_updates.keys())}, "- f"batched={set(batched_diff.set_once_updates.keys())}"- )- assert set(single_diff.unset_updates.keys()) == set(batched_diff.unset_updates.keys()), (- f"unset_updates keys differ: single={set(single_diff.unset_updates.keys())}, "- f"batched={set(batched_diff.unset_updates.keys())}"- )-- # Same values- for key in single_diff.set_updates:- assert single_diff.set_updates[key].value == batched_diff.set_updates[key].value, (- f"set_updates[{key}] value differs: single={single_diff.set_updates[key].value}, "- f"batched={batched_diff.set_updates[key].value}"- )-- for key in single_diff.set_once_updates:- assert single_diff.set_once_updates[key].value == batched_diff.set_once_updates[key].value, (- f"set_once_updates[{key}] value differs: single={single_diff.set_once_updates[key].value}, "- f"batched={batched_diff.set_once_updates[key].value}"- )-- # Verify expected properties: name set to Name3, email unset- assert "name" in single_diff.set_updates- assert single_diff.set_updates["name"].value == "Name3"- assert "email" in single_diff.unset_updates---# Exercises the persons DB directly (raw reads and writes against the persons writer), so it must-# run against the real persons DB rather than the personhog fake.[email protected]_db(transaction=True)[email protected]_db_direct-class TestBatchCommitsEndToEnd:- """End-to-end integration tests for batch commit functionality.-- These tests use real ClickHouse and Postgres connections to verify- that batch commits work correctly in a production-like environment.- """-- @pytest.fixture- def organization(self):- """Create a test organization."""- from posthog.models import Organization-- return Organization.objects.create(name="Batch Test Organization")-- @pytest.fixture- def team(self, organization):- """Create a test team."""- from posthog.models import Team-- return Team.objects.create(organization=organization, name="Batch Test Team")-- def test_batch_commits_end_to_end(self, cluster: ClickhouseCluster, team):- """- End-to-end test that verifies batch commits work with real databases.-- Creates 5 persons in Postgres, inserts events in ClickHouse,- runs reconciliation with batch_size=2, and verifies:- 1. All persons are updated in Postgres- 2. Commits happen in batches (3 batches: [2, 2, 1])- """- from posthog.dags.person_property_reconciliation import (- get_person_property_updates_from_clickhouse,- process_persons_in_batches,- )- from posthog.test.persons import create_person-- num_persons = 5- batch_size = 2-- # Use a unique team_id to avoid conflicts with other tests- team_id = team.id-- # Time setup- now = datetime.now().replace(microsecond=0)- bug_window_start = now - timedelta(days=10)- event_ts = now - timedelta(days=5)-- # Create persons in Postgres with old property values- persons = []- for i in range(num_persons):- person = create_person(- team_id=team_id,- properties={"email": f"old_{i}@example.com", "counter": i},- properties_last_updated_at={- "email": "2024-01-01T00:00:00+00:00",- "counter": "2024-01-01T00:00:00+00:00",- },- properties_last_operation={"email": "set", "counter": "set"},- version=1,- )- persons.append(person)-- # Insert events in ClickHouse with new property values- events = []- for i, person in enumerate(persons):- events.append(- (- team_id,- f"distinct_id_{i}",- person.uuid,- event_ts,- json.dumps({"$set": {"email": f"new_{i}@example.com"}}),- )- )-- def insert_events(client: Client) -> None:- client.execute(- """INSERT INTO writable_events (team_id, distinct_id, person_id, timestamp, properties)- VALUES""",- events,- )-- cluster.any_host(insert_events).result()-- # Insert persons in ClickHouse (required for the query join)- person_ch_data = []- for i, person in enumerate(persons):- person_ch_data.append(- (- team_id,- person.uuid,- json.dumps({"email": f"old_{i}@example.com", "counter": i}),- 1, # version- 0, # is_deleted- now - timedelta(days=8), # _timestamp- )- )-- def insert_persons_ch(client: Client) -> None:- client.execute(- """INSERT INTO person (team_id, id, properties, version, is_deleted, _timestamp)- VALUES""",- person_ch_data,- )-- cluster.any_host(insert_persons_ch).result()-- # Get person property updates from ClickHouse- person_property_updates = get_person_property_updates_from_clickhouse(- team_id=team_id,- bug_window_start=bug_window_start.strftime("%Y-%m-%d %H:%M:%S"),- )-- assert len(person_property_updates) == num_persons, (- f"Expected {num_persons} persons from ClickHouse query, got {len(person_property_updates)}"- )-- # Track batches- batch_sizes = []-- def on_batch_committed(batch_num: int, batch_persons: list[dict]) -> None:- batch_sizes.append(len(batch_persons))-- from posthog.persons_db import persons_db_connection-- # Non-autocommit: process_persons_in_batches expects a transactional cursor (its commit_fn drives commits).- with persons_db_connection(writer=True) as connection, connection.cursor() as cursor:- # Set up cursor settings- cursor.execute("SET application_name = 'test_batch_commits'")-- result = process_persons_in_batches(- person_property_diffs=person_property_updates,- cursor=cursor,- job_id="test-batch-job",- team_id=team_id,- batch_size=batch_size,- dry_run=False,- backup_enabled=False,- commit_fn=lambda: None, # No-op commit in tests (Django manages transaction)- on_batch_committed=on_batch_committed,- )-- # Verify all persons were processed- assert result.total_processed == num_persons, f"Expected {num_persons} processed, got {result.total_processed}"- assert result.total_updated == num_persons, f"Expected {num_persons} updated, got {result.total_updated}"- assert result.total_skipped == 0, f"Expected 0 skipped, got {result.total_skipped}"-- # Verify batch sizes: [2, 2, 1] for 5 persons with batch_size=2- assert result.total_commits == 3, f"Expected 3 commits, got {result.total_commits}"- assert batch_sizes == [2, 2, 1], f"Expected batch sizes [2, 2, 1], got {batch_sizes}"-- # Verify Postgres was actually updated with correct properties and metadata- for i, person in enumerate(persons):- refresh_person_from_persons_db(person)-- # Property value should be updated- assert person.properties["email"] == f"new_{i}@example.com", (- f"Person {i} email not updated. Expected 'new_{i}@example.com', got '{person.properties.get('email')}'"- )-- # Counter should be unchanged (wasn't in the update)- assert person.properties["counter"] == i, (- f"Person {i} counter changed unexpectedly. Expected {i}, got {person.properties.get('counter')}"- )-- # Version should be incremented- assert person.version == 2, f"Person {i} version not incremented. Expected 2, got {person.version}"-- # properties_last_updated_at should have new timestamp for email- assert person.properties_last_updated_at is not None- assert "email" in person.properties_last_updated_at, (- f"Person {i} properties_last_updated_at missing 'email' key"- )- # The timestamp should be from the event (event_ts), not the old value- email_updated_at = person.properties_last_updated_at["email"]- assert email_updated_at != "2024-01-01T00:00:00+00:00", (- f"Person {i} email timestamp not updated. Still has old value: {email_updated_at}"- )-- # properties_last_operation should be 'set' for email- assert person.properties_last_operation is not None- assert person.properties_last_operation.get("email") == "set", (- f"Person {i} properties_last_operation['email'] should be 'set', "- f"got '{person.properties_last_operation.get('email')}'"- )-- # Counter's metadata should be unchanged- assert person.properties_last_updated_at.get("counter") == "2024-01-01T00:00:00+00:00", (- f"Person {i} counter timestamp changed unexpectedly"- )-- def test_batch_commits_with_missing_person(self, cluster: ClickhouseCluster, team):- """- Test that missing persons in Postgres are skipped gracefully.-- Creates 4 persons in ClickHouse events but only 3 in Postgres.- Verifies the missing person is skipped and others are updated.- """- from posthog.dags.person_property_reconciliation import (- get_person_property_updates_from_clickhouse,- process_persons_in_batches,- )- from posthog.test.persons import create_person-- team_id = team.id-- now = datetime.now().replace(microsecond=0)- bug_window_start = now - timedelta(days=10)- event_ts = now - timedelta(days=5)-- # Create only 3 persons in Postgres- persons = []- for i in range(3):- person = create_person(- team_id=team_id,- properties={"name": f"old_name_{i}"},- properties_last_updated_at={"name": "2024-01-01T00:00:00+00:00"},- properties_last_operation={"name": "set"},- version=1,- )- persons.append(person)-- # Create a 4th UUID that won't exist in Postgres- missing_uuid = UUID("99999999-9999-9999-9999-999999999999")-- # Insert 4 events in ClickHouse (including one for missing person)- events = []- for i, person in enumerate(persons):- events.append(- (- team_id,- f"distinct_id_{i}",- person.uuid,- event_ts,- json.dumps({"$set": {"name": f"new_name_{i}"}}),- )- )- # Add event for missing person- events.append(- (- team_id,- "distinct_id_missing",- missing_uuid,- event_ts,- json.dumps({"$set": {"name": "new_name_missing"}}),- )- )-- def insert_events(client: Client) -> None:- client.execute(- """INSERT INTO writable_events (team_id, distinct_id, person_id, timestamp, properties)- VALUES""",- events,- )-- cluster.any_host(insert_events).result()-- # Insert 4 persons in ClickHouse (including missing one)- person_ch_data = []- for i, person in enumerate(persons):- person_ch_data.append(- (- team_id,- person.uuid,- json.dumps({"name": f"old_name_{i}"}),- 1,- 0,- now - timedelta(days=8),- )- )- # Add the missing person to ClickHouse- person_ch_data.append(- (- team_id,- missing_uuid,- json.dumps({"name": "old_name_missing"}),- 1,- 0,- now - timedelta(days=8),- )- )-- def insert_persons_ch(client: Client) -> None:- client.execute(- """INSERT INTO person (team_id, id, properties, version, is_deleted, _timestamp)- VALUES""",- person_ch_data,- )-- cluster.any_host(insert_persons_ch).result()-- # Get updates from ClickHouse - should return 4 persons- person_property_updates = get_person_property_updates_from_clickhouse(- team_id=team_id,- bug_window_start=bug_window_start.strftime("%Y-%m-%d %H:%M:%S"),- )-- assert len(person_property_updates) == 4-- batch_sizes = []-- def on_batch_committed(batch_num: int, batch_persons: list[dict]) -> None:- batch_sizes.append(len(batch_persons))-- from posthog.persons_db import persons_db_connection-- # Non-autocommit: process_persons_in_batches expects a transactional cursor (its commit_fn drives commits).- with persons_db_connection(writer=True) as connection, connection.cursor() as cursor:- result = process_persons_in_batches(- person_property_diffs=person_property_updates,- cursor=cursor,- job_id="test-missing-person-job",- team_id=team_id,- batch_size=2,- dry_run=False,- backup_enabled=False,- commit_fn=lambda: None, # No-op commit in tests (Django manages transaction)- on_batch_committed=on_batch_committed,- )-- # 4 persons processed, 1 skipped (missing from Postgres), 3 updated- assert result.total_processed == 4- assert result.total_skipped == 1 # The missing person- assert result.total_updated == 3-- # Verify existing persons were updated with correct properties and metadata- for i, person in enumerate(persons):- refresh_person_from_persons_db(person)-- # Property value should be updated- assert person.properties["name"] == f"new_name_{i}", (- f"Person {i} name not updated. Expected 'new_name_{i}', got '{person.properties.get('name')}'"- )-- # Version should be incremented- assert person.version == 2, f"Person {i} version not incremented. Expected 2, got {person.version}"-- # properties_last_updated_at should have new timestamp- assert person.properties_last_updated_at is not None- assert "name" in person.properties_last_updated_at, (- f"Person {i} properties_last_updated_at missing 'name' key"- )- name_updated_at = person.properties_last_updated_at["name"]- assert name_updated_at != "2024-01-01T00:00:00+00:00", (- f"Person {i} name timestamp not updated. Still has old value: {name_updated_at}"- )-- # properties_last_operation should be 'set'- assert person.properties_last_operation is not None- assert person.properties_last_operation.get("name") == "set", (- f"Person {i} properties_last_operation['name'] should be 'set', "- f"got '{person.properties_last_operation.get('name')}'"- )--[email protected]_db(transaction=True)[email protected]_db_direct-class TestKafkaClickHouseRoundTrip:- """Integration tests that verify person updates flow through Kafka to ClickHouse.-- These tests use real Kafka (not mocked) and verify that:- 1. publish_person_to_kafka produces messages in the correct format- 2. ClickHouse's Kafka engine consumes the messages- 3. The person table in ClickHouse has the correct data-- This ensures the reconciliation job's Kafka message format is compatible- with ClickHouse's expectations (matching Node.js ingestion format).- """-- @pytest.fixture- def organization(self):- """Create a test organization."""- from posthog.models import Organization-- return Organization.objects.create(name="Kafka Test Organization")-- @pytest.fixture- def team(self, organization):- """Create a test team."""- from posthog.models import Team-- return Team.objects.create(organization=organization, name="Kafka Test Team")-- def _wait_for_clickhouse_person(- self,- team_id: int,- person_uuid: str,- expected_version: int,- max_wait_seconds: int = 30,- poll_interval_seconds: float = 0.5,- ) -> dict | None:- """- Poll ClickHouse until the person appears with the expected version.-- Returns the person row as a dict, or None if not found within timeout.- """- import time-- from posthog.clickhouse.client import sync_execute-- start_time = time.time()- while time.time() - start_time < max_wait_seconds:- rows = sync_execute(- """- SELECT id, team_id, properties, is_identified, version, is_deleted, created_at- FROM person FINAL- WHERE team_id = %(team_id)s AND id = %(person_uuid)s- """,- {"team_id": team_id, "person_uuid": person_uuid},- )- if rows:- row = rows[0]- person_data = {- "id": str(row[0]),- "team_id": row[1],- "properties": row[2],- "is_identified": row[3],- "version": row[4],- "is_deleted": row[5],- "created_at": row[6],- }- if person_data["version"] >= expected_version:- return person_data- time.sleep(poll_interval_seconds)- return None-- @pytest.mark.skipif(- os.environ.get("KAFKA_ROUNDTRIP_TESTS") != "1",- reason="Requires real Kafka infrastructure. Set KAFKA_ROUNDTRIP_TESTS=1 to run.",- )- def test_publish_person_to_kafka_updates_clickhouse(self, team):- """- Test that publish_person_to_kafka produces messages that ClickHouse consumes correctly.-- This verifies the full round-trip:- 1. Create a person in Postgres- 2. Publish to Kafka using publish_person_to_kafka (with real Kafka, not mocked)- 3. Wait for ClickHouse to consume the message- 4. Verify ClickHouse has the correct person data- """- from django.test import override_settings-- from posthog.dags.person_property_reconciliation import publish_person_to_kafka- from posthog.kafka_client.client import _KafkaProducer- from posthog.test.persons import create_person-- # Create person in Postgres- person = create_person(- team_id=team.id,- properties={"email": "[email protected]", "name": "Kafka Test User"},- version=1,- is_identified=True,- )-- # Prepare person data for Kafka (simulating what reconciliation job does)- person_data = {- "id": person.uuid,- "team_id": team.id,- "properties": {"email": "[email protected]", "name": "Kafka Test User"},- "is_identified": True,- "is_deleted": 0,- "created_at": person.created_at,- "version": 1,- }-- # Create a real Kafka producer (not mocked)- with override_settings(TEST=False):- producer = _KafkaProducer(test=False)- try:- publish_person_to_kafka(person_data, producer)- producer.flush(timeout=10)- finally:- producer.close()-- # Wait for ClickHouse to consume the message- ch_person = self._wait_for_clickhouse_person(- team_id=team.id,- person_uuid=str(person.uuid),- expected_version=1,- max_wait_seconds=30,- )-- # Verify ClickHouse received the person- assert ch_person is not None, f"Person {person.uuid} not found in ClickHouse after 30 seconds"- assert ch_person["team_id"] == team.id- assert ch_person["version"] == 1- assert ch_person["is_identified"] == 1 # ClickHouse stores as Int8-- # Parse properties (stored as JSON string in ClickHouse)- ch_properties = json.loads(ch_person["properties"])- assert ch_properties["email"] == "[email protected]"- assert ch_properties["name"] == "Kafka Test User"-- @pytest.mark.skipif(- os.environ.get("KAFKA_ROUNDTRIP_TESTS") != "1",- reason="Requires real Kafka infrastructure. Set KAFKA_ROUNDTRIP_TESTS=1 to run.",- )- def test_reconciliation_kafka_message_format_matches_nodejs(self, team, cluster: ClickhouseCluster):- """- Test that the reconciliation job's Kafka message format produces the same- ClickHouse state as Node.js ingestion would.-- This is important because both systems publish to the same Kafka topic,- and ClickHouse must be able to handle messages from both sources.- """- from django.test import override_settings-- from posthog.dags.person_property_reconciliation import publish_person_to_kafka- from posthog.kafka_client.client import _KafkaProducer- from posthog.test.persons import create_person-- # Create person in Postgres with version 1- person = create_person(- team_id=team.id,- properties={"email": "[email protected]"},- version=1,- is_identified=False,- )-- # First, publish version 1 to establish baseline- person_data_v1 = {- "id": person.uuid,- "team_id": team.id,- "properties": {"email": "[email protected]"},- "is_identified": False,- "is_deleted": 0,- "created_at": person.created_at,- "version": 1,- }-- with override_settings(TEST=False):- producer = _KafkaProducer(test=False)- try:- publish_person_to_kafka(person_data_v1, producer)- producer.flush(timeout=10)- finally:- producer.close()-- # Wait for version 1 in ClickHouse- ch_person_v1 = self._wait_for_clickhouse_person(- team_id=team.id,- person_uuid=str(person.uuid),- expected_version=1,- )- assert ch_person_v1 is not None, "Version 1 not found in ClickHouse"-- # Now simulate reconciliation updating the person to version 2- person_data_v2 = {- "id": person.uuid,- "team_id": team.id,- "properties": {"email": "[email protected]", "source": "reconciliation"},- "is_identified": True,- "is_deleted": 0,- "created_at": person.created_at,- "version": 2,- }-- with override_settings(TEST=False):- producer = _KafkaProducer(test=False)- try:- publish_person_to_kafka(person_data_v2, producer)- producer.flush(timeout=10)- finally:- producer.close()-- # Wait for version 2 in ClickHouse- ch_person_v2 = self._wait_for_clickhouse_person(- team_id=team.id,- person_uuid=str(person.uuid),- expected_version=2,- )-- # Verify ClickHouse received the update- assert ch_person_v2 is not None, "Version 2 not found in ClickHouse after 30 seconds"- assert ch_person_v2["version"] == 2- assert ch_person_v2["is_identified"] == 1-- ch_properties = json.loads(ch_person_v2["properties"])- assert ch_properties["email"] == "[email protected]"- assert ch_properties["source"] == "reconciliation"-- @pytest.mark.skipif(- os.environ.get("KAFKA_ROUNDTRIP_TESTS") != "1",- reason="Requires real Kafka infrastructure. Set KAFKA_ROUNDTRIP_TESTS=1 to run.",- )- def test_full_dagster_job_with_real_kafka(self, team, cluster: ClickhouseCluster):- """- Test the full Dagster reconciliation job with real Kafka.-- This is the most comprehensive test - it:- 1. Creates persons in Postgres with outdated properties- 2. Inserts events in ClickHouse with updated properties- 3. Runs the actual Dagster job with real Kafka producer- 4. Waits for ClickHouse to consume the Kafka messages- 5. Verifies both Postgres AND ClickHouse have the correct data- """- from django.test import override_settings-- from posthog.dags.person_property_reconciliation import person_property_reconciliation_job- from posthog.kafka_client.client import _KafkaProducer- from posthog.test.persons import create_person-- # Time setup - create a bug window that includes our test events- now = datetime.now().replace(microsecond=0)- bug_window_start = (now - timedelta(days=10)).strftime("%Y-%m-%d %H:%M:%S")- bug_window_end = (now + timedelta(days=1)).strftime("%Y-%m-%d %H:%M:%S")- event_ts = now - timedelta(days=5)-- # Create person in Postgres with old property value- person = create_person(- team_id=team.id,- properties={"email": "[email protected]", "unchanged": "value"},- properties_last_updated_at={- "email": "2024-01-01T00:00:00+00:00",- "unchanged": "2024-01-01T00:00:00+00:00",- },- properties_last_operation={"email": "set", "unchanged": "set"},- version=1,- )-- # Insert event in ClickHouse with new property value- def insert_event(client: Client) -> None:- client.execute(- """INSERT INTO writable_events (team_id, distinct_id, person_id, timestamp, properties)- VALUES""",- [- (- team.id,- "test_distinct_id",- person.uuid,- event_ts,- json.dumps({"$set": {"email": "[email protected]"}}),- )- ],- )-- cluster.any_host(insert_event).result()-- # Insert person in ClickHouse (required for the reconciliation query join)- def insert_person_ch(client: Client) -> None:- client.execute(- """INSERT INTO person (team_id, id, properties, version, is_deleted, _timestamp)- VALUES""",- [- (- team.id,- person.uuid,- json.dumps({"email": "[email protected]", "unchanged": "value"}),- 1,- 0,- now - timedelta(days=8),- )- ],- )-- cluster.any_host(insert_person_ch).result()-- from posthog.persons_db import persons_db_connection-- # Create real Kafka producer. Non-autocommit: the dagster job drives commit()/rollback() on this resource.- with persons_db_connection(writer=True) as persons_conn, override_settings(TEST=False):- kafka_producer = _KafkaProducer(test=False)-- try:- # Run the actual Dagster job- result = person_property_reconciliation_job.execute_in_process(- run_config={- "ops": {- "get_team_ids_to_reconcile": {- "config": {- "team_ids": [team.id],- "bug_window_start": bug_window_start,- "bug_window_end": bug_window_end,- "dry_run": False,- "backup_enabled": False,- "batch_size": 100,- }- },- "reconcile_team_chunk": {- "config": {- "bug_window_start": bug_window_start,- "dry_run": False,- "backup_enabled": False,- "batch_size": 100,- }- },- }- },- resources={- "cluster": cluster,- "persons_database": persons_conn,- "kafka_producer": kafka_producer,- },- )-- assert result.success, f"Dagster job failed: {result}"-- # Flush kafka to ensure all messages are sent- kafka_producer.flush(timeout=10)-- finally:- kafka_producer.close()-- # Verify Postgres was updated- refresh_person_from_persons_db(person)- assert person.properties["email"] == "[email protected]", (- f"Postgres email not updated. Expected '[email protected]', got '{person.properties.get('email')}'"- )- assert person.properties["unchanged"] == "value", "Unchanged property was modified"- assert person.version == 2, f"Postgres version not incremented. Expected 2, got {person.version}"-- # Wait for ClickHouse to consume the Kafka message- ch_person = self._wait_for_clickhouse_person(- team_id=team.id,- person_uuid=str(person.uuid),- expected_version=2,- max_wait_seconds=30,- )-- # Verify ClickHouse received the update via Kafka- assert ch_person is not None, (- f"Person {person.uuid} version 2 not found in ClickHouse after 30 seconds. "- "This suggests the Kafka message format may not be compatible with ClickHouse."- )- assert ch_person["version"] == 2-- ch_properties = json.loads(ch_person["properties"])- assert ch_properties["email"] == "[email protected]", (- f"ClickHouse email not updated. Expected '[email protected]', got '{ch_properties.get('email')}'"- )- assert ch_properties["unchanged"] == "value", "ClickHouse unchanged property was modified"-- @pytest.mark.skipif(- os.environ.get("KAFKA_ROUNDTRIP_TESTS") != "1",- reason="Requires real Kafka infrastructure. Set KAFKA_ROUNDTRIP_TESTS=1 to run.",- )- def test_full_job_timestamp_window_filtering_all_permutations(self, cluster: ClickhouseCluster):- """End-to-end test of the full Dagster job with all timestamp permutations.-- Tests the complete pipeline:- 1. Creates persons in Postgres with different _timestamp values- 2. Inserts events in ClickHouse with different timestamps- 3. Runs the full Dagster reconciliation job- 4. Verifies only the correct persons were updated in Postgres-- Test matrix (person _timestamp × event timestamp):- | Person _timestamp | Event timestamp | Should be reconciled? |- |-------------------|-----------------|----------------------|- | BEFORE | BEFORE | NO |- | BEFORE | DURING | YES |- | BEFORE | AFTER | YES |- | DURING | BEFORE | NO |- | DURING | DURING | YES |- | DURING | AFTER | YES |- | AFTER | BEFORE | NO |- | AFTER | DURING | YES |- | AFTER | AFTER | YES |-- Key insight: Only EVENT timestamp matters, not person _timestamp.- Events BEFORE bug_window_start are filtered; events DURING or AFTER are included.- """- from django.test import override_settings-- from posthog.dags.person_property_reconciliation import person_property_reconciliation_job- from posthog.kafka_client.client import _KafkaProducer- from posthog.models import Organization, Team- from posthog.test.persons import create_person-- # Create two organizations and teams for isolation- org1 = Organization.objects.create(name="Test Org 1 for timestamp permutations")- org2 = Organization.objects.create(name="Test Org 2 for timestamp permutations")- team1 = Team.objects.create(organization=org1, name="Team 1")- team2 = Team.objects.create(organization=org2, name="Team 2")-- now = datetime.now().replace(microsecond=0)-- # Define time periods- bug_window_start = now - timedelta(days=10)- bug_window_end = now - timedelta(days=5)-- # Timestamps for each period- ts_before = now - timedelta(days=15) # Before bug_window_start- ts_during = now - timedelta(days=7) # Between bug_window_start and bug_window_end- ts_after = now - timedelta(days=2) # After bug_window_end but before now()-- # Test cases for team 1: all 9 permutations- # Format: (suffix, person_ts, event_ts, should_be_reconciled, prop_name)- test_cases_team1 = [- # Person _timestamp BEFORE bug_window_start- ("t1_001", ts_before, ts_before, False, "prop_before_before"),- ("t1_002", ts_before, ts_during, True, "prop_before_during"),- ("t1_003", ts_before, ts_after, True, "prop_before_after"),- # Person _timestamp DURING bug_window- ("t1_004", ts_during, ts_before, False, "prop_during_before"),- ("t1_005", ts_during, ts_during, True, "prop_during_during"),- ("t1_006", ts_during, ts_after, True, "prop_during_after"),- # Person _timestamp AFTER bug_window_end- ("t1_007", ts_after, ts_before, False, "prop_after_before"),- ("t1_008", ts_after, ts_during, True, "prop_after_during"),- ("t1_009", ts_after, ts_after, True, "prop_after_after"),- ]-- # Test cases for team 2: subset to verify team isolation- test_cases_team2 = [- ("t2_001", ts_before, ts_during, True, "team2_prop_1"),- ("t2_002", ts_during, ts_after, True, "team2_prop_2"),- ("t2_003", ts_after, ts_before, False, "team2_prop_3"),- ]-- # Create persons in Postgres for team 1- persons_team1 = {}- for suffix, _person_ts, _event_ts, _expected, prop_name in test_cases_team1:- person = create_person(- team_id=team1.id,- properties={prop_name: "old_value"},- properties_last_updated_at={prop_name: "2020-01-01T00:00:00+00:00"},- properties_last_operation={prop_name: "set"},- version=1,- )- persons_team1[suffix] = person-- # Create persons in Postgres for team 2- persons_team2 = {}- for suffix, _person_ts, _event_ts, _expected, prop_name in test_cases_team2:- person = create_person(- team_id=team2.id,- properties={prop_name: "old_value"},- properties_last_updated_at={prop_name: "2020-01-01T00:00:00+00:00"},- properties_last_operation={prop_name: "set"},- version=1,- )- persons_team2[suffix] = person-- # Insert events in ClickHouse for team 1- events_team1 = []- for suffix, _person_ts, event_ts, _expected, prop_name in test_cases_team1:- person = persons_team1[suffix]- events_team1.append(- (- team1.id,- f"distinct_{suffix}",- person.uuid,- event_ts,- json.dumps({"$set": {prop_name: "new_value"}}),- )- )-- def insert_events_team1(client: Client) -> None:- client.execute(- """INSERT INTO writable_events (team_id, distinct_id, person_id, timestamp, properties)- VALUES""",- events_team1,- )-- cluster.any_host(insert_events_team1).result()-- # Insert events in ClickHouse for team 2- events_team2 = []- for suffix, _person_ts, event_ts, _expected, prop_name in test_cases_team2:- person = persons_team2[suffix]- events_team2.append(- (- team2.id,- f"distinct_{suffix}",- person.uuid,- event_ts,- json.dumps({"$set": {prop_name: "new_value"}}),- )- )-- def insert_events_team2(client: Client) -> None:- client.execute(- """INSERT INTO writable_events (team_id, distinct_id, person_id, timestamp, properties)- VALUES""",- events_team2,- )-- cluster.any_host(insert_events_team2).result()-- # Insert persons in ClickHouse for team 1 (required for the query join)- persons_ch_team1 = []- for suffix, person_ts, _event_ts, _expected, prop_name in test_cases_team1:- person = persons_team1[suffix]- persons_ch_team1.append(- (- team1.id,- person.uuid,- json.dumps({prop_name: "old_value"}),- 1,- 0, # is_deleted- person_ts, # _timestamp- )- )-- def insert_persons_ch_team1(client: Client) -> None:- client.execute(- """INSERT INTO person (team_id, id, properties, version, is_deleted, _timestamp)- VALUES""",- persons_ch_team1,- )-- cluster.any_host(insert_persons_ch_team1).result()-- # Insert persons in ClickHouse for team 2- persons_ch_team2 = []- for suffix, person_ts, _event_ts, _expected, prop_name in test_cases_team2:- person = persons_team2[suffix]- persons_ch_team2.append(- (- team2.id,- person.uuid,- json.dumps({prop_name: "old_value"}),- 1,- 0,- person_ts,- )- )-- def insert_persons_ch_team2(client: Client) -> None:- client.execute(- """INSERT INTO person (team_id, id, properties, version, is_deleted, _timestamp)- VALUES""",- persons_ch_team2,- )-- cluster.any_host(insert_persons_ch_team2).result()-- # Run the Dagster job- from posthog.persons_db import persons_db_connection-- # Non-autocommit: the dagster job drives commit()/rollback() on this resource connection.- with persons_db_connection(writer=True) as persons_conn, override_settings(TEST=False):- kafka_producer = _KafkaProducer(test=False)-- try:- result = person_property_reconciliation_job.execute_in_process(- run_config={- "ops": {- "get_team_ids_to_reconcile": {- "config": {- "team_ids": [team1.id, team2.id],- "bug_window_start": bug_window_start.strftime("%Y-%m-%d %H:%M:%S"),- "bug_window_end": bug_window_end.strftime("%Y-%m-%d %H:%M:%S"),- "dry_run": False,- "backup_enabled": False,- "batch_size": 100,- }- },- "reconcile_team_chunk": {- "config": {- "bug_window_start": bug_window_start.strftime("%Y-%m-%d %H:%M:%S"),- "dry_run": False,- "backup_enabled": False,- "batch_size": 100,- }- },- }- },- resources={- "cluster": cluster,- "persons_database": persons_conn,- "kafka_producer": kafka_producer,- },- )-- assert result.success, f"Dagster job failed: {result}"- kafka_producer.flush(timeout=10)-- finally:- kafka_producer.close()-- # Verify team 1 results in Postgres- for suffix, _person_ts, _event_ts, should_be_reconciled, prop_name in test_cases_team1:- person = persons_team1[suffix]- refresh_person_from_persons_db(person)-- if should_be_reconciled:- assert person.properties[prop_name] == "new_value", (- f"Person {suffix} should have been reconciled. "- f"Expected 'new_value', got '{person.properties.get(prop_name)}'"- )- assert person.version == 2, f"Person {suffix} version should be 2, got {person.version}"- else:- assert person.properties[prop_name] == "old_value", (- f"Person {suffix} should NOT have been reconciled. "- f"Expected 'old_value', got '{person.properties.get(prop_name)}'"- )- assert person.version == 1, f"Person {suffix} version should still be 1, got {person.version}"-- # Verify team 2 results in Postgres- for suffix, _person_ts, _event_ts, should_be_reconciled, prop_name in test_cases_team2:- person = persons_team2[suffix]- refresh_person_from_persons_db(person)-- if should_be_reconciled:- assert person.properties[prop_name] == "new_value", (- f"Person {suffix} should have been reconciled. "- f"Expected 'new_value', got '{person.properties.get(prop_name)}'"- )- assert person.version == 2, f"Person {suffix} version should be 2, got {person.version}"- else:- assert person.properties[prop_name] == "old_value", (- f"Person {suffix} should NOT have been reconciled. "- f"Expected 'old_value', got '{person.properties.get(prop_name)}'"- )- assert person.version == 1, f"Person {suffix} version should still be 1, got {person.version}"-- # Count reconciled vs not reconciled for summary- reconciled_team1 = sum(1 for _, _, _, expected, _ in test_cases_team1 if expected)- not_reconciled_team1 = len(test_cases_team1) - reconciled_team1- reconciled_team2 = sum(1 for _, _, _, expected, _ in test_cases_team2 if expected)- not_reconciled_team2 = len(test_cases_team2) - reconciled_team2-- # Verify we tested the expected number of cases- assert reconciled_team1 == 6, f"Expected 6 reconciled in team 1, got {reconciled_team1}"- assert not_reconciled_team1 == 3, f"Expected 3 not reconciled in team 1, got {not_reconciled_team1}"- assert reconciled_team2 == 2, f"Expected 2 reconciled in team 2, got {reconciled_team2}"- assert not_reconciled_team2 == 1, f"Expected 1 not reconciled in team 2, got {not_reconciled_team2}"---class TestFetchPersonPropertiesFromClickHouse:- """Integration tests for fetch_person_properties_from_clickhouse against real ClickHouse."""-- def test_fetches_oldest_version_gte_min_version(self, cluster: ClickhouseCluster):- """Test that we fetch the oldest available version >= min_version.-- The function uses argMin to get the oldest version that's at least as new as- the requested min_version. This handles cases where the exact version may have- been merged away by ReplacingMergeTree.- """- team_id = 99801- person_id = UUID("aaaaaaaa-aaaa-aaaa-aaaa-000000000001")- now = datetime.now().replace(microsecond=0)-- # Insert each version in separate INSERT statements to create separate parts- def insert_v1(client: Client) -> None:- client.execute(- """INSERT INTO person (team_id, id, properties, version, _timestamp)- VALUES""",- [(team_id, person_id, json.dumps({"email": "[email protected]"}), 1, now - timedelta(days=5))],- )-- def insert_v3(client: Client) -> None:- client.execute(- """INSERT INTO person (team_id, id, properties, version, _timestamp)- VALUES""",- [(team_id, person_id, json.dumps({"email": "[email protected]"}), 3, now - timedelta(days=3))],- )-- def insert_v5(client: Client) -> None:- client.execute(- """INSERT INTO person (team_id, id, properties, version, _timestamp)- VALUES""",- [(team_id, person_id, json.dumps({"email": "[email protected]"}), 5, now - timedelta(days=1))],- )-- cluster.any_host(insert_v1).result()- cluster.any_host(insert_v3).result()- cluster.any_host(insert_v5).result()-- # min_version=1 should get v1 (oldest >= 1)- props = fetch_person_properties_from_clickhouse(team_id, str(person_id), min_version=1)- assert props is not None- assert props["email"] == "[email protected]"-- # min_version=2 should get v3 (oldest >= 2, since v2 doesn't exist)- props = fetch_person_properties_from_clickhouse(team_id, str(person_id), min_version=2)- assert props is not None- assert props["email"] == "[email protected]"-- # min_version=3 should get v3 (oldest >= 3)- props = fetch_person_properties_from_clickhouse(team_id, str(person_id), min_version=3)- assert props is not None- assert props["email"] == "[email protected]"-- # min_version=4 should get v5 (oldest >= 4, since v4 doesn't exist)- props = fetch_person_properties_from_clickhouse(team_id, str(person_id), min_version=4)- assert props is not None- assert props["email"] == "[email protected]"-- # min_version=5 should get v5- props = fetch_person_properties_from_clickhouse(team_id, str(person_id), min_version=5)- assert props is not None- assert props["email"] == "[email protected]"-- def test_returns_none_when_no_version_gte_min(self, cluster: ClickhouseCluster):- """Test that None is returned when no version >= min_version exists."""- team_id = 99802- person_id = UUID("bbbbbbbb-bbbb-bbbb-bbbb-000000000002")- now = datetime.now().replace(microsecond=0)-- # Insert person with only version 5- def insert_person(client: Client) -> None:- client.execute(- """INSERT INTO person (team_id, id, properties, version, _timestamp)- VALUES""",- [(team_id, person_id, json.dumps({"email": "[email protected]"}), 5, now)],- )-- cluster.any_host(insert_person).result()-- # min_version=5 exists- result = fetch_person_properties_from_clickhouse(team_id, str(person_id), min_version=5)- assert result is not None-- # min_version=6 - no version >= 6 exists- assert fetch_person_properties_from_clickhouse(team_id, str(person_id), min_version=6) is None-- # min_version=99 - no version >= 99 exists- assert fetch_person_properties_from_clickhouse(team_id, str(person_id), min_version=99) is None-- def test_returns_none_for_nonexistent_person(self, cluster: ClickhouseCluster):- """Test that None is returned when the person doesn't exist."""- team_id = 99803- nonexistent_person_id = "cccccccc-cccc-cccc-cccc-000000000003"-- result = fetch_person_properties_from_clickhouse(team_id, nonexistent_person_id, min_version=1)- assert result is None-- def test_returns_empty_dict_for_empty_properties(self, cluster: ClickhouseCluster):- """Test that empty dict is returned when properties are empty."""- team_id = 99804- person_id = UUID("dddddddd-dddd-dddd-dddd-000000000004")- now = datetime.now().replace(microsecond=0)-- # Insert person with empty properties- person_data = [(team_id, person_id, json.dumps({}), 1, now)]-- def insert_person(client: Client) -> None:- client.execute(- """INSERT INTO person (team_id, id, properties, version, _timestamp)- VALUES""",- person_data,- )-- cluster.any_host(insert_person).result()-- result = fetch_person_properties_from_clickhouse(team_id, str(person_id), min_version=1)- assert result == {}---class TestReconcileWithConcurrentChanges:- """Unit tests for reconcile_with_concurrent_changes 3-way merge logic."""-- def test_applies_event_changes_when_no_concurrent_postgres_changes(self):- """When Postgres hasn't changed from CH baseline, all event changes apply."""- ch_properties = {"email": "[email protected]", "name": "Original"}- postgres_person = {- "properties": {"email": "[email protected]", "name": "Original"},- "properties_last_updated_at": {},- "properties_last_operation": {},- }- person_diffs = PersonPropertyDiffs(- person_id="test-person",- person_version=1,- set_updates={"email": PropertyValue(timestamp=datetime(2024, 1, 15, 12, 0, 0), value="[email protected]")},- set_once_updates={},- unset_updates={},- )-- result = reconcile_with_concurrent_changes(ch_properties, postgres_person, person_diffs)-- assert result is not None- assert result["properties"]["email"] == "[email protected]"- assert result["properties"]["name"] == "Original" # unchanged-- def test_postgres_wins_on_conflict(self):- """When Postgres has changed a key that events also change, Postgres wins."""- ch_properties = {"email": "[email protected]"}- postgres_person = {- "properties": {"email": "[email protected]"}, # Postgres changed this- "properties_last_updated_at": {"email": "2024-01-16T00:00:00"},- "properties_last_operation": {"email": "set"},- }- person_diffs = PersonPropertyDiffs(- person_id="test-person",- person_version=1,- set_updates={- "email": PropertyValue(timestamp=datetime(2024, 1, 15, 12, 0, 0), value="[email protected]")- },- set_once_updates={},- unset_updates={},- )-- result = reconcile_with_concurrent_changes(ch_properties, postgres_person, person_diffs)-- # No changes - Postgres already has a different value, so our change is skipped- assert result is None-- def test_applies_event_change_to_unconflicted_key(self):- """Event changes apply to keys that Postgres hasn't concurrently modified."""- ch_properties = {"email": "[email protected]", "name": "Original"}- postgres_person = {- "properties": {"email": "[email protected]", "name": "Original"}, # Only email changed- "properties_last_updated_at": {"email": "2024-01-16T00:00:00"},- "properties_last_operation": {"email": "set"},- }- person_diffs = PersonPropertyDiffs(- person_id="test-person",- person_version=1,- set_updates={- "email": PropertyValue(timestamp=datetime(2024, 1, 15, 12, 0, 0), value="[email protected]"),- "name": PropertyValue(timestamp=datetime(2024, 1, 15, 12, 0, 0), value="Event Name"),- },- set_once_updates={},- unset_updates={},- )-- result = reconcile_with_concurrent_changes(ch_properties, postgres_person, person_diffs)-- assert result is not None- # email: Postgres wins (concurrent change)- assert result["properties"]["email"] == "[email protected]"- # name: event wins (no concurrent change)- assert result["properties"]["name"] == "Event Name"-- def test_set_once_applies_for_new_key(self):- """set_once applies when key doesn't exist in Postgres."""- ch_properties: dict[str, Any] = {}- postgres_person: dict[str, Any] = {- "properties": {},- "properties_last_updated_at": {},- "properties_last_operation": {},- }- person_diffs = PersonPropertyDiffs(- person_id="test-person",- person_version=1,- set_updates={},- set_once_updates={"referrer": PropertyValue(timestamp=datetime(2024, 1, 10, 8, 0, 0), value="google.com")},- unset_updates={},- )-- result = reconcile_with_concurrent_changes(ch_properties, postgres_person, person_diffs)-- assert result is not None- assert result["properties"]["referrer"] == "google.com"-- def test_set_once_skipped_when_key_exists_in_postgres(self):- """set_once is skipped when key already exists in Postgres (even if added concurrently)."""- ch_properties: dict[str, Any] = {} # Key didn't exist at CH version- postgres_person: dict[str, Any] = {- "properties": {"referrer": "facebook.com"}, # But now exists in Postgres- "properties_last_updated_at": {"referrer": "2024-01-12T00:00:00"},- "properties_last_operation": {"referrer": "set"},- }- person_diffs = PersonPropertyDiffs(- person_id="test-person",- person_version=1,- set_updates={},- set_once_updates={"referrer": PropertyValue(timestamp=datetime(2024, 1, 10, 8, 0, 0), value="google.com")},- unset_updates={},- )-- result = reconcile_with_concurrent_changes(ch_properties, postgres_person, person_diffs)-- # No change - key exists in Postgres- assert result is None-- def test_unset_applies_when_key_not_concurrently_changed(self):- """$unset removes key when Postgres hasn't concurrently modified it."""- ch_properties = {"email": "[email protected]", "name": "Original"}- postgres_person = {- "properties": {"email": "[email protected]", "name": "Original"},- "properties_last_updated_at": {},- "properties_last_operation": {},- }- person_diffs = PersonPropertyDiffs(- person_id="test-person",- person_version=1,- set_updates={},- set_once_updates={},- unset_updates={"email": PropertyValue(timestamp=datetime(2024, 1, 15, 12, 0, 0), value=None)},- )-- result = reconcile_with_concurrent_changes(ch_properties, postgres_person, person_diffs)-- assert result is not None- assert "email" not in result["properties"]- assert result["properties"]["name"] == "Original"-- def test_unset_skipped_when_key_concurrently_changed(self):- """$unset is skipped when Postgres has concurrently modified the key."""- ch_properties = {"email": "[email protected]"}- postgres_person = {- "properties": {"email": "[email protected]"}, # Postgres changed this- "properties_last_updated_at": {"email": "2024-01-16T00:00:00"},- "properties_last_operation": {"email": "set"},- }- person_diffs = PersonPropertyDiffs(- person_id="test-person",- person_version=1,- set_updates={},- set_once_updates={},- unset_updates={"email": PropertyValue(timestamp=datetime(2024, 1, 15, 12, 0, 0), value=None)},- )-- result = reconcile_with_concurrent_changes(ch_properties, postgres_person, person_diffs)-- # No change - Postgres has concurrent modification- assert result is None-- def test_preserves_postgres_additions(self):- """Keys added to Postgres concurrently should be preserved."""- ch_properties = {"email": "[email protected]"}- postgres_person = {- "properties": {"email": "[email protected]", "new_key": "postgres_added"},- "properties_last_updated_at": {"new_key": "2024-01-16T00:00:00"},- "properties_last_operation": {"new_key": "set"},- }- person_diffs = PersonPropertyDiffs(- person_id="test-person",- person_version=1,- set_updates={"name": PropertyValue(timestamp=datetime(2024, 1, 15, 12, 0, 0), value="Event Name")},- set_once_updates={},- unset_updates={},- )-- result = reconcile_with_concurrent_changes(ch_properties, postgres_person, person_diffs)-- assert result is not None- assert result["properties"]["email"] == "[email protected]"- assert result["properties"]["new_key"] == "postgres_added" # Preserved- assert result["properties"]["name"] == "Event Name" # Event change applied-- def test_mixed_operations_with_conflicts(self):- """Test complex scenario with multiple operations and partial conflicts."""- ch_properties = {"a": "ch_a", "b": "ch_b", "c": "ch_c"}- postgres_person = {- "properties": {"a": "pg_a", "b": "ch_b", "c": "ch_c", "d": "pg_d"}, # a changed, d added- "properties_last_updated_at": {"a": "2024-01-16T00:00:00", "d": "2024-01-16T00:00:00"},- "properties_last_operation": {"a": "set", "d": "set"},- }- person_diffs = PersonPropertyDiffs(- person_id="test-person",- person_version=1,- set_updates={- "a": PropertyValue(timestamp=datetime(2024, 1, 15, 12, 0, 0), value="event_a"), # Conflict- "b": PropertyValue(timestamp=datetime(2024, 1, 15, 12, 0, 0), value="event_b"), # No conflict- },- set_once_updates={- "e": PropertyValue(timestamp=datetime(2024, 1, 10, 8, 0, 0), value="event_e"), # New key- },- unset_updates={- "c": PropertyValue(timestamp=datetime(2024, 1, 15, 12, 0, 0), value=None), # No conflict- },- )-- result = reconcile_with_concurrent_changes(ch_properties, postgres_person, person_diffs)-- assert result is not None- assert result["properties"]["a"] == "pg_a" # Postgres wins (conflict)- assert result["properties"]["b"] == "event_b" # Event wins (no conflict)- assert "c" not in result["properties"] # Unset applied (no conflict)- assert result["properties"]["d"] == "pg_d" # Postgres addition preserved- assert result["properties"]["e"] == "event_e" # set_once applied (new key)---class TestQueryTeamIdsFromClickHouse:- """Integration tests for query_team_ids_from_clickhouse with team_id filters."""-- def test_returns_teams_with_property_events(self, cluster: ClickhouseCluster):- """Basic test that teams with $set/$set_once/$unset events are returned."""- now = datetime.now().replace(microsecond=0)- bug_window_start = now - timedelta(days=10)- bug_window_end = now + timedelta(days=1)-- # Use unique team_ids unlikely to conflict with other tests (88001-88005)- events = [- (- 88001,- "d1",- UUID("11111111-1111-1111-1111-880000000001"),- now - timedelta(days=5),- json.dumps({"$set": {"a": 1}}),- ),- (- 88002,- "d2",- UUID("11111111-1111-1111-1111-880000000002"),- now - timedelta(days=5),- json.dumps({"$set_once": {"b": 2}}),- ),- (- 88003,- "d3",- UUID("11111111-1111-1111-1111-880000000003"),- now - timedelta(days=5),- json.dumps({"$unset": ["c"]}),- ),- (- 88004,- "d4",- UUID("11111111-1111-1111-1111-880000000004"),- now - timedelta(days=5),- json.dumps({"$set": {"d": 4}}),- ),- (- 88005,- "d5",- UUID("11111111-1111-1111-1111-880000000005"),- now - timedelta(days=5),- json.dumps({"other": "no_props"}),- ),- ]-- def insert_events(client: Client) -> None:- client.execute(- "INSERT INTO writable_events (team_id, distinct_id, person_id, timestamp, properties) VALUES",- events,- )-- cluster.any_host(insert_events).result()-- result = query_team_ids_from_clickhouse(- bug_window_start=bug_window_start.strftime("%Y-%m-%d %H:%M:%S"),- bug_window_end=bug_window_end.strftime("%Y-%m-%d %H:%M:%S"),- min_team_id=88001,- max_team_id=88005,- )-- # 88005 has no $set/$set_once/$unset so should not be included- assert result == [88001, 88002, 88003, 88004]-- def test_min_team_id_filter(self, cluster: ClickhouseCluster):- """Test that min_team_id filters out teams below the threshold."""- now = datetime.now().replace(microsecond=0)- bug_window_start = now - timedelta(days=10)- bug_window_end = now + timedelta(days=1)-- events = [- (- 89001,- "d1",- UUID("11111111-1111-1111-1111-890000000001"),- now - timedelta(days=5),- json.dumps({"$set": {"a": 1}}),- ),- (- 89002,- "d2",- UUID("11111111-1111-1111-1111-890000000002"),- now - timedelta(days=5),- json.dumps({"$set": {"b": 2}}),- ),- (- 89003,- "d3",- UUID("11111111-1111-1111-1111-890000000003"),- now - timedelta(days=5),- json.dumps({"$set": {"c": 3}}),- ),- ]-- def insert_events(client: Client) -> None:- client.execute(- "INSERT INTO writable_events (team_id, distinct_id, person_id, timestamp, properties) VALUES",- events,- )-- cluster.any_host(insert_events).result()-- result = query_team_ids_from_clickhouse(- bug_window_start=bug_window_start.strftime("%Y-%m-%d %H:%M:%S"),- bug_window_end=bug_window_end.strftime("%Y-%m-%d %H:%M:%S"),- min_team_id=89002,- max_team_id=89003,- )-- assert result == [89002, 89003]-- def test_max_team_id_filter(self, cluster: ClickhouseCluster):- """Test that max_team_id filters out teams above the threshold."""- now = datetime.now().replace(microsecond=0)- bug_window_start = now - timedelta(days=10)- bug_window_end = now + timedelta(days=1)-- events = [- (- 90001,- "d1",- UUID("11111111-1111-1111-1111-900000000001"),- now - timedelta(days=5),- json.dumps({"$set": {"a": 1}}),- ),- (- 90002,- "d2",- UUID("11111111-1111-1111-1111-900000000002"),- now - timedelta(days=5),- json.dumps({"$set": {"b": 2}}),- ),- (- 90003,- "d3",- UUID("11111111-1111-1111-1111-900000000003"),- now - timedelta(days=5),- json.dumps({"$set": {"c": 3}}),- ),- ]-- def insert_events(client: Client) -> None:- client.execute(- "INSERT INTO writable_events (team_id, distinct_id, person_id, timestamp, properties) VALUES",- events,- )-- cluster.any_host(insert_events).result()-- result = query_team_ids_from_clickhouse(- bug_window_start=bug_window_start.strftime("%Y-%m-%d %H:%M:%S"),- bug_window_end=bug_window_end.strftime("%Y-%m-%d %H:%M:%S"),- min_team_id=90001,- max_team_id=90002,- )-- assert result == [90001, 90002]-- def test_both_min_and_max_team_id_filters(self, cluster: ClickhouseCluster):- """Test that both min and max filters work together."""- now = datetime.now().replace(microsecond=0)- bug_window_start = now - timedelta(days=10)- bug_window_end = now + timedelta(days=1)-- events = [- (- 91001,- "d1",- UUID("11111111-1111-1111-1111-910000000001"),- now - timedelta(days=5),- json.dumps({"$set": {"a": 1}}),- ),- (- 91002,- "d2",- UUID("11111111-1111-1111-1111-910000000002"),- now - timedelta(days=5),- json.dumps({"$set": {"b": 2}}),- ),- (- 91003,- "d3",- UUID("11111111-1111-1111-1111-910000000003"),- now - timedelta(days=5),- json.dumps({"$set": {"c": 3}}),- ),- (- 91004,- "d4",- UUID("11111111-1111-1111-1111-910000000004"),- now - timedelta(days=5),- json.dumps({"$set": {"d": 4}}),- ),- (- 91005,- "d5",- UUID("11111111-1111-1111-1111-910000000005"),- now - timedelta(days=5),- json.dumps({"$set": {"e": 5}}),- ),- ]-- def insert_events(client: Client) -> None:- client.execute(- "INSERT INTO writable_events (team_id, distinct_id, person_id, timestamp, properties) VALUES",- events,- )-- cluster.any_host(insert_events).result()-- result = query_team_ids_from_clickhouse(- bug_window_start=bug_window_start.strftime("%Y-%m-%d %H:%M:%S"),- bug_window_end=bug_window_end.strftime("%Y-%m-%d %H:%M:%S"),- min_team_id=91002,- max_team_id=91004,- )-- assert result == [91002, 91003, 91004]-- def test_team_range_returns_all_matching_teams(self, cluster: ClickhouseCluster):- """Test that team range filter returns all teams with property events within range."""- now = datetime.now().replace(microsecond=0)- bug_window_start = now - timedelta(days=10)- bug_window_end = now + timedelta(days=1)-- events = [- (- 92001,- "d1",- UUID("11111111-1111-1111-1111-920000000001"),- now - timedelta(days=5),- json.dumps({"$set": {"a": 1}}),- ),- (- 92002,- "d2",- UUID("11111111-1111-1111-1111-920000000002"),- now - timedelta(days=5),- json.dumps({"$set": {"b": 2}}),- ),- ]-- def insert_events(client: Client) -> None:- client.execute(- "INSERT INTO writable_events (team_id, distinct_id, person_id, timestamp, properties) VALUES",- events,- )-- cluster.any_host(insert_events).result()-- result = query_team_ids_from_clickhouse(- bug_window_start=bug_window_start.strftime("%Y-%m-%d %H:%M:%S"),- bug_window_end=bug_window_end.strftime("%Y-%m-%d %H:%M:%S"),- min_team_id=92001,- max_team_id=92002,- )-- assert 92001 in result- assert 92002 in result-- def test_exclude_team_ids_filters_out_specified_teams(self, cluster: ClickhouseCluster):- """Test that exclude_team_ids filters out specified teams from results."""- now = datetime.now().replace(microsecond=0)- bug_window_start = now - timedelta(days=10)- bug_window_end = now + timedelta(days=1)-- events = [- (- 93001,- "d1",- UUID("11111111-1111-1111-1111-930000000001"),- now - timedelta(days=5),- json.dumps({"$set": {"a": 1}}),- ),- (- 93002,- "d2",- UUID("11111111-1111-1111-1111-930000000002"),- now - timedelta(days=5),- json.dumps({"$set": {"b": 2}}),- ),- (- 93003,- "d3",- UUID("11111111-1111-1111-1111-930000000003"),- now - timedelta(days=5),- json.dumps({"$set": {"c": 3}}),- ),- (- 93004,- "d4",- UUID("11111111-1111-1111-1111-930000000004"),- now - timedelta(days=5),- json.dumps({"$set": {"d": 4}}),- ),- ]-- def insert_events(client: Client) -> None:- client.execute(- "INSERT INTO writable_events (team_id, distinct_id, person_id, timestamp, properties) VALUES",- events,- )-- cluster.any_host(insert_events).result()-- result = query_team_ids_from_clickhouse(- bug_window_start=bug_window_start.strftime("%Y-%m-%d %H:%M:%S"),- bug_window_end=bug_window_end.strftime("%Y-%m-%d %H:%M:%S"),- min_team_id=93001,- max_team_id=93004,- exclude_team_ids=[93002, 93004],- )-- assert result == [93001, 93003]-- def test_include_team_ids_filters_to_specified_teams(self, cluster: ClickhouseCluster):- """Test that include_team_ids only returns teams in the specified list."""- bug_window_start = datetime.now(UTC) - timedelta(hours=2)- bug_window_end = datetime.now(UTC) + timedelta(hours=1)- event_time = datetime.now(UTC) - timedelta(hours=1)-- # Create events for teams 94001-94005- events = [- (- 94001,- "user_a",- "00000000-0000-0000-0000-000000094001",- event_time,- json.dumps({"$set": {"a": 1}}),- ),- (- 94002,- "user_b",- "00000000-0000-0000-0000-000000094002",- event_time,- json.dumps({"$set": {"b": 2}}),- ),- (- 94003,- "user_c",- "00000000-0000-0000-0000-000000094003",- event_time,- json.dumps({"$set": {"c": 3}}),- ),- (- 94004,- "user_d",- "00000000-0000-0000-0000-000000094004",- event_time,- json.dumps({"$set": {"d": 4}}),- ),- (- 94005,- "user_e",- "00000000-0000-0000-0000-000000094005",- event_time,- json.dumps({"$set": {"e": 5}}),- ),- ]-- def insert_events(client: Client) -> None:- client.execute(- "INSERT INTO writable_events (team_id, distinct_id, person_id, timestamp, properties) VALUES",- events,- )-- cluster.any_host(insert_events).result()-- # Only include teams 94002 and 94004 - should not return 94001, 94003, 94005- result = query_team_ids_from_clickhouse(- bug_window_start=bug_window_start.strftime("%Y-%m-%d %H:%M:%S"),- bug_window_end=bug_window_end.strftime("%Y-%m-%d %H:%M:%S"),- include_team_ids=[94002, 94004],- )-- assert result == [94002, 94004]-- def test_include_team_ids_combined_with_other_filters(self, cluster: ClickhouseCluster):- """Test that include_team_ids works as AND with other filters."""- bug_window_start = datetime.now(UTC) - timedelta(hours=2)- bug_window_end = datetime.now(UTC) + timedelta(hours=1)- event_time = datetime.now(UTC) - timedelta(hours=1)-- # Create events for teams 95001-95004- events = [- (- 95001,- "user_a",- "00000000-0000-0000-0000-000000095001",- event_time,- json.dumps({"$set": {"a": 1}}),- ),- (- 95002,- "user_b",- "00000000-0000-0000-0000-000000095002",- event_time,- json.dumps({"$set": {"b": 2}}),- ),- (- 95003,- "user_c",- "00000000-0000-0000-0000-000000095003",- event_time,- json.dumps({"$set": {"c": 3}}),- ),- (- 95004,- "user_d",- "00000000-0000-0000-0000-000000095004",- event_time,- json.dumps({"$set": {"d": 4}}),- ),- ]-- def insert_events(client: Client) -> None:- client.execute(- "INSERT INTO writable_events (team_id, distinct_id, person_id, timestamp, properties) VALUES",- events,- )-- cluster.any_host(insert_events).result()-- # include_team_ids=[95001, 95002, 95003] AND exclude_team_ids=[95002] -> only 95001, 95003- result = query_team_ids_from_clickhouse(- bug_window_start=bug_window_start.strftime("%Y-%m-%d %H:%M:%S"),- bug_window_end=bug_window_end.strftime("%Y-%m-%d %H:%M:%S"),- include_team_ids=[95001, 95002, 95003],- exclude_team_ids=[95002],- )-- assert result == [95001, 95003]-- def test_invalid_range_raises_error(self):- """Test that min_team_id > max_team_id raises ValueError."""- with pytest.raises(ValueError) as exc_info:- query_team_ids_from_clickhouse(- bug_window_start="2024-01-01 00:00:00",- bug_window_end="2024-01-02 00:00:00",- min_team_id=100,- max_team_id=50,- )-- assert "min_team_id (100) cannot be greater than max_team_id (50)" in str(exc_info.value)---class TestReconcileSingleTeamErrorHandling:- """Tests for error handling in reconcile_single_team and reconcile_team_chunk."""-- def test_team_failure_does_not_crash_chunk(self):- """Test that if one team fails, the chunk continues processing other teams."""- from posthog.dags.person_property_reconciliation import (- PersonPropertyReconciliationConfig,- TeamReconciliationResult,- )-- # Mock reconcile_single_team to fail for team 2 but succeed for teams 1 and 3- call_count = {"value": 0}- original_results = {- 1: TeamReconciliationResult(team_id=1, persons_processed=10, persons_updated=5, persons_skipped=5),- 3: TeamReconciliationResult(team_id=3, persons_processed=20, persons_updated=15, persons_skipped=5),- }-- def mock_reconcile_single_team(team_id, **kwargs):- call_count["value"] += 1- if team_id == 2:- raise Exception("Simulated failure for team 2")- return original_results[team_id]-- with patch(- "posthog.dags.person_property_reconciliation.reconcile_single_team",- side_effect=mock_reconcile_single_team,- ):- # Create mock context with run info- mock_context = MagicMock()- mock_context.run.job_name = "test_job"- mock_context.run.run_id = "test_run_123"-- # Create real config- config = PersonPropertyReconciliationConfig(- bug_window_start="2024-01-01 00:00:00",- batch_size=100,- dry_run=False,- backup_enabled=False,- )-- # Create mock resources- mock_db = MagicMock()- mock_cluster = MagicMock()- mock_kafka = MagicMock()-- # Import the underlying function (not the op wrapper)- from posthog.dags.person_property_reconciliation import reconcile_team_chunk-- # Call the op's underlying function directly via __wrapped__- result = reconcile_team_chunk.__wrapped__( # type: ignore[attr-defined]- context=mock_context,- config=config,- chunk=[1, 2, 3],- persons_database=mock_db,- cluster=mock_cluster,- kafka_producer=mock_kafka,- )-- # All 3 teams should have been attempted- assert call_count["value"] == 3-- # Check overall results- assert result["teams_count"] == 3- assert result["teams_succeeded"] == 2- assert result["teams_failed"] == 1- assert result["persons_processed"] == 30 # 10 + 0 + 20- assert result["persons_updated"] == 20 # 5 + 0 + 15-- # Check individual team results- teams_results = result["teams_results"]- assert len(teams_results) == 3-- team1_result = next(r for r in teams_results if r["team_id"] == 1)- assert team1_result["status"] == "success"- assert team1_result["persons_processed"] == 10-- team2_result = next(r for r in teams_results if r["team_id"] == 2)- assert team2_result["status"] == "failed"- assert "Simulated failure" in team2_result["error"]-- team3_result = next(r for r in teams_results if r["team_id"] == 3)- assert team3_result["status"] == "success"- assert team3_result["persons_processed"] == 20---class TestReconciliationSchedulerSensor:- """Tests for the person_property_reconciliation_scheduler sensor."""-- def test_build_reconciliation_run_config(self):- """Test that run config is built correctly from scheduler config."""- from posthog.dags.person_property_reconciliation import (- ReconciliationSchedulerConfig,- build_reconciliation_run_config,- )-- config = ReconciliationSchedulerConfig(- range_start=1,- range_end=10000,- chunk_size=1000,- max_concurrent_jobs=5,- max_concurrent_tasks=10,- bug_window_start="2024-01-06 20:01:00",- bug_window_end="2024-01-07 14:52:00",- dry_run=False,- backup_enabled=True,- batch_size=100,- )-- run_config = build_reconciliation_run_config(config, min_team_id=1, max_team_id=1000)-- assert run_config["ops"]["get_team_ids_to_reconcile"]["config"]["min_team_id"] == 1- assert run_config["ops"]["get_team_ids_to_reconcile"]["config"]["max_team_id"] == 1000- assert run_config["ops"]["get_team_ids_to_reconcile"]["config"]["bug_window_start"] == "2024-01-06 20:01:00"- assert run_config["ops"]["get_team_ids_to_reconcile"]["config"]["dry_run"] is False- assert run_config["ops"]["reconcile_team_chunk"]["config"]["backup_enabled"] is True- assert run_config["execution"]["config"]["max_concurrent"] == 10 # max_concurrent_tasks-- def test_sensor_no_cursor_returns_skip_reason(self):- """Test that sensor returns SkipReason when no cursor is set."""- from dagster import SkipReason, build_sensor_context-- from posthog.dags.person_property_reconciliation import person_property_reconciliation_scheduler-- context = build_sensor_context(cursor=None)- result = person_property_reconciliation_scheduler(context)-- assert isinstance(result, SkipReason)- assert result.skip_message is not None- assert "No cursor set" in result.skip_message-- def test_sensor_invalid_cursor_json_returns_skip_reason(self):- """Test that sensor returns SkipReason for invalid JSON cursor."""- from dagster import SkipReason, build_sensor_context-- from posthog.dags.person_property_reconciliation import person_property_reconciliation_scheduler-- context = build_sensor_context(cursor="not valid json {")- result = person_property_reconciliation_scheduler(context)-- assert isinstance(result, SkipReason)- assert result.skip_message is not None- assert "Invalid cursor JSON" in result.skip_message-- def test_sensor_completed_range_returns_skip_reason(self):- """Test that sensor returns SkipReason when range is completed."""- from dagster import DagsterInstance, SkipReason, build_sensor_context-- from posthog.dags.person_property_reconciliation import person_property_reconciliation_scheduler-- cursor = json.dumps(- {- "range_start": 1,- "range_end": 1000,- "next_chunk_start": 1001,- "bug_window_start": "2024-01-06 20:01:00",- "bug_window_end": "2024-01-07 14:52:00",- }- )-- mock_instance = MagicMock(spec=DagsterInstance)- mock_instance.get_run_records.return_value = []-- context = build_sensor_context(cursor=cursor, instance=mock_instance)- result = person_property_reconciliation_scheduler(context)-- assert isinstance(result, SkipReason)- assert result.skip_message is not None- assert "complete" in result.skip_message.lower()-- def test_sensor_at_max_concurrency_returns_skip_reason(self):- """Test that sensor returns SkipReason when at max concurrency."""- from dagster import DagsterInstance, SkipReason, build_sensor_context-- from posthog.dags.person_property_reconciliation import person_property_reconciliation_scheduler-- cursor = json.dumps(- {- "range_start": 1,- "range_end": 10000,- "chunk_size": 1000,- "max_concurrent_jobs": 2,- "next_chunk_start": 1,- "bug_window_start": "2024-01-06 20:01:00",- "bug_window_end": "2024-01-07 14:52:00",- }- )-- mock_instance = MagicMock(spec=DagsterInstance)- mock_instance.get_run_records.return_value = [MagicMock(), MagicMock()]-- context = build_sensor_context(cursor=cursor, instance=mock_instance)- result = person_property_reconciliation_scheduler(context)-- assert isinstance(result, SkipReason)- assert result.skip_message is not None- assert "max concurrency" in result.skip_message.lower()-- def test_sensor_yields_runs_up_to_available_slots(self):- """Test that sensor yields correct number of runs based on available slots."""- from dagster import DagsterInstance, SensorResult, build_sensor_context-- from posthog.dags.person_property_reconciliation import person_property_reconciliation_scheduler-- cursor = json.dumps(- {- "range_start": 1,- "range_end": 5000,- "chunk_size": 1000,- "max_concurrent_jobs": 3,- "next_chunk_start": 1,- "bug_window_start": "2024-01-06 20:01:00",- "bug_window_end": "2024-01-07 14:52:00",- }- )-- mock_instance = MagicMock(spec=DagsterInstance)- mock_instance.get_run_records.return_value = [MagicMock()]-- context = build_sensor_context(cursor=cursor, instance=mock_instance)- result = person_property_reconciliation_scheduler(context)-- assert isinstance(result, SensorResult)- assert result.run_requests is not None- assert len(result.run_requests) == 2- assert result.run_requests[0].tags["reconciliation_range"] == "1-1000"- assert result.run_requests[1].tags["reconciliation_range"] == "1001-2000"-- assert result.cursor is not None- new_cursor = json.loads(result.cursor)- assert new_cursor["next_chunk_start"] == 2001-- def test_sensor_handles_partial_final_chunk(self):- """Test that sensor correctly handles a final chunk smaller than chunk_size."""- from dagster import DagsterInstance, SensorResult, build_sensor_context-- from posthog.dags.person_property_reconciliation import person_property_reconciliation_scheduler-- cursor = json.dumps(- {- "range_start": 1,- "range_end": 1500,- "chunk_size": 1000,- "max_concurrent_jobs": 5,- "next_chunk_start": 1001,- "bug_window_start": "2024-01-06 20:01:00",- "bug_window_end": "2024-01-07 14:52:00",- }- )-- mock_instance = MagicMock(spec=DagsterInstance)- mock_instance.get_run_records.return_value = []-- context = build_sensor_context(cursor=cursor, instance=mock_instance)- result = person_property_reconciliation_scheduler(context)-- assert isinstance(result, SensorResult)- assert result.run_requests is not None- assert len(result.run_requests) == 1- assert result.run_requests[0].tags["reconciliation_range"] == "1001-1500"-- assert result.cursor is not None- new_cursor = json.loads(result.cursor)- assert new_cursor["next_chunk_start"] == 1501-- def test_sensor_run_request_has_correct_tags(self):- """Test that run requests include expected tags."""- from dagster import DagsterInstance, SensorResult, build_sensor_context-- from posthog.dags.person_property_reconciliation import person_property_reconciliation_scheduler-- cursor = json.dumps(- {- "range_start": 1,- "range_end": 2000,- "chunk_size": 1000,- "max_concurrent_jobs": 5,- "next_chunk_start": 1,- "bug_window_start": "2024-01-06 20:01:00",- "bug_window_end": "2024-01-07 14:52:00",- }- )-- mock_instance = MagicMock(spec=DagsterInstance)- mock_instance.get_run_records.return_value = []-- context = build_sensor_context(cursor=cursor, instance=mock_instance)- result = person_property_reconciliation_scheduler(context)-- assert isinstance(result, SensorResult)- assert result.run_requests is not None- assert len(result.run_requests) == 2- assert result.run_requests[0].tags["reconciliation_range"] == "1-1000"- assert result.run_requests[0].tags["owner"] == "team-ingestion"-- def test_sensor_validates_range_start_less_than_range_end(self):- """Test that sensor rejects range_start > range_end."""- from dagster import SkipReason, build_sensor_context-- from posthog.dags.person_property_reconciliation import person_property_reconciliation_scheduler-- cursor = json.dumps(- {- "range_start": 1000,- "range_end": 500,- "bug_window_start": "2024-01-06 20:01:00",- "bug_window_end": "2024-01-07 14:52:00",- }- )- context = build_sensor_context(cursor=cursor)- result = person_property_reconciliation_scheduler(context)-- assert isinstance(result, SkipReason)- assert result.skip_message is not None- assert "range_start" in result.skip_message- assert "range_end" in result.skip_message-- def test_sensor_validates_chunk_size_positive(self):- """Test that sensor rejects chunk_size <= 0."""- from dagster import SkipReason, build_sensor_context-- from posthog.dags.person_property_reconciliation import person_property_reconciliation_scheduler-- cursor = json.dumps(- {- "range_start": 1,- "range_end": 1000,- "chunk_size": 0,- "bug_window_start": "2024-01-06 20:01:00",- "bug_window_end": "2024-01-07 14:52:00",- }- )- context = build_sensor_context(cursor=cursor)- result = person_property_reconciliation_scheduler(context)-- assert isinstance(result, SkipReason)- assert result.skip_message is not None- assert "chunk_size" in result.skip_message-- def test_sensor_validates_max_concurrent_jobs_positive(self):- """Test that sensor rejects max_concurrent_jobs <= 0."""- from dagster import SkipReason, build_sensor_context-- from posthog.dags.person_property_reconciliation import person_property_reconciliation_scheduler-- cursor = json.dumps(- {- "range_start": 1,- "range_end": 1000,- "max_concurrent_jobs": 0,- "bug_window_start": "2024-01-06 20:01:00",- "bug_window_end": "2024-01-07 14:52:00",- }- )- context = build_sensor_context(cursor=cursor)- result = person_property_reconciliation_scheduler(context)-- assert isinstance(result, SkipReason)- assert result.skip_message is not None- assert "max_concurrent_jobs" in result.skip_message-- def test_sensor_validates_max_concurrent_jobs_cap(self):- """Test that sensor rejects max_concurrent_jobs exceeding the cap."""- from dagster import SkipReason, build_sensor_context-- from posthog.dags.person_property_reconciliation import person_property_reconciliation_scheduler-- cursor = json.dumps(- {- "range_start": 1,- "range_end": 1000,- "max_concurrent_jobs": 100,- "bug_window_start": "2024-01-06 20:01:00",- "bug_window_end": "2024-01-07 14:52:00",- }- )- context = build_sensor_context(cursor=cursor)- result = person_property_reconciliation_scheduler(context)-- assert isinstance(result, SkipReason)- assert result.skip_message is not None- assert "exceeds cap" in result.skip_message.lower()-- def test_sensor_validates_max_concurrent_tasks_positive(self):- """Test that sensor rejects max_concurrent_tasks <= 0."""- from dagster import SkipReason, build_sensor_context-- from posthog.dags.person_property_reconciliation import person_property_reconciliation_scheduler-- cursor = json.dumps(- {- "range_start": 1,- "range_end": 1000,- "max_concurrent_tasks": 0,- "bug_window_start": "2024-01-06 20:01:00",- "bug_window_end": "2024-01-07 14:52:00",- }- )- context = build_sensor_context(cursor=cursor)- result = person_property_reconciliation_scheduler(context)-- assert isinstance(result, SkipReason)- assert result.skip_message is not None- assert "max_concurrent_tasks" in result.skip_message-- def test_sensor_validates_max_concurrent_tasks_cap(self):- """Test that sensor rejects max_concurrent_tasks exceeding the cap."""- from dagster import SkipReason, build_sensor_context-- from posthog.dags.person_property_reconciliation import person_property_reconciliation_scheduler-- cursor = json.dumps(- {- "range_start": 1,- "range_end": 1000,- "max_concurrent_tasks": 200,- "bug_window_start": "2024-01-06 20:01:00",- "bug_window_end": "2024-01-07 14:52:00",- }- )- context = build_sensor_context(cursor=cursor)- result = person_property_reconciliation_scheduler(context)-- assert isinstance(result, SkipReason)- assert result.skip_message is not None- assert "exceeds cap" in result.skip_message.lower()-- def test_sensor_validates_bug_window_start_required(self):- """Test that sensor rejects missing bug_window_start."""- from dagster import SkipReason, build_sensor_context-- from posthog.dags.person_property_reconciliation import person_property_reconciliation_scheduler-- cursor = json.dumps(- {- "range_start": 1,- "range_end": 1000,- "bug_window_end": "2024-01-07 14:52:00",- }- )- context = build_sensor_context(cursor=cursor)- result = person_property_reconciliation_scheduler(context)-- assert isinstance(result, SkipReason)- assert result.skip_message is not None- assert "bug_window_start" in result.skip_message-- def test_sensor_validates_bug_window_end_required(self):- """Test that sensor rejects missing bug_window_end."""- from dagster import SkipReason, build_sensor_context-- from posthog.dags.person_property_reconciliation import person_property_reconciliation_scheduler-- cursor = json.dumps(- {- "range_start": 1,- "range_end": 1000,- "bug_window_start": "2024-01-06 20:01:00",- }- )- context = build_sensor_context(cursor=cursor)- result = person_property_reconciliation_scheduler(context)-- assert isinstance(result, SkipReason)- assert result.skip_message is not None- assert "bug_window_end" in result.skip_message-- def test_build_reconciliation_run_config_with_team_ids(self):- """Test that run config passes team_ids to op config when provided."""- from posthog.dags.person_property_reconciliation import (- ReconciliationSchedulerConfig,- build_reconciliation_run_config,- )-- config = ReconciliationSchedulerConfig(- team_ids=[1, 5, 10, 20],- chunk_size=100,- max_concurrent_jobs=5,- max_concurrent_tasks=10,- bug_window_start="2024-01-06 20:01:00",- bug_window_end="2024-01-07 14:52:00",- dry_run=True,- backup_enabled=False,- batch_size=50,- )-- run_config = build_reconciliation_run_config(config, team_ids=[1, 5, 10])-- op_config = run_config["ops"]["get_team_ids_to_reconcile"]["config"]- assert op_config["team_ids"] == [1, 5, 10]- assert "min_team_id" not in op_config- assert "max_team_id" not in op_config- assert op_config["bug_window_start"] == "2024-01-06 20:01:00"- assert op_config["dry_run"] is True- assert run_config["ops"]["reconcile_team_chunk"]["config"]["backup_enabled"] is False-- def test_sensor_with_team_ids_yields_chunked_runs(self):- """Test that sensor chunks team_ids list correctly."""- from dagster import DagsterInstance, SensorResult, build_sensor_context-- from posthog.dags.person_property_reconciliation import person_property_reconciliation_scheduler-- cursor = json.dumps(- {- "team_ids": [1, 2, 3, 4, 5, 6, 7, 8, 9, 10],- "chunk_size": 3,- "max_concurrent_jobs": 5,- "next_team_index": 0,- "bug_window_start": "2024-01-06 20:01:00",- "bug_window_end": "2024-01-07 14:52:00",- }- )-- mock_instance = MagicMock(spec=DagsterInstance)- mock_instance.get_run_records.return_value = []-- context = build_sensor_context(cursor=cursor, instance=mock_instance)- result = person_property_reconciliation_scheduler(context)-- assert isinstance(result, SensorResult)- assert result.run_requests is not None- assert len(result.run_requests) == 4 # 10 teams / 3 per chunk = 4 chunks (3+3+3+1)- assert result.run_requests[0].tags["reconciliation_team_ids_range"] == "0-2"- assert result.run_requests[1].tags["reconciliation_team_ids_range"] == "3-5"- assert result.run_requests[2].tags["reconciliation_team_ids_range"] == "6-8"- assert result.run_requests[3].tags["reconciliation_team_ids_range"] == "9-9"-- # Verify first run config has correct team_ids chunk- first_run_config = result.run_requests[0].run_config- assert first_run_config["ops"]["get_team_ids_to_reconcile"]["config"]["team_ids"] == [1, 2, 3]-- def test_sensor_team_ids_mode_updates_cursor(self):- """Test that sensor updates next_team_index in cursor."""- from dagster import DagsterInstance, SensorResult, build_sensor_context-- from posthog.dags.person_property_reconciliation import person_property_reconciliation_scheduler-- cursor = json.dumps(- {- "team_ids": [100, 200, 300, 400, 500],- "chunk_size": 2,- "max_concurrent_jobs": 2, # Limit to 2 runs- "next_team_index": 0,- "bug_window_start": "2024-01-06 20:01:00",- "bug_window_end": "2024-01-07 14:52:00",- }- )-- mock_instance = MagicMock(spec=DagsterInstance)- mock_instance.get_run_records.return_value = []-- context = build_sensor_context(cursor=cursor, instance=mock_instance)- result = person_property_reconciliation_scheduler(context)-- assert isinstance(result, SensorResult)- assert result.run_requests is not None- assert len(result.run_requests) == 2 # Limited by max_concurrent_jobs-- # Cursor should track progress- assert result.cursor is not None- new_cursor = json.loads(result.cursor)- assert new_cursor["next_team_index"] == 4 # Processed 2 chunks of 2 = 4 teams-- def test_sensor_team_ids_completes_when_list_exhausted(self):- """Test that sensor returns SkipReason when all team_ids are processed."""- from dagster import DagsterInstance, SkipReason, build_sensor_context-- from posthog.dags.person_property_reconciliation import person_property_reconciliation_scheduler-- cursor = json.dumps(- {- "team_ids": [1, 2, 3],- "chunk_size": 10,- "next_team_index": 3, # Already processed all 3- "bug_window_start": "2024-01-06 20:01:00",- "bug_window_end": "2024-01-07 14:52:00",- }- )-- mock_instance = MagicMock(spec=DagsterInstance)- mock_instance.get_run_records.return_value = []-- context = build_sensor_context(cursor=cursor, instance=mock_instance)- result = person_property_reconciliation_scheduler(context)-- assert isinstance(result, SkipReason)- assert result.skip_message is not None- assert "complete" in result.skip_message.lower()- assert "3" in result.skip_message # Should mention number of teams-- def test_sensor_validates_either_team_ids_or_range_required(self):- """Test that sensor rejects cursor with neither team_ids nor range."""- from dagster import SkipReason, build_sensor_context-- from posthog.dags.person_property_reconciliation import person_property_reconciliation_scheduler-- cursor = json.dumps(- {- "chunk_size": 100,- "bug_window_start": "2024-01-06 20:01:00",- "bug_window_end": "2024-01-07 14:52:00",- }- )- context = build_sensor_context(cursor=cursor)- result = person_property_reconciliation_scheduler(context)-- assert isinstance(result, SkipReason)- assert result.skip_message is not None- assert "team_ids" in result.skip_message- assert "range_start" in result.skip_message-- def test_sensor_validates_team_ids_and_range_mutually_exclusive(self):- """Test that sensor rejects cursor with both team_ids and range."""- from dagster import SkipReason, build_sensor_context-- from posthog.dags.person_property_reconciliation import person_property_reconciliation_scheduler-- cursor = json.dumps(- {- "team_ids": [1, 2, 3],- "range_start": 1,- "range_end": 100,- "bug_window_start": "2024-01-06 20:01:00",- "bug_window_end": "2024-01-07 14:52:00",- }- )- context = build_sensor_context(cursor=cursor)- result = person_property_reconciliation_scheduler(context)-- assert isinstance(result, SkipReason)- assert result.skip_message is not None- assert "both" in result.skip_message.lower()-- def test_sensor_team_ids_run_request_has_correct_tags(self):- """Test that team_ids mode run requests include expected tags."""- from dagster import DagsterInstance, SensorResult, build_sensor_context-- from posthog.dags.person_property_reconciliation import person_property_reconciliation_scheduler-- cursor = json.dumps(- {- "team_ids": [10, 20, 30],- "chunk_size": 2,- "max_concurrent_jobs": 5,- "next_team_index": 0,- "bug_window_start": "2024-01-06 20:01:00",- "bug_window_end": "2024-01-07 14:52:00",- }- )-- mock_instance = MagicMock(spec=DagsterInstance)- mock_instance.get_run_records.return_value = []-- context = build_sensor_context(cursor=cursor, instance=mock_instance)- result = person_property_reconciliation_scheduler(context)-- assert isinstance(result, SensorResult)- assert result.run_requests is not None- assert len(result.run_requests) == 2- assert result.run_requests[0].tags["reconciliation_team_ids_range"] == "0-1"- assert result.run_requests[0].tags["reconciliation_team_count"] == "2"- assert result.run_requests[0].tags["owner"] == "team-ingestion"diff --git a/posthog/management/commands/sync_persons_to_clickhouse.py b/posthog/management/commands/sync_persons_to_clickhouse.pyindex 422d271d9186..4afad1678ffe 100644--- a/posthog/management/commands/sync_persons_to_clickhouse.py+++ b/posthog/management/commands/sync_persons_to_clickhouse.py@@ -1,8 +1,9 @@"""Reconcile a team's person, distinct-id, and group rows from Postgres to ClickHouse.-LOCAL/DEV USE ONLY for now: this loads the whole team's rows into memory (no batching), so it-is not safe to run against large production teams. It exists to seed local ClickHouse from the-local persons database (e.g. via the generate_persons command).+LOCAL DEV AND SELF-HOSTED ONLY: this loads the whole team's rows into memory (no batching), so it+is not safe to run against large production teams, and it refuses to run on deployed PostHog Cloud.+It exists to seed local ClickHouse from the local persons database (e.g. via the generate_persons+command). On PostHog Cloud, `person_consistency` scans a team and repairs what it finds."""import json@@ -10,7 +11,7 @@ import loggingfrom typing import Literal, castfrom uuid import UUID-from django.core.management.base import BaseCommand+from django.core.management.base import BaseCommand, CommandErrorimport structlog@@ -29,6 +30,7 @@ from posthog.models.person.util import (set_distinct_id_version_floor,)from posthog.persons_db import persons_db_connection+from posthog.run_mode import run_modelogger = structlog.get_logger(__name__)logger.setLevel(logging.INFO)@@ -66,6 +68,11 @@ class Command(BaseCommand):def run(options):+ if run_mode().is_deployed_cloud:+ raise CommandError(+ "sync_persons_to_clickhouse runs only in local dev and self-hosted deployments. "+ "On PostHog Cloud, use person_consistency to scan and repair a team."+ )live_run = options["live_run"]deletes = options["deletes"]@@ -110,7 +117,7 @@ def run_person_sync(team_id: int, live_run: bool, deletes: bool, force: bool = F]rows = sync_execute("""- SELECT id, max(version) FROM person WHERE team_id = %(team_id)s GROUP BY id HAVING max(is_deleted) = 0+ SELECT id, max(version) FROM person WHERE team_id = %(team_id)s GROUP BY id HAVING argMax(is_deleted, version) = 0""",{"team_id": team_id,@@ -297,7 +304,7 @@ def run_distinct_id_sync(team_id: int, live_run: bool, deletes: bool):]rows = sync_execute("""- SELECT distinct_id, max(version) FROM person_distinct_id2 WHERE team_id = %(team_id)s GROUP BY distinct_id HAVING max(is_deleted) = 0+ SELECT distinct_id, max(version) FROM person_distinct_id2 WHERE team_id = %(team_id)s GROUP BY distinct_id HAVING argMax(is_deleted, version) = 0""",{"team_id": team_id,diff --git a/posthog/management/commands/test/test_sync_persons_to_clickhouse.py b/posthog/management/commands/test/test_sync_persons_to_clickhouse.pyindex da98f67fef9b..25d14b0e4d4e 100644--- a/posthog/management/commands/test/test_sync_persons_to_clickhouse.py+++ b/posthog/management/commands/test/test_sync_persons_to_clickhouse.py@@ -6,6 +6,9 @@ import pytestfrom posthog.test.base import ClickhouseTestMixin, NonAtomicBaseTestfrom unittest import mock+from django.core.management import call_command+from django.core.management.base import CommandError+from parameterized import parameterizedfrom psycopg.types.json import Jsonb@@ -213,14 +216,19 @@ class TestSyncPersonsToClickHouse(NonAtomicBaseTest, ClickhouseTestMixin):@parameterized.expand([- # (name, ClickHouse live version, expected row after the sync)- ("stored_version_wins", 5, ('{"abc": 123}', 9, True)),- ("clickhouse_ahead_is_raised_above", 9, ('{"abc": 123}', 10, True)),+ # (name, ClickHouse live version, an older ClickHouse tombstone under it, expected row after the sync)+ ("stored_version_wins", 5, False, ('{"abc": 123}', 9, True)),+ ("clickhouse_ahead_is_raised_above", 9, False, ('{"abc": 123}', 10, True)),+ ("live_above_an_older_tombstone", 5, True, ('{"abc": 123}', 9, True)),])- def test_persons_tombstoned_in_postgres_publish_the_stored_version(self, _name, ch_version, expected):+ def test_persons_tombstoned_in_postgres_publish_the_stored_version(+ self, _name, ch_version, older_tombstone, expected+ ):person_uuid = uuid4()self._seed_tombstoned_person(person_uuid, 9)+ if older_tombstone:+ create_person(uuid=str(person_uuid), team_id=self.team.pk, version=ch_version - 1, is_deleted=True)create_person(uuid=str(person_uuid), team_id=self.team.pk, version=ch_version, properties={"abc": 123})with fake_personhog_client() as personhog:@@ -315,13 +323,17 @@ class TestSyncPersonsToClickHouse(NonAtomicBaseTest, ClickhouseTestMixin):@parameterized.expand([- # (name, ClickHouse live version, mapping revives during the raise, expected version and is_deleted)- ("stored_version_wins", 3, False, (7, True)),- ("clickhouse_ahead_is_raised_above", 7, False, (8, True)),- ("revived_during_the_raise_is_published_live", 7, True, (8, False)),+ # (name, ClickHouse live version, mapping revives during the raise, an older ClickHouse tombstone+ # under the live row, expected version and is_deleted)+ ("stored_version_wins", 3, False, False, (7, True)),+ ("clickhouse_ahead_is_raised_above", 7, False, False, (8, True)),+ ("revived_during_the_raise_is_published_live", 7, True, False, (8, False)),+ ("live_above_an_older_tombstone", 3, False, True, (7, True)),])- def test_distinct_ids_tombstoned_in_postgres_publish_the_primary_row(self, _name, ch_version, revives, expected):+ def test_distinct_ids_tombstoned_in_postgres_publish_the_primary_row(+ self, _name, ch_version, revives, older_tombstone, expected+ ):person_uuid = uuid4()with persons_db_connection(writer=True, autocommit=True) as conn:person_id = insert_seed_person(conn, team_id=self.team.pk, properties={}, version=0, uuid=person_uuid)@@ -331,6 +343,14 @@ class TestSyncPersonsToClickHouse(NonAtomicBaseTest, ClickhouseTestMixin):"UPDATE posthog_persondistinctid SET is_deleted = true WHERE team_id = %s AND distinct_id = %s",[self.team.pk, "test-id"],)+ if older_tombstone:+ create_person_distinct_id(+ team_id=self.team.pk,+ distinct_id="test-id",+ person_id=str(person_uuid),+ is_deleted=True,+ version=ch_version - 1,+ )create_person_distinct_id(team_id=self.team.pk,distinct_id="test-id",@@ -360,6 +380,16 @@ class TestSyncPersonsToClickHouse(NonAtomicBaseTest, ClickhouseTestMixin):)self.assertEqual(cursor.fetchall(), [(version, is_deleted)])+ @parameterized.expand([("US",), ("EU",), ("DEV",)])+ def test_refuses_to_run_on_deployed_cloud(self, cloud_deployment):+ with persons_db_connection(writer=True, autocommit=True) as conn:+ insert_seed_person(conn, team_id=self.team.pk, properties={}, version=0)++ with mock.patch("posthog.settings.CLOUD_DEPLOYMENT", cloud_deployment), self.assertRaises(CommandError):+ call_command("sync_persons_to_clickhouse", team_id=self.team.pk, person=True, live_run=True)++ self.assertEqual(sync_execute("SELECT count() FROM person WHERE team_id = %(t)s", {"t": self.team.pk}), [(0,)])+@mock.patch(f"{posthog.management.commands.sync_persons_to_clickhouse.__name__}.raw_create_group_ch",wraps=posthog.management.commands.sync_persons_to_clickhouse.raw_create_group_ch,diff --git a/posthog/models/person/divergence/inspection.py b/posthog/models/person/divergence/inspection.pyindex 95869b26a07a..83d3fb075e96 100644--- a/posthog/models/person/divergence/inspection.py+++ b/posthog/models/person/divergence/inspection.py@@ -13,7 +13,7 @@ from django.db.models import Qfrom posthog.dataclasses import frozenfrom posthog.models.activity_logging.activity_log import ActivityLog-from posthog.models.person.divergence import lookups, observe+from posthog.models.person.divergence import cleanup_queue, lookups, observefrom posthog.models.person.divergence.flavors import classify_mapping, classify_personfrom posthog.models.person.util import list_person_tombstone_queuefrom posthog.models.team import Team@@ -37,6 +37,8 @@ class Inspection:ch_history: list[dict[str, Any]] | None = Noneactivity: list[dict[str, Any]] | None = Nonetombstone_queue: dict[str, Any] | None = None+ # The persons cleanup queue row: a `blocked_at` means the drain holds the person for a live distinct id.+ cleanup_queue: dict[str, Any] | None = Nonedef hash_distinct_id(team_id: int, distinct_id: str) -> str:@@ -151,6 +153,7 @@ def inspect_persons(pg = observe.pg_person_states(team_id, uuids) if team_exists else {}ch = observe.ch_person_states(team_id, uuids)history = _person_history(team_id, uuids)+ queued = cleanup_queue.queue_entries(team_id, uuids) if team_exists else {}results = []for person_uuid in uuids:pg_state = pg.get(person_uuid)@@ -174,6 +177,7 @@ def inspect_persons(ch_history=history[person_uuid],activity=_activity(team_id, person_uuid, pg_state.id if pg_state else None),tombstone_queue=_tombstone_queue_entry(team_id, person_uuid) if team_exists else None,+ cleanup_queue=queued.get(person_uuid),))return resultsdiff --git a/posthog/models/person/test/test_divergence_inspection.py b/posthog/models/person/test/test_divergence_inspection.pyindex 2159bf21e5d7..768c2af3ec05 100644--- a/posthog/models/person/test/test_divergence_inspection.py+++ b/posthog/models/person/test/test_divergence_inspection.py@@ -7,10 +7,12 @@ from posthog.test.base import BaseTest, ClickhouseTestMixinfrom django.core.management import call_commandfrom posthog.models.activity_logging.activity_log import ActivityLog, Detail+from posthog.models.person.divergence.cleanup_queue import PG_CLEANUP_QUEUE_TABLEfrom posthog.models.person.divergence.inspection import hash_distinct_id, inspect_mappings, inspect_personsfrom posthog.models.person.test.divergence_fixtures import DivergenceFixturesMixinfrom posthog.models.person.util import tombstone_persons_in_postgresfrom posthog.personhog_client.fake_client import get_active_fake+from posthog.persons_db import persons_db_connectionclass TestDivergenceInspection(DivergenceFixturesMixin, ClickhouseTestMixin, BaseTest):@@ -43,6 +45,12 @@ class TestDivergenceInspection(DivergenceFixturesMixin, ClickhouseTestMixin, Basself._ch_person(ghost, 3, {"email": "[email protected]"})self._log_person_activity("deleted", item_id=deleted.pk, name=str(deleted.uuid))+ with persons_db_connection(writer=True) as conn, conn.cursor() as cursor:+ cursor.execute(+ f"INSERT INTO {PG_CLEANUP_QUEUE_TABLE} (team_id, person_uuid, deleted_at, max_version, blocked_at) "+ "VALUES (%s, %s, now(), 3, now())",+ [self.team.pk, str(deleted.uuid)],+ )self._log_person_activity("updated", item_id=stale.pk, name=None)self._log_person_activity("updated", item_id=tie.pk, name=None)@@ -62,6 +70,9 @@ class TestDivergenceInspection(DivergenceFixturesMixin, ClickhouseTestMixin, Basqueued = by_key[str(deleted.uuid)].tombstone_queueassert queued is not None and queued["person_version"] == 3 and queued["tombstoned_at_ms"] > 0assert by_key[str(stale.uuid)].tombstone_queue is None+ drain_row = by_key[str(deleted.uuid)].cleanup_queue+ assert drain_row is not None and drain_row["max_version"] == 3 and drain_row["blocked_at"] is not None+ assert by_key[str(stale.uuid)].cleanup_queue is Noneassert flavors == {str(tie.uuid): ("tie", None, "republish"),str(blanked.uuid): ("blanked", None, "republish"),diff --git a/posthog/persons_seed.py b/posthog/persons_seed.pyindex a25f70f09663..a73105baf6d4 100644--- a/posthog/persons_seed.py+++ b/posthog/persons_seed.py@@ -26,11 +26,21 @@ from psycopg.types.json import Jsonbfrom posthog.models.utils import UUIDTfrom posthog.persons_db import persons_db_connection+from posthog.run_mode import run_mode# Not configurable like the person table; the model uses Django's default db_table.PERSON_DISTINCT_ID_TABLE = "posthog_persondistinctid"+def _refuse_in_deployed_cloud() -> None:+ # A seed command pointed at a real team on PostHog Cloud would write persons outside ingestion and personhog,+ # with no version bump and no tombstone check, so every write here refuses there.+ if run_mode().is_deployed_cloud:+ raise RuntimeError(+ "persons_seed writes only in local dev, tests and self-hosted deployments, not on deployed PostHog Cloud"+ )++def insert_seed_person(conn: psycopg.Connection[Any],*,@@ -52,6 +62,7 @@ def insert_seed_person(``last_seen_at`` and the ``properties_last_*`` maps default to NULL, mirroring``Person.objects.create`` with no override."""+ _refuse_in_deployed_cloud()with conn.cursor() as cursor:cursor.execute(f"INSERT INTO {settings.PERSON_TABLE_NAME} "@@ -88,6 +99,7 @@ def insert_seed_distinct_id(``version`` defaults to 0 but accepts ``None`` to write a NULL version (the column isnullable), mirroring ``PersonDistinctId.objects.create(version=None)``."""+ _refuse_in_deployed_cloud()with conn.cursor() as cursor:cursor.execute(f"INSERT INTO {PERSON_DISTINCT_ID_TABLE} (distinct_id, person_id, team_id, version) VALUES (%s, %s, %s, %s)",@@ -106,6 +118,7 @@ def insert_seed_group(created_at: datetime | None = None,) -> int:"""Insert one group row and return its id. ``created_at`` defaults to ``now()``; ``version`` is NOT NULL."""+ _refuse_in_deployed_cloud()with conn.cursor() as cursor:cursor.execute("INSERT INTO posthog_group "@@ -136,6 +149,7 @@ def insert_seed_group_type_mapping(``created_at`` is written as given (``None`` stays NULL — the model's custom ``save()`` stampscreated_at only on insert, which this bypasses); callers pass ``now()`` for the normal case."""+ _refuse_in_deployed_cloud()with conn.cursor() as cursor:cursor.execute("INSERT INTO posthog_grouptypemapping "@@ -168,6 +182,7 @@ def update_seed_person(is_identified: bool,) -> None:"""Overwrite ``properties`` and ``is_identified`` for an existing person."""+ _refuse_in_deployed_cloud()with conn.cursor() as cursor:cursor.execute(f"UPDATE {settings.PERSON_TABLE_NAME} "diff --git a/posthog/test/repo_invariants/dataclass_frozen_baseline.txt b/posthog/test/repo_invariants/dataclass_frozen_baseline.txtindex a735353c9354..d3b9e160dc55 100644--- a/posthog/test/repo_invariants/dataclass_frozen_baseline.txt+++ b/posthog/test/repo_invariants/dataclass_frozen_baseline.txt@@ -413,7 +413,6 @@2 posthog/dags/data_deletion_requests.py2 posthog/dags/part_breaker.py2 posthog/dags/person_overrides.py-2 posthog/dags/scripts/extract_reconciliation_results.py2 posthog/helpers/email_utils.py1 posthog/hogql/context.py2 posthog/hogql/database/models.py@@ -593,7 +592,6 @@4 products/warehouse_sources/backend/temporal/data_imports/sources/common/base.py5 posthog/api/proxy_record_diagnostics.py5 posthog/api/services/llm_prompt.py-5 posthog/dags/person_property_reconciliation.py5 posthog/temporal/ai_observability/evaluation_clustering/models.py5 posthog/temporal/ai_observability/run_trace_evaluation.py5 posthog/temporal/proxy_service/create.pydiff --git a/posthog/test/test_persons_seed.py b/posthog/test/test_persons_seed.pyindex 6afecf197129..cd87b7a1afcb 100644--- a/posthog/test/test_persons_seed.py+++ b/posthog/test/test_persons_seed.py@@ -1,13 +1,20 @@from typing import Anyimport pytest+from unittest.mock import MagicMock, patchfrom django.conf import settingsimport psycopgfrom posthog.persons_db import persons_db_connection-from posthog.persons_seed import insert_seed_distinct_id, insert_seed_person, update_seed_person+from posthog.persons_seed import (+ insert_seed_distinct_id,+ insert_seed_group,+ insert_seed_group_type_mapping,+ insert_seed_person,+ update_seed_person,+)# Sentinel team id used only by these tests. The seed helpers write through a raw# psycopg connection that commits independently of Django's test transaction, so each@@ -24,6 +31,36 @@ def _cleanup(conn: psycopg.Connection[Any]) -> None:class TestPersonsSeedHelpers:+ @pytest.mark.parametrize(+ "write",+ [+ lambda conn: insert_seed_person(conn, team_id=SEED_TEST_TEAM_ID, properties={}),+ lambda conn: insert_seed_distinct_id(conn, team_id=SEED_TEST_TEAM_ID, person_id=1, distinct_id="d"),+ lambda conn: insert_seed_group(+ conn, team_id=SEED_TEST_TEAM_ID, group_key="g", group_type_index=0, group_properties={}+ ),+ lambda conn: insert_seed_group_type_mapping(+ conn, project_id=SEED_TEST_TEAM_ID, team_id=SEED_TEST_TEAM_ID, group_type="org", group_type_index=0+ ),+ lambda conn: update_seed_person(+ conn,+ team_id=SEED_TEST_TEAM_ID,+ uuid="00000000-0000-0000-0000-000000000001",+ properties={},+ is_identified=False,+ ),+ ],+ ids=["insert_person", "insert_distinct_id", "insert_group", "insert_group_type_mapping", "update_person"],+ )+ def test_writes_refuse_on_deployed_cloud(self, write):+ conn = MagicMock()+ with (+ patch("posthog.settings.CLOUD_DEPLOYMENT", "US"),+ pytest.raises(RuntimeError, match="deployed PostHog Cloud"),+ ):+ write(conn)+ conn.cursor.assert_not_called()+def test_insert_person_persists_fields_and_sets_required_columns(self):with persons_db_connection(writer=True) as conn:try:diff --git a/products/mcp_analytics/backend/management/commands/seed_mcp_sessions.py b/products/mcp_analytics/backend/management/commands/seed_mcp_sessions.pyindex 3e8e5f648d6e..512b96388c71 100644--- a/products/mcp_analytics/backend/management/commands/seed_mcp_sessions.py+++ b/products/mcp_analytics/backend/management/commands/seed_mcp_sessions.py@@ -19,6 +19,7 @@ from posthog.models.utils import UUIDT, uuid7from posthog.personhog_client.caller_tag import personhog_caller_tagfrom posthog.persons_db import persons_db_connectionfrom posthog.persons_seed import insert_seed_distinct_id, insert_seed_person, update_seed_person+from posthog.run_mode import run_modefrom products.mcp_analytics.backend.models import MCPSession@@ -1257,6 +1258,10 @@ class Command(BaseCommand):)def handle(self, *args: Any, **options: Any) -> None:+ if run_mode().is_deployed_cloud:+ raise CommandError(+ "seed_mcp_sessions writes fabricated events and persons; it does not run on PostHog Cloud."+ )team_id: int = options["team_id"]session_count: int = options["sessions"]min_calls: int = options["min_calls"]