Skip to content

:hedgehog: PostHog is the leading platform for building self-driving products. Our developer tools – AI observability, analytics, session replay, flags, experiments, error tracking, logs, and more – capture all the context agents need to diagnose problems, uncover opportunities, and ship fixes. Steer it all from Slack, web, desktop, or the MCP.

chore(persons): retire one-off person repair jobs and guard dev seed tools on cloud (#126689)

Eli Reisman4cc8200599ab
diff --git a/.semgrep/rules/security/no-direct-kafka-producer.yaml b/.semgrep/rules/security/no-direct-kafka-producer.yaml
index 526df01665dc..c71f4df2f1b8 100644
--- a/.semgrep/rules/security/no-direct-kafka-producer.yaml
+++ b/.semgrep/rules/security/no-direct-kafka-producer.yaml
@@ -24,7 +24,6 @@ rules:
- /posthog/kafka_client/test/test_client.py
# Existing integration tests that explicitly opt out of the test stub
# to exercise a real broker. These are gated behind KAFKA_ROUNDTRIP_TESTS.
- - /posthog/dags/tests/test_person_property_reconciliation.py
- /posthog/temporal/tests/dlq_replay/test_activities.py
- /posthog/temporal/tests/common/test_logger.py
# DLQ replay needs per-message header passthrough, which confluent's
diff --git a/posthog/dags/PERSON_PROPERTY_RECONCILIATION.md b/posthog/dags/PERSON_PROPERTY_RECONCILIATION.md
deleted file mode 100644
index 57dc6de1b7ff..000000000000
--- a/posthog/dags/PERSON_PROPERTY_RECONCILIATION.md
+++ /dev/null
@@ -1,401 +0,0 @@
-# Person Property Reconciliation Job
-
-This document provides a comprehensive overview of the person property reconciliation job, including architecture diagrams, parameter reference, and example configurations.
-
-For detailed query explanations, see [person_property_reconciliation_query_explanation.md](person_property_reconciliation_query_explanation.md).
-
-## Overview
-
-The reconciliation job fixes person properties that were missed due to a bug where `PERSON_BATCH_WRITING_DB_WRITE_MODE=ASSERT_VERSION` caused `updatePersonAssertVersion()` to not properly merge properties.
-
-The job:
-
-1. Reads events from ClickHouse to find property updates (`$set`, `$set_once`, `$unset`) within a bug window
-2. Compares with current person properties in ClickHouse
-3. Applies any missed updates to Postgres
-4. Publishes updated persons to Kafka for ClickHouse ingestion
-
-## Architecture
-
-### Non-Windowed Mode (Default)
-
-Best for: Smaller teams or short bug windows where a single ClickHouse query can handle the data.
-
-```text
-┌─────────────────────────────────────────────────────────────────────────────┐
-│ Non-Windowed Flow │
-└─────────────────────────────────────────────────────────────────────────────┘
-
- ┌──────────────────────────────────────────────────────────────────┐
- │ ClickHouse: Single Query │
- │ ┌────────────────────────────────────────────────────────────┐ │
- │ │ get_person_property_updates_from_clickhouse() │ │
- │ │ │ │
- │ │ • Scans events from bug_window_start to now() │ │
- │ │ • Joins with person_distinct_id_overrides (merge handling) │ │
- │ │ • Joins with person table for current state │ │
- │ │ • Computes diffs IN SQL (efficient) │ │
- │ │ • Returns only persons with actual differences │ │
- │ └────────────────────────────────────────────────────────────┘ │
- └──────────────────────────────────────────────────────────────────┘
- │
- ▼
- ┌───────────────────────────────┐
- │ filter_event_person_properties │
- │ (resolve set/unset conflicts) │
- └───────────────────────────────┘
- │
- ▼
- ┌───────────────────────────────┐
- │ process_persons_in_batches │
- │ • Update Postgres │
- │ • Publish to Kafka │
- │ • Commit in batches │
- └───────────────────────────────┘
-```
-
-### Windowed Batched Mode
-
-Best for: Large teams with millions of persons where a single query would OOM.
-
-```text
-┌─────────────────────────────────────────────────────────────────────────────┐
-│ Windowed Batched Flow │
-└─────────────────────────────────────────────────────────────────────────────┘
-
- ┌──────────────────────────────────────────────────────────────────┐
- │ Step 1: Identify Affected Persons (BOUNDED) │
- │ ┌────────────────────────────────────────────────────────────┐ │
- │ │ get_affected_person_ids_from_clickhouse() │ │
- │ │ │ │
- │ │ • Aggregates event properties within the bug window │ │
- │ │ • Compares against current person state in CH │ │
- │ │ • Returns ONLY person_ids with actual diffs │ │
- │ └────────────────────────────────────────────────────────────┘ │
- └──────────────────────────────────────────────────────────────────┘
- │
- ▼
- ┌──────────────────────────────────────────────────────────────────┐
- │ Step 2: Process Persons in Batches (~10K each) │
- │ │
- │ FOR EACH person_batch (chunk of ~10K person_ids): │
- │ │
- │ ┌────────────────────────────────────────────────────────────┐ │
- │ │ Step 2a: Windowed Event Aggregation │ │
- │ │ │ │
- │ │ FOR EACH time_window (1 hour chunks from start to now): │ │
- │ │ • Query events filtered to this batch's person_ids │ │
- │ │ • Merge into batch accumulator (timestamp-based) │ │
- │ └────────────────────────────────────────────────────────────┘ │
- │ │ │
- │ ▼ │
- │ ┌────────────────────────────────────────────────────────────┐ │
- │ │ Step 2b: Compare with Person State │ │
- │ │ │ │
- │ │ • Query current person properties from ClickHouse │ │
- │ │ • Filter to actual diffs (set: value differs, etc.) │ │
- │ └────────────────────────────────────────────────────────────┘ │
- │ │ │
- │ ▼ │
- │ ┌────────────────────────────────────────────────────────────┐ │
- │ │ Step 2c: Process & Commit Batch │ │
- │ │ │ │
- │ │ • Filter set/unset conflicts │ │
- │ │ • Update Postgres │ │
- │ │ • Publish to Kafka │ │
- │ │ • Commit transaction │ │
- │ │ • Clear memory for next batch │ │
- │ └────────────────────────────────────────────────────────────┘ │
- │ │
- │ END FOR EACH person_batch │
- └──────────────────────────────────────────────────────────────────┘
-```
-
-### Why Windowed Batched Mode?
-
-Step 1 aggregates event properties within the bug window and compares against current person state,
-returning only person_ids with actual diffs.
-Step 2 then reads events up to `now()` to capture all property updates for those persons.
-
-This prevents the problem where the original windowed implementation returned ALL persons
-with property-setting events (regardless of whether they had diffs), causing 100x over-selection and OOM.
-
-## Parameter Reference
-
-### Job Configuration (`PersonPropertyReconciliationConfig`)
-
-| Parameter | Type | Default | Description |
-| ------------------------------------ | ------------------- | -------- | ------------------------------------------------------------------------ |
-| `bug_window_start` | `str` | Required | Start of bug window (ClickHouse format: "YYYY-MM-DD HH:MM:SS", UTC) |
-| `bug_window_end` | `str \| None` | `None` | End of bug window. Required if `team_ids` not supplied |
-| `team_ids` | `list[int] \| None` | `None` | Explicit list of team IDs to process |
-| `min_team_id` | `int \| None` | `None` | Minimum team_id (inclusive) for range scanning |
-| `max_team_id` | `int \| None` | `None` | Maximum team_id (inclusive) for range scanning |
-| `exclude_team_ids` | `list[int] \| None` | `None` | Team IDs to exclude from processing |
-| `dry_run` | `bool` | `False` | Log changes without applying to Postgres |
-| `backup_enabled` | `bool` | `True` | Store before/after state in backup table |
-| `batch_size` | `int` | `100` | Commit Postgres transaction every N persons (0 = single commit) |
-| `teams_per_chunk` | `int` | `100` | Teams per k8s task (reduces pod overhead) |
-| `team_ch_props_fetch_window_seconds` | `int` | `0` | **Mode selector**: 0 = non-windowed, >0 = windowed with N-second windows |
-| `person_batch_size` | `int` | `10000` | Persons per batch in windowed mode (keeps IN clauses under limit) |
-
-### Sensor Configuration (`ReconciliationSchedulerConfig`)
-
-The sensor automates launching multiple reconciliation jobs across teams.
-
-| Parameter | Type | Default | Description |
-| ------------------------------------ | ------------------- | ----------------------- | ----------------------------------------- |
-| **Team Selection** | | | |
-| `team_ids` | `list[int] \| None` | `None` | Explicit list of team IDs (Option 1) |
-| `range_start` | `int \| None` | `None` | First team_id for range scan (Option 2) |
-| `range_end` | `int \| None` | `None` | Last team_id for range scan (Option 2) |
-| **Concurrency** | | | |
-| `chunk_size` | `int` | `1000` | Teams per job run |
-| `max_concurrent_jobs` | `int` | `5` | Max reconciliation jobs at once (cap: 50) |
-| `max_concurrent_tasks` | `int` | `10` | Max k8s pods per job (cap: 100) |
-| **Bug Window** | | | |
-| `bug_window_start` | `str` | Required | Start of bug window |
-| `bug_window_end` | `str` | Required | End of bug window |
-| **Processing** | | | |
-| `dry_run` | `bool` | `False` | Log changes without applying |
-| `backup_enabled` | `bool` | `True` | Store before/after state |
-| `batch_size` | `int` | `100` | Postgres commit batch size |
-| `teams_per_chunk` | `int` | `100` | Teams per k8s task |
-| `team_ch_props_fetch_window_seconds` | `int` | `0` | Mode selector (0 = non-windowed) |
-| `person_batch_size` | `int` | `10000` | Persons per batch in windowed mode |
-| **Resources** | | | |
-| `persons_db_env_var` | `str` | `PERSONS_DB_WRITER_URL` | Env var for Postgres connection |
-
-## Example Sensor Cursor Configurations
-
-Set the sensor cursor in Dagster UI to one of these JSON configurations.
-
-### Non-Windowed Mode (Small Teams)
-
-Best for teams with < 100K affected persons or short bug windows.
-
-```json
-{
- "team_ids": [123, 456, 789],
- "bug_window_start": "2026-01-06 20:01:00",
- "bug_window_end": "2026-01-07 14:52:00",
- "team_ch_props_fetch_window_seconds": 0,
- "dry_run": true,
- "backup_enabled": true,
- "batch_size": 100
-}
-```
-
-### Windowed Batched Mode (Large Teams)
-
-Best for teams with 100K+ affected persons. Uses 1-hour windows and 10K person batches.
-
-```json
-{
- "team_ids": [2],
- "bug_window_start": "2026-01-06 20:01:00",
- "bug_window_end": "2026-01-07 14:52:00",
- "team_ch_props_fetch_window_seconds": 3600,
- "person_batch_size": 10000,
- "dry_run": true,
- "backup_enabled": true,
- "batch_size": 100
-}
-```
-
-### Range-Based Scanning (All Teams)
-
-Process all teams in a team_id range, in chunks.
-
-```json
-{
- "range_start": 1,
- "range_end": 100000,
- "chunk_size": 1000,
- "bug_window_start": "2026-01-06 20:01:00",
- "bug_window_end": "2026-01-07 14:52:00",
- "team_ch_props_fetch_window_seconds": 0,
- "max_concurrent_jobs": 5,
- "max_concurrent_tasks": 10,
- "dry_run": false
-}
-```
-
-### Production Run with Windowed Mode
-
-Full production configuration for a large team.
-
-```json
-{
- "team_ids": [2],
- "bug_window_start": "2026-01-06 20:01:00",
- "bug_window_end": "2026-01-07 14:52:00",
- "team_ch_props_fetch_window_seconds": 3600,
- "person_batch_size": 10000,
- "dry_run": false,
- "backup_enabled": true,
- "batch_size": 100,
- "teams_per_chunk": 1,
- "max_concurrent_jobs": 1,
- "max_concurrent_tasks": 1
-}
-```
-
-## Choosing the Right Mode
-
-| Scenario | Recommended Mode | Config |
-| ---------------------------- | ---------------- | ------------------------------------------ |
-| Small team (< 100K persons) | Non-windowed | `team_ch_props_fetch_window_seconds: 0` |
-| Short bug window (< 1 day) | Non-windowed | `team_ch_props_fetch_window_seconds: 0` |
-| Large team (100K+ persons) | Windowed batched | `team_ch_props_fetch_window_seconds: 3600` |
-| Long bug window (days/weeks) | Windowed batched | `team_ch_props_fetch_window_seconds: 3600` |
-| Unknown size, being cautious | Windowed batched | Start with `dry_run: true` |
-
-## Recommended Workflow
-
-For a full reconciliation across all teams, use this two-pass approach:
-
-### Pass 1: Non-Windowed Mode for Most Teams
-
-Run the sensor with non-windowed mode to process the majority of teams quickly:
-
-```json
-{
- "range_start": 1,
- "range_end": 100000,
- "chunk_size": 1000,
- "bug_window_start": "2026-01-06 20:01:00",
- "bug_window_end": "2026-01-07 14:52:00",
- "team_ch_props_fetch_window_seconds": 0,
- "max_concurrent_jobs": 5,
- "max_concurrent_tasks": 10,
- "dry_run": false,
- "backup_enabled": true
-}
-```
-
-This will complete successfully for most teams but may fail (OOM or timeout) for very large teams.
-
-### Extract Failed Teams
-
-Use the extraction script to identify which teams failed:
-
-```bash
-export DAGSTER_CLOUD_TOKEN="<your-user-token>"
-
-# Extract failed teams from recent runs
-python posthog/dags/scripts/extract_reconciliation_results.py \
- --org posthog \
- --deployment prod-us \
- --since "2026-01-27" \
- --until "2026-01-30" \
- --verbose \
- --output-csv failed_teams.csv
-
-# The script outputs:
-# - Summary of succeeded/failed teams
-# - List of failed team IDs
-# - CSV with error details for each failed team
-```
-
-See the script's `--help` for additional options (filter by status, specific run ID, etc.).
-
-### Pass 2: Windowed Mode for Failed Teams
-
-Re-run the sensor targeting only the failed teams using windowed batched mode:
-
-```json
-{
- "team_ids": [2, 15847, 23456],
- "bug_window_start": "2026-01-06 20:01:00",
- "bug_window_end": "2026-01-07 14:52:00",
- "team_ch_props_fetch_window_seconds": 3600,
- "person_batch_size": 10000,
- "max_concurrent_jobs": 1,
- "max_concurrent_tasks": 1,
- "dry_run": false,
- "backup_enabled": true,
- "teams_per_chunk": 1
-}
-```
-
-Key differences for Pass 2:
-
-- `team_ids`: Explicit list of failed teams from the extraction script
-- `team_ch_props_fetch_window_seconds: 3600`: Enable windowed mode (1-hour windows)
-- `person_batch_size: 10000`: Process persons in batches to avoid OOM
-- `max_concurrent_jobs: 1` and `max_concurrent_tasks: 1`: Serialize for large teams
-- `teams_per_chunk: 1`: One team per task for better isolation
-
-If a team still fails, try reducing `person_batch_size` (e.g., 5000) or check the error logs for specific issues.
-
-## Monitoring and Troubleshooting
-
-### Logs to Watch
-
-In windowed batched mode, look for these log patterns:
-
-```text
-# Step 1: Affected persons query
-Querying affected persons: team_id={id}, bug_window=[{start}, {end}]
-Found {N} affected persons for team_id={id}
-
-# Step 2: Per-batch progress
-Starting batched windowed processing: team_id={id}, {N} batches of ~10000 persons, ~{M} windows of 3600s each
-Processing batch {n}/{total} for team_id={id}, {N} persons
-Batch {n}: processed window {w}/{total_windows}, accumulated {N} persons
-Batch {n}: comparing {N} persons against current state
-Batch {n}: yielding {N} persons with diffs
-
-# Completion
-Completed team_id={id} (windowed batched): processed={N}, updated={M}, skipped={K}, person_batches={B}
-```
-
-### Error Logs
-
-When failures occur, errors are logged via `logger.exception()` which includes the full stack trace. Look for these message patterns:
-
-```text
-# Failed to query affected persons
-Failed to query affected persons for team_id={id}, bug_window=[{start}, {end}]
-[full stack trace follows]
-
-# Failed during a specific window query
-Batch {n}: failed to query window {w}/{total} [{window_start}, {window_end}) for team_id={id}
-[full stack trace follows]
-Batch {n}: failed during windowed event collection for team_id={id} after {w} windows, accumulated {N} persons
-[full stack trace follows]
-
-# Failed during person state comparison
-Batch {n}: failed to compare {N} persons against current state for team_id={id}
-[full stack trace follows]
-```
-
-**Partial Progress**: If a failure occurs mid-processing, already-committed batches are persisted to Postgres and Kafka. The job will be marked as failed, but partial progress is saved.
-
-### Common Issues
-
-1. **OOM in windowed mode**: Reduce `person_batch_size` (e.g., 5000 instead of 10000)
-2. **Query size exceeded**: The code handles this by batching person_ids, but if you see this error, reduce `person_batch_size`
-3. **Slow progress**: Increase `team_ch_props_fetch_window_seconds` (e.g., 7200 for 2-hour windows) to reduce query count
-
-### Metrics
-
-The job emits these metrics to ClickHouse:
-
-- `person_property_reconciliation_persons_processed_total`
-- `person_property_reconciliation_persons_updated_total`
-- `person_property_reconciliation_persons_skipped_total`
-- `person_property_reconciliation_teams_succeeded_total`
-- `person_property_reconciliation_teams_failed_total`
-- `person_property_reconciliation_duration_seconds_total`
-- `person_property_reconciliation_error` (on failures)
-
-## Backup and Recovery
-
-When `backup_enabled: true`, the job stores before/after state in `posthog_person_reconciliation_backup`:
-
-- `job_id`: Dagster run ID
-- `properties` / `properties_after`: Before/after JSON
-- `version` / `version_after`: Before/after version numbers
-- `pending_operations`: Array of operations that were applied
diff --git a/posthog/dags/common/resources.py b/posthog/dags/common/resources.py
index 7148fe30f847..567f8980eae6 100644
--- a/posthog/dags/common/resources.py
+++ b/posthog/dags/common/resources.py
@@ -1,10 +1,8 @@
import json
import socket
import asyncio
-from collections.abc import Generator
from dataclasses import dataclass
from typing import Any
-from urllib.parse import urlparse
import dagster
import psycopg2
@@ -23,9 +21,6 @@ from posthog.clickhouse.client.connection import (
is_file_backed_user,
)
from posthog.clickhouse.cluster import ClickhouseCluster, RetryPolicy, get_cluster
-from posthog.kafka_client.client import _KafkaProducer
-from posthog.kafka_client.profiles import KafkaClusterProfile
-from posthog.kafka_client.routing import get_producer
from posthog.redis import get_client, redis
from posthog.utils import initialize_self_capture_api_token
@@ -284,27 +279,6 @@ class PostHogAnalyticsResource(dagster.ConfigurableResource):
return None
-class PostgresURLResource(dagster.ConfigurableResource):
- """
- Postgres connection that parses a connection URL.
- Delegates to PostgresResource for actual connection logic.
- Expects format: postgres://user:pass@host:port/dbname
- """
-
- connection_url: str
-
- def create_resource(self, context: dagster.InitResourceContext) -> psycopg2.extensions.connection:
- parsed = urlparse(self.connection_url)
- pg = PostgresResource(
- host=parsed.hostname or "",
- port=str(parsed.port or 5432),
- database=parsed.path.lstrip("/"),
- user=parsed.username or "",
- password=parsed.password or "",
- )
- return pg.create_resource(context)
-
-
class PostgresURL(dagster.ConfigurableResource):
"""A Postgres connection URL handed to the op unopened.
@@ -319,25 +293,6 @@ class PostgresURL(dagster.ConfigurableResource):
return self.connection_url
-def kafka_producer_resource(context: dagster.InitResourceContext) -> Generator[_KafkaProducer]:
- """Yield a singleton Kafka producer bound to the INGESTION (WarpStream) profile; flush on teardown.
-
- Consumers of this resource produce to `clickhouse_person` / `clickhouse_person_distinct_id`,
- which the routing map sends to the INGESTION profile. Binding the resource here keeps
- that explicit so a chart misconfiguration (missing `KAFKA_INGESTION_HOSTS`)
- fails loud rather than silently dropping writes via the DEFAULT fallback.
-
- Ops producing to topics on a different profile must call
- `posthog.kafka_client.routing.get_producer(topic=...)` directly.
- """
- producer = get_producer(profile=KafkaClusterProfile.INGESTION)
- try:
- yield producer
- finally:
- producer.flush()
-
-
def _is_retryable_http_error(exception: BaseException) -> bool:
"""Check if an HTTP error should be retried."""
if isinstance(exception, requests.exceptions.HTTPError):
diff --git a/posthog/dags/fix_person_id_overrides.py b/posthog/dags/fix_person_id_overrides.py
deleted file mode 100644
index 687072b9b5a9..000000000000
--- a/posthog/dags/fix_person_id_overrides.py
+++ /dev/null
@@ -1,182 +0,0 @@
-"""Dagster job to insert person_id overrides for distinct_ids.
-
-Given a list of "bugged" distinct_ids, for each one:
-1. Find its person_id in person_distinct_id2
-2. Find ALL other distinct_ids mapped to that same person_id
-3. Insert ALL of them into person_distinct_id_overrides
-"""
-
-from typing import Optional
-
-import dagster
-import pydantic
-
-from posthog.clickhouse.client import sync_execute
-from posthog.dags.common import JobOwners
-
-
-class FixPersonIdOverridesConfig(dagster.Config):
- """Configuration for the fix person_id overrides job."""
-
- team_id: int = pydantic.Field(description="Team ID to process")
- distinct_ids: str = pydantic.Field(description="Comma-separated list of bugged distinct_ids")
- dry_run: bool = pydantic.Field(
- default=True,
- description="If true, don't actually insert, just log what would be inserted",
- )
-
-
-def get_person_id_from_pdi2(team_id: int, distinct_id: str) -> Optional[tuple[str, int]]:
- """Get the current person_id and version for a distinct_id from person_distinct_id2."""
- result = sync_execute(
- """
- SELECT
- argMax(person_id, version) as pid,
- max(version) as max_version
- FROM person_distinct_id2
- WHERE team_id = %(team_id)s
- AND distinct_id = %(distinct_id)s
- GROUP BY distinct_id
- HAVING argMax(is_deleted, version) = 0
- """,
- {"team_id": team_id, "distinct_id": distinct_id},
- )
- if result and result[0]:
- return str(result[0][0]), int(result[0][1])
- return None
-
-
-def get_all_distinct_ids_for_person(team_id: int, person_id: str) -> list[tuple[str, int]]:
- """Get ALL distinct_ids mapped to a person_id in person_distinct_id2."""
- result = sync_execute(
- """
- SELECT
- distinct_id,
- max(version) as max_version
- FROM person_distinct_id2
- WHERE team_id = %(team_id)s
- GROUP BY distinct_id
- HAVING argMax(person_id, version) = %(person_id)s
- AND argMax(is_deleted, version) = 0
- """,
- {"team_id": team_id, "person_id": person_id},
- )
- return [(row[0], int(row[1])) for row in result] if result else []
-
-
-def get_existing_override(team_id: int, distinct_id: str) -> Optional[tuple[str, int]]:
- """Check if an override already exists."""
- result = sync_execute(
- """
- SELECT
- argMax(person_id, version) as pid,
- max(version) as max_version
- FROM person_distinct_id_overrides
- WHERE team_id = %(team_id)s
- AND distinct_id = %(distinct_id)s
- GROUP BY distinct_id
- HAVING argMax(is_deleted, version) = 0
- """,
- {"team_id": team_id, "distinct_id": distinct_id},
- )
- if result and result[0]:
- return str(result[0][0]), int(result[0][1])
- return None
-
-
-def insert_override_batch(overrides: list[tuple[int, str, str, int]]) -> None:
- """Insert a batch of records into person_distinct_id_overrides.
-
- Args:
- overrides: List of (team_id, distinct_id, person_id, version) tuples
- """
- if not overrides:
- return
-
- from datetime import datetime
-
- now = datetime.now()
- sync_execute(
- """
- INSERT INTO person_distinct_id_overrides
- (team_id, distinct_id, person_id, is_deleted, version, _timestamp, _offset, _partition)
- VALUES
- """,
- [
- (team_id, distinct_id, person_id, 0, version, now, 0, 0)
- for team_id, distinct_id, person_id, version in overrides
- ],
- )
-
-
-def fix_person_id_overrides_op(
- context: dagster.OpExecutionContext,
- config: FixPersonIdOverridesConfig,
-) -> None:
- """
- For each bugged distinct_id:
- 1. Find its person_id in person_distinct_id2
- 2. Find ALL distinct_ids mapped to that person_id
- 3. Insert ALL mappings into person_distinct_id_overrides
- """
- # Parse comma-separated distinct_ids
- bugged_distinct_ids = [d.strip() for d in config.distinct_ids.split(",") if d.strip()]
-
- context.log.info(f"Processing {len(bugged_distinct_ids)} bugged distinct_ids")
- context.log.info(f"Dry run: {config.dry_run}")
-
- # Track which person_ids we've already processed to avoid duplicates
- processed_person_ids: set[str] = set()
-
- # Batch of overrides to insert: (team_id, distinct_id, person_id, version)
- pending_overrides: list[tuple[int, str, str, int]] = []
-
- for _i, bugged_distinct_id in enumerate(bugged_distinct_ids):
- # Step 1: Get person_id for this bugged distinct_id
- pdi2_result = get_person_id_from_pdi2(config.team_id, bugged_distinct_id)
- if not pdi2_result:
- context.log.warning(f"No person found for bugged distinct_id={bugged_distinct_id}")
- continue
-
- person_id, _ = pdi2_result
-
- # Skip if we already processed this person
- if person_id in processed_person_ids:
- context.log.info(f"Already processed person_id={person_id}... for distinct_id={bugged_distinct_id}")
- continue
-
- processed_person_ids.add(person_id)
-
- # Step 2: Get ALL distinct_ids mapped to this person
- all_distinct_ids = get_all_distinct_ids_for_person(config.team_id, person_id)
- context.log.info(
- f"Person {person_id}... has {len(all_distinct_ids)} distinct_ids: {', '.join(did for did, _ in all_distinct_ids)} "
- )
-
- # Step 3: Collect overrides for each distinct_id
- for distinct_id, version in all_distinct_ids:
- # Check if override already exists
- existing = get_existing_override(config.team_id, distinct_id)
- if existing:
- continue
-
- # Use version > 0 so it gets picked up by the overrides MV
- new_version = max(version, 1)
-
- if config.dry_run:
- context.log.info(f" [DRY RUN] Would insert -> Distinct ID: {distinct_id}, Person ID: {person_id}...")
- else:
- pending_overrides.append((config.team_id, distinct_id, person_id, new_version))
- context.log.info(f" Queued: {distinct_id} -> {person_id}...")
-
- # Insert all overrides at once
- if not config.dry_run and pending_overrides:
- insert_override_batch(pending_overrides)
- context.log.info(f"Inserted {len(pending_overrides)} overrides")
-
-
[email protected](tags={"owner": JobOwners.TEAM_CLICKHOUSE.value})
-def fix_person_id_overrides_job():
- """Job to insert person_id overrides for specified distinct_ids."""
- fix_person_id_overrides_op()
diff --git a/posthog/dags/locations/__init__.py b/posthog/dags/locations/__init__.py
index 432c49fec540..1ceb2ee2e40a 100644
--- a/posthog/dags/locations/__init__.py
+++ b/posthog/dags/locations/__init__.py
@@ -11,10 +11,8 @@ from posthog.dags.common.resources import (
ClickhouseClusterResource,
PostgresResource,
PostgresURL,
- PostgresURLResource,
PostHogAnalyticsResource,
RedisResource,
- kafka_producer_resource,
)
# Default loggers for every code location's jobs. Overrides Dagster's
@@ -46,15 +44,9 @@ resources_by_env = {
"posthoganalytics": dagster.ResourceDefinition.none_resource(
description="Dummy PostHogAnalytics resource since posthoganalytics is configured properly in production."
),
- # Persons DB resource (parses connection URL)
- "persons_database": PostgresURLResource(
- connection_url=dagster.EnvVar("PERSONS_DB_WRITER_URL"),
- ),
"persons_database_url": PostgresURL(
connection_url=dagster.EnvVar("PERSONS_DB_WRITER_URL"),
),
- # Kafka producer (auto-configured from Django settings)
- "kafka_producer": kafka_producer_resource,
# Clay webhook for job switchers pipeline
"clay_webhook_job_switchers": ClayWebhookResource(
webhook_url=dagster.EnvVar("CLAY_JOB_SWITCHERS_WEBHOOK_URL"),
@@ -85,15 +77,9 @@ resources_by_env = {
password=dagster.EnvVar("POSTGRES_PASSWORD"),
),
"posthoganalytics": PostHogAnalyticsResource(personal_api_key=dagster.EnvVar("PERSONAL_API_KEY")),
- # Persons DB resource (parses connection URL)
- "persons_database": PostgresURLResource(
- connection_url=dagster.EnvVar("PERSONS_DB_WRITER_URL"),
- ),
"persons_database_url": PostgresURL(
connection_url=dagster.EnvVar("PERSONS_DB_WRITER_URL"),
),
- # Kafka producer (auto-configured from Django settings)
- "kafka_producer": kafka_producer_resource,
# Clay webhook for job switchers pipeline
"clay_webhook_job_switchers": ClayWebhookResource(
webhook_url=dagster.EnvVar("CLAY_JOB_SWITCHERS_WEBHOOK_URL"),
diff --git a/posthog/dags/locations/clickhouse.py b/posthog/dags/locations/clickhouse.py
index b65d5d788cde..ed1c4ff2abe4 100644
--- a/posthog/dags/locations/clickhouse.py
+++ b/posthog/dags/locations/clickhouse.py
@@ -11,7 +11,6 @@ from posthog.dags import (
deletes,
drop_materialized_column,
export_query_log_archive_to_s3,
- fix_person_id_overrides,
flag_evaluations_backfill,
flag_evaluations_person_id_repair,
part_breaker,
@@ -45,7 +44,6 @@ defs = dagster.Definitions(
deletes.monthly_retention_events_cleanup_job,
export_query_log_archive_to_s3.export_query_log_archive_to_s3,
backfill_materialized_column.backfill_materialized_column,
- fix_person_id_overrides.fix_person_id_overrides_job,
flag_evaluations_backfill.flag_evaluations_backfill_job,
flag_evaluations_person_id_repair.cleanup_orphaned_flag_evaluations_person_id_repair,
flag_evaluations_person_id_repair.flag_evaluations_person_id_repair_job,
diff --git a/posthog/dags/locations/ingestion.py b/posthog/dags/locations/ingestion.py
index 13fb3456f155..b77c2b17c2fc 100644
--- a/posthog/dags/locations/ingestion.py
+++ b/posthog/dags/locations/ingestion.py
@@ -1,6 +1,6 @@
import dagster
-from posthog.dags import distinct_id_usage, ingestion_assets, person_property_reconciliation
+from posthog.dags import distinct_id_usage, ingestion_assets
from . import loggers, resources
@@ -10,14 +10,10 @@ defs = dagster.Definitions(
],
jobs=[
distinct_id_usage.distinct_id_usage_monitoring,
- person_property_reconciliation.person_property_reconciliation_job,
],
schedules=[
distinct_id_usage.distinct_id_usage_monitoring_schedule,
],
- sensors=[
- person_property_reconciliation.person_property_reconciliation_scheduler,
- ],
loggers=loggers,
resources=resources,
)
diff --git a/posthog/dags/owners.yaml b/posthog/dags/owners.yaml
index 65a0f1eec679..813dd2db800f 100644
--- a/posthog/dags/owners.yaml
+++ b/posthog/dags/owners.yaml
@@ -1,11 +1,6 @@
version: 1
owners: []
rules:
- - match:
- - '/fix_person_id_overrides.py'
- - '/person_property_reconciliation.py'
- - '/tests/test_person_property_reconciliation.py'
- owners: team-ingestion
- match: '/sessions.py'
owners: team-analytics-platform
- match:
diff --git a/posthog/dags/person_property_reconciliation.py b/posthog/dags/person_property_reconciliation.py
deleted file mode 100644
index 1a948d145257..000000000000
--- a/posthog/dags/person_property_reconciliation.py
+++ /dev/null
@@ -1,2584 +0,0 @@
-"""Dagster job for reconciling person properties that were missed due to a bug where
-PERSON_BATCH_WRITING_DB_WRITE_MODE=ASSERT_VERSION caused updatePersonAssertVersion()
-to not properly merge properties.
-
-This job reads events from ClickHouse to find property updates ($set, $set_once, $unset)
-within a bug window, compares timestamps with properties_last_updated_at in Postgres,
-and applies any missed updates.
-
-For comprehensive documentation including architecture diagrams, parameter reference,
-and example sensor configurations, see:
- PERSON_PROPERTY_RECONCILIATION.md
-
-For detailed query explanations, see:
- person_property_reconciliation_query_explanation.md
-"""
-
-import json
-import time
-from collections.abc import Iterator
-from dataclasses import dataclass
-from datetime import UTC, datetime, timedelta
-from typing import Any
-
-from django.conf import settings
-
-import dagster
-import psycopg2
-from dagster_k8s import k8s_job_executor
-
-from posthog.clickhouse.client import sync_execute
-from posthog.clickhouse.cluster import ClickhouseCluster
-from posthog.clickhouse.custom_metrics import MetricsClient
-from posthog.dags.common import JobOwners
-from posthog.dataclasses import frozen
-from posthog.kafka_client.client import _KafkaProducer
-from posthog.kafka_client.topics import KAFKA_PERSON
-
-# Use in_process_executor locally (k8s_job_executor doesn't work outside k8s)
-executor_def = dagster.in_process_executor if settings.DEBUG else k8s_job_executor
-
-
-class PersonPropertyReconciliationConfig(dagster.Config):
- """Configuration for the person property reconciliation job."""
-
- bug_window_start: str # ClickHouse format: "YYYY-MM-DD HH:MM:SS" (assumed UTC)
- team_ids: list[int] | None = None # Optional: filter to specific teams
- bug_window_end: str | None = None # Optional: required if team_ids not supplied
- min_team_id: int | None = None # Optional: only process teams with id >= this value
- max_team_id: int | None = None # Optional: only process teams with id <= this value
- exclude_team_ids: list[int] | None = None # Optional: exclude specific team_ids
- dry_run: bool = False # Log changes without applying
- backup_enabled: bool = True # Store before/after state in backup table
- batch_size: int = 100 # Commit Postgres transaction every N persons (0 = single commit at end)
- teams_per_chunk: int = 100 # Number of teams to process per task (reduces task overhead)
- team_ch_props_fetch_window_seconds: int = 0 # 0 = single query; >0 = split into N-second windows
- person_batch_size: int = 10000 # Persons per batch in windowed mode (keeps IN clauses small)
-
-
-@dataclass
-class PropertyValue:
- """A property value with its timestamp."""
-
- timestamp: datetime # UTC datetime
- value: Any | None # None for unset operations
-
-
-@dataclass
-class PersonPropertyDiffs:
- """Property diffs for a single person, organized by operation type."""
-
- person_id: str
- person_version: int # Version from CH person table (baseline for conflict detection)
- set_updates: dict[str, PropertyValue] # key -> PropertyValue
- set_once_updates: dict[str, PropertyValue] # key -> PropertyValue
- unset_updates: dict[str, PropertyValue] # key -> PropertyValue (value is always None)
-
-
-@dataclass
-class RawPersonPropertyUpdates:
- """Raw property updates from events, before comparison with person state.
-
- Used in windowed queries where we aggregate across multiple time windows
- before doing a single comparison against person properties.
- """
-
- person_id: str
- set_updates: dict[str, PropertyValue] # key -> PropertyValue
- set_once_updates: dict[str, PropertyValue] # key -> PropertyValue
- unset_updates: dict[str, PropertyValue] # key -> PropertyValue (value is always None)
-
-
-class SkipReason:
- """Reasons why a person update was skipped."""
-
- SUCCESS = "success" # Not skipped - successfully updated
- NO_CHANGES = "no_changes" # No changes needed after reconciliation
- NOT_FOUND = "not_found" # Person not found in Postgres
- VERSION_CONFLICT = "version_conflict" # Version mismatch after max retries
-
-
-# Properties that should NOT trigger a person update on their own.
-# These change frequently but aren't valuable enough to update the person record for.
-# Keep in sync with: nodejs/src/common/persons/person-property-utils.ts
-FILTERED_PERSON_UPDATE_PROPERTIES = frozenset(
- [
- # URL/navigation properties - change on every page view
- "$current_url",
- "$pathname",
- "$referring_domain",
- "$referrer",
- # Screen/viewport dimensions - can change on window resize
- "$screen_height",
- "$screen_width",
- "$viewport_height",
- "$viewport_width",
- # Browser/device properties - change less frequently but still filtered
- "$browser",
- "$browser_version",
- "$device_type",
- "$raw_user_agent",
- "$os",
- "$os_name",
- "$os_version",
- # GeoIP properties - filtered because they change frequently
- # Note: $geoip_country_name and $geoip_city_name DO trigger updates (not listed here)
- "$geoip_postal_code",
- "$geoip_time_zone",
- "$geoip_latitude",
- "$geoip_longitude",
- "$geoip_accuracy_radius",
- "$geoip_subdivision_1_code",
- "$geoip_subdivision_1_name",
- "$geoip_subdivision_2_code",
- "$geoip_subdivision_2_name",
- "$geoip_subdivision_3_code",
- "$geoip_subdivision_3_name",
- "$geoip_city_confidence",
- "$geoip_country_confidence",
- "$geoip_postal_code_confidence",
- "$geoip_subdivision_1_confidence",
- "$geoip_subdivision_2_confidence",
- ]
-)
-
-
-def ensure_utc_datetime(ts: datetime) -> datetime:
- """Ensure timestamp is a UTC-aware datetime."""
- if ts.tzinfo is None:
- return ts.replace(tzinfo=UTC)
- return ts
-
-
-def get_person_property_updates_from_clickhouse(
- team_id: int,
- bug_window_start: str,
-) -> list[PersonPropertyDiffs]:
- """
- Query ClickHouse to get property updates per person that differ from current state.
-
- This query:
- 1. Joins events with person_distinct_id_overrides to resolve merged persons
- 2. Extracts $set (argMax for latest), $set_once (argMin for first), and $unset properties
- 3. Compares with current person properties in ClickHouse
- 4. Returns only persons where properties differ, are missing, or need removal
-
- Returns:
- List of PersonPropertyDiffs, each containing 3 maps (set, set_once, unset) keyed by property key
- """
- query = """
- -- CTE 1: Get all overrides for the team (no filtering)
- WITH overrides AS (
- SELECT
- argMax(person_distinct_id_overrides.person_id, person_distinct_id_overrides.version) AS person_id,
- person_distinct_id_overrides.distinct_id AS distinct_id
- FROM person_distinct_id_overrides
- WHERE equals(person_distinct_id_overrides.team_id, %(team_id)s)
- GROUP BY person_distinct_id_overrides.distinct_id
- HAVING ifNull(equals(argMax(person_distinct_id_overrides.is_deleted, person_distinct_id_overrides.version), 0), 0)
- ),
- -- CTE 2: Extract properties from events, grouped by RESOLVED person_id (after overrides)
- -- This ensures argMax/argMin aggregates across ALL distinct_ids for the same person
- event_properties_raw AS (
- SELECT
- person_id,
- arrayMap(x -> x.1, arrayFilter(x -> x.4 = 'set', grouped_props)) AS set_keys,
- arrayMap(x -> x.2, arrayFilter(x -> x.4 = 'set', grouped_props)) AS set_values,
- arrayMap(x -> x.3, arrayFilter(x -> x.4 = 'set', grouped_props)) AS set_timestamps,
- arrayMap(x -> x.1, arrayFilter(x -> x.4 = 'set_once', grouped_props)) AS set_once_keys,
- arrayMap(x -> x.2, arrayFilter(x -> x.4 = 'set_once', grouped_props)) AS set_once_values,
- arrayMap(x -> x.3, arrayFilter(x -> x.4 = 'set_once', grouped_props)) AS set_once_timestamps,
- arrayMap(x -> x.1, arrayFilter(x -> x.4 = 'unset', grouped_props)) AS unset_keys,
- arrayMap(x -> x.3, arrayFilter(x -> x.4 = 'unset', grouped_props)) AS unset_timestamps
- FROM (
- SELECT
- person_id,
- groupArray(tuple(key, value, kv_timestamp, prop_type)) AS grouped_props
- FROM (
- SELECT
- -- Apply overrides to get resolved person_id
- if(notEmpty(o.distinct_id), o.person_id, e.person_id) AS person_id,
- kv_tuple.2 AS key,
- kv_tuple.1 AS prop_type,
- -- $set: newest event wins, $set_once: first event wins, $unset: newest event wins
- if(kv_tuple.1 = 'set',
- argMaxIf(kv_tuple.3, e.timestamp, kv_tuple.3 IS NOT NULL AND kv_tuple.3 != ''),
- argMinIf(kv_tuple.3, e.timestamp, kv_tuple.3 IS NOT NULL AND kv_tuple.3 != '')
- ) AS value,
- if(kv_tuple.1 = 'set_once', min(e.timestamp), max(e.timestamp)) AS kv_timestamp
- FROM events e
- LEFT JOIN overrides o ON e.distinct_id = o.distinct_id
- ARRAY JOIN
- arrayConcat(
- arrayFilter(x -> x.3 IS NOT NULL AND x.3 != '' AND x.3 != 'null' AND x.2 NOT IN %(filtered_properties)s,
- arrayMap(x -> tuple('set', x.1, toString(x.2)),
- arrayFilter(x -> x.2 IS NOT NULL, JSONExtractKeysAndValuesRaw(e.properties, '$set'))
- )
- ),
- arrayFilter(x -> x.3 IS NOT NULL AND x.3 != '' AND x.3 != 'null' AND x.2 NOT IN %(filtered_properties)s,
- arrayMap(x -> tuple('set_once', x.1, toString(x.2)),
- arrayFilter(x -> x.2 IS NOT NULL, JSONExtractKeysAndValuesRaw(e.properties, '$set_once'))
- )
- ),
- arrayFilter(x -> x.2 NOT IN %(filtered_properties)s,
- arrayMap(x -> tuple('unset', JSON_VALUE(x, '$'), ''),
- JSONExtractArrayRaw(e.properties, '$unset')
- )
- )
- ) AS kv_tuple
- WHERE e.team_id = %(team_id)s
- AND e.timestamp >= %(bug_window_start)s
- AND e.timestamp < now()
- AND (JSONExtractString(e.properties, '$set') != '' OR JSONExtractString(e.properties, '$set_once') != '' OR notEmpty(JSONExtractArrayRaw(e.properties, '$unset')))
- -- Group by resolved person_id (not distinct_id) so argMax works across all distinct_ids
- GROUP BY if(notEmpty(o.distinct_id), o.person_id, e.person_id), kv_tuple.2, kv_tuple.1
- )
- GROUP BY person_id
- )
- ),
- -- CTE 3: Filter to only persons with non-empty property sets
- event_properties_flat AS (
- SELECT *
- FROM event_properties_raw
- WHERE length(set_keys) > 0 OR length(set_once_keys) > 0 OR length(unset_keys) > 0
- ),
- -- CTE 4: Get person properties and version only for affected persons
- person_props AS (
- SELECT
- id,
- argMax(properties, version) as person_properties,
- argMax(version, version) as person_version
- FROM person
- WHERE team_id = %(team_id)s
- AND id IN (SELECT person_id FROM event_properties_flat)
- GROUP BY id
- HAVING argMax(is_deleted, version) = 0
- )
- SELECT
- ep.person_id,
- p.person_version,
- -- For $set: only include properties where the key exists in person properties AND the value differs
- arrayMap(i -> (ep.set_keys[i], ep.set_values[i], ep.set_timestamps[i]), arrayFilter(
- i -> (
- indexOf(keys2, ep.set_keys[i]) > 0
- AND ep.set_values[i] != vals2[indexOf(keys2, ep.set_keys[i])]
- ),
- arrayEnumerate(ep.set_keys)
- )) AS set_diff,
- -- For $set_once: only include properties where the key does NOT exist in person properties
- arrayFilter(
- kv -> indexOf(keys2, kv.1) = 0,
- arrayMap(i -> (ep.set_once_keys[i], ep.set_once_values[i], ep.set_once_timestamps[i]), arrayEnumerate(ep.set_once_keys))
- ) AS set_once_diff,
- -- For $unset: only include keys that EXIST in person properties (need removal)
- arrayFilter(
- kv -> indexOf(keys2, kv.1) > 0,
- arrayMap(i -> (ep.unset_keys[i], ep.unset_timestamps[i]), arrayEnumerate(ep.unset_keys))
- ) AS unset_diff
- FROM event_properties_flat ep
- INNER JOIN (
- SELECT
- id,
- person_version,
- person_properties,
- arrayMap(x -> x.1, JSONExtractKeysAndValuesRaw(person_properties)) AS keys2,
- arrayMap(x -> toString(x.2), JSONExtractKeysAndValuesRaw(person_properties)) AS vals2
- FROM person_props
- ) AS p ON p.id = ep.person_id
- WHERE length(set_diff) > 0 OR length(set_once_diff) > 0 OR length(unset_diff) > 0
- ORDER BY ep.person_id
- SETTINGS
- readonly=2,
- max_execution_time=1200,
- allow_experimental_object_type=1,
- format_csv_allow_double_quotes=0,
- max_ast_elements=4000000,
- max_expanded_ast_elements=4000000,
- transform_null_in=1,
- optimize_min_equality_disjunction_chain_length=4294967295,
- optimize_min_inequality_conjunction_chain_length=4294967295,
- allow_experimental_join_condition=1,
- use_hive_partitioning=0
- """
-
- params = {
- "team_id": team_id,
- "bug_window_start": bug_window_start,
- "filtered_properties": tuple(FILTERED_PERSON_UPDATE_PROPERTIES),
- }
-
- rows = sync_execute(query, params)
-
- results: list[PersonPropertyDiffs] = []
- for row in rows:
- person_id, person_version, set_diff, set_once_diff, unset_diff = row
-
- set_updates: dict[str, PropertyValue] = {}
- for key, value, timestamp in set_diff:
- set_updates[str(key)] = PropertyValue(
- timestamp=ensure_utc_datetime(timestamp),
- value=json.loads(value),
- )
-
- set_once_updates: dict[str, PropertyValue] = {}
- for key, value, timestamp in set_once_diff:
- set_once_updates[str(key)] = PropertyValue(
- timestamp=ensure_utc_datetime(timestamp),
- value=json.loads(value),
- )
-
- unset_updates: dict[str, PropertyValue] = {}
- for key, timestamp in unset_diff:
- unset_updates[str(key)] = PropertyValue(
- timestamp=ensure_utc_datetime(timestamp),
- value=None,
- )
-
- if set_updates or set_once_updates or unset_updates:
- results.append(
- PersonPropertyDiffs(
- person_id=str(person_id),
- person_version=int(person_version),
- set_updates=set_updates,
- set_once_updates=set_once_updates,
- unset_updates=unset_updates,
- )
- )
-
- return results
-
-
-def get_raw_person_property_updates_from_clickhouse(
- team_id: int,
- bug_window_start: str,
- bug_window_end: str,
- person_ids: tuple[str, ...],
-) -> list[RawPersonPropertyUpdates]:
- """
- Query ClickHouse to get raw property updates from events WITHOUT comparing to person state.
-
- This is used for windowed queries where we need to aggregate events across multiple
- time windows before doing a single comparison against person properties.
-
- Returns raw aggregated event data:
- - $set: argMax for latest value per key
- - $set_once: argMin for first value per key
- - $unset: max timestamp per key
-
- Args:
- team_id: Team ID to query
- bug_window_start: Start of time window (ClickHouse format: "YYYY-MM-DD HH:MM:SS")
- bug_window_end: End of time window (ClickHouse format: "YYYY-MM-DD HH:MM:SS")
- person_ids: Tuple of person_ids to filter to (required for batched processing)
-
- Returns:
- List of RawPersonPropertyUpdates with aggregated event data (no person comparison)
- """
- query = """
- -- CTE 1: Get all overrides for the team
- WITH overrides AS (
- SELECT
- argMax(person_distinct_id_overrides.person_id, person_distinct_id_overrides.version) AS person_id,
- person_distinct_id_overrides.distinct_id AS distinct_id
- FROM person_distinct_id_overrides
- WHERE equals(person_distinct_id_overrides.team_id, %(team_id)s)
- GROUP BY person_distinct_id_overrides.distinct_id
- HAVING ifNull(equals(argMax(person_distinct_id_overrides.is_deleted, person_distinct_id_overrides.version), 0), 0)
- ),
- -- CTE 2: Extract and aggregate properties from events
- event_properties_raw AS (
- SELECT
- person_id,
- arrayMap(x -> x.1, arrayFilter(x -> x.4 = 'set', grouped_props)) AS set_keys,
- arrayMap(x -> x.2, arrayFilter(x -> x.4 = 'set', grouped_props)) AS set_values,
- arrayMap(x -> x.3, arrayFilter(x -> x.4 = 'set', grouped_props)) AS set_timestamps,
- arrayMap(x -> x.1, arrayFilter(x -> x.4 = 'set_once', grouped_props)) AS set_once_keys,
- arrayMap(x -> x.2, arrayFilter(x -> x.4 = 'set_once', grouped_props)) AS set_once_values,
- arrayMap(x -> x.3, arrayFilter(x -> x.4 = 'set_once', grouped_props)) AS set_once_timestamps,
- arrayMap(x -> x.1, arrayFilter(x -> x.4 = 'unset', grouped_props)) AS unset_keys,
- arrayMap(x -> x.3, arrayFilter(x -> x.4 = 'unset', grouped_props)) AS unset_timestamps
- FROM (
- SELECT
- person_id,
- groupArray(tuple(key, value, kv_timestamp, prop_type)) AS grouped_props
- FROM (
- SELECT
- if(notEmpty(o.distinct_id), o.person_id, e.person_id) AS person_id,
- kv_tuple.2 AS key,
- kv_tuple.1 AS prop_type,
- if(kv_tuple.1 = 'set',
- argMaxIf(kv_tuple.3, e.timestamp, kv_tuple.3 IS NOT NULL AND kv_tuple.3 != ''),
- argMinIf(kv_tuple.3, e.timestamp, kv_tuple.3 IS NOT NULL AND kv_tuple.3 != '')
- ) AS value,
- if(kv_tuple.1 = 'set_once', min(e.timestamp), max(e.timestamp)) AS kv_timestamp
- FROM events e
- LEFT JOIN overrides o ON e.distinct_id = o.distinct_id
- ARRAY JOIN
- arrayConcat(
- arrayFilter(x -> x.3 IS NOT NULL AND x.3 != '' AND x.3 != 'null' AND x.2 NOT IN %(filtered_properties)s,
- arrayMap(x -> tuple('set', x.1, toString(x.2)),
- arrayFilter(x -> x.2 IS NOT NULL, JSONExtractKeysAndValuesRaw(e.properties, '$set'))
- )
- ),
- arrayFilter(x -> x.3 IS NOT NULL AND x.3 != '' AND x.3 != 'null' AND x.2 NOT IN %(filtered_properties)s,
- arrayMap(x -> tuple('set_once', x.1, toString(x.2)),
- arrayFilter(x -> x.2 IS NOT NULL, JSONExtractKeysAndValuesRaw(e.properties, '$set_once'))
- )
- ),
- arrayFilter(x -> x.2 NOT IN %(filtered_properties)s,
- arrayMap(x -> tuple('unset', JSON_VALUE(x, '$'), ''),
- JSONExtractArrayRaw(e.properties, '$unset')
- )
- )
- ) AS kv_tuple
- WHERE e.team_id = %(team_id)s
- AND e.timestamp >= %(bug_window_start)s
- AND e.timestamp < %(bug_window_end)s
- AND (JSONExtractString(e.properties, '$set') != '' OR JSONExtractString(e.properties, '$set_once') != '' OR notEmpty(JSONExtractArrayRaw(e.properties, '$unset')))
- AND if(notEmpty(o.distinct_id), o.person_id, e.person_id) IN %(person_ids)s
- GROUP BY if(notEmpty(o.distinct_id), o.person_id, e.person_id), kv_tuple.2, kv_tuple.1
- )
- GROUP BY person_id
- )
- )
- -- Return raw aggregated event data (no person comparison)
- SELECT
- person_id,
- arrayMap(i -> (set_keys[i], set_values[i], set_timestamps[i]), arrayEnumerate(set_keys)) AS set_data,
- arrayMap(i -> (set_once_keys[i], set_once_values[i], set_once_timestamps[i]), arrayEnumerate(set_once_keys)) AS set_once_data,
- arrayMap(i -> (unset_keys[i], unset_timestamps[i]), arrayEnumerate(unset_keys)) AS unset_data
- FROM event_properties_raw
- WHERE length(set_keys) > 0 OR length(set_once_keys) > 0 OR length(unset_keys) > 0
- ORDER BY person_id
- SETTINGS
- readonly=2,
- max_execution_time=600,
- allow_experimental_object_type=1,
- format_csv_allow_double_quotes=0,
- max_ast_elements=4000000,
- max_expanded_ast_elements=4000000,
- transform_null_in=1,
- optimize_min_equality_disjunction_chain_length=4294967295,
- optimize_min_inequality_conjunction_chain_length=4294967295,
- allow_experimental_join_condition=1,
- use_hive_partitioning=0
- """
-
- params = {
- "team_id": team_id,
- "bug_window_start": bug_window_start,
- "bug_window_end": bug_window_end,
- "filtered_properties": tuple(FILTERED_PERSON_UPDATE_PROPERTIES),
- "person_ids": person_ids,
- }
-
- rows = sync_execute(query, params)
-
- results: list[RawPersonPropertyUpdates] = []
- for row in rows:
- person_id, set_data, set_once_data, unset_data = row
-
- set_updates: dict[str, PropertyValue] = {}
- for key, value, timestamp in set_data:
- set_updates[str(key)] = PropertyValue(
- timestamp=ensure_utc_datetime(timestamp),
- value=json.loads(value),
- )
-
- set_once_updates: dict[str, PropertyValue] = {}
- for key, value, timestamp in set_once_data:
- set_once_updates[str(key)] = PropertyValue(
- timestamp=ensure_utc_datetime(timestamp),
- value=json.loads(value),
- )
-
- unset_updates: dict[str, PropertyValue] = {}
- for key, timestamp in unset_data:
- unset_updates[str(key)] = PropertyValue(
- timestamp=ensure_utc_datetime(timestamp),
- value=None,
- )
-
- if set_updates or set_once_updates or unset_updates:
- results.append(
- RawPersonPropertyUpdates(
- person_id=str(person_id),
- set_updates=set_updates,
- set_once_updates=set_once_updates,
- unset_updates=unset_updates,
- )
- )
-
- return results
-
-
-def get_affected_person_ids_from_clickhouse(
- team_id: int,
- bug_window_start: str,
- bug_window_end: str,
-) -> list[str]:
- """
- Query ClickHouse for person_ids whose events in the bug window produce property
- diffs when compared against current person state.
-
- Uses the same aggregation + person-state comparison structure as the
- non-windowed query (get_person_property_updates_from_clickhouse) but:
- - Bounded by bug_window_end (not now()) to keep the scan lightweight
- - Returns only person_ids (no full diff data)
-
- This means only persons whose bug-window event aggregation actually differs
- from their current person properties are returned, dramatically reducing the
- set compared to a simple event-presence check.
-
- Possible false positives (acceptable): a person whose bug-window aggregation
- shows a diff that disappears when events up to now() are included. These are
- filtered out by the downstream windowed comparison step.
-
- Args:
- team_id: Team ID to query
- bug_window_start: Start of bug window (ClickHouse format: "YYYY-MM-DD HH:MM:SS")
- bug_window_end: End of bug window (ClickHouse format: "YYYY-MM-DD HH:MM:SS")
-
- Returns:
- List of person_id strings (UUIDs) affected in the bug window
- """
- query = """
- -- CTE 1: Get all overrides for the team (no filtering)
- WITH overrides AS (
- SELECT
- argMax(person_distinct_id_overrides.person_id, person_distinct_id_overrides.version) AS person_id,
- person_distinct_id_overrides.distinct_id AS distinct_id
- FROM person_distinct_id_overrides
- WHERE equals(person_distinct_id_overrides.team_id, %(team_id)s)
- GROUP BY person_distinct_id_overrides.distinct_id
- HAVING ifNull(equals(argMax(person_distinct_id_overrides.is_deleted, person_distinct_id_overrides.version), 0), 0)
- ),
- -- CTE 2: Extract properties from events, grouped by RESOLVED person_id (after overrides)
- event_properties_raw AS (
- SELECT
- person_id,
- arrayMap(x -> x.1, arrayFilter(x -> x.4 = 'set', grouped_props)) AS set_keys,
- arrayMap(x -> x.2, arrayFilter(x -> x.4 = 'set', grouped_props)) AS set_values,
- arrayMap(x -> x.1, arrayFilter(x -> x.4 = 'set_once', grouped_props)) AS set_once_keys,
- arrayMap(x -> x.1, arrayFilter(x -> x.4 = 'unset', grouped_props)) AS unset_keys
- FROM (
- SELECT
- person_id,
- groupArray(tuple(key, value, kv_timestamp, prop_type)) AS grouped_props
- FROM (
- SELECT
- if(notEmpty(o.distinct_id), o.person_id, e.person_id) AS person_id,
- kv_tuple.2 AS key,
- kv_tuple.1 AS prop_type,
- if(kv_tuple.1 = 'set',
- argMaxIf(kv_tuple.3, e.timestamp, kv_tuple.3 IS NOT NULL AND kv_tuple.3 != ''),
- argMinIf(kv_tuple.3, e.timestamp, kv_tuple.3 IS NOT NULL AND kv_tuple.3 != '')
- ) AS value,
- if(kv_tuple.1 = 'set_once', min(e.timestamp), max(e.timestamp)) AS kv_timestamp
- FROM events e
- LEFT JOIN overrides o ON e.distinct_id = o.distinct_id
- ARRAY JOIN
- arrayConcat(
- arrayFilter(x -> x.3 IS NOT NULL AND x.3 != '' AND x.3 != 'null' AND x.2 NOT IN %(filtered_properties)s,
- arrayMap(x -> tuple('set', x.1, toString(x.2)),
- arrayFilter(x -> x.2 IS NOT NULL, JSONExtractKeysAndValuesRaw(e.properties, '$set'))
- )
- ),
- arrayFilter(x -> x.3 IS NOT NULL AND x.3 != '' AND x.3 != 'null' AND x.2 NOT IN %(filtered_properties)s,
- arrayMap(x -> tuple('set_once', x.1, toString(x.2)),
- arrayFilter(x -> x.2 IS NOT NULL, JSONExtractKeysAndValuesRaw(e.properties, '$set_once'))
- )
- ),
- arrayFilter(x -> x.2 NOT IN %(filtered_properties)s,
- arrayMap(x -> tuple('unset', JSON_VALUE(x, '$'), ''),
- JSONExtractArrayRaw(e.properties, '$unset')
- )
- )
- ) AS kv_tuple
- WHERE e.team_id = %(team_id)s
- AND e.timestamp >= %(bug_window_start)s
- AND e.timestamp < %(bug_window_end)s
- AND (JSONExtractString(e.properties, '$set') != '' OR JSONExtractString(e.properties, '$set_once') != '' OR notEmpty(JSONExtractArrayRaw(e.properties, '$unset')))
- GROUP BY if(notEmpty(o.distinct_id), o.person_id, e.person_id), kv_tuple.2, kv_tuple.1
- )
- GROUP BY person_id
- )
- ),
- -- CTE 3: Filter to only persons with non-empty property sets
- event_properties_flat AS (
- SELECT *
- FROM event_properties_raw
- WHERE length(set_keys) > 0 OR length(set_once_keys) > 0 OR length(unset_keys) > 0
- ),
- -- CTE 4: Get person properties only for affected persons
- person_props AS (
- SELECT
- id,
- argMax(properties, version) as person_properties
- FROM person
- WHERE team_id = %(team_id)s
- AND id IN (SELECT person_id FROM event_properties_flat)
- GROUP BY id
- HAVING argMax(is_deleted, version) = 0
- )
- -- Return only person_ids that have actual diffs against current state
- SELECT ep.person_id
- FROM event_properties_flat ep
- INNER JOIN (
- SELECT
- id,
- person_properties,
- arrayMap(x -> x.1, JSONExtractKeysAndValuesRaw(person_properties)) AS keys2,
- arrayMap(x -> toString(x.2), JSONExtractKeysAndValuesRaw(person_properties)) AS vals2
- FROM person_props
- ) AS p ON p.id = ep.person_id
- WHERE
- -- $set: key exists in person AND value differs
- length(arrayFilter(
- i -> (
- indexOf(p.keys2, ep.set_keys[i]) > 0
- AND ep.set_values[i] != p.vals2[indexOf(p.keys2, ep.set_keys[i])]
- ),
- arrayEnumerate(ep.set_keys)
- )) > 0
- -- $set_once: key does NOT exist in person
- OR length(arrayFilter(
- i -> indexOf(p.keys2, ep.set_once_keys[i]) = 0,
- arrayEnumerate(ep.set_once_keys)
- )) > 0
- -- $unset: key EXISTS in person
- OR length(arrayFilter(
- i -> indexOf(p.keys2, ep.unset_keys[i]) > 0,
- arrayEnumerate(ep.unset_keys)
- )) > 0
- ORDER BY ep.person_id
- SETTINGS
- readonly=2,
- max_execution_time=1200,
- allow_experimental_object_type=1,
- format_csv_allow_double_quotes=0,
- max_ast_elements=4000000,
- max_expanded_ast_elements=4000000,
- transform_null_in=1,
- optimize_min_equality_disjunction_chain_length=4294967295,
- optimize_min_inequality_conjunction_chain_length=4294967295,
- allow_experimental_join_condition=1,
- use_hive_partitioning=0
- """
-
- params = {
- "team_id": team_id,
- "bug_window_start": bug_window_start,
- "bug_window_end": bug_window_end,
- "filtered_properties": tuple(FILTERED_PERSON_UPDATE_PROPERTIES),
- }
-
- rows = sync_execute(query, params)
- return [str(row[0]) for row in rows]
-
-
-# Default batch size for processing persons in windowed mode.
-# Keeps IN clauses under ClickHouse's max_query_size limit.
-# 10K UUIDs × ~40 chars ≈ 400KB query text.
-WINDOWED_PERSON_BATCH_SIZE = 10000
-
-
-def merge_raw_person_property_updates(
- accumulated: dict[str, RawPersonPropertyUpdates],
- new_updates: list[RawPersonPropertyUpdates],
-) -> None:
- """
- Merge new raw updates into accumulated dict, keyed by person_id.
-
- This is used when iterating through time windows to accumulate raw property updates
- across multiple query results BEFORE comparing with person state.
-
- Merge rules:
- - For $set: newer timestamp replaces older
- - For $set_once: earlier timestamp wins (first-writer-wins semantics)
- - For $unset: newer timestamp replaces older
-
- Cross-map conflicts (set/set_once vs unset) are resolved downstream by
- filter_event_person_properties after comparison with person state.
-
- Args:
- accumulated: Dict of person_id -> RawPersonPropertyUpdates to merge into (mutated in place)
- new_updates: List of new RawPersonPropertyUpdates from the latest window query
- """
- for new_update in new_updates:
- person_id = new_update.person_id
-
- if person_id not in accumulated:
- accumulated[person_id] = RawPersonPropertyUpdates(
- person_id=person_id,
- set_updates=dict(new_update.set_updates),
- set_once_updates=dict(new_update.set_once_updates),
- unset_updates=dict(new_update.unset_updates),
- )
- continue
-
- existing = accumulated[person_id]
-
- # Process $set updates - newer timestamp wins
- for key, new_pv in new_update.set_updates.items():
- if key in existing.set_updates:
- if new_pv.timestamp > existing.set_updates[key].timestamp:
- existing.set_updates[key] = new_pv
- else:
- existing.set_updates[key] = new_pv
-
- # Process $set_once updates - earlier timestamp wins
- for key, new_pv in new_update.set_once_updates.items():
- if key in existing.set_once_updates:
- if new_pv.timestamp < existing.set_once_updates[key].timestamp:
- existing.set_once_updates[key] = new_pv
- else:
- existing.set_once_updates[key] = new_pv
-
- # Process $unset updates - newer timestamp wins
- for key, new_pv in new_update.unset_updates.items():
- if key in existing.unset_updates:
- if new_pv.timestamp > existing.unset_updates[key].timestamp:
- existing.unset_updates[key] = new_pv
- else:
- existing.unset_updates[key] = new_pv
-
-
-# Max number of person_ids per batch when querying person state.
-# Prevents "Query size exceeded" errors when IN clause becomes too large.
-# 10K UUIDs × ~40 chars ≈ 400KB query text, under typical max_query_size limits.
-PERSON_STATE_BATCH_SIZE = 10000
-
-
-def compare_raw_updates_with_person_state(
- team_id: int,
- raw_updates: list[RawPersonPropertyUpdates],
-) -> list[PersonPropertyDiffs]:
- """
- Compare merged raw event updates against current person state in ClickHouse.
-
- This is the second step of the windowed query flow:
- 1. get_raw_person_property_updates_from_clickhouse() per window
- 2. merge_raw_person_property_updates() to combine windows
- 3. compare_raw_updates_with_person_state() to filter to actual diffs
-
- Comparison rules (matching the original single-query behavior):
- - $set: only include if key EXISTS in person AND value DIFFERS
- - $set_once: only include if key does NOT exist in person
- - $unset: only include if key EXISTS in person
-
- Note on comparison semantics:
- This function uses Python object comparison, so semantically equal values
- like 123 (int) and 123.0 (float) are considered equal. This differs from
- the non-windowed SQL path (get_person_property_updates_from_clickhouse)
- which compares raw JSON string representations. The Python approach is
- preferable for reconciliation as it avoids unnecessary updates when
- values are semantically equivalent.
-
- Note on batching:
- For high-volume teams with many affected persons, we batch the person
- state queries to avoid "Query size exceeded" errors from overly large
- IN clauses. Each batch queries up to PERSON_STATE_BATCH_SIZE persons.
-
- Args:
- team_id: Team ID
- raw_updates: Merged raw updates from all time windows
-
- Returns:
- List of PersonPropertyDiffs with only actual differences
- """
- if not raw_updates:
- return []
-
- # Extract person_ids for batching
- person_ids = [u.person_id for u in raw_updates]
-
- # Query person properties and versions in batches to avoid "Query size exceeded"
- person_data: dict[str, tuple[dict, int]] = {}
-
- for batch_start in range(0, len(person_ids), PERSON_STATE_BATCH_SIZE):
- batch_person_ids = person_ids[batch_start : batch_start + PERSON_STATE_BATCH_SIZE]
-
- query = """
- SELECT
- id,
- argMax(properties, version) as properties,
- argMax(version, version) as person_version
- FROM person
- WHERE team_id = %(team_id)s
- AND id IN %(person_ids)s
- GROUP BY id
- HAVING argMax(is_deleted, version) = 0
- """
-
- params = {
- "team_id": team_id,
- "person_ids": tuple(batch_person_ids),
- }
-
- rows = sync_execute(query, params)
-
- for row in rows:
- person_id, properties_str, version = row
- if properties_str:
- properties = json.loads(properties_str) if isinstance(properties_str, str) else properties_str
- else:
- properties = {}
- person_data[str(person_id)] = (properties, int(version))
-
- results: list[PersonPropertyDiffs] = []
-
- for raw_update in raw_updates:
- if raw_update.person_id not in person_data:
- continue
-
- person_properties, person_version = person_data[raw_update.person_id]
- person_keys = set(person_properties.keys())
-
- # Filter $set: key must exist in person AND value must differ
- filtered_set: dict[str, PropertyValue] = {}
- for key, pv in raw_update.set_updates.items():
- if key in person_keys:
- person_val = person_properties[key]
- if pv.value != person_val:
- filtered_set[key] = pv
-
- # Filter $set_once: key must NOT exist in person
- filtered_set_once: dict[str, PropertyValue] = {}
- for key, pv in raw_update.set_once_updates.items():
- if key not in person_keys:
- filtered_set_once[key] = pv
-
- # Filter $unset: key must exist in person
- filtered_unset: dict[str, PropertyValue] = {}
- for key, pv in raw_update.unset_updates.items():
- if key in person_keys:
- filtered_unset[key] = pv
-
- if filtered_set or filtered_set_once or filtered_unset:
- results.append(
- PersonPropertyDiffs(
- person_id=raw_update.person_id,
- person_version=person_version,
- set_updates=filtered_set,
- set_once_updates=filtered_set_once,
- unset_updates=filtered_unset,
- )
- )
-
- return results
-
-
-def parse_ch_timestamp(ts: str) -> datetime:
- """Parse a ClickHouse timestamp string to a UTC datetime."""
- return datetime.strptime(ts, "%Y-%m-%d %H:%M:%S").replace(tzinfo=UTC)
-
-
-def format_ch_timestamp(dt: datetime) -> str:
- """Format a datetime to ClickHouse timestamp string (UTC)."""
- if dt.tzinfo is None:
- # Assume naive datetimes are already UTC
- return dt.strftime("%Y-%m-%d %H:%M:%S")
- # Convert to UTC before formatting
- utc_dt = dt.astimezone(UTC)
- return utc_dt.strftime("%Y-%m-%d %H:%M:%S")
-
-
-def get_person_property_updates_windowed(
- team_id: int,
- bug_window_start: str,
- window_seconds: int,
- bug_window_end: str | None = None,
- logger: Any = None,
-) -> list[PersonPropertyDiffs]:
- """
- Fetch person property updates, optionally in time windows.
-
- When window_seconds > 0, this function uses a two-step flow:
- 1. Query affected person_ids first (returns early if none)
- 2. Query raw event aggregates per window (no person comparison)
- 3. Merge raw updates across windows with timestamp-based deduplication
- 4. Compare final merged updates against current person state
-
- When window_seconds <= 0, uses the original single-query approach.
-
- Args:
- team_id: Team ID to query
- bug_window_start: Start of time window (ClickHouse format: "YYYY-MM-DD HH:MM:SS")
- window_seconds: Size of each query window in seconds. If <= 0, single query is used.
- bug_window_end: End of time window (defaults to now if not provided)
- logger: Optional logger for progress tracking
-
- Returns:
- List of PersonPropertyDiffs with merged results across all windows
- """
- if window_seconds <= 0:
- if logger:
- logger.info(f"Using single-query mode for team_id={team_id}")
- return get_person_property_updates_from_clickhouse(team_id, bug_window_start)
-
- now = datetime.now(UTC)
- effective_end = bug_window_end or format_ch_timestamp(now)
-
- # Step 1: Get affected person_ids first (early exit if none)
- affected_person_ids = get_affected_person_ids_from_clickhouse(
- team_id=team_id,
- bug_window_start=bug_window_start,
- bug_window_end=effective_end,
- )
-
- if logger:
- logger.info(f"Found {len(affected_person_ids)} affected persons for team_id={team_id}")
-
- if not affected_person_ids:
- return []
-
- person_ids_tuple = tuple(affected_person_ids)
-
- # Step 2: Query raw updates per window and merge
- accumulated: dict[str, RawPersonPropertyUpdates] = {}
- current_start = parse_ch_timestamp(bug_window_start)
- window_count = 0
- total_windows = int((now - current_start).total_seconds() / window_seconds) + 1
-
- if logger:
- logger.info(f"Starting windowed query: team_id={team_id}, ~{total_windows} windows of {window_seconds}s each")
-
- while current_start < now:
- current_end = min(current_start + timedelta(seconds=window_seconds), now)
- window_updates = get_raw_person_property_updates_from_clickhouse(
- team_id,
- format_ch_timestamp(current_start),
- format_ch_timestamp(current_end),
- person_ids=person_ids_tuple,
- )
- merge_raw_person_property_updates(accumulated, window_updates)
- window_count += 1
-
- if logger and window_count % 10 == 0:
- logger.info(
- f"Processed window {window_count}/{total_windows} for team_id={team_id}, "
- f"accumulated {len(accumulated)} unique persons"
- )
-
- current_start = current_end
-
- if logger:
- logger.info(
- f"Completed all {window_count} windows for team_id={team_id}, total unique persons: {len(accumulated)}"
- )
-
- # Step 3: Compare merged raw updates against person state
- if logger:
- logger.info(f"Comparing {len(accumulated)} persons against current state for team_id={team_id}")
-
- return compare_raw_updates_with_person_state(team_id, list(accumulated.values()))
-
-
-def get_person_property_updates_windowed_batched(
- team_id: int,
- bug_window_start: str,
- bug_window_end: str,
- window_seconds: int,
- person_batch_size: int = WINDOWED_PERSON_BATCH_SIZE,
- logger: Any = None,
-) -> Iterator[list[PersonPropertyDiffs]]:
- """
- Yield batches of PersonPropertyDiffs, processing persons in chunks.
-
- This is the memory-efficient version of get_person_property_updates_windowed()
- that processes persons in batches to prevent OOM on large teams.
-
- Flow:
- 1. Get affected person_ids (bounded by bug_window_end)
- 2. For each batch of person_ids:
- a. For each time window [start → now()]:
- - Query raw events filtered to this batch's person_ids
- - Merge into batch accumulator
- b. Compare batch with person state
- c. Yield batch results (allows caller to process and clear memory)
-
- Args:
- team_id: Team ID to query
- bug_window_start: Start of bug window (ClickHouse format: "YYYY-MM-DD HH:MM:SS")
- bug_window_end: End of bug window - used to bound affected persons
- window_seconds: Size of each query window in seconds
- person_batch_size: Number of persons per batch (default 10K)
- logger: Optional logger for progress tracking
-
- Yields:
- List of PersonPropertyDiffs for each batch of persons
- """
- # Step 1: Get affected person_ids (bounded by bug_window_end)
- if logger:
- logger.info(f"Querying affected persons: team_id={team_id}, bug_window=[{bug_window_start}, {bug_window_end}]")
-
- try:
- affected_person_ids = get_affected_person_ids_from_clickhouse(
- team_id=team_id,
- bug_window_start=bug_window_start,
- bug_window_end=bug_window_end,
- )
- except Exception:
- if logger:
- logger.exception(
- f"Failed to query affected persons for team_id={team_id}, "
- f"bug_window=[{bug_window_start}, {bug_window_end}]"
- )
- raise
-
- if logger:
- logger.info(f"Found {len(affected_person_ids)} affected persons for team_id={team_id}")
-
- if not affected_person_ids:
- return
-
- # Calculate total batches and windows for logging
- total_batches = (len(affected_person_ids) + person_batch_size - 1) // person_batch_size
- now = datetime.now(UTC)
- current_start_for_count = parse_ch_timestamp(bug_window_start)
- total_windows = int((now - current_start_for_count).total_seconds() / window_seconds) + 1
-
- if logger:
- logger.info(
- f"Starting batched windowed processing: team_id={team_id}, "
- f"{total_batches} batches of ~{person_batch_size} persons, "
- f"~{total_windows} windows of {window_seconds}s each"
- )
-
- # Step 2: Process each batch of persons
- for batch_idx in range(0, len(affected_person_ids), person_batch_size):
- batch_person_ids = affected_person_ids[batch_idx : batch_idx + person_batch_size]
- batch_num = (batch_idx // person_batch_size) + 1
-
- if logger:
- logger.info(
- f"Processing batch {batch_num}/{total_batches} for team_id={team_id}, {len(batch_person_ids)} persons"
- )
-
- # Step 2a: Query raw updates per window for this batch and merge
- accumulated: dict[str, RawPersonPropertyUpdates] = {}
- current_start = parse_ch_timestamp(bug_window_start)
- window_count = 0
-
- try:
- while current_start < now:
- current_end = min(current_start + timedelta(seconds=window_seconds), now)
- try:
- window_updates = get_raw_person_property_updates_from_clickhouse(
- team_id,
- format_ch_timestamp(current_start),
- format_ch_timestamp(current_end),
- person_ids=tuple(batch_person_ids),
- )
- except Exception:
- if logger:
- logger.exception(
- f"Batch {batch_num}: failed to query window {window_count + 1}/{total_windows} "
- f"[{format_ch_timestamp(current_start)}, {format_ch_timestamp(current_end)}) "
- f"for team_id={team_id}"
- )
- raise
-
- merge_raw_person_property_updates(accumulated, window_updates)
- window_count += 1
-
- if logger and window_count % 10 == 0:
- logger.info(
- f"Batch {batch_num}: processed window {window_count}/{total_windows}, "
- f"accumulated {len(accumulated)} persons"
- )
-
- current_start = current_end
-
- except Exception:
- if logger:
- logger.exception(
- f"Batch {batch_num}: failed during windowed event collection for team_id={team_id} "
- f"after {window_count} windows, accumulated {len(accumulated)} persons"
- )
- raise
-
- if logger:
- logger.info(
- f"Batch {batch_num}: completed {window_count} windows, total persons with updates: {len(accumulated)}"
- )
-
- # Step 2b: Compare batch with person state
- if accumulated:
- if logger:
- logger.info(f"Batch {batch_num}: comparing {len(accumulated)} persons against current state")
-
- try:
- batch_diffs = compare_raw_updates_with_person_state(team_id, list(accumulated.values()))
- except Exception:
- if logger:
- logger.exception(
- f"Batch {batch_num}: failed to compare {len(accumulated)} persons "
- f"against current state for team_id={team_id}"
- )
- raise
-
- if batch_diffs:
- if logger:
- logger.info(f"Batch {batch_num}: yielding {len(batch_diffs)} persons with diffs")
- # Step 2c: Yield batch results
- yield batch_diffs
- else:
- if logger:
- logger.info(f"Batch {batch_num}: no diffs found after comparison")
- else:
- if logger:
- logger.info(f"Batch {batch_num}: no updates found in time windows")
-
- if logger:
- logger.info(f"Completed all {total_batches} batches for team_id={team_id}")
-
-
-def filter_event_person_properties(
- person_diffs: list[PersonPropertyDiffs],
-) -> list[PersonPropertyDiffs]:
- """
- Filter conflicting set/unset operations by timestamp.
-
- For each key that appears in both set_updates and unset_updates:
- - If set timestamp > unset timestamp: keep set, remove unset
- - If unset timestamp >= set timestamp: keep unset, remove set
-
- Returns filtered PersonPropertyDiffs with conflicts resolved.
- """
- results: list[PersonPropertyDiffs] = []
-
- for diffs in person_diffs:
- # Copy the maps so we don't mutate the input
- set_updates = dict(diffs.set_updates)
- set_once_updates = dict(diffs.set_once_updates)
- unset_updates = dict(diffs.unset_updates)
-
- # Find keys that exist in both maps
- # Use original unset keys for iteration so deletions in first loop don't affect second loop
- original_unset_keys = list(diffs.unset_updates.keys())
-
- for key in original_unset_keys:
- if key in set_updates:
- set_ts = set_updates[key].timestamp
- unset_ts = diffs.unset_updates[key].timestamp
-
- if set_ts > unset_ts:
- # set is more recent - remove unset
- del unset_updates[key]
- else:
- # unset is more recent or equal - remove set
- del set_updates[key]
-
- # Filter set_once vs unset conflicts
- for key in original_unset_keys:
- if key in set_once_updates:
- set_once_ts = set_once_updates[key].timestamp
- unset_ts = diffs.unset_updates[key].timestamp
-
- if set_once_ts > unset_ts:
- # Only delete if not already removed by set loop
- if key in unset_updates:
- del unset_updates[key]
- else:
- del set_once_updates[key]
-
- results.append(
- PersonPropertyDiffs(
- person_id=diffs.person_id,
- person_version=diffs.person_version,
- set_updates=set_updates,
- set_once_updates=set_once_updates,
- unset_updates=unset_updates,
- )
- )
-
- return results
-
-
-def reconcile_person_properties(
- person: dict,
- person_property_diffs: PersonPropertyDiffs,
-) -> dict | None:
- """
- Compute updated properties by comparing ClickHouse updates with current Postgres state.
-
- The CH query pre-filters to only return:
- - $set properties where the CH value differs from current person state
- - $set_once properties that are missing from current person state
- - $unset properties that exist in current person state
-
- Args:
- person: From Postgres with uuid, properties, properties_last_updated_at, properties_last_operation
- person_property_diffs: Property diffs from ClickHouse
-
- Returns:
- Dict with updated properties/metadata if changes needed, None otherwise.
- """
- properties = dict(person["properties"] or {})
- properties_last_updated_at = dict(person["properties_last_updated_at"] or {})
- properties_last_operation = dict(person["properties_last_operation"] or {})
-
- changed = False
-
- # 1. set_once: only update if key not in properties
- for key, pv in person_property_diffs.set_once_updates.items():
- if key not in properties:
- properties[key] = pv.value
- properties_last_updated_at[key] = pv.timestamp.isoformat()
- properties_last_operation[key] = "set_once"
- changed = True
-
- # 2. set: always update
- for key, pv in person_property_diffs.set_updates.items():
- properties[key] = pv.value
- properties_last_updated_at[key] = pv.timestamp.isoformat()
- properties_last_operation[key] = "set"
- changed = True
-
- # 3. unset: delete from all maps
- for key in person_property_diffs.unset_updates.keys():
- if key in properties:
- del properties[key]
- if key in properties_last_updated_at:
- del properties_last_updated_at[key]
- if key in properties_last_operation:
- del properties_last_operation[key]
- changed = True
-
- if changed:
- return {
- "properties": properties,
- "properties_last_updated_at": properties_last_updated_at,
- "properties_last_operation": properties_last_operation,
- }
- return None
-
-
-def fetch_person_properties_from_clickhouse(team_id: int, person_uuid: str, min_version: int) -> dict | None:
- """
- Fetch person properties from ClickHouse for conflict resolution.
-
- Fetches the oldest available version >= min_version. This handles the case where
- the exact version we computed diffs against may have been merged away by
- ReplacingMergeTree background merges.
-
- Returns the properties at the oldest available version >= min_version, or None if not found.
- """
- query = """
- SELECT argMin(properties, version) as properties
- FROM person
- WHERE team_id = %(team_id)s AND id = %(person_id)s AND version >= %(min_version)s
- GROUP BY id
- """
- rows = sync_execute(query, {"team_id": team_id, "person_id": person_uuid, "min_version": min_version})
- if not rows:
- return None
- properties_str = rows[0][0]
- if not properties_str:
- return {}
- return json.loads(properties_str) if isinstance(properties_str, str) else properties_str
-
-
-def reconcile_with_concurrent_changes(
- ch_properties: dict,
- postgres_person: dict,
- person_property_diffs: PersonPropertyDiffs,
-) -> dict | None:
- """
- 3-way merge: apply event changes while respecting concurrent Postgres changes.
-
- - Base: ch_properties (the state when event diffs were computed)
- - Theirs: postgres_person properties (current state, may have concurrent changes)
- - Ours: event_diffs (changes from events)
-
- Conflict resolution: Postgres wins (concurrent changes take precedence).
- This is conservative - we don't overwrite changes made by other processes.
- """
- postgres_properties = dict(postgres_person["properties"] or {})
- properties_last_updated_at = dict(postgres_person["properties_last_updated_at"] or {})
- properties_last_operation = dict(postgres_person["properties_last_operation"] or {})
-
- # Identify what changed in Postgres since CH (concurrent changes)
- concurrent_changed_keys: set[str] = set()
- all_keys = set(ch_properties.keys()) | set(postgres_properties.keys())
- for key in all_keys:
- ch_val = ch_properties.get(key)
- pg_val = postgres_properties.get(key)
- if ch_val != pg_val:
- concurrent_changed_keys.add(key)
-
- changed = False
-
- # 1. set_once: only update if key not in properties AND not concurrently changed
- for key, pv in person_property_diffs.set_once_updates.items():
- if key not in postgres_properties and key not in concurrent_changed_keys:
- postgres_properties[key] = pv.value
- properties_last_updated_at[key] = pv.timestamp.isoformat()
- properties_last_operation[key] = "set_once"
- changed = True
-
- # 2. set: only update if not concurrently changed
- for key, pv in person_property_diffs.set_updates.items():
- if key not in concurrent_changed_keys:
- postgres_properties[key] = pv.value
- properties_last_updated_at[key] = pv.timestamp.isoformat()
- properties_last_operation[key] = "set"
- changed = True
-
- # 3. unset: only delete if not concurrently changed
- for key in person_property_diffs.unset_updates.keys():
- if key in postgres_properties and key not in concurrent_changed_keys:
- del postgres_properties[key]
- if key in properties_last_updated_at:
- del properties_last_updated_at[key]
- if key in properties_last_operation:
- del properties_last_operation[key]
- changed = True
-
- if changed:
- return {
- "properties": postgres_properties,
- "properties_last_updated_at": properties_last_updated_at,
- "properties_last_operation": properties_last_operation,
- }
- return None
-
-
-def publish_person_to_kafka(person_data: dict, producer: _KafkaProducer) -> None:
- """
- Publish a person update to the Kafka topic for ClickHouse ingestion.
-
- Args:
- person_data: Dict with id, team_id, properties, is_identified, is_deleted, created_at, version
- producer: Kafka producer instance (injected resource)
- """
- from django.utils.timezone import now
-
- # Format data for Kafka/ClickHouse
- created_at = person_data.get("created_at")
- if created_at is not None and hasattr(created_at, "strftime"):
- created_at_str = created_at.strftime("%Y-%m-%d %H:%M:%S.%f")
- else:
- created_at_str = str(created_at) if created_at else now().strftime("%Y-%m-%d %H:%M:%S.%f")
-
- kafka_data = {
- "id": str(person_data["id"]),
- "team_id": person_data["team_id"],
- "properties": json.dumps(person_data["properties"]),
- "is_identified": int(person_data.get("is_identified", False)),
- "is_deleted": int(person_data.get("is_deleted", 0)),
- "created_at": created_at_str,
- "version": person_data["version"],
- }
-
- producer.produce(topic=KAFKA_PERSON, data=kafka_data)
-
-
-def fetch_person_from_postgres(cursor, team_id: int, person_uuid: str) -> dict | None:
- """Fetch a single person record from Postgres.
-
- Works with both RealDictCursor (returns dict rows with auto-parsed JSONB)
- and regular cursor (returns tuple rows with JSONB as strings).
- """
-
- columns = [
- "id",
- "uuid",
- "properties",
- "properties_last_updated_at",
- "properties_last_operation",
- "version",
- "is_identified",
- "created_at",
- ]
-
- cursor.execute(
- """
- SELECT
- id,
- uuid::text,
- properties,
- properties_last_updated_at,
- properties_last_operation,
- version,
- is_identified,
- created_at
- FROM posthog_person
- WHERE team_id = %s AND uuid = %s::uuid AND is_deleted = false
- """,
- (team_id, person_uuid),
- )
- row = cursor.fetchone()
- if row is None:
- return None
- # Handle both dict rows (RealDictCursor) and tuple rows (regular cursor)
- if isinstance(row, dict):
- result = dict(row)
- else:
- result = dict(zip(columns, row))
-
- # Parse JSONB columns if they're strings (Django cursor doesn't auto-parse)
- for json_col in ["properties", "properties_last_updated_at", "properties_last_operation"]:
- if isinstance(result.get(json_col), str):
- result[json_col] = json.loads(result[json_col])
-
- return result
-
-
-def backup_person_with_computed_state(
- cursor,
- job_id: str,
- team_id: int,
- person: dict,
- person_property_diffs: PersonPropertyDiffs,
- computed_update: dict,
- new_version: int,
-) -> bool:
- """
- Store person state in backup table with both before and after states.
- Called after computing the update but before applying it.
- Returns True if backup was successful, False otherwise.
- """
- pending_operations = []
- for key, pv in person_property_diffs.set_updates.items():
- pending_operations.append(
- {"key": key, "value": pv.value, "timestamp": pv.timestamp.isoformat(), "operation": "set"}
- )
- for key, pv in person_property_diffs.set_once_updates.items():
- pending_operations.append(
- {"key": key, "value": pv.value, "timestamp": pv.timestamp.isoformat(), "operation": "set_once"}
- )
- for key, pv in person_property_diffs.unset_updates.items():
- pending_operations.append(
- {"key": key, "value": None, "timestamp": pv.timestamp.isoformat(), "operation": "unset"}
- )
-
- cursor.execute(
- """
- INSERT INTO posthog_person_reconciliation_backup (
- job_id, team_id, person_id, uuid,
- properties, properties_last_updated_at, properties_last_operation,
- version, is_identified, created_at, is_user_id,
- pending_operations,
- properties_after, properties_last_updated_at_after,
- properties_last_operation_after, version_after
- ) VALUES (
- %s, %s, %s, %s::uuid,
- %s, %s, %s,
- %s, %s, %s, %s,
- %s,
- %s, %s, %s, %s
- )
- ON CONFLICT (job_id, team_id, person_id) DO NOTHING
- """,
- (
- job_id,
- team_id,
- person["id"],
- str(person["uuid"]),
- json.dumps(person.get("properties", {})),
- json.dumps(person.get("properties_last_updated_at")),
- json.dumps(person.get("properties_last_operation")),
- person.get("version"),
- person.get("is_identified", False),
- person.get("created_at"),
- person.get("is_user_id"),
- json.dumps(pending_operations),
- json.dumps(computed_update["properties"]),
- json.dumps(computed_update["properties_last_updated_at"]),
- json.dumps(computed_update["properties_last_operation"]),
- new_version,
- ),
- )
- return cursor.rowcount > 0
-
-
-@frozen
-class PersonUpdateResult:
- success: bool
- updated_person_data: dict | None
- backup_created: bool
- skip_reason: str
-
-
-def update_person_with_version_check(
- cursor,
- job_id: str,
- team_id: int,
- person_uuid: str,
- person_property_diffs: PersonPropertyDiffs,
- dry_run: bool = False,
- backup_enabled: bool = True,
- max_retries: int = 3,
-) -> PersonUpdateResult:
- """
- Update a person's properties with optimistic locking and conflict resolution.
-
- Flow:
- 1. Fetch person from Postgres
- 2. If Postgres version == CH version: apply diffs normally, UPDATE WHERE version = ch_version
- 3. If versions differ (conflict): fetch CH properties, do 3-way merge, UPDATE WHERE version = postgres_version
- 4. On UPDATE failure, retry
-
- Args:
- cursor: Database cursor
- job_id: Dagster run ID for backup tracking
- team_id: Team ID
- person_uuid: Person UUID
- person_property_diffs: Property diffs from ClickHouse (includes person_version)
- dry_run: If True, don't actually write the UPDATE
- backup_enabled: If True, store before/after state in backup table
- max_retries: Maximum retry attempts on version mismatch
-
- Returns:
- PersonUpdateResult where:
- - updated_person_data contains the final state for Kafka publishing
- - backup_created indicates if a backup row was inserted
- - skip_reason indicates why the person was skipped (see SkipReason class)
- """
- ch_version = person_property_diffs.person_version
-
- for _attempt in range(max_retries):
- # Fetch current person state from Postgres
- person = fetch_person_from_postgres(cursor, team_id, person_uuid)
- if not person:
- return PersonUpdateResult(
- success=False, updated_person_data=None, backup_created=False, skip_reason=SkipReason.NOT_FOUND
- )
-
- postgres_version = person.get("version") or 0
-
- # Check if Postgres and CH are in sync
- if postgres_version == ch_version:
- # Simple case: no concurrent changes, apply diffs directly
- update = reconcile_person_properties(person, person_property_diffs)
- target_version = ch_version
- else:
- # Conflict: Postgres has different version than CH
- # Fetch CH properties at the version we computed diffs against for 3-way merge
- ch_properties = fetch_person_properties_from_clickhouse(team_id, person_uuid, ch_version)
- if ch_properties is None:
- # Person version doesn't exist in CH anymore, skip
- return PersonUpdateResult(
- success=False, updated_person_data=None, backup_created=False, skip_reason=SkipReason.NOT_FOUND
- )
-
- # 3-way merge: apply our changes while respecting concurrent Postgres changes
- update = reconcile_with_concurrent_changes(ch_properties, person, person_property_diffs)
- target_version = postgres_version
-
- if not update:
- # No changes needed (either no diffs or all conflicts resolved to Postgres values)
- return PersonUpdateResult(
- success=True, updated_person_data=None, backup_created=False, skip_reason=SkipReason.NO_CHANGES
- )
-
- # Backup before and after state for audit/rollback
- backup_created = False
- if backup_enabled:
- backup_created = backup_person_with_computed_state(
- cursor, job_id, team_id, person, person_property_diffs, update, target_version + 1
- )
-
- if dry_run:
- return PersonUpdateResult(
- success=True, updated_person_data=None, backup_created=backup_created, skip_reason=SkipReason.SUCCESS
- )
-
- # Write with version check
- cursor.execute(
- """
- UPDATE posthog_person SET
- properties = %s,
- properties_last_updated_at = %s,
- properties_last_operation = %s,
- version = %s
- WHERE team_id = %s AND uuid = %s::uuid AND version = %s
- RETURNING uuid
- """,
- (
- json.dumps(update["properties"]),
- json.dumps(update["properties_last_updated_at"]),
- json.dumps(update["properties_last_operation"]),
- target_version + 1,
- team_id,
- person_uuid,
- target_version,
- ),
- )
-
- if cursor.rowcount > 0:
- # Success - return data for Kafka publishing
- return PersonUpdateResult(
- success=True,
- updated_person_data={
- "id": person_uuid,
- "team_id": team_id,
- "properties": update["properties"],
- "is_identified": person.get("is_identified", False),
- "is_deleted": 0,
- "created_at": person.get("created_at"),
- "version": target_version + 1,
- },
- backup_created=backup_created,
- skip_reason=SkipReason.SUCCESS,
- )
-
- # Version mismatch during UPDATE - retry with fresh data
- # (loop will re-fetch person)
-
- # Exhausted retries
- return PersonUpdateResult(
- success=False, updated_person_data=None, backup_created=False, skip_reason=SkipReason.VERSION_CONFLICT
- )
-
-
-@dataclass
-class BatchProcessingResult:
- """Result of processing persons in batches."""
-
- total_processed: int
- total_updated: int
- total_skipped: int
- total_commits: int
-
-
-def process_persons_in_batches(
- person_property_diffs: list[PersonPropertyDiffs],
- cursor: Any,
- job_id: str,
- team_id: int,
- batch_size: int,
- dry_run: bool,
- backup_enabled: bool,
- commit_fn: Any, # Callable to commit transaction
- on_person_updated: Any = None, # Optional callback for Kafka publishing etc
- on_batch_committed: Any = None, # Optional callback after each batch commit
- logger: Any = None, # Optional logger for logging skipped persons
-) -> BatchProcessingResult:
- """
- Process person property updates in batches with configurable commit frequency.
-
- This function is separated from the Dagster op for testability.
-
- Args:
- person_property_diffs: List of persons with their property diffs
- cursor: Database cursor (already open in transaction)
- job_id: Job identifier for backups
- team_id: Team being processed
- batch_size: Number of persons per batch (0 = all in one batch)
- dry_run: If True, don't actually apply changes
- backup_enabled: If True, create backup rows
- commit_fn: Function to call for committing the transaction
- on_person_updated: Optional callback(person_data) when a person is updated
- on_batch_committed: Optional callback(batch_num, batch_persons) after each commit
- logger: Optional logger for logging skipped persons (e.g., context.log from Dagster)
-
- Returns:
- BatchProcessingResult with counts
- """
- if not person_property_diffs:
- return BatchProcessingResult(
- total_processed=0,
- total_updated=0,
- total_skipped=0,
- total_commits=0,
- )
-
- # Determine effective batch size
- effective_batch_size = batch_size if batch_size > 0 else len(person_property_diffs)
-
- total_processed = 0
- total_updated = 0
- total_skipped = 0
- total_commits = 0
-
- batch_persons_to_publish: list[dict] = []
- batch_has_backups = False
- batch_num = 1
-
- def commit_batch() -> None:
- nonlocal total_commits, batch_persons_to_publish, batch_has_backups, batch_num
-
- if not batch_persons_to_publish and not batch_has_backups:
- return
-
- # Commit if we have updates OR backups
- should_commit = (batch_persons_to_publish and not dry_run) or batch_has_backups
- if should_commit:
- commit_fn()
- total_commits += 1
-
- # Notify callback
- if on_batch_committed:
- on_batch_committed(batch_num, batch_persons_to_publish)
-
- # Reset for next batch
- batch_persons_to_publish = []
- batch_has_backups = False
- batch_num += 1
-
- for i, person_diffs in enumerate(person_property_diffs):
- update_result = update_person_with_version_check(
- cursor=cursor,
- job_id=job_id,
- team_id=team_id,
- person_uuid=person_diffs.person_id,
- person_property_diffs=person_diffs,
- dry_run=dry_run,
- backup_enabled=backup_enabled,
- )
-
- if update_result.backup_created:
- batch_has_backups = True
-
- if not update_result.success:
- total_skipped += 1
- if logger:
- logger.warning(
- f"Skipped person uuid={person_diffs.person_id} team_id={team_id} reason={update_result.skip_reason}"
- )
- elif update_result.updated_person_data:
- batch_persons_to_publish.append(update_result.updated_person_data)
- total_updated += 1
- if on_person_updated:
- on_person_updated(update_result.updated_person_data)
- else:
- # No changes needed (success=True but updated_person_data=None)
- total_skipped += 1
-
- total_processed += 1
-
- # Commit batch if we've reached batch_size
- persons_in_batch = (i % effective_batch_size) + 1
- if persons_in_batch == effective_batch_size:
- commit_batch()
-
- # Commit final batch (if any remaining)
- if batch_persons_to_publish or batch_has_backups:
- commit_batch()
-
- return BatchProcessingResult(
- total_processed=total_processed,
- total_updated=total_updated,
- total_skipped=total_skipped,
- total_commits=total_commits,
- )
-
-
-def query_team_ids_from_clickhouse(
- bug_window_start: str,
- bug_window_end: str,
- min_team_id: int | None = None,
- max_team_id: int | None = None,
- exclude_team_ids: list[int] | None = None,
- include_team_ids: list[int] | None = None,
-) -> list[int]:
- """
- Query ClickHouse for distinct team_ids with property-setting events in the bug window.
-
- Args:
- bug_window_start: Start of bug window (CH format: "YYYY-MM-DD HH:MM:SS")
- bug_window_end: End of bug window (CH format: "YYYY-MM-DD HH:MM:SS")
- min_team_id: Optional minimum team_id (inclusive)
- max_team_id: Optional maximum team_id (inclusive)
- exclude_team_ids: Optional list of team_ids to exclude
- include_team_ids: Optional list of team_ids to include (only these teams will be queried)
-
- Returns:
- List of team_ids sorted ascending
-
- Raises:
- ValueError: If min_team_id > max_team_id
- """
- if min_team_id is not None and max_team_id is not None and min_team_id > max_team_id:
- raise ValueError(
- f"Invalid team_id range: min_team_id ({min_team_id}) cannot be greater than max_team_id ({max_team_id})"
- )
-
- team_id_filters = []
- params: dict[str, Any] = {
- "bug_window_start": bug_window_start,
- "bug_window_end": bug_window_end,
- }
-
- if include_team_ids:
- team_id_filters.append("team_id IN %(include_team_ids)s")
- params["include_team_ids"] = tuple(include_team_ids)
-
- if min_team_id is not None:
- team_id_filters.append("team_id >= %(min_team_id)s")
- params["min_team_id"] = min_team_id
-
- if max_team_id is not None:
- team_id_filters.append("team_id <= %(max_team_id)s")
- params["max_team_id"] = max_team_id
-
- if exclude_team_ids:
- team_id_filters.append("team_id NOT IN %(exclude_team_ids)s")
- params["exclude_team_ids"] = tuple(exclude_team_ids)
-
- team_id_filter_clause = (" AND " + " AND ".join(team_id_filters)) if team_id_filters else ""
-
- query = f"""
- SELECT DISTINCT team_id
- FROM events
- WHERE timestamp >= %(bug_window_start)s
- AND timestamp < %(bug_window_end)s
- AND (
- JSONHas(properties, '$set') = 1
- OR JSONHas(properties, '$set_once') = 1
- OR JSONHas(properties, '$unset') = 1
- ){team_id_filter_clause}
- ORDER BY team_id
- """
-
- results = sync_execute(query, params)
- return [int(row[0]) for row in results]
-
-
-def get_team_ids_to_reconcile(
- context: dagster.OpExecutionContext,
- config: PersonPropertyReconciliationConfig,
- cluster: dagster.ResourceParam[ClickhouseCluster],
-) -> list[int]:
- """
- Query ClickHouse for distinct team_ids with property-setting events in the bug window.
- """
- if not config.bug_window_end:
- raise dagster.Failure(
- description="bug_window_end must be provided",
- metadata={
- "bug_window_start": dagster.MetadataValue.text(config.bug_window_start),
- },
- )
-
- filter_info_parts = []
- if config.team_ids:
- filter_info_parts.append(f"team_ids: {config.team_ids}")
- if config.min_team_id is not None or config.max_team_id is not None:
- filter_info_parts.append(f"range: {config.min_team_id or 'any'} to {config.max_team_id or 'any'}")
- if config.exclude_team_ids:
- filter_info_parts.append(f"excluding: {config.exclude_team_ids}")
- filter_info = f" ({', '.join(filter_info_parts)})" if filter_info_parts else ""
-
- context.log.info(
- f"Querying for team_ids with property events between {config.bug_window_start} and {config.bug_window_end}{filter_info}"
- )
-
- team_ids = query_team_ids_from_clickhouse(
- bug_window_start=config.bug_window_start,
- bug_window_end=config.bug_window_end,
- min_team_id=config.min_team_id,
- max_team_id=config.max_team_id,
- exclude_team_ids=config.exclude_team_ids,
- include_team_ids=config.team_ids,
- )
-
- if not team_ids:
- context.log.info("No team IDs found with property events in bug window")
- return []
-
- context.log.info(f"Found {len(team_ids)} teams with property events")
- context.log.info(f"Sample of teams to process: {team_ids[:10]}" + ("..." if len(team_ids) > 10 else ""))
- context.add_output_metadata({"team_count": dagster.MetadataValue.int(len(team_ids))})
-
- return team_ids
-
-
[email protected](out=dagster.DynamicOut(list[int]))
-def create_team_chunks(
- context: dagster.OpExecutionContext,
- config: PersonPropertyReconciliationConfig,
- team_ids: list[int],
-):
- """
- Create chunks of team_ids based on teams_per_chunk config.
- Yields DynamicOutput for each chunk containing one or more teams.
- """
- if not team_ids:
- context.log.info("No teams to process")
- return
-
- teams_per_chunk = max(1, config.teams_per_chunk)
- num_chunks = (len(team_ids) + teams_per_chunk - 1) // teams_per_chunk
- context.log.info(f"Creating {num_chunks} chunks for {len(team_ids)} teams (teams_per_chunk={teams_per_chunk})")
-
- for i in range(0, len(team_ids), teams_per_chunk):
- chunk = team_ids[i : i + teams_per_chunk]
- chunk_key = f"teams_{chunk[0]}_{chunk[-1]}" if len(chunk) > 1 else f"team_{chunk[0]}"
- context.log.info(
- f"Yielding chunk with {len(chunk)} teams: {chunk[0]}-{chunk[-1] if len(chunk) > 1 else chunk[0]}"
- )
- yield dagster.DynamicOutput(
- value=chunk,
- mapping_key=chunk_key,
- )
-
-
-@dataclass
-class TeamReconciliationResult:
- """Result of reconciling a single team."""
-
- team_id: int
- persons_processed: int
- persons_updated: int
- persons_skipped: int
-
-
-def reconcile_single_team(
- team_id: int,
- bug_window_start: str,
- bug_window_end: str,
- run_id: str,
- batch_size: int,
- dry_run: bool,
- backup_enabled: bool,
- team_ch_props_fetch_window_seconds: int,
- person_batch_size: int,
- persons_database: psycopg2.extensions.connection,
- kafka_producer: _KafkaProducer,
- logger: Any,
-) -> TeamReconciliationResult:
- """
- Reconcile person properties for all affected persons in a single team.
-
- This function supports two modes:
- - Non-windowed (team_ch_props_fetch_window_seconds <= 0): Single query to ClickHouse
- - Windowed batched (team_ch_props_fetch_window_seconds > 0): Process persons in batches,
- each batch iterating through time windows. This prevents OOM on large teams.
-
- This function does not catch exceptions - the caller is responsible for error handling.
- """
-
- # Callback for batch commits - handles Kafka publishing after each batch
- def on_batch_committed(batch_num: int, batch_persons: list[dict]) -> None:
- if batch_persons and not dry_run:
- for person_data in batch_persons:
- try:
- publish_person_to_kafka(person_data, kafka_producer)
- except Exception as kafka_error:
- logger.warning(f"Failed to publish person to Kafka: {person_data['id']}, error: {kafka_error}")
- try:
- kafka_producer.flush()
- except Exception as flush_error:
- logger.warning(f"Failed to flush Kafka producer: {flush_error}")
-
- logger.info(f"Batch {batch_num}: committed {len(batch_persons)} updates for team_id={team_id}")
- elif batch_persons and dry_run:
- logger.info(f"[DRY RUN] Batch {batch_num}: would apply {len(batch_persons)} updates for team_id={team_id}")
-
- # Use windowed batched mode for large teams (window_seconds > 0)
- if team_ch_props_fetch_window_seconds > 0:
- logger.info(
- f"Using windowed batched mode: team_id={team_id}, "
- f"bug_window=[{bug_window_start}, {bug_window_end}], "
- f"window_seconds={team_ch_props_fetch_window_seconds}, "
- f"person_batch_size={person_batch_size}"
- )
-
- total_processed = 0
- total_updated = 0
- total_skipped = 0
- person_batch_num = 0
-
- with persons_database.cursor() as cursor:
- cursor.execute("SET application_name = 'person_property_reconciliation'")
- cursor.execute("SET lock_timeout = '5s'")
- cursor.execute("SET statement_timeout = '30min'")
- cursor.execute("SET synchronous_commit = off")
-
- # Process each batch of persons from the iterator
- for person_batch in get_person_property_updates_windowed_batched(
- team_id=team_id,
- bug_window_start=bug_window_start,
- bug_window_end=bug_window_end,
- window_seconds=team_ch_props_fetch_window_seconds,
- person_batch_size=person_batch_size,
- logger=logger,
- ):
- person_batch_num += 1
-
- # Filter conflicting set/unset operations for this batch
- filtered_batch = filter_event_person_properties(person_batch)
-
- if not filtered_batch:
- logger.info(f"Person batch {person_batch_num}: no diffs after filtering for team_id={team_id}")
- continue
-
- logger.info(
- f"Person batch {person_batch_num}: processing {len(filtered_batch)} persons for team_id={team_id}"
- )
-
- # Process this batch
- result = process_persons_in_batches(
- person_property_diffs=filtered_batch,
- cursor=cursor,
- job_id=run_id,
- team_id=team_id,
- batch_size=batch_size,
- dry_run=dry_run,
- backup_enabled=backup_enabled,
- commit_fn=persons_database.commit,
- on_batch_committed=on_batch_committed,
- logger=logger,
- )
-
- total_processed += result.total_processed
- total_updated += result.total_updated
- total_skipped += result.total_skipped
-
- logger.info(
- f"Completed team_id={team_id} (windowed batched): processed={total_processed}, "
- f"updated={total_updated}, skipped={total_skipped}, person_batches={person_batch_num}"
- )
-
- return TeamReconciliationResult(
- team_id=team_id,
- persons_processed=total_processed,
- persons_updated=total_updated,
- persons_skipped=total_skipped,
- )
-
- # Non-windowed mode (original behavior)
- logger.info(f"Using non-windowed mode: team_id={team_id}, bug_window_start={bug_window_start}")
- person_property_diffs = get_person_property_updates_from_clickhouse(team_id, bug_window_start)
- logger.info(f"Found {len(person_property_diffs)} persons with property diffs for team_id={team_id}")
-
- # Filter conflicting set/unset operations
- person_property_diffs = filter_event_person_properties(person_property_diffs)
-
- if not person_property_diffs:
- logger.info(f"No persons to reconcile for team_id={team_id}")
- return TeamReconciliationResult(
- team_id=team_id,
- persons_processed=0,
- persons_updated=0,
- persons_skipped=0,
- )
-
- logger.info(f"Processing {len(person_property_diffs)} persons for team_id={team_id}")
-
- with persons_database.cursor() as cursor:
- cursor.execute("SET application_name = 'person_property_reconciliation'")
- cursor.execute("SET lock_timeout = '5s'")
- cursor.execute("SET statement_timeout = '30min'")
- cursor.execute("SET synchronous_commit = off")
-
- # Process persons in batches using the helper function
- result = process_persons_in_batches(
- person_property_diffs=person_property_diffs,
- cursor=cursor,
- job_id=run_id,
- team_id=team_id,
- batch_size=batch_size,
- dry_run=dry_run,
- backup_enabled=backup_enabled,
- commit_fn=persons_database.commit,
- on_batch_committed=on_batch_committed,
- logger=logger,
- )
-
- logger.info(
- f"Completed team_id={team_id}: processed={result.total_processed}, "
- f"updated={result.total_updated}, skipped={result.total_skipped}"
- )
-
- return TeamReconciliationResult(
- team_id=team_id,
- persons_processed=result.total_processed,
- persons_updated=result.total_updated,
- persons_skipped=result.total_skipped,
- )
-
-
-def reconcile_team_chunk(
- context: dagster.OpExecutionContext,
- config: PersonPropertyReconciliationConfig,
- chunk: list[int],
- persons_database: dagster.ResourceParam[psycopg2.extensions.connection],
- cluster: dagster.ResourceParam[ClickhouseCluster],
- kafka_producer: dagster.ResourceParam[_KafkaProducer],
-) -> dict[str, Any]:
- """
- Reconcile person properties for all affected persons in a chunk of teams.
- """
- team_ids = chunk
- chunk_id = f"teams_{team_ids[0]}_{team_ids[-1]}" if len(team_ids) > 1 else f"team_{team_ids[0]}"
- job_name = context.run.job_name
- run_id = context.run.run_id
-
- metrics_client = MetricsClient(cluster)
-
- context.log.info(
- f"Starting reconciliation for {len(team_ids)} teams: {team_ids}, "
- f"bug_window_start: {config.bug_window_start}, "
- f"dry_run: {config.dry_run}"
- )
-
- total_persons_processed = 0
- total_persons_updated = 0
- total_persons_skipped = 0
- total_teams_succeeded = 0
- total_teams_failed = 0
- teams_results: list[dict[str, Any]] = []
-
- start_time = time.time()
-
- for team_id in team_ids:
- team_result: dict[str, Any] = {
- "team_id": team_id,
- "status": "success",
- "persons_processed": 0,
- "persons_updated": 0,
- "persons_skipped": 0,
- }
-
- try:
- result = reconcile_single_team(
- team_id=team_id,
- bug_window_start=config.bug_window_start,
- bug_window_end=config.bug_window_end or config.bug_window_start, # Fallback for backward compat
- run_id=run_id,
- batch_size=config.batch_size,
- dry_run=config.dry_run,
- backup_enabled=config.backup_enabled,
- team_ch_props_fetch_window_seconds=config.team_ch_props_fetch_window_seconds,
- person_batch_size=config.person_batch_size,
- persons_database=persons_database,
- kafka_producer=kafka_producer,
- logger=context.log,
- )
- team_result["persons_processed"] = result.persons_processed
- team_result["persons_updated"] = result.persons_updated
- team_result["persons_skipped"] = result.persons_skipped
- total_teams_succeeded += 1
-
- except Exception as e:
- team_result["status"] = "failed"
- team_result["error"] = str(e)
- total_teams_failed += 1
-
- context.log.exception(f"Failed team_id={team_id}: {e}")
-
- try:
- metrics_client.increment(
- "person_property_reconciliation_error",
- labels={"job_name": job_name, "chunk_id": chunk_id, "team_id": str(team_id), "reason": "error"},
- value=1.0,
- ).result()
- except Exception:
- pass
-
- total_persons_processed += team_result["persons_processed"]
- total_persons_updated += team_result["persons_updated"]
- total_persons_skipped += team_result["persons_skipped"]
- teams_results.append(team_result)
-
- # Track metrics for the entire chunk
- duration = time.time() - start_time
- try:
- metrics_client.increment(
- "person_property_reconciliation_persons_processed_total",
- labels={"job_name": job_name, "chunk_id": chunk_id},
- value=float(total_persons_processed),
- ).result()
- metrics_client.increment(
- "person_property_reconciliation_duration_seconds_total",
- labels={"job_name": job_name, "chunk_id": chunk_id},
- value=duration,
- ).result()
- metrics_client.increment(
- "person_property_reconciliation_persons_updated_total",
- labels={"job_name": job_name, "chunk_id": chunk_id},
- value=float(total_persons_updated),
- ).result()
- metrics_client.increment(
- "person_property_reconciliation_persons_skipped_total",
- labels={"job_name": job_name, "chunk_id": chunk_id},
- value=float(total_persons_skipped),
- ).result()
- metrics_client.increment(
- "person_property_reconciliation_teams_succeeded_total",
- labels={"job_name": job_name, "chunk_id": chunk_id},
- value=float(total_teams_succeeded),
- ).result()
- metrics_client.increment(
- "person_property_reconciliation_teams_failed_total",
- labels={"job_name": job_name, "chunk_id": chunk_id},
- value=float(total_teams_failed),
- ).result()
- except Exception:
- pass
-
- # Log failed teams for easy searching
- failed_teams = [r for r in teams_results if r["status"] == "failed"]
- if failed_teams:
- failed_team_ids = [r["team_id"] for r in failed_teams]
- context.log.warning(f"Failed teams in chunk {chunk_id}: {failed_team_ids}")
-
- context.log.info(
- f"Completed chunk {chunk_id}: teams={len(team_ids)} (succeeded={total_teams_succeeded}, failed={total_teams_failed}), "
- f"persons: processed={total_persons_processed}, updated={total_persons_updated}, skipped={total_persons_skipped}"
- )
-
- context.add_output_metadata(
- {
- "team_ids": dagster.MetadataValue.text(str(team_ids)),
- "teams_count": dagster.MetadataValue.int(len(team_ids)),
- "teams_succeeded": dagster.MetadataValue.int(total_teams_succeeded),
- "teams_failed": dagster.MetadataValue.int(total_teams_failed),
- "failed_team_ids": dagster.MetadataValue.text(str([r["team_id"] for r in failed_teams])),
- "persons_processed": dagster.MetadataValue.int(total_persons_processed),
- "persons_updated": dagster.MetadataValue.int(total_persons_updated),
- "persons_skipped": dagster.MetadataValue.int(total_persons_skipped),
- "teams_results": dagster.MetadataValue.json(teams_results),
- }
- )
-
- return {
- "team_ids": team_ids,
- "teams_count": len(team_ids),
- "teams_succeeded": total_teams_succeeded,
- "teams_failed": total_teams_failed,
- "persons_processed": total_persons_processed,
- "persons_updated": total_persons_updated,
- "persons_skipped": total_persons_skipped,
- "teams_results": teams_results,
- }
-
-
- tags={"owner": JobOwners.TEAM_INGESTION.value},
- executor_def=executor_def,
-)
-def person_property_reconciliation_job():
- """
- One-time job to reconcile person properties that were missed due to the
- ASSERT_VERSION bug.
-
- IMPORTANT: This job uses k8s_job_executor which spawns a separate Kubernetes pod
- for each team. Without concurrency limits, this could spin up thousands of pods
- simultaneously. Before running:
-
- 1. Set step concurrency limits in the Dagster UI (Deployment > Configuration)
- or via the run coordinator's concurrency settings
- 2. Consider using the `team_ids` config to process a subset of teams first
- 3. Monitor cluster resources during execution
- """
- team_ids = get_team_ids_to_reconcile()
- chunks = create_team_chunks(team_ids)
- chunks.map(reconcile_team_chunk)
-
-
-# --- Sensor-based scheduler for automated batch reconciliation ---
-
-
-class ReconciliationSchedulerConfig(dagster.Config):
- """Configuration for the reconciliation scheduler sensor.
-
- This sensor automates launching multiple reconciliation jobs across a range
- of team IDs while respecting concurrency limits.
-
- You must provide EITHER:
- - team_ids: explicit list of team IDs to process
- - OR both range_start and range_end: to scan a team ID range
- """
-
- # Option 1: Explicit list of team IDs
- team_ids: list[int] | None = None
-
- # Option 2: Range-based scanning
- range_start: int | None = None # First team_id (inclusive)
- range_end: int | None = None # Last team_id (inclusive)
-
- chunk_size: int = 1000 # Teams per job run (for range mode) or list chunk size (for team_ids mode)
- max_concurrent_jobs: int = 5 # Max reconciliation jobs the sensor can schedule at once
- max_concurrent_tasks: int = 10 # Max k8s pods per job (executor concurrency)
-
- # Base job configuration (applied to all runs)
- bug_window_start: str # ClickHouse format: "YYYY-MM-DD HH:MM:SS"
- bug_window_end: str # ClickHouse format: "YYYY-MM-DD HH:MM:SS"
- dry_run: bool = False
- backup_enabled: bool = True
- batch_size: int = 100
- teams_per_chunk: int = 100 # Number of teams to process per task
- team_ch_props_fetch_window_seconds: int = 0 # 0 = single query; >0 = split into N-second windows
- person_batch_size: int = 10000 # Persons per batch in windowed mode (keeps IN clauses small)
-
- # Resource configuration - the env var name for the PG connection string
- persons_db_env_var: str = "PERSONS_DB_WRITER_URL" # Env var for Postgres connection URL
-
-
-def build_reconciliation_run_config(
- config: ReconciliationSchedulerConfig,
- min_team_id: int | None = None,
- max_team_id: int | None = None,
- team_ids: list[int] | None = None,
-) -> dict[str, Any]:
- """Build run config for a single reconciliation job.
-
- Either provide team_ids (explicit list) OR min_team_id/max_team_id (range scan).
- """
- op_config: dict[str, Any] = {
- "bug_window_start": config.bug_window_start,
- "bug_window_end": config.bug_window_end,
- "dry_run": config.dry_run,
- "backup_enabled": config.backup_enabled,
- "batch_size": config.batch_size,
- "teams_per_chunk": config.teams_per_chunk,
- "team_ch_props_fetch_window_seconds": config.team_ch_props_fetch_window_seconds,
- "person_batch_size": config.person_batch_size,
- }
-
- if team_ids is not None:
- op_config["team_ids"] = team_ids
- else:
- op_config["min_team_id"] = min_team_id
- op_config["max_team_id"] = max_team_id
-
- return {
- "execution": {"config": {"max_concurrent": config.max_concurrent_tasks}},
- "ops": {
- "get_team_ids_to_reconcile": {"config": op_config},
- "create_team_chunks": {"config": op_config},
- "reconcile_team_chunk": {"config": op_config},
- },
- "resources": {
- "cluster": {
- "config": {
- "client_settings": {
- "lightweight_deletes_sync": "0",
- "max_execution_time": "0",
- "max_memory_usage": "0",
- "mutations_sync": "0",
- "receive_timeout": "900",
- }
- }
- },
- "persons_database": {"config": {"connection_url": {"env": config.persons_db_env_var}}},
- },
- }
-
-
- job=person_property_reconciliation_job,
- minimum_interval_seconds=30,
- default_status=dagster.DefaultSensorStatus.STOPPED,
-)
-def person_property_reconciliation_scheduler(context: dagster.SensorEvaluationContext):
- """
- Sensor that automatically schedules person property reconciliation jobs.
-
- Supports two modes:
- 1. team_ids mode: Process an explicit list of team IDs
- 2. range mode: Scan a team_id range (range_start to range_end)
-
- Splits work into chunks and launches jobs up to max_concurrent_jobs,
- tracking progress via cursor. Enable via Dagster UI after configuring.
-
- Cursor format (team_ids mode):
- {"team_ids": [1, 2, 3, ...], "next_team_index": 0, ...}
- Cursor format (range mode):
- {"range_start": 1, "range_end": 10000, "next_chunk_start": 1, ...}
-
- Configuration (set via Dagster UI or launchpad):
- - team_ids: Explicit list of team IDs (mutually exclusive with range_start/range_end)
- - range_start/range_end: Team ID range to process (inclusive)
- - chunk_size: Number of teams per job (default 1000)
- - max_concurrent_jobs: Max reconciliation jobs sensor can schedule at once (default 5, cap 50)
- - max_concurrent_tasks: Max k8s pods per job / executor concurrency (default 10, cap 100)
- - bug_window_start/end: Time window for the reconciliation
- - dry_run, backup_enabled, batch_size: Passed to each job
- """
- sensor_config_raw = context.cursor
-
- if not sensor_config_raw:
- return dagster.SkipReason(
- "No cursor set. Initialize by setting cursor to JSON. "
- 'For team_ids mode: {"team_ids": [1, 2, 3], "chunk_size": 100, ...}. '
- 'For range mode: {"range_start": 1, "range_end": 10000, "chunk_size": 1000, ...}. '
- 'Common fields: "max_concurrent_jobs", "max_concurrent_tasks", "bug_window_start", "bug_window_end", '
- '"dry_run", "backup_enabled", "batch_size"'
- )
-
- try:
- cursor_data = json.loads(sensor_config_raw)
- except json.JSONDecodeError:
- return dagster.SkipReason(f"Invalid cursor JSON: {sensor_config_raw[:100]}...")
-
- # Extract common config
- chunk_size = cursor_data.get("chunk_size", 1000)
- max_concurrent_jobs = cursor_data.get("max_concurrent_jobs", 5)
- max_concurrent_tasks = cursor_data.get("max_concurrent_tasks", 10)
- bug_window_start = cursor_data.get("bug_window_start", "")
- bug_window_end = cursor_data.get("bug_window_end", "")
-
- # Determine mode: team_ids vs range
- team_ids = cursor_data.get("team_ids")
- range_start = cursor_data.get("range_start")
- range_end = cursor_data.get("range_end")
-
- has_team_ids = team_ids is not None and len(team_ids) > 0
- has_range = range_start is not None and range_end is not None
-
- # Validate: must have exactly one mode
- if has_team_ids and has_range:
- return dagster.SkipReason(
- "Invalid config: cannot specify both team_ids and range_start/range_end. Choose one mode."
- )
- if not has_team_ids and not has_range:
- return dagster.SkipReason(
- "Invalid config: must specify either team_ids (list) OR both range_start and range_end"
- )
-
- # Validate common config
- MAX_CONCURRENT_JOBS_CAP = 50
- MAX_CONCURRENT_TASKS_CAP = 100
- if chunk_size <= 0:
- return dagster.SkipReason(f"Invalid config: chunk_size must be > 0, got {chunk_size}")
- if max_concurrent_jobs <= 0:
- return dagster.SkipReason(f"Invalid config: max_concurrent_jobs must be > 0, got {max_concurrent_jobs}")
- if max_concurrent_jobs > MAX_CONCURRENT_JOBS_CAP:
- return dagster.SkipReason(
- f"Invalid config: max_concurrent_jobs ({max_concurrent_jobs}) exceeds cap of {MAX_CONCURRENT_JOBS_CAP}"
- )
- if max_concurrent_tasks <= 0:
- return dagster.SkipReason(f"Invalid config: max_concurrent_tasks must be > 0, got {max_concurrent_tasks}")
- if max_concurrent_tasks > MAX_CONCURRENT_TASKS_CAP:
- return dagster.SkipReason(
- f"Invalid config: max_concurrent_tasks ({max_concurrent_tasks}) exceeds cap of {MAX_CONCURRENT_TASKS_CAP}"
- )
- if not bug_window_start:
- return dagster.SkipReason("Invalid config: bug_window_start is required")
- if not bug_window_end:
- return dagster.SkipReason("Invalid config: bug_window_end is required")
-
- # Mode-specific validation
- if has_range and range_start > range_end:
- return dagster.SkipReason(
- f"Invalid config: range_start ({range_start}) cannot be greater than range_end ({range_end})"
- )
-
- # Build config object for run config generation
- config = ReconciliationSchedulerConfig(
- team_ids=team_ids if has_team_ids else None,
- range_start=range_start if has_range else None,
- range_end=range_end if has_range else None,
- chunk_size=chunk_size,
- max_concurrent_jobs=max_concurrent_jobs,
- max_concurrent_tasks=max_concurrent_tasks,
- bug_window_start=bug_window_start,
- bug_window_end=bug_window_end,
- dry_run=cursor_data.get("dry_run", False),
- backup_enabled=cursor_data.get("backup_enabled", True),
- batch_size=cursor_data.get("batch_size", 100),
- teams_per_chunk=cursor_data.get("teams_per_chunk", 100),
- team_ch_props_fetch_window_seconds=cursor_data.get("team_ch_props_fetch_window_seconds", 0),
- person_batch_size=cursor_data.get("person_batch_size", 10000),
- persons_db_env_var=cursor_data.get("persons_db_env_var", "PERSONS_DB_WRITER_URL"),
- )
-
- # Count active runs for this job
- active_runs = context.instance.get_run_records(
- dagster.RunsFilter(
- job_name="person_property_reconciliation_job",
- statuses=[
- dagster.DagsterRunStatus.QUEUED,
- dagster.DagsterRunStatus.NOT_STARTED,
- dagster.DagsterRunStatus.STARTING,
- dagster.DagsterRunStatus.STARTED,
- ],
- )
- )
- active_count = len(active_runs)
-
- available_slots = max(0, max_concurrent_jobs - active_count)
- if available_slots == 0:
- return dagster.SkipReason(f"At max concurrency ({active_count}/{max_concurrent_jobs} jobs), waiting for slots")
-
- context.log.info(f"Active runs: {active_count}/{max_concurrent_jobs}, available slots: {available_slots}")
-
- # Generate run requests based on mode
- run_requests: list[dagster.RunRequest] = []
-
- if has_team_ids:
- # Team IDs mode: chunk the list
- next_team_index = cursor_data.get("next_team_index", 0)
-
- if next_team_index >= len(team_ids):
- return dagster.SkipReason(f"Reconciliation complete: processed all {len(team_ids)} teams")
-
- while len(run_requests) < available_slots and next_team_index < len(team_ids):
- chunk_end_index = min(next_team_index + chunk_size, len(team_ids))
- chunk_team_ids = team_ids[next_team_index:chunk_end_index]
-
- run_config = build_reconciliation_run_config(
- config=config,
- team_ids=chunk_team_ids,
- )
-
- context.log.info(
- f"Scheduling run for team_ids[{next_team_index}:{chunk_end_index}] ({len(chunk_team_ids)} teams)"
- )
-
- run_requests.append(
- dagster.RunRequest(
- run_config=run_config,
- tags={
- "reconciliation_team_ids_range": f"{next_team_index}-{chunk_end_index - 1}",
- "reconciliation_team_count": str(len(chunk_team_ids)),
- "owner": JobOwners.TEAM_INGESTION.value,
- },
- )
- )
-
- next_team_index = chunk_end_index
-
- # Update cursor
- cursor_data["next_team_index"] = next_team_index
- new_cursor = json.dumps(cursor_data)
- context.log.info(f"Scheduled {len(run_requests)} runs, next_team_index: {next_team_index}")
-
- else:
- # Range mode: existing behavior
- next_chunk_start = cursor_data.get("next_chunk_start", range_start)
-
- if next_chunk_start > range_end:
- return dagster.SkipReason(f"Reconciliation complete: processed all teams up to {range_end}")
-
- current_start = next_chunk_start
-
- while len(run_requests) < available_slots and current_start <= range_end:
- chunk_end = min(current_start + chunk_size - 1, range_end)
-
- run_config = build_reconciliation_run_config(
- config=config,
- min_team_id=current_start,
- max_team_id=chunk_end,
- )
-
- context.log.info(f"Scheduling run for teams {current_start}-{chunk_end}")
-
- run_requests.append(
- dagster.RunRequest(
- run_config=run_config,
- tags={
- "reconciliation_range": f"{current_start}-{chunk_end}",
- "owner": JobOwners.TEAM_INGESTION.value,
- },
- )
- )
-
- current_start = chunk_end + 1
-
- # Update cursor
- cursor_data["next_chunk_start"] = current_start
- new_cursor = json.dumps(cursor_data)
- context.log.info(f"Scheduled {len(run_requests)} runs, next_chunk_start: {current_start}")
-
- return dagster.SensorResult(run_requests=run_requests, cursor=new_cursor)
diff --git a/posthog/dags/person_property_reconciliation_query_explanation.md b/posthog/dags/person_property_reconciliation_query_explanation.md
deleted file mode 100644
index 035093cfd314..000000000000
--- a/posthog/dags/person_property_reconciliation_query_explanation.md
+++ /dev/null
@@ -1,506 +0,0 @@
-# Person Property Reconciliation Query Explanation
-
-This document explains the ClickHouse query in `get_person_property_updates_from_clickhouse` (person_property_reconciliation.py:98-211).
-
-The query finds person properties that need reconciliation by comparing event-derived properties with current person state.
-
-## Query Structure Overview
-
-The query has 5 nested levels:
-
-1. **Core Event Extraction** - Extract properties from events, resolve merged persons, aggregate per property
-2. **Group into Arrays** - Collect all properties per person into one array
-3. **Split by Operation Type** - Separate `$set` and `$set_once` into parallel arrays
-4. **Join with Person State** - Get current person properties from ClickHouse
-5. **Compute Diff** - Find properties that differ or are missing
-
----
-
-## Level 1: Core Event Extraction with Person Resolution (Innermost)
-
-```sql
-SELECT
- if(notEmpty(overrides.distinct_id), overrides.person_id, e.person_id) AS person_id,
- kv_tuple.2 AS key,
- kv_tuple.1 AS prop_type,
- if(kv_tuple.1 = 'set',
- argMaxIf(kv_tuple.3, e.timestamp, kv_tuple.3 IS NOT NULL AND kv_tuple.3 != ''),
- argMinIf(kv_tuple.3, e.timestamp, kv_tuple.3 IS NOT NULL AND kv_tuple.3 != '')
- ) AS value,
- if(kv_tuple.1 = 'set', max(e.timestamp), min(e.timestamp)) AS kv_timestamp
-FROM events e
-LEFT OUTER JOIN (
- SELECT
- argMax(person_distinct_id_overrides.person_id, person_distinct_id_overrides.version) AS person_id,
- person_distinct_id_overrides.distinct_id AS distinct_id
- FROM person_distinct_id_overrides
- WHERE equals(person_distinct_id_overrides.team_id, %(team_id)s)
- GROUP BY person_distinct_id_overrides.distinct_id
- HAVING ifNull(equals(argMax(person_distinct_id_overrides.is_deleted, person_distinct_id_overrides.version), 0), 0)
-) AS overrides ON e.distinct_id = overrides.distinct_id
-ARRAY JOIN
- arrayConcat(
- arrayFilter(x -> x.3 IS NOT NULL AND x.3 != '' AND x.3 != 'null',
- arrayMap(x -> tuple('set', x.1, toString(x.2)),
- arrayFilter(x -> x.2 IS NOT NULL, JSONExtractKeysAndValuesRaw(e.properties, '$set'))
- )
- ),
- arrayFilter(x -> x.3 IS NOT NULL AND x.3 != '' AND x.3 != 'null',
- arrayMap(x -> tuple('set_once', x.1, toString(x.2)),
- arrayFilter(x -> x.2 IS NOT NULL, JSONExtractKeysAndValuesRaw(e.properties, '$set_once'))
- )
- )
- ) AS kv_tuple
-WHERE e.team_id = %(team_id)s
- AND e.timestamp > %(bug_window_start)s
- AND e.timestamp < %(bug_window_end)s
-GROUP BY person_id, kv_tuple.2, kv_tuple.1
-```
-
-### The ARRAY JOIN
-
-`ARRAY JOIN` "unrolls" an array into multiple rows. One input row becomes N output rows (one per array element).
-
-The right side of the ARRAY JOIN builds an array of property tuples from each event:
-
-```sql
-arrayConcat(
- -- $set properties
- arrayFilter(x -> x.3 IS NOT NULL AND x.3 != '' AND x.3 != 'null',
- arrayMap(x -> tuple('set', x.1, toString(x.2)),
- arrayFilter(x -> x.2 IS NOT NULL, JSONExtractKeysAndValuesRaw(e.properties, '$set'))
- )
- ),
- -- $set_once properties (same structure)
- arrayFilter(x -> x.3 IS NOT NULL AND x.3 != '' AND x.3 != 'null',
- arrayMap(x -> tuple('set_once', x.1, toString(x.2)),
- arrayFilter(x -> x.2 IS NOT NULL, JSONExtractKeysAndValuesRaw(e.properties, '$set_once'))
- )
- )
-) AS kv_tuple
-```
-
-Reading innermost to outermost (for the `$set` branch):
-
-1. **`JSONExtractKeysAndValuesRaw(e.properties, '$set')`** - Extract `$set` object as `[(key, raw_json_value), ...]`
-2. **`arrayFilter(x -> x.2 IS NOT NULL, ...)`** - Drop pairs where value is NULL
-3. **`arrayMap(x -> tuple('set', x.1, toString(x.2)), ...)`** - Transform each `(key, value)` into `('set', key, value_string)`
-4. **`arrayFilter(x -> x.3 IS NOT NULL AND x.3 != '' AND x.3 != 'null', ...)`** - Drop tuples where the string value is empty or literal `'null'`
-5. **`arrayConcat(..., ...)`** - Combine `$set` and `$set_once` arrays into one
-
-The resulting `kv_tuple` structure:
-
-- `kv_tuple.1` = operation type (`'set'` or `'set_once'` - only these two values, hardcoded)
-- `kv_tuple.2` = property key
-- `kv_tuple.3` = property value (as JSON string)
-
-Example: if an event has:
-
-```json
-{
- "$set": { "email": "[email protected]", "name": null },
- "$set_once": { "created": "2024-01-01" }
-}
-```
-
-The result is:
-
-```text
-[('set', 'email', '"[email protected]"'), ('set_once', 'created', '"2024-01-01"')]
-```
-
-(The `name` property is filtered out because its value is null)
-
-### The LEFT OUTER JOIN (Person Merge Resolution)
-
-```sql
-LEFT OUTER JOIN (
- SELECT
- argMax(person_distinct_id_overrides.person_id, person_distinct_id_overrides.version) AS person_id,
- person_distinct_id_overrides.distinct_id AS distinct_id
- FROM person_distinct_id_overrides
- WHERE equals(person_distinct_id_overrides.team_id, %(team_id)s)
- GROUP BY person_distinct_id_overrides.distinct_id
- HAVING ifNull(equals(argMax(person_distinct_id_overrides.is_deleted, person_distinct_id_overrides.version), 0), 0)
-) AS overrides ON e.distinct_id = overrides.distinct_id
-```
-
-This handles **person merges**. When two persons are merged, the `person_distinct_id_overrides` table records that a `distinct_id` now belongs to a different `person_id`.
-
-The subquery:
-
-1. **`GROUP BY distinct_id`** - One row per distinct_id
-2. **`argMax(person_id, version)`** - Get the person_id from the row with the highest version (most recent assignment)
-3. **`HAVING ifNull(equals(argMax(is_deleted, version), 0), 0)`** - Only include overrides where the latest version is not deleted
-
-The HAVING clause breakdown:
-
-- `argMax(is_deleted, version)` - Get `is_deleted` value from row with highest version
-- `equals(..., 0)` - Returns `1` if not deleted, `0` if deleted, `NULL` if NULL
-- `ifNull(..., 0)` - Convert NULL to 0 (falsy), so NULL values get filtered out
-- Result: `1` (truthy) keeps the row, `0` (falsy) filters it out
-
-The join:
-
-- `LEFT OUTER JOIN` keeps all events, even without an override
-- Then: `if(notEmpty(overrides.distinct_id), overrides.person_id, e.person_id)` uses the override if it exists
-
-### The SELECT Columns
-
-Grouped by `(person_id, property_key, operation_type)`, producing one row per person per property per operation:
-
-1. **`person_id`** - Use override if exists, else event's person_id
-2. **`key`** - Property key (`kv_tuple.2`)
-3. **`prop_type`** - `'set'` or `'set_once'` (`kv_tuple.1`)
-4. **`value`** - The winning value:
- - For `$set`: `argMaxIf(value, timestamp, ...)` → value from the **latest** event
- - For `$set_once`: `argMinIf(value, timestamp, ...)` → value from the **earliest** event
- - The `If` suffix adds a condition: only consider rows where value is not null/empty
-5. **`kv_timestamp`** - The timestamp of the winning event:
- - For `$set`: `max(timestamp)` → latest
- - For `$set_once`: `min(timestamp)` → earliest
-
-This implements the semantics: `$set` overwrites (latest wins), `$set_once` only sets if not already set (first wins).
-
----
-
-## Level 2: Group Properties into Arrays per Person
-
-```sql
-SELECT
- person_id,
- groupArray(tuple(key, value, kv_timestamp, prop_type)) AS grouped_props
-FROM (... level 1 query ...)
-GROUP BY person_id
-```
-
-**Input:** Multiple rows per person (one for each property/operation combo)
-
-| person_id | key | value | kv_timestamp | prop_type |
-| --------- | ------- | ------------ | ------------ | --------- |
-| abc | email | "[email protected]" | 2024-01-05 | set |
-| abc | plan | "pro" | 2024-01-03 | set |
-| abc | created | "2024-01-01" | 2024-01-01 | set_once |
-| xyz | email | "[email protected]" | 2024-01-04 | set |
-
-**Output:** One row per person, all properties in an array of tuples
-
-| person_id | grouped_props |
-| --------- | -------------------------------------------------------------------------------------------------------------------------------------- |
-| abc | [('email', '"[email protected]"', 2024-01-05, 'set'), ('plan', '"pro"', 2024-01-03, 'set'), ('created', '"2024-01-01"', 2024-01-01, 'set_once')] |
-| xyz | [('email', '"[email protected]"', 2024-01-04, 'set')] |
-
-`groupArray()` is ClickHouse's aggregation function that collects all values in the group into an array.
-
----
-
-## Level 3: Split Arrays by Operation Type
-
-```sql
-SELECT
- person_id,
- arrayMap(x -> x.1, arrayFilter(x -> x.4 = 'set', grouped_props)) AS set_keys,
- arrayMap(x -> x.2, arrayFilter(x -> x.4 = 'set', grouped_props)) AS set_values,
- arrayMap(x -> x.3, arrayFilter(x -> x.4 = 'set', grouped_props)) AS set_timestamps,
- arrayMap(x -> x.1, arrayFilter(x -> x.4 = 'set_once', grouped_props)) AS set_once_keys,
- arrayMap(x -> x.2, arrayFilter(x -> x.4 = 'set_once', grouped_props)) AS set_once_values,
- arrayMap(x -> x.3, arrayFilter(x -> x.4 = 'set_once', grouped_props)) AS set_once_timestamps
-FROM (... level 2 query ...)
-```
-
-Separates `$set` and `$set_once` properties into parallel arrays (keys, values, timestamps) for easier comparison in the diff step.
-
----
-
-## Level 4: Join with Current Person Properties
-
-```sql
-SELECT ...
-FROM (...) AS merged
-INNER JOIN (
- SELECT
- id,
- argMax(properties, version) as person_properties
- FROM person
- WHERE team_id = %(team_id)s
- AND _timestamp > %(bug_window_start)s
- AND _timestamp < %(bug_window_end)s
- GROUP BY id
-) AS p ON p.id = merged.person_id
-```
-
-Gets the current person properties from ClickHouse's `person` table (using `argMax` to get the latest version).
-
----
-
-## Level 5: Compute the Diff (Outermost)
-
-```sql
-SELECT
- with_person_props.person_id,
- -- For $set: only include properties where the key exists AND value differs
- arrayMap(i -> (set_keys[i], set_values[i], set_timestamps[i]), arrayFilter(
- i -> (
- indexOf(keys2, set_keys[i]) > 0
- AND set_values[i] != vals2[indexOf(keys2, set_keys[i])]
- ),
- arrayEnumerate(set_keys)
- )) AS set_diff,
- -- For $set_once: only include properties where the key does NOT exist
- arrayFilter(
- kv -> indexOf(keys2, kv.1) = 0,
- arrayMap(i -> (set_once_keys[i], set_once_values[i], set_once_timestamps[i]), arrayEnumerate(set_once_keys))
- ) AS set_once_diff
-FROM (...) AS with_person_props
-WHERE length(set_diff) > 0 OR length(set_once_diff) > 0
-```
-
-- **`set_diff`**: Properties from events where the value **differs** from current person properties
-- **`set_once_diff`**: Properties from events that are **missing** from current person properties
-
-Only returns persons where at least one property needs reconciliation.
-
----
-
-## Summary
-
-The query flows like this:
-
-1. **Extract** `$set`/`$set_once` from events, resolving merged persons
-2. **Aggregate** to find the winning value per property (latest for `$set`, first for `$set_once`)
-3. **Group** all properties per person into arrays
-4. **Join** with current person state from ClickHouse
-5. **Diff** to find properties that need updating (different values for `$set`, missing keys for `$set_once`)
-
----
-
-## Open Concerns (TODO: write tests)
-
-### 1. Deleted persons filtered in person join (FIXED)
-
-The query now filters out deleted persons using `HAVING argMax(is_deleted, version) = 0`:
-
-```sql
-INNER JOIN (
- SELECT
- id,
- argMax(properties, version) as person_properties
- FROM person
- WHERE team_id = %(team_id)s
- AND _timestamp > %(bug_window_start)s
- AND _timestamp < %(bug_window_end)s
- GROUP BY id
- -- Filter out deleted persons (latest version has is_deleted=1)
- HAVING argMax(is_deleted, version) = 0
-) AS p ON p.id = merged.person_id
-```
-
-Test: `test_deleted_person_filtered_out`
-
-### 2. Person state only from bug window - potential to overwrite newer updates
-
-The person join only selects persons modified within the bug window (`_timestamp > bug_window_start AND _timestamp < bug_window_end`).
-
-If a person received legitimate property updates **after** the bug window ended, will this reconciliation overwrite those newer updates with older values from the bug window?
-
-The reconciliation logic in `reconcile_person_properties()` does compare timestamps via `properties_last_updated_at`, but need to verify this handles the edge case correctly.
-
----
-
-## $unset Support (Implemented)
-
-The reconciliation job now supports `$unset` operations. Here's how it works:
-
-### Query changes
-
-`$unset` extraction in the ARRAY JOIN (note: `$unset` is an array of keys, not key-value pairs):
-
-```sql
--- In ARRAY JOIN arrayConcat:
-arrayMap(x -> tuple('unset', x, ''),
- JSONExtractArrayRaw(e.properties, '$unset')
-)
-```
-
-Add `unset_*` arrays in Level 3:
-
-```sql
-arrayMap(x -> x.1, arrayFilter(x -> x.4 = 'unset', grouped_props)) AS unset_keys,
-arrayMap(x -> x.3, arrayFilter(x -> x.4 = 'unset', grouped_props)) AS unset_timestamps
-```
-
-Add `unset_diff` in Level 5 (keys where unset happened and key still exists):
-
-```sql
-arrayFilter(
- kv -> indexOf(keys2, kv.1) > 0,
- arrayMap(i -> (unset_keys[i], unset_timestamps[i]), arrayEnumerate(unset_keys))
-) AS unset_diff
-```
-
-### Python resolution logic
-
-Operation semantics:
-
-- `$set_once`: "If not set, set it" - operation-level check, not a permanent property attribute
-- `$set`: "Set it to this value" - always overwrites
-- `$unset`: "Remove it" - always removes
-
-Simulate applying operations in timestamp order:
-
-```python
-def resolve_property_updates(set_diff, set_once_diff, unset_diff):
- """
- Simulate applying operations in timestamp order.
-
- - set_once: sets value only if property is currently unset
- - set: always sets value
- - unset: always removes property
- """
- results_by_key = {}
-
- # Collect all operations by key
- ops_by_key = defaultdict(list)
-
- for key, value, ts in set_diff:
- ops_by_key[key].append(('set', value, ts))
-
- for key, value, ts in set_once_diff:
- ops_by_key[key].append(('set_once', value, ts))
-
- for key, ts in unset_diff:
- ops_by_key[key].append(('unset', None, ts))
-
- for key, ops in ops_by_key.items():
- # Sort by timestamp ascending
- ops.sort(key=lambda x: x[2])
-
- # Simulate applying each operation
- current_value = None # None means unset
- current_ts = None
-
- for op_type, value, ts in ops:
- if op_type == 'set':
- current_value = value
- current_ts = ts
- elif op_type == 'set_once':
- if current_value is None:
- current_value = value
- current_ts = ts
- # else: ignore, property already has a value
- elif op_type == 'unset':
- current_value = None
- current_ts = ts
-
- results_by_key[key] = (current_value, current_ts)
-
- # Split into updates and deletions
- updates = {k: v for k, v in results_by_key.items() if v[0] is not None}
- deletions = {k for k, v in results_by_key.items() if v[0] is None}
-
- return updates, deletions
-```
-
-Example timeline for key `email`:
-
-```text
-t1: set_once "[email protected]" → value = "[email protected]"
-t2: set_once "[email protected]" → ignored (already set)
-t3: set "[email protected]" → value = "[email protected]" (set overwrites)
-t4: unset → value = None
-t5: set_once "[email protected]" → value = "[email protected]" (was unset, so set_once applies)
-```
-
-### Potential inconsistency with $unset
-
-**Problem:** If an `$unset` happened during the bug window, but a `$set` happened AFTER the bug window, this reconciliation would incorrectly delete the property.
-
-Example:
-
-```text
-t1 (bug window): $unset "email" → resolved as deletion
-t2 (after window): $set "email"="x" → not seen by reconciliation
-```
-
-The reconciliation would delete `email`, even though it was legitimately set after the bug window.
-
-**Solution:** Compare resolved operation timestamp against `properties_last_updated_at` from Postgres before applying:
-
-```python
-def reconcile_person_properties(person, resolved_updates, resolved_deletions):
- properties = dict(person["properties"] or {})
- properties_last_updated_at = dict(person["properties_last_updated_at"] or {})
-
- # Apply updates (only if our timestamp is newer)
- for key, (value, event_ts) in resolved_updates.items():
- existing_ts = properties_last_updated_at.get(key)
- if existing_ts is None or event_ts > parse_datetime(existing_ts):
- properties[key] = value
- properties_last_updated_at[key] = event_ts.isoformat()
-
- # Apply deletions (only if our timestamp is newer)
- for key, event_ts in resolved_deletions.items():
- existing_ts = properties_last_updated_at.get(key)
- if existing_ts is None or event_ts > parse_datetime(existing_ts):
- properties.pop(key, None)
- properties_last_updated_at.pop(key, None)
-
- return properties, properties_last_updated_at
-```
-
-This ensures we only apply operations that are actually newer than what's currently in Postgres, preventing overwrites of legitimate post-bug-window updates.
-
-### Critical issue: properties_last_updated_at is NOT maintained
-
-**Discovery:** `properties_last_updated_at` is only set at person creation time (`person-create-service.ts:34-42`). It is NOT updated when properties change after creation.
-
-This means:
-
-- If a property is added after person creation, it has no timestamp in `properties_last_updated_at`
-- The reconciliation logic checks `if existing_ts_str is None` and applies the update
-- This causes post-bug-window updates to be incorrectly overwritten
-
-**Test case:** `TestPostBugWindowUpdatePreservation::test_post_bug_window_update_should_not_be_overwritten`
-
-Timeline:
-
-```text
-t1: Person created WITHOUT property P
-t2: Bug window starts
-t2.5: Event sets P=V1 (missed due to bug)
-t3: Bug window ends
-t3.5: Event sets P=V2 (correctly applied)
-t4: Reconciliation runs
-```
-
-Current state in Postgres:
-
-- `properties["P"] = "V2"` (from t3.5)
-- `properties_last_updated_at["P"] = undefined` (not set because P was added after creation)
-
-Reconciliation logic:
-
-```python
-existing_ts_str = properties_last_updated_at.get("P") # None!
-if existing_ts_str is None or event_ts > parse_datetime(existing_ts_str):
- properties["P"] = "V1" # BUG: overwrites V2 with older V1
-```
-
-**This test currently FAILS** - documenting the bug that needs to be fixed.
-
-**Integration tests confirming this behavior:**
-`nodejs/src/ingestion/person-properties-metadata.e2e.test.ts`
-
-These tests verify that:
-
-For `properties_last_updated_at`:
-
-1. IS set at person creation (via `$set` or `$identify`)
-2. Is NOT updated when an existing property is changed
-3. Does NOT get a new entry when a new property is added after creation
-
-For `properties_last_operation`:
-
-1. IS set at person creation with correct operation type (`set` or `set_once`)
-2. Is NOT updated when properties change after creation
-3. Does NOT get a new entry when a new property is added after creation
diff --git a/posthog/dags/scripts/extract_reconciliation_results.py b/posthog/dags/scripts/extract_reconciliation_results.py
deleted file mode 100644
index 8a3f8f942f80..000000000000
--- a/posthog/dags/scripts/extract_reconciliation_results.py
+++ /dev/null
@@ -1,648 +0,0 @@
-#!/usr/bin/env python3
-# ruff: noqa: T201 - CLI script uses print for output
-"""
-Extract failed team IDs from person property reconciliation job runs via Dagster Cloud GraphQL API.
-
-Usage:
- export DAGSTER_CLOUD_TOKEN="<USER_TOKEN>"
- python posthog/dags/scripts/extract_reconciliation_results.py \
- --org posthog \
- --deployment <prod-us | prod-eu> \
- [--all | --limit <INT> | --since "2026-01-01" --until "2026-01-12"]
-
-Prerequisites:
- 1. Get a Dagster Cloud user token:
- - Go to Dagster Cloud UI (e.g., https://posthog.dagster.cloud/)
- - Click your avatar (top-right) > "Organization settings" > "Tokens" tab
- - Create a User Token and copy it
- - REMEMBER: revoke the token when you're done!
- 2. Set the DAGSTER_CLOUD_TOKEN environment variable
-"""
-
-import os
-import ast
-import csv
-import sys
-import json
-import argparse
-from dataclasses import dataclass
-from datetime import datetime
-from typing import Any
-
-import requests
-
-# GraphQL query to get runs for the reconciliation job
-RUNS_QUERY = """
-query GetRuns($limit: Int!, $cursor: String, $filter: RunsFilter!) {
- runsOrError(filter: $filter, limit: $limit, cursor: $cursor) {
- __typename
- ... on Runs {
- results {
- runId
- status
- startTime
- endTime
- tags {
- key
- value
- }
- }
- }
- ... on InvalidPipelineRunsFilterError {
- message
- }
- ... on PythonError {
- message
- stack
- }
- }
-}
-"""
-
-# GraphQL query to get a single run by ID
-RUN_QUERY = """
-query GetRun($runId: ID!) {
- runOrError(runId: $runId) {
- __typename
- ... on Run {
- runId
- status
- startTime
- endTime
- tags {
- key
- value
- }
- }
- ... on RunNotFoundError {
- message
- }
- ... on PythonError {
- message
- stack
- }
- }
-}
-"""
-
-# GraphQL query to get step output events for a run
-STEP_OUTPUTS_QUERY = """
-query GetStepOutputs($runId: ID!) {
- logsForRun(runId: $runId) {
- __typename
- ... on EventConnection {
- events {
- __typename
- ... on ExecutionStepOutputEvent {
- stepKey
- metadataEntries {
- __typename
- label
- ... on TextMetadataEntry {
- text
- }
- ... on JsonMetadataEntry {
- jsonString
- }
- ... on IntMetadataEntry {
- intValue
- }
- }
- }
- }
- }
- ... on PythonError {
- message
- stack
- }
- }
-}
-"""
-
-
-@dataclass
-class TeamResult:
- """Result for a single team from a reconciliation run."""
-
- team_id: int
- status: str
- persons_processed: int
- persons_updated: int
- persons_skipped: int
- error: str | None
- run_id: str
- step_key: str
-
-
-@dataclass
-class RunSummary:
- """Summary of a single reconciliation job run."""
-
- run_id: str
- status: str
- start_time: datetime | None
- end_time: datetime | None
- team_range: str | None
- teams_count: int
- teams_succeeded: int
- teams_failed: int
- failed_team_ids: list[int]
- team_results: list[TeamResult]
- persons_processed: int
- persons_updated: int
- persons_skipped: int
-
-
-class DagsterCloudClient:
- """Client for Dagster Cloud GraphQL API."""
-
- def __init__(self, org: str, deployment: str, token: str):
- self.base_url = f"https://{org}.dagster.cloud/{deployment}/graphql"
- self.headers = {
- "Dagster-Cloud-Api-Token": token,
- "Content-Type": "application/json",
- }
-
- def _execute_query(self, query: str, variables: dict[str, Any] | None = None) -> dict[str, Any]:
- """Execute a GraphQL query and return the result."""
- payload: dict[str, Any] = {"query": query}
- if variables:
- payload["variables"] = variables
-
- response = requests.post(self.base_url, headers=self.headers, json=payload, timeout=60)
- response.raise_for_status()
-
- result = response.json()
- if "errors" in result:
- raise RuntimeError(f"GraphQL errors: {result['errors']}")
-
- return result["data"]
-
- def get_runs(
- self,
- job_name: str = "person_property_reconciliation_job",
- statuses: list[str] | None = None,
- created_after: float | None = None,
- created_before: float | None = None,
- limit: int = 100,
- cursor: str | None = None,
- ) -> tuple[list[dict[str, Any]], str | None]:
- """
- Get runs for the specified job.
-
- Args:
- job_name: Name of the Dagster job
- statuses: Filter by run statuses
- created_after: Unix timestamp - only return runs created after this time
- created_before: Unix timestamp - only return runs created before this time
- limit: Max runs to return per page
- cursor: Pagination cursor
-
- Returns:
- Tuple of (runs list, next cursor for pagination)
- """
- run_filter: dict[str, Any] = {"pipelineName": job_name}
- if statuses:
- run_filter["statuses"] = statuses
- if created_after is not None:
- # Dagster API uses "updatedAfter" (no "createdAfter" exists)
- run_filter["updatedAfter"] = created_after
- if created_before is not None:
- run_filter["createdBefore"] = created_before
-
- variables = {
- "filter": run_filter,
- "limit": limit,
- "cursor": cursor,
- }
-
- data = self._execute_query(RUNS_QUERY, variables)
- runs_result = data["runsOrError"]
-
- if runs_result["__typename"] == "Runs":
- results = runs_result["results"]
- # Get cursor for next page (last run's runId if we got a full page)
- next_cursor = results[-1]["runId"] if len(results) == limit else None
- return results, next_cursor
- elif runs_result["__typename"] == "InvalidPipelineRunsFilterError":
- raise RuntimeError(f"Invalid filter: {runs_result['message']}")
- else:
- raise RuntimeError(f"Query error: {runs_result.get('message', 'Unknown error')}")
-
- def get_run(self, run_id: str) -> dict[str, Any] | None:
- """Get a single run by ID. Returns None if not found."""
- data = self._execute_query(RUN_QUERY, {"runId": run_id})
- result = data["runOrError"]
-
- if result["__typename"] == "Run":
- return result
- elif result["__typename"] == "RunNotFoundError":
- return None
- else:
- raise RuntimeError(f"Query error: {result.get('message', 'Unknown error')}")
-
- def get_step_outputs(self, run_id: str) -> list[dict[str, Any]]:
- """Get step output events for a run, filtered to reconcile_team_chunk steps."""
- data = self._execute_query(STEP_OUTPUTS_QUERY, {"runId": run_id})
- logs_result = data["logsForRun"]
-
- if logs_result["__typename"] == "PythonError":
- raise RuntimeError(f"Query error: {logs_result.get('message', 'Unknown error')}")
-
- events = logs_result.get("events", [])
-
- # Filter to only ExecutionStepOutputEvent for reconcile_team_chunk steps
- step_outputs = []
- for event in events:
- if event["__typename"] == "ExecutionStepOutputEvent":
- step_key = event.get("stepKey", "")
- # Match reconcile_team_chunk steps (they have dynamic suffixes like [teams_1_100])
- if "reconcile_team_chunk" in step_key:
- step_outputs.append(event)
-
- return step_outputs
-
-
-def parse_metadata_entries(entries: list[dict[str, Any]]) -> dict[str, Any]:
- """Parse metadata entries into a dictionary."""
- result: dict[str, Any] = {}
- for entry in entries:
- label = entry.get("label", "")
- typename = entry.get("__typename", "")
-
- if typename == "TextMetadataEntry":
- result[label] = entry.get("text", "")
- elif typename == "JsonMetadataEntry":
- json_str = entry.get("jsonString", "")
- try:
- result[label] = json.loads(json_str) if json_str else None
- except json.JSONDecodeError:
- result[label] = json_str
- elif typename == "IntMetadataEntry":
- result[label] = entry.get("intValue")
-
- return result
-
-
-def parse_failed_team_ids(failed_team_ids_str: str) -> list[int]:
- """Parse the failed_team_ids string (Python list repr) into a list of ints."""
- if not failed_team_ids_str or failed_team_ids_str == "[]":
- return []
-
- try:
- # The string is a Python list repr like "[123, 456, 789]"
- parsed = ast.literal_eval(failed_team_ids_str)
- if isinstance(parsed, list):
- return [int(x) for x in parsed]
- except (ValueError, SyntaxError):
- pass
-
- return []
-
-
-def extract_run_results(client: DagsterCloudClient, run: dict[str, Any]) -> RunSummary:
- """Extract results from a single run."""
- run_id = run["runId"]
- status = run["status"]
-
- # Parse timestamps
- start_time = None
- end_time = None
- if run.get("startTime"):
- start_time = datetime.fromtimestamp(run["startTime"])
- if run.get("endTime"):
- end_time = datetime.fromtimestamp(run["endTime"])
-
- # Extract team range from tags
- team_range = None
- for tag in run.get("tags", []):
- if tag["key"] == "reconciliation_range":
- team_range = tag["value"]
- break
-
- # Get step outputs
- step_outputs = client.get_step_outputs(run_id)
-
- all_failed_team_ids: list[int] = []
- all_team_results: list[TeamResult] = []
- total_teams_count = 0
- total_teams_succeeded = 0
- total_teams_failed = 0
- total_persons_processed = 0
- total_persons_updated = 0
- total_persons_skipped = 0
-
- for step_output in step_outputs:
- step_key = step_output.get("stepKey", "")
- metadata = parse_metadata_entries(step_output.get("metadataEntries", []))
-
- # Aggregate counts
- total_teams_count += metadata.get("teams_count", 0) or 0
- total_teams_succeeded += metadata.get("teams_succeeded", 0) or 0
- total_teams_failed += metadata.get("teams_failed", 0) or 0
-
- # Parse failed team IDs
- failed_ids_str = metadata.get("failed_team_ids", "[]")
- failed_ids = parse_failed_team_ids(failed_ids_str)
- all_failed_team_ids.extend(failed_ids)
-
- # Parse detailed team results
- teams_results = metadata.get("teams_results", [])
- if isinstance(teams_results, list):
- for team_result in teams_results:
- if isinstance(team_result, dict):
- persons_processed = team_result.get("persons_processed", 0) or 0
- persons_updated = team_result.get("persons_updated", 0) or 0
- persons_skipped = team_result.get("persons_skipped", 0) or 0
-
- total_persons_processed += persons_processed
- total_persons_updated += persons_updated
- total_persons_skipped += persons_skipped
-
- all_team_results.append(
- TeamResult(
- team_id=team_result.get("team_id", 0),
- status=team_result.get("status", "unknown"),
- persons_processed=persons_processed,
- persons_updated=persons_updated,
- persons_skipped=persons_skipped,
- error=team_result.get("error"),
- run_id=run_id,
- step_key=step_key,
- )
- )
-
- return RunSummary(
- run_id=run_id,
- status=status,
- start_time=start_time,
- end_time=end_time,
- team_range=team_range,
- teams_count=total_teams_count,
- teams_succeeded=total_teams_succeeded,
- teams_failed=total_teams_failed,
- failed_team_ids=sorted(set(all_failed_team_ids)),
- team_results=all_team_results,
- persons_processed=total_persons_processed,
- persons_updated=total_persons_updated,
- persons_skipped=total_persons_skipped,
- )
-
-
-def main():
- parser = argparse.ArgumentParser(
- description="Extract failed team IDs from person property reconciliation job runs",
- formatter_class=argparse.RawDescriptionHelpFormatter,
- epilog=__doc__,
- )
- parser.add_argument("--org", required=True, help="Dagster Cloud organization name (e.g., 'posthog')")
- parser.add_argument("--deployment", required=True, help="Dagster Cloud deployment name (e.g., 'prod')")
- parser.add_argument(
- "--token",
- default=os.environ.get("DAGSTER_CLOUD_TOKEN"),
- help="Dagster Cloud API token (defaults to DAGSTER_CLOUD_TOKEN env var)",
- )
- parser.add_argument(
- "--status",
- nargs="*",
- choices=["QUEUED", "NOT_STARTED", "STARTING", "STARTED", "SUCCESS", "FAILURE", "CANCELING", "CANCELED"],
- help="Filter runs by status (default: all statuses)",
- )
- parser.add_argument(
- "--limit",
- type=int,
- default=100,
- help="Maximum number of runs to fetch (default: 100, use --all for unlimited)",
- )
- parser.add_argument(
- "--all",
- action="store_true",
- dest="fetch_all",
- help="Fetch ALL runs (ignores --limit)",
- )
- parser.add_argument(
- "--since",
- help="Only include runs created after this date/time in local timezone (format: YYYY-MM-DD or YYYY-MM-DD HH:MM:SS)",
- )
- parser.add_argument(
- "--until",
- help="Only include runs created before this date/time in local timezone (format: YYYY-MM-DD or YYYY-MM-DD HH:MM:SS)",
- )
- parser.add_argument(
- "--run-id",
- help="Extract results from a specific run ID only",
- )
- parser.add_argument(
- "--output-csv",
- help="Output failed teams to a CSV file",
- )
- parser.add_argument(
- "--verbose",
- "-v",
- action="store_true",
- help="Show detailed output including per-team results",
- )
-
- args = parser.parse_args()
-
- if not args.token:
- print("Error: DAGSTER_CLOUD_TOKEN environment variable not set and --token not provided", file=sys.stderr)
- print("\nTo get a token:", file=sys.stderr)
- print(" 1. Go to Dagster Cloud UI (e.g., https://posthog.dagster.cloud/)", file=sys.stderr)
- print(" 2. Click your avatar (top-right) > 'Organization settings' > 'Tokens' tab", file=sys.stderr)
- print(" 3. Create and copy the token", file=sys.stderr)
- sys.exit(1)
-
- client = DagsterCloudClient(args.org, args.deployment, args.token)
-
- def parse_date_arg(value: str, arg_name: str, end_of_day: bool = False) -> float:
- """Parse a date string to unix timestamp. Assumes input is in local timezone."""
- for fmt in ["%Y-%m-%d %H:%M:%S", "%Y-%m-%d"]:
- try:
- dt = datetime.strptime(value, fmt)
- if end_of_day and fmt == "%Y-%m-%d":
- dt = dt.replace(hour=23, minute=59, second=59)
- return dt.timestamp()
- except ValueError:
- continue
- print(f"Error: Invalid {arg_name} date format: {value}", file=sys.stderr)
- print("Use ISO format: YYYY-MM-DD or YYYY-MM-DD HH:MM:SS", file=sys.stderr)
- sys.exit(1)
-
- created_after = parse_date_arg(args.since, "--since") if args.since else None
- created_before = parse_date_arg(args.until, "--until", end_of_day=True) if args.until else None
-
- all_summaries: list[RunSummary] = []
- all_team_results_by_id: dict[int, TeamResult] = {}
-
- if args.run_id:
- print(f"Fetching results for run {args.run_id}...")
- run = client.get_run(args.run_id)
- if not run:
- print(f"Error: Run {args.run_id} not found", file=sys.stderr)
- sys.exit(1)
- runs_to_process = [run]
- else:
- # Fetch runs with pagination
- filter_info = []
- if args.since:
- filter_info.append(f"since {args.since}")
- if args.until:
- filter_info.append(f"until {args.until}")
- if args.status:
- filter_info.append(f"status={args.status}")
- filter_str = f" ({', '.join(filter_info)})" if filter_info else ""
-
- limit_str = "all" if args.fetch_all else str(args.limit)
- print(f"Fetching {limit_str} runs for person_property_reconciliation_job{filter_str}...")
-
- runs_to_process = []
- cursor = None
- max_runs = float("inf") if args.fetch_all else args.limit
-
- while len(runs_to_process) < max_runs:
- batch_limit = 100 if args.fetch_all else min(100, int(max_runs) - len(runs_to_process))
- runs, cursor = client.get_runs(
- statuses=args.status,
- created_after=created_after,
- created_before=created_before,
- limit=batch_limit,
- cursor=cursor,
- )
- runs_to_process.extend(runs)
- print(f" Fetched {len(runs_to_process)} runs so far...")
- if not cursor or len(runs) == 0:
- break
-
- print(f"Found {len(runs_to_process)} runs total")
-
- # Sort runs by start time (newest first) so we process most recent results first
- runs_to_process.sort(key=lambda r: r.get("startTime") or 0, reverse=True)
-
- # Process each run
- for i, run in enumerate(runs_to_process, 1):
- run_id = run["runId"]
- print(f"Processing run {i}/{len(runs_to_process)}: {run_id} ({run['status']})...")
-
- try:
- summary = extract_run_results(client, run)
- all_summaries.append(summary)
-
- # Collect team results, keeping only the first (most recent) result per team
- for result in summary.team_results:
- if result.team_id not in all_team_results_by_id:
- all_team_results_by_id[result.team_id] = result
-
- except Exception as e:
- print(f" Error processing run {run_id}: {e}", file=sys.stderr)
-
- # Compute deduplicated aggregates (most recent result per team)
- deduped_results = list(all_team_results_by_id.values())
- deduped_teams_count = len(deduped_results)
- deduped_teams_succeeded = sum(1 for r in deduped_results if r.status == "success")
- deduped_teams_failed = sum(1 for r in deduped_results if r.status == "failed")
- deduped_persons_processed = sum(r.persons_processed for r in deduped_results)
- deduped_persons_updated = sum(r.persons_updated for r in deduped_results)
- deduped_persons_skipped = sum(r.persons_skipped for r in deduped_results)
- deduped_failed_team_ids = sorted(r.team_id for r in deduped_results if r.status == "failed")
-
- # Compute raw totals (includes retries)
- raw_teams_count = sum(s.teams_count for s in all_summaries)
-
- # Print summary
- print("\n" + "=" * 80)
- print("SUMMARY (deduplicated by team - most recent run per team)")
- print("=" * 80)
-
- print(f"Unique teams processed: {deduped_teams_count}")
- print(f"Teams succeeded: {deduped_teams_succeeded}")
- print(f"Teams failed: {deduped_teams_failed}")
- print(f"Persons processed: {deduped_persons_processed}")
- print(f"Persons updated: {deduped_persons_updated}")
- print(f"Persons skipped: {deduped_persons_skipped}")
-
- if deduped_failed_team_ids:
- print(f"\nFailed team IDs: {deduped_failed_team_ids}")
-
- print("\n" + "-" * 80)
- print("RAW TOTALS (all runs, includes retries)")
- print("-" * 80)
- print(f"Runs analyzed: {len(all_summaries)}")
- print(f"Raw team count: {raw_teams_count}")
-
- # Per-run breakdown
- if args.verbose and all_summaries:
- print("\n" + "-" * 80)
- print("PER-RUN BREAKDOWN")
- print("-" * 80)
-
- for summary in all_summaries:
- start_str = summary.start_time.strftime("%Y-%m-%d %H:%M:%S") if summary.start_time else "N/A"
- print(f"\nRun: {summary.run_id}")
- print(f" Status: {summary.status}")
- print(f" Started: {start_str}")
- print(f" Team range: {summary.team_range or 'N/A'}")
- print(
- f" Teams: {summary.teams_count} total, {summary.teams_succeeded} succeeded, {summary.teams_failed} failed"
- )
- print(
- f" Persons: {summary.persons_processed} processed, {summary.persons_updated} updated, {summary.persons_skipped} skipped"
- )
-
- if summary.failed_team_ids:
- print(f" Failed team IDs: {summary.failed_team_ids}")
-
- # Show error details for failed teams (from deduplicated results)
- failed_team_results = [r for r in deduped_results if r.status == "failed"]
- if args.verbose and failed_team_results:
- print("\n" + "-" * 80)
- print("FAILED TEAM DETAILS")
- print("-" * 80)
-
- for result in failed_team_results:
- print(f"\nTeam {result.team_id}:")
- print(f" Run: {result.run_id}")
- print(f" Step: {result.step_key}")
- print(f" Error: {result.error or 'No error message'}")
-
- # Export to CSV if requested (uses deduplicated results)
- if args.output_csv and failed_team_results:
- print(f"\nWriting failed teams to {args.output_csv}...")
- with open(args.output_csv, "w", newline="") as csvfile:
- writer = csv.DictWriter(
- csvfile,
- fieldnames=[
- "team_id",
- "status",
- "error",
- "run_id",
- "step_key",
- "persons_processed",
- "persons_updated",
- "persons_skipped",
- ],
- )
- writer.writeheader()
- for result in failed_team_results:
- writer.writerow(
- {
- "team_id": result.team_id,
- "status": result.status,
- "error": result.error or "",
- "run_id": result.run_id,
- "step_key": result.step_key,
- "persons_processed": result.persons_processed,
- "persons_updated": result.persons_updated,
- "persons_skipped": result.persons_skipped,
- }
- )
- print(f"Wrote {len(failed_team_results)} failed team records to {args.output_csv}")
-
- # Exit with error code if there were failures
- if deduped_failed_team_ids:
- sys.exit(1)
- sys.exit(0)
-
-
-if __name__ == "__main__":
- main()
diff --git a/posthog/dags/tests/test_fix_person_id_overrides.py b/posthog/dags/tests/test_fix_person_id_overrides.py
deleted file mode 100644
index 5101c3fb94c0..000000000000
--- a/posthog/dags/tests/test_fix_person_id_overrides.py
+++ /dev/null
@@ -1,318 +0,0 @@
-from datetime import datetime
-from uuid import UUID
-
-from posthog.test.base import BaseTest, ClickhouseTestMixin
-
-from posthog.clickhouse.client import sync_execute
-from posthog.dags.fix_person_id_overrides import (
- fix_person_id_overrides_job,
- get_all_distinct_ids_for_person,
- get_existing_override,
- get_person_id_from_pdi2,
- insert_override_batch,
-)
-
-
-def insert_pdi2_records(records: list[tuple[int, str, UUID, int, int]]) -> None:
- """Insert records into person_distinct_id2: (team_id, distinct_id, person_id, version, is_deleted)"""
- sync_execute(
- "INSERT INTO person_distinct_id2 (team_id, distinct_id, person_id, version, is_deleted) VALUES",
- records,
- )
-
-
-def insert_override_records(records: list[tuple[int, str, UUID, int, int]]) -> None:
- """Insert records into person_distinct_id_overrides: (team_id, distinct_id, person_id, version, is_deleted)"""
- sync_execute(
- "INSERT INTO person_distinct_id_overrides (team_id, distinct_id, person_id, version, is_deleted, _timestamp, _offset, _partition) VALUES",
- [(r[0], r[1], r[2], r[3], r[4], datetime(2025, 1, 1), 0, 0) for r in records],
- )
-
-
-def get_all_overrides(team_id: int) -> list[tuple[str, str]]:
- """Get all overrides for a team: (distinct_id, person_id)"""
- result = sync_execute(
- """
- SELECT distinct_id, argMax(person_id, version) as person_id
- FROM person_distinct_id_overrides
- WHERE team_id = %(team_id)s
- GROUP BY distinct_id
- HAVING argMax(is_deleted, version) = 0
- """,
- {"team_id": team_id},
- )
- return [(row[0], str(row[1])) for row in result]
-
-
-class TestGetPersonIdFromPdi2(ClickhouseTestMixin, BaseTest):
- def test_returns_person_id_and_version(self):
- person_id = UUID(int=1)
- insert_pdi2_records([(self.team.id, "user_123", person_id, 1, 0)])
-
- result = get_person_id_from_pdi2(self.team.id, "user_123")
-
- assert result is not None
- assert result[0] == str(person_id)
- assert result[1] == 1
-
- def test_returns_none_for_missing_distinct_id(self):
- result = get_person_id_from_pdi2(self.team.id, "nonexistent")
- assert result is None
-
- def test_returns_none_for_deleted_mapping(self):
- person_id = UUID(int=2)
- insert_pdi2_records(
- [
- (self.team.id, "deleted_user", person_id, 1, 0),
- (self.team.id, "deleted_user", person_id, 2, 1), # deleted
- ]
- )
-
- result = get_person_id_from_pdi2(self.team.id, "deleted_user")
- assert result is None
-
- def test_returns_latest_person_id_by_version(self):
- old_person_id = UUID(int=10)
- new_person_id = UUID(int=20)
- insert_pdi2_records(
- [
- (self.team.id, "merged_user", old_person_id, 1, 0),
- (self.team.id, "merged_user", new_person_id, 2, 0), # newer version
- ]
- )
-
- result = get_person_id_from_pdi2(self.team.id, "merged_user")
-
- assert result is not None
- assert result[0] == str(new_person_id)
- assert result[1] == 2
-
-
-class TestGetAllDistinctIdsForPerson(ClickhouseTestMixin, BaseTest):
- def test_returns_all_distinct_ids_for_person(self):
- person_id = UUID(int=100)
- insert_pdi2_records(
- [
- (self.team.id, "anon_id", person_id, 1, 0),
- (self.team.id, "[email protected]", person_id, 1, 0),
- (self.team.id, "phone_123", person_id, 1, 0),
- ]
- )
-
- result = get_all_distinct_ids_for_person(self.team.id, str(person_id))
-
- distinct_ids = {did for did, _ in result}
- assert distinct_ids == {"anon_id", "[email protected]", "phone_123"}
-
- def test_excludes_deleted_mappings(self):
- person_id = UUID(int=101)
- insert_pdi2_records(
- [
- (self.team.id, "active_id", person_id, 1, 0),
- (self.team.id, "deleted_id", person_id, 1, 0),
- (self.team.id, "deleted_id", person_id, 2, 1), # deleted
- ]
- )
-
- result = get_all_distinct_ids_for_person(self.team.id, str(person_id))
-
- distinct_ids = {did for did, _ in result}
- assert distinct_ids == {"active_id"}
-
- def test_returns_empty_for_nonexistent_person(self):
- result = get_all_distinct_ids_for_person(self.team.id, str(UUID(int=999)))
- assert result == []
-
-
-class TestGetExistingOverride(ClickhouseTestMixin, BaseTest):
- def test_returns_override_when_exists(self):
- person_id = UUID(int=200)
- insert_override_records([(self.team.id, "override_user", person_id, 1, 0)])
-
- result = get_existing_override(self.team.id, "override_user")
-
- assert result is not None
- assert result[0] == str(person_id)
- assert result[1] == 1
-
- def test_returns_none_when_no_override(self):
- result = get_existing_override(self.team.id, "no_override_user")
- assert result is None
-
- def test_returns_none_for_deleted_override(self):
- person_id = UUID(int=201)
- insert_override_records(
- [
- (self.team.id, "deleted_override", person_id, 1, 0),
- (self.team.id, "deleted_override", person_id, 2, 1), # deleted
- ]
- )
-
- result = get_existing_override(self.team.id, "deleted_override")
- assert result is None
-
-
-class TestInsertOverrideBatch(ClickhouseTestMixin, BaseTest):
- def test_inserts_batch_of_overrides(self):
- person_id_1 = UUID(int=300)
- person_id_2 = UUID(int=301)
-
- insert_override_batch(
- [
- (self.team.id, "batch_user_1", str(person_id_1), 1),
- (self.team.id, "batch_user_2", str(person_id_1), 1),
- (self.team.id, "batch_user_3", str(person_id_2), 2),
- ]
- )
-
- result = get_existing_override(self.team.id, "batch_user_1")
- assert result is not None
- assert result[0] == str(person_id_1)
-
- result = get_existing_override(self.team.id, "batch_user_2")
- assert result is not None
- assert result[0] == str(person_id_1)
-
- result = get_existing_override(self.team.id, "batch_user_3")
- assert result is not None
- assert result[0] == str(person_id_2)
-
- def test_handles_empty_batch(self):
- insert_override_batch([])
-
-
-class TestFixPersonIdOverridesJob(ClickhouseTestMixin, BaseTest):
- def test_inserts_overrides_for_all_distinct_ids_of_person(self):
- person_id = UUID(int=400)
- insert_pdi2_records(
- [
- (self.team.id, "anon_abc123", person_id, 1, 0),
- (self.team.id, "[email protected]", person_id, 1, 0),
- ]
- )
-
- fix_person_id_overrides_job.execute_in_process(
- run_config={
- "ops": {
- "fix_person_id_overrides_op": {
- "config": {
- "team_id": self.team.id,
- "distinct_ids": "anon_abc123",
- "dry_run": False,
- }
- }
- }
- }
- )
-
- overrides = get_all_overrides(self.team.id)
- override_dict = dict(overrides)
-
- assert "anon_abc123" in override_dict
- assert "[email protected]" in override_dict
- assert override_dict["anon_abc123"] == str(person_id)
- assert override_dict["[email protected]"] == str(person_id)
-
- def test_dry_run_does_not_insert(self):
- person_id = UUID(int=401)
- insert_pdi2_records([(self.team.id, "dry_run_user", person_id, 1, 0)])
-
- fix_person_id_overrides_job.execute_in_process(
- run_config={
- "ops": {
- "fix_person_id_overrides_op": {
- "config": {
- "team_id": self.team.id,
- "distinct_ids": "dry_run_user",
- "dry_run": True,
- }
- }
- }
- }
- )
-
- overrides = get_all_overrides(self.team.id)
- assert len(overrides) == 0
-
- def test_skips_existing_overrides(self):
- person_id = UUID(int=402)
- existing_person_id = UUID(int=403)
- insert_pdi2_records(
- [
- (self.team.id, "existing_override_user", person_id, 1, 0),
- (self.team.id, "new_user", person_id, 1, 0),
- ]
- )
- insert_override_records([(self.team.id, "existing_override_user", existing_person_id, 1, 0)])
-
- fix_person_id_overrides_job.execute_in_process(
- run_config={
- "ops": {
- "fix_person_id_overrides_op": {
- "config": {
- "team_id": self.team.id,
- "distinct_ids": "existing_override_user",
- "dry_run": False,
- }
- }
- }
- }
- )
-
- overrides = get_all_overrides(self.team.id)
- override_dict = dict(overrides)
-
- # Existing override should not be changed
- assert override_dict["existing_override_user"] == str(existing_person_id)
- # New user should have override inserted
- assert override_dict["new_user"] == str(person_id)
-
- def test_handles_multiple_distinct_ids_mapping_to_same_person(self):
- person_id = UUID(int=404)
- insert_pdi2_records(
- [
- (self.team.id, "id_a", person_id, 1, 0),
- (self.team.id, "id_b", person_id, 1, 0),
- (self.team.id, "id_c", person_id, 1, 0),
- ]
- )
-
- # Pass multiple distinct_ids that map to the same person
- fix_person_id_overrides_job.execute_in_process(
- run_config={
- "ops": {
- "fix_person_id_overrides_op": {
- "config": {
- "team_id": self.team.id,
- "distinct_ids": "id_a,id_b",
- "dry_run": False,
- }
- }
- }
- }
- )
-
- overrides = get_all_overrides(self.team.id)
- override_dict = dict(overrides)
-
- # All three should have overrides (even id_c which wasn't in the input)
- assert len(override_dict) == 3
- assert all(v == str(person_id) for v in override_dict.values())
-
- def test_handles_nonexistent_distinct_id(self):
- # Should complete without error
- result = fix_person_id_overrides_job.execute_in_process(
- run_config={
- "ops": {
- "fix_person_id_overrides_op": {
- "config": {
- "team_id": self.team.id,
- "distinct_ids": "nonexistent_id",
- "dry_run": False,
- }
- }
- }
- }
- )
-
- assert result.success
diff --git a/posthog/dags/tests/test_person_property_reconciliation.py b/posthog/dags/tests/test_person_property_reconciliation.py
deleted file mode 100644
index 732c6074b5d8..000000000000
--- a/posthog/dags/tests/test_person_property_reconciliation.py
+++ /dev/null
@@ -1,8236 +0,0 @@
-"""Tests for the person property reconciliation job.
-
-For comprehensive documentation including architecture diagrams and example configs, see:
- posthog/dags/PERSON_PROPERTY_RECONCILIATION.md
-"""
-
-import os
-import json
-from datetime import UTC, datetime, timedelta
-from typing import Any
-from uuid import UUID
-
-import pytest
-from unittest.mock import MagicMock, patch
-
-from clickhouse_driver import Client
-
-from posthog.clickhouse.cluster import ClickhouseCluster
-from posthog.dags.person_property_reconciliation import (
- FILTERED_PERSON_UPDATE_PROPERTIES,
- PersonPropertyDiffs,
- PropertyValue,
- RawPersonPropertyUpdates,
- SkipReason,
- compare_raw_updates_with_person_state,
- fetch_person_properties_from_clickhouse,
- filter_event_person_properties,
- format_ch_timestamp,
- get_affected_person_ids_from_clickhouse,
- get_person_property_updates_from_clickhouse,
- get_person_property_updates_windowed,
- get_person_property_updates_windowed_batched,
- get_raw_person_property_updates_from_clickhouse,
- merge_raw_person_property_updates,
- parse_ch_timestamp,
- query_team_ids_from_clickhouse,
- reconcile_person_properties,
- reconcile_with_concurrent_changes,
- update_person_with_version_check,
-)
-from posthog.dags.tests.conftest import isolated_clickhouse_cluster, refresh_person_from_persons_db
-
-
-class TestClickHouseResultParsing:
- """Test that ClickHouse query results are correctly parsed into PersonPropertyDiffs objects."""
-
- def test_parses_set_diff_tuples(self):
- """Test that set_diff array of tuples is correctly parsed."""
- # Simulate ClickHouse returning: (person_id, person_version, set_diff, set_once_diff, unset_diff)
- # set_diff is an array of (key, value, timestamp) tuples
- # Values are raw JSON strings that get parsed via json.loads()
- mock_rows: list[
- tuple[
- str, int, list[tuple[str, Any, datetime]], list[tuple[str, Any, datetime]], list[tuple[str, datetime]]
- ]
- ] = [
- (
- "018d1234-5678-0000-0000-000000000001", # person_id (UUID as string)
- 1, # person_version
- [ # set_diff - array of (key, raw_json_value, timestamp) tuples
- ("email", '"[email protected]"', datetime(2024, 1, 15, 12, 0, 0)),
- ("name", '"John Doe"', datetime(2024, 1, 15, 12, 30, 0)),
- ],
- [], # set_once_diff - empty
- [], # unset_diff - empty
- )
- ]
-
- with patch("posthog.dags.person_property_reconciliation.sync_execute", return_value=mock_rows):
- results = get_person_property_updates_from_clickhouse(
- team_id=1,
- bug_window_start="2024-01-01T00:00:00Z",
- )
-
- assert len(results) == 1
- person_diffs = results[0]
- assert person_diffs.person_id == "018d1234-5678-0000-0000-000000000001"
- assert person_diffs.person_version == 1
- assert len(person_diffs.set_updates) == 2
- assert len(person_diffs.set_once_updates) == 0
- assert len(person_diffs.unset_updates) == 0
-
- # Verify set updates - values are parsed from raw JSON to native types
- assert "email" in person_diffs.set_updates
- assert person_diffs.set_updates["email"].value == "[email protected]"
- assert person_diffs.set_updates["email"].timestamp == datetime(2024, 1, 15, 12, 0, 0, tzinfo=UTC)
-
- assert "name" in person_diffs.set_updates
- assert person_diffs.set_updates["name"].value == "John Doe"
- assert person_diffs.set_updates["name"].timestamp == datetime(2024, 1, 15, 12, 30, 0, tzinfo=UTC)
-
- def test_parses_set_once_diff_tuples(self):
- """Test that set_once_diff array of tuples is correctly parsed."""
- # Values are raw JSON strings that get parsed via json.loads()
- mock_rows: list[
- tuple[
- str, int, list[tuple[str, Any, datetime]], list[tuple[str, Any, datetime]], list[tuple[str, datetime]]
- ]
- ] = [
- (
- "018d1234-5678-0000-0000-000000000002",
- 1, # person_version
- [], # set_diff - empty
- [ # set_once_diff - array of (key, raw_json_value, timestamp) tuples
- ("initial_referrer", '"google.com"', datetime(2024, 1, 10, 8, 0, 0)),
- ("first_seen", '"2024-01-10"', datetime(2024, 1, 10, 8, 0, 0)),
- ],
- [], # unset_diff - empty
- )
- ]
-
- with patch("posthog.dags.person_property_reconciliation.sync_execute", return_value=mock_rows):
- results = get_person_property_updates_from_clickhouse(
- team_id=1,
- bug_window_start="2024-01-01T00:00:00Z",
- )
-
- assert len(results) == 1
- person_diffs = results[0]
- assert len(person_diffs.set_updates) == 0
- assert len(person_diffs.set_once_updates) == 2
- assert len(person_diffs.unset_updates) == 0
-
- # Verify set_once updates - values are parsed from raw JSON to native types
- assert "initial_referrer" in person_diffs.set_once_updates
- assert person_diffs.set_once_updates["initial_referrer"].value == "google.com"
- assert person_diffs.set_once_updates["initial_referrer"].timestamp == datetime(2024, 1, 10, 8, 0, 0, tzinfo=UTC)
-
- assert "first_seen" in person_diffs.set_once_updates
- assert person_diffs.set_once_updates["first_seen"].value == "2024-01-10"
- assert person_diffs.set_once_updates["first_seen"].timestamp == datetime(2024, 1, 10, 8, 0, 0, tzinfo=UTC)
-
- def test_parses_mixed_set_and_set_once(self):
- """Test parsing when both set_diff and set_once_diff have values."""
- # Values are raw JSON strings that get parsed via json.loads()
- mock_rows: list[
- tuple[
- str, int, list[tuple[str, Any, datetime]], list[tuple[str, Any, datetime]], list[tuple[str, datetime]]
- ]
- ] = [
- (
- "018d1234-5678-0000-0000-000000000003",
- 1, # person_version
- [("email", '"[email protected]"', datetime(2024, 1, 15, 12, 0, 0))], # set_diff
- [("initial_source", '"organic"', datetime(2024, 1, 10, 8, 0, 0))], # set_once_diff
- [], # unset_diff - empty
- )
- ]
-
- with patch("posthog.dags.person_property_reconciliation.sync_execute", return_value=mock_rows):
- results = get_person_property_updates_from_clickhouse(
- team_id=1,
- bug_window_start="2024-01-01T00:00:00Z",
- )
-
- assert len(results) == 1
- person_diffs = results[0]
- assert len(person_diffs.set_updates) == 1
- assert len(person_diffs.set_once_updates) == 1
- assert len(person_diffs.unset_updates) == 0
-
- # Verify set update
- assert "email" in person_diffs.set_updates
- assert person_diffs.set_updates["email"].value == "[email protected]"
-
- # Verify set_once update
- assert "initial_source" in person_diffs.set_once_updates
- assert person_diffs.set_once_updates["initial_source"].value == "organic"
-
- def test_handles_multiple_persons(self):
- """Test parsing results for multiple persons."""
- # Values are raw JSON strings that get parsed via json.loads()
- mock_rows: list[
- tuple[
- str, int, list[tuple[str, Any, datetime]], list[tuple[str, Any, datetime]], list[tuple[str, datetime]]
- ]
- ] = [
- (
- "018d1234-0000-0000-0000-000000000001",
- 1, # person_version
- [("prop1", '"val1"', datetime(2024, 1, 15, 12, 0, 0))],
- [],
- [], # unset_diff
- ),
- (
- "018d1234-0000-0000-0000-000000000002",
- 2, # person_version
- [("prop2", '"val2"', datetime(2024, 1, 15, 13, 0, 0))],
- [],
- [], # unset_diff
- ),
- (
- "018d1234-0000-0000-0000-000000000003",
- 3, # person_version
- [],
- [("prop3", '"val3"', datetime(2024, 1, 10, 8, 0, 0))],
- [], # unset_diff
- ),
- ]
-
- with patch("posthog.dags.person_property_reconciliation.sync_execute", return_value=mock_rows):
- results = get_person_property_updates_from_clickhouse(
- team_id=1,
- bug_window_start="2024-01-01T00:00:00Z",
- )
-
- assert len(results) == 3
-
- # First person has set update
- assert results[0].person_id == "018d1234-0000-0000-0000-000000000001"
- assert results[0].set_updates["prop1"].value == "val1"
-
- # Second person has set update
- assert results[1].person_id == "018d1234-0000-0000-0000-000000000002"
- assert results[1].set_updates["prop2"].value == "val2"
-
- # Third person has set_once update
- assert results[2].person_id == "018d1234-0000-0000-0000-000000000003"
- assert results[2].set_once_updates["prop3"].value == "val3"
-
- def test_skips_persons_with_no_updates(self):
- """Test that persons with empty set_diff, set_once_diff, and unset_diff are skipped."""
- mock_rows: list[
- tuple[
- str, int, list[tuple[str, Any, datetime]], list[tuple[str, Any, datetime]], list[tuple[str, datetime]]
- ]
- ] = [
- (
- "018d1234-0000-0000-0000-000000000001",
- 1, # person_version
- [], # empty set_diff
- [], # empty set_once_diff
- [], # empty unset_diff
- ),
- ]
-
- with patch("posthog.dags.person_property_reconciliation.sync_execute", return_value=mock_rows):
- results = get_person_property_updates_from_clickhouse(
- team_id=1,
- bug_window_start="2024-01-01T00:00:00Z",
- )
-
- # Person with no updates should be filtered out
- assert len(results) == 0
-
- def test_handles_empty_results(self):
- """Test handling when ClickHouse returns no rows."""
- with patch("posthog.dags.person_property_reconciliation.sync_execute", return_value=[]):
- results = get_person_property_updates_from_clickhouse(
- team_id=1,
- bug_window_start="2024-01-01T00:00:00Z",
- )
-
- assert results == []
-
- def test_parses_raw_json_value_types(self):
- """Test that raw JSON values from CH are correctly parsed to Python types."""
- # Raw JSON representations are parsed via json.loads() to native Python types:
- # - strings: "hello" -> "hello"
- # - numbers: 123, 3.14 -> int, float
- # - booleans: true, false -> True, False
- # - null -> None
- mock_rows: list[
- tuple[
- str, int, list[tuple[str, Any, datetime]], list[tuple[str, Any, datetime]], list[tuple[str, datetime]]
- ]
- ] = [
- (
- "018d1234-5678-0000-0000-000000000001",
- 1, # person_version
- [
- ("string_prop", '"hello"', datetime(2024, 1, 15, 12, 0, 0)),
- ("int_prop", "123", datetime(2024, 1, 15, 12, 0, 0)),
- ("float_prop", "3.14", datetime(2024, 1, 15, 12, 0, 0)),
- ("bool_true_prop", "true", datetime(2024, 1, 15, 12, 0, 0)),
- ("bool_false_prop", "false", datetime(2024, 1, 15, 12, 0, 0)),
- ("null_prop", "null", datetime(2024, 1, 15, 12, 0, 0)),
- ],
- [], # set_once_diff
- [], # unset_diff
- )
- ]
-
- with patch("posthog.dags.person_property_reconciliation.sync_execute", return_value=mock_rows):
- results = get_person_property_updates_from_clickhouse(
- team_id=1,
- bug_window_start="2024-01-01T00:00:00Z",
- )
-
- assert len(results) == 1
- person_diffs = results[0]
- assert len(person_diffs.set_updates) == 6
-
- # String - parsed from JSON string
- assert person_diffs.set_updates["string_prop"].value == "hello"
- assert isinstance(person_diffs.set_updates["string_prop"].value, str)
-
- # Integer
- assert person_diffs.set_updates["int_prop"].value == 123
- assert isinstance(person_diffs.set_updates["int_prop"].value, int)
-
- # Float
- assert person_diffs.set_updates["float_prop"].value == 3.14
- assert isinstance(person_diffs.set_updates["float_prop"].value, float)
-
- # Boolean true
- assert person_diffs.set_updates["bool_true_prop"].value is True
-
- # Boolean false
- assert person_diffs.set_updates["bool_false_prop"].value is False
-
- # Null
- assert person_diffs.set_updates["null_prop"].value is None
-
- def test_parses_unset_diff_tuples(self):
- """Test that unset_diff array of tuples is correctly parsed."""
- # unset_diff is an array of (key, timestamp) tuples
- # Keys are already parsed in the query with JSON_VALUE (consistent with $set/$set_once)
- mock_rows: list[
- tuple[
- str, int, list[tuple[str, Any, datetime]], list[tuple[str, Any, datetime]], list[tuple[str, datetime]]
- ]
- ] = [
- (
- "018d1234-5678-0000-0000-000000000001",
- 1, # person_version
- [], # set_diff - empty
- [], # set_once_diff - empty
- [ # unset_diff - array of (key, timestamp) tuples
- ("email", datetime(2024, 1, 15, 12, 0, 0)),
- ("old_property", datetime(2024, 1, 15, 12, 30, 0)),
- ],
- )
- ]
-
- with patch("posthog.dags.person_property_reconciliation.sync_execute", return_value=mock_rows):
- results = get_person_property_updates_from_clickhouse(
- team_id=1,
- bug_window_start="2024-01-01T00:00:00Z",
- )
-
- assert len(results) == 1
- person_diffs = results[0]
- assert len(person_diffs.set_updates) == 0
- assert len(person_diffs.set_once_updates) == 0
- assert len(person_diffs.unset_updates) == 2
-
- # Verify first unset
- assert "email" in person_diffs.unset_updates
- assert person_diffs.unset_updates["email"].value is None
- assert person_diffs.unset_updates["email"].timestamp == datetime(2024, 1, 15, 12, 0, 0, tzinfo=UTC)
-
- # Verify second unset
- assert "old_property" in person_diffs.unset_updates
- assert person_diffs.unset_updates["old_property"].value is None
- assert person_diffs.unset_updates["old_property"].timestamp == datetime(2024, 1, 15, 12, 30, 0, tzinfo=UTC)
-
- def test_parses_mixed_set_set_once_and_unset(self):
- """Test parsing when all three operation types have values."""
- mock_rows: list[
- tuple[
- str, int, list[tuple[str, Any, datetime]], list[tuple[str, Any, datetime]], list[tuple[str, datetime]]
- ]
- ] = [
- (
- "018d1234-5678-0000-0000-000000000004",
- 1, # person_version
- [("email", '"[email protected]"', datetime(2024, 1, 15, 12, 0, 0))], # set_diff
- [("initial_source", '"organic"', datetime(2024, 1, 10, 8, 0, 0))], # set_once_diff
- [("old_field", datetime(2024, 1, 14, 10, 0, 0))], # unset_diff - keys are already parsed
- )
- ]
-
- with patch("posthog.dags.person_property_reconciliation.sync_execute", return_value=mock_rows):
- results = get_person_property_updates_from_clickhouse(
- team_id=1,
- bug_window_start="2024-01-01T00:00:00Z",
- )
-
- assert len(results) == 1
- person_diffs = results[0]
- assert len(person_diffs.set_updates) == 1
- assert len(person_diffs.set_once_updates) == 1
- assert len(person_diffs.unset_updates) == 1
-
- # Verify set update
- assert "email" in person_diffs.set_updates
- assert person_diffs.set_updates["email"].value == "[email protected]"
-
- # Verify set_once update
- assert "initial_source" in person_diffs.set_once_updates
- assert person_diffs.set_once_updates["initial_source"].value == "organic"
-
- # Verify unset update
- assert "old_field" in person_diffs.unset_updates
- assert person_diffs.unset_updates["old_field"].value is None
-
- def test_no_bug_window_end_parameter_regression(self):
- """Regression test: get_person_property_updates_from_clickhouse should NOT accept bug_window_end.
-
- The function uses ClickHouse's now() for the upper bound, ensuring all events
- from bug_window_start until the current time are included. This is intentional:
- - bug_window_start/end is used to FIND teams with affected events
- - When processing a team, we read ALL events from bug_window_start to now()
-
- This prevents missing events that occurred after the initial bug window discovery.
- """
- import inspect
-
- sig = inspect.signature(get_person_property_updates_from_clickhouse)
- param_names = list(sig.parameters.keys())
-
- assert "bug_window_end" not in param_names, (
- "bug_window_end should not be a parameter. The function should use ClickHouse's now() for the upper bound."
- )
- assert "team_id" in param_names
- assert "bug_window_start" in param_names
-
-
-class TestReconcilePersonProperties:
- """Test the reconcile_person_properties function."""
-
- def test_set_creates_new_property(self):
- """Test that $set creates a property that doesn't exist in PG."""
- person = {
- "uuid": "018d1234-5678-0000-0000-000000000001",
- "properties": {},
- "properties_last_updated_at": {},
- "properties_last_operation": {},
- }
- person_diffs = PersonPropertyDiffs(
- person_id="018d1234-5678-0000-0000-000000000001",
- person_version=1,
- set_updates={"email": PropertyValue(timestamp=datetime(2024, 1, 15, 12, 0, 0), value="[email protected]")},
- set_once_updates={},
- unset_updates={},
- )
-
- result = reconcile_person_properties(person, person_diffs)
-
- assert result is not None
- assert result["properties"]["email"] == "[email protected]"
- assert "email" in result["properties_last_updated_at"]
- assert result["properties_last_operation"]["email"] == "set"
-
- def test_set_always_applies(self):
- """Test that $set always applies (CH query pre-filters to only return diffs)."""
- person = {
- "uuid": "018d1234-5678-0000-0000-000000000001",
- "properties": {"email": "[email protected]"},
- "properties_last_updated_at": {"email": "2024-01-10T00:00:00+00:00"},
- "properties_last_operation": {"email": "set"},
- }
- person_diffs = PersonPropertyDiffs(
- person_id="018d1234-5678-0000-0000-000000000001",
- person_version=1,
- set_updates={
- "email": PropertyValue(timestamp=datetime(2024, 1, 15, 12, 0, 0, tzinfo=UTC), value="[email protected]")
- },
- set_once_updates={},
- unset_updates={},
- )
-
- result = reconcile_person_properties(person, person_diffs)
-
- assert result is not None
- assert result["properties"]["email"] == "[email protected]"
-
- def test_set_once_creates_missing_property(self):
- """Test that $set_once creates a property that doesn't exist in PG."""
- person = {
- "uuid": "018d1234-5678-0000-0000-000000000001",
- "properties": {"other_prop": "value"},
- "properties_last_updated_at": {},
- "properties_last_operation": {},
- }
- person_diffs = PersonPropertyDiffs(
- person_id="018d1234-5678-0000-0000-000000000001",
- person_version=1,
- set_updates={},
- set_once_updates={
- "initial_referrer": PropertyValue(timestamp=datetime(2024, 1, 10, 8, 0, 0), value="google.com")
- },
- unset_updates={},
- )
-
- result = reconcile_person_properties(person, person_diffs)
-
- assert result is not None
- assert result["properties"]["initial_referrer"] == "google.com"
- assert result["properties_last_operation"]["initial_referrer"] == "set_once"
-
- def test_set_once_skips_existing_property(self):
- """Test that $set_once doesn't overwrite an existing property."""
- person = {
- "uuid": "018d1234-5678-0000-0000-000000000001",
- "properties": {"initial_referrer": "facebook.com"},
- "properties_last_updated_at": {"initial_referrer": "2024-01-05T00:00:00+00:00"},
- "properties_last_operation": {"initial_referrer": "set_once"},
- }
- person_diffs = PersonPropertyDiffs(
- person_id="018d1234-5678-0000-0000-000000000001",
- person_version=1,
- set_updates={},
- set_once_updates={
- "initial_referrer": PropertyValue(timestamp=datetime(2024, 1, 10, 8, 0, 0), value="google.com")
- },
- unset_updates={},
- )
-
- result = reconcile_person_properties(person, person_diffs)
-
- # No changes - property already exists
- assert result is None
-
- def test_handles_multiple_updates(self):
- """Test processing multiple updates for different properties."""
- person = {
- "uuid": "018d1234-5678-0000-0000-000000000001",
- "properties": {"existing": "value"},
- "properties_last_updated_at": {},
- "properties_last_operation": {},
- }
- person_diffs = PersonPropertyDiffs(
- person_id="018d1234-5678-0000-0000-000000000001",
- person_version=1,
- set_updates={
- "prop1": PropertyValue(timestamp=datetime(2024, 1, 15, 12, 0, 0), value="val1"),
- "prop2": PropertyValue(timestamp=datetime(2024, 1, 15, 12, 0, 0), value="val2"),
- },
- set_once_updates={
- "prop3": PropertyValue(timestamp=datetime(2024, 1, 10, 8, 0, 0), value="val3"),
- },
- unset_updates={},
- )
-
- result = reconcile_person_properties(person, person_diffs)
-
- assert result is not None
- assert result["properties"]["prop1"] == "val1"
- assert result["properties"]["prop2"] == "val2"
- assert result["properties"]["prop3"] == "val3"
- # Original property preserved
- assert result["properties"]["existing"] == "value"
-
- def test_handles_null_properties(self):
- """Test handling when person has None for properties fields."""
- person = {
- "uuid": "018d1234-5678-0000-0000-000000000001",
- "properties": None,
- "properties_last_updated_at": None,
- "properties_last_operation": None,
- }
- person_diffs = PersonPropertyDiffs(
- person_id="018d1234-5678-0000-0000-000000000001",
- person_version=1,
- set_updates={"email": PropertyValue(timestamp=datetime(2024, 1, 15, 12, 0, 0), value="[email protected]")},
- set_once_updates={},
- unset_updates={},
- )
-
- result = reconcile_person_properties(person, person_diffs)
-
- assert result is not None
- assert result["properties"]["email"] == "[email protected]"
-
- def test_returns_none_when_empty_diffs(self):
- """Test that None is returned when diffs are empty."""
- person = {
- "uuid": "018d1234-5678-0000-0000-000000000001",
- "properties": {"email": "[email protected]"},
- "properties_last_updated_at": {"email": "2024-01-20T00:00:00+00:00"},
- "properties_last_operation": {"email": "set"},
- }
- person_diffs = PersonPropertyDiffs(
- person_id="018d1234-5678-0000-0000-000000000001",
- person_version=1,
- set_updates={},
- set_once_updates={},
- unset_updates={},
- )
-
- result = reconcile_person_properties(person, person_diffs)
- assert result is None
-
- def test_unset_removes_existing_property(self):
- """Test that $unset removes a property."""
- person = {
- "uuid": "018d1234-5678-0000-0000-000000000001",
- "properties": {"email": "[email protected]", "name": "Test User"},
- "properties_last_updated_at": {"email": "2024-01-10T00:00:00+00:00", "name": "2024-01-10T00:00:00+00:00"},
- "properties_last_operation": {"email": "set", "name": "set"},
- }
- person_diffs = PersonPropertyDiffs(
- person_id="018d1234-5678-0000-0000-000000000001",
- person_version=1,
- set_updates={},
- set_once_updates={},
- unset_updates={"email": PropertyValue(timestamp=datetime(2024, 1, 15, 12, 0, 0, tzinfo=UTC), value=None)},
- )
-
- result = reconcile_person_properties(person, person_diffs)
-
- assert result is not None
- assert "email" not in result["properties"]
- assert "email" not in result["properties_last_updated_at"]
- assert "email" not in result["properties_last_operation"]
- # Other properties should remain unchanged
- assert result["properties"]["name"] == "Test User"
-
- def test_unset_marks_changed_even_when_property_not_exists(self):
- """Test that $unset marks changed even when property doesn't exist in properties map."""
- person = {
- "uuid": "018d1234-5678-0000-0000-000000000001",
- "properties": {"name": "Test User"},
- "properties_last_updated_at": {"name": "2024-01-10T00:00:00+00:00"},
- "properties_last_operation": {"name": "set"},
- }
- person_diffs = PersonPropertyDiffs(
- person_id="018d1234-5678-0000-0000-000000000001",
- person_version=1,
- set_updates={},
- set_once_updates={},
- unset_updates={"email": PropertyValue(timestamp=datetime(2024, 1, 15, 12, 0, 0, tzinfo=UTC), value=None)},
- )
-
- result = reconcile_person_properties(person, person_diffs)
-
- # Unset always marks changed (even if key wasn't in properties)
- assert result is not None
-
- def test_unset_removes_property_with_no_timestamp(self):
- """Test that $unset removes property when there's no existing timestamp."""
- person = {
- "uuid": "018d1234-5678-0000-0000-000000000001",
- "properties": {"email": "[email protected]"},
- "properties_last_updated_at": {}, # No timestamp for email
- "properties_last_operation": {},
- }
- person_diffs = PersonPropertyDiffs(
- person_id="018d1234-5678-0000-0000-000000000001",
- person_version=1,
- set_updates={},
- set_once_updates={},
- unset_updates={"email": PropertyValue(timestamp=datetime(2024, 1, 15, 12, 0, 0, tzinfo=UTC), value=None)},
- )
-
- result = reconcile_person_properties(person, person_diffs)
-
- assert result is not None
- assert "email" not in result["properties"]
-
-
-class TestUpdatePersonWithVersionCheck:
- """Test the update_person_with_version_check function."""
-
- def create_mock_cursor(self, person_data=None, update_success=True):
- """Create a mock cursor with configurable behavior."""
- cursor = MagicMock()
-
- # Mock fetchone to return person data
- cursor.fetchone.return_value = person_data
-
- # Mock rowcount for UPDATE success/failure
- cursor.rowcount = 1 if update_success else 0
-
- return cursor
-
- def test_successful_update(self):
- """Test successful update with version check."""
- person_data = {
- "id": 123,
- "uuid": "018d1234-5678-0000-0000-000000000001",
- "properties": {"existing": "value"},
- "properties_last_updated_at": {},
- "properties_last_operation": {},
- "version": 5,
- "is_identified": True,
- "created_at": datetime(2024, 1, 1, 0, 0, 0),
- }
- cursor = self.create_mock_cursor(person_data=person_data, update_success=True)
-
- person_diffs = PersonPropertyDiffs(
- person_id="018d1234-5678-0000-0000-000000000001",
- person_version=5,
- set_updates={"email": PropertyValue(timestamp=datetime(2024, 1, 15, 12, 0, 0), value="[email protected]")},
- set_once_updates={},
- unset_updates={},
- )
-
- result = update_person_with_version_check(
- cursor=cursor,
- job_id="test-job-id",
- team_id=1,
- person_uuid="018d1234-5678-0000-0000-000000000001",
- person_property_diffs=person_diffs,
- dry_run=False,
- )
-
- assert result.success is True
- assert result.updated_person_data is not None
- assert result.updated_person_data["version"] == 6 # version incremented
- assert result.updated_person_data["properties"]["email"] == "[email protected]"
- assert result.updated_person_data["properties"]["existing"] == "value"
-
- # Verify UPDATE was executed
- update_calls = [call for call in cursor.execute.call_args_list if "UPDATE posthog_person" in str(call)]
- assert len(update_calls) == 1
-
- # Verify backup INSERT was executed
- backup_calls = [
- call
- for call in cursor.execute.call_args_list
- if "INSERT INTO posthog_person_reconciliation_backup" in str(call)
- ]
- assert len(backup_calls) == 1
-
- # Verify backup contains correct data
- backup_call_args = backup_calls[0][0] # (query, params)
- backup_params = backup_call_args[1]
- assert backup_params[0] == "test-job-id" # job_id
- assert backup_params[1] == 1 # team_id
- assert backup_params[2] == 123 # person_id
- assert backup_params[3] == "018d1234-5678-0000-0000-000000000001" # uuid
- # Before state
- assert '"existing": "value"' in backup_params[4] # properties (before)
- assert backup_params[7] == 5 # version (before)
- # After state
- assert '"email": "[email protected]"' in backup_params[12] # properties_after
- assert backup_params[15] == 6 # version_after
-
- def test_dry_run_does_not_write(self):
- """Test that dry_run=True doesn't execute UPDATE."""
- person_data = {
- "id": 123,
- "uuid": "018d1234-5678-0000-0000-000000000001",
- "properties": {},
- "properties_last_updated_at": {},
- "properties_last_operation": {},
- "version": 1,
- "is_identified": False,
- "created_at": datetime(2024, 1, 1, 0, 0, 0),
- }
- cursor = self.create_mock_cursor(person_data=person_data)
-
- person_diffs = PersonPropertyDiffs(
- person_id="018d1234-5678-0000-0000-000000000001",
- person_version=1,
- set_updates={"email": PropertyValue(timestamp=datetime(2024, 1, 15, 12, 0, 0), value="[email protected]")},
- set_once_updates={},
- unset_updates={},
- )
-
- result = update_person_with_version_check(
- cursor=cursor,
- job_id="test-job-id",
- team_id=1,
- person_uuid="018d1234-5678-0000-0000-000000000001",
- person_property_diffs=person_diffs,
- dry_run=True,
- )
-
- assert result.success is True
- assert result.updated_person_data is None # No data returned for Kafka in dry run
-
- # Verify UPDATE was NOT executed
- update_calls = [call for call in cursor.execute.call_args_list if "UPDATE posthog_person" in str(call)]
- assert len(update_calls) == 0
-
- # Verify backup INSERT was STILL executed (for audit trail)
- backup_calls = [
- call
- for call in cursor.execute.call_args_list
- if "INSERT INTO posthog_person_reconciliation_backup" in str(call)
- ]
- assert len(backup_calls) == 1
-
- def test_person_not_found(self):
- """Test handling when person doesn't exist in PG."""
- cursor = self.create_mock_cursor(person_data=None)
-
- person_diffs = PersonPropertyDiffs(
- person_id="018d1234-5678-0000-0000-nonexistent",
- person_version=1,
- set_updates={"email": PropertyValue(timestamp=datetime(2024, 1, 15, 12, 0, 0), value="[email protected]")},
- set_once_updates={},
- unset_updates={},
- )
-
- result = update_person_with_version_check(
- cursor=cursor,
- job_id="test-job-id",
- team_id=1,
- person_uuid="018d1234-5678-0000-0000-nonexistent",
- person_property_diffs=person_diffs,
- )
-
- assert result.success is False
- assert result.updated_person_data is None
- assert result.skip_reason == SkipReason.NOT_FOUND
-
- @patch("posthog.dags.person_property_reconciliation.fetch_person_properties_from_clickhouse")
- def test_version_mismatch_retry(self, mock_fetch_ch_properties):
- """Test retry on version mismatch (concurrent modification).
-
- When the first UPDATE fails due to version mismatch:
- 1. On retry, if Postgres version differs from CH version (person_version in diffs)
- 2. We fetch CH properties for 3-way merge
- 3. Apply changes and retry UPDATE with new target version
- """
- person_data_v1 = {
- "id": 123,
- "uuid": "018d1234-5678-0000-0000-000000000001",
- "properties": {},
- "properties_last_updated_at": {},
- "properties_last_operation": {},
- "version": 1,
- "is_identified": False,
- "created_at": datetime(2024, 1, 1, 0, 0, 0),
- }
- person_data_v2 = {
- "id": 123,
- "uuid": "018d1234-5678-0000-0000-000000000001",
- "properties": {"other": "concurrent_change"}, # Concurrent change to Postgres
- "properties_last_updated_at": {"other": "2024-01-14T00:00:00"},
- "properties_last_operation": {"other": "set"},
- "version": 2, # Version changed by concurrent update
- "is_identified": False,
- "created_at": datetime(2024, 1, 1, 0, 0, 0),
- }
-
- # CH properties at version 1 (baseline for 3-way merge)
- mock_fetch_ch_properties.return_value = {}
-
- cursor = MagicMock()
- # First fetch returns v1, second fetch returns v2
- cursor.fetchone.side_effect = [person_data_v1, person_data_v2]
- # First update fails (version mismatch), second succeeds
- cursor.rowcount = 0 # Will be set by side_effect
-
- update_attempt = [0]
-
- def execute_side_effect(query, *args):
- if "UPDATE posthog_person" in query:
- update_attempt[0] += 1
- # First attempt fails, second succeeds
- cursor.rowcount = 1 if update_attempt[0] > 1 else 0
-
- cursor.execute.side_effect = execute_side_effect
-
- person_diffs = PersonPropertyDiffs(
- person_id="018d1234-5678-0000-0000-000000000001",
- person_version=1,
- set_updates={"email": PropertyValue(timestamp=datetime(2024, 1, 15, 12, 0, 0), value="[email protected]")},
- set_once_updates={},
- unset_updates={},
- )
-
- result = update_person_with_version_check(
- cursor=cursor,
- job_id="test-job-id",
- team_id=1,
- person_uuid="018d1234-5678-0000-0000-000000000001",
- person_property_diffs=person_diffs,
- max_retries=3,
- )
-
- assert result.success is True
- assert result.updated_person_data is not None
- assert result.updated_person_data["version"] == 3 # v2 + 1
- # Concurrent change should be preserved, our change should be applied
- assert result.updated_person_data["properties"]["other"] == "concurrent_change"
- assert result.updated_person_data["properties"]["email"] == "[email protected]"
-
- def test_exhausted_retries(self):
- """Test failure after exhausting all retries."""
- person_data = {
- "id": 123,
- "uuid": "018d1234-5678-0000-0000-000000000001",
- "properties": {},
- "properties_last_updated_at": {},
- "properties_last_operation": {},
- "version": 1,
- "is_identified": False,
- "created_at": datetime(2024, 1, 1, 0, 0, 0),
- }
-
- cursor = MagicMock()
- cursor.fetchone.return_value = person_data
- cursor.rowcount = 0 # Always fail version check
-
- person_diffs = PersonPropertyDiffs(
- person_id="018d1234-5678-0000-0000-000000000001",
- person_version=1,
- set_updates={"email": PropertyValue(timestamp=datetime(2024, 1, 15, 12, 0, 0), value="[email protected]")},
- set_once_updates={},
- unset_updates={},
- )
-
- result = update_person_with_version_check(
- cursor=cursor,
- job_id="test-job-id",
- team_id=1,
- person_uuid="018d1234-5678-0000-0000-000000000001",
- person_property_diffs=person_diffs,
- max_retries=3,
- )
-
- assert result.success is False
- assert result.updated_person_data is None
- assert result.skip_reason == SkipReason.VERSION_CONFLICT
-
-
-class TestBatchCommits:
- """Test that batch commits work correctly with different batch sizes.
-
- These tests use the process_persons_in_batches helper function directly,
- which is separated from the Dagster op for testability.
- """
-
- @pytest.mark.parametrize(
- "num_persons,batch_size,expected_commits",
- [
- # 5 persons, batch_size=2: batches of [2, 2, 1] = 3 commits
- (5, 2, 3),
- # 4 persons, batch_size=2: batches of [2, 2] = 2 commits (no partial)
- (4, 2, 2),
- # 3 persons, batch_size=5: batches of [3] = 1 commit (all in one partial batch)
- (3, 5, 1),
- # 1 person, batch_size=100: batches of [1] = 1 commit
- (1, 100, 1),
- # 6 persons, batch_size=0 (disabled): single commit at end
- (6, 0, 1),
- ],
- )
- def test_batch_commits_correct_number_of_times(self, num_persons: int, batch_size: int, expected_commits: int):
- """
- Test that commits happen the correct number of times based on batch_size.
-
- With batch_size=2 and 5 persons:
- - Batch 1: persons 0,1 → commit
- - Batch 2: persons 2,3 → commit
- - Batch 3: person 4 → commit (partial final batch)
- Total: 3 commits
- """
- from posthog.dags.person_property_reconciliation import process_persons_in_batches
-
- # Create mock persons with updates (timestamps must be timezone-aware)
- person_diffs_list = []
- for i in range(num_persons):
- person_diffs_list.append(
- PersonPropertyDiffs(
- person_id=f"person-uuid-{i}",
- person_version=1,
- set_updates={
- "prop": PropertyValue(timestamp=datetime(2024, 1, 15, 12, 0, 0, tzinfo=UTC), value=f"value_{i}")
- },
- set_once_updates={},
- unset_updates={},
- )
- )
-
- # Mock cursor
- mock_cursor = MagicMock()
- mock_cursor.fetchone.return_value = {
- "id": 123,
- "uuid": "test-uuid",
- "properties": {"prop": "old_value"},
- "properties_last_updated_at": {"prop": "2024-01-01T00:00:00+00:00"},
- "properties_last_operation": {"prop": "set"},
- "version": 1,
- "is_identified": False,
- "created_at": datetime(2024, 1, 1, 0, 0, 0),
- }
- mock_cursor.rowcount = 1 # UPDATE succeeds
-
- # Track commits
- commit_count = [0]
-
- def mock_commit():
- commit_count[0] += 1
-
- # Track batch callbacks
- batch_callbacks = []
-
- def on_batch_committed(batch_num, batch_persons):
- batch_callbacks.append((batch_num, len(batch_persons)))
-
- # Call the helper function directly
- result = process_persons_in_batches(
- person_property_diffs=person_diffs_list,
- cursor=mock_cursor,
- job_id="test-job-id",
- team_id=1,
- batch_size=batch_size,
- dry_run=False,
- backup_enabled=False,
- commit_fn=mock_commit,
- on_batch_committed=on_batch_committed,
- )
-
- # Verify commit was called the expected number of times
- assert result.total_commits == expected_commits, (
- f"Expected {expected_commits} commits for {num_persons} persons with batch_size={batch_size}, "
- f"got {result.total_commits}"
- )
-
- # Verify all persons were processed
- assert result.total_processed == num_persons
- assert result.total_updated == num_persons
-
- # Verify batch callback was called for each batch
- assert len(batch_callbacks) == expected_commits
-
- def test_batch_commits_partial_final_batch_all_persons_updated(self):
- """
- Specifically test that the partial final batch commits correctly
- and all persons are updated (not just full batches).
-
- 5 persons with batch_size=2:
- - Batch 1: persons 0,1 → commit ✓
- - Batch 2: persons 2,3 → commit ✓
- - Batch 3: person 4 → commit ✓ (THIS is the partial batch we're testing)
- """
- from posthog.dags.person_property_reconciliation import process_persons_in_batches
-
- num_persons = 5
- batch_size = 2
-
- person_uuids = [f"uuid-{i}" for i in range(num_persons)]
- person_diffs_list = [
- PersonPropertyDiffs(
- person_id=uuid,
- person_version=1,
- set_updates={
- "email": PropertyValue(
- timestamp=datetime(2024, 1, 15, 12, 0, 0, tzinfo=UTC), value=f"user{i}@example.com"
- )
- },
- set_once_updates={},
- unset_updates={},
- )
- for i, uuid in enumerate(person_uuids)
- ]
-
- mock_cursor = MagicMock()
- mock_cursor.fetchone.return_value = {
- "id": 123,
- "uuid": "test-uuid",
- "properties": {"email": "[email protected]"},
- "properties_last_updated_at": {"email": "2024-01-01T00:00:00+00:00"},
- "properties_last_operation": {"email": "set"},
- "version": 1,
- "is_identified": False,
- "created_at": datetime(2024, 1, 1, 0, 0, 0),
- }
- mock_cursor.rowcount = 1
-
- commit_count = [0]
-
- def mock_commit():
- commit_count[0] += 1
-
- # Track batch sizes to verify partial final batch
- batch_sizes = []
-
- def on_batch_committed(batch_num, batch_persons):
- batch_sizes.append(len(batch_persons))
-
- result = process_persons_in_batches(
- person_property_diffs=person_diffs_list,
- cursor=mock_cursor,
- job_id="test-job-id",
- team_id=1,
- batch_size=batch_size,
- dry_run=False,
- backup_enabled=False,
- commit_fn=mock_commit,
- on_batch_committed=on_batch_committed,
- )
-
- # All 5 persons should be processed and updated
- assert result.total_processed == 5
- assert result.total_updated == 5
- assert result.total_skipped == 0
-
- # 3 commits: batch 1 (2), batch 2 (2), batch 3 (1)
- assert result.total_commits == 3
-
- # Verify batch sizes: [2, 2, 1]
- assert batch_sizes == [2, 2, 1], f"Expected batches [2, 2, 1], got {batch_sizes}"
-
- def test_empty_person_list_no_commits(self):
- """Test that empty person list results in zero commits."""
- from posthog.dags.person_property_reconciliation import process_persons_in_batches
-
- mock_cursor = MagicMock()
- commit_count = [0]
-
- def mock_commit():
- commit_count[0] += 1
-
- result = process_persons_in_batches(
- person_property_diffs=[],
- cursor=mock_cursor,
- job_id="test-job-id",
- team_id=1,
- batch_size=10,
- dry_run=False,
- backup_enabled=False,
- commit_fn=mock_commit,
- )
-
- assert result.total_processed == 0
- assert result.total_commits == 0
- assert commit_count[0] == 0
-
- def test_dry_run_no_commits(self):
- """Test that dry_run=True doesn't commit when there are no backups."""
- from posthog.dags.person_property_reconciliation import process_persons_in_batches
-
- person_diffs_list = [
- PersonPropertyDiffs(
- person_id="person-uuid-0",
- person_version=1,
- set_updates={
- "prop": PropertyValue(timestamp=datetime(2024, 1, 15, 12, 0, 0, tzinfo=UTC), value="value_0")
- },
- set_once_updates={},
- unset_updates={},
- )
- ]
-
- mock_cursor = MagicMock()
- # Return None from fetchone to simulate person not found (no backup, no update)
- mock_cursor.fetchone.return_value = None
-
- commit_count = [0]
-
- def mock_commit():
- commit_count[0] += 1
-
- result = process_persons_in_batches(
- person_property_diffs=person_diffs_list,
- cursor=mock_cursor,
- job_id="test-job-id",
- team_id=1,
- batch_size=10,
- dry_run=True,
- backup_enabled=False,
- commit_fn=mock_commit,
- )
-
- # Person not found, so skipped
- assert result.total_processed == 1
- assert result.total_skipped == 1
- assert result.total_commits == 0
-
-
-class TestBackupFunctionality:
- """Test the backup functionality for person property reconciliation."""
-
- def test_backup_contains_pending_operations(self):
- """Test that backup stores the pending operations correctly."""
- import json
-
- person_data = {
- "id": 123,
- "uuid": "018d1234-5678-0000-0000-000000000001",
- "properties": {"existing": "value"},
- "properties_last_updated_at": {"existing": "2024-01-01T00:00:00"},
- "properties_last_operation": {"existing": "set"},
- "version": 5,
- "is_identified": True,
- "created_at": datetime(2024, 1, 1, 0, 0, 0),
- }
-
- cursor = MagicMock()
- cursor.fetchone.return_value = person_data
- cursor.rowcount = 1
-
- person_diffs = PersonPropertyDiffs(
- person_id="018d1234-5678-0000-0000-000000000001",
- person_version=5,
- set_updates={"email": PropertyValue(timestamp=datetime(2024, 1, 15, 12, 0, 0), value="[email protected]")},
- set_once_updates={"name": PropertyValue(timestamp=datetime(2024, 1, 15, 12, 0, 0), value="Test User")},
- unset_updates={},
- )
-
- update_person_with_version_check(
- cursor=cursor,
- job_id="test-job-id",
- team_id=1,
- person_uuid="018d1234-5678-0000-0000-000000000001",
- person_property_diffs=person_diffs,
- dry_run=False,
- )
-
- # Find the backup INSERT call
- backup_calls = [
- call
- for call in cursor.execute.call_args_list
- if "INSERT INTO posthog_person_reconciliation_backup" in str(call)
- ]
- assert len(backup_calls) == 1
-
- backup_params = backup_calls[0][0][1]
- pending_operations_json = backup_params[11] # pending_operations is at index 11
- pending_operations = json.loads(pending_operations_json)
-
- assert len(pending_operations) == 2
- # Find operations by key since dict iteration order may vary
- email_op = next(op for op in pending_operations if op["key"] == "email")
- name_op = next(op for op in pending_operations if op["key"] == "name")
- assert email_op["value"] == "[email protected]"
- assert email_op["operation"] == "set"
- assert name_op["value"] == "Test User"
- assert name_op["operation"] == "set_once"
-
- def test_backup_preserves_before_and_after_state(self):
- """Test that backup correctly stores both before and after state."""
- import json
-
- person_data = {
- "id": 456,
- "uuid": "018d1234-5678-0000-0000-000000000002",
- "properties": {"name": "Old Name", "count": 10},
- "properties_last_updated_at": {},
- "properties_last_operation": {},
- "version": 3,
- "is_identified": False,
- "is_user_id": None,
- "created_at": datetime(2024, 1, 1, 0, 0, 0),
- }
-
- cursor = MagicMock()
- cursor.fetchone.return_value = person_data
- cursor.rowcount = 1
-
- person_diffs = PersonPropertyDiffs(
- person_id="018d1234-5678-0000-0000-000000000002",
- person_version=3,
- set_updates={"name": PropertyValue(timestamp=datetime(2024, 1, 15, 12, 0, 0), value="New Name")},
- set_once_updates={},
- unset_updates={},
- )
-
- update_person_with_version_check(
- cursor=cursor,
- job_id="run-abc-123",
- team_id=42,
- person_uuid="018d1234-5678-0000-0000-000000000002",
- person_property_diffs=person_diffs,
- dry_run=False,
- )
-
- backup_calls = [
- call
- for call in cursor.execute.call_args_list
- if "INSERT INTO posthog_person_reconciliation_backup" in str(call)
- ]
- assert len(backup_calls) == 1
-
- backup_params = backup_calls[0][0][1]
-
- # Verify identifiers
- assert backup_params[0] == "run-abc-123" # job_id
- assert backup_params[1] == 42 # team_id
- assert backup_params[2] == 456 # person_id
-
- # Verify before state
- properties_before = json.loads(backup_params[4])
- assert properties_before == {"name": "Old Name", "count": 10}
- assert backup_params[7] == 3 # version before
-
- # Verify after state
- properties_after = json.loads(backup_params[12])
- assert properties_after["name"] == "New Name"
- assert properties_after["count"] == 10 # unchanged property preserved
- assert backup_params[15] == 4 # version after (3 + 1)
-
- def test_backup_disabled_does_not_create_backup(self):
- """Test that backup_enabled=False skips backup creation."""
- person_data = {
- "id": 123,
- "uuid": "018d1234-5678-0000-0000-000000000001",
- "properties": {"existing": "value"},
- "properties_last_updated_at": {},
- "properties_last_operation": {},
- "version": 5,
- "is_identified": True,
- "created_at": datetime(2024, 1, 1, 0, 0, 0),
- }
-
- cursor = MagicMock()
- cursor.fetchone.return_value = person_data
- cursor.rowcount = 1
-
- person_diffs = PersonPropertyDiffs(
- person_id="018d1234-5678-0000-0000-000000000001",
- person_version=5,
- set_updates={"email": PropertyValue(timestamp=datetime(2024, 1, 15, 12, 0, 0), value="[email protected]")},
- set_once_updates={},
- unset_updates={},
- )
-
- result = update_person_with_version_check(
- cursor=cursor,
- job_id="test-job-id",
- team_id=1,
- person_uuid="018d1234-5678-0000-0000-000000000001",
- person_property_diffs=person_diffs,
- dry_run=False,
- backup_enabled=False,
- )
-
- assert result.success is True
- assert result.updated_person_data is not None
- assert result.backup_created is False
-
- # Verify backup INSERT was NOT executed
- backup_calls = [
- call
- for call in cursor.execute.call_args_list
- if "INSERT INTO posthog_person_reconciliation_backup" in str(call)
- ]
- assert len(backup_calls) == 0
-
- # But UPDATE should still happen
- update_calls = [call for call in cursor.execute.call_args_list if "UPDATE posthog_person" in str(call)]
- assert len(update_calls) == 1
-
- def test_backup_created_return_value_true_when_enabled(self):
- """Test that backup_created=True when backup is enabled and changes made."""
- person_data = {
- "id": 123,
- "uuid": "018d1234-5678-0000-0000-000000000001",
- "properties": {},
- "properties_last_updated_at": {},
- "properties_last_operation": {},
- "version": 1,
- "is_identified": False,
- "created_at": datetime(2024, 1, 1, 0, 0, 0),
- }
-
- cursor = MagicMock()
- cursor.fetchone.return_value = person_data
- cursor.rowcount = 1
-
- person_diffs = PersonPropertyDiffs(
- person_id="018d1234-5678-0000-0000-000000000001",
- person_version=1,
- set_updates={"email": PropertyValue(timestamp=datetime(2024, 1, 15, 12, 0, 0), value="[email protected]")},
- set_once_updates={},
- unset_updates={},
- )
-
- result = update_person_with_version_check(
- cursor=cursor,
- job_id="test-job-id",
- team_id=1,
- person_uuid="018d1234-5678-0000-0000-000000000001",
- person_property_diffs=person_diffs,
- dry_run=False,
- backup_enabled=True,
- )
-
- assert result.success is True
- assert result.backup_created is True
-
- def test_backup_created_false_when_conflict(self):
- """
- Regression test: backup_created should be False when ON CONFLICT DO NOTHING
- means no row was actually inserted (duplicate backup for same person/job).
- """
- person_data = {
- "id": 123,
- "uuid": "018d1234-5678-0000-0000-000000000001",
- "properties": {},
- "properties_last_updated_at": {},
- "properties_last_operation": {},
- "version": 1,
- "is_identified": False,
- "created_at": datetime(2024, 1, 1, 0, 0, 0),
- }
-
- cursor = MagicMock()
- cursor.fetchone.return_value = person_data
-
- # Track which query is being executed to return different rowcounts
- call_count = [0]
- original_execute = cursor.execute
-
- def execute_side_effect(*args, **kwargs):
- call_count[0] += 1
- query = args[0] if args else ""
- if "INSERT INTO posthog_person_reconciliation_backup" in query:
- # Simulate ON CONFLICT DO NOTHING - no row inserted
- cursor.rowcount = 0
- else:
- # Person SELECT/UPDATE succeeds
- cursor.rowcount = 1
- return original_execute(*args, **kwargs)
-
- cursor.execute = MagicMock(side_effect=execute_side_effect)
-
- person_diffs = PersonPropertyDiffs(
- person_id="018d1234-5678-0000-0000-000000000001",
- person_version=1,
- set_updates={"email": PropertyValue(timestamp=datetime(2024, 1, 15, 12, 0, 0), value="[email protected]")},
- set_once_updates={},
- unset_updates={},
- )
-
- result = update_person_with_version_check(
- cursor=cursor,
- job_id="test-job-id",
- team_id=1,
- person_uuid="018d1234-5678-0000-0000-000000000001",
- person_property_diffs=person_diffs,
- dry_run=False,
- backup_enabled=True,
- )
-
- assert result.success is True
- # Key assertion: backup_created should be False because rowcount was 0
- assert result.backup_created is False
-
-
-class TestFilterEventPersonProperties:
- """Test the filter_event_person_properties function for conflict resolution."""
-
- def test_set_wins_when_newer_than_unset(self):
- """Test that $set wins over $unset when set timestamp is newer."""
- person_diffs = [
- PersonPropertyDiffs(
- person_id="018d1234-5678-0000-0000-000000000001",
- person_version=1,
- set_updates={
- "email": PropertyValue(
- timestamp=datetime(2024, 1, 15, 12, 0, 0, tzinfo=UTC), value="[email protected]"
- )
- },
- set_once_updates={},
- unset_updates={
- "email": PropertyValue(timestamp=datetime(2024, 1, 10, 12, 0, 0, tzinfo=UTC), value=None)
- },
- )
- ]
-
- result = filter_event_person_properties(person_diffs)
-
- assert len(result) == 1
- # set should win (newer), unset should be removed
- assert "email" in result[0].set_updates
- assert "email" not in result[0].unset_updates
-
- def test_unset_wins_when_newer_than_set(self):
- """Test that $unset wins over $set when unset timestamp is newer or equal."""
- person_diffs = [
- PersonPropertyDiffs(
- person_id="018d1234-5678-0000-0000-000000000001",
- person_version=1,
- set_updates={
- "email": PropertyValue(
- timestamp=datetime(2024, 1, 10, 12, 0, 0, tzinfo=UTC), value="[email protected]"
- )
- },
- set_once_updates={},
- unset_updates={
- "email": PropertyValue(timestamp=datetime(2024, 1, 15, 12, 0, 0, tzinfo=UTC), value=None)
- },
- )
- ]
-
- result = filter_event_person_properties(person_diffs)
-
- assert len(result) == 1
- # unset should win (newer), set should be removed
- assert "email" not in result[0].set_updates
- assert "email" in result[0].unset_updates
-
- def test_set_once_wins_when_newer_than_unset(self):
- """Test that $set_once wins over $unset when set_once timestamp is newer."""
- person_diffs = [
- PersonPropertyDiffs(
- person_id="018d1234-5678-0000-0000-000000000001",
- person_version=1,
- set_updates={},
- set_once_updates={
- "referrer": PropertyValue(timestamp=datetime(2024, 1, 15, 12, 0, 0, tzinfo=UTC), value="google.com")
- },
- unset_updates={
- "referrer": PropertyValue(timestamp=datetime(2024, 1, 10, 12, 0, 0, tzinfo=UTC), value=None)
- },
- )
- ]
-
- result = filter_event_person_properties(person_diffs)
-
- assert len(result) == 1
- # set_once should win (newer), unset should be removed
- assert "referrer" in result[0].set_once_updates
- assert "referrer" not in result[0].unset_updates
-
- def test_unset_wins_when_newer_than_set_once(self):
- """Test that $unset wins over $set_once when unset timestamp is newer."""
- person_diffs = [
- PersonPropertyDiffs(
- person_id="018d1234-5678-0000-0000-000000000001",
- person_version=1,
- set_updates={},
- set_once_updates={
- "referrer": PropertyValue(timestamp=datetime(2024, 1, 10, 12, 0, 0, tzinfo=UTC), value="google.com")
- },
- unset_updates={
- "referrer": PropertyValue(timestamp=datetime(2024, 1, 15, 12, 0, 0, tzinfo=UTC), value=None)
- },
- )
- ]
-
- result = filter_event_person_properties(person_diffs)
-
- assert len(result) == 1
- # unset should win (newer), set_once should be removed
- assert "referrer" not in result[0].set_once_updates
- assert "referrer" in result[0].unset_updates
-
- def test_no_conflict_passes_through(self):
- """Test that operations on different keys pass through unchanged."""
- person_diffs = [
- PersonPropertyDiffs(
- person_id="018d1234-5678-0000-0000-000000000001",
- person_version=1,
- set_updates={
- "email": PropertyValue(
- timestamp=datetime(2024, 1, 15, 12, 0, 0, tzinfo=UTC), value="[email protected]"
- )
- },
- set_once_updates={
- "referrer": PropertyValue(timestamp=datetime(2024, 1, 10, 8, 0, 0, tzinfo=UTC), value="google.com")
- },
- unset_updates={
- "old_field": PropertyValue(timestamp=datetime(2024, 1, 14, 10, 0, 0, tzinfo=UTC), value=None)
- },
- )
- ]
-
- result = filter_event_person_properties(person_diffs)
-
- assert len(result) == 1
- assert "email" in result[0].set_updates
- assert "referrer" in result[0].set_once_updates
- assert "old_field" in result[0].unset_updates
-
- def test_multiple_persons_filtered_independently(self):
- """Test that multiple persons are filtered independently."""
- person_diffs = [
- PersonPropertyDiffs(
- person_id="person1",
- person_version=1,
- set_updates={
- "email": PropertyValue(
- timestamp=datetime(2024, 1, 15, 12, 0, 0, tzinfo=UTC), value="[email protected]"
- )
- },
- set_once_updates={},
- unset_updates={
- "email": PropertyValue(timestamp=datetime(2024, 1, 10, 12, 0, 0, tzinfo=UTC), value=None)
- },
- ),
- PersonPropertyDiffs(
- person_id="person2",
- person_version=1,
- set_updates={
- "email": PropertyValue(
- timestamp=datetime(2024, 1, 10, 12, 0, 0, tzinfo=UTC), value="[email protected]"
- )
- },
- set_once_updates={},
- unset_updates={
- "email": PropertyValue(timestamp=datetime(2024, 1, 15, 12, 0, 0, tzinfo=UTC), value=None)
- },
- ),
- ]
-
- result = filter_event_person_properties(person_diffs)
-
- assert len(result) == 2
- # Person 1: set wins (newer)
- assert "email" in result[0].set_updates
- assert "email" not in result[0].unset_updates
- # Person 2: unset wins (newer)
- assert "email" not in result[1].set_updates
- assert "email" in result[1].unset_updates
-
-
-class TestMergeRawPersonPropertyUpdates:
- """Test the merge_raw_person_property_updates function for windowed query accumulation."""
-
- def test_first_person_added_to_empty_accumulator(self):
- """Test that first person updates are added directly to empty accumulator."""
- accumulated: dict[str, RawPersonPropertyUpdates] = {}
- new_updates = [
- RawPersonPropertyUpdates(
- person_id="person-1",
- set_updates={"email": PropertyValue(datetime(2024, 1, 15, 12, 0, 0, tzinfo=UTC), "[email protected]")},
- set_once_updates={},
- unset_updates={},
- )
- ]
-
- merge_raw_person_property_updates(accumulated, new_updates)
-
- assert "person-1" in accumulated
- assert accumulated["person-1"].set_updates["email"].value == "[email protected]"
-
- def test_set_newer_timestamp_replaces_older(self):
- """Test that $set with newer timestamp replaces older value."""
- accumulated: dict[str, RawPersonPropertyUpdates] = {
- "person-1": RawPersonPropertyUpdates(
- person_id="person-1",
- set_updates={"email": PropertyValue(datetime(2024, 1, 15, 12, 0, 0, tzinfo=UTC), "[email protected]")},
- set_once_updates={},
- unset_updates={},
- )
- }
- new_updates = [
- RawPersonPropertyUpdates(
- person_id="person-1",
- set_updates={"email": PropertyValue(datetime(2024, 1, 15, 13, 0, 0, tzinfo=UTC), "[email protected]")},
- set_once_updates={},
- unset_updates={},
- )
- ]
-
- merge_raw_person_property_updates(accumulated, new_updates)
-
- assert accumulated["person-1"].set_updates["email"].value == "[email protected]"
-
- def test_set_once_earlier_timestamp_wins(self):
- """Test that $set_once with earlier timestamp wins (first-writer-wins)."""
- accumulated: dict[str, RawPersonPropertyUpdates] = {
- "person-1": RawPersonPropertyUpdates(
- person_id="person-1",
- set_updates={},
- set_once_updates={
- "signup_source": PropertyValue(datetime(2024, 1, 15, 13, 0, 0, tzinfo=UTC), "late-source")
- },
- unset_updates={},
- )
- }
- new_updates = [
- RawPersonPropertyUpdates(
- person_id="person-1",
- set_updates={},
- set_once_updates={
- "signup_source": PropertyValue(datetime(2024, 1, 15, 12, 0, 0, tzinfo=UTC), "early-source")
- },
- unset_updates={},
- )
- ]
-
- merge_raw_person_property_updates(accumulated, new_updates)
-
- assert accumulated["person-1"].set_once_updates["signup_source"].value == "early-source"
-
- def test_set_and_unset_both_retained_for_downstream_filtering(self):
- """Test that both $set and $unset are retained for downstream conflict resolution."""
- accumulated: dict[str, RawPersonPropertyUpdates] = {
- "person-1": RawPersonPropertyUpdates(
- person_id="person-1",
- set_updates={"email": PropertyValue(datetime(2024, 1, 15, 12, 0, 0, tzinfo=UTC), "[email protected]")},
- set_once_updates={},
- unset_updates={},
- )
- }
- new_updates = [
- RawPersonPropertyUpdates(
- person_id="person-1",
- set_updates={},
- set_once_updates={},
- unset_updates={"email": PropertyValue(datetime(2024, 1, 15, 13, 0, 0, tzinfo=UTC), None)},
- )
- ]
-
- merge_raw_person_property_updates(accumulated, new_updates)
-
- assert "email" in accumulated["person-1"].set_updates
- assert "email" in accumulated["person-1"].unset_updates
-
- def test_unset_and_set_both_retained_for_downstream_filtering(self):
- """Test that both $unset and $set are retained for downstream conflict resolution."""
- accumulated: dict[str, RawPersonPropertyUpdates] = {
- "person-1": RawPersonPropertyUpdates(
- person_id="person-1",
- set_updates={},
- set_once_updates={},
- unset_updates={"email": PropertyValue(datetime(2024, 1, 15, 12, 0, 0, tzinfo=UTC), None)},
- )
- }
- new_updates = [
- RawPersonPropertyUpdates(
- person_id="person-1",
- set_updates={"email": PropertyValue(datetime(2024, 1, 15, 13, 0, 0, tzinfo=UTC), "[email protected]")},
- set_once_updates={},
- unset_updates={},
- )
- ]
-
- merge_raw_person_property_updates(accumulated, new_updates)
-
- assert "email" in accumulated["person-1"].set_updates
- assert "email" in accumulated["person-1"].unset_updates
-
- def test_multiple_persons_merged_independently(self):
- """Test that multiple persons are merged independently."""
- accumulated: dict[str, RawPersonPropertyUpdates] = {
- "person-1": RawPersonPropertyUpdates(
- person_id="person-1",
- set_updates={"key1": PropertyValue(datetime(2024, 1, 15, 12, 0, 0, tzinfo=UTC), "value1")},
- set_once_updates={},
- unset_updates={},
- )
- }
- new_updates = [
- RawPersonPropertyUpdates(
- person_id="person-1",
- set_updates={"key1": PropertyValue(datetime(2024, 1, 15, 13, 0, 0, tzinfo=UTC), "value1-updated")},
- set_once_updates={},
- unset_updates={},
- ),
- RawPersonPropertyUpdates(
- person_id="person-2",
- set_updates={"key2": PropertyValue(datetime(2024, 1, 15, 12, 0, 0, tzinfo=UTC), "value2")},
- set_once_updates={},
- unset_updates={},
- ),
- ]
-
- merge_raw_person_property_updates(accumulated, new_updates)
-
- assert len(accumulated) == 2
- assert accumulated["person-1"].set_updates["key1"].value == "value1-updated"
- assert accumulated["person-2"].set_updates["key2"].value == "value2"
-
- def test_cross_operation_conflict_resolved_by_downstream_filter(self):
- """Verify set/unset conflicts from different windows are correctly resolved downstream.
-
- This test documents the intentional design: merge_raw_person_property_updates
- retains both operations, and filter_event_person_properties resolves conflicts.
- This matches the non-windowed CH query path behavior.
- """
- accumulated: dict[str, RawPersonPropertyUpdates] = {}
-
- # Simulate window 1: set(email) at t=12:00
- window1 = [
- RawPersonPropertyUpdates(
- person_id="person-1",
- set_updates={"email": PropertyValue(datetime(2024, 1, 15, 12, 0, 0, tzinfo=UTC), "[email protected]")},
- set_once_updates={},
- unset_updates={},
- )
- ]
- merge_raw_person_property_updates(accumulated, window1)
-
- # Simulate window 2: unset(email) at t=13:00
- window2 = [
- RawPersonPropertyUpdates(
- person_id="person-1",
- set_updates={},
- set_once_updates={},
- unset_updates={"email": PropertyValue(datetime(2024, 1, 15, 13, 0, 0, tzinfo=UTC), None)},
- )
- ]
- merge_raw_person_property_updates(accumulated, window2)
-
- # After merge: both operations retained (no cross-map resolution in merge)
- assert "email" in accumulated["person-1"].set_updates
- assert "email" in accumulated["person-1"].unset_updates
-
- # Convert to PersonPropertyDiffs (simulating what compare_raw_updates_with_person_state returns)
- diffs = [
- PersonPropertyDiffs(
- person_id="person-1",
- person_version=1,
- set_updates=accumulated["person-1"].set_updates,
- set_once_updates=accumulated["person-1"].set_once_updates,
- unset_updates=accumulated["person-1"].unset_updates,
- )
- ]
-
- # After filter_event_person_properties: unset wins (newer timestamp)
- result = filter_event_person_properties(diffs)
- assert len(result) == 1
- assert "email" not in result[0].set_updates
- assert "email" in result[0].unset_updates
-
-
-class TestCompareRawUpdatesWithPersonState:
- """Test the compare_raw_updates_with_person_state function for windowed query flow.
-
- This function compares merged raw event updates against current person state in ClickHouse.
- It filters to only return actual differences that need to be applied.
- """
-
- @patch("posthog.dags.person_property_reconciliation.sync_execute")
- def test_empty_raw_updates_returns_empty(self, mock_sync_execute):
- """Test that empty raw_updates returns empty result without querying CH."""
- result = compare_raw_updates_with_person_state(team_id=1, raw_updates=[])
-
- assert result == []
- mock_sync_execute.assert_not_called()
-
- @patch("posthog.dags.person_property_reconciliation.sync_execute")
- def test_person_not_found_excluded_from_results(self, mock_sync_execute):
- """Test that persons not found in CH are excluded from results."""
- mock_sync_execute.return_value = [] # No person found
-
- raw_updates = [
- RawPersonPropertyUpdates(
- person_id="person-1",
- set_updates={"email": PropertyValue(datetime(2024, 1, 15, 12, 0, 0, tzinfo=UTC), "[email protected]")},
- set_once_updates={},
- unset_updates={},
- )
- ]
-
- result = compare_raw_updates_with_person_state(team_id=1, raw_updates=raw_updates)
-
- assert result == []
-
- @patch("posthog.dags.person_property_reconciliation.sync_execute")
- def test_set_key_exists_value_differs_included(self, mock_sync_execute):
- """Test that $set is included when key exists in person AND value differs."""
- mock_sync_execute.return_value = [("person-1", '{"email": "[email protected]"}', 5)]
-
- raw_updates = [
- RawPersonPropertyUpdates(
- person_id="person-1",
- set_updates={"email": PropertyValue(datetime(2024, 1, 15, 12, 0, 0, tzinfo=UTC), "[email protected]")},
- set_once_updates={},
- unset_updates={},
- )
- ]
-
- result = compare_raw_updates_with_person_state(team_id=1, raw_updates=raw_updates)
-
- assert len(result) == 1
- assert "email" in result[0].set_updates
- assert result[0].set_updates["email"].value == "[email protected]"
-
- @patch("posthog.dags.person_property_reconciliation.sync_execute")
- def test_set_key_exists_value_same_excluded(self, mock_sync_execute):
- """Test that $set is excluded when key exists but value is the same."""
- mock_sync_execute.return_value = [("person-1", '{"email": "[email protected]"}', 5)]
-
- raw_updates = [
- RawPersonPropertyUpdates(
- person_id="person-1",
- set_updates={"email": PropertyValue(datetime(2024, 1, 15, 12, 0, 0, tzinfo=UTC), "[email protected]")},
- set_once_updates={},
- unset_updates={},
- )
- ]
-
- result = compare_raw_updates_with_person_state(team_id=1, raw_updates=raw_updates)
-
- assert result == []
-
- @patch("posthog.dags.person_property_reconciliation.sync_execute")
- def test_set_key_not_exists_excluded(self, mock_sync_execute):
- """Test that $set is excluded when key does NOT exist in person properties."""
- mock_sync_execute.return_value = [("person-1", '{"other_key": "value"}', 5)]
-
- raw_updates = [
- RawPersonPropertyUpdates(
- person_id="person-1",
- set_updates={"email": PropertyValue(datetime(2024, 1, 15, 12, 0, 0, tzinfo=UTC), "[email protected]")},
- set_once_updates={},
- unset_updates={},
- )
- ]
-
- result = compare_raw_updates_with_person_state(team_id=1, raw_updates=raw_updates)
-
- assert result == []
-
- @patch("posthog.dags.person_property_reconciliation.sync_execute")
- def test_set_once_key_not_exists_included(self, mock_sync_execute):
- """Test that $set_once is included when key does NOT exist in person."""
- mock_sync_execute.return_value = [("person-1", '{"other_key": "value"}', 5)]
-
- raw_updates = [
- RawPersonPropertyUpdates(
- person_id="person-1",
- set_updates={},
- set_once_updates={"referrer": PropertyValue(datetime(2024, 1, 15, 12, 0, 0, tzinfo=UTC), "google.com")},
- unset_updates={},
- )
- ]
-
- result = compare_raw_updates_with_person_state(team_id=1, raw_updates=raw_updates)
-
- assert len(result) == 1
- assert "referrer" in result[0].set_once_updates
- assert result[0].set_once_updates["referrer"].value == "google.com"
-
- @patch("posthog.dags.person_property_reconciliation.sync_execute")
- def test_set_once_key_exists_excluded(self, mock_sync_execute):
- """Test that $set_once is excluded when key already exists in person."""
- mock_sync_execute.return_value = [("person-1", '{"referrer": "existing.com"}', 5)]
-
- raw_updates = [
- RawPersonPropertyUpdates(
- person_id="person-1",
- set_updates={},
- set_once_updates={"referrer": PropertyValue(datetime(2024, 1, 15, 12, 0, 0, tzinfo=UTC), "new.com")},
- unset_updates={},
- )
- ]
-
- result = compare_raw_updates_with_person_state(team_id=1, raw_updates=raw_updates)
-
- assert result == []
-
- @patch("posthog.dags.person_property_reconciliation.sync_execute")
- def test_unset_key_exists_included(self, mock_sync_execute):
- """Test that $unset is included when key exists in person."""
- mock_sync_execute.return_value = [("person-1", '{"email": "[email protected]"}', 5)]
-
- raw_updates = [
- RawPersonPropertyUpdates(
- person_id="person-1",
- set_updates={},
- set_once_updates={},
- unset_updates={"email": PropertyValue(datetime(2024, 1, 15, 12, 0, 0, tzinfo=UTC), None)},
- )
- ]
-
- result = compare_raw_updates_with_person_state(team_id=1, raw_updates=raw_updates)
-
- assert len(result) == 1
- assert "email" in result[0].unset_updates
-
- @patch("posthog.dags.person_property_reconciliation.sync_execute")
- def test_unset_key_not_exists_excluded(self, mock_sync_execute):
- """Test that $unset is excluded when key does NOT exist in person."""
- mock_sync_execute.return_value = [("person-1", '{"other_key": "value"}', 5)]
-
- raw_updates = [
- RawPersonPropertyUpdates(
- person_id="person-1",
- set_updates={},
- set_once_updates={},
- unset_updates={"email": PropertyValue(datetime(2024, 1, 15, 12, 0, 0, tzinfo=UTC), None)},
- )
- ]
-
- result = compare_raw_updates_with_person_state(team_id=1, raw_updates=raw_updates)
-
- assert result == []
-
- @patch("posthog.dags.person_property_reconciliation.sync_execute")
- def test_multiple_persons_processed_independently(self, mock_sync_execute):
- """Test that multiple persons are processed independently."""
- mock_sync_execute.return_value = [
- ("person-1", '{"email": "[email protected]"}', 5),
- ("person-2", "{}", 3), # Empty properties
- ]
-
- raw_updates = [
- RawPersonPropertyUpdates(
- person_id="person-1",
- set_updates={"email": PropertyValue(datetime(2024, 1, 15, 12, 0, 0, tzinfo=UTC), "[email protected]")},
- set_once_updates={},
- unset_updates={},
- ),
- RawPersonPropertyUpdates(
- person_id="person-2",
- set_updates={},
- set_once_updates={"source": PropertyValue(datetime(2024, 1, 15, 12, 0, 0, tzinfo=UTC), "organic")},
- unset_updates={},
- ),
- ]
-
- result = compare_raw_updates_with_person_state(team_id=1, raw_updates=raw_updates)
-
- assert len(result) == 2
- # Person 1: $set included (key exists, value differs)
- p1 = next(r for r in result if r.person_id == "person-1")
- assert "email" in p1.set_updates
- # Person 2: $set_once included (key doesn't exist)
- p2 = next(r for r in result if r.person_id == "person-2")
- assert "source" in p2.set_once_updates
-
- @patch("posthog.dags.person_property_reconciliation.sync_execute")
- def test_person_version_propagated_correctly(self, mock_sync_execute):
- """Test that person_version from CH is propagated to PersonPropertyDiffs."""
- mock_sync_execute.return_value = [
- ("person-1", '{"email": "[email protected]"}', 42) # version=42
- ]
-
- raw_updates = [
- RawPersonPropertyUpdates(
- person_id="person-1",
- set_updates={"email": PropertyValue(datetime(2024, 1, 15, 12, 0, 0, tzinfo=UTC), "[email protected]")},
- set_once_updates={},
- unset_updates={},
- )
- ]
-
- result = compare_raw_updates_with_person_state(team_id=1, raw_updates=raw_updates)
-
- assert len(result) == 1
- assert result[0].person_version == 42
-
- @patch("posthog.dags.person_property_reconciliation.sync_execute")
- def test_type_coercion_equality(self, mock_sync_execute):
- """Test that Python type coercion treats 123 and 123.0 as equal.
-
- This documents the semantic comparison behavior: the windowed path uses
- Python object comparison, so numerically equivalent values like 123 (int)
- and 123.0 (float) are considered equal. This differs from the non-windowed
- SQL path which compares raw JSON strings.
-
- The Python approach is preferable for reconciliation as it avoids
- unnecessary updates when values are semantically equivalent.
- """
- # Person has integer 123, event sets float 123.0
- mock_sync_execute.return_value = [("person-1", '{"count": 123}', 5)]
-
- raw_updates = [
- RawPersonPropertyUpdates(
- person_id="person-1",
- set_updates={"count": PropertyValue(datetime(2024, 1, 15, 12, 0, 0, tzinfo=UTC), 123.0)},
- set_once_updates={},
- unset_updates={},
- )
- ]
-
- result = compare_raw_updates_with_person_state(team_id=1, raw_updates=raw_updates)
-
- # 123 == 123.0 in Python, so no diff is generated
- assert result == []
-
- @patch("posthog.dags.person_property_reconciliation.PERSON_STATE_BATCH_SIZE", 2)
- @patch("posthog.dags.person_property_reconciliation.sync_execute")
- def test_batches_large_person_lists(self, mock_sync_execute):
- """Test that large person lists are batched to avoid query size limits.
-
- When there are more persons than PERSON_STATE_BATCH_SIZE, the function
- should split the queries into multiple batches and aggregate results.
- """
- # With batch size of 2, 5 persons should result in 3 batches
- mock_sync_execute.side_effect = [
- # Batch 1: persons 1-2
- [
- ("person-1", '{"email": "[email protected]"}', 5),
- ("person-2", '{"email": "[email protected]"}', 6),
- ],
- # Batch 2: persons 3-4
- [
- ("person-3", '{"email": "[email protected]"}', 7),
- ("person-4", "{}", 8), # Empty properties
- ],
- # Batch 3: person 5
- [
- ("person-5", '{"email": "[email protected]"}', 9),
- ],
- ]
-
- raw_updates = [
- RawPersonPropertyUpdates(
- person_id=f"person-{i}",
- set_updates={
- "email": PropertyValue(datetime(2024, 1, 15, 12, 0, 0, tzinfo=UTC), f"new{i}@example.com")
- },
- set_once_updates={},
- unset_updates={},
- )
- for i in range(1, 6)
- ]
-
- result = compare_raw_updates_with_person_state(team_id=1, raw_updates=raw_updates)
-
- # Should have called sync_execute 3 times (one per batch)
- assert mock_sync_execute.call_count == 3
-
- # Should return 4 results (person-4 has empty properties so $set is excluded)
- assert len(result) == 4
- result_ids = {r.person_id for r in result}
- assert result_ids == {"person-1", "person-2", "person-3", "person-5"}
-
-
-class TestGetPersonPropertyUpdatesWindowed:
- """Test the get_person_property_updates_windowed function."""
-
- @patch("posthog.dags.person_property_reconciliation.get_person_property_updates_from_clickhouse")
- def test_window_seconds_zero_calls_original_once(self, mock_get_updates):
- """Test that window_seconds=0 calls original function once."""
- mock_get_updates.return_value = [
- PersonPropertyDiffs(
- person_id="person-1",
- person_version=1,
- set_updates={"email": PropertyValue(datetime(2024, 1, 15, 12, 0, 0, tzinfo=UTC), "[email protected]")},
- set_once_updates={},
- unset_updates={},
- )
- ]
-
- result = get_person_property_updates_windowed(
- team_id=1,
- bug_window_start="2024-01-01 00:00:00",
- window_seconds=0,
- )
-
- mock_get_updates.assert_called_once_with(1, "2024-01-01 00:00:00")
- assert len(result) == 1
-
- @patch("posthog.dags.person_property_reconciliation.get_person_property_updates_from_clickhouse")
- def test_window_seconds_negative_calls_original_once(self, mock_get_updates):
- """Test that negative window_seconds calls original function once."""
- mock_get_updates.return_value = []
-
- get_person_property_updates_windowed(
- team_id=1,
- bug_window_start="2024-01-01 00:00:00",
- window_seconds=-100,
- )
-
- mock_get_updates.assert_called_once_with(1, "2024-01-01 00:00:00")
-
- @patch("posthog.dags.person_property_reconciliation.compare_raw_updates_with_person_state")
- @patch("posthog.dags.person_property_reconciliation.datetime")
- @patch("posthog.dags.person_property_reconciliation.get_raw_person_property_updates_from_clickhouse")
- @patch("posthog.dags.person_property_reconciliation.get_affected_person_ids_from_clickhouse")
- def test_window_seconds_positive_creates_windows(
- self, mock_get_affected, mock_get_raw, mock_datetime, mock_compare
- ):
- """Test that positive window_seconds creates multiple query windows."""
- mock_now = datetime(2024, 1, 1, 2, 0, 0, tzinfo=UTC)
- mock_datetime.now.return_value = mock_now
- mock_datetime.strptime = datetime.strptime
- mock_get_affected.return_value = ["person-1"]
- mock_get_raw.return_value = []
- mock_compare.return_value = []
-
- get_person_property_updates_windowed(
- team_id=1,
- bug_window_start="2024-01-01 00:00:00",
- window_seconds=3600, # 1 hour windows
- )
-
- # Should first fetch affected person_ids
- mock_get_affected.assert_called_once()
- # Should have 2 calls: 00:00-01:00 and 01:00-02:00, with person_ids filter
- assert mock_get_raw.call_count == 2
- calls = mock_get_raw.call_args_list
- assert calls[0][0] == (1, "2024-01-01 00:00:00", "2024-01-01 01:00:00")
- assert calls[0][1] == {"person_ids": ("person-1",)}
- assert calls[1][0] == (1, "2024-01-01 01:00:00", "2024-01-01 02:00:00")
- assert calls[1][1] == {"person_ids": ("person-1",)}
- # compare_raw_updates_with_person_state should be called once at the end
- mock_compare.assert_called_once()
-
- @patch("posthog.dags.person_property_reconciliation.compare_raw_updates_with_person_state")
- @patch("posthog.dags.person_property_reconciliation.datetime")
- @patch("posthog.dags.person_property_reconciliation.get_raw_person_property_updates_from_clickhouse")
- @patch("posthog.dags.person_property_reconciliation.get_affected_person_ids_from_clickhouse")
- def test_window_merges_results_across_windows(self, mock_get_affected, mock_get_raw, mock_datetime, mock_compare):
- """Test that results are properly merged across windows."""
- mock_now = datetime(2024, 1, 1, 2, 0, 0, tzinfo=UTC)
- mock_datetime.now.return_value = mock_now
- mock_datetime.strptime = datetime.strptime
- mock_get_affected.return_value = ["person-1"]
-
- # First window returns one update, second window returns a newer update for same key
- mock_get_raw.side_effect = [
- [
- RawPersonPropertyUpdates(
- person_id="person-1",
- set_updates={"email": PropertyValue(datetime(2024, 1, 1, 0, 30, 0, tzinfo=UTC), "[email protected]")},
- set_once_updates={},
- unset_updates={},
- )
- ],
- [
- RawPersonPropertyUpdates(
- person_id="person-1",
- set_updates={"email": PropertyValue(datetime(2024, 1, 1, 1, 30, 0, tzinfo=UTC), "[email protected]")},
- set_once_updates={},
- unset_updates={},
- )
- ],
- ]
-
- # Mock the comparison to return the expected result
- mock_compare.return_value = [
- PersonPropertyDiffs(
- person_id="person-1",
- person_version=1,
- set_updates={"email": PropertyValue(datetime(2024, 1, 1, 1, 30, 0, tzinfo=UTC), "[email protected]")},
- set_once_updates={},
- unset_updates={},
- )
- ]
-
- result = get_person_property_updates_windowed(
- team_id=1,
- bug_window_start="2024-01-01 00:00:00",
- window_seconds=3600,
- )
-
- assert len(result) == 1
- assert result[0].set_updates["email"].value == "[email protected]"
-
- # Verify compare was called with merged raw updates (newer email value)
- compare_call_args = mock_compare.call_args[0]
- assert compare_call_args[0] == 1 # team_id
- raw_updates_list = compare_call_args[1]
- assert len(raw_updates_list) == 1
- assert raw_updates_list[0].set_updates["email"].value == "[email protected]"
-
-
-class TestGetAffectedPersonIdsFromClickhouse:
- """Test the get_affected_person_ids_from_clickhouse function.
-
- This query aggregates event properties within the bug window, compares against
- current person state, and returns only person_ids with actual diffs.
- """
-
- @patch("posthog.dags.person_property_reconciliation.sync_execute")
- def test_returns_person_ids_in_bug_window(self, mock_sync_execute):
- """Test that query is bounded by bug_window_end."""
- mock_sync_execute.return_value = [
- ("person-1",),
- ("person-2",),
- ("person-3",),
- ]
-
- result = get_affected_person_ids_from_clickhouse(
- team_id=1,
- bug_window_start="2024-01-01 00:00:00",
- bug_window_end="2024-01-02 00:00:00",
- )
-
- assert result == ["person-1", "person-2", "person-3"]
-
- # Verify the query uses bug_window_end (not now())
- query = mock_sync_execute.call_args[0][0]
- params = mock_sync_execute.call_args[0][1]
- assert "bug_window_end" in params
- assert params["bug_window_end"] == "2024-01-02 00:00:00"
- assert "e.timestamp < %(bug_window_end)s" in query
-
- @patch("posthog.dags.person_property_reconciliation.sync_execute")
- def test_returns_empty_list_when_no_persons(self, mock_sync_execute):
- """Test that empty list is returned when no affected persons."""
- mock_sync_execute.return_value = []
-
- result = get_affected_person_ids_from_clickhouse(
- team_id=1,
- bug_window_start="2024-01-01 00:00:00",
- bug_window_end="2024-01-02 00:00:00",
- )
-
- assert result == []
-
- @patch("posthog.dags.person_property_reconciliation.sync_execute")
- def test_filters_out_filtered_person_update_properties(self, mock_sync_execute):
- """Detection query must exclude events that only set FILTERED_PERSON_UPDATE_PROPERTIES."""
- mock_sync_execute.return_value = []
-
- get_affected_person_ids_from_clickhouse(
- team_id=1,
- bug_window_start="2024-01-01 00:00:00",
- bug_window_end="2024-01-02 00:00:00",
- )
-
- query = mock_sync_execute.call_args[0][0]
- params = mock_sync_execute.call_args[0][1]
-
- assert "filtered_properties" in params
- assert params["filtered_properties"] == tuple(FILTERED_PERSON_UPDATE_PROPERTIES)
- assert "NOT IN %(filtered_properties)s" in query
-
-
-class TestGetRawPersonPropertyUpdatesWithPersonIds:
- """Test the person_ids filter parameter in get_raw_person_property_updates_from_clickhouse."""
-
- @patch("posthog.dags.person_property_reconciliation.sync_execute")
- def test_filters_by_person_ids(self, mock_sync_execute):
- """Test that query filters to specific person_ids (required param)."""
- mock_sync_execute.return_value = []
-
- get_raw_person_property_updates_from_clickhouse(
- team_id=1,
- bug_window_start="2024-01-01 00:00:00",
- bug_window_end="2024-01-02 00:00:00",
- person_ids=("person-1", "person-2"),
- )
-
- params = mock_sync_execute.call_args[0][1]
- # person_ids is required and used directly in IN clause
- assert params["person_ids"] == ("person-1", "person-2")
-
-
-class TestGetPersonPropertyUpdatesWindowedBatched:
- """Test the get_person_property_updates_windowed_batched iterator function."""
-
- @patch("posthog.dags.person_property_reconciliation.compare_raw_updates_with_person_state")
- @patch("posthog.dags.person_property_reconciliation.get_raw_person_property_updates_from_clickhouse")
- @patch("posthog.dags.person_property_reconciliation.datetime")
- @patch("posthog.dags.person_property_reconciliation.get_affected_person_ids_from_clickhouse")
- def test_yields_batches_of_persons(self, mock_get_affected, mock_datetime, mock_get_raw, mock_compare):
- """Test that function yields batches when person count exceeds batch size."""
- # Setup: 3 affected persons, batch size of 2 = 2 batches
- mock_get_affected.return_value = ["person-1", "person-2", "person-3"]
-
- mock_now = datetime(2024, 1, 1, 1, 0, 0, tzinfo=UTC)
- mock_datetime.now.return_value = mock_now
- mock_datetime.strptime = datetime.strptime
-
- # Return raw updates so accumulated dict is populated
- mock_get_raw.side_effect = [
- # First batch, first window
- [
- RawPersonPropertyUpdates(
- person_id="person-1",
- set_updates={"key": PropertyValue(datetime(2024, 1, 1, 0, 30, 0, tzinfo=UTC), "val1")},
- set_once_updates={},
- unset_updates={},
- )
- ],
- # Second batch, first window
- [
- RawPersonPropertyUpdates(
- person_id="person-3",
- set_updates={"key": PropertyValue(datetime(2024, 1, 1, 0, 30, 0, tzinfo=UTC), "val3")},
- set_once_updates={},
- unset_updates={},
- )
- ],
- ]
- mock_compare.side_effect = [
- [ # First batch result
- PersonPropertyDiffs(
- person_id="person-1",
- person_version=1,
- set_updates={"key": PropertyValue(datetime(2024, 1, 1, 0, 30, 0, tzinfo=UTC), "val1")},
- set_once_updates={},
- unset_updates={},
- ),
- ],
- [ # Second batch result
- PersonPropertyDiffs(
- person_id="person-3",
- person_version=1,
- set_updates={"key": PropertyValue(datetime(2024, 1, 1, 0, 30, 0, tzinfo=UTC), "val3")},
- set_once_updates={},
- unset_updates={},
- ),
- ],
- ]
-
- results = list(
- get_person_property_updates_windowed_batched(
- team_id=1,
- bug_window_start="2024-01-01 00:00:00",
- bug_window_end="2024-01-01 01:00:00",
- window_seconds=3600,
- person_batch_size=2, # Small batch size for testing
- )
- )
-
- assert len(results) == 2 # Two batches yielded
- assert len(results[0]) == 1 # First batch has 1 person with diffs
- assert results[0][0].person_id == "person-1"
- assert len(results[1]) == 1 # Second batch has 1 person with diffs
- assert results[1][0].person_id == "person-3"
-
- @patch("posthog.dags.person_property_reconciliation.get_affected_person_ids_from_clickhouse")
- def test_yields_nothing_when_no_affected_persons(self, mock_get_affected):
- """Test that function yields nothing when no affected persons."""
- mock_get_affected.return_value = []
-
- results = list(
- get_person_property_updates_windowed_batched(
- team_id=1,
- bug_window_start="2024-01-01 00:00:00",
- bug_window_end="2024-01-02 00:00:00",
- window_seconds=3600,
- )
- )
-
- assert results == []
-
- @patch("posthog.dags.person_property_reconciliation.compare_raw_updates_with_person_state")
- @patch("posthog.dags.person_property_reconciliation.get_raw_person_property_updates_from_clickhouse")
- @patch("posthog.dags.person_property_reconciliation.datetime")
- @patch("posthog.dags.person_property_reconciliation.get_affected_person_ids_from_clickhouse")
- def test_filters_raw_queries_by_person_batch(self, mock_get_affected, mock_datetime, mock_get_raw, mock_compare):
- """Test that windowed queries are filtered to the current person batch."""
- mock_get_affected.return_value = ["person-1", "person-2", "person-3"]
-
- mock_now = datetime(2024, 1, 1, 2, 0, 0, tzinfo=UTC)
- mock_datetime.now.return_value = mock_now
- mock_datetime.strptime = datetime.strptime
-
- mock_get_raw.return_value = []
- mock_compare.return_value = []
-
- # Consume the iterator with batch_size=2
- list(
- get_person_property_updates_windowed_batched(
- team_id=1,
- bug_window_start="2024-01-01 00:00:00",
- bug_window_end="2024-01-01 01:00:00",
- window_seconds=3600,
- person_batch_size=2,
- )
- )
-
- # Check that get_raw was called with person_ids filter
- # First batch should have person-1, person-2
- # Second batch should have person-3
- calls = mock_get_raw.call_args_list
-
- # There should be 4 calls total: 2 windows × 2 batches
- assert len(calls) == 4
-
- # Check first batch calls include person_ids filter
- first_batch_call = calls[0]
- assert first_batch_call[1]["person_ids"] == ("person-1", "person-2")
-
- # Check second batch calls include person_ids filter
- third_call = calls[2] # First call of second batch
- assert third_call[1]["person_ids"] == ("person-3",)
-
- @patch("posthog.dags.person_property_reconciliation.compare_raw_updates_with_person_state")
- @patch("posthog.dags.person_property_reconciliation.get_raw_person_property_updates_from_clickhouse")
- @patch("posthog.dags.person_property_reconciliation.datetime")
- @patch("posthog.dags.person_property_reconciliation.get_affected_person_ids_from_clickhouse")
- def test_uses_bug_window_end_for_affected_persons_query(
- self, mock_get_affected, mock_datetime, mock_get_raw, mock_compare
- ):
- """Test that affected persons query uses bug_window_end (bounded)."""
- mock_get_affected.return_value = []
-
- list(
- get_person_property_updates_windowed_batched(
- team_id=1,
- bug_window_start="2024-01-01 00:00:00",
- bug_window_end="2024-01-02 00:00:00", # Specific end time
- window_seconds=3600,
- )
- )
-
- # Verify get_affected_person_ids_from_clickhouse was called with bug_window_end
- mock_get_affected.assert_called_once_with(
- team_id=1,
- bug_window_start="2024-01-01 00:00:00",
- bug_window_end="2024-01-02 00:00:00",
- )
-
- @patch("posthog.dags.person_property_reconciliation.get_affected_person_ids_from_clickhouse")
- def test_logs_error_when_affected_persons_query_fails(self, mock_get_affected):
- """Test that errors during affected persons query are logged with context."""
- mock_get_affected.side_effect = Exception("ClickHouse connection failed")
- mock_logger = MagicMock()
-
- with pytest.raises(Exception, match="ClickHouse connection failed"):
- list(
- get_person_property_updates_windowed_batched(
- team_id=1,
- bug_window_start="2024-01-01 00:00:00",
- bug_window_end="2024-01-02 00:00:00",
- window_seconds=3600,
- logger=mock_logger,
- )
- )
-
- # Verify error was logged with context
- mock_logger.exception.assert_called()
- error_call = mock_logger.exception.call_args[0][0]
- assert "team_id=1" in error_call
- assert "Failed to query affected persons" in error_call
-
- @patch("posthog.dags.person_property_reconciliation.get_raw_person_property_updates_from_clickhouse")
- @patch("posthog.dags.person_property_reconciliation.datetime")
- @patch("posthog.dags.person_property_reconciliation.get_affected_person_ids_from_clickhouse")
- def test_logs_error_when_window_query_fails(self, mock_get_affected, mock_datetime, mock_get_raw):
- """Test that errors during window queries are logged with batch and window context."""
- mock_get_affected.return_value = ["person-1"]
-
- mock_now = datetime(2024, 1, 1, 2, 0, 0, tzinfo=UTC)
- mock_datetime.now.return_value = mock_now
- mock_datetime.strptime = datetime.strptime
-
- # First window succeeds, second window fails
- mock_get_raw.side_effect = [
- [], # First window OK
- Exception("Query timeout"), # Second window fails
- ]
- mock_logger = MagicMock()
-
- with pytest.raises(Exception, match="Query timeout"):
- list(
- get_person_property_updates_windowed_batched(
- team_id=1,
- bug_window_start="2024-01-01 00:00:00",
- bug_window_end="2024-01-01 01:00:00",
- window_seconds=3600,
- logger=mock_logger,
- )
- )
-
- # Verify error was logged with window context
- error_calls = list(mock_logger.exception.call_args_list)
- assert len(error_calls) >= 1
- # Should have both the inner window error and the outer batch error
- error_messages = [c[0][0] for c in error_calls]
- assert any("failed to query window" in msg for msg in error_messages)
- assert any("Batch 1" in msg for msg in error_messages)
- assert any("team_id=1" in msg for msg in error_messages)
-
- @patch("posthog.dags.person_property_reconciliation.compare_raw_updates_with_person_state")
- @patch("posthog.dags.person_property_reconciliation.get_raw_person_property_updates_from_clickhouse")
- @patch("posthog.dags.person_property_reconciliation.datetime")
- @patch("posthog.dags.person_property_reconciliation.get_affected_person_ids_from_clickhouse")
- def test_logs_error_when_compare_fails(self, mock_get_affected, mock_datetime, mock_get_raw, mock_compare):
- """Test that errors during person state comparison are logged with context."""
- mock_get_affected.return_value = ["person-1"]
-
- mock_now = datetime(2024, 1, 1, 1, 0, 0, tzinfo=UTC)
- mock_datetime.now.return_value = mock_now
- mock_datetime.strptime = datetime.strptime
-
- mock_get_raw.return_value = [
- RawPersonPropertyUpdates(
- person_id="person-1",
- set_updates={"key": PropertyValue(datetime(2024, 1, 1, 0, 30, 0, tzinfo=UTC), "val")},
- set_once_updates={},
- unset_updates={},
- )
- ]
- mock_compare.side_effect = Exception("Person table query failed")
- mock_logger = MagicMock()
-
- with pytest.raises(Exception, match="Person table query failed"):
- list(
- get_person_property_updates_windowed_batched(
- team_id=1,
- bug_window_start="2024-01-01 00:00:00",
- bug_window_end="2024-01-01 01:00:00",
- window_seconds=3600,
- logger=mock_logger,
- )
- )
-
- # Verify error was logged with comparison context
- error_calls = [c[0][0] for c in mock_logger.exception.call_args_list]
- assert any("failed to compare" in msg for msg in error_calls)
- assert any("1 persons" in msg for msg in error_calls)
-
- @patch("posthog.dags.person_property_reconciliation.compare_raw_updates_with_person_state")
- @patch("posthog.dags.person_property_reconciliation.get_raw_person_property_updates_from_clickhouse")
- @patch("posthog.dags.person_property_reconciliation.datetime")
- @patch("posthog.dags.person_property_reconciliation.get_affected_person_ids_from_clickhouse")
- def test_yields_nothing_when_compare_returns_empty(
- self, mock_get_affected, mock_datetime, mock_get_raw, mock_compare
- ):
- """Test that batch yields nothing when compare finds no actual diffs."""
- mock_get_affected.return_value = ["person-1"]
-
- mock_now = datetime(2024, 1, 1, 1, 0, 0, tzinfo=UTC)
- mock_datetime.now.return_value = mock_now
- mock_datetime.strptime = datetime.strptime
-
- # Raw updates exist but compare finds no actual diffs (person state matches)
- mock_get_raw.return_value = [
- RawPersonPropertyUpdates(
- person_id="person-1",
- set_updates={"key": PropertyValue(datetime(2024, 1, 1, 0, 30, 0, tzinfo=UTC), "val")},
- set_once_updates={},
- unset_updates={},
- )
- ]
- mock_compare.return_value = [] # No diffs after comparison
-
- results = list(
- get_person_property_updates_windowed_batched(
- team_id=1,
- bug_window_start="2024-01-01 00:00:00",
- bug_window_end="2024-01-01 01:00:00",
- window_seconds=3600,
- )
- )
-
- assert results == []
-
- @patch("posthog.dags.person_property_reconciliation.compare_raw_updates_with_person_state")
- @patch("posthog.dags.person_property_reconciliation.get_raw_person_property_updates_from_clickhouse")
- @patch("posthog.dags.person_property_reconciliation.datetime")
- @patch("posthog.dags.person_property_reconciliation.get_affected_person_ids_from_clickhouse")
- def test_merges_across_windows_correctly(self, mock_get_affected, mock_datetime, mock_get_raw, mock_compare):
- """Test that $set takes newest timestamp and $set_once takes earliest when merging windows."""
- mock_get_affected.return_value = ["person-1"]
-
- mock_now = datetime(2024, 1, 1, 2, 0, 0, tzinfo=UTC)
- mock_datetime.now.return_value = mock_now
- mock_datetime.strptime = datetime.strptime
-
- # First window: older set value, older set_once value
- # Second window: newer set value, newer set_once value (should be ignored for set_once)
- mock_get_raw.side_effect = [
- [
- RawPersonPropertyUpdates(
- person_id="person-1",
- set_updates={"prop": PropertyValue(datetime(2024, 1, 1, 0, 30, 0, tzinfo=UTC), "old_val")},
- set_once_updates={
- "first_prop": PropertyValue(datetime(2024, 1, 1, 0, 15, 0, tzinfo=UTC), "first_val")
- },
- unset_updates={},
- )
- ],
- [
- RawPersonPropertyUpdates(
- person_id="person-1",
- set_updates={"prop": PropertyValue(datetime(2024, 1, 1, 1, 30, 0, tzinfo=UTC), "new_val")},
- set_once_updates={
- "first_prop": PropertyValue(datetime(2024, 1, 1, 1, 15, 0, tzinfo=UTC), "ignored_val")
- },
- unset_updates={},
- )
- ],
- ]
-
- # Capture what compare receives
- captured_raw_updates = []
-
- def capture_compare(team_id, raw_updates):
- captured_raw_updates.extend(raw_updates)
- return []
-
- mock_compare.side_effect = capture_compare
-
- list(
- get_person_property_updates_windowed_batched(
- team_id=1,
- bug_window_start="2024-01-01 00:00:00",
- bug_window_end="2024-01-01 01:00:00",
- window_seconds=3600,
- )
- )
-
- # Verify merge semantics: $set takes newest, $set_once takes earliest
- assert len(captured_raw_updates) == 1
- merged = captured_raw_updates[0]
- assert merged.set_updates["prop"].value == "new_val" # Newer timestamp wins
- assert merged.set_once_updates["first_prop"].value == "first_val" # Earlier timestamp wins
-
-
-class TestParseFormatChTimestamp:
- """Test the parse_ch_timestamp and format_ch_timestamp helper functions."""
-
- def test_parse_ch_timestamp(self):
- """Test parsing ClickHouse timestamp string to datetime."""
- result = parse_ch_timestamp("2024-01-15 12:30:45")
- assert result == datetime(2024, 1, 15, 12, 30, 45, tzinfo=UTC)
- assert result.tzinfo == UTC
-
- def test_format_ch_timestamp(self):
- """Test formatting datetime to ClickHouse timestamp string."""
- dt = datetime(2024, 1, 15, 12, 30, 45, tzinfo=UTC)
- result = format_ch_timestamp(dt)
- assert result == "2024-01-15 12:30:45"
-
- def test_roundtrip(self):
- """Test that parse and format are inverse operations."""
- original = "2024-06-20 08:15:30"
- parsed = parse_ch_timestamp(original)
- formatted = format_ch_timestamp(parsed)
- assert formatted == original
-
- def test_format_ch_timestamp_converts_to_utc(self):
- """Test that non-UTC datetime is converted to UTC before formatting."""
- from datetime import timedelta, timezone
-
- est = timezone(timedelta(hours=-5))
- dt_est = datetime(2024, 1, 15, 12, 30, 45, tzinfo=est) # 12:30 EST = 17:30 UTC
- result = format_ch_timestamp(dt_est)
- assert result == "2024-01-15 17:30:45"
-
-
-class TestPropertyValueDataclass:
- """Test the PropertyValue dataclass structure."""
-
- def test_property_value_fields(self):
- """Test that PropertyValue has expected fields."""
- pv = PropertyValue(
- timestamp=datetime(2024, 1, 15, 12, 0, 0),
- value="test_value",
- )
-
- assert pv.timestamp == datetime(2024, 1, 15, 12, 0, 0)
- assert pv.value == "test_value"
-
- def test_property_value_accepts_any_value_type(self):
- """Test that value field accepts various types."""
- # String
- pv1 = PropertyValue(timestamp=datetime.now(), value="string")
- assert pv1.value == "string"
-
- # Number
- pv2 = PropertyValue(timestamp=datetime.now(), value=123)
- assert pv2.value == 123
-
- # Boolean
- pv3 = PropertyValue(timestamp=datetime.now(), value=True)
- assert pv3.value is True
-
- # None (for unset)
- pv4 = PropertyValue(timestamp=datetime.now(), value=None)
- assert pv4.value is None
-
-
-class TestPersonPropertyDiffsDataclass:
- """Test the PersonPropertyDiffs dataclass structure."""
-
- def test_person_property_diffs_fields(self):
- """Test that PersonPropertyDiffs has expected fields."""
- person_diffs = PersonPropertyDiffs(
- person_id="018d1234-5678-0000-0000-000000000001",
- person_version=1,
- set_updates={"email": PropertyValue(timestamp=datetime.now(), value="[email protected]")},
- set_once_updates={"referrer": PropertyValue(timestamp=datetime.now(), value="google.com")},
- unset_updates={"old_field": PropertyValue(timestamp=datetime.now(), value=None)},
- )
-
- assert person_diffs.person_id == "018d1234-5678-0000-0000-000000000001"
- assert person_diffs.person_version == 1
- assert len(person_diffs.set_updates) == 1
- assert len(person_diffs.set_once_updates) == 1
- assert len(person_diffs.unset_updates) == 1
- assert person_diffs.set_updates["email"].value == "[email protected]"
- assert person_diffs.set_once_updates["referrer"].value == "google.com"
- assert person_diffs.unset_updates["old_field"].value is None
-
-
-class TestClickHouseQueryIntegration:
- """Integration tests that insert data into ClickHouse and run the actual query."""
-
- @pytest.fixture(scope="class")
- def cluster(self, django_db_setup):
- with isolated_clickhouse_cluster() as clickhouse_cluster:
- yield clickhouse_cluster
-
- def test_unset_uses_latest_timestamp_regression(self, cluster: ClickhouseCluster):
- """
- Regression test: $unset should use max(timestamp), not min(timestamp).
-
- If a person has two $unset operations on the same key at different times,
- the returned timestamp should be the latest one (max), not the earliest (min).
- This was a bug where $unset incorrectly used min(e.timestamp).
- """
-
- team_id = 99901
- person_id = UUID("11111111-1111-1111-1111-000000000001")
- # Use naive datetimes since ClickHouse returns naive datetimes
- now = datetime.now().replace(microsecond=0)
- bug_window_start = now - timedelta(days=10)
-
- # Two timestamps for the same $unset operation
- earlier_ts = now - timedelta(days=5)
- later_ts = now - timedelta(days=2)
-
- # Insert events with $unset at different timestamps
- events = [
- # First $unset at earlier timestamp
- (
- team_id,
- "distinct_id_1",
- person_id,
- earlier_ts,
- json.dumps({"$unset": ["email"]}),
- ),
- # Second $unset at later timestamp (should be the winner)
- (
- team_id,
- "distinct_id_1",
- person_id,
- later_ts,
- json.dumps({"$unset": ["email"]}),
- ),
- ]
-
- def insert_events(client: Client) -> None:
- client.execute(
- """INSERT INTO writable_events (team_id, distinct_id, person_id, timestamp, properties)
- VALUES""",
- events,
- )
-
- cluster.any_host(insert_events).result()
-
- # Insert person with the "email" property (so $unset has something to unset)
- person_data = [
- (
- team_id,
- person_id,
- json.dumps({"email": "[email protected]", "name": "Test User"}),
- 1, # version
- now - timedelta(days=8), # _timestamp within bug window
- )
- ]
-
- def insert_person(client: Client) -> None:
- client.execute(
- """INSERT INTO person (team_id, id, properties, version, _timestamp)
- VALUES""",
- person_data,
- )
-
- cluster.any_host(insert_person).result()
-
- # Run the actual query
- results = get_person_property_updates_from_clickhouse(
- team_id=team_id,
- bug_window_start=bug_window_start.strftime("%Y-%m-%d %H:%M:%S"),
- )
-
- assert len(results) == 1
- person_diffs = results[0]
- assert str(person_diffs.person_id) == str(person_id)
-
- # Find the unset operation
- assert len(person_diffs.unset_updates) == 1
- assert "email" in person_diffs.unset_updates
- unset_update = person_diffs.unset_updates["email"]
-
- assert unset_update.value is None
-
- # Regression check: timestamp should be the LATER one (max), not earlier (min)
- # Allow 1 second tolerance for timestamp comparison
- # Strip timezone for comparison since CH returns tz-aware but we created naive timestamps
- result_ts = unset_update.timestamp.replace(tzinfo=None)
- assert abs((result_ts - later_ts).total_seconds()) < 1, (
- f"$unset should use max(timestamp). "
- f"Expected ~{later_ts}, got {result_ts}. "
- f"Earlier timestamp was {earlier_ts}."
- )
-
- def test_set_uses_latest_timestamp(self, cluster: ClickhouseCluster):
- """$set should use max(timestamp) - latest value wins."""
-
- team_id = 99902
- person_id = UUID("22222222-2222-2222-2222-000000000002")
- # Use naive datetimes since ClickHouse returns naive datetimes
- now = datetime.now().replace(microsecond=0)
- bug_window_start = now - timedelta(days=10)
-
- earlier_ts = now - timedelta(days=5)
- later_ts = now - timedelta(days=2)
-
- events = [
- (team_id, "distinct_id_2", person_id, earlier_ts, json.dumps({"$set": {"name": "First Name"}})),
- (team_id, "distinct_id_2", person_id, later_ts, json.dumps({"$set": {"name": "Latest Name"}})),
- ]
-
- def insert_events(client: Client) -> None:
- client.execute(
- """INSERT INTO writable_events (team_id, distinct_id, person_id, timestamp, properties)
- VALUES""",
- events,
- )
-
- cluster.any_host(insert_events).result()
-
- # Person with different value for "name" so set_diff includes it
- person_data = [(team_id, person_id, json.dumps({"name": "Old Name"}), 1, now - timedelta(days=8))]
-
- def insert_person(client: Client) -> None:
- client.execute(
- """INSERT INTO person (team_id, id, properties, version, _timestamp)
- VALUES""",
- person_data,
- )
-
- cluster.any_host(insert_person).result()
-
- results = get_person_property_updates_from_clickhouse(
- team_id=team_id,
- bug_window_start=bug_window_start.strftime("%Y-%m-%d %H:%M:%S"),
- )
-
- assert len(results) == 1
- person_diffs = results[0]
- assert len(person_diffs.set_updates) == 1
- assert "name" in person_diffs.set_updates
-
- # Should have latest value and latest timestamp
- assert person_diffs.set_updates["name"].value == "Latest Name"
- result_ts = person_diffs.set_updates["name"].timestamp.replace(tzinfo=None)
- assert abs((result_ts - later_ts).total_seconds()) < 1
-
- def test_set_once_uses_earliest_timestamp(self, cluster: ClickhouseCluster):
- """$set_once should use min(timestamp) - first value wins."""
-
- team_id = 99903
- person_id = UUID("33333333-3333-3333-3333-000000000003")
- # Use naive datetimes since ClickHouse returns naive datetimes
- now = datetime.now().replace(microsecond=0)
- bug_window_start = now - timedelta(days=10)
-
- earlier_ts = now - timedelta(days=5)
- later_ts = now - timedelta(days=2)
-
- events = [
- (team_id, "distinct_id_3", person_id, earlier_ts, json.dumps({"$set_once": {"referrer": "google.com"}})),
- (team_id, "distinct_id_3", person_id, later_ts, json.dumps({"$set_once": {"referrer": "facebook.com"}})),
- ]
-
- def insert_events(client: Client) -> None:
- client.execute(
- """INSERT INTO writable_events (team_id, distinct_id, person_id, timestamp, properties)
- VALUES""",
- events,
- )
-
- cluster.any_host(insert_events).result()
-
- # Person WITHOUT "referrer" so set_once_diff includes it
- person_data = [(team_id, person_id, json.dumps({"other_prop": "value"}), 1, now - timedelta(days=8))]
-
- def insert_person(client: Client) -> None:
- client.execute(
- """INSERT INTO person (team_id, id, properties, version, _timestamp)
- VALUES""",
- person_data,
- )
-
- cluster.any_host(insert_person).result()
-
- results = get_person_property_updates_from_clickhouse(
- team_id=team_id,
- bug_window_start=bug_window_start.strftime("%Y-%m-%d %H:%M:%S"),
- )
-
- assert len(results) == 1
- person_diffs = results[0]
- assert len(person_diffs.set_once_updates) == 1
- assert "referrer" in person_diffs.set_once_updates
-
- # Should have first value and earliest timestamp
- assert person_diffs.set_once_updates["referrer"].value == "google.com"
- result_ts = person_diffs.set_once_updates["referrer"].timestamp.replace(tzinfo=None)
- assert abs((result_ts - earlier_ts).total_seconds()) < 1
-
- def test_multiple_operations_same_key_in_batch(self, cluster: ClickhouseCluster):
- """
- Test multiple operation types on same key: $set then $unset.
-
- Both operations are for the same key but different operation types.
- Each should be returned separately.
- """
-
- team_id = 99904
- person_id = UUID("44444444-4444-4444-4444-000000000004")
- # Use naive datetimes since ClickHouse returns naive datetimes
- now = datetime.now().replace(microsecond=0)
- bug_window_start = now - timedelta(days=10)
-
- set_ts = now - timedelta(days=5)
- unset_ts = now - timedelta(days=2)
-
- events = [
- # First $set the property
- (team_id, "distinct_id_4", person_id, set_ts, json.dumps({"$set": {"email": "[email protected]"}})),
- # Then $unset the same property (later)
- (team_id, "distinct_id_4", person_id, unset_ts, json.dumps({"$unset": ["email"]})),
- ]
-
- def insert_events(client: Client) -> None:
- client.execute(
- """INSERT INTO writable_events (team_id, distinct_id, person_id, timestamp, properties)
- VALUES""",
- events,
- )
-
- cluster.any_host(insert_events).result()
-
- # Person with existing email (different value so $set is in diff, and exists so $unset is in diff)
- person_data = [(team_id, person_id, json.dumps({"email": "[email protected]"}), 1, now - timedelta(days=8))]
-
- def insert_person(client: Client) -> None:
- client.execute(
- """INSERT INTO person (team_id, id, properties, version, _timestamp)
- VALUES""",
- person_data,
- )
-
- cluster.any_host(insert_person).result()
-
- results = get_person_property_updates_from_clickhouse(
- team_id=team_id,
- bug_window_start=bug_window_start.strftime("%Y-%m-%d %H:%M:%S"),
- )
-
- assert len(results) == 1
- person_diffs = results[0]
-
- # Both should be returned - $set with value, $unset without
- assert len(person_diffs.set_updates) == 1
- assert "email" in person_diffs.set_updates
- assert person_diffs.set_updates["email"].value == "[email protected]"
-
- assert len(person_diffs.unset_updates) == 1
- assert "email" in person_diffs.unset_updates
- assert person_diffs.unset_updates["email"].value is None
-
- def test_mixed_operations_different_keys(self, cluster: ClickhouseCluster):
- """Test $set, $set_once, and $unset on different keys in same event."""
-
- team_id = 99905
- person_id = UUID("55555555-5555-5555-5555-000000000005")
- # Use naive datetimes since ClickHouse returns naive datetimes
- now = datetime.now().replace(microsecond=0)
- bug_window_start = now - timedelta(days=10)
-
- event_ts = now - timedelta(days=3)
-
- events = [
- (
- team_id,
- "distinct_id_5",
- person_id,
- event_ts,
- json.dumps(
- {
- "$set": {"name": "New Name"},
- "$set_once": {"first_visit": "2024-01-01"},
- "$unset": ["deprecated_field"],
- }
- ),
- ),
- ]
-
- def insert_events(client: Client) -> None:
- client.execute(
- """INSERT INTO writable_events (team_id, distinct_id, person_id, timestamp, properties)
- VALUES""",
- events,
- )
-
- cluster.any_host(insert_events).result()
-
- # Person with name (different), no first_visit, has deprecated_field
- person_data = [
- (
- team_id,
- person_id,
- json.dumps({"name": "Old Name", "deprecated_field": "to_remove"}),
- 1,
- now - timedelta(days=8),
- )
- ]
-
- def insert_person(client: Client) -> None:
- client.execute(
- """INSERT INTO person (team_id, id, properties, version, _timestamp)
- VALUES""",
- person_data,
- )
-
- cluster.any_host(insert_person).result()
-
- results = get_person_property_updates_from_clickhouse(
- team_id=team_id,
- bug_window_start=bug_window_start.strftime("%Y-%m-%d %H:%M:%S"),
- )
-
- assert len(results) == 1
- person_diffs = results[0]
-
- assert len(person_diffs.set_updates) == 1
- assert "name" in person_diffs.set_updates
- assert person_diffs.set_updates["name"].value == "New Name"
-
- assert len(person_diffs.set_once_updates) == 1
- assert "first_visit" in person_diffs.set_once_updates
- assert person_diffs.set_once_updates["first_visit"].value == "2024-01-01"
-
- assert len(person_diffs.unset_updates) == 1
- assert "deprecated_field" in person_diffs.unset_updates
- assert person_diffs.unset_updates["deprecated_field"].value is None
-
- def test_set_with_various_json_types(self, cluster: ClickhouseCluster):
- """Test $set with various JSON value types: string, number, boolean, null, array, object."""
- team_id = 99906
- person_id = UUID("66666666-6666-6666-6666-000000000006")
- now = datetime.now().replace(microsecond=0)
- bug_window_start = now - timedelta(days=10)
-
- event_ts = now - timedelta(days=3)
-
- # Event with various JSON types in $set
- # Note: null values are filtered out in the query (use $unset for removal)
- events = [
- (
- team_id,
- "distinct_id_6",
- person_id,
- event_ts,
- json.dumps(
- {
- "$set": {
- "string_prop": "hello world",
- "int_prop": 42,
- "float_prop": 3.14159,
- "bool_true_prop": True,
- "bool_false_prop": False,
- "array_prop": [1, "two", True],
- "object_prop": {"nested": "value", "count": 123},
- }
- }
- ),
- ),
- ]
-
- def insert_events(client):
- client.execute(
- """INSERT INTO writable_events (team_id, distinct_id, person_id, timestamp, properties)
- VALUES""",
- events,
- )
-
- cluster.any_host(insert_events).result()
-
- # Person with different values for all properties (so set_diff includes them)
- person_data = [
- (
- team_id,
- person_id,
- json.dumps(
- {
- "string_prop": "old",
- "int_prop": 0,
- "float_prop": 0.0,
- "bool_true_prop": False,
- "bool_false_prop": True,
- "array_prop": [],
- "object_prop": {},
- }
- ),
- 1,
- now - timedelta(days=8),
- )
- ]
-
- def insert_person(client):
- client.execute(
- """INSERT INTO person (team_id, id, properties, version, _timestamp)
- VALUES""",
- person_data,
- )
-
- cluster.any_host(insert_person).result()
-
- results = get_person_property_updates_from_clickhouse(
- team_id=team_id,
- bug_window_start=bug_window_start.strftime("%Y-%m-%d %H:%M:%S"),
- )
-
- assert len(results) == 1
- updates = {key: pv.value for key, pv in results[0].set_updates.items()}
-
- # String
- assert updates["string_prop"] == "hello world"
- assert isinstance(updates["string_prop"], str)
-
- # Integer
- assert updates["int_prop"] == 42
- assert isinstance(updates["int_prop"], int)
-
- # Float
- assert updates["float_prop"] == 3.14159
- assert isinstance(updates["float_prop"], float)
-
- # Boolean true
- assert updates["bool_true_prop"] is True
- assert isinstance(updates["bool_true_prop"], bool)
-
- # Boolean false
- assert updates["bool_false_prop"] is False
- assert isinstance(updates["bool_false_prop"], bool)
-
- # Array
- assert updates["array_prop"] == [1, "two", True]
- assert isinstance(updates["array_prop"], list)
-
- # Object
- assert updates["object_prop"] == {"nested": "value", "count": 123}
- assert isinstance(updates["object_prop"], dict)
-
- def test_set_with_null_value_is_filtered_out(self, cluster: ClickhouseCluster):
- """Test that $set with null value is filtered out (use $unset for removal)."""
- team_id = 99907
- person_id = UUID("77777777-7777-7777-7777-000000000007")
- now = datetime.now().replace(microsecond=0)
- bug_window_start = now - timedelta(days=10)
-
- event_ts = now - timedelta(days=3)
-
- # Event with null value in $set - should be filtered out
- events = [
- (
- team_id,
- "distinct_id_7",
- person_id,
- event_ts,
- json.dumps(
- {
- "$set": {
- "keep_prop": "value",
- "null_prop": None, # This should be filtered out
- }
- }
- ),
- ),
- ]
-
- def insert_events(client):
- client.execute(
- """INSERT INTO writable_events (team_id, distinct_id, person_id, timestamp, properties)
- VALUES""",
- events,
- )
-
- cluster.any_host(insert_events).result()
-
- # Person with different values (so set_diff includes them)
- person_data = [
- (
- team_id,
- person_id,
- json.dumps(
- {
- "keep_prop": "old_value",
- "null_prop": "existing_value", # This exists, but $set null should NOT update it
- }
- ),
- 1,
- now - timedelta(days=8),
- )
- ]
-
- def insert_person(client):
- client.execute(
- """INSERT INTO person (team_id, id, properties, version, _timestamp)
- VALUES""",
- person_data,
- )
-
- cluster.any_host(insert_person).result()
-
- results = get_person_property_updates_from_clickhouse(
- team_id=team_id,
- bug_window_start=bug_window_start.strftime("%Y-%m-%d %H:%M:%S"),
- )
-
- assert len(results) == 1
- updates = {key: pv.value for key, pv in results[0].set_updates.items()}
-
- # keep_prop should be included
- assert "keep_prop" in updates
- assert updates["keep_prop"] == "value"
-
- # null_prop should NOT be included (filtered out because value is null)
- assert "null_prop" not in updates, (
- "$set with null value should be filtered out. Use $unset to remove properties."
- )
-
- def test_set_once_with_various_json_types(self, cluster: ClickhouseCluster):
- """Test $set_once with various JSON value types: string, number, boolean, array, object."""
- team_id = 99908
- person_id = UUID("88888888-8888-8888-8888-000000000008")
- now = datetime.now().replace(microsecond=0)
- bug_window_start = now - timedelta(days=10)
-
- event_ts = now - timedelta(days=3)
-
- # Event with various JSON types in $set_once
- # Note: null values are filtered out in the query
- events = [
- (
- team_id,
- "distinct_id_8",
- person_id,
- event_ts,
- json.dumps(
- {
- "$set_once": {
- "string_prop": "hello world",
- "int_prop": 42,
- "float_prop": 3.14159,
- "bool_true_prop": True,
- "bool_false_prop": False,
- "array_prop": [1, "two", True],
- "object_prop": {"nested": "value", "count": 123},
- }
- }
- ),
- ),
- ]
-
- def insert_events(client):
- client.execute(
- """INSERT INTO writable_events (team_id, distinct_id, person_id, timestamp, properties)
- VALUES""",
- events,
- )
-
- cluster.any_host(insert_events).result()
-
- # Person WITHOUT these properties (so set_once_diff includes them)
- person_data = [
- (
- team_id,
- person_id,
- json.dumps({"other_prop": "value"}), # None of the $set_once keys exist
- 1,
- now - timedelta(days=8),
- )
- ]
-
- def insert_person(client):
- client.execute(
- """INSERT INTO person (team_id, id, properties, version, _timestamp)
- VALUES""",
- person_data,
- )
-
- cluster.any_host(insert_person).result()
-
- results = get_person_property_updates_from_clickhouse(
- team_id=team_id,
- bug_window_start=bug_window_start.strftime("%Y-%m-%d %H:%M:%S"),
- )
-
- assert len(results) == 1
- updates = {key: pv.value for key, pv in results[0].set_once_updates.items()}
-
- # String
- assert updates["string_prop"] == "hello world"
- assert isinstance(updates["string_prop"], str)
-
- # Integer
- assert updates["int_prop"] == 42
- assert isinstance(updates["int_prop"], int)
-
- # Float
- assert updates["float_prop"] == 3.14159
- assert isinstance(updates["float_prop"], float)
-
- # Boolean true
- assert updates["bool_true_prop"] is True
- assert isinstance(updates["bool_true_prop"], bool)
-
- # Boolean false
- assert updates["bool_false_prop"] is False
- assert isinstance(updates["bool_false_prop"], bool)
-
- # Array
- assert updates["array_prop"] == [1, "two", True]
- assert isinstance(updates["array_prop"], list)
-
- # Object
- assert updates["object_prop"] == {"nested": "value", "count": 123}
- assert isinstance(updates["object_prop"], dict)
-
- def test_set_once_with_null_value_is_filtered_out(self, cluster: ClickhouseCluster):
- """Test that $set_once with null value is filtered out."""
- team_id = 99909
- person_id = UUID("99999999-9999-9999-9999-000000000009")
- now = datetime.now().replace(microsecond=0)
- bug_window_start = now - timedelta(days=10)
-
- event_ts = now - timedelta(days=3)
-
- # Event with null value in $set_once - should be filtered out
- events = [
- (
- team_id,
- "distinct_id_9",
- person_id,
- event_ts,
- json.dumps(
- {
- "$set_once": {
- "keep_prop": "value",
- "null_prop": None, # This should be filtered out
- }
- }
- ),
- ),
- ]
-
- def insert_events(client):
- client.execute(
- """INSERT INTO writable_events (team_id, distinct_id, person_id, timestamp, properties)
- VALUES""",
- events,
- )
-
- cluster.any_host(insert_events).result()
-
- # Person without these properties (so set_once_diff would include them if not filtered)
- person_data = [
- (
- team_id,
- person_id,
- json.dumps({"other_prop": "value"}),
- 1,
- now - timedelta(days=8),
- )
- ]
-
- def insert_person(client):
- client.execute(
- """INSERT INTO person (team_id, id, properties, version, _timestamp)
- VALUES""",
- person_data,
- )
-
- cluster.any_host(insert_person).result()
-
- results = get_person_property_updates_from_clickhouse(
- team_id=team_id,
- bug_window_start=bug_window_start.strftime("%Y-%m-%d %H:%M:%S"),
- )
-
- assert len(results) == 1
- updates = {key: pv.value for key, pv in results[0].set_once_updates.items()}
-
- # keep_prop should be included
- assert "keep_prop" in updates
- assert updates["keep_prop"] == "value"
-
- # null_prop should NOT be included (filtered out because value is null)
- assert "null_prop" not in updates, "$set_once with null value should be filtered out."
-
- # ==================== Person Merge Tests ====================
-
- def test_person_merge_uses_override_person_id(self, cluster: ClickhouseCluster):
- """
- When a distinct_id has an override in person_distinct_id_overrides,
- events should be attributed to the override person_id, not the original.
-
- This test verifies that:
- 1. Events with an overridden distinct_id are attributed to the merged person
- 2. The $set property update is correctly associated with the merged person
- """
- team_id = 99910
- original_person_id = UUID("aaaa0000-0000-0000-0000-000000000001")
- merged_person_id = UUID("aaaa0000-0000-0000-0000-000000000002")
- now = datetime.now().replace(microsecond=0)
- bug_window_start = now - timedelta(days=10)
-
- event_ts = now - timedelta(days=3)
-
- # Event with original person_id, but distinct_id will be overridden
- # Using $set on a property that EXISTS in the merged person but has DIFFERENT value
- events = [
- (
- team_id,
- "merged_distinct_id",
- original_person_id, # Original person_id in event
- event_ts,
- json.dumps({"$set": {"existing": "new_value"}}), # Update existing property
- ),
- ]
-
- def insert_events(client):
- client.execute(
- """INSERT INTO writable_events (team_id, distinct_id, person_id, timestamp, properties)
- VALUES""",
- events,
- )
-
- cluster.any_host(insert_events).result()
-
- # Insert override: merged_distinct_id -> merged_person_id
- # Must include _timestamp for the data to be visible
- override_data = [
- (
- team_id,
- "merged_distinct_id",
- merged_person_id, # Override to this person
- now - timedelta(days=4), # _timestamp - must be provided
- 1, # version
- 0, # is_deleted = false
- ),
- ]
-
- def insert_override(client):
- client.execute(
- """INSERT INTO person_distinct_id_overrides
- (team_id, distinct_id, person_id, _timestamp, version, is_deleted)
- VALUES""",
- override_data,
- )
-
- cluster.any_host(insert_override).result()
-
- # Insert the MERGED person (not the original)
- # Has "existing" property with old value that event will update
- person_data = [
- (
- team_id,
- merged_person_id,
- json.dumps({"existing": "old_value"}),
- 1,
- now - timedelta(days=8),
- )
- ]
-
- def insert_person(client):
- client.execute(
- """INSERT INTO person (team_id, id, properties, version, _timestamp)
- VALUES""",
- person_data,
- )
-
- cluster.any_host(insert_person).result()
-
- results = get_person_property_updates_from_clickhouse(
- team_id=team_id,
- bug_window_start=bug_window_start.strftime("%Y-%m-%d %H:%M:%S"),
- )
-
- # Should get results for merged_person_id, not original_person_id
- assert len(results) == 1
- assert results[0].person_id == str(merged_person_id)
-
- updates = {key: pv.value for key, pv in results[0].set_updates.items()}
- assert "existing" in updates
- assert updates["existing"] == "new_value"
-
- def test_multiple_distinct_ids_same_person(self, cluster: ClickhouseCluster):
- """
- Events from multiple distinct_ids that map to the same person
- should be aggregated together.
- """
- team_id = 99911
- person_id = UUID("bbbb0000-0000-0000-0000-000000000001")
- now = datetime.now().replace(microsecond=0)
- bug_window_start = now - timedelta(days=10)
-
- earlier_ts = now - timedelta(days=5)
- later_ts = now - timedelta(days=2)
-
- # Two events from different distinct_ids, same person
- events = [
- (team_id, "distinct_id_A", person_id, earlier_ts, json.dumps({"$set": {"name": "First"}})),
- (team_id, "distinct_id_B", person_id, later_ts, json.dumps({"$set": {"name": "Latest"}})),
- ]
-
- def insert_events(client):
- client.execute(
- """INSERT INTO writable_events (team_id, distinct_id, person_id, timestamp, properties)
- VALUES""",
- events,
- )
-
- cluster.any_host(insert_events).result()
-
- person_data = [(team_id, person_id, json.dumps({"name": "Old"}), 1, now - timedelta(days=8))]
-
- def insert_person(client):
- client.execute(
- """INSERT INTO person (team_id, id, properties, version, _timestamp)
- VALUES""",
- person_data,
- )
-
- cluster.any_host(insert_person).result()
-
- results = get_person_property_updates_from_clickhouse(
- team_id=team_id,
- bug_window_start=bug_window_start.strftime("%Y-%m-%d %H:%M:%S"),
- )
-
- # Should have one result with the latest value (aggregated from both distinct_ids)
- assert len(results) == 1
- updates = {key: pv.value for key, pv in results[0].set_updates.items()}
- assert updates["name"] == "Latest"
-
- def test_deleted_override_not_used(self, cluster: ClickhouseCluster):
- """
- When an override has is_deleted=1, it should not be used.
- Events should use the original person_id from the event.
- """
- team_id = 99920
- original_person_id = UUID("eeee0000-0000-0000-0000-000000000001")
- merged_person_id = UUID("eeee0000-0000-0000-0000-000000000002")
- now = datetime.now().replace(microsecond=0)
- bug_window_start = now - timedelta(days=10)
-
- event_ts = now - timedelta(days=3)
-
- events = [
- (
- team_id,
- "deleted_override_distinct",
- original_person_id,
- event_ts,
- json.dumps({"$set": {"prop": "value"}}),
- ),
- ]
-
- def insert_events(client):
- client.execute(
- """INSERT INTO writable_events (team_id, distinct_id, person_id, timestamp, properties)
- VALUES""",
- events,
- )
-
- cluster.any_host(insert_events).result()
-
- # Insert a DELETED override (is_deleted=1)
- override_data = [
- (
- team_id,
- "deleted_override_distinct",
- merged_person_id,
- now - timedelta(days=4), # _timestamp
- 1, # version
- 1, # is_deleted = TRUE
- ),
- ]
-
- def insert_override(client):
- client.execute(
- """INSERT INTO person_distinct_id_overrides
- (team_id, distinct_id, person_id, _timestamp, version, is_deleted)
- VALUES""",
- override_data,
- )
-
- cluster.any_host(insert_override).result()
-
- # Insert the ORIGINAL person (not the merged one)
- person_data = [(team_id, original_person_id, json.dumps({"prop": "old"}), 1, now - timedelta(days=8))]
-
- def insert_person(client):
- client.execute(
- """INSERT INTO person (team_id, id, properties, version, _timestamp)
- VALUES""",
- person_data,
- )
-
- cluster.any_host(insert_person).result()
-
- results = get_person_property_updates_from_clickhouse(
- team_id=team_id,
- bug_window_start=bug_window_start.strftime("%Y-%m-%d %H:%M:%S"),
- )
-
- # Since override is deleted, should use original_person_id
- assert len(results) == 1
- assert results[0].person_id == str(original_person_id)
-
- def test_override_version_ordering(self, cluster: ClickhouseCluster):
- """
- When multiple override versions exist, the highest version should win.
- """
- team_id = 99921
- original_person_id = UUID("ffff0000-0000-0000-0000-000000000001")
- first_merged_person_id = UUID("ffff0000-0000-0000-0000-000000000002")
- final_merged_person_id = UUID("ffff0000-0000-0000-0000-000000000003")
- now = datetime.now().replace(microsecond=0)
- bug_window_start = now - timedelta(days=10)
-
- event_ts = now - timedelta(days=3)
-
- events = [
- (
- team_id,
- "multi_version_distinct",
- original_person_id,
- event_ts,
- json.dumps({"$set": {"prop": "value"}}),
- ),
- ]
-
- def insert_events(client):
- client.execute(
- """INSERT INTO writable_events (team_id, distinct_id, person_id, timestamp, properties)
- VALUES""",
- events,
- )
-
- cluster.any_host(insert_events).result()
-
- # Insert multiple override versions - version 2 should win
- override_data = [
- (team_id, "multi_version_distinct", first_merged_person_id, now - timedelta(days=5), 1, 0),
- (team_id, "multi_version_distinct", final_merged_person_id, now - timedelta(days=4), 2, 0),
- ]
-
- def insert_override(client):
- client.execute(
- """INSERT INTO person_distinct_id_overrides
- (team_id, distinct_id, person_id, _timestamp, version, is_deleted)
- VALUES""",
- override_data,
- )
-
- cluster.any_host(insert_override).result()
-
- # Insert the FINAL merged person
- person_data = [(team_id, final_merged_person_id, json.dumps({"prop": "old"}), 1, now - timedelta(days=8))]
-
- def insert_person(client):
- client.execute(
- """INSERT INTO person (team_id, id, properties, version, _timestamp)
- VALUES""",
- person_data,
- )
-
- cluster.any_host(insert_person).result()
-
- results = get_person_property_updates_from_clickhouse(
- team_id=team_id,
- bug_window_start=bug_window_start.strftime("%Y-%m-%d %H:%M:%S"),
- )
-
- # Should use final_merged_person_id (version 2)
- assert len(results) == 1
- assert results[0].person_id == str(final_merged_person_id)
-
- def test_deleted_person_filtered_out(self, cluster: ClickhouseCluster):
- """
- Persons with is_deleted=1 in the person table should be filtered out.
- There's no point updating properties on a deleted person.
- """
- team_id = 99922
- person_id = UUID("11110000-0000-0000-0000-000000000001")
- now = datetime.now().replace(microsecond=0)
- bug_window_start = now - timedelta(days=10)
-
- event_ts = now - timedelta(days=3)
-
- events = [
- (team_id, "deleted_person_distinct", person_id, event_ts, json.dumps({"$set": {"prop": "new_value"}})),
- ]
-
- def insert_events(client):
- client.execute(
- """INSERT INTO writable_events (team_id, distinct_id, person_id, timestamp, properties)
- VALUES""",
- events,
- )
-
- cluster.any_host(insert_events).result()
-
- # Insert a DELETED person (is_deleted=1)
- person_data = [
- (
- team_id,
- person_id,
- json.dumps({"prop": "old_value"}),
- 1, # version
- 1, # is_deleted = TRUE
- now - timedelta(days=8), # _timestamp
- )
- ]
-
- def insert_person(client):
- client.execute(
- """INSERT INTO person (team_id, id, properties, version, is_deleted, _timestamp)
- VALUES""",
- person_data,
- )
-
- cluster.any_host(insert_person).result()
-
- results = get_person_property_updates_from_clickhouse(
- team_id=team_id,
- bug_window_start=bug_window_start.strftime("%Y-%m-%d %H:%M:%S"),
- )
-
- # Deleted persons should NOT be processed
- assert len(results) == 0
-
- def test_person_multiple_versions_uses_latest(self, cluster: ClickhouseCluster):
- """
- When a person has multiple versions in CH, argMax(properties, version)
- should select the properties from the highest version.
- """
- team_id = 99923
- person_id = UUID("22220000-0000-0000-0000-000000000002")
- now = datetime.now().replace(microsecond=0)
- bug_window_start = now - timedelta(days=10)
-
- event_ts = now - timedelta(days=3)
-
- events = [
- (
- team_id,
- "multi_version_person_distinct",
- person_id,
- event_ts,
- json.dumps({"$set": {"prop": "event_value"}}),
- ),
- ]
-
- def insert_events(client):
- client.execute(
- """INSERT INTO writable_events (team_id, distinct_id, person_id, timestamp, properties)
- VALUES""",
- events,
- )
-
- cluster.any_host(insert_events).result()
-
- # Insert multiple versions of the same person
- # Version 2 has different properties than version 1
- person_data = [
- (team_id, person_id, json.dumps({"prop": "v1_value"}), 1, now - timedelta(days=8)),
- (team_id, person_id, json.dumps({"prop": "v2_value"}), 2, now - timedelta(days=7)),
- ]
-
- def insert_person(client):
- client.execute(
- """INSERT INTO person (team_id, id, properties, version, _timestamp)
- VALUES""",
- person_data,
- )
-
- cluster.any_host(insert_person).result()
-
- results = get_person_property_updates_from_clickhouse(
- team_id=team_id,
- bug_window_start=bug_window_start.strftime("%Y-%m-%d %H:%M:%S"),
- )
-
- # Should compare against v2_value (latest version)
- # Event has "event_value", person v2 has "v2_value" - different, so should be in diff
- assert len(results) == 1
- updates = {key: pv.value for key, pv in results[0].set_updates.items()}
- assert updates["prop"] == "event_value"
-
- # ==================== Same Key Operation Tests ====================
-
- def test_set_and_set_once_on_same_key(self, cluster: ClickhouseCluster):
- """
- When $set and $set_once both target the same key, both should be returned
- as separate operations (query doesn't merge them, Python logic handles precedence).
- """
- team_id = 99912
- person_id = UUID("cccc0000-0000-0000-0000-000000000001")
- now = datetime.now().replace(microsecond=0)
- bug_window_start = now - timedelta(days=10)
-
- set_once_ts = now - timedelta(days=5)
- set_ts = now - timedelta(days=2)
-
- events = [
- (team_id, "distinct_1", person_id, set_once_ts, json.dumps({"$set_once": {"email": "[email protected]"}})),
- (team_id, "distinct_1", person_id, set_ts, json.dumps({"$set": {"email": "[email protected]"}})),
- ]
-
- def insert_events(client):
- client.execute(
- """INSERT INTO writable_events (team_id, distinct_id, person_id, timestamp, properties)
- VALUES""",
- events,
- )
-
- cluster.any_host(insert_events).result()
-
- # Person WITHOUT email (so set_once_diff includes it)
- # Person WITH different email for set_diff
- # Actually, to get both in results, we need person to NOT have email (for set_once)
- # and have a different email (for set). That's contradictory.
- # Let's test with person having a different email - set_diff should include it,
- # set_once_diff should NOT (key exists).
- person_data = [(team_id, person_id, json.dumps({"email": "[email protected]"}), 1, now - timedelta(days=8))]
-
- def insert_person(client):
- client.execute(
- """INSERT INTO person (team_id, id, properties, version, _timestamp)
- VALUES""",
- person_data,
- )
-
- cluster.any_host(insert_person).result()
-
- results = get_person_property_updates_from_clickhouse(
- team_id=team_id,
- bug_window_start=bug_window_start.strftime("%Y-%m-%d %H:%M:%S"),
- )
-
- assert len(results) == 1
-
- # $set should be included (value differs from person)
- assert len(results[0].set_updates) == 1
- assert "email" in results[0].set_updates
- assert results[0].set_updates["email"].value == "[email protected]"
-
- # $set_once should NOT be included (key already exists in person)
- assert len(results[0].set_once_updates) == 0
-
- # ==================== Diff Filtering Tests ====================
-
- def test_set_with_same_value_not_in_diff(self, cluster: ClickhouseCluster):
- """
- When $set value equals current person property value,
- it should NOT be in set_diff (no change needed).
- """
- team_id = 99913
- person_id = UUID("dddd0000-0000-0000-0000-000000000001")
- now = datetime.now().replace(microsecond=0)
- bug_window_start = now - timedelta(days=10)
-
- event_ts = now - timedelta(days=3)
-
- events = [
- (team_id, "distinct_1", person_id, event_ts, json.dumps({"$set": {"name": "Same Value"}})),
- ]
-
- def insert_events(client):
- client.execute(
- """INSERT INTO writable_events (team_id, distinct_id, person_id, timestamp, properties)
- VALUES""",
- events,
- )
-
- cluster.any_host(insert_events).result()
-
- # Person with SAME value
- person_data = [(team_id, person_id, json.dumps({"name": "Same Value"}), 1, now - timedelta(days=8))]
-
- def insert_person(client):
- client.execute(
- """INSERT INTO person (team_id, id, properties, version, _timestamp)
- VALUES""",
- person_data,
- )
-
- cluster.any_host(insert_person).result()
-
- results = get_person_property_updates_from_clickhouse(
- team_id=team_id,
- bug_window_start=bug_window_start.strftime("%Y-%m-%d %H:%M:%S"),
- )
-
- # Should be empty - no diff when values are the same
- assert len(results) == 0
-
- def test_set_on_missing_property_not_in_diff(self, cluster: ClickhouseCluster):
- """
- When $set targets a property that doesn't exist in person,
- it should NOT be in set_diff (set_diff only updates existing different values).
- """
- team_id = 99914
- person_id = UUID("eeee0000-0000-0000-0000-000000000001")
- now = datetime.now().replace(microsecond=0)
- bug_window_start = now - timedelta(days=10)
-
- event_ts = now - timedelta(days=3)
-
- events = [
- (team_id, "distinct_1", person_id, event_ts, json.dumps({"$set": {"new_prop": "value"}})),
- ]
-
- def insert_events(client):
- client.execute(
- """INSERT INTO writable_events (team_id, distinct_id, person_id, timestamp, properties)
- VALUES""",
- events,
- )
-
- cluster.any_host(insert_events).result()
-
- # Person WITHOUT new_prop
- person_data = [(team_id, person_id, json.dumps({"other_prop": "value"}), 1, now - timedelta(days=8))]
-
- def insert_person(client):
- client.execute(
- """INSERT INTO person (team_id, id, properties, version, _timestamp)
- VALUES""",
- person_data,
- )
-
- cluster.any_host(insert_person).result()
-
- results = get_person_property_updates_from_clickhouse(
- team_id=team_id,
- bug_window_start=bug_window_start.strftime("%Y-%m-%d %H:%M:%S"),
- )
-
- # Should be empty - set_diff doesn't include missing properties
- assert len(results) == 0
-
- def test_unset_on_missing_property_not_in_diff(self, cluster: ClickhouseCluster):
- """
- When $unset targets a property that doesn't exist in person,
- it should NOT be in unset_diff.
- """
- team_id = 99915
- person_id = UUID("ffff0000-0000-0000-0000-000000000001")
- now = datetime.now().replace(microsecond=0)
- bug_window_start = now - timedelta(days=10)
-
- event_ts = now - timedelta(days=3)
-
- events = [
- (team_id, "distinct_1", person_id, event_ts, json.dumps({"$unset": ["nonexistent_prop"]})),
- ]
-
- def insert_events(client):
- client.execute(
- """INSERT INTO writable_events (team_id, distinct_id, person_id, timestamp, properties)
- VALUES""",
- events,
- )
-
- cluster.any_host(insert_events).result()
-
- # Person WITHOUT nonexistent_prop
- person_data = [(team_id, person_id, json.dumps({"other_prop": "value"}), 1, now - timedelta(days=8))]
-
- def insert_person(client):
- client.execute(
- """INSERT INTO person (team_id, id, properties, version, _timestamp)
- VALUES""",
- person_data,
- )
-
- cluster.any_host(insert_person).result()
-
- results = get_person_property_updates_from_clickhouse(
- team_id=team_id,
- bug_window_start=bug_window_start.strftime("%Y-%m-%d %H:%M:%S"),
- )
-
- # Should be empty - unset_diff doesn't include missing properties
- assert len(results) == 0
-
- # ==================== Empty Value Tests ====================
-
- def test_empty_set_object_no_results(self, cluster: ClickhouseCluster):
- """Empty $set object should not cause issues or return results."""
- team_id = 99916
- person_id = UUID("11110000-0000-0000-0000-000000000001")
- now = datetime.now().replace(microsecond=0)
- bug_window_start = now - timedelta(days=10)
-
- event_ts = now - timedelta(days=3)
-
- events = [
- (team_id, "distinct_1", person_id, event_ts, json.dumps({"$set": {}})),
- ]
-
- def insert_events(client):
- client.execute(
- """INSERT INTO writable_events (team_id, distinct_id, person_id, timestamp, properties)
- VALUES""",
- events,
- )
-
- cluster.any_host(insert_events).result()
-
- person_data = [(team_id, person_id, json.dumps({"prop": "value"}), 1, now - timedelta(days=8))]
-
- def insert_person(client):
- client.execute(
- """INSERT INTO person (team_id, id, properties, version, _timestamp)
- VALUES""",
- person_data,
- )
-
- cluster.any_host(insert_person).result()
-
- results = get_person_property_updates_from_clickhouse(
- team_id=team_id,
- bug_window_start=bug_window_start.strftime("%Y-%m-%d %H:%M:%S"),
- )
-
- # Empty $set should produce no results
- assert len(results) == 0
-
- def test_empty_string_value_is_valid(self, cluster: ClickhouseCluster):
- """Empty string is a valid value in $set - different from null."""
- team_id = 99917
- person_id = UUID("22220000-0000-0000-0000-000000000001")
- now = datetime.now().replace(microsecond=0)
- bug_window_start = now - timedelta(days=10)
-
- event_ts = now - timedelta(days=3)
-
- events = [
- (
- team_id,
- "distinct_1",
- person_id,
- event_ts,
- json.dumps({"$set": {"empty_prop": "", "valid_prop": "value"}}),
- ),
- ]
-
- def insert_events(client):
- client.execute(
- """INSERT INTO writable_events (team_id, distinct_id, person_id, timestamp, properties)
- VALUES""",
- events,
- )
-
- cluster.any_host(insert_events).result()
-
- person_data = [
- (team_id, person_id, json.dumps({"empty_prop": "old", "valid_prop": "old"}), 1, now - timedelta(days=8))
- ]
-
- def insert_person(client):
- client.execute(
- """INSERT INTO person (team_id, id, properties, version, _timestamp)
- VALUES""",
- person_data,
- )
-
- cluster.any_host(insert_person).result()
-
- results = get_person_property_updates_from_clickhouse(
- team_id=team_id,
- bug_window_start=bug_window_start.strftime("%Y-%m-%d %H:%M:%S"),
- )
-
- assert len(results) == 1
- updates = {key: pv.value for key, pv in results[0].set_updates.items()}
-
- # valid_prop should be included (changed from "old" to "value")
- assert "valid_prop" in updates
- assert updates["valid_prop"] == "value"
-
- # empty_prop should be included - empty string is a valid value, not filtered like null
- # It changes from "old" to "" (empty string)
- assert "empty_prop" in updates
- assert updates["empty_prop"] == ""
-
- # ==================== Window Filtering Tests ====================
-
- def test_events_before_bug_window_start_filtered(self, cluster: ClickhouseCluster):
- """Events with timestamp before bug_window_start should be filtered.
-
- Events after bug_window_start are included (up to now()).
- """
- team_id = 99918
- person_id = UUID("33330000-0000-0000-0000-000000000001")
- now = datetime.now().replace(microsecond=0)
- bug_window_start = now - timedelta(days=5)
-
- # Event BEFORE bug_window_start - should be filtered
- before_ts = now - timedelta(days=7)
- # Event AFTER bug_window_start - should be included (reads up to now())
- after_start_ts = now - timedelta(days=1)
- # Event IN window (also after bug_window_start)
- in_window_ts = now - timedelta(days=3)
-
- events = [
- (team_id, "distinct_1", person_id, before_ts, json.dumps({"$set": {"before": "value"}})),
- (team_id, "distinct_1", person_id, after_start_ts, json.dumps({"$set": {"after_start": "value"}})),
- (team_id, "distinct_1", person_id, in_window_ts, json.dumps({"$set": {"in_window": "value"}})),
- ]
-
- def insert_events(client):
- client.execute(
- """INSERT INTO writable_events (team_id, distinct_id, person_id, timestamp, properties)
- VALUES""",
- events,
- )
-
- cluster.any_host(insert_events).result()
-
- person_data = [
- (
- team_id,
- person_id,
- json.dumps({"before": "old", "after_start": "old", "in_window": "old"}),
- 1,
- now - timedelta(days=4), # Person _timestamp after bug_window_start
- )
- ]
-
- def insert_person(client):
- client.execute(
- """INSERT INTO person (team_id, id, properties, version, _timestamp)
- VALUES""",
- person_data,
- )
-
- cluster.any_host(insert_person).result()
-
- results = get_person_property_updates_from_clickhouse(
- team_id=team_id,
- bug_window_start=bug_window_start.strftime("%Y-%m-%d %H:%M:%S"),
- )
-
- assert len(results) == 1
- updates = {key: pv.value for key, pv in results[0].set_updates.items()}
-
- # Events after bug_window_start should be included
- assert "in_window" in updates
- assert "after_start" in updates
- # Events before bug_window_start should NOT be included
- assert "before" not in updates
-
- def test_person_created_before_bug_window_with_events_after_is_included(self, cluster: ClickhouseCluster):
- """Regression test: Person created before bug_window_start with events after should be included.
-
- Scenario:
- - Person was created/last modified BEFORE bug_window_start (their _timestamp is old)
- - After bug_window_start, there are $set events for this person
- - These events SHOULD be included in the reconciliation
-
- This is important because we want to reconcile ALL persons that have property-setting
- events after bug_window_start, regardless of when the person was originally created.
- The bug window is about when the ingestion bug occurred, not when persons were created.
- """
- team_id = 99919
- person_id = UUID("55550000-0000-0000-0000-000000000002")
- now = datetime.now().replace(microsecond=0)
- bug_window_start = now - timedelta(days=5)
-
- # Event AFTER bug_window_start - should be included
- event_ts = now - timedelta(days=3)
-
- events = [
- (team_id, "distinct_1", person_id, event_ts, json.dumps({"$set": {"email": "[email protected]"}})),
- ]
-
- def insert_events(client):
- client.execute(
- """INSERT INTO writable_events (team_id, distinct_id, person_id, timestamp, properties)
- VALUES""",
- events,
- )
-
- cluster.any_host(insert_events).result()
-
- # Person _timestamp BEFORE bug_window_start (person was created before the bug window)
- # but with a property that differs from the event's $set value
- person_data = [(team_id, person_id, json.dumps({"email": "[email protected]"}), 1, now - timedelta(days=10))]
-
- def insert_person(client):
- client.execute(
- """INSERT INTO person (team_id, id, properties, version, _timestamp)
- VALUES""",
- person_data,
- )
-
- cluster.any_host(insert_person).result()
-
- results = get_person_property_updates_from_clickhouse(
- team_id=team_id,
- bug_window_start=bug_window_start.strftime("%Y-%m-%d %H:%M:%S"),
- )
-
- # Person SHOULD be included because they have events after bug_window_start
- # The person's _timestamp being before bug_window_start should NOT exclude them
- assert len(results) == 1
- assert str(results[0].person_id) == str(person_id)
- assert "email" in results[0].set_updates
- assert results[0].set_updates["email"].value == "[email protected]"
-
- def test_timestamp_window_filtering_all_permutations(self, cluster: ClickhouseCluster):
- """Comprehensive test of all permutations of person _timestamp and event timestamp.
-
- Tests the matrix of:
- - Person _timestamp: BEFORE / DURING / AFTER bug window
- - Event timestamp: BEFORE / DURING / AFTER bug window
-
- Expected behavior:
- - Person _timestamp does NOT affect filtering (we removed that filter)
- - Events BEFORE bug_window_start are filtered out
- - Events DURING or AFTER bug_window_start are included (query uses now() as upper bound)
-
- So expected results:
- | Person _timestamp | Event timestamp | Included? |
- |-------------------|-----------------|-----------|
- | BEFORE | BEFORE | NO |
- | BEFORE | DURING | YES |
- | BEFORE | AFTER | YES |
- | DURING | BEFORE | NO |
- | DURING | DURING | YES |
- | DURING | AFTER | YES |
- | AFTER | BEFORE | NO |
- | AFTER | DURING | YES |
- | AFTER | AFTER | YES |
- """
- # Use two teams to ensure isolation
- team_id_1 = 99950
- team_id_2 = 99951
-
- now = datetime.now().replace(microsecond=0)
-
- # Define time periods
- # Bug window is conceptually from bug_window_start to bug_window_end (for team discovery)
- # But the CH query reads from bug_window_start to now()
- bug_window_start = now - timedelta(days=10)
- _bug_window_end = now - timedelta(days=5) # Used conceptually, not in query
-
- # Timestamps for each period
- ts_before = now - timedelta(days=15) # Before bug_window_start
- ts_during = now - timedelta(days=7) # Between bug_window_start and bug_window_end
- ts_after = now - timedelta(days=2) # After bug_window_end but before now()
-
- # Create 9 persons for team 1 - all permutations
- # Format: (person_id_suffix, person_ts, event_ts, should_be_included)
- test_cases_team1 = [
- # Person _timestamp BEFORE bug_window_start
- ("001", ts_before, ts_before, False, "person_before_event_before"),
- ("002", ts_before, ts_during, True, "person_before_event_during"),
- ("003", ts_before, ts_after, True, "person_before_event_after"),
- # Person _timestamp DURING bug_window
- ("004", ts_during, ts_before, False, "person_during_event_before"),
- ("005", ts_during, ts_during, True, "person_during_event_during"),
- ("006", ts_during, ts_after, True, "person_during_event_after"),
- # Person _timestamp AFTER bug_window_end
- ("007", ts_after, ts_before, False, "person_after_event_before"),
- ("008", ts_after, ts_during, True, "person_after_event_during"),
- ("009", ts_after, ts_after, True, "person_after_event_after"),
- ]
-
- # Create 3 persons for team 2 - subset to verify team isolation
- test_cases_team2 = [
- ("101", ts_before, ts_during, True, "team2_person_before_event_during"),
- ("102", ts_during, ts_after, True, "team2_person_during_event_after"),
- ("103", ts_after, ts_before, False, "team2_person_after_event_before"),
- ]
-
- # Helper to create UUID from suffix
- def make_uuid(suffix: str) -> UUID:
- return UUID(f"99990000-0000-0000-0000-000000000{suffix}")
-
- # Insert events for team 1
- events_team1 = []
- for suffix, _person_ts, event_ts, _expected, prop_name in test_cases_team1:
- person_id = make_uuid(suffix)
- events_team1.append(
- (
- team_id_1,
- f"distinct_{suffix}",
- person_id,
- event_ts,
- json.dumps({"$set": {prop_name: "new_value"}}),
- )
- )
-
- def insert_events_team1(client):
- client.execute(
- """INSERT INTO writable_events (team_id, distinct_id, person_id, timestamp, properties)
- VALUES""",
- events_team1,
- )
-
- cluster.any_host(insert_events_team1).result()
-
- # Insert events for team 2
- events_team2 = []
- for suffix, _person_ts, event_ts, _expected, prop_name in test_cases_team2:
- person_id = make_uuid(suffix)
- events_team2.append(
- (
- team_id_2,
- f"distinct_{suffix}",
- person_id,
- event_ts,
- json.dumps({"$set": {prop_name: "new_value"}}),
- )
- )
-
- def insert_events_team2(client):
- client.execute(
- """INSERT INTO writable_events (team_id, distinct_id, person_id, timestamp, properties)
- VALUES""",
- events_team2,
- )
-
- cluster.any_host(insert_events_team2).result()
-
- # Insert persons for team 1 (with different property values so we get a diff)
- persons_team1 = []
- for suffix, person_ts, _event_ts, _expected, prop_name in test_cases_team1:
- person_id = make_uuid(suffix)
- persons_team1.append(
- (
- team_id_1,
- person_id,
- json.dumps({prop_name: "old_value"}),
- 1, # version
- person_ts, # _timestamp
- )
- )
-
- def insert_persons_team1(client):
- client.execute(
- """INSERT INTO person (team_id, id, properties, version, _timestamp)
- VALUES""",
- persons_team1,
- )
-
- cluster.any_host(insert_persons_team1).result()
-
- # Insert persons for team 2
- persons_team2 = []
- for suffix, person_ts, _event_ts, _expected, prop_name in test_cases_team2:
- person_id = make_uuid(suffix)
- persons_team2.append(
- (
- team_id_2,
- person_id,
- json.dumps({prop_name: "old_value"}),
- 1,
- person_ts,
- )
- )
-
- def insert_persons_team2(client):
- client.execute(
- """INSERT INTO person (team_id, id, properties, version, _timestamp)
- VALUES""",
- persons_team2,
- )
-
- cluster.any_host(insert_persons_team2).result()
-
- # Query for team 1
- results_team1 = get_person_property_updates_from_clickhouse(
- team_id=team_id_1,
- bug_window_start=bug_window_start.strftime("%Y-%m-%d %H:%M:%S"),
- )
-
- # Query for team 2
- results_team2 = get_person_property_updates_from_clickhouse(
- team_id=team_id_2,
- bug_window_start=bug_window_start.strftime("%Y-%m-%d %H:%M:%S"),
- )
-
- # Verify team 1 results
- result_person_ids_team1 = {str(r.person_id) for r in results_team1}
- expected_person_ids_team1 = {
- str(make_uuid(suffix))
- for suffix, _person_ts, _event_ts, expected, _prop_name in test_cases_team1
- if expected
- }
-
- assert result_person_ids_team1 == expected_person_ids_team1, (
- f"Team 1 mismatch.\n"
- f"Expected: {expected_person_ids_team1}\n"
- f"Got: {result_person_ids_team1}\n"
- f"Missing: {expected_person_ids_team1 - result_person_ids_team1}\n"
- f"Extra: {result_person_ids_team1 - expected_person_ids_team1}"
- )
-
- # Verify team 2 results
- result_person_ids_team2 = {str(r.person_id) for r in results_team2}
- expected_person_ids_team2 = {
- str(make_uuid(suffix))
- for suffix, _person_ts, _event_ts, expected, _prop_name in test_cases_team2
- if expected
- }
-
- assert result_person_ids_team2 == expected_person_ids_team2, (
- f"Team 2 mismatch.\nExpected: {expected_person_ids_team2}\nGot: {result_person_ids_team2}"
- )
-
- # Verify the property values are correct for included persons
- for result in results_team1:
- # Each person should have exactly one set_update with value "new_value"
- assert len(result.set_updates) == 1, f"Person {result.person_id} should have 1 update"
- prop_value = next(iter(result.set_updates.values()))
- assert prop_value.value == "new_value", f"Person {result.person_id} should have new_value"
-
- # Verify specific cases to make the test more explicit
- # Person 002: _timestamp BEFORE, event DURING → should be included
- assert str(make_uuid("002")) in result_person_ids_team1, (
- "Person with _timestamp BEFORE but event DURING should be included"
- )
-
- # Person 003: _timestamp BEFORE, event AFTER → should be included
- assert str(make_uuid("003")) in result_person_ids_team1, (
- "Person with _timestamp BEFORE but event AFTER should be included"
- )
-
- # Person 007: _timestamp AFTER, event BEFORE → should NOT be included
- assert str(make_uuid("007")) not in result_person_ids_team1, (
- "Person with event BEFORE bug_window_start should NOT be included"
- )
-
- # Person 001: _timestamp BEFORE, event BEFORE → should NOT be included
- assert str(make_uuid("001")) not in result_person_ids_team1, (
- "Person with event BEFORE bug_window_start should NOT be included"
- )
-
- # ==================== Edge Case Tests ====================
-
- def test_same_timestamp_deterministic(self, cluster: ClickhouseCluster):
- """
- When multiple events have exact same timestamp,
- argMax/argMin should still produce deterministic results.
- """
- team_id = 99924
- person_id = UUID("55550000-0000-0000-0000-000000000001")
- now = datetime.now().replace(microsecond=0)
- bug_window_start = now - timedelta(days=10)
-
- # Same timestamp for both events
- same_ts = now - timedelta(days=3)
-
- events = [
- (team_id, "distinct_1", person_id, same_ts, json.dumps({"$set": {"name": "Value A"}})),
- (team_id, "distinct_2", person_id, same_ts, json.dumps({"$set": {"name": "Value B"}})),
- ]
-
- def insert_events(client):
- client.execute(
- """INSERT INTO writable_events (team_id, distinct_id, person_id, timestamp, properties)
- VALUES""",
- events,
- )
-
- cluster.any_host(insert_events).result()
-
- person_data = [(team_id, person_id, json.dumps({"name": "Old"}), 1, now - timedelta(days=8))]
-
- def insert_person(client):
- client.execute(
- """INSERT INTO person (team_id, id, properties, version, _timestamp)
- VALUES""",
- person_data,
- )
-
- cluster.any_host(insert_person).result()
-
- results = get_person_property_updates_from_clickhouse(
- team_id=team_id,
- bug_window_start=bug_window_start.strftime("%Y-%m-%d %H:%M:%S"),
- )
-
- # Should get exactly one result with one of the values
- assert len(results) == 1
- updates = {key: pv.value for key, pv in results[0].set_updates.items()}
- assert updates["name"] in ["Value A", "Value B"]
-
- def test_special_characters_in_property_keys(self, cluster: ClickhouseCluster):
- """Property keys with special characters should work correctly."""
- team_id = 99925
- person_id = UUID("66660000-0000-0000-0000-000000000001")
- now = datetime.now().replace(microsecond=0)
- bug_window_start = now - timedelta(days=10)
-
- event_ts = now - timedelta(days=3)
-
- events = [
- (
- team_id,
- "distinct_1",
- person_id,
- event_ts,
- json.dumps(
- {
- "$set": {
- "key with spaces": "value1",
- "key.with.dots": "value2",
- "key-with-dashes": "value3",
- "key_with_underscores": "value4",
- "キー": "unicode value", # Japanese "key"
- }
- }
- ),
- ),
- ]
-
- def insert_events(client):
- client.execute(
- """INSERT INTO writable_events (team_id, distinct_id, person_id, timestamp, properties)
- VALUES""",
- events,
- )
-
- cluster.any_host(insert_events).result()
-
- # Person with different values for these keys
- person_data = [
- (
- team_id,
- person_id,
- json.dumps(
- {
- "key with spaces": "old1",
- "key.with.dots": "old2",
- "key-with-dashes": "old3",
- "key_with_underscores": "old4",
- "キー": "old unicode",
- }
- ),
- 1,
- now - timedelta(days=8),
- )
- ]
-
- def insert_person(client):
- client.execute(
- """INSERT INTO person (team_id, id, properties, version, _timestamp)
- VALUES""",
- person_data,
- )
-
- cluster.any_host(insert_person).result()
-
- results = get_person_property_updates_from_clickhouse(
- team_id=team_id,
- bug_window_start=bug_window_start.strftime("%Y-%m-%d %H:%M:%S"),
- )
-
- assert len(results) == 1
- updates = {key: pv.value for key, pv in results[0].set_updates.items()}
-
- assert updates["key with spaces"] == "value1"
- assert updates["key.with.dots"] == "value2"
- assert updates["key-with-dashes"] == "value3"
- assert updates["key_with_underscores"] == "value4"
- assert updates["キー"] == "unicode value"
-
- def test_filtered_properties_are_excluded(self, cluster: ClickhouseCluster):
- """Properties in FILTERED_PERSON_UPDATE_PROPERTIES should be excluded from results.
-
- These are high-frequency properties like $current_url that change often
- but aren't valuable enough to trigger person updates.
- """
- team_id = 99926
- person_id = UUID("77770000-0000-0000-0000-000000000001")
- now = datetime.now().replace(microsecond=0)
- bug_window_start = now - timedelta(days=10)
-
- event_ts = now - timedelta(days=3)
-
- # Event with both filtered and non-filtered properties
- events = [
- (
- team_id,
- "distinct_1",
- person_id,
- event_ts,
- json.dumps(
- {
- "$set": {
- # Filtered properties - should NOT appear in results
- "$current_url": "https://example.com/page",
- "$pathname": "/page",
- "$browser": "Chrome",
- "$os": "Mac OS X",
- # Non-filtered properties - SHOULD appear in results
- "email": "[email protected]",
- "name": "Test User",
- },
- "$set_once": {
- # Filtered
- "$referring_domain": "google.com",
- # Non-filtered
- "initial_campaign": "summer_sale",
- },
- }
- ),
- ),
- ]
-
- def insert_events(client):
- client.execute(
- """INSERT INTO writable_events (team_id, distinct_id, person_id, timestamp, properties)
- VALUES""",
- events,
- )
-
- cluster.any_host(insert_events).result()
-
- # Person with different values so diffs are detected
- person_data = [
- (
- team_id,
- person_id,
- json.dumps(
- {
- "$current_url": "https://example.com/old",
- "email": "[email protected]",
- "name": "Old Name",
- }
- ),
- 1,
- now - timedelta(days=8),
- )
- ]
-
- def insert_person(client):
- client.execute(
- """INSERT INTO person (team_id, id, properties, version, _timestamp)
- VALUES""",
- person_data,
- )
-
- cluster.any_host(insert_person).result()
-
- results = get_person_property_updates_from_clickhouse(
- team_id=team_id,
- bug_window_start=bug_window_start.strftime("%Y-%m-%d %H:%M:%S"),
- )
-
- assert len(results) == 1
- person_diffs = results[0]
-
- # Non-filtered properties should be present
- set_keys = set(person_diffs.set_updates.keys())
- set_once_keys = set(person_diffs.set_once_updates.keys())
-
- assert "email" in set_keys, "Non-filtered property 'email' should be in results"
- assert "name" in set_keys, "Non-filtered property 'name' should be in results"
- assert "initial_campaign" in set_once_keys, "Non-filtered property 'initial_campaign' should be in results"
-
- # Filtered properties should NOT be present
- assert "$current_url" not in set_keys, "Filtered property '$current_url' should NOT be in results"
- assert "$pathname" not in set_keys, "Filtered property '$pathname' should NOT be in results"
- assert "$browser" not in set_keys, "Filtered property '$browser' should NOT be in results"
- assert "$os" not in set_keys, "Filtered property '$os' should NOT be in results"
- assert "$referring_domain" not in set_once_keys, (
- "Filtered property '$referring_domain' should NOT be in results"
- )
-
- def test_person_ids_filter_only_returns_specified_persons(self, cluster: ClickhouseCluster):
- """
- Integration test: verify person_ids filter correctly restricts query results.
-
- This tests the fix for the NULL IS NULL bug - when person_ids is provided,
- only those persons should be returned.
- """
- team_id = 99970
- now = datetime.now().replace(microsecond=0)
- bug_window_start = now - timedelta(days=10)
- event_ts = now - timedelta(days=5)
-
- # Create 5 persons, but we'll only query for 2 of them
- person_ids = [UUID(f"77770000-0000-0000-0000-00000000000{i}") for i in range(1, 6)]
- target_person_ids = (str(person_ids[0]), str(person_ids[2])) # Person 1 and 3
-
- # Insert events for all 5 persons
- events = []
- for i, pid in enumerate(person_ids):
- events.append(
- (
- team_id,
- f"distinct_{i}",
- pid,
- event_ts,
- json.dumps({"$set": {"email": f"person{i}@example.com"}}),
- )
- )
-
- def insert_events(client: Client) -> None:
- client.execute(
- """INSERT INTO writable_events (team_id, distinct_id, person_id, timestamp, properties)
- VALUES""",
- events,
- )
-
- cluster.any_host(insert_events).result()
-
- # Insert all 5 persons in ClickHouse with different email (so diff is detected)
- person_data = []
- for i, pid in enumerate(person_ids):
- person_data.append(
- (
- team_id,
- pid,
- json.dumps({"email": f"old{i}@example.com"}),
- 1,
- now - timedelta(days=8),
- )
- )
-
- def insert_persons(client: Client) -> None:
- client.execute(
- """INSERT INTO person (team_id, id, properties, version, _timestamp)
- VALUES""",
- person_data,
- )
-
- cluster.any_host(insert_persons).result()
-
- # Query with person_ids filter - should only return the 2 specified persons
- results = get_raw_person_property_updates_from_clickhouse(
- team_id=team_id,
- bug_window_start=bug_window_start.strftime("%Y-%m-%d %H:%M:%S"),
- bug_window_end=now.strftime("%Y-%m-%d %H:%M:%S"),
- person_ids=target_person_ids,
- )
-
- # Should only have 2 results (the filtered persons)
- result_person_ids = {r.person_id for r in results}
- expected_person_ids = set(target_person_ids)
-
- assert result_person_ids == expected_person_ids, (
- f"person_ids filter failed.\n"
- f"Expected: {expected_person_ids}\n"
- f"Got: {result_person_ids}\n"
- f"This may indicate the NULL IS NULL bug has regressed!"
- )
-
- # Verify the data is correct for the returned persons
- for result in results:
- assert len(result.set_updates) == 1
- assert "email" in result.set_updates
-
- def test_interleaved_events_multiple_persons_realistic_scenario(self, cluster: ClickhouseCluster):
- """
- Integration test: realistic scenario with multiple persons and interleaved events.
-
- Simulates a real-world scenario where:
- - Multiple users are active simultaneously
- - Events arrive in arbitrary order (not grouped by person)
- - Different operation types ($set, $set_once, $unset) are mixed
- - Some properties conflict (same key, different operations)
-
- This tests that the query correctly aggregates per-person despite
- events being interleaved in the events table.
- """
- team_id = 99971
- now = datetime.now().replace(microsecond=0)
- bug_window_start = now - timedelta(days=10)
-
- # Three users with different activity patterns
- alice = UUID("aaaa0000-0000-0000-0000-000000000001")
- bob = UUID("bbbb0000-0000-0000-0000-000000000002")
- charlie = UUID("cccc0000-0000-0000-0000-000000000003")
-
- # Events arrive interleaved (not grouped by person) - this is realistic
- # Time ordering simulates events arriving from different user sessions
- events = [
- # t+1h: Alice sets email
- (team_id, "alice_1", alice, now - timedelta(hours=9), json.dumps({"$set": {"email": "[email protected]"}})),
- # t+1h: Bob sets name
- (team_id, "bob_1", bob, now - timedelta(hours=9), json.dumps({"$set": {"name": "Bob v1"}})),
- # t+2h: Charlie sets email and set_once for referrer
- (
- team_id,
- "charlie_1",
- charlie,
- now - timedelta(hours=8),
- json.dumps({"$set": {"email": "[email protected]"}, "$set_once": {"referrer": "google"}}),
- ),
- # t+3h: Alice updates email (should win over v1)
- (team_id, "alice_2", alice, now - timedelta(hours=7), json.dumps({"$set": {"email": "[email protected]"}})),
- # t+3h: Bob also tries set_once referrer (should use first value)
- (team_id, "bob_2", bob, now - timedelta(hours=7), json.dumps({"$set_once": {"referrer": "facebook"}})),
- # t+4h: Alice sets name
- (team_id, "alice_3", alice, now - timedelta(hours=6), json.dumps({"$set": {"name": "Alice"}})),
- # t+4h: Bob updates name (should win over v1)
- (team_id, "bob_3", bob, now - timedelta(hours=6), json.dumps({"$set": {"name": "Bob v2"}})),
- # t+5h: Charlie tries set_once referrer again (should NOT override google)
- (
- team_id,
- "charlie_2",
- charlie,
- now - timedelta(hours=5),
- json.dumps({"$set_once": {"referrer": "twitter"}}),
- ),
- # t+6h: Alice unsets email (should override the $set)
- (team_id, "alice_4", alice, now - timedelta(hours=4), json.dumps({"$unset": ["email"]})),
- # t+7h: Bob's set_once for referrer again (still should be facebook - first one)
- (team_id, "bob_4", bob, now - timedelta(hours=3), json.dumps({"$set_once": {"referrer": "linkedin"}})),
- ]
-
- def insert_events(client: Client) -> None:
- client.execute(
- """INSERT INTO writable_events (team_id, distinct_id, person_id, timestamp, properties)
- VALUES""",
- events,
- )
-
- cluster.any_host(insert_events).result()
-
- # Create persons with existing properties
- # Alice: has old email (different, so $set shows in diff) and no name
- # Bob: has old name (different), no referrer
- # Charlie: has old email (different), no referrer
- person_data = [
- (team_id, alice, json.dumps({"email": "[email protected]"}), 1, now - timedelta(days=8)),
- (team_id, bob, json.dumps({"name": "Old Bob"}), 1, now - timedelta(days=8)),
- (team_id, charlie, json.dumps({"email": "[email protected]"}), 1, now - timedelta(days=8)),
- ]
-
- def insert_persons(client: Client) -> None:
- client.execute(
- """INSERT INTO person (team_id, id, properties, version, _timestamp)
- VALUES""",
- person_data,
- )
-
- cluster.any_host(insert_persons).result()
-
- # Query and verify
- results = get_person_property_updates_from_clickhouse(
- team_id=team_id,
- bug_window_start=bug_window_start.strftime("%Y-%m-%d %H:%M:%S"),
- )
-
- results_by_person = {str(r.person_id): r for r in results}
-
- # Alice assertions:
- # - email: $set [email protected] at t+3h, then $unset at t+6h → both should appear
- # - name: $set "Alice" at t+4h → but no name in person, so NOT in set_diff
- alice_result = results_by_person[str(alice)]
- assert "email" in alice_result.set_updates, "Alice should have email in set_updates"
- assert alice_result.set_updates["email"].value == "[email protected]", "Alice email should be v2"
- assert "email" in alice_result.unset_updates, "Alice should have email in unset_updates"
- # name not in person, so $set won't create a diff
- assert "name" not in alice_result.set_updates, "Alice name should NOT be in set_updates (key doesn't exist)"
-
- # Bob assertions:
- # - name: latest is "Bob v2" at t+4h, person has different value
- # - referrer: first set_once is "facebook" at t+3h, person doesn't have it
- bob_result = results_by_person[str(bob)]
- assert "name" in bob_result.set_updates, "Bob should have name in set_updates"
- assert bob_result.set_updates["name"].value == "Bob v2", "Bob name should be v2"
- assert "referrer" in bob_result.set_once_updates, "Bob should have referrer in set_once"
- assert bob_result.set_once_updates["referrer"].value == "facebook", "Bob referrer should be facebook (first)"
-
- # Charlie assertions:
- # - email: "[email protected]", person has different value
- # - referrer: "google" (first set_once), person doesn't have it
- charlie_result = results_by_person[str(charlie)]
- assert "email" in charlie_result.set_updates, "Charlie should have email"
- assert charlie_result.set_updates["email"].value == "[email protected]"
- assert "referrer" in charlie_result.set_once_updates, "Charlie should have referrer"
- assert charlie_result.set_once_updates["referrer"].value == "google", (
- "Charlie referrer should be google (first)"
- )
-
- def test_unicode_and_emoji_in_property_keys_and_values(self, cluster: ClickhouseCluster):
- """
- Integration test: verify Unicode and emoji characters work in property keys and values.
-
- Real users often have international names, emoji in custom properties, etc.
- This ensures the query handles UTF-8 properly throughout the pipeline.
- """
- team_id = 99972
- person_id = UUID("88880000-0000-0000-0000-000000000001")
- now = datetime.now().replace(microsecond=0)
- bug_window_start = now - timedelta(days=10)
- event_ts = now - timedelta(days=5)
-
- # Unicode and emoji test cases
- events = [
- (
- team_id,
- "distinct_unicode",
- person_id,
- event_ts,
- json.dumps(
- {
- "$set": {
- "名前": "田中太郎", # Japanese: name -> Taro Tanaka
- "город": "Москва", # Russian: city -> Moscow
- "emoji_status": "🎉🚀💯", # Emoji value
- "Ñoño": "Señor", # Spanish with ñ
- "العربية": "مرحبا", # Arabic
- },
- "$set_once": {
- "🔑_key": "emoji_key_value", # Emoji in key
- "first_emoji": "👋",
- },
- }
- ),
- )
- ]
-
- def insert_events(client: Client) -> None:
- client.execute(
- """INSERT INTO writable_events (team_id, distinct_id, person_id, timestamp, properties)
- VALUES""",
- events,
- )
-
- cluster.any_host(insert_events).result()
-
- # Person with different values for the Unicode keys
- person_data = [
- (
- team_id,
- person_id,
- json.dumps(
- {
- "名前": "山田花子", # Different Japanese name
- "город": "Санкт-Петербург", # Different Russian city
- "emoji_status": "😢",
- "Ñoño": "Señora",
- "العربية": "وداعا",
- }
- ),
- 1,
- now - timedelta(days=8),
- )
- ]
-
- def insert_persons(client: Client) -> None:
- client.execute(
- """INSERT INTO person (team_id, id, properties, version, _timestamp)
- VALUES""",
- person_data,
- )
-
- cluster.any_host(insert_persons).result()
-
- results = get_person_property_updates_from_clickhouse(
- team_id=team_id,
- bug_window_start=bug_window_start.strftime("%Y-%m-%d %H:%M:%S"),
- )
-
- assert len(results) == 1
- person_diffs = results[0]
-
- # Verify all Unicode keys are present with correct values
- assert "名前" in person_diffs.set_updates
- assert person_diffs.set_updates["名前"].value == "田中太郎"
-
- assert "город" in person_diffs.set_updates
- assert person_diffs.set_updates["город"].value == "Москва"
-
- assert "emoji_status" in person_diffs.set_updates
- assert person_diffs.set_updates["emoji_status"].value == "🎉🚀💯"
-
- assert "Ñoño" in person_diffs.set_updates
- assert person_diffs.set_updates["Ñoño"].value == "Señor"
-
- assert "العربية" in person_diffs.set_updates
- assert person_diffs.set_updates["العربية"].value == "مرحبا"
-
- # Verify set_once with emoji key
- assert "🔑_key" in person_diffs.set_once_updates
- assert person_diffs.set_once_updates["🔑_key"].value == "emoji_key_value"
-
- assert "first_emoji" in person_diffs.set_once_updates
- assert person_diffs.set_once_updates["first_emoji"].value == "👋"
-
- def test_no_affected_persons_returns_empty_results(self, cluster: ClickhouseCluster):
- """
- Integration test: verify graceful handling when no persons are affected.
-
- This ensures the query doesn't error when:
- - No events exist in the bug window
- - Events exist but properties match (no diff)
- - Team has no data at all
- """
- team_id = 99973
- now = datetime.now().replace(microsecond=0)
-
- # Bug window that has no events (far in the past, unique team_id)
- bug_window_start = now - timedelta(days=1000)
- bug_window_end = now - timedelta(days=999)
-
- # Query should return empty, not error
- results = get_person_property_updates_from_clickhouse(
- team_id=team_id,
- bug_window_start=bug_window_start.strftime("%Y-%m-%d %H:%M:%S"),
- )
-
- assert results == [], f"Expected empty results for team with no data, got {len(results)} results"
-
- # Also test get_affected_person_ids_from_clickhouse
- affected_ids = get_affected_person_ids_from_clickhouse(
- team_id=team_id,
- bug_window_start=bug_window_start.strftime("%Y-%m-%d %H:%M:%S"),
- bug_window_end=bug_window_end.strftime("%Y-%m-%d %H:%M:%S"),
- )
-
- assert affected_ids == [], f"Expected no affected persons, got {len(affected_ids)}"
-
- def test_rapid_fire_events_within_same_second(self, cluster: ClickhouseCluster):
- """
- Integration test: multiple events for same person within the same second.
-
- Real-world scenario: rapid client-side events, SDK batching, etc.
- Verifies deterministic behavior when timestamps collide.
- """
- team_id = 99974
- person_id = UUID("99990000-0000-0000-0000-000000000001")
- now = datetime.now().replace(microsecond=0)
- bug_window_start = now - timedelta(days=10)
-
- # Same timestamp for all events (within same second)
- same_ts = now - timedelta(days=5)
-
- # Multiple rapid-fire $set events for same key with same timestamp
- # ClickHouse argMax should pick one deterministically
- events = [
- (team_id, "rapid_1", person_id, same_ts, json.dumps({"$set": {"counter": 1}})),
- (team_id, "rapid_2", person_id, same_ts, json.dumps({"$set": {"counter": 2}})),
- (team_id, "rapid_3", person_id, same_ts, json.dumps({"$set": {"counter": 3}})),
- (team_id, "rapid_4", person_id, same_ts, json.dumps({"$set": {"counter": 4}})),
- (team_id, "rapid_5", person_id, same_ts, json.dumps({"$set": {"counter": 5}})),
- ]
-
- def insert_events(client: Client) -> None:
- client.execute(
- """INSERT INTO writable_events (team_id, distinct_id, person_id, timestamp, properties)
- VALUES""",
- events,
- )
-
- cluster.any_host(insert_events).result()
-
- # Person with different counter value
- person_data = [(team_id, person_id, json.dumps({"counter": 0}), 1, now - timedelta(days=8))]
-
- def insert_persons(client: Client) -> None:
- client.execute(
- """INSERT INTO person (team_id, id, properties, version, _timestamp)
- VALUES""",
- person_data,
- )
-
- cluster.any_host(insert_persons).result()
-
- # Run query multiple times - result should be consistent (deterministic)
- results_1 = get_person_property_updates_from_clickhouse(
- team_id=team_id,
- bug_window_start=bug_window_start.strftime("%Y-%m-%d %H:%M:%S"),
- )
- results_2 = get_person_property_updates_from_clickhouse(
- team_id=team_id,
- bug_window_start=bug_window_start.strftime("%Y-%m-%d %H:%M:%S"),
- )
-
- assert len(results_1) == 1
- assert len(results_2) == 1
-
- # Should be deterministic - same value both times
- counter_1 = results_1[0].set_updates["counter"].value
- counter_2 = results_2[0].set_updates["counter"].value
-
- assert counter_1 == counter_2, (
- f"Rapid-fire events should produce deterministic results. Got {counter_1} then {counter_2}"
- )
-
- # Value should be one of the inserted values (1-5)
- assert counter_1 in [1, 2, 3, 4, 5], f"Counter should be one of 1-5, got {counter_1}"
-
- def test_affected_person_ids_excludes_filtered_property_only_events(self, cluster: ClickhouseCluster):
- """
- Integration test: get_affected_person_ids_from_clickhouse should exclude persons
- whose events only set FILTERED_PERSON_UPDATE_PROPERTIES ($browser, $os, etc.).
-
- Person A: events with ONLY filtered properties -> excluded
- Person B: events with at least one non-filtered property that differs from person state -> included
- """
- team_id = 99980
- person_a = UUID("aaaa0000-0000-0000-0000-000000000001")
- person_b = UUID("bbbb0000-0000-0000-0000-000000000001")
- now = datetime.now().replace(microsecond=0)
- bug_window_start = now - timedelta(hours=3)
- bug_window_end = now - timedelta(hours=1)
- event_ts = now - timedelta(hours=2)
-
- events = [
- # Person A: only filtered properties
- (
- team_id,
- "filtered_only_user",
- person_a,
- event_ts,
- json.dumps(
- {
- "$set": {
- "$browser": "Chrome",
- "$os": "Mac OS X",
- "$current_url": "https://example.com/page",
- "$pathname": "/page",
- },
- }
- ),
- ),
- # Person B: has a non-filtered property with a value that differs from person state
- (
- team_id,
- "real_update_user",
- person_b,
- event_ts,
- json.dumps(
- {
- "$set": {
- "$browser": "Firefox",
- "email": "[email protected]",
- },
- }
- ),
- ),
- ]
-
- def insert_events(client: Client) -> None:
- client.execute(
- """INSERT INTO writable_events (team_id, distinct_id, person_id, timestamp, properties)
- VALUES""",
- events,
- )
-
- cluster.any_host(insert_events).result()
-
- person_data = [
- (team_id, person_a, json.dumps({"$browser": "Safari"}), 1, now - timedelta(hours=4)),
- (team_id, person_b, json.dumps({"email": "[email protected]"}), 1, now - timedelta(hours=4)),
- ]
-
- def insert_persons(client: Client) -> None:
- client.execute(
- """INSERT INTO person (team_id, id, properties, version, _timestamp)
- VALUES""",
- person_data,
- )
-
- cluster.any_host(insert_persons).result()
-
- affected_ids = get_affected_person_ids_from_clickhouse(
- team_id=team_id,
- bug_window_start=bug_window_start.strftime("%Y-%m-%d %H:%M:%S"),
- bug_window_end=bug_window_end.strftime("%Y-%m-%d %H:%M:%S"),
- )
-
- assert str(person_b) in affected_ids, "Person with non-filtered property diff should be detected"
- assert str(person_a) not in affected_ids, "Person with only filtered properties should be excluded"
-
- def test_affected_person_ids_includes_set_once_and_unset_with_non_filtered_keys(self, cluster: ClickhouseCluster):
- """
- Integration test: detection covers $set_once and $unset with non-filtered keys
- that produce actual diffs against person state.
-
- person_set_once: $set_once for key not on person -> included
- person_unset: $unset for key that exists on person -> included
- """
- team_id = 99981
- person_set_once = UUID("cccc0000-0000-0000-0000-000000000001")
- person_unset = UUID("dddd0000-0000-0000-0000-000000000002")
- now = datetime.now().replace(microsecond=0)
- bug_window_start = now - timedelta(hours=3)
- bug_window_end = now - timedelta(hours=1)
- event_ts = now - timedelta(hours=2)
-
- events = [
- (
- team_id,
- "set_once_user",
- person_set_once,
- event_ts,
- json.dumps({"$set_once": {"initial_campaign": "summer_sale"}}),
- ),
- (
- team_id,
- "unset_user",
- person_unset,
- event_ts,
- json.dumps({"$unset": ["email"]}),
- ),
- ]
-
- def insert_events(client: Client) -> None:
- client.execute(
- """INSERT INTO writable_events (team_id, distinct_id, person_id, timestamp, properties)
- VALUES""",
- events,
- )
-
- cluster.any_host(insert_events).result()
-
- person_data = [
- # person_set_once: does NOT have "initial_campaign" -> $set_once will produce a diff
- (team_id, person_set_once, json.dumps({"name": "Alice"}), 1, now - timedelta(hours=4)),
- # person_unset: HAS "email" -> $unset will produce a diff
- (team_id, person_unset, json.dumps({"email": "[email protected]"}), 1, now - timedelta(hours=4)),
- ]
-
- def insert_persons(client: Client) -> None:
- client.execute(
- """INSERT INTO person (team_id, id, properties, version, _timestamp)
- VALUES""",
- person_data,
- )
-
- cluster.any_host(insert_persons).result()
-
- affected_ids = get_affected_person_ids_from_clickhouse(
- team_id=team_id,
- bug_window_start=bug_window_start.strftime("%Y-%m-%d %H:%M:%S"),
- bug_window_end=bug_window_end.strftime("%Y-%m-%d %H:%M:%S"),
- )
-
- assert str(person_set_once) in affected_ids, "$set_once with non-filtered key should be detected"
- assert str(person_unset) in affected_ids, "$unset with non-filtered key should be detected"
-
- def test_affected_person_ids_excludes_persons_whose_properties_already_match(self, cluster: ClickhouseCluster):
- """
- Integration test: persons whose events set non-filtered properties to values
- that already match the current person state should NOT be returned.
-
- This is the core fix for the 100x over-selection bug: the old query returned
- anyone with property-setting events, the new query compares against person
- state and only returns persons with actual diffs.
-
- person_match: $set email to same value already on person -> excluded
- person_diff: $set email to different value -> included
- person_set_once_exists: $set_once for key already on person -> excluded
- person_unset_missing: $unset for key NOT on person -> excluded
- """
- team_id = 99983
- person_match = UUID("ff000000-0000-0000-0000-000000000001")
- person_diff = UUID("ff000000-0000-0000-0000-000000000002")
- person_set_once_exists = UUID("ff000000-0000-0000-0000-000000000003")
- person_unset_missing = UUID("ff000000-0000-0000-0000-000000000004")
- now = datetime.now().replace(microsecond=0)
- bug_window_start = now - timedelta(hours=3)
- bug_window_end = now - timedelta(hours=1)
- event_ts = now - timedelta(hours=2)
-
- events = [
- # person_match: $set email to value that already matches person state
- (team_id, "match_user", person_match, event_ts, json.dumps({"$set": {"email": "[email protected]"}})),
- # person_diff: $set email to different value
- (team_id, "diff_user", person_diff, event_ts, json.dumps({"$set": {"email": "[email protected]"}})),
- # person_set_once_exists: $set_once for key already on person (no-op)
- (
- team_id,
- "set_once_exists_user",
- person_set_once_exists,
- event_ts,
- json.dumps({"$set_once": {"signup_source": "google"}}),
- ),
- # person_unset_missing: $unset for key not on person (no-op)
- (
- team_id,
- "unset_missing_user",
- person_unset_missing,
- event_ts,
- json.dumps({"$unset": ["nonexistent_key"]}),
- ),
- ]
-
- def insert_events(client: Client) -> None:
- client.execute(
- """INSERT INTO writable_events (team_id, distinct_id, person_id, timestamp, properties)
- VALUES""",
- events,
- )
-
- cluster.any_host(insert_events).result()
-
- person_data = [
- # person_match: email already matches the event value
- (team_id, person_match, json.dumps({"email": "[email protected]"}), 1, now - timedelta(hours=4)),
- # person_diff: email differs from event value
- (team_id, person_diff, json.dumps({"email": "[email protected]"}), 1, now - timedelta(hours=4)),
- # person_set_once_exists: already has the key -> $set_once is a no-op
- (team_id, person_set_once_exists, json.dumps({"signup_source": "facebook"}), 1, now - timedelta(hours=4)),
- # person_unset_missing: does NOT have the key -> $unset is a no-op
- (team_id, person_unset_missing, json.dumps({"name": "Alice"}), 1, now - timedelta(hours=4)),
- ]
-
- def insert_persons(client: Client) -> None:
- client.execute(
- """INSERT INTO person (team_id, id, properties, version, _timestamp)
- VALUES""",
- person_data,
- )
-
- cluster.any_host(insert_persons).result()
-
- affected_ids = get_affected_person_ids_from_clickhouse(
- team_id=team_id,
- bug_window_start=bug_window_start.strftime("%Y-%m-%d %H:%M:%S"),
- bug_window_end=bug_window_end.strftime("%Y-%m-%d %H:%M:%S"),
- )
-
- assert str(person_diff) in affected_ids, "$set with different value should be detected"
- assert str(person_match) not in affected_ids, "$set with matching value should be excluded"
- assert str(person_set_once_exists) not in affected_ids, "$set_once for existing key should be excluded"
- assert str(person_unset_missing) not in affected_ids, "$unset for missing key should be excluded"
-
- def test_windowed_query_produces_same_result_as_single_query(self, cluster: ClickhouseCluster):
- """
- Integration test: windowed queries should produce equivalent results to single query.
-
- This test inserts events spread across multiple time windows with $set, $set_once,
- and $unset operations, then verifies that:
- 1. A single query returns the expected merged result
- 2. A windowed query (multiple smaller queries merged) returns the same result
-
- The CH query behavior:
- - $set: only returns keys that EXIST in person AND have different values
- - $set_once: only returns keys that DON'T exist in person
- - $unset: only returns keys that EXIST in person
- """
- team_id = 99930
- person_id = UUID("11111111-1111-1111-1111-000000000030")
- now = datetime.now().replace(microsecond=0)
- bug_window_start = now - timedelta(hours=6)
-
- # Events spread across 6 hours (will use 2-hour windows = 3 windows)
- # Window 1: hour 0-2
- # - $set email to "[email protected]" at hour 1
- # - $set name to "Name1" at hour 1
- # - $set_once signup_source to "source1" at hour 1
- # Window 2: hour 2-4
- # - $set email to "[email protected]" at hour 3 (newer, should win)
- # - $set_once signup_source to "source2" at hour 3 (older should win - source1)
- # Window 3: hour 4-6
- # - $unset email at hour 5 (newest, should remove email from set)
- # - $set name to "Name3" at hour 5 (newest, should win)
-
- events = [
- # Window 1 events (hour 1)
- (
- team_id,
- "distinct_id_1",
- person_id,
- bug_window_start + timedelta(hours=1),
- json.dumps(
- {
- "$set": {"email": "[email protected]", "name": "Name1"},
- "$set_once": {"signup_source": "source1"},
- }
- ),
- ),
- # Window 2 events (hour 3)
- (
- team_id,
- "distinct_id_1",
- person_id,
- bug_window_start + timedelta(hours=3),
- json.dumps({"$set": {"email": "[email protected]"}, "$set_once": {"signup_source": "source2"}}),
- ),
- # Window 3 events (hour 5)
- (
- team_id,
- "distinct_id_1",
- person_id,
- bug_window_start + timedelta(hours=5),
- json.dumps({"$unset": ["email"], "$set": {"name": "Name3"}}),
- ),
- ]
-
- def insert_events(client: Client) -> None:
- client.execute(
- """INSERT INTO writable_events (team_id, distinct_id, person_id, timestamp, properties)
- VALUES""",
- events,
- )
-
- cluster.any_host(insert_events).result()
-
- # Person has email and name (so $set/$unset can modify them)
- # Person does NOT have signup_source (so $set_once will add it)
- person_data = [
- (
- team_id,
- person_id,
- json.dumps({"email": "[email protected]", "name": "OldName"}),
- 1,
- now - timedelta(hours=7),
- )
- ]
-
- def insert_person(client: Client) -> None:
- client.execute(
- """INSERT INTO person (team_id, id, properties, version, _timestamp)
- VALUES""",
- person_data,
- )
-
- cluster.any_host(insert_person).result()
-
- bug_window_start_str = bug_window_start.strftime("%Y-%m-%d %H:%M:%S")
-
- # Run single query (current behavior)
- single_results = get_person_property_updates_from_clickhouse(
- team_id=team_id,
- bug_window_start=bug_window_start_str,
- )
-
- # Run windowed query with 2-hour windows
- windowed_results = get_person_property_updates_windowed(
- team_id=team_id,
- bug_window_start=bug_window_start_str,
- window_seconds=7200, # 2 hours
- )
-
- # Both should return exactly one person
- assert len(single_results) == 1
- assert len(windowed_results) == 1
-
- single_diff = single_results[0]
- windowed_diff = windowed_results[0]
-
- # Both should have the same person_id
- assert single_diff.person_id == windowed_diff.person_id == str(person_id)
-
- # Core assertion: windowed query should produce equivalent results to single query
- # The key properties we expect (based on event data):
- # - name: $set to "Name3" at hour 5 (argMax picks latest)
- # - email: $unset at hour 5 (person has email, events want to unset)
-
- # Verify both queries return the same keys in each category
- assert set(single_diff.set_updates.keys()) == set(windowed_diff.set_updates.keys()), (
- f"set_updates keys differ: single={set(single_diff.set_updates.keys())}, "
- f"windowed={set(windowed_diff.set_updates.keys())}"
- )
- assert set(single_diff.set_once_updates.keys()) == set(windowed_diff.set_once_updates.keys()), (
- f"set_once_updates keys differ: single={set(single_diff.set_once_updates.keys())}, "
- f"windowed={set(windowed_diff.set_once_updates.keys())}"
- )
- assert set(single_diff.unset_updates.keys()) == set(windowed_diff.unset_updates.keys()), (
- f"unset_updates keys differ: single={set(single_diff.unset_updates.keys())}, "
- f"windowed={set(windowed_diff.unset_updates.keys())}"
- )
-
- # Verify values match for keys that exist
- for key in single_diff.set_updates:
- assert single_diff.set_updates[key].value == windowed_diff.set_updates[key].value, (
- f"set_updates[{key}] value differs"
- )
-
- for key in single_diff.set_once_updates:
- assert single_diff.set_once_updates[key].value == windowed_diff.set_once_updates[key].value, (
- f"set_once_updates[{key}] value differs"
- )
-
- # Verify expected properties based on test data
- assert "name" in single_diff.set_updates, "name should be in set_updates"
- assert single_diff.set_updates["name"].value == "Name3", "name should be Name3"
- assert "email" in single_diff.unset_updates, "email should be in unset_updates"
-
- def test_windowed_batched_produces_same_result_as_single_query(self, cluster: ClickhouseCluster):
- """
- Integration test: the production windowed-batched path
- (get_person_property_updates_windowed_batched) must produce identical
- diffs to the non-windowed single-query path.
-
- This test includes:
- - Events with both filtered and non-filtered properties
- - Events spread across multiple time windows
- - A person with ONLY filtered-property events (should be excluded by both)
- - A person with non-filtered properties that MATCH current state (no diff, excluded by both)
- - $set, $set_once, and $unset operations
- """
- team_id = 99982
- person_real = UUID("eeee0000-0000-0000-0000-000000000001")
- person_filtered_only = UUID("eeee0000-0000-0000-0000-000000000002")
- person_no_diff = UUID("eeee0000-0000-0000-0000-000000000003")
- now = datetime.now().replace(microsecond=0)
- bug_window_start = now - timedelta(hours=6)
- bug_window_end = now - timedelta(minutes=5)
-
- events = [
- # Person with real (non-filtered) property updates across windows
- # Window 1 (hour 1): $set email, $set_once signup_source, plus filtered props
- (
- team_id,
- "real_user",
- person_real,
- bug_window_start + timedelta(hours=1),
- json.dumps(
- {
- "$set": {
- "email": "[email protected]",
- "name": "Name1",
- "$browser": "Chrome",
- "$os": "Mac OS X",
- },
- "$set_once": {"signup_source": "google"},
- }
- ),
- ),
- # Window 2 (hour 3): $set email overwrites, $set_once ignored (first wins)
- (
- team_id,
- "real_user",
- person_real,
- bug_window_start + timedelta(hours=3),
- json.dumps(
- {
- "$set": {"email": "[email protected]"},
- "$set_once": {"signup_source": "bing"},
- }
- ),
- ),
- # Window 3 (hour 5): $unset email, $set name overwrites
- (
- team_id,
- "real_user",
- person_real,
- bug_window_start + timedelta(hours=5),
- json.dumps(
- {
- "$unset": ["email"],
- "$set": {"name": "Name3"},
- }
- ),
- ),
- # Person with ONLY filtered-property events (should not appear in results)
- (
- team_id,
- "filtered_user",
- person_filtered_only,
- bug_window_start + timedelta(hours=2),
- json.dumps(
- {
- "$set": {
- "$browser": "Firefox",
- "$os": "Windows",
- "$current_url": "https://example.com",
- },
- }
- ),
- ),
- # Person with non-filtered properties that already match person state (no diff).
- # This is the exact scenario causing 100x over-selection in production:
- # the old affected persons query returned these, the new one should not.
- (
- team_id,
- "no_diff_user",
- person_no_diff,
- bug_window_start + timedelta(hours=2),
- json.dumps(
- {
- "$set": {"plan": "enterprise", "company": "Acme Corp"},
- "$set_once": {"signup_source": "organic"},
- }
- ),
- ),
- ]
-
- def insert_events(client: Client) -> None:
- client.execute(
- """INSERT INTO writable_events (team_id, distinct_id, person_id, timestamp, properties)
- VALUES""",
- events,
- )
-
- cluster.any_host(insert_events).result()
-
- person_data = [
- (
- team_id,
- person_real,
- json.dumps({"email": "[email protected]", "name": "OldName"}),
- 1,
- now - timedelta(hours=7),
- ),
- (
- team_id,
- person_filtered_only,
- json.dumps({"$browser": "Safari", "$os": "Linux"}),
- 1,
- now - timedelta(hours=7),
- ),
- # person_no_diff: properties already match what the events would set
- (
- team_id,
- person_no_diff,
- json.dumps({"plan": "enterprise", "company": "Acme Corp", "signup_source": "organic"}),
- 1,
- now - timedelta(hours=7),
- ),
- ]
-
- def insert_persons(client: Client) -> None:
- client.execute(
- """INSERT INTO person (team_id, id, properties, version, _timestamp)
- VALUES""",
- person_data,
- )
-
- cluster.any_host(insert_persons).result()
-
- bug_window_start_str = bug_window_start.strftime("%Y-%m-%d %H:%M:%S")
- bug_window_end_str = bug_window_end.strftime("%Y-%m-%d %H:%M:%S")
-
- # Run non-windowed single query
- single_results = get_person_property_updates_from_clickhouse(
- team_id=team_id,
- bug_window_start=bug_window_start_str,
- )
-
- # Run windowed-batched (production path) with 2-hour windows
- batched_results: list[PersonPropertyDiffs] = []
- for batch in get_person_property_updates_windowed_batched(
- team_id=team_id,
- bug_window_start=bug_window_start_str,
- bug_window_end=bug_window_end_str,
- window_seconds=7200,
- person_batch_size=100,
- ):
- batched_results.extend(batch)
-
- # Both should return exactly one person (the real one, not the others)
- assert len(single_results) == 1, f"single query returned {len(single_results)}, expected 1"
- assert len(batched_results) == 1, f"batched windowed returned {len(batched_results)}, expected 1"
-
- single_diff = single_results[0]
- batched_diff = batched_results[0]
-
- # Same person_id
- assert single_diff.person_id == batched_diff.person_id == str(person_real)
-
- # No-diff and filtered-only persons must not appear in either mode
- result_person_ids = {r.person_id for r in single_results} | {r.person_id for r in batched_results}
- assert str(person_filtered_only) not in result_person_ids
- assert str(person_no_diff) not in result_person_ids, (
- "Person whose properties already match event values should be excluded by both modes"
- )
-
- # Same keys in each diff category
- assert set(single_diff.set_updates.keys()) == set(batched_diff.set_updates.keys()), (
- f"set_updates keys differ: single={set(single_diff.set_updates.keys())}, "
- f"batched={set(batched_diff.set_updates.keys())}"
- )
- assert set(single_diff.set_once_updates.keys()) == set(batched_diff.set_once_updates.keys()), (
- f"set_once_updates keys differ: single={set(single_diff.set_once_updates.keys())}, "
- f"batched={set(batched_diff.set_once_updates.keys())}"
- )
- assert set(single_diff.unset_updates.keys()) == set(batched_diff.unset_updates.keys()), (
- f"unset_updates keys differ: single={set(single_diff.unset_updates.keys())}, "
- f"batched={set(batched_diff.unset_updates.keys())}"
- )
-
- # Same values
- for key in single_diff.set_updates:
- assert single_diff.set_updates[key].value == batched_diff.set_updates[key].value, (
- f"set_updates[{key}] value differs: single={single_diff.set_updates[key].value}, "
- f"batched={batched_diff.set_updates[key].value}"
- )
-
- for key in single_diff.set_once_updates:
- assert single_diff.set_once_updates[key].value == batched_diff.set_once_updates[key].value, (
- f"set_once_updates[{key}] value differs: single={single_diff.set_once_updates[key].value}, "
- f"batched={batched_diff.set_once_updates[key].value}"
- )
-
- # Verify expected properties: name set to Name3, email unset
- assert "name" in single_diff.set_updates
- assert single_diff.set_updates["name"].value == "Name3"
- assert "email" in single_diff.unset_updates
-
-
-# Exercises the persons DB directly (raw reads and writes against the persons writer), so it must
-# run against the real persons DB rather than the personhog fake.
[email protected]_db(transaction=True)
-class TestBatchCommitsEndToEnd:
- """End-to-end integration tests for batch commit functionality.
-
- These tests use real ClickHouse and Postgres connections to verify
- that batch commits work correctly in a production-like environment.
- """
-
- @pytest.fixture
- def organization(self):
- """Create a test organization."""
- from posthog.models import Organization
-
- return Organization.objects.create(name="Batch Test Organization")
-
- @pytest.fixture
- def team(self, organization):
- """Create a test team."""
- from posthog.models import Team
-
- return Team.objects.create(organization=organization, name="Batch Test Team")
-
- def test_batch_commits_end_to_end(self, cluster: ClickhouseCluster, team):
- """
- End-to-end test that verifies batch commits work with real databases.
-
- Creates 5 persons in Postgres, inserts events in ClickHouse,
- runs reconciliation with batch_size=2, and verifies:
- 1. All persons are updated in Postgres
- 2. Commits happen in batches (3 batches: [2, 2, 1])
- """
- from posthog.dags.person_property_reconciliation import (
- get_person_property_updates_from_clickhouse,
- process_persons_in_batches,
- )
- from posthog.test.persons import create_person
-
- num_persons = 5
- batch_size = 2
-
- # Use a unique team_id to avoid conflicts with other tests
- team_id = team.id
-
- # Time setup
- now = datetime.now().replace(microsecond=0)
- bug_window_start = now - timedelta(days=10)
- event_ts = now - timedelta(days=5)
-
- # Create persons in Postgres with old property values
- persons = []
- for i in range(num_persons):
- person = create_person(
- team_id=team_id,
- properties={"email": f"old_{i}@example.com", "counter": i},
- properties_last_updated_at={
- "email": "2024-01-01T00:00:00+00:00",
- "counter": "2024-01-01T00:00:00+00:00",
- },
- properties_last_operation={"email": "set", "counter": "set"},
- version=1,
- )
- persons.append(person)
-
- # Insert events in ClickHouse with new property values
- events = []
- for i, person in enumerate(persons):
- events.append(
- (
- team_id,
- f"distinct_id_{i}",
- person.uuid,
- event_ts,
- json.dumps({"$set": {"email": f"new_{i}@example.com"}}),
- )
- )
-
- def insert_events(client: Client) -> None:
- client.execute(
- """INSERT INTO writable_events (team_id, distinct_id, person_id, timestamp, properties)
- VALUES""",
- events,
- )
-
- cluster.any_host(insert_events).result()
-
- # Insert persons in ClickHouse (required for the query join)
- person_ch_data = []
- for i, person in enumerate(persons):
- person_ch_data.append(
- (
- team_id,
- person.uuid,
- json.dumps({"email": f"old_{i}@example.com", "counter": i}),
- 1, # version
- 0, # is_deleted
- now - timedelta(days=8), # _timestamp
- )
- )
-
- def insert_persons_ch(client: Client) -> None:
- client.execute(
- """INSERT INTO person (team_id, id, properties, version, is_deleted, _timestamp)
- VALUES""",
- person_ch_data,
- )
-
- cluster.any_host(insert_persons_ch).result()
-
- # Get person property updates from ClickHouse
- person_property_updates = get_person_property_updates_from_clickhouse(
- team_id=team_id,
- bug_window_start=bug_window_start.strftime("%Y-%m-%d %H:%M:%S"),
- )
-
- assert len(person_property_updates) == num_persons, (
- f"Expected {num_persons} persons from ClickHouse query, got {len(person_property_updates)}"
- )
-
- # Track batches
- batch_sizes = []
-
- def on_batch_committed(batch_num: int, batch_persons: list[dict]) -> None:
- batch_sizes.append(len(batch_persons))
-
- from posthog.persons_db import persons_db_connection
-
- # Non-autocommit: process_persons_in_batches expects a transactional cursor (its commit_fn drives commits).
- with persons_db_connection(writer=True) as connection, connection.cursor() as cursor:
- # Set up cursor settings
- cursor.execute("SET application_name = 'test_batch_commits'")
-
- result = process_persons_in_batches(
- person_property_diffs=person_property_updates,
- cursor=cursor,
- job_id="test-batch-job",
- team_id=team_id,
- batch_size=batch_size,
- dry_run=False,
- backup_enabled=False,
- commit_fn=lambda: None, # No-op commit in tests (Django manages transaction)
- on_batch_committed=on_batch_committed,
- )
-
- # Verify all persons were processed
- assert result.total_processed == num_persons, f"Expected {num_persons} processed, got {result.total_processed}"
- assert result.total_updated == num_persons, f"Expected {num_persons} updated, got {result.total_updated}"
- assert result.total_skipped == 0, f"Expected 0 skipped, got {result.total_skipped}"
-
- # Verify batch sizes: [2, 2, 1] for 5 persons with batch_size=2
- assert result.total_commits == 3, f"Expected 3 commits, got {result.total_commits}"
- assert batch_sizes == [2, 2, 1], f"Expected batch sizes [2, 2, 1], got {batch_sizes}"
-
- # Verify Postgres was actually updated with correct properties and metadata
- for i, person in enumerate(persons):
- refresh_person_from_persons_db(person)
-
- # Property value should be updated
- assert person.properties["email"] == f"new_{i}@example.com", (
- f"Person {i} email not updated. Expected 'new_{i}@example.com', got '{person.properties.get('email')}'"
- )
-
- # Counter should be unchanged (wasn't in the update)
- assert person.properties["counter"] == i, (
- f"Person {i} counter changed unexpectedly. Expected {i}, got {person.properties.get('counter')}"
- )
-
- # Version should be incremented
- assert person.version == 2, f"Person {i} version not incremented. Expected 2, got {person.version}"
-
- # properties_last_updated_at should have new timestamp for email
- assert person.properties_last_updated_at is not None
- assert "email" in person.properties_last_updated_at, (
- f"Person {i} properties_last_updated_at missing 'email' key"
- )
- # The timestamp should be from the event (event_ts), not the old value
- email_updated_at = person.properties_last_updated_at["email"]
- assert email_updated_at != "2024-01-01T00:00:00+00:00", (
- f"Person {i} email timestamp not updated. Still has old value: {email_updated_at}"
- )
-
- # properties_last_operation should be 'set' for email
- assert person.properties_last_operation is not None
- assert person.properties_last_operation.get("email") == "set", (
- f"Person {i} properties_last_operation['email'] should be 'set', "
- f"got '{person.properties_last_operation.get('email')}'"
- )
-
- # Counter's metadata should be unchanged
- assert person.properties_last_updated_at.get("counter") == "2024-01-01T00:00:00+00:00", (
- f"Person {i} counter timestamp changed unexpectedly"
- )
-
- def test_batch_commits_with_missing_person(self, cluster: ClickhouseCluster, team):
- """
- Test that missing persons in Postgres are skipped gracefully.
-
- Creates 4 persons in ClickHouse events but only 3 in Postgres.
- Verifies the missing person is skipped and others are updated.
- """
- from posthog.dags.person_property_reconciliation import (
- get_person_property_updates_from_clickhouse,
- process_persons_in_batches,
- )
- from posthog.test.persons import create_person
-
- team_id = team.id
-
- now = datetime.now().replace(microsecond=0)
- bug_window_start = now - timedelta(days=10)
- event_ts = now - timedelta(days=5)
-
- # Create only 3 persons in Postgres
- persons = []
- for i in range(3):
- person = create_person(
- team_id=team_id,
- properties={"name": f"old_name_{i}"},
- properties_last_updated_at={"name": "2024-01-01T00:00:00+00:00"},
- properties_last_operation={"name": "set"},
- version=1,
- )
- persons.append(person)
-
- # Create a 4th UUID that won't exist in Postgres
- missing_uuid = UUID("99999999-9999-9999-9999-999999999999")
-
- # Insert 4 events in ClickHouse (including one for missing person)
- events = []
- for i, person in enumerate(persons):
- events.append(
- (
- team_id,
- f"distinct_id_{i}",
- person.uuid,
- event_ts,
- json.dumps({"$set": {"name": f"new_name_{i}"}}),
- )
- )
- # Add event for missing person
- events.append(
- (
- team_id,
- "distinct_id_missing",
- missing_uuid,
- event_ts,
- json.dumps({"$set": {"name": "new_name_missing"}}),
- )
- )
-
- def insert_events(client: Client) -> None:
- client.execute(
- """INSERT INTO writable_events (team_id, distinct_id, person_id, timestamp, properties)
- VALUES""",
- events,
- )
-
- cluster.any_host(insert_events).result()
-
- # Insert 4 persons in ClickHouse (including missing one)
- person_ch_data = []
- for i, person in enumerate(persons):
- person_ch_data.append(
- (
- team_id,
- person.uuid,
- json.dumps({"name": f"old_name_{i}"}),
- 1,
- 0,
- now - timedelta(days=8),
- )
- )
- # Add the missing person to ClickHouse
- person_ch_data.append(
- (
- team_id,
- missing_uuid,
- json.dumps({"name": "old_name_missing"}),
- 1,
- 0,
- now - timedelta(days=8),
- )
- )
-
- def insert_persons_ch(client: Client) -> None:
- client.execute(
- """INSERT INTO person (team_id, id, properties, version, is_deleted, _timestamp)
- VALUES""",
- person_ch_data,
- )
-
- cluster.any_host(insert_persons_ch).result()
-
- # Get updates from ClickHouse - should return 4 persons
- person_property_updates = get_person_property_updates_from_clickhouse(
- team_id=team_id,
- bug_window_start=bug_window_start.strftime("%Y-%m-%d %H:%M:%S"),
- )
-
- assert len(person_property_updates) == 4
-
- batch_sizes = []
-
- def on_batch_committed(batch_num: int, batch_persons: list[dict]) -> None:
- batch_sizes.append(len(batch_persons))
-
- from posthog.persons_db import persons_db_connection
-
- # Non-autocommit: process_persons_in_batches expects a transactional cursor (its commit_fn drives commits).
- with persons_db_connection(writer=True) as connection, connection.cursor() as cursor:
- result = process_persons_in_batches(
- person_property_diffs=person_property_updates,
- cursor=cursor,
- job_id="test-missing-person-job",
- team_id=team_id,
- batch_size=2,
- dry_run=False,
- backup_enabled=False,
- commit_fn=lambda: None, # No-op commit in tests (Django manages transaction)
- on_batch_committed=on_batch_committed,
- )
-
- # 4 persons processed, 1 skipped (missing from Postgres), 3 updated
- assert result.total_processed == 4
- assert result.total_skipped == 1 # The missing person
- assert result.total_updated == 3
-
- # Verify existing persons were updated with correct properties and metadata
- for i, person in enumerate(persons):
- refresh_person_from_persons_db(person)
-
- # Property value should be updated
- assert person.properties["name"] == f"new_name_{i}", (
- f"Person {i} name not updated. Expected 'new_name_{i}', got '{person.properties.get('name')}'"
- )
-
- # Version should be incremented
- assert person.version == 2, f"Person {i} version not incremented. Expected 2, got {person.version}"
-
- # properties_last_updated_at should have new timestamp
- assert person.properties_last_updated_at is not None
- assert "name" in person.properties_last_updated_at, (
- f"Person {i} properties_last_updated_at missing 'name' key"
- )
- name_updated_at = person.properties_last_updated_at["name"]
- assert name_updated_at != "2024-01-01T00:00:00+00:00", (
- f"Person {i} name timestamp not updated. Still has old value: {name_updated_at}"
- )
-
- # properties_last_operation should be 'set'
- assert person.properties_last_operation is not None
- assert person.properties_last_operation.get("name") == "set", (
- f"Person {i} properties_last_operation['name'] should be 'set', "
- f"got '{person.properties_last_operation.get('name')}'"
- )
-
-
[email protected]_db(transaction=True)
-class TestKafkaClickHouseRoundTrip:
- """Integration tests that verify person updates flow through Kafka to ClickHouse.
-
- These tests use real Kafka (not mocked) and verify that:
- 1. publish_person_to_kafka produces messages in the correct format
- 2. ClickHouse's Kafka engine consumes the messages
- 3. The person table in ClickHouse has the correct data
-
- This ensures the reconciliation job's Kafka message format is compatible
- with ClickHouse's expectations (matching Node.js ingestion format).
- """
-
- @pytest.fixture
- def organization(self):
- """Create a test organization."""
- from posthog.models import Organization
-
- return Organization.objects.create(name="Kafka Test Organization")
-
- @pytest.fixture
- def team(self, organization):
- """Create a test team."""
- from posthog.models import Team
-
- return Team.objects.create(organization=organization, name="Kafka Test Team")
-
- def _wait_for_clickhouse_person(
- self,
- team_id: int,
- person_uuid: str,
- expected_version: int,
- max_wait_seconds: int = 30,
- poll_interval_seconds: float = 0.5,
- ) -> dict | None:
- """
- Poll ClickHouse until the person appears with the expected version.
-
- Returns the person row as a dict, or None if not found within timeout.
- """
- import time
-
- from posthog.clickhouse.client import sync_execute
-
- start_time = time.time()
- while time.time() - start_time < max_wait_seconds:
- rows = sync_execute(
- """
- SELECT id, team_id, properties, is_identified, version, is_deleted, created_at
- FROM person FINAL
- WHERE team_id = %(team_id)s AND id = %(person_uuid)s
- """,
- {"team_id": team_id, "person_uuid": person_uuid},
- )
- if rows:
- row = rows[0]
- person_data = {
- "id": str(row[0]),
- "team_id": row[1],
- "properties": row[2],
- "is_identified": row[3],
- "version": row[4],
- "is_deleted": row[5],
- "created_at": row[6],
- }
- if person_data["version"] >= expected_version:
- return person_data
- time.sleep(poll_interval_seconds)
- return None
-
- @pytest.mark.skipif(
- os.environ.get("KAFKA_ROUNDTRIP_TESTS") != "1",
- reason="Requires real Kafka infrastructure. Set KAFKA_ROUNDTRIP_TESTS=1 to run.",
- )
- def test_publish_person_to_kafka_updates_clickhouse(self, team):
- """
- Test that publish_person_to_kafka produces messages that ClickHouse consumes correctly.
-
- This verifies the full round-trip:
- 1. Create a person in Postgres
- 2. Publish to Kafka using publish_person_to_kafka (with real Kafka, not mocked)
- 3. Wait for ClickHouse to consume the message
- 4. Verify ClickHouse has the correct person data
- """
- from django.test import override_settings
-
- from posthog.dags.person_property_reconciliation import publish_person_to_kafka
- from posthog.kafka_client.client import _KafkaProducer
- from posthog.test.persons import create_person
-
- # Create person in Postgres
- person = create_person(
- team_id=team.id,
- properties={"email": "[email protected]", "name": "Kafka Test User"},
- version=1,
- is_identified=True,
- )
-
- # Prepare person data for Kafka (simulating what reconciliation job does)
- person_data = {
- "id": person.uuid,
- "team_id": team.id,
- "properties": {"email": "[email protected]", "name": "Kafka Test User"},
- "is_identified": True,
- "is_deleted": 0,
- "created_at": person.created_at,
- "version": 1,
- }
-
- # Create a real Kafka producer (not mocked)
- with override_settings(TEST=False):
- producer = _KafkaProducer(test=False)
- try:
- publish_person_to_kafka(person_data, producer)
- producer.flush(timeout=10)
- finally:
- producer.close()
-
- # Wait for ClickHouse to consume the message
- ch_person = self._wait_for_clickhouse_person(
- team_id=team.id,
- person_uuid=str(person.uuid),
- expected_version=1,
- max_wait_seconds=30,
- )
-
- # Verify ClickHouse received the person
- assert ch_person is not None, f"Person {person.uuid} not found in ClickHouse after 30 seconds"
- assert ch_person["team_id"] == team.id
- assert ch_person["version"] == 1
- assert ch_person["is_identified"] == 1 # ClickHouse stores as Int8
-
- # Parse properties (stored as JSON string in ClickHouse)
- ch_properties = json.loads(ch_person["properties"])
- assert ch_properties["email"] == "[email protected]"
- assert ch_properties["name"] == "Kafka Test User"
-
- @pytest.mark.skipif(
- os.environ.get("KAFKA_ROUNDTRIP_TESTS") != "1",
- reason="Requires real Kafka infrastructure. Set KAFKA_ROUNDTRIP_TESTS=1 to run.",
- )
- def test_reconciliation_kafka_message_format_matches_nodejs(self, team, cluster: ClickhouseCluster):
- """
- Test that the reconciliation job's Kafka message format produces the same
- ClickHouse state as Node.js ingestion would.
-
- This is important because both systems publish to the same Kafka topic,
- and ClickHouse must be able to handle messages from both sources.
- """
- from django.test import override_settings
-
- from posthog.dags.person_property_reconciliation import publish_person_to_kafka
- from posthog.kafka_client.client import _KafkaProducer
- from posthog.test.persons import create_person
-
- # Create person in Postgres with version 1
- person = create_person(
- team_id=team.id,
- properties={"email": "[email protected]"},
- version=1,
- is_identified=False,
- )
-
- # First, publish version 1 to establish baseline
- person_data_v1 = {
- "id": person.uuid,
- "team_id": team.id,
- "properties": {"email": "[email protected]"},
- "is_identified": False,
- "is_deleted": 0,
- "created_at": person.created_at,
- "version": 1,
- }
-
- with override_settings(TEST=False):
- producer = _KafkaProducer(test=False)
- try:
- publish_person_to_kafka(person_data_v1, producer)
- producer.flush(timeout=10)
- finally:
- producer.close()
-
- # Wait for version 1 in ClickHouse
- ch_person_v1 = self._wait_for_clickhouse_person(
- team_id=team.id,
- person_uuid=str(person.uuid),
- expected_version=1,
- )
- assert ch_person_v1 is not None, "Version 1 not found in ClickHouse"
-
- # Now simulate reconciliation updating the person to version 2
- person_data_v2 = {
- "id": person.uuid,
- "team_id": team.id,
- "properties": {"email": "[email protected]", "source": "reconciliation"},
- "is_identified": True,
- "is_deleted": 0,
- "created_at": person.created_at,
- "version": 2,
- }
-
- with override_settings(TEST=False):
- producer = _KafkaProducer(test=False)
- try:
- publish_person_to_kafka(person_data_v2, producer)
- producer.flush(timeout=10)
- finally:
- producer.close()
-
- # Wait for version 2 in ClickHouse
- ch_person_v2 = self._wait_for_clickhouse_person(
- team_id=team.id,
- person_uuid=str(person.uuid),
- expected_version=2,
- )
-
- # Verify ClickHouse received the update
- assert ch_person_v2 is not None, "Version 2 not found in ClickHouse after 30 seconds"
- assert ch_person_v2["version"] == 2
- assert ch_person_v2["is_identified"] == 1
-
- ch_properties = json.loads(ch_person_v2["properties"])
- assert ch_properties["email"] == "[email protected]"
- assert ch_properties["source"] == "reconciliation"
-
- @pytest.mark.skipif(
- os.environ.get("KAFKA_ROUNDTRIP_TESTS") != "1",
- reason="Requires real Kafka infrastructure. Set KAFKA_ROUNDTRIP_TESTS=1 to run.",
- )
- def test_full_dagster_job_with_real_kafka(self, team, cluster: ClickhouseCluster):
- """
- Test the full Dagster reconciliation job with real Kafka.
-
- This is the most comprehensive test - it:
- 1. Creates persons in Postgres with outdated properties
- 2. Inserts events in ClickHouse with updated properties
- 3. Runs the actual Dagster job with real Kafka producer
- 4. Waits for ClickHouse to consume the Kafka messages
- 5. Verifies both Postgres AND ClickHouse have the correct data
- """
- from django.test import override_settings
-
- from posthog.dags.person_property_reconciliation import person_property_reconciliation_job
- from posthog.kafka_client.client import _KafkaProducer
- from posthog.test.persons import create_person
-
- # Time setup - create a bug window that includes our test events
- now = datetime.now().replace(microsecond=0)
- bug_window_start = (now - timedelta(days=10)).strftime("%Y-%m-%d %H:%M:%S")
- bug_window_end = (now + timedelta(days=1)).strftime("%Y-%m-%d %H:%M:%S")
- event_ts = now - timedelta(days=5)
-
- # Create person in Postgres with old property value
- person = create_person(
- team_id=team.id,
- properties={"email": "[email protected]", "unchanged": "value"},
- properties_last_updated_at={
- "email": "2024-01-01T00:00:00+00:00",
- "unchanged": "2024-01-01T00:00:00+00:00",
- },
- properties_last_operation={"email": "set", "unchanged": "set"},
- version=1,
- )
-
- # Insert event in ClickHouse with new property value
- def insert_event(client: Client) -> None:
- client.execute(
- """INSERT INTO writable_events (team_id, distinct_id, person_id, timestamp, properties)
- VALUES""",
- [
- (
- team.id,
- "test_distinct_id",
- person.uuid,
- event_ts,
- json.dumps({"$set": {"email": "[email protected]"}}),
- )
- ],
- )
-
- cluster.any_host(insert_event).result()
-
- # Insert person in ClickHouse (required for the reconciliation query join)
- def insert_person_ch(client: Client) -> None:
- client.execute(
- """INSERT INTO person (team_id, id, properties, version, is_deleted, _timestamp)
- VALUES""",
- [
- (
- team.id,
- person.uuid,
- json.dumps({"email": "[email protected]", "unchanged": "value"}),
- 1,
- 0,
- now - timedelta(days=8),
- )
- ],
- )
-
- cluster.any_host(insert_person_ch).result()
-
- from posthog.persons_db import persons_db_connection
-
- # Create real Kafka producer. Non-autocommit: the dagster job drives commit()/rollback() on this resource.
- with persons_db_connection(writer=True) as persons_conn, override_settings(TEST=False):
- kafka_producer = _KafkaProducer(test=False)
-
- try:
- # Run the actual Dagster job
- result = person_property_reconciliation_job.execute_in_process(
- run_config={
- "ops": {
- "get_team_ids_to_reconcile": {
- "config": {
- "team_ids": [team.id],
- "bug_window_start": bug_window_start,
- "bug_window_end": bug_window_end,
- "dry_run": False,
- "backup_enabled": False,
- "batch_size": 100,
- }
- },
- "reconcile_team_chunk": {
- "config": {
- "bug_window_start": bug_window_start,
- "dry_run": False,
- "backup_enabled": False,
- "batch_size": 100,
- }
- },
- }
- },
- resources={
- "cluster": cluster,
- "persons_database": persons_conn,
- "kafka_producer": kafka_producer,
- },
- )
-
- assert result.success, f"Dagster job failed: {result}"
-
- # Flush kafka to ensure all messages are sent
- kafka_producer.flush(timeout=10)
-
- finally:
- kafka_producer.close()
-
- # Verify Postgres was updated
- refresh_person_from_persons_db(person)
- assert person.properties["email"] == "[email protected]", (
- f"Postgres email not updated. Expected '[email protected]', got '{person.properties.get('email')}'"
- )
- assert person.properties["unchanged"] == "value", "Unchanged property was modified"
- assert person.version == 2, f"Postgres version not incremented. Expected 2, got {person.version}"
-
- # Wait for ClickHouse to consume the Kafka message
- ch_person = self._wait_for_clickhouse_person(
- team_id=team.id,
- person_uuid=str(person.uuid),
- expected_version=2,
- max_wait_seconds=30,
- )
-
- # Verify ClickHouse received the update via Kafka
- assert ch_person is not None, (
- f"Person {person.uuid} version 2 not found in ClickHouse after 30 seconds. "
- "This suggests the Kafka message format may not be compatible with ClickHouse."
- )
- assert ch_person["version"] == 2
-
- ch_properties = json.loads(ch_person["properties"])
- assert ch_properties["email"] == "[email protected]", (
- f"ClickHouse email not updated. Expected '[email protected]', got '{ch_properties.get('email')}'"
- )
- assert ch_properties["unchanged"] == "value", "ClickHouse unchanged property was modified"
-
- @pytest.mark.skipif(
- os.environ.get("KAFKA_ROUNDTRIP_TESTS") != "1",
- reason="Requires real Kafka infrastructure. Set KAFKA_ROUNDTRIP_TESTS=1 to run.",
- )
- def test_full_job_timestamp_window_filtering_all_permutations(self, cluster: ClickhouseCluster):
- """End-to-end test of the full Dagster job with all timestamp permutations.
-
- Tests the complete pipeline:
- 1. Creates persons in Postgres with different _timestamp values
- 2. Inserts events in ClickHouse with different timestamps
- 3. Runs the full Dagster reconciliation job
- 4. Verifies only the correct persons were updated in Postgres
-
- Test matrix (person _timestamp × event timestamp):
- | Person _timestamp | Event timestamp | Should be reconciled? |
- |-------------------|-----------------|----------------------|
- | BEFORE | BEFORE | NO |
- | BEFORE | DURING | YES |
- | BEFORE | AFTER | YES |
- | DURING | BEFORE | NO |
- | DURING | DURING | YES |
- | DURING | AFTER | YES |
- | AFTER | BEFORE | NO |
- | AFTER | DURING | YES |
- | AFTER | AFTER | YES |
-
- Key insight: Only EVENT timestamp matters, not person _timestamp.
- Events BEFORE bug_window_start are filtered; events DURING or AFTER are included.
- """
- from django.test import override_settings
-
- from posthog.dags.person_property_reconciliation import person_property_reconciliation_job
- from posthog.kafka_client.client import _KafkaProducer
- from posthog.models import Organization, Team
- from posthog.test.persons import create_person
-
- # Create two organizations and teams for isolation
- org1 = Organization.objects.create(name="Test Org 1 for timestamp permutations")
- org2 = Organization.objects.create(name="Test Org 2 for timestamp permutations")
- team1 = Team.objects.create(organization=org1, name="Team 1")
- team2 = Team.objects.create(organization=org2, name="Team 2")
-
- now = datetime.now().replace(microsecond=0)
-
- # Define time periods
- bug_window_start = now - timedelta(days=10)
- bug_window_end = now - timedelta(days=5)
-
- # Timestamps for each period
- ts_before = now - timedelta(days=15) # Before bug_window_start
- ts_during = now - timedelta(days=7) # Between bug_window_start and bug_window_end
- ts_after = now - timedelta(days=2) # After bug_window_end but before now()
-
- # Test cases for team 1: all 9 permutations
- # Format: (suffix, person_ts, event_ts, should_be_reconciled, prop_name)
- test_cases_team1 = [
- # Person _timestamp BEFORE bug_window_start
- ("t1_001", ts_before, ts_before, False, "prop_before_before"),
- ("t1_002", ts_before, ts_during, True, "prop_before_during"),
- ("t1_003", ts_before, ts_after, True, "prop_before_after"),
- # Person _timestamp DURING bug_window
- ("t1_004", ts_during, ts_before, False, "prop_during_before"),
- ("t1_005", ts_during, ts_during, True, "prop_during_during"),
- ("t1_006", ts_during, ts_after, True, "prop_during_after"),
- # Person _timestamp AFTER bug_window_end
- ("t1_007", ts_after, ts_before, False, "prop_after_before"),
- ("t1_008", ts_after, ts_during, True, "prop_after_during"),
- ("t1_009", ts_after, ts_after, True, "prop_after_after"),
- ]
-
- # Test cases for team 2: subset to verify team isolation
- test_cases_team2 = [
- ("t2_001", ts_before, ts_during, True, "team2_prop_1"),
- ("t2_002", ts_during, ts_after, True, "team2_prop_2"),
- ("t2_003", ts_after, ts_before, False, "team2_prop_3"),
- ]
-
- # Create persons in Postgres for team 1
- persons_team1 = {}
- for suffix, _person_ts, _event_ts, _expected, prop_name in test_cases_team1:
- person = create_person(
- team_id=team1.id,
- properties={prop_name: "old_value"},
- properties_last_updated_at={prop_name: "2020-01-01T00:00:00+00:00"},
- properties_last_operation={prop_name: "set"},
- version=1,
- )
- persons_team1[suffix] = person
-
- # Create persons in Postgres for team 2
- persons_team2 = {}
- for suffix, _person_ts, _event_ts, _expected, prop_name in test_cases_team2:
- person = create_person(
- team_id=team2.id,
- properties={prop_name: "old_value"},
- properties_last_updated_at={prop_name: "2020-01-01T00:00:00+00:00"},
- properties_last_operation={prop_name: "set"},
- version=1,
- )
- persons_team2[suffix] = person
-
- # Insert events in ClickHouse for team 1
- events_team1 = []
- for suffix, _person_ts, event_ts, _expected, prop_name in test_cases_team1:
- person = persons_team1[suffix]
- events_team1.append(
- (
- team1.id,
- f"distinct_{suffix}",
- person.uuid,
- event_ts,
- json.dumps({"$set": {prop_name: "new_value"}}),
- )
- )
-
- def insert_events_team1(client: Client) -> None:
- client.execute(
- """INSERT INTO writable_events (team_id, distinct_id, person_id, timestamp, properties)
- VALUES""",
- events_team1,
- )
-
- cluster.any_host(insert_events_team1).result()
-
- # Insert events in ClickHouse for team 2
- events_team2 = []
- for suffix, _person_ts, event_ts, _expected, prop_name in test_cases_team2:
- person = persons_team2[suffix]
- events_team2.append(
- (
- team2.id,
- f"distinct_{suffix}",
- person.uuid,
- event_ts,
- json.dumps({"$set": {prop_name: "new_value"}}),
- )
- )
-
- def insert_events_team2(client: Client) -> None:
- client.execute(
- """INSERT INTO writable_events (team_id, distinct_id, person_id, timestamp, properties)
- VALUES""",
- events_team2,
- )
-
- cluster.any_host(insert_events_team2).result()
-
- # Insert persons in ClickHouse for team 1 (required for the query join)
- persons_ch_team1 = []
- for suffix, person_ts, _event_ts, _expected, prop_name in test_cases_team1:
- person = persons_team1[suffix]
- persons_ch_team1.append(
- (
- team1.id,
- person.uuid,
- json.dumps({prop_name: "old_value"}),
- 1,
- 0, # is_deleted
- person_ts, # _timestamp
- )
- )
-
- def insert_persons_ch_team1(client: Client) -> None:
- client.execute(
- """INSERT INTO person (team_id, id, properties, version, is_deleted, _timestamp)
- VALUES""",
- persons_ch_team1,
- )
-
- cluster.any_host(insert_persons_ch_team1).result()
-
- # Insert persons in ClickHouse for team 2
- persons_ch_team2 = []
- for suffix, person_ts, _event_ts, _expected, prop_name in test_cases_team2:
- person = persons_team2[suffix]
- persons_ch_team2.append(
- (
- team2.id,
- person.uuid,
- json.dumps({prop_name: "old_value"}),
- 1,
- 0,
- person_ts,
- )
- )
-
- def insert_persons_ch_team2(client: Client) -> None:
- client.execute(
- """INSERT INTO person (team_id, id, properties, version, is_deleted, _timestamp)
- VALUES""",
- persons_ch_team2,
- )
-
- cluster.any_host(insert_persons_ch_team2).result()
-
- # Run the Dagster job
- from posthog.persons_db import persons_db_connection
-
- # Non-autocommit: the dagster job drives commit()/rollback() on this resource connection.
- with persons_db_connection(writer=True) as persons_conn, override_settings(TEST=False):
- kafka_producer = _KafkaProducer(test=False)
-
- try:
- result = person_property_reconciliation_job.execute_in_process(
- run_config={
- "ops": {
- "get_team_ids_to_reconcile": {
- "config": {
- "team_ids": [team1.id, team2.id],
- "bug_window_start": bug_window_start.strftime("%Y-%m-%d %H:%M:%S"),
- "bug_window_end": bug_window_end.strftime("%Y-%m-%d %H:%M:%S"),
- "dry_run": False,
- "backup_enabled": False,
- "batch_size": 100,
- }
- },
- "reconcile_team_chunk": {
- "config": {
- "bug_window_start": bug_window_start.strftime("%Y-%m-%d %H:%M:%S"),
- "dry_run": False,
- "backup_enabled": False,
- "batch_size": 100,
- }
- },
- }
- },
- resources={
- "cluster": cluster,
- "persons_database": persons_conn,
- "kafka_producer": kafka_producer,
- },
- )
-
- assert result.success, f"Dagster job failed: {result}"
- kafka_producer.flush(timeout=10)
-
- finally:
- kafka_producer.close()
-
- # Verify team 1 results in Postgres
- for suffix, _person_ts, _event_ts, should_be_reconciled, prop_name in test_cases_team1:
- person = persons_team1[suffix]
- refresh_person_from_persons_db(person)
-
- if should_be_reconciled:
- assert person.properties[prop_name] == "new_value", (
- f"Person {suffix} should have been reconciled. "
- f"Expected 'new_value', got '{person.properties.get(prop_name)}'"
- )
- assert person.version == 2, f"Person {suffix} version should be 2, got {person.version}"
- else:
- assert person.properties[prop_name] == "old_value", (
- f"Person {suffix} should NOT have been reconciled. "
- f"Expected 'old_value', got '{person.properties.get(prop_name)}'"
- )
- assert person.version == 1, f"Person {suffix} version should still be 1, got {person.version}"
-
- # Verify team 2 results in Postgres
- for suffix, _person_ts, _event_ts, should_be_reconciled, prop_name in test_cases_team2:
- person = persons_team2[suffix]
- refresh_person_from_persons_db(person)
-
- if should_be_reconciled:
- assert person.properties[prop_name] == "new_value", (
- f"Person {suffix} should have been reconciled. "
- f"Expected 'new_value', got '{person.properties.get(prop_name)}'"
- )
- assert person.version == 2, f"Person {suffix} version should be 2, got {person.version}"
- else:
- assert person.properties[prop_name] == "old_value", (
- f"Person {suffix} should NOT have been reconciled. "
- f"Expected 'old_value', got '{person.properties.get(prop_name)}'"
- )
- assert person.version == 1, f"Person {suffix} version should still be 1, got {person.version}"
-
- # Count reconciled vs not reconciled for summary
- reconciled_team1 = sum(1 for _, _, _, expected, _ in test_cases_team1 if expected)
- not_reconciled_team1 = len(test_cases_team1) - reconciled_team1
- reconciled_team2 = sum(1 for _, _, _, expected, _ in test_cases_team2 if expected)
- not_reconciled_team2 = len(test_cases_team2) - reconciled_team2
-
- # Verify we tested the expected number of cases
- assert reconciled_team1 == 6, f"Expected 6 reconciled in team 1, got {reconciled_team1}"
- assert not_reconciled_team1 == 3, f"Expected 3 not reconciled in team 1, got {not_reconciled_team1}"
- assert reconciled_team2 == 2, f"Expected 2 reconciled in team 2, got {reconciled_team2}"
- assert not_reconciled_team2 == 1, f"Expected 1 not reconciled in team 2, got {not_reconciled_team2}"
-
-
-class TestFetchPersonPropertiesFromClickHouse:
- """Integration tests for fetch_person_properties_from_clickhouse against real ClickHouse."""
-
- def test_fetches_oldest_version_gte_min_version(self, cluster: ClickhouseCluster):
- """Test that we fetch the oldest available version >= min_version.
-
- The function uses argMin to get the oldest version that's at least as new as
- the requested min_version. This handles cases where the exact version may have
- been merged away by ReplacingMergeTree.
- """
- team_id = 99801
- person_id = UUID("aaaaaaaa-aaaa-aaaa-aaaa-000000000001")
- now = datetime.now().replace(microsecond=0)
-
- # Insert each version in separate INSERT statements to create separate parts
- def insert_v1(client: Client) -> None:
- client.execute(
- """INSERT INTO person (team_id, id, properties, version, _timestamp)
- VALUES""",
- [(team_id, person_id, json.dumps({"email": "[email protected]"}), 1, now - timedelta(days=5))],
- )
-
- def insert_v3(client: Client) -> None:
- client.execute(
- """INSERT INTO person (team_id, id, properties, version, _timestamp)
- VALUES""",
- [(team_id, person_id, json.dumps({"email": "[email protected]"}), 3, now - timedelta(days=3))],
- )
-
- def insert_v5(client: Client) -> None:
- client.execute(
- """INSERT INTO person (team_id, id, properties, version, _timestamp)
- VALUES""",
- [(team_id, person_id, json.dumps({"email": "[email protected]"}), 5, now - timedelta(days=1))],
- )
-
- cluster.any_host(insert_v1).result()
- cluster.any_host(insert_v3).result()
- cluster.any_host(insert_v5).result()
-
- # min_version=1 should get v1 (oldest >= 1)
- props = fetch_person_properties_from_clickhouse(team_id, str(person_id), min_version=1)
- assert props is not None
- assert props["email"] == "[email protected]"
-
- # min_version=2 should get v3 (oldest >= 2, since v2 doesn't exist)
- props = fetch_person_properties_from_clickhouse(team_id, str(person_id), min_version=2)
- assert props is not None
- assert props["email"] == "[email protected]"
-
- # min_version=3 should get v3 (oldest >= 3)
- props = fetch_person_properties_from_clickhouse(team_id, str(person_id), min_version=3)
- assert props is not None
- assert props["email"] == "[email protected]"
-
- # min_version=4 should get v5 (oldest >= 4, since v4 doesn't exist)
- props = fetch_person_properties_from_clickhouse(team_id, str(person_id), min_version=4)
- assert props is not None
- assert props["email"] == "[email protected]"
-
- # min_version=5 should get v5
- props = fetch_person_properties_from_clickhouse(team_id, str(person_id), min_version=5)
- assert props is not None
- assert props["email"] == "[email protected]"
-
- def test_returns_none_when_no_version_gte_min(self, cluster: ClickhouseCluster):
- """Test that None is returned when no version >= min_version exists."""
- team_id = 99802
- person_id = UUID("bbbbbbbb-bbbb-bbbb-bbbb-000000000002")
- now = datetime.now().replace(microsecond=0)
-
- # Insert person with only version 5
- def insert_person(client: Client) -> None:
- client.execute(
- """INSERT INTO person (team_id, id, properties, version, _timestamp)
- VALUES""",
- [(team_id, person_id, json.dumps({"email": "[email protected]"}), 5, now)],
- )
-
- cluster.any_host(insert_person).result()
-
- # min_version=5 exists
- result = fetch_person_properties_from_clickhouse(team_id, str(person_id), min_version=5)
- assert result is not None
-
- # min_version=6 - no version >= 6 exists
- assert fetch_person_properties_from_clickhouse(team_id, str(person_id), min_version=6) is None
-
- # min_version=99 - no version >= 99 exists
- assert fetch_person_properties_from_clickhouse(team_id, str(person_id), min_version=99) is None
-
- def test_returns_none_for_nonexistent_person(self, cluster: ClickhouseCluster):
- """Test that None is returned when the person doesn't exist."""
- team_id = 99803
- nonexistent_person_id = "cccccccc-cccc-cccc-cccc-000000000003"
-
- result = fetch_person_properties_from_clickhouse(team_id, nonexistent_person_id, min_version=1)
- assert result is None
-
- def test_returns_empty_dict_for_empty_properties(self, cluster: ClickhouseCluster):
- """Test that empty dict is returned when properties are empty."""
- team_id = 99804
- person_id = UUID("dddddddd-dddd-dddd-dddd-000000000004")
- now = datetime.now().replace(microsecond=0)
-
- # Insert person with empty properties
- person_data = [(team_id, person_id, json.dumps({}), 1, now)]
-
- def insert_person(client: Client) -> None:
- client.execute(
- """INSERT INTO person (team_id, id, properties, version, _timestamp)
- VALUES""",
- person_data,
- )
-
- cluster.any_host(insert_person).result()
-
- result = fetch_person_properties_from_clickhouse(team_id, str(person_id), min_version=1)
- assert result == {}
-
-
-class TestReconcileWithConcurrentChanges:
- """Unit tests for reconcile_with_concurrent_changes 3-way merge logic."""
-
- def test_applies_event_changes_when_no_concurrent_postgres_changes(self):
- """When Postgres hasn't changed from CH baseline, all event changes apply."""
- ch_properties = {"email": "[email protected]", "name": "Original"}
- postgres_person = {
- "properties": {"email": "[email protected]", "name": "Original"},
- "properties_last_updated_at": {},
- "properties_last_operation": {},
- }
- person_diffs = PersonPropertyDiffs(
- person_id="test-person",
- person_version=1,
- set_updates={"email": PropertyValue(timestamp=datetime(2024, 1, 15, 12, 0, 0), value="[email protected]")},
- set_once_updates={},
- unset_updates={},
- )
-
- result = reconcile_with_concurrent_changes(ch_properties, postgres_person, person_diffs)
-
- assert result is not None
- assert result["properties"]["email"] == "[email protected]"
- assert result["properties"]["name"] == "Original" # unchanged
-
- def test_postgres_wins_on_conflict(self):
- """When Postgres has changed a key that events also change, Postgres wins."""
- ch_properties = {"email": "[email protected]"}
- postgres_person = {
- "properties": {"email": "[email protected]"}, # Postgres changed this
- "properties_last_updated_at": {"email": "2024-01-16T00:00:00"},
- "properties_last_operation": {"email": "set"},
- }
- person_diffs = PersonPropertyDiffs(
- person_id="test-person",
- person_version=1,
- set_updates={
- "email": PropertyValue(timestamp=datetime(2024, 1, 15, 12, 0, 0), value="[email protected]")
- },
- set_once_updates={},
- unset_updates={},
- )
-
- result = reconcile_with_concurrent_changes(ch_properties, postgres_person, person_diffs)
-
- # No changes - Postgres already has a different value, so our change is skipped
- assert result is None
-
- def test_applies_event_change_to_unconflicted_key(self):
- """Event changes apply to keys that Postgres hasn't concurrently modified."""
- ch_properties = {"email": "[email protected]", "name": "Original"}
- postgres_person = {
- "properties": {"email": "[email protected]", "name": "Original"}, # Only email changed
- "properties_last_updated_at": {"email": "2024-01-16T00:00:00"},
- "properties_last_operation": {"email": "set"},
- }
- person_diffs = PersonPropertyDiffs(
- person_id="test-person",
- person_version=1,
- set_updates={
- "email": PropertyValue(timestamp=datetime(2024, 1, 15, 12, 0, 0), value="[email protected]"),
- "name": PropertyValue(timestamp=datetime(2024, 1, 15, 12, 0, 0), value="Event Name"),
- },
- set_once_updates={},
- unset_updates={},
- )
-
- result = reconcile_with_concurrent_changes(ch_properties, postgres_person, person_diffs)
-
- assert result is not None
- # email: Postgres wins (concurrent change)
- assert result["properties"]["email"] == "[email protected]"
- # name: event wins (no concurrent change)
- assert result["properties"]["name"] == "Event Name"
-
- def test_set_once_applies_for_new_key(self):
- """set_once applies when key doesn't exist in Postgres."""
- ch_properties: dict[str, Any] = {}
- postgres_person: dict[str, Any] = {
- "properties": {},
- "properties_last_updated_at": {},
- "properties_last_operation": {},
- }
- person_diffs = PersonPropertyDiffs(
- person_id="test-person",
- person_version=1,
- set_updates={},
- set_once_updates={"referrer": PropertyValue(timestamp=datetime(2024, 1, 10, 8, 0, 0), value="google.com")},
- unset_updates={},
- )
-
- result = reconcile_with_concurrent_changes(ch_properties, postgres_person, person_diffs)
-
- assert result is not None
- assert result["properties"]["referrer"] == "google.com"
-
- def test_set_once_skipped_when_key_exists_in_postgres(self):
- """set_once is skipped when key already exists in Postgres (even if added concurrently)."""
- ch_properties: dict[str, Any] = {} # Key didn't exist at CH version
- postgres_person: dict[str, Any] = {
- "properties": {"referrer": "facebook.com"}, # But now exists in Postgres
- "properties_last_updated_at": {"referrer": "2024-01-12T00:00:00"},
- "properties_last_operation": {"referrer": "set"},
- }
- person_diffs = PersonPropertyDiffs(
- person_id="test-person",
- person_version=1,
- set_updates={},
- set_once_updates={"referrer": PropertyValue(timestamp=datetime(2024, 1, 10, 8, 0, 0), value="google.com")},
- unset_updates={},
- )
-
- result = reconcile_with_concurrent_changes(ch_properties, postgres_person, person_diffs)
-
- # No change - key exists in Postgres
- assert result is None
-
- def test_unset_applies_when_key_not_concurrently_changed(self):
- """$unset removes key when Postgres hasn't concurrently modified it."""
- ch_properties = {"email": "[email protected]", "name": "Original"}
- postgres_person = {
- "properties": {"email": "[email protected]", "name": "Original"},
- "properties_last_updated_at": {},
- "properties_last_operation": {},
- }
- person_diffs = PersonPropertyDiffs(
- person_id="test-person",
- person_version=1,
- set_updates={},
- set_once_updates={},
- unset_updates={"email": PropertyValue(timestamp=datetime(2024, 1, 15, 12, 0, 0), value=None)},
- )
-
- result = reconcile_with_concurrent_changes(ch_properties, postgres_person, person_diffs)
-
- assert result is not None
- assert "email" not in result["properties"]
- assert result["properties"]["name"] == "Original"
-
- def test_unset_skipped_when_key_concurrently_changed(self):
- """$unset is skipped when Postgres has concurrently modified the key."""
- ch_properties = {"email": "[email protected]"}
- postgres_person = {
- "properties": {"email": "[email protected]"}, # Postgres changed this
- "properties_last_updated_at": {"email": "2024-01-16T00:00:00"},
- "properties_last_operation": {"email": "set"},
- }
- person_diffs = PersonPropertyDiffs(
- person_id="test-person",
- person_version=1,
- set_updates={},
- set_once_updates={},
- unset_updates={"email": PropertyValue(timestamp=datetime(2024, 1, 15, 12, 0, 0), value=None)},
- )
-
- result = reconcile_with_concurrent_changes(ch_properties, postgres_person, person_diffs)
-
- # No change - Postgres has concurrent modification
- assert result is None
-
- def test_preserves_postgres_additions(self):
- """Keys added to Postgres concurrently should be preserved."""
- ch_properties = {"email": "[email protected]"}
- postgres_person = {
- "properties": {"email": "[email protected]", "new_key": "postgres_added"},
- "properties_last_updated_at": {"new_key": "2024-01-16T00:00:00"},
- "properties_last_operation": {"new_key": "set"},
- }
- person_diffs = PersonPropertyDiffs(
- person_id="test-person",
- person_version=1,
- set_updates={"name": PropertyValue(timestamp=datetime(2024, 1, 15, 12, 0, 0), value="Event Name")},
- set_once_updates={},
- unset_updates={},
- )
-
- result = reconcile_with_concurrent_changes(ch_properties, postgres_person, person_diffs)
-
- assert result is not None
- assert result["properties"]["email"] == "[email protected]"
- assert result["properties"]["new_key"] == "postgres_added" # Preserved
- assert result["properties"]["name"] == "Event Name" # Event change applied
-
- def test_mixed_operations_with_conflicts(self):
- """Test complex scenario with multiple operations and partial conflicts."""
- ch_properties = {"a": "ch_a", "b": "ch_b", "c": "ch_c"}
- postgres_person = {
- "properties": {"a": "pg_a", "b": "ch_b", "c": "ch_c", "d": "pg_d"}, # a changed, d added
- "properties_last_updated_at": {"a": "2024-01-16T00:00:00", "d": "2024-01-16T00:00:00"},
- "properties_last_operation": {"a": "set", "d": "set"},
- }
- person_diffs = PersonPropertyDiffs(
- person_id="test-person",
- person_version=1,
- set_updates={
- "a": PropertyValue(timestamp=datetime(2024, 1, 15, 12, 0, 0), value="event_a"), # Conflict
- "b": PropertyValue(timestamp=datetime(2024, 1, 15, 12, 0, 0), value="event_b"), # No conflict
- },
- set_once_updates={
- "e": PropertyValue(timestamp=datetime(2024, 1, 10, 8, 0, 0), value="event_e"), # New key
- },
- unset_updates={
- "c": PropertyValue(timestamp=datetime(2024, 1, 15, 12, 0, 0), value=None), # No conflict
- },
- )
-
- result = reconcile_with_concurrent_changes(ch_properties, postgres_person, person_diffs)
-
- assert result is not None
- assert result["properties"]["a"] == "pg_a" # Postgres wins (conflict)
- assert result["properties"]["b"] == "event_b" # Event wins (no conflict)
- assert "c" not in result["properties"] # Unset applied (no conflict)
- assert result["properties"]["d"] == "pg_d" # Postgres addition preserved
- assert result["properties"]["e"] == "event_e" # set_once applied (new key)
-
-
-class TestQueryTeamIdsFromClickHouse:
- """Integration tests for query_team_ids_from_clickhouse with team_id filters."""
-
- def test_returns_teams_with_property_events(self, cluster: ClickhouseCluster):
- """Basic test that teams with $set/$set_once/$unset events are returned."""
- now = datetime.now().replace(microsecond=0)
- bug_window_start = now - timedelta(days=10)
- bug_window_end = now + timedelta(days=1)
-
- # Use unique team_ids unlikely to conflict with other tests (88001-88005)
- events = [
- (
- 88001,
- "d1",
- UUID("11111111-1111-1111-1111-880000000001"),
- now - timedelta(days=5),
- json.dumps({"$set": {"a": 1}}),
- ),
- (
- 88002,
- "d2",
- UUID("11111111-1111-1111-1111-880000000002"),
- now - timedelta(days=5),
- json.dumps({"$set_once": {"b": 2}}),
- ),
- (
- 88003,
- "d3",
- UUID("11111111-1111-1111-1111-880000000003"),
- now - timedelta(days=5),
- json.dumps({"$unset": ["c"]}),
- ),
- (
- 88004,
- "d4",
- UUID("11111111-1111-1111-1111-880000000004"),
- now - timedelta(days=5),
- json.dumps({"$set": {"d": 4}}),
- ),
- (
- 88005,
- "d5",
- UUID("11111111-1111-1111-1111-880000000005"),
- now - timedelta(days=5),
- json.dumps({"other": "no_props"}),
- ),
- ]
-
- def insert_events(client: Client) -> None:
- client.execute(
- "INSERT INTO writable_events (team_id, distinct_id, person_id, timestamp, properties) VALUES",
- events,
- )
-
- cluster.any_host(insert_events).result()
-
- result = query_team_ids_from_clickhouse(
- bug_window_start=bug_window_start.strftime("%Y-%m-%d %H:%M:%S"),
- bug_window_end=bug_window_end.strftime("%Y-%m-%d %H:%M:%S"),
- min_team_id=88001,
- max_team_id=88005,
- )
-
- # 88005 has no $set/$set_once/$unset so should not be included
- assert result == [88001, 88002, 88003, 88004]
-
- def test_min_team_id_filter(self, cluster: ClickhouseCluster):
- """Test that min_team_id filters out teams below the threshold."""
- now = datetime.now().replace(microsecond=0)
- bug_window_start = now - timedelta(days=10)
- bug_window_end = now + timedelta(days=1)
-
- events = [
- (
- 89001,
- "d1",
- UUID("11111111-1111-1111-1111-890000000001"),
- now - timedelta(days=5),
- json.dumps({"$set": {"a": 1}}),
- ),
- (
- 89002,
- "d2",
- UUID("11111111-1111-1111-1111-890000000002"),
- now - timedelta(days=5),
- json.dumps({"$set": {"b": 2}}),
- ),
- (
- 89003,
- "d3",
- UUID("11111111-1111-1111-1111-890000000003"),
- now - timedelta(days=5),
- json.dumps({"$set": {"c": 3}}),
- ),
- ]
-
- def insert_events(client: Client) -> None:
- client.execute(
- "INSERT INTO writable_events (team_id, distinct_id, person_id, timestamp, properties) VALUES",
- events,
- )
-
- cluster.any_host(insert_events).result()
-
- result = query_team_ids_from_clickhouse(
- bug_window_start=bug_window_start.strftime("%Y-%m-%d %H:%M:%S"),
- bug_window_end=bug_window_end.strftime("%Y-%m-%d %H:%M:%S"),
- min_team_id=89002,
- max_team_id=89003,
- )
-
- assert result == [89002, 89003]
-
- def test_max_team_id_filter(self, cluster: ClickhouseCluster):
- """Test that max_team_id filters out teams above the threshold."""
- now = datetime.now().replace(microsecond=0)
- bug_window_start = now - timedelta(days=10)
- bug_window_end = now + timedelta(days=1)
-
- events = [
- (
- 90001,
- "d1",
- UUID("11111111-1111-1111-1111-900000000001"),
- now - timedelta(days=5),
- json.dumps({"$set": {"a": 1}}),
- ),
- (
- 90002,
- "d2",
- UUID("11111111-1111-1111-1111-900000000002"),
- now - timedelta(days=5),
- json.dumps({"$set": {"b": 2}}),
- ),
- (
- 90003,
- "d3",
- UUID("11111111-1111-1111-1111-900000000003"),
- now - timedelta(days=5),
- json.dumps({"$set": {"c": 3}}),
- ),
- ]
-
- def insert_events(client: Client) -> None:
- client.execute(
- "INSERT INTO writable_events (team_id, distinct_id, person_id, timestamp, properties) VALUES",
- events,
- )
-
- cluster.any_host(insert_events).result()
-
- result = query_team_ids_from_clickhouse(
- bug_window_start=bug_window_start.strftime("%Y-%m-%d %H:%M:%S"),
- bug_window_end=bug_window_end.strftime("%Y-%m-%d %H:%M:%S"),
- min_team_id=90001,
- max_team_id=90002,
- )
-
- assert result == [90001, 90002]
-
- def test_both_min_and_max_team_id_filters(self, cluster: ClickhouseCluster):
- """Test that both min and max filters work together."""
- now = datetime.now().replace(microsecond=0)
- bug_window_start = now - timedelta(days=10)
- bug_window_end = now + timedelta(days=1)
-
- events = [
- (
- 91001,
- "d1",
- UUID("11111111-1111-1111-1111-910000000001"),
- now - timedelta(days=5),
- json.dumps({"$set": {"a": 1}}),
- ),
- (
- 91002,
- "d2",
- UUID("11111111-1111-1111-1111-910000000002"),
- now - timedelta(days=5),
- json.dumps({"$set": {"b": 2}}),
- ),
- (
- 91003,
- "d3",
- UUID("11111111-1111-1111-1111-910000000003"),
- now - timedelta(days=5),
- json.dumps({"$set": {"c": 3}}),
- ),
- (
- 91004,
- "d4",
- UUID("11111111-1111-1111-1111-910000000004"),
- now - timedelta(days=5),
- json.dumps({"$set": {"d": 4}}),
- ),
- (
- 91005,
- "d5",
- UUID("11111111-1111-1111-1111-910000000005"),
- now - timedelta(days=5),
- json.dumps({"$set": {"e": 5}}),
- ),
- ]
-
- def insert_events(client: Client) -> None:
- client.execute(
- "INSERT INTO writable_events (team_id, distinct_id, person_id, timestamp, properties) VALUES",
- events,
- )
-
- cluster.any_host(insert_events).result()
-
- result = query_team_ids_from_clickhouse(
- bug_window_start=bug_window_start.strftime("%Y-%m-%d %H:%M:%S"),
- bug_window_end=bug_window_end.strftime("%Y-%m-%d %H:%M:%S"),
- min_team_id=91002,
- max_team_id=91004,
- )
-
- assert result == [91002, 91003, 91004]
-
- def test_team_range_returns_all_matching_teams(self, cluster: ClickhouseCluster):
- """Test that team range filter returns all teams with property events within range."""
- now = datetime.now().replace(microsecond=0)
- bug_window_start = now - timedelta(days=10)
- bug_window_end = now + timedelta(days=1)
-
- events = [
- (
- 92001,
- "d1",
- UUID("11111111-1111-1111-1111-920000000001"),
- now - timedelta(days=5),
- json.dumps({"$set": {"a": 1}}),
- ),
- (
- 92002,
- "d2",
- UUID("11111111-1111-1111-1111-920000000002"),
- now - timedelta(days=5),
- json.dumps({"$set": {"b": 2}}),
- ),
- ]
-
- def insert_events(client: Client) -> None:
- client.execute(
- "INSERT INTO writable_events (team_id, distinct_id, person_id, timestamp, properties) VALUES",
- events,
- )
-
- cluster.any_host(insert_events).result()
-
- result = query_team_ids_from_clickhouse(
- bug_window_start=bug_window_start.strftime("%Y-%m-%d %H:%M:%S"),
- bug_window_end=bug_window_end.strftime("%Y-%m-%d %H:%M:%S"),
- min_team_id=92001,
- max_team_id=92002,
- )
-
- assert 92001 in result
- assert 92002 in result
-
- def test_exclude_team_ids_filters_out_specified_teams(self, cluster: ClickhouseCluster):
- """Test that exclude_team_ids filters out specified teams from results."""
- now = datetime.now().replace(microsecond=0)
- bug_window_start = now - timedelta(days=10)
- bug_window_end = now + timedelta(days=1)
-
- events = [
- (
- 93001,
- "d1",
- UUID("11111111-1111-1111-1111-930000000001"),
- now - timedelta(days=5),
- json.dumps({"$set": {"a": 1}}),
- ),
- (
- 93002,
- "d2",
- UUID("11111111-1111-1111-1111-930000000002"),
- now - timedelta(days=5),
- json.dumps({"$set": {"b": 2}}),
- ),
- (
- 93003,
- "d3",
- UUID("11111111-1111-1111-1111-930000000003"),
- now - timedelta(days=5),
- json.dumps({"$set": {"c": 3}}),
- ),
- (
- 93004,
- "d4",
- UUID("11111111-1111-1111-1111-930000000004"),
- now - timedelta(days=5),
- json.dumps({"$set": {"d": 4}}),
- ),
- ]
-
- def insert_events(client: Client) -> None:
- client.execute(
- "INSERT INTO writable_events (team_id, distinct_id, person_id, timestamp, properties) VALUES",
- events,
- )
-
- cluster.any_host(insert_events).result()
-
- result = query_team_ids_from_clickhouse(
- bug_window_start=bug_window_start.strftime("%Y-%m-%d %H:%M:%S"),
- bug_window_end=bug_window_end.strftime("%Y-%m-%d %H:%M:%S"),
- min_team_id=93001,
- max_team_id=93004,
- exclude_team_ids=[93002, 93004],
- )
-
- assert result == [93001, 93003]
-
- def test_include_team_ids_filters_to_specified_teams(self, cluster: ClickhouseCluster):
- """Test that include_team_ids only returns teams in the specified list."""
- bug_window_start = datetime.now(UTC) - timedelta(hours=2)
- bug_window_end = datetime.now(UTC) + timedelta(hours=1)
- event_time = datetime.now(UTC) - timedelta(hours=1)
-
- # Create events for teams 94001-94005
- events = [
- (
- 94001,
- "user_a",
- "00000000-0000-0000-0000-000000094001",
- event_time,
- json.dumps({"$set": {"a": 1}}),
- ),
- (
- 94002,
- "user_b",
- "00000000-0000-0000-0000-000000094002",
- event_time,
- json.dumps({"$set": {"b": 2}}),
- ),
- (
- 94003,
- "user_c",
- "00000000-0000-0000-0000-000000094003",
- event_time,
- json.dumps({"$set": {"c": 3}}),
- ),
- (
- 94004,
- "user_d",
- "00000000-0000-0000-0000-000000094004",
- event_time,
- json.dumps({"$set": {"d": 4}}),
- ),
- (
- 94005,
- "user_e",
- "00000000-0000-0000-0000-000000094005",
- event_time,
- json.dumps({"$set": {"e": 5}}),
- ),
- ]
-
- def insert_events(client: Client) -> None:
- client.execute(
- "INSERT INTO writable_events (team_id, distinct_id, person_id, timestamp, properties) VALUES",
- events,
- )
-
- cluster.any_host(insert_events).result()
-
- # Only include teams 94002 and 94004 - should not return 94001, 94003, 94005
- result = query_team_ids_from_clickhouse(
- bug_window_start=bug_window_start.strftime("%Y-%m-%d %H:%M:%S"),
- bug_window_end=bug_window_end.strftime("%Y-%m-%d %H:%M:%S"),
- include_team_ids=[94002, 94004],
- )
-
- assert result == [94002, 94004]
-
- def test_include_team_ids_combined_with_other_filters(self, cluster: ClickhouseCluster):
- """Test that include_team_ids works as AND with other filters."""
- bug_window_start = datetime.now(UTC) - timedelta(hours=2)
- bug_window_end = datetime.now(UTC) + timedelta(hours=1)
- event_time = datetime.now(UTC) - timedelta(hours=1)
-
- # Create events for teams 95001-95004
- events = [
- (
- 95001,
- "user_a",
- "00000000-0000-0000-0000-000000095001",
- event_time,
- json.dumps({"$set": {"a": 1}}),
- ),
- (
- 95002,
- "user_b",
- "00000000-0000-0000-0000-000000095002",
- event_time,
- json.dumps({"$set": {"b": 2}}),
- ),
- (
- 95003,
- "user_c",
- "00000000-0000-0000-0000-000000095003",
- event_time,
- json.dumps({"$set": {"c": 3}}),
- ),
- (
- 95004,
- "user_d",
- "00000000-0000-0000-0000-000000095004",
- event_time,
- json.dumps({"$set": {"d": 4}}),
- ),
- ]
-
- def insert_events(client: Client) -> None:
- client.execute(
- "INSERT INTO writable_events (team_id, distinct_id, person_id, timestamp, properties) VALUES",
- events,
- )
-
- cluster.any_host(insert_events).result()
-
- # include_team_ids=[95001, 95002, 95003] AND exclude_team_ids=[95002] -> only 95001, 95003
- result = query_team_ids_from_clickhouse(
- bug_window_start=bug_window_start.strftime("%Y-%m-%d %H:%M:%S"),
- bug_window_end=bug_window_end.strftime("%Y-%m-%d %H:%M:%S"),
- include_team_ids=[95001, 95002, 95003],
- exclude_team_ids=[95002],
- )
-
- assert result == [95001, 95003]
-
- def test_invalid_range_raises_error(self):
- """Test that min_team_id > max_team_id raises ValueError."""
- with pytest.raises(ValueError) as exc_info:
- query_team_ids_from_clickhouse(
- bug_window_start="2024-01-01 00:00:00",
- bug_window_end="2024-01-02 00:00:00",
- min_team_id=100,
- max_team_id=50,
- )
-
- assert "min_team_id (100) cannot be greater than max_team_id (50)" in str(exc_info.value)
-
-
-class TestReconcileSingleTeamErrorHandling:
- """Tests for error handling in reconcile_single_team and reconcile_team_chunk."""
-
- def test_team_failure_does_not_crash_chunk(self):
- """Test that if one team fails, the chunk continues processing other teams."""
- from posthog.dags.person_property_reconciliation import (
- PersonPropertyReconciliationConfig,
- TeamReconciliationResult,
- )
-
- # Mock reconcile_single_team to fail for team 2 but succeed for teams 1 and 3
- call_count = {"value": 0}
- original_results = {
- 1: TeamReconciliationResult(team_id=1, persons_processed=10, persons_updated=5, persons_skipped=5),
- 3: TeamReconciliationResult(team_id=3, persons_processed=20, persons_updated=15, persons_skipped=5),
- }
-
- def mock_reconcile_single_team(team_id, **kwargs):
- call_count["value"] += 1
- if team_id == 2:
- raise Exception("Simulated failure for team 2")
- return original_results[team_id]
-
- with patch(
- "posthog.dags.person_property_reconciliation.reconcile_single_team",
- side_effect=mock_reconcile_single_team,
- ):
- # Create mock context with run info
- mock_context = MagicMock()
- mock_context.run.job_name = "test_job"
- mock_context.run.run_id = "test_run_123"
-
- # Create real config
- config = PersonPropertyReconciliationConfig(
- bug_window_start="2024-01-01 00:00:00",
- batch_size=100,
- dry_run=False,
- backup_enabled=False,
- )
-
- # Create mock resources
- mock_db = MagicMock()
- mock_cluster = MagicMock()
- mock_kafka = MagicMock()
-
- # Import the underlying function (not the op wrapper)
- from posthog.dags.person_property_reconciliation import reconcile_team_chunk
-
- # Call the op's underlying function directly via __wrapped__
- result = reconcile_team_chunk.__wrapped__( # type: ignore[attr-defined]
- context=mock_context,
- config=config,
- chunk=[1, 2, 3],
- persons_database=mock_db,
- cluster=mock_cluster,
- kafka_producer=mock_kafka,
- )
-
- # All 3 teams should have been attempted
- assert call_count["value"] == 3
-
- # Check overall results
- assert result["teams_count"] == 3
- assert result["teams_succeeded"] == 2
- assert result["teams_failed"] == 1
- assert result["persons_processed"] == 30 # 10 + 0 + 20
- assert result["persons_updated"] == 20 # 5 + 0 + 15
-
- # Check individual team results
- teams_results = result["teams_results"]
- assert len(teams_results) == 3
-
- team1_result = next(r for r in teams_results if r["team_id"] == 1)
- assert team1_result["status"] == "success"
- assert team1_result["persons_processed"] == 10
-
- team2_result = next(r for r in teams_results if r["team_id"] == 2)
- assert team2_result["status"] == "failed"
- assert "Simulated failure" in team2_result["error"]
-
- team3_result = next(r for r in teams_results if r["team_id"] == 3)
- assert team3_result["status"] == "success"
- assert team3_result["persons_processed"] == 20
-
-
-class TestReconciliationSchedulerSensor:
- """Tests for the person_property_reconciliation_scheduler sensor."""
-
- def test_build_reconciliation_run_config(self):
- """Test that run config is built correctly from scheduler config."""
- from posthog.dags.person_property_reconciliation import (
- ReconciliationSchedulerConfig,
- build_reconciliation_run_config,
- )
-
- config = ReconciliationSchedulerConfig(
- range_start=1,
- range_end=10000,
- chunk_size=1000,
- max_concurrent_jobs=5,
- max_concurrent_tasks=10,
- bug_window_start="2024-01-06 20:01:00",
- bug_window_end="2024-01-07 14:52:00",
- dry_run=False,
- backup_enabled=True,
- batch_size=100,
- )
-
- run_config = build_reconciliation_run_config(config, min_team_id=1, max_team_id=1000)
-
- assert run_config["ops"]["get_team_ids_to_reconcile"]["config"]["min_team_id"] == 1
- assert run_config["ops"]["get_team_ids_to_reconcile"]["config"]["max_team_id"] == 1000
- assert run_config["ops"]["get_team_ids_to_reconcile"]["config"]["bug_window_start"] == "2024-01-06 20:01:00"
- assert run_config["ops"]["get_team_ids_to_reconcile"]["config"]["dry_run"] is False
- assert run_config["ops"]["reconcile_team_chunk"]["config"]["backup_enabled"] is True
- assert run_config["execution"]["config"]["max_concurrent"] == 10 # max_concurrent_tasks
-
- def test_sensor_no_cursor_returns_skip_reason(self):
- """Test that sensor returns SkipReason when no cursor is set."""
- from dagster import SkipReason, build_sensor_context
-
- from posthog.dags.person_property_reconciliation import person_property_reconciliation_scheduler
-
- context = build_sensor_context(cursor=None)
- result = person_property_reconciliation_scheduler(context)
-
- assert isinstance(result, SkipReason)
- assert result.skip_message is not None
- assert "No cursor set" in result.skip_message
-
- def test_sensor_invalid_cursor_json_returns_skip_reason(self):
- """Test that sensor returns SkipReason for invalid JSON cursor."""
- from dagster import SkipReason, build_sensor_context
-
- from posthog.dags.person_property_reconciliation import person_property_reconciliation_scheduler
-
- context = build_sensor_context(cursor="not valid json {")
- result = person_property_reconciliation_scheduler(context)
-
- assert isinstance(result, SkipReason)
- assert result.skip_message is not None
- assert "Invalid cursor JSON" in result.skip_message
-
- def test_sensor_completed_range_returns_skip_reason(self):
- """Test that sensor returns SkipReason when range is completed."""
- from dagster import DagsterInstance, SkipReason, build_sensor_context
-
- from posthog.dags.person_property_reconciliation import person_property_reconciliation_scheduler
-
- cursor = json.dumps(
- {
- "range_start": 1,
- "range_end": 1000,
- "next_chunk_start": 1001,
- "bug_window_start": "2024-01-06 20:01:00",
- "bug_window_end": "2024-01-07 14:52:00",
- }
- )
-
- mock_instance = MagicMock(spec=DagsterInstance)
- mock_instance.get_run_records.return_value = []
-
- context = build_sensor_context(cursor=cursor, instance=mock_instance)
- result = person_property_reconciliation_scheduler(context)
-
- assert isinstance(result, SkipReason)
- assert result.skip_message is not None
- assert "complete" in result.skip_message.lower()
-
- def test_sensor_at_max_concurrency_returns_skip_reason(self):
- """Test that sensor returns SkipReason when at max concurrency."""
- from dagster import DagsterInstance, SkipReason, build_sensor_context
-
- from posthog.dags.person_property_reconciliation import person_property_reconciliation_scheduler
-
- cursor = json.dumps(
- {
- "range_start": 1,
- "range_end": 10000,
- "chunk_size": 1000,
- "max_concurrent_jobs": 2,
- "next_chunk_start": 1,
- "bug_window_start": "2024-01-06 20:01:00",
- "bug_window_end": "2024-01-07 14:52:00",
- }
- )
-
- mock_instance = MagicMock(spec=DagsterInstance)
- mock_instance.get_run_records.return_value = [MagicMock(), MagicMock()]
-
- context = build_sensor_context(cursor=cursor, instance=mock_instance)
- result = person_property_reconciliation_scheduler(context)
-
- assert isinstance(result, SkipReason)
- assert result.skip_message is not None
- assert "max concurrency" in result.skip_message.lower()
-
- def test_sensor_yields_runs_up_to_available_slots(self):
- """Test that sensor yields correct number of runs based on available slots."""
- from dagster import DagsterInstance, SensorResult, build_sensor_context
-
- from posthog.dags.person_property_reconciliation import person_property_reconciliation_scheduler
-
- cursor = json.dumps(
- {
- "range_start": 1,
- "range_end": 5000,
- "chunk_size": 1000,
- "max_concurrent_jobs": 3,
- "next_chunk_start": 1,
- "bug_window_start": "2024-01-06 20:01:00",
- "bug_window_end": "2024-01-07 14:52:00",
- }
- )
-
- mock_instance = MagicMock(spec=DagsterInstance)
- mock_instance.get_run_records.return_value = [MagicMock()]
-
- context = build_sensor_context(cursor=cursor, instance=mock_instance)
- result = person_property_reconciliation_scheduler(context)
-
- assert isinstance(result, SensorResult)
- assert result.run_requests is not None
- assert len(result.run_requests) == 2
- assert result.run_requests[0].tags["reconciliation_range"] == "1-1000"
- assert result.run_requests[1].tags["reconciliation_range"] == "1001-2000"
-
- assert result.cursor is not None
- new_cursor = json.loads(result.cursor)
- assert new_cursor["next_chunk_start"] == 2001
-
- def test_sensor_handles_partial_final_chunk(self):
- """Test that sensor correctly handles a final chunk smaller than chunk_size."""
- from dagster import DagsterInstance, SensorResult, build_sensor_context
-
- from posthog.dags.person_property_reconciliation import person_property_reconciliation_scheduler
-
- cursor = json.dumps(
- {
- "range_start": 1,
- "range_end": 1500,
- "chunk_size": 1000,
- "max_concurrent_jobs": 5,
- "next_chunk_start": 1001,
- "bug_window_start": "2024-01-06 20:01:00",
- "bug_window_end": "2024-01-07 14:52:00",
- }
- )
-
- mock_instance = MagicMock(spec=DagsterInstance)
- mock_instance.get_run_records.return_value = []
-
- context = build_sensor_context(cursor=cursor, instance=mock_instance)
- result = person_property_reconciliation_scheduler(context)
-
- assert isinstance(result, SensorResult)
- assert result.run_requests is not None
- assert len(result.run_requests) == 1
- assert result.run_requests[0].tags["reconciliation_range"] == "1001-1500"
-
- assert result.cursor is not None
- new_cursor = json.loads(result.cursor)
- assert new_cursor["next_chunk_start"] == 1501
-
- def test_sensor_run_request_has_correct_tags(self):
- """Test that run requests include expected tags."""
- from dagster import DagsterInstance, SensorResult, build_sensor_context
-
- from posthog.dags.person_property_reconciliation import person_property_reconciliation_scheduler
-
- cursor = json.dumps(
- {
- "range_start": 1,
- "range_end": 2000,
- "chunk_size": 1000,
- "max_concurrent_jobs": 5,
- "next_chunk_start": 1,
- "bug_window_start": "2024-01-06 20:01:00",
- "bug_window_end": "2024-01-07 14:52:00",
- }
- )
-
- mock_instance = MagicMock(spec=DagsterInstance)
- mock_instance.get_run_records.return_value = []
-
- context = build_sensor_context(cursor=cursor, instance=mock_instance)
- result = person_property_reconciliation_scheduler(context)
-
- assert isinstance(result, SensorResult)
- assert result.run_requests is not None
- assert len(result.run_requests) == 2
- assert result.run_requests[0].tags["reconciliation_range"] == "1-1000"
- assert result.run_requests[0].tags["owner"] == "team-ingestion"
-
- def test_sensor_validates_range_start_less_than_range_end(self):
- """Test that sensor rejects range_start > range_end."""
- from dagster import SkipReason, build_sensor_context
-
- from posthog.dags.person_property_reconciliation import person_property_reconciliation_scheduler
-
- cursor = json.dumps(
- {
- "range_start": 1000,
- "range_end": 500,
- "bug_window_start": "2024-01-06 20:01:00",
- "bug_window_end": "2024-01-07 14:52:00",
- }
- )
- context = build_sensor_context(cursor=cursor)
- result = person_property_reconciliation_scheduler(context)
-
- assert isinstance(result, SkipReason)
- assert result.skip_message is not None
- assert "range_start" in result.skip_message
- assert "range_end" in result.skip_message
-
- def test_sensor_validates_chunk_size_positive(self):
- """Test that sensor rejects chunk_size <= 0."""
- from dagster import SkipReason, build_sensor_context
-
- from posthog.dags.person_property_reconciliation import person_property_reconciliation_scheduler
-
- cursor = json.dumps(
- {
- "range_start": 1,
- "range_end": 1000,
- "chunk_size": 0,
- "bug_window_start": "2024-01-06 20:01:00",
- "bug_window_end": "2024-01-07 14:52:00",
- }
- )
- context = build_sensor_context(cursor=cursor)
- result = person_property_reconciliation_scheduler(context)
-
- assert isinstance(result, SkipReason)
- assert result.skip_message is not None
- assert "chunk_size" in result.skip_message
-
- def test_sensor_validates_max_concurrent_jobs_positive(self):
- """Test that sensor rejects max_concurrent_jobs <= 0."""
- from dagster import SkipReason, build_sensor_context
-
- from posthog.dags.person_property_reconciliation import person_property_reconciliation_scheduler
-
- cursor = json.dumps(
- {
- "range_start": 1,
- "range_end": 1000,
- "max_concurrent_jobs": 0,
- "bug_window_start": "2024-01-06 20:01:00",
- "bug_window_end": "2024-01-07 14:52:00",
- }
- )
- context = build_sensor_context(cursor=cursor)
- result = person_property_reconciliation_scheduler(context)
-
- assert isinstance(result, SkipReason)
- assert result.skip_message is not None
- assert "max_concurrent_jobs" in result.skip_message
-
- def test_sensor_validates_max_concurrent_jobs_cap(self):
- """Test that sensor rejects max_concurrent_jobs exceeding the cap."""
- from dagster import SkipReason, build_sensor_context
-
- from posthog.dags.person_property_reconciliation import person_property_reconciliation_scheduler
-
- cursor = json.dumps(
- {
- "range_start": 1,
- "range_end": 1000,
- "max_concurrent_jobs": 100,
- "bug_window_start": "2024-01-06 20:01:00",
- "bug_window_end": "2024-01-07 14:52:00",
- }
- )
- context = build_sensor_context(cursor=cursor)
- result = person_property_reconciliation_scheduler(context)
-
- assert isinstance(result, SkipReason)
- assert result.skip_message is not None
- assert "exceeds cap" in result.skip_message.lower()
-
- def test_sensor_validates_max_concurrent_tasks_positive(self):
- """Test that sensor rejects max_concurrent_tasks <= 0."""
- from dagster import SkipReason, build_sensor_context
-
- from posthog.dags.person_property_reconciliation import person_property_reconciliation_scheduler
-
- cursor = json.dumps(
- {
- "range_start": 1,
- "range_end": 1000,
- "max_concurrent_tasks": 0,
- "bug_window_start": "2024-01-06 20:01:00",
- "bug_window_end": "2024-01-07 14:52:00",
- }
- )
- context = build_sensor_context(cursor=cursor)
- result = person_property_reconciliation_scheduler(context)
-
- assert isinstance(result, SkipReason)
- assert result.skip_message is not None
- assert "max_concurrent_tasks" in result.skip_message
-
- def test_sensor_validates_max_concurrent_tasks_cap(self):
- """Test that sensor rejects max_concurrent_tasks exceeding the cap."""
- from dagster import SkipReason, build_sensor_context
-
- from posthog.dags.person_property_reconciliation import person_property_reconciliation_scheduler
-
- cursor = json.dumps(
- {
- "range_start": 1,
- "range_end": 1000,
- "max_concurrent_tasks": 200,
- "bug_window_start": "2024-01-06 20:01:00",
- "bug_window_end": "2024-01-07 14:52:00",
- }
- )
- context = build_sensor_context(cursor=cursor)
- result = person_property_reconciliation_scheduler(context)
-
- assert isinstance(result, SkipReason)
- assert result.skip_message is not None
- assert "exceeds cap" in result.skip_message.lower()
-
- def test_sensor_validates_bug_window_start_required(self):
- """Test that sensor rejects missing bug_window_start."""
- from dagster import SkipReason, build_sensor_context
-
- from posthog.dags.person_property_reconciliation import person_property_reconciliation_scheduler
-
- cursor = json.dumps(
- {
- "range_start": 1,
- "range_end": 1000,
- "bug_window_end": "2024-01-07 14:52:00",
- }
- )
- context = build_sensor_context(cursor=cursor)
- result = person_property_reconciliation_scheduler(context)
-
- assert isinstance(result, SkipReason)
- assert result.skip_message is not None
- assert "bug_window_start" in result.skip_message
-
- def test_sensor_validates_bug_window_end_required(self):
- """Test that sensor rejects missing bug_window_end."""
- from dagster import SkipReason, build_sensor_context
-
- from posthog.dags.person_property_reconciliation import person_property_reconciliation_scheduler
-
- cursor = json.dumps(
- {
- "range_start": 1,
- "range_end": 1000,
- "bug_window_start": "2024-01-06 20:01:00",
- }
- )
- context = build_sensor_context(cursor=cursor)
- result = person_property_reconciliation_scheduler(context)
-
- assert isinstance(result, SkipReason)
- assert result.skip_message is not None
- assert "bug_window_end" in result.skip_message
-
- def test_build_reconciliation_run_config_with_team_ids(self):
- """Test that run config passes team_ids to op config when provided."""
- from posthog.dags.person_property_reconciliation import (
- ReconciliationSchedulerConfig,
- build_reconciliation_run_config,
- )
-
- config = ReconciliationSchedulerConfig(
- team_ids=[1, 5, 10, 20],
- chunk_size=100,
- max_concurrent_jobs=5,
- max_concurrent_tasks=10,
- bug_window_start="2024-01-06 20:01:00",
- bug_window_end="2024-01-07 14:52:00",
- dry_run=True,
- backup_enabled=False,
- batch_size=50,
- )
-
- run_config = build_reconciliation_run_config(config, team_ids=[1, 5, 10])
-
- op_config = run_config["ops"]["get_team_ids_to_reconcile"]["config"]
- assert op_config["team_ids"] == [1, 5, 10]
- assert "min_team_id" not in op_config
- assert "max_team_id" not in op_config
- assert op_config["bug_window_start"] == "2024-01-06 20:01:00"
- assert op_config["dry_run"] is True
- assert run_config["ops"]["reconcile_team_chunk"]["config"]["backup_enabled"] is False
-
- def test_sensor_with_team_ids_yields_chunked_runs(self):
- """Test that sensor chunks team_ids list correctly."""
- from dagster import DagsterInstance, SensorResult, build_sensor_context
-
- from posthog.dags.person_property_reconciliation import person_property_reconciliation_scheduler
-
- cursor = json.dumps(
- {
- "team_ids": [1, 2, 3, 4, 5, 6, 7, 8, 9, 10],
- "chunk_size": 3,
- "max_concurrent_jobs": 5,
- "next_team_index": 0,
- "bug_window_start": "2024-01-06 20:01:00",
- "bug_window_end": "2024-01-07 14:52:00",
- }
- )
-
- mock_instance = MagicMock(spec=DagsterInstance)
- mock_instance.get_run_records.return_value = []
-
- context = build_sensor_context(cursor=cursor, instance=mock_instance)
- result = person_property_reconciliation_scheduler(context)
-
- assert isinstance(result, SensorResult)
- assert result.run_requests is not None
- assert len(result.run_requests) == 4 # 10 teams / 3 per chunk = 4 chunks (3+3+3+1)
- assert result.run_requests[0].tags["reconciliation_team_ids_range"] == "0-2"
- assert result.run_requests[1].tags["reconciliation_team_ids_range"] == "3-5"
- assert result.run_requests[2].tags["reconciliation_team_ids_range"] == "6-8"
- assert result.run_requests[3].tags["reconciliation_team_ids_range"] == "9-9"
-
- # Verify first run config has correct team_ids chunk
- first_run_config = result.run_requests[0].run_config
- assert first_run_config["ops"]["get_team_ids_to_reconcile"]["config"]["team_ids"] == [1, 2, 3]
-
- def test_sensor_team_ids_mode_updates_cursor(self):
- """Test that sensor updates next_team_index in cursor."""
- from dagster import DagsterInstance, SensorResult, build_sensor_context
-
- from posthog.dags.person_property_reconciliation import person_property_reconciliation_scheduler
-
- cursor = json.dumps(
- {
- "team_ids": [100, 200, 300, 400, 500],
- "chunk_size": 2,
- "max_concurrent_jobs": 2, # Limit to 2 runs
- "next_team_index": 0,
- "bug_window_start": "2024-01-06 20:01:00",
- "bug_window_end": "2024-01-07 14:52:00",
- }
- )
-
- mock_instance = MagicMock(spec=DagsterInstance)
- mock_instance.get_run_records.return_value = []
-
- context = build_sensor_context(cursor=cursor, instance=mock_instance)
- result = person_property_reconciliation_scheduler(context)
-
- assert isinstance(result, SensorResult)
- assert result.run_requests is not None
- assert len(result.run_requests) == 2 # Limited by max_concurrent_jobs
-
- # Cursor should track progress
- assert result.cursor is not None
- new_cursor = json.loads(result.cursor)
- assert new_cursor["next_team_index"] == 4 # Processed 2 chunks of 2 = 4 teams
-
- def test_sensor_team_ids_completes_when_list_exhausted(self):
- """Test that sensor returns SkipReason when all team_ids are processed."""
- from dagster import DagsterInstance, SkipReason, build_sensor_context
-
- from posthog.dags.person_property_reconciliation import person_property_reconciliation_scheduler
-
- cursor = json.dumps(
- {
- "team_ids": [1, 2, 3],
- "chunk_size": 10,
- "next_team_index": 3, # Already processed all 3
- "bug_window_start": "2024-01-06 20:01:00",
- "bug_window_end": "2024-01-07 14:52:00",
- }
- )
-
- mock_instance = MagicMock(spec=DagsterInstance)
- mock_instance.get_run_records.return_value = []
-
- context = build_sensor_context(cursor=cursor, instance=mock_instance)
- result = person_property_reconciliation_scheduler(context)
-
- assert isinstance(result, SkipReason)
- assert result.skip_message is not None
- assert "complete" in result.skip_message.lower()
- assert "3" in result.skip_message # Should mention number of teams
-
- def test_sensor_validates_either_team_ids_or_range_required(self):
- """Test that sensor rejects cursor with neither team_ids nor range."""
- from dagster import SkipReason, build_sensor_context
-
- from posthog.dags.person_property_reconciliation import person_property_reconciliation_scheduler
-
- cursor = json.dumps(
- {
- "chunk_size": 100,
- "bug_window_start": "2024-01-06 20:01:00",
- "bug_window_end": "2024-01-07 14:52:00",
- }
- )
- context = build_sensor_context(cursor=cursor)
- result = person_property_reconciliation_scheduler(context)
-
- assert isinstance(result, SkipReason)
- assert result.skip_message is not None
- assert "team_ids" in result.skip_message
- assert "range_start" in result.skip_message
-
- def test_sensor_validates_team_ids_and_range_mutually_exclusive(self):
- """Test that sensor rejects cursor with both team_ids and range."""
- from dagster import SkipReason, build_sensor_context
-
- from posthog.dags.person_property_reconciliation import person_property_reconciliation_scheduler
-
- cursor = json.dumps(
- {
- "team_ids": [1, 2, 3],
- "range_start": 1,
- "range_end": 100,
- "bug_window_start": "2024-01-06 20:01:00",
- "bug_window_end": "2024-01-07 14:52:00",
- }
- )
- context = build_sensor_context(cursor=cursor)
- result = person_property_reconciliation_scheduler(context)
-
- assert isinstance(result, SkipReason)
- assert result.skip_message is not None
- assert "both" in result.skip_message.lower()
-
- def test_sensor_team_ids_run_request_has_correct_tags(self):
- """Test that team_ids mode run requests include expected tags."""
- from dagster import DagsterInstance, SensorResult, build_sensor_context
-
- from posthog.dags.person_property_reconciliation import person_property_reconciliation_scheduler
-
- cursor = json.dumps(
- {
- "team_ids": [10, 20, 30],
- "chunk_size": 2,
- "max_concurrent_jobs": 5,
- "next_team_index": 0,
- "bug_window_start": "2024-01-06 20:01:00",
- "bug_window_end": "2024-01-07 14:52:00",
- }
- )
-
- mock_instance = MagicMock(spec=DagsterInstance)
- mock_instance.get_run_records.return_value = []
-
- context = build_sensor_context(cursor=cursor, instance=mock_instance)
- result = person_property_reconciliation_scheduler(context)
-
- assert isinstance(result, SensorResult)
- assert result.run_requests is not None
- assert len(result.run_requests) == 2
- assert result.run_requests[0].tags["reconciliation_team_ids_range"] == "0-1"
- assert result.run_requests[0].tags["reconciliation_team_count"] == "2"
- assert result.run_requests[0].tags["owner"] == "team-ingestion"
diff --git a/posthog/management/commands/sync_persons_to_clickhouse.py b/posthog/management/commands/sync_persons_to_clickhouse.py
index 422d271d9186..4afad1678ffe 100644
--- a/posthog/management/commands/sync_persons_to_clickhouse.py
+++ b/posthog/management/commands/sync_persons_to_clickhouse.py
@@ -1,8 +1,9 @@
"""Reconcile a team's person, distinct-id, and group rows from Postgres to ClickHouse.
-LOCAL/DEV USE ONLY for now: this loads the whole team's rows into memory (no batching), so it
-is not safe to run against large production teams. It exists to seed local ClickHouse from the
-local persons database (e.g. via the generate_persons command).
+LOCAL DEV AND SELF-HOSTED ONLY: this loads the whole team's rows into memory (no batching), so it
+is not safe to run against large production teams, and it refuses to run on deployed PostHog Cloud.
+It exists to seed local ClickHouse from the local persons database (e.g. via the generate_persons
+command). On PostHog Cloud, `person_consistency` scans a team and repairs what it finds.
"""
import json
@@ -10,7 +11,7 @@ import logging
from typing import Literal, cast
from uuid import UUID
-from django.core.management.base import BaseCommand
+from django.core.management.base import BaseCommand, CommandError
import structlog
@@ -29,6 +30,7 @@ from posthog.models.person.util import (
set_distinct_id_version_floor,
)
from posthog.persons_db import persons_db_connection
+from posthog.run_mode import run_mode
logger = structlog.get_logger(__name__)
logger.setLevel(logging.INFO)
@@ -66,6 +68,11 @@ class Command(BaseCommand):
def run(options):
+ if run_mode().is_deployed_cloud:
+ raise CommandError(
+ "sync_persons_to_clickhouse runs only in local dev and self-hosted deployments. "
+ "On PostHog Cloud, use person_consistency to scan and repair a team."
+ )
live_run = options["live_run"]
deletes = options["deletes"]
@@ -110,7 +117,7 @@ def run_person_sync(team_id: int, live_run: bool, deletes: bool, force: bool = F
]
rows = sync_execute(
"""
- SELECT id, max(version) FROM person WHERE team_id = %(team_id)s GROUP BY id HAVING max(is_deleted) = 0
+ SELECT id, max(version) FROM person WHERE team_id = %(team_id)s GROUP BY id HAVING argMax(is_deleted, version) = 0
""",
{
"team_id": team_id,
@@ -297,7 +304,7 @@ def run_distinct_id_sync(team_id: int, live_run: bool, deletes: bool):
]
rows = sync_execute(
"""
- SELECT distinct_id, max(version) FROM person_distinct_id2 WHERE team_id = %(team_id)s GROUP BY distinct_id HAVING max(is_deleted) = 0
+ SELECT distinct_id, max(version) FROM person_distinct_id2 WHERE team_id = %(team_id)s GROUP BY distinct_id HAVING argMax(is_deleted, version) = 0
""",
{
"team_id": team_id,
diff --git a/posthog/management/commands/test/test_sync_persons_to_clickhouse.py b/posthog/management/commands/test/test_sync_persons_to_clickhouse.py
index da98f67fef9b..25d14b0e4d4e 100644
--- a/posthog/management/commands/test/test_sync_persons_to_clickhouse.py
+++ b/posthog/management/commands/test/test_sync_persons_to_clickhouse.py
@@ -6,6 +6,9 @@ import pytest
from posthog.test.base import ClickhouseTestMixin, NonAtomicBaseTest
from unittest import mock
+from django.core.management import call_command
+from django.core.management.base import CommandError
+
from parameterized import parameterized
from psycopg.types.json import Jsonb
@@ -213,14 +216,19 @@ class TestSyncPersonsToClickHouse(NonAtomicBaseTest, ClickhouseTestMixin):
@parameterized.expand(
[
- # (name, ClickHouse live version, expected row after the sync)
- ("stored_version_wins", 5, ('{"abc": 123}', 9, True)),
- ("clickhouse_ahead_is_raised_above", 9, ('{"abc": 123}', 10, True)),
+ # (name, ClickHouse live version, an older ClickHouse tombstone under it, expected row after the sync)
+ ("stored_version_wins", 5, False, ('{"abc": 123}', 9, True)),
+ ("clickhouse_ahead_is_raised_above", 9, False, ('{"abc": 123}', 10, True)),
+ ("live_above_an_older_tombstone", 5, True, ('{"abc": 123}', 9, True)),
]
)
- def test_persons_tombstoned_in_postgres_publish_the_stored_version(self, _name, ch_version, expected):
+ def test_persons_tombstoned_in_postgres_publish_the_stored_version(
+ self, _name, ch_version, older_tombstone, expected
+ ):
person_uuid = uuid4()
self._seed_tombstoned_person(person_uuid, 9)
+ if older_tombstone:
+ create_person(uuid=str(person_uuid), team_id=self.team.pk, version=ch_version - 1, is_deleted=True)
create_person(uuid=str(person_uuid), team_id=self.team.pk, version=ch_version, properties={"abc": 123})
with fake_personhog_client() as personhog:
@@ -315,13 +323,17 @@ class TestSyncPersonsToClickHouse(NonAtomicBaseTest, ClickhouseTestMixin):
@parameterized.expand(
[
- # (name, ClickHouse live version, mapping revives during the raise, expected version and is_deleted)
- ("stored_version_wins", 3, False, (7, True)),
- ("clickhouse_ahead_is_raised_above", 7, False, (8, True)),
- ("revived_during_the_raise_is_published_live", 7, True, (8, False)),
+ # (name, ClickHouse live version, mapping revives during the raise, an older ClickHouse tombstone
+ # under the live row, expected version and is_deleted)
+ ("stored_version_wins", 3, False, False, (7, True)),
+ ("clickhouse_ahead_is_raised_above", 7, False, False, (8, True)),
+ ("revived_during_the_raise_is_published_live", 7, True, False, (8, False)),
+ ("live_above_an_older_tombstone", 3, False, True, (7, True)),
]
)
- def test_distinct_ids_tombstoned_in_postgres_publish_the_primary_row(self, _name, ch_version, revives, expected):
+ def test_distinct_ids_tombstoned_in_postgres_publish_the_primary_row(
+ self, _name, ch_version, revives, older_tombstone, expected
+ ):
person_uuid = uuid4()
with persons_db_connection(writer=True, autocommit=True) as conn:
person_id = insert_seed_person(conn, team_id=self.team.pk, properties={}, version=0, uuid=person_uuid)
@@ -331,6 +343,14 @@ class TestSyncPersonsToClickHouse(NonAtomicBaseTest, ClickhouseTestMixin):
"UPDATE posthog_persondistinctid SET is_deleted = true WHERE team_id = %s AND distinct_id = %s",
[self.team.pk, "test-id"],
)
+ if older_tombstone:
+ create_person_distinct_id(
+ team_id=self.team.pk,
+ distinct_id="test-id",
+ person_id=str(person_uuid),
+ is_deleted=True,
+ version=ch_version - 1,
+ )
create_person_distinct_id(
team_id=self.team.pk,
distinct_id="test-id",
@@ -360,6 +380,16 @@ class TestSyncPersonsToClickHouse(NonAtomicBaseTest, ClickhouseTestMixin):
)
self.assertEqual(cursor.fetchall(), [(version, is_deleted)])
+ @parameterized.expand([("US",), ("EU",), ("DEV",)])
+ def test_refuses_to_run_on_deployed_cloud(self, cloud_deployment):
+ with persons_db_connection(writer=True, autocommit=True) as conn:
+ insert_seed_person(conn, team_id=self.team.pk, properties={}, version=0)
+
+ with mock.patch("posthog.settings.CLOUD_DEPLOYMENT", cloud_deployment), self.assertRaises(CommandError):
+ call_command("sync_persons_to_clickhouse", team_id=self.team.pk, person=True, live_run=True)
+
+ self.assertEqual(sync_execute("SELECT count() FROM person WHERE team_id = %(t)s", {"t": self.team.pk}), [(0,)])
+
@mock.patch(
f"{posthog.management.commands.sync_persons_to_clickhouse.__name__}.raw_create_group_ch",
wraps=posthog.management.commands.sync_persons_to_clickhouse.raw_create_group_ch,
diff --git a/posthog/models/person/divergence/inspection.py b/posthog/models/person/divergence/inspection.py
index 95869b26a07a..83d3fb075e96 100644
--- a/posthog/models/person/divergence/inspection.py
+++ b/posthog/models/person/divergence/inspection.py
@@ -13,7 +13,7 @@ from django.db.models import Q
from posthog.dataclasses import frozen
from posthog.models.activity_logging.activity_log import ActivityLog
-from posthog.models.person.divergence import lookups, observe
+from posthog.models.person.divergence import cleanup_queue, lookups, observe
from posthog.models.person.divergence.flavors import classify_mapping, classify_person
from posthog.models.person.util import list_person_tombstone_queue
from posthog.models.team import Team
@@ -37,6 +37,8 @@ class Inspection:
ch_history: list[dict[str, Any]] | None = None
activity: list[dict[str, Any]] | None = None
tombstone_queue: dict[str, Any] | None = None
+ # The persons cleanup queue row: a `blocked_at` means the drain holds the person for a live distinct id.
+ cleanup_queue: dict[str, Any] | None = None
def hash_distinct_id(team_id: int, distinct_id: str) -> str:
@@ -151,6 +153,7 @@ def inspect_persons(
pg = observe.pg_person_states(team_id, uuids) if team_exists else {}
ch = observe.ch_person_states(team_id, uuids)
history = _person_history(team_id, uuids)
+ queued = cleanup_queue.queue_entries(team_id, uuids) if team_exists else {}
results = []
for person_uuid in uuids:
pg_state = pg.get(person_uuid)
@@ -174,6 +177,7 @@ def inspect_persons(
ch_history=history[person_uuid],
activity=_activity(team_id, person_uuid, pg_state.id if pg_state else None),
tombstone_queue=_tombstone_queue_entry(team_id, person_uuid) if team_exists else None,
+ cleanup_queue=queued.get(person_uuid),
)
)
return results
diff --git a/posthog/models/person/test/test_divergence_inspection.py b/posthog/models/person/test/test_divergence_inspection.py
index 2159bf21e5d7..768c2af3ec05 100644
--- a/posthog/models/person/test/test_divergence_inspection.py
+++ b/posthog/models/person/test/test_divergence_inspection.py
@@ -7,10 +7,12 @@ from posthog.test.base import BaseTest, ClickhouseTestMixin
from django.core.management import call_command
from posthog.models.activity_logging.activity_log import ActivityLog, Detail
+from posthog.models.person.divergence.cleanup_queue import PG_CLEANUP_QUEUE_TABLE
from posthog.models.person.divergence.inspection import hash_distinct_id, inspect_mappings, inspect_persons
from posthog.models.person.test.divergence_fixtures import DivergenceFixturesMixin
from posthog.models.person.util import tombstone_persons_in_postgres
from posthog.personhog_client.fake_client import get_active_fake
+from posthog.persons_db import persons_db_connection
class TestDivergenceInspection(DivergenceFixturesMixin, ClickhouseTestMixin, BaseTest):
@@ -43,6 +45,12 @@ class TestDivergenceInspection(DivergenceFixturesMixin, ClickhouseTestMixin, Bas
self._ch_person(ghost, 3, {"email": "[email protected]"})
self._log_person_activity("deleted", item_id=deleted.pk, name=str(deleted.uuid))
+ with persons_db_connection(writer=True) as conn, conn.cursor() as cursor:
+ cursor.execute(
+ f"INSERT INTO {PG_CLEANUP_QUEUE_TABLE} (team_id, person_uuid, deleted_at, max_version, blocked_at) "
+ "VALUES (%s, %s, now(), 3, now())",
+ [self.team.pk, str(deleted.uuid)],
+ )
self._log_person_activity("updated", item_id=stale.pk, name=None)
self._log_person_activity("updated", item_id=tie.pk, name=None)
@@ -62,6 +70,9 @@ class TestDivergenceInspection(DivergenceFixturesMixin, ClickhouseTestMixin, Bas
queued = by_key[str(deleted.uuid)].tombstone_queue
assert queued is not None and queued["person_version"] == 3 and queued["tombstoned_at_ms"] > 0
assert by_key[str(stale.uuid)].tombstone_queue is None
+ drain_row = by_key[str(deleted.uuid)].cleanup_queue
+ assert drain_row is not None and drain_row["max_version"] == 3 and drain_row["blocked_at"] is not None
+ assert by_key[str(stale.uuid)].cleanup_queue is None
assert flavors == {
str(tie.uuid): ("tie", None, "republish"),
str(blanked.uuid): ("blanked", None, "republish"),
diff --git a/posthog/persons_seed.py b/posthog/persons_seed.py
index a25f70f09663..a73105baf6d4 100644
--- a/posthog/persons_seed.py
+++ b/posthog/persons_seed.py
@@ -26,11 +26,21 @@ from psycopg.types.json import Jsonb
from posthog.models.utils import UUIDT
from posthog.persons_db import persons_db_connection
+from posthog.run_mode import run_mode
# Not configurable like the person table; the model uses Django's default db_table.
PERSON_DISTINCT_ID_TABLE = "posthog_persondistinctid"
+def _refuse_in_deployed_cloud() -> None:
+ # A seed command pointed at a real team on PostHog Cloud would write persons outside ingestion and personhog,
+ # with no version bump and no tombstone check, so every write here refuses there.
+ if run_mode().is_deployed_cloud:
+ raise RuntimeError(
+ "persons_seed writes only in local dev, tests and self-hosted deployments, not on deployed PostHog Cloud"
+ )
+
+
def insert_seed_person(
conn: psycopg.Connection[Any],
*,
@@ -52,6 +62,7 @@ def insert_seed_person(
``last_seen_at`` and the ``properties_last_*`` maps default to NULL, mirroring
``Person.objects.create`` with no override.
"""
+ _refuse_in_deployed_cloud()
with conn.cursor() as cursor:
cursor.execute(
f"INSERT INTO {settings.PERSON_TABLE_NAME} "
@@ -88,6 +99,7 @@ def insert_seed_distinct_id(
``version`` defaults to 0 but accepts ``None`` to write a NULL version (the column is
nullable), mirroring ``PersonDistinctId.objects.create(version=None)``.
"""
+ _refuse_in_deployed_cloud()
with conn.cursor() as cursor:
cursor.execute(
f"INSERT INTO {PERSON_DISTINCT_ID_TABLE} (distinct_id, person_id, team_id, version) VALUES (%s, %s, %s, %s)",
@@ -106,6 +118,7 @@ def insert_seed_group(
created_at: datetime | None = None,
) -> int:
"""Insert one group row and return its id. ``created_at`` defaults to ``now()``; ``version`` is NOT NULL."""
+ _refuse_in_deployed_cloud()
with conn.cursor() as cursor:
cursor.execute(
"INSERT INTO posthog_group "
@@ -136,6 +149,7 @@ def insert_seed_group_type_mapping(
``created_at`` is written as given (``None`` stays NULL — the model's custom ``save()`` stamps
created_at only on insert, which this bypasses); callers pass ``now()`` for the normal case.
"""
+ _refuse_in_deployed_cloud()
with conn.cursor() as cursor:
cursor.execute(
"INSERT INTO posthog_grouptypemapping "
@@ -168,6 +182,7 @@ def update_seed_person(
is_identified: bool,
) -> None:
"""Overwrite ``properties`` and ``is_identified`` for an existing person."""
+ _refuse_in_deployed_cloud()
with conn.cursor() as cursor:
cursor.execute(
f"UPDATE {settings.PERSON_TABLE_NAME} "
diff --git a/posthog/test/repo_invariants/dataclass_frozen_baseline.txt b/posthog/test/repo_invariants/dataclass_frozen_baseline.txt
index a735353c9354..d3b9e160dc55 100644
--- a/posthog/test/repo_invariants/dataclass_frozen_baseline.txt
+++ b/posthog/test/repo_invariants/dataclass_frozen_baseline.txt
@@ -413,7 +413,6 @@
2 posthog/dags/data_deletion_requests.py
2 posthog/dags/part_breaker.py
2 posthog/dags/person_overrides.py
-2 posthog/dags/scripts/extract_reconciliation_results.py
2 posthog/helpers/email_utils.py
1 posthog/hogql/context.py
2 posthog/hogql/database/models.py
@@ -593,7 +592,6 @@
4 products/warehouse_sources/backend/temporal/data_imports/sources/common/base.py
5 posthog/api/proxy_record_diagnostics.py
5 posthog/api/services/llm_prompt.py
-5 posthog/dags/person_property_reconciliation.py
5 posthog/temporal/ai_observability/evaluation_clustering/models.py
5 posthog/temporal/ai_observability/run_trace_evaluation.py
5 posthog/temporal/proxy_service/create.py
diff --git a/posthog/test/test_persons_seed.py b/posthog/test/test_persons_seed.py
index 6afecf197129..cd87b7a1afcb 100644
--- a/posthog/test/test_persons_seed.py
+++ b/posthog/test/test_persons_seed.py
@@ -1,13 +1,20 @@
from typing import Any
import pytest
+from unittest.mock import MagicMock, patch
from django.conf import settings
import psycopg
from posthog.persons_db import persons_db_connection
-from posthog.persons_seed import insert_seed_distinct_id, insert_seed_person, update_seed_person
+from posthog.persons_seed import (
+ insert_seed_distinct_id,
+ insert_seed_group,
+ insert_seed_group_type_mapping,
+ insert_seed_person,
+ update_seed_person,
+)
# Sentinel team id used only by these tests. The seed helpers write through a raw
# psycopg connection that commits independently of Django's test transaction, so each
@@ -24,6 +31,36 @@ def _cleanup(conn: psycopg.Connection[Any]) -> None:
class TestPersonsSeedHelpers:
+ @pytest.mark.parametrize(
+ "write",
+ [
+ lambda conn: insert_seed_person(conn, team_id=SEED_TEST_TEAM_ID, properties={}),
+ lambda conn: insert_seed_distinct_id(conn, team_id=SEED_TEST_TEAM_ID, person_id=1, distinct_id="d"),
+ lambda conn: insert_seed_group(
+ conn, team_id=SEED_TEST_TEAM_ID, group_key="g", group_type_index=0, group_properties={}
+ ),
+ lambda conn: insert_seed_group_type_mapping(
+ conn, project_id=SEED_TEST_TEAM_ID, team_id=SEED_TEST_TEAM_ID, group_type="org", group_type_index=0
+ ),
+ lambda conn: update_seed_person(
+ conn,
+ team_id=SEED_TEST_TEAM_ID,
+ uuid="00000000-0000-0000-0000-000000000001",
+ properties={},
+ is_identified=False,
+ ),
+ ],
+ ids=["insert_person", "insert_distinct_id", "insert_group", "insert_group_type_mapping", "update_person"],
+ )
+ def test_writes_refuse_on_deployed_cloud(self, write):
+ conn = MagicMock()
+ with (
+ patch("posthog.settings.CLOUD_DEPLOYMENT", "US"),
+ pytest.raises(RuntimeError, match="deployed PostHog Cloud"),
+ ):
+ write(conn)
+ conn.cursor.assert_not_called()
+
def test_insert_person_persists_fields_and_sets_required_columns(self):
with persons_db_connection(writer=True) as conn:
try:
diff --git a/products/mcp_analytics/backend/management/commands/seed_mcp_sessions.py b/products/mcp_analytics/backend/management/commands/seed_mcp_sessions.py
index 3e8e5f648d6e..512b96388c71 100644
--- a/products/mcp_analytics/backend/management/commands/seed_mcp_sessions.py
+++ b/products/mcp_analytics/backend/management/commands/seed_mcp_sessions.py
@@ -19,6 +19,7 @@ from posthog.models.utils import UUIDT, uuid7
from posthog.personhog_client.caller_tag import personhog_caller_tag
from posthog.persons_db import persons_db_connection
from posthog.persons_seed import insert_seed_distinct_id, insert_seed_person, update_seed_person
+from posthog.run_mode import run_mode
from products.mcp_analytics.backend.models import MCPSession
@@ -1257,6 +1258,10 @@ class Command(BaseCommand):
)
def handle(self, *args: Any, **options: Any) -> None:
+ if run_mode().is_deployed_cloud:
+ raise CommandError(
+ "seed_mcp_sessions writes fabricated events and persons; it does not run on PostHog Cloud."
+ )
team_id: int = options["team_id"]
session_count: int = options["sessions"]
min_calls: int = options["min_calls"]