Skip to content

:hedgehog: PostHog is the leading platform for building self-driving products. Our developer tools – AI observability, analytics, session replay, flags, experiments, error tracking, logs, and more – capture all the context agents need to diagnose problems, uncover opportunities, and ship fixes. Steer it all from Slack, web, desktop, or the MCP.

feat(signals): add activity summaries and embeddings assets (#120034)

posthog[bot]6c0f31fc16bc
diff --git a/.github/workflows/ci-dagster.yml b/.github/workflows/ci-dagster.yml
index f2abc71be01a..7040065af017 100644
--- a/.github/workflows/ci-dagster.yml
+++ b/.github/workflows/ci-dagster.yml
@@ -178,6 +178,8 @@ jobs:
- 'posthog/git.py'
- 'posthog/job_owners.py'
- 'posthog/llm/**'
+ - 'posthog/api/embedding_worker.py'
+ - 'posthog/security/outbound_proxy.py'
- 'posthog/person_db_router.py'
- 'posthog/persons_db.py'
- 'posthog/personhog_client/**'
diff --git a/posthog/dags/locations/signals.py b/posthog/dags/locations/signals.py
index 4aa2ed73fa11..63adf44281b1 100644
--- a/posthog/dags/locations/signals.py
+++ b/posthog/dags/locations/signals.py
@@ -33,6 +33,8 @@ if is_inbox_ranking_registered():
inbox_ranking_training.inbox_ranking_unseen_graded,
activity_summary.activity_summary_inputs,
activity_summary.activity_summary_cost_projection,
+ activity_summary.activity_summaries,
+ activity_summary.activity_summary_embeddings,
],
jobs=[
inbox_ranking_dataset.inbox_ranking_dataset_job,
diff --git a/products/signals/dags/inbox_ranking/README.md b/products/signals/dags/inbox_ranking/README.md
index d24dfb750d9b..a41c7b39748d 100644
--- a/products/signals/dags/inbox_ranking/README.md
+++ b/products/signals/dags/inbox_ranking/README.md
@@ -18,7 +18,7 @@ inbox_ranking/
│ └── promotion.py # the candidate grading rule
├── multihead/ # the multi-head network, its ONNX export, and the offline prototype
├── nn/ # the `nn` family: examples → candidate → champion assets, on the `signals_ml` location
-├── activity_summary/ # per-person activity summary inputs and their cost projection (no LLM call)
+├── activity_summary/ # per-person activity summaries: inputs, cost projection, LLM summaries and their vectors
└── tests/
```
@@ -376,8 +376,8 @@ dagster job launch -w .dagster_home/workspace.yaml --location posthog.dags.locat
## The activity summary dag
-`activity_summary/` prepares per-person activity summaries for inbox ranking, without calling an LLM.
-`activity_summary_job` runs two assets on the shared daily partition, over the 28 days that end with D.
+`activity_summary/` builds per-person activity summaries and their vectors for inbox ranking.
+`activity_summary_job` runs four assets on the shared daily partition, over the 28 days that end with D.
Its weekly schedule ships stopped, so run the job by hand.
- `activity_summary_inputs/v1/dt=D/` holds one row per (project, member) for active members of the dogfood organization with an `@posthog.com` email, in that organization's own projects only.
@@ -390,6 +390,16 @@ Its weekly schedule ships stopped, so run the job by hand.
It has one row per `MIN_UNITS` and `MIN_DAYS` option, with eligible pairs, input token percentiles and monthly volume.
The task and activity sections go in at their capped maximum (`capped_sections_tokens`), so the projection errs high.
Set `input_price_per_million_tokens` and `output_price_per_million_tokens` in the run config to fill `projected_monthly_cost_usd`.
+- `activity_summaries/v1/dt=D/` holds one LLM summary per inputs row, made through `build_openai_client` with the prompt in `prompt.py`.
+ The model gets the rendered input and the pair's previous summary: its successful row in the newest earlier partition, within `PREVIOUS_SUMMARY_MAX_AGE_DAYS`.
+ A malformed answer is retried once, then the row is written with `status = failed`. When every row fails, the asset fails and leaves the partition as it was.
+ `summary_text` is the text that gets embedded, so bump `PROMPT_VERSION` with any change to the prompt, the normalization or `summary_text`.
+ Run config: `model`, `use_previous_summary` (off compares the stateless summaries), the two token prices for `cost_usd`, and `max_concurrency`.
+ Each row also captures an `activity summary generated` event into the dogfood project, with counts only unless `capture_summary_text` is set.
+- `activity_summary_embeddings/v1/dt=D/` holds the `text-embedding-3-small` vector of each successful summary.
+ The asset sends a summary to the embedding worker only when `document_embeddings` has no vector of its current text, waits up to `wait_seconds`, and copies out what landed.
+ A vector that lands later is picked up by the next materialization; `missing_vectors` in the metadata counts them.
+ The source table drops a row three months after the partition day, so this Parquet is the durable copy, and a re-run that would write fewer rows is refused.
Most activity recorded under a person is not theirs: scouts, agents and scheduled jobs act with the person's identity.
Every filter in `render.py` is a named rule in `Drop`, and each inputs row records in `drops` how many units or calls each rule removed.
diff --git a/products/signals/dags/inbox_ranking/activity_summary/dag.py b/products/signals/dags/inbox_ranking/activity_summary/dag.py
index f5b8d6bf5ce4..4c2c9e87430d 100644
--- a/products/signals/dags/inbox_ranking/activity_summary/dag.py
+++ b/products/signals/dags/inbox_ranking/activity_summary/dag.py
@@ -1,9 +1,11 @@
-"""Inputs and a cost projection for per-person activity summaries, with no LLM call.
+"""Per-person activity summaries: their inputs, a cost projection, the summaries and their vectors.
-Two assets on the shared daily partition, each writing Parquet under the dataset prefix:
+Four assets on the shared daily partition, each writing Parquet under the dataset prefix:
activity_summary_inputs/v1/dt=D/ one filtered, rendered input per (project, member)
activity_summary_cost_projection/v1/dt=D/ eligible pairs, token percentiles and monthly volume
+ activity_summaries/v1/dt=D/ one LLM summary per inputs row
+ activity_summary_embeddings/v1/dt=D/ one vector per summary, kept past the source table's TTL
The inputs asset covers only members of the dogfood project's organization with an `@posthog.com`
email, in that organization's own projects, so no customer project's activity gets in. It reads
@@ -13,6 +15,7 @@ activity log. The projection covers every MCP pair in the events and writes coun
"""
import json
+import time
import datetime
from collections import defaultdict
from collections.abc import Collection
@@ -30,6 +33,18 @@ from posthog.dataclasses import frozen
from posthog.models import ActivityLog, OrganizationMembership, Team, User
from products.signals.dags.inbox_ranking.activity_summary import queries
+from products.signals.dags.inbox_ranking.activity_summary.embeddings import (
+ EMBEDDING_MODEL,
+ LandedVector,
+ SummaryDocument,
+ document_id,
+ emit_summary_embedding,
+ flush_embedding_requests,
+ landed_vectors,
+ pending_documents,
+ within_source_ttl,
+)
+from products.signals.dags.inbox_ranking.activity_summary.prompt import PROMPT_VERSION
from products.signals.dags.inbox_ranking.activity_summary.render import (
MAX_ACTIVITY_ROWS_PER_PERSON,
MAX_TASKS_PER_PERSON,
@@ -49,14 +64,30 @@ from products.signals.dags.inbox_ranking.activity_summary.render import (
projection_rows,
sampling_seed,
)
+from products.signals.dags.inbox_ranking.activity_summary.summaries import (
+ DEFAULT_MODEL,
+ PIPELINE_DISTINCT_ID,
+ STATUS_OK,
+ SummaryPrices,
+ SummaryRequest,
+ gateway_client,
+ summarize,
+ summarize_all,
+ summary_events,
+ summary_row,
+)
from products.signals.dags.inbox_ranking.common import (
+ DATASET_VERSION,
+ PARQUET_PART_NAME,
S3_BUCKET_ENV,
dataset_bucket,
dataset_unconfigured,
ensure_utc,
+ object_row_count,
owner_tags,
partition_def,
partition_object_key,
+ partition_write_allowed,
read_parquet_if_exists,
s3_client,
skip_unconfigured,
@@ -64,10 +95,17 @@ from products.signals.dags.inbox_ranking.common import (
write_parquet,
)
from products.signals.dags.inbox_ranking.dataset.queries import LABELS_TEAM_ID, labels_team
+from products.signals.dags.inbox_ranking.training.telemetry import capture_training_events
from products.tasks.backend.facade.user_task_summaries import list_user_task_summaries
INPUTS_TABLE = "activity_summary_inputs"
COST_PROJECTION_TABLE = "activity_summary_cost_projection"
+SUMMARIES_TABLE = "activity_summaries"
+SUMMARY_EMBEDDINGS_TABLE = "activity_summary_embeddings"
+
+# The newest earlier summary of a pair counts as its previous summary only within this many days. An
+# older one describes work that has gone quiet, so the pair starts over.
+PREVIOUS_SUMMARY_MAX_AGE_DAYS = 60
COHORT_EMAIL_DOMAIN = "@posthog.com"
# Tasks a person started: in PostHog AI, in Slack, or by hand in the web or desktop app. Scouts,
@@ -114,6 +152,33 @@ _COST_PROJECTION_FIELDS: list[tuple[str, pa.DataType]] = [
]
COST_PROJECTION_SCHEMA = pa.schema(_COST_PROJECTION_FIELDS)
+_SUMMARIES_FIELDS: list[tuple[str, pa.DataType]] = [
+ ("team_id", pa.int64()),
+ ("user_id", pa.int64()),
+ # The normalized answer: summary, focus_areas, products and recent_shift. Null when status is failed.
+ ("summary_json", pa.string()),
+ ("summary_text", pa.string()),
+ ("used_previous_summary", pa.bool_()),
+ ("model", pa.string()),
+ ("prompt_version", pa.int32()),
+ ("tokens_in", pa.int64()),
+ ("tokens_out", pa.int64()),
+ ("latency_ms", pa.int64()),
+ ("cost_usd", pa.float64()),
+ ("status", pa.string()),
+ ("error", pa.string()),
+]
+SUMMARIES_SCHEMA = pa.schema(_SUMMARIES_FIELDS)
+
+_SUMMARY_EMBEDDINGS_FIELDS: list[tuple[str, pa.DataType]] = [
+ ("team_id", pa.int64()),
+ ("user_id", pa.int64()),
+ ("prompt_version", pa.int32()),
+ ("embedding", pa.list_(pa.float32())),
+ ("embedding_model", pa.string()),
+]
+SUMMARY_EMBEDDINGS_SCHEMA = pa.schema(_SUMMARY_EMBEDDINGS_FIELDS)
+
ASSET_KWARGS: dict[str, Any] = {
"group_name": "activity_summary",
"partitions_def": partition_def,
@@ -137,6 +202,28 @@ class CostProjectionConfig(dagster.Config):
output_tokens_per_update: int = 500
+class SummariesConfig(dagster.Config):
+ """Prices are per million tokens and stay unset by default, like the cost projection's, so
+ `cost_usd` stays null until a run fills in the current price of `model`."""
+
+ model: str = DEFAULT_MODEL
+ # Off to compare the stateful and the stateless summaries on the same inputs.
+ use_previous_summary: bool = True
+ # Puts the summary text and focus topics on the events. Off by default, because an event cannot be
+ # edited or deleted the way a partition can.
+ capture_summary_text: bool = False
+ input_price_per_million_tokens: float | None = None
+ output_price_per_million_tokens: float | None = None
+ max_concurrency: int = 8
+
+
+class SummaryEmbeddingsConfig(dagster.Config):
+ # How long the asset waits for the embedding worker to write the vectors it asked for. A vector
+ # that lands later is picked up by the next materialization of the partition.
+ wait_seconds: int = 600
+ poll_interval_seconds: int = 30
+
+
@frozen
class Window:
start: datetime.datetime
@@ -462,9 +549,288 @@ def activity_summary_cost_projection(context: dagster.AssetExecutionContext, con
)
+def summary_partitions_before(client, bucket: str, prefix: str, partition_key: str) -> list[str]:
+ """The written summaries partitions in the `PREVIOUS_SUMMARY_MAX_AGE_DAYS` before `partition_key`,
+ newest first."""
+ folder = f"{prefix}/{SUMMARIES_TABLE}/{DATASET_VERSION}/dt="
+ earliest = (
+ datetime.date.fromisoformat(partition_key) - datetime.timedelta(days=PREVIOUS_SUMMARY_MAX_AGE_DAYS)
+ ).isoformat()
+ partitions: set[str] = set()
+ for page in client.get_paginator("list_objects_v2").paginate(Bucket=bucket, Prefix=folder):
+ for item in page.get("Contents", []):
+ partition, _, name = item["Key"].removeprefix(folder).partition("/")
+ if name == PARQUET_PART_NAME and earliest <= partition < partition_key:
+ partitions.add(partition)
+ return sorted(partitions, reverse=True)
+
+
+def usable_summaries(rows: list[dict[str, Any]], wanted: Collection[MemberProject]) -> dict[MemberProject, dict]:
+ """The successful summaries among one partition's rows, for the pairs in `wanted`."""
+ return {
+ member: json.loads(row["summary_json"])
+ for row in rows
+ if row["status"] == STATUS_OK
+ and row["summary_json"]
+ and (member := MemberProject(team_id=row["team_id"], user_id=row["user_id"])) in wanted
+ }
+
+
+def previous_summaries(
+ client, bucket: str, prefix: str, partition_key: str, pairs: Collection[MemberProject]
+) -> dict[MemberProject, dict]:
+ """Each pair's summary from the newest earlier partition that holds one for it. Only earlier
+ partitions are read, so a re-run of a partition sees the same previous summaries."""
+ wanted = set(pairs)
+ found: dict[MemberProject, dict] = {}
+ for partition in summary_partitions_before(client, bucket, prefix, partition_key):
+ if not wanted:
+ break
+ table = read_parquet_if_exists(
+ client,
+ bucket,
+ partition_object_key(prefix, SUMMARIES_TABLE, partition),
+ ["team_id", "user_id", "summary_json", "status"],
+ )
+ if table is None:
+ continue
+ newer = usable_summaries(table.to_pylist(), wanted)
+ found |= newer
+ wanted -= newer.keys()
+ return found
+
+
+def keep_earlier_successes(
+ rows: list[dict[str, Any]], existing: list[dict[str, Any]]
+) -> tuple[list[dict[str, Any]], int]:
+ """A re-run's rows, with each failed row replaced by the pair's successful row from the earlier
+ write of the partition, so a transient failure does not destroy a summary. Returns the rows and
+ how many were kept from the earlier write."""
+ earlier = {
+ MemberProject(team_id=row["team_id"], user_id=row["user_id"]): row
+ for row in existing
+ if row["status"] == STATUS_OK
+ }
+ merged: list[dict[str, Any]] = []
+ kept = 0
+ for row in rows:
+ previous = earlier.get(MemberProject(team_id=row["team_id"], user_id=row["user_id"]))
+ if row["status"] != STATUS_OK and previous is not None:
+ merged.append(previous)
+ kept += 1
+ else:
+ merged.append(row)
+ return merged, kept
+
+
[email protected](name=SUMMARIES_TABLE, deps=[INPUTS_TABLE], **ASSET_KWARGS)
+def activity_summaries(context: dagster.AssetExecutionContext, config: SummariesConfig) -> None:
+ """One LLM summary per `activity_summary_inputs` row, with its tokens, latency and cost.
+
+ A row whose answer stays malformed after one retry, or whose request fails, is written with
+ `status = failed`, unless an earlier write of the partition holds a successful row for that pair.
+ When every row fails, the asset fails and leaves the partition as it was."""
+ if skip_unconfigured(context):
+ return
+ partition_key = context.partition_key
+ client = s3_client()
+ bucket = dataset_bucket()
+ prefix = settings.INBOX_RANKING_DATASET_S3_PREFIX
+ inputs = read_parquet_if_exists(
+ client,
+ bucket,
+ partition_object_key(prefix, INPUTS_TABLE, partition_key),
+ ["team_id", "user_id", "rendered_input", "n_tasks", "n_mcp_sessions", "n_activity_lines", "drops"],
+ )
+ if inputs is None:
+ context.log.warning(f"{INPUTS_TABLE} has no object for dt={partition_key}; nothing to summarize")
+ return
+ input_rows_ = inputs.to_pylist()
+ members = [MemberProject(team_id=row["team_id"], user_id=row["user_id"]) for row in input_rows_]
+ previous = previous_summaries(client, bucket, prefix, partition_key, members) if config.use_previous_summary else {}
+ requests = [
+ SummaryRequest(
+ team_id=member.team_id,
+ user_id=member.user_id,
+ rendered_input=row["rendered_input"],
+ previous_summary=previous.get(member),
+ )
+ for member, row in zip(members, input_rows_, strict=True)
+ ]
+
+ llm = gateway_client(partition_key)
+ as_of = datetime.date.fromisoformat(partition_key)
+ outcomes = summarize_all(
+ requests, lambda request: summarize(llm, request, model=config.model, as_of=as_of), config.max_concurrency
+ )
+ prices = SummaryPrices(
+ input_price_per_million_tokens=config.input_price_per_million_tokens,
+ output_price_per_million_tokens=config.output_price_per_million_tokens,
+ )
+ rows = [
+ summary_row(request, outcome, model=config.model, prices=prices)
+ for request, outcome in zip(requests, outcomes, strict=True)
+ ]
+ failed = sum(1 for row in rows if row["status"] != STATUS_OK)
+ if rows and failed == len(rows):
+ raise dagster.Failure(
+ f"Every summary failed for dt={partition_key}, so the partition was not written",
+ metadata={"rows": len(rows), "first_error": str(rows[0]["error"])},
+ )
+
+ key = partition_object_key(prefix, SUMMARIES_TABLE, partition_key)
+ existing = read_parquet_if_exists(client, bucket, key)
+ written_rows, kept = keep_earlier_successes(rows, [] if existing is None else existing.to_pylist())
+ write_parquet(client, bucket, key, pa.Table.from_pylist(written_rows, schema=SUMMARIES_SCHEMA))
+ capture_training_events(
+ context,
+ partition_key,
+ summary_events(rows, input_rows_, capture_summary_text=config.capture_summary_text),
+ distinct_id=PIPELINE_DISTINCT_ID,
+ )
+ costs = [row["cost_usd"] for row in rows if row["cost_usd"] is not None]
+ context.add_output_metadata(
+ {
+ "rows": dagster.MetadataValue.int(len(rows)),
+ "failed": dagster.MetadataValue.int(failed),
+ "kept_from_earlier_run": dagster.MetadataValue.int(kept),
+ "used_previous_summary": dagster.MetadataValue.int(sum(1 for row in rows if row["used_previous_summary"])),
+ "tokens_in": dagster.MetadataValue.int(sum(row["tokens_in"] for row in rows)),
+ "tokens_out": dagster.MetadataValue.int(sum(row["tokens_out"] for row in rows)),
+ "cost_usd": dagster.MetadataValue.float(sum(costs)) if costs else dagster.MetadataValue.null(),
+ "model": dagster.MetadataValue.text(config.model),
+ "prompt_version": dagster.MetadataValue.int(PROMPT_VERSION),
+ "s3_key": dagster.MetadataValue.text(f"s3://{bucket}/{key}"),
+ }
+ )
+
+
+@frozen
+class EmbeddingTarget:
+ user_id: int
+ prompt_version: int
+ text: str
+
+
+def embedding_targets(rows: list[dict[str, Any]], partition_key: str) -> dict[SummaryDocument, EmbeddingTarget]:
+ """The successful summaries of a partition, keyed by their embedding document."""
+ return {
+ SummaryDocument(
+ team_id=row["team_id"], document_id=document_id(row["user_id"], partition_key)
+ ): EmbeddingTarget(user_id=row["user_id"], prompt_version=row["prompt_version"], text=row["summary_text"])
+ for row in rows
+ if row["status"] == STATUS_OK and row["summary_text"]
+ }
+
+
+def summary_embedding_rows(
+ targets: dict[SummaryDocument, EmbeddingTarget], landed: dict[SummaryDocument, LandedVector]
+) -> list[dict[str, Any]]:
+ """One row per summary whose vector landed and embeds the summary's current text."""
+ rows: list[dict[str, Any]] = []
+ for document, target in sorted(targets.items()):
+ vector = landed.get(document)
+ if vector is None or vector.content != target.text:
+ continue
+ rows.append(
+ {
+ "team_id": document.team_id,
+ "user_id": target.user_id,
+ "prompt_version": target.prompt_version,
+ "embedding": vector.embedding,
+ "embedding_model": EMBEDDING_MODEL.value,
+ }
+ )
+ return rows
+
+
+ name=SUMMARY_EMBEDDINGS_TABLE,
+ # Reads the summaries partition, and also keeps the vectors out of the LLM step's run pod memory.
+ deps=[SUMMARIES_TABLE],
+ **ASSET_KWARGS,
+)
+def activity_summary_embeddings(context: dagster.AssetExecutionContext, config: SummaryEmbeddingsConfig) -> None:
+ """The vector of each successful summary, copied out of `document_embeddings`.
+
+ The source table drops a row three months after its timestamp, which is the partition day, so
+ this Parquet is the history a backtest reads. Only a summary with no vector yet, or a vector of
+ other text, is sent to the worker, so a re-run does not embed the same text twice. A re-run that
+ would write fewer rows than the partition holds is refused: past the source TTL, the vectors it
+ lacks cannot come back."""
+ if skip_unconfigured(context):
+ return
+ partition_key = context.partition_key
+ if not within_source_ttl(partition_key, datetime.datetime.now(datetime.UTC).date()):
+ context.log.warning(
+ f"dt={partition_key} is too old: its vectors would expire from document_embeddings before they "
+ "could be read back; nothing to embed"
+ )
+ return
+ client = s3_client()
+ bucket = dataset_bucket()
+ prefix = settings.INBOX_RANKING_DATASET_S3_PREFIX
+ summaries = read_parquet_if_exists(
+ client,
+ bucket,
+ partition_object_key(prefix, SUMMARIES_TABLE, partition_key),
+ ["team_id", "user_id", "prompt_version", "summary_text", "status"],
+ )
+ if summaries is None:
+ context.log.warning(f"{SUMMARIES_TABLE} has no object for dt={partition_key}; nothing to embed")
+ return
+ targets = embedding_targets(summaries.to_pylist(), partition_key)
+ wanted = {document: target.text for document, target in targets.items()}
+ day_start, _ = snapshot_bounds(partition_key)
+
+ landed = landed_vectors(wanted.keys(), day_start)
+ pending = pending_documents(wanted, landed)
+ emitted = len(pending)
+ if pending:
+ for document in pending:
+ target = targets[document]
+ emit_summary_embedding(
+ team_id=document.team_id,
+ user_id=target.user_id,
+ partition_key=partition_key,
+ prompt_version=target.prompt_version,
+ text=target.text,
+ day_start=day_start,
+ )
+ flush_embedding_requests()
+ deadline = time.monotonic() + config.wait_seconds
+ while pending and time.monotonic() < deadline:
+ time.sleep(config.poll_interval_seconds)
+ landed |= landed_vectors(pending, day_start)
+ pending = pending_documents({document: wanted[document] for document in pending}, landed)
+
+ rows = summary_embedding_rows(targets, landed)
+ key = partition_object_key(prefix, SUMMARY_EMBEDDINGS_TABLE, partition_key)
+ existing_rows = object_row_count(client, bucket, key)
+ written = partition_write_allowed(existing_rows, len(rows))
+ if written:
+ write_parquet(client, bucket, key, pa.Table.from_pylist(rows, schema=SUMMARY_EMBEDDINGS_SCHEMA))
+ else:
+ context.log.warning(
+ f"dt={partition_key} already holds {existing_rows} vectors and this run has {len(rows)}; "
+ "the partition was not overwritten"
+ )
+ context.add_output_metadata(
+ {
+ "summaries": dagster.MetadataValue.int(summaries.num_rows),
+ "skipped_failed_summaries": dagster.MetadataValue.int(summaries.num_rows - len(targets)),
+ "emitted": dagster.MetadataValue.int(emitted),
+ "missing_vectors": dagster.MetadataValue.int(len(pending)),
+ "rows": dagster.MetadataValue.int(len(rows)),
+ "written": dagster.MetadataValue.bool(written),
+ "s3_key": dagster.MetadataValue.text(f"s3://{bucket}/{key}"),
+ }
+ )
+
+
activity_summary_job = dagster.define_asset_job(
name="activity_summary_job",
- selection=[INPUTS_TABLE, COST_PROJECTION_TABLE],
+ selection=[INPUTS_TABLE, COST_PROJECTION_TABLE, SUMMARIES_TABLE, SUMMARY_EMBEDDINGS_TABLE],
tags={**owner_tags, "dagster/max_runtime": str(60 * 60)},
)
diff --git a/products/signals/dags/inbox_ranking/activity_summary/embeddings.py b/products/signals/dags/inbox_ranking/activity_summary/embeddings.py
new file mode 100644
index 000000000000..c22fa9600445
--- /dev/null
+++ b/products/signals/dags/inbox_ranking/activity_summary/embeddings.py
@@ -0,0 +1,153 @@
+"""The embedding round trip behind `activity_summary_embeddings`.
+
+Each summary goes through the embedding worker, the path the report embeddings take, so a summary
+vector shares its model and space with the report vectors. The worker writes to
+`document_embeddings` asynchronously, so the asset emits, waits, and reads back what landed.
+"""
+
+import datetime
+from collections.abc import Collection
+from typing import Any, cast
+
+from posthog.api.embedding_worker import emit_embedding_request
+from posthog.clickhouse.client import sync_execute
+from posthog.clickhouse.client.connection import ClickHouseUser
+from posthog.dataclasses import frozen
+from posthog.kafka_client.routing import get_producer
+from posthog.kafka_client.topics import KAFKA_DOCUMENT_EMBEDDINGS_INPUT_TOPIC
+from posthog.schema_enums import EmbeddingModelName
+
+from products.signals.backend.report_embeddings import EMBEDDING_PRODUCT
+from products.signals.dags.inbox_ranking.dataset.queries import EMBEDDINGS_TABLE, etl_workload
+
+EMBEDDING_DOCUMENT_TYPE = "activity_summary"
+EMBEDDING_RENDERING = "summary_text"
+# The model `EMBEDDINGS_TABLE` holds. The worker also writes the other models, like the report rows.
+EMBEDDING_MODEL = EmbeddingModelName.TEXT_EMBEDDING_3_SMALL_1536
+KAFKA_FLUSH_TIMEOUT_SECONDS = 60.0
+# `document_embeddings` drops a row three months after its timestamp, which is the partition day. Three
+# months is never less than 89 days, and the margin leaves room for the worker and the read-back.
+SOURCE_TTL_DAYS = 89
+SOURCE_TTL_MARGIN_DAYS = 7
+
+# `timestamp` is in the table's sort key as `toDate(timestamp)`, and every row of a partition carries
+# the partition day, so the day bound prunes the read to one day of the table.
+SUMMARY_EMBEDDINGS_SQL = f"""
+SELECT
+ team_id,
+ document_id,
+ argMax(embedding, inserted_at) AS embedding,
+ argMax(content, inserted_at) AS content
+FROM {EMBEDDINGS_TABLE}
+WHERE product = %(product)s
+ AND document_type = %(document_type)s
+ AND rendering = %(rendering)s
+ AND team_id IN %(team_ids)s
+ AND document_id IN %(document_ids)s
+ AND timestamp >= %(day_start)s
+ AND timestamp < %(day_end)s
+GROUP BY team_id, document_id
+"""
+SUMMARY_EMBEDDINGS_QUERY_SETTINGS: dict[str, int] = {"max_execution_time": 120}
+
+
+@frozen(order=True)
+class SummaryDocument:
+ team_id: int
+ document_id: str
+
+
+@frozen
+class LandedVector:
+ embedding: list[float]
+ content: str
+
+
+def within_source_ttl(partition_key: str, today: datetime.date) -> bool:
+ """Whether a vector stamped with this partition day still lives in `document_embeddings` long
+ enough to be read back. An older backfill would lose its vectors and write a partial partition."""
+ age_days = (today - datetime.date.fromisoformat(partition_key)).days
+ return age_days <= SOURCE_TTL_DAYS - SOURCE_TTL_MARGIN_DAYS
+
+
+def document_id(user_id: int, partition_key: str) -> str:
+ """One document per (user, partition). The team is part of the table's key on its own."""
+ return f"{user_id}:{partition_key}"
+
+
+def emit_summary_embedding(
+ *, team_id: int, user_id: int, partition_key: str, prompt_version: int, text: str, day_start: datetime.datetime
+) -> None:
+ """Queue one summary for the worker.
+
+ `timestamp` is the partition day, not the emission time: the table replaces on a key that holds
+ `toDate(timestamp)`, so a re-run supersedes its own row instead of adding a second one."""
+ emit_embedding_request(
+ content=text,
+ team_id=team_id,
+ product=EMBEDDING_PRODUCT,
+ document_type=EMBEDDING_DOCUMENT_TYPE,
+ rendering=EMBEDDING_RENDERING,
+ document_id=document_id(user_id, partition_key),
+ models=[model.value for model in EmbeddingModelName],
+ timestamp=day_start,
+ metadata={
+ "team_id": team_id,
+ "user_id": user_id,
+ "partition": partition_key,
+ "prompt_version": prompt_version,
+ },
+ )
+
+
+def flush_embedding_requests() -> None:
+ """Deliver every queued request before the asset waits for the vectors, so a broken producer
+ fails the run instead of looking like a slow worker."""
+ undelivered = get_producer(topic=KAFKA_DOCUMENT_EMBEDDINGS_INPUT_TOPIC).flush(KAFKA_FLUSH_TIMEOUT_SECONDS)
+ if undelivered:
+ raise RuntimeError(f"{undelivered} embedding requests were not delivered to Kafka")
+
+
+def landed_vectors(
+ documents: Collection[SummaryDocument], day_start: datetime.datetime
+) -> dict[SummaryDocument, LandedVector]:
+ """The latest vector and the text it embeds for each document that has one."""
+ if not documents:
+ return {}
+ rows = cast(
+ list[tuple[Any, ...]],
+ sync_execute(
+ SUMMARY_EMBEDDINGS_SQL,
+ {
+ "product": EMBEDDING_PRODUCT,
+ "document_type": EMBEDDING_DOCUMENT_TYPE,
+ "rendering": EMBEDDING_RENDERING,
+ "team_ids": sorted({document.team_id for document in documents}),
+ "document_ids": sorted({document.document_id for document in documents}),
+ "day_start": day_start.replace(tzinfo=None),
+ "day_end": (day_start + datetime.timedelta(days=1)).replace(tzinfo=None),
+ },
+ settings=SUMMARY_EMBEDDINGS_QUERY_SETTINGS,
+ workload=etl_workload(),
+ ch_user=ClickHouseUser.SIGNALS,
+ )
+ or [],
+ )
+ return {
+ SummaryDocument(team_id=int(team_id), document_id=str(doc_id)): LandedVector(
+ embedding=list(embedding), content=str(content)
+ )
+ for team_id, doc_id, embedding, content in rows
+ }
+
+
+def pending_documents(
+ wanted: dict[SummaryDocument, str], landed: dict[SummaryDocument, LandedVector]
+) -> list[SummaryDocument]:
+ """The documents with no vector yet, or with a vector of other text, such as one written before
+ the summaries partition was re-run."""
+ return sorted(
+ document
+ for document, text in wanted.items()
+ if (vector := landed.get(document)) is None or vector.content != text
+ )
diff --git a/products/signals/dags/inbox_ranking/activity_summary/prompt.py b/products/signals/dags/inbox_ranking/activity_summary/prompt.py
new file mode 100644
index 000000000000..a38baff65940
--- /dev/null
+++ b/products/signals/dags/inbox_ranking/activity_summary/prompt.py
@@ -0,0 +1,166 @@
+"""The summary prompt, the output contract and the normalization of the model's answer.
+
+`summary_text` is the text the embeddings asset embeds, so its format is part of the contract with
+every vector already written. Bump `PROMPT_VERSION` when the prompt, the normalization or
+`summary_text` changes.
+"""
+
+import json
+import datetime
+from typing import Any
+
+PROMPT_VERSION = 1
+
+SUMMARY_MAX_WORDS = 60
+MAX_FOCUS_AREAS = 6
+MAX_PRODUCTS = 10
+STRENGTHS = ("high", "medium", "low")
+DEFAULT_STRENGTH = "low"
+# Strings a model writes in place of a JSON null.
+_NULL_STRINGS = frozenset({"", "none", "null", "n/a", "na", "nil"})
+
+SYSTEM_PROMPT = """You maintain a short summary of what one person has been working on in one PostHog project.
+You get their previous summary, if any, and a sample of their recent activity in up to three sections: tasks they gave to PostHog AI or the Slack agent, sessions of their own coding agent over MCP, and things they created or changed in PostHog.
+Update the summary to reflect what they are focused on now. Keep what is still true, drop what has gone quiet, and add what is new. Weigh themes that repeat across sections above one-off items. Treat bulk changes of the same kind as one theme, not a list.
+Name concrete products, features, metrics, pipelines and workflows. Never write generic phrases like "data analysis".
+Write about the work, not the person: no names, emails, IDs, secrets, or quotes from the input. Open with the work itself.
+Return JSON: `summary` (at most 60 words), `focus_areas` (up to 6, each with `topic`, `strength` high, medium or low, and `last_seen` as YYYY-MM-DD), `products`, and `recent_shift` (one sentence, or null).
+Set `recent_shift` only when there is a previous summary and the activity shows a real change from it. Otherwise it is null."""
+
+# Strict structured output. The normalization below still checks every rule, because a provider
+# behind the gateway may not enforce the schema.
+RESPONSE_FORMAT: Any = {
+ "type": "json_schema",
+ "json_schema": {
+ "name": "activity_summary",
+ "strict": True,
+ "schema": {
+ "type": "object",
+ "properties": {
+ "summary": {"type": "string"},
+ "focus_areas": {
+ "type": "array",
+ "items": {
+ "type": "object",
+ "properties": {
+ "topic": {"type": "string"},
+ "strength": {"type": "string", "enum": list(STRENGTHS)},
+ "last_seen": {"type": "string"},
+ },
+ "required": ["topic", "strength", "last_seen"],
+ "additionalProperties": False,
+ },
+ },
+ "products": {"type": "array", "items": {"type": "string"}},
+ "recent_shift": {"type": ["string", "null"]},
+ },
+ "required": ["summary", "focus_areas", "products", "recent_shift"],
+ "additionalProperties": False,
+ },
+ },
+}
+
+
+class MalformedSummaryError(ValueError):
+ """The model's answer cannot be turned into a summary. The message never quotes the answer."""
+
+
+def user_message(rendered_input: str, previous_summary: dict[str, Any] | None, as_of: datetime.date) -> str:
+ """The user turn: the day the activity ends, the previous summary and the rendered input.
+
+ The rendered input comes from `activity_summary_inputs` as is, and the previous summary is this
+ asset's own output, so no other text reaches the model."""
+ previous = (
+ json.dumps(
+ {key: previous_summary.get(key) for key in ("summary", "focus_areas", "products")},
+ ensure_ascii=False,
+ sort_keys=True,
+ )
+ if previous_summary
+ else "None"
+ )
+ return f"Today is {as_of.isoformat()}.\n\nPrevious summary:\n{previous}\n\nRecent activity:\n{rendered_input}"
+
+
+def _clean(value: Any) -> str | None:
+ if not isinstance(value, str):
+ return None
+ text = " ".join(value.split())
+ return None if text.lower() in _NULL_STRINGS else text
+
+
+def cap_words(text: str, max_words: int = SUMMARY_MAX_WORDS) -> str:
+ words = text.split()
+ return text if len(words) <= max_words else " ".join(words[:max_words])
+
+
+def _last_seen(value: Any) -> str | None:
+ text = _clean(value)
+ if text is None:
+ return None
+ try:
+ return datetime.date.fromisoformat(text).isoformat()
+ except ValueError:
+ return None
+
+
+def _focus_areas(value: Any) -> list[dict[str, str | None]]:
+ if not isinstance(value, list):
+ return []
+ areas: list[dict[str, str | None]] = []
+ for item in value:
+ if not isinstance(item, dict) or (topic := _clean(item.get("topic"))) is None:
+ continue
+ strength = (_clean(item.get("strength")) or "").lower()
+ areas.append(
+ {
+ "topic": topic,
+ "strength": strength if strength in STRENGTHS else DEFAULT_STRENGTH,
+ "last_seen": _last_seen(item.get("last_seen")),
+ }
+ )
+ if len(areas) == MAX_FOCUS_AREAS:
+ break
+ return areas
+
+
+def _products(value: Any) -> list[str]:
+ if not isinstance(value, list):
+ return []
+ products: list[str] = []
+ for item in value:
+ product = _clean(item)
+ if product is not None and product not in products:
+ products.append(product)
+ return products[:MAX_PRODUCTS]
+
+
+def normalize_summary(content: str | None, *, had_previous_summary: bool) -> dict[str, Any]:
+ """The model's JSON answer under the output rules, or `MalformedSummaryError`.
+
+ Only a missing or empty `summary` makes an answer malformed. Every other field degrades to an
+ empty or null value, so one bad focus area does not cost the row."""
+ if not content:
+ raise MalformedSummaryError("empty response")
+ try:
+ parsed = json.loads(content)
+ except json.JSONDecodeError as error:
+ raise MalformedSummaryError("response is not JSON") from error
+ if not isinstance(parsed, dict):
+ raise MalformedSummaryError("response is not a JSON object")
+ summary = _clean(parsed.get("summary"))
+ if summary is None:
+ raise MalformedSummaryError("response has no summary")
+ recent_shift = _clean(parsed.get("recent_shift")) if had_previous_summary else None
+ return {
+ "summary": cap_words(summary),
+ "focus_areas": _focus_areas(parsed.get("focus_areas")),
+ "products": _products(parsed.get("products")),
+ "recent_shift": recent_shift,
+ }
+
+
+def summary_text(summary: dict[str, Any]) -> str:
+ """The text that gets embedded: the summary, then the focus area topics in the model's order."""
+ topics = [area["topic"] for area in summary.get("focus_areas", [])]
+ return f"{summary['summary']}\nFocus areas: {'; '.join(topics)}" if topics else summary["summary"]
diff --git a/products/signals/dags/inbox_ranking/activity_summary/summaries.py b/products/signals/dags/inbox_ranking/activity_summary/summaries.py
new file mode 100644
index 000000000000..4a573b959c01
--- /dev/null
+++ b/products/signals/dags/inbox_ranking/activity_summary/summaries.py
@@ -0,0 +1,193 @@
+"""The LLM step behind `activity_summaries`: one call per inputs row, its cost, and its event.
+
+The call goes through `build_openai_client`, so the deployment picks the gateway: the Go AI gateway
+when its settings are set, else the Python gateway on the `signals` route.
+"""
+
+import json
+import time
+import datetime
+from collections.abc import Callable, Mapping, Sequence
+from concurrent.futures import ThreadPoolExecutor
+from typing import Any
+
+from openai import OpenAI, OpenAIError
+from openai.types.chat import ChatCompletionMessageParam
+
+from posthog.dataclasses import frozen
+from posthog.llm.gateway_client import build_openai_client, trace_id_from_seed
+
+from products.signals.dags.inbox_ranking.activity_summary.prompt import (
+ PROMPT_VERSION,
+ RESPONSE_FORMAT,
+ SYSTEM_PROMPT,
+ MalformedSummaryError,
+ normalize_summary,
+ summary_text,
+ user_message,
+)
+from products.signals.dags.inbox_ranking.training.telemetry import TrainingEvent
+
+AI_PRODUCT = "activity_summary"
+# Not a person: the gateway attributes every call to the pipeline, and the events use the same id.
+PIPELINE_DISTINCT_ID = "inbox_ranking_activity_summary"
+DEFAULT_MODEL = "gpt-6-luna"
+# One retry on a malformed answer. The SDK already retries a failed request on its own.
+MAX_ATTEMPTS = 2
+REQUEST_TIMEOUT_SECONDS = 120.0
+
+STATUS_OK = "ok"
+STATUS_FAILED = "failed"
+
+SUMMARY_GENERATED_EVENT = "activity summary generated"
+
+
+@frozen
+class SummaryPrices:
+ input_price_per_million_tokens: float | None
+ output_price_per_million_tokens: float | None
+
+
+def cost_usd(tokens_in: int, tokens_out: int, prices: SummaryPrices) -> float | None:
+ """None without both prices, so a run that does not set them records no guess."""
+ if prices.input_price_per_million_tokens is None or prices.output_price_per_million_tokens is None:
+ return None
+ return (
+ tokens_in * prices.input_price_per_million_tokens + tokens_out * prices.output_price_per_million_tokens
+ ) / 1_000_000
+
+
+@frozen
+class SummaryRequest:
+ team_id: int
+ user_id: int
+ rendered_input: str
+ previous_summary: dict[str, Any] | None
+
+
+@frozen
+class SummaryOutcome:
+ summary: dict[str, Any] | None
+ tokens_in: int
+ tokens_out: int
+ latency_ms: int
+ error: str | None
+
+
+def gateway_client(partition_key: str) -> OpenAI:
+ return build_openai_client(
+ "signals",
+ ai_product=AI_PRODUCT,
+ trace_id=trace_id_from_seed(f"{AI_PRODUCT}:{partition_key}"),
+ properties={"partition": partition_key},
+ distinct_id=PIPELINE_DISTINCT_ID,
+ )
+
+
+def summarize(client: OpenAI, request: SummaryRequest, *, model: str, as_of: datetime.date) -> SummaryOutcome:
+ """One summary. Tokens and latency add up over the attempts, because the retry costs too."""
+ messages: list[ChatCompletionMessageParam] = [
+ {"role": "system", "content": SYSTEM_PROMPT},
+ {"role": "user", "content": user_message(request.rendered_input, request.previous_summary, as_of)},
+ ]
+ tokens_in = tokens_out = 0
+ error: str | None = None
+ started = time.monotonic()
+ for _ in range(MAX_ATTEMPTS):
+ try:
+ response = client.chat.completions.create(
+ model=model,
+ messages=messages,
+ user=PIPELINE_DISTINCT_ID,
+ response_format=RESPONSE_FORMAT,
+ timeout=REQUEST_TIMEOUT_SECONDS,
+ )
+ except OpenAIError as api_error:
+ error = f"api_error: {type(api_error).__name__}"
+ break
+ if response.usage is not None:
+ tokens_in += response.usage.prompt_tokens
+ tokens_out += response.usage.completion_tokens
+ content = response.choices[0].message.content if response.choices else None
+ try:
+ summary = normalize_summary(content, had_previous_summary=request.previous_summary is not None)
+ except MalformedSummaryError as malformed:
+ error = f"malformed_output: {malformed}"
+ continue
+ return SummaryOutcome(
+ summary=summary,
+ tokens_in=tokens_in,
+ tokens_out=tokens_out,
+ latency_ms=round((time.monotonic() - started) * 1000),
+ error=None,
+ )
+ return SummaryOutcome(
+ summary=None,
+ tokens_in=tokens_in,
+ tokens_out=tokens_out,
+ latency_ms=round((time.monotonic() - started) * 1000),
+ error=error,
+ )
+
+
+def summarize_all(
+ requests: Sequence[SummaryRequest], summarize_one: Callable[[SummaryRequest], SummaryOutcome], max_concurrency: int
+) -> list[SummaryOutcome]:
+ """The outcomes in request order, at most `max_concurrency` calls at once."""
+ with ThreadPoolExecutor(max_workers=max(1, max_concurrency)) as pool:
+ return list(pool.map(summarize_one, requests))
+
+
+def summary_row(request: SummaryRequest, outcome: SummaryOutcome, *, model: str, prices: SummaryPrices) -> dict:
+ return {
+ "team_id": request.team_id,
+ "user_id": request.user_id,
+ "summary_json": json.dumps(outcome.summary, ensure_ascii=False, sort_keys=True) if outcome.summary else None,
+ "summary_text": summary_text(outcome.summary) if outcome.summary else None,
+ "used_previous_summary": request.previous_summary is not None,
+ "model": model,
+ "prompt_version": PROMPT_VERSION,
+ "tokens_in": outcome.tokens_in,
+ "tokens_out": outcome.tokens_out,
+ "latency_ms": outcome.latency_ms,
+ "cost_usd": cost_usd(outcome.tokens_in, outcome.tokens_out, prices),
+ "status": STATUS_OK if outcome.summary else STATUS_FAILED,
+ "error": outcome.error,
+ }
+
+
+_EVENT_ROW_KEYS = (
+ "team_id",
+ "user_id",
+ "model",
+ "prompt_version",
+ "status",
+ "error",
+ "used_previous_summary",
+ "tokens_in",
+ "tokens_out",
+ "cost_usd",
+ "latency_ms",
+)
+_EVENT_INPUT_KEYS = ("n_tasks", "n_mcp_sessions", "n_activity_lines")
+
+
+def summary_events(
+ rows: Sequence[Mapping[str, Any]], inputs: Sequence[Mapping[str, Any]], *, capture_summary_text: bool
+) -> list[TrainingEvent]:
+ """One event per summary row, `rows[i]` made from `inputs[i]`.
+
+ Counts only by default. The summary text goes on the event only when `capture_summary_text` is
+ set, because an event cannot be edited or deleted the way a partition can. The `error` value is
+ one of this module's own fixed strings, never model output."""
+ events: list[TrainingEvent] = []
+ for row, input_row in zip(rows, inputs, strict=True):
+ properties: dict[str, object] = {key: row[key] for key in _EVENT_ROW_KEYS}
+ properties |= {key: input_row[key] for key in _EVENT_INPUT_KEYS}
+ properties |= {f"drops_{rule}": count for rule, count in json.loads(input_row["drops"] or "{}").items()}
+ if capture_summary_text and row["summary_json"]:
+ summary = json.loads(row["summary_json"])
+ properties["summary_text"] = row["summary_text"]
+ properties["focus_topics"] = [area["topic"] for area in summary["focus_areas"]]
+ events.append(TrainingEvent(event=SUMMARY_GENERATED_EVENT, properties=properties))
+ return events
diff --git a/products/signals/dags/inbox_ranking/tests/test_activity_summary.py b/products/signals/dags/inbox_ranking/tests/test_activity_summary.py
index 7db32fbc2699..e7d21111c1a4 100644
--- a/products/signals/dags/inbox_ranking/tests/test_activity_summary.py
+++ b/products/signals/dags/inbox_ranking/tests/test_activity_summary.py
@@ -1,12 +1,33 @@
+import io
import json
import datetime
from collections import Counter
+from types import SimpleNamespace
+from typing import Any
import pytest
+import time_machine
+import dagster
+import pyarrow as pa
+from botocore.exceptions import ClientError
+from openai import APIConnectionError
from parameterized import parameterized
+from posthog import settings
+
from products.signals.dags.inbox_ranking.activity_summary import dag
+from products.signals.dags.inbox_ranking.activity_summary.embeddings import (
+ LandedVector,
+ SummaryDocument,
+ within_source_ttl,
+)
+from products.signals.dags.inbox_ranking.activity_summary.prompt import (
+ MalformedSummaryError,
+ normalize_summary,
+ summary_text,
+ user_message,
+)
from products.signals.dags.inbox_ranking.activity_summary.render import (
ACTIVITY_HEADER,
ACTIVITY_LINE_CHARS,
@@ -40,6 +61,15 @@ from products.signals.dags.inbox_ranking.activity_summary.render import (
sampling_seed,
task_units,
)
+from products.signals.dags.inbox_ranking.activity_summary.summaries import (
+ PIPELINE_DISTINCT_ID,
+ SummaryPrices,
+ SummaryRequest,
+ cost_usd,
+ summarize,
+ summary_events,
+ summary_row,
+)
AS_OF = datetime.datetime(2026, 10, 19, tzinfo=datetime.UTC)
@@ -361,3 +391,443 @@ def test_asset_rows_match_their_schemas_and_skip_projects_outside_the_organizati
),
)
assert set(projection[0]) | {"window_start", "window_end"} == set(dag.COST_PROJECTION_SCHEMA.names)
+
+
+PARTITION = "2026-10-18"
+PREFIX = "inbox_ranking"
+PREVIOUS = {"summary": "Builds the signup funnel.", "focus_areas": [], "products": ["product_analytics"]}
+
+
[email protected]([("with a previous summary", PREVIOUS), ("cold start", None)])
+def test_user_message_sends_only_the_rendered_input_and_the_previous_summary(
+ _name: str, previous: dict[str, Any] | None
+) -> None:
+ message = user_message("- 2026-10-17 Build a funnel", previous, datetime.date(2026, 10, 18))
+ assert message.startswith("Today is 2026-10-18.")
+ assert message.endswith("Recent activity:\n- 2026-10-17 Build a funnel")
+ if previous:
+ assert json.dumps(PREVIOUS["summary"]) in message
+ else:
+ assert "Previous summary:\nNone" in message
+
+
+def _answer(**overrides: Any) -> str:
+ return json.dumps(
+ {
+ "summary": "Builds the signup funnel.",
+ "focus_areas": [{"topic": "signup funnel", "strength": "high", "last_seen": "2026-10-17"}],
+ "products": ["product_analytics"],
+ "recent_shift": "Moved from flags to funnels.",
+ **overrides,
+ }
+ )
+
+
+ [
+ ("recent_shift is dropped without a previous summary", _answer(), False, {"recent_shift": None}),
+ (
+ "recent_shift is kept with a previous summary",
+ _answer(),
+ True,
+ {"recent_shift": "Moved from flags to funnels."},
+ ),
+ ("a 'None' recent_shift becomes null", _answer(recent_shift="None"), True, {"recent_shift": None}),
+ ("a 'null' recent_shift becomes null", _answer(recent_shift=" null "), True, {"recent_shift": None}),
+ (
+ "the summary is cut to the word cap",
+ _answer(summary="word " * 80),
+ False,
+ {"summary": " ".join(["word"] * 60)},
+ ),
+ (
+ "at most six focus areas, an unknown strength is low and a bad date is null",
+ _answer(focus_areas=[{"topic": f"t{i}", "strength": "huge", "last_seen": "soon"} for i in range(8)]),
+ False,
+ {"focus_areas": [{"topic": f"t{i}", "strength": "low", "last_seen": None} for i in range(6)]},
+ ),
+ (
+ "a focus area without a topic is dropped",
+ _answer(focus_areas=[{"strength": "high"}]),
+ False,
+ {"focus_areas": []},
+ ),
+ ]
+)
+def test_normalize_summary(_name: str, content: str, had_previous: bool, expected: dict[str, Any]) -> None:
+ assert normalize_summary(content, had_previous_summary=had_previous).items() >= expected.items()
+
+
+ [
+ ("empty", None),
+ ("not JSON", "Here is the summary: ..."),
+ ("not an object", "[1, 2]"),
+ ("no summary", _answer(summary=" ")),
+ ]
+)
+def test_normalize_summary_rejects_a_malformed_answer(_name: str, content: str | None) -> None:
+ with pytest.raises(MalformedSummaryError):
+ normalize_summary(content, had_previous_summary=False)
+
+
+def test_summary_text_is_deterministic_and_keeps_the_focus_order() -> None:
+ summary = normalize_summary(
+ _answer(
+ focus_areas=[
+ {"topic": "b", "strength": "high", "last_seen": ""},
+ {"topic": "a", "strength": "low", "last_seen": ""},
+ ]
+ ),
+ had_previous_summary=False,
+ )
+ reloaded = json.loads(json.dumps(summary, sort_keys=True))
+ assert summary_text(summary) == summary_text(reloaded) == "Builds the signup funnel.\nFocus areas: b; a"
+ assert summary_text({**summary, "focus_areas": []}) == "Builds the signup funnel."
+
+
+ [
+ (
+ "both prices",
+ 1_000_000,
+ 200_000,
+ SummaryPrices(input_price_per_million_tokens=0.1, output_price_per_million_tokens=0.5),
+ 0.2,
+ ),
+ (
+ "no output price",
+ 1_000,
+ 100,
+ SummaryPrices(input_price_per_million_tokens=0.1, output_price_per_million_tokens=None),
+ None,
+ ),
+ (
+ "no prices",
+ 1_000,
+ 100,
+ SummaryPrices(input_price_per_million_tokens=None, output_price_per_million_tokens=None),
+ None,
+ ),
+ ]
+)
+def test_cost_usd(_name: str, tokens_in: int, tokens_out: int, prices: SummaryPrices, expected: float | None) -> None:
+ assert cost_usd(tokens_in, tokens_out, prices) == (pytest.approx(expected) if expected is not None else None)
+
+
+class _FakeLLM:
+ def __init__(self, answers: list[str | Exception]):
+ self.answers = list(answers)
+ self.requests: list[dict[str, Any]] = []
+ self.chat = SimpleNamespace(completions=SimpleNamespace(create=self.create))
+
+ def create(self, **kwargs: Any) -> SimpleNamespace:
+ self.requests.append(kwargs)
+ answer = self.answers.pop(0) if len(self.answers) > 1 else self.answers[0]
+ if isinstance(answer, Exception):
+ raise answer
+ return SimpleNamespace(
+ usage=SimpleNamespace(prompt_tokens=100, completion_tokens=20),
+ choices=[SimpleNamespace(message=SimpleNamespace(content=answer))],
+ )
+
+
+_CONNECTION_ERROR = APIConnectionError(request=SimpleNamespace()) # type: ignore[arg-type]
+
+
+ [
+ ("a malformed answer is retried once", ["not json", _answer()], "ok", 2, None),
+ (
+ "two malformed answers fail the row",
+ ["not json", "still not json"],
+ "failed",
+ 2,
+ "malformed_output: response is not JSON",
+ ),
+ (
+ "a failed request fails the row without a retry",
+ [_CONNECTION_ERROR],
+ "failed",
+ 1,
+ "api_error: APIConnectionError",
+ ),
+ ]
+)
+def test_summarize_retries_a_malformed_answer_once(
+ _name: str, answers: list[str | Exception], status: str, attempts: int, error: str | None
+) -> None:
+ llm = _FakeLLM(answers)
+ request = SummaryRequest(team_id=2, user_id=7, rendered_input="- a line", previous_summary=None)
+ outcome = summarize(llm, request, model="gpt-6-luna", as_of=datetime.date(2026, 10, 18)) # type: ignore[arg-type]
+ row = summary_row(
+ request,
+ outcome,
+ model="gpt-6-luna",
+ prices=SummaryPrices(input_price_per_million_tokens=None, output_price_per_million_tokens=None),
+ )
+
+ assert len(llm.requests) == attempts
+ assert {req["user"] for req in llm.requests} == {PIPELINE_DISTINCT_ID}
+ assert (row["status"], row["error"]) == (status, error)
+ successful_calls = attempts if not isinstance(answers[0], Exception) else 0
+ assert (row["tokens_in"], row["tokens_out"]) == (100 * successful_calls, 20 * successful_calls)
+ assert set(row) == set(dag.SUMMARIES_SCHEMA.names)
+
+
[email protected]([("text off by default", False), ("text on when asked", True)])
+def test_summary_events_carry_free_text_only_when_asked(_name: str, capture_text: bool) -> None:
+ request = SummaryRequest(team_id=2, user_id=7, rendered_input="- a line", previous_summary=None)
+ outcome = summarize(_FakeLLM([_answer()]), request, model="gpt-6-luna", as_of=datetime.date(2026, 10, 18)) # type: ignore[arg-type]
+ row = summary_row(
+ request,
+ outcome,
+ model="gpt-6-luna",
+ prices=SummaryPrices(input_price_per_million_tokens=None, output_price_per_million_tokens=None),
+ )
+ input_row = {"n_tasks": 1, "n_mcp_sessions": 0, "n_activity_lines": 3, "drops": json.dumps({"burst_cap": 4})}
+
+ (event,) = summary_events([row], [input_row], capture_summary_text=capture_text)
+
+ assert {
+ "team_id": 2,
+ "user_id": 7,
+ "status": "ok",
+ "n_activity_lines": 3,
+ "drops_burst_cap": 4,
+ }.items() <= event.properties.items()
+ free_text = {"summary_text", "focus_topics"} & event.properties.keys()
+ assert free_text == ({"summary_text", "focus_topics"} if capture_text else set())
+ if not capture_text:
+ assert not any(isinstance(value, str) and "signup" in value for value in event.properties.values())
+
+
+class _MemoryS3:
+ def __init__(self) -> None:
+ self.objects: dict[str, tuple[bytes, dict[str, str]]] = {}
+
+ def upload_fileobj(self, fileobj: io.IOBase, bucket: str, key: str, ExtraArgs: dict[str, Any]) -> None:
+ self.objects[key] = (fileobj.read(), ExtraArgs["Metadata"])
+
+ def _get(self, key: str, operation: str) -> tuple[bytes, dict[str, str]]:
+ if key not in self.objects:
+ raise ClientError({"Error": {"Code": "NoSuchKey"}}, operation)
+ return self.objects[key]
+
+ def get_object(self, *, Bucket: str, Key: str) -> dict[str, Any]:
+ return {"Body": io.BytesIO(self._get(Key, "GetObject")[0])}
+
+ def head_object(self, *, Bucket: str, Key: str) -> dict[str, Any]:
+ return {"Metadata": self._get(Key, "HeadObject")[1]}
+
+ def get_paginator(self, _name: str) -> "_MemoryS3":
+ return self
+
+ def paginate(self, *, Bucket: str, Prefix: str): # noqa: ANN201
+ yield {"Contents": [{"Key": key} for key in sorted(self.objects) if key.startswith(Prefix)]}
+
+ def write(self, table: str, partition: str, rows: list[dict[str, Any]], schema: pa.Schema) -> None:
+ key = dag.partition_object_key(PREFIX, table, partition)
+ dag.write_parquet(self, "bucket", key, pa.Table.from_pylist(rows, schema=schema))
+
+ def read(self, table: str, partition: str) -> list[dict[str, Any]]:
+ table_ = dag.read_parquet_if_exists(self, "bucket", dag.partition_object_key(PREFIX, table, partition))
+ return [] if table_ is None else table_.to_pylist()
+
+
+class _FailingCapture:
+ def capture(self, **kwargs: Any) -> None:
+ raise RuntimeError("capture is down")
+
+ def shutdown(self) -> None:
+ pass
+
+
+def memory_s3(monkeypatch: pytest.MonkeyPatch) -> _MemoryS3:
+ s3 = _MemoryS3()
+ monkeypatch.setattr(dag, "skip_unconfigured", lambda context: False)
+ monkeypatch.setattr(dag, "s3_client", lambda: s3)
+ monkeypatch.setattr(dag, "dataset_bucket", lambda: "bucket")
+ monkeypatch.setattr(settings, "INBOX_RANKING_DATASET_S3_PREFIX", PREFIX)
+ return s3
+
+
+def _input_row(team_id: int, user_id: int, text: str) -> dict[str, Any]:
+ window = dag.activity_window(PARTITION)
+ return {
+ "team_id": team_id,
+ "user_id": user_id,
+ "window_start": window.start,
+ "window_end": window.end,
+ "n_tasks": 0,
+ "n_mcp_sessions": 1,
+ "n_activity_lines": 0,
+ "rendered_input": text,
+ "task_chars": 0,
+ "mcp_chars": len(text),
+ "activity_chars": 0,
+ "input_chars": len(text),
+ "input_tokens_estimate": 1,
+ "drops": "{}",
+ "render_version": 2,
+ }
+
+
+def _summary_row(team_id: int, user_id: int, status: str, summary: dict[str, Any] | None) -> dict[str, Any]:
+ return {
+ "team_id": team_id,
+ "user_id": user_id,
+ "summary_json": json.dumps(summary) if summary else None,
+ "summary_text": summary_text(summary) if summary else None,
+ "used_previous_summary": False,
+ "model": "gpt-6-luna",
+ "prompt_version": 1,
+ "tokens_in": 1,
+ "tokens_out": 1,
+ "latency_ms": 1,
+ "cost_usd": None,
+ "status": status,
+ "error": None if summary else "malformed_output: empty response",
+ }
+
+
+def test_summaries_asset_reads_earlier_partitions_only_and_survives_a_failed_capture(
+ memory_s3: _MemoryS3, monkeypatch: pytest.MonkeyPatch
+) -> None:
+ memory_s3.write(
+ dag.INPUTS_TABLE,
+ PARTITION,
+ [_input_row(2, 7, "- input seven"), _input_row(2, 8, "- input eight")],
+ dag.INPUTS_SCHEMA,
+ )
+ older = {"summary": "Older work on flags.", "focus_areas": [], "products": []}
+ memory_s3.write(dag.SUMMARIES_TABLE, "2026-10-10", [_summary_row(2, 7, "ok", older)], dag.SUMMARIES_SCHEMA)
+ memory_s3.write(
+ dag.SUMMARIES_TABLE,
+ "2026-10-17",
+ [_summary_row(2, 7, "ok", PREVIOUS), _summary_row(2, 8, "failed", None)],
+ dag.SUMMARIES_SCHEMA,
+ )
+ memory_s3.write(dag.SUMMARIES_TABLE, "2026-10-19", [_summary_row(2, 8, "ok", older)], dag.SUMMARIES_SCHEMA)
+ llm = _FakeLLM([_answer()])
+ monkeypatch.setattr(dag, "gateway_client", lambda partition_key: llm)
+ monkeypatch.setattr("products.signals.dags.inbox_ranking.training.telemetry.is_cloud", lambda: True)
+ monkeypatch.setattr(
+ "products.signals.dags.inbox_ranking.training.telemetry.get_client", lambda region, **kwargs: _FailingCapture()
+ )
+
+ for _ in range(2):
+ dag.activity_summaries(dagster.build_asset_context(partition_key=PARTITION), dag.SummariesConfig())
+
+ rows = memory_s3.read(dag.SUMMARIES_TABLE, PARTITION)
+ assert [(row["user_id"], row["status"], row["used_previous_summary"]) for row in rows] == [
+ (7, "ok", True),
+ (8, "ok", False),
+ ]
+ messages = {
+ req["messages"][1]["content"].rsplit("\n", 1)[-1]: req["messages"][1]["content"] for req in llm.requests
+ }
+ assert "Builds the signup funnel." in messages["- input seven"]
+ assert "Previous summary:\nNone" in messages["- input eight"]
+
+
+class _FailingForLLM(_FakeLLM):
+ def __init__(self, failing_input: str):
+ super().__init__([_answer()])
+ self.failing_input = failing_input
+
+ def create(self, **kwargs: Any) -> SimpleNamespace:
+ if kwargs["messages"][1]["content"].endswith(self.failing_input):
+ self.requests.append(kwargs)
+ raise _CONNECTION_ERROR
+ return super().create(**kwargs)
+
+
[email protected]("users", [[7], [7, 8]], ids=["every row fails", "one row fails"])
+def test_summaries_asset_keeps_earlier_successes_when_a_rerun_fails(
+ users: list[int], memory_s3: _MemoryS3, monkeypatch: pytest.MonkeyPatch
+) -> None:
+ names = {7: "seven", 8: "eight"}
+ memory_s3.write(
+ dag.INPUTS_TABLE,
+ PARTITION,
+ [_input_row(2, user, f"- input {names[user]}") for user in users],
+ dag.INPUTS_SCHEMA,
+ )
+ memory_s3.write(dag.SUMMARIES_TABLE, PARTITION, [_summary_row(2, 7, "ok", PREVIOUS)], dag.SUMMARIES_SCHEMA)
+ monkeypatch.setattr(dag, "gateway_client", lambda partition_key: _FailingForLLM("- input seven"))
+ context = dagster.build_asset_context(partition_key=PARTITION)
+
+ if len(users) == 1:
+ with pytest.raises(dagster.Failure):
+ dag.activity_summaries(context, dag.SummariesConfig())
+ else:
+ dag.activity_summaries(context, dag.SummariesConfig())
+
+ rows = memory_s3.read(dag.SUMMARIES_TABLE, PARTITION)
+ assert [(row["user_id"], row["status"]) for row in rows] == [(user, "ok") for user in users]
+ assert rows[0]["summary_json"] == json.dumps(PREVIOUS)
+
+
+ [
+ ("recent partition", "2026-10-18", True),
+ ("at the cutoff", "2026-07-29", True),
+ ("past the cutoff", "2026-07-28", False),
+ ]
+)
+def test_within_source_ttl(_name: str, partition_key: str, expected: bool) -> None:
+ assert within_source_ttl(partition_key, datetime.date(2026, 10, 19)) is expected
+
+
+@time_machine.travel(datetime.datetime(2026, 10, 19, 12, tzinfo=datetime.UTC), tick=False)
+def test_embeddings_asset_embeds_only_new_text_and_never_shrinks_a_written_partition(
+ memory_s3: _MemoryS3, monkeypatch: pytest.MonkeyPatch
+) -> None:
+ current = {"summary": "Builds the signup funnel.", "focus_areas": [], "products": []}
+ memory_s3.write(
+ dag.SUMMARIES_TABLE,
+ PARTITION,
+ [_summary_row(2, 7, "ok", current), _summary_row(2, 8, "ok", current), _summary_row(2, 9, "failed", None)],
+ dag.SUMMARIES_SCHEMA,
+ )
+ vector = [0.5] * 4
+ store = {
+ SummaryDocument(team_id=2, document_id=f"7:{PARTITION}"): LandedVector(
+ embedding=vector, content=summary_text(current)
+ ),
+ SummaryDocument(team_id=2, document_id=f"8:{PARTITION}"): LandedVector(
+ embedding=vector, content="text before a re-run"
+ ),
+ }
+ emitted: list[tuple[int, str]] = []
+
+ def emit(
+ *, team_id: int, user_id: int, partition_key: str, prompt_version: int, text: str, day_start: datetime.datetime
+ ) -> None:
+ emitted.append((user_id, text))
+ store[SummaryDocument(team_id=team_id, document_id=f"{user_id}:{partition_key}")] = LandedVector(
+ embedding=vector, content=text
+ )
+
+ monkeypatch.setattr(dag, "emit_summary_embedding", emit)
+ monkeypatch.setattr(dag, "flush_embedding_requests", lambda: None)
+ monkeypatch.setattr(dag.time, "sleep", lambda seconds: None)
+ monkeypatch.setattr(
+ dag, "landed_vectors", lambda documents, day_start: {d: store[d] for d in documents if d in store}
+ )
+
+ dag.activity_summary_embeddings(dagster.build_asset_context(partition_key=PARTITION), dag.SummaryEmbeddingsConfig())
+
+ assert emitted == [(8, summary_text(current))]
+ rows = memory_s3.read(dag.SUMMARY_EMBEDDINGS_TABLE, PARTITION)
+ assert [row["user_id"] for row in rows] == [7, 8]
+ assert set(rows[0]) == set(dag.SUMMARY_EMBEDDINGS_SCHEMA.names)
+
+ store.clear()
+ monkeypatch.setattr(dag, "emit_summary_embedding", lambda **kwargs: None)
+ dag.activity_summary_embeddings(
+ dagster.build_asset_context(partition_key=PARTITION), dag.SummaryEmbeddingsConfig(wait_seconds=0)
+ )
+ assert [row["user_id"] for row in memory_s3.read(dag.SUMMARY_EMBEDDINGS_TABLE, PARTITION)] == [7, 8]
diff --git a/products/signals/dags/inbox_ranking/training/telemetry.py b/products/signals/dags/inbox_ranking/training/telemetry.py
index e045fe87fe7c..18c0219eee2d 100644
--- a/products/signals/dags/inbox_ranking/training/telemetry.py
+++ b/products/signals/dags/inbox_ranking/training/telemetry.py
@@ -26,7 +26,8 @@ from products.signals.dags.inbox_ranking.training.unseen import CANDIDATE_ROLE,
# Not a person: one fixed id for the whole dag, and no person profile is created for it. Local dev
# runs get their own id so they never blend into the prod series.
DISTINCT_ID = "inbox_ranking_training"
-LOCAL_DISTINCT_ID = "inbox_ranking_training_local"
+LOCAL_SUFFIX = "_local"
+LOCAL_DISTINCT_ID = f"{DISTINCT_ID}{LOCAL_SUFFIX}"
LOCAL_ENVIRONMENT = "local"
CANDIDATE_TRAINED_EVENT = "inbox_ranking_candidate_trained"
EXAMPLES_BUILT_EVENT = "inbox_ranking_examples_built"
@@ -325,15 +326,18 @@ def capture_training_events(
events: list[TrainingEvent],
*,
timestamp: datetime.datetime | None = None,
+ distinct_id: str = DISTINCT_ID,
) -> None:
"""Best-effort: a failed capture is logged, never raised, so telemetry cannot fail a training
run. Local runs are marked twice, by `environment` and by their own distinct id, so a chart
that forgets to filter still shows them as a separate series. `timestamp` defaults to the
- partition's midday; an event about a person's action passes the moment it happened."""
+ partition's midday; an event about a person's action passes the moment it happened. Another
+ pipeline of this dag passes its own fixed `distinct_id`."""
if not capture_allowed():
return
environment = settings.CLOUD_DEPLOYMENT or LOCAL_ENVIRONMENT
- distinct_id = DISTINCT_ID if environment != LOCAL_ENVIRONMENT else LOCAL_DISTINCT_ID
+ if environment == LOCAL_ENVIRONMENT:
+ distinct_id = f"{distinct_id}{LOCAL_SUFFIX}"
timestamp = timestamp or event_timestamp(partition_key)
base = {"$process_person_profile": False, "partition": partition_key, "environment": environment}
# A fresh client per call, flushed on shutdown, for the same reason `ph_scoped_capture` builds