:hedgehog: PostHog is the leading platform for building self-driving products. Our developer tools – AI observability, analytics, session replay, flags, experiments, error tracking, logs, and more – capture all the context agents need to diagnose problems, uncover opportunities, and ship fixes. Steer it all from Slack, web, desktop, or the MCP.
Python ab-testingai-analyticsanalyticscdpdata-warehouseexperimentsfeature-flagsjavascriptproduct-analyticspythonreactsession-replaysurveystypescriptweb-analytics
fix(signals): check source liveness before anomaly scout close-out (#124640)
posthog[bot]4ad9722bccc8
diff --git a/products/signals/skills/AGENTS.md b/products/signals/skills/AGENTS.mdindex 79c9b0fe8f54..41ddc59099b8 100644--- a/products/signals/skills/AGENTS.md+++ b/products/signals/skills/AGENTS.md@@ -46,8 +46,9 @@ A scout is a skill that holds a `SignalScoutConfig`. The harness globs `signals-- `signals-scout-insight-alerts/` — digest + triage layer over the team's own configured insight alerts (threshold / detector alerts on insights). Doesn't re-detect anomalies — the user already set the thresholds — it reads each alert's recent firing history (`alerts-list` to triage all alerts cheaply, `alert-get` to deep-read the firing checks of the few candidates) and surfaces the recent firings a human most likely missed. Its discriminator is missed-ness × materiality × persistence, weighting hardest toward firings the standard notification path stayed silent on (no subscribers, suppressed by the investigation agent, or an alert silently Errored and no longer evaluating); skips snoozed / disabled / flapping / already-notified-and-resolved. Complements `observability-gaps` (which recommends _creating_ alerts) and `anomaly-detection` (which scores insights the team _views_, whether or not they're alerted) — this one owns the firings of alerts that already exist.- `signals-scout-csp-violations/` — anomaly watcher for Content Security Policy violations (`$csp_violation` blocked-URL clusters, per-directive bursts, post-deploy page-scoped regressions, suspicious third-party domains).- `signals-scout-anomaly-detection/` — anomaly watcher for the dashboards and insights a team actually views. Discovers high-traffic insights (view counts + dashboard access), curates a durable scratchpad watchlist, and balances re-checking known items (exploit) against discovering new ones (explore) across runs; scores the latest complete bucket by robust (MAD) deviation from each insight's own seasonality-matched baseline. On the **report channel** — it files each anomaly as a finished 1:1 inbox report via `emit_report` / `edit_report` (its `allowed_tools`) rather than a weak signal.+ Before its quick close-out on a project nobody views, it probes whether previously active product sources still send events, so a capture failure is not mistaken for disuse.Bundles its own references- (`anomaly-methods.md`, `watchlist-and-memory.md`, `report-contract.md`).+ (`anomaly-methods.md`, `watchlist-and-memory.md`, `report-contract.md`, `source-liveness.md`).- `signals-scout-health-checks/` — the judgment layer over PostHog's own health checks. Reads the project's active health issues (`health-issues-summary` / `-list` / `-get`) rather than re-running detection, and decides which are worth surfacing: bundles same-kind clusters into one finding, weights by real blast radius (cross-referenced against event volume / reach / SDK-version share), and prioritizes issues an agent can resolve via the MCP over credential-gated ones.Its discriminator is kind-concentration × severity × agent-fixability × persistence, not raw firing count. This includes `ingestion_warning` issues — the `ingestion_warning` check files one health issue per warning type (with `warning_type` / `category` / `severity` / `affected_count` in the payload), so the scout owns that triage end-to-end, corroborating blast radius via `execute-sql` against `system.ingestion_warnings` and pointing fixes at the `resolving-ingestion-warnings` skill.- `signals-scout-inbox-validation/` — follow-up watcher for the inbox itself.@@ -139,7 +140,7 @@ The legacy signal channel (weak `emit_signal` findings for the pipeline to clustThe user-facing skills (`authoring-scouts`, `exploring-scouts`) no longer teach it; new scouts that author reports list the report tools.The specialists each carry their own domain discriminator + investigation patterns.-Most are a single self-contained `SKILL.md`; a few bundle surface-specific references read on demand — `signals-scout-anomaly-detection` (`anomaly-methods.md`, `watchlist-and-memory.md`, `report-contract.md`), `signals-scout-ai-observability` (`lenses.md`), `signals-scout-apm` (`performance-issues.md`), `signals-scout-surveys` (`response-querying.md`), `signals-scout-web-vitals` (`remediation.md`, `onset-correlation.md`), `signals-scout-mcp-tool-calls` (`queries.md`, `metrics-dashboard.md`, `structured-output.schema.json`), `signals-scout-tasks` (`queries.md`), `signals-scout-pr-follow-up` (`sources.md`, `deploy-ladder.md`, `probes.md`), `signals-scout-product-analytics` (`funnel-maturity.md`), and `signals-scout-unused-features` (`queries.md`); `ls signals-scout-*/references/` is the current list. Treat the generalist as the reference shape. Note that a scout can only read its own bundled files at runtime (each team's `LLMSkill` row carries just that skill's files) — which is why a scout that needs the dedupe/memory conventions in depth bundles its own copy rather than pointing at the generalist's. The report-channel contract is the exception to that copy-it-locally rule: it rides in the harness prompt, not a file, so report scouts share one definition without each bundling a copy.+Most are a single self-contained `SKILL.md`; a few bundle surface-specific references read on demand — `signals-scout-anomaly-detection` (`anomaly-methods.md`, `watchlist-and-memory.md`, `report-contract.md`, `source-liveness.md`), `signals-scout-ai-observability` (`lenses.md`), `signals-scout-apm` (`performance-issues.md`), `signals-scout-surveys` (`response-querying.md`), `signals-scout-web-vitals` (`remediation.md`, `onset-correlation.md`), `signals-scout-mcp-tool-calls` (`queries.md`, `metrics-dashboard.md`, `structured-output.schema.json`), `signals-scout-tasks` (`queries.md`), `signals-scout-pr-follow-up` (`sources.md`, `deploy-ladder.md`, `probes.md`), `signals-scout-product-analytics` (`funnel-maturity.md`), and `signals-scout-unused-features` (`queries.md`); `ls signals-scout-*/references/` is the current list. Treat the generalist as the reference shape. Note that a scout can only read its own bundled files at runtime (each team's `LLMSkill` row carries just that skill's files) — which is why a scout that needs the dedupe/memory conventions in depth bundles its own copy rather than pointing at the generalist's. The report-channel contract is the exception to that copy-it-locally rule: it rides in the harness prompt, not a file, so report scouts share one definition without each bundling a copy.## When editing skills in this directorydiff --git a/products/signals/skills/signals-scout-anomaly-detection/SKILL.md b/products/signals/skills/signals-scout-anomaly-detection/SKILL.mdindex e877df89f045..0e5a151a9d6d 100644--- a/products/signals/skills/signals-scout-anomaly-detection/SKILL.md+++ b/products/signals/skills/signals-scout-anomaly-detection/SKILL.md@@ -29,7 +29,14 @@ You cannot scan a whole project in one run. Your leverage comes from a **durable## Quick close-out: is anything worth checking?-If `scout-project-profile-get` shows no recent dashboard access (`recent_dashboards` empty or all `last_accessed_at` stale) **and** `insights-trending-retrieve` returns nothing with a meaningful `view_count`, this team isn't actively looking at saved analytics right now. Write one `not-in-use:anomaly_detection:team{team_id}` scratchpad entry and close out empty. Re-running with the same key idempotently refreshes the timestamp.+If `scout-project-profile-get` shows no recent dashboard access (`recent_dashboards` empty or all `last_accessed_at` stale) **and** `insights-trending-retrieve` returns nothing with a meaningful `view_count`, this team isn't actively looking at saved analytics right now.+That proves low saved-analytics use. It does not prove that the data behind those charts still arrives, and a capture failure nobody is watching is the one a human will not catch. So check two things before you close out:++1. **Due watchlist items.** Read the watchlist first (see Get oriented). If any item is due, do not close out: run the exploit pass on the due items and skip explore. Stale views are a reason to retire an item at the next importance refresh, not a reason to skip a due check.+2. **Source liveness.** Run the bounded probe in [`references/source-liveness.md`](references/source-liveness.md). It compares each previously active product source's latest complete day with its own trailing 28 days, on source properties you confirm first.++Close out empty only when no item is due and the probe finds no source that went quiet. Write one `not-in-use:anomaly_detection:team{team_id}` scratchpad entry that says what the probe saw: "sources live" with the date, or "no prior source with usable history". A project with no usable history is not broken, so it keeps the quiet exit. Re-running with the same key idempotently refreshes the timestamp.+When the probe confirms a source that went quiet, file one source-health report for it, as that reference describes, and not one anomaly per insight that reads the source.## How a run works@@ -100,7 +107,7 @@ One paragraph: which watchlist items you checked, what you added, which anomalie- **Seasonal swings** — the regular daily/weekly rhythm (weekday vs weekend, business-hours vs overnight). Only real once the move clears the **seasonality-matched** baseline.- **The current partial bucket** — the in-progress hour/day is incomplete; never score it.-- **Data-pipeline gaps, not real drops** — a metric that flat-lines to zero across _every_ insight at the same timestamp is almost always missing/late data or a deploy gap, not a product anomaly. Note it (it may be worth its own report) but don't report it as a metric anomaly per insight.+- **Data-pipeline gaps, not real drops** — a metric that flat-lines to zero across _every_ insight at the same timestamp is almost always missing/late data or a deploy gap, not a product anomaly. Don't report it as a metric anomaly per insight. Run the source probe in [`references/source-liveness.md`](references/source-liveness.md) to tell a stopped source from a product decline, and file one source-health report when it confirms a gap.- **Low-count noise** — series whose baseline counts are tiny; a few events of movement is not signal. Enforce the minimum relative-change and minimum-absolute-count floors.- **Dev / test / internal-only segments** — bursts whose `properties.$environment` or service is `dev`/`local`/`test`, or single-user/single-session quirks.- **Expected one-offs the team already knows about** — launches, migrations, backfills, known experiments. If a `noise:` / `addressed:` entry names it, skip.@@ -135,7 +142,7 @@ Harness-level: `scout-project-profile-get`, `scout-scratchpad-search`, `scout-ru## When to stop-- Nothing worth checking (quick close-out) → close out empty.+- Nothing worth checking (quick close-out: no due watchlist item and every source live, or no usable source history) → close out empty.- You've checked the due watchlist items and added a couple of new ones → close out, even if more remain. Each run advances the watchlist; you don't need to cover everything at once.- A candidate matches a `noise:` / `addressed:` / `dedupe:` entry → skip.diff --git a/products/signals/skills/signals-scout-anomaly-detection/references/source-liveness.md b/products/signals/skills/signals-scout-anomaly-detection/references/source-liveness.mdnew file mode 100644index 000000000000..6e45760b361e--- /dev/null+++ b/products/signals/skills/signals-scout-anomaly-detection/references/source-liveness.md@@ -0,0 +1,142 @@+# Source liveness: is the data behind the charts still arriving?++Saved-analytics use and capture health are two different questions.+Stale dashboards and empty view counts mean nobody is looking at the charts.+They do not mean that the sources that fed those charts still send events.+A capture failure on a project nobody watches is the one no human will notice, so check it before the quick close-out.++The probe is one grouped query plus one confirmation query per candidate.+It compares each previously active source's latest complete day with its own trailing 28 complete days.++## 0. Is there any history?++If the profile's `top_events` is empty, the project has no recent events to compare.+Skip the probe and record the limit in the `not-in-use:` entry ("no events in the profile, no source history to check").+A project that never sent events is not broken.++## 1. Confirm the source properties++Do not assume that a property exists on every project.+`$lib` and `$host` are optional: a server SDK sends no `$host`, and a direct call to the capture API can send neither.+Call `read-data-schema` with `event_properties` for the top three events in the profile's `top_events`.+A property is confirmed when it appears on at least one of them.++Build `<source>` from confirmed properties only.+`'(none)'` holds events that have no value for the property.++- `$lib` and `$host`: `concat(coalesce(nullIf(toString(properties.$lib), ''), '(none)'), ' @ ', coalesce(nullIf(toString(properties.$host), ''), '(none)'))`+- `$lib` only: `coalesce(nullIf(toString(properties.$lib), ''), '(none)')`+- `$host` only: `coalesce(nullIf(toString(properties.$host), ''), '(none)')`+- Neither: `event`. Each event name is then its own source.++Do not use an app version property as a source.+Every release retires the previous version, so each old version reads as a source that stopped.++Write the `<source>` expression to `baseline:anomaly_detection:sources` and reuse it on later runs, so each `source_id` stays stable.+Build it again only when the schema adds or drops one of these properties.++## 2. Probe every source in one query++Use complete days only.+`toStartOfDay(now())` is the start of today, the partial bucket, and the query stops there.++```sql+SELECT+ <source> AS source,+ cityHash64(source) AS source_id,+ round(countIf(timestamp < toStartOfDay(now()) - INTERVAL 1 DAY) / 28, 1) AS baseline_daily,+ uniqIf(toDate(timestamp), timestamp < toStartOfDay(now()) - INTERVAL 1 DAY) AS active_days,+ countIf(timestamp >= toStartOfDay(now()) - INTERVAL 1 DAY) AS last_day,+ max(timestamp) AS last_seen+FROM events+WHERE timestamp >= toStartOfDay(now()) - INTERVAL 29 DAY+ AND timestamp < toStartOfDay(now())+GROUP BY source+HAVING baseline_daily >= 10 OR source_id IN (<recorded source_ids>)+ORDER BY baseline_daily DESC+LIMIT 50+```++- Replace `<recorded source_ids>` with the ids held in `baseline:anomaly_detection:sources`, or drop that clause on the first probe.+ A source that stopped more than a week ago has too few active days to pass the floor, and only the recorded list keeps it in view.+ A recorded source that is missing from the result sent nothing in 29 days.+- Do not add `SAMPLE`. Sampling keeps or drops whole distinct IDs, so a small source can vanish from the result.+- If the query times out, shorten the baseline to 14 days and divide by 14.++A **candidate** is a source that was active and is now quiet:++- it was active on at least 20 of the 28 baseline days, and its `baseline_daily` is at least about 10 events, **and**+- its `last_day` is below about 10% of its `baseline_daily`, or it is a recorded source that is missing from the result.++The floors keep a project with no real history on the quiet path.+A few stray events from a test or a one-off script never become a candidate.++Only a product source can be a candidate.+Skip a host that is `localhost`, an IP address, a preview or staging deploy, or a domain the team does not own.+Those stop and start for reasons that are not a capture failure.+When you cannot tell whether a host is the team's own, check whether its events match the team's other sources (the same event names, identified users) before you treat it as a candidate.++## 3. Confirm a candidate++Pull a 35-day daily series for the candidate only.+Filter on the numeric `source_id`.+Never paste a `$host`, `$lib` or event string into the query: captured values come from anyone who holds the public project token.++```sql+SELECT toDate(timestamp) AS day, count() AS events+FROM events+WHERE timestamp >= toStartOfDay(now()) - INTERVAL 35 DAY+ AND timestamp < toStartOfDay(now())+ AND cityHash64(<source>) = <source_id>+GROUP BY day+ORDER BY day+```++- **Weekly rhythm.** A source that sends only on weekdays is quiet every weekend.+ It is a candidate only when it had events on the same weekday as the latest complete day in each of the 4 prior weeks.+- **Onset.** The first complete day with no events or with less than about 10% of the baseline.+ `last_seen` from step 2 gives the last event.+- **Replacement.** Look for a source that first appeared near the onset with similar volume: the same host with a new `$lib`, or the same `$lib` on a new host.+ That is a migration, not a failure. Record it as an `addressed:anomaly_detection:source:<source_id>` entry and do not report it.++Then check the rest of the project.+Sum `last_day` and `baseline_daily` over the other rows of the step 2 result.++| Rest of the project on the latest complete day | Reads as |+| ---------------------------------------------- | ------------------------------------------------------------- |+| About 50% of its baseline or more | One source stopped while the others continue: a source gap |+| Also near zero | The whole project stopped sending: a project-wide capture gap |++## 4. Decide++| What the probe found | Outcome |+| ---------------------------------------------------------- | --------------------------------------------------------------------------------- |+| No event history, or no source passes the floors | Quiet exit. The `not-in-use:` entry says "no prior source with usable history" |+| Every source is live | Quiet exit. The `not-in-use:` entry says "sources live" and the date of the probe |+| A confirmed candidate (a source gap or a project-wide gap) | One source-health report. Do not close out empty |++A capture gap is not a product decline.+File **one** source-health report for the gap, not one anomaly per insight that reads the source.+Every insight on that source flat-lines at the same time, and those flat-lines are the gap, not separate findings.++Before you author, search the inbox (`inbox-reports-list`) and your `report:anomaly_detection:source:` pointers.+A project-wide gap can already be in the inbox: the `no_live_events` health check flags a project with no pageviews or screens in 30 days, and the health-checks scout reports it.+If a live report already covers this gap, edit it or skip.++Report shape:++- **Title**: the source, what stopped, and the onset date, for example "Events from posthog-node stopped on Oct 26 while web events continue".+- **Evidence**: the source (`$lib` and `$host` values, or the event name), `last_seen`, `baseline_daily`, `active_days`, the latest complete day's count, and the rest of the project against its baseline.+- **Chart**: the 35-day daily series for the source next to the rest of the project.+- **Next steps** that often fit: check the deploy or config change near the onset for that SDK or host, the project token and API host in that client, an ad blocker or proxy change, and a server job that stopped running.++Follow [`report-contract.md`](report-contract.md) for the notebook and the rest of the contract.+Edit the live report while the source stays quiet, and again when it comes back.++## Memory++| Key | Holds |+| ------------------------------------------------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------- |+| `baseline:anomaly_detection:sources` | Date of the last probe, the `<source>` expression, and each live source's `source_id`, readable source, and `baseline_daily`. Rewrite it after every probe |+| `report:anomaly_detection:source:<source_id>` | The `report_id` of the source-health report, so a later run edits it instead of filing a duplicate |+| `addressed:anomaly_detection:source:<source_id>` | A source the team retired or replaced on purpose. Skip it until it sends events again |diff --git a/products/signals/skills/signals-scout-anomaly-detection/references/watchlist-and-memory.md b/products/signals/skills/signals-scout-anomaly-detection/references/watchlist-and-memory.mdindex bc3456938b9c..ae07805c6013 100644--- a/products/signals/skills/signals-scout-anomaly-detection/references/watchlist-and-memory.md+++ b/products/signals/skills/signals-scout-anomaly-detection/references/watchlist-and-memory.md@@ -23,6 +23,8 @@ to update a baseline or a `last_checked` timestamp without creating duplicates).| `watchlist:anomaly_detection:importance-refresh` | Memo: when the watchlist's importance ranking was last reconciled + what changed. || `baseline:anomaly_detection:insight:<short_id>` | The learned normal: median + MAD per seasonal bucket, so scoring is cheap. || `report:anomaly_detection:insight:<short_id>` | Pointer to the inbox report you authored for this anomaly: `report_id` + re-escalation condition. Add a `:<series-or-direction>` suffix when one insight carries genuinely distinct concurrent anomalies (multi-series / breakdown, or an opposite-direction move) so they don't collapse onto one report. |+| `baseline:anomaly_detection:sources` | The source-liveness probe's state: last probe date, the `<source>` expression, and each live source's `source_id` and daily baseline (see `source-liveness.md`). |+| `report:anomaly_detection:source:<source_id>` | Pointer to the source-health report for a source that stopped sending, so the next run edits it instead of filing a duplicate. || `retired:anomaly_detection:<suffix of the retired row>` | Tombstone for a watchlist item you stopped scoring: why, and what would bring it back. One line. || `reviewer:anomaly_detection:<area>` | A cached owner: the bare lowercase GitHub login for a dashboard / metric area. || `noise:anomaly_detection:<topic>` | A pattern to ignore (a chronically erratic insight, a seasonal quirk). |