Skip to content

:hedgehog: PostHog is the leading platform for building self-driving products. Our developer tools – AI observability, analytics, session replay, flags, experiments, error tracking, logs, and more – capture all the context agents need to diagnose problems, uncover opportunities, and ship fixes. Steer it all from Slack, web, desktop, or the MCP.

fix(signals): check source liveness before anomaly scout close-out (#124640)

posthog[bot]4ad9722bccc8
diff --git a/products/signals/skills/AGENTS.md b/products/signals/skills/AGENTS.md
index 79c9b0fe8f54..41ddc59099b8 100644
--- a/products/signals/skills/AGENTS.md
+++ b/products/signals/skills/AGENTS.md
@@ -46,8 +46,9 @@ A scout is a skill that holds a `SignalScoutConfig`. The harness globs `signals-
- `signals-scout-insight-alerts/` — digest + triage layer over the team's own configured insight alerts (threshold / detector alerts on insights). Doesn't re-detect anomalies — the user already set the thresholds — it reads each alert's recent firing history (`alerts-list` to triage all alerts cheaply, `alert-get` to deep-read the firing checks of the few candidates) and surfaces the recent firings a human most likely missed. Its discriminator is missed-ness × materiality × persistence, weighting hardest toward firings the standard notification path stayed silent on (no subscribers, suppressed by the investigation agent, or an alert silently Errored and no longer evaluating); skips snoozed / disabled / flapping / already-notified-and-resolved. Complements `observability-gaps` (which recommends _creating_ alerts) and `anomaly-detection` (which scores insights the team _views_, whether or not they're alerted) — this one owns the firings of alerts that already exist.
- `signals-scout-csp-violations/` — anomaly watcher for Content Security Policy violations (`$csp_violation` blocked-URL clusters, per-directive bursts, post-deploy page-scoped regressions, suspicious third-party domains).
- `signals-scout-anomaly-detection/` — anomaly watcher for the dashboards and insights a team actually views. Discovers high-traffic insights (view counts + dashboard access), curates a durable scratchpad watchlist, and balances re-checking known items (exploit) against discovering new ones (explore) across runs; scores the latest complete bucket by robust (MAD) deviation from each insight's own seasonality-matched baseline. On the **report channel** — it files each anomaly as a finished 1:1 inbox report via `emit_report` / `edit_report` (its `allowed_tools`) rather than a weak signal.
+ Before its quick close-out on a project nobody views, it probes whether previously active product sources still send events, so a capture failure is not mistaken for disuse.
Bundles its own references
- (`anomaly-methods.md`, `watchlist-and-memory.md`, `report-contract.md`).
+ (`anomaly-methods.md`, `watchlist-and-memory.md`, `report-contract.md`, `source-liveness.md`).
- `signals-scout-health-checks/` — the judgment layer over PostHog's own health checks. Reads the project's active health issues (`health-issues-summary` / `-list` / `-get`) rather than re-running detection, and decides which are worth surfacing: bundles same-kind clusters into one finding, weights by real blast radius (cross-referenced against event volume / reach / SDK-version share), and prioritizes issues an agent can resolve via the MCP over credential-gated ones.
Its discriminator is kind-concentration × severity × agent-fixability × persistence, not raw firing count. This includes `ingestion_warning` issues — the `ingestion_warning` check files one health issue per warning type (with `warning_type` / `category` / `severity` / `affected_count` in the payload), so the scout owns that triage end-to-end, corroborating blast radius via `execute-sql` against `system.ingestion_warnings` and pointing fixes at the `resolving-ingestion-warnings` skill.
- `signals-scout-inbox-validation/` — follow-up watcher for the inbox itself.
@@ -139,7 +140,7 @@ The legacy signal channel (weak `emit_signal` findings for the pipeline to clust
The user-facing skills (`authoring-scouts`, `exploring-scouts`) no longer teach it; new scouts that author reports list the report tools.
The specialists each carry their own domain discriminator + investigation patterns.
-Most are a single self-contained `SKILL.md`; a few bundle surface-specific references read on demand — `signals-scout-anomaly-detection` (`anomaly-methods.md`, `watchlist-and-memory.md`, `report-contract.md`), `signals-scout-ai-observability` (`lenses.md`), `signals-scout-apm` (`performance-issues.md`), `signals-scout-surveys` (`response-querying.md`), `signals-scout-web-vitals` (`remediation.md`, `onset-correlation.md`), `signals-scout-mcp-tool-calls` (`queries.md`, `metrics-dashboard.md`, `structured-output.schema.json`), `signals-scout-tasks` (`queries.md`), `signals-scout-pr-follow-up` (`sources.md`, `deploy-ladder.md`, `probes.md`), `signals-scout-product-analytics` (`funnel-maturity.md`), and `signals-scout-unused-features` (`queries.md`); `ls signals-scout-*/references/` is the current list. Treat the generalist as the reference shape. Note that a scout can only read its own bundled files at runtime (each team's `LLMSkill` row carries just that skill's files) — which is why a scout that needs the dedupe/memory conventions in depth bundles its own copy rather than pointing at the generalist's. The report-channel contract is the exception to that copy-it-locally rule: it rides in the harness prompt, not a file, so report scouts share one definition without each bundling a copy.
+Most are a single self-contained `SKILL.md`; a few bundle surface-specific references read on demand — `signals-scout-anomaly-detection` (`anomaly-methods.md`, `watchlist-and-memory.md`, `report-contract.md`, `source-liveness.md`), `signals-scout-ai-observability` (`lenses.md`), `signals-scout-apm` (`performance-issues.md`), `signals-scout-surveys` (`response-querying.md`), `signals-scout-web-vitals` (`remediation.md`, `onset-correlation.md`), `signals-scout-mcp-tool-calls` (`queries.md`, `metrics-dashboard.md`, `structured-output.schema.json`), `signals-scout-tasks` (`queries.md`), `signals-scout-pr-follow-up` (`sources.md`, `deploy-ladder.md`, `probes.md`), `signals-scout-product-analytics` (`funnel-maturity.md`), and `signals-scout-unused-features` (`queries.md`); `ls signals-scout-*/references/` is the current list. Treat the generalist as the reference shape. Note that a scout can only read its own bundled files at runtime (each team's `LLMSkill` row carries just that skill's files) — which is why a scout that needs the dedupe/memory conventions in depth bundles its own copy rather than pointing at the generalist's. The report-channel contract is the exception to that copy-it-locally rule: it rides in the harness prompt, not a file, so report scouts share one definition without each bundling a copy.
## When editing skills in this directory
diff --git a/products/signals/skills/signals-scout-anomaly-detection/SKILL.md b/products/signals/skills/signals-scout-anomaly-detection/SKILL.md
index e877df89f045..0e5a151a9d6d 100644
--- a/products/signals/skills/signals-scout-anomaly-detection/SKILL.md
+++ b/products/signals/skills/signals-scout-anomaly-detection/SKILL.md
@@ -29,7 +29,14 @@ You cannot scan a whole project in one run. Your leverage comes from a **durable
## Quick close-out: is anything worth checking?
-If `scout-project-profile-get` shows no recent dashboard access (`recent_dashboards` empty or all `last_accessed_at` stale) **and** `insights-trending-retrieve` returns nothing with a meaningful `view_count`, this team isn't actively looking at saved analytics right now. Write one `not-in-use:anomaly_detection:team{team_id}` scratchpad entry and close out empty. Re-running with the same key idempotently refreshes the timestamp.
+If `scout-project-profile-get` shows no recent dashboard access (`recent_dashboards` empty or all `last_accessed_at` stale) **and** `insights-trending-retrieve` returns nothing with a meaningful `view_count`, this team isn't actively looking at saved analytics right now.
+That proves low saved-analytics use. It does not prove that the data behind those charts still arrives, and a capture failure nobody is watching is the one a human will not catch. So check two things before you close out:
+
+1. **Due watchlist items.** Read the watchlist first (see Get oriented). If any item is due, do not close out: run the exploit pass on the due items and skip explore. Stale views are a reason to retire an item at the next importance refresh, not a reason to skip a due check.
+2. **Source liveness.** Run the bounded probe in [`references/source-liveness.md`](references/source-liveness.md). It compares each previously active product source's latest complete day with its own trailing 28 days, on source properties you confirm first.
+
+Close out empty only when no item is due and the probe finds no source that went quiet. Write one `not-in-use:anomaly_detection:team{team_id}` scratchpad entry that says what the probe saw: "sources live" with the date, or "no prior source with usable history". A project with no usable history is not broken, so it keeps the quiet exit. Re-running with the same key idempotently refreshes the timestamp.
+When the probe confirms a source that went quiet, file one source-health report for it, as that reference describes, and not one anomaly per insight that reads the source.
## How a run works
@@ -100,7 +107,7 @@ One paragraph: which watchlist items you checked, what you added, which anomalie
- **Seasonal swings** — the regular daily/weekly rhythm (weekday vs weekend, business-hours vs overnight). Only real once the move clears the **seasonality-matched** baseline.
- **The current partial bucket** — the in-progress hour/day is incomplete; never score it.
-- **Data-pipeline gaps, not real drops** — a metric that flat-lines to zero across _every_ insight at the same timestamp is almost always missing/late data or a deploy gap, not a product anomaly. Note it (it may be worth its own report) but don't report it as a metric anomaly per insight.
+- **Data-pipeline gaps, not real drops** — a metric that flat-lines to zero across _every_ insight at the same timestamp is almost always missing/late data or a deploy gap, not a product anomaly. Don't report it as a metric anomaly per insight. Run the source probe in [`references/source-liveness.md`](references/source-liveness.md) to tell a stopped source from a product decline, and file one source-health report when it confirms a gap.
- **Low-count noise** — series whose baseline counts are tiny; a few events of movement is not signal. Enforce the minimum relative-change and minimum-absolute-count floors.
- **Dev / test / internal-only segments** — bursts whose `properties.$environment` or service is `dev`/`local`/`test`, or single-user/single-session quirks.
- **Expected one-offs the team already knows about** — launches, migrations, backfills, known experiments. If a `noise:` / `addressed:` entry names it, skip.
@@ -135,7 +142,7 @@ Harness-level: `scout-project-profile-get`, `scout-scratchpad-search`, `scout-ru
## When to stop
-- Nothing worth checking (quick close-out) → close out empty.
+- Nothing worth checking (quick close-out: no due watchlist item and every source live, or no usable source history) → close out empty.
- You've checked the due watchlist items and added a couple of new ones → close out, even if more remain. Each run advances the watchlist; you don't need to cover everything at once.
- A candidate matches a `noise:` / `addressed:` / `dedupe:` entry → skip.
diff --git a/products/signals/skills/signals-scout-anomaly-detection/references/source-liveness.md b/products/signals/skills/signals-scout-anomaly-detection/references/source-liveness.md
new file mode 100644
index 000000000000..6e45760b361e
--- /dev/null
+++ b/products/signals/skills/signals-scout-anomaly-detection/references/source-liveness.md
@@ -0,0 +1,142 @@
+# Source liveness: is the data behind the charts still arriving?
+
+Saved-analytics use and capture health are two different questions.
+Stale dashboards and empty view counts mean nobody is looking at the charts.
+They do not mean that the sources that fed those charts still send events.
+A capture failure on a project nobody watches is the one no human will notice, so check it before the quick close-out.
+
+The probe is one grouped query plus one confirmation query per candidate.
+It compares each previously active source's latest complete day with its own trailing 28 complete days.
+
+## 0. Is there any history?
+
+If the profile's `top_events` is empty, the project has no recent events to compare.
+Skip the probe and record the limit in the `not-in-use:` entry ("no events in the profile, no source history to check").
+A project that never sent events is not broken.
+
+## 1. Confirm the source properties
+
+Do not assume that a property exists on every project.
+`$lib` and `$host` are optional: a server SDK sends no `$host`, and a direct call to the capture API can send neither.
+Call `read-data-schema` with `event_properties` for the top three events in the profile's `top_events`.
+A property is confirmed when it appears on at least one of them.
+
+Build `<source>` from confirmed properties only.
+`'(none)'` holds events that have no value for the property.
+
+- `$lib` and `$host`: `concat(coalesce(nullIf(toString(properties.$lib), ''), '(none)'), ' @ ', coalesce(nullIf(toString(properties.$host), ''), '(none)'))`
+- `$lib` only: `coalesce(nullIf(toString(properties.$lib), ''), '(none)')`
+- `$host` only: `coalesce(nullIf(toString(properties.$host), ''), '(none)')`
+- Neither: `event`. Each event name is then its own source.
+
+Do not use an app version property as a source.
+Every release retires the previous version, so each old version reads as a source that stopped.
+
+Write the `<source>` expression to `baseline:anomaly_detection:sources` and reuse it on later runs, so each `source_id` stays stable.
+Build it again only when the schema adds or drops one of these properties.
+
+## 2. Probe every source in one query
+
+Use complete days only.
+`toStartOfDay(now())` is the start of today, the partial bucket, and the query stops there.
+
+```sql
+SELECT
+ <source> AS source,
+ cityHash64(source) AS source_id,
+ round(countIf(timestamp < toStartOfDay(now()) - INTERVAL 1 DAY) / 28, 1) AS baseline_daily,
+ uniqIf(toDate(timestamp), timestamp < toStartOfDay(now()) - INTERVAL 1 DAY) AS active_days,
+ countIf(timestamp >= toStartOfDay(now()) - INTERVAL 1 DAY) AS last_day,
+ max(timestamp) AS last_seen
+FROM events
+WHERE timestamp >= toStartOfDay(now()) - INTERVAL 29 DAY
+ AND timestamp < toStartOfDay(now())
+GROUP BY source
+HAVING baseline_daily >= 10 OR source_id IN (<recorded source_ids>)
+ORDER BY baseline_daily DESC
+LIMIT 50
+```
+
+- Replace `<recorded source_ids>` with the ids held in `baseline:anomaly_detection:sources`, or drop that clause on the first probe.
+ A source that stopped more than a week ago has too few active days to pass the floor, and only the recorded list keeps it in view.
+ A recorded source that is missing from the result sent nothing in 29 days.
+- Do not add `SAMPLE`. Sampling keeps or drops whole distinct IDs, so a small source can vanish from the result.
+- If the query times out, shorten the baseline to 14 days and divide by 14.
+
+A **candidate** is a source that was active and is now quiet:
+
+- it was active on at least 20 of the 28 baseline days, and its `baseline_daily` is at least about 10 events, **and**
+- its `last_day` is below about 10% of its `baseline_daily`, or it is a recorded source that is missing from the result.
+
+The floors keep a project with no real history on the quiet path.
+A few stray events from a test or a one-off script never become a candidate.
+
+Only a product source can be a candidate.
+Skip a host that is `localhost`, an IP address, a preview or staging deploy, or a domain the team does not own.
+Those stop and start for reasons that are not a capture failure.
+When you cannot tell whether a host is the team's own, check whether its events match the team's other sources (the same event names, identified users) before you treat it as a candidate.
+
+## 3. Confirm a candidate
+
+Pull a 35-day daily series for the candidate only.
+Filter on the numeric `source_id`.
+Never paste a `$host`, `$lib` or event string into the query: captured values come from anyone who holds the public project token.
+
+```sql
+SELECT toDate(timestamp) AS day, count() AS events
+FROM events
+WHERE timestamp >= toStartOfDay(now()) - INTERVAL 35 DAY
+ AND timestamp < toStartOfDay(now())
+ AND cityHash64(<source>) = <source_id>
+GROUP BY day
+ORDER BY day
+```
+
+- **Weekly rhythm.** A source that sends only on weekdays is quiet every weekend.
+ It is a candidate only when it had events on the same weekday as the latest complete day in each of the 4 prior weeks.
+- **Onset.** The first complete day with no events or with less than about 10% of the baseline.
+ `last_seen` from step 2 gives the last event.
+- **Replacement.** Look for a source that first appeared near the onset with similar volume: the same host with a new `$lib`, or the same `$lib` on a new host.
+ That is a migration, not a failure. Record it as an `addressed:anomaly_detection:source:<source_id>` entry and do not report it.
+
+Then check the rest of the project.
+Sum `last_day` and `baseline_daily` over the other rows of the step 2 result.
+
+| Rest of the project on the latest complete day | Reads as |
+| ---------------------------------------------- | ------------------------------------------------------------- |
+| About 50% of its baseline or more | One source stopped while the others continue: a source gap |
+| Also near zero | The whole project stopped sending: a project-wide capture gap |
+
+## 4. Decide
+
+| What the probe found | Outcome |
+| ---------------------------------------------------------- | --------------------------------------------------------------------------------- |
+| No event history, or no source passes the floors | Quiet exit. The `not-in-use:` entry says "no prior source with usable history" |
+| Every source is live | Quiet exit. The `not-in-use:` entry says "sources live" and the date of the probe |
+| A confirmed candidate (a source gap or a project-wide gap) | One source-health report. Do not close out empty |
+
+A capture gap is not a product decline.
+File **one** source-health report for the gap, not one anomaly per insight that reads the source.
+Every insight on that source flat-lines at the same time, and those flat-lines are the gap, not separate findings.
+
+Before you author, search the inbox (`inbox-reports-list`) and your `report:anomaly_detection:source:` pointers.
+A project-wide gap can already be in the inbox: the `no_live_events` health check flags a project with no pageviews or screens in 30 days, and the health-checks scout reports it.
+If a live report already covers this gap, edit it or skip.
+
+Report shape:
+
+- **Title**: the source, what stopped, and the onset date, for example "Events from posthog-node stopped on Oct 26 while web events continue".
+- **Evidence**: the source (`$lib` and `$host` values, or the event name), `last_seen`, `baseline_daily`, `active_days`, the latest complete day's count, and the rest of the project against its baseline.
+- **Chart**: the 35-day daily series for the source next to the rest of the project.
+- **Next steps** that often fit: check the deploy or config change near the onset for that SDK or host, the project token and API host in that client, an ad blocker or proxy change, and a server job that stopped running.
+
+Follow [`report-contract.md`](report-contract.md) for the notebook and the rest of the contract.
+Edit the live report while the source stays quiet, and again when it comes back.
+
+## Memory
+
+| Key | Holds |
+| ------------------------------------------------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| `baseline:anomaly_detection:sources` | Date of the last probe, the `<source>` expression, and each live source's `source_id`, readable source, and `baseline_daily`. Rewrite it after every probe |
+| `report:anomaly_detection:source:<source_id>` | The `report_id` of the source-health report, so a later run edits it instead of filing a duplicate |
+| `addressed:anomaly_detection:source:<source_id>` | A source the team retired or replaced on purpose. Skip it until it sends events again |
diff --git a/products/signals/skills/signals-scout-anomaly-detection/references/watchlist-and-memory.md b/products/signals/skills/signals-scout-anomaly-detection/references/watchlist-and-memory.md
index bc3456938b9c..ae07805c6013 100644
--- a/products/signals/skills/signals-scout-anomaly-detection/references/watchlist-and-memory.md
+++ b/products/signals/skills/signals-scout-anomaly-detection/references/watchlist-and-memory.md
@@ -23,6 +23,8 @@ to update a baseline or a `last_checked` timestamp without creating duplicates).
| `watchlist:anomaly_detection:importance-refresh` | Memo: when the watchlist's importance ranking was last reconciled + what changed. |
| `baseline:anomaly_detection:insight:<short_id>` | The learned normal: median + MAD per seasonal bucket, so scoring is cheap. |
| `report:anomaly_detection:insight:<short_id>` | Pointer to the inbox report you authored for this anomaly: `report_id` + re-escalation condition. Add a `:<series-or-direction>` suffix when one insight carries genuinely distinct concurrent anomalies (multi-series / breakdown, or an opposite-direction move) so they don't collapse onto one report. |
+| `baseline:anomaly_detection:sources` | The source-liveness probe's state: last probe date, the `<source>` expression, and each live source's `source_id` and daily baseline (see `source-liveness.md`). |
+| `report:anomaly_detection:source:<source_id>` | Pointer to the source-health report for a source that stopped sending, so the next run edits it instead of filing a duplicate. |
| `retired:anomaly_detection:<suffix of the retired row>` | Tombstone for a watchlist item you stopped scoring: why, and what would bring it back. One line. |
| `reviewer:anomaly_detection:<area>` | A cached owner: the bare lowercase GitHub login for a dashboard / metric area. |
| `noise:anomaly_detection:<topic>` | A pattern to ignore (a chronically erratic insight, a seasonal quirk). |