Skip to content

:hedgehog: PostHog is the leading platform for building self-driving products. Our developer tools – AI observability, analytics, session replay, flags, experiments, error tracking, logs, and more – capture all the context agents need to diagnose problems, uncover opportunities, and ship fixes. Steer it all from Slack, web, desktop, or the MCP.

chore(signals): correct scout pre-check observation guidance (#120489)

posthog[bot]db26250e32f2
diff --git a/products/signals/skills/AGENTS.md b/products/signals/skills/AGENTS.md
index 12b8092b2fed..91cb9136133d 100644
--- a/products/signals/skills/AGENTS.md
+++ b/products/signals/skills/AGENTS.md
@@ -127,8 +127,13 @@ The generalist (`signals-scout-general`) is **report-only** — it authors `Sign
- **`references/discovery.md`** — the concrete moves of the bounded discovery pass behind its coverage map.
- **`references/new-traffic.md`** — the periodic new-traffic check: a by-source volume step against older weeks, the shapes that separate growth from foreign traffic (a forked client, a copied token, a bot, a runaway loop), and the pinned pre-surge baseline that keeps a slow surge visible.
-The entire canonical fleet is on the **report channel** (ported one scout per PR, biggest reach first — see the `scouts-emit-reports` spec). A report-channel scout is report-only — its frontmatter `allowed_tools` lists `emit_report` / `edit_report` — and it carries only its _domain-specific_ report framing **inline in its body** (what's report-shaped for its surface, its `reviewer:<domain>` / `report:<domain>` scratchpad keys, a tailored title example); the channel contract comes from the prompt, so a ported scout bundles **no** `report.md`. The exception is `signals-scout-anomaly-detection`, which keeps a slimmed `references/report-contract.md` for its genuinely scout-specific **notebook write-up + embedded-chart recipe** (it defers the generic contract to the prompt). The legacy signal channel (weak `emit_signal` findings for the pipeline to cluster) still exists in the harness for custom scouts that never opted in via `allowed_tools`, but it is deprecated:
-no canonical scout uses it, and the user-facing skills (`authoring-scouts`, `exploring-scouts`) no longer teach it — new scouts always list the report tools.
+Most canonical scouts author reports on the **report channel**.
+`signals-scout-workflows` writes workflow suggestions and deliberately uses neither signal nor report output; preserve its frontmatter and permissions when editing fleet guidance.
+A report-channel scout is report-only: its frontmatter `allowed_tools` lists `emit_report` / `edit_report`, and it carries only its _domain-specific_ report framing **inline in its body** (what's report-shaped for its surface, its `reviewer:<domain>` / `report:<domain>` scratchpad keys, a tailored title example).
+The channel contract comes from the prompt, so a report-channel scout bundles **no** `report.md`.
+`signals-scout-anomaly-detection` keeps a slimmed `references/report-contract.md` for its scout-specific **notebook write-up + embedded-chart recipe** (it defers the generic contract to the prompt).
+The legacy signal channel (weak `emit_signal` findings for the pipeline to cluster) still exists in the harness for custom scouts that never opted in via `allowed_tools`, but it is deprecated.
+The user-facing skills (`authoring-scouts`, `exploring-scouts`) no longer teach it; new scouts that author reports list the report tools.
The specialists each carry their own domain discriminator + investigation patterns.
Most are a single self-contained `SKILL.md`; a few bundle surface-specific references read on demand — `signals-scout-anomaly-detection` (`anomaly-methods.md`, `watchlist-and-memory.md`, `report-contract.md`), `signals-scout-ai-observability` (`lenses.md`), `signals-scout-surveys` (`response-querying.md`), `signals-scout-web-vitals` (`remediation.md`, `onset-correlation.md`), `signals-scout-mcp-tool-calls` (`queries.md`, `metrics-dashboard.md`, `structured-output.schema.json`), `signals-scout-tasks` (`queries.md`), `signals-scout-pr-follow-up` (`sources.md`, `deploy-ladder.md`, `probes.md`), and `signals-scout-unused-features` (`queries.md`); `ls signals-scout-*/references/` is the current list. Treat the generalist as the reference shape. Note that a scout can only read its own bundled files at runtime (each team's `LLMSkill` row carries just that skill's files) — which is why a scout that needs the dedupe/memory conventions in depth bundles its own copy rather than pointing at the generalist's. The report-channel contract is the exception to that copy-it-locally rule: it rides in the harness prompt, not a file, so report scouts share one definition without each bundling a copy.
diff --git a/products/signals/skills/authoring-scouts/references/scheduled-precheck.md b/products/signals/skills/authoring-scouts/references/scheduled-precheck.md
index 581f28ea6fce..43074f977608 100644
--- a/products/signals/skills/authoring-scouts/references/scheduled-precheck.md
+++ b/products/signals/skills/authoring-scouts/references/scheduled-precheck.md
@@ -125,14 +125,25 @@ A manual run has no block, so the body must still work without one.
## Diagnose a tick with no run row
-A skipped tick writes nothing. The coordinator still advances `last_run_at` (the dispatch stamp), so a fresh `last_run_at` with no matching run row can be a pre-check skip.
+A skipped tick creates no run row, sandbox, or model call, but stores its pre-check outcome on the config.
Before you treat missing runs as a failure:
-1. Read `effective_precheck_query` and `precheck_query_source` in `scout-config-list`.
- When the source is `off`, the pre-check does not explain the gap.
-2. Run `scout-config-precheck-test` without a query.
- `would_run: false` means the next scheduled run skips, so the gap is expected.
-3. Remember that the dry run reads as you, and the scheduled check reads as the acting user.
+1. Read the config in `scout-config-list`:
+ - `last_precheck_at`: the time of the latest scheduled pre-check.
+ - `last_precheck_outcome`: `skip` (the query found nothing new), `run` (the query returned rows that allow a run), or `error` (the query failed, so the gate allows a run).
+ - `last_precheck_row_count`: how many rows it returned. Zero rows or one false value can both skip, so use the outcome to tell them from a run.
+2. Compare `last_precheck_at` with the scout's schedule and the tick you are investigating.
+ A recent recorded `skip` for that tick can explain the missing run row.
+ Stale or absent outcome data leaves the cause unknown. These fields hold only the latest evaluation, not a history of every tick.
+ A recorded `run` or `error` means the pre-check allowed execution; check `scout-runs-list` to confirm a run actually started and read its status.
+3. Read `effective_precheck_query` and `precheck_query_source` to understand the current gate (`config`, `skill_default`, or `off`).
+ An `off` source means the next scheduled run has no pre-check; the current source does not establish what happened on an earlier tick.
+4. Use `scout-config-precheck-test` without a query to test the current gate when needed.
+ Its `would_run` describes a check made now. It cannot establish why an earlier tick has no run row, and it does not update the recorded scheduled outcome.
+ The dry run reads as you, while the scheduled check reads as the scout's acting user.
+ Their access to `system.*` and warehouse tables can differ, so a successful dry run can still fail on schedule (see the access note in [Recipe: test, then save](#recipe-test-then-save)).
+
+The project profile's `scout_fleet` entries also carry `last_precheck_at` and `last_precheck_outcome`, with the current query source in `precheck_source`.
A skip shows that the query found nothing new. It does not show that the scout is healthy.
A gate that is too strict hides real activity, so check the query against the data the scout must see.
diff --git a/products/signals/skills/exploring-scouts/SKILL.md b/products/signals/skills/exploring-scouts/SKILL.md
index 26b067c1a8a1..696858c53f5c 100644
--- a/products/signals/skills/exploring-scouts/SKILL.md
+++ b/products/signals/skills/exploring-scouts/SKILL.md
@@ -106,15 +106,13 @@ Read the result against three cases:
Child dispatch is fire-and-forget, so if workers are saturated or down the children just queue and no run ever materializes, yet `last_run_at` keeps marching forward each tick.
So a recent `last_run_at` means "dispatched this tick," **not** "a run is genuinely happening."
The authoritative liveness signal is the newest actual **run row** in `scout-runs-list`, not the config stamp.
- Cross-check them: if `last_run_at` is fresh (minutes ago) but no run row has appeared for that scout in well over its `run_interval_minutes`, the fleet is **dispatching but not running** — workers backed up / down, or runs stranded — a real reliability problem, not a live scout.
+ Cross-check them: if `last_run_at` is fresh but no run row has appeared for that scout beyond its expected schedule, check the recorded pre-check outcome before diagnosing queued or stranded runs.
Don't report "it's running" off `last_run_at` alone.
- **A pre-check skip also leaves a fresh `last_run_at` with no run row.** When a scout has a pre-check and the query finds nothing new, the scheduled run skips before it creates a run row, a sandbox, or a model call.
- So before you call missing runs a failure, read `effective_precheck_query` and `precheck_query_source` on the config row. When the source is `off`, the pre-check does not explain the gap.
- Otherwise run `scout-config-precheck-test` with only the config `id`: `would_run: false` means the next scheduled run skips too, so the gap is expected.
- The dry run runs as you, but the scheduled check runs as the scout's acting user, so their results can differ when the query reads access-scoped tables.
- A skip shows only that the query found nothing new. It does not show that the scout is healthy.
- The pre-check recipe and its rules are in `authoring-scouts` (its scheduled pre-check reference).
+ **A scheduled pre-check can skip without a run row.** First read `last_precheck_at` and `last_precheck_outcome` on the config, following `authoring-scouts` → `references/scheduled-precheck.md` → "Diagnose a tick with no run row".
+ A recent recorded `skip` for the tick can explain quiet; stale or absent outcome data leaves the cause unknown.
+ Keep `scout-config-precheck-test` for query testing: a check made now cannot establish what happened on an earlier tick, and your access can differ from the scheduled check's acting user.
+ A skip does not establish overall scout health.
A scout that is `enabled: true` but `emit: false` is the most common source of "my scout isn't doing anything" confusion: it _is_ running and reasoning every tick, it just isn't allowed to post reports yet.
Always surface the `emit` posture when reporting on a scout.
diff --git a/products/signals/skills/working-with-scouts/SKILL.md b/products/signals/skills/working-with-scouts/SKILL.md
index ff48ea61eb5c..dc66e1a873a5 100644
--- a/products/signals/skills/working-with-scouts/SKILL.md
+++ b/products/signals/skills/working-with-scouts/SKILL.md
@@ -198,7 +198,10 @@ A fleet left alone drifts; a fleet reviewed occasionally compounds.
Every few weeks (or when someone says "are the scouts even worth it?"), run a calibration pass:
1. **Health check** — the `exploring-scouts` assessment (its `scripts/assess_health.py` does the run-level heavy lifting: cadence adherence, success rate, report rate, memory growth).
- Two blind spots to cover yourself: the script only assesses scouts with runs in the window, so walk the roster for enabled scouts with no recent run rows (often the most broken ones, but a scout whose pre-check skips its ticks writes no run row either, so check `precheck_query_source` and dry-run the pre-check before you call it broken; `exploring-scouts` has the steps); and it counts report writes, not outcomes, so judge signal-to-noise by resolving written reports via `posthog:inbox-reports-list` and reading their statuses.
+ The script only assesses scouts with runs in the window, so walk the roster for enabled scouts with no recent run rows.
+ Follow `authoring-scouts` → `references/scheduled-precheck.md` → "Diagnose a tick with no run row": inspect the recorded `last_precheck_at` and `last_precheck_outcome` before diagnosing missing runs.
+ A recent recorded `skip` can explain quiet without proving overall health; stale or absent outcome data leaves the cause unknown. Keep dry runs for query testing.
+ The script also counts report writes, not outcomes, so judge signal-to-noise by resolving written reports via `posthog:inbox-reports-list` and reading their statuses.
Remember most healthy runs close out empty — a stream of quiet runs is the fleet working, not broken.
2. **Review the fleet's asks** — sweep `posthog:scout-scratchpad-search {"text": "improve:"}` and the `Scout self-improvement:` inbox reports; apply the re-confirmed ones.
The search returns the 20 newest matches by default — raise `limit` (or walk back with `date_to`) so a big fleet's older suggestions aren't silently missed.