:hedgehog: PostHog is the leading platform for building self-driving products. Our developer tools – AI observability, analytics, session replay, flags, experiments, error tracking, logs, and more – capture all the context agents need to diagnose problems, uncover opportunities, and ship fixes. Steer it all from Slack, web, desktop, or the MCP.
Python ab-testingai-analyticsanalyticscdpdata-warehouseexperimentsfeature-flagsjavascriptproduct-analyticspythonreactsession-replaysurveystypescriptweb-analytics
chore(signals): correct scout pre-check observation guidance (#120489)
posthog[bot]db26250e32f2
diff --git a/products/signals/skills/AGENTS.md b/products/signals/skills/AGENTS.mdindex 12b8092b2fed..91cb9136133d 100644--- a/products/signals/skills/AGENTS.md+++ b/products/signals/skills/AGENTS.md@@ -127,8 +127,13 @@ The generalist (`signals-scout-general`) is **report-only** — it authors `Sign- **`references/discovery.md`** — the concrete moves of the bounded discovery pass behind its coverage map.- **`references/new-traffic.md`** — the periodic new-traffic check: a by-source volume step against older weeks, the shapes that separate growth from foreign traffic (a forked client, a copied token, a bot, a runaway loop), and the pinned pre-surge baseline that keeps a slow surge visible.-The entire canonical fleet is on the **report channel** (ported one scout per PR, biggest reach first — see the `scouts-emit-reports` spec). A report-channel scout is report-only — its frontmatter `allowed_tools` lists `emit_report` / `edit_report` — and it carries only its _domain-specific_ report framing **inline in its body** (what's report-shaped for its surface, its `reviewer:<domain>` / `report:<domain>` scratchpad keys, a tailored title example); the channel contract comes from the prompt, so a ported scout bundles **no** `report.md`. The exception is `signals-scout-anomaly-detection`, which keeps a slimmed `references/report-contract.md` for its genuinely scout-specific **notebook write-up + embedded-chart recipe** (it defers the generic contract to the prompt). The legacy signal channel (weak `emit_signal` findings for the pipeline to cluster) still exists in the harness for custom scouts that never opted in via `allowed_tools`, but it is deprecated:-no canonical scout uses it, and the user-facing skills (`authoring-scouts`, `exploring-scouts`) no longer teach it — new scouts always list the report tools.+Most canonical scouts author reports on the **report channel**.+`signals-scout-workflows` writes workflow suggestions and deliberately uses neither signal nor report output; preserve its frontmatter and permissions when editing fleet guidance.+A report-channel scout is report-only: its frontmatter `allowed_tools` lists `emit_report` / `edit_report`, and it carries only its _domain-specific_ report framing **inline in its body** (what's report-shaped for its surface, its `reviewer:<domain>` / `report:<domain>` scratchpad keys, a tailored title example).+The channel contract comes from the prompt, so a report-channel scout bundles **no** `report.md`.+`signals-scout-anomaly-detection` keeps a slimmed `references/report-contract.md` for its scout-specific **notebook write-up + embedded-chart recipe** (it defers the generic contract to the prompt).+The legacy signal channel (weak `emit_signal` findings for the pipeline to cluster) still exists in the harness for custom scouts that never opted in via `allowed_tools`, but it is deprecated.+The user-facing skills (`authoring-scouts`, `exploring-scouts`) no longer teach it; new scouts that author reports list the report tools.The specialists each carry their own domain discriminator + investigation patterns.Most are a single self-contained `SKILL.md`; a few bundle surface-specific references read on demand — `signals-scout-anomaly-detection` (`anomaly-methods.md`, `watchlist-and-memory.md`, `report-contract.md`), `signals-scout-ai-observability` (`lenses.md`), `signals-scout-surveys` (`response-querying.md`), `signals-scout-web-vitals` (`remediation.md`, `onset-correlation.md`), `signals-scout-mcp-tool-calls` (`queries.md`, `metrics-dashboard.md`, `structured-output.schema.json`), `signals-scout-tasks` (`queries.md`), `signals-scout-pr-follow-up` (`sources.md`, `deploy-ladder.md`, `probes.md`), and `signals-scout-unused-features` (`queries.md`); `ls signals-scout-*/references/` is the current list. Treat the generalist as the reference shape. Note that a scout can only read its own bundled files at runtime (each team's `LLMSkill` row carries just that skill's files) — which is why a scout that needs the dedupe/memory conventions in depth bundles its own copy rather than pointing at the generalist's. The report-channel contract is the exception to that copy-it-locally rule: it rides in the harness prompt, not a file, so report scouts share one definition without each bundling a copy.diff --git a/products/signals/skills/authoring-scouts/references/scheduled-precheck.md b/products/signals/skills/authoring-scouts/references/scheduled-precheck.mdindex 581f28ea6fce..43074f977608 100644--- a/products/signals/skills/authoring-scouts/references/scheduled-precheck.md+++ b/products/signals/skills/authoring-scouts/references/scheduled-precheck.md@@ -125,14 +125,25 @@ A manual run has no block, so the body must still work without one.## Diagnose a tick with no run row-A skipped tick writes nothing. The coordinator still advances `last_run_at` (the dispatch stamp), so a fresh `last_run_at` with no matching run row can be a pre-check skip.+A skipped tick creates no run row, sandbox, or model call, but stores its pre-check outcome on the config.Before you treat missing runs as a failure:-1. Read `effective_precheck_query` and `precheck_query_source` in `scout-config-list`.- When the source is `off`, the pre-check does not explain the gap.-2. Run `scout-config-precheck-test` without a query.- `would_run: false` means the next scheduled run skips, so the gap is expected.-3. Remember that the dry run reads as you, and the scheduled check reads as the acting user.+1. Read the config in `scout-config-list`:+ - `last_precheck_at`: the time of the latest scheduled pre-check.+ - `last_precheck_outcome`: `skip` (the query found nothing new), `run` (the query returned rows that allow a run), or `error` (the query failed, so the gate allows a run).+ - `last_precheck_row_count`: how many rows it returned. Zero rows or one false value can both skip, so use the outcome to tell them from a run.+2. Compare `last_precheck_at` with the scout's schedule and the tick you are investigating.+ A recent recorded `skip` for that tick can explain the missing run row.+ Stale or absent outcome data leaves the cause unknown. These fields hold only the latest evaluation, not a history of every tick.+ A recorded `run` or `error` means the pre-check allowed execution; check `scout-runs-list` to confirm a run actually started and read its status.+3. Read `effective_precheck_query` and `precheck_query_source` to understand the current gate (`config`, `skill_default`, or `off`).+ An `off` source means the next scheduled run has no pre-check; the current source does not establish what happened on an earlier tick.+4. Use `scout-config-precheck-test` without a query to test the current gate when needed.+ Its `would_run` describes a check made now. It cannot establish why an earlier tick has no run row, and it does not update the recorded scheduled outcome.+ The dry run reads as you, while the scheduled check reads as the scout's acting user.+ Their access to `system.*` and warehouse tables can differ, so a successful dry run can still fail on schedule (see the access note in [Recipe: test, then save](#recipe-test-then-save)).++The project profile's `scout_fleet` entries also carry `last_precheck_at` and `last_precheck_outcome`, with the current query source in `precheck_source`.A skip shows that the query found nothing new. It does not show that the scout is healthy.A gate that is too strict hides real activity, so check the query against the data the scout must see.diff --git a/products/signals/skills/exploring-scouts/SKILL.md b/products/signals/skills/exploring-scouts/SKILL.mdindex 26b067c1a8a1..696858c53f5c 100644--- a/products/signals/skills/exploring-scouts/SKILL.md+++ b/products/signals/skills/exploring-scouts/SKILL.md@@ -106,15 +106,13 @@ Read the result against three cases:Child dispatch is fire-and-forget, so if workers are saturated or down the children just queue and no run ever materializes, yet `last_run_at` keeps marching forward each tick.So a recent `last_run_at` means "dispatched this tick," **not** "a run is genuinely happening."The authoritative liveness signal is the newest actual **run row** in `scout-runs-list`, not the config stamp.- Cross-check them: if `last_run_at` is fresh (minutes ago) but no run row has appeared for that scout in well over its `run_interval_minutes`, the fleet is **dispatching but not running** — workers backed up / down, or runs stranded — a real reliability problem, not a live scout.+ Cross-check them: if `last_run_at` is fresh but no run row has appeared for that scout beyond its expected schedule, check the recorded pre-check outcome before diagnosing queued or stranded runs.Don't report "it's running" off `last_run_at` alone.- **A pre-check skip also leaves a fresh `last_run_at` with no run row.** When a scout has a pre-check and the query finds nothing new, the scheduled run skips before it creates a run row, a sandbox, or a model call.- So before you call missing runs a failure, read `effective_precheck_query` and `precheck_query_source` on the config row. When the source is `off`, the pre-check does not explain the gap.- Otherwise run `scout-config-precheck-test` with only the config `id`: `would_run: false` means the next scheduled run skips too, so the gap is expected.- The dry run runs as you, but the scheduled check runs as the scout's acting user, so their results can differ when the query reads access-scoped tables.- A skip shows only that the query found nothing new. It does not show that the scout is healthy.- The pre-check recipe and its rules are in `authoring-scouts` (its scheduled pre-check reference).+ **A scheduled pre-check can skip without a run row.** First read `last_precheck_at` and `last_precheck_outcome` on the config, following `authoring-scouts` → `references/scheduled-precheck.md` → "Diagnose a tick with no run row".+ A recent recorded `skip` for the tick can explain quiet; stale or absent outcome data leaves the cause unknown.+ Keep `scout-config-precheck-test` for query testing: a check made now cannot establish what happened on an earlier tick, and your access can differ from the scheduled check's acting user.+ A skip does not establish overall scout health.A scout that is `enabled: true` but `emit: false` is the most common source of "my scout isn't doing anything" confusion: it _is_ running and reasoning every tick, it just isn't allowed to post reports yet.Always surface the `emit` posture when reporting on a scout.diff --git a/products/signals/skills/working-with-scouts/SKILL.md b/products/signals/skills/working-with-scouts/SKILL.mdindex ff48ea61eb5c..dc66e1a873a5 100644--- a/products/signals/skills/working-with-scouts/SKILL.md+++ b/products/signals/skills/working-with-scouts/SKILL.md@@ -198,7 +198,10 @@ A fleet left alone drifts; a fleet reviewed occasionally compounds.Every few weeks (or when someone says "are the scouts even worth it?"), run a calibration pass:1. **Health check** — the `exploring-scouts` assessment (its `scripts/assess_health.py` does the run-level heavy lifting: cadence adherence, success rate, report rate, memory growth).- Two blind spots to cover yourself: the script only assesses scouts with runs in the window, so walk the roster for enabled scouts with no recent run rows (often the most broken ones, but a scout whose pre-check skips its ticks writes no run row either, so check `precheck_query_source` and dry-run the pre-check before you call it broken; `exploring-scouts` has the steps); and it counts report writes, not outcomes, so judge signal-to-noise by resolving written reports via `posthog:inbox-reports-list` and reading their statuses.+ The script only assesses scouts with runs in the window, so walk the roster for enabled scouts with no recent run rows.+ Follow `authoring-scouts` → `references/scheduled-precheck.md` → "Diagnose a tick with no run row": inspect the recorded `last_precheck_at` and `last_precheck_outcome` before diagnosing missing runs.+ A recent recorded `skip` can explain quiet without proving overall health; stale or absent outcome data leaves the cause unknown. Keep dry runs for query testing.+ The script also counts report writes, not outcomes, so judge signal-to-noise by resolving written reports via `posthog:inbox-reports-list` and reading their statuses.Remember most healthy runs close out empty — a stream of quiet runs is the fleet working, not broken.2. **Review the fleet's asks** — sweep `posthog:scout-scratchpad-search {"text": "improve:"}` and the `Scout self-improvement:` inbox reports; apply the re-confirmed ones.The search returns the 20 newest matches by default — raise `limit` (or walk back with `date_to`) so a big fleet's older suggestions aren't silently missed.