Skip to content

:hedgehog: PostHog is the leading platform for building self-driving products. Our developer tools – AI observability, analytics, session replay, flags, experiments, error tracking, logs, and more – capture all the context agents need to diagnose problems, uncover opportunities, and ship fixes. Steer it all from Slack, web, desktop, or the MCP.

12.5 KB
Raw Blame

Engineering analytics

Owner: team-devex. Engineering contract: SPEC.md.

The one-sentence version

It's product analytics, but the "users" are pull requests and the "events" are what happens to them on the way to production.

Everything else is mechanics.

The analogy

If you understand PostHog product analytics, you understand this. Map it one-to-one:

Product analytics Engineering analytics
A person does things in your app A PR moves through your pipeline
Events: pageview, signup, purchase Events: opened, ready_for_review, merged
"How long from signup to purchase?" "How long from opened to merged?"
"What % of signups convert?" (funnel) "What % of PRs make it to prod?" (funnel)
"Is conversion getting better?" (trend) "Is CI getting faster?" (trend)
Segment by country, plan, browser Segment by repo, workflow, file path

Same product, different noun.

The shape of the thing

A PR's life, as a stream of timestamped events:

opened ──> ready_for_review ──> ci_passed ──> approved ──> merged ──> deployed
  │              │                  │            │           │           │
t=0          t=2h              t=2h15m        t=4h        t=4h30m      t=5h

Engineering analytics measures every gap:
  opened→ready     = how long it sat as a draft
  ready→merged     = THE NORTH STAR (shorten this)
  any→ci_passed    = how long CI took to go green
  merged→deployed  = ship latency

Then averages each gap across all PRs, and trends it over weeks.
DevEx's job: watch those gaps shrink.

The product is: measure every gap in a PR's life, average it, and watch it shrink.

Why this exists

The middle of PostHog's AI-to-prod loop (CI / review / merge / deploy) is invisible to both ends. This product fills it:

PostHog Desktop              engineering_analytics             PostHog product analytics
(generates code,      →   (CI / review / merge / deploy)  →    (events, errors, flags,
 opens PRs)                                                     surveys, replays)
        ↑                                                              │
        └───────────  Signals feed back to PostHog Desktop  ───────────┘

Two consumer classes, one tool set: engineers driving an agent (Claude Code, Cursor, PostHog AI) over their monorepo (we dogfood on PostHog/posthog), and PostHog Desktop calling the same tools autonomously on its own PRs. Return shapes are typed contracts whose caveats ride in honest field names (open_to_merge_seconds, never cycle_time): legible to an agent, never free-form prose it might paraphrase wrong.

The system

graph LR
    subgraph "Data in"
        WH[("warehouse: GitHub source<br/>pull_requests / workflow_runs<br/>workflow_jobs / team_members<br/>issue_events")]
        LOGS[("Logs: github-ci-logs<br/>thinned CI failure lines")]
        TRACES[("Traces: per-test CI spans<br/>emitted by Backend and Frontend CI")]
    end
    JL["Temporal job-logs pipeline<br/>fetches + thins failed jobs' logs"] --> LOGS
    CUR["curated HogQL builders<br/>run privately, never global views"]
    WH --> CUR
    LOGS --> CUR
    TRACES --> CUR
    EP["named typed DRF endpoints"]
    CUR --> EP
    EP --> MCP["MCP tools<br/>enabled subset in mcp/tools.yaml"]
    EP --> UI["UI scenes"]
    CUR --> WV["managed warehouse views<br/>job_costs / ci_job_history / ci_failures / pr_friction<br/>ci_runs / ci_jobs (stored)<br/>for insights + SQL"]
  • Job-level CI: per-job duration, queue time, runner tier, and dollar cost ride workflow_jobs (webhook stream plus bounded backfill).
  • One write surface: test quarantine. The quarantine/request endpoint files a tracking issue plus a PR against the repo's checked-in .test_quarantine.json through the team's GitHub App; it is reachable via the API and MCP only, with no in-app UI. Everything else is read-only.
  • The Tests tab is the Trunk quarantine debt scoreboard: every test Trunk currently quarantines, attributed to the team that owns its file in the repository today, aged against a TTL, and rolled up per team. A quarantine masks a failure without fixing it, so the board is the standing work queue of masked tests. The flaky-test queue itself (failures plus same-commit recovery evidence from the pytest and Jest suites, counted per CI run, ranked by blast radius) remains an endpoint and MCP tool; the weekly Slack report requests each runner separately and shows up to 10 pytest tests followed by up to 10 Jest tests; it is informational, so suppressed tests stay listed with their quarantine state. With the FLAKY_REPORT_TEAM_CHANNELS repository variable set to true, each owning team also gets its rows in its own notifications channel, linked from the report thread; a week without Trunk state keeps those rows in the thread. Logs-based rescue counts and links remain pytest-only.
  • Access control: per-user warehouse RBAC at the source resolver, engineering_analytics:read scopes on tools, feature-flag gated. Background work (view sync, the test census, CI failure log collection) runs only for teams that enabled the product, which happens when a user opens it with a GitHub source connected.

The goal: CI Signals for PostHog Desktop

Backend pytest retries retain their failed-attempt diagnostics even when the job passes. The job-log collector uses recovered-test spans to include those successful jobs, and keeps their original conclusion. See Backend test retries for the retry budget and reporting path.

Valuable CI conditions ("this check is flaky", "master went red at SHA X", "this PR is wedged on a failing required check") become Signals: grouped, researched against the repository, and handed to PostHog Desktop for autonomous remediation. Detection is defined once in logic/ over the read layer, so the emitter and the MCP tools share one definition. Shortening ready-for-review-to-merge is the headline metric this serves.

The data boundary

Question Substrate
CI and job durations, queue time, cost, failure logs, flaky and broken tests Warehouse + Logs + Traces
Open to merge time (coarse: open_to_merge_seconds) PR snapshot
Ready to merge time (ready_to_merge_seconds), draft/ready transitions Warehouse github_issue_events (bounded window, grows forward)
Approvals, review submissions Warehouse github_reviews (optional; the author page reads approvals)
Deploys and DORA Warehouse github_deployments + github_deployment_statuses

The warehouse snapshots overwrite state on update, so transition timing is unrecoverable from them. Immutable lifecycle events are the only thing the deferred events destination is for. Deploys left that deferral: unlike the snapshots, github_deployment_statuses is an append-oriented, webhook-fed status history (one row per transition), so DORA reads don't need the events destination. When a deployment's merge SHA is absent from the PR snapshot, the deploy metrics reuse workflow-run commit_pr_number attribution for the exact deployed commit on the same repository's default branch. The candidate PR must have merged into that branch and must carry no merge commit of its own. One that names some other commit is evidence the suffix is wrong, not evidence the deploy carries that PR. That drops a cherry-pick: a commit forward-ported onto the default branch keeps the original subject line, so its (#N) suffix still names the PR it came from, which landed elsewhere. This fallback depends on the merge-message PR suffix and synced workflow history; unresolved deployments remain unattributed. Change failure rate and time-to-restore still lack an incident link, so they ship as honest deploy-status proxies — the caveat rides in the field docs (failed_deployment_share, median_failed_deploy_to_next_success_seconds), the aggregate-endpoint pattern; a typed metric_quality field is reserved for deep tools like pr_lifecycle.

Locked decisions

Change one only in a separate PR with a written reason. Engineering-level decisions live in SPEC.md → Locked decisions.

  • Two motivations: (A) DevEx dogfood, (B) close the dark middle of PostHog's AI-to-prod loop. Every design decision serves both.
  • Unit of value = the open PR. Phase model: draft (low rigor) vs ready-for-review (high stakes).
  • North star: actionable CI Signals for PostHog Desktop. Ready-for-review-to-merge time is the headline metric, not the end in itself.
  • Two first-class surfaces, one endpoint set: the in-app UI and MCP tools. Named typed endpoints run the curated read layer privately (no global HogQL views, core imports only the viewset); keep mcp/tools.yaml current whenever endpoints change.
  • One sanctioned write: the test-health sidecar (quarantine, as an issue plus PR through the team's GitHub App). The write now lives behind the API and MCP tool only; the Tests tab became the Trunk quarantine debt scoreboard because Trunk's auto-quarantine outran the file-based flow as the thing teams need to see. No saved views or stateful filters; persisted surfaces are a separate decision.
  • Data path: HogQL over the warehouse, plus reads from Logs and Traces. PR lifecycle event ingestion deferred. Product Postgres DB stays empty.
  • No developer performance rankings, ever. The surveillance risk is a performance scorecard: output or speed turned into an ordered population of people. An engineer inspecting their own friction is not that risk, and neither is a deliberate comparison with a colleague they already know. So author figures may be read one author at a time, and compared on purpose with another author, a GitHub team, or the repository. The product must never rank, sort, score or enumerate authors by delivery performance. Friction is not performance: scoring and ordering the friction developers meet (red CI they did not cause, merge-queue kickouts, review waits, rework) finds the people the dev loop costs most, and a friction figure never counts how much or how fast someone ships. The author page (one author's PRs, CI cost, delivery timing and lead time against the repository) is reached from PR-row author links, and a colleague's page may be opened directly for such a comparison. The author page and the pull request page also set the ready-to-merge time next to the author's own team, as an aggregate. The team page renders the same delivery figures for one GitHub team, as an aggregate and never a per-member figure. It leaves out the author page's per-pull-request day view, which is too long to read at team size.
  • Bots and drafts excluded by default in throughput / cycle-time reads; bot detection = handle.endswith("[bot]") OR handle in KNOWN_BOT_HANDLES.
  • Author identity = Author{handle, display_name, avatar_url, is_bot}. No PostHog-user mapping.
  • Time to merge = open_to_merge_seconds = merged_at - created_at: coarse, and named so. ready_to_merge_seconds is the precise companion (last observed ready-for-review to merge, from github_issue_events); NULL means "not observed", never zero.
  • The GitHub reviews endpoint syncs; the author page is its first reader (the approval split and the review states of a PR timeline).

Glossary

Term Definition
Good / bad friction Checks that catch real problems (keep, optimize) vs checks engineers ignore (remove)
Wedge End-to-end code visibility, served via MCP to PostHog Desktop and engineers driving agents
The dark middle The CI / review / merge / deploy steps between PostHog Desktop and PostHog product analytics