Skip to content

πŸͺ’ Open source agent evals & observability: Trace, evaluate, and improve LLM applications with one open platform.

feat(seeder): add incident-session scenario for the session timeline (#18061)

Nikita Kabardined60d213a172
diff --git a/.agents/skills/seed-test-data/SKILL.md b/.agents/skills/seed-test-data/SKILL.md
index d37eddda1b..7ecfa58a12 100644
--- a/.agents/skills/seed-test-data/SKILL.md
+++ b/.agents/skills/seed-test-data/SKILL.md
@@ -37,6 +37,7 @@ this.
| A row carrying every annotation at once, to judge whether the timeline is overloaded | `pnpm run seed -- timeline-annotated --v4` (ONE trace, 12 observations: one/two/four scores β€” the last collapsing into `+1` β€” one and twelve comments, a streaming first-token mark, costs and durations spread so the heat map paints some rows, plain rows beside them for contrast, and all three label placements on screen together) |
| A multi-HOUR agent run (wall clock dominated by waiting, as production ones are) | `pnpm run seed -- agent-timeline --turns 120 --turn-gap-ms 60000 --v4` (~2.5h; the gap is jittered up to 2x, so the work lands at a few percent of the trace instead of packing every span into a few seconds) |
| A demo-grade, real-looking agent trace (videos, screenshots, docs) | `pnpm run seed -- support-agent --v4 --id-prefix <hex>` (one fixed, fully handcrafted support-copilot refund run: guardrails, parallel context fan-out, 3-turn ReAct loop with real payloads/costs; deterministic β€” reseed with a FRESH prefix for a clean take; the prefix is the trace id, so a hex prefix reads like production) |
+| A demo-grade, real-looking multi-user SESSION (session timeline videos, screenshots, docs) | `pnpm run seed -- incident-session --id-prefix inc-4471-checkout-latency` (one fixed, handcrafted incident channel: 7 turns, 4 named users, accumulating chat history with reasoning and markdown, tool calls paired to TOOL rows plus two untraced ones, a WARNING and an ERROR tool, a nested sub-agent, a refusal, typed session scores and comments; v4 on by default; the prefix is the session id) |
| A plain trace with no agentic types (collapsed-by-default graph panel) | add `--plain` to `trace-tree` (SPAN/GENERATION/EVENT only) |
| A trace with MORE observations than the detail view loads (v4 caps the tree at 10k, `startTime ASC` β€” the chronological tail is missing) | `pnpm run seed -- trace-tree --observations 12000 --stride-ms 10 --v4` (`--stride-ms` starts each observation index Γ— N ms in, so start times are unique and strictly increasing: observation index < 10000 loads, `-obs-10000` and up fall past the cap. Without it thousands of rows share one millisecond and the boundary is arbitrary. Re-runs that CHANGE timing flags need a fresh `--id-prefix`: `start_time` is an events ORDER BY key, so both versions persist in the same trace) |
| An extremely DEEP single-chain trace (tree depth = observation count; layout stress) | `pnpm run seed -- deep-chain --v4` (1401 sequential generations, each the sole child of the previous β€” the mis-parented-instrumentation shape from LFE-10959 that collapses tree/timeline layouts; `--observations N` to change depth) |
diff --git a/packages/shared/scripts/seeder/AGENTS.md b/packages/shared/scripts/seeder/AGENTS.md
index fb3db214a8..4afbe230bd 100644
--- a/packages/shared/scripts/seeder/AGENTS.md
+++ b/packages/shared/scripts/seeder/AGENTS.md
@@ -20,6 +20,7 @@ pnpm run seed -- timeline-shapes --v4 # a dozen SMALL traces, one per timeline
pnpm run seed -- timeline-shapes --shape retry-backoff --v4 # just one of them
pnpm run seed -- timeline-annotated --v4 # ONE trace with every row annotation at once (scores incl. +N overflow, 1 and 12 comments, costs, heat map, first-token mark) β€” for judging visual load
pnpm run seed -- support-agent --v4 --id-prefix <hex> # demo-grade handcrafted support-copilot run (videos/screenshots)
+pnpm run seed -- incident-session --id-prefix inc-4471-checkout-latency # demo-grade multi-user v4 SESSION: 7 turns, 4 users, tool calls, nested sub-agent, refusal, typed scores + comments (session timeline videos/screenshots)
pnpm run seed -- long-session --traces 300 --observations-per-trace 8
pnpm run seed -- session-shapes --shape all # chat / coding-agent / mixed / media v4 sessions
pnpm run seed -- session-shapes --shape media # messages carrying @@@langfuseMedia:...@@@ refs (needs MinIO)
diff --git a/packages/shared/scripts/seeder/README.md b/packages/shared/scripts/seeder/README.md
index 4e2a455c0c..dbaeeb4cf3 100644
--- a/packages/shared/scripts/seeder/README.md
+++ b/packages/shared/scripts/seeder/README.md
@@ -52,6 +52,7 @@ programmatic calls from the dx seed chain.
| `custom-models` | project-level model definitions (one tiered with a condition-gated second tier and a usage type priced at 0, one single-tier) plus a trace whose generations link to them, and one generation whose model matches no definition so its badge opens the create dialog | β€” |
| `evaluator-gallery` | project-owned code evaluators for gallery pagination and infinite-scroll testing; reconciles deterministic names through the seeded public API key so local and PR preview environments are supported | `--count` |
| `support-agent` | one demo-grade, fully handcrafted trace: a customer-support copilot resolving a duplicate-charge refund β€” input guardrail β†’ intent classification β†’ parallel context fan-out (CRM/billing/tickets) β†’ 3-turn ReAct loop (llm.chat + Stripe tools) β†’ drafted reply β†’ output guardrail β†’ send | β€” |
+| `incident-session` | one demo-grade, fully handcrafted multi-user v4 session: an on-call copilot and four engineers work a checkout-latency incident over seven turns β€” parallel, untraced and failing tool calls, a nested sub-agent, reasoning, markdown answers, a guardrail-flagged refusal, typed session scores and comments | `--v4` (default true) |
| `timeline-shapes` | a dozen SMALL traces (4-25 observations each), one per timeline morphology: rag answer, streamed chat, parallel fan-out, retry backoff with widening gaps, a 13-minute wait on a human, one slow tool dwarfing everything, an error cascade with failover, in-flight spans, zero-duration checkpoints, a ten-level ladder, 24 flat siblings, and a three-turn agent loop | `--shape all\|rag-answer\|streaming-chat\|parallel-fanout\|retry-backoff\|waiting-on-approval\|slow-tool\|error-cascade\|still-running\|checkpoint-marks\|deep-ladder\|flat-siblings\|mixed-loop` |
Common flags: `--project` (defaults to the seeded example project),
diff --git a/packages/shared/scripts/seeder/scenarios/incident-session.ts b/packages/shared/scripts/seeder/scenarios/incident-session.ts
new file mode 100644
index 0000000000..56a61a9eb6
--- /dev/null
+++ b/packages/shared/scripts/seeder/scenarios/incident-session.ts
@@ -0,0 +1,1523 @@
+import { prisma } from "../../../src/db";
+import {
+ convertCallsToArrays,
+ convertDefinitionsToMap,
+ createEventsCh,
+ createObservation,
+ createObservationsCh,
+ createScoresCh,
+ createSessionScore,
+ createTrace,
+ createTraceScore,
+ createTracesCh,
+ extractToolsFromObservation,
+ EventRecordInsertType,
+ ObservationRecordInsertType,
+ ScoreRecordInsertType,
+ TraceRecordInsertType,
+} from "../../../src/server";
+import { ObservationType } from "../../../src/domain";
+import { observationToEvent, traceToEvent } from "./event-mirror";
+import { utcDayStartMs } from "./rng";
+import {
+ chunk,
+ ScenarioContext,
+ ScenarioDefinition,
+ SeedError,
+ SeedSummary,
+} from "./types";
+import { countRows, sessionLink, traceLink } from "./verify";
+
+/**
+ * A demo-grade, fully handcrafted multi-user SESSION for the session timeline:
+ * an on-call copilot working a checkout-latency incident with four engineers
+ * in a Slack incident channel, seven turns over ~56 minutes.
+ *
+ * Where `support-agent` is one trace, this is the session-level counterpart:
+ * every turn carries real-looking content and together they exercise what the
+ * session view renders β€”
+ *
+ * - four users (header user chips + overflow) and named senders (`name` on
+ * each user message)
+ * - OpenAI chat history that accumulates across turns, with reasoning parts,
+ * markdown answers (tables, diff blocks, lists) and one refusal
+ * - tool calls matched to TOOL observations via `metadata.toolCallId`,
+ * parallel calls, and two calls the app ran without tracing (the UI rolls
+ * those up under their generation)
+ * - a TOOL that delegates to a nested sub-agent (AGENT β†’ GENERATION,
+ * RETRIEVER, TOOLs) β€” the "tool row with nested observations" shape
+ * - WARNING and ERROR rows with status messages, guardrails, an EVENT
+ * - three models with usage/cost including `total`, short follow-ups next to
+ * 5-20 minute idle gaps
+ * - typed session scores (numeric/categorical/boolean; API, EVAL and
+ * ANNOTATION sources, comments, metadata; the ANNOTATION one backed by a
+ * project score config), trace feedback scores, and session + trace
+ * comments
+ *
+ * Deterministic: no randomness; ids derive from --id-prefix (the prefix IS the
+ * session id) and every timestamp is a fixed offset from 22:00 UTC of the
+ * previous UTC day, so same-day re-runs overwrite in place.
+ */
+
+const MODEL_PRICES = {
+ "gpt-5.4": { input: 1.25e-6, output: 1e-5 },
+ "claude-sonnet-4-5": { input: 3e-6, output: 1.5e-5 },
+ "claude-haiku-4-5": { input: 1e-6, output: 5e-6 },
+} as const;
+
+type ModelName = keyof typeof MODEL_PRICES;
+
+type ToolCallSpec = {
+ id: string;
+ name: string;
+ args: Record<string, unknown>;
+ /** objects are stored pretty-printed (TOOL rows render I/O verbatim), strings as-is */
+ result: unknown;
+ ms: number;
+ /** false: the app ran the tool without tracing it, so there is no TOOL observation */
+ traced?: boolean;
+ level?: "WARNING" | "ERROR";
+ statusMessage?: string;
+ /** a delegated sub-agent run nested under this TOOL observation */
+ subagent?: SubagentSpec;
+};
+
+type LlmStep = {
+ kind: "llm";
+ name: string;
+ model: ModelName;
+ ms: number;
+ ttft: number;
+ temperature?: number;
+ reasoning?: string;
+ text?: string;
+ refusal?: string;
+ toolCalls?: ToolCallSpec[];
+};
+
+type SpanStep = {
+ kind: "guardrail" | "retriever";
+ name: string;
+ ms: number;
+ input: unknown;
+ output: unknown;
+ level?: "WARNING";
+ statusMessage?: string;
+};
+
+type Step =
+ | LlmStep
+ | SpanStep
+ | { kind: "wait"; ms: number }
+ | { kind: "event"; name: string; metadata: Record<string, string> };
+
+type SubagentSpec = {
+ name: string;
+ system: string;
+ task: string;
+ steps: Step[];
+};
+
+type TurnSpec = {
+ /** trace name β€” shown in bold per turn in the session sidebar */
+ name: string;
+ /** offset from the session start (22:00 UTC) */
+ atMs: number;
+ userId: string;
+ /** chat display name, carried as `name` on the user message */
+ sender: string;
+ message: string;
+ tags: string[];
+ bookmarked?: boolean;
+ steps: Step[];
+};
+
+const USERS = {
+ maya: { id: "maya.okafor", sender: "maya" },
+ jonas: { id: "jonas.weber", sender: "jonas" },
+ priya: { id: "priya.raman", sender: "priya" },
+ alex: { id: "alex.kim", sender: "alex" },
+} as const;
+
+const SEVERITY_CONFIG = {
+ name: "incident-severity",
+ categories: [
+ { label: "SEV-1", value: 1 },
+ { label: "SEV-2", value: 2 },
+ { label: "SEV-3", value: 3 },
+ ],
+ description:
+ "Incident severity assigned in review; SEV-1 is the most severe.",
+};
+
+const SESSION_METADATA = {
+ scenario: "incident-session",
+ incident: "INC-4471",
+ channel: "#inc-4471-checkout-latency",
+ workspace: "northwind",
+};
+
+const COPILOT_SYSTEM_PROMPT = [
+ "You are Northwind's on-call copilot in Slack incident channels.",
+ "Tools: query_metrics, list_deploys, search_logs, get_pull_request_diff, get_rollout_status, delegate_task, create_issue, resolve_incident, publish_status_update.",
+ "Cite evidence (numbers, deploy ids, UTC times). Keep answers short and scannable.",
+ "Never make destructive production changes from chat β€” propose a change request instead.",
+].join("\n");
+
+const MINUTE = 60_000;
+
+const TURNS: TurnSpec[] = [
+ {
+ name: "triage-what-changed",
+ atMs: 0,
+ userId: USERS.maya.id,
+ sender: USERS.maya.sender,
+ message:
+ "@copilot checkout p95 just jumped from ~350ms to 4s+ and the error rate is climbing. What changed in the last hour?",
+ tags: ["triage"],
+ steps: [
+ {
+ kind: "llm",
+ name: "plan",
+ model: "gpt-5.4",
+ ms: 1830,
+ ttft: 610,
+ reasoning:
+ "Latency and errors rose together on checkout-api. Pin the start time from the metric, list deploys across checkout's dependencies in the last hour, and pull error logs to see which dependency is failing β€” all three in parallel.",
+ toolCalls: [
+ {
+ id: "call_mQ8e2vLr4TzA",
+ name: "query_metrics",
+ args: {
+ query: "p95:trace.http.request.duration{service:checkout-api}",
+ from: "now-60m",
+ rollup: "1m",
+ },
+ ms: 910,
+ result: {
+ metric: "p95:trace.http.request.duration{service:checkout-api}",
+ unit: "ms",
+ baseline_p95: 348,
+ current_p95: 4210,
+ change_point: "21:52 UTC",
+ error_rate: { baseline: "0.2%", current: "6.1%" },
+ series_tail: [352, 349, 361, 1840, 3920, 4105, 4210],
+ },
+ },
+ {
+ id: "call_d3Kp7wNc9HsB",
+ name: "list_deploys",
+ args: {
+ services: ["checkout-api", "payments-service", "checkout-web"],
+ since: "now-60m",
+ },
+ ms: 580,
+ result: {
+ deploys: [
+ {
+ service: "payments-service",
+ id: "#4812",
+ at: "21:51 UTC",
+ author: "jonas.weber",
+ title: "Tune DB connection pool for pgbouncer (max 50 β†’ 10)",
+ status: "success",
+ },
+ {
+ service: "checkout-web",
+ id: "#9921",
+ at: "21:34 UTC",
+ author: "lena.park",
+ title: "Update promo banner copy",
+ status: "success",
+ },
+ ],
+ },
+ },
+ {
+ id: "call_l5Xz1qRt6YmC",
+ name: "search_logs",
+ args: {
+ query: "service:checkout-api status:error",
+ from: "now-30m",
+ limit: 500,
+ },
+ ms: 2710,
+ level: "WARNING",
+ statusMessage:
+ "Result truncated: returned the first 500 of 12,904 matching lines",
+ result: {
+ matched: 12904,
+ returned: 500,
+ top_errors: [
+ {
+ count: 11873,
+ message:
+ "TimeoutError: timed out acquiring connection from pool after 3000ms",
+ origin: "payments-service ← checkout-api POST /v1/charge",
+ },
+ {
+ count: 1031,
+ message: "UpstreamError: payments-service responded 503",
+ origin: "checkout-api",
+ },
+ ],
+ first_seen: "21:52:07 UTC",
+ },
+ },
+ ],
+ },
+ {
+ kind: "llm",
+ name: "respond",
+ model: "gpt-5.4",
+ ms: 9450,
+ ttft: 820,
+ text: [
+ "**Most likely cause: payments-service deploy #4812 (21:51 UTC).**",
+ "",
+ "| Time (UTC) | Change | Service |",
+ "| --- | --- | --- |",
+ "| 21:34 | #9921 promo banner copy | checkout-web |",
+ "| **21:51** | **#4812 DB pool max 50 β†’ 10** | **payments-service** |",
+ "| 21:52 | p95 348 ms β†’ 4.2 s, errors 0.2% β†’ 6.1% | checkout-api |",
+ "",
+ "- 92% of errors (11,873) are `TimeoutError: timed out acquiring connection from pool after 3000ms`, all on calls into payments-service.",
+ "- The spike starts one minute after #4812 rolled out; #9921 only touched banner copy.",
+ "",
+ "**Suggested next step:** roll back #4812 (owner: @jonas.weber). Want me to pull the diff first?",
+ ].join("\n"),
+ },
+ ],
+ },
+ {
+ name: "triage-show-diff",
+ atMs: 2 * MINUTE + 10_000,
+ userId: USERS.maya.id,
+ sender: USERS.maya.sender,
+ message: "Yes β€” show me the diff for #4812.",
+ tags: ["triage"],
+ steps: [
+ {
+ kind: "llm",
+ name: "plan",
+ model: "gpt-5.4",
+ ms: 940,
+ ttft: 390,
+ toolCalls: [
+ {
+ id: "call_g7Hd2kPq1XwD",
+ name: "get_pull_request_diff",
+ args: { repo: "northwind/payments-service", number: 4812 },
+ ms: 460,
+ result: [
+ "diff --git a/config/database.yml b/config/database.yml",
+ "@@ -12,7 +12,7 @@ production:",
+ " adapter: postgresql",
+ ' host: <%= ENV["PGBOUNCER_HOST"] %>',
+ "- pool: 50",
+ "+ pool: 10",
+ " checkout_timeout: 3",
+ " prepared_statements: false",
+ ].join("\n"),
+ },
+ ],
+ },
+ {
+ kind: "llm",
+ name: "respond",
+ model: "gpt-5.4",
+ ms: 7120,
+ ttft: 700,
+ text: [
+ "It's a one-line change in `config/database.yml`:",
+ "",
+ "```diff",
+ "- pool: 50",
+ "+ pool: 10",
+ " checkout_timeout: 3",
+ "```",
+ "",
+ "At peak each `POST /v1/charge` fans out to ~24 concurrent queries across 4 Puma workers, so a pool of 10 saturates immediately and requests wait the full `checkout_timeout` (3 s) β€” exactly the 3000 ms timeouts in the logs.",
+ "",
+ "The PR says the value was meant for pgbouncer transaction pooling, which isn't live in production yet.",
+ ].join("\n"),
+ },
+ ],
+ },
+ {
+ name: "mitigate-watch-rollback",
+ atMs: 9 * MINUTE + 5_000,
+ userId: USERS.jonas.id,
+ sender: USERS.jonas.sender,
+ message:
+ "Payments here πŸ‘‹ rolling back #4812 now. Can you watch the rollout and tell me when error rates recover?",
+ tags: ["mitigation"],
+ bookmarked: true,
+ steps: [
+ {
+ kind: "llm",
+ name: "plan",
+ model: "gpt-5.4",
+ ms: 1120,
+ ttft: 450,
+ reasoning:
+ "Watch the payments-service rollout first; recovery metrics only mean something once all pods run the previous image.",
+ toolCalls: [
+ {
+ id: "call_r2Tn8vBx5KqE",
+ name: "get_rollout_status",
+ args: { namespace: "payments", deployment: "payments-service" },
+ ms: 640,
+ level: "ERROR",
+ statusMessage:
+ "Rollout paused: 1/6 pods in CrashLoopBackOff (payments-service-7c9f4d-x2kqp)",
+ result: {
+ status: "paused",
+ ready: "5/6",
+ revision: 212,
+ image: "payments-service:2026.09.28-4811",
+ unhealthy_pods: [
+ {
+ name: "payments-service-7c9f4d-x2kqp",
+ state: "CrashLoopBackOff",
+ node: "ip-10-2-7-19",
+ last_log:
+ "readiness probe failed: dial tcp 10.2.7.19:8080: connect: connection refused",
+ },
+ ],
+ },
+ },
+ ],
+ },
+ {
+ kind: "llm",
+ name: "plan",
+ model: "gpt-5.4",
+ ms: 1480,
+ ttft: 520,
+ reasoning:
+ "5/6 pods are healthy on the old image. The failing pod sits on ip-10-2-7-19, which the cluster autoscaler is draining β€” it will be rescheduled, so this is not a bad rollback. Wait a minute, then re-check the rollout and error rate together.",
+ text: "One pod is crash-looping on a node that's being drained β€” checking again in 60s.",
+ toolCalls: [],
+ },
+ { kind: "wait", ms: 60_000 },
+ {
+ kind: "llm",
+ name: "plan",
+ model: "gpt-5.4",
+ ms: 860,
+ ttft: 380,
+ toolCalls: [
+ {
+ id: "call_s9Jw4mLc2PzF",
+ name: "get_rollout_status",
+ args: { namespace: "payments", deployment: "payments-service" },
+ ms: 590,
+ result: {
+ status: "complete",
+ ready: "6/6",
+ revision: 212,
+ image: "payments-service:2026.09.28-4811",
+ rescheduled: [
+ {
+ pod: "payments-service-7c9f4d-x2kqp",
+ from: "ip-10-2-7-19",
+ to: "ip-10-2-4-88",
+ },
+ ],
+ },
+ },
+ {
+ id: "call_q4Vb7nHs8RtG",
+ name: "query_metrics",
+ args: {
+ query:
+ "p95:trace.http.request.duration{service:checkout-api}, error_rate{service:checkout-api}",
+ from: "now-10m",
+ rollup: "1m",
+ },
+ ms: 820,
+ result: {
+ p95_ms: { peak: 4210, now: 610, trend: "falling" },
+ error_rate: { peak: "6.1%", now: "0.4%", trend: "falling" },
+ },
+ },
+ ],
+ },
+ {
+ kind: "llm",
+ name: "respond",
+ model: "gpt-5.4",
+ ms: 6980,
+ ttft: 640,
+ text: [
+ "βœ… **Rollback complete** β€” 6/6 pods ready on `payments-service:2026.09.28-4811`.",
+ "",
+ "One pod briefly crash-looped because its node (`ip-10-2-7-19`) was being drained by the autoscaler; it was rescheduled to `ip-10-2-4-88` and is healthy. Unrelated to the rollback.",
+ "",
+ "| checkout-api | Peak | Now |",
+ "| --- | --- | --- |",
+ "| p95 latency | 4.2 s | 610 ms ↓ |",
+ "| Error rate | 6.1% | 0.4% ↓ |",
+ "",
+ "I'll keep watching until we're back at baseline (~350 ms p95, 0.2% errors).",
+ ].join("\n"),
+ },
+ ],
+ },
+ {
+ name: "comms-status-page",
+ atMs: 17 * MINUTE + 40_000,
+ userId: USERS.priya.id,
+ sender: USERS.priya.sender,
+ message:
+ "Support lead here β€” tickets are coming in. Can you draft a status page update? Customer-friendly, no internal details please.",
+ tags: ["comms"],
+ steps: [
+ {
+ kind: "llm",
+ name: "draft-status-update",
+ model: "claude-sonnet-4-5",
+ ms: 5260,
+ ttft: 910,
+ temperature: 0.5,
+ reasoning:
+ "External audience: no service names, deploy ids or hostnames. State the impact window, that it's mitigated, and when the next update comes.",
+ text: [
+ "Here's a draft:",
+ "",
+ "> **Degraded checkout performance β€” Monitoring**",
+ ">",
+ "> Between **21:52 and 22:10 UTC**, some customers experienced slow or failed checkouts. We identified the cause, reverted a recent configuration change, and checkout is performing normally again.",
+ ">",
+ "> We're monitoring closely and will post a final update within the hour. We're sorry for the disruption.",
+ ].join("\n"),
+ },
+ {
+ kind: "guardrail",
+ name: "guardrail.external-comms",
+ ms: 180,
+ input: {
+ checks: ["internal_identifiers", "customer_data", "blame", "tone"],
+ audience: "public-status-page",
+ },
+ output: { verdict: "pass", flags: [], reading_level: "grade 7" },
+ },
+ ],
+ },
+ {
+ name: "postmortem-draft",
+ atMs: 31 * MINUTE + 20_000,
+ userId: USERS.maya.id,
+ sender: USERS.maya.sender,
+ message:
+ "p95 has been at baseline for 15 min. Please draft the postmortem timeline and file follow-ups for the pool config and the late alert.",
+ tags: ["postmortem"],
+ steps: [
+ {
+ kind: "llm",
+ name: "plan",
+ model: "gpt-5.4",
+ ms: 1260,
+ ttft: 480,
+ reasoning:
+ "Hand the write-up to the postmortem sub-agent β€” it has the incident timeline and channel history tools β€” then file the two follow-ups myself.",
+ toolCalls: [
+ {
+ id: "call_p6Ym3tQw9LvH",
+ name: "delegate_task",
+ args: {
+ agent: "postmortem-writer",
+ incident: "INC-4471",
+ deliverable: "timeline + root cause + follow-ups",
+ },
+ ms: 0,
+ result: {
+ status: "done",
+ doc: "https://wiki.northwind.dev/postmortems/INC-4471",
+ sections: ["Summary", "Timeline", "Root cause", "Follow-ups"],
+ },
+ subagent: {
+ name: "postmortem-writer",
+ system:
+ "You write blameless incident postmortems. Use UTC times, cite sources, and keep follow-ups concrete with an owner each.",
+ task: "Draft the postmortem for INC-4471 (checkout latency). Use the incident timeline, the #inc-4471 channel history and similar past incidents.",
+ steps: [
+ {
+ kind: "llm",
+ name: "gather-context",
+ model: "claude-haiku-4-5",
+ ms: 1340,
+ ttft: 430,
+ toolCalls: [
+ {
+ id: "toolu_01XkT7fPq3Rw9",
+ name: "get_incident_timeline",
+ args: { incident: "INC-4471" },
+ ms: 520,
+ result: {
+ incident: "INC-4471",
+ severity: "SEV-2",
+ events: [
+ {
+ at: "21:51",
+ event: "payments-service #4812 deployed",
+ },
+ {
+ at: "21:52",
+ event: "checkout p95 > 4 s, errors 6.1%",
+ },
+ { at: "22:00", event: "on-call engaged the copilot" },
+ { at: "22:09", event: "rollback of #4812 started" },
+ { at: "22:10", event: "rollback complete" },
+ {
+ at: "22:16",
+ event: "latency and errors at baseline",
+ },
+ ],
+ },
+ },
+ {
+ id: "toolu_01Hc2WmZ8sLn4",
+ name: "get_channel_history",
+ args: { channel: "#inc-4471-checkout-latency" },
+ ms: 690,
+ result: {
+ messages: 23,
+ participants: [
+ "maya.okafor",
+ "jonas.weber",
+ "priya.raman",
+ "alex.kim",
+ ],
+ decisions: [
+ "rollback #4812 (jonas.weber)",
+ "status page update (priya.raman)",
+ ],
+ },
+ },
+ ],
+ },
+ {
+ kind: "retriever",
+ name: "similar-incidents",
+ ms: 240,
+ input: {
+ query: "connection pool exhaustion after config change",
+ top_k: 3,
+ },
+ output: {
+ documents: [
+ {
+ id: "INC-3902",
+ title: "Orders API pool exhaustion",
+ score: 0.87,
+ },
+ {
+ id: "INC-3518",
+ title: "pgbouncer rollout latency",
+ score: 0.74,
+ },
+ ],
+ },
+ },
+ {
+ kind: "llm",
+ name: "write-postmortem",
+ model: "claude-haiku-4-5",
+ ms: 8420,
+ ttft: 560,
+ text: [
+ "## INC-4471 β€” Checkout latency from DB pool exhaustion",
+ "",
+ "**Impact:** 21:52–22:10 UTC, slow or failed checkouts for ~18 minutes. No data loss.",
+ "",
+ "**Root cause:** payments-service #4812 lowered the connection pool from 50 to 10 ahead of a pgbouncer migration that wasn't live yet; peak fan-out needs ~24 connections.",
+ "",
+ "**Follow-ups**",
+ "1. Validate pool size against peak query fan-out in CI (payments).",
+ "2. Alert on connection-pool wait time (observability) β€” the latency SLO alert fired 11 minutes late.",
+ ].join("\n"),
+ },
+ ],
+ },
+ },
+ ],
+ },
+ {
+ kind: "llm",
+ name: "file-follow-ups",
+ model: "gpt-5.4",
+ ms: 1540,
+ ttft: 520,
+ toolCalls: [
+ {
+ id: "call_c1Nf5xKd7WbJ",
+ name: "create_issue",
+ args: {
+ team: "PAY",
+ title: "Validate DB pool size against peak query fan-out in CI",
+ labels: ["incident:INC-4471"],
+ },
+ ms: 410,
+ traced: false,
+ result: {
+ id: "PAY-1187",
+ url: "https://tracker.northwind.dev/PAY-1187",
+ },
+ },
+ {
+ id: "call_c2Ze8yMg4VtK",
+ name: "create_issue",
+ args: {
+ team: "OBS",
+ title:
+ "Alert on connection-pool wait time for checkout dependencies",
+ labels: ["incident:INC-4471"],
+ },
+ ms: 380,
+ traced: false,
+ result: {
+ id: "OBS-342",
+ url: "https://tracker.northwind.dev/OBS-342",
+ },
+ },
+ ],
+ },
+ {
+ kind: "llm",
+ name: "respond",
+ model: "gpt-5.4",
+ ms: 8870,
+ ttft: 690,
+ text: [
+ "πŸ“ **Postmortem draft:** [INC-4471 β€” Checkout latency from DB pool exhaustion](https://wiki.northwind.dev/postmortems/INC-4471)",
+ "",
+ "| Time (UTC) | Event |",
+ "| --- | --- |",
+ "| 21:51 | payments-service #4812 deployed (pool 50 β†’ 10) |",
+ "| 21:52 | checkout p95 4.2 s, errors 6.1% |",
+ "| 22:00 | @maya engages the copilot |",
+ "| 22:09 | @jonas starts the rollback |",
+ "| 22:10 | rollback complete |",
+ "| 22:16 | back to baseline |",
+ "",
+ "**Follow-ups filed**",
+ "- **PAY-1187** β€” validate pool size against peak query fan-out in CI",
+ "- **OBS-342** β€” alert on connection-pool wait time (the SLO alert fired 11 min late)",
+ "",
+ "Time to mitigate: **18 min**.",
+ ].join("\n"),
+ },
+ ],
+ },
+ {
+ name: "change-request",
+ atMs: 35 * MINUTE + 50_000,
+ userId: USERS.alex.id,
+ sender: USERS.alex.sender,
+ message:
+ "Should we just delete the old pool settings from the prod config so this can't happen again? Can you do it?",
+ tags: ["change-request"],
+ steps: [
+ {
+ kind: "guardrail",
+ name: "guardrail.prod-change",
+ ms: 210,
+ input: {
+ checks: ["destructive_action", "production_scope"],
+ target: "payments-service/config/database.yml",
+ },
+ output: {
+ verdict: "flag",
+ policy: "prod-change-requires-approval",
+ action: "route_to_change_request",
+ },
+ level: "WARNING",
+ statusMessage:
+ "Destructive production change requested β€” routed to the change-request flow",
+ },
+ {
+ kind: "llm",
+ name: "respond",
+ model: "gpt-5.4",
+ ms: 4630,
+ ttft: 560,
+ refusal:
+ "I can't change or delete production configuration from chat β€” prod config changes need an approved change request.",
+ text: [
+ "Deleting the settings wouldn't prevent a repeat β€” the default pool is 5, which is even smaller. Here's the safe path:",
+ "",
+ "1. Open a change request that sets `pool` explicitly from peak fan-out (I can pre-fill it from today's numbers).",
+ "2. Land the CI check from **PAY-1187** first so a too-small pool fails the build.",
+ "3. Roll out behind the pgbouncer flag once transaction pooling is live.",
+ ].join("\n"),
+ },
+ ],
+ },
+ {
+ name: "resolve-incident",
+ atMs: 56 * MINUTE,
+ userId: USERS.maya.id,
+ sender: USERS.maya.sender,
+ message:
+ "All clear for 40 minutes. Closing INC-4471 β€” please resolve it and publish the final status update.",
+ tags: ["resolution"],
+ steps: [
+ {
+ kind: "llm",
+ name: "plan",
+ model: "gpt-5.4",
+ ms: 1020,
+ ttft: 410,
+ toolCalls: [
+ {
+ id: "call_v8Rk1pWn6CyL",
+ name: "resolve_incident",
+ args: { incident: "INC-4471", resolution: "rolled back #4812" },
+ ms: 350,
+ result: {
+ incident: "INC-4471",
+ status: "resolved",
+ resolved_at: "22:56 UTC",
+ duration_min: 64,
+ },
+ },
+ {
+ id: "call_u3Hs9bTe2QxM",
+ name: "publish_status_update",
+ args: {
+ page: "status.northwind.dev",
+ state: "resolved",
+ title: "Degraded checkout performance",
+ },
+ ms: 470,
+ result: {
+ published: true,
+ url: "https://status.northwind.dev/incidents/q7kd2",
+ },
+ },
+ ],
+ },
+ {
+ kind: "event",
+ name: "incident.resolved",
+ metadata: { incident: "INC-4471" },
+ },
+ {
+ kind: "llm",
+ name: "respond",
+ model: "gpt-5.4",
+ ms: 3910,
+ ttft: 520,
+ text: [
+ "**INC-4471 resolved** at 22:56 UTC. Customer impact 21:52–22:10 UTC (18 min).",
+ "",
+ "- Final update published on [status.northwind.dev](https://status.northwind.dev/incidents/q7kd2)",
+ "- Postmortem review: Thursday 15:00 UTC with @maya and @jonas",
+ "",
+ "Thanks everyone πŸ™",
+ ].join("\n"),
+ },
+ ],
+ },
+];
+
+type PlannedObservation = {
+ id: string;
+ parentId: string | null;
+ type: ObservationType;
+ name: string;
+ start: number;
+ end: number;
+ input?: unknown;
+ output?: unknown;
+ /** stored verbatim instead of JSON-encoded (TOOL rows show I/O as stored) */
+ rawInput?: string;
+ rawOutput?: string;
+ model?: ModelName;
+ modelParameters?: Record<string, unknown>;
+ usage?: [number, number];
+ ttft?: number;
+ level?: "WARNING" | "ERROR";
+ statusMessage?: string;
+ metadata: Record<string, string>;
+};
+
+type ChatMessage = Record<string, unknown>;
+
+const estimateTokens = (value: unknown): number =>
+ Math.max(1, Math.round(JSON.stringify(value).length / 3.7));
+
+const assistantOutput = (step: LlmStep): ChatMessage => {
+ const content: Array<Record<string, string>> = [];
+ if (step.reasoning) content.push({ type: "reasoning", text: step.reasoning });
+ if (step.text) content.push({ type: "text", text: step.text });
+ const toolCalls = step.toolCalls ?? [];
+ return {
+ role: "assistant",
+ content: content.length > 0 ? content : null,
+ ...(step.refusal ? { refusal: step.refusal } : {}),
+ ...(toolCalls.length > 0
+ ? {
+ tool_calls: toolCalls.map((call) => ({
+ id: call.id,
+ type: "function",
+ function: { name: call.name, arguments: JSON.stringify(call.args) },
+ })),
+ }
+ : {}),
+ };
+};
+
+const toolResultContent = (result: unknown): string =>
+ typeof result === "string" ? result : JSON.stringify(result);
+
+/**
+ * Lays out one agent loop (the turn's steps, or a sub-agent's) on a timeline
+ * starting at `cursorStart`, appending observations under `parentId` and
+ * extending `history` the way the app would resend it. Returns the end offset.
+ */
+const planSteps = (args: {
+ steps: Step[];
+ parentId: string;
+ cursorStart: number;
+ history: ChatMessage[];
+ observations: PlannedObservation[];
+ nextId: () => string;
+ metadata: Record<string, string>;
+}): number => {
+ const { steps, parentId, history, observations, nextId, metadata } = args;
+ let cursor = args.cursorStart;
+
+ for (const step of steps) {
+ if (step.kind === "wait") {
+ cursor += step.ms;
+ continue;
+ }
+ if (step.kind === "event") {
+ cursor += 30;
+ observations.push({
+ id: nextId(),
+ parentId,
+ type: "EVENT",
+ name: step.name,
+ start: cursor,
+ end: cursor,
+ metadata: { ...metadata, ...step.metadata },
+ });
+ continue;
+ }
+ if (step.kind !== "llm") {
+ const start = cursor + 30;
+ observations.push({
+ id: nextId(),
+ parentId,
+ type: step.kind === "guardrail" ? "GUARDRAIL" : "RETRIEVER",
+ name: step.name,
+ start,
+ end: start + step.ms,
+ input: step.input,
+ output: step.output,
+ level: step.level,
+ statusMessage: step.statusMessage,
+ metadata,
+ });
+ cursor = start + step.ms;
+ continue;
+ }
+
+ // LLM call: the input is everything the app resends; the output is the
+ // assistant message (reasoning, text, refusal, tool calls).
+ const input = [...history];
+ const output = assistantOutput(step);
+ const start = cursor + 60;
+ observations.push({
+ id: nextId(),
+ parentId,
+ type: "GENERATION",
+ name: step.name,
+ start,
+ end: start + step.ms,
+ input,
+ output,
+ model: step.model,
+ modelParameters: {
+ temperature: step.temperature ?? 0.2,
+ max_tokens: 2048,
+ ...(step.toolCalls?.length ? { tool_choice: "auto" } : {}),
+ },
+ usage: [estimateTokens(input) + 350, estimateTokens(output)],
+ ttft: step.ttft,
+ metadata,
+ });
+ cursor = start + step.ms;
+
+ const toolCalls = step.toolCalls ?? [];
+ history.push({
+ role: "assistant",
+ content: step.text ?? null,
+ ...(toolCalls.length > 0
+ ? { tool_calls: (output.tool_calls as unknown[]) ?? [] }
+ : {}),
+ });
+
+ // Tool calls run in parallel, each traced as a TOOL observation carrying
+ // the call id so the session view pairs it with the rolled-up call.
+ let toolsEnd = cursor;
+ toolCalls.forEach((call, index) => {
+ const toolStart = cursor + 40 + index * 6;
+ let toolEnd = toolStart + call.ms;
+ if (call.traced !== false) {
+ const toolId = nextId();
+ if (call.subagent) {
+ const agentId = nextId();
+ const agentStart = toolStart + 20;
+ const subHistory: ChatMessage[] = [
+ { role: "system", content: call.subagent.system },
+ { role: "user", content: call.subagent.task },
+ ];
+ const agentObservations: PlannedObservation[] = [];
+ const agentEnd =
+ planSteps({
+ steps: call.subagent.steps,
+ parentId: agentId,
+ cursorStart: agentStart,
+ history: subHistory,
+ observations: agentObservations,
+ nextId,
+ metadata: { ...metadata, agent: call.subagent.name },
+ }) + 20;
+ toolEnd = agentEnd + 25;
+ observations.push(
+ {
+ id: toolId,
+ parentId,
+ type: "TOOL",
+ name: call.name,
+ start: toolStart,
+ end: toolEnd,
+ rawInput: JSON.stringify(call.args, null, 2),
+ rawOutput: JSON.stringify(call.result, null, 2),
+ metadata: { ...metadata, toolCallId: call.id },
+ },
+ {
+ id: agentId,
+ parentId: toolId,
+ type: "AGENT",
+ name: call.subagent.name,
+ start: agentStart,
+ end: agentEnd,
+ metadata: { ...metadata, agent: call.subagent.name },
+ },
+ ...agentObservations,
+ );
+ } else {
+ observations.push({
+ id: toolId,
+ parentId,
+ type: "TOOL",
+ name: call.name,
+ start: toolStart,
+ end: toolEnd,
+ rawInput: JSON.stringify(call.args, null, 2),
+ rawOutput:
+ typeof call.result === "string"
+ ? call.result
+ : JSON.stringify(call.result, null, 2),
+ level: call.level,
+ statusMessage: call.statusMessage,
+ metadata: { ...metadata, toolCallId: call.id },
+ });
+ }
+ }
+ history.push({
+ role: "tool",
+ tool_call_id: call.id,
+ content: toolResultContent(call.result),
+ });
+ toolsEnd = Math.max(toolsEnd, toolEnd);
+ });
+ cursor = toolsEnd;
+ }
+ return cursor;
+};
+
+const lastLlmStep = (turn: TurnSpec): LlmStep | undefined =>
+ turn.steps.filter((step): step is LlmStep => step.kind === "llm").at(-1);
+
+/** Final assistant text of a turn (trace output): the answer and/or refusal. */
+const finalAnswer = (turn: TurnSpec): string => {
+ const last = lastLlmStep(turn);
+ return [last?.refusal, last?.text].filter(Boolean).join("\n\n");
+};
+
+const run = async (
+ ctx: ScenarioContext,
+ params: Record<string, string | number | boolean>,
+): Promise<SeedSummary> => {
+ const startedAt = Date.now();
+ const withV4 = params["v4"] as boolean;
+
+ // The prefix IS the session id, so `--id-prefix inc-4471-checkout-latency`
+ // reads like a production session in the header.
+ const sessionId = ctx.idPrefix;
+ const traceIds = TURNS.map((_, index) => `${ctx.idPrefix}-t${index + 1}`);
+ // 22:00 UTC of the previous UTC day: always in the past, identical for
+ // same-day re-runs (ORDER BY keys stay stable), and matches the UTC times
+ // quoted in the payloads.
+ const sessionStart = utcDayStartMs() - 2 * 60 * MINUTE;
+
+ if (ctx.dryRun) {
+ return {
+ scenario: "incident-session",
+ target: "clickhouse",
+ params,
+ projectId: ctx.projectId,
+ environment: ctx.environment,
+ traceIds,
+ sessionIds: [sessionId],
+ counts: { sessions: 1, traces: TURNS.length },
+ verified: {},
+ links: [sessionLink(ctx, sessionId)],
+ dryRun: true,
+ durationMs: Date.now() - startedAt,
+ };
+ }
+
+ const traces: TraceRecordInsertType[] = [];
+ const observations: ObservationRecordInsertType[] = [];
+ const history: ChatMessage[] = [
+ { role: "system", content: COPILOT_SYSTEM_PROMPT },
+ ];
+
+ TURNS.forEach((turn, turnIndex) => {
+ const traceId = traceIds[turnIndex];
+ const traceStart = sessionStart + turn.atMs;
+ let observationIndex = 0;
+ const nextId = () => `${traceId}-o${observationIndex++}`;
+ const metadata = {
+ scenario: "incident-session",
+ turn: String(turnIndex + 1),
+ };
+
+ history.push({ role: "user", name: turn.sender, content: turn.message });
+
+ const rootId = nextId();
+ const planned: PlannedObservation[] = [];
+ const end = planSteps({
+ steps: turn.steps,
+ parentId: rootId,
+ cursorStart: 0,
+ history,
+ observations: planned,
+ nextId,
+ metadata,
+ });
+ // Root AGENT without I/O: the session timeline shows its children expanded.
+ planned.unshift({
+ id: rootId,
+ parentId: null,
+ type: "AGENT",
+ name: "oncall-copilot",
+ start: 0,
+ end: end + 40,
+ metadata: { ...metadata, channel: SESSION_METADATA.channel },
+ });
+
+ const trace = createTrace({
+ id: traceId,
+ project_id: ctx.projectId,
+ environment: ctx.environment,
+ session_id: sessionId,
+ user_id: turn.userId,
+ name: turn.name,
+ timestamp: traceStart,
+ release: "copilot-2026.09.3",
+ version: "oncall-copilot-v4",
+ tags: ["incident", "INC-4471", ...turn.tags],
+ public: false,
+ bookmarked: turn.bookmarked ?? false,
+ metadata: { ...SESSION_METADATA, turn: String(turnIndex + 1) },
+ input: turn.message,
+ output: finalAnswer(turn),
+ created_at: Date.now(),
+ updated_at: Date.now(),
+ event_ts: Date.now(),
+ });
+ traces.push(trace);
+
+ for (const p of planned) {
+ const prices = p.model ? MODEL_PRICES[p.model] : null;
+ const [usageInput, usageOutput] = p.usage ?? [0, 0];
+ const inputCost = prices ? usageInput * prices.input : 0;
+ const outputCost = prices ? usageOutput * prices.output : 0;
+ // Direct ClickHouse writes bypass ingestion, so apply the same tool
+ // extraction it would (tool_calls / tool_call_names columns).
+ const { toolDefinitions, toolArguments } = extractToolsFromObservation(
+ p.input,
+ p.output,
+ );
+ const toolCallArrays = convertCallsToArrays(toolArguments);
+
+ observations.push(
+ createObservation({
+ tool_definitions: convertDefinitionsToMap(toolDefinitions),
+ tool_calls: toolCallArrays.tool_calls,
+ tool_call_names: toolCallArrays.tool_call_names,
+ id: p.id,
+ trace_id: traceId,
+ project_id: ctx.projectId,
+ environment: ctx.environment,
+ type: p.type,
+ parent_observation_id: p.parentId,
+ name: p.name,
+ start_time: traceStart + p.start,
+ end_time: traceStart + p.end,
+ completion_start_time:
+ p.ttft !== undefined ? traceStart + p.start + p.ttft : null,
+ level: p.level ?? "DEFAULT",
+ status_message: p.statusMessage ?? null,
+ version: null,
+ input:
+ p.rawInput ??
+ (p.input !== undefined ? JSON.stringify(p.input) : null),
+ output:
+ p.rawOutput ??
+ (p.output !== undefined ? JSON.stringify(p.output) : null),
+ metadata: p.metadata,
+ provided_model_name: p.model ?? null,
+ internal_model_id: null,
+ model_parameters: p.modelParameters
+ ? JSON.stringify(p.modelParameters)
+ : "{}",
+ // Explicit empties for non-generations: the factory would otherwise
+ // fill non-empty usage/cost defaults. Totals are what the session
+ // header sums.
+ ...(prices
+ ? {
+ provided_usage_details: {
+ input: usageInput,
+ output: usageOutput,
+ total: usageInput + usageOutput,
+ },
+ usage_details: {
+ input: usageInput,
+ output: usageOutput,
+ total: usageInput + usageOutput,
+ },
+ provided_cost_details: { input: inputCost, output: outputCost },
+ cost_details: {
+ input: inputCost,
+ output: outputCost,
+ total: inputCost + outputCost,
+ },
+ total_cost: inputCost + outputCost,
+ }
+ : {
+ provided_usage_details: {},
+ usage_details: {},
+ provided_cost_details: {},
+ cost_details: {},
+ total_cost: null,
+ }),
+ prompt_id: null,
+ prompt_name: null,
+ prompt_version: null,
+ created_at: Date.now(),
+ updated_at: Date.now(),
+ event_ts: Date.now(),
+ }),
+ );
+ }
+
+ // The next turn resends the final answer, not the intermediate steps β€”
+ // in the shape the model returned it (content and refusal stay separate
+ // fields), which is what lets the timeline recognize it as history.
+ const last = lastLlmStep(turn);
+ const turnStart = history.findIndex(
+ (message) =>
+ message.role === "user" &&
+ message.name === turn.sender &&
+ message.content === turn.message,
+ );
+ history.splice(turnStart + 1);
+ history.push({
+ role: "assistant",
+ content: last?.text ?? null,
+ ...(last?.refusal ? { refusal: last.refusal } : {}),
+ });
+ });
+
+ // Postgres ids are global rather than per project, so they carry the project
+ // id: the same --id-prefix seeded into two projects must not share rows.
+ const postgresId = (suffix: string) => `${ctx.projectId}-${suffix}`;
+ // One project-level config, as for real annotation scores: the Annotate
+ // panel only renders ANNOTATION scores whose config_id resolves.
+ const severityConfigId = postgresId("incident-severity-config");
+
+ const scoreTimestamp = sessionStart + 57 * MINUTE;
+ const scores: ScoreRecordInsertType[] = [
+ createSessionScore({
+ id: `${sessionId}-score-severity`,
+ project_id: ctx.projectId,
+ session_id: sessionId,
+ environment: ctx.environment,
+ name: SEVERITY_CONFIG.name,
+ value: 2,
+ string_value: "SEV-2",
+ data_type: "CATEGORICAL",
+ source: "ANNOTATION",
+ config_id: severityConfigId,
+ comment: "Customer-facing checkout degradation for 18 min; no data loss.",
+ metadata: { reviewer: "maya.okafor" },
+ timestamp: scoreTimestamp,
+ }),
+ createSessionScore({
+ id: `${sessionId}-score-mitigation`,
+ project_id: ctx.projectId,
+ session_id: sessionId,
+ environment: ctx.environment,
+ name: "time-to-mitigate-min",
+ value: 18,
+ data_type: "NUMERIC",
+ source: "API",
+ comment: null,
+ metadata: {},
+ timestamp: scoreTimestamp,
+ }),
+ createSessionScore({
+ id: `${sessionId}-score-helpfulness`,
+ project_id: ctx.projectId,
+ session_id: sessionId,
+ environment: ctx.environment,
+ name: "copilot-helpfulness",
+ value: 0.92,
+ data_type: "NUMERIC",
+ source: "EVAL",
+ comment:
+ "Named the causal deploy in its first reply and correctly called the pod crash unrelated.",
+ metadata: { judge_model: "gpt-5.4", rubric: "incident-copilot-v2" },
+ timestamp: scoreTimestamp,
+ }),
+ createSessionScore({
+ id: `${sessionId}-score-postmortem`,
+ project_id: ctx.projectId,
+ session_id: sessionId,
+ environment: ctx.environment,
+ name: "postmortem-filed",
+ value: 1,
+ // score_booleans filters on the True/False string
+ string_value: "True",
+ data_type: "BOOLEAN",
+ source: "API",
+ comment: null,
+ metadata: {},
+ timestamp: scoreTimestamp,
+ }),
+ createTraceScore({
+ id: `${traceIds[0]}-score-feedback`,
+ project_id: ctx.projectId,
+ trace_id: traceIds[0],
+ environment: ctx.environment,
+ name: "user-feedback",
+ value: 1,
+ string_value: "True",
+ data_type: "BOOLEAN",
+ source: "API",
+ comment: "Found it in 15 seconds πŸ™Œ",
+ metadata: {},
+ timestamp: sessionStart + 30_000,
+ }),
+ createTraceScore({
+ id: `${traceIds[5]}-score-feedback`,
+ project_id: ctx.projectId,
+ trace_id: traceIds[5],
+ environment: ctx.environment,
+ name: "user-feedback",
+ value: 0,
+ string_value: "False",
+ data_type: "BOOLEAN",
+ source: "API",
+ comment: "Just wanted a quick yes or no",
+ metadata: {},
+ timestamp: sessionStart + TURNS[5].atMs + 20_000,
+ }),
+ ];
+
+ const comments = [
+ {
+ id: postgresId(`${sessionId}-comment-1`),
+ objectType: "SESSION" as const,
+ objectId: sessionId,
+ authorUserId: "user-1",
+ content:
+ "Great example for the on-call training deck: the copilot named the causal deploy in its first reply.",
+ },
+ {
+ id: postgresId(`${sessionId}-comment-2`),
+ objectType: "SESSION" as const,
+ objectId: sessionId,
+ authorUserId: "user-2",
+ content:
+ "Turn 6 is exactly the behavior we want β€” refusal plus a concrete change-request path. Adding it to the eval dataset.",
+ },
+ {
+ id: postgresId(`${sessionId}-comment-3`),
+ objectType: "TRACE" as const,
+ objectId: traceIds[2],
+ authorUserId: "user-1",
+ content:
+ "The CrashLoopBackOff was the node drain, not the rollback β€” correctly called out.",
+ },
+ ];
+
+ const events: EventRecordInsertType[] = withV4
+ ? [
+ ...traces.map((trace) => traceToEvent(trace)),
+ ...observations.map((observation) =>
+ observationToEvent(
+ observation,
+ traces.find((trace) => trace.id === observation.trace_id)!,
+ ),
+ ),
+ ]
+ : [];
+
+ ctx.log(
+ `writing 1 session, ${traces.length} traces, ${observations.length} observations, ${scores.length} scores, ${comments.length} comments${
+ withV4 ? `, ${events.length} events` : ""
+ }`,
+ );
+
+ // The v3 session page needs the trace_sessions row; v4 reads it for
+ // bookmark/public state.
+ await prisma.traceSession.upsert({
+ where: { id_projectId: { id: sessionId, projectId: ctx.projectId } },
+ update: {},
+ create: {
+ id: sessionId,
+ projectId: ctx.projectId,
+ environment: ctx.environment,
+ createdAt: new Date(sessionStart),
+ },
+ });
+ const severityConfig = {
+ name: SEVERITY_CONFIG.name,
+ dataType: "CATEGORICAL" as const,
+ categories: SEVERITY_CONFIG.categories,
+ description: SEVERITY_CONFIG.description,
+ isArchived: false,
+ };
+ await prisma.scoreConfig.upsert({
+ where: { id: severityConfigId },
+ update: severityConfig,
+ create: {
+ id: severityConfigId,
+ projectId: ctx.projectId,
+ ...severityConfig,
+ },
+ });
+ for (const comment of comments) {
+ await prisma.comment.upsert({
+ where: { id: comment.id },
+ update: { content: comment.content, authorUserId: comment.authorUserId },
+ create: { ...comment, projectId: ctx.projectId },
+ });
+ }
+
+ await createTracesCh(traces);
+ for (const batch of chunk(observations, 1000)) {
+ await createObservationsCh(batch);
+ }
+ await createScoresCh(scores);
+ for (const batch of chunk(events, 500)) {
+ await createEventsCh(batch);
+ }
+
+ const verified: Record<string, number> = {
+ traces: await countRows(
+ "traces",
+ `project_id = {projectId: String} AND session_id = {sessionId: String}`,
+ { projectId: ctx.projectId, sessionId },
+ "uniqExact(id)",
+ ),
+ observations: await countRows(
+ "observations",
+ `project_id = {projectId: String} AND trace_id IN {traceIds: Array(String)}`,
+ { projectId: ctx.projectId, traceIds },
+ "uniqExact(id)",
+ ),
+ scores: await countRows(
+ "scores",
+ `project_id = {projectId: String} AND id IN {scoreIds: Array(String)}`,
+ { projectId: ctx.projectId, scoreIds: scores.map((score) => score.id) },
+ "uniqExact(id)",
+ ),
+ };
+ if (withV4) {
+ verified.events = await countRows(
+ "events_full",
+ `project_id = {projectId: String} AND session_id = {sessionId: String}`,
+ { projectId: ctx.projectId, sessionId },
+ "uniqExact(span_id)",
+ );
+ }
+
+ const expected: Record<string, number> = {
+ traces: traces.length,
+ observations: observations.length,
+ scores: scores.length,
+ ...(withV4 ? { events: events.length } : {}),
+ };
+ for (const [key, count] of Object.entries(expected)) {
+ if ((verified[key] ?? 0) < count) {
+ throw new SeedError(
+ `Readback mismatch: expected ${count} ${key}, found ${verified[key] ?? 0}`,
+ );
+ }
+ }
+
+ return {
+ scenario: "incident-session",
+ target: "clickhouse",
+ params,
+ projectId: ctx.projectId,
+ environment: ctx.environment,
+ traceIds,
+ sessionIds: [sessionId],
+ counts: {
+ sessions: 1,
+ traces: traces.length,
+ observations: observations.length,
+ scores: scores.length,
+ comments: comments.length,
+ scoreConfigs: 1,
+ events: events.length,
+ },
+ verified,
+ links: [
+ sessionLink(ctx, sessionId),
+ traceLink(ctx, traceIds[0], sessionStart),
+ ],
+ dryRun: false,
+ durationMs: Date.now() - startedAt,
+ };
+};
+
+export const incidentSessionScenario: ScenarioDefinition = {
+ name: "incident-session",
+ description:
+ "One demo-grade, fully handcrafted multi-user SESSION for the session timeline: an on-call copilot working a checkout-latency incident with four engineers over seven turns β€” parallel and untraced tool calls, a WARNING and an ERROR tool with a retry, a TOOL delegating to a nested sub-agent, reasoning, markdown answers, a guardrail-flagged refusal, three models with costs, typed session scores (API/EVAL/ANNOTATION) and session/trace comments. Deterministic; the id prefix is the session id. Built for videos/screenshots.",
+ supportsV4: true,
+ flags: [
+ {
+ flag: "v4",
+ type: "boolean",
+ default: true,
+ description:
+ "mirror into v4 events tables (on by default: the session timeline is v4-only)",
+ },
+ ],
+ run,
+};
diff --git a/packages/shared/scripts/seeder/scenarios/index.ts b/packages/shared/scripts/seeder/scenarios/index.ts
index 0ac5adb401..61e9afa2a3 100644
--- a/packages/shared/scripts/seeder/scenarios/index.ts
+++ b/packages/shared/scripts/seeder/scenarios/index.ts
@@ -5,6 +5,7 @@ import { customModelsScenario } from "./custom-models";
import { deepChainScenario } from "./deep-chain";
import { evaluatorGalleryScenario } from "./evaluator-gallery";
import { experimentIoScenario } from "./experiment-io";
+import { incidentSessionScenario } from "./incident-session";
import { longSessionScenario } from "./long-session";
import { manyTracesScenario } from "./many-traces";
import { outlierTrafficScenario } from "./outlier-traffic";
@@ -36,6 +37,7 @@ export const scenarios: Record<string, ScenarioDefinition> = {
"annotation-queue": annotationQueueScenario,
"custom-models": customModelsScenario,
"support-agent": supportAgentScenario,
+ "incident-session": incidentSessionScenario,
"timeline-annotated": timelineAnnotatedScenario,
"timeline-shapes": timelineShapesScenario,
};