πͺ’ Open source agent evals & observability: Trace, evaluate, and improve LLM applications with one open platform.
TypeScript analyticsautogenevaluationlangchainlarge-language-modelsllama-indexllmllm-evaluationllm-observabilityllmopsmonitoringobservabilityopen-sourceopenaiplaygroundprompt-engineeringprompt-managementself-hostedycombinator
docs(seeder): document all scenarios in the README table (#16326)
Nikita Kabardinc405ca702cdb
diff --git a/packages/shared/scripts/seeder/README.md b/packages/shared/scripts/seeder/README.mdindex 94fdfdcf5e..6ad577fd6c 100644--- a/packages/shared/scripts/seeder/README.md+++ b/packages/shared/scripts/seeder/README.md@@ -36,12 +36,21 @@ programmatic calls from the dx seed chain.## Scenarios-| Scenario | Covers | Key flags |-| ------------------ | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------- |-| `trace-tree` | one trace with a large, branching observation tree: all ten observation kinds always present, guaranteed depth backbone, hub node with many children, errors/retries/missing end times | `--observations`, `--depth`, `--breadth`, `--payload-bytes`, `--payload-style json\|text\|malformed\|unicode`, `--v4` |-| `long-session` | one session with many traces for session-detail and virtualization work; creates the Postgres `trace_sessions` row the session page requires | `--traces`, `--observations-per-trace`, `--payload-bytes`, `--minutes`, `--session-id`, `--v4` |-| `many-traces` | trace-list and filter performance via `numbers()` bulk SQL; parent/score/prompt/session links all resolve | `--count`, `--days`, `--observations-per-trace`, `--scores-per-trace`, `--rich-payloads` |-| `annotation-queue` | two human-annotation queues for the annotate UI: a "core types" queue with one of every score-field render path (categorical toggle/combobox, boolean, ranged/decimal/unranged numeric, text) over fresh trace items, and an "edge cases" queue adding archived/stale/partial scores, comments, and observation/session/deleted/completed items | `--core-items`, `--v4` (default true) |+| Scenario | Covers | Key flags |+| ------------------ | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |+| `trace-tree` | one trace with a large, branching observation tree: all ten observation kinds always present, guaranteed depth backbone, hub node with many children, errors/retries/missing end times | `--observations`, `--depth`, `--breadth`, `--payload-bytes`, `--payload-style json\|text\|malformed\|unicode`, `--v4` |+| `agent-timeline` | one trace: a realistic LangGraph-style refine-loop agent (planner β retriever β generator β critic β loop) unrolled over N turns, with observations staggered across a real timeline and langgraph_node/step metadata | `--turns`, `--turn-gap-ms`, `--timing-only` |+| `agent-graph` | one trace that is large as a GRAPH, not as a tree: many distinct langgraph_node names with parallel branches per super-step, so the aggregated graph gets thousands of distinct node-pair connections from a few hundred observations | `--nodes`, `--steps`, `--parallel` |+| `deep-chain` | one trace whose observations form a single deep parent chain of sequential generations (child starts after parent ends; depth = observation count) β the mis-parented-instrumentation shape that collapses tree/timeline layouts at extreme depth | `--observations` |+| `long-session` | one session with many traces for session-detail and virtualization work; creates the Postgres `trace_sessions` row the session page requires | `--traces`, `--observations-per-trace`, `--payload-bytes`, `--minutes`, `--session-id`, `--v4` |+| `many-traces` | trace-list and filter performance via `numbers()` bulk SQL; parent/score/prompt/session links all resolve | `--count`, `--days`, `--observations-per-trace`, `--scores-per-trace`, `--rich-payloads` |+| `outlier-traffic` | diurnal base traffic over the past N days with deterministic cost/latency/token outliers and hour-long latency incidents β built for the outlier chart strip above the trace table | `--days`, `--traces-per-day`, `--incidents` |+| `scored-traces` | standalone traces with mixed current/legacy Python and JavaScript SDK attribution, each carrying numeric + categorical scores whose names contain SPACES (e.g. "Rouge Score") at observation and trace level, plus dual-level score names ("confidence", "verdict") present at both levels on the same trace | `--traces` |+| `session-shapes` | diverse v4 session shapes for the session-detail view: a clean multi-turn CHAT session (renders as chat), a coding/AGENT session whose I/O lives on AGENT/TOOL observations with NO GENERATION, a MIXED session, and a MEDIA session whose messages carry langfuseMedia references | `--shape chat\|agent\|mixed\|media\|all`, `--turns` |+| `annotation-queue` | two human-annotation queues for the annotate UI: a "core types" queue with one of every score-field render path (categorical toggle/combobox, boolean, ranged/decimal/unranged numeric, text) over fresh trace items, and an "edge cases" queue adding archived/stale/partial scores, comments, and observation/session/deleted/completed items | `--core-items`, `--v4` (default true) |+| `custom-models` | project-level model definitions (one tiered with a condition-gated second tier and a usage type priced at 0, one single-tier) plus a trace whose generations link to them, and one generation whose model matches no definition so its badge opens the create dialog | β |+| `support-agent` | one demo-grade, fully handcrafted trace: a customer-support copilot resolving a duplicate-charge refund β input guardrail β intent classification β parallel context fan-out (CRM/billing/tickets) β 3-turn ReAct loop (llm.chat + Stripe tools) β drafted reply β output guardrail β send | β |+| `timeline-shapes` | a dozen SMALL traces (4-25 observations each), one per timeline morphology: rag answer, streamed chat, parallel fan-out, retry backoff with widening gaps, a 13-minute wait on a human, one slow tool dwarfing everything, an error cascade with failover, in-flight spans, zero-duration checkpoints, a ten-level ladder, 24 flat siblings, and a three-turn agent loop | `--shape all\|rag-answer\|streaming-chat\|parallel-fanout\|retry-backoff\|waiting-on-approval\|slow-tool\|error-cascade\|still-running\|checkpoint-marks\|deep-ladder\|flat-siblings\|mixed-loop` |Common flags: `--project` (defaults to the seeded example project),`--environment`, `--seed`, `--id-prefix`, `--dry-run` (instant, arithmetic