πͺ’ Open source agent evals & observability: Trace, evaluate, and improve LLM applications with one open platform.
TypeScript analyticsautogenevaluationlangchainlarge-language-modelsllama-indexllmllm-evaluationllm-observabilityllmopsmonitoringobservabilityopen-sourceopenaiplaygroundprompt-engineeringprompt-managementself-hostedycombinator
feat: return conversation history and current turn per thread (#17613)
marliessophie0ad6f869c392
diff --git a/packages/shared/src/server/transcript/README.md b/packages/shared/src/server/transcript/README.mdindex 31d755ca67..3e023dca1d 100644--- a/packages/shared/src/server/transcript/README.md+++ b/packages/shared/src/server/transcript/README.md@@ -2,8 +2,7 @@PLEASE DO NOT USE IN PRODUCTION YET. This is a v1 implementation of the transcript builder and remains work in progress.-Builds a conversation transcript from the observations of one trace, or of-every trace in one session.+Builds a conversation transcript from the observations of one trace.Status: generation-led builder with tool responses matched by ID or name and order.@@ -13,13 +12,17 @@ Status: generation-led builder with tool responses matched by ID or name and ord## Interface```ts-getTranscript(- observations: Observation[]-): Transcript | null;+orderObservations(observations: Observation[]): Observation[];+assembleTranscript(orderedObservations: Observation[]): Transcript | null;type Transcript = { threads: Thread[] };type Thread = {+ conversationHistory: NormalizedMessage[]; // replayed from earlier turns, no provenance+ currentTurn: Turn;+};++type Turn = {messages: ThreadMessage[];observations: { id: string; traceId: string }[]; // observations that contributed, in order};@@ -30,15 +33,47 @@ type ThreadMessage = NormalizedMessage & {};```-The input is the domain `Observation` (see `domain/observations.ts`), which-the repositories produce from ClickHouse rows. Returns `null` when no eligible-generations produce messages.+Consumers load the domain `Observation`s (see `domain/observations.ts`)+themselves, order them with `orderObservations`, and hand them to+`assembleTranscript`, which consumes the given order and returns `null` when+no eligible generations produce messages. For one trace, read it through+`getObservationsForTraceFromEventsTable`, the same repository function and+time bounds the trace tree uses: once for the structure of every observation+without I/O, once for the `GENERATION` and `TOOL` observations with I/O, then+merge the two by id so the walk order comes from the structure and the+messages from the content.++## Ordering++The transcript walks observations the way the trace tree does: depth first,+with roots and siblings by start time. That differs from plain start-time order+when a span that started earlier contains a generation that started later than+a sibling span's generation. `orderObservations` produces this order from the+full structure, following the web tree builder's rules: one row per id with the+earliest start winning, and a row whose parent is missing becomes a root. The+assembler never sorts; callers, including the fixtures test, order first.A **transcript** contains the conversation threads inferred from the suppliedobservations. A **thread** is a sequence of messages connected by shared inputhistory; it can span multiple traces. The consumer handles multiple threadsand decides which, if any, is the main conversation.+## Conversation history and current turn++A trace is one turn. The input its generations replay from earlier turns, up+to and including the last assistant or tool message before the trace's first+output, is `conversationHistory` without provenance; everything after it is+`currentTurn`, each message with the observation that emitted it.++```+input: User: Refund my order.+ Assistant: [refund call]+ Tool: Refund succeeded.+ Assistant: Your refund is complete. <- conversation history ends here+ User: When will it arrive?+output: Assistant: Within five business days.+```+## Which observations contribute?- `GENERATION` observations with a non-null trace ID establish threads.@@ -49,9 +84,9 @@ and decides which, if any, is the main conversation.Unknown explicit IDs do not fall back to names; unmatched tools are skipped.Preserve all normalized output parts; tool inputs are ignored. Provider-specificpayload interpretation belongs to normalized IO, not the transcript builder.-- The caller supplies observations from one trace or session. The builder- orders generations and tools by start time across the supplied traces.-- Each observation is normalized once in this chronological pass. Generations+- The caller supplies one trace's observations, already in transcript order+ (see Ordering).+- Each observation is normalized once in this ordered pass. Generationsestablish threads and register output tool-call IDs; tools enrich registeredcalls. Input messages are processed before output messages for generations.- Generations producing no messages are skipped and do not create empty threads.@@ -145,26 +180,26 @@ and observation provenance are excluded. All fields inside parts are included.```transcript/βββ README.md-βββ index.ts getTranscript+βββ index.ts public surface: assembleTranscript, orderObservations, types+βββ ordering.ts orderObservations, the trace tree walk+βββ ordering.test.ts ordering rules+βββ transcript.ts assembleTranscriptβββ threads.ts thread selection and message deduplicationβββ tool-calls.ts tool matching and response association-βββ types.ts Transcript, Thread, ThreadMessage+βββ types.ts Transcript, Thread, Turn, ThreadMessageβββ fixtures/- βββ README.md how to turn a trace JSON export into a fixtureβββ fixture-types.ts TranscriptFixtureβββ format-transcript.ts chat-shaped printout used by the test- βββ index.ts registry: trace-scoped and session-scoped fixtures+ βββ index.ts registry of fixturesβββ fixtures.test.ts structural checks and behavior assertion per fixture- βββ trace/ one file per trace-scoped fixture- βββ session/ one file per session-scoped fixture+ βββ trace/ one file per fixture```## Fixtures and verification-Fixtures contain observation trees and an optional expected transcript.-**Fixtures with `expected: undefined` exercise parsing and structural checks,-but do not verify transcript correctness.** Their printed output is for manual-review; expectations are authored by hand once the desired behavior is decided.+Each fixture is one trace with an expected transcript, which the test asserts+as a whole. Fixtures whose generations replay earlier turns pin the split+between conversation history and current turn.Run with console output enabled to see it:diff --git a/packages/shared/src/server/transcript/fixtures/README.md b/packages/shared/src/server/transcript/fixtures/README.mddeleted file mode 100644index bb35dc019f..0000000000--- a/packages/shared/src/server/transcript/fixtures/README.md+++ /dev/null@@ -1,222 +0,0 @@-# Building a transcript fixture from a trace JSON export--This guide is written so an agent can follow it end to end. Hand it a trace-export and this file, and it should return one fixture file plus a registry-edit, with the expectation left open.--## Input--A trace JSON export, downloaded from the trace page (`β―` menu β Download-JSON). The file is named `trace-<traceId>.json` and has this shape:--```json-{- "observations": [- {- "id": "β¦",- "traceId": "β¦",- "sessionId": "β¦",- "parentObservationId": "β¦",- "type": "GENERATION",- "name": "β¦",- "startTime": "2026-08-31T12:33:14.205Z",- "endTime": "2026-08-31T12:33:14.818Z",- "level": "DEFAULT",- "statusMessage": "",- "model": "gpt-5.4",- "input": "{\"messages\":[β¦]}",- "output": "{\"content\":null,\"tool_calls\":[β¦]}",- "metadata": { "β¦": "β¦" },- "β¦": "many more fields the fixture does not need"- }- ],- "scores": []-}-```--For a session-scoped fixture, export every trace of the session separately-and concatenate their `observations` arrays. The session page download does-not include observations.--Along with the file, the requester may say which observations to drop, which-payloads to shorten, and what the case is meant to exercise. Without such-notes, keep everything.--## Output--One file `fixtures/trace/<kebab-case-name>.ts` (or `fixtures/session/β¦`)-exporting a single `<camelCaseName>Fixture` that `satisfies TranscriptFixture`,-plus a one-line registration in `fixtures/index.ts`. Use-`trace/support-copilot-refund-loop.ts` as the reference for tone and layout.--## Procedure--1. **Draw the tree.** Build the parent β child tree from `parentObservationId`- and write it as the file's leading doc comment in this style:-- ```- [SPAN] name id root- [AGENT] name id- [GENERATION] name model id in: <message roles> out: <what came back>- [TOOL] name id in: <relation to a call> out: <summary>- ```-- Annotate each `GENERATION` with the roles in its input and a short summary- of its output. Annotate each `TOOL` with how its input relates to a- tool call nearby. Note when the export order differs from tree order.--2. **Keep every observation.** Types the builder ignores (`SPAN`,- `GUARDRAIL`, `EVENT`, β¦) stay in the fixture. Filtering is the builder's- job and must be tested.--3. **Keep the export order.** Do not sort by time or by tree position. The- builder must not rely on input order.--4. **Map fields.** Copy these and drop everything else:-- | Export field | Fixture field | Rule |- | --------------------- | --------------------- | --------------------------------------------- |- | `id` | `id` | verbatim |- | `traceId` | `traceId` | verbatim; hoist into a const |- | `sessionId` | `sessionId` | verbatim when present; hoist into a const |- | `parentObservationId` | `parentObservationId` | `""` becomes `null` |- | `type` | `type` | verbatim |- | `name` | `name` | verbatim |- | `startTime` | `startTime` | verbatim ISO string |- | `endTime` | `endTime` | verbatim ISO string or `null` |- | `level` | `level` | only when not `DEFAULT` |- | `statusMessage` | `statusMessage` | only when non-empty |- | `model` | `model` | only when non-empty |- | `input` | `input` | see encoding rules |- | `output` | `output` | see encoding rules |- | `metadata` | `metadata` | verbatim object; hoist and spread when shared |--5. **Encode I/O exactly as stored.** The export carries `input` and `output`- as one JSON-encoded string each. In the fixture, write the decoded value as- an object literal wrapped in `JSON.stringify(...)`. This keeps the stored- encoding while making the content readable:-- ```ts- input: JSON.stringify({- messages: [{ role: "user", content: customerMessage }],- }),- ```-- Strings nested inside that value stay strings. Tool-call `arguments` and- tool-message `content` are often JSON text themselves. Write them as- `JSON.stringify(...)` inline, never as decoded objects:-- ```ts- function: {- name: "stripe_find_charges",- arguments: JSON.stringify(findChargesArguments),- },- ```-- If a stored value is plain text rather than JSON, keep it as a string- literal. If it is `null`, keep `null`.--6. **Hoist repeated values.** System prompts, user messages, tool-call ids,- tool arguments, and the final answer usually appear in several- observations. Give each a `const` and reuse it, so byte-equality across- observations is visible in code. When a `TOOL` observation's input equals- a tool call's arguments, use the same const for both.--7. **Shorten with care.** Long payloads (retrieved documents, big system- prompts, base64) may be replaced by a short bracketed placeholder such as- `"[reduced: 78 KB of retrieved docs]"`, with these constraints:- - Transcript deduplication relies on message equality. A value that appears- in more than one observation must be shortened identically everywhere,- or not at all.- - Substring relations matter too. If one message quotes another (a user- turn that says "Rate this joke: <joke>"), shorten the quoted text and the- quoting text consistently, or leave both intact.- - Never change roles, part types, ids, ordering, or the number of- messages.--8. **Redact consistently.** Replace personal data (emails, real names, keys)- with stable stand-ins, applied to every occurrence. Seeded demo data needs- no redaction.--9. **Define the expectation.** Review message order, thread boundaries, tool- results, and provenance against the input. Store a static `expected`- transcript; do not derive it from the transcript builder in the test.--10. **Write the description.** Two to five sentences on what makes this tree- interesting for transcript semantics: history replay or its absence,- where tool results live, disjoint generations, repeated system prompts,- non-message I/O, unusual nesting, export order quirks.--11. **Register and verify.** Add the export to `traceTranscriptFixtures` or- `sessionTranscriptFixtures` in `fixtures/index.ts`, then run:-- ```bash- pnpm --filter @langfuse/shared run test src/utils/transcript/fixtures/fixtures.test.ts- ```-- ```bash- pnpm --filter @langfuse/shared run typecheck- ```--## Template--```ts-import type { TranscriptObservation } from "../../types";-import type { TranscriptFixture } from "../fixture-types";--/**- * <One line: what the application does in this trace. Where it came from.>- *- * <Tree, as described in step 1.>- */--const traceId = "β¦";-const sessionId = "β¦";--// Hoisted repeated values (step 6).--const metadata = {- /* β¦ */-};--const common = { traceId, sessionId, metadata };--const observations: TranscriptObservation[] = [- {- ...common,- id: "β¦",- parentObservationId: null,- type: "SPAN",- name: "β¦",- startTime: "β¦",- endTime: "β¦",- input: JSON.stringify({- /* β¦ */- }),- output: JSON.stringify({- /* β¦ */- }),- },- // β¦-];--export const <camelCaseName>Fixture = {- name: "<tree shape in a few words>",- scope: "trace",- description: "<step 10>",- observations,- expected: { threads: [/* reviewed threads and messages */] },-} satisfies TranscriptFixture;-```--## Checklist before handing back--- [ ] Tree comment present and consistent with `parentObservationId`.-- [ ] Every observation from the export is present, in export order.-- [ ] `""` parent ids became `null`.-- [ ] Every `input` / `output` that was a JSON string is `JSON.stringify(...)`- of a literal; nested JSON strings are still strings.-- [ ] Repeated values are hoisted; shortened values are shortened everywhere.-- [ ] `level`, `statusMessage`, `model` only where the rules say.-- [ ] Explicit, reviewed `expected` transcript (or `null` for no transcript).-- [ ] Registered in `fixtures/index.ts`; integrity test and typecheck pass.diff --git a/packages/shared/src/server/transcript/fixtures/fixture-types.ts b/packages/shared/src/server/transcript/fixtures/fixture-types.tsindex aaf5eded27..bc781338f3 100644--- a/packages/shared/src/server/transcript/fixtures/fixture-types.ts+++ b/packages/shared/src/server/transcript/fixtures/fixture-types.ts@@ -2,25 +2,25 @@ import { createObservation } from "../../test-utils";import type { Transcript } from "../types";/**- * One transcript test case: a full observation tree as `createObservation`- * seed records, plus the transcript a human expects the builder to return.- * The behavior test completes each seed with `createObservation` and converts- * it to a domain `Observation` with `convertObservation` before building.+ * One transcript test case: one trace as `createObservation` seed records,+ * plus the transcript a human expects the builder to return. The behavior+ * test completes each seed with `createObservation` and converts it to a+ * domain `Observation` with `convertObservation` before building.*- * Expectations pin reviewed message ordering, thread boundaries, and provenance.+ * Expectations pin reviewed message ordering, thread boundaries, the split+ * between conversation history and current turn, and provenance.*/export type TranscriptFixture = {/** Test name. Describes the tree shape the case exercises. */name: string;- /** `trace`: one trace. `session`: every trace of one session. */- scope: "trace" | "session";/** Why this tree is interesting for transcript semantics. */description: string;/**- * Every observation of the trace(s), including types the builder ignores,- * in the order the export lists them. The builder must not rely on input+ * Every observation of the trace, including types the builder ignores, in+ * the order the export lists them. The builder must not rely on input* order, so fixtures never re-sort.*/observations: Parameters<typeof createObservation>[0][];+ /** The transcript of the trace, or `null` when it yields none. */expected: Transcript | null;};diff --git a/packages/shared/src/server/transcript/fixtures/fixtures.test.ts b/packages/shared/src/server/transcript/fixtures/fixtures.test.tsindex ce67367bf0..be9772c61c 100644--- a/packages/shared/src/server/transcript/fixtures/fixtures.test.ts+++ b/packages/shared/src/server/transcript/fixtures/fixtures.test.ts@@ -2,19 +2,20 @@ import { describe, expect, it } from "vitest";import { transcriptFixtures } from "./index";import { createObservation } from "../../test-utils";import { convertObservation } from "../../repositories/observations_converters";-import { getTranscript } from "../index";+import { orderObservations } from "../ordering";+import { assembleTranscript } from "../transcript";import { formatTranscript } from "./format-transcript";-import { orderSupportRoutingFixture } from "./session/order-support-routing";/*** Structural checks on fixtures and exact transcript expectations.*/describe("transcript fixtures", () => {+ const traceId = "trace";const generation = (id: string, input: string[], output: string[]) =>convertObservation(createObservation({id,- trace_id: `trace-${id}`,+ trace_id: traceId,type: "GENERATION",start_time: `2026-01-01T12:00:0${id}.000Z`,input: JSON.stringify(@@ -28,40 +29,44 @@ describe("transcript fixtures", () => {it("skips generations without messages", () => {const empty = generation("1", [], []);- expect(getTranscript([empty])).toBeNull();- const transcript = getTranscript([empty, generation("2", ["A"], [])]);+ expect(assembleTranscript([empty])).toBeNull();+ const transcript = assembleTranscript([empty, generation("2", ["A"], [])]);expect(transcript?.threads).toHaveLength(1);- expect(transcript?.threads[0].observations).toEqual([- { id: "2", traceId: "trace-2" },+ expect(transcript?.threads[0].currentTurn.observations).toEqual([+ { id: "2", traceId },]);});it("continues the newest matching thread without duplicating history", () => {- const transcript = getTranscript([+ const transcript = assembleTranscript([generation("1", ["A"], []),generation("2", ["B"], []),generation("3", ["B", "A"], ["C"]),]);- expect(transcript?.threads.map((thread) => thread.observations)).toEqual([- [{ id: "1", traceId: "trace-1" }],+ expect(+ transcript?.threads.map((thread) => thread.currentTurn.observations),+ ).toEqual([+ [{ id: "1", traceId }],[- { id: "2", traceId: "trace-2" },- { id: "3", traceId: "trace-3" },+ { id: "2", traceId },+ { id: "3", traceId },],]);expect(- transcript?.threads[1].messages.map((message) => message.observationId),+ transcript?.threads[1].currentTurn.messages.map(+ (message) => message.observationId,+ ),).toEqual(["2", "3", "3"]);});it("does not list a replay-only generation as a contributor", () => {- const transcript = getTranscript([+ const transcript = assembleTranscript([generation("1", ["A"], ["B"]),generation("2", ["A", "B"], []),]);expect(transcript?.threads).toHaveLength(1);- expect(transcript?.threads[0].observations).toEqual([- { id: "1", traceId: "trace-1" },+ expect(transcript?.threads[0].currentTurn.observations).toEqual([+ { id: "1", traceId },]);});@@ -80,38 +85,27 @@ describe("transcript fixtures", () => {expect(new Set(ids).size).toBe(ids.length);});- it("matches its scope", () => {- if (fixture.scope === "trace") {- expect(traceIds.size).toBe(1);- return;- }-- // Session scope spans several traces; session identity lives on the- // trace, not on observation records, so we only assert the shape here.- expect(traceIds.size).toBeGreaterThan(1);+ it("belongs to one trace", () => {+ expect(traceIds.size).toBe(1);});// Complete each seed into a full ClickHouse observation record and convert// it to a domain `Observation`, then build the transcript.- // Routing history is an opaque string; its transcript expectation is deferred.- it.skipIf(fixture === orderSupportRoutingFixture)(- "returns the expected transcript",- () => {- const observations = fixture.observations.map((observation) =>- convertObservation(createObservation(observation)),- );- const transcript = getTranscript(observations);+ it("returns the expected transcript", () => {+ const observations = fixture.observations.map((observation) =>+ convertObservation(createObservation(observation)),+ );+ const transcript = assembleTranscript(orderObservations(observations));- console.log("----------Formatted Transcript-------------------");- console.log(- formatTranscript(fixture.name, transcript, observations, {- hideReasoning: true,- }),- );- console.log("-------------------------------------------------");+ console.log("----------Formatted Transcript-------------------");+ console.log(+ formatTranscript(fixture.name, transcript, observations, {+ hideReasoning: true,+ }),+ );+ console.log("-------------------------------------------------");- expect(transcript).toEqual(fixture.expected);- },- );+ expect(transcript).toEqual(fixture.expected);+ });});});diff --git a/packages/shared/src/server/transcript/fixtures/format-transcript.ts b/packages/shared/src/server/transcript/fixtures/format-transcript.tsindex 850d048a24..3646739e54 100644--- a/packages/shared/src/server/transcript/fixtures/format-transcript.ts+++ b/packages/shared/src/server/transcript/fixtures/format-transcript.ts@@ -1,5 +1,6 @@import type { Observation } from "../../../domain";import type {+ NormalizedMessage,NormalizedMessagePart,NormalizedMessageRole,} from "../../../utils/normalized-io";@@ -65,8 +66,9 @@ function partLines(part: NormalizedMessagePart): string[] {/*** Chat-shaped rendering of a transcript for eyeballing during development:- * one block per message with a role label and the emitting generation, tool- * calls and results as parameter lists, everything else as text.+ * conversation history first, then one block per current-turn message with a+ * role label and the emitting generation; tool calls and results as parameter+ * lists, everything else as text.*/export function formatTranscript(title: string,@@ -87,26 +89,35 @@ export function formatTranscript(return lines.join("\n");}- transcript.threads.forEach((thread, threadIndex) => {+ const pushMessage = (message: NormalizedMessage, origin: string) => {+ const parts = options.hideReasoning+ ? message.parts.filter((part) => part.type !== "reasoning")+ : message.parts;+ if (options.hideReasoning && parts.length === 0) return;+ const sender = message.senderName ? ` (${message.senderName})` : "";lines.push(- "",- `β thread ${threadIndex + 1} Β· trace(s) ${[...new Set(thread.observations.map(({ traceId }) => traceId))].join(", ")}`,- `β generations: ${thread.observations.map(({ id }) => label(id)).join(", ")}`,+ "β",+ `β ${ROLE_LABEL[message.role]}${sender} Β· ${origin} Β· ${message.source}`,);+ for (const part of parts) {+ for (const line of partLines(part)) lines.push(`β ${line}`);+ }+ };- for (const message of thread.messages) {- const parts = options.hideReasoning- ? message.parts.filter((part) => part.type !== "reasoning")- : message.parts;- if (options.hideReasoning && parts.length === 0) continue;- const sender = message.senderName ? ` (${message.senderName})` : "";- lines.push(- "β",- `β ${ROLE_LABEL[message.role]}${sender} Β· ${label(message.observationId)} Β· ${message.source}`,- );- for (const part of parts) {- for (const line of partLines(part)) lines.push(`β ${line}`);- }+ transcript.threads.forEach(({ conversationHistory, currentTurn }, index) => {+ lines.push(+ "",+ `β thread ${index + 1} Β· trace(s) ${[...new Set(currentTurn.observations.map(({ traceId }) => traceId))].join(", ")}`,+ `β generations: ${currentTurn.observations.map(({ id }) => label(id)).join(", ")}`,+ );+ if (conversationHistory.length) {+ lines.push("β", "β ββ conversation history ββ");+ for (const message of conversationHistory)+ pushMessage(message, "history");+ lines.push("β", "β ββ current turn ββ");+ }+ for (const message of currentTurn.messages) {+ pushMessage(message, label(message.observationId));}lines.push("β");});diff --git a/packages/shared/src/server/transcript/fixtures/index.ts b/packages/shared/src/server/transcript/fixtures/index.tsindex 27d23dcd71..86285578ca 100644--- a/packages/shared/src/server/transcript/fixtures/index.ts+++ b/packages/shared/src/server/transcript/fixtures/index.ts@@ -1,32 +1,20 @@import type { TranscriptFixture } from "./fixture-types";-import { cumulativeHistoryFixture } from "./session/cumulative-history";-import { orderSupportRoutingFixture } from "./session/order-support-routing";-import { reorderedHistoryFixture } from "./session/reordered-history";-import { supportCopilotFollowUpFixture } from "./session/support-copilot-follow-up";import { openaiAgentsSpanishHandoffFixture } from "./trace/openai-agents-spanish-handoff";import { vercelAiSdkDocsToolLoopFixture } from "./trace/vercel-ai-sdk-docs-tool-loop";import { openaiAgentsJokeAndRatingFixture } from "./trace/openai-agents-joke-and-rating";import { supportCopilotRefundLoopFixture } from "./trace/support-copilot-refund-loop";+import { supportCopilotFollowUpFixture } from "./trace/support-copilot-follow-up";import { standaloneToolObservationFixture } from "./trace/standalone-tool-observation";+import { inheritedConversationHistoryFixture } from "./trace/inherited-conversation-history";export type { TranscriptFixture } from "./fixture-types";-const traceTranscriptFixtures: TranscriptFixture[] = [+export const transcriptFixtures: TranscriptFixture[] = [standaloneToolObservationFixture,openaiAgentsSpanishHandoffFixture,vercelAiSdkDocsToolLoopFixture,openaiAgentsJokeAndRatingFixture,supportCopilotRefundLoopFixture,-];--const sessionTranscriptFixtures: TranscriptFixture[] = [supportCopilotFollowUpFixture,- orderSupportRoutingFixture,- cumulativeHistoryFixture,- reorderedHistoryFixture,-];--export const transcriptFixtures: TranscriptFixture[] = [- ...traceTranscriptFixtures,- ...sessionTranscriptFixtures,+ inheritedConversationHistoryFixture,];diff --git a/packages/shared/src/server/transcript/fixtures/session/cumulative-history.ts b/packages/shared/src/server/transcript/fixtures/session/cumulative-history.tsdeleted file mode 100644index d6657e55df..0000000000--- a/packages/shared/src/server/transcript/fixtures/session/cumulative-history.ts+++ /dev/null@@ -1,128 +0,0 @@-import type { TranscriptFixture } from "../fixture-types";--/**- * Two root generations in separate traces of one synthetic session.- * Based on SessionConversationTimeline/fns/processTimelineMessages.clienttest.ts.- */-export const cumulativeHistoryFixture = {- name: "Cumulative history across two session traces",- scope: "session",- description:- "Adapted from Ben's cumulative-history test. Expect one thread with the initial exchange and follow-up once each; replayed messages retain the first trace and generation IDs.",- observations: [- {- id: "cumulative-history-generation-1",- trace_id: "cumulative-history-trace-1",- project_id: "transcript-fixture-project",- parent_observation_id: null,- type: "GENERATION",- name: "Session turn 1",- start_time: "2026-01-01T12:00:00.000Z",- end_time: "2026-01-01T12:00:01.000Z",- input: JSON.stringify([- {- role: "user",- content: "Initial request",- },- ]),- output: JSON.stringify({- role: "assistant",- content: "First response",- }),- },- {- id: "cumulative-history-generation-2",- trace_id: "cumulative-history-trace-2",- project_id: "transcript-fixture-project",- parent_observation_id: null,- type: "GENERATION",- name: "Session turn 2",- start_time: "2026-01-01T12:01:00.000Z",- end_time: "2026-01-01T12:01:01.000Z",- input: JSON.stringify([- {- role: "user",- content: "Initial request",- },- {- role: "assistant",- content: "First response",- },- {- role: "user",- content: "Follow-up request",- },- ]),- output: JSON.stringify({- role: "assistant",- content: "Second response",- }),- },- ],- expected: {- threads: [- {- messages: [- {- role: "user",- parts: [- {- type: "text",- text: "Initial request",- },- ],- source: "input",- observationId: "cumulative-history-generation-1",- traceId: "cumulative-history-trace-1",- },- {- role: "assistant",- parts: [- {- type: "text",- text: "First response",- },- ],- source: "output",- observationId: "cumulative-history-generation-1",- traceId: "cumulative-history-trace-1",- },- {- role: "user",- parts: [- {- type: "text",- text: "Follow-up request",- },- ],- source: "input",- observationId: "cumulative-history-generation-2",- traceId: "cumulative-history-trace-2",- },- {- role: "assistant",- parts: [- {- type: "text",- text: "Second response",- },- ],- source: "output",- observationId: "cumulative-history-generation-2",- traceId: "cumulative-history-trace-2",- },- ],- observations: [- {- id: "cumulative-history-generation-1",- traceId: "cumulative-history-trace-1",- },- {- id: "cumulative-history-generation-2",- traceId: "cumulative-history-trace-2",- },- ],- },- ],- },-} satisfies TranscriptFixture;diff --git a/packages/shared/src/server/transcript/fixtures/session/order-support-routing.ts b/packages/shared/src/server/transcript/fixtures/session/order-support-routing.tsdeleted file mode 100644index 99fb470b7f..0000000000--- a/packages/shared/src/server/transcript/fixtures/session/order-support-routing.ts+++ /dev/null@@ -1,219 +0,0 @@-import type { TranscriptFixture } from "../fixture-types";--/**- * Synthetic reconstruction: IDs, timestamps, names and payloads are placeholders.- * Each trace has this hierarchy (optional children noted):- * [SPAN] Support assistant β user/assistant text- * [SPAN] Gather context- * [SPAN] Respond to request- * [SPAN] Escalation classification- * [GENERATION] Classify escalation β system/user β structured classification- * [GENERATION] Analyze (trace 1 only; I/O unknown)- * [SPAN] Check incident- * [GENERATION] Select workflow (trace 1 only) β user/system β tool call- * [SPAN] Run order workflow- * [GENERATION] Manage customer order (trace 1) β system/user β tool call- * [GENERATION] Clarify issue (trace 3) β system/user β text- * All non-root spans intentionally have undefined I/O.- */--type Seed = TranscriptFixture["observations"][number];--function observation(- turn: number,- id: string,- parent: string | null,- name: string,- offset: number,- details: Partial<Seed> = {},-): Seed {- return {- project_id: "transcript-fixture-project",- trace_id: `order-support-trace-${turn}`,- id: `order-support-${turn}-${id}`,- parent_observation_id: parent ? `order-support-${turn}-${parent}` : null,- type: "SPAN",- name,- start_time: new Date(Date.UTC(2026, 0, 1, 12, turn, offset)).toISOString(),- end_time: new Date(Date.UTC(2026, 0, 1, 12, turn, 20)).toISOString(),- input: undefined,- output: undefined,- ...details,- };-}--const classification = {- role: "assistant",- tool_calls: [],- content: { classification: "NO_ESCALATION_REQUIRED" },-};-const conversation =- "USER: Issue with my order --- ASSISTANT: Select the order you need help with: --- ASSISTANT: User is shown the following orders to pick from: .... --- USER: Item 123.";--function baseTrace(turn: number, userContent: unknown): Seed[] {- return [- observation(turn, "root", null, "Support assistant", 0, {- input: "user text",- output: "assistant text",- }),- observation(turn, "context", "root", "Gather context", 1),- observation(turn, "respond", "root", "Respond to request", 2),- observation(turn, "escalation", "respond", "Escalation classification", 3),- observation(turn, "classify", "escalation", "Classify escalation", 4, {- type: "GENERATION",- input: JSON.stringify([- { role: "system", content: "instruction 1" },- { role: "user", content: userContent },- ]),- output: JSON.stringify(classification),- }),- ];-}--export const orderSupportRoutingFixture = {- name: "Order support routing across three session traces",- scope: "session",- description:- "Synthetic three-turn session with classification, workflow selection and tool calls. Later classifiers embed conversation history in a structured user message rather than replaying message arrays. Root text is on spans, one generation has unknown I/O, and separate generations use different system instructions; transcript expectations remain open.",- observations: [- ...baseTrace(1, "question"),- observation(1, "analyze", "respond", "Analyze utterance", 5, {- type: "GENERATION",- }),- observation(1, "incident", "respond", "Check incident", 6),- observation(1, "select", "respond", "Select workflow", 7, {- type: "GENERATION",- input: JSON.stringify({- messages: [- { role: "user", content: "question" },- { role: "system", content: "instruction 2" },- ],- tools: [- {- type: "function",- function: {- parameters: {},- strict: false,- description: "Tool to manage customer order",- name: "manage_customer_order",- },- },- {- type: "function",- function: {- parameters: {},- strict: false,- description:- "Default tool provided out of the box. Use it if specialized tools are less applicable or unavailable.",- name: "answer_generation",- },- },- ],- }),- output: JSON.stringify({- role: "assistant",- tool_calls: [- {- id: "call_1234567890",- type: "function",- function: { name: "manage_customer_order", arguments: {} },- },- ],- content: "",- }),- }),- observation(1, "workflow", "respond", "Run order workflow", 8),- observation(1, "manage", "workflow", "Manage customer order", 9, {- type: "GENERATION",- input: JSON.stringify({- messages: [- { role: "system", content: "Instructions 3" },- { role: "user", content: "Issue with my order" },- ],- tools: [- {- type: "function",- function: {- parameters: {- type: "object",- required: [],- additionalProperties: false,- properties: {},- },- strict: true,- description: "Retrieves a list of orders for the current user",- name: "findUserOrders",- },- },- {- type: "function",- function: {- parameters: {- type: "object",- required: ["orderId"],- additionalProperties: false,- properties: {- orderId: {- description: "The ID of the order to select",- type: "string",- },- },- },- strict: true,- description: "Selects an order from data store.",- name: "selectUserOrder",- },- },- {- type: "function",- function: {- parameters: {- type: "object",- required: [],- additionalProperties: false,- properties: {},- },- strict: true,- description: "Answer the user's question",- name: "answerTool",- },- },- ],- }),- output: JSON.stringify({- role: "assistant",- tool_calls: [- {- id: "call_0987654321",- type: "function",- function: { name: "findUserOrders", arguments: {} },- },- ],- content: "",- }),- }),- ...baseTrace(2, { conversation }),- observation(2, "incident", "respond", "Check incident", 6),- observation(2, "workflow", "respond", "Run order workflow", 8),- ...baseTrace(3, {- conversation: `${conversation} --- ASSISTANT: Great, I can help with that. Please explain what assistance you need. --- USER: No more, thank you.`,- }),- observation(3, "incident", "respond", "Check incident", 6),- observation(3, "workflow", "respond", "Run order workflow", 8),- observation(3, "clarify", "workflow", "Clarify issue", 9, {- type: "GENERATION",- input: JSON.stringify([- { role: "system", content: "instruction 4" },- { role: "user", content: "No more, thank you." },- ]),- output: JSON.stringify({- role: "assistant",- tool_calls: [],- content: "false-positives",- }),- }),- ],- // Opaque conversation strings do not replay structured message history.- // Keep this example for reference; transcript expectations are deferred.- expected: null,-} satisfies TranscriptFixture;diff --git a/packages/shared/src/server/transcript/fixtures/session/reordered-history.ts b/packages/shared/src/server/transcript/fixtures/session/reordered-history.tsdeleted file mode 100644index f63477838a..0000000000--- a/packages/shared/src/server/transcript/fixtures/session/reordered-history.ts+++ /dev/null@@ -1,149 +0,0 @@-import type { TranscriptFixture } from "../fixture-types";--/**- * Two root generations in separate traces of one synthetic session.- * Based on SessionConversationTimeline/fns/processTimelineMessages.clienttest.ts.- */-export const reorderedHistoryFixture = {- name: "Reordered history across two session traces",- scope: "session",- description:- "Desired result is one thread with A, B, C, New, Answer; this deliberately exercises order-insensitive reconciliation beyond prefix matching.",- observations: [- {- id: "reordered-history-generation-1",- trace_id: "reordered-history-trace-1",- project_id: "transcript-fixture-project",- parent_observation_id: null,- type: "GENERATION",- name: "Session turn 1",- start_time: "2026-01-01T12:00:00.000Z",- end_time: "2026-01-01T12:00:01.000Z",- input: JSON.stringify([- {- role: "user",- content: "A",- },- {- role: "user",- content: "B",- },- {- role: "user",- content: "C",- },- ]),- output: null,- },- {- id: "reordered-history-generation-2",- trace_id: "reordered-history-trace-2",- project_id: "transcript-fixture-project",- parent_observation_id: null,- type: "GENERATION",- name: "Session turn 2",- start_time: "2026-01-01T12:01:00.000Z",- end_time: "2026-01-01T12:01:01.000Z",- input: JSON.stringify([- {- role: "user",- content: "B",- },- {- role: "user",- content: "C",- },- {- role: "user",- content: "A",- },- {- role: "user",- content: "New",- },- ]),- output: JSON.stringify({- role: "assistant",- content: "Answer",- }),- },- ],- expected: {- threads: [- {- messages: [- {- role: "user",- parts: [- {- type: "text",- text: "A",- },- ],- source: "input",- observationId: "reordered-history-generation-1",- traceId: "reordered-history-trace-1",- },- {- role: "user",- parts: [- {- type: "text",- text: "B",- },- ],- source: "input",- observationId: "reordered-history-generation-1",- traceId: "reordered-history-trace-1",- },- {- role: "user",- parts: [- {- type: "text",- text: "C",- },- ],- source: "input",- observationId: "reordered-history-generation-1",- traceId: "reordered-history-trace-1",- },- {- role: "user",- parts: [- {- type: "text",- text: "New",- },- ],- source: "input",- observationId: "reordered-history-generation-2",- traceId: "reordered-history-trace-2",- },- {- role: "assistant",- parts: [- {- type: "text",- text: "Answer",- },- ],- source: "output",- observationId: "reordered-history-generation-2",- traceId: "reordered-history-trace-2",- },- ],- observations: [- {- id: "reordered-history-generation-1",- traceId: "reordered-history-trace-1",- },- {- id: "reordered-history-generation-2",- traceId: "reordered-history-trace-2",- },- ],- },- ],- },-} satisfies TranscriptFixture;diff --git a/packages/shared/src/server/transcript/fixtures/session/support-copilot-follow-up.ts b/packages/shared/src/server/transcript/fixtures/session/support-copilot-follow-up.tsdeleted file mode 100644index b541d4dc58..0000000000--- a/packages/shared/src/server/transcript/fixtures/session/support-copilot-follow-up.ts+++ /dev/null@@ -1,74 +0,0 @@-import type { TranscriptFixture } from "../fixture-types";-import { supportCopilotRefundLoopFixture } from "../trace/support-copilot-refund-loop";--const firstTrace = supportCopilotRefundLoopFixture;-const firstThread = firstTrace.expected.threads[0];-const draft = firstTrace.observations.find((observation) =>- observation.id.endsWith("-drf"),-)!;-const completedHistory = [- ...JSON.parse(draft.input!).messages,- JSON.parse(draft.output!),-];--// Repeat the same workflow for another charge, with distinct trace/call IDs.-function followUpCopy<T>(value: T): T {- return JSON.parse(- JSON.stringify(value)- .replaceAll("sa2026083102o", "sa2026083102p")- .replaceAll("call_find_2", "call_find_3")- .replaceAll("call_act_2", "call_act_3")- .replaceAll("ch_3PqK", "ch_4PqK")- .replaceAll("re_18Lk1Q", "re_28Lk1Q")- .replaceAll("inv_20260709", "inv_20260809")- .replaceAll(- "Hi β I was charged twice",- "Thanks for resolving that. I was also charged twice",- )- .replaceAll("2026-07", "2026-08")- .replaceAll("2026-08-31T12:33:", "2026-08-31T12:35:"),- );-}--const followUpObservations = firstTrace.observations.map((original) => {- const observation = followUpCopy(original);- // Both traces belong to the same project.- observation.project_id = original.project_id;- if (observation.type === "GENERATION") {- const input = JSON.parse(observation.input!);- observation.input = JSON.stringify({- ...input,- messages: [- ...completedHistory,- ...input.messages.filter(- (message: { role: string }) => message.role !== "system",- ),- ],- });- }- return observation;-});--export const supportCopilotFollowUpFixture = {- name: "support copilot follow-up replays history across two traces",- scope: "session",- description:- "Two complete refund workflows share a conversation. The second trace replays the first trace's final history in every generation; repeated history retains its first attribution, and new calls/results retain the second trace's IDs.",- observations: [...firstTrace.observations, ...followUpObservations],- expected: {- threads: [- {- observations: [- ...firstThread.observations,- ...followUpCopy(firstThread.observations),- ],- messages: [- ...firstThread.messages,- ...followUpCopy(- firstThread.messages.filter((message) => message.role !== "system"),- ),- ],- },- ],- },-} satisfies TranscriptFixture;diff --git a/packages/shared/src/server/transcript/fixtures/trace/inherited-conversation-history.ts b/packages/shared/src/server/transcript/fixtures/trace/inherited-conversation-history.tsnew file mode 100644index 0000000000..f8a4b414b3--- /dev/null+++ b/packages/shared/src/server/transcript/fixtures/trace/inherited-conversation-history.ts@@ -0,0 +1,74 @@+import type { TranscriptFixture } from "../fixture-types";++/**+ * One root generation whose input replays an earlier exchange before the new+ * user message. Based on SessionConversationTimeline/fns/processTimelineMessages.clienttest.ts.+ */+export const inheritedConversationHistoryFixture = {+ name: "Inherited conversation history before a follow-up request",+ description:+ "The generation receives the first exchange and then a new request. The replay up to the assistant's reply becomes the conversation history; the new exchange is the current turn.",+ observations: [+ {+ id: "inherited-history-generation",+ trace_id: "inherited-history-trace",+ project_id: "transcript-fixture-project",+ parent_observation_id: null,+ type: "GENERATION",+ name: "Session turn 2",+ start_time: "2026-01-01T12:01:00.000Z",+ end_time: "2026-01-01T12:01:01.000Z",+ input: JSON.stringify([+ { role: "user", content: "Initial request" },+ { role: "assistant", content: "First response" },+ { role: "user", content: "Follow-up request" },+ ]),+ output: JSON.stringify({+ role: "assistant",+ content: "Second response",+ }),+ },+ ],+ expected: {+ threads: [+ {+ conversationHistory: [+ {+ role: "user",+ parts: [{ type: "text", text: "Initial request" }],+ source: "input",+ },+ {+ role: "assistant",+ parts: [{ type: "text", text: "First response" }],+ source: "input",+ },+ ],+ currentTurn: {+ messages: [+ {+ role: "user",+ parts: [{ type: "text", text: "Follow-up request" }],+ source: "input",+ observationId: "inherited-history-generation",+ traceId: "inherited-history-trace",+ },+ {+ role: "assistant",+ parts: [{ type: "text", text: "Second response" }],+ source: "output",+ observationId: "inherited-history-generation",+ traceId: "inherited-history-trace",+ },+ ],+ observations: [+ {+ id: "inherited-history-generation",+ traceId: "inherited-history-trace",+ },+ ],+ },+ },+ ],+ },+} satisfies TranscriptFixture;diff --git a/packages/shared/src/server/transcript/fixtures/trace/openai-agents-joke-and-rating.ts b/packages/shared/src/server/transcript/fixtures/trace/openai-agents-joke-and-rating.tsindex 37a6e396d2..fbfc35397e 100644--- a/packages/shared/src/server/transcript/fixtures/trace/openai-agents-joke-and-rating.ts+++ b/packages/shared/src/server/transcript/fixtures/trace/openai-agents-joke-and-rating.ts@@ -203,7 +203,6 @@ const generatorAttributes = JSON.stringify({export const openaiAgentsJokeAndRatingFixture = {name: "OpenAI Agents joke generation followed by rating",- scope: "trace",description:"Separate agent executions generate a joke and rate it. The second prompt quotes the joke inside a fresh user message rather than replaying the first assistant message.",observations: [@@ -371,80 +370,86 @@ export const openaiAgentsJokeAndRatingFixture = {expected: {threads: [{- messages: [- {- role: "user",- parts: [- {- type: "text",- text: "Tell me a joke",- },- ],- source: "input",- observationId: "f8fdd8164d6f8dbe",- traceId: "afa1ae379b6c3a5333e2bab357f31cdc",- },- {- id: "msg_05e10c6f9849abe80068db9d07270c81a0ab2f2738f61c1d11",- role: "assistant",- parts: [- {- type: "text",- text: "Sure! Hereβs one for you:\n\nWhy did the scarecrow win an award?\n\nBecause he was outstanding in his field! πΎπ",- providerMetadata: {- logprobs: [],+ conversationHistory: [],+ currentTurn: {+ messages: [+ {+ role: "user",+ parts: [+ {+ type: "text",+ text: "Tell me a joke",},- },- ],- source: "output",- observationId: "f8fdd8164d6f8dbe",- traceId: "afa1ae379b6c3a5333e2bab357f31cdc",- },- ],- observations: [- {- id: "f8fdd8164d6f8dbe",- traceId: "afa1ae379b6c3a5333e2bab357f31cdc",- },- ],+ ],+ source: "input",+ observationId: "f8fdd8164d6f8dbe",+ traceId: "afa1ae379b6c3a5333e2bab357f31cdc",+ },+ {+ id: "msg_05e10c6f9849abe80068db9d07270c81a0ab2f2738f61c1d11",+ role: "assistant",+ parts: [+ {+ type: "text",+ text: "Sure! Hereβs one for you:\n\nWhy did the scarecrow win an award?\n\nBecause he was outstanding in his field! πΎπ",+ providerMetadata: {+ logprobs: [],+ },+ },+ ],+ source: "output",+ observationId: "f8fdd8164d6f8dbe",+ traceId: "afa1ae379b6c3a5333e2bab357f31cdc",+ },+ ],+ observations: [+ {+ id: "f8fdd8164d6f8dbe",+ traceId: "afa1ae379b6c3a5333e2bab357f31cdc",+ },+ ],+ },},{- messages: [- {- role: "user",- parts: [- {- type: "text",- text: "Rate this joke: Sure! Hereβs one for you:\n\nWhy did the scarecrow win an award?\n\nBecause he was outstanding in his field! πΎπ",- },- ],- source: "input",- observationId: "496efdc060a5399b",- traceId: "afa1ae379b6c3a5333e2bab357f31cdc",- },- {- id: "msg_03e1816217fbc8a30068db9d07dbe481a28fae7f2a41f76827",- role: "assistant",- parts: [- {- type: "text",- text: "Great choice! Thatβs a classic jokeβpunny, wholesome, and easy to get. Hereβs how Iβd rate it:\n\n**Joke Rating: 8/10**\n\n**Why?**\n- **Punniness:** Strong, farm-fresh wordplay!\n- **Audience:** Universally funnyβworks for kids and adults.\n- **Originality:** Popular, but still a crowd-pleaser.\n\n**Bonus:** The addition of the emoji πΎ gives it an extra playful touch! Want to hear a joke from me now?",- providerMetadata: {- logprobs: [],+ conversationHistory: [],+ currentTurn: {+ messages: [+ {+ role: "user",+ parts: [+ {+ type: "text",+ text: "Rate this joke: Sure! Hereβs one for you:\n\nWhy did the scarecrow win an award?\n\nBecause he was outstanding in his field! πΎπ",},- },- ],- source: "output",- observationId: "496efdc060a5399b",- traceId: "afa1ae379b6c3a5333e2bab357f31cdc",- },- ],- observations: [- {- id: "496efdc060a5399b",- traceId: "afa1ae379b6c3a5333e2bab357f31cdc",- },- ],+ ],+ source: "input",+ observationId: "496efdc060a5399b",+ traceId: "afa1ae379b6c3a5333e2bab357f31cdc",+ },+ {+ id: "msg_03e1816217fbc8a30068db9d07dbe481a28fae7f2a41f76827",+ role: "assistant",+ parts: [+ {+ type: "text",+ text: "Great choice! Thatβs a classic jokeβpunny, wholesome, and easy to get. Hereβs how Iβd rate it:\n\n**Joke Rating: 8/10**\n\n**Why?**\n- **Punniness:** Strong, farm-fresh wordplay!\n- **Audience:** Universally funnyβworks for kids and adults.\n- **Originality:** Popular, but still a crowd-pleaser.\n\n**Bonus:** The addition of the emoji πΎ gives it an extra playful touch! Want to hear a joke from me now?",+ providerMetadata: {+ logprobs: [],+ },+ },+ ],+ source: "output",+ observationId: "496efdc060a5399b",+ traceId: "afa1ae379b6c3a5333e2bab357f31cdc",+ },+ ],+ observations: [+ {+ id: "496efdc060a5399b",+ traceId: "afa1ae379b6c3a5333e2bab357f31cdc",+ },+ ],+ },},],},diff --git a/packages/shared/src/server/transcript/fixtures/trace/openai-agents-spanish-handoff.ts b/packages/shared/src/server/transcript/fixtures/trace/openai-agents-spanish-handoff.tsindex 5024ff3804..5bacf2e97a 100644--- a/packages/shared/src/server/transcript/fixtures/trace/openai-agents-spanish-handoff.ts+++ b/packages/shared/src/server/transcript/fixtures/trace/openai-agents-spanish-handoff.ts@@ -7,7 +7,6 @@ import type { TranscriptFixture } from "../fixture-types";*/export const openaiAgentsSpanishHandoffFixture = {name: "OpenAI Agents Spanish handoff",- scope: "trace",description:"Triage hands off to a Spanish agent. This exercises changing instructions and agent handoff. Duplicate observation/trace ID pairs are removed, preserving first-occurrence order.",observations: [@@ -469,79 +468,82 @@ export const openaiAgentsSpanishHandoffFixture = {expected: {threads: [{- messages: [- {- role: "user",- parts: [- {- type: "text",- text: "Hola, ΒΏcΓ³mo estΓ‘s?",- },- ],- source: "input",- observationId: "94a1dc00d7067ae8",- traceId: "c376e44920527b875add9d97b4ed9312",- },- {- role: "assistant",- parts: [- {- type: "tool-call",- toolCallId: "call_i81vsF01rKH8cDZ0SLQQBUA2",- toolName: "transfer_to_spanish_agent",- input: {},- toolType: "function_call",- providerMetadata: {- status: "completed",+ conversationHistory: [],+ currentTurn: {+ messages: [+ {+ role: "user",+ parts: [+ {+ type: "text",+ text: "Hola, ΒΏcΓ³mo estΓ‘s?",},- },- ],- source: "output",- observationId: "94a1dc00d7067ae8",- traceId: "c376e44920527b875add9d97b4ed9312",- },- {- role: "tool",- source: "output",- parts: [- {- type: "tool-result",- toolCallId: "call_i81vsF01rKH8cDZ0SLQQBUA2",- output: {- assistant: "Spanish agent",+ ],+ source: "input",+ observationId: "94a1dc00d7067ae8",+ traceId: "c376e44920527b875add9d97b4ed9312",+ },+ {+ role: "assistant",+ parts: [+ {+ type: "tool-call",+ toolCallId: "call_i81vsF01rKH8cDZ0SLQQBUA2",+ toolName: "transfer_to_spanish_agent",+ input: {},+ toolType: "function_call",+ providerMetadata: {+ status: "completed",+ },},- },- ],- observationId: "a0391ae77593ad0c",- traceId: "c376e44920527b875add9d97b4ed9312",- },- {- id: "msg_000230c027f4a45e0068db9cf9d14481a3a04289f3f1b0f9a1",- role: "assistant",- parts: [- {- type: "text",- text: "Β‘Hola! Estoy muy bien, gracias. ΒΏY tΓΊ, cΓ³mo estΓ‘s?",- providerMetadata: {- logprobs: [],+ ],+ source: "output",+ observationId: "94a1dc00d7067ae8",+ traceId: "c376e44920527b875add9d97b4ed9312",+ },+ {+ role: "tool",+ source: "output",+ parts: [+ {+ type: "tool-result",+ toolCallId: "call_i81vsF01rKH8cDZ0SLQQBUA2",+ output: {+ assistant: "Spanish agent",+ },},- },- ],- source: "output",- observationId: "a0391ae77593ad0c",- traceId: "c376e44920527b875add9d97b4ed9312",- },- ],- observations: [- {- id: "94a1dc00d7067ae8",- traceId: "c376e44920527b875add9d97b4ed9312",- },- {- id: "a0391ae77593ad0c",- traceId: "c376e44920527b875add9d97b4ed9312",- },- ],+ ],+ observationId: "a0391ae77593ad0c",+ traceId: "c376e44920527b875add9d97b4ed9312",+ },+ {+ id: "msg_000230c027f4a45e0068db9cf9d14481a3a04289f3f1b0f9a1",+ role: "assistant",+ parts: [+ {+ type: "text",+ text: "Β‘Hola! Estoy muy bien, gracias. ΒΏY tΓΊ, cΓ³mo estΓ‘s?",+ providerMetadata: {+ logprobs: [],+ },+ },+ ],+ source: "output",+ observationId: "a0391ae77593ad0c",+ traceId: "c376e44920527b875add9d97b4ed9312",+ },+ ],+ observations: [+ {+ id: "94a1dc00d7067ae8",+ traceId: "c376e44920527b875add9d97b4ed9312",+ },+ {+ id: "a0391ae77593ad0c",+ traceId: "c376e44920527b875add9d97b4ed9312",+ },+ ],+ },},],},diff --git a/packages/shared/src/server/transcript/fixtures/trace/standalone-tool-observation.ts b/packages/shared/src/server/transcript/fixtures/trace/standalone-tool-observation.tsindex 3e7fc7b62a..347ebc465e 100644--- a/packages/shared/src/server/transcript/fixtures/trace/standalone-tool-observation.ts+++ b/packages/shared/src/server/transcript/fixtures/trace/standalone-tool-observation.ts@@ -3,7 +3,6 @@ import type { TranscriptFixture } from "../fixture-types";// Tool execution is recorded only as a TOOL observation, not a model tool call.export const standaloneToolObservationFixture = {name: "Standalone tool execution without a generation tool call",- scope: "trace",description:"The generation contains only text. A subsequent TOOL observation has input and output but no matching generation tool call, so its result is omitted from the transcript.",observations: [@@ -41,25 +40,28 @@ export const standaloneToolObservationFixture = {expected: {threads: [{- messages: [- {- role: "user",- parts: [{ type: "text", text: "Find my orders." }],- source: "input",- observationId: "standalone-generation",- traceId: "standalone-tool-trace",- },- {- role: "assistant",- parts: [{ type: "text", text: "I'll look up your orders." }],- source: "output",- observationId: "standalone-generation",- traceId: "standalone-tool-trace",- },- ],- observations: [- { id: "standalone-generation", traceId: "standalone-tool-trace" },- ],+ conversationHistory: [],+ currentTurn: {+ messages: [+ {+ role: "user",+ parts: [{ type: "text", text: "Find my orders." }],+ source: "input",+ observationId: "standalone-generation",+ traceId: "standalone-tool-trace",+ },+ {+ role: "assistant",+ parts: [{ type: "text", text: "I'll look up your orders." }],+ source: "output",+ observationId: "standalone-generation",+ traceId: "standalone-tool-trace",+ },+ ],+ observations: [+ { id: "standalone-generation", traceId: "standalone-tool-trace" },+ ],+ },},],},diff --git a/packages/shared/src/server/transcript/fixtures/trace/support-copilot-follow-up.ts b/packages/shared/src/server/transcript/fixtures/trace/support-copilot-follow-up.tsnew file mode 100644index 0000000000..0527d0f515--- /dev/null+++ b/packages/shared/src/server/transcript/fixtures/trace/support-copilot-follow-up.ts@@ -0,0 +1,167 @@+import type { NormalizedMessage } from "../../../../utils/normalized-io";+import type { TranscriptFixture } from "../fixture-types";+import {+ charges,+ classification,+ copilotSystemPrompt,+ createRefundArguments,+ createRefundToolCallId,+ customerMessage,+ finalReply,+ findChargesArguments,+ findChargesToolCallId,+ refund,+ supportCopilotRefundLoopFixture,+} from "./support-copilot-refund-loop";++const firstTrace = supportCopilotRefundLoopFixture;+const firstThread = firstTrace.expected.threads[0];+const draft = firstTrace.observations.find((observation) =>+ observation.id.endsWith("-drf"),+)!;+const completedHistory = [+ ...JSON.parse(draft.input!).messages,+ JSON.parse(draft.output!),+];++// Repeat the same workflow for another charge, with distinct trace/call IDs.+function followUpCopy<T>(value: T): T {+ return JSON.parse(+ JSON.stringify(value)+ .replaceAll("sa2026083102o", "sa2026083102p")+ .replaceAll("call_find_2", "call_find_3")+ .replaceAll("call_act_2", "call_act_3")+ .replaceAll("ch_3PqK", "ch_4PqK")+ .replaceAll("re_18Lk1Q", "re_28Lk1Q")+ .replaceAll("inv_20260709", "inv_20260809")+ .replaceAll(+ "Hi β I was charged twice",+ "Thanks for resolving that. I was also charged twice",+ )+ .replaceAll("2026-07", "2026-08")+ .replaceAll("2026-08-31T12:33:", "2026-08-31T12:35:"),+ );+}++const followUpObservations = firstTrace.observations.map((original) => {+ const observation = followUpCopy(original);+ // Both traces belong to the same project.+ observation.project_id = original.project_id;+ if (observation.type === "GENERATION") {+ const input = JSON.parse(observation.input!);+ observation.input = JSON.stringify({+ ...input,+ messages: [+ ...completedHistory,+ ...input.messages.filter(+ (message: { role: string }) => message.role !== "system",+ ),+ ],+ });+ }+ return observation;+});++/** The first conversation as every generation of the second trace receives it. */+const replayedHistory: NormalizedMessage[] = [+ {+ role: "system",+ source: "input",+ parts: [{ type: "text", text: copilotSystemPrompt }],+ },+ {+ role: "user",+ source: "input",+ parts: [{ type: "text", text: customerMessage }],+ },+ {+ role: "assistant",+ source: "input",+ parts: [{ type: "text", text: classification.content }],+ },+ {+ role: "assistant",+ source: "input",+ parts: [+ {+ type: "tool-call",+ toolCallId: findChargesToolCallId,+ toolName: "stripe_find_charges",+ input: findChargesArguments,+ toolType: "function",+ },+ ],+ },+ {+ role: "tool",+ source: "input",+ parts: [+ {+ type: "tool-result",+ toolCallId: findChargesToolCallId,+ output: charges,+ },+ ],+ },+ {+ role: "assistant",+ source: "input",+ parts: [+ {+ type: "tool-call",+ toolCallId: createRefundToolCallId,+ toolName: "stripe_create_refund",+ input: createRefundArguments,+ toolType: "function",+ },+ ],+ },+ {+ role: "tool",+ source: "input",+ parts: [+ {+ type: "tool-result",+ toolCallId: createRefundToolCallId,+ output: refund,+ },+ ],+ },+ {+ role: "assistant",+ source: "input",+ parts: [{ type: "text", text: "Resolution complete." }],+ },+ {+ role: "user",+ source: "input",+ parts: [{ type: "text", text: "Draft the reply." }],+ },+ {+ role: "assistant",+ source: "input",+ parts: [{ type: "text", text: finalReply }],+ },+];++export const supportCopilotFollowUpFixture = {+ name: "support copilot follow-up inherits the first workflow as history",+ description:+ "A second refund workflow whose generations replay the first trace's final history before the new customer message. That replay becomes the conversation history; the current turn holds only this trace's workflow with its own IDs.",+ observations: followUpObservations,+ expected: {+ threads: [+ {+ conversationHistory: replayedHistory,+ currentTurn: {+ messages: followUpCopy(+ firstThread.currentTurn.messages.filter(+ (message) => message.role !== "system",+ ),+ ),+ observations: followUpCopy(firstThread.currentTurn.observations),+ },+ },+ ],+ },+} satisfies TranscriptFixture;diff --git a/packages/shared/src/server/transcript/fixtures/trace/support-copilot-refund-loop.ts b/packages/shared/src/server/transcript/fixtures/trace/support-copilot-refund-loop.tsindex 41b1e35f65..62a553f6e3 100644--- a/packages/shared/src/server/transcript/fixtures/trace/support-copilot-refund-loop.ts+++ b/packages/shared/src/server/transcript/fixtures/trace/support-copilot-refund-loop.ts@@ -15,26 +15,32 @@ const loadContextId = "sa2026083102o-load";const customerId = "cus_LqT4v8";const refundId = "re_18Lk1Q";-const findChargesToolCallId = "call_find_2";-const createRefundToolCallId = "call_act_2";+export const findChargesToolCallId = "call_find_2";+export const createRefundToolCallId = "call_act_2";-const customerMessage =+export const customerMessage ="Hi β I was charged twice for my subscription this month (invoices inv_20260709 and inv_20260709-2, $99 each). Can you refund the duplicate?";-const copilotSystemPrompt =+export const copilotSystemPrompt ="You are Acme's support copilot. Resolve the customer's billing issue end-to-end using the available tools.";-const finalReply =+export const finalReply ="Hi Jordan, thanks for flagging this! I've confirmed the duplicate $99.00 charge and issued a refund (re_18Lk1Q). You should see it back on your card within 5β10 business days.";// Tool-call arguments as the model emitted them (a JSON string inside the// chat-completions payload) and as the TOOL observation received them.-const findChargesArguments = { customer_id: customerId, period: "2026-07" };-const createRefundArguments = { charge_id: "ch_3PqK9b", reason: "duplicate" };+export const findChargesArguments = {+ customer_id: customerId,+ period: "2026-07",+};+export const createRefundArguments = {+ charge_id: "ch_3PqK9b",+ reason: "duplicate",+};const initialHistory = [{ role: "system", content: copilotSystemPrompt },{ role: "user", content: customerMessage },];-const classification = {+export const classification = {role: "assistant",content: JSON.stringify({intent: "billing.duplicate_charge",@@ -56,7 +62,7 @@ const findChargesCall = {},],};-const charges = {+export const charges = {charges: [{ id: "ch_3PqK8r" }, { id: "ch_3PqK9b" }],duplicate_confidence: 0.98,};@@ -79,7 +85,11 @@ const refundCall = {},],};-const refund = { refund_id: refundId, status: "succeeded", amount_usd: 99 };+export const refund = {+ refund_id: refundId,+ status: "succeeded",+ amount_usd: 99,+};const refundResult = {role: "tool",tool_call_id: createRefundToolCallId,@@ -365,104 +375,112 @@ const observations = [export const supportCopilotRefundLoopFixture = {name: "support copilot refund loop with cumulative conversation history",- scope: "trace",description:"Five generations replay all prior messages. Matching TOOL observations supply authoritative results, while replayed history is deduplicated and retains first-emitter provenance.",observations,expected: {threads: [{- observations: ["cls", "llm1", "find", "llm2", "ref", "llm3", "drf"].map(- (suffix) => ({ id: `sa2026083102o-${suffix}`, traceId }),- ),- messages: [- {- role: "system",- source: "input",- observationId: "sa2026083102o-cls",- traceId,- parts: [{ type: "text", text: copilotSystemPrompt }],- },- {- role: "user",- source: "input",- observationId: "sa2026083102o-cls",- traceId,- parts: [{ type: "text", text: customerMessage }],- },- {- role: "assistant",- source: "output",- observationId: "sa2026083102o-cls",- traceId,- parts: [{ type: "text", text: classification.content }],- },- {- role: "assistant",- source: "output",- observationId: "sa2026083102o-llm1",- traceId,- parts: [- {- type: "tool-call",- toolCallId: findChargesToolCallId,- toolName: "stripe_find_charges",- input: findChargesArguments,- toolType: "function",- },- ],- },- {- role: "tool",- source: "output",- observationId: "sa2026083102o-find",- traceId,- parts: [{ type: "data", value: charges }],- },- {- role: "assistant",- source: "output",- observationId: "sa2026083102o-llm2",- traceId,- parts: [- {- type: "tool-call",- toolCallId: createRefundToolCallId,- toolName: "stripe_create_refund",- input: createRefundArguments,- toolType: "function",- },- ],- },- {- role: "tool",- source: "output",- observationId: "sa2026083102o-ref",- traceId,- parts: [{ type: "data", value: refund }],- },- {- role: "assistant",- source: "output",- observationId: "sa2026083102o-llm3",- traceId,- parts: [{ type: "text", text: "Resolution complete." }],- },- {- role: "user",- source: "input",- observationId: "sa2026083102o-drf",- traceId,- parts: [{ type: "text", text: "Draft the reply." }],- },- {- role: "assistant",- source: "output",- observationId: "sa2026083102o-drf",- traceId,- parts: [{ type: "text", text: finalReply }],- },- ],+ conversationHistory: [],+ currentTurn: {+ observations: [+ "cls",+ "llm1",+ "find",+ "llm2",+ "ref",+ "llm3",+ "drf",+ ].map((suffix) => ({ id: `sa2026083102o-${suffix}`, traceId })),+ messages: [+ {+ role: "system",+ source: "input",+ observationId: "sa2026083102o-cls",+ traceId,+ parts: [{ type: "text", text: copilotSystemPrompt }],+ },+ {+ role: "user",+ source: "input",+ observationId: "sa2026083102o-cls",+ traceId,+ parts: [{ type: "text", text: customerMessage }],+ },+ {+ role: "assistant",+ source: "output",+ observationId: "sa2026083102o-cls",+ traceId,+ parts: [{ type: "text", text: classification.content }],+ },+ {+ role: "assistant",+ source: "output",+ observationId: "sa2026083102o-llm1",+ traceId,+ parts: [+ {+ type: "tool-call",+ toolCallId: findChargesToolCallId,+ toolName: "stripe_find_charges",+ input: findChargesArguments,+ toolType: "function",+ },+ ],+ },+ {+ role: "tool",+ source: "output",+ observationId: "sa2026083102o-find",+ traceId,+ parts: [{ type: "data", value: charges }],+ },+ {+ role: "assistant",+ source: "output",+ observationId: "sa2026083102o-llm2",+ traceId,+ parts: [+ {+ type: "tool-call",+ toolCallId: createRefundToolCallId,+ toolName: "stripe_create_refund",+ input: createRefundArguments,+ toolType: "function",+ },+ ],+ },+ {+ role: "tool",+ source: "output",+ observationId: "sa2026083102o-ref",+ traceId,+ parts: [{ type: "data", value: refund }],+ },+ {+ role: "assistant",+ source: "output",+ observationId: "sa2026083102o-llm3",+ traceId,+ parts: [{ type: "text", text: "Resolution complete." }],+ },+ {+ role: "user",+ source: "input",+ observationId: "sa2026083102o-drf",+ traceId,+ parts: [{ type: "text", text: "Draft the reply." }],+ },+ {+ role: "assistant",+ source: "output",+ observationId: "sa2026083102o-drf",+ traceId,+ parts: [{ type: "text", text: finalReply }],+ },+ ],+ },},],},diff --git a/packages/shared/src/server/transcript/fixtures/trace/vercel-ai-sdk-docs-tool-loop.ts b/packages/shared/src/server/transcript/fixtures/trace/vercel-ai-sdk-docs-tool-loop.tsindex 8f06fdf35d..1fa4218bcf 100644--- a/packages/shared/src/server/transcript/fixtures/trace/vercel-ai-sdk-docs-tool-loop.ts+++ b/packages/shared/src/server/transcript/fixtures/trace/vercel-ai-sdk-docs-tool-loop.ts@@ -438,7 +438,6 @@ const payload13 = JSON.stringify([export const vercelAiSdkDocsToolLoopFixture: TranscriptFixture = {name: "Vercel AI SDK docs search with replayed tool history",- scope: "trace",description:"Two model steps with a documentation tool call and replayed history. The export also includes a GENERATION-typed application wrapper; its type and all payloads are preserved rather than adjusted to match the design example.",observations: [@@ -705,109 +704,112 @@ export const vercelAiSdkDocsToolLoopFixture: TranscriptFixture = {expected: {threads: [{- messages: [- {- role: "user",- parts: [- {- type: "text",- text: "How do I attach a user score to a trace?",- },- ],- source: "input",- observationId: "1bbf3edb8ef95ab6",- traceId: "75b336f98e39e275a823b0a7ec1ba794",- },- {- role: "assistant",- parts: [- {- type: "reasoning",- content: {- kind: "text",- text: '**Searching documents for scores**\n\nI need to use tools to search the documentation. It looks like Langfuse has a "scores" API that can be linked to traces or observations. There are "Score" entities to consider, which include details like name, value, data type, comments, source, traceId, observationId, and timestamp. The SDKs have methods such as client.score or langfuse.score. I plan to search Langfuse docs to get more information on this.',+ conversationHistory: [],+ currentTurn: {+ messages: [+ {+ role: "user",+ parts: [+ {+ type: "text",+ text: "How do I attach a user score to a trace?",},- },- {- type: "text",- text: "Iβm going to search the Langfuse docs for how to add or attach a score to a trace, including minimal code examples.",- },- {- type: "tool-call",- toolCallId: "call_OQNWWW4PjixkLIhkuHJRwiFV",- toolName: "searchLangfuseDocs",- input: {- query:- "How to attach a score to a trace user score to a trace Langfuse SDK example",+ ],+ source: "input",+ observationId: "1bbf3edb8ef95ab6",+ traceId: "75b336f98e39e275a823b0a7ec1ba794",+ },+ {+ role: "assistant",+ parts: [+ {+ type: "reasoning",+ content: {+ kind: "text",+ text: '**Searching documents for scores**\n\nI need to use tools to search the documentation. It looks like Langfuse has a "scores" API that can be linked to traces or observations. There are "Score" entities to consider, which include details like name, value, data type, comments, source, traceId, observationId, and timestamp. The SDKs have methods such as client.score or langfuse.score. I plan to search Langfuse docs to get more information on this.',+ },+ },+ {+ type: "text",+ text: "Iβm going to search the Langfuse docs for how to add or attach a score to a trace, including minimal code examples.",},- toolType: "tool_call",+ {+ type: "tool-call",+ toolCallId: "call_OQNWWW4PjixkLIhkuHJRwiFV",+ toolName: "searchLangfuseDocs",+ input: {+ query:+ "How to attach a score to a trace user score to a trace Langfuse SDK example",+ },+ toolType: "tool_call",+ },+ ],+ finishReason: {+ type: "tool-calls",+ raw: "tool_call",},- ],- finishReason: {- type: "tool-calls",- raw: "tool_call",+ source: "output",+ observationId: "1bbf3edb8ef95ab6",+ traceId: "75b336f98e39e275a823b0a7ec1ba794",},- source: "output",- observationId: "1bbf3edb8ef95ab6",- traceId: "75b336f98e39e275a823b0a7ec1ba794",- },- {- role: "tool",- source: "output",- parts: [- {- type: "text",- text: '{"content": [{"type": "document", "record_type": "documentation", "url": "https://langfuse.com/docs/evaluation/evaluation-methods/scores-via-sdk", "title": "Scores via API/SDK", "source": {"content": [{"type": "text", "text": "Scores via API/SDK - Langfuse\\n[Docs](/docs)\\n[Evaluation](/docs/evaluation/overview)\\n[Evaluation Methods](/docs/evaluation/evaluation-methods/llm-as-a-judge)\\n[Scores via API/SDK](/docs/evaluation/evaluation-methods/scores-via-sdk)\\n# Scores via API/SDK\\nYou can use the Langfuse SDKs or API to add [scores](/docs/evaluation/scores/overview) to traces, observations, sessions and dataset runs. This is an evaluation method that allows you to set up custom evaluation workflows and extend the scoring capabilities of Langfuse. See the [data model](/docs/evaluation/scores/data-model#scores) for full details on the score object.\\nIf you want Langfuse to run deterministic Python or TypeScript logic for you, use [code evaluators](/docs/evaluation/evaluation-methods/code-evaluators). Use this page when your application, pipeline, or CI job computes scores and sends them to Langfuse via API or SDK.\\n## Ingesting Scores via API/SDKs\\nScores can be attached at different levels of granularity: to individual traces, to specific observations within a trace, or to full sessions.\\nSee the [API reference](/docs/api) for full details on the POST and GET endpoints for scores and score configs.\\n### Trace or Observation-level Scores\\nYou can add scores via the Langfuse SDKs or API. Scores can take one of four data types: **Numeric** , **Categorical** , **Boolean** , or **Text**. See [Score Types](/docs/evaluation/scores/overview#score-types) for details.\\nIf a score is ingested manually using a `trace_id` to link the score to a trace, it is not necessary to wait until the trace has been created. The score will show up in the scores table and will be linked to the trace once the trace with the same `trace_id` is created.\\nHere are examples by `Score` data types."}, {"type": "text", "text": "\\nFor trace and observation scores, `trace_id`/`traceId` is required and `observation_id`/`observationId` is optional. If you attach a score to an observation, always provide both the observation ID and the corresponding trace ID.\\nPython SDKJS/TS SDKAPI\\nNumericCategoricalBooleanText\\nNumeric score values must be provided as float.\\n \\n \\n```\\nfrom langfuse import get_client\\nlangfuse = get_client()\\n///# Method 1: Score via low-level method\\nlangfuse.create_score(\\n name=\\"correctness\\",\\n value=0.9,\\n trace_id=\\"trace_id_here\\",\\n observation_id=\\"observation_id_here\\", ///# optional\\n data_type=\\"NUMERIC\\", ///# optional, inferred if not provided\\n comment=\\"Factually correct\\", ///# optional\\n)\\n///# Method 2: Score current span/generation (within context)\\nwith langfuse.start_as_current_observation(as_type=\\"span\\", name=\\"my-operation\\") as span:\\n # Score the current span\\n span.score(\\n name=\\"correctness\\",\\n value=0.9,\\n data_type=\\"NUMERIC\\",\\n comment=\\"Factually correct\\"\\n )\\n # Score the trace\\n span.score_trace(\\n name=\\"overall_quality\\",\\n value=0.95,\\n data_type=\\"NUMERIC\\"\\n )\\n///# Method 3: Score via the current context\\nwith langfuse.start_as_current_observation(as_type=\\"span\\", name=\\"my-operation\\"):\\n # Score the current span\\n langfuse.score_current_span(\\n name=\\"correctness\\",\\n value=0.9,\\n data_type=\\"NUMERIC\\",\\n comment=\\"Factually correct\\"\\n )\\n # Score the trace\\n langfuse.score_current_trace(\\n name=\\"overall_quality\\",\\n value=0.95,\\n data_type=\\"NUMERIC\\"\\n )\\n```\\nCategorical score values must be provided as strings.\\n \\n \\n```\\nfrom langfuse import get_client\\nlangfuse = get_client()\\n///# Method 1: Score via low-level method\\nlangfuse.create_score(\\n name=\\"accuracy\\",\\n value=\\"partially correct\\",\\n trace_id=\\"trace_id_here\\",\\n observation_id=\\"observation_id_here\\", ///# optional\\n data_type=\\"CATEGORICAL\\", ///# optional, inferred if not provided\\n comment=\\"Some factual errors\\", ///# optional\\n)"}, {"type": "text", "text": "\\n///# Method 2: Score current span/generation (within context)\\nwith langfuse.start_as_current_observation(as_type=\\"span\\", name=\\"my-operation\\") as span:\\n # Score the current span\\n span.score(\\n name=\\"accuracy\\",\\n value=\\"partially correct\\",\\n data_type=\\"CATEGORICAL\\",\\n comment=\\"Some factual errors\\"\\n )\\n # Score the trace\\n span.score_trace(\\n name=\\"overall_quality\\",\\n value=\\"partially correct\\",\\n data_type=\\"CATEGORICAL\\"\\n )\\n///# Method 3: Score via the current context\\nwith langfuse.start_as_current_observation(as_type=\\"span\\", name=\\"my-operation\\"):\\n # Score the current span\\n langfuse.score_current_span(\\n name=\\"accuracy\\",\\n value=\\"partially correct\\",\\n data_type=\\"CATEGORICAL\\",\\n comment=\\"Some factual errors\\"\\n )\\n # Score the trace\\n langfuse.score_current_trace(\\n name=\\"overall_quality\\",\\n value=\\"partially correct\\",\\n data_type=\\"CATEGORICAL\\"\\n )\\n```\\nBoolean scores must be provided as a float, where `1` is true and `0` is false. When read via the [v3 scores API](https://api.reference.langfuse.com/#tag/scoresv3/GET/api/public/v3/scores), the value is returned as a boolean. See [API reference](/docs/api) for more details on POST/GET scores endpoints.\\n \\n \\n```\\nfrom langfuse import get_client\\nlangfuse = get_client()\\n///# Method 1: Score via low-level method\\nlangfuse.create_score(\\n name=\\"helpfulness\\",\\n value=0, ///# 0 or 1\\n trace_id=\\"trace_id_here\\",\\n observation_id=\\"observation_id_here\\", ///# optional\\n data_type=\\"BOOLEAN\\", ///# required, numeric values without data type would be inferred as NUMERIC\\n comment=\\"Incorrect answer\\", ///# optional\\n)\\n///# Method 2: Score current span/generation (within context)\\nwith langfuse.start_as_current_observation(as_type=\\"span\\", name=\\"my-operation\\") as span:\\n # Score the current span"}, {"type": "text", "text": "\\n span.score(\\n name=\\"helpfulness\\",\\n value=1, ///# 0 or 1\\n data_type=\\"BOOLEAN\\",\\n comment=\\"Very helpful response\\"\\n )\\n # Score the trace\\n span.score_trace(\\n name=\\"overall_quality\\",\\n value=1, ///# 0 or 1\\n data_type=\\"BOOLEAN\\"\\n )\\n///# Method 3: Score via the current context\\nwith langfuse.start_as_current_observation(as_type=\\"span\\", name=\\"my-operation\\"):\\n # Score the current span\\n langfuse.score_current_span(\\n name=\\"helpfulness\\",\\n value=1, ///# 0 or 1\\n data_type=\\"BOOLEAN\\",\\n comment=\\"Very helpful response\\"\\n )\\n # Score the trace\\n langfuse.score_current_trace(\\n name=\\"overall_quality\\",\\n value=1, ///# 0 or 1\\n data_type=\\"BOOLEAN\\"\\n )\\n```\\nText score values must be provided as strings between 1 and 500 characters.\\n \\n \\n```\\nfrom langfuse import get_client\\nlangfuse = get_client()\\n///# Method 1: Score via low-level method\\nlangfuse.create_score(\\n name=\\"reviewer_notes\\",\\n value=\\"The response was helpful but could be more concise.\\",\\n trace_id=\\"trace_id_here\\",\\n observation_id=\\"observation_id_here\\", ///# optional\\n data_type=\\"TEXT\\", ///# optional, inferred if not provided\\n comment=\\"Reviewed by QA team\\", ///# optional\\n)\\n///# Method 2: Score current span/generation (within context)\\nwith langfuse.start_as_current_observation(as_type=\\"span\\", name=\\"my-operation\\") as span:\\n # Score the current span\\n span.score(\\n name=\\"reviewer_notes\\",\\n value=\\"The response was helpful but could be more concise.\\",\\n data_type=\\"TEXT\\",\\n comment=\\"Reviewed by QA team\\"\\n )\\n # Score the trace\\n span.score_trace(\\n name=\\"overall_notes\\",\\n value=\\"Good quality overall, minor formatting issues.\\",\\n data_type=\\"TEXT\\"\\n )\\n///# Method 3: Score via the current context\\nwith langfuse.start_as_current_observation(as_type=\\"span\\", name=\\"my-operation\\"):\\n # Score the current span"}, {"type": "text", "text": "\\n langfuse.score_current_span(\\n name=\\"reviewer_notes\\",\\n value=\\"The response was helpful but could be more concise.\\",\\n data_type=\\"TEXT\\",\\n comment=\\"Reviewed by QA team\\"\\n )\\n # Score the trace\\n langfuse.score_current_trace(\\n name=\\"overall_notes\\",\\n value=\\"Good quality overall, minor formatting issues.\\",\\n data_type=\\"TEXT\\"\\n )\\n```\\nNumericCategoricalBooleanText\\nNumeric score values must be provided as float.\\n \\n \\n```\\nimport { LangfuseClient } from \\"@langfuse/client\\";\\nconst langfuse = new LangfuseClient();\\nlangfuse.score.create({\\n id: \\"unique_id\\", // optional, can be used as an idempotency key to update the score subsequently\\n traceId: message.traceId,\\n observationId: message.generationId, // optional\\n name: \\"correctness\\",\\n value: 0.9,\\n dataType: \\"NUMERIC\\", // optional, inferred if not provided\\n comment: \\"Factually correct\\", // optional\\n});\\n// Flush the scores in short-lived environments\\nawait langfuse.flush();\\n```\\nCategorical score values must be provided as strings.\\n \\n \\n```\\nimport { LangfuseClient } from \\"@langfuse/client\\";\\nconst langfuse = new LangfuseClient();\\nlangfuse.score.create({\\n id: \\"unique_id\\", // optional, can be used as an idempotency key to update the score subsequently\\n traceId: message.traceId,\\n observationId: message.generationId, // optional\\n name: \\"accuracy\\",\\n value: \\"partially correct\\",\\n dataType: \\"CATEGORICAL\\", // optional, inferred if not provided\\n comment: \\"Factually correct\\", // optional\\n});\\n// Flush the scores in short-lived environments\\nawait langfuse.flush();\\n```"}, {"type": "text", "text": "\\nBoolean scores must be provided as a float, where `1` is true and `0` is false. When read via the [v3 scores API](https://api.reference.langfuse.com/#tag/scoresv3/GET/api/public/v3/scores), the value is returned as a boolean. See [API reference](/docs/api) for more details on POST/GET scores endpoints.\\n \\n \\n```\\nimport { LangfuseClient } from \\"@langfuse/client\\";\\nconst langfuse = new LangfuseClient();\\nlangfuse.score.create({\\n id: \\"unique_id\\", // optional, can be used as an idempotency key to update the score subsequently\\n traceId: message.traceId,\\n observationId: message.generationId, // optional\\n name: \\"helpfulness\\",\\n value: 0, // 0 or 1\\n dataType: \\"BOOLEAN\\", // required, numeric values without data type would be inferred as NUMERIC\\n comment: \\"Incorrect answer\\", // optional\\n});\\n// Flush the scores in short-lived environments\\nawait langfuse.flush();\\n```\\nText score values must be provided as strings between 1 and 500 characters.\\n \\n \\n```\\nimport { LangfuseClient } from \\"@langfuse/client\\";\\nconst langfuse = new LangfuseClient();\\nlangfuse.score.create({\\n id: \\"unique_id\\", // optional, can be used as an idempotency key to update the score subsequently\\n traceId: message.traceId,\\n observationId: message.generationId, // optional\\n name: \\"reviewer_notes\\",\\n value: \\"The response was helpful but could be more concise.\\",\\n dataType: \\"TEXT\\", // optional, inferred if not provided\\n comment: \\"Reviewed by QA team\\", // optional\\n});\\n// Flush the scores in short-lived environments\\nawait langfuse.flush();\\n```"}, {"type": "text", "text": "\\nYou can also create scores directly via the [REST API](https://api.reference.langfuse.com/#tag/legacyscorev1/POST/api/public/scores). Authenticate using HTTP Basic Auth with your Langfuse Public Key as username and Secret Key as password.\\nNumericCategoricalBooleanText\\nNumeric score values must be provided as float.\\n \\n \\n```\\ncurl -X POST https://cloud.langfuse.com/api/public/scores \\\\\\n -u \\"pk-lf-...\\":\\"sk-lf-...\\" \\\\\\n -H \\"Content-Type: application/json\\" \\\\\\n -d \'{\\n \\"traceId\\": \\"trace_id_here\\",\\n \\"observationId\\": \\"observation_id_here\\",\\n \\"name\\": \\"correctness\\",\\n \\"value\\": 0.9,\\n \\"dataType\\": \\"NUMERIC\\",\\n \\"comment\\": \\"Factually correct\\"\\n }\'\\n```\\nCategorical score values must be provided as strings.\\n \\n \\n```\\ncurl -X POST https://cloud.langfuse.com/api/public/scores \\\\\\n -u \\"pk-lf-...\\":\\"sk-lf-...\\" \\\\\\n -H \\"Content-Type: application/json\\" \\\\\\n -d \'{\\n \\"traceId\\": \\"trace_id_here\\",\\n \\"observationId\\": \\"observation_id_here\\",\\n \\"name\\": \\"accuracy\\",\\n \\"value\\": \\"partially correct\\",\\n \\"dataType\\": \\"CATEGORICAL\\",\\n \\"comment\\": \\"Some factual errors\\"\\n }\'\\n```\\nBoolean scores must be provided as a float, where `1` is true and `0` is false. When read via the [v3 scores API](https://api.reference.langfuse.com/#tag/scoresv3/GET/api/public/v3/scores), the value is returned as a boolean.\\n \\n \\n```\\ncurl -X POST https://cloud.langfuse.com/api/public/scores \\\\\\n -u \\"pk-lf-...\\":\\"sk-lf-...\\" \\\\\\n -H \\"Content-Type: application/json\\" \\\\\\n -d \'{\\n \\"traceId\\": \\"trace_id_here\\",\\n \\"observationId\\": \\"observation_id_here\\",\\n \\"name\\": \\"helpfulness\\",\\n \\"value\\": 0,\\n \\"dataType\\": \\"BOOLEAN\\",\\n \\"comment\\": \\"Incorrect answer\\"\\n }\'\\n```\\nText score values must be provided as strings between 1 and 500 characters.\\n \\n \\n```\\ncurl -X POST https://cloud.langfuse.com/api/public/scores \\\\\\n -u \\"pk-lf-...\\":\\"sk-lf-...\\" \\\\"}, {"type": "text", "text": "\\n comment=\\"Factually correct\\",\\n config_id=\\"12345-6565-3453654-43543\\",\\n data_type=\\"CATEGORICAL\\"\\n )\\n```\\nWhen ingesting boolean scores, you can provide the value as a float. If you provide a configId, the score\'s name and config\'s name must match as well as their data types.\\n \\n \\n```\\nfrom langfuse import get_client\\nlangfuse = get_client()\\n///# Method 1: Score via low-level method\\nlangfuse.create_score(\\n trace_id=\\"trace_id_here\\",\\n observation_id=\\"observation_id_here\\", ///# optional\\n name=\\"helpfulness\\",\\n value=1,\\n comment=\\"Factually correct\\", ///# optional\\n score_id=\\"unique_id\\", ///# optional, can be used as an idempotency key to update the score subsequently\\n config_id=\\"93547-6565-3453654-43543\\", ///# optional, can be used to infer the score data type and validate the score value\\n data_type=\\"BOOLEAN\\" # optional, possibly inferred\\n)\\n///# Method 2: Score within context\\nwith langfuse.start_as_current_observation(as_type=\\"span\\", name=\\"my-operation\\") as span:\\n span.score(\\n name=\\"helpfulness\\",\\n value=1,\\n comment=\\"Factually correct\\",\\n config_id=\\"93547-6565-3453654-43543\\",\\n data_type=\\"BOOLEAN\\"\\n )\\n```\\nWhen ingesting text scores, provide the value as a string (1β500 characters). If you provide a configId, the score\'s name and config\'s name must match as well as their data types.\\n \\n \\n```\\nfrom langfuse import get_client\\nlangfuse = get_client()\\n///# Method 1: Score via low-level method\\nlangfuse.create_score(\\n trace_id=\\"trace_id_here\\",\\n observation_id=\\"observation_id_here\\", ///# optional\\n name=\\"reviewer_notes\\",\\n value=\\"The response was helpful but could be more concise.\\",\\n comment=\\"Reviewed by QA team\\", ///# optional"}, {"type": "text", "text": "\\n score_id=\\"unique_id\\", ///# optional, can be used as an idempotency key to update the score subsequently\\n config_id=\\"24680-6565-3453654-43543\\", ///# optional\\n data_type=\\"TEXT\\" # optional, possibly inferred\\n)\\n///# Method 2: Score within context\\nwith langfuse.start_as_current_observation(as_type=\\"span\\", name=\\"my-operation\\") as span:\\n span.score(\\n name=\\"reviewer_notes\\",\\n value=\\"The response was helpful but could be more concise.\\",\\n comment=\\"Reviewed by QA team\\",\\n config_id=\\"24680-6565-3453654-43543\\",\\n data_type=\\"TEXT\\"\\n )\\n```\\nNumeric ScoresCategorical ScoresBoolean ScoresText Scores\\nWhen ingesting numeric scores, you can provide the value as a float. If you provide a configId, the score value will be validated against the config\'s numeric range, which might be defined by a minimum and/or maximum value.\\n \\n \\n```\\nimport { LangfuseClient } from \\"@langfuse/client\\";\\nconst langfuse = new LangfuseClient();\\nlangfuse.score.create({\\n traceId: message.traceId,\\n observationId: message.generationId, // optional\\n name: \\"accuracy\\",\\n value: 0.9,\\n comment: \\"Factually correct\\", // optional\\n id: \\"unique_id\\", // optional, can be used as an idempotency key to update the score subsequently\\n configId: \\"78545-6565-3453654-43543\\", // optional, to ensure that the score follows a specific min/max value range\\n dataType: \\"NUMERIC\\", // optional, possibly inferred\\n});\\n// Flush the scores in short-lived environments\\nawait langfuse.flush();\\n```\\nCategorical scores are used to evaluate data that falls into specific categories. When ingesting categorical scores, you can provide the value as a string. If you provide a configId, the score value will be validated against the config\'s categories.\\n \\n \\n```"}, {"type": "text", "text": "\\nimport { LangfuseClient } from \\"@langfuse/client\\";\\nconst langfuse = new LangfuseClient();\\nlangfuse.score.create({\\n traceId: message.traceId,\\n observationId: message.generationId, // optional\\n name: \\"correctness\\",\\n value: \\"correct\\",\\n comment: \\"Factually correct\\", // optional\\n id: \\"unique_id\\", // optional, can be used as an idempotency key to update the score subsequently\\n configId: \\"12345-6565-3453654-43543\\", // optional, to ensure that a score maps to a specific category defined in a score config\\n dataType: \\"CATEGORICAL\\", // optional, possibly inferred\\n});\\n// Flush the scores in short-lived environments\\nawait langfuse.flush();\\n```\\nWhen ingesting boolean scores, you can provide the value as a float. If you provide a configId, the score\'s name and config\'s name must match as well as their data types.\\n \\n \\n```\\nimport { LangfuseClient } from \\"@langfuse/client\\";\\nconst langfuse = new LangfuseClient();\\nlangfuse.score.create({\\n traceId: message.traceId,\\n observationId: message.generationId, // optional\\n name: \\"helpfulness\\",\\n value: 1,\\n comment: \\"Factually correct\\", // optional\\n id: \\"unique_id\\", // optional, can be used as an idempotency key to update the score subsequently\\n configId: \\"93547-6565-3453654-43543\\", // optional, can be used to infer the score data type and validate the score value\\n dataType: \\"BOOLEAN\\", // optional, possibly inferred\\n});\\n// Flush the scores in short-lived environments\\nawait langfuse.flush();\\n```\\nWhen ingesting text scores, provide the value as a string (1β500 characters). If you provide a configId, the score\'s name and config\'s name must match as well as their data types.\\n \\n \\n```\\nimport { LangfuseClient } from \\"@langfuse/client\\";\\nconst langfuse = new LangfuseClient();\\nlangfuse.score.create({\\n traceId: message.traceId,\\n observationId: message.generationId, // optional\\n name: \\"reviewer_notes\\",\\n value: \\"The response was helpful but could be more concise.\\","}, {"type": "text", "text": "\\nWhen ingesting boolean scores, you can provide the value as a float. If you provide a configId, the score\'s name and config\'s name must match as well as their data types.\\n \\n \\n```\\ncurl -X POST https://cloud.langfuse.com/api/public/scores \\\\\\n -u \\"pk-lf-...\\":\\"sk-lf-...\\" \\\\\\n -H \\"Content-Type: application/json\\" \\\\\\n -d \'{\\n \\"id\\": \\"unique_id\\",\\n \\"traceId\\": \\"trace_id_here\\",\\n \\"observationId\\": \\"observation_id_here\\",\\n \\"name\\": \\"helpfulness\\",\\n \\"value\\": 1,\\n \\"dataType\\": \\"BOOLEAN\\",\\n \\"configId\\": \\"93547-6565-3453654-43543\\",\\n \\"comment\\": \\"Factually correct\\"\\n }\'\\n```\\nWhen ingesting text scores, provide the value as a string (1β500 characters). If you provide a configId, the score\'s name and config\'s name must match as well as their data types.\\n \\n \\n```\\ncurl -X POST https://cloud.langfuse.com/api/public/scores \\\\\\n -u \\"pk-lf-...\\":\\"sk-lf-...\\" \\\\\\n -H \\"Content-Type: application/json\\" \\\\\\n -d \'{\\n \\"id\\": \\"unique_id\\",\\n \\"traceId\\": \\"trace_id_here\\",\\n \\"observationId\\": \\"observation_id_here\\",\\n \\"name\\": \\"reviewer_notes\\",\\n \\"value\\": \\"The response was helpful but could be more concise.\\",\\n \\"dataType\\": \\"TEXT\\",\\n \\"configId\\": \\"24680-6565-3453654-43543\\",\\n \\"comment\\": \\"Reviewed by QA team\\"\\n }\'\\n```\\nSee [API reference](/docs/api) for more details on POST/GET score configs endpoints.\\n### Inferred Score Properties\\nCertain score properties might be inferred based on your input:\\n * **If you don\'t provide a score data type** it will always be inferred. See tables below for details."}, {"type": "text", "text": "\\n * **For boolean and categorical scores** , we will provide the score value in both numerical and string format where possible. The score value format that is not provided as input, i.e. the translated value is referred to as the inferred value in the tables below.\\n * **On read via the[v3 scores API](https://api.reference.langfuse.com/#tag/scoresv3/GET/api/public/v3/scores)**, boolean scores are returned as a single boolean `value`.\\n * **For categorical scores** , the string representation is always provided and a numerical mapping of the category will be produced only if a `ScoreConfig` was provided.\\nDetailed Examples:\\nNumeric ScoresCategorical ScoresBoolean ScoresText Scores\\nFor example, let\'s assume you\'d like to ingest a numeric score to measure **accuracy**. We have included a table of possible score ingestion scenarios below.\\nValue| Data Type| Config Id| Description| Inferred Data Type| Valid \\n---|---|---|---|---|--- \\n`0.9`| `Null`| `Null`| Data type is inferred| `NUMERIC`| Yes \\n`0.9`| `NUMERIC`| `Null`| No properties inferred| | Yes \\n`depth`| `NUMERIC`| `Null`| Error: data type of value does not match provided data type| | No \\n`0.9`| `NUMERIC`| `78545`| No properties inferred| | Conditional on config validation \\n`0.9`| `Null`| `78545`| Data type inferred| `NUMERIC`| Conditional on config validation \\n`depth`| `NUMERIC`| `78545`| Error: data type of value does not match provided data type| | No \\n \\nFor example, let\'s assume you\'d like to ingest a categorical score to measure **correctness**. We have included a table of possible score ingestion scenarios below."}, {"type": "text", "text": "Value| Data Type| Config Id| Description| Inferred Data Type| Inferred Value representation| Valid \\n---|---|---|---|---|---|--- \\n`correct`| `Null`| `Null`| Data type is inferred| `CATEGORICAL`| | Yes \\n`correct`| `CATEGORICAL`| `Null`| No properties inferred| | | Yes \\n`1`| `CATEGORICAL`| `Null`| Error: data type of value does not match provided data type| | | No \\n`correct`| `CATEGORICAL`| `12345`| Numeric value inferred| | `4` numeric config category mapping| Conditional on config validation \\n`correct`| `NULL`| `12345`| Data type inferred| `CATEGORICAL`| | Conditional on config validation \\n`1`| `CATEGORICAL`| `12345`| Error: data type of value does not match provided data type| | | No \\n \\nFor example, let\'s assume you\'d like to ingest a boolean score to measure **helpfulness**. We have included a table of possible score ingestion scenarios below.\\nValue| Data Type| Config Id| Description| Inferred Data Type| Inferred Value representation| Valid \\n---|---|---|---|---|---|--- \\n`1`| `BOOLEAN`| `Null`| Value\'s string equivalent inferred| | `True`| Yes \\n`true`| `BOOLEAN`| `Null`| Error: data type of value does not match provided data type| | | No \\n`3`| `BOOLEAN`| `Null`| Error: boolean data type expects `0` or `1` as input value| | | No \\n`0.9`| `Null`| `93547`| Data type and value\'s string equivalent inferred| `BOOLEAN`| `True`| Conditional on config validation \\n`depth`| `BOOLEAN`| `93547`| Error: data type of value does not match provided data type| | | No \\n \\nFor example, let\'s assume you\'d like to ingest a text score to capture **reviewer notes**. Text scores must be non-empty strings of at most 500 characters.\\nValue| Data Type| Config Id| Description| Inferred Data Type| Valid "}, {"type": "text", "text": "---|---|---|---|---|--- \\n`\\"Good response\\"`| `Null`| `Null`| Data type is inferred| `TEXT`| Yes \\n`\\"Good response\\"`| `TEXT`| `Null`| No properties inferred| | Yes \\n`0.9`| `TEXT`| `Null`| Error: data type of value does not match provided data type| | No \\n`\\"Good response\\"`| `TEXT`| `24680`| No properties inferred| | Conditional on config validation \\n`\\"Good response\\"`| `Null`| `24680`| Data type inferred| `TEXT`| Conditional on config validation \\n`\\"\\"`| `TEXT`| `Null`| Error: text scores must be non-empty| | No \\n \\n## Update Existing Scores via API/SDKs\\nWhen creating a score, you can provide an optional `id` (JS/TS) / `score_id` (Python) parameter. A re-ingested score overwrites an existing one only when its `id`, `name`, and `timestamp` (at date granularity, `toDate(timestamp)`) all match β a matching `id` alone is not sufficient, and any difference creates a duplicate instead of an update.\\nTo overwrite a score without fetching existing scores first, set a stable `id` as an idempotency key when initially creating the score, and keep the `name` and `timestamp` unchanged on subsequent calls. See Preventing Duplicate Scores for details.\\n## Related guides\\nGuides\\n * [Collecting user feedback\\nCapture explicit and implicit user feedback as scores on your traces, including in-app ratings sent via our Browser SDK.\\n](/guides/user-feedback-loop)\\n * [Custom evaluation data pipeline\\nContinuously monitor quality by fetching traces, running custom evaluations, and ingesting scores back into Langfuse.\\n](/guides/cookbook/example_external_evaluation_pipelines)\\n * [Guardrails and security checks"}, {"type": "text", "text": "\\nCheck whether output contains a certain keyword, matches a required structure or format, or exceeds a length limit.\\n](/guides/cookbook/security-and-guardrails)\\n * [Custom internal workflow tooling\\nBuild internal tooling for human-in-the-loop workflows, ingesting scores back into Langfuse following your custom schema.\\n](/guides/human-in-the-loop-scoring)\\n * [Session-level quality tracking\\nScore full conversations, such as support chats or agent threads, by attaching scores via sessionId in the SDK or API.\\n](/guides/cookbook/example_evaluating_multi_turn_conversations)\\n* * *\\n* * *\\nLast edited 7/22/2026\\n"}], "type": "content"}}, {"type": "document", "record_type": "documentation", "url": "https://langfuse.com/docs/evaluation/evaluation-methods/scores-via-sdk", "title": "Scores via API/SDK", "source": {"content": [{"type": "text", "text": "# Scores via API/SDK\\nYou can use the Langfuse SDKs or API to add [scores](/docs/evaluation/scores/overview) to traces, observations, sessions and dataset runs. This is an evaluation method that allows you to set up custom evaluation workflows and extend the scoring capabilities of Langfuse. See the [data model](/docs/evaluation/scores/data-model#scores) for full details on the score object.\\nIf you want Langfuse to run deterministic Python or TypeScript logic for you, use [code evaluators](/docs/evaluation/evaluation-methods/code-evaluators). Use this page when your application, pipeline, or CI job computes scores and sends them to Langfuse via API or SDK.\\n\\n## Ingesting Scores via API/SDKs\\nScores can be attached at different levels of granularity: to individual traces, to specific observations within a trace, or to full sessions.\\nSee the [API reference](/docs/api) for full details on the POST and GET endpoints for scores and score configs.\\n### Trace or Observation-level Scores\\nYou can add scores via the Langfuse SDKs or API. Scores can take one of four data types: **Numeric** , **Categorical** , **Boolean** , or **Text**. See [Score Types](/docs/evaluation/scores/overview#score-types) for details.\\nIf a score is ingested manually using a `trace_id` to link the score to a trace, it is not necessary to wait until the trace has been created. The score will show up in the scores table and will be linked to the trace once the trace with the same `trace_id` is created.\\nHere are examples by `Score` data types.\\nFor trace and observation scores, `trace_id`/`traceId` is required and `observation_id`/`observationId` is optional. If you attach a score to an observation, always provide both the observation ID and the corresponding trace ID.\\nPython SDKJS/TS SDKAPI\\nNumericCategoricalBooleanText\\nNumeric score values must be provided as float.\\n \\n \\n```"}, {"type": "text", "text": "\\nfrom langfuse import get_client\\nlangfuse = get_client()\\n# Method 1: Score via low-level method\\nlangfuse.create_score(\\n name=\\"correctness\\",\\n value=0.9,\\n trace_id=\\"trace_id_here\\",\\n observation_id=\\"observation_id_here\\", # optional\\n data_type=\\"NUMERIC\\", # optional, inferred if not provided\\n comment=\\"Factually correct\\", # optional\\n)\\n# Method 2: Score current span/generation (within context)\\nwith langfuse.start_as_current_observation(as_type=\\"span\\", name=\\"my-operation\\") as span:\\n # Score the current span\\n span.score(\\n name=\\"correctness\\",\\n value=0.9,\\n data_type=\\"NUMERIC\\",\\n comment=\\"Factually correct\\"\\n )\\n # Score the trace\\n span.score_trace(\\n name=\\"overall_quality\\",\\n value=0.95,\\n data_type=\\"NUMERIC\\"\\n )\\n# Method 3: Score via the current context\\nwith langfuse.start_as_current_observation(as_type=\\"span\\", name=\\"my-operation\\"):\\n # Score the current span\\n langfuse.score_current_span(\\n name=\\"correctness\\",\\n value=0.9,\\n data_type=\\"NUMERIC\\",\\n comment=\\"Factually correct\\"\\n )\\n # Score the trace\\n langfuse.score_current_trace(\\n name=\\"overall_quality\\",\\n value=0.95,\\n data_type=\\"NUMERIC\\"\\n )\\n```\\nCategorical score values must be provided as strings.\\n \\n \\n```\\nfrom langfuse import get_client\\nlangfuse = get_client()\\n# Method 1: Score via low-level method\\nlangfuse.create_score(\\n name=\\"accuracy\\",\\n value=\\"partially correct\\",\\n trace_id=\\"trace_id_here\\",\\n observation_id=\\"observation_id_here\\", # optional\\n data_type=\\"CATEGORICAL\\", # optional, inferred if not provided\\n comment=\\"Some factual errors\\", # optional\\n)\\n# Method 2: Score current span/generation (within context)\\nwith langfuse.start_as_current_observation(as_type=\\"span\\", name=\\"my-operation\\") as span:\\n # Score the current span\\n span.score(\\n name=\\"accuracy\\",\\n value=\\"partially correct\\",\\n data_type=\\"CATEGORICAL\\",\\n comment=\\"Some factual errors\\"\\n )\\n # Score the trace\\n span.score_trace(\\n name=\\"overall_quality\\",\\n value=\\"partially correct\\","}, {"type": "text", "text": "\\n data_type=\\"CATEGORICAL\\"\\n )\\n# Method 3: Score via the current context\\nwith langfuse.start_as_current_observation(as_type=\\"span\\", name=\\"my-operation\\"):\\n # Score the current span\\n langfuse.score_current_span(\\n name=\\"accuracy\\",\\n value=\\"partially correct\\",\\n data_type=\\"CATEGORICAL\\",\\n comment=\\"Some factual errors\\"\\n )\\n # Score the trace\\n langfuse.score_current_trace(\\n name=\\"overall_quality\\",\\n value=\\"partially correct\\",\\n data_type=\\"CATEGORICAL\\"\\n )\\n```\\nBoolean scores must be provided as a float, where `1` is true and `0` is false. When read via the [v3 scores API](https://api.reference.langfuse.com/#tag/scoresv3/GET/api/public/v3/scores), the value is returned as a boolean. See [API reference](/docs/api) for more details on POST/GET scores endpoints.\\n \\n \\n```\\nfrom langfuse import get_client\\nlangfuse = get_client()\\n# Method 1: Score via low-level method\\nlangfuse.create_score(\\n name=\\"helpfulness\\",\\n value=0, # 0 or 1\\n trace_id=\\"trace_id_here\\",\\n observation_id=\\"observation_id_here\\", # optional\\n data_type=\\"BOOLEAN\\", # required, numeric values without data type would be inferred as NUMERIC\\n comment=\\"Incorrect answer\\", # optional\\n)\\n# Method 2: Score current span/generation (within context)\\nwith langfuse.start_as_current_observation(as_type=\\"span\\", name=\\"my-operation\\") as span:\\n # Score the current span\\n span.score(\\n name=\\"helpfulness\\",\\n value=1, # 0 or 1\\n data_type=\\"BOOLEAN\\",\\n comment=\\"Very helpful response\\"\\n )\\n # Score the trace\\n span.score_trace(\\n name=\\"overall_quality\\",\\n value=1, # 0 or 1\\n data_type=\\"BOOLEAN\\"\\n )\\n# Method 3: Score via the current context\\nwith langfuse.start_as_current_observation(as_type=\\"span\\", name=\\"my-operation\\"):\\n # Score the current span\\n langfuse.score_current_span(\\n name=\\"helpfulness\\","}, {"type": "text", "text": "\\n value=1, # 0 or 1\\n data_type=\\"BOOLEAN\\",\\n comment=\\"Very helpful response\\"\\n )\\n # Score the trace\\n langfuse.score_current_trace(\\n name=\\"overall_quality\\",\\n value=1, # 0 or 1\\n data_type=\\"BOOLEAN\\"\\n )\\n```\\nText score values must be provided as strings between 1 and 500 characters.\\n \\n \\n```\\nfrom langfuse import get_client\\nlangfuse = get_client()\\n# Method 1: Score via low-level method\\nlangfuse.create_score(\\n name=\\"reviewer_notes\\",\\n value=\\"The response was helpful but could be more concise.\\",\\n trace_id=\\"trace_id_here\\",\\n observation_id=\\"observation_id_here\\", # optional\\n data_type=\\"TEXT\\", # optional, inferred if not provided\\n comment=\\"Reviewed by QA team\\", # optional\\n)\\n# Method 2: Score current span/generation (within context)\\nwith langfuse.start_as_current_observation(as_type=\\"span\\", name=\\"my-operation\\") as span:\\n # Score the current span\\n span.score(\\n name=\\"reviewer_notes\\",\\n value=\\"The response was helpful but could be more concise.\\",\\n data_type=\\"TEXT\\",\\n comment=\\"Reviewed by QA team\\"\\n )\\n # Score the trace\\n span.score_trace(\\n name=\\"overall_notes\\",\\n value=\\"Good quality overall, minor formatting issues.\\",\\n data_type=\\"TEXT\\"\\n )\\n# Method 3: Score via the current context\\nwith langfuse.start_as_current_observation(as_type=\\"span\\", name=\\"my-operation\\"):\\n # Score the current span\\n langfuse.score_current_span(\\n name=\\"reviewer_notes\\",\\n value=\\"The response was helpful but could be more concise.\\",\\n data_type=\\"TEXT\\",\\n comment=\\"Reviewed by QA team\\"\\n )\\n # Score the trace\\n langfuse.score_current_trace(\\n name=\\"overall_notes\\",\\n value=\\"Good quality overall, minor formatting issues.\\",\\n data_type=\\"TEXT\\"\\n )\\n```\\nNumericCategoricalBooleanText\\nNumeric score values must be provided as float.\\n \\n \\n```"}, {"type": "text", "text": "\\nimport { LangfuseClient } from \\"@langfuse/client\\";\\nconst langfuse = new LangfuseClient();\\nlangfuse.score.create({\\n id: \\"unique_id\\", // optional, can be used as an idempotency key to update the score subsequently\\n traceId: message.traceId,\\n observationId: message.generationId, // optional\\n name: \\"correctness\\",\\n value: 0.9,\\n dataType: \\"NUMERIC\\", // optional, inferred if not provided\\n comment: \\"Factually correct\\", // optional\\n});\\n// Flush the scores in short-lived environments\\nawait langfuse.flush();\\n```\\nCategorical score values must be provided as strings.\\n \\n \\n```\\nimport { LangfuseClient } from \\"@langfuse/client\\";\\nconst langfuse = new LangfuseClient();\\nlangfuse.score.create({\\n id: \\"unique_id\\", // optional, can be used as an idempotency key to update the score subsequently\\n traceId: message.traceId,\\n observationId: message.generationId, // optional\\n name: \\"accuracy\\",\\n value: \\"partially correct\\",\\n dataType: \\"CATEGORICAL\\", // optional, inferred if not provided\\n comment: \\"Factually correct\\", // optional\\n});\\n// Flush the scores in short-lived environments\\nawait langfuse.flush();\\n```\\nBoolean scores must be provided as a float, where `1` is true and `0` is false. When read via the [v3 scores API](https://api.reference.langfuse.com/#tag/scoresv3/GET/api/public/v3/scores), the value is returned as a boolean. See [API reference](/docs/api) for more details on POST/GET scores endpoints.\\n \\n \\n```\\nimport { LangfuseClient } from \\"@langfuse/client\\";\\nconst langfuse = new LangfuseClient();\\nlangfuse.score.create({\\n id: \\"unique_id\\", // optional, can be used as an idempotency key to update the score subsequently\\n traceId: message.traceId,\\n observationId: message.generationId, // optional"}, {"type": "text", "text": "\\nnpm install @langfuse/browser\\n```\\n \\n \\n```\\nimport { LangfuseBrowserClient } from \\"@langfuse/browser\\";\\nconst langfuse = new LangfuseBrowserClient({\\n publicKey: process.env.NEXT_PUBLIC_LANGFUSE_PUBLIC_KEY!,\\n baseUrl: process.env.NEXT_PUBLIC_LANGFUSE_BASE_URL, // optional, defaults to https://cloud.langfuse.com\\n});\\nawait langfuse.score({\\n traceId: message.traceId,\\n observationId: message.generationId, // optional\\n id: `user-feedback-${message.traceId}`, // optional, use as an idempotency key\\n name: \\"user-feedback\\",\\n value: 1, // 1 for positive, 0 for negative\\n dataType: \\"BOOLEAN\\",\\n comment: \\"Helpful answer\\", // optional\\n});\\n```\\n`score()` returns `{ id }` after ingestion succeeds. No `flush()` call is required.\\n### Session-level Scores\\nTo score an entire session (without attaching the score to a trace or observation), provide only `session_id` (Python SDK) or `sessionId` (JS/TS SDK and API).\\nPython SDKJS/TS SDKAPI\\n \\n \\n```\\nfrom langfuse import get_client\\nlangfuse = get_client()\\nlangfuse.create_score(\\n name=\\"session_quality\\",\\n value=0.85,\\n session_id=\\"session_id_here\\",\\n data_type=\\"NUMERIC\\",\\n comment=\\"Overall conversation quality\\"\\n)\\n```\\n \\n \\n```\\nimport { LangfuseClient } from \\"@langfuse/client\\";\\nconst langfuse = new LangfuseClient();\\nlangfuse.score.create({\\n name: \\"session_quality\\",\\n value: 0.85,\\n sessionId: \\"session_id_here\\",\\n dataType: \\"NUMERIC\\",\\n comment: \\"Overall conversation quality\\",\\n});\\nawait langfuse.flush();\\n```\\n \\n \\n```\\ncurl -X POST https://cloud.langfuse.com/api/public/scores \\\\\\n -u \\"pk-lf-...\\":\\"sk-lf-...\\" \\\\\\n -H \\"Content-Type: application/json\\" \\\\\\n -d \'{\\n \\"sessionId\\": \\"session_id_here\\",\\n \\"name\\": \\"session_quality\\",\\n \\"value\\": 0.85,\\n \\"dataType\\": \\"NUMERIC\\",\\n \\"comment\\": \\"Overall conversation quality\\"\\n }\'\\n```\\n"}], "type": "content"}}, {"type": "document", "record_type": "documentation", "url": "https://langfuse.com/docs/evaluation/evaluation-methods/scores-via-sdk", "title": "Scores via API/SDK", "source": {"content": [{"type": "text", "text": "## Advanced\\n### Preventing Duplicate Scores\\nBy default, Langfuse allows for multiple scores of the same `name` on the same trace. This is useful if you\'d like to track the evolution of a score over time or if e.g. you\'ve received multiple user feedback scores on the same trace.\\nIn some cases, you want to prevent this behavior or overwrite an existing score. A score is identified by three fields β its `id`, its `name`, and its date (`toDate(timestamp)`) β and a new score only replaces an existing one when all three match. To achieve this, create an **idempotency key** and pass it as the `id` (JS/TS) / `score_id` (Python) when creating the score (for example `trace_id-score_name`), and keep the `name` and `timestamp` unchanged across calls. If any of the three differs β for example the same `id` with a different `name` β you get an additional score rather than a replacement.\\nDo not send partial score updates (only the changed fields) and rely on Langfuse to merge them into the existing record. This merge only happens for a limited window after creation, is deprecated, and will be removed. Always send the complete score.\\nSee [How to update traces, observations, and scores](/faq/all/tracing-data-updates#updating-scores) for more details on Langfuse\'s immutable data model.\\n### Enforcing a Score Config\\nScore configs are helpful when you want to standardize your scores for future analysis.\\nTo enforce a score config, you can provide a `configId` when creating a score to reference a `ScoreConfig` that was previously created. `Score Configs` can be defined in the Langfuse UI or via our API. [See our guide on how to create and manage score configs](/faq/all/manage-score-configs).\\nWhenever you provide a `ScoreConfig`, the score data will be validated against the config. The following rules apply:\\n * **Score Name** : Must equal the config\'s name"}, {"type": "text", "text": "\\n * **Score Data Type** : When provided, must match the config\'s data type\\n * **Score Value when Type is`NUMERIC`**: Value must be within the min and max values defined in the config (if provided, min and max are optional and otherwise are assumed as -β and +β respectively)\\n * **Score Value when Type is`CATEGORICAL`**: Value must map to one of the categories defined in the config\\n * **Score Value when Type is`BOOLEAN`**: Value must equal `0` or `1`\\n * **Score Value when Type is`TEXT`**: Value must be a non-empty string of at most 500 characters\\nPython SDKJS/TS SDKAPI\\nNumeric ScoresCategorical ScoresBoolean ScoresText Scores\\nWhen ingesting numeric scores, you can provide the value as a float. If you provide a configId, the score value will be validated against the config\'s numeric range, which might be defined by a minimum and/or maximum value.\\n \\n \\n```\\nfrom langfuse import get_client\\nlangfuse = get_client()\\n# Method 1: Score via low-level method\\nlangfuse.create_score(\\n trace_id=\\"trace_id_here\\",\\n observation_id=\\"observation_id_here\\", # optional\\n session_id=\\"session_id_here\\", # optional, ID of the session the score relates to\\n name=\\"accuracy\\",\\n value=0.9,\\n comment=\\"Factually correct\\", # optional\\n score_id=\\"unique_id\\", # optional, can be used as an idempotency key to update the score subsequently\\n config_id=\\"78545-6565-3453654-43543\\", # optional, to ensure that the score follows a specific min/max value range\\n data_type=\\"NUMERIC\\" # optional, possibly inferred\\n)\\n# Method 2: Score within context\\nwith langfuse.start_as_current_observation(as_type=\\"span\\", name=\\"my-operation\\") as span:\\n span.score(\\n name=\\"accuracy\\",\\n value=0.9,\\n comment=\\"Factually correct\\",\\n config_id=\\"78545-6565-3453654-43543\\",\\n data_type=\\"NUMERIC\\"\\n )\\n```"}, {"type": "text", "text": "\\nWhen ingesting numeric scores, you can provide the value as a float. If you provide a configId, the score value will be validated against the config\'s numeric range, which might be defined by a minimum and/or maximum value.\\n \\n \\n```\\nimport { LangfuseClient } from \\"@langfuse/client\\";\\nconst langfuse = new LangfuseClient();\\nlangfuse.score.create({\\n traceId: message.traceId,\\n observationId: message.generationId, // optional\\n name: \\"accuracy\\",\\n value: 0.9,\\n comment: \\"Factually correct\\", // optional\\n id: \\"unique_id\\", // optional, can be used as an idempotency key to update the score subsequently\\n configId: \\"78545-6565-3453654-43543\\", // optional, to ensure that the score follows a specific min/max value range\\n dataType: \\"NUMERIC\\", // optional, possibly inferred\\n});\\n// Flush the scores in short-lived environments\\nawait langfuse.flush();\\n```\\nCategorical scores are used to evaluate data that falls into specific categories. When ingesting categorical scores, you can provide the value as a string. If you provide a configId, the score value will be validated against the config\'s categories.\\n \\n \\n```\\nimport { LangfuseClient } from \\"@langfuse/client\\";\\nconst langfuse = new LangfuseClient();\\nlangfuse.score.create({\\n traceId: message.traceId,\\n observationId: message.generationId, // optional\\n name: \\"correctness\\",\\n value: \\"correct\\",\\n comment: \\"Factually correct\\", // optional\\n id: \\"unique_id\\", // optional, can be used as an idempotency key to update the score subsequently\\n configId: \\"12345-6565-3453654-43543\\", // optional, to ensure that a score maps to a specific category defined in a score config\\n dataType: \\"CATEGORICAL\\", // optional, possibly inferred\\n});"}, {"type": "text", "text": "\\nYou can also enforce score configs via the [REST API](https://api.reference.langfuse.com/#tag/legacyscorev1/POST/api/public/scores) by providing a `configId`.\\nNumeric ScoresCategorical ScoresBoolean ScoresText Scores\\nWhen ingesting numeric scores, you can provide the value as a float. If you provide a configId, the score value will be validated against the config\'s numeric range.\\n \\n \\n```\\ncurl -X POST https://cloud.langfuse.com/api/public/scores \\\\\\n -u \\"pk-lf-...\\":\\"sk-lf-...\\" \\\\\\n -H \\"Content-Type: application/json\\" \\\\\\n -d \'{\\n \\"id\\": \\"unique_id\\",\\n \\"traceId\\": \\"trace_id_here\\",\\n \\"observationId\\": \\"observation_id_here\\",\\n \\"name\\": \\"accuracy\\",\\n \\"value\\": 0.9,\\n \\"dataType\\": \\"NUMERIC\\",\\n \\"configId\\": \\"78545-6565-3453654-43543\\",\\n \\"comment\\": \\"Factually correct\\"\\n }\'\\n```\\nCategorical scores are used to evaluate data that falls into specific categories. If you provide a configId, the score value will be validated against the config\'s categories.\\n \\n \\n```\\ncurl -X POST https://cloud.langfuse.com/api/public/scores \\\\\\n -u \\"pk-lf-...\\":\\"sk-lf-...\\" \\\\\\n -H \\"Content-Type: application/json\\" \\\\\\n -d \'{\\n \\"id\\": \\"unique_id\\",\\n \\"traceId\\": \\"trace_id_here\\",\\n \\"observationId\\": \\"observation_id_here\\",\\n \\"name\\": \\"correctness\\",\\n \\"value\\": \\"correct\\",\\n \\"dataType\\": \\"CATEGORICAL\\",\\n \\"configId\\": \\"12345-6565-3453654-43543\\",\\n \\"comment\\": \\"Factually correct\\"\\n }\'\\n```\\nWhen ingesting boolean scores, you can provide the value as a float. If you provide a configId, the score\'s name and config\'s name must match as well as their data types.\\n \\n \\n```\\ncurl -X POST https://cloud.langfuse.com/api/public/scores \\\\\\n -u \\"pk-lf-...\\":\\"sk-lf-...\\" \\\\\\n -H \\"Content-Type: application/json\\" \\\\\\n -d \'{\\n \\"id\\": \\"unique_id\\",\\n \\"traceId\\": \\"trace_id_here\\",\\n \\"observationId\\": \\"observation_id_here\\",\\n \\"name\\": \\"helpfulness\\","}, {"type": "text", "text": "\\n \\"value\\": 1,\\n \\"dataType\\": \\"BOOLEAN\\",\\n \\"configId\\": \\"93547-6565-3453654-43543\\",\\n \\"comment\\": \\"Factually correct\\"\\n }\'\\n```\\nWhen ingesting text scores, provide the value as a string (1β500 characters). If you provide a configId, the score\'s name and config\'s name must match as well as their data types.\\n \\n \\n```\\ncurl -X POST https://cloud.langfuse.com/api/public/scores \\\\\\n -u \\"pk-lf-...\\":\\"sk-lf-...\\" \\\\\\n -H \\"Content-Type: application/json\\" \\\\\\n -d \'{\\n \\"id\\": \\"unique_id\\",\\n \\"traceId\\": \\"trace_id_here\\",\\n \\"observationId\\": \\"observation_id_here\\",\\n \\"name\\": \\"reviewer_notes\\",\\n \\"value\\": \\"The response was helpful but could be more concise.\\",\\n \\"dataType\\": \\"TEXT\\",\\n \\"configId\\": \\"24680-6565-3453654-43543\\",\\n \\"comment\\": \\"Reviewed by QA team\\"\\n }\'\\n```\\nSee [API reference](/docs/api) for more details on POST/GET score configs endpoints.\\n### Inferred Score Properties\\nCertain score properties might be inferred based on your input:\\n * **If you don\'t provide a score data type** it will always be inferred. See tables below for details.\\n * **For boolean and categorical scores** , we will provide the score value in both numerical and string format where possible. The score value format that is not provided as input, i.e. the translated value is referred to as the inferred value in the tables below.\\n * **On read via the[v3 scores API](https://api.reference.langfuse.com/#tag/scoresv3/GET/api/public/v3/scores)**, boolean scores are returned as a single boolean `value`.\\n * **For categorical scores** , the string representation is always provided and a numerical mapping of the category will be produced only if a `ScoreConfig` was provided.\\nDetailed Examples:"}, {"type": "text", "text": "\\nNumeric ScoresCategorical ScoresBoolean ScoresText Scores\\nFor example, let\'s assume you\'d like to ingest a numeric score to measure **accuracy**. We have included a table of possible score ingestion scenarios below.\\nValue| Data Type| Config Id| Description| Inferred Data Type| Valid \\n---|---|---|---|---|--- \\n`0.9`| `Null`| `Null`| Data type is inferred| `NUMERIC`| Yes \\n`0.9`| `NUMERIC`| `Null`| No properties inferred| | Yes \\n`depth`| `NUMERIC`| `Null`| Error: data type of value does not match provided data type| | No \\n`0.9`| `NUMERIC`| `78545`| No properties inferred| | Conditional on config validation \\n`0.9`| `Null`| `78545`| Data type inferred| `NUMERIC`| Conditional on config validation \\n`depth`| `NUMERIC`| `78545`| Error: data type of value does not match provided data type| | No \\n \\nFor example, let\'s assume you\'d like to ingest a categorical score to measure **correctness**. We have included a table of possible score ingestion scenarios below.\\nValue| Data Type| Config Id| Description| Inferred Data Type| Inferred Value representation| Valid \\n---|---|---|---|---|---|--- \\n`correct`| `Null`| `Null`| Data type is inferred| `CATEGORICAL`| | Yes \\n`correct`| `CATEGORICAL`| `Null`| No properties inferred| | | Yes \\n`1`| `CATEGORICAL`| `Null`| Error: data type of value does not match provided data type| | | No \\n`correct`| `CATEGORICAL`| `12345`| Numeric value inferred| | `4` numeric config category mapping| Conditional on config validation \\n`correct`| `NULL`| `12345`| Data type inferred| `CATEGORICAL`| | Conditional on config validation \\n`1`| `CATEGORICAL`| `12345`| Error: data type of value does not match provided data type| | | No \\n "}, {"type": "text", "text": "For example, let\'s assume you\'d like to ingest a boolean score to measure **helpfulness**. We have included a table of possible score ingestion scenarios below.\\nValue| Data Type| Config Id| Description| Inferred Data Type| Inferred Value representation| Valid \\n---|---|---|---|---|---|--- \\n`1`| `BOOLEAN`| `Null`| Value\'s string equivalent inferred| | `True`| Yes \\n`true`| `BOOLEAN`| `Null`| Error: data type of value does not match provided data type| | | No \\n`3`| `BOOLEAN`| `Null`| Error: boolean data type expects `0` or `1` as input value| | | No \\n`0.9`| `Null`| `93547`| Data type and value\'s string equivalent inferred| `BOOLEAN`| `True`| Conditional on config validation \\n`depth`| `BOOLEAN`| `93547`| Error: data type of value does not match provided data type| | | No \\n \\nFor example, let\'s assume you\'d like to ingest a text score to capture **reviewer notes**. Text scores must be non-empty strings of at most 500 characters.\\nValue| Data Type| Config Id| Description| Inferred Data Type| Valid \\n---|---|---|---|---|--- \\n`\\"Good response\\"`| `Null`| `Null`| Data type is inferred| `TEXT`| Yes \\n`\\"Good response\\"`| `TEXT`| `Null`| No properties inferred| | Yes \\n`0.9`| `TEXT`| `Null`| Error: data type of value does not match provided data type| | No \\n`\\"Good response\\"`| `TEXT`| `24680`| No properties inferred| | Conditional on config validation \\n`\\"Good response\\"`| `Null`| `24680`| Data type inferred| `TEXT`| Conditional on config validation \\n`\\"\\"`| `TEXT`| `Null`| Error: text scores must be non-empty| | No \\n \\n\\n## Update Existing Scores via API/SDKs"}, {"type": "text", "text": "When creating a score, you can provide an optional `id` (JS/TS) / `score_id` (Python) parameter. A re-ingested score overwrites an existing one only when its `id`, `name`, and `timestamp` (at date granularity, `toDate(timestamp)`) all match β a matching `id` alone is not sufficient, and any difference creates a duplicate instead of an update.\\nTo overwrite a score without fetching existing scores first, set a stable `id` as an idempotency key when initially creating the score, and keep the `name` and `timestamp` unchanged on subsequent calls. See Preventing Duplicate Scores for details.\\n## Related guides\\nGuides\\n * [Collecting user feedback\\nCapture explicit and implicit user feedback as scores on your traces, including in-app ratings sent via our Browser SDK.\\n](/guides/user-feedback-loop)\\n * [Custom evaluation data pipeline\\nContinuously monitor quality by fetching traces, running custom evaluations, and ingesting scores back into Langfuse.\\n](/guides/cookbook/example_external_evaluation_pipelines)\\n * [Guardrails and security checks\\nCheck whether output contains a certain keyword, matches a required structure or format, or exceeds a length limit.\\n](/guides/cookbook/security-and-guardrails)\\n * [Custom internal workflow tooling\\nBuild internal tooling for human-in-the-loop workflows, ingesting scores back into Langfuse following your custom schema.\\n](/guides/human-in-the-loop-scoring)\\n * [Session-level quality tracking\\nScore full conversations, such as support chats or agent threads, by attaching scores via sessionId in the SDK or API.\\n](/guides/cookbook/example_evaluating_multi_turn_conversations)\\n* * *\\n* * *\\nLast edited 7/22/2026\\n"}], "type": "content"}}, {"type": "document", "record_type": "documentation", "url": "https://langfuse.com/docs/evaluation/scores/overview", "title": "Scores", "source": {"content": [{"type": "text", "text": "Overview - Langfuse\\n[Docs](/docs)\\n[Evaluation](/docs/evaluation/overview)\\n[Scores](/docs/evaluation/scores/overview)\\n[Overview](/docs/evaluation/scores/overview)\\n# Scores\\nScores are Langfuse\'s universal data object for storing evaluation results. Any time you want to assign a quality judgment to an LLM output, whether by a [human annotation](/docs/evaluation/evaluation-methods/scores-via-ui), an [LLM judge](/docs/evaluation/evaluation-methods/llm-as-a-judge), a [programmatic check](/docs/evaluation/evaluation-methods/scores-via-sdk), or end-user feedback, the result is stored as a score.\\nEvery score has a **name** (like `\\"correctness\\"` or `\\"helpfulness\\"`), a **value** , and a **data type**. Scores also support an optional **comment** for additional context.\\nScores can be attached to [traces](/docs/observability/data-model#observations-and-traces), [observations](/docs/observability/data-model#observations-and-traces), [sessions](/docs/observability/data-model#sessions), or [dataset runs](/docs/evaluation/experiments/data-model). Most commonly, scores are attached to traces to evaluate a single end-to-end interaction.\\nOnce you have scores, they show up in [score analytics](/docs/evaluation/scores/score-analytics), can be visualized in [custom dashboards](/docs/metrics/features/custom-dashboards), and can be queried via the [API](/docs/api).\\n## When to Use Scores\\nScores become useful when you want to go beyond observing what your application does and start measuring how well it does it. Common use cases:\\n * **Collecting user feedback** : Capture thumbs up/down or star ratings from your users and attach them to traces. See the [user feedback guide](/docs/observability/features/user-feedback)."}, {"type": "text", "text": "\\n * **Monitoring production quality** : Set up automated evaluators (like [LLM-as-a-Judge](/docs/evaluation/evaluation-methods/llm-as-a-judge)) to continuously score live traces for things like hallucination, relevance, or tone.\\n * **Running guardrails** : Score whether outputs pass safety checks like PII detection, format validation, or content policy compliance.\\n * **Comparing changes with experiments** : When you change a prompt, model, or pipeline, run an [experiment](/docs/evaluation/experiments) to score the new version against a dataset.\\n## Score Types\\nLangfuse supports four score data types:\\nType| Value| Use when \\n---|---|--- \\n`NUMERIC`| Float (e.g. `0.9`)| Continuous judgments like accuracy, relevance, or similarity scores \\n`CATEGORICAL`| String from predefined categories (e.g. `\\"correct\\"`, `\\"partially correct\\"`)| Discrete classifications where the set of possible values is known upfront \\n`BOOLEAN`| `0` or `1`| Pass/fail checks like hallucination detection or format validation \\n`TEXT`| Free-form string (1-500 characters)| Open-ended annotations like reviewer notes or qualitative feedback. Often used for [open coding](https://en.wikipedia.org/wiki/Open_coding) before formalizing into quantifiable scores via [axial coding](https://en.wikipedia.org/wiki/Axial_coding). \\n \\nText scores are designed for qualitative, open-ended scoring. Because free-form text cannot be meaningfully aggregated or compared, text scores are not supported in [experiments](/docs/evaluation/core-concepts#experiments), [LLM-as-a-Judge](/docs/evaluation/evaluation-methods/llm-as-a-judge), or [score analytics](/docs/evaluation/scores/score-analytics).\\n## How to Create Scores"}, {"type": "text", "text": "\\nThere are five ways to add scores:\\n * **LLM-as-a-Judge** : Set up [automated evaluators](/docs/evaluation/evaluation-methods/llm-as-a-judge) that score traces based on custom criteria (e.g. hallucination, tone, relevance). These can return numeric or categorical scores plus reasoning, and can run on live production traces or on experiment results.\\n * **Code evaluators** : Run [custom Python or TypeScript evaluators](/docs/evaluation/evaluation-methods/code-evaluators) in Langfuse for deterministic checks such as exact match, JSON validation, or custom business rules.\\n * **Scores via UI** : Team members [manually score](/docs/evaluation/evaluation-methods/scores-via-ui) traces, observations, or sessions directly in the Langfuse UI. Requires a [score config](/faq/all/manage-score-configs) to be set up first.\\n * **Annotation Queues** : Set up [structured review workflows](/docs/evaluation/evaluation-methods/annotation-queues) where reviewers work through batches of traces.\\n * **Scores via API/SDK** : [Programmatically add scores](/docs/evaluation/evaluation-methods/scores-via-sdk) from your application code. This is the way to go for user feedback (thumbs up/down, star ratings), guardrail results, or custom evaluation pipelines.\\n## Should I Use Scores or Tags?\\n| Scores| Tags \\n---|---|--- \\n **Purpose**| Measure _how good_ something is| Describe _what_ something is \\n **Data**| Numeric, categorical, boolean, or text value| Simple string label \\n **When added**| Can be added at any time, including long after the trace was created| Set during tracing and cannot be changed afterwards \\n **Used for**| Quality measurement, analytics, experiments| Filtering, segmentation, organizing \\n "}, {"type": "text", "text": "As a rule of thumb: if you already know the category at tracing time (e.g. which feature or API endpoint triggered the trace), use a [tag](/docs/observability/features/tags). If you need to classify or evaluate traces later, use a score.\\n## Score Comments\\nEvery score supports an optional **comment** field. Use it to capture reasoning (e.g. why an LLM judge assigned a particular score), reviewer notes, or context that helps others understand the score value. Comments are shown alongside scores in the Langfuse UI.\\nUse a `TEXT` score instead of comments to capture standalone qualitative feedback -- comments are best for additional reasoning on an existing score.\\n* * *\\n* * *\\nLast edited 7/23/2026\\n"}], "type": "content"}}, {"type": "document", "record_type": "github_discussion", "url": "https://github.com/orgs/langfuse/discussions/6016", "title": "Using scores in data sets from the SDK?", "source": {"content": [{"type": "text", "text": "Title: Using scores in data sets from the SDK?\\nCategory: Support\\nAuthor: @howinator\\nTimestamp: March 14, 2025\\nBody:\\nHere is a workflow that I want, but I can\'t figure out how to accomplish it without a lot of manual work.\\r\\n\\r\\n1. Collect traces\\r\\n2. Annotate traces with either manual annotation or programmatic feedback from users. For a simple classification, I would use a boolean score where True means the model returned the correct answer and False when it returned the incorrect answer.\\r\\n4. Create a \\"false positive\\" data set composed of all traces where the score was False. <-- I can\'t figure out how to do this without a lot of clicking.\\r\\n5. Run an eval from that data set and use the score to determine if the eval was correct, e.g., something like `(dataset.item.score && dataset.item.output === evalOutput.output) || (!dataset.item.score && dataset.item.output !== evalOutput.output)`. <-- I can\'t figure out how to do this programmatically at all.\\r\\n\\r\\nIs my workflow completely off here? Am I missing something? Or is this flow just not possible in Langfuse today?\\nComments:\\n\\n Comment #1 from @dosubot on March 14, 2025:\\n\\n <!-- Greeting -->\\nHey there, @howinator! I\'m here to help you out with any bugs, questions, or contributions you have in mind. Let\'s get to work!\\n\\n<!-- Answer -->\\nYour workflow is mostly feasible in Langfuse, and you can achieve it programmatically with some adjustments:\\n\\n1. **Collect Traces**: This step is straightforward and can be done using Langfuse\'s SDKs to capture and store traces.\\n\\n2. **Annotate Traces**: You can programmatically annotate traces with scores using the `langfuse_context.score_current_trace` function. This allows you to attach a boolean score to each trace based on user feedback or other criteria <sup>[[1]](https://langfuse.com/guides/cookbook/integration_langgraph)</sup>.\\n"}, {"type": "text", "text": "\\n3. **Create a \\"False Positive\\" Dataset**: While there isn\'t a direct method to filter dataset items based on scores, you can programmatically create a dataset by iterating over your traces and selecting those with a score of `False`. Use the SDK to create dataset items for these traces <sup>[[2]](https://langfuse.com/docs/datasets/get-started)</sup>.\\n\\n4. **Run an Evaluation**: You can automate the creation of evaluation jobs for your dataset using the `createEvalJobs` function. This function allows you to specify configurations and run evaluations on specific subsets of dataset items <sup>[[3]](https://github.com/langfuse/langfuse/blob/eecf0d680b13a4bde9fe8fed617acc328029307d/worker/src/__tests__/evalService.test.ts)</sup>.\\n\\nFor the evaluation logic, you can implement custom logic to compare the dataset item scores and outputs with the evaluation outputs. This might require some additional scripting to handle the specific logic you described.\\n\\nOverall, your workflow is possible with Langfuse, but it may require some custom scripting to fully automate the process of filtering and evaluating dataset items based on scores.\\n\\n\\n<!-- Dosu Comment Footer -->\\n*To continue talking to [Dosu](https://dosu.dev), mention @dosu.*\\n\\n---"}, {"type": "text", "text": "1. Create a \\"false positive\\" data set composed of all traces where the score was False -> use scores api to fetch all of these\\r\\n2. Run your own eval on them and add it to the same trace via the sdk\\r\\n3. Compare the evals by fetching all of them via the sdk again.\\r\\n\\r\\nWe currently plan to add more configurable dashboards which will allow for the comparison of these evals more easily in the UI.\\n"}], "type": "content"}}, {"type": "document", "record_type": "documentation", "url": "https://langfuse.com/docs/observability/troubleshooting-and-faq", "title": "Troubleshooting and FAQ", "source": {"content": [{"type": "text", "text": "\\n[LLM as a Session-Level Judge with Support for LangGraph Session](https://github.com/orgs/langfuse/discussions/8303 \\"Langfuse Ideas: LLM as a Session-Level Judge with Support for LangGraph Session\\")\\n[Score Configs: Allow editing the categories of a categorical score](https://github.com/orgs/langfuse/discussions/8259 \\"Langfuse Ideas: Score Configs: Allow editing the categories of a categorical score\\")\\n[feat(trace-detail-view): for each span, show the sum of nested observations\' token usage details](https://github.com/orgs/langfuse/discussions/8201 \\"Langfuse Ideas: feat\\\\(trace-detail-view\\\\): for each span, show the sum of nested observations\' token usage details\\")\\n[UI-LLM as a Jury](https://github.com/orgs/langfuse/discussions/8195 \\"Langfuse Ideas: UI-LLM as a Jury\\")\\n[Add Gemini 2.5 Pro model pricing](https://github.com/orgs/langfuse/discussions/8146 \\"Langfuse Ideas: Add Gemini 2.5 Pro model pricing\\")\\n[Deeplinking for Spans inside of a Trace](https://github.com/orgs/langfuse/discussions/8112 \\"Langfuse Ideas: Deeplinking for Spans inside of a Trace\\")\\n[feat(comments): allow attaching images](https://github.com/orgs/langfuse/discussions/8111 \\"Langfuse Ideas: feat\\\\(comments\\\\): allow attaching images\\")\\n[feat(sessions): support session-level tags](https://github.com/orgs/langfuse/discussions/8041 \\"Langfuse Ideas: feat\\\\(sessions\\\\): support session-level tags\\")\\n[Enable Immediate Score Management for User Feedback](https://github.com/orgs/langfuse/discussions/7686 \\"Langfuse Ideas: Enable Immediate Score Management for User Feedback\\")\\n[Allow for structured filtering on inputs / outputs](https://github.com/orgs/langfuse/discussions/7663 \\"Langfuse Ideas: Allow for structured filtering on inputs / outputs\\")"}, {"type": "text", "text": "\\n[Allow for lazy updating of traces in Python SDK (+ possible workaround)](https://github.com/orgs/langfuse/discussions/7655 \\"Langfuse Ideas: Allow for lazy updating of traces in Python SDK \\\\(+ possible workaround\\\\)\\")\\n[Filter by scores in session view](https://github.com/orgs/langfuse/discussions/7528 \\"Langfuse Ideas: Filter by scores in session view\\")\\n[Support break lines on evaluation run tooltip hint](https://github.com/orgs/langfuse/discussions/7452 \\"Langfuse Ideas: Support break lines on evaluation run tooltip hint\\")\\n[Enable sending Langfuse SDK OTEL spans to third party observability backend](https://github.com/orgs/langfuse/discussions/7365 \\"Langfuse Ideas: Enable sending Langfuse SDK OTEL spans to third party observability backend\\")\\n[Add Thinking config in playground for anthropic models](https://github.com/orgs/langfuse/discussions/7267 \\"Langfuse Ideas: Add Thinking config in playground for anthropic models\\")\\n[Better support for big int in metadata.](https://github.com/orgs/langfuse/discussions/7167 \\"Langfuse Ideas: Better support for big int in metadata.\\")\\n[Ability to filter/search comments on traces](https://github.com/orgs/langfuse/discussions/7109 \\"Langfuse Ideas: Ability to filter/search comments on traces\\")\\n[Image / File / Multimedia rendering in dataset items using mediaId](https://github.com/orgs/langfuse/discussions/6883 \\"Langfuse Ideas: Image / File / Multimedia rendering in dataset items using mediaId\\")\\n[Trace Comments support upload image](https://github.com/orgs/langfuse/discussions/6857 \\"Langfuse Ideas: Trace Comments support upload image\\")"}], "type": "content"}}, {"type": "document", "record_type": "github_issue", "url": "https://github.com/langfuse/langfuse/issues/3380", "title": "feat: add fetch_score(s) in the sdks", "source": {"content": [{"type": "text", "text": "Github Issue by @marcklingen created on September 17, 2024. The current state of the issue is: closed\\n\\n### Discussed in https://github.com/orgs/langfuse/discussions/3379\\r\\n\\r\\n<div type=\'discussions-op-text\'>\\r\\n\\r\\n<sup>Originally posted by **Preet1234-ux** September 17, 2024</sup>\\r\\nI want to fetch the score provided by the user from langfuse in my python code . Are there any functionality by which I can fetch the score for a trace ? </div>\\r\\n\\r\\n\\r\\nCurrently, we need to use either langfuse.client.*(potentially breaking) or the api.\\n---\\nComment #1 by @hassiebp:\\n\\nAlready shipped in Python SDK v3\\n\\n"}], "type": "content"}}, {"type": "document", "record_type": "github_discussion", "url": "https://github.com/orgs/langfuse/discussions/6210", "title": "Next.js + Vercel AI SDK: How to get `observationId` and attach it to scores?", "source": {"content": [{"type": "text", "text": "Title: Next.js + Vercel AI SDK: How to get `observationId` and attach it to scores?\\nCategory: Support\\nAuthor: @natterstefan\\nTimestamp: March 26, 2025\\nBody:\\nDear langfuse-Community,\\r\\n\\r\\nI need your help please, as I do not know how to obtain the `observationId` and attach it to a `score`.\\r\\n\\r\\nContext: this is how my `experimental_telemetry` looks like:\\r\\n\\r\\n```js\\r\\n// src: https://langfuse.com/docs/integrations/vercel-ai-sdk#group-multiple-executions-in-one-trace\\r\\n\\r\\nconst parentTraceId = getRandomId()\\r\\n\\r\\nexport async function POST(req: Request) {\\r\\n\\t// ... other code\\r\\n\\r\\n\\tconst trace = langfuse.trace({\\r\\n\\t id: parentTraceId,\\r\\n\\t name: \'cd-chat-messages\',\\r\\n\\t})\\r\\n\\r\\n\\tconsole.log(\'trace\', trace.observationId) // <<-- this returns `null`\\r\\n\\t\\r\\n\\tconst result = streamText({\\r\\n\\t\\t// ....\\r\\n\\t\\texperimental_telemetry: {\\r\\n\\t\\t isEnabled: true,\\r\\n\\t\\t functionId: \'cd-chat-message-\' + messages.length,\\r\\n\\t\\t metadata: {\\r\\n\\t\\t langfuseTraceId: parentTraceId,\\r\\n\\t\\t\\t // Do not update the parent trace with execution results\\r\\n\\t\\t langfuseUpdateParent: false,\\r\\n\\t\\t },\\r\\n\\t\\t },\\r\\n\\t})\\r\\n\\r\\n\\t// ... other code\\r\\n}\\r\\n```\\r\\n\\r\\nThe docs state\\r\\n\\r\\n> The id of the observation to which the score should be attached. Pass from backend trace. E.g. {generation, event, span}.id\\r\\n\\r\\nBut how and where do I get the observationId from, taking into account that I use `streamText` (<https://sdk.vercel.ai/docs/reference/ai-sdk-core/stream-text>)\\r\\n\\r\\n```js\\r\\n// source: https://langfuse.com/docs/sdk/typescript/guide-web#create-scores\\r\\n\\r\\n// pass traceId and observationId to front end\\r\\nawait langfuseWeb.score({\\r\\n traceId: message.traceId,\\r\\n observationId: message.observationId, <<<-- where do I get this?\\r\\n name: \\"user-feedback\\",\\r\\n value: 1,\\r\\n comment: \\"I like how personalized the response is\\",\\r\\n});\\r\\n```\\r\\n\\r"}, {"type": "text", "text": "Another question I could not answer myself is where (= which \\"layer\\". I guess generation, event, span is the correct wording) a score should be attached to. Is there any best practice recommendation or should I always go for the observation (`ai.streamText.doStream` in this example _if_ that\'s the observationId I can obtain)?\\r\\n\\r\\n\\r\\n\\r\\nThank you very much for your time and help!\\nComments:\\n\\n Comment #1 from @dosubot on March 26, 2025:\\n\\n <!-- Greeting -->\\nHey @natterstefan! I\'m here to help you with any bugs, questions, or even becoming a contributor. Let\'s squash those bugs together! π€\\n\\n<!-- Answer -->\\nTo obtain the `observationId` when using the `streamText` function in the Vercel AI SDK, you should look for it in the response from the `streamText` function. The SDK typically uses the ID from the provider response if available, or it generates one if not. Ensure that your telemetry setup is configured to pass the `observationId` from the backend trace, specifically from the context of the `streamText` execution.\\n\\nRegarding where to attach the score, it is generally recommended to attach it to the most relevant observation. In your case, if `ai.streamText.doStream` is the observation you can obtain, it would be appropriate to attach the score there. This aligns with best practices, as it ensures the score is associated with the specific operation or event you are evaluating.\\n"}, {"type": "text", "text": "\\nFor more detailed guidance, you might want to refer to the Langfuse documentation on creating scores and handling observations <sup>[[1]](https://langfuse.com/docs/sdk/typescript)</sup><sup>[[2]](https://langfuse.com/docs/sdk/typescript/guide)</sup>.\\n\\n\\n<!-- Dosu Comment Footer -->\\n*To continue talking to [Dosu](https://dosu.dev), mention @dosu.*\\n\\n---\\n<sup>Help Dosu learn! Give it feedback: [Great Response](https://app.dosu.dev/response-feedback/68b78af0-2c34-4867-b75c-008bcf449b81?feedback_type=great_response) | [Irrelevant Answer](https://app.dosu.dev/response-feedback/68b78af0-2c34-4867-b75c-008bcf449b81?feedback_type=irrelevant_answer) | [Incorrect Sources](https://app.dosu.dev/response-feedback/68b78af0-2c34-4867-b75c-008bcf449b81?feedback_type=incorrect_sources) | [Too Verbose](https://app.dosu.dev/response-feedback/68b78af0-2c34-4867-b75c-008bcf449b81?feedback_type=too_verbose) | [Hallucination](https://app.dosu.dev/response-feedback/68b78af0-2c34-4867-b75c-008bcf449b81?feedback_type=hallucination) | [Bug Report](https://app.dosu.dev/response-feedback/68b78af0-2c34-4867-b75c-008bcf449b81?feedback_type=bug_report) | [Other](https://app.dosu.dev/response-feedback/68b78af0-2c34-4867-b75c-008bcf449b81?feedback_type=other)</sup>\\n\\n\\n Comment #2 from @marcklingen on March 26, 2025:\\n\\n This seems like a feature request for the AI SDK as you\'d need the SDK to expose the span ids of each of the requests.\\n\\n Replies to comment #2:\\n\\n Reply #1 to comment #2 from marcklingen on March 27, 2025:\\n\\n Hi @marcklingen,\\r\\n\\r\\nthanks for your help. Alright, I [moved (aka copied) the discussion to their repo](https://github.com/vercel/ai/discussions/5415) and hope they can help me.\\r\\n\\r\\nIs @dosubot correct with \\"_(...) In your case, if ai.streamText.doStream is the observation you can obtain, it would be appropriate to attach the score there._\\"\\r\\n\\r"}, {"type": "text", "text": "\\n<sup>Help Dosu learn! Give it feedback: [Great Response](https://app.dosu.dev/response-feedback/d9fcd303-8526-48dc-85b4-1432b3d12429?feedback_type=great_response) | [Irrelevant Answer](https://app.dosu.dev/response-feedback/d9fcd303-8526-48dc-85b4-1432b3d12429?feedback_type=irrelevant_answer) | [Incorrect Sources](https://app.dosu.dev/response-feedback/d9fcd303-8526-48dc-85b4-1432b3d12429?feedback_type=incorrect_sources) | [Too Verbose](https://app.dosu.dev/response-feedback/d9fcd303-8526-48dc-85b4-1432b3d12429?feedback_type=too_verbose) | [Hallucination](https://app.dosu.dev/response-feedback/d9fcd303-8526-48dc-85b4-1432b3d12429?feedback_type=hallucination) | [Bug Report](https://app.dosu.dev/response-feedback/d9fcd303-8526-48dc-85b4-1432b3d12429?feedback_type=bug_report) | [Other](https://app.dosu.dev/response-feedback/d9fcd303-8526-48dc-85b4-1432b3d12429?feedback_type=other)</sup>\\n\\n\\n Reply #3 to comment #2 from marcklingen on March 27, 2025:\\n\\n Hi @marcklingen and @dosubot,\\r\\n\\r\\nFollow up question (once I get the `generation.id`/`response.id` whatever vercel/ai calls it): how do I attach it to the current trace to make sure the observation in langfuse has the same id as the score in my code uses?\\n\\n Reply #4 to comment #2 from marcklingen on March 27, 2025:\\n\\n <!-- Answer -->\\nTo attach the `generation.id` or `response.id` from Vercel AI to the current trace in Langfuse, use the `generation.id` as the `observationId` when logging user feedback. This ensures that the observation has the same ID as the score used in your code. Here\'s how you can do it:\\n\\n1. Add the `generation.id` to the streaming response as a custom header:\\n\\n ```javascript\\n return new StreamingTextResponse(stream, {\\n headers: {\\n \\"X-Message-Id\\": lfGeneration.id,"}], "type": "content"}}]}',- },- ],- observationId: "3dbf50cbf2036387",- traceId: "75b336f98e39e275a823b0a7ec1ba794",- },- {- role: "assistant",- parts: [- {- type: "reasoning",- content: {- kind: "text",- text: "**Crafting user score attachment instructions**\n\nI've gathered enough info to help the user with attaching a user score to a trace. They want minimal examples in TypeScript, Python, and a Browser client. I think providing a basic snippet that adds a boolean score named \"user-feedback\" to the traceId would be effective. It's also essential to note that the data type BOOLEAN requires values of 0 or 1. Iβll include concise runnable examples and useful links about scores via API/SDK and user feedback. We should also consider concluding with a follow-up question.",+ {+ role: "tool",+ source: "output",+ parts: [+ {+ type: "text",+ text: '{"content": [{"type": "document", "record_type": "documentation", "url": "https://langfuse.com/docs/evaluation/evaluation-methods/scores-via-sdk", "title": "Scores via API/SDK", "source": {"content": [{"type": "text", "text": "Scores via API/SDK - Langfuse\\n[Docs](/docs)\\n[Evaluation](/docs/evaluation/overview)\\n[Evaluation Methods](/docs/evaluation/evaluation-methods/llm-as-a-judge)\\n[Scores via API/SDK](/docs/evaluation/evaluation-methods/scores-via-sdk)\\n# Scores via API/SDK\\nYou can use the Langfuse SDKs or API to add [scores](/docs/evaluation/scores/overview) to traces, observations, sessions and dataset runs. This is an evaluation method that allows you to set up custom evaluation workflows and extend the scoring capabilities of Langfuse. See the [data model](/docs/evaluation/scores/data-model#scores) for full details on the score object.\\nIf you want Langfuse to run deterministic Python or TypeScript logic for you, use [code evaluators](/docs/evaluation/evaluation-methods/code-evaluators). Use this page when your application, pipeline, or CI job computes scores and sends them to Langfuse via API or SDK.\\n## Ingesting Scores via API/SDKs\\nScores can be attached at different levels of granularity: to individual traces, to specific observations within a trace, or to full sessions.\\nSee the [API reference](/docs/api) for full details on the POST and GET endpoints for scores and score configs.\\n### Trace or Observation-level Scores\\nYou can add scores via the Langfuse SDKs or API. Scores can take one of four data types: **Numeric** , **Categorical** , **Boolean** , or **Text**. See [Score Types](/docs/evaluation/scores/overview#score-types) for details.\\nIf a score is ingested manually using a `trace_id` to link the score to a trace, it is not necessary to wait until the trace has been created. The score will show up in the scores table and will be linked to the trace once the trace with the same `trace_id` is created.\\nHere are examples by `Score` data types."}, {"type": "text", "text": "\\nFor trace and observation scores, `trace_id`/`traceId` is required and `observation_id`/`observationId` is optional. If you attach a score to an observation, always provide both the observation ID and the corresponding trace ID.\\nPython SDKJS/TS SDKAPI\\nNumericCategoricalBooleanText\\nNumeric score values must be provided as float.\\n \\n \\n```\\nfrom langfuse import get_client\\nlangfuse = get_client()\\n///# Method 1: Score via low-level method\\nlangfuse.create_score(\\n name=\\"correctness\\",\\n value=0.9,\\n trace_id=\\"trace_id_here\\",\\n observation_id=\\"observation_id_here\\", ///# optional\\n data_type=\\"NUMERIC\\", ///# optional, inferred if not provided\\n comment=\\"Factually correct\\", ///# optional\\n)\\n///# Method 2: Score current span/generation (within context)\\nwith langfuse.start_as_current_observation(as_type=\\"span\\", name=\\"my-operation\\") as span:\\n # Score the current span\\n span.score(\\n name=\\"correctness\\",\\n value=0.9,\\n data_type=\\"NUMERIC\\",\\n comment=\\"Factually correct\\"\\n )\\n # Score the trace\\n span.score_trace(\\n name=\\"overall_quality\\",\\n value=0.95,\\n data_type=\\"NUMERIC\\"\\n )\\n///# Method 3: Score via the current context\\nwith langfuse.start_as_current_observation(as_type=\\"span\\", name=\\"my-operation\\"):\\n # Score the current span\\n langfuse.score_current_span(\\n name=\\"correctness\\",\\n value=0.9,\\n data_type=\\"NUMERIC\\",\\n comment=\\"Factually correct\\"\\n )\\n # Score the trace\\n langfuse.score_current_trace(\\n name=\\"overall_quality\\",\\n value=0.95,\\n data_type=\\"NUMERIC\\"\\n )\\n```\\nCategorical score values must be provided as strings.\\n \\n \\n```\\nfrom langfuse import get_client\\nlangfuse = get_client()\\n///# Method 1: Score via low-level method\\nlangfuse.create_score(\\n name=\\"accuracy\\",\\n value=\\"partially correct\\",\\n trace_id=\\"trace_id_here\\",\\n observation_id=\\"observation_id_here\\", ///# optional\\n data_type=\\"CATEGORICAL\\", ///# optional, inferred if not provided\\n comment=\\"Some factual errors\\", ///# optional\\n)"}, {"type": "text", "text": "\\n///# Method 2: Score current span/generation (within context)\\nwith langfuse.start_as_current_observation(as_type=\\"span\\", name=\\"my-operation\\") as span:\\n # Score the current span\\n span.score(\\n name=\\"accuracy\\",\\n value=\\"partially correct\\",\\n data_type=\\"CATEGORICAL\\",\\n comment=\\"Some factual errors\\"\\n )\\n # Score the trace\\n span.score_trace(\\n name=\\"overall_quality\\",\\n value=\\"partially correct\\",\\n data_type=\\"CATEGORICAL\\"\\n )\\n///# Method 3: Score via the current context\\nwith langfuse.start_as_current_observation(as_type=\\"span\\", name=\\"my-operation\\"):\\n # Score the current span\\n langfuse.score_current_span(\\n name=\\"accuracy\\",\\n value=\\"partially correct\\",\\n data_type=\\"CATEGORICAL\\",\\n comment=\\"Some factual errors\\"\\n )\\n # Score the trace\\n langfuse.score_current_trace(\\n name=\\"overall_quality\\",\\n value=\\"partially correct\\",\\n data_type=\\"CATEGORICAL\\"\\n )\\n```\\nBoolean scores must be provided as a float, where `1` is true and `0` is false. When read via the [v3 scores API](https://api.reference.langfuse.com/#tag/scoresv3/GET/api/public/v3/scores), the value is returned as a boolean. See [API reference](/docs/api) for more details on POST/GET scores endpoints.\\n \\n \\n```\\nfrom langfuse import get_client\\nlangfuse = get_client()\\n///# Method 1: Score via low-level method\\nlangfuse.create_score(\\n name=\\"helpfulness\\",\\n value=0, ///# 0 or 1\\n trace_id=\\"trace_id_here\\",\\n observation_id=\\"observation_id_here\\", ///# optional\\n data_type=\\"BOOLEAN\\", ///# required, numeric values without data type would be inferred as NUMERIC\\n comment=\\"Incorrect answer\\", ///# optional\\n)\\n///# Method 2: Score current span/generation (within context)\\nwith langfuse.start_as_current_observation(as_type=\\"span\\", name=\\"my-operation\\") as span:\\n # Score the current span"}, {"type": "text", "text": "\\n span.score(\\n name=\\"helpfulness\\",\\n value=1, ///# 0 or 1\\n data_type=\\"BOOLEAN\\",\\n comment=\\"Very helpful response\\"\\n )\\n # Score the trace\\n span.score_trace(\\n name=\\"overall_quality\\",\\n value=1, ///# 0 or 1\\n data_type=\\"BOOLEAN\\"\\n )\\n///# Method 3: Score via the current context\\nwith langfuse.start_as_current_observation(as_type=\\"span\\", name=\\"my-operation\\"):\\n # Score the current span\\n langfuse.score_current_span(\\n name=\\"helpfulness\\",\\n value=1, ///# 0 or 1\\n data_type=\\"BOOLEAN\\",\\n comment=\\"Very helpful response\\"\\n )\\n # Score the trace\\n langfuse.score_current_trace(\\n name=\\"overall_quality\\",\\n value=1, ///# 0 or 1\\n data_type=\\"BOOLEAN\\"\\n )\\n```\\nText score values must be provided as strings between 1 and 500 characters.\\n \\n \\n```\\nfrom langfuse import get_client\\nlangfuse = get_client()\\n///# Method 1: Score via low-level method\\nlangfuse.create_score(\\n name=\\"reviewer_notes\\",\\n value=\\"The response was helpful but could be more concise.\\",\\n trace_id=\\"trace_id_here\\",\\n observation_id=\\"observation_id_here\\", ///# optional\\n data_type=\\"TEXT\\", ///# optional, inferred if not provided\\n comment=\\"Reviewed by QA team\\", ///# optional\\n)\\n///# Method 2: Score current span/generation (within context)\\nwith langfuse.start_as_current_observation(as_type=\\"span\\", name=\\"my-operation\\") as span:\\n # Score the current span\\n span.score(\\n name=\\"reviewer_notes\\",\\n value=\\"The response was helpful but could be more concise.\\",\\n data_type=\\"TEXT\\",\\n comment=\\"Reviewed by QA team\\"\\n )\\n # Score the trace\\n span.score_trace(\\n name=\\"overall_notes\\",\\n value=\\"Good quality overall, minor formatting issues.\\",\\n data_type=\\"TEXT\\"\\n )\\n///# Method 3: Score via the current context\\nwith langfuse.start_as_current_observation(as_type=\\"span\\", name=\\"my-operation\\"):\\n # Score the current span"}, {"type": "text", "text": "\\n langfuse.score_current_span(\\n name=\\"reviewer_notes\\",\\n value=\\"The response was helpful but could be more concise.\\",\\n data_type=\\"TEXT\\",\\n comment=\\"Reviewed by QA team\\"\\n )\\n # Score the trace\\n langfuse.score_current_trace(\\n name=\\"overall_notes\\",\\n value=\\"Good quality overall, minor formatting issues.\\",\\n data_type=\\"TEXT\\"\\n )\\n```\\nNumericCategoricalBooleanText\\nNumeric score values must be provided as float.\\n \\n \\n```\\nimport { LangfuseClient } from \\"@langfuse/client\\";\\nconst langfuse = new LangfuseClient();\\nlangfuse.score.create({\\n id: \\"unique_id\\", // optional, can be used as an idempotency key to update the score subsequently\\n traceId: message.traceId,\\n observationId: message.generationId, // optional\\n name: \\"correctness\\",\\n value: 0.9,\\n dataType: \\"NUMERIC\\", // optional, inferred if not provided\\n comment: \\"Factually correct\\", // optional\\n});\\n// Flush the scores in short-lived environments\\nawait langfuse.flush();\\n```\\nCategorical score values must be provided as strings.\\n \\n \\n```\\nimport { LangfuseClient } from \\"@langfuse/client\\";\\nconst langfuse = new LangfuseClient();\\nlangfuse.score.create({\\n id: \\"unique_id\\", // optional, can be used as an idempotency key to update the score subsequently\\n traceId: message.traceId,\\n observationId: message.generationId, // optional\\n name: \\"accuracy\\",\\n value: \\"partially correct\\",\\n dataType: \\"CATEGORICAL\\", // optional, inferred if not provided\\n comment: \\"Factually correct\\", // optional\\n});\\n// Flush the scores in short-lived environments\\nawait langfuse.flush();\\n```"}, {"type": "text", "text": "\\nBoolean scores must be provided as a float, where `1` is true and `0` is false. When read via the [v3 scores API](https://api.reference.langfuse.com/#tag/scoresv3/GET/api/public/v3/scores), the value is returned as a boolean. See [API reference](/docs/api) for more details on POST/GET scores endpoints.\\n \\n \\n```\\nimport { LangfuseClient } from \\"@langfuse/client\\";\\nconst langfuse = new LangfuseClient();\\nlangfuse.score.create({\\n id: \\"unique_id\\", // optional, can be used as an idempotency key to update the score subsequently\\n traceId: message.traceId,\\n observationId: message.generationId, // optional\\n name: \\"helpfulness\\",\\n value: 0, // 0 or 1\\n dataType: \\"BOOLEAN\\", // required, numeric values without data type would be inferred as NUMERIC\\n comment: \\"Incorrect answer\\", // optional\\n});\\n// Flush the scores in short-lived environments\\nawait langfuse.flush();\\n```\\nText score values must be provided as strings between 1 and 500 characters.\\n \\n \\n```\\nimport { LangfuseClient } from \\"@langfuse/client\\";\\nconst langfuse = new LangfuseClient();\\nlangfuse.score.create({\\n id: \\"unique_id\\", // optional, can be used as an idempotency key to update the score subsequently\\n traceId: message.traceId,\\n observationId: message.generationId, // optional\\n name: \\"reviewer_notes\\",\\n value: \\"The response was helpful but could be more concise.\\",\\n dataType: \\"TEXT\\", // optional, inferred if not provided\\n comment: \\"Reviewed by QA team\\", // optional\\n});\\n// Flush the scores in short-lived environments\\nawait langfuse.flush();\\n```"}, {"type": "text", "text": "\\nYou can also create scores directly via the [REST API](https://api.reference.langfuse.com/#tag/legacyscorev1/POST/api/public/scores). Authenticate using HTTP Basic Auth with your Langfuse Public Key as username and Secret Key as password.\\nNumericCategoricalBooleanText\\nNumeric score values must be provided as float.\\n \\n \\n```\\ncurl -X POST https://cloud.langfuse.com/api/public/scores \\\\\\n -u \\"pk-lf-...\\":\\"sk-lf-...\\" \\\\\\n -H \\"Content-Type: application/json\\" \\\\\\n -d \'{\\n \\"traceId\\": \\"trace_id_here\\",\\n \\"observationId\\": \\"observation_id_here\\",\\n \\"name\\": \\"correctness\\",\\n \\"value\\": 0.9,\\n \\"dataType\\": \\"NUMERIC\\",\\n \\"comment\\": \\"Factually correct\\"\\n }\'\\n```\\nCategorical score values must be provided as strings.\\n \\n \\n```\\ncurl -X POST https://cloud.langfuse.com/api/public/scores \\\\\\n -u \\"pk-lf-...\\":\\"sk-lf-...\\" \\\\\\n -H \\"Content-Type: application/json\\" \\\\\\n -d \'{\\n \\"traceId\\": \\"trace_id_here\\",\\n \\"observationId\\": \\"observation_id_here\\",\\n \\"name\\": \\"accuracy\\",\\n \\"value\\": \\"partially correct\\",\\n \\"dataType\\": \\"CATEGORICAL\\",\\n \\"comment\\": \\"Some factual errors\\"\\n }\'\\n```\\nBoolean scores must be provided as a float, where `1` is true and `0` is false. When read via the [v3 scores API](https://api.reference.langfuse.com/#tag/scoresv3/GET/api/public/v3/scores), the value is returned as a boolean.\\n \\n \\n```\\ncurl -X POST https://cloud.langfuse.com/api/public/scores \\\\\\n -u \\"pk-lf-...\\":\\"sk-lf-...\\" \\\\\\n -H \\"Content-Type: application/json\\" \\\\\\n -d \'{\\n \\"traceId\\": \\"trace_id_here\\",\\n \\"observationId\\": \\"observation_id_here\\",\\n \\"name\\": \\"helpfulness\\",\\n \\"value\\": 0,\\n \\"dataType\\": \\"BOOLEAN\\",\\n \\"comment\\": \\"Incorrect answer\\"\\n }\'\\n```\\nText score values must be provided as strings between 1 and 500 characters.\\n \\n \\n```\\ncurl -X POST https://cloud.langfuse.com/api/public/scores \\\\\\n -u \\"pk-lf-...\\":\\"sk-lf-...\\" \\\\"}, {"type": "text", "text": "\\n comment=\\"Factually correct\\",\\n config_id=\\"12345-6565-3453654-43543\\",\\n data_type=\\"CATEGORICAL\\"\\n )\\n```\\nWhen ingesting boolean scores, you can provide the value as a float. If you provide a configId, the score\'s name and config\'s name must match as well as their data types.\\n \\n \\n```\\nfrom langfuse import get_client\\nlangfuse = get_client()\\n///# Method 1: Score via low-level method\\nlangfuse.create_score(\\n trace_id=\\"trace_id_here\\",\\n observation_id=\\"observation_id_here\\", ///# optional\\n name=\\"helpfulness\\",\\n value=1,\\n comment=\\"Factually correct\\", ///# optional\\n score_id=\\"unique_id\\", ///# optional, can be used as an idempotency key to update the score subsequently\\n config_id=\\"93547-6565-3453654-43543\\", ///# optional, can be used to infer the score data type and validate the score value\\n data_type=\\"BOOLEAN\\" # optional, possibly inferred\\n)\\n///# Method 2: Score within context\\nwith langfuse.start_as_current_observation(as_type=\\"span\\", name=\\"my-operation\\") as span:\\n span.score(\\n name=\\"helpfulness\\",\\n value=1,\\n comment=\\"Factually correct\\",\\n config_id=\\"93547-6565-3453654-43543\\",\\n data_type=\\"BOOLEAN\\"\\n )\\n```\\nWhen ingesting text scores, provide the value as a string (1β500 characters). If you provide a configId, the score\'s name and config\'s name must match as well as their data types.\\n \\n \\n```\\nfrom langfuse import get_client\\nlangfuse = get_client()\\n///# Method 1: Score via low-level method\\nlangfuse.create_score(\\n trace_id=\\"trace_id_here\\",\\n observation_id=\\"observation_id_here\\", ///# optional\\n name=\\"reviewer_notes\\",\\n value=\\"The response was helpful but could be more concise.\\",\\n comment=\\"Reviewed by QA team\\", ///# optional"}, {"type": "text", "text": "\\n score_id=\\"unique_id\\", ///# optional, can be used as an idempotency key to update the score subsequently\\n config_id=\\"24680-6565-3453654-43543\\", ///# optional\\n data_type=\\"TEXT\\" # optional, possibly inferred\\n)\\n///# Method 2: Score within context\\nwith langfuse.start_as_current_observation(as_type=\\"span\\", name=\\"my-operation\\") as span:\\n span.score(\\n name=\\"reviewer_notes\\",\\n value=\\"The response was helpful but could be more concise.\\",\\n comment=\\"Reviewed by QA team\\",\\n config_id=\\"24680-6565-3453654-43543\\",\\n data_type=\\"TEXT\\"\\n )\\n```\\nNumeric ScoresCategorical ScoresBoolean ScoresText Scores\\nWhen ingesting numeric scores, you can provide the value as a float. If you provide a configId, the score value will be validated against the config\'s numeric range, which might be defined by a minimum and/or maximum value.\\n \\n \\n```\\nimport { LangfuseClient } from \\"@langfuse/client\\";\\nconst langfuse = new LangfuseClient();\\nlangfuse.score.create({\\n traceId: message.traceId,\\n observationId: message.generationId, // optional\\n name: \\"accuracy\\",\\n value: 0.9,\\n comment: \\"Factually correct\\", // optional\\n id: \\"unique_id\\", // optional, can be used as an idempotency key to update the score subsequently\\n configId: \\"78545-6565-3453654-43543\\", // optional, to ensure that the score follows a specific min/max value range\\n dataType: \\"NUMERIC\\", // optional, possibly inferred\\n});\\n// Flush the scores in short-lived environments\\nawait langfuse.flush();\\n```\\nCategorical scores are used to evaluate data that falls into specific categories. When ingesting categorical scores, you can provide the value as a string. If you provide a configId, the score value will be validated against the config\'s categories.\\n \\n \\n```"}, {"type": "text", "text": "\\nimport { LangfuseClient } from \\"@langfuse/client\\";\\nconst langfuse = new LangfuseClient();\\nlangfuse.score.create({\\n traceId: message.traceId,\\n observationId: message.generationId, // optional\\n name: \\"correctness\\",\\n value: \\"correct\\",\\n comment: \\"Factually correct\\", // optional\\n id: \\"unique_id\\", // optional, can be used as an idempotency key to update the score subsequently\\n configId: \\"12345-6565-3453654-43543\\", // optional, to ensure that a score maps to a specific category defined in a score config\\n dataType: \\"CATEGORICAL\\", // optional, possibly inferred\\n});\\n// Flush the scores in short-lived environments\\nawait langfuse.flush();\\n```\\nWhen ingesting boolean scores, you can provide the value as a float. If you provide a configId, the score\'s name and config\'s name must match as well as their data types.\\n \\n \\n```\\nimport { LangfuseClient } from \\"@langfuse/client\\";\\nconst langfuse = new LangfuseClient();\\nlangfuse.score.create({\\n traceId: message.traceId,\\n observationId: message.generationId, // optional\\n name: \\"helpfulness\\",\\n value: 1,\\n comment: \\"Factually correct\\", // optional\\n id: \\"unique_id\\", // optional, can be used as an idempotency key to update the score subsequently\\n configId: \\"93547-6565-3453654-43543\\", // optional, can be used to infer the score data type and validate the score value\\n dataType: \\"BOOLEAN\\", // optional, possibly inferred\\n});\\n// Flush the scores in short-lived environments\\nawait langfuse.flush();\\n```\\nWhen ingesting text scores, provide the value as a string (1β500 characters). If you provide a configId, the score\'s name and config\'s name must match as well as their data types.\\n \\n \\n```\\nimport { LangfuseClient } from \\"@langfuse/client\\";\\nconst langfuse = new LangfuseClient();\\nlangfuse.score.create({\\n traceId: message.traceId,\\n observationId: message.generationId, // optional\\n name: \\"reviewer_notes\\",\\n value: \\"The response was helpful but could be more concise.\\","}, {"type": "text", "text": "\\nWhen ingesting boolean scores, you can provide the value as a float. If you provide a configId, the score\'s name and config\'s name must match as well as their data types.\\n \\n \\n```\\ncurl -X POST https://cloud.langfuse.com/api/public/scores \\\\\\n -u \\"pk-lf-...\\":\\"sk-lf-...\\" \\\\\\n -H \\"Content-Type: application/json\\" \\\\\\n -d \'{\\n \\"id\\": \\"unique_id\\",\\n \\"traceId\\": \\"trace_id_here\\",\\n \\"observationId\\": \\"observation_id_here\\",\\n \\"name\\": \\"helpfulness\\",\\n \\"value\\": 1,\\n \\"dataType\\": \\"BOOLEAN\\",\\n \\"configId\\": \\"93547-6565-3453654-43543\\",\\n \\"comment\\": \\"Factually correct\\"\\n }\'\\n```\\nWhen ingesting text scores, provide the value as a string (1β500 characters). If you provide a configId, the score\'s name and config\'s name must match as well as their data types.\\n \\n \\n```\\ncurl -X POST https://cloud.langfuse.com/api/public/scores \\\\\\n -u \\"pk-lf-...\\":\\"sk-lf-...\\" \\\\\\n -H \\"Content-Type: application/json\\" \\\\\\n -d \'{\\n \\"id\\": \\"unique_id\\",\\n \\"traceId\\": \\"trace_id_here\\",\\n \\"observationId\\": \\"observation_id_here\\",\\n \\"name\\": \\"reviewer_notes\\",\\n \\"value\\": \\"The response was helpful but could be more concise.\\",\\n \\"dataType\\": \\"TEXT\\",\\n \\"configId\\": \\"24680-6565-3453654-43543\\",\\n \\"comment\\": \\"Reviewed by QA team\\"\\n }\'\\n```\\nSee [API reference](/docs/api) for more details on POST/GET score configs endpoints.\\n### Inferred Score Properties\\nCertain score properties might be inferred based on your input:\\n * **If you don\'t provide a score data type** it will always be inferred. See tables below for details."}, {"type": "text", "text": "\\n * **For boolean and categorical scores** , we will provide the score value in both numerical and string format where possible. The score value format that is not provided as input, i.e. the translated value is referred to as the inferred value in the tables below.\\n * **On read via the[v3 scores API](https://api.reference.langfuse.com/#tag/scoresv3/GET/api/public/v3/scores)**, boolean scores are returned as a single boolean `value`.\\n * **For categorical scores** , the string representation is always provided and a numerical mapping of the category will be produced only if a `ScoreConfig` was provided.\\nDetailed Examples:\\nNumeric ScoresCategorical ScoresBoolean ScoresText Scores\\nFor example, let\'s assume you\'d like to ingest a numeric score to measure **accuracy**. We have included a table of possible score ingestion scenarios below.\\nValue| Data Type| Config Id| Description| Inferred Data Type| Valid \\n---|---|---|---|---|--- \\n`0.9`| `Null`| `Null`| Data type is inferred| `NUMERIC`| Yes \\n`0.9`| `NUMERIC`| `Null`| No properties inferred| | Yes \\n`depth`| `NUMERIC`| `Null`| Error: data type of value does not match provided data type| | No \\n`0.9`| `NUMERIC`| `78545`| No properties inferred| | Conditional on config validation \\n`0.9`| `Null`| `78545`| Data type inferred| `NUMERIC`| Conditional on config validation \\n`depth`| `NUMERIC`| `78545`| Error: data type of value does not match provided data type| | No \\n \\nFor example, let\'s assume you\'d like to ingest a categorical score to measure **correctness**. We have included a table of possible score ingestion scenarios below."}, {"type": "text", "text": "Value| Data Type| Config Id| Description| Inferred Data Type| Inferred Value representation| Valid \\n---|---|---|---|---|---|--- \\n`correct`| `Null`| `Null`| Data type is inferred| `CATEGORICAL`| | Yes \\n`correct`| `CATEGORICAL`| `Null`| No properties inferred| | | Yes \\n`1`| `CATEGORICAL`| `Null`| Error: data type of value does not match provided data type| | | No \\n`correct`| `CATEGORICAL`| `12345`| Numeric value inferred| | `4` numeric config category mapping| Conditional on config validation \\n`correct`| `NULL`| `12345`| Data type inferred| `CATEGORICAL`| | Conditional on config validation \\n`1`| `CATEGORICAL`| `12345`| Error: data type of value does not match provided data type| | | No \\n \\nFor example, let\'s assume you\'d like to ingest a boolean score to measure **helpfulness**. We have included a table of possible score ingestion scenarios below.\\nValue| Data Type| Config Id| Description| Inferred Data Type| Inferred Value representation| Valid \\n---|---|---|---|---|---|--- \\n`1`| `BOOLEAN`| `Null`| Value\'s string equivalent inferred| | `True`| Yes \\n`true`| `BOOLEAN`| `Null`| Error: data type of value does not match provided data type| | | No \\n`3`| `BOOLEAN`| `Null`| Error: boolean data type expects `0` or `1` as input value| | | No \\n`0.9`| `Null`| `93547`| Data type and value\'s string equivalent inferred| `BOOLEAN`| `True`| Conditional on config validation \\n`depth`| `BOOLEAN`| `93547`| Error: data type of value does not match provided data type| | | No \\n \\nFor example, let\'s assume you\'d like to ingest a text score to capture **reviewer notes**. Text scores must be non-empty strings of at most 500 characters.\\nValue| Data Type| Config Id| Description| Inferred Data Type| Valid "}, {"type": "text", "text": "---|---|---|---|---|--- \\n`\\"Good response\\"`| `Null`| `Null`| Data type is inferred| `TEXT`| Yes \\n`\\"Good response\\"`| `TEXT`| `Null`| No properties inferred| | Yes \\n`0.9`| `TEXT`| `Null`| Error: data type of value does not match provided data type| | No \\n`\\"Good response\\"`| `TEXT`| `24680`| No properties inferred| | Conditional on config validation \\n`\\"Good response\\"`| `Null`| `24680`| Data type inferred| `TEXT`| Conditional on config validation \\n`\\"\\"`| `TEXT`| `Null`| Error: text scores must be non-empty| | No \\n \\n## Update Existing Scores via API/SDKs\\nWhen creating a score, you can provide an optional `id` (JS/TS) / `score_id` (Python) parameter. A re-ingested score overwrites an existing one only when its `id`, `name`, and `timestamp` (at date granularity, `toDate(timestamp)`) all match β a matching `id` alone is not sufficient, and any difference creates a duplicate instead of an update.\\nTo overwrite a score without fetching existing scores first, set a stable `id` as an idempotency key when initially creating the score, and keep the `name` and `timestamp` unchanged on subsequent calls. See Preventing Duplicate Scores for details.\\n## Related guides\\nGuides\\n * [Collecting user feedback\\nCapture explicit and implicit user feedback as scores on your traces, including in-app ratings sent via our Browser SDK.\\n](/guides/user-feedback-loop)\\n * [Custom evaluation data pipeline\\nContinuously monitor quality by fetching traces, running custom evaluations, and ingesting scores back into Langfuse.\\n](/guides/cookbook/example_external_evaluation_pipelines)\\n * [Guardrails and security checks"}, {"type": "text", "text": "\\nCheck whether output contains a certain keyword, matches a required structure or format, or exceeds a length limit.\\n](/guides/cookbook/security-and-guardrails)\\n * [Custom internal workflow tooling\\nBuild internal tooling for human-in-the-loop workflows, ingesting scores back into Langfuse following your custom schema.\\n](/guides/human-in-the-loop-scoring)\\n * [Session-level quality tracking\\nScore full conversations, such as support chats or agent threads, by attaching scores via sessionId in the SDK or API.\\n](/guides/cookbook/example_evaluating_multi_turn_conversations)\\n* * *\\n* * *\\nLast edited 7/22/2026\\n"}], "type": "content"}}, {"type": "document", "record_type": "documentation", "url": "https://langfuse.com/docs/evaluation/evaluation-methods/scores-via-sdk", "title": "Scores via API/SDK", "source": {"content": [{"type": "text", "text": "# Scores via API/SDK\\nYou can use the Langfuse SDKs or API to add [scores](/docs/evaluation/scores/overview) to traces, observations, sessions and dataset runs. This is an evaluation method that allows you to set up custom evaluation workflows and extend the scoring capabilities of Langfuse. See the [data model](/docs/evaluation/scores/data-model#scores) for full details on the score object.\\nIf you want Langfuse to run deterministic Python or TypeScript logic for you, use [code evaluators](/docs/evaluation/evaluation-methods/code-evaluators). Use this page when your application, pipeline, or CI job computes scores and sends them to Langfuse via API or SDK.\\n\\n## Ingesting Scores via API/SDKs\\nScores can be attached at different levels of granularity: to individual traces, to specific observations within a trace, or to full sessions.\\nSee the [API reference](/docs/api) for full details on the POST and GET endpoints for scores and score configs.\\n### Trace or Observation-level Scores\\nYou can add scores via the Langfuse SDKs or API. Scores can take one of four data types: **Numeric** , **Categorical** , **Boolean** , or **Text**. See [Score Types](/docs/evaluation/scores/overview#score-types) for details.\\nIf a score is ingested manually using a `trace_id` to link the score to a trace, it is not necessary to wait until the trace has been created. The score will show up in the scores table and will be linked to the trace once the trace with the same `trace_id` is created.\\nHere are examples by `Score` data types.\\nFor trace and observation scores, `trace_id`/`traceId` is required and `observation_id`/`observationId` is optional. If you attach a score to an observation, always provide both the observation ID and the corresponding trace ID.\\nPython SDKJS/TS SDKAPI\\nNumericCategoricalBooleanText\\nNumeric score values must be provided as float.\\n \\n \\n```"}, {"type": "text", "text": "\\nfrom langfuse import get_client\\nlangfuse = get_client()\\n# Method 1: Score via low-level method\\nlangfuse.create_score(\\n name=\\"correctness\\",\\n value=0.9,\\n trace_id=\\"trace_id_here\\",\\n observation_id=\\"observation_id_here\\", # optional\\n data_type=\\"NUMERIC\\", # optional, inferred if not provided\\n comment=\\"Factually correct\\", # optional\\n)\\n# Method 2: Score current span/generation (within context)\\nwith langfuse.start_as_current_observation(as_type=\\"span\\", name=\\"my-operation\\") as span:\\n # Score the current span\\n span.score(\\n name=\\"correctness\\",\\n value=0.9,\\n data_type=\\"NUMERIC\\",\\n comment=\\"Factually correct\\"\\n )\\n # Score the trace\\n span.score_trace(\\n name=\\"overall_quality\\",\\n value=0.95,\\n data_type=\\"NUMERIC\\"\\n )\\n# Method 3: Score via the current context\\nwith langfuse.start_as_current_observation(as_type=\\"span\\", name=\\"my-operation\\"):\\n # Score the current span\\n langfuse.score_current_span(\\n name=\\"correctness\\",\\n value=0.9,\\n data_type=\\"NUMERIC\\",\\n comment=\\"Factually correct\\"\\n )\\n # Score the trace\\n langfuse.score_current_trace(\\n name=\\"overall_quality\\",\\n value=0.95,\\n data_type=\\"NUMERIC\\"\\n )\\n```\\nCategorical score values must be provided as strings.\\n \\n \\n```\\nfrom langfuse import get_client\\nlangfuse = get_client()\\n# Method 1: Score via low-level method\\nlangfuse.create_score(\\n name=\\"accuracy\\",\\n value=\\"partially correct\\",\\n trace_id=\\"trace_id_here\\",\\n observation_id=\\"observation_id_here\\", # optional\\n data_type=\\"CATEGORICAL\\", # optional, inferred if not provided\\n comment=\\"Some factual errors\\", # optional\\n)\\n# Method 2: Score current span/generation (within context)\\nwith langfuse.start_as_current_observation(as_type=\\"span\\", name=\\"my-operation\\") as span:\\n # Score the current span\\n span.score(\\n name=\\"accuracy\\",\\n value=\\"partially correct\\",\\n data_type=\\"CATEGORICAL\\",\\n comment=\\"Some factual errors\\"\\n )\\n # Score the trace\\n span.score_trace(\\n name=\\"overall_quality\\",\\n value=\\"partially correct\\","}, {"type": "text", "text": "\\n data_type=\\"CATEGORICAL\\"\\n )\\n# Method 3: Score via the current context\\nwith langfuse.start_as_current_observation(as_type=\\"span\\", name=\\"my-operation\\"):\\n # Score the current span\\n langfuse.score_current_span(\\n name=\\"accuracy\\",\\n value=\\"partially correct\\",\\n data_type=\\"CATEGORICAL\\",\\n comment=\\"Some factual errors\\"\\n )\\n # Score the trace\\n langfuse.score_current_trace(\\n name=\\"overall_quality\\",\\n value=\\"partially correct\\",\\n data_type=\\"CATEGORICAL\\"\\n )\\n```\\nBoolean scores must be provided as a float, where `1` is true and `0` is false. When read via the [v3 scores API](https://api.reference.langfuse.com/#tag/scoresv3/GET/api/public/v3/scores), the value is returned as a boolean. See [API reference](/docs/api) for more details on POST/GET scores endpoints.\\n \\n \\n```\\nfrom langfuse import get_client\\nlangfuse = get_client()\\n# Method 1: Score via low-level method\\nlangfuse.create_score(\\n name=\\"helpfulness\\",\\n value=0, # 0 or 1\\n trace_id=\\"trace_id_here\\",\\n observation_id=\\"observation_id_here\\", # optional\\n data_type=\\"BOOLEAN\\", # required, numeric values without data type would be inferred as NUMERIC\\n comment=\\"Incorrect answer\\", # optional\\n)\\n# Method 2: Score current span/generation (within context)\\nwith langfuse.start_as_current_observation(as_type=\\"span\\", name=\\"my-operation\\") as span:\\n # Score the current span\\n span.score(\\n name=\\"helpfulness\\",\\n value=1, # 0 or 1\\n data_type=\\"BOOLEAN\\",\\n comment=\\"Very helpful response\\"\\n )\\n # Score the trace\\n span.score_trace(\\n name=\\"overall_quality\\",\\n value=1, # 0 or 1\\n data_type=\\"BOOLEAN\\"\\n )\\n# Method 3: Score via the current context\\nwith langfuse.start_as_current_observation(as_type=\\"span\\", name=\\"my-operation\\"):\\n # Score the current span\\n langfuse.score_current_span(\\n name=\\"helpfulness\\","}, {"type": "text", "text": "\\n value=1, # 0 or 1\\n data_type=\\"BOOLEAN\\",\\n comment=\\"Very helpful response\\"\\n )\\n # Score the trace\\n langfuse.score_current_trace(\\n name=\\"overall_quality\\",\\n value=1, # 0 or 1\\n data_type=\\"BOOLEAN\\"\\n )\\n```\\nText score values must be provided as strings between 1 and 500 characters.\\n \\n \\n```\\nfrom langfuse import get_client\\nlangfuse = get_client()\\n# Method 1: Score via low-level method\\nlangfuse.create_score(\\n name=\\"reviewer_notes\\",\\n value=\\"The response was helpful but could be more concise.\\",\\n trace_id=\\"trace_id_here\\",\\n observation_id=\\"observation_id_here\\", # optional\\n data_type=\\"TEXT\\", # optional, inferred if not provided\\n comment=\\"Reviewed by QA team\\", # optional\\n)\\n# Method 2: Score current span/generation (within context)\\nwith langfuse.start_as_current_observation(as_type=\\"span\\", name=\\"my-operation\\") as span:\\n # Score the current span\\n span.score(\\n name=\\"reviewer_notes\\",\\n value=\\"The response was helpful but could be more concise.\\",\\n data_type=\\"TEXT\\",\\n comment=\\"Reviewed by QA team\\"\\n )\\n # Score the trace\\n span.score_trace(\\n name=\\"overall_notes\\",\\n value=\\"Good quality overall, minor formatting issues.\\",\\n data_type=\\"TEXT\\"\\n )\\n# Method 3: Score via the current context\\nwith langfuse.start_as_current_observation(as_type=\\"span\\", name=\\"my-operation\\"):\\n # Score the current span\\n langfuse.score_current_span(\\n name=\\"reviewer_notes\\",\\n value=\\"The response was helpful but could be more concise.\\",\\n data_type=\\"TEXT\\",\\n comment=\\"Reviewed by QA team\\"\\n )\\n # Score the trace\\n langfuse.score_current_trace(\\n name=\\"overall_notes\\",\\n value=\\"Good quality overall, minor formatting issues.\\",\\n data_type=\\"TEXT\\"\\n )\\n```\\nNumericCategoricalBooleanText\\nNumeric score values must be provided as float.\\n \\n \\n```"}, {"type": "text", "text": "\\nimport { LangfuseClient } from \\"@langfuse/client\\";\\nconst langfuse = new LangfuseClient();\\nlangfuse.score.create({\\n id: \\"unique_id\\", // optional, can be used as an idempotency key to update the score subsequently\\n traceId: message.traceId,\\n observationId: message.generationId, // optional\\n name: \\"correctness\\",\\n value: 0.9,\\n dataType: \\"NUMERIC\\", // optional, inferred if not provided\\n comment: \\"Factually correct\\", // optional\\n});\\n// Flush the scores in short-lived environments\\nawait langfuse.flush();\\n```\\nCategorical score values must be provided as strings.\\n \\n \\n```\\nimport { LangfuseClient } from \\"@langfuse/client\\";\\nconst langfuse = new LangfuseClient();\\nlangfuse.score.create({\\n id: \\"unique_id\\", // optional, can be used as an idempotency key to update the score subsequently\\n traceId: message.traceId,\\n observationId: message.generationId, // optional\\n name: \\"accuracy\\",\\n value: \\"partially correct\\",\\n dataType: \\"CATEGORICAL\\", // optional, inferred if not provided\\n comment: \\"Factually correct\\", // optional\\n});\\n// Flush the scores in short-lived environments\\nawait langfuse.flush();\\n```\\nBoolean scores must be provided as a float, where `1` is true and `0` is false. When read via the [v3 scores API](https://api.reference.langfuse.com/#tag/scoresv3/GET/api/public/v3/scores), the value is returned as a boolean. See [API reference](/docs/api) for more details on POST/GET scores endpoints.\\n \\n \\n```\\nimport { LangfuseClient } from \\"@langfuse/client\\";\\nconst langfuse = new LangfuseClient();\\nlangfuse.score.create({\\n id: \\"unique_id\\", // optional, can be used as an idempotency key to update the score subsequently\\n traceId: message.traceId,\\n observationId: message.generationId, // optional"}, {"type": "text", "text": "\\nnpm install @langfuse/browser\\n```\\n \\n \\n```\\nimport { LangfuseBrowserClient } from \\"@langfuse/browser\\";\\nconst langfuse = new LangfuseBrowserClient({\\n publicKey: process.env.NEXT_PUBLIC_LANGFUSE_PUBLIC_KEY!,\\n baseUrl: process.env.NEXT_PUBLIC_LANGFUSE_BASE_URL, // optional, defaults to https://cloud.langfuse.com\\n});\\nawait langfuse.score({\\n traceId: message.traceId,\\n observationId: message.generationId, // optional\\n id: `user-feedback-${message.traceId}`, // optional, use as an idempotency key\\n name: \\"user-feedback\\",\\n value: 1, // 1 for positive, 0 for negative\\n dataType: \\"BOOLEAN\\",\\n comment: \\"Helpful answer\\", // optional\\n});\\n```\\n`score()` returns `{ id }` after ingestion succeeds. No `flush()` call is required.\\n### Session-level Scores\\nTo score an entire session (without attaching the score to a trace or observation), provide only `session_id` (Python SDK) or `sessionId` (JS/TS SDK and API).\\nPython SDKJS/TS SDKAPI\\n \\n \\n```\\nfrom langfuse import get_client\\nlangfuse = get_client()\\nlangfuse.create_score(\\n name=\\"session_quality\\",\\n value=0.85,\\n session_id=\\"session_id_here\\",\\n data_type=\\"NUMERIC\\",\\n comment=\\"Overall conversation quality\\"\\n)\\n```\\n \\n \\n```\\nimport { LangfuseClient } from \\"@langfuse/client\\";\\nconst langfuse = new LangfuseClient();\\nlangfuse.score.create({\\n name: \\"session_quality\\",\\n value: 0.85,\\n sessionId: \\"session_id_here\\",\\n dataType: \\"NUMERIC\\",\\n comment: \\"Overall conversation quality\\",\\n});\\nawait langfuse.flush();\\n```\\n \\n \\n```\\ncurl -X POST https://cloud.langfuse.com/api/public/scores \\\\\\n -u \\"pk-lf-...\\":\\"sk-lf-...\\" \\\\\\n -H \\"Content-Type: application/json\\" \\\\\\n -d \'{\\n \\"sessionId\\": \\"session_id_here\\",\\n \\"name\\": \\"session_quality\\",\\n \\"value\\": 0.85,\\n \\"dataType\\": \\"NUMERIC\\",\\n \\"comment\\": \\"Overall conversation quality\\"\\n }\'\\n```\\n"}], "type": "content"}}, {"type": "document", "record_type": "documentation", "url": "https://langfuse.com/docs/evaluation/evaluation-methods/scores-via-sdk", "title": "Scores via API/SDK", "source": {"content": [{"type": "text", "text": "## Advanced\\n### Preventing Duplicate Scores\\nBy default, Langfuse allows for multiple scores of the same `name` on the same trace. This is useful if you\'d like to track the evolution of a score over time or if e.g. you\'ve received multiple user feedback scores on the same trace.\\nIn some cases, you want to prevent this behavior or overwrite an existing score. A score is identified by three fields β its `id`, its `name`, and its date (`toDate(timestamp)`) β and a new score only replaces an existing one when all three match. To achieve this, create an **idempotency key** and pass it as the `id` (JS/TS) / `score_id` (Python) when creating the score (for example `trace_id-score_name`), and keep the `name` and `timestamp` unchanged across calls. If any of the three differs β for example the same `id` with a different `name` β you get an additional score rather than a replacement.\\nDo not send partial score updates (only the changed fields) and rely on Langfuse to merge them into the existing record. This merge only happens for a limited window after creation, is deprecated, and will be removed. Always send the complete score.\\nSee [How to update traces, observations, and scores](/faq/all/tracing-data-updates#updating-scores) for more details on Langfuse\'s immutable data model.\\n### Enforcing a Score Config\\nScore configs are helpful when you want to standardize your scores for future analysis.\\nTo enforce a score config, you can provide a `configId` when creating a score to reference a `ScoreConfig` that was previously created. `Score Configs` can be defined in the Langfuse UI or via our API. [See our guide on how to create and manage score configs](/faq/all/manage-score-configs).\\nWhenever you provide a `ScoreConfig`, the score data will be validated against the config. The following rules apply:\\n * **Score Name** : Must equal the config\'s name"}, {"type": "text", "text": "\\n * **Score Data Type** : When provided, must match the config\'s data type\\n * **Score Value when Type is`NUMERIC`**: Value must be within the min and max values defined in the config (if provided, min and max are optional and otherwise are assumed as -β and +β respectively)\\n * **Score Value when Type is`CATEGORICAL`**: Value must map to one of the categories defined in the config\\n * **Score Value when Type is`BOOLEAN`**: Value must equal `0` or `1`\\n * **Score Value when Type is`TEXT`**: Value must be a non-empty string of at most 500 characters\\nPython SDKJS/TS SDKAPI\\nNumeric ScoresCategorical ScoresBoolean ScoresText Scores\\nWhen ingesting numeric scores, you can provide the value as a float. If you provide a configId, the score value will be validated against the config\'s numeric range, which might be defined by a minimum and/or maximum value.\\n \\n \\n```\\nfrom langfuse import get_client\\nlangfuse = get_client()\\n# Method 1: Score via low-level method\\nlangfuse.create_score(\\n trace_id=\\"trace_id_here\\",\\n observation_id=\\"observation_id_here\\", # optional\\n session_id=\\"session_id_here\\", # optional, ID of the session the score relates to\\n name=\\"accuracy\\",\\n value=0.9,\\n comment=\\"Factually correct\\", # optional\\n score_id=\\"unique_id\\", # optional, can be used as an idempotency key to update the score subsequently\\n config_id=\\"78545-6565-3453654-43543\\", # optional, to ensure that the score follows a specific min/max value range\\n data_type=\\"NUMERIC\\" # optional, possibly inferred\\n)\\n# Method 2: Score within context\\nwith langfuse.start_as_current_observation(as_type=\\"span\\", name=\\"my-operation\\") as span:\\n span.score(\\n name=\\"accuracy\\",\\n value=0.9,\\n comment=\\"Factually correct\\",\\n config_id=\\"78545-6565-3453654-43543\\",\\n data_type=\\"NUMERIC\\"\\n )\\n```"}, {"type": "text", "text": "\\nWhen ingesting numeric scores, you can provide the value as a float. If you provide a configId, the score value will be validated against the config\'s numeric range, which might be defined by a minimum and/or maximum value.\\n \\n \\n```\\nimport { LangfuseClient } from \\"@langfuse/client\\";\\nconst langfuse = new LangfuseClient();\\nlangfuse.score.create({\\n traceId: message.traceId,\\n observationId: message.generationId, // optional\\n name: \\"accuracy\\",\\n value: 0.9,\\n comment: \\"Factually correct\\", // optional\\n id: \\"unique_id\\", // optional, can be used as an idempotency key to update the score subsequently\\n configId: \\"78545-6565-3453654-43543\\", // optional, to ensure that the score follows a specific min/max value range\\n dataType: \\"NUMERIC\\", // optional, possibly inferred\\n});\\n// Flush the scores in short-lived environments\\nawait langfuse.flush();\\n```\\nCategorical scores are used to evaluate data that falls into specific categories. When ingesting categorical scores, you can provide the value as a string. If you provide a configId, the score value will be validated against the config\'s categories.\\n \\n \\n```\\nimport { LangfuseClient } from \\"@langfuse/client\\";\\nconst langfuse = new LangfuseClient();\\nlangfuse.score.create({\\n traceId: message.traceId,\\n observationId: message.generationId, // optional\\n name: \\"correctness\\",\\n value: \\"correct\\",\\n comment: \\"Factually correct\\", // optional\\n id: \\"unique_id\\", // optional, can be used as an idempotency key to update the score subsequently\\n configId: \\"12345-6565-3453654-43543\\", // optional, to ensure that a score maps to a specific category defined in a score config\\n dataType: \\"CATEGORICAL\\", // optional, possibly inferred\\n});"}, {"type": "text", "text": "\\nYou can also enforce score configs via the [REST API](https://api.reference.langfuse.com/#tag/legacyscorev1/POST/api/public/scores) by providing a `configId`.\\nNumeric ScoresCategorical ScoresBoolean ScoresText Scores\\nWhen ingesting numeric scores, you can provide the value as a float. If you provide a configId, the score value will be validated against the config\'s numeric range.\\n \\n \\n```\\ncurl -X POST https://cloud.langfuse.com/api/public/scores \\\\\\n -u \\"pk-lf-...\\":\\"sk-lf-...\\" \\\\\\n -H \\"Content-Type: application/json\\" \\\\\\n -d \'{\\n \\"id\\": \\"unique_id\\",\\n \\"traceId\\": \\"trace_id_here\\",\\n \\"observationId\\": \\"observation_id_here\\",\\n \\"name\\": \\"accuracy\\",\\n \\"value\\": 0.9,\\n \\"dataType\\": \\"NUMERIC\\",\\n \\"configId\\": \\"78545-6565-3453654-43543\\",\\n \\"comment\\": \\"Factually correct\\"\\n }\'\\n```\\nCategorical scores are used to evaluate data that falls into specific categories. If you provide a configId, the score value will be validated against the config\'s categories.\\n \\n \\n```\\ncurl -X POST https://cloud.langfuse.com/api/public/scores \\\\\\n -u \\"pk-lf-...\\":\\"sk-lf-...\\" \\\\\\n -H \\"Content-Type: application/json\\" \\\\\\n -d \'{\\n \\"id\\": \\"unique_id\\",\\n \\"traceId\\": \\"trace_id_here\\",\\n \\"observationId\\": \\"observation_id_here\\",\\n \\"name\\": \\"correctness\\",\\n \\"value\\": \\"correct\\",\\n \\"dataType\\": \\"CATEGORICAL\\",\\n \\"configId\\": \\"12345-6565-3453654-43543\\",\\n \\"comment\\": \\"Factually correct\\"\\n }\'\\n```\\nWhen ingesting boolean scores, you can provide the value as a float. If you provide a configId, the score\'s name and config\'s name must match as well as their data types.\\n \\n \\n```\\ncurl -X POST https://cloud.langfuse.com/api/public/scores \\\\\\n -u \\"pk-lf-...\\":\\"sk-lf-...\\" \\\\\\n -H \\"Content-Type: application/json\\" \\\\\\n -d \'{\\n \\"id\\": \\"unique_id\\",\\n \\"traceId\\": \\"trace_id_here\\",\\n \\"observationId\\": \\"observation_id_here\\",\\n \\"name\\": \\"helpfulness\\","}, {"type": "text", "text": "\\n \\"value\\": 1,\\n \\"dataType\\": \\"BOOLEAN\\",\\n \\"configId\\": \\"93547-6565-3453654-43543\\",\\n \\"comment\\": \\"Factually correct\\"\\n }\'\\n```\\nWhen ingesting text scores, provide the value as a string (1β500 characters). If you provide a configId, the score\'s name and config\'s name must match as well as their data types.\\n \\n \\n```\\ncurl -X POST https://cloud.langfuse.com/api/public/scores \\\\\\n -u \\"pk-lf-...\\":\\"sk-lf-...\\" \\\\\\n -H \\"Content-Type: application/json\\" \\\\\\n -d \'{\\n \\"id\\": \\"unique_id\\",\\n \\"traceId\\": \\"trace_id_here\\",\\n \\"observationId\\": \\"observation_id_here\\",\\n \\"name\\": \\"reviewer_notes\\",\\n \\"value\\": \\"The response was helpful but could be more concise.\\",\\n \\"dataType\\": \\"TEXT\\",\\n \\"configId\\": \\"24680-6565-3453654-43543\\",\\n \\"comment\\": \\"Reviewed by QA team\\"\\n }\'\\n```\\nSee [API reference](/docs/api) for more details on POST/GET score configs endpoints.\\n### Inferred Score Properties\\nCertain score properties might be inferred based on your input:\\n * **If you don\'t provide a score data type** it will always be inferred. See tables below for details.\\n * **For boolean and categorical scores** , we will provide the score value in both numerical and string format where possible. The score value format that is not provided as input, i.e. the translated value is referred to as the inferred value in the tables below.\\n * **On read via the[v3 scores API](https://api.reference.langfuse.com/#tag/scoresv3/GET/api/public/v3/scores)**, boolean scores are returned as a single boolean `value`.\\n * **For categorical scores** , the string representation is always provided and a numerical mapping of the category will be produced only if a `ScoreConfig` was provided.\\nDetailed Examples:"}, {"type": "text", "text": "\\nNumeric ScoresCategorical ScoresBoolean ScoresText Scores\\nFor example, let\'s assume you\'d like to ingest a numeric score to measure **accuracy**. We have included a table of possible score ingestion scenarios below.\\nValue| Data Type| Config Id| Description| Inferred Data Type| Valid \\n---|---|---|---|---|--- \\n`0.9`| `Null`| `Null`| Data type is inferred| `NUMERIC`| Yes \\n`0.9`| `NUMERIC`| `Null`| No properties inferred| | Yes \\n`depth`| `NUMERIC`| `Null`| Error: data type of value does not match provided data type| | No \\n`0.9`| `NUMERIC`| `78545`| No properties inferred| | Conditional on config validation \\n`0.9`| `Null`| `78545`| Data type inferred| `NUMERIC`| Conditional on config validation \\n`depth`| `NUMERIC`| `78545`| Error: data type of value does not match provided data type| | No \\n \\nFor example, let\'s assume you\'d like to ingest a categorical score to measure **correctness**. We have included a table of possible score ingestion scenarios below.\\nValue| Data Type| Config Id| Description| Inferred Data Type| Inferred Value representation| Valid \\n---|---|---|---|---|---|--- \\n`correct`| `Null`| `Null`| Data type is inferred| `CATEGORICAL`| | Yes \\n`correct`| `CATEGORICAL`| `Null`| No properties inferred| | | Yes \\n`1`| `CATEGORICAL`| `Null`| Error: data type of value does not match provided data type| | | No \\n`correct`| `CATEGORICAL`| `12345`| Numeric value inferred| | `4` numeric config category mapping| Conditional on config validation \\n`correct`| `NULL`| `12345`| Data type inferred| `CATEGORICAL`| | Conditional on config validation \\n`1`| `CATEGORICAL`| `12345`| Error: data type of value does not match provided data type| | | No \\n "}, {"type": "text", "text": "For example, let\'s assume you\'d like to ingest a boolean score to measure **helpfulness**. We have included a table of possible score ingestion scenarios below.\\nValue| Data Type| Config Id| Description| Inferred Data Type| Inferred Value representation| Valid \\n---|---|---|---|---|---|--- \\n`1`| `BOOLEAN`| `Null`| Value\'s string equivalent inferred| | `True`| Yes \\n`true`| `BOOLEAN`| `Null`| Error: data type of value does not match provided data type| | | No \\n`3`| `BOOLEAN`| `Null`| Error: boolean data type expects `0` or `1` as input value| | | No \\n`0.9`| `Null`| `93547`| Data type and value\'s string equivalent inferred| `BOOLEAN`| `True`| Conditional on config validation \\n`depth`| `BOOLEAN`| `93547`| Error: data type of value does not match provided data type| | | No \\n \\nFor example, let\'s assume you\'d like to ingest a text score to capture **reviewer notes**. Text scores must be non-empty strings of at most 500 characters.\\nValue| Data Type| Config Id| Description| Inferred Data Type| Valid \\n---|---|---|---|---|--- \\n`\\"Good response\\"`| `Null`| `Null`| Data type is inferred| `TEXT`| Yes \\n`\\"Good response\\"`| `TEXT`| `Null`| No properties inferred| | Yes \\n`0.9`| `TEXT`| `Null`| Error: data type of value does not match provided data type| | No \\n`\\"Good response\\"`| `TEXT`| `24680`| No properties inferred| | Conditional on config validation \\n`\\"Good response\\"`| `Null`| `24680`| Data type inferred| `TEXT`| Conditional on config validation \\n`\\"\\"`| `TEXT`| `Null`| Error: text scores must be non-empty| | No \\n \\n\\n## Update Existing Scores via API/SDKs"}, {"type": "text", "text": "When creating a score, you can provide an optional `id` (JS/TS) / `score_id` (Python) parameter. A re-ingested score overwrites an existing one only when its `id`, `name`, and `timestamp` (at date granularity, `toDate(timestamp)`) all match β a matching `id` alone is not sufficient, and any difference creates a duplicate instead of an update.\\nTo overwrite a score without fetching existing scores first, set a stable `id` as an idempotency key when initially creating the score, and keep the `name` and `timestamp` unchanged on subsequent calls. See Preventing Duplicate Scores for details.\\n## Related guides\\nGuides\\n * [Collecting user feedback\\nCapture explicit and implicit user feedback as scores on your traces, including in-app ratings sent via our Browser SDK.\\n](/guides/user-feedback-loop)\\n * [Custom evaluation data pipeline\\nContinuously monitor quality by fetching traces, running custom evaluations, and ingesting scores back into Langfuse.\\n](/guides/cookbook/example_external_evaluation_pipelines)\\n * [Guardrails and security checks\\nCheck whether output contains a certain keyword, matches a required structure or format, or exceeds a length limit.\\n](/guides/cookbook/security-and-guardrails)\\n * [Custom internal workflow tooling\\nBuild internal tooling for human-in-the-loop workflows, ingesting scores back into Langfuse following your custom schema.\\n](/guides/human-in-the-loop-scoring)\\n * [Session-level quality tracking\\nScore full conversations, such as support chats or agent threads, by attaching scores via sessionId in the SDK or API.\\n](/guides/cookbook/example_evaluating_multi_turn_conversations)\\n* * *\\n* * *\\nLast edited 7/22/2026\\n"}], "type": "content"}}, {"type": "document", "record_type": "documentation", "url": "https://langfuse.com/docs/evaluation/scores/overview", "title": "Scores", "source": {"content": [{"type": "text", "text": "Overview - Langfuse\\n[Docs](/docs)\\n[Evaluation](/docs/evaluation/overview)\\n[Scores](/docs/evaluation/scores/overview)\\n[Overview](/docs/evaluation/scores/overview)\\n# Scores\\nScores are Langfuse\'s universal data object for storing evaluation results. Any time you want to assign a quality judgment to an LLM output, whether by a [human annotation](/docs/evaluation/evaluation-methods/scores-via-ui), an [LLM judge](/docs/evaluation/evaluation-methods/llm-as-a-judge), a [programmatic check](/docs/evaluation/evaluation-methods/scores-via-sdk), or end-user feedback, the result is stored as a score.\\nEvery score has a **name** (like `\\"correctness\\"` or `\\"helpfulness\\"`), a **value** , and a **data type**. Scores also support an optional **comment** for additional context.\\nScores can be attached to [traces](/docs/observability/data-model#observations-and-traces), [observations](/docs/observability/data-model#observations-and-traces), [sessions](/docs/observability/data-model#sessions), or [dataset runs](/docs/evaluation/experiments/data-model). Most commonly, scores are attached to traces to evaluate a single end-to-end interaction.\\nOnce you have scores, they show up in [score analytics](/docs/evaluation/scores/score-analytics), can be visualized in [custom dashboards](/docs/metrics/features/custom-dashboards), and can be queried via the [API](/docs/api).\\n## When to Use Scores\\nScores become useful when you want to go beyond observing what your application does and start measuring how well it does it. Common use cases:\\n * **Collecting user feedback** : Capture thumbs up/down or star ratings from your users and attach them to traces. See the [user feedback guide](/docs/observability/features/user-feedback)."}, {"type": "text", "text": "\\n * **Monitoring production quality** : Set up automated evaluators (like [LLM-as-a-Judge](/docs/evaluation/evaluation-methods/llm-as-a-judge)) to continuously score live traces for things like hallucination, relevance, or tone.\\n * **Running guardrails** : Score whether outputs pass safety checks like PII detection, format validation, or content policy compliance.\\n * **Comparing changes with experiments** : When you change a prompt, model, or pipeline, run an [experiment](/docs/evaluation/experiments) to score the new version against a dataset.\\n## Score Types\\nLangfuse supports four score data types:\\nType| Value| Use when \\n---|---|--- \\n`NUMERIC`| Float (e.g. `0.9`)| Continuous judgments like accuracy, relevance, or similarity scores \\n`CATEGORICAL`| String from predefined categories (e.g. `\\"correct\\"`, `\\"partially correct\\"`)| Discrete classifications where the set of possible values is known upfront \\n`BOOLEAN`| `0` or `1`| Pass/fail checks like hallucination detection or format validation \\n`TEXT`| Free-form string (1-500 characters)| Open-ended annotations like reviewer notes or qualitative feedback. Often used for [open coding](https://en.wikipedia.org/wiki/Open_coding) before formalizing into quantifiable scores via [axial coding](https://en.wikipedia.org/wiki/Axial_coding). \\n \\nText scores are designed for qualitative, open-ended scoring. Because free-form text cannot be meaningfully aggregated or compared, text scores are not supported in [experiments](/docs/evaluation/core-concepts#experiments), [LLM-as-a-Judge](/docs/evaluation/evaluation-methods/llm-as-a-judge), or [score analytics](/docs/evaluation/scores/score-analytics).\\n## How to Create Scores"}, {"type": "text", "text": "\\nThere are five ways to add scores:\\n * **LLM-as-a-Judge** : Set up [automated evaluators](/docs/evaluation/evaluation-methods/llm-as-a-judge) that score traces based on custom criteria (e.g. hallucination, tone, relevance). These can return numeric or categorical scores plus reasoning, and can run on live production traces or on experiment results.\\n * **Code evaluators** : Run [custom Python or TypeScript evaluators](/docs/evaluation/evaluation-methods/code-evaluators) in Langfuse for deterministic checks such as exact match, JSON validation, or custom business rules.\\n * **Scores via UI** : Team members [manually score](/docs/evaluation/evaluation-methods/scores-via-ui) traces, observations, or sessions directly in the Langfuse UI. Requires a [score config](/faq/all/manage-score-configs) to be set up first.\\n * **Annotation Queues** : Set up [structured review workflows](/docs/evaluation/evaluation-methods/annotation-queues) where reviewers work through batches of traces.\\n * **Scores via API/SDK** : [Programmatically add scores](/docs/evaluation/evaluation-methods/scores-via-sdk) from your application code. This is the way to go for user feedback (thumbs up/down, star ratings), guardrail results, or custom evaluation pipelines.\\n## Should I Use Scores or Tags?\\n| Scores| Tags \\n---|---|--- \\n **Purpose**| Measure _how good_ something is| Describe _what_ something is \\n **Data**| Numeric, categorical, boolean, or text value| Simple string label \\n **When added**| Can be added at any time, including long after the trace was created| Set during tracing and cannot be changed afterwards \\n **Used for**| Quality measurement, analytics, experiments| Filtering, segmentation, organizing \\n "}, {"type": "text", "text": "As a rule of thumb: if you already know the category at tracing time (e.g. which feature or API endpoint triggered the trace), use a [tag](/docs/observability/features/tags). If you need to classify or evaluate traces later, use a score.\\n## Score Comments\\nEvery score supports an optional **comment** field. Use it to capture reasoning (e.g. why an LLM judge assigned a particular score), reviewer notes, or context that helps others understand the score value. Comments are shown alongside scores in the Langfuse UI.\\nUse a `TEXT` score instead of comments to capture standalone qualitative feedback -- comments are best for additional reasoning on an existing score.\\n* * *\\n* * *\\nLast edited 7/23/2026\\n"}], "type": "content"}}, {"type": "document", "record_type": "github_discussion", "url": "https://github.com/orgs/langfuse/discussions/6016", "title": "Using scores in data sets from the SDK?", "source": {"content": [{"type": "text", "text": "Title: Using scores in data sets from the SDK?\\nCategory: Support\\nAuthor: @howinator\\nTimestamp: March 14, 2025\\nBody:\\nHere is a workflow that I want, but I can\'t figure out how to accomplish it without a lot of manual work.\\r\\n\\r\\n1. Collect traces\\r\\n2. Annotate traces with either manual annotation or programmatic feedback from users. For a simple classification, I would use a boolean score where True means the model returned the correct answer and False when it returned the incorrect answer.\\r\\n4. Create a \\"false positive\\" data set composed of all traces where the score was False. <-- I can\'t figure out how to do this without a lot of clicking.\\r\\n5. Run an eval from that data set and use the score to determine if the eval was correct, e.g., something like `(dataset.item.score && dataset.item.output === evalOutput.output) || (!dataset.item.score && dataset.item.output !== evalOutput.output)`. <-- I can\'t figure out how to do this programmatically at all.\\r\\n\\r\\nIs my workflow completely off here? Am I missing something? Or is this flow just not possible in Langfuse today?\\nComments:\\n\\n Comment #1 from @dosubot on March 14, 2025:\\n\\n <!-- Greeting -->\\nHey there, @howinator! I\'m here to help you out with any bugs, questions, or contributions you have in mind. Let\'s get to work!\\n\\n<!-- Answer -->\\nYour workflow is mostly feasible in Langfuse, and you can achieve it programmatically with some adjustments:\\n\\n1. **Collect Traces**: This step is straightforward and can be done using Langfuse\'s SDKs to capture and store traces.\\n\\n2. **Annotate Traces**: You can programmatically annotate traces with scores using the `langfuse_context.score_current_trace` function. This allows you to attach a boolean score to each trace based on user feedback or other criteria <sup>[[1]](https://langfuse.com/guides/cookbook/integration_langgraph)</sup>.\\n"}, {"type": "text", "text": "\\n3. **Create a \\"False Positive\\" Dataset**: While there isn\'t a direct method to filter dataset items based on scores, you can programmatically create a dataset by iterating over your traces and selecting those with a score of `False`. Use the SDK to create dataset items for these traces <sup>[[2]](https://langfuse.com/docs/datasets/get-started)</sup>.\\n\\n4. **Run an Evaluation**: You can automate the creation of evaluation jobs for your dataset using the `createEvalJobs` function. This function allows you to specify configurations and run evaluations on specific subsets of dataset items <sup>[[3]](https://github.com/langfuse/langfuse/blob/eecf0d680b13a4bde9fe8fed617acc328029307d/worker/src/__tests__/evalService.test.ts)</sup>.\\n\\nFor the evaluation logic, you can implement custom logic to compare the dataset item scores and outputs with the evaluation outputs. This might require some additional scripting to handle the specific logic you described.\\n\\nOverall, your workflow is possible with Langfuse, but it may require some custom scripting to fully automate the process of filtering and evaluating dataset items based on scores.\\n\\n\\n<!-- Dosu Comment Footer -->\\n*To continue talking to [Dosu](https://dosu.dev), mention @dosu.*\\n\\n---"}, {"type": "text", "text": "1. Create a \\"false positive\\" data set composed of all traces where the score was False -> use scores api to fetch all of these\\r\\n2. Run your own eval on them and add it to the same trace via the sdk\\r\\n3. Compare the evals by fetching all of them via the sdk again.\\r\\n\\r\\nWe currently plan to add more configurable dashboards which will allow for the comparison of these evals more easily in the UI.\\n"}], "type": "content"}}, {"type": "document", "record_type": "documentation", "url": "https://langfuse.com/docs/observability/troubleshooting-and-faq", "title": "Troubleshooting and FAQ", "source": {"content": [{"type": "text", "text": "\\n[LLM as a Session-Level Judge with Support for LangGraph Session](https://github.com/orgs/langfuse/discussions/8303 \\"Langfuse Ideas: LLM as a Session-Level Judge with Support for LangGraph Session\\")\\n[Score Configs: Allow editing the categories of a categorical score](https://github.com/orgs/langfuse/discussions/8259 \\"Langfuse Ideas: Score Configs: Allow editing the categories of a categorical score\\")\\n[feat(trace-detail-view): for each span, show the sum of nested observations\' token usage details](https://github.com/orgs/langfuse/discussions/8201 \\"Langfuse Ideas: feat\\\\(trace-detail-view\\\\): for each span, show the sum of nested observations\' token usage details\\")\\n[UI-LLM as a Jury](https://github.com/orgs/langfuse/discussions/8195 \\"Langfuse Ideas: UI-LLM as a Jury\\")\\n[Add Gemini 2.5 Pro model pricing](https://github.com/orgs/langfuse/discussions/8146 \\"Langfuse Ideas: Add Gemini 2.5 Pro model pricing\\")\\n[Deeplinking for Spans inside of a Trace](https://github.com/orgs/langfuse/discussions/8112 \\"Langfuse Ideas: Deeplinking for Spans inside of a Trace\\")\\n[feat(comments): allow attaching images](https://github.com/orgs/langfuse/discussions/8111 \\"Langfuse Ideas: feat\\\\(comments\\\\): allow attaching images\\")\\n[feat(sessions): support session-level tags](https://github.com/orgs/langfuse/discussions/8041 \\"Langfuse Ideas: feat\\\\(sessions\\\\): support session-level tags\\")\\n[Enable Immediate Score Management for User Feedback](https://github.com/orgs/langfuse/discussions/7686 \\"Langfuse Ideas: Enable Immediate Score Management for User Feedback\\")\\n[Allow for structured filtering on inputs / outputs](https://github.com/orgs/langfuse/discussions/7663 \\"Langfuse Ideas: Allow for structured filtering on inputs / outputs\\")"}, {"type": "text", "text": "\\n[Allow for lazy updating of traces in Python SDK (+ possible workaround)](https://github.com/orgs/langfuse/discussions/7655 \\"Langfuse Ideas: Allow for lazy updating of traces in Python SDK \\\\(+ possible workaround\\\\)\\")\\n[Filter by scores in session view](https://github.com/orgs/langfuse/discussions/7528 \\"Langfuse Ideas: Filter by scores in session view\\")\\n[Support break lines on evaluation run tooltip hint](https://github.com/orgs/langfuse/discussions/7452 \\"Langfuse Ideas: Support break lines on evaluation run tooltip hint\\")\\n[Enable sending Langfuse SDK OTEL spans to third party observability backend](https://github.com/orgs/langfuse/discussions/7365 \\"Langfuse Ideas: Enable sending Langfuse SDK OTEL spans to third party observability backend\\")\\n[Add Thinking config in playground for anthropic models](https://github.com/orgs/langfuse/discussions/7267 \\"Langfuse Ideas: Add Thinking config in playground for anthropic models\\")\\n[Better support for big int in metadata.](https://github.com/orgs/langfuse/discussions/7167 \\"Langfuse Ideas: Better support for big int in metadata.\\")\\n[Ability to filter/search comments on traces](https://github.com/orgs/langfuse/discussions/7109 \\"Langfuse Ideas: Ability to filter/search comments on traces\\")\\n[Image / File / Multimedia rendering in dataset items using mediaId](https://github.com/orgs/langfuse/discussions/6883 \\"Langfuse Ideas: Image / File / Multimedia rendering in dataset items using mediaId\\")\\n[Trace Comments support upload image](https://github.com/orgs/langfuse/discussions/6857 \\"Langfuse Ideas: Trace Comments support upload image\\")"}], "type": "content"}}, {"type": "document", "record_type": "github_issue", "url": "https://github.com/langfuse/langfuse/issues/3380", "title": "feat: add fetch_score(s) in the sdks", "source": {"content": [{"type": "text", "text": "Github Issue by @marcklingen created on September 17, 2024. The current state of the issue is: closed\\n\\n### Discussed in https://github.com/orgs/langfuse/discussions/3379\\r\\n\\r\\n<div type=\'discussions-op-text\'>\\r\\n\\r\\n<sup>Originally posted by **Preet1234-ux** September 17, 2024</sup>\\r\\nI want to fetch the score provided by the user from langfuse in my python code . Are there any functionality by which I can fetch the score for a trace ? </div>\\r\\n\\r\\n\\r\\nCurrently, we need to use either langfuse.client.*(potentially breaking) or the api.\\n---\\nComment #1 by @hassiebp:\\n\\nAlready shipped in Python SDK v3\\n\\n"}], "type": "content"}}, {"type": "document", "record_type": "github_discussion", "url": "https://github.com/orgs/langfuse/discussions/6210", "title": "Next.js + Vercel AI SDK: How to get `observationId` and attach it to scores?", "source": {"content": [{"type": "text", "text": "Title: Next.js + Vercel AI SDK: How to get `observationId` and attach it to scores?\\nCategory: Support\\nAuthor: @natterstefan\\nTimestamp: March 26, 2025\\nBody:\\nDear langfuse-Community,\\r\\n\\r\\nI need your help please, as I do not know how to obtain the `observationId` and attach it to a `score`.\\r\\n\\r\\nContext: this is how my `experimental_telemetry` looks like:\\r\\n\\r\\n```js\\r\\n// src: https://langfuse.com/docs/integrations/vercel-ai-sdk#group-multiple-executions-in-one-trace\\r\\n\\r\\nconst parentTraceId = getRandomId()\\r\\n\\r\\nexport async function POST(req: Request) {\\r\\n\\t// ... other code\\r\\n\\r\\n\\tconst trace = langfuse.trace({\\r\\n\\t id: parentTraceId,\\r\\n\\t name: \'cd-chat-messages\',\\r\\n\\t})\\r\\n\\r\\n\\tconsole.log(\'trace\', trace.observationId) // <<-- this returns `null`\\r\\n\\t\\r\\n\\tconst result = streamText({\\r\\n\\t\\t// ....\\r\\n\\t\\texperimental_telemetry: {\\r\\n\\t\\t isEnabled: true,\\r\\n\\t\\t functionId: \'cd-chat-message-\' + messages.length,\\r\\n\\t\\t metadata: {\\r\\n\\t\\t langfuseTraceId: parentTraceId,\\r\\n\\t\\t\\t // Do not update the parent trace with execution results\\r\\n\\t\\t langfuseUpdateParent: false,\\r\\n\\t\\t },\\r\\n\\t\\t },\\r\\n\\t})\\r\\n\\r\\n\\t// ... other code\\r\\n}\\r\\n```\\r\\n\\r\\nThe docs state\\r\\n\\r\\n> The id of the observation to which the score should be attached. Pass from backend trace. E.g. {generation, event, span}.id\\r\\n\\r\\nBut how and where do I get the observationId from, taking into account that I use `streamText` (<https://sdk.vercel.ai/docs/reference/ai-sdk-core/stream-text>)\\r\\n\\r\\n```js\\r\\n// source: https://langfuse.com/docs/sdk/typescript/guide-web#create-scores\\r\\n\\r\\n// pass traceId and observationId to front end\\r\\nawait langfuseWeb.score({\\r\\n traceId: message.traceId,\\r\\n observationId: message.observationId, <<<-- where do I get this?\\r\\n name: \\"user-feedback\\",\\r\\n value: 1,\\r\\n comment: \\"I like how personalized the response is\\",\\r\\n});\\r\\n```\\r\\n\\r"}, {"type": "text", "text": "Another question I could not answer myself is where (= which \\"layer\\". I guess generation, event, span is the correct wording) a score should be attached to. Is there any best practice recommendation or should I always go for the observation (`ai.streamText.doStream` in this example _if_ that\'s the observationId I can obtain)?\\r\\n\\r\\n\\r\\n\\r\\nThank you very much for your time and help!\\nComments:\\n\\n Comment #1 from @dosubot on March 26, 2025:\\n\\n <!-- Greeting -->\\nHey @natterstefan! I\'m here to help you with any bugs, questions, or even becoming a contributor. Let\'s squash those bugs together! π€\\n\\n<!-- Answer -->\\nTo obtain the `observationId` when using the `streamText` function in the Vercel AI SDK, you should look for it in the response from the `streamText` function. The SDK typically uses the ID from the provider response if available, or it generates one if not. Ensure that your telemetry setup is configured to pass the `observationId` from the backend trace, specifically from the context of the `streamText` execution.\\n\\nRegarding where to attach the score, it is generally recommended to attach it to the most relevant observation. In your case, if `ai.streamText.doStream` is the observation you can obtain, it would be appropriate to attach the score there. This aligns with best practices, as it ensures the score is associated with the specific operation or event you are evaluating.\\n"}, {"type": "text", "text": "\\nFor more detailed guidance, you might want to refer to the Langfuse documentation on creating scores and handling observations <sup>[[1]](https://langfuse.com/docs/sdk/typescript)</sup><sup>[[2]](https://langfuse.com/docs/sdk/typescript/guide)</sup>.\\n\\n\\n<!-- Dosu Comment Footer -->\\n*To continue talking to [Dosu](https://dosu.dev), mention @dosu.*\\n\\n---\\n<sup>Help Dosu learn! Give it feedback: [Great Response](https://app.dosu.dev/response-feedback/68b78af0-2c34-4867-b75c-008bcf449b81?feedback_type=great_response) | [Irrelevant Answer](https://app.dosu.dev/response-feedback/68b78af0-2c34-4867-b75c-008bcf449b81?feedback_type=irrelevant_answer) | [Incorrect Sources](https://app.dosu.dev/response-feedback/68b78af0-2c34-4867-b75c-008bcf449b81?feedback_type=incorrect_sources) | [Too Verbose](https://app.dosu.dev/response-feedback/68b78af0-2c34-4867-b75c-008bcf449b81?feedback_type=too_verbose) | [Hallucination](https://app.dosu.dev/response-feedback/68b78af0-2c34-4867-b75c-008bcf449b81?feedback_type=hallucination) | [Bug Report](https://app.dosu.dev/response-feedback/68b78af0-2c34-4867-b75c-008bcf449b81?feedback_type=bug_report) | [Other](https://app.dosu.dev/response-feedback/68b78af0-2c34-4867-b75c-008bcf449b81?feedback_type=other)</sup>\\n\\n\\n Comment #2 from @marcklingen on March 26, 2025:\\n\\n This seems like a feature request for the AI SDK as you\'d need the SDK to expose the span ids of each of the requests.\\n\\n Replies to comment #2:\\n\\n Reply #1 to comment #2 from marcklingen on March 27, 2025:\\n\\n Hi @marcklingen,\\r\\n\\r\\nthanks for your help. Alright, I [moved (aka copied) the discussion to their repo](https://github.com/vercel/ai/discussions/5415) and hope they can help me.\\r\\n\\r\\nIs @dosubot correct with \\"_(...) In your case, if ai.streamText.doStream is the observation you can obtain, it would be appropriate to attach the score there._\\"\\r\\n\\r"}, {"type": "text", "text": "\\n<sup>Help Dosu learn! Give it feedback: [Great Response](https://app.dosu.dev/response-feedback/d9fcd303-8526-48dc-85b4-1432b3d12429?feedback_type=great_response) | [Irrelevant Answer](https://app.dosu.dev/response-feedback/d9fcd303-8526-48dc-85b4-1432b3d12429?feedback_type=irrelevant_answer) | [Incorrect Sources](https://app.dosu.dev/response-feedback/d9fcd303-8526-48dc-85b4-1432b3d12429?feedback_type=incorrect_sources) | [Too Verbose](https://app.dosu.dev/response-feedback/d9fcd303-8526-48dc-85b4-1432b3d12429?feedback_type=too_verbose) | [Hallucination](https://app.dosu.dev/response-feedback/d9fcd303-8526-48dc-85b4-1432b3d12429?feedback_type=hallucination) | [Bug Report](https://app.dosu.dev/response-feedback/d9fcd303-8526-48dc-85b4-1432b3d12429?feedback_type=bug_report) | [Other](https://app.dosu.dev/response-feedback/d9fcd303-8526-48dc-85b4-1432b3d12429?feedback_type=other)</sup>\\n\\n\\n Reply #3 to comment #2 from marcklingen on March 27, 2025:\\n\\n Hi @marcklingen and @dosubot,\\r\\n\\r\\nFollow up question (once I get the `generation.id`/`response.id` whatever vercel/ai calls it): how do I attach it to the current trace to make sure the observation in langfuse has the same id as the score in my code uses?\\n\\n Reply #4 to comment #2 from marcklingen on March 27, 2025:\\n\\n <!-- Answer -->\\nTo attach the `generation.id` or `response.id` from Vercel AI to the current trace in Langfuse, use the `generation.id` as the `observationId` when logging user feedback. This ensures that the observation has the same ID as the score used in your code. Here\'s how you can do it:\\n\\n1. Add the `generation.id` to the streaming response as a custom header:\\n\\n ```javascript\\n return new StreamingTextResponse(stream, {\\n headers: {\\n \\"X-Message-Id\\": lfGeneration.id,"}], "type": "content"}}]}',},- },- {- type: "reasoning",- content: {- kind: "text",- text: "**Creating minimal code examples**\n\nI want to create concise code samples while mentioning that you need a traceId, and optionally an observationId. It's important to highlight that you can send user feedback even before the trace exists, which is quite interesting! This way, the user gets a clear understanding while also seeing how to implement everything in a few lines of code. Let's keep this straightforward and focus on clarity in the examples!",+ ],+ observationId: "3dbf50cbf2036387",+ traceId: "75b336f98e39e275a823b0a7ec1ba794",+ },+ {+ role: "assistant",+ parts: [+ {+ type: "reasoning",+ content: {+ kind: "text",+ text: "**Crafting user score attachment instructions**\n\nI've gathered enough info to help the user with attaching a user score to a trace. They want minimal examples in TypeScript, Python, and a Browser client. I think providing a basic snippet that adds a boolean score named \"user-feedback\" to the traceId would be effective. It's also essential to note that the data type BOOLEAN requires values of 0 or 1. Iβll include concise runnable examples and useful links about scores via API/SDK and user feedback. We should also consider concluding with a follow-up question.",+ },},+ {+ type: "reasoning",+ content: {+ kind: "text",+ text: "**Creating minimal code examples**\n\nI want to create concise code samples while mentioning that you need a traceId, and optionally an observationId. It's important to highlight that you can send user feedback even before the trace exists, which is quite interesting! This way, the user gets a clear understanding while also seeing how to implement everything in a few lines of code. Let's keep this straightforward and focus on clarity in the examples!",+ },+ },+ {+ type: "text",+ text: 'You can add a user score to a trace via the SDKs or API by creating a Score linked to the traceId (optionally to a specific observationId). For thumbs up/down, use dataType "BOOLEAN" with value 1 or 0.\n\nJS/TS (server)\n- Docs: https://langfuse.com/docs/evaluation/evaluation-methods/scores-via-sdk\n- Example:\n - npm i @langfuse/client\n - \n import { LangfuseClient } from "@langfuse/client";\n const langfuse = new LangfuseClient();\n await langfuse.score.create({\n traceId: "<trace_id>",\n name: "user-feedback",\n value: 1, // 1 = up, 0 = down\n dataType: "BOOLEAN", // required for boolean\n comment: "Helpful answer", // optional\n });\n await langfuse.flush();\n\nBrowser (collect in-app feedback)\n- Docs: https://langfuse.com/docs/evaluation/evaluation-methods/scores-via-sdk\n- Example:\n - npm i @langfuse/browser\n - \n import { LangfuseBrowserClient } from "@langfuse/browser";\n const lf = new LangfuseBrowserClient({ publicKey: "<public_key>", baseUrl: "https://cloud.langfuse.com" });\n await lf.score({\n traceId: "<trace_id>",\n // observationId: "<generation_or_span_id>", // optional if you want to score a specific step\n id: `user-feedback-<trace_id>`, // optional idempotency key\n name: "user-feedback",\n value: 1,\n dataType: "BOOLEAN",\n comment: "Great result!",\n });\n\nPython\n- Docs: https://langfuse.com/docs/evaluation/evaluation-methods/scores-via-sdk\n- Example:\n - \n from langfuse import get_client\n lf = get_client()\n lf.create_score(\n name="user-feedback",\n value=1.0, # float\n data_type="BOOLEAN", # required for boolean\n trace_id="<trace_id>",\n # observation_id="<observation_id>", # optional\n comment="User liked it", # optional\n )\n\nNotes\n- You only need traceId to attach to a trace; observationId is optional.\n- You can send the score before the trace exists; it links once the trace is created.\n- Score types supported: NUMERIC, CATEGORICAL, BOOLEAN (0/1), TEXT.\n- More: Scores overview: https://langfuse.com/docs/evaluation/scores/overview\n\nWant an example of passing traceId/observationId from your backend to the frontend for a βthumbs up/downβ widget?',+ },+ ],+ finishReason: {+ type: "stop",+ raw: "stop",},- {- type: "text",- text: 'You can add a user score to a trace via the SDKs or API by creating a Score linked to the traceId (optionally to a specific observationId). For thumbs up/down, use dataType "BOOLEAN" with value 1 or 0.\n\nJS/TS (server)\n- Docs: https://langfuse.com/docs/evaluation/evaluation-methods/scores-via-sdk\n- Example:\n - npm i @langfuse/client\n - \n import { LangfuseClient } from "@langfuse/client";\n const langfuse = new LangfuseClient();\n await langfuse.score.create({\n traceId: "<trace_id>",\n name: "user-feedback",\n value: 1, // 1 = up, 0 = down\n dataType: "BOOLEAN", // required for boolean\n comment: "Helpful answer", // optional\n });\n await langfuse.flush();\n\nBrowser (collect in-app feedback)\n- Docs: https://langfuse.com/docs/evaluation/evaluation-methods/scores-via-sdk\n- Example:\n - npm i @langfuse/browser\n - \n import { LangfuseBrowserClient } from "@langfuse/browser";\n const lf = new LangfuseBrowserClient({ publicKey: "<public_key>", baseUrl: "https://cloud.langfuse.com" });\n await lf.score({\n traceId: "<trace_id>",\n // observationId: "<generation_or_span_id>", // optional if you want to score a specific step\n id: `user-feedback-<trace_id>`, // optional idempotency key\n name: "user-feedback",\n value: 1,\n dataType: "BOOLEAN",\n comment: "Great result!",\n });\n\nPython\n- Docs: https://langfuse.com/docs/evaluation/evaluation-methods/scores-via-sdk\n- Example:\n - \n from langfuse import get_client\n lf = get_client()\n lf.create_score(\n name="user-feedback",\n value=1.0, # float\n data_type="BOOLEAN", # required for boolean\n trace_id="<trace_id>",\n # observation_id="<observation_id>", # optional\n comment="User liked it", # optional\n )\n\nNotes\n- You only need traceId to attach to a trace; observationId is optional.\n- You can send the score before the trace exists; it links once the trace is created.\n- Score types supported: NUMERIC, CATEGORICAL, BOOLEAN (0/1), TEXT.\n- More: Scores overview: https://langfuse.com/docs/evaluation/scores/overview\n\nWant an example of passing traceId/observationId from your backend to the frontend for a βthumbs up/downβ widget?',- },- ],- finishReason: {- type: "stop",- raw: "stop",+ source: "output",+ observationId: "0c13b686b4cf6ccc",+ traceId: "75b336f98e39e275a823b0a7ec1ba794",},- source: "output",- observationId: "0c13b686b4cf6ccc",- traceId: "75b336f98e39e275a823b0a7ec1ba794",- },- ],- observations: [- {- id: "1bbf3edb8ef95ab6",- traceId: "75b336f98e39e275a823b0a7ec1ba794",- },- {- id: "3dbf50cbf2036387",- traceId: "75b336f98e39e275a823b0a7ec1ba794",- },- {- id: "0c13b686b4cf6ccc",- traceId: "75b336f98e39e275a823b0a7ec1ba794",- },- ],+ ],+ observations: [+ {+ id: "1bbf3edb8ef95ab6",+ traceId: "75b336f98e39e275a823b0a7ec1ba794",+ },+ {+ id: "3dbf50cbf2036387",+ traceId: "75b336f98e39e275a823b0a7ec1ba794",+ },+ {+ id: "0c13b686b4cf6ccc",+ traceId: "75b336f98e39e275a823b0a7ec1ba794",+ },+ ],+ },},],},diff --git a/packages/shared/src/server/transcript/index.ts b/packages/shared/src/server/transcript/index.tsindex 3c1379722c..29831c68f5 100644--- a/packages/shared/src/server/transcript/index.ts+++ b/packages/shared/src/server/transcript/index.ts@@ -1,73 +1,3 @@-import { partition } from "lodash";-import type { Observation } from "../../domain";-import { normalizeIO } from "../normalized-io";-import type { Transcript } from "./types";-import {- append,- findThread,- messageKey,- type ThreadState,- type TranscriptObservation,-} from "./threads";-import { createToolCallRegistry } from "./tool-calls";+export { orderObservations } from "./ordering";+export { assembleTranscript } from "./transcript";export type * from "./types";--/** Check if an observation is relevant for the transcript. */-const isRelevantObservation = (- observation: Observation,-): observation is TranscriptObservation =>- (observation.type === "GENERATION" || observation.type === "TOOL") &&- observation.traceId !== null;--/** Normalize original I/O without interpreting provider-specific envelopes. */-function normalize(observation: TranscriptObservation) {- const isTool = observation.type === "TOOL";- const { messages } = normalizeIO({- kind: "io",- io: {- // Tool inputs are supplied by the generation's tool-call message.- input: isTool ? undefined : observation.input,- output: observation.output,- metadata: observation.metadata,- },- });- return partition(- messages.map((message) => ({ message, key: messageKey(message) })),- ({ message }) => message.source === "input",- );-}--/** Normalize observations, reconcile history, and retain first-seen provenance. */-export function getTranscript(observations: Observation[]): Transcript | null {- const ordered = observations- .filter(isRelevantObservation)- .sort((a, b) => a.startTime.getTime() - b.startTime.getTime());-- const states: ThreadState[] = [];- const toolCalls = createToolCallRegistry();-- for (const observation of ordered) {- const [input, output] = normalize(observation);- if (observation.type === "TOOL") {- toolCalls.attachToolOutput(observation, output);- continue;- }- if (input.length === 0 && output.length === 0) continue;-- // Registered responses do not participate in thread selection.- let state = findThread(states, input);- const isNewThread = !state;- if (!state) {- // Open new thread- state = {- thread: { messages: [], observations: [] },- messages: [],- shownCounts: new Map(),- };- states.push(state);- }- append(state, observation, input, output, isNewThread, toolCalls);- }-- return states.length ? { threads: states.map(({ thread }) => thread) } : null;-}diff --git a/packages/shared/src/server/transcript/ordering.test.ts b/packages/shared/src/server/transcript/ordering.test.tsnew file mode 100644index 0000000000..1f7b045dd1--- /dev/null+++ b/packages/shared/src/server/transcript/ordering.test.ts@@ -0,0 +1,41 @@+import { describe, expect, it } from "vitest";+import type { Observation } from "../../domain";+import { orderObservations } from "./ordering";++describe("orderObservations", () => {+ const observation = (+ id: string,+ startSecond: number,+ parentObservationId: string | null = null,+ ) =>+ ({+ id,+ parentObservationId,+ startTime: new Date(Date.UTC(2026, 0, 1, 12, 0, startSecond)),+ }) as Observation;++ it("walks the tree depth first with roots and siblings by start time", () => {+ // Span A starts first, but its generation starts after span B's.+ const ordered = orderObservations([+ observation("g2", 2, "B"),+ observation("B", 1),+ observation("g1", 5, "A"),+ observation("A", 0),+ ]);+ expect(ordered.map(({ id }) => id)).toEqual(["A", "g1", "B", "g2"]);+ });++ it("keeps the earliest row per id and roots rows without a known parent", () => {+ const ordered = orderObservations([+ observation("x", 3),+ observation("x", 1),+ observation("orphan", 2, "missing"),+ ]);+ expect(+ ordered.map(({ id, startTime }) => [id, startTime.getUTCSeconds()]),+ ).toEqual([+ ["x", 1],+ ["orphan", 2],+ ]);+ });+});diff --git a/packages/shared/src/server/transcript/ordering.ts b/packages/shared/src/server/transcript/ordering.tsnew file mode 100644index 0000000000..509fb0d2be--- /dev/null+++ b/packages/shared/src/server/transcript/ordering.ts@@ -0,0 +1,48 @@+import type { Observation } from "../../domain";++/**+ * Order observations the way the trace tree walks them: depth first, with+ * roots and siblings by start time. Mirrors the web tree builder's rules: one+ * row per id where the earliest start wins, and a row whose parent is not in+ * the list becomes a root. For a flat trace this is plain start-time order.+ */+export function orderObservations(observations: Observation[]): Observation[] {+ const byId = new Map<string, Observation>();+ for (const observation of observations) {+ const existing = byId.get(observation.id);+ if (!existing || observation.startTime < existing.startTime) {+ byId.set(observation.id, observation);+ }+ }++ const children = new Map<string | null, Observation[]>();+ for (const observation of byId.values()) {+ const parentId =+ observation.parentObservationId !== null &&+ byId.has(observation.parentObservationId)+ ? observation.parentObservationId+ : null;+ children.set(parentId, [...(children.get(parentId) ?? []), observation]);+ }++ const byStartTime = (a: Observation, b: Observation) =>+ a.startTime.getTime() - b.startTime.getTime();+ const ordered: Observation[] = [];+ const visited = new Set<string>();+ const walk = (parentId: string | null) => {+ for (const observation of (children.get(parentId) ?? []).sort(+ byStartTime,+ )) {+ if (visited.has(observation.id)) continue;+ visited.add(observation.id);+ ordered.push(observation);+ walk(observation.id);+ }+ };+ walk(null);+ // Rows inside a parent cycle are unreachable from any root; keep them.+ for (const observation of [...byId.values()].sort(byStartTime)) {+ if (!visited.has(observation.id)) ordered.push(observation);+ }+ return ordered;+}diff --git a/packages/shared/src/server/transcript/threads.ts b/packages/shared/src/server/transcript/threads.tsindex 7ea2398467..296ec0a705 100644--- a/packages/shared/src/server/transcript/threads.ts+++ b/packages/shared/src/server/transcript/threads.ts@@ -1,6 +1,6 @@import type { Observation } from "../../domain";import type { NormalizedMessage } from "../../utils/normalized-io";-import type { Thread, ThreadMessage } from "./types";+import type { Thread, ThreadMessage, Turn } from "./types";import type { createToolCallRegistry } from "./tool-calls";export type TranscriptObservation = Observation & { traceId: string };@@ -10,7 +10,8 @@ export type KeyedMessage = {key: string;};export type ThreadState = {- thread: Thread;+ /** Every message of the thread in one list; `splitTurn` derives the public shape. */+ thread: Turn;messages: KeyedMessage[];shownCounts: Map<string, number>;};@@ -112,7 +113,7 @@ export function append(}export function addContributor(- thread: Thread,+ thread: Turn,observation: TranscriptObservation,) {if (@@ -127,3 +128,39 @@ export function addContributor(});}}++/** Split a thread into replayed history and the turn the last trace added. */+export function splitTurn(thread: Turn): Thread {+ const { messages, observations } = thread;+ // Earlier traces of a session are history; the last contributing trace is+ // the current turn.+ const currentTraceId = observations.at(-1)?.traceId;+ const traceStart =+ messages.findLastIndex(({ traceId }) => traceId !== currentTraceId) + 1;+ // Within that trace, input replayed before the first output is history up+ // to the previous turn's last assistant or tool message. What follows is new.+ const trace = messages.slice(traceStart);+ const firstOutput = trace.findIndex(({ source }) => source === "output");+ const replayed = trace.slice(0, firstOutput === -1 ? undefined : firstOutput);+ const turnStart =+ traceStart ++ replayed.findLastIndex(+ ({ role }) => role === "assistant" || role === "tool",+ ) ++ 1;+ const current = messages.slice(turnStart);+ return {+ conversationHistory: messages+ .slice(0, turnStart)+ .map(({ observationId, traceId, ...message }) => message),+ currentTurn: {+ messages: current,+ observations: observations.filter(({ id, traceId }) =>+ current.some(+ (message) =>+ message.observationId === id && message.traceId === traceId,+ ),+ ),+ },+ };+}diff --git a/packages/shared/src/server/transcript/tool-calls.ts b/packages/shared/src/server/transcript/tool-calls.tsindex dbef39061c..3a36e00f05 100644--- a/packages/shared/src/server/transcript/tool-calls.ts+++ b/packages/shared/src/server/transcript/tool-calls.ts@@ -2,7 +2,7 @@ import type {NormalizedMessage,ToolCallPart,} from "../../utils/normalized-io";-import type { Thread, ThreadMessage } from "./types";+import type { Turn, ThreadMessage } from "./types";import {addContributor,type KeyedMessage,@@ -10,7 +10,7 @@ import {} from "./threads";type Call = {- thread: Thread;+ thread: Turn;message: ThreadMessage;response?: ThreadMessage;fromTool?: boolean;@@ -19,14 +19,14 @@ const key = (traceId: string, id: string) => JSON.stringify([traceId, id]);export function createToolCallRegistry() {const calls = new Map<string, Call>();- const callsByThread = new WeakMap<Thread, Map<string, Call[]>>();+ const callsByThread = new WeakMap<Turn, Map<string, Call[]>>();const pending = new Map<string, { calls: Call[]; next: number }>();const responseTails = new WeakMap<ThreadMessage, ThreadMessage>();function register(observation: TranscriptObservation,part: ToolCallPart,- thread: Thread,+ thread: Turn,message: ThreadMessage,) {const id = part.toolCallId@@ -80,7 +80,7 @@ export function createToolCallRegistry() {function consumePart(observation: TranscriptObservation,part: NormalizedMessage["parts"][number],- thread: Thread,+ thread: Turn,message: ThreadMessage,replayCalls: Map<string, number>,replayTotals: Map<string, number>,diff --git a/packages/shared/src/server/transcript/transcript.ts b/packages/shared/src/server/transcript/transcript.tsnew file mode 100644index 0000000000..2e4a876f0d--- /dev/null+++ b/packages/shared/src/server/transcript/transcript.ts@@ -0,0 +1,78 @@+import { partition } from "lodash";+import type { Observation } from "../../domain";+import { normalizeIO } from "../normalized-io";+import type { Transcript } from "./types";+import {+ append,+ findThread,+ messageKey,+ splitTurn,+ type ThreadState,+ type TranscriptObservation,+} from "./threads";+import { createToolCallRegistry } from "./tool-calls";++/** Check if an observation is relevant for the transcript. */+const isRelevantObservation = (+ observation: Observation,+): observation is TranscriptObservation =>+ (observation.type === "GENERATION" || observation.type === "TOOL") &&+ observation.traceId !== null;++/** Normalize original I/O without interpreting provider-specific envelopes. */+function normalize(observation: TranscriptObservation) {+ const isTool = observation.type === "TOOL";+ const { messages } = normalizeIO({+ kind: "io",+ io: {+ // Tool inputs are supplied by the generation's tool-call message.+ input: isTool ? undefined : observation.input,+ output: observation.output,+ metadata: observation.metadata,+ },+ });+ return partition(+ messages.map((message) => ({ message, key: messageKey(message) })),+ ({ message }) => message.source === "input",+ );+}++/**+ * Assemble threads from observations: normalize their I/O, reconcile replayed+ * history, and retain first-seen provenance. The caller supplies the+ * observations in transcript order, see `orderObservations`; they are consumed+ * as given.+ */+export function assembleTranscript(+ orderedObservations: Observation[],+): Transcript | null {+ const states: ThreadState[] = [];+ const toolCalls = createToolCallRegistry();++ for (const observation of orderedObservations.filter(isRelevantObservation)) {+ const [input, output] = normalize(observation);+ if (observation.type === "TOOL") {+ toolCalls.attachToolOutput(observation, output);+ continue;+ }+ if (input.length === 0 && output.length === 0) continue;++ // Registered responses do not participate in thread selection.+ let state = findThread(states, input);+ const isNewThread = !state;+ if (!state) {+ // Open new thread+ state = {+ thread: { messages: [], observations: [] },+ messages: [],+ shownCounts: new Map(),+ };+ states.push(state);+ }+ append(state, observation, input, output, isNewThread, toolCalls);+ }++ return states.length+ ? { threads: states.map(({ thread }) => splitTurn(thread)) }+ : null;+}diff --git a/packages/shared/src/server/transcript/types.ts b/packages/shared/src/server/transcript/types.tsindex ea1cc59d14..0859148689 100644--- a/packages/shared/src/server/transcript/types.ts+++ b/packages/shared/src/server/transcript/types.ts@@ -8,12 +8,20 @@ export type ThreadMessage = NormalizedMessage & {traceId: string;};-export type Thread = {+/** Messages of one turn with the observations that emitted them. */+export type Turn = {messages: ThreadMessage[];- /** Observations that contributed to this thread, in contribution order. */+ /** Observations that contributed to this turn, in contribution order. */observations: { id: string; traceId: string }[];};+export type Thread = {+ /** Messages replayed from earlier turns, in replay order, without provenance. */+ conversationHistory: NormalizedMessage[];+ /** Messages the supplied observations added to the conversation. */+ currentTurn: Turn;+};+export type Transcript = {threads: Thread[];};