Skip to content

πŸͺ’ Open source agent evals & observability: Trace, evaluate, and improve LLM applications with one open platform.

feat(transcript): implement transcript generation from observations (#17371)

marliessophie555d43d05ebb
diff --git a/packages/shared/src/server/index.ts b/packages/shared/src/server/index.ts
index adc85bfb14..80a8849e89 100644
--- a/packages/shared/src/server/index.ts
+++ b/packages/shared/src/server/index.ts
@@ -88,6 +88,7 @@ export * from "./ingestion/processEventBatch";
export * from "../server/ingestion/validateAndInflateScore";
export * from "./ingestion/extractToolsBackend";
export * from "./normalized-io";
+export * from "./transcript";
export * from "../server/queries/public-api-filter-builder";
export * from "../server/pricing-tiers";
export * from "./redis/redis";
diff --git a/packages/shared/src/server/transcript/README.md b/packages/shared/src/server/transcript/README.md
new file mode 100644
index 0000000000..90ce3d1ddc
--- /dev/null
+++ b/packages/shared/src/server/transcript/README.md
@@ -0,0 +1,171 @@
+# Transcript
+
+PLEASE DO NOT USE IN PRODUCTION YET. This is a v1 implementation of the transcript builder and remains work in progress.
+
+Builds a conversation transcript from the observations of one trace, or of
+every trace in one session.
+
+Status: generation-led builder with tool responses matched by ID or name and order.
+
+- Tool outputs contribute only when matched to a preceding generation output's tool call in the same trace.
+- Root span I/O and unmatched tool observations are excluded.
+
+## Interface
+
+```ts
+getTranscript(
+ observations: Observation[]
+): Transcript | null;
+
+type Transcript = { threads: Thread[] };
+
+type Thread = {
+ messages: ThreadMessage[];
+ observations: { id: string; traceId: string }[]; // observations that contributed, in order
+};
+
+type ThreadMessage = NormalizedMessage & {
+ observationId: string; // observation that first emitted the message
+ traceId: string;
+};
+```
+
+The input is the domain `Observation` (see `domain/observations.ts`), which
+the repositories produce from ClickHouse rows. Returns `null` when no eligible
+generations produce messages.
+
+A **transcript** contains the conversation threads inferred from the supplied
+observations. A **thread** is a sequence of messages connected by shared input
+history; it can span multiple traces. The consumer handles multiple threads
+and decides which, if any, is the main conversation.
+
+## Which observations contribute?
+
+- `GENERATION` observations with a non-null trace ID establish threads.
+- Include `TOOL` observations in the supplied observations to recover responses.
+ Match normalized tool-result IDs first. Without an ID, match the observation's
+ exact name to the earliest preceding call not yet claimed by a tool observation
+ with that tool name in the same trace. No parent constraint or fuzzy matching.
+ Unknown explicit IDs do not fall back to names; unmatched tools are skipped.
+ Preserve all normalized output parts; tool inputs are ignored. Provider-specific
+ payload interpretation belongs to normalized IO, not the transcript builder.
+- The caller supplies observations from one trace or session. The builder
+ orders generations and tools by start time across the supplied traces.
+- Each observation is normalized once in this chronological pass. Generations
+ establish threads and register output tool-call IDs; tools enrich registered
+ calls. Input messages are processed before output messages for generations.
+- Generations producing no messages are skipped and do not create empty threads.
+- Each registered tool call has at most one tool result response. A tool observation with a
+ response adds or replaces it immediately after the originating call message,
+ referencing the tool's `observationId` and
+ `traceId`. Later generation replay cannot overwrite it or add another copy.
+ If multiple tool observations respond to one call, the first response wins.
+ Tool inputs do not contribute. Other observation types are ignored.
+
+The main loop has two paths: generations select a thread, append deduplicated
+input and append output; tool observations enrich an existing call. Call
+registration and replayed-result suppression happen inside message appending.
+
+## How does deduplication work?
+
+### 1. Select a thread
+
+Continue a thread when it has at least one non-system message and all its
+non-system generation messages appear in the incoming input, regardless of order.
+Supplemental tool responses are not required for this match.
+Otherwise create a new thread. Matching checks presence, not occurrence counts.
+When several threads match, the most recently created matching thread wins.
+
+### 2. Append messages
+
+A message is identified by stable JSON of **role + parts**. Object-property
+order is ignored; array order matters. `senderName`, `source`, `finishReason`
+and observation provenance are excluded. All fields inside parts are included.
+
+- **Inputs:** append only occurrences beyond the number already shown in the
+ thread, so replayed history disappears but additional identical copies survive (eg user responds "Thank you" twice).
+- **Outputs:** always append, then count them so subsequent inputs do not repeat them.
+- **Registered tool responses:** use call identity, not message occurrence counts.
+ Tool-observation responses take precedence over generation-provided responses.
+ These responses are excluded from thread matching, irrespective of their source.
+- **References:** each message has a reference to the trace and generation ID of the object that emitted it. Replayed inputs never overwrite references on earlier occurrences.
+
+### System messages
+
+- Exclude system messages when deciding which thread matches. Compare the
+ remaining conversation messages.
+- Once the thread is selected, include new system messages there. Keep their
+ first-seen generation and trace IDs.
+- Apply the same occurrence-count deduplication to system messages within that thread.
+- Preserve changed system messages as separate entries, rather than overwriting
+ earlier instructions or combining their contents.
+
+## Supported cases
+
+- Cumulative history across generations and traces, including previous outputs
+ replayed as inputs.
+- Reordered history: an existing `[A, B, C]` matched against input
+ `[B, C, A, New]` adds only `New`. Existing display order stays `[A, B, C]`.
+- Additional identical input occurrences and always-visible outputs.
+- New system instructions within a continuing conversation.
+- Multiple threads when their histories distinguish them.
+
+## Current limitations
+
+- **Compaction or missing history:** can split a conversation into multiple threads.
+- **Embedded history:** a conversation inside one string or object is not matched
+ against separate messages.
+- **Identical conversations:** unrelated conversations with matching history can join.
+- **Whole-message matching:** equivalent content split into different messages
+ or parts may not match. Reordering parts within a message also changes identity.
+ Registered tool responses are the exception: replay is matched by call ID
+ within the same trace, even when grouped with other parts.
+- **Name matching is best-effort:** same-name parallel executions can start in a
+ different order from their calls. Names must match exactly. Metadata-only IDs
+ are not read by the transcript builder; they require support in normalized IO.
+- **Performance:** replayed history is normalized and serialized for each
+ generation.
+
+## Preliminary decisions
+
+- Tool observations contribute to thread messages rather than just enriching generation messages with tool responses. In some cases, the generations messages do not contain any reference of the tool call or the tool result, but tool-observations can provide this information. Recommendation: let tool observations only enrich generation messages with tool responses. Sampled production data does not show strong enough evidence to support this change. Should we find more evidence in production data, this decision should be revisited.
+- Root span I/O does not contribute to the transcript, unless it is of type `GENERATION`.
+- Do not include status messages and errors in the transcript for v1. Only revisit should we find strong evidence in production data that this is a valuable feature, or if consumers (e.g. Topics, Session UI) require this information.
+- Expose a helper method to get the first user message and final assistant message from a given thread. This is useful for consumers (e.g. Topics, Session UI) to display the user question and final assistant answer. Consumers must assess for which thread they want to display this information, and how to handle multiple threads.
+- Differences in part-level `providerMetadata` prevents deduplication when the visible message content is otherwise identical. Should production data show strong enough evidence to support this change, this decision should be revisited.
+
+## Open questions
+
+- How should compacted histories and branches reconnect to existing threads? Will venture to find solid examples in production data to guide this decision.
+
+## Layout
+
+```
+transcript/
+β”œβ”€β”€ README.md
+β”œβ”€β”€ index.ts getTranscript
+β”œβ”€β”€ threads.ts thread selection and message deduplication
+β”œβ”€β”€ tool-calls.ts tool matching and response association
+β”œβ”€β”€ types.ts Transcript, Thread, ThreadMessage
+└── fixtures/
+ β”œβ”€β”€ README.md how to turn a trace JSON export into a fixture
+ β”œβ”€β”€ fixture-types.ts TranscriptFixture
+ β”œβ”€β”€ format-transcript.ts chat-shaped printout used by the test
+ β”œβ”€β”€ index.ts registry: trace-scoped and session-scoped fixtures
+ β”œβ”€β”€ fixtures.test.ts structural checks and behavior assertion per fixture
+ β”œβ”€β”€ trace/ one file per trace-scoped fixture
+ └── session/ one file per session-scoped fixture
+```
+
+## Fixtures and verification
+
+Fixtures contain observation trees and an optional expected transcript.
+**Fixtures with `expected: undefined` exercise parsing and structural checks,
+but do not verify transcript correctness.** Their printed output is for manual
+review; expectations are authored by hand once the desired behavior is decided.
+
+Run with console output enabled to see it:
+
+```bash
+pnpm --filter @langfuse/shared run test src/server/transcript --disableConsoleIntercept
+```
diff --git a/packages/shared/src/server/transcript/fixtures/README.md b/packages/shared/src/server/transcript/fixtures/README.md
new file mode 100644
index 0000000000..6a42eff5c9
--- /dev/null
+++ b/packages/shared/src/server/transcript/fixtures/README.md
@@ -0,0 +1,221 @@
+# Building a transcript fixture from a trace JSON export
+
+This guide is written so an agent can follow it end to end. Hand it a trace
+export and this file, and it should return one fixture file plus a registry
+edit, with the expectation left open.
+
+## Input
+
+A trace JSON export, downloaded from the trace page (`β‹―` menu β†’ Download
+JSON). The file is named `trace-<traceId>.json` and has this shape:
+
+```json
+{
+ "observations": [
+ {
+ "id": "…",
+ "traceId": "…",
+ "sessionId": "…",
+ "parentObservationId": "…",
+ "type": "GENERATION",
+ "name": "…",
+ "startTime": "2026-08-31T12:33:14.205Z",
+ "endTime": "2026-08-31T12:33:14.818Z",
+ "level": "DEFAULT",
+ "statusMessage": "",
+ "model": "gpt-5.4",
+ "input": "{\"messages\":[…]}",
+ "output": "{\"content\":null,\"tool_calls\":[…]}",
+ "metadata": { "…": "…" },
+ "…": "many more fields the fixture does not need"
+ }
+ ],
+ "scores": []
+}
+```
+
+For a session-scoped fixture, export every trace of the session separately
+and concatenate their `observations` arrays. The session page download does
+not include observations.
+
+Along with the file, the requester may say which observations to drop, which
+payloads to shorten, and what the case is meant to exercise. Without such
+notes, keep everything.
+
+## Output
+
+One file `fixtures/trace/<kebab-case-name>.ts` (or `fixtures/session/…`)
+exporting a single `<camelCaseName>Fixture` that `satisfies TranscriptFixture`,
+plus a one-line registration in `fixtures/index.ts`. Use
+`trace/support-copilot-refund-loop.ts` as the reference for tone and layout.
+
+## Procedure
+
+1. **Draw the tree.** Build the parent β†’ child tree from `parentObservationId`
+ and write it as the file's leading doc comment in this style:
+
+ ```
+ [SPAN] name id root
+ [AGENT] name id
+ [GENERATION] name model id in: <message roles> out: <what came back>
+ [TOOL] name id in: <relation to a call> out: <summary>
+ ```
+
+ Annotate each `GENERATION` with the roles in its input and a short summary
+ of its output. Annotate each `TOOL` with how its input relates to a
+ tool call nearby. Note when the export order differs from tree order.
+
+2. **Keep every observation.** Types the builder ignores (`SPAN`,
+ `GUARDRAIL`, `EVENT`, …) stay in the fixture. Filtering is the builder's
+ job and must be tested.
+
+3. **Keep the export order.** Do not sort by time or by tree position. The
+ builder must not rely on input order.
+
+4. **Map fields.** Copy these and drop everything else:
+
+ | Export field | Fixture field | Rule |
+ | --------------------- | --------------------- | --------------------------------------------- |
+ | `id` | `id` | verbatim |
+ | `traceId` | `traceId` | verbatim; hoist into a const |
+ | `sessionId` | `sessionId` | verbatim when present; hoist into a const |
+ | `parentObservationId` | `parentObservationId` | `""` becomes `null` |
+ | `type` | `type` | verbatim |
+ | `name` | `name` | verbatim |
+ | `startTime` | `startTime` | verbatim ISO string |
+ | `endTime` | `endTime` | verbatim ISO string or `null` |
+ | `level` | `level` | only when not `DEFAULT` |
+ | `statusMessage` | `statusMessage` | only when non-empty |
+ | `model` | `model` | only when non-empty |
+ | `input` | `input` | see encoding rules |
+ | `output` | `output` | see encoding rules |
+ | `metadata` | `metadata` | verbatim object; hoist and spread when shared |
+
+5. **Encode I/O exactly as stored.** The export carries `input` and `output`
+ as one JSON-encoded string each. In the fixture, write the decoded value as
+ an object literal wrapped in `JSON.stringify(...)`. This keeps the stored
+ encoding while making the content readable:
+
+ ```ts
+ input: JSON.stringify({
+ messages: [{ role: "user", content: customerMessage }],
+ }),
+ ```
+
+ Strings nested inside that value stay strings. Tool-call `arguments` and
+ tool-message `content` are often JSON text themselves. Write them as
+ `JSON.stringify(...)` inline, never as decoded objects:
+
+ ```ts
+ function: {
+ name: "stripe_find_charges",
+ arguments: JSON.stringify(findChargesArguments),
+ },
+ ```
+
+ If a stored value is plain text rather than JSON, keep it as a string
+ literal. If it is `null`, keep `null`.
+
+6. **Hoist repeated values.** System prompts, user messages, tool-call ids,
+ tool arguments, and the final answer usually appear in several
+ observations. Give each a `const` and reuse it, so byte-equality across
+ observations is visible in code. When a `TOOL` observation's input equals
+ a tool call's arguments, use the same const for both.
+
+7. **Shorten with care.** Long payloads (retrieved documents, big system
+ prompts, base64) may be replaced by a short bracketed placeholder such as
+ `"[reduced: 78 KB of retrieved docs]"`, with these constraints:
+ - Transcript deduplication relies on message equality. A value that appears
+ in more than one observation must be shortened identically everywhere,
+ or not at all.
+ - Substring relations matter too. If one message quotes another (a user
+ turn that says "Rate this joke: <joke>"), shorten the quoted text and the
+ quoting text consistently, or leave both intact.
+ - Never change roles, part types, ids, ordering, or the number of
+ messages.
+
+8. **Redact consistently.** Replace personal data (emails, real names, keys)
+ with stable stand-ins, applied to every occurrence. Seeded demo data needs
+ no redaction.
+
+9. **Leave the expectation open.** `expected: undefined`, always. The
+ requester defines the expected transcript by hand.
+
+10. **Write the description.** Two to five sentences on what makes this tree
+ interesting for transcript semantics: history replay or its absence,
+ where tool results live, disjoint generations, repeated system prompts,
+ non-message I/O, unusual nesting, export order quirks.
+
+11. **Register and verify.** Add the export to `traceTranscriptFixtures` or
+ `sessionTranscriptFixtures` in `fixtures/index.ts`, then run:
+
+ ```bash
+ pnpm --filter @langfuse/shared run test src/utils/transcript/fixtures/fixtures.test.ts
+ ```
+
+ ```bash
+ pnpm --filter @langfuse/shared run typecheck
+ ```
+
+## Template
+
+```ts
+import type { TranscriptObservation } from "../../types";
+import type { TranscriptFixture } from "../fixture-types";
+
+/**
+ * <One line: what the application does in this trace. Where it came from.>
+ *
+ * <Tree, as described in step 1.>
+ */
+
+const traceId = "…";
+const sessionId = "…";
+
+// Hoisted repeated values (step 6).
+
+const metadata = {
+ /* … */
+};
+
+const common = { traceId, sessionId, metadata };
+
+const observations: TranscriptObservation[] = [
+ {
+ ...common,
+ id: "…",
+ parentObservationId: null,
+ type: "SPAN",
+ name: "…",
+ startTime: "…",
+ endTime: "…",
+ input: JSON.stringify({
+ /* … */
+ }),
+ output: JSON.stringify({
+ /* … */
+ }),
+ },
+ // …
+];
+
+export const <camelCaseName>Fixture = {
+ name: "<tree shape in a few words>",
+ scope: "trace",
+ description: "<step 10>",
+ observations,
+ expected: undefined,
+} satisfies TranscriptFixture;
+```
+
+## Checklist before handing back
+
+- [ ] Tree comment present and consistent with `parentObservationId`.
+- [ ] Every observation from the export is present, in export order.
+- [ ] `""` parent ids became `null`.
+- [ ] Every `input` / `output` that was a JSON string is `JSON.stringify(...)`
+ of a literal; nested JSON strings are still strings.
+- [ ] Repeated values are hoisted; shortened values are shortened everywhere.
+- [ ] `level`, `statusMessage`, `model` only where the rules say.
+- [ ] `expected: undefined`.
+- [ ] Registered in `fixtures/index.ts`; integrity test and typecheck pass.
diff --git a/packages/shared/src/server/transcript/fixtures/fixture-types.ts b/packages/shared/src/server/transcript/fixtures/fixture-types.ts
new file mode 100644
index 0000000000..eaf264f3f3
--- /dev/null
+++ b/packages/shared/src/server/transcript/fixtures/fixture-types.ts
@@ -0,0 +1,29 @@
+import { createObservation } from "../../test-utils";
+import type { Transcript } from "../types";
+
+/**
+ * One transcript test case: a full observation tree as `createObservation`
+ * seed records, plus the transcript a human expects the builder to return.
+ * The behavior test completes each seed with `createObservation` and converts
+ * it to a domain `Observation` with `convertObservation` before building.
+ *
+ * `expected` is authored by hand and stays `undefined` until the semantics of
+ * the case are decided. Fixtures without an expectation are still valuable:
+ * they document real trees, run through the integrity checks, and turn into
+ * assertions the moment the expectation is filled in.
+ */
+export type TranscriptFixture = {
+ /** Test name. Describes the tree shape the case exercises. */
+ name: string;
+ /** `trace`: one trace. `session`: every trace of one session. */
+ scope: "trace" | "session";
+ /** Why this tree is interesting for transcript semantics. */
+ description: string;
+ /**
+ * Every observation of the trace(s), including types the builder ignores,
+ * in the order the export lists them. The builder must not rely on input
+ * order, so fixtures never re-sort.
+ */
+ observations: Parameters<typeof createObservation>[0][];
+ expected: Transcript | null | undefined;
+};
diff --git a/packages/shared/src/server/transcript/fixtures/fixtures.test.ts b/packages/shared/src/server/transcript/fixtures/fixtures.test.ts
new file mode 100644
index 0000000000..06c151c762
--- /dev/null
+++ b/packages/shared/src/server/transcript/fixtures/fixtures.test.ts
@@ -0,0 +1,118 @@
+import { describe, expect, it } from "vitest";
+import { transcriptFixtures } from "./index";
+import { createObservation } from "../../test-utils";
+import { convertObservation } from "../../repositories/observations_converters";
+import { getTranscript } from "../index";
+import { formatTranscript } from "./format-transcript";
+
+/**
+ * Structural checks on the fixtures themselves, plus the behavior assertion
+ * against the transcript builder. Every fixture's transcript is printed
+ * (visible with `--disableConsoleIntercept`) so expectations can be authored
+ * from real output; the assertion only runs once `expected` is defined.
+ */
+describe("transcript fixtures", () => {
+ const generation = (id: string, input: string[], output: string[]) =>
+ convertObservation(
+ createObservation({
+ id,
+ trace_id: `trace-${id}`,
+ type: "GENERATION",
+ start_time: `2026-01-01T12:00:0${id}.000Z`,
+ input: JSON.stringify(
+ input.map((content) => ({ role: "user", content })),
+ ),
+ output: JSON.stringify(
+ output.map((content) => ({ role: "user", content })),
+ ),
+ }),
+ );
+
+ it("skips generations without messages", () => {
+ const empty = generation("1", [], []);
+ expect(getTranscript([empty])).toBeNull();
+ const transcript = getTranscript([empty, generation("2", ["A"], [])]);
+ expect(transcript?.threads).toHaveLength(1);
+ expect(transcript?.threads[0].observations).toEqual([
+ { id: "2", traceId: "trace-2" },
+ ]);
+ });
+
+ it("continues the newest matching thread without duplicating history", () => {
+ const transcript = getTranscript([
+ generation("1", ["A"], []),
+ generation("2", ["B"], []),
+ generation("3", ["B", "A"], ["C"]),
+ ]);
+ expect(transcript?.threads.map((thread) => thread.observations)).toEqual([
+ [{ id: "1", traceId: "trace-1" }],
+ [
+ { id: "2", traceId: "trace-2" },
+ { id: "3", traceId: "trace-3" },
+ ],
+ ]);
+ expect(
+ transcript?.threads[1].messages.map((message) => message.observationId),
+ ).toEqual(["2", "3", "3"]);
+ });
+
+ it("does not list a replay-only generation as a contributor", () => {
+ const transcript = getTranscript([
+ generation("1", ["A"], ["B"]),
+ generation("2", ["A", "B"], []),
+ ]);
+ expect(transcript?.threads).toHaveLength(1);
+ expect(transcript?.threads[0].observations).toEqual([
+ { id: "1", traceId: "trace-1" },
+ ]);
+ });
+
+ it("have unique names", () => {
+ const names = transcriptFixtures.map((fixture) => fixture.name);
+ expect(new Set(names).size).toBe(names.length);
+ });
+
+ describe.each(transcriptFixtures)("$name", (fixture) => {
+ const ids = fixture.observations.map((observation) => observation.id);
+ const traceIds = new Set(
+ fixture.observations.map((observation) => observation.trace_id),
+ );
+
+ it("has unique observation ids", () => {
+ expect(new Set(ids).size).toBe(ids.length);
+ });
+
+ it("matches its scope", () => {
+ if (fixture.scope === "trace") {
+ expect(traceIds.size).toBe(1);
+ return;
+ }
+
+ // Session scope spans several traces; session identity lives on the
+ // trace, not on observation records, so we only assert the shape here.
+ expect(traceIds.size).toBeGreaterThan(1);
+ });
+
+ // Complete each seed into a full ClickHouse observation record and convert
+ // it to a domain `Observation`, then build the transcript.
+ it("returns the expected transcript", () => {
+ const observations = fixture.observations.map((observation) =>
+ convertObservation(createObservation(observation)),
+ );
+ const transcript = getTranscript(observations);
+
+ // console.log("----------Transcript-------------------");
+ // console.log(JSON.stringify(transcript, null, 2));
+ console.log("----------Formatted Transcript-------------------");
+ console.log(
+ formatTranscript(fixture.name, transcript, observations, {
+ hideReasoning: true,
+ }),
+ );
+ console.log("-------------------------------------------------");
+
+ if (fixture.expected === undefined) return;
+ expect(transcript).toEqual(fixture.expected);
+ });
+ });
+});
diff --git a/packages/shared/src/server/transcript/fixtures/format-transcript.ts b/packages/shared/src/server/transcript/fixtures/format-transcript.ts
new file mode 100644
index 0000000000..850d048a24
--- /dev/null
+++ b/packages/shared/src/server/transcript/fixtures/format-transcript.ts
@@ -0,0 +1,115 @@
+import type { Observation } from "../../../domain";
+import type {
+ NormalizedMessagePart,
+ NormalizedMessageRole,
+} from "../../../utils/normalized-io";
+import type { Transcript } from "../types";
+
+const MAX_TEXT = 400;
+
+const clip = (text: string) =>
+ text.length > MAX_TEXT ? `${text.slice(0, MAX_TEXT - 1)}…` : text;
+
+const ROLE_LABEL: Record<NormalizedMessageRole, string> = {
+ system: "System",
+ user: "User",
+ assistant: "Assistant",
+ tool: "Tool",
+};
+
+const isRecord = (value: unknown): value is Record<string, unknown> =>
+ typeof value === "object" && value !== null && !Array.isArray(value);
+
+/** `key: value` per field for objects, one line otherwise. */
+function fieldLines(value: unknown): string[] {
+ if (!isRecord(value)) return [clip(JSON.stringify(value) ?? "undefined")];
+ return Object.entries(value).map(([key, field]) => {
+ const rendered =
+ typeof field === "string"
+ ? field
+ : (JSON.stringify(field) ?? "undefined");
+ return `${key}: ${clip(rendered)}`;
+ });
+}
+
+/** Content lines for one part, without the message indent. */
+function partLines(part: NormalizedMessagePart): string[] {
+ const indent = (lines: string[]) => lines.map((line) => ` ${line}`);
+ switch (part.type) {
+ case "text":
+ return [clip(part.refusal ? `(refusal) ${part.text}` : part.text)];
+ case "reasoning":
+ return part.content.kind === "text"
+ ? [`(reasoning) ${clip(part.content.text)}`]
+ : [`(reasoning: ${part.content.kind})`];
+ case "tool-call":
+ return [
+ `βš™ ${part.toolName} ${part.toolCallId ?? "no id"}${part.invalid ? " (invalid)" : ""}`,
+ " Parameters",
+ ...indent(fieldLines(part.input)),
+ ];
+ case "tool-result":
+ return [
+ `β€· ${part.toolName ?? "tool result"} ${part.toolCallId ?? "no id"}${part.isError ? " (error)" : ""}`,
+ " Result",
+ ...indent(fieldLines(part.output)),
+ ];
+ case "file":
+ return [`πŸ“Ž ${part.mediaType ?? "file"} (${part.content.kind})`];
+ case "data":
+ return ["Data", ...indent(fieldLines(part.value))];
+ case "custom":
+ return [`Custom ${part.kind}`, ...indent(fieldLines(part.value))];
+ }
+}
+
+/**
+ * Chat-shaped rendering of a transcript for eyeballing during development:
+ * one block per message with a role label and the emitting generation, tool
+ * calls and results as parameter lists, everything else as text.
+ */
+export function formatTranscript(
+ title: string,
+ transcript: Transcript | null,
+ observations: Observation[] = [],
+ options: { hideReasoning?: boolean } = {},
+): string {
+ const names = new Map(observations.map((o) => [o.id, o.name ?? ""]));
+ const label = (id: string) => {
+ const name = names.get(id);
+ return name ? `${name} (${id})` : id;
+ };
+
+ const lines = [`━━ ${title}`];
+
+ if (transcript === null) {
+ lines.push(" (null transcript)");
+ return lines.join("\n");
+ }
+
+ transcript.threads.forEach((thread, threadIndex) => {
+ lines.push(
+ "",
+ `β”Œ thread ${threadIndex + 1} Β· trace(s) ${[...new Set(thread.observations.map(({ traceId }) => traceId))].join(", ")}`,
+ `β”‚ generations: ${thread.observations.map(({ id }) => label(id)).join(", ")}`,
+ );
+
+ for (const message of thread.messages) {
+ const parts = options.hideReasoning
+ ? message.parts.filter((part) => part.type !== "reasoning")
+ : message.parts;
+ if (options.hideReasoning && parts.length === 0) continue;
+ const sender = message.senderName ? ` (${message.senderName})` : "";
+ lines.push(
+ "β”‚",
+ `β”‚ ${ROLE_LABEL[message.role]}${sender} Β· ${label(message.observationId)} Β· ${message.source}`,
+ );
+ for (const part of parts) {
+ for (const line of partLines(part)) lines.push(`β”‚ ${line}`);
+ }
+ }
+ lines.push("β””");
+ });
+
+ return lines.join("\n");
+}
diff --git a/packages/shared/src/server/transcript/fixtures/index.ts b/packages/shared/src/server/transcript/fixtures/index.ts
new file mode 100644
index 0000000000..6d6b04ed0a
--- /dev/null
+++ b/packages/shared/src/server/transcript/fixtures/index.ts
@@ -0,0 +1,28 @@
+import type { TranscriptFixture } from "./fixture-types";
+import { cumulativeHistoryFixture } from "./session/cumulative-history";
+import { orderSupportRoutingFixture } from "./session/order-support-routing";
+import { reorderedHistoryFixture } from "./session/reordered-history";
+import { openaiAgentsSpanishHandoffFixture } from "./trace/openai-agents-spanish-handoff";
+import { vercelAiSdkDocsToolLoopFixture } from "./trace/vercel-ai-sdk-docs-tool-loop";
+import { openaiAgentsJokeAndRatingFixture } from "./trace/openai-agents-joke-and-rating";
+import { supportCopilotRefundLoopFixture } from "./trace/support-copilot-refund-loop";
+
+export type { TranscriptFixture } from "./fixture-types";
+
+const traceTranscriptFixtures: TranscriptFixture[] = [
+ openaiAgentsSpanishHandoffFixture,
+ vercelAiSdkDocsToolLoopFixture,
+ openaiAgentsJokeAndRatingFixture,
+ supportCopilotRefundLoopFixture,
+];
+
+const sessionTranscriptFixtures: TranscriptFixture[] = [
+ orderSupportRoutingFixture,
+ cumulativeHistoryFixture,
+ reorderedHistoryFixture,
+];
+
+export const transcriptFixtures: TranscriptFixture[] = [
+ ...traceTranscriptFixtures,
+ ...sessionTranscriptFixtures,
+];
diff --git a/packages/shared/src/server/transcript/fixtures/session/cumulative-history.ts b/packages/shared/src/server/transcript/fixtures/session/cumulative-history.ts
new file mode 100644
index 0000000000..d8433b2b7e
--- /dev/null
+++ b/packages/shared/src/server/transcript/fixtures/session/cumulative-history.ts
@@ -0,0 +1,63 @@
+import type { TranscriptFixture } from "../fixture-types";
+
+/**
+ * Two root generations in separate traces of one synthetic session.
+ * Based on SessionConversationTimeline/fns/processTimelineMessages.clienttest.ts.
+ */
+export const cumulativeHistoryFixture = {
+ name: "Cumulative history across two session traces",
+ scope: "session",
+ description:
+ "Adapted from Ben's cumulative-history test. Expect one thread with the initial exchange and follow-up once each; replayed messages retain the first trace and generation IDs.",
+ observations: [
+ {
+ id: "cumulative-history-generation-1",
+ trace_id: "cumulative-history-trace-1",
+ project_id: "transcript-fixture-project",
+ parent_observation_id: null,
+ type: "GENERATION",
+ name: "Session turn 1",
+ start_time: "2026-01-01T12:00:00.000Z",
+ end_time: "2026-01-01T12:00:01.000Z",
+ input: JSON.stringify([
+ {
+ role: "user",
+ content: "Initial request",
+ },
+ ]),
+ output: JSON.stringify({
+ role: "assistant",
+ content: "First response",
+ }),
+ },
+ {
+ id: "cumulative-history-generation-2",
+ trace_id: "cumulative-history-trace-2",
+ project_id: "transcript-fixture-project",
+ parent_observation_id: null,
+ type: "GENERATION",
+ name: "Session turn 2",
+ start_time: "2026-01-01T12:01:00.000Z",
+ end_time: "2026-01-01T12:01:01.000Z",
+ input: JSON.stringify([
+ {
+ role: "user",
+ content: "Initial request",
+ },
+ {
+ role: "assistant",
+ content: "First response",
+ },
+ {
+ role: "user",
+ content: "Follow-up request",
+ },
+ ]),
+ output: JSON.stringify({
+ role: "assistant",
+ content: "Second response",
+ }),
+ },
+ ],
+ expected: undefined,
+} satisfies TranscriptFixture;
diff --git a/packages/shared/src/server/transcript/fixtures/session/order-support-routing.ts b/packages/shared/src/server/transcript/fixtures/session/order-support-routing.ts
new file mode 100644
index 0000000000..cb0ef976f1
--- /dev/null
+++ b/packages/shared/src/server/transcript/fixtures/session/order-support-routing.ts
@@ -0,0 +1,217 @@
+import type { TranscriptFixture } from "../fixture-types";
+
+/**
+ * Synthetic reconstruction: IDs, timestamps, names and payloads are placeholders.
+ * Each trace has this hierarchy (optional children noted):
+ * [SPAN] Support assistant β€” user/assistant text
+ * [SPAN] Gather context
+ * [SPAN] Respond to request
+ * [SPAN] Escalation classification
+ * [GENERATION] Classify escalation β€” system/user β†’ structured classification
+ * [GENERATION] Analyze (trace 1 only; I/O unknown)
+ * [SPAN] Check incident
+ * [GENERATION] Select workflow (trace 1 only) β€” user/system β†’ tool call
+ * [SPAN] Run order workflow
+ * [GENERATION] Manage customer order (trace 1) β€” system/user β†’ tool call
+ * [GENERATION] Clarify issue (trace 3) β€” system/user β†’ text
+ * All non-root spans intentionally have undefined I/O.
+ */
+
+type Seed = TranscriptFixture["observations"][number];
+
+function observation(
+ turn: number,
+ id: string,
+ parent: string | null,
+ name: string,
+ offset: number,
+ details: Partial<Seed> = {},
+): Seed {
+ return {
+ project_id: "transcript-fixture-project",
+ trace_id: `order-support-trace-${turn}`,
+ id: `order-support-${turn}-${id}`,
+ parent_observation_id: parent ? `order-support-${turn}-${parent}` : null,
+ type: "SPAN",
+ name,
+ start_time: new Date(Date.UTC(2026, 0, 1, 12, turn, offset)).toISOString(),
+ end_time: new Date(Date.UTC(2026, 0, 1, 12, turn, 20)).toISOString(),
+ input: undefined,
+ output: undefined,
+ ...details,
+ };
+}
+
+const classification = {
+ role: "assistant",
+ tool_calls: [],
+ content: { classification: "NO_ESCALATION_REQUIRED" },
+};
+const conversation =
+ "USER: Issue with my order --- ASSISTANT: Select the order you need help with: --- ASSISTANT: User is shown the following orders to pick from: .... --- USER: Item 123.";
+
+function baseTrace(turn: number, userContent: unknown): Seed[] {
+ return [
+ observation(turn, "root", null, "Support assistant", 0, {
+ input: "user text",
+ output: "assistant text",
+ }),
+ observation(turn, "context", "root", "Gather context", 1),
+ observation(turn, "respond", "root", "Respond to request", 2),
+ observation(turn, "escalation", "respond", "Escalation classification", 3),
+ observation(turn, "classify", "escalation", "Classify escalation", 4, {
+ type: "GENERATION",
+ input: JSON.stringify([
+ { role: "system", content: "instruction 1" },
+ { role: "user", content: userContent },
+ ]),
+ output: JSON.stringify(classification),
+ }),
+ ];
+}
+
+export const orderSupportRoutingFixture = {
+ name: "Order support routing across three session traces",
+ scope: "session",
+ description:
+ "Synthetic three-turn session with classification, workflow selection and tool calls. Later classifiers embed conversation history in a structured user message rather than replaying message arrays. Root text is on spans, one generation has unknown I/O, and separate generations use different system instructions; transcript expectations remain open.",
+ observations: [
+ ...baseTrace(1, "question"),
+ observation(1, "analyze", "respond", "Analyze utterance", 5, {
+ type: "GENERATION",
+ }),
+ observation(1, "incident", "respond", "Check incident", 6),
+ observation(1, "select", "respond", "Select workflow", 7, {
+ type: "GENERATION",
+ input: JSON.stringify({
+ messages: [
+ { role: "user", content: "question" },
+ { role: "system", content: "instruction 2" },
+ ],
+ tools: [
+ {
+ type: "function",
+ function: {
+ parameters: {},
+ strict: false,
+ description: "Tool to manage customer order",
+ name: "manage_customer_order",
+ },
+ },
+ {
+ type: "function",
+ function: {
+ parameters: {},
+ strict: false,
+ description:
+ "Default tool provided out of the box. Use it if specialized tools are less applicable or unavailable.",
+ name: "answer_generation",
+ },
+ },
+ ],
+ }),
+ output: JSON.stringify({
+ role: "assistant",
+ tool_calls: [
+ {
+ id: "call_1234567890",
+ type: "function",
+ function: { name: "manage_customer_order", arguments: {} },
+ },
+ ],
+ content: "",
+ }),
+ }),
+ observation(1, "workflow", "respond", "Run order workflow", 8),
+ observation(1, "manage", "workflow", "Manage customer order", 9, {
+ type: "GENERATION",
+ input: JSON.stringify({
+ messages: [
+ { role: "system", content: "Instructions 3" },
+ { role: "user", content: "Issue with my order" },
+ ],
+ tools: [
+ {
+ type: "function",
+ function: {
+ parameters: {
+ type: "object",
+ required: [],
+ additionalProperties: false,
+ properties: {},
+ },
+ strict: true,
+ description: "Retrieves a list of orders for the current user",
+ name: "findUserOrders",
+ },
+ },
+ {
+ type: "function",
+ function: {
+ parameters: {
+ type: "object",
+ required: ["orderId"],
+ additionalProperties: false,
+ properties: {
+ orderId: {
+ description: "The ID of the order to select",
+ type: "string",
+ },
+ },
+ },
+ strict: true,
+ description: "Selects an order from data store.",
+ name: "selectUserOrder",
+ },
+ },
+ {
+ type: "function",
+ function: {
+ parameters: {
+ type: "object",
+ required: [],
+ additionalProperties: false,
+ properties: {},
+ },
+ strict: true,
+ description: "Answer the user's question",
+ name: "answerTool",
+ },
+ },
+ ],
+ }),
+ output: JSON.stringify({
+ role: "assistant",
+ tool_calls: [
+ {
+ id: "call_0987654321",
+ type: "function",
+ function: { name: "findUserOrders", arguments: {} },
+ },
+ ],
+ content: "",
+ }),
+ }),
+ ...baseTrace(2, { conversation }),
+ observation(2, "incident", "respond", "Check incident", 6),
+ observation(2, "workflow", "respond", "Run order workflow", 8),
+ ...baseTrace(3, {
+ conversation: `${conversation} --- ASSISTANT: Great, I can help with that. Please explain what assistance you need. --- USER: No more, thank you.`,
+ }),
+ observation(3, "incident", "respond", "Check incident", 6),
+ observation(3, "workflow", "respond", "Run order workflow", 8),
+ observation(3, "clarify", "workflow", "Clarify issue", 9, {
+ type: "GENERATION",
+ input: JSON.stringify([
+ { role: "system", content: "instruction 4" },
+ { role: "user", content: "No more, thank you." },
+ ]),
+ output: JSON.stringify({
+ role: "assistant",
+ tool_calls: [],
+ content: "false-positives",
+ }),
+ }),
+ ],
+ expected: undefined,
+} satisfies TranscriptFixture;
diff --git a/packages/shared/src/server/transcript/fixtures/session/reordered-history.ts b/packages/shared/src/server/transcript/fixtures/session/reordered-history.ts
new file mode 100644
index 0000000000..5eb5804603
--- /dev/null
+++ b/packages/shared/src/server/transcript/fixtures/session/reordered-history.ts
@@ -0,0 +1,72 @@
+import type { TranscriptFixture } from "../fixture-types";
+
+/**
+ * Two root generations in separate traces of one synthetic session.
+ * Based on SessionConversationTimeline/fns/processTimelineMessages.clienttest.ts.
+ */
+export const reorderedHistoryFixture = {
+ name: "Reordered history across two session traces",
+ scope: "session",
+ description:
+ "Desired result is one thread with A, B, C, New, Answer; this deliberately exercises order-insensitive reconciliation beyond prefix matching.",
+ observations: [
+ {
+ id: "reordered-history-generation-1",
+ trace_id: "reordered-history-trace-1",
+ project_id: "transcript-fixture-project",
+ parent_observation_id: null,
+ type: "GENERATION",
+ name: "Session turn 1",
+ start_time: "2026-01-01T12:00:00.000Z",
+ end_time: "2026-01-01T12:00:01.000Z",
+ input: JSON.stringify([
+ {
+ role: "user",
+ content: "A",
+ },
+ {
+ role: "user",
+ content: "B",
+ },
+ {
+ role: "user",
+ content: "C",
+ },
+ ]),
+ output: null,
+ },
+ {
+ id: "reordered-history-generation-2",
+ trace_id: "reordered-history-trace-2",
+ project_id: "transcript-fixture-project",
+ parent_observation_id: null,
+ type: "GENERATION",
+ name: "Session turn 2",
+ start_time: "2026-01-01T12:01:00.000Z",
+ end_time: "2026-01-01T12:01:01.000Z",
+ input: JSON.stringify([
+ {
+ role: "user",
+ content: "B",
+ },
+ {
+ role: "user",
+ content: "C",
+ },
+ {
+ role: "user",
+ content: "A",
+ },
+ {
+ role: "user",
+ content: "New",
+ },
+ ]),
+ output: JSON.stringify({
+ role: "assistant",
+ content: "Answer",
+ }),
+ },
+ ],
+ expected: undefined,
+} satisfies TranscriptFixture;
diff --git a/packages/shared/src/server/transcript/fixtures/trace/openai-agents-joke-and-rating.ts b/packages/shared/src/server/transcript/fixtures/trace/openai-agents-joke-and-rating.ts
new file mode 100644
index 0000000000..09de0da27d
--- /dev/null
+++ b/packages/shared/src/server/transcript/fixtures/trace/openai-agents-joke-and-rating.ts
@@ -0,0 +1,372 @@
+import type { TranscriptFixture } from "../fixture-types";
+
+/**
+ * Separate agent executions generate a joke and rate it. The second prompt
+ * quotes the joke inside a fresh user message rather than replaying the first
+ * assistant message. Export order is preserved below.
+ *
+ * [SPAN] Joke workflow t-afa1ae379b6c3a5333e2bab357f31cdc
+ * [AGENT] Joke workflow 5427d87c68697585
+ * [AGENT] Joke generator b4b63a5f61c8689a
+ * [GENERATION] response f8fdd8164d6f8dbe
+ * [AGENT] Joke generator 0e44622cd302dacc
+ * [GENERATION] response 496efdc060a5399b
+ */
+
+const traceId = "afa1ae379b6c3a5333e2bab357f31cdc";
+const projectId = "transcript-fixture-project";
+
+const payload1 = null;
+
+const payload2 = JSON.stringify([
+ {
+ content: "Tell me a joke",
+ role: "user",
+ },
+]);
+
+const payload3 = JSON.stringify({
+ id: "resp_05e10c6f9849abe80068db9d06bac081a08d695570d759e9df",
+ created_at: 1759223046,
+ error: null,
+ incomplete_details: null,
+ instructions: "Tell funny jokes.",
+ metadata: {},
+ model: "gpt-4.1-2025-04-14",
+ object: "response",
+ output: [
+ {
+ id: "msg_05e10c6f9849abe80068db9d07270c81a0ab2f2738f61c1d11",
+ content: [
+ {
+ annotations: [],
+ text: "Sure! Here’s one for you:\n\nWhy did the scarecrow win an award?\n\nBecause he was outstanding in his field! πŸŒΎπŸ˜„",
+ type: "output_text",
+ logprobs: [],
+ },
+ ],
+ role: "assistant",
+ status: "completed",
+ type: "message",
+ },
+ ],
+ parallel_tool_calls: true,
+ temperature: 1,
+ tool_choice: "auto",
+ tools: [],
+ top_p: 1,
+ background: false,
+ conversation: null,
+ max_output_tokens: null,
+ max_tool_calls: null,
+ previous_response_id: null,
+ prompt: null,
+ prompt_cache_key: null,
+ reasoning: {
+ effort: null,
+ generate_summary: null,
+ summary: null,
+ },
+ safety_identifier: null,
+ service_tier: "default",
+ status: "completed",
+ text: {
+ format: {
+ type: "text",
+ },
+ verbosity: "medium",
+ },
+ top_logprobs: 0,
+ truncation: "disabled",
+ usage: {
+ input_tokens: 19,
+ input_tokens_details: {
+ cached_tokens: 0,
+ },
+ output_tokens: 30,
+ output_tokens_details: {
+ reasoning_tokens: 0,
+ },
+ total_tokens: 49,
+ },
+ user: null,
+ billing: {
+ payer: "developer",
+ },
+ store: true,
+});
+
+const payload4 = JSON.stringify([
+ {
+ content:
+ "Rate this joke: Sure! Here’s one for you:\n\nWhy did the scarecrow win an award?\n\nBecause he was outstanding in his field! πŸŒΎπŸ˜„",
+ role: "user",
+ },
+]);
+
+const payload5 = JSON.stringify({
+ id: "resp_03e1816217fbc8a30068db9d07949c81a29151e95e3c492c7e",
+ created_at: 1759223047,
+ error: null,
+ incomplete_details: null,
+ instructions: "Tell funny jokes.",
+ metadata: {},
+ model: "gpt-4.1-2025-04-14",
+ object: "response",
+ output: [
+ {
+ id: "msg_03e1816217fbc8a30068db9d07dbe481a28fae7f2a41f76827",
+ content: [
+ {
+ annotations: [],
+ text: "Great choice! That’s a classic jokeβ€”punny, wholesome, and easy to get. Here’s how I’d rate it:\n\n**Joke Rating: 8/10**\n\n**Why?**\n- **Punniness:** Strong, farm-fresh wordplay!\n- **Audience:** Universally funnyβ€”works for kids and adults.\n- **Originality:** Popular, but still a crowd-pleaser.\n\n**Bonus:** The addition of the emoji 🌾 gives it an extra playful touch! Want to hear a joke from me now?",
+ type: "output_text",
+ logprobs: [],
+ },
+ ],
+ role: "assistant",
+ status: "completed",
+ type: "message",
+ },
+ ],
+ parallel_tool_calls: true,
+ temperature: 1,
+ tool_choice: "auto",
+ tools: [],
+ top_p: 1,
+ background: false,
+ conversation: null,
+ max_output_tokens: null,
+ max_tool_calls: null,
+ previous_response_id: null,
+ prompt: null,
+ prompt_cache_key: null,
+ reasoning: {
+ effort: null,
+ generate_summary: null,
+ summary: null,
+ },
+ safety_identifier: null,
+ service_tier: "default",
+ status: "completed",
+ text: {
+ format: {
+ type: "text",
+ },
+ verbosity: "medium",
+ },
+ top_logprobs: 0,
+ truncation: "disabled",
+ usage: {
+ input_tokens: 48,
+ input_tokens_details: {
+ cached_tokens: 0,
+ },
+ output_tokens: 111,
+ output_tokens_details: {
+ reasoning_tokens: 0,
+ },
+ total_tokens: 159,
+ },
+ user: null,
+ billing: {
+ payer: "developer",
+ },
+ store: true,
+});
+
+// OTel envelope shared by every observation. ClickHouse stores observation
+// metadata as Map(String, String), so object-valued attributes live as JSON
+// strings and round-trip back through `parseMetadataCHRecordToDomain`.
+const scope = JSON.stringify({
+ name: "openinference.instrumentation.openai_agents",
+ version: "1.3.0",
+ attributes: {},
+});
+
+const resourceAttributes = JSON.stringify({
+ "telemetry.sdk.language": "python",
+ "telemetry.sdk.name": "opentelemetry",
+ "telemetry.sdk.version": "1.37.0",
+ "service.name": "unknown_service",
+});
+
+const agentAttributes = JSON.stringify({
+ "openinference.span.kind": "AGENT",
+});
+
+const generatorAttributes = JSON.stringify({
+ "llm.system": "openai",
+ "graph.node.id": "Joke generator",
+ "openinference.span.kind": "AGENT",
+});
+
+export const openaiAgentsJokeAndRatingFixture = {
+ name: "OpenAI Agents joke generation followed by rating",
+ scope: "trace",
+ description:
+ "Separate agent executions generate a joke and rate it. The second prompt quotes the joke inside a fresh user message rather than replaying the first assistant message.",
+ observations: [
+ {
+ trace_id: traceId,
+ project_id: projectId,
+ id: "t-afa1ae379b6c3a5333e2bab357f31cdc",
+ parent_observation_id: null,
+ type: "SPAN",
+ name: "Joke workflow",
+ start_time: "2025-09-30T09:04:06.678Z",
+ end_time: null,
+ provided_model_name: "",
+ metadata: {
+ scope,
+ resourceAttributes,
+ attributes: agentAttributes,
+ },
+ input: payload1,
+ output: payload1,
+ },
+ {
+ trace_id: traceId,
+ project_id: projectId,
+ id: "5427d87c68697585",
+ parent_observation_id: "t-afa1ae379b6c3a5333e2bab357f31cdc",
+ type: "AGENT",
+ name: "Joke workflow",
+ start_time: "2025-09-30T09:04:06.678Z",
+ end_time: "2025-09-30T09:04:09.031Z",
+ provided_model_name: "",
+ metadata: {
+ scope,
+ resourceAttributes,
+ attributes: agentAttributes,
+ },
+ input: payload1,
+ output: payload1,
+ },
+ {
+ trace_id: traceId,
+ project_id: projectId,
+ id: "b4b63a5f61c8689a",
+ parent_observation_id: "5427d87c68697585",
+ type: "AGENT",
+ name: "Joke generator",
+ start_time: "2025-09-30T09:04:06.679Z",
+ end_time: "2025-09-30T09:04:07.521Z",
+ provided_model_name: "",
+ metadata: {
+ scope,
+ resourceAttributes,
+ attributes: generatorAttributes,
+ },
+ input: payload1,
+ output: payload1,
+ },
+ {
+ trace_id: traceId,
+ project_id: projectId,
+ id: "f8fdd8164d6f8dbe",
+ parent_observation_id: "b4b63a5f61c8689a",
+ type: "GENERATION",
+ name: "response",
+ start_time: "2025-09-30T09:04:06.682Z",
+ end_time: "2025-09-30T09:04:07.517Z",
+ provided_model_name: "gpt-4.1-2025-04-14",
+ metadata: {
+ scope,
+ resourceAttributes,
+ attributes: JSON.stringify({
+ "llm.system": "openai",
+ "output.mime_type": "application/json",
+ "output.value":
+ '{"id":"resp_05e10c6f9849abe80068db9d06bac081a08d695570d759e9df","created_at":1759223046.0,"error":null,"incomplete_details":null,"instructions":"Tell funny jokes.","metadata":{},"model":"gpt-4.1-2025-04-14","object":"response","output":[{"id":"msg_05e10c6f9849abe80068db9d07270c81a0ab2f2738f61c1d11","content":[{"annotations":[],"text":"Sure! Here’s one for you:\\n\\nWhy did the scarecrow win an award?\\n\\nBecause he was outstanding in his field! πŸŒΎπŸ˜„","type":"output_text","logprobs":[]}],"role":"assistant","status":"completed","type":"message"}],"parallel_tool_calls":true,"temperature":1.0,"tool_choice":"auto","tools":[],"top_p":1.0,"background":false,"conversation":null,"max_output_tokens":null,"max_tool_calls":null,"previous_response_id":null,"prompt":null,"prompt_cache_key":null,"reasoning":{"effort":null,"generate_summary":null,"summary":null},"safety_identifier":null,"service_tier":"default","status":"completed","text":{"format":{"type":"text"},"verbosity":"medium"},"top_logprobs":0,"truncation":"disabled","usage":{"input_tokens":19,"input_tokens_details":{"cached_tokens":0},"output_tokens":30,"output_tokens_details":{"reasoning_tokens":0},"total_tokens":49},"user":null,"billing":{"payer":"developer"},"store":true}',
+ "llm.token_count.completion": "30",
+ "llm.token_count.prompt": "19",
+ "llm.token_count.total": "49",
+ "llm.token_count.prompt_details.cache_read": "0",
+ "llm.token_count.completion_details.reasoning": "0",
+ "llm.output_messages.0.message.role": "assistant",
+ "llm.output_messages.0.message.contents.0.message_content.type":
+ "text",
+ "llm.output_messages.0.message.contents.0.message_content.text":
+ "Sure! Here’s one for you:\n\nWhy did the scarecrow win an award?\n\nBecause he was outstanding in his field! πŸŒΎπŸ˜„",
+ "llm.input_messages.0.message.role": "system",
+ "llm.input_messages.0.message.content": "Tell funny jokes.",
+ "llm.model_name": "gpt-4.1-2025-04-14",
+ "llm.invocation_parameters":
+ '{"id": "resp_05e10c6f9849abe80068db9d06bac081a08d695570d759e9df", "created_at": 1759223046.0, "instructions": "Tell funny jokes.", "metadata": {}, "model": "gpt-4.1-2025-04-14", "parallel_tool_calls": true, "temperature": 1.0, "tool_choice": "auto", "top_p": 1.0, "background": false, "reasoning": {}, "service_tier": "default", "text": {"format": {"type": "text"}, "verbosity": "medium"}, "top_logprobs": 0, "truncation": "disabled", "billing": {"payer": "developer"}, "store": true}',
+ "input.mime_type": "application/json",
+ "input.value": '[{"content": "Tell me a joke", "role": "user"}]',
+ "llm.input_messages.1.message.role": "user",
+ "llm.input_messages.1.message.content": "Tell me a joke",
+ "openinference.span.kind": "LLM",
+ }),
+ },
+ input: payload2,
+ output: payload3,
+ },
+ {
+ trace_id: traceId,
+ project_id: projectId,
+ id: "0e44622cd302dacc",
+ parent_observation_id: "5427d87c68697585",
+ type: "AGENT",
+ name: "Joke generator",
+ start_time: "2025-09-30T09:04:07.521Z",
+ end_time: "2025-09-30T09:04:09.030Z",
+ provided_model_name: "",
+ metadata: {
+ scope,
+ resourceAttributes,
+ attributes: generatorAttributes,
+ },
+ input: payload1,
+ output: payload1,
+ },
+ {
+ trace_id: traceId,
+ project_id: projectId,
+ id: "496efdc060a5399b",
+ parent_observation_id: "0e44622cd302dacc",
+ type: "GENERATION",
+ name: "response",
+ start_time: "2025-09-30T09:04:07.522Z",
+ end_time: "2025-09-30T09:04:09.029Z",
+ provided_model_name: "gpt-4.1-2025-04-14",
+ metadata: {
+ scope,
+ resourceAttributes,
+ attributes: JSON.stringify({
+ "llm.system": "openai",
+ "output.mime_type": "application/json",
+ "output.value":
+ '{"id":"resp_03e1816217fbc8a30068db9d07949c81a29151e95e3c492c7e","created_at":1759223047.0,"error":null,"incomplete_details":null,"instructions":"Tell funny jokes.","metadata":{},"model":"gpt-4.1-2025-04-14","object":"response","output":[{"id":"msg_03e1816217fbc8a30068db9d07dbe481a28fae7f2a41f76827","content":[{"annotations":[],"text":"Great choice! That’s a classic jokeβ€”punny, wholesome, and easy to get. Here’s how I’d rate it:\\n\\n**Joke Rating: 8/10**\\n\\n**Why?**\\n- **Punniness:** Strong, farm-fresh wordplay!\\n- **Audience:** Universally funnyβ€”works for kids and adults.\\n- **Originality:** Popular, but still a crowd-pleaser.\\n\\n**Bonus:** The addition of the emoji 🌾 gives it an extra playful touch! Want to hear a joke from me now?","type":"output_text","logprobs":[]}],"role":"assistant","status":"completed","type":"message"}],"parallel_tool_calls":true,"temperature":1.0,"tool_choice":"auto","tools":[],"top_p":1.0,"background":false,"conversation":null,"max_output_tokens":null,"max_tool_calls":null,"previous_response_id":null,"prompt":null,"prompt_cache_key":null,"reasoning":{"effort":null,"generate_summary":null,"summary":null},"safety_identifier":null,"service_tier":"default","status":"completed","text":{"format":{"type":"text"},"verbosity":"medium"},"top_logprobs":0,"truncation":"disabled","usage":{"input_tokens":48,"input_tokens_details":{"cached_tokens":0},"output_tokens":111,"output_tokens_details":{"reasoning_tokens":0},"total_tokens":159},"user":null,"billing":{"payer":"developer"},"store":true}',
+ "llm.token_count.completion": "111",
+ "llm.token_count.prompt": "48",
+ "llm.token_count.total": "159",
+ "llm.token_count.prompt_details.cache_read": "0",
+ "llm.token_count.completion_details.reasoning": "0",
+ "llm.output_messages.0.message.role": "assistant",
+ "llm.output_messages.0.message.contents.0.message_content.type":
+ "text",
+ "llm.output_messages.0.message.contents.0.message_content.text":
+ "Great choice! That’s a classic jokeβ€”punny, wholesome, and easy to get. Here’s how I’d rate it:\n\n**Joke Rating: 8/10**\n\n**Why?**\n- **Punniness:** Strong, farm-fresh wordplay!\n- **Audience:** Universally funnyβ€”works for kids and adults.\n- **Originality:** Popular, but still a crowd-pleaser.\n\n**Bonus:** The addition of the emoji 🌾 gives it an extra playful touch! Want to hear a joke from me now?",
+ "llm.input_messages.0.message.role": "system",
+ "llm.input_messages.0.message.content": "Tell funny jokes.",
+ "llm.model_name": "gpt-4.1-2025-04-14",
+ "llm.invocation_parameters":
+ '{"id": "resp_03e1816217fbc8a30068db9d07949c81a29151e95e3c492c7e", "created_at": 1759223047.0, "instructions": "Tell funny jokes.", "metadata": {}, "model": "gpt-4.1-2025-04-14", "parallel_tool_calls": true, "temperature": 1.0, "tool_choice": "auto", "top_p": 1.0, "background": false, "reasoning": {}, "service_tier": "default", "text": {"format": {"type": "text"}, "verbosity": "medium"}, "top_logprobs": 0, "truncation": "disabled", "billing": {"payer": "developer"}, "store": true}',
+ "input.mime_type": "application/json",
+ "input.value":
+ '[{"content": "Rate this joke: Sure! Here’s one for you:\\n\\nWhy did the scarecrow win an award?\\n\\nBecause he was outstanding in his field! πŸŒΎπŸ˜„", "role": "user"}]',
+ "llm.input_messages.1.message.role": "user",
+ "llm.input_messages.1.message.content":
+ "Rate this joke: Sure! Here’s one for you:\n\nWhy did the scarecrow win an award?\n\nBecause he was outstanding in his field! πŸŒΎπŸ˜„",
+ "openinference.span.kind": "LLM",
+ }),
+ },
+ input: payload4,
+ output: payload5,
+ },
+ ],
+ expected: undefined,
+} satisfies TranscriptFixture;
diff --git a/packages/shared/src/server/transcript/fixtures/trace/openai-agents-spanish-handoff.ts b/packages/shared/src/server/transcript/fixtures/trace/openai-agents-spanish-handoff.ts
new file mode 100644
index 0000000000..01ace8395e
--- /dev/null
+++ b/packages/shared/src/server/transcript/fixtures/trace/openai-agents-spanish-handoff.ts
@@ -0,0 +1,470 @@
+import type { TranscriptFixture } from "../fixture-types";
+
+/**
+ * Agent workflow -> Triage agent -> response + handoff tool;
+ * Agent workflow -> Spanish agent -> response.
+ * Repeated (trace ID, observation ID) pairs retain their first exported row.
+ */
+export const openaiAgentsSpanishHandoffFixture = {
+ name: "OpenAI Agents Spanish handoff",
+ scope: "trace",
+ description:
+ "Triage hands off to a Spanish agent. This exercises changing instructions and agent handoff. Duplicate observation/trace ID pairs are removed, preserving first-occurrence order.",
+ observations: [
+ {
+ project_id: "transcript-fixture-project",
+ id: "t-c376e44920527b875add9d97b4ed9312",
+ trace_id: "c376e44920527b875add9d97b4ed9312",
+ parent_observation_id: null,
+ type: "SPAN",
+ name: "Agent workflow",
+ start_time: "2025-09-30T09:03:51.191Z",
+ end_time: null,
+ metadata: {
+ scope: JSON.stringify({
+ name: "openinference.instrumentation.openai_agents",
+ version: "1.3.0",
+ attributes: {},
+ }),
+ resourceAttributes: JSON.stringify({
+ "telemetry.sdk.language": "python",
+ "telemetry.sdk.name": "opentelemetry",
+ "telemetry.sdk.version": "1.37.0",
+ "service.name": "unknown_service",
+ }),
+ attributes: JSON.stringify({
+ "openinference.span.kind": "AGENT",
+ }),
+ },
+ provided_model_name: "",
+ input: null,
+ output: null,
+ },
+ {
+ project_id: "transcript-fixture-project",
+ id: "566127d0952130b9",
+ trace_id: "c376e44920527b875add9d97b4ed9312",
+ parent_observation_id: "5cc6d9aebded54a0",
+ type: "AGENT",
+ name: "Triage agent",
+ start_time: "2025-09-30T09:03:51.191Z",
+ end_time: "2025-09-30T09:03:52.758Z",
+ metadata: {
+ scope: JSON.stringify({
+ name: "openinference.instrumentation.openai_agents",
+ version: "1.3.0",
+ attributes: {},
+ }),
+ resourceAttributes: JSON.stringify({
+ "telemetry.sdk.language": "python",
+ "telemetry.sdk.name": "opentelemetry",
+ "telemetry.sdk.version": "1.37.0",
+ "service.name": "unknown_service",
+ }),
+ attributes: JSON.stringify({
+ "llm.system": "openai",
+ "graph.node.id": "Triage agent",
+ "openinference.span.kind": "AGENT",
+ }),
+ },
+ provided_model_name: "",
+ input: null,
+ output: null,
+ },
+ {
+ project_id: "transcript-fixture-project",
+ id: "5cc6d9aebded54a0",
+ trace_id: "c376e44920527b875add9d97b4ed9312",
+ parent_observation_id: "t-c376e44920527b875add9d97b4ed9312",
+ type: "AGENT",
+ name: "Agent workflow",
+ start_time: "2025-09-30T09:03:51.191Z",
+ end_time: "2025-09-30T09:03:54.434Z",
+ metadata: {
+ scope: JSON.stringify({
+ name: "openinference.instrumentation.openai_agents",
+ version: "1.3.0",
+ attributes: {},
+ }),
+ resourceAttributes: JSON.stringify({
+ "telemetry.sdk.language": "python",
+ "telemetry.sdk.name": "opentelemetry",
+ "telemetry.sdk.version": "1.37.0",
+ "service.name": "unknown_service",
+ }),
+ attributes: JSON.stringify({
+ "openinference.span.kind": "AGENT",
+ }),
+ },
+ provided_model_name: "",
+ input: null,
+ output: null,
+ },
+ {
+ project_id: "transcript-fixture-project",
+ id: "94a1dc00d7067ae8",
+ trace_id: "c376e44920527b875add9d97b4ed9312",
+ parent_observation_id: "566127d0952130b9",
+ type: "GENERATION",
+ name: "response",
+ start_time: "2025-09-30T09:03:51.192Z",
+ end_time: "2025-09-30T09:03:52.756Z",
+ metadata: {
+ scope: JSON.stringify({
+ name: "openinference.instrumentation.openai_agents",
+ version: "1.3.0",
+ attributes: {},
+ }),
+ resourceAttributes: JSON.stringify({
+ "telemetry.sdk.language": "python",
+ "telemetry.sdk.name": "opentelemetry",
+ "telemetry.sdk.version": "1.37.0",
+ "service.name": "unknown_service",
+ }),
+ attributes: JSON.stringify({
+ "llm.system": "openai",
+ "output.mime_type": "application/json",
+ "output.value":
+ '{"id":"resp_000230c027f4a45e0068db9cf79e6c81a3a60bf6b6233384f5","created_at":1759223031.0,"error":null,"incomplete_details":null,"instructions":"Handoff to the appropriate agent based on the language of the request.","metadata":{},"model":"gpt-4.1-2025-04-14","object":"response","output":[{"arguments":"{}","call_id":"call_i81vsF01rKH8cDZ0SLQQBUA2","name":"transfer_to_spanish_agent","type":"function_call","id":"fc_000230c027f4a45e0068db9cf8930481a3970aa5df55a731e3","status":"completed"}],"parallel_tool_calls":true,"temperature":1.0,"tool_choice":"auto","tools":[{"name":"transfer_to_spanish_agent","parameters":{"additionalProperties":false,"type":"object","properties":{},"required":[]},"strict":true,"type":"function","description":"Handoff to the Spanish agent agent to handle the request. "},{"name":"transfer_to_english_agent","parameters":{"additionalProperties":false,"type":"object","properties":{},"required":[]},"strict":true,"type":"function","description":"Handoff to the English agent agent to handle the request. "}],"top_p":1.0,"background":false,"conversation":null,"max_output_tokens":null,"max_tool_calls":null,"previous_response_id":null,"prompt":null,"prompt_cache_key":null,"reasoning":{"effort":null,"generate_summary":null,"summary":null},"safety_identifier":null,"service_tier":"default","status":"completed","text":{"format":{"type":"text"},"verbosity":"medium"},"top_logprobs":0,"truncation":"disabled","usage":{"input_tokens":87,"input_tokens_details":{"cached_tokens":0},"output_tokens":14,"output_tokens_details":{"reasoning_tokens":0},"total_tokens":101},"user":null,"billing":{"payer":"developer"},"store":true}',
+ "llm.tools.0.tool.json_schema":
+ '{"type": "function", "function": {"name": "transfer_to_spanish_agent", "description": "Handoff to the Spanish agent agent to handle the request. ", "parameters": {"additionalProperties": false, "type": "object", "properties": {}, "required": []}, "strict": true}}',
+ "llm.tools.1.tool.json_schema":
+ '{"type": "function", "function": {"name": "transfer_to_english_agent", "description": "Handoff to the English agent agent to handle the request. ", "parameters": {"additionalProperties": false, "type": "object", "properties": {}, "required": []}, "strict": true}}',
+ "llm.token_count.completion": "14",
+ "llm.token_count.prompt": "87",
+ "llm.token_count.total": "101",
+ "llm.token_count.prompt_details.cache_read": "0",
+ "llm.token_count.completion_details.reasoning": "0",
+ "llm.output_messages.0.message.role": "assistant",
+ "llm.output_messages.0.message.tool_calls.0.tool_call.id":
+ "call_i81vsF01rKH8cDZ0SLQQBUA2",
+ "llm.output_messages.0.message.tool_calls.0.tool_call.function.name":
+ "transfer_to_spanish_agent",
+ "llm.input_messages.0.message.role": "system",
+ "llm.input_messages.0.message.content":
+ "Handoff to the appropriate agent based on the language of the request.",
+ "llm.model_name": "gpt-4.1-2025-04-14",
+ "llm.invocation_parameters":
+ '{"id": "resp_000230c027f4a45e0068db9cf79e6c81a3a60bf6b6233384f5", "created_at": 1759223031.0, "instructions": "Handoff to the appropriate agent based on the language of the request.", "metadata": {}, "model": "gpt-4.1-2025-04-14", "parallel_tool_calls": true, "temperature": 1.0, "tool_choice": "auto", "top_p": 1.0, "background": false, "reasoning": {}, "service_tier": "default", "text": {"format": {"type": "text"}, "verbosity": "medium"}, "top_logprobs": 0, "truncation": "disabled", "billing": {"payer": "developer"}, "store": true}',
+ "input.mime_type": "application/json",
+ "input.value": '[{"content": "Hola, ΒΏcΓ³mo estΓ‘s?", "role": "user"}]',
+ "llm.input_messages.1.message.role": "user",
+ "llm.input_messages.1.message.content": "Hola, ΒΏcΓ³mo estΓ‘s?",
+ "openinference.span.kind": "LLM",
+ }),
+ },
+ provided_model_name: "gpt-4.1-2025-04-14",
+ input: JSON.stringify([
+ {
+ content: "Hola, ΒΏcΓ³mo estΓ‘s?",
+ role: "user",
+ },
+ ]),
+ output: JSON.stringify({
+ id: "resp_000230c027f4a45e0068db9cf79e6c81a3a60bf6b6233384f5",
+ created_at: 1759223031,
+ error: null,
+ incomplete_details: null,
+ instructions:
+ "Handoff to the appropriate agent based on the language of the request.",
+ metadata: {},
+ model: "gpt-4.1-2025-04-14",
+ object: "response",
+ output: [
+ {
+ arguments: "{}",
+ call_id: "call_i81vsF01rKH8cDZ0SLQQBUA2",
+ name: "transfer_to_spanish_agent",
+ type: "function_call",
+ id: "fc_000230c027f4a45e0068db9cf8930481a3970aa5df55a731e3",
+ status: "completed",
+ },
+ ],
+ parallel_tool_calls: true,
+ temperature: 1,
+ tool_choice: "auto",
+ tools: [
+ {
+ name: "transfer_to_spanish_agent",
+ parameters: {
+ additionalProperties: false,
+ type: "object",
+ properties: {},
+ required: [],
+ },
+ strict: true,
+ type: "function",
+ description:
+ "Handoff to the Spanish agent agent to handle the request. ",
+ },
+ {
+ name: "transfer_to_english_agent",
+ parameters: {
+ additionalProperties: false,
+ type: "object",
+ properties: {},
+ required: [],
+ },
+ strict: true,
+ type: "function",
+ description:
+ "Handoff to the English agent agent to handle the request. ",
+ },
+ ],
+ top_p: 1,
+ background: false,
+ conversation: null,
+ max_output_tokens: null,
+ max_tool_calls: null,
+ previous_response_id: null,
+ prompt: null,
+ prompt_cache_key: null,
+ reasoning: {
+ effort: null,
+ generate_summary: null,
+ summary: null,
+ },
+ safety_identifier: null,
+ service_tier: "default",
+ status: "completed",
+ text: {
+ format: {
+ type: "text",
+ },
+ verbosity: "medium",
+ },
+ top_logprobs: 0,
+ truncation: "disabled",
+ usage: {
+ input_tokens: 87,
+ input_tokens_details: {
+ cached_tokens: 0,
+ },
+ output_tokens: 14,
+ output_tokens_details: {
+ reasoning_tokens: 0,
+ },
+ total_tokens: 101,
+ },
+ user: null,
+ billing: {
+ payer: "developer",
+ },
+ store: true,
+ }),
+ },
+ {
+ project_id: "transcript-fixture-project",
+ id: "b1d77be34cecb5b5",
+ trace_id: "c376e44920527b875add9d97b4ed9312",
+ parent_observation_id: "566127d0952130b9",
+ type: "TOOL",
+ name: "handoff to Spanish agent",
+ start_time: "2025-09-30T09:03:52.758Z",
+ end_time: "2025-09-30T09:03:52.758Z",
+ metadata: {
+ scope: JSON.stringify({
+ name: "openinference.instrumentation.openai_agents",
+ version: "1.3.0",
+ attributes: {},
+ }),
+ resourceAttributes: JSON.stringify({
+ "telemetry.sdk.language": "python",
+ "telemetry.sdk.name": "opentelemetry",
+ "telemetry.sdk.version": "1.37.0",
+ "service.name": "unknown_service",
+ }),
+ attributes: JSON.stringify({
+ "llm.system": "openai",
+ "openinference.span.kind": "TOOL",
+ }),
+ },
+ provided_model_name: "",
+ input: null,
+ output: null,
+ },
+ {
+ project_id: "transcript-fixture-project",
+ id: "670e91064be3fb11",
+ trace_id: "c376e44920527b875add9d97b4ed9312",
+ parent_observation_id: "5cc6d9aebded54a0",
+ type: "AGENT",
+ name: "Spanish agent",
+ start_time: "2025-09-30T09:03:52.759Z",
+ end_time: "2025-09-30T09:03:54.434Z",
+ metadata: {
+ scope: JSON.stringify({
+ name: "openinference.instrumentation.openai_agents",
+ version: "1.3.0",
+ attributes: {},
+ }),
+ resourceAttributes: JSON.stringify({
+ "telemetry.sdk.language": "python",
+ "telemetry.sdk.name": "opentelemetry",
+ "telemetry.sdk.version": "1.37.0",
+ "service.name": "unknown_service",
+ }),
+ attributes: JSON.stringify({
+ "llm.system": "openai",
+ "graph.node.id": "Spanish agent",
+ "graph.node.parent_id": "Triage agent",
+ "openinference.span.kind": "AGENT",
+ }),
+ },
+ provided_model_name: "",
+ input: null,
+ output: null,
+ },
+ {
+ project_id: "transcript-fixture-project",
+ id: "a0391ae77593ad0c",
+ trace_id: "c376e44920527b875add9d97b4ed9312",
+ parent_observation_id: "670e91064be3fb11",
+ type: "GENERATION",
+ name: "response",
+ start_time: "2025-09-30T09:03:52.760Z",
+ end_time: "2025-09-30T09:03:54.433Z",
+ metadata: {
+ scope: JSON.stringify({
+ name: "openinference.instrumentation.openai_agents",
+ version: "1.3.0",
+ attributes: {},
+ }),
+ resourceAttributes: JSON.stringify({
+ "telemetry.sdk.language": "python",
+ "telemetry.sdk.name": "opentelemetry",
+ "telemetry.sdk.version": "1.37.0",
+ "service.name": "unknown_service",
+ }),
+ attributes: JSON.stringify({
+ "llm.system": "openai",
+ "output.mime_type": "application/json",
+ "output.value":
+ '{"id":"resp_000230c027f4a45e0068db9cf8d9f481a380b1ab4e2f52dd8a","created_at":1759223033.0,"error":null,"incomplete_details":null,"instructions":"You only speak Spanish.","metadata":{},"model":"gpt-4.1-2025-04-14","object":"response","output":[{"id":"msg_000230c027f4a45e0068db9cf9d14481a3a04289f3f1b0f9a1","content":[{"annotations":[],"text":"Β‘Hola! Estoy muy bien, gracias. ΒΏY tΓΊ, cΓ³mo estΓ‘s?","type":"output_text","logprobs":[]}],"role":"assistant","status":"completed","type":"message"}],"parallel_tool_calls":true,"temperature":1.0,"tool_choice":"auto","tools":[],"top_p":1.0,"background":false,"conversation":null,"max_output_tokens":null,"max_tool_calls":null,"previous_response_id":null,"prompt":null,"prompt_cache_key":null,"reasoning":{"effort":null,"generate_summary":null,"summary":null},"safety_identifier":null,"service_tier":"default","status":"completed","text":{"format":{"type":"text"},"verbosity":"medium"},"top_logprobs":0,"truncation":"disabled","usage":{"input_tokens":54,"input_tokens_details":{"cached_tokens":0},"output_tokens":17,"output_tokens_details":{"reasoning_tokens":0},"total_tokens":71},"user":null,"billing":{"payer":"developer"},"store":true}',
+ "llm.token_count.completion": "17",
+ "llm.token_count.prompt": "54",
+ "llm.token_count.total": "71",
+ "llm.token_count.prompt_details.cache_read": "0",
+ "llm.token_count.completion_details.reasoning": "0",
+ "llm.output_messages.0.message.role": "assistant",
+ "llm.output_messages.0.message.contents.0.message_content.type":
+ "text",
+ "llm.output_messages.0.message.contents.0.message_content.text":
+ "Β‘Hola! Estoy muy bien, gracias. ΒΏY tΓΊ, cΓ³mo estΓ‘s?",
+ "llm.input_messages.0.message.role": "system",
+ "llm.input_messages.0.message.content": "You only speak Spanish.",
+ "llm.model_name": "gpt-4.1-2025-04-14",
+ "llm.invocation_parameters":
+ '{"id": "resp_000230c027f4a45e0068db9cf8d9f481a380b1ab4e2f52dd8a", "created_at": 1759223033.0, "instructions": "You only speak Spanish.", "metadata": {}, "model": "gpt-4.1-2025-04-14", "parallel_tool_calls": true, "temperature": 1.0, "tool_choice": "auto", "top_p": 1.0, "background": false, "reasoning": {}, "service_tier": "default", "text": {"format": {"type": "text"}, "verbosity": "medium"}, "top_logprobs": 0, "truncation": "disabled", "billing": {"payer": "developer"}, "store": true}',
+ "input.mime_type": "application/json",
+ "input.value":
+ '[{"content": "Hola, ΒΏcΓ³mo estΓ‘s?", "role": "user"}, {"arguments": "{}", "call_id": "call_i81vsF01rKH8cDZ0SLQQBUA2", "name": "transfer_to_spanish_agent", "type": "function_call", "id": "fc_000230c027f4a45e0068db9cf8930481a3970aa5df55a731e3", "status": "completed"}, {"call_id": "call_i81vsF01rKH8cDZ0SLQQBUA2", "output": "{\\"assistant\\": \\"Spanish agent\\"}", "type": "function_call_output"}]',
+ "llm.input_messages.1.message.role": "user",
+ "llm.input_messages.1.message.content": "Hola, ΒΏcΓ³mo estΓ‘s?",
+ "llm.input_messages.2.message.role": "assistant",
+ "llm.input_messages.2.message.tool_calls.0.tool_call.id":
+ "call_i81vsF01rKH8cDZ0SLQQBUA2",
+ "llm.input_messages.2.message.tool_calls.0.tool_call.function.name":
+ "transfer_to_spanish_agent",
+ "llm.input_messages.3.message.role": "tool",
+ "llm.input_messages.3.message.tool_call_id":
+ "call_i81vsF01rKH8cDZ0SLQQBUA2",
+ "llm.input_messages.3.message.content":
+ '{"assistant": "Spanish agent"}',
+ "openinference.span.kind": "LLM",
+ }),
+ },
+ provided_model_name: "gpt-4.1-2025-04-14",
+ input: JSON.stringify([
+ {
+ content: "Hola, ΒΏcΓ³mo estΓ‘s?",
+ role: "user",
+ },
+ {
+ arguments: "{}",
+ call_id: "call_i81vsF01rKH8cDZ0SLQQBUA2",
+ name: "transfer_to_spanish_agent",
+ type: "function_call",
+ id: "fc_000230c027f4a45e0068db9cf8930481a3970aa5df55a731e3",
+ status: "completed",
+ },
+ {
+ call_id: "call_i81vsF01rKH8cDZ0SLQQBUA2",
+ output: '{"assistant": "Spanish agent"}',
+ type: "function_call_output",
+ },
+ ]),
+ output: JSON.stringify({
+ id: "resp_000230c027f4a45e0068db9cf8d9f481a380b1ab4e2f52dd8a",
+ created_at: 1759223033,
+ error: null,
+ incomplete_details: null,
+ instructions: "You only speak Spanish.",
+ metadata: {},
+ model: "gpt-4.1-2025-04-14",
+ object: "response",
+ output: [
+ {
+ id: "msg_000230c027f4a45e0068db9cf9d14481a3a04289f3f1b0f9a1",
+ content: [
+ {
+ annotations: [],
+ text: "Β‘Hola! Estoy muy bien, gracias. ΒΏY tΓΊ, cΓ³mo estΓ‘s?",
+ type: "output_text",
+ logprobs: [],
+ },
+ ],
+ role: "assistant",
+ status: "completed",
+ type: "message",
+ },
+ ],
+ parallel_tool_calls: true,
+ temperature: 1,
+ tool_choice: "auto",
+ tools: [],
+ top_p: 1,
+ background: false,
+ conversation: null,
+ max_output_tokens: null,
+ max_tool_calls: null,
+ previous_response_id: null,
+ prompt: null,
+ prompt_cache_key: null,
+ reasoning: {
+ effort: null,
+ generate_summary: null,
+ summary: null,
+ },
+ safety_identifier: null,
+ service_tier: "default",
+ status: "completed",
+ text: {
+ format: {
+ type: "text",
+ },
+ verbosity: "medium",
+ },
+ top_logprobs: 0,
+ truncation: "disabled",
+ usage: {
+ input_tokens: 54,
+ input_tokens_details: {
+ cached_tokens: 0,
+ },
+ output_tokens: 17,
+ output_tokens_details: {
+ reasoning_tokens: 0,
+ },
+ total_tokens: 71,
+ },
+ user: null,
+ billing: {
+ payer: "developer",
+ },
+ store: true,
+ }),
+ },
+ ],
+ expected: undefined,
+} satisfies TranscriptFixture;
diff --git a/packages/shared/src/server/transcript/fixtures/trace/support-copilot-refund-loop.ts b/packages/shared/src/server/transcript/fixtures/trace/support-copilot-refund-loop.ts
new file mode 100644
index 0000000000..694957bcbd
--- /dev/null
+++ b/packages/shared/src/server/transcript/fixtures/trace/support-copilot-refund-loop.ts
@@ -0,0 +1,356 @@
+import type { TranscriptFixture } from "../fixture-types";
+
+/**
+ * Support copilot resolving a duplicate-charge refund. Seeded scenario
+ * (`support-agent-seed`), authored as ClickHouse `observations` seed records
+ * for the shared `createObservation` test factory.
+ * url: https://cloud.langfuse.com/project/clkpwwm0m000gmm094odg11gi/traces/sa2026083102o?observation=sa2026083102o-cls&timestamp=2026-08-31T12:33:16.561Z&traceId=sa2026083102o
+ *
+ * Trace tree (export order differs: the AGENT is listed before its parent SPAN)
+ *
+ * [SPAN] support-copilot t-sa2026083102o root
+ * [AGENT] support-copilot sa2026083102o-root
+ * [GUARDRAIL] guardrail.input -gin
+ * [GENERATION] classify-intent gpt-5.4-mini -cls in: system + user out: JSON object (intent)
+ * [SPAN] load-context -load
+ * [TOOL] crm.get-customer -crm
+ * [TOOL] billing.list-invoices -bill
+ * [TOOL] tickets.search -tix
+ * [GENERATION] llm.chat gpt-5.4 -llm1 in: system + user out: tool_call stripe_find_charges
+ * [TOOL] stripe.find-charges -find in: = llm1 call args out: charges + duplicate_confidence
+ * [GENERATION] llm.chat gpt-5.4 -llm2 in: system + tool(partial result, no tool_call_id)
+ * out: tool_call stripe_create_refund
+ * [TOOL] stripe.create-refund -ref in: = llm2 call args out: refund_id
+ * [GENERATION] llm.chat gpt-5.4 -llm3 in: system only out: "Resolution complete."
+ * [GENERATION] draft-response gpt-5.4 -drf in: user "Draft the reply." out: JSON object {reply}
+ * [GUARDRAIL] guardrail.output -gout
+ * [TOOL] zendesk.send-reply -send
+ */
+
+const traceId = "sa2026083102o";
+const projectId = "sa2026083102o-project";
+const rootSpanId = "t-sa2026083102o";
+const agentId = "sa2026083102o-root";
+const loadContextId = "sa2026083102o-load";
+
+const customerId = "cus_LqT4v8";
+const refundId = "re_18Lk1Q";
+const findChargesToolCallId = "call_find_2";
+const createRefundToolCallId = "call_act_2";
+
+const customerMessage =
+ "Hi β€” I was charged twice for my subscription this month (invoices inv_20260709 and inv_20260709-2, $99 each). Can you refund the duplicate?";
+const copilotSystemPrompt =
+ "You are Acme's support copilot. Resolve the customer's billing issue end-to-end using the available tools.";
+const finalReply =
+ "Hi Jordan, thanks for flagging this! I've confirmed the duplicate $99.00 charge and issued a refund (re_18Lk1Q). You should see it back on your card within 5–10 business days.";
+
+// Tool-call arguments as the model emitted them (a JSON string inside the
+// chat-completions payload) and as the TOOL observation received them.
+const findChargesArguments = { customer_id: customerId, period: "2026-07" };
+const createRefundArguments = { charge_id: "ch_3PqK9b", reason: "duplicate" };
+
+const metadata = {
+ channel: "in-app-chat",
+ customer_id: customerId,
+ kind: "ok",
+ scenario: "support-agent-seed",
+};
+
+const common = {
+ trace_id: traceId,
+ project_id: projectId,
+ metadata,
+};
+
+const observations = [
+ {
+ ...common,
+ metadata: { ...metadata, plan: "pro" },
+ id: agentId,
+ parent_observation_id: rootSpanId,
+ type: "AGENT",
+ name: "support-copilot",
+ provided_model_name: null,
+ start_time: "2026-08-31T12:33:14.000Z",
+ end_time: "2026-08-31T12:33:25.052Z",
+ input: JSON.stringify({
+ message: customerMessage,
+ customer_id: customerId,
+ }),
+ output: JSON.stringify({
+ reply: finalReply,
+ resolution: "refund_issued",
+ refund_id: refundId,
+ }),
+ },
+ {
+ ...common,
+ id: rootSpanId,
+ parent_observation_id: null,
+ type: "SPAN",
+ name: "support-copilot",
+ provided_model_name: null,
+ start_time: "2026-08-31T12:33:14.000Z",
+ end_time: null,
+ input: JSON.stringify({
+ message: customerMessage,
+ customer_id: customerId,
+ }),
+ output: JSON.stringify({
+ reply: finalReply,
+ resolution: "refund_issued",
+ refund_id: refundId,
+ }),
+ },
+ {
+ ...common,
+ id: "sa2026083102o-gin",
+ parent_observation_id: agentId,
+ type: "GUARDRAIL",
+ name: "guardrail.input",
+ provided_model_name: null,
+ start_time: "2026-08-31T12:33:14.045Z",
+ end_time: "2026-08-31T12:33:14.168Z",
+ input: JSON.stringify({
+ text: customerMessage,
+ checks: ["prompt_injection", "pii"],
+ }),
+ output: JSON.stringify({ verdict: "pass", flags: [] }),
+ },
+ {
+ ...common,
+ id: "sa2026083102o-cls",
+ parent_observation_id: agentId,
+ type: "GENERATION",
+ name: "classify-intent",
+ provided_model_name: "gpt-5.4-mini",
+ start_time: "2026-08-31T12:33:14.205Z",
+ end_time: "2026-08-31T12:33:14.818Z",
+ input: JSON.stringify({
+ messages: [
+ { role: "system", content: "Classify the support request." },
+ { role: "user", content: customerMessage },
+ ],
+ }),
+ output: JSON.stringify({
+ intent: "billing.duplicate_charge",
+ urgency: "medium",
+ sentiment: "frustrated",
+ }),
+ },
+ {
+ ...common,
+ id: loadContextId,
+ parent_observation_id: agentId,
+ type: "SPAN",
+ name: "load-context",
+ provided_model_name: null,
+ start_time: "2026-08-31T12:33:14.858Z",
+ end_time: "2026-08-31T12:33:15.512Z",
+ input: JSON.stringify({ customer_id: customerId }),
+ output: JSON.stringify({ sources: ["crm", "billing", "tickets"] }),
+ },
+ {
+ ...common,
+ id: "sa2026083102o-crm",
+ parent_observation_id: loadContextId,
+ type: "TOOL",
+ name: "crm.get-customer",
+ provided_model_name: null,
+ start_time: "2026-08-31T12:33:14.881Z",
+ end_time: "2026-08-31T12:33:15.129Z",
+ input: JSON.stringify({ customer_id: customerId }),
+ output: JSON.stringify({ name: "Jordan Rivera", plan: "pro", mrr_usd: 99 }),
+ },
+ {
+ ...common,
+ id: "sa2026083102o-bill",
+ parent_observation_id: loadContextId,
+ type: "TOOL",
+ name: "billing.list-invoices",
+ provided_model_name: null,
+ start_time: "2026-08-31T12:33:14.886Z",
+ end_time: "2026-08-31T12:33:15.994Z",
+ input: JSON.stringify({
+ customer_id: customerId,
+ period: "2026-06..2026-07",
+ }),
+ output: JSON.stringify({
+ invoices: [{ id: "inv_20260709" }, { id: "inv_20260709-2" }],
+ }),
+ },
+ {
+ ...common,
+ id: "sa2026083102o-tix",
+ parent_observation_id: loadContextId,
+ type: "TOOL",
+ name: "tickets.search",
+ provided_model_name: null,
+ start_time: "2026-08-31T12:33:14.893Z",
+ end_time: "2026-08-31T12:33:15.263Z",
+ input: JSON.stringify({ query: "duplicate charge cus_LqT4v8" }),
+ output: JSON.stringify({ hits: 1 }),
+ },
+ {
+ ...common,
+ id: "sa2026083102o-llm1",
+ parent_observation_id: agentId,
+ type: "GENERATION",
+ name: "llm.chat",
+ provided_model_name: "gpt-5.4",
+ start_time: "2026-08-31T12:33:16.561Z",
+ end_time: "2026-08-31T12:33:17.389Z",
+ input: JSON.stringify({
+ messages: [
+ { role: "system", content: copilotSystemPrompt },
+ { role: "user", content: customerMessage },
+ ],
+ }),
+ output: JSON.stringify({
+ content: null,
+ tool_calls: [
+ {
+ id: findChargesToolCallId,
+ type: "function",
+ function: {
+ name: "stripe_find_charges",
+ arguments: JSON.stringify(findChargesArguments),
+ },
+ },
+ ],
+ }),
+ },
+ {
+ ...common,
+ id: "sa2026083102o-find",
+ parent_observation_id: agentId,
+ type: "TOOL",
+ name: "stripe.find-charges",
+ provided_model_name: null,
+ start_time: "2026-08-31T12:33:17.441Z",
+ end_time: "2026-08-31T12:33:18.118Z",
+ input: JSON.stringify(findChargesArguments),
+ output: JSON.stringify({
+ charges: [{ id: "ch_3PqK8r" }, { id: "ch_3PqK9b" }],
+ duplicate_confidence: 0.98,
+ }),
+ },
+ {
+ ...common,
+ id: "sa2026083102o-llm2",
+ parent_observation_id: agentId,
+ type: "GENERATION",
+ name: "llm.chat",
+ provided_model_name: "gpt-5.4",
+ start_time: "2026-08-31T12:33:18.172Z",
+ end_time: "2026-08-31T12:33:19.924Z",
+ input: JSON.stringify({
+ messages: [
+ { role: "system", content: copilotSystemPrompt },
+ // Partial tool result, no tool_call_id: the full result lives only on
+ // the sibling TOOL observation above.
+ {
+ role: "tool",
+ content: JSON.stringify({ duplicate_confidence: 0.98 }),
+ },
+ ],
+ }),
+ output: JSON.stringify({
+ content: null,
+ tool_calls: [
+ {
+ id: createRefundToolCallId,
+ type: "function",
+ function: {
+ name: "stripe_create_refund",
+ arguments: JSON.stringify(createRefundArguments),
+ },
+ },
+ ],
+ }),
+ },
+ {
+ ...common,
+ id: "sa2026083102o-ref",
+ parent_observation_id: agentId,
+ type: "TOOL",
+ name: "stripe.create-refund",
+ provided_model_name: null,
+ start_time: "2026-08-31T12:33:19.978Z",
+ end_time: "2026-08-31T12:33:20.893Z",
+ input: JSON.stringify(createRefundArguments),
+ output: JSON.stringify({
+ refund_id: refundId,
+ status: "succeeded",
+ amount_usd: 99,
+ }),
+ },
+ {
+ ...common,
+ id: "sa2026083102o-llm3",
+ parent_observation_id: agentId,
+ type: "GENERATION",
+ name: "llm.chat",
+ provided_model_name: "gpt-5.4",
+ start_time: "2026-08-31T12:33:20.947Z",
+ end_time: "2026-08-31T12:33:22.731Z",
+ input: JSON.stringify({
+ messages: [{ role: "system", content: copilotSystemPrompt }],
+ }),
+ output: JSON.stringify({ content: "Resolution complete.", tool_calls: [] }),
+ },
+ {
+ ...common,
+ id: "sa2026083102o-drf",
+ parent_observation_id: agentId,
+ type: "GENERATION",
+ name: "draft-response",
+ provided_model_name: "gpt-5.4",
+ start_time: "2026-08-31T12:33:22.790Z",
+ end_time: "2026-08-31T12:33:24.377Z",
+ input: JSON.stringify({
+ messages: [{ role: "user", content: "Draft the reply." }],
+ }),
+ output: JSON.stringify({ reply: finalReply }),
+ },
+ {
+ ...common,
+ id: "sa2026083102o-gout",
+ parent_observation_id: agentId,
+ type: "GUARDRAIL",
+ name: "guardrail.output",
+ provided_model_name: null,
+ start_time: "2026-08-31T12:33:24.426Z",
+ end_time: "2026-08-31T12:33:24.557Z",
+ input: JSON.stringify({ text: finalReply }),
+ output: JSON.stringify({ verdict: "pass" }),
+ },
+ {
+ ...common,
+ id: "sa2026083102o-send",
+ parent_observation_id: agentId,
+ type: "TOOL",
+ name: "zendesk.send-reply",
+ provided_model_name: null,
+ start_time: "2026-08-31T12:33:24.609Z",
+ end_time: "2026-08-31T12:33:24.881Z",
+ input: JSON.stringify({ thread_id: "thread_7Hf3kX", body: finalReply }),
+ output: JSON.stringify({ message_id: "msg_9uTb4w", status: "sent" }),
+ },
+];
+
+export const supportCopilotRefundLoopFixture = {
+ name: "support copilot refund loop with tool results only on TOOL observations",
+ scope: "trace",
+ description: [
+ "Five generations under one AGENT, none of which replays earlier assistant output in its input.",
+ "Tool results exist only as sibling TOOL observations whose input equals the preceding generation's tool-call arguments;",
+ "the following generation carries a partial result in a `tool` message without a tool_call_id.",
+ "`classify-intent` and `draft-response` output bare JSON objects rather than messages.",
+ "The same system message repeats across the three `llm.chat` generations.",
+ "Non-contributing types (SPAN, GUARDRAIL) are present, and the export lists the AGENT before its parent SPAN.",
+ ].join(" "),
+ observations,
+ expected: undefined,
+} satisfies TranscriptFixture;
diff --git a/packages/shared/src/server/transcript/fixtures/trace/vercel-ai-sdk-docs-tool-loop.ts b/packages/shared/src/server/transcript/fixtures/trace/vercel-ai-sdk-docs-tool-loop.ts
new file mode 100644
index 0000000000..9e4bb5a282
--- /dev/null
+++ b/packages/shared/src/server/transcript/fixtures/trace/vercel-ai-sdk-docs-tool-loop.ts
@@ -0,0 +1,706 @@
+import type { TranscriptFixture } from "../fixture-types";
+
+/**
+ * Two model steps with a documentation tool call and replayed history. The export also includes a GENERATION-typed application wrapper; its type and all payloads are preserved rather than adjusted to match the design example.
+ * Export order is preserved below.
+ *
+ * [GENERATION] handle-chatbot-message 6aa131f38d871f39 (parent outside export)
+ * [SPAN] get-langfuse-prompt 5b9a45797a30e758
+ * [SPAN] create-mcp-client 2f957a45eec7d164
+ * [AGENT] invoke_agent gpt-5 dfae2f190648ad80
+ * [SPAN] step 1 b8553aadce2f531c
+ * [GENERATION] chat gpt-5 1bbf3edb8ef95ab6
+ * [TOOL] searchLangfuseDocs 3dbf50cbf2036387
+ * [SPAN] step 2 91eef3e3dede80b4
+ * [GENERATION] chat gpt-5 0c13b686b4cf6ccc
+ */
+
+const traceId = "75b336f98e39e275a823b0a7ec1ba794";
+const projectId = "transcript-fixture-project";
+
+const payload1 = "How do I attach a user score to a trace?";
+
+const payload2 =
+ 'You can add a user score to a trace via the SDKs or API by creating a Score linked to the traceId (optionally to a specific observationId). For thumbs up/down, use dataType "BOOLEAN" with value 1 or 0.\n\nJS/TS (server)\n- Docs: https://langfuse.com/docs/evaluation/evaluation-methods/scores-via-sdk\n- Example:\n - npm i @langfuse/client\n - \n import { LangfuseClient } from "@langfuse/client";\n const langfuse = new LangfuseClient();\n await langfuse.score.create({\n traceId: "<trace_id>",\n name: "user-feedback",\n value: 1, // 1 = up, 0 = down\n dataType: "BOOLEAN", // required for boolean\n comment: "Helpful answer", // optional\n });\n await langfuse.flush();\n\nBrowser (collect in-app feedback)\n- Docs: https://langfuse.com/docs/evaluation/evaluation-methods/scores-via-sdk\n- Example:\n - npm i @langfuse/browser\n - \n import { LangfuseBrowserClient } from "@langfuse/browser";\n const lf = new LangfuseBrowserClient({ publicKey: "<public_key>", baseUrl: "https://cloud.langfuse.com" });\n await lf.score({\n traceId: "<trace_id>",\n // observationId: "<generation_or_span_id>", // optional if you want to score a specific step\n id: `user-feedback-<trace_id>`, // optional idempotency key\n name: "user-feedback",\n value: 1,\n dataType: "BOOLEAN",\n comment: "Great result!",\n });\n\nPython\n- Docs: https://langfuse.com/docs/evaluation/evaluation-methods/scores-via-sdk\n- Example:\n - \n from langfuse import get_client\n lf = get_client()\n lf.create_score(\n name="user-feedback",\n value=1.0, # float\n data_type="BOOLEAN", # required for boolean\n trace_id="<trace_id>",\n # observation_id="<observation_id>", # optional\n comment="User liked it", # optional\n )\n\nNotes\n- You only need traceId to attach to a trace; observationId is optional.\n- You can send the score before the trace exists; it links once the trace is created.\n- Score types supported: NUMERIC, CATEGORICAL, BOOLEAN (0/1), TEXT.\n- More: Scores overview: https://langfuse.com/docs/evaluation/scores/overview\n\nWant an example of passing traceId/observationId from your backend to the frontend for a β€œthumbs up/down” widget?';
+
+const payload3 = JSON.stringify([
+ "langfuse-docs-assistant-chat",
+ {
+ type: "chat",
+ },
+]);
+
+const payload4 = JSON.stringify(
+ '{"name":"langfuse-docs-assistant-chat","prompt":[{"role":"system","content":"You are a helpful Langfuse support agent. Your goal: give the correct, actionable answer with links and minimal, runnable examples.\\n\\n## Guidelines\\n- Make extensive use of the tools provided to you in order to search the langfuse documentation.\\n- Getting an overview of the docs is almost always a good idea for on-topic questions.\\n- Answer the question using only this and only this information, outputted in markdown format.\\n- Before using any tools, reason about what kinds of tools make the most sense to be helpful.\\n- After tool responses, reason about whether the information is sufficient to answer a question; if not, use the tools again or give up and state that you don\'t know.\\n- If the question is broad or conceptual, stay at the conceptual level. If needed, ask one high-level clarifying question only if it meaningfully sharpens the user’s intent. Avoid jumping into implementation details too soon (e.g., SDK choice) unless the user’s question is already about implementation.\\n- Unless you asked a clarifying question, end your answer with one suggested follow-up. Think like the user and propose the next logical step that would improve their developer experience β€” whether that’s diving deeper into code, pointing to a related concept, or highlighting a useful best practice or feature. Keep it to one sentence with a natural, helpful tone, concise"},{"name":"chat_history","type":"placeholder"}],"version":26,"isFallback":false,"tags":["general-qa-experiment"],"labels":["latest","production"],"type":"chat","config":{"model":"gpt-5","reasoningSummary":"detailed","textVerbosity":"low","reasoningEffort":"low"}}',
+);
+
+const payload5 = null;
+
+const payload6 = JSON.stringify([
+ {
+ role: "system",
+ content:
+ "You are a helpful Langfuse support agent. Your goal: give the correct, actionable answer with links and minimal, runnable examples.\n\n## Guidelines\n- Make extensive use of the tools provided to you in order to search the langfuse documentation.\n- Getting an overview of the docs is almost always a good idea for on-topic questions.\n- Answer the question using only this and only this information, outputted in markdown format.\n- Before using any tools, reason about what kinds of tools make the most sense to be helpful.\n- After tool responses, reason about whether the information is sufficient to answer a question; if not, use the tools again or give up and state that you don't know.\n- If the question is broad or conceptual, stay at the conceptual level. If needed, ask one high-level clarifying question only if it meaningfully sharpens the user’s intent. Avoid jumping into implementation details too soon (e.g., SDK choice) unless the user’s question is already about implementation.\n- Unless you asked a clarifying question, end your answer with one suggested follow-up. Think like the user and propose the next logical step that would improve their developer experience β€” whether that’s diving deeper into code, pointing to a related concept, or highlighting a useful best practice or feature. Keep it to one sentence with a natural, helpful tone, concise",
+ },
+ {
+ role: "user",
+ parts: [
+ {
+ type: "text",
+ content: "How do I attach a user score to a trace?",
+ },
+ ],
+ },
+]);
+
+const payload7 = JSON.stringify([
+ {
+ role: "assistant",
+ parts: [
+ {
+ type: "reasoning",
+ content:
+ "**Crafting user score attachment instructions**\n\nI've gathered enough info to help the user with attaching a user score to a trace. They want minimal examples in TypeScript, Python, and a Browser client. I think providing a basic snippet that adds a boolean score named \"user-feedback\" to the traceId would be effective. It's also essential to note that the data type BOOLEAN requires values of 0 or 1. I’ll include concise runnable examples and useful links about scores via API/SDK and user feedback. We should also consider concluding with a follow-up question.",
+ },
+ {
+ type: "reasoning",
+ content:
+ "**Creating minimal code examples**\n\nI want to create concise code samples while mentioning that you need a traceId, and optionally an observationId. It's important to highlight that you can send user feedback even before the trace exists, which is quite interesting! This way, the user gets a clear understanding while also seeing how to implement everything in a few lines of code. Let's keep this straightforward and focus on clarity in the examples!",
+ },
+ {
+ type: "text",
+ content:
+ 'You can add a user score to a trace via the SDKs or API by creating a Score linked to the traceId (optionally to a specific observationId). For thumbs up/down, use dataType "BOOLEAN" with value 1 or 0.\n\nJS/TS (server)\n- Docs: https://langfuse.com/docs/evaluation/evaluation-methods/scores-via-sdk\n- Example:\n - npm i @langfuse/client\n - \n import { LangfuseClient } from "@langfuse/client";\n const langfuse = new LangfuseClient();\n await langfuse.score.create({\n traceId: "<trace_id>",\n name: "user-feedback",\n value: 1, // 1 = up, 0 = down\n dataType: "BOOLEAN", // required for boolean\n comment: "Helpful answer", // optional\n });\n await langfuse.flush();\n\nBrowser (collect in-app feedback)\n- Docs: https://langfuse.com/docs/evaluation/evaluation-methods/scores-via-sdk\n- Example:\n - npm i @langfuse/browser\n - \n import { LangfuseBrowserClient } from "@langfuse/browser";\n const lf = new LangfuseBrowserClient({ publicKey: "<public_key>", baseUrl: "https://cloud.langfuse.com" });\n await lf.score({\n traceId: "<trace_id>",\n // observationId: "<generation_or_span_id>", // optional if you want to score a specific step\n id: `user-feedback-<trace_id>`, // optional idempotency key\n name: "user-feedback",\n value: 1,\n dataType: "BOOLEAN",\n comment: "Great result!",\n });\n\nPython\n- Docs: https://langfuse.com/docs/evaluation/evaluation-methods/scores-via-sdk\n- Example:\n - \n from langfuse import get_client\n lf = get_client()\n lf.create_score(\n name="user-feedback",\n value=1.0, # float\n data_type="BOOLEAN", # required for boolean\n trace_id="<trace_id>",\n # observation_id="<observation_id>", # optional\n comment="User liked it", # optional\n )\n\nNotes\n- You only need traceId to attach to a trace; observationId is optional.\n- You can send the score before the trace exists; it links once the trace is created.\n- Score types supported: NUMERIC, CATEGORICAL, BOOLEAN (0/1), TEXT.\n- More: Scores overview: https://langfuse.com/docs/evaluation/scores/overview\n\nWant an example of passing traceId/observationId from your backend to the frontend for a β€œthumbs up/down” widget?',
+ },
+ {
+ type: "tool_call",
+ id: "call_OQNWWW4PjixkLIhkuHJRwiFV",
+ name: "searchLangfuseDocs",
+ arguments: {
+ query:
+ "How to attach a score to a trace user score to a trace Langfuse SDK example",
+ },
+ },
+ ],
+ finish_reason: "stop",
+ },
+]);
+
+const payload8 = JSON.stringify({
+ messages: [
+ {
+ role: "user",
+ parts: [
+ {
+ type: "text",
+ content: "How do I attach a user score to a trace?",
+ },
+ ],
+ },
+ ],
+ tools: [
+ {
+ type: "function",
+ name: "submitFeedback",
+ inputSchema: {
+ type: "object",
+ properties: {
+ targetType: {
+ type: "string",
+ enum: ["skill", "mcp-tool", "cli", "docs", "public-api", "other"],
+ description: "Category of the thing the feedback is about.",
+ },
+ target: {
+ type: "string",
+ minLength: 1,
+ maxLength: 200,
+ description:
+ "The specific instance within targetType: the skill name, MCP tool name, CLI command, API endpoint path, or docs page path (e.g. 'queryMetrics', '/docs/mcp'). An identifier, not a sentence.",
+ },
+ feedback: {
+ type: "string",
+ minLength: 1,
+ maxLength: 3000,
+ },
+ goal: {
+ type: "string",
+ minLength: 1,
+ maxLength: 1500,
+ },
+ referenceUrl: {
+ type: "string",
+ format: "uri",
+ maxLength: 2048,
+ },
+ },
+ required: ["targetType", "target", "feedback"],
+ additionalProperties: false,
+ $schema: "http://json-schema.org/draft-07/schema#",
+ },
+ description:
+ "Submit explicit user-approved feedback to the Langfuse team.\nBefore calling, ask permission and show the exact payload.\nIf the user wants a reply, ask them to include their email address in the feedback text; only use an address they explicitly provide and show it in the exact payload preview.\nDo not include secrets, credentials, customer/project data, trace payloads, or unrelated context; the only contact detail to include is an explicitly provided reply email.",
+ },
+ {
+ type: "function",
+ name: "searchLangfuseDocs",
+ inputSchema: {
+ type: "object",
+ properties: {
+ query: {
+ type: "string",
+ description:
+ "The user's question in natural language. Include helpful context like SDK/language (e.g., Python v3, JS v4), self-hosted vs cloud, and short error messages (trim long stack traces). Keep under ~600 characters.",
+ },
+ },
+ required: ["query"],
+ additionalProperties: false,
+ $schema: "http://json-schema.org/draft-07/schema#",
+ },
+ description:
+ "Semantic search (RAG) over the Langfuse documentation. Use this whenever the user asks a broader question that cannot be answered by a specific single page. Returns a concise answer synthesized from relevant docs. The raw provider response is included in _meta. Prefer this before guessing. If a specific page is needed call getLangfuseDocsPage first.",
+ },
+ {
+ type: "function",
+ name: "getLangfuseDocsPage",
+ inputSchema: {
+ type: "object",
+ properties: {
+ pathOrUrl: {
+ type: "string",
+ description:
+ 'Docs path starting with "/" (e.g., /docs/observability/overview) or a full URL on https://langfuse.com. Do not include anchors (#...) or queries (?foo=bar) β€” they will be ignored.',
+ },
+ },
+ required: ["pathOrUrl"],
+ additionalProperties: false,
+ $schema: "http://json-schema.org/draft-07/schema#",
+ },
+ description:
+ "Fetch the raw Markdown for a single Langfuse docs page. Accepts a docs path (e.g., /docs/observability/overview) or a full https://langfuse.com URL. Returns the exact Markdown (may include front matter). Use when you need a specific page content (Integration, Features, API, etc.) or code samples. Prefer searchLangfuseDocs for broader questions where there is not one specific page about it. Changelog pages (/changelog/...) are historical release notes: use them only to confirm that a feature exists or when it shipped, not as an implementation reference. Their code samples reflect the SDK/API at release time and may be outdated, so for implementation follow the current docs and the API/SDK reference instead.",
+ },
+ {
+ type: "function",
+ name: "getLangfuseOverview",
+ inputSchema: {
+ type: "object",
+ properties: {},
+ additionalProperties: false,
+ $schema: "http://json-schema.org/draft-07/schema#",
+ },
+ description:
+ "Get a high-level, machine-readable index by downloading https://langfuse.com/llms.txt. Use this at the start of a session when needed to discover key docs endpoints or to seed follow-up calls to searchLangfuseDocs or getLangfuseDocsPage. Returns the plain text contents of llms.txt. Avoid repeated calls within the same session.",
+ },
+ ],
+});
+
+const payload9 = JSON.stringify([
+ {
+ role: "assistant",
+ parts: [
+ {
+ type: "reasoning",
+ content:
+ '**Searching documents for scores**\n\nI need to use tools to search the documentation. It looks like Langfuse has a "scores" API that can be linked to traces or observations. There are "Score" entities to consider, which include details like name, value, data type, comments, source, traceId, observationId, and timestamp. The SDKs have methods such as client.score or langfuse.score. I plan to search Langfuse docs to get more information on this.',
+ },
+ {
+ type: "text",
+ content:
+ "I’m going to search the Langfuse docs for how to add or attach a score to a trace, including minimal code examples.",
+ },
+ {
+ type: "tool_call",
+ id: "call_OQNWWW4PjixkLIhkuHJRwiFV",
+ name: "searchLangfuseDocs",
+ arguments: {
+ query:
+ "How to attach a score to a trace user score to a trace Langfuse SDK example",
+ },
+ },
+ ],
+ finish_reason: "tool_call",
+ },
+]);
+
+const payload10 = JSON.stringify({
+ query:
+ "How to attach a score to a trace user score to a trace Langfuse SDK example",
+});
+
+const payload11 = JSON.stringify({
+ _meta: {
+ id: "inkeep-chatcmpl-ef102f93-4b30-4232-b714-78a368f5c654",
+ model: "inkeep-rag",
+ choices: [
+ {
+ finish_reason: "stop",
+ log_probs: null,
+ index: 0,
+ message: {
+ content:
+ '{"content": [{"type": "document", "record_type": "documentation", "url": "https://langfuse.com/docs/evaluation/evaluation-methods/scores-via-sdk", "title": "Scores via API/SDK", "source": {"content": [{"type": "text", "text": "Scores via API/SDK - Langfuse\\n[Docs](/docs)\\n[Evaluation](/docs/evaluation/overview)\\n[Evaluation Methods](/docs/evaluation/evaluation-methods/llm-as-a-judge)\\n[Scores via API/SDK](/docs/evaluation/evaluation-methods/scores-via-sdk)\\n# Scores via API/SDK\\nYou can use the Langfuse SDKs or API to add [scores](/docs/evaluation/scores/overview) to traces, observations, sessions and dataset runs. This is an evaluation method that allows you to set up custom evaluation workflows and extend the scoring capabilities of Langfuse. See the [data model](/docs/evaluation/scores/data-model#scores) for full details on the score object.\\nIf you want Langfuse to run deterministic Python or TypeScript logic for you, use [code evaluators](/docs/evaluation/evaluation-methods/code-evaluators). Use this page when your application, pipeline, or CI job computes scores and sends them to Langfuse via API or SDK.\\n## Ingesting Scores via API/SDKs\\nScores can be attached at different levels of granularity: to individual traces, to specific observations within a trace, or to full sessions.\\nSee the [API reference](/docs/api) for full details on the POST and GET endpoints for scores and score configs.\\n### Trace or Observation-level Scores\\nYou can add scores via the Langfuse SDKs or API. Scores can take one of four data types: **Numeric** , **Categorical** , **Boolean** , or **Text**. See [Score Types](/docs/evaluation/scores/overview#score-types) for details.\\nIf a score is ingested manually using a `trace_id` to link the score to a trace, it is not necessary to wait until the trace has been created. The score will show up in the scores table and will be linked to the trace once the trace with the same `trace_id` is created.\\nHere are examples by `Score` data types."}, {"type": "text", "text": "\\nFor trace and observation scores, `trace_id`/`traceId` is required and `observation_id`/`observationId` is optional. If you attach a score to an observation, always provide both the observation ID and the corresponding trace ID.\\nPython SDKJS/TS SDKAPI\\nNumericCategoricalBooleanText\\nNumeric score values must be provided as float.\\n \\n \\n```\\nfrom langfuse import get_client\\nlangfuse = get_client()\\n///# Method 1: Score via low-level method\\nlangfuse.create_score(\\n name=\\"correctness\\",\\n value=0.9,\\n trace_id=\\"trace_id_here\\",\\n observation_id=\\"observation_id_here\\", ///# optional\\n data_type=\\"NUMERIC\\", ///# optional, inferred if not provided\\n comment=\\"Factually correct\\", ///# optional\\n)\\n///# Method 2: Score current span/generation (within context)\\nwith langfuse.start_as_current_observation(as_type=\\"span\\", name=\\"my-operation\\") as span:\\n # Score the current span\\n span.score(\\n name=\\"correctness\\",\\n value=0.9,\\n data_type=\\"NUMERIC\\",\\n comment=\\"Factually correct\\"\\n )\\n # Score the trace\\n span.score_trace(\\n name=\\"overall_quality\\",\\n value=0.95,\\n data_type=\\"NUMERIC\\"\\n )\\n///# Method 3: Score via the current context\\nwith langfuse.start_as_current_observation(as_type=\\"span\\", name=\\"my-operation\\"):\\n # Score the current span\\n langfuse.score_current_span(\\n name=\\"correctness\\",\\n value=0.9,\\n data_type=\\"NUMERIC\\",\\n comment=\\"Factually correct\\"\\n )\\n # Score the trace\\n langfuse.score_current_trace(\\n name=\\"overall_quality\\",\\n value=0.95,\\n data_type=\\"NUMERIC\\"\\n )\\n```\\nCategorical score values must be provided as strings.\\n \\n \\n```\\nfrom langfuse import get_client\\nlangfuse = get_client()\\n///# Method 1: Score via low-level method\\nlangfuse.create_score(\\n name=\\"accuracy\\",\\n value=\\"partially correct\\",\\n trace_id=\\"trace_id_here\\",\\n observation_id=\\"observation_id_here\\", ///# optional\\n data_type=\\"CATEGORICAL\\", ///# optional, inferred if not provided\\n comment=\\"Some factual errors\\", ///# optional\\n)"}, {"type": "text", "text": "\\n///# Method 2: Score current span/generation (within context)\\nwith langfuse.start_as_current_observation(as_type=\\"span\\", name=\\"my-operation\\") as span:\\n # Score the current span\\n span.score(\\n name=\\"accuracy\\",\\n value=\\"partially correct\\",\\n data_type=\\"CATEGORICAL\\",\\n comment=\\"Some factual errors\\"\\n )\\n # Score the trace\\n span.score_trace(\\n name=\\"overall_quality\\",\\n value=\\"partially correct\\",\\n data_type=\\"CATEGORICAL\\"\\n )\\n///# Method 3: Score via the current context\\nwith langfuse.start_as_current_observation(as_type=\\"span\\", name=\\"my-operation\\"):\\n # Score the current span\\n langfuse.score_current_span(\\n name=\\"accuracy\\",\\n value=\\"partially correct\\",\\n data_type=\\"CATEGORICAL\\",\\n comment=\\"Some factual errors\\"\\n )\\n # Score the trace\\n langfuse.score_current_trace(\\n name=\\"overall_quality\\",\\n value=\\"partially correct\\",\\n data_type=\\"CATEGORICAL\\"\\n )\\n```\\nBoolean scores must be provided as a float, where `1` is true and `0` is false. When read via the [v3 scores API](https://api.reference.langfuse.com/#tag/scoresv3/GET/api/public/v3/scores), the value is returned as a boolean. See [API reference](/docs/api) for more details on POST/GET scores endpoints.\\n \\n \\n```\\nfrom langfuse import get_client\\nlangfuse = get_client()\\n///# Method 1: Score via low-level method\\nlangfuse.create_score(\\n name=\\"helpfulness\\",\\n value=0, ///# 0 or 1\\n trace_id=\\"trace_id_here\\",\\n observation_id=\\"observation_id_here\\", ///# optional\\n data_type=\\"BOOLEAN\\", ///# required, numeric values without data type would be inferred as NUMERIC\\n comment=\\"Incorrect answer\\", ///# optional\\n)\\n///# Method 2: Score current span/generation (within context)\\nwith langfuse.start_as_current_observation(as_type=\\"span\\", name=\\"my-operation\\") as span:\\n # Score the current span"}, {"type": "text", "text": "\\n span.score(\\n name=\\"helpfulness\\",\\n value=1, ///# 0 or 1\\n data_type=\\"BOOLEAN\\",\\n comment=\\"Very helpful response\\"\\n )\\n # Score the trace\\n span.score_trace(\\n name=\\"overall_quality\\",\\n value=1, ///# 0 or 1\\n data_type=\\"BOOLEAN\\"\\n )\\n///# Method 3: Score via the current context\\nwith langfuse.start_as_current_observation(as_type=\\"span\\", name=\\"my-operation\\"):\\n # Score the current span\\n langfuse.score_current_span(\\n name=\\"helpfulness\\",\\n value=1, ///# 0 or 1\\n data_type=\\"BOOLEAN\\",\\n comment=\\"Very helpful response\\"\\n )\\n # Score the trace\\n langfuse.score_current_trace(\\n name=\\"overall_quality\\",\\n value=1, ///# 0 or 1\\n data_type=\\"BOOLEAN\\"\\n )\\n```\\nText score values must be provided as strings between 1 and 500 characters.\\n \\n \\n```\\nfrom langfuse import get_client\\nlangfuse = get_client()\\n///# Method 1: Score via low-level method\\nlangfuse.create_score(\\n name=\\"reviewer_notes\\",\\n value=\\"The response was helpful but could be more concise.\\",\\n trace_id=\\"trace_id_here\\",\\n observation_id=\\"observation_id_here\\", ///# optional\\n data_type=\\"TEXT\\", ///# optional, inferred if not provided\\n comment=\\"Reviewed by QA team\\", ///# optional\\n)\\n///# Method 2: Score current span/generation (within context)\\nwith langfuse.start_as_current_observation(as_type=\\"span\\", name=\\"my-operation\\") as span:\\n # Score the current span\\n span.score(\\n name=\\"reviewer_notes\\",\\n value=\\"The response was helpful but could be more concise.\\",\\n data_type=\\"TEXT\\",\\n comment=\\"Reviewed by QA team\\"\\n )\\n # Score the trace\\n span.score_trace(\\n name=\\"overall_notes\\",\\n value=\\"Good quality overall, minor formatting issues.\\",\\n data_type=\\"TEXT\\"\\n )\\n///# Method 3: Score via the current context\\nwith langfuse.start_as_current_observation(as_type=\\"span\\", name=\\"my-operation\\"):\\n # Score the current span"}, {"type": "text", "text": "\\n langfuse.score_current_span(\\n name=\\"reviewer_notes\\",\\n value=\\"The response was helpful but could be more concise.\\",\\n data_type=\\"TEXT\\",\\n comment=\\"Reviewed by QA team\\"\\n )\\n # Score the trace\\n langfuse.score_current_trace(\\n name=\\"overall_notes\\",\\n value=\\"Good quality overall, minor formatting issues.\\",\\n data_type=\\"TEXT\\"\\n )\\n```\\nNumericCategoricalBooleanText\\nNumeric score values must be provided as float.\\n \\n \\n```\\nimport { LangfuseClient } from \\"@langfuse/client\\";\\nconst langfuse = new LangfuseClient();\\nlangfuse.score.create({\\n id: \\"unique_id\\", // optional, can be used as an idempotency key to update the score subsequently\\n traceId: message.traceId,\\n observationId: message.generationId, // optional\\n name: \\"correctness\\",\\n value: 0.9,\\n dataType: \\"NUMERIC\\", // optional, inferred if not provided\\n comment: \\"Factually correct\\", // optional\\n});\\n// Flush the scores in short-lived environments\\nawait langfuse.flush();\\n```\\nCategorical score values must be provided as strings.\\n \\n \\n```\\nimport { LangfuseClient } from \\"@langfuse/client\\";\\nconst langfuse = new LangfuseClient();\\nlangfuse.score.create({\\n id: \\"unique_id\\", // optional, can be used as an idempotency key to update the score subsequently\\n traceId: message.traceId,\\n observationId: message.generationId, // optional\\n name: \\"accuracy\\",\\n value: \\"partially correct\\",\\n dataType: \\"CATEGORICAL\\", // optional, inferred if not provided\\n comment: \\"Factually correct\\", // optional\\n});\\n// Flush the scores in short-lived environments\\nawait langfuse.flush();\\n```"}, {"type": "text", "text": "\\nBoolean scores must be provided as a float, where `1` is true and `0` is false. When read via the [v3 scores API](https://api.reference.langfuse.com/#tag/scoresv3/GET/api/public/v3/scores), the value is returned as a boolean. See [API reference](/docs/api) for more details on POST/GET scores endpoints.\\n \\n \\n```\\nimport { LangfuseClient } from \\"@langfuse/client\\";\\nconst langfuse = new LangfuseClient();\\nlangfuse.score.create({\\n id: \\"unique_id\\", // optional, can be used as an idempotency key to update the score subsequently\\n traceId: message.traceId,\\n observationId: message.generationId, // optional\\n name: \\"helpfulness\\",\\n value: 0, // 0 or 1\\n dataType: \\"BOOLEAN\\", // required, numeric values without data type would be inferred as NUMERIC\\n comment: \\"Incorrect answer\\", // optional\\n});\\n// Flush the scores in short-lived environments\\nawait langfuse.flush();\\n```\\nText score values must be provided as strings between 1 and 500 characters.\\n \\n \\n```\\nimport { LangfuseClient } from \\"@langfuse/client\\";\\nconst langfuse = new LangfuseClient();\\nlangfuse.score.create({\\n id: \\"unique_id\\", // optional, can be used as an idempotency key to update the score subsequently\\n traceId: message.traceId,\\n observationId: message.generationId, // optional\\n name: \\"reviewer_notes\\",\\n value: \\"The response was helpful but could be more concise.\\",\\n dataType: \\"TEXT\\", // optional, inferred if not provided\\n comment: \\"Reviewed by QA team\\", // optional\\n});\\n// Flush the scores in short-lived environments\\nawait langfuse.flush();\\n```"}, {"type": "text", "text": "\\nYou can also create scores directly via the [REST API](https://api.reference.langfuse.com/#tag/legacyscorev1/POST/api/public/scores). Authenticate using HTTP Basic Auth with your Langfuse Public Key as username and Secret Key as password.\\nNumericCategoricalBooleanText\\nNumeric score values must be provided as float.\\n \\n \\n```\\ncurl -X POST https://cloud.langfuse.com/api/public/scores \\\\\\n -u \\"pk-lf-...\\":\\"sk-lf-...\\" \\\\\\n -H \\"Content-Type: application/json\\" \\\\\\n -d \'{\\n \\"traceId\\": \\"trace_id_here\\",\\n \\"observationId\\": \\"observation_id_here\\",\\n \\"name\\": \\"correctness\\",\\n \\"value\\": 0.9,\\n \\"dataType\\": \\"NUMERIC\\",\\n \\"comment\\": \\"Factually correct\\"\\n }\'\\n```\\nCategorical score values must be provided as strings.\\n \\n \\n```\\ncurl -X POST https://cloud.langfuse.com/api/public/scores \\\\\\n -u \\"pk-lf-...\\":\\"sk-lf-...\\" \\\\\\n -H \\"Content-Type: application/json\\" \\\\\\n -d \'{\\n \\"traceId\\": \\"trace_id_here\\",\\n \\"observationId\\": \\"observation_id_here\\",\\n \\"name\\": \\"accuracy\\",\\n \\"value\\": \\"partially correct\\",\\n \\"dataType\\": \\"CATEGORICAL\\",\\n \\"comment\\": \\"Some factual errors\\"\\n }\'\\n```\\nBoolean scores must be provided as a float, where `1` is true and `0` is false. When read via the [v3 scores API](https://api.reference.langfuse.com/#tag/scoresv3/GET/api/public/v3/scores), the value is returned as a boolean.\\n \\n \\n```\\ncurl -X POST https://cloud.langfuse.com/api/public/scores \\\\\\n -u \\"pk-lf-...\\":\\"sk-lf-...\\" \\\\\\n -H \\"Content-Type: application/json\\" \\\\\\n -d \'{\\n \\"traceId\\": \\"trace_id_here\\",\\n \\"observationId\\": \\"observation_id_here\\",\\n \\"name\\": \\"helpfulness\\",\\n \\"value\\": 0,\\n \\"dataType\\": \\"BOOLEAN\\",\\n \\"comment\\": \\"Incorrect answer\\"\\n }\'\\n```\\nText score values must be provided as strings between 1 and 500 characters.\\n \\n \\n```\\ncurl -X POST https://cloud.langfuse.com/api/public/scores \\\\\\n -u \\"pk-lf-...\\":\\"sk-lf-...\\" \\\\"}, {"type": "text", "text": "\\n comment=\\"Factually correct\\",\\n config_id=\\"12345-6565-3453654-43543\\",\\n data_type=\\"CATEGORICAL\\"\\n )\\n```\\nWhen ingesting boolean scores, you can provide the value as a float. If you provide a configId, the score\'s name and config\'s name must match as well as their data types.\\n \\n \\n```\\nfrom langfuse import get_client\\nlangfuse = get_client()\\n///# Method 1: Score via low-level method\\nlangfuse.create_score(\\n trace_id=\\"trace_id_here\\",\\n observation_id=\\"observation_id_here\\", ///# optional\\n name=\\"helpfulness\\",\\n value=1,\\n comment=\\"Factually correct\\", ///# optional\\n score_id=\\"unique_id\\", ///# optional, can be used as an idempotency key to update the score subsequently\\n config_id=\\"93547-6565-3453654-43543\\", ///# optional, can be used to infer the score data type and validate the score value\\n data_type=\\"BOOLEAN\\" # optional, possibly inferred\\n)\\n///# Method 2: Score within context\\nwith langfuse.start_as_current_observation(as_type=\\"span\\", name=\\"my-operation\\") as span:\\n span.score(\\n name=\\"helpfulness\\",\\n value=1,\\n comment=\\"Factually correct\\",\\n config_id=\\"93547-6565-3453654-43543\\",\\n data_type=\\"BOOLEAN\\"\\n )\\n```\\nWhen ingesting text scores, provide the value as a string (1–500 characters). If you provide a configId, the score\'s name and config\'s name must match as well as their data types.\\n \\n \\n```\\nfrom langfuse import get_client\\nlangfuse = get_client()\\n///# Method 1: Score via low-level method\\nlangfuse.create_score(\\n trace_id=\\"trace_id_here\\",\\n observation_id=\\"observation_id_here\\", ///# optional\\n name=\\"reviewer_notes\\",\\n value=\\"The response was helpful but could be more concise.\\",\\n comment=\\"Reviewed by QA team\\", ///# optional"}, {"type": "text", "text": "\\n score_id=\\"unique_id\\", ///# optional, can be used as an idempotency key to update the score subsequently\\n config_id=\\"24680-6565-3453654-43543\\", ///# optional\\n data_type=\\"TEXT\\" # optional, possibly inferred\\n)\\n///# Method 2: Score within context\\nwith langfuse.start_as_current_observation(as_type=\\"span\\", name=\\"my-operation\\") as span:\\n span.score(\\n name=\\"reviewer_notes\\",\\n value=\\"The response was helpful but could be more concise.\\",\\n comment=\\"Reviewed by QA team\\",\\n config_id=\\"24680-6565-3453654-43543\\",\\n data_type=\\"TEXT\\"\\n )\\n```\\nNumeric ScoresCategorical ScoresBoolean ScoresText Scores\\nWhen ingesting numeric scores, you can provide the value as a float. If you provide a configId, the score value will be validated against the config\'s numeric range, which might be defined by a minimum and/or maximum value.\\n \\n \\n```\\nimport { LangfuseClient } from \\"@langfuse/client\\";\\nconst langfuse = new LangfuseClient();\\nlangfuse.score.create({\\n traceId: message.traceId,\\n observationId: message.generationId, // optional\\n name: \\"accuracy\\",\\n value: 0.9,\\n comment: \\"Factually correct\\", // optional\\n id: \\"unique_id\\", // optional, can be used as an idempotency key to update the score subsequently\\n configId: \\"78545-6565-3453654-43543\\", // optional, to ensure that the score follows a specific min/max value range\\n dataType: \\"NUMERIC\\", // optional, possibly inferred\\n});\\n// Flush the scores in short-lived environments\\nawait langfuse.flush();\\n```\\nCategorical scores are used to evaluate data that falls into specific categories. When ingesting categorical scores, you can provide the value as a string. If you provide a configId, the score value will be validated against the config\'s categories.\\n \\n \\n```"}, {"type": "text", "text": "\\nimport { LangfuseClient } from \\"@langfuse/client\\";\\nconst langfuse = new LangfuseClient();\\nlangfuse.score.create({\\n traceId: message.traceId,\\n observationId: message.generationId, // optional\\n name: \\"correctness\\",\\n value: \\"correct\\",\\n comment: \\"Factually correct\\", // optional\\n id: \\"unique_id\\", // optional, can be used as an idempotency key to update the score subsequently\\n configId: \\"12345-6565-3453654-43543\\", // optional, to ensure that a score maps to a specific category defined in a score config\\n dataType: \\"CATEGORICAL\\", // optional, possibly inferred\\n});\\n// Flush the scores in short-lived environments\\nawait langfuse.flush();\\n```\\nWhen ingesting boolean scores, you can provide the value as a float. If you provide a configId, the score\'s name and config\'s name must match as well as their data types.\\n \\n \\n```\\nimport { LangfuseClient } from \\"@langfuse/client\\";\\nconst langfuse = new LangfuseClient();\\nlangfuse.score.create({\\n traceId: message.traceId,\\n observationId: message.generationId, // optional\\n name: \\"helpfulness\\",\\n value: 1,\\n comment: \\"Factually correct\\", // optional\\n id: \\"unique_id\\", // optional, can be used as an idempotency key to update the score subsequently\\n configId: \\"93547-6565-3453654-43543\\", // optional, can be used to infer the score data type and validate the score value\\n dataType: \\"BOOLEAN\\", // optional, possibly inferred\\n});\\n// Flush the scores in short-lived environments\\nawait langfuse.flush();\\n```\\nWhen ingesting text scores, provide the value as a string (1–500 characters). If you provide a configId, the score\'s name and config\'s name must match as well as their data types.\\n \\n \\n```\\nimport { LangfuseClient } from \\"@langfuse/client\\";\\nconst langfuse = new LangfuseClient();\\nlangfuse.score.create({\\n traceId: message.traceId,\\n observationId: message.generationId, // optional\\n name: \\"reviewer_notes\\",\\n value: \\"The response was helpful but could be more concise.\\","}, {"type": "text", "text": "\\nWhen ingesting boolean scores, you can provide the value as a float. If you provide a configId, the score\'s name and config\'s name must match as well as their data types.\\n \\n \\n```\\ncurl -X POST https://cloud.langfuse.com/api/public/scores \\\\\\n -u \\"pk-lf-...\\":\\"sk-lf-...\\" \\\\\\n -H \\"Content-Type: application/json\\" \\\\\\n -d \'{\\n \\"id\\": \\"unique_id\\",\\n \\"traceId\\": \\"trace_id_here\\",\\n \\"observationId\\": \\"observation_id_here\\",\\n \\"name\\": \\"helpfulness\\",\\n \\"value\\": 1,\\n \\"dataType\\": \\"BOOLEAN\\",\\n \\"configId\\": \\"93547-6565-3453654-43543\\",\\n \\"comment\\": \\"Factually correct\\"\\n }\'\\n```\\nWhen ingesting text scores, provide the value as a string (1–500 characters). If you provide a configId, the score\'s name and config\'s name must match as well as their data types.\\n \\n \\n```\\ncurl -X POST https://cloud.langfuse.com/api/public/scores \\\\\\n -u \\"pk-lf-...\\":\\"sk-lf-...\\" \\\\\\n -H \\"Content-Type: application/json\\" \\\\\\n -d \'{\\n \\"id\\": \\"unique_id\\",\\n \\"traceId\\": \\"trace_id_here\\",\\n \\"observationId\\": \\"observation_id_here\\",\\n \\"name\\": \\"reviewer_notes\\",\\n \\"value\\": \\"The response was helpful but could be more concise.\\",\\n \\"dataType\\": \\"TEXT\\",\\n \\"configId\\": \\"24680-6565-3453654-43543\\",\\n \\"comment\\": \\"Reviewed by QA team\\"\\n }\'\\n```\\nSee [API reference](/docs/api) for more details on POST/GET score configs endpoints.\\n### Inferred Score Properties\\nCertain score properties might be inferred based on your input:\\n * **If you don\'t provide a score data type** it will always be inferred. See tables below for details."}, {"type": "text", "text": "\\n * **For boolean and categorical scores** , we will provide the score value in both numerical and string format where possible. The score value format that is not provided as input, i.e. the translated value is referred to as the inferred value in the tables below.\\n * **On read via the[v3 scores API](https://api.reference.langfuse.com/#tag/scoresv3/GET/api/public/v3/scores)**, boolean scores are returned as a single boolean `value`.\\n * **For categorical scores** , the string representation is always provided and a numerical mapping of the category will be produced only if a `ScoreConfig` was provided.\\nDetailed Examples:\\nNumeric ScoresCategorical ScoresBoolean ScoresText Scores\\nFor example, let\'s assume you\'d like to ingest a numeric score to measure **accuracy**. We have included a table of possible score ingestion scenarios below.\\nValue| Data Type| Config Id| Description| Inferred Data Type| Valid \\n---|---|---|---|---|--- \\n`0.9`| `Null`| `Null`| Data type is inferred| `NUMERIC`| Yes \\n`0.9`| `NUMERIC`| `Null`| No properties inferred| | Yes \\n`depth`| `NUMERIC`| `Null`| Error: data type of value does not match provided data type| | No \\n`0.9`| `NUMERIC`| `78545`| No properties inferred| | Conditional on config validation \\n`0.9`| `Null`| `78545`| Data type inferred| `NUMERIC`| Conditional on config validation \\n`depth`| `NUMERIC`| `78545`| Error: data type of value does not match provided data type| | No \\n \\nFor example, let\'s assume you\'d like to ingest a categorical score to measure **correctness**. We have included a table of possible score ingestion scenarios below."}, {"type": "text", "text": "Value| Data Type| Config Id| Description| Inferred Data Type| Inferred Value representation| Valid \\n---|---|---|---|---|---|--- \\n`correct`| `Null`| `Null`| Data type is inferred| `CATEGORICAL`| | Yes \\n`correct`| `CATEGORICAL`| `Null`| No properties inferred| | | Yes \\n`1`| `CATEGORICAL`| `Null`| Error: data type of value does not match provided data type| | | No \\n`correct`| `CATEGORICAL`| `12345`| Numeric value inferred| | `4` numeric config category mapping| Conditional on config validation \\n`correct`| `NULL`| `12345`| Data type inferred| `CATEGORICAL`| | Conditional on config validation \\n`1`| `CATEGORICAL`| `12345`| Error: data type of value does not match provided data type| | | No \\n \\nFor example, let\'s assume you\'d like to ingest a boolean score to measure **helpfulness**. We have included a table of possible score ingestion scenarios below.\\nValue| Data Type| Config Id| Description| Inferred Data Type| Inferred Value representation| Valid \\n---|---|---|---|---|---|--- \\n`1`| `BOOLEAN`| `Null`| Value\'s string equivalent inferred| | `True`| Yes \\n`true`| `BOOLEAN`| `Null`| Error: data type of value does not match provided data type| | | No \\n`3`| `BOOLEAN`| `Null`| Error: boolean data type expects `0` or `1` as input value| | | No \\n`0.9`| `Null`| `93547`| Data type and value\'s string equivalent inferred| `BOOLEAN`| `True`| Conditional on config validation \\n`depth`| `BOOLEAN`| `93547`| Error: data type of value does not match provided data type| | | No \\n \\nFor example, let\'s assume you\'d like to ingest a text score to capture **reviewer notes**. Text scores must be non-empty strings of at most 500 characters.\\nValue| Data Type| Config Id| Description| Inferred Data Type| Valid "}, {"type": "text", "text": "---|---|---|---|---|--- \\n`\\"Good response\\"`| `Null`| `Null`| Data type is inferred| `TEXT`| Yes \\n`\\"Good response\\"`| `TEXT`| `Null`| No properties inferred| | Yes \\n`0.9`| `TEXT`| `Null`| Error: data type of value does not match provided data type| | No \\n`\\"Good response\\"`| `TEXT`| `24680`| No properties inferred| | Conditional on config validation \\n`\\"Good response\\"`| `Null`| `24680`| Data type inferred| `TEXT`| Conditional on config validation \\n`\\"\\"`| `TEXT`| `Null`| Error: text scores must be non-empty| | No \\n \\n## Update Existing Scores via API/SDKs\\nWhen creating a score, you can provide an optional `id` (JS/TS) / `score_id` (Python) parameter. A re-ingested score overwrites an existing one only when its `id`, `name`, and `timestamp` (at date granularity, `toDate(timestamp)`) all match β€” a matching `id` alone is not sufficient, and any difference creates a duplicate instead of an update.\\nTo overwrite a score without fetching existing scores first, set a stable `id` as an idempotency key when initially creating the score, and keep the `name` and `timestamp` unchanged on subsequent calls. See Preventing Duplicate Scores for details.\\n## Related guides\\nGuides\\n * [Collecting user feedback\\nCapture explicit and implicit user feedback as scores on your traces, including in-app ratings sent via our Browser SDK.\\n](/guides/user-feedback-loop)\\n * [Custom evaluation data pipeline\\nContinuously monitor quality by fetching traces, running custom evaluations, and ingesting scores back into Langfuse.\\n](/guides/cookbook/example_external_evaluation_pipelines)\\n * [Guardrails and security checks"}, {"type": "text", "text": "\\nCheck whether output contains a certain keyword, matches a required structure or format, or exceeds a length limit.\\n](/guides/cookbook/security-and-guardrails)\\n * [Custom internal workflow tooling\\nBuild internal tooling for human-in-the-loop workflows, ingesting scores back into Langfuse following your custom schema.\\n](/guides/human-in-the-loop-scoring)\\n * [Session-level quality tracking\\nScore full conversations, such as support chats or agent threads, by attaching scores via sessionId in the SDK or API.\\n](/guides/cookbook/example_evaluating_multi_turn_conversations)\\n* * *\\n* * *\\nLast edited 7/22/2026\\n"}], "type": "content"}}, {"type": "document", "record_type": "documentation", "url": "https://langfuse.com/docs/evaluation/evaluation-methods/scores-via-sdk", "title": "Scores via API/SDK", "source": {"content": [{"type": "text", "text": "# Scores via API/SDK\\nYou can use the Langfuse SDKs or API to add [scores](/docs/evaluation/scores/overview) to traces, observations, sessions and dataset runs. This is an evaluation method that allows you to set up custom evaluation workflows and extend the scoring capabilities of Langfuse. See the [data model](/docs/evaluation/scores/data-model#scores) for full details on the score object.\\nIf you want Langfuse to run deterministic Python or TypeScript logic for you, use [code evaluators](/docs/evaluation/evaluation-methods/code-evaluators). Use this page when your application, pipeline, or CI job computes scores and sends them to Langfuse via API or SDK.\\n\\n## Ingesting Scores via API/SDKs\\nScores can be attached at different levels of granularity: to individual traces, to specific observations within a trace, or to full sessions.\\nSee the [API reference](/docs/api) for full details on the POST and GET endpoints for scores and score configs.\\n### Trace or Observation-level Scores\\nYou can add scores via the Langfuse SDKs or API. Scores can take one of four data types: **Numeric** , **Categorical** , **Boolean** , or **Text**. See [Score Types](/docs/evaluation/scores/overview#score-types) for details.\\nIf a score is ingested manually using a `trace_id` to link the score to a trace, it is not necessary to wait until the trace has been created. The score will show up in the scores table and will be linked to the trace once the trace with the same `trace_id` is created.\\nHere are examples by `Score` data types.\\nFor trace and observation scores, `trace_id`/`traceId` is required and `observation_id`/`observationId` is optional. If you attach a score to an observation, always provide both the observation ID and the corresponding trace ID.\\nPython SDKJS/TS SDKAPI\\nNumericCategoricalBooleanText\\nNumeric score values must be provided as float.\\n \\n \\n```"}, {"type": "text", "text": "\\nfrom langfuse import get_client\\nlangfuse = get_client()\\n# Method 1: Score via low-level method\\nlangfuse.create_score(\\n name=\\"correctness\\",\\n value=0.9,\\n trace_id=\\"trace_id_here\\",\\n observation_id=\\"observation_id_here\\", # optional\\n data_type=\\"NUMERIC\\", # optional, inferred if not provided\\n comment=\\"Factually correct\\", # optional\\n)\\n# Method 2: Score current span/generation (within context)\\nwith langfuse.start_as_current_observation(as_type=\\"span\\", name=\\"my-operation\\") as span:\\n # Score the current span\\n span.score(\\n name=\\"correctness\\",\\n value=0.9,\\n data_type=\\"NUMERIC\\",\\n comment=\\"Factually correct\\"\\n )\\n # Score the trace\\n span.score_trace(\\n name=\\"overall_quality\\",\\n value=0.95,\\n data_type=\\"NUMERIC\\"\\n )\\n# Method 3: Score via the current context\\nwith langfuse.start_as_current_observation(as_type=\\"span\\", name=\\"my-operation\\"):\\n # Score the current span\\n langfuse.score_current_span(\\n name=\\"correctness\\",\\n value=0.9,\\n data_type=\\"NUMERIC\\",\\n comment=\\"Factually correct\\"\\n )\\n # Score the trace\\n langfuse.score_current_trace(\\n name=\\"overall_quality\\",\\n value=0.95,\\n data_type=\\"NUMERIC\\"\\n )\\n```\\nCategorical score values must be provided as strings.\\n \\n \\n```\\nfrom langfuse import get_client\\nlangfuse = get_client()\\n# Method 1: Score via low-level method\\nlangfuse.create_score(\\n name=\\"accuracy\\",\\n value=\\"partially correct\\",\\n trace_id=\\"trace_id_here\\",\\n observation_id=\\"observation_id_here\\", # optional\\n data_type=\\"CATEGORICAL\\", # optional, inferred if not provided\\n comment=\\"Some factual errors\\", # optional\\n)\\n# Method 2: Score current span/generation (within context)\\nwith langfuse.start_as_current_observation(as_type=\\"span\\", name=\\"my-operation\\") as span:\\n # Score the current span\\n span.score(\\n name=\\"accuracy\\",\\n value=\\"partially correct\\",\\n data_type=\\"CATEGORICAL\\",\\n comment=\\"Some factual errors\\"\\n )\\n # Score the trace\\n span.score_trace(\\n name=\\"overall_quality\\",\\n value=\\"partially correct\\","}, {"type": "text", "text": "\\n data_type=\\"CATEGORICAL\\"\\n )\\n# Method 3: Score via the current context\\nwith langfuse.start_as_current_observation(as_type=\\"span\\", name=\\"my-operation\\"):\\n # Score the current span\\n langfuse.score_current_span(\\n name=\\"accuracy\\",\\n value=\\"partially correct\\",\\n data_type=\\"CATEGORICAL\\",\\n comment=\\"Some factual errors\\"\\n )\\n # Score the trace\\n langfuse.score_current_trace(\\n name=\\"overall_quality\\",\\n value=\\"partially correct\\",\\n data_type=\\"CATEGORICAL\\"\\n )\\n```\\nBoolean scores must be provided as a float, where `1` is true and `0` is false. When read via the [v3 scores API](https://api.reference.langfuse.com/#tag/scoresv3/GET/api/public/v3/scores), the value is returned as a boolean. See [API reference](/docs/api) for more details on POST/GET scores endpoints.\\n \\n \\n```\\nfrom langfuse import get_client\\nlangfuse = get_client()\\n# Method 1: Score via low-level method\\nlangfuse.create_score(\\n name=\\"helpfulness\\",\\n value=0, # 0 or 1\\n trace_id=\\"trace_id_here\\",\\n observation_id=\\"observation_id_here\\", # optional\\n data_type=\\"BOOLEAN\\", # required, numeric values without data type would be inferred as NUMERIC\\n comment=\\"Incorrect answer\\", # optional\\n)\\n# Method 2: Score current span/generation (within context)\\nwith langfuse.start_as_current_observation(as_type=\\"span\\", name=\\"my-operation\\") as span:\\n # Score the current span\\n span.score(\\n name=\\"helpfulness\\",\\n value=1, # 0 or 1\\n data_type=\\"BOOLEAN\\",\\n comment=\\"Very helpful response\\"\\n )\\n # Score the trace\\n span.score_trace(\\n name=\\"overall_quality\\",\\n value=1, # 0 or 1\\n data_type=\\"BOOLEAN\\"\\n )\\n# Method 3: Score via the current context\\nwith langfuse.start_as_current_observation(as_type=\\"span\\", name=\\"my-operation\\"):\\n # Score the current span\\n langfuse.score_current_span(\\n name=\\"helpfulness\\","}, {"type": "text", "text": "\\n value=1, # 0 or 1\\n data_type=\\"BOOLEAN\\",\\n comment=\\"Very helpful response\\"\\n )\\n # Score the trace\\n langfuse.score_current_trace(\\n name=\\"overall_quality\\",\\n value=1, # 0 or 1\\n data_type=\\"BOOLEAN\\"\\n )\\n```\\nText score values must be provided as strings between 1 and 500 characters.\\n \\n \\n```\\nfrom langfuse import get_client\\nlangfuse = get_client()\\n# Method 1: Score via low-level method\\nlangfuse.create_score(\\n name=\\"reviewer_notes\\",\\n value=\\"The response was helpful but could be more concise.\\",\\n trace_id=\\"trace_id_here\\",\\n observation_id=\\"observation_id_here\\", # optional\\n data_type=\\"TEXT\\", # optional, inferred if not provided\\n comment=\\"Reviewed by QA team\\", # optional\\n)\\n# Method 2: Score current span/generation (within context)\\nwith langfuse.start_as_current_observation(as_type=\\"span\\", name=\\"my-operation\\") as span:\\n # Score the current span\\n span.score(\\n name=\\"reviewer_notes\\",\\n value=\\"The response was helpful but could be more concise.\\",\\n data_type=\\"TEXT\\",\\n comment=\\"Reviewed by QA team\\"\\n )\\n # Score the trace\\n span.score_trace(\\n name=\\"overall_notes\\",\\n value=\\"Good quality overall, minor formatting issues.\\",\\n data_type=\\"TEXT\\"\\n )\\n# Method 3: Score via the current context\\nwith langfuse.start_as_current_observation(as_type=\\"span\\", name=\\"my-operation\\"):\\n # Score the current span\\n langfuse.score_current_span(\\n name=\\"reviewer_notes\\",\\n value=\\"The response was helpful but could be more concise.\\",\\n data_type=\\"TEXT\\",\\n comment=\\"Reviewed by QA team\\"\\n )\\n # Score the trace\\n langfuse.score_current_trace(\\n name=\\"overall_notes\\",\\n value=\\"Good quality overall, minor formatting issues.\\",\\n data_type=\\"TEXT\\"\\n )\\n```\\nNumericCategoricalBooleanText\\nNumeric score values must be provided as float.\\n \\n \\n```"}, {"type": "text", "text": "\\nimport { LangfuseClient } from \\"@langfuse/client\\";\\nconst langfuse = new LangfuseClient();\\nlangfuse.score.create({\\n id: \\"unique_id\\", // optional, can be used as an idempotency key to update the score subsequently\\n traceId: message.traceId,\\n observationId: message.generationId, // optional\\n name: \\"correctness\\",\\n value: 0.9,\\n dataType: \\"NUMERIC\\", // optional, inferred if not provided\\n comment: \\"Factually correct\\", // optional\\n});\\n// Flush the scores in short-lived environments\\nawait langfuse.flush();\\n```\\nCategorical score values must be provided as strings.\\n \\n \\n```\\nimport { LangfuseClient } from \\"@langfuse/client\\";\\nconst langfuse = new LangfuseClient();\\nlangfuse.score.create({\\n id: \\"unique_id\\", // optional, can be used as an idempotency key to update the score subsequently\\n traceId: message.traceId,\\n observationId: message.generationId, // optional\\n name: \\"accuracy\\",\\n value: \\"partially correct\\",\\n dataType: \\"CATEGORICAL\\", // optional, inferred if not provided\\n comment: \\"Factually correct\\", // optional\\n});\\n// Flush the scores in short-lived environments\\nawait langfuse.flush();\\n```\\nBoolean scores must be provided as a float, where `1` is true and `0` is false. When read via the [v3 scores API](https://api.reference.langfuse.com/#tag/scoresv3/GET/api/public/v3/scores), the value is returned as a boolean. See [API reference](/docs/api) for more details on POST/GET scores endpoints.\\n \\n \\n```\\nimport { LangfuseClient } from \\"@langfuse/client\\";\\nconst langfuse = new LangfuseClient();\\nlangfuse.score.create({\\n id: \\"unique_id\\", // optional, can be used as an idempotency key to update the score subsequently\\n traceId: message.traceId,\\n observationId: message.generationId, // optional"}, {"type": "text", "text": "\\nnpm install @langfuse/browser\\n```\\n \\n \\n```\\nimport { LangfuseBrowserClient } from \\"@langfuse/browser\\";\\nconst langfuse = new LangfuseBrowserClient({\\n publicKey: process.env.NEXT_PUBLIC_LANGFUSE_PUBLIC_KEY!,\\n baseUrl: process.env.NEXT_PUBLIC_LANGFUSE_BASE_URL, // optional, defaults to https://cloud.langfuse.com\\n});\\nawait langfuse.score({\\n traceId: message.traceId,\\n observationId: message.generationId, // optional\\n id: `user-feedback-${message.traceId}`, // optional, use as an idempotency key\\n name: \\"user-feedback\\",\\n value: 1, // 1 for positive, 0 for negative\\n dataType: \\"BOOLEAN\\",\\n comment: \\"Helpful answer\\", // optional\\n});\\n```\\n`score()` returns `{ id }` after ingestion succeeds. No `flush()` call is required.\\n### Session-level Scores\\nTo score an entire session (without attaching the score to a trace or observation), provide only `session_id` (Python SDK) or `sessionId` (JS/TS SDK and API).\\nPython SDKJS/TS SDKAPI\\n \\n \\n```\\nfrom langfuse import get_client\\nlangfuse = get_client()\\nlangfuse.create_score(\\n name=\\"session_quality\\",\\n value=0.85,\\n session_id=\\"session_id_here\\",\\n data_type=\\"NUMERIC\\",\\n comment=\\"Overall conversation quality\\"\\n)\\n```\\n \\n \\n```\\nimport { LangfuseClient } from \\"@langfuse/client\\";\\nconst langfuse = new LangfuseClient();\\nlangfuse.score.create({\\n name: \\"session_quality\\",\\n value: 0.85,\\n sessionId: \\"session_id_here\\",\\n dataType: \\"NUMERIC\\",\\n comment: \\"Overall conversation quality\\",\\n});\\nawait langfuse.flush();\\n```\\n \\n \\n```\\ncurl -X POST https://cloud.langfuse.com/api/public/scores \\\\\\n -u \\"pk-lf-...\\":\\"sk-lf-...\\" \\\\\\n -H \\"Content-Type: application/json\\" \\\\\\n -d \'{\\n \\"sessionId\\": \\"session_id_here\\",\\n \\"name\\": \\"session_quality\\",\\n \\"value\\": 0.85,\\n \\"dataType\\": \\"NUMERIC\\",\\n \\"comment\\": \\"Overall conversation quality\\"\\n }\'\\n```\\n"}], "type": "content"}}, {"type": "document", "record_type": "documentation", "url": "https://langfuse.com/docs/evaluation/evaluation-methods/scores-via-sdk", "title": "Scores via API/SDK", "source": {"content": [{"type": "text", "text": "## Advanced\\n### Preventing Duplicate Scores\\nBy default, Langfuse allows for multiple scores of the same `name` on the same trace. This is useful if you\'d like to track the evolution of a score over time or if e.g. you\'ve received multiple user feedback scores on the same trace.\\nIn some cases, you want to prevent this behavior or overwrite an existing score. A score is identified by three fields β€” its `id`, its `name`, and its date (`toDate(timestamp)`) β€” and a new score only replaces an existing one when all three match. To achieve this, create an **idempotency key** and pass it as the `id` (JS/TS) / `score_id` (Python) when creating the score (for example `trace_id-score_name`), and keep the `name` and `timestamp` unchanged across calls. If any of the three differs β€” for example the same `id` with a different `name` β€” you get an additional score rather than a replacement.\\nDo not send partial score updates (only the changed fields) and rely on Langfuse to merge them into the existing record. This merge only happens for a limited window after creation, is deprecated, and will be removed. Always send the complete score.\\nSee [How to update traces, observations, and scores](/faq/all/tracing-data-updates#updating-scores) for more details on Langfuse\'s immutable data model.\\n### Enforcing a Score Config\\nScore configs are helpful when you want to standardize your scores for future analysis.\\nTo enforce a score config, you can provide a `configId` when creating a score to reference a `ScoreConfig` that was previously created. `Score Configs` can be defined in the Langfuse UI or via our API. [See our guide on how to create and manage score configs](/faq/all/manage-score-configs).\\nWhenever you provide a `ScoreConfig`, the score data will be validated against the config. The following rules apply:\\n * **Score Name** : Must equal the config\'s name"}, {"type": "text", "text": "\\n * **Score Data Type** : When provided, must match the config\'s data type\\n * **Score Value when Type is`NUMERIC`**: Value must be within the min and max values defined in the config (if provided, min and max are optional and otherwise are assumed as -∞ and +∞ respectively)\\n * **Score Value when Type is`CATEGORICAL`**: Value must map to one of the categories defined in the config\\n * **Score Value when Type is`BOOLEAN`**: Value must equal `0` or `1`\\n * **Score Value when Type is`TEXT`**: Value must be a non-empty string of at most 500 characters\\nPython SDKJS/TS SDKAPI\\nNumeric ScoresCategorical ScoresBoolean ScoresText Scores\\nWhen ingesting numeric scores, you can provide the value as a float. If you provide a configId, the score value will be validated against the config\'s numeric range, which might be defined by a minimum and/or maximum value.\\n \\n \\n```\\nfrom langfuse import get_client\\nlangfuse = get_client()\\n# Method 1: Score via low-level method\\nlangfuse.create_score(\\n trace_id=\\"trace_id_here\\",\\n observation_id=\\"observation_id_here\\", # optional\\n session_id=\\"session_id_here\\", # optional, ID of the session the score relates to\\n name=\\"accuracy\\",\\n value=0.9,\\n comment=\\"Factually correct\\", # optional\\n score_id=\\"unique_id\\", # optional, can be used as an idempotency key to update the score subsequently\\n config_id=\\"78545-6565-3453654-43543\\", # optional, to ensure that the score follows a specific min/max value range\\n data_type=\\"NUMERIC\\" # optional, possibly inferred\\n)\\n# Method 2: Score within context\\nwith langfuse.start_as_current_observation(as_type=\\"span\\", name=\\"my-operation\\") as span:\\n span.score(\\n name=\\"accuracy\\",\\n value=0.9,\\n comment=\\"Factually correct\\",\\n config_id=\\"78545-6565-3453654-43543\\",\\n data_type=\\"NUMERIC\\"\\n )\\n```"}, {"type": "text", "text": "\\nWhen ingesting numeric scores, you can provide the value as a float. If you provide a configId, the score value will be validated against the config\'s numeric range, which might be defined by a minimum and/or maximum value.\\n \\n \\n```\\nimport { LangfuseClient } from \\"@langfuse/client\\";\\nconst langfuse = new LangfuseClient();\\nlangfuse.score.create({\\n traceId: message.traceId,\\n observationId: message.generationId, // optional\\n name: \\"accuracy\\",\\n value: 0.9,\\n comment: \\"Factually correct\\", // optional\\n id: \\"unique_id\\", // optional, can be used as an idempotency key to update the score subsequently\\n configId: \\"78545-6565-3453654-43543\\", // optional, to ensure that the score follows a specific min/max value range\\n dataType: \\"NUMERIC\\", // optional, possibly inferred\\n});\\n// Flush the scores in short-lived environments\\nawait langfuse.flush();\\n```\\nCategorical scores are used to evaluate data that falls into specific categories. When ingesting categorical scores, you can provide the value as a string. If you provide a configId, the score value will be validated against the config\'s categories.\\n \\n \\n```\\nimport { LangfuseClient } from \\"@langfuse/client\\";\\nconst langfuse = new LangfuseClient();\\nlangfuse.score.create({\\n traceId: message.traceId,\\n observationId: message.generationId, // optional\\n name: \\"correctness\\",\\n value: \\"correct\\",\\n comment: \\"Factually correct\\", // optional\\n id: \\"unique_id\\", // optional, can be used as an idempotency key to update the score subsequently\\n configId: \\"12345-6565-3453654-43543\\", // optional, to ensure that a score maps to a specific category defined in a score config\\n dataType: \\"CATEGORICAL\\", // optional, possibly inferred\\n});"}, {"type": "text", "text": "\\nYou can also enforce score configs via the [REST API](https://api.reference.langfuse.com/#tag/legacyscorev1/POST/api/public/scores) by providing a `configId`.\\nNumeric ScoresCategorical ScoresBoolean ScoresText Scores\\nWhen ingesting numeric scores, you can provide the value as a float. If you provide a configId, the score value will be validated against the config\'s numeric range.\\n \\n \\n```\\ncurl -X POST https://cloud.langfuse.com/api/public/scores \\\\\\n -u \\"pk-lf-...\\":\\"sk-lf-...\\" \\\\\\n -H \\"Content-Type: application/json\\" \\\\\\n -d \'{\\n \\"id\\": \\"unique_id\\",\\n \\"traceId\\": \\"trace_id_here\\",\\n \\"observationId\\": \\"observation_id_here\\",\\n \\"name\\": \\"accuracy\\",\\n \\"value\\": 0.9,\\n \\"dataType\\": \\"NUMERIC\\",\\n \\"configId\\": \\"78545-6565-3453654-43543\\",\\n \\"comment\\": \\"Factually correct\\"\\n }\'\\n```\\nCategorical scores are used to evaluate data that falls into specific categories. If you provide a configId, the score value will be validated against the config\'s categories.\\n \\n \\n```\\ncurl -X POST https://cloud.langfuse.com/api/public/scores \\\\\\n -u \\"pk-lf-...\\":\\"sk-lf-...\\" \\\\\\n -H \\"Content-Type: application/json\\" \\\\\\n -d \'{\\n \\"id\\": \\"unique_id\\",\\n \\"traceId\\": \\"trace_id_here\\",\\n \\"observationId\\": \\"observation_id_here\\",\\n \\"name\\": \\"correctness\\",\\n \\"value\\": \\"correct\\",\\n \\"dataType\\": \\"CATEGORICAL\\",\\n \\"configId\\": \\"12345-6565-3453654-43543\\",\\n \\"comment\\": \\"Factually correct\\"\\n }\'\\n```\\nWhen ingesting boolean scores, you can provide the value as a float. If you provide a configId, the score\'s name and config\'s name must match as well as their data types.\\n \\n \\n```\\ncurl -X POST https://cloud.langfuse.com/api/public/scores \\\\\\n -u \\"pk-lf-...\\":\\"sk-lf-...\\" \\\\\\n -H \\"Content-Type: application/json\\" \\\\\\n -d \'{\\n \\"id\\": \\"unique_id\\",\\n \\"traceId\\": \\"trace_id_here\\",\\n \\"observationId\\": \\"observation_id_here\\",\\n \\"name\\": \\"helpfulness\\","}, {"type": "text", "text": "\\n \\"value\\": 1,\\n \\"dataType\\": \\"BOOLEAN\\",\\n \\"configId\\": \\"93547-6565-3453654-43543\\",\\n \\"comment\\": \\"Factually correct\\"\\n }\'\\n```\\nWhen ingesting text scores, provide the value as a string (1–500 characters). If you provide a configId, the score\'s name and config\'s name must match as well as their data types.\\n \\n \\n```\\ncurl -X POST https://cloud.langfuse.com/api/public/scores \\\\\\n -u \\"pk-lf-...\\":\\"sk-lf-...\\" \\\\\\n -H \\"Content-Type: application/json\\" \\\\\\n -d \'{\\n \\"id\\": \\"unique_id\\",\\n \\"traceId\\": \\"trace_id_here\\",\\n \\"observationId\\": \\"observation_id_here\\",\\n \\"name\\": \\"reviewer_notes\\",\\n \\"value\\": \\"The response was helpful but could be more concise.\\",\\n \\"dataType\\": \\"TEXT\\",\\n \\"configId\\": \\"24680-6565-3453654-43543\\",\\n \\"comment\\": \\"Reviewed by QA team\\"\\n }\'\\n```\\nSee [API reference](/docs/api) for more details on POST/GET score configs endpoints.\\n### Inferred Score Properties\\nCertain score properties might be inferred based on your input:\\n * **If you don\'t provide a score data type** it will always be inferred. See tables below for details.\\n * **For boolean and categorical scores** , we will provide the score value in both numerical and string format where possible. The score value format that is not provided as input, i.e. the translated value is referred to as the inferred value in the tables below.\\n * **On read via the[v3 scores API](https://api.reference.langfuse.com/#tag/scoresv3/GET/api/public/v3/scores)**, boolean scores are returned as a single boolean `value`.\\n * **For categorical scores** , the string representation is always provided and a numerical mapping of the category will be produced only if a `ScoreConfig` was provided.\\nDetailed Examples:"}, {"type": "text", "text": "\\nNumeric ScoresCategorical ScoresBoolean ScoresText Scores\\nFor example, let\'s assume you\'d like to ingest a numeric score to measure **accuracy**. We have included a table of possible score ingestion scenarios below.\\nValue| Data Type| Config Id| Description| Inferred Data Type| Valid \\n---|---|---|---|---|--- \\n`0.9`| `Null`| `Null`| Data type is inferred| `NUMERIC`| Yes \\n`0.9`| `NUMERIC`| `Null`| No properties inferred| | Yes \\n`depth`| `NUMERIC`| `Null`| Error: data type of value does not match provided data type| | No \\n`0.9`| `NUMERIC`| `78545`| No properties inferred| | Conditional on config validation \\n`0.9`| `Null`| `78545`| Data type inferred| `NUMERIC`| Conditional on config validation \\n`depth`| `NUMERIC`| `78545`| Error: data type of value does not match provided data type| | No \\n \\nFor example, let\'s assume you\'d like to ingest a categorical score to measure **correctness**. We have included a table of possible score ingestion scenarios below.\\nValue| Data Type| Config Id| Description| Inferred Data Type| Inferred Value representation| Valid \\n---|---|---|---|---|---|--- \\n`correct`| `Null`| `Null`| Data type is inferred| `CATEGORICAL`| | Yes \\n`correct`| `CATEGORICAL`| `Null`| No properties inferred| | | Yes \\n`1`| `CATEGORICAL`| `Null`| Error: data type of value does not match provided data type| | | No \\n`correct`| `CATEGORICAL`| `12345`| Numeric value inferred| | `4` numeric config category mapping| Conditional on config validation \\n`correct`| `NULL`| `12345`| Data type inferred| `CATEGORICAL`| | Conditional on config validation \\n`1`| `CATEGORICAL`| `12345`| Error: data type of value does not match provided data type| | | No \\n "}, {"type": "text", "text": "For example, let\'s assume you\'d like to ingest a boolean score to measure **helpfulness**. We have included a table of possible score ingestion scenarios below.\\nValue| Data Type| Config Id| Description| Inferred Data Type| Inferred Value representation| Valid \\n---|---|---|---|---|---|--- \\n`1`| `BOOLEAN`| `Null`| Value\'s string equivalent inferred| | `True`| Yes \\n`true`| `BOOLEAN`| `Null`| Error: data type of value does not match provided data type| | | No \\n`3`| `BOOLEAN`| `Null`| Error: boolean data type expects `0` or `1` as input value| | | No \\n`0.9`| `Null`| `93547`| Data type and value\'s string equivalent inferred| `BOOLEAN`| `True`| Conditional on config validation \\n`depth`| `BOOLEAN`| `93547`| Error: data type of value does not match provided data type| | | No \\n \\nFor example, let\'s assume you\'d like to ingest a text score to capture **reviewer notes**. Text scores must be non-empty strings of at most 500 characters.\\nValue| Data Type| Config Id| Description| Inferred Data Type| Valid \\n---|---|---|---|---|--- \\n`\\"Good response\\"`| `Null`| `Null`| Data type is inferred| `TEXT`| Yes \\n`\\"Good response\\"`| `TEXT`| `Null`| No properties inferred| | Yes \\n`0.9`| `TEXT`| `Null`| Error: data type of value does not match provided data type| | No \\n`\\"Good response\\"`| `TEXT`| `24680`| No properties inferred| | Conditional on config validation \\n`\\"Good response\\"`| `Null`| `24680`| Data type inferred| `TEXT`| Conditional on config validation \\n`\\"\\"`| `TEXT`| `Null`| Error: text scores must be non-empty| | No \\n \\n\\n## Update Existing Scores via API/SDKs"}, {"type": "text", "text": "When creating a score, you can provide an optional `id` (JS/TS) / `score_id` (Python) parameter. A re-ingested score overwrites an existing one only when its `id`, `name`, and `timestamp` (at date granularity, `toDate(timestamp)`) all match β€” a matching `id` alone is not sufficient, and any difference creates a duplicate instead of an update.\\nTo overwrite a score without fetching existing scores first, set a stable `id` as an idempotency key when initially creating the score, and keep the `name` and `timestamp` unchanged on subsequent calls. See Preventing Duplicate Scores for details.\\n## Related guides\\nGuides\\n * [Collecting user feedback\\nCapture explicit and implicit user feedback as scores on your traces, including in-app ratings sent via our Browser SDK.\\n](/guides/user-feedback-loop)\\n * [Custom evaluation data pipeline\\nContinuously monitor quality by fetching traces, running custom evaluations, and ingesting scores back into Langfuse.\\n](/guides/cookbook/example_external_evaluation_pipelines)\\n * [Guardrails and security checks\\nCheck whether output contains a certain keyword, matches a required structure or format, or exceeds a length limit.\\n](/guides/cookbook/security-and-guardrails)\\n * [Custom internal workflow tooling\\nBuild internal tooling for human-in-the-loop workflows, ingesting scores back into Langfuse following your custom schema.\\n](/guides/human-in-the-loop-scoring)\\n * [Session-level quality tracking\\nScore full conversations, such as support chats or agent threads, by attaching scores via sessionId in the SDK or API.\\n](/guides/cookbook/example_evaluating_multi_turn_conversations)\\n* * *\\n* * *\\nLast edited 7/22/2026\\n"}], "type": "content"}}, {"type": "document", "record_type": "documentation", "url": "https://langfuse.com/docs/evaluation/scores/overview", "title": "Scores", "source": {"content": [{"type": "text", "text": "Overview - Langfuse\\n[Docs](/docs)\\n[Evaluation](/docs/evaluation/overview)\\n[Scores](/docs/evaluation/scores/overview)\\n[Overview](/docs/evaluation/scores/overview)\\n# Scores\\nScores are Langfuse\'s universal data object for storing evaluation results. Any time you want to assign a quality judgment to an LLM output, whether by a [human annotation](/docs/evaluation/evaluation-methods/scores-via-ui), an [LLM judge](/docs/evaluation/evaluation-methods/llm-as-a-judge), a [programmatic check](/docs/evaluation/evaluation-methods/scores-via-sdk), or end-user feedback, the result is stored as a score.\\nEvery score has a **name** (like `\\"correctness\\"` or `\\"helpfulness\\"`), a **value** , and a **data type**. Scores also support an optional **comment** for additional context.\\nScores can be attached to [traces](/docs/observability/data-model#observations-and-traces), [observations](/docs/observability/data-model#observations-and-traces), [sessions](/docs/observability/data-model#sessions), or [dataset runs](/docs/evaluation/experiments/data-model). Most commonly, scores are attached to traces to evaluate a single end-to-end interaction.\\nOnce you have scores, they show up in [score analytics](/docs/evaluation/scores/score-analytics), can be visualized in [custom dashboards](/docs/metrics/features/custom-dashboards), and can be queried via the [API](/docs/api).\\n## When to Use Scores\\nScores become useful when you want to go beyond observing what your application does and start measuring how well it does it. Common use cases:\\n * **Collecting user feedback** : Capture thumbs up/down or star ratings from your users and attach them to traces. See the [user feedback guide](/docs/observability/features/user-feedback)."}, {"type": "text", "text": "\\n * **Monitoring production quality** : Set up automated evaluators (like [LLM-as-a-Judge](/docs/evaluation/evaluation-methods/llm-as-a-judge)) to continuously score live traces for things like hallucination, relevance, or tone.\\n * **Running guardrails** : Score whether outputs pass safety checks like PII detection, format validation, or content policy compliance.\\n * **Comparing changes with experiments** : When you change a prompt, model, or pipeline, run an [experiment](/docs/evaluation/experiments) to score the new version against a dataset.\\n## Score Types\\nLangfuse supports four score data types:\\nType| Value| Use when \\n---|---|--- \\n`NUMERIC`| Float (e.g. `0.9`)| Continuous judgments like accuracy, relevance, or similarity scores \\n`CATEGORICAL`| String from predefined categories (e.g. `\\"correct\\"`, `\\"partially correct\\"`)| Discrete classifications where the set of possible values is known upfront \\n`BOOLEAN`| `0` or `1`| Pass/fail checks like hallucination detection or format validation \\n`TEXT`| Free-form string (1-500 characters)| Open-ended annotations like reviewer notes or qualitative feedback. Often used for [open coding](https://en.wikipedia.org/wiki/Open_coding) before formalizing into quantifiable scores via [axial coding](https://en.wikipedia.org/wiki/Axial_coding). \\n \\nText scores are designed for qualitative, open-ended scoring. Because free-form text cannot be meaningfully aggregated or compared, text scores are not supported in [experiments](/docs/evaluation/core-concepts#experiments), [LLM-as-a-Judge](/docs/evaluation/evaluation-methods/llm-as-a-judge), or [score analytics](/docs/evaluation/scores/score-analytics).\\n## How to Create Scores"}, {"type": "text", "text": "\\nThere are five ways to add scores:\\n * **LLM-as-a-Judge** : Set up [automated evaluators](/docs/evaluation/evaluation-methods/llm-as-a-judge) that score traces based on custom criteria (e.g. hallucination, tone, relevance). These can return numeric or categorical scores plus reasoning, and can run on live production traces or on experiment results.\\n * **Code evaluators** : Run [custom Python or TypeScript evaluators](/docs/evaluation/evaluation-methods/code-evaluators) in Langfuse for deterministic checks such as exact match, JSON validation, or custom business rules.\\n * **Scores via UI** : Team members [manually score](/docs/evaluation/evaluation-methods/scores-via-ui) traces, observations, or sessions directly in the Langfuse UI. Requires a [score config](/faq/all/manage-score-configs) to be set up first.\\n * **Annotation Queues** : Set up [structured review workflows](/docs/evaluation/evaluation-methods/annotation-queues) where reviewers work through batches of traces.\\n * **Scores via API/SDK** : [Programmatically add scores](/docs/evaluation/evaluation-methods/scores-via-sdk) from your application code. This is the way to go for user feedback (thumbs up/down, star ratings), guardrail results, or custom evaluation pipelines.\\n## Should I Use Scores or Tags?\\n| Scores| Tags \\n---|---|--- \\n **Purpose**| Measure _how good_ something is| Describe _what_ something is \\n **Data**| Numeric, categorical, boolean, or text value| Simple string label \\n **When added**| Can be added at any time, including long after the trace was created| Set during tracing and cannot be changed afterwards \\n **Used for**| Quality measurement, analytics, experiments| Filtering, segmentation, organizing \\n "}, {"type": "text", "text": "As a rule of thumb: if you already know the category at tracing time (e.g. which feature or API endpoint triggered the trace), use a [tag](/docs/observability/features/tags). If you need to classify or evaluate traces later, use a score.\\n## Score Comments\\nEvery score supports an optional **comment** field. Use it to capture reasoning (e.g. why an LLM judge assigned a particular score), reviewer notes, or context that helps others understand the score value. Comments are shown alongside scores in the Langfuse UI.\\nUse a `TEXT` score instead of comments to capture standalone qualitative feedback -- comments are best for additional reasoning on an existing score.\\n* * *\\n* * *\\nLast edited 7/23/2026\\n"}], "type": "content"}}, {"type": "document", "record_type": "github_discussion", "url": "https://github.com/orgs/langfuse/discussions/6016", "title": "Using scores in data sets from the SDK?", "source": {"content": [{"type": "text", "text": "Title: Using scores in data sets from the SDK?\\nCategory: Support\\nAuthor: @howinator\\nTimestamp: March 14, 2025\\nBody:\\nHere is a workflow that I want, but I can\'t figure out how to accomplish it without a lot of manual work.\\r\\n\\r\\n1. Collect traces\\r\\n2. Annotate traces with either manual annotation or programmatic feedback from users. For a simple classification, I would use a boolean score where True means the model returned the correct answer and False when it returned the incorrect answer.\\r\\n4. Create a \\"false positive\\" data set composed of all traces where the score was False. <-- I can\'t figure out how to do this without a lot of clicking.\\r\\n5. Run an eval from that data set and use the score to determine if the eval was correct, e.g., something like `(dataset.item.score && dataset.item.output === evalOutput.output) || (!dataset.item.score && dataset.item.output !== evalOutput.output)`. <-- I can\'t figure out how to do this programmatically at all.\\r\\n\\r\\nIs my workflow completely off here? Am I missing something? Or is this flow just not possible in Langfuse today?\\nComments:\\n\\n Comment #1 from @dosubot on March 14, 2025:\\n\\n <!-- Greeting -->\\nHey there, @howinator! I\'m here to help you out with any bugs, questions, or contributions you have in mind. Let\'s get to work!\\n\\n<!-- Answer -->\\nYour workflow is mostly feasible in Langfuse, and you can achieve it programmatically with some adjustments:\\n\\n1. **Collect Traces**: This step is straightforward and can be done using Langfuse\'s SDKs to capture and store traces.\\n\\n2. **Annotate Traces**: You can programmatically annotate traces with scores using the `langfuse_context.score_current_trace` function. This allows you to attach a boolean score to each trace based on user feedback or other criteria <sup>[[1]](https://langfuse.com/guides/cookbook/integration_langgraph)</sup>.\\n"}, {"type": "text", "text": "\\n3. **Create a \\"False Positive\\" Dataset**: While there isn\'t a direct method to filter dataset items based on scores, you can programmatically create a dataset by iterating over your traces and selecting those with a score of `False`. Use the SDK to create dataset items for these traces <sup>[[2]](https://langfuse.com/docs/datasets/get-started)</sup>.\\n\\n4. **Run an Evaluation**: You can automate the creation of evaluation jobs for your dataset using the `createEvalJobs` function. This function allows you to specify configurations and run evaluations on specific subsets of dataset items <sup>[[3]](https://github.com/langfuse/langfuse/blob/eecf0d680b13a4bde9fe8fed617acc328029307d/worker/src/__tests__/evalService.test.ts)</sup>.\\n\\nFor the evaluation logic, you can implement custom logic to compare the dataset item scores and outputs with the evaluation outputs. This might require some additional scripting to handle the specific logic you described.\\n\\nOverall, your workflow is possible with Langfuse, but it may require some custom scripting to fully automate the process of filtering and evaluating dataset items based on scores.\\n\\n\\n<!-- Dosu Comment Footer -->\\n*To continue talking to [Dosu](https://dosu.dev), mention @dosu.*\\n\\n---"}, {"type": "text", "text": "1. Create a \\"false positive\\" data set composed of all traces where the score was False -> use scores api to fetch all of these\\r\\n2. Run your own eval on them and add it to the same trace via the sdk\\r\\n3. Compare the evals by fetching all of them via the sdk again.\\r\\n\\r\\nWe currently plan to add more configurable dashboards which will allow for the comparison of these evals more easily in the UI.\\n"}], "type": "content"}}, {"type": "document", "record_type": "documentation", "url": "https://langfuse.com/docs/observability/troubleshooting-and-faq", "title": "Troubleshooting and FAQ", "source": {"content": [{"type": "text", "text": "\\n[LLM as a Session-Level Judge with Support for LangGraph Session](https://github.com/orgs/langfuse/discussions/8303 \\"Langfuse Ideas: LLM as a Session-Level Judge with Support for LangGraph Session\\")\\n[Score Configs: Allow editing the categories of a categorical score](https://github.com/orgs/langfuse/discussions/8259 \\"Langfuse Ideas: Score Configs: Allow editing the categories of a categorical score\\")\\n[feat(trace-detail-view): for each span, show the sum of nested observations\' token usage details](https://github.com/orgs/langfuse/discussions/8201 \\"Langfuse Ideas: feat\\\\(trace-detail-view\\\\): for each span, show the sum of nested observations\' token usage details\\")\\n[UI-LLM as a Jury](https://github.com/orgs/langfuse/discussions/8195 \\"Langfuse Ideas: UI-LLM as a Jury\\")\\n[Add Gemini 2.5 Pro model pricing](https://github.com/orgs/langfuse/discussions/8146 \\"Langfuse Ideas: Add Gemini 2.5 Pro model pricing\\")\\n[Deeplinking for Spans inside of a Trace](https://github.com/orgs/langfuse/discussions/8112 \\"Langfuse Ideas: Deeplinking for Spans inside of a Trace\\")\\n[feat(comments): allow attaching images](https://github.com/orgs/langfuse/discussions/8111 \\"Langfuse Ideas: feat\\\\(comments\\\\): allow attaching images\\")\\n[feat(sessions): support session-level tags](https://github.com/orgs/langfuse/discussions/8041 \\"Langfuse Ideas: feat\\\\(sessions\\\\): support session-level tags\\")\\n[Enable Immediate Score Management for User Feedback](https://github.com/orgs/langfuse/discussions/7686 \\"Langfuse Ideas: Enable Immediate Score Management for User Feedback\\")\\n[Allow for structured filtering on inputs / outputs](https://github.com/orgs/langfuse/discussions/7663 \\"Langfuse Ideas: Allow for structured filtering on inputs / outputs\\")"}, {"type": "text", "text": "\\n[Allow for lazy updating of traces in Python SDK (+ possible workaround)](https://github.com/orgs/langfuse/discussions/7655 \\"Langfuse Ideas: Allow for lazy updating of traces in Python SDK \\\\(+ possible workaround\\\\)\\")\\n[Filter by scores in session view](https://github.com/orgs/langfuse/discussions/7528 \\"Langfuse Ideas: Filter by scores in session view\\")\\n[Support break lines on evaluation run tooltip hint](https://github.com/orgs/langfuse/discussions/7452 \\"Langfuse Ideas: Support break lines on evaluation run tooltip hint\\")\\n[Enable sending Langfuse SDK OTEL spans to third party observability backend](https://github.com/orgs/langfuse/discussions/7365 \\"Langfuse Ideas: Enable sending Langfuse SDK OTEL spans to third party observability backend\\")\\n[Add Thinking config in playground for anthropic models](https://github.com/orgs/langfuse/discussions/7267 \\"Langfuse Ideas: Add Thinking config in playground for anthropic models\\")\\n[Better support for big int in metadata.](https://github.com/orgs/langfuse/discussions/7167 \\"Langfuse Ideas: Better support for big int in metadata.\\")\\n[Ability to filter/search comments on traces](https://github.com/orgs/langfuse/discussions/7109 \\"Langfuse Ideas: Ability to filter/search comments on traces\\")\\n[Image / File / Multimedia rendering in dataset items using mediaId](https://github.com/orgs/langfuse/discussions/6883 \\"Langfuse Ideas: Image / File / Multimedia rendering in dataset items using mediaId\\")\\n[Trace Comments support upload image](https://github.com/orgs/langfuse/discussions/6857 \\"Langfuse Ideas: Trace Comments support upload image\\")"}], "type": "content"}}, {"type": "document", "record_type": "github_issue", "url": "https://github.com/langfuse/langfuse/issues/3380", "title": "feat: add fetch_score(s) in the sdks", "source": {"content": [{"type": "text", "text": "Github Issue by @marcklingen created on September 17, 2024. The current state of the issue is: closed\\n\\n### Discussed in https://github.com/orgs/langfuse/discussions/3379\\r\\n\\r\\n<div type=\'discussions-op-text\'>\\r\\n\\r\\n<sup>Originally posted by **Preet1234-ux** September 17, 2024</sup>\\r\\nI want to fetch the score provided by the user from langfuse in my python code . Are there any functionality by which I can fetch the score for a trace ? </div>\\r\\n\\r\\n\\r\\nCurrently, we need to use either langfuse.client.*(potentially breaking) or the api.\\n---\\nComment #1 by @hassiebp:\\n\\nAlready shipped in Python SDK v3\\n\\n"}], "type": "content"}}, {"type": "document", "record_type": "github_discussion", "url": "https://github.com/orgs/langfuse/discussions/6210", "title": "Next.js + Vercel AI SDK: How to get `observationId` and attach it to scores?", "source": {"content": [{"type": "text", "text": "Title: Next.js + Vercel AI SDK: How to get `observationId` and attach it to scores?\\nCategory: Support\\nAuthor: @natterstefan\\nTimestamp: March 26, 2025\\nBody:\\nDear langfuse-Community,\\r\\n\\r\\nI need your help please, as I do not know how to obtain the `observationId` and attach it to a `score`.\\r\\n\\r\\nContext: this is how my `experimental_telemetry` looks like:\\r\\n\\r\\n```js\\r\\n// src: https://langfuse.com/docs/integrations/vercel-ai-sdk#group-multiple-executions-in-one-trace\\r\\n\\r\\nconst parentTraceId = getRandomId()\\r\\n\\r\\nexport async function POST(req: Request) {\\r\\n\\t// ... other code\\r\\n\\r\\n\\tconst trace = langfuse.trace({\\r\\n\\t id: parentTraceId,\\r\\n\\t name: \'cd-chat-messages\',\\r\\n\\t})\\r\\n\\r\\n\\tconsole.log(\'trace\', trace.observationId) // <<-- this returns `null`\\r\\n\\t\\r\\n\\tconst result = streamText({\\r\\n\\t\\t// ....\\r\\n\\t\\texperimental_telemetry: {\\r\\n\\t\\t isEnabled: true,\\r\\n\\t\\t functionId: \'cd-chat-message-\' + messages.length,\\r\\n\\t\\t metadata: {\\r\\n\\t\\t langfuseTraceId: parentTraceId,\\r\\n\\t\\t\\t // Do not update the parent trace with execution results\\r\\n\\t\\t langfuseUpdateParent: false,\\r\\n\\t\\t },\\r\\n\\t\\t },\\r\\n\\t})\\r\\n\\r\\n\\t// ... other code\\r\\n}\\r\\n```\\r\\n\\r\\nThe docs state\\r\\n\\r\\n> The id of the observation to which the score should be attached. Pass from backend trace. E.g. {generation, event, span}.id\\r\\n\\r\\nBut how and where do I get the observationId from, taking into account that I use `streamText` (<https://sdk.vercel.ai/docs/reference/ai-sdk-core/stream-text>)\\r\\n\\r\\n```js\\r\\n// source: https://langfuse.com/docs/sdk/typescript/guide-web#create-scores\\r\\n\\r\\n// pass traceId and observationId to front end\\r\\nawait langfuseWeb.score({\\r\\n traceId: message.traceId,\\r\\n observationId: message.observationId, <<<-- where do I get this?\\r\\n name: \\"user-feedback\\",\\r\\n value: 1,\\r\\n comment: \\"I like how personalized the response is\\",\\r\\n});\\r\\n```\\r\\n\\r"}, {"type": "text", "text": "Another question I could not answer myself is where (= which \\"layer\\". I guess generation, event, span is the correct wording) a score should be attached to. Is there any best practice recommendation or should I always go for the observation (`ai.streamText.doStream` in this example _if_ that\'s the observationId I can obtain)?\\r\\n\\r\\n![image](https://github.com/user-attachments/assets/e103abd2-3fd5-4607-9c09-6d0ad2b15393)\\r\\n\\r\\nThank you very much for your time and help!\\nComments:\\n\\n Comment #1 from @dosubot on March 26, 2025:\\n\\n <!-- Greeting -->\\nHey @natterstefan! I\'m here to help you with any bugs, questions, or even becoming a contributor. Let\'s squash those bugs together! πŸ€–\\n\\n<!-- Answer -->\\nTo obtain the `observationId` when using the `streamText` function in the Vercel AI SDK, you should look for it in the response from the `streamText` function. The SDK typically uses the ID from the provider response if available, or it generates one if not. Ensure that your telemetry setup is configured to pass the `observationId` from the backend trace, specifically from the context of the `streamText` execution.\\n\\nRegarding where to attach the score, it is generally recommended to attach it to the most relevant observation. In your case, if `ai.streamText.doStream` is the observation you can obtain, it would be appropriate to attach the score there. This aligns with best practices, as it ensures the score is associated with the specific operation or event you are evaluating.\\n"}, {"type": "text", "text": "\\nFor more detailed guidance, you might want to refer to the Langfuse documentation on creating scores and handling observations <sup>[[1]](https://langfuse.com/docs/sdk/typescript)</sup><sup>[[2]](https://langfuse.com/docs/sdk/typescript/guide)</sup>.\\n\\n\\n<!-- Dosu Comment Footer -->\\n*To continue talking to [Dosu](https://dosu.dev), mention @dosu.*\\n\\n---\\n<sup>Help Dosu learn! Give it feedback: [Great Response](https://app.dosu.dev/response-feedback/68b78af0-2c34-4867-b75c-008bcf449b81?feedback_type=great_response) | [Irrelevant Answer](https://app.dosu.dev/response-feedback/68b78af0-2c34-4867-b75c-008bcf449b81?feedback_type=irrelevant_answer) | [Incorrect Sources](https://app.dosu.dev/response-feedback/68b78af0-2c34-4867-b75c-008bcf449b81?feedback_type=incorrect_sources) | [Too Verbose](https://app.dosu.dev/response-feedback/68b78af0-2c34-4867-b75c-008bcf449b81?feedback_type=too_verbose) | [Hallucination](https://app.dosu.dev/response-feedback/68b78af0-2c34-4867-b75c-008bcf449b81?feedback_type=hallucination) | [Bug Report](https://app.dosu.dev/response-feedback/68b78af0-2c34-4867-b75c-008bcf449b81?feedback_type=bug_report) | [Other](https://app.dosu.dev/response-feedback/68b78af0-2c34-4867-b75c-008bcf449b81?feedback_type=other)</sup>\\n\\n\\n Comment #2 from @marcklingen on March 26, 2025:\\n\\n This seems like a feature request for the AI SDK as you\'d need the SDK to expose the span ids of each of the requests.\\n\\n Replies to comment #2:\\n\\n Reply #1 to comment #2 from marcklingen on March 27, 2025:\\n\\n Hi @marcklingen,\\r\\n\\r\\nthanks for your help. Alright, I [moved (aka copied) the discussion to their repo](https://github.com/vercel/ai/discussions/5415) and hope they can help me.\\r\\n\\r\\nIs @dosubot correct with \\"_(...) In your case, if ai.streamText.doStream is the observation you can obtain, it would be appropriate to attach the score there._\\"\\r\\n\\r"}, {"type": "text", "text": "\\n<sup>Help Dosu learn! Give it feedback: [Great Response](https://app.dosu.dev/response-feedback/d9fcd303-8526-48dc-85b4-1432b3d12429?feedback_type=great_response) | [Irrelevant Answer](https://app.dosu.dev/response-feedback/d9fcd303-8526-48dc-85b4-1432b3d12429?feedback_type=irrelevant_answer) | [Incorrect Sources](https://app.dosu.dev/response-feedback/d9fcd303-8526-48dc-85b4-1432b3d12429?feedback_type=incorrect_sources) | [Too Verbose](https://app.dosu.dev/response-feedback/d9fcd303-8526-48dc-85b4-1432b3d12429?feedback_type=too_verbose) | [Hallucination](https://app.dosu.dev/response-feedback/d9fcd303-8526-48dc-85b4-1432b3d12429?feedback_type=hallucination) | [Bug Report](https://app.dosu.dev/response-feedback/d9fcd303-8526-48dc-85b4-1432b3d12429?feedback_type=bug_report) | [Other](https://app.dosu.dev/response-feedback/d9fcd303-8526-48dc-85b4-1432b3d12429?feedback_type=other)</sup>\\n\\n\\n Reply #3 to comment #2 from marcklingen on March 27, 2025:\\n\\n Hi @marcklingen and @dosubot,\\r\\n\\r\\nFollow up question (once I get the `generation.id`/`response.id` whatever vercel/ai calls it): how do I attach it to the current trace to make sure the observation in langfuse has the same id as the score in my code uses?\\n\\n Reply #4 to comment #2 from marcklingen on March 27, 2025:\\n\\n <!-- Answer -->\\nTo attach the `generation.id` or `response.id` from Vercel AI to the current trace in Langfuse, use the `generation.id` as the `observationId` when logging user feedback. This ensures that the observation has the same ID as the score used in your code. Here\'s how you can do it:\\n\\n1. Add the `generation.id` to the streaming response as a custom header:\\n\\n ```javascript\\n return new StreamingTextResponse(stream, {\\n headers: {\\n \\"X-Message-Id\\": lfGeneration.id,"}], "type": "content"}}]}',
+ role: "assistant",
+ tool_calls: null,
+ function_call: null,
+ },
+ },
+ ],
+ usage: {
+ prompt_tokens: 17,
+ completion_tokens: 19665,
+ total_tokens: 19682,
+ },
+ system_fingerprint: "inkeep-fp-17da6d66-ec37-4217-b055-653078fa0344",
+ object: "chat.completion",
+ service_tier: null,
+ created: 1788942428,
+ },
+ content: [
+ {
+ type: "text",
+ text: '{"content": [{"type": "document", "record_type": "documentation", "url": "https://langfuse.com/docs/evaluation/evaluation-methods/scores-via-sdk", "title": "Scores via API/SDK", "source": {"content": [{"type": "text", "text": "Scores via API/SDK - Langfuse\\n[Docs](/docs)\\n[Evaluation](/docs/evaluation/overview)\\n[Evaluation Methods](/docs/evaluation/evaluation-methods/llm-as-a-judge)\\n[Scores via API/SDK](/docs/evaluation/evaluation-methods/scores-via-sdk)\\n# Scores via API/SDK\\nYou can use the Langfuse SDKs or API to add [scores](/docs/evaluation/scores/overview) to traces, observations, sessions and dataset runs. This is an evaluation method that allows you to set up custom evaluation workflows and extend the scoring capabilities of Langfuse. See the [data model](/docs/evaluation/scores/data-model#scores) for full details on the score object.\\nIf you want Langfuse to run deterministic Python or TypeScript logic for you, use [code evaluators](/docs/evaluation/evaluation-methods/code-evaluators). Use this page when your application, pipeline, or CI job computes scores and sends them to Langfuse via API or SDK.\\n## Ingesting Scores via API/SDKs\\nScores can be attached at different levels of granularity: to individual traces, to specific observations within a trace, or to full sessions.\\nSee the [API reference](/docs/api) for full details on the POST and GET endpoints for scores and score configs.\\n### Trace or Observation-level Scores\\nYou can add scores via the Langfuse SDKs or API. Scores can take one of four data types: **Numeric** , **Categorical** , **Boolean** , or **Text**. See [Score Types](/docs/evaluation/scores/overview#score-types) for details.\\nIf a score is ingested manually using a `trace_id` to link the score to a trace, it is not necessary to wait until the trace has been created. The score will show up in the scores table and will be linked to the trace once the trace with the same `trace_id` is created.\\nHere are examples by `Score` data types."}, {"type": "text", "text": "\\nFor trace and observation scores, `trace_id`/`traceId` is required and `observation_id`/`observationId` is optional. If you attach a score to an observation, always provide both the observation ID and the corresponding trace ID.\\nPython SDKJS/TS SDKAPI\\nNumericCategoricalBooleanText\\nNumeric score values must be provided as float.\\n \\n \\n```\\nfrom langfuse import get_client\\nlangfuse = get_client()\\n///# Method 1: Score via low-level method\\nlangfuse.create_score(\\n name=\\"correctness\\",\\n value=0.9,\\n trace_id=\\"trace_id_here\\",\\n observation_id=\\"observation_id_here\\", ///# optional\\n data_type=\\"NUMERIC\\", ///# optional, inferred if not provided\\n comment=\\"Factually correct\\", ///# optional\\n)\\n///# Method 2: Score current span/generation (within context)\\nwith langfuse.start_as_current_observation(as_type=\\"span\\", name=\\"my-operation\\") as span:\\n # Score the current span\\n span.score(\\n name=\\"correctness\\",\\n value=0.9,\\n data_type=\\"NUMERIC\\",\\n comment=\\"Factually correct\\"\\n )\\n # Score the trace\\n span.score_trace(\\n name=\\"overall_quality\\",\\n value=0.95,\\n data_type=\\"NUMERIC\\"\\n )\\n///# Method 3: Score via the current context\\nwith langfuse.start_as_current_observation(as_type=\\"span\\", name=\\"my-operation\\"):\\n # Score the current span\\n langfuse.score_current_span(\\n name=\\"correctness\\",\\n value=0.9,\\n data_type=\\"NUMERIC\\",\\n comment=\\"Factually correct\\"\\n )\\n # Score the trace\\n langfuse.score_current_trace(\\n name=\\"overall_quality\\",\\n value=0.95,\\n data_type=\\"NUMERIC\\"\\n )\\n```\\nCategorical score values must be provided as strings.\\n \\n \\n```\\nfrom langfuse import get_client\\nlangfuse = get_client()\\n///# Method 1: Score via low-level method\\nlangfuse.create_score(\\n name=\\"accuracy\\",\\n value=\\"partially correct\\",\\n trace_id=\\"trace_id_here\\",\\n observation_id=\\"observation_id_here\\", ///# optional\\n data_type=\\"CATEGORICAL\\", ///# optional, inferred if not provided\\n comment=\\"Some factual errors\\", ///# optional\\n)"}, {"type": "text", "text": "\\n///# Method 2: Score current span/generation (within context)\\nwith langfuse.start_as_current_observation(as_type=\\"span\\", name=\\"my-operation\\") as span:\\n # Score the current span\\n span.score(\\n name=\\"accuracy\\",\\n value=\\"partially correct\\",\\n data_type=\\"CATEGORICAL\\",\\n comment=\\"Some factual errors\\"\\n )\\n # Score the trace\\n span.score_trace(\\n name=\\"overall_quality\\",\\n value=\\"partially correct\\",\\n data_type=\\"CATEGORICAL\\"\\n )\\n///# Method 3: Score via the current context\\nwith langfuse.start_as_current_observation(as_type=\\"span\\", name=\\"my-operation\\"):\\n # Score the current span\\n langfuse.score_current_span(\\n name=\\"accuracy\\",\\n value=\\"partially correct\\",\\n data_type=\\"CATEGORICAL\\",\\n comment=\\"Some factual errors\\"\\n )\\n # Score the trace\\n langfuse.score_current_trace(\\n name=\\"overall_quality\\",\\n value=\\"partially correct\\",\\n data_type=\\"CATEGORICAL\\"\\n )\\n```\\nBoolean scores must be provided as a float, where `1` is true and `0` is false. When read via the [v3 scores API](https://api.reference.langfuse.com/#tag/scoresv3/GET/api/public/v3/scores), the value is returned as a boolean. See [API reference](/docs/api) for more details on POST/GET scores endpoints.\\n \\n \\n```\\nfrom langfuse import get_client\\nlangfuse = get_client()\\n///# Method 1: Score via low-level method\\nlangfuse.create_score(\\n name=\\"helpfulness\\",\\n value=0, ///# 0 or 1\\n trace_id=\\"trace_id_here\\",\\n observation_id=\\"observation_id_here\\", ///# optional\\n data_type=\\"BOOLEAN\\", ///# required, numeric values without data type would be inferred as NUMERIC\\n comment=\\"Incorrect answer\\", ///# optional\\n)\\n///# Method 2: Score current span/generation (within context)\\nwith langfuse.start_as_current_observation(as_type=\\"span\\", name=\\"my-operation\\") as span:\\n # Score the current span"}, {"type": "text", "text": "\\n span.score(\\n name=\\"helpfulness\\",\\n value=1, ///# 0 or 1\\n data_type=\\"BOOLEAN\\",\\n comment=\\"Very helpful response\\"\\n )\\n # Score the trace\\n span.score_trace(\\n name=\\"overall_quality\\",\\n value=1, ///# 0 or 1\\n data_type=\\"BOOLEAN\\"\\n )\\n///# Method 3: Score via the current context\\nwith langfuse.start_as_current_observation(as_type=\\"span\\", name=\\"my-operation\\"):\\n # Score the current span\\n langfuse.score_current_span(\\n name=\\"helpfulness\\",\\n value=1, ///# 0 or 1\\n data_type=\\"BOOLEAN\\",\\n comment=\\"Very helpful response\\"\\n )\\n # Score the trace\\n langfuse.score_current_trace(\\n name=\\"overall_quality\\",\\n value=1, ///# 0 or 1\\n data_type=\\"BOOLEAN\\"\\n )\\n```\\nText score values must be provided as strings between 1 and 500 characters.\\n \\n \\n```\\nfrom langfuse import get_client\\nlangfuse = get_client()\\n///# Method 1: Score via low-level method\\nlangfuse.create_score(\\n name=\\"reviewer_notes\\",\\n value=\\"The response was helpful but could be more concise.\\",\\n trace_id=\\"trace_id_here\\",\\n observation_id=\\"observation_id_here\\", ///# optional\\n data_type=\\"TEXT\\", ///# optional, inferred if not provided\\n comment=\\"Reviewed by QA team\\", ///# optional\\n)\\n///# Method 2: Score current span/generation (within context)\\nwith langfuse.start_as_current_observation(as_type=\\"span\\", name=\\"my-operation\\") as span:\\n # Score the current span\\n span.score(\\n name=\\"reviewer_notes\\",\\n value=\\"The response was helpful but could be more concise.\\",\\n data_type=\\"TEXT\\",\\n comment=\\"Reviewed by QA team\\"\\n )\\n # Score the trace\\n span.score_trace(\\n name=\\"overall_notes\\",\\n value=\\"Good quality overall, minor formatting issues.\\",\\n data_type=\\"TEXT\\"\\n )\\n///# Method 3: Score via the current context\\nwith langfuse.start_as_current_observation(as_type=\\"span\\", name=\\"my-operation\\"):\\n # Score the current span"}, {"type": "text", "text": "\\n langfuse.score_current_span(\\n name=\\"reviewer_notes\\",\\n value=\\"The response was helpful but could be more concise.\\",\\n data_type=\\"TEXT\\",\\n comment=\\"Reviewed by QA team\\"\\n )\\n # Score the trace\\n langfuse.score_current_trace(\\n name=\\"overall_notes\\",\\n value=\\"Good quality overall, minor formatting issues.\\",\\n data_type=\\"TEXT\\"\\n )\\n```\\nNumericCategoricalBooleanText\\nNumeric score values must be provided as float.\\n \\n \\n```\\nimport { LangfuseClient } from \\"@langfuse/client\\";\\nconst langfuse = new LangfuseClient();\\nlangfuse.score.create({\\n id: \\"unique_id\\", // optional, can be used as an idempotency key to update the score subsequently\\n traceId: message.traceId,\\n observationId: message.generationId, // optional\\n name: \\"correctness\\",\\n value: 0.9,\\n dataType: \\"NUMERIC\\", // optional, inferred if not provided\\n comment: \\"Factually correct\\", // optional\\n});\\n// Flush the scores in short-lived environments\\nawait langfuse.flush();\\n```\\nCategorical score values must be provided as strings.\\n \\n \\n```\\nimport { LangfuseClient } from \\"@langfuse/client\\";\\nconst langfuse = new LangfuseClient();\\nlangfuse.score.create({\\n id: \\"unique_id\\", // optional, can be used as an idempotency key to update the score subsequently\\n traceId: message.traceId,\\n observationId: message.generationId, // optional\\n name: \\"accuracy\\",\\n value: \\"partially correct\\",\\n dataType: \\"CATEGORICAL\\", // optional, inferred if not provided\\n comment: \\"Factually correct\\", // optional\\n});\\n// Flush the scores in short-lived environments\\nawait langfuse.flush();\\n```"}, {"type": "text", "text": "\\nBoolean scores must be provided as a float, where `1` is true and `0` is false. When read via the [v3 scores API](https://api.reference.langfuse.com/#tag/scoresv3/GET/api/public/v3/scores), the value is returned as a boolean. See [API reference](/docs/api) for more details on POST/GET scores endpoints.\\n \\n \\n```\\nimport { LangfuseClient } from \\"@langfuse/client\\";\\nconst langfuse = new LangfuseClient();\\nlangfuse.score.create({\\n id: \\"unique_id\\", // optional, can be used as an idempotency key to update the score subsequently\\n traceId: message.traceId,\\n observationId: message.generationId, // optional\\n name: \\"helpfulness\\",\\n value: 0, // 0 or 1\\n dataType: \\"BOOLEAN\\", // required, numeric values without data type would be inferred as NUMERIC\\n comment: \\"Incorrect answer\\", // optional\\n});\\n// Flush the scores in short-lived environments\\nawait langfuse.flush();\\n```\\nText score values must be provided as strings between 1 and 500 characters.\\n \\n \\n```\\nimport { LangfuseClient } from \\"@langfuse/client\\";\\nconst langfuse = new LangfuseClient();\\nlangfuse.score.create({\\n id: \\"unique_id\\", // optional, can be used as an idempotency key to update the score subsequently\\n traceId: message.traceId,\\n observationId: message.generationId, // optional\\n name: \\"reviewer_notes\\",\\n value: \\"The response was helpful but could be more concise.\\",\\n dataType: \\"TEXT\\", // optional, inferred if not provided\\n comment: \\"Reviewed by QA team\\", // optional\\n});\\n// Flush the scores in short-lived environments\\nawait langfuse.flush();\\n```"}, {"type": "text", "text": "\\nYou can also create scores directly via the [REST API](https://api.reference.langfuse.com/#tag/legacyscorev1/POST/api/public/scores). Authenticate using HTTP Basic Auth with your Langfuse Public Key as username and Secret Key as password.\\nNumericCategoricalBooleanText\\nNumeric score values must be provided as float.\\n \\n \\n```\\ncurl -X POST https://cloud.langfuse.com/api/public/scores \\\\\\n -u \\"pk-lf-...\\":\\"sk-lf-...\\" \\\\\\n -H \\"Content-Type: application/json\\" \\\\\\n -d \'{\\n \\"traceId\\": \\"trace_id_here\\",\\n \\"observationId\\": \\"observation_id_here\\",\\n \\"name\\": \\"correctness\\",\\n \\"value\\": 0.9,\\n \\"dataType\\": \\"NUMERIC\\",\\n \\"comment\\": \\"Factually correct\\"\\n }\'\\n```\\nCategorical score values must be provided as strings.\\n \\n \\n```\\ncurl -X POST https://cloud.langfuse.com/api/public/scores \\\\\\n -u \\"pk-lf-...\\":\\"sk-lf-...\\" \\\\\\n -H \\"Content-Type: application/json\\" \\\\\\n -d \'{\\n \\"traceId\\": \\"trace_id_here\\",\\n \\"observationId\\": \\"observation_id_here\\",\\n \\"name\\": \\"accuracy\\",\\n \\"value\\": \\"partially correct\\",\\n \\"dataType\\": \\"CATEGORICAL\\",\\n \\"comment\\": \\"Some factual errors\\"\\n }\'\\n```\\nBoolean scores must be provided as a float, where `1` is true and `0` is false. When read via the [v3 scores API](https://api.reference.langfuse.com/#tag/scoresv3/GET/api/public/v3/scores), the value is returned as a boolean.\\n \\n \\n```\\ncurl -X POST https://cloud.langfuse.com/api/public/scores \\\\\\n -u \\"pk-lf-...\\":\\"sk-lf-...\\" \\\\\\n -H \\"Content-Type: application/json\\" \\\\\\n -d \'{\\n \\"traceId\\": \\"trace_id_here\\",\\n \\"observationId\\": \\"observation_id_here\\",\\n \\"name\\": \\"helpfulness\\",\\n \\"value\\": 0,\\n \\"dataType\\": \\"BOOLEAN\\",\\n \\"comment\\": \\"Incorrect answer\\"\\n }\'\\n```\\nText score values must be provided as strings between 1 and 500 characters.\\n \\n \\n```\\ncurl -X POST https://cloud.langfuse.com/api/public/scores \\\\\\n -u \\"pk-lf-...\\":\\"sk-lf-...\\" \\\\"}, {"type": "text", "text": "\\n comment=\\"Factually correct\\",\\n config_id=\\"12345-6565-3453654-43543\\",\\n data_type=\\"CATEGORICAL\\"\\n )\\n```\\nWhen ingesting boolean scores, you can provide the value as a float. If you provide a configId, the score\'s name and config\'s name must match as well as their data types.\\n \\n \\n```\\nfrom langfuse import get_client\\nlangfuse = get_client()\\n///# Method 1: Score via low-level method\\nlangfuse.create_score(\\n trace_id=\\"trace_id_here\\",\\n observation_id=\\"observation_id_here\\", ///# optional\\n name=\\"helpfulness\\",\\n value=1,\\n comment=\\"Factually correct\\", ///# optional\\n score_id=\\"unique_id\\", ///# optional, can be used as an idempotency key to update the score subsequently\\n config_id=\\"93547-6565-3453654-43543\\", ///# optional, can be used to infer the score data type and validate the score value\\n data_type=\\"BOOLEAN\\" # optional, possibly inferred\\n)\\n///# Method 2: Score within context\\nwith langfuse.start_as_current_observation(as_type=\\"span\\", name=\\"my-operation\\") as span:\\n span.score(\\n name=\\"helpfulness\\",\\n value=1,\\n comment=\\"Factually correct\\",\\n config_id=\\"93547-6565-3453654-43543\\",\\n data_type=\\"BOOLEAN\\"\\n )\\n```\\nWhen ingesting text scores, provide the value as a string (1–500 characters). If you provide a configId, the score\'s name and config\'s name must match as well as their data types.\\n \\n \\n```\\nfrom langfuse import get_client\\nlangfuse = get_client()\\n///# Method 1: Score via low-level method\\nlangfuse.create_score(\\n trace_id=\\"trace_id_here\\",\\n observation_id=\\"observation_id_here\\", ///# optional\\n name=\\"reviewer_notes\\",\\n value=\\"The response was helpful but could be more concise.\\",\\n comment=\\"Reviewed by QA team\\", ///# optional"}, {"type": "text", "text": "\\n score_id=\\"unique_id\\", ///# optional, can be used as an idempotency key to update the score subsequently\\n config_id=\\"24680-6565-3453654-43543\\", ///# optional\\n data_type=\\"TEXT\\" # optional, possibly inferred\\n)\\n///# Method 2: Score within context\\nwith langfuse.start_as_current_observation(as_type=\\"span\\", name=\\"my-operation\\") as span:\\n span.score(\\n name=\\"reviewer_notes\\",\\n value=\\"The response was helpful but could be more concise.\\",\\n comment=\\"Reviewed by QA team\\",\\n config_id=\\"24680-6565-3453654-43543\\",\\n data_type=\\"TEXT\\"\\n )\\n```\\nNumeric ScoresCategorical ScoresBoolean ScoresText Scores\\nWhen ingesting numeric scores, you can provide the value as a float. If you provide a configId, the score value will be validated against the config\'s numeric range, which might be defined by a minimum and/or maximum value.\\n \\n \\n```\\nimport { LangfuseClient } from \\"@langfuse/client\\";\\nconst langfuse = new LangfuseClient();\\nlangfuse.score.create({\\n traceId: message.traceId,\\n observationId: message.generationId, // optional\\n name: \\"accuracy\\",\\n value: 0.9,\\n comment: \\"Factually correct\\", // optional\\n id: \\"unique_id\\", // optional, can be used as an idempotency key to update the score subsequently\\n configId: \\"78545-6565-3453654-43543\\", // optional, to ensure that the score follows a specific min/max value range\\n dataType: \\"NUMERIC\\", // optional, possibly inferred\\n});\\n// Flush the scores in short-lived environments\\nawait langfuse.flush();\\n```\\nCategorical scores are used to evaluate data that falls into specific categories. When ingesting categorical scores, you can provide the value as a string. If you provide a configId, the score value will be validated against the config\'s categories.\\n \\n \\n```"}, {"type": "text", "text": "\\nimport { LangfuseClient } from \\"@langfuse/client\\";\\nconst langfuse = new LangfuseClient();\\nlangfuse.score.create({\\n traceId: message.traceId,\\n observationId: message.generationId, // optional\\n name: \\"correctness\\",\\n value: \\"correct\\",\\n comment: \\"Factually correct\\", // optional\\n id: \\"unique_id\\", // optional, can be used as an idempotency key to update the score subsequently\\n configId: \\"12345-6565-3453654-43543\\", // optional, to ensure that a score maps to a specific category defined in a score config\\n dataType: \\"CATEGORICAL\\", // optional, possibly inferred\\n});\\n// Flush the scores in short-lived environments\\nawait langfuse.flush();\\n```\\nWhen ingesting boolean scores, you can provide the value as a float. If you provide a configId, the score\'s name and config\'s name must match as well as their data types.\\n \\n \\n```\\nimport { LangfuseClient } from \\"@langfuse/client\\";\\nconst langfuse = new LangfuseClient();\\nlangfuse.score.create({\\n traceId: message.traceId,\\n observationId: message.generationId, // optional\\n name: \\"helpfulness\\",\\n value: 1,\\n comment: \\"Factually correct\\", // optional\\n id: \\"unique_id\\", // optional, can be used as an idempotency key to update the score subsequently\\n configId: \\"93547-6565-3453654-43543\\", // optional, can be used to infer the score data type and validate the score value\\n dataType: \\"BOOLEAN\\", // optional, possibly inferred\\n});\\n// Flush the scores in short-lived environments\\nawait langfuse.flush();\\n```\\nWhen ingesting text scores, provide the value as a string (1–500 characters). If you provide a configId, the score\'s name and config\'s name must match as well as their data types.\\n \\n \\n```\\nimport { LangfuseClient } from \\"@langfuse/client\\";\\nconst langfuse = new LangfuseClient();\\nlangfuse.score.create({\\n traceId: message.traceId,\\n observationId: message.generationId, // optional\\n name: \\"reviewer_notes\\",\\n value: \\"The response was helpful but could be more concise.\\","}, {"type": "text", "text": "\\nWhen ingesting boolean scores, you can provide the value as a float. If you provide a configId, the score\'s name and config\'s name must match as well as their data types.\\n \\n \\n```\\ncurl -X POST https://cloud.langfuse.com/api/public/scores \\\\\\n -u \\"pk-lf-...\\":\\"sk-lf-...\\" \\\\\\n -H \\"Content-Type: application/json\\" \\\\\\n -d \'{\\n \\"id\\": \\"unique_id\\",\\n \\"traceId\\": \\"trace_id_here\\",\\n \\"observationId\\": \\"observation_id_here\\",\\n \\"name\\": \\"helpfulness\\",\\n \\"value\\": 1,\\n \\"dataType\\": \\"BOOLEAN\\",\\n \\"configId\\": \\"93547-6565-3453654-43543\\",\\n \\"comment\\": \\"Factually correct\\"\\n }\'\\n```\\nWhen ingesting text scores, provide the value as a string (1–500 characters). If you provide a configId, the score\'s name and config\'s name must match as well as their data types.\\n \\n \\n```\\ncurl -X POST https://cloud.langfuse.com/api/public/scores \\\\\\n -u \\"pk-lf-...\\":\\"sk-lf-...\\" \\\\\\n -H \\"Content-Type: application/json\\" \\\\\\n -d \'{\\n \\"id\\": \\"unique_id\\",\\n \\"traceId\\": \\"trace_id_here\\",\\n \\"observationId\\": \\"observation_id_here\\",\\n \\"name\\": \\"reviewer_notes\\",\\n \\"value\\": \\"The response was helpful but could be more concise.\\",\\n \\"dataType\\": \\"TEXT\\",\\n \\"configId\\": \\"24680-6565-3453654-43543\\",\\n \\"comment\\": \\"Reviewed by QA team\\"\\n }\'\\n```\\nSee [API reference](/docs/api) for more details on POST/GET score configs endpoints.\\n### Inferred Score Properties\\nCertain score properties might be inferred based on your input:\\n * **If you don\'t provide a score data type** it will always be inferred. See tables below for details."}, {"type": "text", "text": "\\n * **For boolean and categorical scores** , we will provide the score value in both numerical and string format where possible. The score value format that is not provided as input, i.e. the translated value is referred to as the inferred value in the tables below.\\n * **On read via the[v3 scores API](https://api.reference.langfuse.com/#tag/scoresv3/GET/api/public/v3/scores)**, boolean scores are returned as a single boolean `value`.\\n * **For categorical scores** , the string representation is always provided and a numerical mapping of the category will be produced only if a `ScoreConfig` was provided.\\nDetailed Examples:\\nNumeric ScoresCategorical ScoresBoolean ScoresText Scores\\nFor example, let\'s assume you\'d like to ingest a numeric score to measure **accuracy**. We have included a table of possible score ingestion scenarios below.\\nValue| Data Type| Config Id| Description| Inferred Data Type| Valid \\n---|---|---|---|---|--- \\n`0.9`| `Null`| `Null`| Data type is inferred| `NUMERIC`| Yes \\n`0.9`| `NUMERIC`| `Null`| No properties inferred| | Yes \\n`depth`| `NUMERIC`| `Null`| Error: data type of value does not match provided data type| | No \\n`0.9`| `NUMERIC`| `78545`| No properties inferred| | Conditional on config validation \\n`0.9`| `Null`| `78545`| Data type inferred| `NUMERIC`| Conditional on config validation \\n`depth`| `NUMERIC`| `78545`| Error: data type of value does not match provided data type| | No \\n \\nFor example, let\'s assume you\'d like to ingest a categorical score to measure **correctness**. We have included a table of possible score ingestion scenarios below."}, {"type": "text", "text": "Value| Data Type| Config Id| Description| Inferred Data Type| Inferred Value representation| Valid \\n---|---|---|---|---|---|--- \\n`correct`| `Null`| `Null`| Data type is inferred| `CATEGORICAL`| | Yes \\n`correct`| `CATEGORICAL`| `Null`| No properties inferred| | | Yes \\n`1`| `CATEGORICAL`| `Null`| Error: data type of value does not match provided data type| | | No \\n`correct`| `CATEGORICAL`| `12345`| Numeric value inferred| | `4` numeric config category mapping| Conditional on config validation \\n`correct`| `NULL`| `12345`| Data type inferred| `CATEGORICAL`| | Conditional on config validation \\n`1`| `CATEGORICAL`| `12345`| Error: data type of value does not match provided data type| | | No \\n \\nFor example, let\'s assume you\'d like to ingest a boolean score to measure **helpfulness**. We have included a table of possible score ingestion scenarios below.\\nValue| Data Type| Config Id| Description| Inferred Data Type| Inferred Value representation| Valid \\n---|---|---|---|---|---|--- \\n`1`| `BOOLEAN`| `Null`| Value\'s string equivalent inferred| | `True`| Yes \\n`true`| `BOOLEAN`| `Null`| Error: data type of value does not match provided data type| | | No \\n`3`| `BOOLEAN`| `Null`| Error: boolean data type expects `0` or `1` as input value| | | No \\n`0.9`| `Null`| `93547`| Data type and value\'s string equivalent inferred| `BOOLEAN`| `True`| Conditional on config validation \\n`depth`| `BOOLEAN`| `93547`| Error: data type of value does not match provided data type| | | No \\n \\nFor example, let\'s assume you\'d like to ingest a text score to capture **reviewer notes**. Text scores must be non-empty strings of at most 500 characters.\\nValue| Data Type| Config Id| Description| Inferred Data Type| Valid "}, {"type": "text", "text": "---|---|---|---|---|--- \\n`\\"Good response\\"`| `Null`| `Null`| Data type is inferred| `TEXT`| Yes \\n`\\"Good response\\"`| `TEXT`| `Null`| No properties inferred| | Yes \\n`0.9`| `TEXT`| `Null`| Error: data type of value does not match provided data type| | No \\n`\\"Good response\\"`| `TEXT`| `24680`| No properties inferred| | Conditional on config validation \\n`\\"Good response\\"`| `Null`| `24680`| Data type inferred| `TEXT`| Conditional on config validation \\n`\\"\\"`| `TEXT`| `Null`| Error: text scores must be non-empty| | No \\n \\n## Update Existing Scores via API/SDKs\\nWhen creating a score, you can provide an optional `id` (JS/TS) / `score_id` (Python) parameter. A re-ingested score overwrites an existing one only when its `id`, `name`, and `timestamp` (at date granularity, `toDate(timestamp)`) all match β€” a matching `id` alone is not sufficient, and any difference creates a duplicate instead of an update.\\nTo overwrite a score without fetching existing scores first, set a stable `id` as an idempotency key when initially creating the score, and keep the `name` and `timestamp` unchanged on subsequent calls. See Preventing Duplicate Scores for details.\\n## Related guides\\nGuides\\n * [Collecting user feedback\\nCapture explicit and implicit user feedback as scores on your traces, including in-app ratings sent via our Browser SDK.\\n](/guides/user-feedback-loop)\\n * [Custom evaluation data pipeline\\nContinuously monitor quality by fetching traces, running custom evaluations, and ingesting scores back into Langfuse.\\n](/guides/cookbook/example_external_evaluation_pipelines)\\n * [Guardrails and security checks"}, {"type": "text", "text": "\\nCheck whether output contains a certain keyword, matches a required structure or format, or exceeds a length limit.\\n](/guides/cookbook/security-and-guardrails)\\n * [Custom internal workflow tooling\\nBuild internal tooling for human-in-the-loop workflows, ingesting scores back into Langfuse following your custom schema.\\n](/guides/human-in-the-loop-scoring)\\n * [Session-level quality tracking\\nScore full conversations, such as support chats or agent threads, by attaching scores via sessionId in the SDK or API.\\n](/guides/cookbook/example_evaluating_multi_turn_conversations)\\n* * *\\n* * *\\nLast edited 7/22/2026\\n"}], "type": "content"}}, {"type": "document", "record_type": "documentation", "url": "https://langfuse.com/docs/evaluation/evaluation-methods/scores-via-sdk", "title": "Scores via API/SDK", "source": {"content": [{"type": "text", "text": "# Scores via API/SDK\\nYou can use the Langfuse SDKs or API to add [scores](/docs/evaluation/scores/overview) to traces, observations, sessions and dataset runs. This is an evaluation method that allows you to set up custom evaluation workflows and extend the scoring capabilities of Langfuse. See the [data model](/docs/evaluation/scores/data-model#scores) for full details on the score object.\\nIf you want Langfuse to run deterministic Python or TypeScript logic for you, use [code evaluators](/docs/evaluation/evaluation-methods/code-evaluators). Use this page when your application, pipeline, or CI job computes scores and sends them to Langfuse via API or SDK.\\n\\n## Ingesting Scores via API/SDKs\\nScores can be attached at different levels of granularity: to individual traces, to specific observations within a trace, or to full sessions.\\nSee the [API reference](/docs/api) for full details on the POST and GET endpoints for scores and score configs.\\n### Trace or Observation-level Scores\\nYou can add scores via the Langfuse SDKs or API. Scores can take one of four data types: **Numeric** , **Categorical** , **Boolean** , or **Text**. See [Score Types](/docs/evaluation/scores/overview#score-types) for details.\\nIf a score is ingested manually using a `trace_id` to link the score to a trace, it is not necessary to wait until the trace has been created. The score will show up in the scores table and will be linked to the trace once the trace with the same `trace_id` is created.\\nHere are examples by `Score` data types.\\nFor trace and observation scores, `trace_id`/`traceId` is required and `observation_id`/`observationId` is optional. If you attach a score to an observation, always provide both the observation ID and the corresponding trace ID.\\nPython SDKJS/TS SDKAPI\\nNumericCategoricalBooleanText\\nNumeric score values must be provided as float.\\n \\n \\n```"}, {"type": "text", "text": "\\nfrom langfuse import get_client\\nlangfuse = get_client()\\n# Method 1: Score via low-level method\\nlangfuse.create_score(\\n name=\\"correctness\\",\\n value=0.9,\\n trace_id=\\"trace_id_here\\",\\n observation_id=\\"observation_id_here\\", # optional\\n data_type=\\"NUMERIC\\", # optional, inferred if not provided\\n comment=\\"Factually correct\\", # optional\\n)\\n# Method 2: Score current span/generation (within context)\\nwith langfuse.start_as_current_observation(as_type=\\"span\\", name=\\"my-operation\\") as span:\\n # Score the current span\\n span.score(\\n name=\\"correctness\\",\\n value=0.9,\\n data_type=\\"NUMERIC\\",\\n comment=\\"Factually correct\\"\\n )\\n # Score the trace\\n span.score_trace(\\n name=\\"overall_quality\\",\\n value=0.95,\\n data_type=\\"NUMERIC\\"\\n )\\n# Method 3: Score via the current context\\nwith langfuse.start_as_current_observation(as_type=\\"span\\", name=\\"my-operation\\"):\\n # Score the current span\\n langfuse.score_current_span(\\n name=\\"correctness\\",\\n value=0.9,\\n data_type=\\"NUMERIC\\",\\n comment=\\"Factually correct\\"\\n )\\n # Score the trace\\n langfuse.score_current_trace(\\n name=\\"overall_quality\\",\\n value=0.95,\\n data_type=\\"NUMERIC\\"\\n )\\n```\\nCategorical score values must be provided as strings.\\n \\n \\n```\\nfrom langfuse import get_client\\nlangfuse = get_client()\\n# Method 1: Score via low-level method\\nlangfuse.create_score(\\n name=\\"accuracy\\",\\n value=\\"partially correct\\",\\n trace_id=\\"trace_id_here\\",\\n observation_id=\\"observation_id_here\\", # optional\\n data_type=\\"CATEGORICAL\\", # optional, inferred if not provided\\n comment=\\"Some factual errors\\", # optional\\n)\\n# Method 2: Score current span/generation (within context)\\nwith langfuse.start_as_current_observation(as_type=\\"span\\", name=\\"my-operation\\") as span:\\n # Score the current span\\n span.score(\\n name=\\"accuracy\\",\\n value=\\"partially correct\\",\\n data_type=\\"CATEGORICAL\\",\\n comment=\\"Some factual errors\\"\\n )\\n # Score the trace\\n span.score_trace(\\n name=\\"overall_quality\\",\\n value=\\"partially correct\\","}, {"type": "text", "text": "\\n data_type=\\"CATEGORICAL\\"\\n )\\n# Method 3: Score via the current context\\nwith langfuse.start_as_current_observation(as_type=\\"span\\", name=\\"my-operation\\"):\\n # Score the current span\\n langfuse.score_current_span(\\n name=\\"accuracy\\",\\n value=\\"partially correct\\",\\n data_type=\\"CATEGORICAL\\",\\n comment=\\"Some factual errors\\"\\n )\\n # Score the trace\\n langfuse.score_current_trace(\\n name=\\"overall_quality\\",\\n value=\\"partially correct\\",\\n data_type=\\"CATEGORICAL\\"\\n )\\n```\\nBoolean scores must be provided as a float, where `1` is true and `0` is false. When read via the [v3 scores API](https://api.reference.langfuse.com/#tag/scoresv3/GET/api/public/v3/scores), the value is returned as a boolean. See [API reference](/docs/api) for more details on POST/GET scores endpoints.\\n \\n \\n```\\nfrom langfuse import get_client\\nlangfuse = get_client()\\n# Method 1: Score via low-level method\\nlangfuse.create_score(\\n name=\\"helpfulness\\",\\n value=0, # 0 or 1\\n trace_id=\\"trace_id_here\\",\\n observation_id=\\"observation_id_here\\", # optional\\n data_type=\\"BOOLEAN\\", # required, numeric values without data type would be inferred as NUMERIC\\n comment=\\"Incorrect answer\\", # optional\\n)\\n# Method 2: Score current span/generation (within context)\\nwith langfuse.start_as_current_observation(as_type=\\"span\\", name=\\"my-operation\\") as span:\\n # Score the current span\\n span.score(\\n name=\\"helpfulness\\",\\n value=1, # 0 or 1\\n data_type=\\"BOOLEAN\\",\\n comment=\\"Very helpful response\\"\\n )\\n # Score the trace\\n span.score_trace(\\n name=\\"overall_quality\\",\\n value=1, # 0 or 1\\n data_type=\\"BOOLEAN\\"\\n )\\n# Method 3: Score via the current context\\nwith langfuse.start_as_current_observation(as_type=\\"span\\", name=\\"my-operation\\"):\\n # Score the current span\\n langfuse.score_current_span(\\n name=\\"helpfulness\\","}, {"type": "text", "text": "\\n value=1, # 0 or 1\\n data_type=\\"BOOLEAN\\",\\n comment=\\"Very helpful response\\"\\n )\\n # Score the trace\\n langfuse.score_current_trace(\\n name=\\"overall_quality\\",\\n value=1, # 0 or 1\\n data_type=\\"BOOLEAN\\"\\n )\\n```\\nText score values must be provided as strings between 1 and 500 characters.\\n \\n \\n```\\nfrom langfuse import get_client\\nlangfuse = get_client()\\n# Method 1: Score via low-level method\\nlangfuse.create_score(\\n name=\\"reviewer_notes\\",\\n value=\\"The response was helpful but could be more concise.\\",\\n trace_id=\\"trace_id_here\\",\\n observation_id=\\"observation_id_here\\", # optional\\n data_type=\\"TEXT\\", # optional, inferred if not provided\\n comment=\\"Reviewed by QA team\\", # optional\\n)\\n# Method 2: Score current span/generation (within context)\\nwith langfuse.start_as_current_observation(as_type=\\"span\\", name=\\"my-operation\\") as span:\\n # Score the current span\\n span.score(\\n name=\\"reviewer_notes\\",\\n value=\\"The response was helpful but could be more concise.\\",\\n data_type=\\"TEXT\\",\\n comment=\\"Reviewed by QA team\\"\\n )\\n # Score the trace\\n span.score_trace(\\n name=\\"overall_notes\\",\\n value=\\"Good quality overall, minor formatting issues.\\",\\n data_type=\\"TEXT\\"\\n )\\n# Method 3: Score via the current context\\nwith langfuse.start_as_current_observation(as_type=\\"span\\", name=\\"my-operation\\"):\\n # Score the current span\\n langfuse.score_current_span(\\n name=\\"reviewer_notes\\",\\n value=\\"The response was helpful but could be more concise.\\",\\n data_type=\\"TEXT\\",\\n comment=\\"Reviewed by QA team\\"\\n )\\n # Score the trace\\n langfuse.score_current_trace(\\n name=\\"overall_notes\\",\\n value=\\"Good quality overall, minor formatting issues.\\",\\n data_type=\\"TEXT\\"\\n )\\n```\\nNumericCategoricalBooleanText\\nNumeric score values must be provided as float.\\n \\n \\n```"}, {"type": "text", "text": "\\nimport { LangfuseClient } from \\"@langfuse/client\\";\\nconst langfuse = new LangfuseClient();\\nlangfuse.score.create({\\n id: \\"unique_id\\", // optional, can be used as an idempotency key to update the score subsequently\\n traceId: message.traceId,\\n observationId: message.generationId, // optional\\n name: \\"correctness\\",\\n value: 0.9,\\n dataType: \\"NUMERIC\\", // optional, inferred if not provided\\n comment: \\"Factually correct\\", // optional\\n});\\n// Flush the scores in short-lived environments\\nawait langfuse.flush();\\n```\\nCategorical score values must be provided as strings.\\n \\n \\n```\\nimport { LangfuseClient } from \\"@langfuse/client\\";\\nconst langfuse = new LangfuseClient();\\nlangfuse.score.create({\\n id: \\"unique_id\\", // optional, can be used as an idempotency key to update the score subsequently\\n traceId: message.traceId,\\n observationId: message.generationId, // optional\\n name: \\"accuracy\\",\\n value: \\"partially correct\\",\\n dataType: \\"CATEGORICAL\\", // optional, inferred if not provided\\n comment: \\"Factually correct\\", // optional\\n});\\n// Flush the scores in short-lived environments\\nawait langfuse.flush();\\n```\\nBoolean scores must be provided as a float, where `1` is true and `0` is false. When read via the [v3 scores API](https://api.reference.langfuse.com/#tag/scoresv3/GET/api/public/v3/scores), the value is returned as a boolean. See [API reference](/docs/api) for more details on POST/GET scores endpoints.\\n \\n \\n```\\nimport { LangfuseClient } from \\"@langfuse/client\\";\\nconst langfuse = new LangfuseClient();\\nlangfuse.score.create({\\n id: \\"unique_id\\", // optional, can be used as an idempotency key to update the score subsequently\\n traceId: message.traceId,\\n observationId: message.generationId, // optional"}, {"type": "text", "text": "\\nnpm install @langfuse/browser\\n```\\n \\n \\n```\\nimport { LangfuseBrowserClient } from \\"@langfuse/browser\\";\\nconst langfuse = new LangfuseBrowserClient({\\n publicKey: process.env.NEXT_PUBLIC_LANGFUSE_PUBLIC_KEY!,\\n baseUrl: process.env.NEXT_PUBLIC_LANGFUSE_BASE_URL, // optional, defaults to https://cloud.langfuse.com\\n});\\nawait langfuse.score({\\n traceId: message.traceId,\\n observationId: message.generationId, // optional\\n id: `user-feedback-${message.traceId}`, // optional, use as an idempotency key\\n name: \\"user-feedback\\",\\n value: 1, // 1 for positive, 0 for negative\\n dataType: \\"BOOLEAN\\",\\n comment: \\"Helpful answer\\", // optional\\n});\\n```\\n`score()` returns `{ id }` after ingestion succeeds. No `flush()` call is required.\\n### Session-level Scores\\nTo score an entire session (without attaching the score to a trace or observation), provide only `session_id` (Python SDK) or `sessionId` (JS/TS SDK and API).\\nPython SDKJS/TS SDKAPI\\n \\n \\n```\\nfrom langfuse import get_client\\nlangfuse = get_client()\\nlangfuse.create_score(\\n name=\\"session_quality\\",\\n value=0.85,\\n session_id=\\"session_id_here\\",\\n data_type=\\"NUMERIC\\",\\n comment=\\"Overall conversation quality\\"\\n)\\n```\\n \\n \\n```\\nimport { LangfuseClient } from \\"@langfuse/client\\";\\nconst langfuse = new LangfuseClient();\\nlangfuse.score.create({\\n name: \\"session_quality\\",\\n value: 0.85,\\n sessionId: \\"session_id_here\\",\\n dataType: \\"NUMERIC\\",\\n comment: \\"Overall conversation quality\\",\\n});\\nawait langfuse.flush();\\n```\\n \\n \\n```\\ncurl -X POST https://cloud.langfuse.com/api/public/scores \\\\\\n -u \\"pk-lf-...\\":\\"sk-lf-...\\" \\\\\\n -H \\"Content-Type: application/json\\" \\\\\\n -d \'{\\n \\"sessionId\\": \\"session_id_here\\",\\n \\"name\\": \\"session_quality\\",\\n \\"value\\": 0.85,\\n \\"dataType\\": \\"NUMERIC\\",\\n \\"comment\\": \\"Overall conversation quality\\"\\n }\'\\n```\\n"}], "type": "content"}}, {"type": "document", "record_type": "documentation", "url": "https://langfuse.com/docs/evaluation/evaluation-methods/scores-via-sdk", "title": "Scores via API/SDK", "source": {"content": [{"type": "text", "text": "## Advanced\\n### Preventing Duplicate Scores\\nBy default, Langfuse allows for multiple scores of the same `name` on the same trace. This is useful if you\'d like to track the evolution of a score over time or if e.g. you\'ve received multiple user feedback scores on the same trace.\\nIn some cases, you want to prevent this behavior or overwrite an existing score. A score is identified by three fields β€” its `id`, its `name`, and its date (`toDate(timestamp)`) β€” and a new score only replaces an existing one when all three match. To achieve this, create an **idempotency key** and pass it as the `id` (JS/TS) / `score_id` (Python) when creating the score (for example `trace_id-score_name`), and keep the `name` and `timestamp` unchanged across calls. If any of the three differs β€” for example the same `id` with a different `name` β€” you get an additional score rather than a replacement.\\nDo not send partial score updates (only the changed fields) and rely on Langfuse to merge them into the existing record. This merge only happens for a limited window after creation, is deprecated, and will be removed. Always send the complete score.\\nSee [How to update traces, observations, and scores](/faq/all/tracing-data-updates#updating-scores) for more details on Langfuse\'s immutable data model.\\n### Enforcing a Score Config\\nScore configs are helpful when you want to standardize your scores for future analysis.\\nTo enforce a score config, you can provide a `configId` when creating a score to reference a `ScoreConfig` that was previously created. `Score Configs` can be defined in the Langfuse UI or via our API. [See our guide on how to create and manage score configs](/faq/all/manage-score-configs).\\nWhenever you provide a `ScoreConfig`, the score data will be validated against the config. The following rules apply:\\n * **Score Name** : Must equal the config\'s name"}, {"type": "text", "text": "\\n * **Score Data Type** : When provided, must match the config\'s data type\\n * **Score Value when Type is`NUMERIC`**: Value must be within the min and max values defined in the config (if provided, min and max are optional and otherwise are assumed as -∞ and +∞ respectively)\\n * **Score Value when Type is`CATEGORICAL`**: Value must map to one of the categories defined in the config\\n * **Score Value when Type is`BOOLEAN`**: Value must equal `0` or `1`\\n * **Score Value when Type is`TEXT`**: Value must be a non-empty string of at most 500 characters\\nPython SDKJS/TS SDKAPI\\nNumeric ScoresCategorical ScoresBoolean ScoresText Scores\\nWhen ingesting numeric scores, you can provide the value as a float. If you provide a configId, the score value will be validated against the config\'s numeric range, which might be defined by a minimum and/or maximum value.\\n \\n \\n```\\nfrom langfuse import get_client\\nlangfuse = get_client()\\n# Method 1: Score via low-level method\\nlangfuse.create_score(\\n trace_id=\\"trace_id_here\\",\\n observation_id=\\"observation_id_here\\", # optional\\n session_id=\\"session_id_here\\", # optional, ID of the session the score relates to\\n name=\\"accuracy\\",\\n value=0.9,\\n comment=\\"Factually correct\\", # optional\\n score_id=\\"unique_id\\", # optional, can be used as an idempotency key to update the score subsequently\\n config_id=\\"78545-6565-3453654-43543\\", # optional, to ensure that the score follows a specific min/max value range\\n data_type=\\"NUMERIC\\" # optional, possibly inferred\\n)\\n# Method 2: Score within context\\nwith langfuse.start_as_current_observation(as_type=\\"span\\", name=\\"my-operation\\") as span:\\n span.score(\\n name=\\"accuracy\\",\\n value=0.9,\\n comment=\\"Factually correct\\",\\n config_id=\\"78545-6565-3453654-43543\\",\\n data_type=\\"NUMERIC\\"\\n )\\n```"}, {"type": "text", "text": "\\nWhen ingesting numeric scores, you can provide the value as a float. If you provide a configId, the score value will be validated against the config\'s numeric range, which might be defined by a minimum and/or maximum value.\\n \\n \\n```\\nimport { LangfuseClient } from \\"@langfuse/client\\";\\nconst langfuse = new LangfuseClient();\\nlangfuse.score.create({\\n traceId: message.traceId,\\n observationId: message.generationId, // optional\\n name: \\"accuracy\\",\\n value: 0.9,\\n comment: \\"Factually correct\\", // optional\\n id: \\"unique_id\\", // optional, can be used as an idempotency key to update the score subsequently\\n configId: \\"78545-6565-3453654-43543\\", // optional, to ensure that the score follows a specific min/max value range\\n dataType: \\"NUMERIC\\", // optional, possibly inferred\\n});\\n// Flush the scores in short-lived environments\\nawait langfuse.flush();\\n```\\nCategorical scores are used to evaluate data that falls into specific categories. When ingesting categorical scores, you can provide the value as a string. If you provide a configId, the score value will be validated against the config\'s categories.\\n \\n \\n```\\nimport { LangfuseClient } from \\"@langfuse/client\\";\\nconst langfuse = new LangfuseClient();\\nlangfuse.score.create({\\n traceId: message.traceId,\\n observationId: message.generationId, // optional\\n name: \\"correctness\\",\\n value: \\"correct\\",\\n comment: \\"Factually correct\\", // optional\\n id: \\"unique_id\\", // optional, can be used as an idempotency key to update the score subsequently\\n configId: \\"12345-6565-3453654-43543\\", // optional, to ensure that a score maps to a specific category defined in a score config\\n dataType: \\"CATEGORICAL\\", // optional, possibly inferred\\n});"}, {"type": "text", "text": "\\nYou can also enforce score configs via the [REST API](https://api.reference.langfuse.com/#tag/legacyscorev1/POST/api/public/scores) by providing a `configId`.\\nNumeric ScoresCategorical ScoresBoolean ScoresText Scores\\nWhen ingesting numeric scores, you can provide the value as a float. If you provide a configId, the score value will be validated against the config\'s numeric range.\\n \\n \\n```\\ncurl -X POST https://cloud.langfuse.com/api/public/scores \\\\\\n -u \\"pk-lf-...\\":\\"sk-lf-...\\" \\\\\\n -H \\"Content-Type: application/json\\" \\\\\\n -d \'{\\n \\"id\\": \\"unique_id\\",\\n \\"traceId\\": \\"trace_id_here\\",\\n \\"observationId\\": \\"observation_id_here\\",\\n \\"name\\": \\"accuracy\\",\\n \\"value\\": 0.9,\\n \\"dataType\\": \\"NUMERIC\\",\\n \\"configId\\": \\"78545-6565-3453654-43543\\",\\n \\"comment\\": \\"Factually correct\\"\\n }\'\\n```\\nCategorical scores are used to evaluate data that falls into specific categories. If you provide a configId, the score value will be validated against the config\'s categories.\\n \\n \\n```\\ncurl -X POST https://cloud.langfuse.com/api/public/scores \\\\\\n -u \\"pk-lf-...\\":\\"sk-lf-...\\" \\\\\\n -H \\"Content-Type: application/json\\" \\\\\\n -d \'{\\n \\"id\\": \\"unique_id\\",\\n \\"traceId\\": \\"trace_id_here\\",\\n \\"observationId\\": \\"observation_id_here\\",\\n \\"name\\": \\"correctness\\",\\n \\"value\\": \\"correct\\",\\n \\"dataType\\": \\"CATEGORICAL\\",\\n \\"configId\\": \\"12345-6565-3453654-43543\\",\\n \\"comment\\": \\"Factually correct\\"\\n }\'\\n```\\nWhen ingesting boolean scores, you can provide the value as a float. If you provide a configId, the score\'s name and config\'s name must match as well as their data types.\\n \\n \\n```\\ncurl -X POST https://cloud.langfuse.com/api/public/scores \\\\\\n -u \\"pk-lf-...\\":\\"sk-lf-...\\" \\\\\\n -H \\"Content-Type: application/json\\" \\\\\\n -d \'{\\n \\"id\\": \\"unique_id\\",\\n \\"traceId\\": \\"trace_id_here\\",\\n \\"observationId\\": \\"observation_id_here\\",\\n \\"name\\": \\"helpfulness\\","}, {"type": "text", "text": "\\n \\"value\\": 1,\\n \\"dataType\\": \\"BOOLEAN\\",\\n \\"configId\\": \\"93547-6565-3453654-43543\\",\\n \\"comment\\": \\"Factually correct\\"\\n }\'\\n```\\nWhen ingesting text scores, provide the value as a string (1–500 characters). If you provide a configId, the score\'s name and config\'s name must match as well as their data types.\\n \\n \\n```\\ncurl -X POST https://cloud.langfuse.com/api/public/scores \\\\\\n -u \\"pk-lf-...\\":\\"sk-lf-...\\" \\\\\\n -H \\"Content-Type: application/json\\" \\\\\\n -d \'{\\n \\"id\\": \\"unique_id\\",\\n \\"traceId\\": \\"trace_id_here\\",\\n \\"observationId\\": \\"observation_id_here\\",\\n \\"name\\": \\"reviewer_notes\\",\\n \\"value\\": \\"The response was helpful but could be more concise.\\",\\n \\"dataType\\": \\"TEXT\\",\\n \\"configId\\": \\"24680-6565-3453654-43543\\",\\n \\"comment\\": \\"Reviewed by QA team\\"\\n }\'\\n```\\nSee [API reference](/docs/api) for more details on POST/GET score configs endpoints.\\n### Inferred Score Properties\\nCertain score properties might be inferred based on your input:\\n * **If you don\'t provide a score data type** it will always be inferred. See tables below for details.\\n * **For boolean and categorical scores** , we will provide the score value in both numerical and string format where possible. The score value format that is not provided as input, i.e. the translated value is referred to as the inferred value in the tables below.\\n * **On read via the[v3 scores API](https://api.reference.langfuse.com/#tag/scoresv3/GET/api/public/v3/scores)**, boolean scores are returned as a single boolean `value`.\\n * **For categorical scores** , the string representation is always provided and a numerical mapping of the category will be produced only if a `ScoreConfig` was provided.\\nDetailed Examples:"}, {"type": "text", "text": "\\nNumeric ScoresCategorical ScoresBoolean ScoresText Scores\\nFor example, let\'s assume you\'d like to ingest a numeric score to measure **accuracy**. We have included a table of possible score ingestion scenarios below.\\nValue| Data Type| Config Id| Description| Inferred Data Type| Valid \\n---|---|---|---|---|--- \\n`0.9`| `Null`| `Null`| Data type is inferred| `NUMERIC`| Yes \\n`0.9`| `NUMERIC`| `Null`| No properties inferred| | Yes \\n`depth`| `NUMERIC`| `Null`| Error: data type of value does not match provided data type| | No \\n`0.9`| `NUMERIC`| `78545`| No properties inferred| | Conditional on config validation \\n`0.9`| `Null`| `78545`| Data type inferred| `NUMERIC`| Conditional on config validation \\n`depth`| `NUMERIC`| `78545`| Error: data type of value does not match provided data type| | No \\n \\nFor example, let\'s assume you\'d like to ingest a categorical score to measure **correctness**. We have included a table of possible score ingestion scenarios below.\\nValue| Data Type| Config Id| Description| Inferred Data Type| Inferred Value representation| Valid \\n---|---|---|---|---|---|--- \\n`correct`| `Null`| `Null`| Data type is inferred| `CATEGORICAL`| | Yes \\n`correct`| `CATEGORICAL`| `Null`| No properties inferred| | | Yes \\n`1`| `CATEGORICAL`| `Null`| Error: data type of value does not match provided data type| | | No \\n`correct`| `CATEGORICAL`| `12345`| Numeric value inferred| | `4` numeric config category mapping| Conditional on config validation \\n`correct`| `NULL`| `12345`| Data type inferred| `CATEGORICAL`| | Conditional on config validation \\n`1`| `CATEGORICAL`| `12345`| Error: data type of value does not match provided data type| | | No \\n "}, {"type": "text", "text": "For example, let\'s assume you\'d like to ingest a boolean score to measure **helpfulness**. We have included a table of possible score ingestion scenarios below.\\nValue| Data Type| Config Id| Description| Inferred Data Type| Inferred Value representation| Valid \\n---|---|---|---|---|---|--- \\n`1`| `BOOLEAN`| `Null`| Value\'s string equivalent inferred| | `True`| Yes \\n`true`| `BOOLEAN`| `Null`| Error: data type of value does not match provided data type| | | No \\n`3`| `BOOLEAN`| `Null`| Error: boolean data type expects `0` or `1` as input value| | | No \\n`0.9`| `Null`| `93547`| Data type and value\'s string equivalent inferred| `BOOLEAN`| `True`| Conditional on config validation \\n`depth`| `BOOLEAN`| `93547`| Error: data type of value does not match provided data type| | | No \\n \\nFor example, let\'s assume you\'d like to ingest a text score to capture **reviewer notes**. Text scores must be non-empty strings of at most 500 characters.\\nValue| Data Type| Config Id| Description| Inferred Data Type| Valid \\n---|---|---|---|---|--- \\n`\\"Good response\\"`| `Null`| `Null`| Data type is inferred| `TEXT`| Yes \\n`\\"Good response\\"`| `TEXT`| `Null`| No properties inferred| | Yes \\n`0.9`| `TEXT`| `Null`| Error: data type of value does not match provided data type| | No \\n`\\"Good response\\"`| `TEXT`| `24680`| No properties inferred| | Conditional on config validation \\n`\\"Good response\\"`| `Null`| `24680`| Data type inferred| `TEXT`| Conditional on config validation \\n`\\"\\"`| `TEXT`| `Null`| Error: text scores must be non-empty| | No \\n \\n\\n## Update Existing Scores via API/SDKs"}, {"type": "text", "text": "When creating a score, you can provide an optional `id` (JS/TS) / `score_id` (Python) parameter. A re-ingested score overwrites an existing one only when its `id`, `name`, and `timestamp` (at date granularity, `toDate(timestamp)`) all match β€” a matching `id` alone is not sufficient, and any difference creates a duplicate instead of an update.\\nTo overwrite a score without fetching existing scores first, set a stable `id` as an idempotency key when initially creating the score, and keep the `name` and `timestamp` unchanged on subsequent calls. See Preventing Duplicate Scores for details.\\n## Related guides\\nGuides\\n * [Collecting user feedback\\nCapture explicit and implicit user feedback as scores on your traces, including in-app ratings sent via our Browser SDK.\\n](/guides/user-feedback-loop)\\n * [Custom evaluation data pipeline\\nContinuously monitor quality by fetching traces, running custom evaluations, and ingesting scores back into Langfuse.\\n](/guides/cookbook/example_external_evaluation_pipelines)\\n * [Guardrails and security checks\\nCheck whether output contains a certain keyword, matches a required structure or format, or exceeds a length limit.\\n](/guides/cookbook/security-and-guardrails)\\n * [Custom internal workflow tooling\\nBuild internal tooling for human-in-the-loop workflows, ingesting scores back into Langfuse following your custom schema.\\n](/guides/human-in-the-loop-scoring)\\n * [Session-level quality tracking\\nScore full conversations, such as support chats or agent threads, by attaching scores via sessionId in the SDK or API.\\n](/guides/cookbook/example_evaluating_multi_turn_conversations)\\n* * *\\n* * *\\nLast edited 7/22/2026\\n"}], "type": "content"}}, {"type": "document", "record_type": "documentation", "url": "https://langfuse.com/docs/evaluation/scores/overview", "title": "Scores", "source": {"content": [{"type": "text", "text": "Overview - Langfuse\\n[Docs](/docs)\\n[Evaluation](/docs/evaluation/overview)\\n[Scores](/docs/evaluation/scores/overview)\\n[Overview](/docs/evaluation/scores/overview)\\n# Scores\\nScores are Langfuse\'s universal data object for storing evaluation results. Any time you want to assign a quality judgment to an LLM output, whether by a [human annotation](/docs/evaluation/evaluation-methods/scores-via-ui), an [LLM judge](/docs/evaluation/evaluation-methods/llm-as-a-judge), a [programmatic check](/docs/evaluation/evaluation-methods/scores-via-sdk), or end-user feedback, the result is stored as a score.\\nEvery score has a **name** (like `\\"correctness\\"` or `\\"helpfulness\\"`), a **value** , and a **data type**. Scores also support an optional **comment** for additional context.\\nScores can be attached to [traces](/docs/observability/data-model#observations-and-traces), [observations](/docs/observability/data-model#observations-and-traces), [sessions](/docs/observability/data-model#sessions), or [dataset runs](/docs/evaluation/experiments/data-model). Most commonly, scores are attached to traces to evaluate a single end-to-end interaction.\\nOnce you have scores, they show up in [score analytics](/docs/evaluation/scores/score-analytics), can be visualized in [custom dashboards](/docs/metrics/features/custom-dashboards), and can be queried via the [API](/docs/api).\\n## When to Use Scores\\nScores become useful when you want to go beyond observing what your application does and start measuring how well it does it. Common use cases:\\n * **Collecting user feedback** : Capture thumbs up/down or star ratings from your users and attach them to traces. See the [user feedback guide](/docs/observability/features/user-feedback)."}, {"type": "text", "text": "\\n * **Monitoring production quality** : Set up automated evaluators (like [LLM-as-a-Judge](/docs/evaluation/evaluation-methods/llm-as-a-judge)) to continuously score live traces for things like hallucination, relevance, or tone.\\n * **Running guardrails** : Score whether outputs pass safety checks like PII detection, format validation, or content policy compliance.\\n * **Comparing changes with experiments** : When you change a prompt, model, or pipeline, run an [experiment](/docs/evaluation/experiments) to score the new version against a dataset.\\n## Score Types\\nLangfuse supports four score data types:\\nType| Value| Use when \\n---|---|--- \\n`NUMERIC`| Float (e.g. `0.9`)| Continuous judgments like accuracy, relevance, or similarity scores \\n`CATEGORICAL`| String from predefined categories (e.g. `\\"correct\\"`, `\\"partially correct\\"`)| Discrete classifications where the set of possible values is known upfront \\n`BOOLEAN`| `0` or `1`| Pass/fail checks like hallucination detection or format validation \\n`TEXT`| Free-form string (1-500 characters)| Open-ended annotations like reviewer notes or qualitative feedback. Often used for [open coding](https://en.wikipedia.org/wiki/Open_coding) before formalizing into quantifiable scores via [axial coding](https://en.wikipedia.org/wiki/Axial_coding). \\n \\nText scores are designed for qualitative, open-ended scoring. Because free-form text cannot be meaningfully aggregated or compared, text scores are not supported in [experiments](/docs/evaluation/core-concepts#experiments), [LLM-as-a-Judge](/docs/evaluation/evaluation-methods/llm-as-a-judge), or [score analytics](/docs/evaluation/scores/score-analytics).\\n## How to Create Scores"}, {"type": "text", "text": "\\nThere are five ways to add scores:\\n * **LLM-as-a-Judge** : Set up [automated evaluators](/docs/evaluation/evaluation-methods/llm-as-a-judge) that score traces based on custom criteria (e.g. hallucination, tone, relevance). These can return numeric or categorical scores plus reasoning, and can run on live production traces or on experiment results.\\n * **Code evaluators** : Run [custom Python or TypeScript evaluators](/docs/evaluation/evaluation-methods/code-evaluators) in Langfuse for deterministic checks such as exact match, JSON validation, or custom business rules.\\n * **Scores via UI** : Team members [manually score](/docs/evaluation/evaluation-methods/scores-via-ui) traces, observations, or sessions directly in the Langfuse UI. Requires a [score config](/faq/all/manage-score-configs) to be set up first.\\n * **Annotation Queues** : Set up [structured review workflows](/docs/evaluation/evaluation-methods/annotation-queues) where reviewers work through batches of traces.\\n * **Scores via API/SDK** : [Programmatically add scores](/docs/evaluation/evaluation-methods/scores-via-sdk) from your application code. This is the way to go for user feedback (thumbs up/down, star ratings), guardrail results, or custom evaluation pipelines.\\n## Should I Use Scores or Tags?\\n| Scores| Tags \\n---|---|--- \\n **Purpose**| Measure _how good_ something is| Describe _what_ something is \\n **Data**| Numeric, categorical, boolean, or text value| Simple string label \\n **When added**| Can be added at any time, including long after the trace was created| Set during tracing and cannot be changed afterwards \\n **Used for**| Quality measurement, analytics, experiments| Filtering, segmentation, organizing \\n "}, {"type": "text", "text": "As a rule of thumb: if you already know the category at tracing time (e.g. which feature or API endpoint triggered the trace), use a [tag](/docs/observability/features/tags). If you need to classify or evaluate traces later, use a score.\\n## Score Comments\\nEvery score supports an optional **comment** field. Use it to capture reasoning (e.g. why an LLM judge assigned a particular score), reviewer notes, or context that helps others understand the score value. Comments are shown alongside scores in the Langfuse UI.\\nUse a `TEXT` score instead of comments to capture standalone qualitative feedback -- comments are best for additional reasoning on an existing score.\\n* * *\\n* * *\\nLast edited 7/23/2026\\n"}], "type": "content"}}, {"type": "document", "record_type": "github_discussion", "url": "https://github.com/orgs/langfuse/discussions/6016", "title": "Using scores in data sets from the SDK?", "source": {"content": [{"type": "text", "text": "Title: Using scores in data sets from the SDK?\\nCategory: Support\\nAuthor: @howinator\\nTimestamp: March 14, 2025\\nBody:\\nHere is a workflow that I want, but I can\'t figure out how to accomplish it without a lot of manual work.\\r\\n\\r\\n1. Collect traces\\r\\n2. Annotate traces with either manual annotation or programmatic feedback from users. For a simple classification, I would use a boolean score where True means the model returned the correct answer and False when it returned the incorrect answer.\\r\\n4. Create a \\"false positive\\" data set composed of all traces where the score was False. <-- I can\'t figure out how to do this without a lot of clicking.\\r\\n5. Run an eval from that data set and use the score to determine if the eval was correct, e.g., something like `(dataset.item.score && dataset.item.output === evalOutput.output) || (!dataset.item.score && dataset.item.output !== evalOutput.output)`. <-- I can\'t figure out how to do this programmatically at all.\\r\\n\\r\\nIs my workflow completely off here? Am I missing something? Or is this flow just not possible in Langfuse today?\\nComments:\\n\\n Comment #1 from @dosubot on March 14, 2025:\\n\\n <!-- Greeting -->\\nHey there, @howinator! I\'m here to help you out with any bugs, questions, or contributions you have in mind. Let\'s get to work!\\n\\n<!-- Answer -->\\nYour workflow is mostly feasible in Langfuse, and you can achieve it programmatically with some adjustments:\\n\\n1. **Collect Traces**: This step is straightforward and can be done using Langfuse\'s SDKs to capture and store traces.\\n\\n2. **Annotate Traces**: You can programmatically annotate traces with scores using the `langfuse_context.score_current_trace` function. This allows you to attach a boolean score to each trace based on user feedback or other criteria <sup>[[1]](https://langfuse.com/guides/cookbook/integration_langgraph)</sup>.\\n"}, {"type": "text", "text": "\\n3. **Create a \\"False Positive\\" Dataset**: While there isn\'t a direct method to filter dataset items based on scores, you can programmatically create a dataset by iterating over your traces and selecting those with a score of `False`. Use the SDK to create dataset items for these traces <sup>[[2]](https://langfuse.com/docs/datasets/get-started)</sup>.\\n\\n4. **Run an Evaluation**: You can automate the creation of evaluation jobs for your dataset using the `createEvalJobs` function. This function allows you to specify configurations and run evaluations on specific subsets of dataset items <sup>[[3]](https://github.com/langfuse/langfuse/blob/eecf0d680b13a4bde9fe8fed617acc328029307d/worker/src/__tests__/evalService.test.ts)</sup>.\\n\\nFor the evaluation logic, you can implement custom logic to compare the dataset item scores and outputs with the evaluation outputs. This might require some additional scripting to handle the specific logic you described.\\n\\nOverall, your workflow is possible with Langfuse, but it may require some custom scripting to fully automate the process of filtering and evaluating dataset items based on scores.\\n\\n\\n<!-- Dosu Comment Footer -->\\n*To continue talking to [Dosu](https://dosu.dev), mention @dosu.*\\n\\n---"}, {"type": "text", "text": "1. Create a \\"false positive\\" data set composed of all traces where the score was False -> use scores api to fetch all of these\\r\\n2. Run your own eval on them and add it to the same trace via the sdk\\r\\n3. Compare the evals by fetching all of them via the sdk again.\\r\\n\\r\\nWe currently plan to add more configurable dashboards which will allow for the comparison of these evals more easily in the UI.\\n"}], "type": "content"}}, {"type": "document", "record_type": "documentation", "url": "https://langfuse.com/docs/observability/troubleshooting-and-faq", "title": "Troubleshooting and FAQ", "source": {"content": [{"type": "text", "text": "\\n[LLM as a Session-Level Judge with Support for LangGraph Session](https://github.com/orgs/langfuse/discussions/8303 \\"Langfuse Ideas: LLM as a Session-Level Judge with Support for LangGraph Session\\")\\n[Score Configs: Allow editing the categories of a categorical score](https://github.com/orgs/langfuse/discussions/8259 \\"Langfuse Ideas: Score Configs: Allow editing the categories of a categorical score\\")\\n[feat(trace-detail-view): for each span, show the sum of nested observations\' token usage details](https://github.com/orgs/langfuse/discussions/8201 \\"Langfuse Ideas: feat\\\\(trace-detail-view\\\\): for each span, show the sum of nested observations\' token usage details\\")\\n[UI-LLM as a Jury](https://github.com/orgs/langfuse/discussions/8195 \\"Langfuse Ideas: UI-LLM as a Jury\\")\\n[Add Gemini 2.5 Pro model pricing](https://github.com/orgs/langfuse/discussions/8146 \\"Langfuse Ideas: Add Gemini 2.5 Pro model pricing\\")\\n[Deeplinking for Spans inside of a Trace](https://github.com/orgs/langfuse/discussions/8112 \\"Langfuse Ideas: Deeplinking for Spans inside of a Trace\\")\\n[feat(comments): allow attaching images](https://github.com/orgs/langfuse/discussions/8111 \\"Langfuse Ideas: feat\\\\(comments\\\\): allow attaching images\\")\\n[feat(sessions): support session-level tags](https://github.com/orgs/langfuse/discussions/8041 \\"Langfuse Ideas: feat\\\\(sessions\\\\): support session-level tags\\")\\n[Enable Immediate Score Management for User Feedback](https://github.com/orgs/langfuse/discussions/7686 \\"Langfuse Ideas: Enable Immediate Score Management for User Feedback\\")\\n[Allow for structured filtering on inputs / outputs](https://github.com/orgs/langfuse/discussions/7663 \\"Langfuse Ideas: Allow for structured filtering on inputs / outputs\\")"}, {"type": "text", "text": "\\n[Allow for lazy updating of traces in Python SDK (+ possible workaround)](https://github.com/orgs/langfuse/discussions/7655 \\"Langfuse Ideas: Allow for lazy updating of traces in Python SDK \\\\(+ possible workaround\\\\)\\")\\n[Filter by scores in session view](https://github.com/orgs/langfuse/discussions/7528 \\"Langfuse Ideas: Filter by scores in session view\\")\\n[Support break lines on evaluation run tooltip hint](https://github.com/orgs/langfuse/discussions/7452 \\"Langfuse Ideas: Support break lines on evaluation run tooltip hint\\")\\n[Enable sending Langfuse SDK OTEL spans to third party observability backend](https://github.com/orgs/langfuse/discussions/7365 \\"Langfuse Ideas: Enable sending Langfuse SDK OTEL spans to third party observability backend\\")\\n[Add Thinking config in playground for anthropic models](https://github.com/orgs/langfuse/discussions/7267 \\"Langfuse Ideas: Add Thinking config in playground for anthropic models\\")\\n[Better support for big int in metadata.](https://github.com/orgs/langfuse/discussions/7167 \\"Langfuse Ideas: Better support for big int in metadata.\\")\\n[Ability to filter/search comments on traces](https://github.com/orgs/langfuse/discussions/7109 \\"Langfuse Ideas: Ability to filter/search comments on traces\\")\\n[Image / File / Multimedia rendering in dataset items using mediaId](https://github.com/orgs/langfuse/discussions/6883 \\"Langfuse Ideas: Image / File / Multimedia rendering in dataset items using mediaId\\")\\n[Trace Comments support upload image](https://github.com/orgs/langfuse/discussions/6857 \\"Langfuse Ideas: Trace Comments support upload image\\")"}], "type": "content"}}, {"type": "document", "record_type": "github_issue", "url": "https://github.com/langfuse/langfuse/issues/3380", "title": "feat: add fetch_score(s) in the sdks", "source": {"content": [{"type": "text", "text": "Github Issue by @marcklingen created on September 17, 2024. The current state of the issue is: closed\\n\\n### Discussed in https://github.com/orgs/langfuse/discussions/3379\\r\\n\\r\\n<div type=\'discussions-op-text\'>\\r\\n\\r\\n<sup>Originally posted by **Preet1234-ux** September 17, 2024</sup>\\r\\nI want to fetch the score provided by the user from langfuse in my python code . Are there any functionality by which I can fetch the score for a trace ? </div>\\r\\n\\r\\n\\r\\nCurrently, we need to use either langfuse.client.*(potentially breaking) or the api.\\n---\\nComment #1 by @hassiebp:\\n\\nAlready shipped in Python SDK v3\\n\\n"}], "type": "content"}}, {"type": "document", "record_type": "github_discussion", "url": "https://github.com/orgs/langfuse/discussions/6210", "title": "Next.js + Vercel AI SDK: How to get `observationId` and attach it to scores?", "source": {"content": [{"type": "text", "text": "Title: Next.js + Vercel AI SDK: How to get `observationId` and attach it to scores?\\nCategory: Support\\nAuthor: @natterstefan\\nTimestamp: March 26, 2025\\nBody:\\nDear langfuse-Community,\\r\\n\\r\\nI need your help please, as I do not know how to obtain the `observationId` and attach it to a `score`.\\r\\n\\r\\nContext: this is how my `experimental_telemetry` looks like:\\r\\n\\r\\n```js\\r\\n// src: https://langfuse.com/docs/integrations/vercel-ai-sdk#group-multiple-executions-in-one-trace\\r\\n\\r\\nconst parentTraceId = getRandomId()\\r\\n\\r\\nexport async function POST(req: Request) {\\r\\n\\t// ... other code\\r\\n\\r\\n\\tconst trace = langfuse.trace({\\r\\n\\t id: parentTraceId,\\r\\n\\t name: \'cd-chat-messages\',\\r\\n\\t})\\r\\n\\r\\n\\tconsole.log(\'trace\', trace.observationId) // <<-- this returns `null`\\r\\n\\t\\r\\n\\tconst result = streamText({\\r\\n\\t\\t// ....\\r\\n\\t\\texperimental_telemetry: {\\r\\n\\t\\t isEnabled: true,\\r\\n\\t\\t functionId: \'cd-chat-message-\' + messages.length,\\r\\n\\t\\t metadata: {\\r\\n\\t\\t langfuseTraceId: parentTraceId,\\r\\n\\t\\t\\t // Do not update the parent trace with execution results\\r\\n\\t\\t langfuseUpdateParent: false,\\r\\n\\t\\t },\\r\\n\\t\\t },\\r\\n\\t})\\r\\n\\r\\n\\t// ... other code\\r\\n}\\r\\n```\\r\\n\\r\\nThe docs state\\r\\n\\r\\n> The id of the observation to which the score should be attached. Pass from backend trace. E.g. {generation, event, span}.id\\r\\n\\r\\nBut how and where do I get the observationId from, taking into account that I use `streamText` (<https://sdk.vercel.ai/docs/reference/ai-sdk-core/stream-text>)\\r\\n\\r\\n```js\\r\\n// source: https://langfuse.com/docs/sdk/typescript/guide-web#create-scores\\r\\n\\r\\n// pass traceId and observationId to front end\\r\\nawait langfuseWeb.score({\\r\\n traceId: message.traceId,\\r\\n observationId: message.observationId, <<<-- where do I get this?\\r\\n name: \\"user-feedback\\",\\r\\n value: 1,\\r\\n comment: \\"I like how personalized the response is\\",\\r\\n});\\r\\n```\\r\\n\\r"}, {"type": "text", "text": "Another question I could not answer myself is where (= which \\"layer\\". I guess generation, event, span is the correct wording) a score should be attached to. Is there any best practice recommendation or should I always go for the observation (`ai.streamText.doStream` in this example _if_ that\'s the observationId I can obtain)?\\r\\n\\r\\n![image](https://github.com/user-attachments/assets/e103abd2-3fd5-4607-9c09-6d0ad2b15393)\\r\\n\\r\\nThank you very much for your time and help!\\nComments:\\n\\n Comment #1 from @dosubot on March 26, 2025:\\n\\n <!-- Greeting -->\\nHey @natterstefan! I\'m here to help you with any bugs, questions, or even becoming a contributor. Let\'s squash those bugs together! πŸ€–\\n\\n<!-- Answer -->\\nTo obtain the `observationId` when using the `streamText` function in the Vercel AI SDK, you should look for it in the response from the `streamText` function. The SDK typically uses the ID from the provider response if available, or it generates one if not. Ensure that your telemetry setup is configured to pass the `observationId` from the backend trace, specifically from the context of the `streamText` execution.\\n\\nRegarding where to attach the score, it is generally recommended to attach it to the most relevant observation. In your case, if `ai.streamText.doStream` is the observation you can obtain, it would be appropriate to attach the score there. This aligns with best practices, as it ensures the score is associated with the specific operation or event you are evaluating.\\n"}, {"type": "text", "text": "\\nFor more detailed guidance, you might want to refer to the Langfuse documentation on creating scores and handling observations <sup>[[1]](https://langfuse.com/docs/sdk/typescript)</sup><sup>[[2]](https://langfuse.com/docs/sdk/typescript/guide)</sup>.\\n\\n\\n<!-- Dosu Comment Footer -->\\n*To continue talking to [Dosu](https://dosu.dev), mention @dosu.*\\n\\n---\\n<sup>Help Dosu learn! Give it feedback: [Great Response](https://app.dosu.dev/response-feedback/68b78af0-2c34-4867-b75c-008bcf449b81?feedback_type=great_response) | [Irrelevant Answer](https://app.dosu.dev/response-feedback/68b78af0-2c34-4867-b75c-008bcf449b81?feedback_type=irrelevant_answer) | [Incorrect Sources](https://app.dosu.dev/response-feedback/68b78af0-2c34-4867-b75c-008bcf449b81?feedback_type=incorrect_sources) | [Too Verbose](https://app.dosu.dev/response-feedback/68b78af0-2c34-4867-b75c-008bcf449b81?feedback_type=too_verbose) | [Hallucination](https://app.dosu.dev/response-feedback/68b78af0-2c34-4867-b75c-008bcf449b81?feedback_type=hallucination) | [Bug Report](https://app.dosu.dev/response-feedback/68b78af0-2c34-4867-b75c-008bcf449b81?feedback_type=bug_report) | [Other](https://app.dosu.dev/response-feedback/68b78af0-2c34-4867-b75c-008bcf449b81?feedback_type=other)</sup>\\n\\n\\n Comment #2 from @marcklingen on March 26, 2025:\\n\\n This seems like a feature request for the AI SDK as you\'d need the SDK to expose the span ids of each of the requests.\\n\\n Replies to comment #2:\\n\\n Reply #1 to comment #2 from marcklingen on March 27, 2025:\\n\\n Hi @marcklingen,\\r\\n\\r\\nthanks for your help. Alright, I [moved (aka copied) the discussion to their repo](https://github.com/vercel/ai/discussions/5415) and hope they can help me.\\r\\n\\r\\nIs @dosubot correct with \\"_(...) In your case, if ai.streamText.doStream is the observation you can obtain, it would be appropriate to attach the score there._\\"\\r\\n\\r"}, {"type": "text", "text": "\\n<sup>Help Dosu learn! Give it feedback: [Great Response](https://app.dosu.dev/response-feedback/d9fcd303-8526-48dc-85b4-1432b3d12429?feedback_type=great_response) | [Irrelevant Answer](https://app.dosu.dev/response-feedback/d9fcd303-8526-48dc-85b4-1432b3d12429?feedback_type=irrelevant_answer) | [Incorrect Sources](https://app.dosu.dev/response-feedback/d9fcd303-8526-48dc-85b4-1432b3d12429?feedback_type=incorrect_sources) | [Too Verbose](https://app.dosu.dev/response-feedback/d9fcd303-8526-48dc-85b4-1432b3d12429?feedback_type=too_verbose) | [Hallucination](https://app.dosu.dev/response-feedback/d9fcd303-8526-48dc-85b4-1432b3d12429?feedback_type=hallucination) | [Bug Report](https://app.dosu.dev/response-feedback/d9fcd303-8526-48dc-85b4-1432b3d12429?feedback_type=bug_report) | [Other](https://app.dosu.dev/response-feedback/d9fcd303-8526-48dc-85b4-1432b3d12429?feedback_type=other)</sup>\\n\\n\\n Reply #3 to comment #2 from marcklingen on March 27, 2025:\\n\\n Hi @marcklingen and @dosubot,\\r\\n\\r\\nFollow up question (once I get the `generation.id`/`response.id` whatever vercel/ai calls it): how do I attach it to the current trace to make sure the observation in langfuse has the same id as the score in my code uses?\\n\\n Reply #4 to comment #2 from marcklingen on March 27, 2025:\\n\\n <!-- Answer -->\\nTo attach the `generation.id` or `response.id` from Vercel AI to the current trace in Langfuse, use the `generation.id` as the `observationId` when logging user feedback. This ensures that the observation has the same ID as the score used in your code. Here\'s how you can do it:\\n\\n1. Add the `generation.id` to the streaming response as a custom header:\\n\\n ```javascript\\n return new StreamingTextResponse(stream, {\\n headers: {\\n \\"X-Message-Id\\": lfGeneration.id,"}], "type": "content"}}]}',
+ },
+ ],
+ isError: false,
+});
+
+const payload12 = JSON.stringify({
+ messages: [
+ {
+ role: "user",
+ parts: [
+ {
+ type: "text",
+ content: "How do I attach a user score to a trace?",
+ },
+ ],
+ },
+ {
+ role: "assistant",
+ parts: [
+ {
+ type: "reasoning",
+ content:
+ '**Searching documents for scores**\n\nI need to use tools to search the documentation. It looks like Langfuse has a "scores" API that can be linked to traces or observations. There are "Score" entities to consider, which include details like name, value, data type, comments, source, traceId, observationId, and timestamp. The SDKs have methods such as client.score or langfuse.score. I plan to search Langfuse docs to get more information on this.',
+ },
+ {
+ type: "text",
+ content:
+ "I’m going to search the Langfuse docs for how to add or attach a score to a trace, including minimal code examples.",
+ },
+ {
+ type: "tool_call",
+ id: "call_OQNWWW4PjixkLIhkuHJRwiFV",
+ name: "searchLangfuseDocs",
+ arguments: {
+ query:
+ "How to attach a score to a trace user score to a trace Langfuse SDK example",
+ },
+ },
+ ],
+ },
+ {
+ role: "tool",
+ parts: [
+ {
+ type: "tool_call_response",
+ id: "call_OQNWWW4PjixkLIhkuHJRwiFV",
+ response: {
+ type: "content",
+ value: [
+ {
+ type: "text",
+ text: '{"content": [{"type": "document", "record_type": "documentation", "url": "https://langfuse.com/docs/evaluation/evaluation-methods/scores-via-sdk", "title": "Scores via API/SDK", "source": {"content": [{"type": "text", "text": "Scores via API/SDK - Langfuse\\n[Docs](/docs)\\n[Evaluation](/docs/evaluation/overview)\\n[Evaluation Methods](/docs/evaluation/evaluation-methods/llm-as-a-judge)\\n[Scores via API/SDK](/docs/evaluation/evaluation-methods/scores-via-sdk)\\n# Scores via API/SDK\\nYou can use the Langfuse SDKs or API to add [scores](/docs/evaluation/scores/overview) to traces, observations, sessions and dataset runs. This is an evaluation method that allows you to set up custom evaluation workflows and extend the scoring capabilities of Langfuse. See the [data model](/docs/evaluation/scores/data-model#scores) for full details on the score object.\\nIf you want Langfuse to run deterministic Python or TypeScript logic for you, use [code evaluators](/docs/evaluation/evaluation-methods/code-evaluators). Use this page when your application, pipeline, or CI job computes scores and sends them to Langfuse via API or SDK.\\n## Ingesting Scores via API/SDKs\\nScores can be attached at different levels of granularity: to individual traces, to specific observations within a trace, or to full sessions.\\nSee the [API reference](/docs/api) for full details on the POST and GET endpoints for scores and score configs.\\n### Trace or Observation-level Scores\\nYou can add scores via the Langfuse SDKs or API. Scores can take one of four data types: **Numeric** , **Categorical** , **Boolean** , or **Text**. See [Score Types](/docs/evaluation/scores/overview#score-types) for details.\\nIf a score is ingested manually using a `trace_id` to link the score to a trace, it is not necessary to wait until the trace has been created. The score will show up in the scores table and will be linked to the trace once the trace with the same `trace_id` is created.\\nHere are examples by `Score` data types."}, {"type": "text", "text": "\\nFor trace and observation scores, `trace_id`/`traceId` is required and `observation_id`/`observationId` is optional. If you attach a score to an observation, always provide both the observation ID and the corresponding trace ID.\\nPython SDKJS/TS SDKAPI\\nNumericCategoricalBooleanText\\nNumeric score values must be provided as float.\\n \\n \\n```\\nfrom langfuse import get_client\\nlangfuse = get_client()\\n///# Method 1: Score via low-level method\\nlangfuse.create_score(\\n name=\\"correctness\\",\\n value=0.9,\\n trace_id=\\"trace_id_here\\",\\n observation_id=\\"observation_id_here\\", ///# optional\\n data_type=\\"NUMERIC\\", ///# optional, inferred if not provided\\n comment=\\"Factually correct\\", ///# optional\\n)\\n///# Method 2: Score current span/generation (within context)\\nwith langfuse.start_as_current_observation(as_type=\\"span\\", name=\\"my-operation\\") as span:\\n # Score the current span\\n span.score(\\n name=\\"correctness\\",\\n value=0.9,\\n data_type=\\"NUMERIC\\",\\n comment=\\"Factually correct\\"\\n )\\n # Score the trace\\n span.score_trace(\\n name=\\"overall_quality\\",\\n value=0.95,\\n data_type=\\"NUMERIC\\"\\n )\\n///# Method 3: Score via the current context\\nwith langfuse.start_as_current_observation(as_type=\\"span\\", name=\\"my-operation\\"):\\n # Score the current span\\n langfuse.score_current_span(\\n name=\\"correctness\\",\\n value=0.9,\\n data_type=\\"NUMERIC\\",\\n comment=\\"Factually correct\\"\\n )\\n # Score the trace\\n langfuse.score_current_trace(\\n name=\\"overall_quality\\",\\n value=0.95,\\n data_type=\\"NUMERIC\\"\\n )\\n```\\nCategorical score values must be provided as strings.\\n \\n \\n```\\nfrom langfuse import get_client\\nlangfuse = get_client()\\n///# Method 1: Score via low-level method\\nlangfuse.create_score(\\n name=\\"accuracy\\",\\n value=\\"partially correct\\",\\n trace_id=\\"trace_id_here\\",\\n observation_id=\\"observation_id_here\\", ///# optional\\n data_type=\\"CATEGORICAL\\", ///# optional, inferred if not provided\\n comment=\\"Some factual errors\\", ///# optional\\n)"}, {"type": "text", "text": "\\n///# Method 2: Score current span/generation (within context)\\nwith langfuse.start_as_current_observation(as_type=\\"span\\", name=\\"my-operation\\") as span:\\n # Score the current span\\n span.score(\\n name=\\"accuracy\\",\\n value=\\"partially correct\\",\\n data_type=\\"CATEGORICAL\\",\\n comment=\\"Some factual errors\\"\\n )\\n # Score the trace\\n span.score_trace(\\n name=\\"overall_quality\\",\\n value=\\"partially correct\\",\\n data_type=\\"CATEGORICAL\\"\\n )\\n///# Method 3: Score via the current context\\nwith langfuse.start_as_current_observation(as_type=\\"span\\", name=\\"my-operation\\"):\\n # Score the current span\\n langfuse.score_current_span(\\n name=\\"accuracy\\",\\n value=\\"partially correct\\",\\n data_type=\\"CATEGORICAL\\",\\n comment=\\"Some factual errors\\"\\n )\\n # Score the trace\\n langfuse.score_current_trace(\\n name=\\"overall_quality\\",\\n value=\\"partially correct\\",\\n data_type=\\"CATEGORICAL\\"\\n )\\n```\\nBoolean scores must be provided as a float, where `1` is true and `0` is false. When read via the [v3 scores API](https://api.reference.langfuse.com/#tag/scoresv3/GET/api/public/v3/scores), the value is returned as a boolean. See [API reference](/docs/api) for more details on POST/GET scores endpoints.\\n \\n \\n```\\nfrom langfuse import get_client\\nlangfuse = get_client()\\n///# Method 1: Score via low-level method\\nlangfuse.create_score(\\n name=\\"helpfulness\\",\\n value=0, ///# 0 or 1\\n trace_id=\\"trace_id_here\\",\\n observation_id=\\"observation_id_here\\", ///# optional\\n data_type=\\"BOOLEAN\\", ///# required, numeric values without data type would be inferred as NUMERIC\\n comment=\\"Incorrect answer\\", ///# optional\\n)\\n///# Method 2: Score current span/generation (within context)\\nwith langfuse.start_as_current_observation(as_type=\\"span\\", name=\\"my-operation\\") as span:\\n # Score the current span"}, {"type": "text", "text": "\\n span.score(\\n name=\\"helpfulness\\",\\n value=1, ///# 0 or 1\\n data_type=\\"BOOLEAN\\",\\n comment=\\"Very helpful response\\"\\n )\\n # Score the trace\\n span.score_trace(\\n name=\\"overall_quality\\",\\n value=1, ///# 0 or 1\\n data_type=\\"BOOLEAN\\"\\n )\\n///# Method 3: Score via the current context\\nwith langfuse.start_as_current_observation(as_type=\\"span\\", name=\\"my-operation\\"):\\n # Score the current span\\n langfuse.score_current_span(\\n name=\\"helpfulness\\",\\n value=1, ///# 0 or 1\\n data_type=\\"BOOLEAN\\",\\n comment=\\"Very helpful response\\"\\n )\\n # Score the trace\\n langfuse.score_current_trace(\\n name=\\"overall_quality\\",\\n value=1, ///# 0 or 1\\n data_type=\\"BOOLEAN\\"\\n )\\n```\\nText score values must be provided as strings between 1 and 500 characters.\\n \\n \\n```\\nfrom langfuse import get_client\\nlangfuse = get_client()\\n///# Method 1: Score via low-level method\\nlangfuse.create_score(\\n name=\\"reviewer_notes\\",\\n value=\\"The response was helpful but could be more concise.\\",\\n trace_id=\\"trace_id_here\\",\\n observation_id=\\"observation_id_here\\", ///# optional\\n data_type=\\"TEXT\\", ///# optional, inferred if not provided\\n comment=\\"Reviewed by QA team\\", ///# optional\\n)\\n///# Method 2: Score current span/generation (within context)\\nwith langfuse.start_as_current_observation(as_type=\\"span\\", name=\\"my-operation\\") as span:\\n # Score the current span\\n span.score(\\n name=\\"reviewer_notes\\",\\n value=\\"The response was helpful but could be more concise.\\",\\n data_type=\\"TEXT\\",\\n comment=\\"Reviewed by QA team\\"\\n )\\n # Score the trace\\n span.score_trace(\\n name=\\"overall_notes\\",\\n value=\\"Good quality overall, minor formatting issues.\\",\\n data_type=\\"TEXT\\"\\n )\\n///# Method 3: Score via the current context\\nwith langfuse.start_as_current_observation(as_type=\\"span\\", name=\\"my-operation\\"):\\n # Score the current span"}, {"type": "text", "text": "\\n langfuse.score_current_span(\\n name=\\"reviewer_notes\\",\\n value=\\"The response was helpful but could be more concise.\\",\\n data_type=\\"TEXT\\",\\n comment=\\"Reviewed by QA team\\"\\n )\\n # Score the trace\\n langfuse.score_current_trace(\\n name=\\"overall_notes\\",\\n value=\\"Good quality overall, minor formatting issues.\\",\\n data_type=\\"TEXT\\"\\n )\\n```\\nNumericCategoricalBooleanText\\nNumeric score values must be provided as float.\\n \\n \\n```\\nimport { LangfuseClient } from \\"@langfuse/client\\";\\nconst langfuse = new LangfuseClient();\\nlangfuse.score.create({\\n id: \\"unique_id\\", // optional, can be used as an idempotency key to update the score subsequently\\n traceId: message.traceId,\\n observationId: message.generationId, // optional\\n name: \\"correctness\\",\\n value: 0.9,\\n dataType: \\"NUMERIC\\", // optional, inferred if not provided\\n comment: \\"Factually correct\\", // optional\\n});\\n// Flush the scores in short-lived environments\\nawait langfuse.flush();\\n```\\nCategorical score values must be provided as strings.\\n \\n \\n```\\nimport { LangfuseClient } from \\"@langfuse/client\\";\\nconst langfuse = new LangfuseClient();\\nlangfuse.score.create({\\n id: \\"unique_id\\", // optional, can be used as an idempotency key to update the score subsequently\\n traceId: message.traceId,\\n observationId: message.generationId, // optional\\n name: \\"accuracy\\",\\n value: \\"partially correct\\",\\n dataType: \\"CATEGORICAL\\", // optional, inferred if not provided\\n comment: \\"Factually correct\\", // optional\\n});\\n// Flush the scores in short-lived environments\\nawait langfuse.flush();\\n```"}, {"type": "text", "text": "\\nBoolean scores must be provided as a float, where `1` is true and `0` is false. When read via the [v3 scores API](https://api.reference.langfuse.com/#tag/scoresv3/GET/api/public/v3/scores), the value is returned as a boolean. See [API reference](/docs/api) for more details on POST/GET scores endpoints.\\n \\n \\n```\\nimport { LangfuseClient } from \\"@langfuse/client\\";\\nconst langfuse = new LangfuseClient();\\nlangfuse.score.create({\\n id: \\"unique_id\\", // optional, can be used as an idempotency key to update the score subsequently\\n traceId: message.traceId,\\n observationId: message.generationId, // optional\\n name: \\"helpfulness\\",\\n value: 0, // 0 or 1\\n dataType: \\"BOOLEAN\\", // required, numeric values without data type would be inferred as NUMERIC\\n comment: \\"Incorrect answer\\", // optional\\n});\\n// Flush the scores in short-lived environments\\nawait langfuse.flush();\\n```\\nText score values must be provided as strings between 1 and 500 characters.\\n \\n \\n```\\nimport { LangfuseClient } from \\"@langfuse/client\\";\\nconst langfuse = new LangfuseClient();\\nlangfuse.score.create({\\n id: \\"unique_id\\", // optional, can be used as an idempotency key to update the score subsequently\\n traceId: message.traceId,\\n observationId: message.generationId, // optional\\n name: \\"reviewer_notes\\",\\n value: \\"The response was helpful but could be more concise.\\",\\n dataType: \\"TEXT\\", // optional, inferred if not provided\\n comment: \\"Reviewed by QA team\\", // optional\\n});\\n// Flush the scores in short-lived environments\\nawait langfuse.flush();\\n```"}, {"type": "text", "text": "\\nYou can also create scores directly via the [REST API](https://api.reference.langfuse.com/#tag/legacyscorev1/POST/api/public/scores). Authenticate using HTTP Basic Auth with your Langfuse Public Key as username and Secret Key as password.\\nNumericCategoricalBooleanText\\nNumeric score values must be provided as float.\\n \\n \\n```\\ncurl -X POST https://cloud.langfuse.com/api/public/scores \\\\\\n -u \\"pk-lf-...\\":\\"sk-lf-...\\" \\\\\\n -H \\"Content-Type: application/json\\" \\\\\\n -d \'{\\n \\"traceId\\": \\"trace_id_here\\",\\n \\"observationId\\": \\"observation_id_here\\",\\n \\"name\\": \\"correctness\\",\\n \\"value\\": 0.9,\\n \\"dataType\\": \\"NUMERIC\\",\\n \\"comment\\": \\"Factually correct\\"\\n }\'\\n```\\nCategorical score values must be provided as strings.\\n \\n \\n```\\ncurl -X POST https://cloud.langfuse.com/api/public/scores \\\\\\n -u \\"pk-lf-...\\":\\"sk-lf-...\\" \\\\\\n -H \\"Content-Type: application/json\\" \\\\\\n -d \'{\\n \\"traceId\\": \\"trace_id_here\\",\\n \\"observationId\\": \\"observation_id_here\\",\\n \\"name\\": \\"accuracy\\",\\n \\"value\\": \\"partially correct\\",\\n \\"dataType\\": \\"CATEGORICAL\\",\\n \\"comment\\": \\"Some factual errors\\"\\n }\'\\n```\\nBoolean scores must be provided as a float, where `1` is true and `0` is false. When read via the [v3 scores API](https://api.reference.langfuse.com/#tag/scoresv3/GET/api/public/v3/scores), the value is returned as a boolean.\\n \\n \\n```\\ncurl -X POST https://cloud.langfuse.com/api/public/scores \\\\\\n -u \\"pk-lf-...\\":\\"sk-lf-...\\" \\\\\\n -H \\"Content-Type: application/json\\" \\\\\\n -d \'{\\n \\"traceId\\": \\"trace_id_here\\",\\n \\"observationId\\": \\"observation_id_here\\",\\n \\"name\\": \\"helpfulness\\",\\n \\"value\\": 0,\\n \\"dataType\\": \\"BOOLEAN\\",\\n \\"comment\\": \\"Incorrect answer\\"\\n }\'\\n```\\nText score values must be provided as strings between 1 and 500 characters.\\n \\n \\n```\\ncurl -X POST https://cloud.langfuse.com/api/public/scores \\\\\\n -u \\"pk-lf-...\\":\\"sk-lf-...\\" \\\\"}, {"type": "text", "text": "\\n comment=\\"Factually correct\\",\\n config_id=\\"12345-6565-3453654-43543\\",\\n data_type=\\"CATEGORICAL\\"\\n )\\n```\\nWhen ingesting boolean scores, you can provide the value as a float. If you provide a configId, the score\'s name and config\'s name must match as well as their data types.\\n \\n \\n```\\nfrom langfuse import get_client\\nlangfuse = get_client()\\n///# Method 1: Score via low-level method\\nlangfuse.create_score(\\n trace_id=\\"trace_id_here\\",\\n observation_id=\\"observation_id_here\\", ///# optional\\n name=\\"helpfulness\\",\\n value=1,\\n comment=\\"Factually correct\\", ///# optional\\n score_id=\\"unique_id\\", ///# optional, can be used as an idempotency key to update the score subsequently\\n config_id=\\"93547-6565-3453654-43543\\", ///# optional, can be used to infer the score data type and validate the score value\\n data_type=\\"BOOLEAN\\" # optional, possibly inferred\\n)\\n///# Method 2: Score within context\\nwith langfuse.start_as_current_observation(as_type=\\"span\\", name=\\"my-operation\\") as span:\\n span.score(\\n name=\\"helpfulness\\",\\n value=1,\\n comment=\\"Factually correct\\",\\n config_id=\\"93547-6565-3453654-43543\\",\\n data_type=\\"BOOLEAN\\"\\n )\\n```\\nWhen ingesting text scores, provide the value as a string (1–500 characters). If you provide a configId, the score\'s name and config\'s name must match as well as their data types.\\n \\n \\n```\\nfrom langfuse import get_client\\nlangfuse = get_client()\\n///# Method 1: Score via low-level method\\nlangfuse.create_score(\\n trace_id=\\"trace_id_here\\",\\n observation_id=\\"observation_id_here\\", ///# optional\\n name=\\"reviewer_notes\\",\\n value=\\"The response was helpful but could be more concise.\\",\\n comment=\\"Reviewed by QA team\\", ///# optional"}, {"type": "text", "text": "\\n score_id=\\"unique_id\\", ///# optional, can be used as an idempotency key to update the score subsequently\\n config_id=\\"24680-6565-3453654-43543\\", ///# optional\\n data_type=\\"TEXT\\" # optional, possibly inferred\\n)\\n///# Method 2: Score within context\\nwith langfuse.start_as_current_observation(as_type=\\"span\\", name=\\"my-operation\\") as span:\\n span.score(\\n name=\\"reviewer_notes\\",\\n value=\\"The response was helpful but could be more concise.\\",\\n comment=\\"Reviewed by QA team\\",\\n config_id=\\"24680-6565-3453654-43543\\",\\n data_type=\\"TEXT\\"\\n )\\n```\\nNumeric ScoresCategorical ScoresBoolean ScoresText Scores\\nWhen ingesting numeric scores, you can provide the value as a float. If you provide a configId, the score value will be validated against the config\'s numeric range, which might be defined by a minimum and/or maximum value.\\n \\n \\n```\\nimport { LangfuseClient } from \\"@langfuse/client\\";\\nconst langfuse = new LangfuseClient();\\nlangfuse.score.create({\\n traceId: message.traceId,\\n observationId: message.generationId, // optional\\n name: \\"accuracy\\",\\n value: 0.9,\\n comment: \\"Factually correct\\", // optional\\n id: \\"unique_id\\", // optional, can be used as an idempotency key to update the score subsequently\\n configId: \\"78545-6565-3453654-43543\\", // optional, to ensure that the score follows a specific min/max value range\\n dataType: \\"NUMERIC\\", // optional, possibly inferred\\n});\\n// Flush the scores in short-lived environments\\nawait langfuse.flush();\\n```\\nCategorical scores are used to evaluate data that falls into specific categories. When ingesting categorical scores, you can provide the value as a string. If you provide a configId, the score value will be validated against the config\'s categories.\\n \\n \\n```"}, {"type": "text", "text": "\\nimport { LangfuseClient } from \\"@langfuse/client\\";\\nconst langfuse = new LangfuseClient();\\nlangfuse.score.create({\\n traceId: message.traceId,\\n observationId: message.generationId, // optional\\n name: \\"correctness\\",\\n value: \\"correct\\",\\n comment: \\"Factually correct\\", // optional\\n id: \\"unique_id\\", // optional, can be used as an idempotency key to update the score subsequently\\n configId: \\"12345-6565-3453654-43543\\", // optional, to ensure that a score maps to a specific category defined in a score config\\n dataType: \\"CATEGORICAL\\", // optional, possibly inferred\\n});\\n// Flush the scores in short-lived environments\\nawait langfuse.flush();\\n```\\nWhen ingesting boolean scores, you can provide the value as a float. If you provide a configId, the score\'s name and config\'s name must match as well as their data types.\\n \\n \\n```\\nimport { LangfuseClient } from \\"@langfuse/client\\";\\nconst langfuse = new LangfuseClient();\\nlangfuse.score.create({\\n traceId: message.traceId,\\n observationId: message.generationId, // optional\\n name: \\"helpfulness\\",\\n value: 1,\\n comment: \\"Factually correct\\", // optional\\n id: \\"unique_id\\", // optional, can be used as an idempotency key to update the score subsequently\\n configId: \\"93547-6565-3453654-43543\\", // optional, can be used to infer the score data type and validate the score value\\n dataType: \\"BOOLEAN\\", // optional, possibly inferred\\n});\\n// Flush the scores in short-lived environments\\nawait langfuse.flush();\\n```\\nWhen ingesting text scores, provide the value as a string (1–500 characters). If you provide a configId, the score\'s name and config\'s name must match as well as their data types.\\n \\n \\n```\\nimport { LangfuseClient } from \\"@langfuse/client\\";\\nconst langfuse = new LangfuseClient();\\nlangfuse.score.create({\\n traceId: message.traceId,\\n observationId: message.generationId, // optional\\n name: \\"reviewer_notes\\",\\n value: \\"The response was helpful but could be more concise.\\","}, {"type": "text", "text": "\\nWhen ingesting boolean scores, you can provide the value as a float. If you provide a configId, the score\'s name and config\'s name must match as well as their data types.\\n \\n \\n```\\ncurl -X POST https://cloud.langfuse.com/api/public/scores \\\\\\n -u \\"pk-lf-...\\":\\"sk-lf-...\\" \\\\\\n -H \\"Content-Type: application/json\\" \\\\\\n -d \'{\\n \\"id\\": \\"unique_id\\",\\n \\"traceId\\": \\"trace_id_here\\",\\n \\"observationId\\": \\"observation_id_here\\",\\n \\"name\\": \\"helpfulness\\",\\n \\"value\\": 1,\\n \\"dataType\\": \\"BOOLEAN\\",\\n \\"configId\\": \\"93547-6565-3453654-43543\\",\\n \\"comment\\": \\"Factually correct\\"\\n }\'\\n```\\nWhen ingesting text scores, provide the value as a string (1–500 characters). If you provide a configId, the score\'s name and config\'s name must match as well as their data types.\\n \\n \\n```\\ncurl -X POST https://cloud.langfuse.com/api/public/scores \\\\\\n -u \\"pk-lf-...\\":\\"sk-lf-...\\" \\\\\\n -H \\"Content-Type: application/json\\" \\\\\\n -d \'{\\n \\"id\\": \\"unique_id\\",\\n \\"traceId\\": \\"trace_id_here\\",\\n \\"observationId\\": \\"observation_id_here\\",\\n \\"name\\": \\"reviewer_notes\\",\\n \\"value\\": \\"The response was helpful but could be more concise.\\",\\n \\"dataType\\": \\"TEXT\\",\\n \\"configId\\": \\"24680-6565-3453654-43543\\",\\n \\"comment\\": \\"Reviewed by QA team\\"\\n }\'\\n```\\nSee [API reference](/docs/api) for more details on POST/GET score configs endpoints.\\n### Inferred Score Properties\\nCertain score properties might be inferred based on your input:\\n * **If you don\'t provide a score data type** it will always be inferred. See tables below for details."}, {"type": "text", "text": "\\n * **For boolean and categorical scores** , we will provide the score value in both numerical and string format where possible. The score value format that is not provided as input, i.e. the translated value is referred to as the inferred value in the tables below.\\n * **On read via the[v3 scores API](https://api.reference.langfuse.com/#tag/scoresv3/GET/api/public/v3/scores)**, boolean scores are returned as a single boolean `value`.\\n * **For categorical scores** , the string representation is always provided and a numerical mapping of the category will be produced only if a `ScoreConfig` was provided.\\nDetailed Examples:\\nNumeric ScoresCategorical ScoresBoolean ScoresText Scores\\nFor example, let\'s assume you\'d like to ingest a numeric score to measure **accuracy**. We have included a table of possible score ingestion scenarios below.\\nValue| Data Type| Config Id| Description| Inferred Data Type| Valid \\n---|---|---|---|---|--- \\n`0.9`| `Null`| `Null`| Data type is inferred| `NUMERIC`| Yes \\n`0.9`| `NUMERIC`| `Null`| No properties inferred| | Yes \\n`depth`| `NUMERIC`| `Null`| Error: data type of value does not match provided data type| | No \\n`0.9`| `NUMERIC`| `78545`| No properties inferred| | Conditional on config validation \\n`0.9`| `Null`| `78545`| Data type inferred| `NUMERIC`| Conditional on config validation \\n`depth`| `NUMERIC`| `78545`| Error: data type of value does not match provided data type| | No \\n \\nFor example, let\'s assume you\'d like to ingest a categorical score to measure **correctness**. We have included a table of possible score ingestion scenarios below."}, {"type": "text", "text": "Value| Data Type| Config Id| Description| Inferred Data Type| Inferred Value representation| Valid \\n---|---|---|---|---|---|--- \\n`correct`| `Null`| `Null`| Data type is inferred| `CATEGORICAL`| | Yes \\n`correct`| `CATEGORICAL`| `Null`| No properties inferred| | | Yes \\n`1`| `CATEGORICAL`| `Null`| Error: data type of value does not match provided data type| | | No \\n`correct`| `CATEGORICAL`| `12345`| Numeric value inferred| | `4` numeric config category mapping| Conditional on config validation \\n`correct`| `NULL`| `12345`| Data type inferred| `CATEGORICAL`| | Conditional on config validation \\n`1`| `CATEGORICAL`| `12345`| Error: data type of value does not match provided data type| | | No \\n \\nFor example, let\'s assume you\'d like to ingest a boolean score to measure **helpfulness**. We have included a table of possible score ingestion scenarios below.\\nValue| Data Type| Config Id| Description| Inferred Data Type| Inferred Value representation| Valid \\n---|---|---|---|---|---|--- \\n`1`| `BOOLEAN`| `Null`| Value\'s string equivalent inferred| | `True`| Yes \\n`true`| `BOOLEAN`| `Null`| Error: data type of value does not match provided data type| | | No \\n`3`| `BOOLEAN`| `Null`| Error: boolean data type expects `0` or `1` as input value| | | No \\n`0.9`| `Null`| `93547`| Data type and value\'s string equivalent inferred| `BOOLEAN`| `True`| Conditional on config validation \\n`depth`| `BOOLEAN`| `93547`| Error: data type of value does not match provided data type| | | No \\n \\nFor example, let\'s assume you\'d like to ingest a text score to capture **reviewer notes**. Text scores must be non-empty strings of at most 500 characters.\\nValue| Data Type| Config Id| Description| Inferred Data Type| Valid "}, {"type": "text", "text": "---|---|---|---|---|--- \\n`\\"Good response\\"`| `Null`| `Null`| Data type is inferred| `TEXT`| Yes \\n`\\"Good response\\"`| `TEXT`| `Null`| No properties inferred| | Yes \\n`0.9`| `TEXT`| `Null`| Error: data type of value does not match provided data type| | No \\n`\\"Good response\\"`| `TEXT`| `24680`| No properties inferred| | Conditional on config validation \\n`\\"Good response\\"`| `Null`| `24680`| Data type inferred| `TEXT`| Conditional on config validation \\n`\\"\\"`| `TEXT`| `Null`| Error: text scores must be non-empty| | No \\n \\n## Update Existing Scores via API/SDKs\\nWhen creating a score, you can provide an optional `id` (JS/TS) / `score_id` (Python) parameter. A re-ingested score overwrites an existing one only when its `id`, `name`, and `timestamp` (at date granularity, `toDate(timestamp)`) all match β€” a matching `id` alone is not sufficient, and any difference creates a duplicate instead of an update.\\nTo overwrite a score without fetching existing scores first, set a stable `id` as an idempotency key when initially creating the score, and keep the `name` and `timestamp` unchanged on subsequent calls. See Preventing Duplicate Scores for details.\\n## Related guides\\nGuides\\n * [Collecting user feedback\\nCapture explicit and implicit user feedback as scores on your traces, including in-app ratings sent via our Browser SDK.\\n](/guides/user-feedback-loop)\\n * [Custom evaluation data pipeline\\nContinuously monitor quality by fetching traces, running custom evaluations, and ingesting scores back into Langfuse.\\n](/guides/cookbook/example_external_evaluation_pipelines)\\n * [Guardrails and security checks"}, {"type": "text", "text": "\\nCheck whether output contains a certain keyword, matches a required structure or format, or exceeds a length limit.\\n](/guides/cookbook/security-and-guardrails)\\n * [Custom internal workflow tooling\\nBuild internal tooling for human-in-the-loop workflows, ingesting scores back into Langfuse following your custom schema.\\n](/guides/human-in-the-loop-scoring)\\n * [Session-level quality tracking\\nScore full conversations, such as support chats or agent threads, by attaching scores via sessionId in the SDK or API.\\n](/guides/cookbook/example_evaluating_multi_turn_conversations)\\n* * *\\n* * *\\nLast edited 7/22/2026\\n"}], "type": "content"}}, {"type": "document", "record_type": "documentation", "url": "https://langfuse.com/docs/evaluation/evaluation-methods/scores-via-sdk", "title": "Scores via API/SDK", "source": {"content": [{"type": "text", "text": "# Scores via API/SDK\\nYou can use the Langfuse SDKs or API to add [scores](/docs/evaluation/scores/overview) to traces, observations, sessions and dataset runs. This is an evaluation method that allows you to set up custom evaluation workflows and extend the scoring capabilities of Langfuse. See the [data model](/docs/evaluation/scores/data-model#scores) for full details on the score object.\\nIf you want Langfuse to run deterministic Python or TypeScript logic for you, use [code evaluators](/docs/evaluation/evaluation-methods/code-evaluators). Use this page when your application, pipeline, or CI job computes scores and sends them to Langfuse via API or SDK.\\n\\n## Ingesting Scores via API/SDKs\\nScores can be attached at different levels of granularity: to individual traces, to specific observations within a trace, or to full sessions.\\nSee the [API reference](/docs/api) for full details on the POST and GET endpoints for scores and score configs.\\n### Trace or Observation-level Scores\\nYou can add scores via the Langfuse SDKs or API. Scores can take one of four data types: **Numeric** , **Categorical** , **Boolean** , or **Text**. See [Score Types](/docs/evaluation/scores/overview#score-types) for details.\\nIf a score is ingested manually using a `trace_id` to link the score to a trace, it is not necessary to wait until the trace has been created. The score will show up in the scores table and will be linked to the trace once the trace with the same `trace_id` is created.\\nHere are examples by `Score` data types.\\nFor trace and observation scores, `trace_id`/`traceId` is required and `observation_id`/`observationId` is optional. If you attach a score to an observation, always provide both the observation ID and the corresponding trace ID.\\nPython SDKJS/TS SDKAPI\\nNumericCategoricalBooleanText\\nNumeric score values must be provided as float.\\n \\n \\n```"}, {"type": "text", "text": "\\nfrom langfuse import get_client\\nlangfuse = get_client()\\n# Method 1: Score via low-level method\\nlangfuse.create_score(\\n name=\\"correctness\\",\\n value=0.9,\\n trace_id=\\"trace_id_here\\",\\n observation_id=\\"observation_id_here\\", # optional\\n data_type=\\"NUMERIC\\", # optional, inferred if not provided\\n comment=\\"Factually correct\\", # optional\\n)\\n# Method 2: Score current span/generation (within context)\\nwith langfuse.start_as_current_observation(as_type=\\"span\\", name=\\"my-operation\\") as span:\\n # Score the current span\\n span.score(\\n name=\\"correctness\\",\\n value=0.9,\\n data_type=\\"NUMERIC\\",\\n comment=\\"Factually correct\\"\\n )\\n # Score the trace\\n span.score_trace(\\n name=\\"overall_quality\\",\\n value=0.95,\\n data_type=\\"NUMERIC\\"\\n )\\n# Method 3: Score via the current context\\nwith langfuse.start_as_current_observation(as_type=\\"span\\", name=\\"my-operation\\"):\\n # Score the current span\\n langfuse.score_current_span(\\n name=\\"correctness\\",\\n value=0.9,\\n data_type=\\"NUMERIC\\",\\n comment=\\"Factually correct\\"\\n )\\n # Score the trace\\n langfuse.score_current_trace(\\n name=\\"overall_quality\\",\\n value=0.95,\\n data_type=\\"NUMERIC\\"\\n )\\n```\\nCategorical score values must be provided as strings.\\n \\n \\n```\\nfrom langfuse import get_client\\nlangfuse = get_client()\\n# Method 1: Score via low-level method\\nlangfuse.create_score(\\n name=\\"accuracy\\",\\n value=\\"partially correct\\",\\n trace_id=\\"trace_id_here\\",\\n observation_id=\\"observation_id_here\\", # optional\\n data_type=\\"CATEGORICAL\\", # optional, inferred if not provided\\n comment=\\"Some factual errors\\", # optional\\n)\\n# Method 2: Score current span/generation (within context)\\nwith langfuse.start_as_current_observation(as_type=\\"span\\", name=\\"my-operation\\") as span:\\n # Score the current span\\n span.score(\\n name=\\"accuracy\\",\\n value=\\"partially correct\\",\\n data_type=\\"CATEGORICAL\\",\\n comment=\\"Some factual errors\\"\\n )\\n # Score the trace\\n span.score_trace(\\n name=\\"overall_quality\\",\\n value=\\"partially correct\\","}, {"type": "text", "text": "\\n data_type=\\"CATEGORICAL\\"\\n )\\n# Method 3: Score via the current context\\nwith langfuse.start_as_current_observation(as_type=\\"span\\", name=\\"my-operation\\"):\\n # Score the current span\\n langfuse.score_current_span(\\n name=\\"accuracy\\",\\n value=\\"partially correct\\",\\n data_type=\\"CATEGORICAL\\",\\n comment=\\"Some factual errors\\"\\n )\\n # Score the trace\\n langfuse.score_current_trace(\\n name=\\"overall_quality\\",\\n value=\\"partially correct\\",\\n data_type=\\"CATEGORICAL\\"\\n )\\n```\\nBoolean scores must be provided as a float, where `1` is true and `0` is false. When read via the [v3 scores API](https://api.reference.langfuse.com/#tag/scoresv3/GET/api/public/v3/scores), the value is returned as a boolean. See [API reference](/docs/api) for more details on POST/GET scores endpoints.\\n \\n \\n```\\nfrom langfuse import get_client\\nlangfuse = get_client()\\n# Method 1: Score via low-level method\\nlangfuse.create_score(\\n name=\\"helpfulness\\",\\n value=0, # 0 or 1\\n trace_id=\\"trace_id_here\\",\\n observation_id=\\"observation_id_here\\", # optional\\n data_type=\\"BOOLEAN\\", # required, numeric values without data type would be inferred as NUMERIC\\n comment=\\"Incorrect answer\\", # optional\\n)\\n# Method 2: Score current span/generation (within context)\\nwith langfuse.start_as_current_observation(as_type=\\"span\\", name=\\"my-operation\\") as span:\\n # Score the current span\\n span.score(\\n name=\\"helpfulness\\",\\n value=1, # 0 or 1\\n data_type=\\"BOOLEAN\\",\\n comment=\\"Very helpful response\\"\\n )\\n # Score the trace\\n span.score_trace(\\n name=\\"overall_quality\\",\\n value=1, # 0 or 1\\n data_type=\\"BOOLEAN\\"\\n )\\n# Method 3: Score via the current context\\nwith langfuse.start_as_current_observation(as_type=\\"span\\", name=\\"my-operation\\"):\\n # Score the current span\\n langfuse.score_current_span(\\n name=\\"helpfulness\\","}, {"type": "text", "text": "\\n value=1, # 0 or 1\\n data_type=\\"BOOLEAN\\",\\n comment=\\"Very helpful response\\"\\n )\\n # Score the trace\\n langfuse.score_current_trace(\\n name=\\"overall_quality\\",\\n value=1, # 0 or 1\\n data_type=\\"BOOLEAN\\"\\n )\\n```\\nText score values must be provided as strings between 1 and 500 characters.\\n \\n \\n```\\nfrom langfuse import get_client\\nlangfuse = get_client()\\n# Method 1: Score via low-level method\\nlangfuse.create_score(\\n name=\\"reviewer_notes\\",\\n value=\\"The response was helpful but could be more concise.\\",\\n trace_id=\\"trace_id_here\\",\\n observation_id=\\"observation_id_here\\", # optional\\n data_type=\\"TEXT\\", # optional, inferred if not provided\\n comment=\\"Reviewed by QA team\\", # optional\\n)\\n# Method 2: Score current span/generation (within context)\\nwith langfuse.start_as_current_observation(as_type=\\"span\\", name=\\"my-operation\\") as span:\\n # Score the current span\\n span.score(\\n name=\\"reviewer_notes\\",\\n value=\\"The response was helpful but could be more concise.\\",\\n data_type=\\"TEXT\\",\\n comment=\\"Reviewed by QA team\\"\\n )\\n # Score the trace\\n span.score_trace(\\n name=\\"overall_notes\\",\\n value=\\"Good quality overall, minor formatting issues.\\",\\n data_type=\\"TEXT\\"\\n )\\n# Method 3: Score via the current context\\nwith langfuse.start_as_current_observation(as_type=\\"span\\", name=\\"my-operation\\"):\\n # Score the current span\\n langfuse.score_current_span(\\n name=\\"reviewer_notes\\",\\n value=\\"The response was helpful but could be more concise.\\",\\n data_type=\\"TEXT\\",\\n comment=\\"Reviewed by QA team\\"\\n )\\n # Score the trace\\n langfuse.score_current_trace(\\n name=\\"overall_notes\\",\\n value=\\"Good quality overall, minor formatting issues.\\",\\n data_type=\\"TEXT\\"\\n )\\n```\\nNumericCategoricalBooleanText\\nNumeric score values must be provided as float.\\n \\n \\n```"}, {"type": "text", "text": "\\nimport { LangfuseClient } from \\"@langfuse/client\\";\\nconst langfuse = new LangfuseClient();\\nlangfuse.score.create({\\n id: \\"unique_id\\", // optional, can be used as an idempotency key to update the score subsequently\\n traceId: message.traceId,\\n observationId: message.generationId, // optional\\n name: \\"correctness\\",\\n value: 0.9,\\n dataType: \\"NUMERIC\\", // optional, inferred if not provided\\n comment: \\"Factually correct\\", // optional\\n});\\n// Flush the scores in short-lived environments\\nawait langfuse.flush();\\n```\\nCategorical score values must be provided as strings.\\n \\n \\n```\\nimport { LangfuseClient } from \\"@langfuse/client\\";\\nconst langfuse = new LangfuseClient();\\nlangfuse.score.create({\\n id: \\"unique_id\\", // optional, can be used as an idempotency key to update the score subsequently\\n traceId: message.traceId,\\n observationId: message.generationId, // optional\\n name: \\"accuracy\\",\\n value: \\"partially correct\\",\\n dataType: \\"CATEGORICAL\\", // optional, inferred if not provided\\n comment: \\"Factually correct\\", // optional\\n});\\n// Flush the scores in short-lived environments\\nawait langfuse.flush();\\n```\\nBoolean scores must be provided as a float, where `1` is true and `0` is false. When read via the [v3 scores API](https://api.reference.langfuse.com/#tag/scoresv3/GET/api/public/v3/scores), the value is returned as a boolean. See [API reference](/docs/api) for more details on POST/GET scores endpoints.\\n \\n \\n```\\nimport { LangfuseClient } from \\"@langfuse/client\\";\\nconst langfuse = new LangfuseClient();\\nlangfuse.score.create({\\n id: \\"unique_id\\", // optional, can be used as an idempotency key to update the score subsequently\\n traceId: message.traceId,\\n observationId: message.generationId, // optional"}, {"type": "text", "text": "\\nnpm install @langfuse/browser\\n```\\n \\n \\n```\\nimport { LangfuseBrowserClient } from \\"@langfuse/browser\\";\\nconst langfuse = new LangfuseBrowserClient({\\n publicKey: process.env.NEXT_PUBLIC_LANGFUSE_PUBLIC_KEY!,\\n baseUrl: process.env.NEXT_PUBLIC_LANGFUSE_BASE_URL, // optional, defaults to https://cloud.langfuse.com\\n});\\nawait langfuse.score({\\n traceId: message.traceId,\\n observationId: message.generationId, // optional\\n id: `user-feedback-${message.traceId}`, // optional, use as an idempotency key\\n name: \\"user-feedback\\",\\n value: 1, // 1 for positive, 0 for negative\\n dataType: \\"BOOLEAN\\",\\n comment: \\"Helpful answer\\", // optional\\n});\\n```\\n`score()` returns `{ id }` after ingestion succeeds. No `flush()` call is required.\\n### Session-level Scores\\nTo score an entire session (without attaching the score to a trace or observation), provide only `session_id` (Python SDK) or `sessionId` (JS/TS SDK and API).\\nPython SDKJS/TS SDKAPI\\n \\n \\n```\\nfrom langfuse import get_client\\nlangfuse = get_client()\\nlangfuse.create_score(\\n name=\\"session_quality\\",\\n value=0.85,\\n session_id=\\"session_id_here\\",\\n data_type=\\"NUMERIC\\",\\n comment=\\"Overall conversation quality\\"\\n)\\n```\\n \\n \\n```\\nimport { LangfuseClient } from \\"@langfuse/client\\";\\nconst langfuse = new LangfuseClient();\\nlangfuse.score.create({\\n name: \\"session_quality\\",\\n value: 0.85,\\n sessionId: \\"session_id_here\\",\\n dataType: \\"NUMERIC\\",\\n comment: \\"Overall conversation quality\\",\\n});\\nawait langfuse.flush();\\n```\\n \\n \\n```\\ncurl -X POST https://cloud.langfuse.com/api/public/scores \\\\\\n -u \\"pk-lf-...\\":\\"sk-lf-...\\" \\\\\\n -H \\"Content-Type: application/json\\" \\\\\\n -d \'{\\n \\"sessionId\\": \\"session_id_here\\",\\n \\"name\\": \\"session_quality\\",\\n \\"value\\": 0.85,\\n \\"dataType\\": \\"NUMERIC\\",\\n \\"comment\\": \\"Overall conversation quality\\"\\n }\'\\n```\\n"}], "type": "content"}}, {"type": "document", "record_type": "documentation", "url": "https://langfuse.com/docs/evaluation/evaluation-methods/scores-via-sdk", "title": "Scores via API/SDK", "source": {"content": [{"type": "text", "text": "## Advanced\\n### Preventing Duplicate Scores\\nBy default, Langfuse allows for multiple scores of the same `name` on the same trace. This is useful if you\'d like to track the evolution of a score over time or if e.g. you\'ve received multiple user feedback scores on the same trace.\\nIn some cases, you want to prevent this behavior or overwrite an existing score. A score is identified by three fields β€” its `id`, its `name`, and its date (`toDate(timestamp)`) β€” and a new score only replaces an existing one when all three match. To achieve this, create an **idempotency key** and pass it as the `id` (JS/TS) / `score_id` (Python) when creating the score (for example `trace_id-score_name`), and keep the `name` and `timestamp` unchanged across calls. If any of the three differs β€” for example the same `id` with a different `name` β€” you get an additional score rather than a replacement.\\nDo not send partial score updates (only the changed fields) and rely on Langfuse to merge them into the existing record. This merge only happens for a limited window after creation, is deprecated, and will be removed. Always send the complete score.\\nSee [How to update traces, observations, and scores](/faq/all/tracing-data-updates#updating-scores) for more details on Langfuse\'s immutable data model.\\n### Enforcing a Score Config\\nScore configs are helpful when you want to standardize your scores for future analysis.\\nTo enforce a score config, you can provide a `configId` when creating a score to reference a `ScoreConfig` that was previously created. `Score Configs` can be defined in the Langfuse UI or via our API. [See our guide on how to create and manage score configs](/faq/all/manage-score-configs).\\nWhenever you provide a `ScoreConfig`, the score data will be validated against the config. The following rules apply:\\n * **Score Name** : Must equal the config\'s name"}, {"type": "text", "text": "\\n * **Score Data Type** : When provided, must match the config\'s data type\\n * **Score Value when Type is`NUMERIC`**: Value must be within the min and max values defined in the config (if provided, min and max are optional and otherwise are assumed as -∞ and +∞ respectively)\\n * **Score Value when Type is`CATEGORICAL`**: Value must map to one of the categories defined in the config\\n * **Score Value when Type is`BOOLEAN`**: Value must equal `0` or `1`\\n * **Score Value when Type is`TEXT`**: Value must be a non-empty string of at most 500 characters\\nPython SDKJS/TS SDKAPI\\nNumeric ScoresCategorical ScoresBoolean ScoresText Scores\\nWhen ingesting numeric scores, you can provide the value as a float. If you provide a configId, the score value will be validated against the config\'s numeric range, which might be defined by a minimum and/or maximum value.\\n \\n \\n```\\nfrom langfuse import get_client\\nlangfuse = get_client()\\n# Method 1: Score via low-level method\\nlangfuse.create_score(\\n trace_id=\\"trace_id_here\\",\\n observation_id=\\"observation_id_here\\", # optional\\n session_id=\\"session_id_here\\", # optional, ID of the session the score relates to\\n name=\\"accuracy\\",\\n value=0.9,\\n comment=\\"Factually correct\\", # optional\\n score_id=\\"unique_id\\", # optional, can be used as an idempotency key to update the score subsequently\\n config_id=\\"78545-6565-3453654-43543\\", # optional, to ensure that the score follows a specific min/max value range\\n data_type=\\"NUMERIC\\" # optional, possibly inferred\\n)\\n# Method 2: Score within context\\nwith langfuse.start_as_current_observation(as_type=\\"span\\", name=\\"my-operation\\") as span:\\n span.score(\\n name=\\"accuracy\\",\\n value=0.9,\\n comment=\\"Factually correct\\",\\n config_id=\\"78545-6565-3453654-43543\\",\\n data_type=\\"NUMERIC\\"\\n )\\n```"}, {"type": "text", "text": "\\nWhen ingesting numeric scores, you can provide the value as a float. If you provide a configId, the score value will be validated against the config\'s numeric range, which might be defined by a minimum and/or maximum value.\\n \\n \\n```\\nimport { LangfuseClient } from \\"@langfuse/client\\";\\nconst langfuse = new LangfuseClient();\\nlangfuse.score.create({\\n traceId: message.traceId,\\n observationId: message.generationId, // optional\\n name: \\"accuracy\\",\\n value: 0.9,\\n comment: \\"Factually correct\\", // optional\\n id: \\"unique_id\\", // optional, can be used as an idempotency key to update the score subsequently\\n configId: \\"78545-6565-3453654-43543\\", // optional, to ensure that the score follows a specific min/max value range\\n dataType: \\"NUMERIC\\", // optional, possibly inferred\\n});\\n// Flush the scores in short-lived environments\\nawait langfuse.flush();\\n```\\nCategorical scores are used to evaluate data that falls into specific categories. When ingesting categorical scores, you can provide the value as a string. If you provide a configId, the score value will be validated against the config\'s categories.\\n \\n \\n```\\nimport { LangfuseClient } from \\"@langfuse/client\\";\\nconst langfuse = new LangfuseClient();\\nlangfuse.score.create({\\n traceId: message.traceId,\\n observationId: message.generationId, // optional\\n name: \\"correctness\\",\\n value: \\"correct\\",\\n comment: \\"Factually correct\\", // optional\\n id: \\"unique_id\\", // optional, can be used as an idempotency key to update the score subsequently\\n configId: \\"12345-6565-3453654-43543\\", // optional, to ensure that a score maps to a specific category defined in a score config\\n dataType: \\"CATEGORICAL\\", // optional, possibly inferred\\n});"}, {"type": "text", "text": "\\nYou can also enforce score configs via the [REST API](https://api.reference.langfuse.com/#tag/legacyscorev1/POST/api/public/scores) by providing a `configId`.\\nNumeric ScoresCategorical ScoresBoolean ScoresText Scores\\nWhen ingesting numeric scores, you can provide the value as a float. If you provide a configId, the score value will be validated against the config\'s numeric range.\\n \\n \\n```\\ncurl -X POST https://cloud.langfuse.com/api/public/scores \\\\\\n -u \\"pk-lf-...\\":\\"sk-lf-...\\" \\\\\\n -H \\"Content-Type: application/json\\" \\\\\\n -d \'{\\n \\"id\\": \\"unique_id\\",\\n \\"traceId\\": \\"trace_id_here\\",\\n \\"observationId\\": \\"observation_id_here\\",\\n \\"name\\": \\"accuracy\\",\\n \\"value\\": 0.9,\\n \\"dataType\\": \\"NUMERIC\\",\\n \\"configId\\": \\"78545-6565-3453654-43543\\",\\n \\"comment\\": \\"Factually correct\\"\\n }\'\\n```\\nCategorical scores are used to evaluate data that falls into specific categories. If you provide a configId, the score value will be validated against the config\'s categories.\\n \\n \\n```\\ncurl -X POST https://cloud.langfuse.com/api/public/scores \\\\\\n -u \\"pk-lf-...\\":\\"sk-lf-...\\" \\\\\\n -H \\"Content-Type: application/json\\" \\\\\\n -d \'{\\n \\"id\\": \\"unique_id\\",\\n \\"traceId\\": \\"trace_id_here\\",\\n \\"observationId\\": \\"observation_id_here\\",\\n \\"name\\": \\"correctness\\",\\n \\"value\\": \\"correct\\",\\n \\"dataType\\": \\"CATEGORICAL\\",\\n \\"configId\\": \\"12345-6565-3453654-43543\\",\\n \\"comment\\": \\"Factually correct\\"\\n }\'\\n```\\nWhen ingesting boolean scores, you can provide the value as a float. If you provide a configId, the score\'s name and config\'s name must match as well as their data types.\\n \\n \\n```\\ncurl -X POST https://cloud.langfuse.com/api/public/scores \\\\\\n -u \\"pk-lf-...\\":\\"sk-lf-...\\" \\\\\\n -H \\"Content-Type: application/json\\" \\\\\\n -d \'{\\n \\"id\\": \\"unique_id\\",\\n \\"traceId\\": \\"trace_id_here\\",\\n \\"observationId\\": \\"observation_id_here\\",\\n \\"name\\": \\"helpfulness\\","}, {"type": "text", "text": "\\n \\"value\\": 1,\\n \\"dataType\\": \\"BOOLEAN\\",\\n \\"configId\\": \\"93547-6565-3453654-43543\\",\\n \\"comment\\": \\"Factually correct\\"\\n }\'\\n```\\nWhen ingesting text scores, provide the value as a string (1–500 characters). If you provide a configId, the score\'s name and config\'s name must match as well as their data types.\\n \\n \\n```\\ncurl -X POST https://cloud.langfuse.com/api/public/scores \\\\\\n -u \\"pk-lf-...\\":\\"sk-lf-...\\" \\\\\\n -H \\"Content-Type: application/json\\" \\\\\\n -d \'{\\n \\"id\\": \\"unique_id\\",\\n \\"traceId\\": \\"trace_id_here\\",\\n \\"observationId\\": \\"observation_id_here\\",\\n \\"name\\": \\"reviewer_notes\\",\\n \\"value\\": \\"The response was helpful but could be more concise.\\",\\n \\"dataType\\": \\"TEXT\\",\\n \\"configId\\": \\"24680-6565-3453654-43543\\",\\n \\"comment\\": \\"Reviewed by QA team\\"\\n }\'\\n```\\nSee [API reference](/docs/api) for more details on POST/GET score configs endpoints.\\n### Inferred Score Properties\\nCertain score properties might be inferred based on your input:\\n * **If you don\'t provide a score data type** it will always be inferred. See tables below for details.\\n * **For boolean and categorical scores** , we will provide the score value in both numerical and string format where possible. The score value format that is not provided as input, i.e. the translated value is referred to as the inferred value in the tables below.\\n * **On read via the[v3 scores API](https://api.reference.langfuse.com/#tag/scoresv3/GET/api/public/v3/scores)**, boolean scores are returned as a single boolean `value`.\\n * **For categorical scores** , the string representation is always provided and a numerical mapping of the category will be produced only if a `ScoreConfig` was provided.\\nDetailed Examples:"}, {"type": "text", "text": "\\nNumeric ScoresCategorical ScoresBoolean ScoresText Scores\\nFor example, let\'s assume you\'d like to ingest a numeric score to measure **accuracy**. We have included a table of possible score ingestion scenarios below.\\nValue| Data Type| Config Id| Description| Inferred Data Type| Valid \\n---|---|---|---|---|--- \\n`0.9`| `Null`| `Null`| Data type is inferred| `NUMERIC`| Yes \\n`0.9`| `NUMERIC`| `Null`| No properties inferred| | Yes \\n`depth`| `NUMERIC`| `Null`| Error: data type of value does not match provided data type| | No \\n`0.9`| `NUMERIC`| `78545`| No properties inferred| | Conditional on config validation \\n`0.9`| `Null`| `78545`| Data type inferred| `NUMERIC`| Conditional on config validation \\n`depth`| `NUMERIC`| `78545`| Error: data type of value does not match provided data type| | No \\n \\nFor example, let\'s assume you\'d like to ingest a categorical score to measure **correctness**. We have included a table of possible score ingestion scenarios below.\\nValue| Data Type| Config Id| Description| Inferred Data Type| Inferred Value representation| Valid \\n---|---|---|---|---|---|--- \\n`correct`| `Null`| `Null`| Data type is inferred| `CATEGORICAL`| | Yes \\n`correct`| `CATEGORICAL`| `Null`| No properties inferred| | | Yes \\n`1`| `CATEGORICAL`| `Null`| Error: data type of value does not match provided data type| | | No \\n`correct`| `CATEGORICAL`| `12345`| Numeric value inferred| | `4` numeric config category mapping| Conditional on config validation \\n`correct`| `NULL`| `12345`| Data type inferred| `CATEGORICAL`| | Conditional on config validation \\n`1`| `CATEGORICAL`| `12345`| Error: data type of value does not match provided data type| | | No \\n "}, {"type": "text", "text": "For example, let\'s assume you\'d like to ingest a boolean score to measure **helpfulness**. We have included a table of possible score ingestion scenarios below.\\nValue| Data Type| Config Id| Description| Inferred Data Type| Inferred Value representation| Valid \\n---|---|---|---|---|---|--- \\n`1`| `BOOLEAN`| `Null`| Value\'s string equivalent inferred| | `True`| Yes \\n`true`| `BOOLEAN`| `Null`| Error: data type of value does not match provided data type| | | No \\n`3`| `BOOLEAN`| `Null`| Error: boolean data type expects `0` or `1` as input value| | | No \\n`0.9`| `Null`| `93547`| Data type and value\'s string equivalent inferred| `BOOLEAN`| `True`| Conditional on config validation \\n`depth`| `BOOLEAN`| `93547`| Error: data type of value does not match provided data type| | | No \\n \\nFor example, let\'s assume you\'d like to ingest a text score to capture **reviewer notes**. Text scores must be non-empty strings of at most 500 characters.\\nValue| Data Type| Config Id| Description| Inferred Data Type| Valid \\n---|---|---|---|---|--- \\n`\\"Good response\\"`| `Null`| `Null`| Data type is inferred| `TEXT`| Yes \\n`\\"Good response\\"`| `TEXT`| `Null`| No properties inferred| | Yes \\n`0.9`| `TEXT`| `Null`| Error: data type of value does not match provided data type| | No \\n`\\"Good response\\"`| `TEXT`| `24680`| No properties inferred| | Conditional on config validation \\n`\\"Good response\\"`| `Null`| `24680`| Data type inferred| `TEXT`| Conditional on config validation \\n`\\"\\"`| `TEXT`| `Null`| Error: text scores must be non-empty| | No \\n \\n\\n## Update Existing Scores via API/SDKs"}, {"type": "text", "text": "When creating a score, you can provide an optional `id` (JS/TS) / `score_id` (Python) parameter. A re-ingested score overwrites an existing one only when its `id`, `name`, and `timestamp` (at date granularity, `toDate(timestamp)`) all match β€” a matching `id` alone is not sufficient, and any difference creates a duplicate instead of an update.\\nTo overwrite a score without fetching existing scores first, set a stable `id` as an idempotency key when initially creating the score, and keep the `name` and `timestamp` unchanged on subsequent calls. See Preventing Duplicate Scores for details.\\n## Related guides\\nGuides\\n * [Collecting user feedback\\nCapture explicit and implicit user feedback as scores on your traces, including in-app ratings sent via our Browser SDK.\\n](/guides/user-feedback-loop)\\n * [Custom evaluation data pipeline\\nContinuously monitor quality by fetching traces, running custom evaluations, and ingesting scores back into Langfuse.\\n](/guides/cookbook/example_external_evaluation_pipelines)\\n * [Guardrails and security checks\\nCheck whether output contains a certain keyword, matches a required structure or format, or exceeds a length limit.\\n](/guides/cookbook/security-and-guardrails)\\n * [Custom internal workflow tooling\\nBuild internal tooling for human-in-the-loop workflows, ingesting scores back into Langfuse following your custom schema.\\n](/guides/human-in-the-loop-scoring)\\n * [Session-level quality tracking\\nScore full conversations, such as support chats or agent threads, by attaching scores via sessionId in the SDK or API.\\n](/guides/cookbook/example_evaluating_multi_turn_conversations)\\n* * *\\n* * *\\nLast edited 7/22/2026\\n"}], "type": "content"}}, {"type": "document", "record_type": "documentation", "url": "https://langfuse.com/docs/evaluation/scores/overview", "title": "Scores", "source": {"content": [{"type": "text", "text": "Overview - Langfuse\\n[Docs](/docs)\\n[Evaluation](/docs/evaluation/overview)\\n[Scores](/docs/evaluation/scores/overview)\\n[Overview](/docs/evaluation/scores/overview)\\n# Scores\\nScores are Langfuse\'s universal data object for storing evaluation results. Any time you want to assign a quality judgment to an LLM output, whether by a [human annotation](/docs/evaluation/evaluation-methods/scores-via-ui), an [LLM judge](/docs/evaluation/evaluation-methods/llm-as-a-judge), a [programmatic check](/docs/evaluation/evaluation-methods/scores-via-sdk), or end-user feedback, the result is stored as a score.\\nEvery score has a **name** (like `\\"correctness\\"` or `\\"helpfulness\\"`), a **value** , and a **data type**. Scores also support an optional **comment** for additional context.\\nScores can be attached to [traces](/docs/observability/data-model#observations-and-traces), [observations](/docs/observability/data-model#observations-and-traces), [sessions](/docs/observability/data-model#sessions), or [dataset runs](/docs/evaluation/experiments/data-model). Most commonly, scores are attached to traces to evaluate a single end-to-end interaction.\\nOnce you have scores, they show up in [score analytics](/docs/evaluation/scores/score-analytics), can be visualized in [custom dashboards](/docs/metrics/features/custom-dashboards), and can be queried via the [API](/docs/api).\\n## When to Use Scores\\nScores become useful when you want to go beyond observing what your application does and start measuring how well it does it. Common use cases:\\n * **Collecting user feedback** : Capture thumbs up/down or star ratings from your users and attach them to traces. See the [user feedback guide](/docs/observability/features/user-feedback)."}, {"type": "text", "text": "\\n * **Monitoring production quality** : Set up automated evaluators (like [LLM-as-a-Judge](/docs/evaluation/evaluation-methods/llm-as-a-judge)) to continuously score live traces for things like hallucination, relevance, or tone.\\n * **Running guardrails** : Score whether outputs pass safety checks like PII detection, format validation, or content policy compliance.\\n * **Comparing changes with experiments** : When you change a prompt, model, or pipeline, run an [experiment](/docs/evaluation/experiments) to score the new version against a dataset.\\n## Score Types\\nLangfuse supports four score data types:\\nType| Value| Use when \\n---|---|--- \\n`NUMERIC`| Float (e.g. `0.9`)| Continuous judgments like accuracy, relevance, or similarity scores \\n`CATEGORICAL`| String from predefined categories (e.g. `\\"correct\\"`, `\\"partially correct\\"`)| Discrete classifications where the set of possible values is known upfront \\n`BOOLEAN`| `0` or `1`| Pass/fail checks like hallucination detection or format validation \\n`TEXT`| Free-form string (1-500 characters)| Open-ended annotations like reviewer notes or qualitative feedback. Often used for [open coding](https://en.wikipedia.org/wiki/Open_coding) before formalizing into quantifiable scores via [axial coding](https://en.wikipedia.org/wiki/Axial_coding). \\n \\nText scores are designed for qualitative, open-ended scoring. Because free-form text cannot be meaningfully aggregated or compared, text scores are not supported in [experiments](/docs/evaluation/core-concepts#experiments), [LLM-as-a-Judge](/docs/evaluation/evaluation-methods/llm-as-a-judge), or [score analytics](/docs/evaluation/scores/score-analytics).\\n## How to Create Scores"}, {"type": "text", "text": "\\nThere are five ways to add scores:\\n * **LLM-as-a-Judge** : Set up [automated evaluators](/docs/evaluation/evaluation-methods/llm-as-a-judge) that score traces based on custom criteria (e.g. hallucination, tone, relevance). These can return numeric or categorical scores plus reasoning, and can run on live production traces or on experiment results.\\n * **Code evaluators** : Run [custom Python or TypeScript evaluators](/docs/evaluation/evaluation-methods/code-evaluators) in Langfuse for deterministic checks such as exact match, JSON validation, or custom business rules.\\n * **Scores via UI** : Team members [manually score](/docs/evaluation/evaluation-methods/scores-via-ui) traces, observations, or sessions directly in the Langfuse UI. Requires a [score config](/faq/all/manage-score-configs) to be set up first.\\n * **Annotation Queues** : Set up [structured review workflows](/docs/evaluation/evaluation-methods/annotation-queues) where reviewers work through batches of traces.\\n * **Scores via API/SDK** : [Programmatically add scores](/docs/evaluation/evaluation-methods/scores-via-sdk) from your application code. This is the way to go for user feedback (thumbs up/down, star ratings), guardrail results, or custom evaluation pipelines.\\n## Should I Use Scores or Tags?\\n| Scores| Tags \\n---|---|--- \\n **Purpose**| Measure _how good_ something is| Describe _what_ something is \\n **Data**| Numeric, categorical, boolean, or text value| Simple string label \\n **When added**| Can be added at any time, including long after the trace was created| Set during tracing and cannot be changed afterwards \\n **Used for**| Quality measurement, analytics, experiments| Filtering, segmentation, organizing \\n "}, {"type": "text", "text": "As a rule of thumb: if you already know the category at tracing time (e.g. which feature or API endpoint triggered the trace), use a [tag](/docs/observability/features/tags). If you need to classify or evaluate traces later, use a score.\\n## Score Comments\\nEvery score supports an optional **comment** field. Use it to capture reasoning (e.g. why an LLM judge assigned a particular score), reviewer notes, or context that helps others understand the score value. Comments are shown alongside scores in the Langfuse UI.\\nUse a `TEXT` score instead of comments to capture standalone qualitative feedback -- comments are best for additional reasoning on an existing score.\\n* * *\\n* * *\\nLast edited 7/23/2026\\n"}], "type": "content"}}, {"type": "document", "record_type": "github_discussion", "url": "https://github.com/orgs/langfuse/discussions/6016", "title": "Using scores in data sets from the SDK?", "source": {"content": [{"type": "text", "text": "Title: Using scores in data sets from the SDK?\\nCategory: Support\\nAuthor: @howinator\\nTimestamp: March 14, 2025\\nBody:\\nHere is a workflow that I want, but I can\'t figure out how to accomplish it without a lot of manual work.\\r\\n\\r\\n1. Collect traces\\r\\n2. Annotate traces with either manual annotation or programmatic feedback from users. For a simple classification, I would use a boolean score where True means the model returned the correct answer and False when it returned the incorrect answer.\\r\\n4. Create a \\"false positive\\" data set composed of all traces where the score was False. <-- I can\'t figure out how to do this without a lot of clicking.\\r\\n5. Run an eval from that data set and use the score to determine if the eval was correct, e.g., something like `(dataset.item.score && dataset.item.output === evalOutput.output) || (!dataset.item.score && dataset.item.output !== evalOutput.output)`. <-- I can\'t figure out how to do this programmatically at all.\\r\\n\\r\\nIs my workflow completely off here? Am I missing something? Or is this flow just not possible in Langfuse today?\\nComments:\\n\\n Comment #1 from @dosubot on March 14, 2025:\\n\\n <!-- Greeting -->\\nHey there, @howinator! I\'m here to help you out with any bugs, questions, or contributions you have in mind. Let\'s get to work!\\n\\n<!-- Answer -->\\nYour workflow is mostly feasible in Langfuse, and you can achieve it programmatically with some adjustments:\\n\\n1. **Collect Traces**: This step is straightforward and can be done using Langfuse\'s SDKs to capture and store traces.\\n\\n2. **Annotate Traces**: You can programmatically annotate traces with scores using the `langfuse_context.score_current_trace` function. This allows you to attach a boolean score to each trace based on user feedback or other criteria <sup>[[1]](https://langfuse.com/guides/cookbook/integration_langgraph)</sup>.\\n"}, {"type": "text", "text": "\\n3. **Create a \\"False Positive\\" Dataset**: While there isn\'t a direct method to filter dataset items based on scores, you can programmatically create a dataset by iterating over your traces and selecting those with a score of `False`. Use the SDK to create dataset items for these traces <sup>[[2]](https://langfuse.com/docs/datasets/get-started)</sup>.\\n\\n4. **Run an Evaluation**: You can automate the creation of evaluation jobs for your dataset using the `createEvalJobs` function. This function allows you to specify configurations and run evaluations on specific subsets of dataset items <sup>[[3]](https://github.com/langfuse/langfuse/blob/eecf0d680b13a4bde9fe8fed617acc328029307d/worker/src/__tests__/evalService.test.ts)</sup>.\\n\\nFor the evaluation logic, you can implement custom logic to compare the dataset item scores and outputs with the evaluation outputs. This might require some additional scripting to handle the specific logic you described.\\n\\nOverall, your workflow is possible with Langfuse, but it may require some custom scripting to fully automate the process of filtering and evaluating dataset items based on scores.\\n\\n\\n<!-- Dosu Comment Footer -->\\n*To continue talking to [Dosu](https://dosu.dev), mention @dosu.*\\n\\n---"}, {"type": "text", "text": "1. Create a \\"false positive\\" data set composed of all traces where the score was False -> use scores api to fetch all of these\\r\\n2. Run your own eval on them and add it to the same trace via the sdk\\r\\n3. Compare the evals by fetching all of them via the sdk again.\\r\\n\\r\\nWe currently plan to add more configurable dashboards which will allow for the comparison of these evals more easily in the UI.\\n"}], "type": "content"}}, {"type": "document", "record_type": "documentation", "url": "https://langfuse.com/docs/observability/troubleshooting-and-faq", "title": "Troubleshooting and FAQ", "source": {"content": [{"type": "text", "text": "\\n[LLM as a Session-Level Judge with Support for LangGraph Session](https://github.com/orgs/langfuse/discussions/8303 \\"Langfuse Ideas: LLM as a Session-Level Judge with Support for LangGraph Session\\")\\n[Score Configs: Allow editing the categories of a categorical score](https://github.com/orgs/langfuse/discussions/8259 \\"Langfuse Ideas: Score Configs: Allow editing the categories of a categorical score\\")\\n[feat(trace-detail-view): for each span, show the sum of nested observations\' token usage details](https://github.com/orgs/langfuse/discussions/8201 \\"Langfuse Ideas: feat\\\\(trace-detail-view\\\\): for each span, show the sum of nested observations\' token usage details\\")\\n[UI-LLM as a Jury](https://github.com/orgs/langfuse/discussions/8195 \\"Langfuse Ideas: UI-LLM as a Jury\\")\\n[Add Gemini 2.5 Pro model pricing](https://github.com/orgs/langfuse/discussions/8146 \\"Langfuse Ideas: Add Gemini 2.5 Pro model pricing\\")\\n[Deeplinking for Spans inside of a Trace](https://github.com/orgs/langfuse/discussions/8112 \\"Langfuse Ideas: Deeplinking for Spans inside of a Trace\\")\\n[feat(comments): allow attaching images](https://github.com/orgs/langfuse/discussions/8111 \\"Langfuse Ideas: feat\\\\(comments\\\\): allow attaching images\\")\\n[feat(sessions): support session-level tags](https://github.com/orgs/langfuse/discussions/8041 \\"Langfuse Ideas: feat\\\\(sessions\\\\): support session-level tags\\")\\n[Enable Immediate Score Management for User Feedback](https://github.com/orgs/langfuse/discussions/7686 \\"Langfuse Ideas: Enable Immediate Score Management for User Feedback\\")\\n[Allow for structured filtering on inputs / outputs](https://github.com/orgs/langfuse/discussions/7663 \\"Langfuse Ideas: Allow for structured filtering on inputs / outputs\\")"}, {"type": "text", "text": "\\n[Allow for lazy updating of traces in Python SDK (+ possible workaround)](https://github.com/orgs/langfuse/discussions/7655 \\"Langfuse Ideas: Allow for lazy updating of traces in Python SDK \\\\(+ possible workaround\\\\)\\")\\n[Filter by scores in session view](https://github.com/orgs/langfuse/discussions/7528 \\"Langfuse Ideas: Filter by scores in session view\\")\\n[Support break lines on evaluation run tooltip hint](https://github.com/orgs/langfuse/discussions/7452 \\"Langfuse Ideas: Support break lines on evaluation run tooltip hint\\")\\n[Enable sending Langfuse SDK OTEL spans to third party observability backend](https://github.com/orgs/langfuse/discussions/7365 \\"Langfuse Ideas: Enable sending Langfuse SDK OTEL spans to third party observability backend\\")\\n[Add Thinking config in playground for anthropic models](https://github.com/orgs/langfuse/discussions/7267 \\"Langfuse Ideas: Add Thinking config in playground for anthropic models\\")\\n[Better support for big int in metadata.](https://github.com/orgs/langfuse/discussions/7167 \\"Langfuse Ideas: Better support for big int in metadata.\\")\\n[Ability to filter/search comments on traces](https://github.com/orgs/langfuse/discussions/7109 \\"Langfuse Ideas: Ability to filter/search comments on traces\\")\\n[Image / File / Multimedia rendering in dataset items using mediaId](https://github.com/orgs/langfuse/discussions/6883 \\"Langfuse Ideas: Image / File / Multimedia rendering in dataset items using mediaId\\")\\n[Trace Comments support upload image](https://github.com/orgs/langfuse/discussions/6857 \\"Langfuse Ideas: Trace Comments support upload image\\")"}], "type": "content"}}, {"type": "document", "record_type": "github_issue", "url": "https://github.com/langfuse/langfuse/issues/3380", "title": "feat: add fetch_score(s) in the sdks", "source": {"content": [{"type": "text", "text": "Github Issue by @marcklingen created on September 17, 2024. The current state of the issue is: closed\\n\\n### Discussed in https://github.com/orgs/langfuse/discussions/3379\\r\\n\\r\\n<div type=\'discussions-op-text\'>\\r\\n\\r\\n<sup>Originally posted by **Preet1234-ux** September 17, 2024</sup>\\r\\nI want to fetch the score provided by the user from langfuse in my python code . Are there any functionality by which I can fetch the score for a trace ? </div>\\r\\n\\r\\n\\r\\nCurrently, we need to use either langfuse.client.*(potentially breaking) or the api.\\n---\\nComment #1 by @hassiebp:\\n\\nAlready shipped in Python SDK v3\\n\\n"}], "type": "content"}}, {"type": "document", "record_type": "github_discussion", "url": "https://github.com/orgs/langfuse/discussions/6210", "title": "Next.js + Vercel AI SDK: How to get `observationId` and attach it to scores?", "source": {"content": [{"type": "text", "text": "Title: Next.js + Vercel AI SDK: How to get `observationId` and attach it to scores?\\nCategory: Support\\nAuthor: @natterstefan\\nTimestamp: March 26, 2025\\nBody:\\nDear langfuse-Community,\\r\\n\\r\\nI need your help please, as I do not know how to obtain the `observationId` and attach it to a `score`.\\r\\n\\r\\nContext: this is how my `experimental_telemetry` looks like:\\r\\n\\r\\n```js\\r\\n// src: https://langfuse.com/docs/integrations/vercel-ai-sdk#group-multiple-executions-in-one-trace\\r\\n\\r\\nconst parentTraceId = getRandomId()\\r\\n\\r\\nexport async function POST(req: Request) {\\r\\n\\t// ... other code\\r\\n\\r\\n\\tconst trace = langfuse.trace({\\r\\n\\t id: parentTraceId,\\r\\n\\t name: \'cd-chat-messages\',\\r\\n\\t})\\r\\n\\r\\n\\tconsole.log(\'trace\', trace.observationId) // <<-- this returns `null`\\r\\n\\t\\r\\n\\tconst result = streamText({\\r\\n\\t\\t// ....\\r\\n\\t\\texperimental_telemetry: {\\r\\n\\t\\t isEnabled: true,\\r\\n\\t\\t functionId: \'cd-chat-message-\' + messages.length,\\r\\n\\t\\t metadata: {\\r\\n\\t\\t langfuseTraceId: parentTraceId,\\r\\n\\t\\t\\t // Do not update the parent trace with execution results\\r\\n\\t\\t langfuseUpdateParent: false,\\r\\n\\t\\t },\\r\\n\\t\\t },\\r\\n\\t})\\r\\n\\r\\n\\t// ... other code\\r\\n}\\r\\n```\\r\\n\\r\\nThe docs state\\r\\n\\r\\n> The id of the observation to which the score should be attached. Pass from backend trace. E.g. {generation, event, span}.id\\r\\n\\r\\nBut how and where do I get the observationId from, taking into account that I use `streamText` (<https://sdk.vercel.ai/docs/reference/ai-sdk-core/stream-text>)\\r\\n\\r\\n```js\\r\\n// source: https://langfuse.com/docs/sdk/typescript/guide-web#create-scores\\r\\n\\r\\n// pass traceId and observationId to front end\\r\\nawait langfuseWeb.score({\\r\\n traceId: message.traceId,\\r\\n observationId: message.observationId, <<<-- where do I get this?\\r\\n name: \\"user-feedback\\",\\r\\n value: 1,\\r\\n comment: \\"I like how personalized the response is\\",\\r\\n});\\r\\n```\\r\\n\\r"}, {"type": "text", "text": "Another question I could not answer myself is where (= which \\"layer\\". I guess generation, event, span is the correct wording) a score should be attached to. Is there any best practice recommendation or should I always go for the observation (`ai.streamText.doStream` in this example _if_ that\'s the observationId I can obtain)?\\r\\n\\r\\n![image](https://github.com/user-attachments/assets/e103abd2-3fd5-4607-9c09-6d0ad2b15393)\\r\\n\\r\\nThank you very much for your time and help!\\nComments:\\n\\n Comment #1 from @dosubot on March 26, 2025:\\n\\n <!-- Greeting -->\\nHey @natterstefan! I\'m here to help you with any bugs, questions, or even becoming a contributor. Let\'s squash those bugs together! πŸ€–\\n\\n<!-- Answer -->\\nTo obtain the `observationId` when using the `streamText` function in the Vercel AI SDK, you should look for it in the response from the `streamText` function. The SDK typically uses the ID from the provider response if available, or it generates one if not. Ensure that your telemetry setup is configured to pass the `observationId` from the backend trace, specifically from the context of the `streamText` execution.\\n\\nRegarding where to attach the score, it is generally recommended to attach it to the most relevant observation. In your case, if `ai.streamText.doStream` is the observation you can obtain, it would be appropriate to attach the score there. This aligns with best practices, as it ensures the score is associated with the specific operation or event you are evaluating.\\n"}, {"type": "text", "text": "\\nFor more detailed guidance, you might want to refer to the Langfuse documentation on creating scores and handling observations <sup>[[1]](https://langfuse.com/docs/sdk/typescript)</sup><sup>[[2]](https://langfuse.com/docs/sdk/typescript/guide)</sup>.\\n\\n\\n<!-- Dosu Comment Footer -->\\n*To continue talking to [Dosu](https://dosu.dev), mention @dosu.*\\n\\n---\\n<sup>Help Dosu learn! Give it feedback: [Great Response](https://app.dosu.dev/response-feedback/68b78af0-2c34-4867-b75c-008bcf449b81?feedback_type=great_response) | [Irrelevant Answer](https://app.dosu.dev/response-feedback/68b78af0-2c34-4867-b75c-008bcf449b81?feedback_type=irrelevant_answer) | [Incorrect Sources](https://app.dosu.dev/response-feedback/68b78af0-2c34-4867-b75c-008bcf449b81?feedback_type=incorrect_sources) | [Too Verbose](https://app.dosu.dev/response-feedback/68b78af0-2c34-4867-b75c-008bcf449b81?feedback_type=too_verbose) | [Hallucination](https://app.dosu.dev/response-feedback/68b78af0-2c34-4867-b75c-008bcf449b81?feedback_type=hallucination) | [Bug Report](https://app.dosu.dev/response-feedback/68b78af0-2c34-4867-b75c-008bcf449b81?feedback_type=bug_report) | [Other](https://app.dosu.dev/response-feedback/68b78af0-2c34-4867-b75c-008bcf449b81?feedback_type=other)</sup>\\n\\n\\n Comment #2 from @marcklingen on March 26, 2025:\\n\\n This seems like a feature request for the AI SDK as you\'d need the SDK to expose the span ids of each of the requests.\\n\\n Replies to comment #2:\\n\\n Reply #1 to comment #2 from marcklingen on March 27, 2025:\\n\\n Hi @marcklingen,\\r\\n\\r\\nthanks for your help. Alright, I [moved (aka copied) the discussion to their repo](https://github.com/vercel/ai/discussions/5415) and hope they can help me.\\r\\n\\r\\nIs @dosubot correct with \\"_(...) In your case, if ai.streamText.doStream is the observation you can obtain, it would be appropriate to attach the score there._\\"\\r\\n\\r"}, {"type": "text", "text": "\\n<sup>Help Dosu learn! Give it feedback: [Great Response](https://app.dosu.dev/response-feedback/d9fcd303-8526-48dc-85b4-1432b3d12429?feedback_type=great_response) | [Irrelevant Answer](https://app.dosu.dev/response-feedback/d9fcd303-8526-48dc-85b4-1432b3d12429?feedback_type=irrelevant_answer) | [Incorrect Sources](https://app.dosu.dev/response-feedback/d9fcd303-8526-48dc-85b4-1432b3d12429?feedback_type=incorrect_sources) | [Too Verbose](https://app.dosu.dev/response-feedback/d9fcd303-8526-48dc-85b4-1432b3d12429?feedback_type=too_verbose) | [Hallucination](https://app.dosu.dev/response-feedback/d9fcd303-8526-48dc-85b4-1432b3d12429?feedback_type=hallucination) | [Bug Report](https://app.dosu.dev/response-feedback/d9fcd303-8526-48dc-85b4-1432b3d12429?feedback_type=bug_report) | [Other](https://app.dosu.dev/response-feedback/d9fcd303-8526-48dc-85b4-1432b3d12429?feedback_type=other)</sup>\\n\\n\\n Reply #3 to comment #2 from marcklingen on March 27, 2025:\\n\\n Hi @marcklingen and @dosubot,\\r\\n\\r\\nFollow up question (once I get the `generation.id`/`response.id` whatever vercel/ai calls it): how do I attach it to the current trace to make sure the observation in langfuse has the same id as the score in my code uses?\\n\\n Reply #4 to comment #2 from marcklingen on March 27, 2025:\\n\\n <!-- Answer -->\\nTo attach the `generation.id` or `response.id` from Vercel AI to the current trace in Langfuse, use the `generation.id` as the `observationId` when logging user feedback. This ensures that the observation has the same ID as the score used in your code. Here\'s how you can do it:\\n\\n1. Add the `generation.id` to the streaming response as a custom header:\\n\\n ```javascript\\n return new StreamingTextResponse(stream, {\\n headers: {\\n \\"X-Message-Id\\": lfGeneration.id,"}], "type": "content"}}]}',
+ },
+ ],
+ },
+ },
+ ],
+ },
+ ],
+ tools: [
+ {
+ type: "function",
+ name: "submitFeedback",
+ inputSchema: {
+ type: "object",
+ properties: {
+ targetType: {
+ type: "string",
+ enum: ["skill", "mcp-tool", "cli", "docs", "public-api", "other"],
+ description: "Category of the thing the feedback is about.",
+ },
+ target: {
+ type: "string",
+ minLength: 1,
+ maxLength: 200,
+ description:
+ "The specific instance within targetType: the skill name, MCP tool name, CLI command, API endpoint path, or docs page path (e.g. 'queryMetrics', '/docs/mcp'). An identifier, not a sentence.",
+ },
+ feedback: {
+ type: "string",
+ minLength: 1,
+ maxLength: 3000,
+ },
+ goal: {
+ type: "string",
+ minLength: 1,
+ maxLength: 1500,
+ },
+ referenceUrl: {
+ type: "string",
+ format: "uri",
+ maxLength: 2048,
+ },
+ },
+ required: ["targetType", "target", "feedback"],
+ additionalProperties: false,
+ $schema: "http://json-schema.org/draft-07/schema#",
+ },
+ description:
+ "Submit explicit user-approved feedback to the Langfuse team.\nBefore calling, ask permission and show the exact payload.\nIf the user wants a reply, ask them to include their email address in the feedback text; only use an address they explicitly provide and show it in the exact payload preview.\nDo not include secrets, credentials, customer/project data, trace payloads, or unrelated context; the only contact detail to include is an explicitly provided reply email.",
+ },
+ {
+ type: "function",
+ name: "searchLangfuseDocs",
+ inputSchema: {
+ type: "object",
+ properties: {
+ query: {
+ type: "string",
+ description:
+ "The user's question in natural language. Include helpful context like SDK/language (e.g., Python v3, JS v4), self-hosted vs cloud, and short error messages (trim long stack traces). Keep under ~600 characters.",
+ },
+ },
+ required: ["query"],
+ additionalProperties: false,
+ $schema: "http://json-schema.org/draft-07/schema#",
+ },
+ description:
+ "Semantic search (RAG) over the Langfuse documentation. Use this whenever the user asks a broader question that cannot be answered by a specific single page. Returns a concise answer synthesized from relevant docs. The raw provider response is included in _meta. Prefer this before guessing. If a specific page is needed call getLangfuseDocsPage first.",
+ },
+ {
+ type: "function",
+ name: "getLangfuseDocsPage",
+ inputSchema: {
+ type: "object",
+ properties: {
+ pathOrUrl: {
+ type: "string",
+ description:
+ 'Docs path starting with "/" (e.g., /docs/observability/overview) or a full URL on https://langfuse.com. Do not include anchors (#...) or queries (?foo=bar) β€” they will be ignored.',
+ },
+ },
+ required: ["pathOrUrl"],
+ additionalProperties: false,
+ $schema: "http://json-schema.org/draft-07/schema#",
+ },
+ description:
+ "Fetch the raw Markdown for a single Langfuse docs page. Accepts a docs path (e.g., /docs/observability/overview) or a full https://langfuse.com URL. Returns the exact Markdown (may include front matter). Use when you need a specific page content (Integration, Features, API, etc.) or code samples. Prefer searchLangfuseDocs for broader questions where there is not one specific page about it. Changelog pages (/changelog/...) are historical release notes: use them only to confirm that a feature exists or when it shipped, not as an implementation reference. Their code samples reflect the SDK/API at release time and may be outdated, so for implementation follow the current docs and the API/SDK reference instead.",
+ },
+ {
+ type: "function",
+ name: "getLangfuseOverview",
+ inputSchema: {
+ type: "object",
+ properties: {},
+ additionalProperties: false,
+ $schema: "http://json-schema.org/draft-07/schema#",
+ },
+ description:
+ "Get a high-level, machine-readable index by downloading https://langfuse.com/llms.txt. Use this at the start of a session when needed to discover key docs endpoints or to seed follow-up calls to searchLangfuseDocs or getLangfuseDocsPage. Returns the plain text contents of llms.txt. Avoid repeated calls within the same session.",
+ },
+ ],
+});
+
+const payload13 = JSON.stringify([
+ {
+ role: "assistant",
+ parts: [
+ {
+ type: "reasoning",
+ content:
+ "**Crafting user score attachment instructions**\n\nI've gathered enough info to help the user with attaching a user score to a trace. They want minimal examples in TypeScript, Python, and a Browser client. I think providing a basic snippet that adds a boolean score named \"user-feedback\" to the traceId would be effective. It's also essential to note that the data type BOOLEAN requires values of 0 or 1. I’ll include concise runnable examples and useful links about scores via API/SDK and user feedback. We should also consider concluding with a follow-up question.",
+ },
+ {
+ type: "reasoning",
+ content:
+ "**Creating minimal code examples**\n\nI want to create concise code samples while mentioning that you need a traceId, and optionally an observationId. It's important to highlight that you can send user feedback even before the trace exists, which is quite interesting! This way, the user gets a clear understanding while also seeing how to implement everything in a few lines of code. Let's keep this straightforward and focus on clarity in the examples!",
+ },
+ {
+ type: "text",
+ content:
+ 'You can add a user score to a trace via the SDKs or API by creating a Score linked to the traceId (optionally to a specific observationId). For thumbs up/down, use dataType "BOOLEAN" with value 1 or 0.\n\nJS/TS (server)\n- Docs: https://langfuse.com/docs/evaluation/evaluation-methods/scores-via-sdk\n- Example:\n - npm i @langfuse/client\n - \n import { LangfuseClient } from "@langfuse/client";\n const langfuse = new LangfuseClient();\n await langfuse.score.create({\n traceId: "<trace_id>",\n name: "user-feedback",\n value: 1, // 1 = up, 0 = down\n dataType: "BOOLEAN", // required for boolean\n comment: "Helpful answer", // optional\n });\n await langfuse.flush();\n\nBrowser (collect in-app feedback)\n- Docs: https://langfuse.com/docs/evaluation/evaluation-methods/scores-via-sdk\n- Example:\n - npm i @langfuse/browser\n - \n import { LangfuseBrowserClient } from "@langfuse/browser";\n const lf = new LangfuseBrowserClient({ publicKey: "<public_key>", baseUrl: "https://cloud.langfuse.com" });\n await lf.score({\n traceId: "<trace_id>",\n // observationId: "<generation_or_span_id>", // optional if you want to score a specific step\n id: `user-feedback-<trace_id>`, // optional idempotency key\n name: "user-feedback",\n value: 1,\n dataType: "BOOLEAN",\n comment: "Great result!",\n });\n\nPython\n- Docs: https://langfuse.com/docs/evaluation/evaluation-methods/scores-via-sdk\n- Example:\n - \n from langfuse import get_client\n lf = get_client()\n lf.create_score(\n name="user-feedback",\n value=1.0, # float\n data_type="BOOLEAN", # required for boolean\n trace_id="<trace_id>",\n # observation_id="<observation_id>", # optional\n comment="User liked it", # optional\n )\n\nNotes\n- You only need traceId to attach to a trace; observationId is optional.\n- You can send the score before the trace exists; it links once the trace is created.\n- Score types supported: NUMERIC, CATEGORICAL, BOOLEAN (0/1), TEXT.\n- More: Scores overview: https://langfuse.com/docs/evaluation/scores/overview\n\nWant an example of passing traceId/observationId from your backend to the frontend for a β€œthumbs up/down” widget?',
+ },
+ ],
+ finish_reason: "stop",
+ },
+]);
+
+export const vercelAiSdkDocsToolLoopFixture: TranscriptFixture = {
+ name: "Vercel AI SDK docs search with replayed tool history",
+ scope: "trace",
+ description:
+ "Two model steps with a documentation tool call and replayed history. The export also includes a GENERATION-typed application wrapper; its type and all payloads are preserved rather than adjusted to match the design example.",
+ observations: [
+ {
+ trace_id: traceId,
+ project_id: projectId,
+ id: "6aa131f38d871f39",
+ parent_observation_id: "e8cd3b597bd6f6fe",
+ type: "SPAN",
+ name: "handle-chatbot-message",
+ start_time: "2026-09-09T08:26:57.595Z",
+ end_time: "2026-09-09T08:27:19.094Z",
+ provided_model_name: "",
+ metadata: {
+ "scope.version": "5.9.1",
+ "scope.name": "langfuse-sdk",
+ "resourceAttributes.telemetry.sdk.version": "2.0.1",
+ "resourceAttributes.telemetry.sdk.name": "opentelemetry",
+ "resourceAttributes.telemetry.sdk.language": "nodejs",
+ "resourceAttributes.service.name": "unknown_service:node",
+ },
+ input: payload1,
+ output: payload2,
+ },
+ {
+ trace_id: traceId,
+ project_id: projectId,
+ id: "5b9a45797a30e758",
+ parent_observation_id: "6aa131f38d871f39",
+ type: "SPAN",
+ name: "get-langfuse-prompt",
+ start_time: "2026-09-09T08:26:57.595Z",
+ end_time: "2026-09-09T08:26:57.693Z",
+ provided_model_name: "",
+ metadata: {
+ "scope.version": "5.9.1",
+ "scope.name": "langfuse-sdk",
+ "resourceAttributes.telemetry.sdk.version": "2.0.1",
+ "resourceAttributes.telemetry.sdk.name": "opentelemetry",
+ "resourceAttributes.telemetry.sdk.language": "nodejs",
+ "resourceAttributes.service.name": "unknown_service:node",
+ },
+ input: payload3,
+ output: payload4,
+ },
+ {
+ trace_id: traceId,
+ project_id: projectId,
+ id: "2f957a45eec7d164",
+ parent_observation_id: "6aa131f38d871f39",
+ type: "SPAN",
+ name: "create-mcp-client",
+ start_time: "2026-09-09T08:26:57.696Z",
+ end_time: "2026-09-09T08:26:57.792Z",
+ provided_model_name: "",
+ metadata: {
+ "scope.version": "5.9.1",
+ "scope.name": "langfuse-sdk",
+ "resourceAttributes.telemetry.sdk.version": "2.0.1",
+ "resourceAttributes.telemetry.sdk.name": "opentelemetry",
+ "resourceAttributes.telemetry.sdk.language": "nodejs",
+ "resourceAttributes.service.name": "unknown_service:node",
+ },
+ input: payload5,
+ output: payload5,
+ },
+ {
+ trace_id: traceId,
+ project_id: projectId,
+ id: "dfae2f190648ad80",
+ parent_observation_id: "6aa131f38d871f39",
+ type: "AGENT",
+ name: "invoke_agent gpt-5",
+ start_time: "2026-09-09T08:26:57.832Z",
+ end_time: "2026-09-09T08:27:19.093Z",
+ provided_model_name: "",
+ metadata: {
+ "scope.name": "gen_ai",
+ "resourceAttributes.telemetry.sdk.version": "2.0.1",
+ "resourceAttributes.telemetry.sdk.name": "opentelemetry",
+ "resourceAttributes.telemetry.sdk.language": "nodejs",
+ "resourceAttributes.service.name": "unknown_service:node",
+ "attributes.gen_ai.usage.cache_creation.input_tokens": "0",
+ "attributes.gen_ai.usage.cache_read.input_tokens": "1024",
+ "attributes.gen_ai.usage.output_tokens": "984",
+ "attributes.gen_ai.usage.input_tokens": "22192",
+ "attributes.gen_ai.response.finish_reasons": '["stop"]',
+ "attributes.langfuse.trace.tags": '["qa-chatbot"]',
+ "attributes.langfuse.trace.name": "QA-Chatbot",
+ "attributes.session.id": "chat_f929c6af-9842-4c0c-ba24-cd00cda1cdc3",
+ "attributes.user.id": "u-2cwgMvI2GG_m-MduRMqsB",
+ "attributes.gen_ai.agent.name": "qa-chatbot",
+ "attributes.gen_ai.request.model": "gpt-5",
+ "attributes.gen_ai.provider.name": "openai",
+ "attributes.gen_ai.operation.name": "invoke_agent",
+ },
+ input: payload6,
+ output: payload7,
+ },
+ {
+ trace_id: traceId,
+ project_id: projectId,
+ id: "1bbf3edb8ef95ab6",
+ parent_observation_id: "b8553aadce2f531c",
+ type: "GENERATION",
+ name: "chat gpt-5",
+ start_time: "2026-09-09T08:26:57.837Z",
+ end_time: "2026-09-09T08:27:02.699Z",
+ provided_model_name: "gpt-5",
+ metadata: {
+ "scope.name": "gen_ai",
+ "resourceAttributes.telemetry.sdk.version": "2.0.1",
+ "resourceAttributes.telemetry.sdk.name": "opentelemetry",
+ "resourceAttributes.telemetry.sdk.language": "nodejs",
+ "resourceAttributes.service.name": "unknown_service:node",
+ "attributes.gen_ai.usage.cache_creation.input_tokens": "0",
+ "attributes.gen_ai.usage.cache_read.input_tokens": "0",
+ "attributes.gen_ai.usage.output_tokens": "159",
+ "attributes.gen_ai.usage.input_tokens": "1010",
+ "attributes.gen_ai.response.id":
+ "resp_0acb410cfc4131e7006aa118521b2087d29f62fea08c852c57",
+ "attributes.gen_ai.response.finish_reasons": '["tool-calls"]',
+ "attributes.gen_ai.client.operation.time_per_output_chunk":
+ "0.017943698769230746",
+ "attributes.gen_ai.client.operation.time_to_first_chunk":
+ "2.2376144869999846",
+ "attributes.gen_ai.client.operation.duration": "4.861078823999996",
+ "attributes.langfuse.trace.tags": '["qa-chatbot"]',
+ "attributes.langfuse.trace.name": "QA-Chatbot",
+ "attributes.session.id": "chat_f929c6af-9842-4c0c-ba24-cd00cda1cdc3",
+ "attributes.user.id": "u-2cwgMvI2GG_m-MduRMqsB",
+ "attributes.gen_ai.request.model": "gpt-5",
+ "attributes.gen_ai.provider.name": "openai",
+ "attributes.gen_ai.operation.name": "chat",
+ "attributes.langfuse.observation.prompt.version": "26",
+ "attributes.langfuse.observation.prompt.name":
+ "langfuse-docs-assistant-chat",
+ },
+ input: payload8,
+ output: payload9,
+ },
+ {
+ trace_id: traceId,
+ project_id: projectId,
+ id: "b8553aadce2f531c",
+ parent_observation_id: "dfae2f190648ad80",
+ type: "SPAN",
+ name: "step 1",
+ start_time: "2026-09-09T08:26:57.837Z",
+ end_time: "2026-09-09T08:27:08.262Z",
+ provided_model_name: "",
+ metadata: {
+ "scope.name": "gen_ai",
+ "resourceAttributes.telemetry.sdk.version": "2.0.1",
+ "resourceAttributes.telemetry.sdk.name": "opentelemetry",
+ "resourceAttributes.telemetry.sdk.language": "nodejs",
+ "resourceAttributes.service.name": "unknown_service:node",
+ "attributes.langfuse.trace.tags": '["qa-chatbot"]',
+ "attributes.langfuse.trace.name": "QA-Chatbot",
+ "attributes.session.id": "chat_f929c6af-9842-4c0c-ba24-cd00cda1cdc3",
+ "attributes.user.id": "u-2cwgMvI2GG_m-MduRMqsB",
+ "attributes.gen_ai.operation.name": "agent_step",
+ },
+ input: payload5,
+ output: payload5,
+ },
+ {
+ trace_id: traceId,
+ project_id: projectId,
+ id: "3dbf50cbf2036387",
+ parent_observation_id: "b8553aadce2f531c",
+ type: "TOOL",
+ name: "searchLangfuseDocs",
+ start_time: "2026-09-09T08:27:02.700Z",
+ end_time: "2026-09-09T08:27:08.255Z",
+ provided_model_name: "",
+ metadata: {
+ "scope.name": "gen_ai",
+ "resourceAttributes.telemetry.sdk.version": "2.0.1",
+ "resourceAttributes.telemetry.sdk.name": "opentelemetry",
+ "resourceAttributes.telemetry.sdk.language": "nodejs",
+ "resourceAttributes.service.name": "unknown_service:node",
+ "attributes.gen_ai.execute_tool.duration": "5.55423780599999",
+ "attributes.langfuse.trace.tags": '["qa-chatbot"]',
+ "attributes.langfuse.trace.name": "QA-Chatbot",
+ "attributes.session.id": "chat_f929c6af-9842-4c0c-ba24-cd00cda1cdc3",
+ "attributes.user.id": "u-2cwgMvI2GG_m-MduRMqsB",
+ "attributes.gen_ai.tool.type": "function",
+ "attributes.gen_ai.tool.call.id": "call_OQNWWW4PjixkLIhkuHJRwiFV",
+ "attributes.gen_ai.tool.name": "searchLangfuseDocs",
+ "attributes.gen_ai.operation.name": "execute_tool",
+ },
+ input: payload10,
+ output: payload11,
+ },
+ {
+ trace_id: traceId,
+ project_id: projectId,
+ id: "91eef3e3dede80b4",
+ parent_observation_id: "dfae2f190648ad80",
+ type: "SPAN",
+ name: "step 2",
+ start_time: "2026-09-09T08:27:08.266Z",
+ end_time: "2026-09-09T08:27:19.091Z",
+ provided_model_name: "",
+ metadata: {
+ "scope.name": "gen_ai",
+ "resourceAttributes.telemetry.sdk.version": "2.0.1",
+ "resourceAttributes.telemetry.sdk.name": "opentelemetry",
+ "resourceAttributes.telemetry.sdk.language": "nodejs",
+ "resourceAttributes.service.name": "unknown_service:node",
+ "attributes.langfuse.trace.tags": '["qa-chatbot"]',
+ "attributes.langfuse.trace.name": "QA-Chatbot",
+ "attributes.session.id": "chat_f929c6af-9842-4c0c-ba24-cd00cda1cdc3",
+ "attributes.user.id": "u-2cwgMvI2GG_m-MduRMqsB",
+ "attributes.gen_ai.operation.name": "agent_step",
+ },
+ input: payload5,
+ output: payload5,
+ },
+ {
+ trace_id: traceId,
+ project_id: projectId,
+ id: "0c13b686b4cf6ccc",
+ parent_observation_id: "91eef3e3dede80b4",
+ type: "GENERATION",
+ name: "chat gpt-5",
+ start_time: "2026-09-09T08:27:08.267Z",
+ end_time: "2026-09-09T08:27:19.089Z",
+ provided_model_name: "gpt-5",
+ metadata: {
+ "scope.name": "gen_ai",
+ "resourceAttributes.telemetry.sdk.version": "2.0.1",
+ "resourceAttributes.telemetry.sdk.name": "opentelemetry",
+ "resourceAttributes.telemetry.sdk.language": "nodejs",
+ "resourceAttributes.service.name": "unknown_service:node",
+ "attributes.gen_ai.usage.cache_creation.input_tokens": "0",
+ "attributes.gen_ai.usage.cache_read.input_tokens": "1024",
+ "attributes.gen_ai.usage.output_tokens": "825",
+ "attributes.gen_ai.usage.input_tokens": "21182",
+ "attributes.gen_ai.response.id":
+ "resp_0acb410cfc4131e7006aa1185c67fc87d2b3eab011f085698a",
+ "attributes.gen_ai.response.finish_reasons": '["stop"]',
+ "attributes.gen_ai.client.operation.time_per_output_chunk":
+ "0.007957471177083335",
+ "attributes.gen_ai.client.operation.time_to_first_chunk":
+ "4.444447944999992",
+ "attributes.gen_ai.client.operation.duration": "10.822020600999997",
+ "attributes.langfuse.trace.tags": '["qa-chatbot"]',
+ "attributes.langfuse.trace.name": "QA-Chatbot",
+ "attributes.session.id": "chat_f929c6af-9842-4c0c-ba24-cd00cda1cdc3",
+ "attributes.user.id": "u-2cwgMvI2GG_m-MduRMqsB",
+ "attributes.gen_ai.request.model": "gpt-5",
+ "attributes.gen_ai.provider.name": "openai",
+ "attributes.gen_ai.operation.name": "chat",
+ "attributes.langfuse.observation.prompt.version": "26",
+ "attributes.langfuse.observation.prompt.name":
+ "langfuse-docs-assistant-chat",
+ },
+ input: payload12,
+ output: payload13,
+ },
+ ],
+ expected: undefined,
+};
diff --git a/packages/shared/src/server/transcript/index.ts b/packages/shared/src/server/transcript/index.ts
new file mode 100644
index 0000000000..3c1379722c
--- /dev/null
+++ b/packages/shared/src/server/transcript/index.ts
@@ -0,0 +1,73 @@
+import { partition } from "lodash";
+import type { Observation } from "../../domain";
+import { normalizeIO } from "../normalized-io";
+import type { Transcript } from "./types";
+import {
+ append,
+ findThread,
+ messageKey,
+ type ThreadState,
+ type TranscriptObservation,
+} from "./threads";
+import { createToolCallRegistry } from "./tool-calls";
+export type * from "./types";
+
+/** Check if an observation is relevant for the transcript. */
+const isRelevantObservation = (
+ observation: Observation,
+): observation is TranscriptObservation =>
+ (observation.type === "GENERATION" || observation.type === "TOOL") &&
+ observation.traceId !== null;
+
+/** Normalize original I/O without interpreting provider-specific envelopes. */
+function normalize(observation: TranscriptObservation) {
+ const isTool = observation.type === "TOOL";
+ const { messages } = normalizeIO({
+ kind: "io",
+ io: {
+ // Tool inputs are supplied by the generation's tool-call message.
+ input: isTool ? undefined : observation.input,
+ output: observation.output,
+ metadata: observation.metadata,
+ },
+ });
+ return partition(
+ messages.map((message) => ({ message, key: messageKey(message) })),
+ ({ message }) => message.source === "input",
+ );
+}
+
+/** Normalize observations, reconcile history, and retain first-seen provenance. */
+export function getTranscript(observations: Observation[]): Transcript | null {
+ const ordered = observations
+ .filter(isRelevantObservation)
+ .sort((a, b) => a.startTime.getTime() - b.startTime.getTime());
+
+ const states: ThreadState[] = [];
+ const toolCalls = createToolCallRegistry();
+
+ for (const observation of ordered) {
+ const [input, output] = normalize(observation);
+ if (observation.type === "TOOL") {
+ toolCalls.attachToolOutput(observation, output);
+ continue;
+ }
+ if (input.length === 0 && output.length === 0) continue;
+
+ // Registered responses do not participate in thread selection.
+ let state = findThread(states, input);
+ const isNewThread = !state;
+ if (!state) {
+ // Open new thread
+ state = {
+ thread: { messages: [], observations: [] },
+ messages: [],
+ shownCounts: new Map(),
+ };
+ states.push(state);
+ }
+ append(state, observation, input, output, isNewThread, toolCalls);
+ }
+
+ return states.length ? { threads: states.map(({ thread }) => thread) } : null;
+}
diff --git a/packages/shared/src/server/transcript/threads.ts b/packages/shared/src/server/transcript/threads.ts
new file mode 100644
index 0000000000..3f3be5d857
--- /dev/null
+++ b/packages/shared/src/server/transcript/threads.ts
@@ -0,0 +1,108 @@
+import type { Observation } from "../../domain";
+import type { NormalizedMessage } from "../../utils/normalized-io";
+import type { Thread, ThreadMessage } from "./types";
+import type { createToolCallRegistry } from "./tool-calls";
+
+export type TranscriptObservation = Observation & { traceId: string };
+
+export type KeyedMessage = {
+ message: NormalizedMessage;
+ key: string;
+};
+export type ThreadState = {
+ thread: Thread;
+ messages: KeyedMessage[];
+ shownCounts: Map<string, number>;
+};
+
+const isRecord = (value: unknown): value is Record<string, unknown> =>
+ value !== null && typeof value === "object" && !Array.isArray(value);
+
+/** Object property order and observation provenance do not define identity. */
+export function messageKey(message: NormalizedMessage): string {
+ return JSON.stringify(
+ [message.role, message.parts],
+ (_key, value: unknown) =>
+ isRecord(value)
+ ? Object.fromEntries(
+ Object.keys(value)
+ .sort()
+ .map((key) => [key, value[key]]),
+ )
+ : value,
+ );
+}
+
+/** Find a thread that matches the input messages. */
+export function findThread(states: ThreadState[], input: KeyedMessage[]) {
+ const inputKeys = new Set(input.map(({ key }) => key));
+ return states.findLast(
+ ({ messages }) =>
+ messages.some(({ message }) => message.role !== "system") &&
+ messages.every(
+ ({ message, key }) => message.role === "system" || inputKeys.has(key),
+ ),
+ );
+}
+
+/** Append messages to a thread. */
+export function append(
+ state: ThreadState,
+ observation: TranscriptObservation,
+ input: KeyedMessage[],
+ output: KeyedMessage[],
+ isNewThread: boolean,
+ toolCalls: ReturnType<typeof createToolCallRegistry>,
+) {
+ const { thread, messages, shownCounts } = state;
+ const inputCounts = new Map<string, number>();
+ for (const { message, key: originalKey } of [...input, ...output]) {
+ const isOutput = message.source === "output";
+ const emitted: ThreadMessage = {
+ ...message,
+ parts: [],
+ observationId: observation.id,
+ traceId: observation.traceId,
+ };
+ // Anchor output calls before attaching any results carried by the same message.
+ if (isOutput) thread.messages.push(emitted);
+ for (const part of message.parts) {
+ if (!toolCalls.consumePart(observation, part, thread, emitted))
+ emitted.parts.push(part);
+ }
+ if (!emitted.parts.length) {
+ if (isOutput) thread.messages.splice(thread.messages.indexOf(emitted), 1);
+ continue;
+ }
+ const key =
+ emitted.parts.length === message.parts.length
+ ? originalKey
+ : messageKey(emitted);
+ if (!isOutput) {
+ const occurrence = (inputCounts.get(key) ?? 0) + 1;
+ inputCounts.set(key, occurrence);
+ if (!isNewThread && occurrence <= (shownCounts.get(key) ?? 0)) continue;
+ thread.messages.push(emitted);
+ }
+ messages.push({ message: emitted, key });
+ shownCounts.set(key, (shownCounts.get(key) ?? 0) + 1);
+ addContributor(thread, observation);
+ }
+}
+
+export function addContributor(
+ thread: Thread,
+ observation: TranscriptObservation,
+) {
+ if (
+ !thread.observations.some(
+ ({ id, traceId }) =>
+ id === observation.id && traceId === observation.traceId,
+ )
+ ) {
+ thread.observations.push({
+ id: observation.id,
+ traceId: observation.traceId,
+ });
+ }
+}
diff --git a/packages/shared/src/server/transcript/tool-calls.ts b/packages/shared/src/server/transcript/tool-calls.ts
new file mode 100644
index 0000000000..47142c46d6
--- /dev/null
+++ b/packages/shared/src/server/transcript/tool-calls.ts
@@ -0,0 +1,112 @@
+import type {
+ NormalizedMessage,
+ ToolCallPart,
+} from "../../utils/normalized-io";
+import type { Thread, ThreadMessage } from "./types";
+import {
+ addContributor,
+ type KeyedMessage,
+ type TranscriptObservation,
+} from "./threads";
+
+type Call = {
+ thread: Thread;
+ message: ThreadMessage;
+ response?: ThreadMessage;
+ fromTool?: boolean;
+};
+const key = (traceId: string, id: string) => JSON.stringify([traceId, id]);
+
+export function createToolCallRegistry() {
+ const calls = new Map<string, Call>();
+ const pending = new Map<string, { calls: Call[]; next: number }>();
+ const responseTails = new WeakMap<ThreadMessage, ThreadMessage>();
+
+ function register(
+ observation: TranscriptObservation,
+ part: ToolCallPart,
+ thread: Thread,
+ message: ThreadMessage,
+ ) {
+ const id = part.toolCallId
+ ? key(observation.traceId, part.toolCallId)
+ : undefined;
+ if (id && calls.has(id)) return;
+ const call: Call = { thread, message };
+ if (id) calls.set(id, call);
+ const name = key(observation.traceId, part.toolName);
+ if (!pending.has(name)) pending.set(name, { calls: [], next: 0 });
+ pending.get(name)!.calls.push(call);
+ }
+
+ function setResponse(
+ call: Call,
+ observation: TranscriptObservation,
+ parts: NormalizedMessage["parts"],
+ ) {
+ const fromTool = observation.type === "TOOL";
+ if (call.response && (!fromTool || call.fromTool)) return;
+ const response: ThreadMessage = {
+ role: "tool",
+ source: "output",
+ parts,
+ observationId: observation.id,
+ traceId: observation.traceId,
+ };
+ if (call.response) Object.assign(call.response, response);
+ else {
+ const anchor = responseTails.get(call.message) ?? call.message;
+ call.thread.messages.splice(
+ call.thread.messages.indexOf(anchor) + 1,
+ 0,
+ response,
+ );
+ responseTails.set(call.message, response);
+ call.response = response;
+ }
+ call.fromTool = fromTool;
+ addContributor(call.thread, observation);
+ }
+
+ /** Register output calls or consume a result while its message is appended. */
+ function consumePart(
+ observation: TranscriptObservation,
+ part: NormalizedMessage["parts"][number],
+ thread: Thread,
+ message: ThreadMessage,
+ ): boolean {
+ if (part.type === "tool-call" && message.source === "output")
+ register(observation, part, thread, message);
+ if (part.type !== "tool-result" || !part.toolCallId) return false;
+ const call = calls.get(key(observation.traceId, part.toolCallId));
+ if (!call) return false;
+ setResponse(call, observation, [part]);
+ return true;
+ }
+
+ function attachToolOutput(
+ observation: TranscriptObservation,
+ output: KeyedMessage[],
+ ) {
+ const parts = output.flatMap(({ message }) => message.parts);
+ if (!parts.length) return;
+ let hasId = false;
+ for (const part of parts) {
+ if (part.type !== "tool-result" || !part.toolCallId) continue;
+ hasId = true;
+ const call = calls.get(key(observation.traceId, part.toolCallId));
+ if (call) setResponse(call, observation, [part]);
+ }
+ if (hasId || !observation.name) return;
+ const queue = pending.get(key(observation.traceId, observation.name));
+ if (!queue) return;
+ while (queue.calls[queue.next]?.fromTool) queue.next++;
+ const call = queue.calls[queue.next];
+ if (call) {
+ queue.next++;
+ setResponse(call, observation, parts);
+ }
+ }
+
+ return { consumePart, attachToolOutput };
+}
diff --git a/packages/shared/src/server/transcript/types.ts b/packages/shared/src/server/transcript/types.ts
new file mode 100644
index 0000000000..ea1cc59d14
--- /dev/null
+++ b/packages/shared/src/server/transcript/types.ts
@@ -0,0 +1,19 @@
+import type { NormalizedMessage } from "../../utils/normalized-io/types";
+
+export type ThreadMessage = NormalizedMessage & {
+ /** Observation that first emitted this message. */
+ observationId: string;
+ /** Trace the emitting observation belongs to. Helpful when the transcript
+ * spans several traces, for example one session. */
+ traceId: string;
+};
+
+export type Thread = {
+ messages: ThreadMessage[];
+ /** Observations that contributed to this thread, in contribution order. */
+ observations: { id: string; traceId: string }[];
+};
+
+export type Transcript = {
+ threads: Thread[];
+};