Skip to content

🪢 Open source agent evals & observability: Trace, evaluate, and improve LLM applications with one open platform.

feat(trace-batching): enhance transcript assembly with timing metrics and token estimates (#17839)

marliessophie72dcdb363d21
diff --git a/packages/shared/src/server/transcript/transcript.ts b/packages/shared/src/server/transcript/transcript.ts
index 2e4a876f0d..55afea5b76 100644
--- a/packages/shared/src/server/transcript/transcript.ts
+++ b/packages/shared/src/server/transcript/transcript.ts
@@ -41,16 +41,25 @@ function normalize(observation: TranscriptObservation) {
* Assemble threads from observations: normalize their I/O, reconcile replayed
* history, and retain first-seen provenance. The caller supplies the
* observations in transcript order, see `orderObservations`; they are consumed
- * as given.
+ * as given. Optional timings separate normalization (including initial message
+ * keys) from remaining assembly work, excluding caller-owned observation ordering.
*/
export function assembleTranscript(
orderedObservations: Observation[],
+ onTimings?: (timings: {
+ normalizationMs: number;
+ matchingMs: number;
+ }) => void,
): Transcript | null {
+ const startedAt = onTimings ? performance.now() : 0;
+ let normalizationMs = 0;
const states: ThreadState[] = [];
const toolCalls = createToolCallRegistry();
for (const observation of orderedObservations.filter(isRelevantObservation)) {
+ const normalizationStart = onTimings ? performance.now() : 0;
const [input, output] = normalize(observation);
+ if (onTimings) normalizationMs += performance.now() - normalizationStart;
if (observation.type === "TOOL") {
toolCalls.attachToolOutput(observation, output);
continue;
@@ -72,7 +81,12 @@ export function assembleTranscript(
append(state, observation, input, output, isNewThread, toolCalls);
}
- return states.length
+ const transcript = states.length
? { threads: states.map(({ thread }) => splitTurn(thread)) }
: null;
+ onTimings?.({
+ normalizationMs,
+ matchingMs: performance.now() - startedAt - normalizationMs,
+ });
+ return transcript;
}
diff --git a/worker/AGENTS.md b/worker/AGENTS.md
index 7d374e2dce..121f68cdc9 100644
--- a/worker/AGENTS.md
+++ b/worker/AGENTS.md
@@ -30,7 +30,10 @@
and Redis snapshots independently of dispatch/consumption when either role is
enabled. Global snapshot gauges must not be summed across worker reporters.
`src/features/traceBatching/traceBatchTranscript.ts` measures per-trace assembly
- and token estimates. Allow one pending tokenization promise per batch while
+ phases, thread count, current-turn/history token estimates and their sum.
+ Token partitions run sequentially; tool-response size uses comparable character
+ counts over message parts, without another tokenizer pass.
+ Allow one pending tokenization promise per batch while
buffering the next trace, and drain it even on read failure. Never flush a failed
stream's partial final trace; completion covers the query window, not future arrivals.
- Evaluation terminal-outcome classification: `src/features/evaluation/evalExecutionMetrics.ts`. Keep it aligned with shared code evaluator dispatcher error codes and user-visible error mapping.
diff --git a/worker/src/features/traceBatching/README.md b/worker/src/features/traceBatching/README.md
index 92a865880a..a4d5655a1a 100644
--- a/worker/src/features/traceBatching/README.md
+++ b/worker/src/features/traceBatching/README.md
@@ -122,7 +122,13 @@ These distributions use the `langfuse.trace_batch` prefix:
| Metric | Sample |
| --- | --- |
| `transcript_assembly_duration_ms` | One trace's ordering and assembly time, excluding I/O conversion, stream waits and tokenization; `has_transcript:true\|false`. |
-| `transcript_tokens` | Token estimate of the complete transcript JSON (history, current turn and provenance); zero when no transcript can be assembled. |
+| `transcript_message_tokens` | Sum of current-turn and history token estimates; zero when empty, omitted if either estimate is unavailable. Replaces full-transcript JSON tokenization under a different name. |
+| `transcript_current_turn_tokens` | Token estimate of current-turn messages across all threads, using role and parts only; zero when empty. |
+| `transcript_history_tokens` | Token estimate of history messages across all threads, using the same role/parts representation; zero when empty. |
+| `transcript_content_characters` | Sum of JSON-serialized message-part lengths across history and current turn, in UTF-16 code units; excludes message wrappers and provenance. |
+| `transcript_tool_response_characters` | The subset of content characters belonging to tool-role messages or unmatched tool-result parts; zero when absent. |
+| `transcript_thread_count` | Number of assembled conversation threads, or zero for a null transcript. |
+| `transcript_assembly_phase_duration_ms` | Two non-overlapping samples per trace tagged `phase:normalization` or `phase:matching`. Normalization includes initial message-key construction and partitioning; matching covers remaining assembly work, including tool matching, deduplication, any rebuilt keys and finalization. Observation ordering is outside these phases but remains in total assembly time. |
Transcript token counts use the existing local worker-thread pool and bundled
tiktoken WASM, without a network or model API call. The `gpt-4o` configuration
@@ -130,19 +136,38 @@ selects `o200k_base`, also used by GPT-5 mini and nano
([OpenAI mapping](https://github.com/openai/tiktoken/blob/main/tiktoken/model.py)); metrics are tagged
`tokenizer:o200k_base`. These are serialized-payload estimates, not provider
billing counts or a model's full request framing.
+Current turn and history use identical `{ messages: [{ role, parts }] }` JSON
+framing, without observation provenance. Only nonempty partitions are tokenized,
+at most twice per trace. Their sum is `transcript_message_tokens`; it is not
+directly comparable to the retired `transcript_tokens` full-transcript JSON metric.
+Empty transcripts report zero. Thread counts are numeric samples, never metric tags.
+
+For rough tool-response size share, divide `transcript_tool_response_characters`
+by `transcript_content_characters` (when nonzero). Both sum serialized parts on
+the same basis, including part JSON syntax but excluding message wrappers and
+provenance. This is a character share, not a token share. For an overall traffic
+share, divide the sums rather than averaging per-trace percentages. These metrics
+are emitted before tokenization, including when it fails; no tool tokenizer runs.
Unknown estimates are omitted and counted in `token_estimation_unavailable`.
-Rejected estimates increment `token_estimation_failed` and omit the token sample;
-they do not retry the batch. Only the assembled transcript is tokenized.
+Rejected estimates increment `token_estimation_failed` and stop the remaining
+estimates for that trace; successful earlier samples are retained. Unknown
+estimates do not stop later estimates. The summed estimate is omitted unless
+both partitions are known. Neither failure retries the batch.
+Only current-turn and history message projections are tokenized.
The worker submits one transcript for tokenization while streaming the next
trace's observations. Before submitting another transcript it awaits the previous
-promise, keeping at most one pending token estimate per batch. Success and stream
+promise. Projections are tokenized sequentially, keeping at most one pending
+tokenizer request per batch. Success and stream
failure both drain accepted promises; a stream failure never assembles its
partial final trace. Assembly itself remains synchronous. `read_duration_ms`
includes all this processing.
Memory includes the current trace, the previous transcript being tokenized and
stream buffers, multiplied by active batch jobs. A single trace remains unbounded.
+The transcript remains referenced until both partitions finish. Each message
+belongs to one partition; there is no additional full-transcript or tool-response
+tokenization pass.
The default two-thread tokenizer pool is shared with ingestion; overlap hides
waiting time but does not remove CPU use or contention. Its existing 30-second
timeout rejects the promise without cancelling queued/running encoding, so the
@@ -164,27 +189,33 @@ Metric delivery and percentile configuration must be verified after deployment.
Each completed trace emits a `trace-batch-transcript` child span under the batch
processing span. Its attributes include `langfuse.project.id`, `langfuse.trace.id`,
and `langfuse.trace.url` (a peek link using the configured product base URL).
-Under `langfuse.trace_batch`, the span records `transcript_tokens`, `transcript_characters`,
+Under `langfuse.trace_batch`, the span records `transcript_message_tokens`, `transcript_characters`,
`transcript_assembly_duration_ms`, `observation_count` (rows before observation
deduplication), `has_transcript`, `tokenizer`, and `experiment_id`.
+It also records the token breakdown and content/tool-response character metrics,
+`transcript_thread_count`,
+and each phase as `transcript_assembly_<phase>_duration_ms`.
No transcript content is attached, and IDs are not distribution metric tags.
`transcript_characters` measures the complete transcript JSON's JavaScript string
length (UTF-16 code units, including JSON syntax), or zero for a null transcript.
+This legacy span-only field uses a different basis from the content character
+metrics; use `transcript_content_characters` as the tool-response denominator.
It is recorded before tokenization, so remains available if token estimation fails.
Only recording spans serialize this extra temporary copy; it is not retained
while tokenization runs or included in the assembly-duration measurement.
The span stays open until token estimation settles, so its duration includes
tokenization and pool waits. Use the assembly-duration attribute for assembly
-performance. Missing estimates have `token_estimation:unavailable|failed` and no
-token count; null transcripts have zero tokens. The child span does not become
+performance. Missing estimates have `token_estimation:unavailable|failed`; their
+counts are omitted while successful earlier estimates remain. Null transcripts
+have zero tokens. The child span does not become
active while the next trace streams, and failed streams do not emit a span for
their partial final trace.
In Datadog APM Trace Explorer, search for:
```text
-env:prod-eu service:worker-cpu resource_name:trace-batch-transcript @langfuse.trace_batch.transcript_tokens:>=100000
+env:prod-eu service:worker-cpu resource_name:trace-batch-transcript @langfuse.trace_batch.transcript_message_tokens:>=100000
```
Add token count as a numeric measure to sort largest first, display the project
diff --git a/worker/src/features/traceBatching/traceBatchTranscript.ts b/worker/src/features/traceBatching/traceBatchTranscript.ts
index 7f4ebd0196..a0ebfbaede 100644
--- a/worker/src/features/traceBatching/traceBatchTranscript.ts
+++ b/worker/src/features/traceBatching/traceBatchTranscript.ts
@@ -1,5 +1,5 @@
import { type Model, type Observation } from "@langfuse/shared";
-import { SpanStatusCode } from "@opentelemetry/api";
+import { SpanStatusCode, type Span } from "@opentelemetry/api";
import {
assembleTranscript,
buildTracesPath,
@@ -8,6 +8,7 @@ import {
orderObservations,
recordDistribution,
recordIncrement,
+ type Transcript,
} from "@langfuse/shared/src/server";
import { env } from "../../env";
import { tokenCountAsync } from "../tokenisation/async-usage";
@@ -31,6 +32,96 @@ const TOKENIZER_MODEL: Model = {
unit: "TOKENS",
};
+const TOKEN_METRICS = [
+ "transcript_message_tokens",
+ "transcript_current_turn_tokens",
+ "transcript_history_tokens",
+] as const;
+
+function recordTokens(
+ span: Span,
+ metric: (typeof TOKEN_METRICS)[number],
+ tokens: number,
+) {
+ recordDistribution(`langfuse.trace_batch.${metric}`, tokens, {
+ tokenizer: "o200k_base",
+ });
+ span.setAttribute(`langfuse.trace_batch.${metric}`, tokens);
+}
+
+function recordCharacterCounts(transcript: Transcript | null, span: Span) {
+ let total = 0;
+ let toolResponses = 0;
+ for (const thread of transcript?.threads ?? []) {
+ for (const messages of [
+ thread.conversationHistory,
+ thread.currentTurn.messages,
+ ]) {
+ for (const message of messages) {
+ for (const part of message.parts) {
+ // Use the same serialized-part basis for numerator and denominator;
+ // exclude message wrappers and observation provenance from both.
+ const characters = JSON.stringify(part).length;
+ total += characters;
+ if (message.role === "tool" || part.type === "tool-result") {
+ toolResponses += characters;
+ }
+ }
+ }
+ }
+ }
+ for (const [metric, characters] of [
+ ["transcript_content_characters", total],
+ ["transcript_tool_response_characters", toolResponses],
+ ] as const) {
+ recordDistribution(`langfuse.trace_batch.${metric}`, characters);
+ span.setAttribute(`langfuse.trace_batch.${metric}`, characters);
+ }
+}
+
+async function recordTokenEstimates(transcript: Transcript, span: Span) {
+ const currentTurn = transcript.threads.flatMap(
+ ({ currentTurn }) => currentTurn.messages,
+ );
+ const history = transcript.threads.flatMap(
+ ({ conversationHistory }) => conversationHistory,
+ );
+ let total = 0;
+ let unavailable = false;
+ // Count each message in just one partition. Submit sequentially to keep one
+ // pending tokenizer request per batch in the pool shared with ingestion.
+ for (const [metric, messages] of [
+ ["transcript_current_turn_tokens", currentTurn],
+ ["transcript_history_tokens", history],
+ ] as const) {
+ const tokens = messages.length
+ ? await tokenCountAsync({
+ model: TOKENIZER_MODEL,
+ // Both partitions exclude provenance and use identical JSON framing.
+ text: {
+ messages: messages.map(({ role, parts }) => ({ role, parts })),
+ },
+ })
+ : 0;
+ if (tokens === undefined) {
+ if (!unavailable) {
+ recordIncrement("langfuse.trace_batch.token_estimation_unavailable", 1);
+ span.setAttribute(
+ "langfuse.trace_batch.token_estimation",
+ "unavailable",
+ );
+ }
+ unavailable = true;
+ } else {
+ recordTokens(span, metric, tokens);
+ total += tokens;
+ }
+ }
+ if (!unavailable) {
+ recordTokens(span, "transcript_message_tokens", total);
+ }
+}
+
export function recordTraceBatchTranscript(
observations: Observation[],
): Promise<void> {
@@ -57,7 +148,13 @@ export function recordTraceBatchTranscript(
);
}
const startedAt = performance.now();
- const transcript = assembleTranscript(orderObservations(observations));
+ let phaseTimings = { normalizationMs: 0, matchingMs: 0 };
+ const transcript = assembleTranscript(
+ orderObservations(observations),
+ (timings) => {
+ phaseTimings = timings;
+ },
+ );
const assemblyDurationMs = performance.now() - startedAt;
recordDistribution(
"langfuse.trace_batch.transcript_assembly_duration_ms",
@@ -69,6 +166,30 @@ export function recordTraceBatchTranscript(
assemblyDurationMs,
"langfuse.trace_batch.has_transcript": transcript !== null,
});
+ for (const [phase, duration] of [
+ ["normalization", phaseTimings.normalizationMs],
+ ["matching", phaseTimings.matchingMs],
+ ] as const) {
+ recordDistribution(
+ "langfuse.trace_batch.transcript_assembly_phase_duration_ms",
+ duration,
+ { phase },
+ );
+ span.setAttribute(
+ `langfuse.trace_batch.transcript_assembly_${phase}_duration_ms`,
+ duration,
+ );
+ }
+ const threadCount = transcript?.threads.length ?? 0;
+ recordDistribution(
+ "langfuse.trace_batch.transcript_thread_count",
+ threadCount,
+ );
+ span.setAttribute(
+ "langfuse.trace_batch.transcript_thread_count",
+ threadCount,
+ );
+ recordCharacterCounts(transcript, span);
if (span.isRecording()) {
// JSON string length in UTF-16 code units; do not retain the serialized copy.
span.setAttribute(
@@ -78,33 +199,13 @@ export function recordTraceBatchTranscript(
}
if (transcript === null) {
- recordDistribution("langfuse.trace_batch.transcript_tokens", 0, {
- tokenizer: "o200k_base",
- });
- span.setAttribute("langfuse.trace_batch.transcript_tokens", 0);
+ for (const metric of TOKEN_METRICS) recordTokens(span, metric, 0);
span.end();
return Promise.resolve();
}
// Callbacks capture only the span, releasing observations while the next trace streams.
- return tokenCountAsync({ model: TOKENIZER_MODEL, text: transcript })
- .then((tokens) => {
- if (tokens === undefined) {
- recordIncrement(
- "langfuse.trace_batch.token_estimation_unavailable",
- 1,
- );
- span.setAttribute(
- "langfuse.trace_batch.token_estimation",
- "unavailable",
- );
- return;
- }
- recordDistribution("langfuse.trace_batch.transcript_tokens", tokens, {
- tokenizer: "o200k_base",
- });
- span.setAttribute("langfuse.trace_batch.transcript_tokens", tokens);
- })
+ return recordTokenEstimates(transcript, span)
.catch(() => {
// A missing experiment metric must not retry all reads in the batch.
recordIncrement("langfuse.trace_batch.token_estimation_failed", 1);
diff --git a/worker/src/queues/__tests__/traceBatchQueue.test.ts b/worker/src/queues/__tests__/traceBatchQueue.test.ts
index 2732097f8e..11ad2ba802 100644
--- a/worker/src/queues/__tests__/traceBatchQueue.test.ts
+++ b/worker/src/queues/__tests__/traceBatchQueue.test.ts
@@ -13,6 +13,8 @@ import { context, SpanStatusCode, trace } from "@opentelemetry/api";
import { NodeSDK, tracing } from "@opentelemetry/sdk-node";
import {
getCurrentSpan,
+ convertObservation,
+ createObservation,
getTraceBatchEventStream,
logger,
QueueJobs,
@@ -25,6 +27,7 @@ import {
import { env } from "../../env";
import { tokenCountAsync } from "../../features/tokenisation/async-usage";
import { tokenCount } from "../../features/tokenisation/usage";
+import { recordTraceBatchTranscript } from "../../features/traceBatching/traceBatchTranscript";
import {
recordTraceBatchActiveReads,
traceBatchQueueProcessor,
@@ -80,6 +83,219 @@ afterEach(() => {
});
describe("trace batch queue", () => {
+ it("measures history, current turn, threads and deduplicated tool responses", async () => {
+ const history = [
+ { role: "user", content: "earlier question" },
+ { role: "assistant", content: "earlier answer" },
+ ];
+ const question = { role: "user", content: "current question" };
+ const call = {
+ role: "assistant",
+ content: null,
+ tool_calls: [
+ {
+ id: "call",
+ type: "function",
+ function: { name: "lookup", arguments: "{}" },
+ },
+ ],
+ };
+ const observation = (
+ id: string,
+ input: unknown,
+ output: unknown,
+ type: "GENERATION" | "TOOL" = "GENERATION",
+ ) =>
+ convertObservation(
+ createObservation({
+ id,
+ trace_id: "trace",
+ project_id: "project",
+ type,
+ name: type === "TOOL" ? "lookup" : "generation",
+ start_time: `2026-01-01T12:00:0${id}.000Z`,
+ input: JSON.stringify(input),
+ output: JSON.stringify(output),
+ }),
+ );
+ await recordTraceBatchTranscript([
+ observation("1", [...history, question], call),
+ observation("2", {}, "tool payload", "TOOL"),
+ observation(
+ "3",
+ [
+ ...history,
+ question,
+ call,
+ { role: "tool", tool_call_id: "call", content: "tool payload" },
+ ],
+ "current answer",
+ ),
+ observation(
+ "4",
+ [{ role: "user", content: "independent question" }],
+ "independent answer",
+ ),
+ ]);
+ const estimates = vi
+ .mocked(tokenCountAsync)
+ .mock.calls.map(([params]) => params);
+ expect(estimates).toHaveLength(2);
+ const [current, historyOnly] = estimates.map(({ text }) =>
+ JSON.stringify(text),
+ );
+ expect(current.split("tool payload")).toHaveLength(2);
+ expect(current).not.toContain("earlier question");
+ expect(current).toContain("current question");
+ expect(current).toContain("independent question");
+ expect(historyOnly).toContain("earlier question");
+ expect(historyOnly).not.toContain("current question");
+ expect(recordDistribution).toHaveBeenCalledWith(
+ "langfuse.trace_batch.transcript_thread_count",
+ 2,
+ );
+ for (const [index, suffix] of [
+ "current_turn_tokens",
+ "history_tokens",
+ ].entries()) {
+ expect(recordDistribution).toHaveBeenCalledWith(
+ `langfuse.trace_batch.transcript_${suffix}`,
+ tokenCount(estimates[index]),
+ { tokenizer: "o200k_base" },
+ );
+ }
+ expect(recordDistribution).toHaveBeenCalledWith(
+ "langfuse.trace_batch.transcript_message_tokens",
+ tokenCount(estimates[0])! + tokenCount(estimates[1])!,
+ { tokenizer: "o200k_base" },
+ );
+ const span = exporter.getFinishedSpans()[0];
+ const toolCharacters = JSON.stringify({
+ type: "text",
+ text: "tool payload",
+ }).length;
+ expect(recordDistribution).toHaveBeenCalledWith(
+ "langfuse.trace_batch.transcript_tool_response_characters",
+ toolCharacters,
+ );
+ expect(
+ span.attributes["langfuse.trace_batch.transcript_content_characters"],
+ ).toBeGreaterThan(toolCharacters);
+ let phaseTotal = 0;
+ for (const phase of ["normalization", "matching"]) {
+ const duration = span.attributes[
+ `langfuse.trace_batch.transcript_assembly_${phase}_duration_ms`
+ ] as number;
+ expect(duration).toBeGreaterThanOrEqual(0);
+ phaseTotal += duration;
+ expect(recordDistribution).toHaveBeenCalledWith(
+ "langfuse.trace_batch.transcript_assembly_phase_duration_ms",
+ duration,
+ { phase },
+ );
+ }
+ expect(phaseTotal).toBeLessThanOrEqual(
+ span.attributes[
+ "langfuse.trace_batch.transcript_assembly_duration_ms"
+ ] as number,
+ );
+ });
+
+ it("counts unmatched tool results without counting adjacent user text", async () => {
+ await recordTraceBatchTranscript([
+ convertObservation(
+ createObservation({
+ type: "GENERATION",
+ trace_id: "trace",
+ input: JSON.stringify([
+ {
+ role: "user",
+ content: [
+ { type: "text", text: "Please explain this result" },
+ {
+ type: "tool_result",
+ tool_use_id: "unmatched",
+ content: "tool payload",
+ },
+ ],
+ },
+ ]),
+ output: "Answer",
+ }),
+ ),
+ ]);
+ expect(tokenCountAsync).toHaveBeenCalledTimes(1);
+ const toolCharacters = JSON.stringify({
+ type: "tool-result",
+ toolCallId: "unmatched",
+ output: "tool payload",
+ }).length;
+ const totalCharacters =
+ toolCharacters +
+ JSON.stringify({ type: "text", text: "Please explain this result" })
+ .length +
+ JSON.stringify({ type: "text", text: "Answer" }).length;
+ expect(recordDistribution).toHaveBeenCalledWith(
+ "langfuse.trace_batch.transcript_tool_response_characters",
+ toolCharacters,
+ );
+ expect(recordDistribution).toHaveBeenCalledWith(
+ "langfuse.trace_batch.transcript_content_characters",
+ totalCharacters,
+ );
+ expect(recordDistribution).toHaveBeenCalledWith(
+ "langfuse.trace_batch.transcript_history_tokens",
+ 0,
+ { tokenizer: "o200k_base" },
+ );
+ });
+
+ it.each(["unavailable", "failed"])(
+ "retains current-turn tokens and character counts but omits the sum when history is %s",
+ async (outcome) => {
+ vi.mocked(tokenCountAsync).mockResolvedValueOnce(100);
+ if (outcome === "unavailable")
+ vi.mocked(tokenCountAsync).mockResolvedValueOnce(undefined);
+ else
+ vi.mocked(tokenCountAsync).mockRejectedValueOnce(
+ new Error("tokenizer failed"),
+ );
+ await recordTraceBatchTranscript([
+ convertObservation(
+ createObservation({
+ type: "GENERATION",
+ trace_id: "trace",
+ input: JSON.stringify([
+ { role: "assistant", content: "history" },
+ { role: "user", content: "question" },
+ ]),
+ output: "answer",
+ }),
+ ),
+ ]);
+ const span = exporter.getFinishedSpans()[0];
+ expect(
+ span.attributes["langfuse.trace_batch.transcript_current_turn_tokens"],
+ ).toBe(100);
+ expect(span.attributes).not.toHaveProperty(
+ "langfuse.trace_batch.transcript_history_tokens",
+ );
+ expect(span.attributes).not.toHaveProperty(
+ "langfuse.trace_batch.transcript_message_tokens",
+ );
+ expect(
+ span.attributes["langfuse.trace_batch.transcript_content_characters"],
+ ).toBeGreaterThan(0);
+ expect(span.attributes["langfuse.trace_batch.token_estimation"]).toBe(
+ outcome,
+ );
+ expect(recordIncrement).toHaveBeenCalledWith(
+ `langfuse.trace_batch.token_estimation_${outcome}`,
+ 1,
+ );
+ },
+ );
+
it.each([false, true])(
"overlaps one tokenization with streaming, bounds pending work and drains it (stream fails: %s)",
async (streamFails) => {
@@ -169,7 +385,7 @@ describe("trace batch queue", () => {
expect(spans).toHaveLength(streamFails ? 2 : 3);
expect(spans[0].attributes).toMatchObject({
"langfuse.trace.id": "a",
- "langfuse.trace_batch.transcript_tokens": 100,
+ "langfuse.trace_batch.transcript_message_tokens": 100,
});
expect(spans[1].attributes).toMatchObject({
"langfuse.trace.id": "b",
@@ -177,7 +393,7 @@ describe("trace batch queue", () => {
});
expect(spans[1].status.code).toBe(SpanStatusCode.UNSET);
expect(spans[1].attributes).not.toHaveProperty(
- "langfuse.trace_batch.transcript_tokens",
+ "langfuse.trace_batch.transcript_message_tokens",
);
expect(
spans[1].attributes["langfuse.trace_batch.transcript_characters"],
@@ -275,15 +491,32 @@ describe("trace batch queue", () => {
expect(serializedTranscript.split(content)).toHaveLength(2);
}
expect(recordDistribution).toHaveBeenCalledWith(
- "langfuse.trace_batch.transcript_tokens",
+ "langfuse.trace_batch.transcript_message_tokens",
tokenCount(estimates[0][0]),
{ tokenizer: "o200k_base" },
);
expect(recordDistribution).toHaveBeenCalledWith(
- "langfuse.trace_batch.transcript_tokens",
+ "langfuse.trace_batch.transcript_message_tokens",
0,
{ tokenizer: "o200k_base" },
);
+ for (const suffix of ["current_turn", "history"]) {
+ expect(recordDistribution).toHaveBeenCalledWith(
+ `langfuse.trace_batch.transcript_${suffix}_tokens`,
+ 0,
+ { tokenizer: "o200k_base" },
+ );
+ }
+ for (const suffix of ["content", "tool_response"]) {
+ expect(recordDistribution).toHaveBeenCalledWith(
+ `langfuse.trace_batch.transcript_${suffix}_characters`,
+ 0,
+ );
+ }
+ expect(recordDistribution).toHaveBeenCalledWith(
+ "langfuse.trace_batch.transcript_thread_count",
+ 0,
+ );
for (const hasTranscript of ["true", "false"]) {
expect(recordDistribution).toHaveBeenCalledWith(
"langfuse.trace_batch.transcript_assembly_duration_ms",
@@ -301,10 +534,10 @@ describe("trace batch queue", () => {
"langfuse.trace.id": "shared-trace",
"langfuse.trace_batch.observation_count": 2,
"langfuse.trace_batch.has_transcript": index === 0,
- "langfuse.trace_batch.transcript_tokens":
+ "langfuse.trace_batch.transcript_message_tokens":
index === 0 ? tokenCount(estimates[0][0]) : 0,
"langfuse.trace_batch.transcript_characters":
- index === 0 ? serializedTranscript.length : 0,
+ index === 0 ? expect.any(Number) : 0,
"langfuse.trace_batch.transcript_assembly_duration_ms":
expect.any(Number),
});