🪢 Open source agent evals & observability: Trace, evaluate, and improve LLM applications with one open platform.
TypeScript analyticsautogenevaluationlangchainlarge-language-modelsllama-indexllmllm-evaluationllm-observabilityllmopsmonitoringobservabilityopen-sourceopenaiplaygroundprompt-engineeringprompt-managementself-hostedycombinator
chore(otel): replay raw documents through the production queue (#18217)
Valery Meleshkinb1fc56a8fe11
diff --git a/worker/AGENTS.md b/worker/AGENTS.mdindex c4b3db68ef..142aa72d34 100644--- a/worker/AGENTS.md+++ b/worker/AGENTS.md@@ -47,9 +47,10 @@in `src/initialize.ts`, the health probe call in `src/api/index.ts`. Native coderecords its own metrics and logs; see `../packages/native/AGENTS.md`.- Tests: `src/__tests__/*`, `src/queues/__tests__/*`-- Direct-event replay: `pnpm --filter worker run test:otel-replay` exercises the- production OTEL event phase with isolated ClickHouse tables. Setup and scope:- `src/features/otel-ingestion/README.md`.+- Direct-event replay: `pnpm --filter worker run test:otel-replay` feeds raw S3+ JSON through the production OTEL queue and JSON writer into isolated+ ClickHouse tables, including retained legacy writes in dual-write mode. Setup+ and scope: `src/features/otel-ingestion/README.md`.- Native codec checks: `pnpm --filter worker run test:native-codec` selects`nativeCodec` suites. Build the addon first; the command requires ClickHousewhenever the live parity suite is present.diff --git a/worker/src/__tests__/helpers/otelReplayHarness.ts b/worker/src/__tests__/helpers/otelReplayHarness.tsindex d8b1f03df7..873f334716 100644--- a/worker/src/__tests__/helpers/otelReplayHarness.ts+++ b/worker/src/__tests__/helpers/otelReplayHarness.ts@@ -1,91 +1,250 @@import { randomUUID } from "node:crypto";+import { expect, vi } from "vitest";+import type { Job } from "bullmq";+import {clickhouseClient,+ getS3EventStorageClient,+ IngestionQueue,+ QueueJobs,+ QueueName,+ StorageServiceFactory,type ClickhouseClientType,- type ResourceSpan,- OtelIngestionProcessor,+ type TQueueJobTypes,} from "@langfuse/shared/src/server";import { ClickhouseWriter, TableName } from "../../services/ClickhouseWriter";-import { IngestionService } from "../../services/IngestionService";-import { processOtelEvents } from "../../features/otel-ingestion/processOtelEvents";+import { otelIngestionQueueProcessorBuilder } from "../../queues/otelIngestionQueue";+import { ingestionQueueProcessorBuilder } from "../../queues/ingestionQueue";+import { configureOtelReplayEnvironment } from "./otelReplaySetup";type OtelReplayStoredRow = Record<string, unknown>;+export type OtelReplayResult = {+ storedRows: OtelReplayStoredRow[];+ legacyRows?: Partial<Record<LegacyReplayTable, OtelReplayStoredRow[]>>;+};++type LegacyReplayTable =+ | TableName.Traces+ | TableName.Observations+ | TableName.ObservationsBatchStaging;++type IngestionQueueJob = Job<TQueueJobTypes[QueueName.IngestionQueue]>;++const legacyReplayTables: LegacyReplayTable[] = [+ TableName.Traces,+ TableName.Observations,+ TableName.ObservationsBatchStaging,+];+type RunOtelReplayParams = {- resourceSpans: ResourceSpan[];+ /** The exact S3 document bytes passed to the production queue processor. */+ bytes: Buffer;projectId?: string;+ orgId?: string;fileKey?: string;+ mediaUploadEnabled?: boolean;+ overflowEnabled?: boolean;+ overflowSizeLimitBytes?: number;+ writeMode?: "events_only" | "dual";+ failLegacyQueueProcessing?: boolean;};-type OtelReplayResult = {- storedRows: OtelReplayStoredRow[];+// The legacy processor caches its storage client. Keep this fake's identity stable+// across sequential replays, but release all file contents after each run.+const eventStorage = {+ source: undefined as+ | { path: string; bytes: Buffer; failLegacyDownloads?: boolean }+ | undefined,+ uploaded: new Map<string, string>(),+ async uploadJson(+ path: string,+ body: Record<string, unknown>[] | Record<string, unknown>,+ ) {+ eventStorage.uploaded.set(path, JSON.stringify(body));+ },+ read(path: string): Buffer {+ const uploaded = eventStorage.uploaded.get(path);+ if (uploaded !== undefined) {+ if (eventStorage.source?.failLegacyDownloads) {+ throw new Error("simulated legacy ingestion read failure");+ }+ return Buffer.from(uploaded);+ }+ if (path === eventStorage.source?.path) return eventStorage.source.bytes;+ throw new Error(`Replay event file not found: ${path}`);+ },+ // Only downloads need call recording: the harness asserts the selected raw-input path.+ download: vi.fn(async (path: string) =>+ eventStorage.read(path).toString("utf8"),+ ),+ downloadBytes: vi.fn(async (path: string) =>+ Uint8Array.from(eventStorage.read(path)),+ ),+ async listFiles(prefix: string) {+ return [...eventStorage.uploaded.keys()]+ .filter((path) => path.startsWith(prefix))+ .map((file) => ({ file, createdAt: new Date() }));+ },};/**- * Runs the production OTEL event phase through the production writer and reads- * the rows persisted in an isolated Memory table.+ * Replay an S3 OTEL document through the production queue and JSON writer, then+ * read its events_full rows from an isolated ClickHouse Memory table.*/export async function runOtelReplay(params: RunOtelReplayParams,): Promise<OtelReplayResult> {const projectId = params.projectId ?? "otel-replay-test";const fileKey = params.fileKey ?? "otel-replay/test.json";+ const writeMode = params.writeMode ?? "events_only";const client = clickhouseClient();const suffix = randomUUID().replaceAll("-", "");const tableName = `otel_replay_events_${suffix}`;- let writer: ClickhouseWriter | undefined;- let tableCreated = false;- let writerShutdown = false;+ const restoreEnvironment = configureOtelReplayEnvironment({+ mediaUploadEnabled: params.mediaUploadEnabled,+ overflowEnabled: params.overflowEnabled,+ overflowSizeLimitBytes: params.overflowSizeLimitBytes,+ writeMode,+ });++ eventStorage.source = {+ path: fileKey,+ bytes: Buffer.from(params.bytes),+ failLegacyDownloads: params.failLegacyQueueProcessing,+ };+ const storageClient = eventStorage as unknown as ReturnType<+ typeof getS3EventStorageClient+ >;+ const storageServiceSpy = vi+ .spyOn(StorageServiceFactory, "getInstance")+ .mockReturnValue(storageClient);++ const isolatedTableNames = new Map<TableName, string>([+ [TableName.EventsFull, tableName],+ ]);+ if (writeMode === "dual") {+ for (const legacyTable of legacyReplayTables) {+ isolatedTableNames.set(legacyTable, `${tableName}_${legacyTable}`);+ }+ }++ const createdTables: string[] = [];+ const pendingReplayJobAdds: Promise<IngestionQueueJob>[] = [];+ const restoreQueueAddSpies: Array<() => void> = [];+ let unmappedInsertError: Error | undefined;+ let writer: ReturnType<typeof ClickhouseWriter.getInstance> | undefined;+ let replayResult: OtelReplayResult | undefined;+ let primaryError: Error | undefined;+ const cleanupErrors: unknown[] = [];try {- await client.command({- query: `CREATE TABLE ${tableName} AS events_full ENGINE = Memory`,- });- tableCreated = true;-- writer = ClickhouseWriter.getInstance({- insert: async (params: Parameters<ClickhouseClientType["insert"]>[0]) =>- client.insert({- ...params,- table:- params.table === TableName.EventsFull ? tableName : params.table,- }),- } as ClickhouseClientType);-- const processor = new OtelIngestionProcessor({- projectId,- publicKey: "",- sdkName: "otel-replay",- sdkVersion: "test",- fileKey,- });- const ingestionService = new IngestionService(- {} as never,- {} as never,- writer,- {} as never,- );+ vi.mocked(getS3EventStorageClient).mockReturnValue(storageClient);- await processOtelEvents({- processor,- resourceSpans: params.resourceSpans,- ingestionService,- projectId,- fileKey,- shouldWriteToEventsTable: true,+ for (const [sourceTable, isolatedTable] of isolatedTableNames) {+ await client.command({+ query: `CREATE TABLE ${isolatedTable} AS ${sourceTable} ENGINE = Memory`,+ });+ createdTables.push(isolatedTable);+ }++ const insert = async (+ insertParams: Parameters<ClickhouseClientType["insert"]>[0],+ ) => {+ const isolatedTable = isolatedTableNames.get(+ insertParams.table as TableName,+ );+ if (!isolatedTable) {+ const sourceTable = String(insertParams.table);+ const error = new Error(+ `OTEL replay attempted an insert into unmapped ClickHouse table ${sourceTable}`,+ );+ if (!unmappedInsertError) unmappedInsertError = error;+ throw error;+ }++ return client.insert({ ...insertParams, table: isolatedTable });+ };+ const testClient = new Proxy(client, {+ get(target, property, receiver) {+ if (property === "insert") return insert;+ return Reflect.get(target, property, receiver);+ },});- // Reset the singletons here so cleanup cannot retry rows retained by this drain.await ClickhouseWriter.shutdownAll();- writerShutdown = true;+ writer = ClickhouseWriter.getInstance(testClient);- const pendingRows = writer.queue[TableName.EventsFull].length;- if (pendingRows > 0) {- throw new Error(- `ClickHouse replay writer retained ${pendingRows} EventsFull rows after shutdown`,- );+ const processor = otelIngestionQueueProcessorBuilder(false);+ const job = {+ data: {+ id: `otel-replay-${suffix}`,+ timestamp: new Date(),+ name: QueueJobs.OtelIngestionJob,+ payload: {+ data: { fileKey },+ authCheck: {+ validKey: true,+ scope: {+ projectId,+ orgId: params.orgId ?? "otel-replay-org",+ accessLevel: "project",+ },+ },+ ingestionVersion: "4",+ sdkName: "otel-replay",+ sdkVersion: "test",+ },+ },+ } as Job<TQueueJobTypes[QueueName.OtelIngestionQueue]>;++ if (writeMode === "dual") {+ for (const shardName of IngestionQueue.getShardNames()) {+ const queue = IngestionQueue.getInstance({ shardName });+ if (!queue) continue;++ const originalAdd = queue.add.bind(queue);+ const addSpy = vi.spyOn(queue, "add");+ addSpy.mockImplementation((...args) => {+ const { payload } = args[1];+ const { bucketPrefix, fileKey: queuedFileKey } = payload.data;+ const pendingAdd = originalAdd(...args);+ // Preserve the real queue add; own only this replay's uploaded files.+ if (+ payload.authCheck.scope.projectId === projectId &&+ eventStorage.uploaded.has(+ `${bucketPrefix ?? ""}${queuedFileKey}.json`,+ )+ ) {+ pendingReplayJobAdds.push(pendingAdd);+ }+ return pendingAdd;+ });+ restoreQueueAddSpies.push(() => addSpy.mockRestore());+ }+ }++ await processor(job, undefined);+ expect(eventStorage.download).toHaveBeenCalledExactlyOnceWith(fileKey);+ expect(eventStorage.downloadBytes).not.toHaveBeenCalled();++ if (writeMode === "dual") {+ const legacyProcessor = ingestionQueueProcessorBuilder(false);+ for (const legacyJob of await Promise.all(pendingReplayJobAdds)) {+ await legacyProcessor(legacyJob, undefined);+ }+ }++ await ClickhouseWriter.shutdownAll();+ for (const table of isolatedTableNames.keys()) {+ const pendingRows = writer.queue[table].length;+ if (pendingRows > 0) {+ throw new Error(+ `ClickHouse replay retained ${pendingRows} ${table} rows after shutdown`,+ );+ }}const result = await client.query({@@ -98,27 +257,91 @@ export async function runOtelReplay(`,format: "JSONEachRow",});+ const legacyRows =+ writeMode === "dual"+ ? Object.fromEntries(+ await Promise.all(+ legacyReplayTables.map(async (table) => {+ const legacyResult = await client.query({+ query: `SELECT * FROM ${isolatedTableNames.get(table)}`,+ format: "JSONEachRow",+ });+ return [+ table,+ (await legacyResult.json()) as OtelReplayStoredRow[],+ ];+ }),+ ),+ )+ : undefined;- return {+ replayResult = {storedRows: (await result.json()) as OtelReplayStoredRow[],+ legacyRows,};} catch (error) {- throw new Error(- `OTEL replay harness failed for ClickHouse table ${tableName}: ${formatUnknownError(error)}`,+ primaryError = new Error(+ `OTEL replay failed for ClickHouse table ${tableName}: ${formatUnknownError(error)}`,{ cause: error },);} finally {+ restoreEnvironment();+ const replayJobAdds = await Promise.allSettled(pendingReplayJobAdds);+ for (const restoreQueueAddSpy of restoreQueueAddSpies) {+ restoreQueueAddSpy();+ }+ const queueCleanupResults = await Promise.allSettled(+ replayJobAdds+ .filter(+ (result): result is PromiseFulfilledResult<IngestionQueueJob> =>+ result.status === "fulfilled",+ )+ .map(({ value: job }) => job.remove()),+ );+ for (const result of queueCleanupResults) {+ if (result.status === "rejected") cleanupErrors.push(result.reason);+ }+ storageServiceSpy.mockRestore();+ vi.mocked(getS3EventStorageClient).mockReset();+ eventStorage.download.mockClear();+ eventStorage.downloadBytes.mockClear();+ eventStorage.uploaded.clear();+ eventStorage.source = undefined;+try {- if (writer && !writerShutdown) {- // Each corpus replay needs a fresh singleton and interval timer.- await ClickhouseWriter.shutdownAll();- }- } finally {- if (tableCreated) {- await client.command({ query: `DROP TABLE IF EXISTS ${tableName}` });+ await ClickhouseWriter.shutdownAll();+ } catch (error) {+ cleanupErrors.push(error);+ }+ if (unmappedInsertError) {+ cleanupErrors.push(unmappedInsertError);+ }+ if (createdTables.length > 0) {+ try {+ await Promise.all(+ createdTables.map((table) =>+ client.command({ query: `DROP TABLE IF EXISTS ${table}` }),+ ),+ );+ } catch (error) {+ cleanupErrors.push(error);}}}++ if (cleanupErrors.length > 0) {+ throw new AggregateError(+ primaryError ? [primaryError, ...cleanupErrors] : cleanupErrors,+ primaryError+ ? `OTEL replay cleanup failed after ${tableName}`+ : `OTEL replay cleanup failed for ${tableName}`,+ );+ }+ if (primaryError) throw primaryError;+ if (!replayResult) {+ throw new Error(`OTEL replay produced no result for ${tableName}`);+ }+ return replayResult;}function formatUnknownError(error: unknown): string {diff --git a/worker/src/__tests__/helpers/otelReplaySetup.ts b/worker/src/__tests__/helpers/otelReplaySetup.tsindex 6c436b741d..1ed70249fb 100644--- a/worker/src/__tests__/helpers/otelReplaySetup.ts+++ b/worker/src/__tests__/helpers/otelReplaySetup.ts@@ -3,9 +3,12 @@ import { env } from "../../env";const otelReplayMocks = vi.hoisted(() => ({findModel: vi.fn(),+ getS3EventStorageClient: vi.fn(),+ linkMediaToTraceOrObservation: vi.fn(),uploadMediaForTrace: vi.fn(),getPrompt: vi.fn(),fetchObservationEvalRules: vi.fn(),+ hasNoEvalConfigsCache: vi.fn(),createObservationEvalSchedulerDeps: vi.fn(),scheduleObservationEvals: vi.fn(),}));@@ -32,6 +35,9 @@ vi.mock("../../env", async (importOriginal) => {vi.mock("@langfuse/shared/src/server", async (importOriginal) => ({...(await importOriginal<typeof import("@langfuse/shared/src/server")>()),findModel: otelReplayMocks.findModel,+ hasNoEvalConfigsCache: otelReplayMocks.hasNoEvalConfigsCache,+ getS3EventStorageClient: otelReplayMocks.getS3EventStorageClient,+ linkMediaToTraceOrObservation: otelReplayMocks.linkMediaToTraceOrObservation,PromptService: class {getPrompt(...args: unknown[]) {return otelReplayMocks.getPrompt(...args);@@ -62,12 +68,14 @@ export function configureDefaultOtelReplayMocks(): void {model: null,pricingTiers: [],});+ otelReplayMocks.linkMediaToTraceOrObservation.mockResolvedValue(undefined);otelReplayMocks.uploadMediaForTrace.mockResolvedValue({mediaId: "otel-replay-media",outcome: "uploaded",});otelReplayMocks.getPrompt.mockResolvedValue(null);otelReplayMocks.fetchObservationEvalRules.mockResolvedValue([]);+ otelReplayMocks.hasNoEvalConfigsCache.mockResolvedValue(true);otelReplayMocks.createObservationEvalSchedulerDeps.mockReturnValue({});otelReplayMocks.scheduleObservationEvals.mockResolvedValue(undefined);}@@ -76,6 +84,7 @@ type ReplayEnvironmentOptions = {mediaUploadEnabled?: boolean;overflowEnabled?: boolean;overflowSizeLimitBytes?: number;+ writeMode?: "events_only" | "dual";};/**@@ -94,10 +103,15 @@ export function configureOtelReplayEnvironment(LANGFUSE_OTEL_MEDIA_UPLOAD_ENABLED: env.LANGFUSE_OTEL_MEDIA_UPLOAD_ENABLED,LANGFUSE_S3_MEDIA_UPLOAD_BUCKET: env.LANGFUSE_S3_MEDIA_UPLOAD_BUCKET,LANGFUSE_S3_MEDIA_UPLOAD_PREFIX: env.LANGFUSE_S3_MEDIA_UPLOAD_PREFIX,+ LANGFUSE_ENABLE_BLOB_STORAGE_FILE_LOG:+ env.LANGFUSE_ENABLE_BLOB_STORAGE_FILE_LOG,LANGFUSE_OBSERVATION_FIELD_OVERFLOW_ENABLED:env.LANGFUSE_OBSERVATION_FIELD_OVERFLOW_ENABLED,LANGFUSE_OBSERVATION_FIELD_SIZE_LIMIT_BYTES:env.LANGFUSE_OBSERVATION_FIELD_SIZE_LIMIT_BYTES,+ LANGFUSE_MIGRATION_V4_WRITE_MODE: env.LANGFUSE_MIGRATION_V4_WRITE_MODE,+ LANGFUSE_MIGRATION_V4_NATIVE_OTEL_BEHAVIOUR:+ env.LANGFUSE_MIGRATION_V4_NATIVE_OTEL_BEHAVIOUR,};Object.assign(env, {@@ -109,11 +123,16 @@ export function configureOtelReplayEnvironment(: "false",LANGFUSE_S3_MEDIA_UPLOAD_BUCKET: "otel-replay-test",LANGFUSE_S3_MEDIA_UPLOAD_PREFIX: "otel-replay/",+ LANGFUSE_ENABLE_BLOB_STORAGE_FILE_LOG: "false",LANGFUSE_OBSERVATION_FIELD_OVERFLOW_ENABLED: options.overflowEnabled? "true": "false",LANGFUSE_OBSERVATION_FIELD_SIZE_LIMIT_BYTES:options.overflowSizeLimitBytes ?? 2 * 1024 * 1024,+ LANGFUSE_MIGRATION_V4_WRITE_MODE: options.writeMode ?? "events_only",+ // Keep deployment-level forcing out of the replay; direct tests select+ // that path through the same ingestionVersion=4 signal as production.+ LANGFUSE_MIGRATION_V4_NATIVE_OTEL_BEHAVIOUR: "dual_write",});return () => Object.assign(env, original);diff --git a/worker/src/__tests__/otelReplay.corpus.test.ts b/worker/src/__tests__/otelReplay.corpus.test.tsindex f427900f05..e4c54ed8ad 100644--- a/worker/src/__tests__/otelReplay.corpus.test.ts+++ b/worker/src/__tests__/otelReplay.corpus.test.ts@@ -362,8 +362,9 @@ describe("OTEL replay provider corpus", { retry: 0, timeout: 120_000 }, () => {expect(sourceSpan).toBeDefined();const { storedRows } = await runOtelReplay({- resourceSpans,+ bytes: Buffer.from(JSON.stringify(resourceSpans)),projectId: `otel-replay-${name.toLowerCase().replaceAll(" ", "-")}`,+ mediaUploadEnabled: true,});expect(storedRows).toHaveLength(1);diff --git a/worker/src/__tests__/otelReplay.integration.test.ts b/worker/src/__tests__/otelReplay.integration.test.tsindex 485d2ea2cb..1c756ee3f1 100644--- a/worker/src/__tests__/otelReplay.integration.test.ts+++ b/worker/src/__tests__/otelReplay.integration.test.ts@@ -1,8 +1,17 @@import "./helpers/otelReplaySetup";import { Decimal } from "decimal.js";+import { randomUUID } from "node:crypto";import { afterEach, beforeEach, describe, expect, it } from "vitest";-import type { ResourceSpan } from "@langfuse/shared/src/server";+import { env } from "../env";+import {+ createOrgProjectAndApiKey,+ IngestionQueue,+ QueueJobs,+ TraceUpsertQueue,+ type ResourceSpan,+} from "@langfuse/shared/src/server";+import { prisma } from "@langfuse/shared/src/db";import { runOtelReplay } from "./helpers/otelReplayHarness";import {configureDefaultOtelReplayMocks,@@ -13,9 +22,40 @@ import {const PROJECT_ID = "otel-replay-integration";const FILE_KEY = "otel-replay/integration.json";const TRACE_ID = "11111111111111111111111111111111";+const SECOND_TRACE_ID = "44444444444444444444444444444444";const FIRST_SPAN_ID = "2222222222222222";const SECOND_SPAN_ID = "3333333333333333";+async function getTraceUpsertJobsForProject(projectId: string) {+ const jobsByShard = await Promise.all(+ TraceUpsertQueue.getShardNames().map(async (shardName) => {+ const queue = TraceUpsertQueue.getInstance({ shardName });+ return queue+ ? queue.getJobs(["waiting", "delayed", "prioritized", "active"])+ : [];+ }),+ );++ return jobsByShard+ .flat()+ .filter((job) => job.data.payload.projectId === projectId);+}++async function getIngestionJobsForProject(projectId: string) {+ const jobsByShard = await Promise.all(+ IngestionQueue.getShardNames().map(async (shardName) => {+ const queue = IngestionQueue.getInstance({ shardName });+ return queue+ ? queue.getJobs(["waiting", "delayed", "prioritized", "active"])+ : [];+ }),+ );++ return jobsByShard+ .flat()+ .filter((job) => job.data.payload.authCheck.scope.projectId === projectId);+}+function createBufferId(hex: string): Buffer {return Buffer.from(hex, "hex");}@@ -31,7 +71,9 @@ function intAttribute(key: string, value: number) {};}-function buildResourceSpans(): ResourceSpan[] {+function buildResourceSpans(+ options: { separateTraces?: boolean } = {},+): ResourceSpan[] {const imageDataUri = `data:image/png;base64,${Buffer.from("small-image").toString("base64")}`;const commonAttributes = [stringAttribute("langfuse.observation.type", "generation"),@@ -80,9 +122,13 @@ function buildResourceSpans(): ResourceSpan[] {status: { code: 0 },},{- traceId: createBufferId(TRACE_ID),+ traceId: createBufferId(+ options.separateTraces ? SECOND_TRACE_ID : TRACE_ID,+ ),spanId: createBufferId(SECOND_SPAN_ID),- parentSpanId: createBufferId(FIRST_SPAN_ID),+ ...(options.separateTraces+ ? {}+ : { parentSpanId: createBufferId(FIRST_SPAN_ID) }),name: "replay-generation-with-overflow",kind: 1,startTimeUnixNano: "1714488531686000000",@@ -172,9 +218,12 @@ describe(it("writes both enriched observations and preserves accounting through readback", async () => {const { storedRows } = await runOtelReplay({- resourceSpans: buildResourceSpans(),+ bytes: Buffer.from(JSON.stringify(buildResourceSpans())),projectId: PROJECT_ID,fileKey: FILE_KEY,+ mediaUploadEnabled: true,+ overflowEnabled: true,+ overflowSizeLimitBytes: 256,});expect(storedRows).toHaveLength(2);@@ -258,5 +307,117 @@ describe(expect(Number(row.event_bytes)).toBeGreaterThan(0);}});++ it("preserves JavaScript rounding for unsafe OTLP numbers in the raw document", async () => {+ // Keep the number literal in the S3 document. Serializing a JS object+ // first would round it before the production queue reads the bytes.+ const bytes = Buffer.from(+ `[{"resource":{"attributes":[]},"scopeSpans":[{"scope":{"name":"otel-replay"},"spans":[{"traceId":"${TRACE_ID}","spanId":"${FIRST_SPAN_ID}","name":"unsafe-number","startTimeUnixNano":"1714488530686000000","endTimeUnixNano":"1714488530687000000","attributes":[{"key":"gen_ai.request.temperature","value":{"doubleValue":9007199254740993}}],"status":{"code":0}}]}]}]`,+ );+ const { storedRows } = await runOtelReplay({+ bytes,+ projectId: `${PROJECT_ID}-unsafe-number`,+ fileKey: `${FILE_KEY}.unsafe-number`,+ mediaUploadEnabled: true,+ });++ expect(storedRows).toHaveLength(1);+ const metadataIndex = (storedRows[0].metadata_names as string[]).indexOf(+ "attributes.gen_ai.request.temperature",+ );+ expect(metadataIndex).toBeGreaterThanOrEqual(0);+ expect(storedRows[0].metadata_values?.[metadataIndex]).toBe(+ "9007199254740992",+ );+ });++ it("persists legacy and events_full rows from raw OTEL bytes in dual mode", async () => {+ const { projectId, orgId } = await createOrgProjectAndApiKey();+ const originalBlobStorageFileLogFlag =+ env.LANGFUSE_ENABLE_BLOB_STORAGE_FILE_LOG;+ env.LANGFUSE_ENABLE_BLOB_STORAGE_FILE_LOG = "true";+ try {+ const { storedRows, legacyRows } = await runOtelReplay({+ bytes: Buffer.from(JSON.stringify(buildResourceSpans())),+ projectId,+ orgId,+ fileKey: `${FILE_KEY}.dual-write`,+ writeMode: "dual",+ });++ expect(storedRows).toHaveLength(2);+ expect(legacyRows?.traces?.length).toBeGreaterThan(0);+ expect(legacyRows?.observations).toHaveLength(2);+ expect(legacyRows?.observations_batch_staging).toHaveLength(0);+ expect(await getTraceUpsertJobsForProject(projectId)).toHaveLength(0);+ } finally {+ const traceJobs = await getTraceUpsertJobsForProject(projectId);+ await Promise.allSettled(traceJobs.map((job) => job.remove()));+ env.LANGFUSE_ENABLE_BLOB_STORAGE_FILE_LOG =+ originalBlobStorageFileLogFlag;+ await prisma.project.delete({ where: { id: projectId } });+ await prisma.organization.delete({ where: { id: orgId } });+ }+ });++ it("removes replay-owned ingestion jobs when a legacy replay job fails", async () => {+ const { projectId, orgId } = await createOrgProjectAndApiKey();+ const originalBlobStorageFileLogFlag =+ env.LANGFUSE_ENABLE_BLOB_STORAGE_FILE_LOG;+ env.LANGFUSE_ENABLE_BLOB_STORAGE_FILE_LOG = "false";++ const unrelatedQueue = IngestionQueue.getInstance({+ shardingKey: `${projectId}-unrelated`,+ });+ if (!unrelatedQueue)+ throw new Error("Ingestion queue is not initialized");+ const unrelatedJob = await unrelatedQueue.add(QueueJobs.IngestionJob, {+ id: `otel-replay-unrelated-${randomUUID()}`,+ timestamp: new Date(),+ name: QueueJobs.IngestionJob,+ payload: {+ data: {+ type: "span-create",+ eventBodyId: `otel-replay-unrelated-${randomUUID()}`,+ fileKey: "unrelated.json",+ bucketPrefix: "otel-replay/unrelated/",+ ingestionApiKey: "",+ ingestionSdkName: "otel-replay-test",+ ingestionSdkVersion: "test",+ },+ authCheck: {+ validKey: true,+ scope: { projectId, orgId, accessLevel: "project" },+ },+ },+ });++ try {+ await expect(+ runOtelReplay({+ bytes: Buffer.from(+ JSON.stringify(buildResourceSpans({ separateTraces: true })),+ ),+ projectId,+ orgId,+ fileKey: `${FILE_KEY}.legacy-failure`,+ writeMode: "dual",+ failLegacyQueueProcessing: true,+ }),+ ).rejects.toThrow("simulated legacy ingestion read failure");++ const remainingJobs = await getIngestionJobsForProject(projectId);+ expect(remainingJobs.map((job) => job.id)).toEqual([unrelatedJob.id]);+ } finally {+ const remainingJobs = await getIngestionJobsForProject(projectId);+ await Promise.allSettled(remainingJobs.map((job) => job.remove()));+ const traceJobs = await getTraceUpsertJobsForProject(projectId);+ await Promise.allSettled(traceJobs.map((job) => job.remove()));+ env.LANGFUSE_ENABLE_BLOB_STORAGE_FILE_LOG =+ originalBlobStorageFileLogFlag;+ await prisma.project.delete({ where: { id: projectId } });+ await prisma.organization.delete({ where: { id: orgId } });+ }+ });},);diff --git a/worker/src/__tests__/otelReplay.properties.test.ts b/worker/src/__tests__/otelReplay.properties.test.tsindex c319ea7908..6688ea0076 100644--- a/worker/src/__tests__/otelReplay.properties.test.ts+++ b/worker/src/__tests__/otelReplay.properties.test.ts@@ -642,9 +642,10 @@ const directedCases: ReplayCase[] = [async function assertReplayPersists(params: ReplayCase): Promise<void> {const replay = buildReplayCase(params);const { storedRows } = await runOtelReplay({- resourceSpans: replay.resourceSpans,+ bytes: Buffer.from(JSON.stringify(replay.resourceSpans)),projectId: replay.projectId,fileKey: FILE_KEY,+ mediaUploadEnabled: true,});expect(storedRows).toHaveLength(replay.expectedRows.length);diff --git a/worker/src/features/otel-ingestion/README.md b/worker/src/features/otel-ingestion/README.mdindex e80466409c..374bbb1628 100644--- a/worker/src/features/otel-ingestion/README.md+++ b/worker/src/features/otel-ingestion/README.md@@ -1,47 +1,61 @@# OTEL replay tests-Run the production direct-event processing phase against an isolated ClickHouse-table:+Run the OTEL replay suites against an isolated ClickHouse table:```shpnpm --filter worker run test:otel-replay```-Use the normal worker test environment and a migrated ClickHouse database. The+Use local or disposable test services: migrated ClickHouse and Postgres databases+and Redis without active ingestion consumers. Thetest client uses `CLICKHOUSE_URL`, `CLICKHOUSE_USER`, `CLICKHOUSE_PASSWORD`, and`CLICKHOUSE_DB`; Vitest loads the repository's `.env` without overriding suppliedenvironment variables. The configured user needs to read the `events_full`schema and create, insert into, select from, and drop the test tables. Anunavailable ClickHouse server fails the run.-The tests start with decoded OTLP resource spans and call `processOtelEvents`,-the same function used by the queue. They exercise event normalization, media-transformation, enrichment, overflow handling, and the real ClickHouse writer.-External service responses and uploads are supplied by test doubles. Each run-creates a uniquely named Memory table from `events_full` and uses the production-writer singleton. An insert-only adapter forwards the normal insert parameters-and substitutes the temporary table only for `EventsFull`. The helper shuts-down the writer before reading persisted rows. Cleanup clears the test's-singleton references so each sequential replay starts with the normal writer-settings and a fresh interval timer, then drops the table.--The Memory table exercises ClickHouse types, column defaults, and serialization.-Production MergeTree deduplication and materialized views are outside this run.--This is an integration test of the direct-event phase. HTTP/protobuf decoding,-S3 input loading, masking, routing, and legacy writes remain outside this test-boundary. The existing `test:native-codec` suite compares the TypeScript writer-with Rust Native encoding inside ClickHouse. A future complete Rust processor-can consume the same replay inputs and use that table comparison.--## Remote execution--The same command can run from a checkout on an authorized remote host, including-a bastion with an appropriate runtime and database access, or locally through-an existing tunnel. Install the repository dependencies and build the shared-package as for local worker tests; provide the remote database connection in the-environment. No code needs to execute on the bastion when using a tunnel.--The current corpus is synthetic. Fetching production S3 batches and replaying a-time window are separate extensions; this command does not discover production-credentials, enqueue production jobs, or download customer data.+The corpus, integration, and property suites serialize their fixtures to S3+document bytes and pass those bytes to the production OTEL queue processor. The+queue performs the normal download and JSON decoding; normalization, enrichment,+tokenization, media and overflow handling, and ClickHouse persistence use the+TypeScript implementations. Model and prompt lookups, media uploads, and+evaluation scheduling use controlled test doubles.++Each replay creates isolated Memory tables from the live schema and uses the+production JSON writer singleton. A thin client adapter substitutes the+temporary destination table for production table names and rejects unmapped+inserts. Blob-storage file logging and background trace evaluation jobs are+disabled. The harness drains the writer before reading persisted rows. The+integration suite also covers dual writes by draining only its queued legacy+jobs into isolated trace and observation tables and removing those jobs even+when legacy processing fails. The suites keep generated seeds reproducible and+disable retries. Memory tables cover ClickHouse types, defaults, and serialization;+MergeTree deduplication and production materialized views are outside the run.++## Scope and caveats++- Scratch tables isolate ClickHouse inserts and provide predictable readback+ targets. They do not isolate the whole application from production services.+ Dual tests create Postgres project/session state and real Redis jobs; run them+ only against test services. Another worker consuming those jobs would race the+ harness.+- Each replay processes one document under the direct-v4 route, with optional+ legacy writes. The job explicitly selects ingestion version 4; an empty+ staging table does not test the queue-forwarded events route.+- Legacy merge reads still target ordinary trace and observation tables. Dual+ cases use fresh projects with no existing rows. They do not cover updates+ across replay calls: a later call cannot read the earlier scratch writes.+ Such tests need reads and writes to share the same isolated state first.+- The harness invokes production queue processors in-process, including the+ legacy jobs it enqueues. It does not exercise BullMQ worker consumption,+ Redis deserialization, scheduling, or retries.+- Media uploads and association writes use test doubles. Their calls can verify+ content and destinations, but do not prove object-storage persistence or+ Postgres media/link persistence. Model/prompt lookups and evaluation services+ are also controlled rather than end-to-end.++The current corpus is synthetic. Production S3 capture is a separate extension:+preserve the raw object bytes, then replay against local services with an+explicit routing/configuration context. Running from a bastion is compatible+with that model, but pointing this test command at production databases or+queues is not.