Skip to content

🪢 Open source agent evals & observability: Trace, evaluate, and improve LLM applications with one open platform.

chore(otel): replay raw documents through the production queue (#18217)

Valery Meleshkinb1fc56a8fe11
diff --git a/worker/AGENTS.md b/worker/AGENTS.md
index c4b3db68ef..142aa72d34 100644
--- a/worker/AGENTS.md
+++ b/worker/AGENTS.md
@@ -47,9 +47,10 @@
in `src/initialize.ts`, the health probe call in `src/api/index.ts`. Native code
records its own metrics and logs; see `../packages/native/AGENTS.md`.
- Tests: `src/__tests__/*`, `src/queues/__tests__/*`
-- Direct-event replay: `pnpm --filter worker run test:otel-replay` exercises the
- production OTEL event phase with isolated ClickHouse tables. Setup and scope:
- `src/features/otel-ingestion/README.md`.
+- Direct-event replay: `pnpm --filter worker run test:otel-replay` feeds raw S3
+ JSON through the production OTEL queue and JSON writer into isolated
+ ClickHouse tables, including retained legacy writes in dual-write mode. Setup
+ and scope: `src/features/otel-ingestion/README.md`.
- Native codec checks: `pnpm --filter worker run test:native-codec` selects
`nativeCodec` suites. Build the addon first; the command requires ClickHouse
whenever the live parity suite is present.
diff --git a/worker/src/__tests__/helpers/otelReplayHarness.ts b/worker/src/__tests__/helpers/otelReplayHarness.ts
index d8b1f03df7..873f334716 100644
--- a/worker/src/__tests__/helpers/otelReplayHarness.ts
+++ b/worker/src/__tests__/helpers/otelReplayHarness.ts
@@ -1,91 +1,250 @@
import { randomUUID } from "node:crypto";
+import { expect, vi } from "vitest";
+import type { Job } from "bullmq";
+
import {
clickhouseClient,
+ getS3EventStorageClient,
+ IngestionQueue,
+ QueueJobs,
+ QueueName,
+ StorageServiceFactory,
type ClickhouseClientType,
- type ResourceSpan,
- OtelIngestionProcessor,
+ type TQueueJobTypes,
} from "@langfuse/shared/src/server";
import { ClickhouseWriter, TableName } from "../../services/ClickhouseWriter";
-import { IngestionService } from "../../services/IngestionService";
-import { processOtelEvents } from "../../features/otel-ingestion/processOtelEvents";
+import { otelIngestionQueueProcessorBuilder } from "../../queues/otelIngestionQueue";
+import { ingestionQueueProcessorBuilder } from "../../queues/ingestionQueue";
+import { configureOtelReplayEnvironment } from "./otelReplaySetup";
type OtelReplayStoredRow = Record<string, unknown>;
+export type OtelReplayResult = {
+ storedRows: OtelReplayStoredRow[];
+ legacyRows?: Partial<Record<LegacyReplayTable, OtelReplayStoredRow[]>>;
+};
+
+type LegacyReplayTable =
+ | TableName.Traces
+ | TableName.Observations
+ | TableName.ObservationsBatchStaging;
+
+type IngestionQueueJob = Job<TQueueJobTypes[QueueName.IngestionQueue]>;
+
+const legacyReplayTables: LegacyReplayTable[] = [
+ TableName.Traces,
+ TableName.Observations,
+ TableName.ObservationsBatchStaging,
+];
+
type RunOtelReplayParams = {
- resourceSpans: ResourceSpan[];
+ /** The exact S3 document bytes passed to the production queue processor. */
+ bytes: Buffer;
projectId?: string;
+ orgId?: string;
fileKey?: string;
+ mediaUploadEnabled?: boolean;
+ overflowEnabled?: boolean;
+ overflowSizeLimitBytes?: number;
+ writeMode?: "events_only" | "dual";
+ failLegacyQueueProcessing?: boolean;
};
-type OtelReplayResult = {
- storedRows: OtelReplayStoredRow[];
+// The legacy processor caches its storage client. Keep this fake's identity stable
+// across sequential replays, but release all file contents after each run.
+const eventStorage = {
+ source: undefined as
+ | { path: string; bytes: Buffer; failLegacyDownloads?: boolean }
+ | undefined,
+ uploaded: new Map<string, string>(),
+ async uploadJson(
+ path: string,
+ body: Record<string, unknown>[] | Record<string, unknown>,
+ ) {
+ eventStorage.uploaded.set(path, JSON.stringify(body));
+ },
+ read(path: string): Buffer {
+ const uploaded = eventStorage.uploaded.get(path);
+ if (uploaded !== undefined) {
+ if (eventStorage.source?.failLegacyDownloads) {
+ throw new Error("simulated legacy ingestion read failure");
+ }
+ return Buffer.from(uploaded);
+ }
+ if (path === eventStorage.source?.path) return eventStorage.source.bytes;
+ throw new Error(`Replay event file not found: ${path}`);
+ },
+ // Only downloads need call recording: the harness asserts the selected raw-input path.
+ download: vi.fn(async (path: string) =>
+ eventStorage.read(path).toString("utf8"),
+ ),
+ downloadBytes: vi.fn(async (path: string) =>
+ Uint8Array.from(eventStorage.read(path)),
+ ),
+ async listFiles(prefix: string) {
+ return [...eventStorage.uploaded.keys()]
+ .filter((path) => path.startsWith(prefix))
+ .map((file) => ({ file, createdAt: new Date() }));
+ },
};
/**
- * Runs the production OTEL event phase through the production writer and reads
- * the rows persisted in an isolated Memory table.
+ * Replay an S3 OTEL document through the production queue and JSON writer, then
+ * read its events_full rows from an isolated ClickHouse Memory table.
*/
export async function runOtelReplay(
params: RunOtelReplayParams,
): Promise<OtelReplayResult> {
const projectId = params.projectId ?? "otel-replay-test";
const fileKey = params.fileKey ?? "otel-replay/test.json";
+ const writeMode = params.writeMode ?? "events_only";
const client = clickhouseClient();
const suffix = randomUUID().replaceAll("-", "");
const tableName = `otel_replay_events_${suffix}`;
- let writer: ClickhouseWriter | undefined;
- let tableCreated = false;
- let writerShutdown = false;
+ const restoreEnvironment = configureOtelReplayEnvironment({
+ mediaUploadEnabled: params.mediaUploadEnabled,
+ overflowEnabled: params.overflowEnabled,
+ overflowSizeLimitBytes: params.overflowSizeLimitBytes,
+ writeMode,
+ });
+
+ eventStorage.source = {
+ path: fileKey,
+ bytes: Buffer.from(params.bytes),
+ failLegacyDownloads: params.failLegacyQueueProcessing,
+ };
+ const storageClient = eventStorage as unknown as ReturnType<
+ typeof getS3EventStorageClient
+ >;
+ const storageServiceSpy = vi
+ .spyOn(StorageServiceFactory, "getInstance")
+ .mockReturnValue(storageClient);
+
+ const isolatedTableNames = new Map<TableName, string>([
+ [TableName.EventsFull, tableName],
+ ]);
+ if (writeMode === "dual") {
+ for (const legacyTable of legacyReplayTables) {
+ isolatedTableNames.set(legacyTable, `${tableName}_${legacyTable}`);
+ }
+ }
+
+ const createdTables: string[] = [];
+ const pendingReplayJobAdds: Promise<IngestionQueueJob>[] = [];
+ const restoreQueueAddSpies: Array<() => void> = [];
+ let unmappedInsertError: Error | undefined;
+ let writer: ReturnType<typeof ClickhouseWriter.getInstance> | undefined;
+ let replayResult: OtelReplayResult | undefined;
+ let primaryError: Error | undefined;
+ const cleanupErrors: unknown[] = [];
try {
- await client.command({
- query: `CREATE TABLE ${tableName} AS events_full ENGINE = Memory`,
- });
- tableCreated = true;
-
- writer = ClickhouseWriter.getInstance({
- insert: async (params: Parameters<ClickhouseClientType["insert"]>[0]) =>
- client.insert({
- ...params,
- table:
- params.table === TableName.EventsFull ? tableName : params.table,
- }),
- } as ClickhouseClientType);
-
- const processor = new OtelIngestionProcessor({
- projectId,
- publicKey: "",
- sdkName: "otel-replay",
- sdkVersion: "test",
- fileKey,
- });
- const ingestionService = new IngestionService(
- {} as never,
- {} as never,
- writer,
- {} as never,
- );
+ vi.mocked(getS3EventStorageClient).mockReturnValue(storageClient);
- await processOtelEvents({
- processor,
- resourceSpans: params.resourceSpans,
- ingestionService,
- projectId,
- fileKey,
- shouldWriteToEventsTable: true,
+ for (const [sourceTable, isolatedTable] of isolatedTableNames) {
+ await client.command({
+ query: `CREATE TABLE ${isolatedTable} AS ${sourceTable} ENGINE = Memory`,
+ });
+ createdTables.push(isolatedTable);
+ }
+
+ const insert = async (
+ insertParams: Parameters<ClickhouseClientType["insert"]>[0],
+ ) => {
+ const isolatedTable = isolatedTableNames.get(
+ insertParams.table as TableName,
+ );
+ if (!isolatedTable) {
+ const sourceTable = String(insertParams.table);
+ const error = new Error(
+ `OTEL replay attempted an insert into unmapped ClickHouse table ${sourceTable}`,
+ );
+ if (!unmappedInsertError) unmappedInsertError = error;
+ throw error;
+ }
+
+ return client.insert({ ...insertParams, table: isolatedTable });
+ };
+ const testClient = new Proxy(client, {
+ get(target, property, receiver) {
+ if (property === "insert") return insert;
+ return Reflect.get(target, property, receiver);
+ },
});
- // Reset the singletons here so cleanup cannot retry rows retained by this drain.
await ClickhouseWriter.shutdownAll();
- writerShutdown = true;
+ writer = ClickhouseWriter.getInstance(testClient);
- const pendingRows = writer.queue[TableName.EventsFull].length;
- if (pendingRows > 0) {
- throw new Error(
- `ClickHouse replay writer retained ${pendingRows} EventsFull rows after shutdown`,
- );
+ const processor = otelIngestionQueueProcessorBuilder(false);
+ const job = {
+ data: {
+ id: `otel-replay-${suffix}`,
+ timestamp: new Date(),
+ name: QueueJobs.OtelIngestionJob,
+ payload: {
+ data: { fileKey },
+ authCheck: {
+ validKey: true,
+ scope: {
+ projectId,
+ orgId: params.orgId ?? "otel-replay-org",
+ accessLevel: "project",
+ },
+ },
+ ingestionVersion: "4",
+ sdkName: "otel-replay",
+ sdkVersion: "test",
+ },
+ },
+ } as Job<TQueueJobTypes[QueueName.OtelIngestionQueue]>;
+
+ if (writeMode === "dual") {
+ for (const shardName of IngestionQueue.getShardNames()) {
+ const queue = IngestionQueue.getInstance({ shardName });
+ if (!queue) continue;
+
+ const originalAdd = queue.add.bind(queue);
+ const addSpy = vi.spyOn(queue, "add");
+ addSpy.mockImplementation((...args) => {
+ const { payload } = args[1];
+ const { bucketPrefix, fileKey: queuedFileKey } = payload.data;
+ const pendingAdd = originalAdd(...args);
+ // Preserve the real queue add; own only this replay's uploaded files.
+ if (
+ payload.authCheck.scope.projectId === projectId &&
+ eventStorage.uploaded.has(
+ `${bucketPrefix ?? ""}${queuedFileKey}.json`,
+ )
+ ) {
+ pendingReplayJobAdds.push(pendingAdd);
+ }
+ return pendingAdd;
+ });
+ restoreQueueAddSpies.push(() => addSpy.mockRestore());
+ }
+ }
+
+ await processor(job, undefined);
+ expect(eventStorage.download).toHaveBeenCalledExactlyOnceWith(fileKey);
+ expect(eventStorage.downloadBytes).not.toHaveBeenCalled();
+
+ if (writeMode === "dual") {
+ const legacyProcessor = ingestionQueueProcessorBuilder(false);
+ for (const legacyJob of await Promise.all(pendingReplayJobAdds)) {
+ await legacyProcessor(legacyJob, undefined);
+ }
+ }
+
+ await ClickhouseWriter.shutdownAll();
+ for (const table of isolatedTableNames.keys()) {
+ const pendingRows = writer.queue[table].length;
+ if (pendingRows > 0) {
+ throw new Error(
+ `ClickHouse replay retained ${pendingRows} ${table} rows after shutdown`,
+ );
+ }
}
const result = await client.query({
@@ -98,27 +257,91 @@ export async function runOtelReplay(
`,
format: "JSONEachRow",
});
+ const legacyRows =
+ writeMode === "dual"
+ ? Object.fromEntries(
+ await Promise.all(
+ legacyReplayTables.map(async (table) => {
+ const legacyResult = await client.query({
+ query: `SELECT * FROM ${isolatedTableNames.get(table)}`,
+ format: "JSONEachRow",
+ });
+ return [
+ table,
+ (await legacyResult.json()) as OtelReplayStoredRow[],
+ ];
+ }),
+ ),
+ )
+ : undefined;
- return {
+ replayResult = {
storedRows: (await result.json()) as OtelReplayStoredRow[],
+ legacyRows,
};
} catch (error) {
- throw new Error(
- `OTEL replay harness failed for ClickHouse table ${tableName}: ${formatUnknownError(error)}`,
+ primaryError = new Error(
+ `OTEL replay failed for ClickHouse table ${tableName}: ${formatUnknownError(error)}`,
{ cause: error },
);
} finally {
+ restoreEnvironment();
+ const replayJobAdds = await Promise.allSettled(pendingReplayJobAdds);
+ for (const restoreQueueAddSpy of restoreQueueAddSpies) {
+ restoreQueueAddSpy();
+ }
+ const queueCleanupResults = await Promise.allSettled(
+ replayJobAdds
+ .filter(
+ (result): result is PromiseFulfilledResult<IngestionQueueJob> =>
+ result.status === "fulfilled",
+ )
+ .map(({ value: job }) => job.remove()),
+ );
+ for (const result of queueCleanupResults) {
+ if (result.status === "rejected") cleanupErrors.push(result.reason);
+ }
+ storageServiceSpy.mockRestore();
+ vi.mocked(getS3EventStorageClient).mockReset();
+ eventStorage.download.mockClear();
+ eventStorage.downloadBytes.mockClear();
+ eventStorage.uploaded.clear();
+ eventStorage.source = undefined;
+
try {
- if (writer && !writerShutdown) {
- // Each corpus replay needs a fresh singleton and interval timer.
- await ClickhouseWriter.shutdownAll();
- }
- } finally {
- if (tableCreated) {
- await client.command({ query: `DROP TABLE IF EXISTS ${tableName}` });
+ await ClickhouseWriter.shutdownAll();
+ } catch (error) {
+ cleanupErrors.push(error);
+ }
+ if (unmappedInsertError) {
+ cleanupErrors.push(unmappedInsertError);
+ }
+ if (createdTables.length > 0) {
+ try {
+ await Promise.all(
+ createdTables.map((table) =>
+ client.command({ query: `DROP TABLE IF EXISTS ${table}` }),
+ ),
+ );
+ } catch (error) {
+ cleanupErrors.push(error);
}
}
}
+
+ if (cleanupErrors.length > 0) {
+ throw new AggregateError(
+ primaryError ? [primaryError, ...cleanupErrors] : cleanupErrors,
+ primaryError
+ ? `OTEL replay cleanup failed after ${tableName}`
+ : `OTEL replay cleanup failed for ${tableName}`,
+ );
+ }
+ if (primaryError) throw primaryError;
+ if (!replayResult) {
+ throw new Error(`OTEL replay produced no result for ${tableName}`);
+ }
+ return replayResult;
}
function formatUnknownError(error: unknown): string {
diff --git a/worker/src/__tests__/helpers/otelReplaySetup.ts b/worker/src/__tests__/helpers/otelReplaySetup.ts
index 6c436b741d..1ed70249fb 100644
--- a/worker/src/__tests__/helpers/otelReplaySetup.ts
+++ b/worker/src/__tests__/helpers/otelReplaySetup.ts
@@ -3,9 +3,12 @@ import { env } from "../../env";
const otelReplayMocks = vi.hoisted(() => ({
findModel: vi.fn(),
+ getS3EventStorageClient: vi.fn(),
+ linkMediaToTraceOrObservation: vi.fn(),
uploadMediaForTrace: vi.fn(),
getPrompt: vi.fn(),
fetchObservationEvalRules: vi.fn(),
+ hasNoEvalConfigsCache: vi.fn(),
createObservationEvalSchedulerDeps: vi.fn(),
scheduleObservationEvals: vi.fn(),
}));
@@ -32,6 +35,9 @@ vi.mock("../../env", async (importOriginal) => {
vi.mock("@langfuse/shared/src/server", async (importOriginal) => ({
...(await importOriginal<typeof import("@langfuse/shared/src/server")>()),
findModel: otelReplayMocks.findModel,
+ hasNoEvalConfigsCache: otelReplayMocks.hasNoEvalConfigsCache,
+ getS3EventStorageClient: otelReplayMocks.getS3EventStorageClient,
+ linkMediaToTraceOrObservation: otelReplayMocks.linkMediaToTraceOrObservation,
PromptService: class {
getPrompt(...args: unknown[]) {
return otelReplayMocks.getPrompt(...args);
@@ -62,12 +68,14 @@ export function configureDefaultOtelReplayMocks(): void {
model: null,
pricingTiers: [],
});
+ otelReplayMocks.linkMediaToTraceOrObservation.mockResolvedValue(undefined);
otelReplayMocks.uploadMediaForTrace.mockResolvedValue({
mediaId: "otel-replay-media",
outcome: "uploaded",
});
otelReplayMocks.getPrompt.mockResolvedValue(null);
otelReplayMocks.fetchObservationEvalRules.mockResolvedValue([]);
+ otelReplayMocks.hasNoEvalConfigsCache.mockResolvedValue(true);
otelReplayMocks.createObservationEvalSchedulerDeps.mockReturnValue({});
otelReplayMocks.scheduleObservationEvals.mockResolvedValue(undefined);
}
@@ -76,6 +84,7 @@ type ReplayEnvironmentOptions = {
mediaUploadEnabled?: boolean;
overflowEnabled?: boolean;
overflowSizeLimitBytes?: number;
+ writeMode?: "events_only" | "dual";
};
/**
@@ -94,10 +103,15 @@ export function configureOtelReplayEnvironment(
LANGFUSE_OTEL_MEDIA_UPLOAD_ENABLED: env.LANGFUSE_OTEL_MEDIA_UPLOAD_ENABLED,
LANGFUSE_S3_MEDIA_UPLOAD_BUCKET: env.LANGFUSE_S3_MEDIA_UPLOAD_BUCKET,
LANGFUSE_S3_MEDIA_UPLOAD_PREFIX: env.LANGFUSE_S3_MEDIA_UPLOAD_PREFIX,
+ LANGFUSE_ENABLE_BLOB_STORAGE_FILE_LOG:
+ env.LANGFUSE_ENABLE_BLOB_STORAGE_FILE_LOG,
LANGFUSE_OBSERVATION_FIELD_OVERFLOW_ENABLED:
env.LANGFUSE_OBSERVATION_FIELD_OVERFLOW_ENABLED,
LANGFUSE_OBSERVATION_FIELD_SIZE_LIMIT_BYTES:
env.LANGFUSE_OBSERVATION_FIELD_SIZE_LIMIT_BYTES,
+ LANGFUSE_MIGRATION_V4_WRITE_MODE: env.LANGFUSE_MIGRATION_V4_WRITE_MODE,
+ LANGFUSE_MIGRATION_V4_NATIVE_OTEL_BEHAVIOUR:
+ env.LANGFUSE_MIGRATION_V4_NATIVE_OTEL_BEHAVIOUR,
};
Object.assign(env, {
@@ -109,11 +123,16 @@ export function configureOtelReplayEnvironment(
: "false",
LANGFUSE_S3_MEDIA_UPLOAD_BUCKET: "otel-replay-test",
LANGFUSE_S3_MEDIA_UPLOAD_PREFIX: "otel-replay/",
+ LANGFUSE_ENABLE_BLOB_STORAGE_FILE_LOG: "false",
LANGFUSE_OBSERVATION_FIELD_OVERFLOW_ENABLED: options.overflowEnabled
? "true"
: "false",
LANGFUSE_OBSERVATION_FIELD_SIZE_LIMIT_BYTES:
options.overflowSizeLimitBytes ?? 2 * 1024 * 1024,
+ LANGFUSE_MIGRATION_V4_WRITE_MODE: options.writeMode ?? "events_only",
+ // Keep deployment-level forcing out of the replay; direct tests select
+ // that path through the same ingestionVersion=4 signal as production.
+ LANGFUSE_MIGRATION_V4_NATIVE_OTEL_BEHAVIOUR: "dual_write",
});
return () => Object.assign(env, original);
diff --git a/worker/src/__tests__/otelReplay.corpus.test.ts b/worker/src/__tests__/otelReplay.corpus.test.ts
index f427900f05..e4c54ed8ad 100644
--- a/worker/src/__tests__/otelReplay.corpus.test.ts
+++ b/worker/src/__tests__/otelReplay.corpus.test.ts
@@ -362,8 +362,9 @@ describe("OTEL replay provider corpus", { retry: 0, timeout: 120_000 }, () => {
expect(sourceSpan).toBeDefined();
const { storedRows } = await runOtelReplay({
- resourceSpans,
+ bytes: Buffer.from(JSON.stringify(resourceSpans)),
projectId: `otel-replay-${name.toLowerCase().replaceAll(" ", "-")}`,
+ mediaUploadEnabled: true,
});
expect(storedRows).toHaveLength(1);
diff --git a/worker/src/__tests__/otelReplay.integration.test.ts b/worker/src/__tests__/otelReplay.integration.test.ts
index 485d2ea2cb..1c756ee3f1 100644
--- a/worker/src/__tests__/otelReplay.integration.test.ts
+++ b/worker/src/__tests__/otelReplay.integration.test.ts
@@ -1,8 +1,17 @@
import "./helpers/otelReplaySetup";
import { Decimal } from "decimal.js";
+import { randomUUID } from "node:crypto";
import { afterEach, beforeEach, describe, expect, it } from "vitest";
-import type { ResourceSpan } from "@langfuse/shared/src/server";
+import { env } from "../env";
+import {
+ createOrgProjectAndApiKey,
+ IngestionQueue,
+ QueueJobs,
+ TraceUpsertQueue,
+ type ResourceSpan,
+} from "@langfuse/shared/src/server";
+import { prisma } from "@langfuse/shared/src/db";
import { runOtelReplay } from "./helpers/otelReplayHarness";
import {
configureDefaultOtelReplayMocks,
@@ -13,9 +22,40 @@ import {
const PROJECT_ID = "otel-replay-integration";
const FILE_KEY = "otel-replay/integration.json";
const TRACE_ID = "11111111111111111111111111111111";
+const SECOND_TRACE_ID = "44444444444444444444444444444444";
const FIRST_SPAN_ID = "2222222222222222";
const SECOND_SPAN_ID = "3333333333333333";
+async function getTraceUpsertJobsForProject(projectId: string) {
+ const jobsByShard = await Promise.all(
+ TraceUpsertQueue.getShardNames().map(async (shardName) => {
+ const queue = TraceUpsertQueue.getInstance({ shardName });
+ return queue
+ ? queue.getJobs(["waiting", "delayed", "prioritized", "active"])
+ : [];
+ }),
+ );
+
+ return jobsByShard
+ .flat()
+ .filter((job) => job.data.payload.projectId === projectId);
+}
+
+async function getIngestionJobsForProject(projectId: string) {
+ const jobsByShard = await Promise.all(
+ IngestionQueue.getShardNames().map(async (shardName) => {
+ const queue = IngestionQueue.getInstance({ shardName });
+ return queue
+ ? queue.getJobs(["waiting", "delayed", "prioritized", "active"])
+ : [];
+ }),
+ );
+
+ return jobsByShard
+ .flat()
+ .filter((job) => job.data.payload.authCheck.scope.projectId === projectId);
+}
+
function createBufferId(hex: string): Buffer {
return Buffer.from(hex, "hex");
}
@@ -31,7 +71,9 @@ function intAttribute(key: string, value: number) {
};
}
-function buildResourceSpans(): ResourceSpan[] {
+function buildResourceSpans(
+ options: { separateTraces?: boolean } = {},
+): ResourceSpan[] {
const imageDataUri = `data:image/png;base64,${Buffer.from("small-image").toString("base64")}`;
const commonAttributes = [
stringAttribute("langfuse.observation.type", "generation"),
@@ -80,9 +122,13 @@ function buildResourceSpans(): ResourceSpan[] {
status: { code: 0 },
},
{
- traceId: createBufferId(TRACE_ID),
+ traceId: createBufferId(
+ options.separateTraces ? SECOND_TRACE_ID : TRACE_ID,
+ ),
spanId: createBufferId(SECOND_SPAN_ID),
- parentSpanId: createBufferId(FIRST_SPAN_ID),
+ ...(options.separateTraces
+ ? {}
+ : { parentSpanId: createBufferId(FIRST_SPAN_ID) }),
name: "replay-generation-with-overflow",
kind: 1,
startTimeUnixNano: "1714488531686000000",
@@ -172,9 +218,12 @@ describe(
it("writes both enriched observations and preserves accounting through readback", async () => {
const { storedRows } = await runOtelReplay({
- resourceSpans: buildResourceSpans(),
+ bytes: Buffer.from(JSON.stringify(buildResourceSpans())),
projectId: PROJECT_ID,
fileKey: FILE_KEY,
+ mediaUploadEnabled: true,
+ overflowEnabled: true,
+ overflowSizeLimitBytes: 256,
});
expect(storedRows).toHaveLength(2);
@@ -258,5 +307,117 @@ describe(
expect(Number(row.event_bytes)).toBeGreaterThan(0);
}
});
+
+ it("preserves JavaScript rounding for unsafe OTLP numbers in the raw document", async () => {
+ // Keep the number literal in the S3 document. Serializing a JS object
+ // first would round it before the production queue reads the bytes.
+ const bytes = Buffer.from(
+ `[{"resource":{"attributes":[]},"scopeSpans":[{"scope":{"name":"otel-replay"},"spans":[{"traceId":"${TRACE_ID}","spanId":"${FIRST_SPAN_ID}","name":"unsafe-number","startTimeUnixNano":"1714488530686000000","endTimeUnixNano":"1714488530687000000","attributes":[{"key":"gen_ai.request.temperature","value":{"doubleValue":9007199254740993}}],"status":{"code":0}}]}]}]`,
+ );
+ const { storedRows } = await runOtelReplay({
+ bytes,
+ projectId: `${PROJECT_ID}-unsafe-number`,
+ fileKey: `${FILE_KEY}.unsafe-number`,
+ mediaUploadEnabled: true,
+ });
+
+ expect(storedRows).toHaveLength(1);
+ const metadataIndex = (storedRows[0].metadata_names as string[]).indexOf(
+ "attributes.gen_ai.request.temperature",
+ );
+ expect(metadataIndex).toBeGreaterThanOrEqual(0);
+ expect(storedRows[0].metadata_values?.[metadataIndex]).toBe(
+ "9007199254740992",
+ );
+ });
+
+ it("persists legacy and events_full rows from raw OTEL bytes in dual mode", async () => {
+ const { projectId, orgId } = await createOrgProjectAndApiKey();
+ const originalBlobStorageFileLogFlag =
+ env.LANGFUSE_ENABLE_BLOB_STORAGE_FILE_LOG;
+ env.LANGFUSE_ENABLE_BLOB_STORAGE_FILE_LOG = "true";
+ try {
+ const { storedRows, legacyRows } = await runOtelReplay({
+ bytes: Buffer.from(JSON.stringify(buildResourceSpans())),
+ projectId,
+ orgId,
+ fileKey: `${FILE_KEY}.dual-write`,
+ writeMode: "dual",
+ });
+
+ expect(storedRows).toHaveLength(2);
+ expect(legacyRows?.traces?.length).toBeGreaterThan(0);
+ expect(legacyRows?.observations).toHaveLength(2);
+ expect(legacyRows?.observations_batch_staging).toHaveLength(0);
+ expect(await getTraceUpsertJobsForProject(projectId)).toHaveLength(0);
+ } finally {
+ const traceJobs = await getTraceUpsertJobsForProject(projectId);
+ await Promise.allSettled(traceJobs.map((job) => job.remove()));
+ env.LANGFUSE_ENABLE_BLOB_STORAGE_FILE_LOG =
+ originalBlobStorageFileLogFlag;
+ await prisma.project.delete({ where: { id: projectId } });
+ await prisma.organization.delete({ where: { id: orgId } });
+ }
+ });
+
+ it("removes replay-owned ingestion jobs when a legacy replay job fails", async () => {
+ const { projectId, orgId } = await createOrgProjectAndApiKey();
+ const originalBlobStorageFileLogFlag =
+ env.LANGFUSE_ENABLE_BLOB_STORAGE_FILE_LOG;
+ env.LANGFUSE_ENABLE_BLOB_STORAGE_FILE_LOG = "false";
+
+ const unrelatedQueue = IngestionQueue.getInstance({
+ shardingKey: `${projectId}-unrelated`,
+ });
+ if (!unrelatedQueue)
+ throw new Error("Ingestion queue is not initialized");
+ const unrelatedJob = await unrelatedQueue.add(QueueJobs.IngestionJob, {
+ id: `otel-replay-unrelated-${randomUUID()}`,
+ timestamp: new Date(),
+ name: QueueJobs.IngestionJob,
+ payload: {
+ data: {
+ type: "span-create",
+ eventBodyId: `otel-replay-unrelated-${randomUUID()}`,
+ fileKey: "unrelated.json",
+ bucketPrefix: "otel-replay/unrelated/",
+ ingestionApiKey: "",
+ ingestionSdkName: "otel-replay-test",
+ ingestionSdkVersion: "test",
+ },
+ authCheck: {
+ validKey: true,
+ scope: { projectId, orgId, accessLevel: "project" },
+ },
+ },
+ });
+
+ try {
+ await expect(
+ runOtelReplay({
+ bytes: Buffer.from(
+ JSON.stringify(buildResourceSpans({ separateTraces: true })),
+ ),
+ projectId,
+ orgId,
+ fileKey: `${FILE_KEY}.legacy-failure`,
+ writeMode: "dual",
+ failLegacyQueueProcessing: true,
+ }),
+ ).rejects.toThrow("simulated legacy ingestion read failure");
+
+ const remainingJobs = await getIngestionJobsForProject(projectId);
+ expect(remainingJobs.map((job) => job.id)).toEqual([unrelatedJob.id]);
+ } finally {
+ const remainingJobs = await getIngestionJobsForProject(projectId);
+ await Promise.allSettled(remainingJobs.map((job) => job.remove()));
+ const traceJobs = await getTraceUpsertJobsForProject(projectId);
+ await Promise.allSettled(traceJobs.map((job) => job.remove()));
+ env.LANGFUSE_ENABLE_BLOB_STORAGE_FILE_LOG =
+ originalBlobStorageFileLogFlag;
+ await prisma.project.delete({ where: { id: projectId } });
+ await prisma.organization.delete({ where: { id: orgId } });
+ }
+ });
},
);
diff --git a/worker/src/__tests__/otelReplay.properties.test.ts b/worker/src/__tests__/otelReplay.properties.test.ts
index c319ea7908..6688ea0076 100644
--- a/worker/src/__tests__/otelReplay.properties.test.ts
+++ b/worker/src/__tests__/otelReplay.properties.test.ts
@@ -642,9 +642,10 @@ const directedCases: ReplayCase[] = [
async function assertReplayPersists(params: ReplayCase): Promise<void> {
const replay = buildReplayCase(params);
const { storedRows } = await runOtelReplay({
- resourceSpans: replay.resourceSpans,
+ bytes: Buffer.from(JSON.stringify(replay.resourceSpans)),
projectId: replay.projectId,
fileKey: FILE_KEY,
+ mediaUploadEnabled: true,
});
expect(storedRows).toHaveLength(replay.expectedRows.length);
diff --git a/worker/src/features/otel-ingestion/README.md b/worker/src/features/otel-ingestion/README.md
index e80466409c..374bbb1628 100644
--- a/worker/src/features/otel-ingestion/README.md
+++ b/worker/src/features/otel-ingestion/README.md
@@ -1,47 +1,61 @@
# OTEL replay tests
-Run the production direct-event processing phase against an isolated ClickHouse
-table:
+Run the OTEL replay suites against an isolated ClickHouse table:
```sh
pnpm --filter worker run test:otel-replay
```
-Use the normal worker test environment and a migrated ClickHouse database. The
+Use local or disposable test services: migrated ClickHouse and Postgres databases
+and Redis without active ingestion consumers. The
test client uses `CLICKHOUSE_URL`, `CLICKHOUSE_USER`, `CLICKHOUSE_PASSWORD`, and
`CLICKHOUSE_DB`; Vitest loads the repository's `.env` without overriding supplied
environment variables. The configured user needs to read the `events_full`
schema and create, insert into, select from, and drop the test tables. An
unavailable ClickHouse server fails the run.
-The tests start with decoded OTLP resource spans and call `processOtelEvents`,
-the same function used by the queue. They exercise event normalization, media
-transformation, enrichment, overflow handling, and the real ClickHouse writer.
-External service responses and uploads are supplied by test doubles. Each run
-creates a uniquely named Memory table from `events_full` and uses the production
-writer singleton. An insert-only adapter forwards the normal insert parameters
-and substitutes the temporary table only for `EventsFull`. The helper shuts
-down the writer before reading persisted rows. Cleanup clears the test's
-singleton references so each sequential replay starts with the normal writer
-settings and a fresh interval timer, then drops the table.
-
-The Memory table exercises ClickHouse types, column defaults, and serialization.
-Production MergeTree deduplication and materialized views are outside this run.
-
-This is an integration test of the direct-event phase. HTTP/protobuf decoding,
-S3 input loading, masking, routing, and legacy writes remain outside this test
-boundary. The existing `test:native-codec` suite compares the TypeScript writer
-with Rust Native encoding inside ClickHouse. A future complete Rust processor
-can consume the same replay inputs and use that table comparison.
-
-## Remote execution
-
-The same command can run from a checkout on an authorized remote host, including
-a bastion with an appropriate runtime and database access, or locally through
-an existing tunnel. Install the repository dependencies and build the shared
-package as for local worker tests; provide the remote database connection in the
-environment. No code needs to execute on the bastion when using a tunnel.
-
-The current corpus is synthetic. Fetching production S3 batches and replaying a
-time window are separate extensions; this command does not discover production
-credentials, enqueue production jobs, or download customer data.
+The corpus, integration, and property suites serialize their fixtures to S3
+document bytes and pass those bytes to the production OTEL queue processor. The
+queue performs the normal download and JSON decoding; normalization, enrichment,
+tokenization, media and overflow handling, and ClickHouse persistence use the
+TypeScript implementations. Model and prompt lookups, media uploads, and
+evaluation scheduling use controlled test doubles.
+
+Each replay creates isolated Memory tables from the live schema and uses the
+production JSON writer singleton. A thin client adapter substitutes the
+temporary destination table for production table names and rejects unmapped
+inserts. Blob-storage file logging and background trace evaluation jobs are
+disabled. The harness drains the writer before reading persisted rows. The
+integration suite also covers dual writes by draining only its queued legacy
+jobs into isolated trace and observation tables and removing those jobs even
+when legacy processing fails. The suites keep generated seeds reproducible and
+disable retries. Memory tables cover ClickHouse types, defaults, and serialization;
+MergeTree deduplication and production materialized views are outside the run.
+
+## Scope and caveats
+
+- Scratch tables isolate ClickHouse inserts and provide predictable readback
+ targets. They do not isolate the whole application from production services.
+ Dual tests create Postgres project/session state and real Redis jobs; run them
+ only against test services. Another worker consuming those jobs would race the
+ harness.
+- Each replay processes one document under the direct-v4 route, with optional
+ legacy writes. The job explicitly selects ingestion version 4; an empty
+ staging table does not test the queue-forwarded events route.
+- Legacy merge reads still target ordinary trace and observation tables. Dual
+ cases use fresh projects with no existing rows. They do not cover updates
+ across replay calls: a later call cannot read the earlier scratch writes.
+ Such tests need reads and writes to share the same isolated state first.
+- The harness invokes production queue processors in-process, including the
+ legacy jobs it enqueues. It does not exercise BullMQ worker consumption,
+ Redis deserialization, scheduling, or retries.
+- Media uploads and association writes use test doubles. Their calls can verify
+ content and destinations, but do not prove object-storage persistence or
+ Postgres media/link persistence. Model/prompt lookups and evaluation services
+ are also controlled rather than end-to-end.
+
+The current corpus is synthetic. Production S3 capture is a separate extension:
+preserve the raw object bytes, then replay against local services with an
+explicit routing/configuration context. Running from a bastion is compatible
+with that model, but pointing this test command at production databases or
+queues is not.