🪢 Open source agent evals & observability: Trace, evaluate, and improve LLM applications with one open platform.
TypeScript analyticsautogenevaluationlangchainlarge-language-modelsllama-indexllmllm-evaluationllm-observabilityllmopsmonitoringobservabilityopen-sourceopenaiplaygroundprompt-engineeringprompt-managementself-hostedycombinator
feat(worker): emit export data-freshness lag distribution (#17177)
Niklas Semmlera3d7e290621d
diff --git a/.agents/skills/datadog-query-recipes/SKILL.md b/.agents/skills/datadog-query-recipes/SKILL.mdindex 1f97ef1cce..e03ec09f60 100644--- a/.agents/skills/datadog-query-recipes/SKILL.md+++ b/.agents/skills/datadog-query-recipes/SKILL.md@@ -41,6 +41,8 @@ data domain you need: traces, logs, metrics, and visualizations.[`references/public-api-tenant-usage.md`](references/public-api-tenant-usage.md)- Queue inventory, queue consumers, and queue metrics:[`references/queue-consumers.md`](references/queue-consumers.md)+ - Scheduled-export freshness lag (blob / PostHog / Mixpanel):+ [`references/export-freshness-lag.md`](references/export-freshness-lag.md)3. Start with aggregate queries, grouped by environment, service, route,queue, project, org, status, or error facets as appropriate.4. Fetch raw spans, logs, or traces only after aggregation identifies thediff --git a/.agents/skills/datadog-query-recipes/references/export-freshness-lag.md b/.agents/skills/datadog-query-recipes/references/export-freshness-lag.mdnew file mode 100644index 0000000000..7b77156c80--- /dev/null+++ b/.agents/skills/datadog-query-recipes/references/export-freshness-lag.md@@ -0,0 +1,43 @@+# Export freshness lag++Scheduled integrations (blob, PostHog, Mixpanel) emit a pooled distribution at+the end of each run. Provenance: the shape from LFE-15853.++## Metric++- Name: `langfuse.export.freshness_lag_seconds`+- Type: distribution+- Value: `runStartTime − maxExportedTimestamp` (seconds, clamped at 0)+- Watermark: `lastSyncAt` after a successful window (`maxTimestamp` just+ written). Failing runs use the **unchanged** watermark so stall shows as+ rising lag. Do not treat intended window end as the watermark.+- Tags: `integration` (`blob_storage` / `posthog` / `mixpanel`), `window`+ (`20m` / `1h` / `1d` / `1w`), `status` (`success` / `failure`), `unit:seconds`.+ `env` comes from the agent. **No `project_id`.**+- PostHog and Mixpanel are tagged `window:1h` (hourly cron). Blob uses the+ configured cadence.++## Baseline P95 (both Datadog sites)++EU (`prod-eu`) and US (`prod-us`, `prod-hipaa`, `prod-jp`):++```text+p95:langfuse.export.freshness_lag_seconds{*} by {integration,window,env}+```++Split success vs failure while baselining:++```text+p95:langfuse.export.freshness_lag_seconds{*} by {integration,window,status}+```++Catch-up (historic backfill) sits in the tail. P95 is the planned SLI **if**+those runs stay under ~5% of pooled volume. If they do not, filter runs whose+target window is older than N× the cadence (follow-up, not this metric).++## Emit sites++- `worker/src/features/blobstorage/handleBlobStorageIntegrationProjectJob.ts`+- `worker/src/features/posthog/handlePostHogIntegrationProjectJob.ts`+- `worker/src/features/mixpanel/handleMixpanelIntegrationProjectJob.ts`+- Helper: `worker/src/services/exportFreshnessLagMetric.ts`diff --git a/worker/src/__tests__/blobStorageIntegrationProcessing.test.ts b/worker/src/__tests__/blobStorageIntegrationProcessing.test.tsindex cb9c07bfa4..275693fb93 100644--- a/worker/src/__tests__/blobStorageIntegrationProcessing.test.ts+++ b/worker/src/__tests__/blobStorageIntegrationProcessing.test.ts@@ -15,10 +15,11 @@ const originalCloudRegion = vi.hoisted(() => {return cloudRegion;});-// Override recordIncrement + recordHistogram so the attempt counter and-// per-stage timing metrics are assertable.+// Override recordIncrement + recordHistogram + recordDistribution so the+// attempt counter, per-stage timing, and freshness-lag metrics are assertable.const mockRecordIncrement = vi.hoisted(() => vi.fn());const mockRecordHistogram = vi.hoisted(() => vi.fn());+const mockRecordDistribution = vi.hoisted(() => vi.fn());// Stubbable endpoint preflight — the customer-fault tests reject it to drive an// in-try failure without infra. Defaults to the real impl for other tests.const mockValidateBlobStorageEndpoint = vi.hoisted(() => vi.fn());@@ -34,6 +35,7 @@ vi.mock("@langfuse/shared/src/server", async (importOriginal) => {...actual,recordIncrement: mockRecordIncrement,recordHistogram: mockRecordHistogram,+ recordDistribution: mockRecordDistribution,validateBlobStorageEndpoint: mockValidateBlobStorageEndpoint,};});@@ -54,7 +56,9 @@ import {createEventsCh,StorageService,StorageServiceFactory,+ BlobStorageIntegrationProcessingQueue,} from "@langfuse/shared/src/server";+import { EXPORT_FRESHNESS_LAG_METRIC } from "../services/exportFreshnessLagMetric";import { prisma } from "@langfuse/shared/src/db";import { Job } from "bullmq";import {@@ -1838,6 +1842,78 @@ describe("BlobStorageIntegrationProcessingJob", () => {expect(timeDiff).toBeLessThan(5000); // Within 5 seconds});+ it("records freshness success when catch-up enqueue fails after the watermark advances", async () => {+ const { projectId } = await createOrgProjectAndApiKey();+ s3Prefix = projectId;+ const now = new Date();+ const twoDaysAgo = new Date(now.getTime() - 2 * 24 * 60 * 60 * 1000);++ const trace = createTrace({+ project_id: projectId,+ timestamp: twoDaysAgo.getTime(),+ name: "Old Trace",+ });+ await createTracesCh([trace]);++ await prisma.blobStorageIntegration.create({+ data: {+ projectId,+ type: BlobStorageIntegrationType.S3,+ bucketName,+ prefix: s3Prefix,+ accessKeyId: minioAccessKeyId,+ secretAccessKey: encrypt(minioAccessKeySecret),+ region: region ? region : "auto",+ endpoint: minioEndpoint,+ forcePathStyle:+ env.LANGFUSE_S3_EVENT_UPLOAD_FORCE_PATH_STYLE === "true",+ enabled: true,+ exportFrequency: "hourly",+ lastSyncAt: twoDaysAgo,+ compressed: false,+ },+ });++ const add = vi.fn().mockRejectedValue(new Error("redis down"));+ const getInstanceSpy = vi+ .spyOn(BlobStorageIntegrationProcessingQueue, "getInstance")+ .mockReturnValue({ add } as never);+ mockRecordDistribution.mockClear();++ try {+ await expect(+ handleBlobStorageIntegrationProjectJob({+ data: { payload: { projectId } },+ } as Job),+ ).rejects.toThrow(/redis down/);++ const updatedIntegration =+ await prisma.blobStorageIntegration.findUnique({+ where: { projectId },+ });+ expect(updatedIntegration?.lastSyncAt?.getTime()).toBe(+ twoDaysAgo.getTime() + 60 * 60 * 1000,+ );+ expect(add).toHaveBeenCalled();+ expect(mockRecordDistribution).toHaveBeenCalledWith(+ EXPORT_FRESHNESS_LAG_METRIC,+ expect.any(Number),+ expect.objectContaining({+ integration: "blob_storage",+ window: "1h",+ status: "success",+ }),+ );+ expect(mockRecordDistribution).not.toHaveBeenCalledWith(+ EXPORT_FRESHNESS_LAG_METRIC,+ expect.anything(),+ expect.objectContaining({ status: "failure" }),+ );+ } finally {+ getInstanceSpy.mockRestore();+ }+ });+it("should coalesce a sub-second remainder instead of emitting a colliding chunk", async () => {// Regression for the silent object-key collision: a caught-up run whose// full-interval chunk ends only a sub-second before the frontier used todiff --git a/worker/src/__tests__/exportFreshnessLagMetric.test.ts b/worker/src/__tests__/exportFreshnessLagMetric.test.tsnew file mode 100644index 0000000000..4957c1ac2f--- /dev/null+++ b/worker/src/__tests__/exportFreshnessLagMetric.test.ts@@ -0,0 +1,112 @@+import { describe, it, expect, vi, beforeEach } from "vitest";++const recordDistribution = vi.fn();+vi.mock("@langfuse/shared/src/server", () => ({+ recordDistribution: (...args: unknown[]) => recordDistribution(...args),+}));++import {+ recordExportFreshnessLag,+ windowClassFromBlobFrequency,+ EXPORT_FRESHNESS_LAG_METRIC,+} from "../services/exportFreshnessLagMetric";++describe("recordExportFreshnessLag", () => {+ const runStartTime = new Date("2026-09-08T12:00:00.000Z");++ beforeEach(() => recordDistribution.mockClear());++ it("emits runStart − watermark in seconds as a distribution, without project_id", () => {+ recordExportFreshnessLag({+ integration: "blob_storage",+ window: "20m",+ status: "success",+ runStartTime,+ maxExportedTimestamp: new Date("2026-09-08T11:40:00.000Z"),+ });++ expect(recordDistribution).toHaveBeenCalledWith(+ EXPORT_FRESHNESS_LAG_METRIC,+ 1200,+ {+ integration: "blob_storage",+ window: "20m",+ status: "success",+ unit: "seconds",+ },+ );+ expect(recordDistribution.mock.calls[0][2]).not.toHaveProperty(+ "project_id",+ );+ expect(recordDistribution.mock.calls[0][2]).not.toHaveProperty("projectId");+ });++ it("emits failing runs against the unchanged watermark so stall shows as rising lag", () => {+ recordExportFreshnessLag({+ integration: "posthog",+ window: "1h",+ status: "failure",+ runStartTime,+ maxExportedTimestamp: new Date("2026-09-08T06:00:00.000Z"),+ });++ expect(recordDistribution).toHaveBeenCalledWith(+ EXPORT_FRESHNESS_LAG_METRIC,+ 6 * 60 * 60,+ {+ integration: "posthog",+ window: "1h",+ status: "failure",+ unit: "seconds",+ },+ );+ });++ it("skips when there is no watermark yet", () => {+ recordExportFreshnessLag({+ integration: "mixpanel",+ window: "1h",+ status: "failure",+ runStartTime,+ maxExportedTimestamp: null,+ });+ recordExportFreshnessLag({+ integration: "mixpanel",+ window: "1h",+ status: "success",+ runStartTime,+ maxExportedTimestamp: undefined,+ });+ expect(recordDistribution).not.toHaveBeenCalled();+ });++ it("clamps a watermark ahead of run start to zero", () => {+ recordExportFreshnessLag({+ integration: "blob_storage",+ window: "1d",+ status: "success",+ runStartTime,+ maxExportedTimestamp: new Date("2026-09-08T12:05:00.000Z"),+ });+ expect(recordDistribution).toHaveBeenCalledWith(+ EXPORT_FRESHNESS_LAG_METRIC,+ 0,+ expect.objectContaining({ status: "success", window: "1d" }),+ );+ });+});++describe("windowClassFromBlobFrequency", () => {+ it("maps configured blob cadences to window classes", () => {+ expect(windowClassFromBlobFrequency("every_20_minutes")).toBe("20m");+ expect(windowClassFromBlobFrequency("hourly")).toBe("1h");+ expect(windowClassFromBlobFrequency("daily")).toBe("1d");+ expect(windowClassFromBlobFrequency("weekly")).toBe("1w");+ });++ it("rejects an unknown frequency", () => {+ expect(() => windowClassFromBlobFrequency("monthly")).toThrow(+ /Unsupported export frequency/,+ );+ });+});diff --git a/worker/src/__tests__/mixpanelIntegrationProjectJob.test.ts b/worker/src/__tests__/mixpanelIntegrationProjectJob.test.tsindex a631f40d57..5cd10db86a 100644--- a/worker/src/__tests__/mixpanelIntegrationProjectJob.test.ts+++ b/worker/src/__tests__/mixpanelIntegrationProjectJob.test.ts@@ -128,6 +128,7 @@ vi.mock("@langfuse/shared/src/server", () => ({QueueName: { MixpanelIntegrationProcessingQueue: "mixpanel" },logger: { info: vi.fn(), warn: vi.fn(), error: vi.fn(), debug: vi.fn() },recordIncrement: vi.fn(),+ recordDistribution: vi.fn(),getCurrentSpan: vi.fn(() => undefined),getTracesForAnalyticsIntegrations: vi.fn(() => h.fakeStream("traces")),getGenerationsForAnalyticsIntegrations: vi.fn(() =>@@ -145,7 +146,10 @@ import {import {getScoresForAnalyticsIntegrations,recordIncrement,+ recordDistribution,} from "@langfuse/shared/src/server";+import { decrypt } from "@langfuse/shared/encryption";+import { EXPORT_FRESHNESS_LAG_METRIC } from "../services/exportFreshnessLagMetric";import { env } from "../env";// Mirrors the real queue's defaultJobOptions (mixpanelIntegrationProcessingQueue.ts).@@ -362,4 +366,26 @@ describe("handleMixpanelIntegrationProjectJob customer-fault observability", ()expect(recordIncrement).not.toHaveBeenCalled();});++ it("emits freshness failure when decrypt throws before export work starts", async () => {+ vi.mocked(decrypt).mockImplementationOnce(() => {+ throw new Error("bad token");+ });+ vi.mocked(recordDistribution).mockClear();++ await expect(+ handleMixpanelIntegrationProjectJob(makeJob()),+ ).rejects.toThrow(/bad token/);++ expect(recordDistribution).toHaveBeenCalledWith(+ EXPORT_FRESHNESS_LAG_METRIC,+ expect.any(Number),+ expect.objectContaining({+ integration: "mixpanel",+ window: "1h",+ status: "failure",+ }),+ );+ expect(h.mixpanelIntegrationUpdate).not.toHaveBeenCalled();+ });});diff --git a/worker/src/__tests__/posthogIntegrationProjectJob.test.ts b/worker/src/__tests__/posthogIntegrationProjectJob.test.tsindex e325984abb..104514807b 100644--- a/worker/src/__tests__/posthogIntegrationProjectJob.test.ts+++ b/worker/src/__tests__/posthogIntegrationProjectJob.test.ts@@ -92,6 +92,7 @@ const h = vi.hoisted(() => {// hostname) and the customer notification, both controllable per test.const validateWebhookURL = vi.fn(async () => {});const recordIncrement = vi.fn();+ const recordDistribution = vi.fn();const dispatchProjectNotification = vi.fn(async () => {await new Promise((resolve) => setTimeout(resolve, 0));try {@@ -143,6 +144,7 @@ const h = vi.hoisted(() => {notification,validateWebhookURL,recordIncrement,+ recordDistribution,dispatchProjectNotification,OutboundUrlValidationError,getTraces,@@ -174,6 +176,7 @@ vi.mock("@langfuse/shared/src/server", () => ({QueueName: { PostHogIntegrationProcessingQueue: "posthog" },logger: { info: vi.fn(), warn: vi.fn(), error: vi.fn(), debug: vi.fn() },recordIncrement: h.recordIncrement,+ recordDistribution: h.recordDistribution,getCurrentSpan: vi.fn(() => undefined),validateWebhookURL: h.validateWebhookURL,dispatchProjectNotification: h.dispatchProjectNotification,@@ -277,6 +280,7 @@ function resetSharedState() {h.notification.settled = false;h.notification.rejects = false;h.recordIncrement.mockClear();+ h.recordDistribution.mockClear();h.validateWebhookURL.mockReset();h.validateWebhookURL.mockImplementation(async () => {});h.dispatchProjectNotification.mockClear();diff --git a/worker/src/features/blobstorage/handleBlobStorageIntegrationProjectJob.ts b/worker/src/features/blobstorage/handleBlobStorageIntegrationProjectJob.tsindex ae4d8eab25..d72d24727e 100644--- a/worker/src/features/blobstorage/handleBlobStorageIntegrationProjectJob.ts+++ b/worker/src/features/blobstorage/handleBlobStorageIntegrationProjectJob.ts@@ -75,6 +75,10 @@ import { SpanKind } from "@opentelemetry/api";import { env } from "../../env";import { assertExportSourceWritable } from "../exportWriteModeGuard";import { recordExportVolume } from "../../services/exportVolumeMetric";+import {+ recordExportFreshnessLag,+ windowClassFromBlobFrequency,+} from "../../services/exportFreshnessLagMetric";import {buildBlobExportManifest,buildBlobExportManifestKey,@@ -1216,9 +1220,10 @@ export const handleBlobStorageIntegrationProjectJob = async (return;}+ const runStartTime = new Date();const { count: claimed } = await prisma.blobStorageIntegration.updateMany({where: { projectId },- data: { runStartedAt: new Date() },+ data: { runStartedAt: runStartTime },});if (claimed === 0) {logger.info(@@ -1275,6 +1280,15 @@ export const handleBlobStorageIntegrationProjectJob = async (lastErrorAt: null,},});+ recordExportFreshnessLag({+ integration: "blob_storage",+ window: windowClassFromBlobFrequency(+ blobStorageIntegration.exportFrequency,+ ),+ status: "success",+ runStartTime,+ maxExportedTimestamp: blobStorageIntegration.lastSyncAt,+ });return;}@@ -1282,6 +1296,7 @@ export const handleBlobStorageIntegrationProjectJob = async (// self-hosted), so the deprecation notice below is Cloud-only too.const isCloud = Boolean(env.NEXT_PUBLIC_LANGFUSE_CLOUD_REGION);+ let watermarkAdvanced = false;try {// The catch persists lastError and notifies admins.assertExportSourceWritable(@@ -1510,6 +1525,20 @@ export const handleBlobStorageIntegrationProjectJob = async (return;}+ // The export watermark is committed. Catch-up enqueue below can still+ // fail (Redis); that must not be recorded as an export-freshness failure+ // against the pre-run lastSyncAt.+ recordExportFreshnessLag({+ integration: "blob_storage",+ window: windowClassFromBlobFrequency(+ blobStorageIntegration.exportFrequency,+ ),+ status: "success",+ runStartTime,+ maxExportedTimestamp: maxTimestamp,+ });+ watermarkAdvanced = true;+// If still catching up, immediately queue the next chunk jobif (!caughtUp) {const queue = BlobStorageIntegrationProcessingQueue.getInstance();@@ -1552,6 +1581,18 @@ export const handleBlobStorageIntegrationProjectJob = async (return; // obsolete job: complete it rather than fail it}+ if (!watermarkAdvanced) {+ recordExportFreshnessLag({+ integration: "blob_storage",+ window: windowClassFromBlobFrequency(+ blobStorageIntegration.exportFrequency,+ ),+ status: "failure",+ runStartTime,+ maxExportedTimestamp: blobStorageIntegration.lastSyncAt,+ });+ }+switch (outcome.kind) {case "disabled-by-us":// Awaited: the integration is now off, so the scheduler never revisitsdiff --git a/worker/src/features/mixpanel/handleMixpanelIntegrationProjectJob.ts b/worker/src/features/mixpanel/handleMixpanelIntegrationProjectJob.tsindex 38dcf1b1be..9de828c236 100644--- a/worker/src/features/mixpanel/handleMixpanelIntegrationProjectJob.ts+++ b/worker/src/features/mixpanel/handleMixpanelIntegrationProjectJob.ts@@ -14,6 +14,7 @@ import {import { decrypt } from "@langfuse/shared/encryption";import { MixpanelClient } from "./mixpanelClient";import { recordExportVolume } from "../../services/exportVolumeMetric";+import { recordExportFreshnessLag } from "../../services/exportFreshnessLagMetric";import {transformTraceForMixpanel,transformGenerationForMixpanel,@@ -249,21 +250,23 @@ export const handleMixpanelIntegrationProjectJob = async (return;}- // Fetch relevant data and send it to Mixpanel- const executionConfig: MixpanelExecutionConfig = {- projectId,- projectName: mixpanelIntegration.project.name,- // Start from 2000-01-01 if no lastSyncAt. Workaround because 1970-01-01 leads to subtle bugs in ClickHouse- minTimestamp: mixpanelIntegration.lastSyncAt || new Date("2000-01-01"),- maxTimestamp: new Date(new Date().getTime() - 30 * 60 * 1000), // 30 minutes ago- decryptedMixpanelProjectToken: decrypt(- mixpanelIntegration.encryptedMixpanelProjectToken,- ),- mixpanelRegion: mixpanelIntegration.mixpanelRegion,- useGraceHash: job.attemptsMade > 0,- };+ const runStartTime = new Date();try {+ // Fetch relevant data and send it to Mixpanel+ const executionConfig: MixpanelExecutionConfig = {+ projectId,+ projectName: mixpanelIntegration.project.name,+ // Start from 2000-01-01 if no lastSyncAt. Workaround because 1970-01-01 leads to subtle bugs in ClickHouse+ minTimestamp: mixpanelIntegration.lastSyncAt || new Date("2000-01-01"),+ maxTimestamp: new Date(new Date().getTime() - 30 * 60 * 1000), // 30 minutes ago+ decryptedMixpanelProjectToken: decrypt(+ mixpanelIntegration.encryptedMixpanelProjectToken,+ ),+ mixpanelRegion: mixpanelIntegration.mixpanelRegion,+ useGraceHash: job.attemptsMade > 0,+ };+// Fail loudly before exporting empty data and advancing lastSyncAt// (LFE-10148, LFE-11009); the catch below logs and BullMQ retries.assertExportSourceWritable(@@ -314,10 +317,24 @@ export const handleMixpanelIntegrationProjectJob = async (bytes: mixpanel.getSerializedBytes(),projectId,});+ recordExportFreshnessLag({+ integration: "mixpanel",+ window: "1h",+ status: "success",+ runStartTime,+ maxExportedTimestamp: executionConfig.maxTimestamp,+ });logger.info(`[MIXPANEL] Mixpanel integration processing complete for project ${projectId}`,);} catch (error) {+ recordExportFreshnessLag({+ integration: "mixpanel",+ window: "1h",+ status: "failure",+ runStartTime,+ maxExportedTimestamp: mixpanelIntegration.lastSyncAt,+ });const mixpanelFaultReason = classifyCustomerFault(error);if (mixpanelFaultReason !== undefined) {recordIncrement(MIXPANEL_INTEGRATION_CUSTOMER_FAULT_METRIC, 1, {diff --git a/worker/src/features/posthog/handlePostHogIntegrationProjectJob.ts b/worker/src/features/posthog/handlePostHogIntegrationProjectJob.tsindex 075705b29e..8c8c966c5a 100644--- a/worker/src/features/posthog/handlePostHogIntegrationProjectJob.ts+++ b/worker/src/features/posthog/handlePostHogIntegrationProjectJob.ts@@ -28,6 +28,7 @@ import {import { decrypt } from "@langfuse/shared/encryption";import { PostHog } from "posthog-node";import { recordExportVolume } from "../../services/exportVolumeMetric";+import { recordExportFreshnessLag } from "../../services/exportFreshnessLagMetric";import { assertExportSourceWritable } from "../exportWriteModeGuard";import { classifyCustomerFault } from "../integrations/customerFaultClassification";import { isRecordNotFoundError } from "../integrations/prismaErrors";@@ -284,6 +285,8 @@ export const handlePostHogIntegrationProjectJob = async (return;}+ const runStartTime = new Date();+try {// Validate PostHog hostname to prevent SSRF attacks before sending data.// Rewrap preserving { cause } so the single catch below can classify the@@ -327,6 +330,13 @@ export const handlePostHogIntegrationProjectJob = async (logger.info(`[POSTHOG] Skipping PostHog integration for project ${projectId}: empty sync window (min: ${minTimestamp.toISOString()}, max: ${maxTimestamp.toISOString()})`,);+ recordExportFreshnessLag({+ integration: "posthog",+ window: "1h",+ status: "success",+ runStartTime,+ maxExportedTimestamp: postHogIntegration.lastSyncAt,+ });return;}@@ -425,6 +435,13 @@ export const handlePostHogIntegrationProjectJob = async (bytes: executionConfig.volume.bytes,projectId,});+ recordExportFreshnessLag({+ integration: "posthog",+ window: "1h",+ status: "success",+ runStartTime,+ maxExportedTimestamp: executionConfig.maxTimestamp,+ });logger.info(`[POSTHOG] PostHog integration processing complete for project ${projectId}`,);@@ -444,6 +461,14 @@ export const handlePostHogIntegrationProjectJob = async (error instanceof Error ? error.message : String(error)).slice(0, 1000);+ recordExportFreshnessLag({+ integration: "posthog",+ window: "1h",+ status: "failure",+ runStartTime,+ maxExportedTimestamp: postHogIntegration.lastSyncAt,+ });+if (reason === undefined) {// Defensive fallback: every code today either classifies above and// disables (including one raised on a redirect hop, reached through thediff --git a/worker/src/services/exportFreshnessLagMetric.ts b/worker/src/services/exportFreshnessLagMetric.tsnew file mode 100644index 0000000000..740b2899ec--- /dev/null+++ b/worker/src/services/exportFreshnessLagMetric.ts@@ -0,0 +1,62 @@+import { recordDistribution } from "@langfuse/shared/src/server";++export const EXPORT_FRESHNESS_LAG_METRIC =+ "langfuse.export.freshness_lag_seconds";++export type ExportFreshnessIntegration =+ | "blob_storage"+ | "posthog"+ | "mixpanel";++export type ExportFreshnessWindow = "20m" | "1h" | "1d" | "1w";++export type ExportFreshnessStatus = "success" | "failure";++const BLOB_FREQUENCY_TO_WINDOW: Record<string, ExportFreshnessWindow> = {+ every_20_minutes: "20m",+ hourly: "1h",+ daily: "1d",+ weekly: "1w",+};++export const windowClassFromBlobFrequency = (+ frequency: string,+): ExportFreshnessWindow => {+ const window = BLOB_FREQUENCY_TO_WINDOW[frequency];+ if (!window) {+ throw new Error(`Unsupported export frequency: ${frequency}`);+ }+ return window;+};++/**+ * Seconds the newest successfully-exported timestamp lags this run's start.+ * On success, pass the watermark just written (this run's maxTimestamp).+ * On failure, pass the unchanged lastSyncAt so lag climbs until recovery.+ * First runs with no watermark yet are skipped.+ */+export const recordExportFreshnessLag = ({+ integration,+ window,+ status,+ runStartTime,+ maxExportedTimestamp,+}: {+ integration: ExportFreshnessIntegration;+ window: ExportFreshnessWindow;+ status: ExportFreshnessStatus;+ runStartTime: Date;+ maxExportedTimestamp: Date | null | undefined;+}): void => {+ if (!maxExportedTimestamp) {+ return;+ }+ const lagSeconds =+ (runStartTime.getTime() - maxExportedTimestamp.getTime()) / 1000;+ recordDistribution(EXPORT_FRESHNESS_LAG_METRIC, Math.max(0, lagSeconds), {+ integration,+ window,+ status,+ unit: "seconds",+ });+};