Skip to content

🪢 Open source agent evals & observability: Trace, evaluate, and improve LLM applications with one open platform.

feat(worker): emit export data-freshness lag distribution (#17177)

Niklas Semmlera3d7e290621d
diff --git a/.agents/skills/datadog-query-recipes/SKILL.md b/.agents/skills/datadog-query-recipes/SKILL.md
index 1f97ef1cce..e03ec09f60 100644
--- a/.agents/skills/datadog-query-recipes/SKILL.md
+++ b/.agents/skills/datadog-query-recipes/SKILL.md
@@ -41,6 +41,8 @@ data domain you need: traces, logs, metrics, and visualizations.
[`references/public-api-tenant-usage.md`](references/public-api-tenant-usage.md)
- Queue inventory, queue consumers, and queue metrics:
[`references/queue-consumers.md`](references/queue-consumers.md)
+ - Scheduled-export freshness lag (blob / PostHog / Mixpanel):
+ [`references/export-freshness-lag.md`](references/export-freshness-lag.md)
3. Start with aggregate queries, grouped by environment, service, route,
queue, project, org, status, or error facets as appropriate.
4. Fetch raw spans, logs, or traces only after aggregation identifies the
diff --git a/.agents/skills/datadog-query-recipes/references/export-freshness-lag.md b/.agents/skills/datadog-query-recipes/references/export-freshness-lag.md
new file mode 100644
index 0000000000..7b77156c80
--- /dev/null
+++ b/.agents/skills/datadog-query-recipes/references/export-freshness-lag.md
@@ -0,0 +1,43 @@
+# Export freshness lag
+
+Scheduled integrations (blob, PostHog, Mixpanel) emit a pooled distribution at
+the end of each run. Provenance: the shape from LFE-15853.
+
+## Metric
+
+- Name: `langfuse.export.freshness_lag_seconds`
+- Type: distribution
+- Value: `runStartTime − maxExportedTimestamp` (seconds, clamped at 0)
+- Watermark: `lastSyncAt` after a successful window (`maxTimestamp` just
+ written). Failing runs use the **unchanged** watermark so stall shows as
+ rising lag. Do not treat intended window end as the watermark.
+- Tags: `integration` (`blob_storage` / `posthog` / `mixpanel`), `window`
+ (`20m` / `1h` / `1d` / `1w`), `status` (`success` / `failure`), `unit:seconds`.
+ `env` comes from the agent. **No `project_id`.**
+- PostHog and Mixpanel are tagged `window:1h` (hourly cron). Blob uses the
+ configured cadence.
+
+## Baseline P95 (both Datadog sites)
+
+EU (`prod-eu`) and US (`prod-us`, `prod-hipaa`, `prod-jp`):
+
+```text
+p95:langfuse.export.freshness_lag_seconds{*} by {integration,window,env}
+```
+
+Split success vs failure while baselining:
+
+```text
+p95:langfuse.export.freshness_lag_seconds{*} by {integration,window,status}
+```
+
+Catch-up (historic backfill) sits in the tail. P95 is the planned SLI **if**
+those runs stay under ~5% of pooled volume. If they do not, filter runs whose
+target window is older than N× the cadence (follow-up, not this metric).
+
+## Emit sites
+
+- `worker/src/features/blobstorage/handleBlobStorageIntegrationProjectJob.ts`
+- `worker/src/features/posthog/handlePostHogIntegrationProjectJob.ts`
+- `worker/src/features/mixpanel/handleMixpanelIntegrationProjectJob.ts`
+- Helper: `worker/src/services/exportFreshnessLagMetric.ts`
diff --git a/worker/src/__tests__/blobStorageIntegrationProcessing.test.ts b/worker/src/__tests__/blobStorageIntegrationProcessing.test.ts
index cb9c07bfa4..275693fb93 100644
--- a/worker/src/__tests__/blobStorageIntegrationProcessing.test.ts
+++ b/worker/src/__tests__/blobStorageIntegrationProcessing.test.ts
@@ -15,10 +15,11 @@ const originalCloudRegion = vi.hoisted(() => {
return cloudRegion;
});
-// Override recordIncrement + recordHistogram so the attempt counter and
-// per-stage timing metrics are assertable.
+// Override recordIncrement + recordHistogram + recordDistribution so the
+// attempt counter, per-stage timing, and freshness-lag metrics are assertable.
const mockRecordIncrement = vi.hoisted(() => vi.fn());
const mockRecordHistogram = vi.hoisted(() => vi.fn());
+const mockRecordDistribution = vi.hoisted(() => vi.fn());
// Stubbable endpoint preflight — the customer-fault tests reject it to drive an
// in-try failure without infra. Defaults to the real impl for other tests.
const mockValidateBlobStorageEndpoint = vi.hoisted(() => vi.fn());
@@ -34,6 +35,7 @@ vi.mock("@langfuse/shared/src/server", async (importOriginal) => {
...actual,
recordIncrement: mockRecordIncrement,
recordHistogram: mockRecordHistogram,
+ recordDistribution: mockRecordDistribution,
validateBlobStorageEndpoint: mockValidateBlobStorageEndpoint,
};
});
@@ -54,7 +56,9 @@ import {
createEventsCh,
StorageService,
StorageServiceFactory,
+ BlobStorageIntegrationProcessingQueue,
} from "@langfuse/shared/src/server";
+import { EXPORT_FRESHNESS_LAG_METRIC } from "../services/exportFreshnessLagMetric";
import { prisma } from "@langfuse/shared/src/db";
import { Job } from "bullmq";
import {
@@ -1838,6 +1842,78 @@ describe("BlobStorageIntegrationProcessingJob", () => {
expect(timeDiff).toBeLessThan(5000); // Within 5 seconds
});
+ it("records freshness success when catch-up enqueue fails after the watermark advances", async () => {
+ const { projectId } = await createOrgProjectAndApiKey();
+ s3Prefix = projectId;
+ const now = new Date();
+ const twoDaysAgo = new Date(now.getTime() - 2 * 24 * 60 * 60 * 1000);
+
+ const trace = createTrace({
+ project_id: projectId,
+ timestamp: twoDaysAgo.getTime(),
+ name: "Old Trace",
+ });
+ await createTracesCh([trace]);
+
+ await prisma.blobStorageIntegration.create({
+ data: {
+ projectId,
+ type: BlobStorageIntegrationType.S3,
+ bucketName,
+ prefix: s3Prefix,
+ accessKeyId: minioAccessKeyId,
+ secretAccessKey: encrypt(minioAccessKeySecret),
+ region: region ? region : "auto",
+ endpoint: minioEndpoint,
+ forcePathStyle:
+ env.LANGFUSE_S3_EVENT_UPLOAD_FORCE_PATH_STYLE === "true",
+ enabled: true,
+ exportFrequency: "hourly",
+ lastSyncAt: twoDaysAgo,
+ compressed: false,
+ },
+ });
+
+ const add = vi.fn().mockRejectedValue(new Error("redis down"));
+ const getInstanceSpy = vi
+ .spyOn(BlobStorageIntegrationProcessingQueue, "getInstance")
+ .mockReturnValue({ add } as never);
+ mockRecordDistribution.mockClear();
+
+ try {
+ await expect(
+ handleBlobStorageIntegrationProjectJob({
+ data: { payload: { projectId } },
+ } as Job),
+ ).rejects.toThrow(/redis down/);
+
+ const updatedIntegration =
+ await prisma.blobStorageIntegration.findUnique({
+ where: { projectId },
+ });
+ expect(updatedIntegration?.lastSyncAt?.getTime()).toBe(
+ twoDaysAgo.getTime() + 60 * 60 * 1000,
+ );
+ expect(add).toHaveBeenCalled();
+ expect(mockRecordDistribution).toHaveBeenCalledWith(
+ EXPORT_FRESHNESS_LAG_METRIC,
+ expect.any(Number),
+ expect.objectContaining({
+ integration: "blob_storage",
+ window: "1h",
+ status: "success",
+ }),
+ );
+ expect(mockRecordDistribution).not.toHaveBeenCalledWith(
+ EXPORT_FRESHNESS_LAG_METRIC,
+ expect.anything(),
+ expect.objectContaining({ status: "failure" }),
+ );
+ } finally {
+ getInstanceSpy.mockRestore();
+ }
+ });
+
it("should coalesce a sub-second remainder instead of emitting a colliding chunk", async () => {
// Regression for the silent object-key collision: a caught-up run whose
// full-interval chunk ends only a sub-second before the frontier used to
diff --git a/worker/src/__tests__/exportFreshnessLagMetric.test.ts b/worker/src/__tests__/exportFreshnessLagMetric.test.ts
new file mode 100644
index 0000000000..4957c1ac2f
--- /dev/null
+++ b/worker/src/__tests__/exportFreshnessLagMetric.test.ts
@@ -0,0 +1,112 @@
+import { describe, it, expect, vi, beforeEach } from "vitest";
+
+const recordDistribution = vi.fn();
+vi.mock("@langfuse/shared/src/server", () => ({
+ recordDistribution: (...args: unknown[]) => recordDistribution(...args),
+}));
+
+import {
+ recordExportFreshnessLag,
+ windowClassFromBlobFrequency,
+ EXPORT_FRESHNESS_LAG_METRIC,
+} from "../services/exportFreshnessLagMetric";
+
+describe("recordExportFreshnessLag", () => {
+ const runStartTime = new Date("2026-09-08T12:00:00.000Z");
+
+ beforeEach(() => recordDistribution.mockClear());
+
+ it("emits runStart − watermark in seconds as a distribution, without project_id", () => {
+ recordExportFreshnessLag({
+ integration: "blob_storage",
+ window: "20m",
+ status: "success",
+ runStartTime,
+ maxExportedTimestamp: new Date("2026-09-08T11:40:00.000Z"),
+ });
+
+ expect(recordDistribution).toHaveBeenCalledWith(
+ EXPORT_FRESHNESS_LAG_METRIC,
+ 1200,
+ {
+ integration: "blob_storage",
+ window: "20m",
+ status: "success",
+ unit: "seconds",
+ },
+ );
+ expect(recordDistribution.mock.calls[0][2]).not.toHaveProperty(
+ "project_id",
+ );
+ expect(recordDistribution.mock.calls[0][2]).not.toHaveProperty("projectId");
+ });
+
+ it("emits failing runs against the unchanged watermark so stall shows as rising lag", () => {
+ recordExportFreshnessLag({
+ integration: "posthog",
+ window: "1h",
+ status: "failure",
+ runStartTime,
+ maxExportedTimestamp: new Date("2026-09-08T06:00:00.000Z"),
+ });
+
+ expect(recordDistribution).toHaveBeenCalledWith(
+ EXPORT_FRESHNESS_LAG_METRIC,
+ 6 * 60 * 60,
+ {
+ integration: "posthog",
+ window: "1h",
+ status: "failure",
+ unit: "seconds",
+ },
+ );
+ });
+
+ it("skips when there is no watermark yet", () => {
+ recordExportFreshnessLag({
+ integration: "mixpanel",
+ window: "1h",
+ status: "failure",
+ runStartTime,
+ maxExportedTimestamp: null,
+ });
+ recordExportFreshnessLag({
+ integration: "mixpanel",
+ window: "1h",
+ status: "success",
+ runStartTime,
+ maxExportedTimestamp: undefined,
+ });
+ expect(recordDistribution).not.toHaveBeenCalled();
+ });
+
+ it("clamps a watermark ahead of run start to zero", () => {
+ recordExportFreshnessLag({
+ integration: "blob_storage",
+ window: "1d",
+ status: "success",
+ runStartTime,
+ maxExportedTimestamp: new Date("2026-09-08T12:05:00.000Z"),
+ });
+ expect(recordDistribution).toHaveBeenCalledWith(
+ EXPORT_FRESHNESS_LAG_METRIC,
+ 0,
+ expect.objectContaining({ status: "success", window: "1d" }),
+ );
+ });
+});
+
+describe("windowClassFromBlobFrequency", () => {
+ it("maps configured blob cadences to window classes", () => {
+ expect(windowClassFromBlobFrequency("every_20_minutes")).toBe("20m");
+ expect(windowClassFromBlobFrequency("hourly")).toBe("1h");
+ expect(windowClassFromBlobFrequency("daily")).toBe("1d");
+ expect(windowClassFromBlobFrequency("weekly")).toBe("1w");
+ });
+
+ it("rejects an unknown frequency", () => {
+ expect(() => windowClassFromBlobFrequency("monthly")).toThrow(
+ /Unsupported export frequency/,
+ );
+ });
+});
diff --git a/worker/src/__tests__/mixpanelIntegrationProjectJob.test.ts b/worker/src/__tests__/mixpanelIntegrationProjectJob.test.ts
index a631f40d57..5cd10db86a 100644
--- a/worker/src/__tests__/mixpanelIntegrationProjectJob.test.ts
+++ b/worker/src/__tests__/mixpanelIntegrationProjectJob.test.ts
@@ -128,6 +128,7 @@ vi.mock("@langfuse/shared/src/server", () => ({
QueueName: { MixpanelIntegrationProcessingQueue: "mixpanel" },
logger: { info: vi.fn(), warn: vi.fn(), error: vi.fn(), debug: vi.fn() },
recordIncrement: vi.fn(),
+ recordDistribution: vi.fn(),
getCurrentSpan: vi.fn(() => undefined),
getTracesForAnalyticsIntegrations: vi.fn(() => h.fakeStream("traces")),
getGenerationsForAnalyticsIntegrations: vi.fn(() =>
@@ -145,7 +146,10 @@ import {
import {
getScoresForAnalyticsIntegrations,
recordIncrement,
+ recordDistribution,
} from "@langfuse/shared/src/server";
+import { decrypt } from "@langfuse/shared/encryption";
+import { EXPORT_FRESHNESS_LAG_METRIC } from "../services/exportFreshnessLagMetric";
import { env } from "../env";
// Mirrors the real queue's defaultJobOptions (mixpanelIntegrationProcessingQueue.ts).
@@ -362,4 +366,26 @@ describe("handleMixpanelIntegrationProjectJob customer-fault observability", ()
expect(recordIncrement).not.toHaveBeenCalled();
});
+
+ it("emits freshness failure when decrypt throws before export work starts", async () => {
+ vi.mocked(decrypt).mockImplementationOnce(() => {
+ throw new Error("bad token");
+ });
+ vi.mocked(recordDistribution).mockClear();
+
+ await expect(
+ handleMixpanelIntegrationProjectJob(makeJob()),
+ ).rejects.toThrow(/bad token/);
+
+ expect(recordDistribution).toHaveBeenCalledWith(
+ EXPORT_FRESHNESS_LAG_METRIC,
+ expect.any(Number),
+ expect.objectContaining({
+ integration: "mixpanel",
+ window: "1h",
+ status: "failure",
+ }),
+ );
+ expect(h.mixpanelIntegrationUpdate).not.toHaveBeenCalled();
+ });
});
diff --git a/worker/src/__tests__/posthogIntegrationProjectJob.test.ts b/worker/src/__tests__/posthogIntegrationProjectJob.test.ts
index e325984abb..104514807b 100644
--- a/worker/src/__tests__/posthogIntegrationProjectJob.test.ts
+++ b/worker/src/__tests__/posthogIntegrationProjectJob.test.ts
@@ -92,6 +92,7 @@ const h = vi.hoisted(() => {
// hostname) and the customer notification, both controllable per test.
const validateWebhookURL = vi.fn(async () => {});
const recordIncrement = vi.fn();
+ const recordDistribution = vi.fn();
const dispatchProjectNotification = vi.fn(async () => {
await new Promise((resolve) => setTimeout(resolve, 0));
try {
@@ -143,6 +144,7 @@ const h = vi.hoisted(() => {
notification,
validateWebhookURL,
recordIncrement,
+ recordDistribution,
dispatchProjectNotification,
OutboundUrlValidationError,
getTraces,
@@ -174,6 +176,7 @@ vi.mock("@langfuse/shared/src/server", () => ({
QueueName: { PostHogIntegrationProcessingQueue: "posthog" },
logger: { info: vi.fn(), warn: vi.fn(), error: vi.fn(), debug: vi.fn() },
recordIncrement: h.recordIncrement,
+ recordDistribution: h.recordDistribution,
getCurrentSpan: vi.fn(() => undefined),
validateWebhookURL: h.validateWebhookURL,
dispatchProjectNotification: h.dispatchProjectNotification,
@@ -277,6 +280,7 @@ function resetSharedState() {
h.notification.settled = false;
h.notification.rejects = false;
h.recordIncrement.mockClear();
+ h.recordDistribution.mockClear();
h.validateWebhookURL.mockReset();
h.validateWebhookURL.mockImplementation(async () => {});
h.dispatchProjectNotification.mockClear();
diff --git a/worker/src/features/blobstorage/handleBlobStorageIntegrationProjectJob.ts b/worker/src/features/blobstorage/handleBlobStorageIntegrationProjectJob.ts
index ae4d8eab25..d72d24727e 100644
--- a/worker/src/features/blobstorage/handleBlobStorageIntegrationProjectJob.ts
+++ b/worker/src/features/blobstorage/handleBlobStorageIntegrationProjectJob.ts
@@ -75,6 +75,10 @@ import { SpanKind } from "@opentelemetry/api";
import { env } from "../../env";
import { assertExportSourceWritable } from "../exportWriteModeGuard";
import { recordExportVolume } from "../../services/exportVolumeMetric";
+import {
+ recordExportFreshnessLag,
+ windowClassFromBlobFrequency,
+} from "../../services/exportFreshnessLagMetric";
import {
buildBlobExportManifest,
buildBlobExportManifestKey,
@@ -1216,9 +1220,10 @@ export const handleBlobStorageIntegrationProjectJob = async (
return;
}
+ const runStartTime = new Date();
const { count: claimed } = await prisma.blobStorageIntegration.updateMany({
where: { projectId },
- data: { runStartedAt: new Date() },
+ data: { runStartedAt: runStartTime },
});
if (claimed === 0) {
logger.info(
@@ -1275,6 +1280,15 @@ export const handleBlobStorageIntegrationProjectJob = async (
lastErrorAt: null,
},
});
+ recordExportFreshnessLag({
+ integration: "blob_storage",
+ window: windowClassFromBlobFrequency(
+ blobStorageIntegration.exportFrequency,
+ ),
+ status: "success",
+ runStartTime,
+ maxExportedTimestamp: blobStorageIntegration.lastSyncAt,
+ });
return;
}
@@ -1282,6 +1296,7 @@ export const handleBlobStorageIntegrationProjectJob = async (
// self-hosted), so the deprecation notice below is Cloud-only too.
const isCloud = Boolean(env.NEXT_PUBLIC_LANGFUSE_CLOUD_REGION);
+ let watermarkAdvanced = false;
try {
// The catch persists lastError and notifies admins.
assertExportSourceWritable(
@@ -1510,6 +1525,20 @@ export const handleBlobStorageIntegrationProjectJob = async (
return;
}
+ // The export watermark is committed. Catch-up enqueue below can still
+ // fail (Redis); that must not be recorded as an export-freshness failure
+ // against the pre-run lastSyncAt.
+ recordExportFreshnessLag({
+ integration: "blob_storage",
+ window: windowClassFromBlobFrequency(
+ blobStorageIntegration.exportFrequency,
+ ),
+ status: "success",
+ runStartTime,
+ maxExportedTimestamp: maxTimestamp,
+ });
+ watermarkAdvanced = true;
+
// If still catching up, immediately queue the next chunk job
if (!caughtUp) {
const queue = BlobStorageIntegrationProcessingQueue.getInstance();
@@ -1552,6 +1581,18 @@ export const handleBlobStorageIntegrationProjectJob = async (
return; // obsolete job: complete it rather than fail it
}
+ if (!watermarkAdvanced) {
+ recordExportFreshnessLag({
+ integration: "blob_storage",
+ window: windowClassFromBlobFrequency(
+ blobStorageIntegration.exportFrequency,
+ ),
+ status: "failure",
+ runStartTime,
+ maxExportedTimestamp: blobStorageIntegration.lastSyncAt,
+ });
+ }
+
switch (outcome.kind) {
case "disabled-by-us":
// Awaited: the integration is now off, so the scheduler never revisits
diff --git a/worker/src/features/mixpanel/handleMixpanelIntegrationProjectJob.ts b/worker/src/features/mixpanel/handleMixpanelIntegrationProjectJob.ts
index 38dcf1b1be..9de828c236 100644
--- a/worker/src/features/mixpanel/handleMixpanelIntegrationProjectJob.ts
+++ b/worker/src/features/mixpanel/handleMixpanelIntegrationProjectJob.ts
@@ -14,6 +14,7 @@ import {
import { decrypt } from "@langfuse/shared/encryption";
import { MixpanelClient } from "./mixpanelClient";
import { recordExportVolume } from "../../services/exportVolumeMetric";
+import { recordExportFreshnessLag } from "../../services/exportFreshnessLagMetric";
import {
transformTraceForMixpanel,
transformGenerationForMixpanel,
@@ -249,21 +250,23 @@ export const handleMixpanelIntegrationProjectJob = async (
return;
}
- // Fetch relevant data and send it to Mixpanel
- const executionConfig: MixpanelExecutionConfig = {
- projectId,
- projectName: mixpanelIntegration.project.name,
- // Start from 2000-01-01 if no lastSyncAt. Workaround because 1970-01-01 leads to subtle bugs in ClickHouse
- minTimestamp: mixpanelIntegration.lastSyncAt || new Date("2000-01-01"),
- maxTimestamp: new Date(new Date().getTime() - 30 * 60 * 1000), // 30 minutes ago
- decryptedMixpanelProjectToken: decrypt(
- mixpanelIntegration.encryptedMixpanelProjectToken,
- ),
- mixpanelRegion: mixpanelIntegration.mixpanelRegion,
- useGraceHash: job.attemptsMade > 0,
- };
+ const runStartTime = new Date();
try {
+ // Fetch relevant data and send it to Mixpanel
+ const executionConfig: MixpanelExecutionConfig = {
+ projectId,
+ projectName: mixpanelIntegration.project.name,
+ // Start from 2000-01-01 if no lastSyncAt. Workaround because 1970-01-01 leads to subtle bugs in ClickHouse
+ minTimestamp: mixpanelIntegration.lastSyncAt || new Date("2000-01-01"),
+ maxTimestamp: new Date(new Date().getTime() - 30 * 60 * 1000), // 30 minutes ago
+ decryptedMixpanelProjectToken: decrypt(
+ mixpanelIntegration.encryptedMixpanelProjectToken,
+ ),
+ mixpanelRegion: mixpanelIntegration.mixpanelRegion,
+ useGraceHash: job.attemptsMade > 0,
+ };
+
// Fail loudly before exporting empty data and advancing lastSyncAt
// (LFE-10148, LFE-11009); the catch below logs and BullMQ retries.
assertExportSourceWritable(
@@ -314,10 +317,24 @@ export const handleMixpanelIntegrationProjectJob = async (
bytes: mixpanel.getSerializedBytes(),
projectId,
});
+ recordExportFreshnessLag({
+ integration: "mixpanel",
+ window: "1h",
+ status: "success",
+ runStartTime,
+ maxExportedTimestamp: executionConfig.maxTimestamp,
+ });
logger.info(
`[MIXPANEL] Mixpanel integration processing complete for project ${projectId}`,
);
} catch (error) {
+ recordExportFreshnessLag({
+ integration: "mixpanel",
+ window: "1h",
+ status: "failure",
+ runStartTime,
+ maxExportedTimestamp: mixpanelIntegration.lastSyncAt,
+ });
const mixpanelFaultReason = classifyCustomerFault(error);
if (mixpanelFaultReason !== undefined) {
recordIncrement(MIXPANEL_INTEGRATION_CUSTOMER_FAULT_METRIC, 1, {
diff --git a/worker/src/features/posthog/handlePostHogIntegrationProjectJob.ts b/worker/src/features/posthog/handlePostHogIntegrationProjectJob.ts
index 075705b29e..8c8c966c5a 100644
--- a/worker/src/features/posthog/handlePostHogIntegrationProjectJob.ts
+++ b/worker/src/features/posthog/handlePostHogIntegrationProjectJob.ts
@@ -28,6 +28,7 @@ import {
import { decrypt } from "@langfuse/shared/encryption";
import { PostHog } from "posthog-node";
import { recordExportVolume } from "../../services/exportVolumeMetric";
+import { recordExportFreshnessLag } from "../../services/exportFreshnessLagMetric";
import { assertExportSourceWritable } from "../exportWriteModeGuard";
import { classifyCustomerFault } from "../integrations/customerFaultClassification";
import { isRecordNotFoundError } from "../integrations/prismaErrors";
@@ -284,6 +285,8 @@ export const handlePostHogIntegrationProjectJob = async (
return;
}
+ const runStartTime = new Date();
+
try {
// Validate PostHog hostname to prevent SSRF attacks before sending data.
// Rewrap preserving { cause } so the single catch below can classify the
@@ -327,6 +330,13 @@ export const handlePostHogIntegrationProjectJob = async (
logger.info(
`[POSTHOG] Skipping PostHog integration for project ${projectId}: empty sync window (min: ${minTimestamp.toISOString()}, max: ${maxTimestamp.toISOString()})`,
);
+ recordExportFreshnessLag({
+ integration: "posthog",
+ window: "1h",
+ status: "success",
+ runStartTime,
+ maxExportedTimestamp: postHogIntegration.lastSyncAt,
+ });
return;
}
@@ -425,6 +435,13 @@ export const handlePostHogIntegrationProjectJob = async (
bytes: executionConfig.volume.bytes,
projectId,
});
+ recordExportFreshnessLag({
+ integration: "posthog",
+ window: "1h",
+ status: "success",
+ runStartTime,
+ maxExportedTimestamp: executionConfig.maxTimestamp,
+ });
logger.info(
`[POSTHOG] PostHog integration processing complete for project ${projectId}`,
);
@@ -444,6 +461,14 @@ export const handlePostHogIntegrationProjectJob = async (
error instanceof Error ? error.message : String(error)
).slice(0, 1000);
+ recordExportFreshnessLag({
+ integration: "posthog",
+ window: "1h",
+ status: "failure",
+ runStartTime,
+ maxExportedTimestamp: postHogIntegration.lastSyncAt,
+ });
+
if (reason === undefined) {
// Defensive fallback: every code today either classifies above and
// disables (including one raised on a redirect hop, reached through the
diff --git a/worker/src/services/exportFreshnessLagMetric.ts b/worker/src/services/exportFreshnessLagMetric.ts
new file mode 100644
index 0000000000..740b2899ec
--- /dev/null
+++ b/worker/src/services/exportFreshnessLagMetric.ts
@@ -0,0 +1,62 @@
+import { recordDistribution } from "@langfuse/shared/src/server";
+
+export const EXPORT_FRESHNESS_LAG_METRIC =
+ "langfuse.export.freshness_lag_seconds";
+
+export type ExportFreshnessIntegration =
+ | "blob_storage"
+ | "posthog"
+ | "mixpanel";
+
+export type ExportFreshnessWindow = "20m" | "1h" | "1d" | "1w";
+
+export type ExportFreshnessStatus = "success" | "failure";
+
+const BLOB_FREQUENCY_TO_WINDOW: Record<string, ExportFreshnessWindow> = {
+ every_20_minutes: "20m",
+ hourly: "1h",
+ daily: "1d",
+ weekly: "1w",
+};
+
+export const windowClassFromBlobFrequency = (
+ frequency: string,
+): ExportFreshnessWindow => {
+ const window = BLOB_FREQUENCY_TO_WINDOW[frequency];
+ if (!window) {
+ throw new Error(`Unsupported export frequency: ${frequency}`);
+ }
+ return window;
+};
+
+/**
+ * Seconds the newest successfully-exported timestamp lags this run's start.
+ * On success, pass the watermark just written (this run's maxTimestamp).
+ * On failure, pass the unchanged lastSyncAt so lag climbs until recovery.
+ * First runs with no watermark yet are skipped.
+ */
+export const recordExportFreshnessLag = ({
+ integration,
+ window,
+ status,
+ runStartTime,
+ maxExportedTimestamp,
+}: {
+ integration: ExportFreshnessIntegration;
+ window: ExportFreshnessWindow;
+ status: ExportFreshnessStatus;
+ runStartTime: Date;
+ maxExportedTimestamp: Date | null | undefined;
+}): void => {
+ if (!maxExportedTimestamp) {
+ return;
+ }
+ const lagSeconds =
+ (runStartTime.getTime() - maxExportedTimestamp.getTime()) / 1000;
+ recordDistribution(EXPORT_FRESHNESS_LAG_METRIC, Math.max(0, lagSeconds), {
+ integration,
+ window,
+ status,
+ unit: "seconds",
+ });
+};