Skip to content

Infisical is the open-source platform for securing secrets, certificates, access, and AI Agents.

feat(metrics): implement local refresh metrics for monitoring task execution

Victor Hugo dos Santosfce1ca39a1f3
diff --git a/backend/CLAUDE.md b/backend/CLAUDE.md
index e863c2b402..da8c0f6107 100644
--- a/backend/CLAUDE.md
+++ b/backend/CLAUDE.md
@@ -630,7 +630,7 @@ Recurring work runs through the cron manager in `src/lib/cron/cron-job.ts` (`cro
**Only `general-workers` pods start the manager's timers**, so only they execute a scheduled handler; every pod still calls `register`. The separate `cronJobs` array in `src/server/routes/index.ts` is deliberately **not** gated: those refresh the process's own caches (license, rate limits, env overrides), not fleet work.
-**Per-process refreshes use `startLocalRefresh` (`src/lib/cron/local-refresh.ts`), never the cron manager and never a raw `new CronJob`.** The cron manager runs a job once per fleet, which would leave every other pod's in-memory state stale; a wall-clock cron pattern makes every pod fire in the same second. The helper starts each process at a random offset within one interval (random per process on purpose, unlike the manager's deterministic hash), skips a tick while the previous run is in flight, logs and swallows errors, and unrefs its timers. Its `stop()` handle is what goes in `cronJobs`.
+**Per-process refreshes use `startLocalRefresh` (`src/lib/cron/local-refresh.ts`), never the cron manager and never a raw `new CronJob`.** The cron manager runs a job once per fleet, which would leave every other pod's in-memory state stale; a wall-clock cron pattern makes every pod fire in the same second. The helper starts each process at a random offset within one interval (random per process on purpose, unlike the manager's deterministic hash), skips a tick while the previous run is in flight, logs and swallows errors, and unrefs its timers. Its `stop()` handle is what goes in `cronJobs`. It records `infisical.local_refresh.*` on the `InfisicalCore` meter, labelled by `name` as `job.name`, so a new refresh needs no metric code; keep `name` a fixed string. A failing refresh keeps retrying on every tick rather than giving up after N failures, because stopping would freeze the pod's state at whatever it last loaded. The `consecutive_failures` gauge (reset on success) is the alerting signal.
**Why this exists instead of BullMQ repeatables**: cron runs are coordinated across pods via a slot-election scheme (5 participant slots backed by Redis SET NX/PX) plus per-run redlocks, so each fire executes exactly once across the fleet without the orphaned-scheduler / duplicate-execution failure modes the BullMQ `JobScheduler` had. The manager also handles crash recovery via lease TTLs, hang recovery via per-handler timeouts, and bounded exponential backoff that won't overlap with the next scheduled fire.
diff --git a/backend/src/lib/cron/local-refresh.test.ts b/backend/src/lib/cron/local-refresh.test.ts
index f905c9114d..b0bec5a00f 100644
--- a/backend/src/lib/cron/local-refresh.test.ts
+++ b/backend/src/lib/cron/local-refresh.test.ts
@@ -1,6 +1,7 @@
import { afterEach, beforeEach, describe, expect, test, vi } from "vitest";
import { logger } from "@app/lib/logger";
+import { LocalRefreshOutcome, recordLocalRefreshRunMetric } from "@app/lib/telemetry/metrics";
import { startLocalRefresh, TLocalRefreshHandle } from "./local-refresh";
@@ -8,6 +9,13 @@ vi.mock("@app/lib/logger", () => ({
logger: { info: vi.fn(), warn: vi.fn(), error: vi.fn(), debug: vi.fn() }
}));
+// Stubbed outright: the real module reaches this file again through env -> crypto -> license-service,
+// so a partial mock built from importOriginal() hands local-refresh the unmocked helper.
+vi.mock("@app/lib/telemetry/metrics", () => ({
+ LocalRefreshOutcome: { COMPLETED: "completed", FAILED: "failed", SKIPPED: "skipped" },
+ recordLocalRefreshRunMetric: vi.fn()
+}));
+
const INTERVAL_MS = 5 * 60 * 1000;
const MAX_TIMER_DELAY_MS = 2 ** 31 - 1;
@@ -140,6 +148,52 @@ describe("startLocalRefresh", () => {
expect(logger.error).toHaveBeenCalledWith(expect.any(Error), expect.stringContaining("[name=flaky]"));
});
+ test("counts consecutive failures and resets the count on the next success", async () => {
+ mockRandom(0);
+ const boom = new Error("boom");
+ const task = vi
+ .fn()
+ .mockRejectedValueOnce(boom)
+ .mockRejectedValueOnce(boom)
+ .mockResolvedValueOnce(undefined)
+ .mockRejectedValueOnce(boom);
+
+ handle = startLocalRefresh({ name: "flaky", intervalMs: INTERVAL_MS, task });
+ await vi.advanceTimersByTimeAsync(0);
+ await vi.advanceTimersByTimeAsync(INTERVAL_MS * 3);
+
+ const recorded = vi.mocked(recordLocalRefreshRunMetric).mock.calls.map(([params]) => params);
+ expect(recorded.map(({ outcome, consecutiveFailures }) => [outcome, consecutiveFailures])).toEqual([
+ [LocalRefreshOutcome.FAILED, 1],
+ [LocalRefreshOutcome.FAILED, 2],
+ [LocalRefreshOutcome.COMPLETED, 0],
+ [LocalRefreshOutcome.FAILED, 1]
+ ]);
+ expect(recorded[0]).toEqual(expect.objectContaining({ name: "flaky", error: boom }));
+ expect(recorded[2].error).toBeUndefined();
+ expect(logger.error).toHaveBeenCalledWith(boom, expect.stringContaining("[consecutiveFailures=2]"));
+ });
+
+ test("records run duration, and a skipped tick without one", async () => {
+ mockRandom(0);
+ let release: () => void = () => {};
+ const task = vi.fn(async () => {
+ await new Promise<void>((resolve) => {
+ release = resolve;
+ });
+ });
+
+ handle = startLocalRefresh({ name: "slow", intervalMs: INTERVAL_MS, task });
+ await vi.advanceTimersByTimeAsync(INTERVAL_MS);
+ expect(recordLocalRefreshRunMetric).toHaveBeenCalledWith({ name: "slow", outcome: LocalRefreshOutcome.SKIPPED });
+
+ release();
+ await vi.advanceTimersByTimeAsync(0);
+ expect(recordLocalRefreshRunMetric).toHaveBeenLastCalledWith(
+ expect.objectContaining({ name: "slow", outcome: LocalRefreshOutcome.COMPLETED, durationMs: INTERVAL_MS })
+ );
+ });
+
test("logs duration at debug and warns only above the threshold", async () => {
mockRandom(0);
const task = vi.fn(async () => {
diff --git a/backend/src/lib/cron/local-refresh.ts b/backend/src/lib/cron/local-refresh.ts
index 8bd7583eec..671493b783 100644
--- a/backend/src/lib/cron/local-refresh.ts
+++ b/backend/src/lib/cron/local-refresh.ts
@@ -1,4 +1,5 @@
import { logger } from "@app/lib/logger";
+import { LocalRefreshOutcome, recordLocalRefreshRunMetric } from "@app/lib/telemetry/metrics";
export type TLocalRefreshHandle = {
stop: () => void;
@@ -33,24 +34,33 @@ export const startLocalRefresh = ({
let stopped = false;
let running = false;
+ let consecutiveFailures = 0;
let timer: NodeJS.Timeout | undefined;
const run = async () => {
if (stopped) return;
if (running) {
logger.warn(`Local refresh skipped, previous run still in progress [name=${name}]`);
+ recordLocalRefreshRunMetric({ name, outcome: LocalRefreshOutcome.SKIPPED });
return;
}
running = true;
const startedAt = Date.now();
+ let outcome = LocalRefreshOutcome.COMPLETED;
+ let failure: unknown;
try {
await task();
+ consecutiveFailures = 0;
} catch (error) {
- logger.error(error, `Local refresh failed [name=${name}]`);
+ outcome = LocalRefreshOutcome.FAILED;
+ failure = error;
+ consecutiveFailures += 1;
+ logger.error(error, `Local refresh failed [name=${name}] [consecutiveFailures=${consecutiveFailures}]`);
} finally {
running = false;
const durationMs = Date.now() - startedAt;
+ recordLocalRefreshRunMetric({ name, outcome, durationMs, error: failure, consecutiveFailures });
logger.debug(
{ name, duration_ms: durationMs },
`Local refresh finished [name=${name}] [durationMs=${durationMs}]`
diff --git a/backend/src/lib/telemetry/metrics.test.ts b/backend/src/lib/telemetry/metrics.test.ts
index 9323ee39c5..5c0c359af4 100644
--- a/backend/src/lib/telemetry/metrics.test.ts
+++ b/backend/src/lib/telemetry/metrics.test.ts
@@ -1,6 +1,25 @@
import opentelemetry from "@opentelemetry/api";
-
-import { highCardinalityMeter, normalizeHttpMethod, shouldRecordHighCardinalityMetrics } from "./metrics";
+import {
+ AggregationTemporality,
+ createAllowListAttributesProcessor,
+ type DataPoint,
+ InMemoryMetricExporter,
+ MeterProvider,
+ PeriodicExportingMetricReader
+} from "@opentelemetry/sdk-metrics";
+import type { Knex } from "knex";
+
+import { BadRequestError } from "@app/lib/errors";
+
+import {
+ highCardinalityMeter,
+ LocalRefreshOutcome,
+ normalizeHttpMethod,
+ recordLocalRefreshRunMetric,
+ registerInfrastructureMetrics,
+ shouldRecordHighCardinalityMetrics
+} from "./metrics";
+import { INFISICAL_CORE_METER_ATTRIBUTES } from "./telemetry-attributes";
const mockConfig = {
OTEL_TELEMETRY_COLLECTION_ENABLED: true,
@@ -128,3 +147,95 @@ describe("normalizeHttpMethod", () => {
expect(normalizeHttpMethod("")).toBe("GET");
});
});
+
+describe("local refresh metrics", () => {
+ let reader: PeriodicExportingMetricReader;
+
+ // Instruments bind to the provider on first use and never rebind, so the provider is installed once.
+ beforeAll(() => {
+ opentelemetry.metrics.disable();
+ reader = new PeriodicExportingMetricReader({
+ exporter: new InMemoryMetricExporter(AggregationTemporality.CUMULATIVE),
+ exportIntervalMillis: 60_000
+ });
+ opentelemetry.metrics.setGlobalMeterProvider(
+ new MeterProvider({
+ readers: [reader],
+ views: [
+ {
+ meterName: "InfisicalCore",
+ attributesProcessors: [createAllowListAttributesProcessor(INFISICAL_CORE_METER_ATTRIBUTES)]
+ }
+ ]
+ })
+ );
+ registerInfrastructureMetrics({ client: {} } as unknown as Knex);
+ });
+
+ afterAll(() => {
+ opentelemetry.metrics.disable();
+ });
+
+ beforeEach(() => {
+ mockConfig.OTEL_TELEMETRY_COLLECTION_ENABLED = true;
+ });
+
+ const collectPoints = async (metricName: string) => {
+ const { resourceMetrics } = await reader.collect();
+ return resourceMetrics.scopeMetrics
+ .flatMap((scope) => scope.metrics)
+ .filter((metric) => metric.descriptor.name === metricName)
+ .flatMap((metric) => metric.dataPoints as DataPoint<number>[]);
+ };
+
+ test("labels failed runs with the classified error type and keeps the labels through the allowlist", async () => {
+ recordLocalRefreshRunMetric({
+ name: "labels",
+ outcome: LocalRefreshOutcome.FAILED,
+ durationMs: 1500,
+ error: new BadRequestError({ message: "bad" }),
+ consecutiveFailures: 1
+ });
+ recordLocalRefreshRunMetric({ name: "labels", outcome: LocalRefreshOutcome.SKIPPED });
+
+ const runs = (await collectPoints("infisical.local_refresh.run.count")).filter(
+ (point) => point.attributes["job.name"] === "labels"
+ );
+ expect(runs.map((point) => point.attributes)).toEqual(
+ expect.arrayContaining([
+ { "job.name": "labels", outcome: "failed", "error.type": "validation" },
+ { "job.name": "labels", outcome: "skipped" }
+ ])
+ );
+
+ const durations = (await collectPoints("infisical.local_refresh.run.duration")).filter(
+ (point) => point.attributes["job.name"] === "labels"
+ );
+ expect(durations).toHaveLength(1);
+ expect((durations[0].value as unknown as { sum: number }).sum).toBe(1.5);
+ });
+
+ test("the consecutive failure gauge heals on success", async () => {
+ const gaugeValue = async () =>
+ (await collectPoints("infisical.local_refresh.consecutive_failures")).find(
+ (point) => point.attributes["job.name"] === "gauge"
+ )?.value;
+
+ recordLocalRefreshRunMetric({ name: "gauge", outcome: LocalRefreshOutcome.FAILED, consecutiveFailures: 3 });
+ expect(await gaugeValue()).toBe(3);
+
+ recordLocalRefreshRunMetric({ name: "gauge", outcome: LocalRefreshOutcome.COMPLETED, consecutiveFailures: 0 });
+ expect(await gaugeValue()).toBe(0);
+ });
+
+ test("records nothing while telemetry is disabled", async () => {
+ mockConfig.OTEL_TELEMETRY_COLLECTION_ENABLED = false;
+ recordLocalRefreshRunMetric({ name: "disabled", outcome: LocalRefreshOutcome.FAILED, consecutiveFailures: 1 });
+ mockConfig.OTEL_TELEMETRY_COLLECTION_ENABLED = true;
+
+ const runs = (await collectPoints("infisical.local_refresh.run.count")).filter(
+ (point) => point.attributes["job.name"] === "disabled"
+ );
+ expect(runs).toHaveLength(0);
+ });
+});
diff --git a/backend/src/lib/telemetry/metrics.ts b/backend/src/lib/telemetry/metrics.ts
index bf26cb3ce3..94e15e3d8f 100644
--- a/backend/src/lib/telemetry/metrics.ts
+++ b/backend/src/lib/telemetry/metrics.ts
@@ -309,6 +309,54 @@ export const queueStalledCounter = infisicalCoreMeter.createCounter("infisical.q
unit: "{job}"
});
+// In-process refresh metrics. Wired in lib/cron/local-refresh.ts around every run. Each pod runs its own
+// copy, so a failure here means that pod is serving stale in-memory state, not that a fleet job was lost.
+export enum LocalRefreshOutcome {
+ COMPLETED = "completed",
+ FAILED = "failed",
+ SKIPPED = "skipped"
+}
+
+export const localRefreshRunCounter = infisicalCoreMeter.createCounter("infisical.local_refresh.run.count", {
+ description:
+ "In-process refresh runs by job and outcome. skipped means the previous run was still in progress when the next tick fired.",
+ unit: "{run}"
+});
+
+export const localRefreshRunDurationHistogram = infisicalCoreMeter.createHistogram(
+ "infisical.local_refresh.run.duration",
+ {
+ description: "In-process refresh run duration by job and outcome. Not recorded for skipped runs.",
+ unit: "s"
+ }
+);
+
+// Read by the infisical.local_refresh.consecutive_failures gauge in registerInfrastructureMetrics().
+const localRefreshConsecutiveFailures = new Map<string, number>();
+
+export const recordLocalRefreshRunMetric = (params: {
+ name: string;
+ outcome: LocalRefreshOutcome;
+ durationMs?: number;
+ error?: unknown;
+ consecutiveFailures?: number;
+}) => {
+ safely(() => {
+ if (params.consecutiveFailures !== undefined) {
+ localRefreshConsecutiveFailures.set(params.name, params.consecutiveFailures);
+ }
+ if (!isTelemetryEnabled()) return;
+ const attributes: Record<string, string> = { "job.name": params.name, outcome: params.outcome };
+ if (params.durationMs !== undefined) {
+ localRefreshRunDurationHistogram.record(params.durationMs / 1000, attributes);
+ }
+ if (params.outcome === LocalRefreshOutcome.FAILED) {
+ attributes["error.type"] = classifyError(params.error);
+ }
+ localRefreshRunCounter.add(1, attributes);
+ });
+};
+
// Audit log lifecycle metrics. Wired in audit-log-queue.ts: enqueued when an event is appended to
// the Redis ingest stream, dropped when the request-path push fails, persist duration around the
// batch insert in the unified consumer.
@@ -899,6 +947,20 @@ export const registerInfrastructureMetrics = (db: Knex) => {
const { size, max } = getAgentPoolStats();
result.observe(size, { "pool.max": String(max) });
});
+
+ // Resets to 0 on the next successful run, so a sustained non-zero value means the refresh is stuck
+ // failing rather than flaking. Alert on this instead of on the failure rate.
+ const localRefreshFailuresGauge = meter.createObservableGauge("infisical.local_refresh.consecutive_failures", {
+ description: "Consecutive failed runs of each in-process refresh on this pod. Resets to 0 on success.",
+ unit: "{run}"
+ });
+
+ localRefreshFailuresGauge.addCallback((result) => {
+ if (!isTelemetryEnabled()) return;
+ localRefreshConsecutiveFailures.forEach((count, name) => {
+ result.observe(count, { "job.name": name });
+ });
+ });
};
// -- Legacy root-key usage (InfisicalCore meter) -----------------------------------------------------
diff --git a/docs/self-hosting/guides/monitoring-telemetry.mdx b/docs/self-hosting/guides/monitoring-telemetry.mdx
index e705b3a2f7..dae7607f3c 100644
--- a/docs/self-hosting/guides/monitoring-telemetry.mdx
+++ b/docs/self-hosting/guides/monitoring-telemetry.mdx
@@ -344,7 +344,7 @@ Replace `otel-collector:8889` with the actual hostname and port where your OpenT
Infisical emits metrics on two OpenTelemetry meters simultaneously. Choose which to scrape based on your deployment scale.
- **High-cardinality, per-actor meters:** the `Infisical` and `API` meters' original metrics (`infisical.http.server.request.*`, `infisical.http.server.error.count`, `infisical.secret.read.count`, `infisical.auth.attempt.count`, `infisical.kmip.operation.count`) include per-actor labels such as `user.email`, `identity.name`, `client.address`, `user_agent.original`, `organization.name`, `project.name`, `secret.path`, and `secret.name`. The `SecretSyncs`, `PkiSyncs`, and `Integrations` meters likewise carry unbounded labels such as `syncId`. Useful for self-hosted deployments where you want per-user visibility directly in Grafana. May become expensive at large scale (many users, identities, or IPs) due to label cardinality.
-- **`InfisicalCore` meter (bounded-cardinality):** all newer metrics (queue, audit log, product analytics aggregation, alert dispatch, event outbox, permission cache, secret cache, rate limit, build info, `infisical.core.http.request.count`, `infisical.core.http.error.count`, authentication latency, token renewal, SSO config changes, SCIM provisioning, and database connection pool) use **only bounded enums and static route templates** as labels (e.g. HTTP method, parameterized `http.route`, error type, outcome). A strict allowlist drops every other attribute at the SDK level, so **no tenant or actor identifiers are emitted**; no organization ID, project IDs, names, emails, IPs, or user agents. This keeps the series count independent of how many organizations, users, or identities you have. Per-org and per-actor detail is available in audit logs instead.
+- **`InfisicalCore` meter (bounded-cardinality):** all newer metrics (queue, local refresh, audit log, product analytics aggregation, alert dispatch, event outbox, permission cache, secret cache, rate limit, build info, `infisical.core.http.request.count`, `infisical.core.http.error.count`, authentication latency, token renewal, SSO config changes, SCIM provisioning, and database connection pool) use **only bounded enums and static route templates** as labels (e.g. HTTP method, parameterized `http.route`, error type, outcome). A strict allowlist drops every other attribute at the SDK level, so **no tenant or actor identifiers are emitted**; no organization ID, project IDs, names, emails, IPs, or user agents. This keeps the series count independent of how many organizations, users, or identities you have. Per-org and per-actor detail is available in audit logs instead.
All meters emit at the same time. If you're a self-hosted instance and find the per-actor labels useful, keep using the high-cardinality metrics. For larger or multi-tenant deployments, scrape only the `InfisicalCore` metrics to keep cardinality under control.
@@ -809,6 +809,51 @@ These metrics give per-queue visibility into BullMQ worker health: throughput, l
</Accordion>
</AccordionGroup>
+### Local refresh metrics (InfisicalCore)
+
+Every Infisical instance runs a few background tasks on a timer to refresh settings it keeps in memory, such as the license, rate limits, and SSO configuration. Each instance runs its own copy, so a failing refresh means that instance is serving stale settings while the others may be fine. These metrics are labelled by task, and each instance reports its own values.
+
+<AccordionGroup>
+ <Accordion title="Local Refresh Run Count">
+ **Metric Name**: `infisical.local_refresh.run.count`
+
+ **Type**: Counter
+
+ **Unit**: `{run}`
+
+ **Description**: Refresh runs by outcome. A `skipped` run means the previous run was still in progress when the next one was due.
+
+ **Attributes**:
+ - `job.name` (string): the refresh task, e.g. `self-hosted-license-sync`
+ - `outcome` (string): `completed`, `failed`, or `skipped`
+ - `error.type` (string): only on `failed` runs, with the same values as `infisical.queue.job.failure.count`
+ </Accordion>
+
+ <Accordion title="Local Refresh Run Duration">
+ **Metric Name**: `infisical.local_refresh.run.duration`
+
+ **Type**: Histogram
+
+ **Unit**: `s`
+
+ **Description**: How long each refresh run took. Not recorded for skipped runs.
+
+ **Attributes**: `job.name`, `outcome`
+ </Accordion>
+
+ <Accordion title="Local Refresh Consecutive Failures">
+ **Metric Name**: `infisical.local_refresh.consecutive_failures`
+
+ **Type**: Observable Gauge
+
+ **Unit**: `{run}`
+
+ **Description**: How many runs in a row have failed for each refresh task on this instance. It returns to `0` on the next successful run, so a single failure clears itself. Alert on this gauge rather than on the failure count: a value that stays above `2` or `3` means the task keeps failing and the instance has stopped picking up changes to those settings.
+
+ **Attributes**: `job.name`
+ </Accordion>
+</AccordionGroup>
+
### Audit log metrics (InfisicalCore)
End-to-end audit-log pipeline observability: how many events get enqueued per event type / actor, how long persistence takes, and how many are ultimately dropped.