Infisical is the open-source platform for securing secrets, certificates, access, and AI Agents.
TypeScript acmeagentsai-agentscertificate-managementdevsecopsenvironment-variableskubernetesopen-sourcepampkipostgresprivate-caprivileged-access-managementsecret-managementsecret-managersecretssecrets-managementsecuritysecurity-toolsvault
feat(metrics): implement local refresh metrics for monitoring task execution
Victor Hugo dos Santosfce1ca39a1f3
diff --git a/backend/CLAUDE.md b/backend/CLAUDE.mdindex e863c2b402..da8c0f6107 100644--- a/backend/CLAUDE.md+++ b/backend/CLAUDE.md@@ -630,7 +630,7 @@ Recurring work runs through the cron manager in `src/lib/cron/cron-job.ts` (`cro**Only `general-workers` pods start the manager's timers**, so only they execute a scheduled handler; every pod still calls `register`. The separate `cronJobs` array in `src/server/routes/index.ts` is deliberately **not** gated: those refresh the process's own caches (license, rate limits, env overrides), not fleet work.-**Per-process refreshes use `startLocalRefresh` (`src/lib/cron/local-refresh.ts`), never the cron manager and never a raw `new CronJob`.** The cron manager runs a job once per fleet, which would leave every other pod's in-memory state stale; a wall-clock cron pattern makes every pod fire in the same second. The helper starts each process at a random offset within one interval (random per process on purpose, unlike the manager's deterministic hash), skips a tick while the previous run is in flight, logs and swallows errors, and unrefs its timers. Its `stop()` handle is what goes in `cronJobs`.+**Per-process refreshes use `startLocalRefresh` (`src/lib/cron/local-refresh.ts`), never the cron manager and never a raw `new CronJob`.** The cron manager runs a job once per fleet, which would leave every other pod's in-memory state stale; a wall-clock cron pattern makes every pod fire in the same second. The helper starts each process at a random offset within one interval (random per process on purpose, unlike the manager's deterministic hash), skips a tick while the previous run is in flight, logs and swallows errors, and unrefs its timers. Its `stop()` handle is what goes in `cronJobs`. It records `infisical.local_refresh.*` on the `InfisicalCore` meter, labelled by `name` as `job.name`, so a new refresh needs no metric code; keep `name` a fixed string. A failing refresh keeps retrying on every tick rather than giving up after N failures, because stopping would freeze the pod's state at whatever it last loaded. The `consecutive_failures` gauge (reset on success) is the alerting signal.**Why this exists instead of BullMQ repeatables**: cron runs are coordinated across pods via a slot-election scheme (5 participant slots backed by Redis SET NX/PX) plus per-run redlocks, so each fire executes exactly once across the fleet without the orphaned-scheduler / duplicate-execution failure modes the BullMQ `JobScheduler` had. The manager also handles crash recovery via lease TTLs, hang recovery via per-handler timeouts, and bounded exponential backoff that won't overlap with the next scheduled fire.diff --git a/backend/src/lib/cron/local-refresh.test.ts b/backend/src/lib/cron/local-refresh.test.tsindex f905c9114d..b0bec5a00f 100644--- a/backend/src/lib/cron/local-refresh.test.ts+++ b/backend/src/lib/cron/local-refresh.test.ts@@ -1,6 +1,7 @@import { afterEach, beforeEach, describe, expect, test, vi } from "vitest";import { logger } from "@app/lib/logger";+import { LocalRefreshOutcome, recordLocalRefreshRunMetric } from "@app/lib/telemetry/metrics";import { startLocalRefresh, TLocalRefreshHandle } from "./local-refresh";@@ -8,6 +9,13 @@ vi.mock("@app/lib/logger", () => ({logger: { info: vi.fn(), warn: vi.fn(), error: vi.fn(), debug: vi.fn() }}));+// Stubbed outright: the real module reaches this file again through env -> crypto -> license-service,+// so a partial mock built from importOriginal() hands local-refresh the unmocked helper.+vi.mock("@app/lib/telemetry/metrics", () => ({+ LocalRefreshOutcome: { COMPLETED: "completed", FAILED: "failed", SKIPPED: "skipped" },+ recordLocalRefreshRunMetric: vi.fn()+}));+const INTERVAL_MS = 5 * 60 * 1000;const MAX_TIMER_DELAY_MS = 2 ** 31 - 1;@@ -140,6 +148,52 @@ describe("startLocalRefresh", () => {expect(logger.error).toHaveBeenCalledWith(expect.any(Error), expect.stringContaining("[name=flaky]"));});+ test("counts consecutive failures and resets the count on the next success", async () => {+ mockRandom(0);+ const boom = new Error("boom");+ const task = vi+ .fn()+ .mockRejectedValueOnce(boom)+ .mockRejectedValueOnce(boom)+ .mockResolvedValueOnce(undefined)+ .mockRejectedValueOnce(boom);++ handle = startLocalRefresh({ name: "flaky", intervalMs: INTERVAL_MS, task });+ await vi.advanceTimersByTimeAsync(0);+ await vi.advanceTimersByTimeAsync(INTERVAL_MS * 3);++ const recorded = vi.mocked(recordLocalRefreshRunMetric).mock.calls.map(([params]) => params);+ expect(recorded.map(({ outcome, consecutiveFailures }) => [outcome, consecutiveFailures])).toEqual([+ [LocalRefreshOutcome.FAILED, 1],+ [LocalRefreshOutcome.FAILED, 2],+ [LocalRefreshOutcome.COMPLETED, 0],+ [LocalRefreshOutcome.FAILED, 1]+ ]);+ expect(recorded[0]).toEqual(expect.objectContaining({ name: "flaky", error: boom }));+ expect(recorded[2].error).toBeUndefined();+ expect(logger.error).toHaveBeenCalledWith(boom, expect.stringContaining("[consecutiveFailures=2]"));+ });++ test("records run duration, and a skipped tick without one", async () => {+ mockRandom(0);+ let release: () => void = () => {};+ const task = vi.fn(async () => {+ await new Promise<void>((resolve) => {+ release = resolve;+ });+ });++ handle = startLocalRefresh({ name: "slow", intervalMs: INTERVAL_MS, task });+ await vi.advanceTimersByTimeAsync(INTERVAL_MS);+ expect(recordLocalRefreshRunMetric).toHaveBeenCalledWith({ name: "slow", outcome: LocalRefreshOutcome.SKIPPED });++ release();+ await vi.advanceTimersByTimeAsync(0);+ expect(recordLocalRefreshRunMetric).toHaveBeenLastCalledWith(+ expect.objectContaining({ name: "slow", outcome: LocalRefreshOutcome.COMPLETED, durationMs: INTERVAL_MS })+ );+ });+test("logs duration at debug and warns only above the threshold", async () => {mockRandom(0);const task = vi.fn(async () => {diff --git a/backend/src/lib/cron/local-refresh.ts b/backend/src/lib/cron/local-refresh.tsindex 8bd7583eec..671493b783 100644--- a/backend/src/lib/cron/local-refresh.ts+++ b/backend/src/lib/cron/local-refresh.ts@@ -1,4 +1,5 @@import { logger } from "@app/lib/logger";+import { LocalRefreshOutcome, recordLocalRefreshRunMetric } from "@app/lib/telemetry/metrics";export type TLocalRefreshHandle = {stop: () => void;@@ -33,24 +34,33 @@ export const startLocalRefresh = ({let stopped = false;let running = false;+ let consecutiveFailures = 0;let timer: NodeJS.Timeout | undefined;const run = async () => {if (stopped) return;if (running) {logger.warn(`Local refresh skipped, previous run still in progress [name=${name}]`);+ recordLocalRefreshRunMetric({ name, outcome: LocalRefreshOutcome.SKIPPED });return;}running = true;const startedAt = Date.now();+ let outcome = LocalRefreshOutcome.COMPLETED;+ let failure: unknown;try {await task();+ consecutiveFailures = 0;} catch (error) {- logger.error(error, `Local refresh failed [name=${name}]`);+ outcome = LocalRefreshOutcome.FAILED;+ failure = error;+ consecutiveFailures += 1;+ logger.error(error, `Local refresh failed [name=${name}] [consecutiveFailures=${consecutiveFailures}]`);} finally {running = false;const durationMs = Date.now() - startedAt;+ recordLocalRefreshRunMetric({ name, outcome, durationMs, error: failure, consecutiveFailures });logger.debug({ name, duration_ms: durationMs },`Local refresh finished [name=${name}] [durationMs=${durationMs}]`diff --git a/backend/src/lib/telemetry/metrics.test.ts b/backend/src/lib/telemetry/metrics.test.tsindex 9323ee39c5..5c0c359af4 100644--- a/backend/src/lib/telemetry/metrics.test.ts+++ b/backend/src/lib/telemetry/metrics.test.ts@@ -1,6 +1,25 @@import opentelemetry from "@opentelemetry/api";--import { highCardinalityMeter, normalizeHttpMethod, shouldRecordHighCardinalityMetrics } from "./metrics";+import {+ AggregationTemporality,+ createAllowListAttributesProcessor,+ type DataPoint,+ InMemoryMetricExporter,+ MeterProvider,+ PeriodicExportingMetricReader+} from "@opentelemetry/sdk-metrics";+import type { Knex } from "knex";++import { BadRequestError } from "@app/lib/errors";++import {+ highCardinalityMeter,+ LocalRefreshOutcome,+ normalizeHttpMethod,+ recordLocalRefreshRunMetric,+ registerInfrastructureMetrics,+ shouldRecordHighCardinalityMetrics+} from "./metrics";+import { INFISICAL_CORE_METER_ATTRIBUTES } from "./telemetry-attributes";const mockConfig = {OTEL_TELEMETRY_COLLECTION_ENABLED: true,@@ -128,3 +147,95 @@ describe("normalizeHttpMethod", () => {expect(normalizeHttpMethod("")).toBe("GET");});});++describe("local refresh metrics", () => {+ let reader: PeriodicExportingMetricReader;++ // Instruments bind to the provider on first use and never rebind, so the provider is installed once.+ beforeAll(() => {+ opentelemetry.metrics.disable();+ reader = new PeriodicExportingMetricReader({+ exporter: new InMemoryMetricExporter(AggregationTemporality.CUMULATIVE),+ exportIntervalMillis: 60_000+ });+ opentelemetry.metrics.setGlobalMeterProvider(+ new MeterProvider({+ readers: [reader],+ views: [+ {+ meterName: "InfisicalCore",+ attributesProcessors: [createAllowListAttributesProcessor(INFISICAL_CORE_METER_ATTRIBUTES)]+ }+ ]+ })+ );+ registerInfrastructureMetrics({ client: {} } as unknown as Knex);+ });++ afterAll(() => {+ opentelemetry.metrics.disable();+ });++ beforeEach(() => {+ mockConfig.OTEL_TELEMETRY_COLLECTION_ENABLED = true;+ });++ const collectPoints = async (metricName: string) => {+ const { resourceMetrics } = await reader.collect();+ return resourceMetrics.scopeMetrics+ .flatMap((scope) => scope.metrics)+ .filter((metric) => metric.descriptor.name === metricName)+ .flatMap((metric) => metric.dataPoints as DataPoint<number>[]);+ };++ test("labels failed runs with the classified error type and keeps the labels through the allowlist", async () => {+ recordLocalRefreshRunMetric({+ name: "labels",+ outcome: LocalRefreshOutcome.FAILED,+ durationMs: 1500,+ error: new BadRequestError({ message: "bad" }),+ consecutiveFailures: 1+ });+ recordLocalRefreshRunMetric({ name: "labels", outcome: LocalRefreshOutcome.SKIPPED });++ const runs = (await collectPoints("infisical.local_refresh.run.count")).filter(+ (point) => point.attributes["job.name"] === "labels"+ );+ expect(runs.map((point) => point.attributes)).toEqual(+ expect.arrayContaining([+ { "job.name": "labels", outcome: "failed", "error.type": "validation" },+ { "job.name": "labels", outcome: "skipped" }+ ])+ );++ const durations = (await collectPoints("infisical.local_refresh.run.duration")).filter(+ (point) => point.attributes["job.name"] === "labels"+ );+ expect(durations).toHaveLength(1);+ expect((durations[0].value as unknown as { sum: number }).sum).toBe(1.5);+ });++ test("the consecutive failure gauge heals on success", async () => {+ const gaugeValue = async () =>+ (await collectPoints("infisical.local_refresh.consecutive_failures")).find(+ (point) => point.attributes["job.name"] === "gauge"+ )?.value;++ recordLocalRefreshRunMetric({ name: "gauge", outcome: LocalRefreshOutcome.FAILED, consecutiveFailures: 3 });+ expect(await gaugeValue()).toBe(3);++ recordLocalRefreshRunMetric({ name: "gauge", outcome: LocalRefreshOutcome.COMPLETED, consecutiveFailures: 0 });+ expect(await gaugeValue()).toBe(0);+ });++ test("records nothing while telemetry is disabled", async () => {+ mockConfig.OTEL_TELEMETRY_COLLECTION_ENABLED = false;+ recordLocalRefreshRunMetric({ name: "disabled", outcome: LocalRefreshOutcome.FAILED, consecutiveFailures: 1 });+ mockConfig.OTEL_TELEMETRY_COLLECTION_ENABLED = true;++ const runs = (await collectPoints("infisical.local_refresh.run.count")).filter(+ (point) => point.attributes["job.name"] === "disabled"+ );+ expect(runs).toHaveLength(0);+ });+});diff --git a/backend/src/lib/telemetry/metrics.ts b/backend/src/lib/telemetry/metrics.tsindex bf26cb3ce3..94e15e3d8f 100644--- a/backend/src/lib/telemetry/metrics.ts+++ b/backend/src/lib/telemetry/metrics.ts@@ -309,6 +309,54 @@ export const queueStalledCounter = infisicalCoreMeter.createCounter("infisical.qunit: "{job}"});+// In-process refresh metrics. Wired in lib/cron/local-refresh.ts around every run. Each pod runs its own+// copy, so a failure here means that pod is serving stale in-memory state, not that a fleet job was lost.+export enum LocalRefreshOutcome {+ COMPLETED = "completed",+ FAILED = "failed",+ SKIPPED = "skipped"+}++export const localRefreshRunCounter = infisicalCoreMeter.createCounter("infisical.local_refresh.run.count", {+ description:+ "In-process refresh runs by job and outcome. skipped means the previous run was still in progress when the next tick fired.",+ unit: "{run}"+});++export const localRefreshRunDurationHistogram = infisicalCoreMeter.createHistogram(+ "infisical.local_refresh.run.duration",+ {+ description: "In-process refresh run duration by job and outcome. Not recorded for skipped runs.",+ unit: "s"+ }+);++// Read by the infisical.local_refresh.consecutive_failures gauge in registerInfrastructureMetrics().+const localRefreshConsecutiveFailures = new Map<string, number>();++export const recordLocalRefreshRunMetric = (params: {+ name: string;+ outcome: LocalRefreshOutcome;+ durationMs?: number;+ error?: unknown;+ consecutiveFailures?: number;+}) => {+ safely(() => {+ if (params.consecutiveFailures !== undefined) {+ localRefreshConsecutiveFailures.set(params.name, params.consecutiveFailures);+ }+ if (!isTelemetryEnabled()) return;+ const attributes: Record<string, string> = { "job.name": params.name, outcome: params.outcome };+ if (params.durationMs !== undefined) {+ localRefreshRunDurationHistogram.record(params.durationMs / 1000, attributes);+ }+ if (params.outcome === LocalRefreshOutcome.FAILED) {+ attributes["error.type"] = classifyError(params.error);+ }+ localRefreshRunCounter.add(1, attributes);+ });+};+// Audit log lifecycle metrics. Wired in audit-log-queue.ts: enqueued when an event is appended to// the Redis ingest stream, dropped when the request-path push fails, persist duration around the// batch insert in the unified consumer.@@ -899,6 +947,20 @@ export const registerInfrastructureMetrics = (db: Knex) => {const { size, max } = getAgentPoolStats();result.observe(size, { "pool.max": String(max) });});++ // Resets to 0 on the next successful run, so a sustained non-zero value means the refresh is stuck+ // failing rather than flaking. Alert on this instead of on the failure rate.+ const localRefreshFailuresGauge = meter.createObservableGauge("infisical.local_refresh.consecutive_failures", {+ description: "Consecutive failed runs of each in-process refresh on this pod. Resets to 0 on success.",+ unit: "{run}"+ });++ localRefreshFailuresGauge.addCallback((result) => {+ if (!isTelemetryEnabled()) return;+ localRefreshConsecutiveFailures.forEach((count, name) => {+ result.observe(count, { "job.name": name });+ });+ });};// -- Legacy root-key usage (InfisicalCore meter) -----------------------------------------------------diff --git a/docs/self-hosting/guides/monitoring-telemetry.mdx b/docs/self-hosting/guides/monitoring-telemetry.mdxindex e705b3a2f7..dae7607f3c 100644--- a/docs/self-hosting/guides/monitoring-telemetry.mdx+++ b/docs/self-hosting/guides/monitoring-telemetry.mdx@@ -344,7 +344,7 @@ Replace `otel-collector:8889` with the actual hostname and port where your OpenTInfisical emits metrics on two OpenTelemetry meters simultaneously. Choose which to scrape based on your deployment scale.- **High-cardinality, per-actor meters:** the `Infisical` and `API` meters' original metrics (`infisical.http.server.request.*`, `infisical.http.server.error.count`, `infisical.secret.read.count`, `infisical.auth.attempt.count`, `infisical.kmip.operation.count`) include per-actor labels such as `user.email`, `identity.name`, `client.address`, `user_agent.original`, `organization.name`, `project.name`, `secret.path`, and `secret.name`. The `SecretSyncs`, `PkiSyncs`, and `Integrations` meters likewise carry unbounded labels such as `syncId`. Useful for self-hosted deployments where you want per-user visibility directly in Grafana. May become expensive at large scale (many users, identities, or IPs) due to label cardinality.-- **`InfisicalCore` meter (bounded-cardinality):** all newer metrics (queue, audit log, product analytics aggregation, alert dispatch, event outbox, permission cache, secret cache, rate limit, build info, `infisical.core.http.request.count`, `infisical.core.http.error.count`, authentication latency, token renewal, SSO config changes, SCIM provisioning, and database connection pool) use **only bounded enums and static route templates** as labels (e.g. HTTP method, parameterized `http.route`, error type, outcome). A strict allowlist drops every other attribute at the SDK level, so **no tenant or actor identifiers are emitted**; no organization ID, project IDs, names, emails, IPs, or user agents. This keeps the series count independent of how many organizations, users, or identities you have. Per-org and per-actor detail is available in audit logs instead.+- **`InfisicalCore` meter (bounded-cardinality):** all newer metrics (queue, local refresh, audit log, product analytics aggregation, alert dispatch, event outbox, permission cache, secret cache, rate limit, build info, `infisical.core.http.request.count`, `infisical.core.http.error.count`, authentication latency, token renewal, SSO config changes, SCIM provisioning, and database connection pool) use **only bounded enums and static route templates** as labels (e.g. HTTP method, parameterized `http.route`, error type, outcome). A strict allowlist drops every other attribute at the SDK level, so **no tenant or actor identifiers are emitted**; no organization ID, project IDs, names, emails, IPs, or user agents. This keeps the series count independent of how many organizations, users, or identities you have. Per-org and per-actor detail is available in audit logs instead.All meters emit at the same time. If you're a self-hosted instance and find the per-actor labels useful, keep using the high-cardinality metrics. For larger or multi-tenant deployments, scrape only the `InfisicalCore` metrics to keep cardinality under control.@@ -809,6 +809,51 @@ These metrics give per-queue visibility into BullMQ worker health: throughput, l</Accordion></AccordionGroup>+### Local refresh metrics (InfisicalCore)++Every Infisical instance runs a few background tasks on a timer to refresh settings it keeps in memory, such as the license, rate limits, and SSO configuration. Each instance runs its own copy, so a failing refresh means that instance is serving stale settings while the others may be fine. These metrics are labelled by task, and each instance reports its own values.++<AccordionGroup>+ <Accordion title="Local Refresh Run Count">+ **Metric Name**: `infisical.local_refresh.run.count`++ **Type**: Counter++ **Unit**: `{run}`++ **Description**: Refresh runs by outcome. A `skipped` run means the previous run was still in progress when the next one was due.++ **Attributes**:+ - `job.name` (string): the refresh task, e.g. `self-hosted-license-sync`+ - `outcome` (string): `completed`, `failed`, or `skipped`+ - `error.type` (string): only on `failed` runs, with the same values as `infisical.queue.job.failure.count`+ </Accordion>++ <Accordion title="Local Refresh Run Duration">+ **Metric Name**: `infisical.local_refresh.run.duration`++ **Type**: Histogram++ **Unit**: `s`++ **Description**: How long each refresh run took. Not recorded for skipped runs.++ **Attributes**: `job.name`, `outcome`+ </Accordion>++ <Accordion title="Local Refresh Consecutive Failures">+ **Metric Name**: `infisical.local_refresh.consecutive_failures`++ **Type**: Observable Gauge++ **Unit**: `{run}`++ **Description**: How many runs in a row have failed for each refresh task on this instance. It returns to `0` on the next successful run, so a single failure clears itself. Alert on this gauge rather than on the failure count: a value that stays above `2` or `3` means the task keeps failing and the instance has stopped picking up changes to those settings.++ **Attributes**: `job.name`+ </Accordion>+</AccordionGroup>+### Audit log metrics (InfisicalCore)End-to-end audit-log pipeline observability: how many events get enqueued per event type / actor, how long persistence takes, and how many are ultimately dropped.