Infisical is the open-source platform for securing secrets, certificates, access, and AI Agents.
TypeScript acmeagentsai-agentscertificate-managementdevsecopsenvironment-variableskubernetesopen-sourcepampkipostgresprivate-caprivileged-access-managementsecret-managementsecret-managersecretssecrets-managementsecuritysecurity-toolsvault
feat(telemetry): enhance HTTP metrics and instrumentation
Victor Hugo dos Santos6d198907e37c
diff --git a/backend/CLAUDE.md b/backend/CLAUDE.mdindex ffc63d3a16..ee978aa61f 100644--- a/backend/CLAUDE.md+++ b/backend/CLAUDE.md@@ -696,7 +696,7 @@ OpenTelemetry metric setup lives in `src/lib/telemetry/`. Instruments are define**Meter split by cardinality:**- **`InfisicalCore`** — the meter for all new metrics. A strict attribute allowlist (`INFISICAL_CORE_METER_ATTRIBUTES` in `telemetry-attributes.ts`) is applied via an SDK View, so **only bounded labels survive** — HTTP method, parameterized `http.route` template, and low-cardinality enums. This is the choke point for our own call sites: any attribute passed at a call site that isn't in the allowlist is silently dropped.- **Legacy `Infisical` / `API` / `SecretSyncs` / `PkiSyncs` / `Integrations`** — carry unbounded per-actor labels (`HIGH_CARDINALITY_METER_NAMES`). Disabled wholesale via `OTEL_DROP_HIGH_CARDINALITY_METERS=true` in multi-tenant/cloud.-- **`@opentelemetry/instrumentation-http`** — the second allowlist (`HTTP_INSTRUMENTATION_METER_ATTRIBUTES`). `instrumentation.ts` forces the stable HTTP semconv (`OTEL_SEMCONV_STABILITY_OPT_IN`), because the old conventions label the server metric with the raw `Host` header.+- **`@opentelemetry/instrumentation-http`** — the second allowlist (`HTTP_INSTRUMENTATION_METER_ATTRIBUTES`). `instrumentation.ts` defaults `OTEL_SEMCONV_STABILITY_OPT_IN` to the stable HTTP semconv, because the old conventions label the server metric with the raw `Host` header. An operator's explicit `http/dup` is honoured instead (it is the OTel migration path, emitting both name sets while dashboards move over), which is why the allowlist also carries the old `http.method` / `http.status_code` / `http.flavor` / `http.scheme` names — without them the dup'd metric arrives with an empty attribute set. The `net.*` host and peer names stay off the list in both conventions.**Recording is fire-and-forget.** Every `.add()` / `.record()` and every `record*Metric` helper is wrapped in `safely()` in `metrics.ts`, so a broken exporter, a bad instrument name, or a config read before `initEnvConfig()` can never throw into the code being measured. A measurement is an observation of the work, never a step in it — so call sites do not need their own try/catch, and must not treat a metric as something that can fail. The swallow is silent because the logger is itself initialised from config.@@ -707,7 +707,7 @@ The gate has to sit at the call site because **a `DROP` aggregation does not bou**Rules for InfisicalCore metrics:**- **No per-tenant / per-actor identifiers** as labels — no org id, user id/email, identity id, ip, user agent, request id, or free-form values (e.g. environment slug). These scale series count with customer count, which breaks CloudWatch's 1000-datapoint-per-OTLP-request limit and drives per-GB ingestion cost. Use the **audit log table** for per-org / per-actor breakdowns.- Adding a new label means adding it to the allowlist in `telemetry-attributes.ts`. Only add **bounded** keys (fixed enums / static route templates), and document why.-- `http.route` must be the parameterized template (`req.routeOptions.url`), never the raw request path.+- `http.route` must be the parameterized template (`req.routeOptions.url`), never the raw request path. `api-metrics.ts` skips routes generated by `@fastify/static` for the same reason: in `STANDALONE_MODE` `serve-ui.ts` registers it with `wildcard: false`, so every built asset gets its own route and `http.route` would be a content-hashed filename.### PostHog Product Analyticsdiff --git a/backend/src/lib/telemetry/http-instrumentation.test.ts b/backend/src/lib/telemetry/http-instrumentation.test.tsindex 7a8bc1a37a..9b082ce9cd 100644--- a/backend/src/lib/telemetry/http-instrumentation.test.ts+++ b/backend/src/lib/telemetry/http-instrumentation.test.ts@@ -29,10 +29,13 @@ describe("resolveHttpSemconvOptIn", () => {expect(resolveHttpSemconvOptIn("database,messaging")).toBe("database,messaging,http");});- // http/dup emits the old attribute set alongside the stable one, which puts net.host.name back.- test("strips http/dup rather than letting it through", () => {- expect(resolveHttpSemconvOptIn("http/dup")).toBe("http");- expect(resolveHttpSemconvOptIn("database,HTTP/DUP")).toBe("database,http");+ // http/dup is the OTel migration path: it emits the old metric names alongside the stable ones so an+ // operator can cut dashboards over before the old names go away. An explicit opt-in is theirs to make.+ test("honours an explicitly set http/dup", () => {+ expect(resolveHttpSemconvOptIn("http/dup")).toBe("http/dup");+ expect(resolveHttpSemconvOptIn("database,HTTP/DUP")).toBe("database,HTTP/DUP");+ // http/dup takes precedence in the SDK's own parser, so both tokens together still mean duplicate.+ expect(resolveHttpSemconvOptIn("http,http/dup")).toBe("http,http/dup");});test("does not duplicate an http token that is already present", () => {diff --git a/backend/src/lib/telemetry/http-metrics.test.ts b/backend/src/lib/telemetry/http-metrics.test.tsindex 087cf70d01..ee05a1fdbe 100644--- a/backend/src/lib/telemetry/http-metrics.test.ts+++ b/backend/src/lib/telemetry/http-metrics.test.ts@@ -1,3 +1,8 @@+import fs from "node:fs";+import os from "node:os";+import path from "node:path";++import staticServe from "@fastify/static";import opentelemetry from "@opentelemetry/api";import {AggregationType,@@ -63,7 +68,7 @@ const installMeterProvider = () => {opentelemetry.metrics.setGlobalMeterProvider(new MeterProvider({ readers: [reader], views }));};-const buildApp = async (): Promise<FastifyInstance> => {+const buildApp = async (extend?: (app: FastifyInstance) => Promise<void>): Promise<FastifyInstance> => {// Imported after the provider is installed so the plugin binds its instruments to this test's reader.const { apiMetrics } = await import("@app/server/plugins/api-metrics");const app = Fastify();@@ -72,21 +77,45 @@ const buildApp = async (): Promise<FastifyInstance> => {app.get("/api/v1/boom", async () => {throw new Error("boom");});+ await extend?.(app);await app.ready();return app;};+let collected = false;++const takeSnapshot = async () => {+ if (collected) {+ throw new Error(+ "reader.collect() was already called in this test. The reader is delta, so this snapshot is empty. Take one snapshot and read every metric out of it (see collectPair)."+ );+ }+ collected = true;+ const { resourceMetrics } = await reader.collect();+ return resourceMetrics.scopeMetrics.flatMap((scope) => scope.metrics);+};+// Returns every data point recorded for a metric, across all attribute sets.const collect = async (metricName: string): Promise<DataPoint<number>[]> => {- const { resourceMetrics } = await reader.collect();- return resourceMetrics.scopeMetrics- .flatMap((scope) => scope.metrics)+ const metrics = await takeSnapshot();+ return metrics.filter((metric) => metric.descriptor.name === metricName).flatMap((metric) => metric.dataPoints as DataPoint<number>[]);};+// Reads two metrics out of one snapshot, which is the only way to assert on both.+const collectPair = async (first: string, second: string): Promise<[DataPoint<number>[], DataPoint<number>[]]> => {+ const metrics = await takeSnapshot();+ const pointsFor = (name: string) =>+ metrics+ .filter((metric) => metric.descriptor.name === name)+ .flatMap((metric) => metric.dataPoints as DataPoint<number>[]);+ return [pointsFor(first), pointsFor(second)];+};+describe("api-metrics plugin", () => {beforeEach(() => {+ collected = false;mockConfig.OTEL_TELEMETRY_COLLECTION_ENABLED = true;mockConfig.OTEL_DROP_HIGH_CARDINALITY_METERS = false;});@@ -134,6 +163,21 @@ describe("api-metrics plugin", () => {await app.close();});+ test("normalizes only the core counter, leaving the per-actor meter's method raw", async () => {+ installMeterProvider();+ const app = await buildApp();+ await app.inject({ method: "PROPFIND" as "GET", url: "/api/v1/no-such-route" });++ // The core counter is normalized so its labels join against the OTel HTTP instrumentation. The legacy+ // meter is not, because folding PROPFIND onto _OTHER there would erase the scanner traffic that meter+ // exists to attribute, and the raw method is bounded by Node's own parser anyway.+ const [core, legacy] = await collectPair(CORE_REQUEST_COUNT, LEGACY_REQUEST_COUNT);+ expect(core[0].attributes["http.request.method"]).toBe("_OTHER");+ expect(legacy).toHaveLength(1);+ expect(legacy[0].attributes["http.request.method"]).toBe("PROPFIND");+ await app.close();+ });+test("collapses distinct path parameters onto one series", async () => {installMeterProvider();const app = await buildApp();@@ -170,15 +214,43 @@ describe("api-metrics plugin", () => {await app.inject({ method: "GET", url: "/api/v1/workspace/abc-123" });// The denominator has to survive the kill switch; that is the whole point of the split.- const core = await collect(CORE_REQUEST_COUNT);+ const [core, legacy] = await collectPair(CORE_REQUEST_COUNT, LEGACY_REQUEST_COUNT);expect(core).toHaveLength(1);expect(core[0].value).toBe(1);-- const legacy = await collect(LEGACY_REQUEST_COUNT);expect(legacy).toHaveLength(0);await app.close();});+ test("ignores static asset routes so hashed filenames never become http.route", async () => {+ // STANDALONE_MODE serves the built frontend from the API, and serve-ui.ts registers @fastify/static+ // with wildcard: false, which declares one route per file. Recording those would put a content-hashed+ // filename in http.route on the meter documented as the bounded one, and would make the counter mean+ // something different depending on whether the deployment serves its own UI.+ const assetDir = fs.mkdtempSync(path.join(os.tmpdir(), "api-metrics-assets-"));+ fs.mkdirSync(path.join(assetDir, "assets"));+ const assetNames = ["app-Bmt1dKu1.js", "vendor-YXR_I9Ib.js", "styles-hwHhd4i8.css"];+ assetNames.forEach((name) => fs.writeFileSync(path.join(assetDir, "assets", name), "// built asset"));++ installMeterProvider();+ const app = await buildApp(async (instance) => {+ await instance.register(staticServe, { root: assetDir, wildcard: false, index: false });+ });++ for (const name of assetNames) {+ // eslint-disable-next-line no-await-in-loop+ const res = await app.inject({ method: "GET", url: `/assets/${name}` });+ expect(res.statusCode).toBe(200);+ }+ await app.inject({ method: "GET", url: "/api/v1/workspace/abc-123" });++ const points = await collect(CORE_REQUEST_COUNT);+ expect(points).toHaveLength(1);+ expect(points[0].attributes["http.route"]).toBe("/api/v1/workspace/:workspaceId");++ await app.close();+ fs.rmSync(assetDir, { recursive: true, force: true });+ });+test("records nothing at all when telemetry is disabled", async () => {mockConfig.OTEL_TELEMETRY_COLLECTION_ENABLED = false;installMeterProvider();diff --git a/backend/src/lib/telemetry/http-semconv-dup.test.ts b/backend/src/lib/telemetry/http-semconv-dup.test.tsnew file mode 100644index 0000000000..b6178a3c1d--- /dev/null+++ b/backend/src/lib/telemetry/http-semconv-dup.test.ts@@ -0,0 +1,117 @@+import { createRequire } from "node:module";++import opentelemetry from "@opentelemetry/api";+import { registerInstrumentations } from "@opentelemetry/instrumentation";+import { HttpInstrumentation } from "@opentelemetry/instrumentation-http";+import {+ AggregationTemporality,+ createAllowListAttributesProcessor,+ InMemoryMetricExporter,+ MeterProvider,+ PeriodicExportingMetricReader,+ type ResourceMetrics+} from "@opentelemetry/sdk-metrics";++import {+ HTTP_INSTRUMENTATION_METER_ATTRIBUTES,+ HTTP_INSTRUMENTATION_METER_NAME,+ resolveHttpSemconvOptIn+} from "./telemetry-attributes";++// The third HttpInstrumentation pipeline file, covering the migration configuration an operator opts into+// with OTEL_SEMCONV_STABILITY_OPT_IN=http/dup. Its own file for the same reason as the other two: the+// instrumentation reads the env var in its constructor and patches node:http on require, and Vitest+// isolates by file. What it guards is that honouring http/dup is worth something — the old metric has to+// come back with labels a dashboard can filter on, without the unbounded ones coming back with it.+const getWithHost = (http: typeof import("node:http"), port: number, host: string) =>+ new Promise<void>((resolve, reject) => {+ http+ .get({ host: "127.0.0.1", port, path: "/probe", headers: { Host: host } }, (res) => {+ res.resume();+ res.on("end", resolve);+ })+ .on("error", reject);+ });++const findMetric = (snapshot: ResourceMetrics, name: string) =>+ snapshot.scopeMetrics.flatMap((scope) => scope.metrics).find((metric) => metric.descriptor.name === name);++describe("HttpInstrumentation under an operator's http/dup opt-in", () => {+ let snapshot: ResourceMetrics;+ let server: import("node:http").Server;++ beforeAll(async () => {+ process.env.OTEL_SEMCONV_STABILITY_OPT_IN = resolveHttpSemconvOptIn("http/dup");++ const reader = new PeriodicExportingMetricReader({+ exporter: new InMemoryMetricExporter(AggregationTemporality.CUMULATIVE),+ exportIntervalMillis: 60_000+ });+ const meterProvider = new MeterProvider({+ readers: [reader],+ views: [+ {+ meterName: HTTP_INSTRUMENTATION_METER_NAME,+ attributesProcessors: [createAllowListAttributesProcessor(HTTP_INSTRUMENTATION_METER_ATTRIBUTES)]+ }+ ]+ });+ opentelemetry.metrics.setGlobalMeterProvider(meterProvider);+ registerInstrumentations({ instrumentations: [new HttpInstrumentation()], meterProvider });++ const http = createRequire(`${process.cwd()}/`)("http") as typeof import("node:http");+ server = http.createServer((_req, res) => {+ res.statusCode = 200;+ res.end("ok");+ });+ await new Promise<void>((resolve) => {+ server.listen(0, "127.0.0.1", resolve);+ });+ const { port } = server.address() as { port: number };++ for (const host of ["probe-a.invalid", "probe-b.invalid", "probe-c.invalid"]) {+ // eslint-disable-next-line no-await-in-loop+ await getWithHost(http, port, host);+ }++ snapshot = (await reader.collect()).resourceMetrics;+ });++ afterAll(async () => {+ delete process.env.OTEL_SEMCONV_STABILITY_OPT_IN;+ opentelemetry.metrics.disable();+ await new Promise<void>((resolve) => {+ if (!server) {+ resolve();+ return;+ }+ server.close(() => resolve());+ });+ });++ test("emits both the old and the stable metric name", () => {+ const names = snapshot.scopeMetrics.flatMap((scope) => scope.metrics).map((metric) => metric.descriptor.name);+ expect(names).toContain("http.server.duration");+ expect(names).toContain("http.server.request.duration");+ });++ test("keeps the old metric usable: method and status survive the allowlist", () => {+ const duration = findMetric(snapshot, "http.server.duration");+ expect(duration?.descriptor.unit).toBe("ms");++ const keys = new Set(duration?.dataPoints.flatMap((point) => Object.keys(point.attributes)) ?? []);+ // An empty attribute set here would mean the old metric is emitted but no dashboard can read it.+ expect(keys.has("http.method")).toBe(true);+ expect(keys.has("http.status_code")).toBe(true);+ });++ test("does not let the old names smuggle the Host header back in", () => {+ const duration = findMetric(snapshot, "http.server.duration");+ const keys = new Set(duration?.dataPoints.flatMap((point) => Object.keys(point.attributes)) ?? []);+ expect(keys.has("net.host.name")).toBe(false);+ expect(keys.has("net.host.port")).toBe(false);++ // Three distinct Host headers collapse onto one series, which is the whole point of the View.+ expect(duration?.dataPoints).toHaveLength(1);+ });+});diff --git a/backend/src/lib/telemetry/instrumentation.ts b/backend/src/lib/telemetry/instrumentation.tsindex 8e839e66b5..1478de35de 100644--- a/backend/src/lib/telemetry/instrumentation.ts+++ b/backend/src/lib/telemetry/instrumentation.ts@@ -23,6 +23,7 @@ import {HTTP_INSTRUMENTATION_METER_ATTRIBUTES,HTTP_INSTRUMENTATION_METER_NAME,INFISICAL_CORE_METER_ATTRIBUTES,+ METER_AGGREGATION_CARDINALITY_LIMIT,resolveHttpSemconvOptIn} from "./telemetry-attributes";@@ -85,10 +86,12 @@ const initTelemetryInstrumentation = ({const views: ViewOptions[] = [{meterName: "InfisicalCore",+ aggregationCardinalityLimit: METER_AGGREGATION_CARDINALITY_LIMIT,attributesProcessors: [createAllowListAttributesProcessor(INFISICAL_CORE_METER_ATTRIBUTES)]},{meterName: HTTP_INSTRUMENTATION_METER_NAME,+ aggregationCardinalityLimit: METER_AGGREGATION_CARDINALITY_LIMIT,attributesProcessors: [createAllowListAttributesProcessor(HTTP_INSTRUMENTATION_METER_ATTRIBUTES)]}];@@ -117,7 +120,19 @@ const initTelemetryInstrumentation = ({opentelemetry.metrics.setGlobalMeterProvider(meterProvider);- process.env.OTEL_SEMCONV_STABILITY_OPT_IN = resolveHttpSemconvOptIn(process.env.OTEL_SEMCONV_STABILITY_OPT_IN);+ const requestedSemconvOptIn = process.env.OTEL_SEMCONV_STABILITY_OPT_IN;+ const resolvedSemconvOptIn = resolveHttpSemconvOptIn(requestedSemconvOptIn);+ if (requestedSemconvOptIn !== resolvedSemconvOptIn) {+ // The value an operator set is not the value the instrumentation ends up reading, so say so at boot.+ // The logger is not available here: instrumentation is imported before the app config is loaded.+ // eslint-disable-next-line no-console+ console.log(+ `Telemetry: set OTEL_SEMCONV_STABILITY_OPT_IN to "${resolvedSemconvOptIn}"${+ requestedSemconvOptIn ? ` (from "${requestedSemconvOptIn}")` : ""+ }. Infisical requires the stable HTTP semantic conventions. Set "http/dup" to also emit the old metric names while migrating dashboards.`+ );+ }+ process.env.OTEL_SEMCONV_STABILITY_OPT_IN = resolvedSemconvOptIn;registerInstrumentations({instrumentations: [new HttpInstrumentation(), new RuntimeNodeInstrumentation()]diff --git a/backend/src/lib/telemetry/metric-cardinality.test.ts b/backend/src/lib/telemetry/metric-cardinality.test.tsnew file mode 100644index 0000000000..7eda3cc2b0--- /dev/null+++ b/backend/src/lib/telemetry/metric-cardinality.test.ts@@ -0,0 +1,108 @@+import opentelemetry from "@opentelemetry/api";+import {+ AggregationTemporality,+ createAllowListAttributesProcessor,+ type DataPoint,+ InMemoryMetricExporter,+ MeterProvider,+ PeriodicExportingMetricReader+} from "@opentelemetry/sdk-metrics";++import { INFISICAL_CORE_METER_ATTRIBUTES, METER_AGGREGATION_CARDINALITY_LIMIT } from "./telemetry-attributes";++const OVERFLOW_ATTRIBUTE = "otel.metric.overflow";++// The cardinality ceiling is what bounds an InfisicalCore instrument on the Prometheus path, where series+// are retained for the life of the process. These tests pin the behaviour an operator's alarm depends on:+// that breaching the ceiling is visible, that it costs attribution rather than counts, and that the signal+// survives the allowlist. A low limit stands in for 2000 so the test stays fast; the fold is the same.+const collectWithLimit = async (limit: number, distinctRoutes: number) => {+ opentelemetry.metrics.disable();+ const reader = new PeriodicExportingMetricReader({+ exporter: new InMemoryMetricExporter(AggregationTemporality.CUMULATIVE),+ exportIntervalMillis: 60_000+ });+ const meterProvider = new MeterProvider({+ readers: [reader],+ views: [+ {+ meterName: "InfisicalCore",+ aggregationCardinalityLimit: limit,+ attributesProcessors: [createAllowListAttributesProcessor(INFISICAL_CORE_METER_ATTRIBUTES)]+ }+ ]+ });+ opentelemetry.metrics.setGlobalMeterProvider(meterProvider);++ const counter = meterProvider.getMeter("InfisicalCore").createCounter("infisical.core.http.request.count");+ for (let i = 0; i < distinctRoutes; i += 1) {+ counter.add(1, {+ "http.request.method": "GET",+ "http.route": `/api/v1/route-${i}`,+ "http.response.status_code": 200,+ // Not on the allowlist. Present to prove the ceiling counts what survives the filter, not what the+ // call site passed, which is the difference between a bounded instrument and an unbounded one.+ "infisical.user.email": `user-${i}@example.com`+ });+ }++ const { resourceMetrics } = await reader.collect();+ const points = resourceMetrics.scopeMetrics+ .flatMap((scope) => scope.metrics)+ .filter((metric) => metric.descriptor.name === "infisical.core.http.request.count")+ .flatMap((metric) => metric.dataPoints as DataPoint<number>[]);+ return points;+};++describe("aggregation cardinality ceiling", () => {+ afterEach(() => {+ opentelemetry.metrics.disable();+ });++ test("holds the instrument at the configured ceiling", async () => {+ const points = await collectWithLimit(10, 50);+ expect(points).toHaveLength(10);+ });++ test("folds the excess onto a single overflow point that outlives the allowlist", async () => {+ const points = await collectWithLimit(10, 50);++ // The SDK injects this after the attributes processor has run, so an alarm can see it even though it+ // is not on INFISICAL_CORE_METER_ATTRIBUTES. Adding it to the allowlist would not help and is not why+ // it survives.+ const overflow = points.find((point) => point.attributes[OVERFLOW_ATTRIBUTE] === true);+ expect(overflow).toBeDefined();+ expect(INFISICAL_CORE_METER_ATTRIBUTES).not.toContain(OVERFLOW_ATTRIBUTE);+ });++ test("loses attribution but never counts", async () => {+ const points = await collectWithLimit(10, 50);++ // 50 increments in, 50 increments out. A breach costs the per-route breakdown, not the total, which is+ // why the alarm is a data-quality signal rather than a liveness one.+ const total = points.reduce((sum, point) => sum + point.value, 0);+ expect(total).toBe(50);+ });++ test("counts attribute sets after the allowlist, not before", async () => {+ // Every call site above passed a distinct infisical.user.email. If cardinality were counted before the+ // filter, nine distinct routes could not fit under a ceiling of ten.+ const points = await collectWithLimit(10, 9);+ expect(points.some((point) => point.attributes[OVERFLOW_ATTRIBUTE] === true)).toBe(false);+ expect(points).toHaveLength(9);+ });++ // This repo registers on the order of 1,500 routes, and infisical.core.http.request.count is keyed by+ // method, route and status. One status code per route fits; a second one does not. So the ceiling is not+ // theoretical headroom for this instrument, it is roughly where a fully exercised deployment sits, and on+ // Prometheus the cumulative retention means a long-lived process trends toward the whole surface. That is+ // the trade the ceiling buys: a bounded scrape, paid for with the tail of the route map.+ test("is reached by the route surface this repo registers once a second status code appears", async () => {+ const withinCeiling = await collectWithLimit(METER_AGGREGATION_CARDINALITY_LIMIT, 1500);+ expect(withinCeiling.some((point) => point.attributes[OVERFLOW_ATTRIBUTE] === true)).toBe(false);++ const beyondCeiling = await collectWithLimit(METER_AGGREGATION_CARDINALITY_LIMIT, 3000);+ expect(beyondCeiling.some((point) => point.attributes[OVERFLOW_ATTRIBUTE] === true)).toBe(true);+ expect(beyondCeiling).toHaveLength(METER_AGGREGATION_CARDINALITY_LIMIT);+ });+});diff --git a/backend/src/lib/telemetry/telemetry-attributes.ts b/backend/src/lib/telemetry/telemetry-attributes.tsindex baf703d96a..34c3fccc41 100644--- a/backend/src/lib/telemetry/telemetry-attributes.ts+++ b/backend/src/lib/telemetry/telemetry-attributes.ts@@ -55,23 +55,36 @@ export const INFISICAL_CORE_METER_ATTRIBUTES = [// OTEL_DROP_HIGH_CARDINALITY_METERS. Kept on by default for self-hosted; dropped in multi-tenant/cloud.export const HIGH_CARDINALITY_METER_NAMES = ["Infisical", "API", "SecretSyncs", "PkiSyncs", "Integrations"];+// Per-instrument series ceiling on the two allowlisted Views. This is the SDK's own default made explicit.+export const METER_AGGREGATION_CARDINALITY_LIMIT = 2000;+// Meter used by @opentelemetry/instrumentation-http for http.server.duration / http.client.duration.export const HTTP_INSTRUMENTATION_METER_NAME = "@opentelemetry/instrumentation-http";-// HttpInstrumentation reads OTEL_SEMCONV_STABILITY_OPT_IN in its constructor (no config hook), so instrumentation.ts-// forces the "http" token via resolveHttpSemconvOptIn — not "http/dup", which doubles cardinality; other namespaces are preserved.+// HttpInstrumentation reads OTEL_SEMCONV_STABILITY_OPT_IN at construction; default to stable "http".+// Honour explicit "http/dup" unchanged so old and stable HTTP metrics can run side by side during migration.export const resolveHttpSemconvOptIn = (current?: string): string => {- const otherNamespaces = (current ?? "")+ const tokens = (current ?? "").split(",").map((token) => token.trim())- .filter((token) => token && token.toLowerCase() !== "http" && token.toLowerCase() !== "http/dup");+ .filter(Boolean);++ if (tokens.some((token) => token.toLowerCase() === "http/dup")) return tokens.join(",");++ const otherNamespaces = tokens.filter((token) => token.toLowerCase() !== "http");return [...otherNamespaces, "http"].join(",");};-// This View keeps http.server.duration bounded if an upgrade reintroduces unbounded host related attributes like net.host.name and constant labels like url.scheme+// Allowlist for http.server.duration: bounded labels only; drops unbounded host/peer attrs an upgrade might reintroduce.export const HTTP_INSTRUMENTATION_METER_ATTRIBUTES = ["http.request.method","http.response.status_code","http.route",- "error.type"+ "error.type",+ // Old names for "http/dup" migration; inert under plain "http" but needed so dup'd metrics stay filterable.+ "http.method",+ "http.status_code",+ "http.flavor",+ "http.scheme"+ // net.host.* / net.peer.* omitted on purpose; client-controlled or per-destination, and unbounded.];diff --git a/backend/src/server/plugins/api-metrics.ts b/backend/src/server/plugins/api-metrics.tsindex b8ef0d534c..2109f81de0 100644--- a/backend/src/server/plugins/api-metrics.ts+++ b/backend/src/server/plugins/api-metrics.ts@@ -32,7 +32,13 @@ export const apiMetrics = fp(async (fastify) => {fastify.addHook("onResponse", async (request, reply) => {if (!isTelemetryEnabled()) return;- const method = normalizeHttpMethod(request.method);+ // the line detects @fastify/static routes via config.rootPath and excludes frontend asset traffic+ // so http.route stays low-cardinality and metrics stay comparable across deployment modes.+ if ((request.routeOptions.config as { rootPath?: string } | undefined)?.rootPath) return;++ // Normalized only for the InfisicalCore instrument, we drop the per-actor meters there.+ const coreMethod = normalizeHttpMethod(request.method);+ const { method } = request;const route = request.routeOptions.url;const { statusCode } = reply;@@ -40,7 +46,7 @@ export const apiMetrics = fp(async (fastify) => {// template, never the raw path), and they are the denominator for coreHttpErrorCounter, so a deployment// that drops the per-actor meters must not lose them too.coreHttpRequestCounter.add(1, {- "http.request.method": method,+ "http.request.method": coreMethod,"http.route": route ?? "unknown","http.response.status_code": statusCode});diff --git a/backend/src/server/plugins/error-handler.ts b/backend/src/server/plugins/error-handler.tsindex 755ee23c40..388e54666b 100644--- a/backend/src/server/plugins/error-handler.ts+++ b/backend/src/server/plugins/error-handler.ts@@ -109,7 +109,9 @@ export const fastifyErrHandler = fastifyPlugin(async (server: FastifyZodProvider}if (appCfg.OTEL_TELEMETRY_COLLECTION_ENABLED) {- const method = normalizeHttpMethod(req.method);+ // Normalized only for the InfisicalCore instrument, we drop the per-actor meters there.+ const coreMethod = normalizeHttpMethod(req.method);+ const { method } = req;const route = req.routeOptions.url;if (shouldRecordHighCardinalityMetrics()) {@@ -186,7 +188,7 @@ export const fastifyErrHandler = fastifyPlugin(async (server: FastifyZodProvider}const coreAttrs: Record<string, string | number> = {- "http.request.method": method,+ "http.request.method": coreMethod,"http.route": route ?? "unknown","error.type": classifyError(error)};diff --git a/docs/self-hosting/configuration/envars.mdx b/docs/self-hosting/configuration/envars.mdxindex 3d43ec6f3a..8fc4371115 100644--- a/docs/self-hosting/configuration/envars.mdx+++ b/docs/self-hosting/configuration/envars.mdx@@ -1106,6 +1106,14 @@ If the export type is set to `otlp`, you will have to configure a value for `OTEWhen set to `true`, Infisical stops recording the high-cardinality, per-actor `Infisical`, `API`, `SecretSyncs`, `PkiSyncs`, and `Integrations` meters. Nothing is measured, exported, or held in memory for them. Only bounded-cardinality `InfisicalCore` metrics are emitted. Recommended when those meters are enabled with `OTEL_EXPORT_TYPE=prometheus` and label combinations turn over often (many secret paths, source IPs, or clients), because memory then grows with uptime until the process restarts. See [Monitoring & Telemetry](/self-hosting/guides/monitoring-telemetry#available-metrics) for details.</ParamField>+<ParamField query="OTEL_SEMCONV_STABILITY_OPT_IN" type="string" default="http" optional>+ Which OpenTelemetry semantic conventions the HTTP metrics follow. Leave it unset and Infisical applies `http`, the stable conventions, and logs that it did so at startup.++ Set it to `http/dup` to emit the old metric names alongside the stable ones while you move dashboards across. Both name sets are exported for as long as it's set, so the volume of those metrics roughly doubles. See [Monitoring & Telemetry](/self-hosting/guides/monitoring-telemetry#system-metrics) for the old and new names.++ Tokens for other namespaces, such as `database`, are left alone.+</ParamField>+## Identity auth method<ParamFielddiff --git a/docs/self-hosting/guides/monitoring-telemetry.mdx b/docs/self-hosting/guides/monitoring-telemetry.mdxindex 70989cfbfa..c5796cac8c 100644--- a/docs/self-hosting/guides/monitoring-telemetry.mdx+++ b/docs/self-hosting/guides/monitoring-telemetry.mdx@@ -473,7 +473,7 @@ These metrics track all HTTP API requests to Infisical, including request countsFor HTTP error rate per route, divide the `http.response.status_code >= 400` series by the total on this metric. Do not use `infisical.core.http.error.count` as the numerator: it counts thrown errors, and a response that sets a 4xx or 5xx status without throwing (an unmatched route, for example) never reaches it.**Attributes**:- - `http.request.method` (string): HTTP method+ - `http.request.method` (string): HTTP method, normalized to the ten names the OpenTelemetry HTTP conventions define, with anything else reported as `_OTHER` so this metric joins against `http.server.request.duration`. The Infisical meter keep the method as sent, so use those to see an unusual method such as `PROPFIND` on its own- `http.route` (string): Parameterized route template (e.g. `/api/v1/secrets/:secretName`), never the raw path. `unknown` for a request that matched no route.- `http.response.status_code` (int): HTTP status code</Accordion>@@ -488,7 +488,7 @@ These metrics track all HTTP API requests to Infisical, including request counts**Description**: API errors with bounded error classification. The low-cardinality counterpart to `infisical.http.server.error.count`; same error volume, but labels are restricted to the InfisicalCore allowlist so the series count stays independent of org/user/identity count. Use this for alerting and dashboards at scale; use `infisical.http.server.error.count` (high-cardinality meter) when you need per-actor attribution. For per-org error breakdowns, query the audit log.**Attributes**:- - `http.request.method` (string): HTTP method+ - `http.request.method` (string): HTTP method, normalized to the ten names the OpenTelemetry HTTP conventions define, with anything else reported as `_OTHER` so this metric joins against `http.server.request.duration`. The Infisical meter keep the method as sent, so use those to see an unusual method such as `PROPFIND` on its own- `http.route` (string): Parameterized route template where the error occurred (e.g. `/api/v1/secrets/:secretName`), never the raw path- `error.type` (string): Bounded error classification</Accordion>@@ -1089,6 +1089,21 @@ Heap usage, GC pause, event loop lag, and other Node runtime metrics are auto-emThese low-level HTTP metrics are automatically collected by OpenTelemetry's instrumentation layer, providing baseline performance data for all HTTP traffic.+<Warning>+ Infisical now follows the stable OpenTelemetry HTTP semantic conventions, which renamed both of these metrics and changed their unit from milliseconds to seconds.++ | Old name | New name |+ | --- | --- |+ | `http.server.duration` (ms) | `http.server.request.duration` (s) |+ | `http.client.duration` (ms) | `http.client.request.duration` (s) |++ Their attributes were renamed too. `http.method` became `http.request.method`, and `http.status_code` became `http.response.status_code`.++ A dashboard or alert built on the old names stops returning data once you upgrade, and a latency threshold carried across unchanged is wrong by a factor of 1000. To emit both name sets at the same time while you move dashboards over, set `OTEL_SEMCONV_STABILITY_OPT_IN=http/dup`, then unset it when you're done. That roughly doubles the volume of these two metrics for as long as it's set.++ Under `http/dup` the old metrics keep `http.method` and `http.status_code`, so your existing queries keep working. They don't get `net.host.name` back, because that label is the reason for the change: it carries the client-controlled `Host` header, and it grows a new time series for every value a client sends.+</Warning>+<AccordionGroup><Accordion title="http_server_request_duration">**Metric Name**: `http.server.request.duration`