Infisical is the open-source platform for securing secrets, certificates, access, and AI Agents.
TypeScript acmeagentsai-agentscertificate-managementdevsecopsenvironment-variableskubernetesopen-sourcepampkipostgresprivate-caprivileged-access-managementsecret-managementsecret-managersecretssecrets-managementsecuritysecurity-toolsvault
second try with redis
Dimitris Staratziscbe552bb1934
diff --git a/backend/CLAUDE.md b/backend/CLAUDE.mdindex 741d59af56..985179c64a 100644--- a/backend/CLAUDE.md+++ b/backend/CLAUDE.md@@ -1078,13 +1078,31 @@ only at some later merge, so read with `FINAL` when an exact count matters.### Secret Read Activity-`ee/services/secret-read-activity/` answers "who has read this secret" from audit logs, for the-per-secret access sheet and the Insights Unused Secret Access card. A bulk read only records a folder, so a-read counts when it reaches the secret through that folder, an import of it, or a secret referencing it-(imports and references check the reader's own access to the source, so revoking it would break them).-A read of a parent folder is only a possible read, since whether it was recursive is not logged. Reads-are matched on paths, and a rename or move changes the path, so only reads after the secret's current-path existed count (`trackedSince`, from folder versions, the secret's and environment's timestamps).+`ee/services/secret-read-activity/` answers "who has read this secret" for the per-secret access sheet and+the Insights Unused Secret Access card. It does not use audit logs: ClickHouse is optional on self-hosted+and the events cannot say which folders a recursive read returned. Instead it keeps its own table,+`secret_read_activity`, one row per (object, reader) holding the last read time.++- **Write path.** Read routes call `recordBulkRead` / `recordRead` with `void`, so they never fail or slow a+ read. A bulk read that returned whole folders (no tag or metadata filter, imports and references included)+ records its folder, plus every non-reserved subfolder when recursive, and does so even on a 304. A+ narrower bulk read records the ids of the secrets it returned, so the secrets it left out are not credited.+ A narrow 304 carries no secrets and records nothing, which is safe only because ETags are stored per UTC day,+ so each client's first read of the day is a full response. Single reads record the secret. A read with+ references unexpanded is stored as `secret-unexpanded` and credits only that secret, not the values it+ references, which is why `objectType` is part of the primary key. A Redis `SET NX` gate (25h TTL) keyed on+ the reader, the day and the request (or the hash of the returned ids) lets only the first identical read+ per day through. It is keyed on the request, not the ETag, since older clients never send one. Gated reads go to a+ Redis stream drained the way the audit log ingest stream is (collect, write, then trim, under a+ non-blocking lock), by the `SecretReadActivityFlush` cron every minute into an upsert.+- **No foreign keys and no index on `lastReadAt`.** Rows of deleted objects stop updating and the daily+ `SecretReadActivityPrune` cron removes anything older than 90 days, walking the primary key. Keeping+ `lastReadAt` out of every index keeps the daily upsert a HOT update. A move creates new ids, so the window+ starts no earlier than the secret's and its folder's creation (`trackedSince`), nor before the table existed.+- **Read path.** A read counts when it reaches the secret through its folder, an import of it, a secret+ referencing it, or the source of a replication import (imports and references check the reader's own access+ to the source, so revoking it would break them). The card computes the whole project once and caches it for+ 5 minutes, grouping principals by identical rules so each rule set is checked once per secret.### Audit Log Event Classes and Settingsdiff --git a/backend/src/@types/knex.d.ts b/backend/src/@types/knex.d.tsindex f5ccfd7fcc..74d7257603 100644--- a/backend/src/@types/knex.d.ts+++ b/backend/src/@types/knex.d.ts@@ -608,6 +608,9 @@ import {TSecretImports,TSecretImportsInsert,TSecretImportsUpdate,+ TSecretReadActivity,+ TSecretReadActivityInsert,+ TSecretReadActivityUpdate,TSecretReferences,TSecretReferencesInsert,TSecretReferencesUpdate,@@ -1427,6 +1430,11 @@ declare module "knex/types/tables" {TSecretReferencesV2Insert,TSecretReferencesV2Update>;+ [TableName.SecretReadActivity]: KnexOriginal.CompositeTableType<+ TSecretReadActivity,+ TSecretReadActivityInsert,+ TSecretReadActivityUpdate+ >;// Junction tables[TableName.SecretV2JnTag]: KnexOriginal.CompositeTableType<TSecretV2TagJunction,diff --git a/backend/src/db/migrations/20261015120000_secret-read-activity.ts b/backend/src/db/migrations/20261015120000_secret-read-activity.tsnew file mode 100644index 0000000000..4032d35073--- /dev/null+++ b/backend/src/db/migrations/20261015120000_secret-read-activity.ts@@ -0,0 +1,25 @@+import { Knex } from "knex";++import { TableName } from "../schemas";++export async function up(knex: Knex): Promise<void> {+ if (!(await knex.schema.hasTable(TableName.SecretReadActivity))) {+ await knex.schema.createTable(TableName.SecretReadActivity, (t) => {+ // A folder or secret id. No foreign keys: each would need its own index, and rows of deleted+ // resources stop being updated, so the daily prune removes them anyway.+ t.uuid("objectId").notNullable();+ t.string("objectType", 32).notNullable();+ t.uuid("actorId").notNullable();+ t.string("actorType", 16).notNullable();+ // Left out of every index so that the daily update is a HOT update and never touches an index.+ t.datetime("lastReadAt", { useTz: true }).notNullable();+ // objectType is part of the key because one secret can be read both with and without its references+ // expanded, and the two credit different secrets.+ t.primary(["objectId", "actorId", "objectType"]);+ });+ }+}++export async function down(knex: Knex): Promise<void> {+ await knex.schema.dropTableIfExists(TableName.SecretReadActivity);+}diff --git a/backend/src/db/schemas/index.ts b/backend/src/db/schemas/index.tsindex e7638966b9..b7f5227ddd 100644--- a/backend/src/db/schemas/index.ts+++ b/backend/src/db/schemas/index.ts@@ -237,6 +237,7 @@ export * from "./secret-change-requests";export * from "./secret-folder-versions";export * from "./secret-folders";export * from "./secret-imports";+export * from "./secret-read-activity";export * from "./secret-references";export * from "./secret-references-v2";export * from "./secret-rotation-v2-secret-mappings";diff --git a/backend/src/db/schemas/models.ts b/backend/src/db/schemas/models.tsindex 44d0893d7b..167d6c1fb6 100644--- a/backend/src/db/schemas/models.ts+++ b/backend/src/db/schemas/models.ts@@ -162,6 +162,7 @@ export enum TableName {DynamicSecretLease = "dynamic_secret_leases",SecretV2 = "secrets_v2",SecretReferenceV2 = "secret_references_v2",+ SecretReadActivity = "secret_read_activity",SecretVersionV2 = "secret_versions_v2",SecretApprovalRequestSecretV2 = "secret_approval_requests_secrets_v2",SecretApprovalRequestSecretTagV2 = "secret_approval_request_secret_tags_v2",diff --git a/backend/src/db/schemas/secret-read-activity.ts b/backend/src/db/schemas/secret-read-activity.tsnew file mode 100644index 0000000000..3095237dd7--- /dev/null+++ b/backend/src/db/schemas/secret-read-activity.ts@@ -0,0 +1,20 @@+// Code generated by automation script, DO NOT EDIT.+// Automated by pulling database and generating zod schema+// To update. Just run npm run generate:schema+// Written by akhilmhdh.++import { z } from "zod";++import { TImmutableDBKeys } from "./models";++export const SecretReadActivitySchema = z.object({+ objectId: z.string().uuid(),+ objectType: z.string(),+ actorId: z.string().uuid(),+ actorType: z.string(),+ lastReadAt: z.date()+});++export type TSecretReadActivity = z.infer<typeof SecretReadActivitySchema>;+export type TSecretReadActivityInsert = Omit<z.input<typeof SecretReadActivitySchema>, TImmutableDBKeys>;+export type TSecretReadActivityUpdate = Partial<Omit<z.input<typeof SecretReadActivitySchema>, TImmutableDBKeys>>;diff --git a/backend/src/ee/services/audit-log/audit-log-clickhouse-dal.ts b/backend/src/ee/services/audit-log/audit-log-clickhouse-dal.tsindex 3f95da2739..a932d3f2ae 100644--- a/backend/src/ee/services/audit-log/audit-log-clickhouse-dal.ts+++ b/backend/src/ee/services/audit-log/audit-log-clickhouse-dal.ts@@ -7,7 +7,7 @@ import { DatabaseError } from "@app/lib/errors";import { logger } from "@app/lib/logger";import { ActorType } from "@app/services/auth/auth-type";-import { TFindLastSecretReadsQuery, TLastSecretRead, TPamAuditLogScope } from "./audit-log-dal";+import { TPamAuditLogScope } from "./audit-log-dal";import { ACTOR_TYPE_TO_METADATA_ID_KEY, EventType, filterableSecretEvents } from "./audit-log-types";type TAuditLogWithProjectName = TAuditLogs & { projectName: string | null };@@ -497,77 +497,11 @@ export const clickhouseAuditLogDALFactory = (clickhouseClient: ClickHouseClient,}};- // No FINAL: a duplicate row left by a retried batch cannot change a max().- const findLastSecretReads = async ({- orgId,- projectId,- eventTypes,- environments,- startDate,- endDate- }: TFindLastSecretReadsQuery): Promise<TLastSecretRead[]> => {- if (!environments.length) return [];- const query = `- SELECT- actor,- if(actor = {userActor:String}, JSONExtractString(actorMetadata, 'userId'), JSONExtractString(actorMetadata, 'identityId')) AS actorId,- JSONExtractString(eventMetadata, 'environment') AS environment,- JSONExtractString(eventMetadata, 'secretPath') AS secretPath,- JSONExtractString(eventMetadata, 'secretId') AS secretId,- JSONExtractString(eventMetadata, 'secretKey') AS secretKey,- toUnixTimestamp64Milli(max(createdAt)) AS lastReadAt- FROM ${tableName}- WHERE orgId = {orgId:UUID}- AND projectId = {projectId:String}- AND createdAt >= {startDate:DateTime64(6, 'UTC')}- AND createdAt < {endDate:DateTime64(6, 'UTC')}- AND eventType IN ({eventTypes:Array(String)})- AND actor IN ({actors:Array(String)})- AND JSONExtractString(eventMetadata, 'environment') IN ({environments:Array(String)})- GROUP BY actor, actorId, environment, secretPath, secretId, secretKey- `;-- try {- const result = await clickhouseClient.query({- query,- query_params: {- orgId,- projectId,- startDate: startDate.replace("Z", ""),- endDate: endDate.replace("Z", ""),- eventTypes,- environments,- userActor: ActorType.USER,- actors: [ActorType.USER, ActorType.IDENTITY]- },- clickhouse_settings: {- max_execution_time: 30- },- format: "JSONEachRow"- });-- const rows = await result.json<- Omit<TLastSecretRead, "lastReadAt"> & { secretId: string; secretKey: string; lastReadAt: string }- >();-- return rows.map((row) => ({- ...row,- secretId: row.secretId || null,- secretKey: row.secretKey || null,- lastReadAt: new Date(Number(row.lastReadAt))- }));- } catch (error) {- logger.error(error, `Failed to find last secret reads in ClickHouse [orgId=${orgId}] [projectId=${projectId}]`);- throw new DatabaseError({ error });- }- };-return {find,countByDateForOrg,countByIdentityAuthMethodForOrg,countByDateAndActorForProject,- countByAuthMethodForProject,- findLastSecretReads+ countByAuthMethodForProject};};diff --git a/backend/src/ee/services/audit-log/audit-log-dal.ts b/backend/src/ee/services/audit-log/audit-log-dal.tsindex e32861a705..299948ac96 100644--- a/backend/src/ee/services/audit-log/audit-log-dal.ts+++ b/backend/src/ee/services/audit-log/audit-log-dal.ts@@ -21,19 +21,6 @@ type TAggregateQuery = {endDate: string;};-export type TFindLastSecretReadsQuery = TAggregateQuery & { environments: string[] };--// The latest read per actor and location. secretId is set only for single-secret reads.-export type TLastSecretRead = {- actor: ActorType.USER | ActorType.IDENTITY;- actorId: string;- environment: string;- secretPath: string;- secretId: string | null;- secretKey: string | null;- lastReadAt: Date;-};-export interface TAuditLogDALFactory extends Omit<TOrmify<TableName.AuditLog>, "find"> {pruneAuditLog: () => Promise<void>;getApproximateRowCount: () => Promise<number>;@@ -60,7 +47,6 @@ export interface TAuditLogDALFactory extends Omit<TOrmify<TableName.AuditLog>, "arg: TAggregateQuery,tx?: knex.Knex) => Promise<{ actor: string; actorMetadata: unknown; count: number }[]>;- findLastSecretReads: (arg: TFindLastSecretReadsQuery, tx?: knex.Knex) => Promise<TLastSecretRead[]>;}type TFindQuery = {@@ -444,44 +430,6 @@ export const auditLogDALFactory = (db: TDbClient) => {return rows as { actor: string; actorMetadata: unknown; count: number }[];};- const findLastSecretReads: TAuditLogDALFactory["findLastSecretReads"] = async (- { orgId, projectId, eventTypes, environments, startDate, endDate },- tx- ) => {- if (!environments.length) return [];- try {- const rows = await (tx || db.replicaNode())(TableName.AuditLog)- .where(`${TableName.AuditLog}.orgId`, orgId)- .where(`${TableName.AuditLog}.projectId`, projectId)- .whereIn(`${TableName.AuditLog}.eventType`, eventTypes)- .whereIn(`${TableName.AuditLog}.actor`, [ActorType.USER, ActorType.IDENTITY])- .whereRaw(`"${TableName.AuditLog}"."createdAt" >= ?::timestamptz`, [startDate])- .whereRaw(`"${TableName.AuditLog}"."createdAt" < ?::timestamptz`, [endDate])- .whereRaw(`"eventMetadata"->>'environment' IN (${environments.map(() => "?").join(", ")})`, environments)- .select(- "actor",- db.raw(`COALESCE("actorMetadata"->>'userId', "actorMetadata"->>'identityId') as "actorId"`),- db.raw(`"eventMetadata"->>'environment' as "environment"`),- db.raw(`"eventMetadata"->>'secretPath' as "secretPath"`),- db.raw(`"eventMetadata"->>'secretId' as "secretId"`),- db.raw(`"eventMetadata"->>'secretKey' as "secretKey"`),- db.raw(`MAX("createdAt") as "lastReadAt"`)- )- .groupByRaw("1, 2, 3, 4, 5, 6")- .timeout(1000 * 120);-- return rows as unknown as TLastSecretRead[];- } catch (error) {- if (error instanceof knex.KnexTimeoutError) {- throw new GatewayTimeoutError({- error,- message: "Loading secret read activity timed out. Try a narrower folder or a shorter time window."- });- }- throw new DatabaseError({ error, name: "Find last secret reads" });- }- };-return {...auditLogOrm,create,@@ -491,7 +439,6 @@ export const auditLogDALFactory = (db: TDbClient) => {find,countByDateAndActor,countByIpAddress,- countByAuthMethod,- findLastSecretReads+ countByAuthMethod};};diff --git a/backend/src/ee/services/secret-read-activity/secret-read-activity-dal.ts b/backend/src/ee/services/secret-read-activity/secret-read-activity-dal.tsnew file mode 100644index 0000000000..f9354ac133--- /dev/null+++ b/backend/src/ee/services/secret-read-activity/secret-read-activity-dal.ts@@ -0,0 +1,109 @@+import { TDbClient } from "@app/db";+import { TableName, TSecretReadActivity, TSecretReadActivityInsert } from "@app/db/schemas";+import { DatabaseError } from "@app/lib/errors";+import { chunkArray } from "@app/lib/fn";++const UPSERT_CHUNK_SIZE = 1000;+const LOOKUP_CHUNK_SIZE = 10_000;+const PRUNE_BATCH_SIZE = 10_000;++export type TSecretReadActivityDALFactory = ReturnType<typeof secretReadActivityDALFactory>;++export const secretReadActivityDALFactory = (db: TDbClient) => {+ const upsertLastReads = async (rows: TSecretReadActivityInsert[]) => {+ // Postgres refuses an upsert that touches the same row twice in one statement, so keep the latest per key.+ const latestByKey = new Map<string, TSecretReadActivityInsert>();+ rows.forEach((row) => {+ const key = `${row.objectId}:${row.actorId}:${row.objectType}`;+ const current = latestByKey.get(key);+ if (!current || new Date(row.lastReadAt) > new Date(current.lastReadAt)) latestByKey.set(key, row);+ });++ try {+ for (const chunk of chunkArray([...latestByKey.values()], UPSERT_CHUNK_SIZE)) {+ // eslint-disable-next-line no-await-in-loop+ await db(TableName.SecretReadActivity)+ .insert(chunk)+ .onConflict(["objectId", "actorId", "objectType"])+ .merge(["lastReadAt"])+ .where(`${TableName.SecretReadActivity}.lastReadAt`, "<", db.raw("excluded.??", ["lastReadAt"]));+ }+ } catch (error) {+ throw new DatabaseError({ error, name: "Upsert secret read activity" });+ }+ };++ const findLastReads = async (objectIds: string[], since: Date) => {+ try {+ const rows: TSecretReadActivity[] = [];+ for (const chunk of chunkArray(objectIds, LOOKUP_CHUNK_SIZE)) {+ // eslint-disable-next-line no-await-in-loop+ const batch = await db+ .replicaNode()(TableName.SecretReadActivity)+ .whereIn("objectId", chunk)+ .where("lastReadAt", ">=", since);+ rows.push(...batch);+ }+ return rows;+ } catch (error) {+ throw new DatabaseError({ error, name: "Find secret read activity" });+ }+ };++ // Walks the primary key in batches: lastReadAt has no index, and searching by it would scan the table+ // once per batch.+ const pruneOlderThan = async (cutoff: Date) => {+ let deleted = 0;+ let cursor: { objectId: string; actorId: string; objectType: string } | undefined;+ try {+ for (;;) {+ const after = cursor;+ // eslint-disable-next-line no-await-in-loop+ const batch = await db(TableName.SecretReadActivity)+ .where((qb) => {+ if (after) {+ void qb.whereRaw(`("objectId", "actorId", "objectType") > (?, ?, ?)`, [+ after.objectId,+ after.actorId,+ after.objectType+ ]);+ }+ })+ .orderBy([{ column: "objectId" }, { column: "actorId" }, { column: "objectType" }])+ .limit(PRUNE_BATCH_SIZE)+ .select("objectId", "actorId", "objectType", "lastReadAt");++ const expired = batch.filter((row) => row.lastReadAt < cutoff);+ if (expired.length) {+ // eslint-disable-next-line no-await-in-loop+ deleted += await db(TableName.SecretReadActivity)+ .whereIn(+ ["objectId", "actorId", "objectType"],+ expired.map((row) => [row.objectId, row.actorId, row.objectType])+ )+ .del();+ }++ if (batch.length < PRUNE_BATCH_SIZE) break;+ cursor = batch[batch.length - 1];+ }+ return deleted;+ } catch (error) {+ throw new DatabaseError({ error, name: "Prune secret read activity" });+ }+ };++ // Reads are only known from when the table was created, so a window cannot start earlier than that.+ const findTrackingStartedAt = async () => {+ try {+ const migration = (await db("infisical_migrations")+ .where("name", "like", "%_secret-read-activity.%")+ .first("migration_time")) as { migration_time: Date } | undefined;+ return migration?.migration_time;+ } catch (error) {+ throw new DatabaseError({ error, name: "Find secret read activity tracking start" });+ }+ };++ return { upsertLastReads, findLastReads, pruneOlderThan, findTrackingStartedAt };+};diff --git a/backend/src/ee/services/secret-read-activity/secret-read-activity-fns.test.ts b/backend/src/ee/services/secret-read-activity/secret-read-activity-fns.test.tsindex 5c9904c33c..4907fe0ea2 100644--- a/backend/src/ee/services/secret-read-activity/secret-read-activity-fns.test.ts+++ b/backend/src/ee/services/secret-read-activity/secret-read-activity-fns.test.ts@@ -1,46 +1,28 @@import { describe, expect, test } from "vitest";-import { TLastSecretRead } from "@app/ee/services/audit-log/audit-log-dal";+import { TSecretReadActivity } from "@app/db/schemas";import { ActorType } from "@app/services/auth/auth-type";import { classifySecretReads, computeSecretReach, indexSecretReads } from "./secret-read-activity-fns";-import { SecretReadStatus, TReachSecret, TSecretReachGraph } from "./secret-read-activity-types";--// prod: /, /app, /app/nested, /shared. dev: /, /consumer (imports prod /app/nested).-const folders: Record<string, { parentId: string | null; environment: string; secretPath: string }> = {- prodRoot: { parentId: null, environment: "prod", secretPath: "/" },- app: { parentId: "prodRoot", environment: "prod", secretPath: "/app" },- nested: { parentId: "app", environment: "prod", secretPath: "/app/nested" },- shared: { parentId: "prodRoot", environment: "prod", secretPath: "/shared" },- devRoot: { parentId: null, environment: "dev", secretPath: "/" },- consumer: { parentId: "devRoot", environment: "dev", secretPath: "/consumer" }-};+import { SecretReadObjectType, SecretReadStatus, TReachSecret, TSecretReachGraph } from "./secret-read-activity-types";const dbPass: TReachSecret = { id: "db-pass", folderId: "shared", key: "DB_PASS" };const dbUrl: TReachSecret = { id: "db-url", folderId: "shared", key: "DB_URL" };const connection: TReachSecret = { id: "connection", folderId: "nested", key: "CONNECTION" };// DB_URL references DB_PASS locally, CONNECTION references DB_URL, and CONNECTION back to DB_PASS closes a cycle.-const referencers: Record<string, TReachSecret[]> = {- [dbPass.id]: [dbUrl],- [dbUrl.id]: [connection],- [connection.id]: [dbPass]-};-+// The consumer folder imports the nested folder, and the edge folder imports the consumer folder.const graph: TSecretReachGraph = {- getReferencers: (secret) => referencers[secret.id] ?? [],- getImporterFolderIds: (folderId) => (folderId === "nested" ? ["consumer"] : []),- getParentFolderId: (folderId) => folders[folderId]?.parentId,- getLocation: (folderId) => folders[folderId]+ getReferencers: (secret) =>+ ({ [dbPass.id]: [dbUrl], [dbUrl.id]: [connection], [connection.id]: [dbPass] })[secret.id] ?? [],+ getImporterFolderIds: (folderId) => ({ nested: ["consumer"], consumer: ["edge"] })[folderId] ?? []};-const read = (overrides: Partial<TLastSecretRead>): TLastSecretRead => ({- actor: ActorType.IDENTITY,+const read = (overrides: Partial<TSecretReadActivity>): TSecretReadActivity => ({+ objectId: "shared",+ objectType: SecretReadObjectType.Folder,actorId: "identity-1",- environment: "prod",- secretPath: "/shared",- secretId: null,- secretKey: null,+ actorType: ActorType.IDENTITY,lastReadAt: new Date("2026-10-10T00:00:00.000Z"),...overrides});@@ -54,64 +36,51 @@ describe("computeSecretReach", () => {expect([...reach.secretIds].sort()).toEqual([connection.id, dbPass.id, dbUrl.id].sort());});- test("includes the folders holding referencing secrets and the folders importing them", () => {- expect([...reach.locations].sort()).toEqual(["dev:/consumer", "prod:/app/nested", "prod:/shared"]);- expect([...reach.environments].sort()).toEqual(["dev", "prod"]);- });-- test("lists parent folders separately, never a folder that already returns the secret", () => {- expect([...reach.parentLocations].sort()).toEqual(["dev:/", "prod:/", "prod:/app"]);+ test("includes the folders holding referencing secrets and every folder importing them, transitively", () => {+ expect([...reach.folderIds].sort()).toEqual(["consumer", "edge", "nested", "shared"]);});});describe("classifySecretReads", () => {const reach = computeSecretReach(dbPass, graph);- const classify = (reads: TLastSecretRead[]) =>+ const classify = (reads: TSecretReadActivity[]) =>classifySecretReads(indexSecretReads(reads), reach, trackedSince).get(`${ActorType.IDENTITY}:identity-1`);- test("counts a bulk read of a folder that imports the secret, whatever the path's trailing slash", () => {- expect(classify([read({ environment: "dev", secretPath: "/consumer/" })])?.status).toBe(SecretReadStatus.Read);+ test("counts a read of a folder that imports the secret through a chain", () => {+ expect(classify([read({ objectId: "edge" })])?.status).toBe(SecretReadStatus.Read);});test("counts a single read of a secret that references it", () => {- expect(classify([read({ secretPath: "/app/nested", secretId: connection.id })])?.status).toBe(+ expect(classify([read({ objectId: connection.id, objectType: SecretReadObjectType.Secret })])?.status).toBe(SecretReadStatus.Read);});- test("ignores a single read of an unrelated secret in the same folder", () => {- expect(classify([read({ secretId: "unrelated", secretKey: "UNRELATED" })])).toBeUndefined();+ test("ignores a single read of an unrelated secret, and a folder read recorded under a secret's id", () => {+ expect(classify([read({ objectId: "unrelated", objectType: SecretReadObjectType.Secret })])).toBeUndefined();+ expect(classify([read({ objectId: dbPass.id, objectType: SecretReadObjectType.Folder })])).toBeUndefined();});- test("counts a single read by name through an import, even when it returned a replicated copy", () => {- expect(- classify([- read({ environment: "dev", secretPath: "/consumer", secretId: "replicated-copy", secretKey: "DB_PASS" })- ])?.status- ).toBe(SecretReadStatus.Read);- });-- test("ignores a single read of a different name in a folder that imports the secret", () => {+ test("counts an unexpanded read only for the secret it returned, not the secrets it references", () => {+ expect(classify([read({ objectId: dbPass.id, objectType: SecretReadObjectType.SecretUnexpanded })])?.status).toBe(+ SecretReadStatus.Read+ );expect(- classify([read({ environment: "dev", secretPath: "/consumer", secretId: "other", secretKey: "OTHER_KEY" })])+ classify([read({ objectId: connection.id, objectType: SecretReadObjectType.SecretUnexpanded })])).toBeUndefined();});- test("treats a read of a parent folder as a possible read", () => {- expect(classify([read({ secretPath: "/" })])?.status).toBe(SecretReadStatus.PossiblyRead);- });-- test("ignores reads from before the secret's path existed", () => {+ test("ignores reads from before the secret was tracked", () => {expect(classify([read({ lastReadAt: new Date("2026-09-30T00:00:00.000Z") })])).toBeUndefined();});- test("prefers a definite read over a later possible one and keeps the latest of each", () => {- const latestRead = new Date("2026-10-08T00:00:00.000Z");+ test("keeps the latest read across all the ways the secret was reached", () => {+ const latestRead = new Date("2026-10-12T00:00:00.000Z");expect(classify([- read({ secretPath: "/", lastReadAt: new Date("2026-10-12T00:00:00.000Z") }),read({ lastReadAt: new Date("2026-10-05T00:00:00.000Z") }),- read({ environment: "dev", secretPath: "/consumer", lastReadAt: latestRead })+ read({ objectId: "consumer", lastReadAt: latestRead }),+ read({ objectId: dbUrl.id, objectType: SecretReadObjectType.Secret })])).toEqual({actorType: ActorType.IDENTITY,diff --git a/backend/src/ee/services/secret-read-activity/secret-read-activity-fns.ts b/backend/src/ee/services/secret-read-activity/secret-read-activity-fns.tsindex 6299d78d37..694012e500 100644--- a/backend/src/ee/services/secret-read-activity/secret-read-activity-fns.ts+++ b/backend/src/ee/services/secret-read-activity/secret-read-activity-fns.ts@@ -1,24 +1,21 @@-import { TLastSecretRead } from "@app/ee/services/audit-log/audit-log-dal";-import { EventType } from "@app/ee/services/audit-log/audit-log-types";+import { TSecretReadActivity } from "@app/db/schemas";import { groupBy, prefixWithSlash, removeTrailingSlash } from "@app/lib/fn";import {+ SecretReadObjectType,SecretReadStatus,TReachSecret,TSecretLocation,TSecretReach,TSecretReachGraph,- TSecretReadActivity+ TSecretReaderType,+ TSecretReadResult} from "./secret-read-activity-types";-export const SECRET_READ_EVENT_TYPES = [- EventType.GET_SECRETS,- EventType.GET_SECRET,- EventType.DASHBOARD_GET_SECRET_VALUE-];+export const normalizeSecretPath = (secretPath: string) => removeTrailingSlash(prefixWithSlash(secretPath));export const toLocationKey = ({ environment, secretPath }: TSecretLocation) =>- `${environment}:${removeTrailingSlash(prefixWithSlash(secretPath))}`;+ `${environment}:${normalizeSecretPath(secretPath)}`;// Imports and references check the reader's own access to the source secret, so a read through either// one uses that access and must count.@@ -47,72 +44,36 @@ export const computeSecretReach = (secret: TReachSecret, graph: TSecretReachGrap}}- const reach: TSecretReach = {- secretKey: secret.key,- secretIds,- locations: new Set(),- parentLocations: new Set(),- environments: new Set()- };- folderIds.forEach((folderId) => {- const location = graph.getLocation(folderId);- if (!location) return;- reach.locations.add(toLocationKey(location));- reach.environments.add(location.environment);- });- folderIds.forEach((folderId) => {- const visited = new Set<string>();- let parentId = graph.getParentFolderId(folderId);- while (parentId && !visited.has(parentId)) {- visited.add(parentId);- const location = graph.getLocation(parentId);- if (location && !reach.locations.has(toLocationKey(location))) {- reach.parentLocations.add(toLocationKey(location));- }- parentId = graph.getParentFolderId(parentId);- }- });- return reach;+ return { secretId: secret.id, secretIds, folderIds };};-export const indexSecretReads = (reads: TLastSecretRead[]) => {- const singleReads = reads.filter((read) => read.secretId);- return {- bySecretId: groupBy(singleReads, (read) => read.secretId as string),- // A single read by name through a replication import returns a copy with its own ID, so single reads are- // also matched by name at the location they were made.- byLocationAndKey: groupBy(singleReads, (read) => `${toLocationKey(read)}:${read.secretKey}`),- byLocation: groupBy(- reads.filter((read) => !read.secretId),- (read) => toLocationKey(read)- )- };-};+export const indexSecretReads = (reads: TSecretReadActivity[]) => groupBy(reads, (read) => read.objectId);-// Reads before trackedSince are ignored: the secret, or the path it lives at, did not exist yet.+// Reads before trackedSince are ignored: the secret or its folder did not exist yet.export const classifySecretReads = (- reads: ReturnType<typeof indexSecretReads>,+ readsByObjectId: ReturnType<typeof indexSecretReads>,reach: TSecretReach,trackedSince: Date) => {- const activity = new Map<string, TSecretReadActivity>();- const record = (status: SecretReadStatus, matches: TLastSecretRead[] = []) => {- matches.forEach((read) => {- if (read.lastReadAt < trackedSince) return;- const key = `${read.actor}:${read.actorId}`;+ const activity = new Map<string, TSecretReadResult>();+ const record = (objectType: SecretReadObjectType, objectId: string) => {+ (readsByObjectId[objectId] ?? []).forEach((read) => {+ if (read.objectType !== objectType || read.lastReadAt < trackedSince) return;+ const key = `${read.actorType}:${read.actorId}`;const current = activity.get(key);- // Definite reads are recorded first, so a possible read never replaces one.- if (!current || (current.status === status && read.lastReadAt > current.lastReadAt)) {- activity.set(key, { actorType: read.actor, actorId: read.actorId, status, lastReadAt: read.lastReadAt });+ if (!current || read.lastReadAt > current.lastReadAt) {+ activity.set(key, {+ actorType: read.actorType as TSecretReaderType,+ actorId: read.actorId,+ status: SecretReadStatus.Read,+ lastReadAt: read.lastReadAt+ });}});};- reach.secretIds.forEach((secretId) => record(SecretReadStatus.Read, reads.bySecretId[secretId]));- reach.locations.forEach((location) => {- record(SecretReadStatus.Read, reads.byLocation[location]);- record(SecretReadStatus.Read, reads.byLocationAndKey[`${location}:${reach.secretKey}`]);- });- reach.parentLocations.forEach((location) => record(SecretReadStatus.PossiblyRead, reads.byLocation[location]));+ reach.secretIds.forEach((secretId) => record(SecretReadObjectType.Secret, secretId));+ record(SecretReadObjectType.SecretUnexpanded, reach.secretId);+ reach.folderIds.forEach((folderId) => record(SecretReadObjectType.Folder, folderId));return activity;};diff --git a/backend/src/ee/services/secret-read-activity/secret-read-activity-service.ts b/backend/src/ee/services/secret-read-activity/secret-read-activity-service.tsindex 27d69a9441..ddf3fec792 100644--- a/backend/src/ee/services/secret-read-activity/secret-read-activity-service.ts+++ b/backend/src/ee/services/secret-read-activity/secret-read-activity-service.ts@@ -1,12 +1,4 @@-import { SecretType, TableName, TSecretFolders } from "@app/db/schemas";-import { TClickHouseAuditLogDALFactory } from "@app/ee/services/audit-log/audit-log-clickhouse-dal";-import { TAuditLogDALFactory } from "@app/ee/services/audit-log/audit-log-dal";-import { hasAuditLogRetention } from "@app/ee/services/audit-log/audit-log-retention-fns";-import {- isAuditLogEventEnabled,- TAuditLogSettingsServiceFactory-} from "@app/ee/services/audit-log/audit-log-settings-service";-import { EventType } from "@app/ee/services/audit-log/audit-log-types";+import { SecretType, TableName, TSecretReadActivityInsert } from "@app/db/schemas";import { DAY_IN_MS } from "@app/ee/services/audit-report/audit-report-fns";import { reviveDates } from "@app/ee/services/insights/insights-fns";import { checkInsightsPermission } from "@app/ee/services/insights/insights-service";@@ -20,8 +12,11 @@ import { ProjectPermissionSecretActions } from "@app/ee/services/permission/projimport { KeyStorePrefixes, KeyStoreTtls, TKeyStoreFactory } from "@app/keystore/keystore";import { getCacheTtl, withCache } from "@app/lib/cache/with-cache";import { getConfig } from "@app/lib/config/env";-import { BadRequestError, NotFoundError } from "@app/lib/errors";-import { groupBy } from "@app/lib/fn";+import { CronJobName, TCronJobFactory } from "@app/lib/cron/cron-job";+import { generateCacheKeyFromData } from "@app/lib/crypto/cache";+import { NotFoundError } from "@app/lib/errors";+import { chunkArray, groupBy } from "@app/lib/fn";+import { logger } from "@app/lib/logger";import { OrderByDirection, OrgServiceActor } from "@app/lib/types";import { ActorType } from "@app/services/auth/auth-type";import { TKmsServiceFactory } from "@app/services/kms/kms-service";@@ -29,42 +24,64 @@ import { KmsDataKey } from "@app/services/kms/kms-types";import { TProjectEnvDALFactory } from "@app/services/project-env/project-env-dal";import { TSecretFolderDALFactory } from "@app/services/secret-folder/secret-folder-dal";import { buildFolderIdMap, buildFolderPath } from "@app/services/secret-folder/secret-folder-fns";-import { TSecretFolderVersionDALFactory } from "@app/services/secret-folder/secret-folder-version-dal";+import { ReservedFolders } from "@app/services/secret-folder/secret-folder-types";import { TSecretImportDALFactory } from "@app/services/secret-import/secret-import-dal";import { getAllSecretReferences } from "@app/services/secret-v2-bridge/secret-reference-fns";import { TSecretV2BridgeDALFactory } from "@app/services/secret-v2-bridge/secret-v2-bridge-dal";+import { TSecretReadActivityDALFactory } from "./secret-read-activity-dal";import {classifySecretReads,computeSecretReach,indexSecretReads,- SECRET_READ_EVENT_TYPES,+ normalizeSecretPath,toLocationKey} from "./secret-read-activity-fns";import {+ SecretReadObjectType,SecretReadStatus,TGetSecretReadActivityDTO,TListUnusedSecretAccessDTO,TReadActivityScope,+ TRecordBulkSecretReadDTO,+ TRecordSecretReadDTO,TSecretReachGraph,+ TSecretReaderType,+ TSecretReadStreamEntry,TUnusedSecretAccess} from "./secret-read-activity-types";const SECRET_READ_ACTIVITY_WINDOW_DAYS = 30;+const SECRET_READ_RETENTION_DAYS = 90;+const SECRET_READ_GATE_TTL_SECONDS = 25 * 60 * 60;+const SECRET_READ_STREAM_MAX_LENGTH = 100_000;+const SECRET_READ_FLUSH_BATCH_SIZE = 1000;+const SECRET_READ_FLUSH_MAX_ENTRIES = 50_000;+const SECRET_READ_CONSUMER_LOCK_TTL_MS = 60_000;+const SECRET_READ_STREAM_ENTRY_MAX_SECRETS = 1000;type TSecretReadActivityServiceFactoryDep = {permissionService: Pick<TPermissionServiceFactory, "getProjectPermission" | "getProjectPermissions">;licenseService: Pick<TLicenseServiceFactory, "getPlan">;- auditLogSettingsService: Pick<TAuditLogSettingsServiceFactory, "getEffectiveSettings">;- auditLogDAL: Pick<TAuditLogDALFactory, "findLastSecretReads">;- clickhouseAuditLogDAL?: Pick<TClickHouseAuditLogDALFactory, "findLastSecretReads">;+ secretReadActivityDAL: TSecretReadActivityDALFactory;folderDAL: Pick<TSecretFolderDALFactory, "findByProjectId">;- folderVersionDAL: Pick<TSecretFolderVersionDALFactory, "find">;projectEnvDAL: Pick<TProjectEnvDALFactory, "find">;- secretImportDAL: Pick<TSecretImportDALFactory, "find">;+ secretImportDAL: Pick<TSecretImportDALFactory, "find" | "findByIds">;secretV2BridgeDAL: Pick<TSecretV2BridgeDALFactory, "find" | "findSecretReferencesByProjectId">;kmsService: Pick<TKmsServiceFactory, "createCipherPairWithDataKey">;- keyStore: Pick<TKeyStoreFactory, "setItemWithExpiry" | "getItem" | "ttl">;+ keyStore: Pick<+ TKeyStoreFactory,+ | "setItemWithExpiry"+ | "getItem"+ | "ttl"+ | "setItemWithExpiryNX"+ | "deleteItem"+ | "streamAdd"+ | "streamCollect"+ | "streamTrim"+ | "acquireLock"+ >;+ cronJob: TCronJobFactory;};export type TSecretReadActivityServiceFactory = ReturnType<typeof secretReadActivityServiceFactory>;@@ -72,36 +89,16 @@ export type TSecretReadActivityServiceFactory = ReturnType<typeof secretReadActiexport const secretReadActivityServiceFactory = ({permissionService,licenseService,- auditLogSettingsService,- auditLogDAL,- clickhouseAuditLogDAL,+ secretReadActivityDAL,folderDAL,- folderVersionDAL,projectEnvDAL,secretImportDAL,secretV2BridgeDAL,kmsService,- keyStore+ keyStore,+ cronJob}: TSecretReadActivityServiceFactoryDep) => {- const $getReadActivity = async ({ projectId, orgId, windowDays, secret: requestedSecret }: TReadActivityScope) => {- const settings = await auditLogSettingsService.getEffectiveSettings(orgId, projectId);- if (!isAuditLogEventEnabled(settings, EventType.GET_SECRETS, projectId)) {- throw new BadRequestError({- message:- "Secret reads are not recorded for this project because data access audit logging is turned off. Turn it on in the project's audit log settings to see read activity."- });- }- const plan = await licenseService.getPlan(orgId);- if (!hasAuditLogRetention(plan)) {- throw new BadRequestError({- message: "Your plan does not retain audit logs, so secret read activity is unavailable."- });- }- const now = new Date();- const windowStart = new Date(- now.getTime() - Math.min(windowDays, plan.auditLogsRetentionDays ?? windowDays) * DAY_IN_MS- );-+ const $loadProjectTree = async (projectId: string) => {const [folders, environments] = await Promise.all([folderDAL.findByProjectId(projectId),projectEnvDAL.find({ projectId })@@ -117,30 +114,323 @@ export const secretReadActivityServiceFactory = ({const env = folderById[folderId] && envById.get(folderById[folderId].envId);return env ? { environment: env.slug, secretPath: pathOf(folderId) } : undefined;};- const ancestryOf = (folderId: string) => {- const chain: string[] = [];- let folder: TSecretFolders | undefined = folderById[folderId];- while (folder && !chain.includes(folder.id)) {- chain.push(folder.id);- folder = folder.parentId ? folderById[folder.parentId] : undefined;+ const folderIdByLocation = new Map(+ folders.flatMap((folder) => {+ const location = locationOf(folder.id);+ return location ? [[toLocationKey(location), folder.id] as const] : [];+ })+ );+ return { folders, folderById, envById, pathOf, locationOf, folderIdByLocation };+ };++ let trackingStartedAt: Promise<Date | undefined> | undefined;+ const $getTrackingStartedAt = () => {+ trackingStartedAt ??= secretReadActivityDAL.findTrackingStartedAt().catch((error) => {+ trackingStartedAt = undefined;+ throw error;+ });+ return trackingStartedAt;+ };++ const $isTrackedReader = (actor: ActorType): actor is TSecretReaderType =>+ actor === ActorType.USER || actor === ActorType.IDENTITY;++ // Called once a read's response is built and never awaited by it. Only the first matching read per reader+ // per day reaches the stream, so a client polling every minute costs one Redis call per read.+ const $record = async (+ { actor, actorId, projectId }: { actor: TSecretReaderType; actorId: string; projectId: string },+ scope: string,+ reads: Pick<+ TSecretReadStreamEntry,+ "environment" | "secretPath" | "recursive" | "secretIds" | "referencesExpanded"+ >[]+ ) => {+ const readAt = new Date();+ const gateKey = KeyStorePrefixes.SecretReadActivityGate(readAt.toISOString().slice(0, 10), actorId, scope);++ try {+ if (!(await keyStore.setItemWithExpiryNX(gateKey, SECRET_READ_GATE_TTL_SECONDS, 1))) return;+ for (const read of reads) {+ const entry: TSecretReadStreamEntry = {+ actorType: actor,+ actorId,+ projectId,+ readAt: readAt.toISOString(),+ ...read+ };+ // eslint-disable-next-line no-await-in-loop+ await keyStore.streamAdd(+ KeyStorePrefixes.SecretReadActivityStream,+ "*",+ { data: JSON.stringify(entry) },+ SECRET_READ_STREAM_MAX_LENGTH+ );+ }+ } catch (error) {+ // Left in place, the gate would hide the rest of today's reads of this scope.+ await keyStore.deleteItem(gateKey).catch(() => undefined);+ logger.warn(error, `Failed to record secret read [projectId=${projectId}] [actorId=${actorId}]`);+ }+ };++ const recordRead = async ({ actor, actorId, projectId, secretIds, referencesExpanded }: TRecordSecretReadDTO) => {+ if (!$isTrackedReader(actor) || !secretIds.length) return;++ const uniqueSecretIds = [...new Set(secretIds)].sort();+ await $record(+ { actor, actorId, projectId },+ `secrets:${String(referencesExpanded)}:${generateCacheKeyFromData(uniqueSecretIds)}`,+ chunkArray(uniqueSecretIds, SECRET_READ_STREAM_ENTRY_MAX_SECRETS).map((chunk) => ({+ secretIds: chunk,+ referencesExpanded+ }))+ );+ };++ // A read that returned whole folders (no filters, imports and references included) is recorded against+ // them. A narrower one is recorded against the secrets it returned, so the secrets it left out are not+ // credited.+ const recordBulkRead = async ({+ actor,+ actorId,+ projectId,+ environment,+ secretPath,+ recursive,+ isFiltered,+ includeImports,+ expandSecretReferences,+ notModified,+ secrets,+ imports = []+ }: TRecordBulkSecretReadDTO) => {+ if (!$isTrackedReader(actor)) return;++ if (!isFiltered && includeImports && expandSecretReferences) {+ const normalizedPath = normalizeSecretPath(secretPath);+ await $record(+ { actor, actorId, projectId },+ `folder:${projectId}:${environment}:${normalizedPath}:${String(recursive)}`,+ [{ environment, secretPath: normalizedPath, recursive }]+ );+ return;+ }++ // A 304 carries no secrets. ETags are stored per UTC day, so each client's first read of the day gets a+ // full response, and that one is recorded.+ if (notModified) return;++ await recordRead({+ actor,+ actorId,+ projectId,+ secretIds: [...secrets, ...imports.flatMap((secretImport) => secretImport.secrets)]+ .filter((secret) => !secret.secretValueHidden)+ .map((secret) => secret.id),+ referencesExpanded: expandSecretReferences+ });+ };++ // A single read through a replication import returns the copy, which is checked against the reader's access+ // to the secret it was copied from, so the read is credited to that secret.+ const $findReplicatedSourceIds = async (tree: Awaited<ReturnType<typeof $loadProjectTree>>, secretIds: string[]) => {+ const importIdByReplicaFolderId = new Map(+ tree.folders+ .filter((folder) => folder.isReserved && folder.name.startsWith(ReservedFolders.SecretReplication))+ .map((folder) => [folder.id, folder.name.slice(ReservedFolders.SecretReplication.length)])+ );+ if (!importIdByReplicaFolderId.size) return new Map<string, string>();++ // Looked up by folder, so the query is bounded by the replicas rather than by how many secrets were read.+ const readSecretIds = new Set(secretIds);+ const copies = (+ await secretV2BridgeDAL.find({+ $in: { [`${TableName.SecretV2}.folderId` as "folderId"]: [...importIdByReplicaFolderId.keys()] }+ })+ ).filter((secret) => readSecretIds.has(secret.id));+ if (!copies.length) return new Map<string, string>();++ const imports = await secretImportDAL.findByIds([+ ...new Set(copies.map((copy) => importIdByReplicaFolderId.get(copy.folderId) as string))+ ]);+ const sourceFolderIdByImportId = new Map(+ imports.flatMap((secretImport) => {+ const folderId = tree.folderIdByLocation.get(+ toLocationKey({ environment: secretImport.importEnv.slug, secretPath: secretImport.importPath })+ );+ return folderId ? [[secretImport.id, folderId] as const] : [];+ })+ );+ const sources = await secretV2BridgeDAL.find({+ $in: { [`${TableName.SecretV2}.folderId` as "folderId"]: [...new Set(sourceFolderIdByImportId.values())] },+ [`${TableName.SecretV2}.type` as "type"]: SecretType.Shared+ });+ const sourceIdByFolderAndKey = new Map(sources.map((source) => [`${source.folderId}:${source.key}`, source.id]));++ return new Map(+ copies.flatMap((copy) => {+ const sourceFolderId = sourceFolderIdByImportId.get(importIdByReplicaFolderId.get(copy.folderId) as string);+ const sourceId = sourceFolderId && sourceIdByFolderAndKey.get(`${sourceFolderId}:${copy.key}`);+ return sourceId ? [[copy.id, sourceId] as const] : [];+ })+ );+ };++ const $resolveProjectReads = async (projectId: string, reads: TSecretReadStreamEntry[]) => {+ const tree = await $loadProjectTree(projectId);+ const subtrees = new Map<string, string[]>();+ const subtreeOf = (folderId: string) => {+ if (!subtrees.has(folderId)) {+ const prefix = tree.pathOf(folderId) === "/" ? "/" : `${tree.pathOf(folderId)}/`;+ const { envId } = tree.folderById[folderId];+ subtrees.set(+ folderId,+ tree.folders+ .filter(+ (folder) =>+ folder.envId === envId &&+ !folder.isReserved &&+ (folder.id === folderId || tree.pathOf(folder.id).startsWith(prefix))+ )+ .map((folder) => folder.id)+ );}- return chain;+ return subtrees.get(folderId) as string[];};- const requestedFolder =+ const rows: TSecretReadActivityInsert[] = [];+ const toRow = (read: TSecretReadStreamEntry, objectType: SecretReadObjectType, objectId: string) => ({+ objectId,+ objectType,+ actorId: read.actorId,+ actorType: read.actorType,+ lastReadAt: new Date(read.readAt)+ });++ reads.forEach((read) => {+ if (read.environment === undefined || read.secretPath === undefined) return;+ const folderId = tree.folderIdByLocation.get(+ toLocationKey({ environment: read.environment, secretPath: read.secretPath })+ );+ if (!folderId) return;+ (read.recursive ? subtreeOf(folderId) : [folderId]).forEach((id) =>+ rows.push(toRow(read, SecretReadObjectType.Folder, id))+ );+ });++ const secretReads = reads.filter((read) => read.secretIds?.length);+ if (secretReads.length) {+ const sourceIdByCopyId = await $findReplicatedSourceIds(+ tree,+ secretReads.flatMap((read) => read.secretIds as string[])+ );+ secretReads.forEach((read) => {+ const objectType = read.referencesExpanded+ ? SecretReadObjectType.Secret+ : SecretReadObjectType.SecretUnexpanded;+ (read.secretIds as string[]).forEach((secretId) =>+ rows.push(toRow(read, objectType, sourceIdByCopyId.get(secretId) ?? secretId))+ );+ });+ }+ return rows;+ };++ // Entries are trimmed only after they are written, so a failed batch is retried on the next run; the upsert+ // makes the retry harmless.+ const $flushReadStream = async () => {+ const { entries, lastId } = await keyStore.streamCollect(+ KeyStorePrefixes.SecretReadActivityStream,+ SECRET_READ_FLUSH_BATCH_SIZE,+ SECRET_READ_FLUSH_MAX_ENTRIES+ );+ if (!entries.length || !lastId) return;++ if (entries.length >= SECRET_READ_FLUSH_MAX_ENTRIES) {+ logger.warn(+ `secret-read-activity: stream backlog hit the collection ceiling [collected=${entries.length}] [maxEntries=${SECRET_READ_FLUSH_MAX_ENTRIES}]; the oldest reads risk being dropped by the stream's MAXLEN cap`+ );+ }++ const reads = entries.flatMap(([, fields]) => {+ try {+ return [JSON.parse(fields[fields.indexOf("data") + 1]) as TSecretReadStreamEntry];+ } catch (error) {+ logger.error(error, "secret-read-activity: Skipping unparseable stream entry");+ return [];+ }+ });++ const rows: TSecretReadActivityInsert[] = [];+ for (const [projectId, projectReads] of Object.entries(groupBy(reads, (read) => read.projectId))) {+ // eslint-disable-next-line no-await-in-loop+ rows.push(...(await $resolveProjectReads(projectId, projectReads)));+ }+ await secretReadActivityDAL.upsertLastReads(rows);+ await keyStore.streamTrim(KeyStorePrefixes.SecretReadActivityStream, lastId, true);+ logger.info(`secret-read-activity: Flushed read stream [entries=${entries.length}] [rows=${rows.length}]`);+ };++ // A run can outlast the next fire under load. The upsert makes overlap harmless, but it doubles the work when+ // the stream is already behind, so a run that finds the lock held skips and leaves the backlog to the next one.+ const $consumeReadStream = async () => {+ let lock;+ try {+ lock = await keyStore.acquireLock(+ [KeyStorePrefixes.SecretReadActivityConsumerLock],+ SECRET_READ_CONSUMER_LOCK_TTL_MS,+ { retryCount: 0 }+ );+ } catch {+ return;+ }+ try {+ await $flushReadStream();+ } finally {+ await lock.release();+ }+ };++ const init = () => {+ const appCfg = getConfig();+ cronJob.register({+ name: CronJobName.SecretReadActivityFlush,+ pattern: "* * * * *",+ runHashTtlS: 60 * 60,+ enabled: !appCfg.isSecondaryInstance,+ handler: $consumeReadStream+ });+ cronJob.register({+ name: CronJobName.SecretReadActivityPrune,+ pattern: "15 1 * * *",+ runHashTtlS: 3 * 24 * 60 * 60,+ enabled: !appCfg.isSecondaryInstance,+ handler: async () => {+ const deleted = await secretReadActivityDAL.pruneOlderThan(+ new Date(Date.now() - SECRET_READ_RETENTION_DAYS * DAY_IN_MS)+ );+ logger.info(`cron[${CronJobName.SecretReadActivityPrune}]: pruned ${deleted} rows`);+ }+ });+ };++ const $getReadActivity = async ({ projectId, windowDays, secret: requestedSecret }: TReadActivityScope) => {+ const windowStart = new Date(Date.now() - windowDays * DAY_IN_MS);+ const tree = await $loadProjectTree(projectId);+ const { folders, folderById, locationOf, pathOf } = tree;++ const requestedFolderId =requestedSecret &&- folders.find(- (folder) =>- locationOf(folder.id)?.environment === requestedSecret.environment &&- pathOf(folder.id) === requestedSecret.secretPath+ tree.folderIdByLocation.get(+ toLocationKey({ environment: requestedSecret.environment, secretPath: requestedSecret.secretPath }));- if (requestedSecret && !requestedFolder) {+ if (requestedSecret && !requestedFolderId) {throw new NotFoundError({message: `Folder with path '${requestedSecret.secretPath}' in environment '${requestedSecret.environment}' not found`});}- const scopeFolderIds = requestedFolder- ? [requestedFolder.id]+ const scopeFolderIds = requestedFolderId+ ? [requestedFolderId]: folders.filter((folder) => !folder.isReserved).map((folder) => folder.id);const referencesByLocation = groupBy(@@ -208,9 +498,7 @@ export const secretReadActivityServiceFactory = ({toLocationKey({ environment: folderById[folderId].envId, secretPath: pathOf(folderId) })] ?? []).map((secretImport) => secretImport.folderId)- : [],- getParentFolderId: (folderId) => folderById[folderId]?.parentId,- getLocation: locationOf+ : []};const targets = scopeFolderIds.flatMap((folderId) => {@@ -224,36 +512,23 @@ export const secretReadActivityServiceFactory = ({.map((secret) => ({ secret, ...location }));});const reaches = targets.map(({ secret }) => computeSecretReach(secret, graph));- const auditLogStore =- getConfig().CLICKHOUSE_AUDIT_LOG_ENABLED && clickhouseAuditLogDAL ? clickhouseAuditLogDAL : auditLogDAL;const reads = indexSecretReads(- await auditLogStore.findLastSecretReads({- orgId,- projectId,- eventTypes: SECRET_READ_EVENT_TYPES,- environments: [...new Set(reaches.flatMap((reach) => [...reach.environments]))],- startDate: windowStart.toISOString(),- endDate: now.toISOString()- })- );-- // A read can only match the secret's current path once that path exists, so the window starts no- // earlier than the last time the secret, its folders or its environment were created, renamed or moved.- // Folder versions are written on each of those.- const lastPathChangeAt = new Map<string, number>();- const chainFolderIds = [...new Set(scopeFolderIds.flatMap(ancestryOf))];- chainFolderIds.forEach((folderId) => lastPathChangeAt.set(folderId, folderById[folderId].createdAt.getTime()));- (await folderVersionDAL.find({ $in: { folderId: chainFolderIds } })).forEach(({ folderId, createdAt }) =>- lastPathChangeAt.set(folderId, Math.max(lastPathChangeAt.get(folderId) ?? 0, createdAt.getTime()))+ await secretReadActivityDAL.findLastReads(+ [...new Set(reaches.flatMap((reach) => [...reach.folderIds, ...reach.secretIds]))],+ windowStart+ ));+ // Reads are matched by folder and secret ids, so renames do not matter. A move creates new ids, so the+ // window starts no earlier than the secret and its folder were created, nor before tracking began.+ const trackingStart = (await $getTrackingStartedAt())?.getTime() ?? 0;return targets.map(({ secret, environment, secretPath }, index) => {const trackedSince = new Date(Math.max(windowStart.getTime(),+ trackingStart,secret.createdAt.getTime(),- envById.get(folderById[secret.folderId].envId)?.updatedAt.getTime() ?? 0,- ...ancestryOf(secret.folderId).map((folderId) => lastPathChangeAt.get(folderId) ?? 0)+ folderById[secret.folderId].createdAt.getTime()));return {@@ -274,7 +549,6 @@ export const secretReadActivityServiceFactory = ({const [activity] = await $getReadActivity({projectId,- orgId: actor.orgId,windowDays: SECRET_READ_ACTIVITY_WINDOW_DAYS,secret: { environment, secretPath, secretName }});@@ -295,7 +569,7 @@ export const secretReadActivityServiceFactory = ({// Everyone who can read a secret's value but has not read it, ranked by the secrets with the most of them.const $computeUnusedSecretAccess = async (projectId: string, orgId: string) => {- const activities = await $getReadActivity({ projectId, orgId, windowDays: SECRET_READ_ACTIVITY_WINDOW_DAYS });+ const activities = await $getReadActivity({ projectId, windowDays: SECRET_READ_ACTIVITY_WINDOW_DAYS });const { userPermissions, identityPermissions } = await permissionService.getProjectPermissions(projectId, orgId);// Principals with identical rules always get the same answer, so each distinct rule set is checked once.const principalGroups = Object.values(@@ -328,8 +602,7 @@ export const secretReadActivityServiceFactory = ({).flat().flatMap((principal) => {- const read = reads.get(`${principal.actorType}:${principal.id}`);- if (read?.status === SecretReadStatus.Read) return [];+ if (reads.has(`${principal.actorType}:${principal.id}`)) return [];return {secretKey: secret.key,environment,@@ -337,8 +610,8 @@ export const secretReadActivityServiceFactory = ({principalType: principal.actorType,principalId: principal.id,principalName: principal.name,- readStatus: read?.status ?? SecretReadStatus.NoReads,- lastReadAt: read?.lastReadAt ?? null,+ readStatus: SecretReadStatus.NoReads,+ lastReadAt: null,trackedSince};})@@ -385,5 +658,5 @@ export const secretReadActivityServiceFactory = ({};};- return { getSecretReadActivity, listUnusedSecretAccess };+ return { init, recordRead, recordBulkRead, getSecretReadActivity, listUnusedSecretAccess };};diff --git a/backend/src/ee/services/secret-read-activity/secret-read-activity-types.ts b/backend/src/ee/services/secret-read-activity/secret-read-activity-types.tsindex 655613c9ce..7f7704f66d 100644--- a/backend/src/ee/services/secret-read-activity/secret-read-activity-types.ts+++ b/backend/src/ee/services/secret-read-activity/secret-read-activity-types.ts@@ -1,29 +1,35 @@-import { TLastSecretRead } from "@app/ee/services/audit-log/audit-log-dal";import { OrderByDirection } from "@app/lib/types";+import { ActorType } from "@app/services/auth/auth-type";export enum SecretReadStatus {Read = "read",- // Only a parent folder was read. That returns the secret only if the read was recursive, which the- // audit log does not record.- PossiblyRead = "possibly-read",NoReads = "no-reads"}-export type TSecretReadActivity = {- actorType: TLastSecretRead["actor"];+// A bulk read that returned whole folders is recorded against them. Anything narrower, and every single read,+// is recorded against the secrets it returned, as unexpanded when their references were not resolved: those+// handed over only the secret itself, not the values it references.+export enum SecretReadObjectType {+ Folder = "folder",+ Secret = "secret",+ SecretUnexpanded = "secret-unexpanded"+}++export type TSecretReaderType = ActorType.USER | ActorType.IDENTITY;++export type TSecretReadResult = {+ actorType: TSecretReaderType;actorId: string;status: SecretReadStatus;lastReadAt: Date;};-// Everything whose read hands over this secret's value: the secret, every secret that references it,+// Everything whose read hands over this secret's value: the secret and every secret that references it,// and every folder that holds one of those or imports one that does.export type TSecretReach = {- secretKey?: string;+ secretId: string;secretIds: Set<string>;- locations: Set<string>;- parentLocations: Set<string>;- environments: Set<string>;+ folderIds: Set<string>;};export type TReachSecret = { id: string; folderId: string; key?: string };@@ -33,8 +39,44 @@ export type TSecretLocation = { environment: string; secretPath: string };export type TSecretReachGraph = {getReferencers: (secret: TReachSecret) => TReachSecret[];getImporterFolderIds: (folderId: string) => string[];- getParentFolderId: (folderId: string) => string | null | undefined;- getLocation: (folderId: string) => TSecretLocation | undefined;+};++type TSecretReader = {+ actor: ActorType;+ actorId: string;+ projectId: string;+};++export type TRecordSecretReadDTO = TSecretReader & {+ secretIds: string[];+ referencesExpanded: boolean;+};++type TReturnedSecret = { id: string; secretValueHidden: boolean };++export type TRecordBulkSecretReadDTO = TSecretReader & {+ environment: string;+ secretPath: string;+ recursive: boolean;+ isFiltered: boolean;+ includeImports: boolean;+ expandSecretReferences: boolean;+ notModified?: boolean;+ secrets: TReturnedSecret[];+ imports?: { secrets: TReturnedSecret[] }[];+};++// What the request path appends to the stream; the consumer resolves it to folders and secrets.+export type TSecretReadStreamEntry = {+ actorType: TSecretReaderType;+ actorId: string;+ projectId: string;+ readAt: string;+ environment?: string;+ secretPath?: string;+ recursive?: boolean;+ secretIds?: string[];+ referencesExpanded?: boolean;};export type TGetSecretReadActivityDTO = {@@ -47,7 +89,6 @@ export type TGetSecretReadActivityDTO = {// Without a secret, the whole project is in scope.export type TReadActivityScope = {projectId: string;- orgId: string;windowDays: number;secret?: Omit<TGetSecretReadActivityDTO, "projectId">;};@@ -56,7 +97,7 @@ export type TUnusedSecretAccess = {secretKey: string;environment: string;secretPath: string;- principalType: TLastSecretRead["actor"];+ principalType: TSecretReaderType;principalId: string;principalName: string;readStatus: SecretReadStatus;diff --git a/backend/src/keystore/keystore.ts b/backend/src/keystore/keystore.tsindex c78303d43f..4ef01f2020 100644--- a/backend/src/keystore/keystore.ts+++ b/backend/src/keystore/keystore.ts@@ -229,6 +229,10 @@ export const KeyStorePrefixes = {`telemetry-agg-stream:${event}:${bucketId}` as const,AuditLogStreamFlushDebounce: (streamId: string) => `audit-log-stream:${streamId}:flush-debounce` as const,+ SecretReadActivityGate: (day: string, actorId: string, scope: string) =>+ `secret-read-activity:${day}:${actorId}:${scope}` as const,+ SecretReadActivityStream: "secret-read-activity-stream" as const,+ SecretReadActivityConsumerLock: "secret-read-activity:consumer-lock" as const,AuditLogIngestConsumerLock: "audit-log-ingest:consumer-lock" as const,// period is a YYYY-MM stamp so the monthly notice can only go out once per org per monthdiff --git a/backend/src/lib/cron/cron-job.ts b/backend/src/lib/cron/cron-job.tsindex d86860aab2..28b37707e9 100644--- a/backend/src/lib/cron/cron-job.ts+++ b/backend/src/lib/cron/cron-job.ts@@ -51,7 +51,9 @@ export const CronJobName = {EventOutboxStaleClaimSweeper: "event-outbox-stale-claim-sweeper",EventOutboxCleanup: "event-outbox-cleanup",SecretScanningStuckScanReaper: "secret-scanning-stuck-scan-reaper",- InstanceUpdateCheck: "instance-update-check"+ InstanceUpdateCheck: "instance-update-check",+ SecretReadActivityFlush: "secret-read-activity-flush",+ SecretReadActivityPrune: "secret-read-activity-prune"} as const;const JITTER_INTERVAL_FRACTION = 0.25; // keep below ~0.48 so 30s retry fits before next 1-min firediff --git a/backend/src/server/routes/index.ts b/backend/src/server/routes/index.tsindex 36daf1b4e9..7798af59d1 100644--- a/backend/src/server/routes/index.ts+++ b/backend/src/server/routes/index.ts@@ -207,6 +207,7 @@ import { secretChangeGlobalPolicyBridgeDALFactory } from "@app/ee/services/secreimport { secretChangeGlobalPolicyBridgeServiceFactory } from "@app/ee/services/secret-change-global-policy-bridge/secret-change-global-policy-bridge-service";import { secretChangeGlobalRequestBridgeDALFactory } from "@app/ee/services/secret-change-global-request-bridge/secret-change-global-request-bridge-dal";import { secretChangeGlobalRequestBridgeServiceFactory } from "@app/ee/services/secret-change-global-request-bridge/secret-change-global-request-bridge-service";+import { secretReadActivityDALFactory } from "@app/ee/services/secret-read-activity/secret-read-activity-dal";import { secretReadActivityServiceFactory } from "@app/ee/services/secret-read-activity/secret-read-activity-service";import { secretReplicationServiceFactory } from "@app/ee/services/secret-replication/secret-replication-service";import { secretRotationV2DALFactory } from "@app/ee/services/secret-rotation-v2/secret-rotation-v2-dal";@@ -3484,16 +3485,14 @@ export const registerRoutes = async (const secretReadActivityService = secretReadActivityServiceFactory({permissionService,licenseService,- auditLogSettingsService,- auditLogDAL,- clickhouseAuditLogDAL,+ secretReadActivityDAL: secretReadActivityDALFactory(db),folderDAL,- folderVersionDAL,projectEnvDAL,secretImportDAL,secretV2BridgeDAL,kmsService,- keyStore+ keyStore,+ cronJob});const insightsService = insightsServiceFactory({@@ -4566,6 +4565,7 @@ export const registerRoutes = async (projectCleanupQueue.init();usageEventQueue.init();healthAlert.init();+ secretReadActivityService.init();alertQueue.init();eventOutboxQueue.init();auditLogStreamOutboxQueue.init();diff --git a/backend/src/server/routes/v1/dashboard-router.ts b/backend/src/server/routes/v1/dashboard-router.tsindex 24a369386e..677ac77246 100644--- a/backend/src/server/routes/v1/dashboard-router.ts+++ b/backend/src/server/routes/v1/dashboard-router.ts@@ -2007,6 +2007,14 @@ export const registerDashboardRouter = async (server: FastifyZodProvider) => {return { valueOverride: secret.secretValue };}+ void server.services.secretReadActivity.recordRead({+ actor: req.permission.type,+ actorId: req.permission.id,+ projectId,+ secretIds: [secret.id],+ referencesExpanded: false+ });+await server.services.auditLog.createAuditLog({projectId,...req.auditLogInfo,diff --git a/backend/src/server/routes/v3/deprecated-secret-router.ts b/backend/src/server/routes/v3/deprecated-secret-router.tsindex b63f5b84b2..13b2b55704 100644--- a/backend/src/server/routes/v3/deprecated-secret-router.ts+++ b/backend/src/server/routes/v3/deprecated-secret-router.ts@@ -346,6 +346,24 @@ export const registerDeprecatedSecretRouter = async (server: FastifyZodProvider)const { secrets, imports, etag, notModified } = result;+ // Recorded before the 304 return: a client polling with an ETag is still using the secrets it holds.+ if (req.query.viewSecretValue) {+ void server.services.secretReadActivity.recordBulkRead({+ actor: req.permission.type,+ actorId: req.permission.id,+ projectId: workspaceId,+ environment,+ secretPath,+ recursive: req.query.recursive,+ isFiltered: Boolean(req.query.tagSlugs?.length || req.query.metadataFilter?.length),+ includeImports: req.query.include_imports,+ expandSecretReferences: req.query.expandSecretReferences,+ notModified,+ secrets,+ imports+ });+ }+if (etag) {void reply.header("ETag", etag);}@@ -521,6 +539,16 @@ export const registerDeprecatedSecretRouter = async (server: FastifyZodProvider)version: req.query.version});+ if (req.query.viewSecretValue) {+ void server.services.secretReadActivity.recordRead({+ actor: req.permission.type,+ actorId: req.permission.id,+ projectId: workspaceId,+ secretIds: [secret.id],+ referencesExpanded: req.query.expandSecretReferences+ });+ }+await server.services.auditLog.createAuditLog({projectId: secret.workspace,...req.auditLogInfo,diff --git a/backend/src/server/routes/v4/secret-router.ts b/backend/src/server/routes/v4/secret-router.tsindex 7236943768..a198743f04 100644--- a/backend/src/server/routes/v4/secret-router.ts+++ b/backend/src/server/routes/v4/secret-router.ts@@ -226,6 +226,24 @@ export const registerSecretRouter = async (server: FastifyZodProvider) => {const { secrets, imports, etag, notModified } = result;+ // Recorded before the 304 return: a client polling with an ETag is still using the secrets it holds.+ if (req.query.viewSecretValue) {+ void server.services.secretReadActivity.recordBulkRead({+ actor: req.permission.type,+ actorId: req.permission.id,+ projectId,+ environment,+ secretPath,+ recursive: req.query.recursive,+ isFiltered: Boolean(req.query.tagSlugs?.length || req.query.metadataFilter?.length),+ includeImports: req.query.includeImports,+ expandSecretReferences: req.query.expandSecretReferences,+ notModified,+ secrets,+ imports+ });+ }+if (etag) {void reply.header("ETag", etag);}@@ -382,6 +400,16 @@ export const registerSecretRouter = async (server: FastifyZodProvider) => {version: req.query.version});+ if (req.query.viewSecretValue) {+ void server.services.secretReadActivity.recordRead({+ actor: req.permission.type,+ actorId: req.permission.id,+ projectId,+ secretIds: [secret.id],+ referencesExpanded: req.query.expandSecretReferences+ });+ }+await server.services.auditLog.createAuditLog({projectId,...req.auditLogInfo,@@ -1577,6 +1605,16 @@ export const registerSecretRouter = async (server: FastifyZodProvider) => {environment});+ if (value !== undefined) {+ void server.services.secretReadActivity.recordRead({+ actor: req.permission.type,+ actorId: req.permission.id,+ projectId,+ secretIds: [secret.id],+ referencesExpanded: true+ });+ }+await server.services.auditLog.createAuditLog({projectId,...req.auditLogInfo,diff --git a/frontend/src/components/secrets/SecretReadStatusBadge.tsx b/frontend/src/components/secrets/SecretReadStatusBadge.tsxindex 9bf54f3fe7..5d38d02489 100644--- a/frontend/src/components/secrets/SecretReadStatusBadge.tsx+++ b/frontend/src/components/secrets/SecretReadStatusBadge.tsx@@ -1,6 +1,6 @@-import { formatDistanceToNow } from "date-fns";+import { differenceInCalendarDays } from "date-fns";-import { Badge, Tooltip, TooltipContent, TooltipTrigger } from "@app/components/v3";+import { Badge } from "@app/components/v3";import { SecretReadStatus } from "@app/hooks/api/secrets/types";type Props = {@@ -8,22 +8,17 @@ type Props = {lastReadAt?: string | null;};+// Only a reader's first read of the day is recorded, so the time of day would understate later reads.+const formatReadDay = (lastReadAt: string) => {+ const days = differenceInCalendarDays(new Date(), new Date(lastReadAt));+ if (days <= 0) return "Today";+ if (days === 1) return "Yesterday";+ return `${days} days ago`;+};+export const SecretReadStatusBadge = ({ status, lastReadAt }: Props) => {if (status === SecretReadStatus.Read && lastReadAt) {- return <span>{formatDistanceToNow(new Date(lastReadAt))} ago</span>;- }- if (status === SecretReadStatus.PossiblyRead && lastReadAt) {- return (- <Tooltip>- <TooltipTrigger asChild>- <Badge variant="warning">Possibly read</Badge>- </TooltipTrigger>- <TooltipContent>- A parent folder was read {formatDistanceToNow(new Date(lastReadAt))} ago. That returns- this secret only if the read was recursive.- </TooltipContent>- </Tooltip>- );+ return <span>{formatReadDay(lastReadAt)}</span>;}return <Badge variant="danger">No reads</Badge>;};diff --git a/frontend/src/hooks/api/secrets/types.ts b/frontend/src/hooks/api/secrets/types.tsindex 3bf07e690b..c9561cc3c4 100644--- a/frontend/src/hooks/api/secrets/types.ts+++ b/frontend/src/hooks/api/secrets/types.ts@@ -164,7 +164,6 @@ export type TGetSecretReadActivityDTO = Omit<TGetSecretAccessListDTO, "includeAlexport enum SecretReadStatus {Read = "read",- PossiblyRead = "possibly-read",NoReads = "no-reads"}