AI Agents & MCPs & AI Workflow Automation • (~400 MCP servers for AI agents) • AI Automation / AI Agent with MCPs • AI Workflows & AI Agents • MCPs for AI Agents
TypeScript ai-agentai-agent-toolsai-agentsai-agents-frameworkmcpmcp-servermcp-toolsmcpsn8n-alternativeno-code-automationworkflowworkflow-automationworkflows
feat(audit-logs): keep audit events for a set period, with an hourly cleanup (#16004)
Othman Abu Ajamieh46d541dd6cbd
diff --git a/brain/knowledge/data-storage-observability/audit-logs.md b/brain/knowledge/data-storage-observability/audit-logs.mdindex e260e852619..9edbe3993e9 100644--- a/brain/knowledge/data-storage-observability/audit-logs.md+++ b/brain/knowledge/data-storage-observability/audit-logs.md@@ -7,13 +7,20 @@ icon: 📜Records security-relevant actions for compliance and forensics, persisted to the `audit_event` table and queryable by platform admins. Enterprise/Cloud only, gated by `platform.plan.auditLogEnabled`.### Entities & services-- **ApplicationEvent**: discriminated union of all auditable types; **ApplicationEventName** is a 27-value enum (`flow.created`, `flow.published`, `user.signed.in`, `variable.value.revealed`, etc.).+- **ApplicationEvent**: discriminated union of all auditable types; **ApplicationEventName** is a 40-value enum (`flow.created`, `flow.published`, `user.signed.in`, `variable.value.revealed`, etc.).- `audit_event` entity: `action`, `userEmail`, `userId`, `projectId` (nullable), `data` (jsonb), `ip`. Composite indices on `(platformId, projectId, userId, action)` and narrower.- `audit-event-service.ts`: `setup()` and `list()`.### How it works- `setup()` registers two fire-and-forget listeners on the `applicationEvents` bus — `userEvent` (user actions) and `workerEvent` (background actions) — so events are captured transparently without callers coupling to the audit code.-- `GET /v1/audit-events` (platformAdminOnly) returns `SeekPage<ApplicationEvent>` sorted by `created` desc. Filters: `action[]`, `projectId[]`, `userId`, `createdBefore/After`, cursor/limit.+- `GET /v1/audit-events` (platformAdminOnly) returns `SeekPage<ApplicationEvent>` sorted by `created` desc, or asc with `order=ASC` (the retention dialog reads the oldest event with `order=ASC&limit=1`). Filters: `action[]`, `projectId[]`, `userId`, `createdBefore/After`, cursor/limit.++### Retention+- **Retention ceiling**: `AP_AUDIT_LOG_RETENTION_DAYS`, the instance maximum; empty keeps events forever. Read it only through `auditLogRetentionCeiling` (digits only, 30–3650, anything else is null = keep forever). Its floor matches the platform minimum, so every ceiling leaves an admin at least one period to pick.+- **Platform retention**: nullable `platform.auditLogRetentionDays`, set by a platform admin from the Audit Logs page (at least 30, at most the ceiling or 3650). The effective value is `LEAST(platform, ceiling)`; NULL with NULL skips the platform.+- `SystemJobName.AUDIT_LOG_RETENTION` runs hourly at `:15`, EE/Cloud only. One probe query lists platforms with expired rows in random order. The run then goes round those platforms, at most 100k rows per platform per round, until none has expired rows or the run reaches 1M rows or 10 minutes, so a single-platform install gets the whole budget. Each batch deletes 5000 rows oldest first with `FOR UPDATE SKIP LOCKED`, then pauses for as long as it took.+- **Pause flag**: `AP_AUDIT_LOG_RETENTION_PAUSED=true` skips the job and keeps every retention value. The ceiling cannot pause anything, because `LEAST` only shortens. The flag `AUDIT_LOG_RETENTION_PAUSED` carries it to the Retention dialog, which says the cleanup is paused.+- **Run summary**: `stoppedBy` is `done` only when no platform has expired rows left, and `failed` when every platform still left failed this run. A platform still more than `AUDIT_LOG_RETENTION_BACKLOG_GRACE_DAYS` behind its period goes into one warn line per run (worst 10).### Gotchas- Event capture is decoupled via the event bus — new auditable actions just emit onto `applicationEvents`.@@ -23,15 +30,31 @@ Records security-relevant actions for compliance and forensics, persisted to the- Anything you read about this paginator emitting `DATE_TRUNC('second', created)` cursors is stale — it now selects `created::text` and emits a plain composite cursor `(created < c) OR (created = c AND id < i)`, so the old "events in the same second get skipped across pages" bug is gone.- `summarizeApplicationEvent()` builds detailed summaries (e.g. for `flow.updated`). `buildMockEvent()` yields a typed mock per event name, reused by event-destination test delivery.+- **Never read the retention ceiling with `system.getNumber`.** It uses `parseInt`, so `0` reads as 0 and `1e3` as 1, and the startup validator only warns. A ceiling of 0 would delete every platform's events each hour.+- Retention deletes walk `(platformId, created DESC, id DESC)` backwards. Do not add a `created`-leading index for them; on the Cloud table that build is a manual, hours-long operation (see above).+- Each retention batch carries a keyset cursor `(created, id)` from the last row it deleted. Without it every batch re-walks the dead index entries that earlier batches left at the old end of the index until vacuum runs, which grows with the square of the rows deleted per run. The cursor is a `to_char(..., 'US')` ISO string so it keeps microseconds, and the `id` part matters: events take `created` from JS `new Date()`, so many share a millisecond.+- Each retention batch re-reads the platform's period with `SELECT ... FOR SHARE` on the platform row. The probe's value can be minutes old by the last batch, and an admin who lengthens or clears the period must not lose rows after the save returned. The share lock makes that save wait for one batch at most.+- The job sets `autovacuum_vacuum_scale_factor = 0.02` on `audit_event` the first time it has rows to delete, and only when the table has no value, so an operator's setting wins. It is not a migration: `ALTER TABLE ... SET` waits behind a running anti-wraparound vacuum, and migrations run before the healthcheck. Its `lock_timeout` is 100 ms, under the default 1 s `deadlock_timeout`, so it gives up before Postgres would cancel a running autovacuum to let it in.+- Tests that need a race during a sweep (a period changed mid-run, a batch that fails) install a `BEFORE DELETE` trigger on `audit_event`; PGlite runs PL/pgSQL.+- Count deleted rows inside the statement, as `WITH deleted AS (DELETE ... RETURNING ...) SELECT count(*) OVER () ...`, not with `affected`. TypeORM fills `affected` from `rowCount`, which a PGlite result does not have. A top-level `SELECT` returns rows the same way on both drivers, where a bare `DELETE` through `em.query` does not.+- Flow-run STARTED/FINISHED events, test runs included, carry the whole `FlowRun` and are most of the rows. Retention bounds the table; it does not lower the write rate.+- Every event is saved for every platform, whatever `plan.auditLogEnabled` says. The flag gates the read endpoint and the retention setting, not capture, so only the instance ceiling cleans a platform without the feature.+- The retention job reads `platform_plan.auditLogEnabled` itself, with a `LEFT JOIN`, in the probe and in every batch. A saved period counts only while the plan has audit logs, because the API refuses every retention change (clearing included) on a plan without them, so a downgraded admin could not stop it. A platform with no plan row falls back to the ceiling. On Cloud the plan row only refreshes from Autumn when something reads the plan, so a churned platform nobody opens can keep `auditLogEnabled = true` and its saved period keeps running.+- A `DELETE` does not shrink the table file; Postgres reuses the space. A one-time shrink needs `pg_repack` or `VACUUM FULL`.+- `AP_AUDIT_LOG_RETENTION_PAUSED` is read by the app server that picks up the hourly job, and every app server runs the system-job worker. A pause set on only some servers does not stop the cleanup.+- The piece-upgrade revert (`POST /v1/admin/flows/revert-upgrade`) reads `flow.pieces.upgraded` rows, so it can only revert upgrades still inside the platform's retention period.+### Key files-Entry point: `auditLogService`, wired up in `auditEventModule` which calls `.setup()` and mounts the controller at `/v1/audit-events`.+Entry point: `auditLogService`, wired up in `auditEventModule` which calls `.setup()` and mounts the controller at `/v1/audit-events`. Retention: `auditLogRetention` (the job) and `auditLogRetentionCeiling` (the only reader of the env ceiling).-- `packages/server/api/src/app/ee/audit-logs/` — module, service, and TypeORM entity+- `packages/server/api/src/app/ee/audit-logs/` — module, service, TypeORM entity, and the retention job+- `packages/server/api/src/app/helper/retention/` — the retention ceiling reader (CE, shared with the platform service and flags)- `packages/core/shared/src/lib/ee/audit-events/` — event types, the `ApplicationEvent` union, `summarizeApplicationEvent()`, and `buildMockEvent()`- `packages/web/src/features/platform-admin/api/audit-events-api.ts` — frontend API client- `packages/web/src/features/platform-admin/hooks/audit-log-hooks.ts` — React Query hooks- `packages/web/src/app/routes/platform/security/audit-logs/` — platform admin UI page+- `packages/web/src/features/platform-admin/` — `AuditLogRetentionButton` and `auditLogRetentionUtils`- `packages/server/api/test/integration/cloud/audit-event/` — integration tests- `docs/admin-guide/security/audit-logs/` — one user-facing doc page per event type-Paths verified 2026-07-17.+Paths verified 2026-09-30.diff --git a/bun.lock b/bun.lockindex 42ecdbfbf8b..36d20d172f9 100644--- a/bun.lock+++ b/bun.lock@@ -1,5 +1,6 @@{"lockfileVersion": 1,+ "configVersion": 0,"workspaces": {"": {"name": "activepieces",@@ -166,7 +167,7 @@},"packages/core/shared": {"name": "@activepieces/shared",- "version": "0.195.0",+ "version": "0.196.0","dependencies": {"@activepieces/core-execution": "workspace:*","@activepieces/core-formula": "workspace:*",diff --git a/docs/admin-guide/security/audit-logs/overview.mdx b/docs/admin-guide/security/audit-logs/overview.mdxindex 3b3ff8f6b68..a48227cc584 100644--- a/docs/admin-guide/security/audit-logs/overview.mdx+++ b/docs/admin-guide/security/audit-logs/overview.mdx@@ -23,6 +23,27 @@ Open **Platform Admin → Security → Audit Logs** to browse the full table. FiTo react to events in real time — get a Slack alert when a flow fails, forward audit logs to your SIEM, send a daily digest of new sign-ups — use [Event Streaming](/admin-guide/guides/event-streaming). It POSTs each event to a webhook URL of your choice, which you can point at an external system or at an internal handler flow that routes each event to the channel you care about.+## Keep events for a set period++Audit events are kept forever by default. To keep the table small, open **Platform Admin → Security → Audit Logs**, click the **Retention** button at the top right, and choose how long to keep events: 30 days, 90 days, 6 months, 1 year, or forever. The dialog also shows the date of your oldest event, so you can see when the cleanup has caught up.++- Events older than that period are deleted every hour, oldest first, in small batches with a pause after each one, so your database keeps serving traffic while the cleanup runs.+- Each hourly run deletes up to 1 million events. A large backlog takes several runs: 100 million expired events take about four days. While a platform is behind, the server logs a warning that names it.+- Deletion is permanent. If you need a long-term copy, forward events to your own storage with [Event Streaming](/admin-guide/guides/event-streaming) before they expire.+- The shortest period you can choose is 30 days.++On a self-hosted instance, set [`AP_AUDIT_LOG_RETENTION_DAYS`](/install/reference/environment-variables#limits-&-retention) to cap the period for every platform, from 30 to 3650 days. The cap is a hard limit: a platform can only choose a shorter period. A platform that has not chosen a period uses the cap, so the first hourly cleanup after you set it deletes that platform's older events.++To stop the cleanup for a while, for example during a database migration, set [`AP_AUDIT_LOG_RETENTION_PAUSED`](/install/reference/environment-variables#limits-&-retention) to `true` on every app server and restart them. Any app server can run the cleanup, so a server without the setting keeps deleting. While the cleanup is paused, the Retention dialog says so.++<Note>+Deleting events does not shrink the database files on disk. Postgres reuses the freed space for new events, so the `audit_event` table stops growing. To return the space to the operating system once, run `pg_repack` on the `audit_event` table (online, but it needs free disk about the size of the table and its indexes), or `VACUUM FULL audit_event` during a maintenance window (it locks the table while it runs). Backups and point-in-time recovery archives taken earlier keep the deleted events until they expire.+</Note>++<Note>+The first time the cleanup has events to delete, it sets `autovacuum_vacuum_scale_factor = 0.02` on the `audit_event` table, so Postgres reclaims deleted rows sooner. If you already set a value on that table, the cleanup leaves it alone.+</Note>+## Event catalogThe full list of documented events. We add new ones as the platform grows; the source of truth for every event we emit is the [event schema in the codebase](https://github.com/activepieces/activepieces/blob/main/packages/core/shared/src/lib/ee/audit-events/index.ts).diff --git a/docs/install/reference/environment-variables.mdx b/docs/install/reference/environment-variables.mdxindex 5351d4535d3..664c46cac87 100644--- a/docs/install/reference/environment-variables.mdx+++ b/docs/install/reference/environment-variables.mdx@@ -239,8 +239,10 @@ run data is kept. The **Cloud** values and how these interact are covered in| Variable | Description | Default ||---|---|---|-| `AP_EXECUTION_DATA_RETENTION_DAYS` | Days to retain execution data, logs, and events. Should be **strictly greater** than `AP_PAUSED_FLOW_TIMEOUT_DAYS` — when the two are equal, a valid resume firing on the retention boundary can race the file-cleanup job and find its state files gone. A 1-2 day margin closes the race. | `30` |+| `AP_EXECUTION_DATA_RETENTION_DAYS` | Days to retain execution data, logs, and events. Audit log events have their own setting, `AP_AUDIT_LOG_RETENTION_DAYS`. Should be **strictly greater** than `AP_PAUSED_FLOW_TIMEOUT_DAYS` — when the two are equal, a valid resume firing on the retention boundary can race the file-cleanup job and find its state files gone. A 1-2 day margin closes the race. | `30` || `AP_PAUSED_FLOW_TIMEOUT_DAYS` | Maximum pause duration for a paused flow, in days. Cannot exceed `AP_EXECUTION_DATA_RETENTION_DAYS`, and should be at least 1 day smaller than it (see above). Measured cumulatively from `flow_run.created`, not per-pause — a flow with stacked delays whose sum exceeds this fails at waitpoint creation. Webhook waitpoints without an explicit expiry default to this deadline. | `30` |+| `AP_AUDIT_LOG_RETENTION_DAYS` | Maximum number of days to keep [audit log events](/admin-guide/security/audit-logs/overview#keep-events-for-a-set-period). Leave it empty to keep them forever. This is a hard limit: platform admins can pick a shorter period (at least 30 days) from the Audit Logs page, never a longer one. Platforms that have not picked a period use this value, so the first hourly cleanup after you set it deletes their older events. Must be a whole number from 30 to 3650 — any other value is ignored with a warning at boot, and events are kept. | _(empty)_ |+| `AP_AUDIT_LOG_RETENTION_PAUSED` | Set to `true` to stop the hourly [audit log cleanup](/admin-guide/security/audit-logs/overview#keep-events-for-a-set-period) without changing any retention setting, for example during a database migration. Expired events stay until you set it back to `false`. Set it on every app server and restart them, because any app server can run the cleanup. | `false` || `AP_ISSUE_ARCHIVE_DAYS` | Issues not updated for this many days are automatically archived. | `7` || `AP_MAX_FILE_SIZE_MB` | Maximum size (MB) for files uploaded in steps or triggers. Larger files are rejected. | `25` || `AP_AI_CREDIT_USD_VALUE` | Dollar value of one AI credit. A call on the Activepieces AI provider is billed the dollar cost the provider reports for it, divided by this. A call on your own API key is billed one flat credit and ignores this. | `0.0005` |diff --git a/packages/core/shared/package.json b/packages/core/shared/package.jsonindex 5348e9290d3..d45ef8177e7 100644--- a/packages/core/shared/package.json+++ b/packages/core/shared/package.json@@ -1,6 +1,6 @@{"name": "@activepieces/shared",- "version": "0.195.0",+ "version": "0.196.0","type": "commonjs","sideEffects": false,"main": "./dist/src/index.js",diff --git a/packages/core/shared/src/lib/core/flag/flag.ts b/packages/core/shared/src/lib/core/flag/flag.tsindex fcab4e05f38..5f6a0e45971 100755--- a/packages/core/shared/src/lib/core/flag/flag.ts+++ b/packages/core/shared/src/lib/core/flag/flag.ts@@ -28,6 +28,8 @@ export enum ApFlagId {EMAIL_CODE_AUTH_ENABLED = 'EMAIL_CODE_AUTH_ENABLED',FRONTEND_SENTRY_DSN = 'FRONTEND_SENTRY_DSN',EXECUTION_DATA_RETENTION_DAYS = 'EXECUTION_DATA_RETENTION_DAYS',+ AUDIT_LOG_RETENTION_DAYS = 'AUDIT_LOG_RETENTION_DAYS',+ AUDIT_LOG_RETENTION_PAUSED = 'AUDIT_LOG_RETENTION_PAUSED',ENVIRONMENT = 'ENVIRONMENT',PUBLIC_URL = 'PUBLIC_URL',MCP_URL = 'MCP_URL',diff --git a/packages/core/shared/src/lib/ee/audit-events/index.ts b/packages/core/shared/src/lib/ee/audit-events/index.tsindex 7a0cc77cbec..809a733ef4b 100644--- a/packages/core/shared/src/lib/ee/audit-events/index.ts+++ b/packages/core/shared/src/lib/ee/audit-events/index.ts@@ -12,6 +12,7 @@ export const ListAuditEventsRequest = z.object({userId: z.string().optional(),createdBefore: z.string().optional(),createdAfter: z.string().optional(),+ order: z.enum(['ASC', 'DESC']).optional(),})export type ListAuditEventsRequest = z.infer<typeof ListAuditEventsRequest>@@ -831,6 +832,10 @@ function convertUpdateActionToDetails(event: FlowUpdatedEvent) {}}+export const AUDIT_LOG_RETENTION_MIN_DAYS = 30+export const AUDIT_LOG_RETENTION_MAX_DAYS = 3650+export const AUDIT_LOG_RETENTION_BACKLOG_GRACE_DAYS = 2+export type AgentActionRef = z.infer<typeof AgentActionEventData>['action']export * from './mock-event-builder'diff --git a/packages/core/shared/src/lib/management/platform/platform.model.ts b/packages/core/shared/src/lib/management/platform/platform.model.tsindex c5198a7c1f2..ba14a0ff286 100644--- a/packages/core/shared/src/lib/management/platform/platform.model.ts+++ b/packages/core/shared/src/lib/management/platform/platform.model.ts@@ -212,6 +212,7 @@ export const Platform = z.object({autoCreatePersonalProjects: z.boolean(),pinnedPieces: z.array(z.string()),pieceSelectorConfig: Nullable(PieceSelectorConfig),+ auditLogRetentionDays: Nullable(z.number().int()),})export type Platform = z.infer<typeof Platform>export type PlatformWithoutFederatedAuth = Omit<Platform, 'federatedAuthProviders'>@@ -242,6 +243,7 @@ export const PlatformWithoutSensitiveData = z.object({autoCreatePersonalProjects: z.boolean(),pinnedPieces: z.array(z.string()),pieceSelectorConfig: Nullable(PieceSelectorConfig),+ auditLogRetentionDays: Nullable(z.number().int()),})export type PlatformWithoutSensitiveData = z.infer<typeof PlatformWithoutSensitiveData>diff --git a/packages/core/shared/src/lib/management/platform/platform.request.ts b/packages/core/shared/src/lib/management/platform/platform.request.tsindex 181050e9fdc..6986273d577 100644--- a/packages/core/shared/src/lib/management/platform/platform.request.ts+++ b/packages/core/shared/src/lib/management/platform/platform.request.ts@@ -66,6 +66,7 @@ export const UpdatePlatformRequestBody = z.object({pieceSelectorConfig: NullablePieceSelectorConfigFromMultipart.optional(),allowedEmbedOrigins: z.array(allowedEmbedOriginSchema).optional(),+ auditLogRetentionDays: Nullable(z.number().int().positive()),})export type UpdatePlatformRequestBody = z.infer<typeof UpdatePlatformRequestBody>diff --git a/packages/server/api/src/app/database/migration/postgres/1865000000000-AddAuditLogRetentionDaysToPlatform.ts b/packages/server/api/src/app/database/migration/postgres/1865000000000-AddAuditLogRetentionDaysToPlatform.tsnew file mode 100644index 00000000000..9acb90210e3--- /dev/null+++ b/packages/server/api/src/app/database/migration/postgres/1865000000000-AddAuditLogRetentionDaysToPlatform.ts@@ -0,0 +1,21 @@+import { QueryRunner } from 'typeorm'+import { Migration } from '../../migration'++export class AddAuditLogRetentionDaysToPlatform1865000000000 implements Migration {+ name = 'AddAuditLogRetentionDaysToPlatform1865000000000'+ breaking = false+ release = '0.93.0'++ public async up(queryRunner: QueryRunner): Promise<void> {+ await queryRunner.query(`+ ALTER TABLE "platform"+ ADD COLUMN IF NOT EXISTS "auditLogRetentionDays" integer+ `)+ }++ public async down(queryRunner: QueryRunner): Promise<void> {+ await queryRunner.query(`+ ALTER TABLE "platform" DROP COLUMN IF EXISTS "auditLogRetentionDays"+ `)+ }+}diff --git a/packages/server/api/src/app/database/postgres-connection.ts b/packages/server/api/src/app/database/postgres-connection.tsindex 5218f65bd28..699402cdb28 100644--- a/packages/server/api/src/app/database/postgres-connection.ts+++ b/packages/server/api/src/app/database/postgres-connection.ts@@ -452,6 +452,7 @@ import { AddPieceSetRequiredActions1860000000000 } from './migration/postgres/18import { AddEventDestinationEnabledAndFormat1862000000000 } from './migration/postgres/1862000000000-AddEventDestinationEnabledAndFormat'import { AddEventDestinationHeaders1863000000000 } from './migration/postgres/1863000000000-AddEventDestinationHeaders'import { AddPlatformTierFastAndConversationTier1864000000000 } from './migration/postgres/1864000000000-AddPlatformTierFastAndConversationTier'+import { AddAuditLogRetentionDaysToPlatform1865000000000 } from './migration/postgres/1865000000000-AddAuditLogRetentionDaysToPlatform'const getSslConfig = (): boolean | TlsOptions => {const useSsl = system.get(AppSystemProp.POSTGRES_USE_SSL)@@ -919,6 +920,7 @@ export const getMigrations = (): (new () => Migration)[] => {AddEventDestinationEnabledAndFormat1862000000000,AddEventDestinationHeaders1863000000000,AddPlatformTierFastAndConversationTier1864000000000,+ AddAuditLogRetentionDaysToPlatform1865000000000,]return migrations}diff --git a/packages/server/api/src/app/ee/audit-logs/audit-event-module.ts b/packages/server/api/src/app/ee/audit-logs/audit-event-module.tsindex 1d8931dd339..cde4f1cf3da 100644--- a/packages/server/api/src/app/ee/audit-logs/audit-event-module.ts+++ b/packages/server/api/src/app/ee/audit-logs/audit-event-module.ts@@ -1,11 +1,29 @@import { ListAuditEventsRequest, PrincipalType } from '@activepieces/shared'import { FastifyPluginAsyncZod } from 'fastify-type-provider-zod'import { securityAccess } from '../../core/security/authorization/fastify-security'+import { SystemJobName } from '../../helper/system-jobs/common'+import { systemJobHandlers } from '../../helper/system-jobs/job-handlers'+import { systemJobsSchedule } from '../../helper/system-jobs/system-job'import { platformMustHaveFeatureEnabled } from '../authentication/ee-authorization'import { auditLogService } from './audit-event-service'+import { auditLogRetention } from './audit-log-retention'export const auditEventModule: FastifyPluginAsyncZod = async (app) => {auditLogService(app.log).setup()+ systemJobHandlers.registerJobHandler(SystemJobName.AUDIT_LOG_RETENTION, async () => {+ await auditLogRetention(app.log).sweep()+ })+ await systemJobsSchedule(app.log).upsertJob({+ job: {+ name: SystemJobName.AUDIT_LOG_RETENTION,+ data: {},+ jobId: SystemJobName.AUDIT_LOG_RETENTION,+ },+ schedule: {+ type: 'repeated',+ cron: '15 * * * *',+ },+ })app.addHook('preHandler', platformMustHaveFeatureEnabled((platform) => platform.plan.auditLogEnabled))await app.register(auditEventController, { prefix: '/v1/audit-events' })}@@ -22,6 +40,7 @@ const auditEventController: FastifyPluginAsyncZod = async (app) => {userId: request.query.userId ?? undefined,createdBefore: request.query.createdBefore ?? undefined,createdAfter: request.query.createdAfter ?? undefined,+ order: request.query.order,})})}diff --git a/packages/server/api/src/app/ee/audit-logs/audit-event-service.ts b/packages/server/api/src/app/ee/audit-logs/audit-event-service.tsindex e8e65a5edbe..a07cab4e429 100644--- a/packages/server/api/src/app/ee/audit-logs/audit-event-service.ts+++ b/packages/server/api/src/app/ee/audit-logs/audit-event-service.ts@@ -22,13 +22,13 @@ export const auditLogService = (log: FastifyBaseLogger) => ({},})},- async list({ platformId, cursorRequest, limit, userId, action, projectId, createdBefore, createdAfter }: ListParams): Promise<SeekPage<ApplicationEvent>> {+ async list({ platformId, cursorRequest, limit, userId, action, projectId, createdBefore, createdAfter, order }: ListParams): Promise<SeekPage<ApplicationEvent>> {const decodedCursor = paginationHelper.decodeCursor(cursorRequest)const paginator = buildPaginator({entity: AuditEventEntity,query: {limit,- order: 'DESC',+ order: order ?? 'DESC',afterCursor: decodedCursor.nextCursor,beforeCursor: decodedCursor.previousCursor,},@@ -75,4 +75,5 @@ type ListParams = {projectId?: string[]createdBefore?: stringcreatedAfter?: string+ order?: 'ASC' | 'DESC'}diff --git a/packages/server/api/src/app/ee/audit-logs/audit-log-retention.ts b/packages/server/api/src/app/ee/audit-logs/audit-log-retention.tsnew file mode 100644index 00000000000..601f0634c94--- /dev/null+++ b/packages/server/api/src/app/ee/audit-logs/audit-log-retention.ts@@ -0,0 +1,360 @@+import { isNil, tryCatch } from '@activepieces/core-utils'+import { AUDIT_LOG_RETENTION_BACKLOG_GRACE_DAYS } from '@activepieces/shared'+import dayjs from 'dayjs'+import { FastifyBaseLogger } from 'fastify'+import { EntityManager } from 'typeorm'+import { z } from 'zod'+import { auditLogRetentionCeiling } from '../../helper/retention/audit-log-retention-ceiling'+import { sleep } from '../../helper/sleep'+import { system } from '../../helper/system/system'+import { AppSystemProp } from '../../helper/system/system-props'+import { auditLogRepo } from './audit-event-service'++export const auditLogRetention = (log: FastifyBaseLogger) => ({+ async sweep({+ batchSize = BATCH_SIZE,+ maxRowsPerPlatformPerRound = MAX_ROWS_PER_PLATFORM_PER_ROUND,+ maxRowsPerRun = MAX_ROWS_PER_RUN,+ runBudgetMs = RUN_BUDGET_MS,+ }: SweepParams = {}): Promise<SweepSummary> {+ const startedAt = Date.now()+ if (system.getBoolean(AppSystemProp.AUDIT_LOG_RETENTION_PAUSED) === true) {+ log.info('[auditLogRetention#sweep] Paused by AP_AUDIT_LOG_RETENTION_PAUSED')+ return summarize({ progress: [], deletedCount: 0, maxRowsPerRun, startedAt, paused: true })+ }+ const deadline = startedAt + runBudgetMs+ const ceiling = auditLogRetentionCeiling.get()+ const platformIds = await findPlatformsWithExpiredEvents({ ceiling })+ if (platformIds.length === 0) {+ return summarize({ progress: [], deletedCount: 0, maxRowsPerRun, startedAt, paused: false })+ }+ await ensureAutovacuumScaleFactor({ log })++ const progress = new Map(platformIds.map((platformId) => [platformId, startProgress(platformId)]))+ let deletedCount = 0+ let pendingIds = platformIds+ while (pendingIds.length > 0 && deletedCount < maxRowsPerRun && Date.now() < deadline) {+ for (const platformId of pendingIds) {+ if (deletedCount >= maxRowsPerRun || Date.now() >= deadline) {+ break+ }+ const previous = progress.get(platformId) ?? startProgress(platformId)+ const pass = await deleteExpiredEventsOfPlatform({+ platformId,+ ceiling,+ cursor: previous.cursor,+ batchSize,+ maxRows: Math.min(maxRowsPerPlatformPerRound, maxRowsPerRun - deletedCount),+ deadline,+ })+ deletedCount += pass.deleted+ progress.set(platformId, advanceProgress({ previous, pass }))+ if (!isNil(pass.error)) {+ log.warn({ platform: { id: platformId }, deletedCount: pass.deleted, error: pass.error }, '[auditLogRetention#sweep] Failed to delete expired audit events')+ }+ }+ pendingIds = pendingIds.filter((platformId) => progress.get(platformId)?.status === 'pending')+ }++ const finalProgress = [...progress.values()]+ const summary = summarize({ progress: finalProgress, deletedCount, maxRowsPerRun, startedAt, paused: false })+ log.info(summary, '[auditLogRetention#sweep] Completed')+ const behind = findPlatformsBehind(finalProgress)+ if (behind.length > 0) {+ log.warn({+ platformsBehind: behind.length,+ platforms: behind.slice(0, MAX_PLATFORMS_IN_BACKLOG_WARNING),+ }, '[auditLogRetention#sweep] Deletion is behind the retention period')+ }+ return summary+ },+})++async function findPlatformsWithExpiredEvents({ ceiling }: { ceiling: number | null }): Promise<string[]> {+ const rows: unknown = await auditLogRepo().manager.transaction(async (entityManager) => {+ await entityManager.query(`SET LOCAL statement_timeout = ${PROBE_STATEMENT_TIMEOUT_MS}`)+ return entityManager.query(`+ SELECT p.id AS "platformId"+ FROM "platform" p+ LEFT JOIN "platform_plan" pp ON pp."platformId" = p.id+ WHERE ${EFFECTIVE_RETENTION_DAYS_SQL} IS NOT NULL+ AND EXISTS (+ SELECT 1 FROM "audit_event" a+ WHERE a."platformId" = p.id+ AND a.created < now() - make_interval(days => ${EFFECTIVE_RETENTION_DAYS_SQL})+ )+ ORDER BY random()+ `, [ceiling])+ })+ return PlatformIdRows.parse(rows).map((row) => row.platformId)+}++async function ensureAutovacuumScaleFactor({ log }: { log: FastifyBaseLogger }): Promise<void> {+ const { error } = await tryCatch(() => auditLogRepo().manager.transaction(async (entityManager) => {+ const rows: unknown = await entityManager.query('SELECT reloptions FROM pg_class WHERE oid = \'"audit_event"\'::regclass')+ const options = TableOptionRows.parse(rows)[0]?.reloptions ?? []+ if (options.some((option) => option.startsWith(`${AUTOVACUUM_SCALE_FACTOR_OPTION}=`))) {+ return+ }+ await entityManager.query(`SET LOCAL lock_timeout = ${AUTOVACUUM_LOCK_TIMEOUT_MS}`)+ await entityManager.query(`ALTER TABLE "audit_event" SET (${AUTOVACUUM_SCALE_FACTOR_OPTION} = ${AUTOVACUUM_SCALE_FACTOR})`)+ log.info({ autovacuumScaleFactor: AUTOVACUUM_SCALE_FACTOR }, '[auditLogRetention#sweep] Set the autovacuum scale factor of audit_event')+ }))+ if (!isNil(error)) {+ log.info({ error }, '[auditLogRetention#sweep] Could not set the autovacuum scale factor of audit_event, will retry on the next run')+ }+}++async function deleteExpiredEventsOfPlatform({ platformId, ceiling, cursor, batchSize, maxRows, deadline }: DeleteExpiredEventsOfPlatformParams): Promise<PlatformPass> {+ let deleted = 0+ let lastDeleted = cursor+ let retentionDays: number | null = null+ while (deleted < maxRows && Date.now() < deadline) {+ const limit = Math.min(batchSize, maxRows - deleted)+ const batchStartedAt = Date.now()+ const { data: batch, error } = await tryCatch(() => deleteOneBatch({ platformId, ceiling, cursor: lastDeleted, limit }))+ if (error) {+ return { deleted, cursor: lastDeleted, retentionDays, outcome: 'failed', error }+ }+ deleted += batch.deletedCount+ lastDeleted = batch.lastDeleted ?? lastDeleted+ retentionDays = batch.retentionDays+ if (batch.deletedCount < limit) {+ return { deleted, cursor: lastDeleted, retentionDays, outcome: 'finished' }+ }+ await sleep(Date.now() - batchStartedAt)+ }+ return { deleted, cursor: lastDeleted, retentionDays, outcome: deleted >= maxRows ? 'capped' : 'stopped' }+}++async function deleteOneBatch({ platformId, ceiling, cursor, limit }: DeleteOneBatchParams): Promise<BatchResult> {+ return auditLogRepo().manager.transaction(async (entityManager) => {+ await entityManager.query(`SET LOCAL statement_timeout = ${BATCH_STATEMENT_TIMEOUT_MS}`)+ const retentionDays = await lockRetentionDays({ entityManager, platformId, ceiling })+ if (isNil(retentionDays)) {+ return { deletedCount: 0, lastDeleted: null, retentionDays }+ }+ const rows: unknown = await entityManager.query(`+ WITH deleted AS (+ DELETE FROM "audit_event"+ WHERE id IN (+ SELECT id FROM "audit_event"+ WHERE "platformId" = $1+ AND created < now() - make_interval(days => $2)+ ${isNil(cursor) ? '' : 'AND created >= $4 AND (created, id) > ($4, $5)'}+ ORDER BY created, id+ LIMIT $3+ FOR UPDATE SKIP LOCKED+ )+ RETURNING id, created+ )+ SELECT+ id,+ to_char(created AT TIME ZONE 'UTC', 'YYYY-MM-DD"T"HH24:MI:SS.US"Z"') AS created,+ count(*) OVER ()::int AS "deletedCount"+ FROM deleted+ ORDER BY deleted.created DESC, id DESC+ LIMIT 1+ `, isNil(cursor) ? [platformId, retentionDays, limit] : [platformId, retentionDays, limit, cursor.created, cursor.id])+ const last = LastDeletedRows.parse(rows)[0]+ return {+ deletedCount: last?.deletedCount ?? 0,+ lastDeleted: isNil(last) ? null : { id: last.id, created: last.created },+ retentionDays,+ }+ })+}++async function lockRetentionDays({ entityManager, platformId, ceiling }: { entityManager: EntityManager, platformId: string, ceiling: number | null }): Promise<number | null> {+ const rows: unknown = await entityManager.query(`+ SELECT ${EFFECTIVE_RETENTION_DAYS_SQL} AS "retentionDays"+ FROM "platform" p+ LEFT JOIN "platform_plan" pp ON pp."platformId" = p.id+ WHERE p.id = $2+ FOR SHARE OF p+ `, [ceiling, platformId])+ return RetentionDaysRows.parse(rows)[0]?.retentionDays ?? null+}++function startProgress(platformId: string): PlatformProgress {+ return { platformId, deleted: 0, cursor: null, retentionDays: null, status: 'pending' }+}++function advanceProgress({ previous, pass }: { previous: PlatformProgress, pass: PlatformPass }): PlatformProgress {+ return {+ platformId: previous.platformId,+ deleted: previous.deleted + pass.deleted,+ cursor: pass.cursor,+ retentionDays: pass.retentionDays ?? previous.retentionDays,+ status: toStatus(pass.outcome),+ }+}++function toStatus(outcome: PassOutcome): PlatformStatus {+ switch (outcome) {+ case 'finished':+ return 'done'+ case 'failed':+ return 'failed'+ case 'capped':+ case 'stopped':+ return 'pending'+ }+}++function findPlatformsBehind(progress: PlatformProgress[]): PlatformBacklog[] {+ return progress+ .flatMap((entry) => {+ if (entry.status !== 'pending' || isNil(entry.cursor) || isNil(entry.retentionDays)) {+ return []+ }+ const daysBehind = dayjs().diff(dayjs(entry.cursor.created), 'day') - entry.retentionDays+ return daysBehind > AUDIT_LOG_RETENTION_BACKLOG_GRACE_DAYS+ ? [{ platform: { id: entry.platformId }, retentionDays: entry.retentionDays, deletedUpTo: entry.cursor.created, daysBehind }]+ : []+ })+ .sort((a, b) => b.daysBehind - a.daysBehind)+}++function summarize({ progress, deletedCount, maxRowsPerRun, startedAt, paused }: SummarizeParams): SweepSummary {+ const platformsLeft = progress.filter((entry) => entry.status === 'pending').length+ const platformsFailed = progress.filter((entry) => entry.status === 'failed').length+ return {+ deletedCount,+ platformsDone: progress.filter((entry) => entry.status === 'done').length,+ platformsFailed,+ platformsLeft,+ platformsBehind: findPlatformsBehind(progress).length,+ durationMs: Date.now() - startedAt,+ stoppedBy: resolveStopReason({ paused, platformsLeft, platformsFailed, deletedCount, maxRowsPerRun }),+ }+}++function resolveStopReason({ paused, platformsLeft, platformsFailed, deletedCount, maxRowsPerRun }: ResolveStopReasonParams): StopReason {+ if (paused) {+ return 'paused'+ }+ if (platformsLeft === 0) {+ return platformsFailed === 0 ? 'done' : 'failed'+ }+ return deletedCount >= maxRowsPerRun ? 'rowLimit' : 'timeBudget'+}++const BATCH_SIZE = 5_000+const MAX_ROWS_PER_PLATFORM_PER_ROUND = 100_000+const MAX_ROWS_PER_RUN = 1_000_000+const RUN_BUDGET_MS = 10 * 60 * 1000+const BATCH_STATEMENT_TIMEOUT_MS = 60 * 1000+const PROBE_STATEMENT_TIMEOUT_MS = 5 * 60 * 1000+const AUTOVACUUM_SCALE_FACTOR_OPTION = 'autovacuum_vacuum_scale_factor'+const AUTOVACUUM_SCALE_FACTOR = 0.02+const AUTOVACUUM_LOCK_TIMEOUT_MS = 100+const MAX_PLATFORMS_IN_BACKLOG_WARNING = 10+const EFFECTIVE_RETENTION_DAYS_SQL = 'LEAST(CASE WHEN pp."auditLogEnabled" THEN p."auditLogRetentionDays" END, $1::int)'++const PlatformIdRows = z.array(z.object({+ platformId: z.string(),+}))++const RetentionDaysRows = z.array(z.object({+ retentionDays: z.number().int().positive().nullable(),+}))++const TableOptionRows = z.array(z.object({+ reloptions: z.array(z.string()).nullable(),+}))++const LastDeletedRows = z.array(z.object({+ id: z.string(),+ created: z.string(),+ deletedCount: z.number().int(),+}))++type DeleteCursor = {+ id: string+ created: string+}++type PassOutcome = 'finished' | 'capped' | 'stopped' | 'failed'++type PlatformStatus = 'pending' | 'done' | 'failed'++type StopReason = 'done' | 'failed' | 'rowLimit' | 'timeBudget' | 'paused'++type PlatformPass = {+ deleted: number+ cursor: DeleteCursor | null+ retentionDays: number | null+ outcome: PassOutcome+ error?: Error+}++type PlatformProgress = {+ platformId: string+ deleted: number+ cursor: DeleteCursor | null+ retentionDays: number | null+ status: PlatformStatus+}++type PlatformBacklog = {+ platform: { id: string }+ retentionDays: number+ deletedUpTo: string+ daysBehind: number+}++type BatchResult = {+ deletedCount: number+ lastDeleted: DeleteCursor | null+ retentionDays: number | null+}++type SweepParams = {+ batchSize?: number+ maxRowsPerPlatformPerRound?: number+ maxRowsPerRun?: number+ runBudgetMs?: number+}++type SweepSummary = {+ deletedCount: number+ platformsDone: number+ platformsFailed: number+ platformsLeft: number+ platformsBehind: number+ durationMs: number+ stoppedBy: StopReason+}++type SummarizeParams = {+ progress: PlatformProgress[]+ deletedCount: number+ maxRowsPerRun: number+ startedAt: number+ paused: boolean+}++type ResolveStopReasonParams = {+ paused: boolean+ platformsLeft: number+ platformsFailed: number+ deletedCount: number+ maxRowsPerRun: number+}++type DeleteExpiredEventsOfPlatformParams = {+ platformId: string+ ceiling: number | null+ cursor: DeleteCursor | null+ batchSize: number+ maxRows: number+ deadline: number+}++type DeleteOneBatchParams = {+ platformId: string+ ceiling: number | null+ cursor: DeleteCursor | null+ limit: number+}diff --git a/packages/server/api/src/app/flags/flag.service.ts b/packages/server/api/src/app/flags/flag.service.tsindex b8643ad8946..89d90c1640f 100644--- a/packages/server/api/src/app/flags/flag.service.ts+++ b/packages/server/api/src/app/flags/flag.service.ts@@ -9,6 +9,7 @@ import { repoFactory } from '../core/db/repo-factory'import { federatedAuthnService } from '../ee/authentication/federated-authn/federated-authn-service'import { smtpEmailSender } from '../ee/helper/email/email-sender/smtp-email-sender'import { domainHelper } from '../helper/domain-helper'+import { auditLogRetentionCeiling } from '../helper/retention/audit-log-retention-ceiling'import { system } from '../helper/system/system'import { AppSystemProp } from '../helper/system/system-props'import { knowledgeBaseSchema } from '../knowledge-base/knowledge-base-schema'@@ -122,6 +123,18 @@ export const flagService = (log: FastifyBaseLogger) => ({created,updated,},+ {+ id: ApFlagId.AUDIT_LOG_RETENTION_DAYS,+ value: auditLogRetentionCeiling.get(),+ created,+ updated,+ },+ {+ id: ApFlagId.AUDIT_LOG_RETENTION_PAUSED,+ value: system.getBoolean(AppSystemProp.AUDIT_LOG_RETENTION_PAUSED) === true,+ created,+ updated,+ },{id: ApFlagId.CLOUD_AUTH_ENABLED,value: system.getBoolean(AppSystemProp.CLOUD_AUTH_ENABLED) ?? true,diff --git a/packages/server/api/src/app/helper/retention/audit-log-retention-ceiling.ts b/packages/server/api/src/app/helper/retention/audit-log-retention-ceiling.tsnew file mode 100644index 00000000000..b7f198c58e9--- /dev/null+++ b/packages/server/api/src/app/helper/retention/audit-log-retention-ceiling.ts@@ -0,0 +1,20 @@+import { isNil } from '@activepieces/core-utils'+import { AUDIT_LOG_RETENTION_MAX_DAYS, AUDIT_LOG_RETENTION_MIN_DAYS } from '@activepieces/shared'+import { system } from '../system/system'+import { AppSystemProp } from '../system/system-props'++function get(): number | null {+ return parse(system.get(AppSystemProp.AUDIT_LOG_RETENTION_DAYS))+}++function parse(value: string | undefined): number | null {+ if (isNil(value) || !WHOLE_DAYS_PATTERN.test(value)) {+ return null+ }+ const days = Number(value)+ return days >= AUDIT_LOG_RETENTION_MIN_DAYS && days <= AUDIT_LOG_RETENTION_MAX_DAYS ? days : null+}++export const auditLogRetentionCeiling = { get, parse }++const WHOLE_DAYS_PATTERN = /^\d+$/diff --git a/packages/server/api/src/app/helper/system-jobs/common.ts b/packages/server/api/src/app/helper/system-jobs/common.tsindex c04514726f7..11daf6cb8e4 100644--- a/packages/server/api/src/app/helper/system-jobs/common.ts+++ b/packages/server/api/src/app/helper/system-jobs/common.ts@@ -16,6 +16,7 @@ export enum SystemJobName {TOOL_SEARCH_REINDEX = 'tool-search-reindex',CHAT_STALE_SWEEP = 'chat-stale-sweep',WAITPOINT_DEADLINE_SWEEP = 'waitpoint-deadline-sweep',+ AUDIT_LOG_RETENTION = 'audit-log-retention',}type DeleteFlowDurableSystemJobData = {@@ -58,6 +59,7 @@ type SystemJobDataMap = {[SystemJobName.TOOL_SEARCH_REINDEX]: ToolSearchReindexSystemJobData[SystemJobName.CHAT_STALE_SWEEP]: Record<string, never>[SystemJobName.WAITPOINT_DEADLINE_SWEEP]: Record<string, never>+ [SystemJobName.AUDIT_LOG_RETENTION]: Record<string, never>}export type SystemJobData<T extends SystemJobName = SystemJobName> = T extends SystemJobName ? SystemJobDataMap[T] : neverdiff --git a/packages/server/api/src/app/helper/system-validator.ts b/packages/server/api/src/app/helper/system-validator.tsindex f25030034b4..cf257d55e4c 100644--- a/packages/server/api/src/app/helper/system-validator.ts+++ b/packages/server/api/src/app/helper/system-validator.ts@@ -1,6 +1,6 @@import { inspect } from 'util'import { isNil, tryCatchSync } from '@activepieces/core-utils'-import { ApEdition, ApEnvironment, DefaultProjectRole, ExecutionMode, FileLocation, maxBarrierSignalsBounds, NetworkMode, PieceSyncMode } from '@activepieces/shared'+import { ApEdition, ApEnvironment, AUDIT_LOG_RETENTION_MAX_DAYS, AUDIT_LOG_RETENTION_MIN_DAYS, DefaultProjectRole, ExecutionMode, FileLocation, maxBarrierSignalsBounds, NetworkMode, PieceSyncMode } from '@activepieces/shared'import { FastifyBaseLogger } from 'fastify'import { DatabaseType } from '../database/database-type'import { RedisType } from '../database/redis/types'@@ -8,6 +8,7 @@ import { s3Helper } from '../file/s3-helper'import { encryptUtils } from './encryption'import { jwtUtils } from './jwt-utils'import { networkUtils } from './network-utils'+import { auditLogRetentionCeiling } from './retention/audit-log-retention-ceiling'import { system } from './system/system'import { AppSystemProp, ContainerType, SystemProp } from './system/system-props'@@ -34,6 +35,10 @@ function positiveIntegerValidator(value: string | undefined) {return Number.isInteger(parsed) && parsed > 0 ? true : 'Value must be a positive integer'}+function auditLogRetentionDaysValidator(value: string | undefined) {+ return isNil(auditLogRetentionCeiling.parse(value)) ? `Value must be a whole number of days from ${AUDIT_LOG_RETENTION_MIN_DAYS} to ${AUDIT_LOG_RETENTION_MAX_DAYS}` : true+}+function positiveFiniteNumberValidator(value: string | undefined) {const parsed = isNil(value) ? Number.NaN : Number(value)const isValid = Number.isFinite(parsed) && parsed > 0@@ -149,6 +154,8 @@ const systemPropValidators: {[AppSystemProp.DEV_PIECES]: stringValidator,[AppSystemProp.ENCRYPTION_KEY]: stringValidator,[AppSystemProp.EXECUTION_DATA_RETENTION_DAYS]: numberValidator,+ [AppSystemProp.AUDIT_LOG_RETENTION_DAYS]: auditLogRetentionDaysValidator,+ [AppSystemProp.AUDIT_LOG_RETENTION_PAUSED]: booleanValidator,[AppSystemProp.JWT_SECRET]: stringValidator,[AppSystemProp.DEFAULT_CONCURRENT_JOBS_LIMIT]: numberValidator,[AppSystemProp.PIECES_SYNC_MODE]: enumValidator(Object.values(PieceSyncMode)),diff --git a/packages/server/api/src/app/helper/system/system-props.ts b/packages/server/api/src/app/helper/system/system-props.tsindex 24f2d5e32e0..284a132dc9e 100644--- a/packages/server/api/src/app/helper/system/system-props.ts+++ b/packages/server/api/src/app/helper/system/system-props.ts@@ -14,6 +14,8 @@ export enum AppSystemProp {API_RATE_LIMIT_AUTHN_WINDOW = 'API_RATE_LIMIT_AUTHN_WINDOW',API_RATE_LIMIT_EMAIL_CODE_MAX = 'API_RATE_LIMIT_EMAIL_CODE_MAX',APP_WEBHOOK_SECRETS = 'APP_WEBHOOK_SECRETS',+ AUDIT_LOG_RETENTION_DAYS = 'AUDIT_LOG_RETENTION_DAYS',+ AUDIT_LOG_RETENTION_PAUSED = 'AUDIT_LOG_RETENTION_PAUSED',APPSUMO_TOKEN = 'APPSUMO_TOKEN',APOLLO_API_KEY = 'APOLLO_API_KEY',AUTUMN_CONSOLE_URL = 'AUTUMN_CONSOLE_URL',diff --git a/packages/server/api/src/app/platform/platform.entity.ts b/packages/server/api/src/app/platform/platform.entity.tsindex 8a3e8ed0a8a..9c98260ab39 100644--- a/packages/server/api/src/app/platform/platform.entity.ts+++ b/packages/server/api/src/app/platform/platform.entity.ts@@ -95,6 +95,10 @@ export const PlatformEntity = new EntitySchema<PlatformSchema>({type: 'jsonb',nullable: true,},+ auditLogRetentionDays: {+ type: Number,+ nullable: true,+ },},indices: [{diff --git a/packages/server/api/src/app/platform/platform.service.ts b/packages/server/api/src/app/platform/platform.service.tsindex 4fa1c71567f..9be747c33d1 100644--- a/packages/server/api/src/app/platform/platform.service.ts+++ b/packages/server/api/src/app/platform/platform.service.ts@@ -1,5 +1,5 @@import { ActivepiecesError, apId, ErrorCode, isNil, PlatformId, spreadIfDefined, spreadIfNotUndefined, tryCatch, UserId } from '@activepieces/core-utils'-import { ApEdition, AuthenticationResponse, OPEN_SOURCE_PLAN, Platform, PlatformPlanLimits, PlatformRole, PlatformUsage, PlatformWithoutFederatedAuth, PlatformWithoutSensitiveData, ProjectType, SsoDomainVerification, SsoDomainVerificationStatus, UpdatePlatformRequestBody, User, UserStatus } from '@activepieces/shared'+import { ApEdition, AUDIT_LOG_RETENTION_MAX_DAYS, AUDIT_LOG_RETENTION_MIN_DAYS, AuthenticationResponse, OPEN_SOURCE_PLAN, Platform, PlatformPlanLimits, PlatformRole, PlatformUsage, PlatformWithoutFederatedAuth, PlatformWithoutSensitiveData, ProjectType, SsoDomainVerification, SsoDomainVerificationStatus, UpdatePlatformRequestBody, User, UserStatus } from '@activepieces/shared'import { FastifyBaseLogger } from 'fastify'import { nanoid } from 'nanoid'import { authenticationUtils } from '../authentication/authentication-utils'@@ -10,6 +10,7 @@ import { invalidateSamlClientCache } from '../ee/authentication/saml-authn/saml-import { platformPlanService } from '../ee/platform/platform-plan/platform-plan.service'import { defaultTheme } from '../flags/theme'import { rejectedPromiseHandler } from '../helper/promise-handler'+import { auditLogRetentionCeiling } from '../helper/retention/audit-log-retention-ceiling'import { system } from '../helper/system/system'import { telemetry } from '../helper/telemetry.utils'import { projectService } from '../project/project-service'@@ -168,6 +169,7 @@ export const platformService = (log: FastifyBaseLogger) => ({const platform = params.federatedAuthProviders !== undefined? await this.getOneWithFederatedAuthOrThrow(params.id): await this.getOneOrThrow(params.id)+ await assertAuditLogRetentionDaysAllowed({ log, platform, auditLogRetentionDays: params.auditLogRetentionDays })const federatedAuthProviders = hasFederatedAuth(platform)? {...platform.federatedAuthProviders,@@ -197,6 +199,7 @@ export const platformService = (log: FastifyBaseLogger) => ({...spreadIfDefined('ssoDomainVerification', params.ssoDomainVerification),...spreadIfDefined('pinnedPieces', params.pinnedPieces),...spreadIfNotUndefined('pieceSelectorConfig', params.pieceSelectorConfig),+ ...spreadIfNotUndefined('auditLogRetentionDays', params.auditLogRetentionDays),}if (!isNil(params.plan)) {await platformPlanService(log).update({@@ -209,6 +212,10 @@ export const platformService = (log: FastifyBaseLogger) => ({}log.info({ platform: { id: params.id } }, 'Platform updated')const saved = await platformRepo().save(updatedPlatform)+ const previousRetentionDays = platform.auditLogRetentionDays ?? null+ if (params.auditLogRetentionDays !== undefined && params.auditLogRetentionDays !== previousRetentionDays) {+ log.info({ platform: { id: params.id }, previousRetentionDays, retentionDays: params.auditLogRetentionDays }, 'Audit log retention updated')+ }return stripFederatedAuth(saved)},async getOneOrThrow(id: PlatformId): Promise<PlatformWithoutFederatedAuth> {@@ -386,6 +393,33 @@ async function getBillingEnforced(log: FastifyBaseLogger, platformId: PlatformIdreturn data ?? undefined}+async function assertAuditLogRetentionDaysAllowed({ log, platform, auditLogRetentionDays }: AssertAuditLogRetentionDaysAllowedParams): Promise<void> {+ if (auditLogRetentionDays === undefined) {+ return+ }+ const plan = await getPlan(log, platform)+ if (!plan.auditLogEnabled) {+ throw new ActivepiecesError({+ code: ErrorCode.FEATURE_DISABLED,+ params: {+ message: 'Audit logs are not enabled for this platform',+ },+ })+ }+ if (isNil(auditLogRetentionDays)) {+ return+ }+ const maxDays = auditLogRetentionCeiling.get() ?? AUDIT_LOG_RETENTION_MAX_DAYS+ if (auditLogRetentionDays < AUDIT_LOG_RETENTION_MIN_DAYS || auditLogRetentionDays > maxDays) {+ throw new ActivepiecesError({+ code: ErrorCode.VALIDATION,+ params: {+ message: `auditLogRetentionDays must be between ${AUDIT_LOG_RETENTION_MIN_DAYS} and ${maxDays}`,+ },+ })+ }+}+async function getPlan(log: FastifyBaseLogger, platform: PlatformWithoutFederatedAuth): Promise<PlatformPlanLimits> {const edition = system.getEdition()if (edition === ApEdition.COMMUNITY) {@@ -416,6 +450,12 @@ type AddParams = {type NewPlatform = Omit<Platform, 'created' | 'updated'>+type AssertAuditLogRetentionDaysAllowedParams = {+ log: FastifyBaseLogger+ platform: PlatformWithoutFederatedAuth+ auditLogRetentionDays: number | null | undefined+}+type UpdateParams = UpdatePlatformRequestBody & {id: PlatformIdplan?: Partial<PlatformPlanLimits>diff --git a/packages/server/api/test/helpers/mocks/index.ts b/packages/server/api/test/helpers/mocks/index.tsindex 30df2758714..de6203ab615 100644--- a/packages/server/api/test/helpers/mocks/index.ts+++ b/packages/server/api/test/helpers/mocks/index.ts@@ -216,6 +216,7 @@ export const createMockPlatform = (platform?: Partial<Platform>): Platform => {googleAuthEnabled: platform?.googleAuthEnabled ?? true,ssoDomain: platform?.ssoDomain ?? null,ssoDomainVerification: platform?.ssoDomainVerification ?? null,+ auditLogRetentionDays: platform?.auditLogRetentionDays ?? null,}}diff --git a/packages/server/api/test/integration/cloud/audit-event/audit-log-retention.test.ts b/packages/server/api/test/integration/cloud/audit-event/audit-log-retention.test.tsnew file mode 100644index 00000000000..6b47ef450b1--- /dev/null+++ b/packages/server/api/test/integration/cloud/audit-event/audit-log-retention.test.ts@@ -0,0 +1,363 @@+import { ApFlagId } from '@activepieces/shared'+import dayjs from 'dayjs'+import { FastifyInstance } from 'fastify'+import { StatusCodes } from 'http-status-codes'+import { In } from 'typeorm'+import { databaseConnection } from '../../../../src/app/database/database-connection'+import { auditLogRepo } from '../../../../src/app/ee/audit-logs/audit-event-service'+import { auditLogRetention } from '../../../../src/app/ee/audit-logs/audit-log-retention'+import { db } from '../../../helpers/db'+import { createAuditEvent } from '../../../helpers/mocks'+import { createTestContext } from '../../../helpers/test-context'+import { setupTestEnvironment, teardownTestEnvironment } from '../../../helpers/test-setup'++let app: FastifyInstance | null = null++beforeAll(async () => {+ app = await setupTestEnvironment()+})++afterEach(async () => {+ delete process.env.AP_AUDIT_LOG_RETENTION_DAYS+ delete process.env.AP_AUDIT_LOG_RETENTION_PAUSED+ await databaseConnection().query('DROP TRIGGER IF EXISTS audit_event_test_trigger ON "audit_event"')+ await databaseConnection().query('DROP FUNCTION IF EXISTS audit_event_test_trigger()')+})++afterAll(async () => {+ await teardownTestEnvironment()+})++const daysAgo = (days: number): string => dayjs().subtract(days, 'days').toISOString()++const saveEvents = async ({ platformId, ages }: { platformId: string, ages: number[] }): Promise<string[]> => {+ const events = ages.map((age) => createAuditEvent({ platformId, created: daysAgo(age), updated: daysAgo(age) }))+ await db.save('audit_event', events)+ return events.map((event) => event.id)+}++const installDeleteTrigger = async ({ body }: { body: string }): Promise<void> => {+ await databaseConnection().query(`CREATE FUNCTION audit_event_test_trigger() RETURNS trigger LANGUAGE plpgsql AS $$ BEGIN ${body} RETURN OLD; END $$`)+ await databaseConnection().query('CREATE TRIGGER audit_event_test_trigger BEFORE DELETE ON "audit_event" FOR EACH ROW EXECUTE FUNCTION audit_event_test_trigger()')+}++const autovacuumOptions = async (): Promise<string[]> => {+ const rows: { reloptions: string[] | null }[] = await databaseConnection().query('SELECT reloptions FROM pg_class WHERE oid = \'"audit_event"\'::regclass')+ return rows[0]?.reloptions ?? []+}++const remainingIds = async (ids: string[]): Promise<string[]> => {+ const rows = await auditLogRepo().findBy({ id: In(ids) })+ return rows.map((row) => row.id).sort()+}++const createPlatform = async ({ auditLogRetentionDays, auditLogEnabled = true }: { auditLogRetentionDays: number | null, auditLogEnabled?: boolean }) => {+ return createTestContext(app!, {+ platform: { auditLogRetentionDays },+ plan: { auditLogEnabled },+ })+}++describe('auditLogRetention.sweep', () => {+ beforeEach(async () => {+ await databaseConnection().query('DELETE FROM "audit_event"')+ })++ it('deletes only the expired events of the platform that set a retention, and leaves other platforms alone', async () => {+ const withRetention = await createPlatform({ auditLogRetentionDays: 30 })+ const withoutRetention = await createPlatform({ auditLogRetentionDays: null })+ const [expired, fresh] = await Promise.all([+ saveEvents({ platformId: withRetention.platform.id, ages: [45, 400] }),+ saveEvents({ platformId: withRetention.platform.id, ages: [1, 29] }),+ ])+ const untouched = await saveEvents({ platformId: withoutRetention.platform.id, ages: [45, 400] })++ await auditLogRetention(app!.log).sweep()++ expect(await remainingIds([...expired, ...fresh, ...untouched])).toStrictEqual([...fresh, ...untouched].sort())+ })++ it('keeps every event when neither the platform nor the instance sets a retention', async () => {+ const ctx = await createPlatform({ auditLogRetentionDays: null })+ const ids = await saveEvents({ platformId: ctx.platform.id, ages: [100, 1000, 3000] })++ await auditLogRetention(app!.log).sweep()++ expect(await remainingIds(ids)).toStrictEqual([...ids].sort())+ })++ it('applies the instance ceiling to platforms without a value and to platforms with a longer value', async () => {+ process.env.AP_AUDIT_LOG_RETENTION_DAYS = '60'+ const inherits = await createPlatform({ auditLogRetentionDays: null })+ const longer = await createPlatform({ auditLogRetentionDays: 90 })+ const inheritsExpired = await saveEvents({ platformId: inherits.platform.id, ages: [61] })+ const inheritsFresh = await saveEvents({ platformId: inherits.platform.id, ages: [59] })+ const longerExpired = await saveEvents({ platformId: longer.platform.id, ages: [75] })+ const longerFresh = await saveEvents({ platformId: longer.platform.id, ages: [10] })++ await auditLogRetention(app!.log).sweep()++ expect(await remainingIds([...inheritsExpired, ...inheritsFresh, ...longerExpired, ...longerFresh]))+ .toStrictEqual([...inheritsFresh, ...longerFresh].sort())+ })++ it('ignores the saved period of a platform whose plan has no audit logs', async () => {+ const ctx = await createPlatform({ auditLogRetentionDays: 30, auditLogEnabled: false })+ const ids = await saveEvents({ platformId: ctx.platform.id, ages: [45, 400] })++ await auditLogRetention(app!.log).sweep()++ expect(await remainingIds(ids)).toStrictEqual([...ids].sort())+ })++ it('applies only the instance ceiling to a platform whose plan has no audit logs', async () => {+ process.env.AP_AUDIT_LOG_RETENTION_DAYS = '60'+ const ctx = await createPlatform({ auditLogRetentionDays: 30, auditLogEnabled: false })+ const [fresh, expired] = await saveEvents({ platformId: ctx.platform.id, ages: [45, 400] })++ await auditLogRetention(app!.log).sweep()++ expect(await remainingIds([fresh, expired])).toStrictEqual([fresh])+ })++ it('applies only the instance ceiling to a platform without a plan row', async () => {+ process.env.AP_AUDIT_LOG_RETENTION_DAYS = '60'+ const ctx = await createPlatform({ auditLogRetentionDays: 30 })+ await databaseConnection().query('DELETE FROM "platform_plan" WHERE "platformId" = $1', [ctx.platform.id])+ const [fresh, expired] = await saveEvents({ platformId: ctx.platform.id, ages: [45, 400] })++ await auditLogRetention(app!.log).sweep()++ expect(await remainingIds([fresh, expired])).toStrictEqual([fresh])+ })++ it.each(['0', '7'])('ignores the malformed or too short ceiling %j instead of deleting everything', async (ceiling) => {+ process.env.AP_AUDIT_LOG_RETENTION_DAYS = ceiling+ const ctx = await createPlatform({ auditLogRetentionDays: null })+ const ids = await saveEvents({ platformId: ctx.platform.id, ages: [1, 100] })++ await auditLogRetention(app!.log).sweep()++ expect(await remainingIds(ids)).toStrictEqual([...ids].sort())+ })++ it('gives one platform more rounds while the run has budget, oldest first', async () => {+ const ctx = await createPlatform({ auditLogRetentionDays: 30 })+ const ids = await saveEvents({ platformId: ctx.platform.id, ages: [100, 90, 80, 70, 60, 50, 40] })++ const firstRun = await auditLogRetention(app!.log).sweep({ batchSize: 2, maxRowsPerPlatformPerRound: 3, maxRowsPerRun: 5 })++ expect(firstRun.deletedCount).toBe(5)+ expect(firstRun.stoppedBy).toBe('rowLimit')+ expect(firstRun.platformsLeft).toBe(1)+ expect(await remainingIds(ids)).toStrictEqual(ids.slice(5).sort())++ const secondRun = await auditLogRetention(app!.log).sweep({ batchSize: 2, maxRowsPerPlatformPerRound: 3 })++ expect(secondRun.deletedCount).toBe(2)+ expect(secondRun.stoppedBy).toBe('done')+ expect(secondRun.platformsLeft).toBe(0)+ expect(await remainingIds(ids)).toStrictEqual([])+ })++ it('shares the run between platforms in rounds', async () => {+ const first = await createPlatform({ auditLogRetentionDays: 30 })+ const second = await createPlatform({ auditLogRetentionDays: 30 })+ const firstIds = await saveEvents({ platformId: first.platform.id, ages: [40, 50, 60, 70, 80] })+ const secondIds = await saveEvents({ platformId: second.platform.id, ages: [40, 50, 60, 70, 80] })++ const summary = await auditLogRetention(app!.log).sweep({ batchSize: 2, maxRowsPerPlatformPerRound: 2, maxRowsPerRun: 6 })++ expect(summary.deletedCount).toBe(6)+ expect((await remainingIds(firstIds)).length).toBeLessThanOrEqual(3)+ expect((await remainingIds(secondIds)).length).toBeLessThanOrEqual(3)+ })++ it('deletes every expired event when many share the same created time', async () => {+ const ctx = await createPlatform({ auditLogRetentionDays: 30 })+ const created = daysAgo(40)+ const events = Array.from({ length: 7 }, () => createAuditEvent({ platformId: ctx.platform.id, created, updated: created }))+ await db.save('audit_event', events)++ const summary = await auditLogRetention(app!.log).sweep({ batchSize: 2 })++ expect(summary.deletedCount).toBe(7)+ expect(await remainingIds(events.map((event) => event.id))).toStrictEqual([])+ })++ it('reports the platforms left when the time budget runs out', async () => {+ const ctx = await createPlatform({ auditLogRetentionDays: 30 })+ const ids = await saveEvents({ platformId: ctx.platform.id, ages: [40, 50] })++ const summary = await auditLogRetention(app!.log).sweep({ runBudgetMs: 0 })++ expect(summary.deletedCount).toBe(0)+ expect(summary.platformsLeft).toBe(1)+ expect(summary.stoppedBy).toBe('timeBudget')+ expect(await remainingIds(ids)).toStrictEqual([...ids].sort())+ })++ it('deletes nothing while AP_AUDIT_LOG_RETENTION_PAUSED is true', async () => {+ process.env.AP_AUDIT_LOG_RETENTION_PAUSED = 'true'+ const ctx = await createPlatform({ auditLogRetentionDays: 30 })+ const ids = await saveEvents({ platformId: ctx.platform.id, ages: [40, 50] })++ const summary = await auditLogRetention(app!.log).sweep()++ expect(summary.deletedCount).toBe(0)+ expect(summary.stoppedBy).toBe('paused')+ expect(await remainingIds(ids)).toStrictEqual([...ids].sort())+ })++ it('reads the retention period again before each batch', async () => {+ const ctx = await createPlatform({ auditLogRetentionDays: 30 })+ const ids = await saveEvents({ platformId: ctx.platform.id, ages: [100, 90, 80, 70] })+ await installDeleteTrigger({+ body: `IF OLD.id = '${ids[0]}' THEN UPDATE "platform" SET "auditLogRetentionDays" = NULL WHERE id = '${ctx.platform.id}'; END IF;`,+ })++ const summary = await auditLogRetention(app!.log).sweep({ batchSize: 2 })++ expect(summary.deletedCount).toBe(2)+ expect(await remainingIds(ids)).toStrictEqual(ids.slice(2).sort())+ })++ it('counts the batches a platform deleted before it failed', async () => {+ const ctx = await createPlatform({ auditLogRetentionDays: 30 })+ const ids = await saveEvents({ platformId: ctx.platform.id, ages: [100, 90, 80, 70] })+ await installDeleteTrigger({+ body: `IF OLD.id = '${ids[2]}' THEN RAISE EXCEPTION 'poisoned row'; END IF;`,+ })++ const summary = await auditLogRetention(app!.log).sweep({ batchSize: 2 })++ expect(summary.deletedCount).toBe(2)+ expect(summary.platformsFailed).toBe(1)+ expect(summary.stoppedBy).toBe('failed')+ expect(await remainingIds(ids)).toStrictEqual(ids.slice(2).sort())+ })++ it('sets the autovacuum scale factor of audit_event the first time it has events to delete', async () => {+ await databaseConnection().query('ALTER TABLE "audit_event" RESET (autovacuum_vacuum_scale_factor)')+ const ctx = await createPlatform({ auditLogRetentionDays: 30 })+ await saveEvents({ platformId: ctx.platform.id, ages: [40] })++ await auditLogRetention(app!.log).sweep()++ expect(await autovacuumOptions()).toContain('autovacuum_vacuum_scale_factor=0.02')+ })++ it('keeps an autovacuum scale factor that an operator already set', async () => {+ await databaseConnection().query('ALTER TABLE "audit_event" SET (autovacuum_vacuum_scale_factor = 0.05)')+ const ctx = await createPlatform({ auditLogRetentionDays: 30 })+ await saveEvents({ platformId: ctx.platform.id, ages: [40] })++ await auditLogRetention(app!.log).sweep()++ expect(await autovacuumOptions()).toContain('autovacuum_vacuum_scale_factor=0.05')+ await databaseConnection().query('ALTER TABLE "audit_event" RESET (autovacuum_vacuum_scale_factor)')+ })++ it('stops the whole run at the run limit', async () => {+ const first = await createPlatform({ auditLogRetentionDays: 30 })+ const second = await createPlatform({ auditLogRetentionDays: 30 })+ const ids = [+ ...await saveEvents({ platformId: first.platform.id, ages: [40, 50, 60] }),+ ...await saveEvents({ platformId: second.platform.id, ages: [40, 50, 60] }),+ ]++ const summary = await auditLogRetention(app!.log).sweep({ maxRowsPerRun: 4 })++ expect(summary.deletedCount).toBe(4)+ expect(summary.stoppedBy).toBe('rowLimit')+ expect(await remainingIds(ids)).toHaveLength(2)+ })+})++describe('List audit events', () => {+ it('returns the oldest event first when asked for ascending order', async () => {+ const ctx = await createPlatform({ auditLogRetentionDays: null })+ const other = await createPlatform({ auditLogRetentionDays: null })+ const [newest, oldest, middle] = await saveEvents({ platformId: ctx.platform.id, ages: [10, 300, 50] })+ await saveEvents({ platformId: other.platform.id, ages: [900] })++ const ascending = await ctx.get('/v1/audit-events', { order: 'ASC', limit: 1 })+ const descending = await ctx.get('/v1/audit-events', { limit: 3 })++ expect(ascending.statusCode).toBe(StatusCodes.OK)+ expect(ascending.json().data.map((event: ApplicationEvent) => event.id)).toStrictEqual([oldest])+ expect(descending.json().data.map((event: ApplicationEvent) => event.id)).toStrictEqual([newest, middle, oldest])+ })+})++describe('Update platform audit log retention', () => {+ it('saves a value inside the bounds, returns it, and clears it with null', async () => {+ const ctx = await createPlatform({ auditLogRetentionDays: null })++ const saved = await ctx.post(`/v1/platforms/${ctx.platform.id}`, { auditLogRetentionDays: 90 })++ expect(saved.statusCode).toBe(StatusCodes.OK)+ expect(saved.json().auditLogRetentionDays).toBe(90)++ const cleared = await ctx.post(`/v1/platforms/${ctx.platform.id}`, { auditLogRetentionDays: null })++ expect(cleared.statusCode).toBe(StatusCodes.OK)+ expect(cleared.json().auditLogRetentionDays).toBeNull()+ })++ it.each([29, 3651])('rejects %i days when the instance sets no ceiling', async (days) => {+ const ctx = await createPlatform({ auditLogRetentionDays: null })++ const response = await ctx.post(`/v1/platforms/${ctx.platform.id}`, { auditLogRetentionDays: days })++ expect(response.statusCode).toBe(StatusCodes.CONFLICT)+ })++ it('rejects a value above the instance ceiling', async () => {+ process.env.AP_AUDIT_LOG_RETENTION_DAYS = '365'+ const ctx = await createPlatform({ auditLogRetentionDays: null })++ const response = await ctx.post(`/v1/platforms/${ctx.platform.id}`, { auditLogRetentionDays: 366 })++ expect(response.statusCode).toBe(StatusCodes.CONFLICT)+ })++ it('accepts the minimum when the instance ceiling is under it, because that ceiling is ignored', async () => {+ process.env.AP_AUDIT_LOG_RETENTION_DAYS = '7'+ const ctx = await createPlatform({ auditLogRetentionDays: null })++ const response = await ctx.post(`/v1/platforms/${ctx.platform.id}`, { auditLogRetentionDays: 30 })++ expect(response.statusCode).toBe(StatusCodes.OK)+ })++ it('refuses the change when the plan has no audit logs', async () => {+ const ctx = await createTestContext(app!, { plan: { auditLogEnabled: false } })++ const response = await ctx.post(`/v1/platforms/${ctx.platform.id}`, { auditLogRetentionDays: 90 })++ expect(response.statusCode).toBe(StatusCodes.PAYMENT_REQUIRED)+ const platform = await ctx.get(`/v1/platforms/${ctx.platform.id}`)+ expect(platform.json().auditLogRetentionDays).toBeNull()+ })++ it('exposes the instance ceiling as a flag', async () => {+ process.env.AP_AUDIT_LOG_RETENTION_DAYS = '365'+ const ctx = await createPlatform({ auditLogRetentionDays: null })++ const response = await ctx.get('/v1/flags')++ expect(response.statusCode).toBe(StatusCodes.OK)+ expect(response.json()[ApFlagId.AUDIT_LOG_RETENTION_DAYS]).toBe(365)+ })++ it('exposes the paused cleanup as a flag', async () => {+ process.env.AP_AUDIT_LOG_RETENTION_PAUSED = 'true'+ const ctx = await createPlatform({ auditLogRetentionDays: null })++ const response = await ctx.get('/v1/flags')++ expect(response.statusCode).toBe(StatusCodes.OK)+ expect(response.json()[ApFlagId.AUDIT_LOG_RETENTION_PAUSED]).toBe(true)+ })+})diff --git a/packages/server/api/test/integration/cloud/platform/platform.test.ts b/packages/server/api/test/integration/cloud/platform/platform.test.tsindex e47cbd1040a..0d2b1d74ce0 100644--- a/packages/server/api/test/integration/cloud/platform/platform.test.ts+++ b/packages/server/api/test/integration/cloud/platform/platform.test.ts@@ -578,6 +578,7 @@ describe('Platform API', () => {expect(Object.keys(responseBody).sort()).toStrictEqual(['allowedAuthDomains','allowedEmbedOrigins',+ 'auditLogRetentionDays','autoCreatePersonalProjects','billingEnforced','cloudAuthEnabled',diff --git a/packages/server/api/test/unit/app/helper/audit-log-retention-ceiling.test.ts b/packages/server/api/test/unit/app/helper/audit-log-retention-ceiling.test.tsnew file mode 100644index 00000000000..84e03a81d93--- /dev/null+++ b/packages/server/api/test/unit/app/helper/audit-log-retention-ceiling.test.ts@@ -0,0 +1,24 @@+import { afterEach, describe, expect, it } from 'vitest'+import { auditLogRetentionCeiling } from '../../../../src/app/helper/retention/audit-log-retention-ceiling'++afterEach(() => {+ delete process.env.AP_AUDIT_LOG_RETENTION_DAYS+})++describe('auditLogRetentionCeiling', () => {+ it('keeps audit events forever when the variable is not set', () => {+ expect(auditLogRetentionCeiling.get()).toBeNull()+ })++ it.each(['', '0', '1', '29', '-5', '1e3', '1,000', '1_000', '30.5', ' 30', 'abc', '3651'])('ignores the malformed or out-of-range value %j instead of reading a smaller number', (value) => {+ process.env.AP_AUDIT_LOG_RETENTION_DAYS = value++ expect(auditLogRetentionCeiling.get()).toBeNull()+ })++ it.each([['30', 30], ['365', 365], ['3650', 3650]])('reads the whole number of days %j', (value, days) => {+ process.env.AP_AUDIT_LOG_RETENTION_DAYS = value++ expect(auditLogRetentionCeiling.get()).toBe(days)+ })+})diff --git a/packages/web/public/locales/en/translation.json b/packages/web/public/locales/en/translation.jsonindex 8f5d2d77954..4e40e91c214 100644--- a/packages/web/public/locales/en/translation.json+++ b/packages/web/public/locales/en/translation.json@@ -3322,5 +3322,18 @@"MCP clients will be able to use the delete tools you are turning on in this project. Deleted items cannot be restored.": "MCP clients will be able to use the delete tools you are turning on in this project. Deleted items cannot be restored.","{tier}: {status}": "{tier}: {status}","{tier}, {status}": "{tier}, {status}",- "Show full output": "Show full output"+ "Show full output": "Show full output",+ "Retention: {period}": "Retention: {period}",+ "Audit log retention": "Audit log retention",+ "Events older than the retention period are deleted every hour.": "Events older than the retention period are deleted every hour.",+ "Keep events for": "Keep events for",+ "Cleanup is paused on this instance. No events are deleted until it resumes.": "Cleanup is paused on this instance. No events are deleted until it resumes.",+ "Events older than {period}, created before {date}, will be permanently deleted. This cannot be undone.": "Events older than {period}, created before {date}, will be permanently deleted. This cannot be undone.",+ "Oldest event: {date}": "Oldest event: {date}",+ "Oldest event: {date}. Older events are still being deleted, in batches every hour.": "Oldest event: {date}. Older events are still being deleted, in batches every hour.",+ "Forever": "Forever",+ "Instance limit ({period})": "Instance limit ({period})",+ "6 months": "6 months",+ "1 year": "1 year",+ "retentionPeriodDays": "{days, plural, =1 {1 day} other {# days}}"}diff --git a/packages/web/src/app/routes/platform/security/audit-logs/index.tsx b/packages/web/src/app/routes/platform/security/audit-logs/index.tsxindex 3f456e6bc19..d70b2efe0da 100644--- a/packages/web/src/app/routes/platform/security/audit-logs/index.tsx+++ b/packages/web/src/app/routes/platform/security/audit-logs/index.tsx@@ -42,7 +42,10 @@ import {SheetHeader,SheetTitle,} from '@/components/ui/sheet';-import { auditLogQueries } from '@/features/platform-admin';+import {+ AuditLogRetentionButton,+ auditLogQueries,+} from '@/features/platform-admin';import { platformUserHooks } from '@/features/platform-admin/hooks/platform-user-hooks';import { projectCollectionUtils } from '@/features/projects';import { platformHooks } from '@/hooks/platform-hooks';@@ -121,7 +124,9 @@ export default function AuditLogsPage() {<DashboardPageHeaderdescription={t('Track activities done within your platform')}title={t('Audit Logs')}- />+ >+ {!isSample && <AuditLogRetentionButton />}+ </DashboardPageHeader><DataTableemptyStateTextTitle={t('No audit logs found')}emptyStateTextDescription={t(diff --git a/packages/web/src/features/platform-admin/components/audit-log-retention-dialog.tsx b/packages/web/src/features/platform-admin/components/audit-log-retention-dialog.tsxnew file mode 100644index 00000000000..7530b39be7f--- /dev/null+++ b/packages/web/src/features/platform-admin/components/audit-log-retention-dialog.tsx@@ -0,0 +1,253 @@+import { isNil } from '@activepieces/core-utils';+import { ApFlagId } from '@activepieces/shared';+import { useMutation } from '@tanstack/react-query';+import { t } from 'i18next';+import { CirclePause, Clock, TriangleAlert } from 'lucide-react';+import { useState } from 'react';+import { toast } from 'sonner';++import { platformApi } from '@/api/platforms-api';+import { Alert, AlertDescription } from '@/components/ui/alert';+import { Button } from '@/components/ui/button';+import {+ Dialog,+ DialogContent,+ DialogDescription,+ DialogFooter,+ DialogHeader,+ DialogTitle,+ DialogTrigger,+} from '@/components/ui/dialog';+import {+ Select,+ SelectContent,+ SelectItem,+ SelectTrigger,+ SelectValue,+} from '@/components/ui/select';+import { flagsHooks } from '@/hooks/flags-hooks';+import { platformHooks } from '@/hooks/platform-hooks';+import { formatUtils } from '@/lib/format-utils';++import { auditLogQueries } from '../hooks/audit-log-hooks';+import { auditLogRetentionUtils } from '../lib/audit-log-retention-utils';++export function AuditLogRetentionButton() {+ const [open, setOpen] = useState(false);+ const { platform, refetch } = platformHooks.useCurrentPlatform();+ const { data: ceilingFlag } = flagsHooks.useFlag<number | null>(+ ApFlagId.AUDIT_LOG_RETENTION_DAYS,+ );+ const ceiling = ceilingFlag ?? null;+ const { data: pausedFlag } = flagsHooks.useFlag<boolean>(+ ApFlagId.AUDIT_LOG_RETENTION_PAUSED,+ );+ const savedDays = platform.auditLogRetentionDays ?? null;+ const currentDays = auditLogRetentionUtils.effectiveDays({+ days: savedDays,+ ceiling,+ });++ return (+ <Dialog open={open} onOpenChange={setOpen}>+ <DialogTrigger asChild>+ <Button variant="outline" size="sm">+ <Clock className="size-4" />+ {t('Retention: {period}', { period: formatPeriod(currentDays) })}+ </Button>+ </DialogTrigger>+ <DialogContent>+ <AuditLogRetentionForm+ key={open ? 'open' : 'closed'}+ platformId={platform.id}+ savedDays={savedDays}+ ceiling={ceiling}+ paused={pausedFlag === true}+ refetchPlatform={refetch}+ onClose={() => setOpen(false)}+ />+ </DialogContent>+ </Dialog>+ );+}++function AuditLogRetentionForm({+ platformId,+ savedDays,+ ceiling,+ paused,+ refetchPlatform,+ onClose,+}: AuditLogRetentionFormProps) {+ const options = auditLogRetentionUtils.buildOptions({ savedDays, ceiling });+ const initialValue = toOptionValue(+ auditLogRetentionUtils.initialSelection({ savedDays, ceiling }),+ );+ const [selectedValue, setSelectedValue] = useState(initialValue);+ const selectedDays = fromOptionValue(selectedValue);+ const nextDays = auditLogRetentionUtils.effectiveDays({+ days: selectedDays,+ ceiling,+ });+ const currentDays = auditLogRetentionUtils.effectiveDays({+ days: savedDays,+ ceiling,+ });+ const deletesEvents = auditLogRetentionUtils.deletesEvents({+ next: nextDays,+ current: currentDays,+ });+ const now = new Date();+ const { data: oldestEventCreated } = auditLogQueries.useOldestEventCreated();++ const { mutate, isPending } = useMutation({+ mutationFn: async () => {+ await platformApi.update(+ { auditLogRetentionDays: selectedDays },+ platformId,+ );+ await refetchPlatform();+ },+ onSuccess: () => {+ toast.success(t('Your changes have been saved.'), { duration: 3000 });+ onClose();+ },+ onError: () => {+ toast.error(t('Failed to save changes. Please try again.'));+ },+ });++ return (+ <>+ <DialogHeader>+ <DialogTitle>{t('Audit log retention')}</DialogTitle>+ <DialogDescription>+ {t('Events older than the retention period are deleted every hour.')}+ </DialogDescription>+ </DialogHeader>+ <div className="flex flex-col gap-2">+ <span className="text-sm font-medium">{t('Keep events for')}</span>+ <Select value={selectedValue} onValueChange={setSelectedValue}>+ <SelectTrigger>+ <SelectValue />+ </SelectTrigger>+ <SelectContent>+ {options.map((days) => (+ <SelectItem key={toOptionValue(days)} value={toOptionValue(days)}>+ {formatOption({ days, ceiling })}+ </SelectItem>+ ))}+ </SelectContent>+ </Select>+ </div>+ {paused && (+ <Alert>+ <CirclePause className="size-4" />+ <AlertDescription>+ {t(+ 'Cleanup is paused on this instance. No events are deleted until it resumes.',+ )}+ </AlertDescription>+ </Alert>+ )}+ {!isNil(oldestEventCreated) && (+ <p className="text-sm text-gray-11">+ {!paused &&+ auditLogRetentionUtils.isCleanupPending({+ oldestEventCreated,+ days: currentDays,+ now,+ })+ ? t(+ 'Oldest event: {date}. Older events are still being deleted, in batches every hour.',+ {+ date: formatUtils.formatDateOnly(+ new Date(oldestEventCreated),+ ),+ },+ )+ : t('Oldest event: {date}', {+ date: formatUtils.formatDateOnly(new Date(oldestEventCreated)),+ })}+ </p>+ )}+ {deletesEvents && !isNil(nextDays) && (+ <Alert variant="destructive">+ <TriangleAlert className="size-4" />+ <AlertDescription>+ {t(+ 'Events older than {period}, created before {date}, will be permanently deleted. This cannot be undone.',+ {+ period: formatPeriod(nextDays),+ date: formatUtils.formatDateOnly(+ auditLogRetentionUtils.cutoffDate({ days: nextDays, now }),+ ),+ },+ )}+ </AlertDescription>+ </Alert>+ )}+ <DialogFooter>+ <Button type="button" variant="outline" onClick={onClose}>+ {t('Cancel')}+ </Button>+ <Button+ type="button"+ variant={deletesEvents ? 'destructive' : 'default'}+ disabled={selectedValue === initialValue || isPending}+ loading={isPending}+ onClick={() => mutate()}+ >+ {t('Save')}+ </Button>+ </DialogFooter>+ </>+ );+}++function toOptionValue(days: number | null): string {+ return isNil(days) ? INSTANCE_OPTION_VALUE : String(days);+}++function fromOptionValue(value: string): number | null {+ return value === INSTANCE_OPTION_VALUE ? null : Number(value);+}++function formatOption({+ days,+ ceiling,+}: {+ days: number | null;+ ceiling: number | null;+}): string {+ if (!isNil(days)) {+ return formatPeriod(days);+ }+ return isNil(ceiling)+ ? t('Forever')+ : t('Instance limit ({period})', { period: formatPeriod(ceiling) });+}++function formatPeriod(days: number | null): string {+ if (isNil(days)) {+ return t('Forever');+ }+ if (days === 180) {+ return t('6 months');+ }+ if (days === 365) {+ return t('1 year');+ }+ return t('retentionPeriodDays', { days });+}++const INSTANCE_OPTION_VALUE = 'instance';++type AuditLogRetentionFormProps = {+ platformId: string;+ savedDays: number | null;+ ceiling: number | null;+ paused: boolean;+ refetchPlatform: () => Promise<void>;+ onClose: () => void;+};diff --git a/packages/web/src/features/platform-admin/hooks/audit-log-hooks.ts b/packages/web/src/features/platform-admin/hooks/audit-log-hooks.tsindex ebfe93b26d1..6671ba58376 100644--- a/packages/web/src/features/platform-admin/hooks/audit-log-hooks.ts+++ b/packages/web/src/features/platform-admin/hooks/audit-log-hooks.ts@@ -11,6 +11,7 @@ import { auditEventsApi } from '../api/audit-events-api';export const auditLogKeys = {all: (searchParams: string) => ['audit-logs', searchParams] as const,+ oldest: ['audit-logs', 'oldest'] as const,};export const auditLogQueries = {@@ -40,4 +41,14 @@ export const auditLogQueries = {},});},+ useOldestEventCreated: () => {+ return useQuery({+ queryKey: auditLogKeys.oldest,+ staleTime: 0,+ queryFn: async () => {+ const page = await auditEventsApi.list({ order: 'ASC', limit: 1 });+ return page.data[0]?.created ?? null;+ },+ });+ },};diff --git a/packages/web/src/features/platform-admin/index.ts b/packages/web/src/features/platform-admin/index.tsindex e8d9dd81445..d065ac495b8 100644--- a/packages/web/src/features/platform-admin/index.ts+++ b/packages/web/src/features/platform-admin/index.ts@@ -13,6 +13,7 @@ export {embedSubdomainMutations,embedSubdomainKeys,} from './hooks/embed-subdomain-hooks';+export { AuditLogRetentionButton } from './components/audit-log-retention-dialog';export { NewSigningKeyDialog } from './components/new-signing-key-dialog';export { platformAnalyticsHooks } from './hooks/analytics-hooks';export {diff --git a/packages/web/src/features/platform-admin/lib/audit-log-retention-utils.ts b/packages/web/src/features/platform-admin/lib/audit-log-retention-utils.tsnew file mode 100644index 00000000000..677224a6a23--- /dev/null+++ b/packages/web/src/features/platform-admin/lib/audit-log-retention-utils.ts@@ -0,0 +1,108 @@+import { isNil } from '@activepieces/core-utils';+import {+ AUDIT_LOG_RETENTION_BACKLOG_GRACE_DAYS,+ AUDIT_LOG_RETENTION_MIN_DAYS,+} from '@activepieces/shared';++function buildOptions({+ savedDays,+ ceiling,+}: {+ savedDays: number | null;+ ceiling: number | null;+}): (number | null)[] {+ const presets = PRESET_DAYS.filter((days) =>+ isWithinCeiling({ days, ceiling }),+ );+ const custom =+ !isNil(savedDays) &&+ !PRESET_DAYS.includes(savedDays) &&+ savedDays >= AUDIT_LOG_RETENTION_MIN_DAYS &&+ isWithinCeiling({ days: savedDays, ceiling })+ ? [savedDays]+ : [];+ return [...[...presets, ...custom].sort((a, b) => a - b), null];+}++function initialSelection({+ savedDays,+ ceiling,+}: {+ savedDays: number | null;+ ceiling: number | null;+}): number | null {+ return isWithinCeiling({ days: savedDays, ceiling }) ? savedDays : null;+}++function effectiveDays({+ days,+ ceiling,+}: {+ days: number | null;+ ceiling: number | null;+}): number | null {+ if (isNil(days)) {+ return ceiling;+ }+ return isNil(ceiling) ? days : Math.min(days, ceiling);+}++function deletesEvents({+ next,+ current,+}: {+ next: number | null;+ current: number | null;+}): boolean {+ if (isNil(next)) {+ return false;+ }+ return isNil(current) || next < current;+}++function cutoffDate({ days, now }: { days: number; now: Date }): Date {+ return new Date(now.getTime() - days * DAY_MS);+}++function isCleanupPending({+ oldestEventCreated,+ days,+ now,+}: {+ oldestEventCreated: string | null;+ days: number | null;+ now: Date;+}): boolean {+ if (isNil(oldestEventCreated) || isNil(days)) {+ return false;+ }+ return (+ new Date(oldestEventCreated).getTime() <+ cutoffDate({+ days: days + AUDIT_LOG_RETENTION_BACKLOG_GRACE_DAYS,+ now,+ }).getTime()+ );+}++function isWithinCeiling({+ days,+ ceiling,+}: {+ days: number | null;+ ceiling: number | null;+}): boolean {+ return !isNil(days) && (isNil(ceiling) || days <= ceiling);+}++export const auditLogRetentionUtils = {+ buildOptions,+ initialSelection,+ effectiveDays,+ deletesEvents,+ cutoffDate,+ isCleanupPending,+};++const PRESET_DAYS = [30, 90, 180, 365];+const DAY_MS = 24 * 60 * 60 * 1000;diff --git a/packages/web/test/features/platform-admin/lib/audit-log-retention-utils.test.ts b/packages/web/test/features/platform-admin/lib/audit-log-retention-utils.test.tsnew file mode 100644index 00000000000..d318b4f3614--- /dev/null+++ b/packages/web/test/features/platform-admin/lib/audit-log-retention-utils.test.ts@@ -0,0 +1,118 @@+import { describe, it, expect } from 'vitest';++import { auditLogRetentionUtils } from '@/features/platform-admin/lib/audit-log-retention-utils';++describe('auditLogRetentionUtils.buildOptions', () => {+ it('offers every preset and the forever option when the instance sets no ceiling', () => {+ expect(+ auditLogRetentionUtils.buildOptions({ savedDays: null, ceiling: null }),+ ).toEqual([30, 90, 180, 365, null]);+ });++ it('hides the presets above the ceiling', () => {+ expect(+ auditLogRetentionUtils.buildOptions({ savedDays: null, ceiling: 100 }),+ ).toEqual([30, 90, null]);+ });++ it('keeps a saved value that is not a preset, in order', () => {+ expect(+ auditLogRetentionUtils.buildOptions({ savedDays: 45, ceiling: null }),+ ).toEqual([30, 45, 90, 180, 365, null]);+ });++ it('drops a saved value that a lowered ceiling now exceeds', () => {+ expect(+ auditLogRetentionUtils.buildOptions({ savedDays: 200, ceiling: 100 }),+ ).toEqual([30, 90, null]);+ });+});++describe('auditLogRetentionUtils.initialSelection', () => {+ it('selects the saved value when it is inside the ceiling', () => {+ expect(+ auditLogRetentionUtils.initialSelection({ savedDays: 90, ceiling: 365 }),+ ).toBe(90);+ });++ it('selects the instance option when a lowered ceiling overrides the saved value', () => {+ expect(+ auditLogRetentionUtils.initialSelection({ savedDays: 365, ceiling: 90 }),+ ).toBeNull();+ });+});++describe('auditLogRetentionUtils.effectiveDays', () => {+ it.each([+ { days: null, ceiling: null, expected: null },+ { days: null, ceiling: 365, expected: 365 },+ { days: 90, ceiling: null, expected: 90 },+ { days: 90, ceiling: 365, expected: 90 },+ { days: 730, ceiling: 365, expected: 365 },+ ])(+ 'resolves $days days under a $ceiling ceiling to $expected',+ ({ days, ceiling, expected }) => {+ expect(auditLogRetentionUtils.effectiveDays({ days, ceiling })).toBe(+ expected,+ );+ },+ );+});++describe('auditLogRetentionUtils.deletesEvents', () => {+ it.each([+ { next: 30, current: null, expected: true },+ { next: 30, current: 90, expected: true },+ { next: 90, current: 30, expected: false },+ { next: 90, current: 90, expected: false },+ { next: null, current: 30, expected: false },+ ])(+ 'warns $expected when going from $current to $next days',+ ({ next, current, expected }) => {+ expect(auditLogRetentionUtils.deletesEvents({ next, current })).toBe(+ expected,+ );+ },+ );+});++describe('auditLogRetentionUtils.cutoffDate', () => {+ it('returns the moment that is the retention period before now', () => {+ expect(+ auditLogRetentionUtils+ .cutoffDate({ days: 30, now: new Date('2026-10-31T12:00:00.000Z') })+ .toISOString(),+ ).toBe('2026-10-01T12:00:00.000Z');+ });++ it('counts whole 24-hour days across a daylight-saving change, like the database in UTC', () => {+ expect(+ auditLogRetentionUtils+ .cutoffDate({ days: 30, now: new Date('2026-11-15T00:30:00.000Z') })+ .toISOString(),+ ).toBe('2026-10-16T00:30:00.000Z');+ });+});++describe('auditLogRetentionUtils.isCleanupPending', () => {+ const now = new Date('2026-10-31T12:00:00.000Z');++ it.each([+ { oldestEventCreated: '2026-09-15T12:00:00.000Z', days: 30, expected: true },+ { oldestEventCreated: '2026-09-30T12:00:00.000Z', days: 30, expected: false },+ { oldestEventCreated: '2026-10-20T12:00:00.000Z', days: 30, expected: false },+ { oldestEventCreated: '2020-01-01T00:00:00.000Z', days: null, expected: false },+ { oldestEventCreated: null, days: 30, expected: false },+ ])(+ 'is $expected for an oldest event at $oldestEventCreated with $days days',+ ({ oldestEventCreated, days, expected }) => {+ expect(+ auditLogRetentionUtils.isCleanupPending({+ oldestEventCreated,+ days,+ now,+ }),+ ).toBe(expected);+ },+ );+});