Infisical is the open-source platform for securing secrets, certificates, access, and AI Agents.
TypeScript acmeagentsai-agentscertificate-managementdevsecopsenvironment-variableskubernetesopen-sourcepampkipostgresprivate-caprivileged-access-managementsecret-managementsecret-managersecretssecrets-managementsecuritysecurity-toolsvault
feat: add outbox pattern to be used on the delegated approval system to perform the request creation on the external system
adilsonad55441ac2ac
diff --git a/CLAUDE.md b/CLAUDE.mdindex b7a431abeb..ffff875267 100644--- a/CLAUDE.md+++ b/CLAUDE.md@@ -158,23 +158,28 @@ rules in `assertDelegatedApprovalPolicyConstraints` when another policy type optpolicies may opt in today: `assertPolicyTypeSupportsDelegatedApproval` refuses the rest, and secret changepolicies return it as a 400 rather than dropping the field.-Delivery to the delegated system is asynchronous and lives in `delegated-approval/dispatch/`. The request-bridge creates the `delegated_approval_requests` row inside the request's own transaction and enqueues the-`DelegatedApprovalDispatch` job after commit; the worker moves the row `pending_dispatch` → `waiting_approval` (storing the provider's record id in-`providerRequestId`) or `failed_dispatch` with a `statusReason` a user can read, and BullMQ retries put it back-to `pending_dispatch`. A user resends a `failed_dispatch` request with-`POST /api/v1/access-approvals/requests/:requestId/retry-delegated-dispatch`. The product's bridge checks who may-retry, and `delegatedApprovalService.retryDispatch` checks the request can be retried and only re-enqueues the-fixed-id dispatch job: the worker stays the only writer of the status, so a retry cannot run beside an attempt-BullMQ already has scheduled, and a retry that loses that race leaves the row retryable. A delegated request has-no Infisical approvers, so `POST /api/v1/access-approvals/requests/:requestId/cancel` is the only way to close+Delivery to the delegated system is asynchronous and goes through the event outbox (see `backend/CLAUDE.md`).+`delegatedApprovalService.createRequest` writes the `delegated_approval_requests` row and emits+`delegated-approval.request.dispatch-requested` (`delegated-approval-events.ts`) in the caller's transaction, so+the dispatch is exactly as durable as the request; never enqueue it after commit. The+`delegated-approval-dispatch` consumer (`delegated-approval/dispatch/delegated-approval-dispatch-consumer.ts`)+calls the provider inline and moves the row `pending_dispatch` → `waiting_approval` (storing the provider's+record id in `providerRequestId`) or `failed_dispatch` with a `statusReason` a user can read. A transient+failure returns `Retry`, so the outbox backs off and tries again (up to `MAX_OUTBOX_ATTEMPTS`); the reason text+promises that retry on every attempt because the consumer cannot see the attempt count. A user resends a+`failed_dispatch` request with `POST /api/v1/access-approvals/requests/:requestId/retry-delegated-dispatch`. The+product's bridge checks who may retry, and `delegatedApprovalService.retryDispatch` checks the request can be+retried and emits the same event with no transaction. One flush runs per consumer at a time, so two dispatches+of one request never run together; a second event for a request that was already delivered is marked delivered+without calling the provider. A delegated request has+no Infisical approvers, so `POST /api/v1/access-approvals/requests/:requestId/cancel-delegated-dispatch` is the only way to closeone from Infisical: the requester, or anyone who can edit the policy, sets it `cancelled`, which is also how apolicy blocked by pending requests is freed for editing. Every provider implements `TDelegatedApprovalProvider` against the same`TDelegatedApprovalDispatchInput` and registers in `DELEGATED_APPROVAL_PROVIDER_MAP`(`dispatch/providers/<name>/`); a new policy type adds an arm to `TDelegatedApprovalDispatchDetails` and a-builder in `DELEGATED_APPROVAL_DISPATCH_DETAILS_BUILDER_MAP` rather than branching in a provider. The worker-reads from the primary, never checks the license, and throws bullmq's `UnrecoverableError` for anything a retry-cannot fix. The callback contract (token issuance now, the inbound route and approve / reject later) is its own+builder in `DELEGATED_APPROVAL_DISPATCH_DETAILS_BUILDER_MAP` rather than branching in a provider. The consumer+reads from the primary, never checks the license, and treats bullmq's `UnrecoverableError` as the terminal failure+signal, returning `Failed` for it and `Retry` for anything else. The callback contract (token issuance now, the inbound route and approve / reject later) is its ownmodule, `backend/src/services/delegated-approval-callback/`, so the delegated approval module never grows acallback branch. A request gets exactly one callback token for its whole life: it is random, issued on the firstdispatch attempt, and stored encrypted under the org KMS key in `encryptedCallbackToken` so every retry resendsdiff --git a/backend/CLAUDE.md b/backend/CLAUDE.mdindex 808f6fbc93..1d82e1bd92 100644--- a/backend/CLAUDE.md+++ b/backend/CLAUDE.md@@ -774,7 +774,9 @@ Invariants worth knowing before extending it:an event is exactly as durable as the business write. Enqueueing to Redis after commit loses the eventwhenever the pod dies or Redis is down in that window. It's not alert-specific:`alert-event-consumer.ts` is one consumer on the shared registry, and anything else that needs-at-least-once delivery of a domain event registers next to it.+at-least-once delivery of a domain event registers next to it. The delegated approval dispatch consumer+(`delegated-approval/dispatch/delegated-approval-dispatch-consumer.ts`) is the second: it calls the external+approval system inline from `handle()` and maps the outcome to `Delivered` / `Retry` / `Failed`, with no queue hop.**Naming:** the outbox is an implementation detail to both sides of it. Emitters depend on`TEventEmitter` and pass a `TEventInput`; consumers implement `IEventConsumer`, receive `TEvent[]`diff --git a/backend/src/queue/queue-service.ts b/backend/src/queue/queue-service.tsindex 2c8d320198..829ad846a0 100644--- a/backend/src/queue/queue-service.ts+++ b/backend/src/queue/queue-service.ts@@ -45,7 +45,6 @@ import {TAppConnectionCredentialRotationSendNotificationJobPayload} from "@app/services/app-connection/credential-rotation/app-connection-credential-rotation-types";import { CaType } from "@app/services/certificate-authority/certificate-authority-enums";-import { TDelegatedApprovalDispatchJobPayload } from "@app/services/delegated-approval/dispatch/delegated-approval-dispatch-types";import {ExternalPlatforms,TVaultImportSideEffectsJobPayload@@ -122,7 +121,6 @@ export enum QueueName {PamDiscoveryScan = "pam-discovery-scan",PkiAcmeChallengeValidation = "pki-acme-challenge-validation",PkiDiscoveryScan = "pki-discovery-scan",- DelegatedApprovalDispatch = "delegated-approval-dispatch",AppConnectionCredentialRotation = "app-connection-credential-rotation",AppConnectionCredentialRotationRotate = "app-connection-credential-rotation-rotate",AuditLogClickHouseBatch = "audit-log-clickhouse-batch",@@ -207,7 +205,6 @@ export enum QueueJobs {PkiAcmeChallengeValidation = "pki-acme-challenge-validation",PkiDiscoveryRunScan = "pki-discovery-run-scan",PkiDiscoveryScheduledScan = "pki-discovery-scheduled-scan",- DelegatedApprovalDispatch = "delegated-approval-dispatch-job",AppConnectionCredentialRotationQueueRotations = "app-connection-credential-rotation-queue-rotations",AppConnectionCredentialRotationRotate = "app-connection-credential-rotation-rotate",AppConnectionCredentialRotationSendNotification = "app-connection-credential-rotation-send-notification",@@ -569,10 +566,6 @@ export type TQueueJobTypes = {name: QueueJobs.PkiAcmeChallengeValidation;payload: { challengeId: string };};- [QueueName.DelegatedApprovalDispatch]: {- name: QueueJobs.DelegatedApprovalDispatch;- payload: TDelegatedApprovalDispatchJobPayload;- };[QueueName.PkiDiscoveryScan]:| {name: QueueJobs.PkiDiscoveryRunScan;diff --git a/backend/src/server/routes/index.ts b/backend/src/server/routes/index.tsindex 8e6af14115..f88c63c28a 100644--- a/backend/src/server/routes/index.ts+++ b/backend/src/server/routes/index.ts@@ -357,7 +357,7 @@ import { convertorServiceFactory } from "@app/services/convertor/convertor-serviimport { delegatedApprovalPolicyDALFactory } from "@app/services/delegated-approval/delegated-approval-policy-dal";import { delegatedApprovalRequestDALFactory } from "@app/services/delegated-approval/delegated-approval-request-dal";import { delegatedApprovalServiceFactory } from "@app/services/delegated-approval/delegated-approval-service";-import { delegatedApprovalDispatchQueueFactory } from "@app/services/delegated-approval/dispatch/delegated-approval-dispatch-queue";+import { delegatedApprovalDispatchConsumerFactory } from "@app/services/delegated-approval/dispatch/delegated-approval-dispatch-consumer";import { delegatedApprovalCallbackServiceFactory } from "@app/services/delegated-approval-callback/delegated-approval-callback-service";import { encryptionKeyRotationServiceFactory } from "@app/services/encryption-key-rotation/encryption-key-rotation-service";import { acmeEnrollmentConfigDALFactory } from "@app/services/enrollment-config/acme-enrollment-config-dal";@@ -3244,16 +3244,17 @@ export const registerRoutes = async (kmsService});- const delegatedApprovalDispatchQueue = delegatedApprovalDispatchQueueFactory({- delegatedApprovalRequestDAL,- delegatedApprovalPolicyDAL,- delegatedApprovalCallbackService,- approvalRequestDAL,- projectDAL,- appConnectionDAL,- kmsService,- queueService- });+ eventOutboxRegistry.register(+ delegatedApprovalDispatchConsumerFactory({+ delegatedApprovalRequestDAL,+ delegatedApprovalPolicyDAL,+ delegatedApprovalCallbackService,+ approvalRequestDAL,+ projectDAL,+ appConnectionDAL,+ kmsService+ })+ );const delegatedApprovalService = delegatedApprovalServiceFactory({delegatedApprovalPolicyDAL,@@ -3261,7 +3262,7 @@ export const registerRoutes = async (approvalPolicyDAL,appConnectionDAL,appConnectionService,- delegatedApprovalDispatchQueue+ eventEmitter: eventOutboxService});const secretAccessApprovalGlobalRequestBridgeService = secretAccessApprovalGlobalRequestBridgeServiceFactory({@@ -3279,7 +3280,6 @@ export const registerRoutes = async (secretAccessApprovalGlobalPolicyBridgeDAL,secretAccessApprovalGlobalRequestBridgeDAL,delegatedApprovalService,- delegatedApprovalDispatchQueue,smtpService,notificationService,kmsService,@@ -4620,7 +4620,6 @@ export const registerRoutes = async (healthAlert.init();alertQueue.init();eventOutboxQueue.init();- delegatedApprovalDispatchQueue.init();auditLogStreamOutboxQueue.init();pkiSyncCleanup.init();pkiSyncHealthCheckQueue.init();diff --git a/backend/src/services/delegated-approval/delegated-approval-events.ts b/backend/src/services/delegated-approval/delegated-approval-events.tsnew file mode 100644index 0000000000..be728eea7d--- /dev/null+++ b/backend/src/services/delegated-approval/delegated-approval-events.ts@@ -0,0 +1,29 @@+import { Knex } from "knex";+import { z } from "zod";++import { TEventEmitter } from "@app/services/event-outbox/event-outbox-types";++export const DELEGATED_APPROVAL_DISPATCH_REQUESTED_EVENT = "delegated-approval.request.dispatch-requested";++export const DelegatedApprovalDispatchRequestedPayloadSchema = z.object({+ delegatedRequestId: z.string().uuid()+});++export type TDelegatedApprovalDispatchRequestedPayload = z.infer<+ typeof DelegatedApprovalDispatchRequestedPayloadSchema+>;++// No idempotency key on purpose: a user retry has to insert a fresh row while the delivered one from the+// first attempt is still retained, and the consumer already skips a request that was delivered.+export const emitDelegatedApprovalDispatchRequested = (+ eventEmitter: TEventEmitter,+ { delegatedRequestId }: TDelegatedApprovalDispatchRequestedPayload,+ tx?: Knex+): Promise<void> =>+ eventEmitter.emit(+ {+ eventType: DELEGATED_APPROVAL_DISPATCH_REQUESTED_EVENT,+ payload: { delegatedRequestId }+ },+ tx+ );diff --git a/backend/src/services/delegated-approval/delegated-approval-service.test.ts b/backend/src/services/delegated-approval/delegated-approval-service.test.tsindex 3a6871befc..8f3118b4ab 100644--- a/backend/src/services/delegated-approval/delegated-approval-service.test.ts+++ b/backend/src/services/delegated-approval/delegated-approval-service.test.ts@@ -1,9 +1,10 @@import { describe, expect, test, vi } from "vitest";import { BadRequestError } from "@app/lib/errors";-import { ApprovalRequestStatus } from "@app/services/approval-policy/approval-policy-enums";+import { ApprovalPolicyType, ApprovalRequestStatus } from "@app/services/approval-policy/approval-policy-enums";import { DelegatedApprovalPolicyType, DelegatedApprovalRequestStatus } from "./delegated-approval-enums";+import { DELEGATED_APPROVAL_DISPATCH_REQUESTED_EVENT } from "./delegated-approval-events";import { delegatedApprovalServiceFactory } from "./delegated-approval-service";const failedRequest = { id: "delegated-1", requestId: "req-1", status: DelegatedApprovalRequestStatus.FailedDispatch };@@ -15,72 +16,101 @@ const serviceNowPolicy = {const pendingApprovalRequest = {id: "req-1",policyId: "policy-1",+ type: ApprovalPolicyType.SecretAccess,status: ApprovalRequestStatus.Pending,expiresAt: null};+// emit has to receive the caller's tx untouched; a second connection is how the pool deadlocks.+const TX = { sentinel: true } as never;+const createService = ({delegatedRequest = failedRequest as unknown,delegatedPolicy = serviceNowPolicy as unknown} = {}) => {- const queueDispatch = vi.fn().mockResolvedValue(undefined);+ const emit = vi.fn().mockResolvedValue(undefined);+ const create = vi.fn().mockImplementation(async (row: { requestId: string; status: string }) => ({+ id: "delegated-1",+ ...row+ }));const service = delegatedApprovalServiceFactory({- delegatedApprovalRequestDAL: { findOne: vi.fn().mockResolvedValue(delegatedRequest) },+ delegatedApprovalRequestDAL: { findOne: vi.fn().mockResolvedValue(delegatedRequest), create },delegatedApprovalPolicyDAL: { findOne: vi.fn().mockResolvedValue(delegatedPolicy) },- delegatedApprovalDispatchQueue: { queueDispatch }+ eventEmitter: { emit }} as never);- return { service, queueDispatch };+ return { service, emit, create };+};++const dispatchRequestedEvent = {+ eventType: DELEGATED_APPROVAL_DISPATCH_REQUESTED_EVENT,+ payload: { delegatedRequestId: "delegated-1" }};+describe("delegatedApprovalService.createRequest", () => {+ test("records the dispatch event in the same transaction as the request row", async () => {+ const { service, emit, create } = createService();++ const request = await service.createRequest({ approvalRequest: pendingApprovalRequest as never }, TX);++ expect(request).toMatchObject({ id: "delegated-1", status: DelegatedApprovalRequestStatus.PendingDispatch });+ expect(create).toHaveBeenCalledWith(+ { requestId: "req-1", status: DelegatedApprovalRequestStatus.PendingDispatch },+ TX+ );+ expect(emit).toHaveBeenCalledTimes(1);+ expect(emit).toHaveBeenCalledWith(dispatchRequestedEvent, TX);+ });+});+describe("delegatedApprovalService.retryDispatch", () => {- test("re-enqueues a pending request that failed to reach the delegated system", async () => {- const { service, queueDispatch } = createService();+ test("asks for a new dispatch of a pending request that failed to reach the delegated system", async () => {+ const { service, emit } = createService();await expect(service.retryDispatch({ approvalRequest: pendingApprovalRequest })).resolves.toEqual({delegatedRequest: failedRequest,delegatedPolicy: serviceNowPolicy});- expect(queueDispatch).toHaveBeenCalledWith({ delegatedRequestId: "delegated-1" });+ expect(emit).toHaveBeenCalledWith(dispatchRequestedEvent, undefined);});test("refuses a request without a delegated approval", async () => {- const { service, queueDispatch } = createService({ delegatedRequest: null });+ const { service, emit } = createService({ delegatedRequest: null });await expect(service.retryDispatch({ approvalRequest: pendingApprovalRequest })).rejects.toThrow(new BadRequestError({ message: "Request with ID 'req-1' is not reviewed in an external approval system" }));- expect(queueDispatch).not.toHaveBeenCalled();+ expect(emit).not.toHaveBeenCalled();});test("refuses a request that is no longer pending", async () => {- const { service, queueDispatch } = createService();+ const { service, emit } = createService();await expect(service.retryDispatch({approvalRequest: { ...pendingApprovalRequest, status: ApprovalRequestStatus.Cancelled }})).rejects.toThrow("Only pending requests can be sent again");- expect(queueDispatch).not.toHaveBeenCalled();+ expect(emit).not.toHaveBeenCalled();});test("refuses an expired request", async () => {- const { service, queueDispatch } = createService();+ const { service, emit } = createService();await expect(service.retryDispatch({approvalRequest: { ...pendingApprovalRequest, expiresAt: new Date("2020-01-01T00:00:00.000Z") }})).rejects.toThrow("This request has expired. Create a new request instead.");- expect(queueDispatch).not.toHaveBeenCalled();+ expect(emit).not.toHaveBeenCalled();});test("refuses a request whose policy no longer delegates approval", async () => {- const { service, queueDispatch } = createService({ delegatedPolicy: null });+ const { service, emit } = createService({ delegatedPolicy: null });await expect(service.retryDispatch({ approvalRequest: pendingApprovalRequest })).rejects.toThrow("The policy for this request no longer uses an external approval system. Create a new request instead.");- expect(queueDispatch).not.toHaveBeenCalled();+ expect(emit).not.toHaveBeenCalled();});test.each([@@ -88,11 +118,11 @@ describe("delegatedApprovalService.retryDispatch", () => {DelegatedApprovalRequestStatus.WaitingApproval,DelegatedApprovalRequestStatus.Approved])("refuses a request in the %s state", async (status) => {- const { service, queueDispatch } = createService({ delegatedRequest: { ...failedRequest, status } });+ const { service, emit } = createService({ delegatedRequest: { ...failedRequest, status } });await expect(service.retryDispatch({ approvalRequest: pendingApprovalRequest })).rejects.toThrow("Only requests that failed to reach ServiceNow can be sent again");- expect(queueDispatch).not.toHaveBeenCalled();+ expect(emit).not.toHaveBeenCalled();});});diff --git a/backend/src/services/delegated-approval/delegated-approval-service.ts b/backend/src/services/delegated-approval/delegated-approval-service.tsindex f4aa21d3c5..e2fe2f857a 100644--- a/backend/src/services/delegated-approval/delegated-approval-service.ts+++ b/backend/src/services/delegated-approval/delegated-approval-service.ts@@ -10,8 +10,10 @@ import { APP_CONNECTION_NAME_MAP } from "@app/services/app-connection/app-connecimport { TAppConnectionServiceFactory } from "@app/services/app-connection/app-connection-service";import { TApprovalPolicyDALFactory } from "@app/services/approval-policy/approval-policy-dal";import { ApprovalRequestStatus } from "@app/services/approval-policy/approval-policy-enums";+import { TEventEmitter } from "@app/services/event-outbox/event-outbox-types";import { DelegatedApprovalPolicyType, DelegatedApprovalRequestStatus } from "./delegated-approval-enums";+import { emitDelegatedApprovalDispatchRequested } from "./delegated-approval-events";import { assertPolicyTypeSupportsDelegatedApproval, listDelegatedApprovalOptions } from "./delegated-approval-fns";import { DELEGATED_APPROVAL_CONNECTION_MAP, DELEGATED_APPROVAL_NAME_MAP } from "./delegated-approval-maps";import { TDelegatedApprovalPolicyDALFactory } from "./delegated-approval-policy-dal";@@ -26,7 +28,6 @@ import {TUpdateDelegatedApprovalRequestDTO,TValidateDelegatedApprovalConnectionDTO} from "./delegated-approval-types";-import { TDelegatedApprovalDispatchQueueFactory } from "./dispatch/delegated-approval-dispatch-queue";type TDelegatedApprovalServiceFactoryDep = {delegatedApprovalPolicyDAL: TDelegatedApprovalPolicyDALFactory;@@ -34,7 +35,7 @@ type TDelegatedApprovalServiceFactoryDep = {approvalPolicyDAL: Pick<TApprovalPolicyDALFactory, "findById">;appConnectionDAL: Pick<TAppConnectionDALFactory, "findById">;appConnectionService: Pick<TAppConnectionServiceFactory, "validateAppConnectionUsageById">;- delegatedApprovalDispatchQueue: Pick<TDelegatedApprovalDispatchQueueFactory, "queueDispatch">;+ eventEmitter: TEventEmitter;};export type TDelegatedApprovalServiceFactory = ReturnType<typeof delegatedApprovalServiceFactory>;@@ -45,7 +46,7 @@ export const delegatedApprovalServiceFactory = ({approvalPolicyDAL,appConnectionDAL,appConnectionService,- delegatedApprovalDispatchQueue+ eventEmitter}: TDelegatedApprovalServiceFactoryDep) => {const $assertConnectionMatches = async ({@@ -148,22 +149,16 @@ export const delegatedApprovalServiceFactory = ({return (policy as TDelegatedApprovalPolicy | undefined) ?? null;};- const createRequest = async (- {- approvalRequest,- status = DelegatedApprovalRequestStatus.PendingDispatch,- statusReason- }: TCreateDelegatedApprovalRequestDTO,- tx?: Knex- ) => {+ const createRequest = async ({ approvalRequest }: TCreateDelegatedApprovalRequestDTO, tx?: Knex) => {assertPolicyTypeSupportsDelegatedApproval(approvalRequest.type);try {- const request = await delegatedApprovalRequestDAL.create(- { requestId: approvalRequest.id, status, statusReason },+ const request = (await delegatedApprovalRequestDAL.create(+ { requestId: approvalRequest.id, status: DelegatedApprovalRequestStatus.PendingDispatch },tx- );- return request as TDelegatedApprovalRequest;+ )) as TDelegatedApprovalRequest;+ await emitDelegatedApprovalDispatchRequested(eventEmitter, { delegatedRequestId: request.id }, tx);+ return request;} catch (err) {if (hasPostgresErrorCode(err, DatabaseErrorCode.UniqueViolation)) {throw new BadRequestError({@@ -217,7 +212,7 @@ export const delegatedApprovalServiceFactory = ({});}- await delegatedApprovalDispatchQueue.queueDispatch({ delegatedRequestId: delegatedRequest.id });+ await emitDelegatedApprovalDispatchRequested(eventEmitter, { delegatedRequestId: delegatedRequest.id });return { delegatedRequest, delegatedPolicy };};diff --git a/backend/src/services/delegated-approval/delegated-approval-types.ts b/backend/src/services/delegated-approval/delegated-approval-types.tsindex 74f3bf714e..c051fc5c25 100644--- a/backend/src/services/delegated-approval/delegated-approval-types.ts+++ b/backend/src/services/delegated-approval/delegated-approval-types.ts@@ -40,8 +40,6 @@ export type TUpdateDelegatedApprovalPolicyDTO = {export type TCreateDelegatedApprovalRequestDTO = {approvalRequest: Pick<TApprovalRequests, "id" | "type">;- status?: DelegatedApprovalRequestStatus;- statusReason?: string | null;};export type TRetryDelegatedApprovalRequestDispatchDTO = {diff --git a/backend/src/services/delegated-approval/dispatch/delegated-approval-dispatch-constants.ts b/backend/src/services/delegated-approval/dispatch/delegated-approval-dispatch-constants.tsindex 14a2753080..866dc63a60 100644--- a/backend/src/services/delegated-approval/dispatch/delegated-approval-dispatch-constants.ts+++ b/backend/src/services/delegated-approval/dispatch/delegated-approval-dispatch-constants.ts@@ -1,7 +1 @@-export const DELEGATED_APPROVAL_DISPATCH_ATTEMPTS = 5;--export const DELEGATED_APPROVAL_DISPATCH_BACKOFF_MS = 5_000;--export const DELEGATED_APPROVAL_DISPATCH_CONCURRENCY = 5;-export const DELEGATED_APPROVAL_STATUS_REASON_MAX_CHARS = 1_000;diff --git a/backend/src/services/delegated-approval/dispatch/delegated-approval-dispatch-consumer.test.ts b/backend/src/services/delegated-approval/dispatch/delegated-approval-dispatch-consumer.test.tsnew file mode 100644index 0000000000..a7154f53f7--- /dev/null+++ b/backend/src/services/delegated-approval/dispatch/delegated-approval-dispatch-consumer.test.ts@@ -0,0 +1,323 @@+import { UnrecoverableError } from "bullmq";+import { beforeEach, describe, expect, test, vi } from "vitest";++import { TEventOutbox } from "@app/db/schemas";+import { ApprovalPolicyType, ApprovalRequestStatus } from "@app/services/approval-policy/approval-policy-enums";+import { eventOutboxRegistryFactory } from "@app/services/event-outbox/event-outbox-registry";+import { eventOutboxServiceFactory } from "@app/services/event-outbox/event-outbox-service";+import { EventOutboxStatus, EventResultStatus, TEvent } from "@app/services/event-outbox/event-outbox-types";++import { DelegatedApprovalPolicyType, DelegatedApprovalRequestStatus } from "../delegated-approval-enums";+import { DELEGATED_APPROVAL_DISPATCH_REQUESTED_EVENT } from "../delegated-approval-events";+import { delegatedApprovalDispatchConsumerFactory } from "./delegated-approval-dispatch-consumer";++vi.mock("@app/lib/logger", () => ({+ logger: { info: () => {}, warn: () => {}, error: () => {}, debug: () => {} },+ initLogger: () => {}+}));++vi.mock("@app/lib/telemetry/metrics", () => ({+ recordEventOutboxLagMetric: () => {},+ recordEventOutboxExhaustedMetric: () => {}+}));++const { dispatch } = vi.hoisted(() => ({ dispatch: vi.fn() }));+vi.mock("./delegated-approval-dispatch-fns", () => ({+ getDelegatedApprovalProvider: () => ({ dispatch }),+ getDelegatedApprovalDispatchDetailsBuilder: () => () => ({+ policyType: "secret-access",+ environment: "dev",+ secretPath: "/",+ permissions: ["read"],+ isTemporary: false,+ temporaryRange: null+ })+}));++vi.mock("@app/services/app-connection/app-connection-fns", () => ({+ decryptAppConnection: async (connection: unknown) => connection+}));++const DELEGATED_REQUEST_ID = "11111111-1111-1111-1111-111111111111";+const ORG_ID = "22222222-2222-2222-2222-222222222222";+const RETRY_REASON =+ "ServiceNow did not respond. Infisical will retry automatically. If it keeps failing, send the request again.";++const pendingDelegatedRequest = {+ id: DELEGATED_REQUEST_ID,+ requestId: "req-1",+ status: DelegatedApprovalRequestStatus.PendingDispatch,+ statusReason: null as string | null,+ providerRequestId: null as string | null,+ encryptedCallbackToken: null+};++type TRow = typeof pendingDelegatedRequest;++const approvalRequest = {+ id: "req-1",+ policyId: "policy-1",+ projectId: "project-1",+ organizationId: ORG_ID,+ type: ApprovalPolicyType.SecretAccess,+ status: ApprovalRequestStatus.Pending,+ requesterName: "Ada",+ requesterEmail: "[email protected]",+ justification: null+};++const delegatedPolicy = {+ id: "delegated-policy-1",+ policyId: "policy-1",+ type: DelegatedApprovalPolicyType.ServiceNow,+ connectionId: "connection-1"+};++const PRIMARY = { primary: true } as never;++const buildConsumer = ({+ delegatedRequest = pendingDelegatedRequest as TRow | null,+ approval = approvalRequest as unknown,+ policy = delegatedPolicy as unknown,+ findById = undefined as ((id: string) => Promise<TRow | null>) | undefined+} = {}) => {+ let row: TRow | null = delegatedRequest ? { ...delegatedRequest } : null;++ const update = vi.fn(async (filter: Partial<TRow>, patch: Partial<TRow>) => {+ const current = row;+ if (!current) return [];+ const matches = Object.entries(filter).every(([key, value]) => current[key as keyof TRow] === value);+ if (!matches) return [];+ row = { ...current, ...patch };+ return [row];+ });+ const updateById = vi.fn(async (_id: string, patch: Partial<TRow>) => {+ row = row ? { ...row, ...patch } : row;+ return row;+ });++ const consumer = delegatedApprovalDispatchConsumerFactory({+ delegatedApprovalRequestDAL: {+ findById: findById ?? vi.fn(async () => row),+ primaryNode: () => PRIMARY,+ update,+ updateById+ },+ delegatedApprovalPolicyDAL: { findOne: vi.fn().mockResolvedValue(policy) },+ delegatedApprovalCallbackService: {+ getCallbackUrl: () => "https://infisical.example.com/api/v1/delegated-approvals/callback",+ getOrIssueCallbackToken: vi.fn().mockResolvedValue("callback-token")+ },+ approvalRequestDAL: { findById: vi.fn().mockResolvedValue(approval) },+ projectDAL: { findById: vi.fn().mockResolvedValue({ id: "project-1", name: "Project", type: "secret-manager" }) },+ appConnectionDAL: {+ findById: vi.fn().mockResolvedValue({ id: "connection-1", orgId: ORG_ID, app: "servicenow" })+ },+ kmsService: { createCipherPairWithDataKey: vi.fn() }+ } as never);++ return { consumer, row: () => row, update };+};++const makeEvent = (overrides?: Partial<TEvent>): TEvent => ({+ id: 1,+ eventType: DELEGATED_APPROVAL_DISPATCH_REQUESTED_EVENT,+ payload: { delegatedRequestId: DELEGATED_REQUEST_ID },+ occurredAt: new Date(),+ ...overrides+});++beforeEach(() => {+ dispatch.mockReset();+});++describe("delegatedApprovalDispatchConsumer", () => {+ test("delivers the request and records the provider's record id", async () => {+ dispatch.mockResolvedValueOnce({ providerRequestId: "snow-1" });+ const { consumer, row } = buildConsumer();++ const results = await consumer.handle([makeEvent()]);++ expect(results).toEqual([{ id: "1", status: EventResultStatus.Delivered }]);+ expect(dispatch).toHaveBeenCalledTimes(1);+ expect(dispatch.mock.calls[0][0]).toMatchObject({+ delegatedRequestId: DELEGATED_REQUEST_ID,+ approvalRequestId: "req-1",+ callback: { token: "callback-token" }+ });+ expect(row()).toMatchObject({+ status: DelegatedApprovalRequestStatus.WaitingApproval,+ providerRequestId: "snow-1",+ statusReason: null+ });+ });++ test("asks the outbox to retry a transient provider failure and tells the user it will", async () => {+ dispatch.mockRejectedValueOnce(new Error("ServiceNow did not respond."));+ const { consumer, row } = buildConsumer();++ const results = await consumer.handle([makeEvent()]);++ expect(results).toEqual([{ id: "1", status: EventResultStatus.Retry, error: "ServiceNow did not respond." }]);+ expect(row()).toMatchObject({ status: DelegatedApprovalRequestStatus.FailedDispatch, statusReason: RETRY_REASON });+ });++ test("fails terminally on an unrecoverable error without promising a retry", async () => {+ dispatch.mockRejectedValueOnce(new UnrecoverableError("ServiceNow rejected the credentials."));+ const { consumer, row } = buildConsumer();++ const results = await consumer.handle([makeEvent()]);++ expect(results).toEqual([+ { id: "1", status: EventResultStatus.Failed, error: "ServiceNow rejected the credentials." }+ ]);+ expect(row()).toMatchObject({+ status: DelegatedApprovalRequestStatus.FailedDispatch,+ statusReason: "ServiceNow rejected the credentials."+ });+ });++ test("fails terminally when the approval request is no longer pending", async () => {+ const { consumer, row } = buildConsumer({+ approval: { ...approvalRequest, status: ApprovalRequestStatus.Cancelled }+ });++ const [result] = await consumer.handle([makeEvent()]);++ expect(result.status).toBe(EventResultStatus.Failed);+ expect(dispatch).not.toHaveBeenCalled();+ expect(row()?.status).toBe(DelegatedApprovalRequestStatus.FailedDispatch);+ });++ test.each([+ ["is gone", null],+ ["is no longer awaiting dispatch", { ...pendingDelegatedRequest, status: DelegatedApprovalRequestStatus.Approved }],+ ["was already delivered", { ...pendingDelegatedRequest, providerRequestId: "snow-1" }]+ ])("marks the event delivered without calling the provider when the request %s", async (_, delegatedRequest) => {+ const { consumer } = buildConsumer({ delegatedRequest });++ const results = await consumer.handle([makeEvent()]);++ expect(results).toEqual([{ id: "1", status: EventResultStatus.Delivered }]);+ expect(dispatch).not.toHaveBeenCalled();+ });++ test("fails an unreadable payload terminally", async () => {+ const { consumer } = buildConsumer();++ const [result] = await consumer.handle([makeEvent({ payload: { delegatedRequestId: "not-a-uuid" } })]);++ expect(result.status).toBe(EventResultStatus.Failed);+ expect(result.error).toContain("Unreadable delegated approval dispatch payload");+ expect(dispatch).not.toHaveBeenCalled();+ });+});++// Why this consumer exists: a dispatch that fails is kept by the outbox and tried again, instead of+// depending on an enqueue that ran after the request committed and could be lost with it.+describe("delegatedApprovalDispatchConsumer through the outbox", () => {+ const buildOutbox = (consumer: ReturnType<typeof buildConsumer>["consumer"]) => {+ const registry = eventOutboxRegistryFactory();+ registry.register(consumer);++ const rows: TEventOutbox[] = [];+ const isDue = (row: TEventOutbox) =>+ (row.status === EventOutboxStatus.Pending || row.status === EventOutboxStatus.Retry) &&+ row.nextRetryAt <= new Date();++ const service = eventOutboxServiceFactory({+ eventOutboxDAL: {+ insertEvents: async (inserted: Pick<TEventOutbox, "consumer" | "eventType" | "payload">[]) => {+ inserted.forEach((row) =>+ rows.push({+ id: rows.length + 1,+ status: EventOutboxStatus.Pending,+ attempts: 0,+ nextRetryAt: new Date(0),+ occurredAt: new Date(),+ ...row+ } as TEventOutbox)+ );+ },+ claimBatch: async () => {+ const claimable = rows.filter(isDue);+ claimable.forEach((row) => Object.assign(row, { status: EventOutboxStatus.Processing }));+ return { lockToken: "token", rows: claimable.map((row) => ({ ...row })) };+ },+ extendClaims: async () => {},+ commitResults: async (input: {+ delivered: { ids: string[] }[];+ retriable: { ids: string[]; nextRetryDelayMs: number; error?: string }[];+ failed: { ids: string[]; error?: string }[];+ }) => {+ const apply = (+ groups: { ids: string[]; nextRetryDelayMs?: number; error?: string }[],+ status: EventOutboxStatus+ ) =>+ groups.forEach((group) =>+ group.ids.forEach((id) => {+ const row = rows.find((candidate) => String(candidate.id) === id);+ if (!row) return;+ row.status = status;+ row.attempts += 1;+ row.lastError = group.error ?? null;+ row.nextRetryAt = new Date(Date.now() + (group.nextRetryDelayMs ?? 0));+ })+ );+ apply(input.delivered, EventOutboxStatus.Delivered);+ apply(input.retriable, EventOutboxStatus.Retry);+ apply(input.failed, EventOutboxStatus.Failed);+ return [...input.delivered, ...input.retriable, ...input.failed].reduce(+ (n, group) => n + group.ids.length,+ 0+ );+ }+ } as never,+ eventOutboxRegistry: registry+ });++ const elapseBackoff = () => {+ rows.forEach((row) => Object.assign(row, { nextRetryAt: new Date(0) }));+ };++ return { service, rows, elapseBackoff };+ };++ test("retries a dispatch the provider refused and delivers it on a later flush", async () => {+ dispatch+ .mockRejectedValueOnce(new Error("ServiceNow did not respond."))+ .mockResolvedValueOnce({ providerRequestId: "snow-1" });+ const { consumer, row } = buildConsumer();+ const { service, rows, elapseBackoff } = buildOutbox(consumer);++ await service.emit(+ { eventType: DELEGATED_APPROVAL_DISPATCH_REQUESTED_EVENT, payload: { delegatedRequestId: DELEGATED_REQUEST_ID } },+ PRIMARY+ );+ expect(rows).toHaveLength(1);++ await service.drain({ consumer: consumer.name });++ expect(rows[0]).toMatchObject({+ status: EventOutboxStatus.Retry,+ attempts: 1,+ lastError: "ServiceNow did not respond."+ });+ expect(row()).toMatchObject({ status: DelegatedApprovalRequestStatus.FailedDispatch, statusReason: RETRY_REASON });++ await service.drain({ consumer: consumer.name });+ expect(rows[0].status).toBe(EventOutboxStatus.Retry);+ expect(dispatch).toHaveBeenCalledTimes(1);++ elapseBackoff();+ await service.drain({ consumer: consumer.name });++ expect(rows[0]).toMatchObject({ status: EventOutboxStatus.Delivered, attempts: 2 });+ expect(row()).toMatchObject({+ status: DelegatedApprovalRequestStatus.WaitingApproval,+ providerRequestId: "snow-1",+ statusReason: null+ });+ expect(dispatch).toHaveBeenCalledTimes(2);+ });+});diff --git a/backend/src/services/delegated-approval/dispatch/delegated-approval-dispatch-consumer.ts b/backend/src/services/delegated-approval/dispatch/delegated-approval-dispatch-consumer.tsnew file mode 100644index 0000000000..f556c4ff79--- /dev/null+++ b/backend/src/services/delegated-approval/dispatch/delegated-approval-dispatch-consumer.ts@@ -0,0 +1,279 @@+import { UnrecoverableError } from "bullmq";++import { ProjectType } from "@app/db/schemas";+import { logger } from "@app/lib/logger";+import { TAppConnectionDALFactory } from "@app/services/app-connection/app-connection-dal";+import { decryptAppConnection } from "@app/services/app-connection/app-connection-fns";+import { APP_CONNECTION_NAME_MAP } from "@app/services/app-connection/app-connection-maps";+import { ApprovalRequestStatus } from "@app/services/approval-policy/approval-policy-enums";+import { TApprovalRequestDALFactory } from "@app/services/approval-policy/approval-request-dal";+import { TDelegatedApprovalCallbackServiceFactory } from "@app/services/delegated-approval-callback/delegated-approval-callback-service";+import {+ EventResultStatus,+ IEventConsumer,+ TEvent,+ TEventConsumerResult+} from "@app/services/event-outbox/event-outbox-types";+import { TKmsServiceFactory } from "@app/services/kms/kms-service";+import { TProjectDALFactory } from "@app/services/project/project-dal";++import { DelegatedApprovalPolicyType, DelegatedApprovalRequestStatus } from "../delegated-approval-enums";+import {+ DELEGATED_APPROVAL_DISPATCH_REQUESTED_EVENT,+ DelegatedApprovalDispatchRequestedPayloadSchema,+ TDelegatedApprovalDispatchRequestedPayload+} from "../delegated-approval-events";+import { DELEGATED_APPROVAL_CONNECTION_MAP } from "../delegated-approval-maps";+import { TDelegatedApprovalPolicyDALFactory } from "../delegated-approval-policy-dal";+import { TDelegatedApprovalRequestDALFactory } from "../delegated-approval-request-dal";+import { DELEGATED_APPROVAL_STATUS_REASON_MAX_CHARS } from "./delegated-approval-dispatch-constants";+import {+ getDelegatedApprovalDispatchDetailsBuilder,+ getDelegatedApprovalProvider+} from "./delegated-approval-dispatch-fns";+import { TDelegatedApprovalDispatchInput } from "./delegated-approval-dispatch-types";++export const DELEGATED_APPROVAL_DISPATCH_CONSUMER = "delegated-approval-dispatch";++type TDelegatedApprovalDispatchConsumerDep = {+ delegatedApprovalRequestDAL: Pick<+ TDelegatedApprovalRequestDALFactory,+ "findById" | "primaryNode" | "update" | "updateById"+ >;+ delegatedApprovalPolicyDAL: Pick<TDelegatedApprovalPolicyDALFactory, "findOne">;+ delegatedApprovalCallbackService: Pick<+ TDelegatedApprovalCallbackServiceFactory,+ "getCallbackUrl" | "getOrIssueCallbackToken"+ >;+ approvalRequestDAL: Pick<TApprovalRequestDALFactory, "findById">;+ projectDAL: Pick<TProjectDALFactory, "findById">;+ appConnectionDAL: Pick<TAppConnectionDALFactory, "findById">;+ kmsService: Pick<TKmsServiceFactory, "createCipherPairWithDataKey">;+};++export type TDelegatedApprovalDispatchConsumer = ReturnType<typeof delegatedApprovalDispatchConsumerFactory>;++export const delegatedApprovalDispatchConsumerFactory = ({+ delegatedApprovalRequestDAL,+ delegatedApprovalPolicyDAL,+ delegatedApprovalCallbackService,+ approvalRequestDAL,+ projectDAL,+ appConnectionDAL,+ kmsService+}: TDelegatedApprovalDispatchConsumerDep): IEventConsumer<TDelegatedApprovalDispatchRequestedPayload> => {+ const $markDispatchFailed = async (delegatedRequestId: string, statusReason: string) => {+ try {+ await delegatedApprovalRequestDAL.update(+ { id: delegatedRequestId, status: DelegatedApprovalRequestStatus.PendingDispatch },+ {+ status: DelegatedApprovalRequestStatus.FailedDispatch,+ statusReason: statusReason.slice(0, DELEGATED_APPROVAL_STATUS_REASON_MAX_CHARS)+ }+ );+ } catch (error) {+ logger.error(+ error,+ `Failed to record delegated approval dispatch failure [delegatedRequestId=${delegatedRequestId}]`+ );+ }+ };++ // Both rows were committed together with the event, so they are read from the primary.+ const $loadRequests = async (delegatedRequestId: string) => {+ const primary = delegatedApprovalRequestDAL.primaryNode();++ const delegatedRequest = await delegatedApprovalRequestDAL.findById(delegatedRequestId, primary);+ if (!delegatedRequest) return null;++ const approvalRequest = await approvalRequestDAL.findById(delegatedRequest.requestId, primary);+ const delegatedPolicy = approvalRequest?.policyId+ ? await delegatedApprovalPolicyDAL.findOne({ policyId: approvalRequest.policyId }, primary)+ : null;++ return { delegatedRequest, approvalRequest, delegatedPolicy };+ };++ const $dispatch = async (delegatedRequestId: string): Promise<Omit<TEventConsumerResult, "id">> => {+ const requests = await $loadRequests(delegatedRequestId);+ if (!requests) {+ logger.warn(+ `Delegated approval request no longer exists, skipping dispatch [delegatedRequestId=${delegatedRequestId}]`+ );+ return { status: EventResultStatus.Delivered };+ }++ const { delegatedRequest, approvalRequest, delegatedPolicy } = requests;++ if (+ delegatedRequest.status !== DelegatedApprovalRequestStatus.PendingDispatch &&+ delegatedRequest.status !== DelegatedApprovalRequestStatus.FailedDispatch+ ) {+ logger.info(+ `Delegated approval request is not awaiting dispatch, skipping [delegatedRequestId=${delegatedRequestId}] [status=${delegatedRequest.status}]`+ );+ return { status: EventResultStatus.Delivered };+ }++ if (delegatedRequest.providerRequestId) {+ logger.info(+ `Delegated approval request was already delivered, skipping [delegatedRequestId=${delegatedRequestId}] [providerRequestId=${delegatedRequest.providerRequestId}]`+ );+ return { status: EventResultStatus.Delivered };+ }++ if (delegatedRequest.status === DelegatedApprovalRequestStatus.FailedDispatch) {+ await delegatedApprovalRequestDAL.update(+ { id: delegatedRequestId, status: DelegatedApprovalRequestStatus.FailedDispatch },+ { status: DelegatedApprovalRequestStatus.PendingDispatch, statusReason: null }+ );+ }++ try {+ if (!approvalRequest) {+ throw new UnrecoverableError("The approval request no longer exists.");+ }+ if (approvalRequest.status !== ApprovalRequestStatus.Pending) {+ throw new UnrecoverableError(+ "The approval request is no longer pending, so it was not sent to the delegated approval system."+ );+ }+ if (!delegatedPolicy) {+ throw new UnrecoverableError(+ "The policy's delegated approver was removed before the request could be sent. Configure a delegated approver on the policy and create the request again."+ );+ }++ const provider = getDelegatedApprovalProvider(delegatedPolicy.type);+ const buildDetails = getDelegatedApprovalDispatchDetailsBuilder(approvalRequest.type);+ const details = buildDetails(approvalRequest);++ const callbackUrl = delegatedApprovalCallbackService.getCallbackUrl();+ if (!callbackUrl) {+ throw new UnrecoverableError(+ "SITE_URL must be configured so the delegated approval system can call Infisical back."+ );+ }++ const primary = delegatedApprovalRequestDAL.primaryNode();+ const [project, rawConnection] = await Promise.all([+ projectDAL.findById(approvalRequest.projectId, primary),+ appConnectionDAL.findById(delegatedPolicy.connectionId, primary)+ ]);+ if (!project) {+ throw new UnrecoverableError("The project of the approval request no longer exists.");+ }+ if (!rawConnection || rawConnection.orgId !== approvalRequest.organizationId) {+ const connectionApp = DELEGATED_APPROVAL_CONNECTION_MAP[delegatedPolicy.type as DelegatedApprovalPolicyType];+ throw new UnrecoverableError(+ `App Connection with ID '${delegatedPolicy.connectionId}' was not found. Select a ${APP_CONNECTION_NAME_MAP[connectionApp]} Connection on the policy and create the request again.`+ );+ }++ const connection = await decryptAppConnection(rawConnection, kmsService);++ const callbackToken = await delegatedApprovalCallbackService.getOrIssueCallbackToken({+ delegatedRequest,+ orgId: approvalRequest.organizationId+ });+ if (!callbackToken) {+ throw new UnrecoverableError(+ "A callback token could not be issued for this request. Create the request again."+ );+ }++ const input: TDelegatedApprovalDispatchInput = {+ delegatedRequestId,+ approvalRequestId: approvalRequest.id,+ organizationId: approvalRequest.organizationId,+ productType: project.type as ProjectType,+ project: { id: project.id, name: project.name },+ requester: { name: approvalRequest.requesterName, email: approvalRequest.requesterEmail },+ justification: approvalRequest.justification ?? null,+ callback: { url: callbackUrl, token: callbackToken },+ details+ };++ const { providerRequestId } = await provider.dispatch(input, connection);++ const [waitingRequest] = await delegatedApprovalRequestDAL.update(+ { id: delegatedRequestId, status: DelegatedApprovalRequestStatus.PendingDispatch },+ { status: DelegatedApprovalRequestStatus.WaitingApproval, statusReason: null, providerRequestId }+ );+ if (!waitingRequest) {+ await delegatedApprovalRequestDAL.updateById(delegatedRequestId, { providerRequestId });+ logger.warn(+ `Delegated approval request left pending dispatch while it was being delivered [delegatedRequestId=${delegatedRequestId}] [providerRequestId=${providerRequestId}]`+ );+ }++ logger.info(+ `Delegated approval request delivered [delegatedRequestId=${delegatedRequestId}] [approvalRequestId=${approvalRequest.id}] [provider=${delegatedPolicy.type}] [providerRequestId=${providerRequestId}]`+ );+ return { status: EventResultStatus.Delivered };+ } catch (error) {+ const isTerminal = error instanceof UnrecoverableError;+ const message = error instanceof Error ? error.message : String(error);++ // The outbox decides when retries run out and the consumer never sees the attempt count, so the+ // reason has to read true on every attempt. The row is failed_dispatch throughout, which is what+ // makes the manual resend available while the outbox is still retrying.+ await $markDispatchFailed(+ delegatedRequestId,+ isTerminal+ ? message+ : `${message} Infisical will retry automatically. If it keeps failing, send the request again.`+ );++ const logContext = { err: { name: (error as Error)?.name, message } };+ const logMessage = `Delegated approval dispatch failed [delegatedRequestId=${delegatedRequestId}] [terminal=${isTerminal}]`;+ if (isTerminal) {+ logger.error(logContext, logMessage);+ return { status: EventResultStatus.Failed, error: message };+ }+ logger.warn(logContext, logMessage);+ return { status: EventResultStatus.Retry, error: message };+ }+ };++ const $handleEvent = async (event: TEvent): Promise<TEventConsumerResult> => {+ const id = String(event.id);++ const payload = DelegatedApprovalDispatchRequestedPayloadSchema.safeParse(event.payload);+ if (!payload.success) {+ return {+ id,+ status: EventResultStatus.Failed,+ error: `Unreadable delegated approval dispatch payload: ${payload.error.issues.map((issue) => issue.message).join(", ")}`+ };+ }++ return { id, ...(await $dispatch(payload.data.delegatedRequestId)) };+ };++ const handle = async (events: TEvent[]): Promise<TEventConsumerResult[]> => {+ const results: TEventConsumerResult[] = [];++ for (const event of events) {+ try {+ // eslint-disable-next-line no-await-in-loop -- one dispatch at a time, so two events for the same request never race+ results.push(await $handleEvent(event));+ } catch (error) {+ results.push({+ id: String(event.id),+ status: EventResultStatus.Retry,+ error: error instanceof Error ? error.message : "Unknown error"+ });+ }+ }++ return results;+ };++ return {+ name: DELEGATED_APPROVAL_DISPATCH_CONSUMER,+ payloadSchema: DelegatedApprovalDispatchRequestedPayloadSchema,+ subscribesTo: (eventType: string) => eventType === DELEGATED_APPROVAL_DISPATCH_REQUESTED_EVENT,+ handle+ };+};diff --git a/backend/src/services/delegated-approval/dispatch/delegated-approval-dispatch-queue.ts b/backend/src/services/delegated-approval/dispatch/delegated-approval-dispatch-queue.tsdeleted file mode 100644index 4b7800ccce..0000000000--- a/backend/src/services/delegated-approval/dispatch/delegated-approval-dispatch-queue.ts+++ /dev/null@@ -1,264 +0,0 @@-import { UnrecoverableError } from "bullmq";--import { ProjectType } from "@app/db/schemas";-import { logger } from "@app/lib/logger";-import { QueueJobs, QueueName, TQueueServiceFactory } from "@app/queue";-import { TAppConnectionDALFactory } from "@app/services/app-connection/app-connection-dal";-import { decryptAppConnection } from "@app/services/app-connection/app-connection-fns";-import { APP_CONNECTION_NAME_MAP } from "@app/services/app-connection/app-connection-maps";-import { ApprovalRequestStatus } from "@app/services/approval-policy/approval-policy-enums";-import { TApprovalRequestDALFactory } from "@app/services/approval-policy/approval-request-dal";-import { TDelegatedApprovalCallbackServiceFactory } from "@app/services/delegated-approval-callback/delegated-approval-callback-service";-import { TKmsServiceFactory } from "@app/services/kms/kms-service";-import { TProjectDALFactory } from "@app/services/project/project-dal";--import { DelegatedApprovalPolicyType, DelegatedApprovalRequestStatus } from "../delegated-approval-enums";-import { DELEGATED_APPROVAL_CONNECTION_MAP } from "../delegated-approval-maps";-import { TDelegatedApprovalPolicyDALFactory } from "../delegated-approval-policy-dal";-import { TDelegatedApprovalRequestDALFactory } from "../delegated-approval-request-dal";-import {- DELEGATED_APPROVAL_DISPATCH_ATTEMPTS,- DELEGATED_APPROVAL_DISPATCH_BACKOFF_MS,- DELEGATED_APPROVAL_DISPATCH_CONCURRENCY,- DELEGATED_APPROVAL_STATUS_REASON_MAX_CHARS-} from "./delegated-approval-dispatch-constants";-import {- getDelegatedApprovalDispatchDetailsBuilder,- getDelegatedApprovalProvider-} from "./delegated-approval-dispatch-fns";-import {- TDelegatedApprovalDispatchInput,- TDelegatedApprovalDispatchJobPayload-} from "./delegated-approval-dispatch-types";--type TDelegatedApprovalDispatchQueueFactoryDep = {- delegatedApprovalRequestDAL: Pick<- TDelegatedApprovalRequestDALFactory,- "findById" | "primaryNode" | "update" | "updateById"- >;- delegatedApprovalPolicyDAL: Pick<TDelegatedApprovalPolicyDALFactory, "findOne">;- delegatedApprovalCallbackService: Pick<- TDelegatedApprovalCallbackServiceFactory,- "getCallbackUrl" | "getOrIssueCallbackToken"- >;- approvalRequestDAL: Pick<TApprovalRequestDALFactory, "findById">;- projectDAL: Pick<TProjectDALFactory, "findById">;- appConnectionDAL: Pick<TAppConnectionDALFactory, "findById">;- kmsService: Pick<TKmsServiceFactory, "createCipherPairWithDataKey">;- queueService: Pick<TQueueServiceFactory, "queue" | "start">;-};--export type TDelegatedApprovalDispatchQueueFactory = ReturnType<typeof delegatedApprovalDispatchQueueFactory>;--export const delegatedApprovalDispatchQueueFactory = ({- delegatedApprovalRequestDAL,- delegatedApprovalPolicyDAL,- delegatedApprovalCallbackService,- approvalRequestDAL,- projectDAL,- appConnectionDAL,- kmsService,- queueService-}: TDelegatedApprovalDispatchQueueFactoryDep) => {- const $markDispatchFailed = async (delegatedRequestId: string, statusReason: string) => {- try {- await delegatedApprovalRequestDAL.update(- { id: delegatedRequestId, status: DelegatedApprovalRequestStatus.PendingDispatch },- {- status: DelegatedApprovalRequestStatus.FailedDispatch,- statusReason: statusReason.slice(0, DELEGATED_APPROVAL_STATUS_REASON_MAX_CHARS)- }- );- } catch (error) {- logger.error(- error,- `Failed to record delegated approval dispatch failure [delegatedRequestId=${delegatedRequestId}]`- );- }- };-- const queueDispatch = async ({ delegatedRequestId }: TDelegatedApprovalDispatchJobPayload) => {- try {- await queueService.queue(- QueueName.DelegatedApprovalDispatch,- QueueJobs.DelegatedApprovalDispatch,- { delegatedRequestId },- {- jobId: `delegated-approval-dispatch-${delegatedRequestId}`,- attempts: DELEGATED_APPROVAL_DISPATCH_ATTEMPTS,- backoff: { type: "exponential", delay: DELEGATED_APPROVAL_DISPATCH_BACKOFF_MS },- removeOnComplete: true,- removeOnFail: true- }- );- } catch (error) {- logger.error(error, `Failed to queue delegated approval dispatch [delegatedRequestId=${delegatedRequestId}]`);- await $markDispatchFailed(- delegatedRequestId,- "Infisical could not schedule delivery to the delegated approval system."- );- }- };-- // Both rows were committed together moments before the job runs, so they are read from the primary.- const $loadRequests = async (delegatedRequestId: string) => {- const primary = delegatedApprovalRequestDAL.primaryNode();-- const delegatedRequest = await delegatedApprovalRequestDAL.findById(delegatedRequestId, primary);- if (!delegatedRequest) return null;-- const approvalRequest = await approvalRequestDAL.findById(delegatedRequest.requestId, primary);- const delegatedPolicy = approvalRequest?.policyId- ? await delegatedApprovalPolicyDAL.findOne({ policyId: approvalRequest.policyId }, primary)- : null;-- return { delegatedRequest, approvalRequest, delegatedPolicy };- };-- const init = () => {- queueService.start(- QueueName.DelegatedApprovalDispatch,- async (job) => {- const { delegatedRequestId } = job.data;- const attempt = job.attemptsMade + 1;- const maxAttempts = job.opts.attempts ?? 1;- const isFinalAttempt = attempt >= maxAttempts;-- const requests = await $loadRequests(delegatedRequestId);- if (!requests) {- logger.warn(- `Delegated approval request no longer exists, skipping dispatch [delegatedRequestId=${delegatedRequestId}]`- );- return;- }-- const { delegatedRequest, approvalRequest, delegatedPolicy } = requests;-- if (- delegatedRequest.status !== DelegatedApprovalRequestStatus.PendingDispatch &&- delegatedRequest.status !== DelegatedApprovalRequestStatus.FailedDispatch- ) {- logger.info(- `Delegated approval request is not awaiting dispatch, skipping [delegatedRequestId=${delegatedRequestId}] [status=${delegatedRequest.status}]`- );- return;- }-- if (delegatedRequest.providerRequestId) {- logger.info(- `Delegated approval request was already delivered, skipping [delegatedRequestId=${delegatedRequestId}] [providerRequestId=${delegatedRequest.providerRequestId}]`- );- return;- }-- if (delegatedRequest.status === DelegatedApprovalRequestStatus.FailedDispatch) {- await delegatedApprovalRequestDAL.update(- { id: delegatedRequestId, status: DelegatedApprovalRequestStatus.FailedDispatch },- { status: DelegatedApprovalRequestStatus.PendingDispatch, statusReason: null }- );- }-- try {- if (!approvalRequest) {- throw new UnrecoverableError("The approval request no longer exists.");- }- if (approvalRequest.status !== ApprovalRequestStatus.Pending) {- throw new UnrecoverableError(- "The approval request is no longer pending, so it was not sent to the delegated approval system."- );- }- if (!delegatedPolicy) {- throw new UnrecoverableError(- "The policy's delegated approver was removed before the request could be sent. Configure a delegated approver on the policy and create the request again."- );- }-- const provider = getDelegatedApprovalProvider(delegatedPolicy.type);- const details = getDelegatedApprovalDispatchDetailsBuilder(approvalRequest.type)(approvalRequest);-- const callbackUrl = delegatedApprovalCallbackService.getCallbackUrl();- if (!callbackUrl) {- throw new UnrecoverableError(- "SITE_URL must be configured so the delegated approval system can call Infisical back."- );- }-- const primary = delegatedApprovalRequestDAL.primaryNode();- const [project, rawConnection] = await Promise.all([- projectDAL.findById(approvalRequest.projectId, primary),- appConnectionDAL.findById(delegatedPolicy.connectionId, primary)- ]);- if (!project) {- throw new UnrecoverableError("The project of the approval request no longer exists.");- }- if (!rawConnection || rawConnection.orgId !== approvalRequest.organizationId) {- const connectionApp =- DELEGATED_APPROVAL_CONNECTION_MAP[delegatedPolicy.type as DelegatedApprovalPolicyType];- throw new UnrecoverableError(- `App Connection with ID '${delegatedPolicy.connectionId}' was not found. Select a ${APP_CONNECTION_NAME_MAP[connectionApp]} Connection on the policy and create the request again.`- );- }-- const connection = await decryptAppConnection(rawConnection, kmsService);-- const callbackToken = await delegatedApprovalCallbackService.getOrIssueCallbackToken({- delegatedRequest,- orgId: approvalRequest.organizationId- });- if (!callbackToken) {- throw new UnrecoverableError(- "A callback token could not be issued for this request. Create the request again."- );- }-- const input: TDelegatedApprovalDispatchInput = {- delegatedRequestId,- approvalRequestId: approvalRequest.id,- organizationId: approvalRequest.organizationId,- productType: project.type as ProjectType,- project: { id: project.id, name: project.name },- requester: { name: approvalRequest.requesterName, email: approvalRequest.requesterEmail },- justification: approvalRequest.justification ?? null,- callback: { url: callbackUrl, token: callbackToken },- details- };-- const { providerRequestId } = await provider.dispatch(input, connection);-- const [waitingRequest] = await delegatedApprovalRequestDAL.update(- { id: delegatedRequestId, status: DelegatedApprovalRequestStatus.PendingDispatch },- { status: DelegatedApprovalRequestStatus.WaitingApproval, statusReason: null, providerRequestId }- );- if (!waitingRequest) {- await delegatedApprovalRequestDAL.updateById(delegatedRequestId, { providerRequestId });- logger.warn(- `Delegated approval request left pending dispatch while it was being delivered [delegatedRequestId=${delegatedRequestId}] [providerRequestId=${providerRequestId}]`- );- }-- logger.info(- `Delegated approval request delivered [delegatedRequestId=${delegatedRequestId}] [approvalRequestId=${approvalRequest.id}] [provider=${delegatedPolicy.type}] [providerRequestId=${providerRequestId}]`- );- } catch (error) {- const isTerminal = isFinalAttempt || error instanceof UnrecoverableError;- const message = error instanceof Error ? error.message : String(error);-- await $markDispatchFailed(delegatedRequestId, isTerminal ? message : `${message} Infisical will retry.`);-- const logContext = { err: { name: (error as Error)?.name, message } };- const logMessage = `Delegated approval dispatch failed [delegatedRequestId=${delegatedRequestId}] [attempt=${attempt}/${maxAttempts}] [terminal=${isTerminal}]`;- if (isTerminal) {- logger.error(logContext, logMessage);- } else {- logger.warn(logContext, logMessage);- }-- throw error;- }- },- { concurrency: DELEGATED_APPROVAL_DISPATCH_CONCURRENCY }- );- };-- return { queueDispatch, init };-};diff --git a/backend/src/services/delegated-approval/dispatch/delegated-approval-dispatch-types.ts b/backend/src/services/delegated-approval/dispatch/delegated-approval-dispatch-types.tsindex 15861ade66..39e5838c93 100644--- a/backend/src/services/delegated-approval/dispatch/delegated-approval-dispatch-types.ts+++ b/backend/src/services/delegated-approval/dispatch/delegated-approval-dispatch-types.ts@@ -42,5 +42,3 @@ export type TDelegatedApprovalProvider = {export type TDelegatedApprovalDispatchDetailsBuilder = (request: TApprovalRequests) => TDelegatedApprovalDispatchDetails;--export type TDelegatedApprovalDispatchJobPayload = { delegatedRequestId: string };diff --git a/backend/src/services/secret-access-approval-global-request-bridge/secret-access-approval-global-request-bridge-service.ts b/backend/src/services/secret-access-approval-global-request-bridge/secret-access-approval-global-request-bridge-service.tsindex 18dba57ed6..a3891c9fcf 100644--- a/backend/src/services/secret-access-approval-global-request-bridge/secret-access-approval-global-request-bridge-service.ts+++ b/backend/src/services/secret-access-approval-global-request-bridge/secret-access-approval-global-request-bridge-service.ts@@ -78,7 +78,6 @@ export const secretAccessApprovalGlobalRequestBridgeServiceFactory = ({secretAccessApprovalGlobalPolicyBridgeDAL,secretAccessApprovalGlobalRequestBridgeDAL,delegatedApprovalService,- delegatedApprovalDispatchQueue,smtpService,notificationService,kmsService,@@ -332,7 +331,7 @@ export const secretAccessApprovalGlobalRequestBridgeServiceFactory = ({const parsedMs = requestExpirationTime ? ms(requestExpirationTime) : null;const expiresAt = parsedMs && !Number.isNaN(parsedMs) ? new Date(Date.now() + parsedMs) : null;- const { created, delegatedRequest } = await approvalRequestDAL.transaction(async (tx) => {+ const created = await approvalRequestDAL.transaction(async (tx) => {const approvalRequest = await createApprovalRequestWithSteps({projectId,@@ -351,16 +350,12 @@ export const secretAccessApprovalGlobalRequestBridgeServiceFactory = ({{ approvalRequestDAL, approvalRequestStepsDAL, approvalRequestStepEligibleApproversDAL },tx);- return {- created: approvalRequest,- delegatedRequest: delegatedPolicy ? await delegatedApprovalService.createRequest({ approvalRequest }, tx) : null- };+ if (delegatedPolicy) {+ await delegatedApprovalService.createRequest({ approvalRequest }, tx);+ }+ return approvalRequest;});- if (delegatedRequest) {- await delegatedApprovalDispatchQueue.queueDispatch({ delegatedRequestId: delegatedRequest.id });- }-const request = await $findRequestById(created.id);const [firstStep] = policy.steps;if (firstStep) {diff --git a/backend/src/services/secret-access-approval-global-request-bridge/secret-access-approval-global-request-bridge-types.ts b/backend/src/services/secret-access-approval-global-request-bridge/secret-access-approval-global-request-bridge-types.tsindex 1d80eea497..0e84c1b0da 100644--- a/backend/src/services/secret-access-approval-global-request-bridge/secret-access-approval-global-request-bridge-types.ts+++ b/backend/src/services/secret-access-approval-global-request-bridge/secret-access-approval-global-request-bridge-types.ts@@ -18,7 +18,6 @@ import {import { TSecretAccessApprovalGlobalResource } from "@app/services/approval-policy/secret-access/secret-access-policy-factory";import { TSecretAccessPolicy } from "@app/services/approval-policy/secret-access/secret-access-policy-types";import { TDelegatedApprovalServiceFactory } from "@app/services/delegated-approval/delegated-approval-service";-import { TDelegatedApprovalDispatchQueueFactory } from "@app/services/delegated-approval/dispatch/delegated-approval-dispatch-queue";import { TKmsServiceFactory } from "@app/services/kms/kms-service";import { TMicrosoftTeamsServiceFactory } from "@app/services/microsoft-teams/microsoft-teams-service";import { TProjectMicrosoftTeamsConfigDALFactory } from "@app/services/microsoft-teams/project-microsoft-teams-config-dal";@@ -67,7 +66,6 @@ export type TSecretAccessApprovalGlobalRequestBridgeServiceFactoryDep = {TDelegatedApprovalServiceFactory,"getPolicyByPolicyId" | "createRequest" | "getRequestByRequestId" | "getRequestsByRequestIds" | "retryDispatch">;- delegatedApprovalDispatchQueue: Pick<TDelegatedApprovalDispatchQueueFactory, "queueDispatch">;smtpService: Pick<TSmtpService, "sendMail">;notificationService: Pick<TNotificationServiceFactory, "createUserNotifications">;kmsService: Pick<TKmsServiceFactory, "createCipherPairWithDataKey">;