Skip to content

Infisical is the open-source platform for securing secrets, certificates, access, and AI Agents.

feat: add outbox pattern to be used on the delegated approval system to perform the request creation on the external system

adilsonad55441ac2ac
diff --git a/CLAUDE.md b/CLAUDE.md
index b7a431abeb..ffff875267 100644
--- a/CLAUDE.md
+++ b/CLAUDE.md
@@ -158,23 +158,28 @@ rules in `assertDelegatedApprovalPolicyConstraints` when another policy type opt
policies may opt in today: `assertPolicyTypeSupportsDelegatedApproval` refuses the rest, and secret change
policies return it as a 400 rather than dropping the field.
-Delivery to the delegated system is asynchronous and lives in `delegated-approval/dispatch/`. The request
-bridge creates the `delegated_approval_requests` row inside the request's own transaction and enqueues the
-`DelegatedApprovalDispatch` job after commit; the worker moves the row `pending_dispatch` → `waiting_approval` (storing the provider's record id in
-`providerRequestId`) or `failed_dispatch` with a `statusReason` a user can read, and BullMQ retries put it back
-to `pending_dispatch`. A user resends a `failed_dispatch` request with
-`POST /api/v1/access-approvals/requests/:requestId/retry-delegated-dispatch`. The product's bridge checks who may
-retry, and `delegatedApprovalService.retryDispatch` checks the request can be retried and only re-enqueues the
-fixed-id dispatch job: the worker stays the only writer of the status, so a retry cannot run beside an attempt
-BullMQ already has scheduled, and a retry that loses that race leaves the row retryable. A delegated request has
-no Infisical approvers, so `POST /api/v1/access-approvals/requests/:requestId/cancel` is the only way to close
+Delivery to the delegated system is asynchronous and goes through the event outbox (see `backend/CLAUDE.md`).
+`delegatedApprovalService.createRequest` writes the `delegated_approval_requests` row and emits
+`delegated-approval.request.dispatch-requested` (`delegated-approval-events.ts`) in the caller's transaction, so
+the dispatch is exactly as durable as the request; never enqueue it after commit. The
+`delegated-approval-dispatch` consumer (`delegated-approval/dispatch/delegated-approval-dispatch-consumer.ts`)
+calls the provider inline and moves the row `pending_dispatch` → `waiting_approval` (storing the provider's
+record id in `providerRequestId`) or `failed_dispatch` with a `statusReason` a user can read. A transient
+failure returns `Retry`, so the outbox backs off and tries again (up to `MAX_OUTBOX_ATTEMPTS`); the reason text
+promises that retry on every attempt because the consumer cannot see the attempt count. A user resends a
+`failed_dispatch` request with `POST /api/v1/access-approvals/requests/:requestId/retry-delegated-dispatch`. The
+product's bridge checks who may retry, and `delegatedApprovalService.retryDispatch` checks the request can be
+retried and emits the same event with no transaction. One flush runs per consumer at a time, so two dispatches
+of one request never run together; a second event for a request that was already delivered is marked delivered
+without calling the provider. A delegated request has
+no Infisical approvers, so `POST /api/v1/access-approvals/requests/:requestId/cancel-delegated-dispatch` is the only way to close
one from Infisical: the requester, or anyone who can edit the policy, sets it `cancelled`, which is also how a
policy blocked by pending requests is freed for editing. Every provider implements `TDelegatedApprovalProvider` against the same
`TDelegatedApprovalDispatchInput` and registers in `DELEGATED_APPROVAL_PROVIDER_MAP`
(`dispatch/providers/<name>/`); a new policy type adds an arm to `TDelegatedApprovalDispatchDetails` and a
-builder in `DELEGATED_APPROVAL_DISPATCH_DETAILS_BUILDER_MAP` rather than branching in a provider. The worker
-reads from the primary, never checks the license, and throws bullmq's `UnrecoverableError` for anything a retry
-cannot fix. The callback contract (token issuance now, the inbound route and approve / reject later) is its own
+builder in `DELEGATED_APPROVAL_DISPATCH_DETAILS_BUILDER_MAP` rather than branching in a provider. The consumer
+reads from the primary, never checks the license, and treats bullmq's `UnrecoverableError` as the terminal failure
+signal, returning `Failed` for it and `Retry` for anything else. The callback contract (token issuance now, the inbound route and approve / reject later) is its own
module, `backend/src/services/delegated-approval-callback/`, so the delegated approval module never grows a
callback branch. A request gets exactly one callback token for its whole life: it is random, issued on the first
dispatch attempt, and stored encrypted under the org KMS key in `encryptedCallbackToken` so every retry resends
diff --git a/backend/CLAUDE.md b/backend/CLAUDE.md
index 808f6fbc93..1d82e1bd92 100644
--- a/backend/CLAUDE.md
+++ b/backend/CLAUDE.md
@@ -774,7 +774,9 @@ Invariants worth knowing before extending it:
an event is exactly as durable as the business write. Enqueueing to Redis after commit loses the event
whenever the pod dies or Redis is down in that window. It's not alert-specific:
`alert-event-consumer.ts` is one consumer on the shared registry, and anything else that needs
-at-least-once delivery of a domain event registers next to it.
+at-least-once delivery of a domain event registers next to it. The delegated approval dispatch consumer
+(`delegated-approval/dispatch/delegated-approval-dispatch-consumer.ts`) is the second: it calls the external
+approval system inline from `handle()` and maps the outcome to `Delivered` / `Retry` / `Failed`, with no queue hop.
**Naming:** the outbox is an implementation detail to both sides of it. Emitters depend on
`TEventEmitter` and pass a `TEventInput`; consumers implement `IEventConsumer`, receive `TEvent[]`
diff --git a/backend/src/queue/queue-service.ts b/backend/src/queue/queue-service.ts
index 2c8d320198..829ad846a0 100644
--- a/backend/src/queue/queue-service.ts
+++ b/backend/src/queue/queue-service.ts
@@ -45,7 +45,6 @@ import {
TAppConnectionCredentialRotationSendNotificationJobPayload
} from "@app/services/app-connection/credential-rotation/app-connection-credential-rotation-types";
import { CaType } from "@app/services/certificate-authority/certificate-authority-enums";
-import { TDelegatedApprovalDispatchJobPayload } from "@app/services/delegated-approval/dispatch/delegated-approval-dispatch-types";
import {
ExternalPlatforms,
TVaultImportSideEffectsJobPayload
@@ -122,7 +121,6 @@ export enum QueueName {
PamDiscoveryScan = "pam-discovery-scan",
PkiAcmeChallengeValidation = "pki-acme-challenge-validation",
PkiDiscoveryScan = "pki-discovery-scan",
- DelegatedApprovalDispatch = "delegated-approval-dispatch",
AppConnectionCredentialRotation = "app-connection-credential-rotation",
AppConnectionCredentialRotationRotate = "app-connection-credential-rotation-rotate",
AuditLogClickHouseBatch = "audit-log-clickhouse-batch",
@@ -207,7 +205,6 @@ export enum QueueJobs {
PkiAcmeChallengeValidation = "pki-acme-challenge-validation",
PkiDiscoveryRunScan = "pki-discovery-run-scan",
PkiDiscoveryScheduledScan = "pki-discovery-scheduled-scan",
- DelegatedApprovalDispatch = "delegated-approval-dispatch-job",
AppConnectionCredentialRotationQueueRotations = "app-connection-credential-rotation-queue-rotations",
AppConnectionCredentialRotationRotate = "app-connection-credential-rotation-rotate",
AppConnectionCredentialRotationSendNotification = "app-connection-credential-rotation-send-notification",
@@ -569,10 +566,6 @@ export type TQueueJobTypes = {
name: QueueJobs.PkiAcmeChallengeValidation;
payload: { challengeId: string };
};
- [QueueName.DelegatedApprovalDispatch]: {
- name: QueueJobs.DelegatedApprovalDispatch;
- payload: TDelegatedApprovalDispatchJobPayload;
- };
[QueueName.PkiDiscoveryScan]:
| {
name: QueueJobs.PkiDiscoveryRunScan;
diff --git a/backend/src/server/routes/index.ts b/backend/src/server/routes/index.ts
index 8e6af14115..f88c63c28a 100644
--- a/backend/src/server/routes/index.ts
+++ b/backend/src/server/routes/index.ts
@@ -357,7 +357,7 @@ import { convertorServiceFactory } from "@app/services/convertor/convertor-servi
import { delegatedApprovalPolicyDALFactory } from "@app/services/delegated-approval/delegated-approval-policy-dal";
import { delegatedApprovalRequestDALFactory } from "@app/services/delegated-approval/delegated-approval-request-dal";
import { delegatedApprovalServiceFactory } from "@app/services/delegated-approval/delegated-approval-service";
-import { delegatedApprovalDispatchQueueFactory } from "@app/services/delegated-approval/dispatch/delegated-approval-dispatch-queue";
+import { delegatedApprovalDispatchConsumerFactory } from "@app/services/delegated-approval/dispatch/delegated-approval-dispatch-consumer";
import { delegatedApprovalCallbackServiceFactory } from "@app/services/delegated-approval-callback/delegated-approval-callback-service";
import { encryptionKeyRotationServiceFactory } from "@app/services/encryption-key-rotation/encryption-key-rotation-service";
import { acmeEnrollmentConfigDALFactory } from "@app/services/enrollment-config/acme-enrollment-config-dal";
@@ -3244,16 +3244,17 @@ export const registerRoutes = async (
kmsService
});
- const delegatedApprovalDispatchQueue = delegatedApprovalDispatchQueueFactory({
- delegatedApprovalRequestDAL,
- delegatedApprovalPolicyDAL,
- delegatedApprovalCallbackService,
- approvalRequestDAL,
- projectDAL,
- appConnectionDAL,
- kmsService,
- queueService
- });
+ eventOutboxRegistry.register(
+ delegatedApprovalDispatchConsumerFactory({
+ delegatedApprovalRequestDAL,
+ delegatedApprovalPolicyDAL,
+ delegatedApprovalCallbackService,
+ approvalRequestDAL,
+ projectDAL,
+ appConnectionDAL,
+ kmsService
+ })
+ );
const delegatedApprovalService = delegatedApprovalServiceFactory({
delegatedApprovalPolicyDAL,
@@ -3261,7 +3262,7 @@ export const registerRoutes = async (
approvalPolicyDAL,
appConnectionDAL,
appConnectionService,
- delegatedApprovalDispatchQueue
+ eventEmitter: eventOutboxService
});
const secretAccessApprovalGlobalRequestBridgeService = secretAccessApprovalGlobalRequestBridgeServiceFactory({
@@ -3279,7 +3280,6 @@ export const registerRoutes = async (
secretAccessApprovalGlobalPolicyBridgeDAL,
secretAccessApprovalGlobalRequestBridgeDAL,
delegatedApprovalService,
- delegatedApprovalDispatchQueue,
smtpService,
notificationService,
kmsService,
@@ -4620,7 +4620,6 @@ export const registerRoutes = async (
healthAlert.init();
alertQueue.init();
eventOutboxQueue.init();
- delegatedApprovalDispatchQueue.init();
auditLogStreamOutboxQueue.init();
pkiSyncCleanup.init();
pkiSyncHealthCheckQueue.init();
diff --git a/backend/src/services/delegated-approval/delegated-approval-events.ts b/backend/src/services/delegated-approval/delegated-approval-events.ts
new file mode 100644
index 0000000000..be728eea7d
--- /dev/null
+++ b/backend/src/services/delegated-approval/delegated-approval-events.ts
@@ -0,0 +1,29 @@
+import { Knex } from "knex";
+import { z } from "zod";
+
+import { TEventEmitter } from "@app/services/event-outbox/event-outbox-types";
+
+export const DELEGATED_APPROVAL_DISPATCH_REQUESTED_EVENT = "delegated-approval.request.dispatch-requested";
+
+export const DelegatedApprovalDispatchRequestedPayloadSchema = z.object({
+ delegatedRequestId: z.string().uuid()
+});
+
+export type TDelegatedApprovalDispatchRequestedPayload = z.infer<
+ typeof DelegatedApprovalDispatchRequestedPayloadSchema
+>;
+
+// No idempotency key on purpose: a user retry has to insert a fresh row while the delivered one from the
+// first attempt is still retained, and the consumer already skips a request that was delivered.
+export const emitDelegatedApprovalDispatchRequested = (
+ eventEmitter: TEventEmitter,
+ { delegatedRequestId }: TDelegatedApprovalDispatchRequestedPayload,
+ tx?: Knex
+): Promise<void> =>
+ eventEmitter.emit(
+ {
+ eventType: DELEGATED_APPROVAL_DISPATCH_REQUESTED_EVENT,
+ payload: { delegatedRequestId }
+ },
+ tx
+ );
diff --git a/backend/src/services/delegated-approval/delegated-approval-service.test.ts b/backend/src/services/delegated-approval/delegated-approval-service.test.ts
index 3a6871befc..8f3118b4ab 100644
--- a/backend/src/services/delegated-approval/delegated-approval-service.test.ts
+++ b/backend/src/services/delegated-approval/delegated-approval-service.test.ts
@@ -1,9 +1,10 @@
import { describe, expect, test, vi } from "vitest";
import { BadRequestError } from "@app/lib/errors";
-import { ApprovalRequestStatus } from "@app/services/approval-policy/approval-policy-enums";
+import { ApprovalPolicyType, ApprovalRequestStatus } from "@app/services/approval-policy/approval-policy-enums";
import { DelegatedApprovalPolicyType, DelegatedApprovalRequestStatus } from "./delegated-approval-enums";
+import { DELEGATED_APPROVAL_DISPATCH_REQUESTED_EVENT } from "./delegated-approval-events";
import { delegatedApprovalServiceFactory } from "./delegated-approval-service";
const failedRequest = { id: "delegated-1", requestId: "req-1", status: DelegatedApprovalRequestStatus.FailedDispatch };
@@ -15,72 +16,101 @@ const serviceNowPolicy = {
const pendingApprovalRequest = {
id: "req-1",
policyId: "policy-1",
+ type: ApprovalPolicyType.SecretAccess,
status: ApprovalRequestStatus.Pending,
expiresAt: null
};
+// emit has to receive the caller's tx untouched; a second connection is how the pool deadlocks.
+const TX = { sentinel: true } as never;
+
const createService = ({
delegatedRequest = failedRequest as unknown,
delegatedPolicy = serviceNowPolicy as unknown
} = {}) => {
- const queueDispatch = vi.fn().mockResolvedValue(undefined);
+ const emit = vi.fn().mockResolvedValue(undefined);
+ const create = vi.fn().mockImplementation(async (row: { requestId: string; status: string }) => ({
+ id: "delegated-1",
+ ...row
+ }));
const service = delegatedApprovalServiceFactory({
- delegatedApprovalRequestDAL: { findOne: vi.fn().mockResolvedValue(delegatedRequest) },
+ delegatedApprovalRequestDAL: { findOne: vi.fn().mockResolvedValue(delegatedRequest), create },
delegatedApprovalPolicyDAL: { findOne: vi.fn().mockResolvedValue(delegatedPolicy) },
- delegatedApprovalDispatchQueue: { queueDispatch }
+ eventEmitter: { emit }
} as never);
- return { service, queueDispatch };
+ return { service, emit, create };
+};
+
+const dispatchRequestedEvent = {
+ eventType: DELEGATED_APPROVAL_DISPATCH_REQUESTED_EVENT,
+ payload: { delegatedRequestId: "delegated-1" }
};
+describe("delegatedApprovalService.createRequest", () => {
+ test("records the dispatch event in the same transaction as the request row", async () => {
+ const { service, emit, create } = createService();
+
+ const request = await service.createRequest({ approvalRequest: pendingApprovalRequest as never }, TX);
+
+ expect(request).toMatchObject({ id: "delegated-1", status: DelegatedApprovalRequestStatus.PendingDispatch });
+ expect(create).toHaveBeenCalledWith(
+ { requestId: "req-1", status: DelegatedApprovalRequestStatus.PendingDispatch },
+ TX
+ );
+ expect(emit).toHaveBeenCalledTimes(1);
+ expect(emit).toHaveBeenCalledWith(dispatchRequestedEvent, TX);
+ });
+});
+
describe("delegatedApprovalService.retryDispatch", () => {
- test("re-enqueues a pending request that failed to reach the delegated system", async () => {
- const { service, queueDispatch } = createService();
+ test("asks for a new dispatch of a pending request that failed to reach the delegated system", async () => {
+ const { service, emit } = createService();
await expect(service.retryDispatch({ approvalRequest: pendingApprovalRequest })).resolves.toEqual({
delegatedRequest: failedRequest,
delegatedPolicy: serviceNowPolicy
});
- expect(queueDispatch).toHaveBeenCalledWith({ delegatedRequestId: "delegated-1" });
+ expect(emit).toHaveBeenCalledWith(dispatchRequestedEvent, undefined);
});
test("refuses a request without a delegated approval", async () => {
- const { service, queueDispatch } = createService({ delegatedRequest: null });
+ const { service, emit } = createService({ delegatedRequest: null });
await expect(service.retryDispatch({ approvalRequest: pendingApprovalRequest })).rejects.toThrow(
new BadRequestError({ message: "Request with ID 'req-1' is not reviewed in an external approval system" })
);
- expect(queueDispatch).not.toHaveBeenCalled();
+ expect(emit).not.toHaveBeenCalled();
});
test("refuses a request that is no longer pending", async () => {
- const { service, queueDispatch } = createService();
+ const { service, emit } = createService();
await expect(
service.retryDispatch({
approvalRequest: { ...pendingApprovalRequest, status: ApprovalRequestStatus.Cancelled }
})
).rejects.toThrow("Only pending requests can be sent again");
- expect(queueDispatch).not.toHaveBeenCalled();
+ expect(emit).not.toHaveBeenCalled();
});
test("refuses an expired request", async () => {
- const { service, queueDispatch } = createService();
+ const { service, emit } = createService();
await expect(
service.retryDispatch({
approvalRequest: { ...pendingApprovalRequest, expiresAt: new Date("2020-01-01T00:00:00.000Z") }
})
).rejects.toThrow("This request has expired. Create a new request instead.");
- expect(queueDispatch).not.toHaveBeenCalled();
+ expect(emit).not.toHaveBeenCalled();
});
test("refuses a request whose policy no longer delegates approval", async () => {
- const { service, queueDispatch } = createService({ delegatedPolicy: null });
+ const { service, emit } = createService({ delegatedPolicy: null });
await expect(service.retryDispatch({ approvalRequest: pendingApprovalRequest })).rejects.toThrow(
"The policy for this request no longer uses an external approval system. Create a new request instead."
);
- expect(queueDispatch).not.toHaveBeenCalled();
+ expect(emit).not.toHaveBeenCalled();
});
test.each([
@@ -88,11 +118,11 @@ describe("delegatedApprovalService.retryDispatch", () => {
DelegatedApprovalRequestStatus.WaitingApproval,
DelegatedApprovalRequestStatus.Approved
])("refuses a request in the %s state", async (status) => {
- const { service, queueDispatch } = createService({ delegatedRequest: { ...failedRequest, status } });
+ const { service, emit } = createService({ delegatedRequest: { ...failedRequest, status } });
await expect(service.retryDispatch({ approvalRequest: pendingApprovalRequest })).rejects.toThrow(
"Only requests that failed to reach ServiceNow can be sent again"
);
- expect(queueDispatch).not.toHaveBeenCalled();
+ expect(emit).not.toHaveBeenCalled();
});
});
diff --git a/backend/src/services/delegated-approval/delegated-approval-service.ts b/backend/src/services/delegated-approval/delegated-approval-service.ts
index f4aa21d3c5..e2fe2f857a 100644
--- a/backend/src/services/delegated-approval/delegated-approval-service.ts
+++ b/backend/src/services/delegated-approval/delegated-approval-service.ts
@@ -10,8 +10,10 @@ import { APP_CONNECTION_NAME_MAP } from "@app/services/app-connection/app-connec
import { TAppConnectionServiceFactory } from "@app/services/app-connection/app-connection-service";
import { TApprovalPolicyDALFactory } from "@app/services/approval-policy/approval-policy-dal";
import { ApprovalRequestStatus } from "@app/services/approval-policy/approval-policy-enums";
+import { TEventEmitter } from "@app/services/event-outbox/event-outbox-types";
import { DelegatedApprovalPolicyType, DelegatedApprovalRequestStatus } from "./delegated-approval-enums";
+import { emitDelegatedApprovalDispatchRequested } from "./delegated-approval-events";
import { assertPolicyTypeSupportsDelegatedApproval, listDelegatedApprovalOptions } from "./delegated-approval-fns";
import { DELEGATED_APPROVAL_CONNECTION_MAP, DELEGATED_APPROVAL_NAME_MAP } from "./delegated-approval-maps";
import { TDelegatedApprovalPolicyDALFactory } from "./delegated-approval-policy-dal";
@@ -26,7 +28,6 @@ import {
TUpdateDelegatedApprovalRequestDTO,
TValidateDelegatedApprovalConnectionDTO
} from "./delegated-approval-types";
-import { TDelegatedApprovalDispatchQueueFactory } from "./dispatch/delegated-approval-dispatch-queue";
type TDelegatedApprovalServiceFactoryDep = {
delegatedApprovalPolicyDAL: TDelegatedApprovalPolicyDALFactory;
@@ -34,7 +35,7 @@ type TDelegatedApprovalServiceFactoryDep = {
approvalPolicyDAL: Pick<TApprovalPolicyDALFactory, "findById">;
appConnectionDAL: Pick<TAppConnectionDALFactory, "findById">;
appConnectionService: Pick<TAppConnectionServiceFactory, "validateAppConnectionUsageById">;
- delegatedApprovalDispatchQueue: Pick<TDelegatedApprovalDispatchQueueFactory, "queueDispatch">;
+ eventEmitter: TEventEmitter;
};
export type TDelegatedApprovalServiceFactory = ReturnType<typeof delegatedApprovalServiceFactory>;
@@ -45,7 +46,7 @@ export const delegatedApprovalServiceFactory = ({
approvalPolicyDAL,
appConnectionDAL,
appConnectionService,
- delegatedApprovalDispatchQueue
+ eventEmitter
}: TDelegatedApprovalServiceFactoryDep) => {
const $assertConnectionMatches = async (
{
@@ -148,22 +149,16 @@ export const delegatedApprovalServiceFactory = ({
return (policy as TDelegatedApprovalPolicy | undefined) ?? null;
};
- const createRequest = async (
- {
- approvalRequest,
- status = DelegatedApprovalRequestStatus.PendingDispatch,
- statusReason
- }: TCreateDelegatedApprovalRequestDTO,
- tx?: Knex
- ) => {
+ const createRequest = async ({ approvalRequest }: TCreateDelegatedApprovalRequestDTO, tx?: Knex) => {
assertPolicyTypeSupportsDelegatedApproval(approvalRequest.type);
try {
- const request = await delegatedApprovalRequestDAL.create(
- { requestId: approvalRequest.id, status, statusReason },
+ const request = (await delegatedApprovalRequestDAL.create(
+ { requestId: approvalRequest.id, status: DelegatedApprovalRequestStatus.PendingDispatch },
tx
- );
- return request as TDelegatedApprovalRequest;
+ )) as TDelegatedApprovalRequest;
+ await emitDelegatedApprovalDispatchRequested(eventEmitter, { delegatedRequestId: request.id }, tx);
+ return request;
} catch (err) {
if (hasPostgresErrorCode(err, DatabaseErrorCode.UniqueViolation)) {
throw new BadRequestError({
@@ -217,7 +212,7 @@ export const delegatedApprovalServiceFactory = ({
});
}
- await delegatedApprovalDispatchQueue.queueDispatch({ delegatedRequestId: delegatedRequest.id });
+ await emitDelegatedApprovalDispatchRequested(eventEmitter, { delegatedRequestId: delegatedRequest.id });
return { delegatedRequest, delegatedPolicy };
};
diff --git a/backend/src/services/delegated-approval/delegated-approval-types.ts b/backend/src/services/delegated-approval/delegated-approval-types.ts
index 74f3bf714e..c051fc5c25 100644
--- a/backend/src/services/delegated-approval/delegated-approval-types.ts
+++ b/backend/src/services/delegated-approval/delegated-approval-types.ts
@@ -40,8 +40,6 @@ export type TUpdateDelegatedApprovalPolicyDTO = {
export type TCreateDelegatedApprovalRequestDTO = {
approvalRequest: Pick<TApprovalRequests, "id" | "type">;
- status?: DelegatedApprovalRequestStatus;
- statusReason?: string | null;
};
export type TRetryDelegatedApprovalRequestDispatchDTO = {
diff --git a/backend/src/services/delegated-approval/dispatch/delegated-approval-dispatch-constants.ts b/backend/src/services/delegated-approval/dispatch/delegated-approval-dispatch-constants.ts
index 14a2753080..866dc63a60 100644
--- a/backend/src/services/delegated-approval/dispatch/delegated-approval-dispatch-constants.ts
+++ b/backend/src/services/delegated-approval/dispatch/delegated-approval-dispatch-constants.ts
@@ -1,7 +1 @@
-export const DELEGATED_APPROVAL_DISPATCH_ATTEMPTS = 5;
-
-export const DELEGATED_APPROVAL_DISPATCH_BACKOFF_MS = 5_000;
-
-export const DELEGATED_APPROVAL_DISPATCH_CONCURRENCY = 5;
-
export const DELEGATED_APPROVAL_STATUS_REASON_MAX_CHARS = 1_000;
diff --git a/backend/src/services/delegated-approval/dispatch/delegated-approval-dispatch-consumer.test.ts b/backend/src/services/delegated-approval/dispatch/delegated-approval-dispatch-consumer.test.ts
new file mode 100644
index 0000000000..a7154f53f7
--- /dev/null
+++ b/backend/src/services/delegated-approval/dispatch/delegated-approval-dispatch-consumer.test.ts
@@ -0,0 +1,323 @@
+import { UnrecoverableError } from "bullmq";
+import { beforeEach, describe, expect, test, vi } from "vitest";
+
+import { TEventOutbox } from "@app/db/schemas";
+import { ApprovalPolicyType, ApprovalRequestStatus } from "@app/services/approval-policy/approval-policy-enums";
+import { eventOutboxRegistryFactory } from "@app/services/event-outbox/event-outbox-registry";
+import { eventOutboxServiceFactory } from "@app/services/event-outbox/event-outbox-service";
+import { EventOutboxStatus, EventResultStatus, TEvent } from "@app/services/event-outbox/event-outbox-types";
+
+import { DelegatedApprovalPolicyType, DelegatedApprovalRequestStatus } from "../delegated-approval-enums";
+import { DELEGATED_APPROVAL_DISPATCH_REQUESTED_EVENT } from "../delegated-approval-events";
+import { delegatedApprovalDispatchConsumerFactory } from "./delegated-approval-dispatch-consumer";
+
+vi.mock("@app/lib/logger", () => ({
+ logger: { info: () => {}, warn: () => {}, error: () => {}, debug: () => {} },
+ initLogger: () => {}
+}));
+
+vi.mock("@app/lib/telemetry/metrics", () => ({
+ recordEventOutboxLagMetric: () => {},
+ recordEventOutboxExhaustedMetric: () => {}
+}));
+
+const { dispatch } = vi.hoisted(() => ({ dispatch: vi.fn() }));
+vi.mock("./delegated-approval-dispatch-fns", () => ({
+ getDelegatedApprovalProvider: () => ({ dispatch }),
+ getDelegatedApprovalDispatchDetailsBuilder: () => () => ({
+ policyType: "secret-access",
+ environment: "dev",
+ secretPath: "/",
+ permissions: ["read"],
+ isTemporary: false,
+ temporaryRange: null
+ })
+}));
+
+vi.mock("@app/services/app-connection/app-connection-fns", () => ({
+ decryptAppConnection: async (connection: unknown) => connection
+}));
+
+const DELEGATED_REQUEST_ID = "11111111-1111-1111-1111-111111111111";
+const ORG_ID = "22222222-2222-2222-2222-222222222222";
+const RETRY_REASON =
+ "ServiceNow did not respond. Infisical will retry automatically. If it keeps failing, send the request again.";
+
+const pendingDelegatedRequest = {
+ id: DELEGATED_REQUEST_ID,
+ requestId: "req-1",
+ status: DelegatedApprovalRequestStatus.PendingDispatch,
+ statusReason: null as string | null,
+ providerRequestId: null as string | null,
+ encryptedCallbackToken: null
+};
+
+type TRow = typeof pendingDelegatedRequest;
+
+const approvalRequest = {
+ id: "req-1",
+ policyId: "policy-1",
+ projectId: "project-1",
+ organizationId: ORG_ID,
+ type: ApprovalPolicyType.SecretAccess,
+ status: ApprovalRequestStatus.Pending,
+ requesterName: "Ada",
+ requesterEmail: "[email protected]",
+ justification: null
+};
+
+const delegatedPolicy = {
+ id: "delegated-policy-1",
+ policyId: "policy-1",
+ type: DelegatedApprovalPolicyType.ServiceNow,
+ connectionId: "connection-1"
+};
+
+const PRIMARY = { primary: true } as never;
+
+const buildConsumer = ({
+ delegatedRequest = pendingDelegatedRequest as TRow | null,
+ approval = approvalRequest as unknown,
+ policy = delegatedPolicy as unknown,
+ findById = undefined as ((id: string) => Promise<TRow | null>) | undefined
+} = {}) => {
+ let row: TRow | null = delegatedRequest ? { ...delegatedRequest } : null;
+
+ const update = vi.fn(async (filter: Partial<TRow>, patch: Partial<TRow>) => {
+ const current = row;
+ if (!current) return [];
+ const matches = Object.entries(filter).every(([key, value]) => current[key as keyof TRow] === value);
+ if (!matches) return [];
+ row = { ...current, ...patch };
+ return [row];
+ });
+ const updateById = vi.fn(async (_id: string, patch: Partial<TRow>) => {
+ row = row ? { ...row, ...patch } : row;
+ return row;
+ });
+
+ const consumer = delegatedApprovalDispatchConsumerFactory({
+ delegatedApprovalRequestDAL: {
+ findById: findById ?? vi.fn(async () => row),
+ primaryNode: () => PRIMARY,
+ update,
+ updateById
+ },
+ delegatedApprovalPolicyDAL: { findOne: vi.fn().mockResolvedValue(policy) },
+ delegatedApprovalCallbackService: {
+ getCallbackUrl: () => "https://infisical.example.com/api/v1/delegated-approvals/callback",
+ getOrIssueCallbackToken: vi.fn().mockResolvedValue("callback-token")
+ },
+ approvalRequestDAL: { findById: vi.fn().mockResolvedValue(approval) },
+ projectDAL: { findById: vi.fn().mockResolvedValue({ id: "project-1", name: "Project", type: "secret-manager" }) },
+ appConnectionDAL: {
+ findById: vi.fn().mockResolvedValue({ id: "connection-1", orgId: ORG_ID, app: "servicenow" })
+ },
+ kmsService: { createCipherPairWithDataKey: vi.fn() }
+ } as never);
+
+ return { consumer, row: () => row, update };
+};
+
+const makeEvent = (overrides?: Partial<TEvent>): TEvent => ({
+ id: 1,
+ eventType: DELEGATED_APPROVAL_DISPATCH_REQUESTED_EVENT,
+ payload: { delegatedRequestId: DELEGATED_REQUEST_ID },
+ occurredAt: new Date(),
+ ...overrides
+});
+
+beforeEach(() => {
+ dispatch.mockReset();
+});
+
+describe("delegatedApprovalDispatchConsumer", () => {
+ test("delivers the request and records the provider's record id", async () => {
+ dispatch.mockResolvedValueOnce({ providerRequestId: "snow-1" });
+ const { consumer, row } = buildConsumer();
+
+ const results = await consumer.handle([makeEvent()]);
+
+ expect(results).toEqual([{ id: "1", status: EventResultStatus.Delivered }]);
+ expect(dispatch).toHaveBeenCalledTimes(1);
+ expect(dispatch.mock.calls[0][0]).toMatchObject({
+ delegatedRequestId: DELEGATED_REQUEST_ID,
+ approvalRequestId: "req-1",
+ callback: { token: "callback-token" }
+ });
+ expect(row()).toMatchObject({
+ status: DelegatedApprovalRequestStatus.WaitingApproval,
+ providerRequestId: "snow-1",
+ statusReason: null
+ });
+ });
+
+ test("asks the outbox to retry a transient provider failure and tells the user it will", async () => {
+ dispatch.mockRejectedValueOnce(new Error("ServiceNow did not respond."));
+ const { consumer, row } = buildConsumer();
+
+ const results = await consumer.handle([makeEvent()]);
+
+ expect(results).toEqual([{ id: "1", status: EventResultStatus.Retry, error: "ServiceNow did not respond." }]);
+ expect(row()).toMatchObject({ status: DelegatedApprovalRequestStatus.FailedDispatch, statusReason: RETRY_REASON });
+ });
+
+ test("fails terminally on an unrecoverable error without promising a retry", async () => {
+ dispatch.mockRejectedValueOnce(new UnrecoverableError("ServiceNow rejected the credentials."));
+ const { consumer, row } = buildConsumer();
+
+ const results = await consumer.handle([makeEvent()]);
+
+ expect(results).toEqual([
+ { id: "1", status: EventResultStatus.Failed, error: "ServiceNow rejected the credentials." }
+ ]);
+ expect(row()).toMatchObject({
+ status: DelegatedApprovalRequestStatus.FailedDispatch,
+ statusReason: "ServiceNow rejected the credentials."
+ });
+ });
+
+ test("fails terminally when the approval request is no longer pending", async () => {
+ const { consumer, row } = buildConsumer({
+ approval: { ...approvalRequest, status: ApprovalRequestStatus.Cancelled }
+ });
+
+ const [result] = await consumer.handle([makeEvent()]);
+
+ expect(result.status).toBe(EventResultStatus.Failed);
+ expect(dispatch).not.toHaveBeenCalled();
+ expect(row()?.status).toBe(DelegatedApprovalRequestStatus.FailedDispatch);
+ });
+
+ test.each([
+ ["is gone", null],
+ ["is no longer awaiting dispatch", { ...pendingDelegatedRequest, status: DelegatedApprovalRequestStatus.Approved }],
+ ["was already delivered", { ...pendingDelegatedRequest, providerRequestId: "snow-1" }]
+ ])("marks the event delivered without calling the provider when the request %s", async (_, delegatedRequest) => {
+ const { consumer } = buildConsumer({ delegatedRequest });
+
+ const results = await consumer.handle([makeEvent()]);
+
+ expect(results).toEqual([{ id: "1", status: EventResultStatus.Delivered }]);
+ expect(dispatch).not.toHaveBeenCalled();
+ });
+
+ test("fails an unreadable payload terminally", async () => {
+ const { consumer } = buildConsumer();
+
+ const [result] = await consumer.handle([makeEvent({ payload: { delegatedRequestId: "not-a-uuid" } })]);
+
+ expect(result.status).toBe(EventResultStatus.Failed);
+ expect(result.error).toContain("Unreadable delegated approval dispatch payload");
+ expect(dispatch).not.toHaveBeenCalled();
+ });
+});
+
+// Why this consumer exists: a dispatch that fails is kept by the outbox and tried again, instead of
+// depending on an enqueue that ran after the request committed and could be lost with it.
+describe("delegatedApprovalDispatchConsumer through the outbox", () => {
+ const buildOutbox = (consumer: ReturnType<typeof buildConsumer>["consumer"]) => {
+ const registry = eventOutboxRegistryFactory();
+ registry.register(consumer);
+
+ const rows: TEventOutbox[] = [];
+ const isDue = (row: TEventOutbox) =>
+ (row.status === EventOutboxStatus.Pending || row.status === EventOutboxStatus.Retry) &&
+ row.nextRetryAt <= new Date();
+
+ const service = eventOutboxServiceFactory({
+ eventOutboxDAL: {
+ insertEvents: async (inserted: Pick<TEventOutbox, "consumer" | "eventType" | "payload">[]) => {
+ inserted.forEach((row) =>
+ rows.push({
+ id: rows.length + 1,
+ status: EventOutboxStatus.Pending,
+ attempts: 0,
+ nextRetryAt: new Date(0),
+ occurredAt: new Date(),
+ ...row
+ } as TEventOutbox)
+ );
+ },
+ claimBatch: async () => {
+ const claimable = rows.filter(isDue);
+ claimable.forEach((row) => Object.assign(row, { status: EventOutboxStatus.Processing }));
+ return { lockToken: "token", rows: claimable.map((row) => ({ ...row })) };
+ },
+ extendClaims: async () => {},
+ commitResults: async (input: {
+ delivered: { ids: string[] }[];
+ retriable: { ids: string[]; nextRetryDelayMs: number; error?: string }[];
+ failed: { ids: string[]; error?: string }[];
+ }) => {
+ const apply = (
+ groups: { ids: string[]; nextRetryDelayMs?: number; error?: string }[],
+ status: EventOutboxStatus
+ ) =>
+ groups.forEach((group) =>
+ group.ids.forEach((id) => {
+ const row = rows.find((candidate) => String(candidate.id) === id);
+ if (!row) return;
+ row.status = status;
+ row.attempts += 1;
+ row.lastError = group.error ?? null;
+ row.nextRetryAt = new Date(Date.now() + (group.nextRetryDelayMs ?? 0));
+ })
+ );
+ apply(input.delivered, EventOutboxStatus.Delivered);
+ apply(input.retriable, EventOutboxStatus.Retry);
+ apply(input.failed, EventOutboxStatus.Failed);
+ return [...input.delivered, ...input.retriable, ...input.failed].reduce(
+ (n, group) => n + group.ids.length,
+ 0
+ );
+ }
+ } as never,
+ eventOutboxRegistry: registry
+ });
+
+ const elapseBackoff = () => {
+ rows.forEach((row) => Object.assign(row, { nextRetryAt: new Date(0) }));
+ };
+
+ return { service, rows, elapseBackoff };
+ };
+
+ test("retries a dispatch the provider refused and delivers it on a later flush", async () => {
+ dispatch
+ .mockRejectedValueOnce(new Error("ServiceNow did not respond."))
+ .mockResolvedValueOnce({ providerRequestId: "snow-1" });
+ const { consumer, row } = buildConsumer();
+ const { service, rows, elapseBackoff } = buildOutbox(consumer);
+
+ await service.emit(
+ { eventType: DELEGATED_APPROVAL_DISPATCH_REQUESTED_EVENT, payload: { delegatedRequestId: DELEGATED_REQUEST_ID } },
+ PRIMARY
+ );
+ expect(rows).toHaveLength(1);
+
+ await service.drain({ consumer: consumer.name });
+
+ expect(rows[0]).toMatchObject({
+ status: EventOutboxStatus.Retry,
+ attempts: 1,
+ lastError: "ServiceNow did not respond."
+ });
+ expect(row()).toMatchObject({ status: DelegatedApprovalRequestStatus.FailedDispatch, statusReason: RETRY_REASON });
+
+ await service.drain({ consumer: consumer.name });
+ expect(rows[0].status).toBe(EventOutboxStatus.Retry);
+ expect(dispatch).toHaveBeenCalledTimes(1);
+
+ elapseBackoff();
+ await service.drain({ consumer: consumer.name });
+
+ expect(rows[0]).toMatchObject({ status: EventOutboxStatus.Delivered, attempts: 2 });
+ expect(row()).toMatchObject({
+ status: DelegatedApprovalRequestStatus.WaitingApproval,
+ providerRequestId: "snow-1",
+ statusReason: null
+ });
+ expect(dispatch).toHaveBeenCalledTimes(2);
+ });
+});
diff --git a/backend/src/services/delegated-approval/dispatch/delegated-approval-dispatch-consumer.ts b/backend/src/services/delegated-approval/dispatch/delegated-approval-dispatch-consumer.ts
new file mode 100644
index 0000000000..f556c4ff79
--- /dev/null
+++ b/backend/src/services/delegated-approval/dispatch/delegated-approval-dispatch-consumer.ts
@@ -0,0 +1,279 @@
+import { UnrecoverableError } from "bullmq";
+
+import { ProjectType } from "@app/db/schemas";
+import { logger } from "@app/lib/logger";
+import { TAppConnectionDALFactory } from "@app/services/app-connection/app-connection-dal";
+import { decryptAppConnection } from "@app/services/app-connection/app-connection-fns";
+import { APP_CONNECTION_NAME_MAP } from "@app/services/app-connection/app-connection-maps";
+import { ApprovalRequestStatus } from "@app/services/approval-policy/approval-policy-enums";
+import { TApprovalRequestDALFactory } from "@app/services/approval-policy/approval-request-dal";
+import { TDelegatedApprovalCallbackServiceFactory } from "@app/services/delegated-approval-callback/delegated-approval-callback-service";
+import {
+ EventResultStatus,
+ IEventConsumer,
+ TEvent,
+ TEventConsumerResult
+} from "@app/services/event-outbox/event-outbox-types";
+import { TKmsServiceFactory } from "@app/services/kms/kms-service";
+import { TProjectDALFactory } from "@app/services/project/project-dal";
+
+import { DelegatedApprovalPolicyType, DelegatedApprovalRequestStatus } from "../delegated-approval-enums";
+import {
+ DELEGATED_APPROVAL_DISPATCH_REQUESTED_EVENT,
+ DelegatedApprovalDispatchRequestedPayloadSchema,
+ TDelegatedApprovalDispatchRequestedPayload
+} from "../delegated-approval-events";
+import { DELEGATED_APPROVAL_CONNECTION_MAP } from "../delegated-approval-maps";
+import { TDelegatedApprovalPolicyDALFactory } from "../delegated-approval-policy-dal";
+import { TDelegatedApprovalRequestDALFactory } from "../delegated-approval-request-dal";
+import { DELEGATED_APPROVAL_STATUS_REASON_MAX_CHARS } from "./delegated-approval-dispatch-constants";
+import {
+ getDelegatedApprovalDispatchDetailsBuilder,
+ getDelegatedApprovalProvider
+} from "./delegated-approval-dispatch-fns";
+import { TDelegatedApprovalDispatchInput } from "./delegated-approval-dispatch-types";
+
+export const DELEGATED_APPROVAL_DISPATCH_CONSUMER = "delegated-approval-dispatch";
+
+type TDelegatedApprovalDispatchConsumerDep = {
+ delegatedApprovalRequestDAL: Pick<
+ TDelegatedApprovalRequestDALFactory,
+ "findById" | "primaryNode" | "update" | "updateById"
+ >;
+ delegatedApprovalPolicyDAL: Pick<TDelegatedApprovalPolicyDALFactory, "findOne">;
+ delegatedApprovalCallbackService: Pick<
+ TDelegatedApprovalCallbackServiceFactory,
+ "getCallbackUrl" | "getOrIssueCallbackToken"
+ >;
+ approvalRequestDAL: Pick<TApprovalRequestDALFactory, "findById">;
+ projectDAL: Pick<TProjectDALFactory, "findById">;
+ appConnectionDAL: Pick<TAppConnectionDALFactory, "findById">;
+ kmsService: Pick<TKmsServiceFactory, "createCipherPairWithDataKey">;
+};
+
+export type TDelegatedApprovalDispatchConsumer = ReturnType<typeof delegatedApprovalDispatchConsumerFactory>;
+
+export const delegatedApprovalDispatchConsumerFactory = ({
+ delegatedApprovalRequestDAL,
+ delegatedApprovalPolicyDAL,
+ delegatedApprovalCallbackService,
+ approvalRequestDAL,
+ projectDAL,
+ appConnectionDAL,
+ kmsService
+}: TDelegatedApprovalDispatchConsumerDep): IEventConsumer<TDelegatedApprovalDispatchRequestedPayload> => {
+ const $markDispatchFailed = async (delegatedRequestId: string, statusReason: string) => {
+ try {
+ await delegatedApprovalRequestDAL.update(
+ { id: delegatedRequestId, status: DelegatedApprovalRequestStatus.PendingDispatch },
+ {
+ status: DelegatedApprovalRequestStatus.FailedDispatch,
+ statusReason: statusReason.slice(0, DELEGATED_APPROVAL_STATUS_REASON_MAX_CHARS)
+ }
+ );
+ } catch (error) {
+ logger.error(
+ error,
+ `Failed to record delegated approval dispatch failure [delegatedRequestId=${delegatedRequestId}]`
+ );
+ }
+ };
+
+ // Both rows were committed together with the event, so they are read from the primary.
+ const $loadRequests = async (delegatedRequestId: string) => {
+ const primary = delegatedApprovalRequestDAL.primaryNode();
+
+ const delegatedRequest = await delegatedApprovalRequestDAL.findById(delegatedRequestId, primary);
+ if (!delegatedRequest) return null;
+
+ const approvalRequest = await approvalRequestDAL.findById(delegatedRequest.requestId, primary);
+ const delegatedPolicy = approvalRequest?.policyId
+ ? await delegatedApprovalPolicyDAL.findOne({ policyId: approvalRequest.policyId }, primary)
+ : null;
+
+ return { delegatedRequest, approvalRequest, delegatedPolicy };
+ };
+
+ const $dispatch = async (delegatedRequestId: string): Promise<Omit<TEventConsumerResult, "id">> => {
+ const requests = await $loadRequests(delegatedRequestId);
+ if (!requests) {
+ logger.warn(
+ `Delegated approval request no longer exists, skipping dispatch [delegatedRequestId=${delegatedRequestId}]`
+ );
+ return { status: EventResultStatus.Delivered };
+ }
+
+ const { delegatedRequest, approvalRequest, delegatedPolicy } = requests;
+
+ if (
+ delegatedRequest.status !== DelegatedApprovalRequestStatus.PendingDispatch &&
+ delegatedRequest.status !== DelegatedApprovalRequestStatus.FailedDispatch
+ ) {
+ logger.info(
+ `Delegated approval request is not awaiting dispatch, skipping [delegatedRequestId=${delegatedRequestId}] [status=${delegatedRequest.status}]`
+ );
+ return { status: EventResultStatus.Delivered };
+ }
+
+ if (delegatedRequest.providerRequestId) {
+ logger.info(
+ `Delegated approval request was already delivered, skipping [delegatedRequestId=${delegatedRequestId}] [providerRequestId=${delegatedRequest.providerRequestId}]`
+ );
+ return { status: EventResultStatus.Delivered };
+ }
+
+ if (delegatedRequest.status === DelegatedApprovalRequestStatus.FailedDispatch) {
+ await delegatedApprovalRequestDAL.update(
+ { id: delegatedRequestId, status: DelegatedApprovalRequestStatus.FailedDispatch },
+ { status: DelegatedApprovalRequestStatus.PendingDispatch, statusReason: null }
+ );
+ }
+
+ try {
+ if (!approvalRequest) {
+ throw new UnrecoverableError("The approval request no longer exists.");
+ }
+ if (approvalRequest.status !== ApprovalRequestStatus.Pending) {
+ throw new UnrecoverableError(
+ "The approval request is no longer pending, so it was not sent to the delegated approval system."
+ );
+ }
+ if (!delegatedPolicy) {
+ throw new UnrecoverableError(
+ "The policy's delegated approver was removed before the request could be sent. Configure a delegated approver on the policy and create the request again."
+ );
+ }
+
+ const provider = getDelegatedApprovalProvider(delegatedPolicy.type);
+ const buildDetails = getDelegatedApprovalDispatchDetailsBuilder(approvalRequest.type);
+ const details = buildDetails(approvalRequest);
+
+ const callbackUrl = delegatedApprovalCallbackService.getCallbackUrl();
+ if (!callbackUrl) {
+ throw new UnrecoverableError(
+ "SITE_URL must be configured so the delegated approval system can call Infisical back."
+ );
+ }
+
+ const primary = delegatedApprovalRequestDAL.primaryNode();
+ const [project, rawConnection] = await Promise.all([
+ projectDAL.findById(approvalRequest.projectId, primary),
+ appConnectionDAL.findById(delegatedPolicy.connectionId, primary)
+ ]);
+ if (!project) {
+ throw new UnrecoverableError("The project of the approval request no longer exists.");
+ }
+ if (!rawConnection || rawConnection.orgId !== approvalRequest.organizationId) {
+ const connectionApp = DELEGATED_APPROVAL_CONNECTION_MAP[delegatedPolicy.type as DelegatedApprovalPolicyType];
+ throw new UnrecoverableError(
+ `App Connection with ID '${delegatedPolicy.connectionId}' was not found. Select a ${APP_CONNECTION_NAME_MAP[connectionApp]} Connection on the policy and create the request again.`
+ );
+ }
+
+ const connection = await decryptAppConnection(rawConnection, kmsService);
+
+ const callbackToken = await delegatedApprovalCallbackService.getOrIssueCallbackToken({
+ delegatedRequest,
+ orgId: approvalRequest.organizationId
+ });
+ if (!callbackToken) {
+ throw new UnrecoverableError(
+ "A callback token could not be issued for this request. Create the request again."
+ );
+ }
+
+ const input: TDelegatedApprovalDispatchInput = {
+ delegatedRequestId,
+ approvalRequestId: approvalRequest.id,
+ organizationId: approvalRequest.organizationId,
+ productType: project.type as ProjectType,
+ project: { id: project.id, name: project.name },
+ requester: { name: approvalRequest.requesterName, email: approvalRequest.requesterEmail },
+ justification: approvalRequest.justification ?? null,
+ callback: { url: callbackUrl, token: callbackToken },
+ details
+ };
+
+ const { providerRequestId } = await provider.dispatch(input, connection);
+
+ const [waitingRequest] = await delegatedApprovalRequestDAL.update(
+ { id: delegatedRequestId, status: DelegatedApprovalRequestStatus.PendingDispatch },
+ { status: DelegatedApprovalRequestStatus.WaitingApproval, statusReason: null, providerRequestId }
+ );
+ if (!waitingRequest) {
+ await delegatedApprovalRequestDAL.updateById(delegatedRequestId, { providerRequestId });
+ logger.warn(
+ `Delegated approval request left pending dispatch while it was being delivered [delegatedRequestId=${delegatedRequestId}] [providerRequestId=${providerRequestId}]`
+ );
+ }
+
+ logger.info(
+ `Delegated approval request delivered [delegatedRequestId=${delegatedRequestId}] [approvalRequestId=${approvalRequest.id}] [provider=${delegatedPolicy.type}] [providerRequestId=${providerRequestId}]`
+ );
+ return { status: EventResultStatus.Delivered };
+ } catch (error) {
+ const isTerminal = error instanceof UnrecoverableError;
+ const message = error instanceof Error ? error.message : String(error);
+
+ // The outbox decides when retries run out and the consumer never sees the attempt count, so the
+ // reason has to read true on every attempt. The row is failed_dispatch throughout, which is what
+ // makes the manual resend available while the outbox is still retrying.
+ await $markDispatchFailed(
+ delegatedRequestId,
+ isTerminal
+ ? message
+ : `${message} Infisical will retry automatically. If it keeps failing, send the request again.`
+ );
+
+ const logContext = { err: { name: (error as Error)?.name, message } };
+ const logMessage = `Delegated approval dispatch failed [delegatedRequestId=${delegatedRequestId}] [terminal=${isTerminal}]`;
+ if (isTerminal) {
+ logger.error(logContext, logMessage);
+ return { status: EventResultStatus.Failed, error: message };
+ }
+ logger.warn(logContext, logMessage);
+ return { status: EventResultStatus.Retry, error: message };
+ }
+ };
+
+ const $handleEvent = async (event: TEvent): Promise<TEventConsumerResult> => {
+ const id = String(event.id);
+
+ const payload = DelegatedApprovalDispatchRequestedPayloadSchema.safeParse(event.payload);
+ if (!payload.success) {
+ return {
+ id,
+ status: EventResultStatus.Failed,
+ error: `Unreadable delegated approval dispatch payload: ${payload.error.issues.map((issue) => issue.message).join(", ")}`
+ };
+ }
+
+ return { id, ...(await $dispatch(payload.data.delegatedRequestId)) };
+ };
+
+ const handle = async (events: TEvent[]): Promise<TEventConsumerResult[]> => {
+ const results: TEventConsumerResult[] = [];
+
+ for (const event of events) {
+ try {
+ // eslint-disable-next-line no-await-in-loop -- one dispatch at a time, so two events for the same request never race
+ results.push(await $handleEvent(event));
+ } catch (error) {
+ results.push({
+ id: String(event.id),
+ status: EventResultStatus.Retry,
+ error: error instanceof Error ? error.message : "Unknown error"
+ });
+ }
+ }
+
+ return results;
+ };
+
+ return {
+ name: DELEGATED_APPROVAL_DISPATCH_CONSUMER,
+ payloadSchema: DelegatedApprovalDispatchRequestedPayloadSchema,
+ subscribesTo: (eventType: string) => eventType === DELEGATED_APPROVAL_DISPATCH_REQUESTED_EVENT,
+ handle
+ };
+};
diff --git a/backend/src/services/delegated-approval/dispatch/delegated-approval-dispatch-queue.ts b/backend/src/services/delegated-approval/dispatch/delegated-approval-dispatch-queue.ts
deleted file mode 100644
index 4b7800ccce..0000000000
--- a/backend/src/services/delegated-approval/dispatch/delegated-approval-dispatch-queue.ts
+++ /dev/null
@@ -1,264 +0,0 @@
-import { UnrecoverableError } from "bullmq";
-
-import { ProjectType } from "@app/db/schemas";
-import { logger } from "@app/lib/logger";
-import { QueueJobs, QueueName, TQueueServiceFactory } from "@app/queue";
-import { TAppConnectionDALFactory } from "@app/services/app-connection/app-connection-dal";
-import { decryptAppConnection } from "@app/services/app-connection/app-connection-fns";
-import { APP_CONNECTION_NAME_MAP } from "@app/services/app-connection/app-connection-maps";
-import { ApprovalRequestStatus } from "@app/services/approval-policy/approval-policy-enums";
-import { TApprovalRequestDALFactory } from "@app/services/approval-policy/approval-request-dal";
-import { TDelegatedApprovalCallbackServiceFactory } from "@app/services/delegated-approval-callback/delegated-approval-callback-service";
-import { TKmsServiceFactory } from "@app/services/kms/kms-service";
-import { TProjectDALFactory } from "@app/services/project/project-dal";
-
-import { DelegatedApprovalPolicyType, DelegatedApprovalRequestStatus } from "../delegated-approval-enums";
-import { DELEGATED_APPROVAL_CONNECTION_MAP } from "../delegated-approval-maps";
-import { TDelegatedApprovalPolicyDALFactory } from "../delegated-approval-policy-dal";
-import { TDelegatedApprovalRequestDALFactory } from "../delegated-approval-request-dal";
-import {
- DELEGATED_APPROVAL_DISPATCH_ATTEMPTS,
- DELEGATED_APPROVAL_DISPATCH_BACKOFF_MS,
- DELEGATED_APPROVAL_DISPATCH_CONCURRENCY,
- DELEGATED_APPROVAL_STATUS_REASON_MAX_CHARS
-} from "./delegated-approval-dispatch-constants";
-import {
- getDelegatedApprovalDispatchDetailsBuilder,
- getDelegatedApprovalProvider
-} from "./delegated-approval-dispatch-fns";
-import {
- TDelegatedApprovalDispatchInput,
- TDelegatedApprovalDispatchJobPayload
-} from "./delegated-approval-dispatch-types";
-
-type TDelegatedApprovalDispatchQueueFactoryDep = {
- delegatedApprovalRequestDAL: Pick<
- TDelegatedApprovalRequestDALFactory,
- "findById" | "primaryNode" | "update" | "updateById"
- >;
- delegatedApprovalPolicyDAL: Pick<TDelegatedApprovalPolicyDALFactory, "findOne">;
- delegatedApprovalCallbackService: Pick<
- TDelegatedApprovalCallbackServiceFactory,
- "getCallbackUrl" | "getOrIssueCallbackToken"
- >;
- approvalRequestDAL: Pick<TApprovalRequestDALFactory, "findById">;
- projectDAL: Pick<TProjectDALFactory, "findById">;
- appConnectionDAL: Pick<TAppConnectionDALFactory, "findById">;
- kmsService: Pick<TKmsServiceFactory, "createCipherPairWithDataKey">;
- queueService: Pick<TQueueServiceFactory, "queue" | "start">;
-};
-
-export type TDelegatedApprovalDispatchQueueFactory = ReturnType<typeof delegatedApprovalDispatchQueueFactory>;
-
-export const delegatedApprovalDispatchQueueFactory = ({
- delegatedApprovalRequestDAL,
- delegatedApprovalPolicyDAL,
- delegatedApprovalCallbackService,
- approvalRequestDAL,
- projectDAL,
- appConnectionDAL,
- kmsService,
- queueService
-}: TDelegatedApprovalDispatchQueueFactoryDep) => {
- const $markDispatchFailed = async (delegatedRequestId: string, statusReason: string) => {
- try {
- await delegatedApprovalRequestDAL.update(
- { id: delegatedRequestId, status: DelegatedApprovalRequestStatus.PendingDispatch },
- {
- status: DelegatedApprovalRequestStatus.FailedDispatch,
- statusReason: statusReason.slice(0, DELEGATED_APPROVAL_STATUS_REASON_MAX_CHARS)
- }
- );
- } catch (error) {
- logger.error(
- error,
- `Failed to record delegated approval dispatch failure [delegatedRequestId=${delegatedRequestId}]`
- );
- }
- };
-
- const queueDispatch = async ({ delegatedRequestId }: TDelegatedApprovalDispatchJobPayload) => {
- try {
- await queueService.queue(
- QueueName.DelegatedApprovalDispatch,
- QueueJobs.DelegatedApprovalDispatch,
- { delegatedRequestId },
- {
- jobId: `delegated-approval-dispatch-${delegatedRequestId}`,
- attempts: DELEGATED_APPROVAL_DISPATCH_ATTEMPTS,
- backoff: { type: "exponential", delay: DELEGATED_APPROVAL_DISPATCH_BACKOFF_MS },
- removeOnComplete: true,
- removeOnFail: true
- }
- );
- } catch (error) {
- logger.error(error, `Failed to queue delegated approval dispatch [delegatedRequestId=${delegatedRequestId}]`);
- await $markDispatchFailed(
- delegatedRequestId,
- "Infisical could not schedule delivery to the delegated approval system."
- );
- }
- };
-
- // Both rows were committed together moments before the job runs, so they are read from the primary.
- const $loadRequests = async (delegatedRequestId: string) => {
- const primary = delegatedApprovalRequestDAL.primaryNode();
-
- const delegatedRequest = await delegatedApprovalRequestDAL.findById(delegatedRequestId, primary);
- if (!delegatedRequest) return null;
-
- const approvalRequest = await approvalRequestDAL.findById(delegatedRequest.requestId, primary);
- const delegatedPolicy = approvalRequest?.policyId
- ? await delegatedApprovalPolicyDAL.findOne({ policyId: approvalRequest.policyId }, primary)
- : null;
-
- return { delegatedRequest, approvalRequest, delegatedPolicy };
- };
-
- const init = () => {
- queueService.start(
- QueueName.DelegatedApprovalDispatch,
- async (job) => {
- const { delegatedRequestId } = job.data;
- const attempt = job.attemptsMade + 1;
- const maxAttempts = job.opts.attempts ?? 1;
- const isFinalAttempt = attempt >= maxAttempts;
-
- const requests = await $loadRequests(delegatedRequestId);
- if (!requests) {
- logger.warn(
- `Delegated approval request no longer exists, skipping dispatch [delegatedRequestId=${delegatedRequestId}]`
- );
- return;
- }
-
- const { delegatedRequest, approvalRequest, delegatedPolicy } = requests;
-
- if (
- delegatedRequest.status !== DelegatedApprovalRequestStatus.PendingDispatch &&
- delegatedRequest.status !== DelegatedApprovalRequestStatus.FailedDispatch
- ) {
- logger.info(
- `Delegated approval request is not awaiting dispatch, skipping [delegatedRequestId=${delegatedRequestId}] [status=${delegatedRequest.status}]`
- );
- return;
- }
-
- if (delegatedRequest.providerRequestId) {
- logger.info(
- `Delegated approval request was already delivered, skipping [delegatedRequestId=${delegatedRequestId}] [providerRequestId=${delegatedRequest.providerRequestId}]`
- );
- return;
- }
-
- if (delegatedRequest.status === DelegatedApprovalRequestStatus.FailedDispatch) {
- await delegatedApprovalRequestDAL.update(
- { id: delegatedRequestId, status: DelegatedApprovalRequestStatus.FailedDispatch },
- { status: DelegatedApprovalRequestStatus.PendingDispatch, statusReason: null }
- );
- }
-
- try {
- if (!approvalRequest) {
- throw new UnrecoverableError("The approval request no longer exists.");
- }
- if (approvalRequest.status !== ApprovalRequestStatus.Pending) {
- throw new UnrecoverableError(
- "The approval request is no longer pending, so it was not sent to the delegated approval system."
- );
- }
- if (!delegatedPolicy) {
- throw new UnrecoverableError(
- "The policy's delegated approver was removed before the request could be sent. Configure a delegated approver on the policy and create the request again."
- );
- }
-
- const provider = getDelegatedApprovalProvider(delegatedPolicy.type);
- const details = getDelegatedApprovalDispatchDetailsBuilder(approvalRequest.type)(approvalRequest);
-
- const callbackUrl = delegatedApprovalCallbackService.getCallbackUrl();
- if (!callbackUrl) {
- throw new UnrecoverableError(
- "SITE_URL must be configured so the delegated approval system can call Infisical back."
- );
- }
-
- const primary = delegatedApprovalRequestDAL.primaryNode();
- const [project, rawConnection] = await Promise.all([
- projectDAL.findById(approvalRequest.projectId, primary),
- appConnectionDAL.findById(delegatedPolicy.connectionId, primary)
- ]);
- if (!project) {
- throw new UnrecoverableError("The project of the approval request no longer exists.");
- }
- if (!rawConnection || rawConnection.orgId !== approvalRequest.organizationId) {
- const connectionApp =
- DELEGATED_APPROVAL_CONNECTION_MAP[delegatedPolicy.type as DelegatedApprovalPolicyType];
- throw new UnrecoverableError(
- `App Connection with ID '${delegatedPolicy.connectionId}' was not found. Select a ${APP_CONNECTION_NAME_MAP[connectionApp]} Connection on the policy and create the request again.`
- );
- }
-
- const connection = await decryptAppConnection(rawConnection, kmsService);
-
- const callbackToken = await delegatedApprovalCallbackService.getOrIssueCallbackToken({
- delegatedRequest,
- orgId: approvalRequest.organizationId
- });
- if (!callbackToken) {
- throw new UnrecoverableError(
- "A callback token could not be issued for this request. Create the request again."
- );
- }
-
- const input: TDelegatedApprovalDispatchInput = {
- delegatedRequestId,
- approvalRequestId: approvalRequest.id,
- organizationId: approvalRequest.organizationId,
- productType: project.type as ProjectType,
- project: { id: project.id, name: project.name },
- requester: { name: approvalRequest.requesterName, email: approvalRequest.requesterEmail },
- justification: approvalRequest.justification ?? null,
- callback: { url: callbackUrl, token: callbackToken },
- details
- };
-
- const { providerRequestId } = await provider.dispatch(input, connection);
-
- const [waitingRequest] = await delegatedApprovalRequestDAL.update(
- { id: delegatedRequestId, status: DelegatedApprovalRequestStatus.PendingDispatch },
- { status: DelegatedApprovalRequestStatus.WaitingApproval, statusReason: null, providerRequestId }
- );
- if (!waitingRequest) {
- await delegatedApprovalRequestDAL.updateById(delegatedRequestId, { providerRequestId });
- logger.warn(
- `Delegated approval request left pending dispatch while it was being delivered [delegatedRequestId=${delegatedRequestId}] [providerRequestId=${providerRequestId}]`
- );
- }
-
- logger.info(
- `Delegated approval request delivered [delegatedRequestId=${delegatedRequestId}] [approvalRequestId=${approvalRequest.id}] [provider=${delegatedPolicy.type}] [providerRequestId=${providerRequestId}]`
- );
- } catch (error) {
- const isTerminal = isFinalAttempt || error instanceof UnrecoverableError;
- const message = error instanceof Error ? error.message : String(error);
-
- await $markDispatchFailed(delegatedRequestId, isTerminal ? message : `${message} Infisical will retry.`);
-
- const logContext = { err: { name: (error as Error)?.name, message } };
- const logMessage = `Delegated approval dispatch failed [delegatedRequestId=${delegatedRequestId}] [attempt=${attempt}/${maxAttempts}] [terminal=${isTerminal}]`;
- if (isTerminal) {
- logger.error(logContext, logMessage);
- } else {
- logger.warn(logContext, logMessage);
- }
-
- throw error;
- }
- },
- { concurrency: DELEGATED_APPROVAL_DISPATCH_CONCURRENCY }
- );
- };
-
- return { queueDispatch, init };
-};
diff --git a/backend/src/services/delegated-approval/dispatch/delegated-approval-dispatch-types.ts b/backend/src/services/delegated-approval/dispatch/delegated-approval-dispatch-types.ts
index 15861ade66..39e5838c93 100644
--- a/backend/src/services/delegated-approval/dispatch/delegated-approval-dispatch-types.ts
+++ b/backend/src/services/delegated-approval/dispatch/delegated-approval-dispatch-types.ts
@@ -42,5 +42,3 @@ export type TDelegatedApprovalProvider = {
export type TDelegatedApprovalDispatchDetailsBuilder = (
request: TApprovalRequests
) => TDelegatedApprovalDispatchDetails;
-
-export type TDelegatedApprovalDispatchJobPayload = { delegatedRequestId: string };
diff --git a/backend/src/services/secret-access-approval-global-request-bridge/secret-access-approval-global-request-bridge-service.ts b/backend/src/services/secret-access-approval-global-request-bridge/secret-access-approval-global-request-bridge-service.ts
index 18dba57ed6..a3891c9fcf 100644
--- a/backend/src/services/secret-access-approval-global-request-bridge/secret-access-approval-global-request-bridge-service.ts
+++ b/backend/src/services/secret-access-approval-global-request-bridge/secret-access-approval-global-request-bridge-service.ts
@@ -78,7 +78,6 @@ export const secretAccessApprovalGlobalRequestBridgeServiceFactory = ({
secretAccessApprovalGlobalPolicyBridgeDAL,
secretAccessApprovalGlobalRequestBridgeDAL,
delegatedApprovalService,
- delegatedApprovalDispatchQueue,
smtpService,
notificationService,
kmsService,
@@ -332,7 +331,7 @@ export const secretAccessApprovalGlobalRequestBridgeServiceFactory = ({
const parsedMs = requestExpirationTime ? ms(requestExpirationTime) : null;
const expiresAt = parsedMs && !Number.isNaN(parsedMs) ? new Date(Date.now() + parsedMs) : null;
- const { created, delegatedRequest } = await approvalRequestDAL.transaction(async (tx) => {
+ const created = await approvalRequestDAL.transaction(async (tx) => {
const approvalRequest = await createApprovalRequestWithSteps(
{
projectId,
@@ -351,16 +350,12 @@ export const secretAccessApprovalGlobalRequestBridgeServiceFactory = ({
{ approvalRequestDAL, approvalRequestStepsDAL, approvalRequestStepEligibleApproversDAL },
tx
);
- return {
- created: approvalRequest,
- delegatedRequest: delegatedPolicy ? await delegatedApprovalService.createRequest({ approvalRequest }, tx) : null
- };
+ if (delegatedPolicy) {
+ await delegatedApprovalService.createRequest({ approvalRequest }, tx);
+ }
+ return approvalRequest;
});
- if (delegatedRequest) {
- await delegatedApprovalDispatchQueue.queueDispatch({ delegatedRequestId: delegatedRequest.id });
- }
-
const request = await $findRequestById(created.id);
const [firstStep] = policy.steps;
if (firstStep) {
diff --git a/backend/src/services/secret-access-approval-global-request-bridge/secret-access-approval-global-request-bridge-types.ts b/backend/src/services/secret-access-approval-global-request-bridge/secret-access-approval-global-request-bridge-types.ts
index 1d80eea497..0e84c1b0da 100644
--- a/backend/src/services/secret-access-approval-global-request-bridge/secret-access-approval-global-request-bridge-types.ts
+++ b/backend/src/services/secret-access-approval-global-request-bridge/secret-access-approval-global-request-bridge-types.ts
@@ -18,7 +18,6 @@ import {
import { TSecretAccessApprovalGlobalResource } from "@app/services/approval-policy/secret-access/secret-access-policy-factory";
import { TSecretAccessPolicy } from "@app/services/approval-policy/secret-access/secret-access-policy-types";
import { TDelegatedApprovalServiceFactory } from "@app/services/delegated-approval/delegated-approval-service";
-import { TDelegatedApprovalDispatchQueueFactory } from "@app/services/delegated-approval/dispatch/delegated-approval-dispatch-queue";
import { TKmsServiceFactory } from "@app/services/kms/kms-service";
import { TMicrosoftTeamsServiceFactory } from "@app/services/microsoft-teams/microsoft-teams-service";
import { TProjectMicrosoftTeamsConfigDALFactory } from "@app/services/microsoft-teams/project-microsoft-teams-config-dal";
@@ -67,7 +66,6 @@ export type TSecretAccessApprovalGlobalRequestBridgeServiceFactoryDep = {
TDelegatedApprovalServiceFactory,
"getPolicyByPolicyId" | "createRequest" | "getRequestByRequestId" | "getRequestsByRequestIds" | "retryDispatch"
>;
- delegatedApprovalDispatchQueue: Pick<TDelegatedApprovalDispatchQueueFactory, "queueDispatch">;
smtpService: Pick<TSmtpService, "sendMail">;
notificationService: Pick<TNotificationServiceFactory, "createUserNotifications">;
kmsService: Pick<TKmsServiceFactory, "createCipherPairWithDataKey">;