Skip to content

Infisical is the open-source platform for securing secrets, certificates, access, and AI Agents.

address final comments

adilsonf6705ce438b9
diff --git a/CLAUDE.md b/CLAUDE.md
index a74c2aa08d..3d174126bb 100644
--- a/CLAUDE.md
+++ b/CLAUDE.md
@@ -165,8 +165,12 @@ the dispatch is exactly as durable as the request; never enqueue it after commit
`delegated-approval-dispatch` consumer (`delegated-approval/dispatch/delegated-approval-dispatch-consumer.ts`)
calls the provider inline and moves the row `pending_dispatch` → `waiting_approval` (storing the provider's
record id in `providerRequestId`) or `failed_dispatch` with a `statusReason` a user can read. A transient
-failure returns `Retry`, so the outbox backs off and tries again (up to `MAX_OUTBOX_ATTEMPTS`); the reason text
-promises that retry on every attempt because the consumer cannot see the attempt count. A user resends a
+failure returns `Retry`, so the outbox backs off and tries again (up to `MAX_OUTBOX_ATTEMPTS`), and the reason
+text promises that retry. The outbox flags the last attempt (`TEvent.isFinalAttempt`), and a transient failure
+on it closes the request instead: the delegated row goes `failed` with a reason that says so, the approval
+request goes `cancelled`, and the consumer returns `Failed`. A terminal failure the consumer raises itself (an
+`UnrecoverableError`: SITE_URL unset, the app connection gone, the policy no longer delegated) stays
+`failed_dispatch` on every attempt, because an admin can fix the cause and resend. A user resends a
`failed_dispatch` request with `POST /api/v1/access-approvals/requests/:requestId/retry-delegated-dispatch`. The
product's bridge checks who may retry, and `delegatedApprovalService.retryDispatch` checks the request can be
retried and emits the same event with no transaction. The emit's idempotency key is the event, the delegated
@@ -181,13 +185,15 @@ policy blocked by pending requests is freed for editing. Every provider implemen
(`dispatch/providers/<name>/`); a new policy type adds an arm to `TDelegatedApprovalDispatchDetails` and a
builder in `DELEGATED_APPROVAL_DISPATCH_DETAILS_BUILDER_MAP` rather than branching in a provider. The consumer
reads from the primary, never checks the license, and treats bullmq's `UnrecoverableError` as the terminal failure
-signal, returning `Failed` for it and `Retry` for anything else. The callback contract (token issuance now, the inbound route and approve / reject later) is its own
+signal, returning `Failed` for it and `Retry` for anything else until the last attempt. The callback contract (token issuance now, the inbound route and approve / reject later) is its own
module, `backend/src/services/delegated-approval-callback/`, so the delegated approval module never grows a
callback branch. A request gets exactly one callback token for its whole life: it is random, issued on the first
-dispatch attempt, and stored encrypted under the org KMS key in `encryptedCallbackToken` so every retry resends
-the same value; the inbound side finds the row by the provider's record id and compares the decrypted token with
-the presented one. Never send a provider a token that differs from the one stored for the request; the provider
-keeps the first token it was given and cannot be told another, so a mismatch is unrecoverable.
+dispatch attempt, and stored in `encryptedCallbackToken` under the SecretManager KMS key of the request's project
+(`getOrIssueCallbackToken` resolves the project from the delegated request itself, since the inbound route knows
+only that row) so every retry resends the same value; the inbound side finds the row by the provider's record id
+and compares the decrypted token with the presented one. Never send a provider a token that differs from the one
+stored for the request; the provider keeps the first token it was given and cannot be told another, so a mismatch
+is unrecoverable.
### Object Storage
diff --git a/backend/CLAUDE.md b/backend/CLAUDE.md
index 69f9eb6bfb..51dc6ee328 100644
--- a/backend/CLAUDE.md
+++ b/backend/CLAUDE.md
@@ -815,6 +815,10 @@ await someDAL.transaction(async (tx) => {
no per-event state for a consumer: what a retry should skip is recorded wherever the consumer already
keeps delivery records, keyed by `TEvent.id` (stable across attempts). The alert consumer files each
run under that id in `alert_history.eventId` and the engine skips channels already recorded there.
+ The one piece of retry state a consumer sees is `TEvent.isFinalAttempt`: the row is `failed` after that
+ attempt whatever the consumer returns, so a consumer with state of its own records the terminal outcome
+ then (the delegated approval dispatch closes its request). A give-up from the stale-claim sweep never
+ reaches a consumer.
Backoff is exponential with jitter from 30s, and `MAX_OUTBOX_ATTEMPTS` puts the last attempt about an
hour after the first, because a `failed` row is a notification nobody will receive. To replay failed
rows by hand: `status = 'retry', attempts = 0, nextRetryAt = now()`.
diff --git a/backend/src/server/routes/index.ts b/backend/src/server/routes/index.ts
index 963831af0e..1f492a891a 100644
--- a/backend/src/server/routes/index.ts
+++ b/backend/src/server/routes/index.ts
@@ -3243,6 +3243,7 @@ export const registerRoutes = async (
const delegatedApprovalCallbackService = delegatedApprovalCallbackServiceFactory({
delegatedApprovalRequestDAL,
+ approvalRequestDAL,
kmsService
});
diff --git a/backend/src/services/alert/alert-event-consumer.test.ts b/backend/src/services/alert/alert-event-consumer.test.ts
index 7ef5ef92b5..0342925398 100644
--- a/backend/src/services/alert/alert-event-consumer.test.ts
+++ b/backend/src/services/alert/alert-event-consumer.test.ts
@@ -30,6 +30,7 @@ const makeEvent = (overrides?: Partial<TEvent>): TEvent =>
eventType: EVENT_TYPE,
payload: makePayload(),
occurredAt: new Date(),
+ isFinalAttempt: false,
...overrides
}) as TEvent;
diff --git a/backend/src/services/delegated-approval-callback/delegated-approval-callback-service.test.ts b/backend/src/services/delegated-approval-callback/delegated-approval-callback-service.test.ts
index 58f77f0c56..3e20ed8d5e 100644
--- a/backend/src/services/delegated-approval-callback/delegated-approval-callback-service.test.ts
+++ b/backend/src/services/delegated-approval-callback/delegated-approval-callback-service.test.ts
@@ -17,15 +17,25 @@ const CIPHER_PREFIX = Buffer.from("enc:");
const encrypt = (plainText: Buffer) => Buffer.concat([CIPHER_PREFIX, plainText]);
const decrypt = (cipherTextBlob: Buffer) => cipherTextBlob.subarray(CIPHER_PREFIX.length);
+const PRIMARY = { primary: true };
+
+const delegatedRequest = { id: "delegated-1", requestId: "req-1", encryptedCallbackToken: null };
+
const makeService = ({
stored = true,
- currentRow = null
-}: { stored?: boolean; currentRow?: { encryptedCallbackToken: Buffer | null } | null } = {}) => {
+ currentRow = null,
+ approvalRequest = { id: "req-1", projectId: "project-1" }
+}: {
+ stored?: boolean;
+ currentRow?: { encryptedCallbackToken: Buffer | null } | null;
+ approvalRequest?: { id: string; projectId: string } | null;
+} = {}) => {
const delegatedApprovalRequestDAL = {
update: vi.fn(async () => (stored ? [{ id: "delegated-1" }] : [])),
- primaryNode: vi.fn(() => ({})),
+ primaryNode: vi.fn(() => PRIMARY),
findById: vi.fn(async () => currentRow)
};
+ const approvalRequestDAL = { findById: vi.fn(async () => approvalRequest) };
const kmsService = {
createCipherPairWithDataKey: vi.fn(async () => ({
encryptor: ({ plainText }: { plainText: Buffer }) => ({ cipherTextBlob: encrypt(plainText) }),
@@ -35,10 +45,11 @@ const makeService = ({
const service = delegatedApprovalCallbackServiceFactory({
delegatedApprovalRequestDAL: delegatedApprovalRequestDAL as never,
+ approvalRequestDAL: approvalRequestDAL as never,
kmsService: kmsService as never
});
- return { service, delegatedApprovalRequestDAL, kmsService };
+ return { service, delegatedApprovalRequestDAL, approvalRequestDAL, kmsService };
};
describe("getCallbackUrl", () => {
@@ -50,18 +61,16 @@ describe("getCallbackUrl", () => {
});
describe("getOrIssueCallbackToken", () => {
- test("issues a token once and stores it encrypted under the org key", async () => {
- const { service, delegatedApprovalRequestDAL, kmsService } = makeService();
+ test("issues a token once and stores it encrypted under the project key", async () => {
+ const { service, delegatedApprovalRequestDAL, approvalRequestDAL, kmsService } = makeService();
- const token = await service.getOrIssueCallbackToken({
- delegatedRequest: { id: "delegated-1", encryptedCallbackToken: null },
- orgId: "org-1"
- });
+ const token = await service.getOrIssueCallbackToken({ delegatedRequest });
expect(token).toBeTruthy();
+ expect(approvalRequestDAL.findById).toHaveBeenCalledWith("req-1", PRIMARY);
expect(kmsService.createCipherPairWithDataKey).toHaveBeenCalledWith({
- type: KmsDataKey.Organization,
- orgId: "org-1"
+ type: KmsDataKey.SecretManager,
+ projectId: "project-1"
});
expect(delegatedApprovalRequestDAL.update).toHaveBeenCalledWith(
{ id: "delegated-1", encryptedCallbackToken: null },
@@ -73,8 +82,7 @@ describe("getOrIssueCallbackToken", () => {
const { service, delegatedApprovalRequestDAL } = makeService();
const token = await service.getOrIssueCallbackToken({
- delegatedRequest: { id: "delegated-1", encryptedCallbackToken: encrypt(Buffer.from("first-token")) },
- orgId: "org-1"
+ delegatedRequest: { ...delegatedRequest, encryptedCallbackToken: encrypt(Buffer.from("first-token")) }
});
expect(token).toBe("first-token");
@@ -87,10 +95,7 @@ describe("getOrIssueCallbackToken", () => {
currentRow: { encryptedCallbackToken: encrypt(Buffer.from("winner-token")) }
});
- const token = await service.getOrIssueCallbackToken({
- delegatedRequest: { id: "delegated-1", encryptedCallbackToken: null },
- orgId: "org-1"
- });
+ const token = await service.getOrIssueCallbackToken({ delegatedRequest });
expect(token).toBe("winner-token");
});
@@ -98,11 +103,18 @@ describe("getOrIssueCallbackToken", () => {
test("returns null when the request vanished before a token could be stored", async () => {
const { service } = makeService({ stored: false, currentRow: { encryptedCallbackToken: null } });
- const token = await service.getOrIssueCallbackToken({
- delegatedRequest: { id: "delegated-1", encryptedCallbackToken: null },
- orgId: "org-1"
- });
+ const token = await service.getOrIssueCallbackToken({ delegatedRequest });
expect(token).toBeNull();
});
+
+ test("returns null without issuing a token when the approval request is gone", async () => {
+ const { service, delegatedApprovalRequestDAL, kmsService } = makeService({ approvalRequest: null });
+
+ const token = await service.getOrIssueCallbackToken({ delegatedRequest });
+
+ expect(token).toBeNull();
+ expect(kmsService.createCipherPairWithDataKey).not.toHaveBeenCalled();
+ expect(delegatedApprovalRequestDAL.update).not.toHaveBeenCalled();
+ });
});
diff --git a/backend/src/services/delegated-approval-callback/delegated-approval-callback-service.ts b/backend/src/services/delegated-approval-callback/delegated-approval-callback-service.ts
index 59b919649e..e1057bbb5d 100644
--- a/backend/src/services/delegated-approval-callback/delegated-approval-callback-service.ts
+++ b/backend/src/services/delegated-approval-callback/delegated-approval-callback-service.ts
@@ -12,6 +12,7 @@ export type TDelegatedApprovalCallbackServiceFactory = ReturnType<typeof delegat
export const delegatedApprovalCallbackServiceFactory = ({
delegatedApprovalRequestDAL,
+ approvalRequestDAL,
kmsService
}: TDelegatedApprovalCallbackServiceFactoryDep) => {
const getCallbackUrl = () => {
@@ -19,10 +20,14 @@ export const delegatedApprovalCallbackServiceFactory = ({
return SITE_URL ? `${SITE_URL}${DELEGATED_APPROVAL_CALLBACK_PATH}` : null;
};
- const getOrIssueCallbackToken = async ({ delegatedRequest, orgId }: TGetOrIssueCallbackTokenDTO) => {
+ const getOrIssueCallbackToken = async ({ delegatedRequest }: TGetOrIssueCallbackTokenDTO) => {
+ const primary = delegatedApprovalRequestDAL.primaryNode();
+ const approvalRequest = await approvalRequestDAL.findById(delegatedRequest.requestId, primary);
+ if (!approvalRequest) return null;
+
const { encryptor, decryptor } = await kmsService.createCipherPairWithDataKey({
- type: KmsDataKey.Organization,
- orgId
+ type: KmsDataKey.SecretManager,
+ projectId: approvalRequest.projectId
});
if (delegatedRequest.encryptedCallbackToken) {
@@ -36,10 +41,7 @@ export const delegatedApprovalCallbackServiceFactory = ({
);
if (issued) return token;
- const current = await delegatedApprovalRequestDAL.findById(
- delegatedRequest.id,
- delegatedApprovalRequestDAL.primaryNode()
- );
+ const current = await delegatedApprovalRequestDAL.findById(delegatedRequest.id, primary);
if (!current?.encryptedCallbackToken) return null;
return decryptor({ cipherTextBlob: current.encryptedCallbackToken }).toString();
diff --git a/backend/src/services/delegated-approval-callback/delegated-approval-callback-types.ts b/backend/src/services/delegated-approval-callback/delegated-approval-callback-types.ts
index 643d2e29c0..9b638cba1f 100644
--- a/backend/src/services/delegated-approval-callback/delegated-approval-callback-types.ts
+++ b/backend/src/services/delegated-approval-callback/delegated-approval-callback-types.ts
@@ -1,13 +1,14 @@
import { TDelegatedApprovalRequests } from "@app/db/schemas";
+import { TApprovalRequestDALFactory } from "@app/services/approval-policy/approval-request-dal";
import { TDelegatedApprovalRequestDALFactory } from "@app/services/delegated-approval/delegated-approval-request-dal";
import { TKmsServiceFactory } from "@app/services/kms/kms-service";
export type TDelegatedApprovalCallbackServiceFactoryDep = {
delegatedApprovalRequestDAL: Pick<TDelegatedApprovalRequestDALFactory, "findById" | "primaryNode" | "update">;
+ approvalRequestDAL: Pick<TApprovalRequestDALFactory, "findById">;
kmsService: Pick<TKmsServiceFactory, "createCipherPairWithDataKey">;
};
export type TGetOrIssueCallbackTokenDTO = {
- delegatedRequest: Pick<TDelegatedApprovalRequests, "id" | "encryptedCallbackToken">;
- orgId: string;
+ delegatedRequest: Pick<TDelegatedApprovalRequests, "id" | "requestId" | "encryptedCallbackToken">;
};
diff --git a/backend/src/services/delegated-approval/dispatch/delegated-approval-dispatch-consumer.test.ts b/backend/src/services/delegated-approval/dispatch/delegated-approval-dispatch-consumer.test.ts
index d334a5e08e..ce5c299cdf 100644
--- a/backend/src/services/delegated-approval/dispatch/delegated-approval-dispatch-consumer.test.ts
+++ b/backend/src/services/delegated-approval/dispatch/delegated-approval-dispatch-consumer.test.ts
@@ -5,7 +5,12 @@ import { TEventOutbox } from "@app/db/schemas";
import { ApprovalPolicyType, ApprovalRequestStatus } from "@app/services/approval-policy/approval-policy-enums";
import { eventOutboxRegistryFactory } from "@app/services/event-outbox/event-outbox-registry";
import { eventOutboxServiceFactory } from "@app/services/event-outbox/event-outbox-service";
-import { EventOutboxStatus, EventResultStatus, TEvent } from "@app/services/event-outbox/event-outbox-types";
+import {
+ EventOutboxStatus,
+ EventResultStatus,
+ MAX_OUTBOX_ATTEMPTS,
+ TEvent
+} from "@app/services/event-outbox/event-outbox-types";
import { DelegatedApprovalPolicyType, DelegatedApprovalRequestStatus } from "../delegated-approval-enums";
import { DELEGATED_APPROVAL_DISPATCH_REQUESTED_EVENT } from "../delegated-approval-events";
@@ -74,6 +79,7 @@ const delegatedPolicy = {
};
const PRIMARY = { primary: true } as never;
+const TX = { tx: true } as never;
const buildConsumer = ({
delegatedRequest = pendingDelegatedRequest as TRow | null,
@@ -96,10 +102,13 @@ const buildConsumer = ({
return row;
});
+ const approvalUpdate = vi.fn(async () => [approval]);
+
const consumer = delegatedApprovalDispatchConsumerFactory({
delegatedApprovalRequestDAL: {
findById: findById ?? vi.fn(async () => row),
primaryNode: () => PRIMARY,
+ transaction: async (cb: (tx: never) => Promise<unknown>) => cb(TX),
update,
updateById
},
@@ -108,7 +117,7 @@ const buildConsumer = ({
getCallbackUrl: () => "https://infisical.example.com/api/v1/delegated-approvals/callback",
getOrIssueCallbackToken: vi.fn().mockResolvedValue("callback-token")
},
- approvalRequestDAL: { findById: vi.fn().mockResolvedValue(approval) },
+ approvalRequestDAL: { findById: vi.fn().mockResolvedValue(approval), update: approvalUpdate },
projectDAL: { findById: vi.fn().mockResolvedValue({ id: "project-1", name: "Project", type: "secret-manager" }) },
appConnectionDAL: {
findById: vi.fn().mockResolvedValue({ id: "connection-1", orgId: ORG_ID, app: "servicenow" })
@@ -116,7 +125,7 @@ const buildConsumer = ({
kmsService: { createCipherPairWithDataKey: vi.fn() }
} as never);
- return { consumer, row: () => row, update };
+ return { consumer, row: () => row, update, approvalUpdate };
};
const makeEvent = (overrides?: Partial<TEvent>): TEvent => ({
@@ -124,6 +133,7 @@ const makeEvent = (overrides?: Partial<TEvent>): TEvent => ({
eventType: DELEGATED_APPROVAL_DISPATCH_REQUESTED_EVENT,
payload: { delegatedRequestId: DELEGATED_REQUEST_ID },
occurredAt: new Date(),
+ isFinalAttempt: false,
...overrides
});
@@ -177,6 +187,38 @@ describe("delegatedApprovalDispatchConsumer", () => {
});
});
+ test("closes the request when a transient failure lands on the last attempt", async () => {
+ dispatch.mockRejectedValueOnce(new Error("ServiceNow did not respond."));
+ const { consumer, row, approvalUpdate } = buildConsumer();
+
+ const results = await consumer.handle([makeEvent({ isFinalAttempt: true })]);
+
+ expect(results).toEqual([{ id: "1", status: EventResultStatus.Failed, error: "ServiceNow did not respond." }]);
+ expect(row()).toMatchObject({
+ status: DelegatedApprovalRequestStatus.Failed,
+ statusReason: `ServiceNow did not respond. Infisical could not send this request after ${MAX_OUTBOX_ATTEMPTS} attempts and closed it. Create a new request.`
+ });
+ expect(approvalUpdate).toHaveBeenCalledWith(
+ { id: "req-1", status: ApprovalRequestStatus.Pending },
+ { status: ApprovalRequestStatus.Cancelled },
+ TX
+ );
+ });
+
+ test("keeps a terminal failure on the last attempt resendable", async () => {
+ dispatch.mockRejectedValueOnce(new UnrecoverableError("ServiceNow rejected the credentials."));
+ const { consumer, row, approvalUpdate } = buildConsumer();
+
+ const [result] = await consumer.handle([makeEvent({ isFinalAttempt: true })]);
+
+ expect(result.status).toBe(EventResultStatus.Failed);
+ expect(row()).toMatchObject({
+ status: DelegatedApprovalRequestStatus.FailedDispatch,
+ statusReason: "ServiceNow rejected the credentials."
+ });
+ expect(approvalUpdate).not.toHaveBeenCalled();
+ });
+
test("fails terminally when the approval request is no longer pending", async () => {
const { consumer, row } = buildConsumer({
approval: { ...approvalRequest, status: ApprovalRequestStatus.Cancelled }
@@ -409,4 +451,31 @@ describe("delegatedApprovalDispatchConsumer through the outbox", () => {
});
expect(dispatch).toHaveBeenCalledTimes(2);
});
+
+ test("closes the request once the outbox has used up its attempts", async () => {
+ dispatch.mockRejectedValue(new Error("ServiceNow did not respond."));
+ const { consumer, row, approvalUpdate } = buildConsumer();
+ const { service, rows, elapseBackoff } = buildOutbox(consumer);
+
+ await service.emit(
+ { eventType: DELEGATED_APPROVAL_DISPATCH_REQUESTED_EVENT, payload: { delegatedRequestId: DELEGATED_REQUEST_ID } },
+ PRIMARY
+ );
+
+ for (let attempt = 1; attempt < MAX_OUTBOX_ATTEMPTS; attempt += 1) {
+ // eslint-disable-next-line no-await-in-loop -- one drain per attempt
+ await service.drain({ consumer: consumer.name });
+ expect(rows[0].status).toBe(EventOutboxStatus.Retry);
+ expect(row()?.status).toBe(DelegatedApprovalRequestStatus.FailedDispatch);
+ expect(approvalUpdate).not.toHaveBeenCalled();
+ elapseBackoff();
+ }
+
+ await service.drain({ consumer: consumer.name });
+
+ expect(dispatch).toHaveBeenCalledTimes(MAX_OUTBOX_ATTEMPTS);
+ expect(rows[0]).toMatchObject({ status: EventOutboxStatus.Failed, attempts: MAX_OUTBOX_ATTEMPTS });
+ expect(row()?.status).toBe(DelegatedApprovalRequestStatus.Failed);
+ expect(approvalUpdate).toHaveBeenCalledTimes(1);
+ });
});
diff --git a/backend/src/services/delegated-approval/dispatch/delegated-approval-dispatch-consumer.ts b/backend/src/services/delegated-approval/dispatch/delegated-approval-dispatch-consumer.ts
index 8322a0c325..b81dad2903 100644
--- a/backend/src/services/delegated-approval/dispatch/delegated-approval-dispatch-consumer.ts
+++ b/backend/src/services/delegated-approval/dispatch/delegated-approval-dispatch-consumer.ts
@@ -12,6 +12,7 @@ import { TDelegatedApprovalCallbackServiceFactory } from "@app/services/delegate
import {
EventResultStatus,
IEventConsumer,
+ MAX_OUTBOX_ATTEMPTS,
TEvent,
TEventConsumerResult
} from "@app/services/event-outbox/event-outbox-types";
@@ -41,14 +42,14 @@ const DELEGATED_APPROVAL_DISPATCH_CONCURRENCY = 10;
type TDelegatedApprovalDispatchConsumerDep = {
delegatedApprovalRequestDAL: Pick<
TDelegatedApprovalRequestDALFactory,
- "findById" | "primaryNode" | "update" | "updateById"
+ "findById" | "primaryNode" | "transaction" | "update" | "updateById"
>;
delegatedApprovalPolicyDAL: Pick<TDelegatedApprovalPolicyDALFactory, "findOne">;
delegatedApprovalCallbackService: Pick<
TDelegatedApprovalCallbackServiceFactory,
"getCallbackUrl" | "getOrIssueCallbackToken"
>;
- approvalRequestDAL: Pick<TApprovalRequestDALFactory, "findById">;
+ approvalRequestDAL: Pick<TApprovalRequestDALFactory, "findById" | "update">;
projectDAL: Pick<TProjectDALFactory, "findById">;
appConnectionDAL: Pick<TAppConnectionDALFactory, "findById">;
kmsService: Pick<TKmsServiceFactory, "createCipherPairWithDataKey">;
@@ -82,6 +83,39 @@ export const delegatedApprovalDispatchConsumerFactory = ({
}
};
+ const $closeAfterExhaustedAttempts = async ({
+ delegatedRequestId,
+ approvalRequestId,
+ statusReason
+ }: {
+ delegatedRequestId: string;
+ approvalRequestId: string;
+ statusReason: string;
+ }) => {
+ try {
+ await delegatedApprovalRequestDAL.transaction(async (tx) => {
+ await delegatedApprovalRequestDAL.update(
+ { id: delegatedRequestId, status: DelegatedApprovalRequestStatus.PendingDispatch },
+ {
+ status: DelegatedApprovalRequestStatus.Failed,
+ statusReason: statusReason.slice(0, DELEGATED_APPROVAL_STATUS_REASON_MAX_CHARS)
+ },
+ tx
+ );
+ await approvalRequestDAL.update(
+ { id: approvalRequestId, status: ApprovalRequestStatus.Pending },
+ { status: ApprovalRequestStatus.Cancelled },
+ tx
+ );
+ });
+ } catch (error) {
+ logger.error(
+ error,
+ `Failed to close a delegated approval request whose dispatch attempts ran out [delegatedRequestId=${delegatedRequestId}] [approvalRequestId=${approvalRequestId}]`
+ );
+ }
+ };
+
// Both rows were committed together with the event, so they are read from the primary.
const $loadRequests = async (delegatedRequestId: string) => {
const primary = delegatedApprovalRequestDAL.primaryNode();
@@ -97,7 +131,10 @@ export const delegatedApprovalDispatchConsumerFactory = ({
return { delegatedRequest, approvalRequest, delegatedPolicy };
};
- const $dispatch = async (delegatedRequestId: string): Promise<Omit<TEventConsumerResult, "id">> => {
+ const $dispatch = async (
+ delegatedRequestId: string,
+ isFinalAttempt: boolean
+ ): Promise<Omit<TEventConsumerResult, "id">> => {
const requests = await $loadRequests(delegatedRequestId);
if (!requests) {
logger.warn(
@@ -180,10 +217,7 @@ export const delegatedApprovalDispatchConsumerFactory = ({
const connection = await decryptAppConnection(rawConnection, kmsService);
- const callbackToken = await delegatedApprovalCallbackService.getOrIssueCallbackToken({
- delegatedRequest,
- orgId: approvalRequest.organizationId
- });
+ const callbackToken = await delegatedApprovalCallbackService.getOrIssueCallbackToken({ delegatedRequest });
if (!callbackToken) {
throw new UnrecoverableError(
"A callback token could not be issued for this request. Create the request again."
@@ -222,10 +256,23 @@ export const delegatedApprovalDispatchConsumerFactory = ({
} catch (error) {
const isTerminal = error instanceof UnrecoverableError;
const message = error instanceof Error ? error.message : String(error);
+ const logContext = { err: { name: (error as Error)?.name, message } };
- // The outbox decides when retries run out and the consumer never sees the attempt count, so the
- // reason has to read true on every attempt. The row is failed_dispatch throughout, which is what
- // makes the manual resend available while the outbox is still retrying.
+ if (!isTerminal && isFinalAttempt && approvalRequest) {
+ await $closeAfterExhaustedAttempts({
+ delegatedRequestId,
+ approvalRequestId: approvalRequest.id,
+ statusReason: `${message} Infisical could not send this request after ${MAX_OUTBOX_ATTEMPTS} attempts and closed it. Create a new request.`
+ });
+ logger.error(
+ logContext,
+ `Delegated approval dispatch attempts ran out, request closed [delegatedRequestId=${delegatedRequestId}] [approvalRequestId=${approvalRequest.id}]`
+ );
+ return { status: EventResultStatus.Failed, error: message };
+ }
+
+ // The row is failed_dispatch throughout, which is what makes the manual resend available while the
+ // outbox is still retrying.
await $markDispatchFailed(
delegatedRequestId,
isTerminal
@@ -233,7 +280,6 @@ export const delegatedApprovalDispatchConsumerFactory = ({
: `${message} Infisical will retry automatically. If it keeps failing, send the request again.`
);
- const logContext = { err: { name: (error as Error)?.name, message } };
const logMessage = `Delegated approval dispatch failed [delegatedRequestId=${delegatedRequestId}] [terminal=${isTerminal}]`;
if (isTerminal) {
logger.error(logContext, logMessage);
@@ -256,7 +302,7 @@ export const delegatedApprovalDispatchConsumerFactory = ({
};
}
- return { id, ...(await $dispatch(payload.data.delegatedRequestId)) };
+ return { id, ...(await $dispatch(payload.data.delegatedRequestId, event.isFinalAttempt)) };
};
const $runEvent = async (event: TEvent): Promise<TEventConsumerResult> => {
diff --git a/backend/src/services/event-outbox/event-outbox-service.test.ts b/backend/src/services/event-outbox/event-outbox-service.test.ts
index 2cce8ceb34..9316073af9 100644
--- a/backend/src/services/event-outbox/event-outbox-service.test.ts
+++ b/backend/src/services/event-outbox/event-outbox-service.test.ts
@@ -270,6 +270,24 @@ describe("event outbox drain", () => {
expect(commits[0].retriable).toEqual([]);
});
+ test("hands the consumer isFinalAttempt only on its last attempt", async () => {
+ const received: { id: string; isFinalAttempt: boolean }[] = [];
+ const { service } = buildDrain({
+ batches: [[makeRow({ id: 1, attempts: 0 }), makeRow({ id: 2, attempts: MAX_OUTBOX_ATTEMPTS - 1 })]],
+ handle: async (events) => {
+ events.forEach((event) => received.push({ id: String(event.id), isFinalAttempt: event.isFinalAttempt }));
+ return events.map((event) => ({ id: String(event.id), status: EventResultStatus.Retry }));
+ }
+ });
+
+ await service.drain(KEY);
+
+ expect(received).toEqual([
+ { id: "1", isFinalAttempt: false },
+ { id: "2", isFinalAttempt: true }
+ ]);
+ });
+
// One UPDATE per distinct outcome, so a hundred deliveries is one statement. Rows on the same attempt
// share a retry delay, so they group too.
test("groups rows that share an outcome into one commit entry", async () => {
diff --git a/backend/src/services/event-outbox/event-outbox-service.ts b/backend/src/services/event-outbox/event-outbox-service.ts
index 595fd4cbcc..b5370e29b4 100644
--- a/backend/src/services/event-outbox/event-outbox-service.ts
+++ b/backend/src/services/event-outbox/event-outbox-service.ts
@@ -15,11 +15,13 @@ import {
EventResultStatus,
FAILED_RETENTION_MS,
IEventConsumer,
+ isFinalOutboxAttempt,
MAX_BATCHES_PER_FLUSH,
MAX_OUTBOX_ATTEMPTS,
MAX_OUTBOX_PAYLOAD_BYTES,
OUTBOX_CLAIM_BATCH_SIZE,
STALE_CLAIM_THRESHOLD_MS,
+ TEvent,
TEventConsumerResult,
TEventInput,
TOutboxFlushKey
@@ -192,7 +194,7 @@ export const eventOutboxServiceFactory = ({ eventOutboxDAL, eventOutboxRegistry
if (result.status === EventResultStatus.Delivered) {
finalStatus = EventOutboxStatus.Delivered;
delivered.push({ id });
- } else if (result.status === EventResultStatus.Failed || attempts >= MAX_OUTBOX_ATTEMPTS) {
+ } else if (result.status === EventResultStatus.Failed || isFinalOutboxAttempt(row.attempts)) {
finalStatus = EventOutboxStatus.Failed;
failed.push({ id, error: result.error });
} else {
@@ -241,6 +243,13 @@ export const eventOutboxServiceFactory = ({ eventOutboxDAL, eventOutboxRegistry
lockToken: string
): Promise<TEventConsumerResult[]> => {
const ids = claimed.map((row) => String(row.id));
+ const events: TEvent[] = claimed.map((row) => ({
+ id: row.id,
+ eventType: row.eventType,
+ payload: row.payload,
+ occurredAt: row.occurredAt,
+ isFinalAttempt: isFinalOutboxAttempt(row.attempts)
+ }));
const heartbeat = setInterval(() => {
eventOutboxDAL.extendClaims(ids, lockToken).catch((error) => {
logger.warn(error, `event-outbox: failed to extend a claim [consumer=${key.consumer}]`);
@@ -249,7 +258,7 @@ export const eventOutboxServiceFactory = ({ eventOutboxDAL, eventOutboxRegistry
try {
return await withTimeout(
- consumer.handle(claimed),
+ consumer.handle(events),
CONSUMER_HANDLE_TIMEOUT_MS,
() => new ConsumerHandleTimeoutError(key.consumer, CONSUMER_HANDLE_TIMEOUT_MS)
);
diff --git a/backend/src/services/event-outbox/event-outbox-types.ts b/backend/src/services/event-outbox/event-outbox-types.ts
index efd1a58a5a..b7613bca99 100644
--- a/backend/src/services/event-outbox/event-outbox-types.ts
+++ b/backend/src/services/event-outbox/event-outbox-types.ts
@@ -24,6 +24,8 @@ export const OUTBOX_CLAIM_BATCH_SIZE = 100;
export const MAX_OUTBOX_ATTEMPTS = 8;
+export const isFinalOutboxAttempt = (attemptsMade: number): boolean => attemptsMade + 1 >= MAX_OUTBOX_ATTEMPTS;
+
export const OUTBOX_BACKOFF_BASE_MS = 30_000;
export const computeBackoffMs = (attemptsAfterIncrement: number): number => {
@@ -61,8 +63,10 @@ export type TEventEmitter = {
emit: (event: TEventInput, tx?: Knex) => Promise<void>;
};
-// What a consumer sees. Lock, attempt and status columns are the outbox's business.
-export type TEvent = Pick<TEventOutbox, "id" | "eventType" | "payload" | "occurredAt">;
+// What a consumer sees. Lock and status columns are the outbox's business.
+export type TEvent = Pick<TEventOutbox, "id" | "eventType" | "payload" | "occurredAt"> & {
+ isFinalAttempt: boolean;
+};
export enum EventResultStatus {
Delivered = "delivered",
diff --git a/backend/src/services/secret-access-approval-global-request-bridge/secret-access-approval-global-request-bridge-service.test.ts b/backend/src/services/secret-access-approval-global-request-bridge/secret-access-approval-global-request-bridge-service.test.ts
index 5cfd054503..9aba3ae5b1 100644
--- a/backend/src/services/secret-access-approval-global-request-bridge/secret-access-approval-global-request-bridge-service.test.ts
+++ b/backend/src/services/secret-access-approval-global-request-bridge/secret-access-approval-global-request-bridge-service.test.ts
@@ -1,8 +1,10 @@
import { createMongoAbility, ForbiddenError } from "@casl/ability";
import { describe, expect, test, vi } from "vitest";
+import { ProjectMembershipRole } from "@app/db/schemas";
import { ApprovalStatus } from "@app/ee/services/access-approval-request/access-approval-request-types";
import { ProjectPermissionActions, ProjectPermissionSub } from "@app/ee/services/permission/project-permission";
+import { ForbiddenRequestError } from "@app/lib/errors";
import { ApprovalRequestStatus } from "@app/services/approval-policy/approval-policy-enums";
import { ActorType, AuthMethod } from "@app/services/auth/auth-type";
import { DelegatedApprovalRequestStatus } from "@app/services/delegated-approval/delegated-approval-enums";
@@ -56,22 +58,31 @@ const createService = ({
lockedRequest = pendingRequest as unknown,
delegated = delegatedRequest as unknown,
policy = delegatedPolicy as unknown,
- permission = cannotEditPolicies
+ permission = cannotEditPolicies,
+ memberships = [] as { roles: { role: string }[] }[],
+ projectAccess = "member" as "member" | "none"
} = {}) => {
let current = { ...pendingRequest };
const updateById = vi.fn().mockImplementation(async (_id: string, patch: Partial<typeof pendingRequest>) => {
current = { ...current, ...patch };
return current;
});
+ const findSecretAccessPolicies = vi.fn().mockResolvedValue(policy ? [policy] : []);
+ const getProjectPermission =
+ projectAccess === "member"
+ ? vi.fn().mockResolvedValue({ permission, memberships })
+ : vi
+ .fn()
+ .mockRejectedValue(
+ new ForbiddenRequestError({ message: "You are not a member of this project with ID project-1" })
+ );
const service = secretAccessApprovalGlobalRequestBridgeServiceFactory({
secretAccessApprovalGlobalRequestBridgeDAL: {
findSecretAccessRequestById: vi.fn().mockImplementation(async () => current)
},
- secretAccessApprovalGlobalPolicyBridgeDAL: {
- findSecretAccessPolicies: vi.fn().mockResolvedValue(policy ? [policy] : [])
- },
- permissionService: { getProjectPermission: vi.fn().mockResolvedValue({ permission }) },
+ secretAccessApprovalGlobalPolicyBridgeDAL: { findSecretAccessPolicies },
+ permissionService: { getProjectPermission },
delegatedApprovalService: { getRequestByRequestId: vi.fn().mockResolvedValue(delegated) },
approvalRequestDAL: {
transaction: vi.fn().mockImplementation(async (cb: (tx: string) => unknown) => cb("tx")),
@@ -80,7 +91,7 @@ const createService = ({
}
} as never);
- return { service, updateById };
+ return { service, updateById, findSecretAccessPolicies };
};
const cancelAs = (service: ReturnType<typeof createService>["service"], actorId: string) =>
@@ -174,4 +185,24 @@ describe("secretAccessApprovalGlobalRequestBridge.reviewAccessApprovalRequest",
);
expect(updateById).not.toHaveBeenCalled();
});
+
+ test("answers a non-member with the membership refusal, not the external review message", async () => {
+ const { service, findSecretAccessPolicies } = createService({ projectAccess: "none" });
+
+ await expect(reviewAs(service, ApprovalStatus.APPROVED)).rejects.toThrow(
+ "You are not a member of this project with ID project-1"
+ );
+ expect(findSecretAccessPolicies).not.toHaveBeenCalled();
+ });
+
+ test("refuses a member whose only active role is NoAccess before reading the policy", async () => {
+ const { service, findSecretAccessPolicies } = createService({
+ memberships: [{ roles: [{ role: ProjectMembershipRole.NoAccess }] }]
+ });
+
+ await expect(reviewAs(service, ApprovalStatus.APPROVED)).rejects.toThrow(
+ "You are not authorized to review this request"
+ );
+ expect(findSecretAccessPolicies).not.toHaveBeenCalled();
+ });
});
diff --git a/backend/src/services/secret-access-approval-global-request-bridge/secret-access-approval-global-request-bridge-service.ts b/backend/src/services/secret-access-approval-global-request-bridge/secret-access-approval-global-request-bridge-service.ts
index 557018ed31..1bd5828c15 100644
--- a/backend/src/services/secret-access-approval-global-request-bridge/secret-access-approval-global-request-bridge-service.ts
+++ b/backend/src/services/secret-access-approval-global-request-bridge/secret-access-approval-global-request-bridge-service.ts
@@ -440,23 +440,6 @@ export const secretAccessApprovalGlobalRequestBridgeServiceFactory = ({
actorAuthMethod
}: TReviewSecretAccessApprovalGlobalRequestDTO) => {
const request = await $findRequestById(requestId);
- const delegatedRequest = await delegatedApprovalService.getRequestByRequestId(request.id);
- if (delegatedRequest) {
- throw new BadRequestError({ message: externallyReviewedAccessRequestMessage(requestId) });
- }
- if (!request.policyId) {
- throw new BadRequestError({ message: "The policy associated with this access request has been deleted." });
- }
- const policy = await $findPolicyById(
- request.policyId,
- "The policy associated with this access request has been deleted."
- );
- if (policy.delegatedApproval) {
- throw new BadRequestError({ message: externallyReviewedAccessRequestMessage(requestId) });
- }
- if (!request.requesterId) {
- throw new NotFoundError({ message: "The user who created this access request no longer exists" });
- }
const { memberships } = await permissionService.getProjectPermission({
actor,
@@ -477,6 +460,20 @@ export const secretAccessApprovalGlobalRequestBridgeServiceFactory = ({
throw new ForbiddenRequestError({ message: "You are not authorized to review this request" });
}
+ if (!request.policyId) {
+ throw new BadRequestError({ message: "The policy associated with this access request has been deleted." });
+ }
+ const policy = await $findPolicyById(
+ request.policyId,
+ "The policy associated with this access request has been deleted."
+ );
+ if (policy.delegatedApproval) {
+ throw new BadRequestError({ message: externallyReviewedAccessRequestMessage(requestId) });
+ }
+ if (!request.requesterId) {
+ throw new NotFoundError({ message: "The user who created this access request no longer exists" });
+ }
+
if (request.status !== ApprovalRequestStatus.Pending) {
throw new BadRequestError({ message: "The request has been closed" });
}