Infisical is the open-source platform for securing secrets, certificates, access, and AI Agents.
TypeScript acmeagentsai-agentscertificate-managementdevsecopsenvironment-variableskubernetesopen-sourcepampkipostgresprivate-caprivileged-access-managementsecret-managementsecret-managersecretssecrets-managementsecuritysecurity-toolsvault
feat(secret-rotation): enhance GCP service account key rotation with improved error handling and lock management
Dimitris Staratzise906fbea1b83
diff --git a/backend/src/ee/services/secret-rotation-v2/gcp-service-account-key/gcp-service-account-key-rotation-fns.test.ts b/backend/src/ee/services/secret-rotation-v2/gcp-service-account-key/gcp-service-account-key-rotation-fns.test.tsindex 4b6e0d36d5..de611fa90d 100644--- a/backend/src/ee/services/secret-rotation-v2/gcp-service-account-key/gcp-service-account-key-rotation-fns.test.ts+++ b/backend/src/ee/services/secret-rotation-v2/gcp-service-account-key/gcp-service-account-key-rotation-fns.test.ts@@ -3,10 +3,11 @@ import { beforeEach, describe, expect, it, vi } from "vitest";import { TSecretRotationV2Raw } from "@app/ee/services/secret-rotation-v2/secret-rotation-v2-types";-const { postMock, deleteMock, authorizeMock } = vi.hoisted(() => ({+const { postMock, deleteMock, authorizeMock, delayMock } = vi.hoisted(() => ({postMock: vi.fn<(url: string, body?: unknown, config?: unknown) => Promise<unknown>>(),deleteMock: vi.fn<(url: string, config?: unknown) => Promise<unknown>>(),- authorizeMock: vi.fn<() => Promise<unknown>>()+ authorizeMock: vi.fn<() => Promise<unknown>>(),+ delayMock: vi.fn<(ms: number) => Promise<void>>()}));vi.mock("@app/lib/config/request", () => ({@@ -22,7 +23,7 @@ vi.mock("google-auth-library", () => ({}}));vi.mock("@app/lib/delay", () => ({- delay: vi.fn(async () => {})+ delay: delayMock}));vi.mock("@app/lib/logger", () => ({logger: { error: vi.fn(), info: vi.fn(), warn: vi.fn(), debug: vi.fn() }diff --git a/backend/src/ee/services/secret-rotation-v2/gcp-service-account-key/gcp-service-account-key-rotation-fns.ts b/backend/src/ee/services/secret-rotation-v2/gcp-service-account-key/gcp-service-account-key-rotation-fns.tsindex d473a19769..7bd57031f6 100644--- a/backend/src/ee/services/secret-rotation-v2/gcp-service-account-key/gcp-service-account-key-rotation-fns.ts+++ b/backend/src/ee/services/secret-rotation-v2/gcp-service-account-key/gcp-service-account-key-rotation-fns.ts@@ -23,18 +23,28 @@ import {TGcpServiceAccountKeyRotationWithConnection} from "./gcp-service-account-key-rotation-types";-// GCP documents that a new key can take 60 seconds or more before it is accepted, but the rotation-// holds a 60 second lock, so polling stops well short of that and the attempt fails instead.-const KEY_VERIFICATION_MAX_ATTEMPTS = 12;-const KEY_VERIFICATION_INTERVAL_MS = 3000;+// GCP documents that a new key can take 60 seconds or more before it is accepted. Every check has to+// finish inside this rotation's lock, which SECRET_ROTATION_LOCK_TTL_MS sets to 5 minutes.+const KEY_VERIFICATION_MAX_ATTEMPTS = 5;+const KEY_VERIFICATION_INTERVAL_MS = 60 * 1000;type TGoogleApiError = {- error?: { status?: string; message?: string };+ error?: {+ status?: string;+ message?: string;+ details?: { "@type"?: string; reason?: string }[];+ };};const getGoogleApiError = (error: unknown) =>error instanceof AxiosError ? (error.response?.data as TGoogleApiError | undefined)?.error : undefined;+// The machine-readable cause of a Google API error (e.g. SERVICE_DISABLED), which unlike the message is part of+// the API contract.+const getGoogleApiErrorReason = (error: unknown) =>+ getGoogleApiError(error)?.details?.find((detail) => detail["@type"] === "type.googleapis.com/google.rpc.ErrorInfo")+ ?.reason;+const getErrorMessage = (error: unknown): string => {if (error instanceof AxiosError) return getGoogleApiError(error)?.message ?? error.message;@@ -66,13 +76,13 @@ export const gcpServiceAccountKeyRotationFactory: TRotationFactory<const message = getErrorMessage(error);const status = getErrorStatus(error);- if (status === 403) {- if (message.includes("has not been used in project") || message.includes("is disabled")) {- return new BadRequestError({- message: `Failed to ${action}: the Identity and Access Management (IAM) API is not enabled on the GCP project of the connection's service account. Enable it in the Google Cloud console and try again.`- });- }+ if (getGoogleApiErrorReason(error) === "SERVICE_DISABLED") {+ return new BadRequestError({+ message: `Failed to ${action}: the Identity and Access Management (IAM) API is not enabled on the GCP project of the connection's service account. Enable it in the Google Cloud console and try again.`+ });+ }+ if (status === 403) {return new BadRequestError({message: `Failed to ${action}: the connection's service account is not allowed to manage keys of GCP service account "${serviceAccountEmail}". Grant it the Service Account Key Admin role (roles/iam.serviceAccountKeyAdmin) on that service account and try again.`});@@ -241,6 +251,9 @@ export const gcpServiceAccountKeyRotationFactory: TRotationFactory<> = async (credentialsToRevoke, callback, _activeCredentials, options) => {const accessToken = await getGcpConnectionAuthToken(connection);+ // The new key is created before the previous one is deleted, so a failed create or a key GCP never+ // accepts can't leave the service account with fewer working keys than before. The cost is room for a+ // third key under GCP's 10 key limit while the rotation runs.const newCredentials = await $createKey(accessToken, Boolean(options?.isBackgroundJob));// Delete before committing, so a failure leaves GCP and the rotation agreeing with each other and thediff --git a/backend/src/ee/services/secret-rotation-v2/secret-rotation-v2-maps.ts b/backend/src/ee/services/secret-rotation-v2/secret-rotation-v2-maps.tsindex 0e0a14ce69..1c8bfb6b64 100644--- a/backend/src/ee/services/secret-rotation-v2/secret-rotation-v2-maps.ts+++ b/backend/src/ee/services/secret-rotation-v2/secret-rotation-v2-maps.ts@@ -66,3 +66,11 @@ export const SECRET_ROTATION_CONNECTION_MAP: Record<SecretRotation, AppConnectio[SecretRotation.StripeApiKey]: AppConnection.Stripe,[SecretRotation.GcpServiceAccountKey]: AppConnection.GCP};++export const DEFAULT_SECRET_ROTATION_LOCK_TTL_MS = 60 * 1000;++// For rotations that wait on the provider after creating a credential, so a rotation still running can't+// have its lock expire and another rotation start on the same credentials.+export const SECRET_ROTATION_LOCK_TTL_MS: Partial<Record<SecretRotation, number>> = {+ [SecretRotation.GcpServiceAccountKey]: 5 * 60 * 1000+};diff --git a/backend/src/ee/services/secret-rotation-v2/secret-rotation-v2-service.ts b/backend/src/ee/services/secret-rotation-v2/secret-rotation-v2-service.tsindex 5beddb14f7..e99e62a249 100644--- a/backend/src/ee/services/secret-rotation-v2/secret-rotation-v2-service.ts+++ b/backend/src/ee/services/secret-rotation-v2/secret-rotation-v2-service.ts@@ -38,7 +38,9 @@ import {throwOnImmutableParameterUpdate} from "@app/ee/services/secret-rotation-v2/secret-rotation-v2-fns";import {+ DEFAULT_SECRET_ROTATION_LOCK_TTL_MS,SECRET_ROTATION_CONNECTION_MAP,+ SECRET_ROTATION_LOCK_TTL_MS,SECRET_ROTATION_NAME_MAP} from "@app/ee/services/secret-rotation-v2/secret-rotation-v2-maps";import {@@ -1309,7 +1311,10 @@ export const secretRotationV2ServiceFactory = ({try {try {- lock = await keyStore.acquireLock([KeyStorePrefixes.SecretRotationLock(rotationId)], 60 * 1000);+ lock = await keyStore.acquireLock(+ [KeyStorePrefixes.SecretRotationLock(rotationId)],+ SECRET_ROTATION_LOCK_TTL_MS[type as SecretRotation] ?? DEFAULT_SECRET_ROTATION_LOCK_TTL_MS+ );} catch (e) {throw new InternalServerError({message: "Failed to acquire rotation lock."diff --git a/docs/documentation/platform/secret-rotation/gcp-service-account-key.mdx b/docs/documentation/platform/secret-rotation/gcp-service-account-key.mdxindex 9f6fdd048b..bc4e52cd11 100644--- a/docs/documentation/platform/secret-rotation/gcp-service-account-key.mdx+++ b/docs/documentation/platform/secret-rotation/gcp-service-account-key.mdx@@ -41,7 +41,7 @@ Infisical needs a GCP connection whose service account has the **Service AccountIn the Google Cloud console, go to **IAM & Admin** > **Service Accounts** and select the service account you want to rotate.</Step><Step>- On the **Permissions** tab, select **Grant access**.+ Open the **Principals with access** tab and select **Grant access**.</Step><Step>In **New principals**, enter the email of the connection's service account. Under **Select a role**, choose **Service Account Key Admin**, then select **Save**.@@ -153,7 +153,7 @@ Infisical needs a GCP connection whose service account has the **Service AccountInfisical doesn't change or delete keys that it didn't create. After your applications read the key from Infisical, delete any keys you created by hand so they stop working.</Accordion><Accordion title="Why does a scheduled rotation take longer than a manual one?">- GCP can take a minute or more before it accepts a new key. A scheduled rotation waits until GCP accepts the new key before it updates the secret, so applications don't read a key that GCP still rejects. If GCP doesn't accept the key within about 35 seconds, Infisical deletes the new key and the rotation fails, and the current key stays in the secret.+ GCP can take a minute or more before it accepts a new key. A scheduled rotation waits until GCP accepts the new key before it updates the secret, so applications don't read a key that GCP still rejects. If GCP doesn't accept the key within about 4 minutes, Infisical deletes the new key and the rotation fails. The current key stays in the secret and keeps working.If you rotate the key manually with **Rotate Secret**, or create a new rotation, Infisical updates the secret without waiting. For a short time after that, applications that read the new key can fail to authenticate. After a manual rotation, the previous key keeps working during that time.</Accordion>