Skip to content

Infisical is the open-source platform for securing secrets, certificates, access, and AI Agents.

remove old includeRepos

Matheus Nogueira49aeead35c47
diff --git a/backend/CLAUDE.md b/backend/CLAUDE.md
index 70be28ce39..516ff0c3bf 100644
--- a/backend/CLAUDE.md
+++ b/backend/CLAUDE.md
@@ -1059,7 +1059,7 @@ EE routes register before community routes so they can override/extend endpoints
**Agent Vault**: the same applies to the `agent-vault-*` services and routers; the concept map is [`src/ee/services/agent-vault/CLAUDE.md`](src/ee/services/agent-vault/CLAUDE.md). PAM and Agent Vault are the two **org-scoped products**: one implicit project per org, resolved lazily, whose roles collapse to admin or member. Anything that branches on `ProjectType.PAM` (metering emits, predefined roles, the billable-project count, invite grants) almost always needs an Agent Vault arm too.
-**Secret Scanning** is single-project (see the root `CLAUDE.md`): `secret-scanning-v2-project-bootstrap.ts` creates the project with the org when it has a creator to make admin (like PAM and Agent Vault), `secret-scanning-v2-project-resolver.ts` picks the active project and lazily creates it, with every org admin as project admin, for older orgs and for orgs created from the admin console, whose admins are only invited after the org exists, and routes that take `projectId` read `req.internalSecretScanningProjectId`.
+**Secret Scanning** is single-project (see the root `CLAUDE.md`): `secret-scanning-v2-project-bootstrap.ts` creates the project with the org when it has a creator to make admin (like PAM and Agent Vault), `secret-scanning-v2-project-resolver.ts` picks the active project and lazily creates it, with every org admin as project admin, for older orgs and for orgs created from the admin console, whose admins are only invited after the org exists, and routes that take `projectId` read `req.internalSecretScanningProjectId`. A data source covers the resources it has rows for (added, included at create, or discovered); `config.includeRepos` / `includeProjects` are only read on create and on API updates that change them, so scans, "included" state and GitLab account-scope webhooks all key off rows.
**Dynamic secret lease secrets needed at revoke time go in `dynamic_secret_leases.encryptedLeaseData`**, never in `externalEntityId` or `config` (both plaintext, and `externalEntityId` is returned by every lease route and logged in audit events). A provider opts in by listing the `create()` `data` keys in `persistedLeaseFields`; the lease service encrypts them with the project's SecretManager key and every revoke path passes them back as `metadata.leaseData`. Lease routes return `SanitizedDynamicSecretLeaseSchema` so the column never reaches a response. Secret *config* fields are listed separately in `DYNAMIC_SECRET_SECRET_FIELDS` (`providers/redact.ts`), as dotted paths so nested secrets work (`clientAuth.clientSecret`): reads strip them via `redactStoredInputs`, and an update that omits one keeps the stored value via `restoreOmittedSecretFields`, because the update merge is shallow.
diff --git a/backend/src/ee/routes/v2/secret-scanning-v2-routers/secret-scanning-v2-endpoints.ts b/backend/src/ee/routes/v2/secret-scanning-v2-routers/secret-scanning-v2-endpoints.ts
index 773315f315..6302db9faf 100644
--- a/backend/src/ee/routes/v2/secret-scanning-v2-routers/secret-scanning-v2-endpoints.ts
+++ b/backend/src/ee/routes/v2/secret-scanning-v2-routers/secret-scanning-v2-endpoints.ts
@@ -484,6 +484,149 @@ export const registerSecretScanningEndpoints = <
}
});
+ server.route({
+ method: "POST",
+ url: "/:dataSourceId/resources",
+ config: {
+ rateLimit: writeLimit
+ },
+ schema: {
+ hide: true,
+ operationId: `add${sourceTypeId}DataSourceResource`,
+ tags: [ApiDocsTags.SecretScanning],
+ description: `Add a resource to the specified ${sourceType} Data Source so it is scanned.`,
+ params: z.object({
+ dataSourceId: z.string().uuid().describe(SecretScanningDataSources.ADD_RESOURCE(type).dataSourceId)
+ }),
+ body: z.object({
+ resourceName: z
+ .string()
+ .trim()
+ .min(1)
+ .max(255)
+ .describe(SecretScanningDataSources.ADD_RESOURCE(type).resourceName)
+ }),
+ response: {
+ 200: z.object({ resources: SecretScanningResourceSchema.array() })
+ }
+ },
+ onRequest: verifyAuth([AuthMode.JWT, AuthMode.IDENTITY_ACCESS_TOKEN]),
+ handler: async (req) => {
+ const { dataSourceId } = req.params;
+
+ const { resources, projectId } = await server.services.secretScanningV2.addSecretScanningResource(
+ { type, dataSourceId, resourceName: req.body.resourceName },
+ req.permission
+ );
+
+ await server.services.auditLog.createAuditLog({
+ ...req.auditLogInfo,
+ projectId,
+ event: {
+ type: EventType.SECRET_SCANNING_RESOURCE_ADD,
+ metadata: {
+ type,
+ dataSourceId,
+ resources: resources.map((resource) => ({ resourceId: resource.id, resourceName: resource.name }))
+ }
+ }
+ });
+
+ return { resources };
+ }
+ });
+
+ // not exposed, for UI only. An action rather than a create: the resources come from the provider, not the request.
+ server.route({
+ method: "POST",
+ url: "/:dataSourceId/resources/all",
+ config: {
+ rateLimit: writeLimit
+ },
+ schema: {
+ hide: true,
+ operationId: `addAll${sourceTypeId}DataSourceResources`,
+ tags: [ApiDocsTags.SecretScanning],
+ description: `Add every resource the App Connection can access to the specified ${sourceType} Data Source.`,
+ params: z.object({
+ dataSourceId: z.string().uuid().describe(SecretScanningDataSources.ADD_RESOURCE(type).dataSourceId)
+ }),
+ response: {
+ 200: z.object({ resources: SecretScanningResourceSchema.array() })
+ }
+ },
+ onRequest: verifyAuth([AuthMode.JWT]),
+ handler: async (req) => {
+ const { dataSourceId } = req.params;
+
+ const { resources, projectId } = await server.services.secretScanningV2.addAllSecretScanningResources(
+ { type, dataSourceId },
+ req.permission
+ );
+
+ await server.services.auditLog.createAuditLog({
+ ...req.auditLogInfo,
+ projectId,
+ event: {
+ type: EventType.SECRET_SCANNING_RESOURCE_ADD,
+ metadata: {
+ type,
+ dataSourceId,
+ resources: resources.map((resource) => ({ resourceId: resource.id, resourceName: resource.name }))
+ }
+ }
+ });
+
+ return { resources };
+ }
+ });
+
+ server.route({
+ method: "DELETE",
+ url: "/:dataSourceId/resources/:resourceId",
+ config: {
+ rateLimit: writeLimit
+ },
+ schema: {
+ hide: true,
+ operationId: `remove${sourceTypeId}DataSourceResource`,
+ tags: [ApiDocsTags.SecretScanning],
+ description: `Remove a resource from the specified ${sourceType} Data Source, deleting its scans and findings.`,
+ params: z.object({
+ dataSourceId: z.string().uuid().describe(SecretScanningDataSources.REMOVE_RESOURCE(type).dataSourceId),
+ resourceId: z.string().uuid().describe(SecretScanningDataSources.REMOVE_RESOURCE(type).resourceId)
+ }),
+ response: {
+ 200: z.object({ resource: SecretScanningResourceSchema })
+ }
+ },
+ onRequest: verifyAuth([AuthMode.JWT, AuthMode.IDENTITY_ACCESS_TOKEN]),
+ handler: async (req) => {
+ const { dataSourceId, resourceId } = req.params;
+
+ const { resource, projectId } = await server.services.secretScanningV2.removeSecretScanningResource(
+ { type, dataSourceId, resourceId },
+ req.permission
+ );
+
+ await server.services.auditLog.createAuditLog({
+ ...req.auditLogInfo,
+ projectId,
+ event: {
+ type: EventType.SECRET_SCANNING_RESOURCE_REMOVE,
+ metadata: {
+ type,
+ dataSourceId,
+ resourceId: resource.id,
+ resourceName: resource.name
+ }
+ }
+ });
+
+ return { resource };
+ }
+ });
+
server.route({
method: "GET",
url: "/:dataSourceId/scans",
diff --git a/backend/src/ee/services/audit-log/audit-log-types.ts b/backend/src/ee/services/audit-log/audit-log-types.ts
index 678c6dec63..e96a2817c5 100644
--- a/backend/src/ee/services/audit-log/audit-log-types.ts
+++ b/backend/src/ee/services/audit-log/audit-log-types.ts
@@ -709,6 +709,8 @@ export enum EventType {
SECRET_SCANNING_DATA_SOURCE_TRIGGER_SCAN = "secret-scanning-data-source-trigger-scan",
SECRET_SCANNING_DATA_SOURCE_SCAN = "secret-scanning-data-source-scan",
SECRET_SCANNING_RESOURCE_LIST = "secret-scanning-resource-list",
+ SECRET_SCANNING_RESOURCE_ADD = "secret-scanning-resource-add",
+ SECRET_SCANNING_RESOURCE_REMOVE = "secret-scanning-resource-remove",
SECRET_SCANNING_SCAN_LIST = "secret-scanning-scan-list",
SECRET_SCANNING_FINDING_LIST = "secret-scanning-finding-list",
SECRET_SCANNING_FINDING_UPDATE = "secret-scanning-finding-update",
@@ -5761,6 +5763,25 @@ interface SecretScanningResourceListEvent {
};
}
+interface SecretScanningResourceAddEvent {
+ type: EventType.SECRET_SCANNING_RESOURCE_ADD;
+ metadata: {
+ type: SecretScanningDataSource;
+ dataSourceId: string;
+ resources: { resourceId: string; resourceName: string }[];
+ };
+}
+
+interface SecretScanningResourceRemoveEvent {
+ type: EventType.SECRET_SCANNING_RESOURCE_REMOVE;
+ metadata: {
+ type: SecretScanningDataSource;
+ dataSourceId: string;
+ resourceId: string;
+ resourceName: string;
+ };
+}
+
interface SecretScanningScanListEvent {
type: EventType.SECRET_SCANNING_SCAN_LIST;
metadata: {
@@ -8483,6 +8504,8 @@ export type Event =
| SecretScanningDataSourceTriggerScanEvent
| SecretScanningDataSourceScanEvent
| SecretScanningResourceListEvent
+ | SecretScanningResourceAddEvent
+ | SecretScanningResourceRemoveEvent
| SecretScanningScanListEvent
| SecretScanningFindingListEvent
| SecretScanningFindingUpdateEvent
diff --git a/backend/src/ee/services/secret-scanning-v2/gitlab/gitlab-secret-scanning-factory.ts b/backend/src/ee/services/secret-scanning-v2/gitlab/gitlab-secret-scanning-factory.ts
index eefc71ee8c..03f92c28ae 100644
--- a/backend/src/ee/services/secret-scanning-v2/gitlab/gitlab-secret-scanning-factory.ts
+++ b/backend/src/ee/services/secret-scanning-v2/gitlab/gitlab-secret-scanning-factory.ts
@@ -274,14 +274,13 @@ export const GitLabSecretScanningFactory = ({ appConnectionDAL, kmsService }: TS
const setupWebhook: TSecretScanningFactorySetupWebhook<
TGitLabDataSourceWithConnection,
TGitLabDataSourceCredentials
- > = async ({ dataSource: { id, name, connection, config } }) => {
+ > = async ({ dataSource: { id, name, connection, config }, resourceNames }) => {
if (config.scope === GitLabDataSourceScope.Account) {
- const projectNames = getHookedProjectNames(config);
- if (!projectNames.length) return undefined;
+ if (!resourceNames.length) return undefined;
const accountClient = await getGitLabConnectionClient(connection, appConnectionDAL, kmsService);
const accountToken = alphaNumericNanoId(64);
- const projectHooks = await addProjectHooks(accountClient, projectNames, accountToken, id);
+ const projectHooks = await addProjectHooks(accountClient, resourceNames, accountToken, id);
return { token: accountToken, projectHooks };
}
@@ -297,13 +296,12 @@ export const GitLabSecretScanningFactory = ({ appConnectionDAL, kmsService }: TS
const updateWebhook: TSecretScanningFactoryUpdateWebhook<
TGitLabDataSourceWithConnection,
TGitLabDataSourceCredentials
- > = async ({ dataSource: { id, connection }, config, credentials }) => {
+ > = async ({ dataSource: { id, connection, config }, resourceNames, credentials }) => {
if (config.scope !== GitLabDataSourceScope.Account || !("projectHooks" in credentials)) return undefined;
- const projectNames = getHookedProjectNames(config);
- const kept = credentials.projectHooks.filter((hook) => projectNames.includes(hook.projectName));
- const stale = credentials.projectHooks.filter((hook) => !projectNames.includes(hook.projectName));
- const missing = projectNames.filter((projectName) => !kept.some((hook) => hook.projectName === projectName));
+ const kept = credentials.projectHooks.filter((hook) => resourceNames.includes(hook.projectName));
+ const stale = credentials.projectHooks.filter((hook) => !resourceNames.includes(hook.projectName));
+ const missing = resourceNames.filter((projectName) => !kept.some((hook) => hook.projectName === projectName));
if (!stale.length && !missing.length) return undefined;
diff --git a/backend/src/ee/services/secret-scanning-v2/secret-scanning-v2-queue.ts b/backend/src/ee/services/secret-scanning-v2/secret-scanning-v2-queue.ts
index 818c09d60b..720922e35d 100644
--- a/backend/src/ee/services/secret-scanning-v2/secret-scanning-v2-queue.ts
+++ b/backend/src/ee/services/secret-scanning-v2/secret-scanning-v2-queue.ts
@@ -14,6 +14,7 @@ import {
} from "@app/ee/services/secret-scanning/secret-scanning-queue/secret-scanning-fns";
import {
assertClonedRepositoryWithinSizeLimit,
+ getConfigIncludingAllResources,
parseScanErrorMessage,
planCommitBatches,
repositoryHasCommits,
@@ -174,11 +175,15 @@ export const secretScanningV2QueueServiceFactory = ({
{
resourceExternalId,
triggeredByUserId,
- shouldScanResource
+ shouldScanResource,
+ existingResourcesOnly
}: {
resourceExternalId?: string;
triggeredByUserId?: string;
shouldScanResource?: (resource: Pick<TSecretScanningResourcesInsert, "externalId" | "name">) => boolean;
+ // The data source's resources are the ones added to it, not what its include list matches. The provider is
+ // still listed so a renamed resource is cloned under its current name and a deleted one is skipped.
+ existingResourcesOnly?: boolean;
} = {}
) => {
try {
@@ -189,10 +194,18 @@ export const secretScanningV2QueueServiceFactory = ({
appConnectionDAL
});
- const rawResources = await factory.listRawResources(dataSource);
+ const rawResources = await factory.listRawResources(
+ existingResourcesOnly ? getConfigIncludingAllResources(dataSource) : dataSource
+ );
let filteredRawResources = rawResources;
+ if (existingResourcesOnly) {
+ const existingResources = await secretScanningV2DAL.resources.find({ dataSourceId: dataSource.id });
+ const existingExternalIds = new Set(existingResources.map((resource) => resource.externalId));
+ filteredRawResources = filteredRawResources.filter((resource) => existingExternalIds.has(resource.externalId));
+ }
+
// TODO: should add individual resource fetch to factory
if (resourceExternalId) {
filteredRawResources = rawResources.filter((resource) => resource.externalId === resourceExternalId);
@@ -204,7 +217,9 @@ export const secretScanningV2QueueServiceFactory = ({
if (!filteredRawResources.length) {
throw new BadRequestError({
- message: `${resourceExternalId ? `Resource with "ID" ${resourceExternalId} could not be found.` : "Data source has no resources to scan"}. Ensure your data source config is correct and not filtering out scanning resources.`
+ message: resourceExternalId
+ ? "This resource could not be found. It may have been deleted, or the App Connection may no longer have access to it."
+ : "This data source has no resources to scan. Add one, then try again."
});
}
diff --git a/backend/src/ee/services/secret-scanning-v2/secret-scanning-v2-service.ts b/backend/src/ee/services/secret-scanning-v2/secret-scanning-v2-service.ts
index 79694cfdd2..1fe874b422 100644
--- a/backend/src/ee/services/secret-scanning-v2/secret-scanning-v2-service.ts
+++ b/backend/src/ee/services/secret-scanning-v2/secret-scanning-v2-service.ts
@@ -17,7 +17,11 @@ import {
writeTextToFile
} from "@app/ee/services/secret-scanning/secret-scanning-queue/secret-scanning-fns";
import { githubSecretScanningService } from "@app/ee/services/secret-scanning-v2/github/github-secret-scanning-service";
-import { SecretScanningFindingStatus } from "@app/ee/services/secret-scanning-v2/secret-scanning-v2-enums";
+import { GitLabDataSourceScope } from "@app/ee/services/secret-scanning-v2/gitlab/gitlab-secret-scanning-enums";
+import {
+ SecretScanningDataSource,
+ SecretScanningFindingStatus
+} from "@app/ee/services/secret-scanning-v2/secret-scanning-v2-enums";
import { SECRET_SCANNING_FACTORY_MAP } from "@app/ee/services/secret-scanning-v2/secret-scanning-v2-factory";
import {
doesConfigUpdateIncludeNewResources,
@@ -29,15 +33,19 @@ import {
listSecretScanningDataSourceOptions
} from "@app/ee/services/secret-scanning-v2/secret-scanning-v2-fns";
import {
+ AUTO_SYNC_DESCRIPTION_HELPER,
SECRET_SCANNING_DATA_SOURCE_CONNECTION_MAP,
SECRET_SCANNING_DATA_SOURCE_NAME_MAP
} from "@app/ee/services/secret-scanning-v2/secret-scanning-v2-maps";
import {
+ TAddAllSecretScanningResourcesDTO,
+ TAddSecretScanningResourceDTO,
TCreateSecretScanningDataSourceDTO,
TDeleteSecretScanningDataSourceDTO,
TFindSecretScanningDataSourceByIdDTO,
TFindSecretScanningDataSourceByNameDTO,
TListSecretScanningDataSourcesByProjectId,
+ TRemoveSecretScanningResourceDTO,
TSecretScanningDataSource,
TSecretScanningDataSourceCredentials,
TSecretScanningDataSourceInput,
@@ -234,6 +242,48 @@ export const secretScanningV2ServiceFactory = ({
return dataSource as TSecretScanningDataSource;
};
+ // Providers that hook each resource (GitLab account scope) keep their webhooks in line with the data source's
+ // resources. Returns undefined when nothing changes.
+ const prepareResourceWebhooks = async (
+ dataSource: TSecretScanningDataSourceWithConnection,
+ encryptedCredentials: Buffer | null | undefined,
+ resourceNames: string[]
+ ) => {
+ const factory = SECRET_SCANNING_FACTORY_MAP[dataSource.type]({ appConnectionDAL, kmsService });
+ if (!factory.updateWebhook || !isDataSourceWebhookRequired(dataSource)) return undefined;
+
+ const { encryptor, decryptor } = await kmsService.createCipherPairWithDataKey({
+ type: KmsDataKey.SecretManager,
+ projectId: dataSource.projectId
+ });
+
+ if (!encryptedCredentials) {
+ const credentials = await factory.setupWebhook({ dataSource, resourceNames });
+ if (!credentials) return undefined;
+
+ return {
+ encryptedCredentials: encryptor({ plainText: Buffer.from(JSON.stringify(credentials)) }).cipherTextBlob,
+ finalize: async () => {},
+ revert: () => factory.teardown({ dataSource, credentials })
+ };
+ }
+
+ const update = await factory.updateWebhook({
+ dataSource,
+ resourceNames,
+ credentials: JSON.parse(
+ decryptor({ cipherTextBlob: encryptedCredentials }).toString()
+ ) as TSecretScanningDataSourceCredentials
+ });
+ if (!update) return undefined;
+
+ return {
+ encryptedCredentials: encryptor({ plainText: Buffer.from(JSON.stringify(update.credentials)) }).cipherTextBlob,
+ finalize: update.finalize,
+ revert: update.revert
+ };
+ };
+
const createSecretScanningDataSource = async (
{ isAutoScanEnabled, ...dto }: TCreateSecretScanningDataSourceDTO,
actor: OrgServiceActor
@@ -338,23 +388,33 @@ export const secretScanningV2ServiceFactory = ({
);
// A data source created with an empty include list has nothing to scan yet; queueing would only fail.
- if (
- payload.isHistoricalScanEnabled &&
- doesDataSourceIncludeResources(payload as TSecretScanningDataSourceTypeAndConfig)
- ) {
+ if (doesDataSourceIncludeResources(payload as TSecretScanningDataSourceTypeAndConfig)) {
+ const createdDataSourceWithConnection = {
+ ...createdDataSource,
+ connection
+ } as TSecretScanningDataSourceWithConnection;
+
try {
- await secretScanningV2Queue.queueDataSourceFullScan(
- {
- ...createdDataSource,
- connection
- } as TSecretScanningDataSourceWithConnection,
- { triggeredByUserId: actor.type === ActorType.USER ? actor.id : undefined }
- );
+ // The include list is only read here: from now on the data source covers the resources it has rows for.
+ if (payload.isHistoricalScanEnabled) {
+ await secretScanningV2Queue.queueDataSourceFullScan(createdDataSourceWithConnection, {
+ triggeredByUserId: actor.type === ActorType.USER ? actor.id : undefined
+ });
+ } else {
+ const rawResources = await factory.listRawResources(createdDataSourceWithConnection);
+
+ if (rawResources.length) {
+ await secretScanningV2DAL.resources.upsert(
+ rawResources.map((resource) => ({ ...resource, dataSourceId: createdDataSource.id })),
+ ["externalId", "dataSourceId"]
+ );
+ }
+ }
} catch (error) {
- // Deliberately non-fatal — a scan that can't be queued shouldn't block creating the data source.
+ // Deliberately non-fatal — the data source is saved, and its resources can still be added from it.
logger.error(
error,
- `secretScanningV2Service: Failed to queue initial full scan on data source creation [dataSourceId=${createdDataSource.id}] [projectId=${payload.projectId}]`
+ `secretScanningV2Service: Failed to set up resources on data source creation [dataSourceId=${createdDataSource.id}] [projectId=${payload.projectId}]`
);
}
}
@@ -449,7 +509,7 @@ export const secretScanningV2ServiceFactory = ({
}
const dataSourceWithConnection = { ...dataSource, connection } as TSecretScanningDataSourceWithConnection;
- // Hooks are placed from the include list, so creating them has to see the config this update saves.
+ // A webhook created here has to see the config this update saves.
const nextDataSourceWithConnection = (
payload.config ? { ...dataSourceWithConnection, config: payload.config } : dataSourceWithConnection
) as TSecretScanningDataSourceWithConnection;
@@ -466,10 +526,13 @@ export const secretScanningV2ServiceFactory = ({
// Stored credentials are what tells us a webhook exists, so they are added or cleared alongside it.
let createdCredentials: TSecretScanningDataSourceCredentials;
let encryptedCredentials: Buffer | null | undefined;
- let webhookUpdate: Awaited<ReturnType<NonNullable<typeof factory.updateWebhook>>>;
if (isWebhookRequired && !dataSource.encryptedCredentials) {
- createdCredentials = await factory.setupWebhook({ dataSource: nextDataSourceWithConnection });
+ const resources = await secretScanningV2DAL.resources.find({ dataSourceId });
+ createdCredentials = await factory.setupWebhook({
+ dataSource: nextDataSourceWithConnection,
+ resourceNames: resources.map((resource) => resource.name)
+ });
if (createdCredentials) {
encryptedCredentials = encryptor({ plainText: Buffer.from(JSON.stringify(createdCredentials)) }).cipherTextBlob;
@@ -482,20 +545,6 @@ export const secretScanningV2ServiceFactory = ({
) as TSecretScanningDataSourceCredentials
});
encryptedCredentials = null;
- } else if (isWebhookRequired && dataSource.encryptedCredentials && payload.config && factory.updateWebhook) {
- webhookUpdate = await factory.updateWebhook({
- dataSource: dataSourceWithConnection,
- config: nextDataSourceWithConnection.config,
- credentials: JSON.parse(
- decryptor({ cipherTextBlob: dataSource.encryptedCredentials }).toString()
- ) as TSecretScanningDataSourceCredentials
- });
-
- if (webhookUpdate) {
- encryptedCredentials = encryptor({
- plainText: Buffer.from(JSON.stringify(webhookUpdate.credentials))
- }).cipherTextBlob;
- }
}
let updatedDataSource: TSecretScanningDataSource;
@@ -510,8 +559,6 @@ export const secretScanningV2ServiceFactory = ({
await factory.teardown({ dataSource: nextDataSourceWithConnection, credentials: createdCredentials });
}
- await webhookUpdate?.revert();
-
if (err instanceof DatabaseError && (err.error as { code: string })?.code === DatabaseErrorCode.UniqueViolation) {
throw new BadRequestError({
message: `A Secret Scanning Data Source with the name "${payload.name}" already exists for the project with ID "${dataSource.projectId}"`
@@ -521,8 +568,6 @@ export const secretScanningV2ServiceFactory = ({
throw err;
}
- await webhookUpdate?.finalize();
-
if (
doesConfigUpdateIncludeNewResources(
dataSource as TSecretScanningDataSourceTypeAndConfig,
@@ -556,6 +601,26 @@ export const secretScanningV2ServiceFactory = ({
);
}
}
+
+ const resources = await secretScanningV2DAL.resources.find({ dataSourceId });
+ const savedDataSource = await secretScanningV2DAL.dataSources.findById(dataSourceId);
+ const webhook = await prepareResourceWebhooks(
+ updatedDataSourceWithConnection,
+ savedDataSource?.encryptedCredentials,
+ resources.map((resource) => resource.name)
+ );
+
+ if (webhook) {
+ try {
+ await secretScanningV2DAL.dataSources.updateById(dataSourceId, {
+ encryptedCredentials: webhook.encryptedCredentials
+ });
+ } catch (err) {
+ await webhook.revert();
+ throw err;
+ }
+ await webhook.finalize();
+ }
} catch (error) {
// Deliberately non-fatal: the include list is saved, and the resources can still be scanned manually.
logger.error(
@@ -703,12 +768,253 @@ export const secretScanningV2ServiceFactory = ({
...dataSource,
connection
} as TSecretScanningDataSourceWithConnection,
- { resourceExternalId, triggeredByUserId: actor.type === ActorType.USER ? actor.id : undefined }
+ {
+ resourceExternalId,
+ triggeredByUserId: actor.type === ActorType.USER ? actor.id : undefined,
+ existingResourcesOnly: true
+ }
);
return dataSource as TSecretScanningDataSource;
};
+ // Adds one resource by name, or every resource the connection can access when no name is given.
+ const $addSecretScanningResources = async (
+ { type, dataSourceId, resourceName }: TAddAllSecretScanningResourcesDTO & { resourceName?: string },
+ actor: OrgServiceActor
+ ) => {
+ const plan = await licenseService.getPlan(actor.orgId);
+
+ if (!plan.secretScanning)
+ throw new BadRequestError({
+ message:
+ "Failed to add Secret Scanning Resources due to plan restriction. Upgrade plan to enable Secret Scanning."
+ });
+
+ const dataSource = await secretScanningV2DAL.dataSources.findById(dataSourceId);
+ const sourceName = SECRET_SCANNING_DATA_SOURCE_NAME_MAP[type];
+
+ if (!dataSource)
+ throw new NotFoundError({
+ message: `Could not find ${sourceName} Data Source with ID "${dataSourceId}"`
+ });
+
+ const { permission } = await permissionService.getProjectPermission({
+ actor: actor.type,
+ actorId: actor.id,
+ actorAuthMethod: actor.authMethod,
+ actorOrgId: actor.orgId,
+ actionProjectType: ActionProjectType.SecretScanning,
+ projectId: dataSource.projectId
+ });
+
+ ForbiddenError.from(permission).throwUnlessCan(
+ ProjectPermissionSecretScanningDataSourceActions.Edit,
+ ProjectPermissionSub.SecretScanningDataSources
+ );
+
+ if (type !== dataSource.type)
+ throw new BadRequestError({
+ message: `Secret Scanning Data Source with ID "${dataSourceId}" is not configured for ${sourceName}`
+ });
+
+ if (dataSource.isDisconnected)
+ throw new BadRequestError({
+ message: `Data Source "${dataSource.name}" is disconnected from ${sourceName}, so resources can't be added to it. Delete it and connect ${sourceName} again.`
+ });
+
+ if (
+ dataSource.type === SecretScanningDataSource.GitLab &&
+ (dataSource.config as { scope?: string }).scope === GitLabDataSourceScope.Project
+ )
+ throw new BadRequestError({
+ message: `Data Source "${dataSource.name}" scans a single GitLab project, so projects can't be added to it.`
+ });
+
+ // GitLab has no account-level webhook, so adding everything would mean a webhook on every project the
+ // connection can access.
+ if (
+ resourceName === undefined &&
+ dataSource.type === SecretScanningDataSource.GitLab &&
+ (dataSource.config as { scope?: string }).scope === GitLabDataSourceScope.Account
+ )
+ throw new BadRequestError({
+ message: `Data Source "${dataSource.name}" isn't scoped to a GitLab group, so projects have to be added one at a time.`
+ });
+
+ let connection: TAppConnection | null = null;
+ if (dataSource.connectionId) {
+ connection = await appConnectionService.validateAppConnectionUsageById(
+ SECRET_SCANNING_DATA_SOURCE_CONNECTION_MAP[dataSource.type],
+ { connectionId: dataSource.connectionId, projectId: dataSource.projectId },
+ actor
+ );
+ }
+
+ const dataSourceWithConnection = { ...dataSource, connection } as TSecretScanningDataSourceWithConnection;
+ const factory = SECRET_SCANNING_FACTORY_MAP[dataSource.type]({ appConnectionDAL, kmsService });
+ const { noun } = AUTO_SYNC_DESCRIPTION_HELPER[dataSource.type];
+
+ let availableResources: Awaited<ReturnType<typeof factory.listRawResources>>;
+ try {
+ availableResources = await factory.listRawResources(getConfigIncludingAllResources(dataSourceWithConnection));
+ } catch (error) {
+ if (error instanceof BadRequestError) throw error;
+
+ logger.error(
+ error,
+ `secretScanningV2Service: Failed to list resources to add [dataSourceId=${dataSourceId}] [projectId=${dataSource.projectId}]`
+ );
+ throw new BadRequestError({
+ message: `Could not list ${noun} from ${sourceName} for Data Source "${dataSource.name}". Check that its App Connection still has access, then try again.`
+ });
+ }
+
+ let requestedResources = availableResources;
+
+ if (resourceName !== undefined) {
+ const resource = availableResources.find((available) => available.name === resourceName);
+
+ if (!resource)
+ throw new BadRequestError({
+ message: `The App Connection for Data Source "${dataSource.name}" can't access "${resourceName}". Check the name, or give the connection access in ${sourceName}.`
+ });
+
+ requestedResources = [resource];
+ }
+
+ const existingResources = await secretScanningV2DAL.resources.find({ dataSourceId });
+ const existingExternalIds = new Set(existingResources.map((resource) => resource.externalId));
+ const newResources = requestedResources.filter((resource) => !existingExternalIds.has(resource.externalId));
+
+ if (!newResources.length) return { resources: [], projectId: dataSource.projectId };
+
+ const webhook = await prepareResourceWebhooks(dataSourceWithConnection, dataSource.encryptedCredentials, [
+ ...existingResources.map((resource) => resource.name),
+ ...newResources.map((resource) => resource.name)
+ ]);
+
+ let addedResources: Awaited<ReturnType<typeof secretScanningV2DAL.resources.upsert>>;
+ try {
+ addedResources = await secretScanningV2DAL.resources.transaction(async (tx) => {
+ if (webhook) {
+ await secretScanningV2DAL.dataSources.updateById(
+ dataSourceId,
+ { encryptedCredentials: webhook.encryptedCredentials },
+ tx
+ );
+ }
+
+ return secretScanningV2DAL.resources.upsert(
+ newResources.map((resource) => ({ ...resource, dataSourceId })),
+ ["externalId", "dataSourceId"],
+ tx
+ );
+ });
+ } catch (err) {
+ await webhook?.revert();
+ throw err;
+ }
+
+ await webhook?.finalize();
+
+ if (dataSource.isHistoricalScanEnabled) {
+ const newExternalIds = new Set(newResources.map((resource) => resource.externalId));
+
+ try {
+ await secretScanningV2Queue.queueDataSourceFullScan(dataSourceWithConnection, {
+ triggeredByUserId: actor.type === ActorType.USER ? actor.id : undefined,
+ existingResourcesOnly: true,
+ shouldScanResource: (resource) => newExternalIds.has(resource.externalId)
+ });
+ } catch (error) {
+ // Deliberately non-fatal: the resources are added, and can be scanned from the data source.
+ logger.error(
+ error,
+ `secretScanningV2Service: Failed to queue historical scan of added resources [dataSourceId=${dataSourceId}] [projectId=${dataSource.projectId}]`
+ );
+ }
+ }
+
+ return { resources: addedResources, projectId: dataSource.projectId };
+ };
+
+ const addSecretScanningResource = ({ resourceName, ...dto }: TAddSecretScanningResourceDTO, actor: OrgServiceActor) =>
+ $addSecretScanningResources({ ...dto, resourceName }, actor);
+
+ const addAllSecretScanningResources = (dto: TAddAllSecretScanningResourcesDTO, actor: OrgServiceActor) =>
+ $addSecretScanningResources(dto, actor);
+
+ // Not gated on the plan: a downgraded org must still be able to remove what it added.
+ const removeSecretScanningResource = async (
+ { type, dataSourceId, resourceId }: TRemoveSecretScanningResourceDTO,
+ actor: OrgServiceActor
+ ) => {
+ const dataSource = await secretScanningV2DAL.dataSources.findById(dataSourceId);
+ const sourceName = SECRET_SCANNING_DATA_SOURCE_NAME_MAP[type];
+
+ if (!dataSource)
+ throw new NotFoundError({
+ message: `Could not find ${sourceName} Data Source with ID "${dataSourceId}"`
+ });
+
+ const { permission } = await permissionService.getProjectPermission({
+ actor: actor.type,
+ actorId: actor.id,
+ actorAuthMethod: actor.authMethod,
+ actorOrgId: actor.orgId,
+ actionProjectType: ActionProjectType.SecretScanning,
+ projectId: dataSource.projectId
+ });
+
+ ForbiddenError.from(permission).throwUnlessCan(
+ ProjectPermissionSecretScanningDataSourceActions.Edit,
+ ProjectPermissionSub.SecretScanningDataSources
+ );
+
+ if (type !== dataSource.type)
+ throw new BadRequestError({
+ message: `Secret Scanning Data Source with ID "${dataSourceId}" is not configured for ${sourceName}`
+ });
+
+ const resource = await secretScanningV2DAL.resources.findOne({ id: resourceId, dataSourceId });
+
+ if (!resource)
+ throw new NotFoundError({
+ message: `Could not find Secret Scanning Resource with ID "${resourceId}" for Data Source with ID "${dataSourceId}"`
+ });
+
+ // A disconnected data source can't reach the provider, so its hook is left behind rather than blocking removal.
+ let webhook: Awaited<ReturnType<typeof prepareResourceWebhooks>>;
+ if (dataSource.connection && !dataSource.isDisconnected) {
+ const connection = await decryptAppConnection(dataSource.connection, kmsService);
+ const remainingResources = await secretScanningV2DAL.resources.find({ dataSourceId });
+
+ webhook = await prepareResourceWebhooks(
+ { ...dataSource, connection } as TSecretScanningDataSourceWithConnection,
+ dataSource.encryptedCredentials,
+ remainingResources.filter((remaining) => remaining.id !== resourceId).map((remaining) => remaining.name)
+ );
+ }
+
+ await secretScanningV2DAL.resources.transaction(async (tx) => {
+ if (webhook) {
+ await secretScanningV2DAL.dataSources.updateById(
+ dataSourceId,
+ { encryptedCredentials: webhook.encryptedCredentials },
+ tx
+ );
+ }
+
+ // Cascades to the resource's scans and findings.
+ await secretScanningV2DAL.resources.deleteById(resourceId, tx);
+ });
+
+ await webhook?.finalize();
+
+ return { resource, projectId: dataSource.projectId };
+ };
+
const listSecretScanningResourcesByDataSourceId = async (
{ type, dataSourceId }: TFindSecretScanningDataSourceByIdDTO,
actor: OrgServiceActor
@@ -820,10 +1126,13 @@ export const secretScanningV2ServiceFactory = ({
});
}
+ const existingResources = await secretScanningV2DAL.resources.find({ dataSourceId });
+ const existingExternalIds = new Set(existingResources.map((resource) => resource.externalId));
+
const resources = rawResources
.map((resource) => ({
...resource,
- isIncluded: isResourceIncludedByConfig(dataSource as TSecretScanningDataSourceTypeAndConfig, resource.name)
+ isIncluded: existingExternalIds.has(resource.externalId)
}))
.sort((a, b) => a.name.localeCompare(b.name));
@@ -1174,6 +1483,9 @@ export const secretScanningV2ServiceFactory = ({
updateSecretScanningDataSource,
deleteSecretScanningDataSource,
triggerSecretScanningDataSourceScan,
+ addSecretScanningResource,
+ addAllSecretScanningResources,
+ removeSecretScanningResource,
listSecretScanningResourcesByDataSourceId,
listSecretScanningAvailableResourcesByDataSourceId,
listSecretScanningScansByDataSourceId,
diff --git a/backend/src/ee/services/secret-scanning-v2/secret-scanning-v2-types.ts b/backend/src/ee/services/secret-scanning-v2/secret-scanning-v2-types.ts
index 6b78f852b6..244b0b93f7 100644
--- a/backend/src/ee/services/secret-scanning-v2/secret-scanning-v2-types.ts
+++ b/backend/src/ee/services/secret-scanning-v2/secret-scanning-v2-types.ts
@@ -138,6 +138,23 @@ export type TTriggerSecretScanningDataSourceDTO = {
resourceId?: string;
};
+export type TAddSecretScanningResourceDTO = {
+ type: SecretScanningDataSource;
+ dataSourceId: string;
+ resourceName: string;
+};
+
+export type TAddAllSecretScanningResourcesDTO = {
+ type: SecretScanningDataSource;
+ dataSourceId: string;
+};
+
+export type TRemoveSecretScanningResourceDTO = {
+ type: SecretScanningDataSource;
+ dataSourceId: string;
+ resourceId: string;
+};
+
export type TQueueSecretScanningDataSourceFullScan = {
dataSourceId: string;
resourceId: string;
@@ -233,14 +250,15 @@ export type TSecretScanningFactoryTeardown<
export type TSecretScanningFactorySetupWebhook<
T extends TSecretScanningDataSourceWithConnection,
C extends TSecretScanningDataSourceCredentials = undefined
-> = (params: { dataSource: T }) => Promise<C | undefined>;
+> = (params: { dataSource: T; resourceNames: string[] }) => Promise<C | undefined>;
-// Brings existing webhooks in line with a new config. Hooks it adds exist before the data source is saved, so
-// revert removes them if the save fails; hooks it drops are only removed by finalize, after the save.
+// Brings existing webhooks in line with the resources a data source covers, for providers that hook each resource.
+// Hooks it adds exist before the change is saved, so revert removes them if the save fails; hooks it drops are
+// only removed by finalize, after the save.
export type TSecretScanningFactoryUpdateWebhook<
T extends TSecretScanningDataSourceWithConnection,
C extends TSecretScanningDataSourceCredentials = undefined
-> = (params: { dataSource: T; config: T["config"]; credentials: C }) => Promise<
+> = (params: { dataSource: T; resourceNames: string[]; credentials: C }) => Promise<
| {
credentials: C;
finalize: () => Promise<void>;
diff --git a/backend/src/lib/api-docs/constants.ts b/backend/src/lib/api-docs/constants.ts
index 866c76e961..e26b5eb138 100644
--- a/backend/src/lib/api-docs/constants.ts
+++ b/backend/src/lib/api-docs/constants.ts
@@ -3994,6 +3994,14 @@ export const SecretScanningDataSources = {
LIST_SCANS: (type: SecretScanningDataSource) => ({
dataSourceId: `The ID of the ${SECRET_SCANNING_DATA_SOURCE_NAME_MAP[type]} Data Source to list scans for.`
}),
+ ADD_RESOURCE: (type: SecretScanningDataSource) => ({
+ dataSourceId: `The ID of the ${SECRET_SCANNING_DATA_SOURCE_NAME_MAP[type]} Data Source to add resources to.`,
+ resourceName: `The full name of the ${SECRET_SCANNING_DATA_SOURCE_NAME_MAP[type]} resource to scan, as the App Connection lists it. A resource already added is ignored.`
+ }),
+ REMOVE_RESOURCE: (type: SecretScanningDataSource) => ({
+ dataSourceId: `The ID of the ${SECRET_SCANNING_DATA_SOURCE_NAME_MAP[type]} Data Source to remove the resource from.`,
+ resourceId: "The ID of the resource to remove. Its scans and findings are deleted with it."
+ }),
CONFIG: {
GITHUB: {
includeRepos:
diff --git a/frontend/src/components/secret-scanning/forms/CreateSecretScanningDataSourceForm.tsx b/frontend/src/components/secret-scanning/forms/CreateSecretScanningDataSourceForm.tsx
index 55afe47381..176ac816e6 100644
--- a/frontend/src/components/secret-scanning/forms/CreateSecretScanningDataSourceForm.tsx
+++ b/frontend/src/components/secret-scanning/forms/CreateSecretScanningDataSourceForm.tsx
@@ -1,5 +1,5 @@
import { useEffect, useState } from "react";
-import { FieldPath, FormProvider, useForm, useWatch } from "react-hook-form";
+import { DefaultValues, FieldPath, FormProvider, useForm, useWatch } from "react-hook-form";
import { zodResolver } from "@hookform/resolvers/zod";
import { createNotification } from "@app/components/notifications";
@@ -68,7 +68,7 @@ const getDefaultValues = (type: SecretScanningDataSource) =>
isScanNewCommitsEnabled: true,
isDiscoveryEnabled: doesDataSourceSupportDiscovery({ type, config: getDefaultConfig(type) }),
config: getDefaultConfig(type)
- }) as Partial<TSecretScanningDataSourceForm>;
+ }) as DefaultValues<TSecretScanningDataSourceForm>;
const CONNECTION_STEP_FIELDS: Record<SecretScanningDataSource, TFormField[]> = {
[SecretScanningDataSource.GitHub]: ["connection"],
@@ -212,7 +212,10 @@ export const CreateSecretScanningDataSourceForm = ({
const formMethods = useForm<TSecretScanningDataSourceForm>({
resolver: zodResolver(SecretScanningDataSourceSchema),
- defaultValues: { ...getDefaultValues(initialType), ...initialFormData },
+ defaultValues: {
+ ...getDefaultValues(initialType),
+ ...initialFormData
+ } as DefaultValues<TSecretScanningDataSourceForm>,
// Step validation uses trigger(), which does not put the form into its submitted state.
// Validate changes from the start so a field's error clears as soon as it is corrected.
mode: "onChange",
diff --git a/frontend/src/helpers/secretScanningV2.ts b/frontend/src/helpers/secretScanningV2.ts
index 736fa1e43f..2deafb8420 100644
--- a/frontend/src/helpers/secretScanningV2.ts
+++ b/frontend/src/helpers/secretScanningV2.ts
@@ -52,35 +52,14 @@ export const doesDataSourceSupportDiscovery = ({
(config as { scope?: string } | undefined)?.scope !== GitLabDataSourceScope.Group
);
-// a project-scoped GitLab data source scans exactly one project, so it has no include list to add to
+// a project-scoped GitLab data source scans exactly one project, so projects can't be added or removed
export const canAddSecretScanningResources = (dataSource: TSecretScanningDataSource) =>
!(
dataSource.type === SecretScanningDataSource.GitLab &&
dataSource.config.scope === GitLabDataSourceScope.Project
);
-export const getConfigWithIncludedResource = (
- dataSource: TSecretScanningDataSource,
- resourceName: string
-): TSecretScanningDataSource["config"] | null => {
- const append = (names: string[]) => [...names.filter((name) => name !== "*"), resourceName];
-
- switch (dataSource.type) {
- case SecretScanningDataSource.Bitbucket:
- case SecretScanningDataSource.GitHub:
- return { ...dataSource.config, includeRepos: append(dataSource.config.includeRepos) };
- case SecretScanningDataSource.GitLab:
- if (dataSource.config.scope === GitLabDataSourceScope.Project) return null;
- return {
- ...dataSource.config,
- includeProjects: append(dataSource.config.includeProjects)
- };
- default:
- return null;
- }
-};
-
-// mirrors what the backend does with resources added to the include list
+// mirrors what the backend does with resources added to a data source
export const getAddedResourceScanDescription = (
dataSource: Pick<TSecretScanningDataSource, "isHistoricalScanEnabled" | "isScanNewCommitsEnabled">
) => {
@@ -89,61 +68,6 @@ export const getAddedResourceScanDescription = (
return "Nothing will be scanned until Historical Scan or Scan New Commits is turned on in Settings.";
};
-export const getConfigWithAllResourcesIncluded = (
- dataSource: TSecretScanningDataSource
-): TSecretScanningDataSource["config"] | null => {
- switch (dataSource.type) {
- case SecretScanningDataSource.Bitbucket:
- case SecretScanningDataSource.GitHub:
- return { ...dataSource.config, includeRepos: ["*"] };
- case SecretScanningDataSource.GitLab:
- if (dataSource.config.scope === GitLabDataSourceScope.Project) return null;
- return { ...dataSource.config, includeProjects: ["*"] };
- default:
- return null;
- }
-};
-
-export const areAllSecretScanningResourcesIncluded = (dataSource: TSecretScanningDataSource) => {
- switch (dataSource.type) {
- case SecretScanningDataSource.Bitbucket:
- case SecretScanningDataSource.GitHub:
- return dataSource.config.includeRepos.includes("*");
- case SecretScanningDataSource.GitLab:
- return (
- dataSource.config.scope === GitLabDataSourceScope.Project ||
- dataSource.config.includeProjects.includes("*")
- );
- default:
- return false;
- }
-};
-
-export const isSecretScanningResourceIncluded = (
- dataSource: TSecretScanningDataSource,
- resourceName: string
-) => {
- switch (dataSource.type) {
- case SecretScanningDataSource.Bitbucket:
- case SecretScanningDataSource.GitHub:
- return (
- dataSource.config.includeRepos.includes("*") ||
- dataSource.config.includeRepos.includes(resourceName)
- );
- case SecretScanningDataSource.GitLab:
- if (dataSource.config.scope === GitLabDataSourceScope.Project) return true;
-
- return (
- dataSource.config.includeProjects.includes("*") ||
- dataSource.config.includeProjects.includes(resourceName)
- );
- default:
- throw new Error(
- `Unhandled Data Source Type: ${(dataSource as TSecretScanningDataSource).type}`
- );
- }
-};
-
export const RESOURCE_DESCRIPTION_HELPER: Record<
SecretScanningDataSource,
{
diff --git a/frontend/src/hooks/api/auditLogs/constants.tsx b/frontend/src/hooks/api/auditLogs/constants.tsx
index 8b01fbd474..55ef8a81e8 100644
--- a/frontend/src/hooks/api/auditLogs/constants.tsx
+++ b/frontend/src/hooks/api/auditLogs/constants.tsx
@@ -267,6 +267,8 @@ export const eventToNameMap: { [K in EventType]: string } = {
[EventType.SECRET_SCANNING_DATA_SOURCE_TRIGGER_SCAN]: "Trigger Secret Scanning Data Source Scan",
[EventType.SECRET_SCANNING_DATA_SOURCE_SCAN]: "Secret Scanning Data Source Scan",
[EventType.SECRET_SCANNING_RESOURCE_LIST]: "List Secret Scanning Resources",
+ [EventType.SECRET_SCANNING_RESOURCE_ADD]: "Add Secret Scanning Resources",
+ [EventType.SECRET_SCANNING_RESOURCE_REMOVE]: "Remove Secret Scanning Resource",
[EventType.SECRET_SCANNING_SCAN_LIST]: "List Secret Scanning Scans",
[EventType.SECRET_SCANNING_FINDING_LIST]: "List Secret Scanning Findings",
[EventType.SECRET_SCANNING_FINDING_UPDATE]: "Update Secret Scanning Finding Status",
diff --git a/frontend/src/hooks/api/auditLogs/enums.tsx b/frontend/src/hooks/api/auditLogs/enums.tsx
index 0915b412ab..14486c3bd5 100644
--- a/frontend/src/hooks/api/auditLogs/enums.tsx
+++ b/frontend/src/hooks/api/auditLogs/enums.tsx
@@ -262,6 +262,8 @@ export enum EventType {
SECRET_SCANNING_DATA_SOURCE_TRIGGER_SCAN = "secret-scanning-data-source-trigger-scan",
SECRET_SCANNING_DATA_SOURCE_SCAN = "secret-scanning-data-source-scan",
SECRET_SCANNING_RESOURCE_LIST = "secret-scanning-resource-list",
+ SECRET_SCANNING_RESOURCE_ADD = "secret-scanning-resource-add",
+ SECRET_SCANNING_RESOURCE_REMOVE = "secret-scanning-resource-remove",
SECRET_SCANNING_SCAN_LIST = "secret-scanning-scan-list",
SECRET_SCANNING_FINDING_LIST = "secret-scanning-finding-list",
SECRET_SCANNING_FINDING_UPDATE = "secret-scanning-finding-update",
diff --git a/frontend/src/hooks/api/secretScanningV2/mutations.tsx b/frontend/src/hooks/api/secretScanningV2/mutations.tsx
index beb1cffa08..51031a2179 100644
--- a/frontend/src/hooks/api/secretScanningV2/mutations.tsx
+++ b/frontend/src/hooks/api/secretScanningV2/mutations.tsx
@@ -4,9 +4,12 @@ import { apiRequest } from "@app/config/request";
import { secretScanningV2Keys } from "./queries";
import {
+ TAddAllSecretScanningResourcesDTO,
+ TAddSecretScanningResourceDTO,
TCreateSecretScanningDataSourceDTO,
TDeleteSecretScanningDataSourceDTO,
TGetSecretScanningConfigResponse,
+ TRemoveSecretScanningResourceDTO,
TSecretScanningConfig,
TSecretScanningDataSourceResponse,
TSecretScanningFindingResponse,
@@ -115,6 +118,86 @@ export const useTriggerSecretScanningDataSource = () => {
});
};
+const useInvalidateAddedResources = () => {
+ const queryClient = useQueryClient();
+ return ({ projectId, dataSourceId }: { projectId: string; dataSourceId: string }) => {
+ queryClient.invalidateQueries({
+ queryKey: secretScanningV2Keys.listDataSources(projectId)
+ });
+ queryClient.invalidateQueries({
+ queryKey: secretScanningV2Keys.dataSourceById(dataSourceId)
+ });
+ queryClient.invalidateQueries({
+ queryKey: secretScanningV2Keys.listResources(dataSourceId)
+ });
+ queryClient.invalidateQueries({
+ queryKey: secretScanningV2Keys.listScans(dataSourceId)
+ });
+ };
+};
+
+export const useAddSecretScanningResource = () => {
+ const invalidate = useInvalidateAddedResources();
+ return useMutation({
+ mutationFn: async ({ type, dataSourceId, resourceName }: TAddSecretScanningResourceDTO) => {
+ const { data } = await apiRequest.post<{ resources: { id: string; name: string }[] }>(
+ `/api/v2/secret-scanning/data-sources/${type}/${dataSourceId}/resources`,
+ { resourceName }
+ );
+
+ return data.resources;
+ },
+ onSuccess: (_, dto) => invalidate(dto)
+ });
+};
+
+export const useAddAllSecretScanningResources = () => {
+ const invalidate = useInvalidateAddedResources();
+ return useMutation({
+ mutationFn: async ({ type, dataSourceId }: TAddAllSecretScanningResourcesDTO) => {
+ const { data } = await apiRequest.post<{ resources: { id: string; name: string }[] }>(
+ `/api/v2/secret-scanning/data-sources/${type}/${dataSourceId}/resources/all`
+ );
+
+ return data.resources;
+ },
+ onSuccess: (_, dto) => invalidate(dto)
+ });
+};
+
+export const useRemoveSecretScanningResource = () => {
+ const queryClient = useQueryClient();
+ return useMutation({
+ mutationFn: async ({ type, dataSourceId, resourceId }: TRemoveSecretScanningResourceDTO) => {
+ const { data } = await apiRequest.delete<{ resource: { id: string; name: string } }>(
+ `/api/v2/secret-scanning/data-sources/${type}/${dataSourceId}/resources/${resourceId}`
+ );
+
+ return data.resource;
+ },
+ onSuccess: (_, { projectId, dataSourceId }) => {
+ queryClient.invalidateQueries({
+ queryKey: secretScanningV2Keys.listDataSources(projectId)
+ });
+ queryClient.invalidateQueries({
+ queryKey: secretScanningV2Keys.dataSourceById(dataSourceId)
+ });
+ queryClient.invalidateQueries({
+ queryKey: secretScanningV2Keys.listResources(dataSourceId)
+ });
+ queryClient.invalidateQueries({
+ queryKey: secretScanningV2Keys.listScans(dataSourceId)
+ });
+ queryClient.invalidateQueries({
+ queryKey: secretScanningV2Keys.listFindings(projectId)
+ });
+ queryClient.invalidateQueries({
+ queryKey: secretScanningV2Keys.findingCount(projectId)
+ });
+ }
+ });
+};
+
// If possible, use useUpdateMultipleSecretScanningFinding instead.
export const useUpdateSecretScanningFinding = () => {
const queryClient = useQueryClient();
diff --git a/frontend/src/hooks/api/secretScanningV2/types/index.ts b/frontend/src/hooks/api/secretScanningV2/types/index.ts
index 046f0bc03f..ae51c8ba11 100644
--- a/frontend/src/hooks/api/secretScanningV2/types/index.ts
+++ b/frontend/src/hooks/api/secretScanningV2/types/index.ts
@@ -86,6 +86,29 @@ export type TTriggerSecretScanningDataSourceDTO = {
projectId: string;
};
+export type TAddSecretScanningResourceDTO = {
+ type: SecretScanningDataSource;
+ dataSourceId: string;
+ resourceName: string;
+ // required for query invalidation
+ projectId: string;
+};
+
+export type TAddAllSecretScanningResourcesDTO = {
+ type: SecretScanningDataSource;
+ dataSourceId: string;
+ // required for query invalidation
+ projectId: string;
+};
+
+export type TRemoveSecretScanningResourceDTO = {
+ type: SecretScanningDataSource;
+ dataSourceId: string;
+ resourceId: string;
+ // required for query invalidation
+ projectId: string;
+};
+
export type TUpdateSecretScanningFinding = {
findingId: string;
status: SecretScanningFindingStatus;
diff --git a/frontend/src/pages/secret-scanning/SecretScanningDataSourceByIdPage/components/AddSecretScanningResourceSheet.tsx b/frontend/src/pages/secret-scanning/SecretScanningDataSourceByIdPage/components/AddSecretScanningResourceSheet.tsx
index 9c343b56f5..6aaf9bbe14 100644
--- a/frontend/src/pages/secret-scanning/SecretScanningDataSourceByIdPage/components/AddSecretScanningResourceSheet.tsx
+++ b/frontend/src/pages/secret-scanning/SecretScanningDataSourceByIdPage/components/AddSecretScanningResourceSheet.tsx
@@ -27,14 +27,13 @@ import {
} from "@app/components/v3";
import {
getAddedResourceScanDescription,
- getConfigWithIncludedResource,
RESOURCE_DESCRIPTION_HELPER,
SECRET_SCANNING_DATA_SOURCE_MAP
} from "@app/helpers/secretScanningV2";
import {
TSecretScanningDataSource,
- useListSecretScanningAvailableResources,
- useUpdateSecretScanningDataSource
+ useAddSecretScanningResource,
+ useListSecretScanningAvailableResources
} from "@app/hooks/api/secretScanningV2";
type Props = {
@@ -52,7 +51,7 @@ export const AddSecretScanningResourceSheet = ({ dataSource, isOpen, onOpenChang
{ enabled: isOpen }
);
- const updateDataSource = useUpdateSecretScanningDataSource();
+ const addResource = useAddSecretScanningResource();
const resourceDetails = RESOURCE_DESCRIPTION_HELPER[dataSource.type];
const { name: platform } = SECRET_SCANNING_DATA_SOURCE_MAP[dataSource.type];
@@ -68,16 +67,13 @@ export const AddSecretScanningResourceSheet = ({ dataSource, isOpen, onOpenChang
const hasExcludedResources = resources.some(({ isIncluded }) => !isIncluded);
const handleEnableScanning = async (resourceName: string) => {
- const config = getConfigWithIncludedResource(dataSource, resourceName);
- if (!config) return;
-
setPendingName(resourceName);
try {
- await updateDataSource.mutateAsync({
+ await addResource.mutateAsync({
dataSourceId: dataSource.id,
type: dataSource.type,
projectId: dataSource.projectId,
- config
+ resourceName
});
} finally {
setPendingName(null);
diff --git a/frontend/src/pages/secret-scanning/SecretScanningDataSourceByIdPage/components/AddSecretScanningResourcesButton.tsx b/frontend/src/pages/secret-scanning/SecretScanningDataSourceByIdPage/components/AddSecretScanningResourcesButton.tsx
index 53045377e0..b559c24264 100644
--- a/frontend/src/pages/secret-scanning/SecretScanningDataSourceByIdPage/components/AddSecretScanningResourcesButton.tsx
+++ b/frontend/src/pages/secret-scanning/SecretScanningDataSourceByIdPage/components/AddSecretScanningResourcesButton.tsx
@@ -24,16 +24,14 @@ import {
import { ProjectPermissionSub } from "@app/context";
import { ProjectPermissionSecretScanningDataSourceActions } from "@app/context/ProjectPermissionContext/types";
import {
- areAllSecretScanningResourcesIncluded,
getAddedResourceScanDescription,
- getConfigWithAllResourcesIncluded,
RESOURCE_DESCRIPTION_HELPER,
SECRET_SCANNING_DATA_SOURCE_MAP
} from "@app/helpers/secretScanningV2";
import {
SecretScanningDataSource,
TSecretScanningDataSource,
- useUpdateSecretScanningDataSource
+ useAddAllSecretScanningResources
} from "@app/hooks/api/secretScanningV2";
import { GitLabDataSourceScope } from "@app/hooks/api/secretScanningV2/types/gitlab-data-source";
@@ -47,11 +45,10 @@ export const AddSecretScanningResourcesButton = ({ dataSource }: Props) => {
const [isSheetOpen, setIsSheetOpen] = useState(false);
const [isAddAllOpen, setIsAddAllOpen] = useState(false);
- const updateDataSource = useUpdateSecretScanningDataSource();
+ const addAllResources = useAddAllSecretScanningResources();
const { singularTitle, singularNoun, pluralTitle, pluralNoun } =
RESOURCE_DESCRIPTION_HELPER[dataSource.type];
- const isEverythingIncluded = areAllSecretScanningResourcesIncluded(dataSource);
// GitLab has no account-level webhook, so new commits are only reported for projects added one by one;
// hooking every project the connection can access could mean thousands of them.
const isAddAllUnsupported =
@@ -59,19 +56,17 @@ export const AddSecretScanningResourcesButton = ({ dataSource }: Props) => {
dataSource.config.scope === GitLabDataSourceScope.Account;
const handleAddAll = async () => {
- const config = getConfigWithAllResourcesIncluded(dataSource);
- if (!config) return;
-
- await updateDataSource.mutateAsync({
+ const added = await addAllResources.mutateAsync({
dataSourceId: dataSource.id,
type: dataSource.type,
- projectId: dataSource.projectId,
- config
+ projectId: dataSource.projectId
});
setIsAddAllOpen(false);
createNotification({
- text: `Added all ${pluralNoun}. ${getAddedResourceScanDescription(dataSource)}`,
+ text: added.length
+ ? `Added ${added.length} ${added.length === 1 ? singularNoun : pluralNoun}. ${getAddedResourceScanDescription(dataSource)}`
+ : `Every ${singularNoun} the connection can access is already added.`,
type: "success"
});
};
@@ -91,21 +86,14 @@ export const AddSecretScanningResourcesButton = ({ dataSource }: Props) => {
</Button>
</DropdownMenuTrigger>
<DropdownMenuContent align="end">
- <DropdownMenuItem
- isDisabled={isEverythingIncluded}
- onClick={() => setIsSheetOpen(true)}
- >
+ <DropdownMenuItem onClick={() => setIsSheetOpen(true)}>
<PlusIcon />
Add {singularTitle}
</DropdownMenuItem>
- <Tooltip
- hoverable
- delayDuration={50}
- open={isAddAllUnsupported && !isEverythingIncluded ? undefined : false}
- >
+ <Tooltip hoverable delayDuration={50} open={isAddAllUnsupported ? undefined : false}>
<TooltipTrigger asChild>
<DropdownMenuItem
- isDisabled={isEverythingIncluded || isAddAllUnsupported}
+ isDisabled={isAddAllUnsupported}
isDisabledFocusable={isAddAllUnsupported}
onClick={() => setIsAddAllOpen(true)}
>
@@ -139,10 +127,10 @@ export const AddSecretScanningResourcesButton = ({ dataSource }: Props) => {
</AlertDialogDescription>
</AlertDialogHeader>
<AlertDialogFooter>
- <AlertDialogCancel isDisabled={updateDataSource.isPending}>Cancel</AlertDialogCancel>
+ <AlertDialogCancel isDisabled={addAllResources.isPending}>Cancel</AlertDialogCancel>
<AlertDialogAction
variant="ss"
- isPending={updateDataSource.isPending}
+ isPending={addAllResources.isPending}
onClick={(event) => {
event.preventDefault();
handleAddAll();
diff --git a/frontend/src/pages/secret-scanning/SecretScanningDataSourceByIdPage/components/SecretScanningResourceRow.tsx b/frontend/src/pages/secret-scanning/SecretScanningDataSourceByIdPage/components/SecretScanningResourceRow.tsx
index 4d4fc828f5..abb6b830b9 100644
--- a/frontend/src/pages/secret-scanning/SecretScanningDataSourceByIdPage/components/SecretScanningResourceRow.tsx
+++ b/frontend/src/pages/secret-scanning/SecretScanningDataSourceByIdPage/components/SecretScanningResourceRow.tsx
@@ -1,17 +1,26 @@
+import { useState } from "react";
import { formatDistanceToNowStrict } from "date-fns";
import {
AlertTriangleIcon,
- BanIcon,
CheckIcon,
CopyIcon,
MoreHorizontalIcon,
- ScanSearchIcon
+ ScanSearchIcon,
+ TrashIcon
} from "lucide-react";
import { createNotification } from "@app/components/notifications";
import { ProjectPermissionCan } from "@app/components/permissions";
import { SecretScanningScanStatusBadge } from "@app/components/secret-scanning";
import {
+ AlertDialog,
+ AlertDialogAction,
+ AlertDialogCancel,
+ AlertDialogContent,
+ AlertDialogDescription,
+ AlertDialogFooter,
+ AlertDialogHeader,
+ AlertDialogTitle,
Badge,
DropdownMenu,
DropdownMenuContent,
@@ -19,23 +28,21 @@ import {
DropdownMenuTrigger,
IconButton,
TableCell,
- TableRow,
- Tooltip,
- TooltipContent,
- TooltipTrigger
+ TableRow
} from "@app/components/v3";
import {
ProjectPermissionSecretScanningDataSourceActions,
ProjectPermissionSub
} from "@app/context/ProjectPermissionContext/types";
import {
- isSecretScanningResourceIncluded,
+ canAddSecretScanningResources,
RESOURCE_DESCRIPTION_HELPER
} from "@app/helpers/secretScanningV2";
import {
SecretScanningScanStatus,
TSecretScanningDataSource,
TSecretScanningResourceWithDetails,
+ useRemoveSecretScanningResource,
useTriggerSecretScanningDataSource
} from "@app/hooks/api/secretScanningV2";
@@ -49,10 +56,14 @@ export const SecretScanningResourceRow = ({ resource, dataSource, onViewFindings
const { id, name, lastScannedAt, lastScanStatus, unresolvedFindings, lastScanStatusMessage } =
resource;
- const isIncluded = isSecretScanningResourceIncluded(dataSource, name);
+ const [isRemoveOpen, setIsRemoveOpen] = useState(false);
+
const resourceDetails = RESOURCE_DESCRIPTION_HELPER[dataSource.type];
+ // a project-scoped GitLab data source scans exactly one project, so removing it would leave nothing
+ const canRemove = canAddSecretScanningResources(dataSource);
const triggerDataSourceScan = useTriggerSecretScanningDataSource();
+ const removeResource = useRemoveSecretScanningResource();
const handleTriggerScan = async () => {
await triggerDataSourceScan.mutateAsync({
@@ -68,29 +79,27 @@ export const SecretScanningResourceRow = ({ resource, dataSource, onViewFindings
});
};
+ const handleRemove = async () => {
+ await removeResource.mutateAsync({
+ dataSourceId: dataSource.id,
+ type: dataSource.type,
+ projectId: dataSource.projectId,
+ resourceId: id
+ });
+
+ setIsRemoveOpen(false);
+ createNotification({
+ text: `Removed "${name}"`,
+ type: "success"
+ });
+ };
+
const handleCopyId = () => {
navigator.clipboard.writeText(id);
createNotification({ text: "Resource ID copied to clipboard", type: "info" });
};
const renderStatus = () => {
- if (!isIncluded) {
- return (
- <Tooltip>
- <TooltipTrigger asChild>
- <Badge variant="neutral">
- <BanIcon />
- Excluded
- </Badge>
- </TooltipTrigger>
- <TooltipContent className="max-w-sm">
- This {resourceDetails.singularNoun} is excluded in the data source configuration and
- won&apos;t be scanned.
- </TooltipContent>
- </Tooltip>
- );
- }
-
if (lastScanStatus && lastScanStatus !== SecretScanningScanStatus.Completed) {
return (
<SecretScanningScanStatusBadge
@@ -155,7 +164,7 @@ export const SecretScanningResourceRow = ({ resource, dataSource, onViewFindings
>
{(isAllowed: boolean) => (
<DropdownMenuItem
- isDisabled={!isAllowed || !isIncluded || dataSource.isDisconnected}
+ isDisabled={!isAllowed || dataSource.isDisconnected}
onClick={handleTriggerScan}
>
<ScanSearchIcon />
@@ -163,8 +172,51 @@ export const SecretScanningResourceRow = ({ resource, dataSource, onViewFindings
</DropdownMenuItem>
)}
</ProjectPermissionCan>
+ {canRemove && (
+ <ProjectPermissionCan
+ I={ProjectPermissionSecretScanningDataSourceActions.Edit}
+ a={ProjectPermissionSub.SecretScanningDataSources}
+ >
+ {(isAllowed: boolean) => (
+ <DropdownMenuItem
+ variant="danger"
+ isDisabled={!isAllowed}
+ onClick={() => setIsRemoveOpen(true)}
+ >
+ <TrashIcon />
+ Remove {resourceDetails.singularTitle}
+ </DropdownMenuItem>
+ )}
+ </ProjectPermissionCan>
+ )}
</DropdownMenuContent>
</DropdownMenu>
+ <AlertDialog open={isRemoveOpen} onOpenChange={setIsRemoveOpen}>
+ <AlertDialogContent>
+ <AlertDialogHeader>
+ <AlertDialogTitle>Remove {name}?</AlertDialogTitle>
+ <AlertDialogDescription>
+ This {resourceDetails.singularNoun} won&apos;t be scanned anymore, and its scans and
+ findings are permanently deleted.
+ {dataSource.isDiscoveryEnabled &&
+ ` Discovery is on, so it's added back the next time someone pushes to it.`}
+ </AlertDialogDescription>
+ </AlertDialogHeader>
+ <AlertDialogFooter>
+ <AlertDialogCancel isDisabled={removeResource.isPending}>Cancel</AlertDialogCancel>
+ <AlertDialogAction
+ variant="danger"
+ isPending={removeResource.isPending}
+ onClick={(event) => {
+ event.preventDefault();
+ handleRemove();
+ }}
+ >
+ Remove {resourceDetails.singularTitle}
+ </AlertDialogAction>
+ </AlertDialogFooter>
+ </AlertDialogContent>
+ </AlertDialog>
</TableCell>
</TableRow>
);
diff --git a/frontend/src/pages/secret-scanning/SecretScanningDataSourceByIdPage/components/SecretScanningResourceTable.tsx b/frontend/src/pages/secret-scanning/SecretScanningDataSourceByIdPage/components/SecretScanningResourceTable.tsx
index 36e8a58134..8db1ab8b02 100644
--- a/frontend/src/pages/secret-scanning/SecretScanningDataSourceByIdPage/components/SecretScanningResourceTable.tsx
+++ b/frontend/src/pages/secret-scanning/SecretScanningDataSourceByIdPage/components/SecretScanningResourceTable.tsx
@@ -1,21 +1,14 @@
-import { useMemo, useState } from "react";
-import { ChevronDownIcon, FilterIcon, FolderGitIcon, LockIcon, SearchIcon } from "lucide-react";
+import { useMemo } from "react";
+import { ChevronDownIcon, FolderGitIcon, LockIcon, SearchIcon } from "lucide-react";
import { twMerge } from "tailwind-merge";
import {
- Button,
Card,
CardAction,
CardContent,
CardDescription,
CardHeader,
CardTitle,
- DropdownMenu,
- DropdownMenuContent,
- DropdownMenuRadioGroup,
- DropdownMenuRadioItem,
- DropdownMenuSeparator,
- DropdownMenuTrigger,
Empty,
EmptyDescription,
EmptyHeader,
@@ -38,7 +31,6 @@ import { ProjectPermissionSub, useProjectPermission } from "@app/context";
import { ProjectPermissionSecretScanningDataSourceActions } from "@app/context/ProjectPermissionContext/types";
import {
canAddSecretScanningResources,
- isSecretScanningResourceIncluded,
RESOURCE_DESCRIPTION_HELPER,
SECRET_SCANNING_DATA_SOURCE_MAP
} from "@app/helpers/secretScanningV2";
@@ -60,18 +52,6 @@ enum ResourcesOrderBy {
LastScanned = "last-scanned"
}
-enum ResourceStatusFilter {
- All = "all",
- Included = "included",
- Excluded = "excluded"
-}
-
-const RESOURCE_STATUS_FILTER_LABELS: Record<ResourceStatusFilter, string> = {
- [ResourceStatusFilter.All]: "All",
- [ResourceStatusFilter.Included]: "Included",
- [ResourceStatusFilter.Excluded]: "Excluded"
-};
-
const SKELETON_ROWS = ["first", "second", "third"];
const SKELETON_CELLS = ["name", "status", "last-scan", "actions"];
@@ -113,8 +93,6 @@ export const SecretScanningResourcesTable = ({ dataSource, onViewFindings }: Pro
ProjectPermissionSub.SecretScanningDataSources
);
- const [statusFilter, setStatusFilter] = useState(ResourceStatusFilter.All);
-
const { data: resources = [], isPending } = useListSecretScanningResources(
{ dataSourceId: dataSource.id, type: dataSource.type },
{
@@ -143,20 +121,13 @@ export const SecretScanningResourcesTable = ({ dataSource, onViewFindings }: Pro
const searchValue = search.trim().toLowerCase();
return resources
- .filter(({ name }) => {
- if (statusFilter !== ResourceStatusFilter.All) {
- const isIncluded = isSecretScanningResourceIncluded(dataSource, name);
- if (isIncluded !== (statusFilter === ResourceStatusFilter.Included)) return false;
- }
-
- return name.toLowerCase().includes(searchValue);
- })
+ .filter(({ name }) => name.toLowerCase().includes(searchValue))
.sort((a, b) =>
orderDirection === OrderByDirection.ASC
? sortResources(a, b, orderBy)
: sortResources(b, a, orderBy)
);
- }, [resources, orderDirection, search, orderBy, statusFilter, dataSource]);
+ }, [resources, orderDirection, search, orderBy]);
useResetPageHelper({
totalCount: filteredResources.length,
@@ -275,33 +246,6 @@ export const SecretScanningResourcesTable = ({ dataSource, onViewFindings }: Pro
placeholder={`Search ${resourceDetails.pluralNoun}...`}
/>
</InputGroup>
- <DropdownMenu>
- <DropdownMenuTrigger asChild>
- <Button variant="outline">
- <FilterIcon />
- Status: {RESOURCE_STATUS_FILTER_LABELS[statusFilter]}
- <ChevronDownIcon />
- </Button>
- </DropdownMenuTrigger>
- <DropdownMenuContent align="end">
- <DropdownMenuRadioGroup
- value={statusFilter}
- onValueChange={(value) => {
- setStatusFilter(value as ResourceStatusFilter);
- setPage(1);
- }}
- >
- <DropdownMenuRadioItem value={ResourceStatusFilter.Included}>
- Included
- </DropdownMenuRadioItem>
- <DropdownMenuRadioItem value={ResourceStatusFilter.Excluded}>
- Excluded
- </DropdownMenuRadioItem>
- <DropdownMenuSeparator />
- <DropdownMenuRadioItem value={ResourceStatusFilter.All}>All</DropdownMenuRadioItem>
- </DropdownMenuRadioGroup>
- </DropdownMenuContent>
- </DropdownMenu>
</div>
</CardHeader>
<CardContent>