Infisical is the open-source platform for securing secrets, certificates, access, and AI Agents.
TypeScript acmeagentsai-agentscertificate-managementdevsecopsenvironment-variableskubernetesopen-sourcepampkipostgresprivate-caprivileged-access-managementsecret-managementsecret-managersecretssecrets-managementsecuritysecurity-toolsvault
get paginated files for realtime scans
Matheus Nogueira557e52811015
diff --git a/backend/AGENTS.md b/backend/AGENTS.mdindex a857cf7ff5..2b768432fb 100644--- a/backend/AGENTS.md+++ b/backend/AGENTS.md@@ -1233,7 +1233,7 @@ EE routes register before community routes so they can override/extend endpoints**Secret Scanning** is single-project (see the root `AGENTS.md`): `secret-scanning-v2-project-bootstrap.ts` creates the project with the org when it has a creator to make admin (like PAM and Agent Vault), `secret-scanning-v2-project-resolver.ts` picks the active project and lazily creates it, with every org admin as project admin, for older orgs and for orgs created from the admin console, whose admins are only invited after the org exists, and routes that take `projectId` read `req.internalSecretScanningProjectId`. A data source covers the resources it has rows for (added by a user or discovered from a push); its `config` only says where to look (Bitbucket workspace, GitLab group), never which resources, so scans, "included" state and GitLab account-scope webhooks all key off rows. Stored configs may still carry the old `includeRepos` / `includeProjects` keys, which nothing reads.-**Secret scanning scans record what started them in `trigger`** (`SecretScanningScanTrigger`: `manual`, `resources-added`, `push`, `pull-request`), separate from `type`, which says how much was scanned (`historical` / `realtime`). Rows from before the column are null. `metadata` (jsonb) holds trigger-specific data and is parsed by trigger; it is omitted from API responses. **GitHub pull request checks** are realtime scans with `trigger: pull-request`, gated on the data source's `isScanNewCommitsEnabled`. A PR on a repository that isn't a resource yet goes through `discoverResource`, the same discovery a push gets (skipped when discovery is off, history scanned when historical scan is on), so both webhooks add repositories the same way. `secret-scanning-v2-queue.ts` stays provider-agnostic: provider behavior goes behind hooks on the data source factory, never behind a provider check in the queue. A scan reported back to where it was requested implements `TSecretScanningScanReport` (`start`, `getCompletedScanMetadata`, `complete`, `fail`, `cancel`), which the factory builds from the diff job payload (`getDiffScanReport`, so a scan whose data source was deleted can still close it) or from the saved scan (`getScanReport`, for the stuck-scan reaper). `getDiffScanFindingsPayload` returns `{ findings, details? }`, where `details` is what the report needs beyond the findings (GitHub: commit counts). Webhook events other than pushes go through `queueDataSourceEvent` as a `TQueueSecretScanningDataSourceEvent` and are handled by the provider factory's optional `processEvent`, which gets `discoverResource` and `queueReportedDiffScan` per call (as `initialize` gets the DAL) (inserts the scan under the ID its report already carries, and closes the report if queueing fails). The `pull_request`, `check_run.rerequested`, `check_suite.rerequested` and `check_run.requested_action` webhook handlers in `github-secret-scanning-service.ts` only enqueue a reduced `TQueueGitHubPullRequestEvent` that way (job ID from the GitHub delivery ID, 3 attempts); `processGitHubPullRequestEvent` in `github-secret-scanning-fns.ts` processes it in the worker. Everything before the check run is created is safe to repeat and nothing after it throws, which is what makes the retries safe. The processor creates the check run *before* the scan row: the scan's uuid is generated in the app and sent as the check run's `external_id`, so a `check_run.rerequested` event (whose `pull_requests` is empty for forks) loads the scan by primary key and reads the PR from `metadata` (`GitHubPullRequestScanMetadataSchema`). `check_suite.rerequested` ("Re-run all checks") names only the suite, so it lists the suite's latest check run with our name and takes the same path from its `external_id`. Pull requests from forks (`head.repo` differs from the base repository, or was deleted) aren't scanned and don't discover the repository: anyone can open one on a public repository, so scanning them would let any GitHub account spend the shared scan workers and the installation's API quota and plant findings. They get a check run with the scan's check name created already completed with conclusion `skipped`, which passes branch protection, unless the head commit already has one of our check runs: checks belong to the commit, and the newest run counts, so a skipped check would otherwise replace a same-repository PR's failed scan of that commit. Their commits are scanned by the push scan once merged. If the check run can't be created (typically an installation that hasn't accepted the Checks permission), nothing is scanned. The scan lists commits with the compare API pinned to the event's base and head SHAs (not `pulls.listCommits`, which caps at 250 and drifts as commits land) and reuses the push path's per-commit scan, so fingerprints match and findings dedupe with push scans. The check fails on every finding in the PR's commits, not only the rows its own scan inserted: a finding a push scan saved first keeps that scan's `scanId`. Findings triaged `false-positive` or `ignore` don't fail the check: the result is judged on the rows the upsert returns, which carry their stored status, and the scan stores every finding ID in `metadata.findingIds` when it completes. A failing check offers **Skip: False positive** and **Skip: Ignore** (`PULL_REQUEST_CHECK_SKIP_ACTIONS`); the `check_run.requested_action` event verifies the scan belongs to the event's installation and repository, sets that status on the PR's findings like Infisical's own triage (audited per finding, GitHub login in `triageComment`), and starts a new check run. GitHub shows those buttons to anyone with write access to the repository. Every exit of `handleDiffScan` (including a failure before the scan starts) and the stuck-scan reaper closes the check run, since a check left `queued` blocks merging when it is required. The reaper fails realtime scans left `queued` past the stuck-scan timeout as well as stuck `scanning` ones, because a realtime job's payload isn't stored and can't be redelivered; check reporting is best effort after the scan's outcome is committed (diff scans run with one attempt).+**Secret scanning scans record what started them in `trigger`** (`SecretScanningScanTrigger`: `manual`, `resources-added`, `push`, `pull-request`), separate from `type`, which says how much was scanned (`historical` / `realtime`). Rows from before the column are null. `metadata` (jsonb) holds trigger-specific data and is parsed by trigger; it is omitted from API responses. **GitHub pull request checks** are realtime scans with `trigger: pull-request`, gated on the data source's `isScanNewCommitsEnabled`. A PR on a repository that isn't a resource yet goes through `discoverResource`, the same discovery a push gets (skipped when discovery is off, history scanned when historical scan is on), so both webhooks add repositories the same way. `secret-scanning-v2-queue.ts` stays provider-agnostic: provider behavior goes behind hooks on the data source factory, never behind a provider check in the queue. A scan reported back to where it was requested implements `TSecretScanningScanReport` (`start`, `getCompletedScanMetadata`, `complete`, `fail`, `cancel`), which the factory builds from the diff job payload (`getDiffScanReport`, so a scan whose data source was deleted can still close it) or from the saved scan (`getScanReport`, for the stuck-scan reaper). `getDiffScanFindingsPayload` returns `{ findings, details? }`, where `details` is what the report needs beyond the findings (GitHub: commit counts). Webhook events other than pushes go through `queueDataSourceEvent` as a `TQueueSecretScanningDataSourceEvent` and are handled by the provider factory's optional `processEvent`, which gets `discoverResource` and `queueReportedDiffScan` per call (as `initialize` gets the DAL) (inserts the scan under the ID its report already carries, and closes the report if queueing fails). The `pull_request`, `check_run.rerequested`, `check_suite.rerequested` and `check_run.requested_action` webhook handlers in `github-secret-scanning-service.ts` only enqueue a reduced `TQueueGitHubPullRequestEvent` that way (job ID from the GitHub delivery ID, 3 attempts); `processGitHubPullRequestEvent` in `github-secret-scanning-fns.ts` processes it in the worker. Everything before the check run is created is safe to repeat and nothing after it throws, which is what makes the retries safe. The processor creates the check run *before* the scan row: the scan's uuid is generated in the app and sent as the check run's `external_id`, so a `check_run.rerequested` event (whose `pull_requests` is empty for forks) loads the scan by primary key and reads the PR from `metadata` (`GitHubPullRequestScanMetadataSchema`). `check_suite.rerequested` ("Re-run all checks") names only the suite, so it lists the suite's latest check run with our name and takes the same path from its `external_id`. Pull requests from forks (`head.repo` differs from the base repository, or was deleted) aren't scanned and don't discover the repository: anyone can open one on a public repository, so scanning them would let any GitHub account spend the shared scan workers and the installation's API quota and plant findings. They get a check run with the scan's check name created already completed with conclusion `skipped`, which passes branch protection, unless the head commit already has one of our check runs: checks belong to the commit, and the newest run counts, so a skipped check would otherwise replace a same-repository PR's failed scan of that commit. Their commits are scanned by the push scan once merged. If the check run can't be created (typically an installation that hasn't accepted the Checks permission), nothing is scanned. The scan lists commits with the compare API pinned to the event's base and head SHAs (not `pulls.listCommits`, which caps at 250 and drifts as commits land) and reuses the push path's per-commit scan, so fingerprints match and findings dedupe with push scans. That scan pages through each commit's files (one `getCommit` response holds only the first page); GitHub lists at most 3,000 files per commit, so a commit beyond that fails a PR scan (`requireCompleteCommits`) while a push scan scans the files it gets. The check fails on every finding in the PR's commits, not only the rows its own scan inserted: a finding a push scan saved first keeps that scan's `scanId`. Findings triaged `false-positive` or `ignore` don't fail the check: the result is judged on the rows the upsert returns, which carry their stored status, and the scan stores every finding ID in `metadata.findingIds` when it completes. A failing check offers **Skip: False positive** and **Skip: Ignore** (`PULL_REQUEST_CHECK_SKIP_ACTIONS`); the `check_run.requested_action` event verifies the scan belongs to the event's installation and repository, sets that status on the PR's findings like Infisical's own triage (audited per finding, GitHub login in `triageComment`), and starts a new check run. GitHub shows those buttons to anyone with write access to the repository. Every exit of `handleDiffScan` (including a failure before the scan starts) and the stuck-scan reaper closes the check run, since a check left `queued` blocks merging when it is required. The reaper fails realtime scans left `queued` past the stuck-scan timeout as well as stuck `scanning` ones, because a realtime job's payload isn't stored and can't be redelivered; check reporting is best effort after the scan's outcome is committed (diff scans run with one attempt).**Dynamic secret lease secrets needed at revoke time go in `dynamic_secret_leases.encryptedLeaseData`**, never in `externalEntityId` or `config` (both plaintext, and `externalEntityId` is returned by every lease route and logged in audit events). A provider opts in by listing the `create()` `data` keys in `persistedLeaseFields`; the lease service encrypts them with the project's SecretManager key and every revoke path passes them back as `metadata.leaseData`. Lease routes return `SanitizedDynamicSecretLeaseSchema` so the column never reaches a response. Secret *config* fields are listed separately in `DYNAMIC_SECRET_SECRET_FIELDS` (`providers/redact.ts`), as dotted paths so nested secrets work (`clientAuth.clientSecret`): reads strip them via `redactStoredInputs`, and an update that omits one keeps the stored value via `restoreOmittedSecretFields`, because the update merge is shallow.diff --git a/backend/src/ee/services/secret-scanning-v2/github/github-secret-scanning-fns.test.ts b/backend/src/ee/services/secret-scanning-v2/github/github-secret-scanning-fns.test.tsindex 1fdba9b389..e042b5c5a9 100644--- a/backend/src/ee/services/secret-scanning-v2/github/github-secret-scanning-fns.test.ts+++ b/backend/src/ee/services/secret-scanning-v2/github/github-secret-scanning-fns.test.ts@@ -245,6 +245,74 @@ describe("scanGitHubCommitsAndGetFindings", () => {const scanned = scanContentAndGetFindings.mock.calls.map(([content]) => content.trim());expect(scanned).toEqual(["added", "modified", "renamed", "copied", "changed"]);});++ const commit = { sha: "c".repeat(40), authorName: "a", authorEmail: "[email protected]", date: "", message: "m" };+ const makeFiles = (count: number, offset: number) =>+ Array.from({ length: count }, (_, index) => ({+ status: "modified",+ filename: `file-${offset + index}.txt`,+ patch: `@@ -0,0 +1 @@\n+file-${offset + index}`+ }));+ const asOctokit = (getCommit: unknown) =>+ ({ repos: { getCommit } }) as unknown as Parameters<typeof scanGitHubCommitsAndGetFindings>[0]["octokit"];++ test("scans every page of a commit's files", async () => {+ const getCommit = vi+ .fn()+ .mockResolvedValueOnce({ data: { files: makeFiles(100, 0) } })+ .mockResolvedValueOnce({ data: { files: makeFiles(100, 100) } })+ .mockResolvedValueOnce({ data: { files: makeFiles(5, 200) } });++ await scanGitHubCommitsAndGetFindings({+ octokit: asOctokit(getCommit),+ repositoryFullName: "acme/api",+ resourceName: "acme/api",+ commits: [commit],+ requireCompleteCommits: true+ });++ expect(getCommit).toHaveBeenCalledTimes(3);+ expect(getCommit).toHaveBeenLastCalledWith(expect.objectContaining({ page: 3, per_page: 100 }));+ expect(scanContentAndGetFindings).toHaveBeenCalledTimes(205);+ });++ test("fails a scan that requires every file when a commit has more files than GitHub lists", async () => {+ let page = 0;+ const getCommit = vi.fn().mockImplementation(() => {+ page += 1;+ return Promise.resolve({ data: { files: makeFiles(100, (page - 1) * 100) } });+ });++ await expect(+ scanGitHubCommitsAndGetFindings({+ octokit: asOctokit(getCommit),+ repositoryFullName: "acme/api",+ resourceName: "acme/api",+ commits: [commit],+ requireCompleteCommits: true+ })+ ).rejects.toThrow("changes more than 3,000 files");++ expect(getCommit).toHaveBeenCalledTimes(30);+ expect(scanContentAndGetFindings).not.toHaveBeenCalled();+ });++ test("scans the files GitHub lists when the scan doesn't require every file", async () => {+ let page = 0;+ const getCommit = vi.fn().mockImplementation(() => {+ page += 1;+ return Promise.resolve({ data: { files: makeFiles(100, (page - 1) * 100) } });+ });++ await scanGitHubCommitsAndGetFindings({+ octokit: asOctokit(getCommit),+ repositoryFullName: "acme/api",+ resourceName: "acme/api",+ commits: [commit]+ });++ expect(scanContentAndGetFindings).toHaveBeenCalledTimes(3000);+ });});describe("createSkippedPullRequestCheckRun", () => {diff --git a/backend/src/ee/services/secret-scanning-v2/github/github-secret-scanning-fns.ts b/backend/src/ee/services/secret-scanning-v2/github/github-secret-scanning-fns.tsindex ead1f4bedc..66e75a1166 100644--- a/backend/src/ee/services/secret-scanning-v2/github/github-secret-scanning-fns.ts+++ b/backend/src/ee/services/secret-scanning-v2/github/github-secret-scanning-fns.ts@@ -42,6 +42,9 @@ export const GITHUB_PULL_REQUEST_CHECK_NAME = "Infisical Secret Scanning";const COMPARE_COMMITS_PER_PAGE = 100;const COMPARE_COMMITS_MAX_PAGES = 20;+const COMMIT_FILES_PER_PAGE = 100;+// GitHub lists at most 3,000 of a commit's files, across pages.+const COMMIT_MAX_LISTED_FILES = 3000;// GitHub rejects a check run update carrying more than 50 annotations; more are sent as further updates.const CHECK_RUN_ANNOTATIONS_PER_REQUEST = 50;const CHECK_RUN_MAX_ANNOTATIONS = 500;@@ -112,18 +115,44 @@ export const isGitHubPullRequestScanPayload = (payload: TQueueSecretScanningResourceDiffScan["payload"]): payload is TGitHubPullRequestScanPayload => "pullRequest" in payload && "checkRunId" in payload;+// A commit's files are paginated; one response alone holds only the first page of them.+const listCommitFiles = async (+ octokit: TGitHubOctokit,+ { owner, repo, sha }: { owner: string; repo: string; sha: string }+) => {+ const files: NonNullable<Awaited<ReturnType<TGitHubOctokit["repos"]["getCommit"]>>["data"]["files"]> = [];++ for (let page = 1; files.length < COMMIT_MAX_LISTED_FILES; page += 1) {+ // eslint-disable-next-line no-await-in-loop+ const { data } = await octokit.repos.getCommit({ owner, repo, ref: sha, per_page: COMMIT_FILES_PER_PAGE, page });+ const pageFiles = data.files ?? [];++ files.push(...pageFiles);++ if (pageFiles.length < COMMIT_FILES_PER_PAGE) break;+ }++ return { files, isComplete: files.length < COMMIT_MAX_LISTED_FILES };+};++/**+ * A commit that changes more files than GitHub lists can't be fully scanned from the API. A pull request check passes+ * `requireCompleteCommits` so it fails rather than passing on part of the commit; a push scan scans what it can.+ */export const scanGitHubCommitsAndGetFindings = async ({octokit,repositoryFullName,resourceName,commits,- configPath+ configPath,+ requireCompleteCommits}: {octokit: TGitHubOctokit;repositoryFullName: string;resourceName: string;commits: TGitHubCommitToScan[];configPath?: string;+ requireCompleteCommits?: boolean;}): Promise<TFindingsPayload> => {const { owner, repo } = splitRepositoryFullName(repositoryFullName);@@ -131,16 +160,21 @@ export const scanGitHubCommitsAndGetFindings = async ({for (const commit of commits) {// eslint-disable-next-line no-await-in-loop- const commitData = await octokit.repos.getCommit({- owner,- repo,- ref: commit.sha- });+ const { files, isComplete } = await listCommitFiles(octokit, { owner, repo, sha: commit.sha });++ if (!isComplete) {+ if (requireCompleteCommits) {+ throw new Error(+ `Commit ${commit.sha.slice(0, 7)} changes more than ${COMMIT_MAX_LISTED_FILES.toLocaleString("en-US")} files, more than GitHub lists, so it can't be fully scanned.`+ );+ }- // eslint-disable-next-line no-continue- if (!commitData.data.files) continue;+ logger.warn(+ `secretScanningV2DiffScan: GitHub - commit files truncated, scanning the first ${files.length} [repository=${repositoryFullName}] [commit=${commit.sha}]`+ );+ }- for (const file of commitData.data.files) {+ for (const file of files) {// A renamed, copied or changed file's patch can add lines too; a removed file's adds none.if (file.status !== "removed" && file.patch) {// eslint-disable-next-line@@ -254,7 +288,8 @@ export const scanGitHubPullRequest = async ({repositoryFullName: payload.repository.full_name,resourceName,commits,- configPath+ configPath,+ requireCompleteCommits: true});return { findings, details: { scannedCommitCount: commits.length, totalCommitCount: totalCommits } };