The open-source app everyone uses to manage agents at work
TypeScript acpxagent-orchestrationagent-skillsagentic-aiai-agentsai-engineeringai-governanceai-toolsautonomous-agentsclaudeclaude-codeclaude-skillscodexhermesllmmcpmulti-agentopenclawproject-managementworkflow-automation
refactor(tools): isolate GitHub access discovery (#17144)
Dotta9db3a5726e70
diff --git a/doc/plans/2026-10-21-tool-access-boundaries.md b/doc/plans/2026-10-21-tool-access-boundaries.mdnew file mode 100644index 0000000000..a7b8b1eb8f--- /dev/null+++ b/doc/plans/2026-10-21-tool-access-boundaries.md@@ -0,0 +1,46 @@+# Tool access boundaries++Baseline: `efc2ddc7bf19ec319b21cd294e71e5a4932c16f4`.+`server/src/services/tool-access.ts` has 21,525 lines and 829,519 bytes.+Imports account for 293 lines. The service factory spans 18,366 lines,+including its public method implementations and service wiring.+The largest functions are connection setup (1,468 lines), OAuth callback+completion (644), OAuth start (613), and grant refresh (611).++The first batch contains three dependent PRs. Each PR targets its predecessor.+Existing exports remain compatible. No branch is merged as part of this batch.++| Order | Responsibility and destination | Estimated move / net reduction | Contract | State and effects | Verification |+| --- | --- | --- | --- | --- | --- |+| 1 | GitHub access discovery, `services/tool-access/github-access-discovery.ts` | 290 / 288 lines | Token or headers and request function to normalized repository/account metadata | GitHub pagination and response validation; no stored grants or shared state | Existing public exports, paginated PAT and managed access, malformed responses, retry classification, GitHub service integration |+| 2 | Connection method configuration and projection, `services/tool-access/connection-methods.ts` | About 330 / 310 lines | Reviewed method and raw settings to validated settings; connection settings and tool input to projected headers/schema/arguments | Pure transforms; provider-specific argument policy stays in its existing module | Configuration rejection, managed defaults, header safety, legacy methods, setup and discovery integration |+| 3 | OAuth protocol handling, `services/tool-access/oauth/` | About 1,500 / 1,400 lines | Guarded requests and connection metadata to discovered endpoints; registration request to validated client material | Network interpretation and endpoint policy; persistence and refresh leases stay in the coordinator | Existing discovery, DCR, CIMD, scope, redaction, issuer and endpoint-security tests; real local protocol fixture |+| 4 | Profile resolution and review, `services/tool-access/profiles/` | About 1,500 / 1,400 lines | Company, agent, selectors and current catalog to effective profiles and review decisions | Scoped queries and profile mutations; preserve assignment locks and caller transactions | Binding precedence, exclusions, company isolation, new-tool review and lightweight projections |+| 5 | Connection token brokering, `services/tool-access/token-broker/` | About 1,400 / 1,300 lines | Authorized run and token request to issuance result | Run/membership locks, rate limit, credential exchange, hashed issuance audit | Revocation races, scope restriction, unavailable/disabled broker, exchange failure and secret redaction |++Rows 4 and 5 are follow-up candidates, not part of the initial batch. Reassess+their contracts after the first three cuts. Keep authorization, current grant+checks, company isolation, audit, and transaction ownership with their existing+owners. Do not pass the full service context into a new module.++Existing extracted owners include tool-access policy, profile entry validation,+binding precedence, credential bindings, MCP transport and endpoint guards,+runtime supervision, managed Cloud connectors, Vercel Connect, and individual+provider policies. New modules import those owners directly and never import+the compatibility service entry point.++Run relevant tests before and after each move, with one database test worker.+Compare moved bodies against the base. Run repository typecheck, full tests,+and build at the batch handoff. Record setup failures separately from behavior+failures, and leave unavailable checks unverified.++## First extraction++GitHub access discovery preserves all six moved function bodies exactly.+The service retains grant selection, credential resolution, refresh retry,+membership locking, revision checks, persistence, and webhook updates.+The original service becomes 21,237 lines / 820,116 bytes; the new module is+294 lines / 9,722 bytes. Public-path tests pass before and after: 13 focused+tests and 11 selected GitHub service cases using disposable PostgreSQL.+The initial test attempt could not resolve the unbuilt plugin SDK; building+the existing workspace prerequisites resolved that setup failure.diff --git a/server/src/__tests__/github-grant-metadata.test.ts b/server/src/__tests__/github-grant-metadata.test.tsindex 6ea824a081..16f04a2c9f 100644--- a/server/src/__tests__/github-grant-metadata.test.ts+++ b/server/src/__tests__/github-grant-metadata.test.ts@@ -92,4 +92,27 @@ describe("GitHub grant metadata", () => {details: expect.objectContaining({ code: "github_access_check_failed" }),});});++ it.each([+ [401, "oauth_reauthorization_required"],+ [503, "github_access_check_failed"],+ ])("preserves the retry decision for HTTP %s without reflecting provider text", async (status, code) => {+ const request = vi.fn<typeof fetch>().mockResolvedValue(new Response("private provider detail", { status }));+ await expect(loadGitHubGrantMetadata("ghu_secret", request)).rejects.toMatchObject({+ status: 422,+ message: "GitHub could not verify this account's installation access",+ details: { code },+ });+ expect(request).toHaveBeenCalledTimes(1);+ });++ it("rejects an empty next page before reporting an incomplete access snapshot", async () => {+ const request = vi.fn<typeof fetch>()+ .mockResolvedValueOnce(json({ id: 42, login: "octocat" }))+ .mockResolvedValueOnce(json({ installations: [] }, true));+ await expect(loadGitHubGrantMetadata("ghu_secret", request)).rejects.toMatchObject({+ details: { code: "github_bad_response" },+ });+ expect(request).toHaveBeenCalledTimes(2);+ });});diff --git a/server/src/services/tool-access.ts b/server/src/services/tool-access.tsindex 1f5a02b08c..a8b9835d7a 100644--- a/server/src/services/tool-access.ts+++ b/server/src/services/tool-access.ts@@ -1,3 +1,5 @@+import { loadGitHubGrantMetadata, loadGitHubTokenRepositories } from "./tool-access/github-access-discovery.js";+export { loadGitHubGrantMetadata, loadGitHubTokenRepositories } from "./tool-access/github-access-discovery.js";import { getAppDefinitionForApplicationKey, isConnectorFeatureEnabled } from "@paperclipai/shared";import { stableHash, descriptorHash } from "./tool-catalog-hash.js";import { assertProfileEntryInputs } from "./tool-profile-entry-validation.js";@@ -2602,296 +2604,6 @@ function managedConnectorProfile(value: string | undefined): {return null;}-function gitHubRepositoryOwnerType(repository: Record<string, unknown>) {- const type = recordValue(repository.owner) ? repository.owner.type : undefined;- return type === "Organization" ? "organization" as const- : type === "User" ? "personal" as const : undefined;-}--export async function loadGitHubTokenRepositories(- headers: Record<string, string>,- request: typeof fetch = fetch,-) {- const repositories: Array<{- id: string;- fullName: string;- ownerType?: "personal" | "organization";- private?: boolean;- }> = [];- for (let page = 1; ; page += 1) {- const response = await request(- `https://api.github.com/user/repos?per_page=100&page=${page}`,- {- headers: {- ...headers,- accept: "application/vnd.github+json",- "user-agent": "Paperclip",- "x-github-api-version": "2022-11-28",- },- signal: AbortSignal.timeout(15_000),- },- );- if (!response.ok)- throw unprocessable(- "Could not load GitHub repositories. Reconnect GitHub and try again.",- );- const rows: unknown = await response.json();- if (!Array.isArray(rows))- throw unprocessable("GitHub returned invalid repositories");- for (const row of rows) {- if (- !recordValue(row) ||- !githubId(row.id) ||- typeof row.full_name !== "string" ||- !/^[A-Za-z0-9][A-Za-z0-9-]*\/(?!\.{1,2}$)[A-Za-z0-9_.-]+$/.test(- row.full_name,- )- ) {- throw unprocessable("GitHub returned invalid repository metadata");- }- repositories.push({- id: githubId(row.id)!,- fullName: row.full_name,- ...(gitHubRepositoryOwnerType(row) ? { ownerType: gitHubRepositoryOwnerType(row) } : {}),- ...(typeof row.private === "boolean" ? { private: row.private } : {}),- });- }- if (!/;\s*rel="next"/.test(response.headers.get("link") ?? ""))- return repositories;- if (!rows.length) throw unprocessable("GitHub returned invalid pagination");- }-}--export async function loadGitHubGrantMetadata(- accessToken: string,- request: typeof fetch = fetch,- appSlug?: string,-): Promise<{- userId: string;- login: string;- avatarUrl?: string;- installationCount: number;- repositoryCount: number;- repositorySelection: "all" | "selected" | "mixed" | "none";- installationIds: string[];- installationOwnerLogins: string[];- repositories: Array<{- id: string;- fullName: string;- installationId: string;- ownerType?: "personal" | "organization";- private?: boolean;- }>;- installationUrl: string;- managementUrl: string;- appSlug?: string;- accessRevision: string;- lastAccessRefreshAt: string;- webhookHealth: "pending";-}> {- let resolvedAppSlug = appSlug;- const accessRefreshStartedAt = new Date().toISOString();- const github = async (- path: string,- ): Promise<{ data: Record<string, unknown>; hasNext: boolean }> => {- const response = await request(`https://api.github.com${path}`, {- headers: {- accept: "application/vnd.github+json",- authorization: `Bearer ${accessToken}`,- "user-agent": "Paperclip",- "x-github-api-version": "2022-11-28",- },- signal: AbortSignal.timeout(15_000),- });- if (!response.ok) {- throw unprocessable(- "GitHub could not verify this account's installation access",- {- code:- response.status === 401- ? "oauth_reauthorization_required"- : "github_access_check_failed",- },- );- }- const value = (await response.json()) as unknown;- if (!recordValue(value))- throw unprocessable("GitHub returned invalid account metadata", {- code: "github_bad_response",- });- return {- data: value,- hasNext: /;\s*rel="next"/.test(response.headers.get("link") ?? ""),- };- };- const list = async (- path: string,- key: string,- ): Promise<Record<string, unknown>[]> => {- const items: Record<string, unknown>[] = [];- for (let page = 1; ; page += 1) {- const { data, hasNext } = await github(- `${path}?per_page=100&page=${page}`,- );- const batch = data[key];- if (- !Array.isArray(batch) ||- !batch.every(recordValue) ||- (hasNext && batch.length === 0)- ) {- throw unprocessable("GitHub returned invalid access metadata", {- code: "github_bad_response",- });- }- items.push(...batch);- if (!hasNext) return items;- }- };- const { data: user } = await github("/user");- const userId = githubId(user.id);- const login = typeof user.login === "string" ? user.login : null;- if (!userId || !login)- throw unprocessable("GitHub returned invalid account metadata", {- code: "github_bad_response",- });- const installations = await list("/user/installations", "installations");- const installationIds: string[] = [];- const owners = new Set<string>();- const selections = new Set<"all" | "selected">();- const managementUrls = new Set<string>();- const repositories = new Map<- string,- { id: string; fullName: string; installationId: string; ownerType?: "personal" | "organization"; private?: boolean }- >();- for (const installation of installations) {- const installationId = githubId(installation.id);- if (!installationId) continue;- installationIds.push(installationId);- // Older grants predate the broker's appSlug field. GitHub's installation- // response identifies this token's app without choosing an environment.- if (- !resolvedAppSlug &&- typeof installation.app_slug === "string" &&- /^[a-z0-9-]{1,100}$/.test(installation.app_slug)- ) {- resolvedAppSlug = installation.app_slug;- }- if (- installation.repository_selection === "all" ||- installation.repository_selection === "selected"- ) {- selections.add(installation.repository_selection);- }- const account = recordValue(installation.account)- ? installation.account- : null;- if (typeof account?.login === "string") owners.add(account.login);- const managementUrl = githubInstallationManagementUrl(- installation.html_url,- );- if (managementUrl) managementUrls.add(managementUrl);- for (const repository of await list(- `/user/installations/${installationId}/repositories`,- "repositories",- )) {- const id = githubId(repository.id);- const fullName =- typeof repository.full_name === "string" ? repository.full_name : "";- if (- !id ||- !/^[A-Za-z0-9][A-Za-z0-9-]*\/(?!\.{1,2}$)[A-Za-z0-9_.-]+$/.test(- fullName,- )- ) {- throw unprocessable("GitHub returned invalid repository metadata", {- code: "github_bad_response",- });- }- repositories.set(id, {- id,- fullName,- installationId,- ...(gitHubRepositoryOwnerType(repository) ? { ownerType: gitHubRepositoryOwnerType(repository) } : {}),- ...(typeof repository.private === "boolean"- ? { private: repository.private }- : {}),- });- }- }- const repositoryCount = repositories.size;- if (installationIds.length === 0 || repositoryCount === 0) {- const installationUrl = resolvedAppSlug- ? `https://github.com/apps/${resolvedAppSlug}/installations/new`- : "https://github.com/settings/installations";- throw unprocessable(- "GitHub access is required. Install Paperclip and grant at least one repository before refreshing access.",- {- code: "github_installation_required",- installationUrl,- managementUrl: "https://github.com/settings/installations",- },- );- }- const installationUrl = resolvedAppSlug- ? `https://github.com/apps/${resolvedAppSlug}/installations/new`- : "https://github.com/settings/installations";- return {- userId,- login,- ...(typeof user.avatar_url === "string"- ? { avatarUrl: user.avatar_url }- : {}),- installationCount: installationIds.length,- repositoryCount,- repositorySelection:- selections.size > 1- ? "mixed"- : (selections.values().next().value ?? "none"),- installationIds,- installationOwnerLogins: [...owners],- repositories: [...repositories.values()].sort((a, b) =>- a.fullName.localeCompare(b.fullName),- ),- installationUrl,- managementUrl:- managementUrls.size === 1- ? managementUrls.values().next().value!- : "https://github.com/settings/installations",- ...(resolvedAppSlug ? { appSlug: resolvedAppSlug } : {}),- accessRevision: randomUUID(),- lastAccessRefreshAt: accessRefreshStartedAt,- webhookHealth: "pending",- };-}--function githubInstallationManagementUrl(value: unknown): string | null {- if (typeof value !== "string" || value.length > 2_000) return null;- try {- const url = new URL(value);- if (- url.protocol !== "https:" ||- url.hostname.toLowerCase() !== "github.com"- )- return null;- return url.pathname.includes("/settings/installations/")- ? url.toString()- : null;- } catch {- return null;- }-}--function githubId(value: unknown): string | null {- if (typeof value === "number" && Number.isSafeInteger(value) && value > 0)- return String(value);- if (typeof value === "string" && /^[1-9][0-9]{0,30}$/.test(value))- return value;- return null;-}--function recordValue(value: unknown): value is Record<string, unknown> {- return typeof value === "object" && value !== null && !Array.isArray(value);-}/**diff --git a/server/src/services/tool-access/github-access-discovery.ts b/server/src/services/tool-access/github-access-discovery.tsnew file mode 100644index 0000000000..05083337fd--- /dev/null+++ b/server/src/services/tool-access/github-access-discovery.ts@@ -0,0 +1,294 @@+import { randomUUID } from "node:crypto";+import { unprocessable } from "../../errors.js";++function gitHubRepositoryOwnerType(repository: Record<string, unknown>) {+ const type = recordValue(repository.owner) ? repository.owner.type : undefined;+ return type === "Organization" ? "organization" as const+ : type === "User" ? "personal" as const : undefined;+}++export async function loadGitHubTokenRepositories(+ headers: Record<string, string>,+ request: typeof fetch = fetch,+) {+ const repositories: Array<{+ id: string;+ fullName: string;+ ownerType?: "personal" | "organization";+ private?: boolean;+ }> = [];+ for (let page = 1; ; page += 1) {+ const response = await request(+ `https://api.github.com/user/repos?per_page=100&page=${page}`,+ {+ headers: {+ ...headers,+ accept: "application/vnd.github+json",+ "user-agent": "Paperclip",+ "x-github-api-version": "2022-11-28",+ },+ signal: AbortSignal.timeout(15_000),+ },+ );+ if (!response.ok)+ throw unprocessable(+ "Could not load GitHub repositories. Reconnect GitHub and try again.",+ );+ const rows: unknown = await response.json();+ if (!Array.isArray(rows))+ throw unprocessable("GitHub returned invalid repositories");+ for (const row of rows) {+ if (+ !recordValue(row) ||+ !githubId(row.id) ||+ typeof row.full_name !== "string" ||+ !/^[A-Za-z0-9][A-Za-z0-9-]*\/(?!\.{1,2}$)[A-Za-z0-9_.-]+$/.test(+ row.full_name,+ )+ ) {+ throw unprocessable("GitHub returned invalid repository metadata");+ }+ repositories.push({+ id: githubId(row.id)!,+ fullName: row.full_name,+ ...(gitHubRepositoryOwnerType(row) ? { ownerType: gitHubRepositoryOwnerType(row) } : {}),+ ...(typeof row.private === "boolean" ? { private: row.private } : {}),+ });+ }+ if (!/;\s*rel="next"/.test(response.headers.get("link") ?? ""))+ return repositories;+ if (!rows.length) throw unprocessable("GitHub returned invalid pagination");+ }+}++export async function loadGitHubGrantMetadata(+ accessToken: string,+ request: typeof fetch = fetch,+ appSlug?: string,+): Promise<{+ userId: string;+ login: string;+ avatarUrl?: string;+ installationCount: number;+ repositoryCount: number;+ repositorySelection: "all" | "selected" | "mixed" | "none";+ installationIds: string[];+ installationOwnerLogins: string[];+ repositories: Array<{+ id: string;+ fullName: string;+ installationId: string;+ ownerType?: "personal" | "organization";+ private?: boolean;+ }>;+ installationUrl: string;+ managementUrl: string;+ appSlug?: string;+ accessRevision: string;+ lastAccessRefreshAt: string;+ webhookHealth: "pending";+}> {+ let resolvedAppSlug = appSlug;+ const accessRefreshStartedAt = new Date().toISOString();+ const github = async (+ path: string,+ ): Promise<{ data: Record<string, unknown>; hasNext: boolean }> => {+ const response = await request(`https://api.github.com${path}`, {+ headers: {+ accept: "application/vnd.github+json",+ authorization: `Bearer ${accessToken}`,+ "user-agent": "Paperclip",+ "x-github-api-version": "2022-11-28",+ },+ signal: AbortSignal.timeout(15_000),+ });+ if (!response.ok) {+ throw unprocessable(+ "GitHub could not verify this account's installation access",+ {+ code:+ response.status === 401+ ? "oauth_reauthorization_required"+ : "github_access_check_failed",+ },+ );+ }+ const value = (await response.json()) as unknown;+ if (!recordValue(value))+ throw unprocessable("GitHub returned invalid account metadata", {+ code: "github_bad_response",+ });+ return {+ data: value,+ hasNext: /;\s*rel="next"/.test(response.headers.get("link") ?? ""),+ };+ };+ const list = async (+ path: string,+ key: string,+ ): Promise<Record<string, unknown>[]> => {+ const items: Record<string, unknown>[] = [];+ for (let page = 1; ; page += 1) {+ const { data, hasNext } = await github(+ `${path}?per_page=100&page=${page}`,+ );+ const batch = data[key];+ if (+ !Array.isArray(batch) ||+ !batch.every(recordValue) ||+ (hasNext && batch.length === 0)+ ) {+ throw unprocessable("GitHub returned invalid access metadata", {+ code: "github_bad_response",+ });+ }+ items.push(...batch);+ if (!hasNext) return items;+ }+ };+ const { data: user } = await github("/user");+ const userId = githubId(user.id);+ const login = typeof user.login === "string" ? user.login : null;+ if (!userId || !login)+ throw unprocessable("GitHub returned invalid account metadata", {+ code: "github_bad_response",+ });+ const installations = await list("/user/installations", "installations");+ const installationIds: string[] = [];+ const owners = new Set<string>();+ const selections = new Set<"all" | "selected">();+ const managementUrls = new Set<string>();+ const repositories = new Map<+ string,+ { id: string; fullName: string; installationId: string; ownerType?: "personal" | "organization"; private?: boolean }+ >();+ for (const installation of installations) {+ const installationId = githubId(installation.id);+ if (!installationId) continue;+ installationIds.push(installationId);+ // Older grants predate the broker's appSlug field. GitHub's installation+ // response identifies this token's app without choosing an environment.+ if (+ !resolvedAppSlug &&+ typeof installation.app_slug === "string" &&+ /^[a-z0-9-]{1,100}$/.test(installation.app_slug)+ ) {+ resolvedAppSlug = installation.app_slug;+ }+ if (+ installation.repository_selection === "all" ||+ installation.repository_selection === "selected"+ ) {+ selections.add(installation.repository_selection);+ }+ const account = recordValue(installation.account)+ ? installation.account+ : null;+ if (typeof account?.login === "string") owners.add(account.login);+ const managementUrl = githubInstallationManagementUrl(+ installation.html_url,+ );+ if (managementUrl) managementUrls.add(managementUrl);+ for (const repository of await list(+ `/user/installations/${installationId}/repositories`,+ "repositories",+ )) {+ const id = githubId(repository.id);+ const fullName =+ typeof repository.full_name === "string" ? repository.full_name : "";+ if (+ !id ||+ !/^[A-Za-z0-9][A-Za-z0-9-]*\/(?!\.{1,2}$)[A-Za-z0-9_.-]+$/.test(+ fullName,+ )+ ) {+ throw unprocessable("GitHub returned invalid repository metadata", {+ code: "github_bad_response",+ });+ }+ repositories.set(id, {+ id,+ fullName,+ installationId,+ ...(gitHubRepositoryOwnerType(repository) ? { ownerType: gitHubRepositoryOwnerType(repository) } : {}),+ ...(typeof repository.private === "boolean"+ ? { private: repository.private }+ : {}),+ });+ }+ }+ const repositoryCount = repositories.size;+ if (installationIds.length === 0 || repositoryCount === 0) {+ const installationUrl = resolvedAppSlug+ ? `https://github.com/apps/${resolvedAppSlug}/installations/new`+ : "https://github.com/settings/installations";+ throw unprocessable(+ "GitHub access is required. Install Paperclip and grant at least one repository before refreshing access.",+ {+ code: "github_installation_required",+ installationUrl,+ managementUrl: "https://github.com/settings/installations",+ },+ );+ }+ const installationUrl = resolvedAppSlug+ ? `https://github.com/apps/${resolvedAppSlug}/installations/new`+ : "https://github.com/settings/installations";+ return {+ userId,+ login,+ ...(typeof user.avatar_url === "string"+ ? { avatarUrl: user.avatar_url }+ : {}),+ installationCount: installationIds.length,+ repositoryCount,+ repositorySelection:+ selections.size > 1+ ? "mixed"+ : (selections.values().next().value ?? "none"),+ installationIds,+ installationOwnerLogins: [...owners],+ repositories: [...repositories.values()].sort((a, b) =>+ a.fullName.localeCompare(b.fullName),+ ),+ installationUrl,+ managementUrl:+ managementUrls.size === 1+ ? managementUrls.values().next().value!+ : "https://github.com/settings/installations",+ ...(resolvedAppSlug ? { appSlug: resolvedAppSlug } : {}),+ accessRevision: randomUUID(),+ lastAccessRefreshAt: accessRefreshStartedAt,+ webhookHealth: "pending",+ };+}++function githubInstallationManagementUrl(value: unknown): string | null {+ if (typeof value !== "string" || value.length > 2_000) return null;+ try {+ const url = new URL(value);+ if (+ url.protocol !== "https:" ||+ url.hostname.toLowerCase() !== "github.com"+ )+ return null;+ return url.pathname.includes("/settings/installations/")+ ? url.toString()+ : null;+ } catch {+ return null;+ }+}++function githubId(value: unknown): string | null {+ if (typeof value === "number" && Number.isSafeInteger(value) && value > 0)+ return String(value);+ if (typeof value === "string" && /^[1-9][0-9]{0,30}$/.test(value))+ return value;+ return null;+}++function recordValue(value: unknown): value is Record<string, unknown> {+ return typeof value === "object" && value !== null && !Array.isArray(value);+}+