Composio powers 1000+ toolkits, tool search, context management, authentication, and a sandboxed workbench to help you build AI agents that turn intent into action.
TypeScript agentic-aiagentsaiai-agentsaiagentsdeveloper-toolsfunction-callinggpt-4javascriptjsllmllmopsmcppythonremote-mcp-serverssetypescript
fix(docs): paginate toolkits fetch so the full catalog reaches the site (#3745)
Alberto Schiabel8e8ce95eb3fb
diff --git a/.github/workflows/docs-update-data.yml b/.github/workflows/docs-update-data.ymlindex f271668b0..96e866809 100644--- a/.github/workflows/docs-update-data.yml+++ b/.github/workflows/docs-update-data.yml@@ -96,14 +96,14 @@ jobs:id: changesrun: |cd ..- git add -N docs/public/data/ docs/public/openapi.json docs/public/openapi-v3.json docs/content/reference/api-reference/ docs/content/reference/v3/api-reference/ docs/content/reference/meta-tools/ 2>/dev/null || true- if git diff --quiet docs/public/data/ docs/public/openapi.json docs/public/openapi-v3.json docs/content/reference/api-reference/ docs/content/reference/v3/api-reference/ docs/content/reference/meta-tools/ 2>/dev/null; then+ git add -N docs/public/data/ docs/public/openapi.json docs/public/openapi-v3.json docs/content/reference/api-reference/ docs/content/reference/v3/api-reference/ docs/content/toolkits/meta-tools/ 2>/dev/null || true+ if git diff --quiet docs/public/data/ docs/public/openapi.json docs/public/openapi-v3.json docs/content/reference/api-reference/ docs/content/reference/v3/api-reference/ docs/content/toolkits/meta-tools/ 2>/dev/null; thenecho "has_changes=false" >> "$GITHUB_OUTPUT"echo "No changes detected"elseecho "has_changes=true" >> "$GITHUB_OUTPUT"echo "Changes detected:"- git diff --stat docs/public/data/ docs/public/openapi.json docs/public/openapi-v3.json docs/content/reference/api-reference/ docs/content/reference/v3/api-reference/ docs/content/reference/meta-tools/ 2>/dev/null || true+ git diff --stat docs/public/data/ docs/public/openapi.json docs/public/openapi-v3.json docs/content/reference/api-reference/ docs/content/reference/v3/api-reference/ docs/content/toolkits/meta-tools/ 2>/dev/null || truefi- name: Create Pull Request@@ -122,7 +122,7 @@ jobs:- **Toolkit catalog** (`docs/public/data/toolkits.json`, `toolkits-list.json`) — refreshed list of available toolkits, auth schemes, and tools from the backend API- **OpenAPI specs** (`docs/public/openapi.json`, `docs/public/openapi-v3.json`) — latest v3.1 and v3.0 API specifications fetched from staging- **API reference pages** (`docs/content/reference/api-reference/`, `docs/content/reference/v3/api-reference/`) — regenerated index pages for both API versions- - **Meta tools reference** (`docs/public/data/meta-tools.json`, `docs/content/reference/meta-tools/*.mdx`) — updated meta tool schemas and reference docs+ - **Meta tools reference** (`docs/public/data/meta-tools.json`, `docs/content/toolkits/meta-tools/*.mdx`) — updated meta tool schemas and reference docsbranch: docs/auto-update-database: nextadd-paths: |@@ -131,7 +131,7 @@ jobs:docs/public/openapi-v3.jsondocs/content/reference/api-reference/docs/content/reference/v3/api-reference/- docs/content/reference/meta-tools/+ docs/content/toolkits/meta-tools/- name: Request review from trigger actorif: steps.create-pr.outputs.pull-request-numberdiff --git a/docs/scripts/README.md b/docs/scripts/README.mdindex 07dfbc2e5..17c0edf14 100644--- a/docs/scripts/README.md+++ b/docs/scripts/README.md@@ -9,8 +9,9 @@ responses, honors the `Retry-After` header when present, otherwise falls back toexponential backoff with jitter, and caps attempts so CI still fails fast whenthe backend is genuinely down.-This matters because `generate-toolkits.ts` issues ~3000 requests per run-(2 + 3 per toolkit), which exceeds the staging limit of 2000 requests/minute.+This matters because `generate-toolkits.ts` issues ~6500 requests per run+(a few catalog pages + 3 per toolkit across a ~2.1k catalog), which exceeds the+staging limit of 2000 requests/minute.Before this helper, runs failed with `429`, and `generate-meta-tools.ts` — whichruns immediately after — inherited the exhausted rate-limit window.diff --git a/docs/scripts/fetch-with-retry.ts b/docs/scripts/fetch-with-retry.tsindex 24cad7cd2..cf883e1be 100644--- a/docs/scripts/fetch-with-retry.ts+++ b/docs/scripts/fetch-with-retry.ts@@ -13,8 +13,9 @@* Network-level errors thrown by `fetch` (DNS, connection reset, …) are retried* the same way and re-thrown once attempts are exhausted.*- * Why this exists: `generate-toolkits.ts` issues ~3000 requests per run, which- * exceeds the staging limit of 2000 requests/minute. Without backoff the run+ * Why this exists: `generate-toolkits.ts` issues ~6500 requests per run (3 per+ * toolkit across a ~2.1k catalog), which exceeds the staging limit of 2000+ * requests/minute. Without backoff the run* fails with `429`, and `generate-meta-tools.ts` (which runs next) inherits the* exhausted window. See docs/scripts/README.md.*diff --git a/docs/scripts/generate-toolkits.ts b/docs/scripts/generate-toolkits.tsindex b1bf1ae77..aa9bfc378 100644--- a/docs/scripts/generate-toolkits.ts+++ b/docs/scripts/generate-toolkits.ts@@ -74,22 +74,57 @@ interface Toolkit {authConfigDetails?: AuthConfigDetail[];}+// The backend silently caps `limit` at 1000 per page and defaults to usage+// ordering, so a single request returns only the top-1000 toolkits — about half+// the catalog. Request the cap and follow `next_cursor` until exhausted.+// MAX_PAGES is a runaway guard well above the real catalog (~2.1k → 3 pages).+const TOOLKITS_PAGE_LIMIT = 1000;+const TOOLKITS_MAX_PAGES = 12;+async function fetchToolkits(): Promise<any[]> {console.log('Fetching toolkits from API...');- const response = await fetchWithRetry(`${API_BASE}/toolkits`, {- headers: {- 'Content-Type': 'application/json',- 'x-api-key': API_KEY!,- },- });+ const items: any[] = [];+ // Pages can overlap when the catalog shifts between cursor fetches; keep the+ // first occurrence so API-provided ordering stays stable.+ const seen = new Set<string>();+ let cursor: string | undefined;- if (!response.ok) {- throw new Error(`Failed to fetch toolkits: ${response.status} ${response.statusText}`);+ for (let page = 0; page < TOOLKITS_MAX_PAGES; page++) {+ const params = new URLSearchParams({ limit: String(TOOLKITS_PAGE_LIMIT) });+ if (cursor) params.set('cursor', cursor);++ const response = await fetchWithRetry(`${API_BASE}/toolkits?${params}`, {+ headers: {+ 'Content-Type': 'application/json',+ 'x-api-key': API_KEY!,+ },+ });++ if (!response.ok) {+ throw new Error(`Failed to fetch toolkits: ${response.status} ${response.statusText}`);+ }++ const data = await response.json();+ const pageItems: any[] = data.items || data;+ for (const item of pageItems) {+ const slug = item?.slug?.toLowerCase();+ if (slug) {+ if (seen.has(slug)) continue;+ seen.add(slug);+ }+ items.push(item);+ }++ cursor = data.next_cursor ?? undefined;+ if (!cursor) return items;}- const data = await response.json();- return data.items || data;+ // Failing beats silently publishing a truncated catalog — that is the exact+ // bug this pagination loop exists to prevent.+ throw new Error(+ `Toolkit catalog exceeds ${TOOLKITS_MAX_PAGES} pages of ${TOOLKITS_PAGE_LIMIT}; raise TOOLKITS_MAX_PAGES`+ );}async function fetchToolkitChangelog(): Promise<Map<string, string>> {