Skip to content

Composio powers 1000+ toolkits, tool search, context management, authentication, and a sandboxed workbench to help you build AI agents that turn intent into action.

fix(docs): paginate toolkits fetch so the full catalog reaches the site (#3745)

Alberto Schiabel8e8ce95eb3fb
diff --git a/.github/workflows/docs-update-data.yml b/.github/workflows/docs-update-data.yml
index f271668b0..96e866809 100644
--- a/.github/workflows/docs-update-data.yml
+++ b/.github/workflows/docs-update-data.yml
@@ -96,14 +96,14 @@ jobs:
id: changes
run: |
cd ..
- git add -N docs/public/data/ docs/public/openapi.json docs/public/openapi-v3.json docs/content/reference/api-reference/ docs/content/reference/v3/api-reference/ docs/content/reference/meta-tools/ 2>/dev/null || true
- if git diff --quiet docs/public/data/ docs/public/openapi.json docs/public/openapi-v3.json docs/content/reference/api-reference/ docs/content/reference/v3/api-reference/ docs/content/reference/meta-tools/ 2>/dev/null; then
+ git add -N docs/public/data/ docs/public/openapi.json docs/public/openapi-v3.json docs/content/reference/api-reference/ docs/content/reference/v3/api-reference/ docs/content/toolkits/meta-tools/ 2>/dev/null || true
+ if git diff --quiet docs/public/data/ docs/public/openapi.json docs/public/openapi-v3.json docs/content/reference/api-reference/ docs/content/reference/v3/api-reference/ docs/content/toolkits/meta-tools/ 2>/dev/null; then
echo "has_changes=false" >> "$GITHUB_OUTPUT"
echo "No changes detected"
else
echo "has_changes=true" >> "$GITHUB_OUTPUT"
echo "Changes detected:"
- git diff --stat docs/public/data/ docs/public/openapi.json docs/public/openapi-v3.json docs/content/reference/api-reference/ docs/content/reference/v3/api-reference/ docs/content/reference/meta-tools/ 2>/dev/null || true
+ git diff --stat docs/public/data/ docs/public/openapi.json docs/public/openapi-v3.json docs/content/reference/api-reference/ docs/content/reference/v3/api-reference/ docs/content/toolkits/meta-tools/ 2>/dev/null || true
fi
- name: Create Pull Request
@@ -122,7 +122,7 @@ jobs:
- **Toolkit catalog** (`docs/public/data/toolkits.json`, `toolkits-list.json`) — refreshed list of available toolkits, auth schemes, and tools from the backend API
- **OpenAPI specs** (`docs/public/openapi.json`, `docs/public/openapi-v3.json`) — latest v3.1 and v3.0 API specifications fetched from staging
- **API reference pages** (`docs/content/reference/api-reference/`, `docs/content/reference/v3/api-reference/`) — regenerated index pages for both API versions
- - **Meta tools reference** (`docs/public/data/meta-tools.json`, `docs/content/reference/meta-tools/*.mdx`) — updated meta tool schemas and reference docs
+ - **Meta tools reference** (`docs/public/data/meta-tools.json`, `docs/content/toolkits/meta-tools/*.mdx`) — updated meta tool schemas and reference docs
branch: docs/auto-update-data
base: next
add-paths: |
@@ -131,7 +131,7 @@ jobs:
docs/public/openapi-v3.json
docs/content/reference/api-reference/
docs/content/reference/v3/api-reference/
- docs/content/reference/meta-tools/
+ docs/content/toolkits/meta-tools/
- name: Request review from trigger actor
if: steps.create-pr.outputs.pull-request-number
diff --git a/docs/scripts/README.md b/docs/scripts/README.md
index 07dfbc2e5..17c0edf14 100644
--- a/docs/scripts/README.md
+++ b/docs/scripts/README.md
@@ -9,8 +9,9 @@ responses, honors the `Retry-After` header when present, otherwise falls back to
exponential backoff with jitter, and caps attempts so CI still fails fast when
the backend is genuinely down.
-This matters because `generate-toolkits.ts` issues ~3000 requests per run
-(2 + 3 per toolkit), which exceeds the staging limit of 2000 requests/minute.
+This matters because `generate-toolkits.ts` issues ~6500 requests per run
+(a few catalog pages + 3 per toolkit across a ~2.1k catalog), which exceeds the
+staging limit of 2000 requests/minute.
Before this helper, runs failed with `429`, and `generate-meta-tools.ts` — which
runs immediately after — inherited the exhausted rate-limit window.
diff --git a/docs/scripts/fetch-with-retry.ts b/docs/scripts/fetch-with-retry.ts
index 24cad7cd2..cf883e1be 100644
--- a/docs/scripts/fetch-with-retry.ts
+++ b/docs/scripts/fetch-with-retry.ts
@@ -13,8 +13,9 @@
* Network-level errors thrown by `fetch` (DNS, connection reset, …) are retried
* the same way and re-thrown once attempts are exhausted.
*
- * Why this exists: `generate-toolkits.ts` issues ~3000 requests per run, which
- * exceeds the staging limit of 2000 requests/minute. Without backoff the run
+ * Why this exists: `generate-toolkits.ts` issues ~6500 requests per run (3 per
+ * toolkit across a ~2.1k catalog), which exceeds the staging limit of 2000
+ * requests/minute. Without backoff the run
* fails with `429`, and `generate-meta-tools.ts` (which runs next) inherits the
* exhausted window. See docs/scripts/README.md.
*
diff --git a/docs/scripts/generate-toolkits.ts b/docs/scripts/generate-toolkits.ts
index b1bf1ae77..aa9bfc378 100644
--- a/docs/scripts/generate-toolkits.ts
+++ b/docs/scripts/generate-toolkits.ts
@@ -74,22 +74,57 @@ interface Toolkit {
authConfigDetails?: AuthConfigDetail[];
}
+// The backend silently caps `limit` at 1000 per page and defaults to usage
+// ordering, so a single request returns only the top-1000 toolkits — about half
+// the catalog. Request the cap and follow `next_cursor` until exhausted.
+// MAX_PAGES is a runaway guard well above the real catalog (~2.1k → 3 pages).
+const TOOLKITS_PAGE_LIMIT = 1000;
+const TOOLKITS_MAX_PAGES = 12;
+
async function fetchToolkits(): Promise<any[]> {
console.log('Fetching toolkits from API...');
- const response = await fetchWithRetry(`${API_BASE}/toolkits`, {
- headers: {
- 'Content-Type': 'application/json',
- 'x-api-key': API_KEY!,
- },
- });
+ const items: any[] = [];
+ // Pages can overlap when the catalog shifts between cursor fetches; keep the
+ // first occurrence so API-provided ordering stays stable.
+ const seen = new Set<string>();
+ let cursor: string | undefined;
- if (!response.ok) {
- throw new Error(`Failed to fetch toolkits: ${response.status} ${response.statusText}`);
+ for (let page = 0; page < TOOLKITS_MAX_PAGES; page++) {
+ const params = new URLSearchParams({ limit: String(TOOLKITS_PAGE_LIMIT) });
+ if (cursor) params.set('cursor', cursor);
+
+ const response = await fetchWithRetry(`${API_BASE}/toolkits?${params}`, {
+ headers: {
+ 'Content-Type': 'application/json',
+ 'x-api-key': API_KEY!,
+ },
+ });
+
+ if (!response.ok) {
+ throw new Error(`Failed to fetch toolkits: ${response.status} ${response.statusText}`);
+ }
+
+ const data = await response.json();
+ const pageItems: any[] = data.items || data;
+ for (const item of pageItems) {
+ const slug = item?.slug?.toLowerCase();
+ if (slug) {
+ if (seen.has(slug)) continue;
+ seen.add(slug);
+ }
+ items.push(item);
+ }
+
+ cursor = data.next_cursor ?? undefined;
+ if (!cursor) return items;
}
- const data = await response.json();
- return data.items || data;
+ // Failing beats silently publishing a truncated catalog — that is the exact
+ // bug this pagination loop exists to prevent.
+ throw new Error(
+ `Toolkit catalog exceeds ${TOOLKITS_MAX_PAGES} pages of ${TOOLKITS_PAGE_LIMIT}; raise TOOLKITS_MAX_PAGES`
+ );
}
async function fetchToolkitChangelog(): Promise<Map<string, string>> {