Composio powers 1000+ toolkits, tool search, context management, authentication, and a sandboxed workbench to help you build AI agents that turn intent into action.
TypeScript agentic-aiagentsaiai-agentsaiagentsdeveloper-toolsfunction-callinggpt-4javascriptjsllmllmopsmcppythonremote-mcp-serverssetypescript
docs: migrate search to Algolia (#3452)
Rahul Tarak7bd3e0669ceb
diff --git a/.github/workflows/docs-search-sync.yml b/.github/workflows/docs-search-sync.ymlnew file mode 100644index 000000000..4b36ce82c--- /dev/null+++ b/.github/workflows/docs-search-sync.yml@@ -0,0 +1,46 @@+name: Docs - Sync Algolia Search++on:+ push:+ branches:+ - next+ paths:+ - 'docs/**'+ workflow_dispatch:++permissions:+ contents: read++jobs:+ sync-search:+ runs-on: ubuntu-latest+ defaults:+ run:+ working-directory: ./docs+ env:+ ALGOLIA_APP_ID: ${{ vars.ALGOLIA_APP_ID || '62HI9PQZ1L' }}+ ALGOLIA_ADMIN_API_KEY: ${{ secrets.ALGOLIA_ADMIN_API_KEY }}+ ALGOLIA_INDEX_NAME: ${{ vars.ALGOLIA_INDEX_NAME || 'docs_composio_dev_62hi9pqz1l_pages' }}++ steps:+ - name: Checkout repository+ uses: actions/checkout@v6++ - name: Setup Bun+ uses: oven-sh/setup-bun@v2+ with:+ bun-version-file: '.bun-version'++ - name: Install dependencies+ run: bun install++ - name: Validate search index generation+ run: bun run sync:search --dry-run++ - name: Sync Algolia search index+ if: env.ALGOLIA_APP_ID != '' && env.ALGOLIA_ADMIN_API_KEY != ''+ run: bun run sync:search++ - name: Skip sync without Algolia secrets+ if: env.ALGOLIA_APP_ID == '' || env.ALGOLIA_ADMIN_API_KEY == ''+ run: echo "Skipping Algolia sync because ALGOLIA_APP_ID or ALGOLIA_ADMIN_API_KEY is not configured."diff --git a/docs/CLAUDE.md b/docs/CLAUDE.mdindex 4862683d2..2706420ca 100644--- a/docs/CLAUDE.md+++ b/docs/CLAUDE.md@@ -9,6 +9,7 @@ bun install # Install dependenciesbun run dev # Dev server (http://localhost:3000)bun run build # Production build (validates TS code blocks)bun run types:check # Type check+bun run sync:search # Sync Algolia docs search index (requires Algolia env vars)```## Project Structure@@ -83,6 +84,12 @@ Detailed documentation for Claude is organized in `.claude/`:`content/docs/glossary.mdx` defines key Composio terms (auth config, session, toolkit, etc.) using `<Glossary>` and `<GlossaryTerm name="...">` components (`components/glossary.tsx`). The component renders a filterable two-column table. The markdown converter in `lib/source.ts` converts `<GlossaryTerm>` tags to `### Term` headings for LLM-friendly output. When adding new Composio concepts, add a `<GlossaryTerm>` entry and update the `keywords` frontmatter array.+## Search++The docs search dialog uses Algolia in production when `NEXT_PUBLIC_ALGOLIA_SEARCH_API_KEY` is configured. `NEXT_PUBLIC_ALGOLIA_APP_ID` defaults to `62HI9PQZ1L`, and `NEXT_PUBLIC_ALGOLIA_INDEX_NAME` defaults to `docs_composio_dev_62hi9pqz1l_pages`. Without a search API key it falls back to `/api/search` for local development and tests. Algolia searches request `clickAnalytics`, and the client sends search result view/click events with `search-insights`.++Run `ALGOLIA_APP_ID=62HI9PQZ1L ALGOLIA_ADMIN_API_KEY=... ALGOLIA_INDEX_NAME=docs_composio_dev_62hi9pqz1l_pages bun run sync:search` from `docs/` to rebuild the Algolia index.+## API VersioningThe API reference supports v3.1 (latest, default) and v3.0 side-by-side:diff --git a/docs/README.md b/docs/README.mdindex 6da6874e2..23f0915a8 100644--- a/docs/README.md+++ b/docs/README.md@@ -74,6 +74,24 @@ All TypeScript code blocks in MDX files are **type-checked at build time** usingSee `CLAUDE.md` for detailed patterns and troubleshooting.+## Search++Docs search uses Algolia when `NEXT_PUBLIC_ALGOLIA_SEARCH_API_KEY` is set; otherwise it falls back to the local Fumadocs `/api/search` endpoint for development and tests. The Algolia client requests `clickAnalytics` and sends search result view/click events with `search-insights`.++```bash+NEXT_PUBLIC_ALGOLIA_APP_ID=62HI9PQZ1L # optional; default shown+NEXT_PUBLIC_ALGOLIA_SEARCH_API_KEY=...+NEXT_PUBLIC_ALGOLIA_INDEX_NAME=docs_composio_dev_62hi9pqz1l_pages # optional; default shown+```++Sync the Algolia index with an admin key:++```bash+ALGOLIA_APP_ID=62HI9PQZ1L+ALGOLIA_ADMIN_API_KEY=...+ALGOLIA_INDEX_NAME=docs_composio_dev_62hi9pqz1l_pages bun run sync:search+```+## Commands| Command | Description |@@ -81,4 +99,5 @@ See `CLAUDE.md` for detailed patterns and troubleshooting.| `bun run dev` | Dev server || `bun run build` | Production build (validates TS code blocks) || `bun run types:check` | Type check |+| `bun run sync:search` | Sync docs search records to Algolia |diff --git a/docs/app/api/search/route.ts b/docs/app/api/search/route.tsindex ba05af37b..be0355baa 100644--- a/docs/app/api/search/route.ts+++ b/docs/app/api/search/route.ts@@ -1,89 +1,8 @@-// Use direct imports from collections to avoid top-level await in lib/source.ts-import { docs, reference, cookbooks, toolkits, changelog } from 'fumadocs-mdx:collections/server';import { createSearchAPI } from 'fumadocs-core/search/server';-import { loader, multiple } from 'fumadocs-core/source';-import { lucideIconsPlugin } from 'fumadocs-core/source/lucide-icons';-import { openapiSource, openapiPlugin } from 'fumadocs-openapi/server';-import { openapi } from '@/lib/openapi';-import { getAllToolkitsSync } from '@/lib/toolkit-data';+import { getDocsSearchIndexes } from '@/lib/search-index';-// Create loaders directly here to avoid the problematic lib/source.ts import-const docsSource = loader({- baseUrl: '/docs',- source: docs.toFumadocsSource(),- plugins: [lucideIconsPlugin()],-});--const cookbooksSource = loader({- baseUrl: '/cookbooks',- source: cookbooks.toFumadocsSource(),- plugins: [lucideIconsPlugin()],-});--const toolkitsSource = loader({- baseUrl: '/toolkits',- source: toolkits.toFumadocsSource(),- plugins: [lucideIconsPlugin()],-});--// Dynamic toolkit entries from toolkits.json-const mdxToolkitSlugs = new Set(- toolkitsSource.getPages().map((page) => page.slugs.join('/')),-);--const dynamicToolkitIndexes = getAllToolkitsSync()- .filter((toolkit) => !mdxToolkitSlugs.has(toolkit.slug))- .map((toolkit) => ({- id: `/toolkits/${toolkit.slug}`,- title: toolkit.name,- description: toolkit.description,- url: `/toolkits/${toolkit.slug}`,- structuredData: { headings: [] as never[], contents: [] as never[] },- keywords: [toolkit.slug, toolkit.category].filter(Boolean) as string[],- }));--// Changelog entries-const changelogIndexes = changelog.map((entry) => ({- id: `/docs/changelog/${entry.date.replace(/-/g, '/')}#${entry.title}`,- title: entry.title,- description: entry.description ?? '',- url: `/docs/changelog/${entry.date.replace(/-/g, '/')}`,- structuredData: { headings: [] as never[], contents: [] as never[] },- keywords: ['changelog'],-}));--// Use dynamic indexes to support async OpenAPI page loading+// Fallback/local search endpoint. Production search uses Algolia from the client+// when NEXT_PUBLIC_ALGOLIA_* variables are configured.export const { GET } = createSearchAPI('advanced', {- indexes: async () => {- // Load OpenAPI pages and build full reference source- const openapiPages = await openapiSource(openapi, {- groupBy: 'tag',- baseDir: 'api-reference',- });-- const fullReferenceSource = loader({- baseUrl: '/reference',- source: multiple({- mdx: reference.toFumadocsSource(),- openapi: openapiPages,- }),- plugins: [lucideIconsPlugin(), openapiPlugin()],- });-- const mdxIndexes = [- ...docsSource.getPages(),- ...cookbooksSource.getPages(),- ...toolkitsSource.getPages(),- ...fullReferenceSource.getPages(),- ].map((page) => ({- id: page.url,- title: page.data.title ?? 'Untitled',- description: page.data.description,- url: page.url,- structuredData: page.data.structuredData,- keywords: 'keywords' in page.data ? (page.data.keywords as string[]) : undefined,- }));-- return [...mdxIndexes, ...dynamicToolkitIndexes, ...changelogIndexes];- },+ indexes: getDocsSearchIndexes,});diff --git a/docs/bun.lock b/docs/bun.lockindex 96f1e5674..ba8797694 100644--- a/docs/bun.lock+++ b/docs/bun.lock@@ -1,5 +1,6 @@{"lockfileVersion": 1,+ "configVersion": 0,"workspaces": {"": {"name": "fumadocs",@@ -7,6 +8,7 @@"@radix-ui/react-tabs": "^1.1.13","@shikijs/twoslash": "^3.21.0","@vercel/analytics": "^1.6.1",+ "algoliasearch": "^5","class-variance-authority": "^0.7.1","clsx": "^2.1.1","fumadocs-core": "^16.4.2",@@ -19,6 +21,7 @@"posthog-js": "^1.319.0","react": "^19.2.3","react-dom": "^19.2.3",+ "search-insights": "^2","shiki": "^3.20.0","tailwind-merge": "^3.4.0",},@@ -95,6 +98,34 @@"@ai-sdk/ui-utils-v5": ["@ai-sdk/[email protected]", "", { "dependencies": { "@ai-sdk/provider": "1.1.3", "@ai-sdk/provider-utils": "2.2.8", "zod-to-json-schema": "^3.24.1" }, "peerDependencies": { "zod": "^3.23.8" } }, "sha512-3zcwCc8ezzFlwp3ZD15wAPjf2Au4s3vAbKsXQVyhxODHcmu0iyPO2Eua6D/vicq/AUm/BAo60r97O6HU+EI0+w=="],+ "@algolia/abtesting": ["@algolia/[email protected]", "", { "dependencies": { "@algolia/client-common": "5.52.1", "@algolia/requester-browser-xhr": "5.52.1", "@algolia/requester-fetch": "5.52.1", "@algolia/requester-node-http": "5.52.1" } }, "sha512-aehCadlWOGvrT91KUIZpC0MbB8KBW9yUuvTJFd2xesR7le/IsT4nJUnjCCZ4ZqZCeTcPHPV5mo//fZ5oxcSVYw=="],++ "@algolia/client-abtesting": ["@algolia/[email protected]", "", { "dependencies": { "@algolia/client-common": "5.52.1", "@algolia/requester-browser-xhr": "5.52.1", "@algolia/requester-fetch": "5.52.1", "@algolia/requester-node-http": "5.52.1" } }, "sha512-HmXOGBOAOJPounpBzBpuY0zDYeiCpxgHnQmuA7JO6ScukcBdGp3/XM9zJk5pJx/xNGD68mbPGXWpDxGtl6BwDQ=="],++ "@algolia/client-analytics": ["@algolia/[email protected]", "", { "dependencies": { "@algolia/client-common": "5.52.1", "@algolia/requester-browser-xhr": "5.52.1", "@algolia/requester-fetch": "5.52.1", "@algolia/requester-node-http": "5.52.1" } }, "sha512-5oo4+I8iixie9vXhCyNFCzeIr8pqA3FQ//VsLHTDvZAV4ttYOPGvYHGQq5NSalrLx5Jc3dRro/5uDOlnUMcBJg=="],++ "@algolia/client-common": ["@algolia/[email protected]", "", {}, "sha512-qCDoZfx5MpX7XQzvQ3bC4tSEMkQWQMaF/ABtLuoze03Y/flR563CCSws02qIJ23oX7lxl92LsilZjINVyTdtLw=="],++ "@algolia/client-insights": ["@algolia/[email protected]", "", { "dependencies": { "@algolia/client-common": "5.52.1", "@algolia/requester-browser-xhr": "5.52.1", "@algolia/requester-fetch": "5.52.1", "@algolia/requester-node-http": "5.52.1" } }, "sha512-hnGs0/lsFJ2PWDxNBz7pxreXo/Xz7gxYRcfePBUjsH26ad0kU/sgnVZd9LwWBpsQv65z2jlb5dkyaB9WE9M9FQ=="],++ "@algolia/client-personalization": ["@algolia/[email protected]", "", { "dependencies": { "@algolia/client-common": "5.52.1", "@algolia/requester-browser-xhr": "5.52.1", "@algolia/requester-fetch": "5.52.1", "@algolia/requester-node-http": "5.52.1" } }, "sha512-2VxxNc/uBysyKvGeBdSM5n9eIDKH8kWD7wd9/yqbJAiVwU4Yv6tU1LSJusHKrXV/aCu1KW7t9Gug9QyeEmtn/Q=="],++ "@algolia/client-query-suggestions": ["@algolia/[email protected]", "", { "dependencies": { "@algolia/client-common": "5.52.1", "@algolia/requester-browser-xhr": "5.52.1", "@algolia/requester-fetch": "5.52.1", "@algolia/requester-node-http": "5.52.1" } }, "sha512-O6mPtsw3xEfNOe6gWFpYLeAZAIljNa4Hgna3bq15PwyN7nbjTY0wXJFRbzs/0YVf75Br+SbOQUmjKxXYjDiSiQ=="],++ "@algolia/client-search": ["@algolia/[email protected]", "", { "dependencies": { "@algolia/client-common": "5.52.1", "@algolia/requester-browser-xhr": "5.52.1", "@algolia/requester-fetch": "5.52.1", "@algolia/requester-node-http": "5.52.1" } }, "sha512-gA8oJOV1LnQQkDf91iebNnFInHuW0gRPEgLSOQ7EfipCEjYTHm5swm1DlH9H5RaRw4RrHuzHBegnlzc0MAstcg=="],++ "@algolia/ingestion": ["@algolia/[email protected]", "", { "dependencies": { "@algolia/client-common": "5.52.1", "@algolia/requester-browser-xhr": "5.52.1", "@algolia/requester-fetch": "5.52.1", "@algolia/requester-node-http": "5.52.1" } }, "sha512-U9zZfc5xIu9wRxZkt+HceJUAD4VKHKbAyLSloJdEyMRmphXeibfrY9cxqIXBcmPeZzGhn3Imb35Dq8l19PkJhw=="],++ "@algolia/monitoring": ["@algolia/[email protected]", "", { "dependencies": { "@algolia/client-common": "5.52.1", "@algolia/requester-browser-xhr": "5.52.1", "@algolia/requester-fetch": "5.52.1", "@algolia/requester-node-http": "5.52.1" } }, "sha512-a3SGNceHmkQfq77iG8Ka+w1pvwfZa/0lzEIgse30fL0kD+yKnd/dg0dQvSfFPAEt2f21DMcGkDSSeJlO3KdQjQ=="],++ "@algolia/recommend": ["@algolia/[email protected]", "", { "dependencies": { "@algolia/client-common": "5.52.1", "@algolia/requester-browser-xhr": "5.52.1", "@algolia/requester-fetch": "5.52.1", "@algolia/requester-node-http": "5.52.1" } }, "sha512-z98QEguCFDpxb4S/PyrUK1igqF8tPsdbqOUUO6ON91vJ58w+Gwa6ncrI0oNXSFcrkxA5EqPKPQ2A1PBCn08TYQ=="],++ "@algolia/requester-browser-xhr": ["@algolia/[email protected]", "", { "dependencies": { "@algolia/client-common": "5.52.1" } }, "sha512-CI7+/0I11QeZM59Uc8whd2or0kqzFVjpaPn9Qpwll/krHcBAxk24WkAQ6WX+IwDVMfpont4YGbKwAmCre3vE8Q=="],++ "@algolia/requester-fetch": ["@algolia/[email protected]", "", { "dependencies": { "@algolia/client-common": "5.52.1" } }, "sha512-S6bDuw9byfOvm3T71cgdoZgrgnZq6hpdMLkx52Louh57nUAmvGQESz2aojOynQHjbTiV55smvAFbgn0qT4tJrg=="],++ "@algolia/requester-node-http": ["@algolia/[email protected]", "", { "dependencies": { "@algolia/client-common": "5.52.1" } }, "sha512-tqZXM+54rWo4mk5jL5Z/flE11nPmNEdXwFBM5py9DkOmbjeCNemfVd45FyM97XdzfZ0dl9uOJC6PYn1FpkeyQg=="],+"@alloc/quick-lru": ["@alloc/[email protected]", "", {}, "sha512-UrcABB+4bUrFABwbluTIBErXwvbsU/V7TZWfmbgJfbkwiBuziS9gxdODUyuiecfdGQ85jglMW6juS3+z5TsKLw=="],"@antfu/install-pkg": ["@antfu/[email protected]", "", { "dependencies": { "package-manager-detector": "^1.3.0", "tinyexec": "^1.0.1" } }, "sha512-MGQsmw10ZyI+EJo45CdSER4zEb+p31LpDAFp2Z3gkSd1yqVZGi0Ebx++YTEMonJy4oChEMLsxZ64j8FH6sSqtQ=="],@@ -865,6 +896,8 @@"ajv-formats": ["[email protected]", "", { "dependencies": { "ajv": "^8.0.0" } }, "sha512-8iUql50EUR+uUcdRQ3HDqa6EVyo3docL8g5WJ3FNcWmu62IbkGUue/pEyLBW8VGKKucTPgqeks4fIU1DA4yowQ=="],+ "algoliasearch": ["[email protected]", "", { "dependencies": { "@algolia/abtesting": "1.18.1", "@algolia/client-abtesting": "5.52.1", "@algolia/client-analytics": "5.52.1", "@algolia/client-common": "5.52.1", "@algolia/client-insights": "5.52.1", "@algolia/client-personalization": "5.52.1", "@algolia/client-query-suggestions": "5.52.1", "@algolia/client-search": "5.52.1", "@algolia/ingestion": "1.52.1", "@algolia/monitoring": "1.52.1", "@algolia/recommend": "5.52.1", "@algolia/requester-browser-xhr": "5.52.1", "@algolia/requester-fetch": "5.52.1", "@algolia/requester-node-http": "5.52.1" } }, "sha512-fHA8+kXTbjagw3jkLiaS7KKrH8qe2DyOsiUhGlN4cdT77PEsfqXZl7ewDk1hsg+pJnPlnE50XtLxjR91iJOpmg=="],+"alien-signals": ["[email protected]", "", {}, "sha512-d9dYqZTS90WLiU0I5c6DHj/HcKkF8ZyGN3G5x8wSbslulz70KOxaqCT0hQCo9KOyhVqzqGojvNdJXoTumZOtcw=="],"ansi-regex": ["[email protected]", "", {}, "sha512-Bq3SmSpyFHaWjPk8If9yc6svM8c56dB5BAtW4Qbw5jHTwwXXcTLoRMkpDJp6VL0XzlWaCHTXrkFURMYmD0sLqg=="],@@ -1999,6 +2032,8 @@"scroll-into-view-if-needed": ["[email protected]", "", { "dependencies": { "compute-scroll-into-view": "^3.0.2" } }, "sha512-49oNpRjWRvnU8NyGVmUaYG4jtTkNonFZI86MmGRDqBphEK2EXT9gdEUoQPZhuBM8yWHxCWbobltqYO5M4XrUvQ=="],+ "search-insights": ["[email protected]", "", {}, "sha512-RQPdCYTa8A68uM2jwxoY842xDhvx3E5LFL1LxvxCNMev4o5mLuokczhzjAgGwUZBAmOKZknArSxLKmXtIi2AxQ=="],+"section-matter": ["[email protected]", "", { "dependencies": { "extend-shallow": "^2.0.1", "kind-of": "^6.0.0" } }, "sha512-vfD3pmTzGpufjScBh50YHKzEu2lxBWhVEHsNGoEXmCmn2hKGfeNLYMzCJpe8cD7gqX7TJluOVpBkAequ6dgMmA=="],"secure-json-parse": ["[email protected]", "", {}, "sha512-6aU+Rwsezw7VR8/nyvKTx8QpWH9FrcYiXXlqC4z5d5XQBDRqtbfsRjnwGyqbi3gddNtWHuEk9OANUotL26qKUw=="],diff --git a/docs/components/custom-search-dialog.tsx b/docs/components/custom-search-dialog.tsxindex 388f2c389..0c87ac229 100644--- a/docs/components/custom-search-dialog.tsx+++ b/docs/components/custom-search-dialog.tsx@@ -1,7 +1,9 @@'use client';-import { useEffect, useState } from 'react';+import { useCallback, useEffect, useMemo, useRef, useState } from 'react';import Link from 'next/link';+import { liteClient } from 'algoliasearch/lite';+import aa from 'search-insights';import {SearchDialog,SearchDialogClose,@@ -15,14 +17,20 @@ import {} from 'fumadocs-ui/components/dialog/search';import { useI18n } from 'fumadocs-ui/contexts/i18n';import { useDocsSearch } from 'fumadocs-core/search/client';+import type { BaseIndex } from 'fumadocs-core/search/algolia';import { BotMessageSquare } from 'lucide-react';import { toggleDecimalWidget, detectMac } from './ask-ai-button';function MetaKey() {const [key, setKey] = useState('⌘');+useEffect(() => {- if (!detectMac()) setKey('Ctrl');+ if (detectMac()) return;++ const id = window.setTimeout(() => setKey('Ctrl'), 0);+ return () => window.clearTimeout(id);}, []);+return key;}@@ -37,17 +45,136 @@ interface CustomSearchDialogProps extends SharedProps {api?: string;}+type AlgoliaHit = {+ objectID: string;+ url?: string;+};++type AlgoliaSearchResponse = {+ hits: AlgoliaHit[];+ queryID?: string;+};++type AlgoliaHitMeta = {+ objectID: string;+ position: number;+ queryID?: string;+};+export default function CustomSearchDialog({defaultLinks = [],api = '/api/search',...props}: CustomSearchDialogProps) {const { locale } = useI18n();- const { search, setSearch, query } = useDocsSearch({- type: 'fetch',- locale,- api,- });+ const algoliaHitMetaRef = useRef(new Map<string, AlgoliaHitMeta>());+ const insightsInitializedRef = useRef(false);++ const ensureAlgoliaInsights = useCallback((appId: string, searchApiKey: string) => {+ if (insightsInitializedRef.current) return;++ aa('init', {+ appId,+ apiKey: searchApiKey,+ useCookie: true,+ });+ insightsInitializedRef.current = true;+ }, []);++ const clientOptions = useMemo(() => {+ const appId = process.env.NEXT_PUBLIC_ALGOLIA_APP_ID ?? '62HI9PQZ1L';+ const searchApiKey = process.env.NEXT_PUBLIC_ALGOLIA_SEARCH_API_KEY;+ const indexName = process.env.NEXT_PUBLIC_ALGOLIA_INDEX_NAME ?? 'docs_composio_dev_62hi9pqz1l_pages';++ if (appId && searchApiKey) {+ const client = liteClient(appId, searchApiKey);++ return {+ type: 'algolia' as const,+ client,+ indexName,+ locale,+ onSearch: async (query: string, tag?: string) => {+ ensureAlgoliaInsights(appId, searchApiKey);++ const result = await client.searchForHits<BaseIndex>({+ requests: [+ {+ type: 'default',+ indexName,+ query,+ distinct: 5,+ hitsPerPage: 10,+ filters: tag ? `tag:${tag}` : undefined,+ clickAnalytics: true,+ },+ ],+ });+ const response = result.results[0] as AlgoliaSearchResponse;+ const metadata = new Map<string, AlgoliaHitMeta>();++ response.hits.forEach((hit, index) => {+ if (!hit.url || metadata.has(hit.url)) return;+ metadata.set(hit.url, {+ objectID: hit.objectID,+ position: index + 1,+ queryID: response.queryID,+ });+ });++ algoliaHitMetaRef.current = metadata;++ const viewedObjectIDs = Array.from(metadata.values())+ .slice(0, 20)+ .map((hit) => hit.objectID);++ if (viewedObjectIDs.length > 0) {+ aa('viewedObjectIDs', {+ index: indexName,+ eventName: 'Docs Search Results Viewed',+ objectIDs: viewedObjectIDs,+ });+ }++ return result;+ },+ };+ }++ return {+ type: 'fetch' as const,+ locale,+ api,+ };+ }, [api, ensureAlgoliaInsights, locale]);++ const { search, setSearch, query } = useDocsSearch(clientOptions);++ const trackAlgoliaClick = useCallback((href: string) => {+ const url = new URL(href, window.location.origin);+ const path = `${url.pathname}${url.hash}`;+ const hit = algoliaHitMetaRef.current.get(path) ?? algoliaHitMetaRef.current.get(url.pathname);+ const indexName = process.env.NEXT_PUBLIC_ALGOLIA_INDEX_NAME ?? 'docs_composio_dev_62hi9pqz1l_pages';++ if (!hit) return;++ if (hit.queryID) {+ aa('clickedObjectIDsAfterSearch', {+ index: indexName,+ eventName: 'Docs Search Result Clicked',+ queryID: hit.queryID,+ objectIDs: [hit.objectID],+ positions: [hit.position],+ });+ return;+ }++ aa('clickedObjectIDs', {+ index: indexName,+ eventName: 'Docs Search Result Clicked',+ objectIDs: [hit.objectID],+ });+ }, []);return (<SearchDialog@@ -63,7 +190,19 @@ export default function CustomSearchDialog({<SearchDialogInput /><SearchDialogClose /></SearchDialogHeader>- <div className="search-scroll-container max-md:min-h-0 max-md:flex-1 max-md:overflow-y-auto">+ <div+ className="search-scroll-container max-md:min-h-0 max-md:flex-1 max-md:overflow-y-auto"+ onClickCapture={(event) => {+ const target = event.target;+ if (!(target instanceof Element)) return;++ const anchor = target.closest('a[href]');+ const href = anchor?.getAttribute('href');+ if (!href) return;++ trackAlgoliaClick(href);+ }}+ >{query.data === 'empty' && defaultLinks.length > 0 ? (<div className="flex flex-col p-2">{defaultLinks.map((link) => (diff --git a/docs/lib/search-index.ts b/docs/lib/search-index.tsnew file mode 100644index 000000000..e6ae68258--- /dev/null+++ b/docs/lib/search-index.ts@@ -0,0 +1,283 @@+// Use direct imports from collections to avoid top-level await in lib/source.ts.+import { existsSync, readdirSync, readFileSync } from 'node:fs';+import { join, relative } from 'node:path';+import { docs, reference, cookbooks, toolkits } from 'fumadocs-mdx:collections/server';+import { loader, multiple } from 'fumadocs-core/source';+import { lucideIconsPlugin } from 'fumadocs-core/source/lucide-icons';+import { openapiSource, openapiPlugin } from 'fumadocs-openapi/server';+import type { AdvancedIndex } from 'fumadocs-core/search/server';+import type { DocumentRecord } from 'fumadocs-core/search/algolia';+import { openapi } from '@/lib/openapi';+import { getAllToolkitsSync } from '@/lib/toolkit-data';++// Create loaders directly here to avoid the problematic lib/source.ts import.+const docsSource = loader({+ baseUrl: '/docs',+ source: docs.toFumadocsSource(),+ plugins: [lucideIconsPlugin()],+});++const cookbooksSource = loader({+ baseUrl: '/cookbooks',+ source: cookbooks.toFumadocsSource(),+ plugins: [lucideIconsPlugin()],+});++const toolkitsSource = loader({+ baseUrl: '/toolkits',+ source: toolkits.toFumadocsSource(),+ plugins: [lucideIconsPlugin()],+});++// Dynamic toolkit entries from toolkits.json.+const mdxToolkitSlugs = new Set(+ toolkitsSource.getPages().map((page) => page.slugs.join('/')),+);++const dynamicToolkitIndexes = getAllToolkitsSync()+ .filter((toolkit) => !mdxToolkitSlugs.has(toolkit.slug))+ .map((toolkit) => ({+ id: `/toolkits/${toolkit.slug}`,+ title: toolkit.name,+ description: toolkit.description,+ url: `/toolkits/${toolkit.slug}`,+ structuredData: { headings: [], contents: [] },+ keywords: [toolkit.slug, toolkit.category].filter(Boolean) as string[],+ } satisfies SearchIndex));++function getFrontmatterValue(frontmatter: string, key: string): string | undefined {+ const match = frontmatter.match(new RegExp(`^${key}:\\s*(?:"([^"]*)"|'([^']*)'|(.+))$`, 'm'));+ const value = match?.[1] ?? match?.[2] ?? match?.[3];+ return value?.trim();+}++function getChangelogIndexes(): SearchIndex[] {+ const changelogDir = join(process.cwd(), 'content/changelog');+ if (!existsSync(changelogDir)) return [];++ return readdirSync(changelogDir)+ .filter((file) => file.endsWith('.mdx'))+ .flatMap((file) => {+ const source = readFileSync(join(changelogDir, file), 'utf8');+ const frontmatter = source.match(/^---\n([\s\S]*?)\n---/)?.[1];+ if (!frontmatter) return [];++ const date = getFrontmatterValue(frontmatter, 'date');+ const title = getFrontmatterValue(frontmatter, 'title');+ if (!date || !title) return [];++ const url = `/docs/changelog/${date.replace(/-/g, '/')}`;++ return [{+ id: `${url}#${title}`,+ title,+ description: getFrontmatterValue(frontmatter, 'description') ?? '',+ url,+ structuredData: { headings: [], contents: [] },+ keywords: ['changelog'],+ } satisfies SearchIndex];+ });+}++type SearchIndex = AdvancedIndex & {+ keywords?: string[];+};++export async function getDocsSearchIndexes(): Promise<SearchIndex[]> {+ // Load OpenAPI pages and build full reference source.+ const openapiPages = await openapiSource(openapi, {+ groupBy: 'tag',+ baseDir: 'api-reference',+ });++ const fullReferenceSource = loader({+ baseUrl: '/reference',+ source: multiple({+ mdx: reference.toFumadocsSource(),+ openapi: openapiPages,+ }),+ plugins: [lucideIconsPlugin(), openapiPlugin()],+ });++ const mdxIndexes = [+ ...docsSource.getPages(),+ ...cookbooksSource.getPages(),+ ...toolkitsSource.getPages(),+ ...fullReferenceSource.getPages(),+ ].map((page) => ({+ id: page.url,+ title: page.data.title ?? 'Untitled',+ description: page.data.description,+ url: page.url,+ structuredData: page.data.structuredData,+ keywords: 'keywords' in page.data ? (page.data.keywords as string[]) : undefined,+ } satisfies SearchIndex));++ return [...mdxIndexes, ...dynamicToolkitIndexes, ...getChangelogIndexes()];+}++function listContentFiles(dir: string): string[] {+ if (!existsSync(dir)) return [];++ return readdirSync(dir, { withFileTypes: true }).flatMap((entry) => {+ const path = join(dir, entry.name);+ if (entry.isDirectory()) return listContentFiles(path);+ if (entry.isFile() && /\.mdx?$/.test(entry.name)) return [path];+ return [];+ });+}++function slugifyHeading(value: string): string {+ return value+ .toLowerCase()+ .trim()+ .replace(/[<>]/g, '')+ .replace(/[`*_~\[\]()]/g, '')+ .replace(/&[a-z]+;/g, '')+ .replace(/[^\w\s-]/g, '')+ .replace(/\s+/g, '-')+ .replace(/-+/g, '-');+}++function stripMdx(value: string): string {+ return value+ .replace(/^---\n[\s\S]*?\n---\n?/, '')+ .replace(/```[\s\S]*?```/g, '')+ .replace(/[<>]/g, ' ')+ .replace(/\{[^}]*\}/g, ' ')+ .replace(/!\[[^\]]*\]\([^)]*\)/g, '')+ .replace(/\[([^\]]+)\]\([^)]*\)/g, '$1')+ .replace(/[`*_>#-]/g, ' ')+ .replace(/\s+/g, ' ')+ .trim();+}++function buildStructuredData(markdown: string) {+ const body = markdown.replace(/^---\n[\s\S]*?\n---\n?/, '');+ const headings: { content: string; id: string }[] = [];+ const contents: { content: string; heading: string | undefined }[] = [];+ const lines = body.split('\n');+ let currentHeading: string | undefined;+ let current: string[] = [];++ function flush() {+ const content = stripMdx(current.join('\n'));+ if (content) contents.push({ content, heading: currentHeading });+ current = [];+ }++ for (const line of lines) {+ const match = line.match(/^(#{2,6})\s+(.+)$/);+ if (match) {+ flush();+ const content = stripMdx(match[2] ?? '');+ if (content) {+ const id = slugifyHeading(content);+ headings.push({ content, id });+ currentHeading = id;+ }+ continue;+ }++ current.push(line);+ }++ flush();+ if (contents.length === 0) {+ const content = stripMdx(body);+ if (content) contents.push({ content, heading: undefined });+ }++ return { headings, contents };+}++function urlFromContentPath(path: string): string | undefined {+ const rel = relative(join(process.cwd(), 'content'), path).replace(/\\/g, '/');+ const withoutExt = rel.replace(/\.mdx?$/, '');+ const parts = withoutExt.split('/');+ const collection = parts.shift();+ if (!collection) return undefined;++ if (collection === 'docs') return `/docs/${parts.join('/')}`.replace(/\/index$/, '');+ if (collection === 'cookbooks') return `/cookbooks/${parts.join('/')}`.replace(/\/index$/, '');+ if (collection === 'reference') return `/reference/${parts.join('/')}`.replace(/\/index$/, '');+ if (collection === 'toolkits') {+ if (parts[0] === 'faq') return undefined;+ return `/toolkits/${parts.join('/')}`.replace(/\/index$/, '');+ }+ if (collection === 'changelog') return undefined;++ return undefined;+}++function getFilesystemIndexes(): SearchIndex[] {+ const contentDir = join(process.cwd(), 'content');+ return listContentFiles(contentDir).flatMap((file) => {+ const url = urlFromContentPath(file);+ if (!url) return [];++ const source = readFileSync(file, 'utf8');+ const frontmatter = source.match(/^---\n([\s\S]*?)\n---/)?.[1] ?? '';+ const title = getFrontmatterValue(frontmatter, 'title');+ if (!title) return [];++ return [{+ id: url,+ title,+ description: getFrontmatterValue(frontmatter, 'description'),+ url,+ structuredData: buildStructuredData(source),+ keywords: getFrontmatterValue(frontmatter, 'keywords')?.split(',').map((keyword) => keyword.trim()),+ } satisfies SearchIndex];+ });+}++async function getOpenApiSearchIndexes(): Promise<SearchIndex[]> {+ const openapiPages = await openapiSource(openapi, {+ groupBy: 'tag',+ baseDir: 'api-reference',+ });++ const openapiOnlySource = loader({+ baseUrl: '/reference',+ source: openapiPages,+ plugins: [lucideIconsPlugin(), openapiPlugin()],+ });++ return openapiOnlySource.getPages().map((page) => ({+ id: page.url,+ title: page.data.title ?? 'Untitled',+ description: page.data.description,+ url: page.url,+ structuredData: page.data.structuredData,+ } satisfies SearchIndex));+}++function toAlgoliaDocument(index: SearchIndex): DocumentRecord {+ const searchableKeywords = index.keywords?.filter(Boolean).join(' ');+ const description = [index.description, searchableKeywords].filter(Boolean).join('\n');++ return {+ _id: index.id,+ title: index.title,+ description,+ breadcrumbs: index.breadcrumbs,+ url: index.url,+ structured: index.structuredData,+ tag: Array.isArray(index.tag) ? index.tag[0] : index.tag,+ } satisfies DocumentRecord;+}++export async function getAlgoliaSearchDocuments(): Promise<DocumentRecord[]> {+ const indexes = [+ ...getFilesystemIndexes(),+ ...dynamicToolkitIndexes,+ ...getChangelogIndexes(),+ ...await getOpenApiSearchIndexes(),+ ];++ const deduped = new Map<string, SearchIndex>();+ for (const index of indexes) deduped.set(index.id, index);++ return Array.from(deduped.values()).map(toAlgoliaDocument);+}diff --git a/docs/package.json b/docs/package.jsonindex 3585453aa..bd01f51ac 100644--- a/docs/package.json+++ b/docs/package.json@@ -10,6 +10,7 @@"postinstall": "fumadocs-mdx","lint": "eslint","lint:links": "bun scripts/validate-links.ts",+ "sync:search": "bun scripts/sync-algolia-search.ts","generate:toolkits": "bun scripts/generate-toolkits.ts","generate:meta-tools": "bun scripts/generate-meta-tools.ts","generate:api-index": "bun scripts/generate-api-index.ts",@@ -20,6 +21,7 @@"@radix-ui/react-tabs": "^1.1.13","@shikijs/twoslash": "^3.21.0","@vercel/analytics": "^1.6.1",+ "algoliasearch": "^5","class-variance-authority": "^0.7.1","clsx": "^2.1.1","fumadocs-core": "^16.4.2",@@ -32,6 +34,7 @@"posthog-js": "^1.319.0","react": "^19.2.3","react-dom": "^19.2.3",+ "search-insights": "^2","shiki": "^3.20.0","tailwind-merge": "^3.4.0"},diff --git a/docs/scripts/sync-algolia-search.ts b/docs/scripts/sync-algolia-search.tsnew file mode 100644index 000000000..8710e0bc8--- /dev/null+++ b/docs/scripts/sync-algolia-search.ts@@ -0,0 +1,40 @@+import { algoliasearch } from 'algoliasearch';+import { sync } from 'fumadocs-core/search/algolia';+import { getAlgoliaSearchDocuments } from '@/lib/search-index';++function requireEnv(name: string, fallback?: string): string {+ const value = process.env[name] ?? fallback;+ if (!value) {+ throw new Error(`Missing required environment variable: ${name}`);+ }+ return value;+}++const dryRun = process.argv.includes('--dry-run');+const indexName =+ process.env.ALGOLIA_INDEX_NAME ?? process.env.NEXT_PUBLIC_ALGOLIA_INDEX_NAME ?? 'docs_composio_dev_62hi9pqz1l_pages';++const documents = await getAlgoliaSearchDocuments();++if (documents.length === 0) {+ throw new Error('Refusing to sync an empty Algolia docs index.');+}++if (dryRun) {+ console.log(`Prepared ${documents.length} docs pages for Algolia index "${indexName}".`);+ process.exit(0);+}++const appId = requireEnv(+ 'ALGOLIA_APP_ID',+ process.env.NEXT_PUBLIC_ALGOLIA_APP_ID ?? '62HI9PQZ1L',+);+const adminApiKey = requireEnv('ALGOLIA_ADMIN_API_KEY');+const client = algoliasearch(appId, adminApiKey);++console.log(`Syncing ${documents.length} docs pages to Algolia index "${indexName}"...`);+await sync(client, {+ indexName,+ documents,+});+console.log(`Synced Algolia docs search index "${indexName}".`);