Composio powers 1000+ toolkits, tool search, context management, authentication, and a sandboxed workbench to help you build AI agents that turn intent into action.
TypeScript agentic-aiagentsaiai-agentsaiagentsdeveloper-toolsfunction-callinggpt-4javascriptjsllmllmopsmcppythonremote-mcp-serverssetypescript
docs: reapply Algolia docs search (#3454)
Rahul Tarak18c3b98de690
diff --git a/.github/workflows/docs-search-sync.yml b/.github/workflows/docs-search-sync.ymlnew file mode 100644index 000000000..4b36ce82c--- /dev/null+++ b/.github/workflows/docs-search-sync.yml@@ -0,0 +1,46 @@+name: Docs - Sync Algolia Search++on:+ push:+ branches:+ - next+ paths:+ - 'docs/**'+ workflow_dispatch:++permissions:+ contents: read++jobs:+ sync-search:+ runs-on: ubuntu-latest+ defaults:+ run:+ working-directory: ./docs+ env:+ ALGOLIA_APP_ID: ${{ vars.ALGOLIA_APP_ID || '62HI9PQZ1L' }}+ ALGOLIA_ADMIN_API_KEY: ${{ secrets.ALGOLIA_ADMIN_API_KEY }}+ ALGOLIA_INDEX_NAME: ${{ vars.ALGOLIA_INDEX_NAME || 'docs_composio_dev_62hi9pqz1l_pages' }}++ steps:+ - name: Checkout repository+ uses: actions/checkout@v6++ - name: Setup Bun+ uses: oven-sh/setup-bun@v2+ with:+ bun-version-file: '.bun-version'++ - name: Install dependencies+ run: bun install++ - name: Validate search index generation+ run: bun run sync:search --dry-run++ - name: Sync Algolia search index+ if: env.ALGOLIA_APP_ID != '' && env.ALGOLIA_ADMIN_API_KEY != ''+ run: bun run sync:search++ - name: Skip sync without Algolia secrets+ if: env.ALGOLIA_APP_ID == '' || env.ALGOLIA_ADMIN_API_KEY == ''+ run: echo "Skipping Algolia sync because ALGOLIA_APP_ID or ALGOLIA_ADMIN_API_KEY is not configured."diff --git a/docs/CLAUDE.md b/docs/CLAUDE.mdindex 4862683d2..52f3df02f 100644--- a/docs/CLAUDE.md+++ b/docs/CLAUDE.md@@ -9,6 +9,8 @@ bun install # Install dependenciesbun run dev # Dev server (http://localhost:3000)bun run build # Production build (validates TS code blocks)bun run types:check # Type check+bun run sync:search # Build and sync Algolia docs search index (requires Algolia env vars)+bun run test:search # Test live Algolia search relevance from the terminal```## Project Structure@@ -83,6 +85,12 @@ Detailed documentation for Claude is organized in `.claude/`:`content/docs/glossary.mdx` defines key Composio terms (auth config, session, toolkit, etc.) using `<Glossary>` and `<GlossaryTerm name="...">` components (`components/glossary.tsx`). The component renders a filterable two-column table. The markdown converter in `lib/source.ts` converts `<GlossaryTerm>` tags to `### Term` headings for LLM-friendly output. When adding new Composio concepts, add a `<GlossaryTerm>` entry and update the `keywords` frontmatter array.+## Search++The docs search dialog uses Algolia in production when `NEXT_PUBLIC_ALGOLIA_SEARCH_API_KEY` is configured. `NEXT_PUBLIC_ALGOLIA_APP_ID` defaults to `62HI9PQZ1L`, and `NEXT_PUBLIC_ALGOLIA_INDEX_NAME` defaults to `docs_composio_dev_62hi9pqz1l_pages`. Without a search API key it falls back to `/api/search` for local development and tests. Algolia searches request `clickAnalytics`, and the client sends search result view/click events with `search-insights`.++Run `ALGOLIA_APP_ID=62HI9PQZ1L ALGOLIA_ADMIN_API_KEY=... ALGOLIA_INDEX_NAME=docs_composio_dev_62hi9pqz1l_pages bun run sync:search` from `docs/` to rebuild the Algolia index. The sync script does not use Algolia Crawler: it creates section-sized records from MDX/OpenAPI/toolkit data, configures searchable attributes, `customRanking`, and `attributeForDistinct`, then replaces the index objects. Use `bun run sync:search --dry-run --samples` to inspect generated records and `ALGOLIA_SEARCH_API_KEY=... bun run test:search "oauth auth config"` to test live relevance.+## API VersioningThe API reference supports v3.1 (latest, default) and v3.0 side-by-side:diff --git a/docs/README.md b/docs/README.mdindex 6da6874e2..182775229 100644--- a/docs/README.md+++ b/docs/README.md@@ -74,6 +74,31 @@ All TypeScript code blocks in MDX files are **type-checked at build time** usingSee `CLAUDE.md` for detailed patterns and troubleshooting.+## Search++Docs search uses Algolia when `NEXT_PUBLIC_ALGOLIA_SEARCH_API_KEY` is set; otherwise it falls back to the local Fumadocs `/api/search` endpoint for development and tests. The sync script builds a first-party index from MDX/OpenAPI/toolkit data (no crawler required), splits long pages into section-sized records, configures searchable attributes/custom ranking/distinct, requests `clickAnalytics`, and sends search result view/click events with `search-insights`.++```bash+NEXT_PUBLIC_ALGOLIA_APP_ID=62HI9PQZ1L # optional; default shown+NEXT_PUBLIC_ALGOLIA_SEARCH_API_KEY=...+NEXT_PUBLIC_ALGOLIA_INDEX_NAME=docs_composio_dev_62hi9pqz1l_pages # optional; default shown+```++Sync the Algolia index with an admin key:++```bash+ALGOLIA_APP_ID=62HI9PQZ1L+ALGOLIA_ADMIN_API_KEY=...+ALGOLIA_INDEX_NAME=docs_composio_dev_62hi9pqz1l_pages bun run sync:search+```++Preview the generated records or test live relevance:++```bash+bun run sync:search --dry-run --samples+ALGOLIA_SEARCH_API_KEY=... bun run test:search "oauth auth config" "gmail send email"+```+## Commands| Command | Description |@@ -81,4 +106,6 @@ See `CLAUDE.md` for detailed patterns and troubleshooting.| `bun run dev` | Dev server || `bun run build` | Production build (validates TS code blocks) || `bun run types:check` | Type check |+| `bun run sync:search` | Sync docs search records to Algolia |+| `bun run test:search` | Query the configured Algolia index from the terminal |diff --git a/docs/app/api/search/route.ts b/docs/app/api/search/route.tsindex ba05af37b..be0355baa 100644--- a/docs/app/api/search/route.ts+++ b/docs/app/api/search/route.ts@@ -1,89 +1,8 @@-// Use direct imports from collections to avoid top-level await in lib/source.ts-import { docs, reference, cookbooks, toolkits, changelog } from 'fumadocs-mdx:collections/server';import { createSearchAPI } from 'fumadocs-core/search/server';-import { loader, multiple } from 'fumadocs-core/source';-import { lucideIconsPlugin } from 'fumadocs-core/source/lucide-icons';-import { openapiSource, openapiPlugin } from 'fumadocs-openapi/server';-import { openapi } from '@/lib/openapi';-import { getAllToolkitsSync } from '@/lib/toolkit-data';+import { getDocsSearchIndexes } from '@/lib/search-index';-// Create loaders directly here to avoid the problematic lib/source.ts import-const docsSource = loader({- baseUrl: '/docs',- source: docs.toFumadocsSource(),- plugins: [lucideIconsPlugin()],-});--const cookbooksSource = loader({- baseUrl: '/cookbooks',- source: cookbooks.toFumadocsSource(),- plugins: [lucideIconsPlugin()],-});--const toolkitsSource = loader({- baseUrl: '/toolkits',- source: toolkits.toFumadocsSource(),- plugins: [lucideIconsPlugin()],-});--// Dynamic toolkit entries from toolkits.json-const mdxToolkitSlugs = new Set(- toolkitsSource.getPages().map((page) => page.slugs.join('/')),-);--const dynamicToolkitIndexes = getAllToolkitsSync()- .filter((toolkit) => !mdxToolkitSlugs.has(toolkit.slug))- .map((toolkit) => ({- id: `/toolkits/${toolkit.slug}`,- title: toolkit.name,- description: toolkit.description,- url: `/toolkits/${toolkit.slug}`,- structuredData: { headings: [] as never[], contents: [] as never[] },- keywords: [toolkit.slug, toolkit.category].filter(Boolean) as string[],- }));--// Changelog entries-const changelogIndexes = changelog.map((entry) => ({- id: `/docs/changelog/${entry.date.replace(/-/g, '/')}#${entry.title}`,- title: entry.title,- description: entry.description ?? '',- url: `/docs/changelog/${entry.date.replace(/-/g, '/')}`,- structuredData: { headings: [] as never[], contents: [] as never[] },- keywords: ['changelog'],-}));--// Use dynamic indexes to support async OpenAPI page loading+// Fallback/local search endpoint. Production search uses Algolia from the client+// when NEXT_PUBLIC_ALGOLIA_* variables are configured.export const { GET } = createSearchAPI('advanced', {- indexes: async () => {- // Load OpenAPI pages and build full reference source- const openapiPages = await openapiSource(openapi, {- groupBy: 'tag',- baseDir: 'api-reference',- });-- const fullReferenceSource = loader({- baseUrl: '/reference',- source: multiple({- mdx: reference.toFumadocsSource(),- openapi: openapiPages,- }),- plugins: [lucideIconsPlugin(), openapiPlugin()],- });-- const mdxIndexes = [- ...docsSource.getPages(),- ...cookbooksSource.getPages(),- ...toolkitsSource.getPages(),- ...fullReferenceSource.getPages(),- ].map((page) => ({- id: page.url,- title: page.data.title ?? 'Untitled',- description: page.data.description,- url: page.url,- structuredData: page.data.structuredData,- keywords: 'keywords' in page.data ? (page.data.keywords as string[]) : undefined,- }));-- return [...mdxIndexes, ...dynamicToolkitIndexes, ...changelogIndexes];- },+ indexes: getDocsSearchIndexes,});diff --git a/docs/bun.lock b/docs/bun.lockindex 96f1e5674..ba8797694 100644--- a/docs/bun.lock+++ b/docs/bun.lock@@ -1,5 +1,6 @@{"lockfileVersion": 1,+ "configVersion": 0,"workspaces": {"": {"name": "fumadocs",@@ -7,6 +8,7 @@"@radix-ui/react-tabs": "^1.1.13","@shikijs/twoslash": "^3.21.0","@vercel/analytics": "^1.6.1",+ "algoliasearch": "^5","class-variance-authority": "^0.7.1","clsx": "^2.1.1","fumadocs-core": "^16.4.2",@@ -19,6 +21,7 @@"posthog-js": "^1.319.0","react": "^19.2.3","react-dom": "^19.2.3",+ "search-insights": "^2","shiki": "^3.20.0","tailwind-merge": "^3.4.0",},@@ -95,6 +98,34 @@"@ai-sdk/ui-utils-v5": ["@ai-sdk/[email protected]", "", { "dependencies": { "@ai-sdk/provider": "1.1.3", "@ai-sdk/provider-utils": "2.2.8", "zod-to-json-schema": "^3.24.1" }, "peerDependencies": { "zod": "^3.23.8" } }, "sha512-3zcwCc8ezzFlwp3ZD15wAPjf2Au4s3vAbKsXQVyhxODHcmu0iyPO2Eua6D/vicq/AUm/BAo60r97O6HU+EI0+w=="],+ "@algolia/abtesting": ["@algolia/[email protected]", "", { "dependencies": { "@algolia/client-common": "5.52.1", "@algolia/requester-browser-xhr": "5.52.1", "@algolia/requester-fetch": "5.52.1", "@algolia/requester-node-http": "5.52.1" } }, "sha512-aehCadlWOGvrT91KUIZpC0MbB8KBW9yUuvTJFd2xesR7le/IsT4nJUnjCCZ4ZqZCeTcPHPV5mo//fZ5oxcSVYw=="],++ "@algolia/client-abtesting": ["@algolia/[email protected]", "", { "dependencies": { "@algolia/client-common": "5.52.1", "@algolia/requester-browser-xhr": "5.52.1", "@algolia/requester-fetch": "5.52.1", "@algolia/requester-node-http": "5.52.1" } }, "sha512-HmXOGBOAOJPounpBzBpuY0zDYeiCpxgHnQmuA7JO6ScukcBdGp3/XM9zJk5pJx/xNGD68mbPGXWpDxGtl6BwDQ=="],++ "@algolia/client-analytics": ["@algolia/[email protected]", "", { "dependencies": { "@algolia/client-common": "5.52.1", "@algolia/requester-browser-xhr": "5.52.1", "@algolia/requester-fetch": "5.52.1", "@algolia/requester-node-http": "5.52.1" } }, "sha512-5oo4+I8iixie9vXhCyNFCzeIr8pqA3FQ//VsLHTDvZAV4ttYOPGvYHGQq5NSalrLx5Jc3dRro/5uDOlnUMcBJg=="],++ "@algolia/client-common": ["@algolia/[email protected]", "", {}, "sha512-qCDoZfx5MpX7XQzvQ3bC4tSEMkQWQMaF/ABtLuoze03Y/flR563CCSws02qIJ23oX7lxl92LsilZjINVyTdtLw=="],++ "@algolia/client-insights": ["@algolia/[email protected]", "", { "dependencies": { "@algolia/client-common": "5.52.1", "@algolia/requester-browser-xhr": "5.52.1", "@algolia/requester-fetch": "5.52.1", "@algolia/requester-node-http": "5.52.1" } }, "sha512-hnGs0/lsFJ2PWDxNBz7pxreXo/Xz7gxYRcfePBUjsH26ad0kU/sgnVZd9LwWBpsQv65z2jlb5dkyaB9WE9M9FQ=="],++ "@algolia/client-personalization": ["@algolia/[email protected]", "", { "dependencies": { "@algolia/client-common": "5.52.1", "@algolia/requester-browser-xhr": "5.52.1", "@algolia/requester-fetch": "5.52.1", "@algolia/requester-node-http": "5.52.1" } }, "sha512-2VxxNc/uBysyKvGeBdSM5n9eIDKH8kWD7wd9/yqbJAiVwU4Yv6tU1LSJusHKrXV/aCu1KW7t9Gug9QyeEmtn/Q=="],++ "@algolia/client-query-suggestions": ["@algolia/[email protected]", "", { "dependencies": { "@algolia/client-common": "5.52.1", "@algolia/requester-browser-xhr": "5.52.1", "@algolia/requester-fetch": "5.52.1", "@algolia/requester-node-http": "5.52.1" } }, "sha512-O6mPtsw3xEfNOe6gWFpYLeAZAIljNa4Hgna3bq15PwyN7nbjTY0wXJFRbzs/0YVf75Br+SbOQUmjKxXYjDiSiQ=="],++ "@algolia/client-search": ["@algolia/[email protected]", "", { "dependencies": { "@algolia/client-common": "5.52.1", "@algolia/requester-browser-xhr": "5.52.1", "@algolia/requester-fetch": "5.52.1", "@algolia/requester-node-http": "5.52.1" } }, "sha512-gA8oJOV1LnQQkDf91iebNnFInHuW0gRPEgLSOQ7EfipCEjYTHm5swm1DlH9H5RaRw4RrHuzHBegnlzc0MAstcg=="],++ "@algolia/ingestion": ["@algolia/[email protected]", "", { "dependencies": { "@algolia/client-common": "5.52.1", "@algolia/requester-browser-xhr": "5.52.1", "@algolia/requester-fetch": "5.52.1", "@algolia/requester-node-http": "5.52.1" } }, "sha512-U9zZfc5xIu9wRxZkt+HceJUAD4VKHKbAyLSloJdEyMRmphXeibfrY9cxqIXBcmPeZzGhn3Imb35Dq8l19PkJhw=="],++ "@algolia/monitoring": ["@algolia/[email protected]", "", { "dependencies": { "@algolia/client-common": "5.52.1", "@algolia/requester-browser-xhr": "5.52.1", "@algolia/requester-fetch": "5.52.1", "@algolia/requester-node-http": "5.52.1" } }, "sha512-a3SGNceHmkQfq77iG8Ka+w1pvwfZa/0lzEIgse30fL0kD+yKnd/dg0dQvSfFPAEt2f21DMcGkDSSeJlO3KdQjQ=="],++ "@algolia/recommend": ["@algolia/[email protected]", "", { "dependencies": { "@algolia/client-common": "5.52.1", "@algolia/requester-browser-xhr": "5.52.1", "@algolia/requester-fetch": "5.52.1", "@algolia/requester-node-http": "5.52.1" } }, "sha512-z98QEguCFDpxb4S/PyrUK1igqF8tPsdbqOUUO6ON91vJ58w+Gwa6ncrI0oNXSFcrkxA5EqPKPQ2A1PBCn08TYQ=="],++ "@algolia/requester-browser-xhr": ["@algolia/[email protected]", "", { "dependencies": { "@algolia/client-common": "5.52.1" } }, "sha512-CI7+/0I11QeZM59Uc8whd2or0kqzFVjpaPn9Qpwll/krHcBAxk24WkAQ6WX+IwDVMfpont4YGbKwAmCre3vE8Q=="],++ "@algolia/requester-fetch": ["@algolia/[email protected]", "", { "dependencies": { "@algolia/client-common": "5.52.1" } }, "sha512-S6bDuw9byfOvm3T71cgdoZgrgnZq6hpdMLkx52Louh57nUAmvGQESz2aojOynQHjbTiV55smvAFbgn0qT4tJrg=="],++ "@algolia/requester-node-http": ["@algolia/[email protected]", "", { "dependencies": { "@algolia/client-common": "5.52.1" } }, "sha512-tqZXM+54rWo4mk5jL5Z/flE11nPmNEdXwFBM5py9DkOmbjeCNemfVd45FyM97XdzfZ0dl9uOJC6PYn1FpkeyQg=="],+"@alloc/quick-lru": ["@alloc/[email protected]", "", {}, "sha512-UrcABB+4bUrFABwbluTIBErXwvbsU/V7TZWfmbgJfbkwiBuziS9gxdODUyuiecfdGQ85jglMW6juS3+z5TsKLw=="],"@antfu/install-pkg": ["@antfu/[email protected]", "", { "dependencies": { "package-manager-detector": "^1.3.0", "tinyexec": "^1.0.1" } }, "sha512-MGQsmw10ZyI+EJo45CdSER4zEb+p31LpDAFp2Z3gkSd1yqVZGi0Ebx++YTEMonJy4oChEMLsxZ64j8FH6sSqtQ=="],@@ -865,6 +896,8 @@"ajv-formats": ["[email protected]", "", { "dependencies": { "ajv": "^8.0.0" } }, "sha512-8iUql50EUR+uUcdRQ3HDqa6EVyo3docL8g5WJ3FNcWmu62IbkGUue/pEyLBW8VGKKucTPgqeks4fIU1DA4yowQ=="],+ "algoliasearch": ["[email protected]", "", { "dependencies": { "@algolia/abtesting": "1.18.1", "@algolia/client-abtesting": "5.52.1", "@algolia/client-analytics": "5.52.1", "@algolia/client-common": "5.52.1", "@algolia/client-insights": "5.52.1", "@algolia/client-personalization": "5.52.1", "@algolia/client-query-suggestions": "5.52.1", "@algolia/client-search": "5.52.1", "@algolia/ingestion": "1.52.1", "@algolia/monitoring": "1.52.1", "@algolia/recommend": "5.52.1", "@algolia/requester-browser-xhr": "5.52.1", "@algolia/requester-fetch": "5.52.1", "@algolia/requester-node-http": "5.52.1" } }, "sha512-fHA8+kXTbjagw3jkLiaS7KKrH8qe2DyOsiUhGlN4cdT77PEsfqXZl7ewDk1hsg+pJnPlnE50XtLxjR91iJOpmg=="],+"alien-signals": ["[email protected]", "", {}, "sha512-d9dYqZTS90WLiU0I5c6DHj/HcKkF8ZyGN3G5x8wSbslulz70KOxaqCT0hQCo9KOyhVqzqGojvNdJXoTumZOtcw=="],"ansi-regex": ["[email protected]", "", {}, "sha512-Bq3SmSpyFHaWjPk8If9yc6svM8c56dB5BAtW4Qbw5jHTwwXXcTLoRMkpDJp6VL0XzlWaCHTXrkFURMYmD0sLqg=="],@@ -1999,6 +2032,8 @@"scroll-into-view-if-needed": ["[email protected]", "", { "dependencies": { "compute-scroll-into-view": "^3.0.2" } }, "sha512-49oNpRjWRvnU8NyGVmUaYG4jtTkNonFZI86MmGRDqBphEK2EXT9gdEUoQPZhuBM8yWHxCWbobltqYO5M4XrUvQ=="],+ "search-insights": ["[email protected]", "", {}, "sha512-RQPdCYTa8A68uM2jwxoY842xDhvx3E5LFL1LxvxCNMev4o5mLuokczhzjAgGwUZBAmOKZknArSxLKmXtIi2AxQ=="],+"section-matter": ["[email protected]", "", { "dependencies": { "extend-shallow": "^2.0.1", "kind-of": "^6.0.0" } }, "sha512-vfD3pmTzGpufjScBh50YHKzEu2lxBWhVEHsNGoEXmCmn2hKGfeNLYMzCJpe8cD7gqX7TJluOVpBkAequ6dgMmA=="],"secure-json-parse": ["[email protected]", "", {}, "sha512-6aU+Rwsezw7VR8/nyvKTx8QpWH9FrcYiXXlqC4z5d5XQBDRqtbfsRjnwGyqbi3gddNtWHuEk9OANUotL26qKUw=="],diff --git a/docs/components/custom-search-dialog.tsx b/docs/components/custom-search-dialog.tsxindex 388f2c389..9f1108ea7 100644--- a/docs/components/custom-search-dialog.tsx+++ b/docs/components/custom-search-dialog.tsx@@ -1,7 +1,9 @@'use client';-import { useEffect, useState } from 'react';+import { useCallback, useEffect, useMemo, useRef, useState } from 'react';import Link from 'next/link';+import { liteClient } from 'algoliasearch/lite';+import aa from 'search-insights';import {SearchDialog,SearchDialogClose,@@ -15,14 +17,20 @@ import {} from 'fumadocs-ui/components/dialog/search';import { useI18n } from 'fumadocs-ui/contexts/i18n';import { useDocsSearch } from 'fumadocs-core/search/client';+import type { BaseIndex } from 'fumadocs-core/search/algolia';import { BotMessageSquare } from 'lucide-react';import { toggleDecimalWidget, detectMac } from './ask-ai-button';function MetaKey() {const [key, setKey] = useState('⌘');+useEffect(() => {- if (!detectMac()) setKey('Ctrl');+ if (detectMac()) return;++ const id = window.setTimeout(() => setKey('Ctrl'), 0);+ return () => window.clearTimeout(id);}, []);+return key;}@@ -37,17 +45,182 @@ interface CustomSearchDialogProps extends SharedProps {api?: string;}+type AlgoliaHit = {+ objectID: string;+ url?: string;+};++type AlgoliaSearchResponse = {+ hits: AlgoliaHit[];+ queryID?: string;+};++type AlgoliaHitMeta = {+ objectID: string;+ position: number;+ queryID?: string;+};++type SearchResultItem = {+ content: unknown;+ breadcrumbs?: unknown[];+};++const SEARCH_BREADCRUMB_LABELS: Record<string, string> = {+ toolkits: 'Toolkit',+ toolkit: 'Toolkit',+ cookbooks: 'Cookbook',+ cookbook: 'Cookbook',+ docs: 'Docs',+ reference: 'Reference',+ 'api-reference': 'API Reference',+ changelog: 'Changelog',+};++function normalizeSearchBreadcrumb(value: string): string {+ return SEARCH_BREADCRUMB_LABELS[value.toLowerCase()] ?? value;+}++function normalizeSearchResult<T extends SearchResultItem>(result: T): T {+ if (!result.breadcrumbs) return result;++ const breadcrumbs = result.breadcrumbs.map((breadcrumb) =>+ typeof breadcrumb === 'string' ? normalizeSearchBreadcrumb(breadcrumb) : breadcrumb,+ );++ const content = typeof result.content === 'string' ? result.content.toLowerCase() : null;+ const dedupedBreadcrumbs = breadcrumbs.filter((breadcrumb, index) => {+ if (index !== breadcrumbs.length - 1 || content === null || typeof breadcrumb !== 'string') {+ return true;+ }++ return breadcrumb.toLowerCase() !== content;+ });++ return {+ ...result,+ breadcrumbs: dedupedBreadcrumbs,+ };+}+export default function CustomSearchDialog({defaultLinks = [],api = '/api/search',...props}: CustomSearchDialogProps) {const { locale } = useI18n();- const { search, setSearch, query } = useDocsSearch({- type: 'fetch',- locale,- api,- });+ const algoliaHitMetaRef = useRef(new Map<string, AlgoliaHitMeta>());+ const insightsInitializedRef = useRef(false);++ const ensureAlgoliaInsights = useCallback((appId: string, searchApiKey: string) => {+ if (insightsInitializedRef.current) return;++ aa('init', {+ appId,+ apiKey: searchApiKey,+ useCookie: true,+ });+ insightsInitializedRef.current = true;+ }, []);++ const clientOptions = useMemo(() => {+ const appId = process.env.NEXT_PUBLIC_ALGOLIA_APP_ID ?? '62HI9PQZ1L';+ const searchApiKey = process.env.NEXT_PUBLIC_ALGOLIA_SEARCH_API_KEY;+ const indexName = process.env.NEXT_PUBLIC_ALGOLIA_INDEX_NAME ?? 'docs_composio_dev_62hi9pqz1l_pages';++ if (appId && searchApiKey) {+ const client = liteClient(appId, searchApiKey);++ return {+ type: 'algolia' as const,+ client,+ indexName,+ locale,+ onSearch: async (query: string, tag?: string) => {+ ensureAlgoliaInsights(appId, searchApiKey);++ const result = await client.searchForHits<BaseIndex>({+ requests: [+ {+ type: 'default',+ indexName,+ query,+ distinct: true,+ hitsPerPage: 10,+ filters: tag ? `tag:${tag}` : undefined,+ clickAnalytics: true,+ },+ ],+ });+ const response = result.results[0] as AlgoliaSearchResponse;+ const metadata = new Map<string, AlgoliaHitMeta>();++ response.hits.forEach((hit, index) => {+ if (!hit.url || metadata.has(hit.url)) return;+ metadata.set(hit.url, {+ objectID: hit.objectID,+ position: index + 1,+ queryID: response.queryID,+ });+ });++ algoliaHitMetaRef.current = metadata;++ const viewedObjectIDs = Array.from(metadata.values())+ .slice(0, 20)+ .map((hit) => hit.objectID);++ if (viewedObjectIDs.length > 0) {+ aa('viewedObjectIDs', {+ index: indexName,+ eventName: 'Docs Search Results Viewed',+ objectIDs: viewedObjectIDs,+ });+ }++ return result;+ },+ };+ }++ return {+ type: 'fetch' as const,+ locale,+ api,+ };+ }, [api, ensureAlgoliaInsights, locale]);++ const { search, setSearch, query } = useDocsSearch(clientOptions);+ const searchResults = useMemo(() => {+ if (query.data === 'empty' || !query.data) return query.data;+ return query.data.map(normalizeSearchResult);+ }, [query.data]);++ const trackAlgoliaClick = useCallback((href: string) => {+ const url = new URL(href, window.location.origin);+ const path = `${url.pathname}${url.hash}`;+ const hit = algoliaHitMetaRef.current.get(path) ?? algoliaHitMetaRef.current.get(url.pathname);+ const indexName = process.env.NEXT_PUBLIC_ALGOLIA_INDEX_NAME ?? 'docs_composio_dev_62hi9pqz1l_pages';++ if (!hit) return;++ if (hit.queryID) {+ aa('clickedObjectIDsAfterSearch', {+ index: indexName,+ eventName: 'Docs Search Result Clicked',+ queryID: hit.queryID,+ objectIDs: [hit.objectID],+ positions: [hit.position],+ });+ return;+ }++ aa('clickedObjectIDs', {+ index: indexName,+ eventName: 'Docs Search Result Clicked',+ objectIDs: [hit.objectID],+ });+ }, []);return (<SearchDialog@@ -63,7 +236,19 @@ export default function CustomSearchDialog({<SearchDialogInput /><SearchDialogClose /></SearchDialogHeader>- <div className="search-scroll-container max-md:min-h-0 max-md:flex-1 max-md:overflow-y-auto">+ <div+ className="search-scroll-container max-md:min-h-0 max-md:flex-1 max-md:overflow-y-auto"+ onClickCapture={(event) => {+ const target = event.target;+ if (!(target instanceof Element)) return;++ const anchor = target.closest('a[href]');+ const href = anchor?.getAttribute('href');+ if (!href) return;++ trackAlgoliaClick(href);+ }}+ >{query.data === 'empty' && defaultLinks.length > 0 ? (<div className="flex flex-col p-2">{defaultLinks.map((link) => (@@ -81,7 +266,7 @@ export default function CustomSearchDialog({))}</div>) : (- <SearchDialogList items={query.data === 'empty' ? null : query.data} />+ <SearchDialogList items={searchResults === 'empty' ? null : searchResults} />)}</div><div className="flex items-center justify-between border-t px-3 py-2">diff --git a/docs/lib/search-index.ts b/docs/lib/search-index.tsnew file mode 100644index 000000000..7fa0c4935--- /dev/null+++ b/docs/lib/search-index.ts@@ -0,0 +1,618 @@+// Use direct imports from collections to avoid top-level await in lib/source.ts.+import { createHash } from 'node:crypto';+import { existsSync, readdirSync, readFileSync } from 'node:fs';+import { join, relative } from 'node:path';+import { docs, reference, cookbooks, toolkits } from 'fumadocs-mdx:collections/server';+import { loader, multiple } from 'fumadocs-core/source';+import { lucideIconsPlugin } from 'fumadocs-core/source/lucide-icons';+import { openapiSource, openapiPlugin } from 'fumadocs-openapi/server';+import type { AdvancedIndex } from 'fumadocs-core/search/server';+import type { BaseIndex } from 'fumadocs-core/search/algolia';+import { openapi } from '@/lib/openapi';+import { mdxToCleanMarkdown } from '@/lib/source';+import { getAllToolkitsSync } from '@/lib/toolkit-data';++export const ALGOLIA_DEFAULT_APP_ID = '62HI9PQZ1L';+export const ALGOLIA_DEFAULT_INDEX_NAME = 'docs_composio_dev_62hi9pqz1l_pages';++const MAX_CHUNK_CHARS = 3_800;+const MAX_CHUNK_BYTES = 9_000;+const MAX_TOOL_ALIAS_ITEMS = 80;+const MAX_TOOL_ALIAS_BYTES = 2_500;++// Create loaders directly here to avoid the problematic lib/source.ts import in the+// fallback route. This route is intentionally still Fumadocs/Orama-backed for local+// development when Algolia env vars aren't configured.+const docsSource = loader({+ baseUrl: '/docs',+ source: docs.toFumadocsSource(),+ plugins: [lucideIconsPlugin()],+});++const cookbooksSource = loader({+ baseUrl: '/cookbooks',+ source: cookbooks.toFumadocsSource(),+ plugins: [lucideIconsPlugin()],+});++const toolkitsSource = loader({+ baseUrl: '/toolkits',+ source: toolkits.toFumadocsSource(),+ plugins: [lucideIconsPlugin()],+});++type SearchIndex = AdvancedIndex & {+ keywords?: string[];+};++export type AlgoliaDocsRecord = BaseIndex & {+ objectID: string;+ description?: string;+ keywords?: string[];+ slug?: string;+ headings?: string[];+ tool_names?: string[];+ tool_slugs?: string[];+ type: string;+ lang: string;+ page_rank: number;+ toolkit_popularity: number;+ section_rank: number;+ position: number;+ depth: number;+ tags?: string[];+};++function getFrontmatter(source: string): string {+ return source.match(/^---\n([\s\S]*?)\n---/)?.[1] ?? '';+}++function getFrontmatterValue(frontmatter: string, key: string): string | undefined {+ const match = frontmatter.match(new RegExp(`^${key}:\\s*(?:"([^"]*)"|'([^']*)'|(.+))$`, 'm'));+ const value = match?.[1] ?? match?.[2] ?? match?.[3];+ return value?.trim();+}++function getFrontmatterList(frontmatter: string, key: string): string[] {+ const raw = getFrontmatterValue(frontmatter, key);+ if (!raw) return [];++ if (raw.startsWith('[') && raw.endsWith(']')) {+ return raw+ .slice(1, -1)+ .split(',')+ .map((value) => value.trim().replace(/^['"]|['"]$/g, ''))+ .filter(Boolean);+ }++ return [raw];+}++function listContentFiles(dir: string): string[] {+ if (!existsSync(dir)) return [];++ return readdirSync(dir, { withFileTypes: true }).flatMap((entry) => {+ const path = join(dir, entry.name);+ if (entry.isDirectory()) return listContentFiles(path);+ if (entry.isFile() && /\.mdx?$/.test(entry.name)) return [path];+ return [];+ });+}++function slugify(value: string): string {+ return value+ .toLowerCase()+ .trim()+ .replace(/[<>]/g, '')+ .replace(/[`*_~\[\]()]/g, '')+ .replace(/&[a-z]+;/g, '')+ .replace(/[^\w\s-]/g, '')+ .replace(/\s+/g, '-')+ .replace(/-+/g, '-')+ .replace(/^-|-$/g, '');+}++function uniqueSlug(value: string, seen: Map<string, number>): string {+ const base = slugify(value) || 'section';+ const count = seen.get(base) ?? 0;+ seen.set(base, count + 1);+ return count === 0 ? base : `${base}-${count + 1}`;+}++function splitIntoChunks(text: string): string[] {+ const paragraphs = text+ .split(/\n{2,}/)+ .map((paragraph) => paragraph.trim())+ .filter(Boolean);++ const chunks: string[] = [];+ let current = '';++ const pushCurrent = () => {+ const trimmed = current.trim();+ if (trimmed) chunks.push(trimmed);+ current = '';+ };++ for (const paragraph of paragraphs) {+ const next = current ? `${current}\n\n${paragraph}` : paragraph;+ if (next.length <= MAX_CHUNK_CHARS && Buffer.byteLength(next, 'utf8') <= MAX_CHUNK_BYTES) {+ current = next;+ continue;+ }++ pushCurrent();++ if (paragraph.length <= MAX_CHUNK_CHARS && Buffer.byteLength(paragraph, 'utf8') <= MAX_CHUNK_BYTES) {+ current = paragraph;+ continue;+ }++ for (let i = 0; i < paragraph.length; i += MAX_CHUNK_CHARS) {+ chunks.push(paragraph.slice(i, i + MAX_CHUNK_CHARS).trim());+ }+ }++ pushCurrent();+ return chunks;+}++function contentHash(value: string): string {+ return createHash('sha1').update(value).digest('hex').slice(0, 10);+}++function urlFromContentPath(path: string): { url: string; type: string } | undefined {+ const rel = relative(join(process.cwd(), 'content'), path).replace(/\\/g, '/');+ const withoutExt = rel.replace(/\.mdx?$/, '');+ const parts = withoutExt.split('/');+ const collection = parts.shift();+ if (!collection) return undefined;++ if (collection === 'docs') {+ return { url: `/docs/${parts.join('/')}`.replace(/\/index$/, ''), type: 'docs' };+ }+ if (collection === 'cookbooks') {+ return { url: `/cookbooks/${parts.join('/')}`.replace(/\/index$/, ''), type: 'cookbooks' };+ }+ if (collection === 'reference') {+ const url = `/reference/${parts.join('/')}`.replace(/\/index$/, '');+ return { url, type: url.startsWith('/reference/v3') ? 'v3-reference' : 'reference' };+ }+ if (collection === 'toolkits') {+ if (parts[0] === 'faq') return undefined;+ return { url: `/toolkits/${parts.join('/')}`.replace(/\/index$/, ''), type: 'toolkits' };+ }+ if (collection === 'changelog') return undefined;++ return undefined;+}++function slugTokens(url: string): string {+ return url+ .split(/[/-]/)+ .map((part) => part.replace(/[_#?=&.]/g, ' '))+ .join(' ')+ .replace(/\s+/g, ' ')+ .trim();+}++const typeLabels: Record<string, string> = {+ docs: 'Docs',+ cookbooks: 'Cookbook',+ reference: 'Reference',+ 'v3-reference': 'Legacy v3 Reference',+ toolkits: 'Toolkit',+ changelog: 'Changelog',+ 'api-reference': 'API Reference',+};++function titleizeSlug(value: string): string {+ return value+ .split('-')+ .filter(Boolean)+ .map((part) => part.charAt(0).toUpperCase() + part.slice(1))+ .join(' ')+ .replace(/\bApi\b/g, 'API')+ .replace(/\bMcp\b/g, 'MCP')+ .replace(/\bSdk\b/g, 'SDK')+ .replace(/\bOauth\b/g, 'OAuth');+}++function breadcrumbsForUrl(url: string, type: string): string[] {+ const label = typeLabels[type] ?? titleizeSlug(type);++ if (type === 'toolkits' || type === 'cookbooks' || type === 'changelog') {+ return [label];+ }++ const parts = url.split('/').filter(Boolean);+ const parentParts = parts.slice(1, -1);+ return [label, ...parentParts.map(titleizeSlug)].filter(Boolean);+}++const TOOLKIT_POPULARITY_OVERRIDES: Record<string, number> = {+ gmail: 1_000,+ github: 980,+ slack: 960,+ googledrive: 940,+ googlecalendar: 930,+ notion: 920,+ linear: 900,+ jira: 890,+ hubspot: 860,+ salesforce: 850,+ resend: 650,+ zoho_mail: 520,+ mailchimp: 500,+ sendgrid: 480,+ mailsoftly: 260,+};++function getToolkitSlugFromUrl(url: string): string | null {+ return url.match(/^\/toolkits\/([^/#?]+)/)?.[1] ?? null;+}++function toolkitPopularity(url: string, type: string): number {+ if (type !== 'toolkits') return 0;++ const slug = getToolkitSlugFromUrl(url);+ if (!slug) return 0;++ const toolkit = toolkitBySlug.get(slug);+ const override = TOOLKIT_POPULARITY_OVERRIDES[slug] ?? 0;+ const managedAuthBoost = toolkit?.composioManagedAuthSchemes?.length ? 120 : 0;+ const authBoost = toolkit?.authSchemes?.includes('OAUTH2') ? 40 : 0;+ const triggerBoost = Math.min((toolkit?.triggerCount ?? 0) * 10, 80);+ const toolCountBoost = Math.min((toolkit?.toolCount ?? 0), 100);++ return override + managedAuthBoost + authBoost + triggerBoost + toolCountBoost;+}++function pageRank(url: string, type: string): number {+ // Prefer conceptual docs over generated/reference material when textual+ // relevance is otherwise close. Toolkit aliases can still win earlier via+ // searchableAttributes when the query matches a tool name/slug exactly.+ if (type === 'docs') {+ if (url === '/docs' || url === '/docs/') return 2_400;+ if (url.includes('/quickstart')) return 2_300;+ if (url.includes('/authentication')) return 2_220;+ if (url.includes('/tools-and-toolkits')) return 2_180;+ if (url.includes('/configuring-sessions')) return 2_120;+ return 2_000;+ }++ if (type === 'cookbooks') return 1_500;+ if (type === 'toolkits') return 1_250;+ // Current v3.1 reference should be available, but conceptual docs should+ // win whenever both match. Legacy v3 reference is only a last-resort result.+ if (type === 'reference') return 650;+ if (type === 'api-reference') return 560;+ if (type === 'v3-reference') return 25;+ if (type === 'changelog') return 300;+ return 400;+}++function recordsFromMarkdownPage(input: {+ url: string;+ type: string;+ title: string;+ description?: string;+ keywords?: string[];+ markdown: string;+ breadcrumbs?: string[];+ tags?: string[];+ toolNames?: string[];+ toolSlugs?: string[];+}): AlgoliaDocsRecord[] {+ const clean = mdxToCleanMarkdown(input.markdown);+ const lines = clean.split('\n');+ const headingSlugs = new Map<string, number>();+ const headings: string[] = [];+ const sections: Array<{ heading?: string; section_id?: string; depth: number; text: string; position: number }> = [];+ let currentHeading: string | undefined;+ let currentSectionId: string | undefined;+ let currentDepth = 0;+ let currentLines: string[] = [];+ let sectionPosition = 0;++ const flush = () => {+ const text = currentLines.join('\n').trim();+ if (!text) return;+ sections.push({+ heading: currentHeading,+ section_id: currentSectionId,+ depth: currentDepth,+ text,+ position: sectionPosition++,+ });+ currentLines = [];+ };++ for (const line of lines) {+ const match = line.match(/^(#{1,6})\s+(.+)$/);+ if (match) {+ flush();+ const depth = match[1].length;+ const heading = match[2].trim();+ currentHeading = heading;+ currentSectionId = uniqueSlug(heading, headingSlugs);+ currentDepth = depth;+ headings.push(heading);+ currentLines.push(line);+ continue;+ }++ currentLines.push(line);+ }+ flush();++ const fallbackText = clean.trim() || [input.title, input.description, ...(input.keywords ?? [])].filter(Boolean).join('\n');+ if (sections.length === 0 && fallbackText) {+ sections.push({ text: fallbackText, position: 0, depth: 0 });+ }++ let recordPosition = 0;+ return sections.flatMap((section) => {+ const chunks = splitIntoChunks(section.text);+ return chunks.map((chunk, chunkIndex) => {+ const sectionPart = section.section_id ?? 'overview';+ const objectID = `${input.url}__${sectionPart}__${chunkIndex}__${contentHash(chunk)}`;+ const sectionRank = Math.max(10, 120 - section.depth * 12 - chunkIndex * 2);++ const position = recordPosition++;+ const includeToolkitAliases = input.type === 'toolkits' && position === 0;++ return {+ objectID,+ title: input.title,+ description: input.description,+ breadcrumbs: input.breadcrumbs,+ url: input.url,+ page_id: input.url,+ section: section.heading,+ section_id: section.section_id,+ content: chunk,+ keywords: input.keywords,+ slug: slugTokens(input.url),+ headings,+ tool_names: includeToolkitAliases ? input.toolNames : undefined,+ tool_slugs: includeToolkitAliases ? input.toolSlugs : undefined,+ type: input.type,+ lang: 'en',+ tags: input.tags,+ page_rank: pageRank(input.url, input.type),+ toolkit_popularity: toolkitPopularity(input.url, input.type),+ section_rank: sectionRank,+ position,+ depth: section.depth,+ } satisfies AlgoliaDocsRecord;+ });+ });+}++const toolkitBySlug = new Map(getAllToolkitsSync().map((toolkit) => [toolkit.slug, toolkit]));++function limitToolkitAliases(values: string[]): string[] {+ const aliases: string[] = [];+ let bytes = 0;++ for (const value of values) {+ const trimmed = value.trim();+ if (!trimmed) continue;++ const nextBytes = Buffer.byteLength(JSON.stringify(trimmed), 'utf8');+ if (aliases.length >= MAX_TOOL_ALIAS_ITEMS || bytes + nextBytes > MAX_TOOL_ALIAS_BYTES) break;++ aliases.push(trimmed);+ bytes += nextBytes;+ }++ return aliases;+}++function getToolkitSearchFields(slug: string): Pick<Parameters<typeof recordsFromMarkdownPage>[0], 'toolNames' | 'toolSlugs' | 'tags'> {+ const toolkit = toolkitBySlug.get(slug);+ if (!toolkit) return {};++ return {+ toolNames: limitToolkitAliases(toolkit.tools.map((tool) => tool.name).filter(Boolean)),+ toolSlugs: limitToolkitAliases(toolkit.tools.map((tool) => tool.slug).filter(Boolean)),+ tags: [toolkit.category].filter(Boolean) as string[],+ };+}++function getFilesystemRecords(): AlgoliaDocsRecord[] {+ const contentDir = join(process.cwd(), 'content');+ return listContentFiles(contentDir).flatMap((file) => {+ const route = urlFromContentPath(file);+ if (!route) return [];++ const source = readFileSync(file, 'utf8');+ const frontmatter = getFrontmatter(source);+ const title = getFrontmatterValue(frontmatter, 'title');+ if (!title) return [];++ const toolkitFields = route.type === 'toolkits'+ ? getToolkitSearchFields(route.url.replace(/^\/toolkits\//, ''))+ : {};++ return recordsFromMarkdownPage({+ url: route.url,+ type: route.type,+ title,+ description: getFrontmatterValue(frontmatter, 'description'),+ keywords: getFrontmatterList(frontmatter, 'keywords'),+ markdown: source,+ breadcrumbs: breadcrumbsForUrl(route.url, route.type),+ ...toolkitFields,+ });+ });+}++// Dynamic toolkit entries from toolkits.json. We include a compact sample of tool+// names/slugs so queries like "gmail send email" can still find the Gmail toolkit+// page without indexing the full tool catalog as noisy giant records.+const mdxToolkitSlugs = new Set(+ toolkitsSource.getPages().map((page) => page.slugs.join('/')),+);++const dynamicToolkitIndexes = getAllToolkitsSync()+ .filter((toolkit) => !mdxToolkitSlugs.has(toolkit.slug))+ .map((toolkit) => ({+ id: `/toolkits/${toolkit.slug}`,+ title: toolkit.name,+ description: toolkit.description,+ url: `/toolkits/${toolkit.slug}`,+ structuredData: { headings: [], contents: [] },+ keywords: [toolkit.slug, toolkit.category].filter(Boolean) as string[],+ } satisfies SearchIndex));++function getDynamicToolkitRecords(): AlgoliaDocsRecord[] {+ return getAllToolkitsSync()+ .filter((toolkit) => !mdxToolkitSlugs.has(toolkit.slug))+ .flatMap((toolkit) => {+ const toolsText = (toolkit.tools ?? [])+ .slice(0, 60)+ .map((tool) => `${tool.slug ?? ''} ${tool.name ?? ''}`)+ .join('\n');++ return recordsFromMarkdownPage({+ url: `/toolkits/${toolkit.slug}`,+ type: 'toolkits',+ title: toolkit.name,+ description: toolkit.description,+ keywords: [toolkit.slug, toolkit.category].filter(Boolean) as string[],+ markdown: `# ${toolkit.name}\n\n${toolkit.description ?? ''}\n\n## Available tools\n\n${toolsText}`,+ breadcrumbs: breadcrumbsForUrl(`/toolkits/${toolkit.slug}`, 'toolkits'),+ ...getToolkitSearchFields(toolkit.slug),+ });+ });+}++function getChangelogIndexes(): SearchIndex[] {+ const changelogDir = join(process.cwd(), 'content/changelog');+ if (!existsSync(changelogDir)) return [];++ return readdirSync(changelogDir)+ .filter((file) => file.endsWith('.mdx'))+ .flatMap((file) => {+ const source = readFileSync(join(changelogDir, file), 'utf8');+ const frontmatter = getFrontmatter(source);+ const date = getFrontmatterValue(frontmatter, 'date');+ const title = getFrontmatterValue(frontmatter, 'title');+ if (!date || !title) return [];++ const url = `/docs/changelog/${date.replace(/-/g, '/')}`;++ return [{+ id: `${url}#${title}`,+ title,+ description: getFrontmatterValue(frontmatter, 'description') ?? '',+ url,+ structuredData: { headings: [], contents: [] },+ keywords: ['changelog'],+ } satisfies SearchIndex];+ });+}++function getChangelogRecords(): AlgoliaDocsRecord[] {+ const changelogDir = join(process.cwd(), 'content/changelog');+ if (!existsSync(changelogDir)) return [];++ return readdirSync(changelogDir)+ .filter((file) => file.endsWith('.mdx'))+ .flatMap((file) => {+ const source = readFileSync(join(changelogDir, file), 'utf8');+ const frontmatter = getFrontmatter(source);+ const date = getFrontmatterValue(frontmatter, 'date');+ const title = getFrontmatterValue(frontmatter, 'title');+ if (!date || !title) return [];++ return recordsFromMarkdownPage({+ url: `/docs/changelog/${date.replace(/-/g, '/')}`,+ type: 'changelog',+ title,+ description: getFrontmatterValue(frontmatter, 'description'),+ keywords: ['changelog', date],+ markdown: source,+ breadcrumbs: breadcrumbsForUrl(`/docs/changelog/${date.replace(/-/g, '/')}`, 'changelog'),+ });+ });+}++export async function getDocsSearchIndexes(): Promise<SearchIndex[]> {+ // Load OpenAPI pages and build full reference source.+ const openapiPages = await openapiSource(openapi, {+ groupBy: 'tag',+ baseDir: 'api-reference',+ });++ const fullReferenceSource = loader({+ baseUrl: '/reference',+ source: multiple({+ mdx: reference.toFumadocsSource(),+ openapi: openapiPages,+ }),+ plugins: [lucideIconsPlugin(), openapiPlugin()],+ });++ const mdxIndexes = [+ ...docsSource.getPages(),+ ...cookbooksSource.getPages(),+ ...toolkitsSource.getPages(),+ ...fullReferenceSource.getPages(),+ ].map((page) => ({+ id: page.url,+ title: page.data.title ?? 'Untitled',+ description: page.data.description,+ url: page.url,+ structuredData: page.data.structuredData,+ keywords: 'keywords' in page.data ? (page.data.keywords as string[]) : undefined,+ } satisfies SearchIndex));++ return [...mdxIndexes, ...dynamicToolkitIndexes, ...getChangelogIndexes()];+}++async function getOpenApiRecords(): Promise<AlgoliaDocsRecord[]> {+ const openapiPages = await openapiSource(openapi, {+ groupBy: 'tag',+ baseDir: 'api-reference',+ });++ const openapiOnlySource = loader({+ baseUrl: '/reference',+ source: openapiPages,+ plugins: [lucideIconsPlugin(), openapiPlugin()],+ });++ return openapiOnlySource.getPages().flatMap((page) => {+ const contents = [+ page.data.description,+ ...(page.data.structuredData?.headings ?? []).map((heading) => heading.content),+ ...(page.data.structuredData?.contents ?? []).map((content) => content.content),+ ].filter(Boolean).join('\n\n');++ return recordsFromMarkdownPage({+ url: page.url,+ type: 'api-reference',+ title: page.data.title ?? 'Untitled',+ description: page.data.description,+ markdown: `# ${page.data.title ?? 'Untitled'}\n\n${contents}`,+ breadcrumbs: breadcrumbsForUrl(page.url, 'api-reference'),+ });+ });+}++export async function getAlgoliaSearchDocuments(): Promise<AlgoliaDocsRecord[]> {+ const records = [+ ...getFilesystemRecords(),+ ...getDynamicToolkitRecords(),+ ...getChangelogRecords(),+ ...await getOpenApiRecords(),+ ];++ const deduped = new Map<string, AlgoliaDocsRecord>();+ for (const record of records) deduped.set(record.objectID, record);++ return Array.from(deduped.values());+}diff --git a/docs/package.json b/docs/package.jsonindex 3585453aa..a7b1f231b 100644--- a/docs/package.json+++ b/docs/package.json@@ -10,6 +10,8 @@"postinstall": "fumadocs-mdx","lint": "eslint","lint:links": "bun scripts/validate-links.ts",+ "sync:search": "bun scripts/sync-algolia-search.ts",+ "test:search": "bun scripts/test-algolia-search.ts","generate:toolkits": "bun scripts/generate-toolkits.ts","generate:meta-tools": "bun scripts/generate-meta-tools.ts","generate:api-index": "bun scripts/generate-api-index.ts",@@ -20,6 +22,7 @@"@radix-ui/react-tabs": "^1.1.13","@shikijs/twoslash": "^3.21.0","@vercel/analytics": "^1.6.1",+ "algoliasearch": "^5","class-variance-authority": "^0.7.1","clsx": "^2.1.1","fumadocs-core": "^16.4.2",@@ -32,6 +35,7 @@"posthog-js": "^1.319.0","react": "^19.2.3","react-dom": "^19.2.3",+ "search-insights": "^2","shiki": "^3.20.0","tailwind-merge": "^3.4.0"},diff --git a/docs/scripts/sync-algolia-search.ts b/docs/scripts/sync-algolia-search.tsnew file mode 100644index 000000000..1ddc278a4--- /dev/null+++ b/docs/scripts/sync-algolia-search.ts@@ -0,0 +1,102 @@+import { algoliasearch } from 'algoliasearch';+import {+ ALGOLIA_DEFAULT_APP_ID,+ ALGOLIA_DEFAULT_INDEX_NAME,+ getAlgoliaSearchDocuments,+} from '@/lib/search-index';++function requireEnv(name: string, fallback?: string): string {+ const value = process.env[name] ?? fallback;+ if (!value) {+ throw new Error(`Missing required environment variable: ${name}`);+ }+ return value;+}++const dryRun = process.argv.includes('--dry-run');+const showSamples = process.argv.includes('--samples');+const indexName =+ process.env.ALGOLIA_INDEX_NAME ?? process.env.NEXT_PUBLIC_ALGOLIA_INDEX_NAME ?? ALGOLIA_DEFAULT_INDEX_NAME;++const records = await getAlgoliaSearchDocuments();++if (records.length === 0) {+ throw new Error('Refusing to sync an empty Algolia docs index.');+}++if (dryRun) {+ const uniquePages = new Set(records.map((record) => record.page_id)).size;+ console.log(`Prepared ${records.length} Algolia records across ${uniquePages} docs pages for index "${indexName}".`);++ if (showSamples) {+ console.log(JSON.stringify(records.slice(0, 5), null, 2));+ }++ process.exit(0);+}++const appId = requireEnv(+ 'ALGOLIA_APP_ID',+ process.env.NEXT_PUBLIC_ALGOLIA_APP_ID ?? ALGOLIA_DEFAULT_APP_ID,+);+const adminApiKey = requireEnv('ALGOLIA_ADMIN_API_KEY');+const client = algoliasearch(appId, adminApiKey);++console.log(`Configuring Algolia index "${indexName}"...`);+await client.setSettings({+ indexName,+ indexSettings: {+ attributeForDistinct: 'page_id',+ attributesToRetrieve: [+ 'objectID',+ 'title',+ 'description',+ 'section',+ 'content',+ 'url',+ 'section_id',+ 'breadcrumbs',+ 'page_id',+ 'type',+ ],+ searchableAttributes: [+ 'unordered(title)',+ 'unordered(keywords)',+ 'unordered(slug)',+ 'unordered(section)',+ 'unordered(headings)',+ 'unordered(tool_names)',+ 'unordered(tool_slugs)',+ 'unordered(description)',+ 'unordered(content)',+ ],+ customRanking: [+ 'desc(page_rank)',+ 'desc(toolkit_popularity)',+ 'desc(section_rank)',+ 'asc(position)',+ ],+ attributesForFaceting: [+ 'filterOnly(type)',+ 'filterOnly(lang)',+ 'searchable(tags)',+ ],+ attributesToHighlight: ['title', 'section', 'content'],+ attributesToSnippet: ['content:30'],+ highlightPreTag: '<mark>',+ highlightPostTag: '</mark>',+ ignorePlurals: true,+ minProximity: 1,+ removeStopWords: false,+ removeWordsIfNoResults: 'lastWords',+ typoTolerance: true,+ advancedSyntax: true,+ },+});++console.log(`Replacing ${records.length} records in Algolia index "${indexName}"...`);+await client.replaceAllObjects({+ indexName,+ objects: records.map((record) => ({ ...record })),+});+console.log(`Synced Algolia docs search index "${indexName}".`);diff --git a/docs/scripts/test-algolia-search.ts b/docs/scripts/test-algolia-search.tsnew file mode 100644index 000000000..3e583356a--- /dev/null+++ b/docs/scripts/test-algolia-search.ts@@ -0,0 +1,56 @@+import { liteClient } from 'algoliasearch/lite';+import type { AlgoliaDocsRecord } from '@/lib/search-index';+import { ALGOLIA_DEFAULT_APP_ID, ALGOLIA_DEFAULT_INDEX_NAME } from '@/lib/search-index';++function requireEnv(name: string, fallback?: string): string {+ const value = process.env[name] ?? fallback;+ if (!value) throw new Error(`Missing required environment variable: ${name}`);+ return value;+}++const appId = requireEnv('ALGOLIA_APP_ID', process.env.NEXT_PUBLIC_ALGOLIA_APP_ID ?? ALGOLIA_DEFAULT_APP_ID);+const searchApiKey = requireEnv(+ 'ALGOLIA_SEARCH_API_KEY',+ process.env.NEXT_PUBLIC_ALGOLIA_SEARCH_API_KEY ?? process.env.ALGOLIA_ADMIN_API_KEY,+);+const indexName = process.env.ALGOLIA_INDEX_NAME ?? process.env.NEXT_PUBLIC_ALGOLIA_INDEX_NAME ?? ALGOLIA_DEFAULT_INDEX_NAME;+const queries = process.argv.slice(2);++if (queries.length === 0) {+ queries.push(+ 'quickstart',+ 'oauth auth config',+ 'create connected account',+ 'gmail send email',+ 'tool router session search',+ 'webhook verification',+ );+}++const client = liteClient(appId, searchApiKey);++for (const query of queries) {+ const result = await client.searchForHits<AlgoliaDocsRecord>({+ requests: [+ {+ type: 'default',+ indexName,+ query,+ hitsPerPage: 5,+ distinct: true,+ getRankingInfo: true,+ clickAnalytics: true,+ },+ ],+ });++ const response = result.results[0];+ console.log(`\n## ${query}`);+ response.hits.forEach((hit, index) => {+ const section = hit.section ? ` — ${hit.section}` : '';+ const snippet = hit.content.replace(/\s+/g, ' ').slice(0, 180);+ console.log(`${index + 1}. ${hit.title}${section}`);+ console.log(` ${hit.url}`);+ console.log(` ${snippet}${snippet.length === 180 ? '…' : ''}`);+ });+}