An AI-powered search engine with a generative UI
TypeScript generative-aigenerative-uinextjsollamareactsearxngshadcn-uitailwindcsstavilytypescriptupstashvercel-ai-sdk
feat(fetch): add self-hosted crawl4ai backend (#1067)
relic664efcafd20aee8
diff --git a/.env.local.example b/.env.local.exampleindex f3af581d..32f2e698 100644--- a/.env.local.example+++ b/.env.local.example@@ -157,6 +157,10 @@ ANONYMOUS_USER_ID=anonymous-user# Default: Tavily Extract API# Alternative: Jina AI - Get your API key at: https://jina.ai/# JINA_API_KEY=[YOUR_JINA_API_KEY]+# Alternative: self-hosted crawl4ai (handles all fetches when set, including+# PDFs; takes precedence over Jina and Tavily)+# CRAWL4AI_BASE_URL=http://localhost:11235+# CRAWL4AI_API_KEY=[YOUR_CRAWL4AI_API_KEY]# -----------------------------------------------------------------------------# Observability & Monitoringdiff --git a/docs/CONFIGURATION.md b/docs/CONFIGURATION.mdindex ad1621be..e44eeccd 100644--- a/docs/CONFIGURATION.md+++ b/docs/CONFIGURATION.md@@ -483,3 +483,11 @@ Use Jina for enhanced content extraction:```bashJINA_API_KEY=[YOUR_API_KEY]```++Alternatively, a self-hosted [crawl4ai](https://github.com/unclecode/crawl4ai) instance can serve as the fetch backend. When `CRAWL4AI_BASE_URL` is set it handles every fetch the tool performs — both regular and API extraction, PDFs included — and takes precedence over Jina and Tavily. The URL is sent to the crawler's `/crawl` pipeline in two passes: first as a page with the same fit filter the server's `/md` endpoint applies, then, if that yields nothing (Chromium cannot render PDFs), with the server's dedicated PDF scraping strategy. No request to the target is made from the Morphic server itself; detection runs through the crawler:++```bash+CRAWL4AI_BASE_URL=http://localhost:11235+# Optional: only needed when the instance requires an API key+# CRAWL4AI_API_KEY=[YOUR_CRAWL4AI_API_KEY]+```diff --git a/docs/PRIVACY.md b/docs/PRIVACY.mdindex ede60d11..a9c4624d 100644--- a/docs/PRIVACY.md+++ b/docs/PRIVACY.md@@ -52,7 +52,7 @@ SearXNG keeps this on your own infrastructure, and the Docker Compose setup ship### Reading pages-By default the `fetch` tool retrieves a URL directly from your server, so only the origin site sees the request. It routes through an extraction service when the agent asks for API extraction, which it does for PDFs and JavaScript-rendered pages: Jina when `JINA_API_KEY` is set, Tavily Extract otherwise.+By default the `fetch` tool retrieves a URL directly from your server, so only the origin site sees the request. It routes through an extraction service when the agent asks for API extraction, which it does for PDFs and JavaScript-rendered pages: your self-hosted crawl4ai instance when `CRAWL4AI_BASE_URL` is set, Jina when `JINA_API_KEY` is set, Tavily Extract otherwise. When `CRAWL4AI_BASE_URL` is set, crawl4ai handles every fetch — regular and API extraction both, PDF detection included — so no page is retrieved directly from your server and the origin only ever sees requests from your crawler.The Tavily Extract path does not check for a key first, so an instance with no `TAVILY_API_KEY` still posts the URL to Tavily and gets an authentication error back. If you want no requests to Tavily at all, set `JINA_API_KEY` or keep the agent off API extraction.diff --git a/lib/agents/prompts/search-mode-prompts.ts b/lib/agents/prompts/search-mode-prompts.tsindex acfae6bc..d0e386c9 100644--- a/lib/agents/prompts/search-mode-prompts.ts+++ b/lib/agents/prompts/search-mode-prompts.ts@@ -7,6 +7,21 @@ import {isGeneralSearchProviderAvailable} from '@/lib/utils/search-config'+// Whether the fetch tool's configured backend extracts PDFs itself. When a+// self-hosted crawl4ai instance is set, every fetch type runs through its+// /crawl pipeline and PDFs come back as text on either type; without it the+// regular path is a plain HTML fetch that hard-fails on a PDF body, so the+// agent must still route PDFs to the api extraction type.+function fetchBackendExtractsPdfs(): boolean {+ return !!process.env.CRAWL4AI_BASE_URL+}++function getPdfFetchGuidance(): string {+ return fetchBackendExtractsPdfs()+ ? '- **For PDF URLs**: use \`type: "regular"\` - PDFs are extracted and converted to text automatically'+ : '- **For PDF URLs (ending in .pdf)**: ALWAYS use \`type: "api"\` - regular type will fail on PDFs'+}+function getIdentityGuidance(): string {return `Identity:- You are Morphic, an AI-powered answer engine.@@ -88,7 +103,7 @@ Fetch tool usage:- **ONLY use fetch tool when a URL is directly provided by the user in their query**- Do NOT use fetch to get more details from search results- This keeps responses fast and efficient-- **For PDF URLs (ending in .pdf)**: ALWAYS use \`type: "api"\` - regular type will fail on PDFs+${getPdfFetchGuidance()}- **For regular web pages**: Use default \`type: "regular"\` for fast HTML fetchingCitation Format (MANDATORY):@@ -251,7 +266,7 @@ Fetch tool usage:- Use when you need deeper content analysis beyond search snippets- Fetch the top 2-3 most relevant/recent URLs for comprehensive coverage- Especially important for news, current events, and time-sensitive information-- **For PDF URLs (ending in .pdf)**: ALWAYS use \`type: "api"\` - regular type will fail on PDFs+${getPdfFetchGuidance()}- **For complex JavaScript-rendered pages**: Use \`type: "api"\` for better extraction- **For regular web pages**: Use default \`type: "regular"\` for fast HTML fetchingdiff --git a/lib/tools/__tests__/fetch-ssrf.test.ts b/lib/tools/__tests__/fetch-ssrf.test.tsindex 51d7dd7e..7080f8b0 100644--- a/lib/tools/__tests__/fetch-ssrf.test.ts+++ b/lib/tools/__tests__/fetch-ssrf.test.ts@@ -87,6 +87,18 @@ describe('fetch tool SSRF guard', () => {expect(fetchMock).not.toHaveBeenCalled()})+ // The same settlement applies when the extraction service is crawl4ai: the+ // target is checked before it leaves the server, independently of who fetches.+ it('refuses a name that resolves into private space on the crawl4ai path', async () => {+ const fetchMock = vi.fn()+ vi.stubGlobal('fetch', fetchMock)+ vi.stubEnv('CRAWL4AI_BASE_URL', 'http://crawl4ai.internal:11235')++ await expectBlocked('http://localhost/admin', 'api')++ expect(fetchMock).not.toHaveBeenCalled()+ })+it('never reaches a live private host, and never returns its body', async () => {vi.spyOn(console, 'error').mockImplementation(() => {})diff --git a/lib/tools/__tests__/fetch.test.ts b/lib/tools/__tests__/fetch.test.tsindex 8ee10402..73a25a8c 100644--- a/lib/tools/__tests__/fetch.test.ts+++ b/lib/tools/__tests__/fetch.test.ts@@ -101,3 +101,347 @@ describe('regular fetch content types', () => {)})})++describe('crawl4ai extraction', () => {+ const baseUrl = 'http://crawl4ai.internal:11235'+ const pageResponse = {+ success: true,+ results: [+ {+ url,+ redirected_url: url,+ success: true,+ markdown: {+ fit_markdown: '# Fitted page\n\nBody text.',+ raw_markdown: '# Fitted page\n\nBody text.'+ },+ metadata: { title: 'Fitted page' }+ }+ ]+ }++ function fetchCrawl4ai(handler: (init: RequestInit) => Response) {+ type FetchLike = (+ url: string | URL | Request,+ init?: RequestInit+ ) => Promise<Response>+ const fetchMock = vi.fn<FetchLike>(async (_url, init) =>+ handler(init ?? {})+ )+ vi.stubGlobal('fetch', fetchMock)+ vi.stubEnv('CRAWL4AI_BASE_URL', baseUrl)+ return fetchMock+ }++ type FetchMock = {+ mock: { calls: Array<[string | URL | Request, RequestInit?]> }+ }++ function crawlBody(call: 0 | 1): {+ urls: string[]+ crawler_config: Record<string, unknown>+ } {+ const fetchMock = globalThis.fetch as unknown as FetchMock+ return JSON.parse(fetchMock.mock.calls[call]?.[1]?.body as string) as {+ urls: string[]+ crawler_config: Record<string, unknown>+ }+ }++ async function runCrawl4ai(type: 'regular' | 'api' = 'api') {+ const result = fetchTool.execute?.(+ { url, type },+ { toolCallId: 'fetch', messages: [], context: {} }+ )+ const iterator = (result as AsyncIterable<unknown>)[Symbol.asyncIterator]()++ await iterator.next()+ return iterator.next()+ }++ afterEach(() => {+ vi.unstubAllGlobals()+ vi.unstubAllEnvs()+ vi.restoreAllMocks()+ })++ it('routes regular-type fetches through crawl4ai when configured', async () => {+ const fetchMock = fetchCrawl4ai(+ () => new Response(JSON.stringify(pageResponse))+ )++ const result = fetchTool.execute?.(+ { url, type: 'regular' },+ { toolCallId: 'fetch', messages: [], context: {} }+ )+ const iterator = (result as AsyncIterable<unknown>)[Symbol.asyncIterator]()+ await iterator.next()+ const complete = await iterator.next()++ expect(fetchMock.mock.calls[0]?.[0]).toBe(`${baseUrl}/crawl`)+ expect(complete.value).toMatchObject({+ state: 'complete',+ results: [+ {+ title: 'Fitted page',+ content: pageResponse.results[0].markdown.fit_markdown,+ url+ }+ ]+ })+ })++ it('renders pages with the fit generator on the first pass', async () => {+ fetchCrawl4ai(() => new Response(JSON.stringify(pageResponse)))++ await runCrawl4ai()++ expect(crawlBody(0).crawler_config).toMatchObject({+ markdown_generator: {+ type: 'DefaultMarkdownGenerator',+ params: {+ content_filter: { type: 'PruningContentFilterLXML', params: {} }+ }+ }+ })+ })++ it('retries a PDF with the dedicated strategy when the page pass yields nothing', async () => {+ const pdfUrl = 'https://example.com/paper.pdf'+ const crawlResponse = {+ success: true,+ results: [+ {+ url: pdfUrl,+ redirected_url: pdfUrl,+ success: true,+ markdown: { raw_markdown: 'PDF text content', fit_markdown: '' },+ metadata: { title: 'A Paper' }+ }+ ]+ }+ const fetchMock = fetchCrawl4ai(init =>+ JSON.parse(String(init?.body)).crawler_config.scraping_strategy+ ? new Response(JSON.stringify(crawlResponse))+ : new Response(+ JSON.stringify({+ success: true,+ results: [+ {+ url: pdfUrl,+ success: true,+ markdown: { fit_markdown: '', raw_markdown: '' }+ }+ ]+ })+ )+ )++ const result = fetchTool.execute?.(+ { url: pdfUrl, type: 'regular' },+ { toolCallId: 'fetch', messages: [], context: {} }+ )+ const iterator = (result as AsyncIterable<unknown>)[Symbol.asyncIterator]()+ await iterator.next()+ const complete = await iterator.next()++ expect(fetchMock).toHaveBeenCalledTimes(2)+ expect(fetchMock.mock.calls[1]?.[0]).toBe(`${baseUrl}/crawl`)+ expect(crawlBody(1).crawler_config).toMatchObject({+ scraping_strategy: { type: 'PDFContentScrapingStrategy', params: {} }+ })+ expect(complete.value).toMatchObject({+ state: 'complete',+ results: [{ title: 'A Paper', content: 'PDF text content', url: pdfUrl }]+ })+ })++ it('falls back to raw markdown when the fit pruner drops everything', async () => {+ const sparse = {+ success: true,+ results: [+ {+ url,+ success: true,+ markdown: { fit_markdown: '', raw_markdown: 'just a line' },+ metadata: null+ }+ ]+ }+ fetchCrawl4ai(() => new Response(JSON.stringify(sparse)))++ const complete = await runCrawl4ai()++ expect(complete.value).toMatchObject({+ results: [{ title: 'example.com', content: 'just a line' }]+ })+ })++ it('uses the redirected URL from the crawl result', async () => {+ const moved = {+ success: true,+ results: [+ {+ url,+ redirected_url: 'https://www.example.com/final',+ success: true,+ markdown: { fit_markdown: 'moved content' },+ metadata: null+ }+ ]+ }+ fetchCrawl4ai(() => new Response(JSON.stringify(moved)))++ const complete = await runCrawl4ai()++ expect(complete.value).toMatchObject({+ results: [{ url: 'https://www.example.com/final' }]+ })+ })++ it('posts the page pass to /crawl with fit mode', async () => {+ const fetchMock = fetchCrawl4ai(+ () => new Response(JSON.stringify(pageResponse))+ )++ await runCrawl4ai()++ expect(crawlBody(0).urls).toEqual([url])+ })++ it('prefers crawl4ai over Jina when both are configured', async () => {+ const fetchMock = fetchCrawl4ai(+ () => new Response(JSON.stringify(pageResponse))+ )+ vi.stubEnv('JINA_API_KEY', 'test-jina-key')++ await runCrawl4ai()++ expect(fetchMock.mock.calls[0]?.[0]).toBe(`${baseUrl}/crawl`)+ })++ it('normalizes a trailing slash on the configured base URL', async () => {+ const fetchMock = fetchCrawl4ai(+ () => new Response(JSON.stringify(pageResponse))+ )+ vi.stubEnv('CRAWL4AI_BASE_URL', `${baseUrl}/`)++ await runCrawl4ai()++ expect(fetchMock.mock.calls[0]?.[0]).toBe(`${baseUrl}/crawl`)+ })++ it('normalizes multiple trailing slashes on the configured base URL', async () => {+ const fetchMock = fetchCrawl4ai(+ () => new Response(JSON.stringify(pageResponse))+ )+ vi.stubEnv('CRAWL4AI_BASE_URL', `${baseUrl}///`)++ await runCrawl4ai()++ expect(fetchMock.mock.calls[0]?.[0]).toBe(`${baseUrl}/crawl`)+ })++ it('sends the API key when one is configured', async () => {+ vi.stubEnv('CRAWL4AI_API_KEY', 'test-crawl4ai-key')+ const fetchMock = fetchCrawl4ai(+ () => new Response(JSON.stringify(pageResponse))+ )++ await runCrawl4ai()++ expect(fetchMock.mock.calls[0]?.[1]?.headers).toMatchObject({+ Authorization: 'Bearer test-crawl4ai-key'+ })+ })++ it('falls back to Tavily Extract without CRAWL4AI_BASE_URL', async () => {+ vi.spyOn(console, 'error').mockImplementation(() => {})+ type FetchLike = (url: string | URL | Request) => Promise<Response>+ const fetchMock = vi.fn<FetchLike>(+ async () => new Response(JSON.stringify({ results: [] }), { status: 403 })+ )+ vi.stubGlobal('fetch', fetchMock)+ vi.stubEnv('TAVILY_API_KEY', 'test-tavily-key')++ await expect(runCrawl4ai()).rejects.toThrow(+ 'No results returned from content extraction service'+ )+ expect(fetchMock.mock.calls[0]?.[0]).toBe('https://api.tavily.com/extract')+ })++ it('keeps regular fetches direct without CRAWL4AI_BASE_URL', async () => {+ type FetchLike = (url: string | URL | Request) => Promise<Response>+ const fetchMock = vi.fn<FetchLike>(+ async () =>+ new Response(+ '<html><title>Page title</title><body>Hello</body></html>',+ {+ headers: { 'content-type': 'text/html' }+ }+ )+ )+ vi.stubGlobal('fetch', fetchMock)++ const result = fetchTool.execute?.(+ { url, type: 'regular' },+ { toolCallId: 'fetch', messages: [], context: {} }+ )+ const iterator = (result as AsyncIterable<unknown>)[Symbol.asyncIterator]()+ await iterator.next()+ const complete = await iterator.next()++ expect(fetchMock.mock.calls[0]?.[0]).toBe(url)+ expect(complete.value).toMatchObject({+ results: [{ content: 'Page title Hello' }]+ })+ })++ it('rejects an unsuccessful crawl4ai response', async () => {+ vi.spyOn(console, 'error').mockImplementation(() => {})+ fetchCrawl4ai(+ () => new Response(JSON.stringify({ detail: 'boom' }), { status: 500 })+ )++ await expect(runCrawl4ai()).rejects.toThrow(+ 'crawl4ai request failed: HTTP 500'+ )+ })++ it('reports the server error when both passes yield nothing', async () => {+ vi.spyOn(console, 'error').mockImplementation(() => {})+ const failing = {+ success: true,+ results: [+ {+ url,+ success: false,+ error_message: 'seek of closed file',+ markdown: null+ }+ ]+ }+ fetchCrawl4ai(() => new Response(JSON.stringify(failing)))++ await expect(runCrawl4ai()).rejects.toThrow('seek of closed file')+ })++ it('refuses a private result URL reported by the crawler', async () => {+ vi.spyOn(console, 'error').mockImplementation(() => {})+ const internal = {+ success: true,+ results: [+ {+ url,+ redirected_url: 'http://169.254.169.254/latest/meta-data/',+ success: true,+ markdown: { fit_markdown: 'stolen credentials' },+ metadata: null+ }+ ]+ }+ fetchCrawl4ai(() => new Response(JSON.stringify(internal)))++ await expect(runCrawl4ai()).rejects.toThrow('Blocked fetch URL.')+ })+})diff --git a/lib/tools/fetch.ts b/lib/tools/fetch.tsindex 1ae0dbcc..418408a1 100644--- a/lib/tools/fetch.ts+++ b/lib/tools/fetch.ts@@ -187,15 +187,192 @@ async function fetchTavilyExtractData(url: string): Promise<SearchResultsType> {}}+// Everything goes through the server's /crawl pipeline, in two deliberate+// passes. The first pass renders the page in a browser with the same fit+// filter the server's /md endpoint pins (DefaultMarkdownGenerator ++// PruningContentFilterLXML), and its response carries the content type and+// metadata the old /md string collapsed away. Chromium cannot render PDFs, so+// anything the first pass fails to extract gets a second pass with the PDF+// scraping strategy, which makes the server swap in its dedicated PDF crawler.+// Detection therefore happens through the configured backend: no request to+// the origin is made from Morphic itself, and a mis-suffixed URL costs one+// fast failed pass rather than misrouting the fetch.+interface Crawl4aiCrawlResult {+ url?: string+ redirected_url?: string | null+ success?: boolean+ error_message?: string | null+ metadata?: { title?: string | null } | null+ markdown?: {+ raw_markdown?: string | null+ fit_markdown?: string | null+ } | null+}++const CRAWL4AI_PAGE_TIMEOUT_MS = 30000+// PDFs download and parse server-side, which is slower than a page render.+const CRAWL4AI_PDF_TIMEOUT_MS = 60000++async function postCrawl4ai(+ baseUrl: string,+ url: string,+ crawlerConfig: Record<string, unknown>,+ timeoutMs: number+): Promise<{+ success?: boolean+ results?: Crawl4aiCrawlResult[]+}> {+ try {+ const response = await fetch(`${baseUrl}/crawl`, {+ method: 'POST',+ headers: {+ Authorization: `Bearer ${process.env.CRAWL4AI_API_KEY ?? ''}`,+ 'Content-Type': 'application/json'+ },+ body: JSON.stringify({ urls: [url], crawler_config: crawlerConfig }),+ // AbortSignal.timeout stays armed through body consumption, so a server+ // that sends headers and stalls mid-body still hits the deadline.+ signal: AbortSignal.timeout(timeoutMs)+ })++ if (!response.ok) {+ throw new Error(+ `crawl4ai request failed: HTTP ${response.status} ${response.statusText}`+ )+ }++ return await response.json()+ } catch (error) {+ if (+ error instanceof Error &&+ (error.name === 'AbortError' || error.name === 'TimeoutError')+ ) {+ throw new Error(+ `crawl4ai request timed out after ${timeoutMs / 1000} seconds`+ )+ }+ throw error+ }+}++function crawl4aiMarkdown(result: Crawl4aiCrawlResult): string {+ // fit_markdown is the pruned read the fit filter produces; it is empty for+ // PDFs (nothing to prune) and for sparse pages the pruner rejects, where the+ // raw conversion is still worth reading back.+ return result.markdown?.fit_markdown || result.markdown?.raw_markdown || ''+}++function crawl4aiSearchResults(+ url: string,+ result: Crawl4aiCrawlResult,+ markdown: string+): SearchResultsType {+ const content = markdown.slice(0, CONTENT_CHARACTER_LIMIT)+ const resultUrl = result.redirected_url || result.url || url+ // The server validated its own hops, but this URL is what comes back into+ // Morphic for display, so it gets the same public-space gate every URL in a+ // result must pass. It throws on a private or malformed reported target.+ assertPublicUrl(resultUrl)+ const rawTitle = result.metadata?.title?.trim() || new URL(resultUrl).hostname+ const title =+ rawTitle.length > TITLE_CHARACTER_LIMIT+ ? rawTitle.substring(0, TITLE_CHARACTER_LIMIT) + '...'+ : rawTitle++ return {+ results: [+ {+ title,+ content,+ url: resultUrl+ }+ ],+ query: '',+ images: []+ }+}++function crawl4aiPayload(json: {+ success?: boolean+ results?: Crawl4aiCrawlResult[]+}): { success: boolean; result: Crawl4aiCrawlResult; markdown: string } {+ // The envelope carries the request-level success; the per-URL result holds+ // the markdown, its own success, and any server-side error message.+ const result = json.results?.[0] ?? {}+ return {+ success: Boolean(json.success && result.success),+ result,+ markdown: crawl4aiMarkdown(result)+ }+}++async function fetchCrawl4aiData(url: string): Promise<SearchResultsType> {+ // Strip trailing slashes so `http://crawler:11235/` never degrades into+ // `//crawl`: this backend takes over every fetch, so a malformed route+ // would break all of them at once.+ const baseUrl = process.env.CRAWL4AI_BASE_URL?.replace(/\/+$/, '')+ if (!baseUrl) {+ throw new Error('CRAWL4AI_BASE_URL is not configured')+ }++ const first = crawl4aiPayload(+ await postCrawl4ai(+ baseUrl,+ url,+ {+ markdown_generator: {+ type: 'DefaultMarkdownGenerator',+ params: {+ content_filter: { type: 'PruningContentFilterLXML', params: {} }+ }+ }+ },+ CRAWL4AI_PAGE_TIMEOUT_MS+ )+ )+ if (first.success && first.markdown) {+ return crawl4aiSearchResults(url, first.result, first.markdown)+ }++ const second = crawl4aiPayload(+ await postCrawl4ai(+ baseUrl,+ url,+ { scraping_strategy: { type: 'PDFContentScrapingStrategy', params: {} } },+ CRAWL4AI_PDF_TIMEOUT_MS+ )+ )+ if (second.success && second.markdown) {+ return crawl4aiSearchResults(url, second.result, second.markdown)+ }++ // The second pass is the deliberate look, so its server-side message is the+ // most specific; fall back to the first pass's for context.+ throw new Error(+ second.result.error_message ||+ first.result.error_message ||+ 'No data returned from crawl4ai'+ )+}+export function createFetchTool(options?: {labelAllocator?: CitationLabelAllocator}) {const labelAllocator =options?.labelAllocator ?? createCitationLabelAllocator()+ // The PDF guidance must match the configured backend: a self-hosted+ // crawl4ai instance extracts PDFs on either type, while the plain HTML+ // path (and the Jina/Tavily extraction services behind "api") only serve+ // PDFs through "api". Without crawl4ai the description is byte-identical+ // to the one on main: deployments that never set the variable must not+ // see their prompts move with this change.+ const description = process.env.CRAWL4AI_BASE_URL+ ? 'Fetch content from any URL and get clean markdown back. Use "regular" (default) for web pages and "api" for pages that need advanced handling. PDFs are extracted and converted to text on either type. The "api" type also gives deeper extraction for complex JavaScript-rendered pages, better markdown formatting, and table extraction. When a crawl4ai instance is configured via CRAWL4AI_BASE_URL it serves both types; otherwise "api" uses Jina Reader if available and falls back to Tavily Extract.'+ : 'Fetch content from any URL. By default uses "regular" type which performs fast, direct HTML fetching without external APIs - ideal for most websites. IMPORTANT: "regular" type does NOT support PDFs and will fail on PDF URLs. Use "api" type when you need: 1) PDF content extraction (required for .pdf URLs), 2) Complex JavaScript-rendered pages, 3) Better markdown formatting, 4) Table extraction. The "api" type requires Jina or Tavily API keys and uses Jina Reader if available, otherwise falls back to Tavily Extract.'+return tool({- description:- 'Fetch content from any URL. By default uses "regular" type which performs fast, direct HTML fetching without external APIs - ideal for most websites. IMPORTANT: "regular" type does NOT support PDFs and will fail on PDF URLs. Use "api" type when you need: 1) PDF content extraction (required for .pdf URLs), 2) Complex JavaScript-rendered pages, 3) Better markdown formatting, 4) Table extraction. The "api" type requires Jina or Tavily API keys and uses Jina Reader if available, otherwise falls back to Tavily Extract.',+ description,inputSchema: fetchSchema,async *execute({ url, type = 'regular' }) {assertFetchableUrl(url)@@ -218,7 +395,18 @@ export function createFetchTool(options?: {// Only the retrieval is wrapped: a yield rejects when the consumer stops// reading, and an aborted stream is not a failure of the page.try {- if (type === 'regular') {+ if (process.env.CRAWL4AI_BASE_URL) {+ // A configured crawl4ai instance is the fetch backend for every+ // type: it takes over regular fetching too, not just API extraction.+ // Settlement before the handoff stops Morphic from pointing the+ // crawler at private space; the crawler's own redirect hops and+ // connected peers are guarded server-side (seed validation, the+ // pinning egress proxy for browser hops, per-hop validators for the+ // PDF crawler), and anything it reports back passes the public gate+ // again inside fetchCrawl4aiData before it reaches a result.+ await assertResolvedPublicUrl(url)+ results = await fetchCrawl4aiData(url)+ } else if (type === 'regular') {// Use regular fetch for direct HTML retrievalresults = await fetchRegularData(url)} else {@@ -226,7 +414,8 @@ export function createFetchTool(options?: {// be settled before it is handed over rather than at connect time.await assertResolvedPublicUrl(url)- // Use API-based extraction (Jina or Tavily)+ // Use API-based extraction (Jina or Tavily; crawl4ai, when+ // configured, already took the branch above for every type)const useJina = process.env.JINA_API_KEYif (useJina) {results = await fetchJinaReaderData(url)diff --git a/lib/utils/safe-fetch.ts b/lib/utils/safe-fetch.tsindex d4d10b39..c812f4e6 100644--- a/lib/utils/safe-fetch.ts+++ b/lib/utils/safe-fetch.ts@@ -243,6 +243,7 @@ export async function assertResolvedPublicUrl(value: string): Promise<void> {}interface SafeFetchInit {+ method?: stringheaders?: Record<string, string>signal?: AbortSignal}