Official Crawl4AI Cloud: a hosted web scraping, extraction, and search API for AI agents. Free API key, 1,000 credits a month, no card. Fully open source: this repo is the Apache-2.0 crawler that powers it, and you can self-host it.
Python aiai-agentscrawl4ai-cloudcrawlerdata-extractionllmmarkdownmcpopen-sourceplaywrightpythonragscraperweb-crawlerweb-crawlingweb-scraping
docs: fix 33 broken links across the docs site
unclecode5fae7a4092b9
diff --git a/docs/md_v2/advanced/adaptive-strategies.md b/docs/md_v2/advanced/adaptive-strategies.mdindex 11c5585a..d91465d1 100644--- a/docs/md_v2/advanced/adaptive-strategies.md+++ b/docs/md_v2/advanced/adaptive-strategies.md@@ -395,6 +395,5 @@ print(f"Query term coverage: {coverage_ratio:.0%}")## Next Steps-- Explore [Custom Strategy Implementation](../tutorials/custom-adaptive-strategies.md)-- Learn about [Knowledge Base Management](../tutorials/knowledge-base-management.md)-- See [Performance Benchmarks](../benchmarks/adaptive-performance.md)\ No newline at end of file+- Start with the [Adaptive Crawling guide](../core/adaptive-crawling.md)+- Read [Adaptive Crawling: the idea behind it](../blog/articles/adaptive-crawling-revolution.md)\ No newline at end of filediff --git a/docs/md_v2/advanced/virtual-scroll.md b/docs/md_v2/advanced/virtual-scroll.mdindex 271a9564..55cc85df 100644--- a/docs/md_v2/advanced/virtual-scroll.md+++ b/docs/md_v2/advanced/virtual-scroll.md@@ -293,7 +293,7 @@ If the container isn't found, crawling continues normally without virtual scroll## Complete Example-See our [comprehensive example](/docs/examples/virtual_scroll_example.py) that demonstrates:+See our [comprehensive example](https://github.com/unclecode/crawl4ai/blob/main/docs/examples/virtual_scroll_example.py) that demonstrates:- Twitter-like feeds- Instagram grids- Traditional infinite scrolldiff --git a/docs/md_v2/api/c4a-script-reference.md b/docs/md_v2/api/c4a-script-reference.mdindex 57de39b2..c0cff83d 100644--- a/docs/md_v2/api/c4a-script-reference.md+++ b/docs/md_v2/api/c4a-script-reference.md@@ -989,4 +989,4 @@ async with AsyncWebCrawler() as crawler:print(result.markdown)```-This reference covers all available C4A-Script commands and patterns. For interactive learning, try the [tutorial](../examples/c4a_script/tutorial/) or [live demo](https://docs.crawl4ai.com/c4a-script/demo).\ No newline at end of file+This reference covers all available C4A-Script commands and patterns. For interactive learning, try the [tutorial](https://github.com/unclecode/crawl4ai/tree/main/docs/examples/c4a_script/tutorial) or [live demo](https://docs.crawl4ai.com/apps/c4a-script/).\ No newline at end of filediff --git a/docs/md_v2/api/parameters.md b/docs/md_v2/api/parameters.mdindex f9f759a5..b9f139be 100644--- a/docs/md_v2/api/parameters.md+++ b/docs/md_v2/api/parameters.md@@ -271,7 +271,7 @@ config = CrawlerRunConfig(- Use `virtual_scroll_config` when content is **replaced** during scroll (Twitter, Instagram)- Use `scan_full_page` when content is **appended** during scroll (traditional infinite scroll)-See [Virtual Scroll documentation](../../advanced/virtual-scroll.md) for detailed examples.+See [Virtual Scroll documentation](../advanced/virtual-scroll.md) for detailed examples.---diff --git a/docs/md_v2/apps/c4a-script/README.md b/docs/md_v2/apps/c4a-script/README.mdindex 2d6940bb..f2fa26c2 100644--- a/docs/md_v2/apps/c4a-script/README.md+++ b/docs/md_v2/apps/c4a-script/README.md@@ -31,7 +31,7 @@ A comprehensive web-based tutorial for learning and experimenting with C4A-Scriphttp://localhost:8000```-**🌐 Try Online**: [Live Demo](https://docs.crawl4ai.com/c4a-script/demo)+**🌐 Try Online**: [Live Demo](https://docs.crawl4ai.com/apps/c4a-script/)### 2. Try Your First Script@@ -365,7 +365,7 @@ Enable detailed logging by setting `DEBUG = True` in `assets/app.js`- **[C4A-Script Documentation](../../md_v2/core/c4a-script.md)** - Complete language guide- **[API Reference](../../md_v2/api/c4a-script-reference.md)** - Detailed command documentation-- **[Live Demo](https://docs.crawl4ai.com/c4a-script/demo)** - Try without installation+- **[Live Demo](https://docs.crawl4ai.com/apps/c4a-script/)** - Try without installation- **[Example Scripts](../)** - More automation examples## 🤝 Contributingdiff --git a/docs/md_v2/apps/crawl4ai-assistant/index.html b/docs/md_v2/apps/crawl4ai-assistant/index.htmlindex abe082dc..00b20975 100644--- a/docs/md_v2/apps/crawl4ai-assistant/index.html+++ b/docs/md_v2/apps/crawl4ai-assistant/index.html@@ -26,16 +26,6 @@</div><div class="content">- <!-- Video Section -->- <section class="video-section">- <div class="video-wrapper">- <video autoplay loop muted playsinline class="demo-video">- <source src="demo.mp4" type="video/mp4">- Your browser does not support the video tag.- </video>- </div>- </section>-<!-- Cloud Announcement Banner --><section class="cloud-banner-section"><div class="cloud-banner">diff --git a/docs/md_v2/blog/articles/virtual-scroll-revolution.md b/docs/md_v2/blog/articles/virtual-scroll-revolution.mdindex e2736ed6..b3964ba6 100644--- a/docs/md_v2/blog/articles/virtual-scroll-revolution.md+++ b/docs/md_v2/blog/articles/virtual-scroll-revolution.md@@ -348,7 +348,7 @@ Welcome to the future of web scraping. Welcome to a world where virtual scroll i## Learn More- 📖 [Virtual Scroll Documentation](https://docs.crawl4ai.com/advanced/virtual-scroll) - Complete API reference and configuration options-- 💻 [Interactive Examples](https://docs.crawl4ai.com/examples/virtual_scroll_example.py) - Try it yourself with our test server+- 💻 [Interactive Examples](https://github.com/unclecode/crawl4ai/blob/main/docs/examples/virtual_scroll_example.py) - Try it yourself with our test server- 🚀 [Get Started with Crawl4AI](https://docs.crawl4ai.com/core/quickstart) - Full installation and setup guide- 🤝 [Join our Community](https://github.com/unclecode/crawl4ai) - Share your experiences and get helpdiff --git a/docs/md_v2/blog/index.md b/docs/md_v2/blog/index.mdindex 30a9dbe1..501e409d 100644--- a/docs/md_v2/blog/index.md+++ b/docs/md_v2/blog/index.md@@ -20,7 +20,7 @@ Ever wondered why your AI coding assistant struggles with your library despite c## Latest Release-### [Crawl4AI v0.9.2 – Maintenance Bug Fixes](../blog/release-v0.9.2.md)+### [Crawl4AI v0.9.2 – Maintenance Bug Fixes](releases/v0.9.2.md)*July 2026*Crawl4AI v0.9.2 is a maintenance patch release with bug fixes across the dispatcher, Docker, and GPU builds.@@ -30,11 +30,11 @@ Key highlights:- **🐳 Docker Fixes**: Playground "Advanced Config" 400, Monitor WebSocket auth 500, and Playwright headless-shell packaging- **🖥️ GPU Builds**: `ENABLE_GPU=true` Docker builds fixed (CUDA toolkit `non-free` apt source)-[Read full release notes →](../blog/release-v0.9.2.md)+[Read full release notes →](releases/v0.9.2.md)## Recent Releases-### [Crawl4AI v0.9.1 – Bug Fixes & PruningContentFilter Whitelist](../blog/release-v0.9.1.md)+### [Crawl4AI v0.9.1 – Bug Fixes & PruningContentFilter Whitelist](releases/v0.9.1.md)*July 2026*Crawl4AI v0.9.1 is a patch release with 12 bug fixes and a new `preserve_classes`/`preserve_tags` whitelist for PruningContentFilter.@@ -44,16 +44,16 @@ Key highlights:- **🐛 12 Bug Fixes**: Docker auth gate UI, Windows browser crash, HTTP timeout unit mismatch, and more- **📦 Dependency**: lxml ceiling widened to allow 6.x-[Read full release notes →](../blog/release-v0.9.1.md)+[Read full release notes →](releases/v0.9.1.md)-### [Crawl4AI v0.8.5 – Anti-Bot Detection, Shadow DOM & 60+ Bug Fixes](../blog/release-v0.8.5.md)+### [Crawl4AI v0.8.5 – Anti-Bot Detection, Shadow DOM & 60+ Bug Fixes](releases/v0.8.5.md)*March 2026*Crawl4AI v0.8.5 is the biggest release since v0.8.0, bringing automatic anti-bot detection with proxy escalation, Shadow DOM flattening, deep crawl cancellation, and over 60 bug fixes.-[Read full release notes →](../blog/release-v0.8.5.md)+[Read full release notes →](releases/v0.8.5.md)-### [Crawl4AI v0.8.0 – Crash Recovery & Prefetch Mode](../blog/release-v0.8.0.md)+### [Crawl4AI v0.8.0 – Crash Recovery & Prefetch Mode](releases/v0.8.0.md)*January 2026*Crawl4AI v0.8.0 introduces crash recovery for deep crawls, a new prefetch mode for fast URL discovery, and critical security fixes for Docker deployments.@@ -63,9 +63,9 @@ Key highlights:- **⚡ Prefetch Mode**: `prefetch=True` for 5-10x faster URL discovery, perfect for two-phase crawling patterns- **🔒 Security Fixes**: Hooks disabled by default, `file://` URLs blocked on Docker API, `__import__` removed from sandbox-[Read full release notes →](../blog/release-v0.8.0.md)+[Read full release notes →](releases/v0.8.0.md)-### [Crawl4AI v0.7.8 – Stability & Bug Fix Release](../blog/release-v0.7.8.md)+### [Crawl4AI v0.7.8 – Stability & Bug Fix Release](releases/v0.7.8.md)*December 2025*Crawl4AI v0.7.8 is a focused stability release addressing 11 bugs reported by the community. Fixes for Docker deployments, LLM extraction, URL handling, and dependency compatibility.@@ -75,7 +75,7 @@ Key highlights:- **🤖 LLM Improvements**: Configurable rate limiter backoff, HTML input format support- **📦 Dependencies**: Replaced deprecated PyPDF2 with pypdf, Pydantic v2 ConfigDict compatibility-[Read full release notes →](../blog/release-v0.7.8.md)+[Read full release notes →](releases/v0.7.8.md)---@@ -83,13 +83,13 @@ Key highlights:| Version | Date | Highlights ||---------|------|------------|-| [v0.7.7](../blog/release-v0.7.7.md) | November 2025 | Self-hosting platform, real-time monitoring, smart browser pool |-| [v0.7.6](../blog/release-v0.7.6.md) | October 2025 | Webhook infrastructure, reliable delivery, custom auth |-| [v0.7.5](../blog/release-v0.7.5.md) | September 2025 | Docker Hooks System, enhanced LLM integration, HTTPS preservation |-| [v0.7.4](../blog/release-v0.7.4.md) | August 2025 | LLM-powered table extraction, performance improvements |-| [v0.7.3](../blog/release-v0.7.3.md) | July 2025 | Undetected browser, multi-URL config, memory monitoring |-| [v0.7.1](../blog/release-v0.7.1.md) | June 2025 | Bug fixes and stability improvements |-| [v0.7.0](../blog/release-v0.7.0.md) | May 2025 | Adaptive crawling, virtual scroll, link analysis |+| [v0.7.7](releases/v0.7.7.md) | November 2025 | Self-hosting platform, real-time monitoring, smart browser pool |+| [v0.7.6](releases/0.7.6.md) | October 2025 | Webhook infrastructure, reliable delivery, custom auth |+| [v0.7.5](releases/v0.7.5.md) | September 2025 | Docker Hooks System, enhanced LLM integration, HTTPS preservation |+| [v0.7.4](https://github.com/unclecode/crawl4ai/releases/tag/v0.7.4) | August 2025 | LLM-powered table extraction, performance improvements |+| [v0.7.3](releases/0.7.3.md) | July 2025 | Undetected browser, multi-URL config, memory monitoring |+| [v0.7.1](releases/0.7.1.md) | June 2025 | Bug fixes and stability improvements |+| [v0.7.0](releases/0.7.0.md) | May 2025 | Adaptive crawling, virtual scroll, link analysis |## Project Historydiff --git a/docs/md_v2/blog/releases/0.5.0.md b/docs/md_v2/blog/releases/0.5.0.mdindex e55bde20..c8ad4328 100644--- a/docs/md_v2/blog/releases/0.5.0.md+++ b/docs/md_v2/blog/releases/0.5.0.md@@ -224,7 +224,7 @@ crwl https://example.com -q "What is the main topic?"crwl --example```-See the [CLI documentation](../docs/md_v2/core/cli.md) for more details.+See the [CLI documentation](../../core/cli.md) for more details.### 6. LXML Scraping Modediff --git a/docs/md_v2/branding/index.md b/docs/md_v2/branding/index.mdindex 5741344d..47d808bb 100644--- a/docs/md_v2/branding/index.md+++ b/docs/md_v2/branding/index.md@@ -1294,9 +1294,8 @@ Use these CSS variables for consistency across all styles:### Download Assets-- [Dank Mono Font Files](/docs/md_v2/assets/) (Regular, Bold, Italic)-- [Brand CSS Template](/docs/md_v2/branding/assets/brand-examples.css)-- [Component Library](/docs/md_v2/apps/)+- [Dank Mono Font Files](https://github.com/unclecode/crawl4ai/tree/main/docs/md_v2/assets) (Regular, Bold, Italic)+- [Component Library](../apps/index.md)### Reference Filesdiff --git a/docs/md_v2/core/c4a-script.md b/docs/md_v2/core/c4a-script.mdindex 1af3da4e..57959188 100644--- a/docs/md_v2/core/c4a-script.md+++ b/docs/md_v2/core/c4a-script.md@@ -275,7 +275,7 @@ C4A-Script includes a powerful visual programming interface built on Google Bloc- **Error Prevention**: Visual connections prevent syntax errors- **Comment Support**: Add visual comment blocks for documentation-Try the visual editor in our [live demo](https://docs.crawl4ai.com/c4a-script/demo) or [local tutorial](/examples/c4a_script/tutorial/).+Try the visual editor in our [live demo](https://docs.crawl4ai.com/apps/c4a-script/) or [local tutorial](https://github.com/unclecode/crawl4ai/tree/main/docs/examples/c4a_script/tutorial).## Advanced Features@@ -375,18 +375,18 @@ SET `#email` $test_email## Getting Help-- **📖 [Complete Examples](/examples/c4a_script/)** - Real-world automation scripts-- **🎮 [Interactive Tutorial](/examples/c4a_script/tutorial/)** - Hands-on learning environment+- **📖 [Complete Examples](https://github.com/unclecode/crawl4ai/tree/main/docs/examples/c4a_script)** - Real-world automation scripts+- **🎮 [Interactive Tutorial](https://github.com/unclecode/crawl4ai/tree/main/docs/examples/c4a_script/tutorial)** - Hands-on learning environment- **📋 [API Reference](/api/c4a-script-reference/)** - Detailed command documentation-- **🌐 [Live Demo](https://docs.crawl4ai.com/c4a-script/demo)** - Try it in your browser+- **🌐 [Live Demo](https://docs.crawl4ai.com/apps/c4a-script/)** - Try it in your browser## What's Next?Ready to dive deeper? Check out:1. **[API Reference](/api/c4a-script-reference/)** - Complete command documentation-2. **[Tutorial Examples](/examples/c4a_script/)** - Copy-paste ready scripts-3. **[Local Tutorial Setup](/examples/c4a_script/tutorial/)** - Run the full development environment+2. **[Tutorial Examples](https://github.com/unclecode/crawl4ai/tree/main/docs/examples/c4a_script)** - Copy-paste ready scripts+3. **[Local Tutorial Setup](https://github.com/unclecode/crawl4ai/tree/main/docs/examples/c4a_script/tutorial)** - Run the full development environmentC4A-Script makes web automation accessible to everyone. Whether you're a developer automating tests, a designer creating interactive demos, or a business user streamlining repetitive tasks, C4A-Script has the tools you need.diff --git a/docs/md_v2/core/examples.md b/docs/md_v2/core/examples.mdindex d5d58e02..acc49cdc 100644--- a/docs/md_v2/core/examples.md+++ b/docs/md_v2/core/examples.md@@ -15,7 +15,7 @@ This page provides a comprehensive list of example scripts that demonstrate vari| Example | Description | Link ||----------|--------------|------|-| **NSTProxy** | [NSTProxy](https://www.nstproxy.com/?utm_source=crawl4ai) Seamlessly integrates with crawl4ai — no setup required. Access high-performance residential, datacenter, ISP, and IPv6 proxies with smart rotation and anti-blocking technology. Starts from $0.1/GB. Use code crawl4ai for 10% off. | [View Code](https://github.com/unclecode/crawl4ai/tree/main/docs/examples/proxy) |+| **NSTProxy** | [NSTProxy](https://www.nstproxy.com/?utm_source=crawl4ai) Seamlessly integrates with crawl4ai — no setup required. Access high-performance residential, datacenter, ISP, and IPv6 proxies with smart rotation and anti-blocking technology. Starts from $0.1/GB. Use code crawl4ai for 10% off. | [View Code](https://github.com/unclecode/crawl4ai/tree/main/docs/examples/nst_proxy) |## Browser & Crawling Featuresdiff --git a/docs/md_v2/core/llmtxt.md b/docs/md_v2/core/llmtxt.mdindex b2c67f3a..6181315c 100644--- a/docs/md_v2/core/llmtxt.md+++ b/docs/md_v2/core/llmtxt.md@@ -1,5 +1,5 @@I<div class="llmtxt-container">-<iframe id="llmtxt-frame" src="../../llmtxt/index.html" width="100%" style="border:none; display: block;" title="Crawl4AI LLM Context Builder"></iframe>+<iframe id="llmtxt-frame" src="../../apps/llmtxt/index.html" width="100%" style="border:none; display: block;" title="Crawl4AI LLM Context Builder"></iframe></div><script>diff --git a/docs/md_v2/overrides/partials/favicon.html b/docs/md_v2/overrides/partials/favicon.htmlnew file mode 100644index 00000000..224df640--- /dev/null+++ b/docs/md_v2/overrides/partials/favicon.html@@ -0,0 +1,3 @@+<!-- Overrides the terminal theme's favicon partial: it links six icon files, and only these two exist in docs/md_v2/img. -->+<link rel="shortcut icon" href="{{ 'img/favicon.ico' | url }}" />+<link rel="icon" type="image/png" sizes="32x32" href="{{ 'img/favicon-32x32.png' | url }}" />diff --git a/docs/md_v2/terms.md b/docs/md_v2/terms.mdindex 9f84fd86..5d5a572c 100644--- a/docs/md_v2/terms.md+++ b/docs/md_v2/terms.md@@ -18,7 +18,7 @@ You must be at least 16 years old, or the age of digital consent in your jurisdi## 2. Plans, credits, and billing-The Service is offered as a freemium model with paid plans. Each plan defines a daily credit limit, rate limits, storage allowance, and feature set. Current plans are listed at [crawl4ai.com/pricing](https://crawl4ai.com/pricing).+The Service is offered as a freemium model with paid plans. Each plan defines a daily credit limit, rate limits, storage allowance, and feature set. Current plans are listed at [crawl4ai.com/#pricing](https://crawl4ai.com/#pricing).**Billing.** Paid plans are billed in advance on a recurring basis (monthly or annual). All fees are exclusive of taxes, which you are responsible for. Payment is processed through Stripe.