Official Crawl4AI Cloud: a hosted web scraping, extraction, and search API for AI agents. Free API key, 1,000 credits a month, no card. Fully open source: this repo is the Apache-2.0 crawler that powers it, and you can self-host it.
Python aiai-agentscrawl4ai-cloudcrawlerdata-extractionllmmarkdownmcpopen-sourceplaywrightpythonragscraperweb-crawlerweb-crawlingweb-scraping
release: Crawl4AI v0.9.2
ntohidi055e2ecdc702
diff --git a/CONTRIBUTORS.md b/CONTRIBUTORS.mdindex 1eadb852..c2926ec8 100644--- a/CONTRIBUTORS.md+++ b/CONTRIBUTORS.md@@ -58,9 +58,11 @@ We would like to thank the following people for their contributions to Crawl4AI:- [microHoffman](https://github.com/microHoffman) - feat: add --json-ensure-ascii CLI flag for Unicode handling [#1668](https://github.com/unclecode/crawl4ai/pull/1668)- [fstark96](https://github.com/fstark96) - fix: skip channel='chromium' on Windows to prevent Chrome exit code 0 [#2051](https://github.com/unclecode/crawl4ai/pull/2051)- [TobiasWallura-xitaso](https://github.com/TobiasWallura-xitaso) - fix: use writable directories for supervisord pid and redis data in Docker [#2047](https://github.com/unclecode/crawl4ai/pull/2047)-- [harshmathurx](https://github.com/harshmathurx) - fix: authenticate rate limit Redis storage in Docker [#2043](https://github.com/unclecode/crawl4ai/pull/2043)+- [harshmathurx](https://github.com/harshmathurx) - fix: authenticate rate limit Redis storage in Docker [#2043](https://github.com/unclecode/crawl4ai/pull/2043), fix: add non-free apt source for nvidia-cuda-toolkit GPU builds [#2057](https://github.com/unclecode/crawl4ai/pull/2057)- [RajanChavada](https://github.com/RajanChavada) - chore: remove dead normalize_url definitions and accidental adaptive_crawler copy [#2042](https://github.com/unclecode/crawl4ai/pull/2042)- [bibi-creator](https://github.com/bibi-creator) - docs: split sponsors into Strategic Partners and Enterprise Sponsors [#2056](https://github.com/unclecode/crawl4ai/pull/2056), fix: update sponsor logos [#2052](https://github.com/unclecode/crawl4ai/pull/2052)+- [reallav0](https://github.com/reallav0) - fix: clean up MemoryAdaptiveDispatcher tasks on stream close [#2072](https://github.com/unclecode/crawl4ai/pull/2072), fix: include Playwright headless shell in Docker image [#2067](https://github.com/unclecode/crawl4ai/pull/2067)+- [Pitchfork-and-Torch](https://github.com/Pitchfork-and-Torch) - fix: Docker playground config type and monitor WebSocket auth [#2063](https://github.com/unclecode/crawl4ai/pull/2063)#### Feb-Alpha-1- [sufianuddin](https://github.com/sufianuddin) - fix: [Documentation for JsonCssExtractionStrategy](https://github.com/unclecode/crawl4ai/issues/651)diff --git a/Dockerfile b/Dockerfileindex 92f1404f..3b39e307 100644--- a/Dockerfile+++ b/Dockerfile@@ -1,7 +1,7 @@FROM python:3.12-slim-bookworm AS build# C4ai version-ARG C4AI_VER=0.9.1+ARG C4AI_VER=0.9.2ENV C4AI_VERSION=$C4AI_VERLABEL c4ai.version=$C4AI_VERdiff --git a/README.md b/README.mdindex 20367f5f..705f42ac 100644--- a/README.md+++ b/README.md@@ -37,9 +37,9 @@ Limited slots._Crawl4AI turns the web into clean, LLM ready Markdown for RAG, agents, and data pipelines. Fast, controllable, battle tested by a 50k+ star community.-[✨ Check out latest update v0.9.1](#-recent-updates)+[✨ Check out latest update v0.9.2](#-recent-updates)-✨ **New in v0.9.1**: Patch release with 12 bug fixes across Docker, browser, and core. Adds `preserve_classes`/`preserve_tags` whitelist for PruningContentFilter, fixes Windows browser crash, Docker auth gate UI, HTTP timeout unit mismatch, and more. [Release notes →](https://github.com/unclecode/crawl4ai/blob/main/docs/blog/release-v0.9.1.md)+✨ **New in v0.9.2**: Maintenance patch release. Fixes a `MemoryAdaptiveDispatcher` task/page leak when a streaming crawl is closed, Docker Playground "Advanced Config" and Monitor WebSocket auth, Playwright headless-shell packaging, and GPU (`ENABLE_GPU=true`) Docker builds. [Release notes →](https://github.com/unclecode/crawl4ai/blob/main/docs/blog/release-v0.9.2.md)✨ Recent v0.9.0: Major secure-by-default release of the Docker API server. Auth is on by default, the server binds loopback unless given a token, and the request body is now an untrusted trust boundary. [Release notes →](https://github.com/unclecode/crawl4ai/blob/main/docs/blog/release-v0.9.0.md)@@ -565,6 +565,21 @@ async def test_news_crawl():## ✨ Recent Updates<details open>+<summary><strong>Version 0.9.2 Release Highlights - Maintenance Bug Fixes</strong></summary>++A maintenance patch release with bug fixes across the dispatcher, Docker, and GPU builds. `MemoryAdaptiveDispatcher` no longer leaks crawl tasks and browser pages when a streaming crawl is closed. Docker fixes cover the Playground "Advanced Config" 400, the Monitor WebSocket 500 under JWT auth, and Playwright headless-shell packaging. `ENABLE_GPU=true` Docker builds no longer fail on the CUDA toolkit.++No new features, no breaking changes.++```bash+pip install -U crawl4ai+```++[Full v0.9.2 Release Notes →](https://github.com/unclecode/crawl4ai/blob/main/docs/blog/release-v0.9.2.md)++</details>++<details><summary><strong>Version 0.9.1 Release Highlights - Bug Fixes & PruningContentFilter Whitelist</strong></summary>A patch release with 12 bug fixes and one new feature. The new `preserve_classes` / `preserve_tags` parameters for `PruningContentFilter` let you whitelist CSS classes or HTML tags that should never be pruned — useful for protecting short metadata elements like author names and timestamps.diff --git a/crawl4ai/__version__.py b/crawl4ai/__version__.pyindex 6dac9465..505ca152 100644--- a/crawl4ai/__version__.py+++ b/crawl4ai/__version__.py@@ -1,7 +1,7 @@# crawl4ai/__version__.py# This is the version that will be used for stable releases-__version__ = "0.9.1"+__version__ = "0.9.2"# For nightly builds, this gets set during build process__nightly_version__ = Nonediff --git a/deploy/docker/README.md b/deploy/docker/README.mdindex a711c720..7ce0dcd0 100644--- a/deploy/docker/README.md+++ b/deploy/docker/README.md@@ -59,10 +59,10 @@ Pull and run images directly from Docker Hub without building locally.#### 1. Pull the Image-Our latest stable release is `0.8.6`. Images are built with multi-arch manifests, so Docker automatically pulls the correct version for your system.+Our latest stable release is `0.9.2`. Images are built with multi-arch manifests, so Docker automatically pulls the correct version for your system.```bash-# Pull the latest stable version (0.8.6)+# Pull the latest stable version (0.9.2)docker pull unclecode/crawl4ai:0.8.6# Or use the latest tagdiff --git a/docs/blog/release-v0.9.2.md b/docs/blog/release-v0.9.2.mdnew file mode 100644index 00000000..95a47aa5--- /dev/null+++ b/docs/blog/release-v0.9.2.md@@ -0,0 +1,67 @@+# Crawl4AI v0.9.2: Maintenance Bug Fixes++*July 2026 - 2 min read*++---++I'm releasing Crawl4AI v0.9.2, a small maintenance patch that cleans up a dispatcher resource leak and fixes a handful of Docker and GPU build issues. No new features, no breaking changes.++If you're on v0.9.1, upgrade freely.++## What's new at a glance++- **Dispatcher cleanup**: `MemoryAdaptiveDispatcher` no longer leaves crawl tasks and browser pages running after a streaming crawl is closed+- **Docker Playground**: "Advanced Config" no longer returns a 400+- **Docker Monitor**: the dashboard WebSocket no longer returns a 500 under JWT auth+- **Docker image**: Playwright headless shell is now included+- **GPU builds**: `ENABLE_GPU=true` Docker builds no longer fail on the CUDA toolkit++## Bug fixes++### Dispatcher (1 fix)++- **Streaming cleanup leak**: When the async generator from `MemoryAdaptiveDispatcher.run_urls_stream()` was closed or cancelled mid-stream, its per-URL crawl tasks kept running in the background. A later `arun_many()` on the same crawler could then overlap the old tasks while they still held the same browser, contexts, and pages — leaking pages and emitting stray Playwright `TargetClosedError`s. Closing the stream now cancels and awaits every in-flight task and drains any queued-but-unstarted URLs before returning. (#2071, thanks @reallav0)++### Docker (3 fixes)++- **Playground "Advanced Config" 400**: `/config/dump` requires a `type`, but the playground's `pyConfigToJson` only sent `code`. The request now includes the config type, aligns the stream fallback with the dump shape, and teaches `shouldUseStream` both nestings. (#2059, thanks @Pitchfork-and-Torch)+- **Monitor WebSocket 500**: the monitor dashboard's `/monitor/ws` endpoint 500'd because a router-level `token_dep` is HTTP-request-only and can't inject into WebSocket scopes. Auth is now enforced solely by `AuthGateMiddleware`; admin routes keep `require_admin`. (#2060, thanks @Pitchfork-and-Torch)+- **Headless shell packaging**: the Playwright headless shell is now installed in the Docker image. (#2067, thanks @reallav0)++### GPU / Build (1 fix)++- **CUDA toolkit on Bookworm**: `ENABLE_GPU=true` builds failed with "Package has no installation candidate" because `nvidia-cuda-toolkit` lives in Debian Bookworm's `non-free` component, which the `python:3.12-slim-bookworm` base image omits. The Dockerfile now adds the `non-free` (and contrib) apt sources before the GPU install block. (#2020, thanks @harshmathurx)++## Tests++- Regression test added for the dispatcher stream-close cleanup, asserting in-flight tasks are cancelled, the queue is drained, and session count returns to zero.+- Updated the monitor auth test for the WebSocket-safe router config, and corrected a regex group name and the Playwright cache-copy target in the Docker tests.++## Breaking changes++None.++## Upgrade++```bash+pip install -U crawl4ai+crawl4ai-doctor # verify installation+```++Docker users: pull the latest image once the Docker release workflow finishes.++## Verification++```bash+python docs/releases_review/demo_v0.9.2.py+```++## Acknowledgments++Thanks to the community contributors who made this release possible: @reallav0 (#2071, #2067), @Pitchfork-and-Torch (#2059, #2060), @harshmathurx (#2020).++## Support & Resources++- [Documentation](https://docs.crawl4ai.com)+- [GitHub Issues](https://github.com/unclecode/crawl4ai/issues)+- [Discord Community](https://discord.gg/crawl4ai)diff --git a/docs/md_v2/blog/index.md b/docs/md_v2/blog/index.mdindex 4d3dd1b5..30a9dbe1 100644--- a/docs/md_v2/blog/index.md+++ b/docs/md_v2/blog/index.md@@ -20,6 +20,20 @@ Ever wondered why your AI coding assistant struggles with your library despite c## Latest Release+### [Crawl4AI v0.9.2 – Maintenance Bug Fixes](../blog/release-v0.9.2.md)+*July 2026*++Crawl4AI v0.9.2 is a maintenance patch release with bug fixes across the dispatcher, Docker, and GPU builds.++Key highlights:+- **🧹 Dispatcher Cleanup**: `MemoryAdaptiveDispatcher` no longer leaks crawl tasks and browser pages when a streaming crawl is closed+- **🐳 Docker Fixes**: Playground "Advanced Config" 400, Monitor WebSocket auth 500, and Playwright headless-shell packaging+- **🖥️ GPU Builds**: `ENABLE_GPU=true` Docker builds fixed (CUDA toolkit `non-free` apt source)++[Read full release notes →](../blog/release-v0.9.2.md)++## Recent Releases+### [Crawl4AI v0.9.1 – Bug Fixes & PruningContentFilter Whitelist](../blog/release-v0.9.1.md)*July 2026*@@ -32,8 +46,6 @@ Key highlights:[Read full release notes →](../blog/release-v0.9.1.md)-## Recent Releases-### [Crawl4AI v0.8.5 – Anti-Bot Detection, Shadow DOM & 60+ Bug Fixes](../blog/release-v0.8.5.md)*March 2026*diff --git a/docs/md_v2/blog/releases/v0.9.2.md b/docs/md_v2/blog/releases/v0.9.2.mdnew file mode 100644index 00000000..95a47aa5--- /dev/null+++ b/docs/md_v2/blog/releases/v0.9.2.md@@ -0,0 +1,67 @@+# Crawl4AI v0.9.2: Maintenance Bug Fixes++*July 2026 - 2 min read*++---++I'm releasing Crawl4AI v0.9.2, a small maintenance patch that cleans up a dispatcher resource leak and fixes a handful of Docker and GPU build issues. No new features, no breaking changes.++If you're on v0.9.1, upgrade freely.++## What's new at a glance++- **Dispatcher cleanup**: `MemoryAdaptiveDispatcher` no longer leaves crawl tasks and browser pages running after a streaming crawl is closed+- **Docker Playground**: "Advanced Config" no longer returns a 400+- **Docker Monitor**: the dashboard WebSocket no longer returns a 500 under JWT auth+- **Docker image**: Playwright headless shell is now included+- **GPU builds**: `ENABLE_GPU=true` Docker builds no longer fail on the CUDA toolkit++## Bug fixes++### Dispatcher (1 fix)++- **Streaming cleanup leak**: When the async generator from `MemoryAdaptiveDispatcher.run_urls_stream()` was closed or cancelled mid-stream, its per-URL crawl tasks kept running in the background. A later `arun_many()` on the same crawler could then overlap the old tasks while they still held the same browser, contexts, and pages — leaking pages and emitting stray Playwright `TargetClosedError`s. Closing the stream now cancels and awaits every in-flight task and drains any queued-but-unstarted URLs before returning. (#2071, thanks @reallav0)++### Docker (3 fixes)++- **Playground "Advanced Config" 400**: `/config/dump` requires a `type`, but the playground's `pyConfigToJson` only sent `code`. The request now includes the config type, aligns the stream fallback with the dump shape, and teaches `shouldUseStream` both nestings. (#2059, thanks @Pitchfork-and-Torch)+- **Monitor WebSocket 500**: the monitor dashboard's `/monitor/ws` endpoint 500'd because a router-level `token_dep` is HTTP-request-only and can't inject into WebSocket scopes. Auth is now enforced solely by `AuthGateMiddleware`; admin routes keep `require_admin`. (#2060, thanks @Pitchfork-and-Torch)+- **Headless shell packaging**: the Playwright headless shell is now installed in the Docker image. (#2067, thanks @reallav0)++### GPU / Build (1 fix)++- **CUDA toolkit on Bookworm**: `ENABLE_GPU=true` builds failed with "Package has no installation candidate" because `nvidia-cuda-toolkit` lives in Debian Bookworm's `non-free` component, which the `python:3.12-slim-bookworm` base image omits. The Dockerfile now adds the `non-free` (and contrib) apt sources before the GPU install block. (#2020, thanks @harshmathurx)++## Tests++- Regression test added for the dispatcher stream-close cleanup, asserting in-flight tasks are cancelled, the queue is drained, and session count returns to zero.+- Updated the monitor auth test for the WebSocket-safe router config, and corrected a regex group name and the Playwright cache-copy target in the Docker tests.++## Breaking changes++None.++## Upgrade++```bash+pip install -U crawl4ai+crawl4ai-doctor # verify installation+```++Docker users: pull the latest image once the Docker release workflow finishes.++## Verification++```bash+python docs/releases_review/demo_v0.9.2.py+```++## Acknowledgments++Thanks to the community contributors who made this release possible: @reallav0 (#2071, #2067), @Pitchfork-and-Torch (#2059, #2060), @harshmathurx (#2020).++## Support & Resources++- [Documentation](https://docs.crawl4ai.com)+- [GitHub Issues](https://github.com/unclecode/crawl4ai/issues)+- [Discord Community](https://discord.gg/crawl4ai)diff --git a/docs/releases_review/demo_v0.9.2.py b/docs/releases_review/demo_v0.9.2.pynew file mode 100644index 00000000..aa028d91--- /dev/null+++ b/docs/releases_review/demo_v0.9.2.py@@ -0,0 +1,194 @@+#!/usr/bin/env python3+"""+Crawl4AI v0.9.2 Release Demo - Fix Verification Tests+======================================================++This demo ACTUALLY RUNS and VERIFIES the key changes in v0.9.2.+v0.9.2 is a maintenance patch release with no new features, so each test+exercises a bug-fix path end-to-end.++Fixes Verified (Python-testable):+1. MemoryAdaptiveDispatcher cleans up tasks when a stream is closed (#2071)+2. Basic real-crawl smoke test (the release build crawls end-to-end)++Not covered here (validated by CI image builds, not this script):+- Docker Playground "Advanced Config" 400 (#2059)+- Docker Monitor WebSocket auth 500 (#2060)+- Playwright headless-shell packaging (#2067)+- GPU ENABLE_GPU=true build / CUDA toolkit non-free apt source (#2020)++Usage:+ python docs/releases_review/demo_v0.9.2.py+"""++import asyncio+import sys+from dataclasses import dataclass+from types import SimpleNamespace+++@dataclass+class TestResult:+ name: str+ feature: str+ passed: bool+ message: str+ skipped: bool = False+++results: list[TestResult] = []+++def print_header(title: str):+ print(f"\n{'=' * 70}")+ print(f"{title}")+ print(f"{'=' * 70}")+++def print_test(name: str, feature: str):+ print(f"\n[TEST] {name} ({feature})")+ print("-" * 50)+++def record_result(name: str, feature: str, passed: bool, message: str, skipped: bool = False):+ results.append(TestResult(name, feature, passed, message, skipped))+ status = "SKIP" if skipped else ("PASS" if passed else "FAIL")+ print(f" [{status}] {message}")+++# ── Test 1: Dispatcher stream-close cleanup (#2071) ───────────────────++async def test_dispatcher_stream_cleanup():+ """Closing a streaming MemoryAdaptiveDispatcher must cancel in-flight+ per-URL tasks, drain queued URLs, and restore concurrent_sessions."""+ print_test("Dispatcher stream-close cleanup", "MemoryAdaptiveDispatcher #2071")++ from crawl4ai import CrawlerRunConfig, MemoryAdaptiveDispatcher++ class BlockingCrawler:+ """Return one result immediately; block all other URLs forever."""++ async def arun(self, url, config=None, session_id=None):+ if url == "fast":+ return SimpleNamespace(success=True, status_code=200, error_message="")+ await asyncio.Event().wait()++ class TrackingDispatcher(MemoryAdaptiveDispatcher):+ def __init__(self):+ super().__init__(max_session_permit=3)+ self.tasks = {}++ async def crawl_url(self, url, config, task_id, retry_count=0):+ self.tasks[url] = asyncio.current_task()+ return await super().crawl_url(url, config, task_id, retry_count)++ dispatcher = TrackingDispatcher()+ stream = dispatcher.run_urls_stream(+ ["fast", "blocked-1", "blocked-2", "queued"],+ BlockingCrawler(),+ CrawlerRunConfig(),+ )++ try:+ first = await asyncio.wait_for(stream.__anext__(), timeout=5)+ assert first.url == "fast", f"expected first result 'fast', got {first.url!r}"++ # Close the stream while blocked-1 / blocked-2 are still running.+ await asyncio.wait_for(stream.aclose(), timeout=5)++ started = set(dispatcher.tasks)+ all_done = all(t.done() for t in dispatcher.tasks.values())+ blocked_cancelled = (+ dispatcher.tasks["blocked-1"].cancelled()+ and dispatcher.tasks["blocked-2"].cancelled()+ )++ if (+ started == {"fast", "blocked-1", "blocked-2"}+ and all_done+ and blocked_cancelled+ and dispatcher.concurrent_sessions == 0+ and dispatcher.task_queue.empty()+ ):+ record_result(+ "dispatcher_stream_cleanup", "dispatcher", True,+ "aclose() cancelled in-flight tasks, drained queue, sessions=0",+ )+ else:+ record_result(+ "dispatcher_stream_cleanup", "dispatcher", False,+ f"leaked state: started={started}, all_done={all_done}, "+ f"blocked_cancelled={blocked_cancelled}, "+ f"sessions={dispatcher.concurrent_sessions}, "+ f"queue_empty={dispatcher.task_queue.empty()}",+ )+ finally:+ # Belt-and-suspenders: never let the demo itself leak tasks.+ for t in dispatcher.tasks.values():+ if not t.done():+ t.cancel()+ await asyncio.gather(*dispatcher.tasks.values(), return_exceptions=True)+++# ── Test 2: Basic real-crawl smoke test ───────────────────────────────++async def test_basic_crawl_smoke():+ """The release build should crawl a real page end-to-end and produce+ markdown. Skips gracefully if no browser/network is available."""+ print_test("Basic crawl smoke test", "arun end-to-end")++ from crawl4ai import AsyncWebCrawler, BrowserConfig, CrawlerRunConfig++ try:+ async with AsyncWebCrawler(config=BrowserConfig(headless=True)) as crawler:+ result = await crawler.arun(+ url="https://example.com",+ config=CrawlerRunConfig(),+ )+ if result.success and result.markdown and len(str(result.markdown)) > 0:+ record_result(+ "basic_crawl_smoke", "core", True,+ f"crawled example.com, markdown length={len(str(result.markdown))}",+ )+ else:+ record_result(+ "basic_crawl_smoke", "core", False,+ f"crawl failed: success={result.success}, error={result.error_message}",+ )+ except Exception as e:+ record_result(+ "basic_crawl_smoke", "core", False,+ f"browser/network unavailable — skipped: {e}", skipped=True,+ )+++# ── Main ──────────────────────────────────────────────────────────────++async def main():+ print_header("Crawl4AI v0.9.2 Release Verification")++ await test_dispatcher_stream_cleanup()+ await test_basic_crawl_smoke()++ # Summary+ print_header("RESULTS SUMMARY")+ total = len(results)+ passed = sum(1 for r in results if r.passed and not r.skipped)+ failed = sum(1 for r in results if not r.passed and not r.skipped)+ skipped = sum(1 for r in results if r.skipped)++ for r in results:+ status = "SKIP" if r.skipped else ("PASS" if r.passed else "FAIL")+ print(f" [{status}] {r.name}: {r.message}")++ print(f"\nTotal: {total} | Passed: {passed} | Failed: {failed} | Skipped: {skipped}")++ if failed > 0:+ print("\n⚠️ Some tests FAILED. Review before release.")+ sys.exit(1)+ else:+ print("\n✅ All tests passed!")+++if __name__ == "__main__":+ asyncio.run(main())