Official Crawl4AI Cloud: a hosted web scraping, extraction, and search API for AI agents. Free API key, 1,000 credits a month, no card. Fully open source: this repo is the Apache-2.0 crawler that powers it, and you can self-host it.
Python aiai-agentscrawl4ai-cloudcrawlerdata-extractionllmmarkdownmcpopen-sourceplaywrightpythonragscraperweb-crawlerweb-crawlingweb-scraping
fix: default max_pages_before_recycle in DEFAULT_CONFIG
ntohidi15f4b29b8ba2
diff --git a/deploy/docker/utils.py b/deploy/docker/utils.pyindex 8f8ecb18..0fc2e69c 100644--- a/deploy/docker/utils.py+++ b/deploy/docker/utils.py@@ -64,7 +64,7 @@ DEFAULT_CONFIG = {"memory_threshold_percent": 95.0,"rate_limiter": {"enabled": True, "base_delay": [1.0, 2.0]},"timeouts": {"stream_init": 30.0, "batch_process": 300.0},- "pool": {"max_pages": 40, "idle_ttl_sec": 300},+ "pool": {"max_pages": 40, "idle_ttl_sec": 300, "max_pages_before_recycle": 200},"browser": {"kwargs": {"headless": True, "text_mode": True},"extra_args": [