Official Crawl4AI Cloud: a hosted web scraping, extraction, and search API for AI agents. Free API key, 1,000 credits a month, no card. Fully open source: this repo is the Apache-2.0 crawler that powers it, and you can self-host it.
Python aiai-agentscrawl4ai-cloudcrawlerdata-extractionllmmarkdownmcpopen-sourceplaywrightpythonragscraperweb-crawlerweb-crawlingweb-scraping
Commits on main
Merge pull request #1922 from unclecode/fix/deep-crawl-streaming-contextvar-1917
5e56e34 Nasrin committed 7 months ago
fix: wire semaphore_count into auto-created MemoryAdaptiveDispatcher (#1927)
d3c92ee hafezparast committed 7 months ago
fix(security): SSRF protection on all crawl/md/llm URL entry points
f77c0a8 unclecode committed 7 months ago
docs: add Privacy Policy, Terms of Service, and Support pages
0e92b5e unclecode committed 7 months ago
fix: add LLMTableExtraction to Docker API deserialization allowlist
c8c2dc3 Gab committed 7 months ago
Merge pull request #1913 from unclecode/fix/nlp-sentence-chunking-1909
4e86399 Nasrin committed 7 months ago
fix(deep-crawl): use set(False) instead of reset(token) for ContextVar (#1917)
3d4bda1 ntohidi committed 7 months ago
chore: add GitHub Security Advisory payloads for release day
c991469 unclecode committed 7 months ago
fix(security): batch 2 - JWT secret, eval removal, execute_js, hook sandbox
0f20f8b unclecode committed 7 months ago
fix(security): patch 4 vulns - file write, SSRF, monitor auth, XSS
7976b45 unclecode committed 7 months ago
fix(chunking): preserve sentence order in NlpSentenceChunking (#1909)
c837c0d ntohidi committed 7 months ago
feat: add preserve_classes/preserve_tags to PruningContentFilter (#1900)
8ff2fcb hafezparast committed 7 months ago
fix: correct arun() return type from RunManyReturn to CrawlResultContainer (#1898)
c5612f7 hafezparast committed 7 months ago
Merge pull request #1852 from hafezparast/feat/maysam-arun-many-config-list-1837
3d02d75 Nasrin committed 7 months ago
fix: default LLMExtractionStrategy extraction_type to schema
ec560f1 unclecode committed 7 months ago
fix: convert page_timeout from ms to seconds for aiohttp ClientTimeout (#1894)
0a51902 hafezparast committed 7 months ago
fix(security): complete AST sandbox escape remediation (CVSS 9.8)
e326da9 unclecode committed 7 months ago
fix(security): remove eval() from computed fields, harden config deserializer
2fc39cb unclecode committed 7 months ago
docs: update version references to 0.8.6 in README and Docker docs
bcbccbe ntohidi committed 7 months ago
Merge pull request #1882 from hafezparast/fix/crawler-config-dict-validation-1880
7e7533e Nasrin committed 7 months ago
fix: validate markdown_generator type in CrawlerRunConfig to catch bad JSON format (#1880)
e9f8322 hafezparast committed 7 months ago
fix: replace litellm with unclecode-litellm due to PyPI supply chain compromise
4e4a996 unclecode committed 8 months ago
fix: pin litellm to safe fork (v1.81.13) due to PyPI supply chain compromise
01c685c unclecode committed 8 months ago