Official Crawl4AI Cloud: a hosted web scraping, extraction, and search API for AI agents. Free API key, 1,000 credits a month, no card. Fully open source: this repo is the Apache-2.0 crawler that powers it, and you can self-host it.
Python aiai-agentscrawl4ai-cloudcrawlerdata-extractionllmmarkdownmcpopen-sourceplaywrightpythonragscraperweb-crawlerweb-crawlingweb-scraping
Commits on main
Merge pull request #2335 from ege-arhan/fix/cli-json-output-help
29c28eb Nasrin committed 1 month ago
Merge pull request #2362 from unclecode/fix/cli-json-output-2334
0335a2e Nasrin committed 1 month ago
Merge pull request #2363 from unclecode/fix/noscript-count-skip-raw-text
a6e3bb3 Nasrin committed 1 month ago
fix: keep the raw-text strip linear on unclosed script, style and comments
3442707 ntohidi committed 1 month ago
fix(scraping): keep a space where a whitespace-only element is removed
c381413 drakeo338 committed 1 month ago
fix: ignore <noscript> text inside script, style and comments when counting tags
b59dc9c ntohidi committed 1 month ago
Merge pull request #2361 from alkaz-nodemaven/fix/redact-proxy-credentials-in-http-errors
01d5565 Nasrin committed 1 month ago
Merge pull request #2327 from omarbounawarapy/bugfix/http-proxy-credentials-encoding
bcad146 Nasrin committed 1 month ago
fix(cli): fail early when -o json has no extraction strategy
1e54fa7 Soham Kukreti committed 1 month ago
Decode percent-encoded credentials in ProxyConfig.from_string
8b3a75d omarbounawarapy committed 1 month ago
Percent-encode proxy credentials in the HTTP crawler proxy URL
2de19c8 omarbounawarapy committed 1 month ago
Merge pull request #2290 from talelboussetta/bugfix/crawler-configs-single-url-and-stream
4268340 Nasrin committed 1 month ago
docs(cli): clarify that --output json requires an extraction strategy for structured content
36cea05 ege-arhan committed 1 month ago
fix(docker): apply base_config to the crawler_configs list on the streaming path
974275d Talel Boussetta committed 1 month ago
Merge pull request #2320 from Kylinny/fix-pdf-page-number-race-2319
3bd2450 Nasrin committed 1 month ago
Merge pull request #2317 from Artechie0806/fix/2309-reset-pages-crawled
82af20f Nasrin committed 1 month ago
Merge pull request #2360 from unclecode/fix/2293-fit-html-noscript
89c2642 Nasrin committed 1 month ago
fix(utils): keep fit_html body nested in an unclosed <noscript>
ba8b722 ntohidi committed 1 month ago
style(docker): black-format the new crawler_configs code
4d645a4 Talel Boussetta committed 1 month ago
fix(docker): let a per-URL config list decide the PDF crawler
5ce402f Talel Boussetta committed 1 month ago
fix(docker): apply crawler_configs to single-URL and streaming requests
81efcc3 Talel Boussetta committed 1 month ago
Merge pull request #2284 from tommypacker/bugfix/unclosed-noscript-empties-cleaned-html
07ff307 Nasrin committed 1 month ago
Merge pull request #2359 from unclecode/fix/2288-per-seed-refusal
3e12a96 Nasrin committed 1 month ago