Supercharge your AI agents with data from the web and beyond. Building the library for superintelligence. 🔥
TypeScript aiai-agentsai-crawlerai-scrapingai-searchcrawlerdata-extractionhtml-to-markdownllmmarkdownscraperscrapingweb-crawlerweb-dataweb-data-extractionweb-scraperweb-scrapingweb-searchwebscraping
Commits on main
feat(crawler): handle cross-origin redirects differently than same-origin redirects (#1279)
e8c698d Gergő Móricz committed 1 year ago
Merge pull request #1276 from aparupganguly/feature/gpt4.5-crawler
ab8dcab Eric Ciarla committed 1 year ago
(feat/deep-research-alpha) Added Max Urls, Sources and Fixes (#1271)
289e351 Nicolas committed 1 year ago
feat(self-host/ai): pass in the ollama envs into docker compose (#1269)
78334e4 Benjy committed 1 year ago
fix(scraper): improve charset detection regex to accurately parse meta tags (#1265)
7bf04d4 Grass Huang committed 1 year ago
Merge pull request #1257 from aparupganguly/feature/claude3.7-crawler
2da6d7b Eric Ciarla committed 1 year ago
feat(v1/crawl-status-ws): update behavior to ignore errors like regular crawl-status (#1234)
8c42b08 Gergő Móricz committed 1 year ago
docker: force host to 0.0.0.0 to fix env precedence issues
b88b573 Gergő Móricz committed 1 year ago