Supercharge your AI agents with data from the web and beyond. Building the library for superintelligence. 🔥
TypeScript aiai-agentsai-crawlerai-scrapingai-searchcrawlerdata-extractionhtml-to-markdownllmmarkdownscraperscrapingweb-crawlerweb-dataweb-data-extractionweb-scraperweb-scrapingweb-searchwebscraping
Commits on main
refactor: increase max limit for search request schema from 50 to 100 (#1545)
06189b9 AdemÃlson Tonato committed 1 year ago
Revert "Revert "fix(queue-worker, scrape): match billing logic and add billing for stealth proxies (#1521)""
0fd05a6 Gergő Móricz committed 1 year ago
feat(queue-worker): add more logs around crawl finishing logic
fdeb018 Gergő Móricz committed 1 year ago
[Bug Fix] Make WaitAction milliseconds field optional in firecrawl-py (#1533)
21adf04 devin-ai-integration[bot] committed 1 year ago
Revert "fix(queue-worker, scrape): match billing logic and add billing for stealth proxies (#1521)"
017a915 Gergő Móricz committed 1 year ago
fix(queue-worker, scrape): match billing logic and add billing for stealth proxies (#1521)
e06c7cc Gergő Móricz committed 1 year ago
fix(queue-jobs): never cc timeout jobs that are crawl-associated (makes no sense)
0f32500 Gergő Móricz committed 1 year ago
fix(concurrency-limit): rework cc queue to work by time not priority (#1526)
7ad9a00 Gergő Móricz committed 1 year ago
Merge pull request #1523 from mendableai/refactor/map-limit
5d07ccc AdemÃlson Tonato committed 1 year ago
refactor: maximum links limit for map endpoint from 5000 to 30000
ae12c32 AdemÃlson Tonato committed 1 year ago
Add delay parameter to crawl options in all SDKs (#1514)
0512ad6 devin-ai-integration[bot] committed 1 year ago
Add crawl delay functionality with per-crawl concurrency limiting (FIR-249) (#1413)
411ecdf devin-ai-integration[bot] committed 1 year ago
Merge pull request #1510 from mendableai/devin/1746062769-qwen3-web-crawler-example
a0ed76d Eric Ciarla committed 1 year ago
[feat] Implement GCS storage option for scrape results across controllers an… (#1500)
eee613d Rafael Miller committed 1 year ago
Fix: Handle both dict and model instances in actions parameter (#1508)
f0b1507 devin-ai-integration[bot] committed 1 year ago