Supercharge your AI agents with data from the web and beyond. Building the library for superintelligence. 🔥
TypeScript aiai-agentsai-crawlerai-scrapingai-searchcrawlerdata-extractionhtml-to-markdownllmmarkdownscraperscrapingweb-crawlerweb-dataweb-data-extractionweb-scraperweb-scrapingweb-searchwebscraping
Commits on main
fix(scrapeURL/fire-engine): handle "Operation timed out" error correctly
52e3313 Gergő Móricz committed 1 year ago
(python-sdk): get_crawl_errors and active_crawls, got rid of useless tests
d2b325f rafaelmmiller committed 1 year ago
fix(queue-worker): disable cacheable lookup for liveness check
904bba6 Gergő Móricz committed 1 year ago
feat(crawl-redis): reduce `crawl:<id>:visited` size in Redis by 16x (#1936)
7611f81 Gergő Móricz committed 1 year ago
fix: prevent PDF scrapes with parsePDF:false from being indexed (#1933)
bfbff89 Gergő Móricz committed 1 year ago
Revert "fix(queue-worker): decrease concurrency to avoid stalling other workers"
1a98428 Gergő Móricz committed 1 year ago
fix(queue-worker): decrease concurrency to avoid stalling other workers
33bf7eb Gergő Móricz committed 1 year ago
feat(python-sdk): add agent parameter support to scrape_url method (#1919)
71829db devin-ai-integration[bot] committed 1 year ago
ENG-3088: Change parsers parameter from object to array format (#1931)
cb1415a Gergő Móricz committed 1 year ago
chore: update pnpm-lock.yaml with new package versions and dependencies
5998744 Ademílson Tonato committed 1 year ago
fix(queue-worker): lower pressure by doubling lock duration
3b25432 Gergő Móricz committed 1 year ago
fix: handle --max-old-space-size flag for worker threads (#1922)
75e3cd5 Josh-M42 committed 1 year ago