Supercharge your AI agents with data from the web and beyond. Building the library for superintelligence. 🔥
TypeScript aiai-agentsai-crawlerai-scrapingai-searchcrawlerdata-extractionhtml-to-markdownllmmarkdownscraperscrapingweb-crawlerweb-dataweb-data-extractionweb-scraperweb-scrapingweb-searchwebscraping
Commits on main
feat(api/worker): liveness check in queueing -- don't take jobs when the worker is dying (#1725)
57b8e66 Gergő Móricz committed 1 year ago
Fix unreachable allowSubdomains code in crawler filterURL method (#1719)
070d1c1 devin-ai-integration[bot] committed 1 year ago
feat(scrapeURL/skipTlsVerification): improve error message (#1723)
55d5c1f Gergő Móricz committed 1 year ago
feat(python-sdk): add parsePDF parameter support (#1713)
1919799 devin-ai-integration[bot] committed 1 year ago
Add temporary exception for Faire team ID to bypass job expiration (#1716)
89e57ac devin-ai-integration[bot] committed 1 year ago
fix(api): cached acuc didn't have the is_extract flag set (#1712)
3d04c20 Gergő Móricz committed 1 year ago
fix(concurrency-limit): scan instead of taking jobs (#1708)
cc3afa2 Gergő Móricz committed 1 year ago
fix(api/batch/scrape): maxConcurrency field support when using ignoreInvalidURLs (#1705)
11f4694 Gergő Móricz committed 1 year ago
fix(api/v1/crawl/ongoing): only crawls, no batch scrape (#1701)
439619f Gergő Móricz committed 1 year ago
feat(api): optimize job count query and improve error handling (#1700)
1fdf959 Gergő Móricz committed 1 year ago
fix(api): handle errors better in redis-less crawl status (#1699)
c311724 Gergő Móricz committed 1 year ago
fix(api): enhance error handler with optional ACUC data (#1698)
66cde50 Gergő Móricz committed 1 year ago
fix(api): improve error logging with structured error object (#1697)
e06ec2d Gergő Móricz committed 1 year ago
feat(scrapeURL): separate URL rewrites to different function
7ed19c0 Gergő Móricz committed 1 year ago