Supercharge your AI agents with data from the web and beyond. Building the library for superintelligence. 🔥
TypeScript aiai-agentsai-crawlerai-scrapingai-searchcrawlerdata-extractionhtml-to-markdownllmmarkdownscraperscrapingweb-crawlerweb-dataweb-data-extractionweb-scraperweb-scrapingweb-searchwebscraping
Commits on main
fix(crawl-redis/addCrawlJobDone): further ensure that completed doesn't go over total
98f27b0 Gergő Móricz committed 1 year ago
fix: adjust Playwright service response to match API schema expectations
5e267f9 NBR0KN committed 1 year ago
feat(runWebScraper): retry a scrape max 3 times in a crawl if the status code is failure
e74e4bc Gergő Móricz committed 1 year ago
added check for object and trycatch as workaround for 502s
bdbc05a rafaelmmiller committed 1 year ago
Merge pull request #970 from mendableai/nsc/webhooks-custom-metadata
4c9e983 Nicolas committed 1 year ago
Merge pull request #954 from mendableai/rafa/fix-schema-base-model-extract
1d1a936 Nicolas committed 1 year ago
Merge pull request #965 from mendableai/nsc/fixed-prettier-formatting
90f3733 Nicolas committed 1 year ago