Supercharge your AI agents with data from the web and beyond. Building the library for superintelligence. 🔥
TypeScript aiai-agentsai-crawlerai-scrapingai-searchcrawlerdata-extractionhtml-to-markdownllmmarkdownscraperscrapingweb-crawlerweb-dataweb-data-extractionweb-scraperweb-scrapingweb-searchwebscraping
Commits on main
Merge pull request #1084 from mendableai/added-today-to-extract-prompts
3f9b8a0 Rafael Miller committed 1 year ago
feat(crawl-status): allow for jobs to expire out of the redis
95ce3c3 Gergő Móricz committed 1 year ago
fix(crawler): move sitemap deduplication to deeper in the process
7219812 Móricz Gergő committed 1 year ago
Nick: increased timeouts on extract + reduced extract redis usage
ccb74a2 Nicolas committed 1 year ago