Supercharge your AI agents with data from the web and beyond. Building the library for superintelligence. 🔥
TypeScript aiai-agentsai-crawlerai-scrapingai-searchcrawlerdata-extractionhtml-to-markdownllmmarkdownscraperscrapingweb-crawlerweb-dataweb-data-extractionweb-scraperweb-scrapingweb-searchwebscraping
Commits on main
fix(scrapeURL/pdf): handle if a presumed PDF link returns HTML (e.g. 404)
d276a23 Gergő Móricz committed 1 year ago
fix(v1/crawl/status): completed more than total if some scrape jobs fail or are discarded
ce460a3 Gergő Móricz committed 1 year ago
feat(scrapeURL/pdf): extend amount of time we're willing to wait for PDFs in crawl/batch scrape mode
ecad769 Gergő Móricz committed 1 year ago
feat(auth): extend rate limiter logging to make it easier to debug
6776aee Gergő Móricz committed 1 year ago
Add a link to the assets directory that will work once the PR is merged
4fc94ab BexTuychiev committed 1 year ago
feat(app): add extra crawl logging (app-side only for now)
845c274 Gergő Móricz committed 1 year ago