Supercharge your AI agents with data from the web and beyond. Building the library for superintelligence. 🔥
TypeScript aiai-agentsai-crawlerai-scrapingai-searchcrawlerdata-extractionhtml-to-markdownllmmarkdownscraperscrapingweb-crawlerweb-dataweb-data-extractionweb-scraperweb-scrapingweb-searchwebscraping
Commits on main
Remove old cache mechanisms (redis cache, PDF cache, crawl maps, etc.) (FIR-2266) (#1667)
3f0b8b8 Gergő Móricz committed 1 year ago
feat: add followInternalLinks parameter as semantic replacement for allowBackwardLinks (#1684)
09aabbe devin-ai-integration[bot] committed 1 year ago
Concurrency limit refactor + `maxConcurrency` parameter (FIR-2191) (#1643)
f8983ff Gergő Móricz committed 1 year ago
feat(scrape, extract): creditsUsed, tokensUsed fields (FIR-2336) (#1683)
a8e3c29 Gergő Móricz committed 1 year ago
fix(scrape): log FIRE-1 credits billed on failures properly (FIR-2331) (#1682)
fbd81b4 Gergő Móricz committed 1 year ago
feat(crawl-status): refactor to work after a redis flush (#1664)
ebc1de9 Gergő Móricz committed 1 year ago
Add deployment type field to bug report template (#1681)
cd2e0f8 devin-ai-integration[bot] committed 1 year ago
fix: send the request to new mu version before the main one to achieve better sync
f46f845 Thomas Kosmas committed 1 year ago
Improve URL filtering error messages with specific denial reasons (FIR-2352) (#1676)
9710bdf devin-ai-integration[bot] committed 1 year ago
feat(vertex): fix vertex ai provider bug and update model references to use "gemini-2.5-pro" (#1668)
38c5795 Thomas Kosmas committed 1 year ago
feat(index): add domain splitting for improved map querying (#1666)
0bf2307 Gergő Móricz committed 1 year ago
fix(api): remove query parameter sanitization that was breaking extracts (#1661)
89e42b1 Gergő Móricz committed 1 year ago
Fix PLAYWRIGHT_MICROSERVICE_URL env var to use /scrape endpoint (#1654)
255be2a Pulkit Saini committed 1 year ago