Supercharge your AI agents with data from the web and beyond. Building the library for superintelligence. 🔥
TypeScript aiai-agentsai-crawlerai-scrapingai-searchcrawlerdata-extractionhtml-to-markdownllmmarkdownscraperscrapingweb-crawlerweb-dataweb-data-extractionweb-scraperweb-scrapingweb-searchwebscraping
Commits on main
feat(scrapeURL): handle contentType JSON better in markdown conversion (#1604)
38c96b5 Gergő Móricz committed 1 year ago
feat(apps/api): add support for a separate, non-eviction Redis (#1600)
a5efff0 Gergő Móricz committed 1 year ago
feat(scrapeURL): add unnormalizedSourceURL for url matching DX (FIR-2137) (#1601)
a36c6a4 Gergő Móricz committed 1 year ago
fix(extract-status): be able to get extract status even after TTL lapses (#1599)
a3145cc Gergő Móricz committed 1 year ago
fix(html-transformer): bad outName for og:locale:alternate (FIR-2101) (#1597)
8389a1a Gergő Móricz committed 1 year ago
fix(v1): avoid overwriting rateLimiterMode with FIRE-1 rate limiter (#1593)
3ec17e2 Gergő Móricz committed 1 year ago
feat(queue-worker/afterJobDone): improved ccq insert logic (#1595)
3df687e Gergő Móricz committed 1 year ago
Revert "feat(queue-worker/afterJobDone): improved ccq insert logic"
8571b5a Gergő Móricz committed 1 year ago
feat(queue-worker/afterJobDone): improved ccq insert logic
97c6356 Gergő Móricz committed 1 year ago
feat: parse PDFs on fc side and reject if too long for timeout (FIR-2083) (#1592)
b03670a Gergő Móricz committed 1 year ago
feat(api/v1/authMiddleware): add log to debug extract agent preview mode
2e863da Gergő Móricz committed 1 year ago
feat(concurrency-log): add cclog endpoint (FIR-2067) (#1589)
3e736f1 Gergő Móricz committed 1 year ago
feat(scrapeURL, logJob): log pdf page count to db (FIR-2068) (#1587)
fd74299 Gergő Móricz committed 1 year ago