Supercharge your AI agents with data from the web and beyond. Building the library for superintelligence. 🔥
TypeScript aiai-agentsai-crawlerai-scrapingai-searchcrawlerdata-extractionhtml-to-markdownllmmarkdownscraperscrapingweb-crawlerweb-dataweb-data-extractionweb-scraperweb-scrapingweb-searchwebscraping
Commits on main
feat(scrapeURL): log cache age in request frequency (#1784)
6337afe Gergő Móricz committed 1 year ago
feat(index): store request frequency for precrawling (#1782)
eab6909 Gergő Móricz committed 1 year ago
fix(scrapeURL/index): horrible no-good very bad index url bug (#1780)
d7204e4 Gergő Móricz committed 1 year ago
Revert "fix(api/html-transformer): relative base tag URL handling (#1776)"
9cad755 Gergő Móricz committed 1 year ago
fix(api/html-transformer): relative base tag URL handling (#1776)
4e81246 Gergő Móricz committed 1 year ago
fix(api/html-transformer): stop panicing on arabic sites (#1773)
71c8224 Gergő Móricz committed 1 year ago
Add created_at field to /crawl/active endpoint response (#1718)
8922d7e devin-ai-integration[bot] committed 1 year ago
fix(logger): correct method names in logger.child calls (#1731)
5808196 ahnafatef committed 1 year ago
Make worker Express server port configurable via environment variable (#1748)
b28d4f7 devin-ai-integration[bot] committed 1 year ago
fix: replace hardcoded search billing with calculateCreditsToBeBilled (#1714)
82e76a7 devin-ai-integration[bot] committed 1 year ago