Supercharge your AI agents with data from the web and beyond. Building the library for superintelligence. 🔥
TypeScript aiai-agentsai-crawlerai-scrapingai-searchcrawlerdata-extractionhtml-to-markdownllmmarkdownscraperscrapingweb-crawlerweb-dataweb-data-extractionweb-scraperweb-scrapingweb-searchwebscraping
Commits on main
feat(api/extract): show extract as origin for scrapes originating from it (#1584)
6478754 Gergő Móricz committed 1 year ago
fix(queue-worker): finish crawl if all addable URLs were already locked (#1582)
a3aee9b Gergő Móricz committed 1 year ago
docs: add MAX_RAM and MAX_CPU environment variables documentation (#1581)
c7bb9e7 devin-ai-integration[bot] committed 1 year ago
FIR-2006: Fix maxUrls and timeLimit parameters in Deep Research API (#1569)
9949403 devin-ai-integration[bot] committed 1 year ago
feat(python-sdk/CrawlWatcher): remove max payload size from WebSocket (FIR-2038) (#1577)
513f469 Gergő Móricz committed 1 year ago
Fix: Concatenate metadata arrays into strings with exceptions (#1574)
a5a915d devin-ai-integration[bot] committed 1 year ago
hotfix: kill zombie workers, respect timeouts better (FIR-2034) (#1575)
f838190 Gergő Móricz committed 1 year ago
feat(scrapeURL/pdf/mu): add timeout and created_at (#1570)
192d056 Gergő Móricz committed 1 year ago
FIR-1951: Add automatic URL encoding in preprocessing for special characters in query parameters (#1547)
8eeb3c5 devin-ai-integration[bot] committed 1 year ago
Fix LLMs.txt cache bug with subdomains and add bypass option (#1557)
7ccbbec devin-ai-integration[bot] committed 1 year ago
Fix Supabase client configuration errors when USE_DB_AUTHENTICATION is false (#1534)
ab30c8e devin-ai-integration[bot] committed 1 year ago
Add caching for RunPod PDF markdown results in GCS (#1561)
526165e devin-ai-integration[bot] committed 1 year ago