Supercharge your AI agents with data from the web and beyond. Building the library for superintelligence. 🔥
TypeScript aiai-agentsai-crawlerai-scrapingai-searchcrawlerdata-extractionhtml-to-markdownllmmarkdownscraperscrapingweb-crawlerweb-dataweb-data-extractionweb-scraperweb-scrapingweb-searchwebscraping
Commits on main
queue-worker: move scrape worker to separate threads (ENG-3008) (#1879)
0f44d94 Gergő Móricz committed 1 year ago
feat(crawler): replace robotstxt library with texting_robots for ENG-3016 (#1895)
e7e2c02 devin-ai-integration[bot] committed 1 year ago
Revert "Revert "fix: get rid of debug redis reinstantiation""
78d008c Gergő Móricz committed 1 year ago
fix(queue-jobs): go back to doing setinterval in waitforjob
fa42177 Gergő Móricz committed 1 year ago
feat: add crawlTtlHours team flag to replace teamIdsExcludedFromExpiry (#1899)
cf5eea0 devin-ai-integration[bot] committed 1 year ago
Revert "revert all haphazard changes made while bughunting"
50e88cf Gergő Móricz committed 1 year ago
Fix Pydantic field name shadowing issues causing import NameError (#1800)
a7aa0cb devin-ai-integration[bot] committed 1 year ago
Revert "fix(go): add mutex to prevent concurrent access issues in html-to-markdown (#1883)"
96a9210 Gergő Móricz committed 1 year ago
fix(js-sdk): add retry logic for socket hang up errors in monitorJobStatus (ENG-3029) (#1893)
049d2e6 devin-ai-integration[bot] committed 1 year ago
fix(go): add mutex to prevent concurrent access issues in html-to-markdown (#1883)
52e7d0d Gergő Móricz committed 1 year ago