Supercharge your AI agents with data from the web and beyond. Building the library for superintelligence. 🔥
TypeScript aiai-agentsai-crawlerai-scrapingai-searchcrawlerdata-extractionhtml-to-markdownllmmarkdownscraperscrapingweb-crawlerweb-dataweb-data-extractionweb-scraperweb-scrapingweb-searchwebscraping
Commits on main
feat: Add iframe selector transformation for includeTags and excludeTags (#1850)
659e05b devin-ai-integration[bot] committed 1 year ago
Add __experimental_omceDomain flag for debugging and benchmarking (#1844)
a08f343 devin-ai-integration[bot] committed 1 year ago
Fix ignoreQueryParameters being ignored in URL deduplication - ENG-2804 (#1846)
4268470 devin-ai-integration[bot] committed 1 year ago
Add allowTeammateInvites flag to TeamFlags type (#1847)
4df5018 devin-ai-integration[bot] committed 1 year ago
Fix robots.txt parser panic with content type validation (#1843)
0f477be devin-ai-integration[bot] committed 1 year ago
feat: rewrite sitemap XML parsing from JavaScript to Rust (ENG-2904) (#1840)
e0b26f8 devin-ai-integration[bot] committed 1 year ago
fix(queue-worker): improve stalled job cleaner (ENG-2907) (#1839)
5506dd9 Gergő Móricz committed 1 year ago
fix(queue-worker): clean up stalled jobs to not get crawls stuck (#1838)
30347c8 Gergő Móricz committed 1 year ago
fix(crawl): only extend URLs at pre-finish if changeTracking is on (#1837)
3ecbc2f Gergő Móricz committed 1 year ago
chore(deps): bump esbuild and tsx in /apps/js-sdk (#1834)
af8892b dependabot[bot] committed 1 year ago
chore(deps): bump axios from 1.6.8 to 1.11.0 in /apps/js-sdk (#1833)
3a9445a dependabot[bot] committed 1 year ago
chore(deps): bump form-data from 4.0.0 to 4.0.4 in /apps/js-sdk (#1832)
a2f5c1f dependabot[bot] committed 1 year ago
fix(python-sdk): add max_age parameter to scrape_url validation (#1825)
26926e5 devin-ai-integration[bot] committed 1 year ago
feat(types): add JSON schema validation to schema options in extractOptions and baseScrapeOptions (#1803)
b850064 Rafael Miller committed 1 year ago
(feat/rtxt) Improved robots control on scrape via flags (#1820)
b23535c Nicolas committed 1 year ago
fix(api/v1/types): depth check throws error if URL is invalid (#1821)
6849ab1 Gergő Móricz committed 1 year ago