Supercharge your AI agents with data from the web and beyond. Building the library for superintelligence. 🔥
TypeScript aiai-agentsai-crawlerai-scrapingai-searchcrawlerdata-extractionhtml-to-markdownllmmarkdownscraperscrapingweb-crawlerweb-dataweb-data-extractionweb-scraperweb-scrapingweb-searchwebscraping
Commits on main
feat: add waitFor validation rule to enforce waitFor <= timeout/2 (#1751)
251e42c devin-ai-integration[bot] committed 1 year ago
Add end-to-end test for deep research functionality (ENG-2627) (#1760)
b5b6ca0 devin-ai-integration[bot] committed 1 year ago
Implement fallback mechanism for onlyMainContent scraping (ENG-2499) (#1759)
46850d1 devin-ai-integration[bot] committed 1 year ago
fix(html-transformer): check base tag when resolving relative URLs (#1761)
8d81209 Gergő Móricz committed 1 year ago
Implement JavaScript-only base href handling (ENG-2302) (#1756)
de1bc96 devin-ai-integration[bot] committed 1 year ago
Make error message for backwards crawling more clear (#1758)
4c1af8b Micah Stairs committed 1 year ago
fix(api/deep-research): bad default values leading to bug on self-host (#1754)
6d8e771 Gergő Móricz committed 1 year ago
Add URL depth validation to crawl requests (ENG-2617) (#1753)
a9f4b1f devin-ai-integration[bot] committed 1 year ago
Add health check endpoint for Playwright service (ENG-2585) (#1752)
de4094b devin-ai-integration[bot] committed 1 year ago
bugfix zero_data_retention param and certifi dependency (#1749)
1797371 Rafael Miller committed 1 year ago
Merge pull request #1715 from mendableai/feat/integration-param-pythonsdk
8b54a37 Ademílson Tonato committed 1 year ago
Fix queue worker liveness endpoint for self-hosted environments (#1747)
72b9074 devin-ai-integration[bot] committed 1 year ago
feat(firecrawl): add integration parameter support and enhance kwargs handling
30b7e17 Ademílson Tonato committed 1 year ago
Add local environment configuration to docker-compose services (#1742)
dcbed18 Nicolas committed 1 year ago
feat(queue-worker): decrease job lock duration to pick up jobs on dead workers faster (#1737)
ebf98e3 Gergő Móricz committed 1 year ago
feat(scrapeURL): ask user to increase timeout if there's a DOM.getDocument or queryAXTree error (#1739)
400d497 Gergő Móricz committed 1 year ago
feat: implement IDN support with Punycode encoding (#1735)
1816cfc devin-ai-integration[bot] committed 1 year ago