Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.
TypeScript apifyautomationcrawlercrawlingheadlessheadless-chromejavascriptnodejsnpmplaywrightpuppeteerscraperscrapingtypescriptweb-crawlerweb-crawlingweb-scraping
Commits on master
docs: fix search modal styles after docsearch v4 update (#3175)
9176a21 Martin Adámek committed 1 year ago
chore(release): update internal dependencies [skip ci]
fae891d Apify Release Bot committed 1 year ago
ci: Remove Node.js 16 from test matrices while preserving runtime compatibility (#3172)
f7c25dc Copilot committed 1 year ago
fix: use correct config for storage classes to avoid memory leaks (#3144)
911a2eb Edgars Beigarts committed 1 year ago
chore(deps): update actions/github-script action to v8 (#3154)
2094fdd renovate[bot] committed 1 year ago
perf: Improve glob performance by reusing minimatch objects (#3168)
e5632e2 Edgars Beigarts committed 1 year ago
chore(release): update internal dependencies [skip ci]
3f9be81 Apify Release Bot committed 1 year ago
chore(deps): update dependency axios to v1.12.0 [security]
cd8521c renovate[bot] committed 1 year ago
fix: respect `maxCrawlDepth` with a custom enqueueLinks `transformRequestFunction` (#3159)
e2ecb74 Lukáš Průša committed 1 year ago
feat: add `TandemRequestProvider` for combined `RequestList` and `RequestQueue` usage (#2914)
4ca450f हिमांशु committed 1 year ago