Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.
TypeScript apifyautomationcrawlercrawlingheadlessheadless-chromejavascriptnodejsnpmplaywrightpuppeteerscraperscrapingtypescriptweb-crawlerweb-crawlingweb-scraping
Commits on master
ci: add the apify-factory review workflow (#4266)
6a51c89 Apify Service Account committed 1 month ago
fix!: stop raising the global `EventEmitter.defaultMaxListeners` on import (#4164)
8cb2eb6 Noah committed 1 month ago
fix(utils): keep sitemap URLs written as CDATA surrounded by whitespace (#4250)
829fd57 Jay Hemnani committed 1 month ago
chore: enforce file extensions in relative imports (#4246)
dc52c6d Martin Adámek committed 1 month ago
docs: Mention initialConcurrency in v4 upgrading guide (#4245)
5ca137f Jan Buchar committed 1 month ago
chore: move package-level tests into the root test folder (#4236)
3a8fd8e Martin Adámek committed 1 month ago
chore(deps): update dependency puppeteer to v25.13.0 (#4238)
386048d renovate[bot] committed 1 month ago
chore: rename source and test files to kebab-case (#4234)
9c9e7a5 Martin Adámek committed 1 month ago
docs: split the v4 upgrading guide into multiple pages (#4223)
a5b4794 Martin Adámek committed 1 month ago
refactor: Extract remote logic from abstract browser pool (#4197)
d893412 Jan Buchar committed 1 month ago
chore(deps): update vitest monorepo to v5 and adapt the tests (#4217)
8f57dd3 renovate[bot] committed 1 month ago
chore(deps): update lerna to v10 and drop its pnpm overrides (#4209)
8713eae renovate[bot] committed 1 month ago
chore(deps): update dependency whatwg-mimetype to v5 (#3357)
9b813f6 renovate[bot] committed 1 month ago
chore(deps): update dependency @types/mime-types to v3 (#4201)
38a75e8 renovate[bot] committed 1 month ago
chore(deps): update content-type to v3 and drop @types/content-type (#4200)
4fcfa4d renovate[bot] committed 1 month ago
chore: keep transitive playwright in sync with the pinned version (#4199)
bf7f904 Martin Adámek committed 1 month ago