Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.
TypeScript apifyautomationcrawlercrawlingheadlessheadless-chromejavascriptnodejsnpmplaywrightpuppeteerscraperscrapingtypescriptweb-crawlerweb-crawlingweb-scraping
Commits on master
chore: limit maxRequestsPerCrawl number in starter templates (#2165)
b59ee50 Jan Buchar committed 3 years ago
chore(deps): update amannn/action-semantic-pull-request action to v5.4.0
316fd3d renovate[bot] committed 3 years ago
chore: fix command for global install of apify CLI (`-G` -> `-g`)
430f025 Martin Adámek committed 3 years ago
docs: mention Apify Academy in blocking guide & link it to proxy and … (#2156)
9ce3f17 Lukáš Křivka committed 3 years ago
chore(deps): update dependency lint-staged to v15 (#2131)
e192a0d renovate[bot] committed 3 years ago
chore(deps): update dependency linkedom to ^0.16.0 (#2139)
abf1841 renovate[bot] committed 3 years ago
fix: add `skipNavigation` option to `enqueueLinks` (#2153)
118515d Gustavo Rudiger committed 3 years ago
fix: provide more detailed error messages for browser launch errors (#2157)
f188ebe Vlad Frangu committed 3 years ago
fix(core): respect some advanced options for `RequestList.open()` + improve docs (#2158)
c5a1b07 Matěj Volf committed 3 years ago
docs: Add Arch linux instructions to contributing guide (#2155)
435d44e Jan Buchar committed 3 years ago
fix(BrowserPool): ignore `--no-sandbox` flag for webkit launcher (#2148)
1eb2f08 Vlad Frangu committed 3 years ago