Skip to content

Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.

Commits on master

3229016
Feature/autoscaling fix (#90)
Marek Trunkát committed 8 years ago
ae2d681
Updated CHANGELOG.md
Jan Curn committed 8 years ago
70fc649
8983546
Client version bump
Marek Trunkat committed 8 years ago
b1acf33
Increasing pageOpsTimeoutMillis to 300s
Marek Trunkat committed 8 years ago
256c74a
Fixed comments
Jan Curn committed 8 years ago
02409dd
Fixed comments
Jan Curn committed 8 years ago
146d84f
Fixing docs
Marek Trunkat committed 8 years ago
b83fc51
Version bump
Marek Trunkat committed 8 years ago
9b4e6cf
Feature/at home (#81)
Marek Trunkát committed 8 years ago
44ef507
Updated DEFAULT_USER_AGENT
Jan Curn committed 8 years ago
0b2dee6
Fix/proxy url regex (#85)
Jan Čurn committed 8 years ago
1d6ee0d
Feature/autoscaling (#82)
Marek Trunkát committed 8 years ago
1e9abde
Updated homepage in package.json
Jan Curn committed 8 years ago
b576ed0
Version bump
JakubDrobnik committed 8 years ago
bed6e42
Copiying opts in launchPuppeteer
Marek Trunkat committed 8 years ago
811ad4d
Fixing changelog
Marek Trunkat committed 8 years ago
46d8f47
Fixing changelog
Marek Trunkat committed 8 years ago
6f08c2b
Version bump
Marek Trunkat committed 8 years ago
bb1976f
Renaming params of getApifyProxyUrl() (#78)
Marek Trunkát committed 8 years ago
c9862f1
Version bump
Marek Trunkat committed 8 years ago
57c6ff7
ApifyClient version bump
Marek Trunkat committed 8 years ago
b476e4b
Updating packages and configuring code coverage (#77)
Marek Trunkát committed 8 years ago
3f1f716
RequestQueue - caching requests locally (#76)
Marek Trunkát committed 8 years ago
9a9977f