Skip to content

Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.

Commits on master

Updating README
Marek Trunkat committed 8 years ago
031c17a
Updating README
Marek Trunkat committed 8 years ago
39dac95
Updating README
Marek Trunkat committed 8 years ago
42762fa
Updating README
Marek Trunkat committed 8 years ago
d5b02ab
Updating README
Marek Trunkat committed 8 years ago
2912563
Updated comments in tests
Jan Curn committed 8 years ago
2be1191
Updating README
Marek Trunkat committed 8 years ago
848d13d
Merging README-new
Marek Trunkat committed 8 years ago
2a83e20
Updating README.md
Marek Trunkat committed 8 years ago
90da62d
Fixed lint error
Jan Curn committed 8 years ago
77e94fd
Less verbose logging in tests
Jan Curn committed 8 years ago
f6c0507
46f4fb7
Updated comments
Jan Curn committed 8 years ago
8f91ab1
8e3c9c6
2fb5c6c
Updated DEFAULT_USER_AGENT and USER_AGENT_LIST
Jan Curn committed 8 years ago
104a045
Merge branch 'develop' into feature/env_vars
Marek Trunkát committed 8 years ago
dd295fb
Updating readme
Marek Trunkat committed 8 years ago
9b88000
Fixing typo
Marek Trunkat committed 8 years ago
6865140
Updating readme
Marek Trunkat committed 8 years ago
1710ef5
Udpating README.md
Marek Trunkat committed 8 years ago
9300f24
Udpating README.md
Marek Trunkat committed 8 years ago
4b8ad1e
Adding a TODO
Marek Trunkat committed 8 years ago
80c5503
Udpating env vars docs in readme
Marek Trunkat committed 8 years ago
3e75cb0
Fixing tests
Marek Trunkat committed 8 years ago
884eb42
Fix example running steps in README
Ondra Urban committed 8 years ago
a7e93bb
Fix typo
Ondra Urban committed 8 years ago
b83fc09
Finalize example revamp
Ondra Urban committed 8 years ago
ab0c06a
4a08730