Skip to content

Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.

Commits on master

moved abortFunction to DEFAULTS
Petr Patek committed 7 years ago
b9f9236
Lint docs, build readme
Travis CI committed 7 years ago
6e17e5f
Optimize docusaurus
Travis CI committed 7 years ago
15ab9e2
Removed unnecessary iltorb dependency.
Petr Patek committed 7 years ago
12b0da7
Update prettier config and lint
Travis CI committed 7 years ago
b5644aa
SDK tut tests pass
Travis CI committed 7 years ago
0503a31
Update CHANGELOG.md
Ondra Urban committed 7 years ago
6656466
44ef7a1
Better logging of timeouts in crawlers
Jan Curn committed 7 years ago
995f644
added headers order check.
Petr Patek committed 7 years ago
87daf7b
Merge pull request #418 from davidturnbull/master
Ondra Urban committed 7 years ago
c0a457c
fixed comment
Petr Patek committed 7 years ago
e25ba5d
fixed user-agents.
Petr Patek committed 7 years ago
39bb6db
fixed imports.
Petr Patek committed 7 years ago
470c5dc
Rewrote to use the http request.
Petr Patek committed 7 years ago
1b16f07
Format README with Prettier
David Turnbull committed 7 years ago
e2e770e
Format documentation with Prettier
David Turnbull committed 7 years ago
277463b
Add Markdown formatting to “Examples”
David Turnbull committed 7 years ago
0a1429d
Update travis scripts
Travis CI committed 7 years ago
8e8c93d
Fix docker build in travis
Travis CI committed 7 years ago
7691ecb
Do not update xvfb package.json in release
Travis CI committed 7 years ago
ba1af72
Update release-images script
Travis CI committed 7 years ago
878dfcb
Update travis configuration
Ondra Urban committed 7 years ago
e60ec42
Update travis configuration
Ondra Urban committed 7 years ago
30e6b6d
Update travis configuration
Ondra Urban committed 7 years ago
060265d
Add Markdown formatting with Prettier to build process
David Turnbull committed 7 years ago
3826c8f