Skip to content

Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.

Commits on master

Build docs
Ondra Urban committed 7 years ago
0998861
1fef1c0
Regenerated docs
Jan Curn committed 7 years ago
eaf8337
cd52be8
4d70d25
Updated BLOCK_TAGS_REGEX
Jan Curn committed 7 years ago
7c6a6a0
4e2b8e2
Updated comments
Jan Curn committed 7 years ago
e3425c4
Updated comments
Jan Curn committed 7 years ago
66b0712
Fix reusePages with request interception
Ondra Urban committed 7 years ago
70bbb4e
Build docs
Ondra Urban committed 7 years ago
fca35cf
Bump package version
Ondra Urban committed 7 years ago
ee006c8
a15d2b9
Fix flaky test
Ondra Urban committed 7 years ago
745a75d
Fix reusePages memory leak and config
Ondra Urban committed 7 years ago
2ec3f27
Updated package.json file
Jan Curn committed 7 years ago
9d9a244
2652030
Improved SKIP_TAGS_REGEX in Apify.utils
Jan Curn committed 7 years ago
dad07ea
Fix changelog
Ondra Urban committed 7 years ago
7692b69
Bump package version
Ondra Urban committed 7 years ago
3fc5a0c
Update changelog, fix missing parameter
Ondra Urban committed 7 years ago
da6c86f
Fix version in changelog
Ondra Urban committed 7 years ago
a00d3ac
Fix wrong package version bump
Ondra Urban committed 7 years ago
7ffdd14
Update docs
Ondra Urban committed 7 years ago
12c09d7
Update hello_world example
Ondra Urban committed 7 years ago
1bb1a76
Build readme
Ondra Urban committed 7 years ago
809b666
Update readme build script
Ondra Urban committed 7 years ago
3b9177f
Build docs
Ondra Urban committed 7 years ago
b9b6249
Bump package version
Ondra Urban committed 7 years ago
f357caf
Improve maxRequestPerCrawl log message
Ondra Urban committed 7 years ago
e0b6714