Skip to content

Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.

Commits on master

Improve getRandomUserAgent() docs
Ondra Urban committed 8 years ago
fb90a25
Add getRandomUserAgent() to public utils
Ondra Urban committed 8 years ago
b19c913
Add a list of common user agents
Ondra Urban committed 8 years ago
7d77e6d
Removing duplicate from COMMON_LOCAL_FILE_EXTENSIONS
Marek Trunkát committed 8 years ago
9f65786
Adding some extensions to COMMON_LOCAL_FILE_EXTENSIONS
Marek Trunkát committed 8 years ago
e2cbf27
Catch only ENOENT errors
Ondra Urban committed 8 years ago
dc3ae47
Update tests
Ondra Urban committed 8 years ago
1bf3b25
d894d79
Add test
Ondra Urban committed 8 years ago
54c1f6b
Add regex factory
Ondra Urban committed 8 years ago
3778cc1
Improve pushData docs
Ondra Urban committed 8 years ago
ac33b36
Support XML extension in local KeyValueStore
Ondra Urban committed 8 years ago
3aa03f0
Lint comments
Ondra Urban committed 8 years ago
1b29678
Add unit tests for new functions; Handle edge cases
Ondra Urban committed 8 years ago
1298780
Improve chunkBySize loop
Ondra Urban committed 8 years ago
b0272d7
Update error message
Ondra Urban committed 8 years ago
04b938c
Refactor based on PR comments
Ondra Urban committed 8 years ago
b50c849
Update changelog
Ondra Urban committed 8 years ago
3bcca99
Implement PR comments
Ondra Urban committed 8 years ago
8c1fe72
Fix lint issue
Ondra Urban committed 8 years ago
eb324ca
Add tests
Ondra Urban committed 8 years ago
eabf2d3
b234d7d
34628bf
Remove random element from tests
Ondra Urban committed 8 years ago
b30828c
abce5a0
Rephrase docs
Ondra Urban committed 8 years ago
ffbd985
Add live view documentation
Ondra Urban committed 8 years ago
fb1936a