Skip to content

Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.

Commits on master

Refactor useChrome test, remove Selenium support
Ondra Urban committed 7 years ago
3724e17
Upgrade ubuntu
Ondra Urban committed 7 years ago
1544810
More precise tagline
Jan Curn committed 7 years ago
4373a8d
Better logging of memory overload errors
Jan Curn committed 7 years ago
81c9e6f
Better logging of memory overload errors
Jan Curn committed 7 years ago
21cb53e
620321f
Fix lint issue
Ondra Urban committed 7 years ago
3117419
b847a2b
Improve error handling in blockRequests()
Ondra Urban committed 7 years ago
9883f28
Improve docs in CheerioCrawler requestOptions.
Ondra Urban committed 7 years ago
c65b853
fixed ectractUrls docs
vdrmota committed 7 years ago
21061c0
a35584c
Build docs
Ondra Urban committed 7 years ago
dfb346a
8336ea4
299b7f5
Docs update
Marek Trunkat committed 7 years ago
4219fe8
Version bump
Marek Trunkat committed 7 years ago
fcc63b8
5144e40
RequestQueue improvements, more unit test
Jan Curn committed 7 years ago
3c90f1f
Update live view readme docs
Ondra Urban committed 7 years ago
30b4b00
Merge branch 'develop' into fix/request-queue
Jan Curn committed 7 years ago
dae3fb1
Update changelog, bump version
Ondra Urban committed 7 years ago
7e287a8
Fixed link in docs
Jan Curn committed 7 years ago
7627004
RequestQueue - fixes from PR review
Jan Curn committed 7 years ago
e624b10
Fix bugs in LiveView and do some type experiments
Ondra Urban committed 7 years ago
0dadce6
e72e8db