Skip to content

Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.

Commits on master

Tuning up the docs (#589)
Ondra Urban committed 6 years ago
28f5795
Updated docs
Jan Curn committed 6 years ago
9fd2dc4
9cf05dc
e0fa831
Added a TODO to Apify.utils.htmlToText()
Jan Curn committed 6 years ago
5b300a4
Added a TODO to Apify.utils.htmlToText()
Jan Curn committed 6 years ago
b3fd463
Attempt to fix CheerioCrawlerOptions docs
Jan Curn committed 6 years ago
e32f155
Updated docs
Jan Curn committed 6 years ago
b9d136f
30349cc
Improved hide navigator hiding trick.
Petr Patek committed 6 years ago
0f9d563
Rename lenient to insecure, tone down doc warning
Ondra Urban committed 6 years ago
20c6400
Update test error message for Node 10
Ondra Urban committed 6 years ago
a010ced
Update changelog, bump version, build
Ondra Urban committed 6 years ago
efb1347
Add tests for lenient HTTP parser
Ondra Urban committed 6 years ago
be92f65
788df43
Fix build of docs not working on Node 12
Ondra Urban committed 6 years ago
1b0c774
Fix flaky memory info test
Ondra Urban committed 6 years ago
0609fbd
Remove useless integration tests
Ondra Urban committed 6 years ago
0b17da5
Remove unused CheerioCrawler tests
Ondra Urban committed 6 years ago
f9da492
Log exception on PuppeteerCrawler failed request
Ondra Urban committed 6 years ago
3644e19
Refactor CheerioCrawler default options
Ondra Urban committed 6 years ago
cea243f
Improved RequestList docs (#573)
Jan Čurn committed 6 years ago
f600875
15df545
Fix Session tests
Ondra Urban committed 6 years ago
2878a15
More updates to Session docs
Ondra Urban committed 6 years ago
14207b4
Build Session docs
Ondra Urban committed 6 years ago
037cf04
Improve Session docs; remove unused method
Ondra Urban committed 6 years ago
df739a8
Add session.setPuppeteerCookies test
Ondra Urban committed 6 years ago
6e7a953