Skip to content

Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.

Commits on master

Fix flaky test
Ondra Urban committed 7 years ago
35df7ca
Improve deprecation message
Ondra Urban committed 7 years ago
8a55f02
Improve UX
Ondra Urban committed 7 years ago
566fdb2
Clean up old live view
Ondra Urban committed 7 years ago
92b3c1e
Finalize LiveViewServer tests
Ondra Urban committed 7 years ago
53299d9
Add trivial tests
Ondra Urban committed 7 years ago
13f6c7b
Merge branch 'develop' into chore/live-view
Ondra Urban committed 7 years ago
50fac5e
Refactor live view
Ondra Urban committed 7 years ago
df622c6
Added tunnel agent fix and improved code.
Petr Patek committed 7 years ago
5e93a7a
Update puppeteers_request_interception.js
Marek Trunkát committed 7 years ago
0e5a4e2
Improved requestExtended and requestLikeBrowser.
Petr Patek committed 7 years ago
68414a0
Updated docs
Jan Curn committed 7 years ago
c534bdd
Client upgrade and version bump
Marek Trunkat committed 7 years ago
01ffce1
Lintfix
Marek Trunkat committed 7 years ago
7ccdf7f
Finalizing request middlewares and tests
Marek Trunkat committed 7 years ago
26fb876
Adding tests for resource blocking
Marek Trunkat committed 7 years ago
bcea729
Implementing request middlewares
Marek Trunkat committed 7 years ago
ae79c6c
Merge develop, resolve conflicts
Ondra Urban committed 7 years ago
d532a78
Added response stream destroy.
Petr Patek committed 7 years ago
e966a7a
Improved tests.
Petr Patek committed 7 years ago
bfe0fe5
Fixed default values
Petr Patek committed 7 years ago
12304fb
Added abortFunction.
Petr Patek committed 7 years ago
c5e7708
Removed unnecessary "tmp" dev dependency
Jan Curn committed 7 years ago
7274bb3
Package versino bump and client update
Marek Trunkat committed 7 years ago
84b48db
fixed lint comments.
Petr Patek committed 7 years ago
b6b8f2f