Skip to content

Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.

Commits on master

Version bump
Marek Trunkat committed 8 years ago
3e603c5
Feature/get proxy url opts (#118)
Marek Trunkát committed 8 years ago
1bc28aa
Version bump
Marek Trunkat committed 8 years ago
2bd4da2
Feature/dataset local fix (#117)
Marek Trunkát committed 8 years ago
994dfdf
Feature/call memory (#114)
Marek Trunkát committed 8 years ago
cace317
Fixing typos in local request queue
Marek Trunkat committed 8 years ago
9407859
Change server start API to allow tests.
Ondrej Urban committed 8 years ago
447f439
add documentation comments
Ondrej Urban committed 8 years ago
5ff39c6
add missing client commands, clean up code a little
Ondrej Urban committed 8 years ago
4766caa
implement client side websocket handling
Ondrej Urban committed 8 years ago
b9e29e5
Fixed broken link in comments
Jan Curn committed 8 years ago
7ef5887
refactoring and basic WebSocket implementation
Ondrej Urban committed 8 years ago
8b947b6
Merge branch 'master' into feature/live-view
Ondrej Urban committed 8 years ago
85c41b0
Version bump
Marek Trunkat committed 8 years ago
b67365c
564133c
Version bump (#109)
Jakub Drobník committed 8 years ago
e6fa3a4
37d313c
Removing unneeded import
Marek Trunkat committed 8 years ago
c0bac45
Removing unneeded import
Marek Trunkat committed 8 years ago
eab8a89
Version bump
Marek Trunkat committed 8 years ago
af433d7
Adding page.close() timeout to PuppeteerCrawler (#107)
Marek Trunkát committed 8 years ago
4e30146
Version bump
Marek Trunkat committed 8 years ago
b3beaa6
Adding map/forEach/reduce methods to dataset (#105)
Marek Trunkát committed 8 years ago
5f8590a