Skip to content

Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.

Commits on master

Fix lint issues and misnamings
Ondra Urban committed 8 years ago
04e083a
Merge branch 'develop' into bug/invalid-url-regex
Ondra Urban committed 8 years ago
bacf7fd
Add tests for invalid URLs. Update RegExp
Ondra Urban committed 8 years ago
242160b
7a7da4f
Version bump
Marek Trunkat committed 8 years ago
bd5882d
Fix default value when null in extractUrls()
Ondra Urban committed 8 years ago
ab351e8
bump version, update CHANGELOG
Ondra Urban committed 8 years ago
1de5c51
Merge branch 'develop' into bug/invalid-url-regex
Ondra Urban committed 8 years ago
233d335
ec96632
Unnecessary comments deleted
Vratislav Bartonicek committed 8 years ago
b2d1a5c
ESLint syntax fixes
Vratislav Bartonicek committed 8 years ago
aada6c3
Adding .editorconfig and 'lint:fix' command
Marek Trunkat committed 8 years ago
2a11880
Live view - variables replaced by constants
Vratislav Bartonicek committed 8 years ago
03dbe9a
Live view styles
Vratislav Bartonicek committed 8 years ago
4a7ffbe
Writing README.md
Jan Curn committed 8 years ago
b0d3a0a
Live view - frontend structure and styles
Vratislav Bartonicek committed 8 years ago
63733f7
Writing README.md
Jan Curn committed 8 years ago
bdda696
Writing README.md
Jan Curn committed 8 years ago
b4f2f04
Version bump
Marek Trunkat committed 8 years ago
e8edae2
Fixing liveview with HTTP
Marek Trunkat committed 8 years ago
c25de1c
Merging develop
Marek Trunkat committed 8 years ago
9bc5764
Updating puppeteer
Marek Trunkat committed 8 years ago
9c8ef7d
cd797ad
Fixed unit tests for PuppeteerCrawler
Jan Curn committed 8 years ago
f907a6a
Fixed unit tests for PuppeteerCrawler
Jan Curn committed 8 years ago
ac70d65
Updated CHANGELOG.md
Jan Curn committed 8 years ago
849926a
bc1b1b3