Skip to content

Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.

Commits on master

40cbefe
Fix typos
Ondra Urban committed 7 years ago
54cf199
Update check-user-agent.js
Ondra Urban committed 7 years ago
1b50159
Add tests for deprecated storage .delete() methods
Ondra Urban committed 7 years ago
b83e16b
Fix lint, build docs
Ondra Urban committed 7 years ago
4aa205c
Refactor storage .delete methods to .drop
Ondra Urban committed 7 years ago
384220a
builded docs.
Petr Patek committed 7 years ago
c1d8c7f
added process exit to end.
Petr Patek committed 7 years ago
26b0732
fixed yaml
Petr Patek committed 7 years ago
376da8e
fixed yaml
Petr Patek committed 7 years ago
9580bdd
fixed Yaml
Petr Patek committed 7 years ago
9f15177
test failed allow script to fail.
Petr Patek committed 7 years ago
906754a
test failed
Petr Patek committed 7 years ago
661e013
different stage used.
Petr Patek committed 7 years ago
aa236ff
2c820ea
Use shared ESLint config and resolve issues
Ondra Urban committed 7 years ago
6798066
Merge branch 'master' into chore/[email protected]
Ondra Urban committed 7 years ago
3a2a51d
Update changelog
Ondra Urban committed 7 years ago
bcb2872
Bump puppeteer to 1.19.0; Use pipes again
Ondra Urban committed 7 years ago
c6a4d4d
56e3a32
d89a061
Bump version [skip ci]
Ondra Urban committed 7 years ago
d2fb426
Add gotoTimeoutSecs and remove deprecated fields
Ondra Urban committed 7 years ago
8c246a6