Skip to content

Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.

Commits on master

Bump @types/puppeteer from 2.1.1 to 3.0.0 (#688)
dependabot-preview[bot] committed 6 years ago
4f41247
Fix flaky test
Ondra Urban committed 6 years ago
9db7d44
Bump puppeteer from 3.0.4 to 3.1.0 (#686)
dependabot-preview[bot] committed 6 years ago
b235ec6
Add survey link to docs homepage [skip ci]
Ondra Urban committed 6 years ago
451eb62
Bump @types/fs-extra from 8.1.1 to 9.0.0 (#683)
dependabot-preview[bot] committed 6 years ago
97aaaf7
Bump ts-jest from 25.5.1 to 26.0.0 (#684)
dependabot-preview[bot] committed 6 years ago
db58dc5
Proxy configuration (#681)
Ondra Urban committed 6 years ago
1f7bf23
fix tests (#680)
Ondra Urban committed 6 years ago
3f1c159
Remove Node 14 check on Travis until we can fix
Ondra Urban committed 6 years ago
8438e86
Update packages, migrate jest
Ondra Urban committed 6 years ago
5a9f5b4
Add Travis tests for Node 14
Ondra Urban committed 6 years ago
91ebcca
Remove remnants of HTML from docs [skip ci]
Ondra Urban committed 6 years ago
b12dc6a
Bump eslint from 6.8.0 to 7.0.0 (#676)
dependabot-preview[bot] committed 6 years ago
51cb733
958e829
Bump puppeteer from 3.0.2 to 3.0.4 (#673)
dependabot-preview[bot] committed 6 years ago
d6b9b0b
Bump version, update changelog
Ondra Urban committed 6 years ago
6473c28
Bump puppeteer from 3.0.1 to 3.0.2 (#670)
dependabot-preview[bot] committed 6 years ago
aceabb1
53ac1f6
Update http-request to fix upstream error
Ondra Urban committed 6 years ago
9803719
Build readme [skip ci]
Ondra Urban committed 6 years ago
d9a90e9
Update example in quick start
Ondra Urban committed 6 years ago
c9ef154
Fix hideconstructor in docs build [skip ci]
Ondra Urban committed 6 years ago
c43388b
Bump puppeteer from 3.0.0 to 3.0.1 (#666)
dependabot-preview[bot] committed 6 years ago
19b2ffd
cookie expires uses seconds not milliseconds (#659)
Paulo Cesar committed 6 years ago
ea9eb91
Bump puppeteer from 2.1.1 to 3.0.0 (#655)
dependabot-preview[bot] committed 6 years ago
30cbde8
Document that userId can be used in call (#654)
Lukáš Křivka committed 6 years ago
f2d6e5c
add namespaced loggers (#642)
Paulo Cesar committed 6 years ago
ffd44d5