Skip to content

Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.

Commits on master

feat: add `keepAlive` to crawler options (#1452)
Martin Adámek committed 4 years ago
084b6b2
chore(deps): update dependency playwright to v1.25.0
renovate[bot] committed 4 years ago
822ac1b
chore(deps): update npm to v8.17.0
renovate[bot] committed 4 years ago
b25cf06
feat: dom-crawler (#1451)
Szymon Marczak committed 4 years ago
ddb6cf0
feat: http-crawler (#1440)
Szymon Marczak committed 4 years ago
8c303f7
docs: add giscus discussions to docs pages
Martin Adámek committed 4 years ago
3f09eb1
chore(deps): update dependency turbo to v1.4.3 (#1448)
renovate[bot] committed 4 years ago
7e491e9
47591ff
docs: fix code description on homepage
Martin Adámek committed 4 years ago
a2c200c
docs: use root section headers only for API docs
Martin Adámek committed 4 years ago
c7648a8
docs: add root section headers to sidebar
Martin Adámek committed 4 years ago
721687d
51dd3c4
docs: fix linter
Martin Adámek committed 4 years ago
36afcb8
61e3964
feat: add `utils.playwright.blockRequests()` (#1447)
Jindřich Bär committed 4 years ago
1c6a8e4
chore: fix linter
Martin Adámek committed 4 years ago
f98f514
docs: fix typedoc rendering
Martin Adámek committed 4 years ago
50aa343
docs: use custom order of packages in API docs
Martin Adámek committed 4 years ago
d36ceaa
chore: allow consistent quoting of properties
Martin Adámek committed 4 years ago
5db9f9c
fix: sendRequest types (#1445)
Szymon Marczak committed 4 years ago
751563b
docs: update docusaurus
Martin Adámek committed 4 years ago
6b4c967
docs: add configuration guide (#1433)
Andrey Bykov committed 4 years ago
43cb957
chore: fix TS dockerfiles - do not copy everything
Martin Adámek committed 4 years ago
f60ce2f
f45758d