Skip to content

Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.

Commits on master

refactor: tweak jsdom settings a bit 3
Martin Adámek committed 3 years ago
643f01b
refactor: tweak jsdom settings a bit 2
Martin Adámek committed 3 years ago
6b28e8e
refactor: tweak jsdom settings a bit
Martin Adámek committed 3 years ago
60872b6
dc60f0c
chore(deps): update patch/minor dependencies
renovate[bot] committed 3 years ago
111b7c5
2e81618
feat(jsdom): add `parseWithCheerio` context helper
Martin Adámek committed 3 years ago
c8f0796
chore(deps): update dependency minimatch to v8 (#1861)
renovate[bot] committed 3 years ago
4b8b77e
chore(deps): update dependency tldts to v6 (#1862)
renovate[bot] committed 3 years ago
1c753cd
chore(deps): update dependency puppeteer to v19.8.3
renovate[bot] committed 3 years ago
6ad6840
chore: do not log status message on start
Martin Adámek committed 3 years ago
8c9c3fa
b336a73
5d1df7a
chore(deps): update patch/minor dependencies
renovate[bot] committed 3 years ago
bee1593
docs: add robots.txt
Martin Adámek committed 3 years ago
c981a96
chore(deps): update patch/minor dependencies
renovate[bot] committed 3 years ago
4061677
chore(deps): update dependency playwright to v1.32.1
renovate[bot] committed 3 years ago
c3525c2
chore(deps): update dependency puppeteer to v19.8.0
renovate[bot] committed 3 years ago
2f9aa4e
chore(deps): update patch/minor dependencies to v2.4.0
renovate[bot] committed 3 years ago
23187a1
fix: status message - total requests (#1842)
Jindřich Bär committed 3 years ago
710f734
chore(deps): update dependency playwright to v1.32.0
renovate[bot] committed 3 years ago
4d21ab6
chore(deps): update dependency turbo to v1.8.5
renovate[bot] committed 3 years ago
f81db12
29929df