Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.
TypeScript apifyautomationcrawlercrawlingheadlessheadless-chromejavascriptnodejsnpmplaywrightpuppeteerscraperscrapingtypescriptweb-crawlerweb-crawlingweb-scraping
Commits on master
fix: handle multiple `BasicCrawler.stop()` calls correctly (#3324)
9c0580b Jindřich Bär committed 10 months ago
docs: update `beta` install example with version tags (#3312)
37b6602 Jindřich Bär committed 11 months ago
docs: fix the overlapping `LLMButtons` dropdown (#3307)
2229ab6 Richard Biroš committed 11 months ago
fix: certain redirect responses change request methods (#3296)
efac644 Jindřich Bär committed 11 months ago
chore(deps): update actions/cache action to v5 (#3303)
c647a8b renovate[bot] committed 11 months ago
feat: make `handleCloudflareChallenge` more configurable (#3247)
629daf8 Yash Hirgude committed 11 months ago
chore(deps): update dependency @types/express to v5 (#3290)
a8177d3 renovate[bot] committed 11 months ago
chore(deps): bump mdast-util-to-hast from 13.2.0 to 13.2.1 in /website (#3289)
5a901f7 dependabot[bot] committed 11 months ago
chore(deps): update dependency idcac-playwright to ^0.2.0 (#3279)
bf503fb renovate[bot] committed 11 months ago
chore: invalidate CloudFront cache when web changes (#3281)
cd5f43b Marko Kubrachenko committed 11 months ago
chore(CI): typecheck TS examples in `docs` automatically (#3278)
e20b4bf Jindřich Bär committed 11 months ago
chore: remove direct dependency on `idcac-playwright` (#3259)
6c76d32 Jindřich Bär committed 12 months ago
chore(deps): update actions/checkout action to v6 (#3260)
5b143b2 renovate[bot] committed 12 months ago
chore(deps): bump js-yaml from 3.14.1 to 3.14.2 in /website (#3258)
465c246 dependabot[bot] committed 12 months ago
chore(deps): update dependency react-lite-youtube-embed to v3 (#3249)
a6b2ca8 renovate[bot] committed 12 months ago
chore(e2e): fix faulty assertion in `storage-open-return-storage-object` e2e test (#3253)
e8ea3e2 Jindřich Bär committed 1 year ago
chore: mention the Apify $1M challenge in the announcement bar (#3244)
c732711 Jindřich Bär committed 1 year ago