Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.
TypeScript apifyautomationcrawlercrawlingheadlessheadless-chromejavascriptnodejsnpmplaywrightpuppeteerscraperscrapingtypescriptweb-crawlerweb-crawlingweb-scraping
Commits on master
fix: using GET for candidate checks - sitemaps (#3464)
f82eb04 nikitachapovskii-dev committed 8 months ago
chore(e2e): restrict git ceiling directory for actor push (#3467)
e976265 Richard Solar committed 8 months ago
chore(deps): bump @modelcontextprotocol/sdk from 1.25.2 to 1.26.0 (#3394)
6f3cd87 dependabot[bot] committed 8 months ago
chore(deps): bump minimatch from 9.0.5 to 9.0.7 (#3468)
1ee8b3b dependabot[bot] committed 8 months ago
chore(deps): bump @hono/node-server from 1.19.7 to 1.19.10 (#3466)
5d14a96 dependabot[bot] committed 8 months ago
fix: log warning when Sitemap.load returns empty sitemap due to loading error (#3462)
2cf9c1c Copilot committed 8 months ago
fix: add network timeouts to discoverValidSitemaps to prevent indefinite hangs (#3429)
ee90835 不做了睡大觉 committed 8 months ago
chore(deps): batch merge dependabot version bumps (#3449)
adfa13b Jindřich Bär committed 8 months ago
fix: prevent v3 sitemap discovery init hangs in discoverValidSitemaps (#3434)
6c04f92 nikitachapovskii-dev committed 8 months ago
fix: bind `Browser.pages()` correctly in `PuppeteerPlugin` (#3439)
c3a4b3b Jindřich Bär committed 8 months ago
fix: parse proxy URLs correctly when using `useIncognitoPages` (#3433)
db2bb68 Copilot committed 8 months ago
docs: Update clipboard handling for Safari compatibility in LLMButtons component (#3421)
1f31779 Richard Biroš committed 9 months ago
feat: add `ignoreProxyCertificate` option for the internal `proxy-chain` instance (#3418)
02eec66 Jindřich Bär committed 9 months ago
chore(deps): bump qs from 6.14.1 to 6.14.2 in /website (#3415)
070c258 dependabot[bot] committed 9 months ago
fix: change sequential approach to parallel for Iterator first page (#3402)
adf3dae Richard Solar committed 9 months ago
chore(deps): bump axios from 1.13.2 to 1.13.5 in /website (#3403)
136d368 dependabot[bot] committed 9 months ago
docs: mention new `impit` fingerprints in the docs (#3400)
bfb861c Jindřich Bär committed 9 months ago
chore(release): update internal dependencies [skip ci]
e645174 Apify Release Bot committed 9 months ago
chore(deps): update dependency @stylistic/eslint-plugin to v5 (#3385)
ba60f04 renovate[bot] committed 9 months ago