Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.
TypeScript apifyautomationcrawlercrawlingheadlessheadless-chromejavascriptnodejsnpmplaywrightpuppeteerscraperscrapingtypescriptweb-crawlerweb-crawlingweb-scraping
Commits on master
chore(release): update internal dependencies [skip ci]
49c115e Apify Release Bot committed 3 months ago
fix(core): keep crawler enqueue limits when options carry explicit undefined (#3927)
5f1b0ab Martin Adámek committed 3 months ago
chore(deps): bump brace-expansion from 1.1.16 to 1.1.18 in /docs (#3960)
7dba89e dependabot[bot] committed 3 months ago
chore(deps): bump brace-expansion from 1.1.16 to 1.1.18 in /website (#3959)
7245307 dependabot[bot] committed 3 months ago
chore(deps): bump fast-uri from 3.1.4 to 3.1.5 in /website (#3958)
7f4df88 dependabot[bot] committed 3 months ago
chore(deps): bump hono from 4.12.31 to 4.12.34 (#3955)
999a353 dependabot[bot] committed 3 months ago
chore(deps): bump fast-uri from 3.1.4 to 3.1.5 (#3956)
26e5e4a dependabot[bot] committed 3 months ago
chore(deps): bump undici from 6.27.0 to 6.28.0 (#3957)
5661333 dependabot[bot] committed 3 months ago
chore(deps): bump ip-address from 10.2.0 to 10.4.0 (#3954)
b62ef01 dependabot[bot] committed 3 months ago
chore(deps): update dependency camoufox-js to ^0.12.0 (#3950)
5401ab9 renovate[bot] committed 3 months ago
chore(deps): bump brace-expansion from 1.1.16 to 1.1.18 in /website/versioned_docs/version-3.17 (#3948)
9b9cd54 dependabot[bot] committed 3 months ago
chore(deps): update patch/minor dependencies to v1.62.1
ce2e6d7 renovate[bot] committed 3 months ago
test(e2e): crawl local fixtures instead of the live demo store (#3922)
5de7162 Martin Adámek committed 3 months ago
fix(impit-client): honor redirect handler for session cookie parity (#3938)
9371301 Felix-Ayush committed 3 months ago
fix: respect falsy transformRequestFunction return in context enqueueLinks (#3925)
6cb4fc3 Martin Adámek committed 3 months ago
chore(deps): bump brace-expansion in website lockfile to resolve security alert (#3933)
9b38e9c Martin Adámek committed 3 months ago
fix(core): ignore explicitly undefined baseUrl in enqueueLinks helpers (#3924)
a58a9c1 Martin Adámek committed 3 months ago
fix(core): keep same-domain filtering after an off-domain redirect (#3923)
9b312ef Martin Adámek committed 3 months ago
feat(core): infer default-route handler userData from its schema (#3919)
37d0ccb Martin Adámek committed 3 months ago
chore(deps): refresh lockfiles to resolve dependabot alerts (#3913)
7013001 Jindřich Bär committed 3 months ago
chore(deps): update patch/minor dependencies to v1.62.0
d7d5d9b renovate[bot] committed 3 months ago
fix(sitemap): recognize sitemap_index as a valid input sitemap url (#3909)
eacab5e Luigi Ruocco committed 3 months ago
chore(deps): update dependency tar to v7.5.19 [security]
cb63a2f renovate[bot] committed 3 months ago
chore(deps): bump hono from 4.12.25 to 4.12.31 (#3894)
9dbb405 dependabot[bot] committed 3 months ago
chore: allow explicit `id` for Statistics persistence (#3879)
45fa7e5 nikitachapovskii-dev committed 4 months ago