Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.
TypeScript apifyautomationcrawlercrawlingheadlessheadless-chromejavascriptnodejsnpmplaywrightpuppeteerscraperscrapingtypescriptweb-crawlerweb-crawlingweb-scraping
Commits on master
chore: make the regexp option optional when using `requestsFromUrl` with RQ
a51abd4 Martin Adámek committed 3 years ago
feat(core): add `RequestQueue.addRequestsBatched()` that is non-blocking (#1996)
c85485d Martin Adámek committed 3 years ago
fix(basic-crawler): limit `internalTimeoutMillis` in addition to `requestHandlerTimeoutMillis` (#1981)
8122622 Sviatozar Petrenko committed 3 years ago
chore(deps): update dependency docusaurus-plugin-typedoc-api to v3 (#1848)
4ba42fd renovate[bot] committed 3 years ago
chore(deps): update dependency linkedom to ^0.15.0 (#1989)
b518e2f renovate[bot] committed 3 years ago
chore(deps): update dependency puppeteer to v20.8.2 (#1982)
ea5d44e renovate[bot] committed 3 years ago
fix(http-crawler): replace `IncomingMessage` with `PlainResponse` for context's `response` (#1973)
2a1cc7f Sviatozar Petrenko committed 3 years ago
chore(deps): update dependency unist-util-visit to v5 (#1963)
522a9b8 renovate[bot] committed 3 years ago
chore(deps): bump semver from 5.7.1 to 5.7.2 in /website (#1968)
0faf00d dependabot[bot] committed 3 years ago
test: fix assertion in the failing linkedom e2e test (#1972)
5826f75 Martin Adámek committed 3 years ago