Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.
TypeScript apifyautomationcrawlercrawlingheadlessheadless-chromejavascriptnodejsnpmplaywrightpuppeteerscraperscrapingtypescriptweb-crawlerweb-crawlingweb-scraping
Update changelog, bump version
Ondra Urbanec2e01db522a
diff --git a/CHANGELOG.md b/CHANGELOG.mdindex 79311a48e..433e0d1c1 100644--- a/CHANGELOG.md+++ b/CHANGELOG.md@@ -1,5 +1,9 @@-xxx+0.9.0 / 2018-11-07===================+- Added the ability of `CheerioCrawler` to request and download only `text/html` responses.+- Added a workaround for a long standing `tunnel-agent` package error to `CheerioCrawler`.+- Added `request.doNotRetry()` function to prevent further retries of a `request`.+- Deprecated `request.ignoreErrors` option. Use `request.doNotRetry`.- Fixed `Apify.utils.puppeteer.enqueueLinks` to allow `null` value for `pseudoUrls` param- Fixed `RequestQueue.addRequest()` to gracefully handle invalid URLs- Renamed `RequestOperationInfo` to `QueueOperationInfo`diff --git a/package.json b/package.jsonindex 13a6f945c..9c0428698 100644--- a/package.json+++ b/package.json@@ -1,6 +1,6 @@{"name": "apify",- "version": "0.8.19",+ "version": "0.9.0","description": "The scalable web crawling and scraping library for JavaScript. Enables development of data extraction and web automation jobs (not only) with headless Chrome and Puppeteer.","engines": {"node": ">=8.0.0"@@ -22,7 +22,8 @@},"contributors": ["Jan Curn <[email protected]>",- "Marek Trunkat <[email protected]>"+ "Marek Trunkat <[email protected]>",+ "Ondra Urban <[email protected]>"],"license": "Apache-2.0","repository": {@@ -50,7 +51,7 @@"dependencies": {"@apify/ps-tree": "^1.1.2","apify-client": "^0.4.0",- "apify-shared": "^0.1.7",+ "apify-shared": "^0.1.9","bluebird": "^3.5.0","cheerio": "^1.0.0-rc.2","content-type": "^1.0.3",