Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.
TypeScript apifyautomationcrawlercrawlingheadlessheadless-chromejavascriptnodejsnpmplaywrightpuppeteerscraperscrapingtypescriptweb-crawlerweb-crawlingweb-scraping
Merge master, resolve conflicts
Ondra Urbandc4ed7c21e3a
diff --cc CHANGELOG.mdindex 72ad1a781,fb3c964b9..6679b955e--- a/CHANGELOG.md+++ b/CHANGELOG.md@@@ -1,10 -1,11 +1,14 @@@- xxx+ NEXT====================- - Bugfix in BasicCrawler: async calls in `isFinishedFunction` were not awaited- - Better logging of memory overload errors- - Better error message in `Apify.call()`+ - **BREAKING CHANGE** Removed support for Web Driver (Selenium) since no further updates are planned.+ If you wish to continue using Web Driver, please stay on Apify SDK version ^0.14.15+- **BREAKING CHANGE**: `Dataset.getData()` throws an error if user provides an unsupported option+ when using local disk storage.+ - Improve logging of memory overload errors.++- Improve error message in `Apify.call()`.+ - Fix multiple log lines appearing when a crawler was about to finish.+ - Add `Apify.utils.puppeteer.enqueueLinksByClickingElements()` function which enables you+ to add requests to the queue from pure JavaScript navigations, form submissions etc.0.14.15 / 2019-05-31====================diff --cc src/dataset.jsindex 076ba0977,b43b86445..e2bb2a67c--- a/src/dataset.js+++ b/src/dataset.js@@@ -199,13 -199,18 +199,23 @@@ export class Dataset}/**- * Returns items in the dataset based on the provided options.+ * Returns items in the dataset based on the provided parameters. The returned object+ * has the following structure:+ *+ * ```javascript+ * {+ * items, // Array|String|Buffer based on chosen format parameter.+ * total, // Number+ * limit, // Number+ * offset, // Number+ * }+ * ```*+ * **NOTE**: If using dataset with local disk storage, the `format` option must be `json` and+ * the following options are not supported:+ * `unwind`, `disableBodyParser`, `attachment`, `bom` and `simplified`.+ * If you try to use them, you will receive an error.+ ** @param {Object} [options] All `getData()` parameters are passed* via an options object with the following keys:* @param {String} [options.format='json']