Skip to content

Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.

Improved Dataset.getData() docs

Jan Curn42e20df84d00
diff --git a/src/dataset.js b/src/dataset.js
index 9ecce9ddd..a339bd59a 100644
--- a/src/dataset.js
+++ b/src/dataset.js
@@ -251,13 +251,28 @@ export class Dataset {
* By default, the element name is `page` or `result`, depending on the value of the `simplified` option.
* @param {Boolean} [options.skipHeaderRow=false]
* If set to `true` then header row in CSV format is skipped.
+ * @param {Boolean} [options.clean=false]
+ * If `true` then the function returns only non-empty items and skips hidden fields (i.e. fields starting with `#` character).
+ * Note that the `clean` parameter is a shortcut for `skipHidden: true` and `skipEmpty: true` options.
+ * @param {Boolean} [options.skipHidden=false]
+ * If `true` then the function doesn't return hidden fields (fields starting with "#" character).
+ * @param {Boolean} [options.skipEmpty=false]
+ * If `true` then the function doesn't return empty items.
+ * Note that in this case the returned number of items might be lower than limit parameter and pagination must be done using the `limit` value.
* @param {Boolean} [options.simplified]
- * If set to `true` then function applies the `fields: ['url','pageFunctionResult','errorInfo']` and `unwind: 'pageFunctionResult'` options.
+ * If `true` then function applies the `fields: ['url','pageFunctionResult','errorInfo']` and `unwind: 'pageFunctionResult'` options.
+ * This feature is used to emulate simplified results provided by Apify API version 1 used for
+ * the legacy Apify Crawler and it's not recommended to use it in new integrations.
* @param {Boolean} [options.skipFailedPages]
- * If set to `true` then all the items with errorInfo property will be skipped from the output.
+ * If `true` then, the all the items with errorInfo property will be skipped from the output.
+ * This feature is here to emulate functionality of Apify API version 1 used for
+ * the legacy Apify Crawler product and it's not recommended to use it in new integrations.
* @return {Promise<Object>}
*/
getData(options = {}) {
+ // TODO (JC): Do we really need this function? It only works with API but not locally,
+ // and it's just 1:1 copy of what apify-client provides, and returns { items } which can
+ // be a Buffer ... it doesn't really make much sense
const { datasetId } = this;
const params = Object.assign({ datasetId }, options);