Skip to content

Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.

Better error messages in Apify.call(), Dataset.getData() throws on unsupported options

Jan Curn764d495849c8
diff --git a/CHANGELOG.md b/CHANGELOG.md
index c89c7a821..72ad1a781 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -2,6 +2,9 @@ xxx
====================
- Bugfix in BasicCrawler: async calls in `isFinishedFunction` were not awaited
- Better logging of memory overload errors
+- Better error message in `Apify.call()`
+- **BREAKING CHANGE**: `Dataset.getData()` throws an error if user provides an unsupported option
+ when using local disk storage.
0.14.15 / 2019-05-31
====================
diff --git a/src/actor.js b/src/actor.js
index 7539b9bfe..b76a731cf 100644
--- a/src/actor.js
+++ b/src/actor.js
@@ -67,8 +67,8 @@ const waitForRunToFinish = async ({ actId, runId, token, waitSecs, taskId }) =>
&& status !== ACT_JOB_STATUSES.READY
) {
const message = taskId
- ? `The actor task ${taskId} invoked by Apify.call() did not succeed`
- : `The actor ${actId} invoked by Apify.call() did not succeed`;
+ ? `The actor task ${taskId} invoked by Apify.call() did not succeed. For details, see https://my.apify.com/view/runs/${runId}`
+ : `The actor ${actId} invoked by Apify.call() did not succeed. For details, see https://my.apify.com/view/runs/${runId}`;
throw new ApifyCallError(updatedRun, message);
}
diff --git a/src/dataset.js b/src/dataset.js
index ec01c3dd2..d41adf0aa 100644
--- a/src/dataset.js
+++ b/src/dataset.js
@@ -199,7 +199,11 @@ export class Dataset {
}
/**
- * Returns items in the dataset based on the provided parameters.
+ * Returns items in the dataset based on the provided options.
+ *
+ * **NOTE**: If using dataset with local disk storage, the following options are not supported:
+ * `unwind`, `disableBodyParser`, `attachment`, `bom` and `simplified`, and
+ * the `format` option must be `json`. If you try to use them, you will receive an error.
*
* @param {Object} [options] All `getData()` parameters are passed
* via an options object with the following keys:
@@ -480,6 +484,14 @@ export class DatasetLocal {
checkParamOrThrow(opts.limit, 'opts.limit', 'Maybe Number');
checkParamOrThrow(opts.offset, 'opts.offset', 'Maybe Number');
+ if (opts.format && opts.format !== 'json') {
+ throw new Error(`Datasets with local disk storage only support the "json" format (was "${opts.format}")`);
+ }
+ if (opts.unwind || opts.disableBodyParser || opts.attachment || opts.bom || opts.simplified) {
+ // eslint-disable-next-line max-len
+ throw new Error('Datasets with local disk storage do not support the following options: unwind, disableBodyParser, attachment, bom, simplified');
+ }
+
if (!opts.limit) opts.limit = LOCAL_GET_ITEMS_DEFAULT_LIMIT;
if (!opts.offset) opts.offset = 0;