Skip to content

Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.

docs: update ConfigurationOptions description + more (#1437)

Andrey Bykovf45758dd1e22
diff --git a/docs/guides/apify_platform.mdx b/docs/guides/apify_platform.mdx
index 2670fcb0a..c1b02855d 100644
--- a/docs/guides/apify_platform.mdx
+++ b/docs/guides/apify_platform.mdx
@@ -113,7 +113,8 @@ For running Crawlee code as an actor on [Apify platform](https://apify.com/actor
- or wrap it into [`Actor.main()`](https://apify.github.io/apify-sdk-js/api/apify/class/Actor#main) function.
:::info NOTE
-Adding [`Actor.init()`](https://apify.github.io/apify-sdk-js/api/apify/class/Actor#init) to your code or wrapping it into [`Actor.main()`](https://apify.github.io/apify-sdk-js/api/apify/class/Actor#main) is the only important thing needed to run it on Apify platform as an actor. Some things mentioned below are either used for your convenience or for some specific use cases.
+- Adding [`Actor.init()`](https://apify.github.io/apify-sdk-js/api/apify/class/Actor#init) and [`Actor.exit()`](https://apify.github.io/apify-sdk-js/api/apify/class/Actor#exit) to your code are the only two important things needed to run it on Apify platform as an actor. `Actor.init()` is needed to initialize your actor (e.g. to set the correct storage implementation), while without `Actor.exit()` the process will simply never stop.
+- [`Actor.main()`](https://apify.github.io/apify-sdk-js/api/apify/class/Actor#main) is an alternative to `Actor.init()` and `Actor.exit()` as it calls both behind the scenes.
:::
Let's look at the `CheerioCrawler` example from the [Quick Start](../quick-start) guide:
diff --git a/packages/core/src/configuration.ts b/packages/core/src/configuration.ts
index fff7b47db..215e1e39a 100644
--- a/packages/core/src/configuration.ts
+++ b/packages/core/src/configuration.ts
@@ -11,24 +11,146 @@ import type { EventManager } from './events';
import { LocalEventManager } from './events';
export interface ConfigurationOptions {
+ /**
+ * Defines storage client to be used.
+ * @default {@link MemoryStorage}
+ */
storageClient?: StorageClient;
+
+ /**
+ * Defines the Event Manager to be used.
+ * @default {@link EventManager}
+ */
eventManager?: EventManager;
+
+ /**
+ * Could be used to adjust the storage client behavior
+ * e.g. {@link MemoryStorageOptions} could be used to adjust the {@link MemoryStorage} behavior.
+ */
storageClientOptions?: Dictionary;
+
+ /**
+ * Default dataset id.
+ *
+ * Alternative to `CRAWLEE_DEFAULT_DATASET_ID` environment variable.
+ * @default 'default'
+ */
defaultDatasetId?: string;
+
+ /**
+ * Defines whether to purge the default storage folders before starting the crawler run.
+ *
+ * Alternative to `CRAWLEE_PURGE_ON_START` environment variable.
+ * @default true
+ */
purgeOnStart?: boolean;
+
+ /**
+ * Default key-value store id.
+ *
+ * Alternative to `CRAWLEE_DEFAULT_KEY_VALUE_STORE_ID` environment variable.
+ * @default 'default'
+ */
defaultKeyValueStoreId?: string;
+
+ /**
+ * Default request queue id.
+ *
+ * Alternative to `CRAWLEE_DEFAULT_REQUEST_QUEUE_ID` environment variable.
+ * @default 'default'
+ */
defaultRequestQueueId?: string;
+
+ /**
+ * Sets the ratio, defining the maximum CPU usage.
+ * When the CPU usage is higher than the provided ratio, the CPU is considered overloaded.
+ * @default 0.95
+ */
maxUsedCpuRatio?: number;
+
+ /**
+ * Sets the ratio, defining the amount of system memory that could be used by the {@link AutoscaledPool}.
+ * When the memory usage is more than the provided ratio, the memory is considered overloaded.
+ *
+ * Alternative to `CRAWLEE_AVAILABLE_MEMORY_RATIO` environment variable.
+ * @default 0.25
+ */
availableMemoryRatio?: number;
+
+ /**
+ * Sets the amount of system memory in megabytes to be used by the {@link AutoscaledPool}.
+ * By default, the maximum memory is set to one quarter of total system memory.
+ *
+ * Alternative to `CRAWLEE_MEMORY_MBYTES` environment variable.
+ */
memoryMbytes?: number;
+
+ /**
+ * Defines the interval of emitting the `persistState` event.
+ *
+ * Alternative to `CRAWLEE_PERSIST_STATE_INTERVAL_MILLIS` environment variable.
+ * @default 60_000
+ */
persistStateIntervalMillis?: number;
+
+ /**
+ Defines the interval of emitting the `systemInfo` event.
+ @default 60_000
+ */
systemInfoIntervalMillis?: number;
+
+ /**
+ * Defines the default input key, i.e. the key that is used to get the crawler input value
+ * from the default {@link KeyValueStore} associated with the current crawler run.
+ *
+ * Alternative to `CRAWLEE_INPUT_KEY` environment variable.
+ * @default 'INPUT'
+ */
inputKey?: string;
+
+ /**
+ * Defines whether web browsers launched by Crawlee will run in the headless mode.
+ *
+ * Alternative to `CRAWLEE_HEADLESS` environment variable.
+ * @default true
+ */
headless?: boolean;
+
+ /**
+ * Defines whether to run X virtual framebuffer on the web browsers launched by Crawlee.
+ *
+ * Alternative to `CRAWLEE_XVFB` environment variable.
+ * @default false
+ */
xvfb?: boolean;
+
+ /**
+ * Defines a path to Chrome executable.
+ *
+ * Alternative to `CRAWLEE_CHROME_EXECUTABLE_PATH` environment variable.
+ */
chromeExecutablePath?: string;
+
+ /**
+ * Defines a path to default browser executable.
+ *
+ * Alternative to `CRAWLEE_DEFAULT_BROWSER_PATH` environment variable.
+ */
defaultBrowserPath?: string;
+
+ /**
+ * Defines whether to disable browser sandbox by adding `--no-sandbox` flag to `launchOptions`.
+ *
+ * Alternative to `CRAWLEE_DISABLE_BROWSER_SANDBOX` environment variable.
+ */
disableBrowserSandbox?: boolean;
+
+ /**
+ * Sets the log level to the given value.
+ *
+ * Alternative to `CRAWLEE_LOG_LEVEL`.
+ * @default 'INFO'
+ */
logLevel?: LogLevel | LogLevel[keyof LogLevel];
}