Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.
TypeScript apifyautomationcrawlercrawlingheadlessheadless-chromejavascriptnodejsnpmplaywrightpuppeteerscraperscrapingtypescriptweb-crawlerweb-crawlingweb-scraping
chore: Remove default storage id config options (#4125)
Jan Buchar3a248f4c0b18
diff --git a/docs/guides/configuration.mdx b/docs/guides/configuration.mdxindex 94717bd5d..68cca7501 100644--- a/docs/guides/configuration.mdx+++ b/docs/guides/configuration.mdx@@ -76,18 +76,6 @@ can be changed significantly by setting or unsetting them.Defines the path to a local directory where <ApiLink to="core/class/KeyValueStore">`KeyValueStore`</ApiLink>, <ApiLink to="core/class/Dataset">`Dataset`</ApiLink>, and <ApiLink to="core/class/RequestQueue">`RequestQueue`</ApiLink> store their data. By default, it is set to `./storage`.-#### `CRAWLEE_DEFAULT_DATASET_ID`--The default dataset has ID `default`. Setting this environment variable overrides the default dataset ID with the provided value.--#### `CRAWLEE_DEFAULT_KEY_VALUE_STORE_ID`--The default key-value store has ID `default`. Setting this environment variable overrides the default key-value store ID with the provided value.--#### `CRAWLEE_DEFAULT_REQUEST_QUEUE_ID`--The default request queue has ID `default`. Setting this environment variable overrides the default request queue ID with the provided value.-#### `CRAWLEE_PURGE_ON_START`Run-scoped storage directories (the default storage and any opened with an `alias`) are purged by default; named storages are never purged. If set to `false` - local storage directories would not be purged automatically at the start of the crawler run or before opening of some storage explicitly (e.g. via `Dataset.open()`). Useful if we're trying e.g. to add more items to dataset with each next run (and keep the previously saved/scraped items).diff --git a/docs/guides/request_storage.mdx b/docs/guides/request_storage.mdxindex 702d60ab7..803450c1b 100644--- a/docs/guides/request_storage.mdx+++ b/docs/guides/request_storage.mdx@@ -34,7 +34,7 @@ By default, the request queue is managed by the <ApiLink to="fs-storage/class/Fi:::note-`{QUEUE_ID}` is the name or ID of the request queue. The default queue has ID `default`, unless we override it by setting the `CRAWLEE_DEFAULT_REQUEST_QUEUE_ID` environment variable.+`{QUEUE_ID}` is the name or ID of the request queue. The default queue has ID `default`.:::diff --git a/docs/guides/result_storage.mdx b/docs/guides/result_storage.mdxindex 67aa83416..c6f9b5379 100644--- a/docs/guides/result_storage.mdx+++ b/docs/guides/result_storage.mdx@@ -26,7 +26,7 @@ The data is stored in the directory specified by the `CRAWLEE_STORAGE_DIR` envir:::note-`{STORE_ID}` is the name or the ID of the key-value store. The default key-value store has ID `default`, unless we override it by setting the `CRAWLEE_DEFAULT_KEY_VALUE_STORE_ID` environment variable. The `{KEY}` is the key of the record and `{EXT}` corresponds to the MIME content type of the data value.+`{STORE_ID}` is the name or the ID of the key-value store. The default key-value store has ID `default`. The `{KEY}` is the key of the record and `{EXT}` corresponds to the MIME content type of the data value.:::@@ -76,7 +76,7 @@ The data is stored in the directory specified by the `CRAWLEE_STORAGE_DIR` envir:::note-`{DATASET_ID}` is the name or the ID of the dataset. The default dataset has ID `default`, unless we override it by setting the `CRAWLEE_DEFAULT_DATASET_ID` environment variable. Each dataset item is stored as a separate JSON file, where `{INDEX}` is a zero-based index of the item in the dataset.+`{DATASET_ID}` is the name or the ID of the dataset. The default dataset has ID `default`. Each dataset item is stored as a separate JSON file, where `{INDEX}` is a zero-based index of the item in the dataset.:::diff --git a/docs/public-api/crawlee-core.api.md b/docs/public-api/crawlee-core.api.mdindex 1d598e816..82900a74e 100644--- a/docs/public-api/crawlee-core.api.md+++ b/docs/public-api/crawlee-core.api.md@@ -146,10 +146,7 @@ export type ConfigurationOptions = ConfigurationInput;// @public (undocumented)export const crawleeConfigFields: {- defaultDatasetId: ConfigField<z.ZodDefault<z.ZodString>>;purgeOnStart: ConfigField<z.ZodDefault<z.ZodPreprocess<z.ZodBoolean, unknown>>>;- defaultKeyValueStoreId: ConfigField<z.ZodDefault<z.ZodString>>;- defaultRequestQueueId: ConfigField<z.ZodDefault<z.ZodString>>;maxUsedCpuRatio: ConfigField<z.ZodDefault<z.ZodPreprocess<z.ZodNumber, unknown>>>;availableMemoryRatio: ConfigField<z.ZodDefault<z.ZodPreprocess<z.ZodNumber, unknown>>>;memoryMbytes: ConfigField<z.ZodOptional<z.ZodPreprocess<z.ZodNumber, unknown>>>;diff --git a/docs/upgrading/upgrading_v4.md b/docs/upgrading/upgrading_v4.mdindex 86a7d869b..38e4f2470 100644--- a/docs/upgrading/upgrading_v4.md+++ b/docs/upgrading/upgrading_v4.md@@ -2224,6 +2224,7 @@ The full list of removed exports and members, for ctrl-F purposes. Where a repla- `context.blockResources` and `context.cacheResponses` — no longer attached to the crawling context. The functionality is still available as deprecated functions, accessible both via the `puppeteerUtils` namespace (`puppeteerUtils.blockResources`, `puppeteerUtils.cacheResponses`) and as top-level exports from `@crawlee/puppeteer` (`import { blockResources, cacheResponses } from '@crawlee/puppeteer'`). Unlike the old context helpers, these take an explicit `page` argument — e.g. `await blockResources(page)`. Both are `@deprecated` and will be removed in a future release, so migrate away from them.- `context.closeCookieModals`, `playwrightUtils.closeCookieModals` and `puppeteerUtils.closeCookieModals` — removed along with the optional `idcac-playwright` peer dependency (see [Crawling context no longer includes `closeCookieModals`](#crawling-context-no-longer-includes-closecookiemodals) and the [cookie modals guide](../guides/cookie-modals))- `Configuration.systemInfoV2` / `CRAWLEE_SYSTEM_INFO_V2` environment variable — the v2 behavior is now the default (see [Available resource detection](#available-resource-detection))+- `Configuration.defaultDatasetId` / `defaultKeyValueStoreId` / `defaultRequestQueueId` and their `CRAWLEE_DEFAULT_*_ID` environment variables — the default storage is addressed by a reserved alias, not by a configurable ID. Open a storage by name if you need a specific one.- `checkAndSerialize` and `chunkBySize` functions (from `@crawlee/core`) — value (de)serialization now lives in the `KeyValueStore` frontend; use `serializeValue` / `parseValue` (see [`maybeStringify` is removed](#maybestringify-is-removed))- `BASIC_CRAWLER_TIMEOUT_BUFFER_SECS` constant (from `@crawlee/basic`) — was an internal timeout buffer, no longer exported- `HttpResponse`, `HttpResponseWithoutBody`, `StreamingHttpResponse`, `ResponseTypes`, `BaseHttpResponseData`, `SimpleHeaders`, `processHttpRequestOptions`, and `GotScrapingHttpClient` (from `@crawlee/core`) — the HTTP client surface moved to `@crawlee/http-client` / `@crawlee/got-scraping-client` (see [HTTP client packages and `BaseHttpClient` reshaped](#http-client-packages-and-basehttpclient-reshaped))diff --git a/packages/core/src/configuration.ts b/packages/core/src/configuration.tsindex d186d7521..7ac3832ec 100644--- a/packages/core/src/configuration.ts+++ b/packages/core/src/configuration.ts@@ -52,14 +52,8 @@ const logLevelSchema = z.preprocess((val) => {// --- Crawlee config field definitions ---export const crawleeConfigFields = {- /** @default 'default' */- defaultDatasetId: field(z.string().default('default'), 'CRAWLEE_DEFAULT_DATASET_ID'),/** @default true */purgeOnStart: field(coerceBoolean.default(true), 'CRAWLEE_PURGE_ON_START'),- /** @default 'default' */- defaultKeyValueStoreId: field(z.string().default('default'), 'CRAWLEE_DEFAULT_KEY_VALUE_STORE_ID'),- /** @default 'default' */- defaultRequestQueueId: field(z.string().default('default'), 'CRAWLEE_DEFAULT_REQUEST_QUEUE_ID'),/** @default 0.95 */maxUsedCpuRatio: field(coerceNumber.default(0.95)),/** @default 0.25 */@@ -156,9 +150,6 @@ export interface Configuration extends ResolvedConfigValues {}* `memoryMbytes` | `CRAWLEE_MEMORY_MBYTES` | -* `logLevel` | `CRAWLEE_LOG_LEVEL` | -* `headless` | `CRAWLEE_HEADLESS` | `true`- * `defaultDatasetId` | `CRAWLEE_DEFAULT_DATASET_ID` | `'default'`- * `defaultKeyValueStoreId` | `CRAWLEE_DEFAULT_KEY_VALUE_STORE_ID` | `'default'`- * `defaultRequestQueueId` | `CRAWLEE_DEFAULT_REQUEST_QUEUE_ID` | `'default'`* `persistStateIntervalMillis` | `CRAWLEE_PERSIST_STATE_INTERVAL_MILLIS` | `60_000`* `internalTimeoutMillis` | `CRAWLEE_INTERNAL_TIMEOUT` | -* `purgeOnStart` | `CRAWLEE_PURGE_ON_START` | `true`diff --git a/packages/core/src/storages/key_value_store.ts b/packages/core/src/storages/key_value_store.tsindex 8e3520172..f4f228ffa 100644--- a/packages/core/src/storages/key_value_store.ts+++ b/packages/core/src/storages/key_value_store.ts@@ -73,8 +73,7 @@ const openOptionsSchema = z.strictObject({* ```* {CRAWLEE_STORAGE_DIR}/key_value_stores/{STORE_ID}/{INDEX}.{EXT}* ```- * Note that `{STORE_ID}` is the name or ID of the key-value store. The default key-value store has ID: `default`,- * unless you override it by setting the `CRAWLEE_DEFAULT_KEY_VALUE_STORE_ID` environment variable.+ * Note that `{STORE_ID}` is the name or ID of the key-value store. The default key-value store has ID: `default`.* The `{KEY}` is the key of the record and `{EXT}` corresponds to the MIME content type of the data value.** **Example usage:**diff --git a/packages/core/test/core/configuration.test.ts b/packages/core/test/core/configuration.test.tsindex a382e8a41..fb2cce96d 100644--- a/packages/core/test/core/configuration.test.ts+++ b/packages/core/test/core/configuration.test.ts@@ -35,9 +35,6 @@ describe('Configuration', () => {describe('defaults', () => {it('returns schema defaults when nothing is set', () => {const config = new Configuration();- expect(config.defaultDatasetId).toBe('default');- expect(config.defaultKeyValueStoreId).toBe('default');- expect(config.defaultRequestQueueId).toBe('default');expect(config.inputKey).toBe('INPUT');expect(config.headless).toBe(true);expect(config.xvfb).toBe(false);@@ -85,9 +82,9 @@ describe('Configuration', () => {});it('constructor options override env vars for string fields', () => {- setEnv('CRAWLEE_DEFAULT_DATASET_ID', 'from-env');- const config = new Configuration({ defaultDatasetId: 'from-constructor' });- expect(config.defaultDatasetId).toBe('from-constructor');+ setEnv('CRAWLEE_INPUT_KEY', 'from-env');+ const config = new Configuration({ inputKey: 'from-constructor' });+ expect(config.inputKey).toBe('from-constructor');});it('constructor options override env vars for number fields', () => {@@ -151,9 +148,9 @@ describe('Configuration', () => {setEnv('CRAWLEE_PERSIST_STATE_INTERVAL_MILLIS', '');expect(new Configuration().persistStateIntervalMillis).toBe(60_000);- // Empty string env var falls through to default ('default'), not coerced to ''- setEnv('CRAWLEE_DEFAULT_DATASET_ID', '');- expect(new Configuration().defaultDatasetId).toBe('default');+ // Empty string env var falls through to default ('INPUT'), not coerced to ''+ setEnv('CRAWLEE_INPUT_KEY', '');+ expect(new Configuration().inputKey).toBe('INPUT');// Optional fields with no default stay undefinedsetEnv('CRAWLEE_MEMORY_MBYTES', '');@@ -188,11 +185,11 @@ describe('Configuration', () => {it('exposes resolved values as instance properties', () => {const config = new Configuration({headless: false,- defaultDatasetId: 'my-dataset',+ inputKey: 'my-input-key',persistStateIntervalMillis: 5_000,});expect(config.headless).toBe(false);- expect(config.defaultDatasetId).toBe('my-dataset');+ expect(config.inputKey).toBe('my-input-key');expect(config.persistStateIntervalMillis).toBe(5_000);});});@@ -249,19 +246,19 @@ describe('Configuration', () => {});it('loads values from crawlee.json', () => {- writeFileSync(crawleeJsonPath, JSON.stringify({ defaultDatasetId: 'from-file' }));+ writeFileSync(crawleeJsonPath, JSON.stringify({ inputKey: 'from-file' }));fileCreated = true;const config = new Configuration();- expect(config.defaultDatasetId).toBe('from-file');+ expect(config.inputKey).toBe('from-file');});it('constructor options override crawlee.json', () => {- writeFileSync(crawleeJsonPath, JSON.stringify({ defaultDatasetId: 'from-file' }));+ writeFileSync(crawleeJsonPath, JSON.stringify({ inputKey: 'from-file' }));fileCreated = true;- const config = new Configuration({ defaultDatasetId: 'from-constructor' });- expect(config.defaultDatasetId).toBe('from-constructor');+ const config = new Configuration({ inputKey: 'from-constructor' });+ expect(config.inputKey).toBe('from-constructor');});it('env vars override crawlee.json', () => {@@ -286,7 +283,7 @@ describe('Configuration', () => {it('handles missing crawlee.json gracefully', () => {// No file created — should fall through to defaultsconst config = new Configuration();- expect(config.defaultDatasetId).toBe('default');+ expect(config.inputKey).toBe('INPUT');});it('handles malformed crawlee.json gracefully', () => {@@ -294,7 +291,7 @@ describe('Configuration', () => {fileCreated = true;const config = new Configuration();- expect(config.defaultDatasetId).toBe('default');+ expect(config.inputKey).toBe('INPUT');});});