Skip to content

Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.

fix: rename default storage folder to just `storage` (#1403)

Martin Adámekc253985e534b
diff --git a/.gitignore b/.gitignore
index 7b8b56bfd..6ccbf77e2 100644
--- a/.gitignore
+++ b/.gitignore
@@ -23,4 +23,5 @@ docs/typedefs
tsconfig.tsbuildinfo
apify_storage
crawlee_storage
+storage
.turbo
diff --git a/MIGRATIONS.md b/MIGRATIONS.md
index cbce19370..022e1c175 100644
--- a/MIGRATIONS.md
+++ b/MIGRATIONS.md
@@ -1,3 +1,6 @@
+# Migration from 2.x.x to 3.0.0
+Check the v3 [upgrading guide](https://crawlee.dev/docs/upgrading/upgrading-to-v3).
+
# Migration from 1.x.x to 2.0.0
There should be no changes needed apart from upgrading your Node.js version to >= 15.10. If you encounter issues with `cheerio`, [read their CHANGELOG](https://github.com/cheeriojs/cheerio/releases). We bumped it from `rc.3` to `rc.10`.
diff --git a/README.md b/README.md
index 44ee3b485..a712f0173 100644
--- a/README.md
+++ b/README.md
@@ -100,7 +100,7 @@ When you run the example, you should see Crawlee automating a Chrome browser.
![Chrome Scrape](https://crawlee.dev/img/chrome_scrape.gif)
-By default, Crawlee stores data to `./crawlee_storage` in the current working directory. You can override this directory via `CRAWLEE_STORAGE_DIR` env var. For details, see [Environment variables](https://crawlee.dev/docs/guides/environment-variables), [Request storage](https://crawlee.dev/docs/guides/request-storage) and [Result storage](https://crawlee.dev/docs/guides/result-storage).
+By default, Crawlee stores data to `./storage` in the current working directory. You can override this directory via `CRAWLEE_STORAGE_DIR` env var. For details, see [Environment variables](https://crawlee.dev/docs/guides/environment-variables), [Request storage](https://crawlee.dev/docs/guides/request-storage) and [Result storage](https://crawlee.dev/docs/guides/result-storage).
### Local usage with Crawlee command-line interface (CLI)
@@ -119,7 +119,7 @@ cd my-hello-world
npx crawlee run
```
-By default, the crawling data will be stored in a local directory at `./crawlee_storage`. For example, the input JSON file for the actor is expected to be in the default key-value store in `./crawlee_storage/key_value_stores/default/INPUT.json`.
+By default, the crawling data will be stored in a local directory at `./storage`. For example, the input JSON file for the actor is expected to be in the default key-value store in `./storage/key_value_stores/default/INPUT.json`.
### Usage on the Apify platform
diff --git a/docs/examples/accept_user_input.mdx b/docs/examples/accept_user_input.mdx
index 67c076f42..ddd872f0a 100644
--- a/docs/examples/accept_user_input.mdx
+++ b/docs/examples/accept_user_input.mdx
@@ -15,7 +15,7 @@ This example accepts and logs user input:
To provide the actor with input, create a `INPUT.json` file inside the "default" key-value store:
```bash
-{PROJECT_FOLDER}/crawlee_storage/key_value_stores/default/INPUT.json
+{PROJECT_FOLDER}/storage/key_value_stores/default/INPUT.json
```
Anything in this file will be available to the actor when it runs.
diff --git a/docs/examples/add_data_to_dataset.mdx b/docs/examples/add_data_to_dataset.mdx
index c1262fef6..9ca752d11 100644
--- a/docs/examples/add_data_to_dataset.mdx
+++ b/docs/examples/add_data_to_dataset.mdx
@@ -17,5 +17,5 @@ You can save data to custom datasets by using <ApiLink to="core/class/Dataset#op
Each item in this dataset will be saved to its own file in the following directory:
```bash
-{PROJECT_FOLDER}/crawlee_storage/datasets/default/
+{PROJECT_FOLDER}/storage/datasets/default/
```
diff --git a/docs/examples/basic_crawler.mdx b/docs/examples/basic_crawler.mdx
index 056c70cd3..df9071197 100644
--- a/docs/examples/basic_crawler.mdx
+++ b/docs/examples/basic_crawler.mdx
@@ -12,7 +12,7 @@ like <ApiLink to="cheerio-crawler/class/CheerioCrawler">`CheerioCrawler`</ApiLin
The script simply downloads several web pages with plain HTTP requests using the <ApiLink to="basic-crawler/interface/BasicCrawlingContext#sendRequest">`sendRequest`</ApiLink> utility function (which uses the [`got-scraping`](https://github.com/apify/got-scraping)
npm module internally) and stores their raw HTML and URL in the default dataset. In local configuration, the data will be stored as JSON files in
-`./crawlee_storage/datasets/default`.
+`./storage/datasets/default`.
<CodeBlock className="language-js">
{BasicCrawlerSource}
diff --git a/docs/examples/cheerio_crawler.ts b/docs/examples/cheerio_crawler.ts
index b486d967a..84b98e065 100644
--- a/docs/examples/cheerio_crawler.ts
+++ b/docs/examples/cheerio_crawler.ts
@@ -44,7 +44,7 @@ const crawler = new CheerioCrawler({
});
// Store the results to the dataset. In local configuration,
- // the data will be stored as JSON files in ./crawlee_storage/datasets/default
+ // the data will be stored as JSON files in ./storage/datasets/default
await dataset.pushData({
url: request.url,
title,
diff --git a/docs/examples/forms.mdx b/docs/examples/forms.mdx
index 78ee0142c..281853a70 100644
--- a/docs/examples/forms.mdx
+++ b/docs/examples/forms.mdx
@@ -11,7 +11,7 @@ This example demonstrates how to use <ApiLink to="puppeteer-crawler/class/Puppet
automatically fill and submit a search form to look up repositories on [GitHub](https://github.com) using headless Chrome / Puppeteer.
The crawler first fills in the search term, repository owner, start date and language of the repository, then submits the form
and prints out the results. Finally, the results are saved either on the Apify platform to the
-default <ApiLink to="core/class/Dataset">`dataset`</ApiLink> or on the local machine as JSON files in `./crawlee_storage/datasets/default`.
+default <ApiLink to="core/class/Dataset">`dataset`</ApiLink> or on the local machine as JSON files in `./storage/datasets/default`.
:::tip
diff --git a/docs/examples/map_and_reduce.mdx b/docs/examples/map_and_reduce.mdx
index 00d6ab484..4f57795b0 100644
--- a/docs/examples/map_and_reduce.mdx
+++ b/docs/examples/map_and_reduce.mdx
@@ -18,7 +18,7 @@ the dataset in any way.
Examples for both methods are demonstrated on a simple dataset containing the results scraped from a page: the `URL` and a hypothetical number of
`h1` - `h3` header elements under the `headingCount` key.
-This data structure is stored in the default dataset under `{PROJECT_FOLDER}/crawlee_storage/datasets/default/`. If you want to simulate the
+This data structure is stored in the default dataset under `{PROJECT_FOLDER}/storage/datasets/default/`. If you want to simulate the
functionality, you can use the <ApiLink to="core/class/Dataset#pushData">`dataset.pushData()`</ApiLink>
method to save the example `JSON array` to your dataset.
diff --git a/docs/examples/playwright_crawler.mdx b/docs/examples/playwright_crawler.mdx
index bcd2fc4e7..c75e50d1c 100644
--- a/docs/examples/playwright_crawler.mdx
+++ b/docs/examples/playwright_crawler.mdx
@@ -9,7 +9,7 @@ import CrawlSource from '!!raw-loader!./playwright_crawler.ts';
This example demonstrates how to use <ApiLink to="playwright-crawler/class/PlaywrightCrawler">`PlaywrightCrawler`</ApiLink> in combination with <ApiLink to="core/class/RequestQueue">`RequestQueue`</ApiLink> to recursively scrape the [Hacker News website](https://news.ycombinator.com) using headless Chrome / Playwright.
-The crawler starts with a single URL, finds links to next pages, enqueues them and continues until no more desired links are available. The results are stored to the default dataset. In local configuration, the results are stored as JSON files in `./crawlee_storage/datasets/default`.
+The crawler starts with a single URL, finds links to next pages, enqueues them and continues until no more desired links are available. The results are stored to the default dataset. In local configuration, the results are stored as JSON files in `./storage/datasets/default`.
:::tip
diff --git a/docs/examples/puppeteer_crawler.mdx b/docs/examples/puppeteer_crawler.mdx
index aa9b19884..e82c25d53 100644
--- a/docs/examples/puppeteer_crawler.mdx
+++ b/docs/examples/puppeteer_crawler.mdx
@@ -12,7 +12,7 @@ with <ApiLink to="core/class/RequestQueue">`RequestQueue`</ApiLink>
to recursively scrape the [Hacker News website](https://news.ycombinator.com) using headless Chrome / Puppeteer.
The crawler starts with a single URL, finds links to next pages, enqueues them and continues until no more desired links are available. The results
-are stored to the default dataset. In local configuration, the results are stored as JSON files in `./crawlee_storage/datasets/default`
+are stored to the default dataset. In local configuration, the results are stored as JSON files in `./storage/datasets/default`
:::tip
diff --git a/docs/guides/environment_variables.mdx b/docs/guides/environment_variables.mdx
index 14ae050d8..034186805 100644
--- a/docs/guides/environment_variables.mdx
+++ b/docs/guides/environment_variables.mdx
@@ -17,7 +17,7 @@ can be changed significantly by setting or unsetting them.
### `CRAWLEE_STORAGE_DIR`
-Defines the path to a local directory where <ApiLink to="core/class/KeyValueStore">`KeyValueStore`</ApiLink>, <ApiLink to="core/class/Dataset">`Dataset`</ApiLink>, and <ApiLink to="core/class/RequestQueue">`RequestQueue`</ApiLink> store their data. By default, it is set to `./crawlee_storage`.
+Defines the path to a local directory where <ApiLink to="core/class/KeyValueStore">`KeyValueStore`</ApiLink>, <ApiLink to="core/class/Dataset">`Dataset`</ApiLink>, and <ApiLink to="core/class/RequestQueue">`RequestQueue`</ApiLink> store their data. By default, it is set to `./storage`.
### `CRAWLEE_DEFAULT_DATASET_ID`
diff --git a/docs/guides/request_storage.mdx b/docs/guides/request_storage.mdx
index a0a7df854..c14ce4095 100644
--- a/docs/guides/request_storage.mdx
+++ b/docs/guides/request_storage.mdx
@@ -17,7 +17,7 @@ import CrawlerSource from '!!raw-loader!./request_storage_queue_crawler.ts';
import RequestQueueListSource from '!!raw-loader!./request_storage_queue_list.ts';
import RequestQueueAddRequestsSource from '!!raw-loader!./request_storage_queue_only.ts';
-Crawlee has several request storage types that are useful for specific tasks. The requests are stored on local disk to a directory defined by the `CRAWLEE_STORAGE_DIR` environment variable. If this variable is not defined, by default Crawlee sets `CRAWLEE_STORAGE_DIR` to `./crawlee_storage` in the current working directory.
+Crawlee has several request storage types that are useful for specific tasks. The requests are stored on local disk to a directory defined by the `CRAWLEE_STORAGE_DIR` environment variable. If this variable is not defined, by default Crawlee sets `CRAWLEE_STORAGE_DIR` to `./storage` in the current working directory.
## Request queue
diff --git a/docs/guides/result_storage.mdx b/docs/guides/result_storage.mdx
index 54ab18cfb..eff313aef 100644
--- a/docs/guides/result_storage.mdx
+++ b/docs/guides/result_storage.mdx
@@ -6,7 +6,7 @@ description: Where are you going to store all of that juicy scraped data?!
import ApiLink from '@site/src/components/ApiLink';
-Crawlee has several result storage types that are useful for specific tasks. The data is stored on a local disk to the directory defined by the `CRAWLEE_STORAGE_DIR` environment variable. If this variable is not defined, by default Crawlee sets `CRAWLEE_STORAGE_DIR` to `./crawlee_storage` in the current working directory.
+Crawlee has several result storage types that are useful for specific tasks. The data is stored on a local disk to the directory defined by the `CRAWLEE_STORAGE_DIR` environment variable. If this variable is not defined, by default Crawlee sets `CRAWLEE_STORAGE_DIR` to `./storage` in the current working directory.
Crawlee storage is managed by <ApiLink to="memory-storage/class/MemoryStorage">`MemoryStorage`</ApiLink> class. During the crawler run all information is stored in memory, while also being off-loaded to the local files in respective storage type folders.
diff --git a/docs/introduction/05-realworld-example.mdx b/docs/introduction/05-realworld-example.mdx
index 3e0d2ac62..e9e941d17 100644
--- a/docs/introduction/05-realworld-example.mdx
+++ b/docs/introduction/05-realworld-example.mdx
@@ -535,10 +535,10 @@ await crawler.run();
#### Finding your saved data
-It might not be perfectly obvious where the data you saved using the previous command went. Unless you changed the environment variables that Crawlee uses locally, which would suggest that you knew what you were doing, and you didn't need this tutorial anyway, you'll find your data in the `crawlee_storage` directory:
+It might not be perfectly obvious where the data you saved using the previous command went. Unless you changed the environment variables that Crawlee uses locally, which would suggest that you knew what you were doing, and you didn't need this tutorial anyway, you'll find your data in the `storage` directory:
```
-{PROJECT_FOLDER}/crawlee_storage/datasets/default/
+{PROJECT_FOLDER}/storage/datasets/default/
```
The above folder will hold all your saved data in numbered files, as they were pushed into the dataset. Each file represents one invocation of `Dataset.pushData()` or one table row.
@@ -562,7 +562,7 @@ const input = await KeyValueStore.getInput();
You need to place an `INPUT.json` file in your default key-value store for this to work.
```
-{PROJECT_FOLDER}/crawlee_storage/key_value_stores/default/INPUT.json
+{PROJECT_FOLDER}/storage/key_value_stores/default/INPUT.json
```
#### Use `INPUT` to seed your crawler with users
diff --git a/docs/quick-start/index.mdx b/docs/quick-start/index.mdx
index 976ae98ac..b8f58224e 100644
--- a/docs/quick-start/index.mdx
+++ b/docs/quick-start/index.mdx
@@ -84,7 +84,7 @@ Besides the logs, you should also see Crawlee automating the browser:
</TabItem>
</Tabs>
-By default, Crawlee stores data to `./crawlee_storage` in the current working directory. You can override this behavior by setting the `CRAWLEE_STORAGE_DIR` environment variable.
+By default, Crawlee stores data to `./storage` in the current working directory. You can override this behavior by setting the `storage_DIR` environment variable.
More examples showcasing various features of Crawlee could be found in [Examples](./examples) section of the documentation.
@@ -116,4 +116,4 @@ cd my-cheerio-crawler
npm start
```
-By default, the crawling data will be stored in a local directory at `./crawlee_storage`. For example, the input JSON file for the actor is expected to be in the default key-value store in `./crawlee_storage/key_value_stores/default/INPUT.json`.
+By default, the crawling data will be stored in a local directory at `./storage`. For example, the input JSON file for the actor is expected to be in the default key-value store in `./storage/key_value_stores/default/INPUT.json`.
diff --git a/lerna.json b/lerna.json
index 42036ec2e..0a16c7fcd 100644
--- a/lerna.json
+++ b/lerna.json
@@ -2,7 +2,7 @@
"packages": [
"packages/*"
],
- "version": "3.0.0",
+ "version": "3.0.1",
"command": {
"version": {
"conventionalCommits": true,
diff --git a/package-lock.json b/package-lock.json
index 6404cae8d..39980e803 100644
--- a/package-lock.json
+++ b/package-lock.json
@@ -1,12 +1,12 @@
{
"name": "crawlee",
- "version": "3.0.0",
+ "version": "3.0.1",
"lockfileVersion": 2,
"requires": true,
"packages": {
"": {
"name": "crawlee",
- "version": "3.0.0",
+ "version": "3.0.1",
"hasInstallScript": true,
"license": "Apache-2.0",
"workspaces": [
diff --git a/package.json b/package.json
index 64628226a..80d069f49 100644
--- a/package.json
+++ b/package.json
@@ -1,7 +1,7 @@
{
"name": "crawlee",
"private": true,
- "version": "3.0.0",
+ "version": "3.0.1",
"description": "The scalable web crawling and scraping library for JavaScript/Node.js. Enables development of data extraction and web automation jobs (not only) with headless Chrome and Puppeteer.",
"workspaces": [
"packages/*"
diff --git a/packages/core/src/storages/utils.ts b/packages/core/src/storages/utils.ts
index 7a805d612..0576aa972 100644
--- a/packages/core/src/storages/utils.ts
+++ b/packages/core/src/storages/utils.ts
@@ -2,7 +2,7 @@ import type { StorageClient } from '@crawlee/types';
import { Configuration } from '../configuration';
/**
- * Cleans up the local storage folder (defaults to `./crawlee_storage`) created when running code locally.
+ * Cleans up the local storage folder (defaults to `./storage`) created when running code locally.
* Purging will remove all the files in all storages except for INPUT.json in the default KV store.
*
* Purging of storages is happening automatically when we run our crawler (or when we open some storage
diff --git a/packages/memory-storage/src/memory-storage.ts b/packages/memory-storage/src/memory-storage.ts
index 006e0eccb..0230a6df2 100644
--- a/packages/memory-storage/src/memory-storage.ts
+++ b/packages/memory-storage/src/memory-storage.ts
@@ -2,7 +2,7 @@
import type * as storage from '@crawlee/types';
import type { Dictionary } from '@crawlee/types';
import { s } from '@sapphire/shapeshift';
-import { pathExists } from 'fs-extra';
+import { pathExistsSync } from 'fs-extra';
import { readdir, rm } from 'node:fs/promises';
import { join, resolve } from 'node:path';
import { DatasetClient } from './resource-clients/dataset';
@@ -16,9 +16,10 @@ import { initWorkerIfNeeded } from './workers/instance';
export interface MemoryStorageOptions {
/**
* Path to directory where the data will also be saved.
- * @default process.env.CRAWLEE_STORAGE_DIR ?? './crawlee_storage'
+ * @default process.env.CRAWLEE_STORAGE_DIR ?? './storage'
*/
localDataDirectory?: string;
+
/**
* Whether to also write optional metadata files when storing to disk.
* @default process.env.DEBUG?.includes('*') ?? process.env.DEBUG?.includes('crawlee:memory-storage') ?? false
@@ -43,7 +44,18 @@ export class MemoryStorage implements storage.StorageClient {
writeMetadata: s.boolean.optional,
}).parse(options);
- this.localDataDirectory = options.localDataDirectory ?? process.env.CRAWLEE_STORAGE_DIR ?? './crawlee_storage';
+ // v3.0.0 used `crawlee_storage` as the default, we changed this in v3.0.1 to just `storage`,
+ // this function handles it without making BC breaks - it respects existing `crawlee_storage`
+ // directories, and uses the `storage` only if it's not there.
+ const defaultStorageDir = () => {
+ if (pathExistsSync(resolve('./crawlee_storage'))) {
+ return './crawlee_storage';
+ }
+
+ return './storage';
+ };
+
+ this.localDataDirectory = options.localDataDirectory ?? process.env.CRAWLEE_STORAGE_DIR ?? defaultStorageDir();
this.datasetsDirectory = resolve(this.localDataDirectory, 'datasets');
this.keyValueStoresDirectory = resolve(this.localDataDirectory, 'key_value_stores');
this.requestQueuesDirectory = resolve(this.localDataDirectory, 'request_queues');
@@ -113,7 +125,7 @@ export class MemoryStorage implements storage.StorageClient {
}
private async removeFiles(folder: string): Promise<void> {
- const storagePathExists = await pathExists(folder);
+ const storagePathExists = pathExistsSync(folder);
if (storagePathExists) {
const direntNames = await readdir(folder);
diff --git a/packages/templates/scripts/copy-templates.mjs b/packages/templates/scripts/copy-templates.mjs
index 4999551c3..298df8a37 100644
--- a/packages/templates/scripts/copy-templates.mjs
+++ b/packages/templates/scripts/copy-templates.mjs
@@ -6,7 +6,7 @@ const templates = await readdir('./templates');
await copy('./manifest.json', './dist/manifest.json', { override: true });
console.info(`Successfully copied 'manifest.json' to dist`);
-const ignoreFolders = ['node_modules', 'dist', 'crawlee_storage', 'apify_storage', 'package-lock.json'];
+const ignoreFolders = ['node_modules', 'dist', 'crawlee_storage', 'storage', 'apify_storage', 'package-lock.json'];
for (const tpl of templates) {
console.info(tpl);
diff --git a/packages/templates/templates/cheerio-js/.gitignore b/packages/templates/templates/cheerio-js/.gitignore
index 3065074f7..02a7e1020 100644
--- a/packages/templates/templates/cheerio-js/.gitignore
+++ b/packages/templates/templates/cheerio-js/.gitignore
@@ -5,3 +5,4 @@ dist
node_modules
apify_storage
crawlee_storage
+storage
diff --git a/packages/templates/templates/cheerio-ts/.gitignore b/packages/templates/templates/cheerio-ts/.gitignore
index 3065074f7..02a7e1020 100644
--- a/packages/templates/templates/cheerio-ts/.gitignore
+++ b/packages/templates/templates/cheerio-ts/.gitignore
@@ -5,3 +5,4 @@ dist
node_modules
apify_storage
crawlee_storage
+storage
diff --git a/packages/templates/templates/playwright-js/.gitignore b/packages/templates/templates/playwright-js/.gitignore
index 3065074f7..02a7e1020 100644
--- a/packages/templates/templates/playwright-js/.gitignore
+++ b/packages/templates/templates/playwright-js/.gitignore
@@ -5,3 +5,4 @@ dist
node_modules
apify_storage
crawlee_storage
+storage
diff --git a/packages/templates/templates/playwright-ts/.gitignore b/packages/templates/templates/playwright-ts/.gitignore
index 3065074f7..02a7e1020 100644
--- a/packages/templates/templates/playwright-ts/.gitignore
+++ b/packages/templates/templates/playwright-ts/.gitignore
@@ -5,3 +5,4 @@ dist
node_modules
apify_storage
crawlee_storage
+storage
diff --git a/packages/templates/templates/puppeteer-js/.gitignore b/packages/templates/templates/puppeteer-js/.gitignore
index 3065074f7..02a7e1020 100644
--- a/packages/templates/templates/puppeteer-js/.gitignore
+++ b/packages/templates/templates/puppeteer-js/.gitignore
@@ -5,3 +5,4 @@ dist
node_modules
apify_storage
crawlee_storage
+storage
diff --git a/packages/templates/templates/puppeteer-ts/.gitignore b/packages/templates/templates/puppeteer-ts/.gitignore
index 3065074f7..02a7e1020 100644
--- a/packages/templates/templates/puppeteer-ts/.gitignore
+++ b/packages/templates/templates/puppeteer-ts/.gitignore
@@ -5,3 +5,4 @@ dist
node_modules
apify_storage
crawlee_storage
+storage
diff --git a/scripts/copy.ts b/scripts/copy.ts
index ff127bdf4..19c499b1b 100644
--- a/scripts/copy.ts
+++ b/scripts/copy.ts
@@ -23,36 +23,49 @@ function rewrite(path: string, replacer: (from: string) => string): void {
}
}
+let rootVersion: string;
+
+function getRootVersion(): string {
+ if (rootVersion) {
+ return rootVersion;
+ }
+
+ // eslint-disable-next-line @typescript-eslint/no-var-requires,import/no-dynamic-require,global-require
+ rootVersion = require(resolve(root, './packages/core/package.json')).version.replace(/^(\d+\.\d+\.\d+)-?.*$/, '$1');
+
+ return rootVersion;
+}
+
/**
* Checks next dev version number based on the `@crawlee/core` package via `npm show`.
* We always use this package, so we ensure the version is the same for each package in the monorepo.
*/
function getNextVersion() {
const versions: string[] = [];
- // eslint-disable-next-line @typescript-eslint/no-var-requires,import/no-dynamic-require,global-require
- const pkgJson = require(resolve(root, 'package.json'));
try {
- const versionString = execSync(`npm show ${pkgJson.name} versions --json`, { encoding: 'utf8', stdio: 'pipe' });
+ const versionString = execSync(`npm show crawlee versions --json`, { encoding: 'utf8', stdio: 'pipe' });
const parsed = JSON.parse(versionString) as string[];
versions.push(...parsed);
} catch {
// the package might not have been published yet
}
- if (versions.some((v) => v === pkgJson.version)) {
+ const version = getRootVersion();
+
+ if (versions.some((v) => v === version)) {
// eslint-disable-next-line no-console
- console.error(`before-deploy: A release with version ${pkgJson.version} already exists. Please increment version accordingly.`);
+ console.error(`before-deploy: A release with version ${version} already exists. Please increment version accordingly.`);
process.exit(1);
}
const preid = options.preid ?? 'alpha';
const prereleaseNumbers = versions
- .filter((v) => v.startsWith(`${pkgJson.version}-${preid}.`))
+ .filter((v) => v.startsWith(`${version}-${preid}.`))
.map((v) => Number(v.match(/\.(\d+)$/)?.[1]));
const lastPrereleaseNumber = Math.max(-1, ...prereleaseNumbers);
- return `${pkgJson.version}-${preid}.${lastPrereleaseNumber + 1}`;
+ return `${version}-${preid}.${lastPrereleaseNumber + 1}`;
}
// as we publish only the dist folder, we need to copy some meta files inside (readme/license/package.json)
diff --git a/test/e2e/autoscaling-max-tasks-per-minute/actor/.gitignore b/test/e2e/autoscaling-max-tasks-per-minute/actor/.gitignore
index 3b1ab3bb0..ced7cbfc5 100644
--- a/test/e2e/autoscaling-max-tasks-per-minute/actor/.gitignore
+++ b/test/e2e/autoscaling-max-tasks-per-minute/actor/.gitignore
@@ -3,3 +3,5 @@
node_modules
package-lock.json
apify_storage
+crawlee_storage
+storage
diff --git a/test/e2e/cheerio-default/actor/.gitignore b/test/e2e/cheerio-default/actor/.gitignore
index 3b1ab3bb0..ced7cbfc5 100644
--- a/test/e2e/cheerio-default/actor/.gitignore
+++ b/test/e2e/cheerio-default/actor/.gitignore
@@ -3,3 +3,5 @@
node_modules
package-lock.json
apify_storage
+crawlee_storage
+storage
diff --git a/test/e2e/cheerio-enqueue-links/actor/.gitignore b/test/e2e/cheerio-enqueue-links/actor/.gitignore
index 3b1ab3bb0..ced7cbfc5 100644
--- a/test/e2e/cheerio-enqueue-links/actor/.gitignore
+++ b/test/e2e/cheerio-enqueue-links/actor/.gitignore
@@ -3,3 +3,5 @@
node_modules
package-lock.json
apify_storage
+crawlee_storage
+storage
diff --git a/test/e2e/cheerio-ignore-ssl-errors/actor/.gitignore b/test/e2e/cheerio-ignore-ssl-errors/actor/.gitignore
index 3b1ab3bb0..ced7cbfc5 100644
--- a/test/e2e/cheerio-ignore-ssl-errors/actor/.gitignore
+++ b/test/e2e/cheerio-ignore-ssl-errors/actor/.gitignore
@@ -3,3 +3,5 @@
node_modules
package-lock.json
apify_storage
+crawlee_storage
+storage
diff --git a/test/e2e/cheerio-initial-cookies/actor/.gitignore b/test/e2e/cheerio-initial-cookies/actor/.gitignore
index 3b1ab3bb0..ced7cbfc5 100644
--- a/test/e2e/cheerio-initial-cookies/actor/.gitignore
+++ b/test/e2e/cheerio-initial-cookies/actor/.gitignore
@@ -3,3 +3,5 @@
node_modules
package-lock.json
apify_storage
+crawlee_storage
+storage
diff --git a/test/e2e/cheerio-max-requests/actor/.gitignore b/test/e2e/cheerio-max-requests/actor/.gitignore
index 3b1ab3bb0..ced7cbfc5 100644
--- a/test/e2e/cheerio-max-requests/actor/.gitignore
+++ b/test/e2e/cheerio-max-requests/actor/.gitignore
@@ -3,3 +3,5 @@
node_modules
package-lock.json
apify_storage
+crawlee_storage
+storage
diff --git a/test/e2e/cheerio-page-info/actor/.gitignore b/test/e2e/cheerio-page-info/actor/.gitignore
index 3b1ab3bb0..ced7cbfc5 100644
--- a/test/e2e/cheerio-page-info/actor/.gitignore
+++ b/test/e2e/cheerio-page-info/actor/.gitignore
@@ -3,3 +3,5 @@
node_modules
package-lock.json
apify_storage
+crawlee_storage
+storage
diff --git a/test/e2e/cheerio-throw-on-ssl-errors/actor/.gitignore b/test/e2e/cheerio-throw-on-ssl-errors/actor/.gitignore
index 3b1ab3bb0..ced7cbfc5 100644
--- a/test/e2e/cheerio-throw-on-ssl-errors/actor/.gitignore
+++ b/test/e2e/cheerio-throw-on-ssl-errors/actor/.gitignore
@@ -3,3 +3,5 @@
node_modules
package-lock.json
apify_storage
+crawlee_storage
+storage
diff --git a/test/e2e/playwright-default/actor/.gitignore b/test/e2e/playwright-default/actor/.gitignore
index 3b1ab3bb0..ced7cbfc5 100644
--- a/test/e2e/playwright-default/actor/.gitignore
+++ b/test/e2e/playwright-default/actor/.gitignore
@@ -3,3 +3,5 @@
node_modules
package-lock.json
apify_storage
+crawlee_storage
+storage
diff --git a/test/e2e/proxy-rotation/actor/.gitignore b/test/e2e/proxy-rotation/actor/.gitignore
index 3b1ab3bb0..ced7cbfc5 100644
--- a/test/e2e/proxy-rotation/actor/.gitignore
+++ b/test/e2e/proxy-rotation/actor/.gitignore
@@ -3,3 +3,5 @@
node_modules
package-lock.json
apify_storage
+crawlee_storage
+storage
diff --git a/test/e2e/puppeteer-default/actor/.gitignore b/test/e2e/puppeteer-default/actor/.gitignore
index 3b1ab3bb0..ced7cbfc5 100644
--- a/test/e2e/puppeteer-default/actor/.gitignore
+++ b/test/e2e/puppeteer-default/actor/.gitignore
@@ -3,3 +3,5 @@
node_modules
package-lock.json
apify_storage
+crawlee_storage
+storage
diff --git a/test/e2e/puppeteer-enqueue-links/actor/.gitignore b/test/e2e/puppeteer-enqueue-links/actor/.gitignore
index 3b1ab3bb0..ced7cbfc5 100644
--- a/test/e2e/puppeteer-enqueue-links/actor/.gitignore
+++ b/test/e2e/puppeteer-enqueue-links/actor/.gitignore
@@ -3,3 +3,5 @@
node_modules
package-lock.json
apify_storage
+crawlee_storage
+storage
diff --git a/test/e2e/puppeteer-ignore-ssl-errors/actor/.gitignore b/test/e2e/puppeteer-ignore-ssl-errors/actor/.gitignore
index 3b1ab3bb0..ced7cbfc5 100644
--- a/test/e2e/puppeteer-ignore-ssl-errors/actor/.gitignore
+++ b/test/e2e/puppeteer-ignore-ssl-errors/actor/.gitignore
@@ -3,3 +3,5 @@
node_modules
package-lock.json
apify_storage
+crawlee_storage
+storage
diff --git a/test/e2e/puppeteer-initial-cookies/actor/.gitignore b/test/e2e/puppeteer-initial-cookies/actor/.gitignore
index 3b1ab3bb0..ced7cbfc5 100644
--- a/test/e2e/puppeteer-initial-cookies/actor/.gitignore
+++ b/test/e2e/puppeteer-initial-cookies/actor/.gitignore
@@ -3,3 +3,5 @@
node_modules
package-lock.json
apify_storage
+crawlee_storage
+storage
diff --git a/test/e2e/puppeteer-page-info/actor/.gitignore b/test/e2e/puppeteer-page-info/actor/.gitignore
index 3b1ab3bb0..ced7cbfc5 100644
--- a/test/e2e/puppeteer-page-info/actor/.gitignore
+++ b/test/e2e/puppeteer-page-info/actor/.gitignore
@@ -3,3 +3,5 @@
node_modules
package-lock.json
apify_storage
+crawlee_storage
+storage
diff --git a/test/e2e/puppeteer-store-pagination-jquery/actor/.gitignore b/test/e2e/puppeteer-store-pagination-jquery/actor/.gitignore
index 3b1ab3bb0..ced7cbfc5 100644
--- a/test/e2e/puppeteer-store-pagination-jquery/actor/.gitignore
+++ b/test/e2e/puppeteer-store-pagination-jquery/actor/.gitignore
@@ -3,3 +3,5 @@
node_modules
package-lock.json
apify_storage
+crawlee_storage
+storage
diff --git a/test/e2e/puppeteer-store-pagination/actor/.gitignore b/test/e2e/puppeteer-store-pagination/actor/.gitignore
index 3b1ab3bb0..ced7cbfc5 100644
--- a/test/e2e/puppeteer-store-pagination/actor/.gitignore
+++ b/test/e2e/puppeteer-store-pagination/actor/.gitignore
@@ -3,3 +3,5 @@
node_modules
package-lock.json
apify_storage
+crawlee_storage
+storage
diff --git a/test/e2e/puppeteer-throw-on-ssl-errors/actor/.gitignore b/test/e2e/puppeteer-throw-on-ssl-errors/actor/.gitignore
index 3b1ab3bb0..ced7cbfc5 100644
--- a/test/e2e/puppeteer-throw-on-ssl-errors/actor/.gitignore
+++ b/test/e2e/puppeteer-throw-on-ssl-errors/actor/.gitignore
@@ -3,3 +3,5 @@
node_modules
package-lock.json
apify_storage
+crawlee_storage
+storage
diff --git a/test/e2e/request-queue-zero-concurrency/actor/.gitignore b/test/e2e/request-queue-zero-concurrency/actor/.gitignore
index 3b1ab3bb0..ced7cbfc5 100644
--- a/test/e2e/request-queue-zero-concurrency/actor/.gitignore
+++ b/test/e2e/request-queue-zero-concurrency/actor/.gitignore
@@ -3,3 +3,5 @@
node_modules
package-lock.json
apify_storage
+crawlee_storage
+storage
diff --git a/test/e2e/request-skip-navigation/actor/.gitignore b/test/e2e/request-skip-navigation/actor/.gitignore
index 3b1ab3bb0..ced7cbfc5 100644
--- a/test/e2e/request-skip-navigation/actor/.gitignore
+++ b/test/e2e/request-skip-navigation/actor/.gitignore
@@ -3,3 +3,5 @@
node_modules
package-lock.json
apify_storage
+crawlee_storage
+storage
diff --git a/test/e2e/run.mjs b/test/e2e/run.mjs
index 7969259c0..c2f87204e 100644
--- a/test/e2e/run.mjs
+++ b/test/e2e/run.mjs
@@ -16,7 +16,7 @@ process.env.APIFY_CONTAINER_PORT ??= '8000';
/**
* Depending on STORAGE_IMPLEMENTATION the workflow of the tests slightly differs:
- * - for 'MEMORY': the 'crawlee_storage' folder should be removed after the test actor finishes;
+ * - for 'MEMORY': the 'storage' folder should be removed after the test actor finishes;
* - for 'LOCAL': the 'apify_storage' folder should be removed after the test actor finishes;
* - for 'PLATFORM': SDK packages should be copied to respective test actor folders
* (and also should be removed after pushing the actor to platform and starting the test run there)
diff --git a/test/e2e/tools.mjs b/test/e2e/tools.mjs
index 64275ca2e..555a705f1 100644
--- a/test/e2e/tools.mjs
+++ b/test/e2e/tools.mjs
@@ -25,7 +25,7 @@ export const colors = {
export function getStorage(dirName) {
let folderName;
if (process.env.STORAGE_IMPLEMENTATION === 'LOCAL') folderName = 'apify_storage';
- if (process.env.STORAGE_IMPLEMENTATION === 'MEMORY') folderName = 'crawlee_storage';
+ if (process.env.STORAGE_IMPLEMENTATION === 'MEMORY') folderName = 'storage';
return join(dirName, folderName);
}
@@ -161,7 +161,7 @@ export async function clearPackages(dirName) {
export async function clearStorage(dirName) {
let folderName;
if (process.env.STORAGE_IMPLEMENTATION === 'LOCAL') folderName = 'apify_storage';
- if (process.env.STORAGE_IMPLEMENTATION === 'MEMORY') folderName = 'crawlee_storage';
+ if (process.env.STORAGE_IMPLEMENTATION === 'MEMORY') folderName = 'storage';
const destPackagesDir = join(dirName, 'actor', folderName);
await fs.remove(destPackagesDir);
}