Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.
TypeScript apifyautomationcrawlercrawlingheadlessheadless-chromejavascriptnodejsnpmplaywrightpuppeteerscraperscrapingtypescriptweb-crawlerweb-crawlingweb-scraping
feat: rename KVS objects from `SDK_` to `CRAWLEE_` (#3783)
Jindřich Bär8af187d240f7
diff --git a/docs/guides/configuration.mdx b/docs/guides/configuration.mdxindex 0e16c845d..8ec11c466 100644--- a/docs/guides/configuration.mdx+++ b/docs/guides/configuration.mdx@@ -57,7 +57,7 @@ crawler.router.addDefaultHandler(async ({ request }) => {await crawler.run(['https://www.example.com/1']);```-If you run this example (assuming you placed the `crawlee.json` file with `persistStateIntervalMillis` and `logLevel` specified there in the root of your project), you will find the `SDK_CRAWLER_STATISTICS` file in default Key-Value store,+If you run this example (assuming you placed the `crawlee.json` file with `persistStateIntervalMillis` and `logLevel` specified there in the root of your project), you will find the `CRAWLEE_CRAWLER_STATISTICS` file in default Key-Value store,which would show, that there's 1 finished request and crawler runtime was ~10 seconds.This confirms that the state was persisted after 10 seconds, as it was set in `crawlee.json`.Besides, you should see `DEBUG` logs in addition to `INFO` ones in your terminal, as `logLevel` was set to `DEBUG` in the `crawlee.json`, meaning Crawlee picked both provided options correctly.@@ -174,14 +174,14 @@ crawler.router.addDefaultHandler(async ({ request }) => {await crawler.run(['https://www.example.com/1']);```-If you run this example - you will find the `SDK_CRAWLER_STATISTICS` file in default Key-Value store,+If you run this example - you will find the `CRAWLEE_CRAWLER_STATISTICS` file in default Key-Value store,which would show the same number of finished requests (one) and the same crawler runtime (~10 seconds).This confirms that provided parameters worked: the state was persisted after 10 seconds, as it was set in the configuration.:::noteAfter running the same example without the custom configuration, there will be-no `SDK_CRAWLER_STATISTICS` file stored in the default Key-Value store:+no `CRAWLEE_CRAWLER_STATISTICS` file stored in the default Key-Value store:as we did not change the `persistStateIntervalMillis`, Crawlee used the default value of 60 seconds,and the crawler was forcefully aborted after ~15 seconds of run time before it persisted the state for the first time.@@ -222,13 +222,13 @@ await crawler.run(['https://www.example.com/1']);```If you run this example - it would work exactly the same as before,-with the same `SDK_CRAWLER_STATISTICS` file in default Key-Value store after the run,+with the same `CRAWLEE_CRAWLER_STATISTICS` file in default Key-Value store after the run,showing the same number of finished requests and the same crawler run time.:::noteIf you would not pass the configuration to the crawler, there again will be-no `SDK_CRAWLER_STATISTICS` file stored in the default Key-Value store, this time for a different reason though.+no `CRAWLEE_CRAWLER_STATISTICS` file stored in the default Key-Value store, this time for a different reason though.Since we did not pass the configuration to the crawler,the crawler will use the global configuration, which is using the default `persistStateIntervalMillis`.So again, the run was aborted before the state was persisted for the first time.diff --git a/docs/upgrading/upgrading_v4.md b/docs/upgrading/upgrading_v4.mdindex d00a9837c..24eda63f8 100644--- a/docs/upgrading/upgrading_v4.md+++ b/docs/upgrading/upgrading_v4.md@@ -504,6 +504,10 @@ The protected `HttpCrawler._applyCookies` method is removed. If you were overridThe `persistCookiesPerSession` crawler option has been renamed to `saveResponseCookies` on both `HttpCrawler` (and its subclasses like `CheerioCrawler`, `JSDOMCrawler`, etc.) and `BrowserCrawler`. The behavior is unchanged - when enabled (the default), response `Set-Cookie` headers are stored in the session's cookie jar so they're sent on subsequent requests using the same session. Rename the option in your crawler constructor options to migrate.+## Internal KVS keys renamed++Several internal Crawlee keys were prefixed with the `SDK_` prefix for legacy reasons - these keys now start with `CRAWLEE_` instead. These are, e.g., `CRAWLEE_SESSION_POOL_STATE` or `CRAWLEE_CRAWLER_STATISTICS_{n}`.+## `StorageClient` interface simplifiedThe `StorageClient` interface (from `@crawlee/types`) has been redesigned to match the simplified architecture from Crawlee for Python. A new storage backend now needs **4 classes** instead of the previous 7.diff --git a/packages/core/src/crawlers/statistics.ts b/packages/core/src/crawlers/statistics.tsindex 8056c2344..818b893a2 100644--- a/packages/core/src/crawlers/statistics.ts+++ b/packages/core/src/crawlers/statistics.ts@@ -49,7 +49,7 @@ export interface PersistenceOptions {* statistics for requests.** All statistic information is saved on key value store- * under the key `SDK_CRAWLER_STATISTICS_*`, persists between+ * under the key `CRAWLEE_CRAWLER_STATISTICS_*`, persists between* migrations and abort/resurrect** @category Crawlers@@ -130,7 +130,7 @@ export class Statistics {} = options;this.id = id ?? String(Statistics.id++);- this.persistStateKey = `SDK_CRAWLER_STATISTICS_${this.id}`;+ this.persistStateKey = `CRAWLEE_CRAWLER_STATISTICS_${this.id}`;this.log = (options.log ?? serviceLocator.getLogger()).child({ prefix: 'Statistics' });this.errorTracker = new ErrorTracker({ ...errorTrackerConfig, saveErrorSnapshots });diff --git a/packages/core/src/session_pool/consts.ts b/packages/core/src/session_pool/consts.tsindex 8fdfe50d6..7fbe1ed1b 100644--- a/packages/core/src/session_pool/consts.ts+++ b/packages/core/src/session_pool/consts.ts@@ -1,3 +1,3 @@export const BLOCKED_STATUS_CODES = [401, 403, 429];-export const PERSIST_STATE_KEY = 'SDK_SESSION_POOL_STATE';+export const PERSIST_STATE_KEY = 'CRAWLEE_SESSION_POOL_STATE';export const MAX_POOL_SIZE = 1000;diff --git a/packages/core/src/session_pool/session_pool.ts b/packages/core/src/session_pool/session_pool.tsindex 5e2185e6d..a771e493d 100644--- a/packages/core/src/session_pool/session_pool.ts+++ b/packages/core/src/session_pool/session_pool.ts@@ -48,7 +48,7 @@ export interface SessionPoolOptions {/*** Session pool persists its state under this key in Key value store.- * @default SDK_SESSION_POOL_STATE_{id}+ * @default CRAWLEE_SESSION_POOL_STATE_{id}*/persistStateKey?: string;diff --git a/packages/core/src/storages/request_list.ts b/packages/core/src/storages/request_list.tsindex 7f253fbed..26870b532 100644--- a/packages/core/src/storages/request_list.ts+++ b/packages/core/src/storages/request_list.ts@@ -332,8 +332,8 @@ export class RequestList implements IRequestLoader {}),);- this.persistStateKey = persistStateKey ? `SDK_${persistStateKey}` : persistStateKey;- this.persistRequestsKey = persistRequestsKey ? `SDK_${persistRequestsKey}` : persistRequestsKey;+ this.persistStateKey = persistStateKey ? `CRAWLEE_${persistStateKey}` : persistStateKey;+ this.persistRequestsKey = persistRequestsKey ? `CRAWLEE_${persistRequestsKey}` : persistRequestsKey;this.initialState = state;this.httpClient = httpClient;diff --git a/test/core/crawlers/statistics.test.ts b/test/core/crawlers/statistics.test.tsindex 9f4535939..c017a3835 100644--- a/test/core/crawlers/statistics.test.ts+++ b/test/core/crawlers/statistics.test.ts@@ -36,7 +36,7 @@ describe('Statistics', () => {// @ts-expect-error Accessing private propexpect(Statistics.id).toEqual(1);// @ts-expect-error Accessing private prop- expect(stats.persistStateKey).toEqual('SDK_CRAWLER_STATISTICS_0');+ expect(stats.persistStateKey).toEqual('CRAWLEE_CRAWLER_STATISTICS_0');const [n1, n2] = [new Statistics(), new Statistics()];expect(n1.id).toEqual('1');expect(n2.id).toEqual('2');diff --git a/test/core/request_list.test.ts b/test/core/request_list.test.tsindex c5cb7536f..bc1f5c412 100644--- a/test/core/request_list.test.ts+++ b/test/core/request_list.test.ts@@ -581,15 +581,15 @@ describe('RequestList', () => {const setValueSpy = vitest.spyOn(KeyValueStore.prototype, 'setValue');const name = 'xxx';- const SDK_KEY = `SDK_${name}`;+ const CRAWLEE_KEY = `CRAWLEE_${name}`;const sources = [{ url: 'https://example.com' }];const rl = await RequestList.open(name, sources);expect(rl).toBeInstanceOf(RequestList);// @ts-expect-error accessing private var- expect(rl.persistStateKey.startsWith(SDK_KEY)).toBe(true);+ expect(rl.persistStateKey.startsWith(CRAWLEE_KEY)).toBe(true);// @ts-expect-error accessing private var- expect(rl.persistRequestsKey.startsWith(SDK_KEY)).toBe(true);+ expect(rl.persistRequestsKey.startsWith(CRAWLEE_KEY)).toBe(true);// @ts-expect-error accessing private varexpect(rl.sources).toEqual([]);// @ts-expect-error accessing private var@@ -604,16 +604,16 @@ describe('RequestList', () => {const setValueSpy = vitest.spyOn(KeyValueStore.prototype, 'setValue');const name = 'xxx';- const SDK_KEY = `SDK_${name}`;+ const CRAWLEE_KEY = `CRAWLEE_${name}`;const sources = ['https://example.com'];const requests = sources.map((url) => ({ url, uniqueKey: url }));const rl = await RequestList.open(name, sources);expect(rl).toBeInstanceOf(RequestList);// @ts-expect-error accessing private var- expect(rl.persistStateKey.startsWith(SDK_KEY)).toBe(true);+ expect(rl.persistStateKey.startsWith(CRAWLEE_KEY)).toBe(true);// @ts-expect-error accessing private var- expect(rl.persistRequestsKey.startsWith(SDK_KEY)).toBe(true);+ expect(rl.persistRequestsKey.startsWith(CRAWLEE_KEY)).toBe(true);expect(rl.requests).toEqual(requests);// @ts-expect-error accessing private varexpect(rl.isInitialized).toBe(true);@@ -627,7 +627,7 @@ describe('RequestList', () => {const setValueSpy = vitest.spyOn(KeyValueStore.prototype, 'setValue');const name = 'xxx';- const SDK_KEY = `SDK_${name}`;+ const CRAWLEE_KEY = `CRAWLEE_${name}`;let counter = 0;const sources = [{ url: 'https://example.com' }];const requests = sources.map(({ url }) => ({ url, uniqueKey: `${url}-${counter++}` }));@@ -639,9 +639,9 @@ describe('RequestList', () => {const rl = await RequestList.open(name, sources, options);expect(rl).toBeInstanceOf(RequestList);// @ts-expect-error accessing private var- expect(rl.persistStateKey.startsWith(SDK_KEY)).toBe(true);+ expect(rl.persistStateKey.startsWith(CRAWLEE_KEY)).toBe(true);// @ts-expect-error accessing private var- expect(rl.persistRequestsKey.startsWith(SDK_KEY)).toBe(true);+ expect(rl.persistRequestsKey.startsWith(CRAWLEE_KEY)).toBe(true);expect(rl.requests).toEqual(requests);// @ts-expect-error accessing private varexpect(rl.isInitialized).toBe(true);diff --git a/test/e2e/tools.mjs b/test/e2e/tools.mjsindex 515bb5fd0..acfed8c78 100644--- a/test/e2e/tools.mjs+++ b/test/e2e/tools.mjs@@ -24,12 +24,12 @@ function execSync(command, options) {* the base crawler created and installs its own, which lands on the next id.* @param {string} name*/-const isCrawlerStatisticsKey = (name) => name.startsWith('SDK_CRAWLER_STATISTICS_');+const isCrawlerStatisticsKey = (name) => name.startsWith('CRAWLEE_CRAWLER_STATISTICS_');/*** @param {string} name*/-const isPrivateEntry = (name) => isCrawlerStatisticsKey(name) || name === 'SDK_SESSION_POOL_STATE';+const isPrivateEntry = (name) => isCrawlerStatisticsKey(name) || name === 'CRAWLEE_SESSION_POOL_STATE';export const SKIPPED_TEST_CLOSE_CODE = 404;