Skip to content

Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.

refactor(core)!: persist RequestList and SessionPool state through RecoverableState (#4068)

Vlad Frangub70693fab8fb
diff --git a/docs/public-api/crawlee-core.api.md b/docs/public-api/crawlee-core.api.md
index aab338a54..0a842f687 100644
--- a/docs/public-api/crawlee-core.api.md
+++ b/docs/public-api/crawlee-core.api.md
@@ -41,7 +41,6 @@ import type { RequestQueueBackend } from '@crawlee/types';
import type { RequestQueueInfo } from '@crawlee/types';
import type { SendRequestOptions } from '@crawlee/types';
import type { SessionFingerprint } from '@crawlee/types';
-import { SessionState } from '@crawlee/types';
import type { StandardSchemaV1 } from '@standard-schema/spec';
import type * as storage from '@crawlee/types';
import { StorageBackend } from '@crawlee/types';
@@ -1661,7 +1660,6 @@ export class Session implements ISession {
get fingerprint(): SessionFingerprint | undefined;
set fingerprint(fingerprint: SessionFingerprint | undefined);
getCookieString(url: string): Promise<string>;
- getState(): SessionState;
// (undocumented)
readonly id: string;
isBlocked(): boolean;
@@ -1724,21 +1722,14 @@ export class SessionPool implements ISessionPool {
constructor(options?: SessionPoolOptions);
addSession(options?: Session | SessionOptions): Promise<void>;
getSession(sessionId?: string): Promise<Session | undefined>;
- getState(): Promise<{
- usableSessionsCount: number;
- retiredSessionsCount: number;
- sessions: SessionState[];
- }>;
// (undocumented)
readonly id: string;
newSession(sessionOptions?: SessionOptions): Promise<Session>;
- persistState(options?: PersistenceOptions): Promise<void>;
- // (undocumented)
- resetStore(options?: PersistenceOptions): Promise<void>;
+ persistState(): Promise<void>;
+ reset(): void;
+ resetStore(): Promise<void>;
retiredSessionsCount(): Promise<number>;
- teardown(input?: {
- persistState?: boolean;
- }): Promise<void>;
+ teardown(): Promise<void>;
usableSessionsCount(): Promise<number>;
}
diff --git a/docs/public-api/crawlee-types.api.md b/docs/public-api/crawlee-types.api.md
index 4e58d6444..04b8e449c 100644
--- a/docs/public-api/crawlee-types.api.md
+++ b/docs/public-api/crawlee-types.api.md
@@ -529,36 +529,6 @@ export interface SessionFingerprint {
platform?: 'windows' | 'macos' | 'linux' | 'android' | 'ios';
}
-// @public
-export interface SessionState {
- // (undocumented)
- cookieJar: SerializedCookieJar;
- // (undocumented)
- createdAt: string;
- // (undocumented)
- errorScore: number;
- // (undocumented)
- errorScoreDecrement: number;
- // (undocumented)
- expiresAt: string;
- // (undocumented)
- fingerprint?: SessionFingerprint;
- // (undocumented)
- id: string;
- // (undocumented)
- maxErrorScore: number;
- // (undocumented)
- maxUsageCount: number;
- // (undocumented)
- proxyInfo?: ProxyInfo;
- // (undocumented)
- retired: boolean;
- // (undocumented)
- usageCount: number;
- // (undocumented)
- userData: object;
-}
-
// @public
export interface SetStatusMessageOptions {
isStatusMessageTerminal?: boolean;
diff --git a/docs/upgrading/upgrading_v4.md b/docs/upgrading/upgrading_v4.md
index 1d4a1ba96..bef92b89c 100644
--- a/docs/upgrading/upgrading_v4.md
+++ b/docs/upgrading/upgrading_v4.md
@@ -384,22 +384,32 @@ const crawler = new BasicCrawler({
`SessionPool.open()` static factory method is removed. Create instances with `new SessionPool(options)` instead — all public methods automatically initialize the pool on first use.
-`SessionPool.usableSessionsCount` and `SessionPool.retiredSessionsCount` are now async methods instead of synchronous getters. `SessionPool.getState()` is also async now.
+`SessionPool.usableSessionsCount` and `SessionPool.retiredSessionsCount` are now async methods instead of synchronous getters.
**Before:**
```typescript
const sessionPool = await SessionPool.open({ maxPoolSize: 100 });
const count = sessionPool.usableSessionsCount;
-const state = sessionPool.getState();
```
**After:**
```typescript
const sessionPool = new SessionPool({ maxPoolSize: 100 });
const count = await sessionPool.usableSessionsCount();
-const state = await sessionPool.getState();
```
+### `SessionPool.persistState()`, `resetStore()` and `teardown()` no longer take options
+
+The `PersistenceOptions` argument of `persistState()` and `resetStore()` and the `{ persistState }` argument of `teardown()` are removed. Neither had any effect, so there is nothing to replace them with.
+
+`resetStore()` now throws while the pool is running, since the next periodic write would put the record straight back. Call `teardown()` first, or use the new `reset()` to discard the in-memory sessions instead.
+
+### `RequestList` prefers the persisted record over the `state` option
+
+With both `state` and `persistStateKey` set, the record now wins. Previously `state` did. The option is also validated up front: `nextIndex` must be a non-negative integer and `inProgress` an array of unique keys. The `@internal` `isStatePersisted` flag is gone.
+
+Both `SessionPool` and `RequestList` now persist through `RecoverableState`, like `Statistics`. The persisted records keep their shape, so records written by v3 still load.
+
### `retireOnBlockedStatusCodes` is removed from `Session`
`Session.retireOnBlockedStatusCodes` is removed. Blocked status code handling is now internal to the crawler. Configure blocked status codes via the `blockedStatusCodes` crawler option (moved from `sessionPoolOptions`).
diff --git a/packages/basic-crawler/src/internals/basic-crawler.ts b/packages/basic-crawler/src/internals/basic-crawler.ts
index 8c5bdfd3a..7ba2f8c60 100644
--- a/packages/basic-crawler/src/internals/basic-crawler.ts
+++ b/packages/basic-crawler/src/internals/basic-crawler.ts
@@ -1705,7 +1705,11 @@ export class BasicCrawler<
stats.reset();
await stats.resetStore();
});
- await this.#sessionPoolDep.ifOwned((pool) => pool.resetStore());
+ // A session pool the crawler built follows the same blank-slate rule as its statistics.
+ await this.#sessionPoolDep.ifOwned(async (pool) => {
+ pool.reset();
+ await pool.resetStore();
+ });
}
this.#unexpectedStop = false;
@@ -2981,6 +2985,10 @@ export class BasicCrawler<
* To stop the crawler gracefully (waiting for all running requests to finish), use {@apilink BasicCrawler.stop|`crawler.stop()`} instead.
*/
async teardown(): Promise<void> {
+ // First, so that the pool writes its final state once - its teardown stops it listening before the
+ // persistence event below goes out.
+ await this.#sessionPoolDep.ifOwned(async (pool) => pool.teardown());
+
// When this crawler initialized the event manager, its close() call emits
// the final persistence event after the crawler-specific state has been
// saved. External event managers still need an explicit event here.
@@ -2988,8 +2996,6 @@ export class BasicCrawler<
serviceLocator.getEventManager().emit(EventType.PERSIST_STATE, { isMigrating: false });
}
- await this.#sessionPoolDep.ifOwned(async (pool) => pool.teardown({ persistState: this.#closeEvents ?? false }));
-
if (this.#closeEvents) {
await serviceLocator.getEventManager().close();
}
diff --git a/packages/core/src/recoverable_state.ts b/packages/core/src/recoverable_state.ts
index de8211191..fcf6035d3 100644
--- a/packages/core/src/recoverable_state.ts
+++ b/packages/core/src/recoverable_state.ts
@@ -131,6 +131,7 @@ export class RecoverableState<TStateModel = Record<string, unknown>, TPersistedS
readonly #defaultState: () => TStateModel;
#state: TStateModel | null = null;
#initialized = false;
+ #recordLoaded = false;
#listening = false;
readonly #persistenceEnabled: boolean;
readonly #persistStateKey: string;
@@ -163,6 +164,13 @@ export class RecoverableState<TStateModel = Record<string, unknown>, TPersistedS
this.#serialize = this.#toConversion(options.serialize);
this.#deserialize = this.#toConversion(options.deserialize);
+ // A pending open() may reject before anything here awaits it. Observe the rejection so it surfaces on
+ // first use rather than as an unhandled rejection - most immediately when persistence is disabled and
+ // nothing ever awaits the store.
+ if (this.#keyValueStore !== null) {
+ Promise.resolve(this.#keyValueStore).then(undefined, () => {});
+ }
+
// The automatic persists, where a rejection has nowhere useful to go - the event manager does not catch
// listener errors, and throwing from teardown would bury the outcome of the work it cleans up after.
this.#persistStateQuietly = async (eventData) =>
@@ -200,7 +208,8 @@ export class RecoverableState<TStateModel = Record<string, unknown>, TPersistedS
* is no record to restore.
*
* Calling this again after a {@apilink RecoverableState.teardown} starts a new persistence window - the
- * listener is registered again and the record reloaded.
+ * listener is registered again. The record is not reloaded: the in-memory state is what the teardown wrote, and
+ * reloading it would only drop whatever the deserialization leaves out.
*
* @returns The loaded state object
*/
@@ -210,10 +219,7 @@ export class RecoverableState<TStateModel = Record<string, unknown>, TPersistedS
}
if (this.#persistenceEnabled) {
- this.#keyValueStore ??= KeyValueStore.open(null, {
- configuration: this.#configuration ?? serviceLocator.getConfiguration(),
- });
- await this.#resolveKeyValueStore();
+ await this.#openKeyValueStore();
serviceLocator.getEventManager().on(EventType.PERSIST_STATE, this.#persistStateQuietly);
this.#listening = true;
}
@@ -223,7 +229,10 @@ export class RecoverableState<TStateModel = Record<string, unknown>, TPersistedS
this.#initialized = true;
this.#state ??= this.#defaultState();
- await this.#loadSavedState();
+ if (!this.#recordLoaded) {
+ this.#recordLoaded = true;
+ await this.#loadSavedState();
+ }
return this.currentValue;
}
@@ -281,7 +290,7 @@ export class RecoverableState<TStateModel = Record<string, unknown>, TPersistedS
* one would write the record straight back. Use {@apilink RecoverableState.reset} to reset the state itself,
* or {@apilink RecoverableState.teardown} before clearing the record.
*
- * A no-op if persistence is disabled or no KeyValueStore is available yet.
+ * A no-op if persistence is disabled.
*/
async resetStore(): Promise<void> {
if (this.#listening) {
@@ -294,11 +303,7 @@ export class RecoverableState<TStateModel = Record<string, unknown>, TPersistedS
return;
}
- const keyValueStore = await this.#resolveKeyValueStore();
-
- if (keyValueStore === null) {
- return;
- }
+ const keyValueStore = await this.#openKeyValueStore();
await this.#withTimeout(
async () => keyValueStore.setValue(this.#persistStateKey, null),
@@ -336,6 +341,15 @@ export class RecoverableState<TStateModel = Record<string, unknown>, TPersistedS
);
}
+ /** Falls back to the default store when none was supplied - persistence has to write somewhere. */
+ async #openKeyValueStore(): Promise<KeyValueStore> {
+ this.#keyValueStore ??= KeyValueStore.open(null, {
+ configuration: this.#configuration ?? serviceLocator.getConfiguration(),
+ });
+
+ return (await this.#resolveKeyValueStore())!;
+ }
+
/** Awaits a store handed over as a pending `open()`, keeping the resolved instance for later calls. */
async #resolveKeyValueStore(): Promise<KeyValueStore | null> {
if (this.#keyValueStore === null) {
diff --git a/packages/core/src/session_pool/session.ts b/packages/core/src/session_pool/session.ts
index e37e048ac..f795526ff 100644
--- a/packages/core/src/session_pool/session.ts
+++ b/packages/core/src/session_pool/session.ts
@@ -265,7 +265,7 @@ export class Session implements ISession {
/**
* Gets session state for persistence in KeyValueStore.
- * @returns Represents session internal state.
+ * @internal
*/
getState(): SessionState {
return {
diff --git a/packages/core/src/session_pool/session_pool.ts b/packages/core/src/session_pool/session_pool.ts
index 3d6972b6a..4114e8e93 100644
--- a/packages/core/src/session_pool/session_pool.ts
+++ b/packages/core/src/session_pool/session_pool.ts
@@ -1,11 +1,10 @@
-import type { Dictionary, ISessionPool } from '@crawlee/types';
+import type { Dictionary, ISessionPool, SessionState } from '@crawlee/types';
import { AsyncQueue } from '@sapphire/async-queue';
import { z } from 'zod';
import type { PersistenceOptions } from '../crawlers/statistics.js';
-import type { EventManager } from '../events/event_manager.js';
-import { EventType } from '../events/event_manager.js';
import type { CrawleeLogger } from '../log.js';
+import { RecoverableState } from '../recoverable_state.js';
import { serviceLocator } from '../service_locator.js';
import { KeyValueStore } from '../storages/key_value_store.js';
import { parseArgument, schemas, validators } from '../validators.js';
@@ -34,6 +33,68 @@ const createSessionOptionsSchema = z.strictObject({
sessionOptions: schemas.anyObject.default(() => ({})),
});
+// Only what the recreation reads is pinned down - the rest of a session's state is the session's own business.
+const persistedSessionState = z.custom<SessionState>(
+ (value) =>
+ typeof value === 'object' &&
+ value !== null &&
+ typeof (value as Dictionary).createdAt === 'string' &&
+ typeof (value as Dictionary).expiresAt === 'string',
+ 'Expected a persisted session with `createdAt` and `expiresAt` timestamps',
+);
+
+const persistedSessionPoolState = z.object({
+ usableSessionsCount: z.number(),
+ retiredSessionsCount: z.number(),
+ sessions: z.array(persistedSessionState),
+});
+
+const sessionPoolInternalState = z.object({
+ sessions: z.array(z.instanceof(Session)),
+});
+
+/** The live state of a {@apilink SessionPool}, as kept by its {@apilink RecoverableState}. */
+type SessionPoolInternalState = z.infer<typeof sessionPoolInternalState>;
+
+/**
+ * The shape of the persisted {@apilink SessionPool} record, as returned by {@apilink SessionPool.getState}.
+ * @internal
+ */
+export type SessionPoolPersistedState = z.infer<typeof persistedSessionPoolState>;
+
+/**
+ * The conversion between the pool's sessions and its persisted record. Decoding recreates the sessions through
+ * the pool's factory and keeps only the usable ones, which is why the codec is built per pool.
+ */
+function buildSessionPoolStateCodec(pool: {
+ recreateSession: (sessionState: Dictionary) => Promise<Session>;
+ toRecord: (sessions: Session[]) => SessionPoolPersistedState;
+ log: CrawleeLogger;
+}) {
+ return z.codec(persistedSessionPoolState, sessionPoolInternalState, {
+ decode: async (record) => {
+ const sessions: Session[] = [];
+
+ for (const sessionState of record.sessions) {
+ const session = await pool.recreateSession({
+ ...sessionState,
+ createdAt: new Date(sessionState.createdAt),
+ expiresAt: new Date(sessionState.expiresAt),
+ });
+
+ if (session.isUsable()) {
+ sessions.push(session);
+ }
+ }
+
+ pool.log.debug(`${sessions.length} active sessions loaded from KeyValueStore`);
+
+ return { sessions };
+ },
+ encode: ({ sessions }) => pool.toRecord(sessions),
+ });
+}
+
/**
* Factory user-function which creates customized {@apilink Session} instances.
*/
@@ -153,19 +214,22 @@ export class SessionPool implements ISessionPool {
readonly id: string;
#log: CrawleeLogger;
- #sessions: Session[] = [];
#maxPoolSize: number;
#createSessionFunction: CreateSession;
- #keyValueStore?: KeyValueStore;
#sessionMap = new Map<string, Session>();
#sessionOptions: SessionOptions;
#persistStateKeyValueStoreId?: string;
#persistStateKey: string;
- #listener?: () => Promise<void>;
- #events: EventManager;
#persistenceOptions: PersistenceOptions;
#sessionReuseStrategy: SessionReuseStrategy;
+ /**
+ * The sessions live in a {@apilink RecoverableState} that handles the loading, recreation and periodic
+ * persistence of the record under `persistStateKey`.
+ */
+ readonly #state: RecoverableState<SessionPoolInternalState, SessionPoolPersistedState>;
+ readonly #stateCodec: ReturnType<typeof buildSessionPoolStateCodec>;
+
#initPromise?: Promise<void>;
#queue = new AsyncQueue();
#roundRobinIndex = 0;
@@ -185,7 +249,6 @@ export class SessionPool implements ISessionPool {
this.id = id != null ? String(id) : String(SessionPool.#nextId++);
this.#sessionReuseStrategy = sessionReuseStrategy;
- this.#events = serviceLocator.getEventManager();
this.#log = log.child({ prefix: 'SessionPool' });
this.#persistenceOptions = persistenceOptions;
@@ -204,6 +267,30 @@ export class SessionPool implements ISessionPool {
// Session keyValueStore
this.#persistStateKeyValueStoreId = persistStateKeyValueStoreId;
this.#persistStateKey = persistStateKey ?? `${PERSIST_STATE_KEY}_${this.id}`;
+
+ this.#stateCodec = buildSessionPoolStateCodec({
+ recreateSession: (sessionState) => this.invokeCreateSessionFunction(sessionState),
+ toRecord: (sessions) => this.#buildPersistedState(sessions),
+ log: this.#log,
+ });
+
+ this.#state = new RecoverableState({
+ defaultState: () => ({ sessions: [] as Session[] }),
+ persistStateKey: this.#persistStateKey,
+ persistenceEnabled: !!this.#persistenceOptions.enable,
+ logger: this.#log,
+ keyValueStore:
+ this.#persistenceOptions.enable && persistStateKeyValueStoreId
+ ? KeyValueStore.open(
+ { id: persistStateKeyValueStoreId },
+ { configuration: serviceLocator.getConfiguration() },
+ )
+ : undefined,
+ // The codec validates in the decode direction, so it is a Standard Schema as-is; encoding needs a call.
+ deserialize: this.#stateCodec,
+ serialize: (state) => this.#stateCodec.encode(state),
+ });
+ this.#state.reset();
}
/**
@@ -211,7 +298,7 @@ export class SessionPool implements ISessionPool {
*/
async usableSessionsCount(): Promise<number> {
await this.ensureInitialized();
- return this.#sessions.filter((session) => session.isUsable()).length;
+ return this.#state.currentValue.sessions.filter((session) => session.isUsable()).length;
}
/**
@@ -219,7 +306,7 @@ export class SessionPool implements ISessionPool {
*/
async retiredSessionsCount(): Promise<number> {
await this.ensureInitialized();
- return this.#sessions.filter((session) => !session.isUsable()).length;
+ return this.#state.currentValue.sessions.filter((session) => !session.isUsable()).length;
}
/**
@@ -234,28 +321,11 @@ export class SessionPool implements ISessionPool {
}
private async setupPool(): Promise<void> {
- if (!this.#persistenceOptions.enable) {
- return;
- }
-
- this.#keyValueStore = await KeyValueStore.open(
- this.#persistStateKeyValueStoreId ? { id: this.#persistStateKeyValueStoreId } : null,
- {
- configuration: serviceLocator.getConfiguration(),
- },
- );
+ await this.#state.initialize();
- if (!this.#persistStateKeyValueStoreId) {
- this.#log.debug(
- `No 'persistStateKeyValueStoreId' options specified, this session pool's data has been saved in the KeyValueStore with the id: ${this.#keyValueStore.id}`,
- );
+ for (const session of this.#state.currentValue.sessions) {
+ this.#sessionMap.set(session.id, session);
}
-
- // in case of migration happened and SessionPool state should be restored from the keyValueStore.
- await this.maybeLoadSessionPool();
-
- this.#listener = this.persistState.bind(this);
- this.#events.on(EventType.PERSIST_STATE, this.#listener);
}
/**
@@ -332,37 +402,42 @@ export class SessionPool implements ISessionPool {
}
/**
- * @param options - Override the persistence options provided in the constructor
+ * Discards all sessions in the pool, both usable and retired, resetting it to a blank state.
+ *
+ * The persisted record is left alone - use {@apilink SessionPool.resetStore} to clear that as well.
*/
- async resetStore(options?: PersistenceOptions) {
- if (!this.#persistenceOptions.enable && !options?.enable) {
- return;
- }
+ reset(): void {
+ this.#state.reset();
+ this.#sessionMap.clear();
+ this.#roundRobinIndex = 0;
+ }
- await this.ensureInitialized();
- await this.#keyValueStore?.setValue(this.#persistStateKey, null);
+ /**
+ * Clears the persisted pool record, leaving the in-memory sessions alone.
+ *
+ * Throws while the state is still being persisted periodically - the next PERSIST_STATE event would write
+ * the record straight back. Call {@apilink SessionPool.teardown} first, or use {@apilink SessionPool.reset}
+ * to discard the sessions themselves. A no-op if persistence is disabled.
+ */
+ async resetStore(): Promise<void> {
+ await this.#state.resetStore();
}
/**
* Returns an object representing the internal state of the `SessionPool` instance.
- * Note that the object's fields can change in future releases.
+ * @internal
*/
- async getState() {
+ async getState(): Promise<SessionPoolPersistedState> {
await this.ensureInitialized();
- return {
- usableSessionsCount: await this.usableSessionsCount(),
- retiredSessionsCount: await this.retiredSessionsCount(),
- sessions: this.#sessions.map((session) => session.getState()),
- };
+ return this.#buildPersistedState(this.#state.currentValue.sessions);
}
/**
* Persists the current state of the `SessionPool` into the default {@apilink KeyValueStore}.
* The state is persisted automatically in regular intervals.
- * @param options - Override the persistence options provided in the constructor
*/
- async persistState(options?: PersistenceOptions): Promise<void> {
- if (!this.#persistenceOptions.enable && !options?.enable) {
+ async persistState(): Promise<void> {
+ if (!this.#persistenceOptions.enable) {
return;
}
@@ -373,38 +448,36 @@ export class SessionPool implements ISessionPool {
persistStateKey: this.#persistStateKey,
});
- await this.#keyValueStore
- ?.setValue(this.#persistStateKey, await this.getState())
+ await this.#state
+ .persistState()
.catch((error) =>
this.#log.warning(`Failed to persist the session pool stats to ${this.#persistStateKey}`, { error }),
);
}
async [Symbol.asyncDispose](): Promise<void> {
- await this.teardown({ persistState: true });
+ await this.teardown();
}
/**
- * Removes listener from `persistState` event.
+ * Stops the periodic state persistence and persists the state one last time.
* This function should be called after you are done with using the `SessionPool` instance.
- * @param options - Set `persistState` to false when the final state was already persisted by the event manager.
+ * Using the pool again afterwards initializes it anew.
*/
- async teardown({ persistState = true }: { persistState?: boolean } = {}): Promise<void> {
+ async teardown(): Promise<void> {
if (!this.#initPromise) return;
await this.ensureInitialized();
- if (this.#listener) {
- this.#events.off(EventType.PERSIST_STATE, this.#listener);
- }
- if (persistState) {
- await this.persistState();
- }
+ await this.#state.teardown();
+ // The next use initializes the pool again, restoring the record and resuming the periodic persistence -
+ // a crawler that is run twice gets a persisting pool both times.
+ this.#initPromise = undefined;
}
/**
* Removes retired `Session` instances from `SessionPool`.
*/
private removeRetiredSessions() {
- this.#sessions = this.#sessions.filter((storedSession) => {
+ this.#state.currentValue.sessions = this.#state.currentValue.sessions.filter((storedSession) => {
if (storedSession.isUsable()) return true;
this.#sessionMap.delete(storedSession.id);
@@ -419,7 +492,7 @@ export class SessionPool implements ISessionPool {
* @param newSession `Session` instance to be added.
*/
private registerSession(newSession: Session) {
- this.#sessions.push(newSession);
+ this.#state.currentValue.sessions.push(newSession);
this.#sessionMap.set(newSession.id, newSession);
}
@@ -427,7 +500,7 @@ export class SessionPool implements ISessionPool {
* Gets random index.
*/
private getRandomIndex(): number {
- return Math.floor(Math.random() * this.#sessions.length);
+ return Math.floor(Math.random() * this.#state.currentValue.sessions.length);
}
/**
@@ -477,7 +550,7 @@ export class SessionPool implements ISessionPool {
* Decides whether there is enough space for creating new session.
*/
private hasSpaceForSession(): boolean {
- return this.#sessions.length < this.#maxPoolSize;
+ return this.#state.currentValue.sessions.length < this.#maxPoolSize;
}
/**
@@ -488,48 +561,27 @@ export class SessionPool implements ISessionPool {
if (this.#sessionReuseStrategy !== 'use-until-failure' && this.hasSpaceForSession()) return undefined;
if (this.#sessionReuseStrategy === 'use-until-failure') {
- return this.#sessions.find((session) => session.isUsable());
+ return this.#state.currentValue.sessions.find((session) => session.isUsable());
}
let picked: Session;
if (this.#sessionReuseStrategy === 'round-robin') {
- const index = this.#roundRobinIndex % this.#sessions.length;
+ const index = this.#roundRobinIndex % this.#state.currentValue.sessions.length;
this.#roundRobinIndex = index + 1;
- picked = this.#sessions[index];
+ picked = this.#state.currentValue.sessions[index];
} else {
- picked = this.#sessions[this.getRandomIndex()];
+ picked = this.#state.currentValue.sessions[this.getRandomIndex()];
}
return picked.isUsable() ? picked : undefined;
}
- /**
- * Potentially loads `SessionPool`.
- * If the state was persisted it loads the `SessionPool` from the persisted state.
- */
- private async maybeLoadSessionPool(): Promise<void> {
- const loadedSessionPool = await this.#keyValueStore?.getValue<{ sessions: Dictionary[] }>(
- this.#persistStateKey,
- );
-
- if (!loadedSessionPool) return;
-
- // Invalidate old sessions and load active sessions only
- this.#log.debug('Recreating state from KeyValueStore', {
- persistStateKeyValueStoreId: this.#persistStateKeyValueStoreId,
- persistStateKey: this.#persistStateKey,
- });
-
- for (const sessionObject of loadedSessionPool.sessions) {
- sessionObject.createdAt = new Date(sessionObject.createdAt as string);
- sessionObject.expiresAt = new Date(sessionObject.expiresAt as string);
- const recreatedSession = await this.invokeCreateSessionFunction(sessionObject);
-
- if (recreatedSession.isUsable()) {
- this.registerSession(recreatedSession);
- }
- }
-
- this.#log.debug(`${this.#sessions.length} active sessions loaded from KeyValueStore`);
+ /** The persisted form of the pool, which is also what {@apilink SessionPool.getState} returns. */
+ #buildPersistedState(sessions: Session[]): SessionPoolPersistedState {
+ return {
+ usableSessionsCount: sessions.filter((session) => session.isUsable()).length,
+ retiredSessionsCount: sessions.filter((session) => !session.isUsable()).length,
+ sessions: sessions.map((session) => session.getState()),
+ };
}
}
diff --git a/packages/core/src/storages/request_list.ts b/packages/core/src/storages/request_list.ts
index 5fa875702..fa8affde6 100644
--- a/packages/core/src/storages/request_list.ts
+++ b/packages/core/src/storages/request_list.ts
@@ -4,9 +4,9 @@ import { downloadListOfUrls } from '@crawlee/utils';
import { z } from 'zod';
import type { Configuration } from '../configuration.js';
-import { EventType } from '../events/event_manager.js';
import type { CrawleeLogger } from '../log.js';
import type { IProxyConfiguration } from '../proxy_configuration.js';
+import { convertStateSync, RecoverableState } from '../recoverable_state.js';
import { type InternalSource, Request, type RequestOptions, type Source } from '../request.js';
import { createDeserialize, serializeArray } from '../serialization.js';
import { serviceLocator } from '../service_locator.js';
@@ -24,18 +24,28 @@ export const REQUESTS_PERSISTENCE_KEY = 'REQUEST_LIST_REQUESTS';
const CONTENT_TYPE_BINARY = 'application/octet-stream';
+const persistedRequestListState = z.object({
+ nextIndex: z.number().int().nonnegative(),
+ nextUniqueKey: z.string().nullable(),
+ inProgress: z.array(z.string()),
+});
+
+const requestListInternalState = z.object({
+ /** Position of the next request to be processed. */
+ nextIndex: z.number().int().nonnegative(),
+ /** `uniqueKey`s of the requests being processed at the moment. */
+ inProgress: z.set(z.string()),
+});
+
+/** The live crawling position of a {@apilink RequestList}, as kept by its {@apilink RecoverableState}. */
+type RequestListInternalState = z.infer<typeof requestListInternalState>;
+
const requestListOptionsSchema = z.strictObject({
sources: schemas.anyArray.optional(), // check only for array and not subtypes to avoid iteration over the whole thing
sourcesFunction: schemas.anyFunction.optional(),
persistStateKey: z.string().optional(),
persistRequestsKey: z.string().optional(),
- state: z
- .strictObject({
- nextIndex: schemas.anyNumber,
- nextUniqueKey: z.string(),
- inProgress: schemas.anyObject, // persisted as an array of unique keys
- })
- .optional(),
+ state: persistedRequestListState.optional(),
keepDuplicateUrls: z.boolean().default(false),
proxyConfiguration: validators.proxyConfiguration.optional(),
httpClient: schemas.httpClient.optional(),
@@ -43,6 +53,80 @@ const requestListOptionsSchema = z.strictObject({
const listNameSchema = z.string().nullish();
const openOptionsSchema = z.looseObject({});
+/**
+ * The conversion between the crawling position and its persisted record. Decoding also checks the record against
+ * the loaded requests, which is why the codec is built per list.
+ */
+function buildRequestListStateCodec(list: {
+ requests: readonly (Request | RequestOptions)[];
+ indexOf: (uniqueKey: string) => number | undefined;
+ log: CrawleeLogger;
+}) {
+ return z.codec(persistedRequestListState, requestListInternalState, {
+ decode: (record, ctx) => {
+ const { requests } = list;
+ const { nextIndex, nextUniqueKey } = record;
+ const inconsistent = (message: string) => {
+ ctx.issues.push({ code: 'custom', message, input: record });
+ return z.NEVER;
+ };
+
+ if (nextIndex > requests.length) {
+ return inconsistent('The state object is not consistent with RequestList, too few requests loaded.');
+ }
+ if (nextIndex < requests.length && requests[nextIndex].uniqueKey !== nextUniqueKey) {
+ return inconsistent(
+ 'The state object is not consistent with RequestList the order of URLs seems to have changed.',
+ );
+ }
+
+ const inProgress = new Set(record.inProgress);
+ const deleteFromInProgress: string[] = [];
+ for (const uniqueKey of inProgress) {
+ const index = list.indexOf(uniqueKey);
+ if (index === undefined) {
+ return inconsistent(
+ 'The state object is not consistent with RequestList. Unknown uniqueKey is present in the state.',
+ );
+ }
+ if (index >= nextIndex) {
+ deleteFromInProgress.push(uniqueKey);
+ }
+ }
+
+ // WORKAROUND:
+ // It happened to some users that state object contained something like:
+ // {
+ // "nextIndex": 11308,
+ // "nextUniqueKey": "https://www.anychart.com",
+ // "inProgress": {
+ // "https://www.ams360.com": true,
+ // ...
+ // "https://www.anychart.com": true,
+ // }
+ // Which then caused error "The request is not being processed (uniqueKey: https://www.anychart.com)"
+ // As a workaround, we just remove all inProgress requests whose index >= nextIndex,
+ // since they will be crawled again.
+ if (deleteFromInProgress.length) {
+ list.log.warning(
+ "RequestList's in-progress field is not consistent, skipping invalid in-progress entries",
+ { deleteFromInProgress },
+ );
+ for (const uniqueKey of deleteFromInProgress) {
+ inProgress.delete(uniqueKey);
+ }
+ }
+
+ return { nextIndex, inProgress };
+ },
+ encode: ({ nextIndex, inProgress }) => ({
+ nextIndex,
+ nextUniqueKey: nextIndex < list.requests.length ? list.requests[nextIndex].uniqueKey! : null,
+ inProgress: [...inProgress],
+ }),
+ });
+}
+
export interface RequestListOptions {
/**
* An array of sources of URLs for the {@apilink RequestList}. It can be either an array of strings,
@@ -166,7 +250,8 @@ export interface RequestListOptions {
* ```
*
* Note that the preferred (and simpler) way to persist the state of crawling of the `RequestList`
- * is to use the `stateKeyPrefix` parameter instead.
+ * is to use the `stateKeyPrefix` parameter instead. When both are set, a record persisted under
+ * {@apilink RequestListOptions.persistStateKey} takes precedence over this option.
*/
state?: RequestListState;
@@ -269,17 +354,23 @@ export class RequestList implements IRequestLoader {
*/
readonly requests: (Request | RequestOptions)[] = [];
- /** Index to the next item in requests array to fetch. All previous requests are either handled or in progress. */
- #nextIndex = 0;
-
/** Dictionary, key is Request.uniqueKey, value is corresponding index in the requests array. */
#uniqueKeyToIndex: Record<string, number> = {};
+ /**
+ * The crawling position, kept in a {@apilink RecoverableState} that handles the loading, validation and
+ * periodic persistence of the record under `persistStateKey`.
+ */
+ readonly #state: RecoverableState<RequestListInternalState, RequestListState>;
+ readonly #stateCodec: ReturnType<typeof buildRequestListStateCodec>;
+
/**
* Set of `uniqueKey`s of requests that were returned by fetchNextRequest().
* @internal
*/
- inProgress = new Set<string>();
+ get inProgress(): Set<string> {
+ return this.#state.currentValue.inProgress;
+ }
/**
* `uniqueKey`s of requests that were in progress when the state was last persisted and thus need to be
@@ -288,12 +379,6 @@ export class RequestList implements IRequestLoader {
*/
#requestsToRetry: string[] = [];
- /**
- * Starts as true because until we handle the first request, the list is effectively persisted by doing nothing.
- * @internal
- */
- isStatePersisted = true;
-
/**
* Starts as false because we don't know yet and sources might change in the meantime (eg. download from live list).
* @internal
@@ -304,7 +389,6 @@ export class RequestList implements IRequestLoader {
#isInitialized = false;
#persistStateKey?: string;
#persistRequestsKey?: string;
- #initialState?: RequestListState;
#store?: KeyValueStore;
#keepDuplicateUrls: boolean;
#sources: RequestListSource[];
@@ -335,9 +419,33 @@ export class RequestList implements IRequestLoader {
this.#persistStateKey = persistStateKey ? `CRAWLEE_${persistStateKey}` : persistStateKey;
this.#persistRequestsKey = persistRequestsKey ? `CRAWLEE_${persistRequestsKey}` : persistRequestsKey;
- this.#initialState = state;
this.#httpClient = httpClient;
+ this.#stateCodec = buildRequestListStateCodec({
+ requests: this.requests,
+ indexOf: (uniqueKey) => {
+ const index = this.#uniqueKeyToIndex[uniqueKey];
+ return typeof index === 'number' ? index : undefined;
+ },
+ log: this.#log,
+ });
+
+ const stateKey = this.#persistStateKey ?? STATE_PERSISTENCE_KEY;
+ this.#state = new RecoverableState({
+ // The `state` option is where the list starts when there is no persisted record to restore. The factory
+ // runs on `initialize()`, once the requests it is validated against are loaded.
+ defaultState: () =>
+ state
+ ? convertStateSync(this.#stateCodec, state, stateKey)
+ : { nextIndex: 0, inProgress: new Set<string>() },
+ persistStateKey: stateKey,
+ persistenceEnabled: !!this.#persistStateKey,
+ logger: this.#log,
+ // The codec validates in the decode direction, so it is a Standard Schema as-is; encoding needs a call.
+ deserialize: this.#stateCodec,
+ serialize: (internalState) => this.#stateCodec.encode(internalState),
+ });
+
// If this option is set then all requests will get a pre-generated unique ID and duplicate URLs will be kept in the list.
this.#keepDuplicateUrls = keepDuplicateUrls;
@@ -347,8 +455,6 @@ export class RequestList implements IRequestLoader {
// The proxy configuration used for `requestsFromUrl` requests.
this.#proxyConfiguration = proxyConfiguration;
-
- this.persistState = this.persistState.bind(this);
}
/**
@@ -363,22 +469,28 @@ export class RequestList implements IRequestLoader {
this.#isLoading = true;
await purgeDefaultStorages({ onlyPurgeOnce: true });
- const [state, persistedRequests] = await this.loadStateAndPersistedRequests();
+ let persistedRequests: Buffer | undefined;
+ if (this.#persistRequestsKey) {
+ persistedRequests = await this.getPersistedState(this.#persistRequestsKey);
+ if (persistedRequests)
+ this.#log.debug('Loaded requests from key value store using the persistRequestsKey.');
+ }
// Add persisted requests / new sources in a memory efficient way because with very
// large lists, we were running out of memory.
if (persistedRequests) {
- await this.addPersistedRequests(persistedRequests as Buffer);
+ await this.addPersistedRequests(persistedRequests);
} else {
await this.addRequestsFromSources();
}
- this.restoreState(state as RequestListState);
+ // Restores the crawling position - the requests have to be loaded first, as the record is validated
+ // against them. Also starts the periodic state persistence when `persistStateKey` is set.
+ await this.#state.initialize();
+ // All in-progress requests were interrupted and need to be re-crawled.
+ this.#requestsToRetry = [...this.inProgress];
this.#isInitialized = true;
if (this.#persistRequestsKey && !this.areRequestsPersisted) await this.persistRequests();
- if (this.#persistStateKey) {
- serviceLocator.getEventManager().on(EventType.PERSIST_STATE, this.persistState);
- }
return this;
}
@@ -457,11 +569,8 @@ export class RequestList implements IRequestLoader {
if (!this.#persistStateKey) {
throw new Error('Cannot persist state. options.persistStateKey is not set.');
}
- if (this.isStatePersisted) return;
try {
- this.#store ??= await KeyValueStore.open();
- await this.#store.setValue(this.#persistStateKey, this.getState());
- this.isStatePersisted = true;
+ await this.#state.persistState();
} catch (e) {
const err = e as Error;
this.#log.exception(err, 'Attempted to persist state, but failed.');
@@ -474,11 +583,7 @@ export class RequestList implements IRequestLoader {
* leaking the listener (and the requests it retains) on the shared event manager.
*/
async teardown(): Promise<void> {
- serviceLocator.getEventManager().off(EventType.PERSIST_STATE, this.persistState);
-
- if (this.#persistStateKey) {
- await this.persistState();
- }
+ await this.#state.teardown();
}
/**
@@ -493,98 +598,6 @@ export class RequestList implements IRequestLoader {
this.areRequestsPersisted = true;
}
- /**
- * Restores RequestList state from a state object.
- */
- private restoreState(state?: RequestListState): void {
- // If there's no state it means we've not persisted any (yet).
- if (!state) return;
- // Restore previous state.
- if (typeof state.nextIndex !== 'number' || state.nextIndex < 0) {
- throw new Error('The state object is invalid: nextIndex must be a non-negative number.');
- }
- if (state.nextIndex > this.requests.length) {
- throw new Error('The state object is not consistent with RequestList, too few requests loaded.');
- }
- if (
- state.nextIndex < this.requests.length &&
- this.requests[state.nextIndex].uniqueKey !== state.nextUniqueKey
- ) {
- throw new Error(
- 'The state object is not consistent with RequestList the order of URLs seems to have changed.',
- );
- }
-
- const deleteFromInProgress: string[] = [];
- state.inProgress.forEach((uniqueKey) => {
- const index = this.#uniqueKeyToIndex[uniqueKey];
- if (typeof index !== 'number') {
- throw new Error(
- 'The state object is not consistent with RequestList. Unknown uniqueKey is present in the state.',
- );
- }
- if (index >= state.nextIndex) {
- deleteFromInProgress.push(uniqueKey);
- }
- });
-
- this.#nextIndex = state.nextIndex;
- this.inProgress = new Set(state.inProgress);
-
- // WORKAROUND:
- // It happened to some users that state object contained something like:
- // {
- // "nextIndex": 11308,
- // "nextUniqueKey": "https://www.anychart.com",
- // "inProgress": {
- // "https://www.ams360.com": true,
- // ...
- // "https://www.anychart.com": true,
- // }
- // Which then caused error "The request is not being processed (uniqueKey: https://www.anychart.com)"
- // As a workaround, we just remove all inProgress requests whose index >= nextIndex,
- // since they will be crawled again.
- if (deleteFromInProgress.length) {
- this.#log.warning(
- "RequestList's in-progress field is not consistent, skipping invalid in-progress entries",
- {
- deleteFromInProgress,
- },
- );
- for (const uniqueKey of deleteFromInProgress) {
- this.inProgress.delete(uniqueKey);
- }
- }
-
- // All in-progress requests were interrupted and need to be re-crawled.
- this.#requestsToRetry = [...this.inProgress];
- }
-
- /**
- * Attempts to load state and requests using the `RequestList` configuration
- * and returns a tuple of [state, requests] where each may be null if not loaded.
- */
- private async loadStateAndPersistedRequests(): Promise<[RequestListState, Buffer]> {
- let state!: RequestListState;
- let persistedRequests!: Buffer;
-
- if (this.#initialState) {
- state = this.#initialState;
- this.#log.debug('Loaded state from options.state argument.');
- } else if (this.#persistStateKey) {
- state = await this.getPersistedState(this.#persistStateKey);
- if (state) this.#log.debug('Loaded state from key value store using the persistStateKey.');
- }
-
- if (this.#persistRequestsKey) {
- persistedRequests = await this.getPersistedState(this.#persistRequestsKey);
- if (persistedRequests)
- this.#log.debug('Loaded requests from key value store using the persistRequestsKey.');
- }
-
- return [state, persistedRequests];
- }
-
/**
* Returns an object representing the internal state of the `RequestList` instance.
* Note that the object's fields can change in future releases.
@@ -592,11 +605,7 @@ export class RequestList implements IRequestLoader {
getState(): RequestListState {
this.ensureIsInitialized();
- return {
- nextIndex: this.#nextIndex,
- nextUniqueKey: this.#nextIndex < this.requests.length ? this.requests[this.#nextIndex].uniqueKey! : null,
- inProgress: [...this.inProgress],
- };
+ return this.#stateCodec.encode(this.#state.currentValue);
}
/**
@@ -605,7 +614,7 @@ export class RequestList implements IRequestLoader {
async checkReadiness(): Promise<RequestLoaderStatus> {
this.ensureIsInitialized();
- if (this.#requestsToRetry.length > 0 || this.#nextIndex < this.requests.length) {
+ if (this.#requestsToRetry.length > 0 || this.#state.currentValue.nextIndex < this.requests.length) {
return { status: 'ready' };
}
@@ -627,12 +636,12 @@ export class RequestList implements IRequestLoader {
}
// Otherwise return next request.
- if (this.#nextIndex < this.requests.length) {
- const index = this.#nextIndex;
+ const state = this.#state.currentValue;
+ if (state.nextIndex < this.requests.length) {
+ const index = state.nextIndex;
const request = this.requests[index];
- this.inProgress.add(request.uniqueKey!);
- this.#nextIndex++;
- this.isStatePersisted = false;
+ state.inProgress.add(request.uniqueKey!);
+ state.nextIndex++;
return this.ensureRequest(request, index);
}
@@ -670,7 +679,6 @@ export class RequestList implements IRequestLoader {
this.ensureIsInitialized();
this.inProgress.delete(uniqueKey);
- this.isStatePersisted = false;
}
/**
@@ -815,7 +823,7 @@ export class RequestList implements IRequestLoader {
async getPendingCount(): Promise<number> {
this.ensureIsInitialized();
- return this.requests.length - (this.#nextIndex - this.inProgress.size);
+ return this.requests.length - (this.#state.currentValue.nextIndex - this.inProgress.size);
}
/**
@@ -837,7 +845,7 @@ export class RequestList implements IRequestLoader {
async getHandledCount(): Promise<number> {
this.ensureIsInitialized();
- return this.#nextIndex - this.inProgress.size;
+ return this.#state.currentValue.nextIndex - this.inProgress.size;
}
/**
diff --git a/packages/types/src/session.ts b/packages/types/src/session.ts
index a0faa9438..628de404e 100644
--- a/packages/types/src/session.ts
+++ b/packages/types/src/session.ts
@@ -85,6 +85,7 @@ export interface SessionFingerprint {
/**
* Persistable {@apilink Session} state.
+ * @internal
*/
export interface SessionState {
id: string;
diff --git a/test/core/crawlers/basic_crawler.test.ts b/test/core/crawlers/basic_crawler.test.ts
index 86f1665c6..bd12bf58e 100644
--- a/test/core/crawlers/basic_crawler.test.ts
+++ b/test/core/crawlers/basic_crawler.test.ts
@@ -27,6 +27,7 @@ import {
Request,
RequestList,
RequestManagerTandem,
+ STATE_PERSISTENCE_KEY,
RequestQueue,
RequestValidationError,
Router,
@@ -1031,8 +1032,11 @@ describe('BasicCrawler', () => {
});
let finished = false;
- // Mock the call to persist state.
- setValueSpy.mockImplementationOnce(persistResolve as any);
+ // Resolve on the RequestList's state write triggered by the event - the periodic persistence also
+ // writes it once at run start, before anything was processed.
+ setValueSpy.mockImplementation(async (key) => {
+ if (key.endsWith(STATE_PERSISTENCE_KEY) && processed.length > 0) persistResolve();
+ });
// The crawler will pause after 200 requests
const runPromise = basicCrawler.run();
void runPromise.then(() => {
diff --git a/test/core/recoverable_state.test.ts b/test/core/recoverable_state.test.ts
index 116555df2..4d165ab75 100644
--- a/test/core/recoverable_state.test.ts
+++ b/test/core/recoverable_state.test.ts
@@ -165,6 +165,26 @@ describe('RecoverableState', () => {
await recoverableState.teardown();
});
+ test('should keep the in-memory state when initialized again after a teardown', async () => {
+ const store = await KeyValueStore.open();
+
+ const recoverableState = new RecoverableState({
+ defaultState,
+ persistStateKey: 'test-key',
+ persistenceEnabled: true,
+ });
+
+ await recoverableState.initialize();
+ recoverableState.currentValue.counter = 42;
+ await recoverableState.teardown();
+
+ // Something the deserialization would not bring back - here, a record changed behind our back.
+ await store.setValue('test-key', { ...defaultState, counter: 7 });
+
+ expect((await recoverableState.initialize()).counter).toBe(42);
+ await recoverableState.teardown();
+ });
+
test('should warn rather than throw when a periodic persist fails', async () => {
const store = await KeyValueStore.open();
vi.spyOn(store, 'setValue').mockRejectedValue(new Error('store is on fire'));
@@ -563,14 +583,21 @@ describe('RecoverableState', () => {
await expect(recoverableState.resetStore()).resolves.not.toThrow();
});
- test('resetStore should be a no-op without a store', async () => {
+ test('resetStore before the first initialize should clear the record in the default store', async () => {
+ const store = await KeyValueStore.open();
+ await store.setValue('test-key', { ...defaultState, counter: 42 });
+
const recoverableState = new RecoverableState({
defaultState,
persistStateKey: 'test-key',
persistenceEnabled: true,
});
- await expect(recoverableState.resetStore()).resolves.not.toThrow();
+ await recoverableState.resetStore();
+
+ expect(await store.getValue('test-key')).toBeNull();
+ expect(await recoverableState.initialize()).toMatchObject({ counter: 0 });
+ await recoverableState.teardown();
});
});
@@ -644,6 +671,20 @@ describe('RecoverableState', () => {
});
});
+ test('should surface a rejected pending open() on first use, not as an unhandled rejection', async () => {
+ const recoverableState = new RecoverableState({
+ defaultState,
+ persistStateKey: 'test-key',
+ persistenceEnabled: true,
+ keyValueStore: Promise.reject(new Error('store exploded')),
+ });
+
+ // Give an unhandled rejection the chance to fire - vitest would fail the test if it did.
+ await sleep(10);
+
+ await expect(recoverableState.initialize()).rejects.toThrow('store exploded');
+ });
+
test('should time out a persistence call that takes too long', async () => {
const store = await KeyValueStore.open();
vi.spyOn(store, 'setValue').mockImplementation(async () => sleep(1000));
diff --git a/test/core/request_list.test.ts b/test/core/request_list.test.ts
index a659ea2a9..38be9ec81 100644
--- a/test/core/request_list.test.ts
+++ b/test/core/request_list.test.ts
@@ -11,6 +11,7 @@ import {
REQUESTS_PERSISTENCE_KEY,
serviceLocator,
STATE_PERSISTENCE_KEY,
+ StateValidationError,
} from '@crawlee/core';
import { BaseHttpClient } from '@crawlee/http-client';
import { sleep } from '@crawlee/utils';
@@ -336,27 +337,23 @@ describe('RequestList', () => {
const optsCopy = JSON.parse(JSON.stringify(opts));
const requestList = await RequestList.open(opts);
- expect(requestList.isStatePersisted).toBe(true);
-
- // Fetch one request and check that state is not persisted.
await requestList.fetchNextRequest();
- expect(requestList.isStatePersisted).toBe(false);
// Persist state.
setValueSpy.mockResolvedValueOnce();
serviceLocator.getEventManager().emit(EventType.PERSIST_STATE);
await sleep(20);
- expect(requestList.isStatePersisted).toBe(true);
+ expect(setValueSpy).toHaveBeenCalledTimes(1);
+ expect(setValueSpy).toHaveBeenLastCalledWith(`CRAWLEE_${PERSIST_STATE_KEY}`, requestList.getState());
// Do some other changes and persist it again.
const request2 = await requestList.fetchNextRequest();
- expect(requestList.isStatePersisted).toBe(false);
await requestList.markRequestAsHandled(request2!);
- expect(requestList.isStatePersisted).toBe(false);
setValueSpy.mockResolvedValueOnce();
serviceLocator.getEventManager().emit(EventType.PERSIST_STATE);
await sleep(20);
- expect(requestList.isStatePersisted).toBe(true);
+ expect(setValueSpy).toHaveBeenCalledTimes(2);
+ expect(setValueSpy).toHaveBeenLastCalledWith(`CRAWLEE_${PERSIST_STATE_KEY}`, requestList.getState());
// Now initiate new request list from saved state and check that it's same as state
// of original request list.
@@ -365,6 +362,37 @@ describe('RequestList', () => {
expect(requestList2.getState()).toEqual(requestList.getState());
});
+ test('a persisted record that does not match the sources fails validation', async () => {
+ const sources = [1, 2, 3].map((i) => ({ url: `https://example.com/${i}` }));
+ const store = await KeyValueStore.open();
+ await store.setValue('CRAWLEE_state-key', {
+ nextIndex: 1,
+ nextUniqueKey: 'https://example.com/3',
+ inProgress: [],
+ });
+
+ await expect(RequestList.open({ sources, persistStateKey: 'state-key' })).rejects.toThrow(StateValidationError);
+ });
+
+ test('a persisted record takes precedence over the state option', async () => {
+ const sources = [1, 2, 3].map((i) => ({ url: `https://example.com/${i}` }));
+ const store = await KeyValueStore.open();
+ await store.setValue('CRAWLEE_state-key', {
+ nextIndex: 2,
+ nextUniqueKey: 'https://example.com/3',
+ inProgress: [],
+ });
+
+ const requestList = await RequestList.open({
+ sources,
+ persistStateKey: 'state-key',
+ state: { nextIndex: 1, nextUniqueKey: 'https://example.com/2', inProgress: [] },
+ });
+
+ expect(requestList.getState().nextIndex).toBe(2);
+ await requestList.teardown();
+ });
+
test('teardown removes the persist state listener when persistStateKey is set', async () => {
const events = serviceLocator.getEventManager();
const listenerCountBefore = events.listenerCount(EventType.PERSIST_STATE);
@@ -587,8 +615,8 @@ describe('RequestList', () => {
// The persistence keys are derived from the list name, which shows in the keys it reads and writes.
expect(keysPassedTo(getValueSpy)).toEqual([
- `${CRAWLEE_KEY}-${STATE_PERSISTENCE_KEY}`,
`${CRAWLEE_KEY}-${REQUESTS_PERSISTENCE_KEY}`,
+ `${CRAWLEE_KEY}-${STATE_PERSISTENCE_KEY}`,
]);
expect(keysPassedTo(setValueSpy)).toEqual([`${CRAWLEE_KEY}-${REQUESTS_PERSISTENCE_KEY}`]);
});
@@ -608,8 +636,8 @@ describe('RequestList', () => {
await expect(rl.checkReadiness()).resolves.toEqual({ status: 'ready' });
expect(keysPassedTo(getValueSpy)).toEqual([
- `${CRAWLEE_KEY}-${STATE_PERSISTENCE_KEY}`,
`${CRAWLEE_KEY}-${REQUESTS_PERSISTENCE_KEY}`,
+ `${CRAWLEE_KEY}-${STATE_PERSISTENCE_KEY}`,
]);
expect(keysPassedTo(setValueSpy)).toEqual([`${CRAWLEE_KEY}-${REQUESTS_PERSISTENCE_KEY}`]);
});
@@ -636,8 +664,8 @@ describe('RequestList', () => {
// The list name wins over the `persistStateKey` option.
expect(keysPassedTo(getValueSpy)).toEqual([
- `${CRAWLEE_KEY}-${STATE_PERSISTENCE_KEY}`,
`${CRAWLEE_KEY}-${REQUESTS_PERSISTENCE_KEY}`,
+ `${CRAWLEE_KEY}-${STATE_PERSISTENCE_KEY}`,
]);
expect(keysPassedTo(setValueSpy)).toEqual([`${CRAWLEE_KEY}-${REQUESTS_PERSISTENCE_KEY}`]);
});
diff --git a/test/core/session_pool/session_pool.test.ts b/test/core/session_pool/session_pool.test.ts
index 3600b75a1..d3f625054 100644
--- a/test/core/session_pool/session_pool.test.ts
+++ b/test/core/session_pool/session_pool.test.ts
@@ -1,4 +1,12 @@
-import { EventType, KeyValueStore, MemoryStorageBackend, serviceLocator, Session, SessionPool } from '@crawlee/core';
+import {
+ EventType,
+ KeyValueStore,
+ MemoryStorageBackend,
+ serviceLocator,
+ Session,
+ SessionPool,
+ StateValidationError,
+} from '@crawlee/core';
import type { SessionOptions } from '@crawlee/core';
describe('SessionPool - testing session pool', () => {
@@ -18,6 +26,87 @@ describe('SessionPool - testing session pool', () => {
expect(sessionPool.id).toBeDefined();
});
+ test('a corrupt persisted record fails validation instead of crashing the recreation', async () => {
+ const persistStateKey = 'CORRUPT_TEST';
+ const kvStore = await KeyValueStore.open();
+ await kvStore.setValue(persistStateKey, { sessions: [{ id: 'no-timestamps' }] });
+
+ sessionPool = new SessionPool({ persistStateKey });
+
+ await expect(sessionPool.getSession()).rejects.toThrow(StateValidationError);
+ });
+
+ test('should initialize again on use after teardown, resuming the periodic persistence', async () => {
+ const events = serviceLocator.getEventManager();
+ sessionPool = new SessionPool({ persistStateKey: 'REINIT_TEST' });
+
+ await sessionPool.getSession();
+ const listenersWhileRunning = events.listenerCount(EventType.PERSIST_STATE);
+
+ await sessionPool.teardown();
+ expect(events.listenerCount(EventType.PERSIST_STATE)).toBe(listenersWhileRunning - 1);
+
+ expect((await sessionPool.getState()).sessions).toHaveLength(1);
+ expect(events.listenerCount(EventType.PERSIST_STATE)).toBe(listenersWhileRunning);
+
+ await sessionPool.teardown();
+ });
+
+ test('reset should discard the sessions but leave the persisted record alone', async () => {
+ const persistStateKey = 'RESET_TEST';
+ sessionPool = new SessionPool({ persistStateKey });
+
+ const session = await sessionPool.getSession();
+ await sessionPool.persistState();
+
+ sessionPool.reset();
+
+ expect((await sessionPool.getState()).sessions).toEqual([]);
+ expect(await sessionPool.getSession(session!.id)).toBeUndefined();
+
+ const kvStore = await KeyValueStore.open();
+ expect(await kvStore.getValue(persistStateKey)).toBeDefined();
+
+ await sessionPool.teardown();
+ });
+
+ test('resetStore should throw while persisting periodically and clear the record after teardown', async () => {
+ const persistStateKey = 'RESET_STORE_TEST';
+ sessionPool = new SessionPool({ persistStateKey });
+
+ await sessionPool.getSession();
+ await sessionPool.persistState();
+
+ await expect(sessionPool.resetStore()).rejects.toThrow('Use reset() to reset the state itself');
+
+ await sessionPool.teardown();
+ const kvStore = await KeyValueStore.open();
+ expect(await kvStore.getValue(persistStateKey)).toBeDefined();
+
+ await sessionPool.resetStore();
+ expect(await kvStore.getValue(persistStateKey)).toBeNull();
+ });
+
+ test('resetStore before first use should clear the record without restoring old sessions', async () => {
+ const persistStateKey = 'RESET_STORE_FRESH_TEST';
+
+ const firstPool = new SessionPool({ persistStateKey });
+ await firstPool.getSession();
+ await firstPool.teardown();
+
+ const kvStore = await KeyValueStore.open();
+ expect(await kvStore.getValue(persistStateKey)).toBeDefined();
+
+ // A custom store id makes the record reachable before the pool initializes.
+ const secondPool = new SessionPool({ persistStateKey, persistStateKeyValueStoreId: kvStore.id });
+ await secondPool.resetStore();
+
+ expect(await kvStore.getValue(persistStateKey)).toBeNull();
+ expect((await secondPool.getState()).sessions).toEqual([]);
+
+ await secondPool.teardown();
+ });
+
test('should override default values', async () => {
let customFunctionCalled = false;
const persistStateKey = 'CUSTOM_KEY';
diff --git a/test/otel/log-forwarding.test.ts b/test/otel/log-forwarding.test.ts
index cf97e8638..3ddfbbfdd 100644
--- a/test/otel/log-forwarding.test.ts
+++ b/test/otel/log-forwarding.test.ts
@@ -68,9 +68,9 @@ describe('log forwarding against the real Crawlee logger', () => {
// `setStatusMessage` is the one place in Crawlee that logs at a level chosen at runtime. It used to reach for
// `logWithLevel`, which is abstract on `BaseCrawleeLogger` and so cannot be patched - the periodic status
// messages never reached OpenTelemetry at all.
- const record = exporter.getFinishedLogRecords().at(-1)!;
- expect(record.body).toBe('Crawled 40/100 pages, 2 failed.');
- expect(record.severityNumber).toBe(SeverityNumber.INFO);
+ const record = exporter.getFinishedLogRecords().find((r) => r.body === 'Crawled 40/100 pages, 2 failed.');
+ expect(record).toBeDefined();
+ expect(record!.severityNumber).toBe(SeverityNumber.INFO);
});
test('maps each level onto the matching severity', () => {