Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.
TypeScript apifyautomationcrawlercrawlingheadlessheadless-chromejavascriptnodejsnpmplaywrightpuppeteerscraperscrapingtypescriptweb-crawlerweb-crawlingweb-scraping
improved code added docs.
Petr Patek42c2297d6b24
diff --git a/docs/api/Session.md b/docs/api/Session.mdnew file mode 100644index 000000000..ae5bb8b07--- /dev/null+++ b/docs/api/Session.md@@ -0,0 +1,157 @@+---+id: session+title: Session+---++<a name="Session"></a>++Class aggregating data for session. Sessions are used to store information such as cookies and can be used for generating fingerprints and proxy+sessions. You can think of a session as one specific user. Session internal state can be enriched with custom user data for example some authorization+tokens and specific headers in general.++- [Session](session)+ - [`new exports.Session(options)`](#new_Session_new)+ - [`.isBlocked()`](#Session+isBlocked) ⇒ `boolean`+ - [`.isExpired()`](#Session+isExpired) ⇒ `boolean`+ - [`.isMaxUsageCountReached()`](#Session+isMaxUsageCountReached) ⇒ `boolean`+ - [`.isUsable()`](#Session+isUsable) ⇒ `boolean`+ - [`.reclaim()`](#Session+reclaim)+ - [`.getState()`](#Session+getState) ⇒ `Object`+ - [`.retire()`](#Session+retire)+ - [`.fail()`](#Session+fail)++<a name="new_Session_new"></a>++## `new exports.Session(options)`++Session configuration.++<table>+<thead>+<tr>+<th>Param</th><th>Type</th>+</tr>+</thead>+<tbody>+<tr>+<td><code>options</code></td><td></td>+</tr>+<tr>+<td colspan="3"></td></tr><tr>+<td><code>options.id</code></td><td><code>String</code></td>+</tr>+<tr>+<td colspan="3"><p>Id of session used for generating fingerprints. It is used as proxy session name.</p>+</td></tr><tr>+<td><code>options.cookies</code></td><td><code>Array</code></td>+</tr>+<tr>+<td colspan="3"><p>Cookies storage per session.</p>+</td></tr><tr>+<td><code>options.maxAgeSecs</code></td><td><code>Number</code></td>+</tr>+<tr>+<td colspan="3"><p>Number of seconds after which the session is considered as expired.</p>+</td></tr><tr>+<td><code>options.userData</code></td><td><code>Object</code></td>+</tr>+<tr>+<td colspan="3"><p>Object where custom user data can be stored. For example custom headers.</p>+</td></tr><tr>+<td><code>options.maxErrorScore</code></td><td><code>number</code></td>+</tr>+<tr>+<td colspan="3"><p>Maximum number of failed session usage.+If the <code>errorScore</code> reaches the <code>maxErrorScore</code> session is marked as block and it is thrown away.</p>+</td></tr><tr>+<td><code>options.errorScoreDecrement</code></td><td><code>number</code></td>+</tr>+<tr>+<td colspan="3"><p>It is used for healing the session.+For example: if your session fails two times, but it is successful on the third attempt it's errorScore is decremented by this number.</p>+</td></tr><tr>+<td><code>options.createdAt</code></td><td><code>Date</code></td>+</tr>+<tr>+<td colspan="3"><p>Date of creation.</p>+</td></tr><tr>+<td><code>options.expiredAt</code></td><td><code>Date</code></td>+</tr>+<tr>+<td colspan="3"><p>Date of expiration.</p>+</td></tr><tr>+<td><code>options.usageCount</code></td><td><code>Number</code></td>+</tr>+<tr>+<td colspan="3"><p>Indicates how many times the session has been used.</p>+</td></tr><tr>+<td><code>options.errorCount</code></td><td><code>Number</code></td>+</tr>+<tr>+<td colspan="3"><p>Indicates how many times the session failed.</p>+</td></tr><tr>+<td><code>options.maxSessionUsageCount</code></td><td><code>Number</code></td>+</tr>+<tr>+<td colspan="3"><p>Session should be used only a limited amount of times.+This number indicates how many times the session is going to be used, before it is thrown away.</p>+</td></tr><tr>+<td><code>options.sessionPool</code></td><td><code>EventEmitter</code></td>+</tr>+<tr>+<td colspan="3"><p>SessionPool instance. Session will emit the <code>sessionRetired</code> event on this instance.</p>+</td></tr></tbody>+</table>+<a name="Session+isBlocked"></a>++## `session.isBlocked()` ⇒ `boolean`++indicates whether the session is blocked. Session is blocked once it reaches the `maxErrorScore`.++<a name="Session+isExpired"></a>++## `session.isExpired()` ⇒ `boolean`++Indicates whether the session is expired. Session expiration is determined by the `maxAgeSecs`. Once the session is older than+`createdAt + maxAgeSecs` the session is considered expired.++<a name="Session+isMaxUsageCountReached"></a>++## `session.isMaxUsageCountReached()` ⇒ `boolean`++Indicates whether the session is used maximum number of times. Session maximum usage count can be changed by `maxSessionUsageCount` parameter.++<a name="Session+isUsable"></a>++## `session.isUsable()` ⇒ `boolean`++Indicates whether the session can be used for next requests. Session is usable when it is not expired, not blocked and the maximum usage count has not+be reached.++<a name="Session+reclaim"></a>++## `session.reclaim()`++This method should be called after a successful session usage. It increases `usageCount` and potentially lowers the `errorScore` by the+`errorScoreDecrement`.++<a name="Session+getState"></a>++## `session.getState()` ⇒ `Object`++Gets session state for persistence in KeyValueStore.++**Returns**: `Object` - represents session internal state.+<a name="Session+retire"></a>++## `session.retire()`++Marks session as blocked and emits event on the `SessionPool` This method should be used if the session usage was unsuccessful and you are sure that+it is because of the session configuration and not any external matters. For example when server returns 403 status code. If the session does not work+due to some external factors as server error such as 5XX you probably want to use `fail` method.++<a name="Session+fail"></a>++## `session.fail()`++Increases usage and error count. Should be used when the session has been used unsuccessfully. For example because of timeouts.diff --git a/docs/api/SessionPool.md b/docs/api/SessionPool.mdnew file mode 100644index 000000000..6fd137cf1--- /dev/null+++ b/docs/api/SessionPool.md@@ -0,0 +1,146 @@+---+id: sessionpool+title: SessionPool+---++<a name="SessionPool"></a>++Handles the sessions rotation, creation and persistence. Creates a pool of [`Session`](session) instances, that are randomly rotated. When some+session is marked as blocked. It is removed and new one is created instead.++Session pool is by default persisted in default [`KeyValueStore`](keyvaluestore). If you want to have one pool for all runs you have to specify+`persistStateKeyValueStoreId`.++**Example usage:**++```javascript+const sessionPool = new SessionPool({+ maxPoolSize: 25,+ maxSessionAgeSecs: 10,+ maxSessionAgeSecs: 10,+ maxSessionUsageCount: 150, // for example when you know that the site blocks after 150 requests.+ persistStateKeyValueStoreId: 'my-key-value-store-for-sessions',+ persistStateKey: 'my-session-pool',+});++// Now you have to initialize the `SessionPool`.+// If you already have a persisted state in the selected `KeyValueState`.+// The Session pool is recreated, otherwise it creates a new one.+// It also attaches listener to `Apify.events` so it is persisted periodically and not after every change.+await sessionPool.initialize();++// Get random session from the pool+const session1 = await sessionPool.retrieveSession();+const session2 = await sessionPool.retrieveSession();+const session3 = await sessionPool.retrieveSession();++// Now you can mark the session either failed of successful++// Fails session -> it increases error count (soft retire)+session1.fail();++// Marks as successful.+session2.reclaim();++// Retires session -> session is removed from the pool+session3.retire();+```++- [SessionPool](sessionpool)+ - [`new exports.SessionPool(options)`](#new_SessionPool_new)+ - [`.usableSessionsCount`](#SessionPool+usableSessionsCount) ⇒ `number`+ - [`.retiredSessionsCount`](#SessionPool+retiredSessionsCount) ⇒ `number`+ - [`.initialize()`](#SessionPool+initialize) ⇒ `Promise<void>`+ - [`.retrieveSession()`](#SessionPool+retrieveSession) ⇒ [`Promise<Session>`](session)+ - [`.getState()`](#SessionPool+getState) ⇒ `Object`+ - [`.persistState()`](#SessionPool+persistState) ⇒ `Promise`++<a name="new_SessionPool_new"></a>++## `new exports.SessionPool(options)`++Session pool configuration.++<table>+<thead>+<tr>+<th>Param</th><th>Type</th>+</tr>+</thead>+<tbody>+<tr>+<td><code>options</code></td><td></td>+</tr>+<tr>+<td colspan="3"></td></tr><tr>+<td><code>options.maxPoolSize</code></td><td><code>Number</code></td>+</tr>+<tr>+<td colspan="3"><p>Maximum size of the pool.+Indicates how many sessions are rotated.</p>+</td></tr><tr>+<td><code>options.maxSessionAgeSecs</code></td><td><code>Number</code></td>+</tr>+<tr>+<td colspan="3"><p>Number of seconds after which the session is marked as expired.</p>+</td></tr><tr>+<td><code>options.maxSessionUsageCount</code></td><td><code>Number</code></td>+</tr>+<tr>+<td colspan="3"><p>Maximum number of uses per session.+It useful, when you know the site rate-limits, so you can retire the session before it gets blocked and let it cool down.</p>+</td></tr><tr>+<td><code>options.persistStateKeyValueStoreId</code></td><td><code>String</code></td>+</tr>+<tr>+<td colspan="3"><p>Name or Id of <code>KeyValueStore</code> where is the <code>SessionPool</code> state stored.</p>+</td></tr><tr>+<td><code>options.persistStateKey</code></td><td><code>String</code></td>+</tr>+<tr>+<td colspan="3"><p>Session pool persists it's state under this key in Key value store.</p>+</td></tr><tr>+<td><code>options.createSessionFunction</code></td><td><code>function</code></td>+</tr>+<tr>+<td colspan="3"><p>Custom function that should return Session instance.</p>+</td></tr></tbody>+</table>+<a name="SessionPool+usableSessionsCount"></a>++## `sessionPool.usableSessionsCount` ⇒ `number`++Gets count of usable sessions in the pool.++<a name="SessionPool+retiredSessionsCount"></a>++## `sessionPool.retiredSessionsCount` ⇒ `number`++Gets count of blocked sessions in the pool.++<a name="SessionPool+initialize"></a>++## `sessionPool.initialize()` ⇒ `Promise<void>`++Starts periodic state persistence and potentially loads SessionPool state from [`KeyValueStore`](keyvaluestore). This function must be called before+you can start using the instance in a meaningful way.++<a name="SessionPool+retrieveSession"></a>++## `sessionPool.retrieveSession()` ⇒ [`Promise<Session>`](session)++Gets session. If there is space for new session, it creates and return new session. If the session pool is full, it picks a session from the pool, If+the picked session is usable it is returned, otherwise it creates and returns a new one.++<a name="SessionPool+getState"></a>++## `sessionPool.getState()` ⇒ `Object`++Returns an object representing the internal state of the `SessionPool` instance. Note that the object's fields can change in future releases.++<a name="SessionPool+persistState"></a>++## `sessionPool.persistState()` ⇒ `Promise`++Persists the current state of the `SessionPool` into the default [`KeyValueStore`](keyvaluestore). The state is persisted automatically in regular+intervals, but calling this method manuallydiff --git a/src/session_pool/events.js b/src/session_pool/events.jsindex 5893200db..08cf9ad35 100644--- a/src/session_pool/events.js+++ b/src/session_pool/events.js@@ -1,3 +1,3 @@export default {- DISCARDED: 'sessionRetired',+ SESSION_RETIRED: 'sessionRetired',};diff --git a/src/session_pool/session.js b/src/session_pool/session.jsindex f56631122..3f0ff1471 100644--- a/src/session_pool/session.js+++ b/src/session_pool/session.js@@ -8,20 +8,40 @@ import EVENTS from './events';/**- * Class aggregating data for `Session`.- * Session internal state can be enriched with custom user data for example some authorization tokens.+ * Class aggregating data for session.+ * Sessions are used to store information such as cookies and can be used for generating fingerprints and proxy sessions.+ * You can think of a session as one specific user.+ * Session internal state can be enriched with custom user data for example some authorization tokens and specific headers in general.*/-export default class Session {+export class Session {+ /**+ * Session configuration.+ * @param options+ * @param options.id {String} - Id of session used for generating fingerprints. It is used as proxy session name.+ * @param options.cookies {Array} - Cookies storage per session.+ * @param options.maxAgeSecs {Number} - Number of seconds after which the session is considered as expired.+ * @param options.userData {Object} - Object where custom user data can be stored. For example custom headers.+ * @param options.maxErrorScore {number} - Maximum number of failed session usage.+ * If the `errorScore` reaches the `maxErrorScore` session is marked as block and it is thrown away.+ * @param options.errorScoreDecrement {number} - It is used for healing the session.+ * For example: if your session fails two times, but it is successful on the third attempt it's errorScore is decremented by this number.+ * @param options.createdAt {Date} - Date of creation.+ * @param options.expiredAt {Date} - Date of expiration.+ * @param options.usageCount {Number} - Indicates how many times the session has been used.+ * @param options.errorCount {Number} - Indicates how many times the session failed.+ * @param options.maxSessionUsageCount {Number} - Session should be used only a limited amount of times.+ * This number indicates how many times the session is going to be used, before it is thrown away.+ * @param options.sessionPool {EventEmitter} - SessionPool instance. Session will emit the `sessionRetired` event on this instance.+ */constructor(options = {}) {const {- id = cryptoRandomObjectId(),+ id = `session-${cryptoRandomObjectId(10)}`,cookies = [],- fingerprintSeed = cryptoRandomObjectId(),maxAgeSecs = 3000,userData = {},maxErrorScore = 3,errorScoreDecrement = 0.5,- createdAt = moment().toISOString(),+ createdAt = moment(),usageCount = 0,errorScore = 0,maxSessionUsageCount = 50,@@ -33,12 +53,11 @@ export default class Session {// ValidationcheckParamOrThrow(id, 'options.id', 'Maybe String');checkParamOrThrow(cookies, 'options.cookies', 'Maybe Array');- checkParamOrThrow(fingerprintSeed, 'options.fingerprintSeed', 'Maybe String');checkParamOrThrow(maxAgeSecs, 'options.maxAgeSecs', 'Maybe Number');checkParamOrThrow(userData, 'options.userData', 'Maybe Object');checkParamOrThrow(maxErrorScore, 'options.maxErrorScore', 'Maybe Number');- checkParamOrThrow(expiresAt, 'options.expiresAt', 'Maybe String');- checkParamOrThrow(createdAt, 'options.createdAt', 'Maybe String');+ checkParamOrThrow(expiresAt, 'options.expiresAt', 'Maybe Object');+ checkParamOrThrow(createdAt, 'options.createdAt', 'Maybe Object');checkParamOrThrow(usageCount, 'options.usageCount', 'Maybe Number');checkParamOrThrow(errorScore, 'options.errorScore', 'Maybe Number');checkParamOrThrow(maxSessionUsageCount, 'options.maxSessionUsageCount', 'Maybe Number');@@ -51,30 +70,29 @@ export default class Session {}if (!expiresAt) {- expiresAt = moment().add(maxAgeSecs, 'seconds').toISOString();+ expiresAt = moment().add(maxAgeSecs, 'seconds');}// Configurablethis.id = id;this.cookies = cookies;- this.fingerprintSeed = fingerprintSeed;this.maxAgeSecs = maxAgeSecs;this.userData = userData;this.maxErrorScore = maxErrorScore;- this.errorScoreDecrement = errorScoreDecrement; // TODO: Better Naming+ this.errorScoreDecrement = errorScoreDecrement;// Internalthis.expiresAt = expiresAt;this.createdAt = createdAt;- this.usageCount = usageCount;- this.errorScore = errorScore;+ this.usageCount = usageCount; // indicates how many times the session has been used+ this.errorScore = errorScore; // indicates number of failed request with the sessionthis.maxSessionUsageCount = maxSessionUsageCount;this.sessionPool = sessionPool;}/**- * Decides whether the `Session` is blocked.- * `Session` is blocked once it reached the maximum error count.+ * indicates whether the session is blocked.+ * Session is blocked once it reaches the `maxErrorScore`.* @return {boolean}*/isBlocked() {@@ -82,7 +100,9 @@ export default class Session {}/**- * Decides whether the `Session` is expired.+ * Indicates whether the session is expired.+ * Session expiration is determined by the `maxAgeSecs`.+ * Once the session is older than `createdAt + maxAgeSecs` the session is considered expired.* @return {boolean}*/isExpired() {@@ -92,24 +112,26 @@ export default class Session {}/**- * Decides whether the `Session` is used maximum number of times.+ * Indicates whether the session is used maximum number of times.+ * Session maximum usage count can be changed by `maxSessionUsageCount` parameter.* @return {boolean}*/- isMaxUseCountReached() {+ isMaxUsageCountReached() {return this.usageCount >= this.maxSessionUsageCount;}/**- * Decides whether the `Session` can be used for next requests.+ * Indicates whether the session can be used for next requests.+ * Session is usable when it is not expired, not blocked and the maximum usage count has not be reached.* @return {boolean}*/isUsable() {- return !this.isBlocked() && !this.isExpired() && !this.isMaxUseCountReached();+ return !this.isBlocked() && !this.isExpired() && !this.isMaxUsageCountReached();}/**- * Marks the `Session` after successful request.- * Increases usage count and if the `Session` had failed in the previous requests it lowers the errorScore to 'heel' itself.+ * This method should be called after a successful session usage.+ * It increases `usageCount` and potentially lowers the `errorScore` by the `errorScoreDecrement`.*/reclaim() {this.usageCount += 1;@@ -121,18 +143,8 @@ export default class Session {}/**- * Gets `Session` state for persistence in KeyValueStore.- * @return {{- * createdAt: string | *,- * userData: {},- * errorScoreDecrement: number,- * maxErrorScore: number,- * id: String,- * cookies: ([]|Array),- * expiresAt: string | *,- * usageCount: number,- * errorScore: number- * }}+ * Gets session state for persistence in KeyValueStore.+ * @return {Object} represents session internal state.*/getState() {return {@@ -141,8 +153,8 @@ export default class Session {userData: this.userData,maxErrorScore: this.maxErrorScore,errorScoreDecrement: this.errorScoreDecrement,- expiresAt: this.expiresAt,- createdAt: this.createdAt,+ expiresAt: this.expiresAt.toISOString(),+ createdAt: this.createdAt.toISOString(),usageCount: this.usageCount,errorScore: this.errorScore,};@@ -150,6 +162,10 @@ export default class Session {/*** Marks session as blocked and emits event on the `SessionPool`+ * This method should be used if the session usage was unsuccessful+ * and you are sure that it is because of the session configuration and not any external matters.+ * For example when server returns 403 status code.+ * If the session does not work due to some external factors as server error such as 5XX you probably want to use `fail` method.*/retire() {// mark it as an invalid by increasing the error score count.@@ -157,12 +173,12 @@ export default class Session {this.usageCount += 1;// emit event so we can retire browser in puppeteer pool- this.sessionPool.emit(EVENTS.DISCARDED, this);+ this.sessionPool.emit(EVENTS.SESSION_RETIRED, this);}/**- * Increases usage and error count- * Should be used unsuccessful request/use with the session+ * Increases usage and error count.+ * Should be used when the session has been used unsuccessfully. For example because of timeouts.*/fail() {this.errorScore += 1;diff --git a/src/session_pool/session_pool.js b/src/session_pool/session_pool.jsindex b8b9c93ba..176b80e07 100644--- a/src/session_pool/session_pool.js+++ b/src/session_pool/session_pool.js@@ -2,36 +2,75 @@ import EventEmitter from 'events';import log from 'apify-shared/log';import { checkParamOrThrow } from 'apify-client/build/utils';+import moment from 'moment';import { openKeyValueStore } from '../key_value_store';-import Session from './session';+import { Session } from './session';import events from '../events';import { ACTOR_EVENT_NAMES_EX } from '../constants';/*** Handles the sessions rotation, creation and persistence.- * Creates a pool of `Session` instances, that are randomly rotated.- * When some `Session` is marked as blocked. It is removed and new one is created instead.+ * Creates a pool of {@link Session} instances, that are randomly rotated.+ * When some session is marked as blocked. It is removed and new one is created instead.*+ * Session pool is by default persisted in default {@link KeyValueStore}.+ * If you want to have one pool for all runs you have to specify `persistStateKeyValueStoreId`.+ *+ * **Example usage:**+ *+ * ```javascript+ * const sessionPool = new SessionPool({+ * maxPoolSize: 25,+ * maxSessionAgeSecs: 10,+ * maxSessionAgeSecs: 10,+ * maxSessionUsageCount: 150, // for example when you know that the site blocks after 150 requests.+ * persistStateKeyValueStoreId: 'my-key-value-store-for-sessions',+ * persistStateKey: 'my-session-pool',+ * });+ *+ * // Now you have to initialize the `SessionPool`.+ * // If you already have a persisted state in the selected `KeyValueState`.+ * // The Session pool is recreated, otherwise it creates a new one.+ * // It also attaches listener to `Apify.events` so it is persisted periodically and not after every change.+ * await sessionPool.initialize();+ *+ * // Get random session from the pool+ * const session1 = await sessionPool.retrieveSession();+ * const session2 = await sessionPool.retrieveSession();+ * const session3 = await sessionPool.retrieveSession();+ *+ * // Now you can mark the session either failed of successful+ *+ * // Fails session -> it increases error count (soft retire)+ * session1.fail()+ *+ * // Marks as successful.+ * session2.reclaim()+ *+ * // Retires session -> session is removed from the pool+ * session3.retire()+ *+ * ```*/--// TODO: We should probably add some debug loging-// TODO: Validation-// TODO: Class docs will be filled once the integration is done.-export default class SessionPool extends EventEmitter {+export class SessionPool extends EventEmitter {/**- *+ * Session pool configuration.* @param options- * @param options.maxPoolSize {Number} - Maximum size of the pool- * @param options.maxSessionAgeSecs {Number}- * @param options.maxSessionReuseCount {Number}- * @param options.persistStateKeyValueStoreId {String}+ * @param options.maxPoolSize {Number} - Maximum size of the pool.+ * Indicates how many sessions are rotated.+ * @param options.maxSessionAgeSecs {Number} - Number of seconds after which the session is marked as expired.+ * @param options.maxSessionUsageCount {Number} - Maximum number of uses per session.+ * It useful, when you know the site rate-limits, so you can retire the session before it gets blocked and let it cool down.+ * @param options.persistStateKeyValueStoreId {String} - Name or Id of `KeyValueStore` where is the `SessionPool` state stored.+ * @param options.persistStateKey {String} - Session pool persists it's state under this key in Key value store.+ * @param options.createSessionFunction {function} - Custom function that should return Session instance.*/constructor(options = {}) {const {maxPoolSize = 1000,maxSessionAgeSecs = 3000,- maxSessionReuseCount = 50,+ maxSessionUsageCount = 50,persistStateKeyValueStoreId = null,persistStateKey = 'SESSION_POOL_STATE',@@ -45,7 +84,7 @@ export default class SessionPool extends EventEmitter {// ValidationcheckParamOrThrow(maxPoolSize, 'options.maxPoolSize', 'Maybe Number');checkParamOrThrow(maxSessionAgeSecs, 'options.maxSessionAgeSecs', 'Maybe Number');- checkParamOrThrow(maxSessionReuseCount, 'options.maxSessionReuseCount', 'Maybe Number');+ checkParamOrThrow(maxSessionUsageCount, 'options.maxSessionUsageCount', 'Maybe Number');checkParamOrThrow(persistStateKeyValueStoreId, 'options.persistStateKeyValueStoreId', 'Maybe String');checkParamOrThrow(persistStateKey, 'options.persistStateKey', 'Maybe String');checkParamOrThrow(createSessionFunction, 'options.createSessionFunction', 'Maybe Function');@@ -57,7 +96,7 @@ export default class SessionPool extends EventEmitter {// Session configuration// @TODO: Maybe options.sessionOptions / this.sessionOptions?this.maxSessionAgeSecs = maxSessionAgeSecs;- this.maxSessionReuseCount = maxSessionReuseCount;+ this.maxSessionUsageCount = maxSessionUsageCount;// Session keyValueStorethis.persistStateKeyValueStoreId = persistStateKeyValueStoreId;@@ -66,28 +105,26 @@ export default class SessionPool extends EventEmitter {// Operative statesthis.keyValueStore = null;this.sessions = [];-- // Maybe we can add onSessionRetired function to configuration ?}/**- * Gets number of active sessions in the pool.+ * Gets count of usable sessions in the pool.* @return {number}*/- get activeSessionsCount() {+ get usableSessionsCount() {return this.sessions.filter(session => session.isUsable()).length;}/**- * Gets number of blocked sessions in the pool.+ * Gets count of blocked sessions in the pool.* @return {number}*/- get blockedSessionsCount() {+ get retiredSessionsCount() {return this.sessions.filter(session => !session.isUsable()).length;}/**- * Starts periodic state persistence and potentially loads SessionPool state from {@link KeyValueStore}.`+ * Starts periodic state persistence and potentially loads SessionPool state from {@link KeyValueStore}.* This function must be called before you can start using the instance in a meaningful way.** @return {Promise<void>}@@ -96,16 +133,16 @@ export default class SessionPool extends EventEmitter {this.keyValueStore = await openKeyValueStore(this.persistStateKeyValueStoreId);// in case of migration happened and SessionPool state should be restored from the keyValueStore.- await this._maybeRecreateSessionPool();+ await this._maybeLoadSessionPool();events.on(ACTOR_EVENT_NAMES_EX.PERSIST_STATE, this.persistState.bind(this));}/**- * Gets `Session`.- * If there is space for new `Session`, it creates and return new `Session`.- * If the `SessionPool` is full, it picks a `Session` from the pool,- * If the picked `Session` is usable it is returned, otherwise it creates and returns a new `Session`.+ * Gets session.+ * If there is space for new session, it creates and return new session.+ * If the session pool is full, it picks a session from the pool,+ * If the picked session is usable it is returned, otherwise it creates and returns a new one.** @return {Promise<Session>}*/@@ -129,19 +166,6 @@ export default class SessionPool extends EventEmitter {return this._createSession();}- /**- * Gets SessionPool statistics.- * These function could be use in the statistics logging in actor run to know how much is the website blocking- *- * @return {{blockedSessionsCount: number, activeSessionsCount: number}}- */- getStats() {- return {- activeSessionsCount: this.activeSessionsCount,- blockedSessionsCount: this.blockedSessionsCount,- };- }-/*** Returns an object representing the internal state of the `SessionPool` instance.* Note that the object's fields can change in future releases.@@ -150,7 +174,8 @@ export default class SessionPool extends EventEmitter {*/getState() {return {- ...this.getStats(),+ usableSessionsCount: this.usableSessionsCount,+ retiredSessionsCount: this.retiredSessionsCount,sessions: this.sessions.map(session => session.getState()),};}@@ -198,20 +223,20 @@ export default class SessionPool extends EventEmitter {}/**- * Creates new `Session` without any extra behavior.+ * Creates new session without any extra behavior.* @return {Session} - New session.* @private*/_defaultCreateSessionFunction() {return new Session({maxSessionAgeSecs: this.maxSessionAgeSecs,- maxSessionReuseCount: this.maxSessionReuseCount,+ maxSessionUsageCount: this.maxSessionUsageCount,sessionPool: this,});}/**- * Creates new `Session` and adds it to the pool.+ * Creates new session and adds it to the pool.* @return {Promise<Session>} - Newly created `Session` instance.* @private*/@@ -225,7 +250,7 @@ export default class SessionPool extends EventEmitter {}/**- * Decides whether there is enough space for creating new `Session`.+ * Decides whether there is enough space for creating new session.* @return {boolean}* @private*/@@ -234,7 +259,7 @@ export default class SessionPool extends EventEmitter {}/**- * Picks random `Session` from the `SessionPool`.+ * Picks random session from the `SessionPool`.* @return {Session} - Picked `Session`* @private*/@@ -243,12 +268,12 @@ export default class SessionPool extends EventEmitter {}/**- * Potentially recreates `SessionPool`.- * If the state was persisted it recreates the `SessionPool` from the persisted state.+ * Potentially loads `SessionPool`.+ * If the state was persisted it loads the `SessionPool` from the persisted state.* @return {Promise<void>}* @private*/- async _maybeRecreateSessionPool() {+ async _maybeLoadSessionPool() {const loadedSessionPool = await this.keyValueStore.getValue(this.persistStateKey);if (!loadedSessionPool) return;@@ -258,6 +283,8 @@ export default class SessionPool extends EventEmitter {for (const sessionObject of loadedSessionPool.sessions) {sessionObject.sessionPool = this;+ sessionObject.createdAt = moment(sessionObject.createdAt);+ sessionObject.expiresAt = moment(sessionObject.expiresAt);const recreatedSession = new Session(sessionObject);if (recreatedSession.isUsable()) {diff --git a/test/session_pool/session.js b/test/session_pool/session.jsindex fe7fff77f..2ebaba3f2 100644--- a/test/session_pool/session.js+++ b/test/session_pool/session.js@@ -1,7 +1,8 @@import { expect } from 'chai';-import Session from '../../build/session_pool/session';-import SessionPool from '../../build/session_pool/session_pool';+import * as moment from 'moment';+import { Session } from '../../build/session_pool/session';+import { SessionPool } from '../../build/session_pool/session_pool';import EVENTS from '../../build/session_pool/events';import Apify from '../../build';@@ -54,7 +55,7 @@ describe('Session - testing session behaviour ', async () => {it('should max out session usage', () => {session.maxSessionUsageCount = 1;session.reclaim();- expect(session.isMaxUseCountReached()).to.be.eql(true);+ expect(session.isMaxUsageCountReached()).to.be.eql(true);expect(session.isUsable()).to.be.eql(false);});@@ -72,7 +73,7 @@ describe('Session - testing session behaviour ', async () => {it('should retire session', () => {let discarded = false;- sessionPool.on(EVENTS.DISCARDED, (ses) => {+ sessionPool.on(EVENTS.SESSION_RETIRED, (ses) => {expect(ses instanceof Session).to.be.eql(true);discarded = true;});@@ -96,7 +97,11 @@ describe('Session - testing session behaviour ', async () => {Object.entries(state).forEach(([key, value]) => {- expect(session[key]).to.be.eql(value);+ if (moment.isMoment(session[key])) {+ expect(session[key].toISOString()).to.be.eql(value);+ } else {+ expect(session[key]).to.be.eql(value);+ }});});});diff --git a/test/session_pool/session_pool.js b/test/session_pool/session_pool.jsindex eab7946c7..bb38ed79c 100644--- a/test/session_pool/session_pool.js+++ b/test/session_pool/session_pool.js@@ -1,7 +1,7 @@import { expect } from 'chai';-import log from 'apify-shared/log';+import * as moment from 'moment';import { LOCAL_STORAGE_DIR, emptyLocalStorageSubdir } from '../_helper';-import SessionPool from '../../build/session_pool/session_pool';+import { SessionPool } from '../../build/session_pool/session_pool';import Apify from '../../build';import events from '../../build/events';@@ -27,7 +27,7 @@ describe('SessionPool - testing session pool', async () => {expect(sessionPool.sessions.length).to.exist; // eslint-disable-lineexpect(sessionPool.maxPoolSize).to.exist; // eslint-disable-lineexpect(sessionPool.maxSessionAgeSecs).to.exist; // eslint-disable-line- expect(sessionPool.maxSessionReuseCount).to.exist; // eslint-disable-line+ expect(sessionPool.maxSessionUsageCount).to.exist; // eslint-disable-lineexpect(sessionPool.persistStateKey).to.exist; // eslint-disable-lineexpect(sessionPool.createSessionFunction).to.be.eql(sessionPool._defaultCreateSessionFunction); // eslint-disable-line});@@ -36,7 +36,7 @@ describe('SessionPool - testing session pool', async () => {const opts = {maxPoolSize: 3000,maxSessionAgeSecs: 100,- maxSessionReuseCount: 1,+ maxSessionUsageCount: 1,persistStateKeyValueStoreId: 'TEST',persistStateKey: 'SESSION_POOL_STATE2',@@ -58,7 +58,6 @@ describe('SessionPool - testing session pool', async () => {expect(sessionPool.sessions.length).to.be.eql(1);expect(session.id).to.exist; // eslint-disable-lineexpect(session.cookies.length).to.be.eql(0);- expect(session.fingerprintSeed).to.exist; // eslint-disable-lineexpect(session.maxAgeSecs).to.eql(sessionPool.maxSessionAgeSecs);expect(session.maxAgeSecs).to.eql(sessionPool.maxSessionAgeSecs);expect(session.sessionPool).to.eql(sessionPool);@@ -119,7 +118,11 @@ describe('SessionPool - testing session pool', async () => {expect(sessionPoolSaved.sessions.length).to.be.eql(sessionPool.sessions.length);sessionPoolSaved.sessions.forEach((session, index) => {Object.entries(session).forEach(([key, value]) => {- expect(value).to.be.eql(sessionPool.sessions[index][key]);+ if (moment.isMoment(sessionPool.sessions[index][key])) {+ expect(value).to.be.eql(sessionPool.sessions[index][key].toISOString());+ } else {+ expect(sessionPool.sessions[index][key]).to.be.eql(value);+ }});});@@ -198,7 +201,7 @@ describe('SessionPool - testing session pool', async () => {invalidSessionsCount += 1;}}- expect(sessionPool.blockedSessionsCount).to.be.eql(invalidSessionsCount);+ expect(sessionPool.retiredSessionsCount).to.be.eql(invalidSessionsCount);await sessionPool.persistState();diff --git a/website/i18n/en.json b/website/i18n/en.jsonindex 33caa50ce..755dcc666 100644--- a/website/i18n/en.json+++ b/website/i18n/en.json@@ -50,6 +50,12 @@"api/requestqueue": {"title": "RequestQueue"},+ "api/session": {+ "title": "Session"+ },+ "api/sessionpool": {+ "title": "SessionPool"+ },"api/snapshotter": {"title": "Snapshotter"},diff --git a/website/sidebars.json b/website/sidebars.jsonindex 1ce196fb2..e5fa3e8de 100644--- a/website/sidebars.json+++ b/website/sidebars.json@@ -23,6 +23,8 @@"api/request","api/requestlist","api/requestqueue",+ "api/sessionpool",+ "api/session","api/utils","api/log","api/puppeteer",