Skip to content

Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.

improved session pool docs and parameters. Changed session.id to match the proxy session regex.

Petr Patek0e670b6626a3
diff --git a/docs/api/Session.md b/docs/api/Session.md
index b198fcd98..e24b82db2 100644
--- a/docs/api/Session.md
+++ b/docs/api/Session.md
@@ -10,7 +10,7 @@ sessions. You can imagine each session as a specific user, with its own cookies,
internal state can be enriched with custom user data for example some authorization tokens and specific headers in general.
- [Session](session)
- - [`new exports.Session(options)`](#new_Session_new)
+ - [`new exports.Session()`](#new_Session_new)
- [`.isBlocked()`](#Session+isBlocked) ⇒ `boolean`
- [`.isExpired()`](#Session+isExpired) ⇒ `boolean`
- [`.isMaxUsageCountReached()`](#Session+isMaxUsageCountReached) ⇒ `boolean`
@@ -22,38 +22,34 @@ internal state can be enriched with custom user data for example some authorizat
<a name="new_Session_new"></a>
-## `new exports.Session(options)`
+## `new exports.Session()`
Session configuration.
<table>
<thead>
<tr>
-<th>Param</th><th>Type</th>
+<th>Param</th><th>Type</th><th>Default</th>
</tr>
</thead>
<tbody>
<tr>
-<td><code>options</code></td><td></td>
-</tr>
-<tr>
-<td colspan="3"></td></tr><tr>
-<td><code>options.id</code></td><td><code>String</code></td>
+<td><code>[options.id]</code></td><td><code>String</code></td><td></td>
</tr>
<tr>
<td colspan="3"><p>Id of session used for generating fingerprints. It is used as proxy session name.</p>
</td></tr><tr>
-<td><code>options.maxAgeSecs</code></td><td><code>Number</code></td>
+<td><code>[options.maxAgeSecs]</code></td><td><code>Number</code></td><td><code>3000</code></td>
</tr>
<tr>
<td colspan="3"><p>Number of seconds after which the session is considered as expired.</p>
</td></tr><tr>
-<td><code>options.userData</code></td><td><code>Object</code></td>
+<td><code>options.userData</code></td><td><code>Object</code></td><td></td>
</tr>
<tr>
<td colspan="3"><p>Object where custom user data can be stored. For example custom headers.</p>
</td></tr><tr>
-<td><code>options.maxErrorScore</code></td><td><code>number</code></td>
+<td><code>[options.maxErrorScore]</code></td><td><code>number</code></td><td><code>3</code></td>
</tr>
<tr>
<td colspan="3"><p>Maximum number of marking session as blocked usage.
@@ -61,39 +57,39 @@ If the <code>errorScore</code> reaches the <code>maxErrorScore</code> session is
It starts at 0. Calling the <code>markBad</code> function increases the <code>errorScore</code> by 1.
Calling the <code>markGood</code> will decrease the <code>errorScore</code> by <code>errorScoreDecrement</code></p>
</td></tr><tr>
-<td><code>options.errorScoreDecrement</code></td><td><code>number</code></td>
+<td><code>[options.errorScoreDecrement]</code></td><td><code>number</code></td><td><code>0.5</code></td>
</tr>
<tr>
<td colspan="3"><p>It is used for healing the session.
For example: if your session is marked bad two times, but it is successful on the third attempt it&#39;s errorScore is decremented by this number.</p>
</td></tr><tr>
-<td><code>options.createdAt</code></td><td><code>Date</code></td>
+<td><code>options.createdAt</code></td><td><code>Date</code></td><td></td>
</tr>
<tr>
<td colspan="3"><p>Date of creation.</p>
</td></tr><tr>
-<td><code>options.expiredAt</code></td><td><code>Date</code></td>
+<td><code>options.expiredAt</code></td><td><code>Date</code></td><td></td>
</tr>
<tr>
<td colspan="3"><p>Date of expiration.</p>
</td></tr><tr>
-<td><code>options.usageCount</code></td><td><code>Number</code></td>
+<td><code>[options.usageCount]</code></td><td><code>Number</code></td><td><code>0</code></td>
</tr>
<tr>
<td colspan="3"><p>Indicates how many times the session has been used.</p>
</td></tr><tr>
-<td><code>options.errorCount</code></td><td><code>Number</code></td>
+<td><code>[options.errorCount]</code></td><td><code>Number</code></td><td><code>0</code></td>
</tr>
<tr>
<td colspan="3"><p>Indicates how many times the session is marked bad.</p>
</td></tr><tr>
-<td><code>options.maxSessionUsageCount</code></td><td><code>Number</code></td>
+<td><code>[options.maxUsageCount]</code></td><td><code>Number</code></td><td><code>50</code></td>
</tr>
<tr>
<td colspan="3"><p>Session should be used only a limited amount of times.
This number indicates how many times the session is going to be used, before it is thrown away.</p>
</td></tr><tr>
-<td><code>options.sessionPool</code></td><td><code>EventEmitter</code></td>
+<td><code>options.sessionPool</code></td><td><code>EventEmitter</code></td><td></td>
</tr>
<tr>
<td colspan="3"><p>SessionPool instance. Session will emit the <code>sessionRetired</code> event on this instance.</p>
@@ -116,7 +112,7 @@ Indicates whether the session is expired. Session expiration is determined by th
## `session.isMaxUsageCountReached()` ⇒ `boolean`
-Indicates whether the session is used maximum number of times. Session maximum usage count can be changed by `maxSessionUsageCount` parameter.
+Indicates whether the session is used maximum number of times. Session maximum usage count can be changed by `maxUsageCount` parameter.
<a name="Session+isUsable"></a>
diff --git a/docs/api/SessionPool.md b/docs/api/SessionPool.md
index 8a60f7c54..bd8281db9 100644
--- a/docs/api/SessionPool.md
+++ b/docs/api/SessionPool.md
@@ -16,9 +16,10 @@ Session pool is by default persisted in default [`KeyValueStore`](keyvaluestore)
```javascript
const sessionPool = new SessionPool({
maxPoolSize: 25,
- maxSessionAgeSecs: 10,
- maxSessionAgeSecs: 10,
- maxSessionUsageCount: 150, // for example when you know that the site blocks after 150 requests.
+ sessionOptions:{
+ maxAgeSecs: 10,
+ maxUsageCount: 150, // for example when you know that the site blocks after 150 requests.
+ }
persistStateKeyValueStoreId: 'my-key-value-store-for-sessions',
persistStateKey: 'my-session-pool',
});
@@ -37,17 +38,18 @@ const session3 = await sessionPool.getSession();
// Now you can mark the session either failed of successful
// Marks session as bad after unsuccessful usage -> it increases error count (soft retire)
-session1.markBad();
+session1.markBad()
// Marks as successful.
-session2.markGood();
+session2.markGood()
// Retires session -> session is removed from the pool
-session3.retire();
+session3.retire()
+
```
- [SessionPool](sessionpool)
- - [`new exports.SessionPool(options)`](#new_SessionPool_new)
+ - [`new exports.SessionPool([options])`](#new_SessionPool_new)
- [`.usableSessionsCount`](#SessionPool+usableSessionsCount) ⇒ `number`
- [`.retiredSessionsCount`](#SessionPool+retiredSessionsCount) ⇒ `number`
- [`.initialize()`](#SessionPool+initialize) ⇒ `Promise<void>`
@@ -58,50 +60,44 @@ session3.retire();
<a name="new_SessionPool_new"></a>
-## `new exports.SessionPool(options)`
+## `new exports.SessionPool([options])`
Session pool configuration.
<table>
<thead>
<tr>
-<th>Param</th><th>Type</th>
+<th>Param</th><th>Type</th><th>Default</th>
</tr>
</thead>
<tbody>
<tr>
-<td><code>options</code></td><td></td>
+<td><code>[options]</code></td><td></td><td></td>
</tr>
<tr>
<td colspan="3"></td></tr><tr>
-<td><code>options.maxPoolSize</code></td><td><code>Number</code></td>
+<td><code>[options.maxPoolSize]</code></td><td><code>Number</code></td><td><code>1000</code></td>
</tr>
<tr>
<td colspan="3"><p>Maximum size of the pool.
Indicates how many sessions are rotated.</p>
</td></tr><tr>
-<td><code>options.maxSessionAgeSecs</code></td><td><code>Number</code></td>
-</tr>
-<tr>
-<td colspan="3"><p>Number of seconds after which the session is considered as expired.</p>
-</td></tr><tr>
-<td><code>options.maxSessionUsageCount</code></td><td><code>Number</code></td>
+<td><code>[options.sessionOptions]</code></td><td><code>Object</code></td><td></td>
</tr>
<tr>
-<td colspan="3"><p>Maximum number of uses per session.
-It useful, when you know the site rate-limits, so you can retire the session before it gets blocked and let it cool down.</p>
+<td colspan="3"><p>The <a href="session#new_SessionPool_new"><code>new Session</code></a> options</p>
</td></tr><tr>
-<td><code>options.persistStateKeyValueStoreId</code></td><td><code>String</code></td>
+<td><code>[options.persistStateKeyValueStoreId]</code></td><td><code>String</code></td><td></td>
</tr>
<tr>
<td colspan="3"><p>Name or Id of <code>KeyValueStore</code> where is the <code>SessionPool</code> state stored.</p>
</td></tr><tr>
-<td><code>options.persistStateKey</code></td><td><code>String</code></td>
+<td><code>[options.persistStateKey]</code></td><td><code>String</code></td><td><code>&quot;SESSION_POOL_STATE&quot;</code></td>
</tr>
<tr>
<td colspan="3"><p>Session pool persists it&#39;s state under this key in Key value store.</p>
</td></tr><tr>
-<td><code>options.createSessionFunction</code></td><td><code>function</code></td>
+<td><code>[options.createSessionFunction]</code></td><td><code>function</code></td><td></td>
</tr>
<tr>
<td colspan="3"><p>Custom function that should return <code>Session</code> instance.
diff --git a/src/index.js b/src/index.js
index 66d8d33cd..122ebbfee 100644
--- a/src/index.js
+++ b/src/index.js
@@ -23,6 +23,7 @@ import PseudoUrl from './pseudo_url';
import LiveViewServer from './live_view/live_view_server';
import { requestAsBrowser } from './utils_request';
import { openSessionPool } from './session_pool/session_pool';
+import { Session } from './session_pool/session';
// Increase the global limit for event emitter memory leak warnings.
EventEmitter.defaultMaxListeners = 50;
@@ -95,6 +96,7 @@ module.exports = {
openSessionPool,
LiveViewServer,
+ Session,
utils: Object.assign(publicUtils, {
puppeteer: puppeteerUtils,
diff --git a/src/session_pool/session.js b/src/session_pool/session.js
index 8a91d57fc..af81792d0 100644
--- a/src/session_pool/session.js
+++ b/src/session_pool/session.js
@@ -13,27 +13,26 @@ import EVENTS from './events';
export class Session {
/**
* Session configuration.
- * @param options
- * @param options.id {String} - Id of session used for generating fingerprints. It is used as proxy session name.
- * @param options.maxAgeSecs {Number} - Number of seconds after which the session is considered as expired.
+ * @param [options.id] {String} - Id of session used for generating fingerprints. It is used as proxy session name.
+ * @param [options.maxAgeSecs=3000] {Number} - Number of seconds after which the session is considered as expired.
* @param options.userData {Object} - Object where custom user data can be stored. For example custom headers.
- * @param options.maxErrorScore {number} - Maximum number of marking session as blocked usage.
+ * @param [options.maxErrorScore=3] {number} - Maximum number of marking session as blocked usage.
* If the `errorScore` reaches the `maxErrorScore` session is marked as block and it is thrown away.
* It starts at 0. Calling the `markBad` function increases the `errorScore` by 1.
* Calling the `markGood` will decrease the `errorScore` by `errorScoreDecrement`
- * @param options.errorScoreDecrement {number} - It is used for healing the session.
+ * @param [options.errorScoreDecrement=0.5] {number} - It is used for healing the session.
* For example: if your session is marked bad two times, but it is successful on the third attempt it's errorScore is decremented by this number.
* @param options.createdAt {Date} - Date of creation.
* @param options.expiredAt {Date} - Date of expiration.
- * @param options.usageCount {Number} - Indicates how many times the session has been used.
- * @param options.errorCount {Number} - Indicates how many times the session is marked bad.
- * @param options.maxSessionUsageCount {Number} - Session should be used only a limited amount of times.
+ * @param [options.usageCount=0] {Number} - Indicates how many times the session has been used.
+ * @param [options.errorCount=0] {Number} - Indicates how many times the session is marked bad.
+ * @param [options.maxUsageCount=50] {Number} - Session should be used only a limited amount of times.
* This number indicates how many times the session is going to be used, before it is thrown away.
* @param options.sessionPool {EventEmitter} - SessionPool instance. Session will emit the `sessionRetired` event on this instance.
*/
constructor(options = {}) {
const {
- id = `session-${cryptoRandomObjectId(10)}`,
+ id = `session_${cryptoRandomObjectId(10)}`,
cookies = [],
maxAgeSecs = 3000,
userData = {},
@@ -42,7 +41,7 @@ export class Session {
createdAt = new Date(),
usageCount = 0,
errorScore = 0,
- maxSessionUsageCount = 50,
+ maxUsageCount = 50,
sessionPool,
} = options;
@@ -57,7 +56,7 @@ export class Session {
checkParamOrThrow(createdAt, 'options.createdAt', 'Date');
checkParamOrThrow(usageCount, 'options.usageCount', 'Number');
checkParamOrThrow(errorScore, 'options.errorScore', 'Number');
- checkParamOrThrow(maxSessionUsageCount, 'options.maxSessionUsageCount', 'Number');
+ checkParamOrThrow(maxUsageCount, 'options.maxUsageCount', 'Number');
checkParamOrThrow(sessionPool, 'options.sessionPool', 'Object');
// sessionPool must be instance of SessionPool.
@@ -78,7 +77,7 @@ export class Session {
this.createdAt = createdAt;
this.usageCount = usageCount; // indicates how many times the session has been used
this.errorScore = errorScore; // indicates number of markBaded request with the session
- this.maxSessionUsageCount = maxSessionUsageCount;
+ this.maxUsageCount = maxUsageCount;
this.sessionPool = sessionPool;
}
@@ -103,11 +102,11 @@ export class Session {
/**
* Indicates whether the session is used maximum number of times.
- * Session maximum usage count can be changed by `maxSessionUsageCount` parameter.
+ * Session maximum usage count can be changed by `maxUsageCount` parameter.
* @return {boolean}
*/
isMaxUsageCountReached() {
- return this.usageCount >= this.maxSessionUsageCount;
+ return this.usageCount >= this.maxUsageCount;
}
/**
diff --git a/src/session_pool/session_pool.js b/src/session_pool/session_pool.js
index e57992e07..b8505835b 100644
--- a/src/session_pool/session_pool.js
+++ b/src/session_pool/session_pool.js
@@ -22,9 +22,10 @@ import { ACTOR_EVENT_NAMES_EX } from '../constants';
* ```javascript
* const sessionPool = new SessionPool({
* maxPoolSize: 25,
- * maxSessionAgeSecs: 10,
- * maxSessionAgeSecs: 10,
- * maxSessionUsageCount: 150, // for example when you know that the site blocks after 150 requests.
+ * sessionOptions:{
+ * maxAgeSecs: 10,
+ * maxUsageCount: 150, // for example when you know that the site blocks after 150 requests.
+ * }
* persistStateKeyValueStoreId: 'my-key-value-store-for-sessions',
* persistStateKey: 'my-session-pool',
* });
@@ -57,27 +58,24 @@ import { ACTOR_EVENT_NAMES_EX } from '../constants';
export class SessionPool extends EventEmitter {
/**
* Session pool configuration.
- * @param options
- * @param options.maxPoolSize {Number} - Maximum size of the pool.
+ * @param [options]
+ * @param [options.maxPoolSize=1000] {Number} - Maximum size of the pool.
* Indicates how many sessions are rotated.
- * @param options.maxSessionAgeSecs {Number} - Number of seconds after which the session is considered as expired.
- * @param options.maxSessionUsageCount {Number} - Maximum number of uses per session.
- * It useful, when you know the site rate-limits, so you can retire the session before it gets blocked and let it cool down.
- * @param options.persistStateKeyValueStoreId {String} - Name or Id of `KeyValueStore` where is the `SessionPool` state stored.
- * @param options.persistStateKey {String} - Session pool persists it's state under this key in Key value store.
- * @param options.createSessionFunction {function} - Custom function that should return `Session` instance.
+ * @param [options.sessionOptions] {Object} The [`new Session`](session#new_SessionPool_new) options
+ * @param [options.persistStateKeyValueStoreId] {String} - Name or Id of `KeyValueStore` where is the `SessionPool` state stored.
+ * @param [options.persistStateKey="SESSION_POOL_STATE"] {String} - Session pool persists it's state under this key in Key value store.
+ * @param [options.createSessionFunction] {function} - Custom function that should return `Session` instance.
* Function receives `SessionPool` instance as a parameter
*/
constructor(options = {}) {
const {
maxPoolSize = 1000,
- maxSessionAgeSecs = 3000,
- maxSessionUsageCount = 50,
persistStateKeyValueStoreId = null,
persistStateKey = 'SESSION_POOL_STATE',
createSessionFunction = null,
+ sessionOptions = {},
} = options;
@@ -85,8 +83,7 @@ export class SessionPool extends EventEmitter {
// Validation
checkParamOrThrow(maxPoolSize, 'options.maxPoolSize', 'Number');
- checkParamOrThrow(maxSessionAgeSecs, 'options.maxSessionAgeSecs', 'Number');
- checkParamOrThrow(maxSessionUsageCount, 'options.maxSessionUsageCount', 'Number');
+ checkParamOrThrow(sessionOptions, 'options.maxAgeSecs', 'Object');
checkParamOrThrow(persistStateKeyValueStoreId, 'options.persistStateKeyValueStoreId', 'Maybe String');
checkParamOrThrow(persistStateKey, 'options.persistStateKey', 'String');
checkParamOrThrow(createSessionFunction, 'options.createSessionFunction', 'Maybe Function');
@@ -96,8 +93,7 @@ export class SessionPool extends EventEmitter {
this.createSessionFunction = createSessionFunction || this._defaultCreateSessionFunction;
// Session configuration
- this.maxSessionAgeSecs = maxSessionAgeSecs;
- this.maxSessionUsageCount = maxSessionUsageCount;
+ this.sessionOptions = sessionOptions;
// Session keyValueStore
this.persistStateKeyValueStoreId = persistStateKeyValueStoreId;
@@ -239,8 +235,7 @@ export class SessionPool extends EventEmitter {
*/
_defaultCreateSessionFunction(sessionPool) {
return new Session({
- maxSessionAgeSecs: this.maxSessionAgeSecs,
- maxSessionUsageCount: this.maxSessionUsageCount,
+ ...this.sessionOptions,
sessionPool,
});
}
diff --git a/test/session_pool/session.test.js b/test/session_pool/session.test.js
index 8d625abe8..47f551a02 100644
--- a/test/session_pool/session.test.js
+++ b/test/session_pool/session.test.js
@@ -3,6 +3,7 @@ import { SessionPool } from '../../build/session_pool/session_pool';
import EVENTS from '../../build/session_pool/events';
import Apify from '../../build';
+import { getApifyProxyUrl } from '../../build/actor';
describe('Session - testing session behaviour ', () => {
@@ -53,7 +54,7 @@ describe('Session - testing session behaviour ', () => {
});
test('should max out session usage', () => {
- session.maxSessionUsageCount = 1;
+ session.maxUsageCount = 1;
session.markGood();
expect(session.isMaxUsageCountReached()).toBe(true);
expect(session.isUsable()).toBe(false);
@@ -104,4 +105,16 @@ describe('Session - testing session behaviour ', () => {
}
});
});
+
+ test('should be valid proxy session', () => {
+ session = new Session({ sessionPool });
+ let error;
+ try {
+ getApifyProxyUrl({ session: session.id, password: '12312' });
+ } catch (e) {
+ error = e;
+ }
+
+ expect(error).toBeUndefined();
+ });
});
diff --git a/test/session_pool/session_pool.test.js b/test/session_pool/session_pool.test.js
index 680860fa4..5f2a205b3 100644
--- a/test/session_pool/session_pool.test.js
+++ b/test/session_pool/session_pool.test.js
@@ -24,19 +24,21 @@ describe('SessionPool - testing session pool', () => {
// eslint-disable-line
test('should initialize with default values for first time', async () => {
- expect(sessionPool.sessions.length).toBeDefined(); // eslint-disable-line
- expect(sessionPool.maxPoolSize).toBeDefined(); // eslint-disable-line
- expect(sessionPool.maxSessionAgeSecs).toBeDefined(); // eslint-disable-line
- expect(sessionPool.maxSessionUsageCount).toBeDefined(); // eslint-disable-line
- expect(sessionPool.persistStateKey).toBeDefined(); // eslint-disable-line
- expect(sessionPool.createSessionFunction).toEqual(sessionPool._defaultCreateSessionFunction); // eslint-disable-line
+ expect(sessionPool.sessions.length).toBeDefined();
+ expect(sessionPool.maxPoolSize).toBeDefined();
+ expect(sessionPool.sessionOptions).toBeDefined();
+ expect(sessionPool.persistStateKey).toBeDefined();
+ expect(sessionPool.createSessionFunction).toEqual(sessionPool._defaultCreateSessionFunction);
});
test('should override default values', async () => {
const opts = {
maxPoolSize: 3000,
- maxSessionAgeSecs: 100,
- maxSessionUsageCount: 1,
+ sessionOptions: {
+ maxAgeSecs: 100,
+ maxUsageCount: 1,
+ },
+
persistStateKeyValueStoreId: 'TEST',
persistStateKey: 'SESSION_POOL_STATE2',
@@ -56,8 +58,11 @@ describe('SessionPool - testing session pool', () => {
test('should work using openSessionPool', async () => {
const opts = {
maxPoolSize: 3000,
- maxSessionAgeSecs: 100,
- maxSessionUsageCount: 1,
+
+ sessionOptions: {
+ maxAgeSecs: 100,
+ maxUsageCount: 1,
+ },
persistStateKeyValueStoreId: 'TEST',
persistStateKey: 'SESSION_POOL_STATE2',
@@ -76,12 +81,13 @@ describe('SessionPool - testing session pool', () => {
describe('should retrieve session', () => {
test('should retrieve session with correct shape', async () => {
+ sessionPool = await Apify.openSessionPool({ sessionOptions: { maxAgeSecs: 100, maxUsageCount: 10 } });
const session = await sessionPool.getSession();
expect(sessionPool.sessions.length).toBe(1);
expect(session.id).toBeDefined(); // eslint-disable-line
expect(session.cookies.length).toBe(0);
- expect(session.maxAgeSecs).toEqual(sessionPool.maxSessionAgeSecs);
- expect(session.maxAgeSecs).toEqual(sessionPool.maxSessionAgeSecs);
+ expect(session.maxAgeSecs).toEqual(sessionPool.sessionOptions.maxAgeSecs);
+ expect(session.maxAgeSecs).toEqual(sessionPool.sessionOptions.maxAgeSecs);
expect(session.sessionPool).toEqual(sessionPool);
});
@@ -157,8 +163,6 @@ describe('SessionPool - testing session pool', () => {
await loadedSessionPool.initialize();
expect(sessionPool.sessions).toHaveLength(loadedSessionPool.sessions.length);
expect(sessionPool.maxPoolSize).toEqual(loadedSessionPool.maxPoolSize);
- expect(sessionPool.maxSessionAgeSecs).toEqual(loadedSessionPool.maxSessionAgeSecs);
- expect(sessionPool.maxSessionUsageCount).toEqual(loadedSessionPool.maxSessionUsageCount);
expect(sessionPool.persistStateKey).toEqual(loadedSessionPool.persistStateKey);
sessionPool.teardown();
});