Skip to content

Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.

renamed function and changed docs.

Petr Patekc29b7939429b
diff --git a/docs/api/Apify.md b/docs/api/Apify.md
index e73ce5bbf..589147df2 100644
--- a/docs/api/Apify.md
+++ b/docs/api/Apify.md
@@ -83,7 +83,7 @@ effect.
<td colspan="3"><p>Idempotency key enables you to ensure that a webhook will not be added multiple times in case of
an actor restart or other situation that would cause the <code>addWebhook()</code> function to be called again.
We suggest using the actor run ID as the idempotency key. You can get the run ID by calling
- <a href="apify#module_Apify.getEnv">`Apify.getEnv()</a> function.</p>
+ [`Apify.getEnv()](apify#module_Apify.getEnv) function.</p>
</td></tr></tbody>
</table>
<a name="module_Apify.call"></a>
diff --git a/docs/api/Session.md b/docs/api/Session.md
index b3ffc4e41..81e8b1dcf 100644
--- a/docs/api/Session.md
+++ b/docs/api/Session.md
@@ -15,10 +15,10 @@ internal state can be enriched with custom user data for example some authorizat
- [`.isExpired()`](#Session+isExpired) ⇒ `boolean`
- [`.isMaxUsageCountReached()`](#Session+isMaxUsageCountReached) ⇒ `boolean`
- [`.isUsable()`](#Session+isUsable) ⇒ `boolean`
- - [`.reclaim()`](#Session+reclaim)
+ - [`.markGood()`](#Session+markGood)
- [`.getState()`](#Session+getState) ⇒ `Object`
- [`.retire()`](#Session+retire)
- - [`.fail()`](#Session+fail)
+ - [`.markBad()`](#Session+markBad)
<a name="new_Session_new"></a>
@@ -56,14 +56,14 @@ Session configuration.
<td><code>options.maxErrorScore</code></td><td><code>number</code></td>
</tr>
<tr>
-<td colspan="3"><p>Maximum number of failed session usage.
+<td colspan="3"><p>Maximum number of marking session as blocked usage.
If the <code>errorScore</code> reaches the <code>maxErrorScore</code> session is marked as block and it is thrown away.</p>
</td></tr><tr>
<td><code>options.errorScoreDecrement</code></td><td><code>number</code></td>
</tr>
<tr>
<td colspan="3"><p>It is used for healing the session.
-For example: if your session fails two times, but it is successful on the third attempt it&#39;s errorScore is decremented by this number.</p>
+For example: if your session is marked bad two times, but it is successful on the third attempt it&#39;s errorScore is decremented by this number.</p>
</td></tr><tr>
<td><code>options.createdAt</code></td><td><code>Date</code></td>
</tr>
@@ -83,7 +83,7 @@ For example: if your session fails two times, but it is successful on the third
<td><code>options.errorCount</code></td><td><code>Number</code></td>
</tr>
<tr>
-<td colspan="3"><p>Indicates how many times the session failed.</p>
+<td colspan="3"><p>Indicates how many times the session is marked bad.</p>
</td></tr><tr>
<td><code>options.maxSessionUsageCount</code></td><td><code>Number</code></td>
</tr>
@@ -123,9 +123,9 @@ Indicates whether the session is used maximum number of times. Session maximum u
Indicates whether the session can be used for next requests. Session is usable when it is not expired, not blocked and the maximum usage count has not
be reached.
-<a name="Session+reclaim"></a>
+<a name="Session+markGood"></a>
-## `session.reclaim()`
+## `session.markGood()`
This method should be called after a successful session usage. It increases `usageCount` and potentially lowers the `errorScore` by the
`errorScoreDecrement`.
@@ -143,10 +143,10 @@ Gets session state for persistence in KeyValueStore.
Marks session as blocked and emits event on the `SessionPool` This method should be used if the session usage was unsuccessful and you are sure that
it is because of the session configuration and not any external matters. For example when server returns 403 status code. If the session does not work
-due to some external factors as server error such as 5XX you probably want to use `fail` method.
+due to some external factors as server error such as 5XX you probably want to use `markBad` method.
-<a name="Session+fail"></a>
+<a name="Session+markBad"></a>
-## `session.fail()`
+## `session.markBad()`
Increases usage and error count. Should be used when the session has been used unsuccessfully. For example because of timeouts.
diff --git a/docs/api/SessionPool.md b/docs/api/SessionPool.md
index fcf5ede33..d4da806af 100644
--- a/docs/api/SessionPool.md
+++ b/docs/api/SessionPool.md
@@ -30,17 +30,17 @@ const sessionPool = new SessionPool({
await sessionPool.initialize();
// Get random session from the pool
-const session1 = await sessionPool.retrieveSession();
-const session2 = await sessionPool.retrieveSession();
-const session3 = await sessionPool.retrieveSession();
+const session1 = await sessionPool.getSession();
+const session2 = await sessionPool.getSession();
+const session3 = await sessionPool.getSession();
// Now you can mark the session either failed of successful
-// Fails session -> it increases error count (soft retire)
-session1.fail();
+// Marks session as bad after unsuccessful usage -> it increases error count (soft retire)
+session1.markBad();
// Marks as successful.
-session2.reclaim();
+session2.markGood();
// Retires session -> session is removed from the pool
session3.retire();
@@ -51,7 +51,7 @@ session3.retire();
- [`.usableSessionsCount`](#SessionPool+usableSessionsCount) ⇒ `number`
- [`.retiredSessionsCount`](#SessionPool+retiredSessionsCount) ⇒ `number`
- [`.initialize()`](#SessionPool+initialize) ⇒ `Promise<void>`
- - [`.retrieveSession()`](#SessionPool+retrieveSession) ⇒ [`Promise<Session>`](session)
+ - [`.getSession()`](#SessionPool+getSession) ⇒ [`Promise<Session>`](session)
- [`.getState()`](#SessionPool+getState) ⇒ `Object`
- [`.persistState()`](#SessionPool+persistState) ⇒ `Promise`
@@ -126,9 +126,9 @@ Gets count of retired sessions in the pool.
Starts periodic state persistence and potentially loads SessionPool state from [`KeyValueStore`](keyvaluestore). This function must be called before
you can start using the instance in a meaningful way.
-<a name="SessionPool+retrieveSession"></a>
+<a name="SessionPool+getSession"></a>
-## `sessionPool.retrieveSession()` ⇒ [`Promise<Session>`](session)
+## `sessionPool.getSession()` ⇒ [`Promise<Session>`](session)
Gets session. If there is space for new session, it creates and return new session. If the session pool is full, it picks a session from the pool, If
the picked session is usable it is returned, otherwise it creates and returns a new one.
diff --git a/src/session_pool/session.js b/src/session_pool/session.js
index c7f28e89c..250c7fa9c 100644
--- a/src/session_pool/session.js
+++ b/src/session_pool/session.js
@@ -17,14 +17,14 @@ export class Session {
* @param options.id {String} - Id of session used for generating fingerprints. It is used as proxy session name.
* @param options.maxAgeSecs {Number} - Number of seconds after which the session is considered as expired.
* @param options.userData {Object} - Object where custom user data can be stored. For example custom headers.
- * @param options.maxErrorScore {number} - Maximum number of failed session usage.
+ * @param options.maxErrorScore {number} - Maximum number of marking session as blocked usage.
* If the `errorScore` reaches the `maxErrorScore` session is marked as block and it is thrown away.
* @param options.errorScoreDecrement {number} - It is used for healing the session.
- * For example: if your session fails two times, but it is successful on the third attempt it's errorScore is decremented by this number.
+ * For example: if your session is marked bad two times, but it is successful on the third attempt it's errorScore is decremented by this number.
* @param options.createdAt {Date} - Date of creation.
* @param options.expiredAt {Date} - Date of expiration.
* @param options.usageCount {Number} - Indicates how many times the session has been used.
- * @param options.errorCount {Number} - Indicates how many times the session failed.
+ * @param options.errorCount {Number} - Indicates how many times the session is marked bad.
* @param options.maxSessionUsageCount {Number} - Session should be used only a limited amount of times.
* This number indicates how many times the session is going to be used, before it is thrown away.
* @param options.sessionPool {EventEmitter} - SessionPool instance. Session will emit the `sessionRetired` event on this instance.
@@ -75,7 +75,7 @@ export class Session {
this.expiresAt = expiresAt;
this.createdAt = createdAt;
this.usageCount = usageCount; // indicates how many times the session has been used
- this.errorScore = errorScore; // indicates number of failed request with the session
+ this.errorScore = errorScore; // indicates number of markBaded request with the session
this.maxSessionUsageCount = maxSessionUsageCount;
this.sessionPool = sessionPool;
}
@@ -121,7 +121,7 @@ export class Session {
* This method should be called after a successful session usage.
* It increases `usageCount` and potentially lowers the `errorScore` by the `errorScoreDecrement`.
*/
- reclaim() {
+ markGood() {
this.usageCount += 1;
// We should probably lower the errorScore.
@@ -153,7 +153,7 @@ export class Session {
* This method should be used if the session usage was unsuccessful
* and you are sure that it is because of the session configuration and not any external matters.
* For example when server returns 403 status code.
- * If the session does not work due to some external factors as server error such as 5XX you probably want to use `fail` method.
+ * If the session does not work due to some external factors as server error such as 5XX you probably want to use `markBad` method.
*/
retire() {
// mark it as an invalid by increasing the error score count.
@@ -168,7 +168,7 @@ export class Session {
* Increases usage and error count.
* Should be used when the session has been used unsuccessfully. For example because of timeouts.
*/
- fail() {
+ markBad() {
this.errorScore += 1;
this.usageCount += 1;
}
diff --git a/src/session_pool/session_pool.js b/src/session_pool/session_pool.js
index 99bca1832..e121abbb5 100644
--- a/src/session_pool/session_pool.js
+++ b/src/session_pool/session_pool.js
@@ -35,17 +35,17 @@ import { ACTOR_EVENT_NAMES_EX } from '../constants';
* await sessionPool.initialize();
*
* // Get random session from the pool
- * const session1 = await sessionPool.retrieveSession();
- * const session2 = await sessionPool.retrieveSession();
- * const session3 = await sessionPool.retrieveSession();
+ * const session1 = await sessionPool.getSession();
+ * const session2 = await sessionPool.getSession();
+ * const session3 = await sessionPool.getSession();
*
* // Now you can mark the session either failed of successful
*
- * // Fails session -> it increases error count (soft retire)
- * session1.fail()
+ * // Marks session as bad after unsuccessful usage -> it increases error count (soft retire)
+ * session1.markBad()
*
* // Marks as successful.
- * session2.reclaim()
+ * session2.markGood()
*
* // Retires session -> session is removed from the pool
* session3.retire()
@@ -147,7 +147,7 @@ export class SessionPool extends EventEmitter {
*
* @return {Promise<Session>}
*/
- async retrieveSession() {
+ async getSession() {
if (this._hasSpaceForSession()) {
return this._createSession();
}
diff --git a/test/session_pool/session.js b/test/session_pool/session.js
index ba2d38c2a..9937ea5dd 100644
--- a/test/session_pool/session.js
+++ b/test/session_pool/session.js
@@ -16,14 +16,14 @@ describe('Session - testing session behaviour ', async () => {
session = new Session({ sessionPool });
});
- it('should reclaim session and lower the errorScore', () => {
+ it('should markGood session and lower the errorScore', () => {
expect(session.usageCount).to.be.eql(0);
expect(session.errorScore).to.be.eql(0);
- session.reclaim();
+ session.markGood();
expect(session.usageCount).to.be.eql(1);
expect(session.errorScore).to.be.eql(0);
session.errorScore = 1;
- session.reclaim();
+ session.markGood();
expect(session.errorScore).to.be.eql(0.5);
});
@@ -38,8 +38,8 @@ describe('Session - testing session behaviour ', async () => {
expect(err.message.includes('Session: sessionPool must be instance of SessionPool')).to.be.true; // eslint-disable-line
});
- it('should mark session failed', () => {
- session.fail();
+ it('should mark session markBaded', () => {
+ session.markBad();
expect(session.errorScore).to.be.eql(1);
expect(session.usageCount).to.be.eql(1);
});
@@ -53,7 +53,7 @@ describe('Session - testing session behaviour ', async () => {
it('should max out session usage', () => {
session.maxSessionUsageCount = 1;
- session.reclaim();
+ session.markGood();
expect(session.isMaxUsageCountReached()).to.be.eql(true);
expect(session.isUsable()).to.be.eql(false);
});
@@ -64,8 +64,8 @@ describe('Session - testing session behaviour ', async () => {
expect(session.isUsable()).to.be.eql(false);
});
- it('should reclaim session', () => {
- session.reclaim();
+ it('should markGood session', () => {
+ session.markGood();
expect(session.usageCount).to.be.eql(1);
expect(session.isUsable()).to.be.eql(true);
});
diff --git a/test/session_pool/session_pool.js b/test/session_pool/session_pool.js
index e13d75fa0..c05ad134e 100644
--- a/test/session_pool/session_pool.js
+++ b/test/session_pool/session_pool.js
@@ -69,7 +69,7 @@ describe('SessionPool - testing session pool', async () => {
describe('should retrieve session', () => {
it('should retrieve session with correct shape', async () => {
- const session = await sessionPool.retrieveSession();
+ const session = await sessionPool.getSession();
expect(sessionPool.sessions.length).to.be.eql(1);
expect(session.id).to.exist; // eslint-disable-line
expect(session.cookies.length).to.be.eql(0);
@@ -80,8 +80,8 @@ describe('SessionPool - testing session pool', async () => {
it('should pick session when pool is full', async () => {
sessionPool.maxPoolSize = 2;
- await sessionPool.retrieveSession();
- await sessionPool.retrieveSession();
+ await sessionPool.getSession();
+ await sessionPool.getSession();
let isCalled = false;
const oldPick = sessionPool._pickSession; //eslint-disable-line
@@ -90,14 +90,14 @@ describe('SessionPool - testing session pool', async () => {
return oldPick.bind(sessionPool)();
};
- await sessionPool.retrieveSession();
+ await sessionPool.getSession();
expect(isCalled).to.be.true; //eslint-disable-line
});
it('should delete picked session when it is usable a create a new one', async () => {
sessionPool.maxPoolSize = 1;
- await sessionPool.retrieveSession();
+ await sessionPool.getSession();
const session = sessionPool.sessions[0];
session.errorScore += session.maxErrorScore;
@@ -109,7 +109,7 @@ describe('SessionPool - testing session pool', async () => {
return oldRemove.bind(sessionPool)(session);
};
- await sessionPool.retrieveSession();
+ await sessionPool.getSession();
expect(isCalled).to.be.true; //eslint-disable-line
expect(sessionPool.sessions[0].id === session.id).to.be.false; //eslint-disable-line
@@ -119,7 +119,7 @@ describe('SessionPool - testing session pool', async () => {
it('should persist state and recreate it from storage', async () => {
- await sessionPool.retrieveSession();
+ await sessionPool.getSession();
await sessionPool.persistState();
const kvStore = await Apify.openKeyValueStore();
@@ -151,7 +151,7 @@ describe('SessionPool - testing session pool', async () => {
it('should create only maxPoolSize number of sessions', async () => {
const max = sessionPool.maxPoolSize;
for (let i = 0; i < max + 100; i++) {
- await sessionPool.retrieveSession();
+ await sessionPool.getSession();
}
expect(sessionPool.sessions.length).to.be.eql(sessionPool.maxPoolSize);
});
@@ -175,7 +175,7 @@ describe('SessionPool - testing session pool', async () => {
});
it('on persist event', async () => {
- await sessionPool.retrieveSession();
+ await sessionPool.getSession();
expect(sessionPool.sessions.length).to.be.eql(1);
@@ -199,9 +199,9 @@ describe('SessionPool - testing session pool', async () => {
it('should remove session', async () => {
for (let i = 0; i < 10; i++) {
- await sessionPool.retrieveSession();
+ await sessionPool.getSession();
}
- const picked = sessionPool.retrieveSession();
+ const picked = sessionPool.getSession();
sessionPool._removeSession(picked); // eslint-disable-line
expect(sessionPool.sessions.find(s => s.id === picked.id)).to.be.eql(undefined);
});
@@ -209,7 +209,7 @@ describe('SessionPool - testing session pool', async () => {
it('should recreate only usable sessions', async () => {
let invalidSessionsCount = 0;
for (let i = 0; i < 10; i++) {
- const session = await sessionPool.retrieveSession();
+ const session = await sessionPool.getSession();
if (i % 2 === 0) {
session.errorScore += session.maxErrorScore;
@@ -235,7 +235,7 @@ describe('SessionPool - testing session pool', async () => {
return new Session({ sessionPool: sessionPool2 });
};
const newSessionPool = await Apify.openSessionPool({ createSessionFunction });
- const session = await newSessionPool.retrieveSession();
+ const session = await newSessionPool.getSession();
expect(isCalled).to.be.true; // eslint-disable-line
expect(session.constructor.name).to.be.eql("Session") // eslint-disable-line
});