Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.
TypeScript apifyautomationcrawlercrawlingheadlessheadless-chromejavascriptnodejsnpmplaywrightpuppeteerscraperscrapingtypescriptweb-crawlerweb-crawlingweb-scraping
Improved session docs and session_management.md guides.
Petr Patekcb390b5771d2
diff --git a/docs/api/Session.md b/docs/api/Session.mdindex 9c68fa556..da6117550 100644--- a/docs/api/Session.md+++ b/docs/api/Session.md@@ -19,7 +19,7 @@ internal state can be enriched with custom user data for example some authorizat- [`.getState()`](#Session+getState) ⇒ `Object`- [`.retire()`](#Session+retire)- [`.markBad()`](#Session+markBad)- - [`.checkStatus(statusCode)`](#Session+checkStatus) ⇒ `boolean`+ - [`.retireOnBlockedStatusCodes(statusCode, blockedStatusCodes)`](#Session+retireOnBlockedStatusCodes) ⇒ `boolean`- [`.putResponse(response)`](#Session+putResponse)- [`.putPuppeteerCookies(puppeteerCookies, url)`](#Session+putPuppeteerCookies)- [`.setCookies(cookies, url)`](#Session+setCookies)@@ -156,9 +156,9 @@ due to some external factors as server error such as 5XX you probably want to usIncreases usage and error count. Should be used when the session has been used unsuccessfully. For example because of timeouts.-<a name="Session+checkStatus"></a>+<a name="Session+retireOnBlockedStatusCodes"></a>-## `session.checkStatus(statusCode)` ⇒ `boolean`+## `session.retireOnBlockedStatusCodes(statusCode, blockedStatusCodes)` ⇒ `boolean`Retires session based on status code.@@ -176,6 +176,11 @@ Retires session based on status code.</tr><tr><td colspan="3"><p>HTTP status code</p>+</td></tr><tr>+<td><code>blockedStatusCodes</code></td><td><code>Array<Number></code></td>+</tr>+<tr>+<td colspan="3"><p>Custom HTTP status codes that means blocking on particular website.</p></td></tr></tbody></table><a name="Session+putResponse"></a>diff --git a/docs/guides/session_management.md b/docs/guides/session_management.mdindex 7c379c12d..2dff92406 100644--- a/docs/guides/session_management.md+++ b/docs/guides/session_management.md@@ -55,12 +55,12 @@ const crawler = new Apify.PuppeteerCrawler({useApifyProxy: true,// Activates the Session pool.useSessionPool: true,- // Overrides default Session pool configuration+ // Overrides default Session pool configuration.sessionPoolOptions: {maxPoolSize: 100},// Set to true if you want the crawler to save cookies per session,- // and set the cookie header to request automatically..+ // and set the cookie header to request automatically...persistCookiesPerSession: true,handlePageFunction: async ({request, $, session}) => {const title = $("title");@@ -80,15 +80,82 @@ const crawler = new Apify.PuppeteerCrawler({**Example usage in [`BasicCrawler`](../api/basiccrawler)**```javascript-Finish API first+ const crawler = new Apify.BasicCrawler({+ requestQueue,+ useSessionPool: true,+ sessionPoolOptions: {+ maxPoolSize: 100+ },+ handleRequestFunction: async ({request, session}) => {+ // To use the proxy IP session rotation logic you must turn the proxy usage on.+ const proxyUrl = Apify.getApifyProxyUrl({session});+ const requestOptions = {+ url: request.url,+ proxyUrl,+ throwHttpErrors: false,+ headers: {+ // If you want to use the cookieJar.+ // This way you get the Cookie headers string from session.+ Cookie: session.getCookieString(),+ }+ };+ let response;++ try {+ response = await Apify.utils.requestAsBrowser(requestOptions);+ } catch (e) {+ if (e === "SomeNetworkError") {+ // If network error happens like timeout, socket hangup etc...+ // There is usually a chance that it was just bad luck and the proxy works.+ // No need to throw it away.+ session.markBad();+ }+ throw e;+ }++ // Automatically retires the session based on response HTTP status code.+ session.retireOnBlockedStatusCodes(response.statusCode);++ if (response.body.blocked) {+ // You are sure it is blocking.+ // This will trow away the session.+ session.retire();++ }++ // Everything is ok you can get the data.+ // No need to call session.markGood -> BasicCrawler calls it for you.++ // If you want to use the CookieJar in session you need.+ session.setCookiesFromResponse(response);++ }+ });```**Example solo usage**```javascript-Finish API first-```+Apify.main(async () => {++ const sessionPoolOptions = {+ maxPoolSize: 100+ };+ const sessionPool = await Apify.openSessionPool(sessionPoolOptions);++ // Get session+ const session = sessionPool.getSession();++ // Increase the errorScore.+ session.markBad();+ // Throw away the session+ session.retire();++ // Lower the errorScore and marks the session good.+ session.markGood();+});+```These are the basics of configuring the SessionPool.Please, bear in mind that the Session pool needs some time to find the working IPs and build up the pool,so you will be probably seeing a lot of errors until it gets stabilized.