Skip to content

Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.

builded docs.

Petr Patek77b3613dca20
diff --git a/docs/api/Session.md b/docs/api/Session.md
index 395d8f113..9c68fa556 100644
--- a/docs/api/Session.md
+++ b/docs/api/Session.md
@@ -21,7 +21,11 @@ internal state can be enriched with custom user data for example some authorizat
- [`.markBad()`](#Session+markBad)
- [`.checkStatus(statusCode)`](#Session+checkStatus) ⇒ `boolean`
- [`.putResponse(response)`](#Session+putResponse)
+ - [`.putPuppeteerCookies(puppeteerCookies, url)`](#Session+putPuppeteerCookies)
+ - [`.setCookies(cookies, url)`](#Session+setCookies)
+ - [`.getCookies(url)`](#Session+getCookies) ⇒ `Array<Object>`
- [`.getCookieString(url)`](#Session+getCookieString) ⇒ `String`
+ - [`.getPuppeteerCookies(url)`](#Session+getPuppeteerCookies) ⇒ `*`
<a name="new_Session_new"></a>
@@ -193,13 +197,80 @@ Sets cookies from response to the cookieJar. Parses cookies from `set-cookie` he
<tr>
</tr></tbody>
</table>
+<a name="Session+putPuppeteerCookies"></a>
+
+## `session.putPuppeteerCookies(puppeteerCookies, url)`
+
+Persists puppeteer cookies to session for reuse.
+
+<table>
+<thead>
+<tr>
+<th>Param</th>
+</tr>
+</thead>
+<tbody>
+<tr>
+<td><code>puppeteerCookies</code></td>
+</tr>
+<tr>
+<td colspan="3"><p>cookie from puppeteer <code>page.cookies</code> method.</p>
+</td></tr><tr>
+<td><code>url</code></td>
+</tr>
+<tr>
+<td colspan="3"><p>Loaded url from page function.</p>
+</td></tr></tbody>
+</table>
+<a name="Session+setCookies"></a>
+
+## `session.setCookies(cookies, url)`
+
+Set cookies to session cookieJar. Cookies array should be compatible with tough-cookie.
+
+<table>
+<thead>
+<tr>
+<th>Param</th><th>Type</th>
+</tr>
+</thead>
+<tbody>
+<tr>
+<td><code>cookies</code></td><td><code>Array<Cookie></code></td>
+</tr>
+<tr>
+</tr><tr>
+<td><code>url</code></td><td><code>String</code></td>
+</tr>
+<tr>
+</tr></tbody>
+</table>
+<a name="Session+getCookies"></a>
+
+## `session.getCookies(url)` ⇒ `Array<Object>`
+
+Get cookies. Gets a array of `tough-cookie` Cookie instances.
+
+<table>
+<thead>
+<tr>
+<th>Param</th><th>Type</th>
+</tr>
+</thead>
+<tbody>
+<tr>
+<td><code>url</code></td><td><code>String</code></td>
+</tr>
+<tr>
+</tr></tbody>
+</table>
<a name="Session+getCookieString"></a>
## `session.getCookieString(url)` ⇒ `String`
Wrapper around `tough-cookie` Cookie jar `getCookieString` method.
-**Returns**: `String` - String representing `Cookie` header.
+**Returns**: `String` - - represents `Cookie` header.
<table>
<thead>
@@ -214,3 +285,22 @@ Wrapper around `tough-cookie` Cookie jar `getCookieString` method.
<tr>
</tr></tbody>
</table>
+<a name="Session+getPuppeteerCookies"></a>
+
+## `session.getPuppeteerCookies(url)` ⇒ `*`
+
+Gets cookies in format ready for puppeteer.
+
+<table>
+<thead>
+<tr>
+<th>Param</th><th>Type</th>
+</tr>
+</thead>
+<tbody>
+<tr>
+<td><code>url</code></td><td><code>String</code></td>
+</tr>
+<tr>
+</tr></tbody>
+</table>
diff --git a/docs/api/SessionPool.md b/docs/api/SessionPool.md
index bd8281db9..2f5d108c0 100644
--- a/docs/api/SessionPool.md
+++ b/docs/api/SessionPool.md
@@ -16,10 +16,10 @@ Session pool is by default persisted in default [`KeyValueStore`](keyvaluestore)
```javascript
const sessionPool = new SessionPool({
maxPoolSize: 25,
- sessionOptions:{
- maxAgeSecs: 10,
- maxUsageCount: 150, // for example when you know that the site blocks after 150 requests.
- }
+ sessionOptions: {
+ maxAgeSecs: 10,
+ maxUsageCount: 150, // for example when you know that the site blocks after 150 requests.
+ },
persistStateKeyValueStoreId: 'my-key-value-store-for-sessions',
persistStateKey: 'my-session-pool',
});
@@ -38,14 +38,13 @@ const session3 = await sessionPool.getSession();
// Now you can mark the session either failed of successful
// Marks session as bad after unsuccessful usage -> it increases error count (soft retire)
-session1.markBad()
+session1.markBad();
// Marks as successful.
-session2.markGood()
+session2.markGood();
// Retires session -> session is removed from the pool
-session3.retire()
-
+session3.retire();
```
- [SessionPool](sessionpool)