Skip to content

Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.

Build docs

Ondra Urban8fd3636ecc89
diff --git a/docs/api/Snapshotter.md b/docs/api/Snapshotter.md
index 11bc84905..2c9aacfdc 100644
--- a/docs/api/Snapshotter.md
+++ b/docs/api/Snapshotter.md
@@ -6,7 +6,7 @@ title: Snapshotter
Creates snapshots of system resources at given intervals and marks the resource
as either overloaded or not during the last interval. Keeps a history of the snapshots.
-It tracks the following resources: Memory, EventLoop and CPU.
+It tracks the following resources: Memory, EventLoop, API and CPU.
The class is used by the [`AutoscaledPool`](autoscaledpool) class.
There are differences in behavior when running locally and on the Apify platform,
@@ -19,6 +19,9 @@ Max total memory may be overridden by using the `APIFY_MEMORY_MBYTES` environmen
Event loop becomes overloaded if it slows down by more than the `maxBlockedMillis` option.
+Client becomes overloaded when rate limit errors (429 - Too Many Requests),
+typically received from the request queue exceed the set limit within the set interval.
+
CPU tracking is available only on the Apify platform and the CPU overloaded event is read
directly off the container and is not configurable.
@@ -30,6 +33,7 @@ directly off the container and is not configurable.
* [`.getMemorySample([sampleDurationMillis])`](#Snapshotter+getMemorySample) ⇒ <code>Array</code>
* [`.getEventLoopSample([sampleDurationMillis])`](#Snapshotter+getEventLoopSample) ⇒ <code>Array</code>
* [`.getCpuSample([sampleDurationMillis])`](#Snapshotter+getCpuSample) ⇒ <code>Array</code>
+ * [`.getClientSample(sampleDurationMillis)`](#Snapshotter+getClientSample) ⇒ <code>Array</code>
<a name="new_Snapshotter_new"></a>
@@ -53,6 +57,12 @@ directly off the container and is not configurable.
<tr>
<td colspan="3"><p>Defines the interval of measuring the event loop response time.</p>
</td></tr><tr>
+<td><code>[options.clientSnapshotIntervalSecs]</code></td><td><code>Number</code></td><td><code>1</code></td>
+</tr>
+<tr>
+<td colspan="3"><p>Defines the interval of checking the current state
+ of the remote API client.</p>
+</td></tr><tr>
<td><code>[options.maxBlockedMillis]</code></td><td><code>Number</code></td><td><code>50</code></td>
</tr>
<tr>
@@ -72,6 +82,12 @@ directly off the container and is not configurable.
<td colspan="3"><p>Defines the maximum ratio of total memory that can be used.
Exceeding this limit overloads the memory.</p>
</td></tr><tr>
+<td><code>[options.maxClientErrors]</code></td><td><code>Number</code></td><td><code>1</code></td>
+</tr>
+<tr>
+<td colspan="3"><p>Defines the maximum number of new rate limit errors within
+ the given interval.</p>
+</td></tr><tr>
<td><code>[options.snapshotHistorySecs]</code></td><td><code>Number</code></td><td><code>60</code></td>
</tr>
<tr>
@@ -146,3 +162,22 @@ by the sampleDurationMillis parameter. If omitted, it returns a full snapshot hi
<tr>
</tr></tbody>
</table>
+<a name="Snapshotter+getClientSample"></a>
+
+## `snapshotter.getClientSample(sampleDurationMillis)` ⇒ <code>Array</code>
+Returns a sample of latest Client snapshots, with the size of the sample defined
+by the sampleDurationMillis parameter. If omitted, it returns a full snapshot history.
+
+<table>
+<thead>
+<tr>
+<th>Param</th><th>Type</th>
+</tr>
+</thead>
+<tbody>
+<tr>
+<td><code>sampleDurationMillis</code></td><td><code>Number</code></td>
+</tr>
+<tr>
+</tr></tbody>
+</table>
diff --git a/docs/api/SystemStatus.md b/docs/api/SystemStatus.md
index 0f8d708a0..c72658e66 100644
--- a/docs/api/SystemStatus.md
+++ b/docs/api/SystemStatus.md
@@ -65,11 +65,17 @@ in the [`Snapshotter`](snapshotter) instance.
<td colspan="3"><p>Sets the maximum ratio of overloaded snapshots in an event loop sample.
If the sample exceeds this ratio, the system will be overloaded.</p>
</td></tr><tr>
-<td><code>[options.maxCpuOverloadedRatio]</code></td><td><code>Number</code></td><td><code>0.1</code></td>
+<td><code>[options.maxCpuOverloadedRatio]</code></td><td><code>Number</code></td><td><code>0.4</code></td>
</tr>
<tr>
<td colspan="3"><p>Sets the maximum ratio of overloaded snapshots in a CPU sample.
If the sample exceeds this ratio, the system will be overloaded.</p>
+</td></tr><tr>
+<td><code>[options.maxClientOverloadedRatio]</code></td><td><code>Number</code></td><td><code>0.2</code></td>
+</tr>
+<tr>
+<td colspan="3"><p>Sets the maximum ratio of overloaded snapshots in a Client sample.
+ If the sample exceeds this ratio, the system will be overloaded.</p>
</td></tr></tbody>
</table>
<a name="SystemStatus+getCurrentStatus"></a>
diff --git a/docs/api/social.md b/docs/api/social.md
index 0eada551e..0eba462a6 100644
--- a/docs/api/social.md
+++ b/docs/api/social.md
@@ -34,7 +34,7 @@ const emails = Apify.utils.social.emailsFromText('[email protected] bob@example.
<a name="social.LINKEDIN_REGEX"></a>
-## `social.LINKEDIN\_REGEX` : <code>RegExp</code>
+## `social.LINKEDIN_REGEX` : <code>RegExp</code>
Regular expression to exactly match a single LinkedIn profile URL, without any additional
subdirectories or query parameters. The regular expression has the following form: `/^...$/i`.
@@ -45,55 +45,55 @@ TODO
<a name="social.LINKEDIN_REGEX_GLOBAL"></a>
-## `social.LINKEDIN\_REGEX\_GLOBAL` : <code>RegExp</code>
+## `social.LINKEDIN_REGEX_GLOBAL` : <code>RegExp</code>
Regular expression to find multiple LinkedIn profile URLs in a text.
It has the following form: `/.../ig`.
<a name="social.INSTAGRAM_REGEX"></a>
-## `social.INSTAGRAM\_REGEX` : <code>RegExp</code>
+## `social.INSTAGRAM_REGEX` : <code>RegExp</code>
Regular expression to exactly match a single Instagram profile URL.
It has the following form: `/^...$/i`.
<a name="social.INSTAGRAM_REGEX_GLOBAL"></a>
-## `social.INSTAGRAM\_REGEX\_GLOBAL` : <code>RegExp</code>
+## `social.INSTAGRAM_REGEX_GLOBAL` : <code>RegExp</code>
Regular expression to find multiple Instagram profile URLs in a text.
It has the following form: `/.../ig`.
<a name="social.TWITTER_REGEX"></a>
-## `social.TWITTER\_REGEX` : <code>RegExp</code>
+## `social.TWITTER_REGEX` : <code>RegExp</code>
Regular expression to exactly match a single Instagram profile URL.
It has the following form: `/^...$/i`.
<a name="social.TWITTER_REGEX_GLOBAL"></a>
-## `social.TWITTER\_REGEX\_GLOBAL` : <code>RegExp</code>
+## `social.TWITTER_REGEX_GLOBAL` : <code>RegExp</code>
Regular expression to find multiple Instagram profile URLs in a text.
It has the following form: `/.../ig`.
<a name="social.FACEBOOK_REGEX"></a>
-## `social.FACEBOOK\_REGEX` : <code>RegExp</code>
+## `social.FACEBOOK_REGEX` : <code>RegExp</code>
Regular expression to exactly match a single Facebook user profile URL.
It has the following form: `/^...$/i`.
<a name="social.FACEBOOK_REGEX_GLOBAL"></a>
-## `social.FACEBOOK\_REGEX\_GLOBAL` : <code>RegExp</code>
+## `social.FACEBOOK_REGEX_GLOBAL` : <code>RegExp</code>
Regular expression to find multiple Instagram profile URLs in a text.
It has the following form: `/.../ig`.
<a name="social.EMAIL_REGEX"></a>
-## `social.EMAIL\_REGEX` : <code>RegExp</code>
+## `social.EMAIL_REGEX` : <code>RegExp</code>
Regular expression to exactly match a single email address.
It has the following form: `/^...$/i`.
<a name="social.EMAIL_REGEX_GLOBAL"></a>
-## `social.EMAIL\_REGEX\_GLOBAL` : <code>RegExp</code>
+## `social.EMAIL_REGEX_GLOBAL` : <code>RegExp</code>
Regular expression to find multiple email addresses in a text.
It has the following form: `/.../ig`.
diff --git a/docs/api/utils.md b/docs/api/utils.md
index f1abe4c24..2dd1421fb 100644
--- a/docs/api/utils.md
+++ b/docs/api/utils.md
@@ -28,13 +28,13 @@ await Apify.utils.sleep(1500);
<a name="utils.URL_NO_COMMAS_REGEX"></a>
-## `utils.URL\_NO\_COMMAS\_REGEX`
+## `utils.URL_NO_COMMAS_REGEX`
Default regular expression to match URLs in a string that may be plain text, JSON, CSV or other. It supports common URL characters
and does not support URLs containing commas or spaces. The URLs also may contain Unicode letters (not symbols).
<a name="utils.URL_WITH_COMMAS_REGEX"></a>
-## `utils.URL\_WITH\_COMMAS\_REGEX`
+## `utils.URL_WITH_COMMAS_REGEX`
Regular expression that, in addition to the default regular expression `URL_NO_COMMAS_REGEX`, supports matching commas in URL path and query.
Note, however, that this may prevent parsing URLs from comma delimited lists, or the URLs may become malformed.