Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.
TypeScript apifyautomationcrawlercrawlingheadlessheadless-chromejavascriptnodejsnpmplaywrightpuppeteerscraperscrapingtypescriptweb-crawlerweb-crawlingweb-scraping
Build docs
Ondra Urbanf365ea74e5bc
diff --git a/docs/api/Apify.md b/docs/api/Apify.mdindex 2d97120e8..0e06b5178 100644--- a/docs/api/Apify.md+++ b/docs/api/Apify.md@@ -418,15 +418,17 @@ or the <a href="https://apify.com/docs/proxy" target="_blank">documentation</a>.<a name="module_Apify.getEnv"></a>## `Apify.getEnv()` ⇒ <code>Object</code>-Returns a new object which contains information parsed from the `APIFY_XXX` environment variables.-It has the following properties:+Returns a new object which contains information parsed from all the `APIFY_XXX` environment variables.+It has properties such as the following:```javascript{- // ID of the actor (APIFY_ACT_ID)- actId: String,- // ID of the actor run (APIFY_ACT_RUN_ID)- actRunId: String,+ // ID of the actor (APIFY_ACTOR_ID)+ actorId: String,+ // ID of the actor run (APIFY_ACTOR_RUN_ID)+ actorRunId: String,+ // ID of the actor task (APIFY_ACTOR_TASK_ID)+ actorTaskId: String,// ID of the user who started the actor - note that it might be// different than the owner of the actor (APIFY_USER_ID)userId: String,@@ -906,13 +908,13 @@ const requestList = await Apify.openRequestList('my-name', sources);Additionally, the <code>requestsFromUrl</code> property may be used instead of <code>url</code>,which will instruct <code>RequestList</code> to download the source URLs from a given remote location.The URLs will be parsed from the received response.</p>-<p> For details, see the (<code>RequestList</code>)[requestlist#new-exportsrequestlistoptions]+<p> For details, see the <a href="requestlist#new_RequestList_new"><code>RequestList</code></a>constructor options.</p></td></tr><tr><td><code>[options]</code></td><td><code>Object</code></td></tr><tr>-<td colspan="3"><p>The (<code>new RequestList</code>)(RequestList#new_RequestList_new) options. Note that the listName parameter supersedes+<td colspan="3"><p>The <a href="requestlist#new_RequestList_new"><code>new RequestList</code></a> options. Note that the listName parameter supersedesthe <code>persistStateKey</code> and <code>persistSourcesKey</code> options and the sources parameter supersedes the <code>sources</code> option.</p></td></tr></tbody></table>diff --git a/docs/api/LiveViewServer.md b/docs/api/LiveViewServer.mdindex 50b2f42d3..0e4c18772 100644--- a/docs/api/LiveViewServer.md+++ b/docs/api/LiveViewServer.md@@ -66,8 +66,8 @@ a better view into the browser, including DevTools, but `LiveViewServer` works t</tr><tr><td colspan="3"><p>By default, the screenshots are saved to- the <code>live_view</code> directory in the process' working directory. Provide a different- absolute path to change the settings.</p>+ the <code>live_view</code> directory in the Apify local storage directory.+ Provide a different absolute path to change the settings.</p></td></tr><tr><td><code>[options.maxScreenshotFiles]</code></td><td><code>number</code></td><td><code>10</code></td></tr>diff --git a/docs/api/Snapshotter.md b/docs/api/Snapshotter.mdindex 2c9aacfdc..fbff90453 100644--- a/docs/api/Snapshotter.md+++ b/docs/api/Snapshotter.md@@ -9,21 +9,23 @@ as either overloaded or not during the last interval. Keeps a history of the snaIt tracks the following resources: Memory, EventLoop, API and CPU.The class is used by the [`AutoscaledPool`](autoscaledpool) class.-There are differences in behavior when running locally and on the Apify platform,-but those differences are handled internally by the class and do not affect its interface.+When running on the Apify platform, the CPU and memory statistics are provided by the platform,+as collected from the running Docker container. When running locally, `Snapshotter`+makes its own statistics by querying the OS.++CPU becomes overloaded locally when its current use exceeds the `maxUsedCpuRatio` option or+when Apify platform marks it as overloaded.Memory becomes overloaded if its current use exceeds the `maxUsedMemoryRatio` option.It's computed using the total memory available to the container when running onthe Apify platform and a quarter of total system memory when running locally.-Max total memory may be overridden by using the `APIFY_MEMORY_MBYTES` environment variable.+Max total memory when running locally may be overridden by using the `APIFY_MEMORY_MBYTES`+environment variable.Event loop becomes overloaded if it slows down by more than the `maxBlockedMillis` option.Client becomes overloaded when rate limit errors (429 - Too Many Requests),-typically received from the request queue exceed the set limit within the set interval.--CPU tracking is available only on the Apify platform and the CPU overloaded event is read-directly off the container and is not configurable.+typically received from the request queue, exceed the set limit within the set interval.* [Snapshotter](snapshotter)@@ -69,10 +71,25 @@ directly off the container and is not configurable.<td colspan="3"><p>Maximum allowed delay of the event loop in milliseconds.Exceeding this limit overloads the event loop.</p></td></tr><tr>+<td><code>[options.cpuSnapshotIntervalSecs]</code></td><td><code>Number</code></td><td><code>1</code></td>+</tr>+<tr>+<td colspan="3"><p>Defines the interval of measuring CPU usage.+ This is only used when running locally. On the Apify platform,+ the statistics are provided externally at a fixed interval.</p>+</td></tr><tr>+<td><code>[options.maxUsedCpuRatio]</code></td><td><code>Number</code></td><td><code>0.95</code></td>+</tr>+<tr>+<td colspan="3"><p>Defines the maximum usage of CPU.+ Exceeding this limit overloads the CPU.</p>+</td></tr><tr><td><code>[options.memorySnapshotIntervalSecs]</code></td><td><code>Number</code></td><td><code>1</code></td></tr><tr><td colspan="3"><p>Defines the interval of measuring memory consumption.+ This is only used when running locally. On the Apify platform,+ the statistics are provided externally at a fixed interval.The measurement itself is resource intensive (25 - 50ms async).Therefore, setting this interval below 1 second is not recommended.</p></td></tr><tr>diff --git a/docs/examples/SynchronousRun.md b/docs/examples/SynchronousRun.mdindex 217592d9c..e26813292 100644--- a/docs/examples/SynchronousRun.md+++ b/docs/examples/SynchronousRun.md@@ -12,7 +12,7 @@ can be invoked synchronously using a single HTTP request to directly obtain its<a href="https://apify.com/docs/api/v2#/reference/actors/run-actor-synchronously/without-input" target="_blank">Run actor synchronously</a>Apify API endpoint. The example is also shared as the<a href="https://apify.com/apify/example-golden-gate-webcam" target="_blank">apify/example-golden-gate-webcam</a>-actor in the Apify Store, so you can test it directly there simply by sending a POST request to+actor in the Apify library, so you can test it directly there simply by sending a POST request to```httphttps://api.apify.com/v2/acts/apify~example-golden-gate-webcam/run-sync?token=[YOUR_API_TOKEN]```@@ -48,4 +48,4 @@ Apify.main(async () => {await Apify.setValue('OUTPUT', imageBuffer, { contentType: 'image/jpeg' });console.log('Actor finished.');});-```+```\ No newline at end of file