Skip to content

Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.

builded docs

Petr Patekea98dfaab4cf
diff --git a/docs/api/Apify.md b/docs/api/Apify.md
index 5807e62d3..f880f1176 100644
--- a/docs/api/Apify.md
+++ b/docs/api/Apify.md
@@ -82,7 +82,7 @@ effect.
<td colspan="3"><p>Idempotency key enables you to ensure that a webhook will not be added multiple times in case of
an actor restart or other situation that would cause the <code>addWebhook()</code> function to be called again.
We suggest using the actor run ID as the idempotency key. You can get the run ID by calling
- [`Apify.getEnv()](apify#module_Apify.getEnv) function.</p>
+ <a href="apify#module_Apify.getEnv">`Apify.getEnv()</a> function.</p>
</td></tr></tbody>
</table>
<a name="module_Apify.call"></a>
diff --git a/docs/api/RequestList.md b/docs/api/RequestList.md
index c98df283f..b1e8f39c8 100644
--- a/docs/api/RequestList.md
+++ b/docs/api/RequestList.md
@@ -19,8 +19,9 @@ After that, no more URLs can be added to the list.
URLs to crawl. Unlike [`RequestQueue`](requestqueue), `RequestList` is static but it can contain even millions of URLs.
`RequestList` has an internal state where it stores information about which requests were already handled, which are in progress and which were
-reclaimed. The state may be automatically persisted to the default key-value store by setting the `persistStateKey` option so that if the Node.js
-process is restarted, the crawling can continue where it left off. For more details, see [`KeyValueStore`](keyvaluestore).
+reclaimed. The state may be automatically persisted to the default [`KeyValueStore`](keyvaluestore) by setting the `persistStateKey` option so that if
+the Node.js process is restarted, the crawling can continue where it left off. The automated persisting is launched upon receiving the `persistState`
+event that is periodically emitted by [`Apify.events`](events).
The internal state is closely tied to the provided sources (URLs) to validate it's position in the list after a migration or restart. Therefore, if
the sources change, the state will become corrupted and `RequestList` will raise an exception. This typically happens when using a live list of URLs