Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.
TypeScript apifyautomationcrawlercrawlingheadlessheadless-chromejavascriptnodejsnpmplaywrightpuppeteerscraperscrapingtypescriptweb-crawlerweb-crawlingweb-scraping
Build docs [skip ci]
Ondra Urbanf9c2fd8f243e
diff --git a/docs/api/RequestList.md b/docs/api/RequestList.mdindex c98df283f..b1e8f39c8 100644--- a/docs/api/RequestList.md+++ b/docs/api/RequestList.md@@ -19,8 +19,9 @@ After that, no more URLs can be added to the list.URLs to crawl. Unlike [`RequestQueue`](requestqueue), `RequestList` is static but it can contain even millions of URLs.`RequestList` has an internal state where it stores information about which requests were already handled, which are in progress and which were-reclaimed. The state may be automatically persisted to the default key-value store by setting the `persistStateKey` option so that if the Node.js-process is restarted, the crawling can continue where it left off. For more details, see [`KeyValueStore`](keyvaluestore).+reclaimed. The state may be automatically persisted to the default [`KeyValueStore`](keyvaluestore) by setting the `persistStateKey` option so that if+the Node.js process is restarted, the crawling can continue where it left off. The automated persisting is launched upon receiving the `persistState`+event that is periodically emitted by [`Apify.events`](events).The internal state is closely tied to the provided sources (URLs) to validate it's position in the list after a migration or restart. Therefore, ifthe sources change, the state will become corrupted and `RequestList` will raise an exception. This typically happens when using a live list of URLs