Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.
TypeScript apifyautomationcrawlercrawlingheadlessheadless-chromejavascriptnodejsnpmplaywrightpuppeteerscraperscrapingtypescriptweb-crawlerweb-crawlingweb-scraping
Fix RequestList memory (#606)
Ondra Urban7897bca2f412
diff --git a/CHANGELOG.md b/CHANGELOG.mdindex eda4d9662..e2968add3 100644--- a/CHANGELOG.md+++ b/CHANGELOG.md@@ -7,10 +7,21 @@is `true` by default and forces the function to use a HTTP parser that is less strict thandefault Node 12 parser, but also less secure. It is needed to be able to bypass certainanti-scraping walls and fetch websites that do not comply with HTTP spec.+- **BREAKING:** `RequestList` now removes all the elements from the `sources` array on+ initialization. If you need to use the sources somewhere else, make a copy. This change+ was added as one of several measures to improve memory management of `RequestList`+ in scenarios with very large amount of `Request` instances.+- **DEPRECATED:** `RequestListOptions.persistSourcesKey` is now deprecated. Please use+ `RequestListOptions.persistRequestsKey`.+- `RequestList.sources` can now be an array of `string` URLs as well.+- Added `sourcesFunction` to `RequestListOptions`. It enables dynamic fetching of sources+ and will only be called if persisted `Requests` were not retrieved from key-value store.+ Use it to reduce memory spikes and also to make sure that your sources are not re-created+ on actor restarts.- Improved `Apify.launchPuppeteer()` code to prevent triggering bugs in Puppeteer by passingmore than required options to `puppeteer.launch()`.-- Documented `BasicCrawler.autoscaledPool` property,- and added `CheerioCrawler.autoscaledPool` and `PuppeteerCrawler.autoscaledPool` properties.+- Documented `BasicCrawler.autoscaledPool` property, and added `CheerioCrawler.autoscaledPool`+ and `PuppeteerCrawler.autoscaledPool` properties.- Added TypeScript typings and typedef documentation for all entities used throughout SDK.- Upgraded `proxy-chain` NPM package from 0.2.7 to 0.3.3diff --git a/docs/api/Apify.md b/docs/api/Apify.mdindex 8899304ea..0cf09965c 100644--- a/docs/api/Apify.md+++ b/docs/api/Apify.md@@ -296,7 +296,8 @@ the variables are not defined or are invalid, the corresponding value in the res## `Apify.getInput()`-**Returns**: `Promise<(object|null)>` - Returns a promise that resolves once the record is stored.+**Returns**: `Promise<(object|string|Buffer|null)>` - Returns a promise that resolves to an object, string or+[`Buffer`](https://nodejs.org/api/buffer.html), depending on the MIME content type of the record, or `null` if the record is missing.Gets the actor input value from the default [`KeyValueStore`](/docs/api/key-value-store) associated with the current actor run.@@ -337,7 +338,8 @@ second.## `Apify.getValue(key)`-**Returns**: `Promise<(object|null)>` - Returns a promise that resolves once the record is stored.+**Returns**: `Promise<(object|string|Buffer|null)>` - Returns a promise that resolves to an object, string or+[`Buffer`](https://nodejs.org/api/buffer.html), depending on the MIME content type of the record, or `null` if the record is missing.Gets a value from the default [`KeyValueStore`](/docs/api/key-value-store) associated with the current actor run.@@ -452,7 +454,7 @@ The `Apify.main()` function performs the following actions:<li>When running on the Apify platform (i.e. <code>APIFY_IS_AT_HOME</code> environment variable is set),it sets up a connection to listen for platform events.For example, to get a notification about an imminent migration to another server.- See [](apify#apifyevents) for details.+ See [`Apify.events`](/docs/api/apify#events) for details.</li><li>It checks that either <code>APIFY_TOKEN</code> or <code>APIFY_LOCAL_STORAGE_DIR</code> environment variableis defined. If not, the functions sets <code>APIFY_LOCAL_STORAGE_DIR</code> to <code>./apify_storage</code>@@ -608,9 +610,12 @@ const requestList = await Apify.openRequestList('my-name', sources);If `null`, the list will not be persisted and will only be stored in memory. Process restart will then cause the list to be crawled again from thebeginning. We suggest always using a name.-- **`sources`**: [`SourceInput`](/docs/typedefs/source-input) | `Array<string>` - An array of sources of URLs for the- [`RequestList`](/docs/api/request-list). It can be either an array of plain objects that define at least the `url` property, or an array of- instances of the [`Request`](/docs/api/request) class.+- **`sources`**: [`Array<(RequestOptions|Request|string)>`](/docs/typedefs/request-options) - An array of sources of URLs for the+ [`RequestList`](/docs/api/request-list). It can be either an array of strings, plain objects that define at least the `url` property, or an array+ of [`Request`](/docs/api/request) instances.++ **IMPORTANT:** The `sources` array will be consumed (left empty) after [`RequestList`](/docs/api/request-list) initializes. This is a measure to+ prevent memory leaks in situations when millions of sources are added.Additionally, the `requestsFromUrl` property may be used instead of `url`, which will instruct [`RequestList`](/docs/api/request-list) to download thesource URLs from a given remote location. The URLs will be parsed from the received response. In this case you can limit the URLs using `regex`@@ -620,8 +625,8 @@ For details, see the [`RequestListOptions.sources`](/docs/typedefs/request-list-- **`[options]`**: [`RequestListOptions`](/docs/typedefs/request-list-options) - The [`RequestList`](/docs/api/request-list) options. Note that the`listName` parameter supersedes the [`RequestListOptions.persistStateKey`](/docs/typedefs/request-list-options#persiststatekey) and- [`RequestListOptions.persistSourcesKey`](/docs/typedefs/request-list-options#persistsourceskey) options and the `sources` parameter supersedes the- [`RequestListOptions.sources`](/docs/typedefs/request-list-options#sources) option.+ [`RequestListOptions.persistRequestsKey`](/docs/typedefs/request-list-options#persistrequestskey) options and the `sources` parameter supersedes+ the [`RequestListOptions.sources`](/docs/typedefs/request-list-options#sources) option.---@@ -731,7 +736,7 @@ For more information, see [`Apify.openKeyValueStore()`](/docs/api/apify#openkeyv<li>If `null`, the record in the key-value store is deleted.</li><li>If no `options.contentType` is specified, `value` can be any JavaScript object and it will be stringified to JSON.</li><li>If `options.contentType` is specified, `value` is considered raw data and it must be a `String`- or [](https://nodejs.org/api/buffer.html).</li>+ or [`Buffer`](https://nodejs.org/api/buffer.html).</li></ul>For any other value an error will be thrown.- **`[options]`**: `Object`diff --git a/docs/api/KeyValueStore.md b/docs/api/KeyValueStore.mdindex d110b2dd7..23b868115 100644--- a/docs/api/KeyValueStore.md+++ b/docs/api/KeyValueStore.md@@ -80,14 +80,14 @@ await store.drop();## `keyValueStore.getValue(key)`-**Returns**: `Promise<(object|Buffer|string|null)>` - Returns a promise that resolves to an object, string or [](https://nodejs.org/api/buffer.html),-depending on the MIME content type of the record.+**Returns**: `Promise<(object|Buffer|string|null)>` - Returns a promise that resolves to an object, string or+[`Buffer`](https://nodejs.org/api/buffer.html), depending on the MIME content type of the record.Gets a value from the key-value store.The function returns a `Promise` that resolves to the record value, whose JavaScript type depends on the MIME content type of the record. Records withthe `application/json` content type are automatically parsed and returned as a JavaScript object. Similarly, records with `text/plain` content types-are returned as a string. For all other content types, the value is returned as a raw [](https://nodejs.org/api/buffer.html) instance.+are returned as a string. For all other content types, the value is returned as a raw [`Buffer`](https://nodejs.org/api/buffer.html) instance.If the record does not exist, the function resolves to `null`.@@ -132,7 +132,7 @@ const store = await Apify.openKeyValueStore('my-text-store');await store.setValue('RESULTS', 'my text data', { contentType: 'text/plain' });```-If you set custom content type, `value` must be either a string or [](https://nodejs.org/api/buffer.html), otherwise an error will be thrown.+If you set custom content type, `value` must be either a string or [`Buffer`](https://nodejs.org/api/buffer.html), otherwise an error will be thrown.If `value` is `null`, the record is deleted instead. Note that the `setValue()` function succeeds regardless whether the record existed or not.@@ -150,7 +150,7 @@ stored!<li>If `null`, the record in the key-value store is deleted.</li><li>If no `options.contentType` is specified, `value` can be any JavaScript object and it will be stringified to JSON.</li><li>If `options.contentType` is specified, `value` is considered raw data and it must be either a `String`- or [](https://nodejs.org/api/buffer.html).</li>+ or [`Buffer`](https://nodejs.org/api/buffer.html).</li></ul>For any other value an error will be thrown.- **`[options]`**: `Object`diff --git a/docs/api/RequestList.md b/docs/api/RequestList.mdindex ea99b1655..0ceca0ef2 100644--- a/docs/api/RequestList.md+++ b/docs/api/RequestList.md@@ -26,42 +26,41 @@ the `persistState` event that is periodically emitted by [`Apify.events`](/docs/The internal state is closely tied to the provided sources (URLs). If the sources change on actor restart, the state will become corrupted and`RequestList` will raise an exception. This typically happens when the sources is a list of URLs downloaded from the web. In such case, use the-`persistSourcesKey` option in conjunction with `persistStateKey`, to make the `RequestList` store the initial sources to the default key-value store+`persistRequestsKey` option in conjunction with `persistStateKey`, to make the `RequestList` store the initial sources to the default key-value storeand load them after restart, which will prevent any issues that a live list of URLs might cause.-**Example usage:**+**Basic usage:**```javascript+// Use a helper function to simplify request list initialization.+// State and sources are automatically persisted.+const requestList = await Apify.openRequestList('my-request-list', [+ 'http://www.example.com/page-1',+ { url: 'http://www.example.com/page-2', method: 'POST', userData: { foo: 'bar' } },+ { requestsFromUrl: 'http://www.example.com/my-url-list.txt', userData: { isFromUrl: true } },+]);+```++**Advanced usage:**++```javascript+// Use the constructor to get more control over the initialization.const requestList = new Apify.RequestList({sources: [// Separate requests- { url: 'http://www.example.com/page-1', method: 'GET', headers: {} },- { url: 'http://www.example.com/page-2', userData: { foo: 'bar' } },+ { url: 'http://www.example.com/page-1', method: 'GET', headers: { ... } },+ { url: 'http://www.example.com/page-2', userData: { foo: 'bar' }},// Bulk load of URLs from file `http://www.example.com/my-url-list.txt`// Note that all URLs must start with http:// or https://{ requestsFromUrl: 'http://www.example.com/my-url-list.txt', userData: { isFromUrl: true } },],- // Ensure both the sources and crawling state of the request list is persisted,- // so that on actor restart, the crawling will continue where it left off+ // Persist only state in cases where the original sources are immutable to improve performance.persistStateKey: 'my-state',- persistSourcesKey: 'my-sources',});-// This call loads and parses the URLs from the remote file.await requestList.initialize();--// Get requests from list-const request1 = await requestList.fetchNextRequest();-const request2 = await requestList.fetchNextRequest();-const request3 = await requestList.fetchNextRequest();--// Mark some of them as handled-await requestList.markRequestHandled(request1);--// If processing fails then reclaim it back to the list-await requestList.reclaimRequest(request2);```---diff --git a/docs/api/puppeteer.md b/docs/api/puppeteer.mdindex cca0c06a1..6cc8962dc 100644--- a/docs/api/puppeteer.md+++ b/docs/api/puppeteer.md@@ -72,7 +72,7 @@ await page.goto('http://example.com');**Params**-- **`page`**: `Page` - Puppeteer [](https://pptr.dev/#?product=Puppeteer&show=api-class-page) object.+- **`page`**: `Page` - Puppeteer [`Page`](https://pptr.dev/#?product=Puppeteer&show=api-class-page) object.- **`handler`**: [`InterceptHandler`](/docs/typedefs/intercept-handler) - Request interception handler.---@@ -87,7 +87,7 @@ Removes request interception handler for given page.**Params**-- **`page`**: `Page` - Puppeteer [](https://pptr.dev/#?product=Puppeteer&show=api-class-page) object.+- **`page`**: `Page` - Puppeteer [`Page`](https://pptr.dev/#?product=Puppeteer&show=api-class-page) object.- **`handler`**: [`InterceptHandler`](/docs/typedefs/intercept-handler) - Request interception handler.---@@ -106,7 +106,7 @@ performance.**Params**-- **`page`**: `Page` - Puppeteer [](https://pptr.dev/#?product=Puppeteer&show=api-class-page) object.+- **`page`**: `Page` - Puppeteer [`Page`](https://pptr.dev/#?product=Puppeteer&show=api-class-page) object.- **`request`**: [`Request`](/docs/api/request)- **`gotoOptions`**: `DirectNavigationOptions` - Custom options for `page.goto()`.@@ -122,7 +122,7 @@ Scrolls to the bottom of a page, or until it times out. Loads dynamic content wh**Params**-- **`page`**: `Page` - Puppeteer [](https://pptr.dev/#?product=Puppeteer&show=api-class-page) object.+- **`page`**: `Page` - Puppeteer [`Page`](https://pptr.dev/#?product=Puppeteer&show=api-class-page) object.- **`[options]`**: `Object`- **`[.timeoutSecs]`**: `number` <code> = 0</code> - How many seconds to scroll for. If 0, will scroll until bottom of page.- **`[.waitForSecs]`**: `number` <code> = 4</code> - How many seconds to wait for no new content to load before exit.@@ -139,7 +139,7 @@ Saves a full screenshot and HTML of the current page into a Key-Value store.**Params**-- **`page`**: `Page` - Puppeteer [](https://pptr.dev/#?product=Puppeteer&show=api-class-page) object.+- **`page`**: `Page` - Puppeteer [`Page`](https://pptr.dev/#?product=Puppeteer&show=api-class-page) object.- **`[options]`**: `Object`- **`[.key]`**: `string` <code> = "SNAPSHOT"</code> - Key under which the screenshot and HTML will be saved. `.jpg` will be appendedfor screenshot and `.html` for HTML.@@ -269,7 +269,7 @@ File contents are cached for up to 10 files to limit file system access.**Params**-- **`page`**: `Page` - Puppeteer [](https://pptr.dev/#?product=Puppeteer&show=api-class-page) object.+- **`page`**: `Page` - Puppeteer [`Page`](https://pptr.dev/#?product=Puppeteer&show=api-class-page) object.- **`filePath`**: `string` - File path- **`[options]`**: `Object`- **`[.surviveNavigations]`**: `boolean` - Enables the injected script to survive page navigations and reloads without need to be re-injected@@ -284,8 +284,8 @@ File contents are cached for up to 10 files to limit file system access.**Returns**: `Promise<*>`-Injects the [](https://jquery.com/) library into a Puppeteer page. jQuery is often useful for various web scraping and crawling tasks. For example, it-can help extract text from HTML elements using CSS selectors.+Injects the [jQuery](https://jquery.com/) library into a Puppeteer page. jQuery is often useful for various web scraping and crawling tasks. For+example, it can help extract text from HTML elements using CSS selectors.Beware that the injected jQuery object will be set to the `window.$` variable and thus it might cause conflicts with other libraries included by thepage that use the same variable name (e.g. another version of jQuery). This can affect functionality of page's scripts.@@ -301,11 +301,12 @@ const title = await page.evaluate(() => {});```-Note that `injectJQuery()` does not affect the Puppeteer's [](https://pptr.dev/#?product=Puppeteer&show=api-pageselector) function in any way.+Note that `injectJQuery()` does not affect the Puppeteer's [`page.$()`](https://pptr.dev/#?product=Puppeteer&show=api-pageselector) function in any+way.**Params**-- **`page`**: `Page` - Puppeteer [](https://pptr.dev/#?product=Puppeteer&show=api-class-page) object.+- **`page`**: `Page` - Puppeteer [`Page`](https://pptr.dev/#?product=Puppeteer&show=api-class-page) object.---@@ -315,7 +316,7 @@ Note that `injectJQuery()` does not affect the Puppeteer's [](https://pptr.dev/#**Returns**: `Promise<*>`-Injects the [](https://underscorejs.org/) library into a Puppeteer page.+Injects the [Underscore](https://underscorejs.org/) library into a Puppeteer page.Beware that the injected Underscore object will be set to the `window._` variable and thus it might cause conflicts with libraries included by thepage that use the same variable name. This can affect functionality of page's scripts.@@ -380,7 +381,7 @@ await page.goto('https://cnn.com');**Params**-- **`page`**: `Page` - Puppeteer [](https://pptr.dev/#?product=Puppeteer&show=api-class-page) object.+- **`page`**: `Page` - Puppeteer [`Page`](https://pptr.dev/#?product=Puppeteer&show=api-class-page) object.- **`[options]`**: `Object`- **`[.urlPatterns]`**: `Array<string>` - The patterns of URLs to block from being loaded by the browser. Only `*` can be used as a wildcard. Itis also automatically added to the beginning and end of the pattern. This limitation is enforced by the DevTools protocol. `.png` is the same@@ -406,7 +407,7 @@ mitigated in future iterations of this feature.**Params**-- **`page`**: `Page` - Puppeteer [](https://pptr.dev/#?product=Puppeteer&show=api-class-page) object.+- **`page`**: `Page` - Puppeteer [`Page`](https://pptr.dev/#?product=Puppeteer&show=api-class-page) object.- **`cache`**: `Object` - Object in which responses are stored- **`responseUrlRules`**: `Array<(string|RegExp)>` - List of rules that are used to check if the response should be cached. String rules arecompared as page.url().includes(rule) while RegExp rules are evaluated as rule.test(page.url()).@@ -428,7 +429,7 @@ Compiles a Puppeteer script into an async function that may be executed at any t}```-Where `page` is a Puppeteer [](https://pptr.dev/#?product=Puppeteer&show=api-class-page) and `request` is a [`Request`](/docs/api/request).+Where `page` is a Puppeteer [`Page`](https://pptr.dev/#?product=Puppeteer&show=api-class-page) and `request` is a [`Request`](/docs/api/request).The function is compiled by using the `scriptString` parameter as the function's body, so any limitations to function bodies apply. Return value ofthe compiled function is the return value of the function body = the `scriptString` parameter.diff --git a/docs/typedefs/KeyConsumer.md b/docs/typedefs/KeyConsumer.mdindex a02ae781a..c06aa6c9d 100644--- a/docs/typedefs/KeyConsumer.md+++ b/docs/typedefs/KeyConsumer.md@@ -10,8 +10,8 @@ User-function used in the [`KeyValueStore.forEachKey()`](/docs/api/key-value-sto**Params**- **`key`**: `string` - Current {KeyValue} key being processed.-- **`index`**: `number` - Position of the current key in {KeyValuestore}.-- **`info`**: `object` - Information about the current {KeyValueStore} entry.+- **`index`**: `number` - Position of the current key in [`KeyValueStore`](/docs/api/key-value-store).+- **`info`**: `object` - Information about the current [`KeyValueStore`](/docs/api/key-value-store) entry.- **`.size`**: `number` - Size of the value associated with the current key in bytes.---diff --git a/docs/typedefs/RequestListOptions.md b/docs/typedefs/RequestListOptions.mdindex f3a8d0367..2613ee172 100644--- a/docs/typedefs/RequestListOptions.md+++ b/docs/typedefs/RequestListOptions.md@@ -9,16 +9,24 @@ title: RequestListOptions### `sources`-**Type**: [`SourceInput`](/docs/typedefs/source-input)+**Type**: [`Array<(RequestOptions|Request|string)>`](/docs/typedefs/request-options)-An array of sources of URLs for the `RequestList`. It can be either an array of plain objects that define the `url` property, or an array of instances-of the [`Request`](/docs/api/request) class. Additionally, the `requestsFromUrl` property may be used instead of `url`, which will instruct-`RequestList` to download the source URLs from a given remote location. The URLs will be parsed from the received response.+An array of sources of URLs for the [`RequestList`](/docs/api/request-list). It can be either an array of strings, plain objects that define at least+the `url` property, or an array of [`Request`](/docs/api/request) instances.++**IMPORTANT:** The `sources` array will be consumed (left empty) after `RequestList` initializes. This is a measure to prevent memory leaks in+situations when millions of sources are added.++Additionally, the `requestsFromUrl` property may be used instead of `url`, which will instruct `RequestList` to download the source URLs from a given+remote location. The URLs will be parsed from the received response.```[// A single URL- { method: 'GET', url: 'http://example.com/a/b' },+ 'http://example.com/a/b',++ // Modify Request options+ { method: PUT, 'https://example.com/put, payload: { foo: 'bar' }}// Batch import of URLs from a file hosted on the web,// where the URLs should be requested using the HTTP POST request@@ -35,6 +43,52 @@ of the [`Request`](/docs/api/request) class. Additionally, the `requestsFromUrl`---+### `sourcesFunction`++**Type**: `function`++A function that will be called to get the sources for the `RequestList`, but only if `RequestList` was not able to fetch their persisted version (see+[`RequestListOptions.persistRequestsKey`](/docs/typedefs/request-list-options#persistrequestskey)). It must return an `Array` of+[`Request`](/docs/api/request) or [`RequestOptions`](/docs/typedefs/request-options).++This is very useful in a scenario when getting the sources is a resource intensive or time consuming task, such as fetching URLs from multiple+sitemaps or parsing URLs from large datasets. Using the `sourcesFunction` in combination with `persistStateKey` and `persistRequestsKey` will allow+you to fetch and parse those URLs only once, saving valuable time when your actor migrates or restarts.++If both [`RequestListOptions.sources`](/docs/typedefs/request-list-options#sources) and+[`RequestListOptions.sourcesFunction`](/docs/typedefs/request-list-options#sourcesfunction) are provided, the sources returned by the function will be+added after the `sources`.++**Example:**++```javascript+// Let's say we want to scrape URLs extracted from sitemaps.++const sourcesFunction = async () => {+ // With super large sitemaps, this operation could take very long+ // and big websites typically have multiple sitemaps.+ const sitemaps = await downloadHugeSitemaps();+ return parseUrlsFromSitemaps(sitemaps);+};++// Sitemaps can change in real-time, so it's important to persist+// the URLs we collected. Otherwise we might lose our scraping+// state in case of an actor migration / failure / time-out.+const requestList = new RequestList({+ sourcesFunction,+ persistStateKey: 'state-key',+ persistRequestsKey: 'requests-key',+});++// The sourcesFunction is called now and the Requests are persisted.+// If something goes wrong and we need to start again, RequestList+// will load the persisted Requests from storage and will NOT+// call the sourcesFunction again, saving time and resources.+await requestList.initialize();+```++---+### `persistStateKey`**Type**: `string`@@ -46,15 +100,15 @@ If `persistStateKey` is not set, `RequestList` will always start from the beginn----### `persistSourcesKey`+### `persistRequestsKey`**Type**: `string`-Identifies the key in the default key-value store under which the `RequestList` persists its sources (i.e. the lists of URLs) during the+Identifies the key in the default key-value store under which the `RequestList` persists its Requests during the[`RequestList.initialize()`](/docs/api/request-list#initialize) call. This is necessary if `persistStateKey` is set and the source URLs mightpotentially change, to ensure consistency of the source URLs and state object. However, it comes with some storage and performance overheads.-If `persistSourcesKey` is not set, [`RequestList.initialize()`](/docs/api/request-list#initialize) will always fetch the sources from their origin,+If `persistRequestsKey` is not set, [`RequestList.initialize()`](/docs/api/request-list#initialize) will always fetch the sources from their origin,check that they are consistent with the restored state (if any) and throw an error if they are not.---diff --git a/docs/typedefs/RequestListState.md b/docs/typedefs/RequestListState.mdindex c09cdfd63..7b6e785e8 100644--- a/docs/typedefs/RequestListState.md+++ b/docs/typedefs/RequestListState.md@@ -5,8 +5,9 @@ title: RequestListState<a name="requestliststate"></a>-Represents state of a {RequestList}. It can be used to resume a {RequestList} which has been previously processed. You can obtain the state by calling-[`RequestList.getState()`](/docs/api/request-list#getstate) and receive an object with the following structure:+Represents state of a [`RequestList`](/docs/api/request-list). It can be used to resume a [`RequestList`](/docs/api/request-list) which has been+previously processed. You can obtain the state by calling [`RequestList.getState()`](/docs/api/request-list#getstate) and receive an object with the+following structure:```{diff --git a/docs/typedefs/RequestOptions.md b/docs/typedefs/RequestOptions.mdindex 3dc8dbc86..8ab74b639 100644--- a/docs/typedefs/RequestOptions.md+++ b/docs/typedefs/RequestOptions.md@@ -5,7 +5,7 @@ title: RequestOptions<a name="requestoptions"></a>-Specifies required and optional fields for constructing a [`Request`](../api/request).+Specifies required and optional fields for constructing a [`Request`](/docs/api/request).## Propertiesdiff --git a/package.json b/package.jsonindex 5e4ef75d0..8c513bb1c 100644--- a/package.json+++ b/package.json@@ -74,6 +74,7 @@"rimraf": "^3.0.2","semver": "^7.1.3","socket.io": "^2.2.0",+ "stream-json": "^1.3.3","tough-cookie": "^3.0.1","underscore": "^1.9.1","ws": "^7.2.1"diff --git a/src/actor.js b/src/actor.jsindex 4ffd650de..2ac193f37 100644--- a/src/actor.js+++ b/src/actor.js@@ -171,7 +171,7 @@ export const getEnv = () => {* <li>When running on the Apify platform (i.e. <code>APIFY_IS_AT_HOME</code> environment variable is set),* it sets up a connection to listen for platform events.* For example, to get a notification about an imminent migration to another server.- * See [](apify#apifyevents) for details.+ * See {@link Apify.events} for details.* </li>* <li>It checks that either <code>APIFY_TOKEN</code> or <code>APIFY_LOCAL_STORAGE_DIR</code> environment variable* is defined. If not, the functions sets <code>APIFY_LOCAL_STORAGE_DIR</code> to <code>./apify_storage</code>diff --git a/src/key_value_store.js b/src/key_value_store.jsindex 441e033fe..97557fbde 100644--- a/src/key_value_store.js+++ b/src/key_value_store.js@@ -182,7 +182,7 @@ export class KeyValueStore {* content type are automatically parsed and returned as a JavaScript object.* Similarly, records with `text/plain` content types are returned as a string.* For all other content types, the value is returned as a raw- * [](https://nodejs.org/api/buffer.html) instance.+ * [`Buffer`](https://nodejs.org/api/buffer.html) instance.** If the record does not exist, the function resolves to `null`.*@@ -200,7 +200,7 @@ export class KeyValueStore {* of the following characters: `a`-`z`, `A`-`Z`, `0`-`9` and `!-_.'()`* @returns {Promise<(object|Buffer|string|null)>}* Returns a promise that resolves to an object, string- * or [](https://nodejs.org/api/buffer.html), depending+ * or [`Buffer`](https://nodejs.org/api/buffer.html), depending* on the MIME content type of the record.*/getValue(key) {@@ -235,7 +235,7 @@ export class KeyValueStore {* await store.setValue('RESULTS', 'my text data', { contentType: 'text/plain' });* ```* If you set custom content type, `value` must be either a string or- * [](https://nodejs.org/api/buffer.html), otherwise an error will be thrown.+ * [`Buffer`](https://nodejs.org/api/buffer.html), otherwise an error will be thrown.** If `value` is `null`, the record is deleted instead. Note that the `setValue()` function succeeds* regardless whether the record existed or not.@@ -255,7 +255,7 @@ export class KeyValueStore {* <li>If `null`, the record in the key-value store is deleted.</li>* <li>If no `options.contentType` is specified, `value` can be any JavaScript object and it will be stringified to JSON.</li>* <li>If `options.contentType` is specified, `value` is considered raw data and it must be either a `String`- * or [](https://nodejs.org/api/buffer.html).</li>+ * or [`Buffer`](https://nodejs.org/api/buffer.html).</li>* </ul>* For any other value an error will be thrown.* @param {Object} [options]@@ -630,9 +630,11 @@ export const openKeyValueStore = (storeIdOrName, options = {}) => {** @param {string} key* Unique record key.- * @returns {Promise<(object|null)>}- * Returns a promise that resolves once the record is stored.- *+ * @returns {Promise<object|string|Buffer|null>}+ * Returns a promise that resolves to an object, string+ * or [`Buffer`](https://nodejs.org/api/buffer.html), depending+ * on the MIME content type of the record, or `null`+ * if the record is missing.* @memberof module:Apify* @name getValue* @function@@ -671,7 +673,7 @@ export const getValue = async (key) => {* <li>If `null`, the record in the key-value store is deleted.</li>* <li>If no `options.contentType` is specified, `value` can be any JavaScript object and it will be stringified to JSON.</li>* <li>If `options.contentType` is specified, `value` is considered raw data and it must be a `String`- * or [](https://nodejs.org/api/buffer.html).</li>+ * or [`Buffer`](https://nodejs.org/api/buffer.html).</li>* </ul>* For any other value an error will be thrown.* @param {Object} [options]@@ -706,8 +708,11 @@ export const setValue = async (key, value, options) => {* For more information, see {@link Apify#openKeyValueStore}* and {@link KeyValueStore#getValue}.*- * @returns {Promise<(object|null)>}- * Returns a promise that resolves once the record is stored.+ * @returns {Promise<object|string|Buffer|null>}+ * Returns a promise that resolves to an object, string+ * or [`Buffer`](https://nodejs.org/api/buffer.html), depending+ * on the MIME content type of the record, or `null`+ * if the record is missing.* @memberof module:Apify* @name getInput* @function@@ -721,9 +726,9 @@ export const getInput = async () => getValue(process.env[ENV_VARS.INPUT_KEY] ||* @param {string} key* Current {KeyValue} key being processed.* @param {number} index- * Position of the current key in {KeyValuestore}.+ * Position of the current key in {@link KeyValueStore}.* @param {object} info- * Information about the current {KeyValueStore} entry.+ * Information about the current {@link KeyValueStore} entry.* @param {number} info.size* Size of the value associated with the current key in bytes.*/diff --git a/src/puppeteer_request_interception.js b/src/puppeteer_request_interception.jsindex 75311d609..7ae9d0717 100644--- a/src/puppeteer_request_interception.js+++ b/src/puppeteer_request_interception.js@@ -113,7 +113,7 @@ const handleRequest = (request, interceptRequestHandlers) => {* ```** @param {Page} page- * Puppeteer [](https://pptr.dev/#?product=Puppeteer&show=api-class-page) object.+ * Puppeteer [`Page`](https://pptr.dev/#?product=Puppeteer&show=api-class-page) object.* @param {InterceptHandler} handler Request interception handler.* @return {Promise<void>}* @memberOf puppeteer@@ -148,7 +148,7 @@ export const addInterceptRequestHandler = async (page, handler) => {* Removes request interception handler for given page.** @param {Page} page- * Puppeteer [](https://pptr.dev/#?product=Puppeteer&show=api-class-page) object.+ * Puppeteer [`Page`](https://pptr.dev/#?product=Puppeteer&show=api-class-page) object.* @param {InterceptHandler} handler Request interception handler.* @return {Promise<void>}* @memberOf puppeteerdiff --git a/src/puppeteer_utils.js b/src/puppeteer_utils.jsindex f25d62be1..7f240afe8 100644--- a/src/puppeteer_utils.js+++ b/src/puppeteer_utils.js@@ -27,7 +27,7 @@ const DEFAULT_BLOCK_REQUEST_URL_PATTERNS = ['.css', '.jpg', '.jpeg', '.png', '.s* The function should be called on a newly-created page object before navigating to the target crawled page.** @param {Page} page- * Puppeteer [](https://pptr.dev/#?product=Puppeteer&show=api-class-page) object.+ * Puppeteer [`Page`](https://pptr.dev/#?product=Puppeteer&show=api-class-page) object.* @return {Promise<void>}* @memberOf puppeteer* @ignore@@ -78,7 +78,7 @@ const injectedFilesCache = new LruCache({ maxLength: MAX_INJECT_FILE_CACHE_SIZE* File contents are cached for up to 10 files to limit file system access.** @param {Page} page- * Puppeteer [](https://pptr.dev/#?product=Puppeteer&show=api-class-page) object.+ * Puppeteer [`Page`](https://pptr.dev/#?product=Puppeteer&show=api-class-page) object.* @param {string} filePath File path* @param {Object} [options]* @param {boolean} [options.surviveNavigations]@@ -105,7 +105,7 @@ const injectFile = async (page, filePath, options = {}) => {};/**- * Injects the [](https://jquery.com/) library into a Puppeteer page.+ * Injects the [jQuery](https://jquery.com/) library into a Puppeteer page.* jQuery is often useful for various web scraping and crawling tasks.* For example, it can help extract text from HTML elements using CSS selectors.*@@ -124,11 +124,11 @@ const injectFile = async (page, filePath, options = {}) => {* ```** Note that `injectJQuery()` does not affect the Puppeteer's- * [](https://pptr.dev/#?product=Puppeteer&show=api-pageselector)+ * [`page.$()`](https://pptr.dev/#?product=Puppeteer&show=api-pageselector)* function in any way.** @param {Page} page- * Puppeteer [](https://pptr.dev/#?product=Puppeteer&show=api-class-page) object.+ * Puppeteer [`Page`](https://pptr.dev/#?product=Puppeteer&show=api-class-page) object.* @return {Promise<*>}* @memberOf puppeteer*/@@ -138,7 +138,7 @@ const injectJQuery = (page) => {};/**- * Injects the [](https://underscorejs.org/) library into a Puppeteer page.+ * Injects the [Underscore](https://underscorejs.org/) library into a Puppeteer page.** Beware that the injected Underscore object will be set to the `window._` variable and thus it might cause conflicts with* libraries included by the page that use the same variable name.@@ -230,7 +230,7 @@ const enqueueRequestsFromClickableElements = async (page, selector, purls, reque* ```** @param {Page} page- * Puppeteer [](https://pptr.dev/#?product=Puppeteer&show=api-class-page) object.+ * Puppeteer [`Page`](https://pptr.dev/#?product=Puppeteer&show=api-class-page) object.* @param {Object} [options]* @param {string[]} [options.urlPatterns]* The patterns of URLs to block from being loaded by the browser.@@ -282,7 +282,7 @@ const blockResources = async (page, resourceTypes = ['stylesheet', 'font', 'imag* *IMPORTANT*: Caching responses stores them to memory, so too loose rules could cause memory leaks for longer running crawlers.* This issue should be resolved or atleast mitigated in future iterations of this feature.* @param {Page} page- * Puppeteer [](https://pptr.dev/#?product=Puppeteer&show=api-class-page) object.+ * Puppeteer [`Page`](https://pptr.dev/#?product=Puppeteer&show=api-class-page) object.* @param {Object} cache* Object in which responses are stored* @param {Array<(string|RegExp)>} responseUrlRules@@ -349,7 +349,7 @@ const cacheResponses = async (page, cache, responseUrlRules) => {* request: Request,* }* ```- * Where `page` is a Puppeteer [](https://pptr.dev/#?product=Puppeteer&show=api-class-page)+ * Where `page` is a Puppeteer [`Page`](https://pptr.dev/#?product=Puppeteer&show=api-class-page)* and `request` is a {@link Request}.** The function is compiled by using the `scriptString` parameter as the function's body,@@ -396,7 +396,7 @@ const compileScript = (scriptString, context = Object.create(null)) => {* browser cache which degrades performance.** @param {Page} page- * Puppeteer [](https://pptr.dev/#?product=Puppeteer&show=api-class-page) object.+ * Puppeteer [`Page`](https://pptr.dev/#?product=Puppeteer&show=api-class-page) object.* @param {Request} request* @param {DirectNavigationOptions} gotoOptions Custom options for `page.goto()`.* @return {Promise<(Response | null)>}@@ -441,7 +441,7 @@ export const gotoExtended = async (page, request, gotoOptions = {}) => {* Scrolls to the bottom of a page, or until it times out.* Loads dynamic content when it hits the bottom of a page, and then continues scrolling.* @param {Page} page- * Puppeteer [](https://pptr.dev/#?product=Puppeteer&show=api-class-page) object.+ * Puppeteer [`Page`](https://pptr.dev/#?product=Puppeteer&show=api-class-page) object.* @param {Object} [options]* @param {number} [options.timeoutSecs=0]* How many seconds to scroll for. If 0, will scroll until bottom of page.@@ -510,7 +510,7 @@ export const infiniteScroll = async (page, options = {}) => {/*** Saves a full screenshot and HTML of the current page into a Key-Value store.* @param {Page} page- * Puppeteer [](https://pptr.dev/#?product=Puppeteer&show=api-class-page) object.+ * Puppeteer [`Page`](https://pptr.dev/#?product=Puppeteer&show=api-class-page) object.* @param {Object} [options]* @param {string} [options.key=SNAPSHOT]* Key under which the screenshot and HTML will be saved. `.jpg` will be appended for screenshot and `.html` for HTML.diff --git a/src/request.js b/src/request.jsindex 9ca54dadc..7c87ecb47 100644--- a/src/request.js+++ b/src/request.js@@ -223,7 +223,7 @@ class Request {export default Request;/**- * Specifies required and optional fields for constructing a [`Request`](../api/request).+ * Specifies required and optional fields for constructing a {@link Request}.** @typedef RequestOptions* @property {string} url URL of the web page to crawl. It must be a non-empty string.diff --git a/src/request_list.js b/src/request_list.jsindex 60c50a86e..d7e51c438 100644--- a/src/request_list.js+++ b/src/request_list.js@@ -1,31 +1,34 @@import { checkParamOrThrow } from 'apify-client/build/utils';import * as _ from 'underscore';import { ACTOR_EVENT_NAMES_EX } from './constants';-import Request, { RequestOptions } from './request'; // eslint-disable-line no-unused-vars,import/named+import Request from './request'; // eslint-disable-line import/no-duplicatesimport log from './utils_log';import events from './events';import { getFirstKey, publicUtils } from './utils';import { getValue, setValue } from './key_value_store';+import { serializeArray, createDeserialize } from './serialization';++// TYPE IMPORTS+/* eslint-disable no-unused-vars,import/named,import/no-duplicates,import/order */+import { RequestOptions } from './request';+/* eslint-enable no-unused-vars,import/named,import/no-duplicates,import/order */export const STATE_PERSISTENCE_KEY = 'REQUEST_LIST_STATE';-export const SOURCES_PERSISTENCE_KEY = 'REQUEST_LIST_SOURCES';+export const REQUESTS_PERSISTENCE_KEY = 'REQUEST_LIST_REQUESTS';-/**- * @typedef RequestListInput- * @property {string} [method]- * @property {string} [requestsFromUrl]- * @property {RegExp} [regex]- */+const CONTENT_TYPE_BINARY = 'application/octet-stream';-/**- * @typedef {Array<(RequestListInput|RequestOptions|Request)>} SourceInput- *//*** @typedef RequestListOptions- * @property {SourceInput} sources- * An array of sources of URLs for the `RequestList`. It can be either an array of plain objects that- * define the `url` property, or an array of instances of the {@link Request} class.+ * @property {Array<RequestOptions|Request|string>} [sources]+ * An array of sources of URLs for the {@link RequestList}. It can be either an array of strings,+ * plain objects that define at least the `url` property, or an array of {@link Request} instances.+ *+ * **IMPORTANT:** The `sources` array will be consumed (left empty) after `RequestList` initializes.+ * This is a measure to prevent memory leaks in situations when millions of sources are+ * added.+ ** Additionally, the `requestsFromUrl` property may be used instead of `url`,* which will instruct `RequestList` to download the source URLs from a given remote location.* The URLs will be parsed from the received response.@@ -33,7 +36,10 @@ export const SOURCES_PERSISTENCE_KEY = 'REQUEST_LIST_SOURCES';* ```* [* // A single URL- * { method: 'GET', url: 'http://example.com/a/b' },+ * 'http://example.com/a/b',+ *+ * // Modify Request options+ * { method: PUT, 'https://example.com/put, payload: { foo: 'bar' }}** // Batch import of URLs from a file hosted on the web,* // where the URLs should be requested using the HTTP POST request@@ -47,6 +53,45 @@ export const SOURCES_PERSISTENCE_KEY = 'REQUEST_LIST_SOURCES';* { requestsFromUrl: 'https://docs.google.com/spreadsheets/d/1GA5sSQhQjB_REes8I5IKg31S-TuRcznWOPjcpNqtxmU/gviz/tq?tqx=out:csv' }* ]* ```+ * @property {Function} [sourcesFunction]+ * A function that will be called to get the sources for the `RequestList`, but only if `RequestList`+ * was not able to fetch their persisted version (see {@link RequestListOptions.persistRequestsKey}).+ * It must return an `Array` of {@link Request} or {@link RequestOptions}.+ *+ * This is very useful in a scenario when getting the sources is a resource intensive or time consuming+ * task, such as fetching URLs from multiple sitemaps or parsing URLs from large datasets. Using the+ * `sourcesFunction` in combination with `persistStateKey` and `persistRequestsKey` will allow you to+ * fetch and parse those URLs only once, saving valuable time when your actor migrates or restarts.+ *+ * If both {@link RequestListOptions.sources} and {@link RequestListOptions.sourcesFunction} are provided,+ * the sources returned by the function will be added after the `sources`.+ *+ * **Example:**+ * ```javascript+ * // Let's say we want to scrape URLs extracted from sitemaps.+ *+ * const sourcesFunction = async () => {+ * // With super large sitemaps, this operation could take very long+ * // and big websites typically have multiple sitemaps.+ * const sitemaps = await downloadHugeSitemaps();+ * return parseUrlsFromSitemaps(sitemaps);+ * }+ *+ * // Sitemaps can change in real-time, so it's important to persist+ * // the URLs we collected. Otherwise we might lose our scraping+ * // state in case of an actor migration / failure / time-out.+ * const requestList = new RequestList({+ * sourcesFunction,+ * persistStateKey: 'state-key',+ * persistRequestsKey: 'requests-key',+ * })+ *+ * // The sourcesFunction is called now and the Requests are persisted.+ * // If something goes wrong and we need to start again, RequestList+ * // will load the persisted Requests from storage and will NOT+ * // call the sourcesFunction again, saving time and resources.+ * await requestList.initialize();+ * ```* @property {string} [persistStateKey]* Identifies the key in the default key-value store under which `RequestList` periodically stores its* state (i.e. which URLs were crawled and which not).@@ -55,13 +100,14 @@ export const SOURCES_PERSISTENCE_KEY = 'REQUEST_LIST_SOURCES';** If `persistStateKey` is not set, `RequestList` will always start from the beginning,* and all the source URLs will be crawled again.- * @property {string} [persistSourcesKey]+ * @property {string} [persistRequestsKey]* Identifies the key in the default key-value store under which the `RequestList` persists its- * sources (i.e. the lists of URLs) during the {@link RequestList#initialize} call.+ * Requests during the {@link RequestList#initialize} call.* This is necessary if `persistStateKey` is set and the source URLs might potentially change,- * to ensure consistency of the source URLs and state object. However, it comes with some storage and performance overheads.+ * to ensure consistency of the source URLs and state object. However, it comes with some+ * storage and performance overheads.*- * If `persistSourcesKey` is not set, {@link RequestList#initialize} will always fetch the sources+ * If `persistRequestsKey` is not set, {@link RequestList#initialize} will always fetch the sources* from their origin, check that they are consistent with the restored state (if any)* and throw an error if they are not.* @property {RequestListState} [state]@@ -95,6 +141,7 @@ export const SOURCES_PERSISTENCE_KEY = 'REQUEST_LIST_SOURCES';* `uniqueKey`s however. It is the user's responsibility to ensure uniqueness of their unique keys* if they wish to keep more than just a single copy in the `RequestList`.*/+/*** Represents a static list of URLs to crawl.* The URLs can be provided either in code or parsed from a text file hosted on the web.@@ -121,17 +168,28 @@ export const SOURCES_PERSISTENCE_KEY = 'REQUEST_LIST_SOURCES';** The internal state is closely tied to the provided sources (URLs). If the sources change on actor restart, the state will become corrupted and* `RequestList` will raise an exception. This typically happens when the sources is a list of URLs downloaded from the web.- * In such case, use the `persistSourcesKey` option in conjunction with `persistStateKey`,+ * In such case, use the `persistRequestsKey` option in conjunction with `persistStateKey`,* to make the `RequestList` store the initial sources to the default key-value store and load them after restart,* which will prevent any issues that a live list of URLs might cause.*- * **Example usage:**+ * **Basic usage:**+ * ```javascript+ * // Use a helper function to simplify request list initialization.+ * // State and sources are automatically persisted.+ * const requestList = await Apify.openRequestList('my-request-list', [+ * 'http://www.example.com/page-1',+ * { url: 'http://www.example.com/page-2', method: 'POST', userData: { foo: 'bar' }},+ * { requestsFromUrl: 'http://www.example.com/my-url-list.txt', userData: { isFromUrl: true } },+ * ]);+ * ```*+ * **Advanced usage:*** ```javascript+ * // Use the constructor to get more control over the initialization.* const requestList = new Apify.RequestList({* sources: [* // Separate requests- * { url: 'http://www.example.com/page-1', method: 'GET', headers: {} },+ * { url: 'http://www.example.com/page-1', method: 'GET', headers: { ... } },* { url: 'http://www.example.com/page-2', userData: { foo: 'bar' }},** // Bulk load of URLs from file `http://www.example.com/my-url-list.txt`@@ -139,25 +197,11 @@ export const SOURCES_PERSISTENCE_KEY = 'REQUEST_LIST_SOURCES';* { requestsFromUrl: 'http://www.example.com/my-url-list.txt', userData: { isFromUrl: true } },* ],*- * // Ensure both the sources and crawling state of the request list is persisted,- * // so that on actor restart, the crawling will continue where it left off+ * // Persist only state in cases where the original sources are immutable to improve performance.* persistStateKey: 'my-state',- * persistSourcesKey: 'my-sources'* });*- * // This call loads and parses the URLs from the remote file.* await requestList.initialize();- *- * // Get requests from list- * const request1 = await requestList.fetchNextRequest();- * const request2 = await requestList.fetchNextRequest();- * const request3 = await requestList.fetchNextRequest();- *- * // Mark some of them as handled- * await requestList.markRequestHandled(request1);- *- * // If processing fails then reclaim it back to the list- * await requestList.reclaimRequest(request2);* ```*/export class RequestList {@@ -167,14 +211,33 @@ export class RequestList {constructor(options = {}) {checkParamOrThrow(options, 'options', 'Object');- const { sources, persistStateKey, persistSourcesKey, state, keepDuplicateUrls = false } = options;+ const {+ sources,+ sourcesFunction,+ persistStateKey,+ persistRequestsKey,+ persistSourcesKey,+ state,+ keepDuplicateUrls = false,+ } = options;++ // TODO Deprecated 02/2020+ if (persistSourcesKey) {+ log.deprecated('RequestList: options.persistSourcesKey is deprecated. Use options.persistRequestsKey.');+ }- checkParamOrThrow(sources, 'options.sources', 'Array');+ checkParamOrThrow(sources, 'options.sources', 'Maybe Array');+ checkParamOrThrow(sourcesFunction, 'options.sourcesFunction', 'Maybe Function');checkParamOrThrow(state, 'options.state', 'Maybe Object');checkParamOrThrow(persistStateKey, 'options.persistStateKey', 'Maybe String');+ checkParamOrThrow(persistRequestsKey, 'options.persistRequestsKey', 'Maybe String');checkParamOrThrow(persistSourcesKey, 'options.persistSourcesKey', 'Maybe String');checkParamOrThrow(keepDuplicateUrls, 'options.keepDuplicateUrls', 'Maybe Boolean');+ if (!(sources || sourcesFunction)) {+ throw new Error('RequestList: At least one of "sources" or "sourcesFunction" must be provided.');+ }+// Array of all requests from all sources, in the order as they appeared in sources.// All requests in the array have distinct uniqueKey!this.requests = [];@@ -196,7 +259,7 @@ export class RequestList {this.reclaimed = {};this.persistStateKey = persistStateKey;- this.persistSourcesKey = persistSourcesKey;+ this.persistRequestsKey = persistRequestsKey || persistSourcesKey;this.initialState = state;@@ -206,10 +269,12 @@ export class RequestList {// Starts as true because until we handle the first request, the list is effectively persisted by doing nothing.this.isStatePersisted = true;// Starts as false because we don't know yet and sources might change in the meantime (eg. download from live list).- this.areSourcesPersisted = false;+ this.areRequestsPersisted = false;this.isLoading = false;this.isInitialized = false;- this.sources = sources;+ // Will be empty after initialization to save memory.+ this.sources = sources || [];+ this.sourcesFunction = sourcesFunction;}/**@@ -224,33 +289,85 @@ export class RequestList {}this.isLoading = true;- const [state, sources] = await this._loadStateAndSources();+ const [state, persistedRequests] = await this._loadStateAndPersistedRequests();++ // Add persisted requests / new sources in a memory efficient way because with very+ // large lists, we were running out of memory.+ if (persistedRequests) {+ await this._addPersistedRequests(persistedRequests);+ } else {+ await this._addRequestsFromSources();+ }++ this._restoreState(state);+ this.isInitialized = true;+ if (this.persistRequestsKey && !this.areRequestsPersisted) await this._persistRequests();+ if (this.persistStateKey) {+ events.on(ACTOR_EVENT_NAMES_EX.PERSIST_STATE, this.persistState.bind(this));+ }+ }++ /**+ * Adds previously persisted Requests, as retrieved from the key-value store.+ * This needs to be done in a memory efficient way. We should update the input+ * to a Stream once apify-client supports streams.+ * @param {Buffer} persistedRequests+ * @ignore+ */+ async _addPersistedRequests(persistedRequests) {+ // We don't need the sources so we purge them to+ // prevent them from hanging in memory.+ for (let i = 0; i < this.sources.length; i++) {+ delete this.sources[i];+ }+ this.sources = [];- // If there are no sources, it just means that we've not persisted any (yet).- if (sources) this.areSourcesPersisted = true;- const actualSources = sources || this.sources;+ this.areRequestsPersisted = true;+ const requestStream = createDeserialize(persistedRequests);+ for await (const request of requestStream) {+ this._addRequest(request);+ }+ }+ /**+ * Add Requests from both options.sources and options.sourcesFunction.+ * This function is called only when persisted sources were not loaded.+ * We need to avoid keeping both sources and requests in memory+ * to reduce memory footprint with very large sources.+ * @returns {Promise<void>}+ * @ignore+ */+ async _addRequestsFromSources() {// We'll load all sources in sequence to ensure that they get loaded in the right order.- const sourcesWithRequestsFromUrl = [];- for (const source of actualSources) {+ const sourcesCount = this.sources.length;+ for (let i = 0; i < sourcesCount; i++) {+ const source = this.sources[i];+ // Using delete here to drop the original object ASAP to free memory+ // .pop would reverse the array and .shift is SLOW.+ delete this.sources[i];+if (source.requestsFromUrl) {const fetchedRequests = await this._fetchRequestsFromUrl(source);- sourcesWithRequestsFromUrl.push(...fetchedRequests);await this._addFetchedRequests(source, fetchedRequests);} else {- sourcesWithRequestsFromUrl.push(source);this._addRequest(source);}}- // Replace source with source with request from remote URLs- this.sources = sourcesWithRequestsFromUrl;-- this._restoreState(state);- this.isInitialized = true;- if (this.persistSourcesKey && !this.areSourcesPersisted) await this._persistSources();- if (this.persistStateKey) {- events.on(ACTOR_EVENT_NAMES_EX.PERSIST_STATE, this.persistState.bind(this));+ // Drop the original array full of empty indexes.+ this.sources = [];++ if (this.sourcesFunction) {+ try {+ const sourcesFromFunction = await this.sourcesFunction();+ const sourcesFromFunctionCount = sourcesFromFunction.length;+ for (let i = 0; i < sourcesFromFunctionCount; i++) {+ const source = sourcesFromFunction.shift();+ this._addRequest(source);+ }+ } catch (err) {+ throw new Error(`RequestList: Loading requests with sourcesFunction failed. Cause:\n${err.stack}`);+ }}}@@ -278,15 +395,16 @@ export class RequestList {/*** Unlike persistState(), this is used only internally, since the sources- * are automatically persisted at RequestList initialization (if the persistSourcesKey is set),+ * are automatically persisted at RequestList initialization (if the persistRequestsKey is set),* but there's no reason to persist it again afterwards, because RequestList is immutable.** @return {Promise<void>}* @ignore*/- async _persistSources() {- await setValue(this.persistSourcesKey, this.sources);- this.areSourcesPersisted = true;+ async _persistRequests() {+ const serializedRequests = await serializeArray(this.requests);+ await setValue(this.persistRequestsKey, serializedRequests, { contentType: CONTENT_TYPE_BINARY });+ this.areRequestsPersisted = true;}/**@@ -349,26 +467,28 @@ export class RequestList {}/**- * Attempts to load state and sources using the `RequestList` configuration- * and returns a tuple of [state, sources] where each may be null if not loaded.+ * Attempts to load state and requests using the `RequestList` configuration+ * and returns a tuple of [state, requests] where each may be null if not loaded.** @return {Promise<Array<(RequestListState|null)>>}* @ignore*/- async _loadStateAndSources() {+ async _loadStateAndPersistedRequests() {let state;+ let persistedRequests;if (this.initialState) {- log.debug('RequestList: Loading previous state from options.state argument.');state = this.initialState;+ log.debug('RequestList: Loaded state from options.state argument.');} else if (this.persistStateKey) {- log.debug('RequestList: Loading previous state from key value store using the persistStateKey.');state = getValue(this.persistStateKey);+ if (state) log.debug('RequestList: Loaded state from key value store using the persistStateKey.');}- if (this.persistSourcesKey) {- log.debug('RequestList: Loading sources from key value store using the persistSourcesKey.');- return Promise.all([state, getValue(this.persistSourcesKey)]);+ if (this.persistRequestsKey) {+ persistedRequests = await getValue(this.persistRequestsKey);+ if (persistedRequests) log.debug('RequestList: Loaded requests from key value store using the persistRequestsKey.');}- return [await state, null];+ // Unwraps "state" promise if needed, otherwise no-op.+ return Promise.all([state, persistedRequests]);}/**@@ -494,7 +614,7 @@ export class RequestList {const fetchedCount = fetchedRequests.length;const importedCount = this.requests.length - originalLength;- log.info('RequestList: list fetched.', {+ log.info('RequestList: Fetched and loaded Requests from a remote resource.', {requestsFromUrl,regex,fetchedCount,@@ -534,17 +654,26 @@ export class RequestList {/*** Adds given request.- * If the `opts` parameter is a plain object and not an instance of a `Request`, then the function- * creates a `Request` instance.+ * If the `source` parameter is a string or plain object and not an instance+ * of a `Request`, then the function creates a `Request` instance.*+ * @param {string|Request|object} source* @ignore*/- _addRequest(opts) {- const hasUniqueKey = !!opts.uniqueKey;+ _addRequest(source) {+ let request;+ const type = typeof source;+ if (type === 'string') {+ request = new Request({ url: source });+ } else if (source instanceof Request) {+ request = source;+ } else if (source && type === 'object') {+ request = new Request(source);+ } else {+ throw new Error(`RequestList: Cannot create Request from type: ${type}`);+ }- const request = opts instanceof Request- ? opts- : new Request(opts);+ const hasUniqueKey = !!source.uniqueKey;// Add index to uniqueKey if duplicates are to be keptif (this.keepDuplicateUrls && !hasUniqueKey) {@@ -657,9 +786,13 @@ export class RequestList {** If `null`, the list will not be persisted and will only be stored in memory. Process restart* will then cause the list to be crawled again from the beginning. We suggest always using a name.- * @param {(SourceInput|string[])} sources- * An array of sources of URLs for the {@link RequestList}. It can be either an array of plain objects- * that define at least the `url` property, or an array of instances of the {@link Request} class.+ * @param {Array<RequestOptions|Request|string>} sources+ * An array of sources of URLs for the {@link RequestList}. It can be either an array of strings,+ * plain objects that define at least the `url` property, or an array of {@link Request} instances.+ *+ * **IMPORTANT:** The `sources` array will be consumed (left empty) after {@link RequestList} initializes.+ * This is a measure to prevent memory leaks in situations when millions of sources are+ * added.** Additionally, the `requestsFromUrl` property may be used instead of `url`,* which will instruct {@link RequestList} to download the source URLs from a given remote location.@@ -669,7 +802,7 @@ export class RequestList {* For details, see the {@link RequestListOptions.sources}* @param {RequestListOptions} [options]* The {@link RequestList} options. Note that the `listName` parameter supersedes- * the {@link RequestListOptions.persistStateKey} and {@link RequestListOptions.persistSourcesKey}+ * the {@link RequestListOptions.persistStateKey} and {@link RequestListOptions.persistRequestsKey}* options and the `sources` parameter supersedes the {@link RequestListOptions.sources} option.* @returns {Promise<RequestList>}* @memberof module:Apify@@ -678,17 +811,13 @@ export class RequestList {*/export const openRequestList = async (listName, sources, options = {}) => {checkParamOrThrow(listName, 'listName', 'String | Null');- checkParamOrThrow(sources, 'sources', '[Object | String]');- if (!sources.length) throw new Error('Parameter sources must not be an empty array.');+ checkParamOrThrow(sources, 'sources', 'Array');checkParamOrThrow(options, 'options', 'Object');- // Support both an array of strings and array of objects.- if (typeof sources[0] === 'string') sources = sources.map(url => ({ url }));-const rl = new RequestList({...options,persistStateKey: listName ? `${listName}-${STATE_PERSISTENCE_KEY}` : null,- persistSourcesKey: listName ? `${listName}-${SOURCES_PERSISTENCE_KEY}` : null,+ persistRequestsKey: listName ? `${listName}-${REQUESTS_PERSISTENCE_KEY}` : null,sources,});await rl.initialize();@@ -696,7 +825,7 @@ export const openRequestList = async (listName, sources, options = {}) => {};/**- * Represents state of a {RequestList}. It can be used to resume a {RequestList} which has been previously processed.+ * Represents state of a {@link RequestList}. It can be used to resume a {@link RequestList} which has been previously processed.* You can obtain the state by calling {@link RequestList#getState} and receive an object with* the following structure:*diff --git a/src/serialization.js b/src/serialization.jsnew file mode 100644index 000000000..e556a26c8--- /dev/null+++ b/src/serialization.js@@ -0,0 +1,152 @@+import { checkParamOrThrow } from 'apify-client/build/utils';+import * as stream from 'stream'; // eslint-disable-line import/no-duplicates+import * as StreamArray from 'stream-json/streamers/StreamArray';+import * as util from 'util';+import * as zlib from 'zlib';++// TYPE IMPORTS+/* eslint-disable no-unused-vars,import/named,import/no-duplicates,import/order */+import { Readable } from 'stream';+// eslint-enable-line import/no-duplicates++const pipeline = util.promisify(stream.pipeline);++/**+ * Transforms an array of items to a JSON in a streaming+ * fashion to save memory. It operates in batches to speed+ * up the process.+ * @ignore+ */+class ArrayToJson extends stream.Readable {+ constructor(data, options = {}) {+ super(options);+ const {+ batchSize = 10000,+ } = options;+ this.offset = 0;+ this.batchSize = batchSize;+ this.data = data;+ this.push('[');+ }++ _read() {+ try {+ const items = this.data.slice(this.offset, this.offset + this.batchSize);+ if (items.length) {+ const json = JSON.stringify(items);+ // Strip brackets to flatten the batch.+ const itemString = json.substring(1, json.length - 1);+ if (this.offset > 0) this.push(',', 'utf8');+ this.push(itemString, 'utf8');+ this.offset += this.batchSize;+ } else {+ this.push(']');+ this.push(null);+ this.destroy();+ }+ } catch (err) {+ this.destroy(err);+ }+ }+}++/**+ * Uses Gzip compression to take an array of values, which can be anything+ * from entries in a Dataset to Requests in a RequestList and compresses+ * them to a Buffer in a memory-efficient way (streaming one by one). Ideally,+ * the largest chunk of memory consumed will be the final compressed Buffer.+ * This could be further improved by outputting a Stream, if and when+ * apify-client supports streams.+ *+ * @param {Array} data+ * @returns {Promise<Buffer>}+ * @ignore+ */+export const serializeArray = async (data) => {+ checkParamOrThrow(data, 'data', 'Array');+ const { chunks, collector } = createChunkCollector();+ await pipeline(+ new ArrayToJson(data),+ zlib.createGzip(),+ collector,+ );+ return Buffer.concat(chunks);+};++/**+ * Decompresses a Buffer previously created with compressData (technically,+ * any JSON that is an Array) and collects it into an Array of values+ * in a memory-efficient way (streaming the array items one by one instead+ * of creating a fully decompressed buffer -> full JSON -> full Array all+ * in memory at once. Could be further optimized to ingest a Stream if and+ * when apify-client supports streams.+ *+ * @param {Buffer} compressedData+ * @returns {Promise<Array>}+ * @ignore+ */+export const deserializeArray = async (compressedData) => {+ checkParamOrThrow(compressedData, 'compressedData', 'Buffer');+ const { chunks, collector } = createChunkCollector({ fromValuesStream: true });+ await pipeline(+ stream.Readable.from([compressedData]),+ zlib.createGunzip(),+ StreamArray.withParser(),+ collector,+ );+ return chunks;+};++/**+ * Creates a stream that decompresses a Buffer previously created with+ * compressData (technically, any JSON that is an Array) and collects it+ * into an Array of values in a memory-efficient way (streaming the array+ * items one by one instead of creating a fully decompressed buffer+ * -> full JSON -> full Array all in memory at once. Could be further+ * optimized to ingest a Stream if and when apify-client supports streams.+ * @param compressedData+ * @returns {Readable}+ * @ignore+ */+export const createDeserialize = (compressedData) => {+ checkParamOrThrow(compressedData, 'compressedData', 'Buffer');+ const streamArray = StreamArray.withParser();+ const destination = pluckValue(streamArray);+ stream.pipeline(+ stream.Readable.from([compressedData]),+ zlib.createGunzip(),+ destination,+ err => destination.emit(err),+ );+ return destination;+};++/**+ * @return {{chunks: Array<string|Buffer>, collector: module:stream.internal.Writable}}+ */+function createChunkCollector(options = {}) {+ const { fromValuesStream = false } = options;+ const chunks = [];+ const collector = new stream.Writable({ // eslint-disable-line no-shadow+ decodeStrings: false,+ objectMode: fromValuesStream,+ write(chunk, nil, callback) {+ chunks.push(fromValuesStream ? chunk.value : chunk);+ callback();+ },+ writev(chunkObjects, callback) {+ const buffers = chunkObjects.map(({ chunk }) => {+ return fromValuesStream ? chunk.value : chunk;+ });+ chunkObjects.push(...buffers);+ callback();+ },+ });+ return { collector, chunks };+}++function pluckValue(streamArray) {+ const realPush = streamArray.push.bind(streamArray);+ streamArray.push = obj => realPush(obj && obj.value);+ return streamArray;+}diff --git a/test/.eslintrc.json b/test/.eslintrc.jsonindex 311ecf443..1e6583889 100644--- a/test/.eslintrc.json+++ b/test/.eslintrc.json@@ -10,6 +10,7 @@"beforeEach": false,"afterEach": false,"expect": false,- "test": false+ "test": false,+ "jest": false}}diff --git a/test/autoscaling/snapshotter.test.js b/test/autoscaling/snapshotter.test.jsindex 30dcff499..ca0606987 100644--- a/test/autoscaling/snapshotter.test.js+++ b/test/autoscaling/snapshotter.test.js@@ -42,7 +42,7 @@ describe('Snapshotter', () => {const clientSnapshots = snapshotter.getClientSample();expect(Array.isArray(cpuSnapshots)).toBe(true);- expect(cpuSnapshots).toHaveLength(2);+ expect(cpuSnapshots.length).toBeGreaterThanOrEqual(1);cpuSnapshots.forEach((ss) => {expect(ss.createdAt).toBeInstanceOf(Date);expect(typeof ss.isOverloaded).toBe('boolean');@@ -50,7 +50,7 @@ describe('Snapshotter', () => {});expect(Array.isArray(memorySnapshots)).toBe(true);- expect(memorySnapshots).toHaveLength(2);+ expect(memorySnapshots.length).toBeGreaterThanOrEqual(1);memorySnapshots.forEach((ss) => {expect(ss.createdAt).toBeInstanceOf(Date);expect(typeof ss.isOverloaded).toBe('boolean');@@ -58,7 +58,7 @@ describe('Snapshotter', () => {});expect(Array.isArray(eventLoopSnapshots)).toBe(true);- expect(eventLoopSnapshots).toHaveLength(3);+ expect(eventLoopSnapshots.length).toBeGreaterThanOrEqual(2);eventLoopSnapshots.forEach((ss) => {expect(ss.createdAt).toBeInstanceOf(Date);expect(typeof ss.isOverloaded).toBe('boolean');@@ -66,7 +66,7 @@ describe('Snapshotter', () => {});expect(Array.isArray(clientSnapshots)).toBe(true);- expect(clientSnapshots).toHaveLength(2);+ expect(clientSnapshots.length).toBeGreaterThanOrEqual(1);clientSnapshots.forEach((ss) => {expect(ss.createdAt).toBeInstanceOf(Date);expect(typeof ss.isOverloaded).toBe('boolean');diff --git a/test/crawlers/basic_crawler.test.js b/test/crawlers/basic_crawler.test.jsindex e48cfd91e..5b3034c07 100644--- a/test/crawlers/basic_crawler.test.js+++ b/test/crawlers/basic_crawler.test.js@@ -33,6 +33,7 @@ describe('BasicCrawler', () => {test('should run in parallel thru all the requests', async () => {const sources = _.range(0, 500).map(index => ({ url: `https://example.com/${index}` }));+ const sourcesCopy = JSON.parse(JSON.stringify(sources));const processed = [];const requestList = new Apify.RequestList({ sources });@@ -52,7 +53,7 @@ describe('BasicCrawler', () => {await basicCrawler.run();expect(basicCrawler.autoscaledPool.minConcurrency).toBe(25);- expect(processed).toEqual(sources);+ expect(processed).toEqual(sourcesCopy);expect(await requestList.isFinished()).toBe(true);expect(await requestList.isEmpty()).toBe(true);});@@ -541,6 +542,7 @@ describe('BasicCrawler', () => {sinon.restore();const sources = _.range(1, 10).map(i => ({ url: `http://example.com/${i}` }));+ const sourcesCopy = JSON.parse(JSON.stringify(sources));let requestList = new Apify.RequestList({ sources });await requestList.initialize();stub = sinon@@ -563,7 +565,7 @@ describe('BasicCrawler', () => {expect(count).toBe(7);sinon.restore();- requestList = new Apify.RequestList({ sources });+ requestList = new Apify.RequestList({ sources: sourcesCopy });await requestList.initialize();const listStub = sinon.stub(requestList, 'handledCount')diff --git a/test/crawlers/cheerio_crawler.test.js b/test/crawlers/cheerio_crawler.test.jsindex 8aa80a339..d66df4589 100644--- a/test/crawlers/cheerio_crawler.test.js+++ b/test/crawlers/cheerio_crawler.test.js@@ -157,6 +157,7 @@ describe('CheerioCrawler', () => {{ url: 'http://example.com/?q=5' },{ url: 'http://example.com/?q=6' },];+ const sourcesCopy = JSON.parse(JSON.stringify(sources));const processed = [];const failed = [];const requestList = new Apify.RequestList({ sources });@@ -183,7 +184,7 @@ describe('CheerioCrawler', () => {processed.sort(comparator);processed.forEach((request, id) => {- expect(request.url).toEqual(sources[id].url);+ expect(request.url).toEqual(sourcesCopy[id].url);expect(request.userData.title).toBe('Example Domain');expect(typeof request.userData.html).toBe('string');expect(request.userData.html.length).not.toBe(0);@@ -506,7 +507,7 @@ describe('CheerioCrawler', () => {expect(handledRequests).toEqual(contentTypes.length);});- describe('should works with all content types from options.additionalMimeTypes', () => {+ describe('should work with all content types from options.additionalMimeTypes', () => {const handlePageInvocationParams = [];let handleFailedInvocationCount = 0;beforeAll(async () => {@@ -515,6 +516,7 @@ describe('CheerioCrawler', () => {{ url: `http://${HOST}:${port}/xml-type` },{ url: `http://${HOST}:${port}/image-type` },];+ const sourceCount = sources.length;const requestList = new Apify.RequestList({ sources });await requestList.initialize();const crawler = new Apify.CheerioCrawler({@@ -531,7 +533,7 @@ describe('CheerioCrawler', () => {await crawler.run();expect(handleFailedInvocationCount).toBe(0);- expect(handlePageInvocationParams.length).toEqual(sources.length);+ expect(handlePageInvocationParams.length).toEqual(sourceCount);});test('when response is application/json', async () => {const jsonRequestParams = handlePageInvocationParams[0];@@ -713,7 +715,7 @@ describe('CheerioCrawler', () => {beforeEach(async () => {await localStorageEmulator.clean();- requestList = await Apify.openRequestList('test', sources);+ requestList = await Apify.openRequestList('test', sources.slice());});test('should work', async () => {diff --git a/test/crawlers/puppeteer_crawler.test.js b/test/crawlers/puppeteer_crawler.test.jsindex 2e6f0db57..a12907486 100644--- a/test/crawlers/puppeteer_crawler.test.js+++ b/test/crawlers/puppeteer_crawler.test.js@@ -34,6 +34,7 @@ describe('PuppeteerCrawler', () => {{ url: 'http://example.com/?q=5' },{ url: 'http://example.com/?q=6' },];+ const sourcesCopy = JSON.parse(JSON.stringify(sources));const processed = [];const failed = [];const requestList = new Apify.RequestList({ sources });@@ -61,7 +62,7 @@ describe('PuppeteerCrawler', () => {expect(failed).toHaveLength(0);processed.forEach((request, id) => {- expect(request.url).toEqual(sources[id].url);+ expect(request.url).toEqual(sourcesCopy[id].url);expect(request.userData.title).toBe('Example Domain');});});diff --git a/test/data/sample.json.gz b/test/data/sample.json.gznew file mode 100644index 000000000..b70515212Binary files /dev/null and b/test/data/sample.json.gz differdiff --git a/test/request_list.test.js b/test/request_list.test.jsindex 760bc5f15..3ca7c4e1c 100644--- a/test/request_list.test.js+++ b/test/request_list.test.js@@ -6,6 +6,7 @@ import { ACTOR_EVENT_NAMES_EX } from '../build/constants';import Apify from '../build/index';import * as keyValueStore from '../build/key_value_store';import * as utils from '../build/utils';+import { deserializeArray } from '../build/serialization';describe('Apify.RequestList', () => {let ll;@@ -76,6 +77,7 @@ describe('Apify.RequestList', () => {{ url: 'https://example.com/7' },{ url: 'https://example.com/8' },];+ const sourcesCopy = JSON.parse(JSON.stringify(sources));const originalList = new Apify.RequestList({ sources });await originalList.initialize();@@ -93,7 +95,7 @@ describe('Apify.RequestList', () => {await originalList.reclaimRequest(r5);const newList = new Apify.RequestList({- sources,+ sources: sourcesCopy,state: originalList.getState(),});await newList.initialize();@@ -416,6 +418,7 @@ describe('Apify.RequestList', () => {],persistStateKey: PERSIST_STATE_KEY,};+ const optsCopy = JSON.parse(JSON.stringify(opts));const requestList = new Apify.RequestList(opts);await requestList.initialize();@@ -457,7 +460,7 @@ describe('Apify.RequestList', () => {.once().withArgs(PERSIST_STATE_KEY).returns(Promise.resolve(requestList.getState()));- const requestList2 = new Apify.RequestList(opts);+ const requestList2 = new Apify.RequestList(optsCopy);await requestList2.initialize();expect(requestList2.getState()).toEqual(requestList.getState());@@ -466,10 +469,13 @@ describe('Apify.RequestList', () => {);test(- 'should correctly persist its sources when persistSourcesKey is set',+ 'should correctly persist its sources when persistRequestsKey is set',async () => {- const PERSIST_SOURCES_KEY = 'some-key';- const mock = sinon.mock(keyValueStore);+ const PERSIST_REQUESTS_KEY = 'some-key';+ const getValueStub = sinon.stub(keyValueStore, 'getValue');+ const setValueStub = sinon.stub(keyValueStore, 'setValue');++ let persistedRequests;const opts = {sources: [@@ -477,26 +483,30 @@ describe('Apify.RequestList', () => {{ url: 'https://example.com/2' },{ url: 'https://example.com/3' },],- persistSourcesKey: PERSIST_SOURCES_KEY,+ persistRequestsKey: PERSIST_REQUESTS_KEY,};const requestList = new Apify.RequestList(opts);- expect(requestList.areSourcesPersisted).toBe(false);+ expect(requestList.areRequestsPersisted).toBe(false);// Expect an attempt to load sources.- mock.expects('getValue')- .once()- .withArgs(PERSIST_SOURCES_KEY)- .returns(null);+ getValueStub.withArgs(PERSIST_REQUESTS_KEY)+ .onFirstCall()+ .resolves(null)+ // See second RequestList below.+ .onSecondCall()+ .callsFake(() => {+ return persistedRequests;+ });// Expect persist sources.- mock.expects('setValue')- .once()- .withArgs(PERSIST_SOURCES_KEY, requestList.sources)- .returns(Promise.resolve());+ setValueStub.withArgs(PERSIST_REQUESTS_KEY)+ .callsFake(async (key, value) => {+ persistedRequests = value;+ });await requestList.initialize();- expect(requestList.areSourcesPersisted).toBe(true);+ expect(requestList.areRequestsPersisted).toBe(true);const opts2 = {sources: [@@ -504,37 +514,32 @@ describe('Apify.RequestList', () => {{ url: 'https://test.com/2' },{ url: 'https://test.com/3' },],- persistSourcesKey: PERSIST_SOURCES_KEY,+ persistRequestsKey: PERSIST_REQUESTS_KEY,};const requestList2 = new Apify.RequestList(opts2);- expect(requestList2.areSourcesPersisted).toBe(false);+ expect(requestList2.areRequestsPersisted).toBe(false);// Now initialize new request list from saved sources and check that// they are same as state of original request list.- mock.expects('getValue')- .once()- .withArgs(PERSIST_SOURCES_KEY)- .returns(Promise.resolve(requestList.sources));-- mock.expects('setValue')- .never();-await requestList2.initialize();- expect(requestList2.areSourcesPersisted).toBe(true);+ expect(requestList2.areRequestsPersisted).toBe(true);expect(requestList2.requests).toEqual(requestList.requests);- mock.verify();+ getValueStub.restore();+ setValueStub.restore();},);test(- 'should correctly persist sources from requestsFromUrl if persistSourcesKey is set',+ 'should correctly persist sources from requestsFromUrl if persistRequestsKey is set',async () => {- const PERSIST_SOURCES_KEY = 'some-key';+ const PERSIST_REQUESTS_KEY = 'some-key';const kvsMock = sinon.mock(keyValueStore);const publicUtilsMock = sinon.mock(utils.publicUtils);+ let persistedRequests;+const opts = {sources: [{ url: 'https://example.com/1' },@@ -542,26 +547,27 @@ describe('Apify.RequestList', () => {{ requestsFromUrl: 'http://example.com/list-urls.txt', userData: { isFromUrl: true } },{ url: 'https://example.com/5' },],- persistSourcesKey: PERSIST_SOURCES_KEY,+ persistRequestsKey: PERSIST_REQUESTS_KEY,};const urlsFromTxt = ['http://example.com/3', 'http://example.com/4'];const requestList = new Apify.RequestList(opts);- expect(requestList.areSourcesPersisted).toBe(false);+ expect(requestList.areRequestsPersisted).toBe(false);// Expect an attempt to load sources.kvsMock.expects('getValue').once()- .withArgs(PERSIST_SOURCES_KEY)- .returns(null);+ .withArgs(PERSIST_REQUESTS_KEY)+ .resolves(null);// Expect persist sources.- const sourcesFromTxt = urlsFromTxt.map(url => ({ url, userData: { isFromUrl: true } }));kvsMock.expects('setValue').once()- .withArgs(PERSIST_SOURCES_KEY, [opts.sources[0], opts.sources[1], ...sourcesFromTxt, opts.sources[3]])- .returns(Promise.resolve());+ .withArgs(PERSIST_REQUESTS_KEY)+ .callsFake((key, value) => {+ persistedRequests = value;+ });// Expect downloadListOfUrls returns list of URLspublicUtilsMock.expects('downloadListOfUrls').once()@@ -569,7 +575,10 @@ describe('Apify.RequestList', () => {.returns(Promise.resolve(urlsFromTxt));await requestList.initialize();- expect(requestList.areSourcesPersisted).toBe(true);+ expect(requestList.areRequestsPersisted).toBe(true);+ const requests = await deserializeArray(persistedRequests);+ expect(requestList.requests).toHaveLength(5);+ expect(requests).toEqual(requestList.requests);kvsMock.verify();publicUtilsMock.verify();@@ -675,6 +684,7 @@ describe('Apify.RequestList', () => {{ url: 'https://www.example.com' },{ url: 'https://www.ex2mple.com' },];+ const sourcesCopy = JSON.parse(JSON.stringify(sources));let requestList = new Apify.RequestList({sources,@@ -688,7 +698,7 @@ describe('Apify.RequestList', () => {const logStub = sinon.stub(console, 'warn');requestList = new Apify.RequestList({- sources: sources.concat([+ sources: sourcesCopy.concat([{ url: 'https://www.example.com', uniqueKey: '123' },{ url: 'https://www.example.com', uniqueKey: '123' },{ url: 'https://www.example.com', uniqueKey: '456' },@@ -719,8 +729,8 @@ describe('Apify.RequestList', () => {const rl = await Apify.openRequestList(name, sources);expect(rl).toBeInstanceOf(Apify.RequestList);expect(rl.persistStateKey.startsWith(name)).toBe(true);- expect(rl.persistSourcesKey.startsWith(name)).toBe(true);- expect(rl.sources).toEqual(sources);+ expect(rl.persistRequestsKey.startsWith(name)).toBe(true);+ expect(rl.sources).toEqual([]);expect(rl.isInitialized).toBe(true);mock.verify();@@ -732,12 +742,13 @@ describe('Apify.RequestList', () => {const name = 'xxx';const sources = ['https://example.com'];+ const requests = sources.map(url => new Apify.Request({ url }));const rl = await Apify.openRequestList(name, sources);expect(rl).toBeInstanceOf(Apify.RequestList);expect(rl.persistStateKey.startsWith(name)).toBe(true);- expect(rl.persistSourcesKey.startsWith(name)).toBe(true);- expect(rl.sources).toEqual(sources.map(url => ({ url })));+ expect(rl.persistRequestsKey.startsWith(name)).toBe(true);+ expect(rl.requests).toEqual(requests);expect(rl.isInitialized).toBe(true);mock.verify();@@ -748,17 +759,19 @@ describe('Apify.RequestList', () => {mock.expects('setValue').atLeast(1).resolves();const name = 'xxx';+ let counter = 0;const sources = [{ url: 'https://example.com' }];+ const requests = sources.map(({ url }) => new Apify.Request({ url, uniqueKey: `${url}-${counter++}` }));const options = {keepDuplicateUrls: true,- persistStateKeyPrefix: 'yyy',+ persistStateKey: 'yyy',};const rl = await Apify.openRequestList(name, sources, options);expect(rl).toBeInstanceOf(Apify.RequestList);expect(rl.persistStateKey.startsWith(name)).toBe(true);- expect(rl.persistSourcesKey.startsWith(name)).toBe(true);- expect(rl.sources).toEqual(sources);+ expect(rl.persistRequestsKey.startsWith(name)).toBe(true);+ expect(rl.requests).toEqual(requests);expect(rl.isInitialized).toBe(true);expect(rl.keepDuplicateUrls).toBe(true);@@ -771,12 +784,13 @@ describe('Apify.RequestList', () => {const name = null;const sources = [{ url: 'https://example.com' }];+ const requests = sources.map(({ url }) => new Apify.Request({ url }));const rl = await Apify.openRequestList(name, sources);expect(rl).toBeInstanceOf(Apify.RequestList);- expect(rl.persistStateKey).toBe(null);- expect(rl.persistSourcesKey).toBe(null);- expect(rl.sources).toEqual(sources);+ expect(rl.persistStateKey == null).toBe(true);+ expect(rl.persistRequestsKey == null).toBe(true);+ expect(rl.requests).toEqual(requests);expect(rl.isInitialized).toBe(true);mock.verify();@@ -784,10 +798,9 @@ describe('Apify.RequestList', () => {test('should throw on invalid parameters', async () => {const args = [[],- ['x', []],- ['x', [6]],- ['x', [[]], {}],- ['x', [[]], []],+ ['x', {}],+ ['x', 6, {}],+ ['x', [], []],];for (const arg of args) {try {@@ -801,4 +814,32 @@ describe('Apify.RequestList', () => {}});});++ // This test is here to run locally. It would take too long+ // when running a test suite and in CI with large source arrays+ // and would be flaky with small source arrays, so manual inspection+ // looks like the best idea, since multiple runs with various values+ // need to be tested and compared (read: I'm too lazy to automate this)++ // test('memory consumption does not spike', async () => {+ // function getMemoryInMbytes() {+ // const memory = process.memoryUsage();+ // return (memory.heapUsed + memory.external) / 1024 / 1024;+ // }+ // const sources = [];+ // for (let i = 0; i < 1e6; i++) {+ // sources.push({ url: `https://example.com?page=${i}` });+ // }+ // const startingMemory = getMemoryInMbytes();+ // console.log(startingMemory, 'MB');+ //+ // process.env.APIFY_LOCAL_STORAGE_DIR = 'tmp';+ // const rl = new Apify.RequestList({ sources, persistRequestsKey: null });+ // const instanceMemory = getMemoryInMbytes();+ // console.log(instanceMemory, 'MB');+ //+ // await rl.initialize();+ // const initMemory = getMemoryInMbytes();+ // console.log(initMemory, 'MB');+ // });});diff --git a/test/serialization.test.js b/test/serialization.test.jsnew file mode 100644index 000000000..66e01462f--- /dev/null+++ b/test/serialization.test.js@@ -0,0 +1,62 @@+import { promises as fs } from 'fs';+import path from 'path';+import util from 'util';+import zlib from 'zlib';+import { serializeArray, deserializeArray, createDeserialize } from '../build/serialization';++const TEST_JSON_PATH = path.join(__dirname, 'data', 'sample.json.gz');+++const gunzip = util.promisify(zlib.gunzip);++describe('Data Compression:', () => {+ test('compressData should compress', async () => {+ const compressedTestJson = await fs.readFile(TEST_JSON_PATH);+ const jsonBuffer = await gunzip(compressedTestJson);+ const expectedArray = JSON.parse(jsonBuffer.toString('utf8'));++ const compressed = await serializeArray(expectedArray);+ const decompressed = await gunzip(compressed);+ const decompressedJson = decompressed.toString('utf8');+ const receivedArray = JSON.parse(decompressedJson);++ // Compare objects, to avoid errors with /n /t and other insignificant+ // characters in the JSON strings.+ expect(receivedArray).toEqual(expectedArray);+ });++ test('decompressData should decompress', async () => {+ const compressedTestJson = await fs.readFile(TEST_JSON_PATH);+ const jsonBuffer = await gunzip(compressedTestJson);+ const expectedArray = JSON.parse(jsonBuffer.toString('utf8'));++ const receivedArray = await deserializeArray(compressedTestJson);++ // Compare objects, to avoid errors with /n /t and other insignificant+ // characters in the JSON strings.+ expect(receivedArray).toEqual(expectedArray);+ });++ test('compressData + decompressData should produce original data', async () => {+ const data = [];+ for (let i = 0; i < 10000; i++) {+ data.push({ [`${Math.random()}`]: Math.random() });+ }+ const compressed = await serializeArray(data);+ const decompressed = await deserializeArray(compressed);+ expect(decompressed).toEqual(data);+ });++ test('createDecompress should work', async () => {+ const compressedTestJson = await fs.readFile(TEST_JSON_PATH);+ const jsonBuffer = await gunzip(compressedTestJson);+ const expectedArray = JSON.parse(jsonBuffer.toString('utf8'));++ const decompress = createDeserialize(compressedTestJson);+ const receivedArray = [];+ for await (const item of decompress) {+ receivedArray.push(item);+ }+ expect(receivedArray).toEqual(expectedArray);+ });+});diff --git a/test/utils.test.js b/test/utils.test.jsindex 7a0148f15..f0968466d 100644--- a/test/utils.test.js+++ b/test/utils.test.js@@ -133,11 +133,12 @@ describe('utils.getMemoryInfo()', () => {osMock.expects('freemem')- .once()+ .atLeast(1).returns(222);osMock.expects('totalmem')+ .atLeast(1).returns(333);try {@@ -196,12 +197,12 @@ describe('utils.getMemoryInfo()', () => {osMock.expects('freemem')- .once()+ .atLeast(1).returns(222);osMock.expects('totalmem')- .once()+ .atLeast(1).returns(333);let browser;diff --git a/types/key_value_store.d.ts b/types/key_value_store.d.tsindex f5956c618..5a2f07757 100644--- a/types/key_value_store.d.ts+++ b/types/key_value_store.d.ts@@ -81,7 +81,7 @@ export class KeyValueStore {* content type are automatically parsed and returned as a JavaScript object.* Similarly, records with `text/plain` content types are returned as a string.* For all other content types, the value is returned as a raw- * [](https://nodejs.org/api/buffer.html) instance.+ * [`Buffer`](https://nodejs.org/api/buffer.html) instance.** If the record does not exist, the function resolves to `null`.*@@ -99,7 +99,7 @@ export class KeyValueStore {* of the following characters: `a`-`z`, `A`-`Z`, `0`-`9` and `!-_.'()`* @returns {Promise<(object|Buffer|string|null)>}* Returns a promise that resolves to an object, string- * or [](https://nodejs.org/api/buffer.html), depending+ * or [`Buffer`](https://nodejs.org/api/buffer.html), depending* on the MIME content type of the record.*/getValue(key: string): Promise<any>;@@ -124,7 +124,7 @@ export class KeyValueStore {* await store.setValue('RESULTS', 'my text data', { contentType: 'text/plain' });* ```* If you set custom content type, `value` must be either a string or- * [](https://nodejs.org/api/buffer.html), otherwise an error will be thrown.+ * [`Buffer`](https://nodejs.org/api/buffer.html), otherwise an error will be thrown.** If `value` is `null`, the record is deleted instead. Note that the `setValue()` function succeeds* regardless whether the record existed or not.@@ -144,7 +144,7 @@ export class KeyValueStore {* <li>If `null`, the record in the key-value store is deleted.</li>* <li>If no `options.contentType` is specified, `value` can be any JavaScript object and it will be stringified to JSON.</li>* <li>If `options.contentType` is specified, `value` is considered raw data and it must be either a `String`- * or [](https://nodejs.org/api/buffer.html).</li>+ * or [`Buffer`](https://nodejs.org/api/buffer.html).</li>* </ul>* For any other value an error will be thrown.* @param {Object} [options]diff --git a/types/main.d.ts b/types/main.d.tsindex b065ad420..2e29c5895 100644--- a/types/main.d.ts+++ b/types/main.d.ts@@ -127,7 +127,7 @@ export { LaunchPuppeteerOptions } from "./puppeteer";export { LaunchPuppeteerFunction, PuppeteerPoolOptions } from "./puppeteer_pool";export { InterceptHandler } from "./puppeteer_request_interception";export { RequestOptions } from "./request";-export { RequestListInput, SourceInput, RequestListOptions, RequestListState } from "./request_list";+export { RequestListOptions, RequestListState } from "./request_list";export { QueueOperationInfo } from "./request_queue";export { SessionState, SessionOptions } from "./session_pool/session";export { CreateSession, SessionPoolOptions } from "./session_pool/session_pool";diff --git a/types/request.d.ts b/types/request.d.tsindex f60805b91..a4aa13f10 100644--- a/types/request.d.ts+++ b/types/request.d.ts@@ -8,7 +8,7 @@ export function computeUniqueKey({ url, method, payload, keepUrlFragment, useExtexport function hashPayload(payload: any): string;export default Request;/**- * Specifies required and optional fields for constructing a [`Request`](../api/request).+ * Specifies required and optional fields for constructing a {@link Request}.*/export type RequestOptions = {/**diff --git a/types/request_list.d.ts b/types/request_list.d.tsindex 9c1a0502b..d0802847f 100644--- a/types/request_list.d.ts+++ b/types/request_list.d.ts@@ -1,19 +1,15 @@export const STATE_PERSISTENCE_KEY: "REQUEST_LIST_STATE";-export const SOURCES_PERSISTENCE_KEY: "REQUEST_LIST_SOURCES";-/**- * @typedef RequestListInput- * @property {string} [method]- * @property {string} [requestsFromUrl]- * @property {RegExp} [regex]- */-/**- * @typedef {Array<(RequestListInput|RequestOptions|Request)>} SourceInput- */+export const REQUESTS_PERSISTENCE_KEY: "REQUEST_LIST_REQUESTS";/*** @typedef RequestListOptions- * @property {SourceInput} sources- * An array of sources of URLs for the `RequestList`. It can be either an array of plain objects that- * define the `url` property, or an array of instances of the {@link Request} class.+ * @property {Array<RequestOptions|Request|string>} [sources]+ * An array of sources of URLs for the {@link RequestList}. It can be either an array of strings,+ * plain objects that define at least the `url` property, or an array of {@link Request} instances.+ *+ * **IMPORTANT:** The `sources` array will be consumed (left empty) after `RequestList` initializes.+ * This is a measure to prevent memory leaks in situations when millions of sources are+ * added.+ ** Additionally, the `requestsFromUrl` property may be used instead of `url`,* which will instruct `RequestList` to download the source URLs from a given remote location.* The URLs will be parsed from the received response.@@ -21,7 +17,10 @@ export const SOURCES_PERSISTENCE_KEY: "REQUEST_LIST_SOURCES";* ```* [* // A single URL- * { method: 'GET', url: 'http://example.com/a/b' },+ * 'http://example.com/a/b',+ *+ * // Modify Request options+ * { method: PUT, 'https://example.com/put, payload: { foo: 'bar' }}** // Batch import of URLs from a file hosted on the web,* // where the URLs should be requested using the HTTP POST request@@ -35,6 +34,45 @@ export const SOURCES_PERSISTENCE_KEY: "REQUEST_LIST_SOURCES";* { requestsFromUrl: 'https://docs.google.com/spreadsheets/d/1GA5sSQhQjB_REes8I5IKg31S-TuRcznWOPjcpNqtxmU/gviz/tq?tqx=out:csv' }* ]* ```+ * @property {Function} [sourcesFunction]+ * A function that will be called to get the sources for the `RequestList`, but only if `RequestList`+ * was not able to fetch their persisted version (see {@link RequestListOptions.persistRequestsKey}).+ * It must return an `Array` of {@link Request} or {@link RequestOptions}.+ *+ * This is very useful in a scenario when getting the sources is a resource intensive or time consuming+ * task, such as fetching URLs from multiple sitemaps or parsing URLs from large datasets. Using the+ * `sourcesFunction` in combination with `persistStateKey` and `persistRequestsKey` will allow you to+ * fetch and parse those URLs only once, saving valuable time when your actor migrates or restarts.+ *+ * If both {@link RequestListOptions.sources} and {@link RequestListOptions.sourcesFunction} are provided,+ * the sources returned by the function will be added after the `sources`.+ *+ * **Example:**+ * ```javascript+ * // Let's say we want to scrape URLs extracted from sitemaps.+ *+ * const sourcesFunction = async () => {+ * // With super large sitemaps, this operation could take very long+ * // and big websites typically have multiple sitemaps.+ * const sitemaps = await downloadHugeSitemaps();+ * return parseUrlsFromSitemaps(sitemaps);+ * }+ *+ * // Sitemaps can change in real-time, so it's important to persist+ * // the URLs we collected. Otherwise we might lose our scraping+ * // state in case of an actor migration / failure / time-out.+ * const requestList = new RequestList({+ * sourcesFunction,+ * persistStateKey: 'state-key',+ * persistRequestsKey: 'requests-key',+ * })+ *+ * // The sourcesFunction is called now and the Requests are persisted.+ * // If something goes wrong and we need to start again, RequestList+ * // will load the persisted Requests from storage and will NOT+ * // call the sourcesFunction again, saving time and resources.+ * await requestList.initialize();+ * ```* @property {string} [persistStateKey]* Identifies the key in the default key-value store under which `RequestList` periodically stores its* state (i.e. which URLs were crawled and which not).@@ -43,13 +81,14 @@ export const SOURCES_PERSISTENCE_KEY: "REQUEST_LIST_SOURCES";** If `persistStateKey` is not set, `RequestList` will always start from the beginning,* and all the source URLs will be crawled again.- * @property {string} [persistSourcesKey]+ * @property {string} [persistRequestsKey]* Identifies the key in the default key-value store under which the `RequestList` persists its- * sources (i.e. the lists of URLs) during the {@link RequestList#initialize} call.+ * Requests during the {@link RequestList#initialize} call.* This is necessary if `persistStateKey` is set and the source URLs might potentially change,- * to ensure consistency of the source URLs and state object. However, it comes with some storage and performance overheads.+ * to ensure consistency of the source URLs and state object. However, it comes with some+ * storage and performance overheads.*- * If `persistSourcesKey` is not set, {@link RequestList#initialize} will always fetch the sources+ * If `persistRequestsKey` is not set, {@link RequestList#initialize} will always fetch the sources* from their origin, check that they are consistent with the restored state (if any)* and throw an error if they are not.* @property {RequestListState} [state]@@ -109,17 +148,28 @@ export const SOURCES_PERSISTENCE_KEY: "REQUEST_LIST_SOURCES";** The internal state is closely tied to the provided sources (URLs). If the sources change on actor restart, the state will become corrupted and* `RequestList` will raise an exception. This typically happens when the sources is a list of URLs downloaded from the web.- * In such case, use the `persistSourcesKey` option in conjunction with `persistStateKey`,+ * In such case, use the `persistRequestsKey` option in conjunction with `persistStateKey`,* to make the `RequestList` store the initial sources to the default key-value store and load them after restart,* which will prevent any issues that a live list of URLs might cause.*- * **Example usage:**+ * **Basic usage:**+ * ```javascript+ * // Use a helper function to simplify request list initialization.+ * // State and sources are automatically persisted.+ * const requestList = await Apify.openRequestList('my-request-list', [+ * 'http://www.example.com/page-1',+ * { url: 'http://www.example.com/page-2', method: 'POST', userData: { foo: 'bar' }},+ * { requestsFromUrl: 'http://www.example.com/my-url-list.txt', userData: { isFromUrl: true } },+ * ]);+ * ```*+ * **Advanced usage:*** ```javascript+ * // Use the constructor to get more control over the initialization.* const requestList = new Apify.RequestList({* sources: [* // Separate requests- * { url: 'http://www.example.com/page-1', method: 'GET', headers: {} },+ * { url: 'http://www.example.com/page-1', method: 'GET', headers: { ... } },* { url: 'http://www.example.com/page-2', userData: { foo: 'bar' }},** // Bulk load of URLs from file `http://www.example.com/my-url-list.txt`@@ -127,25 +177,11 @@ export const SOURCES_PERSISTENCE_KEY: "REQUEST_LIST_SOURCES";* { requestsFromUrl: 'http://www.example.com/my-url-list.txt', userData: { isFromUrl: true } },* ],*- * // Ensure both the sources and crawling state of the request list is persisted,- * // so that on actor restart, the crawling will continue where it left off+ * // Persist only state in cases where the original sources are immutable to improve performance.* persistStateKey: 'my-state',- * persistSourcesKey: 'my-sources'* });*- * // This call loads and parses the URLs from the remote file.* await requestList.initialize();- *- * // Get requests from list- * const request1 = await requestList.fetchNextRequest();- * const request2 = await requestList.fetchNextRequest();- * const request3 = await requestList.fetchNextRequest();- *- * // Mark some of them as handled- * await requestList.markRequestHandled(request1);- *- * // If processing fails then reclaim it back to the list- * await requestList.reclaimRequest(request2);* ```*/export class RequestList {@@ -159,14 +195,15 @@ export class RequestList {inProgress: {};reclaimed: {};persistStateKey: string;- persistSourcesKey: string;+ persistRequestsKey: any;initialState: RequestListState;keepDuplicateUrls: boolean;isStatePersisted: boolean;- areSourcesPersisted: boolean;+ areRequestsPersisted: boolean;isLoading: boolean;isInitialized: boolean;- sources: any;+ sources: (string | Request | RequestOptions)[];+ sourcesFunction: Function;/*** Loads all remote sources of URLs and potentially starts periodic state persistence.* This function must be called before you can start using the instance in a meaningful way.@@ -174,6 +211,23 @@ export class RequestList {* @returns {Promise<void>}*/initialize(): Promise<void>;+ /**+ * Adds previously persisted Requests, as retrieved from the key-value store.+ * This needs to be done in a memory efficient way. We should update the input+ * to a Stream once apify-client supports streams.+ * @param {Buffer} persistedRequests+ * @ignore+ */+ _addPersistedRequests(persistedRequests: Buffer): Promise<void>;+ /**+ * Add Requests from both options.sources and options.sourcesFunction.+ * This function is called only when persisted sources were not loaded.+ * We need to avoid keeping both sources and requests in memory+ * to reduce memory footprint with very large sources.+ * @returns {Promise<void>}+ * @ignore+ */+ _addRequestsFromSources(): Promise<void>;/*** Persists the current state of the `RequestList` into the default {@link KeyValueStore}.* The state is persisted automatically in regular intervals, but calling this method manually@@ -186,13 +240,13 @@ export class RequestList {persistState(): Promise<void>;/*** Unlike persistState(), this is used only internally, since the sources- * are automatically persisted at RequestList initialization (if the persistSourcesKey is set),+ * are automatically persisted at RequestList initialization (if the persistRequestsKey is set),* but there's no reason to persist it again afterwards, because RequestList is immutable.** @return {Promise<void>}* @ignore*/- _persistSources(): Promise<void>;+ _persistRequests(): Promise<void>;/*** Restores RequestList state from a state object.*@@ -201,13 +255,13 @@ export class RequestList {*/_restoreState(state: RequestListState): void;/**- * Attempts to load state and sources using the `RequestList` configuration- * and returns a tuple of [state, sources] where each may be null if not loaded.+ * Attempts to load state and requests using the `RequestList` configuration+ * and returns a tuple of [state, requests] where each may be null if not loaded.** @return {Promise<Array<(RequestListState|null)>>}* @ignore*/- _loadStateAndSources(): Promise<(RequestListState | null)[]>;+ _loadStateAndPersistedRequests(): Promise<(RequestListState | null)[]>;/*** Returns an object representing the internal state of the `RequestList` instance.* Note that the object's fields can change in future releases.@@ -270,12 +324,13 @@ export class RequestList {_fetchRequestsFromUrl(source: any): Promise<RequestOptions[]>;/*** Adds given request.- * If the `opts` parameter is a plain object and not an instance of a `Request`, then the function- * creates a `Request` instance.+ * If the `source` parameter is a string or plain object and not an instance+ * of a `Request`, then the function creates a `Request` instance.*+ * @param {string|Request|object} source* @ignore*/- _addRequest(opts: any): void;+ _addRequest(source: any): void;/*** Helper function that validates unique key.* Throws an error if uniqueKey is not a non-empty string.@@ -308,17 +363,16 @@ export class RequestList {*/handledCount(): number;}-export function openRequestList(listName: string | null, sources: string[] | (Request | RequestOptions | RequestListInput)[], options?: RequestListOptions | undefined): Promise<RequestList>;-export type RequestListInput = {- method?: string;- requestsFromUrl?: string;- regex?: RegExp;-};-export type SourceInput = (Request | RequestOptions | RequestListInput)[];+export function openRequestList(listName: string | null, sources: (string | Request | RequestOptions)[], options?: RequestListOptions | undefined): Promise<RequestList>;export type RequestListOptions = {/**- * An array of sources of URLs for the `RequestList`. It can be either an array of plain objects that- * define the `url` property, or an array of instances of the {@link Request} class.+ * An array of sources of URLs for the {@link RequestList}. It can be either an array of strings,+ * plain objects that define at least the `url` property, or an array of {@link Request} instances.+ *+ * **IMPORTANT:** The `sources` array will be consumed (left empty) after `RequestList` initializes.+ * This is a measure to prevent memory leaks in situations when millions of sources are+ * added.+ ** Additionally, the `requestsFromUrl` property may be used instead of `url`,* which will instruct `RequestList` to download the source URLs from a given remote location.* The URLs will be parsed from the received response.@@ -326,7 +380,10 @@ export type RequestListOptions = {* ```* [* // A single URL- * { method: 'GET', url: 'http://example.com/a/b' },+ * 'http://example.com/a/b',+ *+ * // Modify Request options+ * { method: PUT, 'https://example.com/put, payload: { foo: 'bar' }}** // Batch import of URLs from a file hosted on the web,* // where the URLs should be requested using the HTTP POST request@@ -341,7 +398,48 @@ export type RequestListOptions = {* ]* ```*/- sources: (Request | RequestOptions | RequestListInput)[];+ sources?: (string | Request | RequestOptions)[];+ /**+ * A function that will be called to get the sources for the `RequestList`, but only if `RequestList`+ * was not able to fetch their persisted version (see {@link RequestListOptions.persistRequestsKey}).+ * It must return an `Array` of {@link Request} or {@link RequestOptions}.+ *+ * This is very useful in a scenario when getting the sources is a resource intensive or time consuming+ * task, such as fetching URLs from multiple sitemaps or parsing URLs from large datasets. Using the+ * `sourcesFunction` in combination with `persistStateKey` and `persistRequestsKey` will allow you to+ * fetch and parse those URLs only once, saving valuable time when your actor migrates or restarts.+ *+ * If both {@link RequestListOptions.sources} and {@link RequestListOptions.sourcesFunction} are provided,+ * the sources returned by the function will be added after the `sources`.+ *+ * **Example:**+ * ```javascript+ * // Let's say we want to scrape URLs extracted from sitemaps.+ *+ * const sourcesFunction = async () => {+ * // With super large sitemaps, this operation could take very long+ * // and big websites typically have multiple sitemaps.+ * const sitemaps = await downloadHugeSitemaps();+ * return parseUrlsFromSitemaps(sitemaps);+ * }+ *+ * // Sitemaps can change in real-time, so it's important to persist+ * // the URLs we collected. Otherwise we might lose our scraping+ * // state in case of an actor migration / failure / time-out.+ * const requestList = new RequestList({+ * sourcesFunction,+ * persistStateKey: 'state-key',+ * persistRequestsKey: 'requests-key',+ * })+ *+ * // The sourcesFunction is called now and the Requests are persisted.+ * // If something goes wrong and we need to start again, RequestList+ * // will load the persisted Requests from storage and will NOT+ * // call the sourcesFunction again, saving time and resources.+ * await requestList.initialize();+ * ```+ */+ sourcesFunction?: Function;/*** Identifies the key in the default key-value store under which `RequestList` periodically stores its* state (i.e. which URLs were crawled and which not).@@ -354,15 +452,16 @@ export type RequestListOptions = {persistStateKey?: string;/*** Identifies the key in the default key-value store under which the `RequestList` persists its- * sources (i.e. the lists of URLs) during the {@link RequestList#initialize} call.+ * Requests during the {@link RequestList#initialize} call.* This is necessary if `persistStateKey` is set and the source URLs might potentially change,- * to ensure consistency of the source URLs and state object. However, it comes with some storage and performance overheads.+ * to ensure consistency of the source URLs and state object. However, it comes with some+ * storage and performance overheads.*- * If `persistSourcesKey` is not set, {@link RequestList#initialize} will always fetch the sources+ * If `persistRequestsKey` is not set, {@link RequestList#initialize} will always fetch the sources* from their origin, check that they are consistent with the restored state (if any)* and throw an error if they are not.*/- persistSourcesKey?: string;+ persistRequestsKey?: string;/*** The state object that the `RequestList` will be initialized from.* It is in the form as returned by `RequestList.getState()`, such as follows:@@ -399,7 +498,7 @@ export type RequestListOptions = {keepDuplicateUrls?: boolean;};/**- * Represents state of a {RequestList}. It can be used to resume a {RequestList} which has been previously processed.+ * Represents state of a {@link RequestList}. It can be used to resume a {@link RequestList} which has been previously processed.* You can obtain the state by calling {@link RequestList#getState} and receive an object with* the following structure:*diff --git a/types/serialization.d.ts b/types/serialization.d.tsnew file mode 100644index 000000000..05ee8a4e2--- /dev/null+++ b/types/serialization.d.ts@@ -0,0 +1,4 @@+export function serializeArray(data: any[]): Promise<Buffer>;+export function deserializeArray(compressedData: Buffer): Promise<any[]>;+export function createDeserialize(compressedData: any): stream.Readable;+import * as stream from "stream";