Skip to content

Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.

build docs

vdrmota1f71a6872fad
diff --git a/docs/api/utils.md b/docs/api/utils.md
index a8e2cc911..ddf13066f 100644
--- a/docs/api/utils.md
+++ b/docs/api/utils.md
@@ -23,6 +23,7 @@ await Apify.utils.sleep(1500);
* [`.sleep`](#utils.sleep) ⇒ `Promise`
* [`.URL_NO_COMMAS_REGEX`](#utils.URL_NO_COMMAS_REGEX)
* [`.URL_WITH_COMMAS_REGEX`](#utils.URL_WITH_COMMAS_REGEX)
+ * [`.exports.infiniteScroll`](#utils.exports.infiniteScroll) ⇒ `Promise`
* [`.isDocker()`](#utils.isDocker) ⇒ `Promise`
* [`.downloadListOfUrls(options)`](#utils.downloadListOfUrls) ⇒ `Promise<Array<String>>`
* [`.extractUrls(string, [urlRegExp])`](#utils.extractUrls) ⇒ `Array<String>`
@@ -181,6 +182,40 @@ and does not support URLs containing commas or spaces. The URLs also may contain
Regular expression that, in addition to the default regular expression `URL_NO_COMMAS_REGEX`, supports matching commas in URL path and query.
Note, however, that this may prevent parsing URLs from comma delimited lists, or the URLs may become malformed.
+<a name="utils.exports.infiniteScroll"></a>
+
+## `utils.exports.infiniteScroll` ⇒ `Promise`
+Scrolls to the bottom of a page, or until it times out.
+Loads dynamic content when it hits the bottom of a page, and then continues scrolling.
+
+<table>
+<thead>
+<tr>
+<th>Param</th><th>Type</th><th>Default</th>
+</tr>
+</thead>
+<tbody>
+<tr>
+<td><code>options</code></td><td><code>Object</code></td><td></td>
+</tr>
+<tr>
+<td colspan="3"></td></tr><tr>
+<td><code>options.page</code></td><td><code>Object</code></td><td></td>
+</tr>
+<tr>
+<td colspan="3"><p>The page object returned from browser.newPage()</p>
+</td></tr><tr>
+<td><code>[options.timeoutSecs]</code></td><td><code>Number</code></td><td><code>0</code></td>
+</tr>
+<tr>
+<td colspan="3"><p>How many seconds to scroll for. If 0, will scroll until bottom of page.</p>
+</td></tr><tr>
+<td><code>[options.waitForSecs]</code></td><td><code>Number</code></td><td><code>4</code></td>
+</tr>
+<tr>
+<td colspan="3"><p>How many seconds to wait for no new content to load before exit.</p>
+</td></tr></tbody>
+</table>
<a name="utils.isDocker"></a>
## `utils.isDocker()` ⇒ `Promise`
diff --git a/package.json b/package.json
index d0efed72c..d4f16ce1f 100644
--- a/package.json
+++ b/package.json
@@ -60,6 +60,7 @@
"htmlparser2": "^3.10.1",
"iltorb": "^2.4.2",
"jquery": "^3.3.1",
+ "jsdoc-to-markdown": "^5.0.0",
"mime": "^2.3.1",
"proxy-chain": "^0.2.5",
"request": "^2.88.0",