Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.
TypeScript apifyautomationcrawlercrawlingheadlessheadless-chromejavascriptnodejsnpmplaywrightpuppeteerscraperscrapingtypescriptweb-crawlerweb-crawlingweb-scraping
build docs
vdrmota1f71a6872fad
diff --git a/docs/api/utils.md b/docs/api/utils.mdindex a8e2cc911..ddf13066f 100644--- a/docs/api/utils.md+++ b/docs/api/utils.md@@ -23,6 +23,7 @@ await Apify.utils.sleep(1500);* [`.sleep`](#utils.sleep) ⇒ `Promise`* [`.URL_NO_COMMAS_REGEX`](#utils.URL_NO_COMMAS_REGEX)* [`.URL_WITH_COMMAS_REGEX`](#utils.URL_WITH_COMMAS_REGEX)+ * [`.exports.infiniteScroll`](#utils.exports.infiniteScroll) ⇒ `Promise`* [`.isDocker()`](#utils.isDocker) ⇒ `Promise`* [`.downloadListOfUrls(options)`](#utils.downloadListOfUrls) ⇒ `Promise<Array<String>>`* [`.extractUrls(string, [urlRegExp])`](#utils.extractUrls) ⇒ `Array<String>`@@ -181,6 +182,40 @@ and does not support URLs containing commas or spaces. The URLs also may containRegular expression that, in addition to the default regular expression `URL_NO_COMMAS_REGEX`, supports matching commas in URL path and query.Note, however, that this may prevent parsing URLs from comma delimited lists, or the URLs may become malformed.+<a name="utils.exports.infiniteScroll"></a>++## `utils.exports.infiniteScroll` ⇒ `Promise`+Scrolls to the bottom of a page, or until it times out.+Loads dynamic content when it hits the bottom of a page, and then continues scrolling.++<table>+<thead>+<tr>+<th>Param</th><th>Type</th><th>Default</th>+</tr>+</thead>+<tbody>+<tr>+<td><code>options</code></td><td><code>Object</code></td><td></td>+</tr>+<tr>+<td colspan="3"></td></tr><tr>+<td><code>options.page</code></td><td><code>Object</code></td><td></td>+</tr>+<tr>+<td colspan="3"><p>The page object returned from browser.newPage()</p>+</td></tr><tr>+<td><code>[options.timeoutSecs]</code></td><td><code>Number</code></td><td><code>0</code></td>+</tr>+<tr>+<td colspan="3"><p>How many seconds to scroll for. If 0, will scroll until bottom of page.</p>+</td></tr><tr>+<td><code>[options.waitForSecs]</code></td><td><code>Number</code></td><td><code>4</code></td>+</tr>+<tr>+<td colspan="3"><p>How many seconds to wait for no new content to load before exit.</p>+</td></tr></tbody>+</table><a name="utils.isDocker"></a>## `utils.isDocker()` ⇒ `Promise`diff --git a/package.json b/package.jsonindex d0efed72c..d4f16ce1f 100644--- a/package.json+++ b/package.json@@ -60,6 +60,7 @@"htmlparser2": "^3.10.1","iltorb": "^2.4.2","jquery": "^3.3.1",+ "jsdoc-to-markdown": "^5.0.0","mime": "^2.3.1","proxy-chain": "^0.2.5","request": "^2.88.0",