Skip to content

Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.

Move infiniteScroll to utils.puppeteer, improve test and docs

Ondra Urbanf1204cd365a7
diff --cc CHANGELOG.md
index 574e41587,a1ecca4de..bc07837e0
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@@ -1,22 -1,8 +1,22 @@@
-xxx
+NEXT
====================
-- Bugfix in BasicCrawler: async calls in `isFinishedFunction` were not awaited
-- Better logging of memory overload errors
-- Upgraded `apify-client` to 0.5.22
+- **BREAKING CHANGE** Removed support for Web Driver (Selenium) since no further updates are planned.
+ If you wish to continue using Web Driver, please stay on Apify SDK version ^0.14.15
+- **BREAKING CHANGE**: `Dataset.getData()` throws an error if user provides an unsupported option
+ when using local disk storage.
+- **DEPRECATED**: `options.userData` of `Apify.utils.enqueueLinks()` is deprecated.
+ Use `options.transformRequestFunction` instead.
+- Improve logging of memory overload errors.
+- Improve error message in `Apify.call()`.
+- Fix multiple log lines appearing when a crawler was about to finish.
+- Add `Apify.utils.puppeteer.enqueueLinksByClickingElements()` function which enables you
+ to add requests to the queue from pure JavaScript navigations, form submissions etc.
+- The `RequestQueue.handledCount()` function has been resurrected from deprecation,
+ in order to have compatible interface with `RequestList`.
+- Add `useExtendedUniqueKey` option to `Request` constructor to include `method` and `payload`
+ in the `Request`'s computed `uniqueKey`.
+- Updated Puppeteer to 1.18.0
-
++- Updated `apify-client` to 0.5.22
0.14.15 / 2019-05-31
====================
diff --cc docs/api/puppeteer.md
index 150326a46,01eb88741..d1d4e5ba2
--- a/docs/api/puppeteer.md
+++ b/docs/api/puppeteer.md
@@@ -28,6 -27,6 +28,7 @@@ await puppeteer.injectJQuery(page)
* [`.addInterceptRequestHandler`](#puppeteer.addInterceptRequestHandler) ⇒ `Promise`
* [`.removeInterceptRequestHandler`](#puppeteer.removeInterceptRequestHandler) ⇒ `Promise`
* [`.gotoExtended`](#puppeteer.gotoExtended) ⇒ `Promise<Response>`
++ * [`.infiniteScroll`](#puppeteer.infiniteScroll) ⇒ `Promise`
* [`.injectFile(page, filePath, [options])`](#puppeteer.injectFile) ⇒ `Promise`
* [`.injectJQuery(page)`](#puppeteer.injectJQuery) ⇒ `Promise`
* [`.injectUnderscore(page)`](#puppeteer.injectUnderscore) ⇒ `Promise`
@@@ -281,6 -158,6 +282,40 @@@ browser cache which degrades performanc
<td colspan="3"><p>Custom options for <code>page.goto()</code>.</p>
</td></tr></tbody>
</table>
++<a name="puppeteer.infiniteScroll"></a>
++
++## `puppeteer.infiniteScroll` ⇒ `Promise`
++Scrolls to the bottom of a page, or until it times out.
++Loads dynamic content when it hits the bottom of a page, and then continues scrolling.
++
++<table>
++<thead>
++<tr>
++<th>Param</th><th>Type</th><th>Default</th>
++</tr>
++</thead>
++<tbody>
++<tr>
++<td><code>page</code></td><td><code>Object</code></td><td></td>
++</tr>
++<tr>
++<td colspan="3"><p>Puppeteer <a href="https://pptr.dev/#?product=Puppeteer&show=api-class-page" target="_blank"><code>Page</code></a> object.</p>
++</td></tr><tr>
++<td><code>[options]</code></td><td><code>Object</code></td><td></td>
++</tr>
++<tr>
++<td colspan="3"></td></tr><tr>
++<td><code>[options.timeoutSecs]</code></td><td><code>Number</code></td><td><code>0</code></td>
++</tr>
++<tr>
++<td colspan="3"><p>How many seconds to scroll for. If 0, will scroll until bottom of page.</p>
++</td></tr><tr>
++<td><code>[options.waitForSecs]</code></td><td><code>Number</code></td><td><code>4</code></td>
++</tr>
++<tr>
++<td colspan="3"><p>How many seconds to wait for no new content to load before exit.</p>
++</td></tr></tbody>
++</table>
<a name="puppeteer.injectFile"></a>
## `puppeteer.injectFile(page, filePath, [options])` ⇒ `Promise`
diff --cc docs/api/utils.md
index 1389cfdab,a8e2cc911..5c617d274
--- a/docs/api/utils.md
+++ b/docs/api/utils.md
@@@ -23,10 -23,9 +23,9 @@@ await Apify.utils.sleep(1500)
* [`.sleep`](#utils.sleep) ⇒ `Promise`
* [`.URL_NO_COMMAS_REGEX`](#utils.URL_NO_COMMAS_REGEX)
* [`.URL_WITH_COMMAS_REGEX`](#utils.URL_WITH_COMMAS_REGEX)
- * [`.exports.infiniteScroll`](#utils.exports.infiniteScroll) ⇒ `Promise`
* [`.isDocker()`](#utils.isDocker) ⇒ `Promise`
* [`.downloadListOfUrls(options)`](#utils.downloadListOfUrls) ⇒ `Promise<Array<String>>`
- * [`.extractUrls(string, [urlRegExp])`](#utils.extractUrls) ⇒ `Array<String>`
+ * [`.extractUrls(options)`](#utils.extractUrls) ⇒ `Array<String>`
* [`.getRandomUserAgent()`](#utils.getRandomUserAgent) ⇒ `String`
* [`.htmlToText(html)`](#utils.htmlToText) ⇒ `String`
diff --cc src/puppeteer_utils.js
index a7dbd833d,4de3a0a89..e6496ea66
--- a/src/puppeteer_utils.js
+++ b/src/puppeteer_utils.js
@@@ -442,6 -431,6 +442,81 @@@ export const gotoExtended = async (page
return page.goto(request.url, gotoOptions);
};
++/**
++ * Scrolls to the bottom of a page, or until it times out.
++ * Loads dynamic content when it hits the bottom of a page, and then continues scrolling.
++ * @param {Object} page
++ * Puppeteer <a href="https://pptr.dev/#?product=Puppeteer&show=api-class-page" target="_blank"><code>Page</code></a> object.
++ * @param {Object} [options]
++ * @param {Number} [options.timeoutSecs=0]
++ * How many seconds to scroll for. If 0, will scroll until bottom of page.
++ * @param {Number} [options.waitForSecs=4]
++ * How many seconds to wait for no new content to load before exit.
++ * @returns {Promise}
++ * @memberOf puppeteer
++ * @name infiniteScroll
++ */
++export const infiniteScroll = async (page, options = {}) => {
++ const { timeoutSecs = 0, waitForSecs = 4 } = options;
++
++ checkParamOrThrow(page, 'page', 'Object');
++ checkParamOrThrow(timeoutSecs, 'timeoutSecs', 'Number');
++ checkParamOrThrow(waitForSecs, 'waitForSecs', 'Number');
++
++ let finished;
++ const startTime = Date.now();
++ const CHECK_INTERVAL_MILLIS = 1000;
++ const SCROLL_HEIGHT_IF_ZERO = 10000;
++ const maybeResourceTypesInfiniteScroll = ['xhr', 'fetch', 'websocket', 'other'];
++ const resourcesStats = {
++ newRequested: 0,
++ oldRequested: 0,
++ matchNumber: 0,
++ };
++
++ page.on('request', (msg) => {
++ if (maybeResourceTypesInfiniteScroll.includes(msg.resourceType())) {
++ resourcesStats.newRequested++;
++ }
++ });
++
++ const checkFinished = setInterval(() => {
++ // console.log(resourcesStats)
++ if (resourcesStats.oldRequested === resourcesStats.newRequested) {
++ resourcesStats.matchNumber++;
++ if (resourcesStats.matchNumber >= waitForSecs) {
++ clearInterval(checkFinished);
++ finished = true;
++ return;
++ }
++ } else {
++ resourcesStats.matchNumber = 0;
++ resourcesStats.oldRequested = resourcesStats.newRequested;
++ }
++ // check if timeout has been reached
++ if (timeoutSecs !== 0 && (Date.now() - startTime) / 1000 > timeoutSecs) {
++ // console.log("Timeout limit reached, exiting infinite scroll.")
++ clearInterval(checkFinished);
++ finished = true;
++ }
++ }, CHECK_INTERVAL_MILLIS);
++
++
++ /* global window document */
++
++ const doScroll = async () => {
++ /* istanbul ignore next */
++ await page.evaluate(async () => {
++ const delta = document.body.scrollHeight === 0 ? SCROLL_HEIGHT_IF_ZERO : document.body.scrollHeight;
++ window.scrollBy(0, delta);
++ });
++ };
++
++ while (!finished) {
++ await doScroll();
++ }
++};
++
let logEnqueueLinksDeprecationWarning = true;
/**
@@@ -486,4 -474,4 +561,5 @@@ export const puppeteerUtils =
gotoExtended,
addInterceptRequestHandler,
removeInterceptRequestHandler,
++ infiniteScroll,
};
diff --cc test/integration_tests/utils.js
index 18d1894db,000000000..1d253fd61
mode 100644,000000..100644
--- a/test/integration_tests/utils.js
+++ b/test/integration_tests/utils.js
@@@ -1,49 -1,0 +1,34 @@@
++// import { expect } from 'chai';
+import * as utils from '../../build/utils';
-
- const puppeteer = require('puppeteer');
++import Apify from '../../build/index';
+
+describe('utils.infiniteScroll()', () => {
- it('exits after no more to scroll', () => {
- (async () => {
- const browser = await puppeteer.launch({
- headless: true,
- });
- const page = await browser.newPage();
- const contentHTML = '<div>nothing</div>';
- await page.setContent(contentHTML);
- await utils.infiniteScroll({ page });
- await browser.close();
- })();
- });
-
- it('exits after reaches the bottom', () => {
- (async () => {
- const browser = await puppeteer.launch({
- headless: true,
- });
++ xit('exits after it reaches the bottom', async () => {
++ const browser = await Apify.launchPuppeteer({ headless: true });
++ try {
+ const page = await browser.newPage();
+ // Note: external website
+ await page.goto('https://twitter.com/search?src=typd&q=%23fingervein&lang=sv', {
+ waitUntil: 'networkidle2',
+ });
+ await utils.infiniteScroll({ page });
++ } finally {
+ await browser.close();
- })();
++ }
+ });
+
- it('times out if limit is set', () => {
- (async () => {
- const browser = await puppeteer.launch({
- headless: true,
- });
++ xit('times out if limit is set', async () => {
++ const browser = await Apify.launchPuppeteer({ headless: true });
++ try {
+ const page = await browser.newPage();
+ // Note: external website
+ await page.goto('https://medium.com/search?q=biometrics', {
+ waitUntil: 'networkidle2',
+ });
+ const timeoutSecs = 10; // seconds
+ await utils.infiniteScroll({ page, timeoutSecs });
++ } finally {
+ await browser.close();
- })();
++ }
+ });
+});
diff --cc test/puppeteer_utils.js
index 322754c1e,69c57360d..3f6352532
--- a/test/puppeteer_utils.js
+++ b/test/puppeteer_utils.js
@@@ -4,7 -4,7 +4,7 @@@ import Apify from '../build/index'
const { utils: { log } } = Apify;
--/* global window */
++/* global window, document */
describe('Apify.utils.puppeteer', () => {
let ll;
@@@ -350,4 -347,4 +350,30 @@@
await browser.close();
}
});
++
++ it('infiniteScroll() works', async () => {
++ function isAtBottom() {
++ return (window.innerHeight + window.pageYOffset) >= document.body.offsetHeight;
++ }
++ const browser = await Apify.launchPuppeteer({ headless: true });
++ try {
++ const page = await browser.newPage();
++ let count = 0;
++ const content = Array(1000).fill(null).map(() => {
++ return `<div style="border: 1px solid black">Div number: ${count++}</div>`;
++ });
++ const contentHTML = `<html><body>${content}</body></html>`;
++ await page.setContent(contentHTML);
++
++ const before = await page.evaluate(isAtBottom);
++ expect(before).to.be.eql(false);
++
++ await Apify.utils.puppeteer.infiniteScroll(page, { waitForSecs: 0 });
++
++ const after = await page.evaluate(isAtBottom);
++ expect(after).to.be.eql(true);
++ } finally {
++ await browser.close();
++ }
++ });
});