Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.
TypeScript apifyautomationcrawlercrawlingheadlessheadless-chromejavascriptnodejsnpmplaywrightpuppeteerscraperscrapingtypescriptweb-crawlerweb-crawlingweb-scraping
Bump version, update Changelog, build docs
Ondra Urban4d4b72159245
diff --git a/CHANGELOG.md b/CHANGELOG.mdindex e063475e6..84d01440a 100644--- a/CHANGELOG.md+++ b/CHANGELOG.md@@ -1,3 +1,9 @@+0.14.4 / 2019-05-06+===================+- Added the `stealth` option `launchPuppeteerOptions` which decreases headless browser detection chance.+- DEPRECATED: `Apify.utils.puppeteer.hideWebDriver` use `launchPuppeteerOptions.stealth` instead.+- `CheerioCrawler` now parses HTML using streams. This improves performance and memory usage in most cases.+0.14.3 / 2019-05-06===================- Better logging of errors in `Apify.main()`diff --git a/docs/api/puppeteer.md b/docs/api/puppeteer.mdindex 5b4b8edbf..d9d59a602 100644--- a/docs/api/puppeteer.md+++ b/docs/api/puppeteer.md@@ -27,7 +27,6 @@ await puppeteer.injectJQuery(page);* [`.addInterceptRequestHandler`](#puppeteer.addInterceptRequestHandler) ⇒ <code>Promise</code>* [`.removeInterceptRequestHandler`](#puppeteer.removeInterceptRequestHandler) ⇒ <code>Promise</code>* [`.gotoExtended`](#puppeteer.gotoExtended) ⇒ <code>Promise<Response></code>- * [`.hideWebDriver(page)`](#puppeteer.hideWebDriver) ⇒ <code>Promise</code>* [`.injectFile(page, filePath, [options])`](#puppeteer.injectFile) ⇒ <code>Promise</code>* [`.injectJQuery(page)`](#puppeteer.injectJQuery) ⇒ <code>Promise</code>* [`.injectUnderscore(page)`](#puppeteer.injectUnderscore) ⇒ <code>Promise</code>@@ -156,26 +155,6 @@ request parameter that must be an instance of Apify.Request class.<td colspan="3"><p>Custom options for <code>page.goto()</code>.</p></td></tr></tbody></table>-<a name="puppeteer.hideWebDriver"></a>--## `puppeteer.hideWebDriver(page)` ⇒ <code>Promise</code>-Hides certain Puppeteer fingerprints from the page, in order to help avoid detection of the crawler.-The function should be called on a newly-created page object before navigating to the target crawled page.--<table>-<thead>-<tr>-<th>Param</th><th>Type</th>-</tr>-</thead>-<tbody>-<tr>-<td><code>page</code></td><td><code>Page</code></td>-</tr>-<tr>-<td colspan="3"><p>Puppeteer <a href="https://pptr.dev/#?product=Puppeteer&show=api-class-page" target="_blank"><code>Page</code></a> object.</p>-</td></tr></tbody>-</table><a name="puppeteer.injectFile"></a>## `puppeteer.injectFile(page, filePath, [options])` ⇒ <code>Promise</code>diff --git a/docs/typedefs/LaunchPuppeteerOptions.md b/docs/typedefs/LaunchPuppeteerOptions.mdindex ebec20eec..2f9352c41 100644--- a/docs/typedefs/LaunchPuppeteerOptions.md+++ b/docs/typedefs/LaunchPuppeteerOptions.md@@ -85,7 +85,7 @@ function and in addition, all the options available below.<td><code>[stealth]</code></td><td><code>boolean</code></td><td></td></tr><tr>-<td colspan="3"><p>This setting hides most of the know properties that identifies the headless Chrome and makes it nearly undetectable.+<td colspan="3"><p>This setting hides most of the known properties that identify headless Chrome and makes it nearly undetectable.It is recommended to use it together with the <code>useChrome</code> set to <code>true</code>.</p></td></tr><tr><td><code>[stealthOptions]</code></td><td><code><a href="../typedefs/stealthoptions">StealthOptions</a></code></td><td></td>diff --git a/package.json b/package.jsonindex b8fa207d9..72cfb67dc 100644--- a/package.json+++ b/package.json@@ -1,6 +1,6 @@{"name": "apify",- "version": "0.14.3",+ "version": "0.14.4","description": "The scalable web crawling and scraping library for JavaScript. Enables development of data extraction and web automation jobs (not only) with headless Chrome and Puppeteer.","engines": {"node": ">=8.14.0"diff --git a/src/puppeteer.js b/src/puppeteer.jsindex 96b55a31c..c3406d9c4 100644--- a/src/puppeteer.js+++ b/src/puppeteer.js@@ -6,7 +6,7 @@ import { ENV_VARS } from 'apify-shared/consts';import { DEFAULT_USER_AGENT } from './constants';import { getTypicalChromeExecutablePath, isAtHome } from './utils';import { getApifyProxyUrl } from './actor';-import applyStealthFromBrowser from './stealth/stealth';+import applyStealthToBrowser from './stealth/stealth';/* global process, require */@@ -70,7 +70,7 @@ const LAUNCH_PUPPETEER_DEFAULT_VIEWPORT = {* Take caution, because it can cause all kinds of unexpected errors and weird behavior.* Apify SDK is not tested with any other library besides `puppeteer` itself.* @property {boolean} [stealth]- * This setting hides most of the know properties that identifies the headless Chrome and makes it nearly undetectable.+ * This setting hides most of the known properties that identify headless Chrome and makes it nearly undetectable.* It is recommended to use it together with the `useChrome` set to `true`.* @property {StealthOptions} [stealthOptions]* Using this configuration, you can disable some of the hiding tricks.@@ -251,7 +251,7 @@ export const launchPuppeteer = async (options = {}) => {// Add stealthif (optsCopy.stealth) {- browser = applyStealthFromBrowser(browser, optsCopy.stealthOptions);+ browser = applyStealthToBrowser(browser, optsCopy.stealthOptions);}log.info('Launching Puppeteer', _.omit(optsCopy, LAUNCH_PUPPETEER_LOG_OMIT_OPTS));return browser;diff --git a/src/puppeteer_utils.js b/src/puppeteer_utils.jsindex 629d27812..19b816d6b 100644--- a/src/puppeteer_utils.js+++ b/src/puppeteer_utils.js@@ -25,8 +25,10 @@ const MAX_INJECT_FILE_CACHE_SIZE = 10;* Puppeteer <a href="https://pptr.dev/#?product=Puppeteer&show=api-class-page" target="_blank"><code>Page</code></a> object.* @return {Promise}* @memberOf puppeteer+ * @ignore*/const hideWebDriver = async (page) => {+ log.deprecated('Apify.utils.puppeteer.hideWebDriver() is deprecated. Use launchPuppeteerOptions.stealth instead.');checkParamOrThrow(page, 'page', 'Object');await page.evaluateOnNewDocument(() => {