Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.
TypeScript apifyautomationcrawlercrawlingheadlessheadless-chromejavascriptnodejsnpmplaywrightpuppeteerscraperscrapingtypescriptweb-crawlerweb-crawlingweb-scraping
Bump version, update Changelog, build docs
Ondra Urbandc56eda3b053
diff --git a/CHANGELOG.md b/CHANGELOG.mdindex ed6eb7624..df77a907c 100644--- a/CHANGELOG.md+++ b/CHANGELOG.md@@ -1,7 +1,10 @@-xxx+0.14.1 / 2019-04-24===================+- Made UI and UX improvements to `LiveViewServer` functionality.+- `launchPuppeteerOptions.puppeteerModule` now supports `Object` (pre-required modules).- Removed `--enable-resource-load-scheduler=false` Chromium command line flag, it has no effect.See https://bugs.chromium.org/p/chromium/issues/detail?id=723233+- Fixed inconsistency in `prepareRequestFunction` of `CheerioCrawler`.0.14.0 / 2019-04-15===================diff --git a/docs/api/CheerioCrawler.md b/docs/api/CheerioCrawler.mdindex c1e224557..cad848968 100644--- a/docs/api/CheerioCrawler.md+++ b/docs/api/CheerioCrawler.md@@ -147,7 +147,10 @@ await crawler.run();request: Request}</code></pre><p> where the <a href="request"><code>Request</code></a> instance corresponds to the initialized request.</p>-<p> The function should always return <a href="request"><code>Request</code></a>.</p>+<p> The function should modify the properties of the passed <a href="request"><code>Request</code></a> instance+ in place because there are already earlier references to it. Making a copy and returning it from+ this function is therefore not supported, because it would create inconsistencies where+ different parts of SDK would have access to a different <a href="request"><code>Request</code></a> instance.</p></td></tr><tr><td><code>[options.handlePageTimeoutSecs]</code></td><td><code>Number</code></td><td><code>60</code></td></tr>diff --git a/docs/api/LiveViewServer.md b/docs/api/LiveViewServer.mdindex 6afa0de63..50b2f42d3 100644--- a/docs/api/LiveViewServer.md+++ b/docs/api/LiveViewServer.md@@ -8,6 +8,15 @@ title: LiveViewServerthat provides a simple frontend to viewing the captured snapshots. A snapshot consists of threepieces of information, the currently opened URL, the content of the page (HTML) and its screenshot.+```json+{+ "pageUrl": "https://www.example.com",+ "htmlContent": "<html><body> ....",+ "screenshotIndex": 3,+ "createdAt": "2019-04-18T11:50:40.060Z"+}+```+`LiveViewServer` is useful when you want to be able to inspect the current browser status on demand.When no client is connected, the webserver consumes very low resources so it should have a closeto zero impact on performance. Only once a client connects the server will start serving snapshots.@@ -32,6 +41,7 @@ a better view into the browser, including DevTools, but `LiveViewServer` works t* [`.start()`](#LiveViewServer+start) ⇒ <code>Promise</code>* [`.stop()`](#LiveViewServer+stop) ⇒ <code>Promise</code>* [`.serve(page)`](#LiveViewServer+serve) ⇒ <code>Promise</code>+ * [`.getLastSnapshot()`](#LiveViewServer+getLastSnapshot) ⇒ <code>Object</code>* [`.isRunning()`](#LiveViewServer+isRunning) ⇒ <code>boolean</code>* [`.hasClients()`](#LiveViewServer+hasClients) ⇒ <code>boolean</code>@@ -64,6 +74,20 @@ a better view into the browser, including DevTools, but `LiveViewServer` works t<tr><td colspan="3"><p>Limits the number of screenshots storedby the server. This is to prevent using up too much disk space.</p>+</td></tr><tr>+<td><code>[options.snapshotTimeoutSecs]</code></td><td><code>number</code></td><td><code>3</code></td>+</tr>+<tr>+<td colspan="3"><p>If a snapshot is not made within the timeout,+ its creation will be aborted. This is to prevent+ pages from being hung up by a stalled screenshot.</p>+</td></tr><tr>+<td><code>[options.maxSnapshotFrequencySecs]</code></td><td><code>number</code></td><td><code>2</code></td>+</tr>+<tr>+<td colspan="3"><p>Use this parameter to further decrease the resource consumption+ of <code>LiveViewServer</code> by limiting the frequency at which it'll+ serve snapshots.</p></td></tr></tbody></table><a name="LiveViewServer+start"></a>@@ -81,19 +105,12 @@ will not be terminated, but the server will not prevent a process exit.<a name="LiveViewServer+serve"></a>## `liveViewServer.serve(page)` ⇒ <code>Promise</code>-Serves the snapshot to all connected clients.--```json-{- "pageUrl": "https://www.example.com",- "htmlContent": "<html><body> ....",- "screenshotIndex": 3-}-```-+Serves a snapshot to all connected clients.Screenshots are not served directly, only their index numberwhich is used by client to retrieve the screenshot.+Will time out and throw in `options.snapshotTimeoutSecs`.+<table><thead><tr>@@ -107,6 +124,9 @@ which is used by client to retrieve the screenshot.<tr></tr></tbody></table>+<a name="LiveViewServer+getLastSnapshot"></a>++## `liveViewServer.getLastSnapshot()` ⇒ <code>Object</code><a name="LiveViewServer+isRunning"></a>## `liveViewServer.isRunning()` ⇒ <code>boolean</code>diff --git a/docs/examples/PuppeteerCrawler.md b/docs/examples/PuppeteerCrawler.mdindex 23b40fd81..f2b348500 100644--- a/docs/examples/PuppeteerCrawler.md+++ b/docs/examples/PuppeteerCrawler.md@@ -27,8 +27,10 @@ Apify.main(async () => {requestQueue,// Here you can set options that are passed to the Apify.launchPuppeteer() function.- // For example, you can set "slowMo" to slow down Puppeteer operations to simplify debugging- launchPuppeteerOptions: { slowMo: 500 },+ launchPuppeteerOptions: {+ // For example, by adding "slowMo" you'll slow down Puppeteer operations to simplify debugging+ // slowMo: 500,+ },// Stop crawling after several pagesmaxRequestsPerCrawl: 10,diff --git a/docs/typedefs/LaunchPuppeteerOptions.md b/docs/typedefs/LaunchPuppeteerOptions.mdindex 80096d6ba..7fea7492e 100644--- a/docs/typedefs/LaunchPuppeteerOptions.md+++ b/docs/typedefs/LaunchPuppeteerOptions.md@@ -73,10 +73,13 @@ function and in addition, all the options available below.The identifier can only contain the following characters: <code>0-9</code>, <code>a-z</code>, <code>A-Z</code>, <code>"."</code>, <code>"_"</code> and <code>"~"</code>.Only applied if the <code>useApifyProxy</code> option is <code>true</code>.</p></td></tr><tr>-<td><code>[puppeteerModule]</code></td><td><code>String</code></td><td></td>+<td><code>[puppeteerModule]</code></td><td><code>string</code> | <code>Object</code></td><td></td></tr><tr>-<td colspan="3"><p>Require path to a module to be used instead of default <code>puppeteer</code>. This enables usage- of various Puppeteer wrappers such as <code>puppeteer-extra</code>.</p>+<td colspan="3"><p>Either a require path (<code>string</code>) to a package to be used instead of default <code>puppeteer</code>,+ or an already required module (<code>Object</code>). This enables usage of various Puppeteer+ wrappers such as <code>puppeteer-extra</code>.</p>+<p> Take caution, because it can cause all kinds of unexpected errors and weird behavior.+ Apify SDK is not tested with any other library besides <code>puppeteer</code> itself.</p></td></tr></tbody></table>diff --git a/package.json b/package.jsonindex 27dae7931..626634d3b 100644--- a/package.json+++ b/package.json@@ -1,6 +1,6 @@{"name": "apify",- "version": "0.14.0",+ "version": "0.14.1","description": "The scalable web crawling and scraping library for JavaScript. Enables development of data extraction and web automation jobs (not only) with headless Chrome and Puppeteer.","engines": {"node": ">=8.14.0"@@ -88,7 +88,7 @@"jsdoc": "^3.5.5","jsdoc-export-default-interop": "^0.3.1","mocha": "^5.2.0",- "nyc": "^13.3.0",+ "nyc": "^14.0.0","portastic": "^1.0.1","proxy": "^0.2.4","sinon": "^7.1.1",@@ -97,7 +97,7 @@"why-is-node-running": "^2.0.3"},"optionalDependencies": {- "puppeteer": "1.13.0",+ "puppeteer": "1.14.0","selenium-webdriver": "^3.6.0"}}