Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.
TypeScript apifyautomationcrawlercrawlingheadlessheadless-chromejavascriptnodejsnpmplaywrightpuppeteerscraperscrapingtypescriptweb-crawlerweb-crawlingweb-scraping
Improve docs and migration guide [skip ci]
Ondra Urban453bd005fcd1
diff --git a/MIGRATIONS.md b/MIGRATIONS.mdindex 69d5620e6..d0a622c07 100644--- a/MIGRATIONS.md+++ b/MIGRATIONS.md@@ -66,8 +66,38 @@ const browser = await Apify.launchPuppeteer({})```-### Usage with your own proxies:+#### PuppeteerCrawler warning+`ProxyConfiguration` integrates seamlessly with `PuppeteerCrawler`, but beware+wrong usage of `launchPuppeteerFunction`. It receives an options parameter+that should not be ignored. This is not new, it has always been like that,+but until now, only non-critical features were dependent on it, so you+might have been using it wrongly and never really noticed.++**Correct:**+```javascript+async function launchPuppeteerFunction(options) {+ const newOpts = {+ ...options,+ foo: 'bar',+ }+ // do some other things+ return Apify.launchPuppeteer(newOpts);+}+```+**Incorrect:**+```javascript+async function launchPuppeteerFunction() {+ const opts = {+ foo: 'bar',+ }+ // Because we ignored the options, correct parameters+ // will not make it to the browser. This prevents+ // proxyConfiguration from working correctly.+ return Apify.launchPuppeteer(opts);+}+```+### Usage with your own proxies:Using your own proxies was possible, but it was difficult to find where toenter your URLs to make use of them. Now it's the same as with Apify Proxy,you just add them to the `ProxyConfiguration`. As a bonus, your custom proxiesdiff --git a/src/puppeteer_pool.js b/src/puppeteer_pool.jsindex be4f08559..8a6c8dbdc 100644--- a/src/puppeteer_pool.js+++ b/src/puppeteer_pool.js@@ -102,12 +102,39 @@ class PuppeteerInstance { // TODO: this is in progress and it will be refactored* whole browser is closed too. This parameter defines a time limit between the last tab was opened and* before the browser is closed even if there are pending open tabs.* @property {LaunchPuppeteerFunction} [launchPuppeteerFunction]- * Overrides the default function to launch a new Puppeteer instance.- * The function must return a promise resolving to+ * Overrides the default function to launch a new Puppeteer instance. It must return a promise resolving to* [`Browser`](https://github.com/puppeteer/puppeteer/blob/master/docs/api.md#class-browser) instance.- * See the source code on- * [GitHub](https://github.com/apifytech/apify-js/blob/master/src/puppeteer_pool.js#L28)- * for the default implementation.+ *+ * The function receives one parameter that includes launch options, such as `proxyUrl` and other options+ * generated by the `PuppeteerPool` instance. You can extend or update this options object, but you must+ * not ignore it.+ *+ * **Correct:**+ * ```javascript+ * async function launchPuppeteerFunction(options) {+ * const newOpts = {+ * ...options,+ * foo: 'bar',+ * }+ *+ * // do some other things+ *+ * return Apify.launchPuppeteer(newOpts);+ * }+ * ```+ * **Incorrect:**+ * ```javascript+ * async function launchPuppeteerFunction() {+ * const opts = {+ * foo: 'bar',+ * }+ * // Because we ignored the options, correct parameters+ * // will not make it to the browser. Eg. this prevents+ * // proxyConfiguration from working correctly.+ * return Apify.launchPuppeteer(opts);+ * }+ * ```+ ** @property {LaunchPuppeteerOptions} [launchPuppeteerOptions]* Options used by {@link Apify#launchPuppeteer} to start new Puppeteer instances.* @property {boolean} [recycleDiskCache=false]