Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.
TypeScript apifyautomationcrawlercrawlingheadlessheadless-chromejavascriptnodejsnpmplaywrightpuppeteerscraperscrapingtypescriptweb-crawlerweb-crawlingweb-scraping
Updated docs and README
Jan Curn3263cc87b6df
diff --git a/README.md b/README.mdindex 21a244971..45112c299 100644--- a/README.md+++ b/README.md@@ -307,7 +307,7 @@ All the examples can be found in the [examples](https://github.com/apifytech/apiin the repository.To run the examples, just copy them into the directory where you installed the Apify SDK using-`npm install apify` and then run them by calling:+`npm install apify` and then run them, for example, by calling:```node basic_crawler.jsdiff --git a/src/actor.js b/src/actor.jsindex 9d783392a..111696591 100644--- a/src/actor.js+++ b/src/actor.js@@ -208,7 +208,7 @@ export const main = (userFunc) => {/**- * Runs an actor on the Apify platform using the current user account (given by the `APIFY_TOKEN` environment variable),+ * Runs an actor on the Apify platform using the current user account (determined by the `APIFY_TOKEN` environment variable),* waits for the actor to finish and fetches its output.** By passing the `waitSecs` option you can reduce the maximum amount of time to wait for the run to finish.diff --git a/src/dataset.js b/src/dataset.jsindex 7ab3736f3..c034bc803 100644--- a/src/dataset.js+++ b/src/dataset.js@@ -96,12 +96,20 @@ export const chunkBySize = (items, limitBytes) => {};/**+ * Represents one page of data items from the {@linkcode Dataset}.+ * For more details, see {@linkcode Dataset#getData|Dataset.getData()}.+ ** @typedef {Object} PaginationList- * @property {Array} items - List of returned objects- * @property {Number} total - Total number of object- * @property {Number} offset - Number of Request objects that was skipped at the start.- * @property {Number} count - Number of returned objects- * @property {Number} limit - Requested limit+ * @property {Array} items+ * Array of returned items.+ * @property {Number} total+ * Total number of object.+ * @property {Number} offset+ * Number of Request objects that was skipped at the start.+ * @property {Number} count+ * Number of returned objects.+ * @property {Number} limit+ * Requested limit on the number of items.*//**diff --git a/src/key_value_store.js b/src/key_value_store.jsindex 43c2e6ea8..db1987465 100644--- a/src/key_value_store.js+++ b/src/key_value_store.js@@ -183,7 +183,7 @@ export class KeyValueStore {** @param {String} key* Key of the record.- * @returns {Promise}+ * @returns {Promise<Object>}* Returns a promise that resolves to an object, string* or <a href="https://nodejs.org/api/buffer.html"><code>Buffer</code></a>, depending* on the MIME content type of the record.@@ -491,7 +491,7 @@ export const openKeyValueStore = (storeIdOrName) => {** @param {String} key* Unique record key.- * @returns {Promise}+ * @returns {Promise<Object>}* Returns a promise that resolves once the record is stored.* @see {@link KeyValueStore}*diff --git a/src/puppeteer.js b/src/puppeteer.jsindex 8744e5ee9..46d2ce2da 100644--- a/src/puppeteer.js+++ b/src/puppeteer.js@@ -16,36 +16,49 @@ const LAUNCH_PUPPETEER_LOG_OMIT_OPTS = [];/**+ * Represents options passed to the {@linkcode Apify#launchPuppeteer|Apify.launchPuppeteer()}+ * function.* @typedef {Object} LaunchPuppeteerOptions- * @property {String} [opts.proxyUrl] URL to a HTTP proxy server. It must define the port number,- * and it might also contain proxy username and password.- * For example: `http://bob:[email protected]:1234`.- * @property {String} [opts.userAgent] HTTP `User-Agent` header used by the browser.- * If not provided, the function sets `User-Agent` to a reasonable default- * to reduce the chance of detection of the crawler.- * @property {Boolean} [opts.useChrome=false] If `true` and `opts.executablePath` is not set,- * Puppeteer will launch full Google Chrome browser available on the machine- * rather than the bundled Chromium. The path to Chrome executable- * is taken from the `APIFY_CHROME_EXECUTABLE_PATH` environment variable if provided,- * or defaults to the typical Google Chrome executable location specific for the operating system.- * By default, this option is `false`.- * @property {Boolean} [opts.useApifyProxy=false] If set to `true`, Puppeteer will be configured to use- * <a href="https://my.apify.com/proxy" target="_blank">Apify Proxy</a> for all connections.- * For more information, see the <a href="https://www.apify.com/docs/proxy">documentation</a>- * @property {String[]} [opts.apifyProxyGroups] An array of proxy groups to be used- * by the <a href="https://www.apify.com/docs/proxy" target="_blank">Apify Proxy</a>.- * Only applied if the `useApifyProxy` option is `true`.- * @property {String} [opts.apifyProxySession] Apify Proxy session identifier to be used by all the Chrome browsers.- * All HTTP requests going through the proxy with the same session identifier- * will use the same target proxy server (i.e. the same IP address).- * The identifier can only contain the following characters: `0-9`, `a-z`, `A-Z`, `"."`, `"_"` and `"~"`.- * Only applied if the `useApifyProxy` option is `true`.- * @property {Boolean} [opts.liveView=false] If set to `true`, a PuppeteerLiveViewServer will be started to enable- * screenshot and html capturing of visited pages using PuppeteerLiveViewBrowser.- * @property {Object} [opts.liveViewOptions] Settings for PuppeteerLiveViewBrowser started using launchPuppeteer().- * @property {String} [opts.liveViewOptions.id] Custom ID of a browser instance.- * @property {Number} [opts.liveViewOptions.screenshotTimeoutMillis] Time in milliseconds before a screenshot capturing- * will time out and the actor continues with execution. Screenshot capturing pauses execution within the given page.+ * @property {String} [opts.proxyUrl]+ * URL to a HTTP proxy server. It must define the port number,+ * and it might also contain proxy username and password.+ *+ * For example: `http://bob:[email protected]:1234`.+ * @property {String} [opts.userAgent]+ * The `User-Agent` HTTP header used by the browser.+ * If not provided, the function sets `User-Agent` to a reasonable default+ * to reduce the chance of detection of the crawler.+ * @property {Boolean} [opts.useChrome=false]+ * If `true` and `opts.executablePath` is not set,+ * Puppeteer will launch full Google Chrome browser available on the machine+ * rather than the bundled Chromium. The path to Chrome executable+ * is taken from the `APIFY_CHROME_EXECUTABLE_PATH` environment variable if provided,+ * or defaults to the typical Google Chrome executable location specific for the operating system.+ * By default, this option is `false`.+ * @property {Boolean} [opts.useApifyProxy=false]+ * If set to `true`, Puppeteer will be configured to use+ * <a href="https://my.apify.com/proxy" target="_blank">Apify Proxy</a> for all connections.+ * For more information, see the <a href="https://www.apify.com/docs/proxy">documentation</a>+ * @property {String[]} [opts.apifyProxyGroups]+ * An array of proxy groups to be used+ * by the <a href="https://www.apify.com/docs/proxy" target="_blank">Apify Proxy</a>.+ * Only applied if the `useApifyProxy` option is `true`.+ * @property {String} [opts.apifyProxySession]+ * Apify Proxy session identifier to be used by all the Chrome browsers.+ * All HTTP requests going through the proxy with the same session identifier+ * will use the same target proxy server (i.e. the same IP address).+ * The identifier can only contain the following characters: `0-9`, `a-z`, `A-Z`, `"."`, `"_"` and `"~"`.+ * Only applied if the `useApifyProxy` option is `true`.+ * @property {Boolean} [opts.liveView=false]+ * If set to `true`, a PuppeteerLiveViewServer will be started to enable+ * screenshot and html capturing of visited pages using PuppeteerLiveViewBrowser.+ * @property {Object} [opts.liveViewOptions]+ * Settings for `PuppeteerLiveViewBrowser` started using `launchPuppeteer()`.+ * @property {String} [opts.liveViewOptions.id]+ * Custom ID of a browser instance in live view.+ * @property {Number} [opts.liveViewOptions.screenshotTimeoutMillis]+ * Time in milliseconds before a screenshot capturing+ * will time out and the actor continues with execution. Screenshot capturing pauses execution within the given page.*//**