Skip to content

Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.

Remove remnants of HTML from docs [skip ci]

Ondra Urbanb12dc6acaf97
diff --git a/src/events.js b/src/events.js
index eecdaed6b..ab9da2eb6 100644
--- a/src/events.js
+++ b/src/events.js
@@ -39,54 +39,24 @@ let persistStateInterval = null;
* });
* ```
*
- * The following table shows all currently emitted events:
- * <table>
- * <thead>
- * <tr>
- * <th>Event name</th>
- * <th>Data</th>
- * </thead>
- * <tbody>
- * <tr>
- * <td><code>cpuInfo</code></td>
- * <td><code>{ "isCpuOverloaded": Boolean }</code></td>
- * </tr>
- * <tr>
- * <td colspan="2">
- * The event is emitted approximately every second
- * and it indicates whether the actor is using the maximum of available CPU resources.
- * If that's the case, the actor should not add more workload.
- * For example, this event is used by the {@link AutoscaledPool} class.
- * </td>
- * </tr>
- * <tr>
- * <td><code>migrating</code></td>
- * <td>None</td>
- * </tr>
- * <tr>
- * <td colspan="2">
- * Emitted when the actor running on the Apify platform is going to be migrated to another worker server soon.
- * You can use it to persist the state of the actor and abort the run, to speed up migration.
- * For example, this is used by the {@link RequestList} class.
- * </td>
- * </tr>
- * <tr>
- * <td><code>persistState</code></td>
- * <td><code>{ "isMigrating": Boolean }</code></td>
- * </tr>
- * <tr>
- * <td colspan="2">
- * Emitted in regular intervals (by default 60 seconds) to notify all components of Apify SDK that it is time to persist
- * their state, in order to avoid repeating all work when the actor restarts.
- * This event is automatically emitted together with the <code>migrating</code> event,
- * in which case the <code>isMigrating</code> flag is set to <code>true</code>. Otherwise the flag is <code>false</code>.
- * <br><br>
- * Note that the <code>persistState</code> event is provided merely for user convenience,
- * you can achieve the same effect using <code>setInterval()</code> and listening for the <code>migrating</code> event.
- * </td>
- * </tr>
- * </tbody>
- * </table>
+ * The following events are emitted:
+ *
+ * - `cpuInfo`: `{ "isCpuOverloaded": Boolean }`
+ * The event is emitted approximately every second
+ * and it indicates whether the actor is using the maximum of available CPU resources.
+ * If that's the case, the actor should not add more workload.
+ * For example, this event is used by the {@link AutoscaledPool} class.
+ * - `migrating`: `void`
+ * Emitted when the actor running on the Apify platform is going to be migrated to another worker server soon.
+ * You can use it to persist the state of the actor and abort the run, to speed up migration.
+ * For example, this is used by the {@link RequestList} class.
+ * - `persistState`: `{ "isMigrating": Boolean }`
+ * Emitted in regular intervals (by default 60 seconds) to notify all components of Apify SDK that it is time to persist
+ * their state, in order to avoid repeating all work when the actor restarts.
+ * This event is automatically emitted together with the `migrating` event,
+ * in which case the `isMigrating` flag is set to `true`. Otherwise the flag is `false`.
+ * Note that the `persistState` event is provided merely for user convenience,
+ * you can achieve the same effect using `setInterval()` and listening for the `migrating` event.
*
* @memberof module:Apify
* @name events
diff --git a/src/key_value_store.js b/src/key_value_store.js
index 3dd809c96..c75a250d0 100644
--- a/src/key_value_store.js
+++ b/src/key_value_store.js
@@ -252,12 +252,9 @@ export class KeyValueStore {
* of the following characters: `a`-`z`, `A`-`Z`, `0`-`9` and `!-_.'()`
* @param {(Object|string|Buffer|null)} value
* Record data, which can be one of the following values:
- * <ul>
- * <li>If `null`, the record in the key-value store is deleted.</li>
- * <li>If no `options.contentType` is specified, `value` can be any JavaScript object and it will be stringified to JSON.</li>
- * <li>If `options.contentType` is specified, `value` is considered raw data and it must be either a `String`
- * or [`Buffer`](https://nodejs.org/api/buffer.html).</li>
- * </ul>
+ * - If `null`, the record in the key-value store is deleted.
+ * - If no `options.contentType` is specified, `value` can be any JavaScript object and it will be stringified to JSON.
+ * - If `options.contentType` is set, `value` is taken as is and it must be a `String` or [`Buffer`](https://nodejs.org/api/buffer.html).
* For any other value an error will be thrown.
* @param {Object} [options]
* @param {string} [options.contentType]
@@ -671,12 +668,9 @@ export const getValue = async (key) => {
* Unique record key.
* @param {object} value
* Record data, which can be one of the following values:
- * <ul>
- * <li>If `null`, the record in the key-value store is deleted.</li>
- * <li>If no `options.contentType` is specified, `value` can be any JavaScript object and it will be stringified to JSON.</li>
- * <li>If `options.contentType` is specified, `value` is considered raw data and it must be a `String`
- * or [`Buffer`](https://nodejs.org/api/buffer.html).</li>
- * </ul>
+ * - If `null`, the record in the key-value store is deleted.
+ * - If no `options.contentType` is specified, `value` can be any JavaScript object and it will be stringified to JSON.
+ * - If `options.contentType` is set, `value` is taken as is and it must be a `String` or [`Buffer`](https://nodejs.org/api/buffer.html).
* For any other value an error will be thrown.
* @param {Object} [options]
* @param {string} [options.contentType]
diff --git a/src/pseudo_url.js b/src/pseudo_url.js
index 8033f0c7d..148d57318 100644
--- a/src/pseudo_url.js
+++ b/src/pseudo_url.js
@@ -65,11 +65,9 @@ const parsePurl = (purl) => {
*
* For example, a PURL `http://www.example.com/pages/[(\w|-)*]` will match all of the following URLs:
*
- * <ul>
- * <li><code>http://www.example.com/pages/</code></li>
- * <li><code>http://www.example.com/pages/my-awesome-page</code></li>
- * <li><code>http://www.example.com/pages/something</code></li>
- * </ul>
+ * - `http://www.example.com/pages/`
+ * - `http://www.example.com/pages/my-awesome-page`
+ * - `http://www.example.com/pages/something`
*
* Be careful to correctly escape special characters in the pseudo-URL string.
* If either `[` or `]` is part of the normal query string, it must be encoded as `[\x5B]` or `[\x5D]`,
diff --git a/src/puppeteer.js b/src/puppeteer.js
index ff3f2288d..012b5f54f 100644
--- a/src/puppeteer.js
+++ b/src/puppeteer.js
@@ -164,38 +164,26 @@ const getPuppeteerOrThrow = (puppeteerModule = 'puppeteer') => {
*
* The `launchPuppeteer()` function alters the following Puppeteer options:
*
- * <ul>
- * <li>
- * Passes the setting from the <code>APIFY_HEADLESS</code> environment variable to the <code>headless</code> option,
- * unless it was already defined by the caller or <code>APIFY_XVFB</code> environment variable is set to <code>1</code>.
- * Note that Apify Actor cloud platform automatically sets <code>APIFY_HEADLESS=1</code> to all running actors.
- * </li>
- * <li>
- * Takes the <code>proxyUrl</code> option, validates it and adds it to <code>args</code> as <code>--proxy-server=XXX</code>.
- * The proxy URL must define a port number and have one of the following schemes: <code>http://</code>,
- * <code>https://</code>, <code>socks4://</code> or <code>socks5://</code>.
- * If the proxy is HTTP (i.e. has the <code>http://</code> scheme) and contains username or password,
- * the <code>launchPuppeteer</code> functions sets up an anonymous proxy HTTP
- * to make the proxy work with headless Chrome. For more information, read the
- * <a href="https://blog.apify.com/how-to-make-headless-chrome-and-puppeteer-use-a-proxy-server-with-authentication-249a21a79212"
- * target="_blank">blog post about proxy-chain library</a>.
- * </li>
- * <li>
- * If <code>options.useApifyProxy</code> is <code>true</code> then the function generates a URL of
- * [Apify Proxy](https://docs.apify.com/proxy)
- * based on <code>options.apifyProxyGroups</code> and <code>options.apifyProxySession</code> and passes it as <code>options.proxyUrl</code>.
- * </li>
- * <li>
- * The function adds <code>--no-sandbox</code> to <code>args</code> to enable running
- * headless Chrome in a Docker container on the Apify platform.
- * </li>
- * <li>
- * Sets <code>defaultViewport</code> Puppeteer option (if not already set)
- * to a more reasonable default for screenshots and debugging.
- * You can set <code>options.defaultViewport</code> to <code>null</code> if you prefer to let Puppeteer
- * choose the default viewport size.
- * </li>
- * </ul>
+ * - Passes the setting from the `APIFY_HEADLESS` environment variable to the `headless` option,
+ * unless it was already defined by the caller or `APIFY_XVFB` environment variable is set to `1`.
+ * Note that Apify Actor cloud platform automatically sets `APIFY_HEADLESS=1` to all running actors.
+ * - Takes the `proxyUrl` option, validates it and adds it to `args` as `--proxy-server=XXX`.
+ * The proxy URL must define a port number and have one of the following schemes: `http://`,
+ * `https://`, `socks4://` or `socks5://`.
+ * If the proxy is HTTP (i.e. has the `http://` scheme) and contains username or password,
+ * the `launchPuppeteer` functions sets up an anonymous proxy HTTP
+ * to make the proxy work with headless Chrome. For more information, read the
+ * <a href="https://blog.apify.com/how-to-make-headless-chrome-and-puppeteer-use-a-proxy-server-with-authentication-249a21a79212"
+ * target="_blank">blog post about proxy-chain library</a>.
+ * - If `options.useApifyProxy` is `true` then the function generates a URL of
+ * [Apify Proxy](https://docs.apify.com/proxy)
+ * based on `options.apifyProxyGroups` and `options.apifyProxySession` and passes it as `options.proxyUrl`.
+ * - The function adds `--no-sandbox` to `args` to enable running
+ * headless Chrome in a Docker container on the Apify platform.
+ * - Sets `defaultViewport` Puppeteer option (if not already set)
+ * to a more reasonable default for screenshots and debugging.
+ * You can set `options.defaultViewport` to `null` if you prefer to let Puppeteer
+ * choose the default viewport size.
*
* To use this function, you need to have the [puppeteer](https://www.npmjs.com/package/puppeteer)
* NPM package installed in your project.