Skip to content

Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.

Update docs [skip ci]

Ondra Urban8d989ffec422
diff --git a/docs/api/puppeteer.md b/docs/api/puppeteer.md
index 668b15278..39244527a 100644
--- a/docs/api/puppeteer.md
+++ b/docs/api/puppeteer.md
@@ -438,14 +438,15 @@ const escapedHtml = await page.evaluate(() => {
Forces the Puppeteer browser tab to block loading URLs that match a provided pattern. This is useful to speed up crawling of websites, since it
reduces the amount of data that needs to be downloaded from the web, but it may break some websites or unexpectedly prevent loading of resources.
-If the `options.urlPatterns` parameter is not provided, by default the function blocks URLs that include these patterns:
+By default, the function will block all URLs including the following patterns:
```json
[".css", ".jpg", ".jpeg", ".png", ".svg", ".gif", ".woff", ".pdf", ".zip"]
```
-The defaults will be concatenated with the patterns you provide in `options.urlPatterns`. If you want to remove the defaults, use
-`options.includeDefaults: false`.
+If you want to extend this list further, use the `extraUrlPatterns` option, which will keep blocking the default patterns, as well as add your custom
+ones. If you would like to block only specific patterns, use the `urlPatterns` option, which will override the defaults and block only URLs with your
+custom patterns.
This function does not use Puppeteer's request interception and therefore does not interfere with browser cache. It's also faster than blocking
requests using interception, because the blocking happens directly in the browser without the round-trip to Node.js, but it does not provide the extra
diff --git a/src/puppeteer_utils.js b/src/puppeteer_utils.js
index 845b840ce..5aaad6721 100644
--- a/src/puppeteer_utils.js
+++ b/src/puppeteer_utils.js
@@ -193,15 +193,16 @@ const enqueueRequestsFromClickableElements = async (page, selector, purls, reque
* of data that needs to be downloaded from the web, but it may break some websites
* or unexpectedly prevent loading of resources.
*
- * If the `options.urlPatterns` parameter is not provided,
- * by default the function blocks URLs that include these patterns:
+ * By default, the function will block all URLs including the following patterns:
*
* ```json
* [".css", ".jpg", ".jpeg", ".png", ".svg", ".gif", ".woff", ".pdf", ".zip"]
* ```
*
- * The defaults will be concatenated with the patterns you provide in `options.urlPatterns`.
- * If you want to remove the defaults, use `options.includeDefaults: false`.
+ * If you want to extend this list further, use the `extraUrlPatterns` option,
+ * which will keep blocking the default patterns, as well as add your custom ones.
+ * If you would like to block only specific patterns, use the `urlPatterns` option,
+ * which will override the defaults and block only URLs with your custom patterns.
*
* This function does not use Puppeteer's request interception and therefore does not interfere
* with browser cache. It's also faster than blocking requests using interception,