Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.
TypeScript apifyautomationcrawlercrawlingheadlessheadless-chromejavascriptnodejsnpmplaywrightpuppeteerscraperscrapingtypescriptweb-crawlerweb-crawlingweb-scraping
Remove Puppeteer pipes, fix docs
Ondra Urbanb3222030eca2
diff --git a/CHANGELOG.md b/CHANGELOG.mdindex f0557c798..f9af67f92 100644--- a/CHANGELOG.md+++ b/CHANGELOG.md@@ -10,6 +10,7 @@ NEXT- Add `Apify.utils.puppeteer.saveSnapshot()` helper to quickly save HTML and screenshot of a page.- Add the ability to pass `got` supported options to `requestOptions` in `CheerioCrawler`thus supporting things such as `cookieJar` again.+- Switch Puppeteer to web socket again due to suspected `pipe` errors.- Fix an issue where some encodings were not correctly parsed in `CheerioCrawler`.- Fix parsing bad Content-Type headers for `CheerioCrawler`.- Fix custom headers not being correctly applied in `Apify.utils.requestAsBrowser()`.diff --git a/docs/api/CheerioCrawler.md b/docs/api/CheerioCrawler.mdindex 5979f83e6..0fe49ae90 100644--- a/docs/api/CheerioCrawler.md+++ b/docs/api/CheerioCrawler.md@@ -102,9 +102,9 @@ await crawler.run();<pre><code>{$: Cheerio, // the Cheerio object with parsed HTMLbody: String|Buffer // the request body of the web page- json: Object, // the parsed object from JSON string if the response contains the content type application/json+ json: Object, // parsed JSON when Content-Type: application/jsonrequest: Request,- contentType: Object, // Parsed `Content-Type` header+ contentType: Object, // Parsed Content-Type header: { type, encoding }response: Object // An instance of Node's http.IncomingMessage object,autoscaledPool: AutoscaledPool}</code></pre><p> Type of <code>body</code> depends on web page <code>Content-Type</code> header.</p>diff --git a/src/crawlers/cheerio_crawler.js b/src/crawlers/cheerio_crawler.jsindex 32bc4fc0b..9e879b63c 100644--- a/src/crawlers/cheerio_crawler.js+++ b/src/crawlers/cheerio_crawler.js@@ -124,9 +124,9 @@ const DEFAULT_OPTIONS = {* {* $: Cheerio, // the Cheerio object with parsed HTML* body: String|Buffer // the request body of the web page- * json: Object, // the parsed object from JSON string if the response contains the content type application/json+ * json: Object, // parsed JSON when Content-Type: application/json* request: Request,- * contentType: Object, // Parsed `Content-Type` header+ * contentType: Object, // Parsed Content-Type header: { type, encoding }* response: Object // An instance of Node's http.IncomingMessage object,* autoscaledPool: AutoscaledPool* }diff --git a/src/puppeteer.js b/src/puppeteer.jsindex 4cdd28025..9ecf3fbed 100644--- a/src/puppeteer.js+++ b/src/puppeteer.js@@ -260,8 +260,6 @@ export const launchPuppeteer = async (options = {}) => {optsCopy.args.push(`--user-agent=${userAgent}`);}- if (optsCopy.pipe !== false) optsCopy.pipe = true;-let browser;if (optsCopy.proxyUrl) {// The log for launching with proxyUrl is inside launchPuppeteerWithProxy