Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.
TypeScript apifyautomationcrawlercrawlingheadlessheadless-chromejavascriptnodejsnpmplaywrightpuppeteerscraperscrapingtypescriptweb-crawlerweb-crawlingweb-scraping
fixed lint comments.
Petr Patekb6b8f2fd7d4b
diff --git a/src/utils_request.js b/src/utils_request.jsindex 03bc158bf..5a5ef27f2 100644--- a/src/utils_request.js+++ b/src/utils_request.js@@ -6,8 +6,8 @@ import * as url from 'url';import { decompressStream } from 'iltorb';const FIREFOX_MOBILE_USER_AGENT = 'Mozilla/5.0 (Android 9.0; Mobile; rv:66.0) Gecko/66.0 Firefox/66.0';-const FIREFOX_DESKTOP_USER_AGENT = 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_3) AppleWebKit/605.1.15 (KHTML, like Gecko)' +- ' Version/11.1.1 Safari/605.1.15';+const FIREFOX_DESKTOP_USER_AGENT = 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_3) AppleWebKit/605.1.15 (KHTML, like Gecko)'+ + ' Version/11.1.1 Safari/605.1.15';/*** Sends a HTTP request and returns the response.