Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.
TypeScript apifyautomationcrawlercrawlingheadlessheadless-chromejavascriptnodejsnpmplaywrightpuppeteerscraperscrapingtypescriptweb-crawlerweb-crawlingweb-scraping
Updated DEFAULT_USER_AGENT and LAUNCH_PUPPETEER_DEFAULT_VIEWPORT constants
Jan Curn1460be3e5efc
diff --git a/src/constants.js b/src/constants.jsindex 976c7b366..96b590bf7 100644--- a/src/constants.js+++ b/src/constants.js@@ -4,7 +4,7 @@ import { ACTOR_EVENT_NAMES } from 'apify-shared/consts';* The default user agent used by `Apify.launchPuppeteer`.* Last updated on 2018-09-10.*/-export const DEFAULT_USER_AGENT = 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/69.0.3497.100 Safari/537.36'; // eslint-disable-line max-len+export const DEFAULT_USER_AGENT = 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.102 Safari/537.36'; // eslint-disable-line max-len/*** Exit codes for the actor process.diff --git a/src/puppeteer.js b/src/puppeteer.jsindex d438c00fd..4f5b3d5bc 100644--- a/src/puppeteer.js+++ b/src/puppeteer.js@@ -16,8 +16,8 @@ const LAUNCH_PUPPETEER_LOG_OMIT_OPTS = [];const LAUNCH_PUPPETEER_DEFAULT_VIEWPORT = {- width: 1200,- height: 900,+ width: 1366,+ height: 768,};/**