Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.
TypeScript apifyautomationcrawlercrawlingheadlessheadless-chromejavascriptnodejsnpmplaywrightpuppeteerscraperscrapingtypescriptweb-crawlerweb-crawlingweb-scraping
Update changelog, bump default user agent
Ondra Urban2db78133a3d0
diff --git a/CHANGELOG.md b/CHANGELOG.mdindex 192bb6bfd..5976f575d 100644--- a/CHANGELOG.md+++ b/CHANGELOG.md@@ -1,6 +1,8 @@0.21.8 / 2020/10/07====================-- Fix an error in `PuppeteerCrawler` caused by `page.goto()` randomly returning `null`.+- Make `PuppeteerCrawler` safe against malformed Puppeteer responses.+- Update default user agent to Chrome 86+- Bump Puppeteer to 5.3.1 with Chromium 860.21.7 / 2020/10/03====================diff --git a/src/constants.js b/src/constants.jsindex 89fe4df25..56ba7f8bf 100644--- a/src/constants.js+++ b/src/constants.js@@ -5,7 +5,7 @@ import { ACTOR_EVENT_NAMES } from 'apify-shared/consts';* Last updated on 2020-05-22.*/// eslint-disable-next-line max-len-export const DEFAULT_USER_AGENT = 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/85.0.4183.83 Safari/537.36';+export const DEFAULT_USER_AGENT = 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/86.0.4240.75 Safari/537.36';/*** Exit codes for the actor process.