Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.
TypeScript apifyautomationcrawlercrawlingheadlessheadless-chromejavascriptnodejsnpmplaywrightpuppeteerscraperscrapingtypescriptweb-crawlerweb-crawlingweb-scraping
Use `got-scraping@3` (#1118)
Szymon Marczaka31a2b111ff9
diff --git a/CHANGELOG.md b/CHANGELOG.mdindex 4c930c561..4964002c8 100644--- a/CHANGELOG.md+++ b/CHANGELOG.md@@ -2,6 +2,7 @@====================- **BREAKING IN EDGE CASES** - We removed `forceUrlEncoding` in `requestAsBrowser` because we found out that recent versions of the underlying HTTP client `got` already encode URLs and `forceUrlEncoding` could lead to weird behavior. We think of this as fixing a bug, so we're not bumping the major version.- Limit `handleRequestTimeoutMillis` to max valid value to prevent Node.js fallback to `1`.+- Use `got-scraping@^3.0.0`2.0.2 / 2021/08/12====================diff --git a/package.json b/package.jsonindex 5cdfc5f39..dd06d4ebf 100644--- a/package.json+++ b/package.json@@ -67,7 +67,7 @@"content-type": "^1.0.4","express": "^4.17.1","fs-extra": "^10.0.0",- "got-scraping": "2.1.3",+ "got-scraping": "^3.0.0","htmlparser2": "^6.1.0","iconv-lite": "^0.6.3","jquery": "^3.6.0",diff --git a/src/utils_request.js b/src/utils_request.jsindex fb33c5aea..e4697228b 100644--- a/src/utils_request.js+++ b/src/utils_request.js@@ -1,4 +1,4 @@-import * as gotScraping from 'got-scraping';+import { gotScraping } from 'got-scraping';import ow from 'ow';import log from './utils_log';