Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.
TypeScript apifyautomationcrawlercrawlingheadlessheadless-chromejavascriptnodejsnpmplaywrightpuppeteerscraperscrapingtypescriptweb-crawlerweb-crawlingweb-scraping
feat: use session token with got-scraping (#1122)
Szymon Marczakd3261ca53e67
diff --git a/CHANGELOG.md b/CHANGELOG.mdindex 4964002c8..b66e9da09 100644--- a/CHANGELOG.md+++ b/CHANGELOG.md@@ -1,8 +1,14 @@-2.0.3 / BETA+2.0.4 / 2021/08/23+====================+- Use `sessionToken` with `got-scraping`++2.0.3 / 2021/08/20====================- **BREAKING IN EDGE CASES** - We removed `forceUrlEncoding` in `requestAsBrowser` because we found out that recent versions of the underlying HTTP client `got` already encode URLs and `forceUrlEncoding` could lead to weird behavior. We think of this as fixing a bug, so we're not bumping the major version.- Limit `handleRequestTimeoutMillis` to max valid value to prevent Node.js fallback to `1`.-- Use `got-scraping@^3.0.0`+- Use `got-scraping@^3.0.1`+- Disable SSL validation on MITM proxie+- Limit `handleRequestTimeoutMillis` to max valid value2.0.2 / 2021/08/12====================diff --git a/package.json b/package.jsonindex b7ecacdb5..dbdc47386 100644--- a/package.json+++ b/package.json@@ -1,6 +1,6 @@{"name": "apify",- "version": "2.0.3",+ "version": "2.0.4","description": "The scalable web crawling and scraping library for JavaScript/Node.js. Enables development of data extraction and web automation jobs (not only) with headless Chrome and Puppeteer.","engines": {"node": ">=15.10.0"@@ -67,7 +67,7 @@"content-type": "^1.0.4","express": "^4.17.1","fs-extra": "^10.0.0",- "got-scraping": "^3.0.1",+ "got-scraping": "^3.1.0","htmlparser2": "^6.1.0","iconv-lite": "^0.6.3","jquery": "^3.6.0",diff --git a/src/crawlers/cheerio_crawler.js b/src/crawlers/cheerio_crawler.jsindex 32ebecd77..49f42ce51 100644--- a/src/crawlers/cheerio_crawler.js+++ b/src/crawlers/cheerio_crawler.js@@ -719,6 +719,7 @@ class CheerioCrawler extends BasicCrawler {ignoreSslErrors: this.ignoreSslErrors,proxyUrl,timeoutSecs: this.requestTimeoutMillis / 1000,+ sessionToken: session,...requestAsBrowserOptions,stream: true,abortFunction: (res) => {diff --git a/src/utils_request.js b/src/utils_request.jsindex e4697228b..9df3dfb46 100644--- a/src/utils_request.js+++ b/src/utils_request.js@@ -49,6 +49,9 @@ import { IncomingMessage } from 'http';* expect HTTP2 connections, because browsers use HTTP2 by default. It will automatically downgrade* to HTTP/1.1 for websites that do not support HTTP2. For Node 10 this option is always set to `false`* because Node 10 does not support HTTP2 very well. Upgrade to Node 12 for better performance.+ * @property {object} [sessionToken]+ * A unique object used to generate browser headers. By default, new headers are generated on every call.+ * Set this option to make these headers persistent.*//**@@ -119,6 +122,7 @@ export const requestAsBrowser = async (options = {}) => {headerGeneratorOptions: ow.optional.object,stream: ow.optional.boolean,decodeBody: ow.optional.boolean,+ sessionToken: ow.optional.object,}));ow(options, 'RequestAsBrowserOptions', ow.object.validate((opts) => ({