Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.
TypeScript apifyautomationcrawlercrawlingheadlessheadless-chromejavascriptnodejsnpmplaywrightpuppeteerscraperscrapingtypescriptweb-crawlerweb-crawlingweb-scraping
Add object serialization to Request constructor
Ondra Urban04f4a2f9a7ff
diff --git a/CHANGELOG.md b/CHANGELOG.mdindex 931c4203e..49313a41c 100644--- a/CHANGELOG.md+++ b/CHANGELOG.md@@ -1,3 +1,8 @@+NEXT+====================+- Add `SessionPool` implemenation to `CheerioCrawler`.+- Fix `Request` constructor not making a copy of objects such as `userData` and `headers`.+0.17.0 / 2019-11-25====================- **BREAKING**: Node 8 and 9 are no longer supported. Please use Node 10.17.0 or higher.diff --git a/docs/examples/BasicCrawler.md b/docs/examples/BasicCrawler.mdindex 68a2455d3..ab644b4ad 100644--- a/docs/examples/BasicCrawler.md+++ b/docs/examples/BasicCrawler.md@@ -8,7 +8,7 @@ and [`RequestList`](../api/requestlist) classes. The script just downloads sever<a href="https://www.npmjs.com/package/request-promise" target="_blank">request-promise</a> library) and stores their raw HTML and URL to the defaultdataset. In local configuration, the data will be stored as JSON files in `./apify_storage/datasets/default`.-To run this example on the Apify Platform, select the `Node.js 8 on Alpine Linux (apify/actor-node-basic)` base image on the source tab of your actor+To run this example on the Apify Platform, select the `Node.js 10 on Alpine Linux (apify/actor-node-basic)` base image on the source tab of your actorconfiguration.```javascriptdiff --git a/docs/examples/CallActor.md b/docs/examples/CallActor.mdindex f24a598b0..46f0dca3b 100644--- a/docs/examples/CallActor.md+++ b/docs/examples/CallActor.md@@ -13,7 +13,7 @@ To make the example work, you'll need an <a href="https://my.apify.com/" target=`APIFY_TOKEN` [environment variable](../guides/environmentvariables), or run the script using the CLI. If you deploy this actor to the Apify Cloudthen you can set up a scheduler for early morning.-To run this example on the Apify Platform, select the `Node.js 8 + Chrome on Debian (apify/actor-node-chrome)` base image on the source tab of your+To run this example on the Apify Platform, select the `Node.js 10 + Chrome on Debian (apify/actor-node-chrome)` base image on the source tab of youractor configuration.```javascriptdiff --git a/docs/examples/CheerioCrawler.md b/docs/examples/CheerioCrawler.mdindex 536959d0d..49a1f4333 100644--- a/docs/examples/CheerioCrawler.md+++ b/docs/examples/CheerioCrawler.md@@ -7,7 +7,7 @@ This example demonstrates how to use [`CheerioCrawler`](../api/cheeriocrawler) tplain HTTP request, parse the HTML using <a href="https://www.npmjs.com/package/cheerio" target="_blank">cheerio</a> and extract some data from it:the page title and all H1 tags.-To run this example on the Apify Platform, select the `Node.js 8 on Alpine Linux (apify/actor-node-basic)` base image on the source tab of your actor+To run this example on the Apify Platform, select the `Node.js 10 on Alpine Linux (apify/actor-node-basic)` base image on the source tab of your actorconfiguration.```javascriptdiff --git a/docs/examples/Forms.md b/docs/examples/Forms.mdindex bd8722416..da96d0cd6 100644--- a/docs/examples/Forms.md+++ b/docs/examples/Forms.md@@ -9,7 +9,7 @@ search term, repository owner, start date and language of the repository, then ssaved either on the Apify platform to the default <a href="https://sdk.apify.com/docs/api/dataset" target="_blank">dataset</a> or on the local machineas JSON files in ./apify_storage/datasets/default.-To run this example on the Apify Platform, select the `Node.js 8 + Chrome on Debian (apify/actor-node-chrome)` base image on the source tab of your+To run this example on the Apify Platform, select the `Node.js 10 + Chrome on Debian (apify/actor-node-chrome)` base image on the source tab of youractor configuration.```javascriptdiff --git a/docs/examples/HelloWorld.md b/docs/examples/HelloWorld.mdindex 88cede7f3..f0a763222 100644--- a/docs/examples/HelloWorld.md+++ b/docs/examples/HelloWorld.md@@ -5,7 +5,7 @@ title: Hello WorldRun the following example to perform a recursive crawl of a website using Puppeteer.-To run this example on the Apify Platform, select the `Node.js 8 + Chrome on Debian (apify/actor-node-chrome)` base image on the source tab of your+To run this example on the Apify Platform, select the `Node.js 10 + Chrome on Debian (apify/actor-node-chrome)` base image on the source tab of youractor configuration.```javascriptdiff --git a/docs/examples/PuppeteerCrawler.md b/docs/examples/PuppeteerCrawler.mdindex 1a7d3d1ce..f8014e092 100644--- a/docs/examples/PuppeteerCrawler.md+++ b/docs/examples/PuppeteerCrawler.md@@ -8,7 +8,7 @@ recursively scrape the <a href="https://news.ycombinator.com" target="_blank">Hastarts with a single URL, finds links to next pages, enqueues them and continues until no more desired links are available. The results are stored tothe default dataset. In local configuration, the results are stored as JSON files in `./apify_storage/datasets/default`-To run this example on the Apify Platform, select the `Node.js 8 + Chrome on Debian (apify/actor-node-chrome)` base image on the source tab of your+To run this example on the Apify Platform, select the `Node.js 10 + Chrome on Debian (apify/actor-node-chrome)` base image on the source tab of youractor configuration.```javascriptdiff --git a/docs/examples/PuppeteerSitemap.md b/docs/examples/PuppeteerSitemap.mdindex 149e6462b..11acd5315 100644--- a/docs/examples/PuppeteerSitemap.md+++ b/docs/examples/PuppeteerSitemap.md@@ -7,7 +7,7 @@ This example demonstrates how to use [`PuppeteerCrawler`](../api/puppeteercrawleextracts page title and URL from each page and stores them as a record to the default dataset. In local configuration, the results are stored as JSONfiles in `./apify_storage/datasets/default`-To run this example on the Apify Platform, select the `Node.js 8 + Chrome on Debian (apify/actor-node-chrome)` base image on the source tab of your+To run this example on the Apify Platform, select the `Node.js 10 + Chrome on Debian (apify/actor-node-chrome)` base image on the source tab of youractor configuration.```javascriptdiff --git a/docs/examples/PuppeteerWithProxy.md b/docs/examples/PuppeteerWithProxy.mdindex f4258b130..e7805ac6b 100644--- a/docs/examples/PuppeteerWithProxy.md+++ b/docs/examples/PuppeteerWithProxy.md@@ -8,7 +8,7 @@ Proxy</a>. To make it work, you'll need an Apify Account that has access to the<a href="https://my.apify.com/proxy" target="_blank">Proxy</a> page in the app. Just set it to the `APIFY_PROXY_PASSWORD`[environment variable](../guides/environmentvariables) or run the script using the CLI.-To run this example on the Apify Platform, select the `Node.js 8 + Chrome on Debian (apify/actor-node-chrome)` base image on the source tab of your+To run this example on the Apify Platform, select the `Node.js 10 + Chrome on Debian (apify/actor-node-chrome)` base image on the source tab of youractor configuration.```javascriptdiff --git a/docs/examples/Screenshots.md b/docs/examples/Screenshots.mdindex 47e74a2f9..2cde1e444 100644--- a/docs/examples/Screenshots.md+++ b/docs/examples/Screenshots.md@@ -18,7 +18,7 @@ On the Apify cloud, the input can be either set manually in the UI app or passed<a href="https://apify.com/docs/api/v2#/reference/actors/run-collection/run-actor" target="_blank">Run actor API call</a>. For more details, see<a href="https://apify.com/docs/actor#input-output" target="_blank">Input and output</a> in the Apify Actor documentation.-To run this example on the Apify Platform, select the `Node.js 8 + Chrome on Debian (apify/actor-node-chrome)` base image on the source tab of your+To run this example on the Apify Platform, select the `Node.js 10 + Chrome on Debian (apify/actor-node-chrome)` base image on the source tab of youractor configuration.```javascriptdiff --git a/docs/examples/SynchronousRun.md b/docs/examples/SynchronousRun.mdindex ffaf9ef64..083da1004 100644--- a/docs/examples/SynchronousRun.md+++ b/docs/examples/SynchronousRun.md@@ -17,7 +17,7 @@ can test it directly there simply by sending a POST request tohttps://api.apify.com/v2/acts/apify~example-golden-gate-webcam/run-sync?token=[YOUR_API_TOKEN]```-To run this example on the Apify Platform, select the `Node.js 8 + Chrome on Debian (apify/actor-node-chrome)` base image on the source tab of your+To run this example on the Apify Platform, select the `Node.js 10 + Chrome on Debian (apify/actor-node-chrome)` base image on the source tab of youractor configuration.```javascriptdiff --git a/package.json b/package.jsonindex 40b0ffa32..f4db6e405 100644--- a/package.json+++ b/package.json@@ -1,6 +1,6 @@{"name": "apify",- "version": "0.17.0",+ "version": "0.17.1","description": "The scalable web crawling and scraping library for JavaScript/Node.js. Enables development of data extraction and web automation jobs (not only) with headless Chrome and Puppeteer.","engines": {"node": ">=10.17.0"diff --git a/src/request.js b/src/request.jsindex 88e1c4b15..6a025c996 100644--- a/src/request.js+++ b/src/request.js@@ -179,9 +179,9 @@ class Request {this.payload = payload;this.noRetry = noRetry;this.retryCount = retryCount;- this.errorMessages = errorMessages;- this.headers = headers;- this.userData = userData;+ this.errorMessages = JSON.parse(JSON.stringify(errorMessages));+ this.headers = JSON.parse(JSON.stringify(headers));+ this.userData = JSON.parse(JSON.stringify(userData));// Requests received from API will have ISOString dates,// but we want to have a Date instance.