Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.
TypeScript apifyautomationcrawlercrawlingheadlessheadless-chromejavascriptnodejsnpmplaywrightpuppeteerscraperscrapingtypescriptweb-crawlerweb-crawlingweb-scraping
Upgraded proxy-chain NPM package from 0.2.7 to 0.3.3
Jan Curn9cf05dc8d60a
diff --git a/CHANGELOG.md b/CHANGELOG.mdindex 7a4036af3..eda4d9662 100644--- a/CHANGELOG.md+++ b/CHANGELOG.md@@ -3,7 +3,7 @@- **BREAKING:** `Apify.utils.requestAsBrowser()` no longer aborts request on status code 406or when other than `text/html` type is received. Use `options.abortFunction` if you want toretain this functionality.-- **BREAKING:** Added `useLenientHttpParser` option to `Apify.utils.requestAsBrowser()` which+- **BREAKING:** Added `useInsecureHttpParser` option to `Apify.utils.requestAsBrowser()` whichis `true` by default and forces the function to use a HTTP parser that is less strict thandefault Node 12 parser, but also less secure. It is needed to be able to bypass certainanti-scraping walls and fetch websites that do not comply with HTTP spec.@@ -12,6 +12,7 @@- Documented `BasicCrawler.autoscaledPool` property,and added `CheerioCrawler.autoscaledPool` and `PuppeteerCrawler.autoscaledPool` properties.- Added TypeScript typings and typedef documentation for all entities used throughout SDK.+- Upgraded `proxy-chain` NPM package from 0.2.7 to 0.3.30.19.1 / 2020-01-30====================diff --git a/package.json b/package.jsonindex 0ffc2a400..95cfaf639 100644--- a/package.json+++ b/package.json@@ -67,7 +67,7 @@"iconv-lite": "^0.5.0","jquery": "^3.4.1","mime-types": "^2.1.24",- "proxy-chain": "^0.2.7",+ "proxy-chain": "^0.3.3","request": "^2.88.0","request-promise-native": "^1.0.5","rimraf": "^2.7.1",