Skip to content

Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.

Merge branch 'chore/v0.21.10'

Ondra Urbanab04771ab3e3
diff --cc CHANGELOG.md
index d40f76c7a,dbefcf1d2..a973f48d0
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@@ -1,83 -1,7 +1,87 @@@
+0.22.0 / XXX
+====================
+This is the last major release before SDK v1.0.0. We're committed to deliver v1 at the
+end of 2020 so stay tuned. Besides Playwright integration via a new `BrowserPool`,
+it will be the first release of SDK that we'll support for an extended period of time.
+We will not make any breaking changes until 2.0.0, which will come at the end of
+2021. But enough about v1, let's see the changes in 0.22.0.
+
+In this release we've changed a lot of code, but you may not even notice.
+We've updated the underlying `apify-client` package which powers all communication with
+the Apify API to version `1.0.0`. This means a completely new API for all internal calls.
+If you use `Apify.client` calls in your code, this will be a large breaking change for you.
+Visit the [client docs](https://github.com/apify/apify-client-js/blob/master/README.md)
+to see what's new in the client, but also note that we removed the default client
+available under `Apify.client` and replaced it with `Apify.newClient()` function.
+We think it's better to have separate clients for users and internal use.
+
+Until now, local emulation of Apify Storages has been a part of the SDK. We moved the logic
+into a separate package `@apify/storage-local` which shares interface with `apify-client`.
+`RequestQueue` is now powered by `SQLite3` instead of file system, which improves
+reliability and performance quite a bit. `Dataset` and `KeyValueStore` still use file
+system, for easy browsing of data. The structure of `apify_storage` folder remains unchanged.
+
+After collecting common developer mistakes, we've decided to make argument validation stricter.
+You will no longer be able to pass extra arguments to functions and constructors. This is
+to alleviate frustration, when you mistakenly pass `useChrome` to `PuppeteerPoolOptions`
+instead of `LaunchPuppeteerOptions` and don't realize it. Before this version, SDK wouldn't
+let you know and would silently continue with Chromium. Now, it will throw an error saying
+that `useChrome` is not an allowed property of `PuppeteerPoolOptions`.
+
+Based on developer feedback, we decided to remove `--no-sandbox` from the default Puppeteer
+launch args. It will only be used on Apify Platform. This gives you the chance to use
+your own sandboxing strategy.
+
+`LiveViewServer` and `puppeteerPoolOptions.useLiveView` were never very user-friendly
+or performant solutions, due to the inherent performance issues with rapidly taking many
+screenshots in Puppeteer. We've decided to remove it. If you need similar functionality,
+try the `devtools-server` NPM package, which utilizes the Chrome DevTools Frontend for
+screen-casting live view of the running browser.
+
+Full list of changes:
+
+- **BREAKING:** Updated `apify-client` to `1.0.0` with a completely new interface.
+ We also removed the `Apify.client` property and replaced it with an `Apify.newClient()`
+ function that creates a new `ApifyClient` instance.
+- **BREAKING:** Removed `--no-sandbox` from default Puppeteer launch arguments.
+ This will most likely be breaking for Linux and Docker users.
+- **BREAKING:** Function argument validation is now more strict and will not accept extra
+ parameters which are not defined by the functions' signatures.
+- **DEPRECATED:** `puppeteerPoolOptions.useLiveView` is now deprecated.
+ Use the `devtools-server` NPM package instead.
+
+- Added `postResponseFunction` to `CheerioCrawlerOptions`. It allows you to override
+ properties on the HTTP response before processing by `CheerioCrawler`.
+- Added HTTP2 support to `utils.requestAsBrowser()`. Set `useHttp2` to `true`
+ in `RequestAsBrowserOptions` to enable it.
+- Fixed handling of XML content types in `CheerioCrawler`.
+- Fixed capitalization of headers when using `utils.puppeteer.addInterceptRequestHandler`.
+- Fixed `utils.puppeteer.saveSnapshot()` overwriting screenshots with HTML on local.
+- Updated `puppeteer` to version `5.4.1` with Chrom(ium) 87.
+- Removed `RequestQueueLocal` in favor of `@apify/storage-local` API emulator.
+- Removed `KeyValueStoreLocal` in favor of `@apify/storage-local` API emulator.
+- Removed `DatasetLocal` in favor of `@apify/storage-local` API emulator.
+- Removed the `userData` option from `Apify.utils.enqueueLinks` (deprecated in Jun 2019).
+ Use `transformRequestFunction` instead.
+- Removed `instanceKillerIntervalMillis` and `killInstanceAfterMillis` (deprecated in Feb 2019).
+ Use `instanceKillerIntervalSecs` and `killInstanceAfterSecs` instead.
+- Removed the `memory` option from `Apify.call` `options` which was (deprecated in 2018).
+ Use `memoryMbytes` instead.
+- Removed `delete()` methods from `Dataset`, `KeyValueStore` and `RequestQueue` (deprecated in Jul 2019).
+ Use `.drop()`.
+- Removed `utils.puppeteer.hideWebDriver()` (deprecated in May 2019).
+ Use `LaunchPuppeteerOptions.stealth`.
+- Removed `utils.puppeteer.enqueueRequestsFromClickableElements()` (deprecated in 2018).
+ Use `utils.puppeteer.enqueueLinksByClickingElements`.
+- Removed `request.doNotRetry()` (deprecated in June 2019)
+ Use `request.noRetry = true`.
+- Removed `RequestListOptions.persistSourcesKey` (deprecated in Feb 2020)
+ Use `persistRequestsKey`.
+
+ 0.21.10 / 2020/12/07
+ ====================
+ - Bump Puppeteer to 5.5.0 and Chrom(ium) 88.
+
0.21.9 / 2020/11/03
====================
- Fix various issues in `stealth`.
diff --cc package.json
index 9c72f0876,4a5c82185..aa5811cc1
--- a/package.json
+++ b/package.json
@@@ -65,59 -64,58 +65,59 @@@
"cheerio": "^1.0.0-rc.3",
"content-type": "^1.0.4",
"express": "^4.17.1",
- "fs-extra": "^9.0.0",
- "htmlparser2": "^4.1.0",
- "iconv-lite": "^0.6.0",
+ "fs-extra": "^9.0.1",
+ "htmlparser2": "^5.0.1",
+ "iconv-lite": "^0.6.2",
"jquery": "^3.5.1",
"mime-types": "^2.1.27",
- "proxy-chain": "^0.4.4",
+ "ow": "^0.20.0",
+ "proxy-chain": "^0.4.5",
"rimraf": "^3.0.2",
"semver": "^7.3.2",
- "socket.io": "^2.3.0",
- "stream-json": "^1.5.0",
+ "socket.io": "^3.0.3",
+ "stream-json": "^1.7.1",
"tough-cookie": "^4.0.0",
- "underscore": "^1.10.2",
- "ws": "^7.3.0"
+ "underscore": "^1.11.0",
+ "ws": "^7.3.1"
},
"devDependencies": {
- "@apify/eslint-config": "^0.0.3",
- "@babel/cli": "^7.8.4",
- "@babel/core": "^7.9.6",
- "@babel/preset-env": "^7.9.6",
- "@babel/register": "^7.9.0",
+ "@apify/eslint-config": "^0.1.3",
+ "@babel/cli": "^7.11.6",
+ "@babel/core": "^7.11.6",
+ "@babel/preset-env": "^7.11.5",
+ "@babel/register": "^7.11.5",
"@types/content-type": "^1.1.3",
- "@types/fs-extra": "^9.0.0",
- "@types/htmlparser2": "^3.10.1",
- "@types/jest": "^26.0.0",
+ "@types/fs-extra": "^9.0.2",
+ "@types/htmlparser2": "^3.10.2",
+ "@types/jest": "^26.0.14",
"@types/request-promise-native": "^1.0.17",
"@types/rimraf": "^3.0.0",
- "@types/semver": "^7.2.0",
- "@types/sinon": "^9.0.0",
- "@types/underscore": "^1.10.0",
- "@types/ws": "^7.2.4",
+ "@types/semver": "^7.3.4",
+ "@types/sinon": "^9.0.8",
+ "@types/underscore": "^1.10.24",
+ "@types/ws": "^7.2.7",
"basic-auth-parser": "^0.0.2",
"body-parser": "^1.19.0",
- "csv-parse": "^4.9.1",
- "eslint": "^7.0.0",
+ "csv-parse": "^4.12.0",
+ "eslint": "^7.11.0",
"fpcollect": "^1.0.4",
"fpscanner": "^0.1.5",
- "jest": "^26.0.1",
- "jest-circus": "^26.0.1",
- "jsdoc": "^3.6.4",
+ "jest": "^26.5.3",
+ "jest-circus": "^26.5.3",
+ "jsdoc": "^3.6.6",
"jsdoc-export-default-interop": "^0.3.1",
"portastic": "^1.0.1",
- "prettier": "^2.0.5",
+ "prettier": "^2.1.2",
"proxy": "^1.0.2",
- "sinon": "^9.0.2",
+ "sinon": "^9.2.0",
"sinon-stub-promise": "^4.0.0",
- "socket.io-client": "^2.3.0",
- "ts-jest": "^26.0.0",
+ "socket.io-client": "^3.0.3",
+ "ts-jest": "^26.4.1",
"typescript": "3.8.3",
- "ua-parser-js": "^0.7.21",
- "why-is-node-running": "^2.1.2"
+ "ua-parser-js": "^0.7.22",
+ "why-is-node-running": "^2.2.0"
},
"optionalDependencies": {
- "puppeteer": "5.4.1"
+ "puppeteer": "5.5.0"
}
}