Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.
TypeScript apifyautomationcrawlercrawlingheadlessheadless-chromejavascriptnodejsnpmplaywrightpuppeteerscraperscrapingtypescriptweb-crawlerweb-crawlingweb-scraping
Fix merge conflicts
Ondra Urban9627b2f92f08
diff --cc CHANGELOG.mdindex f3e5f3d8c,1909c8a3a..6c6f5c6d7--- a/CHANGELOG.md+++ b/CHANGELOG.md@@@ -1,28 -1,9 +1,29 @@@-<<<<<<< HEADxxx===================-- Added `getPublicUrl` function to `KeyValueStoreLocal`-=======-0.11.5 / 2018-01-18+- Added memory overload warning log message.++- Added `keyValueStore.getPublicUrl` function.++0.11.8 / 2019-02-05+===================+- Fix the `keyValueStore.forEachKey()` method.+- Fix version of `puppeteer` to prevent errors with automatic updates.++0.11.7 / 2019-01-30+===================+- Apify SDK now logs basic system info when `required`.+- Added `utils.createRequestDebugInfo()` function to create a standardized debug info from request and response.+- `PseudoUrl` can now be constructed with a `RegExp`.+- `Apify.utils.enqueueLinks()` now accepts `RegExp` instances in its `pseudoUrls` parameter.+- `Apify.utils.enqueueLinks()` now accepts a `baseUrl` option that enables resolution of relative URLs+ when parsing a Cheerio object. (It's done automatically in browser when using Puppeteer).+- Better error message for an invalid `launchPuppeteerFunction` passed to `PuppeteerPool`.++0.11.6 / 2019-01-24+===================+- **DEPRECATION WARNING** `Apify.utils.puppeteer.enqueueLinks()` was moved to `Apify.utils.enqueueLinks()`.+- `Apify.utils.enqueueLinks()` now supports `options.$` property to enqueue links from a Cheerio object.++0.11.5 / 2019-01-18===================- Disabled the `PuppeteerPool` `reusePages` option for now, due to a memory leak.- Added a `keyValueStore.forEachKey()` method to iterate all keys in the store.diff --cc src/key_value_store.jsindex d76695e4b,c235af98f..06a9dee4b--- a/src/key_value_store.js+++ b/src/key_value_store.js@@@ -292,9 -292,15 +293,15 @@@ export class KeyValueStore});}++ getPublicUrl(fileName) {+ ensureTokenOrLocalStorageEnvExists('key value store');+ return `${publicUrl}/${this.storeId}/records/${fileName}`;+ }+/*** Iterates over key value store keys, yielding each in turn to an `iteratee` function.- * Each invocation of `iteratee` is called with two arguments: `(item, index)`.+ * Each invocation of `iteratee` is called with three arguments: `(item, index, info)`.** If the `iteratee` function returns a Promise then it is awaited before the next call.* If it throws an error, the iteration is aborted and the `forEachKey` function throws the error.