Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.
TypeScript apifyautomationcrawlercrawlingheadlessheadless-chromejavascriptnodejsnpmplaywrightpuppeteerscraperscrapingtypescriptweb-crawlerweb-crawlingweb-scraping
Update default user agent, remove random user agents
Ondra Urbanf4b4f359ec4c
diff --git a/CHANGELOG.md b/CHANGELOG.mdindex 57bcd5d23..f23af0381 100644--- a/CHANGELOG.md+++ b/CHANGELOG.md@@ -1,20 +1,32 @@0.21.0 / NEXT====================-This release comes with a redesigned proxy configuration. All crawlers now accept-a `proxyConfiguration` parameter, which is an instance of `ProxyConfiguration`.-Apify Proxy is now exclusively serviced by this class. In the next release,-custom proxy management will move to `ProxyConfiguration` as well.+This release comes with **breaking changes** that will affect most,+if not all of your projects. See the [migration guide](./MIGRATIONS.md)+for more information and examples.++First large change is a redesigned proxy configuration. `Cheerio` and `Puppeteer` crawlers+now accept a `proxyConfiguration` parameter, which is an instance of `ProxyConfiguration`.+This class now exclusively manages both Apify Proxy and custom proxies.++We also removed `Apify.utils.getRandomUserAgent()` as it was no longer effective+in avoiding bot detection.- **BREAKING:** Removed `Apify.getApifyProxyUrl()`. To get an Apify Proxy url,- use `proxyConfiguration.getUrl([sessionId])`.+ use `proxyConfiguration.newUrl([sessionId])`.- **BREAKING:** Removed `useApifyProxy`, `apifyProxyGroups` and `apifyProxySession` parametersfrom all applications in the SDK. Use `proxyConfiguration` in crawlers and `proxyUrl`in `requestAsBrowser` and `Apify.launchPuppeteer`.+- **BREAKING:** Removed `Apify.utils.getRandomUserAgent()` as it was no longer effective+ in avoiding bot detection.- Add `Apify.createProxyConfiguration()` `async` function to create `ProxyConfiguration`instances. `ProxyConfiguration` itself is not exposed.-- Add `proxyConfiguration` to `BasicCrawlerOptions`, `CheerioCrawlerOptions`+- Add `proxyConfiguration` to `CheerioCrawlerOptions`and `PuppeteerCrawlerOptions`.+- Add `proxyInfo` to `CheerioHandlePageInputs` and `PuppeteerHandlePageInputs`.+ You can use this object to retrieve information about the currently used proxy+ in `Puppeteer` and `Cheerio` crawlers.- Fixed a bug where intercepted requests would never continue.+- Fixed a bug where `Apify.utils.requestAsBrowser()` would get into redirect loops.0.20.4 / 2020-05-11diff --git a/MIGRATIONS.md b/MIGRATIONS.mdindex 19036a012..69d5620e6 100644--- a/MIGRATIONS.md+++ b/MIGRATIONS.md@@ -1,37 +1,142 @@-# Migration from 0.7.x to 0.8.0--## Breaking change 1: AutoscaledPool configuration changed--Originally, AutoscaledPool was just a single class that handled everything. Now it's split into 3 different classes.-As such, the configuration changed dramatically. The API is the same however.--Mainly the parameter `workerFunction` was replaced with combination of 3 parameters `runTaskFunction`, `isTaskReadyFunction` and `isFinishedFunction`. For more information and examples check <a href="https://sdk.apify.com/docs/api/autoscaledpool#docsNav" target="_blank">AutoscaledPool documentation</a>.--If you were using any of the following options, you should see the current documentation and migrate accordingly:--1. `options.workerFunction`- - REMOVED; see `options.runTaskFunction` for similar functionality-2. `options.finishWhenEmpty`- - REMOVED; see `options.isFinishedFunction` for similar functionality-3. `options.ignoreMainProcess`- - REMOVED;-4. `options.maxMemoryMbytes`- - MOVED; use `options.snapshotterOptions.maxMemoryMbytes`-5. `options.minFreeMemoryRatio`- - MOVED AND CHANGED; use `options.snapshotterOptions.maxUsedMemoryRatio`-6. `options.maybeRunIntervalMillis`- - CHANGED; use `options.maybeRunIntervalSecs`-7. `options.loggingIntervalMillis`- - CHANGED; use `options.loggingIntervalSecs`--For more configuration options, see Snapshotter and SystemStatus documentation.--## Breaking change 2: Crawlers no longer support passing AutoscaledPool options directly--Previously, setting e.g. `options.isFinishedFunction` directly on the options object passed to-a Crawler constructor would configure the underlying AutoscaledPool. From v0.8.0,-the options are available under `options.autoscaledPoolOptions`. Setting a custom `isFinishedFunction`-is therefore done using `options.autoscaledPoolOptions.isFinishedFunction`.--Frequently used properties `minConcurrency` and `maxConcurrency` are exempted and can still be used-directly as `options.minConcurrency`.+# Migration from 0.20.x to 0.21.0+There are 2 key changes to watch out for in version 0.21.0. We redesigned the way proxies+are used throughout the SDK, and we removed `Apify.utils.getRandomUserAgent()`.++## Proxy Configuration+Configuring proxies the way you were used to will no longer work in SDK 0.21.0.+To improve the proxy experience, we removed all the various proxy configuration+options scattered throughout the SDK and replaced them with a single+`ProxyConfiguration` class.++### Removal of `Apify.getApifyProxyUrl()`+This function has been fully replaced by the new `ProxyConfiguration` class.+Make sure to remove any references to it before upgrading.++### Usage with Apify Proxy:+The usage of Apify Proxy in SDK was inconsistent. Now, everything is pretty simple+and as a bonus, we validate your configuration at creation, so if there's something+wrong, you'll know right away, instead of seeing your crawlers fail with cryptic errors.++1. You create a `proxyConfiguration` instance.+2. Are you using `CheerioCrawler` or `PuppeteerCrawler`?+ - YES: You plug it into the crawler, and you're done.+ - NO: You call `proxyConfiguration.newUrl([sessionId])` to get your URL.++> `BasicCrawler` does not automatically use `ProxyConfiguration` because it does not make+any network requests (automatically).++That's it. See the examples.++```js+// before+const crawler = new Apify.PuppeteerCrawler({+ launchPuppeteerOptions: {+ useApifyProxy: true,+ apifyProxyGroups: ['GROUP1', 'GROUP2'],+ }+ // ...+})++// now+const proxyConfiguration = await Apify.createProxyConfiguration({+ groups: ['GROUP1', 'GROUP2'],+})++const crawler = new Apify.PuppeteerCrawler({+ proxyConfiguration,+ // ...+})+```++```js+// before+const browser = await Apify.launchPuppeteer({+ useApifyProxy: true,+ apifyProxyGroups: ['GROUP1', 'GROUP2'],+ apifyProxySession: 'session-123'+})++// now+const proxyConfiguration = await Apify.createProxyConfiguration({+ groups: ['GROUP1', 'GROUP2'],+});++const browser = await Apify.launchPuppeteer({+ proxyUrl: proxyConfiguration.newUrl('session-123')+})+```++### Usage with your own proxies:++Using your own proxies was possible, but it was difficult to find where to+enter your URLs to make use of them. Now it's the same as with Apify Proxy,+you just add them to the `ProxyConfiguration`. As a bonus, your custom proxies+can now also be managed by `SessionPool` which was not possible before.+So if one of your proxies goes bad, `SessionPool` will automatically+retire it from use.++```js+// before+const crawler = new Apify.PuppeteerCrawler({+ puppeteerPoolOptions: {+ proxyUrls: [+ 'http://proxy1.com',+ 'http://proxy2.com',+ ]+ }+ // ...+})++// now+const proxyConfiguration = await Apify.createProxyConfiguration({+ proxyUrls: [+ 'http://proxy1.com',+ 'http://proxy2.com',+ ]+})++const crawler = new Apify.PuppeteerCrawler({+ proxyConfiguration,+ // ...+})+```++```js+// before+const browser1 = await Apify.launchPuppeteer({+ proxyUrl: 'http://proxy1:com',+})++// now+const proxyConfiguration = await Apify.createProxyConfiguration({+ proxyUrls: [+ 'http://proxy1.com',+ 'http://proxy2.com',+ ]+});++const browser = await Apify.launchPuppeteer({+ proxyUrl: proxyConfiguration.newUrl('session-123'),+})+```++## Removal of random user agents++Nowadays, bot walls cannot be bypassed simply by rotating user agents.+Quite the opposite is true, using a user agent that does not match+the used browser / network stack, will most likely lead to+red flags and bans due to detected inconsistencies. For that reason,+we've decided to remove the `Apify.utils.getRandomUserAgent()` function,+effective immediately. Leaving a deprecation period would only have+your scrapers get blocked for longer. Please make sure you remove all+references to the function from your code.++```js+// before+const browser = await Apify.launchPuppeteer({+ userAgent: Apify.utils.getRandomUserAgent(),+})++// now, keep the default+const browser = await Apify.launchPuppeteer();+```diff --git a/src/constants.js b/src/constants.jsindex e6845ba1a..375fec6ae 100644--- a/src/constants.js+++ b/src/constants.js@@ -2,9 +2,9 @@ import { ACTOR_EVENT_NAMES } from 'apify-shared/consts';/*** The default user agent used by `Apify.launchPuppeteer`.- * Last updated on 2020-04-08.+ * Last updated on 2020-05-22.*/-export const DEFAULT_USER_AGENT = 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/81.0.4044.92 Safari/537.36'; // eslint-disable-line max-len+export const DEFAULT_USER_AGENT = 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/83.0.4103.61 Safari/537.36';/*** Exit codes for the actor process.@@ -24,92 +24,6 @@ export const EXIT_CODES = {*/export const ACTOR_EVENT_NAMES_EX = Object.assign({}, ACTOR_EVENT_NAMES, { PERSIST_STATE: 'persistState' });-/**- * Most common user agents from https://techblog.willshouse.com/2012/01/03/most-common-user-agents/- *- * Last updated on 2019-02-18.- */-export const USER_AGENT_LIST = [- 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/71.0.3578.98 Safari/537.36',- 'Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/71.0.3578.98 Safari/537.36',- 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/71.0.3578.98 Safari/537.36',- 'Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:64.0) Gecko/20100101 Firefox/64.0',- 'Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:65.0) Gecko/20100101 Firefox/65.0',- 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_2) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/12.0.2 Safari/605.1.15',- 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/71.0.3578.98 Safari/537.36',- 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/72.0.3626.109 Safari/537.36',- 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/64.0.3282.140 Safari/537.36 Edge/17.17134',- 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/72.0.3626.96 Safari/537.36',- 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/71.0.3578.98 Safari/537.36',- 'Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:64.0) Gecko/20100101 Firefox/64.0',- 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/72.0.3626.81 Safari/537.36',- 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_3) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/12.0.3 Safari/605.1.15',- 'Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:65.0) Gecko/20100101 Firefox/65.0',- 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_3) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/71.0.3578.98 Safari/537.36',- 'Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:64.0) Gecko/20100101 Firefox/64.0',- 'Mozilla/5.0 (X11; Linux x86_64; rv:60.0) Gecko/20100101 Firefox/60.0',- 'Mozilla/5.0 (Windows NT 6.3; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/71.0.3578.98 Safari/537.36',- 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/71.0.3578.98 Safari/537.36',- 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/71.0.3578.98 Safari/537.36',- 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10.14; rv:65.0) Gecko/20100101 Firefox/65.0',- 'Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:65.0) Gecko/20100101 Firefox/65.0',- 'Mozilla/5.0 (Windows NT 6.1; WOW64; Trident/7.0; rv:11.0) like Gecko',- 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_6) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/12.0.3 Safari/605.1.15',- 'Mozilla/5.0 (Windows NT 10.0; WOW64; Trident/7.0; rv:11.0) like Gecko',- 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10.14; rv:64.0) Gecko/20100101 Firefox/64.0',- 'Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/71.0.3578.98 Safari/537.36',- 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/71.0.3578.98 Safari/537.36',- 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.102 Safari/537.36 OPR/57.0.3098.116',- 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Ubuntu Chromium/71.0.3578.98 Chrome/71.0.3578.98 Safari/537.36',- 'Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:60.0) Gecko/20100101 Firefox/60.0',- 'Mozilla/5.0 (X11; Linux x86_64; rv:65.0) Gecko/20100101 Firefox/65.0',- 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/12.0 Safari/605.1.15',- 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/64.0.3282.140 Safari/537.36 Edge/18.17763',- 'Mozilla/5.0 (X11; Linux x86_64; rv:64.0) Gecko/20100101 Firefox/64.0',- 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/72.0.3626.81 Safari/537.36',- 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_3) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/72.0.3626.96 Safari/537.36',- 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.77 Safari/537.36',- 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/71.0.3578.98 Safari/537.36',- 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/58.0.3029.110 Safari/537.36 Edge/16.16299',- 'Mozilla/5.0 (Windows NT 6.1; rv:60.0) Gecko/20100101 Firefox/60.0',- 'Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/72.0.3626.109 Safari/537.36',- 'Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/72.0.3626.96 Safari/537.36',- 'Mozilla/5.0 (iPad; CPU OS 12_1_1 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/12.0 Mobile/15E148 Safari/604.1',- 'Mozilla/5.0 (Windows NT 6.3; Win64; x64; rv:64.0) Gecko/20100101 Firefox/64.0',- 'Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/71.0.3578.98 Safari/537.36',- 'Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/72.0.3626.81 Safari/537.36',- 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_6) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/12.0.2 Safari/605.1.15',- 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10.13; rv:65.0) Gecko/20100101 Firefox/65.0',- 'Mozilla/5.0 (Windows NT 6.3; Win64; x64; rv:65.0) Gecko/20100101 Firefox/65.0',- 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/72.0.3626.96 Safari/537.36',- 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_1) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/12.0.1 Safari/605.1.15',- 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_6) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/12.0.3 Safari/605.1.15',- 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/71.0.3578.98 Safari/537.36',- 'Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:60.0) Gecko/20100101 Firefox/60.0',- 'Mozilla/5.0 (Windows NT 10.0; WOW64; Trident/7.0; Touch; rv:11.0) like Gecko',- 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_3) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/72.0.3626.109 Safari/537.36',- 'Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/71.0.3578.99 YaBrowser/19.1.0.2644 Yowser/2.5 Safari/537.36',- 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/72.0.3626.96 Safari/537.36',- 'Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.102 Safari/537.36 OPR/57.0.3098.116',- 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10.13; rv:64.0) Gecko/20100101 Firefox/64.0',- 'Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:66.0) Gecko/20100101 Firefox/66.0',- 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/72.0.3626.81 Safari/537.36',- 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.110 Safari/537.36',- 'Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.102 YaBrowser/18.11.1.805 Yowser/2.5 Safari/537.36',- 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_6) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/12.0.2 Safari/605.1.15',- 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/51.0.2704.106 Safari/537.36',- 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_6) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/11.1.2 Safari/605.1.15',- 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_6) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/11.1.2 Safari/605.1.15',- 'Mozilla/5.0 (Windows NT 10.0; WOW64; rv:60.0) Gecko/20100101 Firefox/60.0',- 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/71.0.3578.80 Safari/537.36',- 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/72.0.3626.81 Safari/537.36',- 'Mozilla/5.0 (Windows NT 10.0; WOW64; rv:65.0) Gecko/20100101 Firefox/65.0',- 'Mozilla/5.0 (X11; Fedora; Linux x86_64; rv:64.0) Gecko/20100101 Firefox/64.0',- 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.110 Safari/537.36',- 'Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.102 YaBrowser/18.11.1.805 Yowser/2.5 Safari/537.36',- 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_3) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/72.0.3626.81 Safari/537.36',-];-/*** Base URL of Apify's API endpoints.* @type {string}diff --git a/src/utils.js b/src/utils.jsindex 2c913e1f8..68d50cc96 100644--- a/src/utils.js+++ b/src/utils.js@@ -4,7 +4,6 @@ import * as ApifyClient from 'apify-client';import { checkParamOrThrow } from 'apify-client/build/utils';import { version as apifyClientVersion } from 'apify-client/package.json';import { ACT_JOB_TERMINAL_STATUSES, ENV_VARS, LOCAL_ENV_VARS } from 'apify-shared/consts';-import { getRandomInt } from 'apify-shared/utilities';import * as cheerio from 'cheerio';import * as contentTypeParser from 'content-type';import * as fs from 'fs';@@ -16,7 +15,6 @@ import * as semver from 'semver';import * as _ from 'underscore';import { URL } from 'url';import * as util from 'util';-import { USER_AGENT_LIST } from './constants';import * as requestUtils from './utils_request';import log from './utils_log';import { version as apifyVersion } from '../package.json';@@ -463,16 +461,6 @@ const extractUrls = ({ string, urlRegExp = URL_NO_COMMAS_REGEX }) => {return string.match(urlRegExp) || [];};-/**- * Returns a randomly selected User-Agent header out of a list of the most common headers.- * @returns {string}- * @memberOf utils- */-const getRandomUserAgent = () => {- const index = getRandomInt(USER_AGENT_LIST.length);- return USER_AGENT_LIST[index];-};-/*** Helper function to open local storage.*@@ -800,7 +788,6 @@ export const publicUtils = {sleep,downloadListOfUrls,extractUrls,- getRandomUserAgent,htmlToText,URL_NO_COMMAS_REGEX,URL_WITH_COMMAS_REGEX,diff --git a/test/utils.test.js b/test/utils.test.jsindex acbed7ff4..f363ff1ce 100644--- a/test/utils.test.js+++ b/test/utils.test.js@@ -522,14 +522,6 @@ describe('Apify.utils.downloadListOfUrls()', () => {});});-describe('Apify.utils.getRandomUserAgent()', () => {- test('works', () => {- const agent = utils.publicUtils.getRandomUserAgent();- expect(typeof agent).toBe('string');- expect(agent.length).not.toBe(0);- });-});-describe('utils.openLocalStorage()', () => {test('should return item from cache if available and create new one otherwise',