Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.
TypeScript apifyautomationcrawlercrawlingheadlessheadless-chromejavascriptnodejsnpmplaywrightpuppeteerscraperscrapingtypescriptweb-crawlerweb-crawlingweb-scraping
Rename lenient to insecure, tone down doc warning
Ondra Urban20c64009ab2b
diff --git a/docs/api/utils.md b/docs/api/utils.mdindex 01236cdab..6231e3d50 100644--- a/docs/api/utils.md+++ b/docs/api/utils.md@@ -139,9 +139,9 @@ objects.## `utils.requestAsBrowser` ⇒ `Promise<(http.IncomingMessage|stream.Readable)>`-**IMPORTANT WARNING:** This function uses an insecure version of HTTP parser by default and also ignores SSL/TLS errors. This is very useful in-scraping, because it allows bypassing certain anti-scraping walls, but it also makes it vulnerable to attacks. Do not use this function in other than-scraping scenarios, unless you set `useLenientHttpParser: false` and `ignoreSslErrors: false`.+**IMPORTANT:** This function uses an insecure version of HTTP parser by default and also ignores SSL/TLS errors. This is very useful in scraping,+because it allows bypassing certain anti-scraping walls, but it also exposes some vulnerability. For other than scraping scenarios, please set+`useInsecureHttpParser: false` and `ignoreSslErrors: false`.Sends an HTTP request that looks like a request sent by a web browser, fully emulating browser's HTTP headers.diff --git a/docs/typedefs/RequestAsBrowserOptions.md b/docs/typedefs/RequestAsBrowserOptions.mdindex 6333b3047..822b6f13e 100644--- a/docs/typedefs/RequestAsBrowserOptions.md+++ b/docs/typedefs/RequestAsBrowserOptions.md@@ -64,13 +64,13 @@ title: RequestAsBrowserOptions<tr><td colspan="3"><p>If set to true, SSL/TLS certificate errors will be ignored.</p></td></tr><tr>-<td><code>[useLenientHttpParser]</code></td><td><code>boolean</code></td><td><code>true</code></td>+<td><code>[useInsecureHttpParser]</code></td><td><code>boolean</code></td><td><code>true</code></td></tr><tr><td colspan="3"><p>Node.js HTTP parser is stricter than Browser parsers. This prevents fetching of some websiteswhose servers do not comply with HTTP specs or employ anti-scraping protections due tovarious parse errors, typically <code>invalid header value char</code> error. This option forces- the parser to ignore certain errors and allow those websites to be scraped.+ the parser to ignore certain errors which allows those websites to be scraped.However, it will also open your application to various security vulnerabilities.</p></td></tr></tbody></table>diff --git a/src/utils_request.js b/src/utils_request.jsindex 3b39cc9a7..d6d59a8f1 100644--- a/src/utils_request.js+++ b/src/utils_request.js@@ -39,20 +39,19 @@ const DEFAULT_HTTP_REQUEST_OPTIONS = {* (@apify/http-request)[https://www.npmjs.com/package/@apify/http-request] NPM package.* @property {boolean} [ignoreSslErrors=true]* If set to true, SSL/TLS certificate errors will be ignored.- * @property {boolean} [useLenientHttpParser=true]+ * @property {boolean} [useInsecureHttpParser=true]* Node.js HTTP parser is stricter than Browser parsers. This prevents fetching of some websites* whose servers do not comply with HTTP specs or employ anti-scraping protections due to* various parse errors, typically `invalid header value char` error. This option forces- * the parser to ignore certain errors and allow those websites to be scraped.+ * the parser to ignore certain errors which allows those websites to be scraped.* However, it will also open your application to various security vulnerabilities.*//**- * **IMPORTANT WARNING:** This function uses an insecure version of HTTP parser by default+ * **IMPORTANT:** This function uses an insecure version of HTTP parser by default* and also ignores SSL/TLS errors. This is very useful in scraping, because it allows bypassing- * certain anti-scraping walls, but it also makes it vulnerable to attacks. Do not use this- * function in other than scraping scenarios, unless you set `useLenientHttpParser: false`- * and `ignoreSslErrors: false`.+ * certain anti-scraping walls, but it also exposes some vulnerability. For other than scraping+ * scenarios, please set `useInsecureHttpParser: false` and `ignoreSslErrors: false`.** Sends an HTTP request that looks like a request sent by a web browser,* fully emulating browser's HTTP headers.@@ -91,7 +90,7 @@ export const requestAsBrowser = async (options) => {useMobileVersion = false,abortFunction,ignoreSslErrors = true,- useLenientHttpParser = true,+ useInsecureHttpParser = true,...otherParams} = options;@@ -117,7 +116,7 @@ export const requestAsBrowser = async (options) => {proxyUrl,abortFunction,ignoreSslErrors,- insecureHTTPParser: useLenientHttpParser,+ insecureHTTPParser: useInsecureHttpParser,};try {diff --git a/test/utils_request.test.js b/test/utils_request.test.jsindex 94b88ee98..133bb99ec 100644--- a/test/utils_request.test.js+++ b/test/utils_request.test.js@@ -214,7 +214,7 @@ describe('Apify.utils_request', () => {});try {await requestAsBrowser({- useLenientHttpParser: false,+ useInsecureHttpParser: false,url,});} catch (err) {