Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.
TypeScript apifyautomationcrawlercrawlingheadlessheadless-chromejavascriptnodejsnpmplaywrightpuppeteerscraperscrapingtypescriptweb-crawlerweb-crawlingweb-scraping
Improve proxy conf docs [skip ci]
Ondra Urbanecb989faa2f1
diff --git a/src/crawlers/cheerio_crawler.js b/src/crawlers/cheerio_crawler.jsindex aa2fa8ede..78d6dc11e 100644--- a/src/crawlers/cheerio_crawler.js+++ b/src/crawlers/cheerio_crawler.js@@ -726,8 +726,13 @@ export default CheerioCrawler;/*** @typedef PrepareRequestInputs- * @property {Request} request Original instance fo the {Request} object. Must be modified in-place.- * @property {Session} [session] The current session+ * @property {Request} request+ * Original instance fo the {Request} object. Must be modified in-place.+ * @property {Session} [session]+ * The current session+ * @property {ProxyInfo} [proxyInfo]+ * An object with information about currently used proxy by the crawler+ * and configured by the {@link ProxyConfiguration} class.*//**diff --git a/src/proxy_configuration.js b/src/proxy_configuration.jsindex 984d6c83f..cfd35517a 100644--- a/src/proxy_configuration.js+++ b/src/proxy_configuration.js@@ -217,7 +217,7 @@ export class ProxyConfiguration {** All the HTTP requests going through the proxy with the same session identifier* will use the same target proxy server (i.e. the same IP address).- * The identifier can only contain the following characters: `0-9`, `a-z`, `A-Z`, `"."`, `"_"` and `"~"`.+ * The identifier must not be longer than 50 characters and include only the following: `0-9`, `a-z`, `A-Z`, `"."`, `"_"` and `"~"`.* @return {ProxyInfo} represents information about used proxy and its configuration.*/newProxyInfo(sessionId) {@@ -238,14 +238,14 @@ export class ProxyConfiguration {}/**- * Returns a new stringified URL of the proxy server.+ * Returns a new proxy URL based on provided configuration options and the `sessionId` parameter.* @param {string} [sessionId]* Represents the identifier of user {@link Session} that can be managed by the {@link SessionPool} or* you can use the Apify Proxy [Session](https://docs.apify.com/proxy/datacenter-proxy#session-persistence) identifier.** All the HTTP requests going through the proxy with the same session identifier* will use the same target proxy server (i.e. the same IP address).- * The identifier can only contain the following characters: `0-9`, `a-z`, `A-Z`, `"."`, `"_"` and `"~"`.+ * The identifier must not be longer than 50 characters and include only the following: `0-9`, `a-z`, `A-Z`, `"."`, `"_"` and `"~"`.* @return {string} represents the proxy URL.*/newUrl(sessionId) {