Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.
TypeScript apifyautomationcrawlercrawlingheadlessheadless-chromejavascriptnodejsnpmplaywrightpuppeteerscraperscrapingtypescriptweb-crawlerweb-crawlingweb-scraping
Updated docs
Jan Curn88dfd2e65be1
diff --git a/src/dataset.js b/src/dataset.jsindex 2851691db..aae17f2d5 100644--- a/src/dataset.js+++ b/src/dataset.js@@ -661,7 +661,7 @@ export const openDataset = (datasetIdOrName) => {* @param {Object|Array} data Object or array of objects containing data to be stored in the default dataset.* The objects must be serializable to JSON and the JSON representation of each object must be smaller than 9MB.* @returns {Promise} Returns a promise that resolves once the data is saved.- * @see {@linkcode Dataset}+ * @see {@link Dataset}** @memberof module:Apify* @name pushDatadiff --git a/src/request.js b/src/request.jsindex 8a712616c..e4f150100 100644--- a/src/request.js+++ b/src/request.js@@ -5,13 +5,19 @@ import _ from 'underscore';export const computeUniqueKey = (url, keepUrlFragment) => normalizeUrl(url, keepUrlFragment);/**- * Requests class defines a web request to be processed and stores info about error that occurred during the processing.+ * Represents a URL to be crawled, optionally including HTTP method, headers, payload and other metadata.+ * The `Request` object also stores information about errors that occurred during processing of the request.+ *+ * Each `Request` instance has the `uniqueKey` property, which is be either specified+ * manually in constructor or generated automatically from the URL. Two requests with the same `uniqueKey`+ * are considered as pointing to the same web page. This behavior applies to all Apify SDK classes,+ * such as {@linkcode RequestList}, {@linkcode RequestQueue} or {@linkcode PuppeteerCrawler}.** Example use:** ```javascript* const request = new Apify.Request({- * url: 'http://example.com',+ * url: 'http://www.example.com',* headers: { Accept: 'application/json' },* });*@@ -26,17 +32,40 @@ export const computeUniqueKey = (url, keepUrlFragment) => normalizeUrl(url, keep* ```** @param {object} opts- * @param {String} opts.url- * @param {String} [opts.uniqueKey] Unique key identifying request. In not provided then it is computed as normalized URL.+ * @param {String} opts.url URL of the web page to crawl.+ * @param {String} [opts.uniqueKey] A unique key identifying the request.+ * Two requests with the same `uniqueKey` are considered as pointing to the same URL.+ *+ * If `uniqueKey` is not provided, then it is automatically generated by normalizing the URL.+ * For example, the URL of `HTTP://www.EXAMPLE.com/something/` will be generated the `uniqueKey`+ * of `http://www.example.com/something`. The `keepUrlFragment` option+ * determines whether URL hash fragment is included in the `uniqueKey` or not. Beware that the HTTP method+ * and payload is not included in the `uniqueKey`, so requests to the same URL but with different+ * HTTP methods or different POST payloads are all considered equal.+ *+ * You can set `uniqueKey` property to arbitrary non-empty text value in order+ * to override the default behavior and specify which URLs shall be considered equal.* @param {String} [opts.method='GET']- * @param {String|Buffer} [opts.payload] Request payload. If method='GET' then the payload is not allowed.- * @param {Number} [opts.retryCount=0] How many times the url was retried in a case of exception.- * @param {String} [opts.errorMessages] Array of error messages from request processing.- * @param {String} [opts.headers={}] HTTP headers.- * @param {Object} [opts.userData={}] Custom data that user can assign to request.- * @param {Boolean} [opts.keepUrlFragment=false] If false then hash part is removed from url when computing `uniqueKey`.- * @param {String} [opts.ignoreErrors=false] If set to `true` then errors in processing of this will be ignored (request won't be- * retried in a case of an error for example).+ * @param {String|Buffer} [opts.payload]+ * HTTP request payload, e.g. for POST requests.+ * @param {Number} [opts.retryCount=0]+ * Indicates how many times the URL was retried in a case of error.+ * @param {String[]} [opts.errorMessages]+ * An array of error messages from request processing.+ * @param {String} [opts.headers={}]+ * HTTP headers.+ * @param {Object} [opts.userData={}]+ * Custom user data assigned to the request.+ * @param {Boolean} [opts.keepUrlFragment=false]+ * If `false` then hash part is removed from the URL when computing the `uniqueKey` property.+ * For example, this causes the `http://www.example.com#foo` and `http://www.example.com#bar` URLs+ * to have the same `uniqueKey` of `http://www.example.com` and thus the URLs are considered equal.+ * Note that this option only has effect if `uniqueKey` is not set.+ * @param {String} [opts.ignoreErrors=false]+ * If `true` then errors in processing of this will be ignored.+ * For example, the request won't be retried in a case of an error for example.+ * @see {@link RequestList}+ * @see {@link RequestQueue}*/export default class Request {constructor(opts = {}) {@@ -86,7 +115,7 @@ export default class Request {}/**- * Stores information about processing error of this request.+ * Stores information about an error occurred during processing of this request.** @param {Error|String} errorOrMessage Error object or error message to be stored in request.*/