Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.
TypeScript apifyautomationcrawlercrawlingheadlessheadless-chromejavascriptnodejsnpmplaywrightpuppeteerscraperscrapingtypescriptweb-crawlerweb-crawlingweb-scraping
Build docs
Ondra Urban177391a1c310
diff --git a/docs/api/Request.md b/docs/api/Request.mdindex 5a0e68a60..7f3747720 100644--- a/docs/api/Request.md+++ b/docs/api/Request.md@@ -69,7 +69,7 @@ const foo = request.userData.foo;<tr><td colspan="3"><p>HTTP method, e.g. <code>GET</code> or <code>POST</code>.</p></td></tr><tr>-<td><code>payload</code></td><td><code>String</code></td>+<td><code>payload</code></td><td><code>String</code> | <code>Buffer</code></td></tr><tr><td colspan="3"><p>HTTP request payload, e.g. for POST requests.</p>@@ -139,13 +139,12 @@ const foo = request.userData.foo;<tr><td colspan="3"><p>A unique key identifying the request.Two requests with the same <code>uniqueKey</code> are considered as pointing to the same URL.</p>-<p>If <code>uniqueKey</code> is not provided, then it is automatically generated by normalizing the URL.+<p>If <code>uniqueKey</code> is not provided, then for GET requests it is automatically generated by normalizing the URL.For example, the URL of <code>HTTP://www.EXAMPLE.com/something/</code> will produce the <code>uniqueKey</code>of <code>http://www.example.com/something</code>.</p>-<p>The <code>keepUrlFragment</code> option determines whether URL hash fragment is included in the <code>uniqueKey</code> or not.-Beware that the HTTP method and payload are not included in the <code>uniqueKey</code>,-so requests to the same URL but with different HTTP methods or different POST payloads-are all considered equal and will be deduplicated by <a href="requestlist"><code>RequestList</code></a> and <a href="requestqueue"><code>RequestQueue</code></a>.</p>+<p>The <code>keepUrlFragment</code> option determines whether URL hash fragment is included in the <code>uniqueKey</code> or not.</p>+<p>For other than GET requests, method and payload are included in the generated <code>uniqueKey</code>,+producing a <code>uniqueKey</code> in the following format: <code>METHOD(payloadHash):normalizedUrl</code>.</p><p>Pass an arbitrary non-empty text value to the <code>uniqueKey</code> propertyto override the default behavior and specify which URLs shall be considered equal.</p></td></tr><tr>