Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.
TypeScript apifyautomationcrawlercrawlingheadlessheadless-chromejavascriptnodejsnpmplaywrightpuppeteerscraperscrapingtypescriptweb-crawlerweb-crawlingweb-scraping
builded docs.
Petr Patekd62e184d49f2
diff --git a/docs/api/utils.md b/docs/api/utils.mdindex 3e446aea0..c07b4689e 100644--- a/docs/api/utils.md+++ b/docs/api/utils.md@@ -141,6 +141,8 @@ This function is useful for web scraping of websites that send the full HTML inThanks to this function, the target web server has no simple way to find out the requesthasn't been sent by a full web browser. Using a headless browser for such requestsis an order of magnitude more resource-intensive than this function.+By default tt aborts all requests that returns 406 status codes or non-HTML content-types.+You can override this behavior by passing custom `abortFunction`.Currently, the function sends requests the same way as Firefox web browser does.In the future, it might add support for other browsers too.@@ -189,6 +191,13 @@ so see it for more details.</tr><tr><td colspan="3"><p>If <code>true</code>, the function uses User-Agent of a mobile browser.</p>+</td></tr><tr>+<td><code>[options.abortFunction]</code></td><td></td>+</tr>+<tr>+<td colspan="3"><p>Function accepts <code>response</code> object as a single parameter and should return true or false.+ If function returns true request gets aborted. This function is passed to the+ (@apify/http-request)[<a href="https://www.npmjs.com/package/@apify/http-request%5D">https://www.npmjs.com/package/@apify/http-request]</a> NPM package.</p></td></tr></tbody></table><a name="utils.sleep"></a>