Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.
TypeScript apifyautomationcrawlercrawlingheadlessheadless-chromejavascriptnodejsnpmplaywrightpuppeteerscraperscrapingtypescriptweb-crawlerweb-crawlingweb-scraping
Regenerated docs
Jan Curn8807c93e022a
diff --git a/docs/api/BasicCrawler.md b/docs/api/BasicCrawler.mdindex 9b642a394..022050446 100644--- a/docs/api/BasicCrawler.md+++ b/docs/api/BasicCrawler.md@@ -84,7 +84,8 @@ await crawler.run();<pre><code>{request: Request,autoscaledPool: AutoscaledPool-}</code></pre><p> where the <a href="request"><code>Request</code></a> instance represents the URL to crawl.</p>+}+</code></pre><p> where the <a href="request"><code>Request</code></a> instance represents the URL to crawl.</p><p> The function must return a promise, which is then awaited by the crawler.</p><p> If the function throws an exception, the crawler will try to re-crawl therequest later, up to <code>option.maxRequestRetries</code> times.@@ -120,7 +121,8 @@ await crawler.run();<pre><code>{request: Request,error: Error,-}</code></pre><p> where the <a href="request"><code>Request</code></a> instance corresponds to the failed request, and the <code>Error</code> instance+}+</code></pre><p> where the <a href="request"><code>Request</code></a> instance corresponds to the failed request, and the <code>Error</code> instancerepresents the last error thrown during processing of the request.</p><p> See<a href="https://github.com/apifytech/apify-js/blob/master/src/crawlers/basic_crawler.js#L11" target="_blank">source code</a>