Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.
TypeScript apifyautomationcrawlercrawlingheadlessheadless-chromejavascriptnodejsnpmplaywrightpuppeteerscraperscrapingtypescriptweb-crawlerweb-crawlingweb-scraping
docs: improve documentation around `addRequests` and `addRequestsBatched`
Martin Adámek522c44b95453
diff --git a/docs/introduction/02-first-crawler.mdx b/docs/introduction/02-first-crawler.mdxindex 92ddaef74..e7f3dd490 100644--- a/docs/introduction/02-first-crawler.mdx+++ b/docs/introduction/02-first-crawler.mdx@@ -123,6 +123,12 @@ await crawler.run(['https://crawlee.dev']);When you run this code, you'll see exactly the same output as with the earlier, longer example. The `RequestQueue` is still there, it's just managed by the crawler automatically.+:::info++This method not only makes the code shorter, it will help with performance too! It will wait only for the initial batch of 1000 requests to be added to the queue before resolving, which means the processing will start almost instantly. After that, it will continue adding the rest of the requests in the background (again, in batches of 1000 items, once every second).++:::+## Next stepsNext, you'll learn about crawling links. That means finding new URLs on the pages you crawl and adding them to the `RequestQueue` for the crawler to visit.diff --git a/packages/basic-crawler/src/internals/basic-crawler.ts b/packages/basic-crawler/src/internals/basic-crawler.tsindex da07be54f..a8acc75b1 100644--- a/packages/basic-crawler/src/internals/basic-crawler.ts+++ b/packages/basic-crawler/src/internals/basic-crawler.ts@@ -981,6 +981,8 @@ export class BasicCrawler<Context extends CrawlingContext = BasicCrawlingContext* the batches via `waitBetweenBatchesMillis`. If you want to wait for all batches to be added to the queue, you can use* the `waitForAllRequestsToBeAdded` promise you get in the response object.*+ * This is an alias for calling `addRequestsBatched()` on the implicit `RequestQueue` for this crawler instance.+ ** @param requests The requests to add* @param options Options for the request queue*/diff --git a/packages/core/src/storages/request_provider.ts b/packages/core/src/storages/request_provider.tsindex 59735c61f..8d8539e0d 100644--- a/packages/core/src/storages/request_provider.ts+++ b/packages/core/src/storages/request_provider.ts@@ -181,9 +181,12 @@ export abstract class RequestProvider implements IStorage {}/**- * Adds requests to the queue in batches of 25.+ * Adds requests to the queue in batches of 25. This method will wait till all the requests are added+ * to the queue before resolving. You should prefer using `queue.addRequestsBatched()` or `crawler.addRequests()`+ * if you don't want to block the processing, as those methods will only wait for the initial 1000 requests,+ * start processing right after that happens, and continue adding more in the background.*- * If a request that is passed in is already present due to its `uniqueKey` property being the same,+ * If a request passed in is already present due to its `uniqueKey` property being the same,* it will not be updated. You can find out whether this happened by finding the request in the resulting* {@apilink BatchAddRequestsResult} object.*@@ -301,7 +304,7 @@ export abstract class RequestProvider implements IStorage {/*** Adds requests to the queue in batches. By default, it will resolve after the initial batch is added, and continue- * adding the rest in background. You can configure the batch size via `batchSize` option and the sleep time in between+ * adding the rest in the background. You can configure the batch size via `batchSize` option and the sleep time in between* the batches via `waitBetweenBatchesMillis`. If you want to wait for all batches to be added to the queue, you can use* the `waitForAllRequestsToBeAdded` promise you get in the response object.*diff --git a/website/versioned_docs/version-3.10/introduction/02-first-crawler.mdx b/website/versioned_docs/version-3.10/introduction/02-first-crawler.mdxindex 92ddaef74..e7f3dd490 100644--- a/website/versioned_docs/version-3.10/introduction/02-first-crawler.mdx+++ b/website/versioned_docs/version-3.10/introduction/02-first-crawler.mdx@@ -123,6 +123,12 @@ await crawler.run(['https://crawlee.dev']);When you run this code, you'll see exactly the same output as with the earlier, longer example. The `RequestQueue` is still there, it's just managed by the crawler automatically.+:::info++This method not only makes the code shorter, it will help with performance too! It will wait only for the initial batch of 1000 requests to be added to the queue before resolving, which means the processing will start almost instantly. After that, it will continue adding the rest of the requests in the background (again, in batches of 1000 items, once every second).++:::+## Next stepsNext, you'll learn about crawling links. That means finding new URLs on the pages you crawl and adding them to the `RequestQueue` for the crawler to visit.