Skip to content

Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.

fix(core): add requests from URL list (`requestsFromUrl`) to the queue in batches

Martin Adámek418fbf89d868
diff --git a/packages/core/src/storages/request_queue.ts b/packages/core/src/storages/request_queue.ts
index c73a43d6f..fddf9fde0 100644
--- a/packages/core/src/storages/request_queue.ts
+++ b/packages/core/src/storages/request_queue.ts
@@ -904,18 +904,18 @@ export class RequestQueue {
*/
protected async _addFetchedRequests(source: InternalSource, fetchedRequests: RequestOptions[], options: RequestQueueOperationOptions) {
const { requestsFromUrl, regex } = source;
- const { processedRequests } = await this.addRequests(fetchedRequests, options);
+ const { addedRequests } = await this.addRequestsBatched(fetchedRequests, options);
this.log.info('Fetched and loaded Requests from a remote resource.', {
requestsFromUrl,
regex,
fetchedCount: fetchedRequests.length,
- importedCount: processedRequests.length,
- duplicateCount: fetchedRequests.length - processedRequests.length,
+ importedCount: addedRequests.length,
+ duplicateCount: fetchedRequests.length - addedRequests.length,
sample: JSON.stringify(fetchedRequests.slice(0, 5)),
});
- return processedRequests;
+ return addedRequests;
}
/**