Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.
TypeScript apifyautomationcrawlercrawlingheadlessheadless-chromejavascriptnodejsnpmplaywrightpuppeteerscraperscrapingtypescriptweb-crawlerweb-crawlingweb-scraping
Renamed RequestOperationInfo to QueueOperationInfo
Jan Curna39e3079e4bc
diff --git a/CHANGELOG.md b/CHANGELOG.mdindex ea985307d..14664052c 100644--- a/CHANGELOG.md+++ b/CHANGELOG.md@@ -1,7 +1,8 @@xxx===================- Fixed `RequestQueue.addRequest()` to gracefully handle invalid URLs-- Added `request` field to `RequestOperationInfo`+- Renamed `RequestOperationInfo` to `QueueOperationInfo`+- Added `request` field to `QueueOperationInfo`0.8.17 / 2018-10-30===================diff --git a/docs/api/RequestQueue.md b/docs/api/RequestQueue.mdindex 82640eadd..b21523c48 100644--- a/docs/api/RequestQueue.md+++ b/docs/api/RequestQueue.md@@ -66,23 +66,23 @@ await queue.reclaimRequest(request2);* [RequestQueue](requestqueue)- * [`.addRequest(request, [options])`](#RequestQueue+addRequest) ⇒ [<code>RequestOperationInfo</code>](../typedefs/requestoperationinfo)+ * [`.addRequest(request, [options])`](#RequestQueue+addRequest) ⇒ [<code>QueueOperationInfo</code>](../typedefs/queueoperationinfo)* [`.getRequest(requestId)`](#RequestQueue+getRequest) ⇒ [<code>Promise<Request></code>](request)* [`.fetchNextRequest()`](#RequestQueue+fetchNextRequest) ⇒ [<code>Promise<Request></code>](request)- * [`.markRequestHandled(request)`](#RequestQueue+markRequestHandled) ⇒ [<code>Promise<RequestOperationInfo></code>](../typedefs/requestoperationinfo)- * [`.reclaimRequest(request, [options])`](#RequestQueue+reclaimRequest) ⇒ [<code>Promise<RequestOperationInfo></code>](../typedefs/requestoperationinfo)+ * [`.markRequestHandled(request)`](#RequestQueue+markRequestHandled) ⇒ [<code>Promise<QueueOperationInfo></code>](../typedefs/queueoperationinfo)+ * [`.reclaimRequest(request, [options])`](#RequestQueue+reclaimRequest) ⇒ [<code>Promise<QueueOperationInfo></code>](../typedefs/queueoperationinfo)* [`.isEmpty()`](#RequestQueue+isEmpty) ⇒ <code>Promise<Boolean></code>* [`.isFinished()`](#RequestQueue+isFinished) ⇒ <code>Promise<Boolean></code>* [`.delete()`](#RequestQueue+delete) ⇒ <code>Promise</code><a name="RequestQueue+addRequest"></a>-## `requestQueue.addRequest(request, [options])` ⇒ [<code>RequestOperationInfo</code>](../typedefs/requestoperationinfo)+## `requestQueue.addRequest(request, [options])` ⇒ [<code>QueueOperationInfo</code>](../typedefs/queueoperationinfo)Adds a request to the queue.If a request with the same `uniqueKey` property is already present in the queue,it will not be updated. You can find out whether this happened from the resulting-[`RequestOperationInfo`](../typedefs/requestoperationinfo) object.+[`QueueOperationInfo`](../typedefs/queueoperationinfo) object.<table><thead>@@ -133,7 +133,7 @@ Returns next request in the queue to be processed.<a name="RequestQueue+markRequestHandled"></a>-## `requestQueue.markRequestHandled(request)` ⇒ [<code>Promise<RequestOperationInfo></code>](../typedefs/requestoperationinfo)+## `requestQueue.markRequestHandled(request)` ⇒ [<code>Promise<QueueOperationInfo></code>](../typedefs/queueoperationinfo)Marks request handled after successful processing.<table>@@ -151,7 +151,7 @@ Marks request handled after successful processing.</table><a name="RequestQueue+reclaimRequest"></a>-## `requestQueue.reclaimRequest(request, [options])` ⇒ [<code>Promise<RequestOperationInfo></code>](../typedefs/requestoperationinfo)+## `requestQueue.reclaimRequest(request, [options])` ⇒ [<code>Promise<QueueOperationInfo></code>](../typedefs/queueoperationinfo)Reclaims failed request back to the queue,so that it can be processed later again.diff --git a/docs/api/puppeteer.md b/docs/api/puppeteer.mdindex 9843ff673..236dde1c5 100644--- a/docs/api/puppeteer.md+++ b/docs/api/puppeteer.md@@ -27,7 +27,7 @@ await puppeteer.injectJQuery(page);* [`.injectFile(page, filePath)`](#puppeteer.injectFile) ⇒ <code>Promise</code>* [`.injectJQuery(page)`](#puppeteer.injectJQuery) ⇒ <code>Promise</code>* [`.injectUnderscore(page)`](#puppeteer.injectUnderscore) ⇒ <code>Promise</code>- * [`.enqueueLinks(page, selector, requestQueue, [pseudoUrls])`](#puppeteer.enqueueLinks) ⇒ <code>Promise<Array<RequestOperationInfo>></code>+ * [`.enqueueLinks(page, selector, requestQueue, [pseudoUrls])`](#puppeteer.enqueueLinks) ⇒ <code>Promise<Array<QueueOperationInfo>></code>* [`.blockResources(page, resourceTypes)`](#puppeteer.blockResources) ⇒ <code>Promise</code>* [`.cacheResponses(page, cache, responseUrlRules)`](#puppeteer.cacheResponses) ⇒ <code>Promise</code>* [`.compileScript(scriptString, context)`](#puppeteer.compileScript) ⇒ <code>function</code>@@ -125,7 +125,7 @@ libraries included by the page that use the same variable.</table><a name="puppeteer.enqueueLinks"></a>-## `puppeteer.enqueueLinks(page, selector, requestQueue, [pseudoUrls])` ⇒ <code>Promise<Array<RequestOperationInfo>></code>+## `puppeteer.enqueueLinks(page, selector, requestQueue, [pseudoUrls])` ⇒ <code>Promise<Array<QueueOperationInfo>></code>Finds HTML elements matching a CSS selector, clicks the elements and if a redirect is triggered and destination URL matchesone of the provided [`PseudoUrl`](pseudourl)s, then the function enqueues that URL to a given request queue.To create a Request object function uses `requestTemplate` from a matching [`PseudoUrl`](pseudourl).@@ -133,7 +133,7 @@ To create a Request object function uses `requestTemplate` from a matching [`Pse*WARNING*: This is work in progress. Currently the function doesn't click elements and only takes their `href` attribute and sois working only for link (`a`) elements, but not for buttons or JavaScript links.-**Returns**: <code>Promise<Array<RequestOperationInfo>></code> - Promise that resolves to an array of [`RequestOperationInfo`](../typedefs/requestoperationinfo) objects.+**Returns**: <code>Promise<Array<QueueOperationInfo>></code> - Promise that resolves to an array of [`QueueOperationInfo`](../typedefs/queueoperationinfo) objects.<table><thead><tr>@@ -262,7 +262,7 @@ Custom context may also be provided using the `context` parameter. To improve semake sure to only pass the really necessary objects to the context. Preferably makingsecured copies beforehand.-**Returns**: <code>function</code> - `async ({ page, request }) => { scriptString }`+**Returns**: <code>function</code> - `async ({ page, request }) => { scriptString }`<table><thead><tr>diff --git a/docs/typedefs/RequestOperationInfo.md b/docs/typedefs/QueueOperationInfo.mdsimilarity index 89%rename from docs/typedefs/RequestOperationInfo.mdrename to docs/typedefs/QueueOperationInfo.mdindex 507f68a3c..1af3f2a52 100644--- a/docs/typedefs/RequestOperationInfo.md+++ b/docs/typedefs/QueueOperationInfo.md@@ -1,8 +1,8 @@----id: requestoperationinfo-title: RequestOperationInfo+id: queueoperationinfo+title: QueueOperationInfo----<a name="RequestOperationInfo"></a>+<a name="QueueOperationInfo"></a>A helper class that is used to report results from the[`Apify.utils.puppeteer.enqueueLinks()`](../api/puppeteer#puppeteer.enqueueLinks) function.diff --git a/src/puppeteer_utils.js b/src/puppeteer_utils.jsindex 25aa780e9..fb2defec6 100644--- a/src/puppeteer_utils.js+++ b/src/puppeteer_utils.js@@ -129,11 +129,11 @@ const enqueueRequestsFromClickableElements = async (page, selector, purls, requeconst urls = await page.$$eval(selector, getHrefs);const requests = urls.filter(matchesPseudoUrl).map(url => new Request(Object.assign({ url }, requestOpts)));- const requestOperationInfos = [];+ const queueOperationInfos = [];for (const request of requests) {- requestOperationInfos.push(await requestQueue.addRequest(request));+ queueOperationInfos.push(await requestQueue.addRequest(request));}- return requestOperationInfos;+ return queueOperationInfos;};/**@@ -197,8 +197,8 @@ let logDeprecationWarning = true;* or an array of Strings or Objects from which the {@link PseudoUrl}s should be constructed* The Objects must include at least a `purl` property, which holds a pseudoUrl string.* All remaining keys will be used as the `requestTemplate` argument of the {@link PseudoUrl} constructor.- * @return {Promise<RequestOperationInfo[]>}- * Promise that resolves to an array of {@link RequestOperationInfo} objects.+ * @return {Promise<QueueOperationInfo[]>}+ * Promise that resolves to an array of {@link QueueOperationInfo} objects.* @memberOf puppeteer*/const enqueueLinks = async (page, selector, requestQueue, pseudoUrls = []) => {@@ -238,11 +238,11 @@ const enqueueLinks = async (page, selector, requestQueue, pseudoUrls = []) => {requests = urls.map(url => ({ url }));}- const requestOperationInfos = [];+ const queueOperationInfos = [];for (const request of requests) {- requestOperationInfos.push(await requestQueue.addRequest(request));+ queueOperationInfos.push(await requestQueue.addRequest(request));}- return requestOperationInfos;+ return queueOperationInfos;};/**diff --git a/src/request_queue.js b/src/request_queue.jsindex 0ed3ddd39..ba8312101 100644--- a/src/request_queue.js+++ b/src/request_queue.js@@ -113,14 +113,12 @@ const getRequestId = (uniqueKey) => {: str;};-// TODO: Let's rename RequestOperationInfo to QueueOperationInfo, it makes more sense, it's not used anywhere-/*** A helper class that is used to report results from various* [`RequestQueue`](../api/requestqueue) functions as well as* [`Apify.utils.puppeteer.enqueueLinks()`](../api/puppeteer#puppeteer.enqueueLinks).*- * @typedef {Object} RequestOperationInfo+ * @typedef {Object} QueueOperationInfo* @property {Boolean} wasAlreadyPresent Indicates if request was already present in the queue.* @property {Boolean} wasAlreadyHandled Indicates if request was already marked as handled.* @property {String} requestId The ID of the added request@@ -215,12 +213,12 @@ export class RequestQueue {** If a request with the same `uniqueKey` property is already present in the queue,* it will not be updated. You can find out whether this happened from the resulting- * {@link RequestOperationInfo} object.+ * {@link QueueOperationInfo} object.** @param {Request|Object} request {@link Request} object, or an object to construct a `Request` instance from.* @param {Object} [options]* @param {Boolean} [options.forefront=false] If `true`, the request will be added to the foremost position in the queue.- * @return {RequestOperationInfo}+ * @return {QueueOperationInfo}*/addRequest(request, options = {}) {const { forefront, request: newRequest } = validateAddRequestParams(request, options);@@ -250,19 +248,19 @@ export class RequestQueue {queueId: this.queueId,forefront,})- .then((requestOperationInfo) => {- const { requestId, wasAlreadyHandled } = requestOperationInfo;+ .then((queueOperationInfo) => {+ const { requestId, wasAlreadyHandled } = queueOperationInfo;- this._cacheRequest(cacheKey, requestOperationInfo);+ this._cacheRequest(cacheKey, queueOperationInfo);if (forefront && !this.requestIdsInProgress[requestId] && !wasAlreadyHandled) {this.queueHeadDict.add(requestId, requestId, true);}// TODO: Why not set request.id to cachedInfo.id???- requestOperationInfo.request = request;+ queueOperationInfo.request = request;- return requestOperationInfo;+ return queueOperationInfo;});}@@ -320,7 +318,7 @@ export class RequestQueue {* Marks request handled after successful processing.** @param {Request} request- * @return {Promise<RequestOperationInfo>}+ * @return {Promise<QueueOperationInfo>}*/markRequestHandled(request) {validateMarkRequestHandledParams(request);@@ -336,13 +334,13 @@ export class RequestQueue {request,queueId: this.queueId,})- .then((requestOperationInfo) => {+ .then((queueOperationInfo) => {this._removeFromInProgress(request.id);- this._cacheRequest(getRequestId(request.uniqueKey), requestOperationInfo);+ this._cacheRequest(getRequestId(request.uniqueKey), queueOperationInfo);- requestOperationInfo.request = request;+ queueOperationInfo.request = request;- return requestOperationInfo;+ return queueOperationInfo;});}@@ -355,7 +353,7 @@ export class RequestQueue {* @param {Boolean} [options.forefront=false]* If `true` then requests get returned to the start of the queue* and to the back of the queue otherwise.- * @return {Promise<RequestOperationInfo>}+ * @return {Promise<QueueOperationInfo>}*/reclaimRequest(request, options = {}) {const { forefront } = validateReclaimRequestParams(request, options);@@ -366,15 +364,15 @@ export class RequestQueue {queueId: this.queueId,forefront,})- .then((requestOperationInfo) => {+ .then((queueOperationInfo) => {this._removeFromInProgress(request.id);- this._cacheRequest(getRequestId(request.uniqueKey), requestOperationInfo);+ this._cacheRequest(getRequestId(request.uniqueKey), queueOperationInfo);if (forefront) this.queueHeadDict.add(request.id, request.id, true);- requestOperationInfo.request = request;+ queueOperationInfo.request = request;- return requestOperationInfo;+ return queueOperationInfo;});}@@ -411,15 +409,15 @@ export class RequestQueue {** @ignore*/- _cacheRequest(cacheKey, requestOperationInfo) {+ _cacheRequest(cacheKey, queueOperationInfo) {checkParamOrThrow(cacheKey, 'cacheKey', 'String');- checkParamOrThrow(requestOperationInfo, 'requestOperationInfo', 'Object');- checkParamOrThrow(requestOperationInfo.requestId, 'requestOperationInfo.requestId', 'String');- checkParamOrThrow(requestOperationInfo.wasAlreadyHandled, 'requestOperationInfo.wasAlreadyHandled', 'Boolean');+ checkParamOrThrow(queueOperationInfo, 'queueOperationInfo', 'Object');+ checkParamOrThrow(queueOperationInfo.requestId, 'queueOperationInfo.requestId', 'String');+ checkParamOrThrow(queueOperationInfo.wasAlreadyHandled, 'queueOperationInfo.wasAlreadyHandled', 'Boolean');this.requestsCache.add(cacheKey, {- id: requestOperationInfo.requestId,- isHandled: requestOperationInfo.wasAlreadyHandled,+ id: queueOperationInfo.requestId,+ isHandled: queueOperationInfo.wasAlreadyHandled,});}diff --git a/website/i18n/en.json b/website/i18n/en.jsonindex c711b23fc..5586682e0 100644--- a/website/i18n/en.json+++ b/website/i18n/en.json@@ -119,8 +119,8 @@"typedefs/paginationlist": {"title": "PaginationList"},- "typedefs/requestoperationinfo": {- "title": "RequestOperationInfo"+ "typedefs/queueoperationinfo": {+ "title": "QueueOperationInfo"}},"links": {diff --git a/website/sidebars.json b/website/sidebars.jsonindex f1bb14815..4b53541e5 100644--- a/website/sidebars.json+++ b/website/sidebars.json@@ -49,7 +49,7 @@"typedefs/apifycallerror","typedefs/launchpuppeteeroptions","typedefs/paginationlist",- "typedefs/requestoperationinfo"+ "typedefs/queueoperationinfo"]}}