Skip to content

Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.

Fix/in progress (#96)

Jan Čurnddf5e9960c0d
diff --git a/CHANGELOG.md b/CHANGELOG.md
index 9913b0efd..b188c8335 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -1,3 +1,7 @@
+xxx
+===================
+- Bugfix in `RequestList` - skip invalid in-progress entries when restoring state
+
0.5.31 / 2018-05-11
===================
- Bugfix in `Apify.utils.puppeteer.injectXxx` functions
@@ -13,7 +17,7 @@
0.5.27 / 2018-04-30
===================
-- Added `Apify.isAtHome()` function that returns `true` when code is running at Apify platform and `false` otherwise
+- Added `Apify.isAtHome()` function that returns `true` when code is running on Apify platform and `false` otherwise
(for example locally).
- Added `ignoreMainProcess` parameter to `AutoscaledPool`. Check documentation for more info.
- `pageOpsTimeoutMillis` of `PuppeteerCrawler` increased to 300 seconds.
diff --git a/README.md b/README.md
index d159bc8e7..c32fa1aee 100644
--- a/README.md
+++ b/README.md
@@ -9,7 +9,7 @@
either locally or running on <a href="https://www.apify.com/docs/actor" target="_blank">Apify Actor</a> -
a serverless computing platform that enables execution of arbitrary code in the cloud.
- The package provides helper functions to launch web browsers with proxies, access the storage etc. Note that the usage of the package is optional, you can create acts at Apify platform without it.
+ The package provides helper functions to launch web browsers with proxies, access the storage etc. Note that the usage of the package is optional, you can create acts on Apify platform without it.
For more information about the Apify Actor platform, please see https://www.apify.com/docs/actor
</div>
@@ -211,7 +211,7 @@ For more information on Puppeteer see its <a href="https://github.com/GoogleChro
### Storage
-Apify package provides 3 storage types for commons use cases both locally and at Apify platform.
+Apify package provides 3 storage types for commons use cases both locally and on Apify platform.
#### Key-value store
@@ -429,7 +429,7 @@ These examples show how to scrape data from a fix list of urls using
### Call to another act
-This example shows how to call another act at Apify plarform - in this case `apify/send-mail`
+This example shows how to call another act on Apify platform - in this case `apify/send-mail`
to send email.
<a href="https://github.com/apifytech/apify-js/tree/master/examples/call_another_act.js">Check source code here</a>
@@ -438,7 +438,7 @@ to send email.
This example shows an act that has short runtime - just few seconds. It opens a webpage
http://goldengatebridge75.org/news/webcam.html that contains webcam stream from Golden Gate
-bridge, takes a screenshot and saves it as output. This makes act executable at Apify platform
+bridge, takes a screenshot and saves it as output. This makes act executable on Apify platform
synchronously with a single request that also returns its output.
Example is shared in library under https://www.apify.com/apify/example-golden-gate-webcam
diff --git a/examples/synchronous_run.js b/examples/synchronous_run.js
index b08bb67a7..731e703df 100644
--- a/examples/synchronous_run.js
+++ b/examples/synchronous_run.js
@@ -1,7 +1,7 @@
/**
* This example shows shows an act that has short runtime - just few seconds. It opens a webpage
* http://goldengatebridge75.org/news/webcam.html that contains webcam stream from Golden Gate
- * bridge, takes a screenshot and saves it as output. This makes act executable at Apify platform
+ * bridge, takes a screenshot and saves it as output. This makes act executable on Apify platform
* synchronously with a single request that also returns its output.
*
* Example is shared in library under https://www.apify.com/apify/example-golden-gate-webcam
diff --git a/src/autoscaled_pool.js b/src/autoscaled_pool.js
index 66f7e2eda..f37736332 100644
--- a/src/autoscaled_pool.js
+++ b/src/autoscaled_pool.js
@@ -191,7 +191,7 @@ export default class AutoscaledPool {
if (isAtHome()) {
this.memCheckInterval = setInterval(() => this._autoscale(), MEM_CHECK_INTERVAL_MILLIS);
} else {
- log.warning('Autoscaling feature is currently available only when running at Apify platform!');
+ log.warning('Autoscaling feature is currently available only when running on Apify platform!');
log.warning('Use `minConcurrency` parameter if you need to test multiple requests in parallel.');
log.warning('This feature will be enabled soon!');
}
diff --git a/src/request_list.js b/src/request_list.js
index b2214b31d..f9241f069 100644
--- a/src/request_list.js
+++ b/src/request_list.js
@@ -167,12 +167,37 @@ export default class RequestList {
throw new Error('The state object is not consistent with RequestList: the order of URLs seems to have changed.');
}
+ const deleteFromInProgress = [];
_.keys(state.inProgress).forEach((uniqueKey) => {
- if (typeof this.uniqueKeyToIndex[uniqueKey] !== 'number') {
+ const index = this.uniqueKeyToIndex[uniqueKey];
+ if (typeof index !== 'number') {
throw new Error('The state object is not consistent with RequestList: unknown uniqueKey is present in the state.');
}
+ if (index >= state.nextIndex) {
+ deleteFromInProgress.push(uniqueKey);
+ }
});
+ // WORKAROUND:
+ // It happened to some users that state object contained something like:
+ // {
+ // "nextIndex": 11308,
+ // "nextUniqueKey": "https://www.anychart.com",
+ // "inProgress": {
+ // "https://www.ams360.com": true,
+ // ...
+ // "https://www.anychart.com": true,
+ // }
+ // Which then caused error "The request is not being processed (uniqueKey: https://www.anychart.com)"
+ // As a workaround, we just remove all inProgress requests whose index >= nextIndex,
+ // since they will be crawled again.
+ if (deleteFromInProgress.length) {
+ log.warning('RequestList\'s in-progress field is not consistent, skipping invalid in-progress entries', { deleteFromInProgress });
+ _.each(deleteFromInProgress, (uniqueKey) => {
+ delete state.inProgress[uniqueKey];
+ });
+ }
+
this.nextIndex = state.nextIndex;
this.inProgress = state.inProgress;
@@ -350,7 +375,7 @@ export default class RequestList {
regex,
fetchedCount,
importedCount,
- dupliciteCount: fetchedCount - importedCount,
+ duplicateCount: fetchedCount - importedCount,
sample: JSON.stringify(urlsArr.slice(0, 5)),
});
} else {
diff --git a/src/utils.js b/src/utils.js
index 7b540bc16..c12390d04 100644
--- a/src/utils.js
+++ b/src/utils.js
@@ -281,7 +281,7 @@ export const getTypicalChromeExecutablePath = () => {
};
/**
- * Returns `true` when code is running at Apify platform and `false` otherwise (for example locally).
+ * Returns `true` when code is running on Apify platform and `false` otherwise (for example locally).
*
* @returns {Boolean}
*
diff --git a/test/request_list.js b/test/request_list.js
index a920f77e1..8926bfe00 100644
--- a/test/request_list.js
+++ b/test/request_list.js
@@ -1,4 +1,5 @@
import 'babel-polyfill';
+import _ from 'underscore';
import chai, { expect } from 'chai';
import chaiAsPromised from 'chai-as-promised';
import request from 'request-promise';
@@ -450,4 +451,46 @@ describe('Apify.RequestList', () => {
mock.verify();
mock.restore();
});
+
+ it('handles correctly inconsistent inProgress fields in state', async () => {
+ // NOTE: This is a test for the deleteFromInProgress hotfix - see RequestList.initialize()
+
+ const sources = [
+ { url: 'https://www.ams360.com' },
+ { url: 'https://www.anybus.com' },
+ { url: 'https://www.anychart.com' },
+ { url: 'https://www.example.com' },
+ ];
+
+ const state = {
+ nextIndex: 2,
+ nextUniqueKey: 'https://www.anychart.com',
+ inProgress: {
+ 'https://www.ams360.com': true,
+ 'https://www.anybus.com': true,
+ 'https://www.anychart.com': true,
+ },
+ };
+
+ const requestList = new Apify.RequestList({
+ sources,
+ state,
+ });
+
+ await requestList.initialize();
+
+ // Get requests from list
+ let reqs = [];
+ for (let i = 0; i < 5; i++) {
+ const request = await requestList.fetchNextRequest(); // eslint-disable-line
+ if (!request) break;
+ reqs.push(request);
+ }
+
+ reqs = _.shuffle(reqs);
+
+ for (let i = 0; i < reqs.length; i++) {
+ await requestList.reclaimRequest(reqs[i]); // eslint-disable-line
+ }
+ });
});