Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.
TypeScript apifyautomationcrawlercrawlingheadlessheadless-chromejavascriptnodejsnpmplaywrightpuppeteerscraperscrapingtypescriptweb-crawlerweb-crawlingweb-scraping
Update changelog, bump version, fix doctypes
Ondra Urbanc777a0ea3c12
diff --git a/CHANGELOG.md b/CHANGELOG.mdindex c66017024..977eb62db 100644--- a/CHANGELOG.md+++ b/CHANGELOG.md@@ -1,3 +1,22 @@+0.21.4 / 2020/09/02+====================+The statistics that you may remember from logs are now persisted in key-value store,+so you won't lose count when your actor restarts. We've also added a lot of useful+stats in there which can be useful to you after a run finishes. Besides that,+we fixed some bugs and annoyances and improved the TypeScript experience a bit.++- Add persistence to `Statistics` class and automatically persist it in `BasicCrawler`.+- Fix issue where inaccessible Apify Proxy would cause `ProxyConfiguration` to throw+ a timeout error.+- Update default user agent to Chrome 85+- Bump Puppeteer to 5.2.1 which uses Chromium 85+- **TypeScript**: Fix `RequestAsBrowserOptions` missing some values and add `RequestQueueInfo`+ as a return value from `requestQueue.getInfo()`++0.21.3 / 2020/07/27+====================+- Fix useless logging in Session.+0.21.2 / 2020/07/27====================- Fix cookies with leading dot in domain (as extracted from Puppeteer)diff --git a/src/crawlers/statistics.js b/src/crawlers/statistics.jsindex 731182585..f565cfe63 100644--- a/src/crawlers/statistics.js+++ b/src/crawlers/statistics.js@@ -246,7 +246,6 @@ class Statistics {return;}- /** @type {StatisticPersistedState & StatisticState} */const savedState = await this.keyValueStore.getValue(this.persistStateKey);if (!savedState) return;diff --git a/src/request_queue.js b/src/request_queue.jsindex 69cc64bcc..4ed49ab72 100644--- a/src/request_queue.js+++ b/src/request_queue.js@@ -710,7 +710,7 @@ export class RequestQueue {* }* ```*- * @returns {Promise<object>}+ * @returns {Promise<RequestQueueInfo>}*/async getInfo() {return requestQueues.getQueue({ queueId: this.queueId });@@ -1111,3 +1111,16 @@ export const openRequestQueue = (queueIdOrName, options = {}) => {? openLocalStorage(queueIdOrName, ENV_VARS.DEFAULT_REQUEST_QUEUE_ID, RequestQueueLocal, queuesCache): openRemoteStorage(queueIdOrName, ENV_VARS.DEFAULT_REQUEST_QUEUE_ID, RequestQueue, queuesCache, getOrCreateQueue);};++/**+ * @typedef RequestQueueInfo+ * @property {string} id+ * @property {string} name+ * @property {string} userId+ * @property {Date} createdAt+ * @property {Date} modifiedAt+ * @property {Date} accessedAt+ * @property {number} totalRequestCount+ * @property {number} handledRequestCount+ * @property {number} pendingRequestCount+ */