Skip to content

Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.

Fix null response in PuppeteerCrawler

Ondra Urbanec6d66c69721
diff --git a/CHANGELOG.md b/CHANGELOG.md
index 6ceefb022..27596d21d 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -1,3 +1,7 @@
+0.21.7 / 2020/09/03
+====================
+- Fix an error in `PuppeteerCrawler` caused by `page.goto()` randomly returning `null`.
+
0.21.6 / 2020/09/02
====================
It appears that `CheerioCrawler` was correctly retiring sessions on timeouts
diff --git a/package.json b/package.json
index b108b0483..d282320da 100644
--- a/package.json
+++ b/package.json
@@ -1,6 +1,6 @@
{
"name": "apify",
- "version": "0.21.6",
+ "version": "0.21.7",
"description": "The scalable web crawling and scraping library for JavaScript/Node.js. Enables development of data extraction and web automation jobs (not only) with headless Chrome and Puppeteer.",
"engines": {
"node": "^10.17.0 || ^12.3.0"
diff --git a/src/crawlers/puppeteer_crawler.js b/src/crawlers/puppeteer_crawler.js
index 946b4c205..676f08613 100644
--- a/src/crawlers/puppeteer_crawler.js
+++ b/src/crawlers/puppeteer_crawler.js
@@ -382,7 +382,10 @@ class PuppeteerCrawler {
}
}
- if (this.useSessionPool) this._throwOnBlockedRequest(session, response.status());
+ if (this.useSessionPool) {
+ const statusCode = response && response.status();
+ this._throwOnBlockedRequest(session, statusCode);
+ }
await this.puppeteerPool.serveLiveViewSnapshot(page);
request.loadedUrl = page.url();