Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.
TypeScript apifyautomationcrawlercrawlingheadlessheadless-chromejavascriptnodejsnpmplaywrightpuppeteerscraperscrapingtypescriptweb-crawlerweb-crawlingweb-scraping
PuppeteerCrawler's default page load timeout changed from 30 to 60 seconds
Jan Curnc1017353924d
diff --git a/CHANGELOG.md b/CHANGELOG.mdindex 10706de3e..376eefb9a 100644--- a/CHANGELOG.md+++ b/CHANGELOG.md@@ -1,3 +1,7 @@+xxx+===================+- PuppeteerCrawler's default page load timeout changed from 30 to 60 seconds.+0.5.51 / 2018-08-09===================- Fixed unresponsive magnifying glass and improved status tracking in LiveView frontenddiff --git a/src/puppeteer_crawler.js b/src/puppeteer_crawler.jsindex 4d5a7a0ba..79ff83d3e 100644--- a/src/puppeteer_crawler.js+++ b/src/puppeteer_crawler.js@@ -7,7 +7,7 @@ import PuppeteerPool from './puppeteer_pool';import { isPromise } from './utils';const DEFAULT_OPTIONS = {- gotoFunction: ({ request, page }) => page.goto(request.url),+ gotoFunction: ({ request, page }) => page.goto(request.url, { timeout: 60000 }),pageOpsTimeoutMillis: 300000,};@@ -65,10 +65,16 @@ const PAGE_CLOSE_TIMEOUT_MILLIS = 30000;* `page` is an instance of the `Puppeteer.Page` class with `page.goto(request.url)` already called.* @param {Number} [options.pageOpsTimeoutMillis=300000]* Timeout in which the function passed as `options.handlePageFunction` needs to finish.- * @param {Function} [options.gotoFunction=({ request, page }) => page.goto(request.url)]+ * @param {Function} [options.gotoFunction=({ request, page }) => page.goto(request.url, { timeout: 60000 })]* Overrides the function that opens the request in Puppeteer.- * This function should return a result of `page.goto()`, i.e. the Puppeteer's `Response` object.- * Note that one page is only used to process one request, and it is closed afterwards.+ * The function should return a result of Puppeteer's+ * <a href="https://github.com/GoogleChrome/puppeteer/blob/master/docs/api.md#pagegotourl-options">page.goto()</a> function,+ * i.e. a promise resolving to the <a href="https://github.com/GoogleChrome/puppeteer/blob/master/docs/api.md#class-response">Response</a> object.+ *+ * For example, this is useful if you need to extend the page load timeout or select a different criteria+ * to determine that the navigation succeeded.+ *+ * Note that a single page object is only used to process a single request and it is closed afterwards.* @param {Function} [options.handleFailedRequestFunction=({ request }) => log.error('Request failed', _.pick(request, 'url', 'uniqueKey'))]* Function to handle requests that failed more than `option.maxRequestRetries` times. See the `handleFailedRequestFunction`* parameter of `Apify.BasicCrawler` for details.