Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.
TypeScript apifyautomationcrawlercrawlingheadlessheadless-chromejavascriptnodejsnpmplaywrightpuppeteerscraperscrapingtypescriptweb-crawlerweb-crawlingweb-scraping
Remove Node 8 from Travis
Ondra Urban3470d231f4a6
diff --git a/.travis.yml b/.travis.ymlindex 2ee9656a1..ec267debd 100644--- a/.travis.yml+++ b/.travis.yml@@ -5,7 +5,6 @@ if: (tag IS blank) OR tag =~ /^v\d+\.\d+\.\d+$/node_js:- '12'- '10'-- '8'cache:directories:- "$HOME/.npm"diff --git a/CHANGELOG.md b/CHANGELOG.mdindex 44d7d3951..eaef4ad25 100644--- a/CHANGELOG.md+++ b/CHANGELOG.md@@ -5,7 +5,7 @@ NEXT- Add default `SessionPool` implementation to `BasicCrawler`.- Add the ability to create ad-hoc webhooks via `Apify.call()` and `Apify.callTask()`.- Add an example of form filling with `Puppeteer`.-- Add `country` option to `Apify.getProxyUrl()`.+- Add `country` option to `Apify.getApifyProxyUrl()`.- Add `Apify.utils.puppeteer.saveSnapshot()` helper to quickly save HTML and screenshot of a page.- Add the ability to pass `got` supported options to `requestOptions` in `CheerioCrawler`thus supporting things such as `cookieJar` again.diff --git a/test/crawlers/cheerio_crawler.test.js b/test/crawlers/cheerio_crawler.test.jsindex 927b5d59c..588b041eb 100644--- a/test/crawlers/cheerio_crawler.test.js+++ b/test/crawlers/cheerio_crawler.test.js@@ -701,7 +701,7 @@ describe('CheerioCrawler', () => {const opts = crawler._getRequestOptions(request);proxies.push(opts.proxyUrl);// it needs to return something valid- return { dom: {}, responseStream: responseMock };+ return { dom: {}, response: responseMock };};await crawler.run();@@ -739,7 +739,7 @@ describe('CheerioCrawler', () => {const opts = crawler._getRequestOptions(request);proxies.push(opts.proxyUrl);// it needs to return something valid- return { dom: {}, responseStream: responseMock };+ return { dom: {}, response: responseMock };};await crawler.run();