Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.
TypeScript apifyautomationcrawlercrawlingheadlessheadless-chromejavascriptnodejsnpmplaywrightpuppeteerscraperscrapingtypescriptweb-crawlerweb-crawlingweb-scraping
Fixed .travis.yml, new comments
Jan Curnad9ff674bf15
diff --git a/.travis.yml b/.travis.ymlindex 4dd70a111..9720aa8d4 100644--- a/.travis.yml+++ b/.travis.yml@@ -4,10 +4,10 @@ before_install:- curl -sS https://dl-ssl.google.com/linux/linux_signing_key.pub | sudo apt-key add -- sudo apt-get -qq update- sudo apt-get install -y google-chrome-beta- - sudo curl -sS http://chromedriver.storage.googleapis.com/2.29/chromedriver_linux64.zip > /tmp.zip- - sudo unzip /tmp.zip- - sudo rm /tmp.zip- - sudo mv /chromedriver /bin+ - sudo curl -sS http://chromedriver.storage.googleapis.com/2.29/chromedriver_linux64.zip > /tmp/chromedriver.zip+ - sudo unzip /tmp/chromedriver.zip+ - sudo rm /tmp/chromedriver.zip+ - sudo mv /tmp/chromedriver /binlanguage: node_jsnode_js:- "7"diff --git a/src/browser.js b/src/browser.jsindex b76ddfcbe..2a8dc8d2c 100644--- a/src/browser.js+++ b/src/browser.js@@ -87,6 +87,7 @@ export const browse = (url, options = null, callback = null) => {'usernmae': 'johnsmith123','password': 'iliketurtles'}+ or use https://github.com/SeleniumHQ/docker-selenium/wiki/Corporate-Proxiescapabilities = dict(DesiredCapabilities.CHROME)capabilities['proxy'] = {'proxyType': 'MANUAL',