Skip to content

Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.

Fixed .travis.yml, new comments

Jan Curnad9ff674bf15
diff --git a/.travis.yml b/.travis.yml
index 4dd70a111..9720aa8d4 100644
--- a/.travis.yml
+++ b/.travis.yml
@@ -4,10 +4,10 @@ before_install:
- curl -sS https://dl-ssl.google.com/linux/linux_signing_key.pub | sudo apt-key add -
- sudo apt-get -qq update
- sudo apt-get install -y google-chrome-beta
- - sudo curl -sS http://chromedriver.storage.googleapis.com/2.29/chromedriver_linux64.zip > /tmp.zip
- - sudo unzip /tmp.zip
- - sudo rm /tmp.zip
- - sudo mv /chromedriver /bin
+ - sudo curl -sS http://chromedriver.storage.googleapis.com/2.29/chromedriver_linux64.zip > /tmp/chromedriver.zip
+ - sudo unzip /tmp/chromedriver.zip
+ - sudo rm /tmp/chromedriver.zip
+ - sudo mv /tmp/chromedriver /bin
language: node_js
node_js:
- "7"
diff --git a/src/browser.js b/src/browser.js
index b76ddfcbe..2a8dc8d2c 100644
--- a/src/browser.js
+++ b/src/browser.js
@@ -87,6 +87,7 @@ export const browse = (url, options = null, callback = null) => {
'usernmae': 'johnsmith123',
'password': 'iliketurtles'}
+ or use https://github.com/SeleniumHQ/docker-selenium/wiki/Corporate-Proxies
capabilities = dict(DesiredCapabilities.CHROME)
capabilities['proxy'] = {'proxyType': 'MANUAL',