Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.
TypeScript apifyautomationcrawlercrawlingheadlessheadless-chromejavascriptnodejsnpmplaywrightpuppeteerscraperscrapingtypescriptweb-crawlerweb-crawlingweb-scraping
Parallelize lint and tests
Ondra Urban20eb4db4823b
diff --git a/.travis.yml b/.travis.ymlindex 8852c0deb..536759d58 100644--- a/.travis.yml+++ b/.travis.yml@@ -11,10 +11,9 @@ cache: npmenv:global:- DOCKER_USERNAME=mnmkng- - secure: Sp706wRWmS2aM86poOhSTngkEBlpbUydvstEXKEQ8Zz/R4HlmWBsb5vX9lk9FkAC504o21qpCJAIyidHs5FXsw9kpeMf2h+zI/Zs2w1RNaitMDeotSofFYo2J76u7QV6JFUhbPzDYjUXKmgJkPAuffCPeLfj0qszjpjRMz+NdbwE4LVWJpaeyM5lOGmdec51//16juKQBbUsdJ9xPD0bXrQ6mmJs7IJEsVMcQpfOkIgqP70IMXNiDTAHBU/15Oi5yipkMW7Wmn4J6GM8wQmADFUXS8t6oLtbVWEr65YtyzfIN/PjWXT3om7KpfXSouf+lOJdRZAn2RhbXe88PwGIIpgMJKgiIil3Sxgch0UtADTMfWvInylrdBldAw7JSM5QMBFP/vOCRNJEVHurfVOvCmQnlJ2NLLe7KzjF37dDI5LHFyY9u0ysXdoMsHMzLptGy+xPgf3kKkiCbT+GfhrYuAAmdb5MJBcxNkbjsQEszSuy0zRgE+Yu9YAgWA9veQtO3JDo6EmUyOyd2vHWvhCE6aF/o2TD/Nf1CTVblG3M1QwLmzvwADBpH6utDCu8L3NOESShWHUA5pAUsdCfMpTKiJR1rnIEOeCRJFGIVv9CmWbKtdL5asN6nJbhkoq3/Lhf7MC5pxFlKH3LUR1a/nwvO824SnF4ZJEjp9f1aeJU+lU=- secure: pgwE9Nxa27nwuC0zI+IrbpY/P/QNPL9oNxdwk+LMxa90Tn/5EUKl5D/LblGzjRYd39i/Z1ucNf1OJoKThX7YQxs/de4fjX00HGC0DjRzBRcLsU02HhbrNSA/6ASpWyR2WTxFGnJG/15CzHgqP1BM6EvLJixFoyhk6tnF0kqpJos/h8yfoSZV1kfdlcaz1TQU/NdoFD2Agxm5BEf3/b4SKtXGF0pGGj/IdPCMc/NwsJpHFHi2lU90Nr0WVnHD3wyZRHyasQlGG9Yu/f8Tjf27LCY5QDgcJ/JhsFbwabUwXe98JtTLssriOCajXECilwdBYFTpctIhIgfs6bmdO5Xy3rUjrvdMbOjhjNhkD0UHAQjQg5SUEAo5RKssEUgTUtL4S7P6falc1QwhehZZO/65Ui13Jh1zhlBZpYAO1RXLB7m8DHj4gRrgQswn4qBjWpjbiwfC9ocIuH0b/3QUXRGA4cPjB3G4tl+OdFMFykSYvuZGdWC4U6x+z/C6OfEEYwqJaapHKNnafp9429LMNPzd8SSS3KhDaY7N2kLnH5YaoFAVsZ0rRIk0GZpP+xYWBfFlELqCarvcuZi9su3QWverzecjOpfNu+LCHpOYdiPQtGvm8Mzr1eZun/VaJ5DZ3ZIX1y6ODSCg9b9ikGtcz6A1n2GkE4cHN/B1YI52NIbRXN0=+ - secure: q5jH62AWy4FpOEPmRMWeA3zVcoeGobBzQ1yJ9YrbTOjGAFa6Kd86lS0VUJWtgOwAMycEROCTpEZX+oalVhRaQyR8kikZg7h80wpOziPfWAJe0y3wBfoP1eLaotLF4PhDhSw2X/wo88DGufzYIFOnH4icITYQIaVPe+8an/Nj/I+Hl3UAONEhIIzMTwWAXNz1haNrPGEDN0keva3bPEaMcrK/f1BgkaOTvVFoZRNpsBGaIuQnM9B9eXzoJeXwB0WmWmX/7Pr2DXE4DOWmSA0hfzJUuCmZsudR1+DL5S2gG97ppjBJ/hKIvNusn5/O7tj00lf/LF6hOKgOnWTWFFhrLbMC9pbqt5TbG4D1kCybiZqZmq1P27n5HX2RmjFiUEG2LJ39nnt+UIo+Y01IuL9YtjghvKpCelauBGflhrdZPae2nA6rvikEzLPHuk7qyHiV03hYgom6feR6yv8S1A6PACvBrX1onPvoI4ihv6IOvrQ9l3rzEGxuRyD3QDDL+CT3wvdr0G7NbMRorkgZDCsMBN77mjivcg7H0cqp8b/n7DRo/Ua1lWTHSs95Y2xr8C0DjpISMLcEHzNeZxs46NmZbPzbjuqac3VJW90g3RhMlIJ06MOHp0zAZHdLdJIUTkTjN0d45DHGoETtFinXaRe7tqGqkPt54sQ8c2ZRTU4QZD0=stages:-- lint- test- deploy- images@@ -22,8 +21,8 @@ install:- npm install --no-package-lockjobs:include:- - stage: lint- node_js: '10'+ - stage: test+ name: 'Lint'script: npm run lint && node .travis/check-docs.js- stage: deploynode_js: '10'