Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.
TypeScript apifyautomationcrawlercrawlingheadlessheadless-chromejavascriptnodejsnpmplaywrightpuppeteerscraperscrapingtypescriptweb-crawlerweb-crawlingweb-scraping
Commits on master
Added visible_hostname to Squid configuration to avoid warnings
e644862 Jan Curn committed 9 years ago
Cosmetic code improvements, better logging, bumped package.json version
d9aa2e5 Jan Curn committed 9 years ago
Updated README.md, bumped package.json version, commented out old code
b148802 Jan Curn committed 9 years ago
Disabled selenium logging, added unit tests for userAgent setting,
39be771 Jan Curn committed 9 years ago
Bugfix: properly handle situation where APIFY_API_BASE_URL env var is missing, added unit test for that, bumped package.json version
631a8f2 Jan Curn committed 9 years ago
Reorganized Apifier.browser() code for better proxy configuration
ffc6241 Jan Curn committed 9 years ago
Increased min version of "apify-client" dependency, bumped package.json version
04e38ab Jan Curn committed 9 years ago
Switched to new "apify-client" version, bumped package.json version, development of Apifier.browse() proxy settings
cc1f0bf Jan Curn committed 9 years ago
Changed back to "apify-client": ">=0.1.6" dependency, bumped package.json version
1b3400e Jan Curn committed 9 years ago
Accomodating to the changed interface of apify-client, bumped package.json version
815770b Jan Curn committed 9 years ago