Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.
TypeScript apifyautomationcrawlercrawlingheadlessheadless-chromejavascriptnodejsnpmplaywrightpuppeteerscraperscrapingtypescriptweb-crawlerweb-crawlingweb-scraping
Commits on master
Added redactUrl() and redactParsedUrl() functions to utils with unit tests, bumped package.json version
e4e0c55 Jan Curn committed 9 years ago
User-friendly JSON formatting in Apify.setValue(), fixed broken user agent tests, bumped package.json version
db2c540 Jan Curn committed 9 years ago
Added dummy setInterval() to prevent termination of process while awaiting empty promise, bumped package.json version
a90a1c5 Jan Curn committed 9 years ago
Added source for the legacy 'apifier' NPM package to set its README
078de68 Jan Curn committed 9 years ago
Bumped package.json version in an attempt to publish the readme
ebd9770 Jan Curn committed 9 years ago
Changed Apify.setValue() params - replaced contentType with options, updated unit tests and README, bumped package.json version
c308cf1 Jan Curn committed 9 years ago
Updated README.md, bumped package.json version in an attempt to publish the readme (using new version of npm)
d9bd710 Jan Curn committed 9 years ago
Bumped package.json version in an attempt to have NPM readme updated
9a453cd Jan Curn committed 9 years ago
Changed publish.sh and bumped package.json version in an attempt to have NPM readme updated
38198d7 Jan Curn committed 9 years ago
Renamed Apify.getContext() to Apify.getEnv(), the context object is no longer passed to user main function, updated README.md
f5ca6b9 Jan Curn committed 9 years ago
Removed Apify.getInput() and Apify.setOutput(), changed Apify.getContext() and Apify.main() semantics not to handle input/output implicitly to make system more transparent
39f6081 Jan Curn committed 9 years ago
Renamed the package from 'apifier' to 'apify', bumped package.json version to 0.4.0 to avoid conflicts with the previous package 0.3.4, renamed "Apifier" in code to "Apify", partially updated README.md
9f15780 Jan Curn committed 9 years ago
ProxyChain: Randomizing squid port to reduce a chance of collision
b7c9f8c Jan Curn committed 9 years ago