Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.
TypeScript apifyautomationcrawlercrawlingheadlessheadless-chromejavascriptnodejsnpmplaywrightpuppeteerscraperscrapingtypescriptweb-crawlerweb-crawlingweb-scraping
Updated README.md
Jan Curne559d68acd3e
diff --git a/README.md b/README.mdindex 8f6159f93..11563fe48 100644--- a/README.md+++ b/README.md@@ -172,6 +172,11 @@ otherwise the act process might finish before the output is stored and/or storagBesides the key `INPUT` and `OUTPUT`, you can use arbitrary keysto store any data from your act, such as its state or larger results.+During local development of the act, you can emulate the key-value store locally using the+`DEV_KEY_VALUE_STORE_DIR` environment variable. Simply set the variable to a path of some local directory,+and the runtime will read and store values in that directory rather than Apify servers.+By default, the values are considered to have the `application/json; charset=utf-8`+content type, but you can override it by setting the `DEV_KEY_VALUE_STORE_CONTENT_TYPE` environment variable.### Browser