Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.
TypeScript apifyautomationcrawlercrawlingheadlessheadless-chromejavascriptnodejsnpmplaywrightpuppeteerscraperscrapingtypescriptweb-crawlerweb-crawlingweb-scraping
Merge branch 'develop' into feature/sdk-review
Jan Curn825deb19b5be
diff --cc README.mdindex b0b826dfb,ecec5c4d5..87cf5497b--- a/README.md+++ b/README.md@@@ -496,23 -346,88 +497,97 @@@ For more information see complete <a hr<div id="include-readme-3">++ #### Puppeteer Live View++ Enables real time inspection of individual Puppeteer browser instances by starting a web server @ `localhost:4321`. This is especially useful when using headless mode or a remote instance.+ Puppeteer Live View provides the user with a dashboard listing all active browser instances and their active page details. The details show a page screenshot and raw HTML.++ You can use Puppeteer Live View, either directly:+ ```javascript+ const browser = Apify.launchPuppeteer({ liveView: true });+ ```+ or while using PuppeteerCrawler:+ ```javascript+ const crawler = new PuppeteerCrawler({+ launchPuppeteerOptions: { liveView: true },+ // other options+ })+ ```++ For debugging, you may want to add another option: `{ slowMo: 300 }` (see <a href="https://pptr.dev/#?product=Puppeteer&version=v1.6.0&show=api-puppeteerlaunchoptions" target="_blank">Puppeteer documentation</a>).++ After you connect to `localhost:4321` (the port is configurable using `APIFY_CONTAINER_PORT` environment variable),+ Puppeteer Live View will present you with the following screen:+ <img src="https://www.apify.com/ext/sdk_assets/puppeteer-live-view-dashboard.png">++ Click on the magnifying glass icon will take you to a page detail, showing its screenshot and raw HTML:+ <img src="https://www.apify.com/ext/sdk_assets/puppeteer-live-view-detail.png">++ For more information see complete <a href="https://www.apify.com/docs/sdk/apify-runtime-js/latest#PuppeteerLiveViewServer" target="_blank">documentation</a>.++ ## Local usage++ The easiest way how to use apify locally is with <a href="https://github.com/apifytech/apify-cli" target="_blank">Apify CLI</a> as shown+ in <a href="#quick-start">quick start</a> section. Other way is to manually define required environment variables:++ <table class="table table-bordered table-condensed">+ <thead>+ <tr>+ <th>Environment variable</th>+ <th>Description</th>+ </tr>+ </thead>+ <tbody>+ <tr>+ <td><code>APIFY_LOCAL_EMULATION_DIR</code></td>+ <td>+ Directory where apify package locally emulates Apify storages - key-value store and dataset.+ Key-value stores will be emulated in directory+ <code>[APIFY_LOCAL_EMULATION_DIR]/key-value-stores/[STORE_ID]</code>+ and datasets in directory+ <code>[APIFY_LOCAL_EMULATION_DIR]/datasets/[DATESET_ID]</code>.+ </td>+ </tr>+ <tr>+ <td><code>APIFY_DEFAULT_KEY_VALUE_STORE_ID</code></td>+ <td>ID of default key-value store.</td>+ </tr>+ <tr>+ <td><code>APIFY_DEFAULT_DATASET_ID</code></td>+ <td>ID of default dataset.</td>+ </tr>+ <tr>+ <td><code>APIFY_DEFAULT_REQUEST_QUEUE_ID</code></td>+ <td>ID of default request queue.</td>+ </tr>+ </tbody>+ </table>++ Apify will then store key-value store records in files named <code>[KEY].[EXT]</code> where <code>[KEY]</code>+ is the record key and <code>[EXT]</code> is based on the record content type. Dataset items will be stored+ in files named <code>[ID].json</code> where <code>[ID]</code> is sequence number of your dataset item.+ *+ If you want to use <a href="https://www.apify.com/docs/proxy" target="_blank">Apify Proxy</a> locally+ then you must define an environment variable <code>PROXY_PASSWORD</code> with password you find at+ <a href="https://my.apify.com/proxy" target="_blank">https://my.apify.com/proxy</a>.+## Promises vs. callbacks-By default, all asynchronous functions provided by this package return a promise.-But Apify uses a <a href="http://bluebirdjs.com/" target="_blank">Bluebird</a>-promise implementation so you can easily convert any function that returns a Promise-into callback style function.-See <a href="http://bluebirdjs.com/docs/api/promise.promisify.html" target="_blank">Bluebird documentation</a>-for more information.+All asynchronous functions provided by the <code>apify</code> package return+a <a href="http://bluebirdjs.com/" target="_blank" rel="noopener">bluebird</a> Promise.+If you prefer to use callbacks, you can use+the <a href="http://bluebirdjs.com/docs/api/ascallback.html" target="_blank" rel="noopener">.asCallback()</a> function.+For example:++```js+const Apify = require('apify');++Apify.launchPuppeteer().asCallback((err, browser) => {+ // Write you callback code here...+});+```+## Examples