Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.
TypeScript apifyautomationcrawlercrawlingheadlessheadless-chromejavascriptnodejsnpmplaywrightpuppeteerscraperscrapingtypescriptweb-crawlerweb-crawlingweb-scraping
chore: check the documentation builds on every PR (#1426)
Martin Adámek1810103fdacf
diff --git a/.github/workflows/test-and-release.yml b/.github/workflows/test-and-release.ymlindex c7593e30f..3f574ec71 100644--- a/.github/workflows/test-and-release.yml+++ b/.github/workflows/test-and-release.yml@@ -62,6 +62,39 @@ jobs:- name: Testsrun: npm test+ docs:+ name: Docs build+ if: ${{ !contains(github.event.head_commit.message, '[skip ci]') }}+ runs-on: ubuntu-latest+ steps:+ - name: Checkout Source code+ uses: actions/checkout@v3++ - name: Use Node.js 16+ uses: actions/setup-node@v3+ with:+ node-version: 16+ cache: 'npm'+ cache-dependency-path: 'package-lock.json'++ - name: Turbo cache+ id: turbo-cache+ uses: actions/cache@v3+ with:+ path: .turbo+ key: turbo-${{ github.job }}-${{ github.ref_name }}-${{ github.sha }}+ restore-keys: |+ turbo-${{ github.job }}-${{ github.ref_name }}-++ - name: Install Dependencies+ run: npm ci++ - name: Build & deploy docs+ run: |+ cd website+ npm ci+ npm run build+lint:name: Lintruns-on: ubuntu-latestdiff --git a/docs/guides/apify_platform.mdx b/docs/guides/apify_platform.mdxindex 51694d812..12d461dc7 100644--- a/docs/guides/apify_platform.mdx+++ b/docs/guides/apify_platform.mdx@@ -153,7 +153,7 @@ You can also develop your actor in an online code editor directly on the platforThere are several things worth mentioning here.1. Compared to Crawlee, in order to simplify access to the default <ApiLink to="core/class/KeyValueStore">`Key-Value Store`</ApiLink> and <ApiLink to="core/class/Dataset">`Dataset`</ApiLink>, you could also use [`Actor.getValue()`](https://apify.github.io/apify-sdk-js/api/apify/class/Actor#getValue), [`Actor.setValue()`](https://apify.github.io/apify-sdk-js/api/apify/class/Actor#setValue) for the default `Key-Value Store` and [`Actor.pushData()`](https://apify.github.io/apify-sdk-js/api/apify/class/Actor#pushData) for the default `Dataset` directly, instead of using the helper functions of respective storage classes.-2. When you plan to use the platform storage while developing and running your actor locally, you should use [`Actor.openKeyValueStore()`](https://apify.github.io/apify-sdk-js/api/apify/class/Actor#openKeyValueStore), [`Actor.openDataset()`](https://apify.github.io/apify-sdk-js/api/apify/class/Actor#openDataset) and [`Actor.openRequestQueue()`](https://apify.github.io/apify-sdk-js/api/apify/class/Actor#openRequestQueue) to open the <ApiLink to="core/class/KeyValueStore">`KeyValueStore`</ApiLink>, <ApiLink to="core/class/Datase">`Dataset`</ApiLink> and <ApiLink to="core/class/RequestQueue">`RequestQueue`</ApiLink> respectively. Using each of these methods allows to pass the [`OpenStorageOptions`](https://apify.github.io/apify-sdk-js/api/apify/interface/OpenStorageOptions) as a second argument, which has only one optional property: [`forceCloud`](https://apify.github.io/apify-sdk-js/api/apify/interface/OpenStorageOptions#forceCloud). If set to `true` - cloud storage will be used instead of the folder on the local disk.+2. When you plan to use the platform storage while developing and running your actor locally, you should use [`Actor.openKeyValueStore()`](https://apify.github.io/apify-sdk-js/api/apify/class/Actor#openKeyValueStore), [`Actor.openDataset()`](https://apify.github.io/apify-sdk-js/api/apify/class/Actor#openDataset) and [`Actor.openRequestQueue()`](https://apify.github.io/apify-sdk-js/api/apify/class/Actor#openRequestQueue) to open the <ApiLink to="core/class/KeyValueStore">`KeyValueStore`</ApiLink>, <ApiLink to="core/class/Dataset">`Dataset`</ApiLink> and <ApiLink to="core/class/RequestQueue">`RequestQueue`</ApiLink> respectively. Using each of these methods allows to pass the [`OpenStorageOptions`](https://apify.github.io/apify-sdk-js/api/apify/interface/OpenStorageOptions) as a second argument, which has only one optional property: [`forceCloud`](https://apify.github.io/apify-sdk-js/api/apify/interface/OpenStorageOptions#forceCloud). If set to `true` - cloud storage will be used instead of the folder on the local disk.:::noteIf you don't plan to use the platform storage for a local actor - you don't need to use the [`Actor`](https://apify.github.io/apify-sdk-js/api/apify/class/Actor) class, meaning you could use <ApiLink to="core/class/KeyValueStore#open">`KeyValueStore.open()`</ApiLink>, <ApiLink to="core/class/Dataset#open">`Dataset.open()`</ApiLink> and <ApiLink to="core/class/RequestQueue#open">`RequestQueue.open()`</ApiLink> to open the respective storage both on the platform and locally.:::