Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.
TypeScript apifyautomationcrawlercrawlingheadlessheadless-chromejavascriptnodejsnpmplaywrightpuppeteerscraperscrapingtypescriptweb-crawlerweb-crawlingweb-scraping
docs: Add KVS encryption example (#4256)
Jan Buchar712aa19387c6
diff --git a/docs/examples/encrypt-key-value-store.mdx b/docs/examples/encrypt-key-value-store.mdxnew file mode 100644index 000000000..39a6216a3--- /dev/null+++ b/docs/examples/encrypt-key-value-store.mdx@@ -0,0 +1,17 @@+---+id: encrypt-key-value-store+title: Encrypt key-value store records+---++import CodeBlock from '@theme/CodeBlock';+import EncryptKeyValueStoreSource from '!!raw-loader!./encrypt-key-value-store.ts';++Any transform on data going to and from storage — encryption, compression, redaction, caching — belongs in a wrapper around the storage backend. Pass the wrapper wherever a backend goes: the crawler's `storageBackend` option, the `storageBackend` option of `Dataset.open()` / `KeyValueStore.open()` / `RequestQueue.open()`, or `serviceLocator.setStorageBackend()`.++This example encrypts every key-value store record, `useState()` included, and passes datasets and request queues through untouched. To transform those as well, wrap the sub-backend returned by `createDatasetBackend()` (`pushData()` / `getData()`) or `createRequestQueueBackend()` (`addBatchOfRequests()` and the methods that return requests) the same way.++Don't subclass `KeyValueStore` for this. Inside a request handler, its writes are recorded in the [storage transaction](../guides/result-storage.mdx#transactional-storage)'s journal and replayed through `setValue()` again on commit, so an override would run twice. The wrapper sits below the journal: a committed write passes through it once, a rolled-back one never reaches it, and reads inside the handler see their own untransformed writes.++<CodeBlock language="js">+ {EncryptKeyValueStoreSource}+</CodeBlock>diff --git a/docs/examples/encrypt-key-value-store.ts b/docs/examples/encrypt-key-value-store.tsnew file mode 100644index 000000000..c0bddd463--- /dev/null+++ b/docs/examples/encrypt-key-value-store.ts@@ -0,0 +1,49 @@+import { createCipheriv, createDecipheriv, randomBytes } from 'node:crypto';+import { CheerioCrawler, FileSystemStorageBackend } from 'crawlee';+import type { KeyValueStoreBackend, StorageIdentifier } from '@crawlee/types';++const secret = randomBytes(32); // load it from somewhere safe instead++function encrypt(plain: Buffer): Buffer {+ const iv = randomBytes(12);+ const cipher = createCipheriv('aes-256-gcm', secret, iv);+ const body = Buffer.concat([cipher.update(plain), cipher.final()]);+ return Buffer.concat([iv, cipher.getAuthTag(), body]);+}++function decrypt(sealed: Uint8Array): Buffer {+ const decipher = createDecipheriv('aes-256-gcm', secret, sealed.subarray(0, 12));+ decipher.setAuthTag(sealed.subarray(12, 28));+ return Buffer.concat([decipher.update(sealed.subarray(28)), decipher.final()]);+}++class EncryptingStorageBackend extends FileSystemStorageBackend {+ override async createKeyValueStoreBackend(options?: StorageIdentifier): Promise<KeyValueStoreBackend> {+ const store = await super.createKeyValueStoreBackend(options);+ const { getValue, setValue } = Object.getPrototypeOf(store) as KeyValueStoreBackend;++ // Patching the instance keeps the example short; a real implementation would wrap the backend in a class of its own.+ store.getValue = async (key) => {+ const record = await getValue.call(store, key);+ return record && { ...record, value: decrypt(new Uint8Array(record.value)) };+ };+ store.setValue = async (record) => {+ // `Response` drains every input shape a backend can receive, streams included.+ const plain = Buffer.from(await new Response(record.value as BodyInit).arrayBuffer());+ await setValue.call(store, { ...record, value: encrypt(plain) });+ };+ return store;+ }+}++const crawler = new CheerioCrawler({+ storageBackend: new EncryptingStorageBackend({ localDataDirectory: './storage' }),+ async requestHandler({ request, $, useState, getKeyValueStore }) {+ const state = await useState({ visited: 0 });+ state.visited++;+ const store = await getKeyValueStore();+ await store.setValue(request.id, { url: request.url, title: $('title').text() });+ },+});++await crawler.run(['https://crawlee.dev']);diff --git a/docs/guides/request-storage.mdx b/docs/guides/request-storage.mdxindex 8a8fbcd08..cf28cad03 100644--- a/docs/guides/request-storage.mdx+++ b/docs/guides/request-storage.mdx@@ -78,6 +78,10 @@ Sometimes you don't want to start from a dynamic queue, but from a static list oSee the dedicated [Request loaders](./request-loaders) guide for details on loaders, request managers, and how to combine them with a queue into a <ApiLink to="core/class/RequestManagerTandem">`RequestManagerTandem`</ApiLink>.+## Transforming stored data++To transform requests on their way to and from storage — encryption, redaction, caching — wrap the storage backend rather than subclassing `RequestQueue`. The [Encrypt KVS records](../examples/encrypt-key-value-store) example shows the pattern on a key-value store; a request queue works the same way through `createRequestQueueBackend()`.+## Cleaning up the storagesRun-scoped storages - the default one and any opened with an `alias` - are purged before the crawler starts if not specified otherwise. Storages opened with a `name` persist across runs and are never purged. This happens as early as when we try to open some storage (e.g. via `RequestQueue.open()`) or when we try to work with a default storage via one of the helper methods (e.g. `crawler.addRequests()` that under the hood calls `RequestQueue.open()`). If we don't work with storages explicitly in our code, the purging will eventually happen when the `run` method of our crawler is executed. In case we need to purge the storages sooner, we can use the <ApiLink to="core/function/purgeDefaultStorages">`purgeDefaultStorages()`</ApiLink> helper explicitly:diff --git a/docs/guides/result-storage.mdx b/docs/guides/result-storage.mdxindex 280bdea9e..865ce61d9 100644--- a/docs/guides/result-storage.mdx+++ b/docs/guides/result-storage.mdx@@ -110,6 +110,10 @@ await dataset.pushData([{ foo: 'bar2', col2: 'val2' }, { col3: 123 }]);To see how to use the dataset to store crawler results, see the [Cheerio Crawler](../examples/cheerio-crawler) example.+## Transforming stored data++To transform data on its way to and from storage — encryption, compression, caching — wrap the storage backend rather than subclassing `KeyValueStore` or `Dataset`. A `KeyValueStore` subclass in particular would see each write from a request handler twice, once when it is made and again when the [transaction](#transactional-storage) commits. See the [Encrypt KVS records](../examples/encrypt-key-value-store) example.+## Cleaning up the storagesRun-scoped storages - the default one and any opened with an `alias` - are purged before the crawler starts if not specified otherwise. Storages opened with a `name` persist across runs and are never purged. This happens as early as when we try to open some storage (e.g. via `Dataset.open()`) or when we try to work with a default storage via one of the helper methods (e.g. `Dataset.pushData()` that under the hood calls `Dataset.open()`). If we don't work with storages explicitly in our code, the purging will eventually happen when the `run` method of our crawler is executed. In case we need to purge the storages sooner, we can use the <ApiLink to="core/function/purgeDefaultStorages">`purgeDefaultStorages()`</ApiLink> helper explicitly: