Skip to content

armgit / papra

Public Imported from GitHub
0

The minimalistic document archiving platform.

feat(maintenance): add document storage migration script (#934)

Corentin Thomasset69f8f0e13b99
diff --git a/apps/docs/src/content/docs/03-guides/10-migrate-document-storage.mdx b/apps/docs/src/content/docs/03-guides/10-migrate-document-storage.mdx
new file mode 100644
index 00000000..eb9f280a
--- /dev/null
+++ b/apps/docs/src/content/docs/03-guides/10-migrate-document-storage.mdx
@@ -0,0 +1,250 @@
+---
+title: Migrate Document Storage
+description: Move documents between storage backends or change storage settings using the storage migration maintenance script.
+slug: guides/migrate-document-storage
+---
+import { Steps } from '@astrojs/starlight/components';
+import { Aside } from '@astrojs/starlight/components';
+import { Tabs, TabItem } from '@astrojs/starlight/components';
+
+<Aside type="note">
+**Not yet available:** This migration script will be available in the next release.
+</Aside>
+
+The storage migration script copies every document from one storage backend to another, streaming files one at a time to keep memory usage flat. Encryption is handled transparently — documents are decrypted on the way out of the source and re-encrypted (or left plain) on the way into the destination, depending on each side's configuration.
+
+## When to Use This Script
+
+- **Changing storage key pattern** — after updating `DOCUMENT_STORAGE_KEY_PATTERN` or switching off the legacy key system, existing documents keep their old storage keys; use this script to re-copy them so they land under the new pattern (see [Storage Key Patterns](/guides/storage-key-patterns))
+- **Moving to cloud storage** — migrating from a local filesystem to S3, Azure Blob Storage, or another S3-compatible provider
+- **Switching cloud providers** — moving files between S3 and Azure Blob, or between S3-compatible services
+- **Changing storage root** — relocating the filesystem storage directory to a different path or volume
+
+<Aside type="note">
+If you only want to encrypt existing documents without changing anything else, use the dedicated [`maintenance:encrypt-all-documents`](/guides/document-encryption#migrating-existing-documents-to-encrypted-format) script instead. The storage migration script will fail if source and destination resolve to the same path for the same key, because storage drivers refuse to overwrite existing files.
+</Aside>
+
+## How It Works
+
+<Steps>
+
+1. **Source and destination configs are built** from the base environment variables, with your `--from` and `--to` overrides applied on top. Database configuration always comes from the running environment — only storage settings are overridden.
+
+2. **Documents are iterated in batches** (including soft-deleted ones), so the script can handle large document collections without loading everything into memory.
+
+3. **Each document is streamed** from the source storage (decrypting if the source has encryption enabled), then written to the destination (encrypting if the destination has encryption enabled).
+
+4. **The new storage key is computed** using the destination's pattern configuration (via `DOCUMENT_STORAGE_KEY_PATTERN`), and both the storage key and encryption metadata are updated in the database. This is what allows a no-args invocation to re-key all documents after a pattern change.
+
+5. **If `--delete-source` is passed**, the original file is removed from the source backend after a successful copy.
+
+6. **Errors are per-document** — a single failed document is logged and skipped; the rest of the migration continues.
+
+</Steps>
+
+## CLI Reference
+
+```
+maintenance:migrate-document-storage [options]
+
+Options:
+ --from <KEY=VALUE> Source storage environment variable (repeatable)
+ --to <KEY=VALUE> Destination storage environment variable (repeatable)
+ --delete-source Delete source files after a successful copy
+ --dry-run Preview what would be migrated without touching any files
+ -h, --help Show this help message
+```
+
+`--from` and `--to` accept any `DOCUMENT_STORAGE_*` environment variable. Values are merged on top of the current environment, so you only need to specify the settings that differ from your running configuration.
+
+## Running the Migration
+
+<Aside type="caution">
+Back up your documents and database before running the migration. While the script is non-destructive by default (it copies, not moves), a backup is the safest way to recover from unexpected issues.
+</Aside>
+
+<Steps>
+
+1. **Run a dry run first**
+
+ The dry run logs every document that would be migrated without reading or writing any files.
+
+ <Tabs>
+ <TabItem label="Docker Compose">
+ ```bash
+ docker compose exec papra pnpm maintenance:migrate-document-storage \
+ --from DOCUMENT_STORAGE_DRIVER=filesystem \
+ --to DOCUMENT_STORAGE_DRIVER=s3 \
+ --to DOCUMENT_STORAGE_S3_BUCKET_NAME=my-bucket \
+ --to DOCUMENT_STORAGE_S3_REGION=us-east-1 \
+ --dry-run
+ ```
+ </TabItem>
+ <TabItem label="Docker">
+ ```bash
+ docker exec -it papra pnpm maintenance:migrate-document-storage \
+ --from DOCUMENT_STORAGE_DRIVER=filesystem \
+ --to DOCUMENT_STORAGE_DRIVER=s3 \
+ --to DOCUMENT_STORAGE_S3_BUCKET_NAME=my-bucket \
+ --to DOCUMENT_STORAGE_S3_REGION=us-east-1 \
+ --dry-run
+ ```
+ </TabItem>
+ <TabItem label="Source Installation">
+ ```bash
+ pnpm maintenance:migrate-document-storage \
+ --from DOCUMENT_STORAGE_DRIVER=filesystem \
+ --to DOCUMENT_STORAGE_DRIVER=s3 \
+ --to DOCUMENT_STORAGE_S3_BUCKET_NAME=my-bucket \
+ --to DOCUMENT_STORAGE_S3_REGION=us-east-1 \
+ --dry-run
+ ```
+ </TabItem>
+ </Tabs>
+
+2. **Run the actual migration**
+
+ Once you are happy with the dry run output, run without `--dry-run`. Add `--delete-source` if you want source files removed after each successful copy (USE WITH CAUTION).
+
+ <Tabs>
+ <TabItem label="Docker Compose">
+ ```bash
+ docker compose exec papra pnpm maintenance:migrate-document-storage \
+ --from DOCUMENT_STORAGE_DRIVER=filesystem \
+ --to DOCUMENT_STORAGE_DRIVER=s3 \
+ --to DOCUMENT_STORAGE_S3_BUCKET_NAME=my-bucket \
+ --to DOCUMENT_STORAGE_S3_REGION=us-east-1
+ ```
+ </TabItem>
+ <TabItem label="Docker">
+ ```bash
+ docker exec -it papra pnpm maintenance:migrate-document-storage \
+ --from DOCUMENT_STORAGE_DRIVER=filesystem \
+ --to DOCUMENT_STORAGE_DRIVER=s3 \
+ --to DOCUMENT_STORAGE_S3_BUCKET_NAME=my-bucket \
+ --to DOCUMENT_STORAGE_S3_REGION=us-east-1
+ ```
+ </TabItem>
+ <TabItem label="Source Installation">
+ ```bash
+ pnpm maintenance:migrate-document-storage \
+ --from DOCUMENT_STORAGE_DRIVER=filesystem \
+ --to DOCUMENT_STORAGE_DRIVER=s3 \
+ --to DOCUMENT_STORAGE_S3_BUCKET_NAME=my-bucket \
+ --to DOCUMENT_STORAGE_S3_REGION=us-east-1
+ ```
+ </TabItem>
+ </Tabs>
+
+3. **Update your Papra configuration**
+
+ Point your running Papra instance to the new storage backend and restart it.
+
+4. **Verify the migration**
+
+ Open Papra and check that documents are accessible and downloadable. If anything looks wrong, your original files are still in the source backend (unless you used `--delete-source`).
+
+</Steps>
+
+## Examples
+
+### Filesystem → S3
+
+Move documents from a local directory to an S3 bucket, using AWS credentials already in the environment:
+
+```bash
+docker compose exec papra pnpm maintenance:migrate-document-storage \
+ --from DOCUMENT_STORAGE_DRIVER=filesystem \
+ --from DOCUMENT_STORAGE_FILESYSTEM_ROOT=./app-data/documents \
+ --to DOCUMENT_STORAGE_DRIVER=s3 \
+ --to DOCUMENT_STORAGE_S3_BUCKET_NAME=my-papra-bucket \
+ --to DOCUMENT_STORAGE_S3_REGION=us-east-1
+```
+
+### S3 → Azure Blob Storage
+
+```bash
+docker compose exec papra pnpm maintenance:migrate-document-storage \
+ --from DOCUMENT_STORAGE_DRIVER=s3 \
+ --from DOCUMENT_STORAGE_S3_BUCKET_NAME=old-bucket \
+ --from DOCUMENT_STORAGE_S3_REGION=us-east-1 \
+ --to DOCUMENT_STORAGE_DRIVER=azure-blob \
+ --to DOCUMENT_STORAGE_AZURE_BLOB_ACCOUNT_NAME=myaccount \
+ --to DOCUMENT_STORAGE_AZURE_BLOB_CONTAINER_NAME=papra-documents
+```
+
+### S3 → S3-Compatible (e.g. Cloudflare R2, MinIO)
+
+```bash
+docker compose exec papra pnpm maintenance:migrate-document-storage \
+ --from DOCUMENT_STORAGE_DRIVER=s3 \
+ --from DOCUMENT_STORAGE_S3_BUCKET_NAME=old-bucket \
+ --from DOCUMENT_STORAGE_S3_REGION=us-east-1 \
+ --to DOCUMENT_STORAGE_DRIVER=s3 \
+ --to DOCUMENT_STORAGE_S3_BUCKET_NAME=new-bucket \
+ --to DOCUMENT_STORAGE_S3_ENDPOINT=https://<account-id>.r2.cloudflarestorage.com \
+ --to DOCUMENT_STORAGE_S3_REGION=auto
+```
+
+### Add Encryption While Moving to S3
+
+Move from an unencrypted filesystem to an encrypted S3 backend in one step:
+
+```bash
+docker compose exec papra pnpm maintenance:migrate-document-storage \
+ --from DOCUMENT_STORAGE_DRIVER=filesystem \
+ --to DOCUMENT_STORAGE_DRIVER=s3 \
+ --to DOCUMENT_STORAGE_S3_BUCKET_NAME=my-papra-bucket \
+ --to DOCUMENT_STORAGE_S3_REGION=us-east-1 \
+ --to DOCUMENT_STORAGE_ENCRYPTION_IS_ENABLED=true \
+ --to DOCUMENT_STORAGE_DOCUMENT_KEY_ENCRYPTION_KEYS=<your-encryption-key>
+```
+
+After running this, update your Papra environment variables to include the encryption settings so new documents are also encrypted.
+
+### Change Storage Key Pattern
+
+Each document's storage key is saved in the database when it is uploaded, so Papra always knows where to find a file regardless of what the current pattern setting is. Changing `DOCUMENT_STORAGE_KEY_PATTERN` (or disabling the legacy key system) only affects new uploads — existing documents keep their old keys.
+
+To re-key existing documents, first update your Papra configuration to the desired pattern, then run the migration script without any arguments. The script will read each document from its stored key and write it back under the key your new pattern produces, updating the database record in the process.
+
+```bash
+# 1. Update your Papra config (e.g. in .env or docker-compose.yml):
+# DOCUMENT_STORAGE_USE_LEGACY_STORAGE_KEY_DEFINITION_SYSTEM=false
+# DOCUMENT_STORAGE_KEY_PATTERN={{organization.id}}/{{document.name}}
+
+# 2. Run the migration (no --from / --to needed when staying on the same backend):
+docker compose exec papra pnpm maintenance:migrate-document-storage
+```
+
+The `--from` / `--to` overrides are only necessary when you are also changing the backend driver or root path at the same time.
+
+### Change Filesystem Root Path
+
+If you are moving the storage directory to a new volume, override only the path:
+
+```bash
+docker compose exec papra pnpm maintenance:migrate-document-storage \
+ --from DOCUMENT_STORAGE_DRIVER=filesystem \
+ --from DOCUMENT_STORAGE_FILESYSTEM_ROOT=/old-volume/documents \
+ --to DOCUMENT_STORAGE_DRIVER=filesystem \
+ --to DOCUMENT_STORAGE_FILESYSTEM_ROOT=/new-volume/documents
+```
+
+## Troubleshooting
+
+**"File already exists" or overwrite error**
+
+The source and destination resolve to the same storage location. This happens when you pass the same driver and path on both sides. Use the `maintenance:encrypt-all-documents` script for in-place encryption instead.
+
+**Individual documents fail but migration continues**
+
+Failed documents are logged with their ID, name, and error message. The migration reports a final `succeeded / failed` count. After fixing the underlying issue (permissions, connectivity, missing credentials), you can re-run the script — already-migrated files will fail to overwrite, so re-run with the source pointing to the correct set of remaining files.
+
+**Migration is slow**
+
+The script streams one document at a time. Performance depends on document size and the latency between source and destination storage. For large collections, consider running the migration during a maintenance window.
+
+**Container exits before migration finishes**
+
+For long-running migrations in Docker, make sure no timeout is configured on `docker exec`.
\ No newline at end of file
diff --git a/apps/docs/src/content/navigation.ts b/apps/docs/src/content/navigation.ts
index 4390ba8e..de5fb7af 100644
--- a/apps/docs/src/content/navigation.ts
+++ b/apps/docs/src/content/navigation.ts
@@ -41,6 +41,11 @@ export const sidebar = [
label: 'Document Encryption',
slug: 'guides/document-encryption',
},
+ // TODO: uncomment when released
+ // {
+ // label: 'Migrate Document Storage',
+ // slug: 'guides/migrate-document-storage',
+ // },
{
label: 'Tagging Rules',
slug: 'guides/tagging-rules',
@@ -53,7 +58,7 @@ export const sidebar = [
label: 'Advanced Search',
slug: 'guides/advanced-search',
},
- // Uncomment when the release is made
+ // TODO: uncomment when released
// {
// label: 'Storage Key Patterns',
// slug: 'guides/storage-key-patterns',
diff --git a/apps/papra-server/package.json b/apps/papra-server/package.json
index c09aab75..cb4907b9 100644
--- a/apps/papra-server/package.json
+++ b/apps/papra-server/package.json
@@ -36,7 +36,8 @@
"script:send-intake-email": "tsx --env-file-if-exists=.env src/scripts/send-intake-email.script.ts | crowlog-pretty",
"stripe:webhook": "stripe listen --forward-to localhost:1221/api/stripe/webhook",
"script:make-user-admin": "tsx --env-file-if-exists=.env src/scripts/make-user-admin.script.ts",
- "maintenance:encrypt-all-documents": "tsx --env-file-if-exists=.env src/scripts/encrypt-all-documents.script.ts"
+ "maintenance:encrypt-all-documents": "tsx --env-file-if-exists=.env src/scripts/encrypt-all-documents.script.ts",
+ "maintenance:migrate-document-storage": "tsx --env-file-if-exists=.env src/scripts/migrate-document-storage.script.ts"
},
"dependencies": {
"@aws-sdk/client-s3": "^3.835.0",
@@ -46,6 +47,7 @@
"@cadence-mq/core": "^0.2.1",
"@cadence-mq/driver-libsql": "^0.2.4",
"@cadence-mq/driver-memory": "^0.2.0",
+ "@clack/prompts": "1.0.0-alpha.6",
"@corentinth/chisels": "catalog:",
"@corentinth/friendly-ids": "^0.0.1",
"@crowlog/async-context-plugin": "^2.1.0",
diff --git a/apps/papra-server/src/modules/documents/storage/document-storage.usecases.ts b/apps/papra-server/src/modules/documents/storage/document-storage.usecases.ts
index d56931b8..519dfdc9 100644
--- a/apps/papra-server/src/modules/documents/storage/document-storage.usecases.ts
+++ b/apps/papra-server/src/modules/documents/storage/document-storage.usecases.ts
@@ -66,15 +66,17 @@ export async function createStorageKey({
documentId,
documentName,
organizationId,
- now = new Date(),
documentsStorageService,
+ logger,
+ now = new Date(),
}: {
storagePatternConfig: StoragePatternConfig;
documentId: string;
documentName: string;
organizationId: string;
- now?: Date;
documentsStorageService: Pick<DocumentStorageService, 'fileExists'>;
+ logger?: Logger;
+ now?: Date;
}) {
const {
useLegacyStorageKeyDefinitionSystem,
@@ -96,6 +98,7 @@ export async function createStorageKey({
maxIncrementalSuffixAttempts,
enableRandomSuffixFallback,
documentsStorageService,
+ logger,
});
return { storageKey };
diff --git a/apps/papra-server/src/scripts/commons/args.utils.test.ts b/apps/papra-server/src/scripts/commons/args.utils.test.ts
new file mode 100644
index 00000000..4ad1a8eb
--- /dev/null
+++ b/apps/papra-server/src/scripts/commons/args.utils.test.ts
@@ -0,0 +1,33 @@
+import { describe, expect, test } from 'vitest';
+import { ensureBooleanArg } from './args.utils';
+
+describe('args utils', () => {
+ describe('ensureBooleanArg', () => {
+ test('when parsing cli args, the native node parseArgs does not guarantee the type of boolean args, so we need to ensure they are boolean', () => {
+ expect(ensureBooleanArg(true)).toBe(true);
+ expect(ensureBooleanArg(false)).toBe(false);
+ expect(ensureBooleanArg('true')).toBe(true);
+ expect(ensureBooleanArg('false')).toBe(false);
+ expect(ensureBooleanArg('1')).toBe(true);
+ expect(ensureBooleanArg('0')).toBe(false);
+ expect(ensureBooleanArg('yes')).toBe(true);
+ expect(ensureBooleanArg('no')).toBe(false);
+ expect(ensureBooleanArg('y')).toBe(true);
+ expect(ensureBooleanArg('n')).toBe(false);
+ expect(ensureBooleanArg('on')).toBe(true);
+ expect(ensureBooleanArg('off')).toBe(false);
+
+ expect(() => ensureBooleanArg('invalid')).toThrowError();
+ expect(() => ensureBooleanArg(123)).toThrowError();
+ expect(() => ensureBooleanArg(null)).toThrowError();
+ expect(() => ensureBooleanArg(undefined)).toThrowError();
+ });
+
+ test('should be case-insensitive and trim the input', () => {
+ expect(ensureBooleanArg(' TRUE ')).toBe(true);
+ expect(ensureBooleanArg(' false ')).toBe(false);
+ expect(ensureBooleanArg(' Yes ')).toBe(true);
+ expect(ensureBooleanArg(' No ')).toBe(false);
+ });
+ });
+});
diff --git a/apps/papra-server/src/scripts/commons/args.utils.ts b/apps/papra-server/src/scripts/commons/args.utils.ts
new file mode 100644
index 00000000..11d76af4
--- /dev/null
+++ b/apps/papra-server/src/scripts/commons/args.utils.ts
@@ -0,0 +1,23 @@
+export function ensureBooleanArg(arg: unknown): boolean {
+ if (typeof arg === 'boolean') {
+ return arg;
+ }
+
+ if (typeof arg === 'string') {
+ const value = arg.trim().toLowerCase();
+
+ const isTruthy = ['true', '1', 'yes', 'y', 'on'].includes(value);
+
+ if (isTruthy) {
+ return true;
+ }
+
+ const isFalsy = ['false', '0', 'no', 'n', 'off'].includes(value);
+
+ if (isFalsy) {
+ return false;
+ }
+ }
+
+ throw new Error(`Invalid boolean argument: ${String(arg)}`);
+}
diff --git a/apps/papra-server/src/scripts/commons/run-script.ts b/apps/papra-server/src/scripts/commons/run-script.ts
index 918455c3..80256c73 100644
--- a/apps/papra-server/src/scripts/commons/run-script.ts
+++ b/apps/papra-server/src/scripts/commons/run-script.ts
@@ -8,7 +8,8 @@ import { parseConfig } from '../../modules/config/config';
import { createLogger, wrapWithLoggerContext } from '../../modules/shared/logger/logger';
export async function runScriptWithDb(
- { scriptName }: { scriptName: string },
+ { scriptName, silent = false }: { scriptName: string; silent?: boolean },
+
fn: (args: { isDryRun: boolean; logger: Logger; db: Database; config: Config }) => Promise<void> | void,
) {
const isDryRun = process.argv.includes('--dry-run');
@@ -25,13 +26,13 @@ export async function runScriptWithDb(
await ensureLocalDatabaseDirectoryExists({ config });
const { db } = setupDatabase({ ...config.database });
- await executeScript({ logger, fn: async () => fn({ isDryRun, logger, db, config }) });
+ await executeScript({ logger, fn: async () => fn({ isDryRun, logger, db, config }), silent });
},
);
}
export async function runScript(
- { scriptName }: { scriptName: string },
+ { scriptName, silent = false }: { scriptName: string; silent?: boolean },
fn: (args: { isDryRun: boolean; logger: Logger }) => Promise<void> | void,
) {
const isDryRun = process.argv.includes('--dry-run');
@@ -39,16 +40,20 @@ export async function runScript(
await wrapWithLoggerContext({ scriptName, isDryRun }, async () => {
const logger = createLogger({ namespace: 'scripts' });
- await executeScript({ logger, fn: async () => fn({ isDryRun, logger }) });
+ await executeScript({ logger, fn: async () => fn({ isDryRun, logger }), silent });
});
}
-async function executeScript({ logger, fn }: { logger: Logger; fn: () => Promise<unknown> }) {
+async function executeScript({ logger, fn, silent = false }: { logger: Logger; fn: () => Promise<unknown>; silent?: boolean }) {
try {
await fn();
- logger.debug('Script finished');
+ if (!silent) {
+ logger.debug('Script finished');
+ }
} catch (error) {
- logger.error({ error }, 'Script failed');
+ if (!silent) {
+ logger.error({ error }, 'Script failed');
+ }
process.exit(1);
}
}
diff --git a/apps/papra-server/src/scripts/migrate-document-storage.script.ts b/apps/papra-server/src/scripts/migrate-document-storage.script.ts
new file mode 100644
index 00000000..a51c2435
--- /dev/null
+++ b/apps/papra-server/src/scripts/migrate-document-storage.script.ts
@@ -0,0 +1,196 @@
+import type { Database } from '../modules/app/database/database.types';
+import process from 'node:process';
+import { parseArgs, parseEnv } from 'node:util';
+import * as p from '@clack/prompts';
+import { castError } from '@corentinth/chisels';
+import { createNoopLogger } from '@crowlog/logger';
+import { count, eq } from 'drizzle-orm';
+import { createIterator } from '../modules/app/database/database.usecases';
+import { parseConfig } from '../modules/config/config';
+import { documentsTable } from '../modules/documents/documents.table';
+import { createStorageKey } from '../modules/documents/storage/document-storage.usecases';
+import { createDocumentStorageService } from '../modules/documents/storage/documents.storage.services';
+import { ensureBooleanArg } from './commons/args.utils';
+import { runScriptWithDb } from './commons/run-script';
+
+const envArgsToString = (args: (string | boolean)[] = []) => args.filter(arg => typeof arg === 'string').join('\n');
+
+export async function migrateDocumentStorage({
+ db,
+ fromEnv,
+ toEnv,
+ deleteSource,
+ isDryRun,
+ prompts,
+}: {
+ db: Database;
+ fromEnv: Record<string, string>;
+ toEnv: Record<string, string>;
+ deleteSource: boolean;
+ isDryRun: boolean;
+ prompts?: typeof p;
+}) {
+ const { config: fromConfig } = await parseConfig({ env: { ...process.env, ...fromEnv } });
+ const { config: toConfig } = await parseConfig({ env: { ...process.env, ...toEnv } });
+
+ const fromStorageService = createDocumentStorageService({ documentStorageConfig: fromConfig.documentsStorage });
+ const toStorageService = createDocumentStorageService({ documentStorageConfig: toConfig.documentsStorage });
+
+ prompts?.intro('Document Storage Migration');
+
+ if (isDryRun) {
+ prompts?.log.info('This is a dry run, no actual migration will be performed.\nJust logging the documents that would be migrated.');
+ }
+
+ const migrationErrors: { document: { id: string; originalName: string; originalStorageKey: string }; error: unknown }[] = [];
+
+ const [{ count: documentCount = 0 } = {}] = await db.select({ count: count() }).from(documentsTable);
+
+ if (documentCount === 0) {
+ prompts?.outro('No documents found in the database, nothing to migrate');
+ return;
+ }
+
+ prompts?.log.info(`Found ${documentCount} documents in the database`);
+
+ const query = db
+ .select({
+ id: documentsTable.id,
+ originalStorageKey: documentsTable.originalStorageKey,
+ organizationId: documentsTable.organizationId,
+ originalName: documentsTable.originalName,
+ mimeType: documentsTable.mimeType,
+ fileEncryptionKeyWrapped: documentsTable.fileEncryptionKeyWrapped,
+ fileEncryptionKekVersion: documentsTable.fileEncryptionKekVersion,
+ fileEncryptionAlgorithm: documentsTable.fileEncryptionAlgorithm,
+ })
+ .from(documentsTable)
+ .$dynamic();
+
+ const documentIterator = createIterator({ query });
+
+ const progress = prompts?.progress({
+ style: 'heavy',
+ max: documentCount,
+ });
+
+ progress?.start();
+
+ for await (const document of documentIterator) {
+ const {
+ id,
+ originalStorageKey: storageKey,
+ organizationId,
+ originalName,
+ mimeType,
+ fileEncryptionKeyWrapped,
+ fileEncryptionKekVersion,
+ fileEncryptionAlgorithm,
+ } = document;
+
+ progress?.message(`Migrating ${originalName} - ${storageKey}`);
+
+ if (isDryRun) {
+ await new Promise(resolve => setTimeout(resolve, 10));
+ progress?.advance();
+ continue;
+ }
+
+ try {
+ const { fileStream } = await fromStorageService.getFileStream({
+ storageKey,
+ fileEncryptionKeyWrapped,
+ fileEncryptionKekVersion,
+ fileEncryptionAlgorithm,
+ });
+
+ const { storageKey: newStorageKey } = await createStorageKey({
+ storagePatternConfig: toConfig.documentsStorage.pattern,
+ documentId: id,
+ organizationId,
+ documentName: originalName,
+ documentsStorageService: toStorageService,
+ logger: createNoopLogger(),
+ });
+
+ const encryptionFields = await toStorageService.saveFile({
+ fileStream,
+ fileName: originalName,
+ mimeType,
+ storageKey: newStorageKey,
+ });
+
+ await db
+ .update(documentsTable)
+ .set({
+ originalStorageKey: newStorageKey,
+ fileEncryptionKeyWrapped: encryptionFields.fileEncryptionKeyWrapped,
+ fileEncryptionKekVersion: encryptionFields.fileEncryptionKekVersion,
+ fileEncryptionAlgorithm: encryptionFields.fileEncryptionAlgorithm,
+ })
+ .where(eq(documentsTable.id, id));
+
+ if (deleteSource) {
+ await fromStorageService.deleteFile({ storageKey });
+ }
+ } catch (error) {
+ migrationErrors.push({ document, error });
+ }
+
+ progress?.advance();
+ }
+
+ progress?.stop(`Migration completed: ${documentCount - migrationErrors.length} succeeded, ${migrationErrors.length} failed.`);
+
+ if (migrationErrors.length > 0) {
+ migrationErrors.forEach(({ document, error }) => {
+ const { id, originalName, originalStorageKey } = document;
+ prompts?.log.error(`- Document ID: ${id}, Name: ${originalName}, Storage Key: ${originalStorageKey}\n Error: ${castError(error).message}`);
+ });
+ }
+}
+
+await runScriptWithDb(
+ { scriptName: 'migrate-document-storage', silent: true },
+ async ({ db }) => {
+ const { values } = parseArgs({
+ args: process.argv.slice(2),
+ options: {
+ 'from': { type: 'string', multiple: true, default: [] },
+ 'to': { type: 'string', multiple: true, default: [] },
+ 'delete-source': { type: 'boolean', default: false },
+ 'dry-run': { type: 'boolean', default: false },
+ 'help': { type: 'boolean', short: 'h', default: false },
+ },
+ strict: false,
+ });
+
+ if (ensureBooleanArg(values.help)) {
+ console.log(`
+Usage: migrate-document-storage [options]
+
+Options:
+ --from <env> Source environment variables (can be specified multiple times)
+ --to <env> Destination environment variables (can be specified multiple times)
+ --delete-source Delete source file after successful migration (backup recommended)
+ --dry-run Perform a dry run without actual migration, just logs
+ -h, --help Show this help message
+ `);
+ process.exit(0);
+ }
+
+ const fromEnv = parseEnv(envArgsToString(values.from)) as Record<string, string>;
+ const toEnv = parseEnv(envArgsToString(values.to)) as Record<string, string>;
+ const deleteSource = ensureBooleanArg(values['delete-source']);
+ const isDryRun = ensureBooleanArg(values['dry-run']);
+
+ await migrateDocumentStorage({
+ db,
+ fromEnv,
+ toEnv,
+ deleteSource,
+ isDryRun,
+ prompts: p,
+ });
+ },
+);
diff --git a/pnpm-lock.yaml b/pnpm-lock.yaml
index cff7e999..dea038bf 100644
--- a/pnpm-lock.yaml
+++ b/pnpm-lock.yaml
@@ -452,6 +452,9 @@ importers:
'@cadence-mq/driver-memory':
specifier: ^0.2.0
version: 0.2.0(@cadence-mq/[email protected])
+ '@clack/prompts':
+ specifier: 1.0.0-alpha.6
+ version: 1.0.0-alpha.6
'@corentinth/chisels':
specifier: 'catalog:'
version: 2.1.0