Database governance built for humans and agents — controlling changes and access across every major database.
Go cicddata-governancedata-maskingdata-securitydatabase-accessdatabase-governancedevopsdevsecopsflywaygitopsliquibasemongodbmysqloraclepostgresqlschema-migrationssecuritysql-clientsqlservertidb
docs: design retiring the export-data workflow
Danny7f15bfb2492e
diff --git a/docs/design/retire-export-data-workflow.md b/docs/design/retire-export-data-workflow.mdnew file mode 100644index 0000000000..808156ebc1--- /dev/null+++ b/docs/design/retire-export-data-workflow.md@@ -0,0 +1,208 @@+# Retire the Export-Data Workflow++- Status: design for review — decision: full purge+- Date: 2026-07-31+- Target: main (3.22 dev cycle) — not `release/3.21.0`++## Background++The export center was deprecated in #20693. The SQL Editor "Request Export"+button now opens the JIT access-grant drawer, and JIT-approved exports stream+synchronously through `SQLService.Export` — they never touch `export_archive`.+The plan UI blocks export plans with "Data export issue creation is no longer+supported".++The machinery is still fully wired, reachable only via direct v1 API calls:++- `PlanService.CreatePlan` accepts `export_data_config` specs+ (`backend/api/v1/plan_service.go:650`); rollout creation turns them into+ `DATABASE_EXPORT` tasks (`backend/api/v1/rollout_service_task.go:300`).+- The executor is registered (`backend/server/server.go:222`) and writes+ `export_archive` rows (`backend/runner/taskrun/data_export_executor.go`).+- `SQLService.Export` with a rollout name downloads the archive+ (`doExportFromIssue`, `backend/api/v1/sql_service.go:1038`).+- The data cleaner purges archives older than 24h, so the table is in practice+ always near-empty.++## Decision: full purge++Delete legacy `DATABASE_EXPORT` issues and their plans/tasks/runs in a+migration, drop `export_archive`, and remove every proto value — no+deprecated read surface, no legacy UI.++Considered alternatives: tombstone read-only (keep ~40 deprecated proto lines+so history stays legible) and block-creation-only. Rejected in favor of a+clean end-state: export-center usage was near zero (already deprecated), and+the durable audit trail lives in `audit_log` — issue/plan rows are redundant+with it for compliance purposes.++What is irreversibly lost: the issue-level records of past export requests+(title, approval chain rendering, statement sheet linkage). What remains:+`audit_log` rows for the create/approve/export RPCs, and `query_history` rows+for performed exports.++## Migration (`backend/migrator/migration/3.22/xxxx##retire_export_data.sql`)++`issue.type` and `task.type` are text columns. Plan specs are protojson JSONB+(camelCase keys — the spec key is `exportDataConfig`). All tables involved use+composite PKs `(project, id)`; every join below carries the project column.++Export-plan set: plans referenced by a `DATABASE_EXPORT` issue, UNION plans+whose `config->'specs'` contains an `exportDataConfig` key (covers API-created+plans that never got an issue).++Delete order (children before parents, per the FK graph):++```sql+-- 1. task_run_log → task_run → task(type DATABASE_EXPORT)+DELETE FROM task_run_log l USING task_run r, task t+ WHERE l.project = r.project AND l.task_run_id = r.id+ AND r.project = t.project AND r.task_id = t.id+ AND t.type = 'DATABASE_EXPORT';+-- 2. task_run+DELETE FROM task_run r USING task t+ WHERE r.project = t.project AND r.task_id = t.id+ AND t.type = 'DATABASE_EXPORT';+-- 3. task+DELETE FROM task WHERE type = 'DATABASE_EXPORT';+-- 4-8. with export_plan AS (<set above>):+-- plan_check_run, issue_comment (via export issues), issue,+-- plan_webhook_delivery, plan+-- 9. DROP TABLE export_archive;+```++Also: update `LATEST.sql` (drop the `export_archive` table, update the+`issue.type` comment), and bump `TestLatestVersion` in `migrator_test.go`.++Notes:++- The whole file runs in one transaction; DML + `DROP TABLE` is fine in PG.+- The `jsonb_array_elements` scan over `plan` is a one-time full-table scan;+ acceptable at migration time.+- No new store methods → no new `TestCollision_*` obligations.+- Historical migration files (3.4, 3.17) that mention `export_archive` are+ immutable and stay untouched.++## Proto removal (all tags `reserved`, number and name)++| File | Change |+| --- | --- |+| `store/export_archive.proto` | delete file (+ delete stale generated `export_archive*.pb.go`; `buf generate` does not remove them) |+| `store/issue.proto` | remove `Issue.Type.DATABASE_EXPORT = 3` |+| `store/task.proto` | remove `Task.Type.DATABASE_EXPORT = 3` |+| `store/plan.proto` | remove `Spec.export_data_config = 7` oneof arm + `ExportDataConfig` message |+| `store/task_run.proto` | remove `export_archive_id = 9` |+| `v1/issue_service.proto` | remove `Issue.Type.DATABASE_EXPORT = 3` |+| `v1/plan_service.proto` | remove `Spec.export_data_config = 4` + `ExportDataConfig` message; fix spec_type filter doc (`:139`) |+| `v1/rollout_service.proto` | remove `Task.Type.DATABASE_EXPORT = 4`, `database_data_export = 9` + `DatabaseDataExport` message, `ExportArchiveStatus` enum + field 10; fix task_type filter doc (`:229`) |++Then `buf format -w proto && buf lint proto && cd proto && buf generate`+(regenerates backend `generated-go` and frontend `types/proto-es`). If CI runs+`buf breaking`, this PR is intentionally breaking — handle per the pre-PR+breaking-change review.++## Backend removal++Compile-driven after proto regen; the enum removals surface every remaining+reference.++- Delete `runner/taskrun/data_export_executor.go`; unregister in+ `server.go:222`+- Delete `store/export_archive.go`; remove `DeleteExpiredExportArchivesAll`+ (`store/runner_queries.go`) and `cleanupExportArchives` + retention const+ (`runner/cleaner/data_cleaner.go`)+- `api/v1/sql_service.go`: remove `doExportFromIssue` and the rollout-name+ branch of `Export` (RPC then accepts only database targets); remove the zip+ re-encrypt helpers if unused elsewhere (`doEncrypt` is shared with the+ synchronous path — verify before removing)+- `api/v1/rollout_service_task.go`: remove `getTaskCreatesFromExportDataConfig`+ + its spec-switch case+- `api/v1/plan_service.go`: `CreatePlan`/`UpdatePlan` reject+ `export_data_config` specs with an explicit error; remove both converter+ directions+- `api/v1/issue_service.go`: `CreateIssue` rejects `DATABASE_EXPORT`;+ remove converter cases (`issue_service_converter.go`)+- `api/v1/issue_hook.go`: remove the `Issue_DATABASE_EXPORT` branches from+ approval finding+- `api/v1/rollout_service.go:289,1330`: remove creator-only export special+ cases+- `api/v1/rollout_service_converter.go`: remove `ExportArchiveStatus`+ population, `convertToTaskFromDatabaseDataExport`, task-type mapping+- `component/review/evaluator.go`: remove `Issue_DATABASE_EXPORT` branches+ (`:335,1053`); KEEP `buildCELVariablesForDataExport` and the JIT+ access-grant path (`:864` comment documents the split)+- `runner/taskrun/running_scheduler.go:296`,+ `runner/taskrun/database_migrate_executor.go:1042`: prune from+ `exhaustive:enforce` switches+- Tests: prune export cases in `plan_service_test.go`,+ `issue_service_test.go`; add rejection tests (below)++## Frontend removal++Type-check-driven after proto regen.++- Delete `IssueDetailDatabaseExportView.tsx` (~800 lines) and its route wiring+ (`useIssueDetailType.ts`)+- `actionRegistry.ts`: `exportArchiveReady`, `computeExportArchiveReady`,+ download action; `IssueDetailActionBar.tsx`: download/expiry UI+- Export gating in `IssueDetailTaskRolloutActionPanel`,+ `PlanDetailTaskRolloutActionPanel`, `DeployTaskToolbar`+ (`data-export-creator-only`)+- `advanceState.ts` export blocker (backend rejection replaces it)+- `IssueTable.tsx:537`: remove the `DATABASE_EXPORT` filter option+- Prune export branches across `utils/v1/issue/*`, `lib/plan/*`,+ `plan-detail/utils/*` (`createPlan.ts` `targetsForSpec`, `changeReference`,+ `phaseSummary`, `rolloutPreview`, `diffPlanSpecs`, `check`, `workflow`),+ `useRedirects.ts`, `useIssueDetailSpecValidation.ts`,+ `PlanDetailStatementSection.tsx`, `IssueDetailStatementSection.tsx`,+ `types/v1/issue/issue.ts`, `issueDetailRedirect.test.ts`+- Locales: remove `issue.data-export.*`, `task.data-export-creator-only`, and+ any keys that become unused — across all locale files. KEEP `export-data.*`+ keys used by `DataExportButton` and `sql-editor.request-export` (JIT)++## Explicitly kept (shared with live features)++- JIT access-grant flow: `RequestExportButton`, `AccessGrantRequestDrawer`,+ `bb.accessGrants.*`+- Synchronous SQL Editor export: `DataExportButton`, `SQLService.Export` with+ database targets, `doExport`, export masking/permissions+- Risk source `DATA_EXPORT` / CEL namespace `request.data_export` and+ `buildCELVariablesForDataExport` (used by JIT grants and risk rules)+- `query_history` export rows; `audit_log` — untouched+- Historical migration files referencing `export_archive`++## Compatibility++- Breaking v1 API change: `CreatePlan`/`CreateIssue` reject export specs;+ `Issue.Type.DATABASE_EXPORT`, `Task.Type.DATABASE_EXPORT`,+ `Task.DatabaseDataExport`, `TaskRun.export_archive_status`, and+ `Plan.ExportDataConfig` leave the wire. Goes through the pre-PR+ breaking-change review.+- Terraform provider does not manage plans/issues; `bytebase-action` creates+ migration rollouts only — both unaffected.+- In-flight export issues at upgrade time are deleted with the rest; there is+ no cancel-and-keep state under full purge.+- Downgrade: none (standard for migrations).++## Testing & verification++1. Migrator: `TestLatestVersion`; migration applies on a DB seeded with a DONE+ export issue (plan + task + task_run + comment + archive row) and leaves no+ orphans — verify FK integrity and that unrelated issues/plans survive.+2. API rejection tests: `CreatePlan` with `export_data_config` →+ InvalidArgument; `CreateIssue` type `DATABASE_EXPORT` → InvalidArgument;+ `SQLService.Export` with a rollout name → InvalidArgument.+3. Full gates: `gofmt`, `golangci-lint` (repeat until clean), backend build,+ `buf format/lint/generate`, `pnpm fix/check/type-check/test`.+4. Residual grep: `ExportArchive|DATABASE_EXPORT|exportDataConfig|DatabaseDataExport`+ returns nothing outside historical migrations and the kept JIT/risk+ surfaces.++## Implementation order++1. Branch off `origin/main` (currently on `release/3.21.0`)+2. Migration + `LATEST.sql` + migrator test+3. Proto edits + regen (backend and frontend generated code)+4. Backend removal, compile-driven; backend tests+5. Frontend removal, type-check-driven; frontend tests+6. Verification gates + residual grep; PR with breaking-change checklist