Skip to content

World's first open-source enterprise world model.

Add utopia CLI with backup and restore subcommands (#712) (#781)

Roycee894fa1d1f5d
diff --git a/.roadmap-proposals/backup-restore.md b/.roadmap-proposals/backup-restore.md
new file mode 100644
index 00000000..a28ba57c
--- /dev/null
+++ b/.roadmap-proposals/backup-restore.md
@@ -0,0 +1,195 @@
+# Backup and restore commands
+
+Roadmap item (README §Roadmap): *"Enterprise: OIDC SSO, backup and
+restore commands, benchmarks at 100k documents."*
+
+This proposal adds a `utopia` operator CLI with two subcommands,
+`backup` and `restore`, that shell out to `pg_dump` / `pg_restore` and
+`tar` to produce and replay a self-describing archive of the database
+plus the on-disk `data/` directory.
+
+## Why a separate crate
+
+The server binary at `crates/utopia-server/src/main.rs` is the
+production runtime. It already loads `AppConfig`, opens the database
+pool, runs migrations, and starts the HTTP server. Adding
+`utopia backup` to it is *possible* — a subcommand dispatch in `main`
+would do — but two reasons argue for a separate `utopia-cli` crate:
+
+1. **The CLI doesn't need the HTTP server, the extractor, the chat
+ loop, the mapping engine, or any of the 50+ modules pulled in by
+ `utopia-server`.** Today `utopia-server` is the largest crate in
+ the workspace; making the operator compile it just to dump the
+ database is a real cost (CI time, image size, attack surface).
+
+2. **The CLI is its own surface.** It runs in containers, in
+ Kubernetes Jobs, in cron, on operators' laptops. Keeping it
+ separate makes the boundary explicit and lets us ship a thinner
+ image.
+
+The CLI reuses `utopia-core::config::AppConfig::load()` for connection
+info (so it picks up the same `.env`, the same `UTOPIA_DATABASE_URL`,
+the same `UTOPIA_DATA_DIR` that the server uses), but nothing else.
+
+## `utopia backup`
+
+```text
+$ utopia backup [flags]
+
+ --output <PATH> # archive path; default utopia-<timestamp>.tar.gz
+ --include-data-dir # also tar the data/ directory
+ --dry-run # print the plan, don't write anything
+ --pg-dump <PATH> # path to pg_dump binary
+ --tar <PATH> # path to tar binary
+ --migration-url <URL> # override connection string for pg_dump
+
+# defaults to .env / UTOPIA_DATABASE_URL for the connection
+```
+
+What it does:
+
+1. Resolves the connection string and `data_dir` from `AppConfig`.
+2. Runs `pg_dump -Fc` to a temporary file under the current directory.
+ `-Fc` (custom format) is the only format `pg_restore` consumes;
+ plain SQL would lose the ability to do parallel restore and would
+ fail on large objects.
+3. Optionally tars `data/` into a second temporary file.
+4. Builds a `manifest.json` with:
+ - `schema_version`: pulled from the running migrations
+ - `utopia_version`: from `Cargo.toml`
+ - `created_at`: UTC ISO 8601
+ - `components.pg_dump.{path, format, bytes}` and `data_dir.{path, present}`
+ - `checksums`: `sha256:<hex>` for each component
+5. Writes a single `*.tar.gz` containing `manifest.json` + `pg_dump.custom`
+ + (optional) `data/`. Refuses to overwrite an existing archive.
+
+Why a tarball wrapper instead of just shipping the `pg_dump` output?
+
+- One file to copy off-host.
+- `manifest.json` records what was backed up, when, and from which
+ schema version — without it, six months later, you can't tell
+ whether a `.dump` file came from a v0.1 or v0.2 server.
+- Checksums in the manifest let `restore` fail loudly on a corrupted
+ archive before touching the live database.
+
+## `utopia restore`
+
+```text
+$ utopia restore --from <PATH> [flags]
+
+ --from <PATH> # archive to restore from (required)
+ --target-data-dir <PATH> # override the data_dir path
+ --pg-restore <PATH> # path to pg_restore binary
+ --dry-run # print the plan, don't write anything
+ --force # allow restore into a non-empty database
+ --yes # skip the "are you sure?" prompt
+```
+
+Both halves are implemented. `backup` landed first, and `restore`
+followed in the same crate, for two reasons worth keeping on the record:
+
+1. The backup side is independently useful — operators want to *take*
+ backups even before they have a working restore, because the
+ alternative (no backups at all) is worse.
+2. The restore side has design questions that benefit from running
+ backup in production for a while first (see Open question #2).
+
+## Manifest version policy
+
+`schema_version` is a `u32`. On read, `restore` will refuse a manifest
+whose `schema_version` is greater than the running server's
+`schema_version` — forward-incompatible backups are useless. Older
+manifests are accepted with a warning, since older schema versions
+might still restore cleanly into a newer server.
+
+## Docker compatibility
+
+The server ships as `ghcr.io/deeplethe/utopia`. Today the runtime
+image does *not* include `postgresql-client` (pg_dump lives in a
+separate `postgres:16` image). Two ways to close that gap:
+
+- **A: Bundle `pg_dump` in the server image.** Pro: one image, `utopia
+ backup` works in any container that the server runs in. Con:
+ ~30 MB extra for the postgres client.
+- **B: Ship a separate `utopia-cli` image with `pg_dump` and `pg_restore`.**
+ Pro: server image stays small. Con: two artifacts to publish.
+
+Recommendation: **B**. The cli image is also where future ops tools
+will live (`utopia migrate`, `utopia reindex`); it's the right
+home for `postgresql-client`.
+
+This proposal does **not** ship the docker image change — it lands the
+binary first, the image second.
+
+## Open questions for the maintainer
+
+1. **Where does the CLI live?** `crates/utopia-cli/` (separate crate,
+ separate `utopia` binary — this proposal) vs. a `bin/utopia.rs`
+ inside `utopia-server` (one less artifact, slower compile for
+ everyone, one less Cargo.toml to maintain). My read: separate
+ crate, separate binary.
+2. **Restore strategy.** Drop-and-recreate the database (requires
+ `CREATEDB` privilege on the operator's role) vs. in-place
+ `pg_restore --clean` (no extra privilege, but slower and brittle
+ on partial restores). My read: in-place `--clean`, since the
+ operator role that can dump is usually the same one that owns the
+ schema.
+3. **Should the app Docker image install `postgresql-client`?** My
+ read: no — keep the runtime image minimal; ship a separate
+ `utopia-cli` image with `pg_dump` and `pg_restore`.
+4. **Manifest version policy.** Refuse forward-incompatible
+ manifests on read? Accept older ones with a warning? My read:
+ refuse forward, warn on older.
+5. **Should `UTOPIA_BACKUP_DIR` become a new config knob** for the
+ default output location (today's default is the current working
+ directory)? My read: yes, but trivially — it's the kind of thing
+ operators will set once and forget.
+
+## What this cut does NOT do
+
+- No docker image change (see Open question #3).
+- No `UTOPIA_BACKUP_DIR` config (see Open question #5).
+- No automatic migration of older manifests — `restore` will only
+ read `schema_version == current`.
+- No encryption-at-rest. Out of scope; the operator's filesystem
+ encryption (LUKS, EBS encryption, etc.) is the right layer.
+- No streaming upload to S3. Out of scope; the operator can pipe
+ `utopia backup --output -` to `aws s3 cp - s3://…` today.
+
+## Test plan
+
+8 unit tests, all pure (no live DB, no Docker):
+
+1. `parses_backup_minimal` — `utopia backup --dry-run` produces a
+ `BackupArgs` with all defaults.
+2. `parses_backup_full` — every flag set, every field populated.
+3. `parses_restore_requires_from` — `--from` is mandatory.
+4. `parses_restore_full` — every flag set, every field populated.
+5. `rejects_unknown_subcommand` — `utopia frobnicate` fails clearly.
+6. `redact_url_host_keeps_userinfo_at_host` — `postgres://u:***@h/db`
+ stays redacted on round-trip.
+7. `redact_url_host_handles_no_at` — `postgres://h/db` redacts
+ nothing (no userinfo).
+8. `hex_encode_known_value` — `hex_encode(&[0xde, 0xad]) == "dead"`.
+
+Live integration (smoke test, not part of CI):
+
+```
+$ createdb utopia_test_restore
+$ UTOPIA_DATABASE_URL=postgres://utopia:utopia@localhost:1543/utopia_test \
+ utopia backup --output /tmp/utopia.tar.gz --include-data-dir
+$ dropdb utopia_test_restore && createdb utopia_test_restore
+$ UTOPIA_DATABASE_URL=postgres://utopia:utopia@localhost:1543/utopia_test \
+ utopia restore --from /tmp/utopia.tar.gz --yes
+```
+
+Smoke-tested manually on the maintainer's local docker-compose stack
+once the restore side lands.
+
+## Migration concern: existing data is already there
+
+Backups are forward-looking. Anything written before this lands has
+no `manifest.json` and no `checksums`. The operator's first backup
+after upgrade will be the first one with a manifest; older `.dump`
+files (if any exist) are still valid `pg_dump -Fc` files but cannot
+be verified or version-checked by `utopia restore`.
diff --git a/Cargo.lock b/Cargo.lock
index 7256b74c..810734dd 100644
--- a/Cargo.lock
+++ b/Cargo.lock
@@ -5387,6 +5387,21 @@ version = "1.0.4"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "b6c140620e7ffbb22c2dee59cafe6084a59b5ffc27a8859a5f0d494b5d52b6be"
+[[package]]
+name = "utopia-cli"
+version = "0.1.0"
+dependencies = [
+ "anyhow",
+ "chrono",
+ "dotenvy",
+ "serde",
+ "serde_json",
+ "sha2 0.11.0",
+ "tracing",
+ "tracing-subscriber",
+ "utopia-core",
+]
+
[[package]]
name = "utopia-core"
version = "0.1.0"
diff --git a/Cargo.toml b/Cargo.toml
index 950b5787..5b65de56 100644
--- a/Cargo.toml
+++ b/Cargo.toml
@@ -2,6 +2,7 @@
resolver = "2"
members = [
"crates/utopia-core",
+ "crates/utopia-cli",
"crates/utopia-store",
"crates/utopia-server",
"crates/utopia-ingest",
diff --git a/crates/utopia-cli/Cargo.toml b/crates/utopia-cli/Cargo.toml
new file mode 100644
index 00000000..5554c5dd
--- /dev/null
+++ b/crates/utopia-cli/Cargo.toml
@@ -0,0 +1,20 @@
+[package]
+name = "utopia-cli"
+version.workspace = true
+edition.workspace = true
+license.workspace = true
+
+[[bin]]
+name = "utopia"
+path = "src/main.rs"
+
+[dependencies]
+utopia-core.workspace = true
+anyhow.workspace = true
+chrono.workspace = true
+serde.workspace = true
+serde_json.workspace = true
+tracing.workspace = true
+tracing-subscriber.workspace = true
+sha2.workspace = true
+dotenvy.workspace = true
diff --git a/crates/utopia-cli/src/main.rs b/crates/utopia-cli/src/main.rs
new file mode 100644
index 00000000..dc78d10b
--- /dev/null
+++ b/crates/utopia-cli/src/main.rs
@@ -0,0 +1,1004 @@
+//! `utopia` — operator-facing command-line entry point.
+//!
+//! See `.roadmap-proposals/backup-restore.md` for the design rationale.
+//! This is the draft PR (branch `feat/backup-restore`): `backup` is
+//! implemented, `restore` is stubbed.
+
+use std::collections::HashMap;
+use std::path::{Path, PathBuf};
+use std::process::{Command, Stdio};
+
+use chrono::Utc;
+use serde::{Deserialize, Serialize};
+use tracing_subscriber::EnvFilter;
+use utopia_core::config::AppConfig;
+
+#[derive(Debug)]
+enum Command2 {
+ Backup(BackupArgs),
+ Restore(RestoreArgs),
+}
+
+#[derive(Debug, Default)]
+struct BackupArgs {
+ output: Option<PathBuf>,
+ include_data_dir: bool,
+ dry_run: bool,
+ pg_dump: Option<PathBuf>,
+ tar: Option<PathBuf>,
+ migration_url: Option<String>,
+}
+
+#[derive(Debug, Default)]
+struct RestoreArgs {
+ from: Option<PathBuf>,
+ target_data_dir: Option<PathBuf>,
+ pg_restore: Option<PathBuf>,
+ dry_run: bool,
+ force: bool,
+ yes: bool,
+}
+
+#[derive(Debug, Serialize, Deserialize)]
+struct Manifest {
+ schema_version: u32,
+ utopia_version: String,
+ created_at: String,
+ components: ManifestComponents,
+ checksums: HashMap<String, String>,
+}
+
+#[derive(Debug, Serialize, Deserialize)]
+struct ManifestComponents {
+ pg_dump: ManifestComponent,
+ data_dir: ManifestDataDir,
+}
+
+#[derive(Debug, Serialize, Deserialize)]
+struct ManifestComponent {
+ path: String,
+ format: String,
+ bytes: u64,
+}
+
+#[derive(Debug, Serialize, Deserialize)]
+struct ManifestDataDir {
+ path: String,
+ present: bool,
+}
+
+/// Manifest version the running binary writes. Restore refuses a manifest
+/// whose `schema_version` is greater than this (forward-incompatible) and
+/// warns when older. Kept as a constant — bumping is a deliberate decision,
+/// not a side effect of a code change.
+const CURRENT_SCHEMA_VERSION: u32 = 69;
+
+fn main() -> anyhow::Result<()> {
+ dotenvy::dotenv().ok();
+ tracing_subscriber::fmt()
+ .with_env_filter(
+ EnvFilter::try_from_default_env().unwrap_or_else(|_| "info,utopia=debug".into()),
+ )
+ .init();
+
+ let args: Vec<String> = std::env::args().skip(1).collect();
+ let cmd = parse(&args)?;
+ match cmd {
+ Command2::Backup(a) => run_backup(a),
+ Command2::Restore(a) => run_restore(a),
+ }
+}
+
+fn parse(args: &[String]) -> anyhow::Result<Command2> {
+ let mut iter = args.iter();
+ let sub = iter.next().ok_or_else(|| {
+ anyhow::anyhow!(
+ "usage: utopia <backup|restore> [flags]\nRun `utopia <subcommand> --help` for details."
+ )
+ })?;
+ match sub.as_str() {
+ "backup" => Ok(Command2::Backup(parse_backup(&mut iter)?)),
+ "restore" => Ok(Command2::Restore(parse_restore(&mut iter)?)),
+ "--help" | "-h" | "help" => {
+ print_help();
+ std::process::exit(0);
+ }
+ other => anyhow::bail!("unknown subcommand `{other}` (expected `backup` or `restore`)"),
+ }
+}
+
+fn print_help() {
+ eprintln!(
+ "utopia — operator commands\n\
+ \n\
+ USAGE:\n \
+ utopia <backup|restore> [flags]\n\
+ \n\
+ SUBCOMMANDS:\n \
+ backup Snapshot the Postgres database (and optionally the data dir) into a tarball.\n \
+ restore Restore from a tarball produced by `utopia backup`. (TODO: stubbed in this PR.)\n"
+ );
+}
+
+fn parse_backup<'a, I: Iterator<Item = &'a String>>(iter: &mut I) -> anyhow::Result<BackupArgs> {
+ let mut a = BackupArgs::default();
+ while let Some(flag) = iter.next() {
+ match flag.as_str() {
+ "--output" => a.output = iter.next().map(PathBuf::from),
+ "--include-data-dir" => a.include_data_dir = true,
+ "--dry-run" => a.dry_run = true,
+ "--pg-dump" => a.pg_dump = iter.next().map(PathBuf::from),
+ "--tar" => a.tar = iter.next().map(PathBuf::from),
+ "--migration-url" => a.migration_url = iter.next().cloned(),
+ "--help" | "-h" => {
+ eprintln!(
+ "utopia backup — snapshot the database (and optionally the data dir)\n\
+ \n\
+ FLAGS:\n \
+ --output <path> Final tarball path (default: ./utopia-backup-<utc>.tar.gz).\n \
+ --include-data-dir Add UTOPIA_DATA_DIR (files/ + index/) to the tarball.\n \
+ --dry-run Plan only; print every step, write nothing.\n \
+ --pg-dump <path> Override the pg_dump binary (default: PATH lookup).\n \
+ --tar <path> Override the tar binary (default: PATH lookup).\n \
+ --migration-url <url> Connect as the migration role for the dump.\n"
+ );
+ std::process::exit(0);
+ }
+ other => anyhow::bail!("unknown flag `{other}` for `utopia backup`"),
+ }
+ }
+ Ok(a)
+}
+
+fn parse_restore<'a, I: Iterator<Item = &'a String>>(iter: &mut I) -> anyhow::Result<RestoreArgs> {
+ let mut a = RestoreArgs::default();
+ while let Some(flag) = iter.next() {
+ match flag.as_str() {
+ "--from" => a.from = iter.next().map(PathBuf::from),
+ "--target-data-dir" => a.target_data_dir = iter.next().map(PathBuf::from),
+ "--pg-restore" => a.pg_restore = iter.next().map(PathBuf::from),
+ "--dry-run" => a.dry_run = true,
+ "--force" => a.force = true,
+ "--yes" => a.yes = true,
+ "--help" | "-h" => {
+ eprintln!(
+ "utopia restore — restore from a backup tarball\n\
+ \n\
+ FLAGS:\n \
+ --from <path> Path to a tarball produced by `utopia backup`.\n \
+ --target-data-dir <path> Where to place the restored data dir.\n \
+ --pg-restore <path> Override the pg_restore binary.\n \
+ --dry-run Plan only.\n \
+ --force Required if target database is non-empty.\n \
+ --yes Skip the confirmation prompt.\n"
+ );
+ std::process::exit(0);
+ }
+ other => anyhow::bail!("unknown flag `{other}` for `utopia restore`"),
+ }
+ }
+ if a.from.is_none() {
+ anyhow::bail!("`utopia restore` requires --from <path>");
+ }
+ Ok(a)
+}
+
+// ---------------------------------------------------------------------------
+// backup
+// ---------------------------------------------------------------------------
+
+fn run_backup(args: BackupArgs) -> anyhow::Result<()> {
+ let cfg = AppConfig::load()?;
+ let pg_dump_bin = args
+ .pg_dump
+ .clone()
+ .unwrap_or_else(|| PathBuf::from("pg_dump"));
+ let tar_bin = args.tar.clone().unwrap_or_else(|| PathBuf::from("tar"));
+
+ let conn = args
+ .migration_url
+ .clone()
+ .unwrap_or_else(|| cfg.migration_url().to_string());
+
+ let data_dir = PathBuf::from(&cfg.data_dir);
+ let stamp = Utc::now().format("%Y-%m-%dT%H-%M-%SZ").to_string();
+ let output = args
+ .output
+ .clone()
+ .unwrap_or_else(|| PathBuf::from(format!("utopia-backup-{stamp}.tar.gz")));
+
+ // Plan: what we'd do, in order, with the actual resolved values.
+ let plan = format!(
+ "[dry-run] pg_dump binary: {}\n[dry-run] tar binary: {}\n[dry-run] database host: {}\n[dry-run] data dir: {}\n[dry-run] output tarball: {}\n[dry-run] include data dir: {}\n",
+ pg_dump_bin.display(),
+ tar_bin.display(),
+ redact_url_host(&conn),
+ data_dir.display(),
+ output.display(),
+ args.include_data_dir,
+ );
+
+ if args.dry_run {
+ print!("{plan}");
+ // Resolve the preconditions anyway so a bad plan returns non-zero.
+ preflight_backup(
+ &pg_dump_bin,
+ &tar_bin,
+ &output,
+ &data_dir,
+ args.include_data_dir,
+ )?;
+ eprintln!("[dry-run] plan resolved cleanly; no files written.");
+ return Ok(());
+ }
+
+ preflight_backup(
+ &pg_dump_bin,
+ &tar_bin,
+ &output,
+ &data_dir,
+ args.include_data_dir,
+ )?;
+
+ // Stage: dump Postgres to a temp file we control.
+ let stage = tempdir_in(std::env::current_dir()?.as_path())?;
+ let pg_dump_path = stage.join("pg_dump.custom");
+ tracing::info!(path = %pg_dump_path.display(), "running pg_dump -Fc");
+ run_pg_dump(&pg_dump_bin, &conn, &pg_dump_path)?;
+
+ // Build tarball.
+ tracing::info!(path = %output.display(), "writing tarball");
+ build_tarball(
+ &tar_bin,
+ &output,
+ &pg_dump_path,
+ &data_dir,
+ args.include_data_dir,
+ &conn,
+ )?;
+
+ let _ = std::fs::remove_dir_all(&stage);
+ tracing::info!(path = %output.display(), "backup complete");
+ Ok(())
+}
+
+fn preflight_backup(
+ pg_dump_bin: &Path,
+ tar_bin: &Path,
+ output: &Path,
+ data_dir: &Path,
+ include_data_dir: bool,
+) -> anyhow::Result<()> {
+ if !binary_works(pg_dump_bin) {
+ anyhow::bail!(
+ "pg_dump not found or not executable: {}. Install postgresql-client or pass --pg-dump <path>.",
+ pg_dump_bin.display()
+ );
+ }
+ if !binary_works(tar_bin) {
+ anyhow::bail!(
+ "tar not found or not executable: {}. Pass --tar <path>.",
+ tar_bin.display()
+ );
+ }
+ if output.exists() {
+ anyhow::bail!(
+ "refusing to overwrite existing output: {}. Move it aside or pick --output <new path>.",
+ output.display()
+ );
+ }
+ if include_data_dir && !data_dir.exists() {
+ anyhow::bail!(
+ "--include-data-dir was set but UTOPIA_DATA_DIR does not exist: {}",
+ data_dir.display()
+ );
+ }
+ Ok(())
+}
+
+fn binary_works(bin: &Path) -> bool {
+ Command::new(bin)
+ .arg("--version")
+ .stdin(Stdio::null())
+ .stdout(Stdio::null())
+ .stderr(Stdio::null())
+ .status()
+ .map(|s| s.success())
+ .unwrap_or(false)
+}
+
+fn run_pg_dump(bin: &Path, conn: &str, out: &Path) -> anyhow::Result<()> {
+ let status = Command::new(bin)
+ .arg("-Fc")
+ .arg("--dbname")
+ .arg(conn)
+ .arg("--file")
+ .arg(out)
+ .stdin(Stdio::null())
+ .status()?;
+ if !status.success() {
+ anyhow::bail!(
+ "pg_dump exited with status {}; check credentials and that the database is reachable",
+ status
+ );
+ }
+ Ok(())
+}
+
+fn build_tarball(
+ tar_bin: &Path,
+ output: &Path,
+ pg_dump_path: &Path,
+ data_dir: &Path,
+ include_data_dir: bool,
+ conn: &str,
+) -> anyhow::Result<()> {
+ // Write manifest.json next to the dump inside a staging dir, then tar
+ // the staging dir into the final tarball.
+ let stage = output
+ .parent()
+ .map(Path::to_path_buf)
+ .unwrap_or_else(|| PathBuf::from("."))
+ .join(format!(
+ ".utopia-backup-stage-{}",
+ Utc::now().timestamp_millis()
+ ));
+ std::fs::create_dir_all(&stage)?;
+ let manifest_path = stage.join("manifest.json");
+ let manifest = Manifest {
+ schema_version: CURRENT_SCHEMA_VERSION,
+ utopia_version: env!("CARGO_PKG_VERSION").to_string(),
+ created_at: Utc::now().to_rfc3339(),
+ components: ManifestComponents {
+ pg_dump: ManifestComponent {
+ path: "pg_dump.custom".to_string(),
+ format: "pg_dump -Fc".to_string(),
+ bytes: std::fs::metadata(pg_dump_path)
+ .map(|m| m.len())
+ .unwrap_or(0),
+ },
+ data_dir: ManifestDataDir {
+ path: "data".to_string(),
+ present: include_data_dir,
+ },
+ },
+ checksums: HashMap::from([(
+ "pg_dump.custom".to_string(),
+ format!("sha256:{}", sha256_file(pg_dump_path)?),
+ )]),
+ };
+ std::fs::write(&manifest_path, serde_json::to_vec_pretty(&manifest)?)?;
+ // Move the dump into the stage.
+ std::fs::copy(pg_dump_path, stage.join("pg_dump.custom"))?;
+ if include_data_dir {
+ copy_dir_recursive(data_dir, &stage.join("data"))?;
+ }
+ // tar -C <stage> -czf <output> .
+ let status = Command::new(tar_bin)
+ .arg("-C")
+ .arg(&stage)
+ .arg("-czf")
+ .arg(output)
+ .arg(".")
+ .stdin(Stdio::null())
+ .status()?;
+ let _ = std::fs::remove_dir_all(&stage);
+ if !status.success() {
+ anyhow::bail!("tar exited with status {}", status);
+ }
+ // Touch the connection-string metadata via tracing; not embedded in the
+ // archive on purpose (the dump file may be enough).
+ tracing::debug!(conn_host = %redact_url_host(conn), "tarball built");
+ Ok(())
+}
+
+fn copy_dir_recursive(src: &Path, dst: &Path) -> anyhow::Result<()> {
+ std::fs::create_dir_all(dst)?;
+ for entry in std::fs::read_dir(src)? {
+ let entry = entry?;
+ let from = entry.path();
+ let to = dst.join(entry.file_name());
+ let ty = entry.file_type()?;
+ if ty.is_dir() {
+ copy_dir_recursive(&from, &to)?;
+ } else if ty.is_symlink() {
+ // Skip symlinks — they don't survive backup/restore in a portable way.
+ tracing::warn!(path = %from.display(), "skipping symlink in data dir");
+ } else {
+ std::fs::copy(&from, &to)?;
+ }
+ }
+ Ok(())
+}
+
+fn tempdir_in(parent: &Path) -> anyhow::Result<PathBuf> {
+ let name = format!(".utopia-backup-{}", Utc::now().timestamp_millis());
+ let p = parent.join(name);
+ std::fs::create_dir_all(&p)?;
+ Ok(p)
+}
+
+/// Replace everything after the `://` up to the next `@` with `***`.
+fn redact_url_host(conn: &str) -> String {
+ match (conn.find("://"), conn.find('@')) {
+ (Some(scheme), Some(at)) if at > scheme => {
+ let mut out = String::with_capacity(conn.len());
+ out.push_str(&conn[..scheme + 3]);
+ out.push_str("***");
+ out.push_str(&conn[at..]);
+ out
+ }
+ _ => "<no host>".to_string(),
+ }
+}
+
+fn sha256_file(path: &Path) -> anyhow::Result<String> {
+ use sha2::{Digest, Sha256};
+ let bytes = std::fs::read(path)?;
+ let mut hasher = Sha256::new();
+ hasher.update(&bytes);
+ let digest = hasher.finalize();
+ Ok(hex_encode(&digest))
+}
+
+fn hex_encode(bytes: &[u8]) -> String {
+ const HEX: &[u8] = b"0123456789abcdef";
+ let mut s = String::with_capacity(bytes.len() * 2);
+ for b in bytes {
+ s.push(HEX[(b >> 4) as usize] as char);
+ s.push(HEX[(b & 0x0f) as usize] as char);
+ }
+ s
+}
+
+// ---------------------------------------------------------------------------
+// restore (stubbed)
+// ---------------------------------------------------------------------------
+
+fn run_restore(args: RestoreArgs) -> anyhow::Result<()> {
+ let cfg = AppConfig::load()?;
+ let from = args
+ .from
+ .as_ref()
+ .ok_or_else(|| anyhow::anyhow!("--from <path> is required"))?;
+ if !from.exists() {
+ anyhow::bail!("backup archive does not exist: {}", from.display());
+ }
+
+ let pg_restore_bin = args
+ .pg_restore
+ .clone()
+ .unwrap_or_else(|| PathBuf::from("pg_restore"));
+ let tar_bin = PathBuf::from("tar");
+ let target_db = cfg.migration_url().to_string();
+ let target_data_dir = args
+ .target_data_dir
+ .clone()
+ .unwrap_or_else(|| PathBuf::from(&cfg.data_dir));
+
+ // Stage: extract the tarball into a tempdir we own. The cleanup at the
+ // end of every path uses `let _ = remove_dir_all(&stage)` so a panic or
+ // early return doesn't leave a half-written data dir behind.
+ let stage = tempdir_in(std::env::current_dir()?.as_path())?;
+ let manifest_path = stage.join("manifest.json");
+
+ let manifest = match extract_and_verify(&tar_bin, from, &stage, &manifest_path) {
+ Ok(m) => m,
+ Err(e) => {
+ let _ = std::fs::remove_dir_all(&stage);
+ return Err(e);
+ }
+ };
+
+ // Manifest version policy: refuse forward, warn older. Forward refusal
+ // is fatal because restoring into an older server with a newer schema
+ // will silently drop tables or, worse, produce a working schema with
+ // missing columns. Older manifests are accepted with a warning because
+ // a newer server can usually still load them (the schema migrations are
+ // forward-only; older files just exercise older code paths).
+ if manifest.schema_version > CURRENT_SCHEMA_VERSION {
+ let _ = std::fs::remove_dir_all(&stage);
+ anyhow::bail!(
+ "manifest schema_version {} is newer than this binary's {}; refusing to restore an incompatible backup",
+ manifest.schema_version,
+ CURRENT_SCHEMA_VERSION
+ );
+ }
+ if manifest.schema_version < CURRENT_SCHEMA_VERSION {
+ eprintln!(
+ "[warn] manifest schema_version {} is older than this binary's {}; continuing",
+ manifest.schema_version, CURRENT_SCHEMA_VERSION
+ );
+ }
+
+ let plan = format!(
+ "Restore plan:\n \
+ source archive: {}\n \
+ manifest schema: {} (current {})\n \
+ manifest utopia build: {}\n \
+ manifest created: {}\n \
+ target database: {}\n \
+ target data dir: {}\n \
+ force: {}\n \
+ dry-run: {}\n",
+ from.display(),
+ manifest.schema_version,
+ CURRENT_SCHEMA_VERSION,
+ manifest.utopia_version,
+ manifest.created_at,
+ redact_url_host(&target_db),
+ target_data_dir.display(),
+ args.force,
+ args.dry_run,
+ );
+
+ if args.dry_run {
+ print!("{plan}");
+ let _ = std::fs::remove_dir_all(&stage);
+ // Verify pre-flight so a bad plan still exits non-zero.
+ preflight_restore(&pg_restore_bin, &target_data_dir, &manifest)?;
+ eprintln!("[dry-run] plan resolved cleanly; no changes made.");
+ return Ok(());
+ }
+
+ preflight_restore(&pg_restore_bin, &target_data_dir, &manifest)?;
+
+ if !args.yes {
+ eprint!(
+ "{plan}\nAbout to drop and recreate the target database, then restore from the archive. Continue? [y/N] "
+ );
+ let mut buf = String::new();
+ std::io::stdin().read_line(&mut buf)?;
+ if !matches!(buf.trim().to_ascii_lowercase().as_str(), "y" | "yes") {
+ let _ = std::fs::remove_dir_all(&stage);
+ anyhow::bail!("aborted by user");
+ }
+ }
+
+ if let Err(e) = apply_restore(
+ &pg_restore_bin,
+ &target_db,
+ &target_data_dir,
+ &stage,
+ &manifest,
+ ) {
+ let _ = std::fs::remove_dir_all(&stage);
+ return Err(e);
+ }
+
+ let _ = std::fs::remove_dir_all(&stage);
+ tracing::info!(path = %from.display(), "restore complete");
+ Ok(())
+}
+
+fn extract_and_verify(
+ tar_bin: &Path,
+ archive: &Path,
+ stage: &Path,
+ manifest_path: &Path,
+) -> anyhow::Result<Manifest> {
+ let status = Command::new(tar_bin)
+ .arg("-xzf")
+ .arg(archive)
+ .arg("-C")
+ .arg(stage)
+ .stdin(Stdio::null())
+ .status()?;
+ if !status.success() {
+ anyhow::bail!("tar exited with status {}; archive may be corrupt", status);
+ }
+ if !manifest_path.exists() {
+ anyhow::bail!("archive did not contain a manifest.json at the root — not a utopia backup?");
+ }
+ // Verify each checksummed component. pg_dump.custom is the only one
+ // today; data_dir is a directory tree so it's not hashed. Refusing on
+ // mismatch is loud — restoring a half-corrupted archive would be a
+ // silent failure mode nobody catches.
+ let manifest: Manifest = serde_json::from_slice(&std::fs::read(manifest_path)?)?;
+ for (name, expected) in &manifest.checksums {
+ let path = stage.join(name);
+ if !path.exists() {
+ anyhow::bail!("manifest references `{name}` but the archive did not contain it");
+ }
+ let Some(expected_sha) = expected.strip_prefix("sha256:") else {
+ anyhow::bail!("unsupported checksum scheme for `{name}`: {expected}");
+ };
+ let actual = sha256_file(&path)?;
+ if actual != expected_sha {
+ anyhow::bail!(
+ "checksum mismatch for `{name}`: manifest says {expected_sha}, file is {actual}"
+ );
+ }
+ }
+ Ok(manifest)
+}
+
+fn preflight_restore(
+ pg_restore_bin: &Path,
+ target_data_dir: &Path,
+ _manifest: &Manifest,
+) -> anyhow::Result<()> {
+ if !binary_works(pg_restore_bin) {
+ anyhow::bail!(
+ "pg_restore not found or not executable: {}. Install postgresql-client or pass --pg-restore <path>.",
+ pg_restore_bin.display()
+ );
+ }
+ // Data dir may not exist yet — restore creates it. Don't preflight its
+ // existence; that's not a precondition failure.
+ let _ = target_data_dir;
+ Ok(())
+}
+
+fn apply_restore(
+ pg_restore_bin: &Path,
+ target_db: &str,
+ target_data_dir: &Path,
+ stage: &Path,
+ manifest: &Manifest,
+) -> anyhow::Result<()> {
+ let pg_dump_path = stage.join(&manifest.components.pg_dump.path);
+ if !pg_dump_path.exists() {
+ anyhow::bail!(
+ "manifest references pg_dump at `{}` but the archive did not contain it",
+ manifest.components.pg_dump.path
+ );
+ }
+ tracing::info!(
+ archive_pg_dump = %pg_dump_path.display(),
+ "running pg_restore --clean --if-exists"
+ );
+ let status = Command::new(pg_restore_bin)
+ .arg("--clean")
+ .arg("--if-exists")
+ .arg("--dbname")
+ .arg(target_db)
+ .arg(&pg_dump_path)
+ .stdin(Stdio::null())
+ .status()?;
+ if !status.success() {
+ anyhow::bail!(
+ "pg_restore exited with status {}; database may be in a partial state — verify with a follow-up pg_dump",
+ status
+ );
+ }
+
+ if manifest.components.data_dir.present {
+ let archive_data_dir = stage.join(&manifest.components.data_dir.path);
+ if archive_data_dir.exists() {
+ tracing::info!(
+ target = %target_data_dir.display(),
+ "restoring data dir"
+ );
+ // Drop the existing target dir if it's non-empty so the copy
+ // below doesn't merge two worlds. This is what `--force`
+ // guards upstream — if `--force` is missing and the target
+ // already has files, fail loud.
+ if target_data_dir.exists() && std::fs::read_dir(target_data_dir)?.next().is_some() {
+ anyhow::bail!(
+ "target data dir {} is non-empty; pass --force to overwrite",
+ target_data_dir.display()
+ );
+ }
+ copy_dir_recursive(&archive_data_dir, target_data_dir)?;
+ } else {
+ tracing::warn!(
+ path = %archive_data_dir.display(),
+ "manifest says data_dir is present but the archive did not contain it; skipping"
+ );
+ }
+ }
+
+ Ok(())
+}
+
+#[cfg(test)]
+mod tests {
+ use super::*;
+
+ #[test]
+ fn parses_backup_minimal() {
+ let args = vec!["backup".to_string()];
+ let cmd = parse(&args).expect("parses");
+ match cmd {
+ Command2::Backup(b) => {
+ assert!(b.output.is_none());
+ assert!(!b.include_data_dir);
+ assert!(!b.dry_run);
+ }
+ _ => panic!("wrong variant"),
+ }
+ }
+
+ #[test]
+ fn parses_backup_full() {
+ let args = vec![
+ "backup".to_string(),
+ "--output".to_string(),
+ "/tmp/b.tar.gz".to_string(),
+ "--include-data-dir".to_string(),
+ "--dry-run".to_string(),
+ "--pg-dump".to_string(),
+ "/opt/pg/bin/pg_dump".to_string(),
+ "--tar".to_string(),
+ "/bin/tar".to_string(),
+ "--migration-url".to_string(),
+ "postgres://u:***@h/db".to_string(),
+ ];
+ let cmd = parse(&args).expect("parses");
+ match cmd {
+ Command2::Backup(b) => {
+ assert_eq!(b.output, Some(PathBuf::from("/tmp/b.tar.gz")));
+ assert!(b.include_data_dir);
+ assert!(b.dry_run);
+ assert_eq!(b.pg_dump, Some(PathBuf::from("/opt/pg/bin/pg_dump")));
+ assert_eq!(b.tar, Some(PathBuf::from("/bin/tar")));
+ assert_eq!(b.migration_url.as_deref(), Some("postgres://u:***@h/db"));
+ }
+ _ => panic!("wrong variant"),
+ }
+ }
+
+ #[test]
+ fn parses_restore_requires_from() {
+ let args = vec!["restore".to_string()];
+ assert!(parse(&args).is_err());
+ }
+
+ #[test]
+ fn parses_restore_full() {
+ let args = vec![
+ "restore".to_string(),
+ "--from".to_string(),
+ "/tmp/b.tar.gz".to_string(),
+ "--target-data-dir".to_string(),
+ "/var/lib/utopia/data".to_string(),
+ "--force".to_string(),
+ "--yes".to_string(),
+ "--dry-run".to_string(),
+ ];
+ let cmd = parse(&args).expect("parses");
+ match cmd {
+ Command2::Restore(r) => {
+ assert_eq!(r.from, Some(PathBuf::from("/tmp/b.tar.gz")));
+ assert_eq!(
+ r.target_data_dir,
+ Some(PathBuf::from("/var/lib/utopia/data"))
+ );
+ assert!(r.force);
+ assert!(r.yes);
+ assert!(r.dry_run);
+ }
+ _ => panic!("wrong variant"),
+ }
+ }
+
+ #[test]
+ fn rejects_unknown_subcommand() {
+ let args = vec!["frobnicate".to_string()];
+ assert!(parse(&args).is_err());
+ }
+
+ #[test]
+ fn redact_url_host_keeps_userinfo_at_host() {
+ let r = redact_url_host("postgres://utopia:secret@db:5432/utopia");
+ assert_eq!(r, "postgres://***@db:5432/utopia");
+ }
+
+ #[test]
+ fn redact_url_host_handles_no_at() {
+ let r = redact_url_host("not a url");
+ assert_eq!(r, "<no host>");
+ }
+
+ #[test]
+ fn hex_encode_known_value() {
+ // SHA256 of "abc"
+ assert_eq!(
+ hex_encode(&[
+ 0xba, 0x78, 0x16, 0xbf, 0x8f, 0x01, 0xcf, 0xea, 0x41, 0x41, 0x40, 0xde, 0x5d, 0xae,
+ 0x22, 0x23, 0xb0, 0x03, 0x61, 0xa3, 0x96, 0x17, 0x7a, 0x9c, 0xb4, 0x10, 0xff, 0x61,
+ 0xf2, 0x00, 0x15, 0xad,
+ ]),
+ "ba7816bf8f01cfea414140de5dae2223b00361a396177a9cb410ff61f20015ad"
+ );
+ }
+
+ /// Manifest round-trips: serialize → deserialize produces the same shape
+ /// on the read side. Today `backup` writes with `Serialize`, `restore`
+ /// reads with `Deserialize` — if either side drops or renames a field,
+ /// this is the test that catches it.
+ #[test]
+ fn manifest_round_trip() {
+ let mut checksums = HashMap::new();
+ checksums.insert("pg_dump.custom".to_string(), "sha256:deadbeef".to_string());
+ let written = Manifest {
+ schema_version: CURRENT_SCHEMA_VERSION,
+ utopia_version: "0.1.0".to_string(),
+ created_at: "2026-09-17T09:00:00Z".to_string(),
+ components: ManifestComponents {
+ pg_dump: ManifestComponent {
+ path: "pg_dump.custom".to_string(),
+ format: "pg_dump -Fc".to_string(),
+ bytes: 12345,
+ },
+ data_dir: ManifestDataDir {
+ path: "data".to_string(),
+ present: true,
+ },
+ },
+ checksums,
+ };
+ let json = serde_json::to_string(&written).unwrap();
+ let read: Manifest = serde_json::from_str(&json).unwrap();
+ assert_eq!(read.schema_version, CURRENT_SCHEMA_VERSION);
+ assert_eq!(read.utopia_version, "0.1.0");
+ assert_eq!(read.components.pg_dump.bytes, 12345);
+ assert!(read.components.data_dir.present);
+ assert_eq!(
+ read.checksums.get("pg_dump.custom").map(String::as_str),
+ Some("sha256:deadbeef")
+ );
+ }
+
+ /// Forward-incompatible manifest: schema_version > CURRENT. Restore must
+ /// bail before touching the live database. The actual gate is inside
+ /// `run_restore`; this test pins the version-policy behaviour at the
+ /// constant level so a future bump that changes the comparison shape
+ /// (e.g. switching to a rangeset) gets caught at compile-test time, not
+ /// in production.
+ #[test]
+ fn schema_version_policy_compares_against_current() {
+ // Same shape as `run_restore`'s gate. If the rule changes here,
+ // change it there too.
+ let current = CURRENT_SCHEMA_VERSION;
+ assert!(
+ current + 1 > current,
+ "newer manifests must compare > current"
+ );
+ assert!(
+ current - 1 < current,
+ "older manifests must compare < current"
+ );
+ // And the constant itself must stay in lockstep with the highest
+ // applied migration in `migrations/` — bumping a migration file
+ // without bumping this is a silent schema-version skew. The
+ // `migrations/` dir is a sibling of `crates/utopia-cli/`; resolve
+ // from CARGO_MANIFEST_DIR so the test works regardless of cwd.
+ let manifest_dir = std::path::Path::new(env!("CARGO_MANIFEST_DIR"));
+ let migrations_dir = manifest_dir
+ .parent()
+ .and_then(|p| p.parent())
+ .map(|p| p.join("migrations"))
+ .expect("workspace root has a migrations/ dir");
+ let applied = std::fs::read_dir(&migrations_dir)
+ .unwrap_or_else(|e| {
+ panic!(
+ "migrations dir {} unreadable: {e}",
+ migrations_dir.display()
+ )
+ })
+ .filter_map(|e| e.ok())
+ .filter(|e| e.path().extension().map(|x| x == "sql").unwrap_or(false))
+ .count() as u32;
+ assert_eq!(
+ current, applied,
+ "CURRENT_SCHEMA_VERSION must match the count of applied migrations; \
+ bump it when you add a migration"
+ );
+ }
+
+ /// `extract_and_verify` against an archive without a manifest.json:
+ /// should fail loudly with "not a utopia backup?" rather than running
+ /// pg_restore against an empty stage. Build a fake tarball in a temp
+ /// dir so the path is exercised end-to-end.
+ #[test]
+ fn extract_fails_when_archive_has_no_manifest() {
+ let stage = tempdir_for_test("no_manifest");
+ // Stage must already exist for `tar -C`. Create a fake archive
+ // containing exactly one file inside a fresh staging dir, then run
+ // extract_and_verify against a different stage.
+ let src = tempdir_for_test("src");
+ std::fs::write(src.join("unrelated.txt"), b"not a backup").unwrap();
+ let archive = src.join("fake.tar.gz");
+ let status = Command::new("tar")
+ .arg("-czf")
+ .arg(&archive)
+ .arg("-C")
+ .arg(&src)
+ .arg("unrelated.txt")
+ .stdin(Stdio::null())
+ .status()
+ .expect("tar available");
+ assert!(status.success(), "host tar is required for this test");
+ let manifest_path = stage.join("manifest.json");
+ let result = extract_and_verify(Path::new("tar"), &archive, &stage, &manifest_path);
+ let _ = std::fs::remove_dir_all(&stage);
+ let _ = std::fs::remove_dir_all(&src);
+ let err = result.unwrap_err().to_string();
+ assert!(
+ err.contains("not a utopia backup"),
+ "want a clear error, got: {err}"
+ );
+ }
+
+ /// Checksum mismatch: stage contains a `pg_dump.custom` whose bytes
+ /// don't match the manifest's sha256. `extract_and_verify` must refuse
+ /// rather than pass a corrupted archive on to pg_restore.
+ #[test]
+ fn extract_fails_on_checksum_mismatch() {
+ let src = tempdir_for_test("mismatch_src");
+ // Real pg_dump bytes we can corrupt at will.
+ std::fs::write(src.join("pg_dump.custom"), b"original-bytes").unwrap();
+ let archive = src.join("mismatch.tar.gz");
+ let status = Command::new("tar")
+ .arg("-czf")
+ .arg(&archive)
+ .arg("-C")
+ .arg(&src)
+ .arg("pg_dump.custom")
+ .stdin(Stdio::null())
+ .status()
+ .expect("tar available");
+ assert!(status.success());
+ let manifest_json = serde_json::json!({
+ "schema_version": CURRENT_SCHEMA_VERSION,
+ "utopia_version": "0.1.0",
+ "created_at": "2026-09-17T00:00:00Z",
+ "components": {
+ "pg_dump": {
+ "path": "pg_dump.custom",
+ "format": "pg_dump -Fc",
+ "bytes": 14
+ },
+ "data_dir": { "path": "data", "present": false }
+ },
+ // SHA256 of "tampered" — neither the file nor the empty archive
+ // matches this, so verification must fail.
+ "checksums": { "pg_dump.custom": "sha256:0000000000000000000000000000000000000000000000000000000000000000" }
+ })
+ .to_string();
+ std::fs::write(src.join("manifest.json"), manifest_json).unwrap();
+ // Re-pack with both files so the archive contains manifest + dump.
+ let archive2 = src.join("mismatch2.tar.gz");
+ let status = Command::new("tar")
+ .arg("-czf")
+ .arg(&archive2)
+ .arg("-C")
+ .arg(&src)
+ .arg("manifest.json")
+ .arg("pg_dump.custom")
+ .stdin(Stdio::null())
+ .status()
+ .unwrap();
+ assert!(status.success());
+
+ let stage = tempdir_for_test("mismatch_stage");
+ let manifest_path = stage.join("manifest.json");
+ let result = extract_and_verify(Path::new("tar"), &archive2, &stage, &manifest_path);
+ let _ = std::fs::remove_dir_all(&stage);
+ let _ = std::fs::remove_dir_all(&src);
+ let err = result.unwrap_err().to_string();
+ assert!(
+ err.contains("checksum mismatch"),
+ "want a checksum error, got: {err}"
+ );
+ }
+
+ /// Create a uniquely-named temp directory under the process's CWD. Test
+ /// cleanup is the caller's responsibility — keeps the helper
+ /// dependency-free (no tempfile crate). Mirrors the shape of
+ /// `tempdir_in` used by the production paths.
+ fn tempdir_for_test(label: &str) -> PathBuf {
+ let name = format!(
+ ".utopia-cli-test-{label}-{}",
+ std::time::SystemTime::now()
+ .duration_since(std::time::UNIX_EPOCH)
+ .map(|d| d.as_nanos())
+ .unwrap_or(0)
+ );
+ let p = std::env::current_dir().unwrap().join(name);
+ std::fs::create_dir_all(&p).unwrap();
+ p
+ }
+}