Skip to content

~95% on SimpleQA (e.g. Qwen3.6-27B on a 3090). Supports all local and cloud LLMs (llama.cpp, Ollama, Google, ...). 10+ search engines - arXiv, PubMed, your private documents. Everything Local & Encrypted.

15.2 KB
Raw Blame

Database Backup System

Local Deep Research includes an automatic database backup system that creates encrypted backups of your user database after each successful login.

Overview

  • Automatic: Backups run in the background after login without blocking the UI
  • Encrypted: Backups use the same encryption as your main database
  • WAL-aware: Uses SQLCipher's sqlcipher_export() to copy committed data, including data still in the source WAL
  • Salt preservation: New backups include the original matching .salt file when the source uses a per-database salt
  • Configurable: Enable/disable and configure retention via settings
  • Pre-migration: A backup is automatically attempted before any database schema migration

How It Works

  1. When you log in successfully, a background backup is scheduled
  2. Only one verified backup per calendar day is normally created — subsequent logins the same day reuse it. The verification in step 4 runs when a backup is created. The application then remembers each verification result, passing or failing, for as long as the backup and its salt keep the same file identity, size and modification time; an application restart or a password change forgets them. A later login reuses a remembered pass without verifying any backup. Only when none of the day's backups has one does it verify the backups that have no remembered result, newest first, stopping at the first that passes. A backup whose database or salt file is removed or changed during that check does not count. An older backup missing its required salt does not prevent creation of a complete replacement, and when the database has its own salt, such a backup is rejected without deriving a key
  3. The backup runs in a separate thread (non-blocking)
  4. Uses sqlcipher_export() to create an encrypted copy, preserves the matching salt, and verifies the copy using the backup's own salt. A copy is removed, and no older backup is deleted, when its database or salt file is removed or changed while it is being verified, or when, by the end of its publication, either file has gone missing or differs in size from the verified one, or a salt file has appeared beside a copy verified without one. A change made after the verification that keeps each file's size is not detected at publication
  5. Old backups are automatically cleaned up based on your retention settings
  6. Before database migrations, a backup is always attempted regardless of the daily limit. If it fails, the log shows Pre-migration backup failed and the migration runs anyway. A pre-migration backup that was made is not kept: with the default Max Backups of 1, the login backup on a later UTC day replaces it with a copy of the migrated database, and a password change deletes it. Copy it out of the backup directory if you may need to roll back; see the upgrade and rollback guide

Backup Location

Backups are stored in:

{data_directory}/encrypted_databases/backups/{user_hash}/

Where {user_hash} is the first 16 hex characters of the SHA-256 hash of your username.

Each new backup has a timestamp and a unique suffix, so forced backups in the same second cannot overwrite one another:

ldr_backup_20250125_143022_ab12cd34.db
ldr_backup_20250125_143022_ab12cd34.db.salt

Preserve both files together. The salt is required to derive the database key from the password. It is not a replacement for the password, and generating a new salt will not recover an existing database. Retention and password-change cleanup remove each backup together with its salt.

Legacy databases that use LDR's built-in shared salt have no .salt sidecar. New backups of those databases remain readable through the legacy derivation path. Do not assume a missing salt means a backup is legacy: verify it with the matching application version and cipher configuration before relying on it.

Settings

The backup status panel shows the latest failed attempt and a fixed error description, including failures before a migration or after a password change. A successful new backup clears an ordinary backup failure; reusing a daily backup does not erase it. An incomplete password-change purge remains visible after ordinary successful backups. Only another purge that verifies all old backup files were removed can clear that warning. Listing failures, denied deletions and a directory becoming unreadable during cleanup are failures to confirm the purge, even when a fresh current-password backup succeeds. A password change whose purge is skipped (the rekey lock stayed held, a later change took over, the change could not be fenced, or the backup service resolved another database) or fails before it starts (a rekey-lock or epoch fault, or the backup directory cannot be set up) is recorded the same way, because nothing was removed. Only a purge run by a later password change can clear the warning: the recovery purge at login after an interrupted change does not report status, because it keeps backups it cannot classify.

Attempt timestamps, outcomes and fixed error codes are account activity. They are stored only in the encrypted user database, never in the plaintext central account database. Status survives restart and is shared across workers once it has been written to that encrypted store. Status reporting writes through the user's already-open database connection and never reopens a logged-out database with a cached password. The purge after a password change is handled separately: it runs under the rekey lock, which the open-connection path would wait on, so it records its result over a connection opened with the new password the user just entered, the same way the fresh backup itself is exported. If encryption, the connection, or status storage is unavailable, the serving worker keeps only a bounded memory observation, explicitly reports unavailable persistence, and writes that observation to the encrypted store at the next opportunity (the next backup attempt, the next status read, or the daily backup check at login). It is lost if the worker restarts first.

Workers merge their observations. The newest attempt wins. The password-change purge state is changed only by a purge observation, and a newer one wins, except that on a timestamp tie or clock step-back "incomplete" wins. A stored purge state is never resent, so an ordinary backup never clears a warning and never revives one from a stale copy. A purge observation that a worker could not write yet travels with that worker's next write; if it is "incomplete" it can then override a newer clean purge from another worker. That is deliberate: a spurious warning is cleared by the next verified purge, while a spurious all-clear would hide old-password backups. Passwords, backup paths and exception text are not part of the status record.

The panel distinguishes a backup directory the server cannot list from an empty one, as far as the listing call reports the error, and an unreadable backup setting from a disabled setting. Daily backup reuse and ordinary retention still use Path.glob, which can suppress scan errors, but they cannot clear an incomplete-purge warning. To remove backups made with the previous password, restore directory access and change the password again, which repeats the purge. The purge removes only regular files: a symlink or a directory with a backup name is left in place and is not counted as a remaining backup. An entry it cannot inspect (for example in a directory that can be listed but not searched) is counted, and an unexpected error during cleanup is recorded as an incomplete purge. Reporting failure never changes the actual result of creating a fresh backup.

Configure backup behavior in Settings > Backup:

Setting Default Description
Enable Auto-Backup true Enable/disable automatic backups on login. Disable if disk space is limited.
Max Backups 1 Maximum number of backup files to keep (1-30)
Backup Retention (days) 7 Delete backups older than this many days

Note: Each backup is a full encrypted copy of your database and cannot be compressed. With the default of 1 backup, disk usage equals your database size. Users with large databases (e.g., containing uploaded PDFs) should monitor disk usage and can reduce the backup count or disable backups entirely via the Enable Auto-Backup setting if disk space is limited.

Why sqlcipher_export()?

We use SQLCipher's sqlcipher_export() instead of VACUUM INTO or simple file copy because:

  1. Encryption Preservation: VACUUM INTO does not preserve SQLCipher encryption settings. sqlcipher_export() correctly copies data while maintaining the same encryption key and cipher configuration
  2. WAL Safety: Regular file copy can corrupt databases using WAL (Write-Ahead Logging) mode
  3. Publication: The export uses a unique temporary database. After verification, its salt is renamed first and the database is renamed last. A process interruption between those steps leaves an orphan salt, rather than a listed database with a missing salt. The two files are not a single atomic filesystem transaction; copy completed pairs together to external storage
  4. Integrity Verification: Each backup is verified with PRAGMA quick_check and its own recovery inputs before being finalized

Backup scope

These are per-user database snapshots, not full instance backups. They do not include the account registry (ldr_auth.db), deployment configuration, or files stored outside the user database, such as filesystem-backed library content. Preserve those separately when backing up an entire deployment. A cleanly stopped copy of the full data directory and deployment configuration is the appropriate starting point for an instance recovery rehearsal.

Keep the application version and any custom LDR_DB_CONFIG_* cipher settings with the backup inventory. The database and salt do not encode all settings used for key derivation. Use the same configuration to verify and recover a backup; changing it can prevent a correct password from opening the database.

Offline recovery rehearsal

The procedure below recovers a user database into a new, empty directory. It does not replace a live database or recreate a missing account registry. An in-application restore with coordinated session shutdown is tracked in #5327.

  1. Copy the completed backup and its matching .salt to storage outside the original data directory. For backups made before salt preservation was added, preserve the original database's matching salt as well. This change cannot reconstruct a salt already lost from an older backup.

  2. Use the matching LDR version and cipher configuration in an isolated environment. Keep it stopped; do not point a running instance at the recovery directory.

  3. Create an empty recovery directory. Copy the backup database there using the original user's filename, ldr_user_<user_hash>.db, and copy its salt as ldr_user_<user_hash>.db.salt. The user hash is the first 16 hexadecimal characters of SHA-256 of the exact username. Restrict the directory to the operator and the recovered files to the application account.

  4. Verify the recovered copy without access to the original data directory. Run the following with the matching LDR installation. It prompts for the password instead of putting it in shell history:

    from getpass import getpass
    from pathlib import Path
    from local_deep_research.database.sqlcipher_utils import create_sqlcipher_connection
    
    recovered = Path(input("Recovered database path: ")).expanduser()
    if not recovered.is_file() or recovered.is_symlink():
        raise ValueError("Select the recovered database file")
    connection = create_sqlcipher_connection(recovered, getpass("Backup password: "))
    try:
        result = connection.execute("PRAGMA quick_check").fetchall()
        if result != [("ok",)]:
            raise ValueError("Recovered database failed its integrity check")
        tables = connection.execute(
            "SELECT name FROM sqlite_master WHERE type='table' ORDER BY name"
        ).fetchall()
        print("Integrity check passed. Recovered tables:", tables)
    finally:
        connection.close()
    
  5. In a separate instance recovery rehearsal, restore the corresponding account registry, configuration, and required external files, then check login and representative research/library data through the UI. A database integrity check alone does not establish that the whole instance has been recovered.

Before any eventual replacement of a production database, stop every process using it and preserve the current database, salt, and any WAL/SHM files as a rollback set. Do not copy a backup over a live database or discard a WAL that may contain uncheckpointed writes. Keep recovery and rollback sets separate; never combine an old WAL with a recovered database.

Important: The backup uses the same password as when it was created. If you've changed your password since the backup, you'll need to use the old password to access it.

The automated recovery tests exercise real SQLCipher exports with committed WAL data, remove the original data directory, recover using the archived database/salt pair, check the recovered data, and verify that new writes persist. They also cover legacy databases, a wrong password, missing or damaged backup salts, failed publication, publishing the salt before the database, retention of matching file pairs, and reusing a verified backup on later logins until its database or salt file is replaced or its modification time changes. They check that later logins derive no key when a failing backup is kept beside a verified one, that a backup without a salt file, of a database that has its own salt, is rejected before any key derivation, and that a backup or export whose database or salt file is removed during its check, or an export whose salt is removed before or during publication, is neither reused nor published. They do not exercise live in-application replacement or full account-registry recovery.

Troubleshooting

Backups not being created

  1. Check if backups are enabled in Settings
  2. Check the logs for backup-related errors
  3. Verify sufficient disk space (requires 2x database size)

Disk space issues

The system checks for available disk space before creating a backup. If you see "Insufficient disk space" errors:

  1. Free up disk space
  2. Reduce the max backup count setting
  3. Reduce the retention days setting

Backup verification failed

If you see "Backup verification failed" in logs, the backup may be corrupted. This can happen if:

  1. The disk ran out of space during backup
  2. There was a system crash during backup
  3. The source database is corrupted
  4. The backup's .salt file is missing or damaged (backups made before salts were kept alongside them have none)

A new backup that fails verification is deleted automatically. An existing backup that fails the check at a later login no longer counts as that day's backup and is kept until retention removes it. The failure is remembered, so that backup is not checked again until the application restarts, the password changes, or its database or salt file changes.