Skip to content

Self-evolving Context Database for AI Agents. Unify Agent Memory, Knowledge RAG and Skills.

15.8 KB
Raw Blame

Encryption Guide

This guide describes how to enable and use at-rest data encryption in OpenViking.

Overview

OpenViking can encrypt newly written files with account-specific keys. The storage layer handles encryption and decryption through the existing APIs:

  • Transparent encryption: No API changes, application layer unaware
  • Multi-tenant isolation: Different accounts use independent keys
  • Three key providers: Local, Vault, Volcengine KMS
  • Backward compatible: Unencrypted old files still readable

See Data Encryption for conceptual explanations.

Encryption in Multi-Write Storage

Multi-write storage reuses the same transparent encryption model. Encryption still happens inside RAGFS, so the Python SDK, HTTP API, and CLI do not need to handle encryption or decryption directly.

Rules:

  • When global encryption.enabled=true, the primary backend must be encrypted.
  • Each backup backend may control its own encryption through encryption.enabled.
  • V2 multi-write metadata is stored on the raw primary and bypasses EncryptionWrappedFS.
  • OpenViking does not expose and does not need public encryption APIs for operating on these internal files.

See the Multi-Write Storage Guide for more multi-write configuration details.

Quick Start

1. Initialize Root Key (Local Mode)

ov system crypto init-key --output-file ~/.openviking/master.key

2. Configure Encryption

Merge the following encryption settings into ~/.openviking/ov.conf; retain your model and server settings:

{
  "encryption": {
    "enabled": true,
    "provider": "local",
    "local": {
      "key_file": "~/.openviking/master.key"
    }
  },
  "storage": {
    "workspace": "./data"
  }
}

3. Verify

Restart the server after changing encryption settings, and run this example against that server. Install the Python SDK in the environment running the script.

from pathlib import Path
from openviking_sdk import SyncHTTPClient


def test():
    # OPENVIKING_API_KEY: use a tenant-bound user/admin key when authentication is enabled.
    client = SyncHTTPClient(url="http://localhost:1933")
    try:
        client.initialize()
        sample = Path("./encrypted-sample.txt")
        sample.write_text("Hello, encrypted world!", encoding="utf-8")
        imported = client.add_resource(
            path=str(sample),
            wait=True,
            timeout=120,
        )
        results = client.find(
            query="encrypted", target_uri=imported["root_uri"]
        )
        print(f"Found {len(results.get('resources', []))} resources")
    finally:
        client.close()


test()

It waits for import processing and checks retrieval; successful retrieval alone does not prove encryption at rest. Use Check File Content to inspect the stored file header.

API Key Hashing Configuration

File encryption and API key hashing are independent controls:

Encryption Layer Config Algorithm Reversible Description
File Layer encryption.enabled AES-GCM Yes Protects entire storage files
API Key Field Layer encryption.api_key_hashing.enabled Argon2id No Protects API keys themselves

Default Behavior

By default, encryption.api_key_hashing.enabled = false:

  • API keys are stored in plaintext within JSON files
  • If encryption.enabled = true, the entire file is protected by AES-GCM encryption
  • ov admin list-users can display the full API key when run with --reveal-api-keys
  • If encryption.enabled = true and API key hashing stays disabled, the server writes an informational log line about this at startup

Enabling Argon2id Hashing

Enable Argon2id one-way hashing to stop storing recoverable API key values:

{
  "encryption": {
    "enabled": true,
    "api_key_hashing": {
      "enabled": true
    }
  }
}

Note: When enabled:

  • API keys are stored using Argon2id one-way hashing
  • Plaintext keys cannot be recovered from hash values
  • ov admin list-users only shows key_prefix instead of the full API key (a caller's own row cannot be revealed either, since no plaintext is stored)
  • Plaintext keys are only visible when creating users or regenerating keys

Configuration Example

{
  "encryption": {
    "enabled": true,
    "provider": "local",
    "local": {
      "key_file": "~/.openviking/master.key"
    },
    "api_key_hashing": {
      "enabled": false
    }
  }
}

Choosing a Key Provider

Provider Use Case Pros Cons
Local Dev environments, single-node Simple, no external services Requires local secret permissions and separate key backups
Vault Production, multi-cloud Enterprise-grade KMS, version control Requires deploying and maintaining Vault
Volcengine KMS Volcengine cloud Cloud-native KMS service Volcengine-only

Local Mode Detailed Guide

Initialize Root Key

# Generate and save to specified path
ov system crypto init-key --output-file ~/.openviking/master.key

# Or use short option
ov system crypto init-key -f ~/.openviking/master.key

Output example:

✓ Root key generated successfully
✓ Saved to: /Users/you/.openviking/master.key

Security Tips

  • ⚠️ Keep master.key safe
  • Recommend setting file permissions to 600 (owner-only read/write)
  • Regularly back up the key file
  • Don't commit the key file to version control

Configuration Example

{
  "encryption": {
    "enabled": true,
    "provider": "local",
    "local": {
      "key_file": "~/.openviking/master.key"
    }
  }
}

Inline Root Key

For simple, controlled deployments, the Local provider can read the same 32-byte root key directly from ov.conf instead of a separate file:

{
  "encryption": {
    "enabled": true,
    "provider": "local",
    "local": {
      "root_key": "<64-character hexadecimal root key>"
    }
  }
}

The value is 32 bytes encoded as exactly 64 hexadecimal characters. You can generate one with openssl rand -hex 32, or reuse the contents of an existing master.key when moving between the two Local configurations.

  • Do not configure root_key and key_file together. If both are omitted, the Local provider continues to use ~/.openviking/master.key.
  • An inline key is never generated automatically. Losing or changing it makes existing encrypted data unreadable.
  • Treat ov.conf as a key file: restrict it to the service user, back it up securely, and never commit it to version control.
  • Prefer Vault or KMS when production deployments need centralized secret management.

Vault Mode Detailed Guide

Prerequisites

  1. HashiCorp Vault service deployed; install the Python hvac dependency in the server environment
  2. Transit engine enabled
  3. Vault Token with sufficient permissions

Configure Vault

  1. Enable Transit engine (if not already enabled):
vault secrets enable transit
  1. Enable KV engine (if not already enabled):
# KV v2 (recommended)
vault secrets enable -path=secret -version=2 kv

# Or KV v1
vault secrets enable -path=secret -version=1 kv

Use one KV command matching your deployment, not both. The example below uses the secret mount with KV v2. Pre-create the Transit key with an administrative token:

vault write -f transit/keys/openviking-root-key type=aes256-gcm96
  1. Configure OpenViking:
{
  "encryption": {
    "enabled": true,
    "provider": "vault",
    "vault": {
      "address": "https://vault.example.com:8200",
      "token": "hvs.xxxxxxxxxxxxxxxxxxxxx",
      "mount_point": "transit",
      "kv_mount_point": "secret",
      "kv_version": 2,
      "root_key_name": "openviking-root-key",
      "encrypted_root_key_key": "openviking-encrypted-root-key"
    }
  }
}

Configuration Parameters:

Parameter Description Default
address Vault server address Required
token Vault authentication token Required
mount_point Transit engine mount path "transit"
kv_mount_point KV engine mount path "secret"
kv_version KV engine version (1 or 2) 1
root_key_name Key name in Transit engine "openviking-root-key"
encrypted_root_key_key Path to store encrypted root key in KV engine "openviking-encrypted-root-key"

Vault Permission Recommendations

For the configuration above, the service token needs Transit key metadata access, encrypt/decrypt access, and KV access to store the wrapped root key. With the engines and Transit key pre-created, a policy example is:

path "auth/token/lookup-self" {
  capabilities = ["read"]
}

path "transit/keys/openviking-root-key" {
  capabilities = ["read"]
}

path "secret/data/openviking-encrypted-root-key" {
  capabilities = ["read", "create", "update"]
}

path "transit/encrypt/openviking-root-key" {
  capabilities = ["update"]
}

path "transit/decrypt/openviking-root-key" {
  capabilities = ["update"]
}

For KV v1, replace secret/data/openviking-encrypted-root-key with secret/openviking-encrypted-root-key. The provider also probes sys/mounts; denial is logged as a warning and does not stop startup when Transit is already enabled.


Volcengine KMS Mode Detailed Guide

Prerequisites

  1. Volcengine KMS service activated
  2. Symmetric key created
  3. Valid Access Key and Secret Key

Create KMS Key

  1. Visit Volcengine KMS Console
  2. Click "Create Key"
  3. Select "Symmetric Key", algorithm AES_256
  4. Record the Key ID

Configure OpenViking

{
  "encryption": {
    "enabled": true,
    "provider": "volcengine_kms",
    "volcengine_kms": {
      "key_id": "d926aa0d-xxxx-xxxx-xxxx-xxxxxxxxxxxx",
      "region": "cn-beijing",
      "access_key": "AKLTxxxxxxxxxxxxxxxxxx",
      "secret_key": "Tmpxxxxxxxxxxxxxxxxxxxxxx",
      "endpoint": null,
      "key_file": "~/.openviking/openviking-volcengine-root-key.enc"
    }
  }
}

Configuration Parameters:

Parameter Description Default
key_id KMS Key ID Required
region Region Required
access_key Access Key Required
secret_key Secret Key Required
endpoint Custom KMS endpoint (optional) null (use default endpoint)
key_file Local cache file path for encrypted root key "~/.openviking/openviking-volcengine-root-key.enc"

Permission Recommendations

Configure minimal permissions for the Access Key:

{
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "kms:Encrypt",
        "kms:Decrypt"
      ],
      "Resource": [
        "trn:kms:*:*:key/d926aa0d-xxxx-xxxx-xxxx-xxxxxxxxxxxx"
      ]
    }
  ]
}

Verifying Encryption

Method 1: Check File Content

Inspect the physical backend file, not the API response, which is decrypted. Substitute its actual path below. Encrypted files start with magic number OVE1:

# View first 4 bytes
hexdump -C ./data/agfs/your-file | head -1

Encrypted file:

00000000  4f 56 45 31 01 01 00 00  00 20 8a 7b 2c 9d 1e  |OVE1..... .{,..|

(First 4 bytes are 4f 56 45 31 = "OVE1")

Unencrypted file:

00000000  7b 22 63 6f 6e 74 65 6e  74 73 22 3a 5b 7b 22 70  |{"contents":[{"p|

Method 2: Restart and Read Back

Read the same imported file through the API before and after restarting the server. Verify its content matches and inspect the physical backend file header as above. This checks that the deployment reloads the expected key and can read its ciphertext.

Do not treat an arbitrary exception in an internal provider call as an encryption test: argument errors, missing dependencies, and network errors can all fail before decryption.


Migration Guide

Migrating from Unencrypted to Encrypted

Enabling encryption does not rewrite existing plaintext files. They remain readable for backward compatibility, while new writes use encryption. To encrypt existing public scopes, migrate them through OVPack into a new, empty encrypted storage environment:

  1. Stop application writes and create a logical backup while the original unencrypted environment is running:
ov backup ./backups/before-encryption.ovpack
  1. Stop OpenViking. Enable encryption and point the storage configuration at a new, empty workspace/backend. Keep the original data and encryption key backup until verification is complete.
  2. Start the encrypted environment. In API key mode, first create the target account and a restore operator with an admin key, then point the CLI at the target using that key, as described in Full Backup and Restore. Restore writes the package content through the encrypted storage layer:

Account initialization creates scope directories, so fail would reject this restore. Confirm that the target contains only the newly created account's initial content before using overwrite below. If it already contains business data, stop and follow the target-backup and conflict-review steps in the OVPack guide instead.

ov restore ./backups/before-encryption.ovpack --on-conflict overwrite
  1. Verify resource, user, session, and index data before switching traffic. OVPack excludes runtime/internal state such as queues, uploads, locks, and watches; recreate or validate those separately.

See OVPack Import and Export for supported scopes and restore options.

Switching Key Providers

Keep the original provider and key available while exporting a logical OVPack backup. Restore it into a separate empty deployment configured with the new provider, following the migration steps above. Verify contents, access permissions, and rebuilt indexes before switching traffic. Changing the provider setting in place does not re-encrypt existing files.


Troubleshooting

Key File Not Found

Error: Key file not found: ~/.openviking/master.key

Solution:

  1. Check file path is correct
  2. Use absolute path
  3. Ensure ~ is properly expanded (use expanduser())

Vault Connection Failed

Error: Failed to connect to Vault

Solution:

  1. Check if Vault service is running
  2. Verify address configuration
  3. Check network connectivity and firewall
  4. Confirm Token is valid and not expired

Volcengine KMS Authentication Failed

Error: Invalid credentials

Solution:

  1. Check Access Key and Secret Key are correct
  2. Confirm key has sufficient permissions
  3. Verify region configuration is correct

Existing ciphertext cannot be read

Check that the deployment still has the original root key, provider settings, account identity, and intact ciphertext. For Vault, retain both the Transit key and the KV entry containing the wrapped root key; for KMS, retain the KMS key and local wrapped-key file. Do not generate a replacement root key to repair access to existing ciphertext.

Partial reads load and authenticate the full encrypted file, then return the requested plaintext slice. If a partial read returns ciphertext, record the server version and compare full and partial reads on a copied test file. Check the storage encryption configuration and relevant fixes before upgrading, and keep the original data and keys available for rollback.


Related Documentation

Vault references: KV v2, Transit API.