Skip to content

Database governance built for humans and agents — controlling changes and access across every major database.

7.3 KB
Raw Blame

High availability for Bytebase

This runbook describes the current operator-facing behavior for running Bytebase in a high-availability (HA) topology.

Current support boundary

  • Bytebase can detect and operate with multiple active replicas that share the same metadata database.
  • HA is license-gated. The subscription API exposes this as the ha field on Subscription.
  • The bundled Helm chart in helm-charts/bytebase currently renders a single-replica StatefulSet (replicas: 1). It does not provide a multi-replica deployment switch, though bytebase.option.ha starts the replica it manages with --ha.

In other words: the runtime has HA awareness, but operators must still supply the multi-replica deployment mechanics outside the current Helm chart.

Prerequisites

Before you keep more than one Bytebase server active at the same time, make sure all of the following are true:

  1. HA is enabled in the license.
    • Check GET /v1/subscription and confirm ha: true.
  2. Every HA replica starts with --ha and the same shared metadata database.
    • Active replicas are counted as Bytebase servers sharing the same database.
    • HA mode requires external PostgreSQL, so each replica must use the same PG_URL value instead of the embedded database.
    • Under the bundled Helm chart, set bytebase.option.ha=true to pass the flag to the replica it manages.
  3. All replicas use the same external URL.
    • Start replicas with the same --external-url value so Bytebase reports one externalUrl for user access and callbacks.
  4. Your platform provides traffic management and rollout control.
    • For example, a load balancer plus a rolling update strategy managed by your orchestrator.

AWS RDS IAM authentication for metadata PostgreSQL

When the shared metadata PostgreSQL database is AWS RDS or Aurora PostgreSQL, every Bytebase replica can enable IAM auth by adding the Bytebase AWS parameters to PG_URL. URI-style values are recommended. Direct compact keyword/value values such as host=... port=... are also supported; this URI-style value is the typical shape:

postgres://[email protected]:5432/bytebase?sslmode=verify-full&bytebase_aws_rds_iam=true&bytebase_aws_region=us-east-1

The PostgreSQL user must be granted rds_iam, and the AWS principal used by each Bytebase process must have rds-db:connect for that database user. Use verified TLS, such as sslmode=verify-full, with a trust store that validates the RDS certificate. Do not put AWS access keys in PG_URL; use the standard AWS SDK credential chain through the runtime environment, such as EKS IRSA, ECS task roles, EC2 instance profiles, environment variables, or shared AWS config.

GCP Cloud SQL IAM authentication for metadata PostgreSQL

When the shared metadata PostgreSQL database is GCP Cloud SQL for PostgreSQL, every Bytebase replica can enable Cloud SQL IAM auth by adding the Bytebase GCP parameters to PG_URL. Keyword/value values are recommended because Cloud SQL IAM PostgreSQL user names commonly contain @:

[email protected] dbname=bytebase bytebase_gcp_cloud_sql_iam=true bytebase_gcp_cloud_sql_instance_connection_name=project-id:us-central1:bytebase-metadata

The Cloud SQL IAM database user must exist and have the needed PostgreSQL privileges. The Google principal used by each Bytebase process must have permission to connect to the Cloud SQL instance, typically through ambient Application Default Credentials such as GKE Workload Identity. Do not put Google service account keys in PG_URL.

By default the connector dials the instance's public IP. For a private-IP-only instance (the recommended production posture), add bytebase_gcp_cloud_sql_ip_type=private (or psc for Private Service Connect); public is the default. Bytebase must have a network path to the selected IP — for private IP, run it inside the same VPC (Private Service Access).

How replica detection works

Bytebase tracks live replicas with heartbeats:

  • Each replica writes a heartbeat immediately on startup and then every 10 seconds.
  • A replica is considered active when it has sent a heartbeat within the last 30 seconds.
  • The actuator API exposes the current active replica count as replicaCount on GET /v1/actuator/info.
  • Bytebase always reports at least one active replica for the current server, even if heartbeat counting fails.

What happens when HA is not licensed

If more than one active replica is detected and the license does not enable HA, Bytebase does not permit the HA topology.

The current runtime behavior is to log warnings such as:

multiple replicas detected (<count>) but HA is not enabled in license

When that condition is present, background runners that check the replica limit skip work instead of continuing in an unsupported topology. This includes scheduler and cleaner paths used for task execution, plan checks, schema sync, approvals, and stale-run cleanup.

Operator validation checklist

Use this checklist when enabling or validating HA:

  1. Call GET /v1/subscription and verify ha is true.
  2. Call GET /v1/actuator/info and record:
    • version
    • externalUrl
    • replicaCount
  3. Confirm every replica starts with --ha, points to the same PG_URL, and uses the same external URL.
  4. After adding or restarting replicas, allow at least 30 seconds for the active replica count to settle.
  5. Review logs for HA-license warnings before declaring the topology healthy.

Administrator recovery

Run bytebase recovery beside the running replicas, with the exact same shared PG_URL used by every replica; do not stop them. The command connects directly to the metadata database and does not start an HTTP server, scheduler, or background runner. Replicas started with --ha read users, settings and policies from the metadata database on every request, so they see a recovery change at once and need no restart.

Follow the administrator recovery runbook for backup, action-selection, Kubernetes Job, login, and cleanup requirements. Recovery is not an alternative to HA licensing or replica coordination, and it does not repair the deployment topology.

Troubleshooting

replicaCount stays at 1

Check the following:

  • The additional Bytebase server is actually running.
  • The replica can reach the shared metadata database and write heartbeats.
  • You waited long enough for the new replica to start and publish heartbeats.

replicaCount is lower than expected during a restart

A replica falls out of the active set after roughly 30 seconds without a heartbeat. A brief drop during restarts or node moves can therefore be expected.

Logs show multiple replicas detected ... but HA is not enabled in license

This means the deployment topology and license do not match. Resolve it by doing one of the following:

  • reduce the deployment back to a single active replica, or
  • install a license where GET /v1/subscription returns ha: true.

Old heartbeat rows exist in the database

Stale heartbeat rows are cleaned up separately and do not define the active replica count. Active counting only considers heartbeats from the last 30 seconds.

Related docs