Git designed for AI agents.
gitaly-backup
Overview
The gitaly-backup tool establishes a gRPC connection with a Gitaly instance to facilitate repository backups by requesting
backup data and transferring it to a specified storage sink, either locally or in a remote cloud storage service (such as AWS S3,
Google Cloud Storage, or Azure).
Gitaly supports two backup approaches:
The
gitaly-backupCLI initiates Remote Procedure Calls (RPCs) to the Gitaly server requesting backup data for one or more repositories. The Gitaly server generates the backup data and streams it back togitaly-backupover the RPC connection.gitaly-backupthen transfers the received data to the specified storage destination, as defined by the-pathargument.+------------------+ Repository data +-----------------+ | gitlab-backup-cli| backup job request > | Gitaly | +------------------+ +-----------------+ ^ | | | | Creates backup and | | gRPC Streamed Data to | | <---------------------------------------------- + | | | +------------------------+ +-> write repository data write to | Backup Storage | | (Local / S3 / GCS etc.)| +------------------------+For a detailed overview, see the GitLab backup sequence diagram.
[!note] In setups where Gitaly and
gitaly-backuprun on the same host (such as smaller reference architectures with fewer than 1,000 seats), this approach can reduce network load on Gitaly by offloading remote network egress overhead to thegitaly-backupCLI. Since Gitaly streams backup data locally via gRPC, it minimizes network congestion and improves efficiency.Direct Gitaly Server-Side Backup – This approach prioritizes direct transfer from Gitaly server to the destination sink. CLI initiates Remote Procedure Calls (RPCs) to the Gitaly server requesting backup data for one or more repositories. The Gitaly server generates the backup data and streams this directly to the storage backend configured in the Gitaly configuration file. in the Gitaly
config.toml+------------------+ 1. Repository +------------------+ | gitlab-backup-cli| > data backup > | Gitaly | +------------------+ request +------------------+ | | 2. Creates backup and | directly writes to v +------------------------+ | Backup Storage | | (Local / S3 / GCS etc.)| +------------------------+For a detailed overview, see the Server-side backup sequence diagram
[!note] Server-side backups reduce network and disk pressure and could offer higher speed in running backups.
How to set up for backup
For each project to backup, find the Gitaly storage name and relative or disk path using either:
- The Admin area.
- The repository storage API.
Generate the
GITALY_SERVERSenvironment variable. This variable specifies the address and authentication details of each storage to restore to. The variable takes a base64-encoded JSON object.For example:
export GITALY_SERVERS=`echo '{"default":{"address":"unix:/var/opt/gitlab/gitaly/gitaly.socket","token":""}}' | base64 --wrap=0`Generate the backup job file. The job file consists of a series of JSON objects separated by a new line (
\n).Attribute Type Required Description addressstring no Address of the Gitaly or Gitaly Cluster server. Overrides the address specified in GITALY_SERVERS.tokenstring no Authentication token for the Gitaly server. Overrides the token specified in GITALY_SERVERS.storage_namestring yes Name of the storage where the repository is stored. relative_pathstring yes Relative path of the repository. gl_project_pathstring no Name of the project. Used for logging. For example,
backup_job.json:{ "storage_name":"default", "relative_path":"@hashed/f5/ca/f5ca38f748a1d6eaf726b8a42fb575c3c71f1864a8143301782de13da2d9202b.git", "gl_project_path":"diaspora/diaspora-client" } { "storage_name":"default", "relative_path":"@hashed/6b/86/6b86b273ff34fce19d6b804eff5a3f5747ada4eaa22f1d49c01e52ddb7875b4b.git", "gl_project_path":"brightbox/puppet" }
Gitaly-Backup Client-Orchestrated backup
Pipe the backup job file to
gitaly-backup create. Set up the environment as show in Set up environment step/opt/gitlab/embedded/bin/gitaly-backup create -path $BACKUP_DESTINATION_PATH < backup_job.jsonArgument Type Required Description -pathstring yes Directory where the backup files will be created. -path-prefixstring no Optional directory prefix where backups will be created, useful for time-based rotation of the backup destination. -parallelinteger no Maximum number of parallel backups. -parallel-storageinteger no Maximum number of parallel backups per storage. -idstring no Used to determine a unique path for the backup when a full backup is created. -incrementalbool no Indicates whether to create an incremental backup. -server-sidebool no Indicates whether to use server-side backups.
Gitaly-Backup Client-Orchestrated Restore
Set up the environment as show in Set up environment step
Pipe the restore job file to
gitaly-backup restore./opt/gitlab/embedded/bin/gitaly-backup restore -path $BACKUP_SOURCE_PATH < restore_job.jsonArgument Type Required Description -pathstring yes Directory where the backup files are stored. -path-prefixstring no Optional directory prefix where backups will be created, useful for time-based rotation of the backup destination. -parallelinteger no Maximum number of parallel restores. -parallel-storageinteger no Maximum number of parallel restores per storage. -idstring no ID of full backup to restore. If not specified, the latest backup is restored (default). -remove-all-repositoriescomma-separated list no List of storage names to have all repositories removed from before restoring. You must specify GITALY_SERVERSfor the listed storage names.-server-sidebool no Indicates whether to use server-side backups.
Direct Gitaly Server-side backup
Server-side create stream backup data from the object storage sink defined in the Gitaly config. To use server-side create:
Set up the environment and create backup job file as show in Set up environment step.
Add backup parameters (
enabledandgo_cloud_url) to the Gitalyconfig.tomlfile.[backup] go_cloud_url = "s3://gitaly-backup?region=us-east-1"See object storage for information on how to configure
go_cloud_urlstorage.Ensure the Gitaly user or IAM credentials has access to the object storage i.e:
in aws ensure the role has a policy similar to :
{ "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": [ "s3:PutObject", "s3:GetObject", "s3:ListBucket" ], "Resource": [ "arn:aws:s3:::gitaly-backup", "arn:aws:s3:::gitaly-backup/*" ] } ] }In gcp the Gitaly user/service account
{ "bindings": [ { "role": [ "roles/storage.objectGet", "roles/storage.objectList", "roles/storage.objectPut" ], "members": [ "serviceAccount:<your-service-account>@<your-project>.iam.gserviceaccount.com" ] } ] }
Restart your Gitaly server.
Pipe the backup job file to
gitaly-backup create./opt/gitlab/embedded/bin/gitaly-backup create -server-side < backup_job.jsonArgument Type Required Description -parallelinteger no Maximum number of parallel restores. -parallel-storageinteger no Maximum number of parallel restores per storage. -idstring no ID of full backup to restore. If not specified, the latest backup is restored (default). -remove-all-repositoriescomma-separated list no List of storage names to have all repositories removed from before restoring. You must specify GITALY_SERVERSfor the listed storage names.-server-sidebool yes Indicates whether to use server-side backups. -path-prefixstring no Optional directory prefix where backups will be created, useful for time-based rotation of the backup destination. Important: Do not set
-pathflag. It cannot be used in server-side mode as Gitaly uses thego_cloud_urlset in the Gitaly config file.
WORM (Write Once Read Many) server-side backups
Gitaly does not modify or delete any backup files that are already uploaded. Therefore, it is compatible with WORM storages.
Direct Gitaly Server-side restore
Server-side restore stream backup data from the object storage sink defined in the Gitaly config. To use server-side restore:
Set up the environment and create backup job file as show in Set up environment step
Add backup parameters (
enabledandgo_cloud_url) to the Gitalyconfig.tomlfile.[backup] go_cloud_url = "s3://gitaly-backup?region=us-east-1"See object storage for information on how to configure
go_cloud_urlstorage.Ensure the Gitaly user or IAM credentials has access to the object storage i.e:
in aws ensure the role has a policy similar to :
{ "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": [ "s3:PutObject", "s3:GetObject", "s3:ListBucket" ], "Resource": [ "arn:aws:s3:::gitaly-backup", "arn:aws:s3:::gitaly-backup/*" ] } ] }In gcp the Gitaly user/service account
{ "bindings": [ { "role": [ "roles/storage.objectGet", "roles/storage.objectList", "roles/storage.objectPut" ], "members": [ "serviceAccount:<your-service-account>@<your-project>.iam.gserviceaccount.com" ] } ] }
Restart your Gitaly server.
Pipe the backup job file to
gitaly-backup restore./opt/gitlab/embedded/bin/gitaly-backup restore -server-side < backup_job.jsonArgument Type Required Description -parallelinteger no Maximum number of parallel restores. -parallel-storageinteger no Maximum number of parallel restores per storage. -idstring no ID of full backup to restore. If not specified, the latest backup is restored (default). -remove-all-repositoriescomma-separated list no List of storage names to have all repositories removed from before restoring. You must specify GITALY_SERVERSfor the listed storage names.-server-sidebool yes Indicates whether to use server-side backups. -path-prefixstring no Optional directory prefix where backups will be created, useful for time-based rotation of the backup destination. Important: Do not set
-pathflag. It cannot be used in server-side mode as Gitaly uses thego_cloud_urlset in the Gitaly config file.
Concepts and Flags
Path
Path determines where on the local filesystem or in object storage backup files
are created on or restored from. The path is set using the -path flag.
Local Filesystem
If -path specifies a local filesystem, it is the root of where all backup
files are created.
Object Storage
gitaly-backup supports streaming backup files directly to object storage
using the gocloud.dev/blob library.
-path or go_cloud_url can be used with:
- Amazon S3. For example
go_cloud_url = "s3://my-bucket?region=us-west-1". - Azure Blob Storage. For example
go_cloud_url = "azblob://my-container". - Google Cloud Storage. For example
go_cloud_url = "gs//my-bucket". - memory block storage backend. For example
go_cloud_url = "mem://my-backup". - Filesystem storage backend for example
go_cloud_url = "/tmp//my-backup".
Since not all s3-compatible storage backends support integrity checks, if endpoint
parameter is configured, Gitaly defaults to disabling these checks by using the query
parameter request_checksum_calculation=when_required.
To enable checksum calculation for any third-party s3 provider, which is required for
S3 Object Lock, include the request_checksum_calculation=when_supported query parameter
in the provided go_cloud_url within the Gitaly config.
Reference backends
A backup records nothing about the reference backend of the repository it was taken from. The bundle it contains works with any backend, so a restore that has to recreate the repository creates it the way the Gitaly node creates any other new repository. The default changes between releases, and a restore follows whatever the node uses at the time, so a repository does not necessarily come back on the backend it was backed up from.
Pass --reference-backend to choose a backend instead of letting the node decide. It accepts
files, reftable and mvcc:
gitaly-backup restore --path $BACKUP_DESTINATION_PATH --reference-backend mvcc < repositories.json
The flag only applies to repositories that the restore has to create. A repository that still
exists keeps the backend it already uses. Restoring on the mvcc backend requires the target
Gitaly to ship the MVCC-enabled Git binary, as it is the only one that can read those
repositories.
Backup Layout
The way backup files are arranged on the filesystem or on object storage is determined by the manifest layout. Gitaly uses manifest files to describe where each file exists in a backup to restore the files from the backup. Every backup has one manifest named after the backup ID.
For example, the following structure is created for a repository with the:
- Storage name of
default. - Relative path of
@hashed/4e/c9/4ec9599fc203d176a301536c2e091a19bc852759b255bd6818810a42c5fed14a.git. - Backup ID of
20210930065413. - Backup path prefix of
feb-2026
$BACKUP_DESTINATION_PATH/
feb-2026/
manifests/
default/
@hashed/
4e/
c9/
4ec9599fc203d176a301536c2e091a19bc852759b255bd6818810a42c5fed14a.git/
20210930065413.toml
default/
@hashed/
4e/
c9/
4ec9599fc203d176a301536c2e091a19bc852759b255bd6818810a42c5fed14a.git/
20210930065413/
001.bundle
001.refs
If an incremental backup was then created with a backup ID of 20211001065419,
gitaly-backup would create the following structure:
$BACKUP_DESTINATION_PATH/
feb-2026/
manifests/
default/
@hashed/
4e/
c9/
4ec9599fc203d176a301536c2e091a19bc852759b255bd6818810a42c5fed14a.git/
20210930065413.toml
20211001065419.toml
default/
@hashed/
4e/
c9/
4ec9599fc203d176a301536c2e091a19bc852759b255bd6818810a42c5fed14a.git/
20210930065413/
001.bundle
001.refs
20211001065419/
002.bundle
002.refs
In this case, the manifest file 20211001065419.toml would reference all the
bundle files required to restore the repository.
Important: Incremental backups should have the same -path-prefix as the full backup they are based on. If the path prefix has changed, a new full backup will be created for the first incremental backup run in the new backup path prefix.