Skip to content

Git designed for AI agents.

5.3 KB
Raw Blame

Bundle URI

Git supports bundling a repository into a bundle. A bundle is essentially a file that stores a packfile along with some extra metadata, including a set of refs and a (possibly empty) set of necessary commits.

The bundle URI feature is a mechanism by which Git takes advantage of those bundles to reduce the load on the Git server during a git clone command: assuming a bundle already exists for a repository, the Git server can return the URI of the bundle to the client prior to the negotiation process by which a client asks the server for the objects it needs. Thus, the client can download the bundle, process it, and request from the Git server only the objects it needs that were not in the bundle, which reduces the amount of work the Git server has to do to generate packfiles for this client. This feature can drastically reduce the load on a Git server, especially when frequent clones are made for a large repository.

How Gitaly uses bundle URI

Gitaly supports bundle URI for git clone to reduce the load on the server.

It does so in two different ways.

Manual bundle

Gitaly has a bundle-uri command that can be used to create a bundle for a repository. For more information on the command, see the Gitaly user documentation. Once a bundle has been manually generated for a given repository, any subsequent git clone command for this repository will return the bundle URI provided that the feature flags are turned on.

Automatic bundle

Gitaly can also automatically generate bundles based on heuristics. Because the idea behind bundle-uri is to speed up clones by unloading the Gitaly server from the task of generating the same packfile multiple times, Gitaly can detect periods of time when clones are frequent (such as a running CI/CD pipelines) for a given repository and automatically generate a bundle to allow further clones to use the bundle.

[!NOTE] Gitaly generates a bundle only for the default branch (HEAD).

Enabling bundle URI

Enabling bundle URI in Gitaly is a two-step process.

1. Configure the bundle bucket

You must configure a bucket to store the bundles. Here is a snippet of the configuration to use:

# Bundle-URI
[bundle_uri]
# The destination object-storage URL.
go_cloud_url = "gs://my-bundle-uri-bucket"

The value of bundle_uri.go_cloud_url must be a cloud bucket URL. We use the gocloud.dev library under the hood.

[!important] If the bucket is not set (or set to an empty string), bundle URI is disabled.

2. Enable feature flags

You must enable two feature flags to enable bundle URI:

  • bundle_generation: enables auto-generation of bundles.
  • bundle_uri: makes Gitaly advertise bundle URIs when they are available (either manually created or auto-generated) and allows the user to manually generate bundles.

Automatic bundle generation

Gitaly can generate bundles automatically using heuristics. Gitaly makes that decision through an abstraction called a Strategy, which is currently implemented in OccurenceStrategy.

This strategy has an Evaluate method that can be called to determine if a bundle should be generated for a given repository. To perform that evaluation, the strategy keeps track of the number of times (occurrences) the Evaluate method was called for each repository. If the number of occurrences reaches a certain threshold in a given time interval, a bundle is generated. The strategy also keeps track of the last time Gitaly generated a bundle for a repository. When a new bundle should be generated, based on the occurrences, the strategy looks at the last time a bundle was generated for the given repository. It will only generate a new bundle if the existing bundle is older than some maxBundleAge configuration.

This strategy assumes that if clones are frequent for a given repository in a short amount of time, it must be that some process such as a CI/CD pipeline is running for this repository. Thus, generating a bundle during that time will allow future CI/CD jobs in the pipeline to use the bundle, thus reducing the load on the server. The bundle will also be available for the next pipelines until it is cleaned up from the cloud storage.

Cleaning up old bundles from the storage

Currently, Gitaly does not handle cleaning up old bundles from the cloud storage. This responsibility is delegated to the cloud storage solution. The bucket used to store the bundles should be configured with object lifecycle management policies that allows for the cleaning up of old resources. On GitLab.com, bundles are deleted after 24 hours from their creation timestamp.

For GCP, you can take a look at the object lifecycle management documentation.