Git designed for AI agents.
Gitaly Benchmarking Tool
What is this?
An Ansible script for running RPC-level benchmarks against Gitaly.
Benchmarks are organised into "experiments", defind as subdirectories within the experiments/ folder. Each experiment
consists of four configuration files:
config.yml- sets up the infrastructure and repositories used for the benchmark. Repository-level configuration like the reference backend and test data are defined here.k6-benchmark.js- defines the offered workload to Gitaly during the benchmark. Workloads in K6 are organised as scenarios which can run concurrently. Refer to the k6 documentation for more information. The runtime is upper-bounded by theworkload_wait_durationdefned inroles/benchmark/vars/main.yml.plot.py- ingests the Gitaly JSON logfile and uses thepandasandplotninelibraries to produce graphs.profile-gitaly.sh- executes concurrently with K6 to profile the Gitaly host under load. The profiling duration is defined asprofile_durationinroles/benchmark/vars/main.yml. This duration should be <= the workload duration.
These files can be customised for each experiment and committed back to the Gitaly repository, allowing experiments to
be easily reproduced. When starting a new experiment, you can simply make a copy of the master experiment as a
starting point.
Steps for use
All scripts must be run with the EXPERIMENT environment variable set to the name of a directory within experiments/.
Experiments are deployed using Terraform workspaces, so it's possible to deploy multiple experiments simultaneously.
Feel free to export the environment varible instead of redefining it for each command.
Three deployment topologies are supported, selected via the SETUP environment variable
(defaults to vm):
SETUP |
Where Gitaly runs | How k6 reaches Gitaly |
|---|---|---|
vm |
One or more dedicated GCE VMs (built from source). | Direct gRPC to the Gitaly VM's internal IP. |
kubernetes |
A GKE cluster's dedicated nodepool (Gitaly built on-node). | Direct gRPC to the underlying node's internal IP. |
incluster |
A GKE cluster running the full GitLab Helm chart, provisioned via gl-gitaly/tf-gke-gitlab-bootstrap branch elastic-poc-3 (published as ~> 2.0). |
k6 runs as a Job in the cluster and hits the Gitaly Service directly over the cluster network. |
In-cluster benchmarks (targets an existing Kubernetes cluster)
Use this flow when Gitaly is deployed in an existing Kubernetes cluster (e.g. via the GitLab Helm chart). The benchmark client runs as a Job inside the same cluster and talks to Gitaly over the cluster network.
Prerequisites
kubectlconfigured against the target cluster.gcloudauthenticated; Artifact Registry and GCS in the same GCP project.- A results bucket and the GSA the Job uploads as. The bucket owner grants
the GSA access once; if a shared bucket already exists, skip this and
just point
BENCHMARK_RESULTS_BUCKETat it. Runs are namespaced by a per-runbenchmark-<timestamp>/subpath, so they don't overwrite.
# --member is the GSA email (same value as WORKLOAD_IDENTITY_GSA below),
# Note you DO NOT need to run this, it has already been run
# once. You only need to if you create your own bucket OR own service account.
# This grants permission for the Kubernete's JOB's service account
# to deposit the benchmarking results to the bucket.
gcloud storage buckets add-iam-policy-binding gs://bucket-name \
--member="serviceAccount:xxxxxxx@${PROJECT}.iam.gserviceaccount.com" \
--role=roles/storage.objectAdmin
Build and push the client image
This is the image the Kubernetes Job (rendered from
incluster/job.yaml.j2) runs the experiment with: a k6 binary plus the
entrypoint.sh that drives the run and harvests results (built from
docker/benchmark-client/). The image also includes
angle-grinder (agrind), which
computes the per-RPC latency metrics from the harvested Gitaly logs.
make -C _support/benchmarking push-benchmark-image
Image is tagged <git-short-sha>-<unix-timestamp> (plus a floating :latest)
and pushed to the Artifact Registry repo configured in the Makefile.
Commit before building so the SHA matches the image contents. The timestamp makes each build unique, so concurrent builds never overwrite each other. The exact ref is recorded in
.last-pushed-image(gitignored), which the run step uses automatically — setBENCHMARK_IMAGEto override (e.g. a@sha256:...digest).
Provision repos script
cd _support/benchmarking
python3 -m venv env && source env/bin/activate && pip install -r requirements.txt
EXPERIMENT=incluster-master make provision-repos
This creates the projects via the Rails console in the toolbox
pod and writes a repo list the experiment config consumes. For each
repo entry in the experiment's config.yml it:
- creates (or finds) the project under the
benchgroup and imports it from its upstream URL; - pins the default branch to the
revisionfrom the config and creates theadditional_refssynthetic refs, so every run starts from the same ref state; - writes the resulting hashed disk path and
gl_repositoryback intoexperiments/$EXPERIMENT/config.yml(that's the--write-config).
To check both Gitalys run with exactly the same settings,
incluster/config-parity diffs their live config.toml files, ignoring
expected per-deployment differences (paths, addresses, secrets). It
takes the two configs as arguments; fetch each from its pod with
kubectl exec <pod> -c gitaly -- cat /etc/gitaly/config.toml.
When the benchmark-control storage is registered in Rails, the same
pipeline provisions a second identical project set (bench-control/*)
onto the control Gitaly (control_disk_path / control_gl_repository
in config.yml), so A/B comparison runs start from identically
constructed repositories on both sides. It also mints the credentials the
Job authenticates with, recreated every run and read through
secretKeyRef: a 30-day admin PAT in gitaly-benchmark-pat for http,
and an ed25519 keypair in gitaly-benchmark-ssh-key for ssh. Re-run
provisioning when the PAT expires.
The benchmark-control storage serves repos from and directs requests
to the control Gitaly; the default storage does the same for the
MVCC-enabled Gitaly. This is assumed to have been set up already by the
Terraform code that configures the benchmarking cluster.
Re-running is safe: existing projects are reused and repos already at
the pinned state are skipped. Pass --reset to destroy the projects
and re-import them from scratch. Comparison runs need this between
benchmarks: the write workloads pump objects and refs into the repos,
and the MVCC side has no reliable garbage collection to reclaim them,
so the A and B copies drift apart and the comparison stops being fair.
Re-importing is the easiest way back to identical state on both sides.
Push sections use a real git client
The push sections drive a genuine git push per iteration: the
entrypoint shallow-clones each test repository, and benchmark-push
commits a random blob and pushes it as new benchmark-* branches. Each
push section runs once per transport in PUSH_TRANSPORTS:
ssh:git pushover real ssh to gitlab-shell, which authenticates the key, authorizes through Rails and calls Gitaly'sSSHReceivePack.http:git pushthrough workhorse/webservice with the provisioned PAT, so Rails auth and workhorse are in the measured window.
In comparison runs the push sections run on both sides with both
transports; each side has its own project set on its own Rails storage,
so pushes land on the Gitaly the run measures. push_duration records
the wall time of git push alone; k6's iteration_duration includes
clone/commit prep, so don't read it as push latency.
Run a benchmark
# BENCHMARK_RESULTS_BUCKET: bucket name only, no gs:// prefix.
# Default: gitaly-benchmark-results (export to override).
# WORKLOAD_IDENTITY_GSA: GSA email (same as --member). Auto discovered if unset.
export BENCHMARK_RESULTS_BUCKET=gitaly-benchmark-results
export WORKLOAD_IDENTITY_GSA=<sa>@<project>.iam.gserviceaccount.com
EXPERIMENT=incluster-master _support/benchmarking/run-benchmarks-incluster
run-benchmarks-incluster renders the manifests in incluster/, applies them with kubectl, follows the Job logs, and exits with the Job's status. Results are uploaded to gs://$BENCHMARK_RESULTS_BUCKET/.
By default it uses the image from the last make push-benchmark-image
(via .last-pushed-image). Set BENCHMARK_IMAGE to override.
Common vars for in-cluster benchmarks
| Variable | Default | Purpose |
|---|---|---|
BENCHMARK_IMAGE |
ref from .last-pushed-image, else <AR repo>/benchmark-client:latest |
Override the auto-resolved image — e.g. pin a @sha256:... digest or replay a specific <sha>-<timestamp> tag. |
KUBE_NAMESPACE |
gitlab |
Namespace for both the Job and Gitaly (always co-located). |
GITALY_SERVICE / GITALY_PORT |
gitlab-gitaly / 8075 |
Gitaly target. |
WORKLOAD_DURATION |
120s |
k6 run time. |
STARTUP_GRACE |
600 |
Seconds added to the Job's activeDeadlineSeconds to cover pod startup before the workload runs (scale-from-zero spot-pool node provisioning + image pulls). Raise if a cold spot pool is slow to come up. |
PROFILE / PROFILE_DURATION |
true / 120 |
Scrape pprof from each Gitaly pod over HTTP. |
PUSH_TRANSPORTS |
ssh,http |
Which transports the push sections run: ssh (gitlab-shell → SSHReceivePack; needs the SSH key from provisioning) and/or http (workhorse → PostReceivePack; needs the PAT). |
GITLAB_HTTP_URL |
webservice Service in KUBE_NAMESPACE, port 8181 |
Workhorse base URL for repo clones and http pushes. |
GITLAB_SSH_URL |
gitlab-shell Service in KUBE_NAMESPACE, port 22 |
Base URL for the ssh transport. |
BENCH_GROUP |
bench |
Group of the provisioned projects; must match provision-repos. |
CLEAR_GITALY_CACHE |
false |
Restart the Gitaly StatefulSet between runs for cold-cache numbers. |
CLEAR_PAGE_CACHE |
false |
Also drop the node page cache (requires CLEAR_GITALY_CACHE=true and kubectl debug node permission). |
Running on a dedicated node pool
The benchmark Job runs on a dedicated node pool, isolated from the Gitaly,
GitLab, and system pods it measures. The nodeSelector and tolerations that
pin the Job to this pool are baked into incluster/job.yaml.j2, so the normal
invocation needs no scheduling flags:
EXPERIMENT=incluster-master _support/benchmarking/run-benchmarks-incluster
Without isolation, a higher-priority pod scheduling onto a full shared node can preempt (evict) the benchmark mid-run, and the benchmark would be a noisy neighbour to the very Gitaly it measures.
The pool is provisioned in the cluster's Terraform
(gl-gitaly/tf-gke-gitlab-bootstrap)
with label workload=benchmark and taint dedicated=benchmark:NoSchedule. It
autoscales from zero, so it costs nothing until a run lands on it — the
first benchmark Job triggers a node to spin up, and the pool scales back down
when idle.
If your pool's label/taint differ, edit the nodeSelector/tolerations in
incluster/job.yaml.j2.
Cleanup
Cleanup is automatic. run-benchmarks-incluster uploads results to
gs://$BENCHMARK_RESULTS_BUCKET/<run-name>/ (via the uploader sidecar) and
then deletes the Job and its ConfigMap once results are harvested. As a
backstop, the Job also sets ttlSecondsAfterFinished: 3600, so Kubernetes
garbage-collects it an hour after completion even if the wrapper is
interrupted.
You only need to clean up manually if the wrapper was killed mid-run before it could delete the resources (logs remain inspectable in the meantime):
make -C _support/benchmarking clean-incluster
(or kubectl delete -n "$KUBE_NAMESPACE" job,configmap,serviceaccount -l app=gitaly-benchmark)
The ServiceAccount, Role, and RoleBinding are reused across runs and are not
deleted automatically; the clean-incluster target removes them when you're
done with the cluster.
Automated in Gitaly's CICD Pipeline
- Simply create a branch name that has the same name as the experiment name directory created inside the
_support/benchmarking/experimentsdirectory. If your branch is namedb/experiment-one, the directory name will beb-experiment-one. Ensure the branch name is short as it gets used when creating the instance and there is a 63 character limit imposed on instance names in Google Cloud Platform. Eg. {branch_name}-ci-{YYYYMMDD}-gitaly-{instance_name}
The directory must have the minimum following files:
.
├── config.yml
├── k6-benchmark.js
├── plot.py
├── profile-gitaly.sh
- When you raise a merge request, there will be a
qa:benchmark:performancejob that can be run. It is an optional job so click run to trigger it. - Artifacts can be downloaded straight from the job artifacts.
In-cluster job and result reporting
The qa:benchmark:performance:incluster job runs the in-cluster flow against
the shared GKE cluster. It is a manual job, runs one at a time
(resource_group), and is restricted to the canonical gitlab-org/gitaly
project — fork pipelines are blocked.
The experiment is auto-detected from the files changed under experiments/
versus the MR target branch; if nothing experiment-specific changed it falls
back to incluster-master. Set the EXPERIMENT variable in the manual-job
form to override, or when more than one experiment changed at once.
After the run, results are published three ways:
- MR metrics report.
report.pyrenders an OpenMetrics file (metrics.txt) wired asartifacts:reports:metrics. GitLab shows the values in the MR widget and diffs each metric against the target branch's latest run. Metrics are emitted both overall and per RPC:gitaly_rpc_duration_ms— latency, with astatlabel (p50/p90/p95/max/avg) and, per RPC, anrpclabel.gitaly_benchmark_checks_pass_ratioandgitaly_benchmark_checks_{passes,fails}_total— overall and per RPC.gitaly_benchmark_iterations_total/_per_second/gitaly_benchmark_dropped_iterations_total— throughput counters.
- MR note. The same data rendered as a Markdown comment (summary table, per-RPC latency table, per-RPC checks table). Append-only — each run posts a fresh note so the history is preserved across re-runs.
- Job artifacts. The full results tree (k6 summary, per-pod Gitaly logs
and their
gitaly-stats/, plots, pprof) is kept for 30 days.
Per-RPC latency comes from the harvested Gitaly logs: the benchmark
entrypoint runs angle-grinder
(agrind) over every gitaly-N.log (and their concatenation), aggregating
the server-side grpc.time_ms field per grpc.method into gitaly-stats/
(all-gitalies-latency{,-overall}.json plus per-pod files and per-status-code
counts). report.py reads its latency figures exclusively from these files.
This job benchmarks whatever Gitaly the cluster already runs — not the MR's
code (build-from-source benchmarking is what the vm/kubernetes topologies
are for). To make that explicit, the wrapper records the deployed Gitaly image
(gitaly-version.txt in the results dir); report.py shows it as
Gitaly under test in the note and emits a gitaly_benchmark_build_info
metric line.
Running manually on local machine
1. Setup your environment
- Ensure that
gcloudis installed and available on your path. - Ensure that
pythonandterraformare installed, or useasdfto install them (recommended). - Create a new Python virtualenv:
python3 -m venv env - Activate the virtualenv:
source env/bin/activate - Install Ansible:
python3 -m pip install -r requirements.txt - Create a
config.ymlto customize the machine types and repositories used for benchmarking. If you don't have access to the default GCP project, you can point the configuration to your own. - Initialize terraform to install required dependencies
cd terraform && terraform init. - Intitalize
gcloudand authenticate and set the projectgcloud auth login && gcloud auth application-default login && gcloud config set project <project_here>.
2. Create instance
EXPERIMENT=master ./create-benchmark-instance
This will create the Gitaly nodes and a small client node to send requests to Gitaly over gRPC. This will prompt for the instance name and public SSH key to use for connections.
Use the gitaly_bench user to SSH into the instance if desired:
ssh gitaly_bench@<INSTANCE_ADDRESS>
3. Configure instance
EXPERIMENT=master ./configure-benchmark-instance
Build and install Gitaly from source with from desired reference and install
profiling tools. Test repositories will be cloned directly to /mnt/git-repositories
on each Gitaly node during instance startup.
4. Run benchmarks
EXPERIMENT=master ./run-benchmarks
Run the specified experiment, using the scripts defined in the experiments/${EXPERIMENT}
directory. Profiling can be optionally disabled to eliminate overhead, but graphs will still
be generated.
EXPERIMENT=master ./run-benchmarks --extra-vars "profile=false"
On completion a tarball of the benchmark output will be written to
experiments/${EXPERIMENT}/results/benchmark-<BENCH_TIMESTAMP>.tar.gz. The root directory
contains the k6 test summary and associated logs. Each subdirectory is named after
the target Gitaly instance (i.e. gitaly_instances[*].name from the config.yml
file) and contains instance-specific output.
5. Destroy instance
EXPERIMENT=master ./destroy-benchmark-instance
All nodes will be destroyed. As GCP will frequently reuse public IP addresses,
the addresses of the now destroyed instances are automatically removed from
your ~/.ssh/known_hosts file to prevent connection failures on future runs.
Automatic cleanup of resources
Destroying resources using the destroy-benchmark-instance is a manual step. People can sometimes
forget to run the step, which can lead to higher cost. The cleanup directory contains the code
to create a scheduled function in GCP to delete leftover resources.
Creating the scheduled function is a one time thing and must be run manually.
It can be done by running:
cd ./cleanup/infra
terraform init
terraform apply
The tfstate is stored in a bucket on the GCP benchmarking project. This allows multiple
team member to update the function using the same centralized state.
FAQs
How do we change the k6 test duration time?
- Directly inside the
k6-benchmark.jsthere are timings to be changed for read/write RPCs and that control the ramping times of these RPC. By default we are using the ramping option.
const stages_read = [{target: 50, duration: '20s'}, {target: 100, duration: '10s'}, {target: 200, duration: '20s'}, {target: 50, duration: '10s'}]
const stages_write = [{target: 25, duration: '20s'}, {target: 50, duration: '10s'}, {target: 100, duration: '20s'}, {target: 25, duration: '10s'}]
- Edit
_support/benchmarking/roles/benchmark/vars/main.yml
profile_duration: 600
workload_wait_duration: 600
This ensures the ansible workload wait job has sufficient time to wait for till the end of the k6 test run. Note profile_durationshould be <= the workload duration.
Alternatively if your k6-benchmark.js script is using the static option of a constant workload for the RPCs.
- Edit
_support/benchmarking/roles/benchmark/vars/main.yml
profile_duration: 600
workload_duration: "600s"