Git designed for AI agents.
Using Jaeger for Gitaly local development
Distributed tracing is an essential tool for understanding how services interact with each other, especially during local development. This tool is very helpful for tracking a flow of a request from GitLab services (Workhorse, GitLab Shell, GitLab Rails) to Gitaly and identifying any issues that might be affecting the performance of your system.
If you're working on local development for Gitaly, you can use Jaeger, a popular open-source distributed tracing system. Jaeger can:
- Visualize the flow of requests between services and identify any bottlenecks.
- Comprehend data propagation and correlation between them.
When combined with Trace2, Jaeger can provide a much deeper look into how Gitaly spawns and manages Git processes.
Configure Jaeger in local environment
Fortunately, Trace2 and distributed tracing are already enabled in Gitaly by
default. Gitaly reads the GITLAB_TRACING environment variable to configure
its tracing backend. The supported format is:
# OTLP over HTTP (insecure, typical for local development)
GITLAB_TRACING=otlp://localhost:4318
# OTLP over HTTPS (secure)
GITLAB_TRACING=otlps://collector.example.com:4318
To start tracing, all you need to do is set up Jaeger:
- If you are using the GDK for local development, getting Jaeger setup involves
some additional configuration to the
gdk.ymland restarting. You can follow this tutorial for more information. - Otherwise, you'll need to set up Jaeger manually. You can follow this tutorial.
How to use Jaeger for local development
For a video introduction to using Jaeger, see Jaeger for Gitaly local development. The video walks through some common scenarios you might encounter during development.
The follow section describes a simple scenario. The video has more information on other view models, as well as more complicated scenarios.
When Jaeger starts, go to
http://localhost:16686to access the Jaeger dashboard.All resulting traces from any actions you perform are captured by Jaeger through another endpoint using the
GITLAB_TRACINGenvironment variable, which is set up automatically by GDK. As the traces are collected asynchronously, it may take a short moment until the collection is complete.Perform a Git-related action, such as
git clone http://localhost:3000/gitlab-org/gitaly.git.Access the UI dashboard again. The left side includes some filters while the right side shows the timeline and corresponding events. Gitaly is not exposed directly, so any Remote Procedure Calls (RPCs) originate through entry-point components such as Workhorse, GitLab Shell, and GitLab Rails. You’ll see the traces of those services there.
Select a trace to enter the trace detail view. The default view displays the timeline of spans (events) in chronological order, through multiple services. In this example, it shows the details of each
gitcommand spawned by Gitaly.
Each span consists of detailed data that's very helpful for development purposes. Of course, we can add custom instrumentation.

One great value distributed tracing offers is the ability to visualize asynchronous flow in a comprehensive manner. This ability is particularly helpful for debugging a complicated action flowing through multiple services in different orders. Unfortunately, this use case is very common.