Git designed for AI agents.
Pack-objects cache design notes
The purpose of this document is to give more insight into the design choices we made when building the first iteration of the pack-objects cache in https://gitlab.com/groups/gitlab-com/gl-infra/-/epics/372.
Introduction
Please read the administrator documentation for the pack-objects cache if you are not already familiar with what it does or what it is for.
Please read Pack-objects cache for CI Git clones epic for more information about Pack-objects cache.
High-level architecture
sequenceDiagram
participant A as Gitaly (PostUploadPack)
participant B as git-upload-pack
participant C as gitaly-hooks
participant D as Gitaly (PackObjectsHook)
participant E as git-pack-objects
Note over A,B: pipe
A->>B: fetch request
Note over B,C: pipe
B->>C: pack request
Note over C,D: socket/sidechannel
C->>D: gRPC request
Note over D,E: pipe
D->>E: pack request (miss)
Note over E,D: pipe
E-->>D: packfile data
Note over C,D: socket/sidechannel
D-->>C: gRPC response
Note over C,B: pipe
C-->>B: packfile data
Note over A,B: pipe
B-->>A: fetch response
The whole pack-objects cache path depends on
uploadpack.packObjectsHook
option. When upload-pack would run git pack-objects to create a packfile for a
client, it will run gitaly-hooks binary instead. The arguments when calling
gitaly-hooks includes git pack-objects at the beginning. This pattern is
similar to how Gitaly handles Git hooks during a push (such as pre-receive
and post-receive).
Problem scope
We designed this cache to solve a specific problem on GitLab.com: high Gitaly server CPU load due to massively parallel CI fetches.
That means:
- It was OK if some types of fetch traffic did not become faster, as long as they also did not get slower
- It was OK to make specific assumptions about the infrastructure this runs on
Example for (1): we made sure the cache can stream unfinished responses because without that, cache misses would be noticeably slower.
Example for (2): GitLab.com uses 16TB filesystems with at least 2TB of free space to store repositories. If our cache files are on there, and as long as the average size is reasonable, we don't have to worry about peak cache size. The worst case average cache size we projected was 30GB, which is just 1.5% of the 2TB of expected free space.
This is not to say that we think it is "bad" to want to use this cache for different types of traffic, or on different infrastructure, but it just wasn't our goal when we built it.
Streaming and backpressure
One of the main goals was to have both streaming and backpressure. By "streaming" we mean that a consumer of a cache entry can start reading and stream data before the producer is done writing the entry. We wanted this because Git fetches are relatively slow and we did not want to add to the end-to-end latency. By "backpressure" we mean that the producer of a cache entry blocks if the consumers are not reading what the producer wrote. This has two benefits. First of all, if the consumer hangs up, the producer gets a write error and also stops. This way we don't write data to disk that no-one will look at. Second of all, we get a natural limit on how much IO bandwidth we use. On its own, the producer can write data faster than the consumers can read it. If there was no backpressure, the producer would put excessive write IO pressure on the cache storage disk.
Because of streaming and backpressure, the producer and the consumers of each cache entry must communicate with each other. By keeping the cache local to a single Gitaly process, we made this a concurrent programming problem, instead of a distributed systems problem.
The data structure responsible for streaming and backpressure is
internal/streamcache.pipe.
Configuring backpressure
The backpressure mechanism is enabled by default but can be configured:
[pack_objects_cache]
enabled = true # Enable the pack-objects cache
backpressure = true # Control the backpressure mechanism (default: true)
min_occurrences = 1 # Minimum number of cache hits before caching an object
max_age = "5m" # How long to keep items in the cache
While the backpressure mechanism provides the benefits described above, it can have a side effect:
git-pack-objects processes may remain in a waiting state until the fastest client requesting the
same cache key completes its operation. During this waiting period, these processes can occupy
substantial server resources.
Consider disabling backpressure if you observe:
- Hung
git-pack-objectsprocesses occupying excessive server resources - Memory pressure during heavy load scenarios
- Slower overall performance with backpressure enabled
When backpressure is disabled, Gitaly will write to the cache at its maximum rate regardless of client consumption, which may increase I/O but could improve throughput in certain environments, especially when there are many concurrent fetches of the same data.
Storage considerations
Early on in the project we thought we would use object storage to store the cache data but we later decided not to, for several reasons.
- Object storage would introduce a new point of failure to Gitaly
- Local storage gets a boost from the operating system's page cache; there is no page cache for object storage
- Local storage is simpler to work with
To support the second point: during testing on a server with high cache throughput, where the number of bytes read from the cache peaks over 100MB/s once an hour, we see disk reads at near 0 bytes per second, with peaks of less than 1MB. This is possible because the operating system is able to serve all the pack-objects cache reads from the page cache, i.e. from RAM. This server does use a lot of IO bandwidth, but that is all due to cache writes, not reads. With object storage we would not get this extra layer of RAM caching from the operating system.
Local files get a speed boost from RAM, and GitLab.com servers have lots of unused RAM.
Off by default
The pack-objects cache is off by default because in some cases it
significantly increases the number of bytes written to disk. For more
information, see this issue where
we turned on the cache for gitlab-com/www-gitlab-com.
It would be better if the cache was on by default. But, if you don't have CI-like traffic, there is probably no benefit, and if your Gitaly server can just manage its normal workload, the extra disk writes may push it into saturation.