Git designed for AI agents.
Unified Load Management Architecture
Architecture Overview
The system is organized into three layers: data collection, decision, and signal export. The data collection layer gathers resource metrics. The decision layer acts on those metrics to protect the system. The signal export layer communicates Gitaly's state to external consumers.
flowchart TB
subgraph dataCollection [Data Collection]
LM["LoadMonitor\ncgroup stats polling\n(CPU throttle, memory, PSI)"]
BT["BurdenTracker\nper-process /proc polling"]
end
subgraph decision [Decision Layer]
RL["Concurrency Limiter\ncontrols admission of new requests"]
LS["Load Shedder\ncancels highest-cost running requests"]
CA["Cost-Aware Admission\ncost hints for cache-servable RPCs"]
end
subgraph signals [Signal Export]
PT["Pressure Trailer\nx-gitaly-pressure\n(node circuit breaker)"]
CT["Cost Trailer\nx-gitaly-cost\n(rate limiting score)"]
PM["Prometheus Metrics"]
SL["Canonical Log Lines"]
end
subgraph external [External Consumers]
RailsWH["Rails / Workhorse\n(emits X-Score HTTP header\nfor upstream rate limiting)"]
end
LM -->|"resource threshold\nnotifications"| RL
LM -->|"critical threshold\nnotifications"| LS
LM -->|"aggregate pressure"| PT
BT -->|"highest-cost request list"| LS
BT -->|"in-flight metrics"| PM
BT -->|"per-RPC resource fields"| SL
PT -->|"gRPC trailer\n(circuit breaker)"| RailsWH
CT -->|"gRPC trailer\n(rate limit score)"| RailsWH
Data Collection Layer
Two complementary data sources that answer different questions.
LoadMonitor
- Consolidates different watchers we have in Gitaly which helps in
answering how stressed a system really is. Replaces the individual
CgroupCPUWatcher,CgroupMemoryWatcher, andCgroupPressureWatcherthat are currently coupled to theAdaptiveCalculatorthrough theResourceWatcherinterface. This part is now done. They have been replaced byConditions(see:/internal/limiter/condition*.gofiles) - Polls cgroup stats at configurable intervals (CPU throttling, memory usage, memory pressure events, PSI counters)
- Components register their own thresholds and receive notifications only when those thresholds are crossed
- Becomes the single source of truth for system-level resource metrics,
including Prometheus gauges currently emitted by the
cgroupspackage
BurdenMonitor
This component would figure out which RPC/process is causing stress.
- Tracks all in-flight RPCs via gRPC interceptors: registers on entry, deregisters on exit
- Tracks spawned commands via the Git command factory
- Polls per-process stats from
/proc/[pid]/statand/proc/[pid]/statusevery x seconds using a single ticker and a bounded worker pool. This is primarily useful for long-running commands (large clones). Short-lived commands may complete before/procis ever polled. This is acceptable as the commands that cause resource contention are mostly the long-running ones. A single ticker seems to be sufficient: benchmarks on a production node show ~250ms to read/procfor 209 concurrent Git processes. At the observed peak of ~1170 processes during a DDoS incident, sequential polling extrapolates to ~1.4s — within the 2s interval but with thin margin. A bounded worker pool should parallelise the reads to stay well within the interval at peak load. - Maintains ordered lists of running requests sorted by CPU time, memory, and duration
- Also addresses the need for periodic logging of long-running command resource consumption (#7097) - the BurdenMonitor already has the data, it just needs to emit periodic log entries for the top-N consumers
These two components have different polling targets (cgroups vs.
/proc/[pid]) and different granularity (system-wide vs. per-process).
Decision Layer
Three components that act on the data. The concurrency limiter decides what gets in. The load shedder decides what gets cancelled. The cost-aware admission layer helps the rate limiter make better decisions about what to let in.
Concurrency Limiter
Controls admission of new requests. The existing
AdaptiveCalculator +
ConcurrencyLimiter:
- Subscribes to LoadMonitor threshold notifications.
- Requests that exceed the current concurrency limit are rejected before consuming resources.
Load Shedder
The concurrency rate limiter reduces the flow of incoming requests when resources are constrained, while load shedder controls in-flight requests. When resource pressure exceeds critical thresholds despite admission control, the load shedder identifies the highest-cost running requests and cancels them to free resources immediately.
- Subscribes to LoadMonitor notifications with critical thresholds (e.g., OOM events, sustained extreme CPU throttle)
- When triggered, queries BurdenMonitor for the highest-cost running requests
- Cancels specific request contexts to recover capacity
End-to-End Flow
The following diagram shows how a client request will flow through the system.
sequenceDiagram
participant Client
participant CL as ConcurrencyLimiter
participant Handler
participant BM as BurdenMonitor
participant LM as LoadMonitor
participant AC as AdaptiveCalculator
participant LS as LoadShedder
Client->>CL: gRPC request
alt limit not reached
CL->>Handler: Admit request
Handler->>BM: RegisterRPC(ctx, method, repo)
BM->>BM: Start tracking commands via /proc polling
par Handler processes request
Handler->>Handler: Spawn git commands
and LoadMonitor polls
LM->>LM: Poll cgroup stats
LM->>LM: Evaluate conditions
alt Healthy
LM->>AC: No event
AC->>CL: limit += 1
else Backoff severity
LM->>AC: BackoffEvent
AC->>CL: limit = floor(limit x 0.5)
else Critical severity
LM->>AC: BackoffEvent
AC->>CL: limit = floor(limit x 0.5)
LM->>LS: CriticalEvent
LS->>BM: EntriesSortedBy(CPU)
BM-->>LS: Ranked in-flight RPCs
LS->>BM: SnipeTopN()
BM-->>Client: RESOURCE_EXHAUSTED (cancel highest-cost)
end
end
Handler->>BM: DeregisterRPC()
Handler-->>Client: Response
else limit reached
CL->>CL: Queue request
alt queue timeout
CL-->>Client: RESOURCE_EXHAUSTED
else slot available
CL->>Handler: Admit request
end
end
Cost-Aware Admission
A separate admission layer that sits before the concurrency limiter in the
gRPC interceptor chain. If a request can be served from
streamcache, it bypasses the concurrency
limiter entirely and proceeds directly to the handler. Currently this applies
to:
PostUploadPackWithSidechannel(clone/fetch)
Request -> Cost-Aware Admission -> Concurrency Limiter -> Handler
|
(cache hit?)
|
bypass limiter -> Handler directly
The concurrency limiter today treats every request as equally expensive. In practice, requests served from cache consume minimal resources (disk streaming) while cache misses run full Git subprocess pipelines. Rather than feeding the limiter a cost hint (which still consumes a limiter slot), cache-hit requests skip the limiter entirely — keeping the concurrency limiter pure and ensuring limits designed for expensive operations do not penalize cheap ones.
This separation also keeps the concurrency limiter's responsibility narrow: every request that reaches it is treated as expensive.
Signal Export Layer
Gitaly exports two distinct signals per RPC response: a node-level pressure value for circuit breaking, and a per-RPC cost score for rate limiting. These serve different purposes and should not be conflated.
Pressure Trailer (x-gitaly-pressure)
This layer attaches a 0.0-1.0 pressure value as a gRPC response trailer on every
RPC. The pressure value reflects the health of the entire Gitaly node — it is a
global signal, not per-project or per-namespace.
Client maintains a moving average of pressure per Gitaly storage and uses it to trip a circuit breaker for that node when sustained pressure is detected.
Rails will receive the pressure signal even for RPCs that bypass it (clone, fetch, push), because other concurrent requests to the same node pass through Rails and carry the trailer.
The pressure value comes from a PressureAggregator interface:
type PressureAggregator interface {
AggregatedPressure() float64
}
This interface allows the backing data source to evolve independently of
the gRPC interceptor that consumes it. Once the LoadMonitor becomes the single
polling point for all cgroup stats, the PressureAggregator implementation will
collect stats from LoadMonitor.
The CgroupPressureCondition
already reads PSI (Pressure Stall Information) data from the kernel for
memory, IO, and CPU. It classifies severity levels (healthy, warning,
backoff, critical) and has configurable thresholds.
| Signal | What it measures | Normalization |
|---|---|---|
| PSI some avg10 (memory, IO, CPU) | % of time tasks were stalled waiting for a resource | psi_avg10 / critical_threshold per resource (thresholds already defined in CgroupPressureCondition) |
| Anonymous memory ratio | TotalAnon / MemoryLimit -- how close to the memory limit, excluding reclaimable page cache |
anon_ratio / memory_threshold (threshold already defined in CgroupMemoryCondition, default 0.6) |
| CPU throttle ratio | CPUThrottledDuration delta over observation window |
throttle_ratio / cpu_threshold (threshold already defined in CgroupCpuCondition, default 0.5) |
The pressure value takes the maximum across all normalized signals:
pressure = max(
memory_psi_some_avg10 / memory_critical_threshold,
io_psi_some_avg10 / io_critical_threshold,
cpu_psi_some_avg10 / cpu_critical_threshold,
anon_ratio / memory_threshold,
cpu_throttle_ratio / cpu_throttle_threshold,
) clamped to [0.0, 1.0]
Additionally, discrete events spike pressure to 1.0 immediately:
OOMKillscounter increased since last pollMemoryMaxEventscounter increased (memory nearly exceeded hard limit)
The thresholds are already calibrated by the existing watcher implementations, so the pressure value aligns with the points where the system starts to degrade.
RPC Cost Score (x-gitaly-cost)
Gitaly returns a cost score for each RPC as a gRPC response trailer.
Gitaly has the most context about the actual cost of each RPC, making it the right
place to own this value. Rails and Workhorse translate the x-gitaly-cost
trailer into an X-Score HTTP response header, making the cost signal
available to any upstream rate limiter.
On GitLab.com, the
X-Scoreheader feeds Cloudflare's complexity-based rate limiting. Self-managed deployments can use the same header with any upstream rate limiter or ignore it.
Static scores are derived from historical data per RPC type. A static score hides
variance (e.g. PostUploadPackWithSidechannel is the same cost whether it serves
1 MB or 10 GB), but dynamic cost (bytes transferred, object count) is only known
after the RPC completes. The two approaches can be combined: use the static score
as a base, then reconcile actual cost in a follow-up after Gitaly responds.
Prometheus Metrics
- BurdenMonitor: in-flight count and CPU/memory per RPC and cumulative CPU time and memory consumed
- LoadMonitor: current value per signal PSI avg10, anon ratio, CPU throttle and time spent at each severity level per signal.
- Pressure:
gitaly_pressureaggregated 0.0–1.0 per storage andgitaly_pressure_seconds_totalby severity level.
Connection to Rails and Workhorse
Gitaly's role in the broader DoS mitigation system is to be the innermost ring of defense and to communicate two signals outward: node health (for circuit breaking) and per-RPC cost (for rate limiting).
What Gitaly provides:
x-gitaly-pressuregRPC trailer on every response (0.0 = healthy, 1.0 = saturated) — node health signalx-gitaly-costgRPC trailer on every response — per-RPC cost scoreRESOURCE_EXHAUSTEDgRPC errors when requests are rejected by the concurrency limiter or load shedder.
What Rails does with it:
- Reads the
x-gitaly-pressuretrailer via a gRPC client interceptor - Maintains a moving average of pressure per Gitaly storage
- Trips a per-node circuit breaker when sustained pressure is detected
- Translates
x-gitaly-costinto theX-ScoreHTTP response header for upstream consumption
For standard requests, Rails is in the response path:
(upstream rate limiter) <- X-Score header <- Rails <- x-gitaly-cost trailer <- Gitaly
For Git operations (clone, fetch, push), Workhorse streams directly
to/from Gitaly — Rails is only in the auth path. Workhorse reads the
x-gitaly-cost trailer from the Gitaly response and sets X-Score on
the HTTP response to the client:
(upstream rate limiter) <- X-Score header <- Workhorse <- x-gitaly-cost trailer <- Gitaly
|
Rails auth response