Git designed for AI agents.
Release
This document describes how changes to Gitaly are deployed into production on www.gitlab.com.
You can find the current Gitaly release status in:
- The
#gitaly-deploymentsSlack channel. This is updated by the GitLab for Slack app whenever a branch is deployed to a new environment. - The Gitaly Version Tracker Grafana dashboard, which shows the currently deployed versions of Gitaly across environments.
- The Gitaly startup logs in Kibana, as the version is logged when Gitaly boots.
sequenceDiagram
autonumber
actor Gitaly developer
Gitaly developer->>gitlab-org/gitaly: Merge MR
gitlab-org/gitaly->>gitlab-org/security/gitaly: Mirror
activate release-tools bot
release-tools bot->>release-tools bot: Execute UpdateGitaly
release-tools bot->>gitlab-org/security/gitaly: Get latest green commit on default branch
release-tools bot->>gitlab-org/gitlab: Create "Update Gitaly version" MR
deactivate release-tools bot
alt Pipeline succeeded
gitlab-org/gitlab->>gitlab-org/gitlab: Merge and deploy
gitlab-org/gitlab->>Slack gitaly_alerts: Deployment progress
else Pipeline failed
gitlab-org/gitlab->>Slack g_gitaly: Report pipeline failure
Gitaly developer->>Gitaly developer: Address failure
end
- After two approving reviews, the MR is merged by the second reviewer and is automatically replicated to
the
gitlab-org/security/gitalymirror (1 and 2). - Once an hour, the
release-toolsbot executes theUpdateGitalytask (3). UpdateGitalyretrieves the latest green commit on the default branch of the mirror and creates an MR with the "Update Gitaly version" title on thegitlab-org/gitlabrepo. The MR updates theGITALY_SERVER_VERSIONfile with the SHA of the latest green commit (4 and 5). See below for more information on how this is accomplished.- The CI pipeline is executed. If the pipeline succeeds, the MR is automatically merged (6) and the
deployment process (7) to the various environments
begins. If the pipeline fails, the failure is reported on the
#g_gitalychannel (8), and the Gitaly team is tagged in the MR. Failures require manual intervention (9). - Once the deployment begins, progress can be tracked in the
#gitaly_alertschannel. The bot also updates theworkflowlabels in the MR as deployment progresses through each stage.
The passing_build_commit and success? functions
in the release-tools bot are responsible for identifying the latest commit with a passing pipeline. The
@project.auto_deploy_path property is what results in the
security mirror.
being targeted.
Why do pipeline failures occur?
The pipeline is unfortunately prone to failure for a variety of reasons:
- Test flake in Gitaly or another component
- A broken master on
gitlab-org/gitlab - A regression in Gitaly
Test flake
If the pipeline failure is suspected to be caused by a flaky test, re-running the pipeline is a good first step to diagnosing the issue. More often than not, the pipeline will pass in the second run. If the flaky test is within the Gitaly codebase itself, consider opening an issue based on the Flaky Test template and quarantining the test if necessary.
Broken master
If master on gitlab-org/gitlab is broken, the best course of action is to wait
for a fix to be merged, and then close the MR. When the release-tool bot runs again, it will automatically create a
new MR against the fixed master.
The #master-broken channel tracks these failures, and a
workflow exists for resolving them.
Regression in Gitaly
If a regression has occurred, the change can either be reverted or a fix can be attempted. It's generally preferred to
revert as it unblocks other inflight changes from being deployed. Once the revert or fix is merged into gitaly, close
the MR so the release-tools bot can automatically open a new one with the updated GITALY_SERVER_VERSION.
Alternatively, you can choose to manually update the GITALY_SERVER_VERSION in the existing MR and wait for the
pipeline to pass.
Pipeline failures in the security mirror
The gitlab-org/security/gitaly mirror is owned by the
#g_delivery group.
Pipeline failures in the master branch prevent the release-tools bot from creating or amending Update Gitaly version MRs.
If the failure is due to test flakes related to the CI/CD environment (that is, it cannot be addressed by a code change in Gitaly),
reach out to the #g_delivery group so they can re-trigger the pipeline.