Git designed for AI agents.
Declutter the top level of the repository
Federico Eliaac66686cad41
diff --git a/.claude/skills b/.claude/skillsdeleted file mode 120000index 42c5394..0000000--- a/.claude/skills+++ /dev/null@@ -1 +0,0 @@-../skills\ No newline at end of filediff --git a/.codeclimate.yml b/.codeclimate.ymldeleted file mode 100644index ce23235..0000000--- a/.codeclimate.yml+++ /dev/null@@ -1,15 +0,0 @@-----engines:- govet:- enabled: true- golint:- enabled: true- gofmt:- enabled: true- rubocop-gitlab:- enabled: true-ratings:- paths:- - "**.go"-exclude_paths:-- _build/diff --git a/.dockerignore b/.dockerignoredeleted file mode 100644index d1f580d..0000000--- a/.dockerignore+++ /dev/null@@ -1,47 +0,0 @@-# Git-.git-.gitignore-.gitattributes--# Build artifacts (except _build/bin which we need)-_build/cache-_build/deps-_build/licenses-_build/proto*-_build/reports-_build/tools-_build/cover-_build/testrepos-_build/NOTICE-_build/*.gem--# Documentation and examples-doc/-*.md-REVIEWING.md-STYLE.md-LICENSE--# Development files-.vscode/-.idea/-*.swp-*.swo-*~--# Test files-*_test.go-testdata/--# CI/CD-.gitlab-ci.yml-.gitlab/--# Ruby-Gemfile*-.bundle/-vendor/--# Misc-.DS_Store-tmp/diff --git a/.gitlab-ci.yml b/.gitlab-ci.ymldeleted file mode 100644index c4fdd07..0000000--- a/.gitlab-ci.yml+++ /dev/null@@ -1,955 +0,0 @@-stages:- - build- - test- - analyze- - publish- - qa--default:- # This image is generated here: https://gitlab.com/gitlab-org/gitlab-build-images/-/blob/master/.gitlab/ci/gitaly.images.yml- image: registry.gitlab.com/gitlab-org/gitlab-build-images/debian-${DEBIAN_VERSION}-golang-${GO_VERSION}-rust-${RUST_VERSION}:git-${CI_GIT_VERSION}- tags:- - saas-linux-large-amd64- interruptible: true--.versions:- debian: "trixie"- git_default: "default"- go_default: "1.26"- # Follow the following process to upgrade Go version:- # - https://gitlab.com/gitlab-org/gitlab-build-images/-/tree/master#note-regarding-golang-and-fips- # - https://docs.gitlab.com/development/go_guide/go_upgrade/#supporting-multiple-go-versions- # - https://gitlab.com/gitlab-org/gitaly/-/blob/master/doc/go_upgrade_gitaly.md- go_supported: ["1.26", "1.27"]- macos: "macos-26-xcode-26"- pgbouncer: "1.24.0"- postgres_default: "17-alpine"- postgres_minimum: "14-alpine"- ruby: "3.4"- rust: "1.73"- ubi: "9.7"--variables:- FF_USE_FASTZIP: "true"- FF_NETWORK_PER_BUILD: "true"- DEBIAN_VERSION: !reference [.versions, debian]- UBI_VERSION: !reference [.versions, ubi]- CACHE_PREFIX: "debian-${DEBIAN_VERSION}"- CI_GIT_VERSION: "2.51"- # We use Gitaly's Git version by default.- GIT_VERSION: !reference [.versions, git_default]- GO_VERSION: !reference [.versions, go_default]- RUBY_VERSION: !reference [.versions, ruby]- RUST_VERSION: !reference [.versions, rust]- POSTGRES_VERSION: !reference [.versions, postgres_default]- PGBOUNCER_VERSION: !reference [.versions, pgbouncer]- GOCACHE: "${CI_PROJECT_DIR}/_build/cache/go-build"- GOMODCACHE: "${CI_PROJECT_DIR}/_build/cache/go-mod"- SAST_EXCLUDED_PATHS: "spec,test,tests,tmp,_build/**"- # We run the build as an untrusted user in a source directory owned by- # "root". Running Git commands in that repository will thus fail due to- # Git's `safe.directory` protections, and that in turns breaks the Go- # build. We work around this by telling Go to not embed VCS information in- # the binaries.- GOFLAGS: "-buildvcs=false -modcacherw"- # Run tests with an intercepted home directory so that we detect cases where- # Gitaly picks up the gitconfig even though it ought not to.- GITALY_TESTING_INTERCEPT_HOME: "YesPlease"- # TEST_UID is the user ID we use to run tests in an unprivileged way. 9999 is- # chosen as a semi-random value so as to not interfer with any preexisting- # users.- TEST_UID: 9999- RAILS_BRANCH_FOR_JOB: "gitaly-ci-jobs-$CI_PIPELINE_IID"- GITLAB_ADVANCED_SAST_ENABLED: "true"-- # Native clone needs Git >=2.49 and GIT_STRATEGY=clone, and falls back to- # `git init` + `git fetch` when either is missing.- FF_USE_GIT_NATIVE_CLONE: "true"- GIT_STRATEGY: "clone"-- # Packhorse only caches a fetch it can build a key for, which takes a deepen- # between 1 and 30 and no bundle in the way, and the runner turns- # transfer.bundleURI on by default.- GIT_DEPTH: "20"- FF_USE_GIT_BUNDLE_URIS: "false"--include:- - template: Jobs/SAST.latest.gitlab-ci.yml- - template: Security/Dependency-Scanning.gitlab-ci.yml- - template: Security/Secret-Detection.gitlab-ci.yml- - component: ${CI_SERVER_FQDN}/gitlab-org/components/danger-review/[email protected]- inputs:- job_stage: "build"- job_allow_failure: true--workflow:- rules:- - if: $CI_MERGE_REQUEST_IID- - if: $CI_COMMIT_TAG- - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH- - if: '$CI_COMMIT_BRANCH =~ /^[\d-]+-stable$/'- - if: $FORCE_GITALY_CI--.cache_deps:- cache: &cache_deps_configuration- key:- files:- - .gitlab-ci.yml- - Makefile- prefix: ${CACHE_PREFIX}-git-${GIT_VERSION}- paths:- - _build/deps- - _build/tools- policy: pull--.cache_go:- cache: &cache_go_configuration- key:- files:- - .gitlab-ci.yml- - go.sum- prefix: ${CACHE_PREFIX}-go-${GO_VERSION}- paths:- - ${GOCACHE}- - ${GOMODCACHE}- policy: pull--.test_template: &test_definition- needs: []- stage: test- cache:- - *cache_deps_configuration- - *cache_go_configuration- services:- - name: postgres:${POSTGRES_VERSION}- alias: postgres- command: ["postgres", "-c", "max_connections=1000", "-c", "shared_buffers=1GB"]- variables: &test_variables- PGHOST: postgres- PGPORT: 5432- PGUSER: postgres- POSTGRES_DB: praefect_test- POSTGRES_HOST_AUTH_METHOD: trust- TEST_JUNIT_REPORT: "${CI_PROJECT_DIR}/_build/reports/tests-junit.xml"- TEST_COVERAGE_DIR: "${CI_PROJECT_DIR}/_build/reports/coverage"- TEST_COVERAGE_LIST: "${CI_PROJECT_DIR}/_build/reports/coverage-list.txt"- TEST_COVERAGE_REPORT: "${CI_PROJECT_DIR}/_build/reports/coverage-report.html"- TEST_JSON_REPORT: "${CI_PROJECT_DIR}/_build/reports/test-report.json"- TEST_LOG_DIR: "${CI_PROJECT_DIR}/_build/reports/test-logs"- TEST_OPTIONS: "-count=1 -timeout=20m" # a number of tests may exceed the default 10m- TEST_TMP_DIR: "/tmp"- TEST_WITH_COVERAGE: YesPlease- before_script: &test_before_script- # Speed up test execution by writing test data into tmpfs instead of onto- # the underlying disk.- - chmod 1777 "${TEST_TMP_DIR}"- - mount none -t tmpfs -o size=10G "${TEST_TMP_DIR}"- - go version- script:- # Create the build directory for the unprivileged user that we're running- # tests as. This is required because the source directory itself is owned- # by `root`.- - mkdir -p _build- - go mod download -modcacherw- - chown -R ${TEST_UID}:${TEST_UID} _build- # But the actual tests should run unprivileged. This assures that we pay- # proper attention to permission bits and that we don't modify the source- # directory.- - setpriv --reuid=${TEST_UID} --regid=${TEST_UID} --clear-groups --no-new-privs env HOME=/tmp/home make ${TEST_TARGET}- # Run tool tests under a privileged user. These tests sometimes trigger- # a Go toolchain to be downloaded, and that seems to cause permissions issues- # when it's executed under `setpriv` below.- - make test-gitaly-linters- - make test-mod-validator- coverage: /^total:\t+\(statements\)\t+\d+\.\d+%$/- after_script: &test_after_script- - go run "${CI_PROJECT_DIR}/tools/panic-parser"- - |- echo -e "section_start:$(date +%s):coverage[collapsed=true]\r\e[0KCode Coverage"- go tool cover -html "${TEST_COVERAGE_DIR}/all.merged" -o "${TEST_COVERAGE_REPORT}"- go tool cover -func "${TEST_COVERAGE_DIR}/all.merged" -o "${TEST_COVERAGE_LIST}"- echo "Code coverage reports are available at:"- echo "- ${TEST_COVERAGE_REPORT}"- echo "- ${TEST_COVERAGE_LIST}"- echo -e "section_end:$(date +%s):coverage\r\e[0K"-- artifacts: &test_artifacts- paths:- - ${TEST_LOG_DIR}- - ${TEST_JSON_REPORT}- - ${TEST_COVERAGE_REPORT}- - ${TEST_COVERAGE_LIST}- reports:- junit: ${TEST_JUNIT_REPORT}- coverage_report:- coverage_format: cobertura- path: ${TEST_COVERAGE_DIR}/cobertura.xml- when: always- expire_in: 1 week--.rules_run_on_merge: &rules_run_on_merge- # We only execute these rules in the main Gitaly repository. This is- # because jobs which run conditionally when being merged may depend on- # specific runners that aren't available in any other repository.- - if: $CI_PROJECT_PATH != "gitlab-org/gitaly"- when: manual- allow_failure: true- # Automatically run the job when the default branch is changing, e.g. due- # to a direct push.- - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH- # Automatically run the job when a merge request gets queued for merge.- - if: $CI_MERGE_REQUEST_EVENT_TYPE == "merge_train"- # Otherwise, allow for manually scheduling the job in the context of a- # merge request.- - if: $CI_PIPELINE_SOURCE == "merge_request_event"- when: manual- allow_failure: true- # Manual pipeline run outside of merge requests, such as when setting- # custom CI variables via the Pipelines page.- - if: $FORCE_GITALY_CI--.rules_run_strictly_on_default_branch: &rules_run_strictly_on_default_branch- - if: $CI_PROJECT_PATH != "gitlab-org/gitaly"- when: manual- allow_failure: true- - if: $CI_PIPELINE_SOURCE == "merge_request_event"- when: manual- allow_failure: true- - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH- - if: $FORCE_GITALY_CI--build:- needs: []- stage: build- cache:- - <<: *cache_deps_configuration- policy: pull-push- - <<: *cache_go_configuration- policy: pull-push- script:- - go version- # Build the binaries as unprivileged user so that we can reuse the cache- # for our "test" targets, which also run unprivileged.- - mkdir -p _build- - go mod download -modcacherw- - chown -R ${TEST_UID}:${TEST_UID} _build- - setpriv --reuid=${TEST_UID} --regid=${TEST_UID} --clear-groups --no-new-privs env HOME=/tmp/home make build $(test "${GIT_VERSION}" = default && echo build-bundled-git || echo build-git)- - go run ./tools/test-boot ${TEST_BOOT_ARGS}- - go run ./tools/binary-size-checker ${BINARY_SIZE_CHECKER_ARGS}- parallel:- matrix:- - GO_VERSION: !reference [.versions, go_supported]- TEST_BOOT_ARGS: "--bundled-git"- BINARY_SIZE_CHECKER_ARGS: "--binary-size-threshold 375"-build:binaries:- needs: []- stage: build- cache:- - *cache_deps_configuration- - *cache_go_configuration- script:- # Just in case we start running CI builds on other architectures in future- - go version- - make -j$(nproc) build- - cd _build && sha256sum bin/* | tee checksums.sha256.txt- artifacts:- paths:- - _build/checksums.sha256.txt- - _build/bin/- name: "${CI_JOB_NAME}:go-${GO_VERSION}-git-${GIT_VERSION}"- expire_in: 6 months- parallel:- matrix:- - GO_VERSION: !reference [.versions, go_supported]- rules:- - if: $CI_COMMIT_TAG--test:- <<: *test_definition- parallel:- matrix:- # The following jobs all test with our default Git version, which is- # using bundled Git binaries.- - GO_VERSION: !reference [.versions, go_supported]- TEST_TARGET: test- - TEST_TARGET: [test-with-praefect, race-go, test-with-git-master, test-with-git-prev]- # Execute tests with our minimum required Postgres version, as well. If- # the minimum version changes, please change this to the new minimum- # version. Furthermore, please make sure to update the minimum required- # version in `datastore.CheckPostgresVersion()`.- - POSTGRES_VERSION: !reference [.versions, postgres_minimum]- TEST_TARGET: [test, test-with-praefect]--test:pgbouncer:- <<: *test_definition- services:- - name: postgres:${POSTGRES_VERSION}- alias: postgres- command: ["postgres", "-c", "max_connections=500", "-c", "shared_buffers=1GB"]- - name: bitnamilegacy/pgbouncer:${PGBOUNCER_VERSION}- alias: pgbouncer- variables:- <<: *test_variables- # The following variables are used by PgBouncer to connect to Postgres.- POSTGRESQL_HOST: "${PGHOST}"- POSTGRESQL_PORT: "${PGPORT}"- POSTGRESQL_USERNAME: "${PGUSER}"- # These variables define how PgBouncer itself is configured- PGBOUNCER_AUTH_TYPE: trust- PGBOUNCER_AUTH_USER: "${PGUSER}"- PGBOUNCER_DATABASE: "*"- PGBOUNCER_IGNORE_STARTUP_PARAMETERS: extra_float_digits- PGBOUNCER_POOL_MODE: transaction- PGBOUNCER_MAX_CLIENT_CONN: 500- PGBOUNCER_MAX_DB_CONNECTIONS: 100- # And these are finally used by Gitaly's tests.- PGHOST_PGBOUNCER: pgbouncer- PGPORT_PGBOUNCER: 6432- # We need to enable per-build networking such that the PgBouncer service- # can reach Postgres.- TEST_TARGET: "test-with-praefect"--test:reftable:- <<: *test_definition- parallel:- matrix:- - TEST_TARGET: [test]- GITALY_TEST_REF_FORMAT: "reftable"--test:nightly:- <<: *test_definition- allow_failure: true- variables:- <<: *test_variables- OVERRIDE_GIT_VERSION: "v99.99.99"- parallel:- matrix:- - GIT_VERSION: ["master", "next"]- TEST_TARGET: [test, test-with-praefect, test-with-reftable, test-with-sha256]- rules:- - if: '$CI_PIPELINE_SOURCE == "schedule"'- allow_failure: false- - when: manual--test:praefect_smoke:- <<: *test_definition- script:- - make -j$(nproc)- - envsubst < _support/config.praefect.toml.ci-sql-test > config.praefect.toml- - ./_build/bin/praefect -config config.praefect.toml sql-ping- - ./_build/bin/praefect -config config.praefect.toml sql-migrate--test:sha256:- <<: *test_definition- parallel:- matrix:- - TEST_TARGET: [test, test-with-praefect, test-with-reftable]- TEST_WITH_SHA256: "YesPlease"--test:fips:- <<: *test_definition- # This image is generated here: https://gitlab.com/gitlab-org/gitlab-build-images/-/blob/master/.gitlab/ci/gitaly.images.yml- image: registry.gitlab.com/gitlab-org/gitlab-build-images/ubi-${UBI_VERSION}-golang-${GO_VERSION}-rust-${RUST_VERSION}:git-${CI_GIT_VERSION}- tags:- - fips- retry:- max: 1- when: always- cache:- - <<: *cache_deps_configuration- policy: pull-push- - <<: *cache_go_configuration- policy: pull-push- variables:- <<: *test_variables- CACHE_PREFIX: ubi-${UBI_VERSION}- # `-p` caps how many packages' test binaries run at once. It defaults to GOMAXPROCS, which is 32 on the- # FIPS runners. Every concurrent package materializes its own set of Gitaly binaries in ${TEST_TMP_DIR},- # which is a 10GB tmpfs, so 32 of them exhaust it and the tests fail with ENOSPC. 8 matches what the- # non-FIPS runners do implicitly and keeps the peak at roughly 3.3GB.- TEST_OPTIONS: "-count=1 -timeout=30m -p=8" # https://gitlab.com/gitlab-org/gitaly/-/issues/6296- GOTESTSUM_OPTIONS: "--rerun-fails-max-failures=24 --rerun-fails=3"- before_script:- - *test_before_script- # Install minimal pyenv build dependencies- - dnf install -y gcc make patch zlib-devel bzip2-devel openssl-devel libffi-devel- # Install pyenv- - curl https://pyenv.run | bash- - export PYENV_ROOT="$HOME/.pyenv"- - export PATH="$PYENV_ROOT/bin:$PATH"- - eval "$(pyenv init -)"- # Install Python 3.12.13- - pyenv install 3.12.13- - pyenv global 3.12.13- - python3 --version- - test "$(cat /proc/sys/crypto/fips_enabled)" = "1" || (echo "System is not running in FIPS mode" && exit 1)- parallel:- matrix:- - TEST_TARGET: [test, test-with-praefect]- FIPS_MODE: "YesPlease"- GO_VERSION: !reference [.versions, go_supported]- rules:- # Automatically run the job when the FIPS label is set.- - if: $CI_MERGE_REQUEST_LABELS =~ /FIPS/- - *rules_run_on_merge--test:macos:- image: !reference [.versions, macos]- tags:- - saas-macos-large-m2pro- needs: []- stage: test- cache:- - <<: *cache_deps_configuration- policy: pull-push- - <<: *cache_go_configuration- policy: pull-push- variables:- <<: *test_variables- CACHE_PREFIX: !reference [.versions, macos]- PGDATA: /Volumes/RAMDisk/postgres- PGHOST: localhost- PGUSER: gitlab- TEST_TMP_DIR: /Volumes/RAMDisk- # Disable code coverage generation as it is too slow on macOS.- TEST_WITH_COVERAGE: ""- # Homebrew tries to clean up old packages during installation of new packages. The- # runner is ephemeral so we don't need this.- HOMEBREW_NO_INSTALL_CLEANUP: 1- before_script:- # Create a 2GB RAM disk to speed up our tests.- - sudo diskutil apfs create $(hdiutil attach -nomount ram://4194304) RAMDisk- - brew uninstall asdf --force- - rm -rf $HOME/.asdf- - brew install pyenv go@${GO_VERSION} cmake postgresql meson rust- - pyenv install 3.12.13- - pyenv global 3.12.13- - export PATH="$HOME/.pyenv/versions/3.12.13/bin:$PATH"- # Older Go versions may be keg-only and thus aren't linked into `PATH` by- # default. We need to override this mechanism to force this specific Go- # version to become active.- - brew link --force go@${GO_VERSION}- - go version- - python3 --version- - sw_vers- - pg_ctl --version- - ulimit -n 4096 # The default of 256 can lead to 'too many open files' errors when running parallel tests- - pg_ctl init- - pg_ctl start- - pg_ctl status- - createdb -U $PGUSER $POSTGRES_DB- - make -j$(nproc) build prepare-tests- script:- - make test-go ; EXIT_CODE=$? ; pg_ctl stop ; exit $EXIT_CODE- after_script:- - *test_after_script- rules:- - *rules_run_on_merge- artifacts:- <<: *test_artifacts--logging-field-validator:- needs: []- stage: test- cache:- - *cache_go_configuration- script:- - make validate-log-fields- rules:- - if: $CI_MERGE_REQUEST_PROJECT_PATH == 'gitlab-org/gitaly' && $CI_PIPELINE_SOURCE == "merge_request_event" && $CI_MERGE_REQUEST_EVENT_TYPE != "merge_train"--verify:docs:- image: registry.gitlab.com/gitlab-org/technical-writing/docs-gitlab-com/lint-markdown:alpine-3.24-vale-3.21.0-markdownlint2-0.23.2-lychee-0.24.2-rumdl-0.2.69- needs: []- stage: analyze- script:- - make lint-docs--verify:- needs: []- stage: analyze- cache:- - *cache_deps_configuration- - *cache_go_configuration- script:- # Download the dependencies in case there was no cache hit, otherwise- # golang-ci lint will timeout downloading them.- - go mod download- - make -j$(nproc) verify- artifacts:- paths:- - _build/proto.diff- - proto/go/gitalypb/*- when: on_failure--vulnerability:- needs: []- stage: analyze- cache:- - *cache_deps_configuration- - *cache_go_configuration- script:- - make govulncheck- rules:- - *rules_run_strictly_on_default_branch--dbschema:- needs: []- stage: analyze- cache:- - *cache_deps_configuration- - *cache_go_configuration- services:- # The database version we use must match the version of `pg_dump` we have- # available in the build image.- - postgres:11.13-alpine- variables:- <<: *test_variables- script:- - make dump-database-schema no-changes- artifacts:- paths:- - _support/praefect-schema.sql- when: on_failure--semgrep-sast:- needs: []- stage: analyze- cache:- - *cache_go_configuration--gitlab-advanced-sast:- timeout: 20 minutes- needs: []- stage: analyze- cache:- - *cache_go_configuration--secret_detection:- needs: []- stage: analyze- rules:- - if: $SECRET_DETECTION_DISABLED- when: never--build-proto-gem:- needs: []- image: ${GITLAB_DEPENDENCY_PROXY}ruby:${RUBY_VERSION}- stage: publish- before_script: []- variables:- BUILD_GEM_OPTIONS: --skip-verify-tag- script:- - make build-proto-gem- rules:- - when: manual- allow_failure: true- artifacts:- paths:- - _build/gitaly.gem- expire_in: 1 month--publish-proto-gem:- needs: []- image: ${GITLAB_DEPENDENCY_PROXY}ruby:${RUBY_VERSION}- stage: publish- before_script: []- script:- - make publish-proto-gem- rules:- - if: '$CI_SERVER_HOST == "gitlab.com" && $CI_PROJECT_NAMESPACE == "gitlab-org" && $CI_COMMIT_TAG'--pages:- needs: []- stage: publish- allow_failure: true- script:- - make build-proto-docs- - mv _build/proto-docs public- artifacts:- paths:- - public- rules:- - *rules_run_on_merge--start-rails-specs:- needs: []- stage: qa- script:- - GITLAB_PROJECT_ID="$([ "$CI_MERGE_REQUEST_PROJECT_PATH" == 'gitlab-org/security/gitaly' ] && echo 15642544 || echo 278964)"- - |- curl --fail --request POST --header "PRIVATE-TOKEN: $GITLAB_PROJECT_TOKEN" \- "https://gitlab.com/api/v4/projects/$GITLAB_PROJECT_ID/repository/branches?ref=master&branch=$RAILS_BRANCH_FOR_JOB"- - |- curl --fail --request PUT --header "PRIVATE-TOKEN: $GITLAB_PROJECT_TOKEN" \- --data "branch=$RAILS_BRANCH_FOR_JOB" \- --data "content=$CI_COMMIT_SHA" \- --data "commit_message=Updating GITALY_SERVER_VERSION to $CI_COMMIT_SHA" \- "https://gitlab.com/api/v4/projects/$GITLAB_PROJECT_ID/repository/files/GITALY_SERVER_VERSION"- environment:- name: merge-requests/$RAILS_BRANCH_FOR_JOB- on_stop: rails-specs-cleanup- rules:- - if: $CI_PIPELINE_SOURCE == "merge_request_event" && $CI_MERGE_REQUEST_EVENT_TYPE != "merge_train"- changes:- - .tool-versions- - if: $CI_MERGE_REQUEST_LABELS =~ /run-rails-specs/ && $CI_PIPELINE_SOURCE == "merge_request_event" && $CI_MERGE_REQUEST_EVENT_TYPE != "merge_train"- - if: $CI_PIPELINE_SOURCE == "merge_request_event" && $CI_MERGE_REQUEST_EVENT_TYPE != "merge_train"- when: manual- allow_failure: true--.rails-specs-common: &rails-specs-common- stage: qa- needs: ["start-rails-specs"]- trigger: &rails-specs-trigger- branch: "gitaly-ci-jobs-$CI_PIPELINE_IID" # Using $RAILS_BRANCH_FOR_JOB directly fails- strategy: depend- inherit:- variables: false # Otherwise variables will be messed up- variables: &rails-specs-vars- GITALY_TEST: "true"- ENABLE_RSPEC: "true"- ENABLE_RSPEC_UNIT: "true"- ENABLE_RSPEC_INTEGRATION: "true"- ENABLE_RSPEC_SYSTEM: "true"--rails-specs:- <<: *rails-specs-common- trigger:- <<: *rails-specs-trigger- project: gitlab-org/gitlab- variables:- <<: *rails-specs-vars- rules:- # pipeline:run-gitaly-mvcc also runs the gitaly-mvcc variants downstream- - if: $CI_MERGE_REQUEST_PROJECT_PATH == 'gitlab-org/gitaly' && $CI_PIPELINE_SOURCE == "merge_request_event" && $CI_MERGE_REQUEST_EVENT_TYPE != "merge_train" && $CI_MERGE_REQUEST_LABELS =~ /pipeline:run-gitaly-mvcc/- variables:- ENABLE_RSPEC_GITALY_MVCC: "true"- - if: $CI_MERGE_REQUEST_PROJECT_PATH == 'gitlab-org/gitaly' && $CI_PIPELINE_SOURCE == "merge_request_event" && $CI_MERGE_REQUEST_EVENT_TYPE != "merge_train"--rails-specs-security:- <<: *rails-specs-common- trigger:- <<: *rails-specs-trigger- project: gitlab-org/security/gitlab- variables:- <<: *rails-specs-vars- GITALY_REPO_URL: "https://gitlab-bot:[email protected]/gitlab-org/security/gitaly.git"- rules:- - if: $CI_MERGE_REQUEST_PROJECT_PATH == 'gitlab-org/security/gitaly' && $CI_PIPELINE_SOURCE == "merge_request_event" && $CI_MERGE_REQUEST_EVENT_TYPE != "merge_train"--.rails-specs-cleanup-common: &rails-specs-cleanup-common- stage: qa- script:- - GITLAB_PROJECT_ID="$([ "$CI_MERGE_REQUEST_PROJECT_PATH" == 'gitlab-org/security/gitaly' ] && echo 15642544 || echo 278964)"- - |- curl --fail --request DELETE --header "PRIVATE-TOKEN: $GITLAB_PROJECT_TOKEN" \- "https://gitlab.com/api/v4/projects/$GITLAB_PROJECT_ID/repository/branches/$RAILS_BRANCH_FOR_JOB"- environment:- name: merge-requests/$RAILS_BRANCH_FOR_JOB- action: stop--rails-specs-cleanup:- <<: *rails-specs-cleanup-common- needs: ["rails-specs"]- rules:- - if: $CI_MERGE_REQUEST_PROJECT_PATH == 'gitlab-org/gitaly' && $CI_PIPELINE_SOURCE == "merge_request_event" && $CI_MERGE_REQUEST_EVENT_TYPE != "merge_train"- when: manual- allow_failure: true--rails-specs-security-cleanup:- <<: *rails-specs-cleanup-common- needs: ["rails-specs-security"]- rules:- - if: $CI_MERGE_REQUEST_PROJECT_PATH == 'gitlab-org/security/gitaly' && $CI_PIPELINE_SOURCE == "merge_request_event" && $CI_MERGE_REQUEST_EVENT_TYPE != "merge_train"- when: manual- allow_failure: true--.cleanup-rails-spec-test-env-common: &cleanup-rails-spec-test-env-common- stage: qa- before_script:- - apt update- - apt install -y jq- script:- - GITLAB_PROJECT_ID="$([ "$CI_MERGE_REQUEST_PROJECT_PATH" == 'gitlab-org/security/gitaly' ] && echo 15642544 || echo 278964)"- - |- ENV_ID=$(curl -s --request GET --header "PRIVATE-TOKEN: $GITALY_PROJECT_TOKEN" "https://gitlab.com/api/v4/projects/$GITLAB_PROJECT_ID/environments" --data-urlencode "name=merge-requests/$RAILS_BRANCH_FOR_JOB" | jq '.[0].id')- curl --fail -s --request DELETE --header "PRIVATE-TOKEN: $GITALY_PROJECT_TOKEN" "https://gitlab.com/api/v4/projects/$GITLAB_PROJECT_ID/environments/$ENV_ID"--cleanup-rails-spec-test-env:- <<: *cleanup-rails-spec-test-env-common- needs: ["rails-specs-cleanup"]- rules:- - if: $CI_MERGE_REQUEST_PROJECT_PATH == 'gitlab-org/gitaly' && $CI_PIPELINE_SOURCE == "merge_request_event" && $CI_MERGE_REQUEST_EVENT_TYPE != "merge_train"- allow_failure: true--cleanup-rails-spec-security-test-env:- <<: *cleanup-rails-spec-test-env-common- needs: ["rails-specs-security-cleanup"]- rules:- - if: $CI_MERGE_REQUEST_PROJECT_PATH == 'gitlab-org/security/gitaly' && $CI_PIPELINE_SOURCE == "merge_request_event" && $CI_MERGE_REQUEST_EVENT_TYPE != "merge_train"- allow_failure: true--build-package-and-qa:- needs: []- stage: qa- trigger:- project: gitlab-org/build/omnibus-gitlab-mirror- strategy: depend- inherit:- variables: false- variables:- ALTERNATIVE_SOURCES: "true"- GITALY_SERVER_VERSION: $CI_COMMIT_SHA- GITALY_SERVER_ALTERNATIVE_REPO: $CI_PROJECT_URL- ee: "true"- rules:- - if: $CI_MERGE_REQUEST_IID- changes:- - go.sum- variables:- BUILD_ON_ALL_OS: "true"- MANUAL_QA_TEST: "true"- allow_failure: false- - when: manual- allow_failure: true--build-cng-image:- needs: []- stage: qa- trigger:- project: gitlab-org/build/CNG-mirror- strategy: depend- inherit:- variables: false- variables:- GITALY_SERVER_VERSION: $CI_COMMIT_SHA- GITLAB_NAMESPACE: $CI_PROJECT_NAMESPACE- ee: "true"- rules:- - if: $CI_MERGE_REQUEST_IID- when: manual- allow_failure: true--qa:nightly-praefect-migration-test:- needs: []- stage: qa- trigger:- project: gitlab-org/quality/praefect-migration-testing- rules:- - if: '$CI_PIPELINE_SOURCE == "schedule"'--qa:benchmark:performance:- stage: qa- image: google/cloud-sdk:slim- needs: []- environment:- name: benchmark- action: prepare- variables:- BENCHMARK_INSTANCE_NAME: "ci-${CI_PIPELINE_ID}"- BENCHMARK_DURATION: "2m"- GCP_SSH_KEY_NAME: "gitaly-bench"- EXPERIMENT: $CI_COMMIT_REF_SLUG- before_script:- # download secure files to default location .secure_files- - curl --silent "https://gitlab.com/gitlab-org/incubation-engineering/mobile-devops/download-secure-files/-/raw/main/installer" | bash- # auth with gcp project- - export CI_SSH_PRIVATE_KEY_FILE="$CI_PROJECT_DIR/.secure_files/$GCP_SSH_KEY_NAME"- - export CI_SSH_PUBKEY_FILE="$CI_PROJECT_DIR/.secure_files/$GCP_SSH_KEY_NAME.pub"- - gcloud auth activate-service-account --key-file=".secure_files/$GCP_BENCHMARKING_SA_KEY_FILE" --quiet- - gcloud config set project $GCP_BENCHMARKING_PROJECT_NAME --quiet- # Install Terraform from HashiCorp repository- # Download the key- - curl -fsSL https://apt.releases.hashicorp.com/gpg | gpg --dearmor -o /usr/share/keyrings/hashicorp-archive-keyring.gpg- # Add the repository with signed-by option- - echo "deb [signed-by=/usr/share/keyrings/hashicorp-archive-keyring.gpg] https://apt.releases.hashicorp.com $(lsb_release -cs) main" | tee /etc/apt/sources.list.d/hashicorp.list- - apt-get update && apt-get install -y python3 python3-pip ansible unzip openssh-client terraform- - terraform version- # Setup SSH agent and add the key- - eval $(ssh-agent -s)- - chmod 600 "$CI_SSH_PRIVATE_KEY_FILE"- - ssh-add "$CI_SSH_PRIVATE_KEY_FILE"- - mkdir -p ~/.ssh- - echo "StrictHostKeyChecking no" >> ~/.ssh/config- script:- - cd _support/benchmarking- - export EXPERIMENT="${EXPERIMENT}"- - ./create-benchmark-instance- - ./configure-benchmark-instance- - ./run-benchmarks --extra-vars "duration=${BENCHMARK_DURATION}"- after_script:- - cd _support/benchmarking- - ./destroy-benchmark-instance- - cd ../..- - rm -rf .secure_files- artifacts:- paths:- - "_support/benchmarking/experiments/${EXPERIMENT}/results/"- expire_in: 4 week- rules:- - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH- when: manual- allow_failure: true- - if: $CI_MERGE_REQUEST_IID- when: manual- allow_failure: true--qa:benchmark:performance:incluster:- stage: qa- image: gcr.io/google.com/cloudsdktool/google-cloud-cli:debian_component_based- needs: []- resource_group: gitaly-benchmark-cluster- interruptible: false- environment:- name: benchmark- action: prepare- variables:- # EXPERIMENT can be overridden at job-trigger time via the manual job- # variable form. If unset, the before_script auto-detects from changed- # files under _support/benchmarking/experiments/, falling back to- # incluster-master when nothing experiment-specific changed.- EXPERIMENT: ""- DEFAULT_EXPERIMENT: "incluster-master"- CLEAR_GITALY_CACHE: "true"- CLEAR_PAGE_CACHE: "true"- # Pinned static-binary version installed in before_script.- GLAB_VERSION: "1.75.0"- before_script:- # kubectl and the GKE auth plugin are pre-installed in this image; this is- # a fast no-op that guards against them dropping out of the base image- # gke-gcloud-auth-plugin is required to use USE_GKE_GCLOUD_AUTH_PLUGIN- - gcloud components install kubectl gke-gcloud-auth-plugin --quiet- # PyYAML for run-benchmarks-incluster's manifest rendering (report.py is- # stdlib-only).- - apt-get update && apt-get install -y --no-install-recommends python3-yaml- # glab is a single static Go binary; drop it in from its pinned release- # tarball.- - curl -fsSL "https://gitlab.com/gitlab-org/cli/-/releases/v${GLAB_VERSION}/downloads/glab_${GLAB_VERSION}_linux_amd64.tar.gz" | tar -xz -C /tmp bin/glab && install /tmp/bin/glab /usr/local/bin/glab- - glab --version- # 1. Pull the GCP service-account key from secure files (same key the VM- # benchmark job uses). The SA needs container.developer on the cluster- # in addition to its existing Compute / Storage permissions.- - curl --silent "https://gitlab.com/gitlab-org/incubation-engineering/mobile-devops/download-secure-files/-/raw/main/installer" | bash- - gcloud auth activate-service-account --key-file=".secure_files/$GCP_BENCHMARKING_SA_KEY_FILE" --quiet- - gcloud config set project "$GCP_BENCHMARKING_PROJECT_NAME" --quiet- # 2. Fetch GKE credentials so `kubectl` in run-benchmarks-incluster points- # at the right cluster. Uses Application Default Credentials, not the- # deprecated gcloud auth plugin.- - export USE_GKE_GCLOUD_AUTH_PLUGIN=True- - gcloud container clusters get-credentials "$GKE_CLUSTER_NAME" --region "$GKE_CLUSTER_REGION" --project "$GCP_BENCHMARKING_PROJECT_NAME"- - kubectl version --client- # 3. Resolve EXPERIMENT (explicit override > auto-detect > default experiment).- - |- set -eu- if [ -n "$EXPERIMENT" ]; then- echo "[ci] EXPERIMENT explicitly set to '$EXPERIMENT' (override)"- else- # Auto-detect from changed files. In MR pipelines GitLab exposes the- # merge-base SHA directly (CI_MERGE_REQUEST_DIFF_BASE_SHA); use it- # rather than a remote-tracking ref like origin/<branch>, which a- # shallow CI clone does not populate. Outside MRs, fall back to the- # previous commit. Detection is best-effort: any git failure (e.g. the- # base commit is outside the shallow history) degrades to- # DEFAULT_EXPERIMENT instead of aborting the job.- if [ -n "${CI_MERGE_REQUEST_DIFF_BASE_SHA:-}" ]; then- BASE="$CI_MERGE_REQUEST_DIFF_BASE_SHA"- git fetch --depth=1 origin "$BASE" 2>/dev/null || true- else- BASE="HEAD~1"- fi-- CHANGED_EXPS=$(git diff --name-only "$BASE" HEAD -- \- '_support/benchmarking/experiments/' 2>/dev/null \- | awk -F/ 'NF>=4 {print $4}' | sort -u || true)-- COUNT=$(echo -n "$CHANGED_EXPS" | grep -c '^' || true)-- if [ "$COUNT" -eq 0 ]; then- echo "[ci] No experiment dir changes (or base unavailable); using default: $DEFAULT_EXPERIMENT"- EXPERIMENT="$DEFAULT_EXPERIMENT"- elif [ "$COUNT" -eq 1 ]; then- EXPERIMENT="$CHANGED_EXPS"- echo "[ci] Auto-selected EXPERIMENT='$EXPERIMENT' from changed files"- else- echo "[ci] ERROR: multiple experiments changed:"- echo "$CHANGED_EXPS" | sed 's/^/ - /'- echo ""- echo "Only one experiment can run per job. Re-trigger this manual"- echo "job with EXPERIMENT=<name> set in the variables form to pick one."- exit 1- fi- fi- export EXPERIMENT- echo "EXPERIMENT=$EXPERIMENT" > experiment.env- script:- - . ./experiment.env- - _support/benchmarking/run-benchmarks-incluster- # Render the run's k6 summary into a Markdown report. Picks the most recent- # results subdirectory (run-benchmarks-incluster creates one per invocation).- # Non-fatal: a report failure shouldn't fail the whole benchmark job.- - |- set -eu- RESULTS_DIR="_support/benchmarking/experiments/${EXPERIMENT}/results"- RUN_DIR=$(ls -1dt "$RESULTS_DIR"/*/ 2>/dev/null | head -n1 || true)- if [ -z "$RUN_DIR" ] || [ ! -f "${RUN_DIR}k6-summary.json" ]; then- echo "[ci] WARN: no k6-summary.json found under $RESULTS_DIR; skipping report"- else- RUN_NAME=$(basename "$RUN_DIR")- ARTIFACT_BASE="${CI_JOB_URL}/artifacts/raw/${RUN_DIR%/}"- python3 _support/benchmarking/report.py "${RUN_DIR}k6-summary.json" \- --output "${RUN_DIR}summary.md" \- --metrics-output _support/benchmarking/metrics.txt \- --artifact-base-url "$ARTIFACT_BASE" \- --experiment "$EXPERIMENT" \- --run-name "$RUN_NAME" \- || echo "[ci] WARN: report.py failed (non-fatal)"- fi- # Post the rendered report as an MR note via glab. Append-only: every- # benchmark run leaves a new comment so the discussion preserves history- # across re-runs. Non-fatal: failure to post shouldn't fail the benchmark- # job itself - the rendered summary.md is still in artifacts.- - |- set -eu- if [ -z "${CI_MERGE_REQUEST_IID:-}" ]; then- echo "[ci] not an MR pipeline; skipping MR note"- elif [ ! -f "${RUN_DIR}summary.md" ]; then- echo "[ci] no summary.md to post; skipping MR note"- elif [ -z "${GLAB_BENCHMARKING:-}" ]; then- echo "[ci] WARN: GLAB_BENCHMARKING unset; skipping MR note (non-fatal)."- echo "[ci] Set an api-scoped project access token to enable it."- else- if GITLAB_TOKEN="$GLAB_BENCHMARKING" glab mr note "$CI_MERGE_REQUEST_IID" \- --repo "$CI_PROJECT_PATH" \- --message "$(cat "${RUN_DIR}summary.md")"; then- echo "[ci] posted benchmark report as MR note"- else- echo "[ci] WARN: glab mr note failed (non-fatal)."- echo "[ci] Check GLAB_BENCHMARKING is valid, unexpired, and"- echo "[ci] has api scope with Developer+ role on this project."- fi- fi- after_script:- - rm -rf .secure_files- artifacts:- reports:- metrics: _support/benchmarking/metrics.txt- paths:- - "_support/benchmarking/experiments/*/results/"- exclude:- # run-benchmarks-incluster also tars up results/<run>/. Keep the unzipped- # tree (its plots/pprof are linked from the report) and drop the- # redundant tarball so the download isn't every file twice.- - "_support/benchmarking/experiments/*/results/*.tar.gz"- when: always- expire_in: 30 days- rules:- # Block fork pipelines outright- - if: $CI_PROJECT_PATH != "gitlab-org/gitaly"- when: never- - if: $CI_MERGE_REQUEST_IID- when: manual- allow_failure: true- - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH- when: manual- allow_failure: truediff --git a/.gitlab/CODEOWNERS b/.gitlab/CODEOWNERSdeleted file mode 100644index ed0b904..0000000--- a/.gitlab/CODEOWNERS+++ /dev/null@@ -1,10 +0,0 @@-# When adding a group as a code owner, make sure to invite the group to the-# project here: https://gitlab.com/gitlab-org/gitaly/-/project_members-# As described in https://docs.gitlab.com/user/project/codeowners/--[Code]-* @gl-gitaly--^[Documentation]-/doc/ @eread-*.md @ereaddiff --git a/.gitlab/caproni/build-gitaly-helpers.sh b/.gitlab/caproni/build-gitaly-helpers.shdeleted file mode 100755index 925ba67..0000000--- a/.gitlab/caproni/build-gitaly-helpers.sh+++ /dev/null@@ -1,37 +0,0 @@-#!/usr/bin/env bash-# Rebuild the helper binaries for Linux when their source changes. They get-# embedded into the gitaly binary, so this runs before every ko build and-# from skaffold-prepare.sh.--set -euo pipefail--REPO_ROOT=$(cd "$(dirname "${BASH_SOURCE[0]}")/../.." && pwd)-cd "${REPO_ROOT}"--goarch=$(go env GOARCH)--aux=(gitaly-hooks gitaly-gpg gitaly-ssh gitaly-lfs-smudge)--matches_arch() {- case "${goarch}" in- amd64) grep -q 'x86-64' ;;- arm64) grep -q 'aarch64' ;;- esac-}--needs_rebuild() {- local bin="$1" pkg="$2"- if [ ! -x "${bin}" ] || file "${bin}" | grep -q Mach-O || ! file "${bin}" | matches_arch; then- return 0- fi- [ -n "$(find "cmd/${pkg}" internal proto/go -name '*.go' -newer "${bin}" -print -quit)" ]-}--mkdir -p _build/bin--for b in "${aux[@]}"; do- if needs_rebuild "_build/bin/${b}" "${b}"; then- echo "building aux binary ${b} for linux/${goarch}"- GOOS=linux GOARCH="${goarch}" go build -o "_build/bin/${b}" "./cmd/${b}"- fi-donediff --git a/.gitlab/caproni/import-gitaly-images.sh b/.gitlab/caproni/import-gitaly-images.shdeleted file mode 100755index f1f9b6d..0000000--- a/.gitlab/caproni/import-gitaly-images.sh+++ /dev/null@@ -1,60 +0,0 @@-#!/usr/bin/env bash-# Import local gitaly images into cluster containerd so the pod can-# pull the tag skaffold deployed.--set -euo pipefail--k3d_node=$(docker ps --format '{{.Names}}' 2>/dev/null | grep '^k3d-.*-server-0$' | head -1 || true)-if [ -n "${k3d_node}" ]; then- ctr_ls() { docker exec "${k3d_node}" ctr -n k8s.io images ls -q; }- ctr_import() { docker exec -i "${k3d_node}" ctr -n k8s.io images import -; }-elif colima status --profile caproni >/dev/null 2>&1; then- ctr_ls() { colima ssh --profile caproni -- sudo ctr -n k8s.io images ls -q; }- ctr_import() { colima ssh --profile caproni -- sudo ctr -n k8s.io images import -; }-else- echo "error: no k3d node or colima caproni profile found. Is the cluster up (\`caproni up\`)?" >&2- exit 1-fi--if ! existing=$(ctr_ls 2>&1); then- echo "error: could not reach the cluster containerd. Is the cluster up (\`caproni up\`)?" >&2- exit 1-fi--num=0-for img in $(docker images --format '{{.ID}}' registry.gitlab.com/gitlab-org/build/cng/gitaly | sort -u); do- if ! tags=$(docker image inspect --format '{{range .RepoTags}}{{println .}}{{end}}' "$img" 2>/dev/null); then- continue # skip this image since it may have already been retagged or removed- fi-- skip=false- while IFS= read -r tag; do- [ -z "${tag}" ] && continue- tag_name=${tag##*:}- # Skip version tags since the same name can be reused for a new build.- [[ "${tag_name}" != *[!0-9a-f]* && ${#tag_name} -eq 64 ]] || continue- if grep -Fqx "${tag}" <<< "${existing}"; then- echo "already imported: ${tag}"- skip=true- break- fi- done <<< "${tags}"-- if [ "${skip}" = true ]; then- continue- fi-- # Save by tag since that's what the pod actually pulls by.- tag_array=()- while IFS= read -r t; do- [ -z "${t}" ] && continue- tag_array+=("${t}")- done <<< "${tags}"- if docker save "${tag_array[@]}" | ctr_import; then- num=$((num + 1))- fi-done--if [ "${num}" -eq 0 ]; then- echo "no local gitaly images to import"-fidiff --git a/.gitlab/caproni/skaffold-prepare.sh b/.gitlab/caproni/skaffold-prepare.shdeleted file mode 100755index 8be974b..0000000--- a/.gitlab/caproni/skaffold-prepare.sh+++ /dev/null@@ -1,87 +0,0 @@-#!/usr/bin/env bash-# Prepare hook (edit_mode_start): build gitaly and import into cluster containerd.--set -euo pipefail--REPO_ROOT=$(cd "$(dirname "${BASH_SOURCE[0]}")/../.." && pwd)--# Gitaly stores Git data on disk, so it runs in a StatefulSet. We generate it-# from the Helm chart here instead of committing one that drifts.-render_edit_mode_statefulset() {- local out err- out=$(mktemp)- err=$(mktemp)- trap 'rm -f "${out}" "${err}"' RETURN-- if ! caproni tool helm get manifest gitlab --namespace gitlab 2>"${err}" | yq -oy eval '- select(.kind == "StatefulSet" and .metadata.name == "gitlab-gitaly") |- .spec.template.spec.containers[0].command = ["/ko-app/gitaly", "serve", "/etc/gitaly/config.toml"] |- .spec.template.spec.containers[0].volumeMounts |= with(.[]; select(.name == "gitaly-config") | .mountPath = "/etc/gitaly")- ' - > "${out}" || [ ! -s "${out}" ]; then- echo "error: no gitlab-gitaly StatefulSet found. Is the cluster up (\`caproni up\`)?" >&2- if [ -s "${err}" ]; then- cat "${err}" >&2- fi- rm -f "${out}" "${err}"- exit 1- fi-- chmod 644 "${out}"- mv "${out}" "${REPO_ROOT}/k8s/statefulset.yaml"- echo "wrote ${REPO_ROOT}/k8s/statefulset.yaml"-}--render_edit_mode_statefulset-unset -f render_edit_mode_statefulset--case "$(uname -m)" in- x86_64|amd64)- goarch=amd64- ;;- *)- goarch=arm64- ;;-esac--"$(dirname "$0")/build-gitaly-helpers.sh"--match_arch() {- case "${goarch}" in- amd64) grep -q 'x86-64' ;;- arm64) grep -q 'aarch64' ;;- esac-}--# Git binaries are embedded into the gitaly image, so they must be Linux.-git_execs=(git git-http-backend git-remote-http)-git_versions=(mvcc master prev)--need_git_build=0-for e in "${git_execs[@]}"; do- for v in "${git_versions[@]}"; do- f="_build/bin/gitaly-${e}-${v}"- if [ ! -x "${f}" ] || file "${f}" | grep -q Mach-O || ! file "${f}" | match_arch; then- need_git_build=1- break 2- fi- done-done--if [ "${need_git_build}" = 1 ]; then- echo "building linux/${goarch} bundled git binaries"- docker build -q -t ko.local/gitaly-bundled-git-builder - < Dockerfile.bundled-git >/dev/null- docker run --rm -v "$(pwd):/src" -w /src ko.local/gitaly-bundled-git-builder make build-bundled-git-fi--if docker image inspect ko.local/gitaly-base:latest &>/dev/null \- || ! docker images --format '{{.Repository}}:{{.Tag}}' | grep -q '^ko.local/gitaly-base:'; then- :-else- tag=$(docker images --format '{{.Repository}}:{{.Tag}}' | grep '^ko.local/gitaly-base:' | head -1)- docker tag "${tag}" ko.local/gitaly-base:latest-fi--build_json="/tmp/gitaly-skaffold-build.json"-skaffold build --file-output="${build_json}" --platform "linux/${goarch}"--"$(dirname "$0")/import-gitaly-images.sh"diff --git a/.gitlab/changelog_config.yml b/.gitlab/changelog_config.ymldeleted file mode 100644index a2fe0d4..0000000--- a/.gitlab/changelog_config.yml+++ /dev/null@@ -1,38 +0,0 @@-----# Settings for generating changelogs using the GitLab API. See-# https://docs.gitlab.com/api/repositories/#generate-changelog-data for-# more information.-categories:- added: Added- fixed: Fixed- changed: Changed- deprecated: Deprecated- removed: Removed- security: Security- performance: Performance- other: Other-template: |- {% if categories %}- {% each categories %}- ### {{ title }} ({% if single_change %}1 change{% else %}{{ count }} changes{% end %})-- {% each entries %}- - [{{ title }}]({{ commit.reference }})\- {% if author.contributor %} by {{ author.reference }}{% end %}\- {% if commit.trailers.MR %}\- ([merge request]({{ commit.trailers.MR }}))\- {% else %}\- {% if merge_request %}\- ([merge request]({{ merge_request.reference }}))\- {% end %}\- {% end %}\- {% if commit.trailers.EE %}\- **GitLab Enterprise Edition**\- {% end %}-- {% end %}-- {% end %}- {% else %}- No changes.- {% end %}diff --git a/.gitlab/duo/mr-review-instructions.yaml b/.gitlab/duo/mr-review-instructions.yamldeleted file mode 100644index 736076f..0000000--- a/.gitlab/duo/mr-review-instructions.yaml+++ /dev/null@@ -1,65 +0,0 @@-----# Custom instructions for GitLab Duo Code Review-# Based on GitLab's official code review guidelines--# References:-# - Field Standardisation in Observability: https://handbook.gitlab.com/handbook/engineering/architecture/design-documents/observability_field_standardisation/---# This file defines custom review criteria that will be applied to specific files-# during merge request reviews. Instructions are grouped by name and can target-# multiple file patterns using glob syntax.-instructions:- - name: Log Field Standards- fileFilters:- - "**/*.go"- - "!**/*_test.go"- instructions: |- Backend engineers should be complying with the new field standardisation in observability best practices- 1. For any log lines that have been altered:- - Ask: "If you are adding or modifying fields that aren't service specific, please ensure that the field is defined within the LabKit Go Fields package"- - Remind: "Common logging fields should be defined within the LabKit fields package and imported from there if they aren't service specific. Link: https://gitlab.com/gitlab-org/labkit/-/tree/master/fields?ref_type=heads"- 2. For any new fields being added to log messages:- - Check to ensure that these fields are not dynamically generated- - Remind: "We're aiming to standardise the fields that we emit across all of our services at GitLab through this approach. Read more about this in https://handbook.gitlab.com/handbook/engineering/architecture/design-documents/observability_field_standardisation/"-- - name: Unbounded log size- fileFilters:- - "**/*.go"- - "!**/*_test.go"- instructions: |- Gitaly is a long-running server process that produces structured logs. In- production, these logs are typically in JSON format and will potentially- be ingested by log aggregation systems like fluentd.-- Gitaly's logging can be quite verbose, especially for new features which- we want enhanced observability on. Whenever logging fields are added or- modified, check:-- - Whether the new field can experience unbounded growth due to how the- value of the field is being computed. For example, is the source of the- value a slice or a map that only grows in size?- - Whether the cardinality of field names is likely to be high. For- example, field names should not contain arbitrary values like PIDs or- other IDs.-- - name: Dependency updates- fileFilters:- - "**/go.mod"- - "**/go.sum"- instructions: |- Updates to Go packages that are used by Gitaly can have subtle but far-reaching consequences. For- example, a minor version bump to the grpc-go package in https://gitlab.com/gitlab-org/gitaly/-/merge_requests/7531- resulted in significant performance degradation due to a bug and a behavioural change.-- Therefore, it is crucial that dependencies on the critical path of Gitaly requests are thoroughly- vetted for correctness and performance. This can be challenging to do for every dependency update.-- Provided that this MR is a dependency update (check the diff in the go.mod files), then assess the- updated dependency for any possible regressions. Scrutinise the upstream sources and inspect how- the dependency is being used by Gitaly.-- Your report should include a score from 1 to 3 representing the risk that the dependency update- introduces. 1 is high risk, 2 is indeterminate, and 3 is low risk. This provides an indication to- human reviewers on how much time is required to be spent vetting the update.-diff --git a/.gitlab/issue_templates/Config Change.md b/.gitlab/issue_templates/Config Change.mddeleted file mode 100644index be255ca..0000000--- a/.gitlab/issue_templates/Config Change.md+++ /dev/null@@ -1,13 +0,0 @@-# Gitaly/Praefect configuration change checklist--There are a number of projects which depend on Gitaly's or Praefect's configuration. When you change the configuration, please have a look whether the change needs to be propagated to any of the listed projects. If deprecating or removing a configuration, consult the [configuration deprecation policy](https://docs.gitlab.com/omnibus/package-information/deprecation_policy.html#deprecating-configuration).--1. [ ] Update the example configuration files for [Gitaly](https://gitlab.com/gitlab-org/gitaly/-/blob/master/config.toml.example) and [Praefect](https://gitlab.com/gitlab-org/gitaly/-/blob/master/config.praefect.toml.example)-1. [ ] Update in [omnibus-gitlab](https://gitlab.com/gitlab-org/omnibus-gitlab) the configuration templates for [Gitaly](https://gitlab.com/gitlab-org/omnibus-gitlab/-/blob/master/files/gitlab-cookbooks/gitaly/templates/default/gitaly-config.toml.erb) and [Praefect](https://gitlab.com/gitlab-org/omnibus-gitlab/-/blob/master/files/gitlab-cookbooks/praefect/templates/default/praefect-config.toml.erb): `<MR LINK>`- - [ ] If deprecating or removing a configuration, consider adding a [deprecation notice](https://gitlab.com/gitlab-org/omnibus-gitlab/-/blob/master/files/gitlab-cookbooks/package/libraries/deprecations.rb).-1. [ ] Update in [gitlab-development-kit](https://gitlab.com/gitlab-org/gitlab-development-kit) the configuration templates for [Gitaly](https://gitlab.com/gitlab-org/gitlab-development-kit/-/blob/master/support/templates/gitaly.config.toml.erb) and [Praefect](https://gitlab.com/gitlab-org/gitlab-development-kit/-/blob/master/support/templates/praefect.config.toml.erb): `<MR LINK>`-1. [ ] Update in [GitLab Chart](https://gitlab.com/gitlab-org/charts/gitlab) the configuration template for [Gitaly](https://gitlab.com/gitlab-org/charts/gitlab/-/blob/master/charts/gitlab/charts/gitaly/templates/configmap.yml): `<MR LINK>`-1. [ ] Update in [CNG](https://gitlab.com/gitlab-org/build/CNG) the [development configuration for Gitaly](https://gitlab.com/gitlab-org/build/CNG/-/blob/master/dev/gitaly-config/config.toml) and the [default configuration for Gitaly](https://gitlab.com/gitlab-org/build/CNG/-/blob/master/gitaly/config.toml): `<MR LINK>`-1. [ ] Update [GitLab test setup](https://gitlab.com/gitlab-org/gitlab): `<MR LINK>`-1. [ ] Update [gitlab-elasticsearch-indexer](https://gitlab.com/gitlab-org/gitlab-elasticsearch-indexer): `<MR LINK>`-1. [ ] Update [gitlab-workhorse](https://gitlab.com/gitlab-org/gitlab-workhorse): `<MR LINK>`diff --git a/.gitlab/issue_templates/Default.md b/.gitlab/issue_templates/Default.mddeleted file mode 100644index d818d9a..0000000--- a/.gitlab/issue_templates/Default.md+++ /dev/null@@ -1,5 +0,0 @@-<!---If possible, please select a relevant template from the "Choose a template" dropdown above.--->--<!-- template sourced from https://gitlab.com/gitlab-org/gitaly/-/blob/master/.gitlab/issue_templates/Default.md -->diff --git a/.gitlab/issue_templates/Demo.md b/.gitlab/issue_templates/Demo.mddeleted file mode 100644index 8e931de..0000000--- a/.gitlab/issue_templates/Demo.md+++ /dev/null@@ -1,140 +0,0 @@-<!--- Replace Date in title below -->--/title Demo YYYY-MM-DD--<!---## Contributing--When adding new feature demonstrations to the script, follow these guidelines.--For each feature you are verifying, add an H3 section with a link to the issue-to the `## Features` section.--Always add new features near the bottom of this section. This way older issues-will float to the top and allow them to be prioritized during the demo.--Make sure you break down steps into the following sections:--1. prep steps - these are steps needed to correctly set up your demonstration.- These steps are okay for the demo runner to perform before the start of the- demo call.-1. demo steps - these are the steps to perform during the demo call to show- how the feature works-1. verify steps - these are the expected observations required to be seen- in order to verify the prep or feature works as expected--Ideally, all setup steps should before the exercise steps (when possible).-Demo and verification steps may interleave as needed. For example, the-following structure is okay:--1. Prep-1. Prep-1. Verify-1. Prep-1. Demo-1. Verify-1. Demo-1. Demo-1. Verify-1. Verify--Along with the H3 section, it might look like this:--```-### #1234--1. [ ] Prep: install thingy-1. [ ] Verify: thingy works-1. [ ] Prep: turn on gizmo-1. [ ] Demo: press red button-1. [ ] Verify: world should explode-```--When your feature passes all verification steps, submit an MR to remove-it from this issue template.---->--This issue is used to conduct a demo for exhibiting and verifying new behavior-for Gitaly and Praefect. Before the demo, run all `Prep:` steps. During the-demo, run through all remaining `Demo:` and `Verify` steps. Check each-step as completed or verified. Do not check a `Verify:` step if it does not-succeed.--## General Setup--1. [ ] Prep:- - [ ] Check the [latest version of this issue template](https://gitlab.com/gitlab-org/gitaly/-/blob/master/.gitlab/issue_templates/Demo.md)- for any new steps and update this issue accordingly.- - [ ] Checkout the latest changes from Gitaly's default branch- - [ ] `cd _support/terraform`- - [ ] `./create-demo-cluster`- - [ ] `./configure-demo-cluster`- - [ ] Sign in as admin user `root` during the demo- - [ ] Create a new repository on the GitLab instance- - [ ] Log into the GitLab web interface and upload license--## Features--### Variable Replication Factor #2971--Previously Praefect has replicated repositories to every node in a virtual storage. This has made large clusters-unfeasible due to increasing cost of replicating repositories to every storage within a virtual storage. This-also made it impossible to horizontally scale a virtual storage's storage capacity. The virtual storage's storage-capacity would be limited by the smallest storage in the virtual storage as it has to fit every repository.--Variable replication factor allows administrator to set each repository's replication factor individually. This allows-for scaling the storage capacity of the cluster horizontally by allowing a repository's replication factor to be lower-than the storage count in a virtual storage. For important or highly used repositories, administrators can distribute-requests and increase redundancy by setting a higher replication factor.--Variable replication factor is only implemented using repository specific primaries. This is due to the primary node-needing a copy of each repository. Needing to have every repository on a single node would make the single primary a-bottleneck as it would need to contain every repository.--While variable replication factor itself is mostly ready, repository specific primaries still have some issues to solve.-Importantly for the demo, repository creation does not yet work. To work around that limitation, the prep step uses-`sql` elector.--1. Prep (all operations done on a Praefect node):- - [ ] Create two repositories in the demo cluster. `sql` elector must be enabled while doing this due to the- `per_repository` elector not being able to create repositories yet. These will be referred to as repository- A and repository B later.- - [ ] Run `sudo -i` as `gitlab-ctl` commands need to be run as root.- - [ ] Connect to Postgres in another terminal by running `/opt/gitlab/embedded/bin/psql -U praefect -d praefect_production -h <postgres address>` on a Praefect node.- - [ ] Ensure there are entries for every storage for both repositories in the `storage_repositories` table and- that they are all on the same generation.- - [ ] Enable repository specific primaries by setting `praefect['failover_election_strategy'] = 'per_repository'` in `/etc/gitlab/gitlab.rb` on Praefect nodes.- - [ ] Disable the reconciler initially. Set `praefect['reconciliation_scheduling_interval'] = 0` in `/etc/gitlab/gitlab.rb` on Praefect nodes.- - [ ] Reconfigure and restart the Praefect nodes by running `gitlab-ctl reconfigure`.-1. Demo:- - [ ] Attempt to set replication factor 0 for repository A by running `/opt/gitlab/embedded/bin/praefect -config /var/opt/gitlab/praefect/config.toml set-replication-factor -virtual-storage default -repository <relative path A> -replication-factor 0`. This should fail as the minimum replication factor is 0.- - [ ] Attempt to set replication factor 4 for repository A. This should fail as the demo cluster only has 3 storage nodes, meaning it is not possible to reach replication factor of 4.- - [ ] Set replication factor of 1 for repository A. The command should print out the assigned storage. The assigned storage should be the repository's primary. You can verify this by running `SELECT * FROM repositories WHERE relative_path = '<relative path A>';` and checking the `primary` column refers to the same storage.- - [ ] With the replication factor 1 set for repository A, perform a write in repository B. Repository B should still replicate on every node. After the write, you can verify each of the storages of B are on the same generation by running `SELECT * FROM storage_repositories WHERE relative_path = '<relative path B>';` and checking that the generations match.- - [ ] Check the virtual storage's status with dataloss by running `/opt/gitlab/embedded/bin/praefect -config /var/opt/gitlab/praefect/config.toml dataloss -virtual-storage default -partially-replicated`. It should not list any outdated repositories.- - [ ] Perform a write in repository A.- - [ ] Check the virtual storage's status with dataloss again. Everything should be fully up to date as all the assigned storages are up to date. Check repository A's entries in the `storage_repositories` table and verify only the assigned node's generation was incremented. Only the assigned nodes participate in a transaction or get a replication job scheduled for a given write.- - [ ] Set the replication factor of repository A to 2. You should observe a random secondary being assigned.- - [ ] Check the virtual storage's status with dataloss again. It should now list repository A as one of the assigned storages is outdated. You should also see the other secondary listed outdated, but without being designated as assigned.- - [ ] Perform another write in repository A.- - [ ] Check the virtual storage's status with dataloss again. Repository A should not be listed anymore as the new write scheduled a replication job to bring the outdated assigned secondary back to speed.- - [ ] Set the replication factor of repository A to 1 and perform a write.- - [ ] Set the replication factor of repository A back to 2.- - [ ] Check with dataloss that the assigned secondary is listed as outdated. The other secondary should also be listed outdated but not assigned.- - [ ] Enable the reconciler by setting the `reconciliation_scheduling_interval` to '5s'. Reconfigure and restart Praefects.- - [ ] Wait for the reconciler to schedule and execute the replication jobs.- - [ ] Check with dataloss that repository A is no longer considered outdated. The reconciler only targets assigned nodes. Verify from the `storage_repositories` table that the unassigned storage is still on a lower generation than the assigned nodes.- - [ ] Repeatedly increase and decrease the replication factor of repository A from 1 to 3 and back. You should observe the primary node is never unassigned, only the secondaries.--## After Demo--1. [ ] Create any follow up issues discovered during the demo and assign label- ~demo.- - Link the issues as related to this issue-1. [ ] [Follow teardown instructions to remove demo- resources](https://gitlab.com/gitlab-org/gitaly/-/blob/master/_support/terraform/README.md#destroying-a-demo-cluster)-1. [ ] Open a new demo issue and assign to the next demo conductor-1. [ ] Close this issue--/label ~demo ~"group::gitaly" ~"devops::systems"diff --git a/.gitlab/issue_templates/Feature Flag Default Enable.md b/.gitlab/issue_templates/Feature Flag Default Enable.mddeleted file mode 100644index 9a36964..0000000--- a/.gitlab/issue_templates/Feature Flag Default Enable.md+++ /dev/null@@ -1,22 +0,0 @@-/title [Feature flag] Default enable feature flag name--/relate feature flag rollout issue here--## What--Default enable the `:feature_name` feature flag.--## Owners--- Team: Gitaly-- Most appropriate slack channel to reach out to: `#g_gitaly`-- Best individual to reach out to: NAME--## Steps--- [ ] Change the feature flag to default-enabled ([howto](https://gitlab.com/gitlab-org/gitaly/-/blob/master/doc/PROCESS.md#feature-lifecycle-after-it-is-live))--Please refer to the [documentation of feature flags](https://gitlab.com/gitlab-org/gitaly/-/blob/master/doc/PROCESS.md#feature-flags) for further information.--/assign me-/label ~"devops::systems" ~"group::gitaly" ~"feature flag" ~"feature::maintainance" ~"Category:Gitaly" ~"section::enablement" ~"featureflag::production" ~"workflow::ready for development"diff --git a/.gitlab/issue_templates/Feature Flag Removal.md b/.gitlab/issue_templates/Feature Flag Removal.mddeleted file mode 100644index 6925bcb..0000000--- a/.gitlab/issue_templates/Feature Flag Removal.md+++ /dev/null@@ -1,25 +0,0 @@-/title [Feature flag] Removal of feature flag name--/relate feature flag default enable issue here (if exists)-/relate feature flag rollout issue here--## What--Remove the `:feature_name` feature flag.--## Owners--- Team: Gitaly-- Most appropriate slack channel to reach out to: `#g_gitaly`-- Best individual to reach out to: NAME--## Steps--- [ ] Remove the feature flag and the pre-feature-flag code ([howto](https://gitlab.com/gitlab-org/gitaly/-/blob/master/doc/PROCESS.md#feature-lifecycle-after-it-is-live))-- [ ] Wait for the MR to be deployed to production-- [ ] Remove the feature flag via chatops ([howto](https://gitlab.com/gitlab-org/gitaly/-/blob/master/doc/PROCESS.md#remove-the-feature-flag-via-chatops))--Please refer to the [documentation of feature flags](https://gitlab.com/gitlab-org/gitaly/-/blob/master/doc/PROCESS.md#feature-flags) for further information.--/assign me-/label ~"devops::systems" ~"group::gitaly" ~"feature flag" ~"feature::maintainance" ~"Category:Gitaly" ~"section::enablement" ~"workflow::ready for development"diff --git a/.gitlab/issue_templates/Feature Flag Roll Out.md b/.gitlab/issue_templates/Feature Flag Roll Out.mddeleted file mode 100644index d701660..0000000--- a/.gitlab/issue_templates/Feature Flag Roll Out.md+++ /dev/null@@ -1,79 +0,0 @@-/title [Feature flag] Enable description of feature--<!----IMPORTANT: when Gitaly is started in a Rails test, all feature flags are enabled:-https://gitlab.com/gitlab-org/gitlab/blob/28bdb35c83fbfcfca09a9d237bfecb71e55b8f9e/spec/support/helpers/gitaly_setup.rb#L69--Any flags which change user-facing behaviour have a chance of causing test failures, which-in turn will block Gitaly deployment MRs. In these situations, you can do the following:--1. Get the Gitaly change reviewed and merged. Validate the Gitaly pipeline is passing on the default branch.-1. Open a Rails MR that simultaneously updates the commit SHA in- GITALY_SERVER_VERSION (https://gitlab.com/gitlab-org/gitlab/-/blob/master/GITALY_SERVER_VERSION) and- contains the updated assertions. This will allow tests to execute against a version of Gitaly that- contains your changes, while updating the version at the same time.-1. Get the Rails MR reviewed and merged.---->--## What--Enable the `:feature_name` feature flag ...--## Owners--- Team: Gitaly-- Most appropriate slack channel to reach out to: `#g_gitaly`-- Best individual to reach out to: NAME--## Expectations--### What release does this feature occur in first?--### What are we expecting to happen?--### What might happen if this goes wrong?--### What can we monitor to detect problems with this?--<!----Which dashboards from https://dashboards.gitlab.net are most relevant?-Usually you'd just like a link to the method you're changing in the-dashboard at:--https://dashboards.gitlab.net/d/000000199/gitaly-feature-status--I.e.--1. Open that URL-2. Change "method" to your feature, e.g. UserDeleteTag-3. Copy/paste the URL & change gprd to gstd to monitor staging as well as prod---->--## Roll Out Steps--- [ ] Enable on staging- - [ ] Is the required code deployed on staging? ([howto](https://gitlab.com/gitlab-org/gitaly/-/blob/master/doc/PROCESS.md#is-the-required-code-deployed))- - [ ] Enable on staging ([howto](https://gitlab.com/gitlab-org/gitaly/-/blob/master/doc/PROCESS.md#enable-on-staging))- - [ ] Add ~"featureflag::staging" to this issue ([howto](https://gitlab.com/gitlab-org/gitaly/-/blob/master/doc/PROCESS.md#feature-flag-labels))- - [ ] Test on staging ([howto](https://gitlab.com/gitlab-org/gitaly/-/blob/master/doc/PROCESS.md#test-on-staging))- - [ ] Verify the feature flag was used by checking Prometheus metric [`gitaly_feature_flag_checks_total`](https://dashboards.gitlab.net/explore?schemaVersion=1&panes=%7B%22jom%22:%7B%22datasource%22:%22mimir-gitlab-gstg%22,%22queries%22:%5B%7B%22refId%22:%22A%22,%22expr%22:%22rate%28gitaly_feature_flag_checks_total%5B$__rate_interval%5D%29%22,%22range%22:true,%22instant%22:true,%22datasource%22:%7B%22type%22:%22prometheus%22,%22uid%22:%22mimir-gitlab-gstg%22%7D,%22editorMode%22:%22code%22,%22legendFormat%22:%22__auto%22%7D%5D,%22range%22:%7B%22from%22:%22now-1h%22,%22to%22:%22now%22%7D%7D%7D&orgId=1&g0.expr=sum%20by%20%28flag%29%20%28rate%28gitaly_feature_flag_checks_total%5B5m%5D%29%29&g0.tab=1&g0.stacked=0&g0.range_input=1h)-- [ ] Enable on production- - [ ] Is the required code deployed on production? ([howto](https://gitlab.com/gitlab-org/gitaly/-/blob/master/doc/PROCESS.md#is-the-required-code-deployed))- - [ ] Progressively enable in production ([howto](https://gitlab.com/gitlab-org/gitaly/-/blob/master/doc/PROCESS.md#enable-in-production))- - [ ] Add ~"featureflag::production" to this issue- - [ ] Verify the feature flag was used by checking Prometheus metric [`gitaly_feature_flag_checks_total`](https://dashboards.gitlab.net/explore?schemaVersion=1&panes=%7B%22jom%22:%7B%22datasource%22:%22mimir-gitlab-gprd%22,%22queries%22:%5B%7B%22refId%22:%22A%22,%22expr%22:%22rate%28gitaly_feature_flag_checks_total%5B$__rate_interval%5D%29%22,%22range%22:true,%22instant%22:true,%22datasource%22:%7B%22type%22:%22prometheus%22,%22uid%22:%22mimir-gitlab-gprd%22%7D,%22editorMode%22:%22code%22,%22legendFormat%22:%22__auto%22%7D%5D,%22range%22:%7B%22from%22:%22now-1h%22,%22to%22:%22now%22%7D%7D%7D&orgId=1&g0.expr=sum%20by%20%28flag%29%20%28rate%28gitaly_feature_flag_checks_total%5B5m%5D%29%29&g0.tab=1&g0.stacked=0&g0.range_input=1h)-- [ ] Create subsequent issues- - [ ] To default enable the feature flag (optional, only required if backwards-compatibility concerns exist)- - [ ] [Create issue](https://gitlab.com/gitlab-org/gitaly/-/issues/new?issuable_template=Feature%20Flag%20Default%20Enable) using the `Feature Flag Default Enable` template.- - [ ] Set milestone to current+1 release- - [ ] To Remove feature flag- - [ ] [Create issue](https://gitlab.com/gitlab-org/gitaly/-/issues/new?issuable_template=Feature%20Flag%20Removal) using the `Feature Flag Removal` template.- - [ ] Set milestone to current+1 (+2 if we created an issue to default enable the flag).--Please refer to the [documentation of feature flags](https://gitlab.com/gitlab-org/gitaly/-/blob/master/doc/PROCESS.md#feature-flags) for further information.--/label ~"devops::systems" ~"group::gitaly" ~"feature flag" ~"feature::maintainance" ~"Category:Gitaly" ~"section::enablement" ~"featureflag::disabled"diff --git a/.gitlab/issue_templates/Flaky Test.md b/.gitlab/issue_templates/Flaky Test.mddeleted file mode 100644index 34bbb0c..0000000--- a/.gitlab/issue_templates/Flaky Test.md+++ /dev/null@@ -1,16 +0,0 @@-/title [Flaky Test] <test name>--<!---Provide any relevant information about the flaky test. Such as:-- Link to CI job with flaky test failure-- Copy of logs containing failure--->--## Steps-- [ ] This issue has been assigned the current milestone-- [ ] Flaky test has been quarantined with [`testhelper.SkipQuarantinedTest`](https://gitlab.com/gitlab-org/gitaly/-/blob/master/internal/testhelper/testhelper.go#L388-L398).-- [ ] Flaky test has been fixed and quarantine removed.--/label ~"Category:Gitaly" ~"group::gitaly" ~"section::enablement" ~"devops::systems" ~"type::maintenance" ~"maintenance::pipelines" ~"failure::flaky-test" ~"priority::1"-/assign me-/label ~"workflow::ready for development"diff --git a/.gitlab/issue_templates/Git Version Upgrade.md b/.gitlab/issue_templates/Git Version Upgrade.mddeleted file mode 100644index 6d30445..0000000--- a/.gitlab/issue_templates/Git Version Upgrade.md+++ /dev/null@@ -1,40 +0,0 @@-/title Rollout Git version vX.XX.X--## Changelog--<!---Add the changelog related to the new version and how this impacts us. It is especially important to highlight changes that increase the risk for this particular upgrade.-Would be really nice to point out contributions made by the Gitaly team, if any.--->--## Steps--### Introducing a new Git version--- [ ] Add the new bundled Git version in the [Makefile](/Makefile) ([Reference](https://gitlab.com/gitlab-org/gitaly/-/merge_requests/7184)).- - [ ] Update the `build-bundled-git` and `install-bundled-git` targets as necessary.- - [ ] For a minor version bumps and above, add a new target under the `# These targets build specific releases of Git.` section corresponding to the new version.-- [ ] Introduce the new bundled Git execution environment in the [Git package](/internal/git/execution_environment.go) behind a feature flag.- - [ ] Create an issue for the rollout of the feature flag ([Reference](https://gitlab.com/gitlab-org/gitaly/-/issues/5030)).- - [ ] Optional: Create a [change request](https://about.gitlab.com/handbook/engineering/infrastructure/change-management/#change-request-workflows) in case the new Git version contains changes that may cause issues.- - [ ] Roll out the feature flag (Announce to #g_gitaly and #g_git).- - [ ] Default enable the feature flag once the upgrade is deemed stable. You can do this in same release that added the feature flag initially.- - [ ] Remove the feature flag after there has been at least one release with the feature flag default enabled. This is not a requirement since we bundle Git with Gitaly, but more of a safety precaution taken to ensure there is sufficient time for rollbacks/reverts.--### Removing an old Git version--- [ ] Remove the old bundled Git version from the [Makefile](/Makefile)- - [ ] Update the `build-bundled-git` and `install-bundled-git` targets as necessary.- - [ ] Remove any unused targets under the `# These targets build specific releases of Git.` section corresponding to the old version.-- [ ] Remove the old bundled Git execution environment from the [Git package](/internal/git/execution_environment.go).--### Upgrading the minimum required Git version--Optional: This is only needed when we want to start using features that have been introduced with the new Git version.--- [ ] Update the minimum required Git version in the [Git package](/internal/git/version.go). ([Reference](https://gitlab.com/gitlab-org/gitaly/-/merge_requests/5705))-- [ ] Update the minimum required Git version in the [README.md](/README.md).-- [ ] Update the GitLab release notes to reflect the new minimum required Git version. ([Reference](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/107565).--/label ~"Category:Gitaly" ~"group::gitaly" ~"group::gitaly::git" ~"section::enablement" ~"devops::systems" ~"type::maintenance" ~"maintenance::dependency"-/label ~"workflow::ready for development"diff --git a/.gitlab/issue_templates/Gitaly Client Update.md b/.gitlab/issue_templates/Gitaly Client Update.mddeleted file mode 100644index aeb4c8c..0000000--- a/.gitlab/issue_templates/Gitaly Client Update.md+++ /dev/null@@ -1,14 +0,0 @@-# gitaly/client config change checklist--When you make a change in gitlab.com/gitlab-org/gitaly/client you need to perform a number of steps to actually ship the change.--- [ ] update [go.mod in gitlab-shell](https://gitlab.com/gitlab-org/gitlab-shell/-/blob/master/go.mod) MR LINK-- [ ] update [go.mod in gitlab-workhorse](https://gitlab.com/gitlab-org/gitlab-workhorse/-/blob/master/go.mod) MR LINK-- [ ] update [go.mod in gitlab-elasticsearch-indexer](https://gitlab.com/gitlab-org/gitlab-elasticsearch-indexer/-/blob/master/go.mod) MR LINK-- [ ] wait/ask for gitlab-shell release TAG LINK-- [ ] wait/ask for gitlab-workhorse release TAG LINK-- [ ] wait/ask for gitlab-elasticsearch-indexer release TAG LINK-- [ ] update [GITLAB_SHELL_VERSION in gitlab](https://gitlab.com/gitlab-org/gitlab/-/blob/master/GITLAB_SHELL_VERSION) MR LINK-- [ ] update [GITLAB_WORKHORSE_VERSION in gitlab](https://gitlab.com/gitlab-org/gitlab/-/blob/master/GITLAB_WORKHORSE_VERSION) MR LINK-- [ ] update [GITLAB_ELASTICSEARCH_INDEXER_VERSION in gitlab](https://gitlab.com/gitlab-org/gitlab/-/blob/master/GITLAB_ELASTICSEARCH_INDEXER_VERSION) MR LINK-diff --git a/.gitlab/issue_templates/Ignored Vulnerability.md b/.gitlab/issue_templates/Ignored Vulnerability.mddeleted file mode 100644index 6bc20ac..0000000--- a/.gitlab/issue_templates/Ignored Vulnerability.md+++ /dev/null@@ -1,15 +0,0 @@-/title [Ignored Vulnerability] <GO ID>--https://pkg.go.dev/vuln/<GO ID>--## Steps--- [ ] Verify the severity of the vulnerability.-- [ ] Add the vulnerability to the ignore list in `./tools/govulncheck-filter/main.go`.-- [ ] Plan steps for resolving the vulnerability, e.g. arranging for dependencies to be updated.--/label ~"Category:Gitaly" ~"devops::data_access" ~"group::gitaly" ~"section::infrastructure platforms"-/label ~"type::maintenance" ~"maintenance::dependency" ~security-/assign me-/label ~"workflow::ready for development"-/confidentialdiff --git a/.gitlab/issue_templates/Production Training.md b/.gitlab/issue_templates/Production Training.mddeleted file mode 100644index 786ff75..0000000--- a/.gitlab/issue_templates/Production Training.md+++ /dev/null@@ -1,98 +0,0 @@-# Gitaly Team production training--This is a collection of items that should prepare new team members to be effective in understanding production issues and thus join the [on-call rotation](https://about.gitlab.com/handbook/engineering/infrastructure-platforms/tenant-scale/gitaly#gitaly-oncall-rotation).--While this can be started at any time, team members should complete their onboarding first, and have some experience in the codebase before completing this process.--**This is not a test, it's an interactive learning guide.** It's quite normal and expected to ask for help and to discuss different approaches.--## Setup--- [ ] Set the title to `Gitaly Team production training: <your name here>`--## Links--Skim/read through these, and use them as references.--- [Debugging Gitaly](https://handbook.gitlab.com/handbook/engineering/infrastructure-platforms/tenant-scale/gitaly/debug/)-- [Managing monorepos](https://docs.gitlab.com/user/project/repository/monorepos/)-- [GCP project layout for Gitaly VMs](https://gitlab.com/gitlab-com/gl-infra/readiness/-/blob/master/library/gitaly-multi-project/README.md)-- [Gitaly runbooks](https://gitlab.com/gitlab-com/runbooks/-/tree/master/docs/gitaly)--Please help correct, clarify, or otherwise improve any documentation you find lacking (including this template)!--## Questions--Please edit this section like a workbook, adding not just the answer but also how you got there.--- [ ] Find a [senior team member](https://handbook.gitlab.com/handbook/engineering/infrastructure-platforms/tenant-scale/gitaly/) to review and discuss this work, and assign them here.--### Statistics--- [ ] What was Gitaly's SLO availability last month?-- [ ] Which repository had the most errors in `gprd` last week?-- [ ] What was the top error they had? If it's a real bug, please file an issue. :slight_smile:-- [ ] What was the p95 latency of the `SSHUploadPackWithSidechannel` RPC in `gstg` last week?-- [ ] Which Gitaly node had the most performance issues last week?-- [ ] Which RPC handlers spent the most CPU time in the last week? Is there an overarching theme amongst them? Hint: we export profiling metrics to GCP.--### Key Host Metrics--- [ ] Be able to explain every graph in the [host stats](https://dashboards.gitlab.net/d/bd2Kl9Imk/host-stats?orgId=1&from=now-1h&to=now&timezone=utc&var-env=gprd&var-node=gitaly-04-stor-gprd.c.gitlab-gitaly-gprd-87a9.internal) dashboard:- - What is it?- - Why do we care?--### Feature flags--- [ ] When was the last feature flag enabled on `gprd`?-- [ ] ...How would you roll it back?-- [ ] Are there any feature flags currently rolling out? What stage are they in?--### Releases--- [ ] There's a bug in `git` and it needs to be rolled back. Describe the process; think about what stage of being rolled out the broken `git` is in. [HINT](https://gitlab.com/gitlab-org/gitaly/-/blob/master/.gitlab/issue_templates/Git%20Version%20Upgrade.md)-- [ ] What version of Gitaly is running currently in `gprd`?-- [ ] Pick a MR from last month -- did it make it into the last release?--### Git Operations--- [ ] Scenario: clones are "slow". [HINT](https://log.gprd.gitlab.net/app/r/s/zoX53)- - [ ] How much memory was consumed for clones (http or ssh) for the gitlab-org/gitlab repository in the past hour?- - [ ] How much CPU was consumed for all clones (http or ssh) for the gitlab-org/gitlab repository in the past hour?- - [ ] How slow are the slowest clones taking on the gitlab-org/gitlab repository the past day?-- [ ] Were there any git operations rejected by Gitaly due to `GitLab is currently unable to handle this request due to load`?- - [ ] If yes, what was the main reason?--### Tracking calls throughout the GitLab ecosystem--- [ ] Pick a [repository files API- call](https://docs.gitlab.com/api/repository_files/#get-file-from-repository) that happened in the past day. Which Gitaly calls were invoked downstream?-- [ ] Pick a `GetBlobs` RPC call in the past day. Who were the upstream callers?-- [ ] Trace the entire flow when a user clones a repo, over SSH, over HTTP, and through the web editor. What components were involved? On which node is the repo data located?--### Runbooks--- [ ] After a Gitaly node restart, users report elevated "The operation could not be completed. Please try again." errors, with `error_metadata.error_details` showing "cannot lock references". What is the root cause, and what manual fix can you run from the Rails console to recover the affected repositories?-- [ ] A single Gitaly node's CPU and anonymous memory usage spikes well above its peers, and `top` shows many `git pack-objects` processes that are children of `git-upload-pack`, all working on the same repo. Which script in this repo can you run to safely mitigate the immediate pressure, and which two non-kill remedies are recommended for the affected project?-- [ ] Gitaly assigns each repository to a cgroup to bound CPU and memory usage. Under `/sys/fs/cgroup/{cpu,memory}/gitaly/gitaly-<pid>/`, what are the child cgroups named, and which Prometheus metric (exported via cadvisor) tells you how much a repo cgroup is being CPU-throttled? Bonus: which log field tells you which cgroup ran the `git` command for a given RPC?-- [ ] What are the three classic symptoms of Gitaly queuing described in the runbook, and which historical S2 incident (linked from the runbook) is the canonical reference for this failure mode?-- [ ] Given a `git pack-objects` PID on a Gitaly host, which `/proc/<pid>/...` lookups would you use to find (a) the repository directory it is operating on and (b) the GitLab `correlation_id` for the request? Once you have the repo directory, which `git config` key reveals the GitLab project's full path?--## Production--- [ ] Are there any production outages going on right now (Gitaly or not)? If not, what was the last one?--## Follow-up activities--- [ ] [Set up a local ELK and import log files into it](../../doc/setup_local_elk_for_downloaded_logs.md)-- [ ] Read through some recently closed customer issues and the investigation. Follow the reasoning and understand the fix. [Gitaly Customer Issues](https://gitlab.com/groups/gitlab-org/tenant-scale/gitaly/-/epics/2).-- [ ] Join an ongoing investigation, or pick up a new incoming issue. (Add the current milestone and ~"workflow::in dev" while assignig the issue to yourself.) Ask for help and guidance shamelessly. :slight_smile:-- [ ] Monitor `#g_gitaly` for incoming questions, direct them to our [intake flow](https://handbook.gitlab.com/handbook/engineering/infrastructure-platforms/tenant-scale/gitaly#customer-issues)--## Finally--- [ ] Add yourself to the [oncall rotation](https://ops.gitlab.net/gitlab-com/gl-infra/config-mgmt/-/blob/main/environments/pagerduty/gitaly_locals.tf?ref_type=heads) by raising a MR. Set manager and reviewer buddy as reviewers.--/confidential-/assign me-/cc @luciezhaodiff --git a/.gitlab/issue_templates/Security Release.md b/.gitlab/issue_templates/Security Release.mddeleted file mode 100644index 2a25907..0000000--- a/.gitlab/issue_templates/Security Release.md+++ /dev/null@@ -1,74 +0,0 @@-<!---# Read me first!--Create this issue under https://gitlab.com/gitlab-org/security--Set the title to: `Description of the original issue`--->--## Prior to starting the security release work--- [ ] Read the [security process for developers] if you are not familiar with it.-- [ ] Run `scripts/security-harness` in your local repository to prevent accidentally pushing to any remote besides `gitlab.com/gitlab-org/security`.-- Fill out the [Links section](#links):- - [ ] Next to **Issue on Gitaly**, add a link to the `gitlab-org/gitaly` issue that describes the security vulnerability.--## Development--- [ ] Create a new branch prefixing it with `security-`.-- [ ] Create a merge request targeting `master` on `gitlab.com/gitlab-org/security` and use the [Security Release merge request template].-- [ ] Follow the same [code review] process: Assign to a reviewer, then to a maintainer.--After your merge request has been approved according to our [approval guidelines], you're ready to prepare the backports--## Backports--- [ ] Once the MR is ready to be merged, create MRs targeting the latest 3 stable branches- * The 3 stable branches correspond to the versions in the title of the [Security Release Tracking Issue].- * At this point, it might be easy to squash the commits from the MR into one- * You can use the script `bin/secpick` instead of the following steps, to help you cherry-picking. See the [secpick documentation]-- [ ] Create each MR targeting the stable branch `X-Y-stable`, using the [Security Release merge request template].- * Every merge request will have its own set of TODOs, so make sure to complete those.-- [ ] On the "Related merge requests" section, ensure all MRs are linked to this issue.- * This section should only list the merge requests created for this issue: One targeting `master` and the 3 backports.-- [ ] If this issue requires less than `4` merge requests, add the ~"reduced backports" label.--## Assigning to a release--- [ ] When this issue is ready for release (Backports are approved and ready to be merged), apply the ~"security-target" label for it to be evaluated and linked to the next planned security release tracking issue.--## Documentation and final details--- [ ] Ensure the [Links section](#links) is completed.-- [ ] Add the GitLab [versions](https://gitlab.com/gitlab-org/release/docs/-/blob/master/general/security/developer.md#versions-affected) and editions affected to the [details section](#details)- * The Git history of the files affected may help you associate the issue with a [release](https://about.gitlab.com/releases/)-- [ ] Fill in any upgrade notes that users may need to take into account in the [details section](#details)-- [ ] Add the nickname of the external user who found the issue (and/or HackerOne profile) to the Thanks row in the [details section](#details)--## Summary--### Links--| Description | Link |-| -------- | -------- |-| Issue on [Gitaly](https://gitlab.com/gitlab-org/gitaly/issues) | #TODO |-| CVE request | #TODO |--### Details--| Description | Details | Further details|-| -------- | -------- | -------- |-| First Version affected | X.Y | |-| Upgrade notes | | |-| Thanks | | |--[gitaly-ce-version]: https://gitlab.com/gitlab-org/gitlab-ce/blob/master/GITALY_SERVER_VERSION-[security process for developers]: https://gitlab.com/gitlab-org/release/docs/blob/master/general/security/developer.md-[secpick documentation]: https://gitlab.com/gitlab-org/release/docs/-/blob/master/general/security/utilities/secpick_script.md-[security Release merge request template]: https://gitlab.com/gitlab-org/security/gitaly/blob/master/.gitlab/merge_request_templates/Security%20Release.md-[code review process]: https://docs.gitlab.com/development/code_review/-[approval guidelines]: https://docs.gitlab.com/development/code_review/#approval-guidelines-[issue as related]: https://docs.gitlab.com/user/project/issues/related_issues/#add-a-linked-issue-[Security Release Tracking Issue]: https://gitlab.com/gitlab-org/gitlab/-/issues/?label_name%5B%5D=upcoming%20security%20release--/label ~"devops::systems" ~"group::gitaly" ~"security" ~"security-notifications"diff --git a/.gitlab/issue_templates/Team Member Onboarding.md b/.gitlab/issue_templates/Team Member Onboarding.mddeleted file mode 100644index df47062..0000000--- a/.gitlab/issue_templates/Team Member Onboarding.md+++ /dev/null@@ -1,45 +0,0 @@-# Gitaly Team specific onboarding--Welcome to [Gitaly Team](https://handbook.gitlab.com/handbook/engineering/infrastructure-platforms/tenant-scale/gitaly/)!!--The following items are the team-specific extension of your onboarding issue.-Please complete all but the "Reading material" section in your first few working days.--- Join Slack channels:- - [ ] [#git-access-lounge](https://gitlab.slack.com/archives/git-access-lounge) (internal team affairs, generic informal chat)- - [ ] [#g_gitaly](https://gitlab.slack.com/archives/g_gitaly) (important communication also outside the team, alerts on blocked [releases](../../doc/release.md#why-do-pipeline-failures-occur))- - [ ] [#scale-architecture-social](https://gitlab.slack.com/archives/scale-architecture-social) (social channel for the entire org)- - [ ] [#scale-architecture-org](https://gitlab.slack.com/archives/scale-architecture-org) (org-wide communication)- - [ ] [#gitaly-alerts](https://gitlab.slack.com/archives/gitaly-alerts) (Alerts for Gitaly pipeline failures)- - [ ] [#gitaly-deployments](https://gitlab.slack.com/archives/gitaly-deployments) (Notifications for Gitaly deployments)-- Ask your manager or your mentor in order to:- - [ ] get Maintainer access to the [@gl-gitaly](https://gitlab.com/groups/gl-gitaly/-/group_members) group- - [ ] get Maintainer access to the [@gl-gitaly](https://ops.gitlab.net/groups/gl-gitaly/-/group_members) group on ops.gitlab.com- - [ ] be added to the weekly discussion sessions and [async news doc](https://docs.google.com/document/d/1k1hY0ZdeFUnSqF6GUToFaJEtVIN_SjWeaRdx1V0i25E/edit)- - [ ] be added to [Geekbot](https://app.geekbot.com/dashboard) for daily Slack standup updates- - [ ] get Maintainer access to the [Gitaly Retrospective](https://gitlab.com/gl-retrospectives/enablement-section/gitaly) respository- - [ ] get access to the [Gitaly Group calendar](https://calendar.google.com/calendar/embed?src=gitlab.com_mc911ncuk38bfngud133tjb4m8%40group.calendar.google.com&ctz=Europe%2FZurich), then add it by clicking [here](https://calendar.google.com/calendar/u/0?cid=Z2l0bGFiLmNvbV9tYzkxMW5jdWszOGJmbmd1ZDEzM3RqYjRtOEBncm91cC5jYWxlbmRhci5nb29nbGUuY29t)-- Acquire necessary access: a Role Based Access Request should have been opened for you based on [this template](https://gitlab.com/gitlab-com/team-member-epics/access-requests/-/blob/master/.gitlab/issue_templates/role_baseline_access_request_tasks/department_internal_infrastructure/role_backend_engineer_gitaly.md?ref_type=heads).- - [ ] If that's not the case, let a manager know and open one following [these instructions](https://internal.gitlab.com/handbook/it/end-user-services/access-request/baseline-entitlements/#how-can-i-create-a-new-ar-using-a-baseline-template).- - [ ] Complete the `Team Member Self Service` section of that issue, and note what you're getting access to.-- If transferring from another team within GitLab:- - [ ] update your [Team page entry](https://about.gitlab.com/handbook/git-page-update/#12-add-yourself-to-the-team-page)- - [ ] update your Slack profile ("What I do")- - [ ] update your Zoom title-- Integrate your PTO with the Gitaly Team Calendar- - [ ] [follow these instructions](https://about.gitlab.com/handbook/people-group/engineering/team-pto-calendar/#steps), and use `[email protected]` as the Calendar ID.-- Reading material- - [ ] check out the [README](https://gitlab.com/gitlab-org/gitaly/-/blob/master/README.md) at https://gitlab.com/gitlab-org/gitaly, without getting lost in too much detail- - [ ] visit the [Gitaly Training & Resources Youtube channel](https://youtube.com/playlist?list=PL05JrBw4t0KqPMCB7xW1ExIRp6YuD_7_N) for videos on how Gitaly works.- - [ ] checkout the Gitaly doc [README](https://gitlab.com/gitlab-org/gitaly/-/blob/master/doc/README.md)- - [ ] familiarize yourself with the [Merge request workflow](https://docs.gitlab.com/ee/development/contributing/merge_request_workflow.html) specific to working _on_ GitLab (the whole page)- - [ ] check out the [team page](https://handbook.gitlab.com/handbook/engineering/infrastructure-platforms/tenant-scale/gitaly/)- - [ ] for day-to-day work, get familiar with how we [schedule work](https://handbook.gitlab.com/handbook/engineering/infrastructure-platforms/tenant-scale/gitaly/#working-with-product). It may be useful to bookmark the [Gitaly ongoing work issue board](https://gitlab.com/groups/gitlab-org/-/boards/1140874?label_name%5B%5D=group%3A%3Agitaly&milestone_title=Upcoming)- - [ ] for the big picture, see the team [vision and roadmap](https://handbook.gitlab.com/handbook/engineering/infrastructure-platforms/tenant-scale/gitaly/#roadmap) (click through on the links)- - [ ] create an MR in the [chef-repo](https://gitlab.com/gitlab-com/gl-infra/chef-repo/) to add a [data bag](https://gitlab.com/gitlab-com/gl-infra/chef-repo/-/blob/master/doc/user-administration.md#add-a-new-developer)- for yourself to create your user on production machines. Be sure to include- `"gitaly-team"` under the `groups` field.--/confidential-/assign me-/cc @luciezhaodiff --git a/.gitlab/merge_request_templates/RFC.md b/.gitlab/merge_request_templates/RFC.mddeleted file mode 100644index 7f02816..0000000--- a/.gitlab/merge_request_templates/RFC.md+++ /dev/null@@ -1,41 +0,0 @@-/title RFC: <REPLACE TITLE>-/label ~"Gitaly RFC" ~backstage--**Accepting comments until <YYYY-MM-DD> UTC**--## Contributor checklist--Check all items before moving onto the next section.--### Pre-Review--- [ ] Verify you have reviewed and understand the [RFC guidelines](doc/rfcs/README.md).-- [ ] Replace all placeholders in the RFC template:- - [ ] Replace `<REPLACE TITLE>` with the RFC title- - [ ] Replace `<REPLACE ABSTRACT>` with a short summary of the RFC contents.-- [ ] Replace all placeholders in this MR template:- - [ ] Replace `<REPLACE TITLE>` at the top of the MR description- - [ ] Replace `<YYYY-MM-DD>` with the UTC deadline for accepting comments.-- [ ] Follow the [contributor guidelines for reviews](https://gitlab.com/gitlab-org/gitaly/-/blob/master/REVIEWING.md#tips-for-the-contributor)--### Ready for review--- [ ] Require a minimum of 2 maintainer approvals. Increase if warranted.-- [ ] Once ready for review, announce the RFC to the Gitaly team (`/cc @gl-gitaly`)- - [ ] Announce a time window for accepting comments (at least a week).--### Post-Approval--Once the minimum number of maintainers have approved:--- [ ] Wait until the declared time window expires before taking action to give others a chance to comment.-- [ ] Once the time window expires, decide to either merge the RFC as is, or address new feedback.-- [ ] If you choose to make changes, ping the existing approvers so that they may review the changes.-- [ ] Merge when ready. If you do not have write access to the repository, ping a Gitaly maintainer `@gl-gitaly`.--## Reviewer instructions--1. Familiarize yourself with the [RFC Guidelines](doc/rfcs/README.md)-1. Identify other reviewers who can offer constructive feedback. Offer them the opportunity to review.-1. For your review, follow the [Gitaly reviewing guide](https://gitlab.com/gitlab-org/gitaly/-/blob/master/REVIEWING.md).-1. Approve, but do not merge. Let the contributor merge when ready.diff --git a/.gitlab/merge_request_templates/Security Release.md b/.gitlab/merge_request_templates/Security Release.mddeleted file mode 100644index 14611df..0000000--- a/.gitlab/merge_request_templates/Security Release.md+++ /dev/null@@ -1,48 +0,0 @@-<!---# README first!-This MR should be created on `gitlab.com/gitlab-org/security/gitaly`.--See [the general developer security release guidelines](https://gitlab.com/gitlab-org/release/docs/blob/master/general/security/developer.md).---->--## Related issues--<!-- Mention the GitLab Security issue this MR is related to -->--## Developer checklist--- [ ] **On "Related issues" section, write down the [GitLab Security] issue it belongs to (i.e. `Related to <issue_id>`).**-- [ ] Merge request targets `master`, or `X-Y-stable` for backports.-- [ ] Milestone is set for the version this merge request applies to. A closed milestone can be assigned via [quick actions].-- [ ] Title of this merge request is the same as for all backports.-- [ ] A [CHANGELOG entry] has been included, with `Changelog` trailer set to `security`.-- [ ] Assign to a reviewer and maintainer, per our [Code Review process].-- [ ] For the MR targeting `master`:- - [ ] Ensure it's approved according to our [Approval Guidelines].-- [ ] Merge request _must not_ close the corresponding security issue, _unless_ it targets `master`.--**Note:** Reviewer/maintainer should not be a Release Manager--## Maintainer checklist-- [ ] Correct milestone is applied and the title is matching across all backports-- [ ] Assigned to `@gitlab-release-tools-bot` with passing CI pipelines and **when all backports including the MR targeting master are ready.**--## AppSec checklist--- [ ] Assign the right [AppSecWeight](https://handbook.gitlab.com/handbook/security/product-security/application-security/milestone-planning/#weight-labels) label-- [ ] Update the `~AppSecWorkflow::in-progress` to `~AppSecWorkflow::complete`--/label ~security--<!-- AppSec specific labels -->--/label ~"Division::Security" ~"Department::Product Security" ~"Application Security Team"-/label ~"AppSecWorkflow::in-progress" ~"AppSecWorkType::VulnFixVerification"-/label ~"AppSecPriority::1" <!-- This is always a priority to review for us to ensure the fix is good and the release is done on time -->--[GitLab Security]: https://gitlab.com/gitlab-org/security/gitlab-[approval guidelines]: https://docs.gitlab.com/development/code_review/#approval-guidelines-[Code Review process]: https://docs.gitlab.com/development/code_review/-[quick actions]: https://docs.gitlab.com/user/project/quick_actions/#issues-merge-requests-and-epics-[CHANGELOG entry]: https://docs.gitlab.com/development/changelog/#overviewdiff --git a/.ko.yaml b/.ko.yamldeleted file mode 100644index 31c0464..0000000--- a/.ko.yaml+++ /dev/null@@ -1,9 +0,0 @@-defaultBaseImage: ko.local/gitaly-base:latest--builds:-- id: cmd/gitaly- env:- - CGO_ENABLED=0- - GOOS=linux- ldflags:- - -X github.com/feder-cr/open_scion/internal/version.version={{.Env.GITALY_VERSION}}diff --git a/.labkit_logging_todo.yml b/.labkit_logging_todo.ymldeleted file mode 100644index 999367c..0000000--- a/.labkit_logging_todo.yml+++ /dev/null@@ -1,147 +0,0 @@-# LabKit Logging Field Standardization TODO-# AUTO-GENERATED FILE. DO NOT EDIT MANUALLY.-#-# This file tracks deprecated logging fields that need to be migrated to-# standard fields defined in gitlab.com/gitlab-org/labkit/v2/fields.-# Each offense represents a file using a deprecated field name.-#-# How to fix:-# Replace the string literal with the constant from the fields package.-# e.g. log.WithField("source_ip", ...) → log.WithField(fields.RemoteIP, ...)-#-# Adding offenses when an immediate fix is not possible:-# go get gitlab.com/gitlab-org/labkit/v2/cmd/validate-log-fields-# go run gitlab.com/gitlab-org/labkit/v2/cmd/validate-log-fields -update-todo-# go mod tidy-#-# Regenerate entire TODO:-# Delete this file, then run the command above.------offenses:- - callsite: internal/burdenmonitor/load_shedder.go- deprecated_field: reason- standard_field: fields.LogMessage- - callsite: internal/burdenmonitor/poller.go- deprecated_field: error- standard_field: fields.ErrorMessage- - callsite: internal/cli/gitaly/serve.go- deprecated_field: duration_ms- standard_field: fields.DurationS- - callsite: internal/command/subprocess_logger.go- deprecated_field: component- standard_field: fields.ServiceName- - callsite: internal/git/housekeeping/manager/optimize_repository.go- deprecated_field: total_time- standard_field: fields.DurationS- - callsite: internal/git/trace2hooks/log_exporter.go- deprecated_field: component- standard_field: fields.ServiceName- - callsite: internal/git/trace2hooks/log_exporter.go- deprecated_field: start_time- standard_field: fields.Timestamp- - callsite: internal/gitaly/maintenance/daily.go- deprecated_field: actual_duration- standard_field: fields.DurationS- - callsite: internal/gitaly/maintenance/optimize.go- deprecated_field: time_ms- standard_field: fields.DurationS- - callsite: internal/gitaly/server/server.go- deprecated_field: component- standard_field: fields.ServiceName- - callsite: internal/gitaly/service/repository/snapshot.go- deprecated_field: error- standard_field: fields.ErrorMessage- - callsite: internal/gitaly/service/smarthttp/cache.go- deprecated_field: service- standard_field: fields.ServiceName- - callsite: internal/gitaly/service/smarthttp/inforefs.go- deprecated_field: service- standard_field: fields.ServiceName- - callsite: internal/gitaly/storage/keyvalue/databasemgr/db.go- deprecated_field: component- standard_field: fields.ServiceName- - callsite: internal/gitaly/storage/storagemgr/partition/migration/manager.go- deprecated_field: duration- standard_field: fields.DurationS- - callsite: internal/gitaly/storage/storagemgr/partition/snapshot/manager.go- deprecated_field: component- standard_field: fields.ServiceName- - callsite: internal/gitaly/transaction/manager.go- deprecated_field: component- standard_field: fields.ServiceName- - callsite: internal/gitlab/client/gitlabnet.go- deprecated_field: duration_ms- standard_field: fields.DurationS- - callsite: internal/gitlab/http_client.go- deprecated_field: component- standard_field: fields.ServiceName- - callsite: internal/grpc/backchannel/client.go- deprecated_field: component- standard_field: fields.ServiceName- - callsite: internal/grpc/backchannel/server.go- deprecated_field: component- standard_field: fields.ServiceName- - callsite: internal/grpc/middleware/panichandler/panic_handler.go- deprecated_field: error- standard_field: fields.ErrorMessage- - callsite: internal/praefect/datastore/collector.go- deprecated_field: component- standard_field: fields.ServiceName- - callsite: internal/praefect/datastore/collector_test.go- deprecated_field: component- standard_field: fields.ServiceName- - callsite: internal/praefect/datastore/collector_test.go- deprecated_field: error- standard_field: fields.ErrorMessage- - callsite: internal/praefect/datastore/listener.go- deprecated_field: component- standard_field: fields.ServiceName- - callsite: internal/praefect/datastore/storage_provider.go- deprecated_field: component- standard_field: fields.ServiceName- - callsite: internal/praefect/datastore/storage_provider_test.go- deprecated_field: component- standard_field: fields.ServiceName- - callsite: internal/praefect/datastore/storage_provider_test.go- deprecated_field: error- standard_field: fields.ErrorMessage- - callsite: internal/praefect/nodes/health_manager.go- deprecated_field: component- standard_field: fields.ServiceName- - callsite: internal/praefect/nodes/health_manager.go- deprecated_field: error- standard_field: fields.ErrorMessage- - callsite: internal/praefect/reconciler/reconciler.go- deprecated_field: component- standard_field: fields.ServiceName- - callsite: internal/praefect/replicator.go- deprecated_field: component- standard_field: fields.ServiceName- - callsite: internal/praefect/repocleaner/action_log.go- deprecated_field: component- standard_field: fields.ServiceName- - callsite: internal/praefect/repocleaner/action_log_test.go- deprecated_field: component- standard_field: fields.ServiceName- - callsite: internal/praefect/repocleaner/repository.go- deprecated_field: component- standard_field: fields.ServiceName- - callsite: internal/praefect/repocleaner/repository_test.go- deprecated_field: component- standard_field: fields.ServiceName- - callsite: internal/praefect/server.go- deprecated_field: component- standard_field: fields.ServiceName- - callsite: internal/praefect/transactions/manager.go- deprecated_field: component- standard_field: fields.ServiceName- - callsite: internal/praefect/verifier.go- deprecated_field: error- standard_field: fields.ErrorMessage- - callsite: internal/tempdir/clean.go- deprecated_field: time_ms- standard_field: fields.DurationS- - callsite: internal/testhelper/testserver/praefect.go- deprecated_field: component- standard_field: fields.ServiceNamediff --git a/.mailmap b/.mailmapdeleted file mode 100644index ae8e282..0000000--- a/.mailmap+++ /dev/null@@ -1,10 +0,0 @@-Ahmad Sherif <[email protected]> <[email protected]>-Ahmad Sherif <[email protected]> <[email protected]>-Jacob Vosmaer <[email protected]> Jacob Vosmaer (GitLab) <[email protected]>-Andrew Newdigate <[email protected]> <[email protected]>-Zeger-Jan van de Weg<[email protected]> <[email protected]>-Alejandro Rodríguez <[email protected]> Alejandro Rodríguez <[email protected]>-Kim Carlbäcker <[email protected]> Kim "BKC" Carlbäcker <[email protected]>-Kim Carlbäcker <[email protected]> Kim Carlbäcker <[email protected]>-Sytse Sijbrandij <[email protected]> Sid Sijbrandij <[email protected]>-Sytse Sijbrandij <[email protected]> sytses <[email protected]>diff --git a/.markdownlint-cli2.yaml b/.markdownlint-cli2.yamldeleted file mode 100644index 3188eb7..0000000--- a/.markdownlint-cli2.yaml+++ /dev/null@@ -1,165 +0,0 @@-----# See https://github.com/DavidAnson/markdownlint/blob/main/doc/Rules.md for explanations of each rule-config:- # First, set the default- default: true-- # Per-rule settings in alphabetical order- code-block-style: # MD046- style: "fenced"- emphasis-style: false # MD049- header-style: # MD003- style: "atx"- hr-style: # MD035- style: "---"- line-length: # MD013- code_blocks: false- tables: false- headings: true- heading_line_length: 100- line_length: 800- no-duplicate-heading: # MD024- siblings_only: true- no-emphasis-as-heading: false # MD036- no-inline-html: false # MD033- no-trailing-punctuation: # MD026- punctuation: ".,;:!。,;:!"- no-trailing-spaces: false # MD009- ol-prefix: # MD029- style: "one"- reference-links-images: false # MD052- ul-style: # MD004- style: "dash"- table-column-style: false # MD060-- # Keep this item last due to length- proper-names: # MD044- code_blocks: false- html_elements: false- names:- [- "Akismet",- "Alertmanager",- "AlmaLinux",- "API",- "Asana",- "Auth0",- "Azure",- "Bamboo",- "Bitbucket",- "Bugzilla",- "CAS",- "CentOS",- "Consul",- "Debian",- "DevOps",- "Docker",- "DockerSlim",- "Elasticsearch",- "Facebook",- "fastlane",- "fluent-plugin-redis-slowlog",- "GDK",- "Geo",- "Git LFS",- "git-annex",- "git-credential-oauth",- "git-sizer",- "Git",- "Gitaly",- "GitHub",- "gitlab-duo",- "gitlab.vim",- "GitLab chart",- "GitLab Geo",- "GitLab Monitor",- "GitLab Operator",- "GitLab Pages",- "GitLab Rails",- "GitLab Runner",- "GitLab Shell",- "GitLab Workhorse",- "GitLab",- "Gitleaks",- "Gmail",- "Google",- "Grafana",- "Gzip",- "Helm",- "HipChat",- "ID",- "IP",- "Ingress",- "jasmine-jquery",- "JavaScript",- "Jaeger",- "Jenkins",- "Jira",- "Jira Cloud",- "Jira Server",- "jQuery",- "JSON",- "JupyterHub",- "Karma",- "Kerberos",- "Knative",- "Kubernetes",- "LDAP",- "Let's Encrypt",- "Markdown",- "markdownlint",- "Mattermost",- "Microsoft",- "minikube",- "MinIO",- "ModSecurity",- "Neovim",- "NGINX Ingress",- "NGINX",- "OAuth",- "OAuth 2",- "OmniAuth",- "OpenID",- "OpenShift",- "PgBouncer",- "Postfix",- "PostgreSQL",- "PowerShell",- "Praefect",- "Prometheus",- "Puma",- "puma-worker-killer",- "Python",- "Rake",- "Redis",- "Redmine",- "reCAPTCHA",- "Ruby",- "runit",- "Salesforce",- "SAML",- "Sendmail",- "Sentry",- "Service Desk",- "Sidekiq",- "Shibboleth",- "Slack",- "SMTP",- "SpotBugs",- "SSH",- "Tiller",- "Tiptap",- "TOML",- "Trello",- "Trello Power-Ups",- "TypeScript",- "Twitter",- "Ubuntu",- "Ultra Auth",- "Unicorn",- "unicorn-worker-killer",- "URL",- "WebdriverIO",- "YAML",- "YouTrack"- ]diff --git a/.ruby-version b/.ruby-versiondeleted file mode 100644index 84d6c67..0000000--- a/.ruby-version+++ /dev/null@@ -1 +0,0 @@-3.4.10diff --git a/.vale.ini b/.vale.inideleted file mode 100644index 04092d3..0000000--- a/.vale.ini+++ /dev/null@@ -1,12 +0,0 @@-# Vale configuration file.-#-# For more information, see https://vale.sh/docs/vale-ini.--StylesPath = doc/.vale-MinAlertLevel = suggestion--[*.md]-BasedOnStyles = gitlab_base--# Ignore SVG markup-TokenIgnores = (\*\*\{\w*\}\*\*)diff --git a/AGENTS.md b/AGENTS.mdindex 1de6798..3d80bc0 100644--- a/AGENTS.md+++ b/AGENTS.md@@ -9,14 +9,14 @@ This document follows the [AGENTS.md](https://agents.md) format. [Agent Skills](Read the relevant context before starting a task:- Working in the codebase: layout, build/test/lint commands, Go/error/proto conventions →- [doc/development.md](doc/development.md), [STYLE.md](STYLE.md)+ [doc/development.md](doc/development.md), [STYLE.md](doc/contributing/STYLE.md)- Adding or changing a CLI command or flag (`gitaly`/`praefect`) → [doc/help_text_style_guide.md](doc/help_text_style_guide.md)-- Commits, merge requests, and code review → [CONTRIBUTING.md](CONTRIBUTING.md), [REVIEWING.md](REVIEWING.md), [STYLE.md](STYLE.md)-- Running or writing tests → [doc/development.md](doc/development.md), [STYLE.md](STYLE.md)+- Commits, merge requests, and code review → [CONTRIBUTING.md](CONTRIBUTING.md), [REVIEWING.md](doc/contributing/REVIEWING.md), [STYLE.md](doc/contributing/STYLE.md)+- Running or writing tests → [doc/development.md](doc/development.md), [STYLE.md](doc/contributing/STYLE.md)- Observability: logs, metrics, traces, and Git-level instrumentation →[doc/observability.md](doc/observability.md) for which signal answers which question, then[doc/logging.md](doc/logging.md) or [doc/trace2_integration.md](doc/trace2_integration.md).- Logging *conventions* live in [`STYLE.md`](STYLE.md#logging) instead+ Logging *conventions* live in [`STYLE.md`](doc/contributing/STYLE.md#logging) instead- Architecture → [doc/DESIGN.md](doc/DESIGN.md), [doc/design_ha.md](doc/design_ha.md)## Always-on rulesdiff --git a/CONTRIBUTING.md b/CONTRIBUTING.mdindex 87b7714..9d87d22 100644--- a/CONTRIBUTING.md+++ b/CONTRIBUTING.md@@ -44,7 +44,7 @@ available at [https://contributor-covenant.org/version/1/1/0/](https://contribut## Style guides-For Gitaly code style, see [`STYLE.md`](STYLE.md).+For Gitaly code style, see [`STYLE.md`](doc/contributing/STYLE.md).For help text style guidance for this project, see the [help text style guide](doc/help_text_style_guide.md).@@ -169,7 +169,7 @@ pass formatting, linting, and our test suite.### Changelog-Gitaly keeps a [changelog](CHANGELOG.md) that is generated:+Gitaly keeps a [changelog](doc/upstream/CHANGELOG-GITALY.md) that is generated:- When a new release is created.- From commit messages where a specific trailer is used.@@ -246,7 +246,7 @@ member is equally responsible for a successful master pipeline and fixing securiissues.Merge requests are approved by [Gitaly team members](https://gitlab.com/groups/gl-gitaly/-/group_members);-[REVIEWING.md](REVIEWING.md) describes how many approvals a change needs and how+[REVIEWING.md](doc/contributing/REVIEWING.md) describes how many approvals a change needs and howreviewers are chosen.[eng-process]: https://handbook.gitlab.com/handbook/engineering/workflow/@@ -254,7 +254,7 @@ reviewers are chosen.### Review Process-See [REVIEWING.md](REVIEWING.md).+See [REVIEWING.md](doc/contributing/REVIEWING.md).## Gitaly Developer Quick-start Guidediff --git a/Dangerfile b/Dangerfiledeleted file mode 100644index 39c6730..0000000--- a/Dangerfile+++ /dev/null@@ -1,19 +0,0 @@-# frozen_string_literal: true--require 'gitlab-dangerfiles'--Gitlab::Dangerfiles.for_project(self) do |gitlab_dangerfiles|- gitlab_dangerfiles.config.files_to_category = {- %r{\Adoc/.*(\.(md|png|gif|jpg))\z} => :docs,- %r{\A(CONTRIBUTING|LICENSE|README|REVIEWING|STYLE)(\.md)?\z} => :docs,-- %r{.*} => [nil]- }.freeze-- Dir.each_child('danger/rules') do |rule|- danger.import_dangerfile(path: "danger/rules/#{rule}")- end-- gitlab_dangerfiles.import_plugins- gitlab_dangerfiles.import_dangerfiles(except: %w[changes_size])-enddiff --git a/Dockerfile.base b/Dockerfile.basedeleted file mode 100644index 8b76cfc..0000000--- a/Dockerfile.base+++ /dev/null@@ -1,47 +0,0 @@-# Base image for ko builds with all required dependencies-FROM registry.access.redhat.com/ubi9/ubi-minimal:9.8--ARG GITLAB_USER=git-ARG UID=1000--# Install all required dependencies including git-RUN microdnf update -y && \- microdnf install -y --nodocs --best --setopt=install_weak_deps=0 \- shadow-utils \- git \- git-core \- iproute \- libicu \- openssh-clients \- bzip2 \- gzip \- procps-ng \- tzdata && \- microdnf clean all--# Create git user and required directories-RUN adduser -u ${UID} -m ${GITLAB_USER} && \- mkdir -p \- /etc/gitaly \- /var/log/gitaly \- /home/${GITLAB_USER}/repositories \- /home/${GITLAB_USER}/custom_hooks \- /srv/gitlab-shell && \- touch /var/log/gitaly/gitaly.log && \- touch /var/log/gitaly/gitlab-shell.log && \- chown -R ${UID}:0 \- /etc/gitaly \- /var/log/gitaly \- /home/${GITLAB_USER}/repositories \- /home/${GITLAB_USER}/custom_hooks \- /srv/gitlab-shell \- /home/${GITLAB_USER} && \- chmod -R g=u \- /etc/gitaly \- /var/log/gitaly \- /home/${GITLAB_USER}/repositories \- /home/${GITLAB_USER}/custom_hooks \- /srv/gitlab-shell \- /home/${GITLAB_USER}--USER ${UID}diff --git a/Dockerfile.bundled-git b/Dockerfile.bundled-gitdeleted file mode 100644index 6b51802..0000000--- a/Dockerfile.bundled-git+++ /dev/null@@ -1,20 +0,0 @@-# Compiles the bundled Git binaries that get embedded into the gitaly-# image. Uses ubi9-minimal to match the base image gitaly actually-# runs on.-FROM registry.access.redhat.com/ubi9/ubi-minimal:9.8--RUN microdnf install -y --nodocs --setopt=install_weak_deps=0 \- diffutils \- gcc \- git \- libcurl-devel \- meson \- ninja-build \- openssl-devel \- pcre2-devel \- pkg-config \- tar \- zlib-devel && \- microdnf clean all--WORKDIR /srcdiff --git a/Makefile b/Makefileindex d5f7caa..3d65d83 100644--- a/Makefile+++ b/Makefile@@ -512,15 +512,15 @@ lint-fix: ${GOLANGCI_LINT} ${GITALY_PACKED_EXECUTABLES} ${GIT_PACKED_EXECUTABLES.PHONY: lint-docs## Run markdownlint-cli2-config to lint the documentation and lychee to check for broken links.lint-docs:- ${Q}markdownlint-cli2 AGENTS.md CLAUDE.md README.md REVIEWING.md STYLE.md doc/**/*.md doc/*.md skills/**/*.md- ${Q}vale --minAlertLevel error AGENTS.md CLAUDE.md README.md REVIEWING.md STYLE.md doc skills+ ${Q}markdownlint-cli2 AGENTS.md CLAUDE.md README.md doc/**/*.md doc/*.md+ ${Q}vale --minAlertLevel error AGENTS.md CLAUDE.md README.md doc${Q}lychee --version- ${Q}lychee --offline --include-fragments AGENTS.md CLAUDE.md README.md REVIEWING.md STYLE.md doc/**/*.md doc/*.md skills/**/*.md+ ${Q}lychee --offline --include-fragments AGENTS.md CLAUDE.md README.md doc/**/*.md doc/*.md.PHONY: lint-docs-fix## Run markdownlint-cli2-config to lint and fix the documentation.lint-docs-fix:- ${Q}markdownlint-cli2 AGENTS.md CLAUDE.md README.md REVIEWING.md STYLE.md doc/**.md skills/**/*.md --fix+ ${Q}markdownlint-cli2 AGENTS.md CLAUDE.md README.md doc/**.md --fix.PHONY: lint-gitaly-linters## Test Go tests in tools/golangci-lint/gitaly folderdiff --git a/README.md b/README.mdindex 9ce95d9..0f0df49 100644--- a/README.md+++ b/README.md@@ -133,5 +133,5 @@ It started as a fork of [Gitaly](https://gitlab.com/gitlab-org/gitaly), GitLab'sGit RPC server (MIT, Copyright GitLab Inc.), which provides the Git plumbingunderneath; every upstream Gitaly RPC keeps working. The non-MIT `ee/` directoryof upstream Gitaly is not part of this repository. Gitaly's own README is kept in-[doc/UPSTREAM-GITALY.md](doc/UPSTREAM-GITALY.md), and third-party notices are in+[doc/UPSTREAM-GITALY.md](doc/upstream/README-GITALY.md), and third-party notices are in[NOTICE](NOTICE).diff --git a/caproni-project.yaml b/caproni-project.yamldeleted file mode 100644index e6c8e00..0000000--- a/caproni-project.yaml+++ /dev/null@@ -1,19 +0,0 @@-# yaml-language-server: $schema=https://gitlab.com/gitlab-org/caproni/-/raw/main/schema/caproni-project.schema.json-# Per-repo config: gitaly's skaffold reloader and prepare hook.-# Infra, deployers, and cluster live in `gitlab-caproni`.-#-# Loaded when the gitaly repository is in edit mode. Set-# repositories.gitaly.edit_mode: true in caproni.local.yaml.--lifecycle_hooks:- - type: edit_mode_start- script: .gitlab/caproni/skaffold-prepare.sh--reloaders:- gitaly:- type: skaffold- repository: gitaly- edit_mode: false- skaffold:- namespace: gitlab- config_file: skaffold.yamldiff --git a/danger/rules/changes_size/Dangerfile b/danger/rules/changes_size/Dangerfiledeleted file mode 100644index 7e098ac..0000000--- a/danger/rules/changes_size/Dangerfile+++ /dev/null@@ -1,22 +0,0 @@-# frozen_string_literal: true--thresholds = helper.config.code_size_thresholds--old_names = git.renamed_files.map {|f| f[:before] }-new_names = git.renamed_files.map {|f| f[:after] }--lines_changed = (git.added_files + git.modified_files + new_names - old_names).sum do |file|- next 0 if file.end_with?('.pb.go') || file.end_with?('_pb.rb')-- info = git.info_for_file(file)-- next 0 if info.nil?-- info[:insertions]-end--if lines_changed > thresholds[:high]- warn "This merge request is definitely too big (#{lines_changed} lines changed), please split it into multiple merge requests."-elsif lines_changed > thresholds[:medium]- warn "This merge request is quite big (#{lines_changed} lines changed), please consider splitting it into multiple merge requests."-enddiff --git a/danger/rules/labels/Dangerfile b/danger/rules/labels/Dangerfiledeleted file mode 100644index 9c0363d..0000000--- a/danger/rules/labels/Dangerfile+++ /dev/null@@ -1,28 +0,0 @@-INHERITABLE_LABELS = %w[- type::bug- type::feature- type::maintenance- feature::addition- security- performance- Deliverable- group::gitaly::cluster- group::gitaly::git- group::gitaly-]--def inherited_labels- gitlab.api- .merge_request_closes_issues( gitlab.mr_json['project_id'], gitlab.mr_json['iid'])- .flat_map { |i| i.labels }- .compact- .uniq- .select { |label| INHERITABLE_LABELS.include?(label) }-end--GITALY_TEAM = gitlab.api.all_group_members("gl-gitaly").flat_map{ |i| i.username }--required_labels = []-required_labels << "group::gitaly" if GITALY_TEAM.include?(gitlab.mr_author)--helper.labels_to_add.concat(required_labels | inherited_labels)diff --git a/danger/rules/milestones/Dangerfile b/danger/rules/milestones/Dangerfiledeleted file mode 100644index 30ea440..0000000--- a/danger/rules/milestones/Dangerfile+++ /dev/null@@ -1,21 +0,0 @@-unless gitlab.mr_json["milestone"]- issues = gitlab.api.merge_request_closes_issues(helper.mr_target_project_id, helper.mr_iid)-- milestone_id = issues- .map { |i| i.milestone }- .compact- .reject { |m| m.state == "closed" || m.due_date.nil? }- .sort_by { |m| m.due_date }- .first- &.id-- if milestone_id- gitlab.api.update_merge_request(- helper.mr_target_project_id,- helper.mr_iid,- milestone_id: milestone_id- )- else- warn "No milestone was set, nor could it be detected from the issues this merge request closes."- end-enddiff --git a/doc/configuration/praefect.md b/doc/configuration/praefect.mdindex 27ad7a7..1439ffc 100644--- a/doc/configuration/praefect.md+++ b/doc/configuration/praefect.md@@ -38,7 +38,7 @@ name = 'praefect'token = 'secret_token'```-An example [config TOML](../../config.praefect.toml.example) is stored in this repository.+An example [config TOML](../../examples/gitaly/config.praefect.toml.example) is stored in this repository.## Timeout recommendationdiff --git a/REVIEWING.md b/doc/contributing/REVIEWING.mdsimilarity index 98%rename from REVIEWING.mdrename to doc/contributing/REVIEWING.mdindex a6ad588..190627d 100644--- a/REVIEWING.md+++ b/doc/contributing/REVIEWING.md@@ -67,7 +67,7 @@ going on.After you have read the diff of your MR in the GitLab UI, take a momentto consider what the title of your MR should be, and update the MR title if needed.-Imagine your title appearing as a [CHANGELOG](CHANGELOG.md) entry.+Imagine your title appearing as a [CHANGELOG](../upstream/CHANGELOG-GITALY.md) entry.Will your title give a good indication of what changed?#### Your MR description should pass the lunch testdiff --git a/STYLE.md b/doc/contributing/STYLE.mdsimilarity index 100%rename from STYLE.mdrename to doc/contributing/STYLE.mddiff --git a/doc/development.md b/doc/development.mdindex 182a364..2addeb5 100644--- a/doc/development.md+++ b/doc/development.md@@ -4,7 +4,7 @@ This document provides an overview of how to work in the Gitaly codebase. This iMore specific guidance is available in the documents linked to from here.-[STYLE.md](../STYLE.md) defines general Go style and conventions.+[STYLE.md](contributing/STYLE.md) defines general Go style and conventions.## Repository mapdiff --git a/doc/logging.md b/doc/logging.mdindex 0262844..3c474ea 100644--- a/doc/logging.md+++ b/doc/logging.md@@ -2,7 +2,7 @@How Gitaly emits logs. The conventions to follow when writing log statements, such ascontext-based logging, snake case fields, and the `log.Logger` interface, are in-[`STYLE.md`](../STYLE.md#logging).+[`STYLE.md`](contributing/STYLE.md#logging).## Where logs godiff --git a/doc/observability.md b/doc/observability.mdindex 3fda8d7..d307576 100644--- a/doc/observability.md+++ b/doc/observability.md@@ -16,7 +16,7 @@ Git processes emit a fifth, narrower signal of their own. See [Trace2 integratioGitaly writes structured logs to `stdout`, with an access log entry per RPC and a correlation ID onevery request-scoped message. [Logging in Gitaly](logging.md) covers the configuration, what gets-logged, and how correlation IDs tie a request together. [`STYLE.md`](../STYLE.md#logging) covers the+logged, and how correlation IDs tie a request together. [`STYLE.md`](contributing/STYLE.md#logging) covers theconventions to follow when adding log statements.## Metricsdiff --git a/CHANGELOG.md b/doc/upstream/CHANGELOG-GITALY.mdsimilarity index 100%rename from CHANGELOG.mdrename to doc/upstream/CHANGELOG-GITALY.mddiff --git a/doc/UPSTREAM-GITALY.md b/doc/upstream/README-GITALY.mdsimilarity index 94%rename from doc/UPSTREAM-GITALY.mdrename to doc/upstream/README-GITALY.mdindex ace7e22..559715c 100644--- a/doc/UPSTREAM-GITALY.md+++ b/doc/upstream/README-GITALY.md@@ -1,7 +1,7 @@# Upstream Gitaly READMEopen_scion started as a fork of Gitaly. This is the README of upstream Gitaly,-kept for reference; see the [open_scion README](../README.md) for this project.+kept for reference; see the [open_scion README](../../README.md) for this project.## Gitaly@@ -46,7 +46,7 @@ The administration and reference guide is [documented in the GitLab project](htt## Contributing-See [CONTRIBUTING.md](../CONTRIBUTING.md) and a [list of quick win issues](https://gitlab.com/gitlab-org/gitaly/-/issues/?sort=due_date&state=opened&label_name%5B%5D=quick%20win&first_page_size=20).+See [CONTRIBUTING.md](../../CONTRIBUTING.md) and a [list of quick win issues](https://gitlab.com/gitlab-org/gitaly/-/issues/?sort=due_date&state=opened&label_name%5B%5D=quick%20win&first_page_size=20).## Name@@ -69,7 +69,7 @@ Gitaly offers a High Availability solution known as Gitaly Cluster ([product doc## Further reading-More about the project and its processes is [detailed in the docs](README.md).+More about the project and its processes is [detailed in the docs](../README.md).## Distributed Tracingdiff --git a/config.praefect.toml.example b/examples/gitaly/config.praefect.toml.examplesimilarity index 100%rename from config.praefect.toml.examplerename to examples/gitaly/config.praefect.toml.examplediff --git a/config.toml.example b/examples/gitaly/config.toml.examplesimilarity index 100%rename from config.toml.examplerename to examples/gitaly/config.toml.examplediff --git a/k8s/configmap.yaml b/k8s/configmap.yamldeleted file mode 100644index 828c2fe..0000000--- a/k8s/configmap.yaml+++ /dev/null@@ -1,74 +0,0 @@-apiVersion: v1-kind: ConfigMap-metadata:- annotations:- meta.helm.sh/release-name: gitlab- meta.helm.sh/release-namespace: gitlab- labels:- app: gitaly- app.kubernetes.io/managed-by: Helm- heritage: Helm- release: gitlab- name: gitlab-gitaly- namespace: gitlab-data:- config.toml: |- # The directory where Gitaly's executables are stored- bin_dir = "/usr/local/bin"-- # listen on a TCP socket- listen_addr = ":8075"-- # Directory where internal sockets reside- # note: no value will result in a `/tmp/gitlab-internal-*` path- # internal_socket_dir = "/home/git"-- # If metrics collection is enabled, inform gitaly about that- prometheus_listen_addr = ":9236"-- # Graceful shutdown timeout, how long to wait for in-flight requests to complete- graceful_restart_timeout = "25s"-- # Storage configuration- [[storage]]- name = "default"- path = "/home/git/repositories"-- [logging]- format = "json"- dir = "/var/log/gitaly"-- [auth]- # Read authentication token from file mounted via Kubernetes Secret- token_file = "/etc/gitlab-secrets/gitaly/gitaly_token"-- [git]- ignore_gitconfig = true-- [gitlab-shell]- # The directory where gitlab-shell is installed- dir = "/srv/gitlab-shell"-- [gitlab]- # location of shared secret for GitLab Shell / API interaction- secret_file = "/etc/gitlab-secrets/shell/.gitlab_shell_secret"- # URL of API- url = "http://gitlab-webservice-default.gitlab.svc:8181/"-- [gitlab.http-settings]- # read_timeout = 300- # user = someone- # password = somepass- # ca_file = /etc/ssl/cert.pem- # ca_path = /etc/pki/tls/certs-- [hooks]- # directory containing custom hooks- custom_hooks_dir = "/home/git/custom_hooks"- configure: |- set -e- mkdir -p /init-secrets/gitaly /init-secrets/shell- cp -v -r -L /init-config/.gitlab_shell_secret /init-secrets/shell/.gitlab_shell_secret- chmod 600 /init-secrets/shell/.gitlab_shell_secret- cp -v -r -L /init-config/gitaly_token /init-secrets/gitaly/gitaly_token- chmod 600 /init-secrets/gitaly/gitaly_tokendiff --git a/scripts/security-harness b/scripts/security-harnessdeleted file mode 100755index 1df3610..0000000--- a/scripts/security-harness+++ /dev/null@@ -1,100 +0,0 @@-#!/usr/bin/env ruby--# frozen_string_literal: true--require 'digest'-require 'fileutils'--if ENV['NO_COLOR']- SHELL_RED = ''- SHELL_GREEN = ''- SHELL_YELLOW = ''- SHELL_CLEAR = ''-else- SHELL_RED = "\e[1;31m"- SHELL_GREEN = "\e[1;32m"- SHELL_YELLOW = "\e[1;33m"- SHELL_CLEAR = "\e[0m"-end--HOOK_PATH = File.expand_path("../.git/hooks/pre-push", __dir__)-HOOK_DATA = <<~HOOK- #!/usr/bin/env bash-- set -e-- url="$2"- harness=`dirname "$0"`/../security_harness-- if [ -e "$harness" ]- then- if [[ "$url" != *"gitlab-org/security/"* ]]- then- echo "Pushing to remotes other than gitlab.com/gitlab-org/security has been disabled!"- echo "Run scripts/security-harness to disable this check."- echo-- exit 1- fi- fi-HOOK--def write_hook- FileUtils.mkdir_p(File.dirname(HOOK_PATH))- File.open(HOOK_PATH, 'w') do |file|- file.write(HOOK_DATA)- end- File.chmod(0755, HOOK_PATH)-end--# Toggle the harness on or off-def toggle- harness_path = File.expand_path('../.git/security_harness', __dir__)-- if File.exist?(harness_path)- FileUtils.rm(harness_path)-- puts "#{SHELL_YELLOW}Security harness removed -- you can now push to all remotes.#{SHELL_CLEAR}"- else- FileUtils.touch(harness_path)-- puts "#{SHELL_GREEN}Security harness installed -- you will only be able to push to gitlab.com/gitlab-org/security!#{SHELL_CLEAR}"- end-end--# If we were to change the script and then check for a pre-existing hook before-# writing, the check would fail even if the user had an unmodified version of-# the old hook. Checking previous version hashes allows us to safely overwrite a-# script that differs from the current version, as long as it's an old one and-# not custom.-def previous_version?(dest_sum)- # SHA256 hashes of previous iterations of the script contained in `DATA`- %w[- 010bf0363a911ebab2bd5728d80795ed02388da51815f0b2530d08ae8ac574f0- ].include?(dest_sum)-end--if !File.exist?(HOOK_PATH)- write_hook- toggle-else- # Deal with a pre-existing hook- source_sum = Digest::SHA256.hexdigest(HOOK_DATA)- dest_sum = Digest::SHA256.file(HOOK_PATH).hexdigest-- if previous_version?(dest_sum)- # Upgrading from a previous version, update in-place- write_hook- toggle- elsif source_sum != dest_sum- # Pre-existing hook we didn't create; do nothing- puts "#{SHELL_RED}#{HOOK_PATH} exists and is different from our hook!"- puts "Remove it and re-run this script to continue.#{SHELL_CLEAR}"-- exit 1- else- # No hook update needed, just toggle- toggle- end-end-diff --git a/skaffold.yaml b/skaffold.yamldeleted file mode 100644index 9b35d48..0000000--- a/skaffold.yaml+++ /dev/null@@ -1,91 +0,0 @@-apiVersion: skaffold/v4beta11-kind: Config-metadata:- name: gitaly--build:- artifacts:- - image: ko.local/gitaly-base- docker:- dockerfile: Dockerfile.base- # .ko.yaml builds on :latest, but skaffold tags this image with the git tag.- # Lets re-tag the image with latest- hooks:- after:- - command: ["sh", "-c", "docker tag \"$SKAFFOLD_IMAGE\" ko.local/gitaly-base:latest"]- - image: registry.gitlab.com/gitlab-org/build/cng/gitaly- # Build the base image first; skaffold otherwise builds both in parallel- # and ko fails to find ko.local/gitaly-base:latest on a fresh Docker daemon- # cache. In `.ko.yaml`, it builds from `defaultBaseImage: ko.local/gitaly-base:latest`.`- requires:- - image: ko.local/gitaly-base- ko:- main: ./cmd/gitaly- dependencies:- paths:- - cmd/gitaly- - cmd/gitaly-hooks- - cmd/gitaly-ssh- - cmd/gitaly-lfs-smudge- - cmd/gitaly-gpg- - internal- - proto/go- - go.mod- - go.sum- - _build/bin/gitaly-hooks- - _build/bin/gitaly-ssh- - _build/bin/gitaly-lfs-smudge- - _build/bin/gitaly-gpg- - _build/bin/gitaly-git-master- - _build/bin/gitaly-git-prev- - _build/bin/gitaly-git-mvcc- - _build/bin/gitaly-git-http-backend-master- - _build/bin/gitaly-git-http-backend-prev- - _build/bin/gitaly-git-http-backend-mvcc- - _build/bin/gitaly-git-remote-http-master- - _build/bin/gitaly-git-remote-http-prev- - _build/bin/gitaly-git-remote-http-mvcc- env:- - CGO_ENABLED=0- ldflags:- - -X github.com/feder-cr/open_scion/internal/version.version=dev-local- hooks:- before:- - command:- - sh- - -c- - bash ./.gitlab/caproni/build-gitaly-helpers.sh- after:- - command:- - sh- - -c- - bash ./.gitlab/caproni/import-gitaly-images.sh- local:- push: false- useBuildkit: true--manifests:- rawYaml:- - k8s/*.yaml--deploy:- kubectl:- defaultNamespace: gitlab- # Without this, helm won't upgrade gitaly after skaffold has deployed it.- # `caproni up` resets gitaly to the chart's image, so run `caproni run` to- # start edit mode again.- flags:- apply:- - --server-side- - --force-conflicts- - --field-manager=helm--portForward:-- resourceType: service- resourceName: gitlab-gitaly- port: 8075- localPort: 8075-- resourceType: service- resourceName: gitlab-gitaly- port: 9236- localPort: 9236diff --git a/skills/gitaly-rpc-development/SKILL.md b/skills/gitaly-rpc-development/SKILL.mddeleted file mode 100644index 3e47a20..0000000--- a/skills/gitaly-rpc-development/SKILL.md+++ /dev/null@@ -1,117 +0,0 @@-----name: gitaly-rpc-development-description: Use when adding or changing a Gitaly gRPC RPC — editing .proto files, setting RPC annotations, regenerating code with make proto, implementing the handler, and writing tests. Covers the full end-to-end flow and the non-obvious rules.------# Adding or changing a gRPC RPC--The end-to-end workflow for adding or changing a Gitaly RPC.--Gitaly RPCs span three layers: the `.proto` definition, generated Go, and a handler in-`internal/gitaly/service/`. Getting the proto annotations or regeneration step wrong breaks-Praefect routing or CI. Follow this order.--## 1. Define or change the RPC in proto--Service definitions live in `proto/<service>.proto`. Each RPC needs a request and response-message named after it (`FooBar` → `FooBarRequest`, `FooBarResponse`).--```proto-// FetchRemote fetches references from a remote repository into the local repository.-rpc FetchRemote(FetchRemoteRequest) returns (FetchRemoteResponse) {- option (op_type) = {- op: MUTATOR- };-}-```--Rules:--- **Document every service, RPC, message, and field** in comments — enough that a caller need- not read the implementation. Document non-obvious error cases.-- **Name RPCs verb-first:** `GetCommit`, `ListRefs`, `IsRepositoryEmpty`, `CreateRepositoryFromBundle`.- Operation RPCs match the Git verb (`FetchRemote`).-- **Keep messages flat.** Abstract only for real, repeated benefit.-- **Never make backwards-incompatible changes** to an existing RPC or message. Add a new RPC and- deprecate the old one (see [doc/PROCESS.md](../../doc/PROCESS.md)).-- **Pick `string` vs `bytes` by encoding.** A Protobuf `string` must be valid UTF-8, but Git does- not enforce encodings, so most Git data is arbitrary bytes. Use `bytes` for anything Git controls- (`revision`, `ref`, `path` inside a repo). Use `string` only for values Gitaly produces that are- guaranteed UTF-8/ASCII, such as `relative_path` (a path on Gitaly's disk) or `commit_id` (a hex- object ID).-- **Stream unbounded or oversized payloads.** Use a `stream` when the result is an unbounded- sequence (a `repeated` collection that can grow without limit) or a payload that could exceed- gRPC's message-size limit, sending it in batches/chunks. For high-volume byte payloads (e.g. Git- fetch/pack data), Gitaly avoids gRPC messages entirely and uses a **sidechannel**- ([doc/sidechannel.md](../../doc/sidechannel.md)); the full streaming patterns are in- [doc/protobuf.md](../../doc/protobuf.md).--## 2. Set the RPC annotations (required)--Gitaly's middleware reads these annotations (via the proto registry) to locate the target-repository/storage, invalidate caches, apply concurrency limits, and coordinate transactions;-Praefect additionally uses them to route and replicate requests across nodes. Missing or wrong-annotations is a common failure.--- **Operation type** — every RPC: `op: ACCESSOR | MUTATOR | MAINTENANCE`.- - `ACCESSOR`: read-only, no side effects.- - `MUTATOR`: modifies repository state.- - `MAINTENANCE`: best-effort upkeep, non-critical.-- **Scope** (mutators): `option (op_type).scope` of `SERVER | STORAGE | REPOSITORY`.- `REPOSITORY` is the default and needs a repository field tagged `[(target_repository)=true]`:-- ```proto- message FetchRemoteRequest {- Repository repository = 1 [(target_repository)=true];- // ...- }- ```-- `STORAGE` scope needs a storage field tagged `[(storage)=true]`.-- **Mutators must vote transactionally**: compute a deterministic vote over all written data,- with at least a preparatory and a committing vote, so replicas stay consistent. Keep errors- deterministic for the same input.--See [doc/protobuf.md](../../doc/protobuf.md) for the full annotation and streaming reference.--## 3. Regenerate code--```shell-make proto-```--This regenerates Go (`proto/go/gitalypb/`) and Ruby. **Commit the generated code together with the-`.proto` change** — CI's `check-proto` fails otherwise.--## 4. Implement the handler--Handlers live at `internal/gitaly/service/<service>/<rpc>.go` as methods on the service's-`*server`, with a companion `<rpc>_test.go`.--- Validate the request first; return `structerr.NewInvalidArgument(...)` for bad input.-- Spawn Git through `internal/git/gitcmd` (the command factory) with long flags and `flag=value` syntax.-- Use context-based logging: `log.DebugContext(ctx, ...)`.-- Return errors from `internal/structerr` (`NewNotFound`, `NewFailedPrecondition`, `NewInternal`, …),- wrapping causes with `%w`. Avoid `Unavailable` except for retriable network failures.-- For rich, client-parseable errors, attach a proto error-details message (RPC `Frobnicate` → `FrobnicateError`- with a `oneof`).--## 5. Test end to end--Write table-driven tests covering success, failure, and validation. Generate all test data at-runtime with `gittest` helpers (`gittest.CreateRepository`, `WriteCommit`, `WriteBlob`, `WriteRef`,-`WriteTag`, …) — never hardcode object IDs. See [doc/development.md](../../doc/development.md) for-test conventions and how tests run. Run the package you touched:--```shell-TEST_PACKAGES=./internal/gitaly/service/<service> TEST_OPTIONS="-count=1 -run=TestFooBar" make test-```--## Before you finish--```shell-make proto # if any .proto changed (and commit generated code)-make format-make verify # pre-push gate: mod-tidy, NOTICE, proto check, lint-make test-```diff --git a/skills/gitaly-rpc-querying/SKILL.md b/skills/gitaly-rpc-querying/SKILL.mddeleted file mode 100644index 24468e1..0000000--- a/skills/gitaly-rpc-querying/SKILL.md+++ /dev/null@@ -1,19 +0,0 @@-----name: gitaly-rpc-querying-description: Use when calling or inspecting the gRPC API of a running Gitaly or Praefect locally — listing and describing services with grpcurl, minting the V2 auth token, invoking an RPC or exploring it in grpcui, and optionally building a protoset with make build-protoset.------# Querying Gitaly RPCs locally--Read [doc/invoking_grpc_endpoints_locally.md](../../doc/invoking_grpc_endpoints_locally.md) first: it-carries every command, and the socket address form, the token scheme, and the JSON encoding of `bytes`-fields are each easy to get wrong by guessing.--The workflow it documents:--1. **Find the socket** — `socket_path` in the GDK's `gitaly/gitaly.config.toml`, or in one of the- `gitaly/gitaly-*.praefect.toml` files when going through Praefect.-1. **Mint a V2 token**, but only if `token` is set in the `[auth]` section of that same file.-1. **Discover the method** by listing and describing services. This works through reflection- against a running server, so a protoset is optional.-1. **Call it**, starting with `gitaly.ServerService.ServerInfo` to prove out the socket and token.