Skip to content

Git designed for AI agents.

docs: lead the README with the Scion design

Federico Eliafbbd91c32f2b
diff --git a/README.md b/README.md
index eea5544..301335f 100644
--- a/README.md
+++ b/README.md
@@ -1,204 +1,126 @@
# open_scion
-open_scion is an open-source (MIT), vendor-neutral, agent-scale version control
-server. It is a fork of [Gitaly](https://gitlab.com/gitlab-org/gitaly) that
-implements the ideas Anthropic described for its internal "Scion" system in
-[Redesigning source control](https://www.anthropic.com/engineering/redesigning-source-control).
-
-> **Not affiliated with Anthropic or GitLab.** open_scion is an independent
-> project inspired by a public Anthropic engineering article. It is not an
-> Anthropic product, and Anthropic does not endorse or maintain it. "Scion" is
-> the name Anthropic uses for its own internal system.
-
-Every repository stays a normal Git repository, every upstream Gitaly RPC keeps
-working, and the server never talks to a language model: AI features are
-optional and live outside the server, behind a vendor-neutral protocol.
-
-Inside the code base the extensions are called **Graft**: the `[graft]`
-configuration section, the `graft` CLI, the `internal/graft` packages and the
-`doc/graft` documentation.
-
-## What it does
-
-- **Server-centric, always-online clients.** Clients fetch exactly the objects
- and trees they need (`ObjectService`); history operations (blame, path history,
- search, ancestry) run on the server and are cached as derived data.
-- **Landing without a merge queue.** `UserLand` rebases a change onto the tip and
- lands it with a compare-and-swap; concurrent lands on one branch are batched.
- Conflicts can be resolved automatically through a vendor-neutral resolver
- protocol (any LLM provider, or none).
-- **Changeset log.** Every accepted reference update is recorded with its
- operation and actor, and can be listed or watched (`ChangesetService`).
-- **Storage core.** The repository state can live in a strongly consistent
- database (SQLite or PostgreSQL) as a write-ahead log of changesets with a
- compare-and-swap per reference, with content-addressed packs in object storage
- (file, S3, GCS, Azure through gocloud). Server nodes become caches that any node
- can rebuild, so they scale horizontally. Shadow mode lets you run it alongside
- the classic storage before switching. See [STORAGE.md](doc/graft/STORAGE.md).
-- **Agents as first-class users.** Per-agent tokens, an RPC allowlist, branch
- policies, an agent-friendly review service with semantic approval rules, code
- search, post-merge verification with automatic bisect and revert, and
- "superfan" monitoring agents.
-- **Tiered caching** of immutable objects (`graft-cache`).
+**Source control for the age of coding agents.**
+open_scion is an open-source implementation of **Scion**, the version control
+design Anthropic described in
+[Redesigning source control](https://www.anthropic.com/engineering/redesigning-source-control):
+a server-centric system built for hundreds of agents that read, write and land
+code at the same time.
-## Quick start
-
-Requirements: Linux or macOS, Go 1.26, and the build dependencies of Gitaly
-(`make` builds the bundled Git).
-
-```shell
-git clone https://github.com/feder-cr/open_scion.git
-cd open_scion
-make build # builds gitaly, graft, graft-cache, ... into _build/bin
-examples/graft/quickstart.sh # a standalone server, a cache daemon, two agents and a land, in a temporary directory
-./_build/bin/graft # the CLI and its commands
-```
-
-To run your own server, adapt [`examples/graft/config.toml`](examples/graft/config.toml)
-and start it with `_build/bin/gitaly serve config.toml`.
-
-The [Quickstart](doc/graft/QUICKSTART.md) walks through a standalone server,
-agent tokens, the cache tier and the CLI. For the storage core, start with
-`[graft.storage] mode = "shadow"` and read [STORAGE.md](doc/graft/STORAGE.md).
-
-## Documentation
-
-- [Architecture](doc/graft/ARCHITECTURE.md): components, data flows, and how each Scion idea maps to the code.
-- [Quickstart](doc/graft/QUICKSTART.md) and [Configuration](doc/graft/CONFIGURATION.md) (the `[graft]` reference).
-- [Storage core](doc/graft/STORAGE.md): database + object storage, modes, import, operations, benchmark, limits.
-- [Land batching](doc/graft/LAND-BATCHING.md), [derived data](doc/graft/DERIVED.md), [conflict resolution](doc/graft/RESOLVE.md).
-- [Review](doc/graft/REVIEW.md), [search](doc/graft/SEARCH.md), [verification](doc/graft/VERIFY.md), [superfans](doc/graft/SUPERFAN.md) and [Slack chat](doc/graft/SUPERFAN-CHAT.md).
-- [Agent security](doc/graft/AGENT-SECURITY.md), [fuzzing](doc/graft/FUZZING.md), [benchmarks](doc/graft/BENCH.md).
-- [Limitations and roadmap](doc/graft/LIMITATIONS.md).
-- Design documents of each phase: [PLAN.md](doc/graft/PLAN.md) to [PLAN6.md](doc/graft/PLAN6.md).
-
-## Status
-
-Version 0.1.0, the first public release. The features above are implemented and
-tested (unit, end-to-end, multi-node, crash and fuzz tests), but interfaces may
-still change. Known gaps, such as garbage collection of packs in the storage
-core, are listed in [LIMITATIONS.md](doc/graft/LIMITATIONS.md) and
-[STORAGE.md](doc/graft/STORAGE.md#limits).
+MIT licensed. Vendor neutral: the server never calls a language model, and every
+AI feature works with any provider, or none.
-## License
-
-MIT, see [LICENSE](LICENSE). open_scion is based on Gitaly by GitLab Inc.,
-whose MIT license and copyright notices are kept. The `ee/` directory of
-upstream Gitaly (not MIT) is not part of this repository. Third-party notices
-are in [NOTICE](NOTICE).
+<sub>open_scion is an independent implementation of the publicly described design. It is not affiliated with or endorsed by Anthropic.</sub>
---
-# Upstream: Gitaly
-
-The rest of this file is the README of upstream Gitaly, kept for reference.
-
-## Gitaly
-
-![Gitaly](https://gitlab.com/gitlab-org/gitaly/uploads/509123ed56bd51247996038c858db006/gitaly-wordmark-small.png)
-
-**Quick Links**:
- [**Roadmap**][roadmap] |
- [Want to Contribute?](#contributing) |
-
-Gitaly is a Git [RPC](https://en.wikipedia.org/wiki/Remote_procedure_call)
-service for handling all the Git calls made by GitLab.
-
-For documentation generated from the protobuf definitions (in `proto/` directory),
-see [Gitaly RPC documentation](https://gitlab-org.gitlab.io/gitaly/).
-
-To see where it fits in please look at [GitLab architecture](https://docs.gitlab.com/development/architecture/#system-layout).
-
-## Project Goals
-
-Fault-tolerant horizontal scaling of Git storage in GitLab, and particularly, on [GitLab.com](https://gitlab.com).
-
-## Current Status
-
-All application code accesses Git repositories via Gitaly.
-
-Besides "Git over RPC" functionality, Gitaly also offers an optional [high-availability solution](#high-availability).
+## Why
+
+Classic Git hosting was built for humans who clone a repository, work for hours
+and push once. Coding agents work differently: thousands of short-lived tasks,
+each touching a few files, each wanting to land in seconds, all on the same
+branches. Full clones, merge queues and client-side history walks become the
+bottleneck.
+
+Scion's answer is to move the work to the server: clients stay online and fetch
+only what they need, history questions are answered (and cached) by the server,
+changes land through a compare-and-swap instead of a queue, every update is an
+entry in a changeset log, and the repository state lives in a consistent
+database plus object storage so servers scale out. open_scion implements these
+ideas, with normal Git repositories underneath.
+
+## The Scion ideas, implemented
+
+| Scion idea | In open_scion |
+|---|---|
+| **Always-online clients** that fetch exactly what they need | `ObjectService` serves single objects and sparse trees; `graft clone` and `graft sparse` materialize only the paths you ask for. |
+| **Source control as an API**: history runs on the server | Blame, path history, ancestry and code search run server-side and are memoized as derived data, computed once per commit. |
+| **Landing without a merge queue** | `UserLand` rebases a change onto the current tip and lands it with a compare-and-swap; concurrent lands on one branch are batched into a single reference update. |
+| **Automatic conflict resolution** | A conflicting land is resolved by a resolver you choose (any LLM provider, a deterministic tool, or none) through a versioned, vendor-neutral protocol. |
+| **Changeset log** | Every accepted reference update is recorded with its operation and actor in a dense, ordered log that clients list or watch live. |
+| **Storage core: a database and object storage, stateless servers** | Each change is a changeset in a write-ahead log in SQLite or PostgreSQL, with a compare-and-swap per reference; packs live in object storage (file, S3, GCS, Azure). Server nodes are caches that any node can rebuild, so you add nodes to scale. |
+| **Test after landing, revert the culprit** | `graft-verify` tests the tip, bisects a failure and reverts the change that broke it. |
+| **Monitoring agents ("superfans")** | Agents follow the changeset stream and report what matters to each component, also through Slack. |
+| **Agent-friendly review** | Reviews with semantic approval rules: a comment-only change can need fewer approvals than a logic change. |
+| **Agents as first-class users** | Per-agent tokens, an RPC allowlist and branch policies decide what each agent may do. |
+| **Strict cache tiers** | `graft-cache` caches immutable objects in daemon and regional tiers, verifying every object. |
+
+## Architecture
-We are building features according to our [roadmap][roadmap].
-
-## Installation
-
-Most users won't install Gitaly on its own. It is already included in [your GitLab installation](https://about.gitlab.com/install/).
-
-Gitaly requires Go 1.26. Run `make` to compile the executables
-required by Gitaly.
-
-Gitaly uses `git`. Versions `2.47.0` and newer are supported.
-
-## Configuration
-
-The administration and reference guide is [documented in the GitLab project](https://docs.gitlab.com/administration/gitaly/).
-
-## Contributing
-
-See [CONTRIBUTING.md](CONTRIBUTING.md) and a [list of quick win issues](https://gitlab.com/gitlab-org/gitaly/-/issues/?sort=due_date&state=opened&label_name%5B%5D=quick%20win&first_page_size=20).
-
-## Name
-
-Gitaly is a tribute to Git and the town of [Aly](https://en.wikipedia.org/wiki/Aly). Where the town of
-Aly has zero inhabitants most of the year we would like to reduce the number of
-disk operations to zero for most actions. It doesn't hurt that it sounds like
-Italy, the capital of which is [the destination of all roads](https://en.wikipedia.org/wiki/All_roads_lead_to_Rome). All Git actions in
-GitLab end up in Gitaly.
-
-## Design
-
-The [Gitaly and Gitaly cluster](https://docs.gitlab.com/administration/gitaly/) documentation details the architecture
-and design of Gitaly, including a list of known Gitaly consumers.
-
-## High Availability
-
-Gitaly offers a High Availability solution known as Gitaly Cluster ([product documentation](https://docs.gitlab.com/administration/gitaly/)).
-
-- In its current iteration, client traffic goes through [Praefect](https://docs.gitlab.com/administration/gitaly/praefect/), which then replicates data to multiple Gitaly servers, and stores state in a PostgreSQL database (see [Design](#design) above).
-
-## Further reading
-
-More about the project and its processes is [detailed in the docs](doc/README.md).
-
-## Distributed Tracing
+```
+ agents, CI jobs, sparse workspaces graft CLI / Go SDK
+ |
+ v gRPC
+ +-------------------------------------------------+
+ | graft-cache (optional daemon and cache tiers) |
+ +------------------------+------------------------+
+ v
+ +---------------------------------------------------------------+
+ | open_scion server (any number of nodes) |
+ | land - objects - changesets - derived data - search - review |
+ | write path: changeset + compare-and-swap per reference |
+ +-----------+--------------------------------+------------------+
+ | |
+ v v
+ +-----------------------+ +-----------------------------+
+ | SQLite / PostgreSQL | | object storage |
+ | changeset WAL, refs, | | content-addressed packs |
+ | changelog, reviews | | (file, S3, GCS, Azure) |
+ +-----------------------+ +-----------------------------+
+ \ /
+ local Git repositories = caches any node can rebuild
+```
-Gitaly supports distributed tracing through [LabKit](https://gitlab.com/gitlab-org/labkit/) using [OpenTracing APIs](https://opentracing.io).
+Details: [ARCHITECTURE.md](doc/graft/ARCHITECTURE.md) and [STORAGE.md](doc/graft/STORAGE.md).
-By default, no tracing implementation is linked into the binary, but different OpenTracing providers can be linked in using [build tags](https://golang.org/pkg/go/build/#hdr-Build_Constraints)/[build constraints](https://golang.org/pkg/go/build/#hdr-Build_Constraints). This can be done by setting the `BUILD_TAGS` make variable.
+## Quick start
-For more details of the supported providers, see LabKit, but as an example, for Jaeger tracing support, include the tags: `BUILD_TAGS="tracer_static tracer_static_jaeger"`.
+Requirements: Linux or macOS, Go 1.26, a C toolchain, Meson and Ninja (the
+build compiles a bundled Git).
```shell
-make BUILD_TAGS="tracer_static tracer_static_jaeger"
+git clone https://github.com/feder-cr/open_scion.git
+cd open_scion
+make build # server, CLI and tools into _build/bin
+examples/graft/quickstart.sh # a server, a cache daemon, two agents and a land, in a temporary directory
+./_build/bin/graft # the CLI and its commands
```
-Once Gitaly is compiled with an opentracing provider, the tracing configuration is configured via the `GITLAB_TRACING` environment variable.
+To run your own server, adapt [`examples/graft/config.toml`](examples/graft/config.toml)
+and start it with `_build/bin/gitaly serve config.toml`. To put the repository
+state in a database and object storage, add a `[graft.storage]` section: start in
+`shadow` mode, import your repositories, then switch to `authoritative`
+([STORAGE.md](doc/graft/STORAGE.md)).
-For example, to configure Jaeger, you could use the following command:
+Inside the code the extensions are called **Graft**: the `[graft]`
+configuration section, the `graft` CLI, `internal/graft` and `doc/graft`.
-```shell
-GITLAB_TRACING=opentracing://jaeger ./gitaly config.toml
-```
-
-## Continuous Profiling
+## Documentation
-Gitaly supports Continuous Profiling through [LabKit][] using [Stackdriver Profiler](https://cloud.google.com/profiler).
+- [Quickstart](doc/graft/QUICKSTART.md) and [configuration reference](doc/graft/CONFIGURATION.md)
+- [Architecture](doc/graft/ARCHITECTURE.md) and the [storage core](doc/graft/STORAGE.md)
+- [Landing and batching](doc/graft/LAND-BATCHING.md), [conflict resolution](doc/graft/RESOLVE.md), [derived data](doc/graft/DERIVED.md)
+- [Review](doc/graft/REVIEW.md), [search](doc/graft/SEARCH.md), [verification](doc/graft/VERIFY.md), [superfans](doc/graft/SUPERFAN.md) and [Slack chat](doc/graft/SUPERFAN-CHAT.md)
+- [Agent security](doc/graft/AGENT-SECURITY.md), [benchmarks](doc/graft/BENCH.md), [fuzzing](doc/graft/FUZZING.md)
+- [Limitations and roadmap](doc/graft/LIMITATIONS.md)
+- Design documents of each phase: [PLAN.md](doc/graft/PLAN.md) to [PLAN6.md](doc/graft/PLAN6.md)
-For more information on how to set it up, see the [LabKit monitoring docs](https://gitlab.com/gitlab-org/labkit/-/blob/master/monitoring/doc.go).
+## Status
-## Training Videos
+Version 0.1.0, the first public release. Everything above is implemented and
+covered by unit, end-to-end, multi-node, crash and fuzz tests. Interfaces may
+still change before 1.0. Known gaps (for example garbage collection of packs in
+the storage core) are listed in [LIMITATIONS.md](doc/graft/LIMITATIONS.md).
-The [Gitaly Training and Resources Playlist](https://www.youtube.com/playlist?list=PL05JrBw4t0KqPMCB7xW1ExIRp6YuD_7_N)
-contains videos that give an overview into how Gitaly works, as well as dive
-deep into different parts of Gitaly and even Git.
+Contributions, issues and ideas are welcome.
-## Presentations
+## Credits and license
-- [TheConf talk on Scaling GitLab Git storage with Gitaly, 2019-08-16](https://speakerdeck.com/olsfer/how-gitlab-scaled-git-access-with-a-go-service)
-- [Infrastructure Team Update 2017-05-11](https://about.gitlab.com/2017/05/11/functional-group-updates/#infrastructure-team)
-- [Gitaly Basics, 2017-05-01](https://docs.google.com/presentation/d/1cLslUbXVkniOaeJ-r3s5AYF0kQep8VeNfvs0XSGrpA0/edit#slide=id.g1c73db867d_0_0)
-- [Git Paris meetup, 2017-02-22](https://docs.google.com/presentation/d/19OZUalFMIDM8WujXrrIyCuVb_oVeaUzpb-UdGThOvAo/edit?usp=sharing) a high-level overview of what our plans are and where we are.
+open_scion is MIT licensed, see [LICENSE](LICENSE).
-[roadmap]: https://handbook.gitlab.com/handbook/engineering/infrastructure-platforms/tenant-scale/gitaly/#roadmap
-[LabKit]: https://gitlab.com/gitlab-org/labkit/
+It started as a fork of [Gitaly](https://gitlab.com/gitlab-org/gitaly), GitLab's
+Git RPC server (MIT, Copyright GitLab Inc.), which provides the Git plumbing
+underneath; every upstream Gitaly RPC keeps working. The non-MIT `ee/` directory
+of upstream Gitaly is not part of this repository. Gitaly's own README is kept in
+[doc/UPSTREAM-GITALY.md](doc/UPSTREAM-GITALY.md), and third-party notices are in
+[NOTICE](NOTICE).
diff --git a/doc/UPSTREAM-GITALY.md b/doc/UPSTREAM-GITALY.md
new file mode 100644
index 0000000..ace7e22
--- /dev/null
+++ b/doc/UPSTREAM-GITALY.md
@@ -0,0 +1,114 @@
+# Upstream Gitaly README
+
+open_scion started as a fork of Gitaly. This is the README of upstream Gitaly,
+kept for reference; see the [open_scion README](../README.md) for this project.
+
+## Gitaly
+
+![Gitaly](https://gitlab.com/gitlab-org/gitaly/uploads/509123ed56bd51247996038c858db006/gitaly-wordmark-small.png)
+
+**Quick Links**:
+ [**Roadmap**][roadmap] |
+ [Want to Contribute?](#contributing) |
+
+Gitaly is a Git [RPC](https://en.wikipedia.org/wiki/Remote_procedure_call)
+service for handling all the Git calls made by GitLab.
+
+For documentation generated from the protobuf definitions (in `proto/` directory),
+see [Gitaly RPC documentation](https://gitlab-org.gitlab.io/gitaly/).
+
+To see where it fits in please look at [GitLab architecture](https://docs.gitlab.com/development/architecture/#system-layout).
+
+## Project Goals
+
+Fault-tolerant horizontal scaling of Git storage in GitLab, and particularly, on [GitLab.com](https://gitlab.com).
+
+## Current Status
+
+All application code accesses Git repositories via Gitaly.
+
+Besides "Git over RPC" functionality, Gitaly also offers an optional [high-availability solution](#high-availability).
+
+We are building features according to our [roadmap][roadmap].
+
+## Installation
+
+Most users won't install Gitaly on its own. It is already included in [your GitLab installation](https://about.gitlab.com/install/).
+
+Gitaly requires Go 1.26. Run `make` to compile the executables
+required by Gitaly.
+
+Gitaly uses `git`. Versions `2.47.0` and newer are supported.
+
+## Configuration
+
+The administration and reference guide is [documented in the GitLab project](https://docs.gitlab.com/administration/gitaly/).
+
+## Contributing
+
+See [CONTRIBUTING.md](../CONTRIBUTING.md) and a [list of quick win issues](https://gitlab.com/gitlab-org/gitaly/-/issues/?sort=due_date&state=opened&label_name%5B%5D=quick%20win&first_page_size=20).
+
+## Name
+
+Gitaly is a tribute to Git and the town of [Aly](https://en.wikipedia.org/wiki/Aly). Where the town of
+Aly has zero inhabitants most of the year we would like to reduce the number of
+disk operations to zero for most actions. It doesn't hurt that it sounds like
+Italy, the capital of which is [the destination of all roads](https://en.wikipedia.org/wiki/All_roads_lead_to_Rome). All Git actions in
+GitLab end up in Gitaly.
+
+## Design
+
+The [Gitaly and Gitaly cluster](https://docs.gitlab.com/administration/gitaly/) documentation details the architecture
+and design of Gitaly, including a list of known Gitaly consumers.
+
+## High Availability
+
+Gitaly offers a High Availability solution known as Gitaly Cluster ([product documentation](https://docs.gitlab.com/administration/gitaly/)).
+
+- In its current iteration, client traffic goes through [Praefect](https://docs.gitlab.com/administration/gitaly/praefect/), which then replicates data to multiple Gitaly servers, and stores state in a PostgreSQL database (see [Design](#design) above).
+
+## Further reading
+
+More about the project and its processes is [detailed in the docs](README.md).
+
+## Distributed Tracing
+
+Gitaly supports distributed tracing through [LabKit](https://gitlab.com/gitlab-org/labkit/) using [OpenTracing APIs](https://opentracing.io).
+
+By default, no tracing implementation is linked into the binary, but different OpenTracing providers can be linked in using [build tags](https://golang.org/pkg/go/build/#hdr-Build_Constraints)/[build constraints](https://golang.org/pkg/go/build/#hdr-Build_Constraints). This can be done by setting the `BUILD_TAGS` make variable.
+
+For more details of the supported providers, see LabKit, but as an example, for Jaeger tracing support, include the tags: `BUILD_TAGS="tracer_static tracer_static_jaeger"`.
+
+```shell
+make BUILD_TAGS="tracer_static tracer_static_jaeger"
+```
+
+Once Gitaly is compiled with an opentracing provider, the tracing configuration is configured via the `GITLAB_TRACING` environment variable.
+
+For example, to configure Jaeger, you could use the following command:
+
+```shell
+GITLAB_TRACING=opentracing://jaeger ./gitaly config.toml
+```
+
+## Continuous Profiling
+
+Gitaly supports Continuous Profiling through [LabKit][] using [Stackdriver Profiler](https://cloud.google.com/profiler).
+
+For more information on how to set it up, see the [LabKit monitoring docs](https://gitlab.com/gitlab-org/labkit/-/blob/master/monitoring/doc.go).
+
+## Training Videos
+
+The [Gitaly Training and Resources Playlist](https://www.youtube.com/playlist?list=PL05JrBw4t0KqPMCB7xW1ExIRp6YuD_7_N)
+contains videos that give an overview into how Gitaly works, as well as dive
+deep into different parts of Gitaly and even Git.
+
+## Presentations
+
+- [TheConf talk on Scaling GitLab Git storage with Gitaly, 2019-08-16](https://speakerdeck.com/olsfer/how-gitlab-scaled-git-access-with-a-go-service)
+- [Infrastructure Team Update 2017-05-11](https://about.gitlab.com/2017/05/11/functional-group-updates/#infrastructure-team)
+- [Gitaly Basics, 2017-05-01](https://docs.google.com/presentation/d/1cLslUbXVkniOaeJ-r3s5AYF0kQep8VeNfvs0XSGrpA0/edit#slide=id.g1c73db867d_0_0)
+- [Git Paris meetup, 2017-02-22](https://docs.google.com/presentation/d/19OZUalFMIDM8WujXrrIyCuVb_oVeaUzpb-UdGThOvAo/edit?usp=sharing) a high-level overview of what our plans are and where we are.
+
+[roadmap]: https://handbook.gitlab.com/handbook/engineering/infrastructure-platforms/tenant-scale/gitaly/#roadmap
+[LabKit]: https://gitlab.com/gitlab-org/labkit/