Skip to content

Every agent's model. One place. Codex on DeepSeek, Claude Code on Kimi, from the menu bar.

davsync: a GitHub backup too large for one request goes up in parts (Zhang-Wei, Discord)

yetonea6efe07cd797
diff --git a/docs/subsystems/sync-backup.md b/docs/subsystems/sync-backup.md
index ab2531b06..df8708d82 100644
--- a/docs/subsystems/sync-backup.md
+++ b/docs/subsystems/sync-backup.md
@@ -114,6 +114,28 @@ repository can keep the token. The branch is part of the sync state key.
commit is built. A ref update that is not a fast-forward is built again on
the new head, up to three times; if the new head changed the file it is
`errChanged`. A branch deleted meanwhile is an error, never created.
+- A blob goes up as base64 in one JSON body, a third larger than the file,
+ and GitHub refuses a body well under the 100 MB a blob can hold: a
+ 50.8 MB backup (about 68 MB of body) got 422 "Sorry, your input was too
+ large to process" (Zhang-Wei, Discord), while a 19 MB file's Contents
+ PUT (about 25 MB of body) went through; GitHub documents no request
+ size. So a backup over 16 MiB (`githubPart`) goes up in parts, in the
+ same one commit: `magpie.magpie-backup.parts/1`, `/2`, … beside it, and
+ at the backup's own path a short list of them (`githubParts`: format
+ `magpie-backup`, version 2, `split: github-parts`, the total size and
+ each part's blob SHA and size). The list's blob SHA is the version, and
+ it names each part by blob SHA, so one version pins one whole set; a
+ read downloads each part by that SHA, checks it, and returns the backup
+ whole. A list that doesn't add up or a part that isn't the one named is
+ an error, never a backup put together wrong. A write removes the parts
+ the list at the head named and the new backup doesn't use, in the same
+ commit, so going back to one file leaves no part behind. Backups up to
+ 16 MiB stay one file, as before.
+- A magpie from before parts opens the list as a backup with version 2
+ and no key derivation, and stops with "made by a newer magpie; update
+ magpie" — not `ErrCorrupt` or `ErrNotBackup`, so it neither rebuilds
+ over it nor offers to replace it. A backup in one file, from any
+ magpie, still reads.
- Tokens are separate from the library's GitHub token, sent only to
`https://api.github.com`, and must differ from the encryption passphrase.
Repository history retains older encrypted versions.
@@ -123,8 +145,9 @@ repository can keep the token. The branch is part of the sync state key.
initialization, after repository access and the default branch are checked.
The initial write omits `branch` so GitHub creates its default branch; other
conflicts and missing branches are not treated as an empty repository.
-- Large files are downloaded by blob SHA, preserving the version that was
- read. Sync files are limited to 64 MiB. Usage listing fails at the Contents
+- Large files and parts are downloaded by blob SHA, preserving the version
+ that was read. Sync files are limited to 64 MiB in all (four parts), the
+ bound the library's skill budget is sized for. Usage listing fails at the Contents
API's 1,000-file limit rather than treating a truncated list as complete.
- A stale SHA, a backup another commit changed, or a racing initial
creation returns `errChanged`. Other 422
@@ -148,6 +171,17 @@ the initial write. `TestGitHubLargeBackup` writes a 21 MB backup past a fixture
API refuses it, as one commit, and reads it back by blob SHA.
`TestGitHubBranchMovedMeanwhile` covers a head that moved before the ref
update, and `TestGitHubEmptyRepositoryLargeBackup` a large first backup.
+`TestGitHubSplitBackup` writes a 50.8 MB backup past a fixture that refuses
+a blob over 24 MiB, as four parts in one commit, reads it back byte for
+byte, then goes to two parts and back to one file with no part left.
+`TestGitHubOneFileBackupStillReads` reads a 30 MB backup in one file and
+replaces it with parts; `TestGitHubPartsOlderMagpie` checks the list stops
+an older magpie as a newer magpie's backup; `TestGitHubPartsDamaged`,
+`TestGitHubPartsBranchMovedMeanwhile` and
+`TestGitHubEmptyRepositoryPartsBackup` cover a bad list or part, a moved
+head (each blob sent once) and an empty repository;
+`TestGitHubSyncLargeLibrary` syncs a library that seals past one request
+to a second computer.
`TestGitHubNotBackup` covers backend-specific recovery
advice when the repository holds an empty or foreign file.
diff --git a/internal/davsync/github.go b/internal/davsync/github.go
index a750a89f1..08e684291 100644
--- a/internal/davsync/github.go
+++ b/internal/davsync/github.go
@@ -20,6 +20,67 @@ import (
const githubMaxFile = 64 << 20
+// githubPart is the most of a backup one request sends. A blob goes up as
+// base64 in a JSON body, a third bigger than the file, and GitHub refuses a
+// body far under the 100 MB a blob can hold: a 50.8 MB backup (about 68 MB
+// of body) got 422 "Sorry, your input was too large to process" (Zhang-Wei,
+// Discord), while a 19 MB file's Contents PUT, about 25 MB of body, went
+// through. GitHub documents no request size, so a part stays under the
+// largest body seen accepted.
+const githubPart = 16 << 20
+
+// githubSplit names a backup sent in parts. Its file at the backup's path
+// is a short list of the parts, each a blob in a folder beside it, all in
+// the one commit; the list's own blob SHA is the version, and it names
+// every part by its blob SHA, so a version pins one whole set. The list
+// says it is format magpie-backup, version 2, with no key derivation, so a
+// magpie before parts opens it as a newer magpie's backup and stops
+// ("update magpie"), never as a damaged one (ErrCorrupt, ErrNotBackup) to
+// rebuild or offer for replacing.
+const githubSplit = "github-parts"
+
+type githubParts struct {
+ Format string `json:"format"`
+ Version int `json:"version"`
+ Split string `json:"split"`
+ Size int64 `json:"size"`
+ Parts []githubPartRef `json:"parts"`
+}
+
+type githubPartRef struct {
+ SHA string `json:"sha"`
+ Size int64 `json:"size"`
+}
+
+// partName is the backup's i-th part (from 0), beside it in magpie's folder.
+func partName(i int) string { return file + ".parts/" + strconv.Itoa(i+1) }
+
+// parseParts reads b as a list of parts: nil for a file that isn't one (a
+// backup in one file), an error for a list that doesn't add up.
+func parseParts(b []byte) (*githubParts, error) {
+ if len(b) > 64<<10 {
+ return nil, nil
+ }
+ var m githubParts
+ if json.Unmarshal(b, &m) != nil || m.Split == "" {
+ return nil, nil
+ }
+ if m.Format != "magpie-backup" || m.Version != 2 || m.Split != githubSplit {
+ return nil, errors.New("the GitHub backup was sent in parts by a newer magpie; update magpie to read it")
+ }
+ total := int64(0)
+ for _, p := range m.Parts {
+ if len(p.SHA) != 40 || p.Size <= 0 || p.Size > githubPart {
+ return nil, errors.New("the GitHub backup's list of parts is damaged")
+ }
+ total += p.Size
+ }
+ if len(m.Parts) < 2 || total != m.Size || total > githubMaxFile {
+ return nil, errors.New("the GitHub backup's list of parts is damaged")
+ }
+ return &m, nil
+}
+
type github struct {
repo, prefix, branch, token string
ready bool
@@ -274,7 +335,28 @@ func (g *github) get(ctx context.Context, have version) ([]byte, version, error)
return nil, v, errNotModified
}
b, err := g.bytes(ctx, c)
- return b, v, err
+ if err != nil {
+ return nil, v, err
+ }
+ m, err := parseParts(b)
+ if err != nil || m == nil {
+ return b, v, err
+ }
+ // Each part by its blob SHA, the one the list names, so the parts are
+ // the set this version holds even if a new backup is committed while
+ // they download.
+ whole := make([]byte, 0, m.Size)
+ for i, p := range m.Parts {
+ b, err := g.bytes(ctx, &githubContent{SHA: p.SHA, Size: p.Size, Encoding: "none"})
+ if err != nil {
+ return nil, v, fmt.Errorf("reading part %d of the GitHub backup: %w", i+1, err)
+ }
+ if gitBlobSHA(b) != p.SHA {
+ return nil, v, fmt.Errorf("GitHub returned a different part %d of the backup than its list names", i+1)
+ }
+ whole = append(whole, b...)
+ }
+ return whole, v, nil
}
// errTooLarge is the Contents API refusing a file it won't process, which
@@ -286,12 +368,19 @@ var errTooLarge = errors.New("GitHub's Contents API refused the file as too larg
// reached the branch after its head was read.
var errBehind = errors.New("the GitHub branch moved meanwhile")
+// githubWrite is one blob a commit puts at path.
+type githubWrite struct {
+ path, sha string
+ data []byte
+}
+
// commit writes or removes one file in one commit. A repository with
// commits goes through the Git Data API (blob, tree, commit, ref), which
// takes files up to 100 MB, where the Contents API refuses large ones (HTTP
-// 422 "the file is too large to be processed"). An empty repository has no
-// branch to build on, so its first write goes through the Contents API,
-// which creates the default branch.
+// 422 "the file is too large to be processed"). A backup over githubPart
+// goes up in parts, with their list at its path, in that one commit. An
+// empty repository has no branch to build on, so its first write goes
+// through the Contents API, which creates the default branch.
func (g *github) commit(ctx context.Context, name string, data []byte, sha string, remove bool) (version, error) {
if len(data) > githubMaxFile {
return version{}, errors.New("the GitHub sync file exceeds 64 MiB")
@@ -299,10 +388,30 @@ func (g *github) commit(ctx context.Context, name string, data []byte, sha strin
if err := g.prepare(ctx); err != nil {
return version{}, err
}
+ var writes []githubWrite
+ if !remove && name == file && len(data) > githubPart {
+ m := githubParts{Format: "magpie-backup", Version: 2, Split: githubSplit, Size: int64(len(data))}
+ for i, off := 0, 0; off < len(data); i, off = i+1, off+githubPart {
+ p := data[off:min(off+githubPart, len(data))]
+ writes = append(writes, githubWrite{g.at(partName(i)), gitBlobSHA(p), p})
+ m.Parts = append(m.Parts, githubPartRef{gitBlobSHA(p), int64(len(p))})
+ }
+ list, err := json.Marshal(m)
+ if err != nil {
+ return version{}, err
+ }
+ writes = append(writes, githubWrite{g.at(name), gitBlobSHA(list), list})
+ } else if !remove {
+ writes = []githubWrite{{g.at(name), gitBlobSHA(data), data}}
+ }
if g.empty {
- v, err := g.contents(ctx, name, data, sha, remove)
- if !errors.Is(err, errTooLarge) {
- return v, err
+ // One file is tried through the Contents API first; parts can't go
+ // that way, so they start on a first commit of the note's.
+ if len(writes) <= 1 {
+ v, err := g.contents(ctx, name, data, sha, remove)
+ if !errors.Is(err, errTooLarge) {
+ return v, err
+ }
}
// Too large for the Contents API: give the repository a first
// commit of its own, then build the file's commit on it.
@@ -310,9 +419,9 @@ func (g *github) commit(ctx context.Context, name string, data []byte, sha strin
return version{}, err
}
}
- blob := ""
+ sent := map[string]bool{}
for try := 0; ; try++ {
- v, err := g.gitCommit(ctx, name, data, sha, remove, &blob)
+ v, err := g.gitCommit(ctx, name, writes, sha, sent)
if !errors.Is(err, errBehind) {
return v, err
}
@@ -356,12 +465,14 @@ func (g *github) githubJSON(ctx context.Context, method, apiPath string, in, out
return nil, nil
}
-// gitCommit builds one commit on the branch's head that writes data (or
-// removes the file) and moves the branch to it. The file must still be at
-// sha in that head, so another computer's backup is never written over;
-// the ref update without force refuses a head that moved after it was read.
-// blob carries the uploaded blob's SHA across retries.
-func (g *github) gitCommit(ctx context.Context, name string, data []byte, sha string, remove bool, blob *string) (version, error) {
+// gitCommit builds one commit on the branch's head that puts writes (the
+// file, or its parts and their list; none removes it) and moves the branch
+// to it. The file must still be at sha in that head, so another computer's
+// backup is never written over; the ref update without force refuses a
+// head that moved after it was read. Parts the file at the head lists and
+// writes no longer has are removed with it. sent holds the blobs already
+// uploaded, across retries.
+func (g *github) gitCommit(ctx context.Context, name string, writes []githubWrite, sha string, sent map[string]bool) (version, error) {
var ref struct {
Object struct {
SHA string `json:"sha"`
@@ -383,20 +494,34 @@ func (g *github) gitCommit(ctx context.Context, name string, data []byte, sha st
if c != nil && c.SHA != sha || c == nil && sha != "" {
return version{}, errChanged
}
- if !remove && *blob == "" {
+ // The parts the file at the head lists. A list that doesn't read
+ // leaves its parts where they are: removing only what is known to be
+ // there never fails a tree on a path that isn't.
+ had := 0
+ if c != nil && name == file && c.Encoding == "base64" {
+ if b, err := g.bytes(ctx, c); err == nil {
+ if m, err := parseParts(b); err == nil && m != nil {
+ had = len(m.Parts)
+ }
+ }
+ }
+ for _, w := range writes {
+ if sent[w.sha] {
+ continue
+ }
var out struct {
SHA string `json:"sha"`
}
- in := map[string]string{"content": base64.StdEncoding.EncodeToString(data), "encoding": "base64"}
+ in := map[string]string{"content": base64.StdEncoding.EncodeToString(w.data), "encoding": "base64"}
if res, err := g.githubJSON(ctx, http.MethodPost, "git/blobs", in, &out); err != nil {
return version{}, err
} else if res != nil {
return version{}, githubError(res)
}
- if out.SHA != gitBlobSHA(data) {
+ if out.SHA != w.sha {
return version{}, errors.New("GitHub stored a different blob than the one sent")
}
- *blob = out.SHA
+ sent[w.sha] = true
}
var commit struct {
Tree struct {
@@ -418,15 +543,23 @@ func (g *github) gitCommit(ctx context.Context, name string, data []byte, sha st
Type string `json:"type"`
SHA *string `json:"sha"`
}
- e := entry{Path: g.at(name), Mode: "100644", Type: "blob"}
- message := "magpie: remove " + name
- if !remove {
- e.SHA, message = blob, "magpie: sync "+name
+ var entries []entry
+ for _, w := range writes {
+ entries = append(entries, entry{Path: w.path, Mode: "100644", Type: "blob", SHA: &w.sha})
+ }
+ message := "magpie: sync " + name
+ if len(writes) == 0 {
+ entries = append(entries, entry{Path: g.at(name), Mode: "100644", Type: "blob"})
+ message = "magpie: remove " + name
+ }
+ parts := max(len(writes)-1, 0)
+ for i := parts; i < had; i++ {
+ entries = append(entries, entry{Path: g.at(partName(i)), Mode: "100644", Type: "blob"})
}
var tree, made struct {
SHA string `json:"sha"`
}
- if res, err := g.githubJSON(ctx, http.MethodPost, "git/trees", map[string]any{"base_tree": commit.Tree.SHA, "tree": []entry{e}}, &tree); err != nil {
+ if res, err := g.githubJSON(ctx, http.MethodPost, "git/trees", map[string]any{"base_tree": commit.Tree.SHA, "tree": entries}, &tree); err != nil {
return version{}, err
} else if res != nil {
return version{}, githubError(res)
@@ -455,10 +588,10 @@ func (g *github) gitCommit(ctx context.Context, name string, data []byte, sha st
return version{}, githubError(res)
}
g.empty = false
- if remove {
+ if len(writes) == 0 {
return version{}, nil
}
- return version{ETag: *blob}, nil
+ return version{ETag: writes[len(writes)-1].sha}, nil
}
// gitBlobSHA is the SHA git names a blob holding b by: the version GitHub
diff --git a/internal/davsync/github_parts_test.go b/internal/davsync/github_parts_test.go
new file mode 100644
index 000000000..7f8e66b64
--- /dev/null
+++ b/internal/davsync/github_parts_test.go
@@ -0,0 +1,339 @@
+package davsync
+
+import (
+ "context"
+ "encoding/json"
+ "errors"
+ "maps"
+ "slices"
+ "strconv"
+ "strings"
+ "testing"
+
+ "github.com/yetone/magpie/internal/backup"
+ "github.com/yetone/magpie/internal/library"
+)
+
+// filled is n bytes that differ from seed to seed and part to part.
+func filled(n int, seed uint32) []byte {
+ b := make([]byte, n)
+ x := seed*2654435761 + 1
+ for i := range b {
+ x = x*1664525 + 1013904223
+ b[i] = byte(x >> 24)
+ }
+ return b
+}
+
+const backupAt = "main|team/magpie/" + file
+
+func partAt(i int) string { return "main|team/magpie/" + file + ".parts/" + strconv.Itoa(i) }
+
+// parts is how many part files the branch holds beside the backup.
+func (f *githubFixture) parts() int {
+ n := 0
+ for k := range f.objects {
+ if strings.HasPrefix(k, backupAt+".parts/") {
+ n++
+ }
+ }
+ return n
+}
+
+// A backup too large for one blob request goes up in parts, in one commit,
+// and reads back whole (Zhang-Wei, Discord: a 50.8 MB backup, about 68 MB
+// of base64 in one POST /git/blobs, got 422 "Sorry, your input was too
+// large to process").
+func TestGitHubSplitBackup(t *testing.T) {
+ f := newGithubFixture(t)
+ g, err := newGitHub(githubConfig())
+ if err != nil {
+ t.Fatal(err)
+ }
+ ctx := context.Background()
+ small := []byte("an earlier sealed backup")
+ v, err := g.put(ctx, small, "")
+ if err != nil {
+ t.Fatal(err)
+ }
+ big := filled(50_800_000, 1)
+ before := f.writes
+ v, err = g.put(ctx, big, v.ETag)
+ if err != nil {
+ t.Fatalf("a 50.8 MB backup: %v", err)
+ }
+ list := f.objects[backupAt]
+ if f.writes != before+1 || v.ETag != githubSHA(list) || f.parts() != 4 {
+ t.Fatalf("one commit of four parts and their list, versioned by the list: commits %d, %+v, parts %d", f.writes-before, v, f.parts())
+ }
+ if f.biggestBlob > githubPart {
+ t.Fatalf("a blob of %d bytes was sent in one request", f.biggestBlob)
+ }
+ b, got, err := g.get(ctx, version{})
+ if err != nil || got != v || !bytesEqual(b, big) {
+ t.Fatalf("reading the 50.8 MB backup back: size %d, %+v, %v", len(b), got, err)
+ }
+ if _, _, err := g.get(ctx, v); !errors.Is(err, errNotModified) {
+ t.Fatalf("unchanged: %v", err)
+ }
+ if _, err := g.put(ctx, small, "old-sha"); !errors.Is(err, errChanged) || !bytesEqual(f.objects[backupAt], list) {
+ t.Fatalf("a stale version must not replace the backup: %v", err)
+ }
+
+ // A smaller backup in fewer parts takes the parts it no longer has
+ // away in the same commit.
+ mid := filled(20<<20, 2)
+ before = f.writes
+ if v, err = g.put(ctx, mid, v.ETag); err != nil {
+ t.Fatal(err)
+ }
+ if f.writes != before+1 || f.parts() != 2 {
+ t.Fatalf("two parts left after one commit: commits %d, parts %d", f.writes-before, f.parts())
+ }
+ if b, _, err := g.get(ctx, version{}); err != nil || !bytesEqual(b, mid) {
+ t.Fatalf("reading the 20 MB backup back: size %d, %v", len(b), err)
+ }
+
+ // Back to one file: no part is left behind.
+ if v, err = g.put(ctx, small, v.ETag); err != nil {
+ t.Fatal(err)
+ }
+ if f.parts() != 0 || string(f.objects[backupAt]) != string(small) {
+ t.Fatalf("parts left beside a backup in one file: %d", f.parts())
+ }
+ if b, got, err := g.get(ctx, version{}); err != nil || got != v || string(b) != string(small) {
+ t.Fatalf("reading the small backup back: %q %v", b, err)
+ }
+}
+
+// A backup an older magpie wrote as one file, large or small, still reads,
+// and a backup in parts can replace it.
+func TestGitHubOneFileBackupStillReads(t *testing.T) {
+ f := newGithubFixture(t)
+ g, err := newGitHub(githubConfig())
+ if err != nil {
+ t.Fatal(err)
+ }
+ ctx := context.Background()
+ for _, old := range [][]byte{[]byte(`{"format":"magpie-backup","version":1}`), filled(30<<20, 3)} {
+ f.objects[backupAt] = old
+ f.advance("main")
+ b, v, err := g.get(ctx, version{})
+ if err != nil || v.ETag != githubSHA(old) || !bytesEqual(b, old) {
+ t.Fatalf("a backup in one file of %d bytes: size %d, %v", len(old), len(b), err)
+ }
+ }
+ big := filled(40<<20, 4)
+ v, err := g.put(ctx, big, githubSHA(f.objects[backupAt]))
+ if err != nil || f.parts() != 3 {
+ t.Fatalf("parts over a backup in one file: %v, parts %d", err, f.parts())
+ }
+ if b, got, err := g.get(ctx, version{}); err != nil || got != v || !bytesEqual(b, big) {
+ t.Fatalf("reading it back: size %d, %v", len(b), err)
+ }
+}
+
+// A magpie from before parts reads the list as a newer magpie's backup and
+// stops: never a damaged file to rebuild, nor one offered for replacing.
+func TestGitHubPartsOlderMagpie(t *testing.T) {
+ f := newGithubFixture(t)
+ g, err := newGitHub(githubConfig())
+ if err != nil {
+ t.Fatal(err)
+ }
+ if _, err := g.put(context.Background(), filled(githubPart+1, 5), ""); err != nil {
+ t.Fatal(err)
+ }
+ _, err = backup.Open(f.objects[backupAt], githubConfig().Passphrase)
+ if err == nil || errors.Is(err, backup.ErrCorrupt) || errors.Is(err, backup.ErrNotBackup) || !strings.Contains(err.Error(), "newer magpie") {
+ t.Fatalf("an older magpie must read the list as a newer magpie's: %v", err)
+ }
+ if _, ok := described(githubConfig(), f.objects[backupAt], err).(*notBackup); ok {
+ t.Fatal("the list must not be offered for replacing")
+ }
+}
+
+// A list that doesn't add up, or a part that isn't the one it names, is an
+// error, never a backup put together wrong.
+func TestGitHubPartsDamaged(t *testing.T) {
+ setup := func(t *testing.T) (*githubFixture, *github, githubParts) {
+ f := newGithubFixture(t)
+ g, err := newGitHub(githubConfig())
+ if err != nil {
+ t.Fatal(err)
+ }
+ if _, err := g.put(context.Background(), filled(githubPart*2+10, 6), ""); err != nil {
+ t.Fatal(err)
+ }
+ var m githubParts
+ if err := json.Unmarshal(f.objects[backupAt], &m); err != nil || len(m.Parts) != 3 {
+ t.Fatalf("the list: %s %v", f.objects[backupAt], err)
+ }
+ return f, g, m
+ }
+ write := func(f *githubFixture, m githubParts) {
+ b, _ := json.Marshal(m)
+ f.objects[backupAt] = b
+ f.advance("main")
+ }
+ t.Run("a part changed", func(t *testing.T) {
+ f, g, m := setup(t)
+ f.blobs[m.Parts[1].SHA] = filled(int(m.Parts[1].Size), 7)
+ if b, _, err := g.get(context.Background(), version{}); err == nil || b != nil || !strings.Contains(err.Error(), "part 2") {
+ t.Fatalf("a changed part: size %d, %v", len(b), err)
+ }
+ })
+ t.Run("a part missing", func(t *testing.T) {
+ f, g, m := setup(t)
+ delete(f.blobs, m.Parts[2].SHA)
+ delete(f.objects, partAt(3))
+ if b, _, err := g.get(context.Background(), version{}); err == nil || b != nil {
+ t.Fatalf("a missing part: size %d, %v", len(b), err)
+ }
+ })
+ t.Run("sizes", func(t *testing.T) {
+ f, g, m := setup(t)
+ m.Size++
+ write(f, m)
+ if b, _, err := g.get(context.Background(), version{}); err == nil || !strings.Contains(err.Error(), "damaged") {
+ t.Fatalf("sizes that don't add up: size %d, %v", len(b), err)
+ }
+ })
+ t.Run("newer", func(t *testing.T) {
+ f, g, m := setup(t)
+ m.Split = "github-parts-2"
+ write(f, m)
+ if _, _, err := g.get(context.Background(), version{}); err == nil || !strings.Contains(err.Error(), "newer magpie") {
+ t.Fatalf("a newer magpie's parts: %v", err)
+ }
+ })
+}
+
+// The head moving meanwhile: a commit of another file is built on again
+// with the parts already sent; another computer's backup is the merge.
+func TestGitHubPartsBranchMovedMeanwhile(t *testing.T) {
+ setup := func(t *testing.T) (*githubFixture, *github, version) {
+ f := newGithubFixture(t)
+ g, err := newGitHub(githubConfig())
+ if err != nil {
+ t.Fatal(err)
+ }
+ v, err := g.put(context.Background(), []byte("first"), "")
+ if err != nil {
+ t.Fatal(err)
+ }
+ return f, g, v
+ }
+ big := filled(githubPart*2+1, 8)
+ t.Run("another file", func(t *testing.T) {
+ f, g, v := setup(t)
+ f.beforeRef = func() {
+ f.beforeRef = nil
+ f.objects["main|team/magpie/usage/other.magpie-usage"] = []byte("other computer")
+ f.advance("main")
+ }
+ f.requests = nil
+ if _, err := g.put(context.Background(), big, v.ETag); err != nil {
+ t.Fatal(err)
+ }
+ posts := 0
+ for _, r := range f.requests {
+ if r == "POST /repos/alice/private/git/blobs" {
+ posts++
+ }
+ }
+ if posts != 4 || f.parts() != 3 || string(f.objects["main|team/magpie/usage/other.magpie-usage"]) != "other computer" {
+ t.Fatalf("both commits on the branch, each blob sent once: posts %d, parts %d", posts, f.parts())
+ }
+ if b, _, err := g.get(context.Background(), version{}); err != nil || !bytesEqual(b, big) {
+ t.Fatalf("reading it back: %v", err)
+ }
+ })
+ t.Run("the backup", func(t *testing.T) {
+ f, g, v := setup(t)
+ f.beforeRef = func() {
+ f.beforeRef = nil
+ f.objects[backupAt] = []byte("other computer's backup")
+ f.advance("main")
+ }
+ if _, err := g.put(context.Background(), big, v.ETag); !errors.Is(err, errChanged) {
+ t.Fatalf("another computer's backup must go through the merge: %v", err)
+ }
+ if string(f.objects[backupAt]) != "other computer's backup" || f.parts() != 0 {
+ t.Fatalf("another computer's backup was written over: parts %d", f.parts())
+ }
+ })
+}
+
+// An empty repository's first backup in parts follows the note that
+// creates the default branch; nothing large goes through the Contents API.
+func TestGitHubEmptyRepositoryPartsBackup(t *testing.T) {
+ f := newGithubFixture(t)
+ f.empty = true
+ g, err := newGitHub(githubConfig())
+ if err != nil {
+ t.Fatal(err)
+ }
+ ctx := context.Background()
+ big := filled(githubPart+100, 9)
+ v, err := g.put(ctx, big, "")
+ if err != nil {
+ t.Fatalf("a first backup in parts: %v", err)
+ }
+ if f.writes != 2 || len(f.objects["main|team/magpie/README.md"]) == 0 || f.parts() != 2 {
+ t.Fatalf("a note, then the backup: commits %d, parts %d", f.writes, f.parts())
+ }
+ puts := slices.DeleteFunc(slices.Clone(f.requests), func(r string) bool { return !strings.HasPrefix(r, "PUT ") })
+ if !slices.Equal(puts, []string{"PUT /repos/alice/private/contents/team/magpie/README.md"}) {
+ t.Fatalf("only the note goes through the Contents API: %v", puts)
+ }
+ if b, got, err := g.get(ctx, version{}); err != nil || got != v || !bytesEqual(b, big) {
+ t.Fatalf("reading it back: %v", err)
+ }
+}
+
+// The reporter's case end to end: a library that seals to a backup past
+// one request syncs, and another computer joins with all of it.
+func TestGitHubSyncLargeLibrary(t *testing.T) {
+ f := newGithubFixture(t)
+ ctx := context.Background()
+ c := githubConfig()
+ a, b := newComputer(t), newComputer(t)
+ a.use(t)
+ // Printable, so the text is carried as it is: about 38 MB of text
+ // seals to about 51 MB, as the reporter's export.
+ raw := filled(28<<20, 10)
+ text := string(encodeText(raw))
+ if _, err := library.SaveInstructions(library.InstructionsChange{Shared: &text}); err != nil {
+ t.Fatal(err)
+ }
+ if err := Configure(c); err != nil {
+ t.Fatal(err)
+ }
+ if err := Now(ctx); err != nil {
+ t.Fatalf("syncing a large library: %v", err)
+ }
+ if f.parts() < 2 {
+ t.Fatalf("the backup should have gone up in parts: %d", f.parts())
+ }
+ b.use(t)
+ if err := Configure(c); err != nil {
+ t.Fatal(err)
+ }
+ if err := Now(ctx); err != nil {
+ t.Fatalf("the second computer joins: %v", err)
+ }
+ got, err := library.Collect()
+ if err != nil || got == nil || !slices.Contains(slices.Collect(maps.Values(got.Texts)), text) {
+ t.Fatalf("the second computer's library: %v", err)
+ }
+}
+
+func encodeText(b []byte) []byte {
+ const letters = "abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789 \n"
+ out := make([]byte, len(b)*4/3)
+ for i := range out {
+ out[i] = letters[int(b[i*3/4])%len(letters)]
+ }
+ return out
+}
diff --git a/internal/davsync/github_test.go b/internal/davsync/github_test.go
index 21fc39acf..f1eec988a 100644
--- a/internal/davsync/github_test.go
+++ b/internal/davsync/github_test.go
@@ -44,6 +44,13 @@ type githubFixture struct {
// contentsLimit is the largest file the Contents API's PUT takes;
// GitHub answers 422 "too large" over its own (undocumented) limit.
contentsLimit int
+ // blobLimit is the largest blob one POST takes. GitHub documents no
+ // request size; it refused a 50.8 MB blob (about 68 MB of base64 body)
+ // with 422 "Sorry, your input was too large to process" (Discord),
+ // and took a 19 MB file's Contents PUT (about 25 MB of body).
+ blobLimit int
+ // biggestBlob is the largest blob a POST sent.
+ biggestBlob int
// beforeRef runs, under the lock, before a ref update is judged: a
// test commits there as another computer would.
beforeRef func()
@@ -118,7 +125,7 @@ func githubSHA(b []byte) string {
func newGithubFixture(t *testing.T) *githubFixture {
t.Helper()
f := &githubFixture{objects: map[string][]byte{}, defaultBranch: "main", heads: map[string]string{},
- blobs: map[string][]byte{}, trees: map[string]fixtureTree{}, commits: map[string]fixtureCommit{}, contentsLimit: 1 << 20}
+ blobs: map[string][]byte{}, trees: map[string]fixtureTree{}, commits: map[string]fixtureCommit{}, contentsLimit: 1 << 20, blobLimit: 24 << 20}
srv := httptest.NewServer(http.HandlerFunc(f.serve))
t.Cleanup(srv.Close)
target, _ := url.Parse(srv.URL)
@@ -176,12 +183,18 @@ func (f *githubFixture) serve(w http.ResponseWriter, r *http.Request) {
}
if strings.HasPrefix(r.URL.Path, base+"/git/blobs/") {
sha := strings.TrimPrefix(r.URL.Path, base+"/git/blobs/")
- for _, b := range f.objects {
- if githubSHA(b) == sha && r.Header.Get("Accept") == "application/vnd.github.raw+json" {
- w.Write(b)
- return
+ // GitHub serves any blob the repository has by its SHA, one a
+ // later commit no longer holds too.
+ b, ok := f.blobs[sha]
+ for _, o := range f.objects {
+ if !ok && githubSHA(o) == sha {
+ b, ok = o, true
}
}
+ if ok && r.Header.Get("Accept") == "application/vnd.github.raw+json" {
+ w.Write(b)
+ return
+ }
fail(404, "Not Found")
return
}
@@ -291,6 +304,11 @@ func (f *githubFixture) serveGit(w http.ResponseWriter, r *http.Request, base st
fail(422, "Invalid blob")
return true
}
+ if len(b) > f.blobLimit {
+ fail(422, "Sorry, your input was too large to process. Consider creating the blob in a local clone of the repository and then pushing it to GitHub.")
+ return true
+ }
+ f.biggestBlob = max(f.biggestBlob, len(b))
f.blobs[githubSHA(b)] = b
send(201, map[string]string{"sha": githubSHA(b)})
case strings.HasPrefix(rest, "commits/") && r.Method == http.MethodGet:
@@ -769,7 +787,7 @@ func TestGitHubPublicRepositoryRefused(t *testing.T) {
// is read back by its blob SHA.
func TestGitHubLargeBackup(t *testing.T) {
f := newGithubFixture(t)
- f.contentsLimit = 19 << 20
+ f.contentsLimit = 10 << 20
g, err := newGitHub(githubConfig())
if err != nil {
t.Fatal(err)
@@ -780,14 +798,14 @@ func TestGitHubLargeBackup(t *testing.T) {
if err != nil {
t.Fatal(err)
}
- large := make([]byte, 21<<20)
+ large := make([]byte, 15<<20) // over the Contents API's, under githubPart
for i := range large {
large[i] = byte(i*7919 + i>>11)
}
before := f.writes
v, err = g.put(ctx, large, v.ETag)
if err != nil {
- t.Fatalf("a 21 MB backup: %v", err)
+ t.Fatalf("a 15 MB backup: %v", err)
}
if v.ETag != githubSHA(large) || f.writes != before+1 {
t.Fatalf("one commit per update, versioned by the blob: %+v, commits %d", v, f.writes-before)
@@ -797,7 +815,7 @@ func TestGitHubLargeBackup(t *testing.T) {
}
b, got, err := g.get(ctx, version{})
if err != nil || got != v || !bytesEqual(b, large) {
- t.Fatalf("reading the 21 MB backup back: size %d, %+v, %v", len(b), got, err)
+ t.Fatalf("reading the 15 MB backup back: size %d, %+v, %v", len(b), got, err)
}
if !slices.Contains(f.requests, "GET /repos/alice/private/git/blobs/"+v.ETag) {
t.Fatal("a file over 1 MB must be downloaded by its blob SHA")