Every agent's model. One place. Codex on DeepSeek, Claude Code on Kimi, from the menu bar.
davsync: a GitHub backup too large for one request goes up in parts (Zhang-Wei, Discord)
yetonea6efe07cd797
diff --git a/docs/subsystems/sync-backup.md b/docs/subsystems/sync-backup.mdindex ab2531b06..df8708d82 100644--- a/docs/subsystems/sync-backup.md+++ b/docs/subsystems/sync-backup.md@@ -114,6 +114,28 @@ repository can keep the token. The branch is part of the sync state key.commit is built. A ref update that is not a fast-forward is built again onthe new head, up to three times; if the new head changed the file it is`errChanged`. A branch deleted meanwhile is an error, never created.+- A blob goes up as base64 in one JSON body, a third larger than the file,+ and GitHub refuses a body well under the 100 MB a blob can hold: a+ 50.8 MB backup (about 68 MB of body) got 422 "Sorry, your input was too+ large to process" (Zhang-Wei, Discord), while a 19 MB file's Contents+ PUT (about 25 MB of body) went through; GitHub documents no request+ size. So a backup over 16 MiB (`githubPart`) goes up in parts, in the+ same one commit: `magpie.magpie-backup.parts/1`, `/2`, … beside it, and+ at the backup's own path a short list of them (`githubParts`: format+ `magpie-backup`, version 2, `split: github-parts`, the total size and+ each part's blob SHA and size). The list's blob SHA is the version, and+ it names each part by blob SHA, so one version pins one whole set; a+ read downloads each part by that SHA, checks it, and returns the backup+ whole. A list that doesn't add up or a part that isn't the one named is+ an error, never a backup put together wrong. A write removes the parts+ the list at the head named and the new backup doesn't use, in the same+ commit, so going back to one file leaves no part behind. Backups up to+ 16 MiB stay one file, as before.+- A magpie from before parts opens the list as a backup with version 2+ and no key derivation, and stops with "made by a newer magpie; update+ magpie" — not `ErrCorrupt` or `ErrNotBackup`, so it neither rebuilds+ over it nor offers to replace it. A backup in one file, from any+ magpie, still reads.- Tokens are separate from the library's GitHub token, sent only to`https://api.github.com`, and must differ from the encryption passphrase.Repository history retains older encrypted versions.@@ -123,8 +145,9 @@ repository can keep the token. The branch is part of the sync state key.initialization, after repository access and the default branch are checked.The initial write omits `branch` so GitHub creates its default branch; otherconflicts and missing branches are not treated as an empty repository.-- Large files are downloaded by blob SHA, preserving the version that was- read. Sync files are limited to 64 MiB. Usage listing fails at the Contents+- Large files and parts are downloaded by blob SHA, preserving the version+ that was read. Sync files are limited to 64 MiB in all (four parts), the+ bound the library's skill budget is sized for. Usage listing fails at the ContentsAPI's 1,000-file limit rather than treating a truncated list as complete.- A stale SHA, a backup another commit changed, or a racing initialcreation returns `errChanged`. Other 422@@ -148,6 +171,17 @@ the initial write. `TestGitHubLargeBackup` writes a 21 MB backup past a fixtureAPI refuses it, as one commit, and reads it back by blob SHA.`TestGitHubBranchMovedMeanwhile` covers a head that moved before the refupdate, and `TestGitHubEmptyRepositoryLargeBackup` a large first backup.+`TestGitHubSplitBackup` writes a 50.8 MB backup past a fixture that refuses+a blob over 24 MiB, as four parts in one commit, reads it back byte for+byte, then goes to two parts and back to one file with no part left.+`TestGitHubOneFileBackupStillReads` reads a 30 MB backup in one file and+replaces it with parts; `TestGitHubPartsOlderMagpie` checks the list stops+an older magpie as a newer magpie's backup; `TestGitHubPartsDamaged`,+`TestGitHubPartsBranchMovedMeanwhile` and+`TestGitHubEmptyRepositoryPartsBackup` cover a bad list or part, a moved+head (each blob sent once) and an empty repository;+`TestGitHubSyncLargeLibrary` syncs a library that seals past one request+to a second computer.`TestGitHubNotBackup` covers backend-specific recoveryadvice when the repository holds an empty or foreign file.diff --git a/internal/davsync/github.go b/internal/davsync/github.goindex a750a89f1..08e684291 100644--- a/internal/davsync/github.go+++ b/internal/davsync/github.go@@ -20,6 +20,67 @@ import (const githubMaxFile = 64 << 20+// githubPart is the most of a backup one request sends. A blob goes up as+// base64 in a JSON body, a third bigger than the file, and GitHub refuses a+// body far under the 100 MB a blob can hold: a 50.8 MB backup (about 68 MB+// of body) got 422 "Sorry, your input was too large to process" (Zhang-Wei,+// Discord), while a 19 MB file's Contents PUT, about 25 MB of body, went+// through. GitHub documents no request size, so a part stays under the+// largest body seen accepted.+const githubPart = 16 << 20++// githubSplit names a backup sent in parts. Its file at the backup's path+// is a short list of the parts, each a blob in a folder beside it, all in+// the one commit; the list's own blob SHA is the version, and it names+// every part by its blob SHA, so a version pins one whole set. The list+// says it is format magpie-backup, version 2, with no key derivation, so a+// magpie before parts opens it as a newer magpie's backup and stops+// ("update magpie"), never as a damaged one (ErrCorrupt, ErrNotBackup) to+// rebuild or offer for replacing.+const githubSplit = "github-parts"++type githubParts struct {+ Format string `json:"format"`+ Version int `json:"version"`+ Split string `json:"split"`+ Size int64 `json:"size"`+ Parts []githubPartRef `json:"parts"`+}++type githubPartRef struct {+ SHA string `json:"sha"`+ Size int64 `json:"size"`+}++// partName is the backup's i-th part (from 0), beside it in magpie's folder.+func partName(i int) string { return file + ".parts/" + strconv.Itoa(i+1) }++// parseParts reads b as a list of parts: nil for a file that isn't one (a+// backup in one file), an error for a list that doesn't add up.+func parseParts(b []byte) (*githubParts, error) {+ if len(b) > 64<<10 {+ return nil, nil+ }+ var m githubParts+ if json.Unmarshal(b, &m) != nil || m.Split == "" {+ return nil, nil+ }+ if m.Format != "magpie-backup" || m.Version != 2 || m.Split != githubSplit {+ return nil, errors.New("the GitHub backup was sent in parts by a newer magpie; update magpie to read it")+ }+ total := int64(0)+ for _, p := range m.Parts {+ if len(p.SHA) != 40 || p.Size <= 0 || p.Size > githubPart {+ return nil, errors.New("the GitHub backup's list of parts is damaged")+ }+ total += p.Size+ }+ if len(m.Parts) < 2 || total != m.Size || total > githubMaxFile {+ return nil, errors.New("the GitHub backup's list of parts is damaged")+ }+ return &m, nil+}+type github struct {repo, prefix, branch, token stringready bool@@ -274,7 +335,28 @@ func (g *github) get(ctx context.Context, have version) ([]byte, version, error)return nil, v, errNotModified}b, err := g.bytes(ctx, c)- return b, v, err+ if err != nil {+ return nil, v, err+ }+ m, err := parseParts(b)+ if err != nil || m == nil {+ return b, v, err+ }+ // Each part by its blob SHA, the one the list names, so the parts are+ // the set this version holds even if a new backup is committed while+ // they download.+ whole := make([]byte, 0, m.Size)+ for i, p := range m.Parts {+ b, err := g.bytes(ctx, &githubContent{SHA: p.SHA, Size: p.Size, Encoding: "none"})+ if err != nil {+ return nil, v, fmt.Errorf("reading part %d of the GitHub backup: %w", i+1, err)+ }+ if gitBlobSHA(b) != p.SHA {+ return nil, v, fmt.Errorf("GitHub returned a different part %d of the backup than its list names", i+1)+ }+ whole = append(whole, b...)+ }+ return whole, v, nil}// errTooLarge is the Contents API refusing a file it won't process, which@@ -286,12 +368,19 @@ var errTooLarge = errors.New("GitHub's Contents API refused the file as too larg// reached the branch after its head was read.var errBehind = errors.New("the GitHub branch moved meanwhile")+// githubWrite is one blob a commit puts at path.+type githubWrite struct {+ path, sha string+ data []byte+}+// commit writes or removes one file in one commit. A repository with// commits goes through the Git Data API (blob, tree, commit, ref), which// takes files up to 100 MB, where the Contents API refuses large ones (HTTP-// 422 "the file is too large to be processed"). An empty repository has no-// branch to build on, so its first write goes through the Contents API,-// which creates the default branch.+// 422 "the file is too large to be processed"). A backup over githubPart+// goes up in parts, with their list at its path, in that one commit. An+// empty repository has no branch to build on, so its first write goes+// through the Contents API, which creates the default branch.func (g *github) commit(ctx context.Context, name string, data []byte, sha string, remove bool) (version, error) {if len(data) > githubMaxFile {return version{}, errors.New("the GitHub sync file exceeds 64 MiB")@@ -299,10 +388,30 @@ func (g *github) commit(ctx context.Context, name string, data []byte, sha strinif err := g.prepare(ctx); err != nil {return version{}, err}+ var writes []githubWrite+ if !remove && name == file && len(data) > githubPart {+ m := githubParts{Format: "magpie-backup", Version: 2, Split: githubSplit, Size: int64(len(data))}+ for i, off := 0, 0; off < len(data); i, off = i+1, off+githubPart {+ p := data[off:min(off+githubPart, len(data))]+ writes = append(writes, githubWrite{g.at(partName(i)), gitBlobSHA(p), p})+ m.Parts = append(m.Parts, githubPartRef{gitBlobSHA(p), int64(len(p))})+ }+ list, err := json.Marshal(m)+ if err != nil {+ return version{}, err+ }+ writes = append(writes, githubWrite{g.at(name), gitBlobSHA(list), list})+ } else if !remove {+ writes = []githubWrite{{g.at(name), gitBlobSHA(data), data}}+ }if g.empty {- v, err := g.contents(ctx, name, data, sha, remove)- if !errors.Is(err, errTooLarge) {- return v, err+ // One file is tried through the Contents API first; parts can't go+ // that way, so they start on a first commit of the note's.+ if len(writes) <= 1 {+ v, err := g.contents(ctx, name, data, sha, remove)+ if !errors.Is(err, errTooLarge) {+ return v, err+ }}// Too large for the Contents API: give the repository a first// commit of its own, then build the file's commit on it.@@ -310,9 +419,9 @@ func (g *github) commit(ctx context.Context, name string, data []byte, sha strinreturn version{}, err}}- blob := ""+ sent := map[string]bool{}for try := 0; ; try++ {- v, err := g.gitCommit(ctx, name, data, sha, remove, &blob)+ v, err := g.gitCommit(ctx, name, writes, sha, sent)if !errors.Is(err, errBehind) {return v, err}@@ -356,12 +465,14 @@ func (g *github) githubJSON(ctx context.Context, method, apiPath string, in, outreturn nil, nil}-// gitCommit builds one commit on the branch's head that writes data (or-// removes the file) and moves the branch to it. The file must still be at-// sha in that head, so another computer's backup is never written over;-// the ref update without force refuses a head that moved after it was read.-// blob carries the uploaded blob's SHA across retries.-func (g *github) gitCommit(ctx context.Context, name string, data []byte, sha string, remove bool, blob *string) (version, error) {+// gitCommit builds one commit on the branch's head that puts writes (the+// file, or its parts and their list; none removes it) and moves the branch+// to it. The file must still be at sha in that head, so another computer's+// backup is never written over; the ref update without force refuses a+// head that moved after it was read. Parts the file at the head lists and+// writes no longer has are removed with it. sent holds the blobs already+// uploaded, across retries.+func (g *github) gitCommit(ctx context.Context, name string, writes []githubWrite, sha string, sent map[string]bool) (version, error) {var ref struct {Object struct {SHA string `json:"sha"`@@ -383,20 +494,34 @@ func (g *github) gitCommit(ctx context.Context, name string, data []byte, sha stif c != nil && c.SHA != sha || c == nil && sha != "" {return version{}, errChanged}- if !remove && *blob == "" {+ // The parts the file at the head lists. A list that doesn't read+ // leaves its parts where they are: removing only what is known to be+ // there never fails a tree on a path that isn't.+ had := 0+ if c != nil && name == file && c.Encoding == "base64" {+ if b, err := g.bytes(ctx, c); err == nil {+ if m, err := parseParts(b); err == nil && m != nil {+ had = len(m.Parts)+ }+ }+ }+ for _, w := range writes {+ if sent[w.sha] {+ continue+ }var out struct {SHA string `json:"sha"`}- in := map[string]string{"content": base64.StdEncoding.EncodeToString(data), "encoding": "base64"}+ in := map[string]string{"content": base64.StdEncoding.EncodeToString(w.data), "encoding": "base64"}if res, err := g.githubJSON(ctx, http.MethodPost, "git/blobs", in, &out); err != nil {return version{}, err} else if res != nil {return version{}, githubError(res)}- if out.SHA != gitBlobSHA(data) {+ if out.SHA != w.sha {return version{}, errors.New("GitHub stored a different blob than the one sent")}- *blob = out.SHA+ sent[w.sha] = true}var commit struct {Tree struct {@@ -418,15 +543,23 @@ func (g *github) gitCommit(ctx context.Context, name string, data []byte, sha stType string `json:"type"`SHA *string `json:"sha"`}- e := entry{Path: g.at(name), Mode: "100644", Type: "blob"}- message := "magpie: remove " + name- if !remove {- e.SHA, message = blob, "magpie: sync "+name+ var entries []entry+ for _, w := range writes {+ entries = append(entries, entry{Path: w.path, Mode: "100644", Type: "blob", SHA: &w.sha})+ }+ message := "magpie: sync " + name+ if len(writes) == 0 {+ entries = append(entries, entry{Path: g.at(name), Mode: "100644", Type: "blob"})+ message = "magpie: remove " + name+ }+ parts := max(len(writes)-1, 0)+ for i := parts; i < had; i++ {+ entries = append(entries, entry{Path: g.at(partName(i)), Mode: "100644", Type: "blob"})}var tree, made struct {SHA string `json:"sha"`}- if res, err := g.githubJSON(ctx, http.MethodPost, "git/trees", map[string]any{"base_tree": commit.Tree.SHA, "tree": []entry{e}}, &tree); err != nil {+ if res, err := g.githubJSON(ctx, http.MethodPost, "git/trees", map[string]any{"base_tree": commit.Tree.SHA, "tree": entries}, &tree); err != nil {return version{}, err} else if res != nil {return version{}, githubError(res)@@ -455,10 +588,10 @@ func (g *github) gitCommit(ctx context.Context, name string, data []byte, sha streturn version{}, githubError(res)}g.empty = false- if remove {+ if len(writes) == 0 {return version{}, nil}- return version{ETag: *blob}, nil+ return version{ETag: writes[len(writes)-1].sha}, nil}// gitBlobSHA is the SHA git names a blob holding b by: the version GitHubdiff --git a/internal/davsync/github_parts_test.go b/internal/davsync/github_parts_test.gonew file mode 100644index 000000000..7f8e66b64--- /dev/null+++ b/internal/davsync/github_parts_test.go@@ -0,0 +1,339 @@+package davsync++import (+ "context"+ "encoding/json"+ "errors"+ "maps"+ "slices"+ "strconv"+ "strings"+ "testing"++ "github.com/yetone/magpie/internal/backup"+ "github.com/yetone/magpie/internal/library"+)++// filled is n bytes that differ from seed to seed and part to part.+func filled(n int, seed uint32) []byte {+ b := make([]byte, n)+ x := seed*2654435761 + 1+ for i := range b {+ x = x*1664525 + 1013904223+ b[i] = byte(x >> 24)+ }+ return b+}++const backupAt = "main|team/magpie/" + file++func partAt(i int) string { return "main|team/magpie/" + file + ".parts/" + strconv.Itoa(i) }++// parts is how many part files the branch holds beside the backup.+func (f *githubFixture) parts() int {+ n := 0+ for k := range f.objects {+ if strings.HasPrefix(k, backupAt+".parts/") {+ n+++ }+ }+ return n+}++// A backup too large for one blob request goes up in parts, in one commit,+// and reads back whole (Zhang-Wei, Discord: a 50.8 MB backup, about 68 MB+// of base64 in one POST /git/blobs, got 422 "Sorry, your input was too+// large to process").+func TestGitHubSplitBackup(t *testing.T) {+ f := newGithubFixture(t)+ g, err := newGitHub(githubConfig())+ if err != nil {+ t.Fatal(err)+ }+ ctx := context.Background()+ small := []byte("an earlier sealed backup")+ v, err := g.put(ctx, small, "")+ if err != nil {+ t.Fatal(err)+ }+ big := filled(50_800_000, 1)+ before := f.writes+ v, err = g.put(ctx, big, v.ETag)+ if err != nil {+ t.Fatalf("a 50.8 MB backup: %v", err)+ }+ list := f.objects[backupAt]+ if f.writes != before+1 || v.ETag != githubSHA(list) || f.parts() != 4 {+ t.Fatalf("one commit of four parts and their list, versioned by the list: commits %d, %+v, parts %d", f.writes-before, v, f.parts())+ }+ if f.biggestBlob > githubPart {+ t.Fatalf("a blob of %d bytes was sent in one request", f.biggestBlob)+ }+ b, got, err := g.get(ctx, version{})+ if err != nil || got != v || !bytesEqual(b, big) {+ t.Fatalf("reading the 50.8 MB backup back: size %d, %+v, %v", len(b), got, err)+ }+ if _, _, err := g.get(ctx, v); !errors.Is(err, errNotModified) {+ t.Fatalf("unchanged: %v", err)+ }+ if _, err := g.put(ctx, small, "old-sha"); !errors.Is(err, errChanged) || !bytesEqual(f.objects[backupAt], list) {+ t.Fatalf("a stale version must not replace the backup: %v", err)+ }++ // A smaller backup in fewer parts takes the parts it no longer has+ // away in the same commit.+ mid := filled(20<<20, 2)+ before = f.writes+ if v, err = g.put(ctx, mid, v.ETag); err != nil {+ t.Fatal(err)+ }+ if f.writes != before+1 || f.parts() != 2 {+ t.Fatalf("two parts left after one commit: commits %d, parts %d", f.writes-before, f.parts())+ }+ if b, _, err := g.get(ctx, version{}); err != nil || !bytesEqual(b, mid) {+ t.Fatalf("reading the 20 MB backup back: size %d, %v", len(b), err)+ }++ // Back to one file: no part is left behind.+ if v, err = g.put(ctx, small, v.ETag); err != nil {+ t.Fatal(err)+ }+ if f.parts() != 0 || string(f.objects[backupAt]) != string(small) {+ t.Fatalf("parts left beside a backup in one file: %d", f.parts())+ }+ if b, got, err := g.get(ctx, version{}); err != nil || got != v || string(b) != string(small) {+ t.Fatalf("reading the small backup back: %q %v", b, err)+ }+}++// A backup an older magpie wrote as one file, large or small, still reads,+// and a backup in parts can replace it.+func TestGitHubOneFileBackupStillReads(t *testing.T) {+ f := newGithubFixture(t)+ g, err := newGitHub(githubConfig())+ if err != nil {+ t.Fatal(err)+ }+ ctx := context.Background()+ for _, old := range [][]byte{[]byte(`{"format":"magpie-backup","version":1}`), filled(30<<20, 3)} {+ f.objects[backupAt] = old+ f.advance("main")+ b, v, err := g.get(ctx, version{})+ if err != nil || v.ETag != githubSHA(old) || !bytesEqual(b, old) {+ t.Fatalf("a backup in one file of %d bytes: size %d, %v", len(old), len(b), err)+ }+ }+ big := filled(40<<20, 4)+ v, err := g.put(ctx, big, githubSHA(f.objects[backupAt]))+ if err != nil || f.parts() != 3 {+ t.Fatalf("parts over a backup in one file: %v, parts %d", err, f.parts())+ }+ if b, got, err := g.get(ctx, version{}); err != nil || got != v || !bytesEqual(b, big) {+ t.Fatalf("reading it back: size %d, %v", len(b), err)+ }+}++// A magpie from before parts reads the list as a newer magpie's backup and+// stops: never a damaged file to rebuild, nor one offered for replacing.+func TestGitHubPartsOlderMagpie(t *testing.T) {+ f := newGithubFixture(t)+ g, err := newGitHub(githubConfig())+ if err != nil {+ t.Fatal(err)+ }+ if _, err := g.put(context.Background(), filled(githubPart+1, 5), ""); err != nil {+ t.Fatal(err)+ }+ _, err = backup.Open(f.objects[backupAt], githubConfig().Passphrase)+ if err == nil || errors.Is(err, backup.ErrCorrupt) || errors.Is(err, backup.ErrNotBackup) || !strings.Contains(err.Error(), "newer magpie") {+ t.Fatalf("an older magpie must read the list as a newer magpie's: %v", err)+ }+ if _, ok := described(githubConfig(), f.objects[backupAt], err).(*notBackup); ok {+ t.Fatal("the list must not be offered for replacing")+ }+}++// A list that doesn't add up, or a part that isn't the one it names, is an+// error, never a backup put together wrong.+func TestGitHubPartsDamaged(t *testing.T) {+ setup := func(t *testing.T) (*githubFixture, *github, githubParts) {+ f := newGithubFixture(t)+ g, err := newGitHub(githubConfig())+ if err != nil {+ t.Fatal(err)+ }+ if _, err := g.put(context.Background(), filled(githubPart*2+10, 6), ""); err != nil {+ t.Fatal(err)+ }+ var m githubParts+ if err := json.Unmarshal(f.objects[backupAt], &m); err != nil || len(m.Parts) != 3 {+ t.Fatalf("the list: %s %v", f.objects[backupAt], err)+ }+ return f, g, m+ }+ write := func(f *githubFixture, m githubParts) {+ b, _ := json.Marshal(m)+ f.objects[backupAt] = b+ f.advance("main")+ }+ t.Run("a part changed", func(t *testing.T) {+ f, g, m := setup(t)+ f.blobs[m.Parts[1].SHA] = filled(int(m.Parts[1].Size), 7)+ if b, _, err := g.get(context.Background(), version{}); err == nil || b != nil || !strings.Contains(err.Error(), "part 2") {+ t.Fatalf("a changed part: size %d, %v", len(b), err)+ }+ })+ t.Run("a part missing", func(t *testing.T) {+ f, g, m := setup(t)+ delete(f.blobs, m.Parts[2].SHA)+ delete(f.objects, partAt(3))+ if b, _, err := g.get(context.Background(), version{}); err == nil || b != nil {+ t.Fatalf("a missing part: size %d, %v", len(b), err)+ }+ })+ t.Run("sizes", func(t *testing.T) {+ f, g, m := setup(t)+ m.Size+++ write(f, m)+ if b, _, err := g.get(context.Background(), version{}); err == nil || !strings.Contains(err.Error(), "damaged") {+ t.Fatalf("sizes that don't add up: size %d, %v", len(b), err)+ }+ })+ t.Run("newer", func(t *testing.T) {+ f, g, m := setup(t)+ m.Split = "github-parts-2"+ write(f, m)+ if _, _, err := g.get(context.Background(), version{}); err == nil || !strings.Contains(err.Error(), "newer magpie") {+ t.Fatalf("a newer magpie's parts: %v", err)+ }+ })+}++// The head moving meanwhile: a commit of another file is built on again+// with the parts already sent; another computer's backup is the merge.+func TestGitHubPartsBranchMovedMeanwhile(t *testing.T) {+ setup := func(t *testing.T) (*githubFixture, *github, version) {+ f := newGithubFixture(t)+ g, err := newGitHub(githubConfig())+ if err != nil {+ t.Fatal(err)+ }+ v, err := g.put(context.Background(), []byte("first"), "")+ if err != nil {+ t.Fatal(err)+ }+ return f, g, v+ }+ big := filled(githubPart*2+1, 8)+ t.Run("another file", func(t *testing.T) {+ f, g, v := setup(t)+ f.beforeRef = func() {+ f.beforeRef = nil+ f.objects["main|team/magpie/usage/other.magpie-usage"] = []byte("other computer")+ f.advance("main")+ }+ f.requests = nil+ if _, err := g.put(context.Background(), big, v.ETag); err != nil {+ t.Fatal(err)+ }+ posts := 0+ for _, r := range f.requests {+ if r == "POST /repos/alice/private/git/blobs" {+ posts+++ }+ }+ if posts != 4 || f.parts() != 3 || string(f.objects["main|team/magpie/usage/other.magpie-usage"]) != "other computer" {+ t.Fatalf("both commits on the branch, each blob sent once: posts %d, parts %d", posts, f.parts())+ }+ if b, _, err := g.get(context.Background(), version{}); err != nil || !bytesEqual(b, big) {+ t.Fatalf("reading it back: %v", err)+ }+ })+ t.Run("the backup", func(t *testing.T) {+ f, g, v := setup(t)+ f.beforeRef = func() {+ f.beforeRef = nil+ f.objects[backupAt] = []byte("other computer's backup")+ f.advance("main")+ }+ if _, err := g.put(context.Background(), big, v.ETag); !errors.Is(err, errChanged) {+ t.Fatalf("another computer's backup must go through the merge: %v", err)+ }+ if string(f.objects[backupAt]) != "other computer's backup" || f.parts() != 0 {+ t.Fatalf("another computer's backup was written over: parts %d", f.parts())+ }+ })+}++// An empty repository's first backup in parts follows the note that+// creates the default branch; nothing large goes through the Contents API.+func TestGitHubEmptyRepositoryPartsBackup(t *testing.T) {+ f := newGithubFixture(t)+ f.empty = true+ g, err := newGitHub(githubConfig())+ if err != nil {+ t.Fatal(err)+ }+ ctx := context.Background()+ big := filled(githubPart+100, 9)+ v, err := g.put(ctx, big, "")+ if err != nil {+ t.Fatalf("a first backup in parts: %v", err)+ }+ if f.writes != 2 || len(f.objects["main|team/magpie/README.md"]) == 0 || f.parts() != 2 {+ t.Fatalf("a note, then the backup: commits %d, parts %d", f.writes, f.parts())+ }+ puts := slices.DeleteFunc(slices.Clone(f.requests), func(r string) bool { return !strings.HasPrefix(r, "PUT ") })+ if !slices.Equal(puts, []string{"PUT /repos/alice/private/contents/team/magpie/README.md"}) {+ t.Fatalf("only the note goes through the Contents API: %v", puts)+ }+ if b, got, err := g.get(ctx, version{}); err != nil || got != v || !bytesEqual(b, big) {+ t.Fatalf("reading it back: %v", err)+ }+}++// The reporter's case end to end: a library that seals to a backup past+// one request syncs, and another computer joins with all of it.+func TestGitHubSyncLargeLibrary(t *testing.T) {+ f := newGithubFixture(t)+ ctx := context.Background()+ c := githubConfig()+ a, b := newComputer(t), newComputer(t)+ a.use(t)+ // Printable, so the text is carried as it is: about 38 MB of text+ // seals to about 51 MB, as the reporter's export.+ raw := filled(28<<20, 10)+ text := string(encodeText(raw))+ if _, err := library.SaveInstructions(library.InstructionsChange{Shared: &text}); err != nil {+ t.Fatal(err)+ }+ if err := Configure(c); err != nil {+ t.Fatal(err)+ }+ if err := Now(ctx); err != nil {+ t.Fatalf("syncing a large library: %v", err)+ }+ if f.parts() < 2 {+ t.Fatalf("the backup should have gone up in parts: %d", f.parts())+ }+ b.use(t)+ if err := Configure(c); err != nil {+ t.Fatal(err)+ }+ if err := Now(ctx); err != nil {+ t.Fatalf("the second computer joins: %v", err)+ }+ got, err := library.Collect()+ if err != nil || got == nil || !slices.Contains(slices.Collect(maps.Values(got.Texts)), text) {+ t.Fatalf("the second computer's library: %v", err)+ }+}++func encodeText(b []byte) []byte {+ const letters = "abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789 \n"+ out := make([]byte, len(b)*4/3)+ for i := range out {+ out[i] = letters[int(b[i*3/4])%len(letters)]+ }+ return out+}diff --git a/internal/davsync/github_test.go b/internal/davsync/github_test.goindex 21fc39acf..f1eec988a 100644--- a/internal/davsync/github_test.go+++ b/internal/davsync/github_test.go@@ -44,6 +44,13 @@ type githubFixture struct {// contentsLimit is the largest file the Contents API's PUT takes;// GitHub answers 422 "too large" over its own (undocumented) limit.contentsLimit int+ // blobLimit is the largest blob one POST takes. GitHub documents no+ // request size; it refused a 50.8 MB blob (about 68 MB of base64 body)+ // with 422 "Sorry, your input was too large to process" (Discord),+ // and took a 19 MB file's Contents PUT (about 25 MB of body).+ blobLimit int+ // biggestBlob is the largest blob a POST sent.+ biggestBlob int// beforeRef runs, under the lock, before a ref update is judged: a// test commits there as another computer would.beforeRef func()@@ -118,7 +125,7 @@ func githubSHA(b []byte) string {func newGithubFixture(t *testing.T) *githubFixture {t.Helper()f := &githubFixture{objects: map[string][]byte{}, defaultBranch: "main", heads: map[string]string{},- blobs: map[string][]byte{}, trees: map[string]fixtureTree{}, commits: map[string]fixtureCommit{}, contentsLimit: 1 << 20}+ blobs: map[string][]byte{}, trees: map[string]fixtureTree{}, commits: map[string]fixtureCommit{}, contentsLimit: 1 << 20, blobLimit: 24 << 20}srv := httptest.NewServer(http.HandlerFunc(f.serve))t.Cleanup(srv.Close)target, _ := url.Parse(srv.URL)@@ -176,12 +183,18 @@ func (f *githubFixture) serve(w http.ResponseWriter, r *http.Request) {}if strings.HasPrefix(r.URL.Path, base+"/git/blobs/") {sha := strings.TrimPrefix(r.URL.Path, base+"/git/blobs/")- for _, b := range f.objects {- if githubSHA(b) == sha && r.Header.Get("Accept") == "application/vnd.github.raw+json" {- w.Write(b)- return+ // GitHub serves any blob the repository has by its SHA, one a+ // later commit no longer holds too.+ b, ok := f.blobs[sha]+ for _, o := range f.objects {+ if !ok && githubSHA(o) == sha {+ b, ok = o, true}}+ if ok && r.Header.Get("Accept") == "application/vnd.github.raw+json" {+ w.Write(b)+ return+ }fail(404, "Not Found")return}@@ -291,6 +304,11 @@ func (f *githubFixture) serveGit(w http.ResponseWriter, r *http.Request, base stfail(422, "Invalid blob")return true}+ if len(b) > f.blobLimit {+ fail(422, "Sorry, your input was too large to process. Consider creating the blob in a local clone of the repository and then pushing it to GitHub.")+ return true+ }+ f.biggestBlob = max(f.biggestBlob, len(b))f.blobs[githubSHA(b)] = bsend(201, map[string]string{"sha": githubSHA(b)})case strings.HasPrefix(rest, "commits/") && r.Method == http.MethodGet:@@ -769,7 +787,7 @@ func TestGitHubPublicRepositoryRefused(t *testing.T) {// is read back by its blob SHA.func TestGitHubLargeBackup(t *testing.T) {f := newGithubFixture(t)- f.contentsLimit = 19 << 20+ f.contentsLimit = 10 << 20g, err := newGitHub(githubConfig())if err != nil {t.Fatal(err)@@ -780,14 +798,14 @@ func TestGitHubLargeBackup(t *testing.T) {if err != nil {t.Fatal(err)}- large := make([]byte, 21<<20)+ large := make([]byte, 15<<20) // over the Contents API's, under githubPartfor i := range large {large[i] = byte(i*7919 + i>>11)}before := f.writesv, err = g.put(ctx, large, v.ETag)if err != nil {- t.Fatalf("a 21 MB backup: %v", err)+ t.Fatalf("a 15 MB backup: %v", err)}if v.ETag != githubSHA(large) || f.writes != before+1 {t.Fatalf("one commit per update, versioned by the blob: %+v, commits %d", v, f.writes-before)@@ -797,7 +815,7 @@ func TestGitHubLargeBackup(t *testing.T) {}b, got, err := g.get(ctx, version{})if err != nil || got != v || !bytesEqual(b, large) {- t.Fatalf("reading the 21 MB backup back: size %d, %+v, %v", len(b), got, err)+ t.Fatalf("reading the 15 MB backup back: size %d, %+v, %v", len(b), got, err)}if !slices.Contains(f.requests, "GET /repos/alice/private/git/blobs/"+v.ETag) {t.Fatal("a file over 1 MB must be downloaded by its blob SHA")