Skip to content

Every agent's model. One place. Codex on DeepSeek, Claude Code on Kimi, from the menu bar.

plugin: the dist index is read from GitHub beside jsDelivr's copy, so a plugin npm doesn't have is offered its newer version

yetonec48fccfad74b
diff --git a/docs/subsystems/provider-plugins.md b/docs/subsystems/provider-plugins.md
index e4cb759a4..11484bea7 100644
--- a/docs/subsystems/provider-plugins.md
+++ b/docs/subsystems/provider-plugins.md
@@ -174,7 +174,7 @@ New users do not see an unused deprecated built-in in the Add sheet. `unusedSub`
The editor's *Update* button and *Update all* install npm's newest version; a pin the user set is kept until they update it themselves, and is then unpinned. `keepMovedCurrent` is the one path that acts without the user pressing anything, so `installPlugin` leaves what the user chose: a pinned npm version below the move's `min` is not reinstalled, and the move says it is pinned and which version it needs; a git install is reinstalled from its own spec; an unpinned npm one goes through `plugin.Upgrade` (npm's newest, which bun says why it won't install under a `minimumReleaseAge`, where `Add`'s `@latest` would quietly give an older one); a plugin the user turned off is left alone.
-A `@magpie-community` package npm doesn't give is installed from the community repo's `dist` branch ([`dist.go`](../../internal/plugin/dist.go)). npm can't take a new package's first version without an owner's 2FA (Trusted Publishing sets up only a package that exists), so reviewed plugins sat unmerged waiting for someone to publish them (magpie-community/plugins#84); a Trusted Publisher made after 2026-09-03 allows stage publish only, and zed 0.1.7 and zen-free 0.1.7 were refused that way. The repo's `publish.yml` now packs every merged version onto `dist` (`scripts/dist.mjs`: `tgz/<file>` and `index.json` with each version's sha512, manifest and the README) with nobody's credentials, and npm is published to beside it. `install` asks npm first, as before; only when `bun add` fails for a community package at `latest` or an exact version (`distWanted`), and not because of the user's `minimumReleaseAge` (that hold is kept), does `installDist` fetch the tarball, refuse it unless its sha512 is the index's, keep it in `plugins/dist` and `bun add ./dist/<file>`. A later install from npm takes its place and `pruneDist` removes the tarballs neither `package.json` names nor `bun.lock`'s `packages` has a package at: Bun installs from the lock's, which can be behind `package.json`, and Bun 1.4.0 writes a package it adds from a tarball into `package.json` a second time, the new tarball first, where `encoding/json` reads the second. A tarball only the lock's `workspaces` names (Bun 1.4.2 and 1.4.3 leave the one replaced there) goes. A lock behind `package.json` put a package's older version back at every bun command, and, once that tarball was gone from `plugins/dist`, failed every `bun add` and `bun remove`, of any plugin, with `ENOENT extracting tarball from <that package>` (#3622, Windows, Bun 1.4.4), so before each bun command `mendDist` runs one `bun update` of every package the lock has at a dist tarball `package.json` doesn't name, when `package.json`'s is there, which takes the lock back to `package.json`'s without the network (`bun add` doesn't, on Bun 1.4.2 and 1.4.3), and logs it. Both read the lock's first object alone (`lockHead`), as Bun does: a shorter lock written over a longer one without truncating, as `package.json` was (magpie-community/plugins#405), leaves the old lock's packages after it, which would keep their tarballs and send them to `bun update` at every command. `npmAsk` (the market's version, *Update*, the hourly check) takes `dist`'s latest when npm errors or is behind, and `Readme` falls back to its README. The branch is read through the 「国内镜像」 switch's jsDelivr copy when that is on (`fetchJSONFaithful`), else raw.githubusercontent.com. `TestDistInstallsAPackageNPMDoesntHave`, `TestDistServesAVersionNPMRefused`, `TestDistRefusesATarballNotItsIndexs`, `TestDistTarballTheLockNamesStays`, `TestDistLockBehindPackageJSONIsMended`, `TestDistLockTailIsntRead` and `TestUpdateBunHoldsBackSaysWhy` (with `dist` holding the held version) in [`dist_test.go`](../../internal/plugin/dist_test.go) and [`update_held_test.go`](../../internal/plugin/update_held_test.go) cover these.
+A `@magpie-community` package npm doesn't give is installed from the community repo's `dist` branch ([`dist.go`](../../internal/plugin/dist.go)). npm can't take a new package's first version without an owner's 2FA (Trusted Publishing sets up only a package that exists), so reviewed plugins sat unmerged waiting for someone to publish them (magpie-community/plugins#84); a Trusted Publisher made after 2026-09-03 allows stage publish only, and zed 0.1.7 and zen-free 0.1.7 were refused that way. The repo's `publish.yml` now packs every merged version onto `dist` (`scripts/dist.mjs`: `tgz/<file>` and `index.json` with each version's sha512, manifest and the README) with nobody's credentials, and npm is published to beside it. `install` asks npm first, as before; only when `bun add` fails for a community package at `latest` or an exact version (`distWanted`), and not because of the user's `minimumReleaseAge` (that hold is kept), does `installDist` fetch the tarball, refuse it unless its sha512 is the index's, keep it in `plugins/dist` and `bun add ./dist/<file>`. A later install from npm takes its place and `pruneDist` removes the tarballs neither `package.json` names nor `bun.lock`'s `packages` has a package at: Bun installs from the lock's, which can be behind `package.json`, and Bun 1.4.0 writes a package it adds from a tarball into `package.json` a second time, the new tarball first, where `encoding/json` reads the second. A tarball only the lock's `workspaces` names (Bun 1.4.2 and 1.4.3 leave the one replaced there) goes. A lock behind `package.json` put a package's older version back at every bun command, and, once that tarball was gone from `plugins/dist`, failed every `bun add` and `bun remove`, of any plugin, with `ENOENT extracting tarball from <that package>` (#3622, Windows, Bun 1.4.4), so before each bun command `mendDist` runs one `bun update` of every package the lock has at a dist tarball `package.json` doesn't name, when `package.json`'s is there, which takes the lock back to `package.json`'s without the network (`bun add` doesn't, on Bun 1.4.2 and 1.4.3), and logs it. Both read the lock's first object alone (`lockHead`), as Bun does: a shorter lock written over a longer one without truncating, as `package.json` was (magpie-community/plugins#405), leaves the old lock's packages after it, which would keep their tarballs and send them to `bun update` at every command. `npmAsk` (the market's version, *Update*, the hourly check) takes `dist`'s latest when npm errors or is behind, and `Readme` falls back to its README. Tarballs are read through the 「国内镜像」 switch's jsDelivr copy when that is on (`fetchJSONFaithful`), else raw.githubusercontent.com. `index.json` is read from raw.githubusercontent.com, and with the switch on from jsDelivr beside it (`readDistIndex`): a jsDelivr edge serves the branch as it was when it cached it, up to 12 hours before (`s-maxage=43200`), and a Gcore edge served CodeArts' 0.2.3 as latest for hours after 0.2.4 was packed, so for a package npm doesn't have the hourly update, *Check for updates* and *Update* all found nothing newer (star21321321124 on Discord). The two copies are merged (`mergeDist`: every version either names, each package's newer latest), each version keeping the sha512 its own copy names; GitHub is waited for `distGrace` (5s) past the mirror's answer, and where it can't be reached the mirror's copy is read alone. `TestDistOnlyPluginUpdatesPastAStaleMirror`, `TestDistIndexFromTheMirrorAloneWhenGitHubFails` and `TestDistIndexDoesntWaitOutAHangingGitHub` in [`dist_updates_test.go`](../../internal/plugin/dist_updates_test.go), `TestDistInstallsAPackageNPMDoesntHave`, `TestDistServesAVersionNPMRefused`, `TestDistRefusesATarballNotItsIndexs`, `TestDistTarballTheLockNamesStays`, `TestDistLockBehindPackageJSONIsMended`, `TestDistLockTailIsntRead` and `TestUpdateBunHoldsBackSaysWhy` (with `dist` holding the held version) in [`dist_test.go`](../../internal/plugin/dist_test.go) and [`update_held_test.go`](../../internal/plugin/update_held_test.go) cover these.
A plugin on the list whose package isn't in `plugins/` is installed again from its own spec, switched off or not, by `plugin.InstallMissing`. `KeepUpdated` runs it 30 seconds after start and then hourly, before `CheckUpdates`. Nothing else did this. An update skips a plugin with no installed version, so the host said `no such plugin: …\node_modules\…` at every start until the user removed the plugin and added it again (#2002, Windows, after an update). Every bun command that changes `plugins/` runs one at a time under `installMu`: `install` (with the first `package.json` it writes, through temp and rename), `reinstall`'s `bun update`, and `Remove`'s and `add`'s `bun remove`, each after `mendPackageJSON` and then `mendDist`, which reads `package.json`. At start, `KeepUpdated` and `keepMovedCurrent` both run `bun add`. Two at once each read `package.json` before either writes it, so one change was lost (reproduced with Bun 1.4.3 on Windows), and a git plugin whose dependency was dropped can't be found. `TestMissingPluginsInstalledAgain` and `TestBunCommandsOneAtATime` in [`plugin_missing_test.go`](../../internal/plugin/plugin_missing_test.go) cover both.
diff --git a/internal/plugin/dist.go b/internal/plugin/dist.go
index 3a3db0a8b..002da8353 100644
--- a/internal/plugin/dist.go
+++ b/internal/plugin/dist.go
@@ -18,6 +18,9 @@ import (
"strings"
"sync"
"time"
+
+ "github.com/yetone/magpie/internal/source"
+ "github.com/yetone/magpie/internal/update"
)
// The community repo's dist branch: every version of each
@@ -76,10 +79,8 @@ func distPkg(ctx context.Context, name string) *distPackage {
defer distMu.Unlock()
stale := distIdx == nil || time.Since(distAt) > distKeep
if stale && (distOK || distAt.IsZero() || time.Since(distAt) > distWait) {
- b, err := fetchJSONFaithful(ctx, distBase+"/index.json", 32<<20)
- var idx distIndex
- if err == nil && json.Unmarshal(b, &idx) == nil {
- distIdx, distOK = &idx, true
+ if idx, err := readDistIndex(ctx); err == nil {
+ distIdx, distOK = idx, true
} else {
// what was read last stays, until a read works
distOK = false
@@ -96,6 +97,116 @@ func distPkg(ctx context.Context, name string) *distPackage {
return &p
}
+var (
+ // distMirror is the copy of the branch's file at u asked before it,
+ // the 「国内镜像」 switch's jsDelivr, "" when none is (moved by tests)
+ distMirror = source.ChinaFirst
+ // distGrace is how much longer GitHub's own index.json is waited for
+ // once the mirror's has come: where GitHub can't be reached, it isn't
+ // waited for to the end of the request's time
+ distGrace = 5 * time.Second
+)
+
+// readDistIndex reads the branch's index.json. jsDelivr's copy, asked first
+// with the 「国内镜像」 switch on, is the branch as it was when the edge
+// cached it, up to 12 hours before (s-maxage=43200): a Gcore edge served
+// CodeArts' 0.2.3 for hours after 0.2.4 was packed, and with npm not having
+// the package at all, the hourly update, Check for updates and Update each
+// found nothing newer and the page said every plugin was up to date
+// (star21321321124 on Discord). So GitHub's own copy is asked beside it and
+// the two are merged (mergeDist). Each version still comes with the
+// sha512 its own copy of the branch names, which installDist checks the
+// tarball against, as it did. Where GitHub can't be reached, the mirror's
+// alone is read, as before.
+func readDistIndex(ctx context.Context) (*distIndex, error) {
+ u := distBase + "/index.json"
+ m := distMirror(u)
+ if m == "" {
+ return parseDistIndex(fetchJSONFrom(ctx, u, 32<<20, source.DoOfficial))
+ }
+ type read struct {
+ idx *distIndex
+ err error
+ }
+ octx, cancel := context.WithCancel(ctx)
+ defer cancel()
+ official, mirror := make(chan read, 1), make(chan read, 1)
+ go func() {
+ idx, err := parseDistIndex(fetchJSONFrom(octx, u, 32<<20, source.DoOfficial))
+ official <- read{idx, err}
+ }()
+ go func() {
+ idx, err := parseDistIndex(fetchJSONFrom(ctx, m, 32<<20, source.DoOfficial))
+ mirror <- read{idx, err}
+ }()
+ mr := <-mirror
+ var or read
+ if mr.err != nil {
+ or = <-official
+ } else {
+ t := time.NewTimer(distGrace)
+ select {
+ case or = <-official:
+ case <-t.C:
+ cancel()
+ or = <-official
+ }
+ t.Stop()
+ }
+ switch {
+ case or.err != nil && mr.err != nil:
+ return nil, errors.Join(or.err, mr.err)
+ case or.err != nil:
+ return mr.idx, nil
+ case mr.err != nil:
+ return or.idx, nil
+ }
+ return mergeDist(or.idx, mr.idx), nil
+}
+
+func parseDistIndex(b []byte, err error) (*distIndex, error) {
+ if err != nil {
+ return nil, err
+ }
+ var idx distIndex
+ if err := json.Unmarshal(b, &idx); err != nil {
+ return nil, err
+ }
+ return &idx, nil
+}
+
+// mergeDist is two copies of the branch's index as one: every version
+// either names (the branch only ever gains versions), a's entry where both
+// do, and each package's newer latest, with the README of the copy that
+// named it.
+func mergeDist(a, b *distIndex) *distIndex {
+ out := &distIndex{Packages: map[string]distPackage{}}
+ for name, p := range a.Packages {
+ out.Packages[name] = p
+ }
+ for name, pb := range b.Packages {
+ pa, ok := out.Packages[name]
+ if !ok {
+ out.Packages[name] = pb
+ continue
+ }
+ vs := make(map[string]distVersion, len(pa.Versions)+len(pb.Versions))
+ for v, x := range pb.Versions {
+ vs[v] = x
+ }
+ for v, x := range pa.Versions {
+ vs[v] = x
+ }
+ p := pa
+ if update.Newer(pb.Latest, pa.Latest) {
+ p.Latest, p.Readme = pb.Latest, pb.Readme
+ }
+ p.Versions = vs
+ out.Packages[name] = p
+ }
+ return out
+}
+
// distInfo is the package's newest version on the dist branch, as npm's
// /latest would say it, and whether there is one.
func distInfo(ctx context.Context, name string) (NPM, bool) {
diff --git a/internal/plugin/dist_test.go b/internal/plugin/dist_test.go
index 6ab99cad9..326dd3ea1 100644
--- a/internal/plugin/dist_test.go
+++ b/internal/plugin/dist_test.go
@@ -26,6 +26,22 @@ import (
// named its latest); tamper serves every tarball other than the one its
// index names.
func fakeDist(t *testing.T, pkgs map[string][]string, tamper bool) {
+ t.Helper()
+ u := distServer(t, pkgs, tamper)
+ forgetDist := func() {
+ distMu.Lock()
+ distIdx, distAt, distOK = nil, time.Time{}, false
+ distMu.Unlock()
+ }
+ base := distBase
+ distBase = u
+ forgetDist()
+ t.Cleanup(func() { distBase = base; forgetDist() })
+}
+
+// distServer serves a copy of the dist branch, as fakeDist describes, and
+// is its address.
+func distServer(t *testing.T, pkgs map[string][]string, tamper bool) string {
t.Helper()
files := map[string][]byte{}
idx := map[string]any{}
@@ -56,15 +72,7 @@ func fakeDist(t *testing.T, pkgs map[string][]string, tamper bool) {
http.NotFound(w, r)
}))
t.Cleanup(srv.Close)
- forgetDist := func() {
- distMu.Lock()
- distIdx, distAt, distOK = nil, time.Time{}, false
- distMu.Unlock()
- }
- base := distBase
- distBase = srv.URL
- forgetDist()
- t.Cleanup(func() { distBase = base; forgetDist() })
+ return srv.URL
}
// npmHasNothing is an npm registry that has no package at all: a new
diff --git a/internal/plugin/dist_updates_test.go b/internal/plugin/dist_updates_test.go
new file mode 100644
index 000000000..b7dd9f7f8
--- /dev/null
+++ b/internal/plugin/dist_updates_test.go
@@ -0,0 +1,120 @@
+package plugin
+
+import (
+ "context"
+ "net/http"
+ "net/http/httptest"
+ "strings"
+ "testing"
+ "time"
+)
+
+// A community plugin npm has never had (CodeArts and River were merged and
+// packed onto the dist branch, never published to npm) is offered the
+// branch's newer version with the 「国内镜像」 switch on, while jsDelivr's
+// copy of the branch is still the one from before it: the hourly update
+// installs it, Check for updates says "update", and Update installs it
+// (star21321321124 on Discord: CodeArts 0.2.3 "auto-updated", every plugin
+// "up to date", Update doing nothing, with 0.2.4 on the branch and a Gcore
+// edge of jsDelivr serving the index with 0.2.3 as its latest). A plugin
+// with no newer version anywhere stays current, not unknown.
+func TestDistOnlyPluginUpdatesPastAStaleMirror(t *testing.T) {
+ for _, how := range []string{"hourly", "update"} {
+ t.Run(how, func(t *testing.T) {
+ sandbox(t)
+ const pkg = "@magpie-community/opencode-codearts-auth"
+ const river = "@magpie-community/opencode-river-auth"
+ npmHasNothing(t)
+ fakeDist(t, map[string][]string{pkg: {"0.2.3"}, river: {"0.1.0"}}, false)
+ ctx, cancel := context.WithTimeout(context.Background(), 3*time.Minute)
+ defer cancel()
+ for _, p := range []string{pkg, river} {
+ if _, err := Add(ctx, p); err != nil {
+ t.Fatalf("Add %s: %v", p, err)
+ }
+ }
+ if v := Installed(pkg); v != "0.2.3" {
+ t.Fatalf("installed %q, want 0.2.3", v)
+ }
+
+ // the branch gains 0.2.4; the mirror's copy is still the old one
+ stale := distServer(t, map[string][]string{pkg: {"0.2.3"}, river: {"0.1.0"}}, false)
+ fakeDist(t, map[string][]string{pkg: {"0.2.3", "0.2.4"}, river: {"0.1.0"}}, false)
+ was := distMirror
+ distMirror = func(u string) string { return stale + strings.TrimPrefix(u, distBase) }
+ t.Cleanup(func() { distMirror = was })
+
+ checks := CheckNow(ctx)
+ got := map[string]VersionCheck{}
+ for _, c := range checks.Plugins {
+ got[c.Package] = c
+ }
+ if c := got[pkg]; c.Status != "update" || c.Latest != "0.2.4" || c.Version != "0.2.3" {
+ t.Fatalf("Check for updates found %+v for CodeArts, want 0.2.3 with an update to 0.2.4", c)
+ }
+ if c := got[river]; c.Status != "current" || c.Latest != "0.1.0" {
+ t.Fatalf("Check for updates found %+v for River, want 0.1.0 current", c)
+ }
+
+ if how == "hourly" {
+ u, err := CheckUpdates(ctx)
+ if err != nil {
+ t.Fatalf("CheckUpdates: %v", err)
+ }
+ if len(u.Updated) != 1 || u.Updated[0].Package != pkg || u.Updated[0].From != "0.2.3" || u.Updated[0].To != "0.2.4" {
+ t.Fatalf("the hourly look updated %+v, want CodeArts 0.2.3 to 0.2.4", u.Updated)
+ }
+ } else if err := Upgrade(ctx, pkg); err != nil {
+ t.Fatalf("Update: %v", err)
+ }
+ if v := Installed(pkg); v != "0.2.4" {
+ t.Fatalf("installed %q, want 0.2.4", v)
+ }
+ if v := Installed(river); v != "0.1.0" {
+ t.Fatalf("River is at %q, want 0.1.0", v)
+ }
+ })
+ }
+}
+
+// Where GitHub can't be reached, the mirror's copy of the index is read
+// alone, as before.
+func TestDistIndexFromTheMirrorAloneWhenGitHubFails(t *testing.T) {
+ sandbox(t)
+ const pkg = "@magpie-community/opencode-codearts-auth"
+ mirror := distServer(t, map[string][]string{pkg: {"0.2.3", "0.2.4"}}, false)
+ fakeDist(t, nil, false)
+ distBase = "http://127.0.0.1:1" // nothing listens there
+ was := distMirror
+ distMirror = func(u string) string { return mirror + strings.TrimPrefix(u, distBase) }
+ t.Cleanup(func() { distMirror = was })
+ ctx, cancel := context.WithTimeout(context.Background(), 30*time.Second)
+ defer cancel()
+ if info, ok := distInfo(ctx, pkg); !ok || info.Version != "0.2.4" {
+ t.Fatalf("distInfo = %+v, %v; want the mirror's 0.2.4", info, ok)
+ }
+}
+
+// GitHub answering nothing, as where it is blocked, holds the mirror's
+// index back by distGrace only, not the request's whole time.
+func TestDistIndexDoesntWaitOutAHangingGitHub(t *testing.T) {
+ sandbox(t)
+ const pkg = "@magpie-community/opencode-codearts-auth"
+ mirror := distServer(t, map[string][]string{pkg: {"0.2.3", "0.2.4"}}, false)
+ hang := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { <-r.Context().Done() }))
+ t.Cleanup(hang.Close)
+ fakeDist(t, nil, false)
+ distBase = hang.URL
+ was, grace := distMirror, distGrace
+ distMirror = func(u string) string { return mirror + strings.TrimPrefix(u, distBase) }
+ distGrace = 50 * time.Millisecond
+ t.Cleanup(func() { distMirror, distGrace = was, grace })
+ ctx, cancel := context.WithTimeout(context.Background(), time.Minute)
+ defer cancel()
+ if info, ok := distInfo(ctx, pkg); !ok || info.Version != "0.2.4" {
+ t.Fatalf("distInfo = %+v, %v; want the mirror's 0.2.4", info, ok)
+ }
+ if ctx.Err() != nil {
+ t.Fatal("waited out the request's whole time for GitHub")
+ }
+}
diff --git a/internal/source/source.go b/internal/source/source.go
index 4ba394582..dba761b10 100644
--- a/internal/source/source.go
+++ b/internal/source/source.go
@@ -67,6 +67,15 @@ func chinaFirst(raw string) string {
return ""
}
+// ChinaFirst is the mirror in China the 「国内镜像」 switch asks before raw,
+// "" with the switch off or when no mirror serves raw.
+func ChinaFirst(raw string) string {
+ if !China() {
+ return ""
+ }
+ return chinaFirst(raw)
+}
+
// modelMirror is hf-mirror's copy of one of the local masking models'
// files (internal/pii), "" for anything else of Hugging Face's. Those
// files are fetched at the revision their sha256 were pinned at, and what