Every agent's model. One place. Codex on DeepSeek, Claude Code on Kimi, from the menu bar.
library: one market skill from a huge repository installs alone; GitHub's file list is never cut (#2794)
yetone33ccc61884fb
diff --git a/docs/subsystems/library.md b/docs/subsystems/library.mdindex 1eb9cf7e4..7e9104b44 100644--- a/docs/subsystems/library.md+++ b/docs/subsystems/library.md@@ -20,7 +20,7 @@ agents RTK, a CLI that cuts shell output down before the model reads it.| Projects | A project (`Project`) is a folder given some of the library's skills and MCP servers, each for some agents. A skill is linked (or copied) into the folder the agent reads a project's skills from (`projectSkillsDirs`): `.claude/skills` for Claude Code, `.qoder/skills` for Qoder and Qoder CN, `.codebuddy/skills` for CodeBuddy Code, `.commandcode/skills` for Command Code, and `.agents/skills` for Codex, Gemini CLI, OpenCode, Pi, Cursor, Copilot CLI, Antigravity, Kimi, Goose, Grok, Droid and Cline. A server is written into the file the agent reads a project's servers from (`projectMCPFiles`), in the shape its user-wide file has: `.mcp.json` for Claude Code and CodeBuddy Code (CodeBuddy's `mcp.json` when the project has that and no `.mcp.json`, since it reads only the first of the two), `.codex/config.toml`, `.gemini/settings.json`, `.cursor/mcp.json`, `opencode.json`, `.zcode/config.json`, `.pi/mcp.json`, `.qoder/settings.json`'s `mcpServers` for Qoder and Qoder CN, and `.factory/mcp.json` for Droid. Claude Code, Qoder and CodeBuddy ask the user to approve a project's server before using it. Only targets whose vendor source or docs name them are listed (each map's comment cites them); an agent with none (Copilot CLI's MCP, Zed, WorkBuddy, Command Code's MCP, whose `.mcp.json` has its own `transport` shape) can't be picked for a project (eirc on Discord). `Placed`, `Wrote`, `Made` and `MadeFiles` record what magpie put there, the only things it takes away, and what it made is listed in the project's `.gitignore`. | [`projects.go`](../../internal/library/projects.go), [`project_mcp.go`](../../internal/library/project_mcp.go) || Instructions | magpie's part of an agent's instructions file sits between two marker lines. The rest of the file is the user's. Several named sets can be switched between (#106). A set can be kept in a file of the user's own instead of `instruction-sets/` (`InstrSet.File`, 新v学员 on Discord): `~/.agents/AGENTS.md`, which Droid and omp read too. `textPath` reads it, `putText` keeps it aside and writes it but never removes it, removing the set lets go of it, and `linkable` refuses a file magpie writes into (an agent's own instructions, a file with the marker line), the library's own and another set's. The app's `KeepLinked` looks at the file every 30 s and writes a change into the agents (`SyncLinked`). A set's text is read without magpie's part (`setText`), and an agent whose instructions file is, links followed, the file the set on is kept in (`isSetFile`: `instructions.md` linked to `~/.claude/CLAUDE.md`, or both linked to one dotfile) gets no part written into it, and one written before is taken out with every part nested in it: read with its part, each sync wrote the whole file into that part again and it doubled, to gigabytes (#2349, `TestInstructionsSetLinkedToAgentsFile`). `Collect` carries the set's text, not where its file is; `Put` keeps this computer's file for the set and never empties it for a text not carried. Alma's file is `~/.config/alma/USER.md`, which its main chat reads into its system prompt as its owner's profile (0.4.164's `out/main/index.js`; a workspace's `USER.md` is read instead), and whose leading front matter (`name`, `telegram_id`, `discord_id`) is who its bots answer to (#2315, Jie211). The target is a `Profile`: magpie's part goes after the user's, so the front matter stays first, and `ImportInstructions` refuses it, as moving it out would take that front matter out of the file and into every agent's (`TestAlmaInstructionsKeepTheProfile`, `TestAlmaProfileIsNotImported`, `library-alma-profile.test.cjs`). | [`instructions.go`](../../internal/library/instructions.go) || MCP servers | A `Server` is a command or a URL (`http`, `sse`). `mcpFile` encodes it in each agent's own format (one `mcpFormat` per agent family) and merges it with the user's own fields for that entry. A server's `StartupTimeout` (#3168) goes to Codex alone, as `startup_timeout_sec` in seconds (`RawMcpServerConfig` in codex-rs/config's mcp_types.rs; 0 leaves Codex's own 30s), in the user-wide config.toml and a project's alike. magpie keeps the value it wrote (`Applied.Startup`, `Project.Startup`): one in the entry other than that, or a `startup_timeout_ms`, is the user's and stays as it is (`startupTheirs`), and once it is taken out magpie's goes in again. The editor shows the field only when an agent that takes it is here (`AgentView.Startup`, `mcp_startup_test.go`, `mcp-startup-timeout.test.cjs`). `Also` files get the same servers; `Extra` files are only read when importing. `CheckServers` connects as an agent would and keeps the result in memory only: with each tool's name and the first paragraph of its description (`ToolList`, at most `maxToolList`, a description cut at `maxToolDesc` runes), which the page lists when the server's "N tools" is clicked, following tools/list's `nextCursor` up to `maxToolPages` pages (`listTools`). Each step (initialize, notifications/initialized, tools/list; connect for SSE) has its own budget, 60s for a URL and 30s for a command (`remoteStep`, `stdioStep`), and a timeout names its `Step` and how long it `Waited`; a timeout isn't kept, so the next look asks again (#1467). A command named bare that magpie's PATH lacks is started from where `proc.FindTool` finds it (fnm's Nodes on Windows among the folders), with that folder on the check's PATH, since npx's package runs the node beside it by name (`stdioProgram`, #2685, `TestStdioProgramFindsWhatPathLacks`). The page asks once a session for the servers some agent or project has (`inUse` in plugins/library/client.js); one no agent has shows "not checked" and is checked only when its status is clicked (#1597, `library-mcp-health.test.cjs`). A server checked once is checked again when it changes, its sign-in included, so a server signed in to after a click's "needs sign-in" isn't left showing both (新v学员 on Discord, `mcp-signin-health.test.cjs`). The check sends magpie's sign-in as the gateway does: on a 401 it renews the token once and asks again, and a server that refuses even the renewed token has the sign-in marked run out (`mcpauth.Rejected`, also from the gateway's `/mcp/<name>`), which `ServerHealth.SignIn` carries back so the row's tag says "Sign-in ran out" rather than "Signed in" beside "needs sign-in". A server that refuses the check (401 or 403) but that an agent signed in to on its own is named so (`ServerHealth.AgentSignIn`, `readOwnSignIns` in `mcp_agent_signin.go`): each Claude Code target's `.credentials.json` beside its settings.json, a WSL distro's under the distro's home, is read for a `mcpOAuth` entry of the server's name and URL with a token, and the row says "signed in in Claude Code (Ubuntu)" instead of "needs sign-in" (CyberCaius on Discord: signed in to with Claude Code's /mcp in WSL). The tokens are never sent or kept; a file that can't be read or parsed names nobody, a stopped distro isn't started for it, and on a Mac only the file is read, never the Keychain Claude Code keeps them in. They are part of the check's key, so a sign-in in the agent since is checked anew (`TestCheckNamesAgentsSignedInThemselves`, `library-mcp-health-wsl.test.cjs`). A server that needs OAuth is signed in once in magpie, and agents reach it through the gateway's `/mcp/<name>` (#615). The editor's Sign in saves the form first only when the sign-in is to another server than the saved one (a new server, a name or URL changed), and its button then reads "Save and sign in"; anything else edited (headers, env, arguments) stays unsaved until Save, so Cancel leaves the server as it was (新v学员 on Discord, `mcp-editor-cancel.test.cjs`). A server whose authorization server refuses magpie's dynamic client registration with 401 or 403 admits only the clients it lists (Figma's, #1619): the sign-in fails with `mcpauth.ErrClientRefused`, which says so and that each agent signs in itself, and the agents keep the server's own address. A server whose authorization server has no registration endpoint at all (GitHub's remote server, 新v学员 on Discord) fails with `mcpauth.ErrNoRegistration`, which says to make an OAuth app with the vendor and give it in the editor; the editor then opens its "OAuth app" fields. The app is the server's `oauth` (`OAuthApp`: `clientId`, `clientSecret`, `callbackPort`, Claude Code's names for `--client-id`, `--client-secret` and `--callback-port`), checked on save (HTTP only, a client ID, a port from 1 to 65535), and `SignInServer` signs in as it through `mcpauth.StartWith` without registering. The callback is Claude Code's, `http://localhost:<port>/callback` (its docs' "Use pre-configured OAuth credentials"; 2.1.296 builds it so), listened for on 127.0.0.1 alone as Claude Code does, so an app made for Claude Code signs magpie in as it is; a server that matches the registered URL exactly refuses another host, which is why Claude Code's 2.1.231 went back to `localhost` from 2.1.229's `127.0.0.1` (`TestAppMadeForClaudeCode`). Without a `callbackPort` the port is any free one, which a loopback callback registered without a port takes (GitHub's, RFC 8252); a fixed `callbackPort` is for a vendor that matches the port too. magpie's own registration, when a server has one, still names `127.0.0.1`. The client secret is kept in library.json, as a header's or env's value is: a backup without keys leaves it out, and `WithSecrets` puts back this computer's secret only for the same client ID; no agent is given the app, and the TUI's editor, which doesn't show it, keeps it. GitHub answers a token request's error with a 200 and `error` in the body (`bad_refresh_token`, `incorrect_client_credentials`), which `tokenRequest` reads as the error it is, and a dead refresh token marks the sign-in run out (`github_app_test.go`, `mcp-oauth-app.test.cjs`). magpie never registers under another client's name; one that reaches the gateway from another machine (a WSL distro under NAT) is given it with the LAN sharing key as its `Authorization` (`agent.KeyAt`), as its models are, which the gateway takes in place of a sign-in and never passes on; turning sharing on or off, or a new key (`access.LANChanged`), writes such entries again. `ConfigOfServers` is the servers picked as an agent's project file (`projectMCPFiles`) would hold them, written by the same code into a folder of its own and read back, with those the agent can't take there named; the page's Copy config… puts it on the clipboard for a project magpie doesn't keep (#1478). Each listed tool has a switch: the tools turned off are the server's `DisabledTools`, set by `ServerTools`, and written into each agent given the server that can leave out a single tool (see the constraint below). | [`mcp.go`](../../internal/library/mcp.go), [`mcp_check.go`](../../internal/library/mcp_check.go), [`mcp_signin.go`](../../internal/library/mcp_signin.go), [`mcp_agent_signin.go`](../../internal/library/mcp_agent_signin.go), [`mcp_copy.go`](../../internal/library/mcp_copy.go), [`mcp_tools_off.go`](../../internal/library/mcp_tools_off.go) |-| Skills | A skill is a folder with a `SKILL.md`. It comes from GitHub, from any other git repository, from a folder on this machine (linked, so editing the folder edits the skill) or from the market. A git repository (`kind: git`, sunset on Discord: GitLab, a company's own GitLab) is any https, http, ssh or `file://` address, `git@host:group/repo.git`, a bare `host.name/group/repo`, or a GitLab or Gitea page of a folder (`/-/tree/<ref>/<path>`, `/-/blob/…`, `/src/branch/…`; a branch with a `/` in its name is read as its first segment) (`gitSource`). It is fetched by the user's own `git` (`proc.FindTool`), so their credential helpers and SSH keys reach an intranet host: `git init`, then `fetch --depth 1 --no-tags -- <url> <ref>` and `checkout FETCH_HEAD` into a `magpie-skill-*` folder, then its `.git` is removed (`fetchGit`). What the user typed is data: an address starting with `-` and a `transport::` address (`ext::`, `fd::`, `file::`) are refused before git runs, the address follows `--`, git runs with `protocol.ext.allow=never` and `core.symlinks=false`, no shell, `GIT_TERMINAL_PROMPT=0` and (unless the user set an SSH command) `ssh -o BatchMode=yes`, so a sign-in it can't make fails instead of waiting. An address with a password or token in it is refused: library.json is shown and backed up. Git missing, a sign-in refused, a host key unknown, a branch missing and a host unreachable each say so (`gitError`; `TestGitSourceReads`, `TestInstallFromGitRepo`, `TestGitTreeURLInstallsFolder`, `TestGitMissingSaysSo`, `TestGitSignInRefusedSaysSo`, `TestGitRefusesOptionsAndTransports`, `TestGitErrorsSaid`). Each agent gets a link or a marked copy (#896, `SkillHow`). An agent in a WSL distro always gets a copy. A copy's mark keeps the hash of what was copied, so a sync knows a copy unchanged since then is current without reading its files (`copyIsFresh`, #1031); other copies are hashed. A skill an agent has of its own (`foundSkills`, the page's "In your agents") is brought in with `ImportSkill`, or taken out of every agent that has it with `RemoveFoundSkill` (#1303): its folders and copies of the very same files, its shared `~/.agents/skills` entry and a folder of it in the library's own folder go to the backups, its links are taken away, and a folder elsewhere it only links to, or another skill by that name, stays. Several go at once with `RemoveFoundSkills` (`POST /api/library/skills/remove-found-some`, wongxy on Discord): the "In your agents" heading's own Select picks them, and Remove… asks once, naming the count, before one change takes each out as `RemoveFoundSkill` would. A name that can't be taken out is left as it was, in `Result.Unremoved`, and stays picked with its error in its row's place (`foundErrs`); when none can go the change fails and nothing is touched. This Select and the library's are never on together: while one is on the other is dimmed, saying why, so neither click takes the other's bar away from above the pointer (`TestRemoveFoundSkills`, `found-skill-pick-remove.test.cjs`). A repository is fetched as one tarball of at most `maxFetch` (200 MB); a bigger one is fetched file by file from its tree (`fetchSkillFiles`), and a skill over 200 MB on its own is left out rather than failing the rest: the probe lists the others and names it in `Probe.Left`, under the list in the page and in the TUI's flash. Only when every skill is too big does the fetch fail (#2062, mikubaka88/CCFA-Skills' 297 MB ccf-paper-writer; `TestOneSkillTooBigLeavesTheOthers`, `skill-too-big.test.cjs`). A probe keeps its download for the install from it for 15 minutes (`keepProbe`, `probeLife`), and removes it when it is probed again or that time is up, not at the next probe, which may never come. A download a magpie that quit left in the temp dir goes at the next probe once it is an hour old (`sweepStaleDownloads`). The package's TestMain gives the tests a temp dir of their own and fails a run that leaves a `magpie-skill-*` in it (#2321, `TestProbeDownloadEndsWithIt`, `TestStaleDownloadsSwept`). | [`skilltree.go`](../../internal/library/skilltree.go), [`skills.go`](../../internal/library/skills.go), [`skillgit.go`](../../internal/library/skillgit.go), [`skill_how.go`](../../internal/library/skill_how.go), [`skill_edits.go`](../../internal/library/skill_edits.go), [`skill_from.go`](../../internal/library/skill_from.go) |+| Skills | A skill is a folder with a `SKILL.md`. It comes from GitHub, from any other git repository, from a folder on this machine (linked, so editing the folder edits the skill) or from the market. A git repository (`kind: git`, sunset on Discord: GitLab, a company's own GitLab) is any https, http, ssh or `file://` address, `git@host:group/repo.git`, a bare `host.name/group/repo`, or a GitLab or Gitea page of a folder (`/-/tree/<ref>/<path>`, `/-/blob/…`, `/src/branch/…`; a branch with a `/` in its name is read as its first segment) (`gitSource`). It is fetched by the user's own `git` (`proc.FindTool`), so their credential helpers and SSH keys reach an intranet host: `git init`, then `fetch --depth 1 --no-tags -- <url> <ref>` and `checkout FETCH_HEAD` into a `magpie-skill-*` folder, then its `.git` is removed (`fetchGit`). What the user typed is data: an address starting with `-` and a `transport::` address (`ext::`, `fd::`, `file::`) are refused before git runs, the address follows `--`, git runs with `protocol.ext.allow=never` and `core.symlinks=false`, no shell, `GIT_TERMINAL_PROMPT=0` and (unless the user set an SSH command) `ssh -o BatchMode=yes`, so a sign-in it can't make fails instead of waiting. An address with a password or token in it is refused: library.json is shown and backed up. Git missing, a sign-in refused, a host key unknown, a branch missing and a host unreachable each say so (`gitError`; `TestGitSourceReads`, `TestInstallFromGitRepo`, `TestGitTreeURLInstallsFolder`, `TestGitMissingSaysSo`, `TestGitSignInRefusedSaysSo`, `TestGitRefusesOptionsAndTransports`, `TestGitErrorsSaid`). Each agent gets a link or a marked copy (#896, `SkillHow`). An agent in a WSL distro always gets a copy. A copy's mark keeps the hash of what was copied, so a sync knows a copy unchanged since then is current without reading its files (`copyIsFresh`, #1031); other copies are hashed. A skill an agent has of its own (`foundSkills`, the page's "In your agents") is brought in with `ImportSkill`, or taken out of every agent that has it with `RemoveFoundSkill` (#1303): its folders and copies of the very same files, its shared `~/.agents/skills` entry and a folder of it in the library's own folder go to the backups, its links are taken away, and a folder elsewhere it only links to, or another skill by that name, stays. Several go at once with `RemoveFoundSkills` (`POST /api/library/skills/remove-found-some`, wongxy on Discord): the "In your agents" heading's own Select picks them, and Remove… asks once, naming the count, before one change takes each out as `RemoveFoundSkill` would. A name that can't be taken out is left as it was, in `Result.Unremoved`, and stays picked with its error in its row's place (`foundErrs`); when none can go the change fails and nothing is touched. This Select and the library's are never on together: while one is on the other is dimmed, saying why, so neither click takes the other's bar away from above the pointer (`TestRemoveFoundSkills`, `found-skill-pick-remove.test.cjs`). A repository is fetched as one tarball of at most `maxFetch` (200 MB); a bigger one is fetched file by file from its tree (`fetchSkillFiles`), only as far as the caller reads: a link to a folder and an update ask GitHub for that folder's own list (`treeUnder`, git's `ref:path`; the whole list when that fails), the market fetches only the skills whose folder is the one it installs (#2794, sickn33/agentic-awesome-skills' 8,589 skills are 287 MB together), and a fetcher keeps such a part apart from another skill's of the same repository, which would otherwise read as removed (`fetcher.get`, `TestMarketSkillFromAHugeRepository`). GitHub's list is read up to `maxTree` (64 MB; that repository's is 11 MB) and an answer over the cap a caller gives `apiGet`, or one cut off, is an error saying so, never a body cut short for JSON to trip on (ye4241's `TestRepoTreeBigList`, `TestApiGetWholeAnswer`); a list GitHub truncated itself is too big (`TestTruncatedListIsTooBig`). And a skill over 200 MB on its own is left out rather than failing the rest: the probe lists the others and names it in `Probe.Left`, under the list in the page and in the TUI's flash. Only when every skill is too big does the fetch fail (#2062, mikubaka88/CCFA-Skills' 297 MB ccf-paper-writer; `TestOneSkillTooBigLeavesTheOthers`, `skill-too-big.test.cjs`). A probe keeps its download for the install from it for 15 minutes (`keepProbe`, `probeLife`), and removes it when it is probed again or that time is up, not at the next probe, which may never come. A download a magpie that quit left in the temp dir goes at the next probe once it is an hour old (`sweepStaleDownloads`). The package's TestMain gives the tests a temp dir of their own and fails a run that leaves a `magpie-skill-*` in it (#2321, `TestProbeDownloadEndsWithIt`, `TestStaleDownloadsSwept`). | [`skilltree.go`](../../internal/library/skilltree.go), [`skills.go`](../../internal/library/skills.go), [`skillgit.go`](../../internal/library/skillgit.go), [`skill_how.go`](../../internal/library/skill_how.go), [`skill_edits.go`](../../internal/library/skill_edits.go), [`skill_from.go`](../../internal/library/skill_from.go) || Skills market | `MarketSkills` lists skills.sh's most installed and what its search finds. magpie's own skills (`featuredSkills`), in [magpie-community/plugins](https://github.com/magpie-community/plugins/tree/main/skills), go first, marked official: `magpie-setup` has an agent set magpie up as the user asks through its commands (providers, accounts, agents, routing groups and rules, gateway keys and limits, models, the library), and `magpie-quota` lets an agent read `magpie quota --json` and wait with `magpie quota wait`. Their commands live in magpie's CLI, so a change to a command either uses gets the skill updated in that repository. Until skills.sh lists one it has no count of installs and no skills.sh link (`featured`). `InstallMarketSkill` installs any of them from its repository. | [`market.go`](../../internal/library/market.go) || Skills a repository added | A check for updates (`CheckSkills`) also asks, for each GitHub repository and ref the library has skills from, GitHub's list of its files (`repoTree`, one API request, free when the ETag still matches). Only when that list has a skill folder beside the installed ones (under `newBase`, their deepest common folder) that the library doesn't have and hasn't offered does `findNew` fetch the repository for the skills' names and descriptions. `SeenSkills` records per `repo@ref` the paths the install picker showed and the ones set aside, so they aren't offered again. The finds live in memory (`newFound`) and reach the page as `View.NewSkills`; `AddNewSkills` gives each to the agents that have its repository's other skills, and `IgnoreNewSkills` sets it aside (`/api/library/skills/{add-new,ignore-new}`). | [`skillnew.go`](../../internal/library/skillnew.go), [`skillcheck.go`](../../internal/library/skillcheck.go) || Skill groups | The page groups skills by `SkillView.Repo`, the repository each came from (`tracer.repoOf`): its GitHub source, CC Switch's `Origin`, `From`, the skills CLI's lock or the git checkout a linked folder is in. A lock entry of `sourceType: well-known` (a site's kit at its `/.well-known/agent-skills/`) is grouped by its `sourceBaseUrl` as host/path (`wellKnownBase`), not by each skill's own `sourceUrl`: Feishu's lark-* skills are one group and Fish Audio's another, while two kits under one host with different base URLs stay apart. An older entry with no base URL takes the part of its `sourceUrl` before `/.well-known/`, and one with neither is grouped by its `sourceUrl` as before. Such a kit is never fetched from GitHub by Group again, even when it reads like owner/repo (a GitHub owner has no dot), and its heading's link opens the site (#1862, `TestWellKnownKitGrouped`). `From` is written when a repository is probed (installing from it again, `noteFrom`): for a skill with the very same `SKILL.md`, and for one traced to nowhere by a name the repository has when the library has others from there, since a pack added a while ago is mostly older copies edited there since (White Immortal on Discord: 20 of mattpocock/skills' 38). Group again (`RegroupSkills`, `POST /api/library/skills/regroup`) fetches each GitHub repository the skills are traced to and does the same; it fails only when none could be fetched. A skill traced to another repository is never moved. A group's (or part's, or the Select bar's) agent chips count, give and take only the skills magpie decides for each agent: one kept in `~/.agents/skills` is in `SkillView.Always` for the agents that read that folder themselves, so a chip's `agents-some` for such an agent leaves it out, its tooltip and the status after a click say how many it still reads itself, and an agent that reads every one of them itself has a lit, locked chip as the row's is, which sends nothing (`groupChips` in plugins/library/client.js). The heading says how many of its skills are in `~/.agents/skills`. A row's locked chip is lit with a lock on its corner, not greyed like a chip that can't be given the skill (#1862, `skill-group-shared.test.cjs`). Collapse all / Expand all, beside the sort, folds or opens every group the list shows at once (`foldAllButton`): without a filter it sets each group's kept fold (`magpie.libSkillFolds`), as a click on each heading would; while filtering, only the groups the filter shows, till the filter changes (`unfiltered`), leaving the kept folds alone. It reads Collapse all while any shown group is open, is hidden with fewer than two groups (A→Z, or a filter one group matches), leaves a part's own fold as it is and sends nothing (#1862, `skill-fold-all.test.cjs`). | [`skill_from.go`](../../internal/library/skill_from.go), [`plugins/library/client.js`](../../internal/gui/assets/plugins/library/client.js) |diff --git a/internal/library/market.go b/internal/library/market.goindex 7e966e6cd..e7309d6e5 100644--- a/internal/library/market.go+++ b/internal/library/market.go@@ -1043,7 +1043,9 @@ func InstallMarketSkill(source, id string, agents []string) (*Result, error) {if !repoRe.MatchString(source) {return nil, fmt.Errorf("%s isn't a GitHub repository", source)}- p, err := cachedProbe(source)+ // only this skill's folder, when the repository is too big to+ // download whole (#2794)+ p, err := cachedProbeFor(source, id)if err != nil {return nil, err}diff --git a/internal/library/skillcheck.go b/internal/library/skillcheck.goindex c905f17cb..69b267539 100644--- a/internal/library/skillcheck.go+++ b/internal/library/skillcheck.go@@ -129,7 +129,7 @@ func lastCommit(src Source) (*commit, error) {q.Set("sha", src.Ref)}u := githubAPI + "/repos/" + src.Repo + "/commits?" + q.Encode()- body, code, token, err := apiGet(u)+ body, code, token, err := apiGet(u, 8<<20)if err != nil {return nil, err}@@ -154,12 +154,22 @@ func lastCommit(src Source) (*commit, error) {return &list[0], nil}+// cacheBody is the most of an answer kept by URL for a 304 to reuse;+// bigger ones are read but not kept, so a huge file list isn't held for+// the process's life.+const cacheBody = 8 << 20++// errOverCap is an answer over the most its caller reads (apiGet's max).+var errOverCap = errors.New("over the most magpie reads")+// apiGet asks GitHub's API for u, with the ETag of the answer it had// before, if any: GitHub's 304 for an unchanged one doesn't count against// the rate limit, and is answered as the 200 it had. It says whether the// request carried a GitHub token, and a rate limit used up is an-// errLimited.-func apiGet(u string) ([]byte, int, bool, error) {+// errLimited. A rate limit is told from the status and headers even when+// the answer can't be read whole. A 200 over max bytes is an error, never+// a cut-off body handed to a caller, and no answer over cacheBody is kept.+func apiGet(u string, max int64) ([]byte, int, bool, error) {req, _ := http.NewRequest("GET", u, nil)req.Header.Set("User-Agent", "magpie")req.Header.Set("Accept", "application/vnd.github+json")@@ -169,20 +179,31 @@ func apiGet(u string) ([]byte, int, bool, error) {if cached {req.Header.Set("If-None-Match", had.etag)}- c := &http.Client{Timeout: 20 * time.Second}+ // a file list of tens of MB takes longer than 20s on a slow line+ timeout := 20 * time.Second+ if max > cacheBody {+ timeout = 2 * time.Minute+ }+ c := &http.Client{Timeout: timeout}resp, token, err := doGitHub(req, func(r *http.Request) (*http.Response, error) { return source.Do(c, r) })if err != nil {return nil, 0, token, fmt.Errorf("couldn't reach GitHub: %w", err)}defer resp.Body.Close()- body, _ := io.ReadAll(io.LimitReader(resp.Body, 8<<20))+ body, err := io.ReadAll(io.LimitReader(resp.Body, max+1))if e, ok := limitedBy(resp, body, token); ok {return nil, resp.StatusCode, token, e}+ if err != nil {+ return nil, resp.StatusCode, token, fmt.Errorf("couldn't read GitHub's answer: %w", err)+ }if resp.StatusCode == 304 && cached {return had.body, 200, token, nil}- if et := resp.Header.Get("ETag"); resp.StatusCode == 200 && et != "" {+ if resp.StatusCode == 200 && int64(len(body)) > max {+ return nil, resp.StatusCode, token, fmt.Errorf("GitHub's answer was over %s: %w", mb(max), errOverCap)+ }+ if et := resp.Header.Get("ETag"); resp.StatusCode == 200 && et != "" && int64(len(body)) <= cacheBody {etags.Lock()etags.m[u] = etagged{etag: et, body: body}etags.Unlock()diff --git a/internal/library/skillcheck_test.go b/internal/library/skillcheck_test.goindex 2aa05a8db..f04df61fc 100644--- a/internal/library/skillcheck_test.go+++ b/internal/library/skillcheck_test.go@@ -1,6 +1,7 @@package libraryimport (+ "errors""fmt""net/http""net/http/httptest"@@ -182,3 +183,70 @@ func check(t *testing.T) map[string]SkillCheck {}return m}++// An answer over the cap is an error that says so, never a cut body for a+// JSON parse to trip on; a cut-off answer is a read error; a rate limit is+// told from the status and headers even then; and an answer the cap lets+// through but cacheBody doesn't isn't kept for a 304 to answer with.+func TestApiGetWholeAnswer(t *testing.T) {+ var mu sync.Mutex+ var ifNone []string+ srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {+ mu.Lock()+ ifNone = append(ifNone, r.Header.Get("If-None-Match"))+ mu.Unlock()+ w.Header().Set("ETag", `"v1"`)+ fmt.Fprint(w, strings.Repeat(`{"path":"a","type":"blob"},`, 1<<20)) // ~27 MB+ }))+ defer srv.Close()++ if _, _, _, err := apiGet(srv.URL, 8<<20); err == nil || !strings.Contains(err.Error(), "over 8 MB") {+ t.Fatalf("over the cap: %v", err)+ }+ for range 2 {+ if _, _, _, err := apiGet(srv.URL, 64<<20); err != nil {+ t.Fatalf("whole answer under the cap: %v", err)+ }+ }+ mu.Lock()+ defer mu.Unlock()+ if len(ifNone) != 3 || slices.Contains(ifNone, `"v1"`) {+ t.Errorf("an over-cacheBody answer was kept for a 304: %q", ifNone)+ }++ // a connection that drops mid-body is a read error, not a part body+ drop := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {+ w.Header().Set("Content-Length", "100")+ fmt.Fprint(w, `{"sha":`)+ }))+ defer drop.Close()+ if _, _, _, err := apiGet(drop.URL, 8<<20); err == nil || !strings.Contains(err.Error(), "couldn't read GitHub's answer") {+ t.Fatalf("dropped connection: %v", err)+ }++ // so is a cut-off rate limit, but it's told from the status and+ // headers: CheckSkills stops asking once GitHub limits it+ limited := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {+ w.Header().Set("X-RateLimit-Remaining", "0")+ w.Header().Set("X-RateLimit-Reset", "2000000000")+ w.Header().Set("Content-Length", "100")+ w.WriteHeader(403)+ fmt.Fprint(w, `{"message":`)+ }))+ defer limited.Close()+ _, _, _, err := apiGet(limited.URL, 8<<20)+ var el errLimited+ if !errors.As(err, &el) {+ t.Fatalf("cut-off rate limit: %v", err)+ }++ // a non-200's size never masks the status the caller reports+ bad := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {+ w.WriteHeader(502)+ fmt.Fprint(w, strings.Repeat("bad gateway ", 1<<20)) // ~12 MB+ }))+ defer bad.Close()+ if _, code, _, err := apiGet(bad.URL, 8<<20); err != nil || code != 502 {+ t.Fatalf("502 over the cap: code %d, err %v", code, err)+ }+}diff --git a/internal/library/skillnew.go b/internal/library/skillnew.goindex 1a9c17097..cbad1451d 100644--- a/internal/library/skillnew.go+++ b/internal/library/skillnew.go@@ -98,6 +98,11 @@ func newBase(skills []*Skill) string {return base}+// maxTree is the most of a repository's file list magpie reads; a list can+// be tens of MB for a big app (stablyai/orca's is ~10 MB, and GitHub+// answers one whole under its own 100,000-entry cap).+const maxTree = 64 << 20+// gitTree is a repository's files as GitHub's API lists them.type gitTree struct {Truncated bool `json:"truncated"`@@ -169,10 +174,24 @@ func hiddenThroughSkills(parts []string) bool {// repoTree is a repository's files at a ref, from GitHub's API.func repoTree(repo, ref string) (*gitTree, error) {+ return listTree(repo, ref, "")+}++// listTree is GitHub's list of the files at a ref under path (the whole+// repository's for ""), with paths from that folder. A list over maxTree,+// or one cut off, is an error that says so, never a list read in part.+func listTree(repo, ref, path string) (*gitTree, error) {if ref == "" {ref = "HEAD"}- body, code, _, err := apiGet(githubAPI + "/repos/" + repo + "/git/trees/" + url.PathEscape(ref) + "?recursive=1")+ ish := ref+ if path != "" {+ ish += ":" + path // git's ref:path, a folder's own tree+ }+ body, code, _, err := apiGet(githubAPI+"/repos/"+repo+"/git/trees/"+url.PathEscape(ish)+"?recursive=1", maxTree)+ if errors.Is(err, errOverCap) {+ return nil, fmt.Errorf("GitHub's list of %s's files is over %s", repo, mb(maxTree))+ }if err != nil {return nil, err}@@ -186,6 +205,31 @@ func repoTree(repo, ref string) (*gitTree, error) {return &t, nil}+// treeUnder is a repository's files at a ref under path, with their paths+// from its root. With a path it asks for that folder's list alone: one+// skill's folder is a few KB where the whole repository's list is over+// 11 MB for sickn33/agentic-awesome-skills (#2794). When that can't be+// had (no such folder, a ref git's ref:path doesn't read, a list cut+// off) the whole repository's is read, which says for sure what is there.+// A rate limit is said at once.+func treeUnder(repo, ref, path string) (*gitTree, error) {+ if path != "" {+ t, err := listTree(repo, ref, path)+ var e errLimited+ if errors.As(err, &e) {+ return nil, err+ }+ if err == nil && !t.Truncated {+ for i := range t.Tree {+ t.Tree[i].Path = path + "/" + t.Tree[i].Path+ }+ t.Tree = append([]gitEntry{{Path: path, Type: "tree", Mode: "040000"}}, t.Tree...)+ return t, nil+ }+ }+ return repoTree(repo, ref)+}+// findNew looks in one repository at one ref for skills beside the// library's from there that it doesn't have and hasn't offered: GitHub's// list of the repository's files first, and only when that has one, the@@ -210,7 +254,7 @@ func findNew(l *Library, repo, ref string, f *fetcher, limited *atomic.Pointer[eknown[s.Source.Path] = true}}- tree, err := repoTree(repo, ref)+ tree, err := treeUnder(repo, ref, base)if err != nil {var e errLimitedif errors.As(err, &e) {@@ -221,7 +265,9 @@ func findNew(l *Library, repo, ref string, f *fetcher, limited *atomic.Pointer[eif !tree.Truncated && !slices.ContainsFunc(skillDirsIn(tree, base), func(d string) bool { return !known[d] }) {return []NewSkill{}, true}- src := Source{Kind: "github", Repo: repo, Ref: ref}+ // only base's folder: candidates looks nowhere else, and a+ // repository too big to download whole is then fetched only there+ src := Source{Kind: "github", Repo: repo, Ref: ref, Path: base}tmp, err := f.fetch(src)if err != nil {return nil, falsediff --git a/internal/library/skillnew_test.go b/internal/library/skillnew_test.goindex d95e5fce7..023414173 100644--- a/internal/library/skillnew_test.go+++ b/internal/library/skillnew_test.go@@ -4,6 +4,7 @@ import ("crypto/sha256""encoding/hex""encoding/json"+ "fmt""net/http""net/http/httptest""os"@@ -31,10 +32,21 @@ func fakeSkillRepo(t *testing.T, files map[string]string) (asked map[string]int,if r.URL.Query().Get("recursive") != "1" {t.Errorf("tree asked without recursive: %s", r.URL)}+ // git's ref:path is that folder's own list, as GitHub+ // answers it; no such folder is a 404+ _, under, _ := strings.Cut(r.URL.Path[strings.Index(r.URL.Path, "/git/trees/")+len("/git/trees/"):], ":")var tree gitTreepaths := []string{}for p := range files {- paths = append(paths, p)+ if under == "" {+ paths = append(paths, p)+ } else if rest, in := strings.CutPrefix(p, under+"/"); in {+ paths = append(paths, rest)+ }+ }+ if len(paths) == 0 {+ http.NotFound(w, r)+ return}sort.Strings(paths)for _, p := range paths {@@ -365,3 +377,43 @@ func TestRepoWithAPluginCopyListsEachSkillOnce(t *testing.T) {t.Errorf("skills in the tree %v, want %v", got, want)}}++// stablyai/orca's file list answers 200 with ~10 MB of JSON, over the 8 MB+// apiGet once read; the cut-off body then failed to parse as "unexpected+// end of JSON input". A list that size is read and understood.+func TestRepoTreeBigList(t *testing.T) {+ // ~10 MB of entries, the shape GitHub answers for orca+ tree := &gitTree{}+ for i := 0; i < 60_000; i++ {+ tree.Tree = append(tree.Tree, gitEntry{+ Path: fmt.Sprintf("apps/desktop/src/renderer/components/panels/sidebar/file%05d.ts", i),+ Type: "blob", Mode: "100644", SHA: "e488653ba64c6cfc3551ec46454b965421d50176", Size: 321,+ })+ }+ tree.Tree = append(tree.Tree, gitEntry{Path: "skills/orca/SKILL.md", Type: "blob", Mode: "100644", SHA: "a438021d7b90192e2c0c4802e6cff1b0df8bb203"})+ b, err := json.Marshal(tree)+ if err != nil {+ t.Fatal(err)+ }+ if len(b) <= 8<<20 {+ t.Fatalf("fixture is %d bytes, not over the old cap", len(b))+ }+ srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {+ w.Write(b)+ }))+ defer srv.Close()+ oldA := githubAPI+ githubAPI = srv.URL+ defer func() { githubAPI = oldA }()++ got, err := repoTree("stablyai/orca", "")+ if err != nil {+ t.Fatal(err)+ }+ if len(got.Tree) != len(tree.Tree) {+ t.Fatalf("read %d of %d entries", len(got.Tree), len(tree.Tree))+ }+ if dirs := skillDirsIn(got, "skills"); !slices.Equal(dirs, []string{"skills/orca"}) {+ t.Errorf("skills in it: %v", dirs)+ }+}diff --git a/internal/library/skills.go b/internal/library/skills.goindex e98b277d1..3c8345e08 100644--- a/internal/library/skills.go+++ b/internal/library/skills.go@@ -765,10 +765,24 @@ func (s *Source) remote() bool { return s != nil && (s.Kind == "github" || s.Kin// asked of GitHub's API with the user's GitHub token (37FlowAI on X): the// API redirects to codeload, and the token, sent to the API's host alone,// isn't carried over to it.+//+// A repository too big to download whole is fetched only as far as src.Path+// and name narrow it (fetchSkillFiles); part says it was.func fetch(src Source) (string, error) {+ dir, _, err := fetchPart(src, "")+ return dir, err+}++func fetchPart(src Source, name string) (dir string, part bool, err error) {if src.Kind == "git" {- return fetchGit(src)+ dir, err = fetchGit(src)+ return dir, false, err}+ dir, err = fetchWhole(src, name)+ return dir, tooBig.has(src) && (src.Path != "" || name != ""), err+}++func fetchWhole(src Source, name string) (string, error) {c := &http.Client{Timeout: 2 * time.Minute}get := func(u string) (*http.Response, bool, error) {req, _ := http.NewRequest("GET", u, nil)@@ -776,7 +790,7 @@ func fetch(src Source) (string, error) {return doGitHub(req, func(r *http.Request) (*http.Response, error) { return source.DoOfficial(c, r) })}if tooBig.has(src) {- return fetchSkillFiles(src)+ return fetchSkillFiles(src, name)}resp, _, err := get(tarballURL(src.Repo, src.Ref))if err != nil {@@ -827,7 +841,7 @@ func fetch(src Source) (string, error) {// skills beside it, #1941): only the skills' foldersresp.Body.Close()tooBig.add(src)- return fetchSkillFiles(src)+ return fetchSkillFiles(src, name)}return "", err}@@ -1011,7 +1025,13 @@ func skillsCommand(input string) string {// (or a folder in one), any other git repository (GitLab, one on a// company's own server; fetched with the user's git), a folder on this// machine, or a skills CLI command for one of them.-func ProbeSkills(input string) (*Probe, error) {+func ProbeSkills(input string) (*Probe, error) { return probeSkills(input, "") }++// probeSkills is ProbeSkills for the skill called name, when one is+// wanted: a repository too big to download whole is then fetched only for+// the skills by that name (fetchSkillFiles), and the probe is kept apart+// from a whole one (probeKey).+func probeSkills(input, name string) (*Probe, error) {input = strings.TrimSpace(skillsCommand(input))if input == "" {return nil, fmt.Errorf("paste a GitHub or git repository's link or a folder's path")@@ -1033,7 +1053,7 @@ func ProbeSkills(input string) (*Probe, error) {if !ok {src, _, _ = gitSource(input)}- tmp, err := fetch(src)+ tmp, part, err := fetchPart(src, name)if err != nil {return nil, err}@@ -1044,7 +1064,11 @@ func ProbeSkills(input string) (*Probe, error) {}p.Candidates = candidates(tmp, src.Path)p.Left = tookLeftOut(tmp, src.Path)- keepProbe(input, p, tmp)+ key := input+ if part && name != "" {+ key = probeKey(input, name)+ }+ keepProbe(key, p, tmp)} else if _, ok, err := gitSource(input); ok && err != nil {return nil, err} else {@@ -1062,16 +1086,30 @@ func ProbeSkills(input string) (*Probe, error) {return p, nil}-func cachedProbe(input string) (*Probe, error) {+func cachedProbe(input string) (*Probe, error) { return cachedProbeFor(input, "") }++// probeKey is where a probe fetched only for the skill called name is+// kept: a later probe of the whole source never takes it for one.+func probeKey(input, name string) string { return input + "\x00" + name }++// cachedProbeFor is the probe kept for input, a whole one or one made for+// the skill called name, or a new one for name.+func cachedProbeFor(input, name string) (*Probe, error) {+ key := strings.TrimSpace(skillsCommand(input))probes.Lock()- e := probes.m[strings.TrimSpace(skillsCommand(input))]+ es := []*probeEntry{probes.m[key]}+ if name != "" {+ es = append(es, probes.m[probeKey(key, name)])+ }probes.Unlock()- if e != nil {- if _, err := os.Stat(e.tmp); err == nil {- return e.p, nil+ for _, e := range es {+ if e != nil {+ if _, err := os.Stat(e.tmp); err == nil {+ return e.p, nil+ }}}- return ProbeSkills(input)+ return probeSkills(input, name)}// InstallSkills adds the skills at those paths of the source to the library@@ -1391,7 +1429,10 @@ func updatable(s *Skill) bool {}// fetcher downloads each repository at each ref once, however many of-// its skills are updated.+// its skills are updated. A repository too big to download whole is+// fetched only under the folder asked for (fetchPart), so a skill+// elsewhere in it is fetched on its own: read from the first folder, it+// would look removed there (goneFrom).type fetcher struct {mu sync.Mutexgot map[string]*fetched@@ -1399,17 +1440,28 @@ type fetcher struct {}type fetched struct {- once sync.Once- dir string- err error+ once sync.Once+ dir string+ err error+ part bool // only under+ under string // the folder fetched, when part}func (f *fetcher) fetch(src Source) (string, error) {+ key := src.Repo + "@" + src.Ref+ g := f.get(key, src)+ if g.part && !inRepoDir(src.Path, g.under) {+ g = f.get(key+"@"+src.Path, src)+ }+ return g.dir, g.err+}++// get is the fetch kept under key, made from src the first time.+func (f *fetcher) get(key string, src Source) *fetched {f.mu.Lock()if f.got == nil {f.got = map[string]*fetched{}}- key := src.Repo + "@" + src.Refg := f.got[key]if g == nil {g = &fetched{}@@ -1417,14 +1469,15 @@ func (f *fetcher) fetch(src Source) (string, error) {}f.mu.Unlock()g.once.Do(func() {- g.dir, g.err = fetch(src)+ g.dir, g.part, g.err = fetchPart(src, "")+ g.under = src.Pathif g.err == nil {f.mu.Lock()f.dirs = append(f.dirs, g.dir)f.mu.Unlock()}})- return g.dir, g.err+ return g}func (f *fetcher) clean() {@@ -1457,10 +1510,19 @@ func (f *fetcher) prepare(s *Skill) (*skillUpdate, error) {c := *srcsrc = &c}- tmp, err := f.fetch(*src)+ // CC Switch's Path is only the folder's name (origin): the whole+ // repository is looked in+ at := func() Source {+ w := *src+ if adopt {+ w.Path = ""+ }+ return w+ }+ tmp, err := f.fetch(at())if err != nil && adopt && src.Ref != "" {src.Ref = "" // the branch CC Switch recorded is gone: the default one- tmp, err = f.fetch(*src)+ tmp, err = f.fetch(at())}if err != nil {return nil, errdiff --git a/internal/library/skilltree.go b/internal/library/skilltree.goindex a4da72b65..5b4530087 100644--- a/internal/library/skilltree.go+++ b/internal/library/skilltree.go@@ -72,11 +72,15 @@ var rawURL = func(repo, ref, path string) string {}// fetchSkillFiles fetches, into a new folder laid out as the repository-// is, every folder of it with a SKILL.md, whatever is in it, and the-// repository's other folders empty: what candidates, origin and an update-// read of a fetched repository.-func fetchSkillFiles(src Source) (string, error) {- tree, err := repoTree(src.Repo, src.Ref)+// is, every folder of it with a SKILL.md under src.Path, whatever is in+// it, and the other folders there empty: what candidates, origin and an+// update read of a fetched repository. With name, only the skills whose+// folder is called that are fetched, if any is: the market installs one+// skill by its name, and sickn33/agentic-awesome-skills' 8,589 skills+// come to 287 MB, over maxFetch (#2794). A skill's folder is its name+// (agentskills.io's SKILL.md spec).+func fetchSkillFiles(src Source, name string) (string, error) {+ tree, err := treeUnder(src.Repo, src.Ref, src.Path)if err != nil {return "", err}@@ -84,12 +88,20 @@ func fetchSkillFiles(src Source) (string, error) {// too many files for GitHub to list in one answerreturn "", errTooBig}- var skillDirs []string+ var skillDirs, named []stringfor _, e := range tree.Tree {if e.Type == "blob" && slash.Base(e.Path) == "SKILL.md" {- skillDirs = append(skillDirs, slash.Dir(e.Path))+ if d := slash.Dir(e.Path); inRepoDir(d, src.Path) {+ skillDirs = append(skillDirs, d)+ if name != "" && slash.Base(d) == name {+ named = append(named, d)+ }+ }}}+ if len(named) > 0 {+ skillDirs = named+ }inSkill := func(p string) bool {for _, d := range skillDirs {if d == "." || strings.HasPrefix(p, d+"/") {@@ -134,7 +146,9 @@ func fetchSkillFiles(src Source) (string, error) {left = append(left, LeftOut{Path: p, Size: size[d]})}}- if len(left) == len(skillDirs) && len(left) > 0 {+ // a folder asked for whose skills are all too big is listed empty,+ // naming them, as the whole repository's list would+ if len(left) == len(skillDirs) && len(left) > 0 && src.Path == "" {return "", fmt.Errorf("the skills in %s are too big to install (over %s)", src.Repo, mb(maxFetch))}var files []file@@ -157,7 +171,9 @@ func fetchSkillFiles(src Source) (string, error) {return dst, strings.HasPrefix(dst, tmp+string(filepath.Separator)) // never outside}for _, e := range tree.Tree {- if e.Type == "tree" {+ // only the named skills' folders when narrowed to them: the rest+ // of a big repository is thousands of empty folders+ if e.Type == "tree" && inRepoDir(e.Path, src.Path) && (len(named) == 0 || inSkill(e.Path)) {if dst, ok := place(e.Path); ok {if err := os.MkdirAll(dst, 0o755); err != nil {os.RemoveAll(tmp)@@ -208,6 +224,12 @@ func fetchSkillFiles(src Source) (string, error) {return tmp, nil}+// inRepoDir is whether a repository's path p is under the folder dir, or+// is it; everything is under "". A skill at the repository's root is ".".+func inRepoDir(p, dir string) bool {+ return dir == "" || p == dir || strings.HasPrefix(p, dir+"/")+}+// LeftOut is a skill of a repository too big to install on its own// (fetchSkillFiles): where it is, and what its files come to.type LeftOut struct {diff --git a/internal/library/skilltree_test.go b/internal/library/skilltree_test.goindex 0befa135b..cc82a7d8d 100644--- a/internal/library/skilltree_test.go+++ b/internal/library/skilltree_test.go@@ -4,12 +4,14 @@ import ("crypto/sha1""encoding/hex""encoding/json"+ "errors""fmt""net/http""net/http/httptest""os"slash "path""path/filepath"+ "slices""sort""strings""sync"@@ -27,6 +29,8 @@ type fakeBigRepo struct {tarballs intfiles map[string]stringserve map[string]string // what raw hands out, where it differs+ trees []string // the tree-ish each file list was asked at+ pad int // files of the app's own, in the whole list only}func gitSHA(body string) string {@@ -57,10 +61,18 @@ func newFakeBigRepo(t *testing.T) *fakeBigRepo {w.Write(tarball(t, g.files))}))api := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {- if !strings.HasPrefix(r.URL.Path, "/repos/vectorize-io/hindsight/git/trees/") {+ ish, found := strings.CutPrefix(r.URL.Path, "/repos/vectorize-io/hindsight/git/trees/")+ if !found {http.NotFound(w, r)return}+ g.mu.Lock()+ g.trees = append(g.trees, ish)+ pad := g.pad+ g.mu.Unlock()+ // git's ref:path is that folder's own list, its paths from there,+ // as GitHub answers it; no such folder is a 404+ _, under, _ := strings.Cut(ish, ":")// GitHub's answer, as it lists hindsight: folders, then files with// their mode, git hash and sizevar tree []map[string]any@@ -76,16 +88,32 @@ func newFakeBigRepo(t *testing.T) *fakeBigRepo {paths = append(paths, d)}sort.Strings(paths)+ if under != "" && !dirs[under] {+ http.NotFound(w, r)+ return+ }for _, p := range paths {+ at := p+ if under != "" {+ var in bool+ if at, in = strings.CutPrefix(p, under+"/"); !in {+ continue+ }+ }if dirs[p] {- tree = append(tree, map[string]any{"mode": "040000", "path": p, "sha": "e488653ba64c6cfc3551ec46454b965421d50176", "type": "tree"})+ tree = append(tree, map[string]any{"mode": "040000", "path": at, "sha": "e488653ba64c6cfc3551ec46454b965421d50176", "type": "tree"})continue}mode := "100644"if strings.HasSuffix(p, ".sh") {mode = "100755"}- tree = append(tree, map[string]any{"mode": mode, "path": p, "sha": gitSHA(g.files[p]), "size": len(g.files[p]), "type": "blob"})+ tree = append(tree, map[string]any{"mode": mode, "path": at, "sha": gitSHA(g.files[p]), "size": len(g.files[p]), "type": "blob"})+ }+ for i := 0; under == "" && i < pad; i++ {+ tree = append(tree, map[string]any{"mode": "100644", "path": fmt.Sprintf("hindsight-app/src/renderer/components/panels/sidebar/file%05d.ts", i),+ "sha": "e488653ba64c6cfc3551ec46454b965421d50176", "size": 321, "type": "blob",+ "url": "https://api.github.com/repos/vectorize-io/hindsight/git/blobs/e488653ba64c6cfc3551ec46454b965421d50176"})}json.NewEncoder(w).Encode(map[string]any{"sha": "52ef60fad332184c3b8a1f9fff4756247fc07146", "tree": tree, "truncated": false})}))@@ -241,3 +269,68 @@ func TestOneSkillTooBigLeavesTheOthers(t *testing.T) {t.Fatalf("the big skill's folder: %+v, %v", p, err)}}++// #2794 (CaiusLuo): the market installs clean-code from+// sickn33/agentic-awesome-skills, whose whole download is too big, whose+// file list is 11 MB (over the 8 MB GitHub's answers were cut at, so it+// failed as "unexpected end of JSON input"), and whose 8,589 skills come+// to 287 MB, over the limit together. The list is read whole, only the+// skill by that name is fetched, and an update asks for its folder's list+// alone. Two skills of it updated together are each fetched, neither+// taken for removed.+func TestMarketSkillFromAHugeRepository(t *testing.T) {+ sandbox(t)+ t.Setenv("GITHUB_TOKEN", "")+ t.Setenv("GH_TOKEN", "")+ g := newFakeBigRepo(t)+ g.pad = 60_000 // ~11 MB of list, as sickn33/agentic-awesome-skills'+ // each under the limit, together over it+ g.files["skills/hindsight-local/references/big.md"] = strings.Repeat("local ", 10<<10)+ g.files["hindsight-integrations/cursor/skills/hindsight-recall/references/big.md"] = strings.Repeat("recall ", 10<<10)++ ok(t)(InstallMarketSkill("vectorize-io/hindsight", "hindsight-docs", []string{"claude"}))+ if b, err := os.ReadFile(filepath.Join(skillDir("hindsight-docs"), "references/changelog/index.md")); err != nil || string(b) != "# Changelog\n" {+ t.Fatalf("hindsight-docs: %q, %v", b, err)+ }+ g.mu.Lock()+ for _, r := range g.raw {+ if !strings.HasPrefix(r, "skills/hindsight-docs/") {+ t.Errorf("raw was asked for %s, outside the one skill the market installs", r)+ }+ }+ g.raw, g.trees = nil, nil+ g.mu.Unlock()++ ok(t)(InstallMarketSkill("vectorize-io/hindsight", "hindsight-recall", []string{"claude"}))+ g.mu.Lock()+ g.trees = nil+ g.mu.Unlock()+ r, err := UpdateSkills()+ if err != nil || len(r.Gone) > 0 || len(r.Unupdated) > 0 || len(r.Updated) != 2 {+ t.Fatalf("update: %+v, %v", r, err)+ }+ g.mu.Lock()+ defer g.mu.Unlock()+ sort.Strings(g.trees)+ if want := []string{"HEAD:hindsight-integrations/cursor/skills/hindsight-recall", "HEAD:skills/hindsight-docs"}; !slices.Equal(g.trees, want) {+ t.Errorf("an update listed %q, want each skill's own folder %q", g.trees, want)+ }+}++// A list GitHub cut off itself (over 100,000 files) says the repository+// is too big; it is never read as the whole of it.+func TestTruncatedListIsTooBig(t *testing.T) {+ sandbox(t)+ srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {+ fmt.Fprint(w, `{"sha":"x","tree":[{"path":"skills/a/SKILL.md","type":"blob","mode":"100644","sha":"y","size":3}],"truncated":true}`)+ }))+ defer srv.Close()+ oldA := githubAPI+ githubAPI = srv.URL+ defer func() { githubAPI = oldA }()+ for _, path := range []string{"", "skills/a"} {+ if _, err := fetchSkillFiles(Source{Kind: "github", Repo: "o/r", Path: path}, ""); !errors.Is(err, errTooBig) {+ t.Errorf("path %q: %v", path, err)+ }+ }+}