Every agent's model. One place. Codex on DeepSeek, Claude Code on Kimi, from the menu bar.
claude: a Claude Code run holding over its memory budget is let go after its turn, its session resumed (#3256)
yetonea093e9c3a5f2
diff --git a/docs/subsystems/claude-subscription-bridge.md b/docs/subsystems/claude-subscription-bridge.mdindex b2115002d..ddf2cb77c 100644--- a/docs/subsystems/claude-subscription-bridge.md+++ b/docs/subsystems/claude-subscription-bridge.md@@ -473,6 +473,35 @@ for the same room), and the request isn't tried again after a moment. Amember of the group at another provider answers it. Alone, the client getsthe 503.+## How large one Claude Code process may grow (`claudeRunBudget`, #3256)++`claudeRoom` bounds how many Claude Code processes run, not how large each+is. A run is kept for its conversation's next turn, turn after turn, and a+service that is never restarted kept one for days: on a 31 GiB host a+`node-MainThread` in magpie.service reached 8 GiB RSS and the kernel's OOM+killer took the machine's other work with it.++Each time a run's reply ends, `oversized` (`claude_rss.go`) reads the+resident memory of its process tree (`proc.Tree.RSS`: `/proc/<pid>/status`+VmRSS on Linux, `ps -o rss=` on macOS, the working sets of every process in+its job on Windows, where an npm install's `claude.cmd` is a cmd.exe with+the node under it) and logs it with the pid. The budget is an eighth of the+memory magpie may use (`memLimit` on Linux, `proc.MemTotal` elsewhere), at+least 1 GiB, 4 GiB where neither is known. It is read once per process.++- A whole turn (`ended`, stop) over the budget: the run is not kept idle.+ Its session is shelved at the conversation's key (`letGoOversized`), and+ the next turn resumes it in a new Claude Code (`--resume`), told only the+ new messages, so Anthropic's cache is read as before.+- A reply that called the client's tools (`park`) over 1.5× the budget:+ the run is let go, and the results start a new run told the whole+ conversation. It has no session to shelve there, so it waits for more.++Each Claude Code a run starts logs its pid and model when it starts, and its+run time, how it ended and its peak RSS (`proc.PeakRSS`, rusage+`ru_maxrss`; not on Windows) when it ends: an OOM report's pid can be found+in magpie's log.+## Files a run leaves- Its session: `<config>/projects/<work project>/<session>.jsonl`, fordiff --git a/internal/gateway/claude_rss.go b/internal/gateway/claude_rss.gonew file mode 100644index 000000000..5a14eda96--- /dev/null+++ b/internal/gateway/claude_rss.go@@ -0,0 +1,116 @@+package gateway++import (+ "fmt"+ "log"+ "runtime"+ "sync"+ "time"++ "github.com/yetone/magpie/internal/proc"+)++// A Claude Code run is kept for its conversation's next turn, turn after+// turn, for days on a service that is never restarted, and nothing bounded+// what one process grew to: claudeRoom bounds how many run, not how large+// each is. On a 31 GiB host one reached 8 GiB resident after four days of+// a large conversation, and the kernel's OOM killer took the machine's+// other work with it (#3256). Once a turn is over, a run holding more than+// claudeRunBudget is let go instead of kept: its session is shelved, and+// the next turn resumes it in a new Claude Code, which asks Anthropic the+// same cached prefix.+//+// The budget is an eighth of what magpie's processes may use (memLimit on+// Linux, the computer's memory elsewhere), claudeRunFloor at least; where+// neither is known, claudeRunUnknown. A Claude Code holds ~100 MiB of its+// own at the start (claudeRoom).+const (+ claudeRunFloor = 1 << 30+ claudeRunUnknown = 4 << 30+)++// claudeRunRSS reads a run's resident memory, for tests to give one.+var claudeRunRSS = func(t *proc.Tree) (int64, bool) { return t.RSS() }++var runBudget struct {+ sync.Once+ n int64+ why string+}++// claudeRunBudget is the most a Claude Code run may hold resident and+// still be kept for its next turn, and what sized it.+func claudeRunBudget() (int64, string) {+ runBudget.Do(func() {+ limit, why := memLimit(memGOOS, memRoot)+ if limit <= 0 && memGOOS == runtime.GOOS {+ limit, why = proc.MemTotal(), "the computer's memory"+ }+ if limit <= 0 {+ runBudget.n, runBudget.why = claudeRunUnknown, "memory not known"+ return+ }+ runBudget.n = max(limit/8, claudeRunFloor)+ runBudget.why = fmt.Sprintf("an eighth of %d MiB (%s), %d MiB at least", limit>>20, why, claudeRunFloor>>20)+ })+ return runBudget.n, runBudget.why+}++// oversized says the run holds more than claudeRunBudget times share/2+// (2 a whole turn, 3 a run that waits on tool calls), and logs what it+// holds either way: a run's growth across turns is in the log, for an OOM+// to be put down to the process it was. One waiting on its caller's tool+// results has no session to shelve there: let go, it is started anew with+// the whole conversation, which Anthropic's cache has not seen, so it is+// let go only past half as much again.+func (r *subscriptionRun) oversized(share int64) (string, bool) {+ r.mu.Lock()+ t := r.tree+ r.mu.Unlock()+ if t == nil {+ return "", false+ }+ rss, ok := claudeRunRSS(t)+ if !ok {+ return "", false+ }+ budget, why := claudeRunBudget()+ budget = budget * share / 2+ log.Printf("claude: Claude Code pid %d (%s) holds %d MiB after its turn; %d MiB may be kept (%s)", r.pid(), r.model, rss>>20, budget>>20, why)+ if rss <= budget {+ return "", false+ }+ return fmt.Sprintf("its Claude Code held %d MiB, over the %d MiB a run may hold to be kept (%s)", rss>>20, budget>>20, why), true+}++// pid is the run's Claude Code process id, 0 before it starts.+func (r *subscriptionRun) pid() int {+ if r.cmd == nil || r.cmd.Process == nil {+ return 0+ }+ return r.cmd.Process.Pid+}++// letGoOversized lets go of a run whose turn was whole and that holds too+// much to be kept (oversized): its session is shelved at key, as one let go+// past idleMost is, so the next turn resumes it in a new Claude Code.+func (b *subscriptionBridge) letGoOversized(r *subscriptionRun, key, conv, why string) {+ b.mu.Lock()+ r.mu.Lock()+ closed := r.closed+ r.mu.Unlock()+ if !closed {+ if r.idleKey != "" && b.idle[r.idleKey] == r {+ delete(b.idle, r.idleKey)+ }+ r.idleKey, r.idleAt = key, time.Now()+ if conv != "" {+ r.convKey = conv+ }+ b.shelve(r)+ r.idleKey = ""+ }+ b.mu.Unlock()+ log.Printf("claude: Claude Code pid %d (%s) is let go: %s; its conversation's next turn resumes its session in a new one", r.pid(), r.model, why)+ r.abortFor(why)+}diff --git a/internal/gateway/claude_rss_test.go b/internal/gateway/claude_rss_test.gonew file mode 100644index 000000000..92bdd8017--- /dev/null+++ b/internal/gateway/claude_rss_test.go@@ -0,0 +1,134 @@+package gateway++import (+ "context"+ "os"+ "path/filepath"+ "strings"+ "sync"+ "testing"+ "time"++ "github.com/yetone/magpie/internal/proc"+)++// giveRSS has every Claude Code run read as holding rss (over the budget+// when over), what its real process holds left out.+func giveRSS(t *testing.T, over bool, share int64) {+ t.Helper()+ old := claudeRunRSS+ claudeRunRSS = func(*proc.Tree) (int64, bool) {+ budget, _ := claudeRunBudget()+ budget = budget * share / 2+ if over {+ return budget + 1, true+ }+ return budget, true+ }+ t.Cleanup(func() { claudeRunRSS = old })+}++// A run that holds more than claudeRunBudget once its turn is whole is let+// go, not kept for the next turn: its session is shelved, and that turn+// resumes it in a new Claude Code, told only the new message (#3256: one+// kept for days reached 8 GiB and took the machine down).+func TestClaudeOversizedRunIsLetGoAndResumes(t *testing.T) {+ h := newScriptHarness(t, "ok", "")+ freshRoom(t, 4)+ giveRSS(t, true, 2)+ h.ask("rules A", "a1")+ var pidA string+ for pid := range h.runs() {+ pidA = pid+ }+ eventually(t, "the oversized run to end", func() bool {+ h.s.subscription.mu.Lock()+ defer h.s.subscription.mu.Unlock()+ return len(h.s.subscription.runs) == 0 && len(h.s.subscription.idle) == 0+ })+ h.ask("rules A", "a1", "a2")+ runs := h.runs()+ if len(runs) != 2 {+ t.Fatalf("runs: %v", runs)+ }+ resumed := false+ for pid, l := range runs {+ if pid != pidA && strings.Contains(l, "--resume s"+pidA) {+ resumed = true+ for _, line := range strings.Split(l, "\n") {+ if strings.HasPrefix(line, "told ") && strings.Contains(line, "a1") {+ t.Fatalf("the resumed run was told the conversation again:\n%s", l)+ }+ }+ }+ }+ if !resumed {+ t.Fatalf("the next turn didn't resume the oversized run's session s%s: %v", pidA, runs)+ }+}++// One within the budget is kept, and its next turn goes to the same+// Claude Code.+func TestClaudeRunWithinBudgetIsKept(t *testing.T) {+ h := newScriptHarness(t, "ok", "")+ freshRoom(t, 4)+ giveRSS(t, false, 2)+ h.ask("rules A", "a1")+ h.ask("rules A", "a1", "a2")+ if runs := h.runs(); len(runs) != 1 {+ t.Fatalf("a run within its budget wasn't kept: %v", runs)+ }+}++// A run waiting on its caller's tool results is let go past half as much+// again as the budget, and kept up to it.+func TestClaudeOversizedParkedRunIsLetGo(t *testing.T) {+ fakeClaudeCalling(t)+ s := New()+ t.Cleanup(s.subscription.abortAll)+ giveRSS(t, false, 3)+ kept := askForCall(t, s, context.Background(), "read a file")+ giveRSS(t, true, 3)+ over := askForCall(t, s, context.Background(), "read another file")+ if t.Failed() {+ return+ }+ if n := leftover([]int{over}, 3*time.Second); n != 0 {+ t.Fatal("a run over its budget was kept waiting on its tool calls")+ }+ if !alive(kept) {+ t.Fatal("a run within its budget was let go")+ }+}++// The budget is an eighth of the memory magpie may use, a floor under it.+func TestClaudeRunBudgetScalesWithMemory(t *testing.T) {+ root := t.TempDir()+ putMeminfo(t, root, "MemTotal: 32505856 kB\n")+ oldGOOS, oldRoot := memGOOS, memRoot+ memGOOS, memRoot = "linux", root+ t.Cleanup(func() { memGOOS, memRoot = oldGOOS, oldRoot })+ budget := func() int64 {+ runBudget.Once = sync.Once{}+ n, _ := claudeRunBudget()+ return n+ }+ t.Cleanup(func() { runBudget.Once = sync.Once{} })+ if n := budget(); n != 32505856<<10/8 {+ t.Fatalf("31 GiB holds a run of %d MiB", n>>20)+ }+ putMeminfo(t, root, "MemTotal: 2097152 kB\n")+ if n := budget(); n != claudeRunFloor {+ t.Fatalf("2 GiB holds a run of %d MiB, not the floor", n>>20)+ }+}++func putMeminfo(t *testing.T, root, text string) {+ t.Helper()+ if err := os.MkdirAll(filepath.Join(root, "proc"), 0o755); err != nil {+ t.Fatal(err)+ }+ if err := os.WriteFile(filepath.Join(root, "proc", "meminfo"), []byte(text), 0o644); err != nil {+ t.Fatal(err)+ }+}diff --git a/internal/gateway/claude_subscription.go b/internal/gateway/claude_subscription.goindex a2b498cb9..15470c8fb 100644--- a/internal/gateway/claude_subscription.go+++ b/internal/gateway/claude_subscription.go@@ -1328,6 +1328,10 @@ func (r *subscriptionRun) ended(req *Request, said, stop string, ok bool) {// there when the conversation's own next turn comes (resume)r.behindKey, r.behindTurn = back, turnr.bridge.mu.Unlock()+ if why, over := r.oversized(2); over {+ r.bridge.letGoOversized(r, key, conv, why)+ return+ }if !r.bridge.keepIdle(r, key, conv) {return // it ended during its reply (#1667)}@@ -1793,6 +1797,11 @@ func (r *subscriptionRun) park() {r.timer.Reset(parkLongest)return}+ if why, over := r.oversized(3); over {+ log.Printf("claude: Claude Code pid %d (%s) is let go while it waits on tool calls: %s; their results go to a new one told the whole conversation", r.pid(), r.model, why)+ r.abortFor(why)+ return+ }b := r.bridgeif b == nil {return@@ -3765,8 +3774,15 @@ func (r *subscriptionRun) launch() error {r.mu.Lock()r.tree, r.exited = t, exitedr.mu.Unlock()+ started := time.Now()+ log.Printf("claude: Claude Code pid %d started, a run on %s", r.pid(), r.model)go func() {err := t.Wait()+ how := "exited"+ if err != nil {+ how = err.Error()+ }+ log.Printf("claude: Claude Code pid %d (%s) ended after %s (%s), %d MiB resident at most", r.pid(), r.model, time.Since(started).Round(time.Second), how, proc.PeakRSS(r.cmd)>>20)r.drain()// exited closes under mu, which abort reads it under: a run it// counts as leaving the room is one whose room is freed herediff --git a/internal/proc/mem_darwin.go b/internal/proc/mem_darwin.gonew file mode 100644index 000000000..ab551fce1--- /dev/null+++ b/internal/proc/mem_darwin.go@@ -0,0 +1,11 @@+package proc++import "golang.org/x/sys/unix"++func memTotal() int64 {+ n, err := unix.SysctlUint64("hw.memsize")+ if err != nil {+ return 0+ }+ return int64(n)+}diff --git a/internal/proc/mem_linux.go b/internal/proc/mem_linux.gonew file mode 100644index 000000000..d7b0bda60--- /dev/null+++ b/internal/proc/mem_linux.go@@ -0,0 +1,20 @@+package proc++import (+ "os"+ "strings"+)++func memTotal() int64 {+ b, err := os.ReadFile("/proc/meminfo")+ if err != nil {+ return 0+ }+ for _, line := range strings.Split(string(b), "\n") {+ if f := strings.Fields(line); len(f) >= 2 && f[0] == "MemTotal:" {+ n, _ := parseKiB(f[1])+ return n+ }+ }+ return 0+}diff --git a/internal/proc/mem_other.go b/internal/proc/mem_other.gonew file mode 100644index 000000000..366bc3adf--- /dev/null+++ b/internal/proc/mem_other.go@@ -0,0 +1,5 @@+//go:build !darwin && !linux && !windows++package proc++func memTotal() int64 { return 0 }diff --git a/internal/proc/peak_darwin.go b/internal/proc/peak_darwin.gonew file mode 100644index 000000000..3b2a6d1a3--- /dev/null+++ b/internal/proc/peak_darwin.go@@ -0,0 +1,14 @@+package proc++import (+ "os"+ "syscall"+)++// peakRSS is the rusage's ru_maxrss, which macOS says in bytes.+func peakRSS(s *os.ProcessState) int64 {+ if ru, ok := s.SysUsage().(*syscall.Rusage); ok {+ return ru.Maxrss+ }+ return 0+}diff --git a/internal/proc/peak_linux.go b/internal/proc/peak_linux.gonew file mode 100644index 000000000..29473a92a--- /dev/null+++ b/internal/proc/peak_linux.go@@ -0,0 +1,14 @@+package proc++import (+ "os"+ "syscall"+)++// peakRSS is the rusage's ru_maxrss, which Linux says in KiB.+func peakRSS(s *os.ProcessState) int64 {+ if ru, ok := s.SysUsage().(*syscall.Rusage); ok {+ return ru.Maxrss << 10+ }+ return 0+}diff --git a/internal/proc/peak_other.go b/internal/proc/peak_other.gonew file mode 100644index 000000000..13aeedebc--- /dev/null+++ b/internal/proc/peak_other.go@@ -0,0 +1,7 @@+//go:build !darwin && !linux++package proc++import "os"++func peakRSS(*os.ProcessState) int64 { return 0 }diff --git a/internal/proc/rss.go b/internal/proc/rss.gonew file mode 100644index 000000000..287002469--- /dev/null+++ b/internal/proc/rss.go@@ -0,0 +1,29 @@+package proc++import "os/exec"++// RSS is the memory the tree holds resident, in bytes: its command's on+// Unix, every process of its job on Windows, where an npm install's+// claude.cmd is a cmd.exe with the node under it. False when it can't be+// read, or the command was waited for: its pid may be another process's+// by then.+func (t *Tree) RSS() (int64, bool) {+ t.mu.Lock()+ if t.done || t.cmd.Process == nil {+ t.mu.Unlock()+ return 0, false+ }+ return t.rss(t.cmd) // unlocks t.mu+}++// MemTotal is this computer's physical memory, in bytes; 0 when not known.+func MemTotal() int64 { return memTotal() }++// PeakRSS is the most the command held resident, in bytes, once it was+// waited for; 0 where the system doesn't say (Windows).+func PeakRSS(cmd *exec.Cmd) int64 {+ if cmd.ProcessState == nil {+ return 0+ }+ return peakRSS(cmd.ProcessState)+}diff --git a/internal/proc/rss_test.go b/internal/proc/rss_test.gonew file mode 100644index 000000000..2d2198065--- /dev/null+++ b/internal/proc/rss_test.go@@ -0,0 +1,35 @@+package proc++import (+ "os/exec"+ "runtime"+ "testing"+)++// A running tree's resident memory is read, and none once it was waited+// for: its pid may be another process's by then.+func TestTreeRSS(t *testing.T) {+ name, args := "sleep", []string{"30"}+ if runtime.GOOS == "windows" {+ name, args = "ping", []string{"-n", "30", "127.0.0.1"}+ }+ if _, err := exec.LookPath(name); err != nil {+ t.Skip("no " + name)+ }+ tree, err := StartTree(exec.Command(name, args...))+ if err != nil {+ t.Fatal(err)+ }+ rss, ok := tree.RSS()+ if !ok || rss < 64<<10 || rss > 1<<30 {+ t.Fatalf("a running %s holds %d bytes (%v)", name, rss, ok)+ }+ tree.Kill()+ _ = tree.Wait()+ if _, ok := tree.RSS(); ok {+ t.Fatal("a tree waited for was read")+ }+ if MemTotal() <= 0 && (runtime.GOOS == "darwin" || runtime.GOOS == "linux" || runtime.GOOS == "windows") {+ t.Fatal("this computer's memory isn't known")+ }+}diff --git a/internal/proc/rss_unix.go b/internal/proc/rss_unix.gonew file mode 100644index 000000000..668039005--- /dev/null+++ b/internal/proc/rss_unix.go@@ -0,0 +1,48 @@+//go:build !windows++package proc++import (+ "context"+ "os"+ "os/exec"+ "strconv"+ "strings"+ "time"+)++// rss reads the command's resident memory with t.mu let go first: on+// macOS it asks ps, which Wait and Kill don't wait behind.+func (t *Tree) rss(cmd *exec.Cmd) (int64, bool) {+ pid := cmd.Process.Pid+ t.mu.Unlock()+ return pidRSS(pid)+}++// pidRSS is pid's resident memory: Linux's /proc/<pid>/status VmRSS, ps+// elsewhere. Both say it in KiB.+func pidRSS(pid int) (int64, bool) {+ if b, err := os.ReadFile("/proc/" + strconv.Itoa(pid) + "/status"); err == nil {+ for _, line := range strings.Split(string(b), "\n") {+ if f := strings.Fields(line); len(f) >= 2 && f[0] == "VmRSS:" {+ return parseKiB(f[1])+ }+ }+ return 0, false+ }+ ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second)+ defer cancel()+ out, err := CommandContext(ctx, "ps", "-o", "rss=", "-p", strconv.Itoa(pid)).Output()+ if err != nil {+ return 0, false+ }+ return parseKiB(strings.TrimSpace(string(out)))+}++func parseKiB(s string) (int64, bool) {+ kb, err := strconv.ParseInt(s, 10, 64)+ if err != nil || kb <= 0 {+ return 0, false+ }+ return kb << 10, true+}diff --git a/internal/proc/rss_windows.go b/internal/proc/rss_windows.gonew file mode 100644index 000000000..6647f62c0--- /dev/null+++ b/internal/proc/rss_windows.go@@ -0,0 +1,87 @@+package proc++import (+ "os/exec"+ "unsafe"++ "golang.org/x/sys/windows"+)++var (+ processMemoryInfo = windows.NewLazySystemDLL("kernel32.dll").NewProc("K32GetProcessMemoryInfo")+ memoryStatus = windows.NewLazySystemDLL("kernel32.dll").NewProc("GlobalMemoryStatusEx")+)++// processMemoryCounters is PROCESS_MEMORY_COUNTERS.+type processMemoryCounters struct {+ cb uint32+ pageFaultCount uint32+ peakWorkingSetSize uintptr+ workingSetSize uintptr+ quotaPeakPagedPoolUsage uintptr+ quotaPagedPoolUsage uintptr+ quotaPeakNonPagedPoolUsage uintptr+ quotaNonPagedPoolUsage uintptr+ pagefileUsage uintptr+ peakPagefileUsage uintptr+}++// rss adds up the working sets of the job's processes, under t.mu: the+// job's handle is closed by sweep. A command not in a job is read alone.+func (t *Tree) rss(cmd *exec.Cmd) (int64, bool) {+ defer t.mu.Unlock()+ pids := []uint32{uint32(cmd.Process.Pid)}+ if t.job != 0 {+ // JOBOBJECT_BASIC_PROCESS_ID_LIST: two counts, then the ids+ var list struct {+ assigned, listed uint32+ ids [256]uintptr+ }+ if err := windows.QueryInformationJobObject(t.job, windows.JobObjectBasicProcessIdList,+ uintptr(unsafe.Pointer(&list)), uint32(unsafe.Sizeof(list)), nil); err == nil && list.listed > 0 {+ pids = pids[:0]+ for _, id := range list.ids[:min(int(list.listed), len(list.ids))] {+ pids = append(pids, uint32(id))+ }+ }+ }+ var total int64+ read := false+ for _, pid := range pids {+ h, err := windows.OpenProcess(windows.PROCESS_QUERY_LIMITED_INFORMATION, false, pid)+ if err != nil {+ continue+ }+ var c processMemoryCounters+ c.cb = uint32(unsafe.Sizeof(c))+ ok, _, _ := processMemoryInfo.Call(uintptr(h), uintptr(unsafe.Pointer(&c)), uintptr(c.cb))+ _ = windows.CloseHandle(h)+ if ok != 0 {+ total += int64(c.workingSetSize)+ read = true+ }+ }+ return total, read+}++// memoryStatusEx is MEMORYSTATUSEX.+type memoryStatusEx struct {+ length uint32+ memoryLoad uint32+ totalPhys uint64+ availPhys uint64+ totalPageFile uint64+ availPageFile uint64+ totalVirtual uint64+ availVirtual uint64+ availExtendedVirtual uint64+}++func memTotal() int64 {+ s := memoryStatusEx{}+ s.length = uint32(unsafe.Sizeof(s))+ if ok, _, _ := memoryStatus.Call(uintptr(unsafe.Pointer(&s))); ok == 0 {+ return 0+ }+ return int64(s.totalPhys)+}