A local-first multi-agent app development platform, built on PenguinHarness
feat(machines): a machine gets the Model credentials its Agents need (#575)
Myriad-Dreaminb145666b6b2e
diff --git a/changelog/unreleased/2026-09-01-machines-models.md b/changelog/unreleased/2026-09-01-machines-models.mdnew file mode 100644index 00000000..2bf2a69e--- /dev/null+++ b/changelog/unreleased/2026-09-01-machines-models.md@@ -0,0 +1,28 @@+# A machine gets the Model credentials its Agents need++- **Date:** 2026-09-01+- **Type:** feature+- **Scope:** `server`+- **PR:** [#575](https://github.com/Prism-Shadow/penguin-harness/pull/575)++[中文版](2026-09-01-machines-models.zh.md)++An Agent running on a machine calls the model endpoint **from over there**, against **that machine's** config. So a machine with no credentials is connected and unusable: picking a model here and starting the Session there failed with "Model is not in the Project config". Connecting a machine now hands it the Model config of the Projects that use it.++## Which Projects, and why the routes are under one++Only the Projects on this server that actually list that machine. A machine is shared — one host, one program, adopted by whichever Projects want it — but a credential is not: it belongs to the Project that holds it. That asymmetry is the reason the machines routes sit under a Project at all.++Editing a Model here syncs it outward too, to every machine that should have it — the whole-table save, the default switch in Project settings, and a key minted through a provider's sign-in all end in the same follow-up, so a machine hears of a change whichever way it was made. Not awaited: the person editing a key is not the one who should wait on a set of ssh tunnels. An edit that lands while a sync is running is not dropped: one trailing sync runs when it ends, reading the config as it is then.++## Merged, not replaced++The machine's `PUT /models` is a whole-table replace, so every entry that exists only over there is re-sent — **without** its `apiKey`, since omitting it keeps the stored value and a GET reports keys masked. On a collision this side wins. A machine that someone configured by hand keeps what it had, and does not silently lose a key this server has never seen.++Only entries worth carrying are sent: an entry with an inline key or its own base URL is something the machine cannot already have, while a bare catalog entry is a preset every server seeds for itself.++## Over the tunnel, as that machine's admin++The key travels inside the forward, to the machine's own authenticated `PUT /models`, which writes `.project_config.toml` at mode 0600 — whoever can do this already has ssh to that machine and can read that file. An ordinary API call rather than a far-side script, because the endpoint is what validates, invalidates cached runtimes and tells open tabs.++A Project the machine refuses — an id its own rules will not take, a table it will not accept — is reported and skipped, and the Projects after it are still written; only the machine being unreachable fails the sync as a whole. A failed sync is **reported, never fatal**: the connect log says what the machine said, and the machine stays connected. What is on it did not become wrong because a later step did not happen.diff --git a/changelog/unreleased/2026-09-01-machines-models.zh.md b/changelog/unreleased/2026-09-01-machines-models.zh.mdnew file mode 100644index 00000000..47dc8c12--- /dev/null+++ b/changelog/unreleased/2026-09-01-machines-models.zh.md@@ -0,0 +1,28 @@+# 机器拿到它的 Agent 所需的模型凭据++- **Date:** 2026-09-01+- **Type:** feature+- **Scope:** `server`+- **PR:** [#575](https://github.com/Prism-Shadow/penguin-harness/pull/575)++[English](2026-09-01-machines-models.md)++在一台机器上运行的 Agent,是**从那边**去调模型端点的,依据的是**那台机器的**配置。因此一台没有凭据的机器是连上了却用不了:在这边选好模型、到那边开 Session,会以「Model is not in the Project config」失败。现在连接一台机器时,会把使用它的那些 Project 的模型配置一并交过去。++## 交给哪些 Project,以及路由为何归属于 Project++只交给本服务端上确实把该机器列入自己名下的 Project。机器是共享的——一台主机、一份程序,谁想用谁纳入——但凭据不是:它属于持有它的那个 Project。正是这个不对称,决定了 machines 路由为什么要放在 Project 之下。++在这边编辑模型时也会向外同步,同步到所有本应拥有它的机器——整表保存、项目设置里切换默认模型、通过提供商登录铸造的密钥,三条路径都汇入同一个后续动作,所以无论改动从哪条路来,机器都会知道。这个过程不被等待:编辑一把密钥的人,不该是等一组 ssh 隧道的人。同步进行中到达的改动不会被丢掉:本次结束后再跑一次收尾同步,读的是那一刻的配置。++## 是合并,不是覆盖++机器那侧的 `PUT /models` 是整表替换,因此只存在于对面的条目会被原样回送——但**不带** `apiKey`,因为省略它就保留已存的值,而 GET 读回的密钥本就是打码的。发生冲突时以本侧为准。有人手工配过的机器保住它原有的东西,不会悄悄丢掉一把本服务端从未见过的密钥。++只发送值得携带的条目:带内联密钥或自有 base URL 的条目是机器不可能已经有的,而裸的目录条目是每台服务端自己都会播种的预设。++## 走隧道,以那台机器的管理员身份++密钥在转发内部送达机器自己那个需要鉴权的 `PUT /models`,由它写入 `.project_config.toml`,权限 0600——能做这件事的人本就有到那台机器的 ssh,也就本就能读那个文件。用一次普通的 API 调用而不是一段远端脚本,是因为校验、让缓存的运行时失效、通知已打开的标签页,都是那个端点在做的事。++机器拒绝的项目——id 不合它自己的规则、表它不接受——会被报告并跳过,后面的项目照常写入;只有机器不可达才会让整次同步失败。同步失败**如实报告,但从不致命**:连接日志里写的是机器自己说的话,机器仍然保持连接。不会因为后面一步没做成,就让已经在它上面的东西变成错的。diff --git a/packages/server/src/http/routes/model-oauth.ts b/packages/server/src/http/routes/model-oauth.tsindex 0bb40034..689abbcc 100644--- a/packages/server/src/http/routes/model-oauth.ts+++ b/packages/server/src/http/routes/model-oauth.ts@@ -25,7 +25,7 @@ import type { AppEnv } from "../../auth/middleware.js";import type { AppDeps } from "../../app.js";import { HttpError } from "../errors.js";import { badRequest, readJson, requireString, requireValidId } from "../validate.js";-import { publishCredentialsUpdated } from "./models.js";+import { modelConfigChanged } from "./models.js";/** Flow ids are base64url (`randomBytes(32)`); reject anything else before it reaches the store. */const FLOW_ID_RE = /^[A-Za-z0-9_-]{1,128}$/;@@ -105,10 +105,13 @@ p { margin: 0; color: #4b5563; }`;}-/** Credential change: same follow-up the models PUT performs, so live Sessions pick the key up. */+/**+ * Credential change: the same follow-up the models PUT performs — live Sessions pick the+ * key up, open tabs unlock, and the machines this Project uses receive it. One helper for+ * every write path, so a key minted here reaches a machine the same way one pasted does.+ */function credentialsChanged(deps: AppDeps, projectId: string): void {- deps.manager.invalidateProjectRuntimes(projectId);- publishCredentialsUpdated(deps, projectId);+ modelConfigChanged(deps, projectId);}/**diff --git a/packages/server/src/http/routes/models.ts b/packages/server/src/http/routes/models.tsindex 3d89d410..b8c085c2 100644--- a/packages/server/src/http/routes/models.ts+++ b/packages/server/src/http/routes/models.ts@@ -38,6 +38,26 @@ export function publishCredentialsUpdated(deps: AppDeps, projectId: string): voi}}+/**+ * EVERYTHING that follows a change to a Project's model config, in one place, so that no+ * path that writes it can forget one of the three: the whole-table PUT, the default switch,+ * and the key-minting flows all end here.+ *+ * - Effective-value semantics (mirrors the vault route): no hot swap into a Task already in+ * flight, but every cached runtime in this Project is invalidated — the next Task on any+ * of its Sessions re-resumes and reads the new api_key / base_url. Without this, a key+ * edit would not reach a loaded Session until the 30-minute idle sweep or a restart.+ * - Live unlock: open tabs clear their auth-dead composer immediately (no reload needed).+ * - The machines run their own Agents against their own config, so a change here has to+ * reach them or their Sessions keep failing on the old one — or start on the wrong default.+ * Not awaited: the person editing is not the one who should wait for a set of ssh tunnels.+ */+export function modelConfigChanged(deps: AppDeps, projectId: string): void {+ deps.manager.invalidateProjectRuntimes(projectId);+ publishCredentialsUpdated(deps, projectId);+ void deps.machines.syncModelsEverywhere(projectId);+}+/** Validate a paired reference object ({ provider, modelId }); shape mismatch throws 400. */function parseRef(value: unknown, label: string): ModelRefDto {if (value === null || typeof value !== "object" || Array.isArray(value)) {@@ -173,13 +193,7 @@ export function modelsRoutes(deps: AppDeps): Hono<AppEnv> {deps.projectService.requireProjectOwner(c.var.user.userId, projectId);const req = parseModelsUpdate(await readJson(c));const res = await deps.projectConfigService.updateModels(projectId, req);- // Effective-value semantics (mirrors the vault route): no hot swap into a Task already- // in flight, but every cached runtime in this Project is invalidated — the next Task on- // any of its Sessions re-resumes and reads the new api_key / base_url. Without this, a- // key edit would not reach a loaded Session until the 30-minute idle sweep or a restart.- deps.manager.invalidateProjectRuntimes(projectId);- // Live unlock: open tabs clear their auth-dead composer immediately (no reload needed).- publishCredentialsUpdated(deps, projectId);+ modelConfigChanged(deps, projectId);return c.json(res);});@@ -187,8 +201,10 @@ export function modelsRoutes(deps: AppDeps): Hono<AppEnv> {// whole-table PUT above maintains, without resending the table — project settings can// change the default without carrying credentials. The pair must name a configured// entry (400 otherwise, same rule as the whole-table route's defaultModel). No runtime- // invalidation and no credentials_updated: existing Sessions pin their model at- // creation, and no credential changes here.+ // invalidation and no credentials_updated here: existing Sessions pin their model at+ // creation, and no credential changes. The machines DO hear of it — a Session started+ // over there without an explicit model runs on that machine's default, and the default+ // is part of what the sync carries.app.put("/default", async (c) => {const projectId = requireValidId(c, "projectId");deps.projectService.requireProjectOwner(c.var.user.userId, projectId);@@ -198,6 +214,7 @@ export function modelsRoutes(deps: AppDeps): Hono<AppEnv> {modelId: requireString(body, "modelId", { minLen: 1, maxLen: 200 }),};const defaultModel = await deps.projectConfigService.setDefaultModelRef(projectId, ref);+ void deps.machines.syncModelsEverywhere(projectId);return c.json({ defaultModel } satisfies DefaultModelResponse);});diff --git a/packages/server/src/machines/machine-api.ts b/packages/server/src/machines/machine-api.tsnew file mode 100644index 00000000..65266999--- /dev/null+++ b/packages/server/src/machines/machine-api.ts@@ -0,0 +1,101 @@+/**+ * A client for one machine's own API, dialled through the connection this side holds. Every piece+ * of work this server does ON a machine goes through here — the hot update, the model sync.+ *+ * node:http rather than fetch: the Host header has to be the canonical app host+ * (`localhost:<port>`) while the connection goes to 127.0.0.1, and fetch ignores an explicit+ * host header. On a loopback bind the App answers on ONE loopback name and serves previews+ * on the other (app.ts's canonical-host guard), so a request addressed to `127.0.0.1` is+ * refused as a preview-host call to /api.+ */+import http from "node:http";++/** The machine's own API, reached through its tunnel as an authenticated caller. */+export interface MachineApi {+ request(+ method: "GET" | "POST" | "PUT",+ path: string,+ body?: unknown,+ ): Promise<{ status: number; text: string }>;+ /**+ * A POST whose body is bytes rather than JSON, with a deadline of its own. The hot update+ * is megabytes, and the machine unpacks, boots and commits it before answering — a timeout+ * sized for a settings write would call a working upgrade dead halfway through.+ */+ postBytes(+ path: string,+ contentType: string,+ body: Buffer,+ timeoutMs: number,+ ): Promise<{ status: number; text: string }>;+}++/** One client per (machine, session): `agent` dials the machine's server through its session, `port` is the server's port over there. */+export function machineApi(agent: http.Agent, port: number, cookie: string): MachineApi {+ /** One request, answered whole — so both shapes agree on Host, cookie and what a failure is. */+ const send = (+ method: string,+ path: string,+ payload: Buffer | null,+ contentType: string,+ timeoutMs: number,+ ): Promise<{ status: number; text: string }> =>+ new Promise((resolve, reject) => {+ const req = http.request(+ {+ agent,+ host: "127.0.0.1",+ port,+ path,+ method,+ headers: {+ host: `localhost:${port}`,+ cookie,+ ...(payload === null+ ? {}+ : { "content-type": contentType, "content-length": String(payload.length) }),+ },+ timeout: timeoutMs,+ },+ (res) => {+ const chunks: Buffer[] = [];+ res.on("data", (chunk: Buffer) => chunks.push(chunk));+ res.on("end", () =>+ resolve({+ status: res.statusCode ?? 0,+ text: Buffer.concat(chunks).toString("utf8"),+ }),+ );+ // `end` is the only settling event on the happy path. A machine that closes the+ // channel after the headers and before the body — its server restarting under a+ // hot update, the session dropping mid-answer — ends the response without `end`,+ // and the request itself need not error once headers were received. Left alone,+ // the promise would hang the job that made it.+ res.on("error", reject);+ res.on("close", () => {+ if (!res.complete) reject(new Error("the machine's server closed mid-answer"));+ });+ },+ );+ req.on("timeout", () => {+ req.destroy();+ reject(new Error("the machine's server did not answer in time"));+ });+ req.on("error", reject);+ if (payload === null) req.end();+ else req.end(payload);+ });++ return {+ request: (method, path, body) =>+ send(+ method,+ path,+ body === undefined ? null : Buffer.from(JSON.stringify(body)),+ "application/json",+ 30_000,+ ),+ postBytes: (path, contentType, body, timeoutMs) =>+ send("POST", path, body, contentType, timeoutMs),+ };+}diff --git a/packages/server/src/machines/models-sync.ts b/packages/server/src/machines/models-sync.tsnew file mode 100644index 00000000..9b069b0a--- /dev/null+++ b/packages/server/src/machines/models-sync.ts@@ -0,0 +1,241 @@+/**+ * Handing this server's Projects, and their Model config, to a machine they were given to.+ *+ * An Agent running over there calls the model endpoint from over there, so that machine+ * needs the credential. Without this, picking a model here and starting the Session there+ * fails with "Model is not in the Project config".+ *+ * The key travels inside the tunnel to the machine's own `PUT /models`, which writes it to+ * `.project_config.toml` at mode 0600 — whoever can run this already has ssh to that machine+ * and can read that file. An ordinary authenticated call rather than a far-side script: the+ * endpoint validates, invalidates cached runtimes and tells open tabs.+ *+ * MERGED, NOT REPLACED. `PUT /models` is a whole-table replace, so every remote-only entry is+ * re-sent — WITHOUT `apiKey`, since omitting it keeps the stored value and a GET reports keys+ * masked. Ours win on a collision.+ */+import type { MachineApi } from "./machine-api.js";+import type { ModelEntry, ModelRef } from "@prismshadow/penguin-core";+import type {+ ModelInfo,+ ModelRefDto,+ ModelUpdateEntry,+ ModelsResponse,+ ModelsUpdateRequest,+} from "../api/types.js";++/** This side's half of the merge: a Project's configured models, keys in plaintext. */+export interface LocalModels {+ models: ModelEntry[];+ defaultModel?: ModelRef;+ visionModel?: ModelRef;+ /** Display name, so a Project created on the machine reads as the same Project it is. */+ name?: string;+}++/** What a sync did, in the words the connect log shows. */+export type ModelSyncOutcome =+ | {+ kind: "synced";+ /** Projects whose model table was written on that machine (empty = nothing needed it). */+ projects: string[];+ /** Of those, the ones this sync had to create over there first. */+ created: string[];+ /**+ * Projects the machine would not take, each with its own reason — a refusal is an+ * answer about ONE Project's boundary, and the Projects after it still get theirs.+ */+ refused: { projectId: string; detail: string }[];+ }+ | { kind: "failed"; detail: string };++/** Entry key: the `(provider, model_id)` pair, joined by a byte neither half can contain. */+const refKey = (provider: string, modelId: string): string => `${provider}\u0000${modelId}`;++/** One of ours, with its credential — the whole point of the exercise. */+function fromLocal(entry: ModelEntry): ModelUpdateEntry {+ return {+ provider: entry.provider,+ modelId: entry.model_id,+ ...(entry.display_name !== undefined ? { displayName: entry.display_name } : {}),+ ...(entry.context_window !== undefined ? { contextWindow: entry.context_window } : {}),+ ...(entry.client_type !== undefined ? { clientType: entry.client_type } : {}),+ ...(entry.vision !== undefined ? { vision: entry.vision } : {}),+ ...(entry.max_tokens !== undefined ? { maxTokens: entry.max_tokens } : {}),+ ...(entry.fast_mode === true ? { fastMode: true } : {}),+ ...(entry.pricing !== undefined+ ? {+ pricing: {+ cacheRead: entry.pricing.cache_read,+ cacheWrite: entry.pricing.cache_write,+ output: entry.pricing.output,+ },+ }+ : {}),+ // An entry with no inline key authenticates from THIS machine's environment+ // (ANTHROPIC_API_KEY and friends), and an environment variable is not ours to carry+ // anywhere. Omitted rather than cleared: the machine may have its own value for the same+ // pair, and destroying it would be worse than leaving it.+ ...(entry.api_key !== undefined && entry.api_key !== "" ? { apiKey: entry.api_key } : {}),+ ...(entry.base_url !== undefined ? { baseUrl: entry.base_url } : {}),+ };+}++/**+ * One of theirs, carried across the whole-table replace untouched.+ *+ * No `apiKey` — see the header: omitting it is what preserves the key already on that+ * machine. The one field that does not round-trip exactly is `vision`, which a GET reports+ * from the built-in catalog when the entry carries no annotation; re-sending it writes that+ * catalog value down explicitly. Same effective value, one more line in their TOML.+ */+function fromRemote(info: ModelInfo): ModelUpdateEntry {+ return {+ provider: info.provider,+ modelId: info.modelId,+ ...(info.displayName !== undefined ? { displayName: info.displayName } : {}),+ ...(info.contextWindow !== undefined ? { contextWindow: info.contextWindow } : {}),+ ...(info.clientType !== undefined ? { clientType: info.clientType } : {}),+ ...(info.vision !== undefined ? { vision: info.vision } : {}),+ ...(info.maxTokens !== undefined ? { maxTokens: info.maxTokens } : {}),+ ...(info.fastMode === true ? { fastMode: true } : {}),+ ...(info.pricing !== undefined ? { pricing: info.pricing } : {}),+ ...(info.credential?.baseUrl !== undefined ? { baseUrl: info.credential.baseUrl } : {}),+ };+}++/**+ * The table to PUT on a machine: ours, plus everything of theirs we are not replacing.+ *+ * Callers must not reach here with an empty local table — a whole-table replace built from+ * one would delete every model that machine has (syncModelsToMachine is what guards it).+ */+export function planModelSync(local: LocalModels, remote: ModelsResponse): ModelsUpdateRequest {+ const ours = new Set(local.models.map((entry) => refKey(entry.provider, entry.model_id)));+ const models: ModelUpdateEntry[] = [+ ...local.models.map(fromLocal),+ ...remote.models+ .filter((info) => !ours.has(refKey(info.provider, info.modelId)))+ .map(fromRemote),+ ];+ // The pointers follow ours. A Project with this id on that machine IS this Project — that+ // is what giving the machine to it meant — and a default pointing somewhere else is how a+ // Session started without an explicit model quietly runs on the wrong one. Only ever set to+ // a pair in the table being sent; one we cannot satisfy is left as theirs, since omitting+ // the field is what keeps it.+ const follow = (ref: ModelRef | undefined): ModelRefDto | undefined =>+ ref !== undefined && ours.has(refKey(ref.provider, ref.model_id))+ ? { provider: ref.provider, modelId: ref.model_id }+ : undefined;+ const defaultModel = follow(local.defaultModel);+ const visionModel = follow(local.visionModel);+ return {+ models,+ ...(defaultModel !== undefined ? { defaultModel } : {}),+ ...(visionModel !== undefined ? { visionModel } : {}),+ };+}++/** Project ids that machine has, narrowed from its own answer (it may be an older build). */+function projectIdsIn(text: string): string[] {+ const body: unknown = JSON.parse(text);+ if (body === null || typeof body !== "object") return [];+ const list = (body as { projects?: unknown }).projects;+ if (!Array.isArray(list)) return [];+ return list+ .map((entry) =>+ entry !== null && typeof entry === "object"+ ? (entry as { projectId?: unknown }).projectId+ : undefined,+ )+ .filter((id): id is string => typeof id === "string");+}++/**+ * Gives a machine the Projects it was given to, and the models they need.+ *+ * `projects` is the entitlement, and it is the ONLY input that decides what is written: a+ * machine belongs to the Projects that adopted it, and a Project that does not use it has no+ * business putting its keys on it.+ *+ * A Project missing over there is CREATED, with the same id. That is the point of an id being+ * stable: the Project on that machine is not a lookalike, it is this Project, and a Session+ * started on that machine sends this id for it to resolve. Creating it is not this server+ * inventing workspaces on a stranger's host — the host was handed to this Project, and a+ * machine that cannot answer for the Project that owns it is a machine that cannot run+ * anything for it.+ *+ * A creation that is REFUSED is reported and that Project is skipped: the machine saying no+ * (an id already taken by someone else's Project, a shape it will not accept) is an answer+ * about a boundary, and writing models around it is not this function's call to make. Skipped+ * means exactly that — the Projects after it are still written. Only the machine being+ * unreachable, or refusing to say what Projects it has, fails the sync as a whole.+ */+export async function syncModelsToMachine(opts: {+ api: MachineApi;+ /** This side's table for a Project, or null when it has no config for it. */+ loadLocal: (projectId: string) => Promise<LocalModels | null>;+ /** The Projects entitled to this machine — nothing outside this list is touched. */+ projects: string[];+}): Promise<ModelSyncOutcome> {+ let theirs: string[];+ try {+ const listed = await opts.api.request("GET", "/api/projects");+ if (listed.status !== 200) {+ return { kind: "failed", detail: `it answered ${listed.status} when asked its projects` };+ }+ theirs = projectIdsIn(listed.text);+ } catch (err) {+ return { kind: "failed", detail: err instanceof Error ? err.message : String(err) };+ }++ const synced: string[] = [];+ const created: string[] = [];+ const refused: { projectId: string; detail: string }[] = [];+ for (const projectId of opts.projects) {+ const local = await opts.loadLocal(projectId);+ // Nothing configured here is not a reason to touch their table: a replace built from an+ // empty local list would delete every model they have. It is also not a reason to create+ // the Project — an empty shell is not worth a directory on someone's machine.+ if (local === null || local.models.length === 0) continue;+ const path = `/api/projects/${encodeURIComponent(projectId)}/models`;+ try {+ if (!theirs.includes(projectId)) {+ const made = await opts.api.request("POST", "/api/projects", {+ projectId,+ ...(local.name !== undefined && local.name !== "" ? { name: local.name } : {}),+ });+ if (made.status < 200 || made.status >= 300) {+ refused.push({+ projectId,+ detail: `it refused to create ${projectId}: ${made.status} ${made.text.slice(0, 200)}`,+ });+ continue;+ }+ created.push(projectId);+ }+ const current = await opts.api.request("GET", path);+ if (current.status !== 200) {+ refused.push({+ projectId,+ detail: `it answered ${current.status} when asked the models of ${projectId}`,+ });+ continue;+ }+ const plan = planModelSync(local, JSON.parse(current.text) as ModelsResponse);+ const wrote = await opts.api.request("PUT", path, plan);+ if (wrote.status < 200 || wrote.status >= 300) {+ refused.push({+ projectId,+ detail: `it refused the models of ${projectId}: ${wrote.status} ${wrote.text.slice(0, 200)}`,+ });+ continue;+ }+ synced.push(projectId);+ } catch (err) {+ // Not a refusal: the channel itself gave out. Nothing after it would fare better.+ return { kind: "failed", detail: err instanceof Error ? err.message : String(err) };+ }+ }+ return { kind: "synced", projects: synced, created, refused };+}diff --git a/packages/server/src/machines/service.ts b/packages/server/src/machines/service.tsindex 8aaa4571..4c50d77c 100644--- a/packages/server/src/machines/service.ts+++ b/packages/server/src/machines/service.ts@@ -25,7 +25,13 @@*/import fs from "node:fs";import os from "node:os";-import { DEFAULT_SERVER_PORT, VERSION } from "@prismshadow/penguin-core";+import {+ DEFAULT_PROJECT_ID,+ DEFAULT_SERVER_PORT,+ VERSION,+ loadProjectConfig,+} from "@prismshadow/penguin-core";+import type { ProjectConfig } from "@prismshadow/penguin-core";import type { MachineInfo, MachineJob, MachineServerStatus } from "../api/types.js";import { readServerLock } from "../lock.js";import { SESSION_COOKIE } from "../auth/middleware.js";@@ -45,6 +51,9 @@ import type { ExecResult } from "./transport/index.js";import { installOnRemote, resolvePushPlan } from "./install-server.js";import { probeServerState } from "./server-state.js";import { mintTokenOnRemote } from "./remote-token.js";+import { syncModelsToMachine } from "./models-sync.js";+import type { LocalModels } from "./models-sync.js";+import { machineApi } from "./machine-api.js";import { startRemoteServer } from "./server-control.js";import type { MachineRow, MachinesRepo } from "../db/repos/machines.js";@@ -74,6 +83,8 @@ export interface MachinesEffects {session: (address: string) => ShellSession | null;/** An http.Agent that dials that machine's server through its session. */agent: (target: RemoteTarget, remotePort: number) => http.Agent;+ /** This server's own Project config, credentials in plaintext — the source of a model sync. */+ loadConfig: (projectId: string) => Promise<ProjectConfig>;/** Injected so a test can pin the recorded timestamp instead of asserting around the clock. */now: () => Date;}@@ -104,6 +115,13 @@ export class MachinesService {readonly #busy = new Set<string>();/** Sessions minted on machines, by address, reused until near their TTL. */readonly #sessions = new Map<string, { cookie: string; at: number }>();+ /**+ * Machines with a model sync in flight, and whether another was asked for meanwhile. An+ * edit that arrives while a sync is running is not dropped and not queued behind it one+ * by one: ONE trailing sync runs when the current one ends, reading the config as it is+ * then — which is what every edit in between wanted written.+ */+ readonly #syncing = new Map<string, { again: boolean }>();/*** What the proxy's traffic last learned of each machine's API, by address. In memory* like #statuses: a sighting is only true for the moment it was taken.@@ -135,6 +153,7 @@ export class MachinesService {hold: (target) => connectionTo(target).hold(),session: (address) => sessionOf(address),agent: (target, remotePort) => connectionTo(target).agent(remotePort),+ loadConfig: (projectId) => loadProjectConfig(dataRoot, projectId),now: () => new Date(),...effects,};@@ -289,6 +308,23 @@ export class MachinesService {this.#setMember(projectId, address, false);}+ /** Every Project on this server that uses a given machine — who its credentials belong to. */+ #projectsUsing(address: string): string[] {+ let dirs: string[] = [];+ try {+ dirs = fs+ .readdirSync(this.dataRoot, { withFileTypes: true })+ .filter((entry) => entry.isDirectory())+ .map((entry) => entry.name);+ } catch {+ dirs = [];+ }+ // The default Project is always a candidate: every server seeds it, so it may be using+ // a machine before its directory has been created on this root.+ const candidates = dirs.includes(DEFAULT_PROJECT_ID) ? dirs : [...dirs, DEFAULT_PROJECT_ID];+ return candidates.filter((projectId) => this.#members(projectId).includes(address));+ }+/*** The version this server would install, or null when it has no image to push — a dev* checkout that has never been pushed to is the one shape with none. The page asks for it@@ -595,7 +631,7 @@ export class MachinesService {const address = machine.id;const target = this.#targetOf(machine.alias);- // Asked even when a forward is already up: the forward is an ssh process on THIS side,+ // Asked even when the connection is already up: it is an ssh process on THIS side,// and it outlives the far server. Taking it as the answer reported "connected" over a// dead server — and every caller that then found the machine silent asked for another// connect, which said "already connected" again, forever. Reconnecting (to retry a sync@@ -639,6 +675,9 @@ export class MachinesService {if (!connection.ok) return { ok: false, step: "connect", message: connection.detail };this.repo.patch(address, { remotePort });say(`Connected; its server is on port ${remotePort} over there.`);+ // An Agent started over there resolves its model against THAT machine's config, so a+ // machine without our credentials is connected and unusable.+ await this.#syncModels(address, target, remotePort, say, this.#projectsUsing(address));return { ok: true, connected: true };}@@ -676,6 +715,96 @@ export class MachinesService {closeAllConnections();}+ // --- work on a machine ---------------------------------------------------------------------++ /** Hands a machine the Model credentials an Agent over there needs. Failure is reported, never fatal. */+ async #syncModels(+ address: string,+ target: RemoteTarget,+ port: number,+ say: (line: string) => void,+ projects: string[],+ ): Promise<void> {+ if (projects.length === 0) {+ say("No models synced — no Project on this server uses that machine.");+ return;+ }+ const session = await this.#sessionOn(target);+ if (!("cookie" in session)) {+ say(`Models not synced — ${session.detail}`);+ return;+ }+ const outcome = await syncModelsToMachine({+ api: machineApi(this.#effects.agent(target, port), port, session.cookie),+ loadLocal: (projectId) => this.#localModels(projectId),+ projects,+ });+ if (outcome.kind === "failed") {+ say(`Models not synced — ${outcome.detail}`);+ return;+ }+ if (outcome.created.length > 0) say(`Created there: ${outcome.created.join(", ")}.`);+ if (outcome.projects.length > 0) say(`Models synced: ${outcome.projects.join(", ")}.`);+ for (const { projectId, detail } of outcome.refused) {+ say(`Models of ${projectId} not synced — ${detail}`);+ }+ void address;+ }++ /**+ * A sync of the given Projects to a machine, coalesced: while one runs, a second ask marks+ * it to run once more when it ends. The trailing run reads the config fresh, so however+ * many edits arrived during the first, the machine ends on the last of them. Nothing is+ * dropped, and a burst of edits costs two round trips rather than one per edit.+ */+ async #syncCoalesced(+ address: string,+ target: RemoteTarget,+ port: number,+ projects: () => string[],+ ): Promise<void> {+ const running = this.#syncing.get(address);+ if (running !== undefined) {+ running.again = true;+ return;+ }+ const slot = { again: false };+ this.#syncing.set(address, slot);+ try {+ do {+ slot.again = false;+ await this.#syncModels(address, target, port, () => {}, projects());+ } while (slot.again);+ } finally {+ this.#syncing.delete(address);+ }+ }++ /**+ * This side's model table for a Project, narrowed to the entries worth carrying: an entry+ * with an inline key or its own base URL is something the machine cannot already have.+ * Bare catalog entries are skipped — every server seeds the same presets.+ */+ async #localModels(projectId: string): Promise<LocalModels | null> {+ let config: ProjectConfig;+ try {+ config = await this.#effects.loadConfig(projectId);+ } catch {+ return null;+ }+ const models = config.models.filter(+ (entry) => (entry.api_key ?? "") !== "" || (entry.base_url ?? "") !== "",+ );+ return {+ models,+ ...(config.default_model !== undefined ? { defaultModel: config.default_model } : {}),+ ...(config.vision_model !== undefined ? { visionModel: config.vision_model } : {}),+ ...(config.name !== undefined ? { name: config.name } : {}),+ };+ }++ // --- the automatic sweeps, run when an App boots ----------------------------------------------+/** Runs `work` with this machine's slot held, or returns null when it is already busy. */async #withMachine<T>(address: string, work: () => Promise<T>): Promise<T | null> {if (this.#busy.has(address)) return null;@@ -723,4 +852,33 @@ export class MachinesService {},);}++ /** Brings every connected machine up to date with this server's Model config. */+ /**+ * Pushes a Project's models to every connected machine it uses — a key rotated here, a+ * default switched here, reaches them at once. Not through #sweep: that returns null for a+ * machine that is busy and would silently drop the edit; a sync in flight is instead told+ * to run once more (#syncCoalesced), and an install in flight on the machine is not a+ * reason to lose a credential either — the two do not contend for the shell.+ */+ async syncModelsEverywhere(projectId: string): Promise<void> {+ const connected = this.list(projectId).filter(+ (m) => !m.local && m.installed !== null && m.connection !== null,+ );+ await Promise.all(+ connected.map(async (machine) => {+ const address = machine.id;+ const port =+ this.#liveSession(address) === null ? null : (this.repo.get(address)?.remotePort ?? null);+ if (port === null) return;+ try {+ await this.#syncCoalesced(address, this.#targetOf(machine.alias), port, () => [+ projectId,+ ]);+ } catch {+ // Best effort: the connect log is where a sync's own words go; this one has none.+ }+ }),+ );+ }}diff --git a/packages/server/test/machines-api.test.ts b/packages/server/test/machines-api.test.tsindex 0b43248f..c650f393 100644--- a/packages/server/test/machines-api.test.ts+++ b/packages/server/test/machines-api.test.ts@@ -10,6 +10,7 @@*/import fs from "node:fs";import http from "node:http";+import type { AddressInfo } from "node:net";import path from "node:path";import { afterEach, beforeEach, describe, expect, it } from "vitest";import type { MachinesResponse } from "../src/api/types.js";@@ -728,6 +729,63 @@ describe("machines API", () => {});});+ describe("syncing models outward", () => {+ /** A machine's server: counts what it is asked, and can be made slow. */+ const machineServer = async (opts: { delayMs: number }) => {+ const asked: string[] = [];+ const server = http.createServer((req, res) => {+ asked.push(`${req.method} ${req.url}`);+ setTimeout(() => {+ res.writeHead(200, { "content-type": "application/json" });+ res.end(+ req.url === "/api/projects"+ ? JSON.stringify({ projects: [{ projectId: PROJECT }] })+ : JSON.stringify({ models: [] }),+ );+ }, opts.delayMs);+ });+ const port = await new Promise<number>((resolve) =>+ server.listen(0, "127.0.0.1", () => resolve((server.address() as AddressInfo).port)),+ );+ return { asked, port, close: () => server.close() };+ };++ it("an edit that lands while a sync is in flight is written afterwards, once — not dropped", async () => {+ const remote = await machineServer({ delayMs: 60 });+ try {+ await boot({+ loadConfig: async () =>+ ({+ models: [{ provider: "deepseek", model_id: "deepseek-v4-flash", api_key: "sk-x" }],+ }) as never,+ });+ machinesRepo.patch("ssh:nas", {+ version: "9.9.9",+ installedAt: "2026-08-01T00:00:00.000Z",+ remotePort: remote.port,+ });+ machinesRepo.setMembers(PROJECT, ["ssh:nas"]);+ connected.add("ssh:nas");++ // Three edits in a burst: the first starts a sync, the next two land during it.+ const first = t.deps.machines.syncModelsEverywhere(PROJECT);+ await new Promise((resolve) => setTimeout(resolve, 10));+ const second = t.deps.machines.syncModelsEverywhere(PROJECT);+ const third = t.deps.machines.syncModelsEverywhere(PROJECT);+ await Promise.all([first, second, third]);+ // Give the trailing run its own round trips.+ await waitFor(() => remote.asked.filter((a) => a.startsWith("PUT")).length >= 2);++ const puts = remote.asked.filter((a) => a.startsWith("PUT"));+ // Exactly two writes: the one in flight, and ONE trailing run for everything that+ // arrived during it — the config is read fresh then, so the last edit is what lands.+ expect(puts).toHaveLength(2);+ } finally {+ remote.close();+ }+ });+ });+describe("browsing a machine's directories", () => {const ID = "LNrJdHAZJ91G58i0";/** A machine this server holds a connection to, whose id a probe has heard. */diff --git a/packages/server/test/machines-models-sync.test.ts b/packages/server/test/machines-models-sync.test.tsnew file mode 100644index 00000000..97788ab7--- /dev/null+++ b/packages/server/test/machines-models-sync.test.ts@@ -0,0 +1,353 @@+/**+ * Giving a machine the Model credentials its Agents need (machines/models-sync.ts).+ *+ * The whole feature rests on one asymmetry of the models endpoint: a PUT replaces the whole+ * table, and an entry sent WITHOUT `apiKey` keeps the key already stored. Every test that+ * matters here is about that — a merge that forgets to re-send a machine's own entries+ * deletes them, and one that re-sends the masked key it just read overwrites a working+ * credential with `sk-1…abcd`. Neither failure says anything at the time; both are found+ * later, by the machine, as an auth error.+ *+ * The machine's API is faked. What travels between the two servers is HTTP through a tunnel+ * that already exists, so there is nothing ssh-shaped left to exercise at this level.+ */+import http from "node:http";+import type { AddressInfo } from "node:net";+import { describe, expect, it } from "vitest";+import { serve } from "@hono/node-server";+import type { ModelEntry } from "@prismshadow/penguin-core";+import type { ModelsResponse, ModelsUpdateRequest } from "../src/api/types.js";+import { planModelSync, syncModelsToMachine } from "../src/machines/models-sync.js";+import type { LocalModels } from "../src/machines/models-sync.js";+import { machineApi } from "../src/machines/machine-api.js";+import type { MachineApi } from "../src/machines/machine-api.js";+import { createTestApp } from "./helpers.js";++/**+ * One POST to the machine under test. node:http with an explicit Host header, for the reason+ * machineApi gives: the API answers only under the canonical app host while the connection+ * goes to 127.0.0.1, and fetch will not let that header be set.+ */+function post(+ port: number,+ path: string,+ body: unknown,+): Promise<{ status: number; setCookie: string[] }> {+ return new Promise((resolve, reject) => {+ const payload = Buffer.from(JSON.stringify(body));+ const req = http.request(+ {+ host: "127.0.0.1",+ port,+ path,+ method: "POST",+ headers: {+ host: `localhost:${port}`,+ "content-type": "application/json",+ "content-length": String(payload.length),+ },+ },+ (res) => {+ res.resume();+ res.on("end", () =>+ resolve({ status: res.statusCode ?? 0, setCookie: res.headers["set-cookie"] ?? [] }),+ );+ },+ );+ req.on("error", reject);+ req.end(payload);+ });+}++/**+ * Serves an app on a free port. Port 0: the OS picks one, so this never has to guess on a+ * shared machine — and the address is only known once it is listening, which serve() reports+ * through its callback rather than from a synchronous address().+ */+function listening(+ fetch: Parameters<typeof serve>[0]["fetch"],+): Promise<{ server: ReturnType<typeof serve>; port: number }> {+ return new Promise((resolve) => {+ const started = serve({ fetch, port: 0, hostname: "127.0.0.1" }, (info) =>+ resolve({ server: started, port: (info as AddressInfo).port }),+ );+ });+}++const local = (over: Partial<ModelEntry> = {}): ModelEntry => ({+ provider: "deepseek",+ model_id: "deepseek-v4-flash",+ api_key: "sk-local-0123456789",+ ...over,+});++/** What that machine's GET answers: keys masked, exactly as the real endpoint reports them. */+const remoteTable = (over: Partial<ModelsResponse> = {}): ModelsResponse => ({+ models: [],+ ...over,+});++describe("planModelSync", () => {+ it("sends our entry with its key in plaintext", () => {+ const plan = planModelSync({ models: [local()] }, remoteTable());+ expect(plan.models).toEqual([+ { provider: "deepseek", modelId: "deepseek-v4-flash", apiKey: "sk-local-0123456789" },+ ]);+ });++ it("re-sends a machine's own entry, and never its masked key", () => {+ const plan = planModelSync(+ { models: [local()] },+ remoteTable({+ models: [+ {+ provider: "anthropic",+ modelId: "claude-opus-5",+ isDefault: false,+ contextWindow: 200000,+ credential: { apiKeyMasked: "sk-a…wxyz", baseUrl: "https://theirs.example" },+ },+ ],+ }),+ );+ const theirs = plan.models.find((m) => m.modelId === "claude-opus-5");+ expect(theirs).toBeDefined();+ expect(theirs?.contextWindow).toBe(200000);+ expect(theirs?.baseUrl).toBe("https://theirs.example");+ expect(theirs).not.toHaveProperty("apiKey");+ expect(JSON.stringify(plan)).not.toContain("…");+ });++ it("replaces a machine's entry for the same pair with ours", () => {+ const plan = planModelSync(+ { models: [local({ api_key: "sk-newer-0123456789" })] },+ remoteTable({+ models: [+ {+ provider: "deepseek",+ modelId: "deepseek-v4-flash",+ isDefault: true,+ credential: { apiKeyMasked: "sk-o…lder" },+ },+ ],+ }),+ );+ expect(plan.models).toHaveLength(1);+ expect(plan.models[0]?.apiKey).toBe("sk-newer-0123456789");+ });++ it("omits a key we do not have, rather than clearing theirs", () => {+ const plan = planModelSync(+ { models: [local({ api_key: undefined, base_url: "https://gw.example" })] },+ remoteTable(),+ );+ expect(plan.models[0]).not.toHaveProperty("apiKey");+ expect(plan.models[0]).not.toHaveProperty("clearApiKey");+ expect(plan.models[0]?.baseUrl).toBe("https://gw.example");+ });++ it("points the machine's default at ours, even when it already had one", () => {+ const mine: LocalModels = {+ models: [local()],+ defaultModel: { provider: "deepseek", model_id: "deepseek-v4-flash" },+ };+ const theirs = { provider: "anthropic", modelId: "claude-opus-5" };+ const ours = { provider: "deepseek", modelId: "deepseek-v4-flash" };+ expect(planModelSync(mine, remoteTable({ defaultModel: theirs })).defaultModel).toEqual(ours);+ expect(planModelSync(mine, remoteTable()).defaultModel).toEqual(ours);+ });++ it("does not name a pointer that is not in the table it sends", () => {+ const plan = planModelSync(+ {+ models: [local()],+ defaultModel: { provider: "openai", model_id: "gpt-5" },+ visionModel: { provider: "openai", model_id: "gpt-5" },+ },+ remoteTable(),+ );+ expect(plan.defaultModel).toBeUndefined();+ expect(plan.visionModel).toBeUndefined();+ });+});++/** Records every call, answering GETs from a table keyed by path. */+function fakeMachine(answers: Record<string, { status: number; body: unknown }>): {+ api: MachineApi;+ calls: Array<{ method: string; path: string; body?: unknown }>;+ puts: Array<{ path: string; body: ModelsUpdateRequest }>;+} {+ const calls: Array<{ method: string; path: string; body?: unknown }> = [];+ const puts: Array<{ path: string; body: ModelsUpdateRequest }> = [];+ return {+ calls,+ puts,+ api: {+ postBytes: async () => ({ status: 500, text: "not used by the model sync" }),+ request: async (method, path, body) => {+ calls.push({ method, path, ...(body === undefined ? {} : { body }) });+ if (method === "PUT") {+ puts.push({ path, body: body as ModelsUpdateRequest });+ return { status: 200, text: "{}" };+ }+ // A method-qualified key wins, so a POST can be scripted apart from the GET of one path.+ const answer = answers[`${method} ${path}`] ?? answers[path];+ if (answer === undefined) return { status: 404, text: "{}" };+ return { status: answer.status, text: JSON.stringify(answer.body) };+ },+ },+ };+}++describe("syncModelsToMachine, against a scripted machine", () => {+ it("a Project the machine refuses is reported and skipped; the ones after it are still written", async () => {+ const machine = fakeMachine({+ "/api/projects": { status: 200, body: { projects: [{ projectId: "default_project" }] } },+ "/api/projects/default_project/models": { status: 200, body: remoteTable() },+ "/api/projects/alice-lab/models": { status: 200, body: remoteTable() },+ // The machine's own rule: an admin-created id may not carry the namespace hyphen.+ "POST /api/projects": { status: 400, body: { error: { code: "invalid_project_id" } } },+ });+ const outcome = await syncModelsToMachine({+ api: machine.api,+ projects: ["alice-lab", "default_project"],+ loadLocal: async () => ({ models: [local()] }),+ });+ expect(outcome).toEqual({+ kind: "synced",+ projects: ["default_project"],+ created: [],+ refused: [+ { projectId: "alice-lab", detail: expect.stringContaining("refused to create alice-lab") },+ ],+ });+ expect(machine.puts.map((p) => p.path)).toEqual(["/api/projects/default_project/models"]);+ });++ it("the channel giving out fails the sync as a whole — nothing after it would fare better", async () => {+ const api: MachineApi = {+ postBytes: async () => ({ status: 500, text: "" }),+ request: async (method, path) => {+ if (path === "/api/projects") {+ return { status: 200, text: JSON.stringify({ projects: [{ projectId: "a" }] }) };+ }+ if (method === "GET") throw new Error("the machine's server closed mid-answer");+ return { status: 200, text: "{}" };+ },+ };+ const outcome = await syncModelsToMachine({+ api,+ projects: ["a", "b"],+ loadLocal: async () => ({ models: [local()] }),+ });+ expect(outcome).toEqual({ kind: "failed", detail: "the machine's server closed mid-answer" });+ });+});++describe("machineApi", () => {+ it("rejects, rather than hangs, when the machine closes the answer before its body", async () => {+ // Headers, part of a body, then the socket closes: `end` never fires, `close` does.+ const server = http.createServer((_req, res) => {+ res.writeHead(200, { "content-type": "application/json", "content-length": "100" });+ res.write('{"partial":');+ setTimeout(() => res.socket?.destroy(), 20);+ });+ const port = await new Promise<number>((resolve) =>+ server.listen(0, "127.0.0.1", () => resolve((server.address() as AddressInfo).port)),+ );+ try {+ const api = machineApi(new http.Agent(), port, "penguin_session=x");+ await expect(api.request("GET", "/api/projects")).rejects.toThrow(+ /closed mid-answer|socket hang up|aborted/,+ );+ } finally {+ server.close();+ }+ });+});++describe("syncModelsToMachine, against a running server", () => {+ it("lands our key on it and leaves its own entry intact", async () => {+ const machine = await createTestApp();+ const { server, port } = await listening(machine.app.fetch);+ try {+ await machine.deps.projectConfigService.updateModels("default_project", {+ models: [+ { provider: "anthropic", modelId: "claude-opus-5", apiKey: "sk-theirs-9876543210" },+ ],+ });++ const login = await post(port, "/api/auth/login", {+ userId: "admin",+ password: machine.adminPassword,+ });+ expect(login.status).toBe(200);+ const cookie = login.setCookie.map((line) => line.split(";")[0]?.trim() ?? "").join("; ");++ const outcome = await syncModelsToMachine({+ api: machineApi(new http.Agent(), port, cookie),+ projects: ["default_project"],+ loadLocal: async (projectId) =>+ projectId === "default_project"+ ? { models: [local({ api_key: "sk-ours-0123456789" })] }+ : null,+ });+ expect(outcome).toEqual({+ kind: "synced",+ projects: ["default_project"],+ created: [],+ refused: [],+ });++ const config = await machine.deps.projectConfigService.loadConfig("default_project");+ const ours = config.models.find((m) => m.model_id === "deepseek-v4-flash");+ const theirs = config.models.find((m) => m.model_id === "claude-opus-5");+ expect(ours?.api_key).toBe("sk-ours-0123456789");+ expect(theirs?.api_key).toBe("sk-theirs-9876543210");+ } finally {+ server.close();+ await machine.cleanup();+ }+ });++ it("creates the Project over there under the same id, and it works", async () => {+ const machine = await createTestApp();+ const { server, port } = await listening(machine.app.fetch);+ try {+ const login = await post(port, "/api/auth/login", {+ userId: "admin",+ password: machine.adminPassword,+ });+ const cookie = login.setCookie.map((line) => line.split(";")[0]?.trim() ?? "").join("; ");++ const outcome = await syncModelsToMachine({+ api: machineApi(new http.Agent(), port, cookie),+ projects: ["field_work"],+ loadLocal: async () => ({+ models: [local({ api_key: "sk-ours-0123456789" })],+ defaultModel: { provider: "deepseek", model_id: "deepseek-v4-flash" },+ name: "Field work",+ }),+ });+ expect(outcome).toEqual({+ kind: "synced",+ projects: ["field_work"],+ created: ["field_work"],+ refused: [],+ });++ const listed = await machine.deps.projectService.listProjects("admin");+ expect(listed.find((entry) => entry.projectId === "field_work")?.name).toBe("Field work");+ expect((await machine.deps.agentService.listAgents("field_work")).length).toBeGreaterThan(0);++ const config = await machine.deps.projectConfigService.loadConfig("field_work");+ expect(config.default_model).toEqual({ provider: "deepseek", model_id: "deepseek-v4-flash" });+ expect(config.models.find((m) => m.model_id === "deepseek-v4-flash")?.api_key).toBe(+ "sk-ours-0123456789",+ );+ } finally {+ server.close();+ await machine.cleanup();+ }+ });+});diff --git a/packages/server/test/machines-transport-session.test.ts b/packages/server/test/machines-transport-session.test.tsindex 0b47d7f3..2edd79a6 100644--- a/packages/server/test/machines-transport-session.test.ts+++ b/packages/server/test/machines-transport-session.test.ts@@ -123,9 +123,11 @@ exit 1const second = sessionOf("ssh:nas");expect(second).not.toBeNull();expect(second!.pid).not.toBe(first);- expect(spawns()).toHaveLength(2);- // And it is a working session, still held.+ // And it is a working session, still held. The spawn count is read only after a command+ // has completed over it: the stub logs its own invocation from inside the child, after+ // spawn() has already returned a pid, so a session can exist before its line is there.expect(await conn.exec("echo back")).toMatchObject({ code: 0, stdout: "back\n" });+ expect(spawns()).toHaveLength(2);expect(conn.held()).toBe(true);// An explicit close lets go for good: nothing reopens it.