Self-hosted, Self-Improving, Federated Everything App — dev management, agent orchestration, persistent mind, digital twin, second brain, and creative pipeline from a single dashboard over Tailscale to port the user into the OS
Server conventions (server/)
These apply to Express/service/model code. Universal constraints (no try/catch outside the request lifecycle, functional programming, single-line logging, the Security Model, the Storage backend policy, the AI Provider Usage Policy, and the Sensitive Data & Privacy rules) live in the root AGENTS.md and always apply.
Validation and schema parity
Schema parity when adding fields. When you add a field to a sanitizer,
createXxx, or a payload shape, update the corresponding Zod schema (server/lib/aiToolkit/validation.jsfor toolkit shapes,server/lib/validation.jsfor PortOS routes) in the same change. Wire validation into POST and PUT (PUT can useschema.partial()); the PortOS convention is all inputs validated — an exported-but-unwired sub-schema creates false confidence. For polymorphic stores likePUT /api/settingsthat accept any sub-object, validate the relevant slice withschema.partial()when its key is present in the request body (seeserver/routes/settings.jsfor thebackupConfigSchema.partial()pattern). Tolerate UI sentinels (endpoint: ''for CLI providers) withz.preprocess(v => v === '' ? undefined : v, …). When a service migrates legacy keys on read, the schema must still accept the legacy shape so older clients don't 400 before the migration runs.A POST/PUT/PATCH handler that reads
req.bodymust validate it with a schema.validateRequest(schema, req.body ?? {})is the default; put the schema inserver/lib/validation.js(or the domain's*Validation.js). A hand-rolledif (!path) throw …lets a wrong-typed field reach the service layer.server/routeValidationConventions.test.jsscansserver/routes/, the toolkit'slib/aiToolkit/routes/(schemas inlib/aiToolkit/validation.js, which cannot importlib/validation.js) and the CoS runner'scos-runner/index.js(requestSchemas.js, answering its own{ error }400) for this; itsALLOWED_UNVALIDATEDlist (raw-binary and multipart-upload bodies, each with a reason) may only shrink — the test fails when an entry is compliant or gone. Do not add a JSON-body handler to it (#9950).A handler that reads
req.querymust validate the query object with a schema. Same spellings and same trees, on GET/POST/PUT/PATCH/DELETE:validateRequest(schema, req.query)(orvalidate/parseBody/safeParse/Schema.parse), and the data argument must be the query object or a name assigned from it — not one field, and notreq.params.parsePagination(req.query)only clamps limit/offset, so it does not satisfy the guard.UNVALIDATED_QUERY_BASELINEin that same test is the burn-down of handlers that already skip this; it may only shrink. Delete a line when the handler validates or disappears, and do not add one — a new reader gets a schema (#10024).
Offering models vs running them
- A caller that OFFERS a model choice reads providers through
getSelectableProviders()/listSelectableProviders(); a caller that EXECUTES, ACCOUNTS or WRITES readsgetAllProviders()/listProviders(). The scoped pair narrows each record'smodelsto the install's declared entitlement — which of a vendor's advertised catalog this account can actually run, a thing no/modelsresponse reports (seedocs/MODEL_ACCESS.mdandlib/aiToolkit/internal/modelAccess.js). Picking the wrong one is silent both ways: a picker on the unscoped list offers models the account cannot run, and a write path on the scoped list persists a truncated catalog over the real one, which no refresh restores until the policy is cleared. Two methods rather than a flag, because the answer is a property of the caller's purpose and has to be visible at the call site. Route payloads shaped bypresentProviderare already scoped; anything reading the service directly is not.
Write serialization and throughput
- Async PATCH races on shared records — serialize writes server-side. When two write paths can mutate the same record concurrently (e.g. a blur-save plus an explicit "Render" button against
stages.comicPages.cover), client-side guards (refs,onMouseDown, status checks) are unreliable — keyboard activation, status==='unknown' stalls, andloadState → modify → saveStateinterleaving all defeat them. The correct fix is to serialize writes at the file level on the server (every PATCH awaits the previous one to settle before reading state, so it merges against the freshest persisted record). SeequeueSeriesIssuesWriteinserver/services/pipeline/issuesShared.js(built oncreateKeyCachedQueueinserver/lib/createKeyCachedQueue.js). AMap<recordId, Promise>is not enough — two writes to different record ids share the same JSON state file and can still clobber each other; collapse the queue to a single tail per shared file. - High-frequency state writes must batch. Per-line state mutations that round-trip through
withStateLock → loadState → saveState(e.g.appendAgentOutputinserver/services/cosAgentLifecycle.js) are fine for human-pace events but catastrophic when called from a hot loop — PTY output streams, AI tool-call streams, or any producer that can emit dozens of events per second. When wiring a new streaming producer, add a batched variant that takes an array (seeappendAgentOutputLinesfor the pattern) and flush from the caller on a ~250ms debounce. Always drain the pending buffer in the producer'sfinish/cleanuppath before the final state write so completion events don't beat the last output batch to disk.
PostgreSQL parameters
- A parameter bound to a
json/jsonbcolumn or::jsonbcast is serialized withtoPgJson, never bareJSON.stringify. PostgreSQL refuses a lone UTF-16 surrogate (an emoji.slice()d in half) and NUL insidejsonb, and NUL intext; every file store the migrated records came from accepted both, so an unrepaired write throws and the record is lost (a promoted CoS task, an automatic investigation task, a legacy import retained for retry forever, #12815).toPgJsonandnormalizePgStringlive inserver/lib/pgJson.jsand repair a bad unit to U+FFFD, the same lossy fix a UTF-8 file write applied.lib/db.jsappliesnormalizePgParamsto everyquery()andwithTransactionclient call (strings, array elements and plain-object parameters; it never unescapes a string, so atextvalue holding the characters\^@is stored as written), but a pre-stringified JSON string looks like ordinary text to it, so the call site still usestoPgJson.server/pgJsonConventions.test.jsscans the tree for this and carries no allowlist; appliedscripts/db-migrations/are frozen and skipped.
Timers
- An
asynctimer callback must own its rejections. InsidesetTimeout(async …)/setInterval(async …), everyawaitneeds either an enclosingtry { … } catchor a.catch(…)on the awaited chain. A timer fires outside the request lifecycle, so nothing holds the promise the callback returns and a rejection escapes as an unhandled rejection — fatal on Node >= 15. The CoS runner shipped that at boot: its orphan-cleanup timer awaited a state-file read/write, so one truncatedagentsfile killed the runner seconds after start and PM2 restart-looped it against the same file (#5668). Log and continue in the catch unless the callback genuinely owns a failure verdict.server/timerCallbackConventions.test.jsscans the whole server tree for this and carries no allowlist; the worked example is the watchdog inserver/services/mediaJobQueue/index.js. - A statement-level call to an
asyncfunction must beawaited, end in.catch(…), or start withvoid. A bareannounceSelf(peer);leaves a rejection with no owner, which becomes anunhandledRejectionthatautoFixerturns into an AI task for what was best-effort bookkeeping (#12978).voidmeans "this function owns its rejections, fire-and-forget on purpose" — make that true first.server/floatingAsyncConventions.test.jsresolves module-scope async functions (local and relatively imported) across the server tree and carries no allowlist.
Child processes
- Attach a spawned child's terminal listener before the function next
awaits. Node reports a failed spawn (ENOENT, EACCES, EMFILE) onprocess.nextTickand a fast-dying child exits while you are suspended: an unlistenederrorthrows and takes the process down, an unlistenedclosestrands the jobrunning. Do the awaited work (a scanner read, a claim handoff) BEFOREspawn(), or callbufferChildExit(child)(services/videoGen/generateVideoHelpers.js) in the same tick and replay its result once the real handlers are wired (services/imageGen/local.js). The CoS Runner/spawnand/spawn-tuiroutes and the local image render each shipped this ordering bug (#11761, after #4617/#5655/#5791).server/childListenerOrderConventions.test.jsscans the whole server tree for it and carries no allowlist.
Socket handlers
- An
asyncSocket.IO handler must own its rejections. Same rule as the timer callbacks above, same reason: Socket.IO hands the promise a listener returns to nobody, so a rejection escapes as an unhandled rejection and Node terminates the process — taking every agent run, PTY session and media job with it. Everyawaitin anasynchandler underserver/sockets/needs an enclosingtry { … } catch(the shapevoice.jsuses on whole handler bodies) or a.catch(…)on the awaited chain (the shapelogs.jsuses on the lookups it falls back from).server/sockets/asyncHandlerGuard.test.jsscans the directory for this and carries no allowlist; it names the file and the event string of any offender. Precedent:voice:call:detachshipped unguarded (#5661).
Tests and peer fan-out
- Record-creating tests and peer fan-out. Record create/update paths reach peer-sync only through the subscription adapter in
server/services/sharing/recordEvents.js(autoSubscribeRecordToAllPeersetc.) — a silent no-op untilpeerSync.jsregisters the real implementation at module load. A suite that never loadspeerSync.jstherefore gets no fan-out without mocking anything. Suites that DO load the peer-sync graph (importingpeerSync.jsorsharing/index.js, directly or transitively) must still mockservices/instances.jswithmockNoPeers()fromserver/lib/mockPathsDataRoot.jsandservices/sharing/peerSync.jswithmockNoPeerSync()— the mock keeps the real module's registration side effect from wiring live fan-out into the adapter. To assert auto-subscribe behavior in a domain test, register a test double viaregisterSubscriptionAdapter({...})and clean up with__resetSubscriptionAdapter().
Import scoping
- A widely-reached module must not drag a subtree it only needs a constant from. The Linux CI server suite spends more wall time importing modules than running assertions, because import cost multiplies: every test file that transitively reaches module
Xre-instantiates all ofX's closure in its own worker. So one convenient barrel import inside a module 400 suites reach is worth thousands of module instantiations.server/lib/importScoping.test.jspins the narrowings made in #6009 (which cut the suite from ~115.5k static module instantiations to ~94.5k) as negative reachability assertions overstaticImportClosure; add a row there when you narrow another one. The two shapes worth looking for: a constant re-exported through a heavy barrel (import it from the module that declares it —CHECK_SCOPESfromserver/lib/editorial/checkInfra/taxonomy.js, not fromeditorial/checkRegistry.jsand its 13 check-definition modules), and a dependency only a boot/rare path uses (await import()it inside that path —ensureSchemaImpl()composes the DDL that way, keepinglib/db/schema/**out of the ~400 suites that reachlib/db.jsthroughpgFileFacade.js). - A lazy
import()cuts instantiation cost, not CI selection.vitest relatedfollows dynamic imports as well as static ones, so a module that lazily imports a heavy service still links every suite reaching it to every suite reaching that service. In 2026-10 one dynamically connected component aroundservices/cos.jsmeant a change to any module in it selected ~1,130 server suites, although the static reach ofcos.jswas 90 (#12285). To cut selection, move the piece the caller actually needs into a leaf module both sides import (services/readinessSignal.js), or have the heavy service register a callback at boot that the caller looks up (initAutoFixer()→services/providerRecoveryHooks.js, read bypromptRunner.js). Measure withnode scripts/lib/importReach.js <repo-relative source>; the CI plan's server reason prints the same dynamic-aware fan-out for the changed sources. Cut edges are pinned inSELECTION_CUTSinserver/lib/importScoping.test.js. A registry has the mock caveat in reverse: a suite thatvi.mocked the heavy module must register test doubles instead. - In a non-async scope, defer with
.then()rather than giving up. The lazy shape is not limited toasyncfunctions: anew Promiseexecutor or a fire-and-forget branch takesimport('./x.js').then(({ fn }) => fn(…)).catch(handler), which also routes an import failure into the error path the call already had. That is how the TUI branch inservices/promptRunner.jsdeferstuiPromptRunner.js(and with itnode-pty). Prefer putting theimport()INSIDE an existing predicate gate when there is one —runner.jsdefersollamaAgentContext.jsinside theisOllamaBackedProviderternary, so a cloud-provider run pays neither the hop nor the subtree. - The tree-wide backstop is a base-relative growth guard, not an absolute ceiling.
importScoping.test.jscompares every server*.test.jsfile's static import closure against the merge base, using one analyzer for both sides (scripts/lib/importGrowth.js). The count is a proxy: closure size summed across suites, shared modules counted once per suite that reaches them, not a count of runtime instantiations or a wall-clock. New test files are reported as new-suite contribution and do not spend the allowance, so ordinary additions cannot fail the guard just because the repository total grew. The gated number is the sum, over suites that exist on both sides, of how much each closure grew (shrinkage elsewhere does not cancel it). The limit is 2,500, above the old ceiling's ~1,500 ordinary-growth note and below half of the smaller heavy regression this file already records: #6009 removed 19,286 (115,519 → 96,233) and #6156 removed 12,794 (96,233 → 83,439). A failure names the largest suite increases and the largest newly reached dependency closures. Narrow the import, or defer it withawait import()on the path that uses it. RaisingEXISTING_SUITE_GROWTH_LIMITis the intentional-regression path: do it in the same change, name the module that cannot be deferred, and keep the limit under that #6156 band — the pin test rejects a limit high enough to admit those edges. A missing or unreadable base fails the guard; it is not treated as zero growth.benchmarkProxySplitreports synthetic collection time and execution time separately and neither is a per-PR threshold. CI andnpm run pregaterun this comparison: the test is onALWAYS_RUN_TESTS, and pregate forwards the sameCI_BASE_SHAthe planner used. - Narrowing an import must not step around a barrel a suite mocks.
vi.mock('./cos.js')intercepts the specifier, not the function — so re-pointing a caller from./cos.jsto the./cosTaskStore.jsthat declaresaddTaskmakes production code reach the REAL implementation while the suite still believes it is doubled. The failure lands in an unrelated test file with no obvious link to the import you changed. Before narrowing, grep the tree forvi.mockagainst the module you are leaving; if any suite that reaches the caller mocks it, leave the import on the barrel. A lazyawait import()has no such hazard — vitest's mock registry covers dynamic imports too.
Generated manifests
- Derive at first use before committing a manifest. A source-derived inventory that is deterministic, cheap, and reads only files every install ships belongs in memory behind a per-process cache, not in git:
server/lib/apiRouteGraph.jsscans the mounted route graph in ~50 ms on the first/api/api-docs/*request, andsocketEventInventory.jsdoes the same for Socket.IO events. The route catalog used to be a committedapiRouteCatalog.generated.json(retired — it no longer exists in the tree); even content-keyed, itsstatscounts conflicted on every pair of branches that each added a route,maincarried a stalestats.declarationsover a byte-identical route graph, and it took 14 regeneration-only commits in 90 days. A committed copy is justified only when derivation is slow, nondeterministic, or needs inputs an install lacks (promptStageCallSites.generated.jsonreads everygit ls-files-trackedserver/source once per stage key, which no request handler should pay for). - A checked-in generated manifest is addressed by CONTENT, never by POSITION.
server/lib/*.generated.jsonfiles are derived from source and kept honest by a drift test that regenerates and compares. That only works if the manifest changes when the thing it describes changes — and only then. A record that says "declared atfoo.js:412" is rewritten by any edit that inserts a line above 412 (a rename, a comment, an unrelated handler), so the drift test fires on commits that change no route/stage/event at all, each parallel branch regenerates the file differently, and every rebase conflicts on it. - Key each record by the declaring file plus the semantic identity of what it declares — the stage key plus file paths for
promptStageCallSites.generated.json. Positional detail may stay in memory: the route-graph coverage guard inserver/lib/apiRouteGraph.test.jscompares two fresh in-memory scans (scanRouteGraph()vsparseRouteModule(), keyedfile#routerId METHOD /pathbyrouteDeclarationKey), so nothing committed ever points back into the source it was derived from. - Prove it with a property test, not a name list.
shiftSourceText(scripts/lib/positionInvariance.js) prepends a comment header to each source the generator reads; regenerate from the shifted sources and demand byte-identical output — asserting the rule directly, so it catches a position recorded asat,span,row, aloc: [412, 8]tuple, or afoo.js#L412anchor. The prompt-stage generator carries that test.server/lib/generatedManifests.test.jsis the cheap tree-wide backstop: it scans every git-tracked*.generated.jsonforline/column/offset-style keys andpath/to/file.js:412strings, catching a manifest whose generator never adopted the property test. It matches key names, so it is a net, not a proof — write both. That same file also enforces adoption structurally: it lists every git-trackedscripts/*.jsthat writes a.generated.jsonand fails when one has no sibling.test.jsimportingpositionInvariance.js, so a new generator cannot ship without the property test.
Prompt templates
- Stage-prompt template changes need a migration.
scripts/setup-data.jsonly copies missing prompt files todata/prompts/stages/— existing installs keep their old templates. When you add a{{template.variable}}reference to adata.reference/prompts/stages/*.mdfile, also add an entry toscripts/migrations/NNN-…jsthat updates the corresponding installed prompt when its hash still matches the pre-change shipped version. Normalize line endings (\r\n→\n, bare\r→\n) before hashing so the comparison is correct on Windows checkouts. Build it withmakePromptReplaceMigrationand exportACCEPTED_OLD_MD5/NEW_SHIPPED_MD5— never hand-mirror a hash table intoscripts/setup-data.js, which sweeps those exports itself viabuildPromptDriftTables. Worked example:scripts/migrations/394-factual-extraction-lens-prompts.js. AmakeSeedMigrationonly copies MISSING files and will NOT upgrade an installed template. Re-point every earlier migration naming the same file at the new hash (_shippedHashDrift.test.jsfails otherwise) and keep the superseded hash in the new migration's accepted-old set. Migration scripts live inscripts/migrations/; the per-install applied-list isdata/migrations.applied.json(gitignored runtime state). Seescripts/migrations/003-update-pipeline-stage-prompts.js. - A new migration is a frozen snapshot of its transform. Applied migrations are never edited; a new one (numbered above the current head) imports
_lib.jsprimitives,server/libpure helpers, or carries a frozen copy of the transform — neverserver/services, which an install upgrading months later would run under today's semantics. It declares no local copy of a_lib.jshelper (readJsonOrNull,writeJsonAtomic,writeTextAtomic, the dashboard-grid helpers, …), and a purge migration writes itspurge: truemarker as a top-level property of the default export soscripts/run-migrations.jscan read it from source without importing the module.scripts/migrations/migrationConventions.guard.test.jsenforces all three. - A
PROMPT_VERSIONSbump for a scheduled-task prompt (server/services/taskPromptDefaults.js) needs NO migration — its second step isnode scripts/regen-prompt-integrity-snapshot.js, which retires the outgoing hash (rootAGENTS.md"Distribution model").readSchedule()inserver/services/taskScheduleStore.jsalready upgrades every uncustomized stored prompt to the shipped default on the nextloadSchedule()and persists the result — the first schedule read after the update converges the install regardless of whether a migration ran. Six migrations re-implemented that same loop against atask-schedule.jsonat the data root — a path the store never reads;readSchedule()readsdata/cos/task-schedule.json— before this was written down:281-reconcile-missing-releases-prompt.js,293-agents-md-prompt-rename.js,297-release-check-slashdo-review-prompt.js,308-release-check-advisory-review-prompt.js,332-release-check-fix-blockers-prompt.js,338-claim-issue-volunteer-claim-contract.js. They stay — the distribution model forbids deleting applied migrations — but they are closed precedent, not a pattern to repeat.scripts/migrations/promptBumpMigrations.guard.test.jsfails if a seventh copy importstaskPromptDefaults.jsand writestask-schedule.json. - Creative prompts carry the IP-latitude clause automatically — classify the stage or source rather than wrapping a call site. PortOS is a professional artist's tool; a model that silently renames, genericizes, or disclaimers a deliberate reference to existing IP produces defective output, and rights/clearance is the operator's responsibility, not the model's.
server/lib/creativeLatitude.jsowns the clause and the single table that classifies what gets it, applied at each of the three LLM transports:promptService.buildPrompt(stage templates, keyed on the stage name — fails SAFE, so a stage the tables never saw is stamped),promptRunner.runPromptThroughProvider(every hand-rolled prompt, keyed on the runsourcetag it already requires — an allowlist, since ops calls dominate there), andaiProvider.callProviderAISimple(its own fetch, so creative callers passcreative: true). So a new creative stage or LLM-calling service is covered only once its stage name /sourceis classified in that module — add the prefix or exact name there rather than wrapping a call site. Two guard tests force the decision:creativeLatitude.test.jswalksdata.reference/prompts/stage-config.json, andcreativeLatitude.sources.test.jsscans everysource:literal handed to the shared runners — both fail on an unclassified entry. A caller that budgets a context window before the stamp lands (the inline editorial check) must reserveCREATIVE_LATITUDE_TOKENS. The clause is shown, never hidden (#11317): where the user typed the text the LLM receives (prompt Enhance/Refine, Music Designer, series review notes, mood board Generate more), the service stamps withwithCreativeLatitudeitself (idempotent, so the transport passes it through once) and returns the exact sent text plusCREATIVE_LATITUDE_ADDITIONand its instructions aspromptAdditions, which the client lists withSentPromptPanel.services/mediaPromptRefiner.jsis the worked example.