Skip to content

Self-hosted, Self-Improving, Federated Everything App — dev management, agent orchestration, persistent mind, digital twin, second brain, and creative pipeline from a single dashboard over Tailscale to port the user into the OS

24.2 KB
Raw Blame

Server conventions (server/)

These apply to Express/service/model code. Universal constraints (no try/catch outside the request lifecycle, functional programming, single-line logging, the Security Model, the Storage backend policy, the AI Provider Usage Policy, and the Sensitive Data & Privacy rules) live in the root AGENTS.md and always apply.

Validation and schema parity

  • Schema parity when adding fields. When you add a field to a sanitizer, createXxx, or a payload shape, update the corresponding Zod schema (server/lib/aiToolkit/validation.js for toolkit shapes, server/lib/validation.js for PortOS routes) in the same change. Wire validation into POST and PUT (PUT can use schema.partial()); the PortOS convention is all inputs validated — an exported-but-unwired sub-schema creates false confidence. For polymorphic stores like PUT /api/settings that accept any sub-object, validate the relevant slice with schema.partial() when its key is present in the request body (see server/routes/settings.js for the backupConfigSchema.partial() pattern). Tolerate UI sentinels (endpoint: '' for CLI providers) with z.preprocess(v => v === '' ? undefined : v, …). When a service migrates legacy keys on read, the schema must still accept the legacy shape so older clients don't 400 before the migration runs.

  • A POST/PUT/PATCH handler that reads req.body must validate it with a schema. validateRequest(schema, req.body ?? {}) is the default; put the schema in server/lib/validation.js (or the domain's *Validation.js). A hand-rolled if (!path) throw … lets a wrong-typed field reach the service layer. server/routeValidationConventions.test.js scans server/routes/, the toolkit's lib/aiToolkit/routes/ (schemas in lib/aiToolkit/validation.js, which cannot import lib/validation.js) and the CoS runner's cos-runner/index.js (requestSchemas.js, answering its own { error } 400) for this; its ALLOWED_UNVALIDATED list (raw-binary and multipart-upload bodies, each with a reason) may only shrink — the test fails when an entry is compliant or gone. Do not add a JSON-body handler to it (#9950).

  • A handler that reads req.query must validate the query object with a schema. Same spellings and same trees, on GET/POST/PUT/PATCH/DELETE: validateRequest(schema, req.query) (or validate / parseBody / safeParse / Schema.parse), and the data argument must be the query object or a name assigned from it — not one field, and not req.params. parsePagination(req.query) only clamps limit/offset, so it does not satisfy the guard. UNVALIDATED_QUERY_BASELINE in that same test is the burn-down of handlers that already skip this; it may only shrink. Delete a line when the handler validates or disappears, and do not add one — a new reader gets a schema (#10024).

Offering models vs running them

  • A caller that OFFERS a model choice reads providers through getSelectableProviders() / listSelectableProviders(); a caller that EXECUTES, ACCOUNTS or WRITES reads getAllProviders() / listProviders(). The scoped pair narrows each record's models to the install's declared entitlement — which of a vendor's advertised catalog this account can actually run, a thing no /models response reports (see docs/MODEL_ACCESS.md and lib/aiToolkit/internal/modelAccess.js). Picking the wrong one is silent both ways: a picker on the unscoped list offers models the account cannot run, and a write path on the scoped list persists a truncated catalog over the real one, which no refresh restores until the policy is cleared. Two methods rather than a flag, because the answer is a property of the caller's purpose and has to be visible at the call site. Route payloads shaped by presentProvider are already scoped; anything reading the service directly is not.

Write serialization and throughput

  • Async PATCH races on shared records — serialize writes server-side. When two write paths can mutate the same record concurrently (e.g. a blur-save plus an explicit "Render" button against stages.comicPages.cover), client-side guards (refs, onMouseDown, status checks) are unreliable — keyboard activation, status==='unknown' stalls, and loadState → modify → saveState interleaving all defeat them. The correct fix is to serialize writes at the file level on the server (every PATCH awaits the previous one to settle before reading state, so it merges against the freshest persisted record). See queueSeriesIssuesWrite in server/services/pipeline/issuesShared.js (built on createKeyCachedQueue in server/lib/createKeyCachedQueue.js). A Map<recordId, Promise> is not enough — two writes to different record ids share the same JSON state file and can still clobber each other; collapse the queue to a single tail per shared file.
  • High-frequency state writes must batch. Per-line state mutations that round-trip through withStateLock → loadState → saveState (e.g. appendAgentOutput in server/services/cosAgentLifecycle.js) are fine for human-pace events but catastrophic when called from a hot loop — PTY output streams, AI tool-call streams, or any producer that can emit dozens of events per second. When wiring a new streaming producer, add a batched variant that takes an array (see appendAgentOutputLines for the pattern) and flush from the caller on a ~250ms debounce. Always drain the pending buffer in the producer's finish/cleanup path before the final state write so completion events don't beat the last output batch to disk.

PostgreSQL parameters

  • A parameter bound to a json/jsonb column or ::jsonb cast is serialized with toPgJson, never bare JSON.stringify. PostgreSQL refuses a lone UTF-16 surrogate (an emoji .slice()d in half) and NUL inside jsonb, and NUL in text; every file store the migrated records came from accepted both, so an unrepaired write throws and the record is lost (a promoted CoS task, an automatic investigation task, a legacy import retained for retry forever, #12815). toPgJson and normalizePgString live in server/lib/pgJson.js and repair a bad unit to U+FFFD, the same lossy fix a UTF-8 file write applied. lib/db.js applies normalizePgParams to every query() and withTransaction client call (strings, array elements and plain-object parameters; it never unescapes a string, so a text value holding the characters \^@ is stored as written), but a pre-stringified JSON string looks like ordinary text to it, so the call site still uses toPgJson. server/pgJsonConventions.test.js scans the tree for this and carries no allowlist; applied scripts/db-migrations/ are frozen and skipped.

Timers

  • An async timer callback must own its rejections. Inside setTimeout(async …) / setInterval(async …), every await needs either an enclosing try { … } catch or a .catch(…) on the awaited chain. A timer fires outside the request lifecycle, so nothing holds the promise the callback returns and a rejection escapes as an unhandled rejection — fatal on Node >= 15. The CoS runner shipped that at boot: its orphan-cleanup timer awaited a state-file read/write, so one truncated agents file killed the runner seconds after start and PM2 restart-looped it against the same file (#5668). Log and continue in the catch unless the callback genuinely owns a failure verdict. server/timerCallbackConventions.test.js scans the whole server tree for this and carries no allowlist; the worked example is the watchdog in server/services/mediaJobQueue/index.js.
  • A statement-level call to an async function must be awaited, end in .catch(…), or start with void. A bare announceSelf(peer); leaves a rejection with no owner, which becomes an unhandledRejection that autoFixer turns into an AI task for what was best-effort bookkeeping (#12978). void means "this function owns its rejections, fire-and-forget on purpose" — make that true first. server/floatingAsyncConventions.test.js resolves module-scope async functions (local and relatively imported) across the server tree and carries no allowlist.

Child processes

  • Attach a spawned child's terminal listener before the function next awaits. Node reports a failed spawn (ENOENT, EACCES, EMFILE) on process.nextTick and a fast-dying child exits while you are suspended: an unlistened error throws and takes the process down, an unlistened close strands the job running. Do the awaited work (a scanner read, a claim handoff) BEFORE spawn(), or call bufferChildExit(child) (services/videoGen/generateVideoHelpers.js) in the same tick and replay its result once the real handlers are wired (services/imageGen/local.js). The CoS Runner /spawn and /spawn-tui routes and the local image render each shipped this ordering bug (#11761, after #4617/#5655/#5791). server/childListenerOrderConventions.test.js scans the whole server tree for it and carries no allowlist.

Socket handlers

  • An async Socket.IO handler must own its rejections. Same rule as the timer callbacks above, same reason: Socket.IO hands the promise a listener returns to nobody, so a rejection escapes as an unhandled rejection and Node terminates the process — taking every agent run, PTY session and media job with it. Every await in an async handler under server/sockets/ needs an enclosing try { … } catch (the shape voice.js uses on whole handler bodies) or a .catch(…) on the awaited chain (the shape logs.js uses on the lookups it falls back from). server/sockets/asyncHandlerGuard.test.js scans the directory for this and carries no allowlist; it names the file and the event string of any offender. Precedent: voice:call:detach shipped unguarded (#5661).

Tests and peer fan-out

  • Record-creating tests and peer fan-out. Record create/update paths reach peer-sync only through the subscription adapter in server/services/sharing/recordEvents.js (autoSubscribeRecordToAllPeers etc.) — a silent no-op until peerSync.js registers the real implementation at module load. A suite that never loads peerSync.js therefore gets no fan-out without mocking anything. Suites that DO load the peer-sync graph (importing peerSync.js or sharing/index.js, directly or transitively) must still mock services/instances.js with mockNoPeers() from server/lib/mockPathsDataRoot.js and services/sharing/peerSync.js with mockNoPeerSync() — the mock keeps the real module's registration side effect from wiring live fan-out into the adapter. To assert auto-subscribe behavior in a domain test, register a test double via registerSubscriptionAdapter({...}) and clean up with __resetSubscriptionAdapter().

Import scoping

  • A widely-reached module must not drag a subtree it only needs a constant from. The Linux CI server suite spends more wall time importing modules than running assertions, because import cost multiplies: every test file that transitively reaches module X re-instantiates all of X's closure in its own worker. So one convenient barrel import inside a module 400 suites reach is worth thousands of module instantiations. server/lib/importScoping.test.js pins the narrowings made in #6009 (which cut the suite from ~115.5k static module instantiations to ~94.5k) as negative reachability assertions over staticImportClosure; add a row there when you narrow another one. The two shapes worth looking for: a constant re-exported through a heavy barrel (import it from the module that declares it — CHECK_SCOPES from server/lib/editorial/checkInfra/taxonomy.js, not from editorial/checkRegistry.js and its 13 check-definition modules), and a dependency only a boot/rare path uses (await import() it inside that path — ensureSchemaImpl() composes the DDL that way, keeping lib/db/schema/** out of the ~400 suites that reach lib/db.js through pgFileFacade.js).
  • A lazy import() cuts instantiation cost, not CI selection. vitest related follows dynamic imports as well as static ones, so a module that lazily imports a heavy service still links every suite reaching it to every suite reaching that service. In 2026-10 one dynamically connected component around services/cos.js meant a change to any module in it selected ~1,130 server suites, although the static reach of cos.js was 90 (#12285). To cut selection, move the piece the caller actually needs into a leaf module both sides import (services/readinessSignal.js), or have the heavy service register a callback at boot that the caller looks up (initAutoFixer() → services/providerRecoveryHooks.js, read by promptRunner.js). Measure with node scripts/lib/importReach.js <repo-relative source>; the CI plan's server reason prints the same dynamic-aware fan-out for the changed sources. Cut edges are pinned in SELECTION_CUTS in server/lib/importScoping.test.js. A registry has the mock caveat in reverse: a suite that vi.mocked the heavy module must register test doubles instead.
  • In a non-async scope, defer with .then() rather than giving up. The lazy shape is not limited to async functions: a new Promise executor or a fire-and-forget branch takes import('./x.js').then(({ fn }) => fn(…)).catch(handler), which also routes an import failure into the error path the call already had. That is how the TUI branch in services/promptRunner.js defers tuiPromptRunner.js (and with it node-pty). Prefer putting the import() INSIDE an existing predicate gate when there is one — runner.js defers ollamaAgentContext.js inside the isOllamaBackedProvider ternary, so a cloud-provider run pays neither the hop nor the subtree.
  • The tree-wide backstop is a base-relative growth guard, not an absolute ceiling. importScoping.test.js compares every server *.test.js file's static import closure against the merge base, using one analyzer for both sides (scripts/lib/importGrowth.js). The count is a proxy: closure size summed across suites, shared modules counted once per suite that reaches them, not a count of runtime instantiations or a wall-clock. New test files are reported as new-suite contribution and do not spend the allowance, so ordinary additions cannot fail the guard just because the repository total grew. The gated number is the sum, over suites that exist on both sides, of how much each closure grew (shrinkage elsewhere does not cancel it). The limit is 2,500, above the old ceiling's ~1,500 ordinary-growth note and below half of the smaller heavy regression this file already records: #6009 removed 19,286 (115,519 → 96,233) and #6156 removed 12,794 (96,233 → 83,439). A failure names the largest suite increases and the largest newly reached dependency closures. Narrow the import, or defer it with await import() on the path that uses it. Raising EXISTING_SUITE_GROWTH_LIMIT is the intentional-regression path: do it in the same change, name the module that cannot be deferred, and keep the limit under that #6156 band — the pin test rejects a limit high enough to admit those edges. A missing or unreadable base fails the guard; it is not treated as zero growth. benchmarkProxySplit reports synthetic collection time and execution time separately and neither is a per-PR threshold. CI and npm run pregate run this comparison: the test is on ALWAYS_RUN_TESTS, and pregate forwards the same CI_BASE_SHA the planner used.
  • Narrowing an import must not step around a barrel a suite mocks. vi.mock('./cos.js') intercepts the specifier, not the function — so re-pointing a caller from ./cos.js to the ./cosTaskStore.js that declares addTask makes production code reach the REAL implementation while the suite still believes it is doubled. The failure lands in an unrelated test file with no obvious link to the import you changed. Before narrowing, grep the tree for vi.mock against the module you are leaving; if any suite that reaches the caller mocks it, leave the import on the barrel. A lazy await import() has no such hazard — vitest's mock registry covers dynamic imports too.

Generated manifests

  • Derive at first use before committing a manifest. A source-derived inventory that is deterministic, cheap, and reads only files every install ships belongs in memory behind a per-process cache, not in git: server/lib/apiRouteGraph.js scans the mounted route graph in ~50 ms on the first /api/api-docs/* request, and socketEventInventory.js does the same for Socket.IO events. The route catalog used to be a committed apiRouteCatalog.generated.json (retired — it no longer exists in the tree); even content-keyed, its stats counts conflicted on every pair of branches that each added a route, main carried a stale stats.declarations over a byte-identical route graph, and it took 14 regeneration-only commits in 90 days. A committed copy is justified only when derivation is slow, nondeterministic, or needs inputs an install lacks (promptStageCallSites.generated.json reads every git ls-files-tracked server/ source once per stage key, which no request handler should pay for).
  • A checked-in generated manifest is addressed by CONTENT, never by POSITION. server/lib/*.generated.json files are derived from source and kept honest by a drift test that regenerates and compares. That only works if the manifest changes when the thing it describes changes — and only then. A record that says "declared at foo.js:412" is rewritten by any edit that inserts a line above 412 (a rename, a comment, an unrelated handler), so the drift test fires on commits that change no route/stage/event at all, each parallel branch regenerates the file differently, and every rebase conflicts on it.
  • Key each record by the declaring file plus the semantic identity of what it declares — the stage key plus file paths for promptStageCallSites.generated.json. Positional detail may stay in memory: the route-graph coverage guard in server/lib/apiRouteGraph.test.js compares two fresh in-memory scans (scanRouteGraph() vs parseRouteModule(), keyed file#routerId METHOD /path by routeDeclarationKey), so nothing committed ever points back into the source it was derived from.
  • Prove it with a property test, not a name list. shiftSourceText (scripts/lib/positionInvariance.js) prepends a comment header to each source the generator reads; regenerate from the shifted sources and demand byte-identical output — asserting the rule directly, so it catches a position recorded as at, span, row, a loc: [412, 8] tuple, or a foo.js#L412 anchor. The prompt-stage generator carries that test. server/lib/generatedManifests.test.js is the cheap tree-wide backstop: it scans every git-tracked *.generated.json for line/column/offset-style keys and path/to/file.js:412 strings, catching a manifest whose generator never adopted the property test. It matches key names, so it is a net, not a proof — write both. That same file also enforces adoption structurally: it lists every git-tracked scripts/*.js that writes a .generated.json and fails when one has no sibling .test.js importing positionInvariance.js, so a new generator cannot ship without the property test.

Prompt templates

  • Stage-prompt template changes need a migration. scripts/setup-data.js only copies missing prompt files to data/prompts/stages/ — existing installs keep their old templates. When you add a {{template.variable}} reference to a data.reference/prompts/stages/*.md file, also add an entry to scripts/migrations/NNN-…js that updates the corresponding installed prompt when its hash still matches the pre-change shipped version. Normalize line endings (\r\n → \n, bare \r → \n) before hashing so the comparison is correct on Windows checkouts. Build it with makePromptReplaceMigration and export ACCEPTED_OLD_MD5 / NEW_SHIPPED_MD5 — never hand-mirror a hash table into scripts/setup-data.js, which sweeps those exports itself via buildPromptDriftTables. Worked example: scripts/migrations/394-factual-extraction-lens-prompts.js. A makeSeedMigration only copies MISSING files and will NOT upgrade an installed template. Re-point every earlier migration naming the same file at the new hash (_shippedHashDrift.test.js fails otherwise) and keep the superseded hash in the new migration's accepted-old set. Migration scripts live in scripts/migrations/; the per-install applied-list is data/migrations.applied.json (gitignored runtime state). See scripts/migrations/003-update-pipeline-stage-prompts.js.
  • A new migration is a frozen snapshot of its transform. Applied migrations are never edited; a new one (numbered above the current head) imports _lib.js primitives, server/lib pure helpers, or carries a frozen copy of the transform — never server/services, which an install upgrading months later would run under today's semantics. It declares no local copy of a _lib.js helper (readJsonOrNull, writeJsonAtomic, writeTextAtomic, the dashboard-grid helpers, …), and a purge migration writes its purge: true marker as a top-level property of the default export so scripts/run-migrations.js can read it from source without importing the module. scripts/migrations/migrationConventions.guard.test.js enforces all three.
  • A PROMPT_VERSIONS bump for a scheduled-task prompt (server/services/taskPromptDefaults.js) needs NO migration — its second step is node scripts/regen-prompt-integrity-snapshot.js, which retires the outgoing hash (root AGENTS.md "Distribution model"). readSchedule() in server/services/taskScheduleStore.js already upgrades every uncustomized stored prompt to the shipped default on the next loadSchedule() and persists the result — the first schedule read after the update converges the install regardless of whether a migration ran. Six migrations re-implemented that same loop against a task-schedule.json at the data root — a path the store never reads; readSchedule() reads data/cos/task-schedule.json — before this was written down: 281-reconcile-missing-releases-prompt.js, 293-agents-md-prompt-rename.js, 297-release-check-slashdo-review-prompt.js, 308-release-check-advisory-review-prompt.js, 332-release-check-fix-blockers-prompt.js, 338-claim-issue-volunteer-claim-contract.js. They stay — the distribution model forbids deleting applied migrations — but they are closed precedent, not a pattern to repeat. scripts/migrations/promptBumpMigrations.guard.test.js fails if a seventh copy imports taskPromptDefaults.js and writes task-schedule.json.
  • Creative prompts carry the IP-latitude clause automatically — classify the stage or source rather than wrapping a call site. PortOS is a professional artist's tool; a model that silently renames, genericizes, or disclaimers a deliberate reference to existing IP produces defective output, and rights/clearance is the operator's responsibility, not the model's. server/lib/creativeLatitude.js owns the clause and the single table that classifies what gets it, applied at each of the three LLM transports: promptService.buildPrompt (stage templates, keyed on the stage name — fails SAFE, so a stage the tables never saw is stamped), promptRunner.runPromptThroughProvider (every hand-rolled prompt, keyed on the run source tag it already requires — an allowlist, since ops calls dominate there), and aiProvider.callProviderAISimple (its own fetch, so creative callers pass creative: true). So a new creative stage or LLM-calling service is covered only once its stage name / source is classified in that module — add the prefix or exact name there rather than wrapping a call site. Two guard tests force the decision: creativeLatitude.test.js walks data.reference/prompts/stage-config.json, and creativeLatitude.sources.test.js scans every source: literal handed to the shared runners — both fail on an unclassified entry. A caller that budgets a context window before the stamp lands (the inline editorial check) must reserve CREATIVE_LATITUDE_TOKENS. The clause is shown, never hidden (#11317): where the user typed the text the LLM receives (prompt Enhance/Refine, Music Designer, series review notes, mood board Generate more), the service stamps with withCreativeLatitude itself (idempotent, so the transport passes it through once) and returns the exact sent text plus CREATIVE_LATITUDE_ADDITION and its instructions as promptAdditions, which the client lists with SentPromptPanel. services/mediaPromptRefiner.js is the worked example.