Self-hosted, Self-Improving, Federated Everything App — dev management, agent orchestration, persistent mind, digital twin, second brain, and creative pipeline from a single dashboard over Tailscale to port the user into the OS
JavaScript ai-agentsdashboarddeveloper-toolsdigital-twinquantified-selfsecond-brainself-hostedtailscale
fix: version FableLoom runtime and enforce live decision windows (#13106)
Adam Eivy4ac8ea57c7d5
diff --git a/docs/features/fableloom-runtime-v1.md b/docs/features/fableloom-runtime-v1.mdnew file mode 100644index 0000000000..4a83ae0afb--- /dev/null+++ b/docs/features/fableloom-runtime-v1.md@@ -0,0 +1,171 @@+# FableLoom runtime contract v1++Runtime version: **1.0.0** (`FABLELOOM_RUNTIME_VERSION` in+`server/lib/fableLoomPlayback.js`). Hosted session snapshots advertise+`runtimeVersion`. This version describes playback and turn semantics; an export+manifest's `schemaVersion` is a separate version. A consumer must reject an+unsupported runtime major version. Additive optional fields may be ignored;+changes to traversal, admission or event meaning require a new runtime version+and corresponding hosted regression tests.++This is the runtime foundation for [external package export #13104](https://github.com/atomantic/PortOS/issues/13104).+It does not yet implement an exporter, downloadable package or reference player.+PortOS remains a private authoring/rehearsal service. An external host runs its+own backend, credentials and transport; it never relays public calls into PortOS.++## Graph and playback++Episodes are ordered graphs with a `startNodeId`. A node has an `id`, authored+`prose`, a `playbackMode` (`cut` or `decision`), `audienceConnection`,+`protagonistPresence`, `isEnding`, `endingLabel` and ordered transitions.+Each transition has `id`, `intent`, `triggers: string[]`, optional spoiler-safe+`description` and `targetNodeId`. Resolve an edge only within the current+episode. Never invent a transition or follow a dangling target.++Playback phases are `entry | hold | exit | ended`. An entry plays once. A+decision hold loops while awaiting an allowed choice or turn. A cut plays once,+then takes its automatic continuation on media `ended`; it does not open a+conversation merely because a player reports `hold`. A selected transition may+play its exit clip before entering the target. An ending has no next turn.++Typed playback assets are entry, ordered hold loops and exits keyed by transition+id. Legacy `videoHistoryId` serves as the entry/hold fallback; a missing+`playbackMode` defaults to `decision`. See `resolvePlaybackPhaseAsset` for+selection and deterministic hold rotation. Hosted initial phase is `ended` for+an ending, otherwise `entry` for a typed entry, `hold` for typed holds, or+`entry`/`hold` for a legacy cut/decision. An absent video starts in `hold`.++An asset's audio occupancy carries `durationMs`, `characterDialogue`, `music`+and `effects` arrays. Intervals use inclusive start/exclusive end milliseconds+(`startMs`, `endMs`), with optional speaker/character labels and+`blocking: true` on effects. Live voice is unsafe if there is any character+dialogue, any blocking effect or detected clipping. The current gate excludes+the entire occupied hold; it does not carve silent gaps out of intervals.+An explicit `safeForLiveVoice: false` remains a refusal. Recompute safety from+occupancy rather than trusting a stale `true`.++## Two traversal lanes++The existing private authoring endpoint is+`POST /api/fableloom/:id/episodes/:episodeId/play`. These are data shapes,+not an instruction to call that endpoint from a public host.++| Lane | Input | Resolution |+|---|---|---|+| Deterministic | `{nodeId, transitionId}` | No LLM. An interactive node selects that exact current-node edge; an absent edge is `INVALID_TRANSITION`. Passive helper scenes take their first automatic edge. |+| Free text | `{nodeId, message, transcript?}` | A play-turn resolver chooses an existing edge or stays; the host owns its model and credentials. |++For authoring compatibility, `audienceCanParticipate` permits protagonist-mode+read-through choices, while helper-mode read-through input requires a connected+channel. That authoring convenience is distinct from live voice admission below.++The server result is:++```text+{+ action: "move" | "stay",+ transitionId?: string, // only for a valid move+ narration: string,+ node: public scene, // target on move; current on stay+ ended: boolean,+ resolvedBy: "choice" | "graph" | "llm",+ runId?: string // authoring trace only; not portable package data+}+```++An ending or a node with no edges returns `stay`, empty narration,+`ended: true`, `resolvedBy: "graph"` without a provider call. A resolver+selection absent from the current node, or an edge whose target has disappeared,+stays on the current node. Do not trust a model-supplied target node.++A host can expose `resolveTurn(nodeId, message)` returning+`{move: transitionId}` or `{stay: text}`. Map a move through the local graph;+map stay text to narration. This adapter shape is not the private endpoint's+response shape. A choice-only/keyword adapter needs no network or LLM.++## Live-input admission and half duplex++`evaluateFableLoomLiveInput({node, phase, asset})` is the shared pure admission+policy. `revalidateLiveConversationGate` additionally requires an active session.+Only a connected, non-ending decision node in `hold`, with an offscreen (or+legacy unspecified) protagonist and safe active audio, accepts live conversation.+An enabled interaction window's explicit protagonist presence overrides the node+presence. An interaction window does not make a cut, entry or exit interactive.++The policy returns `{allowed: true}` or `{allowed: false, reason}`. Refusal+codes are `NODE_NOT_FOUND`, `STORY_ENDED`, `AUDIENCE_DISCONNECTED`,+`PLAYBACK_NOT_DECISION`, `PLAYBACK_NOT_IN_HOLD_PHASE`,+`PROTAGONIST_ONSCREEN` and `HOLD_ASSET_OCCUPIED_BY_DIALOGUE`; the session+wrapper adds `SESSION_INACTIVE`.++The half-duplex turn states are `idle -> listening -> thinking -> speaking`.+Accept audio/text only while listening. Reserve a turn before asynchronous+work; refuse concurrent starts and episode switches. Move to thinking before+transcription/resolution and to speaking before narration delivery. Stay returns+to listening; a committed move returns to idle until the new scene is admitted.+Abort, expiration and teardown cancel the active turn. Recheck turn/session+liveness after asynchronous work so stale results cannot change a new episode+or emit after session end.++### Speech ordering and the current hosted gap++Portable hosts must finish narration playback before committing the chosen+transition (and before admitting another turn). Bind a playback completion to+the current turn, scene revision and designated output device. Treat cancellation,+playback failure and missing completion as explicit outcomes, never an implicit+successful acknowledgement.++**Hosted v1 currently guarantees synthesis and audio dispatch before commit,+not completed playback.** `processHostedUtterance` emits `hosted:turn:tts`+then immediately changes the node and emits `hosted:story:transition`.+The legacy `hosted:speech:done` handler merely changes speaking to listening;+it neither binds the acknowledgement to a turn nor delays a transition. The+clients currently do not emit that acknowledgement. Repair is tracked in+[#13112](https://github.com/atomantic/PortOS/issues/13112).+Do not infer audible completion from a TTS frame or from a committed story frame.+The regression test pins dispatch-before-commit without claiming playback+completion.++## Transport-neutral event schema++The table defines semantic names and payloads for a host-owned WebSocket, SSE or+in-process event stream. It maps the existing private Socket.IO vocabulary;+semantic names are not new Socket.IO events. Fields marked `?` are optional.+All identifiers below are host-local graph/turn identifiers, not PortOS peer ids,+join tokens or instance ids.++| Semantic event | Payload | Private hosted mapping |+|---|---|---|+| `session:snapshot` | `{runtimeVersion, currentNodeId, playbackPhase, activeHoldIndex, turnPhase, status}` | `hosted:session:sync`; project the bounded fields, never export the full private snapshot/transcript. |+| `player:node-enter` | `{nodeId, phase, activeHoldIndex}` | Host sends `hosted:playback:update` with `nodeId`, `phase`, `activeHoldIndex`; server sends `hosted:playback:sync`. |+| `player:phase` | `{nodeId, phase, activeHoldIndex}` | Same update/sync pair for a phase-only change. |+| `turn:phase` | `{phase: "idle" \| "listening" \| "thinking" \| "speaking", turnId?, text?}` | `hosted:turn:phase`. Text is narration, not a new graph. |+| `turn:speech` | `{turnId, audio, mimeType, target: "host" \| "audience"}` | `hosted:turn:tts`; audio is base64 WAV in the private transport. |+| `story:transition-committed` | `{transitionId, node, playbackPhase}` | `hosted:story:transition`; node is the server-projected target. |+| `session:ended` | `{reason}` | `hosted:session:ended` (also carries a private session id). |+| `runtime:error` | `{code, message}` | `hosted:error`. |++An initial/reconnect snapshot is authoritative. Apply events in stream order.+A host-owned transport must provide ordering/reconnect reconciliation and reject+late results from a different session, episode or turn. The current Socket.IO+stream has no portable sequence/replay envelope; an external host supplies its+own. Private `hosted:turn:transcript`, join/host tokens, peer status and session+addresses are not exported runtime records.++The play-turn system inputs are authored story context, current scene prose,+current transition ids/intents/triggers/descriptions, the host's bounded+conversation history, reader message and narration-format instructions. A model+returns `{action: "move" | "stay", transitionId?, narration}`; validate its edge+against the current graph before building a public result. Package character+inputs and any voice reference are separate explicit opt-ins, never an export+of provider configuration, adapters, LoRAs or machine-local paths.++## Conformance checks++`server/services/fableLoom/hostedSession.test.js` exercises the shared admission+policy through real turn admission, including a connected cut with a stale hold+phase, occupied audio, explicit window presence and the legacy decision default.+Its deferred synthesis test checks the current hosted phase/event ordering.+Existing hosted socket and weave suites cover role-scoped transport, teardown,+stale turns and provider-free deterministic traversal. Speech playback+acknowledgement conformance remains the explicit gap in #13112.diff --git a/docs/features/fableloom.md b/docs/features/fableloom.mdindex d487e9d198..2e77fd1f27 100644--- a/docs/features/fableloom.md+++ b/docs/features/fableloom.md@@ -8,6 +8,10 @@ decision nodes loop while the viewer chooses a path or gives free-textfeedback—when that story's audience role permits it. A loom holds one or more ordered episodes, and the Play surface canrehearse the same experience at any of those three production stages.+The [runtime v1 contract](fableloom-runtime-v1.md) documents traversal, live-input+gates and the mapping from portable events to private hosted sessions. External+package export is tracked in #13104; the contract alone does not provide an exporter.+## Planning a seriesThe series editor has separate, linkable sections for the arc, plot points anddiff --git a/server/lib/README.md b/server/lib/README.mdindex 44c74487d6..19205092de 100644--- a/server/lib/README.md+++ b/server/lib/README.md@@ -63,7 +63,7 @@ The barrel `server/lib/index.js` is a machine-checkable enumeration of every pub| `fableLoomLimits.js` | FableLoom record caps (`LOOM_LIMITS`) — the single source of truth shared by the route schemas and the record sanitizer. A leaf module, below both. || `cameraMovements.js` | Shared camera-movement vocabulary (#10589): `CAMERA_MOVEMENTS` (stable ids with `family`, `energy`, `snap`/`still` flags, start/end framing and the four-part movement/speed/framing/end prose), `normalizeCameraMovement`/`cameraMovementId`/`cameraMovementFromText`, `shotCameraPrompt(camera)` (the four-part i2v camera block for a structured shot camera) and `cameraMovementCatalogForPrompt`. Used by FableLoom, Music Video planning/review and the layered template's `cameraRig.js`. || `fableLoomCameraMovements.js` | FableLoom names for the shared `cameraMovements.js` catalog (generated cut direction and the manual scene editor selector); persisted ids are unchanged. |-| `fableLoomPlayback.js` | FableLoom node playback modes (`cut` auto-advances; `decision` loops while awaiting input), with the backward-compatible default. |+| `fableLoomPlayback.js` | FableLoom runtime version, node playback modes, audio occupancy and `evaluateFableLoomLiveInput`: shared live-input admission for connected decision holds, offscreen presence and safe audio, with legacy playback defaults. || `fableLoomParticipation.js` | FableLoom audience roles (helper vs protagonist), per-scene connection availability, prompt contracts, and backward-compatible defaults. || `fableLoomFormats.js` | FableLoom scene formats (`LOOM_FORMATS`, `asLoomFormat`, `sceneFormatContract`, …) — how a loom writes scene text, read by validation, the sanitizer, and every generative stage. || `fableLoomProduction.js` | FableLoom production batch planning and topological orchestration: shared still/video render-format presets and story-level image/video provider preferences (16:9 default), ordered episode storyboard gates, DAG resolution, asset enumeration, execution stages, and exact-input provenance verification. |diff --git a/server/lib/fableLoomPlayback.js b/server/lib/fableLoomPlayback.jsindex c9983f47ec..016813f3ba 100644--- a/server/lib/fableLoomPlayback.js+++ b/server/lib/fableLoomPlayback.js@@ -10,6 +10,9 @@import { LOOM_LIMITS } from './fableLoomLimits.js';import { isNonBlankStr, trimTo } from './textUtils.js';+// Version of docs/features/fableloom-runtime-v1.md, independent of package schema versions.+export const FABLELOOM_RUNTIME_VERSION = '1.0.0';+export const FABLELOOM_PLAYBACK_MODES = Object.freeze(['cut', 'decision']);export const FABLELOOM_PLAYBACK_MODE_DEFAULT = 'decision';@@ -266,6 +269,36 @@ export function isAssetSafeForLiveVoice(manifest) {return validateAudioOccupancy(manifest).safeForLiveVoice === true;}+/**+ * Transport-neutral live-input admission for runtime v1. A missing playback+ * mode retains the persisted legacy decision default; an explicit cut never+ * opens a conversation even if a stale player reports the hold phase.+ */+export function evaluateFableLoomLiveInput({ node, phase, asset } = {}) {+ if (!node) return { allowed: false, reason: 'NODE_NOT_FOUND' };+ if (node.isEnding) return { allowed: false, reason: 'STORY_ENDED' };+ if (node.audienceConnection !== 'connected') {+ return { allowed: false, reason: 'AUDIENCE_DISCONNECTED' };+ }+ if (asFableLoomPlaybackMode(node.playbackMode) !== 'decision') {+ return { allowed: false, reason: 'PLAYBACK_NOT_DECISION' };+ }+ if (phase !== 'hold') {+ return { allowed: false, reason: 'PLAYBACK_NOT_IN_HOLD_PHASE' };+ }+ if (resolveFableLoomProtagonistPresence(node) === 'onscreen') {+ return { allowed: false, reason: 'PROTAGONIST_ONSCREEN' };+ }+ if (asset) {+ const occupancy = asset.manifest || asset.audioOccupancy || asset;+ if (asset.safeForLiveVoice === false+ || occupancy.safeForLiveVoice === false || !isAssetSafeForLiveVoice(occupancy)) {+ return { allowed: false, reason: 'HOLD_ASSET_OCCUPIED_BY_DIALOGUE' };+ }+ }+ return { allowed: true };+}+/*** Helper to build an audio occupancy manifest.*/diff --git a/server/services/fableLoom/hostedSession.js b/server/services/fableLoom/hostedSession.jsindex fd879b4fa0..36d18598ef 100644--- a/server/services/fableLoom/hostedSession.js+++ b/server/services/fableLoom/hostedSession.js@@ -10,7 +10,7 @@* 5. Authoritative server story/session state and computer playback clock.* 6. Half-duplex turn contract: LISTENING -> THINKING -> SPEAKING -> LISTENING.* 7. Live conversation gate revalidation (helper decision hold, offscreen protagonist, safe audio occupancy).- * 8. Speech completion before story transition commitment.+ * 8. Speech synthesis and dispatch before commitment (playback ACK: #13112).* 9. Reconnect snapshots and session teardown with in-flight abort.*/@@ -27,7 +27,8 @@ import { pcmToWavBuffer } from '../../lib/chiptuneRender.js';import { pcmToFloat } from '../voice/callEndpointing.js';import {FABLELOOM_AUDIO_TARGETS,- isAssetSafeForLiveVoice,+ FABLELOOM_RUNTIME_VERSION,+ evaluateFableLoomLiveInput,resolvePlaybackPhaseAsset,} from '../../lib/fableLoomPlayback.js';import {@@ -81,6 +82,7 @@ export function sanitizeHostedSession(session) {if (!session) return null;return {id: session.id,+ runtimeVersion: FABLELOOM_RUNTIME_VERSION,loomId: session.loomId,episodeId: session.episodeId,universeId: session.universeId || null,@@ -230,28 +232,11 @@ export function revalidateLiveConversationGate({ session, node, asset }) {if (session?.status !== 'active') {return { allowed: false, reason: 'SESSION_INACTIVE' };}- if (!node) {- return { allowed: false, reason: 'NODE_NOT_FOUND' };- }- if (node.isEnding) {- return { allowed: false, reason: 'STORY_ENDED' };- }- if (node.audienceConnection !== 'connected') {- return { allowed: false, reason: 'AUDIENCE_DISCONNECTED' };- }- if (session.playbackPhase !== 'hold') {- return { allowed: false, reason: 'PLAYBACK_NOT_IN_HOLD_PHASE' };- }- if (node.protagonistPresence === 'onscreen') {- return { allowed: false, reason: 'PROTAGONIST_ONSCREEN' };- }- if (asset) {- const raw = asset.manifest || asset;- if (raw.safeForLiveVoice === false || !isAssetSafeForLiveVoice(raw)) {- return { allowed: false, reason: 'HOLD_ASSET_OCCUPIED_BY_DIALOGUE' };- }- }- return { allowed: true };+ return evaluateFableLoomLiveInput({+ node,+ phase: session.playbackPhase,+ asset,+ });}/**diff --git a/server/services/fableLoom/hostedSession.test.js b/server/services/fableLoom/hostedSession.test.jsindex 9ff9ac37ee..238b1dee71 100644--- a/server/services/fableLoom/hostedSession.test.js+++ b/server/services/fableLoom/hostedSession.test.js@@ -18,6 +18,7 @@ import {verifyHostedHostToken,verifyHostedToken,} from './hostedSession.js';+import { FABLELOOM_RUNTIME_VERSION } from '../../lib/fableLoomPlayback.js';import * as records from './records.js';import * as weave from './weave.js';import * as networkExposure from '../../lib/networkExposure.js';@@ -197,6 +198,7 @@ describe('fableLoom hostedSession', () => {expect(result.session).toBeDefined();expect(result.session.id).toBeDefined();expect(result.session.status).toBe('active');+ expect(result.session.runtimeVersion).toBe(FABLELOOM_RUNTIME_VERSION);expect(result.session.audioTarget).toBe('host');expect(result.session.currentNodeId).toBe('node-start');expect(result.token).toBeDefined();@@ -285,40 +287,81 @@ describe('fableLoom hostedSession', () => {});});- describe('half-duplex turn execution', () => {- it('executes full speech-first turn and commits story transition', async () => {+ describe('runtime v1 listening admission', () => {+ // These regressions are checked at turn admission: a stale host phase,+ // explicit window presence or unsafe occupancy must never start providers.+ it.each([+ ['cut node with stale hold phase', { playbackMode: 'cut' }, 'hold', 'PLAYBACK_NOT_DECISION'],+ ['decision entry clip', {}, 'entry', 'PLAYBACK_NOT_IN_HOLD_PHASE'],+ ['decision exit clip', {}, 'exit', 'PLAYBACK_NOT_IN_HOLD_PHASE'],+ ['disconnected decision', { audienceConnection: 'disconnected' }, 'hold', 'AUDIENCE_DISCONNECTED'],+ ['onscreen live window', { interactionWindow: { enabled: true, protagonistPresence: 'onscreen' } }, 'hold', 'PROTAGONIST_ONSCREEN'],+ ['occupied hold', {+ playbackAssets: {+ holdLoopVideoHistoryIds: ['vid-dialogue'],+ audioOccupancy: { 'vid-dialogue': { characterDialogue: [{ startMs: 0, endMs: 1000 }], safeForLiveVoice: true } },+ },+ }, 'hold', 'HOLD_ASSET_OCCUPIED_BY_DIALOGUE'],+ ])('refuses %s before starting a turn', async (_label, patch, phase, code) => {const { session } = await createHostedSession('loom-1', 'ep-1');- const mockIo = {- of: () => ({- to: () => ({- emit: vi.fn(),- }),- }),- };+ const changedLoom = structuredClone(mockLoom);+ Object.assign(changedLoom.episodes[0].nodes[0], patch);+ vi.mocked(records.getLoom).mockResolvedValue(changedLoom);+ _getInternalSession(session.id).playbackPhase = phase;+ vi.spyOn(weave, 'playTurn');+ const io = makeIo();- // 1. Start listening- const listenRes = await startHostedListening(session.id, { io: mockIo });- expect(listenRes.ok).toBe(true);+ await expect(startHostedListening(session.id, { io })).rejects.toMatchObject({ status: 409, code });++ expect(weave.playTurn).not.toHaveBeenCalled();+ expect(stt.transcribe).not.toHaveBeenCalled();+ expect(tts.synthesize).not.toHaveBeenCalled();+ expect(getHostedSession(session.id).turnPhase).toBe('idle');+ expect(io.emits).toEqual([]);+ });++ it('retains the legacy decision default for a connected offscreen hold', async () => {+ const { session } = await createHostedSession('loom-1', 'ep-1');+ const legacyLoom = structuredClone(mockLoom);+ delete legacyLoom.episodes[0].nodes[0].playbackMode;+ vi.mocked(records.getLoom).mockResolvedValue(legacyLoom);++ await expect(startHostedListening(session.id)).resolves.toMatchObject({ ok: true });expect(getHostedSession(session.id).turnPhase).toBe('listening');+ });+ });- // Mock LLM play response+ describe('half-duplex turn execution', () => {+ it('dispatches synthesized speech before committing the story transition', async () => {+ const { session } = await createHostedSession('loom-1', 'ep-1');+ const io = makeIo();+ const speech = deferred();+ vi.mocked(tts.synthesize).mockReturnValue(speech.promise);+ await startHostedListening(session.id, { io });vi.spyOn(weave, 'playTurn').mockResolvedValue({- action: 'move',- transitionId: 'tr-1',+ action: 'move', transitionId: 'tr-1',narration: 'We shall enter the dark woods together.',node: { id: 'node-2', title: 'Deep Woods' },});- // 2. Process audience utterance- const turnRes = await processHostedUtterance(session.id, {- audioBuffer: Buffer.from('fake-audio-bytes'),- io: mockIo,+ const pending = processHostedUtterance(session.id, {+ audioBuffer: Buffer.from('fake-audio-bytes'), io,});+ await vi.waitFor(() => expect(tts.synthesize).toHaveBeenCalled());+ expect(getHostedSession(session.id)).toMatchObject({+ currentNodeId: 'node-start', turnPhase: 'speaking',+ });+ expect(io.emits.some(({ event }) => event === 'hosted:story:transition')).toBe(false);++ speech.resolve({ wav: Buffer.from('RIFFmockwavdata') });+ await expect(pending).resolves.toMatchObject({ ok: true });- expect(turnRes.ok).toBe(true);- const afterSession = getHostedSession(session.id);- expect(afterSession.currentNodeId).toBe('node-2'); // Moved to next node- expect(afterSession.transcript.length).toBeGreaterThan(1);+ expect(io.emits.filter(({ event }) => [+ 'hosted:turn:phase', 'hosted:turn:tts', 'hosted:story:transition',+ ].includes(event)).map(({ event, payload }) => payload.phase || event)).toEqual([+ 'listening', 'thinking', 'speaking', 'hosted:turn:tts', 'hosted:story:transition',+ ]);+ expect(getHostedSession(session.id).currentNodeId).toBe('node-2');});it('synthesizes the selected protagonist with its approved AuK profile regardless of character order', async () => {