Skip to content

Self-hosted, Self-Improving, Federated Everything App — dev management, agent orchestration, persistent mind, digital twin, second brain, and creative pipeline from a single dashboard over Tailscale to port the user into the OS

fix: version FableLoom runtime and enforce live decision windows (#13106)

Adam Eivy4ac8ea57c7d5
diff --git a/docs/features/fableloom-runtime-v1.md b/docs/features/fableloom-runtime-v1.md
new file mode 100644
index 0000000000..4a83ae0afb
--- /dev/null
+++ b/docs/features/fableloom-runtime-v1.md
@@ -0,0 +1,171 @@
+# FableLoom runtime contract v1
+
+Runtime version: **1.0.0** (`FABLELOOM_RUNTIME_VERSION` in
+`server/lib/fableLoomPlayback.js`). Hosted session snapshots advertise
+`runtimeVersion`. This version describes playback and turn semantics; an export
+manifest's `schemaVersion` is a separate version. A consumer must reject an
+unsupported runtime major version. Additive optional fields may be ignored;
+changes to traversal, admission or event meaning require a new runtime version
+and corresponding hosted regression tests.
+
+This is the runtime foundation for [external package export #13104](https://github.com/atomantic/PortOS/issues/13104).
+It does not yet implement an exporter, downloadable package or reference player.
+PortOS remains a private authoring/rehearsal service. An external host runs its
+own backend, credentials and transport; it never relays public calls into PortOS.
+
+## Graph and playback
+
+Episodes are ordered graphs with a `startNodeId`. A node has an `id`, authored
+`prose`, a `playbackMode` (`cut` or `decision`), `audienceConnection`,
+`protagonistPresence`, `isEnding`, `endingLabel` and ordered transitions.
+Each transition has `id`, `intent`, `triggers: string[]`, optional spoiler-safe
+`description` and `targetNodeId`. Resolve an edge only within the current
+episode. Never invent a transition or follow a dangling target.
+
+Playback phases are `entry | hold | exit | ended`. An entry plays once. A
+decision hold loops while awaiting an allowed choice or turn. A cut plays once,
+then takes its automatic continuation on media `ended`; it does not open a
+conversation merely because a player reports `hold`. A selected transition may
+play its exit clip before entering the target. An ending has no next turn.
+
+Typed playback assets are entry, ordered hold loops and exits keyed by transition
+id. Legacy `videoHistoryId` serves as the entry/hold fallback; a missing
+`playbackMode` defaults to `decision`. See `resolvePlaybackPhaseAsset` for
+selection and deterministic hold rotation. Hosted initial phase is `ended` for
+an ending, otherwise `entry` for a typed entry, `hold` for typed holds, or
+`entry`/`hold` for a legacy cut/decision. An absent video starts in `hold`.
+
+An asset's audio occupancy carries `durationMs`, `characterDialogue`, `music`
+and `effects` arrays. Intervals use inclusive start/exclusive end milliseconds
+(`startMs`, `endMs`), with optional speaker/character labels and
+`blocking: true` on effects. Live voice is unsafe if there is any character
+dialogue, any blocking effect or detected clipping. The current gate excludes
+the entire occupied hold; it does not carve silent gaps out of intervals.
+An explicit `safeForLiveVoice: false` remains a refusal. Recompute safety from
+occupancy rather than trusting a stale `true`.
+
+## Two traversal lanes
+
+The existing private authoring endpoint is
+`POST /api/fableloom/:id/episodes/:episodeId/play`. These are data shapes,
+not an instruction to call that endpoint from a public host.
+
+| Lane | Input | Resolution |
+|---|---|---|
+| Deterministic | `{nodeId, transitionId}` | No LLM. An interactive node selects that exact current-node edge; an absent edge is `INVALID_TRANSITION`. Passive helper scenes take their first automatic edge. |
+| Free text | `{nodeId, message, transcript?}` | A play-turn resolver chooses an existing edge or stays; the host owns its model and credentials. |
+
+For authoring compatibility, `audienceCanParticipate` permits protagonist-mode
+read-through choices, while helper-mode read-through input requires a connected
+channel. That authoring convenience is distinct from live voice admission below.
+
+The server result is:
+
+```text
+{
+ action: "move" | "stay",
+ transitionId?: string, // only for a valid move
+ narration: string,
+ node: public scene, // target on move; current on stay
+ ended: boolean,
+ resolvedBy: "choice" | "graph" | "llm",
+ runId?: string // authoring trace only; not portable package data
+}
+```
+
+An ending or a node with no edges returns `stay`, empty narration,
+`ended: true`, `resolvedBy: "graph"` without a provider call. A resolver
+selection absent from the current node, or an edge whose target has disappeared,
+stays on the current node. Do not trust a model-supplied target node.
+
+A host can expose `resolveTurn(nodeId, message)` returning
+`{move: transitionId}` or `{stay: text}`. Map a move through the local graph;
+map stay text to narration. This adapter shape is not the private endpoint's
+response shape. A choice-only/keyword adapter needs no network or LLM.
+
+## Live-input admission and half duplex
+
+`evaluateFableLoomLiveInput({node, phase, asset})` is the shared pure admission
+policy. `revalidateLiveConversationGate` additionally requires an active session.
+Only a connected, non-ending decision node in `hold`, with an offscreen (or
+legacy unspecified) protagonist and safe active audio, accepts live conversation.
+An enabled interaction window's explicit protagonist presence overrides the node
+presence. An interaction window does not make a cut, entry or exit interactive.
+
+The policy returns `{allowed: true}` or `{allowed: false, reason}`. Refusal
+codes are `NODE_NOT_FOUND`, `STORY_ENDED`, `AUDIENCE_DISCONNECTED`,
+`PLAYBACK_NOT_DECISION`, `PLAYBACK_NOT_IN_HOLD_PHASE`,
+`PROTAGONIST_ONSCREEN` and `HOLD_ASSET_OCCUPIED_BY_DIALOGUE`; the session
+wrapper adds `SESSION_INACTIVE`.
+
+The half-duplex turn states are `idle -> listening -> thinking -> speaking`.
+Accept audio/text only while listening. Reserve a turn before asynchronous
+work; refuse concurrent starts and episode switches. Move to thinking before
+transcription/resolution and to speaking before narration delivery. Stay returns
+to listening; a committed move returns to idle until the new scene is admitted.
+Abort, expiration and teardown cancel the active turn. Recheck turn/session
+liveness after asynchronous work so stale results cannot change a new episode
+or emit after session end.
+
+### Speech ordering and the current hosted gap
+
+Portable hosts must finish narration playback before committing the chosen
+transition (and before admitting another turn). Bind a playback completion to
+the current turn, scene revision and designated output device. Treat cancellation,
+playback failure and missing completion as explicit outcomes, never an implicit
+successful acknowledgement.
+
+**Hosted v1 currently guarantees synthesis and audio dispatch before commit,
+not completed playback.** `processHostedUtterance` emits `hosted:turn:tts`
+then immediately changes the node and emits `hosted:story:transition`.
+The legacy `hosted:speech:done` handler merely changes speaking to listening;
+it neither binds the acknowledgement to a turn nor delays a transition. The
+clients currently do not emit that acknowledgement. Repair is tracked in
+[#13112](https://github.com/atomantic/PortOS/issues/13112).
+Do not infer audible completion from a TTS frame or from a committed story frame.
+The regression test pins dispatch-before-commit without claiming playback
+completion.
+
+## Transport-neutral event schema
+
+The table defines semantic names and payloads for a host-owned WebSocket, SSE or
+in-process event stream. It maps the existing private Socket.IO vocabulary;
+semantic names are not new Socket.IO events. Fields marked `?` are optional.
+All identifiers below are host-local graph/turn identifiers, not PortOS peer ids,
+join tokens or instance ids.
+
+| Semantic event | Payload | Private hosted mapping |
+|---|---|---|
+| `session:snapshot` | `{runtimeVersion, currentNodeId, playbackPhase, activeHoldIndex, turnPhase, status}` | `hosted:session:sync`; project the bounded fields, never export the full private snapshot/transcript. |
+| `player:node-enter` | `{nodeId, phase, activeHoldIndex}` | Host sends `hosted:playback:update` with `nodeId`, `phase`, `activeHoldIndex`; server sends `hosted:playback:sync`. |
+| `player:phase` | `{nodeId, phase, activeHoldIndex}` | Same update/sync pair for a phase-only change. |
+| `turn:phase` | `{phase: "idle" \| "listening" \| "thinking" \| "speaking", turnId?, text?}` | `hosted:turn:phase`. Text is narration, not a new graph. |
+| `turn:speech` | `{turnId, audio, mimeType, target: "host" \| "audience"}` | `hosted:turn:tts`; audio is base64 WAV in the private transport. |
+| `story:transition-committed` | `{transitionId, node, playbackPhase}` | `hosted:story:transition`; node is the server-projected target. |
+| `session:ended` | `{reason}` | `hosted:session:ended` (also carries a private session id). |
+| `runtime:error` | `{code, message}` | `hosted:error`. |
+
+An initial/reconnect snapshot is authoritative. Apply events in stream order.
+A host-owned transport must provide ordering/reconnect reconciliation and reject
+late results from a different session, episode or turn. The current Socket.IO
+stream has no portable sequence/replay envelope; an external host supplies its
+own. Private `hosted:turn:transcript`, join/host tokens, peer status and session
+addresses are not exported runtime records.
+
+The play-turn system inputs are authored story context, current scene prose,
+current transition ids/intents/triggers/descriptions, the host's bounded
+conversation history, reader message and narration-format instructions. A model
+returns `{action: "move" | "stay", transitionId?, narration}`; validate its edge
+against the current graph before building a public result. Package character
+inputs and any voice reference are separate explicit opt-ins, never an export
+of provider configuration, adapters, LoRAs or machine-local paths.
+
+## Conformance checks
+
+`server/services/fableLoom/hostedSession.test.js` exercises the shared admission
+policy through real turn admission, including a connected cut with a stale hold
+phase, occupied audio, explicit window presence and the legacy decision default.
+Its deferred synthesis test checks the current hosted phase/event ordering.
+Existing hosted socket and weave suites cover role-scoped transport, teardown,
+stale turns and provider-free deterministic traversal. Speech playback
+acknowledgement conformance remains the explicit gap in #13112.
diff --git a/docs/features/fableloom.md b/docs/features/fableloom.md
index d487e9d198..2e77fd1f27 100644
--- a/docs/features/fableloom.md
+++ b/docs/features/fableloom.md
@@ -8,6 +8,10 @@ decision nodes loop while the viewer chooses a path or gives free-text
feedback—when that story's audience role permits it. A loom holds one or more ordered episodes, and the Play surface can
rehearse the same experience at any of those three production stages.
+The [runtime v1 contract](fableloom-runtime-v1.md) documents traversal, live-input
+gates and the mapping from portable events to private hosted sessions. External
+package export is tracked in #13104; the contract alone does not provide an exporter.
+
## Planning a series
The series editor has separate, linkable sections for the arc, plot points and
diff --git a/server/lib/README.md b/server/lib/README.md
index 44c74487d6..19205092de 100644
--- a/server/lib/README.md
+++ b/server/lib/README.md
@@ -63,7 +63,7 @@ The barrel `server/lib/index.js` is a machine-checkable enumeration of every pub
| `fableLoomLimits.js` | FableLoom record caps (`LOOM_LIMITS`) — the single source of truth shared by the route schemas and the record sanitizer. A leaf module, below both. |
| `cameraMovements.js` | Shared camera-movement vocabulary (#10589): `CAMERA_MOVEMENTS` (stable ids with `family`, `energy`, `snap`/`still` flags, start/end framing and the four-part movement/speed/framing/end prose), `normalizeCameraMovement`/`cameraMovementId`/`cameraMovementFromText`, `shotCameraPrompt(camera)` (the four-part i2v camera block for a structured shot camera) and `cameraMovementCatalogForPrompt`. Used by FableLoom, Music Video planning/review and the layered template's `cameraRig.js`. |
| `fableLoomCameraMovements.js` | FableLoom names for the shared `cameraMovements.js` catalog (generated cut direction and the manual scene editor selector); persisted ids are unchanged. |
-| `fableLoomPlayback.js` | FableLoom node playback modes (`cut` auto-advances; `decision` loops while awaiting input), with the backward-compatible default. |
+| `fableLoomPlayback.js` | FableLoom runtime version, node playback modes, audio occupancy and `evaluateFableLoomLiveInput`: shared live-input admission for connected decision holds, offscreen presence and safe audio, with legacy playback defaults. |
| `fableLoomParticipation.js` | FableLoom audience roles (helper vs protagonist), per-scene connection availability, prompt contracts, and backward-compatible defaults. |
| `fableLoomFormats.js` | FableLoom scene formats (`LOOM_FORMATS`, `asLoomFormat`, `sceneFormatContract`, …) — how a loom writes scene text, read by validation, the sanitizer, and every generative stage. |
| `fableLoomProduction.js` | FableLoom production batch planning and topological orchestration: shared still/video render-format presets and story-level image/video provider preferences (16:9 default), ordered episode storyboard gates, DAG resolution, asset enumeration, execution stages, and exact-input provenance verification. |
diff --git a/server/lib/fableLoomPlayback.js b/server/lib/fableLoomPlayback.js
index c9983f47ec..016813f3ba 100644
--- a/server/lib/fableLoomPlayback.js
+++ b/server/lib/fableLoomPlayback.js
@@ -10,6 +10,9 @@
import { LOOM_LIMITS } from './fableLoomLimits.js';
import { isNonBlankStr, trimTo } from './textUtils.js';
+// Version of docs/features/fableloom-runtime-v1.md, independent of package schema versions.
+export const FABLELOOM_RUNTIME_VERSION = '1.0.0';
+
export const FABLELOOM_PLAYBACK_MODES = Object.freeze(['cut', 'decision']);
export const FABLELOOM_PLAYBACK_MODE_DEFAULT = 'decision';
@@ -266,6 +269,36 @@ export function isAssetSafeForLiveVoice(manifest) {
return validateAudioOccupancy(manifest).safeForLiveVoice === true;
}
+/**
+ * Transport-neutral live-input admission for runtime v1. A missing playback
+ * mode retains the persisted legacy decision default; an explicit cut never
+ * opens a conversation even if a stale player reports the hold phase.
+ */
+export function evaluateFableLoomLiveInput({ node, phase, asset } = {}) {
+ if (!node) return { allowed: false, reason: 'NODE_NOT_FOUND' };
+ if (node.isEnding) return { allowed: false, reason: 'STORY_ENDED' };
+ if (node.audienceConnection !== 'connected') {
+ return { allowed: false, reason: 'AUDIENCE_DISCONNECTED' };
+ }
+ if (asFableLoomPlaybackMode(node.playbackMode) !== 'decision') {
+ return { allowed: false, reason: 'PLAYBACK_NOT_DECISION' };
+ }
+ if (phase !== 'hold') {
+ return { allowed: false, reason: 'PLAYBACK_NOT_IN_HOLD_PHASE' };
+ }
+ if (resolveFableLoomProtagonistPresence(node) === 'onscreen') {
+ return { allowed: false, reason: 'PROTAGONIST_ONSCREEN' };
+ }
+ if (asset) {
+ const occupancy = asset.manifest || asset.audioOccupancy || asset;
+ if (asset.safeForLiveVoice === false
+ || occupancy.safeForLiveVoice === false || !isAssetSafeForLiveVoice(occupancy)) {
+ return { allowed: false, reason: 'HOLD_ASSET_OCCUPIED_BY_DIALOGUE' };
+ }
+ }
+ return { allowed: true };
+}
+
/**
* Helper to build an audio occupancy manifest.
*/
diff --git a/server/services/fableLoom/hostedSession.js b/server/services/fableLoom/hostedSession.js
index fd879b4fa0..36d18598ef 100644
--- a/server/services/fableLoom/hostedSession.js
+++ b/server/services/fableLoom/hostedSession.js
@@ -10,7 +10,7 @@
* 5. Authoritative server story/session state and computer playback clock.
* 6. Half-duplex turn contract: LISTENING -> THINKING -> SPEAKING -> LISTENING.
* 7. Live conversation gate revalidation (helper decision hold, offscreen protagonist, safe audio occupancy).
- * 8. Speech completion before story transition commitment.
+ * 8. Speech synthesis and dispatch before commitment (playback ACK: #13112).
* 9. Reconnect snapshots and session teardown with in-flight abort.
*/
@@ -27,7 +27,8 @@ import { pcmToWavBuffer } from '../../lib/chiptuneRender.js';
import { pcmToFloat } from '../voice/callEndpointing.js';
import {
FABLELOOM_AUDIO_TARGETS,
- isAssetSafeForLiveVoice,
+ FABLELOOM_RUNTIME_VERSION,
+ evaluateFableLoomLiveInput,
resolvePlaybackPhaseAsset,
} from '../../lib/fableLoomPlayback.js';
import {
@@ -81,6 +82,7 @@ export function sanitizeHostedSession(session) {
if (!session) return null;
return {
id: session.id,
+ runtimeVersion: FABLELOOM_RUNTIME_VERSION,
loomId: session.loomId,
episodeId: session.episodeId,
universeId: session.universeId || null,
@@ -230,28 +232,11 @@ export function revalidateLiveConversationGate({ session, node, asset }) {
if (session?.status !== 'active') {
return { allowed: false, reason: 'SESSION_INACTIVE' };
}
- if (!node) {
- return { allowed: false, reason: 'NODE_NOT_FOUND' };
- }
- if (node.isEnding) {
- return { allowed: false, reason: 'STORY_ENDED' };
- }
- if (node.audienceConnection !== 'connected') {
- return { allowed: false, reason: 'AUDIENCE_DISCONNECTED' };
- }
- if (session.playbackPhase !== 'hold') {
- return { allowed: false, reason: 'PLAYBACK_NOT_IN_HOLD_PHASE' };
- }
- if (node.protagonistPresence === 'onscreen') {
- return { allowed: false, reason: 'PROTAGONIST_ONSCREEN' };
- }
- if (asset) {
- const raw = asset.manifest || asset;
- if (raw.safeForLiveVoice === false || !isAssetSafeForLiveVoice(raw)) {
- return { allowed: false, reason: 'HOLD_ASSET_OCCUPIED_BY_DIALOGUE' };
- }
- }
- return { allowed: true };
+ return evaluateFableLoomLiveInput({
+ node,
+ phase: session.playbackPhase,
+ asset,
+ });
}
/**
diff --git a/server/services/fableLoom/hostedSession.test.js b/server/services/fableLoom/hostedSession.test.js
index 9ff9ac37ee..238b1dee71 100644
--- a/server/services/fableLoom/hostedSession.test.js
+++ b/server/services/fableLoom/hostedSession.test.js
@@ -18,6 +18,7 @@ import {
verifyHostedHostToken,
verifyHostedToken,
} from './hostedSession.js';
+import { FABLELOOM_RUNTIME_VERSION } from '../../lib/fableLoomPlayback.js';
import * as records from './records.js';
import * as weave from './weave.js';
import * as networkExposure from '../../lib/networkExposure.js';
@@ -197,6 +198,7 @@ describe('fableLoom hostedSession', () => {
expect(result.session).toBeDefined();
expect(result.session.id).toBeDefined();
expect(result.session.status).toBe('active');
+ expect(result.session.runtimeVersion).toBe(FABLELOOM_RUNTIME_VERSION);
expect(result.session.audioTarget).toBe('host');
expect(result.session.currentNodeId).toBe('node-start');
expect(result.token).toBeDefined();
@@ -285,40 +287,81 @@ describe('fableLoom hostedSession', () => {
});
});
- describe('half-duplex turn execution', () => {
- it('executes full speech-first turn and commits story transition', async () => {
+ describe('runtime v1 listening admission', () => {
+ // These regressions are checked at turn admission: a stale host phase,
+ // explicit window presence or unsafe occupancy must never start providers.
+ it.each([
+ ['cut node with stale hold phase', { playbackMode: 'cut' }, 'hold', 'PLAYBACK_NOT_DECISION'],
+ ['decision entry clip', {}, 'entry', 'PLAYBACK_NOT_IN_HOLD_PHASE'],
+ ['decision exit clip', {}, 'exit', 'PLAYBACK_NOT_IN_HOLD_PHASE'],
+ ['disconnected decision', { audienceConnection: 'disconnected' }, 'hold', 'AUDIENCE_DISCONNECTED'],
+ ['onscreen live window', { interactionWindow: { enabled: true, protagonistPresence: 'onscreen' } }, 'hold', 'PROTAGONIST_ONSCREEN'],
+ ['occupied hold', {
+ playbackAssets: {
+ holdLoopVideoHistoryIds: ['vid-dialogue'],
+ audioOccupancy: { 'vid-dialogue': { characterDialogue: [{ startMs: 0, endMs: 1000 }], safeForLiveVoice: true } },
+ },
+ }, 'hold', 'HOLD_ASSET_OCCUPIED_BY_DIALOGUE'],
+ ])('refuses %s before starting a turn', async (_label, patch, phase, code) => {
const { session } = await createHostedSession('loom-1', 'ep-1');
- const mockIo = {
- of: () => ({
- to: () => ({
- emit: vi.fn(),
- }),
- }),
- };
+ const changedLoom = structuredClone(mockLoom);
+ Object.assign(changedLoom.episodes[0].nodes[0], patch);
+ vi.mocked(records.getLoom).mockResolvedValue(changedLoom);
+ _getInternalSession(session.id).playbackPhase = phase;
+ vi.spyOn(weave, 'playTurn');
+ const io = makeIo();
- // 1. Start listening
- const listenRes = await startHostedListening(session.id, { io: mockIo });
- expect(listenRes.ok).toBe(true);
+ await expect(startHostedListening(session.id, { io })).rejects.toMatchObject({ status: 409, code });
+
+ expect(weave.playTurn).not.toHaveBeenCalled();
+ expect(stt.transcribe).not.toHaveBeenCalled();
+ expect(tts.synthesize).not.toHaveBeenCalled();
+ expect(getHostedSession(session.id).turnPhase).toBe('idle');
+ expect(io.emits).toEqual([]);
+ });
+
+ it('retains the legacy decision default for a connected offscreen hold', async () => {
+ const { session } = await createHostedSession('loom-1', 'ep-1');
+ const legacyLoom = structuredClone(mockLoom);
+ delete legacyLoom.episodes[0].nodes[0].playbackMode;
+ vi.mocked(records.getLoom).mockResolvedValue(legacyLoom);
+
+ await expect(startHostedListening(session.id)).resolves.toMatchObject({ ok: true });
expect(getHostedSession(session.id).turnPhase).toBe('listening');
+ });
+ });
- // Mock LLM play response
+ describe('half-duplex turn execution', () => {
+ it('dispatches synthesized speech before committing the story transition', async () => {
+ const { session } = await createHostedSession('loom-1', 'ep-1');
+ const io = makeIo();
+ const speech = deferred();
+ vi.mocked(tts.synthesize).mockReturnValue(speech.promise);
+ await startHostedListening(session.id, { io });
vi.spyOn(weave, 'playTurn').mockResolvedValue({
- action: 'move',
- transitionId: 'tr-1',
+ action: 'move', transitionId: 'tr-1',
narration: 'We shall enter the dark woods together.',
node: { id: 'node-2', title: 'Deep Woods' },
});
- // 2. Process audience utterance
- const turnRes = await processHostedUtterance(session.id, {
- audioBuffer: Buffer.from('fake-audio-bytes'),
- io: mockIo,
+ const pending = processHostedUtterance(session.id, {
+ audioBuffer: Buffer.from('fake-audio-bytes'), io,
});
+ await vi.waitFor(() => expect(tts.synthesize).toHaveBeenCalled());
+ expect(getHostedSession(session.id)).toMatchObject({
+ currentNodeId: 'node-start', turnPhase: 'speaking',
+ });
+ expect(io.emits.some(({ event }) => event === 'hosted:story:transition')).toBe(false);
+
+ speech.resolve({ wav: Buffer.from('RIFFmockwavdata') });
+ await expect(pending).resolves.toMatchObject({ ok: true });
- expect(turnRes.ok).toBe(true);
- const afterSession = getHostedSession(session.id);
- expect(afterSession.currentNodeId).toBe('node-2'); // Moved to next node
- expect(afterSession.transcript.length).toBeGreaterThan(1);
+ expect(io.emits.filter(({ event }) => [
+ 'hosted:turn:phase', 'hosted:turn:tts', 'hosted:story:transition',
+ ].includes(event)).map(({ event, payload }) => payload.phase || event)).toEqual([
+ 'listening', 'thinking', 'speaking', 'hosted:turn:tts', 'hosted:story:transition',
+ ]);
+ expect(getHostedSession(session.id).currentNodeId).toBe('node-2');
});
it('synthesizes the selected protagonist with its approved AuK profile regardless of character order', async () => {