Skip to content

Self-hosted, Self-Improving, Federated Everything App — dev management, agent orchestration, persistent mind, digital twin, second brain, and creative pipeline from a single dashboard over Tailscale to port the user into the OS

fix: bind Runway reference tags to canon identities (#13105)

Adam Eivy7284acafdad7
diff --git a/server/lib/README.md b/server/lib/README.md
index f3fa707383..4896acd150 100644
--- a/server/lib/README.md
+++ b/server/lib/README.md
@@ -365,6 +365,7 @@ The barrel `server/lib/index.js` is a machine-checkable enumeration of every pub
| `reactorStartingFrame.js` | Fits a reactor.inc `starting_frame` to the fast-h3 session canvas before it is uploaded. `readImageSize(path)` → `{width,height}` or `null` (never a guessed dimension), and `prepareReactorStartingFrame(sourceImagePath, requestedAspect, outputPath)` → `{ aspect, canvas, framePath, fittedPath }`: an absent `requestedAspect` DERIVES the canvas from the frame's own shape (`nearestReactorAspect`), then centre cover-crops the frame to that canvas's exact pixel size so reactor's own "fitted to the session canvas" step is a no-op. `fittedPath` is the temp file the caller deletes; `null` means the original was uploaded unchanged. A frame sharp cannot decode (or a resize that fails) passes through untouched on the default canvas rather than failing a render PortOS merely could not measure — reactor decodes more containers than sharp does. |
| `reactorVideoClip.js` | The reactor.inc `fast-h3` clip contract, dependency-free so the route schema, the FableLoom prompt compiler and the VideoGen picker all read one copy — `REACTOR_MAX_PROMPT_LENGTH` (800; the API rejects a longer prompt rather than truncating it), `REACTOR_MIN_CLIP_SECONDS` / `REACTOR_MAX_CLIP_SECONDS` (5.167–14.375, i.e. 124–345 frames at the `REACTOR_CLIP_FPS` 24 it delivers), `REACTOR_MAX_CLIP_ID_LENGTH`, and the picker list `REACTOR_CLIP_LENGTHS` + `REACTOR_DEFAULT_CLIP_LENGTH` + `reactorClipLengthLabel()` (the two exact endpoints plus every whole second between them, since the accepted range is continuous but starts and ends off a round number). Also the CANVAS contract `set_canvas` accepts: `REACTOR_CANVASES` (`16:9` 1344×768, `4:3` 1024×768, `1:1` 768×768, `9:16` 768×1344 — every one a 768px short edge, so the aspect string is the whole choice), `REACTOR_ASPECTS`, `REACTOR_DEFAULT_ASPECT`, `reactorCanvas(aspect)` and `nearestReactorAspect(width, height)` (closest canvas in log-ratio space; anything unmeasurable answers the default rather than deriving one). Mirrored to `client/src/lib/reactorVideoClip.js`, parity pinned by `reactorVideoClip.test.js`. |
| `runwayVideoModels.js` | Curated Runway VIDEO model catalog for the developer API's `/v1/image_to_video` and (Gen-4.5) `/v1/text_to_video` — `RUNWAY_VIDEO_MODELS` (Gen-4.5 default, Gen-4 Turbo, Kling 3.0 Standard / Pro, Kling 2.5 Turbo Pro, MiniMax H3 Max, WAN 3.0) with per-model prompt caps, duration rules, output sizes (`ratio` or `resolution`), end-frame support and list prices verified on `RUNWAY_VIDEO_PRICING_VERIFIED_AT`; `getRunwayVideoModel()`, `coerceRunwayVideoSeconds()` (round UP to a covering length), `resolveRunwayVideoResolution()`, `runwayVideoLengths()`, `buildRunwayVideoRequest()` (the endpoint and exact body, text-only when no frame and the model has a `textToVideo` rule, `runwayModelTakesText()`; refuses an uncurated model, a missing frame or an over-cap prompt, never cuts it), `runwayVideoClipQuote()` / `estimateRunwayVideoCostUsd()`, `describeRunwayVideoRate()`. Dependency-free; the client re-exports it. |
+| `runwayImageModels.js` | Native Gen-4 image model schemas, supported canvases, prices, hosted disclosure, request validation and stable binding-derived reference tags; shared with browser model pickers. |
| `kimi.js` | Moonshot AI Kimi Code (`kimi`) provider helpers — id constants (`KIMI_CLI_ID`/`KIMI_TUI_ID`), `isKimiCommand`/`isKimiCliProvider`/`isKimiTuiProvider` predicates, `ensureKimiHeadlessArgs` (model flag only — non-interactive mode is implicit in `--prompt`, and kimi has no `--print`/`--afk` and refuses approval flags alongside `--prompt`) / `ensureKimiTuiArgs` (`--yolo`) argv builders (model selection uses the `KIMI_CONFIGURED_DEFAULT` sentinel in `providerModels.js` so PortOS omits `--model` like Grok/Antigravity), and `prepareKimiPrompt` (delivers the prompt as the `--prompt <value>` argv, not stdin). |
| `runners.js` | Image-runner family constants (`RUNNER_FAMILIES`, `loraCompatKey`), video LoRA families, MiniMax H3 runtime predicates, and the shared `AUDIO_TO_VIDEO_RUNTIMES` / `isAudioToVideoRuntime` capability gate (LTX plus H3 Ref2VA). Mirrored to `client/src/lib/runnerFamilies.js`. |
| `codexAccount.js` | The ChatGPT-subscription half of the Codex providers (#5589), pure. The fixed `codex app-server` invocation, the `account/*` method + notification names, `CODEX_ACCOUNT_STATUS` (`runtime-missing` / `unknown` / `signed-out` / `login-pending` / `ready` / `quota-exhausted` / `reauth-required`) and `CODEX_ERROR_CODES`, `isCodexSubscriptionProvider` (re-exported from `providerModels.js`; keyed on the `codex` COMMAND, so a renamed clone still resolves), the `normalizeCodexAccount` / `normalizeCodexRateLimits` / `normalizeCodexLoginStart` payload normalizers, `redactCodexPayload` (every log/error/response path runs through it, so "PortOS never writes down a Codex token" is one function rather than a rule per call site), and `deriveCodexAccountStatus`. Sentinels are load-bearing: `runtimeInstalled: null` = NOT PROBED, `accountFetched: false` ≠ signed out, `rateLimits: null` (failed fetch) ≠ a fetched window with nothing to report. The process half is `services/codexAppServer.js`. |
diff --git a/server/services/fableLoom/visualConditioning.js b/server/services/fableLoom/visualConditioning.js
index 8aa4f5df4a..89bfc9979a 100644
--- a/server/services/fableLoom/visualConditioning.js
+++ b/server/services/fableLoom/visualConditioning.js
@@ -401,7 +401,14 @@ export async function compileFableLoomVisualRequest({
universeId: universe.id,
capability,
bindings: manifestBindings(bindings),
- assets: allocation.accepted.map(({ path: _path, ...asset }) => asset),
+ assets: allocation.accepted.map(({ path: _path, ...asset }) => {
+ if (capability.backend !== 'runway') return asset;
+ const entry = asset.role.startsWith('character')
+ ? bindings.boundCharacters.find(({ character }) => character.id === asset.bindingId)?.character
+ : asset.role === 'environment' ? bindings.boundPlace
+ : bindings.boundObjects.find((object) => object.id === asset.bindingId);
+ return { ...asset, ...(entry?.name ? { label: entry.name } : {}) };
+ }),
adapters,
omitted,
warnings,
diff --git a/server/services/fableLoom/visualConditioning.test.js b/server/services/fableLoom/visualConditioning.test.js
index 4eb8cbd25a..9b0f72e4d5 100644
--- a/server/services/fableLoom/visualConditioning.test.js
+++ b/server/services/fableLoom/visualConditioning.test.js
@@ -81,6 +81,7 @@ describe('FableLoom visual conditioning compiler', () => {
authoredPrompt: 'Two rivals', ...deps(loom),
});
expect(result.visualConditioning.status).toBe('locked');
+ expect(result.visualConditioning.assets.filter((asset) => asset.role === 'character-neutral').map((asset) => asset.label)).toEqual(['Aria', 'Bex']);
expect(result.referenceImagePaths).toEqual(['/approved/opening.png', '/approved/aria-neutral.png', '/approved/bex-neutral.png']);
});
diff --git a/server/services/imageGen/runway.js b/server/services/imageGen/runway.js
index ecad6fdaa1..42bd5a6b4d 100644
--- a/server/services/imageGen/runway.js
+++ b/server/services/imageGen/runway.js
@@ -22,7 +22,7 @@ import { getSettings } from '../settings.js';
import {
awaitRunwayTask, cancelRunwayTask, createRunwayTaskEntry, resolveRunwayApiKey, submitRunwayTask,
} from '../runwayApi.js';
-import { IMAGE_GEN_MODE } from './modes.js';
+import { IMAGE_GEN_MODE, describeFidelity } from './modes.js';
import { resolveInputImages } from './inputImages.js';
import { cloudPromptRequired } from './cloudProviderConfig.js';
import { rejectDegenerateFrame } from './frameGuard.js';
@@ -74,12 +74,17 @@ export async function checkConnection({ model = RUNWAY_IMAGE_DEFAULT_MODEL } = {
return { connected: Boolean(apiKey), mode: MODE, model, ...(apiKey ? {} : { reason: noKeyReason }) };
}
-export function runwayImageSentPrompt({ prompt = '', negativePrompt, inputImages, visualConditioning, promptAdditions = [] }) {
+export function runwayImageSentPrompt({ prompt = '', negativePrompt, inputImages, initImageStrength, visualConditioning, promptAdditions = [] }) {
const references = runwayReferenceTags(inputImages.paths.map((path) => {
const asset = visualConditioning?.assets?.find((item) => item.filename === basename(path));
return { filename: basename(path), ...(asset || {}), path };
}));
- const roles = references.map((ref) => `@${ref.tag}: ${ref.role || (ref.path === inputImages.initPath ? 'source image to edit' : 'visual reference')}${ref.bindingId ? ` (${ref.bindingId})` : ''}.`).join('\n');
+ const roles = references.map((ref) => {
+ const role = ref.path === inputImages.initPath
+ ? `source image to edit — ${describeFidelity(initImageStrength)}`
+ : ref.role || 'visual reference';
+ return `@${ref.tag}: ${role}${ref.label ? ` for ${ref.label}` : ''}.`;
+ }).join('\n');
const added = [promptAddition('reference-tags', 'Tagged Runway reference images', roles), promptAddition('avoid', 'Negative prompt, sent as "Avoid:"', negativePrompt ? `Avoid: ${negativePrompt}` : '')];
const fullPrompt = [prompt, ...added.map((item) => item?.text).filter(Boolean)].join('\n\n');
return { fullPrompt, references, record: sentPromptRecord({ sentPrompt: fullPrompt, carried: carriedPromptAdditions({ promptAdditions }), added }) };
@@ -88,8 +93,10 @@ export function runwayImageSentPrompt({ prompt = '', negativePrompt, inputImages
async function encodeInputImage(path) {
const bytes = await readFile(path);
const mime = detectImageFormat(bytes)?.mime;
- if (!mime || bytes.length > 5 * 1024 * 1024) throw new ServerError('Runway references must be images no larger than 5 MB', { status: 400, code: 'RUNWAY_IMAGE_INPUT_INVALID' });
- return `data:${mime};base64,${bytes.toString('base64')}`;
+ const uri = mime ? `data:${mime};base64,${bytes.toString('base64')}` : null;
+ // Runway's 5 MB ceiling applies to the encoded URI, including base64 overhead.
+ if (!uri || Buffer.byteLength(uri) > 5_000_000) throw new ServerError('Runway reference data URIs must be images no larger than 5 MB after base64 encoding', { status: 400, code: 'RUNWAY_IMAGE_INPUT_INVALID' });
+ return uri;
}
export const generateImage = (params) => preparations.run(params, generateImagePrepared);
@@ -97,7 +104,7 @@ export const generateImage = (params) => preparations.run(params, generateImageP
async function generateImagePrepared({
promptAdditions = [],
model, resolution = '720p', ratio, prompt = '', negativePrompt, width, height, seed,
- initImagePath, referenceImagePaths = [],
+ initImagePath, initImageStrength, referenceImagePaths = [],
visualConditioning = null,
jobId: providedJobId = null,
cleanC2PA = false,
@@ -129,7 +136,7 @@ async function generateImagePrepared({
const filename = `${jobId}.png`;
const outputPath = join(PATHS.images, filename);
if (inputImages.paths.length !== (initImagePath ? 1 : 0) + referenceImagePaths.length) throw new ServerError('Runway reference inputs are unavailable or exceed the three-image limit', { status: 400, code: 'RUNWAY_IMAGE_INPUT_INVALID' });
- const { fullPrompt, references, record: sentRecord } = runwayImageSentPrompt({ prompt, negativePrompt, inputImages, visualConditioning, promptAdditions });
+ const { fullPrompt, references, record: sentRecord } = runwayImageSentPrompt({ prompt, negativePrompt, inputImages, initImageStrength, visualConditioning, promptAdditions });
const request = buildRunwayImageRequest({ modelId: family.id, prompt: fullPrompt, resolution, ratio, width, height, seed, referenceImages: references.map((ref) => ({ uri: '(validated input)', tag: ref.tag })) });
const meta = {
diff --git a/server/services/imageGen/runway.test.js b/server/services/imageGen/runway.test.js
index 7d3c4b4456..e021db7e8b 100644
--- a/server/services/imageGen/runway.test.js
+++ b/server/services/imageGen/runway.test.js
@@ -55,10 +55,12 @@ describe('Runway image workflow', () => {
it('sends both bound character anchors with distinct stable tags and records basenames rather than paths', async () => {
const paths = ['hero.png', 'rival.png'].map((name) => join(PATHS.images, name));
await Promise.all(paths.map((path) => writeFile(path, output)));
- const visualConditioning = { assets: [{ filename: 'hero.png', bindingId: 'char-a', role: 'character-neutral' }, { filename: 'rival.png', bindingId: 'char-b', role: 'character-neutral' }] };
+ const visualConditioning = { assets: [{ filename: 'hero.png', bindingId: 'char-a', role: 'character-neutral', label: 'Hero' }, { filename: 'rival.png', bindingId: 'char-b', role: 'character-neutral', label: 'Rival' }] };
const job = await runway.generateImage({ prompt: 'Two rivals', model: 'gen4_image_turbo', referenceImagePaths: paths, visualConditioning });
expect(await terminal(job.jobId)).toMatchObject({ type: 'completed' });
const refs = calls[0].body.referenceImages;
+ expect(calls[0].body.promptText).toContain(`@${refs[0].tag}: character-neutral for Hero.`);
+ expect(calls[0].body.promptText).toContain(`@${refs[1].tag}: character-neutral for Rival.`);
expect(refs).toHaveLength(2);
expect(refs[0].tag).not.toBe(refs[1].tag);
for (const ref of refs) { expect(ref.uri).toMatch(/^data:image\/jpeg;base64,/); expect(calls[0].body.promptText).toContain(`@${ref.tag}`); }
@@ -66,6 +68,13 @@ describe('Runway image workflow', () => {
expect(sidecar.referenceTags.map((r) => r.filename)).toEqual(['hero.png', 'rival.png']);
expect(JSON.stringify(sidecar)).not.toContain(root);
});
+ it('refuses an oversized encoded reference before any paid submission', async () => {
+ const path = join(PATHS.images, 'large.png');
+ await writeFile(path, Buffer.concat([output, Buffer.alloc(4_000_000)]));
+ const job = await runway.generateImage({ prompt: 'A garden', referenceImagePaths: [path] });
+ expect(await terminal(job.jobId)).toMatchObject({ type: 'failed', error: expect.stringContaining('after base64 encoding') });
+ expect(calls).toEqual([]);
+ });
it('fails a moderated task once with its verbatim reason and permits the next asset', async () => {
nextTask = { status: 'FAILED', failureCode: 'SAFETY.INPUT', failure: 'Input rejected by Runway moderation.' };
const job = await runway.generateImage({ prompt: 'A garden' });