Self-hosted, Self-Improving, Federated Everything App — dev management, agent orchestration, persistent mind, digital twin, second brain, and creative pipeline from a single dashboard over Tailscale to port the user into the OS
JavaScript ai-agentsdashboarddeveloper-toolsdigital-twinquantified-selfsecond-brainself-hostedtailscale
/**
* Sprites — reference-frame normalization + palette extraction (issue #2896).
*
* Node/sharp port of the source pipeline's `normalize_anchor_frame` (Pillow):
* mask the character out of the generated candidate by per-pixel difference
* from the background key color, then re-composite ONLY the masked pixels
* onto a fresh solid-key square canvas. Because the composite always starts
* from a clean canvas, switching key colors at lock time (mask on the
* generation key, fill with the selected key) is free — that's what makes the
* dynamic chroma-key selection possible without regenerating.
*
* Geometry contract (verbatim from the source): character height is 80% of
* the square side (or width fits inside 10% side margins, whichever needs a
* bigger canvas), feet baseline sits 7% above the bottom edge, pixels are
* never rescaled — only the canvas is sized around them. Mask threshold: a
* pixel is foreground when max-channel |pixel − key| > 40 — a DELIBERATE
* deviation from the source's luma metric, which is blind to black-vs-blue
* (see MASK_CHANNEL_THRESHOLD).
*
* Fringe decontamination: the hard mask keeps anti-aliased edge pixels
* that are a BLEND of the character and the generation key. When lock swaps the
* key (dynamic selection composites onto a key different from the one the
* candidate was masked against), those fringe pixels still carry the OLD key's
* tint — a magenta ring on a magenta→blue re-key. Each surviving foreground
* pixel is shifted by keyShare·(newKey − oldKey), the same straight-alpha
* recomposite `recoverAlphaFrame` performs, so the residual old key is replaced
* by the correct amount of the new key. A no-op when the keys match.
*/
import sharp from 'sharp';
import { tryReadFile, copyFileGuarded } from '../../lib/fileUtils.js';
import { hexToRgb, keyChannelSplit, keyShareFn } from './chromaKey.js';
import { describeFrameStats, isDegenerateFrame } from '../../lib/imageFrameStats.js';
import { ServerError } from '../../lib/errorHandler.js';
const FRAME_HEIGHT_FRAC = 0.80;
const FRAME_BOTTOM_FRAC = 0.07;
const FRAME_SIDE_FRAC = 0.10;
// Max-channel distance from the key, NOT the source pipeline's luma-of-diff:
// luma weights blue at 0.114, so against the blue key a BLACK pixel scores
// 255·0.114 ≈ 29 — under any usable threshold — and black outlines/hair
// would be silently erased. Luma was safe only because the source pipeline
// hardcoded magenta; the dynamic key set (#2895) makes the metric wrong.
// Max-channel treats all three keys symmetrically (black vs any pure key
// differs by 255 in at least one channel).
const MASK_CHANNEL_THRESHOLD = 40;
// Re-export for existing consumers/tests; the definition lives in the pure
// color-math module so chromaKey.js can use it without importing sharp.
export { hexToRgb };
/**
* Decode `src` as flat RGB (alpha dropped over white like Pillow's
* convert("RGB"); generated candidates are opaque PNGs so this is a no-op in
* practice) and compute the foreground mask + tight bounding box vs the key
* color. The lock path runs palette extraction AND normalization off ONE
* analysis so a multi-MP candidate is decoded and scanned once.
*/
const DEGENERATE_FRAME_DETAIL = {
'solid-fill': 'a single flat color',
'fully-transparent': 'fully transparent',
'near-empty': 'almost no detail',
};
/**
* Degenerate-frame gate for a sprite candidate (issue #4173).
*
* A blank candidate has no foreground at all, so `normalizeFromAnalysis` /
* `recompositeOnKey` copy it straight through and it gets locked, atlased and
* published as a real reference frame. Refuse it up front instead.
*
* Only a MEASURED degenerate verdict rejects: a candidate whose stats could
* not be computed (`ok: null`) falls through untouched, so an undecodable file
* still reaches the existing `imageError` / `INVALID_IMAGE` handling rather
* than being mislabeled as empty.
*/
export async function assertFrameHasContent(src) {
const stats = await describeFrameStats(src);
if (!isDegenerateFrame(stats)) return stats;
const detail = DEGENERATE_FRAME_DETAIL[stats.reason] || 'no content';
throw new ServerError(
`Candidate frame has no content (${detail}) — regenerate it before locking.`,
{ status: 422, code: 'DEGENERATE_FRAME' },
);
}
export async function analyzeForeground(src, maskKeyHex) {
// Read a path ONCE and hand the same bytes to the content gate and the decode
// it guards, the read-once-verify-in-memory seam the compiler already keeps
// (atlas.js): passing the path twice re-reads the file between the check and
// the use, and also defeats `describeFrameStats`'s content-addressed memo,
// which only keys on buffers (a path is not content). A read that fails falls
// through to the path unchanged so sharp still raises its own decode error —
// the existing INVALID_IMAGE surface, not a new ENOENT.
const source = Buffer.isBuffer(src) ? src : (await tryReadFile(src, null)) ?? src;
await assertFrameHasContent(source);
const key = hexToRgb(maskKeyHex);
const { data, info } = await sharp(source)
.flatten({ background: { r: 255, g: 255, b: 255 } })
.raw()
.toBuffer({ resolveWithObject: true });
const { width, height } = info;
const mask = new Uint8Array(width * height);
let left = width; let top = height; let right = -1; let bottom = -1;
for (let y = 0; y < height; y++) {
for (let x = 0; x < width; x++) {
const i = (y * width + x) * 3;
const dr = Math.abs(data[i] - key.r);
const dg = Math.abs(data[i + 1] - key.g);
const db = Math.abs(data[i + 2] - key.b);
if (Math.max(dr, dg, db) > MASK_CHANNEL_THRESHOLD) {
mask[y * width + x] = 1;
if (x < left) left = x;
if (x > right) right = x;
if (y < top) top = y;
if (y > bottom) bottom = y;
}
}
}
const bbox = right >= 0 ? { left, top, right: right + 1, bottom: bottom + 1 } : null;
// maskKeyHex rides along so the composite can decontaminate fringe pixels
// when it re-keys onto a DIFFERENT canvas key (see normalizeFromAnalysis).
return { data, width, height, mask, bbox, maskKey: maskKeyHex };
}
const clampByte = (v) => Math.max(0, Math.min(255, Math.round(v)));
/**
* Build the per-pixel fringe decontaminator for a mask-key → canvas-key swap,
* or null when no swap happens (keys equal, or the mask key is unknown). Reuses
* the shared `keyShareFn` unmix (the same one recoverAlphaFrame runs) to measure
* how much old key each foreground pixel still carries, then shifts it by
* share·(newKey − oldKey) so the old key is replaced by the correct amount of
* the new one — the opaque source-over analogue of recoverAlphaFrame's straight-
* alpha recomposite.
*/
function buildKeyDecontaminator(maskKeyHex, canvasKeyHex) {
if (typeof maskKeyHex !== 'string' || maskKeyHex.toUpperCase() === canvasKeyHex.toUpperCase()) {
return null;
}
const oldKey = hexToRgb(maskKeyHex);
const newKey = hexToRgb(canvasKeyHex);
return {
delta: [newKey.r - oldKey.r, newKey.g - oldKey.g, newKey.b - oldKey.b],
share: keyShareFn([oldKey.r, oldKey.g, oldKey.b], keyChannelSplit(maskKeyHex)),
};
}
/**
* Write one masked foreground pixel into the destination canvas, shifting it
* off the old key when the composite re-keys onto a different one. Shared by
* both compositors (the reframing `normalizeFromAnalysis` and the
* geometry-preserving `recompositeOnKey`) so the fringe math — the subtlest
* code in this module — has exactly one definition.
*/
function writeForegroundPixel(canvas, dstI, data, srcI, decon) {
if (!decon) {
canvas[dstI] = data[srcI];
canvas[dstI + 1] = data[srcI + 1];
canvas[dstI + 2] = data[srcI + 2];
return;
}
const share = decon.share(data, srcI);
canvas[dstI] = clampByte(data[srcI] + share * decon.delta[0]);
canvas[dstI + 1] = clampByte(data[srcI + 1] + share * decon.delta[1]);
canvas[dstI + 2] = clampByte(data[srcI + 2] + share * decon.delta[2]);
}
/**
* Histogram the foreground (non-key) pixels of an analysis, quantized to 4
* bits per channel so anti-aliased shades collapse into their parent color.
* Returns `[{ r, g, b, count }]` sorted by count desc — pickChromaKey's input.
*/
export function paletteFromAnalysis({ data, width, height, mask }) {
const counts = new Map();
for (let p = 0; p < width * height; p++) {
if (!mask[p]) continue;
const i = p * 3;
const bucket = ((data[i] >> 4) << 8) | ((data[i + 1] >> 4) << 4) | (data[i + 2] >> 4);
counts.set(bucket, (counts.get(bucket) || 0) + 1);
}
return [...counts.entries()]
.map(([bucket, count]) => ({
// Bucket midpoint (e.g. 0xF? → 0xF8) so hue math sees representative values.
r: (((bucket >> 8) & 0xf) << 4) | 0x8,
g: (((bucket >> 4) & 0xf) << 4) | 0x8,
b: ((bucket & 0xf) << 4) | 0x8,
count,
}))
.sort((a, b) => b.count - a.count);
}
/** One-shot palette extraction (decode + histogram). */
export async function extractForegroundPalette(src, maskKeyHex) {
return paletteFromAnalysis(await analyzeForeground(src, maskKeyHex));
}
/**
* Composite a pre-analyzed candidate onto the canonical key-color square.
* `src` is still needed for the no-foreground copy-through path.
*/
export async function normalizeFromAnalysis(analysis, src, dest, canvasKeyHex) {
const { data, width, mask, bbox, maskKey } = analysis;
if (!bbox) {
await copyFileGuarded(src, dest);
return { copiedThrough: true };
}
const charW = bbox.right - bbox.left;
const charH = bbox.bottom - bbox.top;
const side = Math.max(
Math.round(charH / FRAME_HEIGHT_FRAC),
Math.round(charW / (1 - 2 * FRAME_SIDE_FRAC)),
);
const fill = hexToRgb(canvasKeyHex);
const canvas = Buffer.alloc(side * side * 3, Buffer.from([fill.r, fill.g, fill.b]));
const offsetX = Math.floor((side - charW) / 2);
const feetY = side - Math.round(side * FRAME_BOTTOM_FRAC);
const offsetY = feetY - charH;
// Non-null only when the composite re-keys onto a different canvas — then the
// anti-aliased fringe that survived the hard mask is decontaminated of the
// old key it still carries; otherwise foreground pixels copy through verbatim.
const decon = buildKeyDecontaminator(maskKey, canvasKeyHex);
for (let y = bbox.top; y < bbox.bottom; y++) {
for (let x = bbox.left; x < bbox.right; x++) {
if (!mask[y * width + x]) continue;
const srcI = (y * width + x) * 3;
const dstI = ((offsetY + (y - bbox.top)) * side + offsetX + (x - bbox.left)) * 3;
writeForegroundPixel(canvas, dstI, data, srcI, decon);
}
}
await sharp(canvas, { raw: { width: side, height: side, channels: 3 } }).png().toFile(dest);
return { side, charW, charH };
}
/**
* Re-key an analyzed image onto `canvasKeyHex` WITHOUT reframing it (issue
* #2979). Same canvas dimensions, every non-foreground pixel replaced by the
* canonical key, surviving fringe run through the same decontaminator
* normalizeFromAnalysis uses.
*
* This is the turnaround sheet's lock path: `normalizeFromAnalysis`'s
* single-figure geometry (character at 80% of a square, feet 7% up) is
* meaningless for a multi-figure model sheet, but the key swap still matters —
* the locked sheet is the i2i init image for the main and every anchor, whose
* prompts name the canonical key, so a sheet still wearing its generation key
* would contradict the instruction it is attached to.
*
* A no-op re-encode when the mask key already equals the canvas key (the
* decontaminator is null and the fill matches), which is the common case for a
* legacy record whose key was frozen before the sheet was ever generated.
*/
export async function recompositeOnKey(analysis, src, dest, canvasKeyHex) {
const { data, width, height, mask, bbox, maskKey } = analysis;
if (!bbox) {
await copyFileGuarded(src, dest);
return { copiedThrough: true };
}
const fill = hexToRgb(canvasKeyHex);
const canvas = Buffer.alloc(width * height * 3, Buffer.from([fill.r, fill.g, fill.b]));
const decon = buildKeyDecontaminator(maskKey, canvasKeyHex);
for (let p = 0; p < width * height; p++) {
if (!mask[p]) continue;
// Geometry is preserved, so source and destination indices coincide.
writeForegroundPixel(canvas, p * 3, data, p * 3, decon);
}
await sharp(canvas, { raw: { width, height, channels: 3 } }).png().toFile(dest);
return { width, height };
}
/**
* One-shot normalize: mask `src` against the key it was GENERATED on, then
* composite onto the SELECTED key. An image with no detectable foreground
* copies through unchanged, mirroring the source behavior.
*/
export async function normalizeAnchorFrame(src, dest, { maskKeyHex, canvasKeyHex }) {
return normalizeFromAnalysis(await analyzeForeground(src, maskKeyHex), src, dest, canvasKeyHex);
}