Skip to content

Self-hosted, Self-Improving, Federated Everything App — dev management, agent orchestration, persistent mind, digital twin, second brain, and creative pipeline from a single dashboard over Tailscale to port the user into the OS

12.6 KB
Raw Blame
/**
 * Sprites — reference-frame normalization + palette extraction (issue #2896).
 *
 * Node/sharp port of the source pipeline's `normalize_anchor_frame` (Pillow):
 * mask the character out of the generated candidate by per-pixel difference
 * from the background key color, then re-composite ONLY the masked pixels
 * onto a fresh solid-key square canvas. Because the composite always starts
 * from a clean canvas, switching key colors at lock time (mask on the
 * generation key, fill with the selected key) is free — that's what makes the
 * dynamic chroma-key selection possible without regenerating.
 *
 * Geometry contract (verbatim from the source): character height is 80% of
 * the square side (or width fits inside 10% side margins, whichever needs a
 * bigger canvas), feet baseline sits 7% above the bottom edge, pixels are
 * never rescaled — only the canvas is sized around them. Mask threshold: a
 * pixel is foreground when max-channel |pixel − key| > 40 — a DELIBERATE
 * deviation from the source's luma metric, which is blind to black-vs-blue
 * (see MASK_CHANNEL_THRESHOLD).
 *
 * Fringe decontamination: the hard mask keeps anti-aliased edge pixels
 * that are a BLEND of the character and the generation key. When lock swaps the
 * key (dynamic selection composites onto a key different from the one the
 * candidate was masked against), those fringe pixels still carry the OLD key's
 * tint — a magenta ring on a magenta→blue re-key. Each surviving foreground
 * pixel is shifted by keyShare·(newKey − oldKey), the same straight-alpha
 * recomposite `recoverAlphaFrame` performs, so the residual old key is replaced
 * by the correct amount of the new key. A no-op when the keys match.
 */

import sharp from 'sharp';
import { tryReadFile, copyFileGuarded } from '../../lib/fileUtils.js';
import { hexToRgb, keyChannelSplit, keyShareFn } from './chromaKey.js';
import { describeFrameStats, isDegenerateFrame } from '../../lib/imageFrameStats.js';
import { ServerError } from '../../lib/errorHandler.js';

const FRAME_HEIGHT_FRAC = 0.80;
const FRAME_BOTTOM_FRAC = 0.07;
const FRAME_SIDE_FRAC = 0.10;
// Max-channel distance from the key, NOT the source pipeline's luma-of-diff:
// luma weights blue at 0.114, so against the blue key a BLACK pixel scores
// 255·0.114 ≈ 29 — under any usable threshold — and black outlines/hair
// would be silently erased. Luma was safe only because the source pipeline
// hardcoded magenta; the dynamic key set (#2895) makes the metric wrong.
// Max-channel treats all three keys symmetrically (black vs any pure key
// differs by 255 in at least one channel).
const MASK_CHANNEL_THRESHOLD = 40;

// Re-export for existing consumers/tests; the definition lives in the pure
// color-math module so chromaKey.js can use it without importing sharp.
export { hexToRgb };

/**
 * Decode `src` as flat RGB (alpha dropped over white like Pillow's
 * convert("RGB"); generated candidates are opaque PNGs so this is a no-op in
 * practice) and compute the foreground mask + tight bounding box vs the key
 * color. The lock path runs palette extraction AND normalization off ONE
 * analysis so a multi-MP candidate is decoded and scanned once.
 */
const DEGENERATE_FRAME_DETAIL = {
  'solid-fill': 'a single flat color',
  'fully-transparent': 'fully transparent',
  'near-empty': 'almost no detail',
};

/**
 * Degenerate-frame gate for a sprite candidate (issue #4173).
 *
 * A blank candidate has no foreground at all, so `normalizeFromAnalysis` /
 * `recompositeOnKey` copy it straight through and it gets locked, atlased and
 * published as a real reference frame. Refuse it up front instead.
 *
 * Only a MEASURED degenerate verdict rejects: a candidate whose stats could
 * not be computed (`ok: null`) falls through untouched, so an undecodable file
 * still reaches the existing `imageError` / `INVALID_IMAGE` handling rather
 * than being mislabeled as empty.
 */
export async function assertFrameHasContent(src) {
  const stats = await describeFrameStats(src);
  if (!isDegenerateFrame(stats)) return stats;
  const detail = DEGENERATE_FRAME_DETAIL[stats.reason] || 'no content';
  throw new ServerError(
    `Candidate frame has no content (${detail}) — regenerate it before locking.`,
    { status: 422, code: 'DEGENERATE_FRAME' },
  );
}

export async function analyzeForeground(src, maskKeyHex) {
  // Read a path ONCE and hand the same bytes to the content gate and the decode
  // it guards, the read-once-verify-in-memory seam the compiler already keeps
  // (atlas.js): passing the path twice re-reads the file between the check and
  // the use, and also defeats `describeFrameStats`'s content-addressed memo,
  // which only keys on buffers (a path is not content). A read that fails falls
  // through to the path unchanged so sharp still raises its own decode error —
  // the existing INVALID_IMAGE surface, not a new ENOENT.
  const source = Buffer.isBuffer(src) ? src : (await tryReadFile(src, null)) ?? src;
  await assertFrameHasContent(source);
  const key = hexToRgb(maskKeyHex);
  const { data, info } = await sharp(source)
    .flatten({ background: { r: 255, g: 255, b: 255 } })
    .raw()
    .toBuffer({ resolveWithObject: true });
  const { width, height } = info;
  const mask = new Uint8Array(width * height);
  let left = width; let top = height; let right = -1; let bottom = -1;
  for (let y = 0; y < height; y++) {
    for (let x = 0; x < width; x++) {
      const i = (y * width + x) * 3;
      const dr = Math.abs(data[i] - key.r);
      const dg = Math.abs(data[i + 1] - key.g);
      const db = Math.abs(data[i + 2] - key.b);
      if (Math.max(dr, dg, db) > MASK_CHANNEL_THRESHOLD) {
        mask[y * width + x] = 1;
        if (x < left) left = x;
        if (x > right) right = x;
        if (y < top) top = y;
        if (y > bottom) bottom = y;
      }
    }
  }
  const bbox = right >= 0 ? { left, top, right: right + 1, bottom: bottom + 1 } : null;
  // maskKeyHex rides along so the composite can decontaminate fringe pixels
  // when it re-keys onto a DIFFERENT canvas key (see normalizeFromAnalysis).
  return { data, width, height, mask, bbox, maskKey: maskKeyHex };
}

const clampByte = (v) => Math.max(0, Math.min(255, Math.round(v)));

/**
 * Build the per-pixel fringe decontaminator for a mask-key → canvas-key swap,
 * or null when no swap happens (keys equal, or the mask key is unknown). Reuses
 * the shared `keyShareFn` unmix (the same one recoverAlphaFrame runs) to measure
 * how much old key each foreground pixel still carries, then shifts it by
 * share·(newKey − oldKey) so the old key is replaced by the correct amount of
 * the new one — the opaque source-over analogue of recoverAlphaFrame's straight-
 * alpha recomposite.
 */
function buildKeyDecontaminator(maskKeyHex, canvasKeyHex) {
  if (typeof maskKeyHex !== 'string' || maskKeyHex.toUpperCase() === canvasKeyHex.toUpperCase()) {
    return null;
  }
  const oldKey = hexToRgb(maskKeyHex);
  const newKey = hexToRgb(canvasKeyHex);
  return {
    delta: [newKey.r - oldKey.r, newKey.g - oldKey.g, newKey.b - oldKey.b],
    share: keyShareFn([oldKey.r, oldKey.g, oldKey.b], keyChannelSplit(maskKeyHex)),
  };
}

/**
 * Write one masked foreground pixel into the destination canvas, shifting it
 * off the old key when the composite re-keys onto a different one. Shared by
 * both compositors (the reframing `normalizeFromAnalysis` and the
 * geometry-preserving `recompositeOnKey`) so the fringe math — the subtlest
 * code in this module — has exactly one definition.
 */
function writeForegroundPixel(canvas, dstI, data, srcI, decon) {
  if (!decon) {
    canvas[dstI] = data[srcI];
    canvas[dstI + 1] = data[srcI + 1];
    canvas[dstI + 2] = data[srcI + 2];
    return;
  }
  const share = decon.share(data, srcI);
  canvas[dstI] = clampByte(data[srcI] + share * decon.delta[0]);
  canvas[dstI + 1] = clampByte(data[srcI + 1] + share * decon.delta[1]);
  canvas[dstI + 2] = clampByte(data[srcI + 2] + share * decon.delta[2]);
}

/**
 * Histogram the foreground (non-key) pixels of an analysis, quantized to 4
 * bits per channel so anti-aliased shades collapse into their parent color.
 * Returns `[{ r, g, b, count }]` sorted by count desc — pickChromaKey's input.
 */
export function paletteFromAnalysis({ data, width, height, mask }) {
  const counts = new Map();
  for (let p = 0; p < width * height; p++) {
    if (!mask[p]) continue;
    const i = p * 3;
    const bucket = ((data[i] >> 4) << 8) | ((data[i + 1] >> 4) << 4) | (data[i + 2] >> 4);
    counts.set(bucket, (counts.get(bucket) || 0) + 1);
  }
  return [...counts.entries()]
    .map(([bucket, count]) => ({
      // Bucket midpoint (e.g. 0xF? → 0xF8) so hue math sees representative values.
      r: (((bucket >> 8) & 0xf) << 4) | 0x8,
      g: (((bucket >> 4) & 0xf) << 4) | 0x8,
      b: ((bucket & 0xf) << 4) | 0x8,
      count,
    }))
    .sort((a, b) => b.count - a.count);
}

/** One-shot palette extraction (decode + histogram). */
export async function extractForegroundPalette(src, maskKeyHex) {
  return paletteFromAnalysis(await analyzeForeground(src, maskKeyHex));
}

/**
 * Composite a pre-analyzed candidate onto the canonical key-color square.
 * `src` is still needed for the no-foreground copy-through path.
 */
export async function normalizeFromAnalysis(analysis, src, dest, canvasKeyHex) {
  const { data, width, mask, bbox, maskKey } = analysis;
  if (!bbox) {
    await copyFileGuarded(src, dest);
    return { copiedThrough: true };
  }
  const charW = bbox.right - bbox.left;
  const charH = bbox.bottom - bbox.top;
  const side = Math.max(
    Math.round(charH / FRAME_HEIGHT_FRAC),
    Math.round(charW / (1 - 2 * FRAME_SIDE_FRAC)),
  );
  const fill = hexToRgb(canvasKeyHex);
  const canvas = Buffer.alloc(side * side * 3, Buffer.from([fill.r, fill.g, fill.b]));
  const offsetX = Math.floor((side - charW) / 2);
  const feetY = side - Math.round(side * FRAME_BOTTOM_FRAC);
  const offsetY = feetY - charH;
  // Non-null only when the composite re-keys onto a different canvas — then the
  // anti-aliased fringe that survived the hard mask is decontaminated of the
  // old key it still carries; otherwise foreground pixels copy through verbatim.
  const decon = buildKeyDecontaminator(maskKey, canvasKeyHex);
  for (let y = bbox.top; y < bbox.bottom; y++) {
    for (let x = bbox.left; x < bbox.right; x++) {
      if (!mask[y * width + x]) continue;
      const srcI = (y * width + x) * 3;
      const dstI = ((offsetY + (y - bbox.top)) * side + offsetX + (x - bbox.left)) * 3;
      writeForegroundPixel(canvas, dstI, data, srcI, decon);
    }
  }
  await sharp(canvas, { raw: { width: side, height: side, channels: 3 } }).png().toFile(dest);
  return { side, charW, charH };
}

/**
 * Re-key an analyzed image onto `canvasKeyHex` WITHOUT reframing it (issue
 * #2979). Same canvas dimensions, every non-foreground pixel replaced by the
 * canonical key, surviving fringe run through the same decontaminator
 * normalizeFromAnalysis uses.
 *
 * This is the turnaround sheet's lock path: `normalizeFromAnalysis`'s
 * single-figure geometry (character at 80% of a square, feet 7% up) is
 * meaningless for a multi-figure model sheet, but the key swap still matters —
 * the locked sheet is the i2i init image for the main and every anchor, whose
 * prompts name the canonical key, so a sheet still wearing its generation key
 * would contradict the instruction it is attached to.
 *
 * A no-op re-encode when the mask key already equals the canvas key (the
 * decontaminator is null and the fill matches), which is the common case for a
 * legacy record whose key was frozen before the sheet was ever generated.
 */
export async function recompositeOnKey(analysis, src, dest, canvasKeyHex) {
  const { data, width, height, mask, bbox, maskKey } = analysis;
  if (!bbox) {
    await copyFileGuarded(src, dest);
    return { copiedThrough: true };
  }
  const fill = hexToRgb(canvasKeyHex);
  const canvas = Buffer.alloc(width * height * 3, Buffer.from([fill.r, fill.g, fill.b]));
  const decon = buildKeyDecontaminator(maskKey, canvasKeyHex);
  for (let p = 0; p < width * height; p++) {
    if (!mask[p]) continue;
    // Geometry is preserved, so source and destination indices coincide.
    writeForegroundPixel(canvas, p * 3, data, p * 3, decon);
  }
  await sharp(canvas, { raw: { width, height, channels: 3 } }).png().toFile(dest);
  return { width, height };
}

/**
 * One-shot normalize: mask `src` against the key it was GENERATED on, then
 * composite onto the SELECTED key. An image with no detectable foreground
 * copies through unchanged, mirroring the source behavior.
 */
export async function normalizeAnchorFrame(src, dest, { maskKeyHex, canvasKeyHex }) {
  return normalizeFromAnalysis(await analyzeForeground(src, maskKeyHex), src, dest, canvasKeyHex);
}