Self-hosted, Self-Improving, Federated Everything App — dev management, agent orchestration, persistent mind, digital twin, second brain, and creative pipeline from a single dashboard over Tailscale to port the user into the OS
JavaScript ai-agentsdashboarddeveloper-toolsdigital-twinquantified-selfsecond-brainself-hostedtailscale
/**
* Frame-quality scoring — picks the video continuation anchor by measuring
* candidate frames instead of trusting wherever an end-seek happens to land.
*
* A chained render conditions chunk N+1 on ONE frame taken from the tail of
* chunk N. Local video output is motion-heavy by construction (the prompt
* guidance explicitly asks for a camera move and a visible action), so that
* frame is frequently motion-blurred, mid-blink, or inside a fade — and the
* next chunk inherits the blur as the scene's actual content, compounding
* through every subsequent hop.
*
* The scoring is deliberately cheap and dependency-free beyond `sharp` (already
* a server dependency): decode each candidate to a raw single-channel buffer,
* then combine three terms.
*
* focus — log-compressed variance of the first-difference gradient.
* Log compression is what keeps one exceptionally sharp frame
* from swamping the exposure term.
* exposure — penalty on the distance of mean luma from mid-grey, which is
* what rejects a frame inside a fade-to-black or a blown highlight.
* recency — a small linear preference for later frames, so among
* comparably clean candidates the anchor stays as close to the
* cut as possible. Without it the scorer drifts toward the calm
* middle of the window and continuity loosens.
*
* Usability is a SEPARATE question from ranking. `usable` gates on raw gradient
* variance alone, so only a mathematically degenerate frame (a solid fill, a
* fully faded one) is rejected outright and a dark-but-real low-key tail still
* qualifies. Exposure shapes the RANKING, not the gate.
*/
import sharp from 'sharp';
// Tail window the candidates are decoded from, and how densely.
//
// The window is deliberately capped at the reach of the single `-sseof -1.0`
// seek this replaces. Focus tends to be HIGHEST at the oldest candidate (motion,
// and therefore blur, accumulates toward the cut), so a wider window would let
// the scorer routinely anchor further from the cut than the old behavior did —
// widening the backward jump at every chain seam instead of tightening it. The
// recency bonus biases toward the cut but cannot bound the window on its own.
export const TAIL_WINDOW_SECONDS = 1.0;
export const CANDIDATE_FPS = 12;
// Derived, never an independent number: the ffmpeg-side cap (`-frames:v`)
// truncates from the NEWEST end, which is exactly the frames the recency term
// prefers. If the two ever disagreed, the cap would silently discard the pick.
export const MAX_CANDIDATES = Math.round(TAIL_WINDOW_SECONDS * CANDIDATE_FPS);
// Gradient variance that maps to a focus of 1.0. Real sharp frames land in the
// low thousands, so this leaves headroom without clamping ordinary content.
export const FOCUS_SCALE = 20000;
export const EXPOSURE_WEIGHT = 0.6;
export const RECENCY_WEIGHT = 0.15;
// Gradient variance below which a frame carries no signal at all — a solid
// fill, a fully faded frame, or a decode that produced nothing.
//
// This, NOT the composite score, is what `usable` gates on. Gating on a
// score that includes the exposure penalty would reject a legitimately dark,
// low-key tail (mean luma 0.12 alone costs ~0.46 of quality) and silently
// revert every night scene to the unscored end seek. Same property
// `imageFrameStats.js` commits to: reject only the mathematically degenerate
// case, never dark-but-real content. Exposure still does its job in the
// RANKING, where a fade-to-black loses to its neighbours.
//
// 0.25 is (0.5 levels)^2 — half of one 8-bit level, below the smallest
// difference an encoder can represent, mirroring `SOLID_FILL_STDEV_EPSILON`.
export const MIN_SIGNAL_VARIANCE = 0.25;
const clamp01 = (n) => (n < 0 ? 0 : (n > 1 ? 1 : n));
const isFrame = (frame) => !!frame
&& Number.isInteger(frame.width) && Number.isInteger(frame.height)
&& frame.width > 0 && frame.height > 0
&& frame.data?.length >= frame.width * frame.height;
/**
* Variance of the horizontal + vertical first differences over a raw
* single-channel buffer. Higher = sharper. Both difference sets are pooled
* into one sample population so a frame blurred along only one axis (the
* common camera-pan case) still scores low.
*/
export function gradientVariance(frame) {
if (!isFrame(frame)) return 0;
const { width, height, data } = frame;
let sum = 0;
let sumSq = 0;
let n = 0;
for (let y = 0; y < height; y++) {
const row = y * width;
for (let x = 0; x < width; x++) {
const i = row + x;
if (x + 1 < width) {
const dx = data[i + 1] - data[i];
sum += dx; sumSq += dx * dx; n++;
}
if (y + 1 < height) {
const dy = data[i + width] - data[i];
sum += dy; sumSq += dy * dy; n++;
}
}
}
if (!n) return 0;
const mean = sum / n;
return Math.max(0, sumSq / n - mean * mean);
}
/** Mean luma of a raw single-channel buffer, normalized to 0..1. */
export function meanLuma(frame) {
if (!isFrame(frame)) return 0;
const { width, height, data } = frame;
const count = width * height;
let sum = 0;
for (let i = 0; i < count; i++) sum += data[i];
return sum / count / 255;
}
/**
* Score one decoded frame. `recency` is 0 for the oldest candidate in the
* window and 1 for the newest; a single candidate is treated as newest.
*
* Returns every component, not just the total, so a caller can log WHY a tail
* was rejected instead of just that it was.
*/
export function scoreFrame(frame, { recency = 1 } = {}) {
const variance = gradientVariance(frame);
const focus = clamp01(Math.log1p(variance) / Math.log1p(FOCUS_SCALE));
const luma = meanLuma(frame);
// Distance from mid-grey, normalized so pure black and pure white both hit 1.
const exposurePenalty = clamp01(Math.abs(luma - 0.5) / 0.5);
const r = clamp01(Number.isFinite(recency) ? recency : 0);
const quality = focus - EXPOSURE_WEIGHT * exposurePenalty;
return {
variance,
focus,
luma,
exposurePenalty,
recency: r,
quality,
score: quality + RECENCY_WEIGHT * r,
usable: variance >= MIN_SIGNAL_VARIANCE,
};
}
/**
* Decode an image file to a raw greyscale frame. Returns null on ANY decode
* failure — a candidate ffmpeg wrote partially (or not at all) is simply not a
* candidate, and must never abort a render the user already paid GPU time for.
*/
export async function decodeGreyscaleFrame(path, { sharpImpl = sharp } = {}) {
// On sharp 0.35.3 `.greyscale().raw()` already lands ONE channel even for an
// RGBA source (verified: byte-identical with and without removeAlpha). The
// explicit `removeAlpha()` makes that a stated requirement rather than an
// incidental property, and the channel check below is the backstop if it ever
// stops holding: an interleaved buffer is still ≥ width×height bytes, so the
// shape check alone would pass while gradientVariance read alpha samples as
// neighbouring pixels and meanLuma averaged half the image.
const decoded = await sharpImpl(path).removeAlpha().greyscale().raw()
.toBuffer({ resolveWithObject: true })
.catch(() => null);
if (!decoded?.info) return null;
const { width, height, channels } = decoded.info;
// Interleaved data must never reach the scorer; a null simply drops this
// candidate. `channels == null` means the decoder didn't report it, which is
// not evidence of a problem — sentinel discipline, not `!channels`.
if (channels != null && channels !== 1) return null;
const frame = { width, height, data: decoded.data };
return isFrame(frame) ? frame : null;
}
/**
* Score every candidate path (given in timeline order, oldest first) and
* return the best one, or null when nothing decoded or every candidate is
* degenerate. The returned `index` is the candidate's position in the input
* array AS PASSED — non-string entries are skipped, not renumbered — because
* the caller derives the anchor's time offset from it.
*/
export async function pickBestFrame(paths, { sharpImpl = sharp } = {}) {
const list = Array.isArray(paths) ? paths : [];
if (!list.length) return null;
let best = null;
// Indices are positions in the CALLER's array, never in a filtered copy —
// the caller derives the anchor's time offset from `index` against that same
// array, so a renumbering here would silently mis-report the offset. Skipped
// entries are holes in the timeline, not a reason to shift the rest.
//
// Decoded sequentially on purpose: only the SCORE outlives each frame, so
// one raw buffer is live at a time. Decoding the whole window at once would
// hold width × height bytes per candidate simultaneously, which scales with
// the render resolution for no wall-clock that matters next to a GPU render.
for (let i = 0; i < list.length; i++) {
if (typeof list[i] !== 'string' || !list[i]) continue;
// eslint-disable-next-line no-await-in-loop
const frame = await decodeGreyscaleFrame(list[i], { sharpImpl });
if (!frame) continue;
const recency = list.length === 1 ? 1 : i / (list.length - 1);
const scored = scoreFrame(frame, { recency });
if (!scored.usable) continue;
if (!best || scored.score > best.score) best = { path: list[i], index: i, ...scored };
}
return best;
}