Skip to content

Open-source platform for 3D AI agents. Turn text or a photo into a rigged, animated GLB avatar, give it an LLM brain, memory and a wallet, and embed it anywhere with one web component. Remote MCP server, x402 payments, Three.js, browser-native.

9.0 KB
Raw Blame
/**
 * Selfie capture gates: the single source of truth for every quality threshold
 * the /create/selfie viewfinder and photo assessment enforce, plus the pure
 * scoring functions that apply them. No DOM, no MediaPipe: everything here is
 * unit-testable math over numbers the callers extract from frames.
 *
 * Why these numbers and not invented ones: each threshold is traced to a way
 * the reconstruction pipeline actually fails, so the client rejects for the
 * same reasons the backend would. Sources of truth, in the repo:
 *
 * - "no face" is the native worker's ONLY hard input rejection
 *   (workers/avatar-reconstruction/face_pipeline.py raises
 *   InputError("no face detected in any of the provided photos") after a
 *   FaceMesh pass at min_detection_confidence=0.4 and a BlazeFace small-face
 *   rescue at 0.3). The client mirrors it as the only capture-blocking gate.
 * - Yaw: face_pipeline.MAX_MORPH_YAW_DEG = 35. Above ~35 degrees the worker
 *   skips the geometry morph because the self-occluded half of the face is
 *   depth extrapolation, not structure (eval/detection_guard.py: clean
 *   frontals peak at 2.7 degrees, the one benchmark input that breaks the
 *   morph band reads 58.6). The frontal window of +/-15 keeps captures deep
 *   inside the morph-safe zone; side slots target the ~45 degree band the
 *   multi-view texture pass expects.
 * - Blur: the provider reconstruction lanes fail jobs whose error text the
 *   status mapper (src/selfie-pipeline.js friendlyJobError) matches on
 *   "blur", and the worker's robustness benchmark
 *   (workers/avatar-reconstruction/eval/robustness.py `blurred`) models the
 *   defect as a Gaussian blur of radius longestEdge/220. BLUR_STDDEV_MIN is
 *   the live-feed equivalent: the Laplacian response stddev of a 64px face
 *   crop. The value is measured, not assumed. Six real portrait photographs
 *   were framed as 720x1280 phone selfies and scored through this module's
 *   own math, sharp against the benchmark's exact degradations:
 *
 *     sharp                       22.8  23.4  23.7  27.4  38.3  39.4
 *     GaussianBlur r=longest/220  10.6  12.6  13.8  14.0  14.8  15.7
 *     GaussianBlur r=10            6.3   6.8   8.2   8.4   8.5   9.4
 *
 *   The previous floor of 3.5 sat below every one of those readings, so the
 *   gate never fired on any real photograph, however badly blurred: it told
 *   users "Sharp" on a frame smeared past recognition. 12 sits in the empty
 *   band between the blurred and sharp populations, rejecting every
 *   heavily-blurred sample while leaving the least-sharp real face nearly
 *   2x of headroom. The mild benchmark degradation straddles it on purpose:
 *   it costs identity stability but is not a reconstruction failure, and
 *   over-rejecting a usable capture is worse than accepting a soft one.
 *   SHARPNESS_VAR_MIN = 90 is the same idea at still-photo resolution
 *   (variance, not stddev, at up to 1024px), the threshold
 *   src/selfie-refine.js has always applied at submit time.
 * - Lighting: the robustness benchmark's `dim` degradation (brightness x0.42,
 *   "indoor evening") lands a typical selfie's mean face luma at the LUMA_MIN
 *   boundary, and the adversarial set's "hard-backlight, face in deep shadow"
 *   case is the classic path to the worker's no-face InputError. LUMA_MAX
 *   catches the blown-highlight mirror image (a washed-out face loses the
 *   texture the UV warp samples).
 * - Centering: the refine stage (src/selfie-refine.js computeSubjectFrame)
 *   reframes to a head-and-shoulders square around the detected face, so a
 *   badly off-centre face costs crop headroom rather than failing outright.
 *   CENTER_TOL_X/Y bound the nose offset so the reframe never has to choose
 *   between cutting the head and leaving the subject at the frame edge.
 */

/** Yaw windows per capture slot, in degrees. Positive = looking left. */
export const SLOT_PRESETS = Object.freeze({
	frontal: Object.freeze({ label: 'Frontal', min: -15, max: 15 }),
	left: Object.freeze({ label: 'Left ~45°', min: 30, max: 60 }),
	right: Object.freeze({ label: 'Right ~45°', min: -60, max: -30 }),
});

export const GATES = Object.freeze({
	/** Laplacian-response stddev floor on the 64px live face crop. */
	BLUR_STDDEV_MIN: 12,
	/** Variance-of-Laplacian floor for still photos at up to 1024px. */
	SHARPNESS_VAR_MIN: 90,
	/** Mean face luma bounds (Rec. 601, 0..255). */
	LUMA_MIN: 40,
	LUMA_MAX: 218,
	/** Max share of the face crop allowed to be blown to near-white. */
	CLIPPED_FRAC_MAX: 0.18,
	/** A face pixel at or above this luma carries no recoverable texture. */
	CLIP_LEVEL: 250,
	/** Max nose offset from frame centre, as a fraction of frame size. */
	CENTER_TOL_X: 0.22,
	CENTER_TOL_Y: 0.28,
	/** Worker skips the geometry morph above this head yaw (degrees). */
	MORPH_YAW_MAX: 35,
});

/**
 * Mean luma, blown-highlight share, and Laplacian-response stddev of an 8-bit
 * grey buffer. This is the live viewfinder's blur/lighting measurement,
 * extracted pure so it can be pinned by unit tests. `blurStddev` is the
 * standard deviation of a 4-neighbour Laplacian over the interior pixels:
 * sharp facial detail yields strong high-frequency response, defocus and
 * motion blur collapse it. `clippedFrac` is the share of pixels at or above
 * CLIP_LEVEL, which mean luma cannot see: a face half-blown by window glare
 * still averages into the accepted band while carrying no texture where it
 * counts, and clipping raises the Laplacian response rather than lowering it,
 * so the blur gate cannot catch it either.
 *
 * @param {Uint8Array|Uint8ClampedArray|Float32Array|number[]} grey row-major, length w*h
 * @param {number} w
 * @param {number} h
 * @returns {{ luma: number, blurStddev: number, clippedFrac: number }}
 */
export function grayFaceStats(grey, w, h) {
	const n = w * h;
	if (n === 0) return { luma: 0, blurStddev: 0, clippedFrac: 0 };
	let lumaSum = 0;
	let clipped = 0;
	for (let i = 0; i < n; i++) {
		lumaSum += grey[i];
		if (grey[i] >= GATES.CLIP_LEVEL) clipped++;
	}
	const luma = lumaSum / n;
	const clippedFrac = clipped / n;
	if (w < 3 || h < 3) return { luma, blurStddev: 0, clippedFrac };
	let ls = 0;
	let ls2 = 0;
	let ln = 0;
	for (let row = 1; row < h - 1; row++) {
		for (let col = 1; col < w - 1; col++) {
			const c = grey[row * w + col];
			const lap =
				4 * c -
				grey[(row - 1) * w + col] -
				grey[(row + 1) * w + col] -
				grey[row * w + (col - 1)] -
				grey[row * w + (col + 1)];
			ls += lap;
			ls2 += lap * lap;
			ln++;
		}
	}
	const mean = ls / ln;
	const blurStddev = Math.sqrt(Math.max(0, ls2 / ln - mean * mean));
	return { luma, blurStddev, clippedFrac };
}

/**
 * Grade one live viewfinder frame against the capture gates and name exactly
 * what is wrong, in the order the user should fix it. Pure.
 *
 * @param {{
 *   faceFound: boolean,
 *   slot?: 'frontal'|'left'|'right'|string,
 *   yaw?: number|null,
 *   noseX?: number|null, noseY?: number|null,
 *   blur?: number,
 *   luma?: number,
 *   clippedFrac?: number,
 * }} m `blur` is the grayFaceStats blurStddev; noseX/noseY are normalised 0..1.
 * @returns {{
 *   faceFound: boolean, yawOk: boolean, centered: boolean,
 *   blurOk: boolean, lumaOk: boolean, allPass: boolean,
 *   reason: string|null,
 * }} `reason` is the user-facing retake prompt, null when every gate passes.
 */
export function gradeFrame(m) {
	if (!m.faceFound) {
		return {
			faceFound: false,
			yawOk: false,
			centered: false,
			blurOk: false,
			lumaOk: false,
			allPass: false,
			reason: 'No face detected. Face the camera.',
		};
	}
	const slot = m.slot || 'frontal';
	const cfg = SLOT_PRESETS[slot] || SLOT_PRESETS.frontal;
	const yaw = typeof m.yaw === 'number' ? m.yaw : 0;
	const yawOk = yaw >= cfg.min && yaw <= cfg.max;
	const centered =
		typeof m.noseX === 'number' &&
		typeof m.noseY === 'number' &&
		Math.abs(m.noseX - 0.5) < GATES.CENTER_TOL_X &&
		Math.abs(m.noseY - 0.5) < GATES.CENTER_TOL_Y;
	const blurOk = (m.blur ?? 0) >= GATES.BLUR_STDDEV_MIN;
	const tooDark = (m.luma ?? 0) < GATES.LUMA_MIN;
	const blownOut =
		(m.luma ?? 0) > GATES.LUMA_MAX || (m.clippedFrac ?? 0) > GATES.CLIPPED_FRAC_MAX;
	const lumaOk = !tooDark && !blownOut;

	let reason = null;
	if (!yawOk) {
		reason =
			slot === 'frontal'
				? 'Face the camera straight on.'
				: `Turn your head ${slot === 'left' ? 'left' : 'right'} about 45°.`;
	} else if (!centered) {
		reason = 'Center your face in the oval.';
	} else if (tooDark) {
		// Lighting is named before blur on purpose. A dim frame carries little
		// contrast, so its Laplacian response collapses too and it trips the
		// blur gate as well: telling that user to "hold steady" sends them to
		// fix the symptom while the room stays dark. Light is the root cause,
		// and fixing it usually clears the blur reading with it.
		reason = 'Too dark. Find better light.';
	} else if (blownOut) {
		reason = 'Too bright. Move out of direct light or turn away from the window.';
	} else if (!blurOk) {
		reason = 'Hold steady. The image is blurry.';
	}

	return {
		faceFound: true,
		yawOk,
		centered,
		blurOk,
		lumaOk,
		allPass: yawOk && centered && blurOk && lumaOk,
		reason,
	};
}