LightCraft: An open-source non-destructive photo editor written by scratch in pure Rust
LR-PREF-PEOPLE: update pure-Rust recognition and preserve valid model replacements
Daniel Voshart24f87a91a011
diff --git a/Cargo.lock b/Cargo.lockindex 12e7c651c..07014d782 100644--- a/Cargo.lock+++ b/Cargo.lock@@ -2909,11 +2909,14 @@ dependencies = [name = "lightcraft-faces"version = "0.4.0"dependencies = [+ "matrixmultiply",+ "ndarray","proptest","serde","serde_json","sha2","thiserror 2.0.21",+ "web-time",][[package]]@@ -3188,6 +3191,16 @@ dependencies = ["libc",]+[[package]]+name = "matrixmultiply"+version = "0.3.11"+source = "registry+https://github.com/rust-lang/crates.io-index"+checksum = "3f607c237553f086e7043417a51df26b2eb899d3caff94e6a67592ff992fedc7"+dependencies = [+ "autocfg",+ "rawpointer",+]+[[package]]name = "maybe-rayon"version = "0.1.1"@@ -3341,6 +3354,21 @@ dependencies = ["thiserror 2.0.21",]+[[package]]+name = "ndarray"+version = "0.17.2"+source = "registry+https://github.com/rust-lang/crates.io-index"+checksum = "520080814a7a6b4a6e9070823bb24b4531daac8c4627e08ba5de8c5ef2f2752d"+dependencies = [+ "matrixmultiply",+ "num-complex",+ "num-integer",+ "num-traits",+ "portable-atomic",+ "portable-atomic-util",+ "rawpointer",+]+[[package]]name = "ndk"version = "0.9.0"@@ -4559,6 +4587,12 @@ dependencies = ["objc2-quartz-core 0.3.2",]+[[package]]+name = "rawpointer"+version = "0.2.1"+source = "registry+https://github.com/rust-lang/crates.io-index"+checksum = "60a357793950651c4ed0f3f52338f53b2f809f32d83a07f72909fa13e4c6c1e3"+[[package]]name = "rayon"version = "1.12.0"diff --git a/ROADMAP.md b/ROADMAP.mdindex 5e793e8be..980108329 100644--- a/ROADMAP.md+++ b/ROADMAP.md@@ -25,7 +25,7 @@ duplicate Local entries #22, black GPU exports on an Intel iGPU #78).| **Feature checklist** | 79.7% | P0 core and P1 nearly complete: import (Add / Copy / Move, templates, devices), library, grid/loupe/compare/survey, every Edit slider, curves, colour grading, masking tools, crop/Upright, heal/clone, presets/profiles, versions/history, sync, export, menus, shortcuts | P0: full-resolution zoom detail; P1: lens-profile database, content-aware fill (patch synthesis), video playback/trim || **RAW coverage** (formats people shoot) | ~55% | DNG (all kinds), CR2, CR3 lossless Bayer and version 0x100/0x200 C-RAW (M50/R100/R8 pixel-exact corpus), ARW, NEF (uncompressed + Huffman lossless/lossy), RAF (uncompressed, lossless/lossy compressed Bayer and X-Trans), uncompressed ORF, RW2 / RWL / Panasonic RAW (every raw format, checked on 178 files from 118 Panasonic and Leica bodies), PEF; every container's embedded preview (incl. CR3) | **CR3** (every Canon since ~2018), compressed ORF, Nikon lossy-after-split, Canon sRAW, HEIC/AVIF. NEFs labelled compressed but stored uncompressed (Z 6 packed 14-bit, D850 12-bit uncompressed). Per-model verification is thin (~55 corpus files vs >1,000 models) || **Colour & image quality** | ~55–65% | Pipeline is complete and fast; GPU path CPU-exact within 1/255 | **No measured camera calibration database**: ARW, NEF, RW2, RAF, CR3, CR2 and PEF have guarded camera-JPEG colour estimates, with bundled ILCE-7M4, X-H2S and X-T4 profiles (docs/camera-preview-colour.md); other non-DNG raws and rejected estimates use a neutral matrix. Colour fidelity remains incomplete. No lens-profile database. No measured fidelity against Lightroom (tone, highlights, texture/clarity, NR, sharpening are tuned by eye) |-| **AI & computational** | ~15–20% | Assisted culling (focus, bursts), auto tone, HDR/panorama merge; subject/sky/background masks as classical heuristics | Real segmentation masks (subject, sky, people, objects, landscape, depth), AI denoise, super resolution, lens blur, generative remove, faces/people, natural-language search. **Blocked on a model strategy** (licensable weights or our own training; pure-Rust inference is feasible) |+| **AI & computational** | ~15–20% | Assisted culling (focus, bursts), auto tone, HDR/panorama merge; subject/sky/background masks as classical heuristics; **faces (MVP, [docs/faces.md](docs/faces.md))**: a pure-Rust detector (a 232 KB opt-in download), opt-in recognition models (one-click download or add a file; they run on the shared pure-Rust CPU interpreter; pinned SFace/AuraFace embeddings agree with an independent reference runtime on 40 public crops; one-thread medians 34–36 / 233–252 ms (AuraFace remains expensive)), a background scan that finds and embeds faces, name suggestions, a person's page with look-alike faces to confirm, an Unnamed faces section (look-alikes together, select a group and name it at once) | Real segmentation masks (subject, sky, people, objects, landscape, depth), AI denoise, super resolution, lens blur, generative remove, natural-language search; for faces: clusters with separators between groups, small faces in large photos (the detector looks at 640 px), thresholds calibrated on live faces, writing face regions back to XMP. **Blocked on a model strategy** (licensable weights or our own training; pure-Rust inference is feasible) || **Workflow & library** | ~85% (single machine) | Robust catalog (journal + snapshots, background compaction, crash-tested), 85k-photo libraries stay responsive, Local browsing with automatic cleanup, XMP interop, keywords, smart albums, Move import | Opening an 85k library takes 1.7–4.7 s; no cloud sync (out of scope), no tablet companion (#74, roadmap), shared albums, publish services, tethering || **Classic modules** | ~30% | Geotagging from GPX track logs, soft proofing (partial), slideshow (basic) | **Map view, Book, Print, Slideshow module, Web, publish services**: ~40 tracker rows ⬜ || **HDR & video** | 0% | | HDR edit/display/export; video play/trim/edit/export |@@ -59,7 +59,7 @@ Priorities, in order. Each points at tracker rows in [`docs/parity.md`](docs/par(references stay in the local `plan/`), then tune against the numbers.5. **Lens profiles of our own** (LR-EDIT-OPTICS-PROFILE, P1).6. **AI model strategy** (maintainer decision): which permissively licensed models (or our own training) for- segmentation masks and denoise; then pure-Rust inference. Unblocks M12 and Enhance. Object / Describe masks now+ segmentation masks and denoise; then pure-Rust inference. Unblocks M12 and Enhance. Face detection and recognition run on our own checked Rust engine. Object / Describe masks nowrun Meta's SAM 3 in pure Rust (optional, user-consented download under the non-OSI SAM License; CDN mirrors stillto be set up, docs/ai-masks.md); Subject / Sky / People and Enhance are still open.7. **Then:** HDR (Q), the Classic output modules (Print first, then Map view, Book, Slideshow), video (R), localisationdiff --git a/apps/lightcraft-cli/Cargo.toml b/apps/lightcraft-cli/Cargo.tomlindex 3f60fe32a..1c4686d0a 100644--- a/apps/lightcraft-cli/Cargo.toml+++ b/apps/lightcraft-cli/Cargo.toml@@ -25,6 +25,7 @@ serde_json = { workspace = true }dhat = { workspace = true, optional = true }[features]+default = []# Heap profiling: `cargo run --release -p lightcraft-cli --features dhat-heap -- snapshot …` counts# every allocation (live/peak bytes in `library.memory` / `ui.inspect`) and writes dhat-heap.json# (open it in DHAT's viewer: allocation sites by bytes at the peak). Slows allocation-heavy work.diff --git a/apps/lightcraft/Cargo.toml b/apps/lightcraft/Cargo.tomlindex a29299da0..a0e36cb82 100644--- a/apps/lightcraft/Cargo.toml+++ b/apps/lightcraft/Cargo.toml@@ -9,6 +9,9 @@ default-run = "lightcraft"[lints]workspace = true+[features]+default = []+[dependencies]lightcraft-engine = { workspace = true, features = ["sam"] }lightcraft-sysmem = { workspace = true }diff --git a/crates/engine/src/cmd/face_detect.rs b/crates/engine/src/cmd/face_detect.rsindex 9235a10aa..ffdedbef1 100644--- a/crates/engine/src/cmd/face_detect.rs+++ b/crates/engine/src/cmd/face_detect.rs@@ -22,8 +22,8 @@ use crate::{Result, Session};const C: &str = "faces.detect";/// Regions made by this command say so in their description; only those are replaced on a new run.-const MARK: &str = "Detected by ";-const LABEL: &str = "YuNet 2023mar";+pub(crate) const MARK: &str = "Detected by ";+pub(crate) const LABEL: &str = "YuNet 2023mar";/// Long edge of the picture the detector looks at (it shrinks it to its own 640 anyway).const LOOK_EDGE: usize = 1280;/// YuNet is 232 KB; a file this much larger in its folder is not it.@@ -35,7 +35,8 @@ const EXISTING_IOU: f64 = 0.3;const NOT_INSTALLED: &str = "finding faces needs the YuNet face detector (232 KB, MIT): download it in Settings > Faces";/// The installed YuNet, loaded once and kept (a new one is loaded when the model folder changes).-fn detector(s: &Session) -> std::result::Result<Arc<Detector>, String> {+/// (also used by the background scan, which aligns faces by the detector's landmarks and finds faces in photos that have none)+pub(crate) fn detector(s: &Session) -> std::result::Result<Arc<Detector>, String> {static LOADED: Mutex<Option<(PathBuf, Arc<Detector>)>> = Mutex::new(None);let home = s.face_models_dir.as_deref().ok_or("this build has nowhere to keep face models (the desktop app does)")?.join(YUNET_ID);let path = home.join("model.onnx");@@ -80,10 +81,20 @@ fn overlaps_existing(f: &Face, regions: &[Region]) -> bool {})}-fn is_detected(r: &Region) -> bool {+pub(crate) fn is_detected(r: &Region) -> bool {r.description.as_deref().is_some_and(|d| d.starts_with(MARK))}+/// The photo as the faces see it: upright, uncropped, default settings, as 8-bit RGB with its size, its long edge at+/// most `edge`.+pub(crate) fn render_rgb(s: &mut Session, id: lightcraft_catalog::PhotoId, edge: usize) -> std::result::Result<(Vec<u8>, usize, usize), String> {+ let settings = DevelopSettings::default();+ let job = s.preview_job(id, edge, edge, false, &settings).ok_or("no such photo")?;+ let image = job.run().rendered?.image;+ let rgb: Vec<u8> = image.data.iter().flat_map(|px| [px[0], px[1], px[2]]).collect();+ Ok((rgb, image.width, image.height))+}+fn detect(s: &mut Session, p: &Value) -> Result<Value> {let apply = p.get("apply").and_then(Value::as_bool).unwrap_or(true);let defaults = Options::default();@@ -96,17 +107,14 @@ fn detect(s: &mut Session, p: &Value) -> Result<Value> {let started = Instant::now();let (mut results, mut ops) = (Vec::new(), Vec::new());for id in s.targets(p) {- let settings = DevelopSettings::default();- let Some(job) = s.preview_job(id, LOOK_EDGE, LOOK_EDGE, false, &settings) else { continue };- let rendered = match job.run().rendered {- Ok(r) => r.image,+ let (rgb, width, height) = match render_rgb(s, id, LOOK_EDGE) {+ Ok(r) => r,Err(e) => {results.push(json!({"id": id.0, "error": e}));continue;}};- let rgb: Vec<u8> = rendered.data.iter().flat_map(|px| [px[0], px[1], px[2]]).collect();- let faces = match detector.detect(&rgb, rendered.width, rendered.height, &opts) {+ let faces = match detector.detect(&rgb, width, height, &opts) {Ok(f) => f,Err(e) => {results.push(json!({"id": id.0, "error": e.to_string()}));diff --git a/crates/engine/src/cmd/face_models.rs b/crates/engine/src/cmd/face_models.rsindex 8ffdf0681..44b359e4a 100644--- a/crates/engine/src/cmd/face_models.rs+++ b/crates/engine/src/cmd/face_models.rs@@ -9,9 +9,10 @@use std::io::Read;use std::path::{Path, PathBuf};+use lightcraft_faces::catalog::{self, Catalog};use lightcraft_faces::hash::sha256_file;use lightcraft_faces::manifest::{self, MAX_MANIFEST_BYTES, MAX_MODEL_BYTES, ModelManifest, Role};-use lightcraft_faces::suggest::{Suggestion, suggest};+use lightcraft_faces::suggest::{Suggestion, suggest_with};use lightcraft_faces::{known, onnx};use serde::{Deserialize, Serialize};use serde_json::{Value, json};@@ -22,10 +23,10 @@ use crate::{EngineError, Result, Session};#[derive(Default, Serialize, Deserialize)]#[serde(default, rename_all = "camelCase")]-struct FaceSettings {- enabled: bool,+pub(super) struct FaceSettings {+ pub(super) enabled: bool,/// The installed recogniser in use.- embedder: Option<String>,+ pub(super) embedder: Option<String>,}fn fail(what: &str, e: impl std::fmt::Display) -> EngineError {@@ -43,7 +44,7 @@ fn read_capped(path: &Path, max: usize) -> Option<Vec<u8>> {(buf.len() <= max).then_some(buf)}-fn read_settings(dir: &Path) -> FaceSettings {+pub(super) fn read_settings(dir: &Path) -> FaceSettings {read_capped(&dir.join("settings.json"), 4096).and_then(|b| serde_json::from_slice(&b).ok()).unwrap_or_default()}@@ -61,14 +62,14 @@ fn write_settings(dir: &Path, st: &FaceSettings) -> Result<()> {write_atomic(&dir.join("settings.json"), &serde_json::to_vec_pretty(st).map_err(|e| fail("settings", e))?)}-struct Installed {- manifest: ModelManifest,- accepted: Value,+pub(super) struct Installed {+ pub(super) manifest: ModelManifest,+ pub(super) accepted: Value,}/// The models in the folder. A folder that is not a valid model (wrong name, no model file, a manifest that/// does not validate or does not match its folder) is ignored, never trusted.-fn installed_models(dir: &Path) -> Vec<Installed> {+pub(super) fn installed_models(dir: &Path) -> Vec<Installed> {let Ok(entries) = std::fs::read_dir(dir) else { return Vec::new() };let mut out = Vec::new();for e in entries.flatten() {@@ -88,7 +89,29 @@ fn installed_models(dir: &Path) -> Vec<Installed> {out}-fn row(m: &ModelManifest, installed: bool, selected: bool, accepted: &Value) -> Value {+/// Read the user's catalog (`catalog.json` in the models folder) anew: a few kilobytes, so it is always current.+fn refresh_catalog(s: &mut Session) {+ let Some(dir) = s.face_models_dir.clone() else { return };+ let file = dir.join("catalog.json");+ s.face_catalog = match read_capped(&file, catalog::MAX_CATALOG_BYTES) {+ Some(bytes) => catalog::parse(&bytes),+ None if file.is_file() => catalog::parse(&vec![b' '; catalog::MAX_CATALOG_BYTES + 1]),+ None => Catalog::default(),+ };+}++/// The manifests of the catalog's models (what a file's hash is matched against, besides the built-in ones).+fn catalog_manifests(s: &Session) -> Vec<ModelManifest> {+ s.face_catalog.entries.iter().map(|e| e.manifest.clone()).collect()+}++/// How to fetch a model: the built-in address, or the one in the user's catalog.+fn download_spec(s: &Session, id: &str) -> Option<known::Download> {+ known::download(id).or_else(|| s.face_catalog.entries.iter().find(|e| e.manifest.id == id).and_then(catalog::Entry::download))+}++fn row(s: &Session, m: &ModelManifest, installed: bool, selected: bool, accepted: &Value) -> Value {+ let from_catalog = s.face_catalog.entries.iter().any(|e| e.manifest.id == m.id);json!({"id": m.id,"name": m.name,@@ -98,9 +121,15 @@ fn row(m: &ModelManifest, installed: bool, selected: bool, accepted: &Value) ->"source": m.source,"sizeBytes": m.size_bytes,"sha256": m.sha256,- "known": known::all().iter().any(|k| k.id == m.id),+ // how fast it is as a multiple of a ResNet-100 model (computers differ, ratios between models hardly do), and+ // that in words; null when nobody has measured it+ "speed": m.speed,+ "speedText": m.speed.map(manifest::describe_speed).filter(|t| !t.is_empty()),+ "known": from_catalog || known::all().iter().any(|k| k.id == m.id),+ // from the user's own catalog file+ "fromCatalog": from_catalog,// a pinned address LightCraft can fetch it from (the user presses Download), and which site that is- "downloadHost": known::download(&m.id).map(|d| d.host()),+ "downloadHost": download_spec(s, &m.id).map(|d| known::host(&d.url).to_string()),"installed": installed,"selected": selected,"accepted": accepted,@@ -116,14 +145,14 @@ struct Inspection {/// Look at a model file. `known_sha` is its SHA-256 when something already checked it (a finished download), so a/// file of hundreds of megabytes is not read again just to be hashed.-fn inspect_file(path: &str, cmd: &str, known_sha: Option<&str>) -> Result<Inspection> {+fn inspect_file(path: &str, cmd: &str, known_sha: Option<&str>, extra: &[ModelManifest]) -> Result<Inspection> {let p = Path::new(path);let meta = std::fs::metadata(p).map_err(|e| bad(cmd, format!("cannot read `{path}`: {e}")))?;if !meta.is_file() {return Err(bad(cmd, format!("`{path}` is not a file")));}if meta.len() == 0 || meta.len() > MAX_MODEL_BYTES {- return Err(bad(cmd, "a model file must be between 1 byte and 4 GiB"));+ return Err(bad(cmd, "a model file must be between 1 byte and 512 MiB"));}let sha256 = match known_sha {Some(h) => h.to_string(),@@ -131,13 +160,14 @@ fn inspect_file(path: &str, cmd: &str, known_sha: Option<&str>) -> Result<Inspec};let file_name = p.file_name().map(|n| n.to_string_lossy().into_owned()).unwrap_or_default();let suggestion = match onnx::probe_path(p) {- Ok(info) => suggest(&info, &sha256, meta.len(), &file_name),+ Ok(info) => suggest_with(&info, &sha256, meta.len(), &file_name, extra),Err(e) => Suggestion::Unsupported(e.to_string()),};Ok(Inspection { sha256, size: meta.len(), file_name, suggestion })}fn list(s: &mut Session, _: &Value) -> Result<Value> {+ refresh_catalog(s);let dir = s.face_models_dir.clone();let (settings, installed) = match &dir {Some(d) => (read_settings(d), installed_models(d)),@@ -147,27 +177,41 @@ fn list(s: &mut Session, _: &Value) -> Result<Value> {let mut models: Vec<Value> = Vec::new();for k in known::all() {let found = on_disk(&k.id);- models.push(row(&k, found.is_some(), settings.embedder.as_deref() == Some(k.id.as_str()), found.map_or(&Value::Null, |f| &f.accepted)));+ models.push(row(s, &k, found.is_some(), settings.embedder.as_deref() == Some(k.id.as_str()), found.map_or(&Value::Null, |f| &f.accepted)));+ }+ // the user's own catalog+ for e in &s.face_catalog.entries {+ let found = on_disk(&e.manifest.id);+ let chosen = settings.embedder.as_deref() == Some(e.manifest.id.as_str());+ models.push(row(s, &e.manifest, found.is_some(), chosen, found.map_or(&Value::Null, |f| &f.accepted)));}for i in &installed {- if !known::all().iter().any(|k| k.id == i.manifest.id) {- models.push(row(&i.manifest, true, settings.embedder.as_deref() == Some(i.manifest.id.as_str()), &i.accepted));+ let listed = known::all().iter().any(|k| k.id == i.manifest.id) || s.face_catalog.entries.iter().any(|e| e.manifest.id == i.manifest.id);+ if !listed {+ models.push(row(s, &i.manifest, true, settings.embedder.as_deref() == Some(i.manifest.id.as_str()), &i.accepted));}}Ok(json!({"dir": dir.as_ref().map(|d| d.display().to_string()),"enabled": settings.enabled,"embedder": settings.embedder,- // running a model is not built in yet: models can be installed and chosen, not used- "runtime": false,+ // Native builds always have the pure-Rust runner; model files/background scanning remain native-only.+ "runtime": cfg!(not(target_arch = "wasm32")),"models": models,+ // the user's own list of models to download: where it goes, how many it holds, and what was wrong with the rest+ "catalog": {+ "path": dir.as_ref().map(|d| d.join("catalog.json").display().to_string()),+ "models": s.face_catalog.entries.len(),+ "errors": s.face_catalog.errors,+ },}))}fn inspect(s: &mut Session, p: &Value) -> Result<Value> {const C: &str = "faces.models.inspect";let path = str_param(p, "path").ok_or_else(|| bad(C, "missing `path`"))?;- let ins = inspect_file(path, C, None)?;+ refresh_catalog(s);+ let ins = inspect_file(path, C, None, &catalog_manifests(s))?;let installed = s.face_models_dir.as_deref().map(installed_models).unwrap_or_default();let (kind, model, assumptions, reason) = match &ins.suggestion {Suggestion::Known(m) => ("known", Some(m), Vec::new(), None),@@ -181,7 +225,7 @@ fn inspect(s: &mut Session, p: &Value) -> Result<Value> {"sizeBytes": ins.size,"sha256": ins.sha256,"kind": kind,- "model": model.map(|m| row(m, already, false, &Value::Null)),+ "model": model.map(|m| row(s, m, already, false, &Value::Null)),"assumptions": assumptions,"reason": reason,"alreadyInstalled": already,@@ -192,13 +236,17 @@ fn install(s: &mut Session, p: &Value) -> Result<Value> {const C: &str = "faces.models.install";let path = str_param(p, "path").ok_or_else(|| bad(C, "missing `path`"))?;let acknowledged = p.get("acknowledged").and_then(Value::as_bool).unwrap_or(false);- let ins = inspect_file(path, C, None)?;- install_file(s, C, Path::new(path), ins, acknowledged, false)+ let activate = p.get("activate").and_then(Value::as_bool).unwrap_or(true);+ refresh_catalog(s);+ let ins = inspect_file(path, C, None, &catalog_manifests(s))?;+ install_file(s, C, Path::new(path), ins, acknowledged, activate, false)}-/// Install the inspected model file `src`. `verified` is for a download that was already checked against its recorded-/// hash: it is moved into place instead of being copied and read again.-fn install_file(s: &mut Session, c: &str, src: &Path, ins: Inspection, acknowledged: bool, verified: bool) -> Result<Value> {+/// Install the inspected model file `src`. A recognition model that passes its self-test becomes the one in use and+/// recognition is switched on (`activate`, the default): installing a model is how someone says they want it.+/// `verified` is for a download that was already checked against its recorded hash: it is moved into place instead+/// of being copied and read again.+fn install_file(s: &mut Session, c: &str, src: &Path, ins: Inspection, acknowledged: bool, activate: bool, verified: bool) -> Result<Value> {let dir = models_dir(s, c)?;let m = match ins.suggestion {Suggestion::Known(m) | Suggestion::Draft { manifest: m, .. } => m,@@ -224,19 +272,41 @@ fn install_file(s: &mut Session, c: &str, src: &Path, ins: Inspection, acknowledErr(e) => Err(fail("could not check the copy", e)),})};- if let Err(e) = placed.and_then(|_| std::fs::rename(&part, &final_path).map_err(|e| fail("could not finish the copy", e))) {+ if let Err(e) = placed {let _ = std::fs::remove_file(&part);return Err(e);}+ // Validate the candidate before replacing a working installation.+ let test = match self_test(&part, &m) {+ Ok(t) => t,+ Err(e) => {+ let _ = std::fs::remove_file(&part);+ let _ = std::fs::remove_dir(&home);+ return Err(bad(c, e));+ }+ };+ if let Err(e) = std::fs::rename(&part, &final_path) {+ let _ = std::fs::remove_file(&part);+ return Err(fail("could not finish the copy", e));+ }write_atomic(&home.join("face-model.json"), &serde_json::to_vec_pretty(&m).map_err(|e| fail("manifest", e))?)?;- let accepted = json!({"acceptedAt": (s.clock)(), "licence": m.licence.name, "commercial": m.licence.commercial, "fileName": ins.file_name});+ let accepted = json!({"acceptedAt": (s.clock)(), "licence": m.licence.name, "commercial": m.licence.commercial, "fileName": ins.file_name, "selfTest": test});write_atomic(&home.join("installed.json"), &serde_json::to_vec_pretty(&accepted).map_err(|e| fail("record", e))?)?;// a file that was downloaded and waited in the staging folder has served: the installed copy is the model nowif src.parent() == Some(dir.join(face_download::STAGING).as_path()) {let _ = std::fs::remove_file(src);s.face_downloads.staged_file_used(src);}- Ok(json!({"installed": row(&m, true, false, &accepted)}))+ // the newest model is the one in use (the earlier one stays installed, and `faces.models.select` switches back)+ if activate && m.role == Role::Embedder {+ let mut st = read_settings(&dir);+ st.embedder = Some(m.id.clone());+ st.enabled = true;+ write_settings(&dir, &st)?;+ }+ let now = read_settings(&dir);+ let selected = now.embedder.as_deref() == Some(m.id.as_str());+ Ok(json!({"installed": row(s, &m, true, selected, &accepted), "active": {"embedder": now.embedder, "enabled": now.enabled}}))}/// What an error says, without the command it came from (for a line shown to the user).@@ -247,12 +317,13 @@ fn plain(e: &EngineError) -> String {}}-/// Install every download that has arrived: each was accepted when it was started, so it is installed without another-/// question. Called by `faces.models.downloads`, which the UI asks for every frame while it watches a download.-fn finish_downloads(s: &mut Session) {+/// Install every download that has arrived: each was accepted when it was started, so it is installed, chosen and+/// switched on without another question. Called by `faces.models.downloads` and by every frame's `faces.pump`.+pub(super) fn finish_downloads(s: &mut Session) {const C: &str = "faces.models.download";+ let extra = catalog_manifests(s);for (id, path, sha) in s.face_downloads.finished() {- let outcome = inspect_file(&path.to_string_lossy(), C, Some(&sha)).and_then(|ins| install_file(s, C, &path, ins, true, true));+ let outcome = inspect_file(&path.to_string_lossy(), C, Some(&sha), &extra).and_then(|ins| install_file(s, C, &path, ins, true, true, true));match outcome {Ok(_) => s.face_downloads.set_outcome(&id, State::Installed),Err(e) => {@@ -270,7 +341,8 @@ fn download(s: &mut Session, p: &Value) -> Result<Value> {const C: &str = "faces.models.download";let id = str_param(p, "id").ok_or_else(|| bad(C, "missing `id`"))?;let dir = models_dir(s, C)?;- let spec = known::download(id).ok_or_else(|| bad(C, "LightCraft has no download for that model: get the file from its page, then add it"))?;+ refresh_catalog(s);+ let spec = download_spec(s, id).ok_or_else(|| bad(C, "LightCraft has no download for that model: get the file from its page, then add it"))?;if cfg!(target_arch = "wasm32") {return Err(bad(C, "the web build downloads no models"));}@@ -283,12 +355,13 @@ fn download(s: &mut Session, p: &Value) -> Result<Value> {if installed_models(&dir).iter().any(|i| i.manifest.id == spec.id) {return Err(bad(C, "that model is already installed"));}+ let host = spec.host().to_string();s.face_downloads.start(spec, &dir).map_err(|e| bad(C, e))?;- Ok(json!({"started": id, "from": spec.host()}))+ Ok(json!({"started": id, "from": host}))}-fn download_row(id: &str, state: &State) -> Value {- let from = known::download(id).map(|d| d.host());+fn download_row(s: &Session, id: &str, state: &State) -> Value {+ let from = download_spec(s, id).map(|d| d.host().to_string());match state {State::Running { bytes, total } => json!({"id": id, "state": "running", "bytes": bytes, "total": total, "from": from}),State::Done { path, .. } => json!({"id": id, "state": "done", "path": path.display().to_string(), "from": from}),@@ -301,7 +374,7 @@ fn download_row(id: &str, state: &State) -> Value {/// `faces.models.downloads`: where each download stands (a finished one is installed first).fn downloads(s: &mut Session, _: &Value) -> Result<Value> {finish_downloads(s);- let all: Vec<Value> = s.face_downloads.snapshot().iter().map(|(id, st)| download_row(id, st)).collect();+ let all: Vec<Value> = s.face_downloads.snapshot().iter().map(|(id, st)| download_row(s, id, st)).collect();Ok(json!({"running": s.face_downloads.running(), "downloads": all}))}@@ -312,6 +385,62 @@ fn download_cancel(s: &mut Session, p: &Value) -> Result<Value> {Ok(json!({"discarded": s.face_downloads.discard(id)}))}+/// Load the model and run its self-test: `Ok(null)` in a build without the recognition runtime (nothing can+/// be checked), `Ok(result)` when it passes, `Err(reason)` when it does not load or fails a check.+#[cfg(not(target_arch = "wasm32"))]+fn self_test(path: &Path, m: &ModelManifest) -> std::result::Result<Value, String> {+ if m.role == Role::Detector {+ let bytes = read_capped(path, 8 << 20).ok_or("the detector cannot be read within its 8 MiB limit")?;+ let detector = lightcraft_faces::yunet::Detector::new(&bytes).map_err(|e| e.to_string())?;+ if Some(detector.input_side()) != usize::try_from(m.input.width).ok()+ || m.input.width != m.input.height+ || !matches!(&m.output, manifest::OutputSpec::Detector { decoder } if decoder == "yunet-v2")+ {+ return Err("unsupported detector input or output decoder".into());+ }+ detector.detect(&[0u8; 12], 2, 2, &lightcraft_faces::yunet::Options::default()).map_err(|e| e.to_string())?;+ return Ok(json!({"ok": true}));+ }+ let embedder = lightcraft_faces::runtime::Embedder::load(path, m).map_err(|e| e.to_string())?;+ let t = embedder.self_test();+ if !t.ok {+ let failed: Vec<&str> = t.checks.iter().filter(|(_, ok)| !*ok).map(|(c, _)| c.as_str()).collect();+ return Err(format!("the model failed its self-test: it does not {}", failed.join(", and does not ")));+ }+ serde_json::to_value(&t).map_err(|e| e.to_string())+}++#[cfg(target_arch = "wasm32")]+fn self_test(_: &Path, _: &ModelManifest) -> std::result::Result<Value, String> {+ Ok(Value::Null)+}++/// `faces.models.test {id}`: run an installed model's self-test again.+fn test(s: &mut Session, p: &Value) -> Result<Value> {+ const C: &str = "faces.models.test";+ let id = str_param(p, "id").ok_or_else(|| bad(C, "missing `id`"))?;+ let dir = models_dir(s, C)?;+ if !cfg!(not(target_arch = "wasm32")) {+ return Err(bad(C, "this build cannot run face recognition models"));+ }+ let found = installed_models(&dir).into_iter().find(|i| i.manifest.id == id).ok_or_else(|| bad(C, "that model is not installed"))?;+ if found.manifest.role != Role::Embedder {+ return Err(bad(C, "only recognition models have a self-test"));+ }+ let result = self_test(&dir.join(id).join("model.onnx"), &found.manifest);+ let (ok, detail) = match &result {+ Ok(v) => (true, v.clone()),+ Err(e) => (false, json!(e)),+ };+ // keep the latest outcome with the model's record+ let mut record = found.accepted;+ if let Some(o) = record.as_object_mut() {+ o.insert("selfTest".into(), if ok { detail.clone() } else { json!({"ok": false, "error": detail}) });+ write_atomic(&dir.join(id).join("installed.json"), &serde_json::to_vec_pretty(&record).map_err(|e| fail("record", e))?)?;+ }+ Ok(json!({"id": id, "ok": ok, "result": detail}))+}+fn remove(s: &mut Session, p: &Value) -> Result<Value> {const C: &str = "faces.models.remove";let id = str_param(p, "id").ok_or_else(|| bad(C, "missing `id`"))?;@@ -327,10 +456,11 @@ fn remove(s: &mut Session, p: &Value) -> Result<Value> {std::fs::remove_dir_all(&home).map_err(|e| fail("could not remove the model", e))?;let mut st = read_settings(&dir);if st.embedder.as_deref() == Some(id) {- st.embedder = None;+ // another installed recogniser takes over, so recognition keeps working; with none left nothing is chosen+ st.embedder = installed_models(&dir).into_iter().find(|i| i.manifest.id != id && i.manifest.role == Role::Embedder).map(|i| i.manifest.id);write_settings(&dir, &st)?;}- Ok(json!({"removed": id}))+ Ok(json!({"removed": id, "embedder": st.embedder}))}fn select(s: &mut Session, p: &Value) -> Result<Value> {@@ -357,6 +487,11 @@ fn enable(s: &mut Session, p: &Value) -> Result<Value> {let mut st = read_settings(&dir);st.enabled = p.get("enabled").and_then(Value::as_bool).unwrap_or(!st.enabled);write_settings(&dir, &st)?;+ // the background scan notices at once, not within the second it trusts its last look+ #[cfg(not(target_arch = "wasm32"))]+ {+ s.faces.enabled_seen = None;+ }Ok(json!({"enabled": st.enabled}))}@@ -364,10 +499,11 @@ pub fn specs() -> Vec<CommandSpec> {vec![cmd!(query "faces.models.list", "Face Models", [], None, "{} → {dir, enabled, embedder, runtime, models: [{id, name, role, licence{name, commercial, url, notice}, provenance, source, sizeBytes, known, installed, selected}]}", always, list),cmd!(query "faces.models.inspect", "Inspect Face Model File", [], None, "{path} → what a .onnx file is: {kind: known | draft | unsupported, model, assumptions, reason, alreadyInstalled}; installs nothing", always, inspect),- cmd!(query "faces.models.install", "Install Face Model", [], None, "{path, acknowledged: true} — copy a .onnx face model (the detector or a recogniser) into the models folder. `acknowledged` must be true: the user has been shown its licence (see inspect) and accepted it", always, install),- cmd!(query "faces.models.download", "Download Face Model", [], None, "{id, acknowledged: true} → {started, from} — fetch a model LightCraft has a pinned address for (see `downloadHost` in the list), in the background over https. `acknowledged` must be true: the user has been shown the model's terms and accepted them. It is checked against its size and SHA-256, then installed by itself; `faces.models.downloads` shows how far it is", always, download),+ cmd!(query "faces.models.install", "Install Face Model", [], None, "{path, acknowledged: true, activate?: true} → {installed, active: {embedder, enabled}} — copy a .onnx face recognition model into the models folder. `acknowledged` must be true: the user has been shown its licence (see inspect) and accepted it. Unless `activate` is false the model becomes the one in use and recognition is switched on; an earlier model stays installed", always, install),+ cmd!(query "faces.models.download", "Download Face Model", [], None, "{id, acknowledged: true} → {started, from} — fetch a recognition model LightCraft has a pinned address for (see `downloadHost` in the list), in the background over pure-Rust https (lightcraft-fetch). `acknowledged` must be true: the user has been shown the model's terms and accepted them. It is checked against its size and SHA-256, then installed, chosen and switched on by itself; `faces.models.downloads` shows how far it is", always, download),cmd!(query "faces.models.downloads", "Face Model Downloads", [], None, "{} → {running, downloads: [{id, state: running | done | installed | failed | cancelled, bytes, total, error, from}]} — also installs any download that has arrived; `installed` stays listed until `faces.models.downloadCancel` clears it", always, downloads),- cmd!(query "faces.models.downloadCancel", "Cancel Face Model Download", [], None, "{id} → {discarded} — stop a download, or clear a finished one", always, download_cancel),+ cmd!(query "faces.models.downloadCancel", "Cancel Face Model Download", [], None, "{id} → {discarded} — stop a download, or delete a finished one that was not installed", always, download_cancel),+ cmd!(query "faces.models.test", "Test Face Model", [], None, "{id} → {ok, result: {loadMs, embedMs, dimension, checks}} — load an installed recognition model and check it gives sensible faces; needs the recognition runtime", always, test),cmd!(query "faces.models.remove", "Remove Face Model", [], None, "{id} — delete an installed model", always, remove),cmd!(query "faces.models.select", "Choose Face Recognition Model", [], None, "{id: installed recogniser | null}", always, select),cmd!(query "faces.enable", "Face Recognition On/Off", [], None, "{enabled?: bool} (toggles when omitted)", always, enable),diff --git a/crates/engine/src/cmd/face_recognize.rs b/crates/engine/src/cmd/face_recognize.rsnew file mode 100644index 000000000..9efdb146d--- /dev/null+++ b/crates/engine/src/cmd/face_recognize.rs@@ -0,0 +1,1030 @@+//! Recognition commands: `faces.index` (embed the faces in the library), `faces.suggest` (who might an unnamed+//! face be?), `faces.person` (one person's faces, and the unnamed faces that look like them), `faces.unnamed` (every+//! unnamed face, look-alikes together) and `faces.setName` / `faces.nameFaces` (name one face, or many at once, which is+//! what teaches the suggestions).+//!+//! Everything here is a suggestion for the user to confirm; nothing is ever named automatically. All of it is+//! catalog-only (no sidecar is written) and `faces.setName` is undoable.++use lightcraft_catalog::Op;+use serde_json::{Value, json};++use super::{CommandSpec, bad, cmd};+use crate::{Result, Session};++/// The `name` parameter of the naming commands: a name, or none (`null`, empty) to clear it.+fn name_param(p: &Value, c: &str) -> Result<Option<String>> {+ let name = match p.get("name") {+ None | Some(Value::Null) => None,+ Some(v) => Some(v.as_str().ok_or_else(|| bad(c, "`name` must be text or null"))?.trim().to_string()).filter(|n| !n.is_empty()),+ };+ if let Some(n) = &name+ && (n.chars().count() > 200 || n.chars().any(char::is_control))+ {+ return Err(bad(c, "a name is at most 200 characters, without control characters"));+ }+ Ok(name)+}++/// Give a face region its name. Naming makes the face yours: a later "Detect Faces" run no longer replaces it.+fn apply_name(region: &mut lightcraft_meta::Region, name: &Option<String>) {+ region.name = name.clone();+ if name.is_some() && region.description.as_deref().is_some_and(|d| d.starts_with(super::face_detect::MARK)) {+ region.description = None;+ }+}++/// `faces.nameFaces {faces: [{photo, index}], name}`: name (or clear) many face regions at once, as one undo step: what+/// naming a group of look-alikes does.+fn name_faces(s: &mut Session, p: &Value) -> Result<Value> {+ const C: &str = "faces.nameFaces";+ let name = name_param(p, C)?;+ let list = p.get("faces").and_then(Value::as_array).ok_or_else(|| bad(C, "missing `faces`"))?;+ if list.len() > 5000 {+ return Err(bad(C, "at most 5000 faces at a time"));+ }+ let mut by_photo: std::collections::BTreeMap<u64, Vec<usize>> = std::collections::BTreeMap::new();+ for f in list {+ let (Some(photo), Some(index)) =+ (f.get("photo").and_then(Value::as_u64), f.get("index").and_then(Value::as_u64).and_then(|i| usize::try_from(i).ok()))+ else {+ return Err(bad(C, "each face needs a `photo` and an `index`"));+ };+ by_photo.entry(photo).or_default().push(index);+ }+ let (mut ops, mut named) = (Vec::new(), 0usize);+ for (photo, indexes) in by_photo {+ let id = lightcraft_catalog::PhotoId(photo);+ let Some(photo) = s.catalog.photo(id) else { continue };+ let mut meta = photo.meta.clone();+ let before = named;+ for i in indexes {+ if let Some(region) = meta.regions.get_mut(i).filter(|r| r.kind == lightcraft_meta::RegionKind::Face) {+ apply_name(region, &name);+ named += 1;+ }+ }+ if named > before {+ ops.push(Op::SetMeta { id, meta: Box::new(meta) });+ }+ }+ if ops.is_empty() {+ return Err(bad(C, "none of those faces exist"));+ }+ let photos = ops.len();+ s.commit(if name.is_some() { "Name Faces" } else { "Clear Face Names" }, Op::Batch { ops })?;+ s.skip_auto_write = true;+ Ok(json!({"named": named, "photos": photos, "name": name}))+}++/// `faces.setName {id?, index, name}`: name (or, with `null` or an empty name, un-name) one face region. Naming+/// makes the face yours: a later "Detect Faces" run no longer replaces it.+fn set_name(s: &mut Session, p: &Value) -> Result<Value> {+ const C: &str = "faces.setName";+ let id = p.get("id").and_then(Value::as_u64).map(lightcraft_catalog::PhotoId).or_else(|| s.active()).ok_or_else(|| bad(C, "no photo"))?;+ let index = p.get("index").and_then(Value::as_u64).and_then(|i| usize::try_from(i).ok()).ok_or_else(|| bad(C, "missing or invalid `index`"))?;+ let name = name_param(p, C)?;+ let mut meta = s.catalog.photo(id).ok_or_else(|| bad(C, "no such photo"))?.meta.clone();+ let region = meta.regions.get_mut(index).ok_or_else(|| bad(C, "no such region"))?;+ apply_name(region, &name);+ s.commit(if name.is_some() { "Name Face" } else { "Clear Face Name" }, Op::SetMeta { id, meta: Box::new(meta) })?;+ s.skip_auto_write = true;+ Ok(json!({"id": id.0, "index": index, "name": name}))+}++/// Most faces `faces.unnamed` lists.+const MAX_UNNAMED: usize = 3000;++/// How the unnamed faces were laid out.+struct Arranged {+ /// Faces that look alike are next to each other (only the ones already embedded can be).+ ordered: bool,+ /// Recognition is on and a model is loaded.+ ready: bool,+ /// A name for some faces, when the faces already named say so with confidence: `(photo, index)` to `(name, score)`.+ suggestions: std::collections::HashMap<(u64, usize), (String, f32)>,+}++/// `faces.unnamed {limit?: 600}`: the faces nobody has named, to be named a group at a time. With recognition running,+/// faces that look alike are next to each other, and a face the named faces recognise carries the name they suggest.+fn unnamed(s: &mut Session, p: &Value) -> Result<Value> {+ let limit = p.get("limit").and_then(Value::as_u64).map_or(600, |n| usize::try_from(n).unwrap_or(600)).clamp(1, MAX_UNNAMED);+ let mut all: Vec<(u64, usize, lightcraft_geom::Rect)> = Vec::new();+ for ph in s.catalog.photos().filter(|ph| ph.in_library()) {+ for (index, r) in ph.meta.regions.iter().enumerate() {+ if r.kind == lightcraft_meta::RegionKind::Face && r.name.is_none() {+ all.push((ph.id.0, index, r.rect));+ }+ }+ }+ all.sort_by_key(|(photo, index, _)| (*photo, *index));+ let total = all.len();+ let arranged = imp::arrange_unnamed(s, &mut all, limit);+ all.truncate(limit);+ let faces: Vec<Value> = all+ .iter()+ .map(|(photo, index, rect)| {+ let view = s.face_view(lightcraft_catalog::PhotoId(*photo), *rect);+ let suggestion = arranged.suggestions.get(&(*photo, *index)).map(|(name, score)| json!({"name": name, "score": score}));+ json!({"photo": photo, "index": index, "rect": rect_json(rect), "view": rect_json(&view), "suggestion": suggestion})+ })+ .collect();+ Ok(json!({"total": total, "faces": faces, "ordered": arranged.ordered, "ready": arranged.ready}))+}++/// What looking for faces that resemble a person found.+struct Similar {+ /// `{photo, index, rect, score}` of unnamed faces, the most alike first.+ items: Vec<serde_json::Value>,+ /// Recognition is on and a model is loaded, so an empty list means "none yet" rather than "not looking".+ ready: bool,+ /// Photos the background scan has still to do.+ pending: usize,+}++fn rect_json(r: &lightcraft_geom::Rect) -> Value {+ json!({"x0": r.x0, "y0": r.y0, "x1": r.x1, "y1": r.y1})+}++/// Most confirmed faces `faces.person` lists.+const MAX_CONFIRMED: usize = 5000;++/// `faces.person {name, more?}`: the faces named `name` (case-insensitively) across the library, and, when recognition is+/// running, the unnamed faces that look most like them, to be confirmed one by one.+fn person(s: &mut Session, p: &Value) -> Result<Value> {+ const C: &str = "faces.person";+ let name = p.get("name").and_then(Value::as_str).map(str::trim).filter(|n| !n.is_empty()).ok_or_else(|| bad(C, "missing `name`"))?;+ let key = name.to_lowercase();+ let limit = p.get("more").and_then(Value::as_u64).map_or(120, |n| usize::try_from(n).unwrap_or(120)).min(1000);+ let (mut shown, mut total, mut confirmed) = (name.to_string(), 0usize, Vec::new());+ for ph in s.catalog.photos().filter(|ph| ph.in_library()) {+ for (index, r) in ph.meta.regions.iter().enumerate() {+ let Some(n) = r.name.as_deref().filter(|n| r.kind == lightcraft_meta::RegionKind::Face && n.trim().to_lowercase() == key) else {+ continue;+ };+ if total == 0 {+ shown = n.trim().to_string();+ }+ total += 1;+ if confirmed.len() < MAX_CONFIRMED {+ confirmed.push(json!({"photo": ph.id.0, "index": index, "rect": rect_json(&r.rect), "view": rect_json(&s.face_view(ph.id, r.rect))}));+ }+ }+ }+ let similar = if total > 0 { imp::similar(s, &key, limit) } else { Similar { items: Vec::new(), ready: false, pending: 0 } };+ Ok(+ json!({"name": shown, "total": total, "confirmed": confirmed, "more": similar.items, "ready": similar.ready, "pendingPhotos": similar.pending}),+ )+}++#[cfg(not(target_arch = "wasm32"))]+mod imp {+ use std::sync::Arc;+ use std::time::{Duration, Instant};++ use lightcraft_catalog::{Op, Photo, PhotoId};+ use lightcraft_develop::DevelopSettings;+ use lightcraft_faces::matching;+ use lightcraft_faces::runtime::Embedder;+ use lightcraft_geom::Rect;+ use lightcraft_meta::{Region, RegionKind};+ use serde_json::{Value, json};++ use super::super::bad;+ use super::super::face_detect::{LABEL, MARK};+ use super::super::face_models::{finish_downloads, installed_models, read_settings};+ use super::Similar;+ use crate::faces_worker::{Done, EDGE, PREFETCH, Pace, Prepared, Worker, max_workers, process, target_workers};+ use crate::{Result, Session};++ /// Defaults for suggestions, until a model's own threshold is known.+ const DEFAULT_THRESHOLD: f32 = 0.45;+ const DEFAULT_MARGIN: f32 = 0.06;+ /// How long a look at the models folder is trusted by the per-frame `faces.pump`.+ const RECHECK: Duration = Duration::from_secs(1);+ /// How often the background pump writes new embeddings to the cache file (appending is cheap; this keeps it to a few+ /// writes a minute, and a session that ends writes the rest).+ const SAVE_EVERY: Duration = Duration::from_secs(3);++ /// The chosen recognition model, loaded, with the index reset to it when it changed. `fresh` looks at the models+ /// folder again; otherwise a look made in the last second is trusted (the UI asks every frame).+ fn current_with(s: &mut Session, cmd: &str, fresh: bool) -> Result<Arc<Embedder>> {+ if !fresh+ && let Some((_, e)) = &s.faces.embedder+ && s.faces.checked.is_some_and(|t| t.elapsed() < RECHECK)+ {+ return Ok(e.clone());+ }+ let dir = s.face_models_dir.clone().ok_or_else(|| bad(cmd, "this build has nowhere to keep face models"))?;+ let id = read_settings(&dir).embedder.ok_or_else(|| bad(cmd, "no recognition model is chosen: add one in Settings > Faces and press Use"))?;+ let installed = installed_models(&dir)+ .into_iter()+ .find(|i| i.manifest.id == id)+ .ok_or_else(|| bad(cmd, "the chosen recognition model is not installed"))?;+ let tag = format!("{id}@{}", installed.manifest.sha256.as_deref().unwrap_or("unknown"));+ s.faces.checked = Some(Instant::now());+ // which photos the detector has searched is kept beside the library (or in memory when it is not on disk)+ let scan_path = s.library.as_ref().filter(|l| l.on_disk).map(|l| l.dir.join("face-scanned.bin"));+ if s.faces.scanned.path() != scan_path.as_deref() {+ s.faces.scanned.reset(scan_path);+ }+ if let Some((have, e)) = &s.faces.embedder+ && *have == tag+ {+ return Ok(e.clone());+ }+ let embedder = Arc::new(Embedder::load(&dir.join(&id).join("model.onnx"), &installed.manifest).map_err(|e| bad(cmd, e.to_string()))?);+ let dim = match installed.manifest.output {+ lightcraft_faces::OutputSpec::Embedding { dim } => dim as usize,+ _ => return Err(bad(cmd, "the chosen model is not a recognition model")),+ };+ let cache = s.library.as_ref().filter(|l| l.on_disk).map(|l| l.dir.join(format!("face-embeddings-{id}.bin")));+ s.faces.index.reset(&tag, dim, cache);+ s.faces.queue.clear();+ s.faces.queue_stamp = None;+ s.faces.in_flight.clear();+ s.faces.embedder = Some((tag, embedder.clone()));+ Ok(embedder)+ }++ fn current(s: &mut Session, cmd: &str) -> Result<Arc<Embedder>> {+ current_with(s, cmd, true)+ }++ /// Whether the background scan should look for faces in this photo: it is in the library, has no face regions at all+ /// (nothing from XMP, nothing drawn or named, no earlier detection) and has not been searched yet.+ fn needs_search(s: &Session, photo: &Photo) -> bool {+ photo.in_library()+ && !photo.meta.regions.iter().any(|r| r.kind == RegionKind::Face)+ && !s.faces.scanned.contains(photo.id.0)+ && !s.faces.scan_failed.contains(&photo.id.0)+ }++ /// The photo's faces that still need embedding (and, with `search`, the faces to find in it first), with the render job+ /// for its picture. `None` when there is nothing to do.+ fn prepare(s: &mut Session, id: PhotoId, search: bool) -> Option<Prepared> {+ let tag = s.faces.index.tag.clone();+ let photo = s.catalog.photo(id)?.clone();+ let todo: Vec<Rect> =+ photo.meta.regions.iter().filter(|r| r.kind == RegionKind::Face && s.faces.index.needs(id.0, &r.rect)).map(|r| r.rect).collect();+ // finding faces needs the detector the user may not have downloaded: such a photo is left unsearched, not marked as searched+ let detector = crate::cmd::face_detect::detector(s).ok();+ let detect = search && detector.is_some() && needs_search(s, &photo);+ if todo.is_empty() && !detect {+ return None;+ }+ let job = s.preview_job(id, EDGE, EDGE, false, &DevelopSettings::default())?;+ let preview = s.scan_preview_of(&photo);+ Some(Prepared { tag, epoch: s.faces.epoch, id, job, preview, todo, detect, detector })+ }++ /// The faces the detector found in a photo that had none become its face regions: unnamed, marked as detected (so a+ /// manual Detect Faces replaces them), and not an undo step, since the user did not do this.+ fn add_found(s: &mut Session, id: PhotoId, found: &[Rect]) {+ if found.is_empty() {+ return;+ }+ let Some(photo) = s.catalog.photo(id) else { return };+ // the user (or an import) got there first: leave their faces alone+ if photo.meta.regions.iter().any(|r| r.kind == RegionKind::Face) {+ return;+ }+ let mut meta = photo.meta.clone();+ for rect in found {+ meta.regions.push(Region { rect: *rect, kind: RegionKind::Face, name: None, description: Some(format!("{MARK}{LABEL}")) });+ }+ if s.apply_system(Op::SetMeta { id, meta: Box::new(meta) }).is_ok() {+ // regions are not written to XMP, so no sidecar may be rewritten for this+ s.skip_auto_write = true;+ }+ }++ /// Take a finished photo into the index (ignored when it was made for another model or another library). Returns+ /// (embedded, failed).+ fn ingest(s: &mut Session, d: Done) -> (usize, usize) {+ s.faces.in_flight.remove(&d.id);+ if d.epoch != s.faces.epoch || d.tag != s.faces.index.tag {+ return (0, 0);+ }+ if d.detect_wanted {+ match &d.found {+ Some(found) => {+ s.faces.scanned.insert(d.id.0);+ add_found(s, d.id, found);+ }+ None => {+ s.faces.scan_failed.insert(d.id.0);+ }+ }+ }+ let (mut embedded, mut failed) = (0, 0);+ for (rect, v, view) in d.results {+ match v {+ Some(v) if v.len() == s.faces.index.dim => {+ s.faces.index.insert_with_view(d.id.0, &rect, v, &view);+ embedded += 1;+ }+ _ => {+ s.faces.index.skip(d.id.0, &rect);+ failed += 1;+ }+ }+ }+ (embedded, failed)+ }++ /// Take in whatever the worker has finished.+ fn collect(s: &mut Session) -> (usize, usize) {+ let finished = s.faces.worker.as_ref().map(Worker::finished).unwrap_or_default();+ finished.into_iter().fold((0, 0), |(e, f), d| {+ let (de, df) = ingest(s, d);+ (e + de, f + df)+ })+ }++ /// Embed every face region of `id` the index does not have yet, here and now. Returns (embedded, failed).+ fn embed_photo(s: &mut Session, id: PhotoId, e: &Embedder) -> Result<(usize, usize)> {+ let Some(job) = prepare(s, id, false) else { return Ok((0, 0)) };+ Ok(ingest(s, process(job, e)))+ }++ /// Photos with a face that needs embedding (and, with `search`, photos to look for faces in): those the caller asked+ /// for first, then those with named faces (they are the gallery everything else is compared with), then the rest, and+ /// last the photos that only need searching.+ fn pending(s: &Session, first: &[PhotoId], search: bool) -> Vec<PhotoId> {+ let mut out: Vec<(u8, PhotoId)> = Vec::new();+ for p in s.catalog.photos().filter(|p| p.in_library()) {+ let faces: Vec<_> = p.meta.regions.iter().filter(|r| r.kind == RegionKind::Face).collect();+ let embed = faces.iter().any(|r| s.faces.index.needs(p.id.0, &r.rect));+ if embed || (search && needs_search(s, p)) {+ let rank = if first.contains(&p.id) {+ 0+ } else if faces.iter().any(|r| r.name.is_some()) {+ 1+ } else if embed {+ 2+ } else {+ 3+ };+ out.push((rank, p.id));+ }+ }+ out.sort_by_key(|(rank, id)| (*rank, id.0));+ out.into_iter().map(|(_, id)| id).collect()+ }++ pub fn index(s: &mut Session, p: &Value) -> Result<Value> {+ const C: &str = "faces.index";+ let embedder = current(s, C)?;+ collect(s);+ let budget = Duration::from_millis(p.get("budgetMs").and_then(Value::as_u64).unwrap_or(200).min(600_000));+ let first: Vec<PhotoId> =+ p.get("ids").and_then(Value::as_array).map(|a| a.iter().filter_map(Value::as_u64).map(PhotoId).collect()).unwrap_or_default();+ let todo = pending(s, &first, false);+ let started = Instant::now();+ let (mut embedded, mut failed, mut done) = (0, 0, 0);+ for id in &todo {+ // at least one photo per call, so a slow model still makes progress; none when only the status is asked for+ if budget.is_zero() || (done > 0 && started.elapsed() >= budget) {+ break;+ }+ let (e, f) = embed_photo(s, *id, &embedder)?;+ (embedded, failed, done) = (embedded + e, failed + f, done + 1);+ }+ if let Err(e) = s.faces.index.save() {+ log::warn!("could not save the face embeddings: {e}");+ }+ Ok(json!({+ "model": s.faces.index.tag,+ "embedded": embedded,+ "failed": failed,+ "photosDone": done,+ "pendingPhotos": todo.len().saturating_sub(done),+ "indexedFaces": s.faces.index.len(),+ "ms": started.elapsed().as_secs_f64() * 1000.0,+ }))+ }++ fn str_scope(p: &Value) -> Option<&str> {+ p.get("scope").and_then(Value::as_str)+ }++ /// Seconds since 1970 of an ISO capture time (`2024-09-20T10:48:45…`, any zone suffix ignored): only for telling+ /// shots taken close together, so the zone does not matter.+ fn capture_secs(iso: &str) -> Option<i64> {+ let n = |a: usize, b: usize| iso.get(a..b)?.parse::<i64>().ok();+ let (y, m, d, hh, mm, ss) = (n(0, 4)?, n(5, 7)?, n(8, 10)?, n(11, 13)?, n(14, 16)?, n(17, 19)?);+ if !(1..=12).contains(&m) || !(1..=31).contains(&d) {+ return None;+ }+ // days from civil (proleptic Gregorian)+ let y = if m <= 2 { y - 1 } else { y };+ let era = y.div_euclid(400);+ let yoe = y - era * 400;+ let doy = (153 * (if m > 2 { m - 3 } else { m + 9 }) + 2) / 5 + d - 1;+ let doe = yoe * 365 + yoe / 4 - yoe / 100 + doy;+ Some((era * 146_097 + doe - 719_468) * 86_400 + hh * 3600 + mm * 60 + ss)+ }++ /// `faces.evaluate`: leave-one-out over the library's named faces, to see how well recognition works on *your*+ /// photos and which threshold to trust. Each named face in turn is treated as unknown and ranked against the+ /// other named faces, ignoring those from shots taken within `burstSecs` of it (near-duplicates would make+ /// everything look easy). Reports counts only.+ pub fn evaluate(s: &mut Session, p: &Value) -> Result<Value> {+ const C: &str = "faces.evaluate";+ let embedder = current(s, C)?;+ collect(s);+ let burst = p.get("burstSecs").and_then(Value::as_i64).unwrap_or(5).clamp(0, 86_400 * 7);+ let margin = p.get("margin").and_then(Value::as_f64).filter(|v| v.is_finite()).map_or(DEFAULT_MARGIN, |v| v as f32);+ let thresholds: Vec<f32> = p+ .get("thresholds")+ .and_then(Value::as_array)+ .map(|a| a.iter().filter_map(Value::as_f64).filter(|v| v.is_finite()).map(|v| v as f32).take(64).collect())+ .filter(|v: &Vec<f32>| !v.is_empty())+ .unwrap_or_else(|| (4..=16).map(|i| i as f32 * 0.05).collect());+ let budget = Duration::from_millis(p.get("budgetMs").and_then(Value::as_u64).unwrap_or(0).min(3_600_000));+ let started = Instant::now();+ // `scope: "visible"` looks only at the photos the current filter shows (say one year, or one album)+ let scope: Option<std::collections::HashSet<PhotoId>> = (str_scope(p) == Some("visible")).then(|| s.visible_cloned().into_iter().collect());+ let in_scope = |id: PhotoId| scope.as_ref().is_none_or(|v| v.contains(&id));+ let todo: Vec<PhotoId> = pending(s, &[], false).into_iter().filter(|id| in_scope(*id)).collect();+ let mut done = 0;+ for id in &todo {+ if started.elapsed() >= budget {+ break;+ }+ embed_photo(s, *id, &embedder)?;+ done += 1;+ }+ // every named face with its embedding, the photo it is in, and when that was taken+ struct Named {+ name: String,+ emb: Arc<[f32]>,+ photo: PhotoId,+ when: Option<i64>,+ }+ let named: Vec<Named> = s+ .catalog+ .photos()+ .filter(|ph| ph.in_library() && in_scope(ph.id))+ .flat_map(|ph| {+ let when = ph.captured.as_deref().and_then(capture_secs);+ ph.meta+ .regions+ .iter()+ .filter(|r| r.kind == RegionKind::Face)+ .filter_map(|r| Some(Named { name: r.name.clone()?, emb: s.faces.index.get(ph.id.0, &r.rect)?.clone(), photo: ph.id, when }))+ .collect::<Vec<_>>()+ })+ .collect();+ let near = |a: &Named, b: &Named| a.photo == b.photo || matches!((a.when, b.when), (Some(x), Some(y)) if (x - y).abs() <= burst);+ let mut sweep: Vec<(f32, usize, usize)> = thresholds.iter().map(|t| (*t, 0, 0)).collect();+ let (mut queries, mut top1, mut unmatchable) = (0usize, 0usize, 0usize);+ for q in &named {+ let others: Vec<&Named> = named.iter().filter(|g| !near(q, g)).collect();+ if !others.iter().any(|g| g.name.eq_ignore_ascii_case(&q.name)) {+ unmatchable += 1;+ continue;+ }+ let refs: Vec<(&str, &[f32])> = others.iter().map(|g| (g.name.as_str(), &g.emb[..])).collect();+ let ranked = matching::rank(&q.emb, &refs);+ queries += 1;+ if ranked.first().is_some_and(|r| r.name.eq_ignore_ascii_case(&q.name)) {+ top1 += 1;+ }+ for (t, suggested, correct) in sweep.iter_mut() {+ if let Some(sg) = matching::suggest(&ranked, *t, margin) {+ *suggested += 1;+ if sg.name.eq_ignore_ascii_case(&q.name) {+ *correct += 1;+ }+ }+ }+ }+ let pct = |n: usize, d: usize| if d == 0 { Value::Null } else { json!((n as f64 / d as f64 * 1000.0).round() / 1000.0) };+ Ok(json!({+ "model": s.faces.index.tag,+ "namedFaces": named.len(),+ "queries": queries,+ "unmatchable": unmatchable,+ "pendingPhotos": todo.len().saturating_sub(done),+ "top1": pct(top1, queries),+ "margin": margin,+ "burstSecs": burst,+ "sweep": sweep.iter().map(|(t, sg, ok)| json!({"threshold": t, "suggested": sg, "correct": ok, "precision": pct(*ok, *sg), "recall": pct(*ok, queries)})).collect::<Vec<_>>(),+ }))+ }++ /// `faces.pump`: cheap to call every frame. Takes in what the background worker has finished and, when it is idle,+ /// hands it the next photo with faces to embed. Does nothing (and says so) until recognition is switched on in+ /// Settings with a model chosen.+ pub fn pump(s: &mut Session, p: &Value) -> Result<Value> {+ const C: &str = "faces.pump";+ // how hard the scan may work now, from what the user is doing (the app says); nothing new is started while paused,+ // and what is already running finishes+ let pace = Pace::parse(p.get("pace").and_then(Value::as_str));+ let target = target_workers(pace);+ // a model that has finished downloading is installed and switched on here, whether or not recognition was on+ finish_downloads(s);+ let enabled = s.face_models_dir.clone().is_some_and(|d| enabled_cached(s, &d));+ if !enabled {+ return Ok(json!({"active": false}));+ }+ if s.faces.retry_at.is_some_and(|t| Instant::now() < t) {+ return Ok(json!({"active": false}));+ }+ let Ok(embedder) = current_with(s, C, false) else {+ // the chosen model is gone or broken: look again in a few seconds, not every frame+ s.faces.retry_at = Some(Instant::now() + Duration::from_secs(3));+ return Ok(json!({"active": false}));+ };+ s.faces.retry_at = None;+ let (embedded, _) = collect(s);+ if (s.faces.index.has_unsaved() || s.faces.scanned.has_unsaved()) && s.faces.saved_at.is_none_or(|t| t.elapsed() >= SAVE_EVERY) {+ if let Err(e) = s.faces.index.save() {+ log::warn!("could not save the face embeddings: {e}");+ }+ if let Err(e) = s.faces.scanned.save() {+ log::warn!("could not save the list of searched photos: {e}");+ }+ s.faces.saved_at = Some(Instant::now());+ }+ if s.faces.in_flight.is_empty() && s.faces.queue.is_empty() && s.faces.queue_stamp != Some(s.catalog.revision) {+ // the queue is taken from the end, so it is made last first+ let mut queue = pending(s, &[], true);+ queue.reverse();+ s.faces.queue = queue;+ s.faces.queue_stamp = Some(s.catalog.revision);+ }+ if target > 0 && s.faces.worker.is_none() && !s.faces.queue.is_empty() {+ s.faces.worker = Worker::start(max_workers());+ }+ // the workers are allowed `target` photos at once; a couple more wait behind them, so a worker that finishes has its+ // next photo at once instead of waiting for the next frame to hand it one+ if let Some(w) = &s.faces.worker {+ w.allow(target, pace);+ }+ while target > 0 && s.faces.in_flight.len() < target + PREFETCH {+ let Some(id) = s.faces.queue.pop() else { break };+ let Some(job) = prepare(s, id, true) else { continue };+ if s.faces.worker.as_ref().is_some_and(|w| w.submit(embedder.clone(), job)) {+ s.faces.in_flight.insert(id);+ } else {+ break;+ }+ }+ Ok(json!({+ "active": true,+ "inFlight": s.faces.in_flight.len(),+ "pendingPhotos": s.faces.queue.len() + s.faces.in_flight.len(),+ "indexedFaces": s.faces.index.len(),+ "searchedPhotos": s.faces.scanned.len(),+ "workers": target,+ "embedded": embedded,+ }))+ }++ /// The unnamed faces that look most like the person whose lower-cased name is `key`, best first, at most `limit`: each+ /// is scored like a suggestion (the mean of its two closest faces of theirs) and kept above four fifths of the+ /// model's suggestion bar, since a person looks at these before anything is named. Only faces already embedded can be+ /// found; `ready` says whether recognition is running at all.+ pub fn similar(s: &mut Session, key: &str, limit: usize) -> Similar {+ let none = Similar { items: Vec::new(), ready: false, pending: 0 };+ let Some(dir) = s.face_models_dir.clone() else { return none };+ if !enabled_cached(s, &dir) {+ return none;+ }+ let Ok(embedder) = current_with(s, "faces.person", false) else { return none };+ let pending = s.faces.queue.len() + s.faces.in_flight.len();+ let (mut gallery, mut candidates): (Vec<Arc<[f32]>>, Vec<(u64, usize, Rect, Arc<[f32]>)>) = (Vec::new(), Vec::new());+ for ph in s.catalog.photos().filter(|ph| ph.in_library()) {+ for (index, r) in ph.meta.regions.iter().enumerate().filter(|(_, r)| r.kind == RegionKind::Face) {+ let Some(e) = s.faces.index.get(ph.id.0, &r.rect) else { continue };+ match r.name.as_deref() {+ Some(n) if n.trim().to_lowercase() == key => gallery.push(e.clone()),+ Some(_) => {}+ None => candidates.push((ph.id.0, index, r.rect, e.clone())),+ }+ }+ }+ let ready = Similar { items: Vec::new(), ready: true, pending };+ if gallery.is_empty() {+ return ready;+ }+ let bar = embedder.manifest().thresholds.match_cosine.unwrap_or(DEFAULT_THRESHOLD) * 0.8;+ let refs: Vec<(&str, &[f32])> = gallery.iter().map(|e| ("person", &e[..])).collect();+ let mut scored: Vec<(f32, u64, usize, Rect)> = candidates+ .into_iter()+ .filter_map(|(photo, index, rect, e)| {+ let score = matching::rank(&e, &refs).first()?.score;+ (score >= bar).then_some((score, photo, index, rect))+ })+ .collect();+ scored.sort_by(|a, b| b.0.total_cmp(&a.0).then(a.1.cmp(&b.1)).then(a.2.cmp(&b.2)));+ scored.truncate(limit);+ let items = scored+ .into_iter()+ .map(|(score, photo, index, rect)| {+ let view = s.face_view(PhotoId(photo), rect);+ json!({"photo": photo, "index": index, "rect": super::rect_json(&rect), "view": super::rect_json(&view), "score": score})+ })+ .collect();+ Similar { items, ..ready }+ }++ /// Most faces put in order of likeness (the order is a pass over every pair of them).+ const MAX_ARRANGED: usize = 1500;++ /// Faces in an order that puts look-alikes together: start anywhere, then always go to the closest face not yet placed.+ fn nearest_chain(embeddings: &[&[f32]]) -> Vec<usize> {+ let n = embeddings.len();+ let (mut placed, mut order) = (vec![false; n], Vec::with_capacity(n));+ let mut current = 0;+ if n == 0 {+ return order;+ }+ placed[0] = true;+ order.push(0);+ for _ in 1..n {+ let mut best: Option<(usize, f32)> = None;+ for (j, e) in embeddings.iter().enumerate() {+ if placed[j] {+ continue;+ }+ let c = matching::cosine(embeddings[current], e).unwrap_or(-1.0);+ if best.is_none_or(|(_, b)| c > b) {+ best = Some((j, c));+ }+ }+ let Some((j, _)) = best else { break };+ placed[j] = true;+ order.push(j);+ current = j;+ }+ order+ }++ /// Put the unnamed faces `all` in order of likeness (the ones already embedded; the rest follow in photo order) and+ /// find a name for those the named faces recognise, for the first `limit` of them.+ pub fn arrange_unnamed(s: &mut Session, all: &mut Vec<(u64, usize, Rect)>, limit: usize) -> super::Arranged {+ let none = super::Arranged { ordered: false, ready: false, suggestions: Default::default() };+ let Some(dir) = s.face_models_dir.clone() else { return none };+ if !enabled_cached(s, &dir) {+ return none;+ }+ let Ok(embedder) = current_with(s, "faces.unnamed", false) else { return none };+ let (mut embedded, mut plain) = (Vec::new(), Vec::new());+ for f in all.drain(..) {+ match s.faces.index.get(f.0, &f.2) {+ Some(e) => embedded.push((f, e.clone())),+ None => plain.push(f),+ }+ }+ let beyond = embedded.split_off(embedded.len().min(MAX_ARRANGED));+ let order = nearest_chain(&embedded.iter().map(|(_, e)| &e[..]).collect::<Vec<_>>());+ let ordered: Vec<((u64, usize, Rect), Arc<[f32]>)> = order.into_iter().filter_map(|i| embedded.get(i).cloned()).collect();+ // names the faces already named suggest, as `faces.suggest` would+ let gallery: Vec<(String, Arc<[f32]>)> = s+ .catalog+ .photos()+ .filter(|ph| ph.in_library())+ .flat_map(|ph| {+ ph.meta+ .regions+ .iter()+ .filter(|r| r.kind == RegionKind::Face)+ .filter_map(|r| Some((r.name.clone()?, s.faces.index.get(ph.id.0, &r.rect)?.clone())))+ .collect::<Vec<_>>()+ })+ .collect();+ let refs: Vec<(&str, &[f32])> = gallery.iter().map(|(n, e)| (n.as_str(), &e[..])).collect();+ let threshold = embedder.manifest().thresholds.match_cosine.unwrap_or(DEFAULT_THRESHOLD);+ let mut suggestions = std::collections::HashMap::new();+ if !refs.is_empty() {+ for ((photo, index, _), e) in ordered.iter().take(limit) {+ if let Some(x) = matching::suggest(&matching::rank(e, &refs), threshold, DEFAULT_MARGIN) {+ suggestions.insert((*photo, *index), (x.name, x.score));+ }+ }+ }+ let grouped = !ordered.is_empty();+ all.extend(ordered.into_iter().map(|(f, _)| f));+ all.extend(beyond.into_iter().map(|(f, _)| f));+ all.extend(plain);+ super::Arranged { ordered: grouped, ready: true, suggestions }+ }++ /// Whether face recognition is switched on, looked up at most once a second.+ fn enabled_cached(s: &mut Session, dir: &std::path::Path) -> bool {+ if let Some((t, on)) = s.faces.enabled_seen+ && t.elapsed() < RECHECK+ {+ return on;+ }+ let on = read_settings(dir).enabled;+ s.faces.enabled_seen = Some((Instant::now(), on));+ on+ }++ pub fn suggest(s: &mut Session, p: &Value) -> Result<Value> {+ const C: &str = "faces.suggest";+ let embedder = current(s, C)?;+ collect(s);+ let targets = s.targets(p);+ let model_threshold = embedder.manifest().thresholds.match_cosine;+ let num = |k: &str| p.get(k).and_then(Value::as_f64).filter(|v| v.is_finite()).map(|v| v as f32);+ let threshold = num("threshold").or(model_threshold).unwrap_or(DEFAULT_THRESHOLD);+ let margin = num("margin").unwrap_or(DEFAULT_MARGIN);+ // the photos asked about are embedded first, then the named faces they are compared with, within the budget+ let budget = Duration::from_millis(p.get("budgetMs").and_then(Value::as_u64).unwrap_or(1500).min(600_000));+ let started = Instant::now();+ let todo = pending(s, &targets, false);+ let mut done = 0;+ for id in &todo {+ // a budget of zero only looks at what is already embedded+ if budget.is_zero() || (done > 0 && started.elapsed() >= budget) {+ break;+ }+ embed_photo(s, *id, &embedder)?;+ done += 1;+ }+ let pending_photos: Vec<u64> = todo.iter().skip(done).map(|id| id.0).collect();+ let gallery: Vec<(String, Arc<[f32]>)> = s+ .catalog+ .photos()+ .filter(|ph| ph.in_library())+ .flat_map(|ph| {+ ph.meta+ .regions+ .iter()+ .filter(|r| r.kind == RegionKind::Face)+ .filter_map(|r| Some((r.name.clone()?, s.faces.index.get(ph.id.0, &r.rect)?.clone())))+ .collect::<Vec<_>>()+ })+ .collect();+ let refs: Vec<(&str, &[f32])> = gallery.iter().map(|(n, e)| (n.as_str(), &e[..])).collect();+ let mut photos = Vec::new();+ for id in &targets {+ let Some(photo) = s.catalog.photo(*id) else { continue };+ let mut faces = Vec::new();+ for (index, r) in photo.meta.regions.iter().enumerate().filter(|(_, r)| r.kind == RegionKind::Face && r.name.is_none()) {+ let Some(q) = s.faces.index.get(id.0, &r.rect) else {+ faces.push(json!({"index": index, "pending": true}));+ continue;+ };+ let ranked = matching::rank(q, &refs);+ let suggestion = matching::suggest(&ranked, threshold, margin);+ faces.push(json!({+ "index": index,+ "suggestion": suggestion.as_ref().map(|x| json!({"name": x.name, "score": x.score, "runnerUp": x.runner_up.as_ref().map(|(n, sc)| json!({"name": n, "score": sc}))})),+ "candidates": ranked.iter().take(3).map(|c| json!({"name": c.name, "score": c.score, "faces": c.faces})).collect::<Vec<_>>(),+ }));+ }+ photos.push(json!({"id": id.0, "faces": faces}));+ }+ Ok(+ json!({"model": s.faces.index.tag, "threshold": threshold, "margin": margin, "galleryFaces": gallery.len(), "pendingPhotos": pending_photos, "photos": photos}),+ )+ }+ #[cfg(test)]+ mod tests {+ use super::*;+ use crate::tests_face_recognize::{install_yunet, region, set_regions, setup, temp, two_photos};++ fn unit(parts: &[(usize, f32)]) -> Vec<f32> {+ let mut v = vec![0.0; 64];+ for (i, x) in parts {+ v[*i] = *x;+ }+ let n = v.iter().map(|x| x * x).sum::<f32>().sqrt();+ v.iter().map(|x| x / n).collect()+ }++ /// A session with a tiny model loaded (so the index has its model's tag).+ fn loaded(name: &str) -> (Session, std::path::PathBuf) {+ let d = temp(name);+ let (mut s, _) = setup(&d, 64);+ s.execute("faces.index", &json!({"budgetMs": 0})).unwrap();+ (s, d)+ }++ fn finished(s: &Session, id: PhotoId, found: Option<Vec<Rect>>, results: Vec<(Rect, Option<Vec<f32>>, Rect)>) -> Done {+ Done { tag: s.faces.index.tag.clone(), epoch: s.faces.epoch, id, results, detect_wanted: true, found }+ }++ #[test]+ fn faces_found_by_the_scan_become_unnamed_regions_without_an_undo_step() {+ let (mut s, d) = loaded("found");+ let (a, b) = two_photos(&s);+ set_regions(&mut s, b, vec![region(0.1, Some("Ann"))]);+ let (undo, redo) = (s.undo.len(), s.redo.len());+ let found = region(0.3, None).rect;+ let done = finished(&s, a, Some(vec![found]), vec![(found, Some(unit(&[(0, 1.0)])), found)]);+ assert_eq!(ingest(&mut s, done), (1, 0));+ let regions = &s.catalog.photo(a).unwrap().meta.regions;+ assert_eq!(regions.len(), 1);+ assert!(regions[0].kind == RegionKind::Face && regions[0].name.is_none() && regions[0].rect == found);+ assert!(regions[0].description.as_deref().unwrap().starts_with(MARK), "marked as detected");+ // not something the user did, and its embedding is already there+ assert_eq!((s.undo.len(), s.redo.len()), (undo, redo));+ assert!(s.skip_auto_write, "no sidecar is rewritten for it");+ assert!(s.faces.scanned.contains(a.0) && s.faces.index.contains(a.0, &found));+ // a photo that has faces by now (the user, an import) keeps exactly those+ let other = region(0.6, None).rect;+ let done = finished(&s, b, Some(vec![other]), vec![(other, None, other)]);+ ingest(&mut s, done);+ assert_eq!(s.catalog.photo(b).unwrap().meta.regions.len(), 1);+ assert!(s.faces.scanned.contains(b.0));+ let _ = std::fs::remove_dir_all(&d);+ }++ #[test]+ fn a_photo_that_could_not_be_searched_is_left_alone_for_the_session() {+ let (mut s, d) = loaded("scanfail");+ let (a, _) = two_photos(&s);+ assert!(needs_search(&s, s.catalog.photo(a).unwrap()));+ let done = finished(&s, a, None, vec![]);+ ingest(&mut s, done);+ assert!(!s.faces.scanned.contains(a.0), "not remembered as searched: the file may come back");+ assert!(!needs_search(&s, s.catalog.photo(a).unwrap()), "but not retried on every pass");+ assert!(!pending(&s, &[], true).contains(&a));+ let _ = std::fs::remove_dir_all(&d);+ }++ #[test]+ fn only_the_background_scan_looks_for_faces_and_only_in_photos_without_any() {+ let (mut s, d) = loaded("search");+ // finding faces needs the detector, which is not bundled; without it the scan only embeds+ let detector = install_yunet(&mut s);+ let (a, b) = two_photos(&s);+ set_regions(&mut s, b, vec![region(0.1, Some("Ann")), region(0.5, None)]);+ // `faces.index` embeds what is boxed; only the scan (search = true) also searches the rest, last+ assert_eq!(pending(&s, &[], false), vec![b]);+ let scan = pending(&s, &[], true);+ assert_eq!(scan.first(), Some(&b), "named faces first");+ assert!(scan.contains(&a) && scan.len() > 2, "the unsearched photos follow");+ assert!(prepare(&mut s, a, false).is_none(), "nothing to embed in a photo without boxes");+ if detector {+ assert!(prepare(&mut s, a, true).is_some_and(|p| p.detect && p.todo.is_empty()));+ } else {+ assert!(prepare(&mut s, a, true).is_none(), "no detector: nothing to do, and not marked as searched");+ }+ assert!(prepare(&mut s, b, true).is_some_and(|p| !p.detect && p.todo.len() == 2), "a photo with boxes is not searched");+ // once searched, a photo is not searched again+ s.faces.scanned.insert(a.0);+ assert!(!pending(&s, &[], true).contains(&a));+ let _ = std::fs::remove_dir_all(&d);+ }++ #[test]+ fn opening_another_library_forgets_the_old_ones_faces() {+ let (mut s, d) = loaded("newlib");+ let (a, _) = two_photos(&s);+ let rect = region(0.2, None).rect;+ s.faces.index.insert(a.0, &rect, unit(&[(0, 1.0)]));+ s.faces.scanned.insert(a.0);+ let epoch = s.faces.epoch;+ let old = finished(&s, a, Some(vec![rect]), vec![(rect, Some(unit(&[(1, 1.0)])), rect)]);+ s.open_library(d.join("second"), true).unwrap();+ assert_eq!((s.faces.index.len(), s.faces.scanned.len(), s.faces.queue.len(), s.faces.in_flight.len()), (0, 0, 0, 0));+ assert!(s.faces.epoch != epoch && s.faces.embedder.is_none());+ // a worker that was still busy with the old library cannot put its faces into the new one+ assert_eq!(ingest(&mut s, old), (0, 0));+ assert!(s.faces.index.len() == 0 && !s.faces.scanned.contains(a.0));+ let _ = std::fs::remove_dir_all(&d);+ }++ #[test]+ fn the_unnamed_faces_that_look_like_a_person_come_best_first() {+ let (mut s, d) = loaded("similar");+ let (a, b) = two_photos(&s);+ let c = s.catalog.photos().map(|p| p.id).find(|id| *id != a && *id != b).unwrap();+ set_regions(&mut s, a, vec![region(0.1, Some("Ann")), region(0.5, Some("Bob"))]);+ set_regions(&mut s, b, vec![region(0.1, None), region(0.5, None)]);+ set_regions(&mut s, c, vec![region(0.3, None)]);+ let put = |s: &mut Session, id: PhotoId, x: f64, v: Vec<f32>| s.faces.index.insert(id.0, ®ion(x, None).rect, v);+ put(&mut s, a, 0.1, unit(&[(0, 1.0)])); // Ann+ put(&mut s, a, 0.5, unit(&[(1, 1.0)])); // Bob+ put(&mut s, b, 0.1, unit(&[(0, 0.95), (1, 0.05)])); // looks just like Ann+ put(&mut s, b, 0.5, unit(&[(1, 1.0)])); // looks like Bob, not Ann+ put(&mut s, c, 0.3, unit(&[(0, 0.8), (1, 0.6)])); // somewhat like Ann+ let r = similar(&mut s, "ann", 10);+ assert!(r.ready);+ let found: Vec<(u64, u64)> = r.items.iter().map(|i| (i["photo"].as_u64().unwrap(), i["index"].as_u64().unwrap())).collect();+ assert_eq!(found, vec![(b.0, 0), (c.0, 0)], "best first; the face that looks like Bob is left out");+ assert!(r.items[0]["score"].as_f64().unwrap() > r.items[1]["score"].as_f64().unwrap());+ assert_eq!(similar(&mut s, "ann", 1).items.len(), 1, "capped");+ assert!(similar(&mut s, "nobody", 10).items.is_empty());+ let _ = std::fs::remove_dir_all(&d);+ }+ }+}++#[cfg(target_arch = "wasm32")]+mod imp {+ use serde_json::Value;++ use super::super::bad;+ use crate::{Result, Session};++ pub fn index(_: &mut Session, _: &Value) -> Result<Value> {+ Err(bad("faces.index", "this build cannot run face recognition"))+ }++ pub fn suggest(_: &mut Session, _: &Value) -> Result<Value> {+ Err(bad("faces.suggest", "this build cannot run face recognition"))+ }++ pub fn evaluate(_: &mut Session, _: &Value) -> Result<Value> {+ Err(bad("faces.evaluate", "this build cannot run face recognition"))+ }++ pub fn pump(_: &mut Session, _: &Value) -> Result<Value> {+ Ok(serde_json::json!({"active": false}))+ }++ pub fn similar(_: &mut Session, _: &str, _: usize) -> super::Similar {+ super::Similar { items: Vec::new(), ready: false, pending: 0 }+ }++ pub fn arrange_unnamed(_: &mut Session, _: &mut Vec<(u64, usize, lightcraft_geom::Rect)>, _: usize) -> super::Arranged {+ super::Arranged { ordered: false, ready: false, suggestions: Default::default() }+ }+}++pub fn specs() -> Vec<CommandSpec> {+ vec![+ cmd!(+ "faces.index",+ "Index Faces",+ [],+ None,+ "{budgetMs?: 200, ids?: [photo ids to do first]} → {embedded, photosDone, pendingPhotos, indexedFaces, ms} — embed the faces in the library with the chosen recognition model, as many photos as fit in the time (at least one); call again until `pendingPhotos` is 0. `budgetMs: 0` only reports",+ super::always,+ imp::index+ ),+ cmd!(+ "faces.suggest",+ "Suggest Names for Faces",+ [],+ None,+ "{ids?, threshold?, margin?, budgetMs?} → {photos: [{id, faces: [{index, suggestion: {name, score, runnerUp} | null, candidates: [{name, score}]}]}]} — who the unnamed faces in these photos might be, from the faces already named in the library. Only suggestions: nothing is named",+ super::always,+ imp::suggest+ ),+ cmd!(+ query "faces.pump",+ "Index Faces in the Background",+ [],+ None,+ "{pace?: pause | light | normal | full} → {active, inFlight, pendingPhotos, indexedFaces, searchedPhotos, workers} — cheap to call every frame: takes in what the background workers have finished and gives them the next photos, as many at once as the pace allows (none when paused, one when light, half the machine's threads when normal, four fifths when full); inactive until face recognition is on in Settings with a model chosen",+ super::always,+ imp::pump+ ),+ cmd!(+ "faces.evaluate",+ "Evaluate Face Recognition",+ [],+ None,+ "{budgetMs?: 0, burstSecs?: 5, margin?, thresholds?: [..], scope?: \"visible\"} → {queries, top1, sweep: [{threshold, suggested, correct, precision, recall}]} — leave-one-out over your named faces: each is ranked against the others (ignoring shots within burstSecs of it) to show how well the chosen model recognises people in *your* photos and which threshold to trust. Faces not yet embedded are done within budgetMs (`faces.index` does them all); counts only",+ super::always,+ imp::evaluate+ ),+ cmd!(+ query "faces.unnamed",+ "Unnamed Faces",+ [],+ None,+ "{limit?: 600} → {total, faces: [{photo, index, rect, view, suggestion: {name, score} | null}], ordered, ready} — the faces nobody has named; with recognition running, look-alikes are next to each other and a face the named ones recognise carries the name they suggest. `view` is the box to show the face by. Name them with `faces.nameFaces`",+ super::always,+ unnamed+ ),+ cmd!(+ "faces.nameFaces",+ "Name Faces",+ [],+ None,+ "{faces: [{photo, index}], name} → {named, photos} — name (or, with a null or empty name, clear) many face regions as one undoable step; the sidecar is not touched",+ super::always,+ name_faces+ ),+ cmd!(+ query "faces.person",+ "Faces of a Person",+ [],+ None,+ "{name, more?: 120} → {name, total, confirmed: [{photo, index, rect}], more: [{photo, index, rect, score}], ready, pendingPhotos} — the faces named `name` across the library, and the unnamed faces that look most like them (best first, already embedded ones only, once recognition is on); confirm one with `faces.setName`",+ super::always,+ person+ ),+ cmd!(+ "faces.setName",+ "Name Face",+ [],+ None,+ "{id?, index, name} — name a face region (null or empty clears it); undoable; the sidecar is not touched",+ super::always,+ set_name+ ),+ ]+}diff --git a/crates/engine/src/cmd/mod.rs b/crates/engine/src/cmd/mod.rsindex b5909202f..303a92d45 100644--- a/crates/engine/src/cmd/mod.rs+++ b/crates/engine/src/cmd/mod.rs@@ -13,8 +13,9 @@ pub mod curves;mod develop;mod edit;mod export;-mod face_detect;+pub(crate) mod face_detect;mod face_models;+mod face_recognize;pub mod filters;pub mod keywords;pub mod library;@@ -144,6 +145,7 @@ pub fn command_specs() -> &'static [CommandSpec] {v.extend(filters::specs());v.extend(face_models::specs());v.extend(face_detect::specs());+ v.extend(face_recognize::specs());v})}diff --git a/crates/engine/src/face_download.rs b/crates/engine/src/face_download.rsindex cac1b2e64..0c6ed4ee1 100644--- a/crates/engine/src/face_download.rs+++ b/crates/engine/src/face_download.rs@@ -69,13 +69,14 @@ impl Downloads {/// Start fetching `spec` into `<models dir>/.downloads/`. Different models can download side by side; the same/// one is started once.pub fn start(&mut self, spec: Download, models_dir: &Path) -> Result<(), String> {- if self.jobs.get(spec.id).is_some_and(|j| matches!(j.get(), State::Running { .. })) {+ if self.jobs.get(spec.id.as_str()).is_some_and(|j| matches!(j.get(), State::Running { .. })) {return Err("that model is already downloading".into());}let staging = models_dir.join(STAGING);std::fs::create_dir_all(&staging).map_err(|e| format!("could not create the download folder: {e}"))?;let shared = Arc::new(Shared::new(spec.size_bytes));let worker = shared.clone();+ let id = spec.id.clone();std::thread::Builder::new().name("face-model-download".into()).spawn(move || {@@ -85,7 +86,7 @@ impl Downloads {}}).map_err(|e| format!("could not start the download: {e}"))?;- self.jobs.insert(spec.id.to_string(), shared);+ self.jobs.insert(id, shared);Ok(())}@@ -187,7 +188,7 @@ fn explain(e: &lightcraft_fetch::DownloadError) -> String {/// Fetch `spec` from `mirror` (the address without the file name) into `staging`, check it and leave it there.#[cfg(not(target_arch = "wasm32"))]fn fetch(spec: &Download, mirror: &str, staging: &Path, shared: &Shared) -> Result<(), lightcraft_fetch::DownloadError> {- let file = lightcraft_fetch::FileSpec { name: spec.file_name, size: Some(spec.size_bytes), sha256: Some(spec.sha256), max: spec.size_bytes };+ let file = lightcraft_fetch::FileSpec { name: &spec.file_name, size: Some(spec.size_bytes), sha256: Some(&spec.sha256), max: spec.size_bytes };lightcraft_fetch::download(&[file], &[mirror.to_string()], staging, &lightcraft_fetch::Options::default(), &shared.cancel, &mut |p| {shared.set(State::Running { bytes: p.done.min(spec.size_bytes), total: spec.size_bytes });})@@ -200,7 +201,7 @@ fn fetch(_: &Download, _: &str, _: &Path, _: &Shared) -> Result<(), String> {/// The whole job: reuse a verified earlier copy, else fetch, check and stage. Always ends in a final [`State`].fn run(spec: &Download, mirror: &str, staging: &Path, shared: &Shared) {- let target = staging.join(spec.file_name);+ let target = staging.join(&spec.file_name);let part = staging.join(format!("{}.part", spec.file_name));match fetch(spec, mirror, staging, shared) {Ok(()) => shared.set(State::Done { path: target, sha256: spec.sha256.to_string() }),@@ -236,17 +237,13 @@ mod tests {d}- fn leak(s: String) -> &'static str {- Box::leak(s.into_boxed_str())- }-fn spec_for(bytes: &[u8]) -> Download {Download {- id: "test-model",- url: "https://example.invalid/m/test-model.onnx",- file_name: "test-model.onnx",+ id: "test-model".into(),+ url: "https://example.invalid/m/test-model.onnx".into(),+ file_name: "test-model.onnx".into(),size_bytes: bytes.len() as u64,- sha256: leak(sha256_hex(bytes)),+ sha256: sha256_hex(bytes),}}diff --git a/crates/engine/src/faces_index.rs b/crates/engine/src/faces_index.rsnew file mode 100644index 000000000..c250f8683--- /dev/null+++ b/crates/engine/src/faces_index.rs@@ -0,0 +1,455 @@+//! The face index: one embedding per face region, kept in memory and cached beside the library.+//!+//! Embedding a face costs a model run (tens to hundreds of milliseconds), so each region is embedded once and+//! remembered, keyed by its photo and its box. The cache file (`face-embeddings-<model id>.bin` in the library folder,+//! one per model, so switching back to an earlier model does not start over) belongs to one model: its header names the+//! model and its file hash, and a file made by another model, an older format, or a damaged one is simply ignored and+//! rebuilt. Embeddings are kept out of the catalog on purpose: they are large, rebuildable, and not comparable between+//! models.+//!+//! Each face also keeps the box to cut its picture from (its "view"): the detector's box when the detector found the same+//! face, so every face is shown equally close however loosely or tightly its own region was drawn (a Lightroom box, an+//! older tool's box), else the region's own box.+//!+//! Layout (little-endian): `"LCFE"`, version `2`, tag length `u16`, the tag (`<model id>@<sha256>`), dimension `u32`,+//! then records of `photo u64`, four `i32` (the region's box in ten-thousandths), four `i32` (the view box likewise), and+//! `dim` `f32`.++use std::collections::HashMap;+use std::io::Write;+use std::path::{Path, PathBuf};+use std::sync::Arc;++use lightcraft_geom::Rect;++const MAGIC: &[u8; 4] = b"LCFE";+const VERSION: u8 = 2;+/// Largest cache file we read.+const MAX_FILE: u64 = 1 << 30;++pub(crate) type Key = (u64, [i32; 4]);++/// One face's record: its embedding and its view box (quantized like a key).+type Entry = (Arc<[f32]>, [i32; 4]);++/// A quantized box as a rectangle again.+fn rect_of(k: [i32; 4]) -> Rect {+ let f = |v: i32| f64::from(v) / 10_000.0;+ Rect { x0: f(k[0]), y0: f(k[1]), x1: f(k[2]), y1: f(k[3]) }+}++/// A region's box as the integer key it is cached under.+pub(crate) fn region_key(r: &Rect) -> [i32; 4] {+ let q = |v: f64| if v.is_finite() { (v.clamp(-1e3, 1e3) * 10_000.0).round() as i32 } else { 0 };+ [q(r.x0), q(r.y0), q(r.x1), q(r.y1)]+}++#[derive(Default)]+pub(crate) struct Index {+ /// `<model id>@<sha256>` of the model the embeddings belong to.+ pub tag: String,+ pub dim: usize,+ entries: HashMap<Key, Entry>,+ unsaved: Vec<(Key, Entry)>,+ /// Faces that could not be embedded this session (so they are not retried on every call).+ skipped: std::collections::HashSet<Key>,+ /// The cache file on disk belongs to this model, so new records can be appended to it.+ file_matches: bool,+ pub path: Option<PathBuf>,+}++impl Index {+ pub fn len(&self) -> usize {+ self.entries.len()+ }++ pub fn get(&self, photo: u64, rect: &Rect) -> Option<&Arc<[f32]>> {+ self.entries.get(&(photo, region_key(rect))).map(|e| &e.0)+ }++ /// The box to cut this face's picture from, once the scan has looked at it.+ pub fn view(&self, photo: u64, rect: &Rect) -> Option<Rect> {+ self.entries.get(&(photo, region_key(rect))).map(|e| rect_of(e.1))+ }++ #[cfg(test)]+ pub fn contains(&self, photo: u64, rect: &Rect) -> bool {+ self.entries.contains_key(&(photo, region_key(rect)))+ }++ /// Whether this face still has to be embedded.+ pub fn needs(&self, photo: u64, rect: &Rect) -> bool {+ let key = (photo, region_key(rect));+ !self.entries.contains_key(&key) && !self.skipped.contains(&key)+ }++ pub fn skip(&mut self, photo: u64, rect: &Rect) {+ self.skipped.insert((photo, region_key(rect)));+ }++ #[cfg(test)]+ pub fn insert(&mut self, photo: u64, rect: &Rect, embedding: Vec<f32>) {+ self.insert_with_view(photo, rect, embedding, rect);+ }++ /// Remember a face's embedding and the box to show it by.+ pub fn insert_with_view(&mut self, photo: u64, rect: &Rect, embedding: Vec<f32>, view: &Rect) {+ let entry: Entry = (embedding.into(), region_key(view));+ let key = (photo, region_key(rect));+ self.entries.insert(key, entry.clone());+ self.unsaved.push((key, entry));+ }++ /// Whether there are embeddings not yet written to the cache file.+ pub fn has_unsaved(&self) -> bool {+ !self.unsaved.is_empty()+ }++ /// Start over for `tag` (`dim` numbers per face), loading what the cache file at `path` holds for it. What the+ /// previous model's index had not written yet is written first, so switching models loses nothing.+ pub fn reset(&mut self, tag: &str, dim: usize, path: Option<PathBuf>) {+ let _ = self.save();+ self.tag = tag.to_string();+ self.dim = dim;+ self.path = path;+ self.entries.clear();+ self.unsaved.clear();+ self.skipped.clear();+ self.file_matches = false;+ if let Some(p) = self.path.clone() {+ self.load(&p);+ }+ }++ fn load(&mut self, path: &Path) {+ let Ok(meta) = std::fs::metadata(path) else { return };+ if meta.len() > MAX_FILE {+ return;+ }+ let Ok(bytes) = std::fs::read(path) else { return };+ let Some((tag_len, rest)) = bytes+ .strip_prefix(MAGIC)+ .and_then(|r| r.split_first())+ .filter(|(v, _)| **v == VERSION)+ .and_then(|(_, r)| r.split_at_checked(2))+ .map(|(l, r)| (usize::from(u16::from_le_bytes([l[0], l[1]])), r))+ else {+ return;+ };+ let Some((tag, rest)) = rest.split_at_checked(tag_len) else { return };+ let Some((dim, mut rest)) = rest.split_at_checked(4).map(|(d, r)| (u32::from_le_bytes([d[0], d[1], d[2], d[3]]) as usize, r)) else { return };+ if tag != self.tag.as_bytes() || dim != self.dim || dim == 0 {+ return;+ }+ self.file_matches = true;+ let record = 8 + 16 + 16 + dim * 4;+ while let Some((rec, tail)) = rest.split_at_checked(record) {+ rest = tail;+ let photo = u64::from_le_bytes([rec[0], rec[1], rec[2], rec[3], rec[4], rec[5], rec[6], rec[7]]);+ let int = |at: usize| i32::from_le_bytes([rec[at], rec[at + 1], rec[at + 2], rec[at + 3]]);+ let key = [int(8), int(12), int(16), int(20)];+ let view = [int(24), int(28), int(32), int(36)];+ let emb: Vec<f32> = rec[40..].as_chunks::<4>().0.iter().map(|c| f32::from_le_bytes(*c)).collect();+ if emb.iter().all(|v| v.is_finite()) {+ self.entries.insert((photo, key), (emb.into(), view));+ }+ }+ }++ fn header(&self) -> Vec<u8> {+ let tag = self.tag.as_bytes();+ let mut h = Vec::with_capacity(11 + tag.len());+ h.extend_from_slice(MAGIC);+ h.push(VERSION);+ h.extend_from_slice(&u16::try_from(tag.len()).unwrap_or(0).to_le_bytes());+ h.extend_from_slice(tag);+ h.extend_from_slice(&(self.dim as u32).to_le_bytes());+ h+ }++ fn record(&self, key: &Key, entry: &Entry) -> Vec<u8> {+ let mut r = Vec::with_capacity(40 + entry.0.len() * 4);+ r.extend_from_slice(&key.0.to_le_bytes());+ key.1.iter().chain(entry.1.iter()).for_each(|k| r.extend_from_slice(&k.to_le_bytes()));+ entry.0.iter().for_each(|v| r.extend_from_slice(&v.to_le_bytes()));+ r+ }++ /// Write what was added since the last save: appended when the file already belongs to this model, else the+ /// whole index is written fresh (atomically). A failure is returned, the records stay pending.+ pub fn save(&mut self) -> std::io::Result<()> {+ let Some(path) = self.path.clone() else {+ self.unsaved.clear();+ return Ok(());+ };+ if self.unsaved.is_empty() {+ return Ok(());+ }+ if self.file_matches {+ let mut f = std::fs::OpenOptions::new().append(true).open(&path)?;+ let mut buf = Vec::new();+ for (k, e) in &self.unsaved {+ buf.extend(self.record(k, e));+ }+ f.write_all(&buf)?;+ } else {+ let mut buf = self.header();+ for (k, e) in &self.entries {+ buf.extend(self.record(k, e));+ }+ let part = path.with_extension("part");+ std::fs::write(&part, &buf)?;+ std::fs::rename(&part, &path)?;+ self.file_matches = true;+ }+ self.unsaved.clear();+ Ok(())+ }+}++impl Drop for Index {+ fn drop(&mut self) {+ let _ = self.save();+ }+}++const SCAN_MAGIC: &[u8; 4] = b"LCFS";+const SCAN_VERSION: u8 = 1;+/// The detector a search was made with: a record made by another one is ignored (a better detector may find more).+pub(crate) const SCAN_TAG: &str = "yunet-2023mar";+/// Largest record file we read.+const SCAN_MAX_FILE: u64 = 64 << 20;++/// The photos the detector has already searched for faces, so each is searched once, however many times LightCraft+/// starts. Kept beside the library (`face-scanned.bin`: a header naming the detector, then photo ids, appended to).+#[derive(Default)]+pub(crate) struct Scanned {+ set: std::collections::HashSet<u64>,+ unsaved: Vec<u64>,+ path: Option<PathBuf>,+ file_matches: bool,+}++impl Scanned {+ pub fn path(&self) -> Option<&Path> {+ self.path.as_deref()+ }++ pub fn contains(&self, photo: u64) -> bool {+ self.set.contains(&photo)+ }++ pub fn len(&self) -> usize {+ self.set.len()+ }++ pub fn insert(&mut self, photo: u64) {+ if self.set.insert(photo) {+ self.unsaved.push(photo);+ }+ }++ pub fn has_unsaved(&self) -> bool {+ !self.unsaved.is_empty()+ }++ /// Start over with the record at `path` (written first: what was not yet saved is not lost).+ pub fn reset(&mut self, path: Option<PathBuf>) {+ let _ = self.save();+ self.set.clear();+ self.unsaved.clear();+ self.file_matches = false;+ self.path = path;+ if let Some(p) = self.path.clone() {+ self.load(&p);+ }+ }++ fn header() -> Vec<u8> {+ let mut h = Vec::new();+ h.extend_from_slice(SCAN_MAGIC);+ h.push(SCAN_VERSION);+ h.extend_from_slice(&(SCAN_TAG.len() as u16).to_le_bytes());+ h.extend_from_slice(SCAN_TAG.as_bytes());+ h+ }++ fn load(&mut self, path: &Path) {+ let Ok(meta) = std::fs::metadata(path) else { return };+ if meta.len() > SCAN_MAX_FILE {+ return;+ }+ let Ok(bytes) = std::fs::read(path) else { return };+ let head = Self::header();+ let Some(rest) = bytes.strip_prefix(head.as_slice()) else { return };+ self.file_matches = true;+ // a half-written last id (a crash while appending) is ignored+ self.set.extend(rest.as_chunks::<8>().0.iter().map(|c| u64::from_le_bytes(*c)));+ }++ /// Write what was added since the last save: appended when the file is this detector's, else the whole record+ /// is written fresh (atomically). A failure is returned and the ids stay pending.+ pub fn save(&mut self) -> std::io::Result<()> {+ let Some(path) = self.path.clone() else {+ self.unsaved.clear();+ return Ok(());+ };+ if self.unsaved.is_empty() {+ return Ok(());+ }+ if self.file_matches {+ let mut f = std::fs::OpenOptions::new().append(true).open(&path)?;+ let buf: Vec<u8> = self.unsaved.iter().flat_map(|id| id.to_le_bytes()).collect();+ f.write_all(&buf)?;+ } else {+ let mut buf = Self::header();+ buf.extend(self.set.iter().flat_map(|id| id.to_le_bytes()));+ let part = path.with_extension("part");+ std::fs::write(&part, &buf)?;+ std::fs::rename(&part, &path)?;+ self.file_matches = true;+ }+ self.unsaved.clear();+ Ok(())+ }+}++impl Drop for Scanned {+ fn drop(&mut self) {+ let _ = self.save();+ }+}++#[cfg(test)]+mod tests {+ use super::*;++ fn rect(x: f64) -> Rect {+ Rect { x0: x, y0: 0.1, x1: x + 0.2, y1: 0.4 }+ }++ fn temp(name: &str) -> PathBuf {+ std::env::temp_dir().join(format!("lc-faceindex-{name}-{}.bin", std::process::id()))+ }++ #[test]+ fn embeddings_survive_a_save_and_load_and_append() {+ let path = temp("roundtrip");+ let _ = std::fs::remove_file(&path);+ let mut a = Index::default();+ a.reset("m@abc", 3, Some(path.clone()));+ a.insert(7, &rect(0.1), vec![1.0, 0.0, 0.5]);+ a.insert(8, &rect(0.3), vec![0.0, 1.0, 0.25]);+ a.save().unwrap();+ a.insert(9, &rect(0.5), vec![0.5, 0.5, 0.5]);+ a.save().unwrap(); // appended+ let mut b = Index::default();+ b.reset("m@abc", 3, Some(path.clone()));+ assert_eq!(b.len(), 3);+ assert_eq!(b.get(8, &rect(0.3)).map(|e| e.to_vec()), Some(vec![0.0, 1.0, 0.25]));+ assert!(b.contains(9, &rect(0.5)) && !b.contains(9, &rect(0.6)) && !b.contains(1, &rect(0.5)));+ let _ = std::fs::remove_file(path);+ }++ #[test]+ fn another_model_a_damaged_file_or_a_torn_tail_are_handled() {+ let path = temp("other");+ let _ = std::fs::remove_file(&path);+ let mut a = Index::default();+ a.reset("m@abc", 3, Some(path.clone()));+ a.insert(1, &rect(0.1), vec![1.0, 0.0, 0.0]);+ a.insert(2, &rect(0.2), vec![0.0, 1.0, 0.0]);+ a.save().unwrap();+ // another model: nothing is loaded, and saving replaces the file for the new model+ let mut other = Index::default();+ other.reset("n@def", 4, Some(path.clone()));+ assert_eq!(other.len(), 0);+ other.insert(5, &rect(0.1), vec![1.0, 0.0, 0.0, 0.0]);+ other.save().unwrap();+ let mut again = Index::default();+ again.reset("n@def", 4, Some(path.clone()));+ assert_eq!(again.len(), 1);+ let mut old = Index::default();+ old.reset("m@abc", 3, Some(path.clone()));+ assert_eq!(old.len(), 0, "the old model's records are gone");+ // a torn last record loses only that record+ let bytes = std::fs::read(&path).unwrap();+ std::fs::write(&path, &bytes[..bytes.len() - 5]).unwrap();+ let mut torn = Index::default();+ torn.reset("n@def", 4, Some(path.clone()));+ assert_eq!(torn.len(), 0);+ // garbage of every kind is ignored+ for junk in [Vec::new(), b"LCFE".to_vec(), b"LCFE\x01\xff\xff".to_vec(), vec![0xff; 200], b"LCFE\x02\x00\x00".to_vec()] {+ std::fs::write(&path, &junk).unwrap();+ let mut j = Index::default();+ j.reset("n@def", 4, Some(path.clone()));+ assert_eq!(j.len(), 0);+ }+ // and a missing file is fine+ let _ = std::fs::remove_file(&path);+ let mut m = Index::default();+ m.reset("n@def", 4, Some(path.clone()));+ assert_eq!(m.len(), 0);+ }++ #[test]+ fn non_finite_boxes_and_vectors_do_not_break_the_keys() {+ assert_eq!(region_key(&Rect { x0: f64::NAN, y0: f64::INFINITY, x1: 1e300, y1: -1e300 }), [0, 0, 10_000_000, -10_000_000]);+ let path = temp("nan");+ let _ = std::fs::remove_file(&path);+ let mut a = Index::default();+ a.reset("m@abc", 2, Some(path.clone()));+ a.insert(1, &rect(0.1), vec![f32::NAN, 1.0]);+ a.save().unwrap();+ let mut b = Index::default();+ b.reset("m@abc", 2, Some(path.clone()));+ assert_eq!(b.len(), 0, "a non-finite vector is not loaded back");+ let _ = std::fs::remove_file(path);+ }+}++/// What the recognition commands keep between calls: the loaded model and the embeddings made with it.+#[derive(Default)]+pub(crate) struct FacesState {+ /// `<model id>@<sha256>` and the loaded model.+ pub embedder: Option<(String, Arc<lightcraft_faces::runtime::Embedder>)>,+ pub index: Index,+ /// When the models folder was last looked at (the per-frame pump trusts a look for a second).+ pub checked: Option<std::time::Instant>,+ /// Whether recognition was switched on when last looked up, and when.+ pub enabled_seen: Option<(std::time::Instant, bool)>,+ pub worker: Option<crate::faces_worker::Worker>,+ /// After a failure to load the chosen model, the background pump leaves it alone until then.+ pub retry_at: Option<std::time::Instant>,+ /// Photos the worker is busy with.+ pub in_flight: std::collections::HashSet<lightcraft_catalog::PhotoId>,+ /// Photos still to hand to the worker, and the catalog revision this list was made at.+ pub queue: Vec<lightcraft_catalog::PhotoId>,+ pub queue_stamp: Option<u64>,+ /// When the index was last written to its cache file by the background pump.+ pub saved_at: Option<std::time::Instant>,+ /// Bumped when another library is opened: what a worker was making for the old one is dropped.+ pub epoch: u64,+ /// Photos already searched for faces.+ pub scanned: Scanned,+ /// Photos that could not be searched this session (unreadable files): left alone until the next launch.+ pub scan_failed: std::collections::HashSet<u64>,+}++impl FacesState {+ /// Another library is being opened: what was learned about the old one is saved, and nothing of it is kept (photo+ /// ids are only unique within a library). The loaded model stays; the index is reloaded for the new library.+ pub fn library_changed(&mut self) {+ self.index.reset("", 0, None);+ self.scanned.reset(None);+ self.scan_failed.clear();+ self.embedder = None;+ self.checked = None;+ self.queue.clear();+ self.queue_stamp = None;+ self.in_flight.clear();+ self.retry_at = None;+ self.saved_at = None;+ self.epoch = self.epoch.wrapping_add(1);+ }+}diff --git a/crates/engine/src/faces_worker.rs b/crates/engine/src/faces_worker.rsnew file mode 100644index 000000000..81ff6adae--- /dev/null+++ b/crates/engine/src/faces_worker.rs@@ -0,0 +1,483 @@+//! The heavy part of scanning faces, run off the session thread, on a few threads at once.+//!+//! For one photo: render it (upright, uncropped, default settings), find its faces with the YuNet detector (when installed)+//! so each region can be aligned by landmarks, cut each region out and run the recognition model. A photo that has+//! no face regions at all and has not been searched yet also gets its detections back, with their embeddings, so the+//! scan decodes each photo once. The session thread only prepares the job (which needs the catalog) and later ingests+//! the result, so the window never waits for a model.++use std::panic::{AssertUnwindSafe, catch_unwind};+use std::sync::atomic::{AtomicU8, AtomicUsize, Ordering};+use std::sync::mpsc::{Receiver, Sender, channel};+use std::sync::{Arc, Condvar, Mutex, PoisonError};+use std::time::Duration;++use lightcraft_catalog::PhotoId;+use lightcraft_faces::align::{Rgb, align_to_template, crop_box};+use lightcraft_faces::runtime::Embedder;+use lightcraft_faces::yunet::{Detector, Face, Options};+use lightcraft_geom::Rect;++use crate::media::{PreviewLoader, RenderJob};++/// Long edge of the picture faces are cut from.+pub(crate) const EDGE: usize = 2048;+/// A box from an XMP sidecar is cut with this much room around it (there are no landmarks to align by).+pub(crate) const BOX_PAD: f32 = 1.5;+/// How much a detected face must overlap a region to lend it its landmarks.+pub(crate) const MATCH_IOU: f32 = 0.3;+/// A detection must be this sure to become a face region (the bar `faces.detect` uses).+pub(crate) const REGION_SCORE: f32 = 0.6;+/// A raw's embedded preview is used for the scan only when its long edge is at least this (a thumbnail is too small to+/// find faces in; the raw itself is decoded then).+const MIN_PREVIEW_EDGE: usize = 1000;++/// Most photos ever worked on at once.+const MAX_WORKERS: usize = 24;+/// What one photo in progress holds: the file, its decoded picture and the copies the detector and recogniser read.+/// Measured: peak memory grew by about 180 MB for each extra worker.+const WORKER_BYTES: usize = 180 << 20;+/// Photos queued behind the ones running, so a worker that finishes never waits for the next frame to be fed.+pub(crate) const PREFETCH: usize = 2;++/// How hard the background scan may work right now, as the app judges from what the user is doing.+#[derive(Clone, Copy, Debug, PartialEq, Eq)]+pub(crate) enum Pace {+ /// Start nothing new: the user is dragging, typing or scrolling.+ Pause = 0,+ /// One photo at a time: the user is around, or the window is minimized.+ Light = 1,+ /// Half the machine: the user is idle.+ Normal = 2,+ /// Most of the machine: the user is watching the progress.+ Full = 3,+}++impl Pace {+ fn from_u8(n: u8) -> Pace {+ match n {+ 0 => Pace::Pause,+ 1 => Pace::Light,+ 3 => Pace::Full,+ _ => Pace::Normal,+ }+ }++ /// `pause`, `light`, `normal` or `full`; anything else (or nothing) is `normal`.+ pub fn parse(name: Option<&str>) -> Pace {+ match name {+ Some("pause") => Pace::Pause,+ Some("light") => Pace::Light,+ Some("full") => Pace::Full,+ _ => Pace::Normal,+ }+ }+}++/// How many photos are worked on at once at `pace` on a machine with `cores` threads (as the system reports them, which+/// respects container limits and affinity). `cap` is a limit the user set (`LIGHTCRAFT_FACE_THREADS`). Memory is not+/// decided here: decodes wait at the process-wide memory gate ([`crate::memory::work_gate`]), sized from the RAM.+pub(crate) fn workers_for(cores: usize, pace: Pace, cap: Option<usize>) -> usize {+ let n = match pace {+ Pace::Pause => return 0,+ Pace::Light => 1,+ Pace::Normal => cores / 2,+ Pace::Full => cores * 4 / 5,+ }+ .clamp(1, MAX_WORKERS);+ cap.map_or(n, |c| n.min(c.clamp(1, MAX_WORKERS)))+}++fn cores() -> usize {+ std::thread::available_parallelism().map_or(1, |n| n.get())+}++fn cap_from_env() -> Option<usize> {+ std::env::var("LIGHTCRAFT_FACE_THREADS").ok().and_then(|v| v.trim().parse::<usize>().ok())+}++/// The most photos the memory budget can hold in progress at once: half of it (the rest is the window's, the caches',+/// the renders'), a worker at a time. Raising the budget (`LIGHTCRAFT_MEMORY_MB`) raises this.+fn memory_workers() -> usize {+ (crate::memory::budget() / 2 / WORKER_BYTES).max(1)+}++/// Photos to work on at once at `pace` on this machine: by the processor's threads and the pace, and never more than+/// the memory allows. A limit the user sets (`LIGHTCRAFT_FACE_THREADS`) replaces the memory one.+pub(crate) fn target_workers(pace: Pace) -> usize {+ workers_for(cores(), pace, Some(cap_from_env().unwrap_or_else(memory_workers)))+}++/// Threads the worker pool starts with: enough for the fullest pace (the idle ones cost nothing).+pub(crate) fn max_workers() -> usize {+ target_workers(Pace::Full)+}++/// Threads for the scan's own parallel work (decoding a picture, developing it) at `pace` on a machine with `cores` threads:+/// two when light, half the machine when normal, four fifths when watched. A photo's parallel parts are most of its cost,+/// so this, more than the number of photos at once, is how much of the processor the scan takes. Measured on a 32-thread+/// desktop (4 photos at once): 8 threads scanned 2.2 photos a second, 16 threads 5.8, 25 threads 6.4.+fn pool_threads_for(cores: usize, pace: Pace, cap: Option<usize>) -> usize {+ let n = match pace {+ Pace::Pause | Pace::Light => 2,+ Pace::Normal => cores / 2,+ Pace::Full => cores * 4 / 5,+ }+ .clamp(2, 32);+ cap.map_or(n, |c| n.min(c.max(1)))+}++/// One photo's faces to embed (and, if `detect`, to find first).+pub(crate) struct Prepared {+ /// `<model id>@<sha256>` the embeddings are for.+ pub tag: String,+ /// The session's face epoch when this was prepared: another library has been opened since if it differs.+ pub epoch: u64,+ pub id: PhotoId,+ pub job: RenderJob,+ /// A raw's embedded preview, tried before the render job (much faster than decoding the raw).+ pub preview: Option<(String, PreviewLoader)>,+ pub todo: Vec<Rect>,+ /// The photo has no face regions and has not been searched: look for faces too.+ pub detect: bool,+ /// The YuNet detector, when the user has installed it: faces are aligned by its landmarks, and it finds the faces of+ /// photos that have none. Without it a scan only embeds the faces the photos already have, cut by their boxes.+ pub detector: Option<Arc<Detector>>,+}++/// The embeddings made for a [`Prepared`]; `None` for a face that could not be embedded.+pub(crate) struct Done {+ pub tag: String,+ pub epoch: u64,+ pub id: PhotoId,+ /// Each face's region, its embedding (`None` when it could not be made) and the box to show it by.+ pub results: Vec<(Rect, Option<Vec<f32>>, Rect)>,+ /// Faces were to be looked for in this photo.+ pub detect_wanted: bool,+ /// What the detector found (their embeddings are in `results`); `None` when the photo could not be searched.+ pub found: Option<Vec<Rect>>,+}++fn overlap(a: &Rect, f: &Face) -> f32 {+ let (w, h) = ((a.x1.min(f64::from(f.x1)) - a.x0.max(f64::from(f.x0))).max(0.0), (a.y1.min(f64::from(f.y1)) - a.y0.max(f64::from(f.y0))).max(0.0));+ let inter = w * h;+ let union = (a.x1 - a.x0) * (a.y1 - a.y0) + f64::from((f.x1 - f.x0) * (f.y1 - f.y0)) - inter;+ if union > 0.0 { (inter / union) as f32 } else { 0.0 }+}++/// A detection as a region box: inside the photo and not a sliver; `None` for anything else (it is only ever data).+fn face_rect(f: &Face) -> Option<Rect> {+ if ![f.x0, f.y0, f.x1, f.y1].iter().all(|v| v.is_finite()) {+ return None;+ }+ let c = |v: f32| f64::from(v.clamp(0.0, 1.0));+ let r = Rect { x0: c(f.x0), y0: c(f.y0), x1: c(f.x1), y1: c(f.y1) };+ (r.x1 - r.x0 > 0.005 && r.y1 - r.y0 > 0.005).then_some(r)+}++/// Do the work for one photo. Never panics: whatever goes wrong, its faces come back as `None`.+pub(crate) fn process(p: Prepared, embedder: &Embedder) -> Done {+ let Prepared { tag, epoch, id, job, preview, todo, detect, detector } = p;+ let fallback = |todo: Vec<Rect>| todo.into_iter().map(|r| (r, None, r)).collect::<Vec<_>>();+ let rects = todo.clone();+ let run = catch_unwind(AssertUnwindSafe(|| {+ let from_preview = preview.and_then(|(path, load)| load(&path, EDGE)).filter(|img| img.width.max(img.height) >= MIN_PREVIEW_EDGE);+ let image = match from_preview {+ Some(img) => img,+ None => match job.run().rendered {+ Ok(rendered) => rendered.image,+ Err(_) => return (fallback(todo), None),+ },+ };+ let (w, h) = (image.width, image.height);+ let rgb: Vec<u8> = image.data.iter().flat_map(|px| [px[0], px[1], px[2]]).collect();+ // the detector's landmarks, where it finds the same face: faces are aligned by them, not cut by a box+ let detected = detector.as_deref().and_then(|d| d.detect(&rgb, w, h, &Options { score: 0.5, ..Options::default() }).ok());+ let found = detected.clone().unwrap_or_default();+ let img = Rgb { data: &rgb, width: w, height: h };+ let (iw, ih) = embedder.input_size();+ let scaled = |f: &Face| f.landmarks.map(|(x, y)| (x * w as f32, y * h as f32));+ let mut results: Vec<(Rect, Option<Vec<f32>>, Rect)> = todo+ .into_iter()+ .map(|rect| {+ let best = found.iter().map(|f| (overlap(&rect, f), f)).filter(|(o, _)| *o >= MATCH_IOU).max_by(|a, b| a.0.total_cmp(&b.0));+ let crop = match best {+ Some((_, f)) => align_to_template(&img, &scaled(f), iw, ih),+ None => crop_box(+ &img,+ rect.x0 as f32 * w as f32,+ rect.y0 as f32 * h as f32,+ rect.x1 as f32 * w as f32,+ rect.y1 as f32 * h as f32,+ BOX_PAD,+ iw,+ ih,+ ),+ };+ let v = crop.and_then(|c| embedder.embed(&c).ok());+ // shown by the detector's box where it found this face: every face then has the same tightness+ let view = best.and_then(|(_, f)| face_rect(f)).unwrap_or(rect);+ (rect, v, view)+ })+ .collect();+ // a photo with no face regions: what the detector is sure of becomes its faces, embedded in the same pass+ let new_faces = if detect {+ detected.map(|faces| {+ let mut rects = Vec::new();+ for f in faces.iter().filter(|f| f.score >= REGION_SCORE) {+ let Some(rect) = face_rect(f) else { continue };+ let v = align_to_template(&img, &scaled(f), iw, ih).and_then(|c| embedder.embed(&c).ok());+ results.push((rect, v, rect));+ rects.push(rect);+ }+ rects+ })+ } else {+ None+ };+ (results, new_faces)+ }));+ let (results, found) = run.unwrap_or_else(|_| (fallback(rects), None));+ Done { tag, epoch, id, results, detect_wanted: detect, found }+}++type Job = (Arc<Embedder>, Prepared);++/// How many photos may be worked on at once, changed at any moment by the pace: a thread that has a job waits here until+/// fewer than that are running. A pace of zero holds every job where it is, with nothing decoded.+struct Limit {+ allowed: AtomicUsize,+ /// The pace the limit was last set for (a [`Pace`] as a number): the pool a photo's parallel work uses.+ pace: AtomicU8,+ running: Mutex<usize>,+ wake: Condvar,+}++impl Limit {+ fn new() -> Limit {+ Limit { allowed: AtomicUsize::new(0), pace: AtomicU8::new(Pace::Normal as u8), running: Mutex::new(0), wake: Condvar::new() }+ }++ fn set(&self, n: usize, pace: Pace) {+ self.pace.store(pace as u8, Ordering::Relaxed);+ self.allowed.store(n, Ordering::Relaxed);+ self.wake.notify_all();+ }++ fn pace(&self) -> Pace {+ Pace::from_u8(self.pace.load(Ordering::Relaxed))+ }++ /// Wait for a place; the limit is looked at again whenever it changes and at least every 100 ms.+ fn enter(&self) {+ let mut running = self.running.lock().unwrap_or_else(PoisonError::into_inner);+ while *running >= self.allowed.load(Ordering::Relaxed) {+ running = self.wake.wait_timeout(running, Duration::from_millis(100)).unwrap_or_else(PoisonError::into_inner).0;+ }+ *running += 1;+ }++ fn leave(&self) {+ let mut running = self.running.lock().unwrap_or_else(PoisonError::into_inner);+ *running = running.saturating_sub(1);+ drop(running);+ self.wake.notify_one();+ }+}++/// The threads a photo's parallel work (decoding, developing a picture) runs on at `pace`: one pool for each of three sizes+/// ([`pool_threads_for`]), made when first needed, whose idle threads cost nothing. rayon's global pool serves the+/// interactive work (the loupe, exports) and has no priorities, so scan work queued there would make a slider drag wait+/// behind it; here a light scan can use only two threads however large the machine is.+fn scan_pool(pace: Pace) -> Option<&'static rayon::ThreadPool> {+ static LIGHT: std::sync::OnceLock<Option<rayon::ThreadPool>> = std::sync::OnceLock::new();+ static NORMAL: std::sync::OnceLock<Option<rayon::ThreadPool>> = std::sync::OnceLock::new();+ static FULL: std::sync::OnceLock<Option<rayon::ThreadPool>> = std::sync::OnceLock::new();+ let (cell, pace) = match pace {+ Pace::Pause | Pace::Light => (&LIGHT, Pace::Light),+ Pace::Normal => (&NORMAL, Pace::Normal),+ Pace::Full => (&FULL, Pace::Full),+ };+ cell.get_or_init(|| {+ rayon::ThreadPoolBuilder::new()+ .num_threads(pool_threads_for(cores(), pace, cap_from_env()))+ .thread_name(move |i| format!("lightcraft-faces-{}-{i}", pace as u8))+ .build()+ .ok()+ })+ .as_ref()+}++/// Run one photo on the pool for the pace, so its parallel parts stay there.+///+/// It must NOT wait at the memory gate ([`crate::memory::work_gate`], which background decodes do): a job running on a+/// pool thread that waits inside a nested rayon call can pick up another job to run on top of itself, and if that one waits+/// for memory the first holds, neither can finish (seen as a few photos that never come back). The memory a scan may use is+/// instead limited by how many photos run at once ([`memory_workers`]); its decodes count as interactive ones at the gate,+/// which never wait.+fn process_in_background(prepared: Prepared, embedder: &Embedder, pace: Pace) -> Done {+ let work = || process(prepared, embedder);+ match scan_pool(pace) {+ Some(pool) => pool.install(work),+ None => work(),+ }+}++/// Threads that run [`process`] for the jobs they are given, each on one photo at a time.+pub(crate) struct Worker {+ jobs: Sender<Job>,+ done: Receiver<Done>,+ limit: Arc<Limit>,+}++impl Worker {+ pub fn start(threads: usize) -> Option<Worker> {+ let (jobs, job_rx) = channel::<Job>();+ let (done_tx, done) = channel::<Done>();+ let job_rx = Arc::new(Mutex::new(job_rx));+ let limit = Arc::new(Limit::new());+ let mut started = 0;+ for i in 0..threads.clamp(1, MAX_WORKERS) {+ let (rx, tx, limit) = (job_rx.clone(), done_tx.clone(), limit.clone());+ let spawned = std::thread::Builder::new().name(format!("lightcraft-faces-{i}")).spawn(move || {+ loop {+ // the lock is held only while waiting for a job, never while working on one; the threads end when+ // the session (the only sender) is dropped+ let next = rx.lock().unwrap_or_else(PoisonError::into_inner).recv();+ let Ok((embedder, prepared)) = next else { break };+ limit.enter();+ let done = process_in_background(prepared, &embedder, limit.pace());+ limit.leave();+ if tx.send(done).is_err() {+ break;+ }+ }+ });+ if spawned.is_ok() {+ started += 1;+ }+ }+ (started > 0).then_some(Worker { jobs, done, limit })+ }++ /// How many photos may be worked on at once from now on (what is running is not interrupted).+ pub fn allow(&self, n: usize, pace: Pace) {+ self.limit.set(n, pace);+ }++ pub fn submit(&self, embedder: Arc<Embedder>, prepared: Prepared) -> bool {+ self.jobs.send((embedder, prepared)).is_ok()+ }++ /// Everything finished since the last call.+ pub fn finished(&self) -> Vec<Done> {+ self.done.try_iter().collect()+ }+}++#[cfg(test)]+mod tests {+ use super::*;++ #[test]+ fn the_pace_sets_how_much_of_the_machine_the_scan_may_use() {+ use Pace::*;+ // a 32-thread desktop: nothing while the user works, one photo around them, half when idle, most when watched+ assert_eq!([Pause, Light, Normal, Full].map(|p| workers_for(32, p, None)), [0, 1, 16, 24]);+ assert_eq!([Pause, Light, Normal, Full].map(|p| workers_for(8, p, None)), [0, 1, 4, 6]);+ assert_eq!([Pause, Light, Normal, Full].map(|p| workers_for(4, p, None)), [0, 1, 2, 3]);+ // small machines still get one worker (except while paused)+ assert_eq!([Pause, Light, Normal, Full].map(|p| workers_for(1, p, None)), [0, 1, 1, 1]);+ assert_eq!(workers_for(0, Full, None), 1);+ assert_eq!(workers_for(10_000, Full, None), MAX_WORKERS);+ // a limit set by the user caps every pace, and is itself kept sane+ assert_eq!([Light, Normal, Full].map(|p| workers_for(32, p, Some(3))), [1, 3, 3]);+ assert_eq!((workers_for(32, Full, Some(0)), workers_for(32, Pause, Some(5))), (1, 0));+ assert_eq!(workers_for(32, Full, Some(usize::MAX)), 24);+ }++ #[test]+ fn the_limit_holds_threads_to_the_pace_and_can_change_at_any_moment() {+ let limit = Arc::new(Limit::new());+ let (now, peak) = (Arc::new(AtomicUsize::new(0)), Arc::new(AtomicUsize::new(0)));+ let started = Arc::new(AtomicUsize::new(0));+ let threads: Vec<_> = (0..6)+ .map(|_| {+ let (limit, now, peak, started) = (limit.clone(), now.clone(), peak.clone(), started.clone());+ std::thread::spawn(move || {+ limit.enter();+ started.fetch_add(1, Ordering::SeqCst);+ let n = now.fetch_add(1, Ordering::SeqCst) + 1;+ peak.fetch_max(n, Ordering::SeqCst);+ std::thread::sleep(Duration::from_millis(40));+ now.fetch_sub(1, Ordering::SeqCst);+ limit.leave();+ })+ })+ .collect();+ // paused: nothing starts+ std::thread::sleep(Duration::from_millis(150));+ assert_eq!(started.load(Ordering::SeqCst), 0);+ // two at a time, never more+ limit.set(2, Pace::Normal);+ std::thread::sleep(Duration::from_millis(100));+ assert!(started.load(Ordering::SeqCst) >= 2);+ // raised, the rest go; every thread finishes+ limit.set(6, Pace::Full);+ for t in threads {+ t.join().unwrap();+ }+ assert_eq!(started.load(Ordering::SeqCst), 6);+ assert!(peak.load(Ordering::SeqCst) >= 2, "it did run in parallel");+ assert_eq!(*limit.running.lock().unwrap(), 0, "every place was given back");+ }++ #[test]+ fn memory_limits_the_workers_and_a_users_limit_replaces_it() {+ // the default budget (a quarter of the RAM, at most 1.5 GiB) holds a few photos in progress, not dozens+ assert!((1..=8).contains(&memory_workers()), "{}", memory_workers());+ assert!(target_workers(Pace::Full) >= 1);+ assert_eq!(target_workers(Pace::Pause), 0);+ }++ #[test]+ fn pace_names_are_forgiving() {+ assert_eq!(["pause", "light", "normal", "full"].map(|n| Pace::parse(Some(n))), [Pace::Pause, Pace::Light, Pace::Normal, Pace::Full]);+ assert_eq!(+ (Pace::parse(None), Pace::parse(Some("")), Pace::parse(Some("FULL")), Pace::parse(Some("turbo"))),+ (Pace::Normal, Pace::Normal, Pace::Normal, Pace::Normal)+ );+ }++ #[test]+ fn the_scans_parallel_work_gets_a_pool_the_size_of_the_pace() {+ use Pace::*;+ assert_eq!([Pause, Light, Normal, Full].map(|p| pool_threads_for(32, p, None)), [2, 2, 16, 25]);+ assert_eq!([Light, Normal, Full].map(|p| pool_threads_for(8, p, None)), [2, 4, 6]);+ assert_eq!([Light, Normal, Full].map(|p| pool_threads_for(2, p, None)), [2, 2, 2]);+ assert_eq!(pool_threads_for(10_000, Full, None), 32);+ assert_eq!([Light, Normal, Full].map(|p| pool_threads_for(32, p, Some(3))), [2, 3, 3]);+ // the pools exist and are the sizes asked for+ for pace in [Light, Normal, Full] {+ let pool = scan_pool(pace).expect("a pool can be built");+ assert_eq!(pool.install(rayon::current_num_threads), pool_threads_for(cores(), pace, cap_from_env()));+ }+ // a paused scan has the light pool; the pace is remembered as a number and read back+ assert_eq!(scan_pool(Pause).map(|p| p as *const _), scan_pool(Light).map(|p| p as *const _));+ assert_eq!([0, 1, 2, 3, 9].map(Pace::from_u8), [Pause, Light, Normal, Full, Normal]);+ }++ #[test]+ fn detections_become_boxes_only_when_they_make_sense() {+ let f = |x0, y0, x1, y1| Face { x0, y0, x1, y1, score: 0.9, landmarks: [(0.0, 0.0); 5] };+ assert!(face_rect(&f(0.2, 0.2, 0.4, 0.5)).is_some());+ // a box that spills over the edge is brought back inside+ let r = face_rect(&f(-0.1, 0.1, 0.3, 1.2)).unwrap();+ assert_eq!((r.x0, r.y1), (0.0, 1.0));+ // slivers, inverted boxes and non-numbers are dropped, never turned into regions+ for bad in [f(0.2, 0.2, 0.2, 0.5), f(0.4, 0.2, 0.2, 0.5), f(f32::NAN, 0.2, 0.4, 0.5), f(0.2, 0.2, f32::INFINITY, 0.5), f(1.5, 0.2, 2.0, 0.5)]+ {+ assert!(face_rect(&bad).is_none(), "{bad:?}");+ }+ }+}diff --git a/crates/engine/src/files.rs b/crates/engine/src/files.rsindex c620be928..8b7d355ce 100644--- a/crates/engine/src/files.rs+++ b/crates/engine/src/files.rs@@ -404,8 +404,16 @@ fn fit_box(max_edge: usize) -> lightcraft_codecs::DecodeOptions {}/// Filesystem-backed embedded-preview hook (native).+///+/// Reads the whole file to find the preview, so it holds the memory gate for twice the file's size (the file and the+/// decoded preview): background work (the face scan) waits for room, interactive work is counted and never waits.pub fn fs_preview_loader() -> PreviewLoader {- Arc::new(|path: &str, max_edge: usize| embedded_preview_srgb(&std::fs::read(path).ok()?, max_edge))+ Arc::new(|path: &str, max_edge: usize| {+ let len = std::fs::metadata(path).map(|m| usize::try_from(m.len()).unwrap_or(usize::MAX)).unwrap_or(0);+ let gate = crate::memory::work_gate();+ let _permit = if crate::memory::is_background() { gate.acquire(len.saturating_mul(2)) } else { gate.acquire_urgent(len.saturating_mul(2)) };+ embedded_preview_srgb(&std::fs::read(path).ok()?, max_edge)+ })}/// Filesystem-backed hooks (native). On the web the host installs bytes-based hooks instead.@@ -417,7 +425,7 @@ pub fn fs_preview_loader() -> PreviewLoader {/// yields to them.pub fn fs_hooks() -> (FileLoader, FileProbe) {let loader: FileLoader = Arc::new(|path: &str, max_edge: usize| {- let len = std::fs::metadata(path).map(|m| m.len() as usize).unwrap_or(0);+ let len = std::fs::metadata(path).map(|m| usize::try_from(m.len()).unwrap_or(usize::MAX)).unwrap_or(0);let weight = len * if max_edge <= crate::media::SourceLevel::Thumb.max_edge() { 3 } else { 6 };let gate = crate::memory::work_gate();let _permit = if crate::memory::is_background() { gate.acquire(weight) } else { gate.acquire_urgent(weight) };@@ -428,7 +436,7 @@ pub fn fs_hooks() -> (FileLoader, FileProbe) {r});let probe: FileProbe = Arc::new(|path: &str| {- let len = std::fs::metadata(path).map(|m| m.len() as usize).unwrap_or(0);+ let len = std::fs::metadata(path).map(|m| usize::try_from(m.len()).unwrap_or(usize::MAX)).unwrap_or(0);let _permit = crate::memory::work_gate().acquire(len);let bytes = std::fs::read(path).map_err(|e| format!("{path}: {e}"))?;probe_bytes(path, &bytes)diff --git a/crates/engine/src/lib.rs b/crates/engine/src/lib.rsindex 93e41e132..c3fcdd337 100644--- a/crates/engine/src/lib.rs+++ b/crates/engine/src/lib.rs@@ -21,6 +21,10 @@ pub mod demo;pub mod devices;pub mod export;pub mod face_download;+#[cfg(not(target_arch = "wasm32"))]+mod faces_index;+#[cfg(not(target_arch = "wasm32"))]+mod faces_worker;pub mod files;pub mod fonts;pub mod guard;@@ -163,6 +167,11 @@ pub struct Session {pub face_models_dir: Option<std::path::PathBuf>,/// Face model downloads started this session (the staged files wait in `<face_models_dir>/.downloads`).pub face_downloads: face_download::Downloads,+ /// The user's own list of models to download (`catalog.json` in the models folder), as last read.+ pub(crate) face_catalog: lightcraft_faces::catalog::Catalog,+ /// The loaded recognition model and the face embeddings made with it.+ #[cfg(not(target_arch = "wasm32"))]+ pub(crate) faces: faces_index::FacesState,/// Copied develop settings (partial JSON) for Paste.pub clipboard: Option<Value>,/// The folder on disk the [`LibrarySource::Folder`] view browses.@@ -271,6 +280,9 @@ impl Session {skip_auto_write: false,face_models_dir: None,face_downloads: Default::default(),+ face_catalog: Default::default(),+ #[cfg(not(target_arch = "wasm32"))]+ faces: Default::default(),clipboard: None,meta_clipboard: None,browse: None,@@ -411,6 +423,27 @@ impl Session {Ok(())}+ /// The box to cut a face's picture from: the detector's, when the scan has looked at the face (every face is then shown+ /// equally close, however loosely or tightly its own region was drawn), else the region's own box.+ pub fn face_view(&self, id: PhotoId, rect: lightcraft_geom::Rect) -> lightcraft_geom::Rect {+ #[cfg(not(target_arch = "wasm32"))]+ if let Some(v) = self.faces.index.view(id.0, &rect) {+ return v;+ }+ let _ = id;+ rect+ }++ /// Apply an op that is LightCraft's own bookkeeping rather than something the user did (faces found by the background+ /// scan): journaled like any op, but not an undo step, and it leaves the redo stack alone.+ #[cfg(not(target_arch = "wasm32"))]+ pub(crate) fn apply_system(&mut self, op: Op) -> Result<()> {+ let fwd = op.clone();+ self.catalog.apply(op)?;+ self.pending_log.push(fwd);+ Ok(())+ }+/// Leaving photo `id` after editing it: keep its settings as an automatic version (when they/// differ from its latest version; at most [`AUTO_VERSIONS`] auto versions, oldest dropped)./// Saved with the library but not an undo step.@@ -795,6 +828,9 @@ mod tests_export;#[cfg(test)]mod tests_face_models;#[cfg(test)]+#[cfg(not(target_arch = "wasm32"))]+mod tests_face_recognize;+#[cfg(test)]mod tests_folders;#[cfg(test)]mod tests_forget_local;diff --git a/crates/engine/src/library.rs b/crates/engine/src/library.rsindex c1b91c4cd..d01c1b726 100644--- a/crates/engine/src/library.rs+++ b/crates/engine/src/library.rs@@ -297,6 +297,9 @@ impl Session {{log::error!("library: {e}");}+ // what was learned about the old library's faces is saved, and none of it carries over (photo ids are per library)+ #[cfg(not(target_arch = "wasm32"))]+ self.faces.library_changed();let (mut journal, mut catalog, report) = Journal::open(catalog)?;// A loaded catalog counts revisions from 0, like every other one. Caches (sidebar counts,// keyword tree, the grid's list…) are keyed on the revision, so give each library loadeddiff --git a/crates/engine/src/media.rs b/crates/engine/src/media.rsindex 1ed89e473..29b3bd11d 100644--- a/crates/engine/src/media.rs+++ b/crates/engine/src/media.rs@@ -979,6 +979,17 @@ impl crate::Session {}}+ /// The camera's embedded preview of a raw file, for the face scan. Faces are read from the unedited picture, so the+ /// photo's edits do not matter, and the camera's own JPEG (oriented, display-ready) costs a fraction of decoding the+ /// raw. `None` for anything else, and where the host installed no loader.+ #[cfg(not(target_arch = "wasm32"))]+ pub(crate) fn scan_preview_of(&self, p: &Photo) -> Option<(String, PreviewLoader)> {+ match (&p.source, &self.media.preview_loader) {+ (Source::File { path }, Some(l)) if p.kind == MediaKind::Raw => Some((path.clone(), l.clone())),+ _ => None,+ }+ }+/// Something to show in the loupe right away for `id` (see [`QuickJob`]): its cached view/// render, else the embedded preview of an unedited raw, else a cached or fresh thumbnail.pub fn quick_view_job(&mut self, id: PhotoId, max_edge: usize, apply_crop: bool) -> Option<QuickJob> {diff --git a/crates/engine/src/tests_face_models.rs b/crates/engine/src/tests_face_models.rsindex 5fcf9e575..c71f7afeb 100644--- a/crates/engine/src/tests_face_models.rs+++ b/crates/engine/src/tests_face_models.rs@@ -22,7 +22,7 @@ fn session(dir: &std::path::Path) -> Session {fn model_file(dir: &std::path::Path, name: &str, dim: u64) -> String {let p = dir.join(name);- std::fs::write(&p, lightcraft_faces::synthetic::embedder_model(dim)).unwrap();+ std::fs::write(&p, lightcraft_faces::synthetic::tiny_embedder_model(dim)).unwrap();p.to_string_lossy().into_owned()}@@ -36,11 +36,15 @@ fn the_list_knows_the_known_models_and_is_honest_about_the_runtime() {let mut s = session(&d);let l = s.execute("faces.models.list", &json!({})).unwrap();assert_eq!(l["enabled"], false);- assert_eq!(l["runtime"], false);+ assert_eq!(l["runtime"], cfg!(not(target_arch = "wasm32")));assert_eq!(find(&l, "yunet-2023mar")["installed"], false, "nothing ships with LightCraft");assert_eq!(find(&l, "auraface-v1")["installed"], false);assert_eq!(find(&l, "auraface-v1")["licence"]["commercial"], "yes");assert_eq!(find(&l, "sface-2021dec")["licence"]["commercial"], "unknown");+ // speed is a ratio to a ResNet-100 model, in words, for models that are not installed yet too; never milliseconds+ assert_eq!(find(&l, "sface-2021dec")["speedText"], "6.9× faster than a ResNet-100 model");+ assert_eq!(find(&l, "auraface-v1")["speedText"], "Same speed as a ResNet-100 model");+ assert_eq!(find(&l, "yunet-2023mar")["speedText"], Value::Null, "nobody timed the detector against it");// a build with no folder lists too, and cannot installlet mut web = Session::new();assert!(web.execute("faces.models.list", &json!({})).is_ok());@@ -186,6 +190,16 @@ fn demo_with_yunet() -> Option<Session> {Some(s)}+#[test]+fn installing_the_detector_does_not_choose_it_as_the_recogniser_or_switch_recognition_on() {+ let Some(mut s) = demo_with_yunet() else { return };+ let l = s.execute("faces.models.list", &json!({})).unwrap();+ assert_eq!(find(&l, "yunet-2023mar")["installed"], true);+ assert_eq!(l["enabled"], false, "{l}");+ assert!(l["embedder"].is_null(), "{l}");+ assert!(s.execute("faces.models.test", &json!({"id": "yunet-2023mar"})).is_err(), "a detector has no recognition self-test");+}+#[test]fn detect_without_the_model_says_where_to_get_it() {let d = temp("nodetector");@@ -225,7 +239,7 @@ fn a_download_that_has_arrived_is_installed_by_itself_and_its_staged_file_goes()let staging = d.join("models").join(".downloads");std::fs::create_dir_all(&staging).unwrap();let staged = staging.join("model.onnx");- std::fs::write(&staged, lightcraft_faces::synthetic::embedder_model(512)).unwrap();+ std::fs::write(&staged, lightcraft_faces::synthetic::tiny_embedder_model(512)).unwrap();let sha = lightcraft_faces::hash::sha256_file(&staged).unwrap();s.face_downloads.arrived("my-model", staged.clone(), &sha);@@ -358,3 +372,215 @@ fn real_downloads_of_the_recognisers_start_and_can_be_cancelled() {println!("left in staging: {left:?}");let _ = std::fs::remove_dir_all(&d);}++/// With the recognition runtime: installing runs the model's self-test, a model that does not work is refused+/// and leaves nothing behind, and `faces.models.test` runs it again.+#[cfg(not(target_arch = "wasm32"))]+#[test]+fn installing_runs_the_self_test_and_refuses_models_that_do_not_work() {+ let d = temp("selftest");+ let mut s = session(&d);+ // a working model passes; its record keeps the result+ let f = model_file(&d, "Works.onnx", 64);+ let r = s.execute("faces.models.install", &json!({"path": f, "acknowledged": true})).unwrap();+ let id = r["installed"]["id"].as_str().unwrap().to_string();+ assert_eq!(r["installed"]["accepted"]["selfTest"]["ok"], true, "{r}");+ assert_eq!(r["installed"]["accepted"]["selfTest"]["dimension"], 64);+ let t = s.execute("faces.models.test", &json!({"id": id})).unwrap();+ assert_eq!(t["ok"], true, "{t}");+ assert!(t["result"]["embedMs"].as_f64().unwrap() >= 0.0);+ assert!(s.execute("faces.models.test", &json!({"id": "nope"})).is_err());+ assert!(s.execute("faces.models.test", &json!({})).is_err());+ // a graph with no layers passes the shape check but cannot run: refused, and nothing is left on disk+ let broken = d.join("Broken.onnx");+ std::fs::write(&broken, lightcraft_faces::synthetic::embedder_model(64)).unwrap();+ let e = s.execute("faces.models.install", &json!({"path": broken.to_string_lossy(), "acknowledged": true})).unwrap_err().to_string();+ assert!(e.contains("could not be loaded") || e.contains("self-test"), "{e}");+ let leftovers: Vec<_> = std::fs::read_dir(d.join("models"))+ .unwrap()+ .flatten()+ .map(|e| e.file_name().to_string_lossy().into_owned())+ .filter(|n| n != "settings.json")+ .collect();+ assert!(leftovers.len() == 1 && leftovers[0].starts_with("custom-works-"), "only the working model remains: {leftovers:?}");+ let _ = std::fs::remove_dir_all(&d);+}++#[cfg(not(target_arch = "wasm32"))]+#[test]+fn failed_replacement_keeps_the_installed_model_and_records() {+ let d = temp("replacement");+ let models = d.join("models");+ std::fs::create_dir_all(&models).unwrap();+ let good = model_file(&d, "Good.onnx", 64);+ let write_catalog = |path: &std::path::Path| {+ let catalog = json!({"models": [{+ "id": "replacement", "name": "Replacement", "version": "1", "role": "embedder",+ "url": "https://models.example.org/weights.onnx",+ "sha256": lightcraft_faces::hash::sha256_file(path).unwrap(),+ "sizeBytes": std::fs::metadata(path).unwrap().len(),+ "licence": {"name": "MIT", "commercial": "yes"},+ "output": {"kind": "embedding", "dim": 64}+ }]});+ std::fs::write(models.join("catalog.json"), catalog.to_string()).unwrap();+ };+ write_catalog(std::path::Path::new(&good));+ let mut s = session(&d);+ s.execute("faces.models.install", &json!({"path": good, "acknowledged": true})).unwrap();+ let home = models.join("replacement");+ let files = ["model.onnx", "face-model.json", "installed.json"];+ let before: Vec<_> = files.iter().map(|name| std::fs::read(home.join(name)).unwrap()).collect();+ let settings = std::fs::read(models.join("settings.json")).unwrap();+ let bad = d.join("Bad.onnx");+ std::fs::write(&bad, lightcraft_faces::synthetic::embedder_model(64)).unwrap();+ write_catalog(&bad);+ let mut s = session(&d);+ assert!(s.execute("faces.models.install", &json!({"path": bad, "acknowledged": true})).is_err());+ for (name, bytes) in files.iter().zip(before) {+ assert_eq!(std::fs::read(home.join(name)).unwrap(), bytes, "{name} was replaced");+ }+ assert_eq!(std::fs::read(models.join("settings.json")).unwrap(), settings);+ assert!(!home.join("model.onnx.part").exists());+ let _ = std::fs::remove_dir_all(d);+}++fn settings_of(d: &std::path::Path) -> Value {+ serde_json::from_slice(&std::fs::read(d.join("models").join("settings.json")).unwrap()).unwrap()+}++#[test]+fn installing_a_model_chooses_it_and_switches_recognition_on() {+ let d = temp("activate");+ let mut s = session(&d);+ let a = s.execute("faces.models.install", &json!({"path": model_file(&d, "First.onnx", 64), "acknowledged": true})).unwrap();+ let a_id = a["installed"]["id"].as_str().unwrap().to_string();+ assert_eq!(a["active"], json!({"embedder": a_id, "enabled": true}));+ assert_eq!(a["installed"]["selected"], true);+ assert_eq!(settings_of(&d), json!({"enabled": true, "embedder": a_id}));+ // a second model takes over (the newest is the one the user just asked for); the first stays installed+ let b = s.execute("faces.models.install", &json!({"path": model_file(&d, "Second.onnx", 96), "acknowledged": true})).unwrap();+ let b_id = b["installed"]["id"].as_str().unwrap().to_string();+ assert_ne!(a_id, b_id);+ assert_eq!(b["active"]["embedder"], json!(b_id));+ let l = s.execute("faces.models.list", &json!({})).unwrap();+ assert_eq!((find(&l, &a_id)["installed"].clone(), find(&l, &a_id)["selected"].clone()), (json!(true), json!(false)));+ assert_eq!(find(&l, &b_id)["selected"], true);+ // switching back is a choice, and a model added with `activate: false` leaves the choice alone+ s.execute("faces.models.select", &json!({"id": a_id})).unwrap();+ let c = s.execute("faces.models.install", &json!({"path": model_file(&d, "Third.onnx", 128), "acknowledged": true, "activate": false})).unwrap();+ let c_id = c["installed"]["id"].as_str().unwrap().to_string();+ assert_eq!(c["active"]["embedder"], json!(a_id));+ assert_eq!(c["installed"]["selected"], false);+ // a switched-off recognition is switched back on by the next install, not by a refused one+ s.execute("faces.enable", &json!({"enabled": false})).unwrap();+ assert!(s.execute("faces.models.install", &json!({"path": model_file(&d, "Nope.onnx", 64)})).is_err());+ assert_eq!(settings_of(&d)["enabled"], false);+ s.execute("faces.models.install", &json!({"path": model_file(&d, "Fourth.onnx", 64), "acknowledged": true})).unwrap();+ assert_eq!(settings_of(&d)["enabled"], true);+ // removing the model in use hands over to another installed one instead of leaving nothing chosen+ let in_use = settings_of(&d)["embedder"].as_str().unwrap().to_string();+ let r = s.execute("faces.models.remove", &json!({"id": in_use})).unwrap();+ let next = r["embedder"].as_str().unwrap().to_string();+ assert_ne!(next, in_use);+ assert!([&a_id, &b_id, &c_id].contains(&&next) || next.starts_with("custom-"), "{next}");+ assert_eq!(settings_of(&d)["embedder"], json!(next));+ // removing a model that is not in use changes nothing+ let other = [&a_id, &b_id, &c_id].into_iter().find(|i| **i != next && **i != in_use).unwrap().clone();+ s.execute("faces.models.remove", &json!({"id": other})).unwrap();+ assert_eq!(settings_of(&d)["embedder"], json!(next));+ let _ = std::fs::remove_dir_all(&d);+}++#[test]+fn a_download_that_arrives_is_installed_and_switched_on_by_itself() {+ use crate::face_download::{STAGING, State};+ let d = temp("finish");+ let mut s = session(&d);+ let staging = d.join("models").join(STAGING);+ std::fs::create_dir_all(&staging).unwrap();+ // what the download thread leaves when it succeeds: a verified file in the staging folder and a finished job+ let staged = model_file(&staging, "arrived.onnx", 64);+ let sha = lightcraft_faces::hash::sha256_file(std::path::Path::new(&staged)).unwrap();+ s.face_downloads.set_outcome("fake-model", State::Done { path: staged.clone().into(), sha256: sha });+ let r = s.execute("faces.models.downloads", &json!({})).unwrap();+ assert_eq!(r["downloads"][0]["id"], "fake-model");+ assert_eq!(r["downloads"][0]["state"], "installed", "{r}");+ let st = settings_of(&d);+ assert_eq!(st["enabled"], true);+ assert!(st["embedder"].as_str().unwrap().starts_with("custom-arrived-"), "{st}");+ assert!(!std::path::Path::new(&staged).exists(), "the staged file was moved into place");+ // the record stays until it is cleared, and is cleared once+ assert_eq!(s.execute("faces.models.downloads", &json!({})).unwrap()["downloads"][0]["state"], "installed");+ assert_eq!(s.execute("faces.models.downloadCancel", &json!({"id": "fake-model"})).unwrap()["discarded"], true);+ assert_eq!(s.execute("faces.models.downloads", &json!({})).unwrap()["downloads"], json!([]));++ // a download that turns out not to be a usable model is thrown away, and the reason is shown+ let junk = staging.join("junk.onnx");+ std::fs::write(&junk, b"definitely not a model").unwrap();+ let sha = lightcraft_faces::hash::sha256_file(&junk).unwrap();+ s.face_downloads.set_outcome("fake-junk", State::Done { path: junk.clone(), sha256: sha });+ let r = s.execute("faces.models.downloads", &json!({})).unwrap();+ assert_eq!(r["downloads"][0]["state"], "failed", "{r}");+ assert!(r["downloads"][0]["error"].as_str().unwrap().contains("cannot be used"), "{r}");+ assert!(!junk.exists());+ assert_eq!(settings_of(&d)["embedder"], st["embedder"], "a failed download leaves the choice alone");+ let _ = std::fs::remove_dir_all(&d);+}++#[test]+fn the_users_own_catalog_adds_models_with_the_same_download_and_install() {+ use crate::face_download::State;+ let d = temp("catalog");+ let mut s = session(&d);+ let models = d.join("models");+ std::fs::create_dir_all(&models).unwrap();+ // a model file, and a catalog that lists it by its hash (plus an entry that cannot be used)+ let file = model_file(&d, "weights.onnx", 64);+ let (sha, size) = (lightcraft_faces::hash::sha256_file(std::path::Path::new(&file)).unwrap(), std::fs::metadata(&file).unwrap().len());+ let catalog = json!({"models": [+ {"id": "my-research-model", "name": "My research model", "version": "1", "role": "embedder", "url": "https://models.example.org/w/weights.onnx",+ "sha256": sha, "sizeBytes": size, "licence": {"name": "Research only", "commercial": "no", "notice": "Not for commercial use."},+ "provenance": "Some dataset", "output": {"kind": "embedding", "dim": 64}, "thresholds": {"matchCosine": 0.42}},+ {"id": "broken", "name": "No hash", "version": "1", "role": "embedder", "url": "https://models.example.org/x.onnx", "output": {"kind": "embedding", "dim": 64}},+ ]});+ std::fs::write(models.join("catalog.json"), catalog.to_string()).unwrap();+ // it is listed like a built-in model, with its own terms, a download address, and the problem with the other entry+ let l = s.execute("faces.models.list", &json!({})).unwrap();+ let m = find(&l, "my-research-model");+ assert_eq!(+ (m["fromCatalog"].clone(), m["downloadHost"].clone(), m["licence"]["commercial"].clone()),+ (json!(true), json!("models.example.org"), json!("no"))+ );+ assert_eq!(m["installed"], false);+ assert_eq!(find(&l, "broken"), &Value::Null);+ assert_eq!(l["catalog"]["models"], 1);+ assert!(l["catalog"]["errors"][0].as_str().unwrap().contains("broken"), "{}", l["catalog"]);+ // nothing is fetched without the terms being accepted+ assert!(s.execute("faces.models.download", &json!({"id": "my-research-model"})).is_err());+ // a file that matches is recognised as that model (not as a draft), so its terms and settings are used+ let seen = s.execute("faces.models.inspect", &json!({"path": file})).unwrap();+ assert_eq!((seen["kind"].clone(), seen["model"]["id"].clone()), (json!("known"), json!("my-research-model")));+ // a finished download of it installs under the catalog's id, with the catalog's licence record+ let staging = models.join(crate::face_download::STAGING);+ std::fs::create_dir_all(&staging).unwrap();+ let staged = staging.join("weights.onnx");+ std::fs::copy(&file, &staged).unwrap();+ s.face_downloads.set_outcome("my-research-model", State::Done { path: staged, sha256: sha });+ let r = s.execute("faces.models.downloads", &json!({})).unwrap();+ assert_eq!(r["downloads"][0]["state"], "installed", "{r}");+ let st = settings_of(&d);+ assert_eq!((st["embedder"].clone(), st["enabled"].clone()), (json!("my-research-model"), json!(true)));+ let l = s.execute("faces.models.list", &json!({})).unwrap();+ assert_eq!(+ (find(&l, "my-research-model")["installed"].clone(), find(&l, "my-research-model")["accepted"]["commercial"].clone()),+ (json!(true), json!("no"))+ );+ // a broken or hostile catalog never stops the list+ for bytes in [&b"{"[..], b"[]", &vec![b'x'; 400_000]] {+ std::fs::write(models.join("catalog.json"), bytes).unwrap();+ let l = s.execute("faces.models.list", &json!({})).unwrap();+ assert!(l["catalog"]["errors"].as_array().is_some_and(|e| !e.is_empty()), "{}", l["catalog"]);+ assert!(find(&l, "sface-2021dec")["id"] == "sface-2021dec");+ }+ let _ = std::fs::remove_dir_all(&d);+}diff --git a/crates/engine/src/tests_face_recognize.rs b/crates/engine/src/tests_face_recognize.rsnew file mode 100644index 000000000..2fd53c679--- /dev/null+++ b/crates/engine/src/tests_face_recognize.rs@@ -0,0 +1,489 @@+//! Recognition: indexing faces, suggesting names, naming, with a tiny working network standing in for a real+//! model (its embedding depends on the picture's colours, which is enough to test the machinery; how well real+//! models recognise people is measured separately, on real photos).++use std::path::PathBuf;++use lightcraft_catalog::{Op, PhotoId};+use lightcraft_geom::Rect;+use lightcraft_meta::{Region, RegionKind};+use serde_json::{Value, json};++use crate::Session;++pub(crate) fn temp(name: &str) -> PathBuf {+ let d = std::env::temp_dir().join(format!("lc-facerec-{name}-{}", std::process::id()));+ let _ = std::fs::remove_dir_all(&d);+ std::fs::create_dir_all(&d).unwrap();+ d+}++/// A demo session with a tiny model installed and chosen.+pub(crate) fn setup(d: &std::path::Path, dim: u64) -> (Session, String) {+ let mut s = Session::with_demo();+ s.face_models_dir = Some(d.join("models"));+ let model = d.join(format!("tiny{dim}.onnx"));+ std::fs::write(&model, lightcraft_faces::synthetic::tiny_embedder_model(dim)).unwrap();+ let r = s.execute("faces.models.install", &json!({"path": model.to_string_lossy(), "acknowledged": true})).unwrap();+ let id = r["installed"]["id"].as_str().unwrap().to_string();+ s.execute("faces.models.select", &json!({"id": id})).unwrap();+ (s, id)+}++/// Installs the real YuNet from `LC_YUNET_MODEL=<the .onnx file>` (what Settings > Faces downloads; nothing is+/// committed) and says whether it did: the parts of a test that need to find faces run only then.+pub(crate) fn install_yunet(s: &mut Session) -> bool {+ let Some(file) = std::env::var_os("LC_YUNET_MODEL") else {+ eprintln!("LC_YUNET_MODEL is not set: skipping the face-finding part");+ return false;+ };+ let r = s.execute("faces.models.install", &json!({"path": file.to_string_lossy(), "acknowledged": true}));+ assert!(r.is_ok(), "{r:?}");+ true+}++pub(crate) fn region(x: f64, name: Option<&str>) -> Region {+ Region { rect: Rect { x0: x, y0: 0.2, x1: x + 0.25, y1: 0.6 }, kind: RegionKind::Face, name: name.map(str::to_string), description: None }+}++pub(crate) fn set_regions(s: &mut Session, id: PhotoId, regions: Vec<Region>) {+ let mut meta = s.catalog.photo(id).unwrap().meta.clone();+ meta.regions = regions;+ s.commit("setup", Op::SetMeta { id, meta: Box::new(meta) }).unwrap();+}++pub(crate) fn two_photos(s: &Session) -> (PhotoId, PhotoId) {+ let mut ids = s.catalog.photos().map(|p| p.id).collect::<Vec<_>>();+ ids.sort();+ (ids[0], ids[1])+}++#[test]+fn indexing_embeds_each_face_once_and_reports_progress() {+ let d = temp("index");+ let (mut s, _) = setup(&d, 64);+ let (a, b) = two_photos(&s);+ set_regions(&mut s, a, vec![region(0.1, Some("Ann")), region(0.5, None)]);+ set_regions(&mut s, b, vec![region(0.3, None)]);+ // only a report+ let r = s.execute("faces.index", &json!({"budgetMs": 0})).unwrap();+ assert_eq!((r["embedded"].clone(), r["photosDone"].clone(), r["pendingPhotos"].clone()), (json!(0), json!(0), json!(2)), "{r}");+ // a generous budget does it all, the named photo first+ let r = s.execute("faces.index", &json!({"budgetMs": 60_000})).unwrap();+ assert_eq!((r["embedded"].clone(), r["pendingPhotos"].clone(), r["indexedFaces"].clone()), (json!(3), json!(0), json!(3)), "{r}");+ // nothing is embedded twice+ let r = s.execute("faces.index", &json!({"budgetMs": 60_000})).unwrap();+ assert_eq!((r["embedded"].clone(), r["photosDone"].clone()), (json!(0), json!(0)));+ // a small budget still makes progress (at least one photo per call), and `ids` go first+ let c = s.catalog.photos().map(|p| p.id).find(|id| *id != a && *id != b).unwrap();+ set_regions(&mut s, c, vec![region(0.2, None)]);+ set_regions(&mut s, b, vec![region(0.3, None), region(0.6, None)]);+ let r = s.execute("faces.index", &json!({"budgetMs": 1, "ids": [c.0]})).unwrap();+ assert_eq!(r["photosDone"], 1, "{r}");+ assert_eq!(r["pendingPhotos"], 1, "{r}");+ assert!(s.faces.index.contains(c.0, ®ion(0.2, None).rect), "the photo asked for first was done first");+ let _ = std::fs::remove_dir_all(&d);+}++#[test]+fn suggestions_come_from_the_named_faces_and_are_never_applied() {+ let d = temp("suggest");+ let (mut s, _) = setup(&d, 64);+ let (a, b) = two_photos(&s);+ set_regions(&mut s, a, vec![region(0.1, Some("Ann"))]);+ set_regions(&mut s, b, vec![region(0.1, None), region(0.5, Some("Pet?")), region(0.7, None)]);+ // thresholds that accept anything: Ann is the only person with a named face except "Pet?"+ let r = s.execute("faces.suggest", &json!({"ids": [b.0], "threshold": -1.0, "margin": -2.0, "budgetMs": 60_000})).unwrap();+ assert_eq!(r["galleryFaces"], 2, "{r}");+ let faces = r["photos"][0]["faces"].as_array().unwrap();+ assert_eq!(faces.len(), 2, "only the unnamed faces get suggestions: {r}");+ assert!(faces.iter().all(|f| f["suggestion"]["name"].is_string()), "{r}");+ assert_eq!(faces[0]["index"], 0);+ assert_eq!(faces[1]["index"], 2);+ assert!(faces[0]["candidates"].as_array().unwrap().len() == 2);+ // an impossible threshold suggests nobody but still lists the candidates+ let r = s.execute("faces.suggest", &json!({"ids": [b.0], "threshold": 2.0, "budgetMs": 60_000})).unwrap();+ assert!(r["photos"][0]["faces"][0]["suggestion"].is_null());+ assert!(!r["photos"][0]["faces"][0]["candidates"].as_array().unwrap().is_empty());+ // nothing was named by suggesting+ assert!(s.catalog.photo(b).unwrap().meta.regions.iter().filter(|r| r.name.is_none()).count() == 2);+ // pets and other kinds are not people+ let mut meta = s.catalog.photo(b).unwrap().meta.clone();+ meta.regions[1].kind = RegionKind::Pet;+ s.commit("pet", Op::SetMeta { id: b, meta: Box::new(meta) }).unwrap();+ let r = s.execute("faces.suggest", &json!({"ids": [b.0], "threshold": -1.0, "margin": -2.0, "budgetMs": 60_000})).unwrap();+ assert_eq!(r["galleryFaces"], 1, "{r}");+ // odd parameters are tolerated+ for p in [json!({"threshold": "x"}), json!({"margin": null}), json!({"budgetMs": 0}), json!({"ids": []}), json!({"ids": [999999]})] {+ assert!(s.execute("faces.suggest", &p).is_ok(), "{p}");+ }+ let _ = std::fs::remove_dir_all(&d);+}++#[test]+fn naming_a_face_is_undoable_and_makes_it_yours() {+ let d = temp("name");+ let (mut s, _) = setup(&d, 64);+ let (a, _) = two_photos(&s);+ let mut detected = region(0.1, None);+ detected.description = Some("Detected by YuNet 2023mar".into());+ set_regions(&mut s, a, vec![detected, region(0.5, None)]);+ let undo = s.undo.len();+ let r = s.execute("faces.setName", &json!({"id": a.0, "index": 0, "name": " Ann "})).unwrap();+ assert_eq!(r["name"], "Ann");+ let regions = s.catalog.photo(a).unwrap().meta.regions.clone();+ assert_eq!((regions[0].name.as_deref(), regions[0].description.as_deref()), (Some("Ann"), None), "a named detection loses its detected marker");+ assert_eq!(s.undo.len(), undo + 1);+ // a new detection run now leaves it alone+ if install_yunet(&mut s) {+ s.execute("faces.detect", &json!({"id": a.0})).unwrap();+ assert_eq!(s.catalog.photo(a).unwrap().meta.regions.iter().filter(|r| r.name.as_deref() == Some("Ann")).count(), 1);+ }+ // clearing, undo+ s.execute("faces.setName", &json!({"id": a.0, "index": 0, "name": null})).unwrap();+ assert_eq!(s.catalog.photo(a).unwrap().meta.regions[0].name, None);+ s.execute("edit.undo", &json!({})).unwrap();+ assert_eq!(s.catalog.photo(a).unwrap().meta.regions[0].name.as_deref(), Some("Ann"));+ // bad input+ for p in [+ json!({"id": a.0}),+ json!({"id": a.0, "index": 9, "name": "x"}),+ json!({"id": a.0, "index": 0, "name": 5}),+ json!({"id": a.0, "index": 0, "name": "a\u{7}b"}),+ json!({"id": a.0, "index": 0, "name": "x".repeat(201)}),+ json!({"id": 999999, "index": 0, "name": "x"}),+ ] {+ assert!(s.execute("faces.setName", &p).is_err(), "{p}");+ }+ let _ = std::fs::remove_dir_all(&d);+}++#[test]+fn embeddings_are_cached_beside_the_library_and_reset_for_another_model() {+ let d = temp("persist");+ let lib = d.join("lib");+ let models = d.join("models");+ let install = |s: &mut Session, dim: u64| -> String {+ let model = d.join(format!("tiny{dim}.onnx"));+ std::fs::write(&model, lightcraft_faces::synthetic::tiny_embedder_model(dim)).unwrap();+ let r = s.execute("faces.models.install", &json!({"path": model.to_string_lossy(), "acknowledged": true})).unwrap();+ let id = r["installed"]["id"].as_str().unwrap().to_string();+ s.execute("faces.models.select", &json!({"id": id})).unwrap();+ id+ };+ let mut s = Session::new().with_fs();+ s.face_models_dir = Some(models.clone());+ s.open_library(&lib, true).unwrap();+ let (a, _) = two_photos(&s);+ set_regions(&mut s, a, vec![region(0.1, Some("Ann")), region(0.5, None)]);+ install(&mut s, 64);+ s.execute("faces.index", &json!({"budgetMs": 60_000})).unwrap();+ assert!(lib.join(format!("face-embeddings-{}.bin", s.faces.index.tag.split('@').next().unwrap())).is_file());+ s.close_library().unwrap();+ drop(s);++ // a new session finds the embeddings on disk and has nothing left to do+ let mut s2 = Session::new().with_fs();+ s2.face_models_dir = Some(models);+ s2.open_library(&lib, true).unwrap();+ let r = s2.execute("faces.index", &json!({"budgetMs": 60_000})).unwrap();+ assert_eq!((r["embedded"].clone(), r["indexedFaces"].clone(), r["pendingPhotos"].clone()), (json!(0), json!(2), json!(0)), "{r}");+ // choosing another model starts over: its faces are different numbers, the old ones are not used+ install(&mut s2, 32);+ let r = s2.execute("faces.index", &json!({"budgetMs": 0})).unwrap();+ assert_eq!((r["indexedFaces"].clone(), r["pendingPhotos"].clone()), (json!(0), json!(1)), "{r}");+ let r = s2.execute("faces.index", &json!({"budgetMs": 60_000})).unwrap();+ assert_eq!(r["embedded"], 2);+ let _ = std::fs::remove_dir_all(&d);+}++#[test]+fn without_a_chosen_model_recognition_says_so() {+ let d = temp("nomodel");+ let mut s = Session::with_demo();+ assert!(s.execute("faces.index", &json!({})).is_err(), "no models folder");+ s.face_models_dir = Some(d.join("models"));+ for cmd in ["faces.index", "faces.suggest"] {+ let e = s.execute(cmd, &json!({})).unwrap_err().to_string();+ assert!(e.contains("no recognition model is chosen"), "{cmd}: {e}");+ }+ let _: Value = json!(null);+ let _ = std::fs::remove_dir_all(&d);+}++#[test]+fn the_background_pump_indexes_by_itself_once_recognition_is_on() {+ let d = temp("pump");+ let (mut s, _) = setup(&d, 64);+ let (a, b) = two_photos(&s);+ set_regions(&mut s, a, vec![region(0.1, Some("Ann")), region(0.5, None)]);+ set_regions(&mut s, b, vec![region(0.3, None)]);+ // installing the model switched recognition on; switched off, the pump does nothing, and says so+ assert_eq!(s.execute("faces.enable", &json!({"enabled": false})).unwrap()["enabled"], false);+ assert_eq!(s.execute("faces.pump", &json!({})).unwrap()["active"], false);+ assert_eq!(s.faces.index.len(), 0);+ s.execute("faces.enable", &json!({"enabled": true})).unwrap();+ // `enabled` is looked up at most once a second, so the first pump may still see "off"+ let deadline = std::time::Instant::now() + std::time::Duration::from_secs(60);+ let mut last = json!(null);+ while std::time::Instant::now() < deadline {+ last = s.execute("faces.pump", &json!({})).unwrap();+ if last["active"] == true && last["pendingPhotos"] == 0 && last["indexedFaces"] == 3 {+ break;+ }+ std::thread::sleep(std::time::Duration::from_millis(50));+ }+ assert_eq!((last["active"].clone(), last["pendingPhotos"].clone(), last["indexedFaces"].clone()), (json!(true), json!(0), json!(3)), "{last}");+ // a new face is picked up (the catalog changed), and suggestions use what the pump embedded+ set_regions(&mut s, b, vec![region(0.3, None), region(0.6, None)]);+ let deadline = std::time::Instant::now() + std::time::Duration::from_secs(60);+ while std::time::Instant::now() < deadline && s.faces.index.len() < 4 {+ s.execute("faces.pump", &json!({})).unwrap();+ std::thread::sleep(std::time::Duration::from_millis(50));+ }+ assert_eq!(s.faces.index.len(), 4);+ let r = s.execute("faces.suggest", &json!({"ids": [b.0], "threshold": -1.0, "margin": -2.0, "budgetMs": 0})).unwrap();+ assert_eq!(r["photos"][0]["faces"].as_array().unwrap().len(), 2);+ assert!(r["photos"][0]["faces"][0]["suggestion"]["name"].is_string(), "{r}");+ let _ = std::fs::remove_dir_all(&d);+}++#[test]+fn the_background_pump_saves_what_it_embeds_for_the_next_launch() {+ let d = temp("pumpsave");+ let (lib, models) = (d.join("lib"), d.join("models"));+ let mut s = Session::new().with_fs();+ s.face_models_dir = Some(models);+ s.open_library(&lib, true).unwrap();+ let model = d.join("tiny64.onnx");+ std::fs::write(&model, lightcraft_faces::synthetic::tiny_embedder_model(64)).unwrap();+ // installing switches recognition on: nothing else is asked for+ s.execute("faces.models.install", &json!({"path": model.to_string_lossy(), "acknowledged": true})).unwrap();+ // without the detector (not bundled) photos without faces are left unsearched+ let searching = install_yunet(&mut s);+ let (a, b) = two_photos(&s);+ set_regions(&mut s, a, vec![region(0.1, Some("Ann")), region(0.5, None)]);+ set_regions(&mut s, b, vec![region(0.3, None)]);+ let photos = s.catalog.photos().count();+ let searched_photos = if searching { photos - 2 } else { 0 };+ let deadline = std::time::Instant::now() + std::time::Duration::from_secs(120);+ let mut last = json!(null);+ while std::time::Instant::now() < deadline {+ last = s.execute("faces.pump", &json!({})).unwrap();+ if last["pendingPhotos"] == 0 && s.faces.index.len() == 3 && s.faces.scanned.len() == searched_photos {+ break;+ }+ std::thread::sleep(std::time::Duration::from_millis(50));+ }+ assert_eq!((s.faces.index.len(), s.faces.scanned.len()), (3, searched_photos), "{last}");+ let (tag, path) = (s.faces.index.tag.clone(), s.faces.index.path.clone().unwrap());+ assert!(path.file_name().unwrap().to_string_lossy().starts_with("face-embeddings-custom-"), "one cache file per model: {}", path.display());+ // the first batch is written at once; whatever is left is written when the session ends, so a launch never redoes+ // work: neither the embeddings nor the list of photos already searched for faces+ s.close_library().unwrap();+ drop(s);+ let mut again = crate::faces_index::Index::default();+ again.reset(&tag, 64, Some(path));+ assert_eq!(again.len(), 3);+ let mut searched = crate::faces_index::Scanned::default();+ searched.reset(Some(lib.join("face-scanned.bin")));+ assert_eq!(searched.len(), searched_photos, "every photo without faces is remembered as searched");+ let _ = std::fs::remove_dir_all(&d);+}++#[test]+fn faces_person_lists_a_persons_faces_and_looks_for_more_only_while_recognition_runs() {+ let d = temp("person");+ let (mut s, _) = setup(&d, 64);+ s.execute("faces.index", &json!({"budgetMs": 0})).unwrap();+ let (a, b) = two_photos(&s);+ set_regions(&mut s, a, vec![region(0.1, Some("Ann")), region(0.5, Some("Bob"))]);+ set_regions(&mut s, b, vec![region(0.1, None), region(0.5, Some("ann"))]);+ let unit = |parts: &[(usize, f32)]| {+ let mut v = vec![0.0f32; 64];+ for (i, x) in parts {+ v[*i] = *x;+ }+ let n = v.iter().map(|x| x * x).sum::<f32>().sqrt();+ v.iter().map(|x| x / n).collect::<Vec<f32>>()+ };+ for (id, x, v) in [(a, 0.1, unit(&[(0, 1.0)])), (a, 0.5, unit(&[(1, 1.0)])), (b, 0.1, unit(&[(0, 0.9), (1, 0.1)])), (b, 0.5, unit(&[(0, 1.0)]))] {+ s.faces.index.insert(id.0, ®ion(x, None).rect, v);+ }+ // the name is matched without regard to case or surrounding spaces; every face is listed, wherever it is+ let r = s.execute("faces.person", &json!({"name": " ANN "})).unwrap();+ assert_eq!((r["name"].clone(), r["total"].clone()), (json!("Ann"), json!(2)));+ let mut at: Vec<(u64, u64)> =+ r["confirmed"].as_array().unwrap().iter().map(|f| (f["photo"].as_u64().unwrap(), f["index"].as_u64().unwrap())).collect();+ at.sort();+ let mut want = vec![(a.0, 0), (b.0, 1)];+ want.sort();+ assert_eq!(at, want);+ assert!(r["confirmed"].as_array().unwrap().iter().any(|f| f["rect"]["x0"] == 0.1 && f["rect"]["y1"] == 0.6));+ // recognition is on (installing the model switched it on): the unnamed face that looks like Ann is offered+ assert_eq!(r["ready"], true);+ let more = r["more"].as_array().unwrap();+ assert_eq!((more.len(), more[0]["photo"].clone(), more[0]["index"].clone()), (1, json!(b.0), json!(0)), "{r}");+ assert!(more[0]["score"].as_f64().unwrap() > 0.9);+ // confirming it moves it from one list to the other (and is one undo step)+ s.execute("faces.setName", &json!({"id": b.0, "index": 0, "name": "Ann"})).unwrap();+ let r = s.execute("faces.person", &json!({"name": "Ann"})).unwrap();+ assert_eq!((r["total"].clone(), r["more"].as_array().unwrap().len()), (json!(3), 0));+ // switched off, the faces are still listed but nothing is searched for+ s.execute("faces.enable", &json!({"enabled": false})).unwrap();+ let r = s.execute("faces.person", &json!({"name": "Ann"})).unwrap();+ assert_eq!((r["total"].clone(), r["ready"].clone(), r["more"].as_array().unwrap().len()), (json!(3), json!(false), 0));+ // nobody by that name, and bad input, are plain answers and errors+ let r = s.execute("faces.person", &json!({"name": "Nobody"})).unwrap();+ assert_eq!((r["total"].clone(), r["confirmed"].as_array().unwrap().len(), r["ready"].clone()), (json!(0), 0, json!(false)));+ for bad in [json!({}), json!({"name": ""}), json!({"name": " "}), json!({"name": 7})] {+ assert!(s.execute("faces.person", &bad).is_err(), "{bad}");+ }+ assert!(s.execute("faces.person", &json!({"name": "Ann", "more": u64::MAX})).is_ok());+ let _ = std::fs::remove_dir_all(&d);+}++fn unit64(parts: &[(usize, f32)]) -> Vec<f32> {+ let mut v = vec![0.0f32; 64];+ for (i, x) in parts {+ v[*i] = *x;+ }+ let n = v.iter().map(|x| x * x).sum::<f32>().sqrt();+ v.iter().map(|x| x / n).collect()+}++#[test]+fn many_faces_are_named_in_one_undo_step() {+ let d = temp("namemany");+ let (mut s, _) = setup(&d, 64);+ let (a, b) = two_photos(&s);+ let detected = |x: f64| Region { description: Some("Detected by YuNet 2023mar".into()), ..region(x, None) };+ set_regions(&mut s, a, vec![detected(0.1), region(0.5, None)]);+ set_regions(&mut s, b, vec![detected(0.3)]);+ let undo = s.undo.len();+ let r = s+ .execute(+ "faces.nameFaces",+ &json!({"faces": [{"photo": a.0, "index": 0}, {"photo": a.0, "index": 1}, {"photo": b.0, "index": 0}], "name": " Jane Doe "}),+ )+ .unwrap();+ assert_eq!((r["named"].clone(), r["photos"].clone()), (json!(3), json!(2)));+ assert_eq!(s.undo.len(), undo + 1, "one step for the whole group");+ for (id, i) in [(a, 0), (a, 1), (b, 0)] {+ let reg = &s.catalog.photo(id).unwrap().meta.regions[i];+ assert_eq!(reg.name.as_deref(), Some("Jane Doe"));+ assert!(reg.description.is_none() || !reg.description.as_deref().unwrap().starts_with("Detected by"), "a named face is the user's now");+ }+ // undo restores every one of them together+ s.execute("edit.undo", &json!({})).unwrap();+ assert!(s.catalog.photo(a).unwrap().meta.regions.iter().chain(s.catalog.photo(b).unwrap().meta.regions.iter()).all(|r| r.name.is_none()));+ // an empty name clears; faces that do not exist are skipped, and with none left it is an error+ s.execute("faces.nameFaces", &json!({"faces": [{"photo": a.0, "index": 0}], "name": "Ann"})).unwrap();+ let r = s+ .execute(+ "faces.nameFaces",+ &json!({"faces": [{"photo": a.0, "index": 0}, {"photo": a.0, "index": 99}, {"photo": 987_654, "index": 0}], "name": null}),+ )+ .unwrap();+ assert_eq!(r["named"], 1);+ assert!(s.catalog.photo(a).unwrap().meta.regions[0].name.is_none());+ for bad in [+ json!({"name": "x"}),+ json!({"faces": "x", "name": "x"}),+ json!({"faces": [{"photo": a.0}], "name": "x"}),+ json!({"faces": [{"photo": 987_654, "index": 0}], "name": "x"}),+ json!({"faces": [{"photo": a.0, "index": 0}], "name": 7}),+ json!({"faces": [{"photo": a.0, "index": 0}], "name": "bad\u{7}name"}),+ json!({"faces": vec![json!({"photo": a.0, "index": 0}); 5001], "name": "x"}),+ ] {+ assert!(s.execute("faces.nameFaces", &bad).is_err(), "{bad}");+ }+ let _ = std::fs::remove_dir_all(&d);+}++#[test]+fn unnamed_faces_come_with_look_alikes_together_and_the_name_the_named_ones_suggest() {+ let d = temp("unnamed");+ let (mut s, _) = setup(&d, 64);+ s.execute("faces.index", &json!({"budgetMs": 0})).unwrap();+ let (a, b) = two_photos(&s);+ let c = s.catalog.photos().map(|p| p.id).find(|id| *id != a && *id != b).unwrap();+ set_regions(&mut s, a, vec![region(0.1, Some("Ann")), region(0.5, None), region(0.7, None)]);+ set_regions(&mut s, b, vec![region(0.1, None), region(0.5, None)]);+ set_regions(&mut s, c, vec![region(0.3, None)]);+ // three faces like Ann, two like Bob (unnamed), one unembedded+ let put = |s: &mut Session, id: lightcraft_catalog::PhotoId, x: f64, v: Vec<f32>| s.faces.index.insert(id.0, ®ion(x, None).rect, v);+ put(&mut s, a, 0.1, unit64(&[(0, 1.0)])); // Ann, named+ put(&mut s, a, 0.5, unit64(&[(1, 1.0)])); // like Bob+ put(&mut s, b, 0.1, unit64(&[(0, 0.97), (2, 0.2)])); // like Ann+ put(&mut s, b, 0.5, unit64(&[(1, 0.98), (2, 0.2)])); // like Bob+ put(&mut s, c, 0.3, unit64(&[(0, 0.95), (3, 0.3)])); // like Ann+ // (a's last face, at 0.7, is not embedded yet)+ let r = s.execute("faces.unnamed", &json!({})).unwrap();+ assert_eq!((r["total"].clone(), r["ordered"].clone(), r["ready"].clone()), (json!(5), json!(true), json!(true)), "{r}");+ let faces = r["faces"].as_array().unwrap();+ let at = |f: &Value| (f["photo"].as_u64().unwrap(), f["index"].as_u64().unwrap());+ let order: Vec<(u64, u64)> = faces.iter().map(at).collect();+ // look-alikes are neighbours: the two like Bob are adjacent, and the three like Ann are adjacent; the unembedded one is last+ let pos = |x: (u64, u64)| order.iter().position(|o| *o == x).unwrap();+ assert_eq!(pos((a.0, 1)).abs_diff(pos((b.0, 1))), 1, "{order:?}");+ assert_eq!(order.last(), Some(&(a.0, 2)), "{order:?}");+ // a face the named ones recognise carries the name they suggest+ let like_ann = faces.iter().find(|f| at(f) == (b.0, 0)).unwrap();+ assert_eq!(like_ann["suggestion"]["name"], "Ann", "{like_ann}");+ assert!(faces.iter().find(|f| at(f) == (b.0, 1)).unwrap()["suggestion"].is_null(), "Bob is nobody the named faces know");+ // each face says which box to show it by; with no scan view it is its own box+ for k in ["x0", "y0", "x1", "y1"] {+ assert!((like_ann["view"][k].as_f64().unwrap() - like_ann["rect"][k].as_f64().unwrap()).abs() < 1e-3, "{like_ann}");+ }+ // a limit cuts the list, not the count+ let r = s.execute("faces.unnamed", &json!({"limit": 2})).unwrap();+ assert_eq!((r["total"].clone(), r["faces"].as_array().unwrap().len()), (json!(5), 2));+ // switched off, the faces are still listed, in photo order, without suggestions+ s.execute("faces.enable", &json!({"enabled": false})).unwrap();+ let r = s.execute("faces.unnamed", &json!({})).unwrap();+ assert_eq!((r["total"].clone(), r["ordered"].clone(), r["ready"].clone()), (json!(5), json!(false), json!(false)));+ assert!(r["faces"].as_array().unwrap().iter().all(|f| f["suggestion"].is_null()));+ for odd in [json!({"limit": 0}), json!({"limit": "x"}), json!({"limit": u64::MAX})] {+ assert!(s.execute("faces.unnamed", &odd).is_ok(), "{odd}");+ }+ let _ = std::fs::remove_dir_all(&d);+}++#[test]+fn every_scanned_face_is_shown_by_the_same_kind_of_box() {+ let d = temp("view");+ let (mut s, _) = setup(&d, 64);+ s.execute("faces.index", &json!({"budgetMs": 0})).unwrap();+ let (a, _) = two_photos(&s);+ // a loosely drawn box (from some other tool) and the detector's tighter box of the same face+ let loose = lightcraft_geom::Rect { x0: 0.2, y0: 0.1, x1: 0.8, y1: 0.9 };+ let tight = lightcraft_geom::Rect { x0: 0.35, y0: 0.25, x1: 0.65, y1: 0.65 };+ let mut meta = s.catalog.photo(a).unwrap().meta.clone();+ meta.regions = vec![Region { rect: loose, kind: RegionKind::Face, name: Some("Ann".into()), description: None }];+ s.commit("setup", Op::SetMeta { id: a, meta: Box::new(meta) }).unwrap();+ // before the scan has looked at it, the face is shown by its own box+ assert_eq!(s.face_view(a, loose), loose);+ s.faces.index.insert_with_view(a.0, &loose, unit64(&[(0, 1.0)]), &tight);+ let shown = s.face_view(a, loose);+ assert!((shown.x0 - tight.x0).abs() < 1e-3 && (shown.y1 - tight.y1).abs() < 1e-3, "shown by the detector's box once known: {shown:?}");+ // a face nobody has looked at, or in another photo, is shown by its own box+ let other = lightcraft_geom::Rect { x0: 0.1, y0: 0.1, x1: 0.2, y1: 0.2 };+ assert_eq!(s.face_view(a, other), other);+ // the page of the person uses it too+ let r = s.execute("faces.person", &json!({"name": "Ann"})).unwrap();+ let view = &r["confirmed"][0]["view"];+ assert!((view["x0"].as_f64().unwrap() - tight.x0).abs() < 1e-3, "{r}");+ // and it survives a restart: the index file keeps it beside the embedding+ let (tag, path) = (s.faces.index.tag.clone(), d.join("view-cache.bin"));+ let mut written = crate::faces_index::Index::default();+ written.reset(&tag, 64, Some(path.clone()));+ written.insert_with_view(a.0, &loose, unit64(&[(0, 1.0)]), &tight);+ written.save().unwrap();+ drop(written);+ let mut again = crate::faces_index::Index::default();+ again.reset(&tag, 64, Some(path));+ let v = again.view(a.0, &loose).expect("the face is remembered");+ assert!((v.x0 - tight.x0).abs() < 1e-3 && (v.y0 - tight.y0).abs() < 1e-3, "{v:?}");+ let _ = std::fs::remove_dir_all(&d);+}diff --git a/crates/faces/Cargo.toml b/crates/faces/Cargo.tomlindex da04c3d08..112967cee 100644--- a/crates/faces/Cargo.toml+++ b/crates/faces/Cargo.toml@@ -3,12 +3,19 @@ name = "lightcraft-faces"version.workspace = trueedition.workspace = truelicense.workspace = true-description = "LightCraft faces (L1): face-model manifests, validation, the known-model table and an ONNX shape probe"+description = "LightCraft faces (L1): bounded ONNX CPU inference, face-model manifests, validation and detection"[lints]workspace = true+[features]+default = []+[dependencies]+ndarray = { version = "=0.17.2", default-features = false, features = ["std"] }+# Rust intrinsics only, CPU-dispatched; no BLAS, cc/assembly or nested threading.+matrixmultiply = { version = "=0.3.11", default-features = false, features = ["std", "avx512"] }+web-time = { workspace = true }serde = { workspace = true }serde_json = { workspace = true }sha2 = { workspace = true }diff --git a/crates/faces/src/align.rs b/crates/faces/src/align.rsnew file mode 100644index 000000000..4982325ae--- /dev/null+++ b/crates/faces/src/align.rs@@ -0,0 +1,273 @@+//! Cutting a face out of a photo the way recognition models expect it.+//!+//! With the five landmarks the detector finds (eyes, nose tip, mouth corners), the face is turned and scaled+//! (a similarity transform, found by least squares) so those points land on the standard 112 × 112+//! ArcFace template. Without landmarks (a box from an XMP sidecar) a padded square around the box is used+//! instead. Both warp with supersampling, so shrinking a large face does not alias.+//!+//! Pure geometry on 8-bit RGB: no image types, no panics (every access is checked).++/// The five template points of a 112 × 112 aligned face, image-left eye first: eye, eye, nose tip, mouth+/// corner, mouth corner. (The standard ArcFace template.)+pub const TEMPLATE_112: [(f32, f32); 5] = [(38.2946, 51.6963), (73.5318, 51.5014), (56.0252, 71.7366), (41.5493, 92.3655), (70.7299, 92.2041)];++/// A borrowed 8-bit RGB picture.+#[derive(Clone, Copy)]+pub struct Rgb<'a> {+ pub data: &'a [u8],+ pub width: usize,+ pub height: usize,+}++impl Rgb<'_> {+ fn valid(&self) -> bool {+ self.width > 0 && self.height > 0 && self.width.checked_mul(self.height).and_then(|n| n.checked_mul(3)) == Some(self.data.len())+ }++ /// The pixel at integer coordinates, clamped to the picture.+ fn at(&self, x: isize, y: isize) -> [f32; 3] {+ let (x, y) = (x.clamp(0, self.width as isize - 1) as usize, y.clamp(0, self.height as isize - 1) as usize);+ let i = (y * self.width + x) * 3;+ match self.data.get(i..i + 3) {+ Some(p) => [f32::from(p[0]), f32::from(p[1]), f32::from(p[2])],+ None => [0.0; 3],+ }+ }++ /// Bilinear sample at a fractional position (pixel centres at integers).+ fn sample(&self, x: f32, y: f32) -> [f32; 3] {+ let (x0, y0) = (x.floor(), y.floor());+ let (fx, fy) = (x - x0, y - y0);+ let (x0, y0) = (x0 as isize, y0 as isize);+ let (a, b, c, d) = (self.at(x0, y0), self.at(x0 + 1, y0), self.at(x0, y0 + 1), self.at(x0 + 1, y0 + 1));+ let mut out = [0.0; 3];+ for k in 0..3 {+ let top = a[k] * (1.0 - fx) + b[k] * fx;+ let bottom = c[k] * (1.0 - fx) + d[k] * fx;+ out[k] = top * (1.0 - fy) + bottom * fy;+ }+ out+ }+}++/// `q = a · p + t` with complex `a` (scale and rotation) and `t`, mapping source positions to output positions.+#[derive(Clone, Copy, Debug, PartialEq)]+pub struct Similarity {+ pub a: (f32, f32),+ pub t: (f32, f32),+}++impl Similarity {+ pub fn scale(&self) -> f32 {+ self.a.0.hypot(self.a.1)+ }++ /// Where an output position `q` comes from in the source.+ fn invert(&self, q: (f32, f32)) -> Option<(f32, f32)> {+ let n = self.a.0 * self.a.0 + self.a.1 * self.a.1;+ if !(n.is_finite() && n > 1e-12) {+ return None;+ }+ let (dx, dy) = (q.0 - self.t.0, q.1 - self.t.1);+ Some(((dx * self.a.0 + dy * self.a.1) / n, (dy * self.a.0 - dx * self.a.1) / n))+ }+}++/// The least-squares similarity transform taking `from[i]` to `to[i]` (no reflection). `None` when the points+/// do not determine one (all the same point, or non-finite).+pub fn fit_similarity(from: &[(f32, f32)], to: &[(f32, f32)]) -> Option<Similarity> {+ if from.len() != to.len() || from.len() < 2 {+ return None;+ }+ let n = from.len() as f32;+ let mean = |pts: &[(f32, f32)]| (pts.iter().map(|p| p.0).sum::<f32>() / n, pts.iter().map(|p| p.1).sum::<f32>() / n);+ let (mp, mq) = (mean(from), mean(to));+ // a = Σ conj(p') q' / Σ |p'|² (complex numbers)+ let (mut re, mut im, mut norm) = (0.0f32, 0.0f32, 0.0f32);+ for (p, q) in from.iter().zip(to) {+ let (px, py, qx, qy) = (p.0 - mp.0, p.1 - mp.1, q.0 - mq.0, q.1 - mq.1);+ re += px * qx + py * qy;+ im += px * qy - py * qx;+ norm += px * px + py * py;+ }+ if !(norm.is_finite() && norm > 1e-9) {+ return None;+ }+ let a = (re / norm, im / norm);+ let t = (mq.0 - (a.0 * mp.0 - a.1 * mp.1), mq.1 - (a.0 * mp.1 + a.1 * mp.0));+ (a.0.is_finite() && a.1.is_finite() && t.0.is_finite() && t.1.is_finite()).then_some(Similarity { a, t })+}++/// Fill an `out_w` × `out_h` picture by taking, for each output pixel, the source under it, averaging+/// enough sub-samples that shrinking does not alias.+fn warp(img: &Rgb, m: &Similarity, out_w: usize, out_h: usize) -> Option<Vec<u8>> {+ if !img.valid() || out_w == 0 || out_h == 0 || out_w > 4096 || out_h > 4096 {+ return None;+ }+ if ![m.a.0, m.a.1, m.t.0, m.t.1].iter().all(|v| v.is_finite()) {+ return None;+ }+ // how many source pixels one output pixel spans, per side+ let span = 1.0 / m.scale();+ let n = if span.is_finite() { (span.ceil() as usize).clamp(1, 4) } else { 1 };+ let mut out = vec![0u8; out_w * out_h * 3];+ for (i, px) in out.as_chunks_mut::<3>().0.iter_mut().enumerate() {+ let (ox, oy) = ((i % out_w) as f32, (i / out_w) as f32);+ let mut acc = [0.0f32; 3];+ for sy in 0..n {+ for sx in 0..n {+ let q = (ox + (sx as f32 + 0.5) / n as f32 - 0.5, oy + (sy as f32 + 0.5) / n as f32 - 0.5);+ let (px_, py_) = m.invert(q)?;+ let s = img.sample(px_, py_);+ acc.iter_mut().zip(s).for_each(|(a, v)| *a += v);+ }+ }+ let k = (n * n) as f32;+ for (o, a) in px.iter_mut().zip(acc) {+ *o = (a / k).round().clamp(0.0, 255.0) as u8;+ }+ }+ Some(out)+}++/// The face aligned to the ArcFace template, `out_w` × `out_h` (the template is for 112 × 112 and is scaled for+/// other sizes). `landmarks` are in pixels of `img`. `None` when the landmarks cannot define a transform.+pub fn align_to_template(img: &Rgb, landmarks: &[(f32, f32); 5], out_w: usize, out_h: usize) -> Option<Vec<u8>> {+ let (sx, sy) = (out_w as f32 / 112.0, out_h as f32 / 112.0);+ let template: Vec<(f32, f32)> = TEMPLATE_112.iter().map(|(x, y)| (x * sx, y * sy)).collect();+ let m = fit_similarity(landmarks, &template)?;+ // a face smaller than a few pixels, or absurdly large, is not a face we can use+ let scale = m.scale();+ if !(scale.is_finite() && (0.01..=100.0).contains(&scale)) {+ return None;+ }+ warp(img, &m, out_w, out_h)+}++/// A square around a face box (pixels of `img`), `pad` times its larger side, as `out_w` × `out_h`.+pub fn crop_box(img: &Rgb, x0: f32, y0: f32, x1: f32, y1: f32, pad: f32, out_w: usize, out_h: usize) -> Option<Vec<u8>> {+ if ![x0, y0, x1, y1, pad].iter().all(|v| v.is_finite()) {+ return None;+ }+ let side = (x1 - x0).abs().max((y1 - y0).abs()) * pad;+ if !(side.is_finite() && side >= 2.0) {+ return None;+ }+ let (cx, cy) = ((x0 + x1) / 2.0, (y0 + y1) / 2.0);+ let s = out_w.min(out_h) as f32 / side;+ let m = Similarity { a: (s, 0.0), t: (out_w as f32 / 2.0 - s * cx, out_h as f32 / 2.0 - s * cy) };+ warp(img, &m, out_w, out_h)+}++#[cfg(test)]+mod tests {+ use super::*;++ fn close(a: (f32, f32), b: (f32, f32), tol: f32) -> bool {+ (a.0 - b.0).abs() <= tol && (a.1 - b.1).abs() <= tol+ }++ fn apply(m: &Similarity, p: (f32, f32)) -> (f32, f32) {+ (m.a.0 * p.0 - m.a.1 * p.1 + m.t.0, m.a.0 * p.1 + m.a.1 * p.0 + m.t.1)+ }++ #[test]+ fn a_known_similarity_is_recovered() {+ // rotate 20 degrees, scale 0.7, translate+ let (s, th) = (0.7f32, 20f32.to_radians());+ let truth = Similarity { a: (s * th.cos(), s * th.sin()), t: (13.0, -5.0) };+ let from: Vec<(f32, f32)> = TEMPLATE_112.to_vec();+ let to: Vec<(f32, f32)> = from.iter().map(|p| apply(&truth, *p)).collect();+ let m = fit_similarity(&from, &to).unwrap();+ assert!(close(m.a, truth.a, 1e-4) && close(m.t, truth.t, 1e-3), "{m:?} vs {truth:?}");+ // noisy points still give the best fit (the residual is small)+ let noisy: Vec<(f32, f32)> = to.iter().enumerate().map(|(i, p)| (p.0 + (i as f32 - 2.0) * 0.2, p.1 - (i as f32 - 2.0) * 0.1)).collect();+ let m2 = fit_similarity(&from, &noisy).unwrap();+ assert!(from.iter().zip(&noisy).all(|(p, q)| close(apply(&m2, *p), *q, 1.0)));+ }++ #[test]+ fn degenerate_points_give_nothing() {+ let same = vec![(5.0, 5.0); 5];+ assert!(fit_similarity(&same, &TEMPLATE_112).is_none());+ assert!(fit_similarity(&TEMPLATE_112, &TEMPLATE_112[..3]).is_none());+ assert!(fit_similarity(&[(0.0, 0.0)], &[(1.0, 1.0)]).is_none());+ let nan = vec![(f32::NAN, 0.0); 5];+ assert!(fit_similarity(&nan, &TEMPLATE_112).is_none());+ let inf = vec![(f32::INFINITY, 0.0), (1.0, 1.0), (2.0, 2.0), (3.0, 3.0), (4.0, 4.0)];+ assert!(fit_similarity(&inf, &TEMPLATE_112).is_none());+ }++ /// A picture with a bright dot at each of five landmarks, the landmarks being the template moved by a known+ /// similarity: aligning must put the dots back on the template.+ #[test]+ fn alignment_puts_the_landmarks_on_the_template() {+ let (w, h) = (400usize, 300usize);+ let (s, th) = (1.7f32, -15f32.to_radians());+ let place = Similarity { a: (s * th.cos(), s * th.sin()), t: (150.0, 60.0) };+ let landmarks: Vec<(f32, f32)> = TEMPLATE_112.iter().map(|p| apply(&place, *p)).collect();+ let mut data = vec![0u8; w * h * 3];+ for l in &landmarks {+ for dy in -3i32..=3 {+ for dx in -3i32..=3 {+ let (x, y) = ((l.0 as i32 + dx) as usize, (l.1 as i32 + dy) as usize);+ if x < w && y < h {+ data[(y * w + x) * 3] = 255;+ }+ }+ }+ }+ let img = Rgb { data: &data, width: w, height: h };+ let lm: [(f32, f32); 5] = [landmarks[0], landmarks[1], landmarks[2], landmarks[3], landmarks[4]];+ let out = align_to_template(&img, &lm, 112, 112).unwrap();+ assert_eq!(out.len(), 112 * 112 * 3);+ // the red channel's centre of mass near each template point is on the point+ for t in TEMPLATE_112 {+ let (mut sx, mut sy, mut sw) = (0.0f32, 0.0f32, 0.0f32);+ for y in (t.1 as usize).saturating_sub(8)..(t.1 as usize + 9).min(112) {+ for x in (t.0 as usize).saturating_sub(8)..(t.0 as usize + 9).min(112) {+ let v = f32::from(out[(y * 112 + x) * 3]);+ sx += v * x as f32;+ sy += v * y as f32;+ sw += v;+ }+ }+ assert!(sw > 100.0, "no dot near {t:?}");+ assert!(close((sx / sw, sy / sw), t, 1.5), "dot at {:?}, template {t:?}", (sx / sw, sy / sw));+ }+ }++ #[test]+ fn box_crops_are_centred_and_bad_inputs_are_refused() {+ let (w, h) = (200usize, 100usize);+ let mut data = vec![0u8; w * h * 3];+ for y in 40..60 {+ for x in 90..110 {+ data[(y * w + x) * 3 + 1] = 255;+ }+ }+ let img = Rgb { data: &data, width: w, height: h };+ let out = crop_box(&img, 90.0, 40.0, 110.0, 60.0, 2.0, 64, 64).unwrap();+ // the 20 px block fills the middle half of the 64 px output+ let at = |x: usize, y: usize| out[(y * 64 + x) * 3 + 1];+ assert!(at(32, 32) > 250 && at(20, 20) > 250 && at(44, 44) > 250 && at(5, 5) < 5 && at(60, 32) < 5);+ assert!(crop_box(&img, 5.0, 5.0, 5.0, 5.0, 2.0, 64, 64).is_none(), "an empty box");+ assert!(crop_box(&img, f32::NAN, 0.0, 10.0, 10.0, 2.0, 64, 64).is_none());+ assert!(crop_box(&img, 0.0, 0.0, 10.0, 10.0, 2.0, 0, 64).is_none());+ assert!(crop_box(&img, 0.0, 0.0, 10.0, 10.0, 2.0, 100_000, 64).is_none());+ let bad = Rgb { data: &data[..10], width: w, height: h };+ assert!(crop_box(&bad, 0.0, 0.0, 10.0, 10.0, 2.0, 64, 64).is_none(), "wrong data length");+ }++ #[test]+ fn faces_off_the_edge_and_tiny_or_huge_landmarks_do_not_panic() {+ let data = vec![80u8; 50 * 50 * 3];+ let img = Rgb { data: &data, width: 50, height: 50 };+ // a box mostly outside the picture is padded by clamping, not an error+ assert!(crop_box(&img, -30.0, -30.0, 20.0, 20.0, 1.5, 32, 32).is_some());+ let far: [(f32, f32); 5] = [(1e6, 1e6), (1e6 + 5.0, 1e6), (1e6 + 2.0, 1e6 + 3.0), (1e6, 1e6 + 6.0), (1e6 + 5.0, 1e6 + 6.0)];+ let _ = align_to_template(&img, &far, 112, 112);+ let tiny: [(f32, f32); 5] = [(1.0, 1.0), (1.01, 1.0), (1.0, 1.01), (1.01, 1.01), (1.005, 1.005)];+ assert!(align_to_template(&img, &tiny, 112, 112).is_none(), "landmarks a hundredth of a pixel apart");+ }+}diff --git a/crates/faces/src/catalog.rs b/crates/faces/src/catalog.rsnew file mode 100644index 000000000..07a206228--- /dev/null+++ b/crates/faces/src/catalog.rs@@ -0,0 +1,228 @@+//! Your own list of models to download: `catalog.json` in the models folder.+//!+//! LightCraft's built-in list only holds models whose terms let the project point at them. Anything else (a stronger+//! recogniser whose weights are for research use, a model you trained, a mirror you trust) can be added by you: each entry+//! is a model manifest (the same fields as a `face-model.json`) plus the address to fetch it from, and it then gets the same+//! Download button as a built-in one, with its licence notice shown before anything is fetched. LightCraft ships and links to+//! none of them; what the file says, and whether you may use the weights, is yours to check.+//!+//! ```json+//! { "models": [ {+//! "id": "my-recogniser", "name": "My recogniser", "version": "1", "role": "embedder",+//! "url": "https://example.org/weights/recogniser.onnx",+//! "sha256": "…64 lowercase hex digits…", "sizeBytes": 166000000,+//! "licence": { "name": "Research use only", "commercial": "no", "notice": "Not for commercial use." },+//! "provenance": "Trained on …",+//! "output": { "kind": "embedding", "dim": 512 },+//! "thresholds": { "matchCosine": 0.4 },+//! "speed": 1.7+//! } ] }+//! ```+//!+//! `speed` (optional) is how many times faster the model is than a ResNet-100, shown in Settings instead of timings.+//! `input` may be left out (112 × 112 RGB, `(x − 127.5) / 127.5`, the ArcFace convention). The file is untrusted: it is read+//! with a size limit, every entry is validated, and a bad entry is reported and skipped, never a reason to refuse the rest.+//! The `sha256` and `sizeBytes` are required, since they are what a downloaded file is checked against.++use serde::Deserialize;+use serde_json::Value;++use crate::known::{self, Download};+use crate::manifest::{self, ModelManifest, Role};++/// Largest `catalog.json` we read.+pub const MAX_CATALOG_BYTES: usize = 256 * 1024;+/// Most entries we take from it.+pub const MAX_ENTRIES: usize = 50;++/// One model from the catalog: its manifest, where to fetch it and what to call the file meanwhile.+#[derive(Clone, Debug, PartialEq)]+pub struct Entry {+ pub manifest: ModelManifest,+ pub url: String,+ pub file_name: String,+}++impl Entry {+ /// What the download machinery needs. (`sha256` and `size_bytes` are present: [`parse`] requires them.)+ pub fn download(&self) -> Option<Download> {+ Some(Download {+ id: self.manifest.id.clone(),+ url: self.url.clone(),+ file_name: self.file_name.clone(),+ size_bytes: self.manifest.size_bytes?,+ sha256: self.manifest.sha256.clone()?,+ })+ }+}++/// What a catalog file holds.+#[derive(Clone, Debug, Default, PartialEq)]+pub struct Catalog {+ pub entries: Vec<Entry>,+ /// One plain sentence for each entry (or the file) that could not be used.+ pub errors: Vec<String>,+}++#[derive(Deserialize)]+struct Raw {+ url: String,+ #[serde(flatten)]+ manifest: ModelManifest,+}++/// A file name for the download, from the address: letters, digits, `.`, `_`, `-`, ending in `.onnx`.+fn file_name_of(url: &str) -> String {+ let last = url.split(['?', '#']).next().unwrap_or("").rsplit('/').next().unwrap_or("");+ let clean: String = last.chars().filter(|c| c.is_ascii_alphanumeric() || matches!(c, '.' | '_' | '-')).take(80).collect();+ if clean.ends_with(".onnx") && clean.len() > 5 && !clean.starts_with('.') { clean } else { "model.onnx".to_string() }+}++fn entry(v: &Value) -> Result<Entry, String> {+ let raw: Raw = serde_json::from_value(v.clone()).map_err(|e| e.to_string())?;+ manifest::validate(&raw.manifest).map_err(|e| e.to_string())?;+ let m = &raw.manifest;+ if m.role != Role::Embedder {+ return Err("only face recognition models (role \"embedder\") can be downloaded".into());+ }+ if m.sha256.is_none() || m.size_bytes.is_none() {+ return Err("needs `sha256` and `sizeBytes`: a download is checked against them".into());+ }+ if !(raw.url.len() <= 500 && raw.url.starts_with("https://") && !raw.url.chars().any(|c| c.is_control() || c.is_whitespace())) {+ return Err("`url` must be an https:// address of at most 500 characters, without spaces".into());+ }+ if known::all().iter().any(|k| k.id == m.id) {+ return Err("that id belongs to a model LightCraft already knows".into());+ }+ if m.sha256.as_deref().is_some_and(|h| known::lookup(h).is_some()) {+ return Err("LightCraft already knows a model with that file".into());+ }+ Ok(Entry { file_name: file_name_of(&raw.url), url: raw.url, manifest: raw.manifest })+}++/// Read a catalog file's bytes: the usable entries, and why the others were left out.+pub fn parse(bytes: &[u8]) -> Catalog {+ let mut out = Catalog::default();+ if bytes.len() > MAX_CATALOG_BYTES {+ out.errors.push(format!("catalog.json is larger than {} KB: not read", MAX_CATALOG_BYTES / 1024));+ return out;+ }+ // Notepad and older PowerShell begin a UTF-8 file with a byte-order mark+ let bytes = bytes.strip_prefix(&[0xEF, 0xBB, 0xBF]).unwrap_or(bytes);+ let value: Value = match serde_json::from_slice(bytes) {+ Ok(v) => v,+ Err(e) => {+ out.errors.push(format!("catalog.json is not valid JSON: {e}"));+ return out;+ }+ };+ let Some(models) = value.get("models").and_then(Value::as_array) else {+ out.errors.push("catalog.json needs a \"models\" list".into());+ return out;+ };+ if models.len() > MAX_ENTRIES {+ out.errors.push(format!("only the first {MAX_ENTRIES} models of the catalog are read"));+ }+ for (i, v) in models.iter().take(MAX_ENTRIES).enumerate() {+ let label = v+ .get("id")+ .and_then(Value::as_str)+ .map_or_else(|| format!("model {}", i + 1), |id| format!("`{}`", id.chars().take(40).collect::<String>()));+ match entry(v) {+ Ok(e) if out.entries.iter().any(|x| x.manifest.id == e.manifest.id) => {+ out.errors.push(format!("{label}: listed twice, the first is used"))+ }+ Ok(e) => out.entries.push(e),+ Err(why) => out.errors.push(format!("{label}: {why}")),+ }+ }+ out+}++#[cfg(test)]+mod tests {+ use super::*;+ use serde_json::json;++ fn good(id: &str) -> Value {+ json!({+ "id": id, "name": "Test model", "version": "1", "role": "embedder",+ "url": "https://example.org/w/recogniser-r50.onnx?download=1",+ "sha256": "ab".repeat(32), "sizeBytes": 1234,+ "licence": {"name": "Research only", "commercial": "no", "notice": "Not for commercial use."},+ "output": {"kind": "embedding", "dim": 512}, "thresholds": {"matchCosine": 0.4}, "speed": 1.7,+ })+ }++ fn cat(models: Vec<Value>) -> Catalog {+ parse(json!({"models": models}).to_string().as_bytes())+ }++ #[test]+ fn a_manifest_with_an_address_is_a_downloadable_model() {+ let c = cat(vec![good("my-model")]);+ assert_eq!(c.errors, Vec::<String>::new());+ let e = &c.entries[0];+ assert_eq!((e.manifest.id.as_str(), e.file_name.as_str()), ("my-model", "recogniser-r50.onnx"), "the file name comes from the address");+ assert_eq!(e.manifest.licence.commercial, crate::Commercial::No, "a non-commercial model is the user's to add");+ let d = e.download().unwrap();+ assert_eq!((d.size_bytes, d.sha256.len(), d.url.as_str()), (1234, 64, "https://example.org/w/recogniser-r50.onnx?download=1"));+ assert_eq!(e.manifest.input, crate::InputSpec::default(), "the input defaults to the ArcFace convention");+ assert_eq!(e.manifest.speed, Some(1.7), "the speed ratio is the user's to state");+ let mut nonsense = good("nonsense-speed");+ nonsense["speed"] = json!(-3);+ let c = cat(vec![nonsense]);+ assert!(c.entries.is_empty() && c.errors.iter().any(|e| e.contains("speed")), "{:?}", c.errors);+ }++ #[test]+ fn a_file_saved_with_a_byte_order_mark_still_reads() {+ let mut bytes = vec![0xEF, 0xBB, 0xBF];+ bytes.extend_from_slice(json!({"models": [good("with-bom")]}).to_string().as_bytes());+ let c = parse(&bytes);+ assert_eq!((c.entries.len(), c.errors.len()), (1, 0), "{:?}", c.errors);+ }++ #[test]+ fn bad_entries_are_reported_and_the_rest_still_load() {+ let mut no_hash = good("no-hash");+ no_hash.as_object_mut().unwrap().remove("sha256");+ let mut http = good("plain-http");+ http["url"] = json!("http://example.org/m.onnx");+ let mut spaces = good("spaces");+ spaces["url"] = json!("https://example.org/a b.onnx");+ let mut detector = good("a-detector");+ detector["role"] = json!("detector");+ detector["output"] = json!({"kind": "detector", "decoder": "yunet-v2"});+ let mut bad_id = good("Bad/Id");+ bad_id["id"] = json!("../escape");+ let builtin = good("sface-2021dec");+ let mut known_file = good("known-file");+ known_file["sha256"] = json!(known::SFACE_SHA256);+ let c = cat(vec![good("fine"), no_hash, http, spaces, detector, bad_id, builtin, known_file, good("fine"), json!(7), json!({"id": "x"})]);+ assert_eq!(c.entries.iter().map(|e| e.manifest.id.as_str()).collect::<Vec<_>>(), ["fine"]);+ assert_eq!(c.errors.len(), 10, "{:#?}", c.errors);+ assert!(c.errors.iter().any(|e| e.contains("`no-hash`") && e.contains("sha256")));+ assert!(c.errors.iter().any(|e| e.contains("https")));+ assert!(c.errors.iter().any(|e| e.contains("already knows")));+ assert!(c.errors.iter().any(|e| e.contains("listed twice")));+ }++ #[test]+ fn a_hostile_or_broken_file_is_an_error_message_not_a_crash() {+ for bytes in [&b""[..], b"not json", b"[]", b"{}", b"{\"models\": 5}", b"\xff\xfe\x00", &vec![b' '; MAX_CATALOG_BYTES + 1]] {+ let c = parse(bytes);+ assert!(c.entries.is_empty() && !c.errors.is_empty(), "{:?}", String::from_utf8_lossy(&bytes[..bytes.len().min(20)]));+ }+ // more entries than allowed: the extra are ignored, with a note+ let many: Vec<Value> = (0..MAX_ENTRIES + 20).map(|i| good(&format!("m{i}"))).collect();+ let c = cat(many);+ assert_eq!(c.entries.len(), MAX_ENTRIES);+ assert!(c.errors.iter().any(|e| e.contains("first")));+ // file names from odd addresses are safe+ for url in ["https://x.org/a/b/c.onnx", "https://x.org/", "https://x.org/..%2f..%2fevil.onnx", "https://x.org/q.bin", "https://x.org/.onnx"] {+ let got = file_name_of(url);+ assert!(!got.contains(['/', '\\']) && got.ends_with(".onnx") && !got.starts_with('.'), "{url} -> {got}");+ }+ }+}diff --git a/crates/faces/src/graph.rs b/crates/faces/src/graph.rsindex 4ef4ec8d7..a5a22d262 100644--- a/crates/faces/src/graph.rs+++ b/crates/faces/src/graph.rs@@ -1,7 +1,7 @@//! A full, bounded read of a small ONNX model: its nodes, attributes and weights.//!-//! This is for the detector that ships inside LightCraft (a few hundred kilobytes), which [`crate::net`]-//! runs itself. Bigger models are never loaded this way. The bytes are still treated as hostile: the+//! The detector and installed recognisers share this reader and [`crate::net`]. Files are capped at+//! 512 MiB, with at most 64 million weight elements. The bytes are still treated as hostile: the//! number of nodes, weight tensors and elements is capped, every length is checked against its parent,//! and a weight blob whose size disagrees with its shape is an error.@@ -101,6 +101,9 @@ fn packed_varints(w: &mut Walker<Cursor<&[u8]>>, end: u64, out: &mut Vec<u64>) -return malformed("a list is too long");}out.push(w.varint()?);+ if w.pos()? > end {+ return malformed("a packed number runs past its field");+ }}Ok(())}@@ -125,9 +128,17 @@ fn attribute(w: &mut Walker<Cursor<&[u8]>>, end: u64) -> Result<(String, Attr)>// packed floatslet at = w.pos()?;let raw = w.bytes(at, e, (MAX_LIST * 4) as u64)?;+ if !raw.len().is_multiple_of(4) || floats.len().saturating_add(raw.len() / 4) > MAX_LIST {+ return malformed("an attribute float list is invalid or too long");+ }floats.extend(f32s(&raw));}- (7, Wire::Fixed32(v)) => floats.push(f32::from_bits(v)),+ (7, Wire::Fixed32(v)) => {+ if floats.len() >= MAX_LIST {+ return malformed("an attribute float list is too long");+ }+ floats.push(f32::from_bits(v));+ }(8, Wire::Len { end: e, .. }) => {packed_varints(w, e, &mut ints)?;w.seek_to(e)?;@@ -173,7 +184,16 @@ fn node(w: &mut Walker<Cursor<&[u8]>>, end: u64) -> Result<Node> {}(5, Wire::Len { end: e, .. }) => {let (k, v) = attribute(w, e)?;- n.attrs.insert(k, v);+ if k.is_empty() || n.attrs.insert(k, v).is_some() {+ return malformed("an attribute name is empty or repeated");+ }+ w.seek_to(e)?;+ }+ (7, Wire::Len { start, end: e }) => {+ let domain = w.string(start, e)?;+ if !domain.is_empty() && domain != "ai.onnx" {+ return malformed("custom operator domains are unsupported");+ }w.seek_to(e)?;}(_, Wire::Len { end: e, .. }) => w.seek_to(e)?,@@ -190,7 +210,8 @@ fn node(w: &mut Walker<Cursor<&[u8]>>, end: u64) -> Result<Node> {fn tensor(w: &mut Walker<Cursor<&[u8]>>, end: u64, budget: &mut usize) -> Result<(String, Option<Weight>)> {let (mut name, mut dtype) = (String::new(), 0u64);let mut dims: Vec<u64> = Vec::new();- let (mut raw, mut float_data, mut int_data): (Vec<u8>, Vec<f32>, Vec<u64>) = (Vec::new(), Vec::new(), Vec::new());+ let mut raw: Option<&[u8]> = None;+ let (mut float_data, mut int_data): (Vec<f32>, Vec<u64>) = (Vec::new(), Vec::new());let max_bytes = (*budget as u64).saturating_mul(8);while let Some((n, f)) = w.next(end)? {match (n, f) {@@ -203,8 +224,23 @@ fn tensor(w: &mut Walker<Cursor<&[u8]>>, end: u64, budget: &mut usize) -> Result(4, Wire::Len { end: e, .. }) => {let at = w.pos()?;let bytes = w.bytes(at, e, max_bytes)?;+ if !bytes.len().is_multiple_of(4) || float_data.len().saturating_add(bytes.len() / 4) > *budget {+ return malformed("invalid or excessive packed float weights");+ }float_data.extend(f32s(&bytes));}+ (4, Wire::Fixed32(v)) => {+ if float_data.len() >= *budget {+ return malformed("too many float weights");+ }+ float_data.push(f32::from_bits(v));+ }+ (7, Wire::Varint(v)) => {+ if int_data.len() >= *budget {+ return malformed("too many integer weights");+ }+ int_data.push(v);+ }(7, Wire::Len { end: e, .. }) => {packed_varints(w, e, &mut int_data)?;w.seek_to(e)?;@@ -214,8 +250,15 @@ fn tensor(w: &mut Walker<Cursor<&[u8]>>, end: u64, budget: &mut usize) -> Resultw.seek_to(e)?;}(9, Wire::Len { start, end: e }) => {- raw = w.bytes(start, e, max_bytes)?;+ if raw.is_some() || e.saturating_sub(start) > max_bytes {+ return malformed("repeated or excessive raw weights");+ }+ let from = usize::try_from(start).map_err(|_| ProbeError::Malformed("weight offset overflow"))?;+ let to = usize::try_from(e).map_err(|_| ProbeError::Malformed("weight offset overflow"))?;+ raw = Some((*w.r.get_ref()).get(from..to).ok_or(ProbeError::Malformed("short raw weights"))?);+ w.seek_to(e)?;}+ (13, Wire::Len { .. }) | (14, Wire::Varint(1..)) => return malformed("external tensor data is unsupported"),(_, Wire::Len { end: e, .. }) => w.seek_to(e)?,_ => {}}@@ -234,23 +277,43 @@ fn tensor(w: &mut Walker<Cursor<&[u8]>>, end: u64, budget: &mut usize) -> Resultreturn malformed("the weights are too large");}*budget -= elements;+ if raw.is_some() && (!float_data.is_empty() || !int_data.is_empty()) {+ return malformed("ambiguous raw and typed weights");+ }let data = match dtype {1 => {- let v = if raw.is_empty() { float_data } else { f32s(&raw) };+ let v = if let Some(raw) = raw {+ if elements.checked_mul(4) != Some(raw.len()) {+ return malformed("float weight bytes disagree with shape");+ }+ f32s(raw)+ } else {+ float_data+ };if v.len() != elements {return malformed("a weight tensor's size disagrees with its shape");}Some(Data::F32(v))}7 => {- let v = if raw.is_empty() { int_data.into_iter().map(|v| v as i64).collect() } else { i64s(&raw) };+ let v = if let Some(raw) = raw {+ if elements.checked_mul(8) != Some(raw.len()) {+ return malformed("integer weight bytes disagree with shape");+ }+ i64s(raw)+ } else {+ int_data.into_iter().map(|v| v as i64).collect()+ };if v.len() != elements {return malformed("a weight tensor's size disagrees with its shape");}Some(Data::I64(v))}- _ => None,+ _ => return malformed("only float32 and int64 initializers are supported"),};+ if name.is_empty() {+ return malformed("a weight name is empty");+ }Ok((name, data.map(|data| Weight { dims: shape, data })))}@@ -266,7 +329,16 @@ fn value_info(w: &mut Walker<Cursor<&[u8]>>, end: u64) -> Result<(String, Vec<us// TypeProto > tensor_type (1) > shape (2) > dim (1) > dim_value (1)let mut info = crate::onnx::TensorInfo { name: String::new(), elem_type: 0, shape: Vec::new() };crate::onnx::type_proto(w, e, &mut info)?;- shape = info.shape.iter().map(|d| if let crate::onnx::Dim::Fixed(v) = d { usize::try_from(*v).unwrap_or(0) } else { 0 }).collect();+ shape = info+ .shape+ .iter()+ .map(|d| match d {+ crate::onnx::Dim::Fixed(v) => {+ usize::try_from(*v).map_err(|_| ProbeError::Malformed("a declared dimension exceeds this platform"))+ }+ crate::onnx::Dim::Dynamic(_) => Ok(0),+ })+ .collect::<Result<_>>()?;w.seek_to(e)?;}(_, Wire::Len { end: e, .. }) => w.seek_to(e)?,@@ -279,9 +351,22 @@ fn value_info(w: &mut Walker<Cursor<&[u8]>>, end: u64) -> Result<(String, Vec<usOk((name, shape))}-/// Read a whole (small) ONNX model.+/// Read a bounded float32 ONNX inference model.pub fn load(bytes: &[u8]) -> Result<Graph> {let total = bytes.len() as u64;+ if total > 512 * 1024 * 1024 {+ return malformed("the model exceeds 512 MiB");+ }+ let info = crate::onnx::probe_reader(&mut Cursor::new(bytes))?;+ if info.opset.is_none_or(|v| !(10..=17).contains(&v)) {+ return malformed("only ONNX default-domain opsets 10 through 17 are supported");+ }+ if info.inputs.iter().chain(&info.outputs).any(|v| v.shape.iter().any(|d| matches!(d, crate::onnx::Dim::Fixed(0)))) {+ return malformed("a declared input/output has a zero dimension");+ }+ if info.inputs.iter().chain(&info.outputs).any(|v| v.elem_type != 1) {+ return malformed("model inputs and outputs must be float32");+ }if total == 0 {return malformed("the file is empty");}@@ -293,6 +378,9 @@ pub fn load(bytes: &[u8]) -> Result<Graph> {while let Some((n, f)) = w.next(total)? {match (n, f) {(7, Wire::Len { end, .. }) => {+ if saw_graph {+ return malformed("the model contains multiple graphs");+ }saw_graph = true;while let Some((gn, gf)) = w.next(end)? {match (gn, gf) {@@ -308,19 +396,28 @@ pub fn load(bytes: &[u8]) -> Result<Graph> {return malformed("too many weight tensors");}let (name, wt) = tensor(&mut w, e, &mut budget)?;- if let Some(wt) = wt {- weights.insert(name, wt);+ if let Some(wt) = wt+ && weights.insert(name, wt).is_some()+ {+ return malformed("a weight is defined twice");}w.seek_to(e)?;}(11, Wire::Len { end: e, .. }) => {+ if inputs.len() >= MAX_TENSORS {+ return malformed("too many graph inputs");+ }inputs.push(value_info(&mut w, e)?);w.seek_to(e)?;}(12, Wire::Len { end: e, .. }) => {+ if outputs.len() >= 64 {+ return malformed("too many graph outputs");+ }outputs.push(value_info(&mut w, e)?.0);w.seek_to(e)?;}+ (15, Wire::Len { .. }) => return malformed("sparse initializers are unsupported"),(_, Wire::Len { end: e, .. }) => w.seek_to(e)?,_ => {}}@@ -340,3 +437,6 @@ pub fn load(bytes: &[u8]) -> Result<Graph> {}Ok(Graph { nodes, weights, inputs, outputs })}++#[cfg(test)]+mod tests;diff --git a/crates/faces/src/graph/tests.rs b/crates/faces/src/graph/tests.rsnew file mode 100644index 000000000..38635459c--- /dev/null+++ b/crates/faces/src/graph/tests.rs@@ -0,0 +1,117 @@+use super::*;+use crate::synthetic::{len_field, value_info_bytes, varint, varint_field};+fn node() -> Vec<u8> {+ let mut n = Vec::new();+ for (id, v) in [(1, "x"), (2, "y"), (4, "Identity")] {+ len_field(id, v.as_bytes(), &mut n);+ }+ n+}+fn weight(raw: &[u8]) -> Vec<u8> {+ let mut t = Vec::new();+ varint_field(1, 1, &mut t);+ varint_field(2, 1, &mut t);+ len_field(8, b"w", &mut t);+ len_field(9, raw, &mut t);+ t+}+fn model(node: &[u8], weights: &[Vec<u8>]) -> Vec<u8> {+ let mut g = Vec::new();+ len_field(1, node, &mut g);+ for t in weights {+ len_field(5, t, &mut g);+ }+ for (id, name) in [(11, "x"), (12, "y")] {+ len_field(id, &value_info_bytes(name, 1, &[Ok(1), Ok(3), Ok(112), Ok(112)]), &mut g);+ }+ let mut ops = Vec::new();+ varint_field(2, 11, &mut ops);+ let mut out = Vec::new();+ len_field(7, &g, &mut out);+ len_field(8, &ops, &mut out);+ out+}+#[test]+fn raw_and_typed_weights_are_exact_not_truncated_or_ambiguous() {+ assert!(load(&model(&node(), &[weight(&1f32.to_le_bytes())])).is_ok());+ for raw in [vec![0; 3], vec![0; 5], vec![0; 8]] {+ assert!(load(&model(&node(), &[weight(&raw)])).is_err());+ }+ let mut t = weight(&1f32.to_le_bytes());+ varint(4 << 3 | 5, &mut t);+ t.extend_from_slice(&1f32.to_le_bytes());+ assert!(load(&model(&node(), &[t])).is_err());+ let mut t = weight(&1f32.to_le_bytes());+ len_field(9, &1f32.to_le_bytes(), &mut t);+ assert!(load(&model(&node(), &[t])).is_err());+ // The protobuf float_data form must work as well as raw_data.+ let mut t = Vec::new();+ varint_field(1, 1, &mut t);+ varint_field(2, 1, &mut t);+ len_field(8, b"w", &mut t);+ varint(4 << 3 | 5, &mut t);+ t.extend_from_slice(&1f32.to_le_bytes());+ assert!(load(&model(&node(), &[t])).is_ok());+}+#[test]+fn external_sparse_duplicate_custom_domain_and_wrong_dtype_are_errors() {+ let t = weight(&1f32.to_le_bytes());+ assert!(load(&model(&node(), &[t.clone(), t.clone()])).is_err());+ for field in [13, 14] {+ let mut bad = t.clone();+ if field == 13 {+ len_field(field, b"external", &mut bad);+ } else {+ varint_field(field, 1, &mut bad);+ }+ assert!(load(&model(&node(), &[bad])).is_err());+ }+ let mut bad = t.clone();+ varint_field(2, 10, &mut bad);+ assert!(load(&model(&node(), &[bad])).is_err());+ let mut n = node();+ len_field(7, b"vendor.example", &mut n);+ assert!(load(&model(&n, &[])).is_err());+ let mut attr = Vec::new();+ len_field(1, b"ratio", &mut attr);+ varint_field(20, 1, &mut attr);+ let mut n = node();+ len_field(5, &attr, &mut n);+ len_field(5, &attr, &mut n);+ assert!(load(&model(&n, &[])).is_err());+ let mut g = Vec::new();+ len_field(15, b"sparse", &mut g);+ let mut bad = model(&node(), &[]);+ len_field(7, &g, &mut bad);+ assert!(load(&bad).is_err());+}+#[test]+fn overflowing_and_cross_boundary_varints_long_strings_and_bad_shapes_are_errors() {+ assert!(load(&[0x08, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0x02]).is_err());+ let mut t = Vec::new();+ len_field(1, &[0x80], &mut t);+ varint_field(2, 1, &mut t);+ len_field(8, b"w", &mut t);+ assert!(load(&model(&node(), &[t])).is_err());+ let mut n = node();+ len_field(1, &vec![b'x'; 4097], &mut n);+ assert!(load(&model(&n, &[])).is_err());+ let mut n = node();+ len_field(1, &[0xff], &mut n);+ assert!(load(&model(&n, &[])).is_err());+ let mut t = Vec::new();+ varint_field(1, u64::MAX, &mut t);+ varint_field(1, 8, &mut t);+ varint_field(2, 1, &mut t);+ len_field(8, b"w", &mut t);+ assert!(load(&model(&node(), &[t])).is_err());+ let mut bad = model(&node(), &[]);+ let mut ops = Vec::new();+ varint_field(2, 999, &mut ops);+ len_field(8, &ops, &mut bad);+ assert!(load(&bad).is_err());+}+proptest::proptest! {+ #[test]+ fn arbitrary_graph_bytes_do_not_panic(bytes in proptest::collection::vec(proptest::num::u8::ANY,0..4096)) { let _=load(&bytes); }+}diff --git a/crates/faces/src/known.rs b/crates/faces/src/known.rsindex 973d6d494..f1ff541c7 100644--- a/crates/faces/src/known.rs+++ b/crates/faces/src/known.rs@@ -29,20 +29,24 @@ pub fn yunet() -> ModelManifest {"MIT",Commercial::Yes,"https://github.com/opencv/opencv_zoo/blob/main/models/face_detection_yunet/LICENSE",- "MIT licence, copyright Shiqi Yu.",+ "MIT licence, copyright Shiqi Yu. A separate opt-in download.",),source: Some("https://github.com/opencv/opencv_zoo/tree/main/models/face_detection_yunet".into()),sha256: Some(YUNET_SHA256.into()),size_bytes: Some(232_589),- provenance: "WIDER FACE, whose terms are not settled by the weights' MIT licence. A detector: it finds faces, it does not identify anyone."- .into(),+ provenance:+ "Trained on WIDER FACE, whose terms are not settled by the weights' MIT licence. A detector: it finds faces, it does not identify anyone."+ .into(),input: InputSpec { width: 640, height: 640, colour: Colour::Bgr, mean: [0.0; 3], std: [1.0; 3], resize: Resize::Letterbox },output: OutputSpec::Detector { decoder: "yunet-v2".into() },thresholds: Thresholds { match_cosine: None, score: Some(0.6), nms_iou: Some(0.3) },+ speed: None,}}-/// SFace 2021dec (OpenCV Zoo).+/// SFace 2021dec (OpenCV Zoo). Its match threshold is a starting point: on 755 named faces (busts, matched across+/// shots more than five seconds apart) 0.55 gave 97.5% right suggestions for 88% of faces; `faces.evaluate` checks it on+/// your own photos.pub fn sface() -> ModelManifest {ModelManifest {id: "sface-2021dec".into(),@@ -61,11 +65,13 @@ pub fn sface() -> ModelManifest {provenance: "Undocumented. The original SFace repository mentions CASIA-WebFace, VGGFace2 and MS1MV2.".into(),input: InputSpec { width: 112, height: 112, colour: Colour::Rgb, mean: [0.0; 3], std: [1.0; 3], resize: Resize::Stretch },output: OutputSpec::Embedding { dim: 128 },- thresholds: Thresholds::default(),+ thresholds: Thresholds { match_cosine: Some(0.55), ..Thresholds::default() },+ speed: Some(6.9),}}-/// AuraFace v1 `glintr100` (fal.ai).+/// AuraFace v1 `glintr100` (fal.ai). Starting-point threshold: on the same 755 faces 0.40 gave 96.6% right suggestions+/// for 56% of faces (0.30: 97.3% for 85%).pub fn auraface() -> ModelManifest {ModelManifest {id: "auraface-v1".into(),@@ -84,73 +90,74 @@ pub fn auraface() -> ModelManifest {provenance: "Undisclosed: \"a commercial dataset comprising face images from various sources\".".into(),input: InputSpec::default(),output: OutputSpec::Embedding { dim: 512 },- thresholds: Thresholds::default(),+ thresholds: Thresholds { match_cosine: Some(0.40), ..Thresholds::default() },+ speed: Some(1.0),}}-/// Where LightCraft can fetch a model from when the user asks: an address pinned to a commit of the model's own-/// repository, and the exact size and SHA-256 of the file (checked after the download, so the address is never-/// trusted on its own). The file is the last segment of the address, which is what the downloader needs: it fetches-/// `<mirror>/<file name>`, and here the mirror is everything before it.-#[derive(Clone, Copy, Debug, PartialEq, Eq)]-pub struct Download {- pub id: &'static str,- pub url: &'static str,- pub file_name: &'static str,- pub size_bytes: u64,- pub sha256: &'static str,+/// Every model we know by hash.+pub fn all() -> Vec<ModelManifest> {+ vec![yunet(), sface(), auraface()]}-impl Download {- /// The address without the file name (the downloader's "mirror").- pub fn base(&self) -> &'static str {- self.url.strip_suffix(self.file_name).map_or(self.url, |b| b.trim_end_matches('/'))- }+/// The known model whose file has this SHA-256 (lowercase hex).+pub fn lookup(sha256: &str) -> Option<ModelManifest> {+ all().into_iter().find(|m| m.sha256.as_deref() == Some(sha256))+}- /// "github.com": which site the file comes from, as the user is told.- pub fn host(&self) -> &'static str {- self.url.strip_prefix("https://").and_then(|r| r.split('/').next()).unwrap_or("")- }+/// Where the user's "Download" button fetches a model from. The URL is pinned to a commit of the model's own+/// repository, and what arrives is checked against the manifest's size and SHA-256 before anything uses it.+#[derive(Clone, Debug, PartialEq)]+pub struct Download {+ pub id: String,+ pub url: String,+ /// The name the file is kept under while it waits for the user to accept its terms.+ pub file_name: String,+ pub size_bytes: u64,+ pub sha256: String,}-/// Models LightCraft offers to download. A model marked non-commercial is never here: the user brings that file.-pub const SOURCES: &[Download] = &[- Download {- id: YUNET_ID,- url: "https://github.com/opencv/opencv_zoo/raw/25f423d0e04c31a17254620e58febd7386da523b/models/face_detection_yunet/face_detection_yunet_2023mar.onnx",- file_name: "face_detection_yunet_2023mar.onnx",- size_bytes: 232_589,- sha256: YUNET_SHA256,- },- Download {- id: "sface-2021dec",- url: "https://github.com/opencv/opencv_zoo/raw/25f423d0e04c31a17254620e58febd7386da523b/models/face_recognition_sface/face_recognition_sface_2021dec.onnx",- file_name: "face_recognition_sface_2021dec.onnx",- size_bytes: 38_696_353,- sha256: SFACE_SHA256,- },- Download {- id: "auraface-v1",- url: "https://huggingface.co/fal/AuraFace-v1/resolve/af6d057c9b0ec4071d4c49c80e3539258798b609/glintr100.onnx",- file_name: "glintr100.onnx",- size_bytes: 260_694_151,- sha256: AURAFACE_SHA256,- },+/// Pinned download addresses by model id. Only models whose terms allow us to point at them are here: a model+/// that is marked non-commercial is never offered for download (the user brings the file themselves).+const SOURCES: &[(&str, &str, &str)] = &[+ (+ YUNET_ID,+ "https://github.com/opencv/opencv_zoo/raw/25f423d0e04c31a17254620e58febd7386da523b/models/face_detection_yunet/face_detection_yunet_2023mar.onnx",+ "face_detection_yunet_2023mar.onnx",+ ),+ (+ "sface-2021dec",+ "https://github.com/opencv/opencv_zoo/raw/25f423d0e04c31a17254620e58febd7386da523b/models/face_recognition_sface/face_recognition_sface_2021dec.onnx",+ "face_recognition_sface_2021dec.onnx",+ ),+ ("auraface-v1", "https://huggingface.co/fal/AuraFace-v1/resolve/af6d057c9b0ec4071d4c49c80e3539258798b609/glintr100.onnx", "glintr100.onnx"),];/// How to download the model with this id, if LightCraft offers to.pub fn download(id: &str) -> Option<Download> {- SOURCES.iter().find(|d| d.id == id).copied()+ let (_, url, file_name) = SOURCES.iter().find(|(i, _, _)| *i == id)?;+ let m = all().into_iter().find(|m| m.id == id)?;+ if m.licence.commercial == Commercial::No {+ return None;+ }+ Some(Download { id: m.id, url: (*url).into(), file_name: (*file_name).into(), size_bytes: m.size_bytes?, sha256: m.sha256? })}-/// Every model we know by hash.-pub fn all() -> Vec<ModelManifest> {- vec![yunet(), sface(), auraface()]+/// "github.com" for a download address: what the user is told the file comes from.+pub fn host(url: &str) -> &str {+ url.strip_prefix("https://").and_then(|r| r.split('/').next()).unwrap_or("")}-/// The known model whose file has this SHA-256 (lowercase hex).-pub fn lookup(sha256: &str) -> Option<ModelManifest> {- all().into_iter().find(|m| m.sha256.as_deref() == Some(sha256))+impl Download {+ /// The address without the file name (the downloader's "mirror": it fetches `<mirror>/<file name>`).+ pub fn base(&self) -> &str {+ self.url.strip_suffix(self.file_name.as_str()).map_or(self.url.as_str(), |b| b.trim_end_matches('/'))+ }++ /// The site the file comes from, as the user is told.+ pub fn host(&self) -> &str {+ host(&self.url)+ }}#[cfg(test)]@@ -184,21 +191,27 @@ mod tests {#[test]fn downloads_are_pinned_checked_and_only_for_models_we_may_point_at() {- for d in SOURCES {- let m = all().into_iter().find(|m| m.id == d.id).unwrap_or_else(|| panic!("{} is not a known model", d.id));+ for m in all() {+ let Some(d) = download(&m.id) else { continue };assert!(d.url.starts_with("https://"), "{}: https only", d.id);assert!(!d.url.contains("/main/") && !d.url.contains("/master/"), "{}: pinned to a commit, not a branch", d.id);assert!(d.file_name.ends_with(".onnx") && !d.file_name.contains(['/', '\\']), "{}", d.file_name);assert!(d.url.ends_with(&format!("/{}", d.file_name)), "{}: the address ends in the file's name", d.id);assert!(d.base().starts_with("https://") && !d.base().ends_with('/') && !d.base().ends_with(".onnx"), "{}", d.base());- assert_eq!((Some(d.size_bytes), Some(d.sha256)), (m.size_bytes, m.sha256.as_deref()), "{}", d.id);+ assert_eq!((Some(d.size_bytes), Some(d.sha256.as_str())), (m.size_bytes, m.sha256.as_deref()), "{}", d.id);assert!(m.licence.commercial != Commercial::No, "{} is non-commercial and must not be offered", d.id);}- assert_eq!(download("yunet-2023mar").map(|d| d.host()), Some("github.com"));- assert_eq!(download("auraface-v1").map(|d| d.host()), Some("huggingface.co"));+ assert_eq!(download("yunet-2023mar").map(|d| d.host().to_string()).as_deref(), Some("github.com"));+ assert_eq!(download("sface-2021dec").map(|d| host(&d.url).to_string()).as_deref(), Some("github.com"));+ assert_eq!(download("auraface-v1").map(|d| host(&d.url).to_string()).as_deref(), Some("huggingface.co"));+ // unknown ids and path tricks have nothing to downloadfor id in ["nope", "", "../sface-2021dec", "SFACE-2021DEC"] {assert!(download(id).is_none(), "{id}");}+ // every address in the table belongs to a known model+ for (id, _, _) in SOURCES {+ assert!(all().iter().any(|m| m.id == *id), "{id}");+ }}#[test]diff --git a/crates/faces/src/lib.rs b/crates/faces/src/lib.rsindex aace35754..b4b7066d5 100644--- a/crates/faces/src/lib.rs+++ b/crates/faces/src/lib.rs@@ -4,17 +4,21 @@//! own can be one drop and one licence prompt.//!//! Everything here treats model files and manifests as hostile input: sizes are capped, numbers must be-//! finite and sane, and a malformed file is an error, never a panic. Model execution is not part of-//! this crate yet.+//! finite and sane, and a malformed file is an error, never a panic. Detection and recognition run+//! on the same checked, single-thread pure-Rust CPU interpreter.#![forbid(unsafe_code)]#![deny(clippy::unwrap_used, clippy::expect_used, clippy::panic, clippy::unimplemented, clippy::todo, clippy::unreachable)]+pub mod align;+pub mod catalog;pub mod graph;pub mod hash;pub mod known;pub mod manifest;+pub mod matching;pub mod net;pub mod onnx;+pub mod runtime;pub mod suggest;pub mod synthetic;pub mod yunet;diff --git a/crates/faces/src/manifest.rs b/crates/faces/src/manifest.rsindex 36f8878a1..d3d995012 100644--- a/crates/faces/src/manifest.rs+++ b/crates/faces/src/manifest.rs@@ -9,7 +9,7 @@ use serde::{Deserialize, Serialize};/// Largest `face-model.json` we read.pub const MAX_MANIFEST_BYTES: usize = 64 * 1024;/// Largest model file we accept (bytes).-pub const MAX_MODEL_BYTES: u64 = 4 * 1024 * 1024 * 1024;+pub const MAX_MODEL_BYTES: u64 = 512 * 1024 * 1024;#[derive(Clone, Copy, Debug, PartialEq, Eq, Serialize, Deserialize)]#[serde(rename_all = "camelCase")]@@ -129,6 +129,31 @@ pub struct ModelManifest {pub output: OutputSpec,#[serde(default)]pub thresholds: Thresholds,+ /// How fast it is as a multiple of a ResNet-100 model's speed (`1.0`): `2.0` is twice as fast. Shown instead of+ /// timings, which depend on the computer; leave it out when you have not measured it.+ #[serde(default, skip_serializing_if = "Option::is_none")]+ pub speed: Option<f32>,+}++/// The reference model the `speed` ratios are measured against.+pub const SPEED_REFERENCE: &str = "a ResNet-100 model";++/// A speed ratio in plain words, as times faster than the reference ("4.6× faster than a ResNet-100 model").+pub fn describe_speed(speed: f32) -> String {+ if !(speed.is_finite() && speed > 0.0) {+ return String::new();+ }+ let times = |x: f32| {+ let s = format!("{x:.1}");+ s.strip_suffix(".0").map_or(s.clone(), str::to_string)+ };+ if speed >= 1.15 {+ format!("{}× faster than {SPEED_REFERENCE}", times(speed))+ } else if speed > 0.87 {+ format!("Same speed as {SPEED_REFERENCE}")+ } else {+ format!("{}× slower than {SPEED_REFERENCE}", times(1.0 / speed))+ }}#[derive(Debug, PartialEq, thiserror::Error)]@@ -231,6 +256,7 @@ pub fn validate(m: &ModelManifest) -> Result<(), ManifestError> {ratio("thresholds.matchCosine", m.thresholds.match_cosine, -1.0, 1.0)?;ratio("thresholds.score", m.thresholds.score, 0.0, 1.0)?;ratio("thresholds.nmsIou", m.thresholds.nms_iou, 0.0, 1.0)?;+ ratio("speed", m.speed, 0.01, 1000.0)?;Ok(())}@@ -267,6 +293,19 @@ mod tests {input: InputSpec::default(),output: OutputSpec::Embedding { dim: 512 },thresholds: Thresholds { match_cosine: Some(0.4), ..Default::default() },+ speed: Some(1.7),+ }+ }++ #[test]+ fn speed_reads_as_times_faster_than_the_reference() {+ assert_eq!(describe_speed(7.0), "7× faster than a ResNet-100 model");+ assert_eq!(describe_speed(4.6), "4.6× faster than a ResNet-100 model");+ assert_eq!(describe_speed(1.7), "1.7× faster than a ResNet-100 model");+ assert_eq!(describe_speed(1.0), "Same speed as a ResNet-100 model");+ assert_eq!(describe_speed(0.5), "2× slower than a ResNet-100 model");+ for odd in [0.0, -1.0, f32::NAN, f32::INFINITY] {+ assert_eq!(describe_speed(odd), "", "{odd}");}}@@ -318,6 +357,9 @@ mod tests {),("nan threshold", Box::new(|m| m.thresholds.match_cosine = Some(f32::NAN))),("threshold range", Box::new(|m| m.thresholds.score = Some(1.5))),+ ("nan speed", Box::new(|m| m.speed = Some(f32::NAN))),+ ("zero speed", Box::new(|m| m.speed = Some(0.0))),+ ("huge speed", Box::new(|m| m.speed = Some(1e9))),];for (what, f) in cases {let mut m = good();diff --git a/crates/faces/src/matching.rs b/crates/faces/src/matching.rsnew file mode 100644index 000000000..de50ad3e4--- /dev/null+++ b/crates/faces/src/matching.rs@@ -0,0 +1,128 @@+//! Who does this face look like? Rank the people in a gallery of named faces by how close their faces are,+//! and suggest one only when the match is strong *and* clearly ahead of the runner-up.+//!+//! Embeddings are unit vectors, so closeness is the dot product (cosine similarity). A person's score is the+//! mean of their best few faces (one stray, wrongly named face cannot make a stranger look like them). The+//! result is only ever a suggestion for the user to confirm: it is better to leave a face unnamed than to+//! name it wrongly.++use std::cmp::Ordering;++/// How many of a person's best matching faces are averaged into their score.+pub const TOP_FACES: usize = 2;++#[derive(Clone, Debug, PartialEq)]+pub struct Ranked {+ pub name: String,+ pub score: f32,+ /// How many of their faces the score is based on.+ pub faces: usize,+}++#[derive(Clone, Debug, PartialEq)]+pub struct Suggestion {+ pub name: String,+ pub score: f32,+ /// The next best person, and how far behind.+ pub runner_up: Option<(String, f32)>,+}++/// Cosine similarity of two unit vectors. `None` when their lengths differ or either has a non-finite number.+pub fn cosine(a: &[f32], b: &[f32]) -> Option<f32> {+ if a.len() != b.len() || a.is_empty() {+ return None;+ }+ let d: f32 = a.iter().zip(b).map(|(x, y)| x * y).sum();+ d.is_finite().then_some(d)+}++/// Rank the people in `gallery` (a name and one face's embedding per entry; a person appears once per face),+/// best first. Names are compared case-insensitively and shown as first seen.+pub fn rank(query: &[f32], gallery: &[(&str, &[f32])]) -> Vec<Ranked> {+ let mut people: Vec<(String, String, Vec<f32>)> = Vec::new(); // (key, shown name, similarities)+ for (name, emb) in gallery {+ let Some(c) = cosine(query, emb) else { continue };+ let key = name.to_lowercase();+ match people.iter_mut().find(|p| p.0 == key) {+ Some(p) => p.2.push(c),+ None => people.push((key, (*name).to_string(), vec![c])),+ }+ }+ let mut out: Vec<Ranked> = people+ .into_iter()+ .map(|(_, name, mut sims)| {+ sims.sort_by(|a, b| b.partial_cmp(a).unwrap_or(Ordering::Equal));+ let top = sims.get(..sims.len().min(TOP_FACES)).unwrap_or(&[]);+ Ranked { name, score: top.iter().sum::<f32>() / top.len().max(1) as f32, faces: sims.len() }+ })+ .collect();+ out.sort_by(|a, b| b.score.partial_cmp(&a.score).unwrap_or(Ordering::Equal).then_with(|| a.name.cmp(&b.name)));+ out+}++/// The best person if they pass both bars: a score of at least `threshold`, and at least `margin` ahead of the+/// next person (when there is one).+pub fn suggest(ranked: &[Ranked], threshold: f32, margin: f32) -> Option<Suggestion> {+ let best = ranked.first()?;+ let next = ranked.get(1);+ if best.score.is_nan() || best.score < threshold || next.is_some_and(|n| best.score - n.score < margin) {+ return None;+ }+ Some(Suggestion { name: best.name.clone(), score: best.score, runner_up: next.map(|n| (n.name.clone(), n.score)) })+}++#[cfg(test)]+mod tests {+ use super::*;++ fn unit(v: &[f32]) -> Vec<f32> {+ let n = v.iter().map(|x| x * x).sum::<f32>().sqrt();+ v.iter().map(|x| x / n).collect()+ }++ #[test]+ fn people_are_ranked_by_their_closest_faces() {+ let (a1, a2, b1) = (unit(&[1.0, 0.1, 0.0]), unit(&[0.9, 0.2, 0.0]), unit(&[0.0, 1.0, 0.2]));+ let gallery: Vec<(&str, &[f32])> = vec![("Ann", &a1), ("ann", &a2), ("Bob", &b1)];+ let ranked = rank(&unit(&[1.0, 0.0, 0.0]), &gallery);+ assert_eq!(ranked.iter().map(|r| r.name.as_str()).collect::<Vec<_>>(), vec!["Ann", "Bob"], "case-insensitive, shown as first seen");+ assert_eq!(ranked[0].faces, 2);+ assert!(ranked[0].score > 0.95 && ranked[1].score < 0.2);+ // a stray, wrongly named face cannot carry a person: Cy has one far face and one good one+ let (c_good, c_bad) = (unit(&[1.0, 0.0, 0.0]), unit(&[0.0, 0.0, 1.0]));+ let g2: Vec<(&str, &[f32])> = vec![("Cy", &c_good), ("Cy", &c_bad), ("Di", &a1), ("Di", &a2)];+ let r2 = rank(&unit(&[1.0, 0.0, 0.0]), &g2);+ assert_eq!(r2[0].name, "Di", "{r2:?}");+ }++ #[test]+ fn a_suggestion_needs_a_strong_match_clearly_ahead() {+ let r = |name: &str, score: f32| Ranked { name: name.into(), score, faces: 1 };+ // strong and clear+ let s = suggest(&[r("Ann", 0.62), r("Bob", 0.20)], 0.45, 0.08).unwrap();+ assert_eq!((s.name.as_str(), s.runner_up.as_ref().map(|x| x.0.as_str())), ("Ann", Some("Bob")));+ // strong but too close to the runner-up: nobody+ assert!(suggest(&[r("Ann", 0.62), r("Bob", 0.58)], 0.45, 0.08).is_none());+ // clear but not strong enough+ assert!(suggest(&[r("Ann", 0.40), r("Bob", 0.10)], 0.45, 0.08).is_none());+ // a single person has no runner-up to be ahead of+ assert!(suggest(&[r("Ann", 0.5)], 0.45, 0.08).is_some());+ assert!(suggest(&[], 0.45, 0.08).is_none());+ assert!(suggest(&[r("Ann", f32::NAN)], 0.45, 0.08).is_none());+ }++ #[test]+ fn mismatched_or_non_finite_embeddings_are_skipped_not_panics() {+ assert!(cosine(&[1.0], &[1.0, 0.0]).is_none());+ assert!(cosine(&[], &[]).is_none());+ assert!(cosine(&[f32::NAN], &[1.0]).is_none());+ let good = unit(&[1.0, 0.0]);+ let bad = [f32::NAN, 0.0];+ let short = [1.0f32];+ let gallery: Vec<(&str, &[f32])> = vec![("Ann", &good), ("Nan", &bad), ("Short", &short)];+ let ranked = rank(&good, &gallery);+ assert_eq!(ranked.len(), 1);+ assert!(rank(&[], &gallery).is_empty());+ assert!(rank(&good, &[]).is_empty());+ }+}diff --git a/crates/faces/src/net.rs b/crates/faces/src/net.rsindex 2b6c3342c..6222314f2 100644--- a/crates/faces/src/net.rs+++ b/crates/faces/src/net.rs@@ -1,16 +1,18 @@//! A small CPU interpreter for the convolutional networks LightCraft bundles (the YuNet face detector).//!-//! It runs exactly the operators such a network uses (Conv, Relu, Sigmoid, MaxPool, Resize by whole-number-//! nearest-neighbour scales, Add, Transpose, Reshape) on float32 NCHW tensors with a batch of one, and refuses-//! anything else with an error, so an unknown model fails to load rather than computing nonsense. It is pure-//! Rust with no dependencies, and the inner loops are row-wise multiply-adds over slices that the compiler can-//! vectorise. Every shape, stride and size comes from the file, so all of them are checked.+//! It runs checked convolution, affine/activation, broadcast, pooling, matrix and shape operators on+//! float32 tensors. Unsupported semantics fail at load time. Dense convolutions use bounded im2col and+//! single-thread GEMM; depthwise convolutions use row-wise Rust loops. Every file-derived shape is checked.//!-//! This is not a general runtime: models users install run on `tract` (see the plan), not here.+//! The same checked float32 engine runs YuNet and installed SFace/AuraFace recognisers.+//! Unknown operators, attributes and shapes are rejected before recognition starts.use std::collections::HashMap;use std::rc::Rc;+mod kernels;+mod shape;+use crate::graph::{Data, Graph, Node};/// Elements of any one tensor the interpreter will allocate.@@ -43,7 +45,7 @@ pub struct Tensor {impl Tensor {pub fn new(shape: Vec<usize>, data: Vec<f32>) -> Result<Tensor> {let n = shape.iter().try_fold(1usize, |a, d| a.checked_mul(*d));- if n != Some(data.len()) {+ if shape.len() > 8 || shape.contains(&0) || n.is_none_or(|n| n > MAX_TENSOR) || n != Some(data.len()) || data.iter().any(|v| !v.is_finite()) {return shape_err(format!("{} values do not fill the shape {shape:?}", data.len()));}Ok(Tensor { shape, data })@@ -51,7 +53,12 @@ impl Tensor {fn zeros(shape: Vec<usize>) -> Result<Tensor> {match shape.iter().try_fold(1usize, |a, d| a.checked_mul(*d)) {- Some(n) if n <= MAX_TENSOR => Ok(Tensor { shape, data: vec![0.0; n] }),+ Some(n) if n > 0 && n <= MAX_TENSOR && shape.len() <= 8 && !shape.contains(&0) => {+ let mut data = Vec::new();+ data.try_reserve_exact(n).map_err(|_| NetError::Shape("not enough memory for a model tensor".into()))?;+ data.resize(n, 0.0);+ Ok(Tensor { shape, data })+ }_ => shape_err("a tensor would be too large"),}}@@ -62,59 +69,14 @@ pub struct Net {graph: Graph,}-/// Which operators and attribute combinations are supported, checked once when the network is built.+/// Validate semantics rather than silently ignoring unknown or mistyped attributes.fn check(node: &Node, g: &Graph) -> Result<()> {- let weight = |i: usize| node.inputs.get(i).and_then(|n| g.weights.get(n));- match node.op.as_str() {- "Conv" => {- let Some(w) = weight(1) else { return unsupported("a Conv without constant weights") };- if w.dims.len() != 4 || !matches!(w.data, Data::F32(_)) {- return unsupported("a Conv that is not a 2-D float convolution");- }- if node.ints("dilations").is_some_and(|d| d.iter().any(|v| *v != 1)) {- return unsupported("a dilated Conv");- }- if node.string("auto_pad").is_some_and(|s| s != "NOTSET") {- return unsupported("a Conv with auto_pad");- }- if node.inputs.get(2).is_some_and(|b| !b.is_empty() && !g.weights.contains_key(b)) {- return unsupported("a Conv with a computed bias");- }- }- "Relu" | "Sigmoid" | "Add" => {}- "MaxPool" => {- if node.int("ceil_mode").unwrap_or(0) != 0 || node.ints("kernel_shape").is_none_or(|k| k.len() != 2) {- return unsupported("this MaxPool");- }- if node.ints("dilations").is_some_and(|d| d.iter().any(|v| *v != 1)) {- return unsupported("a dilated MaxPool");- }- }- "Resize" => {- if node.string("mode").unwrap_or("nearest") != "nearest" {- return unsupported("a Resize that is not nearest-neighbour");- }- if !node.inputs.get(2).is_some_and(|s| g.weights.contains_key(s)) {- return unsupported("a Resize without constant scales");- }- }- "Transpose" => {- if node.ints("perm").is_none() {- return unsupported("a Transpose without a permutation");- }- }- "Reshape" => {- if !node.inputs.get(1).is_some_and(|s| matches!(g.weights.get(s).map(|w| &w.data), Some(Data::I64(_)))) {- return unsupported("a Reshape with a computed shape");- }- }- other => return unsupported(format!("the operator `{other}`")),- }- Ok(())+ shape::check(node, g)}impl Net {pub fn new(graph: Graph) -> Result<Net> {+ shape::check_graph(&graph)?;for n in &graph.nodes {check(n, &graph)?;}@@ -126,8 +88,18 @@ impl Net {self.graph.inputs.first().map(|(_, s)| s.as_slice())}+ /// Infer and check every intermediate without allocating its pixel data.+ /// Recognition calls this at load time with the manifest's fixed input.+ pub fn validate_input(&self, shape: &[usize]) -> Result<Vec<Vec<usize>>> {+ shape::infer(&self.graph, shape)+ }+/// Run the network on `input` (its single input); returns each output by name.pub fn run(&self, input: Tensor) -> Result<Vec<(String, Tensor)>> {+ shape::tensor(&input.shape, input.data.len(), &input.data)?;+ self.validate_input(&input.shape)?;+ let mut scratch = kernels::Scratch::default();+ let profile = std::env::var_os("LIGHTCRAFT_PROFILE").is_some();let g = &self.graph;let Some((input_name, _)) = g.inputs.first() else { return shape_err("the model has no input") };// when each value is last read, so big intermediates are dropped as soon as nothing needs them@@ -145,19 +117,38 @@ impl Net {for (i, node) in g.nodes.iter().enumerate() {let get = |idx: usize| -> Result<Rc<Tensor>> {let name = node.inputs.get(idx).ok_or_else(|| NetError::Shape(format!("{} is missing an input", node.op)))?;+ if let Some(value) = values.get(name) {+ return Ok(value.clone());+ }+ if let Some(crate::graph::Weight { dims, data: Data::F32(data) }) = g.weights.get(name) {+ return Ok(Rc::new(Tensor::new(dims.clone(), data.clone())?));+ }values.get(name).cloned().ok_or_else(|| NetError::Shape(format!("`{name}` is read before it is computed")))};+ let started = web_time::Instant::now();let out = match node.op.as_str() {- "Conv" => conv(node, &*get(0)?, g)?,+ "Conv" => kernels::conv(node, &*get(0)?, g, &mut scratch)?,"Relu" => map(&*get(0)?, |v| v.max(0.0)),"Sigmoid" => map(&*get(0)?, |v| 1.0 / (1.0 + (-v).exp())),- "Add" => add(&*get(0)?, &*get(1)?)?,+ "Add" | "Sub" | "Mul" => kernels::binary(node.op.as_str(), &*get(0)?, &*get(1)?)?,+ "PRelu" => kernels::prelu(node, &*get(0)?, g)?,+ "BatchNormalization" => kernels::batch_norm(node, &*get(0)?, g)?,+ "Gemm" => kernels::gemm(node, &*get(0)?, g)?,+ "Flatten" => kernels::flatten(node, &*get(0)?)?,+ "Dropout" | "Identity" => (*get(0)?).clone(),+ "GlobalAveragePool" => kernels::global_average(&*get(0)?)?,"MaxPool" => max_pool(node, &*get(0)?)?,"Resize" => resize(node, &*get(0)?, g)?,"Transpose" => transpose(node, &*get(0)?)?,"Reshape" => reshape(node, &*get(0)?, g)?,other => return unsupported(format!("the operator `{other}`")),};+ if out.data.iter().any(|v| !v.is_finite()) {+ return shape_err(format!("{} produced non-finite values", node.op));+ }+ if profile {+ eprintln!("[faces-cpu] layer {i} {} {:?}: {:.3} ms", node.op, out.shape, started.elapsed().as_secs_f64() * 1000.0);+ }let Some(name) = node.outputs.first() else { return shape_err("a node has no output") };values.insert(name.clone(), Rc::new(out));for name in &node.inputs {@@ -169,8 +160,9 @@ impl Net {g.outputs.iter().map(|name| {- let t = values.get(name).ok_or_else(|| NetError::Shape(format!("output `{name}` was never computed")))?;- Ok((name.clone(), (**t).clone()))+ let t = values.remove(name).ok_or_else(|| NetError::Shape(format!("output `{name}` was never computed")))?;+ let t = Rc::try_unwrap(t).map_err(|_| NetError::Shape(format!("output `{name}` is still shared")))?;+ Ok((name.clone(), t))}).collect()}@@ -180,13 +172,6 @@ fn map(x: &Tensor, f: impl Fn(f32) -> f32) -> Tensor {Tensor { shape: x.shape.clone(), data: x.data.iter().map(|v| f(*v)).collect() }}-fn add(a: &Tensor, b: &Tensor) -> Result<Tensor> {- if a.shape != b.shape {- return unsupported("an Add that broadcasts");- }- Ok(Tensor { shape: a.shape.clone(), data: a.data.iter().zip(&b.data).map(|(x, y)| x + y).collect() })-}-/// `[1, c, h, w]` → `(c, h, w)`.fn chw(x: &Tensor) -> Result<(usize, usize, usize)> {match x.shape.as_slice() {@@ -218,7 +203,7 @@ fn pair(node: &Node, name: &str, default: usize) -> Result<(usize, usize)> {}}-fn conv(node: &Node, x: &Tensor, g: &Graph) -> Result<Tensor> {+fn conv_direct(node: &Node, x: &Tensor, g: &Graph) -> Result<Tensor> {let (cin, h, w) = chw(x)?;let weight = node.inputs.get(1).and_then(|n| g.weights.get(n)).ok_or_else(|| NetError::Shape("a Conv lost its weights".into()))?;let Data::F32(wd) = &weight.data else { return unsupported("non-float Conv weights") };@@ -238,7 +223,8 @@ fn conv(node: &Node, x: &Tensor, g: &Graph) -> Result<Tensor> {};let (sh, sw) = pair(node, "strides", 1)?;let (pt, pl, pb, pr) = pads(node)?;- let (ph, pw) = (h + pt + pb, w + pl + pr);+ let ph = h.checked_add(pt).and_then(|v| v.checked_add(pb)).ok_or_else(|| NetError::Shape("Conv padding overflow".into()))?;+ let pw = w.checked_add(pl).and_then(|v| v.checked_add(pr)).ok_or_else(|| NetError::Shape("Conv padding overflow".into()))?;if ph < kh || pw < kw {return shape_err("the Conv kernel is larger than its input");}@@ -353,9 +339,10 @@ fn resize(node: &Node, x: &Tensor, g: &Graph) -> Result<Tensor> {}fn transpose(node: &Node, x: &Tensor) -> Result<Tensor> {+ let default_perm: Vec<i64> = (0..x.shape.len() as i64).rev().collect();let perm: Vec<usize> = node.ints("perm")- .unwrap_or(&[])+ .unwrap_or(&default_perm).iter().map(|p| usize::try_from(*p).map_err(|_| NetError::Shape("a negative permutation".into()))).collect::<Result<_>>()?;@@ -368,7 +355,8 @@ fn transpose(node: &Node, x: &Tensor) -> Result<Tensor> {// strides of the input, in elementslet mut in_strides = vec![1usize; rank];for i in (0..rank.saturating_sub(1)).rev() {- in_strides[i] = in_strides.get(i + 1).copied().unwrap_or(1) * x.shape.get(i + 1).copied().unwrap_or(1);+ let v = in_strides.get(i + 1).copied().unwrap_or(1) * x.shape.get(i + 1).copied().unwrap_or(1);+ *in_strides.get_mut(i).ok_or_else(|| NetError::Shape("missing transpose stride".into()))? = v;}let strides_for_out: Vec<usize> = perm.iter().filter_map(|p| in_strides.get(*p).copied()).collect();let mut out = Tensor::zeros(out_shape.clone())?;@@ -577,3 +565,6 @@ mod tests {}}}++#[cfg(test)]+mod recognition_tests;diff --git a/crates/faces/src/net/kernels.rs b/crates/faces/src/net/kernels.rsnew file mode 100644index 000000000..73cc72b93--- /dev/null+++ b/crates/faces/src/net/kernels.rs@@ -0,0 +1,306 @@+//! Safe CPU kernels shared by detection and recognition. No inner thread pool.+use super::*;+use ndarray::{ArrayView2, ArrayViewMut2, linalg::general_mat_mul};+const BLOCK: usize = 256;+#[derive(Default)]+pub(super) struct Scratch {+ lower: Vec<f32>,+ product: Vec<f32>,+}+fn storage(v: &mut Vec<f32>, n: usize) -> Result<&mut [f32]> {+ if n > MAX_TENSOR {+ return shape_err("convolution scratch exceeds the limit");+ }+ if v.len() < n {+ v.try_reserve_exact(n - v.len()).map_err(|_| NetError::Shape("not enough memory for convolution scratch".into()))?;+ v.resize(n, 0.0);+ }+ v.get_mut(..n).ok_or_else(|| NetError::Shape("missing convolution scratch".into()))+}+/// The safe matrix views assert lengths, so validate exact slices before creating any view.+fn multiply(w: &[f32], x: &[f32], y: &mut [f32], m: usize, k: usize, n: usize) -> Result<()> {+ if m == 0 || k == 0 || n == 0 || m.checked_mul(k) != Some(w.len()) || k.checked_mul(n) != Some(x.len()) || m.checked_mul(n) != Some(y.len()) {+ return shape_err("invalid GEMM dimensions");+ }+ let a = ArrayView2::from_shape((m, k), w).map_err(|e| NetError::Shape(e.to_string()))?;+ let b = ArrayView2::from_shape((k, n), x).map_err(|e| NetError::Shape(e.to_string()))?;+ let mut c = ArrayViewMut2::from_shape((m, n), y).map_err(|e| NetError::Shape(e.to_string()))?;+ general_mat_mul(1.0, &a, &b, 0.0, &mut c);+ Ok(())+}+/// For equal-width, unit-stride maps a shifted plane is one contiguous copy. Only horizontal+/// border stripes need masking; clipping the flat source range also removes vertical padding.+fn shifted_plane(plane: &[f32], width: usize, ky: usize, kx: usize, pt: usize, pl: usize, start: usize, row: &mut [f32]) -> Result<()> {+ let end = start + row.len();+ let delta = (ky as i128 - pt as i128) * width as i128 + kx as i128 - pl as i128;+ let lo = (-delta).clamp(start as i128, end as i128);+ let hi = (plane.len() as i128 - delta).clamp(start as i128, end as i128);+ row.fill(0.0);+ if lo < hi {+ let from = usize::try_from(lo + delta).map_err(|_| NetError::Shape("shifted source overflow".into()))?;+ let at = usize::try_from(lo - start as i128).map_err(|_| NetError::Shape("shifted destination overflow".into()))?;+ let len = usize::try_from(hi - lo).map_err(|_| NetError::Shape("shifted length overflow".into()))?;+ row.get_mut(at..at + len)+ .ok_or_else(|| NetError::Shape("short shifted destination".into()))?+ .copy_from_slice(plane.get(from..from + len).ok_or_else(|| NetError::Shape("short shifted source".into()))?);+ }+ let left = pl.saturating_sub(kx).min(width);+ let right = (width + pl).saturating_sub(kx).min(width);+ if left > 0 || right < width {+ for y in start / width..=(end - 1) / width {+ for (a, b) in [(y * width, y * width + left), (y * width + right, (y + 1) * width)] {+ let lo = a.max(start);+ let hi = b.min(end);+ if lo < hi {+ row.get_mut(lo - start..hi - start).ok_or_else(|| NetError::Shape("short border stripe".into()))?.fill(0.0);+ }+ }+ }+ }+ Ok(())+}+pub(super) fn conv(n: &Node, x: &Tensor, g: &Graph, scratch: &mut Scratch) -> Result<Tensor> {+ let (cin, h, w) = chw(x)?;+ let wt = shape::weight(n, g, 1)?;+ let wd = shape::floats(n, g, 1)?;+ let [cout, cg, kh, kw] = wt.dims.as_slice() else {+ return shape_err("invalid convolution weights");+ };+ let (cout, cg, kh, kw) = (*cout, *cg, *kh, *kw);+ let groups = usize::try_from(n.int("group").unwrap_or(1)).map_err(|_| NetError::Shape("invalid group".into()))?;+ if cg == 1 {+ return conv_direct(n, x, g);+ }+ let (sh, sw) = pair(n, "strides", 1)?;+ let (pt, pl, pb, pr) = pads(n)?;+ let ph = h.checked_add(pt).and_then(|v| v.checked_add(pb)).ok_or_else(|| NetError::Shape("padding overflow".into()))?;+ let pw = w.checked_add(pl).and_then(|v| v.checked_add(pr)).ok_or_else(|| NetError::Shape("padding overflow".into()))?;+ if ph < kh || pw < kw || groups == 0 || cin != groups * cg {+ return shape_err("invalid convolution shape");+ }+ let (oh, ow) = ((ph - kh) / sh + 1, (pw - kw) / sw + 1);+ let mut out = Tensor::zeros(vec![1, cout, oh, ow])?;+ let pixels = oh * ow;+ let k = cg * kh * kw;+ let m = cout / groups;+ let bias = if n.inputs.get(2).is_some_and(|s| !s.is_empty()) { Some(shape::floats(n, g, 2)?) } else { None };+ for group in 0..groups {+ let weights = wd.get(group * m * k..(group + 1) * m * k).ok_or_else(|| NetError::Shape("short convolution weights".into()))?;+ // A pointwise convolution is already a matrix: no lowering, copy or spatial blocking needed.+ if kh == 1 && kw == 1 && sh == 1 && sw == 1 && (pt, pl, pb, pr) == (0, 0, 0, 0) {+ let input = x.data.get(group * cg * pixels..(group + 1) * cg * pixels).ok_or_else(|| NetError::Shape("short pointwise input".into()))?;+ let output =+ out.data.get_mut(group * m * pixels..(group + 1) * m * pixels).ok_or_else(|| NetError::Shape("short pointwise output".into()))?;+ multiply(weights, input, output, m, cg, pixels)?;+ for (co, plane) in output.chunks_exact_mut(pixels).enumerate() {+ let b = bias.and_then(|v| v.get(group * m + co)).copied().unwrap_or(0.0);+ if b != 0.0 {+ for v in plane {+ *v += b;+ }+ }+ }+ continue;+ }+ for start in (0..pixels).step_by(BLOCK) {+ let len = BLOCK.min(pixels - start);+ let lower = storage(&mut scratch.lower, k * len)?;+ for cl in 0..cg {+ let plane = x+ .data+ .get((group * cg + cl) * h * w..(group * cg + cl + 1) * h * w)+ .ok_or_else(|| NetError::Shape("short convolution input".into()))?;+ for ky in 0..kh {+ for kx in 0..kw {+ let row = lower+ .get_mut((cl * kh * kw + ky * kw + kx) * len..(cl * kh * kw + ky * kw + kx + 1) * len)+ .ok_or_else(|| NetError::Shape("short im2col row".into()))?;+ if sh == 1 && sw == 1 && ow == w {+ shifted_plane(+ plane,+ w,+ ky,+ kx,+ pt,+ pl,+ start,+ row.get_mut(..len).ok_or_else(|| NetError::Shape("short lower panel".into()))?,+ )?;+ continue;+ }+ let mut at = 0;+ while at < len {+ let p = start + at;+ let (oy, ox) = (p / ow, p % ow);+ let span = (ow - ox).min(len - at);+ if let Some(yy) = (oy * sh + ky).checked_sub(pt).filter(|&v| v < h) {+ let lo = ox.max(pl.saturating_sub(kx).div_ceil(sw));+ let hi = (ox + span).min((w + pl).saturating_sub(kx).div_ceil(sw));+ if lo < hi {+ let from = yy * w + lo * sw + kx - pl;+ let to = at + lo - ox;+ row.get_mut(at..to).ok_or_else(|| NetError::Shape("short im2col padding".into()))?.fill(0.0);+ let src = plane+ .get(from..from + (hi - lo - 1) * sw + 1)+ .ok_or_else(|| NetError::Shape("short im2col source row".into()))?;+ let dst = row.get_mut(to..to + hi - lo).ok_or_else(|| NetError::Shape("short im2col row".into()))?;+ if sw == 1 {+ dst.copy_from_slice(src);+ } else {+ for (o, &v) in dst.iter_mut().zip(src.iter().step_by(sw)) {+ *o = v;+ }+ }+ row.get_mut(to + hi - lo..at + span).ok_or_else(|| NetError::Shape("short im2col padding".into()))?.fill(0.0);+ } else {+ row.get_mut(at..at + span).ok_or_else(|| NetError::Shape("short im2col row".into()))?.fill(0.0);+ }+ } else {+ row.get_mut(at..at + span).ok_or_else(|| NetError::Shape("short im2col row".into()))?.fill(0.0);+ }+ at += span;+ }+ }+ }+ }+ let product = storage(&mut scratch.product, m * len)?;+ multiply(weights, lower, product, m, k, len)?;+ for (co, row) in product.chunks_exact(len).enumerate() {+ let channel = group * m + co;+ let b = bias.and_then(|v| v.get(channel)).copied().unwrap_or(0.0);+ let dest = out+ .data+ .get_mut(channel * pixels + start..channel * pixels + start + len)+ .ok_or_else(|| NetError::Shape("short convolution output".into()))?;+ for (to, &value) in dest.iter_mut().zip(row) {+ *to = value + b;+ }+ }+ }+ }+ Ok(out)+}+/// Flat offset under ONNX's trailing-dimension broadcasting; shapes are checked before execution.+fn broadcast_offset(mut at: usize, out: &[usize], input: &[usize]) -> usize {+ let mut offset = 0;+ let mut stride = 1;+ for (axis, &size) in out.iter().enumerate().rev() {+ let coordinate = at % size;+ at /= size;+ if let Some(&d) = axis.checked_sub(out.len() - input.len()).and_then(|i| input.get(i)) {+ if d != 1 {+ offset += coordinate * stride;+ }+ stride *= d;+ }+ }+ offset+}+pub(super) fn binary(op: &str, a: &Tensor, b: &Tensor) -> Result<Tensor> {+ let dims = shape::broadcast(&a.shape, &b.shape)?;+ let mut out = Tensor::zeros(dims)?;+ let f = |x: f32, y: f32| match op {+ "Add" => x + y,+ "Sub" => x - y,+ _ => x * y,+ };+ if a.shape == b.shape {+ for ((o, &x), &y) in out.data.iter_mut().zip(&a.data).zip(&b.data) {+ *o = f(x, y);+ }+ } else if b.data.len() == 1 && a.shape == out.shape {+ let y = *b.data.first().ok_or_else(|| NetError::Shape("empty scalar".into()))?;+ for (o, &x) in out.data.iter_mut().zip(&a.data) {+ *o = f(x, y);+ }+ } else {+ for (i, o) in out.data.iter_mut().enumerate() {+ let x = a.data.get(broadcast_offset(i, &out.shape, &a.shape)).ok_or_else(|| NetError::Shape("bad broadcast A".into()))?;+ let y = b.data.get(broadcast_offset(i, &out.shape, &b.shape)).ok_or_else(|| NetError::Shape("bad broadcast B".into()))?;+ *o = f(*x, *y);+ }+ }+ Ok(out)+}+pub(super) fn prelu(n: &Node, x: &Tensor, g: &Graph) -> Result<Tensor> {+ let wt = shape::weight(n, g, 1)?;+ let slope = shape::floats(n, g, 1)?;+ let mut out = Tensor::zeros(x.shape.clone())?;+ let channels = x.shape.get(1).copied().unwrap_or(1);+ let mut expanded = vec![1; x.shape.len().saturating_sub(wt.dims.len())];+ expanded.extend_from_slice(&wt.dims);+ let channel_slope = expanded.iter().enumerate().all(|(i, &d)| d == 1 || i == 1) && slope.len() == channels;+ if channel_slope || slope.len() == 1 {+ let plane = shape::elements(x.shape.get(2..).unwrap_or(&[]))?;+ for (c, (src, dst)) in x.data.chunks_exact(plane).zip(out.data.chunks_exact_mut(plane)).enumerate() {+ let alpha = *slope.get(if slope.len() == 1 { 0 } else { c % channels }).ok_or_else(|| NetError::Shape("short PRelu slope".into()))?;+ for (to, &v) in dst.iter_mut().zip(src) {+ *to = if v < 0.0 { v * alpha } else { v };+ }+ }+ } else {+ for (i, (o, &v)) in out.data.iter_mut().zip(&x.data).enumerate() {+ let alpha = *slope.get(broadcast_offset(i, &x.shape, &wt.dims)).ok_or_else(|| NetError::Shape("bad PRelu broadcast".into()))?;+ *o = if v < 0.0 { v * alpha } else { v };+ }+ }+ Ok(out)+}+pub(super) fn batch_norm(n: &Node, x: &Tensor, g: &Graph) -> Result<Tensor> {+ let channels = *x.shape.get(1).ok_or_else(|| NetError::Shape("BatchNormalization rank".into()))?;+ let (scale, bias, mean, var) = (shape::floats(n, g, 1)?, shape::floats(n, g, 2)?, shape::floats(n, g, 3)?, shape::floats(n, g, 4)?);+ let eps = shape::float(n, "epsilon", 1e-5);+ let plane = shape::elements(x.shape.get(2..).unwrap_or(&[]))?;+ let mut out = Tensor::zeros(x.shape.clone())?;+ for (j, (src, dst)) in x.data.chunks_exact(plane).zip(out.data.chunks_exact_mut(plane)).enumerate() {+ let c = j % channels;+ let coefficient = *scale.get(c).ok_or_else(|| NetError::Shape("short BN scale".into()))?+ / (var.get(c).copied().ok_or_else(|| NetError::Shape("short BN variance".into()))? + eps).sqrt();+ let offset = bias.get(c).copied().ok_or_else(|| NetError::Shape("short BN bias".into()))?+ - mean.get(c).copied().ok_or_else(|| NetError::Shape("short BN mean".into()))? * coefficient;+ for (to, &value) in dst.iter_mut().zip(src) {+ *to = value * coefficient + offset;+ }+ }+ Ok(out)+}+pub(super) fn gemm(n: &Node, a: &Tensor, g: &Graph) -> Result<Tensor> {+ let wt = shape::weight(n, g, 1)?;+ let b = shape::floats(n, g, 1)?;+ let [ar, ac] = a.shape.as_slice() else {+ return shape_err("Gemm A is not a matrix");+ };+ let [br, bc] = wt.dims.as_slice() else {+ return shape_err("Gemm B is not a matrix");+ };+ if ar.checked_mul(*ac) != Some(a.data.len()) || br.checked_mul(*bc) != Some(b.len()) {+ return shape_err("invalid matrix buffers");+ }+ let av = ArrayView2::from_shape((*ar, *ac), a.data.as_slice()).map_err(|e| NetError::Shape(e.to_string()))?;+ let bv = ArrayView2::from_shape((*br, *bc), b).map_err(|e| NetError::Shape(e.to_string()))?;+ let av = if n.int("transA").unwrap_or(0) != 0 { av.reversed_axes() } else { av };+ let bv = if n.int("transB").unwrap_or(0) != 0 { bv.reversed_axes() } else { bv };+ if av.ncols() != bv.nrows() {+ return shape_err("incompatible matrices");+ }+ let mut out = Tensor::zeros(vec![av.nrows(), bv.ncols()])?;+ let mut view = ArrayViewMut2::from_shape((av.nrows(), bv.ncols()), out.data.as_mut_slice()).map_err(|e| NetError::Shape(e.to_string()))?;+ general_mat_mul(shape::float(n, "alpha", 1.0), &av, &bv, 0.0, &mut view);+ if n.inputs.get(2).is_some_and(|s| !s.is_empty()) {+ let wt = shape::weight(n, g, 2)?;+ let bias = shape::floats(n, g, 2)?;+ let beta = shape::float(n, "beta", 1.0);+ for (i, to) in out.data.iter_mut().enumerate() {+ *to += beta * bias.get(broadcast_offset(i, &out.shape, &wt.dims)).copied().ok_or_else(|| NetError::Shape("short matrix bias".into()))?;+ }+ }+ Ok(out)+}+pub(super) fn flatten(n: &Node, x: &Tensor) -> Result<Tensor> {+ let axis = shape::axis(n, x.shape.len())?;+ Tensor::new(vec![shape::elements(x.shape.get(..axis).unwrap_or(&[]))?, shape::elements(x.shape.get(axis..).unwrap_or(&[]))?], x.data.clone())+}+pub(super) fn global_average(x: &Tensor) -> Result<Tensor> {+ let (c, h, w) = chw(x)?;+ Tensor::new(vec![1, c, 1, 1], x.data.chunks_exact(h * w).map(|v| v.iter().sum::<f32>() / (h * w) as f32).collect())+}diff --git a/crates/faces/src/net/recognition_tests.rs b/crates/faces/src/net/recognition_tests.rsnew file mode 100644index 000000000..e245bb333--- /dev/null+++ b/crates/faces/src/net/recognition_tests.rs@@ -0,0 +1,337 @@+use super::*;+use crate::graph::{Attr, Weight};+fn weight(shape: &[usize], values: Vec<f32>) -> Weight {+ Weight { dims: shape.to_vec(), data: Data::F32(values) }+}+fn graph(op: &str, extra: &[&str], weights: Vec<(&str, Weight)>, attrs: Vec<(&str, Attr)>) -> Graph {+ Graph {+ nodes: vec![Node {+ op: op.into(),+ inputs: std::iter::once("x").chain(extra.iter().copied()).map(str::to_string).collect(),+ outputs: vec!["y".into()],+ attrs: attrs.into_iter().map(|(s, a)| (s.into(), a)).collect(),+ }],+ weights: weights.into_iter().map(|(s, w)| (s.into(), w)).collect(),+ inputs: vec![("x".into(), vec![1, 3, 2, 2])],+ outputs: vec!["y".into()],+ }+}+fn execute(g: Graph, shape: &[usize], values: Vec<f32>) -> Tensor {+ Net::new(g).unwrap().run(Tensor::new(shape.to_vec(), values).unwrap()).unwrap().remove(0).1+}+#[test]+fn batch_norm_matches_the_inference_formula_for_images_and_matrices() {+ for shape in [vec![1, 3, 2, 5], vec![2, 3]] {+ let values: Vec<_> = (0..shape.iter().product()).map(|i| i as f32 * 0.07 - 0.8).collect();+ let g = graph(+ "BatchNormalization",+ &["scale", "bias", "mean", "var"],+ vec![+ ("scale", weight(&[3], vec![0.7, -1.1, 2.3])),+ ("bias", weight(&[3], vec![-0.4, 0.6, 0.2])),+ ("mean", weight(&[3], vec![0.5, -0.3, 0.9])),+ ("var", weight(&[3], vec![0.2, 0.0, 1.4])),+ ],+ vec![("epsilon", Attr::Float(1e-3))],+ );+ let out = execute(g.clone(), &shape, values.clone());+ let spatial: usize = shape.get(2..).unwrap_or(&[]).iter().product();+ for (i, (&value, &got)) in values.iter().zip(&out.data).enumerate() {+ let c = (i / spatial) % 3;+ let [s, b, m, v] = ["scale", "bias", "mean", "var"].map(|name| {+ let Data::F32(v) = &g.weights[name].data else { panic!() };+ v[c]+ });+ let want = (value - m) * s / (v + 1e-3).sqrt() + b;+ assert!((got - want).abs() < 1e-5);+ }+ }+}+#[test]+fn prelu_obeys_trailing_broadcasting_including_spatial_slopes() {+ for (shape, slopes) in+ [(vec![3, 1, 1], vec![0.1, 0.2, 0.3]), (vec![1, 3, 1, 1], vec![0.1, 0.2, 0.3]), (vec![1], vec![0.3]), (vec![2, 2], vec![0.1, 0.2, 0.3, 0.4])]+ {+ let values: Vec<_> = (0..12).map(|i| i as f32 - 6.0).collect();+ let out = execute(graph("PRelu", &["s"], vec![("s", weight(&shape, slopes.clone()))], vec![]), &[1, 3, 2, 2], values.clone());+ for (i, (&v, &got)) in values.iter().zip(&out.data).enumerate() {+ let alpha = if slopes.len() == 1 {+ slopes[0]+ } else if slopes.len() == 3 {+ slopes[i / 4]+ } else {+ slopes[i % 4]+ };+ assert_eq!(got, if v < 0.0 { v * alpha } else { v });+ }+ }+}+#[test]+fn gemm_transpositions_scalars_vectors_matrices_and_optional_bias() {+ for ta in [0, 1] {+ for tb in [0, 1] {+ for bias_shape in [vec![], vec![2], vec![1, 2], vec![2, 1], vec![2, 2]] {+ let (ash, bsh) = (if ta == 0 { vec![2, 3] } else { vec![3, 2] }, if tb == 0 { vec![3, 2] } else { vec![2, 3] });+ let a: Vec<_> = (0..6).map(|i| i as f32 * 0.4 - 0.8).collect();+ let b: Vec<_> = (0..6).map(|i| i as f32 * 0.1 - 0.3).collect();+ let bias: Vec<_> = (0..bias_shape.iter().product()).map(|i| i as f32 * 0.25 + 0.3).collect();+ let g = graph(+ "Gemm",+ &["w", "b"],+ vec![("w", weight(&bsh, b.clone())), ("b", weight(&bias_shape, bias.clone()))],+ vec![("transA", Attr::Int(ta)), ("transB", Attr::Int(tb)), ("alpha", Attr::Float(0.7)), ("beta", Attr::Float(-0.4))],+ );+ let out = execute(g, &ash, a.clone());+ assert_eq!(out.shape, vec![2, 2]);+ for i in 0..2 {+ for j in 0..2 {+ let sum = (0..3)+ .map(|k| a[if ta == 0 { i * 3 + k } else { k * 2 + i }] * b[if tb == 0 { k * 2 + j } else { j * 3 + k }])+ .sum::<f32>();+ let off = match bias_shape.as_slice() {+ [] => 0,+ [2] | [1, 2] => j,+ [2, 1] => i,+ _ => i * 2 + j,+ };+ assert!((out.data[i * 2 + j] - (0.7 * sum - 0.4 * bias[off])).abs() < 1e-6);+ }+ }+ }+ }+ }+ let out = execute(graph("Gemm", &["w"], vec![("w", weight(&[2, 1], vec![2.0, 3.0]))], vec![]), &[1, 2], vec![4.0, 5.0]);+ assert_eq!(out.data, vec![23.0]);+}+#[test]+fn arithmetic_broadcasts_and_keeps_operand_order() {+ for op in ["Add", "Sub", "Mul"] {+ let out =+ execute(graph(op, &["w"], vec![("w", weight(&[3, 1], vec![0.5, 1.5, 2.5]))], vec![]), &[1, 3, 2], vec![1.0, 2.0, 3.0, 4.0, 5.0, 6.0]);+ for (i, &got) in out.data.iter().enumerate() {+ let x = (i + 1) as f32;+ let y = (i / 2) as f32 + 0.5;+ let want = match op {+ "Add" => x + y,+ "Sub" => x - y,+ _ => x * y,+ };+ assert_eq!(got, want);+ }+ }+}+#[test]+fn flatten_negative_axis_and_default_transpose_are_correct() {+ let out = execute(graph("Flatten", &[], vec![], vec![("axis", Attr::Int(-1))]), &[1, 3, 2, 5], (0..30).map(|i| i as f32).collect());+ assert_eq!(out.shape, vec![6, 5]);+ let out = execute(graph("Transpose", &[], vec![], vec![]), &[2, 3], (0..6).map(|i| i as f32).collect());+ assert_eq!(out.shape, vec![3, 2]);+ assert_eq!(out.data, vec![0.0, 3.0, 1.0, 4.0, 2.0, 5.0]);+}+#[test]+fn all_recognition_operators_run_on_seeded_synthetic_networks() {+ let input = Tensor::new(vec![1, 3, 112, 112], (0..3 * 112 * 112).map(|i| (i * 37 % 256) as f32).collect()).unwrap();+ for seed in 0..20 {+ let net = Net::new(crate::graph::load(&crate::synthetic::recognition_model(seed)).unwrap()).unwrap();+ let a = net.run(input.clone()).unwrap();+ let b = net.run(input.clone()).unwrap();+ assert_eq!(a, b);+ assert_eq!(a[0].1.shape, vec![1, 8]);+ }+}+#[test]+fn hostile_attributes_weights_shapes_and_public_tensors_return_errors() {+ for (name, attr) in [+ ("strides", Attr::Ints(vec![0, 1])),+ ("pads", Attr::Ints(vec![i64::MAX; 4])),+ ("group", Attr::Int(0)),+ ("kernel_shape", Attr::Ints(vec![7, 7])),+ ("strides", Attr::Float(1.0)),+ ("new_attribute", Attr::Int(1)),+ ] {+ let g = graph("Conv", &["w"], vec![("w", weight(&[3, 3, 1, 1], vec![0.1; 9]))], vec![(name, attr)]);+ let result = Net::new(g).and_then(|n| n.validate_input(&[1, 3, 3, 3]));+ assert!(result.is_err(), "{name}");+ }+ for op in ["Loop", "MatMul", "If", "Concat", "Unknown"] {+ assert!(Net::new(graph(op, &[], vec![], vec![])).is_err());+ }+ for value in [f32::NAN, f32::INFINITY] {+ assert!(Net::new(graph("PRelu", &["w"], vec![("w", weight(&[1], vec![value]))], vec![])).is_err());+ }+ let net = Net::new(graph("Relu", &[], vec![], vec![])).unwrap();+ for input in [+ Tensor { shape: vec![usize::MAX, 2], data: vec![] },+ Tensor { shape: vec![0, 3], data: vec![] },+ Tensor { shape: vec![1], data: vec![f32::NAN] },+ Tensor { shape: vec![1; 9], data: vec![1.0] },+ Tensor { shape: vec![2], data: vec![1.0] },+ ] {+ assert!(net.run(input).is_err());+ }+ let mut g = graph("Relu", &[], vec![], vec![]);+ g.nodes.push(g.nodes[0].clone());+ assert!(Net::new(g).is_err());+ let mut g = graph("Relu", &[], vec![], vec![]);+ g.nodes[0].inputs[0] = "future".into();+ assert!(Net::new(g).is_err());+}+#[test]+fn malformed_normalization_prelu_matrix_dropout_and_axes_are_errors() {+ let bn = || {+ graph(+ "BatchNormalization",+ &["s", "b", "m", "v"],+ vec![+ ("s", weight(&[3], vec![1.0; 3])),+ ("b", weight(&[3], vec![0.0; 3])),+ ("m", weight(&[3], vec![0.0; 3])),+ ("v", weight(&[3], vec![1.0; 3])),+ ],+ vec![],+ )+ };+ for mutation in 0..4 {+ let mut g = bn();+ match mutation {+ 0 => {+ g.nodes[0].attrs.insert("training_mode".into(), Attr::Int(1));+ }+ 1 => {+ g.nodes[0].attrs.insert("epsilon".into(), Attr::Float(0.0));+ }+ 2 => {+ g.weights.insert("v".into(), weight(&[3], vec![-1.0; 3]));+ }+ _ => {+ g.weights.insert("b".into(), weight(&[1, 3], vec![0.0; 3]));+ }+ };+ assert!(Net::new(g).and_then(|n| n.validate_input(&[1, 3, 2, 2])).is_err());+ }+ for g in [+ graph("PRelu", &["s"], vec![("s", weight(&[3], vec![0.1; 3]))], vec![]),+ graph("Gemm", &["w"], vec![("w", weight(&[4, 2], vec![0.1; 8]))], vec![]),+ graph("Flatten", &[], vec![], vec![("axis", Attr::Int(i64::MIN))]),+ graph("Dropout", &[], vec![], vec![("ratio", Attr::Float(1.0))]),+ graph("Transpose", &[], vec![], vec![("perm", Attr::Ints(vec![0, 1, 1, 3]))]),+ ] {+ assert!(Net::new(g).and_then(|n| n.validate_input(&[1, 3, 2, 2])).is_err());+ }+}+#[test]+fn arithmetic_overflow_is_reported_and_next_run_succeeds() {+ let net = Net::new(graph("Mul", &["w"], vec![("w", weight(&[1], vec![f32::MAX]))], vec![])).unwrap();+ assert!(net.run(Tensor::new(vec![1], vec![2.0]).unwrap()).is_err());+ assert!(net.run(Tensor::new(vec![1], vec![0.0]).unwrap()).is_ok());+}++#[test]+fn multiple_batches_prelu_keeps_channels_independent() {+ let out = execute(graph("PRelu", &["s"], vec![("s", weight(&[3, 1, 1], vec![0.1, 0.2, 0.3]))], vec![]), &[2, 3, 2, 2], vec![-1.0; 24]);+ for (i, &v) in out.data.iter().enumerate() {+ assert_eq!(v, -[0.1, 0.2, 0.3][i / 4 % 3]);+ }+}+#[test]+fn integer_activations_and_constant_outputs_fail_at_load() {+ let integer = Weight { dims: vec![1], data: Data::I64(vec![1]) };+ assert!(Net::new(graph("Add", &["w"], vec![("w", integer.clone())], vec![])).is_err());+ let mut g = graph("Relu", &[], vec![("w", integer)], vec![]);+ g.outputs = vec!["w".into()];+ assert!(Net::new(g).is_err());+}+#[test]+fn normalisation_coefficient_and_working_memory_overflows_fail_before_running() {+ let g = graph(+ "BatchNormalization",+ &["s", "b", "m", "v"],+ vec![+ ("s", weight(&[3], vec![f32::MAX; 3])),+ ("b", weight(&[3], vec![0.0; 3])),+ ("m", weight(&[3], vec![f32::MAX; 3])),+ ("v", weight(&[3], vec![0.0; 3])),+ ],+ vec![],+ );+ assert!(Net::new(g).unwrap().validate_input(&[1, 3, 2, 2]).is_err());+ let g = graph("Conv", &["w"], vec![("w", weight(&[4096, 3, 1, 1], vec![0.1; 4096 * 3]))], vec![]);+ assert!(Net::new(g).unwrap().validate_input(&[1, 3, 4096, 4096]).is_err());+}++#[test]+fn empty_broadcast_operands_are_rejected_before_allocation() {+ for op in ["Add", "Sub", "Mul", "PRelu"] {+ let g = graph(op, &["w"], vec![("w", weight(&[0, 3, 1, 1], vec![]))], vec![]);+ assert!(Net::new(g).and_then(|n| n.validate_input(&[1, 3, 2, 2])).is_err());+ }+}++#[test]+fn odd_maps_cross_panel_boundaries_and_reused_padding_is_zero() {+ for (c, co, groups) in [(3, 11, 1), (6, 10, 2)] {+ let weights: Vec<_> = (0..co * (c / groups) * 9).map(|i| (i as f32 % 19.0 - 9.0) * 0.013).collect();+ let g = graph(+ "Conv",+ &["w", "b"],+ vec![("w", weight(&[co, c / groups, 3, 3], weights)), ("b", weight(&[co], vec![0.1; co]))],+ vec![("group", Attr::Int(groups as i64)), ("strides", Attr::Ints(vec![1, 1])), ("pads", Attr::Ints(vec![2, 1, 0, 1]))],+ );+ let mut scratch = kernels::Scratch::default();+ for zero in [false, true, false] {+ let input = Tensor::new(vec![1, c, 23, 29], (0..c * 23 * 29).map(|i| if zero { 0.0 } else { (i as f32 % 29.0 - 14.0) * 0.12 }).collect())+ .unwrap();+ let got = kernels::conv(&g.nodes[0], &input, &g, &mut scratch).unwrap();+ let want = conv_direct(&g.nodes[0], &input, &g).unwrap();+ assert_eq!(got.shape, want.shape);+ for (&a, &b) in got.data.iter().zip(&want.data) {+ assert!((a - b).abs() < 1e-5, "{a} vs {b}");+ }+ }+ }+}++#[test]+fn retained_convolution_scratch_counts_towards_the_working_limit() {+ let network = |output_channels: usize| {+ let pads = vec![("pads", Attr::Ints(vec![1, 0, 0, 0]))];+ let mut g = graph("Conv", &["w"], vec![("w", weight(&[2, 130_000, 2, 1], vec![0.0; 2 * 130_000 * 2]))], pads.clone());+ g.inputs[0].1 = vec![1, 130_000, 2, 256];+ g.weights.insert("v".into(), weight(&[output_channels, 2, 2, 1], vec![0.0; output_channels * 4]));+ g.nodes.push(Node {+ op: "Conv".into(),+ inputs: vec!["y".into(), "v".into()],+ outputs: vec!["z".into()],+ attrs: pads.into_iter().map(|(s, a)| (s.into(), a)).collect(),+ });+ g.outputs = vec!["z".into()];+ Net::new(g).unwrap()+ };+ // The first layer retains a large im2col buffer; the second grows the product buffer.+ // Both layers individually fit, but keeping both scratch peaks would exceed 512 MiB.+ let shape = [1, 130_000, 2, 256];+ assert!(network(131_072).validate_input(&shape).is_err());+ assert!(network(65_536).validate_input(&shape).is_ok());+}++#[test]+fn constant_input_copies_count_towards_the_working_limit() {+ let network = |retained_outputs: usize| {+ let dims = [1, 1, 2048, 2048];+ let mut g = graph("Identity", &[], vec![("w", weight(&dims, vec![0.0; 2048 * 2048]))], vec![]);+ g.inputs[0].1 = dims.to_vec();+ g.nodes.clear();+ g.outputs.clear();+ for i in 0..retained_outputs {+ let name = format!("retained-{i}");+ g.nodes.push(Node { op: "Identity".into(), inputs: vec!["x".into()], outputs: vec![name.clone()], attrs: HashMap::new() });+ g.outputs.push(name);+ }+ g.nodes.push(Node { op: "Add".into(), inputs: vec!["x".into(), "w".into()], outputs: vec!["z".into()], attrs: HashMap::new() });+ g.outputs.push("z".into());+ Net::new(g).unwrap()+ };+ // Infer only: no pixel tensors are allocated. The constant's temporary copy must be counted.+ assert!(network(30).validate_input(&[1, 1, 2048, 2048]).is_err());+ assert!(network(29).validate_input(&[1, 1, 2048, 2048]).is_ok());+}diff --git a/crates/faces/src/net/shape.rs b/crates/faces/src/net/shape.rsnew file mode 100644index 000000000..d76271bc7--- /dev/null+++ b/crates/faces/src/net/shape.rs@@ -0,0 +1,496 @@+//! Bounded, allocation-free (apart from small shape lists) graph checking.+use super::*;+use crate::graph::{Attr, Weight};++pub(super) fn elements(s: &[usize]) -> Result<usize> {+ if s.len() > 8 || s.contains(&0) {+ return shape_err("a tensor has a zero dimension or excessive rank");+ }+ s.iter()+ .try_fold(1usize, |n, &d| n.checked_mul(d))+ .filter(|&n| n <= MAX_TENSOR)+ .ok_or_else(|| NetError::Shape("a tensor exceeds the element limit".into()))+}+pub(super) fn tensor(s: &[usize], n: usize, data: &[f32]) -> Result<()> {+ if elements(s)? != n || data.iter().any(|v| !v.is_finite()) {+ return shape_err("invalid tensor size or non-finite values");+ }+ Ok(())+}+pub(super) fn float(n: &Node, name: &str, default: f32) -> f32 {+ match n.attrs.get(name) {+ Some(Attr::Float(v)) => *v,+ _ => default,+ }+}+pub(super) fn weight<'a>(n: &Node, g: &'a Graph, i: usize) -> Result<&'a Weight> {+ n.inputs.get(i).and_then(|s| g.weights.get(s)).ok_or_else(|| NetError::Unsupported(format!("{} needs constant input {i}", n.op)))+}+pub(super) fn floats<'a>(n: &Node, g: &'a Graph, i: usize) -> Result<&'a [f32]> {+ match &weight(n, g, i)?.data {+ Data::F32(v) => Ok(v),+ _ => unsupported(format!("{} needs float32 weights", n.op)),+ }+}+pub(super) fn check_graph(g: &Graph) -> Result<()> {+ if g.inputs.len() != 1 || g.outputs.is_empty() || g.outputs.len() > 64 || g.nodes.len() > 20_000 || g.weights.len() > 20_000 {+ return unsupported("one input and at most 64 outputs are required");+ }+ let mut known = std::collections::HashSet::new();+ let mut total_weights = 0usize;+ for (name, w) in &g.weights {+ let count = w.dims.iter().try_fold(1usize, |n, &d| n.checked_mul(d));+ let len = match &w.data {+ Data::F32(v) => {+ if v.iter().any(|v| !v.is_finite()) {+ return shape_err("non-finite weights");+ }+ v.len()+ }+ Data::I64(v) => v.len(),+ };+ if name.is_empty()+ || name.len() > 4096+ || w.dims.len() > 8+ || w.dims.iter().any(|&d| d > MAX_TENSOR)+ || count != Some(len)+ || len > MAX_TENSOR+ {+ return shape_err("invalid weight name, shape or size");+ }+ total_weights = total_weights+ .checked_add(len)+ .filter(|&v| v <= MAX_TENSOR)+ .ok_or_else(|| NetError::Shape("combined weights exceed 64 Mi elements".into()))?;+ known.insert(name.as_str());+ }+ let input = g.inputs.first().ok_or_else(|| NetError::Shape("missing input".into()))?;+ if input.0.is_empty() || !known.insert(&input.0) {+ return shape_err("invalid or repeated input name");+ }+ for n in &g.nodes {+ if n.inputs.len() > 64 || n.outputs.len() != 1 || n.outputs.first().is_none_or(|s| s.is_empty() || s.len() > 4096) {+ return unsupported(format!("{} requires one named output", n.op));+ }+ if n.inputs.iter().any(|s| !s.is_empty() && !known.contains(s.as_str())) {+ return shape_err(format!("{} reads an unknown or forward value", n.op));+ }+ if !known.insert(n.outputs.first().ok_or_else(|| NetError::Shape("missing output".into()))?) {+ return shape_err("a graph value is defined twice");+ }+ }+ if g.outputs.iter().any(|s| g.weights.contains_key(s)) {+ return unsupported("constant graph outputs");+ }+ let mut output_names = std::collections::HashSet::new();+ if g.outputs.iter().any(|s| !known.contains(s.as_str()) || !output_names.insert(s)) {+ return shape_err("unknown or repeated graph output");+ }+ Ok(())+}+pub(super) fn check(n: &Node, g: &Graph) -> Result<()> {+ let (min, max, attrs): (usize, usize, &[(&str, u8)]) = match n.op.as_str() {+ "Conv" => (2, 3, &[("group", 0), ("kernel_shape", 2), ("strides", 2), ("pads", 2), ("dilations", 2), ("auto_pad", 3)]),+ "BatchNormalization" => (5, 5, &[("epsilon", 1), ("momentum", 1), ("training_mode", 0)]),+ "PRelu" | "Add" | "Sub" | "Mul" => (2, 2, &[]),+ "Relu" | "Sigmoid" | "GlobalAveragePool" | "Identity" => (1, 1, &[]),+ "Dropout" => (1, 1, &[("ratio", 1), ("seed", 0)]),+ "Flatten" => (1, 1, &[("axis", 0)]),+ "Gemm" => (2, 3, &[("alpha", 1), ("beta", 1), ("transA", 0), ("transB", 0)]),+ "MaxPool" => {+ (1, 1, &[("kernel_shape", 2), ("strides", 2), ("pads", 2), ("dilations", 2), ("ceil_mode", 0), ("auto_pad", 3), ("storage_order", 0)])+ }+ "Resize" => (+ 3,+ 4,+ &[+ ("mode", 3),+ ("coordinate_transformation_mode", 3),+ ("nearest_mode", 3),+ ("cubic_coeff_a", 1),+ ("exclude_outside", 0),+ ("extrapolation_value", 1),+ ],+ ),+ "Transpose" => (1, 1, &[("perm", 2)]),+ "Reshape" => (2, 2, &[("allowzero", 0)]),+ op => return unsupported(format!("the operator `{op}`")),+ };+ if !(min..=max).contains(&n.inputs.len()) || n.inputs.first().is_none_or(String::is_empty) {+ return shape_err(format!("{} has invalid inputs", n.op));+ }+ for (name, value) in &n.attrs {+ let actual = match value {+ Attr::Int(_) => 0,+ Attr::Float(v) if v.is_finite() => 1,+ Attr::Ints(_) => 2,+ Attr::Str(_) => 3,+ _ => 255,+ };+ if attrs.iter().find(|(key, _)| *key == name).is_none_or(|(_, kind)| *kind != actual) {+ return unsupported(format!("{} attribute `{name}` (or its type)", n.op));+ }+ }+ for (i, name) in n.inputs.iter().enumerate() {+ if let Some(w) = g.weights.get(name)+ && matches!(w.data, Data::I64(_))+ && !(n.op == "Reshape" && i == 1)+ {+ return unsupported(format!("{} needs float32 input {i}", n.op));+ }+ }+ match n.op.as_str() {+ "Conv" => {+ let w = weight(n, g, 1)?;+ floats(n, g, 1)?;+ if w.dims.len() != 4 || w.dims.contains(&0) {+ return shape_err("Conv needs nonempty 2-D weights");+ }+ if n.ints("kernel_shape").is_some_and(|v| {+ v.iter().map(|&d| usize::try_from(d).ok()).collect::<Option<Vec<_>>>().as_deref() != Some(w.dims.get(2..).unwrap_or(&[]))+ }) {+ return shape_err("Conv kernel_shape differs from its weights");+ }+ if n.ints("dilations").is_some_and(|d| d != [1, 1]) || n.string("auto_pad").is_some_and(|s| s != "NOTSET") {+ return unsupported("Conv dilation or automatic padding");+ }+ let (sh, sw) = pair(n, "strides", 1)?;+ let (pt, pl, pb, pr) = pads(n)?;+ if [sh, sw, pt, pl, pb, pr].iter().any(|&v| v > 65536) {+ return shape_err("Conv stride or padding exceeds 65536");+ }+ if n.int("group").unwrap_or(1) < 1 {+ return shape_err("Conv group must be positive");+ }+ if n.inputs.get(2).is_some_and(|s| !s.is_empty()) {+ floats(n, g, 2)?;+ }+ }+ "BatchNormalization" => {+ if n.int("training_mode").unwrap_or(0) != 0 {+ return unsupported("training BatchNormalization");+ }+ if float(n, "epsilon", 1e-5) <= 0.0 {+ return shape_err("BatchNormalization epsilon must be positive");+ }+ for i in 1..5 {+ floats(n, g, i)?;+ }+ }+ "PRelu" => {+ floats(n, g, 1)?;+ }+ "Gemm" => {+ floats(n, g, 1)?;+ if n.inputs.get(2).is_some_and(|s| !s.is_empty()) {+ floats(n, g, 2)?;+ }+ }+ "Dropout" => {+ if !(0.0..1.0).contains(&float(n, "ratio", 0.5)) {+ return shape_err("Dropout ratio is invalid");+ }+ }+ "MaxPool" => {+ pair(n, "kernel_shape", 1)?;+ pair(n, "strides", 1)?;+ if n.ints("kernel_shape").is_none()+ || n.int("ceil_mode").unwrap_or(0) != 0+ || n.int("storage_order").unwrap_or(0) != 0+ || n.ints("dilations").is_some_and(|v| v != [1, 1])+ || pads(n)? != (0, 0, 0, 0)+ || n.string("auto_pad").is_some_and(|s| s != "NOTSET")+ {+ return unsupported("this MaxPool padding, dilation or ceil mode");+ }+ }+ "Resize" => {+ floats(n, g, 2)?;+ if n.string("mode").unwrap_or("nearest") != "nearest" || n.inputs.get(3).is_some_and(|s| !s.is_empty()) {+ return unsupported("Resize requires nearest constant scales");+ }+ if n.int("exclude_outside").unwrap_or(0) != 0 || float(n, "extrapolation_value", 0.0) != 0.0 {+ return unsupported("Resize excludes or extrapolates outside values");+ }+ let coord = n.string("coordinate_transformation_mode").unwrap_or("half_pixel");+ let nearest = n.string("nearest_mode").unwrap_or("round_prefer_floor");+ if !matches!((coord, nearest), ("asymmetric", "floor") | ("half_pixel", "round_prefer_floor")) {+ return unsupported("this Resize coordinate/nearest mode");+ }+ if n.inputs.get(1).is_some_and(|s| !s.is_empty()) && !floats(n, g, 1)?.is_empty() {+ return unsupported("Resize with an ROI");+ }+ }+ "Reshape" => {+ if n.int("allowzero").unwrap_or(0) != 0 {+ return unsupported("Reshape allowzero");+ }+ if !matches!(weight(n, g, 1)?.data, Data::I64(_)) {+ return unsupported("Reshape needs a constant int64 shape");+ }+ }+ _ => {}+ }+ Ok(())+}+pub(super) fn broadcast(a: &[usize], b: &[usize]) -> Result<Vec<usize>> {+ elements(a)?;+ elements(b)?;+ let rank = a.len().max(b.len());+ let mut out = vec![1; rank];+ for (i, o) in out.iter_mut().enumerate() {+ let aa = i.checked_sub(rank - a.len()).and_then(|j| a.get(j)).copied().unwrap_or(1);+ let bb = i.checked_sub(rank - b.len()).and_then(|j| b.get(j)).copied().unwrap_or(1);+ if aa != bb && aa != 1 && bb != 1 {+ return shape_err("incompatible broadcast shapes");+ }+ *o = aa.max(bb);+ }+ elements(&out)?;+ Ok(out)+}+pub(super) fn axis(n: &Node, rank: usize) -> Result<usize> {+ let raw = n.int("axis").unwrap_or(1);+ let raw = if raw < 0 { raw.checked_add(rank as i64).ok_or_else(|| NetError::Shape("axis overflow".into()))? } else { raw };+ usize::try_from(raw).ok().filter(|&a| a <= rank).ok_or_else(|| NetError::Shape("Flatten axis is outside its rank".into()))+}+fn image(s: &[usize]) -> Result<(usize, usize, usize)> {+ match s {+ [1, c, h, w] => Ok((*c, *h, *w)),+ _ => unsupported("a spatial operator needs [1,C,H,W]"),+ }+}+pub(super) fn infer(g: &Graph, input: &[usize]) -> Result<Vec<Vec<usize>>> {+ elements(input)?;+ let mut shapes: HashMap<&str, Vec<usize>> = g.weights.iter().map(|(n, w)| (n.as_str(), w.dims.clone())).collect();+ shapes.insert(g.inputs.first().ok_or_else(|| NetError::Shape("no input".into()))?.0.as_str(), input.to_vec());+ let mut live = elements(input)?;+ // Scratch grows independently and remains allocated across all subsequent layers.+ let mut scratch_lower = 0;+ let mut scratch_product = 0;+ let mut last = HashMap::new();+ for (i, n) in g.nodes.iter().enumerate() {+ for s in &n.inputs {+ last.insert(s.as_str(), i);+ }+ }+ for s in &g.outputs {+ last.insert(s.as_str(), usize::MAX);+ }+ for (i, n) in g.nodes.iter().enumerate() {+ let a = n.inputs.first().and_then(|s| shapes.get(s.as_str())).ok_or_else(|| NetError::Shape("missing input shape".into()))?;+ elements(a)?;+ let out = match n.op.as_str() {+ "Conv" => {+ let (cin, h, w) = image(a)?;+ let dims = &weight(n, g, 1)?.dims;+ let [co, cg, kh, kw] = dims.as_slice() else {+ return shape_err("Conv dimensions");+ };+ let groups = usize::try_from(n.int("group").unwrap_or(1)).map_err(|_| NetError::Shape("Conv group".into()))?;+ if groups.checked_mul(*cg) != Some(cin) || !co.is_multiple_of(groups) {+ return shape_err("Conv channels do not match weights/groups");+ }+ if n.inputs.get(2).is_some_and(|s| !s.is_empty()) && weight(n, g, 2)?.dims != [*co] {+ return shape_err("Conv bias is not a channel vector");+ }+ let (sh, sw) = pair(n, "strides", 1)?;+ let (pt, pl, pb, pr) = pads(n)?;+ let padded = |size: usize, p: usize, q: usize, k: usize, s: usize| {+ size.checked_add(p)+ .and_then(|v| v.checked_add(q))+ .and_then(|v| v.checked_sub(k))+ .and_then(|v| (v / s).checked_add(1))+ .ok_or_else(|| NetError::Shape("Conv padding overflow or oversized kernel".into()))+ };+ vec![1, *co, padded(h, pt, pb, *kh, sh)?, padded(w, pl, pr, *kw, sw)?]+ }+ "BatchNormalization" => {+ let channels = *a.get(1).ok_or_else(|| NetError::Shape("BatchNormalization rank must be at least two".into()))?;+ for j in 1..5 {+ if weight(n, g, j)?.dims != [channels] {+ return shape_err("BatchNormalization parameters must be channel vectors");+ }+ }+ if floats(n, g, 4)?.iter().any(|v| *v < 0.0) {+ return shape_err("BatchNormalization has negative variance");+ }+ let eps = float(n, "epsilon", 1e-5);+ let (s, b, m, v) = (floats(n, g, 1)?, floats(n, g, 2)?, floats(n, g, 3)?, floats(n, g, 4)?);+ for (((&s, &b), &m), &v) in s.iter().zip(b).zip(m).zip(v) {+ let variance = v + eps;+ let coefficient = s / variance.sqrt();+ let offset = b - m * coefficient;+ if !variance.is_finite() || !coefficient.is_finite() || !offset.is_finite() {+ return shape_err("BatchNormalization coefficients overflow");+ }+ }++ a.clone()+ }+ "PRelu" => {+ if broadcast(a, &weight(n, g, 1)?.dims)? != *a {+ return shape_err("PRelu slope would enlarge its input");+ }+ a.clone()+ }+ "Add" | "Sub" | "Mul" => {+ let b = n.inputs.get(1).and_then(|s| shapes.get(s.as_str())).ok_or_else(|| NetError::Shape("missing second input".into()))?;+ broadcast(a, b)?+ }+ "Gemm" => {+ let b = &weight(n, g, 1)?.dims;+ let [ar, ac] = a.as_slice() else {+ return shape_err("Gemm needs matrix A");+ };+ let [br, bc] = b.as_slice() else {+ return shape_err("Gemm needs matrix B");+ };+ let (m, k) = if n.int("transA").unwrap_or(0) != 0 { (*ac, *ar) } else { (*ar, *ac) };+ let (bk, cols) = if n.int("transB").unwrap_or(0) != 0 { (*bc, *br) } else { (*br, *bc) };+ if k != bk {+ return shape_err("Gemm inner dimensions differ");+ }+ let s = vec![m, cols];+ if n.inputs.get(2).is_some_and(|s| !s.is_empty()) && broadcast(&s, &weight(n, g, 2)?.dims)? != s {+ return shape_err("Gemm bias would enlarge the output");+ }+ s+ }+ "Flatten" => {+ let at = axis(n, a.len())?;+ vec![+ elements(a.get(..at).ok_or_else(|| NetError::Shape("bad axis".into()))?)?,+ elements(a.get(at..).ok_or_else(|| NetError::Shape("bad axis".into()))?)?,+ ]+ }+ "GlobalAveragePool" => {+ let (c, _, _) = image(a)?;+ vec![1, c, 1, 1]+ }+ "MaxPool" => {+ let (c, h, w) = image(a)?;+ let (kh, kw) = pair(n, "kernel_shape", 1)?;+ let (sh, sw) = pair(n, "strides", 1)?;+ if h < kh || w < kw {+ return shape_err("MaxPool kernel exceeds input");+ }+ vec![1, c, (h - kh) / sh + 1, (w - kw) / sw + 1]+ }+ "Resize" => {+ let (c, h, w) = image(a)?;+ let s = floats(n, g, 2)?;+ if s.len() != 4+ || s.first() != Some(&1.0)+ || s.get(1) != Some(&1.0)+ || s.iter().any(|&v| !(1.0..=16.0).contains(&v) || v.fract() != 0.0)+ {+ return unsupported("Resize supports whole spatial scales from 1 to 16");+ }+ vec![+ 1,+ c,+ h.checked_mul(s.get(2).copied().unwrap_or(0.0) as usize).ok_or_else(|| NetError::Shape("Resize overflow".into()))?,+ w.checked_mul(s.get(3).copied().unwrap_or(0.0) as usize).ok_or_else(|| NetError::Shape("Resize overflow".into()))?,+ ]+ }+ "Transpose" => {+ let perm: Vec<i64> = n.ints("perm").map_or_else(|| (0..a.len() as i64).rev().collect(), Vec::from);+ let mut seen = std::collections::HashSet::new();+ let mut s = Vec::new();+ if perm.len() != a.len() {+ return shape_err("Transpose rank mismatch");+ }+ for d in perm {+ let d = usize::try_from(d).map_err(|_| NetError::Shape("negative permutation".into()))?;+ if !seen.insert(d) {+ return shape_err("repeated permutation axis");+ }+ s.push(*a.get(d).ok_or_else(|| NetError::Shape("bad permutation axis".into()))?);+ }+ s+ }+ "Reshape" => {+ let Data::I64(dims) = &weight(n, g, 1)?.data else {+ return shape_err("Reshape datatype");+ };+ if dims.len() > 8 {+ return shape_err("Reshape rank exceeds eight");+ }+ let total = elements(a)?;+ let mut s = Vec::new();+ let mut infer = None;+ for (j, &v) in dims.iter().enumerate() {+ s.push(match v {+ -1 if infer.is_none() => {+ infer = Some(j);+ 1+ }+ 0 => *a.get(j).ok_or_else(|| NetError::Shape("Reshape copies missing dimension".into()))?,+ v if v > 0 => usize::try_from(v).map_err(|_| NetError::Shape("Reshape dimension overflow".into()))?,+ _ => return shape_err("invalid Reshape dimension"),+ });+ }+ let known = elements(&s)?;+ if let Some(j) = infer {+ if !total.is_multiple_of(known) {+ return shape_err("Reshape does not divide evenly");+ }+ *s.get_mut(j).ok_or_else(|| NetError::Shape("missing inferred dimension".into()))? = total / known;+ }+ if elements(&s)? != total {+ return shape_err("Reshape changes element count");+ }+ s+ }+ _ => a.clone(),+ };+ let out_count = elements(&out)?;+ let mut copied_constants = 0usize;+ for (j, name) in n.inputs.iter().enumerate() {+ // Execution borrows operator weights, but get(0) and binary get(1) materialize constants.+ if (j == 0 || (j == 1 && matches!(n.op.as_str(), "Add" | "Sub" | "Mul")))+ && let Some(w) = g.weights.get(name)+ {+ copied_constants =+ copied_constants.checked_add(elements(&w.dims)?).ok_or_else(|| NetError::Shape("constant copies overflow".into()))?;+ }+ }+ if n.op == "Conv" {+ let wd = &weight(n, g, 1)?.dims;+ let [co, cg, kh, kw] = wd.as_slice() else {+ return shape_err("Conv rank");+ };+ let block = 256.min(out.get(2).copied().unwrap_or(1) * out.get(3).copied().unwrap_or(1));+ let lower = cg+ .checked_mul(*kh)+ .and_then(|v| v.checked_mul(*kw))+ .and_then(|v| v.checked_mul(block))+ .filter(|&v| v <= MAX_TENSOR)+ .ok_or_else(|| NetError::Shape("convolution scratch exceeds the limit".into()))?;+ scratch_lower = scratch_lower.max(lower);+ scratch_product = scratch_product.max(co * block);+ }+ let scratch = scratch_lower + scratch_product;+ if live+ .checked_add(out_count)+ .and_then(|v| v.checked_add(scratch))+ .and_then(|v| v.checked_add(copied_constants))+ .is_none_or(|v| v > 128 * 1024 * 1024)+ {+ return shape_err("tensors and scratch exceed the 512 MiB working limit");+ }+ live = live+ .checked_add(out_count)+ .filter(|&n| n <= 128 * 1024 * 1024)+ .ok_or_else(|| NetError::Shape("live tensors exceed the 512 MiB working limit".into()))?;+ shapes.insert(n.outputs.first().ok_or_else(|| NetError::Shape("no output".into()))?, out);+ let mut removed = std::collections::HashSet::new();+ for name in &n.inputs {+ if last.get(name.as_str()) == Some(&i) && !g.weights.contains_key(name) && removed.insert(name) {+ live = live.saturating_sub(shapes.get(name.as_str()).map_or(0, |s| elements(s).unwrap_or(0)));+ }+ }+ }+ g.outputs.iter().map(|s| shapes.get(s.as_str()).cloned().ok_or_else(|| NetError::Shape("missing output".into()))).collect()+}diff --git a/crates/faces/src/onnx.rs b/crates/faces/src/onnx.rsindex f94f67000..453be2412 100644--- a/crates/faces/src/onnx.rs+++ b/crates/faces/src/onnx.rs@@ -94,9 +94,10 @@ impl<R: Read + Seek> Walker<'_, R> {let mut v = 0u64;for shift in (0..70u32).step_by(7) {let b = self.byte()?;- if shift < 64 {- v |= u64::from(b & 0x7f) << shift;+ if shift == 63 && b > 1 {+ return Err(ProbeError::Malformed("number overflows 64 bits"));}+ v |= u64::from(b & 0x7f) << shift;if b & 0x80 == 0 {return Ok(v);}@@ -116,6 +117,9 @@ impl<R: Read + Seek> Walker<'_, R> {}let tag = self.varint()?;let number = u32::try_from(tag >> 3).map_err(|_| ProbeError::Malformed("field number too large"))?;+ if number == 0 || number > 0x1fff_ffff {+ return Err(ProbeError::Malformed("invalid field number"));+ }let wire = match tag & 7 {0 => Wire::Varint(self.varint()?),1 => {@@ -147,18 +151,21 @@ impl<R: Read + Seek> Walker<'_, R> {}_ => return Err(ProbeError::Malformed("unsupported field type")),};+ if self.pos()? > end {+ return Err(ProbeError::Malformed("field runs past its message"));+ }Ok(Some((number, wire)))}- /// A bounded UTF-8 string occupying `start..end` (empty when it is longer than we care about).+ /// A bounded UTF-8 string occupying `start..end`; never silently truncate graph names.pub(crate) fn string(&mut self, start: u64, end: u64) -> Result<String> {let len = end.saturating_sub(start);if len > MAX_STRING {- return Ok(String::new());+ return Err(ProbeError::Malformed("a string exceeds 4096 bytes"));}let mut buf = Vec::new();self.r.by_ref().take(len).read_to_end(&mut buf)?;- Ok(String::from_utf8_lossy(&buf).into_owned())+ String::from_utf8(buf).map_err(|_| ProbeError::Malformed("a string is not UTF-8"))}/// The bytes of `start..end`, which must be at most `max` long.@@ -321,6 +328,9 @@ fn opset<R: Read + Seek>(w: &mut Walker<R>, end: u64) -> Result<Option<u64>> {/// Read an ONNX model's graph inputs and outputs from any seekable source.pub fn probe_reader<R: Read + Seek>(r: &mut R) -> Result<OnnxInfo> {let total = r.seek(SeekFrom::End(0))?;+ if total > crate::manifest::MAX_MODEL_BYTES {+ return Err(ProbeError::Malformed("the model exceeds 512 MiB"));+ }r.seek(SeekFrom::Start(0))?;if total == 0 {return Err(ProbeError::Malformed("the file is empty"));@@ -486,4 +496,19 @@ pub(crate) mod tests {let _ = probe_reader(&mut Cursor::new(bytes));}}+ #[test]+ fn oversized_models_are_rejected_without_reading_or_allocating() {+ struct Oversized;+ impl std::io::Read for Oversized {+ fn read(&mut self, _: &mut [u8]) -> std::io::Result<usize> {+ panic!("an oversized model must not be read")+ }+ }+ impl std::io::Seek for Oversized {+ fn seek(&mut self, _: std::io::SeekFrom) -> std::io::Result<u64> {+ Ok(crate::manifest::MAX_MODEL_BYTES + 1)+ }+ }+ assert!(probe_reader(&mut Oversized).is_err());+ }}diff --git a/crates/faces/src/runtime.rs b/crates/faces/src/runtime.rsnew file mode 100644index 000000000..bbb1457eb--- /dev/null+++ b/crates/faces/src/runtime.rs@@ -0,0 +1,352 @@+//! Installed recognition models on LightCraft's own checked, single-thread float32 CPU engine.+//! The detector and recognisers share graph validation and convolution kernels.++use std::panic::{AssertUnwindSafe, catch_unwind};+use std::path::Path;+use std::sync::Arc;+use web_time::Instant;++use crate::{+ graph,+ net::{Net, Tensor},+};+use serde::Serialize;++use crate::manifest::{Colour, ModelManifest, OutputSpec};++#[derive(Debug, thiserror::Error)]+pub enum RuntimeError {+ #[error("the model could not be loaded: {0}")]+ Load(String),+ #[error("the model failed while running: {0}")]+ Run(String),+ #[error("{0}")]+ Input(String),+}++/// What `self_test` found.+#[derive(Clone, Debug, PartialEq, Serialize)]+#[serde(rename_all = "camelCase")]+pub struct SelfTest {+ pub ok: bool,+ /// Milliseconds to load and optimise the model.+ pub load_ms: f64,+ /// Median milliseconds to embed one face.+ pub embed_ms: f64,+ pub dimension: usize,+ /// Each check: what was checked and whether it held.+ pub checks: Vec<(String, bool)>,+}++type Plan = Net;++/// A loaded recognition model.+#[derive(Clone)]+pub struct Embedder {+ plan: Arc<Plan>,+ manifest: ModelManifest,+ width: usize,+ height: usize,+ dim: usize,+ load_ms: f64,+}++fn guarded<T>(what: &str, f: impl FnOnce() -> Result<T, RuntimeError>) -> Result<T, RuntimeError> {+ match catch_unwind(AssertUnwindSafe(f)) {+ Ok(r) => r,+ Err(_) => Err(RuntimeError::Run(format!("the runtime gave up on this model while {what}"))),+ }+}++impl Embedder {+ /// Load the `.onnx` at `path` as described by `manifest` (an embedder).+ pub fn load(path: &Path, manifest: &ModelManifest) -> Result<Embedder, RuntimeError> {+ let OutputSpec::Embedding { dim } = manifest.output else {+ return Err(RuntimeError::Load("this is not a recognition model".into()));+ };+ let (w, h) = (manifest.input.width as usize, manifest.input.height as usize);+ let started = Instant::now();+ crate::manifest::validate(manifest).map_err(|e| RuntimeError::Load(e.to_string()))?;+ let mut file = std::fs::File::open(path).map_err(|e| RuntimeError::Load(e.to_string()))?;+ let length = file.metadata().map_err(|e| RuntimeError::Load(e.to_string()))?.len();+ if length == 0 || length > 512 * 1024 * 1024 {+ return Err(RuntimeError::Load("the model file is empty or exceeds 512 MiB".into()));+ }+ let mut bytes = Vec::new();+ bytes+ .try_reserve_exact(usize::try_from(length).map_err(|_| RuntimeError::Load("model size overflow".into()))?)+ .map_err(|_| RuntimeError::Load("not enough memory for the model".into()))?;+ use std::io::Read;+ (&mut file).take(512 * 1024 * 1024 + 1).read_to_end(&mut bytes).map_err(|e| RuntimeError::Load(e.to_string()))?;+ let graph = graph::load(&bytes).map_err(|e| RuntimeError::Load(e.to_string()))?;+ let input = graph.inputs.first().ok_or_else(|| RuntimeError::Load("no input".into()))?;+ let want = [1, 3, h, w];+ if input.1.len() != 4 || input.1.iter().zip(want).any(|(&d, w)| d != 0 && d != w) {+ return Err(RuntimeError::Load("the graph input differs from the manifest's [1,3,H,W]".into()));+ }+ let plan = Net::new(graph).map_err(|e| RuntimeError::Load(e.to_string()))?;+ let shapes = plan.validate_input(&want).map_err(|e| RuntimeError::Load(e.to_string()))?;+ if shapes.len() != 1 || shapes.first().is_none_or(|s| s.first() != Some(&1) || s.iter().product::<usize>() != dim as usize) {+ return Err(RuntimeError::Load("the graph output differs from the manifest's embedding dimension".into()));+ }+ Ok(Embedder {+ plan: Arc::new(plan),+ manifest: manifest.clone(),+ width: w,+ height: h,+ dim: dim as usize,+ load_ms: started.elapsed().as_secs_f64() * 1000.0,+ })+ }++ /// The size of the aligned face picture the model wants (width, height).+ pub fn input_size(&self) -> (usize, usize) {+ (self.width, self.height)+ }++ pub fn manifest(&self) -> &ModelManifest {+ &self.manifest+ }++ /// The face as a unit-length vector. `rgb` is an aligned face at exactly [`Self::input_size`], 8-bit RGB.+ pub fn embed(&self, rgb: &[u8]) -> Result<Vec<f32>, RuntimeError> {+ if rgb.len() != self.width * self.height * 3 {+ return Err(RuntimeError::Input("the face picture is not the size the model wants".into()));+ }+ let (mean, std, bgr) = (self.manifest.input.mean, self.manifest.input.std, self.manifest.input.colour == Colour::Bgr);+ let plane = self.width * self.height;+ let mut data = vec![0.0f32; 3 * plane];+ for (i, px) in rgb.as_chunks::<3>().0.iter().enumerate() {+ for c in 0..3 {+ let src = if bgr { 2 - c } else { c };+ if let (Some(v), Some(m), Some(s), Some(o)) = (px.get(src), mean.get(c), std.get(c), data.get_mut(c * plane + i)) {+ *o = (f32::from(*v) - m) / s;+ }+ }+ }+ let out = guarded("running it", || {+ let input = Tensor::new(vec![1, 3, self.height, self.width], data).map_err(|e| RuntimeError::Input(e.to_string()))?;+ let mut result = self.plan.run(input).map_err(|e| RuntimeError::Run(e.to_string()))?;+ if result.len() != 1 {+ return Err(RuntimeError::Run("the model must give one embedding".into()));+ }+ Ok(result.pop().ok_or_else(|| RuntimeError::Run("no output".into()))?.1.data)+ })?;+ if out.len() != self.dim {+ return Err(RuntimeError::Run(format!("the model gave {} numbers per face, not the {} its description says", out.len(), self.dim)));+ }+ if out.iter().any(|v| !v.is_finite()) {+ return Err(RuntimeError::Run("the model gave numbers that are not finite".into()));+ }+ let norm = out.iter().map(|&v| f64::from(v).powi(2)).sum::<f64>().sqrt();+ if !norm.is_finite() || norm < 1e-9 {+ return Err(RuntimeError::Run("the model gave an all-zero face".into()));+ }+ Ok(out.into_iter().map(|v| (f64::from(v) / norm) as f32).collect())+ }++ /// Check the model does something sensible before it is ever used on a photo: the right number of finite+ /// numbers, the same picture twice gives the same face, and two different pictures give different faces.+ pub fn self_test(&self) -> SelfTest {+ let mut checks: Vec<(String, bool)> = Vec::new();+ let n = self.width * self.height * 3;+ // two deterministic, different test pictures: noise, and a diagonal gradient with a colour cast+ let mut seed = 0x2545_f491u32;+ let noise: Vec<u8> = (0..n)+ .map(|_| {+ seed ^= seed << 13;+ seed ^= seed >> 17;+ seed ^= seed << 5;+ (seed >> 8) as u8+ })+ .collect();+ let gradient: Vec<u8> = (0..n)+ .map(|i| (((i / 3) % self.width + (i / 3) / self.width) * 255 / (self.width + self.height) + (i % 3) * 40).min(255) as u8)+ .collect();+ let mut times = Vec::new();+ let mut run = |img: &[u8]| {+ let t = Instant::now();+ let r = self.embed(img);+ times.push(t.elapsed().as_secs_f64() * 1000.0);+ r+ };+ let (a, a2, b) = (run(&noise), run(&noise), run(&gradient));+ let dot = |x: &[f32], y: &[f32]| x.iter().zip(y).map(|(p, q)| p * q).sum::<f32>();+ match (&a, &a2, &b) {+ (Ok(a), Ok(a2), Ok(b)) => {+ checks.push((format!("gives {} finite numbers per face", a.len()), true));+ checks.push(("gives the same face for the same picture".into(), dot(a, a2) > 0.9999));+ checks.push(("gives different faces for different pictures".into(), dot(a, b) < 0.9995));+ }+ (Err(e), _, _) | (_, Err(e), _) | (_, _, Err(e)) => checks.push((format!("runs: {e}"), false)),+ }+ times.sort_by(|x, y| x.total_cmp(y));+ SelfTest {+ ok: checks.iter().all(|(_, ok)| *ok),+ load_ms: self.load_ms,+ embed_ms: times.get(times.len() / 2).copied().unwrap_or(0.0),+ dimension: self.dim,+ checks,+ }+ }+}++#[cfg(test)]+mod tests {+ use super::*;+ use crate::manifest::{InputSpec, Licence, OutputSpec, Role, Thresholds};++ fn manifest(dim: u32) -> ModelManifest {+ ModelManifest {+ id: "tiny".into(),+ name: "Tiny".into(),+ version: "1".into(),+ role: Role::Embedder,+ licence: Licence::default(),+ source: None,+ sha256: None,+ size_bytes: None,+ provenance: String::new(),+ input: InputSpec::default(),+ output: OutputSpec::Embedding { dim },+ thresholds: Thresholds::default(),+ speed: None,+ }+ }++ fn temp_model(name: &str, bytes: &[u8]) -> std::path::PathBuf {+ let p = std::env::temp_dir().join(format!("lc-faces-runtime-{name}-{}.onnx", std::process::id()));+ std::fs::write(&p, bytes).unwrap();+ p+ }++ #[test]+ fn a_working_model_loads_embeds_and_passes_the_self_test() {+ let p = temp_model("ok", &crate::synthetic::tiny_embedder_model(16));+ let e = Embedder::load(&p, &manifest(16)).unwrap();+ assert_eq!(e.input_size(), (112, 112));+ let v = e.embed(&vec![100u8; 112 * 112 * 3]).unwrap();+ assert_eq!(v.len(), 16);+ assert!((v.iter().map(|x| x * x).sum::<f32>() - 1.0).abs() < 1e-4, "unit length");+ let t = e.self_test();+ assert!(t.ok, "{t:?}");+ assert_eq!(t.dimension, 16);+ let _ = std::fs::remove_file(p);+ }++ #[test]+ fn wrong_sizes_wrong_dimensions_and_broken_files_are_errors() {+ let p = temp_model("dim", &crate::synthetic::tiny_embedder_model(16));+ let e = Embedder::load(&p, &manifest(16)).unwrap();+ assert!(e.embed(&[0u8; 10]).is_err(), "wrong picture size");+ // the manifest promises 32 numbers, the model gives 16+ assert!(Embedder::load(&p, &manifest(32)).is_err());+ // not a model, an empty file, a graph with no layers, a detector-shaped manifest+ for (name, bytes) in [("junk", b"nonsense".to_vec()), ("empty", Vec::new()), ("nolayers", crate::synthetic::embedder_model(16))] {+ let q = temp_model(name, &bytes);+ assert!(Embedder::load(&q, &manifest(16)).is_err(), "{name}");+ let _ = std::fs::remove_file(q);+ }+ let mut detector = manifest(16);+ detector.output = OutputSpec::Detector { decoder: "yunet-v2".into() };+ assert!(Embedder::load(&p, &detector).is_err());+ assert!(Embedder::load(&std::env::temp_dir().join("does-not-exist.onnx"), &manifest(16)).is_err());+ let _ = std::fs::remove_file(p);+ }++ #[test]+ fn truncated_and_corrupted_models_never_panic() {+ let good = crate::synthetic::tiny_embedder_model(16);+ for n in (0..good.len()).step_by(7) {+ let q = temp_model("trunc", &good[..n]);+ let _ = Embedder::load(&q, &manifest(16));+ let _ = std::fs::remove_file(q);+ }+ for at in (0..good.len()).step_by(11) {+ let mut bad = good.clone();+ if let Some(b) = bad.get_mut(at) {+ *b = b.wrapping_add(0x55);+ }+ let q = temp_model("flip", &bad);+ if let Ok(e) = Embedder::load(&q, &manifest(16)) {+ let _ = e.embed(&vec![7u8; 112 * 112 * 3]);+ }+ let _ = std::fs::remove_file(q);+ }+ }++ /// Opt-in: `LC_FACE_MODELS=<folder> cargo test -p lightcraft-faces -- --ignored --nocapture`+ /// self-tests every known recognition model found there.+ #[test]+ #[ignore = "needs real model files: set LC_FACE_MODELS"]+ fn real_models_pass_the_self_test() {+ let Some(dir) = std::env::var_os("LC_FACE_MODELS") else { return };+ for entry in std::fs::read_dir(dir).unwrap().flatten() {+ let path = entry.path();+ let Ok(sha) = crate::hash::sha256_file(&path) else { continue };+ let Some(m) = crate::known::lookup(&sha).filter(|m| m.role == Role::Embedder) else { continue };+ let e = Embedder::load(&path, &m).unwrap();+ let t = e.self_test();+ println!("{}: ok {} | load {:.0} ms | {:.0} ms per face | {} numbers | {:?}", m.id, t.ok, t.load_ms, t.embed_ms, t.dimension, t.checks);+ assert!(t.ok);+ }+ }++ #[test]+ fn bgr_changes_the_input() {+ let p = temp_model("bgr", &crate::synthetic::tiny_embedder_model(16));+ let rgb = Embedder::load(&p, &manifest(16)).unwrap();+ let mut m = manifest(16);+ m.input.colour = crate::manifest::Colour::Bgr;+ let bgr = Embedder::load(&p, &m).unwrap();+ // a picture with different channel levels: swapping the order must change the embedding+ let img: Vec<u8> = (0..112 * 112).flat_map(|_| [200u8, 100, 20]).collect();+ let (a, b) = (rgb.embed(&img).unwrap(), bgr.embed(&img).unwrap());+ let cos: f32 = a.iter().zip(&b).map(|(x, y)| x * y).sum();+ assert!(cos < 0.9999, "{cos}");+ let _ = std::fs::remove_file(p);+ }+ #[test]+ fn large_finite_embeddings_have_a_finite_unit_norm() {+ let p = temp_model("large-norm", &crate::synthetic::norm_overflow_model());+ let e = Embedder::load(&p, &manifest(3)).unwrap();+ let out = e.embed(&vec![255; 112 * 112 * 3]).unwrap();+ assert!(out.iter().all(|v| v.is_finite()));+ assert!((out.iter().map(|v| v * v).sum::<f32>() - 1.0).abs() < 1e-6);+ let _ = std::fs::remove_file(p);+ }+ #[test]+ fn invalid_manifest_numbers_are_rejected_before_preprocessing() {+ let p = temp_model("invalid-manifest", &crate::synthetic::tiny_embedder_model(16));+ for case in 0..5 {+ let mut m = manifest(16);+ match case {+ 0 => m.input.std = [0.0; 3],+ 1 => m.input.mean = [f32::NAN; 3],+ 2 => m.input.width = 0,+ 3 => m.input.width = u32::MAX,+ _ => m.output = OutputSpec::Embedding { dim: 0 },+ };+ assert!(Embedder::load(&p, &m).is_err());+ }+ let _ = std::fs::remove_file(p);+ }+ #[test]+ fn concurrent_faces_share_weights_and_give_deterministic_embeddings() {+ let p = temp_model("concurrent", &crate::synthetic::tiny_embedder_model(16));+ let e = Embedder::load(&p, &manifest(16)).unwrap();+ let input = vec![123; 112 * 112 * 3];+ let want = e.embed(&input).unwrap();+ let threads: Vec<_> = (0..8)+ .map(|_| {+ let e = e.clone();+ let input = input.clone();+ std::thread::spawn(move || e.embed(&input))+ })+ .collect();+ for thread in threads {+ assert_eq!(thread.join().unwrap().unwrap(), want);+ }+ let _ = std::fs::remove_file(p);+ }+}diff --git a/crates/faces/src/suggest.rs b/crates/faces/src/suggest.rsindex eddd07edc..377452e4c 100644--- a/crates/faces/src/suggest.rs+++ b/crates/faces/src/suggest.rs@@ -59,7 +59,12 @@ fn face_input(t: &TensorInfo) -> Option<(u32, u32)> {/// Propose what to do with a probed model. `sha256` and `size` describe the file, `file_name` names it.pub fn suggest(info: &OnnxInfo, sha256: &str, size: u64, file_name: &str) -> Suggestion {- if let Some(m) = crate::known::lookup(sha256) {+ suggest_with(info, sha256, size, file_name, &[])+}++/// [`suggest`], also recognising the models in `extra` (the user's own catalog) by their SHA-256.+pub fn suggest_with(info: &OnnxInfo, sha256: &str, size: u64, file_name: &str, extra: &[ModelManifest]) -> Suggestion {+ if let Some(m) = crate::known::lookup(sha256).or_else(|| extra.iter().find(|m| m.sha256.as_deref() == Some(sha256)).cloned()) {return Suggestion::Known(m);}let [input] = info.inputs.as_slice() else {@@ -98,6 +103,7 @@ pub fn suggest(info: &OnnxInfo, sha256: &str, size: u64, file_name: &str) -> Suginput: InputSpec { width, height, ..InputSpec::default() },output: OutputSpec::Embedding { dim },thresholds: Thresholds::default(),+ speed: None,};let assumptions = vec![format!("Faces are aligned crops of {width} × {height} pixels."),diff --git a/crates/faces/src/synthetic.rs b/crates/faces/src/synthetic.rsindex 7df9acfee..61010a243 100644--- a/crates/faces/src/synthetic.rs+++ b/crates/faces/src/synthetic.rs@@ -68,3 +68,183 @@ pub fn embedder_model(output_dim: u64) -> Vec<u8> {len_field(8, &ops, &mut model);model}++/// A node attribute for [`node_bytes`].+enum Attribute<'a> {+ Int(i64),+ Ints(&'a [i64]),+ Float(f32),+}++fn node_bytes(op: &str, inputs: &[&str], output: &str, attrs: &[(&str, Attribute)]) -> Vec<u8> {+ let mut n = Vec::new();+ for i in inputs {+ len_field(1, i.as_bytes(), &mut n);+ }+ len_field(2, output.as_bytes(), &mut n);+ len_field(4, op.as_bytes(), &mut n);+ for (name, value) in attrs {+ let mut a = Vec::new();+ len_field(1, name.as_bytes(), &mut a);+ match value {+ Attribute::Float(v) => {+ varint(2 << 3 | 5, &mut a);+ a.extend_from_slice(&v.to_le_bytes());+ varint_field(20, 1, &mut a);+ }+ Attribute::Int(v) => {+ varint_field(3, *v as u64, &mut a);+ varint_field(20, 2, &mut a); // INT+ }+ Attribute::Ints(values) => {+ for v in *values {+ varint_field(8, *v as u64, &mut a);+ }+ varint_field(20, 7, &mut a); // INTS+ }+ }+ len_field(5, &a, &mut n);+ }+ n+}++fn float_tensor_bytes(name: &str, dims: &[u64], values: &[f32]) -> Vec<u8> {+ let mut t = Vec::new();+ for d in dims {+ varint_field(1, *d, &mut t);+ }+ varint_field(2, 1, &mut t); // FLOAT+ len_field(8, name.as_bytes(), &mut t);+ let raw: Vec<u8> = values.iter().flat_map(|v| v.to_le_bytes()).collect();+ len_field(9, &raw, &mut t);+ t+}++/// A tiny working network shaped like a face embedder, for tests of the runtime: input `data`+/// `[1, 3, 112, 112]`, a 1 × 1 convolution to `dim` channels, global average pooling and flattening to+/// `[1, dim]`. Its output depends on the picture's per-channel brightness, which is all the tests need.+pub fn tiny_embedder_model(dim: u64) -> Vec<u8> {+ let mut graph = Vec::new();+ len_field(1, &node_bytes("Conv", &["data", "w", "b"], "c", &[("kernel_shape", Attribute::Ints(&[1, 1]))]), &mut graph);+ len_field(1, &node_bytes("GlobalAveragePool", &["c"], "p", &[]), &mut graph);+ len_field(1, &node_bytes("Flatten", &["p"], "emb", &[("axis", Attribute::Int(1))]), &mut graph);+ let w: Vec<f32> = (0..dim * 3).map(|i| ((i * 37 % 19) as f32 - 9.0) / 20.0).collect();+ let b: Vec<f32> = (0..dim).map(|i| (i % 5) as f32 / 10.0 - 0.2).collect();+ len_field(5, &float_tensor_bytes("w", &[dim, 3, 1, 1], &w), &mut graph);+ len_field(5, &float_tensor_bytes("b", &[dim], &b), &mut graph);+ len_field(11, &value_info_bytes("data", 1, &[Ok(1), Ok(3), Ok(112), Ok(112)]), &mut graph);+ len_field(12, &value_info_bytes("emb", 1, &[Ok(1), Ok(dim)]), &mut graph);+ let mut ops = Vec::new();+ len_field(1, b"", &mut ops);+ varint_field(2, 13, &mut ops);+ let mut model = Vec::new();+ varint_field(1, 8, &mut model);+ len_field(7, &graph, &mut model);+ len_field(8, &ops, &mut model);+ model+}++/// A deterministic small recognition graph using all SFace/AuraFace operators (no downloaded weights).+/// Intended for runtime regression and independent reference comparisons.+pub fn recognition_model(mut seed: u32) -> Vec<u8> {+ let mut random = || {+ seed = seed.wrapping_mul(1664525).wrapping_add(1013904223);+ ((seed >> 16) as f32 / 65535.0 - 0.5) * 0.2+ };+ let mut g = Vec::new();+ for (op, inputs, out, attrs) in [+ ("Sub", vec!["data", "center"], "centered", vec![]),+ ("Mul", vec!["centered", "scale"], "scaled", vec![]),+ ("Conv", vec!["scaled", "conv.w", "conv.b"], "conv", vec![("pads", Attribute::Ints(&[1, 1, 1, 1])), ("strides", Attribute::Ints(&[2, 2]))]),+ ("BatchNormalization", vec!["conv", "bn.s", "bn.b", "bn.m", "bn.v"], "bn", vec![("epsilon", Attribute::Float(1e-5))]),+ ("PRelu", vec!["bn", "slope"], "act", vec![]),+ ("Add", vec!["act", "bn"], "skip", vec![]),+ ("Dropout", vec!["skip"], "drop", vec![("ratio", Attribute::Float(0.4))]),+ ("GlobalAveragePool", vec!["drop"], "pool", vec![]),+ ("Flatten", vec!["pool"], "flat", vec![]),+ ("Gemm", vec!["flat", "fc.w", "fc.b"], "emb", vec![("transB", Attribute::Int(1))]),+ ] {+ len_field(1, &node_bytes(op, &inputs, out, &attrs), &mut g);+ }+ for (name, dims, data) in [+ ("center", vec![1], vec![127.5]),+ ("scale", vec![1], vec![1.0 / 127.5]),+ ("conv.w", vec![5, 3, 3, 3], (0..135).map(|_| random()).collect()),+ ("conv.b", vec![5], (0..5).map(|_| random()).collect()),+ ("bn.s", vec![5], (0..5).map(|_| 1.0 + random()).collect()),+ ("bn.b", vec![5], (0..5).map(|_| random()).collect()),+ ("bn.m", vec![5], (0..5).map(|_| random()).collect()),+ ("bn.v", vec![5], (0..5).map(|_| 1.0 + random()).collect()),+ ("slope", vec![5, 1, 1], (0..5).map(|_| 0.2 + random()).collect()),+ ("fc.w", vec![8, 5], (0..40).map(|_| random()).collect()),+ ("fc.b", vec![8], (0..8).map(|_| random()).collect()),+ ] {+ len_field(5, &float_tensor_bytes(name, &dims, &data), &mut g);+ }+ len_field(11, &value_info_bytes("data", 1, &[Ok(1), Ok(3), Ok(112), Ok(112)]), &mut g);+ len_field(12, &value_info_bytes("emb", 1, &[Ok(1), Ok(8)]), &mut g);+ let mut ops = Vec::new();+ varint_field(2, 11, &mut ops);+ let mut model = Vec::new();+ varint_field(1, 8, &mut model);+ len_field(7, &g, &mut model);+ len_field(8, &ops, &mut model);+ model+}++#[cfg(test)]+pub(crate) fn norm_overflow_model() -> Vec<u8> {+ let mut graph = Vec::new();+ len_field(1, &node_bytes("Flatten", &["data"], "flat", &[]), &mut graph);+ len_field(1, &node_bytes("Gemm", &["flat", "w"], "emb", &[("transB", Attribute::Int(1))]), &mut graph);+ let plane = 112 * 112;+ let mut weights = vec![0.0; 3 * 3 * plane];+ for c in 0..3 {+ weights[c * 3 * plane + c * plane] = 5e37;+ }+ len_field(5, &float_tensor_bytes("w", &[3, (3 * plane) as u64], &weights), &mut graph);+ len_field(11, &value_info_bytes("data", 1, &[Ok(1), Ok(3), Ok(112), Ok(112)]), &mut graph);+ len_field(12, &value_info_bytes("emb", 1, &[Ok(1), Ok(3)]), &mut graph);+ let mut ops = Vec::new();+ varint_field(2, 11, &mut ops);+ let mut model = Vec::new();+ len_field(7, &graph, &mut model);+ len_field(8, &ops, &mut model);+ model+}++/// A small YuNet-shaped stand-in that reports no faces, independent of the real downloaded model.+pub fn tiny_detector_model() -> Vec<u8> {+ let mut graph = Vec::new();+ len_field(11, &value_info_bytes("input", 1, &[Ok(1), Ok(3), Ok(64), Ok(64)]), &mut graph);+ for stride in [8u64, 16, 32] {+ let cells = (64 / stride) * (64 / stride);+ for (kind, components) in [("cls", 1u64), ("obj", 1), ("bbox", 4), ("kps", 10)] {+ let name = format!("{kind}_{stride}");+ let weight = format!("weight_{name}");+ let values = vec![0.0; (cells * components) as usize];+ len_field(5, &float_tensor_bytes(&weight, &[1, cells, components], &values), &mut graph);+ len_field(1, &node_bytes("Identity", &[&weight], &name, &[]), &mut graph);+ len_field(12, &value_info_bytes(&name, 1, &[Ok(1), Ok(cells), Ok(components)]), &mut graph);+ }+ }+ let mut model = Vec::new();+ varint_field(1, 8, &mut model);+ len_field(7, &graph, &mut model);+ let mut ops = Vec::new();+ varint_field(2, 13, &mut ops);+ len_field(8, &ops, &mut model);+ model+}++/// Manifest for the stand-in, for model-install and background-scan tests only.+pub fn tiny_detector_manifest() -> crate::manifest::ModelManifest {+ let bytes = tiny_detector_model();+ let mut m = crate::known::yunet();+ m.name = "Synthetic detector (tests only)".into();+ m.input.width = 64;+ m.input.height = 64;+ m.sha256 = Some(crate::hash::sha256_hex(&bytes));+ m.size_bytes = Some(bytes.len() as u64);+ m+}diff --git a/crates/faces/src/yunet.rs b/crates/faces/src/yunet.rsindex 3a24fe18e..b77f318bf 100644--- a/crates/faces/src/yunet.rs+++ b/crates/faces/src/yunet.rs@@ -70,14 +70,31 @@ impl Detector {/// A detector from YuNet-architecture ONNX bytes (a retrained model of the same shape works too).pub fn new(onnx: &[u8]) -> Result<Detector, DetectError> {let g = graph::load(onnx).map_err(|e| DetectError::Model(e.to_string()))?;+ let outputs = g.outputs.clone();let net = Net::new(g)?;let side = match net.input_shape() {Some(&[1, 3, h, w]) if h == w && (64..=4096).contains(&h) && h % 32 == 0 => h,_ => return Err(DetectError::Model("the model's input is not a fixed square image [1, 3, N, N] with N a multiple of 32".into())),};+ let shapes = net.validate_input(&[1, 3, side, side])?;+ for stride in STRIDES {+ for (kind, components) in [("cls", 1), ("obj", 1), ("bbox", 4), ("kps", 10)] {+ let name = format!("{kind}_{stride}");+ let index = outputs.iter().position(|n| n == &name).ok_or_else(|| DetectError::Model(format!("the model has no output `{name}`")))?;+ let count = shapes.get(index).and_then(|s| s.iter().try_fold(1usize, |n, d| n.checked_mul(*d)));+ if count != Some((side / stride) * (side / stride) * components) {+ return Err(DetectError::Model(format!("the output `{name}` has the wrong shape")));+ }+ }+ }Ok(Detector { net, side })}+ /// The square input side, used to check an installed manifest.+ pub fn input_side(&self) -> usize {+ self.side+ }+/// Find faces in an upright `width` × `height` photo of 8-bit RGB pixels (`rgb` is `width * height * 3` bytes).pub fn detect(&self, rgb: &[u8], width: usize, height: usize, opts: &Options) -> Result<Vec<Face>, DetectError> {if width == 0 || height == 0 || width > MAX_SIDE || height > MAX_SIDE {@@ -270,7 +287,7 @@ mod tests {#[test]fn bad_input_is_an_error_not_a_panic() {- let Some(d) = detector() else { return };+ let d = Detector::new(&crate::synthetic::tiny_detector_model()).unwrap();let o = Options::default();assert!(d.detect(&[], 0, 0, &o).is_err());assert!(d.detect(&[0; 12], 2, 3, &o).is_err(), "wrong byte count");@@ -280,8 +297,24 @@ mod tests {assert!(d.detect(&vec![0u8; 12 * 12 * 3], 12, 12, &nan).is_ok(), "non-finite options fall back to defaults");}+ #[test]+ fn missing_decoder_outputs_are_refused_at_load_time() {+ let mut bytes = crate::synthetic::tiny_detector_model();+ let mut i = 0;+ while i + 5 <= bytes.len() {+ if &bytes[i..i + 5] == b"cls_8" {+ bytes[i..i + 5].copy_from_slice(b"bad_8");+ }+ i += 1;+ }+ let error = Detector::new(&bytes).err().unwrap().to_string();+ assert!(error.contains("cls_8"), "{error}");+ }+#[test]fn a_model_that_is_not_yunet_shaped_is_refused() {+ let d = Detector::new(&crate::synthetic::tiny_detector_model()).unwrap();+ assert!(d.detect(&[128; 3], 1, 1, &Options::default()).unwrap().is_empty());assert!(Detector::new(b"nonsense").is_err());assert!(Detector::new(&crate::synthetic::embedder_model(512)).is_err());if let Some(bytes) = model() {diff --git a/crates/ui-egui/src/control.rs b/crates/ui-egui/src/control.rsindex cfcb287fc..fca64f564 100644--- a/crates/ui-egui/src/control.rs+++ b/crates/ui-egui/src/control.rs@@ -105,6 +105,8 @@ pub fn inspect(app: &LightcraftApp, ctx: &egui::Context) -> Value {"status": app.ui.status,"notices": app.notices,"quitPrompt": app.quit_prompt,+ // how hard the background face scan may work right now, and why it is judged so (see `panels::faces::scan_pace`)+ "faceScan": {"pace": app.caches.faces_pace, "focused": app.caches.faces_in_front, "pending": app.caches.faces_pending, "indexed": app.caches.faces_indexed},"unsaved": app.session.unsaved().map(|(n, e)| json!({"ops": n, "error": e})),"libraryProblem": app.library_problem.as_ref().map(crate::panels::library_problem::LibraryProblem::to_json),"scan": app.scan.as_ref().map(crate::import::ScanTask::status),diff --git a/crates/ui-egui/src/headless.rs b/crates/ui-egui/src/headless.rsindex 6278c2771..b9ad67286 100644--- a/crates/ui-egui/src/headless.rs+++ b/crates/ui-egui/src/headless.rs@@ -401,6 +401,58 @@ mod tests {h.settle(SETTLE);}+ /// Naming a face in the loupe: point at an unnamed face, click its "Add name" label, type, Enter. Needs the+ /// recognition runtime only for the background indexer, so the naming itself is tested whatever the build.+ #[test]+ fn naming_a_face_in_the_loupe() {+ use crate::state::NameEdit;+ use lightcraft_catalog::Op;+ let mut h = demo([1400.0, 900.0]);+ let t = Duration::from_secs(10);+ let id = h.app.session.active().unwrap();+ let mut meta = h.app.session.catalog.photo(id).unwrap().meta.clone();+ meta.regions = vec![lightcraft_meta::Region {+ rect: lightcraft_geom::Rect { x0: 0.3, y0: 0.2, x1: 0.55, y1: 0.6 },+ kind: lightcraft_meta::RegionKind::Face,+ name: None,+ description: None,+ }];+ h.app.session.commit("setup", Op::SetMeta { id, meta: Box::new(meta) }).unwrap();+ h.request("ui.set", json!({"view": "detail", "right": "none"}), t);+ h.settle(SETTLE);+ // pointing at the box shows the invitation; clicking it opens the name box+ h.request("ui.pointer", json!({"events": [{"kind": "move", "x": 0.42, "y": 0.4}]}), t);+ h.step();+ h.step();+ let r = h.request("ui.clickWidget", json!({"id": "regionLabel:0"}), t);+ assert_eq!(r["ok"], true, "{r}");+ h.step();+ assert!(matches!(&h.app.ui.name_edit, Some(NameEdit { index: 0, .. })), "{:?}", h.app.ui.name_edit);+ // Escape closes it without naming anything+ let r = h.request("ui.key", json!({"key": "escape"}), t);+ assert_eq!(r["ok"], true, "{r}");+ h.step();+ h.step();+ assert!(h.app.ui.name_edit.is_none(), "{:?}", h.app.ui.name_edit);+ assert_eq!(h.app.session.catalog.photo(id).unwrap().meta.regions[0].name, None);+ // open it again, type a name, Enter: the face is named, in one undo step+ h.request("ui.pointer", json!({"events": [{"kind": "move", "x": 0.42, "y": 0.4}]}), t);+ h.step();+ h.step();+ h.request("ui.clickWidget", json!({"id": "regionLabel:0"}), t);+ h.step();+ h.step();+ let undo = h.app.session.undo.len();+ h.request("ui.text", json!({"text": "Ann"}), t);+ h.step();+ h.request("ui.key", json!({"key": "enter"}), t);+ h.step();+ h.step();+ assert_eq!(h.app.session.catalog.photo(id).unwrap().meta.regions[0].name.as_deref(), Some("Ann"));+ assert_eq!(h.app.session.undo.len(), undo + 1);+ assert!(h.app.ui.name_edit.is_none());+ }+/// Adding a face model: the dialog shows the file's terms, the model is installed only once they are/// accepted, and a file LightCraft cannot use only says why.#[test]@@ -413,7 +465,7 @@ mod tests {std::fs::create_dir_all(&dir).unwrap();h.app.session.face_models_dir = Some(dir.join("models"));let model = dir.join("Mine.onnx");- std::fs::write(&model, lightcraft_faces::synthetic::embedder_model(512)).unwrap();+ std::fs::write(&model, lightcraft_faces::synthetic::tiny_embedder_model(512)).unwrap();let open = |h: &mut Headless, path: &std::path::Path| {h.request("engine.execute", json!({"command": "dialog.faceModel", "params": {"path": path.to_string_lossy()}}), t)};@@ -439,7 +491,10 @@ mod tests {assert_eq!(r["ok"], true, "{r}");let listed = h.app.run("faces.models.list", json!({})).unwrap();assert!(listed["models"].as_array().unwrap().iter().any(|m| m["installed"] == true && m["known"] == false), "{listed}");- assert!(h.app.ui.dialog.is_none());+ // installed, in use and recognition on: back in Settings ▸ Faces, where the scan can be watched+ assert_eq!(h.app.ui.dialog, Some(Dialog::Settings { tab: "faces".into() }));+ assert_eq!(listed["enabled"], true);+ h.app.ui.dialog = None;// a file that is not a usable model says why and offers no installlet junk = dir.join("junk.onnx");@@ -490,6 +545,330 @@ mod tests {let _ = std::fs::remove_dir_all(&dir);}+ /// Pressing Download in Settings shows the model's terms first and fetches nothing until they are accepted; a build+ /// that cannot run recognition models offers no download. The licence dialog replaces Settings, which it was opened from.+ #[test]+ fn download_shows_the_terms_first_and_fetches_nothing_until_accepted() {+ use crate::state::Dialog;+ let mut h = demo([1400.0, 900.0]);+ let t = Duration::from_secs(10);+ let dir = std::env::temp_dir().join(format!("lc-ui-facedl-{}", std::process::id()));+ let _ = std::fs::remove_dir_all(&dir);+ std::fs::create_dir_all(&dir).unwrap();+ h.app.session.face_models_dir = Some(dir.join("models"));+ h.request("engine.execute", json!({"command": "app.settings", "params": {"tab": "faces"}}), t);+ h.settle(SETTLE);+ h.step();+ let runtime = h.app.session.execute("faces.models.list", &json!({})).unwrap()["runtime"] == true;+ let r = h.request("ui.clickWidget", json!({"id": "faces:download:sface-2021dec"}), t);+ if !runtime {+ assert_eq!(r["ok"], false, "a build that cannot run the model offers no download: {r}");+ let _ = std::fs::remove_dir_all(&dir);+ return;+ }+ assert_eq!(r["ok"], true, "{r}");+ h.step();+ h.step();+ let Some(dlg) = h.app.ui.dialog.clone() else { panic!("no dialog") };+ let Dialog::FaceModel { path, info, accepted } = &dlg else { panic!("the terms did not replace Settings: {dlg:?}") };+ assert_eq!((info["download"].as_str(), *accepted, path.as_str()), (Some("sface-2021dec"), false, ""));+ assert_eq!(info["model"]["licence"]["commercial"], "unknown");+ // OK does nothing until the terms are accepted, and nothing has been fetched+ assert!(crate::panels::dialogs::confirm_dialog(&mut h.app, &dlg).is_err());+ assert_eq!(h.app.session.execute("faces.models.downloads", &json!({})).unwrap()["downloads"], json!([]));+ // cancelling leaves it that way+ h.request("ui.key", json!({"key": "escape"}), t);+ h.step();+ assert!(h.app.ui.dialog.is_none());+ assert_eq!(h.app.session.execute("faces.models.downloads", &json!({})).unwrap()["downloads"], json!([]));+ let _ = std::fs::remove_dir_all(&dir);+ }++ /// Until face recognition is set up, the People view and the loupe's name box say so and offer the next step: with no+ /// model, a button that opens Settings ▸ Faces; with a model installed but recognition off, one that switches it on.+ #[test]+ fn people_and_the_name_box_offer_to_set_face_recognition_up() {+ use crate::state::{Dialog, ViewMode};+ use lightcraft_catalog::Op;+ let mut h = demo([1400.0, 900.0]);+ let t = Duration::from_secs(10);+ let dir = std::env::temp_dir().join(format!("lc-ui-facesetup-{}", std::process::id()));+ let _ = std::fs::remove_dir_all(&dir);+ std::fs::create_dir_all(&dir).unwrap();+ h.app.session.face_models_dir = Some(dir.join("models"));+ let runtime = h.app.session.execute("faces.models.list", &json!({})).unwrap()["runtime"] == true;+ let id = h.app.session.active().unwrap();+ let mut meta = h.app.session.catalog.photo(id).unwrap().meta.clone();+ meta.regions = vec![lightcraft_meta::Region {+ rect: lightcraft_geom::Rect { x0: 0.3, y0: 0.2, x1: 0.55, y1: 0.6 },+ kind: lightcraft_meta::RegionKind::Face,+ name: None,+ description: None,+ }];+ h.app.session.commit("setup", Op::SetMeta { id, meta: Box::new(meta) }).unwrap();+ let offers = |h: &mut Headless| h.request("ui.widgets", json!({}), t).to_string().contains("\"faces:setup\"");++ h.request("engine.execute", json!({"command": "view.people"}), t);+ h.settle(SETTLE);+ h.step();+ assert_eq!(h.app.ui.view, ViewMode::People);+ if !runtime {+ assert!(!offers(&mut h), "a build that cannot run recognition offers nothing");+ let _ = std::fs::remove_dir_all(&dir);+ return;+ }+ // no model: the offer is there, and one click lands in Settings ▸ Faces+ assert!(offers(&mut h));+ let r = h.request("ui.clickWidget", json!({"id": "faces:setup"}), t);+ assert_eq!(r["ok"], true, "{r}");+ h.step();+ h.step();+ assert_eq!(h.app.ui.dialog, Some(Dialog::Settings { tab: "faces".into() }));+ h.app.ui.dialog = None;++ // the loupe's name box offers it too, and its button leaves for Settings with the box closed+ h.request("ui.set", json!({"view": "detail", "right": "none"}), t);+ h.settle(SETTLE);+ h.request("ui.pointer", json!({"events": [{"kind": "move", "x": 0.42, "y": 0.4}]}), t);+ h.step();+ h.step();+ h.request("ui.clickWidget", json!({"id": "regionLabel:0"}), t);+ h.step();+ h.step();+ assert!(h.app.ui.name_edit.is_some() && offers(&mut h), "the name box offers the setup");+ let r = h.request("ui.clickWidget", json!({"id": "faces:setup"}), t);+ assert_eq!(r["ok"], true, "{r}");+ h.step();+ h.step();+ assert!(h.app.ui.name_edit.is_none());+ assert_eq!(h.app.ui.dialog, Some(Dialog::Settings { tab: "faces".into() }));+ h.app.ui.dialog = None;++ // a model installed but recognition off: "Turn on" does it on the spot+ let model = dir.join("Mine.onnx");+ std::fs::write(&model, lightcraft_faces::synthetic::tiny_embedder_model(512)).unwrap();+ let installed = h.app.run("faces.models.install", json!({"path": model.to_string_lossy(), "acknowledged": true, "activate": false})).unwrap();+ h.app.run("faces.models.select", json!({"id": installed["installed"]["id"]})).unwrap();+ h.app.caches.faces_epoch += 1;+ h.request("engine.execute", json!({"command": "view.people"}), t);+ h.settle(SETTLE);+ h.step();+ assert!(offers(&mut h));+ assert_eq!(h.app.session.execute("faces.models.list", &json!({})).unwrap()["enabled"], false);+ let r = h.request("ui.clickWidget", json!({"id": "faces:setup"}), t);+ assert_eq!(r["ok"], true, "{r}");+ h.step();+ h.step();+ assert_eq!(h.app.session.execute("faces.models.list", &json!({})).unwrap()["enabled"], true);+ assert_eq!(h.app.ui.dialog, None, "nothing to go to Settings for");+ h.step();+ assert!(!offers(&mut h), "once it is on the offer is gone");+ h.settle(SETTLE);+ let _ = std::fs::remove_dir_all(&dir);+ }++ /// A click on a person's card opens their page, which shows only cropped faces (one per face, not per photo); a click+ /// on one opens its photo; Back, the People button and Escape return to everyone.+ #[test]+ fn a_persons_page_shows_their_faces_and_back_returns_to_everyone() {+ use crate::state::ViewMode;+ use lightcraft_catalog::Op;+ let mut h = demo([1400.0, 900.0]);+ let t = Duration::from_secs(10);+ let ids: Vec<_> = h.app.session.catalog.photos().map(|p| p.id).take(3).collect();+ let face = |x: f64, name: &str| lightcraft_meta::Region {+ rect: lightcraft_geom::Rect { x0: x, y0: 0.2, x1: x + 0.25, y1: 0.6 },+ kind: lightcraft_meta::RegionKind::Face,+ name: Some(name.to_string()),+ description: None,+ };+ // Jane Doe twice in the first photo and once in the second; John Roe in the third+ for (id, regions) in [+ (ids[0], vec![face(0.1, "Jane Doe"), face(0.5, "jane doe")]),+ (ids[1], vec![face(0.3, "Jane Doe")]),+ (ids[2], vec![face(0.3, "John Roe")]),+ ] {+ let mut meta = h.app.session.catalog.photo(id).unwrap().meta.clone();+ meta.regions = regions;+ h.app.session.commit("setup", Op::SetMeta { id, meta: Box::new(meta) }).unwrap();+ }+ let open_people = |h: &mut Headless| {+ h.request("engine.execute", json!({"command": "view.people"}), t);+ h.settle(SETTLE);+ h.step();+ };+ open_people(&mut h);+ assert_eq!((h.app.ui.view, h.app.ui.person_page.clone()), (ViewMode::People, None));+ // the card opens the page: a face tile for each of the three faces, not a tile per photo+ let r = h.request("ui.clickWidget", json!({"id": "person:Jane Doe"}), t);+ assert_eq!(r["ok"], true, "{r}");+ h.step();+ h.step();+ assert_eq!((h.app.ui.view, h.app.ui.person_page.as_deref()), (ViewMode::People, Some("Jane Doe")));+ let widgets = h.request("ui.widgets", json!({}), t).to_string();+ for (id, index) in [(ids[0], 0), (ids[0], 1), (ids[1], 0)] {+ assert!(widgets.contains(&format!("\"person-face:{}:{index}\"", id.0)), "a tile for each face: {widgets}");+ }+ assert!(!widgets.contains(&format!("\"person-face:{}:0\"", ids[2].0)), "someone else's face is not on the page");+ // a face opens its photo in the detail view+ h.request("ui.clickWidget", json!({"id": format!("person-face:{}:0", ids[1].0)}), t);+ h.step();+ h.step();+ assert_eq!((h.app.ui.view, h.app.session.active()), (ViewMode::Detail, Some(ids[1])));+ // back to the page, then each way out of it+ h.request("engine.execute", json!({"command": "view.person", "params": {"name": "Jane Doe"}}), t);+ h.step();+ assert_eq!((h.app.ui.view, h.app.ui.person_page.as_deref()), (ViewMode::People, Some("Jane Doe")));+ let r = h.request("ui.clickWidget", json!({"id": "person:back"}), t);+ assert_eq!(r["ok"], true, "{r}");+ h.step();+ h.step();+ assert_eq!(h.app.ui.person_page, None);+ h.request("engine.execute", json!({"command": "view.person", "params": {"name": "Jane Doe"}}), t);+ h.request("ui.key", json!({"key": "escape"}), t);+ h.step();+ assert_eq!((h.app.ui.view, h.app.ui.person_page.clone()), (ViewMode::People, None), "Escape goes back to everyone");+ h.request("engine.execute", json!({"command": "view.person", "params": {"name": "John Roe"}}), t);+ open_people(&mut h);+ assert_eq!(h.app.ui.person_page, None, "the People button shows everyone");+ // a missing or empty name is refused+ assert_eq!(h.request("engine.execute", json!({"command": "view.person", "params": {}}), t)["ok"], false);+ assert_eq!(h.request("engine.execute", json!({"command": "view.person", "params": {"name": " "}}), t)["ok"], false);+ }++ /// The People view lists the unnamed faces below the named people: select some (a click each, or all), type a name,+ /// press Enter, and they are all named at once, in one undo step; the new person appears among the named.+ #[test]+ fn unnamed_faces_are_selected_and_named_together() {+ use lightcraft_catalog::Op;+ let mut h = demo([1400.0, 900.0]);+ let t = Duration::from_secs(10);+ let ids: Vec<_> = h.app.session.catalog.photos().map(|p| p.id).take(3).collect();+ let face = |x: f64, name: Option<&str>| lightcraft_meta::Region {+ rect: lightcraft_geom::Rect { x0: x, y0: 0.2, x1: x + 0.2, y1: 0.55 },+ kind: lightcraft_meta::RegionKind::Face,+ name: name.map(str::to_string),+ description: None,+ };+ for (id, regions) in [+ (ids[0], vec![face(0.1, Some("Jane Doe")), face(0.5, None)]),+ (ids[1], vec![face(0.3, None), face(0.6, None)]),+ (ids[2], vec![face(0.3, None)]),+ ] {+ let mut meta = h.app.session.catalog.photo(id).unwrap().meta.clone();+ meta.regions = regions;+ h.app.session.commit("setup", Op::SetMeta { id, meta: Box::new(meta) }).unwrap();+ }+ h.request("engine.execute", json!({"command": "view.people"}), t);+ h.settle(SETTLE);+ h.step();+ h.step();+ let tiles = |h: &mut Headless| h.request("ui.widgets", json!({"filter": "unnamed-face:"}), t)["result"].as_array().map_or(0, Vec::len);+ // the named person is a card, and the four unnamed faces are tiles below+ let r = h.request("ui.clickWidget", json!({"id": "person:Jane Doe"}), t);+ assert_eq!(r["ok"], true, "{r}");+ h.request("engine.execute", json!({"command": "view.people"}), t);+ h.step();+ h.step();+ assert_eq!(tiles(&mut h), 4);+ // select two (nothing is named by selecting), and the naming bar appears+ for (id, i) in [(ids[1], 0), (ids[2], 0)] {+ let r = h.request("ui.clickWidget", json!({"id": format!("unnamed-face:{}:{i}", id.0)}), t);+ assert_eq!(r["ok"], true, "{r}");+ h.step();+ h.step();+ }+ assert_eq!(h.app.ui.unnamed_selected.len(), 2);+ let bar = h.request("ui.widgets", json!({"filter": "unnamed:"}), t).to_string();+ assert!(bar.contains("unnamed:name") && bar.contains("unnamed:clear"), "{bar}");+ assert!(h.request("ui.widgets", json!({"filter": "field:unnamedName"}), t).to_string().contains("field:unnamedName"));+ // type a name and press Enter: both faces are named, in one undo step, and the selection is gone+ let undo = h.app.session.undo.len();+ h.request("ui.text", json!({"text": "Ann Example"}), t);+ h.step();+ h.request("ui.key", json!({"key": "enter"}), t);+ h.step();+ h.step();+ let named = |h: &Headless, id: lightcraft_catalog::PhotoId, i: usize| h.app.session.catalog.photo(id).unwrap().meta.regions[i].name.clone();+ assert_eq!((named(&h, ids[1], 0), named(&h, ids[2], 0)), (Some("Ann Example".to_string()), Some("Ann Example".to_string())));+ assert_eq!(named(&h, ids[1], 1), None, "a face that was not selected stays unnamed");+ assert_eq!(h.app.session.undo.len(), undo + 1, "one step for both");+ assert!(h.app.ui.unnamed_selected.is_empty() && h.app.ui.unnamed_name.is_empty());+ h.settle(SETTLE);+ h.step();+ assert_eq!(tiles(&mut h), 2, "the named faces left the unnamed list");+ assert_eq!(h.app.session.catalog.people().len(), 2, "and the new person is among the named");+ // select all, then clear: nothing is named+ h.request("ui.clickWidget", json!({"id": "unnamed:selectAll"}), t);+ h.step();+ h.step();+ assert_eq!(h.app.ui.unnamed_selected.len(), 2);+ h.request("ui.clickWidget", json!({"id": "unnamed:clear"}), t);+ h.step();+ assert!(h.app.ui.unnamed_selected.is_empty());+ // undo gives the two faces back+ h.request("engine.execute", json!({"command": "edit.undo"}), t);+ h.step();+ assert_eq!(named(&h, ids[1], 0), None);+ }++ /// A screenful of faces larger than the picture cache's usual budget (96) is all kept: with a fixed budget the same few+ /// tiles were evicted and re-requested every frame and stayed blank.+ #[test]+ fn a_screenful_of_small_faces_is_not_evicted_and_left_blank() {+ use lightcraft_catalog::Op;+ let mut h = demo([1500.0, 1000.0]);+ let t = Duration::from_secs(20);+ let ids: Vec<_> = h.app.session.catalog.photos().map(|p| p.id).collect();+ // seven unnamed faces in every photo of the demo library+ for id in &ids {+ let mut meta = h.app.session.catalog.photo(*id).unwrap().meta.clone();+ meta.regions = (0..7)+ .map(|k| lightcraft_meta::Region {+ rect: lightcraft_geom::Rect { x0: 0.05 + 0.12 * k as f64, y0: 0.2, x1: 0.15 + 0.12 * k as f64, y1: 0.45 },+ kind: lightcraft_meta::RegionKind::Face,+ name: None,+ description: None,+ })+ .collect();+ h.app.session.commit("setup", Op::SetMeta { id: *id, meta: Box::new(meta) }).unwrap();+ }+ // the smallest faces, so a lot of them fit on the screen+ h.request("ui.set", json!({"thumbSize": 90.0, "view": "people"}), t);+ h.settle(SETTLE);+ h.step();+ let drawn = h.request("ui.widgets", json!({"filter": "unnamed-face:"}), t)["result"].as_array().map_or(0, Vec::len);+ assert!(drawn > 96, "the test needs more faces on screen than the usual budget, got {drawn}");+ // the pictures are made and kept, well beyond the old budget (a few demo photos share a scene, so two faces can share a+ // picture: the count is a little under the number of tiles)+ let started = Instant::now();+ while h.app.renderer.variant_textures() < drawn - 20 && started.elapsed() < Duration::from_secs(20) {+ h.step();+ std::thread::sleep(Duration::from_millis(20));+ }+ let kept = h.app.renderer.variant_textures();+ assert!(kept > 96 + 30, "{kept} pictures kept for {drawn} faces on screen");+ // and they stay: many more frames, and none is evicted to be asked for again+ for _ in 0..30 {+ h.step();+ }+ assert!(h.app.renderer.variant_textures() >= kept, "{} pictures kept after more frames, {kept} before", h.app.renderer.variant_textures());+ }++ /// `ui.inspect` says how hard the face scan is allowed to work and whether the window counts as in front, so a slow+ /// scan can be told from a stuck one.+ #[test]+ fn inspect_reports_the_face_scan_pace() {+ let mut h = demo([1000.0, 700.0]);+ let t = Duration::from_secs(10);+ h.settle(SETTLE);+ h.step();+ let r = h.request("ui.inspect", json!({}), t);+ let scan = &r["result"]["faceScan"];+ assert!(["pause", "light", "normal", "full"].contains(&scan["pace"].as_str().unwrap_or("")), "{scan}");+ assert!(scan["focused"].is_boolean() && scan["pending"].is_u64() && scan["indexed"].is_u64(), "{scan}");+ }+/// Profile browser: live variant thumbnails, hover previews in the loupe without touching the/// photo or its history, click applies, the star toggles the favourite.#[test]diff --git a/crates/ui-egui/src/lib.rs b/crates/ui-egui/src/lib.rsindex 7f1e7cdf2..941c15953 100644--- a/crates/ui-egui/src/lib.rs+++ b/crates/ui-egui/src/lib.rs@@ -762,6 +762,7 @@ impl LightcraftApp {ctx.request_repaint_after(std::time::Duration::from_secs(3));}self.session.persist_if_dirty();+ panels::faces::pump(self, ctx);self.collect_screenshots(ctx);self.issue_screenshots(ctx);if self.fonts_ready {@@ -1051,6 +1052,27 @@ pub struct Caches {pub faces_epoch: u64,/// Face model downloads the user started, followed until installed (see `panels::faces::pump`).pub faces_dl_watch: Vec<String>,+ /// Whether the background face indexer is running, how many faces it has embedded, how many photos are left, and+ /// when to ask it again.+ pub faces_active: bool,+ pub faces_indexed: u64,+ pub faces_pending: u64,+ pub faces_next_pump: f64,+ /// When the user last dragged, typed or scrolled, and when they last moved the pointer (egui time).+ pub last_input: f64,+ /// The pace the face scan was last given, and whether the window was in front then (for `ui.inspect`).+ pub faces_pace: &'static str,+ pub faces_in_front: bool,+ pub last_move: f64,+ /// Name suggestions for the photo in the loupe: (photo, catalog revision, faces indexed, suggestions by region).+ pub face_hints: Option<(u64, u64, u64, std::sync::Arc<panels::faces::Hints>)>,+ /// The open person page: (name, catalog revision, faces indexed, when it was asked for, the page).+ pub person_page: Option<(String, u64, u64, f64, std::sync::Arc<panels::person::PersonPage>)>,+ /// The unnamed faces: (catalog revision, faces indexed, when it was asked for, the list).+ pub unnamed: Option<(u64, u64, f64, std::sync::Arc<panels::unnamed::Unnamed>)>,+ /// The most photos the face scan has had left at once since it last finished (the progress bar's whole).+ pub faces_peak: u64,+ person_names: Option<(u64, std::sync::Arc<Vec<String>>)>,/// The grid's date runs, layout and indexes (by the visible list's generation).pub grid: panels::grid::GridCache,/// What the grid did on its frames (benchmarks and tests check unchanged frames stay cheap).@@ -1139,6 +1161,17 @@ impl Caches {self.counts = Some((cat.revision, c));c}+ /// Everyone named on a face in the library (for completing a name as it is typed).+ pub fn person_names(&mut self, cat: &lightcraft_catalog::Catalog) -> std::sync::Arc<Vec<String>> {+ match &self.person_names {+ Some((r, v)) if *r == cat.revision => v.clone(),+ _ => {+ let v = std::sync::Arc::new(cat.people().into_iter().map(|p| p.name).collect::<Vec<_>>());+ self.person_names = Some((cat.revision, v.clone()));+ v+ }+ }+ }/// The By Date tree.pub fn date_groups(&mut self, cat: &lightcraft_catalog::Catalog) -> std::sync::Arc<Vec<lightcraft_catalog::DateGroup>> {match &self.date_groups {diff --git a/crates/ui-egui/src/menus.rs b/crates/ui-egui/src/menus.rsindex ddc33f5a4..3e807fe1a 100644--- a/crates/ui-egui/src/menus.rs+++ b/crates/ui-egui/src/menus.rs@@ -56,6 +56,7 @@ pub const UI_COMMANDS: &[UiCommand] = &[("view.compare", "Compare", Some("Shift+C"), "View"),("view.survey", "Survey", Some("N"), "View"),("view.people", "People", None, "View"),+ ("view.person", "Show Person", None, ""),("view.faceBoxes", "Face Boxes", None, "View"),("view.reference", "Reference View", Some("Shift+R"), "View"),("photo.setReference", "Set as Reference Photo", None, ""),@@ -318,8 +319,18 @@ pub fn run_ui_command(app: &mut LightcraftApp, id: &str, p: &Value) -> Option<Re}"view.people" => {app.ui.view = ViewMode::People;+ app.ui.person_page = None;Ok(json!({"people": app.session.catalog.people().len()}))}+ "view.person" => {+ // {name}: one person's page in the People view: their faces, and the faces that look like them+ let Some(name) = p.get("name").and_then(Value::as_str).map(str::trim).filter(|n| !n.is_empty()) else {+ return Some(Err("view.person: missing `name`".into()));+ };+ app.ui.view = ViewMode::People;+ app.ui.person_page = Some(name.to_string());+ Ok(json!({"person": name}))+ }"view.survey" => {app.ui.view = ViewMode::Survey;Ok(json!({"photos": crate::panels::compare::survey_photos(app).len()}))@@ -365,6 +376,8 @@ pub fn run_ui_command(app: &mut LightcraftApp, id: &str, p: &Value) -> Option<Reapp.ui.slideshow = None;} else if !app.ui.tool.is_empty() {app.ui.tool.clear();+ } else if app.ui.view == ViewMode::People && app.ui.person_page.is_some() {+ app.ui.person_page = None;} else if matches!(app.ui.view, ViewMode::Compare | ViewMode::Survey) {app.ui.view = ViewMode::Detail;} else if app.ui.view == ViewMode::Detail {diff --git a/crates/ui-egui/src/panels/detail.rs b/crates/ui-egui/src/panels/detail.rsindex c9fb643d4..ea3d782b5 100644--- a/crates/ui-egui/src/panels/detail.rs+++ b/crates/ui-egui/src/panels/detail.rs@@ -631,10 +631,26 @@ pub fn show(app: &mut LightcraftApp, ui: &mut egui::Ui) {filter_pill(app, ui, canvas);}if app.ui.face_boxes {- match region_overlay(ui, &p, &map, &photo, d.orientation) {+ // a name box left open on another photo is dropped+ if app.ui.name_edit.as_ref().is_some_and(|e| e.photo != id.0) {+ app.ui.name_edit = None;+ }+ let hints = super::faces::hints_for(app, id.0);+ let people = if app.ui.name_edit.is_some() { app.caches.person_names(&app.session.catalog) } else { Default::default() };+ // while the name box is open and recognition is not set up, the box offers the next step+ let setup = if app.ui.name_edit.is_some() { super::faces::setup(app, ui.ctx()) } else { super::faces::Setup::Running };+ match region_overlay(ui, &p, &map, &photo, d.orientation, hints.as_deref(), &people, &mut app.ui.name_edit, setup) {+ Some(RegionEdit::SetUp) => {+ app.ui.name_edit = None;+ super::faces::take_step(app, ui.ctx(), setup);+ }Some(RegionEdit::Remove(index)) => {let _ = app.run("photo.removeRegion", json!({"id": id.0, "index": index}));}+ Some(RegionEdit::Name(index, name)) => {+ app.ui.name_edit = None;+ let _ = app.run("faces.setName", json!({"id": id.0, "index": index, "name": name}));+ }Some(RegionEdit::Resize(index, r)) => {let _ = app.run("photo.setRegion", json!({"id": id.0, "index": index, "rect": {"x0": r.x0, "y0": r.y0, "x1": r.x1, "y1": r.y1}}));}@@ -740,6 +756,10 @@ enum RegionEdit {Remove(usize),/// A handle drag ended: the region's new box (normalized, upright frame).Resize(usize, lightcraft_geom::Rect),+ /// A name was typed or picked for the face.+ Name(usize, String),+ /// The name box's offer to set up face recognition was pressed.+ SetUp,}/// Handles of a box: (x, y) as fractions of its width and height, and the cursor they show.@@ -765,6 +785,10 @@ fn region_overlay(map: &CanvasMap,photo: &lightcraft_catalog::Photo,orient: lightcraft_geom::Orientation,+ hints: Option<&super::faces::Hints>,+ people: &[String],+ editing: &mut Option<crate::state::NameEdit>,+ setup: super::faces::Setup,) -> Option<RegionEdit> {let t = Tokens::get(p.ctx());let clip = p.clip_rect();@@ -840,9 +864,20 @@ fn region_overlay(edit = Some(RegionEdit::Remove(index));}}- let Some(name) = &r.name else { continue };+ // what the label says: the name, a guess to confirm ("Jane Doe?"), or on hover an invitation to name the face+ let is_face = r.kind == lightcraft_meta::RegionKind::Face;+ let hint = hints.and_then(|h| h.by_index.get(&index));+ let hovered = pointer.is_some_and(|h| rect.expand(3.0).contains(h));+ let editing_this = editing.as_ref().is_some_and(|e| e.photo == photo.id.0 && e.index == index);+ let (text, text_color, fill_alpha, invitation) = match (&r.name, hint.and_then(|h| h.suggestion.as_ref())) {+ (Some(n), _) => (n.clone(), Color32::from_gray(225), 235, false),+ (None, Some((n, _))) if is_face => (format!("{n}?"), Color32::from_gray(185), 200, false),+ (None, None) if is_face && live.is_none() => ("Add name".to_string(), Color32::from_gray(150), 190, true),+ _ => continue,+ };// the name in a dark label with a caret, centred above the box (below it when there is no room)- let g = p.layout_no_wrap(name.clone(), t.font(13.0), Color32::from_gray(225));+ let name = &text;+ let g = p.layout_no_wrap(name.clone(), t.font(13.0), text_color);let (pad, caret) = (vec2(14.0, 7.0), 5.0);let size = g.size() + pad * 2.0;let above = rect.top() - caret - size.y >= clip.top();@@ -853,12 +888,34 @@ fn region_overlay(};let left = (rect.center().x - size.x / 2.0).clamp(clip.left(), (clip.right() - size.x).max(clip.left()));let label = Rect::from_min_size(pos2(left, top), size);- let fill = Color32::from_rgba_unmultiplied(56, 56, 56, 235);+ // the invitation shows while the pointer is on the box *or on the label* (and the gap between), so it can be reached+ if invitation && !(hovered || editing_this || pointer.is_some_and(|h| rect.union(label).expand(3.0).contains(h))) {+ continue;+ }+ let fill = Color32::from_rgba_unmultiplied(56, 56, 56, fill_alpha);p.rect_filled(label, 3.0, fill);p.rect_stroke(label, 3.0, Stroke::new(1.0, Color32::from_black_alpha(160)), StrokeKind::Inside);let cx = rect.center().x.clamp(label.left() + caret + 4.0, label.right() - caret - 4.0);p.add(egui::Shape::convex_polygon(vec![pos2(cx - caret, base), pos2(cx + caret, base), pos2(cx, tip)], fill, Stroke::NONE));- p.galley(label.min + pad, g, Color32::from_gray(225));+ p.galley(label.min + pad, g, text_color);+ if is_face && live.is_none() {+ // a click on the label opens the name box, filled with the name or the guess+ let resp = ui.interact(label, egui::Id::new(("region-label", index)), Sense::click());+ register(ui.ctx(), format!("regionLabel:{index}"), label);+ if resp.on_hover_text("Click to name this face").clicked() {+ let start = r.name.clone().or_else(|| hint.and_then(|h| h.suggestion.as_ref().map(|(n, _)| n.clone()))).unwrap_or_default();+ *editing = Some(crate::state::NameEdit { photo: photo.id.0, index, text: start, fresh: true });+ }+ }+ if editing_this && let Some(e) = editing.as_mut() {+ let candidates = hint.map(|h| h.candidates.as_slice()).unwrap_or(&[]);+ match super::faces::name_editor(ui.ctx(), pos2(label.left(), rect.bottom() + 8.0), e, candidates, people, setup) {+ super::faces::Editor::Submit(name) => edit = Some(RegionEdit::Name(index, name)),+ super::faces::Editor::Setup => edit = Some(RegionEdit::SetUp),+ super::faces::Editor::Cancel => *editing = None,+ super::faces::Editor::Open => {}+ }+ }}edit}diff --git a/crates/ui-egui/src/panels/faces.rs b/crates/ui-egui/src/panels/faces.rsindex 423a849b5..3117eb607 100644--- a/crates/ui-egui/src/panels/faces.rs+++ b/crates/ui-egui/src/panels/faces.rs@@ -3,9 +3,10 @@//!//! Adding a model is: pick or drop a `.onnx` file → the engine inspects it (`faces.models.inspect`) →//! this dialog shows what it is and its terms → the user ticks "I accept" → `faces.models.install`.-//! LightCraft fetches a model only when the user presses Download on a model it has a pinned address for: the same-//! dialog shows its terms first, and accepting them downloads it (`faces.models.download`: checked against its size and-//! SHA-256, then installed). "Open page" opens the model's own page in the browser.+//! LightCraft fetches a model only when the user presses Download on a model it has a pinned address for: the+//! same dialog shows its terms first, and accepting them downloads it (`faces.models.download`: checked against its+//! size and SHA-256), installs it, starts using it and switches recognition on, with no further question. "Open+//! page" opens the model's own page in the browser for anything else.use egui::RichText;use serde_json::{Value, json};@@ -15,6 +16,37 @@ use crate::LightcraftApp;use crate::theme::Tokens;use crate::widgets::register;+/// How hard the background scan may work (the engine's `faces.pump` pace), from what the user is doing: nothing new+/// while they drag, type or scroll; one photo at a time while they are around or the window is minimized; half the+/// machine once they have been idle for a few seconds, or while another app has the keyboard but this window is still+/// on screen; most of it while they are looking at the scan's progress.+fn scan_pace(app: &LightcraftApp, ctx: &egui::Context, now: f64) -> (&'static str, bool) {+ let (focused, minimized) = ctx.input(|i| (i.focused, i.raw.viewports.get(&i.raw.viewport_id).and_then(|v| v.minimized).unwrap_or(false)));+ let watching =+ matches!(&app.ui.dialog, Some(crate::state::Dialog::Settings { tab }) if tab == "faces") || app.ui.view == crate::state::ViewMode::People;+ (pace_for(focused, minimized, now - app.caches.last_input, now - app.caches.last_move, watching), focused && !minimized)+}++/// The pace for a window that has the keyboard (or not) and is minimized (or not), `worked` seconds after the user last+/// dragged, typed or scrolled and `moved` seconds after they last moved the pointer, while they are (or are not) looking+/// at the scan's progress.+fn pace_for(focused: bool, minimized: bool, worked: f64, moved: f64, watching: bool) -> &'static str {+ if minimized {+ "light"+ } else if !focused {+ // still on screen, but the user is working in another app: no input here to get in the way of+ "normal"+ } else if worked < 0.4 {+ "pause"+ } else if moved < 3.0 {+ "light"+ } else if watching {+ "full"+ } else {+ "normal"+ }+}+/// How long a read of the model list is reused (it is a few small files, but not for every frame).const REFRESH_SECS: f64 = 1.5;@@ -23,20 +55,108 @@ fn list_id() -> egui::Id {}/// The engine's model list, re-read at most every [`REFRESH_SECS`], or at once after an action changed it.-fn models(app: &mut LightcraftApp, ui: &egui::Ui) -> Value {- let now = ui.input(|i| i.time);+fn models(app: &mut LightcraftApp, ctx: &egui::Context) -> Value {+ let now = ctx.input(|i| i.time);let epoch = app.caches.faces_epoch;- if let Some((e, at, v)) = ui.data(|d| d.get_temp::<(u64, f64, Value)>(list_id()))+ if let Some((e, at, v)) = ctx.data(|d| d.get_temp::<(u64, f64, Value)>(list_id()))&& e == epoch&& now - at < REFRESH_SECS{return v;}let v = app.run("faces.models.list", json!({})).unwrap_or(Value::Null);- ui.data_mut(|d| d.insert_temp(list_id(), (epoch, now, v.clone())));+ ctx.data_mut(|d| d.insert_temp(list_id(), (epoch, now, v.clone())));v}+/// What stands between the user and name suggestions, for the prompts where faces are shown or named.+#[derive(Clone, Copy, Debug, PartialEq, Eq)]+pub enum Setup {+ /// Recognition is on (or about to start): nothing to offer.+ Running,+ /// A model is installed and chosen; only the switch is off.+ TurnOn,+ /// No model to use yet: Settings ▸ Faces has the download.+ GetModel,+ /// This build cannot keep or run models: nothing to offer.+ Unavailable,+}++/// Where the user stands with face recognition. Cheap: the model list is read at most every 1.5 s, and not at all while+/// recognition runs.+pub fn setup(app: &mut LightcraftApp, ctx: &egui::Context) -> Setup {+ if app.caches.faces_active {+ return Setup::Running;+ }+ let list = models(app, ctx);+ if list["dir"].is_null() || list["runtime"].as_bool() != Some(true) {+ return Setup::Unavailable;+ }+ let chosen = list["embedder"].as_str();+ let usable = chosen.is_some_and(|id| list["models"].as_array().into_iter().flatten().any(|m| m["id"] == id && m["installed"] == true));+ match (usable, list["enabled"].as_bool() == Some(true)) {+ (true, true) => Setup::Running,+ (true, false) => Setup::TurnOn,+ (false, _) => Setup::GetModel,+ }+}++impl Setup {+ /// The sentence and the button label that offer the next step; none when there is nothing to do.+ pub fn prompt(self) -> Option<(&'static str, &'static str)> {+ match self {+ Setup::TurnOn => Some(("Face recognition is off. Turn it on to get name suggestions and see look-alike faces together.", "Turn on")),+ Setup::GetModel => {+ Some(("Face recognition needs a model to suggest names: a one-time download, in Settings.", "Set up face recognition"))+ }+ Setup::Running | Setup::Unavailable => None,+ }+ }++ /// The same offer as one line of a small box (the name box in the loupe).+ pub fn short(self) -> Option<&'static str> {+ match self {+ Setup::TurnOn => Some("Turn on name suggestions"),+ Setup::GetModel => Some("Set up name suggestions…"),+ Setup::Running | Setup::Unavailable => None,+ }+ }+}++/// The prompt's button: switch recognition on, or open Settings ▸ Faces, where the model is one click away.+pub fn take_step(app: &mut LightcraftApp, ctx: &egui::Context, step: Setup) {+ match step {+ Setup::TurnOn => {+ if app.run("faces.enable", json!({"enabled": true})).is_ok() {+ app.caches.faces_epoch += 1;+ app.ui.status = "Face recognition is on".into();+ app.toast(ctx, "Face recognition is on");+ }+ }+ Setup::GetModel => {+ let _ = app.run("app.settings", json!({"tab": "faces"}));+ }+ Setup::Running | Setup::Unavailable => {}+ }+}++/// A slim line under a view's heading offering to set recognition up, shown only while it is not set up.+pub fn setup_banner(app: &mut LightcraftApp, ui: &mut egui::Ui) {+ let step = setup(app, ui.ctx());+ let Some((message, button)) = step.prompt() else { return };+ let t = Tokens::get(ui.ctx());+ egui::Frame::NONE.inner_margin(egui::Margin::symmetric(12, 6)).show(ui, |ui| {+ ui.horizontal_wrapped(|ui| {+ ui.label(RichText::new(message).font(t.font(12.5)).color(t.text_dim));+ let r = ui.button(button);+ register(ui.ctx(), "faces:setup", r.rect);+ if r.clicked() {+ take_step(app, ui.ctx(), step);+ }+ });+ });+}+/// A download-style size: decimal megabytes, as file managers and model pages show them.fn mb(bytes: Option<u64>) -> String {match bytes {@@ -78,7 +198,7 @@ fn open_page(app: &mut LightcraftApp, url: &str) {/// The Settings ▸ Faces tab.pub fn settings_tab(app: &mut LightcraftApp, ui: &mut egui::Ui, t: &Tokens) {- let list = models(app, ui);+ let list = models(app, ui.ctx());heading(ui, t, "Face recognition");if list["dir"].is_null() {hint(ui, t, "This build has nowhere to keep face models, so the model list is not available. (The desktop app has.)");@@ -89,27 +209,28 @@ pub fn settings_tab(app: &mut LightcraftApp, ui: &mut egui::Ui, t: &Tokens) {let _ = app.run("faces.enable", json!({"enabled": on}));app.caches.faces_epoch += 1;}- hint(- ui,- t,- "Off by default. LightCraft already reads the face names other apps wrote into your photos; this is for models that suggest who is in a photo. Everything stays on your computer.",- );- if list["runtime"].as_bool() != Some(true) {- hint(ui, t, "Models can be added and chosen now. Running them arrives in a later update.");+ hint(ui, t, "Suggests who is in a photo from the faces you have named. Everything stays on your computer.");+ let runtime = list["runtime"].as_bool() == Some(true);+ if !runtime {+ hint(ui, t, "This build cannot run recognition models: they can be added and chosen, not used.");+ } else if on && list["embedder"].is_null() {+ hint(ui, t, "Choose a model below (Use) to start.");+ } else if on && app.caches.faces_active {+ scan_progress(app, ui, t);}let all: Vec<Value> = list["models"].as_array().cloned().unwrap_or_default();let downloads =app.session.execute("faces.models.downloads", &json!({})).map(|v| v["downloads"].as_array().cloned().unwrap_or_default()).unwrap_or_default();heading(ui, t, "Face detector");for m in all.iter().filter(|m| m["role"] == "detector") {- model_row(app, ui, t, m, downloads.iter().find(|d| d["id"] == m["id"]));+ model_row(app, ui, t, m, downloads.iter().find(|d| d["id"] == m["id"]), runtime);}if !all.iter().any(|m| m["role"] == "detector" && m["installed"] == true) {hint(ui, t, "Detect Faces (Photo menu) needs it: a small one-time download.");}heading(ui, t, "Recognition models");for m in all.iter().filter(|m| m["role"] == "embedder") {- model_row(app, ui, t, m, downloads.iter().find(|d| d["id"] == m["id"]));+ model_row(app, ui, t, m, downloads.iter().find(|d| d["id"] == m["id"]), runtime);}ui.add_space(6.0);ui.horizontal(|ui| {@@ -121,14 +242,37 @@ pub fn settings_tab(app: &mut LightcraftApp, ui: &mut egui::Ui, t: &Tokens) {}ui.label(RichText::new("or drop a .onnx file on the window").color(t.text_dim));});+ // what is wrong with the user's own catalog (catalog.json in the models folder), if anything+ for e in list["catalog"]["errors"].as_array().into_iter().flatten().filter_map(Value::as_str).take(3) {+ ui.add(egui::Label::new(RichText::new(format!("catalog.json: {e}")).font(t.font(11.5)).color(t.caution)).wrap());+ }+ if let Some(d) = all.iter().find(|m| m["role"] == "detector") {+ ui.add_space(4.0);+ hint(ui, t, &format!("Faces are found by {} ({}, included).", d["name"].as_str().unwrap_or("the detector"), mb(d["sizeBytes"].as_u64())));+ }}-fn model_row(app: &mut LightcraftApp, ui: &mut egui::Ui, t: &Tokens, m: &Value, dl: Option<&Value>) {+/// How the scan is going: a bar while photos are left, then how many faces it learned.+fn scan_progress(app: &mut LightcraftApp, ui: &mut egui::Ui, t: &Tokens) {+ let (left, peak, faces) = (app.caches.faces_pending, app.caches.faces_peak.max(1), app.caches.faces_indexed);+ if left == 0 {+ hint(ui, t, &format!("{faces} faces learned."));+ return;+ }+ let text = format!("Scanning photos for faces: {left} left · {faces} faces so far");+ let bar = ui.add(+ egui::ProgressBar::new((1.0 - left as f32 / peak as f32).clamp(0.0, 1.0)).desired_width(380.0).text(RichText::new(text).font(t.font(11.5))),+ );+ register(ui.ctx(), "faces:scanProgress", bar.rect);+}++fn model_row(app: &mut LightcraftApp, ui: &mut egui::Ui, t: &Tokens, m: &Value, dl: Option<&Value>, can_run: bool) {let id = m["id"].as_str().unwrap_or("").to_string();- let (installed, selected) = (m["installed"].as_bool() == Some(true), m["selected"].as_bool() == Some(true));let detector = m["role"] == "detector";+ let (installed, selected) = (m["installed"].as_bool() == Some(true), m["selected"].as_bool() == Some(true));let dl_state = if installed { None } else { dl.and_then(|d| d["state"].as_str()) };- let host = m["downloadHost"].as_str().unwrap_or("").to_string();+ // a build that cannot run recognition models has no use for a download+ let host = if can_run { m["downloadHost"].as_str().unwrap_or("").to_string() } else { String::new() };egui::Frame::NONE.inner_margin(egui::Margin::symmetric(0, 3)).show(ui, |ui| {ui.horizontal(|ui| {ui.vertical(|ui| {@@ -149,6 +293,19 @@ fn model_row(app: &mut LightcraftApp, ui: &mut egui::Ui, t: &Tokens, m: &Value,} else {t.caution}));+ // how fast it is, as times faster than a ResNet-100 model (milliseconds would depend on the computer),+ // with whether it passed its test when it has been installed+ let speed = m["speedText"].as_str().filter(|s| !s.is_empty());+ let (line, ok) = match m["accepted"]["selfTest"].as_object() {+ Some(test) if test.get("ok").and_then(Value::as_bool) == Some(true) => {+ (Some(speed.map_or("Works".to_string(), |s| format!("Works · {s}"))), true)+ }+ Some(_) => (Some("Failed its last test".to_string()), false),+ None => (speed.map(str::to_string), true),+ };+ if let Some(line) = line {+ ui.label(RichText::new(line).font(t.font(11.5)).color(if ok { t.text_dim } else { t.caution }));+ }match (dl_state, dl) {(Some("running"), Some(d)) => {let (got, total) = (d["bytes"].as_u64().unwrap_or(0), d["total"].as_u64().unwrap_or(0));@@ -187,34 +344,41 @@ fn model_row(app: &mut LightcraftApp, ui: &mut egui::Ui, t: &Tokens, m: &Value,app.caches.faces_epoch += 1;}}- } else if dl_state == Some("running") {- let r = ui.button("Cancel");- register(ui.ctx(), format!("faces:cancelDownload:{id}"), r.rect);- if r.clicked() {- let _ = app.run("faces.models.downloadCancel", json!({"id": id}));- app.caches.faces_dl_watch.retain(|w| w != &id);- }- } else if dl_state != Some("done") {- // (a download that has just arrived is installed within a moment: nothing to press)- if let Some(url) = m["source"].as_str() {- let r = ui- .button("Open page")- .on_hover_text("Opens the model's own page in your browser, to read about it or get the file yourself.");- register(ui.ctx(), format!("faces:get:{id}"), r.rect);- if r.clicked() {- open_page(app, url);+ } else {+ match dl_state {+ Some("running") => {+ let r = ui.button("Cancel");+ register(ui.ctx(), format!("faces:cancelDownload:{id}"), r.rect);+ if r.clicked() {+ let _ = app.run("faces.models.downloadCancel", json!({"id": id}));+ app.caches.faces_dl_watch.retain(|w| w != &id);+ }}- }- if !host.is_empty() {- let label = if dl_state == Some("failed") { "Try again" } else { "Download" };- let tip = format!(- "Shows the model's terms, then downloads {} from {host} and installs it. Nothing else is sent.",- mb(m["sizeBytes"].as_u64())- );- let r = ui.button(label).on_hover_text(tip);- register(ui.ctx(), format!("faces:download:{id}"), r.rect);- if r.clicked() {- open_download_dialog(app, m);+ // the engine installs it within a moment; nothing to press+ Some("done") => {}+ _ => {+ if let Some(url) = m["source"].as_str() {+ let r = ui+ .button("Open page")+ .on_hover_text("Opens the model's own page in your browser, to read about it or get the file yourself.");+ register(ui.ctx(), format!("faces:get:{id}"), r.rect);+ if r.clicked() {+ open_page(app, url);+ }+ }+ if !host.is_empty() {+ let label = if dl_state == Some("failed") { "Try again" } else { "Download" };+ let tip = format!(+ "Shows the model's terms, then downloads {} from {host} and installs it{}. Nothing else is sent.",+ mb(m["sizeBytes"].as_u64()),+ if detector { "" } else { ", starts using it" }+ );+ let r = ui.button(label).on_hover_text(tip);+ register(ui.ctx(), format!("faces:download:{id}"), r.rect);+ if r.clicked() {+ open_download_dialog(app, m);+ }+ }}}}@@ -249,11 +413,16 @@ pub fn model_dialog(app: &mut LightcraftApp, ui: &mut egui::Ui, t: &Tokens, infoui.label(RichText::new("Already installed. Installing again is harmless.").color(t.text_dim));}let download = info["download"].is_string();+ let detector = m["role"] == "detector";if download {let host = info["host"].as_str().unwrap_or("its own repository");ui.add(egui::Label::new(- RichText::new(format!("Downloads from {host}. Once it has arrived and checked out, LightCraft installs it.")).color(t.text_label),+ RichText::new(format!(+ "Downloads from {host}. Once it has arrived and checked out, LightCraft installs it{}.",+ if detector { "" } else { ", starts using it and turns face recognition on" }+ ))+ .color(t.text_label),).wrap(),);@@ -325,22 +494,204 @@ pub fn open_dialog(app: &mut LightcraftApp, path: &str) -> Result<Value, String>}/// The dialog's OK (the engine refuses without the acceptance): for a download, start it and watch it; for a file,-/// install it.+/// install it. Either way the model ends up installed, in use, with recognition on.pub fn install(app: &mut LightcraftApp, path: &str, info: &Value, accepted: bool) -> Result<Value, String> {if !accepted {return Err("Tick the box to accept the model's terms first".into());}app.caches.faces_epoch += 1;+ let back_to_settings = |app: &mut LightcraftApp| app.ui.dialog = Some(crate::state::Dialog::Settings { tab: "faces".into() });if let Some(id) = info["download"].as_str() {let r = app.run("faces.models.download", json!({"id": id, "acknowledged": true}));if r.is_ok() {app.caches.faces_dl_watch.push(id.to_string());- // the progress is in Settings: go there- app.ui.dialog = Some(crate::state::Dialog::Settings { tab: "faces".into() });+ // the progress is in Settings: stay there+ back_to_settings(app);}return r;}- app.run("faces.models.install", json!({"path": path, "acknowledged": true}))+ let r = app.run("faces.models.install", json!({"path": path, "acknowledged": true}));+ if let Ok(v) = &r {+ let name = v["installed"]["name"].as_str().unwrap_or("The model");+ app.ui.status = if v["installed"]["role"] == "detector" {+ format!("{name} is installed")+ } else {+ format!("{name} is installed and in use: face recognition is on")+ };+ back_to_settings(app);+ }+ r+}++// ---------------------------------------------------------------------------------------------------+// Naming faces in the loupe++/// A suggestion for one unnamed face: the best guess (when it passes the bar) and the closest few people.+#[derive(Clone, Debug, Default, PartialEq)]+pub struct Hint {+ pub suggestion: Option<(String, f32)>,+ pub candidates: Vec<(String, f32)>,+}++/// Suggestions for the unnamed faces of one photo, by region index.+#[derive(Clone, Debug, Default, PartialEq)]+pub struct Hints {+ pub by_index: std::collections::HashMap<usize, Hint>,+}++/// Suggestions for `photo`'s unnamed faces from what is already indexed (nothing is embedded for this: the background+/// indexer does that). `None` while recognition is off. Asked again only when the catalog or the index changed.+pub fn hints_for(app: &mut LightcraftApp, photo: u64) -> Option<std::sync::Arc<Hints>> {+ if !app.caches.faces_active {+ return None;+ }+ let (rev, indexed) = (app.session.catalog.revision, app.caches.faces_indexed);+ if let Some((p, r, i, h)) = &app.caches.face_hints+ && (*p, *r, *i) == (photo, rev, indexed)+ {+ return Some(h.clone());+ }+ let answer = app.session.execute("faces.suggest", &json!({"ids": [photo], "budgetMs": 0})).ok()?;+ let pair = |v: &Value| Some((v["name"].as_str()?.to_string(), v["score"].as_f64()? as f32));+ let mut hints = Hints::default();+ for f in answer["photos"][0]["faces"].as_array().into_iter().flatten() {+ let Some(index) = f["index"].as_u64().and_then(|i| usize::try_from(i).ok()) else { continue };+ hints.by_index.insert(+ index,+ Hint { suggestion: pair(&f["suggestion"]), candidates: f["candidates"].as_array().into_iter().flatten().filter_map(pair).collect() },+ );+ }+ let hints = std::sync::Arc::new(hints);+ app.caches.face_hints = Some((photo, rev, indexed, hints.clone()));+ Some(hints)+}++/// What the name box did this frame.+pub enum Editor {+ Open,+ Submit(String),+ Cancel,+ /// The offer to set up recognition was pressed.+ Setup,+}++/// The inline name box under a face: type a name (completed from the people already named), pick one of the+/// suggestions, Enter to confirm, Escape to cancel. While recognition is not set up it offers the next step (`setup`).+pub fn name_editor(+ ctx: &egui::Context,+ at: egui::Pos2,+ edit: &mut crate::state::NameEdit,+ candidates: &[(String, f32)],+ people: &[String],+ setup: Setup,+) -> Editor {+ let t = Tokens::get(ctx);+ let mut outcome = Editor::Open;+ let opening = edit.fresh;+ let shown = egui::Area::new(egui::Id::new("face-name-editor")).order(egui::Order::Foreground).fixed_pos(at).constrain(true).show(ctx, |ui| {+ egui::Frame::popup(ui.style()).show(ui, |ui| {+ ui.set_min_width(230.0);+ let r = ui.add(egui::TextEdit::singleline(&mut edit.text).hint_text("Name").desired_width(220.0));+ register(ui.ctx(), "field:faceName", r.rect);+ if edit.fresh {+ r.request_focus();+ edit.fresh = false;+ }+ if r.lost_focus() && ui.input(|i| i.key_pressed(egui::Key::Enter)) {+ outcome = Editor::Submit(edit.text.trim().to_string());+ }+ let typed = edit.text.trim().to_lowercase();+ let mut offered: Vec<(String, Option<f32>)> = candidates.iter().take(3).map(|(n, s)| (n.clone(), Some(*s))).collect();+ if !typed.is_empty() {+ for p in people.iter().filter(|p| p.to_lowercase().starts_with(&typed)) {+ if offered.len() >= 7 {+ break;+ }+ if !offered.iter().any(|(n, _)| n.eq_ignore_ascii_case(p)) {+ offered.push((p.clone(), None));+ }+ }+ }+ for (name, score) in offered {+ let label = match score {+ Some(s) => format!("{name} {:.0}%", (s * 100.0).max(0.0)),+ None => name.clone(),+ };+ let b = ui.add(egui::Button::new(RichText::new(label).color(t.text_label)).frame(false).min_size(egui::vec2(220.0, 0.0)));+ register(ui.ctx(), format!("faceName:pick:{name}"), b.rect);+ if b.clicked() {+ outcome = Editor::Submit(name);+ }+ }+ if let Some(label) = setup.short() {+ let b = ui.add(egui::Button::new(RichText::new(label).font(t.font(12.0)).color(t.accent)).frame(false));+ register(ui.ctx(), "faces:setup", b.rect);+ if b.clicked() {+ outcome = Editor::Setup;+ }+ }+ ui.label(RichText::new("Enter to confirm, Esc to cancel").font(t.font(11.0)).color(t.text_dim));+ });+ });+ if ctx.input(|i| i.key_pressed(egui::Key::Escape)) {+ outcome = Editor::Cancel;+ }+ // a click anywhere else closes it (not on the frame it opened, which is the click that opened it)+ if !opening && ctx.input(|i| i.pointer.any_pressed()) && ctx.input(|i| i.pointer.interact_pos()).is_some_and(|p| !shown.response.rect.contains(p))+ {+ outcome = Editor::Cancel;+ }+ outcome+}++/// Called every frame: keeps the background face indexer going while recognition is on, and notes whether it is+/// running, how many faces it has done and how many photos are left (the loupe's suggestions depend on it).+pub fn pump(app: &mut LightcraftApp, ctx: &egui::Context) {+ watch_downloads(app, ctx);+ let now = ctx.input(|i| i.time);+ let (working, moving) = ctx.input(|i| {+ let key_or_scroll = i.events.iter().any(|e| matches!(e, egui::Event::Key { .. } | egui::Event::Text(_) | egui::Event::MouseWheel { .. }));+ (i.pointer.any_down() || key_or_scroll, i.pointer.is_moving())+ });+ if working {+ app.caches.last_input = now;+ }+ if moving || working {+ app.caches.last_move = now;+ }+ if now < app.caches.faces_next_pump {+ return;+ }+ let (pace, in_front) = scan_pace(app, ctx, now);+ app.caches.faces_pace = pace;+ app.caches.faces_in_front = in_front;+ let Ok(v) = app.session.execute("faces.pump", &json!({"pace": pace})) else { return };+ app.caches.faces_active = v["active"] == true;+ app.caches.faces_indexed = v["indexedFaces"].as_u64().unwrap_or(0);+ app.caches.faces_pending = v["pendingPhotos"].as_u64().unwrap_or(0);+ // the progress bar's whole: the most photos that were left at once since the scan last finished+ app.caches.faces_peak = if app.caches.faces_pending == 0 { 0 } else { app.caches.faces_peak.max(app.caches.faces_pending) };+ // Frames are drawn only when something asks for one, so nothing here wakes the window needlessly: while there is work it+ // asks to be called again in 50 ms (the progress in Settings, the next photos for the workers); with recognition on and+ // nothing to do it looks again every few seconds (a few wake-ups a minute); with recognition off it asks for nothing+ // (and is called on every frame, which is cheap: whatever makes a frame, such as switching recognition on, is seen at once).+ let busy = app.caches.faces_active && (app.caches.faces_pending > 0 || v["inFlight"].as_u64().unwrap_or(0) > 0);+ let wait = match (busy, app.caches.faces_active) {+ (true, _) => Some(0.05),+ (false, true) => Some(5.0),+ (false, false) => None,+ };+ app.caches.faces_next_pump = now+ + if busy {+ 0.05+ } else if app.caches.faces_active {+ 1.0+ } else {+ 0.0+ };+ if let Some(secs) = wait {+ ctx.request_repaint_after(std::time::Duration::from_secs_f64(secs));+ }}/// Sort the watched downloads: those that have been installed (to announce), and those still to watch (running, or@@ -357,9 +708,9 @@ fn due(rows: &[Value], watched: Vec<String>) -> (Vec<String>, Vec<String>) {(installed, keep)}-/// Called every frame: follows the downloads the user started. Keeps redrawing while one runs (the progress bar), installs-/// what has arrived (the engine does that when asked) and says so when a model is installed.-pub fn pump(app: &mut LightcraftApp, ctx: &egui::Context) {+/// Follows the downloads the user started: keeps redrawing while one runs (the progress bar), and says so when a+/// model has been installed, is in use and recognition is on (the engine does all of that by itself).+fn watch_downloads(app: &mut LightcraftApp, ctx: &egui::Context) {if app.caches.faces_dl_watch.is_empty() {return;}@@ -381,7 +732,12 @@ pub fn pump(app: &mut LightcraftApp, ctx: &egui::Context) {.unwrap_or_else(|| id.clone());let _ = app.session.execute("faces.models.downloadCancel", &json!({"id": id}));app.caches.faces_epoch += 1;- let text = format!("{name} is installed");+ let detector = app+ .session+ .execute("faces.models.list", &json!({}))+ .ok()+ .is_some_and(|l| l["models"].as_array().is_some_and(|a| a.iter().any(|m| m["id"] == id.as_str() && m["role"] == "detector")));+ let text = if detector { format!("{name} is installed") } else { format!("{name} is installed and in use: face recognition is on") };app.ui.status = text.clone();app.toast(ctx, text);}@@ -392,6 +748,20 @@ pub fn pump(app: &mut LightcraftApp, ctx: &egui::Context) {mod tests {use super::*;+ #[test]+ fn the_scan_works_hard_only_when_nobody_is_in_the_way() {+ // (focused, minimized, seconds since a drag/key/scroll, seconds since the pointer moved, watching the progress)+ assert_eq!(pace_for(true, false, 0.1, 0.1, true), "pause", "dragging a slider: nothing new, even if watching");+ assert_eq!(pace_for(true, false, 2.0, 0.2, false), "light", "the pointer is moving: one photo at a time");+ assert_eq!(pace_for(true, false, 5.0, 2.9, true), "light");+ assert_eq!(pace_for(true, false, 5.0, 4.0, false), "normal", "idle for a few seconds: half the machine");+ assert_eq!(pace_for(true, false, 60.0, 60.0, true), "full", "idle and looking at the progress: most of it");+ // minimized: gentle, whatever the user did last+ assert_eq!([pace_for(true, true, 0.0, 0.0, true), pace_for(false, true, 99.0, 99.0, true)], ["light", "light"]);+ // on screen behind another app: the user is elsewhere, so the scan gets on with it, however recent the last input+ assert_eq!([pace_for(false, false, 0.0, 0.0, true), pace_for(false, false, 99.0, 99.0, false)], ["normal", "normal"]);+ }+fn row(id: &str, state: &str) -> Value {json!({"id": id, "state": state})}diff --git a/crates/ui-egui/src/panels/mod.rs b/crates/ui-egui/src/panels/mod.rsindex cd2ac1a3b..e95f8c7b1 100644--- a/crates/ui-egui/src/panels/mod.rs+++ b/crates/ui-egui/src/panels/mod.rs@@ -16,6 +16,7 @@ pub mod library_problem;pub mod masking;pub mod notices;pub mod people;+pub mod person;pub mod presets;pub mod profiles;pub mod right;@@ -24,6 +25,7 @@ pub mod second;pub mod settings;pub mod strip;pub mod topbar;+pub mod unnamed;use egui::{Align2, Rect, pos2, vec2};diff --git a/crates/ui-egui/src/panels/people.rs b/crates/ui-egui/src/panels/people.rsindex a011af78d..7a7307efd 100644--- a/crates/ui-egui/src/panels/people.rs+++ b/crates/ui-egui/src/panels/people.rs@@ -1,7 +1,11 @@-//! People: a card per person named on faces (read from XMP): a close-up of their largest face, the-//! name and how many photos they are in. A click shows that person's photos in the grid.+//! People: a card per person named on faces (read from XMP): a close-up of their face and how many photos they are in+//! (a small number on the picture). A click opens that person's page (`person.rs`): only their cropped faces, and the+//! faces that look like them. Below the people, the faces nobody has named yet (`unnamed.rs`), to be named a group at a+//! time.//!-//! Only the rows on screen ask for a face render (the engine caches them, memory and disk).+//! The thumbnail-size slider in the bottom bar sizes the faces here too (with its own smaller and larger limits), and+//! every face is cut from the same kind of box (the detector's, once the scan has looked at it), so faces are shown+//! equally close. Only the rows on screen ask for a face render (the engine caches them, memory and disk).use egui::{Align2, Color32, Rect, Sense, Stroke, StrokeKind, pos2, vec2};use lightcraft_catalog::Person;@@ -11,25 +15,69 @@ use crate::LightcraftApp;use crate::theme::Tokens;use crate::widgets::register;-/// A card's face picture (points); the name and count sit below it.-const CARD: f32 = 150.0;-const LABEL_H: f32 = 44.0;-const GAP: f32 = 16.0;-const PAD: f32 = 20.0;+const GAP: f32 = 14.0;+pub(super) const PAD: f32 = 20.0;const HEADER_H: f32 = 44.0;+/// The name under a card's picture.+const NAME_H: f32 = 26.0;+/// The unnamed-faces heading and its hints.+const SECTION_H: f32 = 70.0;++/// `thumb` (the thumbnail slider, 90 to 480) as a position between `low` and `high`.+fn between(thumb: f32, low: f32, high: f32) -> f32 {+ let f = (thumb - 90.0) / (480.0 - 90.0);+ low + if f.is_finite() { f.clamp(0.0, 1.0) } else { 0.4 } * (high - low)+}++/// A person card's face picture (points) for the thumbnail slider's value.+pub fn card_edge(thumb: f32) -> f32 {+ between(thumb, 84.0, 260.0)+}++/// A face tile's picture (points), for the unnamed faces and a person's page.+pub fn tile_edge(thumb: f32) -> f32 {+ between(thumb, 56.0, 180.0)+}++/// How many `edge`-wide cells (with the gap between them) fit across `width`.+pub(super) fn columns(width: f32, edge: f32) -> usize {+ (((width - PAD * 2.0 + GAP) / (edge + GAP)).floor() as usize).max(1)+}++pub(super) const TILE_GAP: f32 = GAP;++/// `text` shortened with "…" until it is no wider than `max_w` in `font` (measured, not guessed).+pub(super) fn fit(painter: &egui::Painter, text: &str, font: egui::FontId, max_w: f32) -> String {+ let width = |s: &str| painter.layout_no_wrap(s.to_string(), font.clone(), Color32::WHITE).size().x;+ if width(text) <= max_w {+ return text.to_string();+ }+ let chars: Vec<char> = text.chars().collect();+ (1..chars.len()).rev().map(|n| chars[..n].iter().collect::<String>() + "…").find(|s| width(s) <= max_w).unwrap_or_else(|| "…".to_string())+}pub fn show(app: &mut LightcraftApp, ui: &mut egui::Ui) {+ if let Some(name) = app.ui.person_page.clone() {+ return super::person::show(app, ui, &name);+ }let t = Tokens::get(ui.ctx());+ let now = ui.input(|i| i.time);+ let thumb = app.ui.thumb_size;let people = app.caches.people(&app.session.catalog, &app.session.filter);let (head, _) = ui.allocate_exact_size(vec2(ui.available_width(), HEADER_H), Sense::hover());ui.painter().text(pos2(head.left() + PAD, head.center().y), Align2::LEFT_CENTER, crate::i18n::tr("Named People"), t.semibold(15.0), t.text);ui.painter().text(pos2(head.right() - PAD, head.center().y), Align2::RIGHT_CENTER, people.len().to_string(), t.font(13.0), t.text_dim);+ // until face recognition is set up: what it takes, and a button that does the next step+ super::faces::setup_banner(app, ui);// the filters narrowing the list (a date, a keyword…), removable herelet chips = lightcraft_engine::filter_chips(&app.session.filter, &app.session.catalog);super::chips::show(app, ui, &chips);- if people.is_empty() {+ // what is selected below, and the box to name it with+ super::unnamed::naming_bar(app, ui);+ let unnamed = super::unnamed::faces(app, now);+ if people.is_empty() && unnamed.faces.is_empty() {let (title, body) = if chips.is_empty() {- ("No named people yet", "Face names written to XMP by Lightroom and other apps show up here.")+ ("No people yet", "Face names written to XMP by Lightroom and other apps show up here, and faces found in your photos can be named.")} else {("No named people in these photos", "Remove a filter above, or choose Clear all")};@@ -38,50 +86,108 @@ pub fn show(app: &mut LightcraftApp, ui: &mut egui::Ui) {}let ppp = ui.ctx().pixels_per_point();let active = app.session.filter.person.clone();+ let (edge, tile) = (card_edge(thumb), tile_edge(thumb));egui::ScrollArea::vertical().auto_shrink(false).show_viewport(ui, |ui, viewport| {let width = ui.available_width();- let cols = (((width - PAD * 2.0 + GAP) / (CARD + GAP)).floor() as usize).max(1);- let row_h = CARD + LABEL_H + GAP;+ let cols = columns(width, edge);+ let row_h = edge + NAME_H + GAP;let rows = people.len().div_ceil(cols);- let (area, _) = ui.allocate_exact_size(vec2(width, PAD * 2.0 + rows as f32 * row_h), Sense::hover());- let first = ((viewport.top() - PAD) / row_h).floor().max(0.0) as usize;- let last = (((viewport.bottom() - PAD) / row_h).ceil().max(0.0) as usize).min(rows);- for row in first..last {+ let named_h = if people.is_empty() { 40.0 } else { PAD + rows as f32 * row_h };+ let (tile_cols, tile_row) = (columns(width, tile), tile + GAP);+ let tile_rows = unnamed.faces.len().div_ceil(tile_cols);+ let section_top = named_h + 6.0;+ let (area, _) = ui.allocate_exact_size(vec2(width, section_top + SECTION_H + tile_rows as f32 * tile_row + PAD), Sense::hover());+ let visible = |top: f32, rows: usize, row_h: f32| {+ let first = ((viewport.top() - top) / row_h).floor().max(0.0) as usize;+ let last = (((viewport.bottom() - top) / row_h).ceil().max(0.0) as usize).min(rows);+ first..last+ };+ if people.is_empty() {+ ui.painter().text(+ pos2(area.left() + PAD, area.top() + 24.0),+ Align2::LEFT_CENTER,+ "No named people yet. Name the faces below to start.",+ t.font(13.0),+ t.text_dim,+ );+ }+ let mut shown = 0;+ for row in visible(PAD, rows, row_h) {for col in 0..cols {let Some(person) = people.get(row * cols + col) else { break };- let min = area.min + vec2(PAD + col as f32 * (CARD + GAP), PAD + row as f32 * row_h);+ shown += 1;+ let min = area.min + vec2(PAD + col as f32 * (edge + GAP), PAD + row as f32 * row_h);let selected = active.as_deref().is_some_and(|a| a.eq_ignore_ascii_case(&person.name));- card(app, ui, person, Rect::from_min_size(min, vec2(CARD, CARD + LABEL_H)), ppp, selected);+ card(app, ui, person, Rect::from_min_size(min, vec2(edge, edge + NAME_H)), edge, ppp, selected);+ }+ }+ // the faces nobody has named+ let top = area.top() + section_top;+ ui.painter().line_segment([pos2(area.left() + PAD, top), pos2(area.right() - PAD, top)], Stroke::new(1.0, t.button_border));+ super::unnamed::section_header(app, ui, &unnamed, pos2(area.left() + PAD, top + 14.0), area.right() - PAD);+ let grid_top = section_top + SECTION_H;+ for row in visible(grid_top, tile_rows, tile_row) {+ for col in 0..tile_cols {+ let i = row * tile_cols + col;+ let Some(face) = unnamed.faces.get(i) else { break };+ shown += 1;+ let min = area.min + vec2(PAD + col as f32 * (tile + GAP), grid_top + row as f32 * tile_row);+ super::unnamed::tile(app, ui, &unnamed, i, Rect::from_min_size(min, vec2(tile, tile)), face, ppp);}}+ // the picture cache keeps all of them (a fixed budget would evict and re-request the same few every frame)+ app.renderer.want_variants(shown);});}-fn card(app: &mut LightcraftApp, ui: &mut egui::Ui, person: &Person, r: Rect, ppp: f32, selected: bool) {+fn card(app: &mut LightcraftApp, ui: &mut egui::Ui, person: &Person, r: Rect, edge: f32, ppp: f32, selected: bool) {let t = Tokens::get(ui.ctx());- let face = Rect::from_min_size(r.min, vec2(CARD, CARD));+ let face = Rect::from_min_size(r.min, vec2(edge, edge));let resp = ui.interact(r, egui::Id::new(("person-card", &person.name)), Sense::click());register(ui.ctx(), format!("person:{}", person.name), r);+ let view = app.session.face_view(person.photo, person.face);let p = ui.painter();p.rect_filled(face, 3.0, t.canvas);- if let Some(job) = app.session.face_job(person.photo, person.face, (CARD * ppp).ceil() as usize)+ if let Some(job) = app.session.face_job(person.photo, view, (edge * ppp).ceil() as usize)&& let Some(tex) = app.renderer.variant(job){p.image(tex.tex.id(), face, Rect::from_min_max(pos2(0.0, 0.0), pos2(1.0, 1.0)), Color32::WHITE);}+ // how many photos they are in, on the picture+ let count = if person.count > 999 { "999+".to_string() } else { person.count.to_string() };+ let g = p.layout_no_wrap(count, t.font(11.5), Color32::WHITE);+ let pill = Rect::from_min_size(pos2(face.right() - g.size().x - 14.0, face.bottom() - g.size().y - 11.0), g.size() + vec2(10.0, 6.0));+ p.rect_filled(pill, 4.0, Color32::from_black_alpha(175));+ p.galley(pill.min + vec2(5.0, 3.0), g, Color32::WHITE);if selected {p.rect_stroke(face, 3.0, Stroke::new(2.0, Color32::WHITE), StrokeKind::Outside);} else if resp.hovered() {p.rect_stroke(face, 3.0, Stroke::new(1.0, t.text_dim), StrokeKind::Outside);}// a long name must not run past the card- let name =- if person.name.chars().count() > 19 { format!("{}…", person.name.chars().take(18).collect::<String>()) } else { person.name.clone() };+ let name = fit(p, &person.name, t.semibold(13.0), edge);p.text(pos2(r.left() + 2.0, face.bottom() + 14.0), Align2::LEFT_CENTER, name, t.semibold(13.0), t.text);let photos = crate::i18n::tr_format!("{n} photo{}", if person.count == 1 { "" } else { "s" }, n = person.count);p.text(pos2(r.left() + 2.0, face.bottom() + 32.0), Align2::LEFT_CENTER, photos, t.font(12.0), t.text_dim);if resp.on_hover_text(crate::i18n::tr_format!("{} — show their photos", person.name)).clicked() {- let _ = app.run("library.filter", json!({"person": person.name}));- let _ = app.run("view.photoGrid", json!({}));+ let _ = app.run("view.person", json!({"name": person.name}));+ }+}++#[cfg(test)]+mod tests {+ use super::*;++ #[test]+ fn faces_follow_the_thumbnail_slider_within_their_own_limits() {+ assert_eq!((card_edge(90.0), card_edge(480.0)), (84.0, 260.0));+ assert_eq!((tile_edge(90.0), tile_edge(480.0)), (56.0, 180.0));+ // between, growing with the slider; beyond its ends, held at the limits; nonsense, a middle size+ assert!(card_edge(220.0) > card_edge(150.0) && tile_edge(300.0) > tile_edge(220.0));+ assert_eq!((card_edge(-5.0), card_edge(10_000.0)), (84.0, 260.0));+ assert!(card_edge(f32::NAN) > 84.0 && card_edge(f32::NAN) < 260.0);+ // there is always at least one column, whatever the width+ assert_eq!((columns(0.0, 260.0), columns(-50.0, 260.0), columns(f32::NAN, 100.0)), (1, 1, 1));+ assert!(columns(1400.0, 84.0) > columns(1400.0, 260.0));}}diff --git a/crates/ui-egui/src/panels/person.rs b/crates/ui-egui/src/panels/person.rsnew file mode 100644index 000000000..73472d430--- /dev/null+++ b/crates/ui-egui/src/panels/person.rs@@ -0,0 +1,300 @@+//! A person's page in the People view: only cropped faces, never whole photos. First the faces already named, then a+//! "More" section with the unnamed faces that look like them (best first), to be confirmed with a click.+//!+//! The engine does the looking (`faces.person`); this draws it. Only the rows on screen ask for a face render.++use std::sync::Arc;++use egui::{Align2, Color32, Pos2, Rect, RichText, Sense, Stroke, StrokeKind, pos2, vec2};+use serde_json::{Value, json};++use crate::LightcraftApp;+use crate::theme::Tokens;+use crate::widgets::register;++const PAD: f32 = 20.0;+const SECTION_H: f32 = 56.0;++/// One face: which photo, which of its regions, the box to show it by, and (for the "More" faces) how alike.+#[derive(Clone, Debug, PartialEq)]+pub struct Face {+ pub photo: u64,+ pub index: usize,+ pub rect: lightcraft_geom::Rect,+ pub score: f32,+}++/// What `faces.person` said about one person.+#[derive(Clone, Debug, Default, PartialEq)]+pub struct PersonPage {+ pub name: String,+ pub total: usize,+ pub confirmed: Vec<Face>,+ pub more: Vec<Face>,+ /// Recognition is on, so an empty "More" means "none yet" and not "not looking".+ pub ready: bool,+ pub pending: usize,+}++fn face_of(v: &Value) -> Option<Face> {+ let rect = |r: &Value| Some(lightcraft_geom::Rect { x0: r["x0"].as_f64()?, y0: r["y0"].as_f64()?, x1: r["x1"].as_f64()?, y1: r["y1"].as_f64()? });+ Some(Face {+ photo: v["photo"].as_u64()?,+ index: usize::try_from(v["index"].as_u64()?).ok()?,+ // shown by the detector's box when the scan has one: every face equally close+ rect: rect(&v["view"]).or_else(|| rect(&v["rect"]))?,+ score: v["score"].as_f64().unwrap_or(0.0) as f32,+ })+}++/// The engine's answer as a page (anything that does not make sense is left out).+pub fn parse(v: &Value) -> PersonPage {+ let faces = |k: &str| v[k].as_array().map(|a| a.iter().filter_map(face_of).collect()).unwrap_or_default();+ PersonPage {+ name: v["name"].as_str().unwrap_or("").to_string(),+ total: v["total"].as_u64().map_or(0, |n| usize::try_from(n).unwrap_or(0)),+ confirmed: faces("confirmed"),+ more: faces("more"),+ ready: v["ready"] == true,+ pending: v["pendingPhotos"].as_u64().map_or(0, |n| usize::try_from(n).unwrap_or(0)),+ }+}++/// The page for `name`, asked for again only when the catalog changed, or (while the scan runs) at most once a second+/// as it learns more faces.+fn page_for(app: &mut LightcraftApp, name: &str, now: f64) -> Arc<PersonPage> {+ let (rev, indexed) = (app.session.catalog.revision, app.caches.faces_indexed);+ if let Some((n, r, i, at, page)) = &app.caches.person_page+ && n == name+ && *r == rev+ && (*i == indexed || now - *at < 1.0)+ {+ return page.clone();+ }+ let v = app.session.execute("faces.person", &json!({"name": name})).unwrap_or(Value::Null);+ let page = Arc::new(parse(&v));+ app.caches.person_page = Some((name.to_string(), rev, indexed, now, page.clone()));+ page+}++#[derive(PartialEq)]+enum Hit {+ Nothing,+ Open,+ Confirm,+ Dismiss,+}++pub fn show(app: &mut LightcraftApp, ui: &mut egui::Ui, name: &str) {+ let t = Tokens::get(ui.ctx());+ let now = ui.input(|i| i.time);+ let page = page_for(app, name, now);+ let shown_name = if page.name.is_empty() { name.to_string() } else { page.name.clone() };+ ui.add_space(8.0);+ ui.horizontal(|ui| {+ ui.add_space(PAD - 4.0);+ let back = ui.button("‹ People");+ register(ui.ctx(), "person:back", back.rect);+ if back.clicked() {+ let _ = app.run("view.people", json!({}));+ }+ ui.label(RichText::new(&shown_name).font(t.semibold(15.0)).color(t.text));+ let count = if page.total == 1 { "1 face".to_string() } else { format!("{} faces", page.total) };+ ui.label(RichText::new(count).color(t.text_dim));+ ui.with_layout(egui::Layout::right_to_left(egui::Align::Center), |ui| {+ ui.add_space(PAD - 4.0);+ let photos = ui.button("Show photos").on_hover_text("Their photos in the grid");+ register(ui.ctx(), "person:photos", photos.rect);+ if photos.clicked() {+ let _ = app.run("library.filter", json!({"person": shown_name}));+ let _ = app.run("view.photoGrid", json!({}));+ }+ });+ });+ ui.add_space(6.0);+ if page.total == 0 {+ super::empty_message(+ ui,+ ui.available_rect_before_wrap(),+ "No faces by that name",+ "They may have been renamed or removed. Go back to People",+ );+ return;+ }+ let dismissed = &app.ui.dismissed_faces;+ let more: Vec<&Face> = page.more.iter().filter(|f| !dismissed.contains(&(f.photo, f.index))).collect();+ // what the "More" section says when it has no faces to show+ let recognising = app.caches.faces_active || page.ready;+ // with a model installed the button switches recognition on; without one it opens Settings ▸ Faces+ let step = if recognising { super::faces::Setup::Running } else { super::faces::setup(app, ui.ctx()) };+ let note: Option<String> = if step == super::faces::Setup::TurnOn {+ Some(format!("Face recognition is off. Turn it on to find more photos of {shown_name}."))+ } else if !recognising {+ Some(format!("Face recognition needs a model, in Settings, to find more photos of {shown_name}."))+ } else if more.is_empty() && page.pending > 0 {+ Some(format!("Looking through your photos… {} left", page.pending))+ } else if more.is_empty() {+ Some(format!("No other faces look like {shown_name} yet. Faces appear here as your photos are scanned."))+ } else {+ None+ };+ let ppp = ui.ctx().pixels_per_point();+ let edge = super::people::tile_edge(app.ui.thumb_size);+ let mut hit: Option<(Hit, Face)> = None;+ egui::ScrollArea::vertical().auto_shrink(false).show_viewport(ui, |ui, viewport| {+ let width = ui.available_width();+ let cols = super::people::columns(width, edge);+ let row_h = edge + super::people::TILE_GAP;+ let confirmed_rows = page.confirmed.len().div_ceil(cols);+ let more_rows = more.len().div_ceil(cols);+ let more_top = PAD + confirmed_rows as f32 * row_h + 12.0;+ let total_h = more_top + SECTION_H + more_rows as f32 * row_h + PAD;+ let (area, _) = ui.allocate_exact_size(vec2(width, total_h), Sense::hover());+ let visible = |top: f32, rows: usize| {+ let first = ((viewport.top() - top) / row_h).floor().max(0.0) as usize;+ let last = (((viewport.bottom() - top) / row_h).ceil().max(0.0) as usize).min(rows);+ first..last+ };+ let mut shown = 0;+ for row in visible(PAD, confirmed_rows) {+ for col in 0..cols {+ let Some(f) = page.confirmed.get(row * cols + col) else { break };+ shown += 1;+ let min = area.min + vec2(PAD + col as f32 * (edge + super::people::TILE_GAP), PAD + row as f32 * row_h);+ if let Hit::Open = tile(app, ui, Rect::from_min_size(min, vec2(edge, edge)), f, ppp, false, &shown_name) {+ hit = Some((Hit::Open, f.clone()));+ }+ }+ }+ // the "More" section+ let heading = pos2(area.left() + PAD, area.top() + more_top + 14.0);+ ui.painter().text(heading, Align2::LEFT_CENTER, "More", t.semibold(15.0), t.text);+ let sub = match ¬e {+ Some(n) => n.clone(),+ None => format!("Faces that look like {shown_name}. Click one to confirm it, or × to hide it."),+ };+ ui.painter().text(pos2(area.left() + PAD, area.top() + more_top + 36.0), Align2::LEFT_CENTER, sub, t.font(12.0), t.text_dim);+ if !recognising {+ let r = Rect::from_min_size(pos2(area.left() + PAD + 330.0, area.top() + more_top + 8.0), vec2(120.0, 24.0));+ let label = if step == super::faces::Setup::TurnOn { "Turn on" } else { "Open Settings" };+ let b = ui.put(r, egui::Button::new(label));+ register(ui.ctx(), "faces:setup", b.rect);+ if b.clicked() {+ if step == super::faces::Setup::TurnOn {+ super::faces::take_step(app, ui.ctx(), step);+ } else {+ let _ = app.run("app.settings", json!({"tab": "faces"}));+ }+ }+ }+ let grid_top = more_top + SECTION_H;+ for row in visible(grid_top, more_rows) {+ for col in 0..cols {+ let Some(f) = more.get(row * cols + col) else { break };+ shown += 1;+ let min = area.min + vec2(PAD + col as f32 * (edge + super::people::TILE_GAP), grid_top + row as f32 * row_h);+ match tile(app, ui, Rect::from_min_size(min, vec2(edge, edge)), f, ppp, true, &shown_name) {+ Hit::Confirm => hit = Some((Hit::Confirm, (*f).clone())),+ Hit::Dismiss => hit = Some((Hit::Dismiss, (*f).clone())),+ _ => {}+ }+ }+ }+ // the picture cache keeps every face on screen+ app.renderer.want_variants(shown);+ });+ if let Some((what, f)) = hit {+ match what {+ Hit::Open => {+ let _ = app.run("library.select", json!({"ids": [f.photo], "active": f.photo}));+ let _ = app.run("view.detail", json!({}));+ }+ Hit::Confirm => {+ let _ = app.run("faces.setName", json!({"id": f.photo, "index": f.index, "name": shown_name}));+ }+ Hit::Dismiss => {+ app.ui.dismissed_faces.insert((f.photo, f.index));+ }+ Hit::Nothing => {}+ }+ }+}++/// One face tile. A confirmed one opens its photo when clicked; a "More" one is confirmed when clicked, and has a × that+/// hides it.+fn tile(app: &mut LightcraftApp, ui: &mut egui::Ui, r: Rect, f: &Face, ppp: f32, more: bool, name: &str) -> Hit {+ let t = Tokens::get(ui.ctx());+ let salt = if more { "more-face" } else { "person-face" };+ let resp = ui.interact(r, egui::Id::new((salt, f.photo, f.index)), Sense::click());+ register(ui.ctx(), format!("{salt}:{}:{}", f.photo, f.index), r);+ // the × is added after the tile, so it wins where they overlap+ let x_rect = Rect::from_min_size(pos2(r.right() - 24.0, r.top() + 4.0), vec2(20.0, 20.0));+ let x_resp = more.then(|| ui.interact(x_rect, egui::Id::new(("more-face-x", f.photo, f.index)), Sense::click()));+ let hovered = resp.hovered() || x_resp.as_ref().is_some_and(|x| x.hovered());+ let p = ui.painter();+ p.rect_filled(r, 3.0, t.canvas);+ if let Some(job) = app.session.face_job(lightcraft_catalog::PhotoId(f.photo), f.rect, (r.width() * ppp).ceil() as usize)+ && let Some(tex) = app.renderer.variant(job)+ {+ p.image(tex.tex.id(), r, Rect::from_min_max(pos2(0.0, 0.0), pos2(1.0, 1.0)), Color32::WHITE);+ }+ if hovered {+ p.rect_stroke(r, 3.0, Stroke::new(1.5, Color32::WHITE), StrokeKind::Outside);+ }+ let mut hit = Hit::Nothing;+ if more && hovered {+ // a check at the bottom right says a click confirms; the × at the top right hides the face+ let badge = Rect::from_min_size(pos2(r.right() - 28.0, r.bottom() - 28.0), vec2(24.0, 24.0));+ p.circle_filled(badge.center(), 12.0, Color32::from_rgba_unmultiplied(30, 30, 30, 220));+ let c = badge.center();+ let check = [pos2(c.x - 5.0, c.y), pos2(c.x - 1.5, c.y + 4.0), pos2(c.x + 5.5, c.y - 4.0)];+ p.line_segment([check[0], check[1]], Stroke::new(2.0, Color32::WHITE));+ p.line_segment([check[1], check[2]], Stroke::new(2.0, Color32::WHITE));+ p.circle_filled(x_rect.center(), 10.0, Color32::from_rgba_unmultiplied(30, 30, 30, 220));+ let xc: Pos2 = x_rect.center();+ p.line_segment([pos2(xc.x - 3.5, xc.y - 3.5), pos2(xc.x + 3.5, xc.y + 3.5)], Stroke::new(1.6, Color32::WHITE));+ p.line_segment([pos2(xc.x - 3.5, xc.y + 3.5), pos2(xc.x + 3.5, xc.y - 3.5)], Stroke::new(1.6, Color32::WHITE));+ }+ if more {+ let tip = format!("Looks like {name} (similarity {:.2}). Click to confirm.", f.score);+ let _ = resp.clone().on_hover_text(tip);+ if x_resp.as_ref().is_some_and(|x| x.clicked()) {+ hit = Hit::Dismiss;+ } else if resp.clicked() {+ hit = Hit::Confirm;+ }+ } else {+ let _ = resp.clone().on_hover_text(format!("{name} — click to open this photo"));+ if resp.clicked() {+ hit = Hit::Open;+ }+ }+ hit+}++#[cfg(test)]+mod tests {+ use super::*;++ #[test]+ fn the_engines_answer_becomes_a_page_and_odd_entries_are_left_out() {+ let v = json!({+ "name": "Jane Doe", "total": 2, "ready": true, "pendingPhotos": 7,+ "confirmed": [+ {"photo": 3, "index": 0, "rect": {"x0": 0.1, "y0": 0.2, "x1": 0.3, "y1": 0.5}},+ {"photo": 4, "index": 1},+ {"photo": "x", "index": 0, "rect": {"x0": 0.1, "y0": 0.2, "x1": 0.3, "y1": 0.5}},+ ],+ "more": [{"photo": 9, "index": 2, "score": 0.5, "rect": {"x0": 0.0, "y0": 0.0, "x1": 1.0, "y1": 1.0}}],+ });+ let page = parse(&v);+ assert_eq!((page.name.as_str(), page.total, page.ready, page.pending), ("Jane Doe", 2, true, 7));+ assert_eq!(page.confirmed.len(), 1, "a face without a box or a photo is dropped");+ assert_eq!((page.confirmed[0].photo, page.confirmed[0].index), (3, 0));+ assert_eq!((page.more.len(), page.more[0].score), (1, 0.5));+ // nothing, or nonsense, is an empty page and not a crash+ for bad in [Value::Null, json!([]), json!({"confirmed": "x", "more": 5, "total": -1})] {+ assert_eq!(parse(&bad), PersonPage::default(), "{bad}");+ }+ }+}diff --git a/crates/ui-egui/src/panels/unnamed.rs b/crates/ui-egui/src/panels/unnamed.rsnew file mode 100644index 000000000..ef50ccaef--- /dev/null+++ b/crates/ui-egui/src/panels/unnamed.rs@@ -0,0 +1,299 @@+//! The unnamed faces in the People view: every face nobody has named, as cropped pictures, with look-alikes next to each+//! other (once recognition has looked at them). Click faces to select them (Shift-click selects a range), type a name in+//! the bar above, press Enter: they are all named at once, in one undo step. A face the named ones recognise carries the+//! name they suggest; clicking that name accepts it for that face alone.+//!+//! The engine does the looking (`faces.unnamed`) and the naming (`faces.nameFaces`); this draws and selects.++use std::sync::Arc;++use egui::{Align2, Color32, Rect, RichText, Sense, Stroke, StrokeKind, pos2, vec2};+use serde_json::{Value, json};++use crate::LightcraftApp;+use crate::theme::Tokens;+use crate::widgets::register;++/// One unnamed face: which photo and region, the box to show it by, and a suggested name.+#[derive(Clone, Debug, PartialEq)]+pub struct UnnamedFace {+ pub photo: u64,+ pub index: usize,+ pub view: lightcraft_geom::Rect,+ pub suggestion: Option<String>,+}++/// What `faces.unnamed` said.+#[derive(Clone, Debug, Default, PartialEq)]+pub struct Unnamed {+ /// Unnamed faces in the library (more than are listed when the list was cut).+ pub total: usize,+ pub faces: Vec<UnnamedFace>,+ /// Look-alikes are next to each other.+ pub ordered: bool,+ /// Recognition is running.+ pub ready: bool,+}++fn rect_of(v: &Value) -> Option<lightcraft_geom::Rect> {+ Some(lightcraft_geom::Rect { x0: v["x0"].as_f64()?, y0: v["y0"].as_f64()?, x1: v["x1"].as_f64()?, y1: v["y1"].as_f64()? })+}++fn face_of(v: &Value) -> Option<UnnamedFace> {+ Some(UnnamedFace {+ photo: v["photo"].as_u64()?,+ index: usize::try_from(v["index"].as_u64()?).ok()?,+ view: rect_of(&v["view"]).or_else(|| rect_of(&v["rect"]))?,+ suggestion: v["suggestion"]["name"].as_str().map(str::to_string),+ })+}++/// The engine's answer as a list (anything that does not make sense is left out).+pub fn parse(v: &Value) -> Unnamed {+ Unnamed {+ total: v["total"].as_u64().map_or(0, |n| usize::try_from(n).unwrap_or(0)),+ faces: v["faces"].as_array().map(|a| a.iter().filter_map(face_of).collect()).unwrap_or_default(),+ ordered: v["ordered"] == true,+ ready: v["ready"] == true,+ }+}++/// The unnamed faces, asked for again only when the catalog changed, or (while the scan runs and learns more faces) at+/// most every few seconds, since putting look-alikes together looks at every pair.+pub fn faces(app: &mut LightcraftApp, now: f64) -> Arc<Unnamed> {+ let (rev, indexed) = (app.session.catalog.revision, app.caches.faces_indexed);+ if let Some((r, i, at, list)) = &app.caches.unnamed+ && *r == rev+ && (*i == indexed || now - *at < 3.0)+ {+ return list.clone();+ }+ let v = app.session.execute("faces.unnamed", &json!({})).unwrap_or(Value::Null);+ let list = Arc::new(parse(&v));+ // what was selected and has since been named (or removed) is no longer selected+ let present: std::collections::HashSet<(u64, usize)> = list.faces.iter().map(|f| (f.photo, f.index)).collect();+ app.ui.unnamed_selected.retain(|k| present.contains(k));+ app.caches.unnamed = Some((rev, indexed, now, list.clone()));+ list+}++/// A click on face number `i` of `list`: select it (or deselect it), or with Shift every face from the last one clicked.+fn click(app: &mut LightcraftApp, list: &Unnamed, i: usize, shift: bool) {+ let Some(f) = list.faces.get(i) else { return };+ match app.ui.unnamed_anchor.filter(|_| shift) {+ Some(a) => {+ for f in list.faces.get(a.min(i)..=a.max(i)).unwrap_or_default() {+ app.ui.unnamed_selected.insert((f.photo, f.index));+ }+ }+ None => {+ let key = (f.photo, f.index);+ if !app.ui.unnamed_selected.remove(&key) {+ app.ui.unnamed_selected.insert(key);+ }+ app.ui.unnamed_anchor = Some(i);+ }+ }+ // typing goes to the name box at once+ app.ui.unnamed_focus = !app.ui.unnamed_selected.is_empty();+}++/// Name `faces` (photo, region) `name`, as one undo step.+fn name(app: &mut LightcraftApp, faces: &[(u64, usize)], name: &str) {+ let list: Vec<Value> = faces.iter().map(|(photo, index)| json!({"photo": photo, "index": index})).collect();+ let _ = app.run("faces.nameFaces", json!({"faces": list, "name": name}));+}++/// The bar above the faces while some are selected: how many, a box to name them all, and the people already named.+pub fn naming_bar(app: &mut LightcraftApp, ui: &mut egui::Ui) {+ if app.ui.unnamed_selected.is_empty() {+ app.ui.unnamed_name.clear();+ return;+ }+ let t = Tokens::get(ui.ctx());+ let selected: Vec<(u64, usize)> = {+ let mut v: Vec<_> = app.ui.unnamed_selected.iter().copied().collect();+ v.sort_unstable();+ v+ };+ // the name the named faces suggest, when every selected face has the same one+ let suggested: Option<String> = {+ let list = app.caches.unnamed.as_ref().map(|c| c.3.clone());+ list.and_then(|l| {+ let names: std::collections::BTreeSet<Option<&str>> =+ l.faces.iter().filter(|f| app.ui.unnamed_selected.contains(&(f.photo, f.index))).map(|f| f.suggestion.as_deref()).collect();+ (names.len() == 1).then(|| names.into_iter().next().flatten().map(str::to_string)).flatten()+ })+ };+ let mut submit: Option<String> = None;+ let mut clear = false;+ ui.horizontal(|ui| {+ ui.add_space(super::people::PAD - 4.0);+ let n = selected.len();+ ui.label(RichText::new(if n == 1 { "1 face selected".to_string() } else { format!("{n} faces selected") }).color(t.text));+ let hint = suggested.clone().unwrap_or_else(|| "Name…".to_string());+ let edit = ui.add(egui::TextEdit::singleline(&mut app.ui.unnamed_name).hint_text(hint).desired_width(240.0));+ register(ui.ctx(), "field:unnamedName", edit.rect);+ if std::mem::take(&mut app.ui.unnamed_focus) {+ edit.request_focus();+ }+ let typed = app.ui.unnamed_name.trim().to_string();+ let chosen = if typed.is_empty() { suggested.clone().unwrap_or_default() } else { typed.clone() };+ if edit.lost_focus() && ui.input(|i| i.key_pressed(egui::Key::Enter)) && !chosen.is_empty() {+ submit = Some(chosen.clone());+ }+ let label = if n == 1 { "Name face".to_string() } else { format!("Name {n} faces") };+ let b = ui.add_enabled(!chosen.is_empty(), egui::Button::new(label));+ register(ui.ctx(), "unnamed:name", b.rect);+ if b.clicked() {+ submit = Some(chosen);+ }+ let c = ui.button("Clear");+ register(ui.ctx(), "unnamed:clear", c.rect);+ clear = c.clicked() || (edit.has_focus() && ui.input(|i| i.key_pressed(egui::Key::Escape)));+ });+ // the people already named, matching what is typed (a click names the faces)+ let typed = app.ui.unnamed_name.trim().to_lowercase();+ if !typed.is_empty() {+ let people = app.caches.person_names(&app.session.catalog);+ let matching: Vec<&String> = people.iter().filter(|p| p.to_lowercase().starts_with(&typed) && p.to_lowercase() != typed).take(6).collect();+ if !matching.is_empty() {+ ui.horizontal(|ui| {+ ui.add_space(super::people::PAD - 4.0);+ for p in matching {+ let b = ui.add(egui::Button::new(RichText::new(p.as_str()).color(t.text_label)).frame(false));+ register(ui.ctx(), format!("unnamed:pick:{p}"), b.rect);+ if b.clicked() {+ submit = Some(p.clone());+ }+ }+ });+ }+ }+ if let Some(who) = submit {+ name(app, &selected, &who);+ app.ui.unnamed_selected.clear();+ app.ui.unnamed_name.clear();+ app.ui.unnamed_anchor = None;+ } else if clear {+ app.ui.unnamed_selected.clear();+ app.ui.unnamed_name.clear();+ app.ui.unnamed_anchor = None;+ }+}++/// The heading of the section, at `at`, and what it says about where the faces come from.+pub fn section_header(app: &mut LightcraftApp, ui: &mut egui::Ui, list: &Unnamed, at: egui::Pos2, right: f32) {+ let t = Tokens::get(ui.ctx());+ let count = if list.total > list.faces.len() { format!("{} of {}", list.faces.len(), list.total) } else { list.total.to_string() };+ let p = ui.painter();+ p.text(at, Align2::LEFT_CENTER, "Unnamed faces", t.semibold(15.0), t.text);+ p.text(pos2(right, at.y), Align2::RIGHT_CENTER, count, t.font(13.0), t.text_dim);+ let sub = if !list.ready && list.faces.is_empty() {+ "Turn on face recognition in Settings to find the faces in your photos.".to_string()+ } else if list.faces.is_empty() && app.caches.faces_pending > 0 {+ format!("Looking through your photos… {} left", app.caches.faces_pending)+ } else if list.faces.is_empty() {+ "No unnamed faces. Faces found in your photos appear here until you name them.".to_string()+ } else if list.ordered {+ "Faces that look alike are next to each other. Click to select, Shift-click for a range, then type a name.".to_string()+ } else {+ "Click faces to select them, Shift-click for a range, then type a name.".to_string()+ };+ p.text(pos2(at.x, at.y + 24.0), Align2::LEFT_CENTER, sub, t.font(12.0), t.text_dim);+ if !list.ready && list.faces.is_empty() {+ let r = Rect::from_min_size(pos2(at.x, at.y + 38.0), vec2(120.0, 22.0));+ let b = ui.put(r, egui::Button::new("Open Settings"));+ register(ui.ctx(), "unnamed:openSettings", b.rect);+ if b.clicked() {+ let _ = app.run("app.settings", json!({"tab": "faces"}));+ }+ } else if !list.faces.is_empty() {+ let r = Rect::from_min_size(pos2(right - 130.0, at.y + 12.0), vec2(130.0, 22.0));+ let all = app.ui.unnamed_selected.len() == list.faces.len();+ let b = ui.put(r, egui::Button::new(if all { "Select none" } else { "Select all" }));+ register(ui.ctx(), "unnamed:selectAll", b.rect);+ if b.clicked() {+ app.ui.unnamed_selected.clear();+ if !all {+ app.ui.unnamed_selected.extend(list.faces.iter().map(|f| (f.photo, f.index)));+ app.ui.unnamed_focus = true;+ }+ }+ }+}++/// One face tile, number `i` of `list`.+pub fn tile(app: &mut LightcraftApp, ui: &mut egui::Ui, list: &Unnamed, i: usize, r: Rect, f: &UnnamedFace, ppp: f32) {+ let t = Tokens::get(ui.ctx());+ let key = (f.photo, f.index);+ let selected = app.ui.unnamed_selected.contains(&key);+ let resp = ui.interact(r, egui::Id::new(("unnamed-face", f.photo, f.index)), Sense::click());+ register(ui.ctx(), format!("unnamed-face:{}:{}", f.photo, f.index), r);+ // a suggested name along the bottom: a click accepts it for this face (added after the tile, so it wins there)+ let strip = Rect::from_min_max(pos2(r.left(), r.bottom() - 20.0), r.right_bottom());+ let accept = f.suggestion.as_ref().map(|_| ui.interact(strip, egui::Id::new(("unnamed-accept", f.photo, f.index)), Sense::click()));+ if let Some(a) = &accept {+ register(ui.ctx(), format!("unnamed-accept:{}:{}", f.photo, f.index), a.rect);+ }+ let hovered = resp.hovered() || accept.as_ref().is_some_and(|a| a.hovered());+ let p = ui.painter();+ p.rect_filled(r, 3.0, t.canvas);+ if let Some(job) = app.session.face_job(lightcraft_catalog::PhotoId(f.photo), f.view, (r.width() * ppp).ceil() as usize)+ && let Some(tex) = app.renderer.variant(job)+ {+ p.image(tex.tex.id(), r, Rect::from_min_max(pos2(0.0, 0.0), pos2(1.0, 1.0)), Color32::WHITE);+ }+ if selected {+ p.rect_filled(r, 3.0, Color32::from_rgba_unmultiplied(70, 130, 255, 55));+ p.rect_stroke(r, 3.0, Stroke::new(2.0, Color32::from_rgb(110, 160, 255)), StrokeKind::Inside);+ // a check at the top left+ let c = pos2(r.left() + 13.0, r.top() + 13.0);+ p.circle_filled(c, 9.0, Color32::from_rgb(70, 130, 255));+ p.line_segment([pos2(c.x - 4.0, c.y), pos2(c.x - 1.0, c.y + 3.5)], Stroke::new(1.8, Color32::WHITE));+ p.line_segment([pos2(c.x - 1.0, c.y + 3.5), pos2(c.x + 4.5, c.y - 3.5)], Stroke::new(1.8, Color32::WHITE));+ } else if hovered {+ p.rect_stroke(r, 3.0, Stroke::new(1.0, Color32::WHITE), StrokeKind::Inside);+ }+ if let Some(name) = &f.suggestion {+ p.rect_filled(strip, 0.0, Color32::from_black_alpha(165));+ let label = super::people::fit(p, name, t.font(11.5), r.width() - 18.0);+ p.text(pos2(r.left() + 5.0, strip.center().y), Align2::LEFT_CENTER, format!("{label}?"), t.font(11.5), Color32::from_gray(225));+ }+ if let (Some(a), Some(who)) = (&accept, &f.suggestion)+ && a.clone().on_hover_text(format!("Name this face {who}")).clicked()+ {+ name(app, &[key], who);+ return;+ }+ if resp.clicked() {+ let shift = ui.input(|i| i.modifiers.shift);+ click(app, list, i, shift);+ }+}++#[cfg(test)]+mod tests {+ use super::*;++ #[test]+ fn the_engines_answer_becomes_a_list_and_odd_entries_are_left_out() {+ let v = json!({+ "total": 9, "ordered": true, "ready": true,+ "faces": [+ {"photo": 3, "index": 1, "rect": {"x0": 0.1, "y0": 0.1, "x1": 0.3, "y1": 0.4}, "view": {"x0": 0.12, "y0": 0.12, "x1": 0.28, "y1": 0.38}, "suggestion": {"name": "Jane Doe", "score": 0.7}},+ {"photo": 4, "index": 0, "rect": {"x0": 0.1, "y0": 0.1, "x1": 0.3, "y1": 0.4}, "suggestion": null},+ {"photo": 5, "index": 0},+ {"index": 0, "rect": {"x0": 0.1, "y0": 0.1, "x1": 0.3, "y1": 0.4}},+ ],+ });+ let list = parse(&v);+ assert_eq!((list.total, list.ordered, list.ready, list.faces.len()), (9, true, true, 2));+ assert_eq!(list.faces[0].suggestion.as_deref(), Some("Jane Doe"));+ assert_eq!(list.faces[0].view.x0, 0.12, "the view box is used when there is one");+ assert_eq!((list.faces[1].view.x0, list.faces[1].suggestion.clone()), (0.1, None), "else the face's own box");+ for bad in [Value::Null, json!([]), json!({"faces": 5, "total": -1})] {+ assert_eq!(parse(&bad), Unnamed::default(), "{bad}");+ }+ }+}diff --git a/crates/ui-egui/src/render.rs b/crates/ui-egui/src/render.rsindex 3f0906a20..f46436941 100644--- a/crates/ui-egui/src/render.rs+++ b/crates/ui-egui/src/render.rs@@ -152,8 +152,11 @@ pub(crate) fn stage_trim_order(sizes: &[(Slot, usize)], budget: usize) -> Vec<Sl/// engine's own caches do not count.const STAGE_BUDGET_SHARE: usize = 4;-/// Variant thumbnails kept as textures (LRU).+/// Variant thumbnails kept as textures (LRU), when nothing asks for more.pub const VARIANT_TEXTURES: usize = 96;+/// The most a view can raise that to ([`Renderer::want_variants`]): a screenful of small faces is a few hundred pictures of+/// a few tens of kilobytes each.+pub const MAX_VARIANT_TEXTURES: usize = 800;/// Priority of variant thumbnail jobs: below on-screen grid thumbnails and the loupe.const VARIANT_PRIORITY: u32 = 6;@@ -199,6 +202,8 @@ pub struct Renderer {prefetched: HashMap<Slot, u64>,/// Variant key → frame it was last asked for (LRU of [`Slot::Variant`] textures).variant_used: HashMap<u64, u64>,+ /// How many variants the last frame's views showed at once (see [`Self::want_variants`]).+ variant_want: usize,/// Frames polled so far.frame: u64,/// Stage caches cleared to stay in budget (see [`stage_trim_order`]).@@ -258,6 +263,7 @@ impl Renderer {catalog_rev: 0,prefetched: HashMap::new(),variant_used: HashMap::new(),+ variant_want: 0,frame: 0,stages_trimmed: 0,#[cfg(test)]@@ -460,6 +466,19 @@ impl Renderer {self.textures.get(&slot)}+ /// A view that shows `n` variants at once (the People view's faces) says so each frame, so the cache holds them all:+ /// with a fixed budget a screen of more than [`VARIANT_TEXTURES`] pictures evicts, and re-requests, the same few every+ /// frame, and those tiles stay blank. The budget is the most asked for in the last frame, a quarter more, up to+ /// [`MAX_VARIANT_TEXTURES`].+ pub fn want_variants(&mut self, n: usize) {+ self.variant_want = self.variant_want.max(n);+ }++ /// How many variant textures are kept now.+ fn variant_budget(&self) -> usize {+ VARIANT_TEXTURES.max(self.variant_want + self.variant_want / 4).min(MAX_VARIANT_TEXTURES)+ }+/// Variant thumbnail textures currently loaded.pub fn variant_textures(&self) -> usize {self.textures.keys().filter(|s| matches!(s, Slot::Variant(_))).count()@@ -489,20 +508,22 @@ impl Renderer {}_ => true,});+ let budget = self.variant_budget();+ self.variant_want = 0;let n = self.variant_textures();- if n > VARIANT_TEXTURES {+ if n > budget {let mut have: Vec<(u64, u64)> = self.textures.keys().filter_map(|s| if let Slot::Variant(k) = s { Some((self.variant_used.get(k).copied().unwrap_or(0), *k)) } else { None }).collect();have.sort_unstable();- for (_, k) in have.into_iter().take(n - VARIANT_TEXTURES) {+ for (_, k) in have.into_iter().take(n - budget) {self.textures.remove(&Slot::Variant(k));self.variant_used.remove(&k);}}- if self.variant_used.len() > 4 * VARIANT_TEXTURES {+ if self.variant_used.len() > 4 * budget {let textures = &self.textures;let pending = &self.pending;self.variant_used.retain(|k, _| textures.contains_key(&Slot::Variant(*k)) || pending.contains_key(&Slot::Variant(*k)));diff --git a/crates/ui-egui/src/shortcuts.rs b/crates/ui-egui/src/shortcuts.rsindex 397ef86fb..cd69b2daa 100644--- a/crates/ui-egui/src/shortcuts.rs+++ b/crates/ui-egui/src/shortcuts.rs@@ -286,7 +286,7 @@ pub fn handle(app: &mut LightcraftApp, ctx: &egui::Context) {app.recording_shortcut = None;}// don't steal keys from text fields or from the keymap editor recording a shortcut- if ctx.egui_wants_keyboard_input() || app.recording_shortcut.is_some() {+ if ctx.egui_wants_keyboard_input() || app.recording_shortcut.is_some() || app.ui.name_edit.is_some() {return;}let mut fire: Vec<String> = Vec::new();diff --git a/crates/ui-egui/src/state.rs b/crates/ui-egui/src/state.rsindex 397e341f9..20e69b008 100644--- a/crates/ui-egui/src/state.rs+++ b/crates/ui-egui/src/state.rs@@ -332,6 +332,25 @@ pub struct UiState {/// Left-sidebar sections folded shut by their header (`albums`, `local`, `byDate`,/// `keywords`); the rest are open.pub collapsed_sidebar: Vec<String>,+ /// A face's name being typed in the loupe.+ #[serde(skip)]+ pub name_edit: Option<NameEdit>,+ /// The person whose page the People view shows (their faces and the faces that look like them); `None`: everyone.+ #[serde(skip)]+ pub person_page: Option<String>,+ /// "More" faces the user hid with ×, for this session: (photo, region index).+ #[serde(skip)]+ pub dismissed_faces: std::collections::HashSet<(u64, usize)>,+ /// The unnamed faces selected in the People view (photo, region), the name being typed for them, the face last+ /// clicked (Shift-click selects a range from it) and whether the name box should take the keyboard.+ #[serde(skip)]+ pub unnamed_selected: std::collections::HashSet<(u64, usize)>,+ #[serde(skip)]+ pub unnamed_name: String,+ #[serde(skip)]+ pub unnamed_anchor: Option<usize>,+ #[serde(skip)]+ pub unnamed_focus: bool,/// Local sidebar locations hidden with “Remove from Local” (folders on disk are untouched).pub hidden_locations: Vec<String>,/// Copies opened in an external editor this session (reloaded when the window is focused@@ -462,6 +481,16 @@ impl Dialog {}}+/// A face's name being typed in the loupe: which face, and what has been typed so far.+#[derive(Clone, Debug, Default, PartialEq)]+pub struct NameEdit {+ pub photo: u64,+ pub index: usize,+ pub text: String,+ /// Just opened: the text box takes the keyboard focus once.+ pub fresh: bool,+}+#[derive(Clone, Debug, Serialize, Deserialize, PartialEq)]#[serde(tag = "kind", rename_all = "camelCase")]pub enum Dialog {@@ -656,6 +685,13 @@ impl Default for UiState {show_counts: true,face_boxes: true,collapsed_sidebar: Vec::new(),+ name_edit: None,+ person_page: None,+ dismissed_faces: Default::default(),+ unnamed_selected: Default::default(),+ unnamed_name: String::new(),+ unnamed_anchor: None,+ unnamed_focus: false,hidden_locations: Vec::new(),dragging_control: None,external_edits: Vec::new(),diff --git a/docs/faces.md b/docs/faces.mdindex d4357995b..3b1692c6a 100644--- a/docs/faces.md+++ b/docs/faces.md@@ -14,6 +14,28 @@ No model weights are part of LightCraft. Each one is a file you choose to get, fidentify anyone. **Photo ▸ Detect Faces** offers the download the first time you use it.- **Face recognition models** are large (AuraFace is 261 MB), and their licences and training data deserve a decision bythe person installing them. Everything works without one: names from XMP, the People view, the face boxes.+- **Recognition runs in LightCraft's own pure-Rust CPU interpreter**, shared with the detector. Dense+ convolutions use bounded im2col and single-thread float32 GEMM (ndarray 0.17.2 / matrixmultiply 0.3.11, MIT or Apache-2.0, default/BLAS/threading features disabled);+ depthwise convolutions use row-wise Rust loops. There is no nested thread pool. No C or assembly is compiled by+ this dependency path. Desktop and CLI builds always include recognition (there is no separate cargo feature for it). The browser retains its existing model-install/background-scan restrictions.++### Supported graphs and limits++The pinned SFace 2021dec and AuraFace v1 files both use ONNX opset 11. Their combined operators are **Conv** (including+SFace's depthwise groups), **BatchNormalization**, **PRelu**, **Add**, **Sub**, **Mul**, **Dropout** (inference identity),+**Flatten**, and **Gemm**. The shared engine also supports the detector's Relu, Sigmoid, MaxPool, nearest whole-number+Resize, Transpose and constant-shape Reshape, plus GlobalAveragePool and Identity. This is a limited inference engine:+other operators (including MatMul and Concat), custom domains, training graphs, unsupported attribute combinations,+external/sparse weights and non-float32 inputs/outputs fail to load with an explicit error.++Shapes and attributes are checked before installation's self-test, including all intermediates and the promised output+size. Files are limited to 512 MiB, combined weights to 64 Mi elements (256 MiB of float32), rank to eight and live tensors plus+retained convolution scratch to 512 MiB. All supplied weights and produced activations must be finite. Embeddings retain the+manifest's RGB/BGR and mean/std preprocessing, and are normalized using a bounded float64 norm to avoid float32 overflow.++The agreement figures below come from a one-off comparison against an independent ONNX reference runtime, run outside+this repository; that tooling is not part of it, so those figures cannot be reproduced from here. What the repository+does check is listed under "Agreement and regression coverage".## Downloading a model@@ -32,6 +54,9 @@ No model weights are part of LightCraft. Each one is a file you choose to get, f## Adding a model file yourself+Adding a model is how you say you want it: once it is installed it is **chosen and recognition is switched on**, with no+further step. The one question is its licence, and that comes first.+1. **Settings ▸ Faces ▸ Add a model file…**, or drop a `.onnx` file on the window.2. LightCraft looks at the file (it never runs it at this point): it recognises models it knows by their SHA-256, andfor any other file it reads the input and output shapes and describes what it assumed (112 × 112 aligned faces,@@ -43,6 +68,42 @@ No model weights are part of LightCraft. Each one is a file you choose to get, fNon-commercial models (InsightFace, for example) can be added this way for your own use; LightCraft never bundles,hosts, downloads or links them from a picker, and the dialog says so.+The newest model is the one in use. An earlier one stays installed, **Use** switches back, and each model keeps its own+cache of embeddings (`face-embeddings-<model id>.bin` in the library folder), so going back does not start over. Removing+the model in use hands over to another installed one. Results of the model that was running when you switched are+discarded, never mixed into the new one's index.++### Bring your own models: `catalog.json`++LightCraft's built-in list only holds models whose terms let the project point at them. For anything else (a stronger+recogniser whose weights are for research use, a model you trained, a mirror you trust) put a `catalog.json` in the models+folder. Each entry is a model manifest plus the address to fetch it from, and it then gets the same **Download** button as+a built-in model, with its licence notice shown before anything is fetched. LightCraft ships and links to none of them: what+the file says, and whether you may use the weights, is yours to check (the dialog shows the notice you wrote, in a warning+colour unless you marked the model `"commercial": "yes"`).++```json+{ "models": [ {+ "id": "my-recogniser", "name": "My recogniser (R50)", "version": "1", "role": "embedder",+ "url": "https://example.org/weights/recogniser.onnx",+ "sha256": "(64 lowercase hex digits)", "sizeBytes": 166000000,+ "licence": { "name": "Research use only", "commercial": "no", "notice": "Not for commercial use." },+ "provenance": "Trained on ...",+ "output": { "kind": "embedding", "dim": 512 },+ "thresholds": { "matchCosine": 0.4 },+ "speed": 1.7+} ] }+```++`speed` is optional: how many times faster the model is than a ResNet-100 (1.7 for a ResNet-50, 7 for a MobileFaceNet; see the+next section). Settings ▸ Faces shows it as "1.7× faster than a ResNet-100 model", also before the model is downloaded; leave it+out if you have not measured it and nothing is shown. `input` may be left out (112 × 112 RGB, `(x − 127.5) / 127.5`, the ArcFace convention that InsightFace-style models use); say+otherwise with `"input": {"width": 112, "height": 112, "colour": "bgr", "mean": [0,0,0], "std": [1,1,1]}`. `sha256` and+`sizeBytes` are required (a download is checked against them), the address must be `https`, and the model must give one+vector per face. A bad entry is reported in Settings ▸ Faces and skipped; the rest still load. The same file by hand+(Add a model file…) is recognised by its hash and gets the terms you wrote. Models you cannot or do not want to list can+still be added by file; non-commercial models are never offered by the built-in list.+The models folder is `<config>/models` (`%APPDATA%\LightCraft\models` on Windows, `~/Library/Application Support/LightCraft/models`on macOS, `~/.config/lightcraft/models` on Linux), or `$LIGHTCRAFT_FACE_MODELS`. The desktop app, the CLI and the MCPserver share it.@@ -57,6 +118,75 @@ server share it."Commercial use allowed" in the dialog is the weights' licence; it says nothing about the training data.+## How fast are the models++The current pure-Rust runner was measured in standard release mode (no `target-cpu=native`) on Windows,+Ryzen 9 7945HX, over **40 aligned photographic crops** from `corpus/public-faces` (public-domain sources in+`SOURCES.tsv`). Each call runs on one thread. Preprocessing, inference and unit normalization are included;+model loading, photo decoding, detection and alignment are excluded.++| Model | Pure Rust / face (median) | Speed (R100 = 1×) |+| --- | ---: | ---: |+| SFace 2021dec (39 MB) | **33.82–36.29 ms** | **6.9×** |+| AuraFace v1 (261 MB) | **233.18–252.14 ms** | **1×** |++The ranges are medians from two runs (the rerun reproduced the same embedding errors).+Five faces cost about **169–181 ms with SFace**, or **1.17–1.26 s with AuraFace**, just for recognition; ten faces cost+338–363 ms / 2.33–2.52 s. Decoding, detecting and aligning the photo add to this. The background scan parallelizes photos+using its existing bounded worker pools; the recognizer never creates an inner pool. These serial estimates do not imply+whole-library throughput. Convolution dominates the profiled model time. Dense kernels lower at most 256 positions+with contiguous plane copies and explicit border masks; depthwise kernels remain direct. CPU-dispatched Rust+AVX-512 intrinsics improve supported x86 CPUs; other CPUs and wasm use the dependency's Rust fallback.+AuraFace remains a costly CPU choice (its load time is 120–172 ms).++SFace's builtin speed metadata now uses the measured 6.9× ratio. User `catalog.json` ratios remain user-provided.+Older published tables for MobileFaceNet/R50/R100 came from a different runtime and are not measurements of this runner; architecture+names alone do not guarantee the same speed, and those additional model files were not verified in this task.++### Agreement and regression coverage++The public-crop embeddings match the reference runtime's outputs:++| Model | Maximum absolute component difference | Mean absolute difference | Minimum cosine | Mean cosine |+| --- | ---: | ---: | ---: | ---: |+| SFace | 2.0266e-6 | 2.0542e-7 | 0.999999999992 | 0.999999999994 |+| AuraFace | 1.2219e-6 | 1.2725e-7 | 0.999999999990 | 0.999999999993 |++Maximum component difference relative to the largest reference component was 8.60e-6 / 7.10e-6, below 1e-3.+Twenty seeded synthetic recognition nets also agree (maximum absolute embedding difference 7.45e-7).+Unit regressions cover normalization, PReLU broadcasting, transposed Gemm/optional bias, odd grouped convolutions,+reuse of padding, retained scratch/constant-copy limits, invalid manifests, overflow, concurrent calls, and malformed/truncated/ambiguous ONNX fields.+These establish inference agreement, not face-identification accuracy or Lightroom parity.++The shared YuNet graph is also compared against the reference runtime on four synthetic pictures. It passes an absolute+1e-6 plus relative 1e-3 tolerance. The reference's sigmoid uses a clamped rational approximation; LightCraft keeps its existing+exponential sigmoid. Tiny confidence outputs differ by at most 4.47e-7 absolute (6.44% relative on the blank picture),+far below the 0.6 detection threshold. Neither pinned recognizer uses Sigmoid. No approximate activation or reduced+precision was introduced to get the speed figures.++`LC_FACE_MODELS=<folder> cargo test --release -p lightcraft-faces runtime::tests::real_models_pass_the_self_test -- --ignored --nocapture`+runs the installation checks against locally supplied, hash-recognized model files. Models/crops stay out of git.++### And how well do they recognise?++Published numbers only, each copied from the model's own page (we have not re-run them). Higher is better. They are+standard face-verification benchmarks: **IJB-C** (hard, mixed-quality photos; true-accept rate at one false accept in+10,000), **CFP-FP** (frontal against profile) and **AgeDB-30** (years apart), both in percent.++| Model | Speed (R100 = 1×) | IJB-C | CFP-FP | AgeDB-30 | Source |+| --- | --- | --- | --- | --- | --- |+| MobileFaceNet (InsightFace buffalo_s / buffalo_sc) | 7× faster | 95.02 | 98.00 | 96.58 | InsightFace model zoo |+| SFace (OpenCV Zoo) | 6.9× faster (current Rust) | not given | not given | not given | OpenCV Zoo says 0.9940 on a set it does not name |+| ResNet-50 (InsightFace buffalo_l) | 1.7× faster | 97.25 | 99.33 | 98.23 | InsightFace model zoo |+| ResNet-100 (AuraFace v1) | 1× | not given | 95.19 | 96.10 | its Hugging Face page |+| TopoFR R50 / R100 / R200 (Glint360K) | 1.7× / 1× / 0.5× | 97.27 / 97.60 / 97.84 | not given | not given | TopoFR repository |++Read with care: the sources report different sets, so the blanks are not zeros and the rows are not all on the same+yardstick. What the numbers do say: a ResNet-50 recogniser is about two points above a MobileFaceNet on IJB-C, a ResNet-100+or 200 adds only a few tenths more, and AuraFace's published CFP-FP and AgeDB-30 are below buffalo_l's ResNet-50 despite being+a ResNet-100 (so it is not the accuracy choice its size suggests). InsightFace's pretrained weights are, in its own words,+"available for non-commercial research purposes only"; TopoFR's page states no licence for its weights.+## Finding faces yourself**Photo ▸ Detect Faces** (`faces.detect`) runs the YuNet detector on the selected photos and adds what it finds as@@ -68,9 +198,101 @@ The detector's output matches OpenCV's own YuNet on a 45-photo public-domain tesmean box overlap 0.97), including marble busts and paintings, with no false boxes on the landscape and architecturephotos in the set.+## Suggesting who is in a photo++With **Recognise faces** switched on in Settings ▸ Faces and a recognition model chosen (**Use**), LightCraft works out, in+the background, what each face in your library looks like to the model, and uses the faces you have already named to+suggest names for the ones you have not:++- An unnamed face with a good match gets a dim label such as **Jane Doe?** in the loupe. Click it and the name box opens+ with the guess filled in; Enter confirms. Any other unnamed face shows **Add name** when you point at it. Clicking a name+ lets you change it, and clearing the box removes it. Typing completes from the people you have already named.+- **Nothing is ever named for you.** A suggestion is only a label until you confirm it, and a suggestion appears only when+ the match is strong *and* clearly ahead of the next person: it is better to leave a face unnamed than to name it wrongly.+ Naming a face also makes it one of the faces the others are compared with, so the suggestions improve as you go.+- Faces come from the names other apps wrote into your photos (read from XMP), from Photo ▸ Detect Faces, or both. A face is+ aligned using the detector's five landmarks (eyes, nose, mouth corners) when it finds the same face, and cut out by its box+ otherwise.+- Everything stays on your computer. The embeddings (one short list of numbers per face) are cached in the library folder in+ `face-embeddings-<model id>.bin`, one file per model, so switching models back and forth does not start over; they are+ not part of the catalog and not written to XMP.++### The background scan++Turning recognition on starts a scan of the whole library, once, in the background:++- A photo that already has face boxes (from XMP, or drawn, or named) has its faces embedded.+- A photo with **no** face boxes at all is searched with the detector, and what it is sure of becomes that photo's unnamed+ face boxes (marked "Detected by YuNet", so a manual Detect Faces replaces them), embedded in the same pass. Photos are+ searched once: `face-scanned.bin` in the library folder remembers which, so a photo whose boxes you remove is not boxed+ again. These boxes are LightCraft's own bookkeeping: not an undo step, never written to a sidecar.+- Photos you have named faces in go first, then the rest. Raw files are read through the camera's embedded preview (much+ faster than decoding the raw); everything else is rendered at 2048 pixels.+- **How hard it works follows what you are doing.** The app tells the engine on every call to `faces.pump`, which is made+ about 20 times a second while there is work and a few times a minute otherwise (the window is not redrawn for it at any+ other time). Dragging, typing or scrolling: nothing new is started. The pointer moving, or the window minimized:+ **light**, one photo at a time on two threads. Idle for three seconds, or the window still on screen but with another+ app in front (you are working elsewhere): **normal**, half of the processor's threads. Idle and looking at the progress+ (Settings ▸ Faces, or the People view): **full**, four fifths. Photos already running are never interrupted, and a+ change of pace takes effect at once.+- **Threads and photos.** Most of a photo's cost is its parallel work (decoding, developing the picture), so the pace sets the+ size of a pool of threads for that work: two, half the machine, four fifths. The scan has pools of its own, not the one the+ loupe and exports use (which has no priorities), so a slider drag never queues behind a scan. Earlier measurements (with a different runner) on a 32-thread+ desktop with 4 photos at once: about 6.4 photos a second at full pace (a pool of 25 threads), 6.0 at normal (16) and 1.8+ when light (2 threads).+- **Memory** limits the number of photos at once as much as the processor does: a photo in progress holds about 180 MB+ (peak memory grew by that much per extra worker), so at most half of the memory budget (a quarter of the RAM, at most+ 1.5 GiB, unless `LIGHTCRAFT_MEMORY_MB` says otherwise) is given to them: about four photos at once on a default setup,+ whatever the core count. Eight at once was not much faster and needed about twice the memory (a 2.0 GB peak against+ 1.1 GB). `LIGHTCRAFT_FACE_THREADS` replaces these limits with a number of your own. A couple more photos wait behind+ the running ones, so a worker that finishes has its next photo at once.+- **Earlier scan speed (a different runner).** A mixed raw and JPEG library of 184 photos (the raws read through their embedded previews, 73 photos searched+ for faces) took about 29 seconds on that machine at full pace: roughly 6 photos a second, so the first scan of 10,000 photos+ is a matter of half an hour; after that only new photos are looked at. Settings ▸ Faces shows how many are left, with a progress bar. After you accept a model's terms+ you are returned to that tab, so the download and then the scan can be watched there; the main window has no status bar.+- Opening another library starts a fresh scan state; nothing learned about one library is used in another.++### The People view++Until recognition is set up, a line at the top says what it takes and has one button: **Set up face recognition** (no model+yet: it opens Settings ▸ Faces, where the download is) or **Turn on** (a model is installed, recognition is off: it switches it+on there and then). The loupe's name box offers the same ("Set up name suggestions…" / "Turn on name suggestions"), and so+does a person's page.++**People** shows a card for each named person (their face, with the number of photos they are in on the picture) and, below+a line, the **Unnamed faces**: every face nobody has named, as cropped pictures. With recognition running, faces that look+alike are next to each other (put in order by looking at every pair, for up to the first 1,500 embedded faces), and a face+the named ones recognise carries the name they suggest along its bottom edge; click that name to accept it for that face.+To name a group: click faces to select them (Shift-click selects a range, **Select all** takes every one listed), type a+name in the bar that appears (people already named complete as you type), press Enter: all of them are named at once, as+one undo step. Selecting names nothing.++The thumbnail-size slider in the bottom bar sizes the faces here too, with limits of their own. Every face is shown by the+same kind of box: the detector's own box once the scan has looked at it (kept beside its embedding), so a loosely drawn box+from another tool does not make one face look farther away than the next; before the scan has looked at a face it is shown+by its own box.++### A person's page++In **People**, a click on a person opens their page: **only cropped faces**, never whole photos. First the faces named+with their name (a click opens that photo), then **More**: the unnamed faces that look like them, most alike first. Click one+to confirm it (that names the face, as one undo step, and it moves up into their faces), or × to hide it for this session.+**Show photos** puts their photos in the grid; **‹ People** or Escape goes back. "More" only holds faces the scan has+already embedded, so it fills in as the scan goes; it shows faces scoring above four fifths of the model's suggestion bar,+since you look at each one before anything is named.++How well it works: on 755 named faces of marble busts from one museum folder (each face hidden in turn and matched against+shots taken more than five seconds apart), SFace named the right person first 95.5% of the time and AuraFace 94.7%; at+the starting thresholds LightCraft suggests (SFace 0.55, AuraFace 0.40) about 97% of the suggestions were right. Busts are+a hard case in some ways (no skin or hair to go by) and an easy one in others (the same sculpture looks the same in every+shot), so check a model on your own photos before trusting it:++`faces.evaluate` tests a model on *your* photos: it hides each named face in turn, asks who it looks like from the others+(ignoring shots taken within a few seconds of it, which would make it too easy) and reports, for each threshold, how many+suggestions it would make and how many were right.+## CommandsAll of this is reachable from the control channel, the CLI and MCP: `faces.models.list`, `faces.models.inspect {path}`,-`faces.models.install {path, acknowledged: true}`, `faces.models.download {id, acknowledged: true}` (then `faces.models.downloads` for-progress; the download is installed when it arrives, and `faces.models.downloadCancel {id}` stops it), `faces.models.remove {id}`, `faces.models.select {id}`,-`faces.enable {enabled?}`, and `faces.detect {ids?, apply?}`. `acknowledged` must be `true`: the caller has shown the user the terms and the user agreed.+`faces.models.install {path, acknowledged: true, activate?}`, `faces.models.download {id, acknowledged: true}` (then `faces.models.downloads`, which also installs what has arrived, and `faces.models.downloadCancel {id}`), `faces.models.remove {id}`, `faces.models.select {id}`,+`faces.enable {enabled?}`, `faces.detect {ids?, apply?}`, `faces.index {budgetMs?, ids?}`, `faces.pump` (what the app calls every frame), `faces.suggest {ids?, threshold?, margin?}`, `faces.person {name, more?}` (a person's faces and the unnamed faces that look like them), `faces.unnamed {limit?}` (every unnamed face, look-alikes together, with suggested names), `faces.setName {id?, index, name}`, `faces.nameFaces {faces: [{photo, index}], name}` (name many at once, one undo step) and `faces.evaluate`. `acknowledged` must be `true`: the caller has shown the user the terms and the user agreed. Installing makes the model the one in use and switches recognition on unless `activate` is `false`.diff --git a/docs/parity.md b/docs/parity.mdindex 1b9447fb2..476cb7dc1 100644--- a/docs/parity.md+++ b/docs/parity.md@@ -87,6 +87,10 @@ Take the first one nobody is working on.Settings → Performance → Preview size (2,560 px by default). Render visible regions at native resolution so100% can reliably show original-pixel detail on larger photos.+Recognition runtime gap closed (LR-PREF-PEOPLE): the pinned SFace and AuraFace models now run on the shared pure-Rust+CPU engine and agree with an independent reference runtime on public crops. Live-face threshold calibration, small-face+coverage and separated clusters remain open; see [faces.md](faces.md).+## Shortcuts: conflicts and missing bindingsCompared `plan/lightroom/06-shortcuts.md` (desktop part) with our bindings: command specs in `crates/engine/src/cmd/`,@@ -160,7 +164,7 @@ under the UI command that wraps it (a no-op error); the UI command now wins.| LR-LIB-FLAG | Pick / reject flags | P0 | ✅ | `cmd:photo.pick`, `cmd:photo.reject`, `cmd:photo.unflag`, `cmd:photo.flag` | P pick, X reject, U clear; Shift+P picks and advances in both grids (Presets elsewhere); Shift+X/U and legacy Shift+Z advance; no flag cycle; rejection does not delete the file || LR-LIB-LABEL | Colour labels | P1 | ✅ | `cmd:photo.label`, `cmd:label.setNames`, `cmd:label.names`, `cmd:dialog.labelNames`, keys 6–9 in `crates/ui-egui/src/shortcuts.rs` | Photo menu and grid context menu (coloured, named), Info-panel swatches, label dot plus translucent colour on square-grid/filmstrip surrounds and photo-grid footers (photo pixels unchanged); colour-label toasts have pale matching backgrounds; setting/clearing through keys (including Shift), menus or Info swatches shows the same bottom toast as ratings, with the colour/custom name; editable label names (undoable, journaled); no purple key (as in Lightroom), no custom extra labels || LR-LIB-KEYWORD | Keywords | P0 | ✅ | `cmd:panel.keywords`, `cmd:photo.setMeta` (`addKeywords`/`removeKeywords`), `cmd:keyword.list`, `cmd:keyword.rename`, `cmd:keyword.delete`, `cmd:keyword.merge`, `cmd:keyword.suggest`, `crates/catalog/src/keywords.rs`, `crates/ui-egui/src/panels/left.rs` | left-panel keyword tree with counts (click filters, children included; context menu rename / merge / delete / add / remove); library-wide ops are one undo step and replay from the op log; suggestions (co-occurring / most used / completions) in the Keywords panel; no keyword drag-and-drop |-| LR-LIB-PEOPLE | People / faces | P2 | 🟡 | `cmd:faces.detect`, `cmd:view.people`, `cmd:view.faceBoxes`, `cmd:photo.removeRegion`, `cmd:photo.setRegion`, `cmd:library.filter`, `crates/ui-egui/src/panels/people.rs`, `crates/catalog/src/query.rs` (`people`), `crates/engine/src/media.rs` (`face_job`), `crates/meta/src/xmp.rs` | MVP: named face regions read from XMP (MWG-RS, read-only, see `docs/xmp-interop.md`; clipped to the photo; Lightroom's `Rotation` of ±π/2 and π handled, mirrored orientations taken as written; boxes follow Rotate Left/Right and flips; a sidecar stating `mwg-rs:Regions`, even empty, replaces the photo's on re-read, one without leaves them) become cards in the People view (View ▸ People, or the People button in the toolbar): a close-up of the person's largest face, name and photo count; a click filters the grid (`person` in `library.filter`, `person:` search token, filter chip). Face boxes in the loupe can be switched off (View ▸ Face Boxes, toolbar) resized with eight drag handles and removed with the × shown on hover (catalog-only and undoable, one undo step per drag; the sidecar is never rewritten for either); no moving a box by dragging it yet. Photo ▸ Detect Faces (`cmd:faces.detect`) finds faces with the YuNet detector (pure Rust, about 200 ms a photo, matches OpenCV's own YuNet on a 45-photo test set; the 232 KB model is a one-time download offered in the same step, never part of LightCraft) and adds them as unnamed boxes; no recognition, no unnamed people / suggestions, no manual tagging, no writing regions back |+| LR-LIB-PEOPLE | People / faces | P2 | 🟡 | `cmd:faces.detect`, `cmd:view.people`, `cmd:view.faceBoxes`, `cmd:photo.removeRegion`, `cmd:photo.setRegion`, `cmd:library.filter`, `crates/ui-egui/src/panels/people.rs`, `crates/catalog/src/query.rs` (`people`), `crates/engine/src/media.rs` (`face_job`), `crates/meta/src/xmp.rs` | MVP: named face regions read from XMP (MWG-RS, read-only, see `docs/xmp-interop.md`; clipped to the photo; Lightroom's `Rotation` of ±π/2 and π handled, mirrored orientations taken as written; boxes follow Rotate Left/Right and flips; a sidecar stating `mwg-rs:Regions`, even empty, replaces the photo's on re-read, one without leaves them) become cards in the People view (View ▸ People, or the People button in the toolbar): a close-up of the person's largest face, name and photo count; a click filters the grid (`person` in `library.filter`, `person:` search token, filter chip). Face boxes in the loupe can be switched off (View ▸ Face Boxes, toolbar) resized with eight drag handles and removed with the × shown on hover (catalog-only and undoable, one undo step per drag; the sidecar is never rewritten for either); no moving a box by dragging it yet. Photo ▸ Detect Faces (`cmd:faces.detect`) finds faces with the YuNet detector (pure Rust, about 200 ms a photo, matches OpenCV's own YuNet on a 45-photo test set; the 232 KB model is a one-time download offered in the same step, never part of LightCraft) and adds them as unnamed boxes; names are suggested from the faces already named (recognition models are opt-in downloads, run on the CPU in a background thread; a face is named only when you confirm it); a person's page shows their cropped faces and the unnamed faces that look like them (LRC-LIB-PEOPLE); with recognition on, a background scan finds faces in photos that have none and embeds every face (one to three worker threads, raws through their embedded preview); an Unnamed faces section orders faces by likeness and names a selected group at once; no separated clusters, no manual tagging, no writing regions back || LR-LIB-STACK | Stacks | P1 | ✅ | `cmd:stack.group`, `cmd:stack.ungroup`, `cmd:stack.toggle`, `cmd:stack.setTop`, `cmd:stack.remove`, `cmd:stack.auto`, `crates/catalog/src/stacks.rs` | grid/filmstrip count badges, expand/collapse, auto-stack by capture time; no visual-similarity auto-stack || LR-LIB-VERSIONS | Versions | P1 | ✅ | `cmd:version.create` | see section M || LR-LIB-DELETE | Delete / Recently Deleted | P0 | ✅ | `cmd:photo.delete`, `cmd:photo.restore`, `cmd:photo.deletePermanently` | Restore and Delete Permanently in the Photo menu and the photo context menus for photos in Recently Deleted (before, the UI had no way to restore); adding a file that is in Recently Deleted again selects it there (side panel opened) and says how to restore it or import it afresh; Photo ▸ Empty Recently Deleted (`cmd:library.emptyRecentlyDeleted`, one undo step, shown while the trash is the source); import takes `onDeleted` skip / restore / fresh for files in the trash (`cmd:library.import`; the report flags `existingDeleted` and lists `restored`; the import review dialog offers Leave them / Restore them / Import as new for files that are in the trash; dropped files and Add Photos without a review still skip and point at Restore); no confirmation dialog, no auto-purge after N days (needs a deleted-at date in the catalog) |@@ -482,7 +486,7 @@ under the UI command that wraps it (a no-op error); the UI command now wins.| LR-PREF-ACCOUNT | Account | OOS | 🚫 | | || LR-PREF-INTERFACE | Interface options | P1 | ✅ | `cmd:app.settings` | filmstrip names/badges, grid badges (auto/always/never), square-grid names, navigator, info overlay || LR-PREF-PERFORMANCE | GPU / performance | P1 | ✅ | `cmd:app.gpu`, `cmd:app.memoryBudget`, `cmd:app.settings` | GPU on/off, preview size (Automatic = the whole-frame loupe render at canvas size, up to 2560 px; or a 1600–5120 px size for it; zoomed views add a window at up to 100 %, issue #323), memory budget (a quarter of it is the idle per-view stage caches' share; the open photo's views are counted and reported, and cut only past four times that, the before side's and the zoom window's first — a Before/After at 1:1 on a 2800 px canvas holds about 1 GB of GPU stages, plus the one 288 MB copy of the original, which `ui.inspect` reports as `sharedSourceBytes`), thumbnail cache size in Settings |-| LR-PREF-PEOPLE | Face recognition | P2 | 🟡 | `cmd:faces.enable`, `cmd:faces.models.list`, `cmd:faces.models.inspect`, `cmd:faces.models.install`, `cmd:faces.models.download`, `cmd:faces.models.downloads`, `cmd:faces.models.downloadCancel`, `cmd:faces.models.remove`, `cmd:faces.models.select`, `cmd:dialog.faceModel`, `crates/faces/src/lib.rs`, `crates/ui-egui/src/panels/faces.rs` | Settings ▸ Faces: an off-by-default toggle and the face model list. Models are opt-in and never bundled: **Download** fetches one from its own repository (pure-Rust https via `lightcraft-fetch`, address pinned to a commit, size and SHA-256 checked, installed by itself) after its terms are shown and accepted, or add a `.onnx` by picking or dropping it (recognised by hash or described from its shape, with its licence terms and training-data note, and an "I accept" box). The YuNet detector is wired (Photo ▸ Detect Faces); nothing runs a recognition model yet (no inference runtime), so nothing recognises faces yet |+| LR-PREF-PEOPLE | Face recognition | P2 | 🟡 | `cmd:faces.enable`, `cmd:faces.models.list`, `cmd:faces.models.inspect`, `cmd:faces.models.install`, `cmd:faces.models.download`, `cmd:faces.models.downloads`, `cmd:faces.models.downloadCancel`, `cmd:faces.models.remove`, `cmd:faces.models.select`, `cmd:dialog.faceModel`, `crates/faces/src/lib.rs`, `crates/ui-egui/src/panels/faces.rs` | Settings ▸ Faces: an off-by-default toggle and the face model list. Models are opt-in and never bundled, the 232 KB YuNet detector included. Press Download (terms first; then a pinned address, size and SHA-256 checked, fetched over pure-Rust https with `lightcraft-fetch`), or add any `.onnx` by picking or dropping it: it is recognised by hash or described from its shape, shows its licence terms and training-data note, runs a self-test, and installs only after an "I accept" box; an installed model is chosen and recognition switched on at once (the newest wins, each model keeps its own embedding cache). Non-commercial models are add-by-file only, never offered for download. Suggestions of names run on the shared pure-Rust CPU interpreter (`crates/faces/src/net.rs`, `crates/faces/src/runtime.rs`; bounded im2col/GEMM, checked ONNX shapes/operators; no external inference runtime; 40 public crops agree within 2.1e-6, one-thread SFace 34–36 ms / AuraFace 233–252 ms, see docs/faces.md) in a background worker and are never applied automatically. Thresholds are starting points calibrated on busts, not yet on live faces| LR-PREF-WATERMARK | Watermark settings | P1 | ✅ | `cmd:export.savePreset`, `crates/ui-egui/src/panels/dialogs.rs` | set in the Export dialog; kept with Export with Previous and in saved export presets || LR-PREF-SHORTCUTS | Shortcut customisation | — | ✅ | `cmd:app.setShortcut`, `cmd:app.resetShortcuts`, `crates/ui-egui/src/panels/keymap.rs` | an extra (the reference app has no keymap editor): Help ▸ Keyboard Shortcuts (⌘/) lists every command; click a shortcut and press the new keys (Esc cancels), × removes, ↺ restores, Reset All; a key another command had moves over; saved in `ui.json`; native menu accelerators follow. Rating/label digits and secondary keys are listed read-only and yield to a key given to a command || LR-PREF-TECHPREVIEW | Early-access toggles | P2 | ⬜ | | |@@ -747,7 +751,7 @@ From `08-lightroom-classic-extras.md` (Classic-only features) and part 2 of `06-| LRC-LIB-KEYWORDS | Hierarchical keywords, sets, painter | P1 | ✅ | `crates/catalog/src/keywords.rs`, `cmd:keyword.list`, `cmd:keyword.sets`, `cmd:keyword.toggleFromSet`, `cmd:keyword.saveSet`, `cmd:tool.keywordPainter` | hierarchical `a\|b\|c` keywords (tree, parent filters include children, rename moves children); keyword sets + Recent Keywords (⌥1–⌥9); keyword painter (Keywords panel ▸ Paint: click photos in the grid to toggle a keyword, Esc stops) || LRC-LIB-METADATA | Metadata panel & presets | P1 | ✅ | `cmd:photo.setMeta`, `cmd:metadata.savePreset`, `cmd:metadata.applyPreset`, `cmd:metadata.presets`, `cmd:metadata.deletePreset`, `cmd:dialog.captureTime` | IPTC core, accessibility, place, capture-time edit; copyright status (unknown / copyrighted / public domain), rights usage terms and copyright info URL (`photo.setMeta` `copyrightStatus`, `usageTerms`, `copyrightUrl`; XMP Rights Management fields, kept by "copyright only" exports, a smart-album rule); metadata presets (Photo → Metadata Preset, Save Metadata Preset… from the active photo; applied on import from Settings) || LRC-LIB-QUICKDEV | Quick develop | P2 | ✅ | `cmd:develop.quickAdjust`, `crates/ui-egui/src/panels/edit.rs` (`quick_develop`) | grid with several photos selected: Edit panel ▸ Quick Develop steps (exposure ⅓ / 1 stop, contrast, highlights, shadows, whites, blacks, clarity, vibrance, temperature) added to each photo's own value, one undo step |-| LRC-LIB-PEOPLE | People view | P2 | 🟡 | `cmd:view.people`, `crates/ui-egui/src/panels/people.rs` | Named People cards (face close-up, name, photos) for the photos the active filters let through, with the filter chips shown above them; no Unnamed People, no per-person page, no confirm / reject |+| LRC-LIB-PEOPLE | People view | P2 | 🟡 | `cmd:view.people`, `cmd:view.person`, `cmd:faces.person`, `cmd:faces.unnamed`, `cmd:faces.nameFaces`, `crates/ui-egui/src/panels/people.rs`, `crates/ui-egui/src/panels/person.rs`, `crates/ui-egui/src/panels/unnamed.rs` | Named People cards (face close-up, the photo count on the picture, name) for the photos the active filters let through, with the filter chips shown above them, sized by the thumbnail slider. Below them the **Unnamed faces**: every face nobody has named as a cropped picture, look-alikes next to each other once recognition has looked at them, a suggested name on a face the named ones recognise; click or Shift-click to select, type a name, Enter names the whole group in one undo step. A click on a person opens their page: only cropped faces, then **More** (unnamed faces that look like them, best first; click to confirm, × to hide). Every face is shown by the detector's box where the scan has one, so faces are equally close. Until recognition is set up a line at the top offers the next step (Settings ▸ Faces, or Turn on), as does the loupe's name box. No clusters with separators between groups, no persistent reject, no merge of two people| LRC-LIB-COMMENTS | Comments panel | P2 | ⬜ | | || LRC-LIB-VISUALSEARCH | Find similar photos | P2 | ✅ | `cmd:library.findSimilar`, `crates/pipeline/src/cull.rs` (`signature`) | photo menu / Photo ▸ Find Similar Photos: look-alike photos (composition and tones), most similar first, as a filter (`only`) || LRC-LIB-MISSING | Missing files & relink | P1 | ✅ | `cmd:library.missing`, `cmd:photo.relink`, `cmd:library.findMissing`, `cmd:file.findMissing`, `cmd:photo.locate`, `crates/engine/src/cmd/missing.rs` (`checked_path`) | File → Find Missing Photos… (anywhere in a folder: same name, size and content hash; renamed files by size + content hash; same name and size alone only when unambiguous — look-alikes are skipped and reported; searched on a worker thread, one undo step), photo menu → Locate Missing File…; unreadable files show "!" in the grid and a reason in the loupe; undo never moves files. Missing Photos in the sidebar (counted on a worker thread; the view, the Info panel, the grid's context menu and thumbnails read file availability from a cache a worker thread fills, so an offline NAS never stalls a frame); it covers library photos only — Local browse records are never checked, for the count, the view, `library.missing` and Find Missing alike |