π End-to-end encrypted cloud for everything.
Dart 2faandroidauthye2eeencryptionend-to-end-encryptionfluttergolanggoogle-photosgoogle-photos-alternativeiosopensourcephotosprivacyprivacy-toolstypescriptzero-knowledge
[photos] Accelerate internal Rust OCR with shared GPU models
laurenspriemf6e82ac769dd
diff --git a/infra/ml/playground/optimizations/README.md b/infra/ml/playground/optimizations/README.mdindex d5ad854a08..89c468eea8 100644--- a/infra/ml/playground/optimizations/README.md+++ b/infra/ml/playground/optimizations/README.md@@ -5,6 +5,97 @@ ML models. The generated CDN artifacts are written under `models/`; ONNX filesare intentionally gitignored, while `model_manifest.json` records thereproducible output metadata.+## OCR models++Generate one shared PP-OCRv5 detector, classifier, and recognizer for CoreML and+native WebGPU with:++```sh+uv run --no-project --with numpy==2.5.3 --with onnx==1.22.0 python \+ infra/ml/playground/optimizations/optimize_ocr_models.py \+ --source-dir infra/ml/playground/.cache/ocr-sources \+ --output-dir infra/ml/playground/optimizations/models/ocr+```++Missing source files are downloaded from `https://models.ente.com/PP-OCRv5`.+The script verifies every source hash and requires every generated model to+match the SHA-256 of its qualified artifact. It writes `det.onnx`, `cls.onnx`,+`rec.onnx`, the unchanged dictionary, and `ocr_model_manifest.json`. Both+platforms download identical model bytes. Upload the ONNX files to new versioned+URLs; the recognizer has a new output contract and must not replace the old URL.++The selected transformations are:++- Detection: fold the final transposed-convolution bias and batch normalization+ into the head weights. Preserve the original backbone affine expressions.+- Classification: canonicalize scalar affine constants and express 18 expanded+ activations as `HardSigmoid(alpha=1/6, beta=0.5) * x`.+- Recognition: canonicalize scalar affine constants, fold input affine operations+ into 12 unpadded convolutions, and replace four expanded normalization blocks+ with `LayerNormalization` using the original parameters and epsilon.+- Detection and recognition: expand `HardSwish` into+ `HardSigmoid(alpha=1/6, beta=0.5) * x` for CoreML compatibility.+- Recognition: pad the final weight matrix's 18,385 output columns to 18,432 for+ the existing WebGPU runtime's vectorized matrix multiplication, then slice+ back to 18,385 before the original bias and softmax.+- Recognition: reduce the original probabilities to the first winning token+ index and its probability for each time step, using FP32 arithmetic supported+ by both GPU providers.++The recognizer returns FP32 `[N,T,2]`, with index first and probability second.+For probabilities `p`, maximum `m`, vocabulary size `V`, and token index `i`, the+compact head computes `min(i + V * ceil(m - p[i]))`. Since probabilities are in+`[0,1]`, the ceiling is zero for maxima and one for every lower value. The+minimum therefore selects the first tied maximum without integer casts or+boolean selection. The other output is the original maximum probability.+See the [ONNX Ceil definition](https://onnx.ai/onnx/operators/onnx__Ceil.html).++The shared models retain FP32 parameters, vocabulary, CTC blank/repeat handling,+confidence averaging, and character-span rules. These transformations preserve+the mathematical operations but can introduce small floating-point differences.++The Rust OCR caller keeps its 960-pixel longest-side detector limit and existing+recognition crop sizes. It specializes sessions for actual dimensions without+changing input pixels and retains two detector shapes, six classifier batch+sizes, and eight recently used recognizer shapes.+CoreML uses MLProgram, CPUAndGPU, and disables low-precision GPU accumulation.+WebGPU uses NHWC internally for detection and NCHW for recognition. GPU session+construction requires full provider coverage; failures retry on a single CPU+thread. OCR never selects XNNPACK. Android classification uses the single-thread+CPU because it was faster than WebGPU on Pixel 8; it uses the same classifier+file as CoreML. Internal iOS uses CoreML for all three stages.++Qualification uses unchanged ONNX Runtime 1.28.1 on Pixel 8 with native+Dawn/Vulkan WebGPU and Apple M4 with CoreML. The saved model comparison covers+317 crop sources from 17 unique photos, 21,256 recognition time steps, 320+classification inputs, and 9,383,936 detector probabilities. Neither provider+changed any winning token, orientation decision, or detector threshold decision+in those cases. Every CoreML case formed one complete partition using GPU+compute devices; both providers completed with CPU fallback disabled. Separate+compact output checks matched all 144 rows exactly, including ties, uniform+probabilities, and adjacent FP32 values. These checks do not establish bit-identical outputs on+all images or performance on an iPhone. Smaller detector inputs were excluded+because they missed text in the supplied photos.++The shared artifacts produced these warm native inference medians on 10+September 2026. Each range contains two trial medians, with 40 calls per trial+and the first 10 discarded. Runs use one CPU inference thread and include the+normal CPU input/output transfer; model loading, compilation, image decoding,+and Rust preprocessing/postprocessing are excluded. CoreML measurements are+from an Apple M4 Mac, not an iPhone.++| Stage and NCHW input | M4 CoreML GPU | Pixel 8 native WebGPU |+| --- | ---: | ---: |+| Empty-image detection, 1Γ3Γ960Γ704 | 10.4β10.9 ms | 128.6β132.2 ms |+| Empty-image detection, 1Γ3Γ960Γ960 | 13.3β13.6 ms | 173.3β175.0 ms |+| Classification, 6Γ3Γ48Γ192 | 1.4β2.0 ms | 19.3β33.9 ms |+| Recognition, 6Γ3Γ48Γ320 | 9.5β9.6 ms | 104.1β104.8 ms |+| Recognition, 6Γ3Γ48Γ640 | 16.8β17.0 ms | 189.5β190.4 ms |++The same classifier took 8.7β8.8 ms on the Pixel's single-thread CPU, which is+why the Android caller keeps that stage on CPU. Both uniform-gray detection+inputs produced no pixels at or above the 0.3 threshold.+## Rebuilding the modelsRun from the repository root:diff --git a/infra/ml/playground/optimizations/models/ocr/.gitignore b/infra/ml/playground/optimizations/models/ocr/.gitignorenew file mode 100644index 0000000000..6801ebbe54--- /dev/null+++ b/infra/ml/playground/optimizations/models/ocr/.gitignore@@ -0,0 +1 @@+ppocrv5_dict.txtdiff --git a/infra/ml/playground/optimizations/models/ocr/ocr_model_manifest.json b/infra/ml/playground/optimizations/models/ocr/ocr_model_manifest.jsonnew file mode 100644index 0000000000..56f26722f7--- /dev/null+++ b/infra/ml/playground/optimizations/models/ocr/ocr_model_manifest.json@@ -0,0 +1,93 @@+{+ "source_base_url": "https://models.ente.com/PP-OCRv5",+ "source_sha256": {+ "cls.onnx": "f4bb53707100c5f3d59ba834eb05bb400369f20aed35d4b26807b1bfadd2a70e",+ "det.onnx": "d7fe3ea74652890722c0f4d02458b7261d9f5ae6c92904d05707c9eb155c7924",+ "ppocrv5_dict.txt": "d1979e9f794c464c0d2e0b70a7fe14dd978e9dc644c0e71f14158cdf8342af1b",+ "rec.onnx": "bf66820f48fa99f779974c4df78e5274a9d8e0458c4137e8c5357e40e2c3faf2"+ },+ "onnx": "1.22.0",+ "models": [+ {+ "file": "det.onnx",+ "sha256": "3e658f85236f1984e186070048d7cdc99ddbb84e12c91a6fd80922b85bd9487a",+ "bytes": 4750896,+ "operators": [+ "Add",+ "BatchNormalization",+ "Concat",+ "Constant",+ "Conv",+ "ConvTranspose",+ "GlobalAveragePool",+ "HardSigmoid",+ "Mul",+ "Relu",+ "Reshape",+ "Resize",+ "Sigmoid"+ ],+ "output": "unchanged"+ },+ {+ "file": "cls.onnx",+ "sha256": "8675903197a1bab7060e830e9bcc63d29c5cf7d372735338bd9501f829e94ed1",+ "bytes": 574875,+ "operators": [+ "Add",+ "BatchNormalization",+ "Cast",+ "Concat",+ "Constant",+ "Conv",+ "GlobalAveragePool",+ "HardSigmoid",+ "MatMul",+ "MaxPool",+ "Mul",+ "Relu",+ "Reshape",+ "Shape",+ "Slice",+ "Softmax"+ ],+ "output": "unchanged"+ },+ {+ "file": "rec.onnx",+ "sha256": "667691bb173a94106b595aa87c7d176b53b68cf831b6ffa09569018754265e57",+ "bytes": 16686404,+ "operators": [+ "Add",+ "AveragePool",+ "BatchNormalization",+ "Ceil",+ "Concat",+ "Constant",+ "Conv",+ "Div",+ "GlobalAveragePool",+ "HardSigmoid",+ "LayerNormalization",+ "MatMul",+ "Mul",+ "Pow",+ "ReduceMax",+ "ReduceMean",+ "ReduceMin",+ "Relu",+ "Reshape",+ "Shape",+ "Sigmoid",+ "Slice",+ "Softmax",+ "Sqrt",+ "Squeeze",+ "Sub",+ "Transpose",+ "Unsqueeze"+ ],+ "output": "FP32 [N,T,2]: first winning index, original winning probability"+ }+ ]+}diff --git a/infra/ml/playground/optimizations/optimize_ocr_models.py b/infra/ml/playground/optimizations/optimize_ocr_models.pynew file mode 100644index 0000000000..72284d76cd--- /dev/null+++ b/infra/ml/playground/optimizations/optimize_ocr_models.py@@ -0,0 +1,547 @@+import argparse+import copy+import hashlib+import json+from pathlib import Path+from urllib.request import urlopen++import numpy as np+import onnx+from onnx import helper, numpy_helper++SOURCE_BASE_URL = "https://models.ente.com/PP-OCRv5"+SOURCE_HASHES = {+ "cls.onnx": "f4bb53707100c5f3d59ba834eb05bb400369f20aed35d4b26807b1bfadd2a70e",+ "det.onnx": "d7fe3ea74652890722c0f4d02458b7261d9f5ae6c92904d05707c9eb155c7924",+ "ppocrv5_dict.txt": "d1979e9f794c464c0d2e0b70a7fe14dd978e9dc644c0e71f14158cdf8342af1b",+ "rec.onnx": "bf66820f48fa99f779974c4df78e5274a9d8e0458c4137e8c5357e40e2c3faf2",+}+OUTPUT_HASHES = {+ "det.onnx": "3e658f85236f1984e186070048d7cdc99ddbb84e12c91a6fd80922b85bd9487a",+ "cls.onnx": "8675903197a1bab7060e830e9bcc63d29c5cf7d372735338bd9501f829e94ed1",+ "rec.onnx": "667691bb173a94106b595aa87c7d176b53b68cf831b6ffa09569018754265e57",+}+++def constant_arrays(model):+ result = {v.name: numpy_helper.to_array(v) for v in model.graph.initializer}+ for node in model.graph.node:+ if node.op_type == "Constant":+ for attr in node.attribute:+ if attr.name == "value":+ result[node.output[0]] = numpy_helper.to_array(attr.t)+ return result+++def remove_unused_nodes(model):+ producers = {v: node for node in model.graph.node for v in node.output}+ needed = set()+ pending = [v.name for v in model.graph.output]+ while pending:+ name = pending.pop()+ if name in needed:+ continue+ needed.add(name)+ if name in producers:+ pending.extend(producers[name].input)+ nodes = [n for n in model.graph.node if any(v in needed for v in n.output)]+ del model.graph.node[:]+ model.graph.node.extend(nodes)+ for field in ["initializer", "value_info"]:+ values = [v for v in getattr(model.graph, field) if v.name in needed]+ del getattr(model.graph, field)[:]+ getattr(model.graph, field).extend(values)+ return model+++def canonicalize_scalar_affines(model):+ values = constant_arrays(model)+ inferred = onnx.shape_inference.infer_shapes(model)+ shapes = {v.name: v.type.tensor_type.shape for v in inferred.graph.value_info}+ count = 0+ for node in model.graph.node:+ if node.op_type not in ["Mul", "Add"]:+ continue+ indices = [+ i for i, n in enumerate(node.input) if n in values and values[n].size == 1+ ]+ if len(indices) != 1:+ continue+ index = indices[0]+ value = values[node.input[index]].reshape(())+ if value.dtype != np.float32:+ continue+ replacement = value if node.op_type == "Mul" else None+ shape = shapes.get(node.output[0])+ if (+ node.op_type == "Add"+ and shape is not None+ and (len(shape.dim) == 4)+ and shape.dim[1].dim_value+ ):+ replacement = np.full(+ (shape.dim[1].dim_value, 1, 1), value, dtype=np.float32+ )+ if replacement is None:+ continue+ name = f"modelopt_affine_{count}"+ model.graph.initializer.append(numpy_helper.from_array(replacement, name))+ node.input[:] = [node.input[1 - index], name]+ count += 1+ return (remove_unused_nodes(model), count)+++def scalar_affine_input(node, values):+ if node.op_type not in ["Mul", "Add"]:+ return None+ indices = [+ i+ for i, n in enumerate(node.input)+ if n in values and values[n].size and np.all(values[n] == values[n].flat[0])+ ]+ if len(indices) != 1:+ return None+ index = indices[0]+ return (node.input[1 - index], float(values[node.input[index]].flat[0]))+++def fold_input_affine(model):+ values = constant_arrays(model)+ producers = {v: n for n in model.graph.node for v in n.output}+ count = 0+ for node in model.graph.node:+ if node.op_type != "Conv" or node.input[1] not in values:+ continue+ attrs = {a.name: helper.get_attribute_value(a) for a in node.attribute}+ if any(attrs.get("pads", [])) or attrs.get("auto_pad", b"NOTSET") != b"NOTSET":+ continue+ current = node.input[0]+ scale, bias, steps = (1.0, 0.0, 0)+ while current in producers:+ parent = producers[current]+ affine = scalar_affine_input(parent, values)+ if affine is None:+ break+ current, value = affine+ if parent.op_type == "Mul":+ scale *= value+ else:+ bias += scale * value+ steps += 1+ if not steps:+ continue+ weights = values[node.input[1]]+ old_bias = (+ values[node.input[2]]+ if len(node.input) > 2+ else np.zeros(weights.shape[0], dtype=np.float32)+ )+ new_weights = (weights.astype(np.float64) * scale).astype(np.float32)+ new_bias = (+ old_bias.astype(np.float64)+ + weights.astype(np.float64).sum(axis=tuple(range(1, weights.ndim))) * bias+ ).astype(np.float32)+ weight_name, bias_name = (+ f"modelopt_input_weight_{count}",+ f"modelopt_input_bias_{count}",+ )+ model.graph.initializer.extend(+ [+ numpy_helper.from_array(new_weights, weight_name),+ numpy_helper.from_array(new_bias, bias_name),+ ]+ )+ node.input[:] = [current, weight_name, bias_name]+ count += 1+ return (remove_unused_nodes(model), count)+++def align_recognizer_projection(model, width):+ values = constant_arrays(model)+ node = next(n for n in model.graph.node if n.name == "MatMul.12")+ weights = values[node.input[1]]+ assert weights.shape == (120, 18385)+ padded = np.pad(weights, ((0, 0), (0, width - 18385)))+ model.graph.initializer.extend(+ [+ numpy_helper.from_array(padded, "modelopt_projection_weight"),+ numpy_helper.from_array(+ np.array([0], dtype=np.int64), "modelopt_slice_start"+ ),+ numpy_helper.from_array(+ np.array([18385], dtype=np.int64), "modelopt_slice_end"+ ),+ numpy_helper.from_array(+ np.array([2], dtype=np.int64), "modelopt_slice_axis"+ ),+ ]+ )+ node.input[1] = "modelopt_projection_weight"+ output = node.output[0]+ node.output[0] = "modelopt_padded_projection"+ index = next((i for i, n in enumerate(model.graph.node) if n.name == node.name))+ model.graph.node.insert(+ index + 1,+ helper.make_node(+ "Slice",+ [+ node.output[0],+ "modelopt_slice_start",+ "modelopt_slice_end",+ "modelopt_slice_axis",+ ],+ [output],+ name="modelopt_projection_slice",+ ),+ )+ return remove_unused_nodes(model)+++def fuse_recognizer_layer_norms(model):+ model = onnx.version_converter.convert_version(model, 17)+ values = constant_arrays(model)+ specs = [+ ("Add.187", "Add.191", "layer_norm_1", "helper.constant.26"),+ ("Add.197", "Add.201", "layer_norm_2", "helper.constant.33"),+ ("Add.207", "Add.211", "layer_norm_3", "helper.constant.49"),+ ("Add.217", "Add.221", "layer_norm_4", "helper.constant.56"),+ ]+ count = 0+ for source, output, prefix, epsilon in specs:+ index = next((i for i, n in enumerate(model.graph.node) if output in n.output))+ weight_name, bias_name = (+ f"{prefix}_modelopt_weight",+ f"{prefix}_modelopt_bias",+ )+ model.graph.initializer.extend(+ [+ numpy_helper.from_array(+ values[prefix + ".w_0"].reshape(-1), weight_name+ ),+ numpy_helper.from_array(values[prefix + ".b_0"].reshape(-1), bias_name),+ ]+ )+ model.graph.node[index].CopyFrom(+ helper.make_node(+ "LayerNormalization",+ [source, weight_name, bias_name],+ [output],+ name=prefix + "_modelopt",+ axis=-1,+ epsilon=float(values[epsilon].item()),+ stash_type=1,+ )+ )+ count += 1+ return (remove_unused_nodes(model), count)+++def expand_hardswish(model):+ nodes = []+ for node in model.graph.node:+ if node.op_type != "HardSwish":+ nodes.append(node)+ continue+ name = node.output[0] + "_modelopt_gate"+ nodes.extend(+ [+ helper.make_node(+ "HardSigmoid",+ [node.input[0]],+ [name],+ name=node.name + "_gate",+ alpha=1 / 6,+ beta=0.5,+ ),+ helper.make_node(+ "Mul",+ [node.input[0], name],+ list(node.output),+ name=node.name + "_multiply",+ ),+ ]+ )+ del model.graph.node[:]+ model.graph.node.extend(nodes)+ return model+++def compact_recognizer_output(model):+ output = model.graph.output[0].name+ model.graph.initializer.extend(+ [+ numpy_helper.from_array(+ np.arange(18385, dtype=np.float32).reshape(1, 1, -1), "shared_positions"+ ),+ numpy_helper.from_array(+ np.array(18385, dtype=np.float32), "shared_sentinel"+ ),+ ]+ )+ model.graph.node.extend(+ [+ helper.make_node(+ "ReduceMax",+ [output],+ ["shared_score"],+ axes=[2],+ keepdims=1,+ name="shared_score",+ ),+ helper.make_node(+ "Sub", ["shared_score", output], ["shared_gap"], name="shared_gap"+ ),+ helper.make_node(+ "Ceil", ["shared_gap"], ["shared_nonwinner"], name="shared_nonwinner"+ ),+ helper.make_node(+ "Mul",+ ["shared_nonwinner", "shared_sentinel"],+ ["shared_penalty"],+ name="shared_penalty",+ ),+ helper.make_node(+ "Add",+ ["shared_positions", "shared_penalty"],+ ["shared_candidates"],+ name="shared_candidates",+ ),+ helper.make_node(+ "ReduceMin",+ ["shared_candidates"],+ ["shared_index"],+ axes=[2],+ keepdims=1,+ name="shared_index",+ ),+ helper.make_node(+ "Concat",+ ["shared_index", "shared_score"],+ ["shared_output"],+ axis=2,+ name="shared_output",+ ),+ ]+ )+ del model.graph.output[:]+ model.graph.output.append(+ helper.make_tensor_value_info(+ "shared_output", onnx.TensorProto.FLOAT, ["N", "T", 2]+ )+ )+ return remove_unused_nodes(model)+++def fold_detector_head(model):+ values = constant_arrays(model)+ nodes = list(model.graph.node)+ for node in nodes:+ if (+ node.op_type == "Reshape"+ and node.input[0] in values+ and (node.input[1] in values)+ ):+ values[node.output[0]] = values[node.input[0]].reshape(+ values[node.input[1]]+ )+ replacements = {}+ for ordinal, node in enumerate([n for n in nodes if n.op_type == "ConvTranspose"]):+ attrs = {a.name: helper.get_attribute_value(a) for a in node.attribute}+ assert attrs["group"] == 1 and attrs["kernel_shape"] == [2, 2]+ assert attrs["strides"] == [2, 2] and (not any(attrs["pads"]))+ add = next(n for n in nodes if node.output[0] in n.input and n.op_type == "Add")+ bias_input = next(v for v in add.input if v != node.output[0])+ weight = values[node.input[1]].astype(np.float64)+ bias = values[bias_input].reshape(-1).astype(np.float64)+ output = add.output[0]+ bn = next(+ (+ n+ for n in nodes+ if output in n.input and n.op_type == "BatchNormalization"+ ),+ None,+ )+ if bn is not None:+ attrs_bn = {a.name: helper.get_attribute_value(a) for a in bn.attribute}+ assert not attrs_bn.get("training_mode", 0)+ scale, offset, mean, variance = [+ values[v].astype(np.float64) for v in bn.input[1:]+ ]+ factor = scale / np.sqrt(variance + attrs_bn["epsilon"])+ weight *= factor[None, :, None, None]+ bias = (bias - mean) * factor + offset+ output = bn.output[0]+ weight_name, bias_name = (+ f"modelopt_deconv_weight_{ordinal}",+ f"modelopt_deconv_bias_{ordinal}",+ )+ fused = copy.deepcopy(node)+ fused.input[:] = [node.input[0], weight_name, bias_name]+ fused.output[:] = [output]+ replacements[node.name] = [fused]+ model.graph.initializer.extend(+ [+ numpy_helper.from_array(weight.astype(np.float32), weight_name),+ numpy_helper.from_array(bias.astype(np.float32), bias_name),+ ]+ )+ replacements[add.name] = []+ if bn is not None:+ replacements[bn.name] = []+ del model.graph.node[:]+ for node in nodes:+ model.graph.node.extend(replacements.get(node.name, [node]))+ return remove_unused_nodes(model)+++def rewrite_classifier_hardswish(model):+ values = constant_arrays(model)+ producers = {v: n for n in model.graph.node for v in n.output}+ nodes = []+ count = 0+ for node in model.graph.node:+ matched = False+ if (+ node.op_type == "Div"+ and node.input[1] in values+ and np.all(values[node.input[1]] == 6)+ ):+ multiply = producers.get(node.input[0])+ if multiply is not None and multiply.op_type == "Mul":+ for index in [0, 1]:+ clip = producers.get(multiply.input[index])+ source = multiply.input[1 - index]+ if clip is None or clip.op_type != "Clip" or len(clip.input) != 3:+ continue+ if (+ clip.input[1] not in values+ or clip.input[2] not in values+ or (not np.all(values[clip.input[1]] == 0))+ or (not np.all(values[clip.input[2]] == 6))+ ):+ continue+ add = producers.get(clip.input[0])+ if add is None or add.op_type != "Add" or source not in add.input:+ continue+ other = add.input[1 - list(add.input).index(source)]+ if other not in values or not np.all(values[other] == 3):+ continue+ gate = f"modelopt_cls_gate_{count}"+ nodes.extend(+ [+ helper.make_node(+ "HardSigmoid",+ [source],+ [gate],+ name=gate,+ alpha=1 / 6,+ beta=0.5,+ ),+ helper.make_node(+ "Mul",+ [source, gate],+ list(node.output),+ name=node.name + "_modelopt",+ ),+ ]+ )+ count += 1+ matched = True+ break+ if not matched:+ nodes.append(node)+ del model.graph.node[:]+ model.graph.node.extend(nodes)+ return (remove_unused_nodes(model), count)+++def load_sources(directory):+ models = {}+ directory.mkdir(parents=True, exist_ok=True)+ for name, expected in SOURCE_HASHES.items():+ path = directory / name+ if not path.exists():+ with urlopen(f"{SOURCE_BASE_URL}/{name}", timeout=120) as response:+ data = response.read()+ if hashlib.sha256(data).hexdigest() != expected:+ raise ValueError(f"Unexpected SHA-256 for downloaded {name}")+ path.write_bytes(data)+ if hashlib.sha256(path.read_bytes()).hexdigest() != expected:+ raise ValueError(f"Unexpected SHA-256 for {path}")+ if path.suffix == ".onnx":+ models[path.stem] = onnx.load(path)+ return models+++def optimize(models):+ recognition, _ = canonicalize_scalar_affines(copy.deepcopy(models["rec"]))+ recognition, folded = fold_input_affine(recognition)+ recognition, normalized = fuse_recognizer_layer_norms(recognition)+ if folded != 12 or normalized != 4:+ raise ValueError(+ f"Unexpected recognizer rewrites: folded={folded!r}, normalized={normalized!r}"+ )+ recognition = expand_hardswish(recognition)+ recognition = align_recognizer_projection(recognition, 18432)+ recognition = compact_recognizer_output(recognition)+ detection = fold_detector_head(copy.deepcopy(models["det"]))+ detection = expand_hardswish(detection)+ classification, _ = canonicalize_scalar_affines(copy.deepcopy(models["cls"]))+ classification, activations = rewrite_classifier_hardswish(classification)+ if activations != 18:+ raise ValueError(f"Unexpected classifier rewrites: activations={activations!r}")+ return {+ "det.onnx": detection,+ "cls.onnx": classification,+ "rec.onnx": recognition,+ }+++def main():+ parser = argparse.ArgumentParser()+ parser.add_argument("--source-dir", type=Path, required=True)+ parser.add_argument("--output-dir", type=Path, required=True)+ args = parser.parse_args()+ if onnx.__version__ != "1.22.0":+ raise ValueError("Reproducible model output requires onnx==1.22.0")+ sources = load_sources(args.source_dir)+ records = []+ for name, model in optimize(sources).items():+ onnx.checker.check_model(model)+ data = model.SerializeToString()+ digest = hashlib.sha256(data).hexdigest()+ if digest != OUTPUT_HASHES[name]:+ raise ValueError(+ f"Generated model differs from qualified artifact: {name} ({digest})"+ )+ path = args.output_dir / name+ path.parent.mkdir(parents=True, exist_ok=True)+ path.write_bytes(data)+ records.append(+ {+ "file": name,+ "sha256": digest,+ "bytes": len(data),+ "operators": sorted({node.op_type for node in model.graph.node}),+ "output": "FP32 [N,T,2]: first winning index, original winning probability"+ if name == "rec.onnx"+ else "unchanged",+ }+ )+ dictionary = args.source_dir / "ppocrv5_dict.txt"+ (args.output_dir / dictionary.name).write_bytes(dictionary.read_bytes())+ metadata = {+ "source_base_url": SOURCE_BASE_URL,+ "source_sha256": SOURCE_HASHES,+ "onnx": onnx.__version__,+ "models": records,+ }+ (args.output_dir / "ocr_model_manifest.json").write_text(+ json.dumps(metadata, indent=2) + "\n"+ )+ print(json.dumps(records, indent=2))+++if __name__ == "__main__":+ main()diff --git a/mobile/apps/photos/changes/ocr-gpu-internal.md b/mobile/apps/photos/changes/ocr-gpu-internal.mdnew file mode 100644index 0000000000..83b2b3d571--- /dev/null+++ b/mobile/apps/photos/changes/ocr-gpu-internal.md@@ -0,0 +1 @@+- Internal users can test GPU-accelerated text recognition on Android and iOS.diff --git a/mobile/apps/photos/lib/services/machine_learning/ocr/rust_ocr_backend.dart b/mobile/apps/photos/lib/services/machine_learning/ocr/rust_ocr_backend.dartindex d2744d2cfe..dfcde2461b 100644--- a/mobile/apps/photos/lib/services/machine_learning/ocr/rust_ocr_backend.dart+++ b/mobile/apps/photos/lib/services/machine_learning/ocr/rust_ocr_backend.dart@@ -5,12 +5,14 @@ import "package:path/path.dart" as p;import "package:path_provider/path_provider.dart";import "package:photos/services/machine_learning/ocr/ocr_backend.dart";import "package:photos/services/machine_learning/ocr/ocr_models.dart";+import "package:photos/services/machine_learning/webgpu_execution_policy.dart";+import "package:photos/src/rust/api/ml_indexing_api.dart";import "package:photos/src/rust/api/ocr_api.dart";import "package:synchronized/synchronized.dart";class RustOcrBackend implements OcrBackend {static final _logger = Logger("RustOcrBackend");- static const _modelVersion = "pp-ocrv5";+ static const _modelVersion = "pp-ocrv5-gpu-v1";final _engineLock = Lock();OcrEngine? _engine;@@ -50,6 +52,9 @@ class RustOcrBackend implements OcrBackend {return;}try {+ await setMlExecutionConfig(+ enableWebgpu: await webGpuExecutionPolicy.isEligible(),+ );_engine = await OcrEngine.create(assetsDir: assetsDir,includeRecognizer: includeRecognizer,diff --git a/mobile/apps/photos/lib/services/machine_learning/ocr_service.dart b/mobile/apps/photos/lib/services/machine_learning/ocr_service.dartindex fc0f05a1ec..82247bfd7d 100644--- a/mobile/apps/photos/lib/services/machine_learning/ocr_service.dart+++ b/mobile/apps/photos/lib/services/machine_learning/ocr_service.dart@@ -52,12 +52,9 @@ class OcrService {if (!_rustOcrEnabled()) {return OcrBackendKind.legacy;}- if (_isAndroid) {+ if (_isAndroid || _isIOS) {return OcrBackendKind.rust;}- if (_isIOS) {- return OcrBackendKind.vision;- }return OcrBackendKind.legacy;}diff --git a/rust/crates/ml/src/ocr/assets.rs b/rust/crates/ml/src/ocr/assets.rsindex 0466e7f6cd..5698d696a7 100644--- a/rust/crates/ml/src/ocr/assets.rs+++ b/rust/crates/ml/src/ocr/assets.rs@@ -6,40 +6,44 @@ use ente_assets::{Asset, AssetFile, AssetStore};use super::OcrModelPaths;use crate::error::{MlError, MlResult};-const MODEL_BASE_URL: &str = "https://models.ente.com/PP-OCRv5";const MODELS: &str = "models";struct OcrModelFile {key: &'static str,name: &'static str,+ url: &'static str,size: u64,sha256: &'static str,}const DETECTION: OcrModelFile = OcrModelFile {- key: "ppocrv5_det",+ key: "ppocrv5_det_gpu_v1",name: "det.onnx",- size: 4_748_769,- sha256: "d7fe3ea74652890722c0f4d02458b7261d9f5ae6c92904d05707c9eb155c7924",+ url: "https://entedevassets.priem.dev/det_opt.onnx",+ size: 4_750_896,+ sha256: "3e658f85236f1984e186070048d7cdc99ddbb84e12c91a6fd80922b85bd9487a",};const CLASSIFICATION: OcrModelFile = OcrModelFile {- key: "ppocrv5_cls",+ key: "ppocrv5_cls_gpu_v1",name: "cls.onnx",- size: 582_663,- sha256: "f4bb53707100c5f3d59ba834eb05bb400369f20aed35d4b26807b1bfadd2a70e",+ url: "https://entedevassets.priem.dev/cls_opt.onnx",+ size: 574_875,+ sha256: "8675903197a1bab7060e830e9bcc63d29c5cf7d372735338bd9501f829e94ed1",};const RECOGNITION: OcrModelFile = OcrModelFile {- key: "ppocrv5_rec",+ key: "ppocrv5_rec_gpu_v1",name: "rec.onnx",- size: 16_517_247,- sha256: "bf66820f48fa99f779974c4df78e5274a9d8e0458c4137e8c5357e40e2c3faf2",+ url: "https://entedevassets.priem.dev/rec_opt.onnx",+ size: 16_686_404,+ sha256: "667691bb173a94106b595aa87c7d176b53b68cf831b6ffa09569018754265e57",};const DICTIONARY: OcrModelFile = OcrModelFile {key: "ppocrv5_dict",name: "ppocrv5_dict.txt",+ url: "https://models.ente.com/PP-OCRv5/ppocrv5_dict.txt",size: 74_012,sha256: "d1979e9f794c464c0d2e0b70a7fe14dd978e9dc644c0e71f14158cdf8342af1b",};@@ -47,10 +51,6 @@ const DICTIONARY: OcrModelFile = OcrModelFile {const CATALOG: [&OcrModelFile; 4] = [&DETECTION, &CLASSIFICATION, &RECOGNITION, &DICTIONARY];impl OcrModelFile {- fn url(&self) -> String {- format!("{MODEL_BASE_URL}/{}", self.name)- }-fn asset(&self) -> Asset {#[expect(clippy::expect_used,@@ -60,7 +60,7 @@ impl OcrModelFile {vec![MODELS.to_string(), self.key.to_string()],AssetFile {name: self.name.to_string(),- url: self.url(),+ url: self.url.to_string(),size: self.size,sha256: self.sha256.to_string(),},@@ -111,7 +111,7 @@ pub async fn ensure_models("ocr models: downloading {} ({} bytes) from {}",file.name,file.size,- file.url()+ file.url);}let start = Instant::now();@@ -178,6 +178,26 @@ mod tests {assert!(!is_detector_downloaded(&store));}+ #[test]+ fn legacy_models_do_not_satisfy_the_optimized_catalog() {+ let root = tempfile::tempdir().unwrap();+ let store = AssetStore::new(root.path());+ for (model, legacy_key) in [+ (&DETECTION, "ppocrv5_det"),+ (&CLASSIFICATION, "ppocrv5_cls"),+ (&RECOGNITION, "ppocrv5_rec"),+ ] {+ let legacy = OcrModelFile {+ key: legacy_key,+ ..*model+ };+ cache_model(&store, &legacy);+ assert!(!store.is_downloaded(&model.asset()));+ assert_ne!(model.path(&store), legacy.path(&store));+ }+ assert!(!is_detector_downloaded(&store));+ }+#[tokio::test]async fn cached_detector_can_be_prepared_without_recognition_models() {let root = tempfile::tempdir().unwrap();diff --git a/rust/crates/ml/src/ocr/classify.rs b/rust/crates/ml/src/ocr/classify.rsindex d6e1fd06d3..b68a3db4b4 100644--- a/rust/crates/ml/src/ocr/classify.rs+++ b/rust/crates/ml/src/ocr/classify.rs@@ -1,12 +1,12 @@use std::sync::{Mutex, PoisonError};+use super::session::{OcrModel, OcrSession};use super::tensor::{BgrNormalization, prepare_crops, write_bgr_planes};use crate::cv;use crate::cv::image::ImageU8;use crate::error::{MlError, MlResult};-use crate::onnx::{ExecutionMode, OnnxSession, PreparedF32Input, SessionRunError, run_f32};+use crate::onnx::{PreparedF32Input, SessionRunError, run_f32};-const MODEL_NAMESPACE: &str = "ocr-classification";const BATCH_SIZE: usize = 6;const INPUT_HEIGHT: i32 = 48;const INPUT_WIDTH: i32 = 192;@@ -32,17 +32,13 @@ pub(crate) struct AngleDecision {}pub(crate) struct AngleClassifier {- session: Mutex<OnnxSession>,+ session: Mutex<OcrSession>,}impl AngleClassifier {pub(crate) fn new(model_path: &str) -> Self {Self {- session: Mutex::new(OnnxSession::new(- model_path,- MODEL_NAMESPACE,- ExecutionMode::CpuOnly,- )),+ session: Mutex::new(OcrSession::new(model_path, OcrModel::Classification)),}}@@ -55,12 +51,9 @@ impl AngleClassifier {let count = batch.len() as i64;let expected_shape = [count, CLASS_COUNT as i64];let mut session = self.session.lock().unwrap_or_else(PoisonError::into_inner);- let (values, _usage) = session.run(|session| {- let (shape, values) = run_f32(- session,- &input,- [count, 3, i64::from(INPUT_HEIGHT), i64::from(INPUT_WIDTH)],- )?;+ let input_shape = [count, 3, i64::from(INPUT_HEIGHT), i64::from(INPUT_WIDTH)];+ let (values, _usage) = session.run(input_shape, |session| {+ let (shape, values) = run_f32(session, &input, input_shape)?;if shape != expected_shape {return Err(SessionRunError::from(MlError::CorruptModel(format!("angle classifier produced output shape {shape:?}, expected {expected_shape:?}"diff --git a/rust/crates/ml/src/ocr/detect.rs b/rust/crates/ml/src/ocr/detect.rsindex 10f1216234..cf440933ea 100644--- a/rust/crates/ml/src/ocr/detect.rs+++ b/rust/crates/ml/src/ocr/detect.rs@@ -4,13 +4,13 @@ use super::Point;use super::geometry::{clip_to_bounds, mean_inside_quad, min_area_rect, min_edge, order_corners, scale_points, unclip,};+use super::session::{OcrModel, OcrSession};use super::tensor::{BgrNormalization, write_bgr_planes};use crate::cv;use crate::cv::image::{Contour, ImageU8};use crate::error::{MlError, MlResult};-use crate::onnx::{ExecutionMode, OnnxSession, PreparedF32Input, SessionRunError, run_f32};+use crate::onnx::{PreparedF32Input, SessionRunError, run_f32};-const MODEL_NAMESPACE: &str = "ocr-detection";const MAX_INPUT_SIDE: i32 = 960;const INPUT_STRIDE: i32 = 32;const BITMAP_THRESHOLD: f32 = 0.3;@@ -40,17 +40,13 @@ pub(crate) struct TextDetection {}pub(crate) struct TextDetector {- session: Mutex<OnnxSession>,+ session: Mutex<OcrSession>,}impl TextDetector {pub(crate) fn new(model_path: &str) -> Self {Self {- session: Mutex::new(OnnxSession::new(- model_path,- MODEL_NAMESPACE,- ExecutionMode::CpuOnly,- )),+ session: Mutex::new(OcrSession::new(model_path, OcrModel::Detection)),}}@@ -82,9 +78,9 @@ impl TextDetector {fn infer(&self, input: &PreparedF32Input, width: i32, height: i32) -> MlResult<Vec<f32>> {let expected_shape = [1i64, 1, i64::from(height), i64::from(width)];let mut session = self.session.lock().unwrap_or_else(PoisonError::into_inner);- let (values, _usage) = session.run(|session| {- let (shape, values) =- run_f32(session, input, [1, 3, i64::from(height), i64::from(width)])?;+ let input_shape = [1, 3, i64::from(height), i64::from(width)];+ let (values, _usage) = session.run(input_shape, |session| {+ let (shape, values) = run_f32(session, input, input_shape)?;if shape != expected_shape {return Err(SessionRunError::from(MlError::CorruptModel(format!("text detector produced output shape {shape:?}, expected {expected_shape:?}"diff --git a/rust/crates/ml/src/ocr/mod.rs b/rust/crates/ml/src/ocr/mod.rsindex 4021d58980..ddaf78c382 100644--- a/rust/crates/ml/src/ocr/mod.rs+++ b/rust/crates/ml/src/ocr/mod.rs@@ -7,6 +7,7 @@ mod detect;mod dictionary;mod geometry;mod recognize;+mod session;mod source;mod tensor;diff --git a/rust/crates/ml/src/ocr/recognize.rs b/rust/crates/ml/src/ocr/recognize.rsindex 2772c5a694..b0dec5eff8 100644--- a/rust/crates/ml/src/ocr/recognize.rs+++ b/rust/crates/ml/src/ocr/recognize.rs@@ -9,11 +9,10 @@ use crate::cv;use crate::cv::image::ImageU8;use crate::error::{MlError, MlResult};use crate::onnx::{- BorrowedFloatTensor, ExecutionMode, FloatTensorData, OnnxSession, PreparedF32Input,- with_prepared_float_output,+ BorrowedFloatTensor, FloatTensorData, PreparedF32Input, with_prepared_float_output,};-const MODEL_NAMESPACE: &str = "ocr-recognition";+use super::session::{OcrModel, OcrSession};const REC_VOCABULARY_SIZE: usize = 18385;const REC_HEIGHT: i32 = 48;const REC_BASE_WIDTH: i32 = 320;@@ -61,18 +60,14 @@ impl LazyDictionary {}pub(crate) struct TextRecognizer {- session: Mutex<OnnxSession>,+ session: Mutex<OcrSession>,dictionary: LazyDictionary,}impl TextRecognizer {pub(crate) fn new(model_path: &str, dictionary_path: &str) -> Self {Self {- session: Mutex::new(OnnxSession::new(- model_path,- MODEL_NAMESPACE,- ExecutionMode::CpuOnly,- )),+ session: Mutex::new(OcrSession::new(model_path, OcrModel::Recognition)),dictionary: LazyDictionary::new(dictionary_path),}}@@ -106,7 +101,7 @@ impl TextRecognizer {i64::from(layout.target_width),];let mut session = self.session.lock().unwrap_or_else(PoisonError::into_inner);- let (recognized, _usage) = session.run(|session| {+ let (recognized, _usage) = session.run(input_shape, |session| {with_prepared_float_output(session, &input, input_shape, |shape, values| match values {BorrowedFloatTensor::F32(values) => {decode_output(shape, values, layout, dictionary)@@ -129,20 +124,59 @@ fn decode_output<'a, T>(where&'a [T]: FloatTensorData,{- let output = SequenceOutput::new(shape, values, layout.content_widths.len(), dictionary.len())?;- Ok(layout+ let compact = shape.get(2) == Some(&2);+ let values_per_step = if compact { 2 } else { dictionary.len() };+ let output = SequenceOutput::new(shape, values, layout.content_widths.len(), values_per_step)?;+ layout.content_widths.iter().zip(output.sequences()).map(|(&content_width, logits)| {- ctc_decode(- logits,- output.vocabulary,- dictionary,- layout.padding_scale(content_width),- )+ let padding_scale = layout.padding_scale(content_width);+ if compact {+ let best = compact_steps(logits, dictionary.len())?;+ Ok(decode_steps(&best, dictionary, padding_scale))+ } else {+ Ok(ctc_decode(+ logits,+ output.vocabulary,+ dictionary,+ padding_scale,+ ))+ }})- .collect())+ .collect()+}++fn compact_steps<'a, T>(values: &'a [T], vocabulary: usize) -> MlResult<Vec<StepBest>>+where+ &'a [T]: FloatTensorData,+{+ values+ .as_chunks::<2>()+ .0+ .iter()+ .map(|step| {+ let step = step.as_slice();+ let index = step.value(0);+ let probability = step.value(1);+ if !index.is_finite()+ || index < 0.0+ || index >= vocabulary as f32+ || index.fract() != 0.0+ || !probability.is_finite()+ || probability < 0.0+ {+ return Err(MlError::CorruptModel(format!(+ "text recognizer produced invalid index/probability [{index}, {probability}]"+ )));+ }+ Ok(StepBest {+ index: index as usize,+ probability,+ })+ })+ .collect()}fn recognize_in_batches(@@ -354,7 +388,11 @@ where&'a [T]: FloatTensorData,{let best = best_per_step(logits, vocabulary);- let spans: Vec<CharacterSpan> = character_runs(&best)+ decode_steps(&best, dictionary, padding_scale)+}++fn decode_steps(best: &[StepBest], dictionary: &[String], padding_scale: f32) -> Recognition {+ let spans: Vec<CharacterSpan> = character_runs(best).iter().filter_map(|run| run.span(best.len(), dictionary, padding_scale)).collect();@@ -534,6 +572,55 @@ mod tests {assert_eq!(recognition, Recognition::default());}+ #[test]+ fn compact_output_preserves_text_confidences_and_character_spans() {+ let sequences = [+ [(1, 0.9), (1, 0.7), (0, 0.8), (1, 0.6), (2, 0.5), (3, 0.9)],+ [(0, 0.9), (0, 0.8), (0, 0.7), (0, 0.6), (0, 0.5), (0, 0.9)],+ ];+ let full: Vec<_> = sequences+ .iter()+ .flat_map(|steps| logits(4, steps))+ .collect();+ let compact: Vec<_> = sequences+ .iter()+ .flatten()+ .flat_map(|&(index, probability)| [index as f32, probability])+ .collect();+ let layout = BatchLayout {+ target_width: 320,+ content_widths: vec![160, 320],+ };+ let expected = decode_output(&[2, 6, 4], &full, &layout, &dictionary()).unwrap();+ let actual = decode_output(&[2, 6, 2], &compact, &layout, &dictionary()).unwrap();+ assert_eq!(actual, expected);+ assert_eq!(actual[0].text, "aabc");+ assert_eq!(actual[1], Recognition::default());+ }++ #[test]+ fn compact_output_rejects_invalid_indices_probabilities_and_shapes() {+ let layout = BatchLayout {+ target_width: 320,+ content_widths: vec![320],+ };+ for values in [+ [-1.0, 0.9],+ [4.0, 0.9],+ [1.5, 0.9],+ [f32::NAN, 0.9],+ [f32::INFINITY, 0.9],+ [1.0, f32::NAN],+ [1.0, f32::INFINITY],+ [1.0, -0.1],+ ] {+ assert!(decode_output(&[1, 1, 2], &values, &layout, &dictionary()).is_err());+ }+ for shape in [[2, 1, 2], [1, 0, 2], [1, 2, 2]] {+ assert!(decode_output(&shape, &[1.0, 0.9], &layout, &dictionary()).is_err());+ }+ }+#[test]fn repeats_separated_by_a_blank_produce_two_characters() {let steps = [(1, 0.9), (1, 0.9), (0, 0.9), (1, 0.6)];diff --git a/rust/crates/ml/src/ocr/session.rs b/rust/crates/ml/src/ocr/session.rsnew file mode 100644index 0000000000..71c7c39cdc--- /dev/null+++ b/rust/crates/ml/src/ocr/session.rs@@ -0,0 +1,168 @@+use std::collections::VecDeque;++use crate::error::{MlError, MlResult};+use crate::onnx::{+ ExecutionMode, GpuOptions, OnnxSession, ProviderUsage, SessionHandle, SessionRunResult,+};++#[derive(Clone, Copy)]+pub(super) enum OcrModel {+ Detection,+ Classification,+ Recognition,+}++impl OcrModel {+ fn cache_capacity(self) -> usize {+ match self {+ Self::Detection => 2,+ Self::Classification => 6,+ Self::Recognition => 8,+ }+ }++ fn namespace(self) -> &'static str {+ match self {+ Self::Detection => "ocr-detection",+ Self::Classification => "ocr-classification",+ Self::Recognition => "ocr-recognition",+ }+ }++ fn mode(self) -> ExecutionMode {+ if matches!(self, Self::Classification)+ && !cfg!(any(target_os = "ios", target_os = "macos"))+ {+ ExecutionMode::CpuOnly+ } else {+ ExecutionMode::GpuPreferred+ }+ }++ fn gpu_options(self, [count, _, height, width]: [i64; 4]) -> GpuOptions {+ let dimensions = match self {+ Self::Detection => vec![+ ("DynamicDimension.0", count),+ ("DynamicDimension.1", height),+ ("DynamicDimension.2", width),+ ],+ Self::Classification => vec![+ ("p2o.DynamicDimension.0", count),+ ("p2o.DynamicDimension.1", height),+ ("p2o.DynamicDimension.2", width),+ ],+ Self::Recognition => vec![("DynamicDimension.0", count), ("DynamicDimension.1", width)],+ };+ GpuOptions {+ dimensions,+ #[cfg(any(target_os = "android", target_os = "linux", target_os = "windows"))]+ prefer_nhwc: matches!(self, Self::Detection),+ }+ }+}++pub(super) struct OcrSession {+ model_path: String,+ model: OcrModel,+ sessions: VecDeque<([i64; 4], OnnxSession)>,+}++impl OcrSession {+ pub(super) fn new(model_path: &str, model: OcrModel) -> Self {+ Self {+ model_path: model_path.to_owned(),+ model,+ sessions: VecDeque::new(),+ }+ }++ pub(super) fn run<T>(+ &mut self,+ shape: [i64; 4],+ operation: impl FnMut(&mut SessionHandle) -> SessionRunResult<T>,+ ) -> MlResult<(T, ProviderUsage)> {+ self.session_for(shape)?.run(operation)+ }++ fn session_for(&mut self, shape: [i64; 4]) -> MlResult<&mut OnnxSession> {+ let mode = self.model.mode();+ let key = if mode == ExecutionMode::CpuOnly {+ [0; 4]+ } else {+ shape+ };+ if let Some(index) = self.sessions.iter().position(|(cached, _)| *cached == key) {+ self.sessions.make_contiguous()[..=index].rotate_right(1);+ } else {+ if self.sessions.len() == self.model.cache_capacity() {+ self.sessions.pop_back();+ }+ let [n, c, h, w] = key;+ let namespace = format!("{}-{n}x{c}x{h}x{w}-gpu-f32-v1", self.model.namespace());+ let mut session = OnnxSession::new(&self.model_path, &namespace, mode);+ if mode == ExecutionMode::GpuPreferred {+ session = session+ .with_gpu_options(self.model.gpu_options(shape))+ .with_unvalidated_acceleration();+ }+ self.sessions.push_front((key, session));+ }+ self.sessions+ .front_mut()+ .map(|(_, session)| session)+ .ok_or_else(|| MlError::Runtime("OCR session cache is empty".to_owned()))+ }+}++#[cfg(test)]+mod tests {+ use super::*;++ #[test]+ fn reuses_shapes_and_evicts_the_least_recently_used_session() {+ let mut cache = OcrSession::new("rec.onnx", OcrModel::Recognition);+ let first = [6, 3, 48, 320];+ cache.session_for(first).unwrap().initialize_load_state();+ for width in 321..328 {+ cache.session_for([6, 3, 48, width]).unwrap();+ }+ assert!(cache.session_for(first).unwrap().has_load_state());+ cache.session_for([1, 3, 48, 320]).unwrap();+ assert_eq!(cache.sessions.len(), OcrModel::Recognition.cache_capacity());+ assert!(cache.session_for(first).unwrap().has_load_state());+ assert!(+ !cache+ .sessions+ .iter()+ .any(|(shape, _)| *shape == [6, 3, 48, 321])+ );+ }++ #[test]+ fn models_override_their_actual_spatial_dimensions() {+ assert_eq!(+ OcrModel::Detection.gpu_options([1, 3, 960, 704]).dimensions,+ vec![+ ("DynamicDimension.0", 1),+ ("DynamicDimension.1", 960),+ ("DynamicDimension.2", 704)+ ]+ );+ assert_eq!(+ OcrModel::Recognition+ .gpu_options([3, 3, 48, 641])+ .dimensions,+ vec![("DynamicDimension.0", 3), ("DynamicDimension.1", 641)]+ );+ assert_eq!(+ OcrModel::Classification+ .gpu_options([3, 3, 48, 192])+ .dimensions,+ vec![+ ("p2o.DynamicDimension.0", 3),+ ("p2o.DynamicDimension.1", 48),+ ("p2o.DynamicDimension.2", 192)+ ]+ );+ }+}diff --git a/rust/crates/ml/src/onnx.rs b/rust/crates/ml/src/onnx.rsindex ae18754305..6e2255895e 100644--- a/rust/crates/ml/src/onnx.rs+++ b/rust/crates/ml/src/onnx.rs@@ -24,6 +24,13 @@ pub(crate) use tensor::{use providers::{ExecutionProvider, ProviderPlan};+#[derive(Clone, Debug)]+pub(crate) struct GpuOptions {+ pub(crate) dimensions: Vec<(&'static str, i64)>,+ #[cfg(any(target_os = "android", target_os = "linux", target_os = "windows"))]+ pub(crate) prefer_nhwc: bool,+}+#[derive(Clone, Copy, Debug, PartialEq, Eq)]enum AccelerationValidation {GoldenRequired,@@ -58,6 +65,7 @@ pub(crate) struct OnnxSession {model_namespace: String,mode: ExecutionMode,validation: AccelerationValidation,+ gpu_options: Option<GpuOptions>,provider_plan: Option<ProviderPlan>,session: Option<(Session, ExecutionProvider)>,first_run_canary: Option<webgpu::ArmedCanary>,@@ -70,6 +78,7 @@ impl OnnxSession {model_namespace: model_namespace.to_string(),mode,validation: AccelerationValidation::GoldenRequired,+ gpu_options: None,provider_plan: None,session: None,first_run_canary: None,@@ -83,6 +92,11 @@ impl OnnxSession {self}+ pub(crate) fn with_gpu_options(mut self, options: GpuOptions) -> Self {+ self.gpu_options = Some(options);+ self+ }+pub(crate) fn model_path(&self) -> &str {&self.model_path}@@ -152,8 +166,13 @@ impl OnnxSession {let model_name = model_file_label(model_path);log::info!("loading {model_name} with {:?} execution", self.mode);let started_at = std::time::Instant::now();- let (loaded, execution_provider) =- build_next_session(model_path, provider_plan, model_namespace, self.validation)?;+ let (loaded, execution_provider) = build_next_session(+ model_path,+ provider_plan,+ model_namespace,+ self.validation,+ self.gpu_options.as_ref(),+ )?;log::info!("loaded {model_name} with {execution_provider:?} in {:?}",started_at.elapsed()@@ -276,9 +295,15 @@ fn build_next_session(plan: &mut ProviderPlan,model_namespace: &str,validation: AccelerationValidation,+ gpu_options: Option<&GpuOptions>,) -> MlResult<(LoadedSession, ExecutionProvider)> {let result = providers::run_provider_plan(plan, |execution_provider| {- let attempt = providers::provider_attempt(execution_provider, model_path, model_namespace);+ let attempt = providers::provider_attempt(+ execution_provider,+ model_path,+ model_namespace,+ gpu_options,+ );if attempt.execution_provider() == ExecutionProvider::WebGpu {#[cfg(any(target_os = "android", target_os = "linux", target_os = "windows"))]{@@ -287,6 +312,7 @@ fn build_next_session(model_namespace,attempt,validation,+ gpu_options,);}#[cfg(not(any(target_os = "android", target_os = "linux", target_os = "windows")))]@@ -294,7 +320,7 @@ fn build_next_session(}let coreml_cache_dir = attempt.coreml_cache_dir().map(Path::to_path_buf);- match build_and_validate_session(model_path, attempt, validation) {+ match build_and_validate_session(model_path, attempt, validation, gpu_options) {Ok(session) => {if let Some(cache_dir) = coreml_cache_dir {coreml_cache::finalize(&cache_dir, model_path);@@ -348,6 +374,7 @@ fn build_cpu_session(model_path: &str) -> MlResult<Session> {&mut plan,"golden-tooling",AccelerationValidation::GoldenRequired,+ None,).map(|(loaded, _)| loaded.session)}@@ -359,6 +386,7 @@ fn build_and_validate_session(model_path: &str,attempt: providers::ProviderAttempt,_validation: AccelerationValidation,+ gpu_options: Option<&GpuOptions>,) -> MlResult<Session> {#[cfg(any(target_os = "android",@@ -369,7 +397,7 @@ fn build_and_validate_session())]let execution_provider = attempt.execution_provider();- let session = match providers::build_session(model_path, attempt) {+ let session = match providers::build_session(model_path, attempt, gpu_options) {Ok(session) => session,Err(error) => {#[cfg(any(@@ -410,6 +438,7 @@ fn build_webgpu_session_with_canary(model_namespace: &str,attempt: providers::ProviderAttempt,validation: AccelerationValidation,+ gpu_options: Option<&GpuOptions>,) -> MlResult<LoadedSession> {// Fail closed: without a durable failure record, a crash during the// attempt would go unnoticed and the crash loop protection would be lost.@@ -448,7 +477,7 @@ fn build_webgpu_session_with_canary(}}}- let mut session = match providers::build_session(model_path, attempt) {+ let mut session = match providers::build_session(model_path, attempt, gpu_options) {Ok(session) => session,Err(error) => {record_provider_attempt_failure(diff --git a/rust/crates/ml/src/onnx/providers.rs b/rust/crates/ml/src/onnx/providers.rsindex 0f85ef0cbf..596cdf04d8 100644--- a/rust/crates/ml/src/onnx/providers.rs+++ b/rust/crates/ml/src/onnx/providers.rs@@ -39,7 +39,7 @@ use super::golden_test;use super::model_file_label;#[cfg(any(target_os = "android", target_os = "linux", target_os = "windows"))]use super::webgpu;-use super::{AccelerationValidation, session_load_error};+use super::{AccelerationValidation, GpuOptions, session_load_error};use crate::error::MlResult;#[cfg(any(target_os = "ios", target_os = "macos"))]@@ -48,6 +48,7 @@ const ENABLE_PERSISTENT_COREML_CACHE: bool = true;#[derive(Clone, Copy, Debug, PartialEq, Eq)]pub(crate) enum ExecutionMode {PlatformDefault,+ GpuPreferred,CpuOnly,}@@ -57,10 +58,6 @@ pub(crate) enum ExecutionMode {pub(super) enum ExecutionProvider {CoreMl,WebGpu,- #[cfg_attr(- not(any(test, target_os = "android")),- expect(dead_code, reason = "XNNPACK is only selected on Android")- )]Xnnpack,Cpu,}@@ -79,6 +76,10 @@ impl ProviderPlan {) -> Self {let providers = match mode {ExecutionMode::PlatformDefault => platform_default_providers(model_path, validation),+ ExecutionMode::GpuPreferred => platform_default_providers(model_path, validation)+ .into_iter()+ .filter(|provider| *provider != ExecutionProvider::Xnnpack)+ .collect(),ExecutionMode::CpuOnly => vec![ExecutionProvider::Cpu],};Self::from_providers(providers)@@ -149,18 +150,23 @@ pub(super) fn provider_attempt(provider: ExecutionProvider,_model_path: &str,_model_namespace: &str,+ _gpu_options: Option<&GpuOptions>,) -> ProviderAttempt {match provider {ExecutionProvider::Cpu => ProviderAttempt::cpu_only(),#[cfg(any(target_os = "ios", target_os = "macos"))]ExecutionProvider::CoreMl => {let (coreml_provider, coreml_cache_dir) =- coreml_provider(_model_path, _model_namespace);+ coreml_provider(_model_path, _model_namespace, _gpu_options.is_some());ProviderAttempt {- providers: vec![- coreml_provider,- CPU::default().with_arena_allocator(true).build(),- ],+ providers: if _gpu_options.is_some() {+ vec![coreml_provider]+ } else {+ vec![+ coreml_provider,+ CPU::default().with_arena_allocator(true).build(),+ ]+ },disable_intra_op_spinning: false,coreml_cache_dir,execution_provider: ExecutionProvider::CoreMl,@@ -168,7 +174,11 @@ pub(super) fn provider_attempt(}#[cfg(any(target_os = "android", target_os = "linux", target_os = "windows"))]ExecutionProvider::WebGpu => ProviderAttempt {- providers: webgpu_attempt_providers(),+ providers: if let Some(options) = _gpu_options {+ vec![webgpu_provider(options.prefer_nhwc)]+ } else {+ webgpu_attempt_providers()+ },disable_intra_op_spinning: true,coreml_cache_dir: None,execution_provider: ExecutionProvider::WebGpu,@@ -179,12 +189,28 @@ pub(super) fn provider_attempt(}}-pub(super) fn build_session(model_path: &str, attempt: ProviderAttempt) -> MlResult<Session> {+pub(super) fn build_session(+ model_path: &str,+ attempt: ProviderAttempt,+ gpu_options: Option<&GpuOptions>,+) -> MlResult<Session> {let mut builder = Session::builder()?.with_optimization_level(GraphOptimizationLevel::All)?.with_intra_threads(1)?.with_inter_threads(1)?;+ if let Some(options) = gpu_options+ && matches!(+ attempt.execution_provider,+ ExecutionProvider::CoreMl | ExecutionProvider::WebGpu+ )+ {+ builder = builder.with_disable_cpu_fallback()?;+ for &(name, value) in &options.dimensions {+ builder = builder.with_dimension_override(name, value)?;+ }+ }+if attempt.disable_intra_op_spinning {builder = builder.with_intra_op_spinning(false)?;}@@ -284,12 +310,20 @@ fn accelerated_provider_allowed(fn coreml_provider(model_path: &str,model_namespace: &str,+ full_gpu: bool,) -> (ExecutionProviderDispatch, Option<PathBuf>) {let mut provider = CoreML::default().with_model_format(ModelFormat::MLProgram).with_compute_units(ComputeUnits::All).with_specialization_strategy(SpecializationStrategy::Default);+ if full_gpu {+ provider = provider+ .with_compute_units(ComputeUnits::CPUAndGPU)+ .with_low_precision_accumulation_on_gpu(false)+ .with_static_input_shapes(true);+ }+let mut prepared_cache_dir = None;if ENABLE_PERSISTENT_COREML_CACHE {match coreml_cache::prepare_directory(model_path, model_namespace) {@@ -312,8 +346,13 @@ fn coreml_provider(}#[cfg(any(target_os = "android", target_os = "linux", target_os = "windows"))]-fn webgpu_provider() -> ExecutionProviderDispatch {- let provider = WebGPU::default().with_preferred_layout(PreferredLayout::NCHW);+fn webgpu_provider(prefer_nhwc: bool) -> ExecutionProviderDispatch {+ let layout = if prefer_nhwc {+ PreferredLayout::NHWC+ } else {+ PreferredLayout::NCHW+ };+ let provider = WebGPU::default().with_preferred_layout(layout);#[cfg(any(target_os = "android", target_os = "linux"))]let provider = provider.with_dawn_backend_type(DawnBackendType::Vulkan);#[cfg(target_os = "windows")]@@ -324,7 +363,7 @@ fn webgpu_provider() -> ExecutionProviderDispatch {#[cfg(target_os = "android")]fn webgpu_attempt_providers() -> Vec<ExecutionProviderDispatch> {vec![- webgpu_provider(),+ webgpu_provider(false),xnnpack_provider().fail_silently(),CPU::default().with_arena_allocator(true).build(),]@@ -333,7 +372,7 @@ fn webgpu_attempt_providers() -> Vec<ExecutionProviderDispatch> {#[cfg(any(target_os = "linux", target_os = "windows"))]fn webgpu_attempt_providers() -> Vec<ExecutionProviderDispatch> {vec![- webgpu_provider(),+ webgpu_provider(false),CPU::default().with_arena_allocator(true).build(),]}@@ -366,6 +405,28 @@ mod tests {AccelerationValidation, ExecutionMode, ExecutionProvider, ProviderPlan, run_provider_plan,};+ #[test]+ fn gpu_preferred_falls_back_directly_to_cpu() {+ let mut plan = ProviderPlan::new(+ ExecutionMode::GpuPreferred,+ "model.onnx",+ AccelerationValidation::Unvalidated,+ );+ let mut attempted = Vec::new();+ let selected = run_provider_plan(&mut plan, |provider| {+ attempted.push(provider);+ if provider == ExecutionProvider::Cpu {+ Ok(provider)+ } else {+ Err(())+ }+ })+ .unwrap();+ assert_eq!(selected, (ExecutionProvider::Cpu, ExecutionProvider::Cpu));+ assert!(!attempted.contains(&ExecutionProvider::Xnnpack));+ assert!(!plan.has_fallback());+ }+#[test]fn construction_falls_through_and_selects_the_successful_provider() {let mut plan = accelerated_provider_plan();